2024-09-13T13:53:57.202 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-13T13:53:57.228 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-13T13:53:57.331 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/akupczyk-2024-09-13_10:00:18-upgrade-wip-aclamk-cbt-combined-distro-default-smithi/7903539 branch: wip-aclamk-cbt-combined description: upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} email: akupczyk@redhat.com first-half-sequence: - cephadm.shell: env: - sha1 mon.a: - ceph config set mgr mgr/cephadm/daemon_cache_timeout 60 - ceph config set global log_to_journald false --force - echo wait for mgr daemons to upgrade - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - echo wait for minority of mons to upgrade - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --limit 1 - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - sleep 60 - echo wait for majority of mons to upgrade - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --limit 1 - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - sleep 60 - echo wait for all mons to upgrade - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - sleep 60 - echo wait for half of osds to upgrade - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types osd --limit 4 - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done first-half-tasks: - workunit: branch: reef clients: client.0: - rbd/import_export.sh env: RBD_CREATE_ARGS: --new-format - print: '**** done rbd/import_export.sh 5-workload' first_in_suite: false job_id: '7903539' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'run basic import/export cli tests for rbd ' - desc: 'run randomized correctness test for rados operations generate write load with rados bench ' - desc: 'run basic cls tests for rbd ' - desc: 'run basic import/export cli tests for rbd ' - desc: 'librbd C and C++ api tests ' - desc: 'randomized correctness test for rados operations on a replicated pool, using only reads, writes, and deletes ' - desc: 'randomized correctness test for rados operations on a replicated pool with snapshot operations ' - desc: 'run basic import/export cli tests for rbd ' name: akupczyk-2024-09-13_10:00:18-upgrade-wip-aclamk-cbt-combined-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-aclamk-cbt-combined ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true create_rbd_pool: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - mons down - mon down - MON_DOWN - out of quorum - PG_AVAILABILITY sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-aclamk-cbt-combined sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 owner: scheduled_akupczyk@teuthology priority: 160 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b second-half-sequence: sequential: - cephadm.shell: env: - sha1 mon.a: - sleep 60 - echo wait for upgrade to complete - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - echo upgrade complete - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 second-half-tasks: - workunit: branch: reef clients: client.0: - rbd/import_export.sh env: RBD_CREATE_ARGS: --new-format - print: '**** done rbd/import_export.sh 5-workload' seed: 6545 sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 sleep_before_teardown: 0 stress-tasks: - thrashosds: aggressive_pg_num_changes: false chance_force_recovery: 0 chance_pgnum_grow: 1 chance_pgpnum_fix: 1 chance_thrash_cluster_full: 0 chance_thrash_pg_upmap: 0 chance_thrash_pg_upmap_items: 0 disable_objectstore_tool_tests: true timeout: 1200 - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - print: '**** done end radosbench.yaml' - workunit: branch: reef clients: client.0: - cls/test_cls_rbd.sh - print: '**** done cls/test_cls_rbd.sh 5-workload' - workunit: branch: reef clients: client.0: - rbd/import_export.sh env: RBD_CREATE_ARGS: --new-format - print: '**** done rbd/import_export.sh 5-workload' - workunit: branch: reef clients: client.0: - rbd/test_librbd.sh env: RBD_FEATURES: '61' - print: '**** done rbd/test_librbd.sh 7-workload' - full_sequential: - rados: clients: - client.0 objects: 500 op_weights: delete: 10 read: 45 write: 45 ops: 4000 write_append_excl: false - print: '**** done rados/readwrite 5-workload' - full_sequential: - rados: clients: - client.0 objects: 50 op_weights: delete: 50 read: 100 rollback: 50 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 write_append_excl: false - print: '**** done rados/snaps-few-objects 5-workload' subset: 682/13280 suite: upgrade suite_branch: wip-aclamk-cbt-combined suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 targets: smithi102.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPd/Xz53IbINORkexgwTyDbQNYddGMvI0XigZZfmjUXY9/enVhdsuBZr49MJPieHS0EQTlgT440k29HJeLmdk4M= smithi195.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLqy+zsGMRFjGVQ+Ul4AuGa2NnkrmaWA6vETJzxcdQrsmKjoGHpMYTmGdk7GM/DlGMprmk9pCbj7r9rnrOEYadU= tasks: - pexec: all: - sudo dnf remove nvme-cli -y - sudo dnf install runc nvmetcli nvme-cli -y - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf - install: branch: reef exclude_packages: - ceph-volume - cephadm: compiled_cephadm_branch: reef conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:reef - cephadm.shell: mon.a: - ceph fs volume create foo - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph.healthy: null - print: '**** upgrading first half of cluster, with stress ****' - parallel: - first-half-tasks - first-half-sequence - print: '**** done upgrading first half of cluster ****' - ceph.healthy: null - print: '**** applying stress + thrashing to mixed-version cluster ****' - parallel: - stress-tasks - ceph.healthy: null - print: '**** finishing upgrade ****' - parallel: - second-half-tasks - second-half-sequence - ceph.healthy: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-13_10:00:18 tube: smithi user: akupczyk verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-13T13:53:57.331 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa; will attempt to use it 2024-09-13T13:53:57.332 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa/tasks 2024-09-13T13:53:57.332 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-13T13:53:57.333 INFO:teuthology.task.internal:Checking packages... 2024-09-13T13:53:57.353 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd58f73069a8157a9ace12a0247ee3e8497707ea9' 2024-09-13T13:53:57.353 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-13T13:53:57.353 INFO:teuthology.packaging:ref: None 2024-09-13T13:53:57.354 INFO:teuthology.packaging:tag: None 2024-09-13T13:53:57.354 INFO:teuthology.packaging:branch: wip-aclamk-cbt-combined 2024-09-13T13:53:57.354 INFO:teuthology.packaging:sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T13:53:57.354 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-aclamk-cbt-combined 2024-09-13T13:53:57.505 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4697.gd58f7306 2024-09-13T13:53:57.506 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-13T13:53:57.514 INFO:teuthology.task.internal:no buildpackages task found 2024-09-13T13:53:57.514 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-13T13:53:57.558 INFO:teuthology.task.internal:Saving configuration 2024-09-13T13:53:57.591 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-13T13:53:57.599 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-13T13:53:57.633 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi102.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/akupczyk-2024-09-13_10:00:18-upgrade-wip-aclamk-cbt-combined-distro-default-smithi/7903539', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 13:48:57.039845', 'locked_by': 'scheduled_akupczyk@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPd/Xz53IbINORkexgwTyDbQNYddGMvI0XigZZfmjUXY9/enVhdsuBZr49MJPieHS0EQTlgT440k29HJeLmdk4M='} 2024-09-13T13:53:57.669 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi195.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/akupczyk-2024-09-13_10:00:18-upgrade-wip-aclamk-cbt-combined-distro-default-smithi/7903539', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 13:48:57.041624', 'locked_by': 'scheduled_akupczyk@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLqy+zsGMRFjGVQ+Ul4AuGa2NnkrmaWA6vETJzxcdQrsmKjoGHpMYTmGdk7GM/DlGMprmk9pCbj7r9rnrOEYadU='} 2024-09-13T13:53:57.669 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-13T13:53:57.676 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-09-13T13:53:57.676 INFO:teuthology.task.internal:roles: ubuntu@smithi195.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-09-13T13:53:57.676 INFO:teuthology.run_tasks:Running task console_log... 2024-09-13T13:53:57.780 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fed21d37be0>, signals=[15]) 2024-09-13T13:53:57.780 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-13T13:53:57.786 INFO:teuthology.task.internal:Opening connections... 2024-09-13T13:53:57.787 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi102.front.sepia.ceph.com 2024-09-13T13:53:57.788 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T13:53:57.877 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi195.front.sepia.ceph.com 2024-09-13T13:53:57.878 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T13:53:57.964 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-13T13:53:57.972 DEBUG:teuthology.orchestra.run.smithi102:> uname -m 2024-09-13T13:53:57.990 INFO:teuthology.orchestra.run.smithi102.stdout:x86_64 2024-09-13T13:53:57.990 DEBUG:teuthology.orchestra.run.smithi102:> cat /etc/os-release 2024-09-13T13:53:58.047 INFO:teuthology.orchestra.run.smithi102.stdout:NAME="CentOS Stream" 2024-09-13T13:53:58.047 INFO:teuthology.orchestra.run.smithi102.stdout:VERSION="9" 2024-09-13T13:53:58.047 INFO:teuthology.orchestra.run.smithi102.stdout:ID="centos" 2024-09-13T13:53:58.047 INFO:teuthology.orchestra.run.smithi102.stdout:ID_LIKE="rhel fedora" 2024-09-13T13:53:58.047 INFO:teuthology.orchestra.run.smithi102.stdout:VERSION_ID="9" 2024-09-13T13:53:58.047 INFO:teuthology.orchestra.run.smithi102.stdout:PLATFORM_ID="platform:el9" 2024-09-13T13:53:58.047 INFO:teuthology.orchestra.run.smithi102.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-13T13:53:58.047 INFO:teuthology.orchestra.run.smithi102.stdout:ANSI_COLOR="0;31" 2024-09-13T13:53:58.047 INFO:teuthology.orchestra.run.smithi102.stdout:LOGO="fedora-logo-icon" 2024-09-13T13:53:58.048 INFO:teuthology.orchestra.run.smithi102.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-13T13:53:58.048 INFO:teuthology.orchestra.run.smithi102.stdout:HOME_URL="https://centos.org/" 2024-09-13T13:53:58.048 INFO:teuthology.orchestra.run.smithi102.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-13T13:53:58.048 INFO:teuthology.orchestra.run.smithi102.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-13T13:53:58.048 INFO:teuthology.orchestra.run.smithi102.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-13T13:53:58.049 INFO:teuthology.lock.ops:Updating smithi102.front.sepia.ceph.com on lock server 2024-09-13T13:53:58.075 DEBUG:teuthology.orchestra.run.smithi195:> uname -m 2024-09-13T13:53:58.094 INFO:teuthology.orchestra.run.smithi195.stdout:x86_64 2024-09-13T13:53:58.094 DEBUG:teuthology.orchestra.run.smithi195:> cat /etc/os-release 2024-09-13T13:53:58.153 INFO:teuthology.orchestra.run.smithi195.stdout:NAME="CentOS Stream" 2024-09-13T13:53:58.153 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION="9" 2024-09-13T13:53:58.153 INFO:teuthology.orchestra.run.smithi195.stdout:ID="centos" 2024-09-13T13:53:58.153 INFO:teuthology.orchestra.run.smithi195.stdout:ID_LIKE="rhel fedora" 2024-09-13T13:53:58.153 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION_ID="9" 2024-09-13T13:53:58.153 INFO:teuthology.orchestra.run.smithi195.stdout:PLATFORM_ID="platform:el9" 2024-09-13T13:53:58.153 INFO:teuthology.orchestra.run.smithi195.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-13T13:53:58.153 INFO:teuthology.orchestra.run.smithi195.stdout:ANSI_COLOR="0;31" 2024-09-13T13:53:58.153 INFO:teuthology.orchestra.run.smithi195.stdout:LOGO="fedora-logo-icon" 2024-09-13T13:53:58.153 INFO:teuthology.orchestra.run.smithi195.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-13T13:53:58.154 INFO:teuthology.orchestra.run.smithi195.stdout:HOME_URL="https://centos.org/" 2024-09-13T13:53:58.154 INFO:teuthology.orchestra.run.smithi195.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-13T13:53:58.154 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-13T13:53:58.154 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-13T13:53:58.154 INFO:teuthology.lock.ops:Updating smithi195.front.sepia.ceph.com on lock server 2024-09-13T13:53:58.183 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-13T13:53:58.193 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-13T13:53:58.225 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-13T13:53:58.226 DEBUG:teuthology.orchestra.run.smithi102:> test '!' -e /home/ubuntu/cephtest 2024-09-13T13:53:58.228 DEBUG:teuthology.orchestra.run.smithi195:> test '!' -e /home/ubuntu/cephtest 2024-09-13T13:53:58.245 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-13T13:53:58.258 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-13T13:53:58.258 DEBUG:teuthology.orchestra.run.smithi102:> test -z $(ls -A /var/lib/ceph) 2024-09-13T13:53:58.285 DEBUG:teuthology.orchestra.run.smithi195:> test -z $(ls -A /var/lib/ceph) 2024-09-13T13:53:58.398 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-13T13:53:58.468 INFO:teuthology.run_tasks:Running task kernel... 2024-09-13T13:53:58.481 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-13T13:53:58.482 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-13T13:53:58.482 DEBUG:teuthology.orchestra.run.smithi102:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T13:53:58.482 DEBUG:teuthology.orchestra.run.smithi195:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T13:53:58.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:53:58.499 DEBUG:teuthology.orchestra.run.smithi102:> uname -r 2024-09-13T13:53:58.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:53:58.501 DEBUG:teuthology.orchestra.run.smithi195:> uname -r 2024-09-13T13:53:58.558 INFO:teuthology.orchestra.run.smithi102.stdout:5.14.0-503.el9.x86_64 2024-09-13T13:53:58.558 INFO:teuthology.task.kernel:Running kernel on smithi102: 5.14.0-503.el9.x86_64 2024-09-13T13:53:58.558 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum install -y kernel 2024-09-13T13:53:58.560 INFO:teuthology.orchestra.run.smithi195.stdout:5.14.0-503.el9.x86_64 2024-09-13T13:53:58.561 INFO:teuthology.task.kernel:Running kernel on smithi195: 5.14.0-503.el9.x86_64 2024-09-13T13:53:58.561 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum install -y kernel 2024-09-13T13:54:00.870 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.2 MB 00:00 2024-09-13T13:54:02.018 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - BaseOS 4.7 MB/s | 8.2 MB 00:01 2024-09-13T13:54:04.419 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - AppStream 15 MB/s | 20 MB 00:01 2024-09-13T13:54:06.621 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - AppStream 4.4 MB/s | 20 MB 00:04 2024-09-13T13:54:08.596 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - CRB 17 MB/s | 6.5 MB 00:00 2024-09-13T13:54:10.146 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - Extras packages 89 kB/s | 19 kB 00:00 2024-09-13T13:54:11.692 INFO:teuthology.orchestra.run.smithi102.stdout:Extra Packages for Enterprise Linux 16 MB/s | 23 MB 00:01 2024-09-13T13:54:12.305 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - CRB 3.4 MB/s | 6.5 MB 00:01 2024-09-13T13:54:14.055 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 19 kB 00:00 2024-09-13T13:54:15.626 INFO:teuthology.orchestra.run.smithi195.stdout:Extra Packages for Enterprise Linux 16 MB/s | 23 MB 00:01 2024-09-13T13:54:17.131 INFO:teuthology.orchestra.run.smithi102.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-13T13:54:19.145 INFO:teuthology.orchestra.run.smithi102.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-13T13:54:19.146 INFO:teuthology.orchestra.run.smithi102.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-13T13:54:19.249 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-09-13T13:54:19.256 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T13:54:19.256 INFO:teuthology.orchestra.run.smithi102.stdout: Package Architecture Version Repository Size 2024-09-13T13:54:19.256 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T13:54:19.256 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-09-13T13:54:19.256 INFO:teuthology.orchestra.run.smithi102.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-13T13:54:19.256 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-09-13T13:54:19.256 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-13T13:54:19.256 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-13T13:54:19.256 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-13T13:54:19.256 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T13:54:19.257 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-09-13T13:54:19.257 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T13:54:19.257 INFO:teuthology.orchestra.run.smithi102.stdout:Install 4 Packages 2024-09-13T13:54:19.257 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T13:54:19.257 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 79 M 2024-09-13T13:54:19.258 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 126 M 2024-09-13T13:54:19.258 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-09-13T13:54:19.583 INFO:teuthology.orchestra.run.smithi102.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 81 kB/s | 23 kB 00:00 2024-09-13T13:54:21.298 INFO:teuthology.orchestra.run.smithi195.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-09-13T13:54:23.338 INFO:teuthology.orchestra.run.smithi195.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-13T13:54:23.338 INFO:teuthology.orchestra.run.smithi195.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-13T13:54:23.420 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-09-13T13:54:23.430 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T13:54:23.430 INFO:teuthology.orchestra.run.smithi195.stdout: Package Architecture Version Repository Size 2024-09-13T13:54:23.430 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T13:54:23.430 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-09-13T13:54:23.430 INFO:teuthology.orchestra.run.smithi195.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-13T13:54:23.430 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-09-13T13:54:23.430 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-13T13:54:23.430 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-13T13:54:23.430 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-13T13:54:23.430 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T13:54:23.431 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-09-13T13:54:23.431 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T13:54:23.431 INFO:teuthology.orchestra.run.smithi195.stdout:Install 4 Packages 2024-09-13T13:54:23.431 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T13:54:23.431 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 79 M 2024-09-13T13:54:23.431 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 126 M 2024-09-13T13:54:23.431 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-09-13T13:54:23.783 INFO:teuthology.orchestra.run.smithi195.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 235 kB/s | 23 kB 00:00 2024-09-13T13:54:28.132 INFO:teuthology.orchestra.run.smithi195.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 3.5 MB/s | 16 MB 00:04 2024-09-13T13:54:28.315 INFO:teuthology.orchestra.run.smithi102.stdout:(2/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 3.8 MB/s | 35 MB 00:09 2024-09-13T13:54:28.932 INFO:teuthology.orchestra.run.smithi102.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 3.1 MB/s | 29 MB 00:09 2024-09-13T13:54:29.474 INFO:teuthology.orchestra.run.smithi102.stdout:(4/4): kernel-core-5.14.0-505.el9.x86_64.rpm 1.5 MB/s | 16 MB 00:10 2024-09-13T13:54:29.474 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:54:29.474 INFO:teuthology.orchestra.run.smithi102.stdout:Total 7.7 MB/s | 79 MB 00:10 2024-09-13T13:54:30.156 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-09-13T13:54:30.257 INFO:teuthology.orchestra.run.smithi195.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 4.4 MB/s | 29 MB 00:06 2024-09-13T13:54:30.502 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-09-13T13:54:30.502 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-09-13T13:54:30.800 INFO:teuthology.orchestra.run.smithi195.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 4.9 MB/s | 35 MB 00:07 2024-09-13T13:54:30.800 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:54:30.800 INFO:teuthology.orchestra.run.smithi195.stdout:Total 11 MB/s | 79 MB 00:07 2024-09-13T13:54:31.473 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-09-13T13:54:31.474 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-09-13T13:54:31.477 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-09-13T13:54:31.794 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-09-13T13:54:31.794 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-09-13T13:54:32.757 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-09-13T13:54:32.757 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-09-13T13:54:33.918 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-09-13T13:54:34.563 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-13T13:54:34.650 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T13:54:35.181 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-09-13T13:54:35.843 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-13T13:54:35.933 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T13:54:37.437 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T13:54:37.571 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T13:54:38.697 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T13:54:38.821 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T13:54:42.385 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T13:54:42.505 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T13:54:43.638 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T13:54:43.737 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T13:54:47.322 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T13:54:48.610 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T13:55:25.525 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T13:55:25.564 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-13T13:55:25.978 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T13:55:25.978 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-13T13:55:25.978 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T13:55:25.978 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T13:55:26.312 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T13:55:26.358 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-13T13:55:26.679 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T13:55:26.679 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-13T13:55:26.679 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T13:55:26.679 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T13:55:26.837 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T13:55:26.838 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T13:55:26.838 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-09-13T13:55:26.838 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-13T13:55:26.838 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-13T13:55:26.838 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-13T13:55:26.838 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-13T13:55:26.838 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T13:55:26.838 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-09-13T13:55:27.195 DEBUG:teuthology.orchestra.run.smithi102:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T13:55:27.270 INFO:teuthology.orchestra.run.smithi102.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T13:55:27.270 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T13:55:27.271 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-13T13:55:27.271 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-13T13:55:27.271 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-13T13:55:27.271 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-13T13:55:27.271 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi102.front.sepia.ceph.com, path=None, version=distro) 2024-09-13T13:55:27.271 DEBUG:teuthology.orchestra.run.smithi102:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T13:55:27.330 INFO:teuthology.orchestra.run.smithi102.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T13:55:27.330 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T13:55:27.330 DEBUG:teuthology.orchestra.run.smithi102:> sudo rpm -qi grub2-tools 2024-09-13T13:55:27.371 INFO:teuthology.orchestra.run.smithi102.stdout:Name : grub2-tools 2024-09-13T13:55:27.371 INFO:teuthology.orchestra.run.smithi102.stdout:Epoch : 1 2024-09-13T13:55:27.372 INFO:teuthology.orchestra.run.smithi102.stdout:Version : 2.06 2024-09-13T13:55:27.372 INFO:teuthology.orchestra.run.smithi102.stdout:Release : 82.el9 2024-09-13T13:55:27.372 INFO:teuthology.orchestra.run.smithi102.stdout:Architecture: x86_64 2024-09-13T13:55:27.372 INFO:teuthology.orchestra.run.smithi102.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-13T13:55:27.372 INFO:teuthology.orchestra.run.smithi102.stdout:Group : Unspecified 2024-09-13T13:55:27.372 INFO:teuthology.orchestra.run.smithi102.stdout:Size : 8274670 2024-09-13T13:55:27.372 INFO:teuthology.orchestra.run.smithi102.stdout:License : GPLv3+ 2024-09-13T13:55:27.372 INFO:teuthology.orchestra.run.smithi102.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-13T13:55:27.372 INFO:teuthology.orchestra.run.smithi102.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-13T13:55:27.372 INFO:teuthology.orchestra.run.smithi102.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-13T13:55:27.372 INFO:teuthology.orchestra.run.smithi102.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-13T13:55:27.372 INFO:teuthology.orchestra.run.smithi102.stdout:Packager : builder@centos.org 2024-09-13T13:55:27.372 INFO:teuthology.orchestra.run.smithi102.stdout:Vendor : CentOS 2024-09-13T13:55:27.373 INFO:teuthology.orchestra.run.smithi102.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-13T13:55:27.373 INFO:teuthology.orchestra.run.smithi102.stdout:Summary : Support tools for GRUB. 2024-09-13T13:55:27.373 INFO:teuthology.orchestra.run.smithi102.stdout:Description : 2024-09-13T13:55:27.373 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T13:55:27.373 INFO:teuthology.orchestra.run.smithi102.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-13T13:55:27.373 INFO:teuthology.orchestra.run.smithi102.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-13T13:55:27.373 INFO:teuthology.orchestra.run.smithi102.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-13T13:55:27.373 INFO:teuthology.orchestra.run.smithi102.stdout:hardware devices. 2024-09-13T13:55:27.373 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T13:55:27.373 INFO:teuthology.orchestra.run.smithi102.stdout:This subpackage provides tools for support of all platforms. 2024-09-13T13:55:27.374 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-13T13:55:27.374 INFO:teuthology.task.kernel:Updating grub on smithi102 to boot 5.14.0-505.el9.x86_64 2024-09-13T13:55:27.374 DEBUG:teuthology.orchestra.run.smithi102:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-13T13:55:27.552 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T13:55:27.552 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T13:55:27.552 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-09-13T13:55:27.552 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-13T13:55:27.552 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-13T13:55:27.552 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-13T13:55:27.552 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-13T13:55:27.553 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T13:55:27.553 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-09-13T13:55:27.904 DEBUG:teuthology.orchestra.run.smithi195:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T13:55:27.985 INFO:teuthology.orchestra.run.smithi195.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T13:55:27.985 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T13:55:27.985 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-13T13:55:27.985 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-13T13:55:27.986 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-13T13:55:27.986 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-13T13:55:27.986 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi195.front.sepia.ceph.com, path=None, version=distro) 2024-09-13T13:55:27.986 DEBUG:teuthology.orchestra.run.smithi195:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T13:55:28.045 INFO:teuthology.orchestra.run.smithi195.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T13:55:28.045 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T13:55:28.045 DEBUG:teuthology.orchestra.run.smithi195:> sudo rpm -qi grub2-tools 2024-09-13T13:55:28.070 INFO:teuthology.orchestra.run.smithi102.stderr:Generating grub configuration file ... 2024-09-13T13:55:28.085 INFO:teuthology.orchestra.run.smithi195.stdout:Name : grub2-tools 2024-09-13T13:55:28.085 INFO:teuthology.orchestra.run.smithi195.stdout:Epoch : 1 2024-09-13T13:55:28.085 INFO:teuthology.orchestra.run.smithi195.stdout:Version : 2.06 2024-09-13T13:55:28.085 INFO:teuthology.orchestra.run.smithi195.stdout:Release : 82.el9 2024-09-13T13:55:28.085 INFO:teuthology.orchestra.run.smithi195.stdout:Architecture: x86_64 2024-09-13T13:55:28.085 INFO:teuthology.orchestra.run.smithi195.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-13T13:55:28.086 INFO:teuthology.orchestra.run.smithi195.stdout:Group : Unspecified 2024-09-13T13:55:28.086 INFO:teuthology.orchestra.run.smithi195.stdout:Size : 8274670 2024-09-13T13:55:28.086 INFO:teuthology.orchestra.run.smithi195.stdout:License : GPLv3+ 2024-09-13T13:55:28.086 INFO:teuthology.orchestra.run.smithi195.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-13T13:55:28.086 INFO:teuthology.orchestra.run.smithi195.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-13T13:55:28.086 INFO:teuthology.orchestra.run.smithi195.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-13T13:55:28.086 INFO:teuthology.orchestra.run.smithi195.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-13T13:55:28.086 INFO:teuthology.orchestra.run.smithi195.stdout:Packager : builder@centos.org 2024-09-13T13:55:28.086 INFO:teuthology.orchestra.run.smithi195.stdout:Vendor : CentOS 2024-09-13T13:55:28.086 INFO:teuthology.orchestra.run.smithi195.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-13T13:55:28.086 INFO:teuthology.orchestra.run.smithi195.stdout:Summary : Support tools for GRUB. 2024-09-13T13:55:28.086 INFO:teuthology.orchestra.run.smithi195.stdout:Description : 2024-09-13T13:55:28.087 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T13:55:28.087 INFO:teuthology.orchestra.run.smithi195.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-13T13:55:28.087 INFO:teuthology.orchestra.run.smithi195.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-13T13:55:28.087 INFO:teuthology.orchestra.run.smithi195.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-13T13:55:28.087 INFO:teuthology.orchestra.run.smithi195.stdout:hardware devices. 2024-09-13T13:55:28.087 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T13:55:28.087 INFO:teuthology.orchestra.run.smithi195.stdout:This subpackage provides tools for support of all platforms. 2024-09-13T13:55:28.088 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-13T13:55:28.088 INFO:teuthology.task.kernel:Updating grub on smithi195 to boot 5.14.0-505.el9.x86_64 2024-09-13T13:55:28.088 DEBUG:teuthology.orchestra.run.smithi195:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-13T13:55:28.673 INFO:teuthology.orchestra.run.smithi195.stderr:Generating grub configuration file ... 2024-09-13T13:55:29.349 INFO:teuthology.orchestra.run.smithi102.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-13T13:55:29.391 INFO:teuthology.orchestra.run.smithi102.stderr:done 2024-09-13T13:55:29.393 DEBUG:teuthology.orchestra.run.smithi102:> mktemp 2024-09-13T13:55:29.409 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/tmp.86mOmz2SQZ 2024-09-13T13:55:29.409 DEBUG:teuthology.orchestra.run.smithi102:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.86mOmz2SQZ 2024-09-13T13:55:29.489 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 0666 /tmp/tmp.86mOmz2SQZ 2024-09-13T13:55:29.641 DEBUG:teuthology.orchestra.remote:smithi102:/tmp/tmp.86mOmz2SQZ is 6KB 2024-09-13T13:55:29.662 DEBUG:teuthology.orchestra.run.smithi102:> rm -fr /tmp/tmp.86mOmz2SQZ 2024-09-13T13:55:29.677 DEBUG:teuthology.orchestra.run.smithi102:> sudo /bin/ls /boot/loader/entries || true 2024-09-13T13:55:29.740 INFO:teuthology.orchestra.run.smithi102.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-13T13:55:29.740 INFO:teuthology.orchestra.run.smithi102.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-13T13:55:29.740 INFO:teuthology.orchestra.run.smithi102.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-13T13:55:29.740 INFO:teuthology.orchestra.run.smithi102.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-13T13:55:29.741 DEBUG:teuthology.orchestra.run.smithi102:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-13T13:55:29.890 DEBUG:teuthology.orchestra.run.smithi102:> sudo shutdown -r now 2024-09-13T13:55:29.965 INFO:teuthology.orchestra.run.smithi195.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-13T13:55:30.009 INFO:teuthology.orchestra.run.smithi195.stderr:done 2024-09-13T13:55:30.011 DEBUG:teuthology.orchestra.run.smithi195:> mktemp 2024-09-13T13:55:30.026 INFO:teuthology.orchestra.run.smithi195.stdout:/tmp/tmp.2miCeQTDP5 2024-09-13T13:55:30.026 DEBUG:teuthology.orchestra.run.smithi195:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.2miCeQTDP5 2024-09-13T13:55:30.093 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 0666 /tmp/tmp.2miCeQTDP5 2024-09-13T13:55:30.234 DEBUG:teuthology.orchestra.remote:smithi195:/tmp/tmp.2miCeQTDP5 is 6KB 2024-09-13T13:55:30.282 DEBUG:teuthology.orchestra.run.smithi195:> rm -fr /tmp/tmp.2miCeQTDP5 2024-09-13T13:55:30.297 DEBUG:teuthology.orchestra.run.smithi195:> sudo /bin/ls /boot/loader/entries || true 2024-09-13T13:55:30.371 INFO:teuthology.orchestra.run.smithi195.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-13T13:55:30.371 INFO:teuthology.orchestra.run.smithi195.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-13T13:55:30.371 INFO:teuthology.orchestra.run.smithi195.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-13T13:55:30.371 INFO:teuthology.orchestra.run.smithi195.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-13T13:55:30.372 DEBUG:teuthology.orchestra.run.smithi195:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-13T13:55:30.505 DEBUG:teuthology.orchestra.run.smithi195:> sudo shutdown -r now 2024-09-13T13:55:59.921 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-13T13:55:59.922 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-09-13T13:55:59.923 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T13:56:00.509 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-13T13:56:00.509 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-09-13T13:56:00.510 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T13:56:18.365 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.102 2024-09-13T13:56:18.877 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.195 2024-09-13T13:56:27.373 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-09-13T13:56:27.374 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T13:56:27.878 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-09-13T13:56:27.879 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T13:56:30.941 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.195 2024-09-13T13:56:42.953 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-09-13T13:56:42.954 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T13:57:14.759 DEBUG:teuthology.orchestra.run.smithi195:> true 2024-09-13T13:57:15.467 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-09-13T13:57:15.467 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-505.el9.x86_64"... 2024-09-13T13:57:15.467 DEBUG:teuthology.orchestra.run.smithi195:> uname -r 2024-09-13T13:57:15.503 INFO:teuthology.orchestra.run.smithi195.stdout:5.14.0-505.el9.x86_64 2024-09-13T13:57:15.503 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-13T13:57:15.503 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T13:57:15.503 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-13T13:57:16.503 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-13T13:57:16.504 DEBUG:teuthology.orchestra.run.smithi195:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-13T13:57:16.828 INFO:teuthology.orchestra.run.smithi195.stdout:ttyS1 2024-09-13T13:57:16.857 DEBUG:teuthology.parallel:result is None 2024-09-13T13:57:27.384 DEBUG:teuthology.orchestra.remote:timed out 2024-09-13T13:57:39.385 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-09-13T13:57:39.386 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T13:57:39.895 DEBUG:teuthology.orchestra.run.smithi102:> true 2024-09-13T13:57:40.287 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-09-13T13:57:40.287 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-13T13:57:40.287 DEBUG:teuthology.orchestra.run.smithi102:> uname -r 2024-09-13T13:57:40.305 INFO:teuthology.orchestra.run.smithi102.stdout:5.14.0-505.el9.x86_64 2024-09-13T13:57:40.305 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-13T13:57:40.305 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T13:57:40.305 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-13T13:57:41.306 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-13T13:57:41.306 DEBUG:teuthology.orchestra.run.smithi102:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-13T13:57:41.655 INFO:teuthology.orchestra.run.smithi102.stdout:ttyS1 2024-09-13T13:57:41.674 DEBUG:teuthology.parallel:result is None 2024-09-13T13:57:41.674 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-13T13:57:41.688 INFO:teuthology.task.internal:Creating test directory... 2024-09-13T13:57:41.688 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T13:57:41.691 DEBUG:teuthology.orchestra.run.smithi195:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T13:57:41.710 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-13T13:57:41.733 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-13T13:57:41.766 INFO:teuthology.task.internal:Creating archive directory... 2024-09-13T13:57:41.766 DEBUG:teuthology.orchestra.run.smithi102:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T13:57:41.769 DEBUG:teuthology.orchestra.run.smithi195:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T13:57:41.821 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-13T13:57:41.829 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-13T13:57:41.830 DEBUG:teuthology.orchestra.run.smithi102:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T13:57:41.832 DEBUG:teuthology.orchestra.run.smithi195:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T13:57:41.861 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T13:57:41.873 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T13:57:41.878 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T13:57:41.890 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T13:57:41.891 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-13T13:57:41.902 INFO:teuthology.task.internal:Configuring sudo... 2024-09-13T13:57:41.903 DEBUG:teuthology.orchestra.run.smithi102:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T13:57:41.917 DEBUG:teuthology.orchestra.run.smithi195:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T13:57:41.959 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-13T13:57:41.972 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-13T13:57:41.973 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T13:57:41.987 DEBUG:teuthology.orchestra.run.smithi195:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T13:57:42.016 DEBUG:teuthology.orchestra.run.smithi102:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T13:57:42.092 DEBUG:teuthology.orchestra.run.smithi102:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T13:57:42.140 DEBUG:teuthology.orchestra.run.smithi102:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T13:57:42.232 DEBUG:teuthology.orchestra.run.smithi102:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T13:57:42.265 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-09-13T13:57:42.265 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T13:57:42.330 DEBUG:teuthology.orchestra.run.smithi195:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T13:57:42.382 DEBUG:teuthology.orchestra.run.smithi195:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T13:57:42.431 DEBUG:teuthology.orchestra.run.smithi195:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T13:57:42.515 DEBUG:teuthology.orchestra.run.smithi195:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T13:57:42.561 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-09-13T13:57:42.561 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T13:57:42.629 DEBUG:teuthology.orchestra.run.smithi102:> sudo service rsyslog restart 2024-09-13T13:57:42.632 DEBUG:teuthology.orchestra.run.smithi195:> sudo service rsyslog restart 2024-09-13T13:57:42.682 INFO:teuthology.orchestra.run.smithi102.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T13:57:42.726 INFO:teuthology.orchestra.run.smithi195.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T13:57:43.035 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-13T13:57:43.051 INFO:teuthology.task.internal:Starting timer... 2024-09-13T13:57:43.051 INFO:teuthology.run_tasks:Running task pcp... 2024-09-13T13:57:43.085 INFO:teuthology.run_tasks:Running task selinux... 2024-09-13T13:57:43.123 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-09-13T13:57:43.124 DEBUG:teuthology.orchestra.run.smithi102:> sudo service auditd rotate 2024-09-13T13:57:43.174 INFO:teuthology.orchestra.run.smithi102.stdout:Rotating logs: 2024-09-13T13:57:43.176 DEBUG:teuthology.orchestra.run.smithi195:> sudo service auditd rotate 2024-09-13T13:57:43.232 INFO:teuthology.orchestra.run.smithi195.stdout:Rotating logs: 2024-09-13T13:57:43.234 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-13T13:57:43.234 DEBUG:teuthology.orchestra.run.smithi102:> /usr/sbin/getenforce 2024-09-13T13:57:43.266 INFO:teuthology.orchestra.run.smithi102.stdout:Permissive 2024-09-13T13:57:43.267 DEBUG:teuthology.orchestra.run.smithi195:> /usr/sbin/getenforce 2024-09-13T13:57:43.291 INFO:teuthology.orchestra.run.smithi195.stdout:Permissive 2024-09-13T13:57:43.292 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi102.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi195.front.sepia.ceph.com': 'permissive'} 2024-09-13T13:57:43.292 DEBUG:teuthology.orchestra.run.smithi102:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-13T13:57:43.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:57:43.334 DEBUG:teuthology.orchestra.run.smithi195:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-13T13:57:43.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:57:43.363 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-13T13:57:43.363 DEBUG:teuthology.orchestra.run.smithi102:> sudo /usr/sbin/setenforce permissive 2024-09-13T13:57:43.407 DEBUG:teuthology.orchestra.run.smithi195:> sudo /usr/sbin/setenforce permissive 2024-09-13T13:57:43.441 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-13T13:57:43.451 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-13T13:57:43.461 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-13T13:57:43.462 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi102.front.sepia.ceph.com,smithi195.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-13T14:01:28.105 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi102.front.sepia.ceph.com'), Remote(name='ubuntu@smithi195.front.sepia.ceph.com')] 2024-09-13T14:01:28.107 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-09-13T14:01:28.108 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T14:01:28.201 DEBUG:teuthology.orchestra.run.smithi102:> true 2024-09-13T14:01:28.235 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-09-13T14:01:28.235 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-09-13T14:01:28.236 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T14:01:28.317 DEBUG:teuthology.orchestra.run.smithi195:> true 2024-09-13T14:01:28.395 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-09-13T14:01:28.396 INFO:teuthology.run_tasks:Running task clock... 2024-09-13T14:01:28.408 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-13T14:01:28.408 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T14:01:28.409 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T14:01:28.412 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T14:01:28.412 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T14:01:28.438 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-13T14:01:28.453 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-13T14:01:28.465 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-13T14:01:28.476 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: ntpd: command not found 2024-09-13T14:01:28.479 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-13T14:01:28.488 INFO:teuthology.orchestra.run.smithi102.stdout:506 Cannot talk to daemon 2024-09-13T14:01:28.502 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-13T14:01:28.503 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: ntpd: command not found 2024-09-13T14:01:28.514 INFO:teuthology.orchestra.run.smithi195.stdout:506 Cannot talk to daemon 2024-09-13T14:01:28.515 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-13T14:01:28.528 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-13T14:01:28.542 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-13T14:01:28.571 INFO:teuthology.orchestra.run.smithi102.stderr:bash: line 1: ntpq: command not found 2024-09-13T14:01:28.574 INFO:teuthology.orchestra.run.smithi102.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T14:01:28.575 INFO:teuthology.orchestra.run.smithi102.stdout:=============================================================================== 2024-09-13T14:01:28.575 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T14:01:28.575 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T14:01:28.575 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T14:01:28.575 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T14:01:28.599 INFO:teuthology.orchestra.run.smithi195.stderr:bash: line 1: ntpq: command not found 2024-09-13T14:01:28.602 INFO:teuthology.orchestra.run.smithi195.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T14:01:28.602 INFO:teuthology.orchestra.run.smithi195.stdout:=============================================================================== 2024-09-13T14:01:28.602 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T14:01:28.603 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T14:01:28.603 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T14:01:28.603 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T14:01:28.604 INFO:teuthology.run_tasks:Running task pexec... 2024-09-13T14:01:28.613 INFO:teuthology.task.pexec:Executing custom commands... 2024-09-13T14:01:28.614 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi102.front.sepia.ceph.com 2024-09-13T14:01:28.614 DEBUG:teuthology.orchestra.run.smithi102:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-13T14:01:28.614 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi195.front.sepia.ceph.com 2024-09-13T14:01:28.614 DEBUG:teuthology.orchestra.run.smithi195:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-13T14:01:28.965 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-09-13T14:01:28.976 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T14:01:28.976 INFO:teuthology.orchestra.run.smithi102.stdout: Package Architecture Version Repository Size 2024-09-13T14:01:28.976 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T14:01:28.976 INFO:teuthology.orchestra.run.smithi102.stdout:Removing: 2024-09-13T14:01:28.976 INFO:teuthology.orchestra.run.smithi102.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-13T14:01:28.977 INFO:teuthology.orchestra.run.smithi102.stdout:Removing unused dependencies: 2024-09-13T14:01:28.977 INFO:teuthology.orchestra.run.smithi102.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-13T14:01:28.977 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:01:28.977 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-09-13T14:01:28.977 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T14:01:28.977 INFO:teuthology.orchestra.run.smithi102.stdout:Remove 2 Packages 2024-09-13T14:01:28.977 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:01:28.977 INFO:teuthology.orchestra.run.smithi102.stdout:Freed space: 5.5 M 2024-09-13T14:01:28.977 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-09-13T14:01:28.979 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-09-13T14:01:28.979 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-09-13T14:01:28.995 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-09-13T14:01:29.006 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T14:01:29.006 INFO:teuthology.orchestra.run.smithi195.stdout: Package Architecture Version Repository Size 2024-09-13T14:01:29.006 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T14:01:29.006 INFO:teuthology.orchestra.run.smithi195.stdout:Removing: 2024-09-13T14:01:29.006 INFO:teuthology.orchestra.run.smithi195.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-13T14:01:29.006 INFO:teuthology.orchestra.run.smithi195.stdout:Removing unused dependencies: 2024-09-13T14:01:29.006 INFO:teuthology.orchestra.run.smithi195.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-13T14:01:29.006 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:01:29.006 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-09-13T14:01:29.006 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T14:01:29.007 INFO:teuthology.orchestra.run.smithi195.stdout:Remove 2 Packages 2024-09-13T14:01:29.007 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:01:29.007 INFO:teuthology.orchestra.run.smithi195.stdout:Freed space: 5.5 M 2024-09-13T14:01:29.007 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-09-13T14:01:29.010 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-09-13T14:01:29.010 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-09-13T14:01:29.037 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-09-13T14:01:29.037 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-09-13T14:01:29.067 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-09-13T14:01:29.068 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-09-13T14:01:29.184 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-09-13T14:01:29.222 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-09-13T14:01:29.268 INFO:teuthology.orchestra.run.smithi102.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-13T14:01:29.296 INFO:teuthology.orchestra.run.smithi195.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-13T14:01:29.338 INFO:teuthology.orchestra.run.smithi102.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-13T14:01:29.361 INFO:teuthology.orchestra.run.smithi195.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-13T14:01:29.810 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-13T14:01:29.811 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-13T14:01:29.864 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-13T14:01:29.864 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-13T14:01:30.071 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-13T14:01:30.072 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:01:30.072 INFO:teuthology.orchestra.run.smithi102.stdout:Removed: 2024-09-13T14:01:30.072 INFO:teuthology.orchestra.run.smithi102.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-13T14:01:30.072 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:01:30.072 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-09-13T14:01:30.103 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-13T14:01:30.103 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:01:30.103 INFO:teuthology.orchestra.run.smithi195.stdout:Removed: 2024-09-13T14:01:30.103 INFO:teuthology.orchestra.run.smithi195.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-13T14:01:30.104 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:01:30.104 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-09-13T14:01:30.887 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:00:48 ago on Fri 13 Sep 2024 02:00:42 PM UTC. 2024-09-13T14:01:30.944 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:00:48 ago on Fri 13 Sep 2024 02:00:42 PM UTC. 2024-09-13T14:01:31.019 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-09-13T14:01:31.020 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T14:01:31.020 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-09-13T14:01:31.020 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T14:01:31.020 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-09-13T14:01:31.020 INFO:teuthology.orchestra.run.smithi102.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-13T14:01:31.020 INFO:teuthology.orchestra.run.smithi102.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-13T14:01:31.020 INFO:teuthology.orchestra.run.smithi102.stdout: runc x86_64 4:1.1.13-4.el9 appstream 3.2 M 2024-09-13T14:01:31.020 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-09-13T14:01:31.021 INFO:teuthology.orchestra.run.smithi102.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-13T14:01:31.021 INFO:teuthology.orchestra.run.smithi102.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-13T14:01:31.021 INFO:teuthology.orchestra.run.smithi102.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-13T14:01:31.021 INFO:teuthology.orchestra.run.smithi102.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-13T14:01:31.021 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:01:31.021 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-09-13T14:01:31.021 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T14:01:31.021 INFO:teuthology.orchestra.run.smithi102.stdout:Install 7 Packages 2024-09-13T14:01:31.021 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:01:31.050 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 5.2 M 2024-09-13T14:01:31.050 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 19 M 2024-09-13T14:01:31.050 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-09-13T14:01:31.080 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-09-13T14:01:31.081 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T14:01:31.081 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-09-13T14:01:31.081 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T14:01:31.081 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-09-13T14:01:31.081 INFO:teuthology.orchestra.run.smithi195.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-13T14:01:31.082 INFO:teuthology.orchestra.run.smithi195.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-13T14:01:31.082 INFO:teuthology.orchestra.run.smithi195.stdout: runc x86_64 4:1.1.13-4.el9 appstream 3.2 M 2024-09-13T14:01:31.082 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-09-13T14:01:31.082 INFO:teuthology.orchestra.run.smithi195.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-13T14:01:31.082 INFO:teuthology.orchestra.run.smithi195.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-13T14:01:31.082 INFO:teuthology.orchestra.run.smithi195.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-13T14:01:31.082 INFO:teuthology.orchestra.run.smithi195.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-13T14:01:31.082 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:01:31.083 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-09-13T14:01:31.083 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T14:01:31.083 INFO:teuthology.orchestra.run.smithi195.stdout:Install 7 Packages 2024-09-13T14:01:31.083 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:01:31.093 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 5.2 M 2024-09-13T14:01:31.093 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 19 M 2024-09-13T14:01:31.093 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-09-13T14:01:31.501 INFO:teuthology.orchestra.run.smithi195.stdout:(1/7): nvmetcli-0.7-3.el9.noarch.rpm 306 kB/s | 44 kB 00:00 2024-09-13T14:01:31.526 INFO:teuthology.orchestra.run.smithi195.stdout:(2/7): libnvme-1.9-3.el9.x86_64.rpm 622 kB/s | 106 kB 00:00 2024-09-13T14:01:31.543 INFO:teuthology.orchestra.run.smithi102.stdout:(1/7): nvmetcli-0.7-3.el9.noarch.rpm 304 kB/s | 44 kB 00:00 2024-09-13T14:01:31.568 INFO:teuthology.orchestra.run.smithi102.stdout:(2/7): libnvme-1.9-3.el9.x86_64.rpm 619 kB/s | 106 kB 00:00 2024-09-13T14:01:31.577 INFO:teuthology.orchestra.run.smithi195.stdout:(3/7): python3-kmod-0.9-32.el9.x86_64.rpm 1.6 MB/s | 84 kB 00:00 2024-09-13T14:01:31.610 INFO:teuthology.orchestra.run.smithi195.stdout:(4/7): python3-configshell-1.1.30-1.el9.noarch. 660 kB/s | 72 kB 00:00 2024-09-13T14:01:31.619 INFO:teuthology.orchestra.run.smithi102.stdout:(3/7): python3-kmod-0.9-32.el9.x86_64.rpm 1.6 MB/s | 84 kB 00:00 2024-09-13T14:01:31.652 INFO:teuthology.orchestra.run.smithi102.stdout:(4/7): python3-configshell-1.1.30-1.el9.noarch. 659 kB/s | 72 kB 00:00 2024-09-13T14:01:31.678 INFO:teuthology.orchestra.run.smithi195.stdout:(5/7): nvme-cli-2.9.1-6.el9.x86_64.rpm 2.8 MB/s | 906 kB 00:00 2024-09-13T14:01:31.711 INFO:teuthology.orchestra.run.smithi102.stdout:(5/7): nvme-cli-2.9.1-6.el9.x86_64.rpm 2.8 MB/s | 906 kB 00:00 2024-09-13T14:01:31.736 INFO:teuthology.orchestra.run.smithi195.stdout:(6/7): python3-urwid-2.1.2-4.el9.x86_64.rpm 5.1 MB/s | 837 kB 00:00 2024-09-13T14:01:31.803 INFO:teuthology.orchestra.run.smithi102.stdout:(6/7): python3-urwid-2.1.2-4.el9.x86_64.rpm 4.4 MB/s | 837 kB 00:00 2024-09-13T14:01:31.945 INFO:teuthology.orchestra.run.smithi102.stdout:(7/7): runc-1.1.13-4.el9.x86_64.rpm 11 MB/s | 3.2 MB 00:00 2024-09-13T14:01:31.945 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:01:31.945 INFO:teuthology.orchestra.run.smithi102.stdout:Total 5.8 MB/s | 5.2 MB 00:00 2024-09-13T14:01:32.054 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-09-13T14:01:32.069 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-09-13T14:01:32.070 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-09-13T14:01:32.353 INFO:teuthology.orchestra.run.smithi195.stdout:(7/7): runc-1.1.13-4.el9.x86_64.rpm 4.3 MB/s | 3.2 MB 00:00 2024-09-13T14:01:32.353 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:01:32.354 INFO:teuthology.orchestra.run.smithi195.stdout:Total 4.1 MB/s | 5.2 MB 00:01 2024-09-13T14:01:32.386 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-09-13T14:01:32.387 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-09-13T14:01:32.463 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-09-13T14:01:32.478 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-09-13T14:01:32.478 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-09-13T14:01:32.783 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-09-13T14:01:32.784 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-09-13T14:01:32.863 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-09-13T14:01:32.939 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/7 2024-09-13T14:01:32.976 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/7 2024-09-13T14:01:33.011 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/7 2024-09-13T14:01:33.228 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/7 2024-09-13T14:01:33.247 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-09-13T14:01:33.267 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-13T14:01:33.330 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/7 2024-09-13T14:01:33.365 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/7 2024-09-13T14:01:33.393 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/7 2024-09-13T14:01:33.609 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/7 2024-09-13T14:01:33.650 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-13T14:01:33.705 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-13T14:01:33.736 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/7 2024-09-13T14:01:33.849 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/7 2024-09-13T14:01:33.914 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-13T14:01:34.092 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-13T14:01:34.119 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/7 2024-09-13T14:01:34.243 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/7 2024-09-13T14:01:34.306 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-13T14:01:34.860 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-13T14:01:34.860 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/7 2024-09-13T14:01:34.860 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/7 2024-09-13T14:01:34.860 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/7 2024-09-13T14:01:34.860 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/7 2024-09-13T14:01:34.860 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/7 2024-09-13T14:01:34.861 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/7 2024-09-13T14:01:35.222 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-13T14:01:35.222 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:01:35.223 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-09-13T14:01:35.223 INFO:teuthology.orchestra.run.smithi102.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-13T14:01:35.223 INFO:teuthology.orchestra.run.smithi102.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-13T14:01:35.223 INFO:teuthology.orchestra.run.smithi102.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-13T14:01:35.223 INFO:teuthology.orchestra.run.smithi102.stdout: runc-4:1.1.13-4.el9.x86_64 2024-09-13T14:01:35.223 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:01:35.223 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-09-13T14:01:35.265 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-13T14:01:35.265 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/7 2024-09-13T14:01:35.265 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/7 2024-09-13T14:01:35.265 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/7 2024-09-13T14:01:35.265 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/7 2024-09-13T14:01:35.265 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/7 2024-09-13T14:01:35.265 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/7 2024-09-13T14:01:35.596 DEBUG:teuthology.parallel:result is None 2024-09-13T14:01:35.615 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-13T14:01:35.616 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:01:35.616 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-09-13T14:01:35.616 INFO:teuthology.orchestra.run.smithi195.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-13T14:01:35.616 INFO:teuthology.orchestra.run.smithi195.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-13T14:01:35.616 INFO:teuthology.orchestra.run.smithi195.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-13T14:01:35.616 INFO:teuthology.orchestra.run.smithi195.stdout: runc-4:1.1.13-4.el9.x86_64 2024-09-13T14:01:35.616 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:01:35.616 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-09-13T14:01:36.007 DEBUG:teuthology.parallel:result is None 2024-09-13T14:01:36.007 INFO:teuthology.run_tasks:Running task install... 2024-09-13T14:01:36.016 DEBUG:teuthology.task.install:project ceph 2024-09-13T14:01:36.017 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}} 2024-09-13T14:01:36.017 DEBUG:teuthology.task.install:config {'branch': 'reef', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'} 2024-09-13T14:01:36.017 INFO:teuthology.task.install:Using flavor: default 2024-09-13T14:01:36.022 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-13T14:01:36.022 INFO:teuthology.task.install:extra packages: [] 2024-09-13T14:01:36.023 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'reef', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9', 'tag': None, 'wait_for_package': False} 2024-09-13T14:01:36.023 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-13T14:01:36.023 INFO:teuthology.packaging:ref: None 2024-09-13T14:01:36.023 INFO:teuthology.packaging:tag: None 2024-09-13T14:01:36.023 INFO:teuthology.packaging:branch: reef 2024-09-13T14:01:36.023 INFO:teuthology.packaging:sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:01:36.024 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-09-13T14:01:36.026 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'reef', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9', 'tag': None, 'wait_for_package': False} 2024-09-13T14:01:36.026 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-13T14:01:36.026 INFO:teuthology.packaging:ref: None 2024-09-13T14:01:36.026 INFO:teuthology.packaging:tag: None 2024-09-13T14:01:36.026 INFO:teuthology.packaging:branch: reef 2024-09-13T14:01:36.026 INFO:teuthology.packaging:sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:01:36.026 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-09-13T14:01:36.264 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/reef/2c01458affee2b2435cd231b6e392435c1c21661/centos/9/flavors/default/ 2024-09-13T14:01:36.265 INFO:teuthology.task.install.rpm:Package version is 18.2.4-843.g2c01458a 2024-09-13T14:01:36.306 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/reef/2c01458affee2b2435cd231b6e392435c1c21661/centos/9/flavors/default/ 2024-09-13T14:01:36.306 INFO:teuthology.task.install.rpm:Package version is 18.2.4-843.g2c01458a 2024-09-13T14:01:36.387 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/reef/2c01458affee2b2435cd231b6e392435c1c21661/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/reef/2c01458affee2b2435cd231b6e392435c1c21661/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/reef/2c01458affee2b2435cd231b6e392435c1c21661/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-13T14:01:36.387 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-09-13T14:01:36.387 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-13T14:01:36.417 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-13T14:01:36.418 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-13T14:01:36.418 INFO:teuthology.packaging:ref: None 2024-09-13T14:01:36.418 INFO:teuthology.packaging:tag: None 2024-09-13T14:01:36.418 INFO:teuthology.packaging:branch: reef 2024-09-13T14:01:36.418 INFO:teuthology.packaging:sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:01:36.418 DEBUG:teuthology.orchestra.run.smithi102:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/reef/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-13T14:01:36.426 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/reef/2c01458affee2b2435cd231b6e392435c1c21661/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/reef/2c01458affee2b2435cd231b6e392435c1c21661/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/reef/2c01458affee2b2435cd231b6e392435c1c21661/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-13T14:01:36.426 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-09-13T14:01:36.426 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-13T14:01:36.465 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-13T14:01:36.465 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-13T14:01:36.465 INFO:teuthology.packaging:ref: None 2024-09-13T14:01:36.465 INFO:teuthology.packaging:tag: None 2024-09-13T14:01:36.465 INFO:teuthology.packaging:branch: reef 2024-09-13T14:01:36.465 INFO:teuthology.packaging:sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:01:36.465 DEBUG:teuthology.orchestra.run.smithi195:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/reef/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-13T14:01:36.486 DEBUG:teuthology.orchestra.run.smithi102:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-13T14:01:36.531 DEBUG:teuthology.orchestra.run.smithi195:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-13T14:01:36.564 DEBUG:teuthology.orchestra.run.smithi102:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-13T14:01:36.605 DEBUG:teuthology.orchestra.run.smithi195:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-13T14:01:36.629 INFO:teuthology.orchestra.run.smithi102.stdout:check_obsoletes = 1 2024-09-13T14:01:36.631 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum clean all 2024-09-13T14:01:36.670 INFO:teuthology.orchestra.run.smithi195.stdout:check_obsoletes = 1 2024-09-13T14:01:36.672 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum clean all 2024-09-13T14:01:36.854 INFO:teuthology.orchestra.run.smithi102.stdout:45 files removed 2024-09-13T14:01:36.882 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-radosgw 2024-09-13T14:01:36.899 INFO:teuthology.orchestra.run.smithi195.stdout:45 files removed 2024-09-13T14:01:36.928 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-radosgw 2024-09-13T14:01:37.436 INFO:teuthology.orchestra.run.smithi102.stdout:ceph packages for x86_64 299 kB/s | 77 kB 00:00 2024-09-13T14:01:37.501 INFO:teuthology.orchestra.run.smithi195.stdout:ceph packages for x86_64 290 kB/s | 77 kB 00:00 2024-09-13T14:01:37.719 INFO:teuthology.orchestra.run.smithi102.stdout:ceph noarch packages 48 kB/s | 11 kB 00:00 2024-09-13T14:01:37.835 INFO:teuthology.orchestra.run.smithi195.stdout:ceph noarch packages 39 kB/s | 11 kB 00:00 2024-09-13T14:01:38.028 INFO:teuthology.orchestra.run.smithi102.stdout:ceph source packages 6.9 kB/s | 1.9 kB 00:00 2024-09-13T14:01:38.136 INFO:teuthology.orchestra.run.smithi195.stdout:ceph source packages 7.2 kB/s | 1.9 kB 00:00 2024-09-13T14:01:38.520 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.2 MB 00:00 2024-09-13T14:01:38.886 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.2 MB 00:00 2024-09-13T14:01:42.371 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - AppStream 7.4 MB/s | 20 MB 00:02 2024-09-13T14:01:43.395 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - AppStream 5.9 MB/s | 20 MB 00:03 2024-09-13T14:01:48.896 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 6.5 MB 00:01 2024-09-13T14:01:50.322 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - Extras packages 88 kB/s | 19 kB 00:00 2024-09-13T14:01:51.959 INFO:teuthology.orchestra.run.smithi195.stdout:Extra Packages for Enterprise Linux 15 MB/s | 23 MB 00:01 2024-09-13T14:01:52.180 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - CRB 1.1 MB/s | 6.5 MB 00:05 2024-09-13T14:01:53.681 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - Extras packages 70 kB/s | 19 kB 00:00 2024-09-13T14:01:55.010 INFO:teuthology.orchestra.run.smithi102.stdout:Extra Packages for Enterprise Linux 19 MB/s | 23 MB 00:01 2024-09-13T14:01:57.557 INFO:teuthology.orchestra.run.smithi195.stdout:lab-extras 5.5 kB/s | 1.7 kB 00:00 2024-09-13T14:01:59.977 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-09-13T14:01:59.978 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T14:01:59.979 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-09-13T14:01:59.979 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T14:01:59.979 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-09-13T14:01:59.979 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-radosgw x86_64 2:18.2.4-843.g2c01458a.el9 ceph 7.7 M 2024-09-13T14:01:59.979 INFO:teuthology.orchestra.run.smithi195.stdout:Upgrading: 2024-09-13T14:01:59.979 INFO:teuthology.orchestra.run.smithi195.stdout: librados2 x86_64 2:18.2.4-843.g2c01458a.el9 ceph 3.3 M 2024-09-13T14:01:59.979 INFO:teuthology.orchestra.run.smithi195.stdout: librbd1 x86_64 2:18.2.4-843.g2c01458a.el9 ceph 3.0 M 2024-09-13T14:01:59.979 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-09-13T14:01:59.979 INFO:teuthology.orchestra.run.smithi195.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-13T14:01:59.979 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-base x86_64 2:18.2.4-843.g2c01458a.el9 ceph 5.1 M 2024-09-13T14:01:59.979 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-common x86_64 2:18.2.4-843.g2c01458a.el9 ceph 18 M 2024-09-13T14:01:59.980 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-selinux x86_64 2:18.2.4-843.g2c01458a.el9 ceph 25 k 2024-09-13T14:01:59.980 INFO:teuthology.orchestra.run.smithi195.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-09-13T14:01:59.980 INFO:teuthology.orchestra.run.smithi195.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-13T14:01:59.980 INFO:teuthology.orchestra.run.smithi195.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-13T14:01:59.980 INFO:teuthology.orchestra.run.smithi195.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-13T14:01:59.980 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs2 x86_64 2:18.2.4-843.g2c01458a.el9 ceph 712 k 2024-09-13T14:01:59.980 INFO:teuthology.orchestra.run.smithi195.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-13T14:01:59.980 INFO:teuthology.orchestra.run.smithi195.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-13T14:01:59.980 INFO:teuthology.orchestra.run.smithi195.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-13T14:01:59.980 INFO:teuthology.orchestra.run.smithi195.stdout: libradosstriper1 x86_64 2:18.2.4-843.g2c01458a.el9 ceph 477 k 2024-09-13T14:01:59.980 INFO:teuthology.orchestra.run.smithi195.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-13T14:01:59.981 INFO:teuthology.orchestra.run.smithi195.stdout: librgw2 x86_64 2:18.2.4-843.g2c01458a.el9 ceph 4.5 M 2024-09-13T14:01:59.981 INFO:teuthology.orchestra.run.smithi195.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-13T14:01:59.981 INFO:teuthology.orchestra.run.smithi195.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-13T14:01:59.981 INFO:teuthology.orchestra.run.smithi195.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-13T14:01:59.981 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ceph-argparse x86_64 2:18.2.4-843.g2c01458a.el9 ceph 45 k 2024-09-13T14:01:59.981 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ceph-common x86_64 2:18.2.4-843.g2c01458a.el9 ceph 129 k 2024-09-13T14:01:59.981 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cephfs x86_64 2:18.2.4-843.g2c01458a.el9 ceph 162 k 2024-09-13T14:01:59.981 INFO:teuthology.orchestra.run.smithi195.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-13T14:01:59.981 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rados x86_64 2:18.2.4-843.g2c01458a.el9 ceph 321 k 2024-09-13T14:01:59.981 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rbd x86_64 2:18.2.4-843.g2c01458a.el9 ceph 301 k 2024-09-13T14:01:59.981 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rgw x86_64 2:18.2.4-843.g2c01458a.el9 ceph 99 k 2024-09-13T14:01:59.981 INFO:teuthology.orchestra.run.smithi195.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-13T14:01:59.981 INFO:teuthology.orchestra.run.smithi195.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-13T14:01:59.982 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:01:59.982 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-09-13T14:01:59.982 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T14:01:59.982 INFO:teuthology.orchestra.run.smithi195.stdout:Install 28 Packages 2024-09-13T14:01:59.982 INFO:teuthology.orchestra.run.smithi195.stdout:Upgrade 2 Packages 2024-09-13T14:01:59.982 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:01:59.982 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 53 M 2024-09-13T14:01:59.982 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-09-13T14:02:00.490 INFO:teuthology.orchestra.run.smithi102.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-13T14:02:00.608 INFO:teuthology.orchestra.run.smithi195.stdout:(1/30): ceph-base-18.2.4-843.g2c01458a.el9.x86_ 9.2 MB/s | 5.1 MB 00:00 2024-09-13T14:02:00.658 INFO:teuthology.orchestra.run.smithi195.stdout:(2/30): ceph-selinux-18.2.4-843.g2c01458a.el9.x 504 kB/s | 25 kB 00:00 2024-09-13T14:02:00.750 INFO:teuthology.orchestra.run.smithi195.stdout:(3/30): ceph-radosgw-18.2.4-843.g2c01458a.el9.x 11 MB/s | 7.7 MB 00:00 2024-09-13T14:02:00.784 INFO:teuthology.orchestra.run.smithi195.stdout:(4/30): libcephfs2-18.2.4-843.g2c01458a.el9.x86 5.6 MB/s | 712 kB 00:00 2024-09-13T14:02:00.826 INFO:teuthology.orchestra.run.smithi195.stdout:(5/30): libradosstriper1-18.2.4-843.g2c01458a.e 6.2 MB/s | 477 kB 00:00 2024-09-13T14:02:00.884 INFO:teuthology.orchestra.run.smithi195.stdout:(6/30): python3-ceph-argparse-18.2.4-843.g2c014 772 kB/s | 45 kB 00:00 2024-09-13T14:02:00.952 INFO:teuthology.orchestra.run.smithi195.stdout:(7/30): python3-ceph-common-18.2.4-843.g2c01458 1.9 MB/s | 129 kB 00:00 2024-09-13T14:02:01.143 INFO:teuthology.orchestra.run.smithi195.stdout:(8/30): ceph-common-18.2.4-843.g2c01458a.el9.x8 17 MB/s | 18 MB 00:01 2024-09-13T14:02:01.168 INFO:teuthology.orchestra.run.smithi195.stdout:(9/30): python3-cephfs-18.2.4-843.g2c01458a.el9 747 kB/s | 162 kB 00:00 2024-09-13T14:02:01.227 INFO:teuthology.orchestra.run.smithi195.stdout:(10/30): librgw2-18.2.4-843.g2c01458a.el9.x86_6 10 MB/s | 4.5 MB 00:00 2024-09-13T14:02:01.253 INFO:teuthology.orchestra.run.smithi195.stdout:(11/30): python3-rados-18.2.4-843.g2c01458a.el9 2.9 MB/s | 321 kB 00:00 2024-09-13T14:02:01.278 INFO:teuthology.orchestra.run.smithi195.stdout:(12/30): python3-rgw-18.2.4-843.g2c01458a.el9.x 1.9 MB/s | 99 kB 00:00 2024-09-13T14:02:01.303 INFO:teuthology.orchestra.run.smithi195.stdout:(13/30): python3-rbd-18.2.4-843.g2c01458a.el9.x 2.2 MB/s | 301 kB 00:00 2024-09-13T14:02:01.395 INFO:teuthology.orchestra.run.smithi195.stdout:(14/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 493 kB/s | 45 kB 00:00 2024-09-13T14:02:01.420 INFO:teuthology.orchestra.run.smithi195.stdout:(15/30): boost-program-options-1.75.0-8.el9.x86 639 kB/s | 107 kB 00:00 2024-09-13T14:02:01.446 INFO:teuthology.orchestra.run.smithi195.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 956 kB/s | 160 kB 00:00 2024-09-13T14:02:01.496 INFO:teuthology.orchestra.run.smithi195.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 848 kB/s | 42 kB 00:00 2024-09-13T14:02:01.522 INFO:teuthology.orchestra.run.smithi195.stdout:(18/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 292 kB 00:00 2024-09-13T14:02:01.555 INFO:teuthology.orchestra.run.smithi195.stdout:(19/30): librdkafka-1.6.1-102.el9.x86_64.rpm 4.1 MB/s | 662 kB 00:00 2024-09-13T14:02:01.613 INFO:teuthology.orchestra.run.smithi195.stdout:(20/30): gperftools-libs-2.9.1-3.el9.x86_64.rpm 3.3 MB/s | 308 kB 00:00 2024-09-13T14:02:01.638 INFO:teuthology.orchestra.run.smithi195.stdout:(21/30): fmt-8.1.1-5.el9.x86_64.rpm 779 kB/s | 111 kB 00:00 2024-09-13T14:02:01.664 INFO:teuthology.orchestra.run.smithi195.stdout:(22/30): libarrow-doc-9.0.0-11.el9.noarch.rpm 515 kB/s | 26 kB 00:00 2024-09-13T14:02:01.689 INFO:teuthology.orchestra.run.smithi195.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 964 kB/s | 49 kB 00:00 2024-09-13T14:02:01.714 INFO:teuthology.orchestra.run.smithi195.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-13T14:02:01.764 INFO:teuthology.orchestra.run.smithi195.stdout:(25/30): re2-20211101-20.el9.x86_64.rpm 3.8 MB/s | 191 kB 00:00 2024-09-13T14:02:01.807 INFO:teuthology.orchestra.run.smithi195.stdout:(26/30): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.0 MB/s | 839 kB 00:00 2024-09-13T14:02:01.990 INFO:teuthology.orchestra.run.smithi195.stdout:(27/30): librados2-18.2.4-843.g2c01458a.el9.x86 18 MB/s | 3.3 MB 00:00 2024-09-13T14:02:02.032 INFO:teuthology.orchestra.run.smithi195.stdout:(28/30): thrift-0.15.0-2.el9.x86_64.rpm 5.9 MB/s | 1.6 MB 00:00 2024-09-13T14:02:02.141 INFO:teuthology.orchestra.run.smithi195.stdout:(29/30): librbd1-18.2.4-843.g2c01458a.el9.x86_6 20 MB/s | 3.0 MB 00:00 2024-09-13T14:02:02.241 INFO:teuthology.orchestra.run.smithi195.stdout:(30/30): libarrow-9.0.0-11.el9.x86_64.rpm 6.6 MB/s | 4.4 MB 00:00 2024-09-13T14:02:02.245 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:02:02.245 INFO:teuthology.orchestra.run.smithi195.stdout:Total 23 MB/s | 53 MB 00:02 2024-09-13T14:02:02.312 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-09-13T14:02:02.346 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-09-13T14:02:02.346 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-09-13T14:02:02.796 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-09-13T14:02:02.797 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-09-13T14:02:02.938 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-09-13T14:02:02.940 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T14:02:02.940 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-09-13T14:02:02.940 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T14:02:02.940 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-09-13T14:02:02.941 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-radosgw x86_64 2:18.2.4-843.g2c01458a.el9 ceph 7.7 M 2024-09-13T14:02:02.941 INFO:teuthology.orchestra.run.smithi102.stdout:Upgrading: 2024-09-13T14:02:02.941 INFO:teuthology.orchestra.run.smithi102.stdout: librados2 x86_64 2:18.2.4-843.g2c01458a.el9 ceph 3.3 M 2024-09-13T14:02:02.941 INFO:teuthology.orchestra.run.smithi102.stdout: librbd1 x86_64 2:18.2.4-843.g2c01458a.el9 ceph 3.0 M 2024-09-13T14:02:02.941 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-09-13T14:02:02.941 INFO:teuthology.orchestra.run.smithi102.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-13T14:02:02.941 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-base x86_64 2:18.2.4-843.g2c01458a.el9 ceph 5.1 M 2024-09-13T14:02:02.941 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-common x86_64 2:18.2.4-843.g2c01458a.el9 ceph 18 M 2024-09-13T14:02:02.941 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-selinux x86_64 2:18.2.4-843.g2c01458a.el9 ceph 25 k 2024-09-13T14:02:02.941 INFO:teuthology.orchestra.run.smithi102.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-09-13T14:02:02.941 INFO:teuthology.orchestra.run.smithi102.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-13T14:02:02.941 INFO:teuthology.orchestra.run.smithi102.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-13T14:02:02.941 INFO:teuthology.orchestra.run.smithi102.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-13T14:02:02.942 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs2 x86_64 2:18.2.4-843.g2c01458a.el9 ceph 712 k 2024-09-13T14:02:02.942 INFO:teuthology.orchestra.run.smithi102.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-13T14:02:02.942 INFO:teuthology.orchestra.run.smithi102.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-13T14:02:02.942 INFO:teuthology.orchestra.run.smithi102.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-13T14:02:02.942 INFO:teuthology.orchestra.run.smithi102.stdout: libradosstriper1 x86_64 2:18.2.4-843.g2c01458a.el9 ceph 477 k 2024-09-13T14:02:02.942 INFO:teuthology.orchestra.run.smithi102.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-13T14:02:02.942 INFO:teuthology.orchestra.run.smithi102.stdout: librgw2 x86_64 2:18.2.4-843.g2c01458a.el9 ceph 4.5 M 2024-09-13T14:02:02.942 INFO:teuthology.orchestra.run.smithi102.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-13T14:02:02.942 INFO:teuthology.orchestra.run.smithi102.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-13T14:02:02.942 INFO:teuthology.orchestra.run.smithi102.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-13T14:02:02.942 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-argparse x86_64 2:18.2.4-843.g2c01458a.el9 ceph 45 k 2024-09-13T14:02:02.942 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-common x86_64 2:18.2.4-843.g2c01458a.el9 ceph 129 k 2024-09-13T14:02:02.942 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cephfs x86_64 2:18.2.4-843.g2c01458a.el9 ceph 162 k 2024-09-13T14:02:02.943 INFO:teuthology.orchestra.run.smithi102.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-13T14:02:02.943 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rados x86_64 2:18.2.4-843.g2c01458a.el9 ceph 321 k 2024-09-13T14:02:02.943 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rbd x86_64 2:18.2.4-843.g2c01458a.el9 ceph 301 k 2024-09-13T14:02:02.943 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rgw x86_64 2:18.2.4-843.g2c01458a.el9 ceph 99 k 2024-09-13T14:02:02.943 INFO:teuthology.orchestra.run.smithi102.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-13T14:02:02.943 INFO:teuthology.orchestra.run.smithi102.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-13T14:02:02.943 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:02:02.943 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-09-13T14:02:02.943 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T14:02:02.943 INFO:teuthology.orchestra.run.smithi102.stdout:Install 28 Packages 2024-09-13T14:02:02.943 INFO:teuthology.orchestra.run.smithi102.stdout:Upgrade 2 Packages 2024-09-13T14:02:02.943 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:02:02.944 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 53 M 2024-09-13T14:02:02.944 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-09-13T14:02:03.917 INFO:teuthology.orchestra.run.smithi102.stdout:(1/30): ceph-base-18.2.4-843.g2c01458a.el9.x86_ 9.1 MB/s | 5.1 MB 00:00 2024-09-13T14:02:04.034 INFO:teuthology.orchestra.run.smithi102.stdout:(2/30): ceph-radosgw-18.2.4-843.g2c01458a.el9.x 11 MB/s | 7.7 MB 00:00 2024-09-13T14:02:04.059 INFO:teuthology.orchestra.run.smithi102.stdout:(3/30): ceph-selinux-18.2.4-843.g2c01458a.el9.x 177 kB/s | 25 kB 00:00 2024-09-13T14:02:04.093 INFO:teuthology.orchestra.run.smithi102.stdout:(4/30): libcephfs2-18.2.4-843.g2c01458a.el9.x86 12 MB/s | 712 kB 00:00 2024-09-13T14:02:04.123 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-09-13T14:02:04.126 INFO:teuthology.orchestra.run.smithi102.stdout:(5/30): libradosstriper1-18.2.4-843.g2c01458a.e 7.0 MB/s | 477 kB 00:00 2024-09-13T14:02:04.192 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-09-13T14:02:04.193 INFO:teuthology.orchestra.run.smithi102.stdout:(6/30): python3-ceph-argparse-18.2.4-843.g2c014 677 kB/s | 45 kB 00:00 2024-09-13T14:02:04.221 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-09-13T14:02:04.279 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-09-13T14:02:04.302 INFO:teuthology.orchestra.run.smithi102.stdout:(7/30): python3-ceph-common-18.2.4-843.g2c01458 1.2 MB/s | 129 kB 00:00 2024-09-13T14:02:04.544 INFO:teuthology.orchestra.run.smithi102.stdout:(8/30): ceph-common-18.2.4-843.g2c01458a.el9.x8 15 MB/s | 18 MB 00:01 2024-09-13T14:02:04.569 INFO:teuthology.orchestra.run.smithi102.stdout:(9/30): python3-cephfs-18.2.4-843.g2c01458a.el9 608 kB/s | 162 kB 00:00 2024-09-13T14:02:04.590 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-09-13T14:02:04.636 INFO:teuthology.orchestra.run.smithi102.stdout:(10/30): librgw2-18.2.4-843.g2c01458a.el9.x86_6 8.2 MB/s | 4.5 MB 00:00 2024-09-13T14:02:04.661 INFO:teuthology.orchestra.run.smithi102.stdout:(11/30): python3-rbd-18.2.4-843.g2c01458a.el9.x 3.2 MB/s | 301 kB 00:00 2024-09-13T14:02:04.687 INFO:teuthology.orchestra.run.smithi102.stdout:(12/30): python3-rados-18.2.4-843.g2c01458a.el9 2.2 MB/s | 321 kB 00:00 2024-09-13T14:02:04.697 INFO:teuthology.orchestra.run.smithi195.stdout: Upgrading : librados2-2:18.2.4-843.g2c01458a.el9.x86_64 5/32 2024-09-13T14:02:04.712 INFO:teuthology.orchestra.run.smithi102.stdout:(13/30): python3-rgw-18.2.4-843.g2c01458a.el9.x 1.3 MB/s | 99 kB 00:00 2024-09-13T14:02:04.788 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librados2-2:18.2.4-843.g2c01458a.el9.x86_64 5/32 2024-09-13T14:02:04.804 INFO:teuthology.orchestra.run.smithi102.stdout:(14/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 494 kB/s | 45 kB 00:00 2024-09-13T14:02:04.829 INFO:teuthology.orchestra.run.smithi102.stdout:(15/30): boost-program-options-1.75.0-8.el9.x86 638 kB/s | 107 kB 00:00 2024-09-13T14:02:04.854 INFO:teuthology.orchestra.run.smithi102.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 958 kB/s | 160 kB 00:00 2024-09-13T14:02:04.872 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rados-2:18.2.4-843.g2c01458a.el9.x86_64 6/32 2024-09-13T14:02:04.896 INFO:teuthology.orchestra.run.smithi102.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-09-13T14:02:04.907 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libcephfs2-2:18.2.4-843.g2c01458a.el9.x86_64 7/32 2024-09-13T14:02:04.921 INFO:teuthology.orchestra.run.smithi102.stdout:(18/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.1 MB/s | 292 kB 00:00 2024-09-13T14:02:04.947 INFO:teuthology.orchestra.run.smithi102.stdout:(19/30): librdkafka-1.6.1-102.el9.x86_64.rpm 4.5 MB/s | 662 kB 00:00 2024-09-13T14:02:04.969 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libcephfs2-2:18.2.4-843.g2c01458a.el9.x86_64 7/32 2024-09-13T14:02:04.997 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-09-13T14:02:05.006 INFO:teuthology.orchestra.run.smithi102.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 1.0 MB/s | 111 kB 00:00 2024-09-13T14:02:05.039 INFO:teuthology.orchestra.run.smithi102.stdout:(21/30): libarrow-doc-9.0.0-11.el9.noarch.rpm 823 kB/s | 26 kB 00:00 2024-09-13T14:02:05.041 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-ceph-argparse-2:18.2.4-843.g2c01458a.el9.x 9/32 2024-09-13T14:02:05.064 INFO:teuthology.orchestra.run.smithi102.stdout:(22/30): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.1 MB/s | 308 kB 00:00 2024-09-13T14:02:05.089 INFO:teuthology.orchestra.run.smithi102.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 971 kB/s | 49 kB 00:00 2024-09-13T14:02:05.107 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cephfs-2:18.2.4-843.g2c01458a.el9.x86_64 10/32 2024-09-13T14:02:05.115 INFO:teuthology.orchestra.run.smithi102.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-13T14:02:05.133 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libradosstriper1-2:18.2.4-843.g2c01458a.el9.x86_64 11/32 2024-09-13T14:02:05.165 INFO:teuthology.orchestra.run.smithi102.stdout:(25/30): re2-20211101-20.el9.x86_64.rpm 3.7 MB/s | 191 kB 00:00 2024-09-13T14:02:05.182 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libradosstriper1-2:18.2.4-843.g2c01458a.el9.x86_64 11/32 2024-09-13T14:02:05.232 INFO:teuthology.orchestra.run.smithi102.stdout:(26/30): parquet-libs-9.0.0-11.el9.x86_64.rpm 5.8 MB/s | 839 kB 00:00 2024-09-13T14:02:05.289 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : re2-1:20211101-20.el9.x86_64 12/32 2024-09-13T14:02:05.332 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 13/32 2024-09-13T14:02:05.373 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-09-13T14:02:05.374 INFO:teuthology.orchestra.run.smithi102.stdout:(27/30): librados2-18.2.4-843.g2c01458a.el9.x86 23 MB/s | 3.3 MB 00:00 2024-09-13T14:02:05.406 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/32 2024-09-13T14:02:05.407 INFO:teuthology.orchestra.run.smithi102.stdout:(28/30): thrift-0.15.0-2.el9.x86_64.rpm 6.5 MB/s | 1.6 MB 00:00 2024-09-13T14:02:05.433 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 16/32 2024-09-13T14:02:05.477 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-09-13T14:02:05.516 INFO:teuthology.orchestra.run.smithi102.stdout:(29/30): librbd1-18.2.4-843.g2c01458a.el9.x86_6 21 MB/s | 3.0 MB 00:00 2024-09-13T14:02:05.526 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-09-13T14:02:05.583 INFO:teuthology.orchestra.run.smithi102.stdout:(30/30): libarrow-9.0.0-11.el9.x86_64.rpm 7.1 MB/s | 4.4 MB 00:00 2024-09-13T14:02:05.587 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:02:05.587 INFO:teuthology.orchestra.run.smithi102.stdout:Total 20 MB/s | 53 MB 00:02 2024-09-13T14:02:05.662 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-09-13T14:02:05.697 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-09-13T14:02:05.698 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-09-13T14:02:05.827 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-09-13T14:02:05.873 INFO:teuthology.orchestra.run.smithi195.stdout: Upgrading : librbd1-2:18.2.4-843.g2c01458a.el9.x86_64 20/32 2024-09-13T14:02:05.947 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librbd1-2:18.2.4-843.g2c01458a.el9.x86_64 20/32 2024-09-13T14:02:05.978 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rbd-2:18.2.4-843.g2c01458a.el9.x86_64 21/32 2024-09-13T14:02:06.013 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-09-13T14:02:06.166 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-09-13T14:02:06.167 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-09-13T14:02:06.419 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 23/32 2024-09-13T14:02:06.442 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librgw2-2:18.2.4-843.g2c01458a.el9.x86_64 24/32 2024-09-13T14:02:06.497 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librgw2-2:18.2.4-843.g2c01458a.el9.x86_64 24/32 2024-09-13T14:02:06.573 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rgw-2:18.2.4-843.g2c01458a.el9.x86_64 25/32 2024-09-13T14:02:06.603 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-ceph-common-2:18.2.4-843.g2c01458a.el9.x86 26/32 2024-09-13T14:02:07.513 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-09-13T14:02:07.592 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-09-13T14:02:07.623 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-09-13T14:02:07.673 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-09-13T14:02:08.044 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-09-13T14:02:08.105 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : librados2-2:18.2.4-843.g2c01458a.el9.x86_64 5/32 2024-09-13T14:02:08.197 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librados2-2:18.2.4-843.g2c01458a.el9.x86_64 5/32 2024-09-13T14:02:08.281 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rados-2:18.2.4-843.g2c01458a.el9.x86_64 6/32 2024-09-13T14:02:08.306 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libcephfs2-2:18.2.4-843.g2c01458a.el9.x86_64 7/32 2024-09-13T14:02:08.369 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libcephfs2-2:18.2.4-843.g2c01458a.el9.x86_64 7/32 2024-09-13T14:02:08.398 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-09-13T14:02:08.441 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-ceph-argparse-2:18.2.4-843.g2c01458a.el9.x 9/32 2024-09-13T14:02:08.507 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cephfs-2:18.2.4-843.g2c01458a.el9.x86_64 10/32 2024-09-13T14:02:08.534 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libradosstriper1-2:18.2.4-843.g2c01458a.el9.x86_64 11/32 2024-09-13T14:02:08.600 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libradosstriper1-2:18.2.4-843.g2c01458a.el9.x86_64 11/32 2024-09-13T14:02:08.688 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-common-2:18.2.4-843.g2c01458a.el9.x86_64 27/32 2024-09-13T14:02:08.715 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : re2-1:20211101-20.el9.x86_64 12/32 2024-09-13T14:02:08.716 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-common-2:18.2.4-843.g2c01458a.el9.x86_64 27/32 2024-09-13T14:02:08.750 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 13/32 2024-09-13T14:02:08.795 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-09-13T14:02:08.831 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/32 2024-09-13T14:02:08.858 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 16/32 2024-09-13T14:02:08.894 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-09-13T14:02:08.927 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-09-13T14:02:09.227 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-09-13T14:02:09.267 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-common-2:18.2.4-843.g2c01458a.el9.x86_64 27/32 2024-09-13T14:02:09.273 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : librbd1-2:18.2.4-843.g2c01458a.el9.x86_64 20/32 2024-09-13T14:02:09.311 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-base-2:18.2.4-843.g2c01458a.el9.x86_64 28/32 2024-09-13T14:02:09.346 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librbd1-2:18.2.4-843.g2c01458a.el9.x86_64 20/32 2024-09-13T14:02:09.378 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rbd-2:18.2.4-843.g2c01458a.el9.x86_64 21/32 2024-09-13T14:02:09.399 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-base-2:18.2.4-843.g2c01458a.el9.x86_64 28/32 2024-09-13T14:02:09.400 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-13T14:02:09.400 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-13T14:02:09.400 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:02:09.422 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-09-13T14:02:09.438 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-selinux-2:18.2.4-843.g2c01458a.el9.x86_64 29/32 2024-09-13T14:02:09.828 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 23/32 2024-09-13T14:02:09.867 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librgw2-2:18.2.4-843.g2c01458a.el9.x86_64 24/32 2024-09-13T14:02:09.921 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librgw2-2:18.2.4-843.g2c01458a.el9.x86_64 24/32 2024-09-13T14:02:09.999 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rgw-2:18.2.4-843.g2c01458a.el9.x86_64 25/32 2024-09-13T14:02:10.028 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-ceph-common-2:18.2.4-843.g2c01458a.el9.x86 26/32 2024-09-13T14:02:12.132 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-common-2:18.2.4-843.g2c01458a.el9.x86_64 27/32 2024-09-13T14:02:12.166 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-common-2:18.2.4-843.g2c01458a.el9.x86_64 27/32 2024-09-13T14:02:12.728 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-common-2:18.2.4-843.g2c01458a.el9.x86_64 27/32 2024-09-13T14:02:12.762 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-base-2:18.2.4-843.g2c01458a.el9.x86_64 28/32 2024-09-13T14:02:12.858 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-base-2:18.2.4-843.g2c01458a.el9.x86_64 28/32 2024-09-13T14:02:12.859 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-13T14:02:12.859 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-13T14:02:12.859 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:02:12.888 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-selinux-2:18.2.4-843.g2c01458a.el9.x86_64 29/32 2024-09-13T14:02:19.154 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-selinux-2:18.2.4-843.g2c01458a.el9.x86_64 29/32 2024-09-13T14:02:19.154 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /sys 2024-09-13T14:02:19.154 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /proc 2024-09-13T14:02:19.154 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /mnt 2024-09-13T14:02:19.154 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /var/tmp 2024-09-13T14:02:19.154 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /home 2024-09-13T14:02:19.154 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /root 2024-09-13T14:02:19.155 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /tmp 2024-09-13T14:02:19.155 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:02:19.886 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-radosgw-2:18.2.4-843.g2c01458a.el9.x86_64 30/32 2024-09-13T14:02:19.911 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-radosgw-2:18.2.4-843.g2c01458a.el9.x86_64 30/32 2024-09-13T14:02:19.911 INFO:teuthology.orchestra.run.smithi195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:02:19.911 INFO:teuthology.orchestra.run.smithi195.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-13T14:02:19.912 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T14:02:19.912 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T14:02:19.912 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:02:19.913 INFO:teuthology.orchestra.run.smithi195.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-09-13T14:02:19.996 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-09-13T14:02:19.997 INFO:teuthology.orchestra.run.smithi195.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-09-13T14:02:21.100 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-09-13T14:02:21.100 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-base-2:18.2.4-843.g2c01458a.el9.x86_64 1/32 2024-09-13T14:02:21.100 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-common-2:18.2.4-843.g2c01458a.el9.x86_64 2/32 2024-09-13T14:02:21.100 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-radosgw-2:18.2.4-843.g2c01458a.el9.x86_64 3/32 2024-09-13T14:02:21.100 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-selinux-2:18.2.4-843.g2c01458a.el9.x86_64 4/32 2024-09-13T14:02:21.101 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libcephfs2-2:18.2.4-843.g2c01458a.el9.x86_64 5/32 2024-09-13T14:02:21.101 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libradosstriper1-2:18.2.4-843.g2c01458a.el9.x86_64 6/32 2024-09-13T14:02:21.101 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librgw2-2:18.2.4-843.g2c01458a.el9.x86_64 7/32 2024-09-13T14:02:21.101 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-ceph-argparse-2:18.2.4-843.g2c01458a.el9.x 8/32 2024-09-13T14:02:21.101 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-ceph-common-2:18.2.4-843.g2c01458a.el9.x86 9/32 2024-09-13T14:02:21.101 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cephfs-2:18.2.4-843.g2c01458a.el9.x86_64 10/32 2024-09-13T14:02:21.101 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rados-2:18.2.4-843.g2c01458a.el9.x86_64 11/32 2024-09-13T14:02:21.101 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rbd-2:18.2.4-843.g2c01458a.el9.x86_64 12/32 2024-09-13T14:02:21.101 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rgw-2:18.2.4-843.g2c01458a.el9.x86_64 13/32 2024-09-13T14:02:21.101 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-09-13T14:02:21.101 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-09-13T14:02:21.101 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-09-13T14:02:21.101 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-09-13T14:02:21.102 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-09-13T14:02:21.102 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-09-13T14:02:21.102 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-09-13T14:02:21.102 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 21/32 2024-09-13T14:02:21.102 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 22/32 2024-09-13T14:02:21.102 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 23/32 2024-09-13T14:02:21.102 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-09-13T14:02:21.102 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-09-13T14:02:21.102 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 26/32 2024-09-13T14:02:21.102 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : re2-1:20211101-20.el9.x86_64 27/32 2024-09-13T14:02:21.102 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-09-13T14:02:21.102 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librados2-2:18.2.4-843.g2c01458a.el9.x86_64 29/32 2024-09-13T14:02:21.102 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-09-13T14:02:21.103 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librbd1-2:18.2.4-843.g2c01458a.el9.x86_64 31/32 2024-09-13T14:02:22.041 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-09-13T14:02:22.041 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:02:22.041 INFO:teuthology.orchestra.run.smithi195.stdout:Upgraded: 2024-09-13T14:02:22.041 INFO:teuthology.orchestra.run.smithi195.stdout: librados2-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:02:22.041 INFO:teuthology.orchestra.run.smithi195.stdout: librbd1-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:02:22.041 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-09-13T14:02:22.041 INFO:teuthology.orchestra.run.smithi195.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-13T14:02:22.041 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-base-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:02:22.042 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-common-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:02:22.042 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-radosgw-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:02:22.042 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-selinux-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:02:22.042 INFO:teuthology.orchestra.run.smithi195.stdout: fmt-8.1.1-5.el9.x86_64 2024-09-13T14:02:22.042 INFO:teuthology.orchestra.run.smithi195.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-13T14:02:22.042 INFO:teuthology.orchestra.run.smithi195.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-13T14:02:22.042 INFO:teuthology.orchestra.run.smithi195.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-13T14:02:22.042 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs2-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:02:22.042 INFO:teuthology.orchestra.run.smithi195.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-13T14:02:22.042 INFO:teuthology.orchestra.run.smithi195.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-13T14:02:22.042 INFO:teuthology.orchestra.run.smithi195.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-13T14:02:22.042 INFO:teuthology.orchestra.run.smithi195.stdout: libradosstriper1-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:02:22.042 INFO:teuthology.orchestra.run.smithi195.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-13T14:02:22.042 INFO:teuthology.orchestra.run.smithi195.stdout: librgw2-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:02:22.043 INFO:teuthology.orchestra.run.smithi195.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-13T14:02:22.043 INFO:teuthology.orchestra.run.smithi195.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-13T14:02:22.043 INFO:teuthology.orchestra.run.smithi195.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-13T14:02:22.043 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ceph-argparse-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:02:22.043 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ceph-common-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:02:22.043 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cephfs-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:02:22.043 INFO:teuthology.orchestra.run.smithi195.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-13T14:02:22.043 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rados-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:02:22.043 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rbd-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:02:22.043 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rgw-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:02:22.043 INFO:teuthology.orchestra.run.smithi195.stdout: re2-1:20211101-20.el9.x86_64 2024-09-13T14:02:22.043 INFO:teuthology.orchestra.run.smithi195.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-13T14:02:22.043 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:02:22.043 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-09-13T14:02:22.413 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-test 2024-09-13T14:02:22.954 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-selinux-2:18.2.4-843.g2c01458a.el9.x86_64 29/32 2024-09-13T14:02:22.954 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /sys 2024-09-13T14:02:22.954 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /proc 2024-09-13T14:02:22.954 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /mnt 2024-09-13T14:02:22.954 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /var/tmp 2024-09-13T14:02:22.954 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /home 2024-09-13T14:02:22.954 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /root 2024-09-13T14:02:22.954 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /tmp 2024-09-13T14:02:22.954 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:02:22.996 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:00:25 ago on Fri 13 Sep 2024 02:01:57 PM UTC. 2024-09-13T14:02:23.137 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-09-13T14:02:23.137 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T14:02:23.137 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-09-13T14:02:23.137 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T14:02:23.137 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-09-13T14:02:23.137 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-test x86_64 2:18.2.4-843.g2c01458a.el9 ceph 40 M 2024-09-13T14:02:23.137 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-09-13T14:02:23.138 INFO:teuthology.orchestra.run.smithi195.stdout: libcephsqlite x86_64 2:18.2.4-843.g2c01458a.el9 ceph 166 k 2024-09-13T14:02:23.138 INFO:teuthology.orchestra.run.smithi195.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-13T14:02:23.138 INFO:teuthology.orchestra.run.smithi195.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-13T14:02:23.138 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:02:23.138 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-09-13T14:02:23.138 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T14:02:23.138 INFO:teuthology.orchestra.run.smithi195.stdout:Install 4 Packages 2024-09-13T14:02:23.138 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:02:23.139 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 40 M 2024-09-13T14:02:23.139 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 164 M 2024-09-13T14:02:23.139 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-09-13T14:02:23.331 INFO:teuthology.orchestra.run.smithi195.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.1 MB/s | 304 kB 00:00 2024-09-13T14:02:23.373 INFO:teuthology.orchestra.run.smithi195.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-13T14:02:23.415 INFO:teuthology.orchestra.run.smithi195.stdout:(3/4): libcephsqlite-18.2.4-843.g2c01458a.el9.x 725 kB/s | 166 kB 00:00 2024-09-13T14:02:23.686 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-radosgw-2:18.2.4-843.g2c01458a.el9.x86_64 30/32 2024-09-13T14:02:23.711 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-radosgw-2:18.2.4-843.g2c01458a.el9.x86_64 30/32 2024-09-13T14:02:23.711 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:02:23.711 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-13T14:02:23.712 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T14:02:23.712 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T14:02:23.712 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:02:23.713 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-09-13T14:02:23.780 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-09-13T14:02:23.780 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-09-13T14:02:24.490 INFO:teuthology.orchestra.run.smithi195.stdout:(4/4): ceph-test-18.2.4-843.g2c01458a.el9.x86_6 30 MB/s | 40 MB 00:01 2024-09-13T14:02:24.492 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:02:24.492 INFO:teuthology.orchestra.run.smithi195.stdout:Total 30 MB/s | 40 MB 00:01 2024-09-13T14:02:24.520 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-09-13T14:02:24.539 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-09-13T14:02:24.540 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-09-13T14:02:24.874 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-09-13T14:02:24.874 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-09-13T14:02:24.894 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-09-13T14:02:24.894 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-base-2:18.2.4-843.g2c01458a.el9.x86_64 1/32 2024-09-13T14:02:24.894 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-common-2:18.2.4-843.g2c01458a.el9.x86_64 2/32 2024-09-13T14:02:24.894 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-radosgw-2:18.2.4-843.g2c01458a.el9.x86_64 3/32 2024-09-13T14:02:24.894 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-selinux-2:18.2.4-843.g2c01458a.el9.x86_64 4/32 2024-09-13T14:02:24.895 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libcephfs2-2:18.2.4-843.g2c01458a.el9.x86_64 5/32 2024-09-13T14:02:24.895 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libradosstriper1-2:18.2.4-843.g2c01458a.el9.x86_64 6/32 2024-09-13T14:02:24.895 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librgw2-2:18.2.4-843.g2c01458a.el9.x86_64 7/32 2024-09-13T14:02:24.895 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-ceph-argparse-2:18.2.4-843.g2c01458a.el9.x 8/32 2024-09-13T14:02:24.895 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-ceph-common-2:18.2.4-843.g2c01458a.el9.x86 9/32 2024-09-13T14:02:24.895 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cephfs-2:18.2.4-843.g2c01458a.el9.x86_64 10/32 2024-09-13T14:02:24.895 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rados-2:18.2.4-843.g2c01458a.el9.x86_64 11/32 2024-09-13T14:02:24.895 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rbd-2:18.2.4-843.g2c01458a.el9.x86_64 12/32 2024-09-13T14:02:24.895 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rgw-2:18.2.4-843.g2c01458a.el9.x86_64 13/32 2024-09-13T14:02:24.895 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-09-13T14:02:24.895 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-09-13T14:02:24.895 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-09-13T14:02:24.895 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-09-13T14:02:24.895 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-09-13T14:02:24.896 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-09-13T14:02:24.896 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-09-13T14:02:24.896 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 21/32 2024-09-13T14:02:24.896 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 22/32 2024-09-13T14:02:24.896 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 23/32 2024-09-13T14:02:24.896 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-09-13T14:02:24.896 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-09-13T14:02:24.896 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 26/32 2024-09-13T14:02:24.896 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : re2-1:20211101-20.el9.x86_64 27/32 2024-09-13T14:02:24.896 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-09-13T14:02:24.896 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librados2-2:18.2.4-843.g2c01458a.el9.x86_64 29/32 2024-09-13T14:02:24.896 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-09-13T14:02:24.897 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librbd1-2:18.2.4-843.g2c01458a.el9.x86_64 31/32 2024-09-13T14:02:25.345 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-09-13T14:02:25.426 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-13T14:02:25.472 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-13T14:02:25.500 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libcephsqlite-2:18.2.4-843.g2c01458a.el9.x86_64 3/4 2024-09-13T14:02:25.952 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-09-13T14:02:25.952 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:02:25.953 INFO:teuthology.orchestra.run.smithi102.stdout:Upgraded: 2024-09-13T14:02:25.953 INFO:teuthology.orchestra.run.smithi102.stdout: librados2-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:02:25.953 INFO:teuthology.orchestra.run.smithi102.stdout: librbd1-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:02:25.953 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-09-13T14:02:25.953 INFO:teuthology.orchestra.run.smithi102.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-13T14:02:25.953 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-base-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:02:25.953 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-common-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:02:25.953 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-radosgw-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:02:25.953 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-selinux-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:02:25.953 INFO:teuthology.orchestra.run.smithi102.stdout: fmt-8.1.1-5.el9.x86_64 2024-09-13T14:02:25.953 INFO:teuthology.orchestra.run.smithi102.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-13T14:02:25.953 INFO:teuthology.orchestra.run.smithi102.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-13T14:02:25.953 INFO:teuthology.orchestra.run.smithi102.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-13T14:02:25.954 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs2-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:02:25.954 INFO:teuthology.orchestra.run.smithi102.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-13T14:02:25.954 INFO:teuthology.orchestra.run.smithi102.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-13T14:02:25.954 INFO:teuthology.orchestra.run.smithi102.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-13T14:02:25.954 INFO:teuthology.orchestra.run.smithi102.stdout: libradosstriper1-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:02:25.954 INFO:teuthology.orchestra.run.smithi102.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-13T14:02:25.954 INFO:teuthology.orchestra.run.smithi102.stdout: librgw2-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:02:25.954 INFO:teuthology.orchestra.run.smithi102.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-13T14:02:25.954 INFO:teuthology.orchestra.run.smithi102.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-13T14:02:25.954 INFO:teuthology.orchestra.run.smithi102.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-13T14:02:25.954 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-argparse-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:02:25.954 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-common-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:02:25.954 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cephfs-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:02:25.955 INFO:teuthology.orchestra.run.smithi102.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-13T14:02:25.955 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rados-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:02:25.955 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rbd-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:02:25.955 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rgw-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:02:25.955 INFO:teuthology.orchestra.run.smithi102.stdout: re2-1:20211101-20.el9.x86_64 2024-09-13T14:02:25.955 INFO:teuthology.orchestra.run.smithi102.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-13T14:02:25.955 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:02:25.955 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-09-13T14:02:26.274 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-test 2024-09-13T14:02:26.852 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:00:26 ago on Fri 13 Sep 2024 02:02:00 PM UTC. 2024-09-13T14:02:26.985 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-09-13T14:02:26.985 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T14:02:26.986 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-09-13T14:02:26.986 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T14:02:26.986 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-09-13T14:02:26.986 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-test x86_64 2:18.2.4-843.g2c01458a.el9 ceph 40 M 2024-09-13T14:02:26.986 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-09-13T14:02:26.986 INFO:teuthology.orchestra.run.smithi102.stdout: libcephsqlite x86_64 2:18.2.4-843.g2c01458a.el9 ceph 166 k 2024-09-13T14:02:26.986 INFO:teuthology.orchestra.run.smithi102.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-13T14:02:26.986 INFO:teuthology.orchestra.run.smithi102.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-13T14:02:26.986 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:02:26.986 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-09-13T14:02:26.986 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T14:02:26.986 INFO:teuthology.orchestra.run.smithi102.stdout:Install 4 Packages 2024-09-13T14:02:26.986 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:02:26.987 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 40 M 2024-09-13T14:02:26.987 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 164 M 2024-09-13T14:02:26.987 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-09-13T14:02:27.223 INFO:teuthology.orchestra.run.smithi102.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.4 MB/s | 304 kB 00:00 2024-09-13T14:02:27.273 INFO:teuthology.orchestra.run.smithi102.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-13T14:02:27.299 INFO:teuthology.orchestra.run.smithi102.stdout:(3/4): libcephsqlite-18.2.4-843.g2c01458a.el9.x 572 kB/s | 166 kB 00:00 2024-09-13T14:02:28.932 INFO:teuthology.orchestra.run.smithi102.stdout:(4/4): ceph-test-18.2.4-843.g2c01458a.el9.x86_6 21 MB/s | 40 MB 00:01 2024-09-13T14:02:28.934 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:02:28.935 INFO:teuthology.orchestra.run.smithi102.stdout:Total 21 MB/s | 40 MB 00:01 2024-09-13T14:02:28.962 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-09-13T14:02:28.981 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-09-13T14:02:28.981 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-09-13T14:02:29.210 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libcephsqlite-2:18.2.4-843.g2c01458a.el9.x86_64 3/4 2024-09-13T14:02:29.302 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-09-13T14:02:29.302 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-09-13T14:02:29.306 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-test-2:18.2.4-843.g2c01458a.el9.x86_64 4/4 2024-09-13T14:02:29.829 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-09-13T14:02:29.941 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-13T14:02:29.990 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-13T14:02:30.018 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libcephsqlite-2:18.2.4-843.g2c01458a.el9.x86_64 3/4 2024-09-13T14:02:31.290 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-test-2:18.2.4-843.g2c01458a.el9.x86_64 4/4 2024-09-13T14:02:31.290 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-test-2:18.2.4-843.g2c01458a.el9.x86_64 1/4 2024-09-13T14:02:31.290 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libcephsqlite-2:18.2.4-843.g2c01458a.el9.x86_64 2/4 2024-09-13T14:02:31.290 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-13T14:02:31.645 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-13T14:02:31.645 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:02:31.645 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-09-13T14:02:31.646 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-test-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:02:31.646 INFO:teuthology.orchestra.run.smithi195.stdout: libcephsqlite-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:02:31.646 INFO:teuthology.orchestra.run.smithi195.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-13T14:02:31.646 INFO:teuthology.orchestra.run.smithi195.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-13T14:02:31.646 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:02:31.646 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-09-13T14:02:31.986 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph 2024-09-13T14:02:32.562 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:00:35 ago on Fri 13 Sep 2024 02:01:57 PM UTC. 2024-09-13T14:02:32.697 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-09-13T14:02:32.699 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T14:02:32.699 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-09-13T14:02:32.699 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T14:02:32.699 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-09-13T14:02:32.699 INFO:teuthology.orchestra.run.smithi195.stdout: ceph x86_64 2:18.2.4-843.g2c01458a.el9 ceph 6.5 k 2024-09-13T14:02:32.699 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-09-13T14:02:32.700 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mds x86_64 2:18.2.4-843.g2c01458a.el9 ceph 2.1 M 2024-09-13T14:02:32.700 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr x86_64 2:18.2.4-843.g2c01458a.el9 ceph 1.5 M 2024-09-13T14:02:32.700 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-modules-core noarch 2:18.2.4-843.g2c01458a.el9 ceph-noarch 246 k 2024-09-13T14:02:32.700 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mon x86_64 2:18.2.4-843.g2c01458a.el9 ceph 4.7 M 2024-09-13T14:02:32.700 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-osd x86_64 2:18.2.4-843.g2c01458a.el9 ceph 17 M 2024-09-13T14:02:32.700 INFO:teuthology.orchestra.run.smithi195.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-13T14:02:32.700 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-13T14:02:32.700 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-13T14:02:32.700 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-13T14:02:32.700 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-13T14:02:32.700 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-13T14:02:32.700 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-13T14:02:32.700 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-13T14:02:32.700 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-13T14:02:32.701 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-13T14:02:32.701 INFO:teuthology.orchestra.run.smithi195.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-13T14:02:32.701 INFO:teuthology.orchestra.run.smithi195.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-13T14:02:32.701 INFO:teuthology.orchestra.run.smithi195.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-13T14:02:32.701 INFO:teuthology.orchestra.run.smithi195.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-13T14:02:32.701 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-13T14:02:32.701 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-13T14:02:32.701 INFO:teuthology.orchestra.run.smithi195.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-13T14:02:32.701 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-13T14:02:32.701 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-13T14:02:32.701 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-13T14:02:32.701 INFO:teuthology.orchestra.run.smithi195.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-13T14:02:32.701 INFO:teuthology.orchestra.run.smithi195.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-13T14:02:32.702 INFO:teuthology.orchestra.run.smithi195.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-13T14:02:32.702 INFO:teuthology.orchestra.run.smithi195.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-13T14:02:32.702 INFO:teuthology.orchestra.run.smithi195.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-13T14:02:32.702 INFO:teuthology.orchestra.run.smithi195.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-13T14:02:32.702 INFO:teuthology.orchestra.run.smithi195.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-13T14:02:32.702 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:02:32.702 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-09-13T14:02:32.702 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T14:02:32.702 INFO:teuthology.orchestra.run.smithi195.stdout:Install 33 Packages 2024-09-13T14:02:32.702 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:02:32.703 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 29 M 2024-09-13T14:02:32.703 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 104 M 2024-09-13T14:02:32.703 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-09-13T14:02:33.585 INFO:teuthology.orchestra.run.smithi195.stdout:(1/33): ceph-18.2.4-843.g2c01458a.el9.x86_64.rp 52 kB/s | 6.5 kB 00:00 2024-09-13T14:02:33.711 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libcephsqlite-2:18.2.4-843.g2c01458a.el9.x86_64 3/4 2024-09-13T14:02:33.760 INFO:teuthology.orchestra.run.smithi195.stdout:(2/33): ceph-mgr-18.2.4-843.g2c01458a.el9.x86_6 4.9 MB/s | 1.5 MB 00:00 2024-09-13T14:02:33.791 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-test-2:18.2.4-843.g2c01458a.el9.x86_64 4/4 2024-09-13T14:02:33.852 INFO:teuthology.orchestra.run.smithi195.stdout:(3/33): ceph-mds-18.2.4-843.g2c01458a.el9.x86_6 5.4 MB/s | 2.1 MB 00:00 2024-09-13T14:02:33.969 INFO:teuthology.orchestra.run.smithi195.stdout:(4/33): ceph-mon-18.2.4-843.g2c01458a.el9.x86_6 12 MB/s | 4.7 MB 00:00 2024-09-13T14:02:33.994 INFO:teuthology.orchestra.run.smithi195.stdout:(5/33): ceph-mgr-modules-core-18.2.4-843.g2c014 1.7 MB/s | 246 kB 00:00 2024-09-13T14:02:34.145 INFO:teuthology.orchestra.run.smithi195.stdout:(6/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.4 MB/s | 253 kB 00:00 2024-09-13T14:02:34.337 INFO:teuthology.orchestra.run.smithi195.stdout:(7/33): ceph-osd-18.2.4-843.g2c01458a.el9.x86_6 29 MB/s | 17 MB 00:00 2024-09-13T14:02:34.362 INFO:teuthology.orchestra.run.smithi195.stdout:(8/33): python3-ply-3.11-14.el9.noarch.rpm 491 kB/s | 106 kB 00:00 2024-09-13T14:02:34.396 INFO:teuthology.orchestra.run.smithi195.stdout:(9/33): python3-cryptography-36.0.1-4.el9.x86_6 3.1 MB/s | 1.2 MB 00:00 2024-09-13T14:02:34.421 INFO:teuthology.orchestra.run.smithi195.stdout:(10/33): python3-requests-2.25.1-8.el9.noarch.r 2.1 MB/s | 125 kB 00:00 2024-09-13T14:02:34.446 INFO:teuthology.orchestra.run.smithi195.stdout:(11/33): python3-urllib3-1.26.5-6.el9.noarch.rp 4.2 MB/s | 215 kB 00:00 2024-09-13T14:02:34.480 INFO:teuthology.orchestra.run.smithi195.stdout:(12/33): python3-pycparser-2.20-6.el9.noarch.rp 950 kB/s | 135 kB 00:00 2024-09-13T14:02:34.563 INFO:teuthology.orchestra.run.smithi195.stdout:(13/33): python3-markupsafe-1.1.1-12.el9.x86_64 297 kB/s | 35 kB 00:00 2024-09-13T14:02:34.589 INFO:teuthology.orchestra.run.smithi195.stdout:(14/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.0 MB/s | 172 kB 00:00 2024-09-13T14:02:34.614 INFO:teuthology.orchestra.run.smithi195.stdout:(15/33): python3-pytz-2021.1-5.el9.noarch.rpm 381 kB/s | 51 kB 00:00 2024-09-13T14:02:34.656 INFO:teuthology.orchestra.run.smithi195.stdout:(16/33): python3-toml-0.10.2-6.el9.noarch.rpm 452 kB/s | 42 kB 00:00 2024-09-13T14:02:34.689 INFO:teuthology.orchestra.run.smithi195.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 432 kB/s | 43 kB 00:00 2024-09-13T14:02:34.715 INFO:teuthology.orchestra.run.smithi195.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 464 kB/s | 11 kB 00:00 2024-09-13T14:02:34.740 INFO:teuthology.orchestra.run.smithi195.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-09-13T14:02:34.765 INFO:teuthology.orchestra.run.smithi195.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 351 kB/s | 18 kB 00:00 2024-09-13T14:02:34.790 INFO:teuthology.orchestra.run.smithi195.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 460 kB/s | 23 kB 00:00 2024-09-13T14:02:34.816 INFO:teuthology.orchestra.run.smithi195.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-09-13T14:02:34.841 INFO:teuthology.orchestra.run.smithi195.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2024-09-13T14:02:34.866 INFO:teuthology.orchestra.run.smithi195.stdout:(24/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.7 MB/s | 358 kB 00:00 2024-09-13T14:02:34.883 INFO:teuthology.orchestra.run.smithi195.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 690 kB/s | 46 kB 00:00 2024-09-13T14:02:34.908 INFO:teuthology.orchestra.run.smithi195.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 1.1 MB/s | 79 kB 00:00 2024-09-13T14:02:34.925 INFO:teuthology.orchestra.run.smithi195.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 391 kB/s | 16 kB 00:00 2024-09-13T14:02:34.942 INFO:teuthology.orchestra.run.smithi195.stdout:(28/33): python3-tempora-5.0.0-2.el9.noarch.rpm 2.1 MB/s | 36 kB 00:00 2024-09-13T14:02:34.968 INFO:teuthology.orchestra.run.smithi195.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-09-13T14:02:34.993 INFO:teuthology.orchestra.run.smithi195.stdout:(30/33): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-09-13T14:02:35.018 INFO:teuthology.orchestra.run.smithi195.stdout:(31/33): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-13T14:02:35.043 INFO:teuthology.orchestra.run.smithi195.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 400 kB/s | 20 kB 00:00 2024-09-13T14:02:35.077 INFO:teuthology.orchestra.run.smithi195.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-13T14:02:35.081 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:02:35.081 INFO:teuthology.orchestra.run.smithi195.stdout:Total 12 MB/s | 29 MB 00:02 2024-09-13T14:02:35.203 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-09-13T14:02:35.241 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-09-13T14:02:35.241 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-09-13T14:02:35.634 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-09-13T14:02:35.635 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-09-13T14:02:35.772 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-test-2:18.2.4-843.g2c01458a.el9.x86_64 4/4 2024-09-13T14:02:35.772 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-test-2:18.2.4-843.g2c01458a.el9.x86_64 1/4 2024-09-13T14:02:35.773 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libcephsqlite-2:18.2.4-843.g2c01458a.el9.x86_64 2/4 2024-09-13T14:02:35.773 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-13T14:02:36.055 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-13T14:02:36.055 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:02:36.055 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-09-13T14:02:36.055 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-test-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:02:36.055 INFO:teuthology.orchestra.run.smithi102.stdout: libcephsqlite-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:02:36.055 INFO:teuthology.orchestra.run.smithi102.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-13T14:02:36.055 INFO:teuthology.orchestra.run.smithi102.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-13T14:02:36.055 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:02:36.055 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-09-13T14:02:36.410 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph 2024-09-13T14:02:36.857 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-09-13T14:02:36.921 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-09-13T14:02:36.971 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-09-13T14:02:36.976 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:00:36 ago on Fri 13 Sep 2024 02:02:00 PM UTC. 2024-09-13T14:02:37.004 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-09-13T14:02:37.097 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-09-13T14:02:37.117 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-09-13T14:02:37.119 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T14:02:37.119 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-09-13T14:02:37.119 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T14:02:37.119 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-09-13T14:02:37.120 INFO:teuthology.orchestra.run.smithi102.stdout: ceph x86_64 2:18.2.4-843.g2c01458a.el9 ceph 6.5 k 2024-09-13T14:02:37.120 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-09-13T14:02:37.120 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mds x86_64 2:18.2.4-843.g2c01458a.el9 ceph 2.1 M 2024-09-13T14:02:37.120 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr x86_64 2:18.2.4-843.g2c01458a.el9 ceph 1.5 M 2024-09-13T14:02:37.120 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-modules-core noarch 2:18.2.4-843.g2c01458a.el9 ceph-noarch 246 k 2024-09-13T14:02:37.120 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mon x86_64 2:18.2.4-843.g2c01458a.el9 ceph 4.7 M 2024-09-13T14:02:37.120 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-osd x86_64 2:18.2.4-843.g2c01458a.el9 ceph 17 M 2024-09-13T14:02:37.120 INFO:teuthology.orchestra.run.smithi102.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-13T14:02:37.120 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-13T14:02:37.120 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-13T14:02:37.120 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-13T14:02:37.120 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-13T14:02:37.120 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-13T14:02:37.121 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-13T14:02:37.121 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-13T14:02:37.121 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-13T14:02:37.121 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-13T14:02:37.121 INFO:teuthology.orchestra.run.smithi102.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-13T14:02:37.121 INFO:teuthology.orchestra.run.smithi102.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-13T14:02:37.121 INFO:teuthology.orchestra.run.smithi102.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-13T14:02:37.121 INFO:teuthology.orchestra.run.smithi102.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-13T14:02:37.121 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-13T14:02:37.121 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-13T14:02:37.121 INFO:teuthology.orchestra.run.smithi102.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-13T14:02:37.121 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-13T14:02:37.121 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-13T14:02:37.122 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-13T14:02:37.122 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-13T14:02:37.122 INFO:teuthology.orchestra.run.smithi102.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-13T14:02:37.122 INFO:teuthology.orchestra.run.smithi102.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-13T14:02:37.122 INFO:teuthology.orchestra.run.smithi102.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-13T14:02:37.122 INFO:teuthology.orchestra.run.smithi102.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-13T14:02:37.122 INFO:teuthology.orchestra.run.smithi102.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-13T14:02:37.122 INFO:teuthology.orchestra.run.smithi102.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-13T14:02:37.122 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:02:37.122 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-09-13T14:02:37.122 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T14:02:37.123 INFO:teuthology.orchestra.run.smithi102.stdout:Install 33 Packages 2024-09-13T14:02:37.123 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:02:37.123 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 29 M 2024-09-13T14:02:37.123 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 104 M 2024-09-13T14:02:37.123 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-09-13T14:02:37.174 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-09-13T14:02:37.214 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/33 2024-09-13T14:02:37.258 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-09-13T14:02:37.287 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-09-13T14:02:37.322 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-09-13T14:02:37.355 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-09-13T14:02:37.396 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-09-13T14:02:37.439 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-09-13T14:02:37.473 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-09-13T14:02:37.519 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-09-13T14:02:37.593 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-09-13T14:02:37.731 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-09-13T14:02:37.803 INFO:teuthology.orchestra.run.smithi102.stdout:(1/33): ceph-18.2.4-843.g2c01458a.el9.x86_64.rp 45 kB/s | 6.5 kB 00:00 2024-09-13T14:02:37.818 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-09-13T14:02:37.880 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 18/33 2024-09-13T14:02:37.919 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-09-13T14:02:37.969 INFO:teuthology.orchestra.run.smithi102.stdout:(2/33): ceph-mgr-18.2.4-843.g2c01458a.el9.x86_6 4.7 MB/s | 1.5 MB 00:00 2024-09-13T14:02:37.976 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-09-13T14:02:38.020 INFO:teuthology.orchestra.run.smithi102.stdout:(3/33): ceph-mds-18.2.4-843.g2c01458a.el9.x86_6 5.8 MB/s | 2.1 MB 00:00 2024-09-13T14:02:38.032 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-09-13T14:02:38.086 INFO:teuthology.orchestra.run.smithi102.stdout:(4/33): ceph-mgr-modules-core-18.2.4-843.g2c014 3.6 MB/s | 246 kB 00:00 2024-09-13T14:02:38.218 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-09-13T14:02:38.229 INFO:teuthology.orchestra.run.smithi102.stdout:(5/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.7 MB/s | 253 kB 00:00 2024-09-13T14:02:38.273 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-09-13T14:02:38.312 INFO:teuthology.orchestra.run.smithi102.stdout:(6/33): python3-cryptography-36.0.1-4.el9.x86_6 15 MB/s | 1.2 MB 00:00 2024-09-13T14:02:38.351 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-09-13T14:02:38.354 INFO:teuthology.orchestra.run.smithi102.stdout:(7/33): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-13T14:02:38.439 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-09-13T14:02:38.487 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-09-13T14:02:38.521 INFO:teuthology.orchestra.run.smithi102.stdout:(8/33): ceph-osd-18.2.4-843.g2c01458a.el9.x86_6 30 MB/s | 17 MB 00:00 2024-09-13T14:02:38.538 INFO:teuthology.orchestra.run.smithi102.stdout:(9/33): python3-pycparser-2.20-6.el9.noarch.rpm 736 kB/s | 135 kB 00:00 2024-09-13T14:02:38.605 INFO:teuthology.orchestra.run.smithi102.stdout:(10/33): ceph-mon-18.2.4-843.g2c01458a.el9.x86_ 5.8 MB/s | 4.7 MB 00:00 2024-09-13T14:02:38.605 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-09-13T14:02:38.630 INFO:teuthology.orchestra.run.smithi102.stdout:(11/33): python3-urllib3-1.26.5-6.el9.noarch.rp 2.3 MB/s | 215 kB 00:00 2024-09-13T14:02:38.664 INFO:teuthology.orchestra.run.smithi102.stdout:(12/33): python3-requests-2.25.1-8.el9.noarch.r 878 kB/s | 125 kB 00:00 2024-09-13T14:02:38.739 INFO:teuthology.orchestra.run.smithi102.stdout:(13/33): python3-markupsafe-1.1.1-12.el9.x86_64 319 kB/s | 35 kB 00:00 2024-09-13T14:02:38.755 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-modules-core-2:18.2.4-843.g2c01458a.el9.n 28/33 2024-09-13T14:02:38.772 INFO:teuthology.orchestra.run.smithi102.stdout:(14/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.0 MB/s | 172 kB 00:00 2024-09-13T14:02:38.783 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-2:18.2.4-843.g2c01458a.el9.x86_64 29/33 2024-09-13T14:02:38.789 INFO:teuthology.orchestra.run.smithi102.stdout:(15/33): python3-toml-0.10.2-6.el9.noarch.rpm 830 kB/s | 42 kB 00:00 2024-09-13T14:02:38.807 INFO:teuthology.orchestra.run.smithi102.stdout:(16/33): python3-pytz-2021.1-5.el9.noarch.rpm 358 kB/s | 51 kB 00:00 2024-09-13T14:02:38.810 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-2:18.2.4-843.g2c01458a.el9.x86_64 29/33 2024-09-13T14:02:38.810 INFO:teuthology.orchestra.run.smithi195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:02:38.810 INFO:teuthology.orchestra.run.smithi195.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-13T14:02:38.810 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T14:02:38.810 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T14:02:38.810 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:02:38.848 INFO:teuthology.orchestra.run.smithi102.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 576 kB/s | 43 kB 00:00 2024-09-13T14:02:38.865 INFO:teuthology.orchestra.run.smithi102.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 716 kB/s | 11 kB 00:00 2024-09-13T14:02:38.891 INFO:teuthology.orchestra.run.smithi102.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-09-13T14:02:38.907 INFO:teuthology.orchestra.run.smithi102.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 422 kB/s | 18 kB 00:00 2024-09-13T14:02:38.933 INFO:teuthology.orchestra.run.smithi102.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 555 kB/s | 23 kB 00:00 2024-09-13T14:02:38.958 INFO:teuthology.orchestra.run.smithi102.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 386 kB/s | 19 kB 00:00 2024-09-13T14:02:38.983 INFO:teuthology.orchestra.run.smithi102.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-09-13T14:02:39.009 INFO:teuthology.orchestra.run.smithi102.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 922 kB/s | 46 kB 00:00 2024-09-13T14:02:39.034 INFO:teuthology.orchestra.run.smithi102.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-13T14:02:39.059 INFO:teuthology.orchestra.run.smithi102.stdout:(26/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.5 MB/s | 358 kB 00:00 2024-09-13T14:02:39.110 INFO:teuthology.orchestra.run.smithi102.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 217 kB/s | 16 kB 00:00 2024-09-13T14:02:39.135 INFO:teuthology.orchestra.run.smithi102.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-09-13T14:02:39.160 INFO:teuthology.orchestra.run.smithi102.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 712 kB/s | 36 kB 00:00 2024-09-13T14:02:39.185 INFO:teuthology.orchestra.run.smithi102.stdout:(30/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.5 MB/s | 272 kB 00:00 2024-09-13T14:02:39.211 INFO:teuthology.orchestra.run.smithi102.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 799 kB/s | 20 kB 00:00 2024-09-13T14:02:39.236 INFO:teuthology.orchestra.run.smithi102.stdout:(32/33): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-13T14:02:39.261 INFO:teuthology.orchestra.run.smithi102.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.1 MB/s | 427 kB 00:00 2024-09-13T14:02:39.265 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:02:39.265 INFO:teuthology.orchestra.run.smithi102.stdout:Total 14 MB/s | 29 MB 00:02 2024-09-13T14:02:39.382 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-09-13T14:02:39.420 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-09-13T14:02:39.420 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-09-13T14:02:39.825 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-09-13T14:02:39.825 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-09-13T14:02:40.225 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-osd-2:18.2.4-843.g2c01458a.el9.x86_64 30/33 2024-09-13T14:02:40.254 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-osd-2:18.2.4-843.g2c01458a.el9.x86_64 30/33 2024-09-13T14:02:40.254 INFO:teuthology.orchestra.run.smithi195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:02:40.254 INFO:teuthology.orchestra.run.smithi195.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-13T14:02:40.254 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T14:02:40.254 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T14:02:40.254 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:02:40.693 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mon-2:18.2.4-843.g2c01458a.el9.x86_64 31/33 2024-09-13T14:02:40.716 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mon-2:18.2.4-843.g2c01458a.el9.x86_64 31/33 2024-09-13T14:02:40.717 INFO:teuthology.orchestra.run.smithi195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:02:40.717 INFO:teuthology.orchestra.run.smithi195.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-13T14:02:40.717 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T14:02:40.717 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T14:02:40.717 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:02:40.926 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mds-2:18.2.4-843.g2c01458a.el9.x86_64 32/33 2024-09-13T14:02:40.950 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mds-2:18.2.4-843.g2c01458a.el9.x86_64 32/33 2024-09-13T14:02:40.950 INFO:teuthology.orchestra.run.smithi195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:02:40.950 INFO:teuthology.orchestra.run.smithi195.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-13T14:02:40.951 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T14:02:40.951 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T14:02:40.951 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:02:41.015 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-2:18.2.4-843.g2c01458a.el9.x86_64 33/33 2024-09-13T14:02:41.095 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-09-13T14:02:41.155 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-09-13T14:02:41.188 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-09-13T14:02:41.221 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-09-13T14:02:41.313 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-09-13T14:02:41.392 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-09-13T14:02:41.439 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/33 2024-09-13T14:02:41.475 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-09-13T14:02:41.505 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-09-13T14:02:41.539 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-09-13T14:02:41.573 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-09-13T14:02:41.622 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-09-13T14:02:41.674 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-09-13T14:02:41.724 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-09-13T14:02:41.767 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-09-13T14:02:41.844 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-09-13T14:02:42.008 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-09-13T14:02:42.088 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-09-13T14:02:42.141 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 18/33 2024-09-13T14:02:42.179 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-09-13T14:02:42.237 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-09-13T14:02:42.291 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-09-13T14:02:42.478 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-09-13T14:02:42.525 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-09-13T14:02:42.603 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-09-13T14:02:42.691 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-09-13T14:02:42.720 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-2:18.2.4-843.g2c01458a.el9.x86_64 33/33 2024-09-13T14:02:42.721 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-2:18.2.4-843.g2c01458a.el9.x86_64 1/33 2024-09-13T14:02:42.721 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mds-2:18.2.4-843.g2c01458a.el9.x86_64 2/33 2024-09-13T14:02:42.721 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-2:18.2.4-843.g2c01458a.el9.x86_64 3/33 2024-09-13T14:02:42.721 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mon-2:18.2.4-843.g2c01458a.el9.x86_64 4/33 2024-09-13T14:02:42.721 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-osd-2:18.2.4-843.g2c01458a.el9.x86_64 5/33 2024-09-13T14:02:42.721 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-modules-core-2:18.2.4-843.g2c01458a.el9.n 6/33 2024-09-13T14:02:42.721 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-09-13T14:02:42.721 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-09-13T14:02:42.721 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-09-13T14:02:42.721 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-09-13T14:02:42.721 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-09-13T14:02:42.721 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/33 2024-09-13T14:02:42.721 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-09-13T14:02:42.722 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-09-13T14:02:42.722 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-09-13T14:02:42.722 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-09-13T14:02:42.722 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-09-13T14:02:42.722 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-09-13T14:02:42.722 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-09-13T14:02:42.722 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-09-13T14:02:42.722 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-09-13T14:02:42.722 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-09-13T14:02:42.722 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-09-13T14:02:42.722 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-09-13T14:02:42.722 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-09-13T14:02:42.722 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-09-13T14:02:42.722 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-09-13T14:02:42.723 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-09-13T14:02:42.723 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-09-13T14:02:42.723 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-09-13T14:02:42.723 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 31/33 2024-09-13T14:02:42.723 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-09-13T14:02:42.727 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-09-13T14:02:42.841 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-09-13T14:02:42.998 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-modules-core-2:18.2.4-843.g2c01458a.el9.n 28/33 2024-09-13T14:02:43.026 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-2:18.2.4-843.g2c01458a.el9.x86_64 29/33 2024-09-13T14:02:43.051 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-2:18.2.4-843.g2c01458a.el9.x86_64 29/33 2024-09-13T14:02:43.051 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:02:43.051 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-13T14:02:43.051 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T14:02:43.051 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T14:02:43.051 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:02:43.714 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-09-13T14:02:43.714 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:02:43.714 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-09-13T14:02:43.714 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:02:43.715 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mds-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:02:43.715 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:02:43.715 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-modules-core-2:18.2.4-843.g2c01458a.el9.noarch 2024-09-13T14:02:43.715 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mon-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:02:43.715 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-osd-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:02:43.715 INFO:teuthology.orchestra.run.smithi195.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-13T14:02:43.715 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-13T14:02:43.715 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-13T14:02:43.715 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-13T14:02:43.715 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-13T14:02:43.715 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-13T14:02:43.715 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-13T14:02:43.716 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-13T14:02:43.716 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-13T14:02:43.716 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-13T14:02:43.716 INFO:teuthology.orchestra.run.smithi195.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-13T14:02:43.716 INFO:teuthology.orchestra.run.smithi195.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-13T14:02:43.717 INFO:teuthology.orchestra.run.smithi195.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-13T14:02:43.717 INFO:teuthology.orchestra.run.smithi195.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-13T14:02:43.717 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-13T14:02:43.717 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ply-3.11-14.el9.noarch 2024-09-13T14:02:43.717 INFO:teuthology.orchestra.run.smithi195.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-13T14:02:43.717 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-13T14:02:43.718 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-13T14:02:43.718 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-13T14:02:43.718 INFO:teuthology.orchestra.run.smithi195.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-13T14:02:43.718 INFO:teuthology.orchestra.run.smithi195.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-13T14:02:43.718 INFO:teuthology.orchestra.run.smithi195.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-13T14:02:43.719 INFO:teuthology.orchestra.run.smithi195.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-13T14:02:43.719 INFO:teuthology.orchestra.run.smithi195.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-13T14:02:43.719 INFO:teuthology.orchestra.run.smithi195.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-13T14:02:43.719 INFO:teuthology.orchestra.run.smithi195.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-13T14:02:43.719 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:02:43.719 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-09-13T14:02:44.118 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-base 2024-09-13T14:02:44.521 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-osd-2:18.2.4-843.g2c01458a.el9.x86_64 30/33 2024-09-13T14:02:44.550 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-osd-2:18.2.4-843.g2c01458a.el9.x86_64 30/33 2024-09-13T14:02:44.551 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:02:44.551 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-13T14:02:44.551 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T14:02:44.551 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T14:02:44.551 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:02:44.690 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:00:47 ago on Fri 13 Sep 2024 02:01:57 PM UTC. 2024-09-13T14:02:44.781 INFO:teuthology.orchestra.run.smithi195.stdout:Package ceph-base-2:18.2.4-843.g2c01458a.el9.x86_64 is already installed. 2024-09-13T14:02:44.823 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-09-13T14:02:44.824 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-09-13T14:02:44.824 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-09-13T14:02:44.894 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install cephadm 2024-09-13T14:02:44.997 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mon-2:18.2.4-843.g2c01458a.el9.x86_64 31/33 2024-09-13T14:02:45.021 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mon-2:18.2.4-843.g2c01458a.el9.x86_64 31/33 2024-09-13T14:02:45.021 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:02:45.021 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-13T14:02:45.021 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T14:02:45.021 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T14:02:45.022 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:02:45.223 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mds-2:18.2.4-843.g2c01458a.el9.x86_64 32/33 2024-09-13T14:02:45.246 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mds-2:18.2.4-843.g2c01458a.el9.x86_64 32/33 2024-09-13T14:02:45.246 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:02:45.246 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-13T14:02:45.247 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T14:02:45.247 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T14:02:45.247 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:02:45.311 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-2:18.2.4-843.g2c01458a.el9.x86_64 33/33 2024-09-13T14:02:45.458 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:00:48 ago on Fri 13 Sep 2024 02:01:57 PM UTC. 2024-09-13T14:02:45.591 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-09-13T14:02:45.592 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T14:02:45.592 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-09-13T14:02:45.593 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T14:02:45.593 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-09-13T14:02:45.593 INFO:teuthology.orchestra.run.smithi195.stdout: cephadm noarch 2:18.2.4-843.g2c01458a.el9 ceph-noarch 226 k 2024-09-13T14:02:45.593 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:02:45.593 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-09-13T14:02:45.593 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T14:02:45.594 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-09-13T14:02:45.594 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:02:45.594 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 226 k 2024-09-13T14:02:45.594 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 217 k 2024-09-13T14:02:45.594 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-09-13T14:02:45.803 INFO:teuthology.orchestra.run.smithi195.stdout:cephadm-18.2.4-843.g2c01458a.el9.noarch.rpm 1.0 MB/s | 226 kB 00:00 2024-09-13T14:02:45.804 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:02:45.804 INFO:teuthology.orchestra.run.smithi195.stdout:Total 1.0 MB/s | 226 kB 00:00 2024-09-13T14:02:45.804 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-09-13T14:02:45.810 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-09-13T14:02:45.810 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-09-13T14:02:45.812 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-09-13T14:02:45.813 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-09-13T14:02:45.881 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-09-13T14:02:46.258 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: cephadm-2:18.2.4-843.g2c01458a.el9.noarch 1/1 2024-09-13T14:02:46.349 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : cephadm-2:18.2.4-843.g2c01458a.el9.noarch 1/1 2024-09-13T14:02:46.691 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: cephadm-2:18.2.4-843.g2c01458a.el9.noarch 1/1 2024-09-13T14:02:46.943 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : cephadm-2:18.2.4-843.g2c01458a.el9.noarch 1/1 2024-09-13T14:02:46.943 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:02:46.943 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-09-13T14:02:46.943 INFO:teuthology.orchestra.run.smithi195.stdout: cephadm-2:18.2.4-843.g2c01458a.el9.noarch 2024-09-13T14:02:46.943 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:02:46.943 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-09-13T14:02:47.026 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-2:18.2.4-843.g2c01458a.el9.x86_64 33/33 2024-09-13T14:02:47.027 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-2:18.2.4-843.g2c01458a.el9.x86_64 1/33 2024-09-13T14:02:47.027 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mds-2:18.2.4-843.g2c01458a.el9.x86_64 2/33 2024-09-13T14:02:47.027 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-2:18.2.4-843.g2c01458a.el9.x86_64 3/33 2024-09-13T14:02:47.027 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mon-2:18.2.4-843.g2c01458a.el9.x86_64 4/33 2024-09-13T14:02:47.027 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-osd-2:18.2.4-843.g2c01458a.el9.x86_64 5/33 2024-09-13T14:02:47.027 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-modules-core-2:18.2.4-843.g2c01458a.el9.n 6/33 2024-09-13T14:02:47.027 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-09-13T14:02:47.027 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-09-13T14:02:47.027 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-09-13T14:02:47.027 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-09-13T14:02:47.027 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-09-13T14:02:47.028 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/33 2024-09-13T14:02:47.028 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-09-13T14:02:47.028 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-09-13T14:02:47.028 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-09-13T14:02:47.028 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-09-13T14:02:47.028 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-09-13T14:02:47.028 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-09-13T14:02:47.028 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-09-13T14:02:47.028 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-09-13T14:02:47.028 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-09-13T14:02:47.028 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-09-13T14:02:47.028 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-09-13T14:02:47.028 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-09-13T14:02:47.029 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-09-13T14:02:47.029 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-09-13T14:02:47.029 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-09-13T14:02:47.029 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-09-13T14:02:47.029 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-09-13T14:02:47.030 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-09-13T14:02:47.030 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 31/33 2024-09-13T14:02:47.030 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-09-13T14:02:47.132 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-immutable-object-cache 2024-09-13T14:02:47.726 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:00:50 ago on Fri 13 Sep 2024 02:01:57 PM UTC. 2024-09-13T14:02:47.861 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-09-13T14:02:47.862 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T14:02:47.862 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2024-09-13T14:02:47.862 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T14:02:47.862 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-09-13T14:02:47.862 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-immutable-object-cache x86_64 2:18.2.4-843.g2c01458a.el9 ceph 143 k 2024-09-13T14:02:47.862 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:02:47.862 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-09-13T14:02:47.862 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T14:02:47.862 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-09-13T14:02:47.862 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:02:47.863 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 143 k 2024-09-13T14:02:47.863 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 395 k 2024-09-13T14:02:47.863 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-09-13T14:02:47.999 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-09-13T14:02:47.999 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:02:47.999 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-09-13T14:02:47.999 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:02:47.999 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mds-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:02:47.999 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:02:47.999 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-modules-core-2:18.2.4-843.g2c01458a.el9.noarch 2024-09-13T14:02:47.999 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mon-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:02:48.000 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-osd-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:02:48.000 INFO:teuthology.orchestra.run.smithi102.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-13T14:02:48.000 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-13T14:02:48.000 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-13T14:02:48.000 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-13T14:02:48.000 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-13T14:02:48.000 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-13T14:02:48.000 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-13T14:02:48.000 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-13T14:02:48.000 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-13T14:02:48.000 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-13T14:02:48.000 INFO:teuthology.orchestra.run.smithi102.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-13T14:02:48.000 INFO:teuthology.orchestra.run.smithi102.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-13T14:02:48.001 INFO:teuthology.orchestra.run.smithi102.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-13T14:02:48.001 INFO:teuthology.orchestra.run.smithi102.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-13T14:02:48.001 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-13T14:02:48.001 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ply-3.11-14.el9.noarch 2024-09-13T14:02:48.001 INFO:teuthology.orchestra.run.smithi102.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-13T14:02:48.001 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-13T14:02:48.001 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-13T14:02:48.001 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-13T14:02:48.001 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-13T14:02:48.001 INFO:teuthology.orchestra.run.smithi102.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-13T14:02:48.001 INFO:teuthology.orchestra.run.smithi102.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-13T14:02:48.001 INFO:teuthology.orchestra.run.smithi102.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-13T14:02:48.001 INFO:teuthology.orchestra.run.smithi102.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-13T14:02:48.001 INFO:teuthology.orchestra.run.smithi102.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-13T14:02:48.001 INFO:teuthology.orchestra.run.smithi102.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-13T14:02:48.002 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:02:48.002 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-09-13T14:02:48.068 INFO:teuthology.orchestra.run.smithi195.stdout:ceph-immutable-object-cache-18.2.4-843.g2c01458 697 kB/s | 143 kB 00:00 2024-09-13T14:02:48.069 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:02:48.069 INFO:teuthology.orchestra.run.smithi195.stdout:Total 691 kB/s | 143 kB 00:00 2024-09-13T14:02:48.069 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-09-13T14:02:48.076 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-09-13T14:02:48.076 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-09-13T14:02:48.133 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-09-13T14:02:48.133 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-09-13T14:02:48.279 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-09-13T14:02:48.364 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-base 2024-09-13T14:02:48.369 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-immutable-object-cache-2:18.2.4-843.g2c01458a.e 1/1 2024-09-13T14:02:48.394 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.4-843.g2c01458a.e 1/1 2024-09-13T14:02:48.394 INFO:teuthology.orchestra.run.smithi195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:02:48.394 INFO:teuthology.orchestra.run.smithi195.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-13T14:02:48.394 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:02:48.961 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:00:48 ago on Fri 13 Sep 2024 02:02:00 PM UTC. 2024-09-13T14:02:49.052 INFO:teuthology.orchestra.run.smithi102.stdout:Package ceph-base-2:18.2.4-843.g2c01458a.el9.x86_64 is already installed. 2024-09-13T14:02:49.093 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-09-13T14:02:49.094 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-09-13T14:02:49.094 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-09-13T14:02:49.164 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install cephadm 2024-09-13T14:02:49.320 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-immutable-object-cache-2:18.2.4-843.g2c01458a.e 1/1 2024-09-13T14:02:49.321 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:02:49.321 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-09-13T14:02:49.321 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-immutable-object-cache-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:02:49.321 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:02:49.321 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-09-13T14:02:49.523 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-mgr 2024-09-13T14:02:49.726 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:00:49 ago on Fri 13 Sep 2024 02:02:00 PM UTC. 2024-09-13T14:02:49.859 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-09-13T14:02:49.860 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T14:02:49.860 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-09-13T14:02:49.860 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T14:02:49.860 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-09-13T14:02:49.860 INFO:teuthology.orchestra.run.smithi102.stdout: cephadm noarch 2:18.2.4-843.g2c01458a.el9 ceph-noarch 226 k 2024-09-13T14:02:49.860 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:02:49.860 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-09-13T14:02:49.861 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T14:02:49.861 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-09-13T14:02:49.861 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:02:49.861 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 226 k 2024-09-13T14:02:49.861 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 217 k 2024-09-13T14:02:49.861 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-09-13T14:02:50.080 INFO:teuthology.orchestra.run.smithi102.stdout:cephadm-18.2.4-843.g2c01458a.el9.noarch.rpm 1.0 MB/s | 226 kB 00:00 2024-09-13T14:02:50.081 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:02:50.081 INFO:teuthology.orchestra.run.smithi102.stdout:Total 1.0 MB/s | 226 kB 00:00 2024-09-13T14:02:50.081 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-09-13T14:02:50.086 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-09-13T14:02:50.087 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-09-13T14:02:50.089 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-09-13T14:02:50.089 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-09-13T14:02:50.105 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:00:53 ago on Fri 13 Sep 2024 02:01:57 PM UTC. 2024-09-13T14:02:50.159 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-09-13T14:02:50.199 INFO:teuthology.orchestra.run.smithi195.stdout:Package ceph-mgr-2:18.2.4-843.g2c01458a.el9.x86_64 is already installed. 2024-09-13T14:02:50.243 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-09-13T14:02:50.244 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-09-13T14:02:50.244 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-09-13T14:02:50.316 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-mgr-dashboard 2024-09-13T14:02:50.584 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: cephadm-2:18.2.4-843.g2c01458a.el9.noarch 1/1 2024-09-13T14:02:50.675 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : cephadm-2:18.2.4-843.g2c01458a.el9.noarch 1/1 2024-09-13T14:02:50.879 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:00:53 ago on Fri 13 Sep 2024 02:01:57 PM UTC. 2024-09-13T14:02:51.009 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-09-13T14:02:51.010 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T14:02:51.010 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-09-13T14:02:51.010 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T14:02:51.010 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-09-13T14:02:51.010 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-dashboard noarch 2:18.2.4-843.g2c01458a.el9 ceph-noarch 3.5 M 2024-09-13T14:02:51.010 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-09-13T14:02:51.010 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-grafana-dashboards noarch 2:18.2.4-843.g2c01458a.el9 ceph-noarch 24 k 2024-09-13T14:02:51.011 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-prometheus-alerts noarch 2:18.2.4-843.g2c01458a.el9 ceph-noarch 16 k 2024-09-13T14:02:51.011 INFO:teuthology.orchestra.run.smithi195.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-13T14:02:51.011 INFO:teuthology.orchestra.run.smithi195.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-13T14:02:51.011 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:02:51.011 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-09-13T14:02:51.011 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T14:02:51.011 INFO:teuthology.orchestra.run.smithi195.stdout:Install 5 Packages 2024-09-13T14:02:51.011 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:02:51.012 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 3.8 M 2024-09-13T14:02:51.012 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 87 M 2024-09-13T14:02:51.012 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-09-13T14:02:51.052 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: cephadm-2:18.2.4-843.g2c01458a.el9.noarch 1/1 2024-09-13T14:02:51.316 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : cephadm-2:18.2.4-843.g2c01458a.el9.noarch 1/1 2024-09-13T14:02:51.317 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:02:51.317 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-09-13T14:02:51.317 INFO:teuthology.orchestra.run.smithi102.stdout: cephadm-2:18.2.4-843.g2c01458a.el9.noarch 2024-09-13T14:02:51.317 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:02:51.317 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-09-13T14:02:51.356 INFO:teuthology.orchestra.run.smithi195.stdout:(1/5): ceph-grafana-dashboards-18.2.4-843.g2c01 133 kB/s | 24 kB 00:00 2024-09-13T14:02:51.380 INFO:teuthology.orchestra.run.smithi195.stdout:(2/5): ceph-prometheus-alerts-18.2.4-843.g2c014 79 kB/s | 16 kB 00:00 2024-09-13T14:02:51.422 INFO:teuthology.orchestra.run.smithi195.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 463 kB/s | 31 kB 00:00 2024-09-13T14:02:51.456 INFO:teuthology.orchestra.run.smithi195.stdout:(4/5): python3-routes-2.5.1-5.el9.noarch.rpm 2.5 MB/s | 188 kB 00:00 2024-09-13T14:02:51.508 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-immutable-object-cache 2024-09-13T14:02:51.623 INFO:teuthology.orchestra.run.smithi195.stdout:(5/5): ceph-mgr-dashboard-18.2.4-843.g2c01458a. 7.8 MB/s | 3.5 MB 00:00 2024-09-13T14:02:51.626 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:02:51.626 INFO:teuthology.orchestra.run.smithi195.stdout:Total 6.2 MB/s | 3.8 MB 00:00 2024-09-13T14:02:51.626 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-09-13T14:02:51.633 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-09-13T14:02:51.633 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-09-13T14:02:51.714 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-09-13T14:02:51.715 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-09-13T14:02:51.970 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-09-13T14:02:52.046 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-09-13T14:02:52.078 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-09-13T14:02:52.087 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:00:52 ago on Fri 13 Sep 2024 02:02:00 PM UTC. 2024-09-13T14:02:52.106 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-prometheus-alerts-2:18.2.4-843.g2c01458a.el9.no 3/5 2024-09-13T14:02:52.224 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-09-13T14:02:52.225 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T14:02:52.225 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repo Size 2024-09-13T14:02:52.225 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T14:02:52.225 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-09-13T14:02:52.225 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-immutable-object-cache x86_64 2:18.2.4-843.g2c01458a.el9 ceph 143 k 2024-09-13T14:02:52.225 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:02:52.226 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-09-13T14:02:52.226 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T14:02:52.226 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-09-13T14:02:52.226 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:02:52.226 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 143 k 2024-09-13T14:02:52.226 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 395 k 2024-09-13T14:02:52.226 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-09-13T14:02:52.644 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-immutable-object-cache-18.2.4-843.g2c01458 341 kB/s | 143 kB 00:00 2024-09-13T14:02:52.645 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:02:52.645 INFO:teuthology.orchestra.run.smithi102.stdout:Total 340 kB/s | 143 kB 00:00 2024-09-13T14:02:52.645 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-09-13T14:02:52.652 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-09-13T14:02:52.653 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-09-13T14:02:52.709 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-09-13T14:02:52.709 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-09-13T14:02:52.847 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-09-13T14:02:52.933 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-immutable-object-cache-2:18.2.4-843.g2c01458a.e 1/1 2024-09-13T14:02:52.958 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.4-843.g2c01458a.e 1/1 2024-09-13T14:02:52.959 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:02:52.959 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-13T14:02:52.959 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:02:53.156 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-grafana-dashboards-2:18.2.4-843.g2c01458a.el9.n 4/5 2024-09-13T14:02:53.201 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-dashboard-2:18.2.4-843.g2c01458a.el9.noarch 5/5 2024-09-13T14:02:53.943 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-immutable-object-cache-2:18.2.4-843.g2c01458a.e 1/1 2024-09-13T14:02:53.943 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:02:53.944 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-09-13T14:02:53.944 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-immutable-object-cache-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:02:53.944 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:02:53.944 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-09-13T14:02:54.165 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-mgr 2024-09-13T14:02:54.252 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.4-843.g2c01458a.el9.noarch 5/5 2024-09-13T14:02:54.252 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-grafana-dashboards-2:18.2.4-843.g2c01458a.el9.n 1/5 2024-09-13T14:02:54.252 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-dashboard-2:18.2.4-843.g2c01458a.el9.noarch 2/5 2024-09-13T14:02:54.252 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-prometheus-alerts-2:18.2.4-843.g2c01458a.el9.no 3/5 2024-09-13T14:02:54.252 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-09-13T14:02:54.508 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-09-13T14:02:54.508 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:02:54.508 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-09-13T14:02:54.508 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-grafana-dashboards-2:18.2.4-843.g2c01458a.el9.noarch 2024-09-13T14:02:54.508 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-dashboard-2:18.2.4-843.g2c01458a.el9.noarch 2024-09-13T14:02:54.508 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-prometheus-alerts-2:18.2.4-843.g2c01458a.el9.noarch 2024-09-13T14:02:54.508 INFO:teuthology.orchestra.run.smithi195.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-13T14:02:54.508 INFO:teuthology.orchestra.run.smithi195.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-13T14:02:54.509 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:02:54.509 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-09-13T14:02:54.735 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-13T14:02:54.745 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:00:54 ago on Fri 13 Sep 2024 02:02:00 PM UTC. 2024-09-13T14:02:54.839 INFO:teuthology.orchestra.run.smithi102.stdout:Package ceph-mgr-2:18.2.4-843.g2c01458a.el9.x86_64 is already installed. 2024-09-13T14:02:54.881 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-09-13T14:02:54.882 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-09-13T14:02:54.882 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-09-13T14:02:54.946 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-mgr-dashboard 2024-09-13T14:02:55.296 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:00:58 ago on Fri 13 Sep 2024 02:01:57 PM UTC. 2024-09-13T14:02:55.430 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-09-13T14:02:55.430 INFO:teuthology.orchestra.run.smithi195.stdout:====================================================================================== 2024-09-13T14:02:55.431 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-09-13T14:02:55.431 INFO:teuthology.orchestra.run.smithi195.stdout:====================================================================================== 2024-09-13T14:02:55.431 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-09-13T14:02:55.431 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.4-843.g2c01458a.el9 ceph-noarch 7.4 M 2024-09-13T14:02:55.431 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-09-13T14:02:55.431 INFO:teuthology.orchestra.run.smithi195.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-13T14:02:55.431 INFO:teuthology.orchestra.run.smithi195.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-13T14:02:55.431 INFO:teuthology.orchestra.run.smithi195.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-13T14:02:55.431 INFO:teuthology.orchestra.run.smithi195.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-13T14:02:55.431 INFO:teuthology.orchestra.run.smithi195.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-13T14:02:55.431 INFO:teuthology.orchestra.run.smithi195.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-13T14:02:55.432 INFO:teuthology.orchestra.run.smithi195.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-13T14:02:55.432 INFO:teuthology.orchestra.run.smithi195.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-13T14:02:55.432 INFO:teuthology.orchestra.run.smithi195.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-13T14:02:55.432 INFO:teuthology.orchestra.run.smithi195.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-13T14:02:55.432 INFO:teuthology.orchestra.run.smithi195.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-13T14:02:55.432 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:02:55.432 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-09-13T14:02:55.432 INFO:teuthology.orchestra.run.smithi195.stdout:====================================================================================== 2024-09-13T14:02:55.432 INFO:teuthology.orchestra.run.smithi195.stdout:Install 12 Packages 2024-09-13T14:02:55.432 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:02:55.433 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 38 M 2024-09-13T14:02:55.433 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 212 M 2024-09-13T14:02:55.433 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-09-13T14:02:55.516 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:00:55 ago on Fri 13 Sep 2024 02:02:00 PM UTC. 2024-09-13T14:02:55.647 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-09-13T14:02:55.648 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T14:02:55.648 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-09-13T14:02:55.648 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T14:02:55.648 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-09-13T14:02:55.648 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-dashboard noarch 2:18.2.4-843.g2c01458a.el9 ceph-noarch 3.5 M 2024-09-13T14:02:55.648 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-09-13T14:02:55.648 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-grafana-dashboards noarch 2:18.2.4-843.g2c01458a.el9 ceph-noarch 24 k 2024-09-13T14:02:55.648 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-prometheus-alerts noarch 2:18.2.4-843.g2c01458a.el9 ceph-noarch 16 k 2024-09-13T14:02:55.649 INFO:teuthology.orchestra.run.smithi102.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-13T14:02:55.649 INFO:teuthology.orchestra.run.smithi102.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-13T14:02:55.649 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:02:55.649 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-09-13T14:02:55.649 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T14:02:55.649 INFO:teuthology.orchestra.run.smithi102.stdout:Install 5 Packages 2024-09-13T14:02:55.649 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:02:55.650 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 3.8 M 2024-09-13T14:02:55.650 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 87 M 2024-09-13T14:02:55.650 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-09-13T14:02:55.857 INFO:teuthology.orchestra.run.smithi102.stdout:(1/5): ceph-grafana-dashboards-18.2.4-843.g2c01 157 kB/s | 24 kB 00:00 2024-09-13T14:02:55.874 INFO:teuthology.orchestra.run.smithi102.stdout:(2/5): ceph-prometheus-alerts-18.2.4-843.g2c014 96 kB/s | 16 kB 00:00 2024-09-13T14:02:55.916 INFO:teuthology.orchestra.run.smithi102.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 525 kB/s | 31 kB 00:00 2024-09-13T14:02:55.949 INFO:teuthology.orchestra.run.smithi102.stdout:(4/5): python3-routes-2.5.1-5.el9.noarch.rpm 2.4 MB/s | 188 kB 00:00 2024-09-13T14:02:56.047 INFO:teuthology.orchestra.run.smithi195.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 923 kB/s | 191 kB 00:00 2024-09-13T14:02:56.083 INFO:teuthology.orchestra.run.smithi102.stdout:(5/5): ceph-mgr-dashboard-18.2.4-843.g2c01458a. 9.3 MB/s | 3.5 MB 00:00 2024-09-13T14:02:56.086 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:02:56.086 INFO:teuthology.orchestra.run.smithi102.stdout:Total 8.7 MB/s | 3.8 MB 00:00 2024-09-13T14:02:56.086 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-09-13T14:02:56.094 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-09-13T14:02:56.094 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-09-13T14:02:56.164 INFO:teuthology.orchestra.run.smithi195.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.4 MB/s | 801 kB 00:00 2024-09-13T14:02:56.178 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-09-13T14:02:56.178 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-09-13T14:02:56.297 INFO:teuthology.orchestra.run.smithi195.stdout:(3/12): ceph-mgr-diskprediction-local-18.2.4-84 16 MB/s | 7.4 MB 00:00 2024-09-13T14:02:56.331 INFO:teuthology.orchestra.run.smithi195.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 115 kB/s | 33 kB 00:00 2024-09-13T14:02:56.429 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-09-13T14:02:56.530 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-09-13T14:02:56.531 INFO:teuthology.orchestra.run.smithi195.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 193 kB/s | 38 kB 00:00 2024-09-13T14:02:56.571 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-09-13T14:02:56.581 INFO:teuthology.orchestra.run.smithi195.stdout:(6/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 64 kB/s | 18 kB 00:00 2024-09-13T14:02:56.600 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-prometheus-alerts-2:18.2.4-843.g2c01458a.el9.no 3/5 2024-09-13T14:02:57.040 INFO:teuthology.orchestra.run.smithi195.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 534 kB/s | 245 kB 00:00 2024-09-13T14:02:57.340 INFO:teuthology.orchestra.run.smithi195.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.5 MB/s | 3.0 MB 00:01 2024-09-13T14:02:57.415 INFO:teuthology.orchestra.run.smithi195.stdout:(9/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 5.6 MB/s | 4.9 MB 00:00 2024-09-13T14:02:57.457 INFO:teuthology.orchestra.run.smithi195.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.1 MB/s | 250 kB 00:00 2024-09-13T14:02:57.645 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-grafana-dashboards-2:18.2.4-843.g2c01458a.el9.n 4/5 2024-09-13T14:02:57.687 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-dashboard-2:18.2.4-843.g2c01458a.el9.noarch 5/5 2024-09-13T14:02:58.249 INFO:teuthology.orchestra.run.smithi195.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.2 MB/s | 5.1 MB 00:01 2024-09-13T14:02:58.781 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.4-843.g2c01458a.el9.noarch 5/5 2024-09-13T14:02:58.781 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-grafana-dashboards-2:18.2.4-843.g2c01458a.el9.n 1/5 2024-09-13T14:02:58.781 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-dashboard-2:18.2.4-843.g2c01458a.el9.noarch 2/5 2024-09-13T14:02:58.781 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-prometheus-alerts-2:18.2.4-843.g2c01458a.el9.no 3/5 2024-09-13T14:02:58.781 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-09-13T14:02:59.034 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-09-13T14:02:59.034 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:02:59.034 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-09-13T14:02:59.034 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-grafana-dashboards-2:18.2.4-843.g2c01458a.el9.noarch 2024-09-13T14:02:59.034 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-dashboard-2:18.2.4-843.g2c01458a.el9.noarch 2024-09-13T14:02:59.034 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-prometheus-alerts-2:18.2.4-843.g2c01458a.el9.noarch 2024-09-13T14:02:59.034 INFO:teuthology.orchestra.run.smithi102.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-13T14:02:59.034 INFO:teuthology.orchestra.run.smithi102.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-13T14:02:59.034 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:02:59.034 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-09-13T14:02:59.251 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-13T14:02:59.818 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:00:59 ago on Fri 13 Sep 2024 02:02:00 PM UTC. 2024-09-13T14:02:59.951 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-09-13T14:02:59.952 INFO:teuthology.orchestra.run.smithi102.stdout:====================================================================================== 2024-09-13T14:02:59.952 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-09-13T14:02:59.952 INFO:teuthology.orchestra.run.smithi102.stdout:====================================================================================== 2024-09-13T14:02:59.952 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-09-13T14:02:59.952 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.4-843.g2c01458a.el9 ceph-noarch 7.4 M 2024-09-13T14:02:59.952 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-09-13T14:02:59.953 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-13T14:02:59.953 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-13T14:02:59.953 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-13T14:02:59.953 INFO:teuthology.orchestra.run.smithi102.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-13T14:02:59.953 INFO:teuthology.orchestra.run.smithi102.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-13T14:02:59.953 INFO:teuthology.orchestra.run.smithi102.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-13T14:02:59.953 INFO:teuthology.orchestra.run.smithi102.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-13T14:02:59.953 INFO:teuthology.orchestra.run.smithi102.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-13T14:02:59.953 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-13T14:02:59.953 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-13T14:02:59.953 INFO:teuthology.orchestra.run.smithi102.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-13T14:02:59.953 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:02:59.953 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-09-13T14:02:59.953 INFO:teuthology.orchestra.run.smithi102.stdout:====================================================================================== 2024-09-13T14:02:59.954 INFO:teuthology.orchestra.run.smithi102.stdout:Install 12 Packages 2024-09-13T14:02:59.954 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:02:59.954 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 38 M 2024-09-13T14:02:59.954 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 212 M 2024-09-13T14:02:59.954 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-09-13T14:03:00.640 INFO:teuthology.orchestra.run.smithi102.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 338 kB/s | 191 kB 00:00 2024-09-13T14:03:00.890 INFO:teuthology.orchestra.run.smithi102.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 982 kB/s | 801 kB 00:00 2024-09-13T14:03:01.265 INFO:teuthology.orchestra.run.smithi102.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 52 kB/s | 33 kB 00:00 2024-09-13T14:03:01.624 INFO:teuthology.orchestra.run.smithi102.stdout:(4/12): ceph-mgr-diskprediction-local-18.2.4-84 4.8 MB/s | 7.4 MB 00:01 2024-09-13T14:03:01.824 INFO:teuthology.orchestra.run.smithi102.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 33 kB/s | 18 kB 00:00 2024-09-13T14:03:02.041 INFO:teuthology.orchestra.run.smithi102.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.6 MB/s | 3.0 MB 00:01 2024-09-13T14:03:02.241 INFO:teuthology.orchestra.run.smithi102.stdout:(7/12): openblas-0.3.26-2.el9.x86_64.rpm 62 kB/s | 38 kB 00:00 2024-09-13T14:03:02.800 INFO:teuthology.orchestra.run.smithi102.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 323 kB/s | 245 kB 00:00 2024-09-13T14:03:03.075 INFO:teuthology.orchestra.run.smithi102.stdout:(9/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 4.0 MB/s | 4.9 MB 00:01 2024-09-13T14:03:03.409 INFO:teuthology.orchestra.run.smithi102.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.4 MB/s | 5.1 MB 00:01 2024-09-13T14:03:03.651 INFO:teuthology.orchestra.run.smithi102.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 295 kB/s | 250 kB 00:00 2024-09-13T14:03:04.534 INFO:teuthology.orchestra.run.smithi102.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-09-13T14:03:04.535 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:03:04.536 INFO:teuthology.orchestra.run.smithi102.stdout:Total 8.3 MB/s | 38 MB 00:04 2024-09-13T14:03:04.900 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-09-13T14:03:04.927 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-09-13T14:03:04.927 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-09-13T14:03:05.426 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-09-13T14:03:05.427 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-09-13T14:03:06.187 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-09-13T14:03:06.260 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-13T14:03:06.299 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-13T14:03:06.409 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-13T14:03:06.450 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-13T14:03:06.635 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-13T14:03:06.657 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-13T14:03:06.734 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-13T14:03:07.393 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-13T14:03:07.515 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-13T14:03:08.764 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T14:03:09.809 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-13T14:03:09.840 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.4-843.g2c0145 12/12 2024-09-13T14:03:12.358 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.4-843.g2c0145 12/12 2024-09-13T14:03:12.358 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.4-843.g2c0145 1/12 2024-09-13T14:03:12.358 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-13T14:03:12.358 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-13T14:03:12.358 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-13T14:03:12.358 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-13T14:03:12.358 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-13T14:03:12.358 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-13T14:03:12.358 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-13T14:03:12.359 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-13T14:03:12.359 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T14:03:12.359 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-13T14:03:12.793 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-13T14:03:12.793 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:03:12.793 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-09-13T14:03:12.794 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-diskprediction-local-2:18.2.4-843.g2c01458a.el9.noarch 2024-09-13T14:03:12.794 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-13T14:03:12.794 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-13T14:03:12.794 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-13T14:03:12.794 INFO:teuthology.orchestra.run.smithi102.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-13T14:03:12.794 INFO:teuthology.orchestra.run.smithi102.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-13T14:03:12.794 INFO:teuthology.orchestra.run.smithi102.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-13T14:03:12.794 INFO:teuthology.orchestra.run.smithi102.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-13T14:03:12.794 INFO:teuthology.orchestra.run.smithi102.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-13T14:03:12.794 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-13T14:03:12.794 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-13T14:03:12.794 INFO:teuthology.orchestra.run.smithi102.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-13T14:03:12.795 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:03:12.795 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-09-13T14:03:13.185 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-mgr-rook 2024-09-13T14:03:13.777 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:13 ago on Fri 13 Sep 2024 02:02:00 PM UTC. 2024-09-13T14:03:13.921 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-09-13T14:03:13.922 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T14:03:13.922 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-09-13T14:03:13.922 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T14:03:13.922 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-09-13T14:03:13.922 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-rook noarch 2:18.2.4-843.g2c01458a.el9 ceph-noarch 49 k 2024-09-13T14:03:13.922 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-09-13T14:03:13.922 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-13T14:03:13.922 INFO:teuthology.orchestra.run.smithi102.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-13T14:03:13.923 INFO:teuthology.orchestra.run.smithi102.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-13T14:03:13.923 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-13T14:03:13.923 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-13T14:03:13.923 INFO:teuthology.orchestra.run.smithi102.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-13T14:03:13.923 INFO:teuthology.orchestra.run.smithi102.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-13T14:03:13.923 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-13T14:03:13.923 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-13T14:03:13.923 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-13T14:03:13.923 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-13T14:03:13.923 INFO:teuthology.orchestra.run.smithi102.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-13T14:03:13.923 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:03:13.923 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-09-13T14:03:13.923 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T14:03:13.924 INFO:teuthology.orchestra.run.smithi102.stdout:Install 13 Packages 2024-09-13T14:03:13.924 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:03:13.924 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 2.2 M 2024-09-13T14:03:13.924 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 26 M 2024-09-13T14:03:13.924 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-09-13T14:03:14.199 INFO:teuthology.orchestra.run.smithi102.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 285 kB/s | 26 kB 00:00 2024-09-13T14:03:14.216 INFO:teuthology.orchestra.run.smithi102.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 180 kB/s | 19 kB 00:00 2024-09-13T14:03:14.333 INFO:teuthology.orchestra.run.smithi102.stdout:(3/13): ceph-mgr-rook-18.2.4-843.g2c01458a.el9. 220 kB/s | 49 kB 00:00 2024-09-13T14:03:14.358 INFO:teuthology.orchestra.run.smithi102.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.1 MB/s | 159 kB 00:00 2024-09-13T14:03:14.383 INFO:teuthology.orchestra.run.smithi102.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.3 MB/s | 222 kB 00:00 2024-09-13T14:03:14.408 INFO:teuthology.orchestra.run.smithi102.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-13T14:03:14.459 INFO:teuthology.orchestra.run.smithi102.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 427 kB/s | 32 kB 00:00 2024-09-13T14:03:14.484 INFO:teuthology.orchestra.run.smithi102.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 206 kB/s | 14 kB 00:00 2024-09-13T14:03:14.517 INFO:teuthology.orchestra.run.smithi102.stdout:(9/13): python3-google-auth-2.34.0-1.el9.noarch 3.8 MB/s | 220 kB 00:00 2024-09-13T14:03:14.543 INFO:teuthology.orchestra.run.smithi102.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-13T14:03:14.568 INFO:teuthology.orchestra.run.smithi102.stdout:(11/13): python3-pyasn1-modules-0.4.8-6.el9.noa 1.2 MB/s | 279 kB 00:00 2024-09-13T14:03:14.610 INFO:teuthology.orchestra.run.smithi102.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 8.1 MB/s | 1.0 MB 00:00 2024-09-13T14:03:14.635 INFO:teuthology.orchestra.run.smithi102.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 971 kB/s | 90 kB 00:00 2024-09-13T14:03:14.637 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:03:14.637 INFO:teuthology.orchestra.run.smithi102.stdout:Total 3.1 MB/s | 2.2 MB 00:00 2024-09-13T14:03:14.706 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-09-13T14:03:14.721 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-09-13T14:03:14.721 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-09-13T14:03:14.808 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-09-13T14:03:14.811 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-09-13T14:03:15.312 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-09-13T14:03:15.473 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-13T14:03:15.536 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-13T14:03:15.600 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-13T14:03:15.637 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-13T14:03:15.679 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-13T14:03:15.789 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-13T14:03:15.890 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-13T14:03:15.939 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-13T14:03:16.662 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-13T14:03:16.720 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-13T14:03:16.753 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-13T14:03:16.801 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-13T14:03:16.823 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-rook-2:18.2.4-843.g2c01458a.el9.noarch 13/13 2024-09-13T14:03:17.602 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.4-843.g2c01458a.el9.noarch 13/13 2024-09-13T14:03:17.602 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-rook-2:18.2.4-843.g2c01458a.el9.noarch 1/13 2024-09-13T14:03:17.602 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-13T14:03:17.603 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-13T14:03:17.603 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-13T14:03:17.603 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-13T14:03:17.603 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-13T14:03:17.603 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-13T14:03:17.603 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-13T14:03:17.603 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-13T14:03:17.603 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-13T14:03:17.603 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-13T14:03:17.603 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-13T14:03:17.933 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-13T14:03:17.933 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:03:17.933 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-09-13T14:03:17.933 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-rook-2:18.2.4-843.g2c01458a.el9.noarch 2024-09-13T14:03:17.933 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-13T14:03:17.933 INFO:teuthology.orchestra.run.smithi102.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-13T14:03:17.933 INFO:teuthology.orchestra.run.smithi102.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-13T14:03:17.933 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-13T14:03:17.933 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-13T14:03:17.933 INFO:teuthology.orchestra.run.smithi102.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-13T14:03:17.933 INFO:teuthology.orchestra.run.smithi102.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-13T14:03:17.933 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-13T14:03:17.934 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-13T14:03:17.934 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-13T14:03:17.934 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-13T14:03:17.934 INFO:teuthology.orchestra.run.smithi102.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-13T14:03:17.934 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:03:17.934 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-09-13T14:03:18.166 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-mgr-cephadm 2024-09-13T14:03:18.730 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:18 ago on Fri 13 Sep 2024 02:02:00 PM UTC. 2024-09-13T14:03:18.864 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-09-13T14:03:18.864 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T14:03:18.864 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-09-13T14:03:18.864 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T14:03:18.864 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-09-13T14:03:18.864 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-cephadm noarch 2:18.2.4-843.g2c01458a.el9 ceph-noarch 139 k 2024-09-13T14:03:18.865 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-09-13T14:03:18.865 INFO:teuthology.orchestra.run.smithi102.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-13T14:03:18.865 INFO:teuthology.orchestra.run.smithi102.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-13T14:03:18.865 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-13T14:03:18.865 INFO:teuthology.orchestra.run.smithi102.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-13T14:03:18.865 INFO:teuthology.orchestra.run.smithi102.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-13T14:03:18.865 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:03:18.865 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-09-13T14:03:18.865 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T14:03:18.865 INFO:teuthology.orchestra.run.smithi102.stdout:Install 6 Packages 2024-09-13T14:03:18.866 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:03:18.866 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 7.0 M 2024-09-13T14:03:18.866 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 33 M 2024-09-13T14:03:18.866 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-09-13T14:03:19.287 INFO:teuthology.orchestra.run.smithi102.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.8 MB/s | 248 kB 00:00 2024-09-13T14:03:19.346 INFO:teuthology.orchestra.run.smithi102.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 9.2 MB/s | 548 kB 00:00 2024-09-13T14:03:19.371 INFO:teuthology.orchestra.run.smithi102.stdout:(3/6): ceph-mgr-cephadm-18.2.4-843.g2c01458a.el 638 kB/s | 139 kB 00:00 2024-09-13T14:03:19.397 INFO:teuthology.orchestra.run.smithi102.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-09-13T14:03:19.438 INFO:teuthology.orchestra.run.smithi102.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.1 MB/s | 74 kB 00:00 2024-09-13T14:03:19.522 INFO:teuthology.orchestra.run.smithi102.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 6.0 MB 00:00 2024-09-13T14:03:19.524 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:03:19.525 INFO:teuthology.orchestra.run.smithi102.stdout:Total 11 MB/s | 7.0 MB 00:00 2024-09-13T14:03:19.594 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-09-13T14:03:19.605 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-09-13T14:03:19.605 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-09-13T14:03:19.705 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-09-13T14:03:19.705 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-09-13T14:03:20.011 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-09-13T14:03:20.172 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-13T14:03:20.224 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-13T14:03:20.761 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-13T14:03:20.805 INFO:teuthology.orchestra.run.smithi195.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 712 kB/s | 16 MB 00:23 2024-09-13T14:03:20.806 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:03:20.806 INFO:teuthology.orchestra.run.smithi195.stdout:Total 1.5 MB/s | 38 MB 00:25 2024-09-13T14:03:20.844 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-13T14:03:20.906 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-13T14:03:20.936 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-cephadm-2:18.2.4-843.g2c01458a.el9.noarch 6/6 2024-09-13T14:03:21.164 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-09-13T14:03:21.190 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-09-13T14:03:21.190 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-09-13T14:03:21.684 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-09-13T14:03:21.684 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-09-13T14:03:21.756 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.4-843.g2c01458a.el9.noarch 6/6 2024-09-13T14:03:21.756 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-cephadm-2:18.2.4-843.g2c01458a.el9.noarch 1/6 2024-09-13T14:03:21.756 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-13T14:03:21.756 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-13T14:03:21.756 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-13T14:03:21.756 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-13T14:03:22.024 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-13T14:03:22.024 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:03:22.024 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-09-13T14:03:22.024 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-cephadm-2:18.2.4-843.g2c01458a.el9.noarch 2024-09-13T14:03:22.024 INFO:teuthology.orchestra.run.smithi102.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-13T14:03:22.024 INFO:teuthology.orchestra.run.smithi102.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-13T14:03:22.025 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-13T14:03:22.025 INFO:teuthology.orchestra.run.smithi102.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-13T14:03:22.025 INFO:teuthology.orchestra.run.smithi102.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-13T14:03:22.025 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:03:22.025 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-09-13T14:03:22.278 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-fuse 2024-09-13T14:03:22.442 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-09-13T14:03:22.526 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-13T14:03:22.569 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-13T14:03:22.677 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-13T14:03:22.721 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-13T14:03:22.841 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:22 ago on Fri 13 Sep 2024 02:02:00 PM UTC. 2024-09-13T14:03:22.907 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-13T14:03:22.936 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-13T14:03:22.974 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-09-13T14:03:22.974 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T14:03:22.974 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-09-13T14:03:22.975 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T14:03:22.975 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-09-13T14:03:22.975 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-fuse x86_64 2:18.2.4-843.g2c01458a.el9 ceph 852 k 2024-09-13T14:03:22.975 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-09-13T14:03:22.975 INFO:teuthology.orchestra.run.smithi102.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-13T14:03:22.975 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:03:22.975 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-09-13T14:03:22.975 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T14:03:22.975 INFO:teuthology.orchestra.run.smithi102.stdout:Install 2 Packages 2024-09-13T14:03:22.975 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:03:22.976 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 932 k 2024-09-13T14:03:22.976 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 2.7 M 2024-09-13T14:03:22.976 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-09-13T14:03:23.014 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-13T14:03:23.516 INFO:teuthology.orchestra.run.smithi102.stdout:(1/2): ceph-fuse-18.2.4-843.g2c01458a.el9.x86_6 3.2 MB/s | 852 kB 00:00 2024-09-13T14:03:23.681 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-13T14:03:23.807 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-13T14:03:23.883 INFO:teuthology.orchestra.run.smithi102.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 127 kB/s | 80 kB 00:00 2024-09-13T14:03:23.883 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:03:23.884 INFO:teuthology.orchestra.run.smithi102.stdout:Total 1.0 MB/s | 932 kB 00:00 2024-09-13T14:03:23.894 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-09-13T14:03:23.916 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-09-13T14:03:23.916 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-09-13T14:03:23.995 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-09-13T14:03:23.996 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-09-13T14:03:24.178 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-09-13T14:03:24.309 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-13T14:03:24.390 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-fuse-2:18.2.4-843.g2c01458a.el9.x86_64 2/2 2024-09-13T14:03:25.040 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T14:03:25.082 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-fuse-2:18.2.4-843.g2c01458a.el9.x86_64 2/2 2024-09-13T14:03:25.082 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-fuse-2:18.2.4-843.g2c01458a.el9.x86_64 1/2 2024-09-13T14:03:25.358 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-13T14:03:25.358 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:03:25.358 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-09-13T14:03:25.358 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-fuse-2:18.2.4-843.g2c01458a.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-13T14:03:25.359 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:03:25.359 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-09-13T14:03:25.678 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install librados-devel 2024-09-13T14:03:26.076 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-13T14:03:26.111 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.4-843.g2c0145 12/12 2024-09-13T14:03:26.257 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:26 ago on Fri 13 Sep 2024 02:02:00 PM UTC. 2024-09-13T14:03:26.397 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-09-13T14:03:26.397 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T14:03:26.398 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repo Size 2024-09-13T14:03:26.398 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T14:03:26.398 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-09-13T14:03:26.398 INFO:teuthology.orchestra.run.smithi102.stdout: librados-devel x86_64 2:18.2.4-843.g2c01458a.el9 ceph 127 k 2024-09-13T14:03:26.398 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:03:26.398 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-09-13T14:03:26.398 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T14:03:26.398 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-09-13T14:03:26.398 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:03:26.398 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 127 k 2024-09-13T14:03:26.398 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 456 k 2024-09-13T14:03:26.398 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-09-13T14:03:26.604 INFO:teuthology.orchestra.run.smithi102.stdout:librados-devel-18.2.4-843.g2c01458a.el9.x86_64. 617 kB/s | 127 kB 00:00 2024-09-13T14:03:26.604 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:03:26.605 INFO:teuthology.orchestra.run.smithi102.stdout:Total 613 kB/s | 127 kB 00:00 2024-09-13T14:03:26.605 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-09-13T14:03:26.610 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-09-13T14:03:26.610 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-09-13T14:03:26.675 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-09-13T14:03:26.675 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-09-13T14:03:26.826 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-09-13T14:03:26.934 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librados-devel-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:03:27.376 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librados-devel-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:03:27.640 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librados-devel-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:03:27.640 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:03:27.640 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-09-13T14:03:27.640 INFO:teuthology.orchestra.run.smithi102.stdout: librados-devel-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:03:27.640 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:03:27.640 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-09-13T14:03:27.854 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install libcephfs2 2024-09-13T14:03:28.433 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:28 ago on Fri 13 Sep 2024 02:02:00 PM UTC. 2024-09-13T14:03:28.523 INFO:teuthology.orchestra.run.smithi102.stdout:Package libcephfs2-2:18.2.4-843.g2c01458a.el9.x86_64 is already installed. 2024-09-13T14:03:28.569 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-09-13T14:03:28.571 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-09-13T14:03:28.571 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-09-13T14:03:28.643 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install libcephfs-devel 2024-09-13T14:03:28.711 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.4-843.g2c0145 12/12 2024-09-13T14:03:28.711 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.4-843.g2c0145 1/12 2024-09-13T14:03:28.711 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-13T14:03:28.711 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-13T14:03:28.712 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-13T14:03:28.712 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-13T14:03:28.712 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-13T14:03:28.712 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-13T14:03:28.712 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-13T14:03:28.712 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-13T14:03:28.712 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T14:03:28.712 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-13T14:03:29.094 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-13T14:03:29.094 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:03:29.095 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-09-13T14:03:29.095 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-diskprediction-local-2:18.2.4-843.g2c01458a.el9.noarch 2024-09-13T14:03:29.095 INFO:teuthology.orchestra.run.smithi195.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-13T14:03:29.095 INFO:teuthology.orchestra.run.smithi195.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-13T14:03:29.095 INFO:teuthology.orchestra.run.smithi195.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-13T14:03:29.095 INFO:teuthology.orchestra.run.smithi195.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-13T14:03:29.095 INFO:teuthology.orchestra.run.smithi195.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-13T14:03:29.095 INFO:teuthology.orchestra.run.smithi195.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-13T14:03:29.095 INFO:teuthology.orchestra.run.smithi195.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-13T14:03:29.095 INFO:teuthology.orchestra.run.smithi195.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-13T14:03:29.095 INFO:teuthology.orchestra.run.smithi195.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-13T14:03:29.095 INFO:teuthology.orchestra.run.smithi195.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-13T14:03:29.095 INFO:teuthology.orchestra.run.smithi195.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-13T14:03:29.095 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:03:29.096 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-09-13T14:03:29.207 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:29 ago on Fri 13 Sep 2024 02:02:00 PM UTC. 2024-09-13T14:03:29.339 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-09-13T14:03:29.340 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T14:03:29.340 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repo Size 2024-09-13T14:03:29.340 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T14:03:29.340 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-09-13T14:03:29.340 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs-devel x86_64 2:18.2.4-843.g2c01458a.el9 ceph 31 k 2024-09-13T14:03:29.341 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:03:29.341 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-09-13T14:03:29.341 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T14:03:29.341 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-09-13T14:03:29.341 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:03:29.341 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 31 k 2024-09-13T14:03:29.341 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 139 k 2024-09-13T14:03:29.341 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-09-13T14:03:29.474 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-mgr-rook 2024-09-13T14:03:29.507 INFO:teuthology.orchestra.run.smithi102.stdout:libcephfs-devel-18.2.4-843.g2c01458a.el9.x86_64 186 kB/s | 31 kB 00:00 2024-09-13T14:03:29.508 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:03:29.508 INFO:teuthology.orchestra.run.smithi102.stdout:Total 185 kB/s | 31 kB 00:00 2024-09-13T14:03:29.508 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-09-13T14:03:29.512 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-09-13T14:03:29.512 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-09-13T14:03:29.528 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-09-13T14:03:29.529 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-09-13T14:03:29.606 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-09-13T14:03:29.729 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libcephfs-devel-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:03:30.050 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:33 ago on Fri 13 Sep 2024 02:01:57 PM UTC. 2024-09-13T14:03:30.139 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libcephfs-devel-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:03:30.184 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-09-13T14:03:30.185 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T14:03:30.185 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-09-13T14:03:30.185 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T14:03:30.186 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-09-13T14:03:30.186 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-rook noarch 2:18.2.4-843.g2c01458a.el9 ceph-noarch 49 k 2024-09-13T14:03:30.186 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-09-13T14:03:30.186 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-13T14:03:30.186 INFO:teuthology.orchestra.run.smithi195.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-13T14:03:30.186 INFO:teuthology.orchestra.run.smithi195.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-13T14:03:30.186 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-13T14:03:30.186 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-13T14:03:30.186 INFO:teuthology.orchestra.run.smithi195.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-13T14:03:30.186 INFO:teuthology.orchestra.run.smithi195.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-13T14:03:30.186 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-13T14:03:30.186 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-13T14:03:30.187 INFO:teuthology.orchestra.run.smithi195.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-13T14:03:30.187 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-13T14:03:30.187 INFO:teuthology.orchestra.run.smithi195.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-13T14:03:30.187 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:03:30.187 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-09-13T14:03:30.187 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T14:03:30.187 INFO:teuthology.orchestra.run.smithi195.stdout:Install 13 Packages 2024-09-13T14:03:30.187 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:03:30.188 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 2.2 M 2024-09-13T14:03:30.188 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 26 M 2024-09-13T14:03:30.188 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-09-13T14:03:30.381 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libcephfs-devel-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:03:30.381 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:03:30.381 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-09-13T14:03:30.381 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs-devel-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:03:30.381 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:03:30.381 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-09-13T14:03:30.587 INFO:teuthology.orchestra.run.smithi195.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 155 kB/s | 19 kB 00:00 2024-09-13T14:03:30.591 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install librados2 2024-09-13T14:03:30.612 INFO:teuthology.orchestra.run.smithi195.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 172 kB/s | 26 kB 00:00 2024-09-13T14:03:30.671 INFO:teuthology.orchestra.run.smithi195.stdout:(3/13): ceph-mgr-rook-18.2.4-843.g2c01458a.el9. 237 kB/s | 49 kB 00:00 2024-09-13T14:03:30.696 INFO:teuthology.orchestra.run.smithi195.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 222 kB 00:00 2024-09-13T14:03:30.721 INFO:teuthology.orchestra.run.smithi195.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-09-13T14:03:30.747 INFO:teuthology.orchestra.run.smithi195.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-13T14:03:30.797 INFO:teuthology.orchestra.run.smithi195.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 427 kB/s | 32 kB 00:00 2024-09-13T14:03:30.822 INFO:teuthology.orchestra.run.smithi195.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 206 kB/s | 14 kB 00:00 2024-09-13T14:03:30.872 INFO:teuthology.orchestra.run.smithi195.stdout:(9/13): python3-google-auth-2.34.0-1.el9.noarch 2.9 MB/s | 220 kB 00:00 2024-09-13T14:03:30.898 INFO:teuthology.orchestra.run.smithi195.stdout:(10/13): python3-pyasn1-modules-0.4.8-6.el9.noa 1.2 MB/s | 279 kB 00:00 2024-09-13T14:03:30.931 INFO:teuthology.orchestra.run.smithi195.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-09-13T14:03:30.973 INFO:teuthology.orchestra.run.smithi195.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.2 MB/s | 90 kB 00:00 2024-09-13T14:03:31.040 INFO:teuthology.orchestra.run.smithi195.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 4.7 MB/s | 1.0 MB 00:00 2024-09-13T14:03:31.042 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:03:31.043 INFO:teuthology.orchestra.run.smithi195.stdout:Total 2.6 MB/s | 2.2 MB 00:00 2024-09-13T14:03:31.109 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-09-13T14:03:31.124 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-09-13T14:03:31.125 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-09-13T14:03:31.152 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:31 ago on Fri 13 Sep 2024 02:02:00 PM UTC. 2024-09-13T14:03:31.212 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-09-13T14:03:31.212 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-09-13T14:03:31.243 INFO:teuthology.orchestra.run.smithi102.stdout:Package librados2-2:18.2.4-843.g2c01458a.el9.x86_64 is already installed. 2024-09-13T14:03:31.286 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-09-13T14:03:31.287 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-09-13T14:03:31.287 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-09-13T14:03:31.360 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install librbd1 2024-09-13T14:03:31.714 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-09-13T14:03:31.884 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-13T14:03:31.918 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:31 ago on Fri 13 Sep 2024 02:02:00 PM UTC. 2024-09-13T14:03:31.950 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-13T14:03:32.008 INFO:teuthology.orchestra.run.smithi102.stdout:Package librbd1-2:18.2.4-843.g2c01458a.el9.x86_64 is already installed. 2024-09-13T14:03:32.014 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-13T14:03:32.051 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-13T14:03:32.051 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-09-13T14:03:32.052 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-09-13T14:03:32.052 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-09-13T14:03:32.092 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-13T14:03:32.124 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install python3-rados 2024-09-13T14:03:32.196 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-13T14:03:32.313 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-13T14:03:32.361 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-13T14:03:32.691 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:32 ago on Fri 13 Sep 2024 02:02:00 PM UTC. 2024-09-13T14:03:32.782 INFO:teuthology.orchestra.run.smithi102.stdout:Package python3-rados-2:18.2.4-843.g2c01458a.el9.x86_64 is already installed. 2024-09-13T14:03:32.824 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-09-13T14:03:32.825 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-09-13T14:03:32.825 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-09-13T14:03:32.900 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install python3-rgw 2024-09-13T14:03:33.090 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-13T14:03:33.141 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-13T14:03:33.175 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-13T14:03:33.223 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-13T14:03:33.253 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-rook-2:18.2.4-843.g2c01458a.el9.noarch 13/13 2024-09-13T14:03:33.460 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:33 ago on Fri 13 Sep 2024 02:02:00 PM UTC. 2024-09-13T14:03:33.550 INFO:teuthology.orchestra.run.smithi102.stdout:Package python3-rgw-2:18.2.4-843.g2c01458a.el9.x86_64 is already installed. 2024-09-13T14:03:33.592 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-09-13T14:03:33.593 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-09-13T14:03:33.593 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-09-13T14:03:33.670 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install python3-cephfs 2024-09-13T14:03:34.035 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.4-843.g2c01458a.el9.noarch 13/13 2024-09-13T14:03:34.036 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-rook-2:18.2.4-843.g2c01458a.el9.noarch 1/13 2024-09-13T14:03:34.036 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-13T14:03:34.036 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-13T14:03:34.036 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-13T14:03:34.036 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-13T14:03:34.036 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-13T14:03:34.036 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-13T14:03:34.036 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-13T14:03:34.036 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-13T14:03:34.036 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-13T14:03:34.036 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-13T14:03:34.036 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-13T14:03:34.231 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:34 ago on Fri 13 Sep 2024 02:02:00 PM UTC. 2024-09-13T14:03:34.321 INFO:teuthology.orchestra.run.smithi102.stdout:Package python3-cephfs-2:18.2.4-843.g2c01458a.el9.x86_64 is already installed. 2024-09-13T14:03:34.349 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-13T14:03:34.350 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:03:34.350 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-09-13T14:03:34.350 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-rook-2:18.2.4-843.g2c01458a.el9.noarch 2024-09-13T14:03:34.350 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-13T14:03:34.350 INFO:teuthology.orchestra.run.smithi195.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-13T14:03:34.350 INFO:teuthology.orchestra.run.smithi195.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-13T14:03:34.351 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-13T14:03:34.351 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-13T14:03:34.351 INFO:teuthology.orchestra.run.smithi195.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-13T14:03:34.351 INFO:teuthology.orchestra.run.smithi195.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-13T14:03:34.351 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-13T14:03:34.351 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-13T14:03:34.351 INFO:teuthology.orchestra.run.smithi195.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-13T14:03:34.351 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-13T14:03:34.351 INFO:teuthology.orchestra.run.smithi195.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-13T14:03:34.351 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:03:34.352 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-09-13T14:03:34.363 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-09-13T14:03:34.364 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-09-13T14:03:34.364 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-09-13T14:03:34.459 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install python3-rbd 2024-09-13T14:03:34.604 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-mgr-cephadm 2024-09-13T14:03:35.023 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:35 ago on Fri 13 Sep 2024 02:02:00 PM UTC. 2024-09-13T14:03:35.113 INFO:teuthology.orchestra.run.smithi102.stdout:Package python3-rbd-2:18.2.4-843.g2c01458a.el9.x86_64 is already installed. 2024-09-13T14:03:35.157 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-09-13T14:03:35.157 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-09-13T14:03:35.157 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-09-13T14:03:35.167 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:38 ago on Fri 13 Sep 2024 02:01:57 PM UTC. 2024-09-13T14:03:35.228 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install rbd-fuse 2024-09-13T14:03:35.300 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-09-13T14:03:35.301 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T14:03:35.301 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-09-13T14:03:35.301 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T14:03:35.301 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-09-13T14:03:35.301 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-cephadm noarch 2:18.2.4-843.g2c01458a.el9 ceph-noarch 139 k 2024-09-13T14:03:35.301 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-09-13T14:03:35.301 INFO:teuthology.orchestra.run.smithi195.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-13T14:03:35.301 INFO:teuthology.orchestra.run.smithi195.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-13T14:03:35.301 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-13T14:03:35.301 INFO:teuthology.orchestra.run.smithi195.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-13T14:03:35.301 INFO:teuthology.orchestra.run.smithi195.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-13T14:03:35.301 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:03:35.302 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-09-13T14:03:35.302 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T14:03:35.302 INFO:teuthology.orchestra.run.smithi195.stdout:Install 6 Packages 2024-09-13T14:03:35.302 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:03:35.302 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 7.0 M 2024-09-13T14:03:35.302 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 33 M 2024-09-13T14:03:35.302 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-09-13T14:03:35.796 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:35 ago on Fri 13 Sep 2024 02:02:00 PM UTC. 2024-09-13T14:03:35.800 INFO:teuthology.orchestra.run.smithi195.stdout:(1/6): ceph-mgr-cephadm-18.2.4-843.g2c01458a.el 741 kB/s | 139 kB 00:00 2024-09-13T14:03:35.826 INFO:teuthology.orchestra.run.smithi195.stdout:(2/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.1 MB/s | 248 kB 00:00 2024-09-13T14:03:35.884 INFO:teuthology.orchestra.run.smithi195.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-09-13T14:03:35.918 INFO:teuthology.orchestra.run.smithi195.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.2 MB/s | 74 kB 00:00 2024-09-13T14:03:35.929 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-09-13T14:03:35.930 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T14:03:35.930 INFO:teuthology.orchestra.run.smithi102.stdout: Package Architecture Version Repository Size 2024-09-13T14:03:35.930 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T14:03:35.930 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-09-13T14:03:35.930 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-fuse x86_64 2:18.2.4-843.g2c01458a.el9 ceph 87 k 2024-09-13T14:03:35.930 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:03:35.930 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-09-13T14:03:35.930 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T14:03:35.931 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-09-13T14:03:35.931 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:03:35.931 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 87 k 2024-09-13T14:03:35.931 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 227 k 2024-09-13T14:03:35.931 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-09-13T14:03:35.960 INFO:teuthology.orchestra.run.smithi195.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 3.4 MB/s | 548 kB 00:00 2024-09-13T14:03:36.120 INFO:teuthology.orchestra.run.smithi102.stdout:rbd-fuse-18.2.4-843.g2c01458a.el9.x86_64.rpm 460 kB/s | 87 kB 00:00 2024-09-13T14:03:36.121 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:03:36.121 INFO:teuthology.orchestra.run.smithi102.stdout:Total 457 kB/s | 87 kB 00:00 2024-09-13T14:03:36.121 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-09-13T14:03:36.127 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-09-13T14:03:36.128 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-09-13T14:03:36.191 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-09-13T14:03:36.191 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-09-13T14:03:36.335 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-09-13T14:03:36.433 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-fuse-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:03:36.798 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: rbd-fuse-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:03:37.051 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-fuse-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:03:37.051 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:03:37.051 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-09-13T14:03:37.051 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-fuse-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:03:37.051 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:03:37.052 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-09-13T14:03:37.320 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install rbd-mirror 2024-09-13T14:03:37.777 INFO:teuthology.orchestra.run.smithi195.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 2.8 MB/s | 6.0 MB 00:02 2024-09-13T14:03:37.778 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:03:37.779 INFO:teuthology.orchestra.run.smithi195.stdout:Total 2.8 MB/s | 7.0 MB 00:02 2024-09-13T14:03:37.853 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-09-13T14:03:37.866 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-09-13T14:03:37.866 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-09-13T14:03:37.898 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:37 ago on Fri 13 Sep 2024 02:02:00 PM UTC. 2024-09-13T14:03:37.970 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-09-13T14:03:37.970 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-09-13T14:03:38.037 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-09-13T14:03:38.038 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T14:03:38.038 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repo Size 2024-09-13T14:03:38.038 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T14:03:38.038 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-09-13T14:03:38.038 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-mirror x86_64 2:18.2.4-843.g2c01458a.el9 ceph 3.0 M 2024-09-13T14:03:38.039 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:03:38.039 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-09-13T14:03:38.039 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T14:03:38.039 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-09-13T14:03:38.039 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:03:38.039 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 3.0 M 2024-09-13T14:03:38.039 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 12 M 2024-09-13T14:03:38.039 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-09-13T14:03:38.277 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-09-13T14:03:38.398 INFO:teuthology.orchestra.run.smithi102.stdout:rbd-mirror-18.2.4-843.g2c01458a.el9.x86_64.rpm 8.3 MB/s | 3.0 MB 00:00 2024-09-13T14:03:38.399 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:03:38.399 INFO:teuthology.orchestra.run.smithi102.stdout:Total 8.3 MB/s | 3.0 MB 00:00 2024-09-13T14:03:38.399 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-09-13T14:03:38.407 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-09-13T14:03:38.407 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-09-13T14:03:38.459 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-13T14:03:38.493 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-09-13T14:03:38.494 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-09-13T14:03:38.511 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-13T14:03:38.934 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-09-13T14:03:39.013 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-mirror-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:03:39.036 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-13T14:03:39.036 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: rbd-mirror-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:03:39.037 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:03:39.037 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-13T14:03:39.037 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T14:03:39.037 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T14:03:39.037 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:03:39.131 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-13T14:03:39.201 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-13T14:03:39.224 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-cephadm-2:18.2.4-843.g2c01458a.el9.noarch 6/6 2024-09-13T14:03:40.042 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.4-843.g2c01458a.el9.noarch 6/6 2024-09-13T14:03:40.042 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-cephadm-2:18.2.4-843.g2c01458a.el9.noarch 1/6 2024-09-13T14:03:40.042 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-13T14:03:40.042 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-13T14:03:40.042 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-13T14:03:40.042 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-13T14:03:40.147 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-mirror-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:03:40.147 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:03:40.147 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-09-13T14:03:40.147 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-mirror-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:03:40.147 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:03:40.147 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-09-13T14:03:40.312 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-13T14:03:40.312 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:03:40.312 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-09-13T14:03:40.313 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-cephadm-2:18.2.4-843.g2c01458a.el9.noarch 2024-09-13T14:03:40.313 INFO:teuthology.orchestra.run.smithi195.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-13T14:03:40.313 INFO:teuthology.orchestra.run.smithi195.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-13T14:03:40.313 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-13T14:03:40.313 INFO:teuthology.orchestra.run.smithi195.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-13T14:03:40.313 INFO:teuthology.orchestra.run.smithi195.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-13T14:03:40.313 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:03:40.313 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-09-13T14:03:40.402 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install rbd-nbd 2024-09-13T14:03:40.540 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-fuse 2024-09-13T14:03:40.991 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:40 ago on Fri 13 Sep 2024 02:02:00 PM UTC. 2024-09-13T14:03:41.111 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:44 ago on Fri 13 Sep 2024 02:01:57 PM UTC. 2024-09-13T14:03:41.126 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-09-13T14:03:41.126 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T14:03:41.127 INFO:teuthology.orchestra.run.smithi102.stdout: Package Architecture Version Repository Size 2024-09-13T14:03:41.127 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T14:03:41.127 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-09-13T14:03:41.127 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-nbd x86_64 2:18.2.4-843.g2c01458a.el9 ceph 172 k 2024-09-13T14:03:41.127 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:03:41.127 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-09-13T14:03:41.127 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-09-13T14:03:41.127 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-09-13T14:03:41.127 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:03:41.128 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 172 k 2024-09-13T14:03:41.128 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 494 k 2024-09-13T14:03:41.128 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-09-13T14:03:41.245 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-09-13T14:03:41.246 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T14:03:41.246 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-09-13T14:03:41.246 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T14:03:41.246 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-09-13T14:03:41.246 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-fuse x86_64 2:18.2.4-843.g2c01458a.el9 ceph 852 k 2024-09-13T14:03:41.246 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-09-13T14:03:41.246 INFO:teuthology.orchestra.run.smithi195.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-13T14:03:41.246 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:03:41.246 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-09-13T14:03:41.246 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T14:03:41.247 INFO:teuthology.orchestra.run.smithi195.stdout:Install 2 Packages 2024-09-13T14:03:41.247 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:03:41.247 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 932 k 2024-09-13T14:03:41.247 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 2.7 M 2024-09-13T14:03:41.247 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-09-13T14:03:41.342 INFO:teuthology.orchestra.run.smithi102.stdout:rbd-nbd-18.2.4-843.g2c01458a.el9.x86_64.rpm 803 kB/s | 172 kB 00:00 2024-09-13T14:03:41.343 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:03:41.344 INFO:teuthology.orchestra.run.smithi102.stdout:Total 796 kB/s | 172 kB 00:00 2024-09-13T14:03:41.344 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-09-13T14:03:41.351 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-09-13T14:03:41.351 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-09-13T14:03:41.414 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-09-13T14:03:41.414 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-09-13T14:03:41.598 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-09-13T14:03:41.628 INFO:teuthology.orchestra.run.smithi195.stdout:(1/2): ceph-fuse-18.2.4-843.g2c01458a.el9.x86_6 3.3 MB/s | 852 kB 00:00 2024-09-13T14:03:41.707 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-nbd-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:03:42.054 INFO:teuthology.orchestra.run.smithi195.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 117 kB/s | 80 kB 00:00 2024-09-13T14:03:42.054 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:03:42.055 INFO:teuthology.orchestra.run.smithi195.stdout:Total 1.1 MB/s | 932 kB 00:00 2024-09-13T14:03:42.068 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-09-13T14:03:42.092 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-09-13T14:03:42.092 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-09-13T14:03:42.121 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: rbd-nbd-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:03:42.175 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-09-13T14:03:42.175 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-09-13T14:03:42.361 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-09-13T14:03:42.368 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-nbd-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:03:42.368 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:03:42.368 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-09-13T14:03:42.368 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-nbd-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:03:42.368 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:03:42.368 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-09-13T14:03:42.485 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-13T14:03:42.561 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-fuse-2:18.2.4-843.g2c01458a.el9.x86_64 2/2 2024-09-13T14:03:42.649 DEBUG:teuthology.parallel:result is None 2024-09-13T14:03:43.222 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-fuse-2:18.2.4-843.g2c01458a.el9.x86_64 2/2 2024-09-13T14:03:43.222 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-fuse-2:18.2.4-843.g2c01458a.el9.x86_64 1/2 2024-09-13T14:03:43.463 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-13T14:03:43.463 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:03:43.463 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-09-13T14:03:43.463 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-fuse-2:18.2.4-843.g2c01458a.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-13T14:03:43.463 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:03:43.463 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-09-13T14:03:43.734 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install librados-devel 2024-09-13T14:03:44.304 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:47 ago on Fri 13 Sep 2024 02:01:57 PM UTC. 2024-09-13T14:03:44.442 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-09-13T14:03:44.442 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T14:03:44.442 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2024-09-13T14:03:44.443 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T14:03:44.443 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-09-13T14:03:44.443 INFO:teuthology.orchestra.run.smithi195.stdout: librados-devel x86_64 2:18.2.4-843.g2c01458a.el9 ceph 127 k 2024-09-13T14:03:44.443 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:03:44.443 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-09-13T14:03:44.443 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T14:03:44.443 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-09-13T14:03:44.443 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:03:44.443 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 127 k 2024-09-13T14:03:44.443 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 456 k 2024-09-13T14:03:44.443 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-09-13T14:03:44.825 INFO:teuthology.orchestra.run.smithi195.stdout:librados-devel-18.2.4-843.g2c01458a.el9.x86_64. 332 kB/s | 127 kB 00:00 2024-09-13T14:03:44.825 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:03:44.826 INFO:teuthology.orchestra.run.smithi195.stdout:Total 331 kB/s | 127 kB 00:00 2024-09-13T14:03:44.826 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-09-13T14:03:44.831 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-09-13T14:03:44.831 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-09-13T14:03:44.895 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-09-13T14:03:44.895 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-09-13T14:03:45.042 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-09-13T14:03:45.146 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librados-devel-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:03:45.548 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librados-devel-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:03:45.787 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librados-devel-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:03:45.788 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:03:45.788 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-09-13T14:03:45.788 INFO:teuthology.orchestra.run.smithi195.stdout: librados-devel-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:03:45.788 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:03:45.788 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-09-13T14:03:46.010 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install libcephfs2 2024-09-13T14:03:46.582 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:49 ago on Fri 13 Sep 2024 02:01:57 PM UTC. 2024-09-13T14:03:46.675 INFO:teuthology.orchestra.run.smithi195.stdout:Package libcephfs2-2:18.2.4-843.g2c01458a.el9.x86_64 is already installed. 2024-09-13T14:03:46.721 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-09-13T14:03:46.721 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-09-13T14:03:46.721 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-09-13T14:03:46.799 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install libcephfs-devel 2024-09-13T14:03:47.365 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:50 ago on Fri 13 Sep 2024 02:01:57 PM UTC. 2024-09-13T14:03:47.500 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-09-13T14:03:47.500 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T14:03:47.501 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2024-09-13T14:03:47.501 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T14:03:47.501 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-09-13T14:03:47.501 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs-devel x86_64 2:18.2.4-843.g2c01458a.el9 ceph 31 k 2024-09-13T14:03:47.501 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:03:47.501 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-09-13T14:03:47.501 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T14:03:47.501 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-09-13T14:03:47.501 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:03:47.501 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 31 k 2024-09-13T14:03:47.501 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 139 k 2024-09-13T14:03:47.501 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-09-13T14:03:47.786 INFO:teuthology.orchestra.run.smithi195.stdout:libcephfs-devel-18.2.4-843.g2c01458a.el9.x86_64 109 kB/s | 31 kB 00:00 2024-09-13T14:03:47.787 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:03:47.787 INFO:teuthology.orchestra.run.smithi195.stdout:Total 108 kB/s | 31 kB 00:00 2024-09-13T14:03:47.787 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-09-13T14:03:47.791 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-09-13T14:03:47.791 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-09-13T14:03:47.810 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-09-13T14:03:47.810 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-09-13T14:03:47.900 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-09-13T14:03:48.016 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libcephfs-devel-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:03:48.383 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libcephfs-devel-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:03:48.602 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libcephfs-devel-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:03:48.602 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:03:48.602 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-09-13T14:03:48.602 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs-devel-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:03:48.602 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:03:48.602 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-09-13T14:03:48.794 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install librados2 2024-09-13T14:03:49.361 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:52 ago on Fri 13 Sep 2024 02:01:57 PM UTC. 2024-09-13T14:03:49.451 INFO:teuthology.orchestra.run.smithi195.stdout:Package librados2-2:18.2.4-843.g2c01458a.el9.x86_64 is already installed. 2024-09-13T14:03:49.493 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-09-13T14:03:49.494 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-09-13T14:03:49.495 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-09-13T14:03:49.573 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install librbd1 2024-09-13T14:03:50.135 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:53 ago on Fri 13 Sep 2024 02:01:57 PM UTC. 2024-09-13T14:03:50.225 INFO:teuthology.orchestra.run.smithi195.stdout:Package librbd1-2:18.2.4-843.g2c01458a.el9.x86_64 is already installed. 2024-09-13T14:03:50.267 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-09-13T14:03:50.268 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-09-13T14:03:50.268 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-09-13T14:03:50.350 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install python3-rados 2024-09-13T14:03:50.919 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:53 ago on Fri 13 Sep 2024 02:01:57 PM UTC. 2024-09-13T14:03:51.009 INFO:teuthology.orchestra.run.smithi195.stdout:Package python3-rados-2:18.2.4-843.g2c01458a.el9.x86_64 is already installed. 2024-09-13T14:03:51.052 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-09-13T14:03:51.053 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-09-13T14:03:51.053 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-09-13T14:03:51.129 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install python3-rgw 2024-09-13T14:03:51.697 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:54 ago on Fri 13 Sep 2024 02:01:57 PM UTC. 2024-09-13T14:03:51.787 INFO:teuthology.orchestra.run.smithi195.stdout:Package python3-rgw-2:18.2.4-843.g2c01458a.el9.x86_64 is already installed. 2024-09-13T14:03:51.831 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-09-13T14:03:51.832 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-09-13T14:03:51.832 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-09-13T14:03:51.909 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install python3-cephfs 2024-09-13T14:03:52.471 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:55 ago on Fri 13 Sep 2024 02:01:57 PM UTC. 2024-09-13T14:03:52.561 INFO:teuthology.orchestra.run.smithi195.stdout:Package python3-cephfs-2:18.2.4-843.g2c01458a.el9.x86_64 is already installed. 2024-09-13T14:03:52.603 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-09-13T14:03:52.604 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-09-13T14:03:52.604 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-09-13T14:03:52.682 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install python3-rbd 2024-09-13T14:03:53.246 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:56 ago on Fri 13 Sep 2024 02:01:57 PM UTC. 2024-09-13T14:03:53.337 INFO:teuthology.orchestra.run.smithi195.stdout:Package python3-rbd-2:18.2.4-843.g2c01458a.el9.x86_64 is already installed. 2024-09-13T14:03:53.379 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-09-13T14:03:53.380 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-09-13T14:03:53.381 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-09-13T14:03:53.463 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install rbd-fuse 2024-09-13T14:03:54.030 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:57 ago on Fri 13 Sep 2024 02:01:57 PM UTC. 2024-09-13T14:03:54.163 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-09-13T14:03:54.163 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T14:03:54.163 INFO:teuthology.orchestra.run.smithi195.stdout: Package Architecture Version Repository Size 2024-09-13T14:03:54.163 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T14:03:54.163 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-09-13T14:03:54.163 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-fuse x86_64 2:18.2.4-843.g2c01458a.el9 ceph 87 k 2024-09-13T14:03:54.163 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:03:54.163 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-09-13T14:03:54.164 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T14:03:54.164 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-09-13T14:03:54.164 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:03:54.164 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 87 k 2024-09-13T14:03:54.164 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 227 k 2024-09-13T14:03:54.164 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-09-13T14:03:54.607 INFO:teuthology.orchestra.run.smithi195.stdout:rbd-fuse-18.2.4-843.g2c01458a.el9.x86_64.rpm 196 kB/s | 87 kB 00:00 2024-09-13T14:03:54.608 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:03:54.608 INFO:teuthology.orchestra.run.smithi195.stdout:Total 195 kB/s | 87 kB 00:00 2024-09-13T14:03:54.608 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-09-13T14:03:54.615 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-09-13T14:03:54.615 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-09-13T14:03:54.679 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-09-13T14:03:54.679 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-09-13T14:03:54.830 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-09-13T14:03:54.938 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : rbd-fuse-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:03:55.322 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: rbd-fuse-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:03:55.567 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : rbd-fuse-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:03:55.567 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:03:55.567 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-09-13T14:03:55.567 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-fuse-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:03:55.567 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:03:55.567 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-09-13T14:03:55.809 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install rbd-mirror 2024-09-13T14:03:56.391 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:59 ago on Fri 13 Sep 2024 02:01:57 PM UTC. 2024-09-13T14:03:56.537 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-09-13T14:03:56.538 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T14:03:56.538 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2024-09-13T14:03:56.538 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T14:03:56.538 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-09-13T14:03:56.539 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-mirror x86_64 2:18.2.4-843.g2c01458a.el9 ceph 3.0 M 2024-09-13T14:03:56.539 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:03:56.539 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-09-13T14:03:56.539 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T14:03:56.539 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-09-13T14:03:56.539 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:03:56.539 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 3.0 M 2024-09-13T14:03:56.539 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 12 M 2024-09-13T14:03:56.539 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-09-13T14:03:58.786 INFO:teuthology.orchestra.run.smithi195.stdout:rbd-mirror-18.2.4-843.g2c01458a.el9.x86_64.rpm 1.3 MB/s | 3.0 MB 00:02 2024-09-13T14:03:58.787 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:03:58.787 INFO:teuthology.orchestra.run.smithi195.stdout:Total 1.3 MB/s | 3.0 MB 00:02 2024-09-13T14:03:58.788 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-09-13T14:03:58.799 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-09-13T14:03:58.799 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-09-13T14:03:58.889 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-09-13T14:03:58.890 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-09-13T14:03:59.324 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-09-13T14:03:59.395 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : rbd-mirror-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:03:59.421 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: rbd-mirror-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:03:59.421 INFO:teuthology.orchestra.run.smithi195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:03:59.421 INFO:teuthology.orchestra.run.smithi195.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-13T14:03:59.421 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T14:03:59.421 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T14:03:59.421 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:04:00.453 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : rbd-mirror-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:04:00.454 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:04:00.454 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-09-13T14:04:00.454 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-mirror-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:04:00.454 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:04:00.454 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-09-13T14:04:00.714 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install rbd-nbd 2024-09-13T14:04:01.302 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:04 ago on Fri 13 Sep 2024 02:01:57 PM UTC. 2024-09-13T14:04:01.437 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-09-13T14:04:01.439 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T14:04:01.439 INFO:teuthology.orchestra.run.smithi195.stdout: Package Architecture Version Repository Size 2024-09-13T14:04:01.439 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T14:04:01.439 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-09-13T14:04:01.439 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-nbd x86_64 2:18.2.4-843.g2c01458a.el9 ceph 172 k 2024-09-13T14:04:01.440 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:04:01.440 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-09-13T14:04:01.440 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-09-13T14:04:01.440 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-09-13T14:04:01.441 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:04:01.441 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 172 k 2024-09-13T14:04:01.441 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 494 k 2024-09-13T14:04:01.441 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-09-13T14:04:01.681 INFO:teuthology.orchestra.run.smithi195.stdout:rbd-nbd-18.2.4-843.g2c01458a.el9.x86_64.rpm 712 kB/s | 172 kB 00:00 2024-09-13T14:04:01.681 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:04:01.681 INFO:teuthology.orchestra.run.smithi195.stdout:Total 707 kB/s | 172 kB 00:00 2024-09-13T14:04:01.682 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-09-13T14:04:01.689 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-09-13T14:04:01.689 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-09-13T14:04:01.752 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-09-13T14:04:01.753 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-09-13T14:04:01.912 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-09-13T14:04:01.995 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : rbd-nbd-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:04:02.396 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: rbd-nbd-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:04:02.639 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : rbd-nbd-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:04:02.639 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:04:02.639 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-09-13T14:04:02.639 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-nbd-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:04:02.639 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:04:02.640 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-09-13T14:04:02.876 DEBUG:teuthology.parallel:result is None 2024-09-13T14:04:02.877 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-13T14:04:02.877 INFO:teuthology.packaging:ref: None 2024-09-13T14:04:02.877 INFO:teuthology.packaging:tag: None 2024-09-13T14:04:02.877 INFO:teuthology.packaging:branch: reef 2024-09-13T14:04:02.877 INFO:teuthology.packaging:sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:04:02.877 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-09-13T14:04:03.110 DEBUG:teuthology.orchestra.run.smithi102:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-13T14:04:03.140 INFO:teuthology.orchestra.run.smithi102.stdout:18.2.4-843.g2c01458a.el9 2024-09-13T14:04:03.141 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-843.g2c01458a.el9 2024-09-13T14:04:03.142 INFO:teuthology.task.install:The correct ceph version 18.2.4-843.g2c01458a is installed. 2024-09-13T14:04:03.144 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-13T14:04:03.144 INFO:teuthology.packaging:ref: None 2024-09-13T14:04:03.144 INFO:teuthology.packaging:tag: None 2024-09-13T14:04:03.144 INFO:teuthology.packaging:branch: reef 2024-09-13T14:04:03.144 INFO:teuthology.packaging:sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:04:03.144 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-09-13T14:04:03.380 DEBUG:teuthology.orchestra.run.smithi195:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-13T14:04:03.403 INFO:teuthology.orchestra.run.smithi195.stdout:18.2.4-843.g2c01458a.el9 2024-09-13T14:04:03.403 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-843.g2c01458a.el9 2024-09-13T14:04:03.404 INFO:teuthology.task.install:The correct ceph version 18.2.4-843.g2c01458a is installed. 2024-09-13T14:04:03.406 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-13T14:04:03.406 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-09-13T14:04:03.407 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T14:04:03.438 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-09-13T14:04:03.438 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T14:04:03.470 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-13T14:04:03.470 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-09-13T14:04:03.471 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T14:04:03.508 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T14:04:03.578 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-09-13T14:04:03.578 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T14:04:03.604 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T14:04:03.669 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-13T14:04:03.670 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-09-13T14:04:03.670 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T14:04:03.697 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T14:04:03.765 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-09-13T14:04:03.765 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T14:04:03.790 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T14:04:03.856 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-13T14:04:03.857 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-09-13T14:04:03.857 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T14:04:03.883 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T14:04:03.950 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-09-13T14:04:03.950 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T14:04:03.975 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T14:04:04.041 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-13T14:04:04.138 INFO:tasks.cephadm:Config: {'compiled_cephadm_branch': 'reef', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}}, 'image': 'quay.ceph.io/ceph-ci/ceph:reef', 'create_rbd_pool': True, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'mons down', 'mon down', 'MON_DOWN', 'out of quorum', 'PG_AVAILABILITY'], 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'} 2024-09-13T14:04:04.139 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:reef 2024-09-13T14:04:04.139 INFO:tasks.cephadm:Cluster fsid is 08cee502-71d9-11ef-bceb-c7b262605968 2024-09-13T14:04:04.139 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-13T14:04:04.139 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.102', 'mon.c': '[v2:172.21.15.102:3301,v1:172.21.15.102:6790]', 'mon.b': '172.21.15.195'} 2024-09-13T14:04:04.140 INFO:tasks.cephadm:First mon is mon.a on smithi102 2024-09-13T14:04:04.140 INFO:tasks.cephadm:First mgr is y 2024-09-13T14:04:04.140 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-13T14:04:04.140 DEBUG:teuthology.orchestra.run.smithi102:> sudo hostname $(hostname -s) 2024-09-13T14:04:04.166 DEBUG:teuthology.orchestra.run.smithi195:> sudo hostname $(hostname -s) 2024-09-13T14:04:04.192 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra for reef 2024-09-13T14:04:04.192 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:04:04.341 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/83028/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.3.0-4697-gd58f7306', 'node_name': '172.21.4.75+confusa13', 'job_name': 'ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.3.0-4697.gd58f7306'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-09-12 23:59:16.034150', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-aclamk-cbt-combined', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/', 'archs': ['source', 'x86_64', 'arm64'], 'distro': 'centos'}] 2024-09-13T14:04:04.618 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref reef, sha1 2c01458affee2b2435cd231b6e392435c1c21661 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&ref=reef 2024-09-13T14:04:04.619 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/reef/2c01458affee2b2435cd231b6e392435c1c21661/centos/9/x86_64/flavors/default/cephadm 2024-09-13T14:04:04.619 INFO:tasks.cephadm:Downloading cephadm from url: https://1.chacra.ceph.com/binaries/ceph/reef/2c01458affee2b2435cd231b6e392435c1c21661/centos/9/x86_64/flavors/default/cephadm 2024-09-13T14:04:04.620 DEBUG:teuthology.orchestra.run.smithi102:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/reef/2c01458affee2b2435cd231b6e392435c1c21661/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-13T14:04:04.967 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r--. 1 ubuntu ubuntu 217218 Sep 13 14:04 /home/ubuntu/cephtest/cephadm 2024-09-13T14:04:04.968 DEBUG:teuthology.orchestra.run.smithi195:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/reef/2c01458affee2b2435cd231b6e392435c1c21661/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-13T14:04:05.319 INFO:teuthology.orchestra.run.smithi195.stdout:-rw-r--r--. 1 ubuntu ubuntu 217218 Sep 13 14:04 /home/ubuntu/cephtest/cephadm 2024-09-13T14:04:05.320 DEBUG:teuthology.orchestra.run.smithi102:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-13T14:04:05.339 DEBUG:teuthology.orchestra.run.smithi195:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-13T14:04:05.370 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:reef on all hosts... 2024-09-13T14:04:05.371 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef pull 2024-09-13T14:04:05.381 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef pull 2024-09-13T14:04:06.962 INFO:teuthology.orchestra.run.smithi195.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-09-13T14:04:07.008 INFO:teuthology.orchestra.run.smithi102.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-09-13T14:04:33.785 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:04:33.785 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph_version": "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)", 2024-09-13T14:04:33.786 INFO:teuthology.orchestra.run.smithi102.stdout: "image_id": "c129f5a2fd4eccce3e6eec43830eff1929af98bc48f3b850cf144ce5b326e8b5", 2024-09-13T14:04:33.786 INFO:teuthology.orchestra.run.smithi102.stdout: "repo_digests": [ 2024-09-13T14:04:33.786 INFO:teuthology.orchestra.run.smithi102.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11" 2024-09-13T14:04:33.786 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:04:33.786 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:04:39.174 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:04:39.174 INFO:teuthology.orchestra.run.smithi195.stdout: "ceph_version": "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)", 2024-09-13T14:04:39.175 INFO:teuthology.orchestra.run.smithi195.stdout: "image_id": "c129f5a2fd4eccce3e6eec43830eff1929af98bc48f3b850cf144ce5b326e8b5", 2024-09-13T14:04:39.175 INFO:teuthology.orchestra.run.smithi195.stdout: "repo_digests": [ 2024-09-13T14:04:39.175 INFO:teuthology.orchestra.run.smithi195.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11" 2024-09-13T14:04:39.175 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:04:39.175 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:04:39.193 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /etc/ceph 2024-09-13T14:04:39.226 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /etc/ceph 2024-09-13T14:04:39.254 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 777 /etc/ceph 2024-09-13T14:04:39.294 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 777 /etc/ceph 2024-09-13T14:04:39.319 INFO:tasks.cephadm:Writing seed config... 2024-09-13T14:04:39.320 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-09-13T14:04:39.320 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-09-13T14:04:39.320 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-13T14:04:39.321 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-13T14:04:39.321 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-09-13T14:04:39.321 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-09-13T14:04:39.321 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-13T14:04:39.321 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-13T14:04:39.321 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-13T14:04:39.321 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-13T14:04:39.321 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-13T14:04:39.322 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-09-13T14:04:39.322 DEBUG:teuthology.orchestra.run.smithi102:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-13T14:04:39.351 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 08cee502-71d9-11ef-bceb-c7b262605968 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-09-13T14:04:39.352 DEBUG:teuthology.orchestra.run.smithi102:mon.a> sudo journalctl -f -n 0 -u ceph-08cee502-71d9-11ef-bceb-c7b262605968@mon.a.service 2024-09-13T14:04:39.394 DEBUG:teuthology.orchestra.run.smithi102:mgr.y> sudo journalctl -f -n 0 -u ceph-08cee502-71d9-11ef-bceb-c7b262605968@mgr.y.service 2024-09-13T14:04:39.438 INFO:tasks.cephadm:Bootstrapping... 2024-09-13T14:04:39.438 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef -v bootstrap --fsid 08cee502-71d9-11ef-bceb-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.102 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-13T14:04:39.590 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:04:39.590 INFO:teuthology.orchestra.run.smithi102.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:reef', '-v', 'bootstrap', '--fsid', '08cee502-71d9-11ef-bceb-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.102', '--skip-admin-label'] 2024-09-13T14:04:39.614 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stdout 5.2.2 2024-09-13T14:04:39.614 INFO:teuthology.orchestra.run.smithi102.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-13T14:04:39.614 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying podman|docker is present... 2024-09-13T14:04:39.637 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stdout 5.2.2 2024-09-13T14:04:39.637 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying lvm2 is present... 2024-09-13T14:04:39.637 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying time synchronization is in place... 2024-09-13T14:04:39.645 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-13T14:04:39.645 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-13T14:04:39.651 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-13T14:04:39.651 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout inactive 2024-09-13T14:04:39.657 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout enabled 2024-09-13T14:04:39.664 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout active 2024-09-13T14:04:39.664 INFO:teuthology.orchestra.run.smithi102.stdout:Unit chronyd.service is enabled and running 2024-09-13T14:04:39.664 INFO:teuthology.orchestra.run.smithi102.stdout:Repeating the final host check... 2024-09-13T14:04:39.687 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stdout 5.2.2 2024-09-13T14:04:39.687 INFO:teuthology.orchestra.run.smithi102.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-13T14:04:39.687 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl is present 2024-09-13T14:04:39.687 INFO:teuthology.orchestra.run.smithi102.stdout:lvcreate is present 2024-09-13T14:04:39.693 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-13T14:04:39.693 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-13T14:04:39.699 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-13T14:04:39.699 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout inactive 2024-09-13T14:04:39.706 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout enabled 2024-09-13T14:04:39.712 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout active 2024-09-13T14:04:39.712 INFO:teuthology.orchestra.run.smithi102.stdout:Unit chronyd.service is enabled and running 2024-09-13T14:04:39.712 INFO:teuthology.orchestra.run.smithi102.stdout:Host looks OK 2024-09-13T14:04:39.712 INFO:teuthology.orchestra.run.smithi102.stdout:Cluster fsid: 08cee502-71d9-11ef-bceb-c7b262605968 2024-09-13T14:04:39.712 INFO:teuthology.orchestra.run.smithi102.stdout:Acquiring lock 139705854455376 on /run/cephadm/08cee502-71d9-11ef-bceb-c7b262605968.lock 2024-09-13T14:04:39.712 INFO:teuthology.orchestra.run.smithi102.stdout:Lock 139705854455376 acquired on /run/cephadm/08cee502-71d9-11ef-bceb-c7b262605968.lock 2024-09-13T14:04:39.713 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying IP 172.21.15.102 port 3300 ... 2024-09-13T14:04:39.713 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying IP 172.21.15.102 port 6789 ... 2024-09-13T14:04:39.713 INFO:teuthology.orchestra.run.smithi102.stdout:Base mon IP(s) is [172.21.15.102:3300, 172.21.15.102:6789], mon addrv is [v2:172.21.15.102:3300,v1:172.21.15.102:6789] 2024-09-13T14:04:39.716 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.102 metric 100 2024-09-13T14:04:39.716 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.102 metric 100 2024-09-13T14:04:39.719 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-13T14:04:39.719 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-09-13T14:04:39.719 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-09-13T14:04:39.721 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-13T14:04:39.721 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-13T14:04:39.721 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-13T14:04:39.721 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-09-13T14:04:39.721 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d769/64 scope link noprefixroute 2024-09-13T14:04:39.721 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-13T14:04:39.722 INFO:teuthology.orchestra.run.smithi102.stdout:Mon IP `172.21.15.102` is in CIDR network `172.21.0.0/20` 2024-09-13T14:04:39.722 INFO:teuthology.orchestra.run.smithi102.stdout:Mon IP `172.21.15.102` is in CIDR network `172.21.0.0/20` 2024-09-13T14:04:39.722 INFO:teuthology.orchestra.run.smithi102.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-09-13T14:04:39.722 INFO:teuthology.orchestra.run.smithi102.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-13T14:04:39.723 INFO:teuthology.orchestra.run.smithi102.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-09-13T14:04:40.229 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stdout c129f5a2fd4eccce3e6eec43830eff1929af98bc48f3b850cf144ce5b326e8b5 2024-09-13T14:04:40.229 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:reef... 2024-09-13T14:04:40.229 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stderr Getting image source signatures 2024-09-13T14:04:40.229 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stderr Copying blob sha256:f9a1328d66f49e17fe78ed666f97dd3c34cc07be60bd7ae6ba8f2402a2a33f7f 2024-09-13T14:04:40.229 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-13T14:04:40.229 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stderr Copying config sha256:c129f5a2fd4eccce3e6eec43830eff1929af98bc48f3b850cf144ce5b326e8b5 2024-09-13T14:04:40.229 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-13T14:04:41.150 INFO:teuthology.orchestra.run.smithi102.stdout:ceph: stdout ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable) 2024-09-13T14:04:41.150 INFO:teuthology.orchestra.run.smithi102.stdout:Ceph version: ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable) 2024-09-13T14:04:41.150 INFO:teuthology.orchestra.run.smithi102.stdout:Extracting ceph user uid/gid from container image... 2024-09-13T14:04:42.006 INFO:teuthology.orchestra.run.smithi102.stdout:stat: stdout 167 167 2024-09-13T14:04:42.006 INFO:teuthology.orchestra.run.smithi102.stdout:Creating initial keys... 2024-09-13T14:04:42.903 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph-authtool: stdout AQB6RuRmae40FBAA6tNjhL8wOutNWl6NX6s7dw== 2024-09-13T14:04:43.867 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph-authtool: stdout AQB7RuRmNe+gDRAA0orBFnvKJamu5bTXTQriuQ== 2024-09-13T14:04:44.790 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph-authtool: stdout AQB8RuRmkVZ6DBAAchPOKcGW6JxrpO0XCV1Yxw== 2024-09-13T14:04:44.790 INFO:teuthology.orchestra.run.smithi102.stdout:Creating initial monmap... 2024-09-13T14:04:45.781 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-13T14:04:45.781 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2024-09-13T14:04:45.781 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 08cee502-71d9-11ef-bceb-c7b262605968 2024-09-13T14:04:45.781 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-13T14:04:45.782 INFO:teuthology.orchestra.run.smithi102.stdout:monmaptool for a [v2:172.21.15.102:3300,v1:172.21.15.102:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-13T14:04:45.782 INFO:teuthology.orchestra.run.smithi102.stdout:setting min_mon_release = pacific 2024-09-13T14:04:45.782 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/monmaptool: set fsid to 08cee502-71d9-11ef-bceb-c7b262605968 2024-09-13T14:04:45.782 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-13T14:04:45.782 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:04:45.782 INFO:teuthology.orchestra.run.smithi102.stdout:Creating mon... 2024-09-13T14:04:46.702 INFO:teuthology.orchestra.run.smithi102.stdout:create mon.a on 2024-09-13T14:04:46.896 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-09-13T14:04:47.048 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-13T14:04:47.218 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-08cee502-71d9-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-08cee502-71d9-11ef-bceb-c7b262605968.target. 2024-09-13T14:04:47.218 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-08cee502-71d9-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-08cee502-71d9-11ef-bceb-c7b262605968.target. 2024-09-13T14:04:47.461 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-08cee502-71d9-11ef-bceb-c7b262605968@mon.a 2024-09-13T14:04:47.461 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Failed to reset failed state of unit ceph-08cee502-71d9-11ef-bceb-c7b262605968@mon.a.service: Unit ceph-08cee502-71d9-11ef-bceb-c7b262605968@mon.a.service not loaded. 2024-09-13T14:04:47.635 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-08cee502-71d9-11ef-bceb-c7b262605968.target.wants/ceph-08cee502-71d9-11ef-bceb-c7b262605968@mon.a.service → /etc/systemd/system/ceph-08cee502-71d9-11ef-bceb-c7b262605968@.service. 2024-09-13T14:04:47.894 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:47 smithi102 podman[28836]: 2024-09-13 14:04:47.786908437 +0000 UTC m=+0.015656393 image pull c129f5a2fd4eccce3e6eec43830eff1929af98bc48f3b850cf144ce5b326e8b5 quay.ceph.io/ceph-ci/ceph:reef 2024-09-13T14:04:47.894 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:47 smithi102 podman[28836]: 2024-09-13 14:04:47.893872642 +0000 UTC m=+0.122620598 container create 4aad925b46a55ec0d12fc858c28816bca65ffd720d061f27b1395a2756df7c88 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-a, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=reef-2c01458, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-09-13T14:04:48.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:48 smithi102 podman[28836]: 2024-09-13 14:04:48.137978092 +0000 UTC m=+0.366726047 container init 4aad925b46a55ec0d12fc858c28816bca65ffd720d061f27b1395a2756df7c88 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-a, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=reef-2c01458, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-09-13T14:04:48.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:48 smithi102 podman[28836]: 2024-09-13 14:04:48.14298838 +0000 UTC m=+0.371736343 container start 4aad925b46a55ec0d12fc858c28816bca65ffd720d061f27b1395a2756df7c88 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-a, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, RELEASE=reef-2c01458, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T14:04:48.227 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-13T14:04:48.227 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout disabled 2024-09-13T14:04:48.234 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-13T14:04:48.234 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout inactive 2024-09-13T14:04:48.234 INFO:teuthology.orchestra.run.smithi102.stdout:firewalld.service is not enabled 2024-09-13T14:04:48.234 INFO:teuthology.orchestra.run.smithi102.stdout:Not possible to enable service . firewalld.service is not available 2024-09-13T14:04:48.234 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for mon to start... 2024-09-13T14:04:48.234 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for mon... 2024-09-13T14:04:48.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:48 smithi102 ceph-mon[28874]: mkfs 08cee502-71d9-11ef-bceb-c7b262605968 2024-09-13T14:04:48.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:48 smithi102 ceph-mon[28874]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-13T14:04:48.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:48 smithi102 bash[28836]: 4aad925b46a55ec0d12fc858c28816bca65ffd720d061f27b1395a2756df7c88 2024-09-13T14:04:48.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:48 smithi102 systemd[1]: Started Ceph mon.a for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:04:49.270 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:49 smithi102 ceph-mon[28874]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-13T14:04:49.270 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:49 smithi102 ceph-mon[28874]: monmap e1: 1 mons at {a=[v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-13T14:04:49.270 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:49 smithi102 ceph-mon[28874]: fsmap 2024-09-13T14:04:49.270 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:49 smithi102 ceph-mon[28874]: osdmap e1: 0 total, 0 up, 0 in 2024-09-13T14:04:49.270 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:49 smithi102 ceph-mon[28874]: mgrmap e1: no daemons active 2024-09-13T14:04:49.270 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:49 smithi102 ceph-mon[28874]: from='client.? 172.21.15.102:0/372093857' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-09-13T14:04:49.272 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout cluster: 2024-09-13T14:04:49.274 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout id: 08cee502-71d9-11ef-bceb-c7b262605968 2024-09-13T14:04:49.274 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-13T14:04:49.274 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-09-13T14:04:49.274 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout services: 2024-09-13T14:04:49.274 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.514809s) 2024-09-13T14:04:49.274 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-13T14:04:49.275 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-13T14:04:49.275 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-09-13T14:04:49.275 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout data: 2024-09-13T14:04:49.275 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-13T14:04:49.275 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-13T14:04:49.275 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-13T14:04:49.275 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout pgs: 2024-09-13T14:04:49.275 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-09-13T14:04:49.275 INFO:teuthology.orchestra.run.smithi102.stdout:mon is available 2024-09-13T14:04:49.275 INFO:teuthology.orchestra.run.smithi102.stdout:Assimilating anything we can from ceph.conf... 2024-09-13T14:04:50.326 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:50 smithi102 ceph-mon[28874]: from='client.? 172.21.15.102:0/2363448041' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-13T14:04:50.326 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:50 smithi102 ceph-mon[28874]: from='client.? 172.21.15.102:0/2363448041' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-13T14:04:50.329 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-09-13T14:04:50.330 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout [global] 2024-09-13T14:04:50.330 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout fsid = 08cee502-71d9-11ef-bceb-c7b262605968 2024-09-13T14:04:50.330 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.102:3300,v1:172.21.15.102:6789] 2024-09-13T14:04:50.330 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-13T14:04:50.330 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-13T14:04:50.330 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-13T14:04:50.330 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-13T14:04:50.330 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-09-13T14:04:50.330 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-13T14:04:50.330 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-13T14:04:50.330 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-09-13T14:04:50.330 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout [osd] 2024-09-13T14:04:50.330 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-13T14:04:50.331 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-13T14:04:50.331 INFO:teuthology.orchestra.run.smithi102.stdout:Generating new minimal ceph.conf... 2024-09-13T14:04:51.394 INFO:teuthology.orchestra.run.smithi102.stdout:Restarting the monitor... 2024-09-13T14:04:51.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:51 smithi102 ceph-mon[28874]: from='client.? 172.21.15.102:0/1833551318' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:04:51.871 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:51 smithi102 systemd[1]: Stopping Ceph mon.a for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T14:04:51.871 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:51 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-a[28851]: 2024-09-13T14:04:51.644+0000 7f4e11178640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T14:04:51.871 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:51 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-a[28851]: 2024-09-13T14:04:51.644+0000 7f4e11178640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-13T14:04:52.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:51 smithi102 podman[29203]: 2024-09-13 14:04:51.870229878 +0000 UTC m=+0.308589243 container died 4aad925b46a55ec0d12fc858c28816bca65ffd720d061f27b1395a2756df7c88 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , RELEASE=reef-2c01458, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-13T14:04:52.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 podman[29203]: 2024-09-13 14:04:52.167946549 +0000 UTC m=+0.606305915 container cleanup 4aad925b46a55ec0d12fc858c28816bca65ffd720d061f27b1395a2756df7c88 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-a, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=reef-2c01458, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True) 2024-09-13T14:04:52.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 bash[29203]: ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-a 2024-09-13T14:04:52.742 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 podman[29220]: 2024-09-13 14:04:52.419312178 +0000 UTC m=+0.545242237 container remove 4aad925b46a55ec0d12fc858c28816bca65ffd720d061f27b1395a2756df7c88 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-a, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=reef-2c01458, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T14:04:52.742 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@mon.a.service: Deactivated successfully. 2024-09-13T14:04:52.742 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 systemd[1]: Stopped Ceph mon.a for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:04:52.742 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 systemd[1]: Starting Ceph mon.a for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T14:04:53.023 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 podman[29317]: 2024-09-13 14:04:52.64416032 +0000 UTC m=+0.015748574 image pull c129f5a2fd4eccce3e6eec43830eff1929af98bc48f3b850cf144ce5b326e8b5 quay.ceph.io/ceph-ci/ceph:reef 2024-09-13T14:04:53.023 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 podman[29317]: 2024-09-13 14:04:52.769990645 +0000 UTC m=+0.141578893 container create 22c604c4fd0269e547915a69d7a65a74f374b0b93816f828974e294fcd2efdd4 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-a, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=reef-2c01458, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-13T14:04:53.023 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 podman[29317]: 2024-09-13 14:04:52.938971349 +0000 UTC m=+0.310559607 container init 22c604c4fd0269e547915a69d7a65a74f374b0b93816f828974e294fcd2efdd4 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-a, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, RELEASE=reef-2c01458, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-13T14:04:53.023 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 podman[29317]: 2024-09-13 14:04:52.943245077 +0000 UTC m=+0.314833326 container start 22c604c4fd0269e547915a69d7a65a74f374b0b93816f828974e294fcd2efdd4 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-a, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=reef-2c01458, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-13T14:04:53.024 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: set uid:gid to 167:167 (ceph:ceph) 2024-09-13T14:04:53.024 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable), process ceph-mon, pid 7 2024-09-13T14:04:53.024 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: pidfile_write: ignore empty --pid-file 2024-09-13T14:04:53.024 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: load: jerasure load: lrc 2024-09-13T14:04:53.024 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: RocksDB version: 7.9.2 2024-09-13T14:04:53.024 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Git sha 0 2024-09-13T14:04:53.024 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Compile date 2024-09-12 19:46:18 2024-09-13T14:04:53.024 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: DB SUMMARY 2024-09-13T14:04:53.024 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: DB Session ID: XLY1I8B6LV79OY2LQ0KT 2024-09-13T14:04:53.024 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: CURRENT file: CURRENT 2024-09-13T14:04:53.025 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: IDENTITY file: IDENTITY 2024-09-13T14:04:53.025 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-09-13T14:04:53.025 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-13T14:04:53.025 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 84570 ; 2024-09-13T14:04:53.025 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.error_if_exists: 0 2024-09-13T14:04:53.025 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.create_if_missing: 0 2024-09-13T14:04:53.025 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.paranoid_checks: 1 2024-09-13T14:04:53.025 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-13T14:04:53.025 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-13T14:04:53.025 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-13T14:04:53.026 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.env: 0x5555aba09bc0 2024-09-13T14:04:53.026 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.fs: PosixFileSystem 2024-09-13T14:04:53.026 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.info_log: 0x5555ac7d6be0 2024-09-13T14:04:53.026 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.max_file_opening_threads: 16 2024-09-13T14:04:53.026 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.statistics: (nil) 2024-09-13T14:04:53.026 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.use_fsync: 0 2024-09-13T14:04:53.026 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.max_log_file_size: 0 2024-09-13T14:04:53.026 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-13T14:04:53.026 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-13T14:04:53.026 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.keep_log_file_num: 1000 2024-09-13T14:04:53.026 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.recycle_log_file_num: 0 2024-09-13T14:04:53.027 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.allow_fallocate: 1 2024-09-13T14:04:53.027 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.allow_mmap_reads: 0 2024-09-13T14:04:53.027 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.allow_mmap_writes: 0 2024-09-13T14:04:53.027 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.use_direct_reads: 0 2024-09-13T14:04:53.027 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-13T14:04:53.027 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.create_missing_column_families: 0 2024-09-13T14:04:53.027 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.db_log_dir: 2024-09-13T14:04:53.027 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.wal_dir: 2024-09-13T14:04:53.027 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-13T14:04:53.027 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-13T14:04:53.027 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-13T14:04:53.027 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-13T14:04:53.027 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-13T14:04:53.028 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-13T14:04:53.028 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.advise_random_on_open: 1 2024-09-13T14:04:53.028 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.db_write_buffer_size: 0 2024-09-13T14:04:53.028 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.write_buffer_manager: 0x5555ac7e4320 2024-09-13T14:04:53.028 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-13T14:04:53.028 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-13T14:04:53.028 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-13T14:04:53.028 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.rate_limiter: (nil) 2024-09-13T14:04:53.028 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-13T14:04:53.028 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.wal_recovery_mode: 2 2024-09-13T14:04:53.028 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.enable_thread_tracking: 0 2024-09-13T14:04:53.028 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.enable_pipelined_write: 0 2024-09-13T14:04:53.030 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.unordered_write: 0 2024-09-13T14:04:53.030 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-13T14:04:53.030 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-13T14:04:53.030 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-13T14:04:53.030 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-13T14:04:53.030 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.row_cache: None 2024-09-13T14:04:53.030 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.wal_filter: None 2024-09-13T14:04:53.031 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-13T14:04:53.031 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.allow_ingest_behind: 0 2024-09-13T14:04:53.031 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.two_write_queues: 0 2024-09-13T14:04:53.031 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.manual_wal_flush: 0 2024-09-13T14:04:53.031 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.wal_compression: 0 2024-09-13T14:04:53.031 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.atomic_flush: 0 2024-09-13T14:04:53.031 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-13T14:04:53.031 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-13T14:04:53.031 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-13T14:04:53.031 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.log_readahead_size: 0 2024-09-13T14:04:53.031 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-13T14:04:53.031 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.best_efforts_recovery: 0 2024-09-13T14:04:53.031 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-13T14:04:53.032 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-13T14:04:53.032 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.allow_data_in_errors: 0 2024-09-13T14:04:53.032 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.db_host_id: __hostname__ 2024-09-13T14:04:53.032 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-13T14:04:53.032 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.max_background_jobs: 2 2024-09-13T14:04:53.032 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.max_background_compactions: -1 2024-09-13T14:04:53.032 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.max_subcompactions: 1 2024-09-13T14:04:53.032 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-13T14:04:53.032 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-13T14:04:53.032 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-13T14:04:53.032 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.max_total_wal_size: 0 2024-09-13T14:04:53.032 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-13T14:04:53.033 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-13T14:04:53.033 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-13T14:04:53.033 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-13T14:04:53.033 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.max_open_files: -1 2024-09-13T14:04:53.033 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.bytes_per_sync: 0 2024-09-13T14:04:53.033 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-13T14:04:53.033 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-13T14:04:53.034 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.compaction_readahead_size: 0 2024-09-13T14:04:53.034 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.max_background_flushes: -1 2024-09-13T14:04:53.034 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Compression algorithms supported: 2024-09-13T14:04:53.034 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: kZSTD supported: 0 2024-09-13T14:04:53.034 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: kXpressCompression supported: 0 2024-09-13T14:04:53.034 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: kBZip2Compression supported: 0 2024-09-13T14:04:53.034 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-13T14:04:53.034 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: kLZ4Compression supported: 1 2024-09-13T14:04:53.034 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: kZlibCompression supported: 1 2024-09-13T14:04:53.034 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: kLZ4HCCompression supported: 1 2024-09-13T14:04:53.034 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: kSnappyCompression supported: 1 2024-09-13T14:04:53.034 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-13T14:04:53.035 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-13T14:04:53.035 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-09-13T14:04:53.035 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-13T14:04:53.035 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-13T14:04:53.035 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.merge_operator: 2024-09-13T14:04:53.035 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.compaction_filter: None 2024-09-13T14:04:53.035 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.compaction_filter_factory: None 2024-09-13T14:04:53.035 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.sst_partitioner_factory: None 2024-09-13T14:04:53.035 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-13T14:04:53.035 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-13T14:04:53.035 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5555ac7d67e0) 2024-09-13T14:04:53.035 INFO:journalctl@ceph.mon.a.smithi102.stdout: cache_index_and_filter_blocks: 1 2024-09-13T14:04:53.036 INFO:journalctl@ceph.mon.a.smithi102.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-13T14:04:53.036 INFO:journalctl@ceph.mon.a.smithi102.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-13T14:04:53.036 INFO:journalctl@ceph.mon.a.smithi102.stdout: pin_top_level_index_and_filter: 1 2024-09-13T14:04:53.036 INFO:journalctl@ceph.mon.a.smithi102.stdout: index_type: 0 2024-09-13T14:04:53.036 INFO:journalctl@ceph.mon.a.smithi102.stdout: data_block_index_type: 0 2024-09-13T14:04:53.036 INFO:journalctl@ceph.mon.a.smithi102.stdout: index_shortening: 1 2024-09-13T14:04:53.036 INFO:journalctl@ceph.mon.a.smithi102.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-13T14:04:53.036 INFO:journalctl@ceph.mon.a.smithi102.stdout: checksum: 4 2024-09-13T14:04:53.036 INFO:journalctl@ceph.mon.a.smithi102.stdout: no_block_cache: 0 2024-09-13T14:04:53.036 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_cache: 0x5555ac7cb350 2024-09-13T14:04:53.036 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_cache_name: BinnedLRUCache 2024-09-13T14:04:53.036 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_cache_options: 2024-09-13T14:04:53.036 INFO:journalctl@ceph.mon.a.smithi102.stdout: capacity : 536870912 2024-09-13T14:04:53.037 INFO:journalctl@ceph.mon.a.smithi102.stdout: num_shard_bits : 4 2024-09-13T14:04:53.037 INFO:journalctl@ceph.mon.a.smithi102.stdout: strict_capacity_limit : 0 2024-09-13T14:04:53.037 INFO:journalctl@ceph.mon.a.smithi102.stdout: high_pri_pool_ratio: 0.000 2024-09-13T14:04:53.037 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_cache_compressed: (nil) 2024-09-13T14:04:53.037 INFO:journalctl@ceph.mon.a.smithi102.stdout: persistent_cache: (nil) 2024-09-13T14:04:53.037 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_size: 4096 2024-09-13T14:04:53.037 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_size_deviation: 10 2024-09-13T14:04:53.037 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_restart_interval: 16 2024-09-13T14:04:53.037 INFO:journalctl@ceph.mon.a.smithi102.stdout: index_block_restart_interval: 1 2024-09-13T14:04:53.037 INFO:journalctl@ceph.mon.a.smithi102.stdout: metadata_block_size: 4096 2024-09-13T14:04:53.037 INFO:journalctl@ceph.mon.a.smithi102.stdout: partition_filters: 0 2024-09-13T14:04:53.037 INFO:journalctl@ceph.mon.a.smithi102.stdout: use_delta_encoding: 1 2024-09-13T14:04:53.038 INFO:journalctl@ceph.mon.a.smithi102.stdout: filter_policy: bloomfilter 2024-09-13T14:04:53.038 INFO:journalctl@ceph.mon.a.smithi102.stdout: whole_key_filtering: 1 2024-09-13T14:04:53.038 INFO:journalctl@ceph.mon.a.smithi102.stdout: verify_compression: 0 2024-09-13T14:04:53.038 INFO:journalctl@ceph.mon.a.smithi102.stdout: read_amp_bytes_per_bit: 0 2024-09-13T14:04:53.038 INFO:journalctl@ceph.mon.a.smithi102.stdout: format_version: 5 2024-09-13T14:04:53.038 INFO:journalctl@ceph.mon.a.smithi102.stdout: enable_index_compression: 1 2024-09-13T14:04:53.038 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_align: 0 2024-09-13T14:04:53.038 INFO:journalctl@ceph.mon.a.smithi102.stdout: max_auto_readahead_size: 262144 2024-09-13T14:04:53.038 INFO:journalctl@ceph.mon.a.smithi102.stdout: prepopulate_block_cache: 0 2024-09-13T14:04:53.038 INFO:journalctl@ceph.mon.a.smithi102.stdout: initial_auto_readahead_size: 8192 2024-09-13T14:04:53.038 INFO:journalctl@ceph.mon.a.smithi102.stdout: num_file_reads_for_auto_readahead: 2 2024-09-13T14:04:53.038 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.write_buffer_size: 33554432 2024-09-13T14:04:53.039 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.max_write_buffer_number: 2 2024-09-13T14:04:53.039 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.compression: NoCompression 2024-09-13T14:04:53.039 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.bottommost_compression: Disabled 2024-09-13T14:04:53.039 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.prefix_extractor: nullptr 2024-09-13T14:04:53.039 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-13T14:04:53.039 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.num_levels: 7 2024-09-13T14:04:53.039 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-13T14:04:53.039 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-13T14:04:53.039 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-13T14:04:53.039 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-13T14:04:53.039 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-13T14:04:53.039 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-13T14:04:53.040 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-13T14:04:53.040 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-13T14:04:53.040 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-13T14:04:53.040 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-13T14:04:53.040 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-13T14:04:53.040 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-13T14:04:53.040 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-13T14:04:53.040 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.compression_opts.level: 32767 2024-09-13T14:04:53.040 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.compression_opts.strategy: 0 2024-09-13T14:04:53.040 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-13T14:04:53.040 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-13T14:04:53.040 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-13T14:04:53.041 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-13T14:04:53.041 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.compression_opts.enabled: false 2024-09-13T14:04:53.041 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-13T14:04:53.041 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-13T14:04:53.041 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-13T14:04:53.041 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-13T14:04:53.041 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.target_file_size_base: 67108864 2024-09-13T14:04:53.041 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-13T14:04:53.041 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-13T14:04:53.041 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-13T14:04:53.041 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-13T14:04:53.041 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-13T14:04:53.041 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-13T14:04:53.042 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-13T14:04:53.042 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-13T14:04:53.042 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-13T14:04:53.042 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-13T14:04:53.042 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-13T14:04:53.042 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-13T14:04:53.042 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-13T14:04:53.042 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-13T14:04:53.042 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.arena_block_size: 1048576 2024-09-13T14:04:53.042 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-13T14:04:53.042 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-13T14:04:53.042 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.disable_auto_compactions: 0 2024-09-13T14:04:53.042 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-13T14:04:53.043 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-13T14:04:53.043 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-13T14:04:53.043 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-13T14:04:53.043 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-13T14:04:53.043 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-13T14:04:53.043 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-13T14:04:53.043 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-13T14:04:53.043 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-13T14:04:53.043 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-13T14:04:53.043 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-13T14:04:53.043 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.inplace_update_support: 0 2024-09-13T14:04:53.043 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-13T14:04:53.043 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-13T14:04:53.043 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-13T14:04:53.044 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-13T14:04:53.044 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.bloom_locality: 0 2024-09-13T14:04:53.044 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.max_successive_merges: 0 2024-09-13T14:04:53.044 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-13T14:04:53.044 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.paranoid_file_checks: 0 2024-09-13T14:04:53.044 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.force_consistency_checks: 1 2024-09-13T14:04:53.044 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.report_bg_io_stats: 0 2024-09-13T14:04:53.044 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.ttl: 2592000 2024-09-13T14:04:53.044 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-13T14:04:53.044 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-13T14:04:53.044 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-13T14:04:53.044 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.enable_blob_files: false 2024-09-13T14:04:53.045 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.min_blob_size: 0 2024-09-13T14:04:53.045 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.blob_file_size: 268435456 2024-09-13T14:04:53.045 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-13T14:04:53.045 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-13T14:04:53.045 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-13T14:04:53.045 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-13T14:04:53.045 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-13T14:04:53.045 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.blob_file_starting_level: 0 2024-09-13T14:04:53.045 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-13T14:04:53.045 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-09-13T14:04:53.045 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-09-13T14:04:53.045 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 02b0d92a-6f0f-4483-8913-1505901ad274 2024-09-13T14:04:53.045 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726236292969007, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-09-13T14:04:53.046 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-09-13T14:04:53.046 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726236292969829, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 81120, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 256, "table_properties": {"data_size": 79270, "index_size": 234, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10267, "raw_average_key_size": 47, "raw_value_size": 73471, "raw_average_value_size": 341, "num_data_blocks": 10, "num_entries": 215, "num_filter_entries": 215, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726236292, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "02b0d92a-6f0f-4483-8913-1505901ad274", "db_session_id": "XLY1I8B6LV79OY2LQ0KT", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-09-13T14:04:53.046 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726236292969931, "job": 1, "event": "recovery_finished"} 2024-09-13T14:04:53.046 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-09-13T14:04:53.046 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-13T14:04:53.046 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5555ac7f6e00 2024-09-13T14:04:53.046 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: DB pointer 0x5555ac8ea000 2024-09-13T14:04:53.046 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-13T14:04:53.046 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-13T14:04:53.046 INFO:journalctl@ceph.mon.a.smithi102.stdout: ** DB Stats ** 2024-09-13T14:04:53.046 INFO:journalctl@ceph.mon.a.smithi102.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T14:04:53.046 INFO:journalctl@ceph.mon.a.smithi102.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-13T14:04:53.046 INFO:journalctl@ceph.mon.a.smithi102.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-13T14:04:53.047 INFO:journalctl@ceph.mon.a.smithi102.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T14:04:53.047 INFO:journalctl@ceph.mon.a.smithi102.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-13T14:04:53.047 INFO:journalctl@ceph.mon.a.smithi102.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-13T14:04:53.047 INFO:journalctl@ceph.mon.a.smithi102.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T14:04:53.047 INFO:journalctl@ceph.mon.a.smithi102.stdout: 2024-09-13T14:04:53.047 INFO:journalctl@ceph.mon.a.smithi102.stdout: ** Compaction Stats [default] ** 2024-09-13T14:04:53.047 INFO:journalctl@ceph.mon.a.smithi102.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-13T14:04:53.047 INFO:journalctl@ceph.mon.a.smithi102.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-13T14:04:53.047 INFO:journalctl@ceph.mon.a.smithi102.stdout: L0 2/0 81.08 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 146.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-13T14:04:53.047 INFO:journalctl@ceph.mon.a.smithi102.stdout: Sum 2/0 81.08 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 146.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-13T14:04:53.047 INFO:journalctl@ceph.mon.a.smithi102.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 146.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-13T14:04:53.047 INFO:journalctl@ceph.mon.a.smithi102.stdout: 2024-09-13T14:04:53.047 INFO:journalctl@ceph.mon.a.smithi102.stdout: ** Compaction Stats [default] ** 2024-09-13T14:04:53.047 INFO:journalctl@ceph.mon.a.smithi102.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-13T14:04:53.048 INFO:journalctl@ceph.mon.a.smithi102.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-13T14:04:53.048 INFO:journalctl@ceph.mon.a.smithi102.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 146.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-13T14:04:53.048 INFO:journalctl@ceph.mon.a.smithi102.stdout: 2024-09-13T14:04:53.048 INFO:journalctl@ceph.mon.a.smithi102.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-13T14:04:53.048 INFO:journalctl@ceph.mon.a.smithi102.stdout: 2024-09-13T14:04:53.048 INFO:journalctl@ceph.mon.a.smithi102.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T14:04:53.048 INFO:journalctl@ceph.mon.a.smithi102.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-13T14:04:53.048 INFO:journalctl@ceph.mon.a.smithi102.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-13T14:04:53.048 INFO:journalctl@ceph.mon.a.smithi102.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-13T14:04:53.048 INFO:journalctl@ceph.mon.a.smithi102.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-13T14:04:53.048 INFO:journalctl@ceph.mon.a.smithi102.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-13T14:04:53.049 INFO:journalctl@ceph.mon.a.smithi102.stdout: Cumulative compaction: 0.00 GB write, 14.02 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T14:04:53.049 INFO:journalctl@ceph.mon.a.smithi102.stdout: Interval compaction: 0.00 GB write, 14.02 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T14:04:53.049 INFO:journalctl@ceph.mon.a.smithi102.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-13T14:04:53.049 INFO:journalctl@ceph.mon.a.smithi102.stdout: Block cache BinnedLRUCache@0x5555ac7cb350#7 capacity: 512.00 MB usage: 1.19 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-13T14:04:53.049 INFO:journalctl@ceph.mon.a.smithi102.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.42 KB,8.04663e-05%) Misc(1,0.00 KB,0%) 2024-09-13T14:04:53.049 INFO:journalctl@ceph.mon.a.smithi102.stdout: 2024-09-13T14:04:53.049 INFO:journalctl@ceph.mon.a.smithi102.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-13T14:04:53.049 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: starting mon.a rank 0 at public addrs [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] at bind addrs [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 08cee502-71d9-11ef-bceb-c7b262605968 2024-09-13T14:04:53.049 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: mon.a@-1(???) e1 preinit fsid 08cee502-71d9-11ef-bceb-c7b262605968 2024-09-13T14:04:53.049 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: mon.a@-1(???).mds e1 new map 2024-09-13T14:04:53.049 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: mon.a@-1(???).mds e1 print_map 2024-09-13T14:04:53.049 INFO:journalctl@ceph.mon.a.smithi102.stdout: e1 2024-09-13T14:04:53.050 INFO:journalctl@ceph.mon.a.smithi102.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-13T14:04:53.050 INFO:journalctl@ceph.mon.a.smithi102.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-13T14:04:53.050 INFO:journalctl@ceph.mon.a.smithi102.stdout: legacy client fscid: -1 2024-09-13T14:04:53.050 INFO:journalctl@ceph.mon.a.smithi102.stdout: 2024-09-13T14:04:53.050 INFO:journalctl@ceph.mon.a.smithi102.stdout: No filesystems configured 2024-09-13T14:04:53.050 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-13T14:04:53.050 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-13T14:04:53.050 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-13T14:04:53.050 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-13T14:04:53.050 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:52 smithi102 ceph-mon[29355]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-13T14:04:53.050 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:53 smithi102 bash[29317]: 22c604c4fd0269e547915a69d7a65a74f374b0b93816f828974e294fcd2efdd4 2024-09-13T14:04:53.050 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:53 smithi102 systemd[1]: Started Ceph mon.a for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:04:53.051 INFO:teuthology.orchestra.run.smithi102.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-13T14:04:53.334 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:53 smithi102 ceph-mon[29355]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-13T14:04:53.335 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:53 smithi102 ceph-mon[29355]: monmap e1: 1 mons at {a=[v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-13T14:04:53.335 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:53 smithi102 ceph-mon[29355]: fsmap 2024-09-13T14:04:53.335 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:53 smithi102 ceph-mon[29355]: osdmap e1: 0 total, 0 up, 0 in 2024-09-13T14:04:53.335 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:53 smithi102 ceph-mon[29355]: mgrmap e1: no daemons active 2024-09-13T14:04:54.074 INFO:teuthology.orchestra.run.smithi102.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-13T14:04:54.095 INFO:teuthology.orchestra.run.smithi102.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-13T14:04:54.095 INFO:teuthology.orchestra.run.smithi102.stdout:Creating mgr... 2024-09-13T14:04:54.095 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-13T14:04:54.096 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-13T14:04:54.353 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-08cee502-71d9-11ef-bceb-c7b262605968@mgr.y 2024-09-13T14:04:54.353 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Failed to reset failed state of unit ceph-08cee502-71d9-11ef-bceb-c7b262605968@mgr.y.service: Unit ceph-08cee502-71d9-11ef-bceb-c7b262605968@mgr.y.service not loaded. 2024-09-13T14:04:54.503 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-08cee502-71d9-11ef-bceb-c7b262605968.target.wants/ceph-08cee502-71d9-11ef-bceb-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-08cee502-71d9-11ef-bceb-c7b262605968@.service. 2024-09-13T14:04:54.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:54 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/487390390' entity='client.admin' 2024-09-13T14:04:54.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:04:54 smithi102 systemd[1]: Starting Ceph mgr.y for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T14:04:55.075 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:04:54 smithi102 podman[29589]: 2024-09-13 14:04:54.634834401 +0000 UTC m=+0.016019520 image pull c129f5a2fd4eccce3e6eec43830eff1929af98bc48f3b850cf144ce5b326e8b5 quay.ceph.io/ceph-ci/ceph:reef 2024-09-13T14:04:55.075 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:04:54 smithi102 podman[29589]: 2024-09-13 14:04:54.747544209 +0000 UTC m=+0.128729354 container create 4b4d7bb68ef4dce6228f7ba6f16f5a2394e0c611e5b7786871b19463ab19b689 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=reef-2c01458, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=) 2024-09-13T14:04:55.075 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:04:54 smithi102 podman[29589]: 2024-09-13 14:04:54.96640181 +0000 UTC m=+0.347586920 container init 4b4d7bb68ef4dce6228f7ba6f16f5a2394e0c611e5b7786871b19463ab19b689 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-2c01458) 2024-09-13T14:04:55.075 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:04:54 smithi102 podman[29589]: 2024-09-13 14:04:54.971292923 +0000 UTC m=+0.352478026 container start 4b4d7bb68ef4dce6228f7ba6f16f5a2394e0c611e5b7786871b19463ab19b689 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=reef-2c01458, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-09-13T14:04:55.076 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:04:55 smithi102 bash[29589]: 4b4d7bb68ef4dce6228f7ba6f16f5a2394e0c611e5b7786871b19463ab19b689 2024-09-13T14:04:55.091 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-13T14:04:55.092 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout disabled 2024-09-13T14:04:55.100 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-13T14:04:55.100 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout inactive 2024-09-13T14:04:55.100 INFO:teuthology.orchestra.run.smithi102.stdout:firewalld.service is not enabled 2024-09-13T14:04:55.100 INFO:teuthology.orchestra.run.smithi102.stdout:Not possible to enable service . firewalld.service is not available 2024-09-13T14:04:55.110 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-13T14:04:55.110 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout disabled 2024-09-13T14:04:55.119 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-13T14:04:55.119 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout inactive 2024-09-13T14:04:55.119 INFO:teuthology.orchestra.run.smithi102.stdout:firewalld.service is not enabled 2024-09-13T14:04:55.119 INFO:teuthology.orchestra.run.smithi102.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-13T14:04:55.119 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for mgr to start... 2024-09-13T14:04:55.119 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for mgr... 2024-09-13T14:04:55.352 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:04:55 smithi102 systemd[1]: Started Ceph mgr.y for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:04:55.352 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:04:55 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:04:55.091+0000 7fccde57a040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-13T14:04:55.352 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:04:55 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:04:55.157+0000 7fccde57a040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-13T14:04:55.352 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:04:55 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:04:55.307+0000 7fccde57a040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-13T14:04:55.612 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:04:55 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:04:55.379+0000 7fccde57a040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-13T14:04:55.612 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:04:55 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:04:55.528+0000 7fccde57a040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:04:55.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:55 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/1440610715' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-13T14:04:55.918 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:04:55 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:04:55.676+0000 7fccde57a040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-13T14:04:56.223 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:04:55 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:04:55.990+0000 7fccde57a040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-13T14:04:56.223 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:04:56 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:04:56.050+0000 7fccde57a040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-13T14:04:56.226 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-09-13T14:04:56.228 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-09-13T14:04:56.228 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "fsid": "08cee502-71d9-11ef-bceb-c7b262605968", 2024-09-13T14:04:56.228 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "health": { 2024-09-13T14:04:56.228 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-13T14:04:56.228 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-13T14:04:56.229 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-13T14:04:56.229 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:04:56.229 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-13T14:04:56.229 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-13T14:04:56.229 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 0 2024-09-13T14:04:56.229 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-09-13T14:04:56.229 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-13T14:04:56.229 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "a" 2024-09-13T14:04:56.229 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-09-13T14:04:56.229 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-09-13T14:04:56.229 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-13T14:04:56.229 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T14:04:56.230 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-09-13T14:04:56.230 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-13T14:04:56.230 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:04:56.230 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-13T14:04:56.230 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T14:04:56.230 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-13T14:04:56.230 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-13T14:04:56.230 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-13T14:04:56.230 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-13T14:04:56.237 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-13T14:04:56.237 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-13T14:04:56.237 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:04:56.237 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-13T14:04:56.237 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-13T14:04:56.237 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-13T14:04:56.237 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-13T14:04:56.237 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-13T14:04:56.237 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-13T14:04:56.237 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-13T14:04:56.237 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-13T14:04:56.237 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-13T14:04:56.238 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:04:56.238 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-13T14:04:56.238 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T14:04:56.238 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-13T14:04:56.238 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-13T14:04:56.238 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:04:56.238 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-13T14:04:56.238 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-13T14:04:56.238 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-13T14:04:56.238 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-13T14:04:56.238 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-13T14:04:56.238 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-13T14:04:56.238 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "restful" 2024-09-13T14:04:56.239 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-09-13T14:04:56.239 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-13T14:04:56.239 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:04:56.239 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-13T14:04:56.239 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T14:04:56.239 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "modified": "2024-09-13T14:04:48.180338+0000", 2024-09-13T14:04:56.239 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-13T14:04:56.239 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:04:56.239 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-13T14:04:56.239 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-09-13T14:04:56.239 INFO:teuthology.orchestra.run.smithi102.stdout:mgr not available, waiting (1/15)... 2024-09-13T14:04:56.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:04:56 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:04:56.235+0000 7fccde57a040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-13T14:04:57.418 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:04:57 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:04:57.095+0000 7fccde57a040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-13T14:04:57.418 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:04:57 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:04:57.160+0000 7fccde57a040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-13T14:04:58.133 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:04:57 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:04:57.877+0000 7fccde57a040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-13T14:04:58.133 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:04:57 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:04:57.992+0000 7fccde57a040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-13T14:04:58.134 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:04:58 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:04:58.132+0000 7fccde57a040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:04:58.418 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:04:58 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:04:58.193+0000 7fccde57a040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-13T14:04:58.418 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:04:58 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:04:58.352+0000 7fccde57a040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-13T14:04:58.697 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:04:58 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:04:58.418+0000 7fccde57a040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-13T14:04:58.697 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:04:58 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:04:58.477+0000 7fccde57a040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-13T14:04:58.697 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:04:58 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:04:58.631+0000 7fccde57a040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-13T14:04:58.697 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:04:58 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:04:58.695+0000 7fccde57a040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-13T14:04:58.960 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:04:58 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:04:58.761+0000 7fccde57a040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-13T14:04:58.960 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:04:58 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:04:58.842+0000 7fccde57a040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-13T14:04:58.960 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:04:58 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:04:58.900+0000 7fccde57a040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-13T14:04:58.960 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:04:58 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:04:58.958+0000 7fccde57a040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-13T14:05:00.012 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:04:59 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:04:59.730+0000 7fccde57a040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-13T14:05:00.012 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:59 smithi102 ceph-mon[29355]: Activating manager daemon y 2024-09-13T14:05:00.012 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:59 smithi102 ceph-mon[29355]: mgrmap e2: y(active, starting, since 0.00306176s) 2024-09-13T14:05:00.012 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:59 smithi102 ceph-mon[29355]: from='mgr.14100 172.21.15.102:0/3295901466' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:05:00.013 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:59 smithi102 ceph-mon[29355]: from='mgr.14100 172.21.15.102:0/3295901466' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:05:00.013 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:59 smithi102 ceph-mon[29355]: from='mgr.14100 172.21.15.102:0/3295901466' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:05:00.013 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:59 smithi102 ceph-mon[29355]: from='mgr.14100 172.21.15.102:0/3295901466' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:05:00.013 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:59 smithi102 ceph-mon[29355]: from='mgr.14100 172.21.15.102:0/3295901466' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:05:00.013 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:59 smithi102 ceph-mon[29355]: Manager daemon y is now available 2024-09-13T14:05:00.013 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:59 smithi102 ceph-mon[29355]: from='mgr.14100 172.21.15.102:0/3295901466' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:05:00.013 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:59 smithi102 ceph-mon[29355]: from='mgr.14100 172.21.15.102:0/3295901466' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:05:00.013 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:59 smithi102 ceph-mon[29355]: from='mgr.14100 172.21.15.102:0/3295901466' entity='mgr.y' 2024-09-13T14:05:00.013 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:59 smithi102 ceph-mon[29355]: from='mgr.14100 172.21.15.102:0/3295901466' entity='mgr.y' 2024-09-13T14:05:00.013 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:04:59 smithi102 ceph-mon[29355]: from='mgr.14100 172.21.15.102:0/3295901466' entity='mgr.y' 2024-09-13T14:05:02.024 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:01 smithi102 ceph-mon[29355]: mgrmap e3: y(active, since 1.00763s) 2024-09-13T14:05:02.024 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:01 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/2899559771' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-13T14:05:03.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:02 smithi102 ceph-mon[29355]: mgrmap e4: y(active, since 2s) 2024-09-13T14:05:07.644 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-09-13T14:05:07.644 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-09-13T14:05:07.644 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "fsid": "08cee502-71d9-11ef-bceb-c7b262605968", 2024-09-13T14:05:07.644 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "health": { 2024-09-13T14:05:07.645 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-13T14:05:07.645 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-13T14:05:07.645 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-13T14:05:07.645 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:05:07.645 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-13T14:05:07.645 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-13T14:05:07.645 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 0 2024-09-13T14:05:07.645 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-09-13T14:05:07.645 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-13T14:05:07.645 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "a" 2024-09-13T14:05:07.645 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-09-13T14:05:07.645 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-09-13T14:05:07.645 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-13T14:05:07.646 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T14:05:07.646 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-09-13T14:05:07.646 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-13T14:05:07.646 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:05:07.646 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-13T14:05:07.646 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T14:05:07.646 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-13T14:05:07.646 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-13T14:05:07.646 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-13T14:05:07.646 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-13T14:05:07.652 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-13T14:05:07.652 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-13T14:05:07.653 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:05:07.653 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-13T14:05:07.653 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-13T14:05:07.653 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-13T14:05:07.653 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-13T14:05:07.653 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-13T14:05:07.653 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-13T14:05:07.653 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-13T14:05:07.653 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-13T14:05:07.654 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-13T14:05:07.654 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:05:07.654 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-13T14:05:07.654 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T14:05:07.654 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-13T14:05:07.654 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-13T14:05:07.654 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:05:07.654 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-13T14:05:07.654 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-13T14:05:07.654 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-13T14:05:07.654 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-13T14:05:07.654 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-13T14:05:07.654 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-13T14:05:07.655 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "restful" 2024-09-13T14:05:07.655 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-09-13T14:05:07.655 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-13T14:05:07.655 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:05:07.655 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-13T14:05:07.655 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T14:05:07.655 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "modified": "2024-09-13T14:04:48.180338+0000", 2024-09-13T14:05:07.655 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-13T14:05:07.655 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:05:07.655 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-13T14:05:07.655 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-09-13T14:05:07.655 INFO:teuthology.orchestra.run.smithi102.stdout:mgr is available 2024-09-13T14:05:09.545 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:09 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/1931928808' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-13T14:05:09.841 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-09-13T14:05:09.841 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout [global] 2024-09-13T14:05:09.842 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout fsid = 08cee502-71d9-11ef-bceb-c7b262605968 2024-09-13T14:05:09.842 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-13T14:05:09.842 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-13T14:05:09.842 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-13T14:05:09.842 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-13T14:05:09.842 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-09-13T14:05:09.842 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-13T14:05:09.842 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-13T14:05:09.842 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-09-13T14:05:09.842 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout [osd] 2024-09-13T14:05:09.842 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-13T14:05:09.842 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-13T14:05:09.842 INFO:teuthology.orchestra.run.smithi102.stdout:Enabling cephadm module... 2024-09-13T14:05:10.890 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:10 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/3896967178' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-09-13T14:05:10.890 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:10 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: ignoring --setuser ceph since I am not root 2024-09-13T14:05:10.890 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:10 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: ignoring --setgroup ceph since I am not root 2024-09-13T14:05:10.890 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:10 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:10.684+0000 7fc33ad1b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-13T14:05:10.890 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:10 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:10.741+0000 7fc33ad1b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-13T14:05:10.890 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:10 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:10.889+0000 7fc33ad1b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-13T14:05:11.168 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:10 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:10.955+0000 7fc33ad1b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-13T14:05:11.168 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:11 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:11.098+0000 7fc33ad1b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:05:11.458 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:11 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:11.242+0000 7fc33ad1b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-13T14:05:11.808 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:11 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/3896967178' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-13T14:05:11.809 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:11 smithi102 ceph-mon[29355]: mgrmap e5: y(active, since 10s) 2024-09-13T14:05:11.809 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:11 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:11.554+0000 7fc33ad1b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-13T14:05:11.809 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:11 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:11.618+0000 7fc33ad1b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-13T14:05:11.809 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:11 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:11.806+0000 7fc33ad1b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-13T14:05:12.437 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-09-13T14:05:12.437 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-13T14:05:12.437 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-13T14:05:12.437 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-13T14:05:12.437 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-13T14:05:12.437 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-09-13T14:05:12.437 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for the mgr to restart... 2024-09-13T14:05:12.437 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for mgr epoch 5... 2024-09-13T14:05:12.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:12 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/854751587' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-13T14:05:13.168 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:12 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:12.682+0000 7fc33ad1b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-13T14:05:13.168 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:12 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:12.751+0000 7fc33ad1b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-13T14:05:13.712 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:13 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:13.458+0000 7fc33ad1b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-13T14:05:13.713 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:13 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:13.571+0000 7fc33ad1b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-13T14:05:13.982 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:13 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:13.710+0000 7fc33ad1b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:05:13.982 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:13 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:13.770+0000 7fc33ad1b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-13T14:05:13.983 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:13 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:13.914+0000 7fc33ad1b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-13T14:05:13.983 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:13 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:13.980+0000 7fc33ad1b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-13T14:05:14.260 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:14 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:14.040+0000 7fc33ad1b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-13T14:05:14.260 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:14 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:14.193+0000 7fc33ad1b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-13T14:05:14.260 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:14 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:14.257+0000 7fc33ad1b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-13T14:05:14.524 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:14 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:14.323+0000 7fc33ad1b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-13T14:05:14.525 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:14 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:14.405+0000 7fc33ad1b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-13T14:05:14.525 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:14 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:14.464+0000 7fc33ad1b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-13T14:05:14.525 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:14 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:14.522+0000 7fc33ad1b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-13T14:05:15.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:15 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:15.283+0000 7fc33ad1b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-13T14:05:15.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:15 smithi102 ceph-mon[29355]: Active manager daemon y restarted 2024-09-13T14:05:15.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:15 smithi102 ceph-mon[29355]: Activating manager daemon y 2024-09-13T14:05:15.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:15 smithi102 ceph-mon[29355]: osdmap e2: 0 total, 0 up, 0 in 2024-09-13T14:05:15.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:15 smithi102 ceph-mon[29355]: mgrmap e6: y(active, starting, since 0.00369283s) 2024-09-13T14:05:15.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:15 smithi102 ceph-mon[29355]: from='mgr.14116 172.21.15.102:0/1085755476' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:05:15.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:15 smithi102 ceph-mon[29355]: from='mgr.14116 172.21.15.102:0/1085755476' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:05:15.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:15 smithi102 ceph-mon[29355]: from='mgr.14116 172.21.15.102:0/1085755476' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:05:15.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:15 smithi102 ceph-mon[29355]: from='mgr.14116 172.21.15.102:0/1085755476' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:05:15.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:15 smithi102 ceph-mon[29355]: from='mgr.14116 172.21.15.102:0/1085755476' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:05:15.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:15 smithi102 ceph-mon[29355]: Manager daemon y is now available 2024-09-13T14:05:15.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:15 smithi102 ceph-mon[29355]: from='mgr.14116 172.21.15.102:0/1085755476' entity='mgr.y' 2024-09-13T14:05:15.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:15 smithi102 ceph-mon[29355]: from='mgr.14116 172.21.15.102:0/1085755476' entity='mgr.y' 2024-09-13T14:05:15.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:15 smithi102 ceph-mon[29355]: from='mgr.14116 172.21.15.102:0/1085755476' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:05:15.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:15 smithi102 ceph-mon[29355]: from='mgr.14116 172.21.15.102:0/1085755476' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:05:15.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:15 smithi102 ceph-mon[29355]: from='mgr.14116 172.21.15.102:0/1085755476' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:05:15.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:15 smithi102 ceph-mon[29355]: from='mgr.14116 172.21.15.102:0/1085755476' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:05:16.606 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:16 smithi102 ceph-mon[29355]: Found migration_current of "None". Setting to last migration. 2024-09-13T14:05:16.606 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:16 smithi102 ceph-mon[29355]: from='mgr.14116 172.21.15.102:0/1085755476' entity='mgr.y' 2024-09-13T14:05:16.606 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:16 smithi102 ceph-mon[29355]: from='mgr.14116 172.21.15.102:0/1085755476' entity='mgr.y' 2024-09-13T14:05:16.606 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:16 smithi102 ceph-mon[29355]: mgrmap e7: y(active, since 1.00536s) 2024-09-13T14:05:16.852 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-09-13T14:05:16.852 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-09-13T14:05:16.852 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-13T14:05:16.852 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-09-13T14:05:16.852 INFO:teuthology.orchestra.run.smithi102.stdout:mgr epoch 5 is available 2024-09-13T14:05:16.852 INFO:teuthology.orchestra.run.smithi102.stdout:Setting orchestrator backend to cephadm... 2024-09-13T14:05:17.243 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:16.970+0000 7fc2a29fb640 -1 log_channel(cephadm) log [ERR] : [13/Sep/2024:14:05:16] ENGINE Error in HTTPServer.serve 2024-09-13T14:05:17.243 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: Traceback (most recent call last): 2024-09-13T14:05:17.243 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:05:17.243 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: self._connections.run(self.expiration_interval) 2024-09-13T14:05:17.243 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:05:17.244 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: self._run(expiration_interval) 2024-09-13T14:05:17.244 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:05:17.244 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:05:17.244 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:05:17.244 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:05:17.244 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:05:17.244 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: s = self.context.wrap_socket( 2024-09-13T14:05:17.244 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:05:17.244 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: return self.sslsocket_class._create( 2024-09-13T14:05:17.244 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:05:17.244 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: self.do_handshake() 2024-09-13T14:05:17.244 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:05:17.244 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: self._sslobj.do_handshake() 2024-09-13T14:05:17.244 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:05:17.245 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:17.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:17 smithi102 ceph-mon[29355]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-13T14:05:17.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:17 smithi102 ceph-mon[29355]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-13T14:05:17.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:17 smithi102 ceph-mon[29355]: from='mgr.14116 172.21.15.102:0/1085755476' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:05:18.357 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:18 smithi102 ceph-mon[29355]: [13/Sep/2024:14:05:16] ENGINE Bus STARTING 2024-09-13T14:05:18.358 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:18 smithi102 ceph-mon[29355]: [13/Sep/2024:14:05:16] ENGINE Serving on http://172.21.15.102:8765 2024-09-13T14:05:18.358 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:18 smithi102 ceph-mon[29355]: [13/Sep/2024:14:05:16] ENGINE Serving on https://172.21.15.102:7150 2024-09-13T14:05:18.358 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:18 smithi102 ceph-mon[29355]: [13/Sep/2024:14:05:16] ENGINE Bus STARTED 2024-09-13T14:05:18.358 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:18 smithi102 ceph-mon[29355]: [13/Sep/2024:14:05:16] ENGINE Error in HTTPServer.serve 2024-09-13T14:05:18.358 INFO:journalctl@ceph.mon.a.smithi102.stdout: Traceback (most recent call last): 2024-09-13T14:05:18.358 INFO:journalctl@ceph.mon.a.smithi102.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:05:18.358 INFO:journalctl@ceph.mon.a.smithi102.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:05:18.358 INFO:journalctl@ceph.mon.a.smithi102.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:05:18.358 INFO:journalctl@ceph.mon.a.smithi102.stdout: self._run(expiration_interval) 2024-09-13T14:05:18.358 INFO:journalctl@ceph.mon.a.smithi102.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:05:18.358 INFO:journalctl@ceph.mon.a.smithi102.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:05:18.358 INFO:journalctl@ceph.mon.a.smithi102.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:05:18.359 INFO:journalctl@ceph.mon.a.smithi102.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:05:18.359 INFO:journalctl@ceph.mon.a.smithi102.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:05:18.359 INFO:journalctl@ceph.mon.a.smithi102.stdout: s = self.context.wrap_socket( 2024-09-13T14:05:18.359 INFO:journalctl@ceph.mon.a.smithi102.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:05:18.359 INFO:journalctl@ceph.mon.a.smithi102.stdout: return self.sslsocket_class._create( 2024-09-13T14:05:18.359 INFO:journalctl@ceph.mon.a.smithi102.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:05:18.359 INFO:journalctl@ceph.mon.a.smithi102.stdout: self.do_handshake() 2024-09-13T14:05:18.359 INFO:journalctl@ceph.mon.a.smithi102.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:05:18.359 INFO:journalctl@ceph.mon.a.smithi102.stdout: self._sslobj.do_handshake() 2024-09-13T14:05:18.359 INFO:journalctl@ceph.mon.a.smithi102.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:05:18.359 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:18 smithi102 ceph-mon[29355]: mgrmap e8: y(active, since 2s) 2024-09-13T14:05:18.359 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:18 smithi102 ceph-mon[29355]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:05:18.360 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:18 smithi102 ceph-mon[29355]: from='mgr.14116 172.21.15.102:0/1085755476' entity='mgr.y' 2024-09-13T14:05:18.360 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:18 smithi102 ceph-mon[29355]: from='mgr.14116 172.21.15.102:0/1085755476' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:05:19.172 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-13T14:05:19.172 INFO:teuthology.orchestra.run.smithi102.stdout:Generating ssh key... 2024-09-13T14:05:19.878 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:19 smithi102 ceph-mon[29355]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:05:20.168 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:20 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: Generating public/private rsa key pair. 2024-09-13T14:05:20.168 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:20 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: Your identification has been saved in /tmp/tmpyad8q7qo/key 2024-09-13T14:05:20.168 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:20 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: Your public key has been saved in /tmp/tmpyad8q7qo/key.pub 2024-09-13T14:05:20.169 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:20 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: The key fingerprint is: 2024-09-13T14:05:20.169 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:20 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: SHA256:eSG8l1vqmFFQK6fD9x9kTlhx6nm7zyEG2X4QkKKhIcg ceph-08cee502-71d9-11ef-bceb-c7b262605968 2024-09-13T14:05:20.169 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:20 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: The key's randomart image is: 2024-09-13T14:05:20.169 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:20 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: +---[RSA 3072]----+ 2024-09-13T14:05:20.169 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:20 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: |. . ... . .| 2024-09-13T14:05:20.169 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:20 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: | E . . o o o. + | 2024-09-13T14:05:20.169 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:20 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: | . o B = .o | 2024-09-13T14:05:20.169 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:20 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: | . o O oo+.. | 2024-09-13T14:05:20.169 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:20 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: | S *ooo* .| 2024-09-13T14:05:20.169 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:20 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: | * =o=...| 2024-09-13T14:05:20.169 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:20 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: | . o .+o+ | 2024-09-13T14:05:20.169 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:20 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: | = ..ooo| 2024-09-13T14:05:20.169 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:20 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: | o . ..+| 2024-09-13T14:05:20.170 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:20 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: +----[SHA256]-----+ 2024-09-13T14:05:21.334 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:21 smithi102 ceph-mon[29355]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:05:21.334 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:21 smithi102 ceph-mon[29355]: Generating ssh key... 2024-09-13T14:05:21.335 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:21 smithi102 ceph-mon[29355]: from='mgr.14116 172.21.15.102:0/1085755476' entity='mgr.y' 2024-09-13T14:05:21.335 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:21 smithi102 ceph-mon[29355]: from='mgr.14116 172.21.15.102:0/1085755476' entity='mgr.y' 2024-09-13T14:05:21.904 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLFi8hd9/DypdyNh6+UxI4MhbvKSi7NuFnc8KO5P+EDT9X+F3uBKoUUxfZU5aDFxQe5dpwuYL9LjzRFnyafJc8+BS4SQvqtyJb7M0pk1ZS46cpReT8mIwZKYP5uI9BCEo8a+XcVgbDeZ6su8al5dbm9G+dQBz8nB0MKQLZ+qUFWCyYNyrBmhlH9y7sKcM4R1SrWCQZnTPjfAo6hUam0drKWxjMG+8mv3iaWL5IAa2qN8eJ6VH3vPcVHvkIsQK77I0Is9LqTpBpqOMUIEXu3EQ1Y6I7z8DLgY3a+FtiYqIKe92WWK+IaNXjdtl7A9HYRMJJQosYwmgoWpUXVEspMXeEIecPr/R9UCHrKkcZfWcQA7pI/vSnmjHlR2jyyznrhE5KvwRBih+OTwIcXHhya9eDF1g4cg/57x17LbLpP6bxzgY/wdUxAfa323f2z5zAG1itJnlPCLRfkNJF9XUKiu9xI5YBYtzJP+z+ENER0a/fy9LmgMdg4bKjUHR4R/lhV4M= ceph-08cee502-71d9-11ef-bceb-c7b262605968 2024-09-13T14:05:21.904 INFO:teuthology.orchestra.run.smithi102.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-13T14:05:21.904 INFO:teuthology.orchestra.run.smithi102.stdout:Adding key to root@localhost authorized_keys... 2024-09-13T14:05:21.907 INFO:teuthology.orchestra.run.smithi102.stdout:Adding host smithi102... 2024-09-13T14:05:22.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:22 smithi102 ceph-mon[29355]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:05:23.306 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:23 smithi102 ceph-mon[29355]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi102", "addr": "172.21.15.102", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:05:24.284 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:24 smithi102 ceph-mon[29355]: Deploying cephadm binary to smithi102 2024-09-13T14:05:25.787 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout Added host 'smithi102' with addr '172.21.15.102' 2024-09-13T14:05:25.787 INFO:teuthology.orchestra.run.smithi102.stdout:Deploying unmanaged mon service... 2024-09-13T14:05:26.382 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:26 smithi102 ceph-mon[29355]: from='mgr.14116 172.21.15.102:0/1085755476' entity='mgr.y' 2024-09-13T14:05:26.382 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:26 smithi102 ceph-mon[29355]: Added host smithi102 2024-09-13T14:05:26.382 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:26 smithi102 ceph-mon[29355]: from='mgr.14116 172.21.15.102:0/1085755476' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:05:27.781 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:27 smithi102 ceph-mon[29355]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:05:27.782 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:27 smithi102 ceph-mon[29355]: Saving service mon spec with placement count:5 2024-09-13T14:05:27.782 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:27 smithi102 ceph-mon[29355]: from='mgr.14116 172.21.15.102:0/1085755476' entity='mgr.y' 2024-09-13T14:05:27.782 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:27 smithi102 ceph-mon[29355]: from='mgr.14116 172.21.15.102:0/1085755476' entity='mgr.y' 2024-09-13T14:05:27.782 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:27 smithi102 ceph-mon[29355]: from='mgr.14116 172.21.15.102:0/1085755476' entity='mgr.y' 2024-09-13T14:05:27.784 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-13T14:05:27.784 INFO:teuthology.orchestra.run.smithi102.stdout:Deploying unmanaged mgr service... 2024-09-13T14:05:29.142 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-13T14:05:29.534 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:29 smithi102 ceph-mon[29355]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:05:29.534 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:29 smithi102 ceph-mon[29355]: Saving service mgr spec with placement count:2 2024-09-13T14:05:29.534 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:29 smithi102 ceph-mon[29355]: from='mgr.14116 172.21.15.102:0/1085755476' entity='mgr.y' 2024-09-13T14:05:30.971 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:30 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/4118102239' entity='client.admin' 2024-09-13T14:05:31.498 INFO:teuthology.orchestra.run.smithi102.stdout:Enabling the dashboard module... 2024-09-13T14:05:31.916 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:31 smithi102 ceph-mon[29355]: from='mgr.14116 172.21.15.102:0/1085755476' entity='mgr.y' 2024-09-13T14:05:31.916 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:31 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/134916744' entity='client.admin' 2024-09-13T14:05:31.916 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:31 smithi102 ceph-mon[29355]: from='mgr.14116 172.21.15.102:0/1085755476' entity='mgr.y' 2024-09-13T14:05:33.123 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:32 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/1747949711' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-09-13T14:05:33.123 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:32 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: ignoring --setuser ceph since I am not root 2024-09-13T14:05:33.123 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:32 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: ignoring --setgroup ceph since I am not root 2024-09-13T14:05:33.123 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:32 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:32.977+0000 7f3dbdb91040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-13T14:05:33.123 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:33 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:33.035+0000 7f3dbdb91040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-13T14:05:33.396 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:33 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:33.184+0000 7f3dbdb91040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-13T14:05:33.397 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:33 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:33.251+0000 7f3dbdb91040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-13T14:05:33.397 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:33 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:33.395+0000 7f3dbdb91040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:05:33.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:33 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:33.539+0000 7f3dbdb91040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-13T14:05:33.991 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:33 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/1747949711' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-13T14:05:33.991 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:33 smithi102 ceph-mon[29355]: mgrmap e9: y(active, since 17s) 2024-09-13T14:05:33.991 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:33 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:33.852+0000 7f3dbdb91040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-13T14:05:33.991 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:33 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:33.914+0000 7f3dbdb91040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-13T14:05:34.411 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:34 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:34.121+0000 7f3dbdb91040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-13T14:05:34.928 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-09-13T14:05:34.928 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-13T14:05:34.928 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-13T14:05:34.928 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-13T14:05:34.928 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-13T14:05:34.928 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-09-13T14:05:34.928 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for the mgr to restart... 2024-09-13T14:05:34.929 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for mgr epoch 9... 2024-09-13T14:05:35.016 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:34 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/902461026' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-13T14:05:35.335 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:35 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:35.015+0000 7f3dbdb91040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-13T14:05:35.336 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:35 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:35.082+0000 7f3dbdb91040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-13T14:05:36.054 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:35 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:35.798+0000 7f3dbdb91040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-13T14:05:36.054 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:35 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:35.912+0000 7f3dbdb91040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-13T14:05:36.326 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:36.052+0000 7f3dbdb91040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:05:36.326 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:36.113+0000 7f3dbdb91040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-13T14:05:36.326 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:36.259+0000 7f3dbdb91040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-13T14:05:36.326 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:36.325+0000 7f3dbdb91040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-13T14:05:36.601 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:36.383+0000 7f3dbdb91040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-13T14:05:36.601 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:36.537+0000 7f3dbdb91040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-13T14:05:36.601 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:36.600+0000 7f3dbdb91040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-13T14:05:36.864 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:36.664+0000 7f3dbdb91040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-13T14:05:36.864 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:36.746+0000 7f3dbdb91040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-13T14:05:36.864 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:36.804+0000 7f3dbdb91040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-13T14:05:36.864 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:36.862+0000 7f3dbdb91040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-13T14:05:37.918 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:37 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:37.632+0000 7f3dbdb91040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-13T14:05:37.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:37 smithi102 ceph-mon[29355]: Active manager daemon y restarted 2024-09-13T14:05:37.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:37 smithi102 ceph-mon[29355]: Activating manager daemon y 2024-09-13T14:05:37.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:37 smithi102 ceph-mon[29355]: osdmap e3: 0 total, 0 up, 0 in 2024-09-13T14:05:37.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:37 smithi102 ceph-mon[29355]: mgrmap e10: y(active, starting, since 0.00336555s) 2024-09-13T14:05:37.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:37 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:05:37.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:37 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:05:37.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:37 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:05:37.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:37 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:05:37.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:37 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:05:37.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:37 smithi102 ceph-mon[29355]: Manager daemon y is now available 2024-09-13T14:05:37.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:37 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:05:37.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:37 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:05:37.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:37 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:05:38.836 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:38 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:38.603+0000 7f3d29826640 -1 log_channel(cephadm) log [ERR] : [13/Sep/2024:14:05:38] ENGINE Error in HTTPServer.serve 2024-09-13T14:05:38.836 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:38 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: Traceback (most recent call last): 2024-09-13T14:05:38.836 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:38 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:05:38.837 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:38 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: self._connections.run(self.expiration_interval) 2024-09-13T14:05:38.837 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:38 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:05:38.837 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:38 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: self._run(expiration_interval) 2024-09-13T14:05:38.837 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:38 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:05:38.837 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:38 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:05:38.837 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:38 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:05:38.837 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:38 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:05:38.837 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:38 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:05:38.837 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:38 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: s = self.context.wrap_socket( 2024-09-13T14:05:38.837 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:38 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:05:38.838 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:38 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: return self.sslsocket_class._create( 2024-09-13T14:05:38.838 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:38 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:05:38.838 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:38 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: self.do_handshake() 2024-09-13T14:05:38.838 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:38 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:05:38.838 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:38 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: self._sslobj.do_handshake() 2024-09-13T14:05:38.838 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:38 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:05:38.838 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:05:38 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:05:39.329 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-09-13T14:05:39.329 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-13T14:05:39.329 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-13T14:05:39.329 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-09-13T14:05:39.329 INFO:teuthology.orchestra.run.smithi102.stdout:mgr epoch 9 is available 2024-09-13T14:05:39.329 INFO:teuthology.orchestra.run.smithi102.stdout:Generating a dashboard self-signed certificate... 2024-09-13T14:05:39.787 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:39 smithi102 ceph-mon[29355]: [13/Sep/2024:14:05:38] ENGINE Bus STARTING 2024-09-13T14:05:39.788 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:39 smithi102 ceph-mon[29355]: [13/Sep/2024:14:05:38] ENGINE Serving on https://172.21.15.102:7150 2024-09-13T14:05:39.788 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:39 smithi102 ceph-mon[29355]: [13/Sep/2024:14:05:38] ENGINE Error in HTTPServer.serve 2024-09-13T14:05:39.788 INFO:journalctl@ceph.mon.a.smithi102.stdout: Traceback (most recent call last): 2024-09-13T14:05:39.788 INFO:journalctl@ceph.mon.a.smithi102.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:05:39.788 INFO:journalctl@ceph.mon.a.smithi102.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:05:39.788 INFO:journalctl@ceph.mon.a.smithi102.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:05:39.788 INFO:journalctl@ceph.mon.a.smithi102.stdout: self._run(expiration_interval) 2024-09-13T14:05:39.788 INFO:journalctl@ceph.mon.a.smithi102.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:05:39.788 INFO:journalctl@ceph.mon.a.smithi102.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:05:39.788 INFO:journalctl@ceph.mon.a.smithi102.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:05:39.789 INFO:journalctl@ceph.mon.a.smithi102.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:05:39.789 INFO:journalctl@ceph.mon.a.smithi102.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:05:39.789 INFO:journalctl@ceph.mon.a.smithi102.stdout: s = self.context.wrap_socket( 2024-09-13T14:05:39.789 INFO:journalctl@ceph.mon.a.smithi102.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:05:39.789 INFO:journalctl@ceph.mon.a.smithi102.stdout: return self.sslsocket_class._create( 2024-09-13T14:05:39.789 INFO:journalctl@ceph.mon.a.smithi102.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:05:39.789 INFO:journalctl@ceph.mon.a.smithi102.stdout: self.do_handshake() 2024-09-13T14:05:39.789 INFO:journalctl@ceph.mon.a.smithi102.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:05:39.789 INFO:journalctl@ceph.mon.a.smithi102.stdout: self._sslobj.do_handshake() 2024-09-13T14:05:39.789 INFO:journalctl@ceph.mon.a.smithi102.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:05:39.789 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:39 smithi102 ceph-mon[29355]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-13T14:05:39.789 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:39 smithi102 ceph-mon[29355]: mgrmap e11: y(active, since 1.00529s) 2024-09-13T14:05:39.789 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:39 smithi102 ceph-mon[29355]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-13T14:05:39.789 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:39 smithi102 ceph-mon[29355]: [13/Sep/2024:14:05:38] ENGINE Serving on http://172.21.15.102:8765 2024-09-13T14:05:39.790 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:39 smithi102 ceph-mon[29355]: [13/Sep/2024:14:05:38] ENGINE Bus STARTED 2024-09-13T14:05:40.577 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-09-13T14:05:40.577 INFO:teuthology.orchestra.run.smithi102.stdout:Creating initial admin user... 2024-09-13T14:05:40.894 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:40 smithi102 ceph-mon[29355]: mgrmap e12: y(active, since 2s) 2024-09-13T14:05:40.895 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:40 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:05:40.895 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:40 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:05:40.895 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:40 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:05:40.895 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:40 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:05:41.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:41 smithi102 ceph-mon[29355]: from='client.14160 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:05:41.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:41 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:05:42.205 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$p8vQSNYzctbMV.cpdmiW6OX6WYbTLSpRhzft1045ZgUawOKI8at8C", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726236341, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-13T14:05:42.205 INFO:teuthology.orchestra.run.smithi102.stdout:Fetching dashboard port number... 2024-09-13T14:05:42.807 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:42 smithi102 ceph-mon[29355]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:05:43.521 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 8443 2024-09-13T14:05:43.530 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-13T14:05:43.531 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout disabled 2024-09-13T14:05:43.537 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-13T14:05:43.537 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout inactive 2024-09-13T14:05:43.537 INFO:teuthology.orchestra.run.smithi102.stdout:firewalld.service is not enabled 2024-09-13T14:05:43.537 INFO:teuthology.orchestra.run.smithi102.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-13T14:05:43.539 INFO:teuthology.orchestra.run.smithi102.stdout:Ceph Dashboard is now available at: 2024-09-13T14:05:43.539 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:05:43.539 INFO:teuthology.orchestra.run.smithi102.stdout: URL: https://smithi102.front.sepia.ceph.com:8443/ 2024-09-13T14:05:43.539 INFO:teuthology.orchestra.run.smithi102.stdout: User: admin 2024-09-13T14:05:43.539 INFO:teuthology.orchestra.run.smithi102.stdout: Password: a7f77ay159 2024-09-13T14:05:43.539 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:05:43.540 INFO:teuthology.orchestra.run.smithi102.stdout:Saving cluster configuration to /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config directory 2024-09-13T14:05:43.540 INFO:teuthology.orchestra.run.smithi102.stdout:Enabling autotune for osd_memory_target 2024-09-13T14:05:43.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:43 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/1194035898' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-09-13T14:05:44.750 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:44 smithi102 ceph-mon[29355]: mgrmap e13: y(active, since 6s) 2024-09-13T14:05:45.943 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-13T14:05:45.943 INFO:teuthology.orchestra.run.smithi102.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-13T14:05:45.943 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:05:45.943 INFO:teuthology.orchestra.run.smithi102.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-13T14:05:45.943 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:05:45.943 INFO:teuthology.orchestra.run.smithi102.stdout:Or, if you are only running a single cluster on this host: 2024-09-13T14:05:45.944 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:05:45.944 INFO:teuthology.orchestra.run.smithi102.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-13T14:05:45.944 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:05:45.944 INFO:teuthology.orchestra.run.smithi102.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-13T14:05:45.944 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:05:45.944 INFO:teuthology.orchestra.run.smithi102.stdout: ceph telemetry on 2024-09-13T14:05:45.944 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:05:45.944 INFO:teuthology.orchestra.run.smithi102.stdout:For more information see: 2024-09-13T14:05:45.944 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:05:45.944 INFO:teuthology.orchestra.run.smithi102.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-09-13T14:05:45.944 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:05:45.944 INFO:teuthology.orchestra.run.smithi102.stdout:Bootstrap complete. 2024-09-13T14:05:45.974 INFO:tasks.cephadm:Fetching config... 2024-09-13T14:05:45.975 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-09-13T14:05:45.975 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-13T14:05:45.991 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-13T14:05:45.991 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-09-13T14:05:45.991 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-13T14:05:46.048 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-13T14:05:46.048 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-09-13T14:05:46.048 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/keyring of=/dev/stdout 2024-09-13T14:05:46.113 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-13T14:05:46.114 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-09-13T14:05:46.114 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-13T14:05:46.169 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-13T14:05:46.170 DEBUG:teuthology.orchestra.run.smithi102:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLFi8hd9/DypdyNh6+UxI4MhbvKSi7NuFnc8KO5P+EDT9X+F3uBKoUUxfZU5aDFxQe5dpwuYL9LjzRFnyafJc8+BS4SQvqtyJb7M0pk1ZS46cpReT8mIwZKYP5uI9BCEo8a+XcVgbDeZ6su8al5dbm9G+dQBz8nB0MKQLZ+qUFWCyYNyrBmhlH9y7sKcM4R1SrWCQZnTPjfAo6hUam0drKWxjMG+8mv3iaWL5IAa2qN8eJ6VH3vPcVHvkIsQK77I0Is9LqTpBpqOMUIEXu3EQ1Y6I7z8DLgY3a+FtiYqIKe92WWK+IaNXjdtl7A9HYRMJJQosYwmgoWpUXVEspMXeEIecPr/R9UCHrKkcZfWcQA7pI/vSnmjHlR2jyyznrhE5KvwRBih+OTwIcXHhya9eDF1g4cg/57x17LbLpP6bxzgY/wdUxAfa323f2z5zAG1itJnlPCLRfkNJF9XUKiu9xI5YBYtzJP+z+ENER0a/fy9LmgMdg4bKjUHR4R/lhV4M= ceph-08cee502-71d9-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-13T14:05:46.244 INFO:teuthology.orchestra.run.smithi102.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLFi8hd9/DypdyNh6+UxI4MhbvKSi7NuFnc8KO5P+EDT9X+F3uBKoUUxfZU5aDFxQe5dpwuYL9LjzRFnyafJc8+BS4SQvqtyJb7M0pk1ZS46cpReT8mIwZKYP5uI9BCEo8a+XcVgbDeZ6su8al5dbm9G+dQBz8nB0MKQLZ+qUFWCyYNyrBmhlH9y7sKcM4R1SrWCQZnTPjfAo6hUam0drKWxjMG+8mv3iaWL5IAa2qN8eJ6VH3vPcVHvkIsQK77I0Is9LqTpBpqOMUIEXu3EQ1Y6I7z8DLgY3a+FtiYqIKe92WWK+IaNXjdtl7A9HYRMJJQosYwmgoWpUXVEspMXeEIecPr/R9UCHrKkcZfWcQA7pI/vSnmjHlR2jyyznrhE5KvwRBih+OTwIcXHhya9eDF1g4cg/57x17LbLpP6bxzgY/wdUxAfa323f2z5zAG1itJnlPCLRfkNJF9XUKiu9xI5YBYtzJP+z+ENER0a/fy9LmgMdg4bKjUHR4R/lhV4M= ceph-08cee502-71d9-11ef-bceb-c7b262605968 2024-09-13T14:05:46.260 DEBUG:teuthology.orchestra.run.smithi195:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLFi8hd9/DypdyNh6+UxI4MhbvKSi7NuFnc8KO5P+EDT9X+F3uBKoUUxfZU5aDFxQe5dpwuYL9LjzRFnyafJc8+BS4SQvqtyJb7M0pk1ZS46cpReT8mIwZKYP5uI9BCEo8a+XcVgbDeZ6su8al5dbm9G+dQBz8nB0MKQLZ+qUFWCyYNyrBmhlH9y7sKcM4R1SrWCQZnTPjfAo6hUam0drKWxjMG+8mv3iaWL5IAa2qN8eJ6VH3vPcVHvkIsQK77I0Is9LqTpBpqOMUIEXu3EQ1Y6I7z8DLgY3a+FtiYqIKe92WWK+IaNXjdtl7A9HYRMJJQosYwmgoWpUXVEspMXeEIecPr/R9UCHrKkcZfWcQA7pI/vSnmjHlR2jyyznrhE5KvwRBih+OTwIcXHhya9eDF1g4cg/57x17LbLpP6bxzgY/wdUxAfa323f2z5zAG1itJnlPCLRfkNJF9XUKiu9xI5YBYtzJP+z+ENER0a/fy9LmgMdg4bKjUHR4R/lhV4M= ceph-08cee502-71d9-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-13T14:05:46.456 INFO:teuthology.orchestra.run.smithi195.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLFi8hd9/DypdyNh6+UxI4MhbvKSi7NuFnc8KO5P+EDT9X+F3uBKoUUxfZU5aDFxQe5dpwuYL9LjzRFnyafJc8+BS4SQvqtyJb7M0pk1ZS46cpReT8mIwZKYP5uI9BCEo8a+XcVgbDeZ6su8al5dbm9G+dQBz8nB0MKQLZ+qUFWCyYNyrBmhlH9y7sKcM4R1SrWCQZnTPjfAo6hUam0drKWxjMG+8mv3iaWL5IAa2qN8eJ6VH3vPcVHvkIsQK77I0Is9LqTpBpqOMUIEXu3EQ1Y6I7z8DLgY3a+FtiYqIKe92WWK+IaNXjdtl7A9HYRMJJQosYwmgoWpUXVEspMXeEIecPr/R9UCHrKkcZfWcQA7pI/vSnmjHlR2jyyznrhE5KvwRBih+OTwIcXHhya9eDF1g4cg/57x17LbLpP6bxzgY/wdUxAfa323f2z5zAG1itJnlPCLRfkNJF9XUKiu9xI5YBYtzJP+z+ENER0a/fy9LmgMdg4bKjUHR4R/lhV4M= ceph-08cee502-71d9-11ef-bceb-c7b262605968 2024-09-13T14:05:46.469 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-13T14:05:46.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:46 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/3783210408' entity='client.admin' 2024-09-13T14:05:46.668 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:05:49.126 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:48 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:05:49.127 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:48 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:05:49.127 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:48 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi102", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:05:49.127 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:48 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:05:49.127 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:48 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:05:49.127 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:48 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:05:49.845 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-13T14:05:49.845 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-13T14:05:50.390 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:05:50.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:50 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/3464982854' entity='client.admin' 2024-09-13T14:05:50.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:50 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:05:50.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:50 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:05:52.942 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi195 2024-09-13T14:05:52.942 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-09-13T14:05:52.942 DEBUG:teuthology.orchestra.run.smithi195:> dd of=/etc/ceph/ceph.conf 2024-09-13T14:05:52.961 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-09-13T14:05:52.961 DEBUG:teuthology.orchestra.run.smithi195:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:05:53.019 INFO:tasks.cephadm:Adding host smithi195 to orchestrator... 2024-09-13T14:05:53.019 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph orch host add smithi195 2024-09-13T14:05:53.213 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:05:53.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:53 smithi102 ceph-mon[29355]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:05:53.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:53 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:05:53.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:53 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:05:53.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:53 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:05:53.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:53 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:05:53.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:53 smithi102 ceph-mon[29355]: Updating smithi102:/etc/ceph/ceph.conf 2024-09-13T14:05:53.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:53 smithi102 ceph-mon[29355]: Updating smithi102:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:05:53.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:53 smithi102 ceph-mon[29355]: Updating smithi102:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:05:53.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:53 smithi102 ceph-mon[29355]: Updating smithi102:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:05:53.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:53 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:05:53.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:53 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:05:53.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:53 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:05:55.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:54 smithi102 ceph-mon[29355]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi195", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:05:56.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:55 smithi102 ceph-mon[29355]: Deploying cephadm binary to smithi195 2024-09-13T14:05:57.432 INFO:teuthology.orchestra.run.smithi102.stdout:Added host 'smithi195' with addr '172.21.15.195' 2024-09-13T14:05:58.058 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph orch host ls --format=json 2024-09-13T14:05:58.246 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:05:58.657 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:58 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:05:58.657 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:58 smithi102 ceph-mon[29355]: Added host smithi195 2024-09-13T14:05:58.657 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:58 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:05:58.658 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:58 smithi102 ceph-mon[29355]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:05:58.658 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:58 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:05:58.658 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:58 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:05:59.881 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:05:59.881 INFO:teuthology.orchestra.run.smithi102.stdout:[{"addr": "172.21.15.102", "hostname": "smithi102", "labels": [], "status": ""}, {"addr": "172.21.15.195", "hostname": "smithi195", "labels": [], "status": ""}] 2024-09-13T14:06:00.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:05:59 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:00.489 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-13T14:06:00.489 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd crush tunables default 2024-09-13T14:06:00.681 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:06:01.045 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:00 smithi102 ceph-mon[29355]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:01.046 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:00 smithi102 ceph-mon[29355]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:06:02.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:02 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:02.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:02 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:02.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:02 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:02.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:02 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:02.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:02 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi195", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:06:02.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:02 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:02.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:02 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:06:02.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:02 smithi102 ceph-mon[29355]: Updating smithi195:/etc/ceph/ceph.conf 2024-09-13T14:06:02.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:02 smithi102 ceph-mon[29355]: Updating smithi195:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:06:02.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:02 smithi102 ceph-mon[29355]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:02.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:02 smithi102 ceph-mon[29355]: Updating smithi195:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:06:02.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:02 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:02.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:02 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:02.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:02 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:03.385 INFO:teuthology.orchestra.run.smithi102.stderr:adjusted tunables profile to default 2024-09-13T14:06:03.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:03 smithi102 ceph-mon[29355]: Updating smithi195:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:06:03.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:03 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/2446257736' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-13T14:06:03.949 INFO:tasks.cephadm:Adding mon.a on smithi102 2024-09-13T14:06:03.949 INFO:tasks.cephadm:Adding mon.c on smithi102 2024-09-13T14:06:03.950 INFO:tasks.cephadm:Adding mon.b on smithi195 2024-09-13T14:06:03.950 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph orch apply mon '3;smithi102:172.21.15.102=a;smithi102:[v2:172.21.15.102:3301,v1:172.21.15.102:6790]=c;smithi195:172.21.15.195=b' 2024-09-13T14:06:04.149 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:06:04.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:04 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/2446257736' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-13T14:06:04.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:04 smithi102 ceph-mon[29355]: osdmap e4: 0 total, 0 up, 0 in 2024-09-13T14:06:04.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:04 smithi102 ceph-mon[29355]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:05.876 INFO:teuthology.orchestra.run.smithi195.stdout:Scheduled mon update... 2024-09-13T14:06:06.456 DEBUG:teuthology.orchestra.run.smithi102:mon.c> sudo journalctl -f -n 0 -u ceph-08cee502-71d9-11ef-bceb-c7b262605968@mon.c.service 2024-09-13T14:06:06.459 DEBUG:teuthology.orchestra.run.smithi195:mon.b> sudo journalctl -f -n 0 -u ceph-08cee502-71d9-11ef-bceb-c7b262605968@mon.b.service 2024-09-13T14:06:06.461 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-13T14:06:06.462 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-13T14:06:07.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:06 smithi102 ceph-mon[29355]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:07.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:06 smithi102 ceph-mon[29355]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi102:172.21.15.102=a;smithi102:[v2:172.21.15.102:3301,v1:172.21.15.102:6790]=c;smithi195:172.21.15.195=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:06:07.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:06 smithi102 ceph-mon[29355]: Saving service mon spec with placement smithi102:172.21.15.102=a;smithi102:[v2:172.21.15.102:3301,v1:172.21.15.102:6790]=c;smithi195:172.21.15.195=b;count:3 2024-09-13T14:06:07.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:06 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:07.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:06 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:06:07.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:06 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:07.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:06 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:06:07.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:06 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:07.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:06 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:06:07.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:06 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:07.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:06 smithi102 ceph-mon[29355]: Deploying daemon mon.b on smithi195 2024-09-13T14:06:07.191 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:06:09.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:08 smithi102 ceph-mon[29355]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:10.376 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:06:10.376 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"08cee502-71d9-11ef-bceb-c7b262605968","modified":"2024-09-13T14:04:45.172288Z","created":"2024-09-13T14:04:45.172288Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:3300","nonce":0},{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-13T14:06:10.376 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-09-13T14:06:10.494 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:10 smithi195 ceph-mon[33205]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-13T14:06:10.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:10 smithi195 bash[33154]: 608ec48476768eb0992ff1fc126d853f5e0de5e1f6d707edd7032b6c35e4afb6 2024-09-13T14:06:10.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:10 smithi195 systemd[1]: Started Ceph mon.b for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:06:12.044 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-13T14:06:12.045 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-13T14:06:12.266 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:06:15.861 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:15 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:06:15.862 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:15 smithi102 ceph-mon[29355]: mon.a calling monitor election 2024-09-13T14:06:15.862 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:15 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:06:15.862 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:15 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:06:15.862 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:15 smithi102 ceph-mon[29355]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:15.862 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:15 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:06:15.862 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:15 smithi102 ceph-mon[29355]: mon.b calling monitor election 2024-09-13T14:06:15.862 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:15 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:06:15.862 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:15 smithi102 ceph-mon[29355]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:15.862 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:15 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:06:15.862 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:15 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:06:15.863 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:15 smithi102 ceph-mon[29355]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-13T14:06:15.863 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:15 smithi102 ceph-mon[29355]: monmap e2: 2 mons at {a=[v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0],b=[v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-13T14:06:15.863 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:15 smithi102 ceph-mon[29355]: fsmap 2024-09-13T14:06:15.863 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:15 smithi102 ceph-mon[29355]: osdmap e4: 0 total, 0 up, 0 in 2024-09-13T14:06:15.863 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:15 smithi102 ceph-mon[29355]: mgrmap e13: y(active, since 37s) 2024-09-13T14:06:15.863 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:15 smithi102 ceph-mon[29355]: overall HEALTH_OK 2024-09-13T14:06:15.863 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:15 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:15.863 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:15 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:15.863 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:15 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:15.863 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:15 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:06:15.863 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:15 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:15.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:15 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:06:15.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:15 smithi195 ceph-mon[33205]: mon.a calling monitor election 2024-09-13T14:06:15.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:15 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:06:15.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:15 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:06:15.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:15 smithi195 ceph-mon[33205]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:15.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:15 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:06:15.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:15 smithi195 ceph-mon[33205]: mon.b calling monitor election 2024-09-13T14:06:15.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:15 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:06:15.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:15 smithi195 ceph-mon[33205]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:15.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:15 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:06:15.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:15 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:06:15.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:15 smithi195 ceph-mon[33205]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-13T14:06:15.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:15 smithi195 ceph-mon[33205]: monmap e2: 2 mons at {a=[v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0],b=[v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-13T14:06:15.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:15 smithi195 ceph-mon[33205]: fsmap 2024-09-13T14:06:15.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:15 smithi195 ceph-mon[33205]: osdmap e4: 0 total, 0 up, 0 in 2024-09-13T14:06:15.999 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:15 smithi195 ceph-mon[33205]: mgrmap e13: y(active, since 37s) 2024-09-13T14:06:15.999 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:15 smithi195 ceph-mon[33205]: overall HEALTH_OK 2024-09-13T14:06:15.999 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:15 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:15.999 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:15 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:15.999 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:15 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:15.999 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:15 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:06:15.999 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:15 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:16.779 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:16 smithi102 ceph-mon[29355]: Deploying daemon mon.c on smithi102 2024-09-13T14:06:16.779 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:16 smithi102 ceph-mon[29355]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:16.779 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:16 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:06:16.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:16 smithi195 ceph-mon[33205]: Deploying daemon mon.c on smithi102 2024-09-13T14:06:16.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:16 smithi195 ceph-mon[33205]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:16.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:16 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:06:17.044 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:06:17.044 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":2,"fsid":"08cee502-71d9-11ef-bceb-c7b262605968","modified":"2024-09-13T14:06:10.466336Z","created":"2024-09-13T14:04:45.172288Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:3300","nonce":0},{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:3300","nonce":0},{"type":"v1","addr":"172.21.15.195:6789","nonce":0}]},"addr":"172.21.15.195:6789/0","public_addr":"172.21.15.195:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-13T14:06:17.044 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 2 2024-09-13T14:06:17.559 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:17 smithi195 ceph-mon[33205]: from='client.? 172.21.15.195:0/250342961' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T14:06:17.819 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:17 smithi102 ceph-mon[29355]: from='client.? 172.21.15.195:0/250342961' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T14:06:18.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 systemd[1]: Starting Ceph mon.c for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T14:06:18.582 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-13T14:06:18.582 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-13T14:06:18.716 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[29355]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:18.716 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 podman[40505]: 2024-09-13 14:06:18.454405033 +0000 UTC m=+0.111293459 container create 6c1fd3099a0575ca34c1bd64c87285020d46fce4f537cf4f32c697cef9e2888b (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-c, org.label-schema.build-date=20240731, RELEASE=reef-2c01458, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-09-13T14:06:18.717 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 podman[40505]: 2024-09-13 14:06:18.35852394 +0000 UTC m=+0.015412317 image pull c129f5a2fd4eccce3e6eec43830eff1929af98bc48f3b850cf144ce5b326e8b5 quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11 2024-09-13T14:06:18.717 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 podman[40505]: 2024-09-13 14:06:18.690021431 +0000 UTC m=+0.346909811 container init 6c1fd3099a0575ca34c1bd64c87285020d46fce4f537cf4f32c697cef9e2888b (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-c, org.label-schema.vendor=CentOS, RELEASE=reef-2c01458, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=) 2024-09-13T14:06:18.717 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 podman[40505]: 2024-09-13 14:06:18.694617658 +0000 UTC m=+0.351506046 container start 6c1fd3099a0575ca34c1bd64c87285020d46fce4f537cf4f32c697cef9e2888b (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-c, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=reef-2c01458, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T14:06:18.717 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: set uid:gid to 167:167 (ceph:ceph) 2024-09-13T14:06:18.717 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable), process ceph-mon, pid 7 2024-09-13T14:06:18.718 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: pidfile_write: ignore empty --pid-file 2024-09-13T14:06:18.777 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:06:18.894 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:18 smithi195 ceph-mon[33205]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:19.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: load: jerasure load: lrc 2024-09-13T14:06:19.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: RocksDB version: 7.9.2 2024-09-13T14:06:19.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Git sha 0 2024-09-13T14:06:19.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Compile date 2024-09-12 19:46:18 2024-09-13T14:06:19.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: DB SUMMARY 2024-09-13T14:06:19.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: DB Session ID: 8M3AS549BIWMXH2RZAY5 2024-09-13T14:06:19.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: CURRENT file: CURRENT 2024-09-13T14:06:19.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: IDENTITY file: IDENTITY 2024-09-13T14:06:19.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-09-13T14:06:19.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-09-13T14:06:19.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2024-09-13T14:06:19.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.error_if_exists: 0 2024-09-13T14:06:19.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.create_if_missing: 0 2024-09-13T14:06:19.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.paranoid_checks: 1 2024-09-13T14:06:19.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-13T14:06:19.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-13T14:06:19.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-13T14:06:19.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.env: 0x55943e237bc0 2024-09-13T14:06:19.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.fs: PosixFileSystem 2024-09-13T14:06:19.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.info_log: 0x55943f75abe0 2024-09-13T14:06:19.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.max_file_opening_threads: 16 2024-09-13T14:06:19.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.statistics: (nil) 2024-09-13T14:06:19.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.use_fsync: 0 2024-09-13T14:06:19.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.max_log_file_size: 0 2024-09-13T14:06:19.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-13T14:06:19.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-13T14:06:19.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.keep_log_file_num: 1000 2024-09-13T14:06:19.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.recycle_log_file_num: 0 2024-09-13T14:06:19.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.allow_fallocate: 1 2024-09-13T14:06:19.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.allow_mmap_reads: 0 2024-09-13T14:06:19.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.allow_mmap_writes: 0 2024-09-13T14:06:19.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.use_direct_reads: 0 2024-09-13T14:06:19.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-13T14:06:19.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.create_missing_column_families: 0 2024-09-13T14:06:19.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.db_log_dir: 2024-09-13T14:06:19.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.wal_dir: 2024-09-13T14:06:19.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-13T14:06:19.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-13T14:06:19.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-13T14:06:19.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-13T14:06:19.173 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-13T14:06:19.173 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-13T14:06:19.173 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.advise_random_on_open: 1 2024-09-13T14:06:19.173 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.db_write_buffer_size: 0 2024-09-13T14:06:19.173 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.write_buffer_manager: 0x55943f768320 2024-09-13T14:06:19.173 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-13T14:06:19.174 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-13T14:06:19.174 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-13T14:06:19.174 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.rate_limiter: (nil) 2024-09-13T14:06:19.174 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-13T14:06:19.174 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.wal_recovery_mode: 2 2024-09-13T14:06:19.174 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.enable_thread_tracking: 0 2024-09-13T14:06:19.174 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.enable_pipelined_write: 0 2024-09-13T14:06:19.174 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.unordered_write: 0 2024-09-13T14:06:19.174 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-13T14:06:19.174 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-13T14:06:19.174 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-13T14:06:19.174 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-13T14:06:19.174 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.row_cache: None 2024-09-13T14:06:19.175 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.wal_filter: None 2024-09-13T14:06:19.175 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-13T14:06:19.175 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.allow_ingest_behind: 0 2024-09-13T14:06:19.175 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.two_write_queues: 0 2024-09-13T14:06:19.175 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.manual_wal_flush: 0 2024-09-13T14:06:19.175 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.wal_compression: 0 2024-09-13T14:06:19.175 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.atomic_flush: 0 2024-09-13T14:06:19.175 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-13T14:06:19.175 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-13T14:06:19.175 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-13T14:06:19.175 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.log_readahead_size: 0 2024-09-13T14:06:19.175 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-13T14:06:19.175 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.best_efforts_recovery: 0 2024-09-13T14:06:19.176 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-13T14:06:19.176 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-13T14:06:19.176 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.allow_data_in_errors: 0 2024-09-13T14:06:19.176 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.db_host_id: __hostname__ 2024-09-13T14:06:19.176 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-13T14:06:19.176 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.max_background_jobs: 2 2024-09-13T14:06:19.176 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.max_background_compactions: -1 2024-09-13T14:06:19.176 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.max_subcompactions: 1 2024-09-13T14:06:19.176 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-13T14:06:19.177 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-13T14:06:19.177 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-13T14:06:19.177 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.max_total_wal_size: 0 2024-09-13T14:06:19.177 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-13T14:06:19.177 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-13T14:06:19.177 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-13T14:06:19.177 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-13T14:06:19.177 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.max_open_files: -1 2024-09-13T14:06:19.177 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.bytes_per_sync: 0 2024-09-13T14:06:19.177 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-13T14:06:19.177 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-13T14:06:19.177 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.compaction_readahead_size: 0 2024-09-13T14:06:19.177 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.max_background_flushes: -1 2024-09-13T14:06:19.177 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Compression algorithms supported: 2024-09-13T14:06:19.178 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: kZSTD supported: 0 2024-09-13T14:06:19.178 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: kXpressCompression supported: 0 2024-09-13T14:06:19.178 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: kBZip2Compression supported: 0 2024-09-13T14:06:19.178 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-13T14:06:19.178 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: kLZ4Compression supported: 1 2024-09-13T14:06:19.178 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: kZlibCompression supported: 1 2024-09-13T14:06:19.178 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: kLZ4HCCompression supported: 1 2024-09-13T14:06:19.178 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: kSnappyCompression supported: 1 2024-09-13T14:06:19.178 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-13T14:06:19.178 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-13T14:06:19.178 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-09-13T14:06:19.178 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-13T14:06:19.178 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-13T14:06:19.178 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.merge_operator: 2024-09-13T14:06:19.179 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.compaction_filter: None 2024-09-13T14:06:19.179 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.compaction_filter_factory: None 2024-09-13T14:06:19.179 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.sst_partitioner_factory: None 2024-09-13T14:06:19.179 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-13T14:06:19.179 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-13T14:06:19.179 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55943f75a7e0) 2024-09-13T14:06:19.179 INFO:journalctl@ceph.mon.c.smithi102.stdout: cache_index_and_filter_blocks: 1 2024-09-13T14:06:19.179 INFO:journalctl@ceph.mon.c.smithi102.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-13T14:06:19.179 INFO:journalctl@ceph.mon.c.smithi102.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-13T14:06:19.179 INFO:journalctl@ceph.mon.c.smithi102.stdout: pin_top_level_index_and_filter: 1 2024-09-13T14:06:19.179 INFO:journalctl@ceph.mon.c.smithi102.stdout: index_type: 0 2024-09-13T14:06:19.179 INFO:journalctl@ceph.mon.c.smithi102.stdout: data_block_index_type: 0 2024-09-13T14:06:19.179 INFO:journalctl@ceph.mon.c.smithi102.stdout: index_shortening: 1 2024-09-13T14:06:19.179 INFO:journalctl@ceph.mon.c.smithi102.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-13T14:06:19.179 INFO:journalctl@ceph.mon.c.smithi102.stdout: checksum: 4 2024-09-13T14:06:19.180 INFO:journalctl@ceph.mon.c.smithi102.stdout: no_block_cache: 0 2024-09-13T14:06:19.180 INFO:journalctl@ceph.mon.c.smithi102.stdout: block_cache: 0x55943f74f350 2024-09-13T14:06:19.180 INFO:journalctl@ceph.mon.c.smithi102.stdout: block_cache_name: BinnedLRUCache 2024-09-13T14:06:19.180 INFO:journalctl@ceph.mon.c.smithi102.stdout: block_cache_options: 2024-09-13T14:06:19.180 INFO:journalctl@ceph.mon.c.smithi102.stdout: capacity : 536870912 2024-09-13T14:06:19.180 INFO:journalctl@ceph.mon.c.smithi102.stdout: num_shard_bits : 4 2024-09-13T14:06:19.180 INFO:journalctl@ceph.mon.c.smithi102.stdout: strict_capacity_limit : 0 2024-09-13T14:06:19.180 INFO:journalctl@ceph.mon.c.smithi102.stdout: high_pri_pool_ratio: 0.000 2024-09-13T14:06:19.180 INFO:journalctl@ceph.mon.c.smithi102.stdout: block_cache_compressed: (nil) 2024-09-13T14:06:19.180 INFO:journalctl@ceph.mon.c.smithi102.stdout: persistent_cache: (nil) 2024-09-13T14:06:19.180 INFO:journalctl@ceph.mon.c.smithi102.stdout: block_size: 4096 2024-09-13T14:06:19.180 INFO:journalctl@ceph.mon.c.smithi102.stdout: block_size_deviation: 10 2024-09-13T14:06:19.180 INFO:journalctl@ceph.mon.c.smithi102.stdout: block_restart_interval: 16 2024-09-13T14:06:19.181 INFO:journalctl@ceph.mon.c.smithi102.stdout: index_block_restart_interval: 1 2024-09-13T14:06:19.181 INFO:journalctl@ceph.mon.c.smithi102.stdout: metadata_block_size: 4096 2024-09-13T14:06:19.181 INFO:journalctl@ceph.mon.c.smithi102.stdout: partition_filters: 0 2024-09-13T14:06:19.181 INFO:journalctl@ceph.mon.c.smithi102.stdout: use_delta_encoding: 1 2024-09-13T14:06:19.181 INFO:journalctl@ceph.mon.c.smithi102.stdout: filter_policy: bloomfilter 2024-09-13T14:06:19.181 INFO:journalctl@ceph.mon.c.smithi102.stdout: whole_key_filtering: 1 2024-09-13T14:06:19.181 INFO:journalctl@ceph.mon.c.smithi102.stdout: verify_compression: 0 2024-09-13T14:06:19.181 INFO:journalctl@ceph.mon.c.smithi102.stdout: read_amp_bytes_per_bit: 0 2024-09-13T14:06:19.181 INFO:journalctl@ceph.mon.c.smithi102.stdout: format_version: 5 2024-09-13T14:06:19.181 INFO:journalctl@ceph.mon.c.smithi102.stdout: enable_index_compression: 1 2024-09-13T14:06:19.181 INFO:journalctl@ceph.mon.c.smithi102.stdout: block_align: 0 2024-09-13T14:06:19.181 INFO:journalctl@ceph.mon.c.smithi102.stdout: max_auto_readahead_size: 262144 2024-09-13T14:06:19.181 INFO:journalctl@ceph.mon.c.smithi102.stdout: prepopulate_block_cache: 0 2024-09-13T14:06:19.181 INFO:journalctl@ceph.mon.c.smithi102.stdout: initial_auto_readahead_size: 8192 2024-09-13T14:06:19.182 INFO:journalctl@ceph.mon.c.smithi102.stdout: num_file_reads_for_auto_readahead: 2 2024-09-13T14:06:19.182 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.write_buffer_size: 33554432 2024-09-13T14:06:19.182 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.max_write_buffer_number: 2 2024-09-13T14:06:19.182 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.compression: NoCompression 2024-09-13T14:06:19.182 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.bottommost_compression: Disabled 2024-09-13T14:06:19.182 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.prefix_extractor: nullptr 2024-09-13T14:06:19.182 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-13T14:06:19.182 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.num_levels: 7 2024-09-13T14:06:19.182 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-13T14:06:19.182 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-13T14:06:19.182 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-13T14:06:19.182 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-13T14:06:19.182 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-13T14:06:19.182 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-13T14:06:19.182 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-13T14:06:19.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-13T14:06:19.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-13T14:06:19.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-13T14:06:19.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-13T14:06:19.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-13T14:06:19.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-13T14:06:19.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.compression_opts.level: 32767 2024-09-13T14:06:19.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.compression_opts.strategy: 0 2024-09-13T14:06:19.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-13T14:06:19.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-13T14:06:19.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-13T14:06:19.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-13T14:06:19.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.compression_opts.enabled: false 2024-09-13T14:06:19.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-13T14:06:19.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-13T14:06:19.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-13T14:06:19.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-13T14:06:19.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.target_file_size_base: 67108864 2024-09-13T14:06:19.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-13T14:06:19.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-13T14:06:19.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-13T14:06:19.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-13T14:06:19.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-13T14:06:19.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-13T14:06:19.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-13T14:06:19.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-13T14:06:19.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-13T14:06:19.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-13T14:06:19.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-13T14:06:19.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-13T14:06:19.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-13T14:06:19.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-13T14:06:19.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.arena_block_size: 1048576 2024-09-13T14:06:19.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-13T14:06:19.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-13T14:06:19.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.disable_auto_compactions: 0 2024-09-13T14:06:19.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-13T14:06:19.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-13T14:06:19.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-13T14:06:19.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-13T14:06:19.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-13T14:06:19.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-13T14:06:19.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-13T14:06:19.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-13T14:06:19.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-13T14:06:19.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-13T14:06:19.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-13T14:06:19.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.inplace_update_support: 0 2024-09-13T14:06:19.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-13T14:06:19.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-13T14:06:19.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-13T14:06:19.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-13T14:06:19.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.bloom_locality: 0 2024-09-13T14:06:19.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.max_successive_merges: 0 2024-09-13T14:06:19.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-13T14:06:19.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.paranoid_file_checks: 0 2024-09-13T14:06:19.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.force_consistency_checks: 1 2024-09-13T14:06:19.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.report_bg_io_stats: 0 2024-09-13T14:06:19.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.ttl: 2592000 2024-09-13T14:06:19.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-13T14:06:19.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-13T14:06:19.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-13T14:06:19.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.enable_blob_files: false 2024-09-13T14:06:19.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.min_blob_size: 0 2024-09-13T14:06:19.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.blob_file_size: 268435456 2024-09-13T14:06:19.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-13T14:06:19.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-13T14:06:19.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-13T14:06:19.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-13T14:06:19.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-13T14:06:19.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.blob_file_starting_level: 0 2024-09-13T14:06:19.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-13T14:06:19.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-13T14:06:19.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-13T14:06:19.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 83a210d1-0432-49a0-a7ce-d40c676b3e42 2024-09-13T14:06:19.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726236378720983, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-13T14:06:19.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-09-13T14:06:19.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726236378721357, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1773, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726236378, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "83a210d1-0432-49a0-a7ce-d40c676b3e42", "db_session_id": "8M3AS549BIWMXH2RZAY5", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-09-13T14:06:19.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726236378721452, "job": 1, "event": "recovery_finished"} 2024-09-13T14:06:19.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-09-13T14:06:19.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-13T14:06:19.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55943f77ae00 2024-09-13T14:06:19.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: DB pointer 0x55943f86e000 2024-09-13T14:06:19.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-13T14:06:19.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-13T14:06:19.189 INFO:journalctl@ceph.mon.c.smithi102.stdout: ** DB Stats ** 2024-09-13T14:06:19.189 INFO:journalctl@ceph.mon.c.smithi102.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T14:06:19.189 INFO:journalctl@ceph.mon.c.smithi102.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-13T14:06:19.189 INFO:journalctl@ceph.mon.c.smithi102.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-13T14:06:19.189 INFO:journalctl@ceph.mon.c.smithi102.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T14:06:19.189 INFO:journalctl@ceph.mon.c.smithi102.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-13T14:06:19.189 INFO:journalctl@ceph.mon.c.smithi102.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-13T14:06:19.189 INFO:journalctl@ceph.mon.c.smithi102.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T14:06:19.189 INFO:journalctl@ceph.mon.c.smithi102.stdout: 2024-09-13T14:06:19.190 INFO:journalctl@ceph.mon.c.smithi102.stdout: ** Compaction Stats [default] ** 2024-09-13T14:06:19.190 INFO:journalctl@ceph.mon.c.smithi102.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-13T14:06:19.190 INFO:journalctl@ceph.mon.c.smithi102.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-13T14:06:19.190 INFO:journalctl@ceph.mon.c.smithi102.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-13T14:06:19.190 INFO:journalctl@ceph.mon.c.smithi102.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-13T14:06:19.190 INFO:journalctl@ceph.mon.c.smithi102.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-13T14:06:19.190 INFO:journalctl@ceph.mon.c.smithi102.stdout: 2024-09-13T14:06:19.190 INFO:journalctl@ceph.mon.c.smithi102.stdout: ** Compaction Stats [default] ** 2024-09-13T14:06:19.190 INFO:journalctl@ceph.mon.c.smithi102.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-13T14:06:19.190 INFO:journalctl@ceph.mon.c.smithi102.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-13T14:06:19.190 INFO:journalctl@ceph.mon.c.smithi102.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-13T14:06:19.190 INFO:journalctl@ceph.mon.c.smithi102.stdout: 2024-09-13T14:06:19.191 INFO:journalctl@ceph.mon.c.smithi102.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-13T14:06:19.191 INFO:journalctl@ceph.mon.c.smithi102.stdout: 2024-09-13T14:06:19.191 INFO:journalctl@ceph.mon.c.smithi102.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T14:06:19.191 INFO:journalctl@ceph.mon.c.smithi102.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-13T14:06:19.191 INFO:journalctl@ceph.mon.c.smithi102.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-13T14:06:19.191 INFO:journalctl@ceph.mon.c.smithi102.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-13T14:06:19.191 INFO:journalctl@ceph.mon.c.smithi102.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-13T14:06:19.191 INFO:journalctl@ceph.mon.c.smithi102.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-13T14:06:19.191 INFO:journalctl@ceph.mon.c.smithi102.stdout: Cumulative compaction: 0.00 GB write, 0.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T14:06:19.191 INFO:journalctl@ceph.mon.c.smithi102.stdout: Interval compaction: 0.00 GB write, 0.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T14:06:19.191 INFO:journalctl@ceph.mon.c.smithi102.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-13T14:06:19.191 INFO:journalctl@ceph.mon.c.smithi102.stdout: Block cache BinnedLRUCache@0x55943f74f350#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-13T14:06:19.191 INFO:journalctl@ceph.mon.c.smithi102.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-09-13T14:06:19.191 INFO:journalctl@ceph.mon.c.smithi102.stdout: 2024-09-13T14:06:19.192 INFO:journalctl@ceph.mon.c.smithi102.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-13T14:06:19.192 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-09-13T14:06:19.192 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: using public_addrv [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] 2024-09-13T14:06:19.192 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: starting mon.c rank -1 at public addrs [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] at bind addrs [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 08cee502-71d9-11ef-bceb-c7b262605968 2024-09-13T14:06:19.192 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: mon.c@-1(???) e0 preinit fsid 08cee502-71d9-11ef-bceb-c7b262605968 2024-09-13T14:06:19.192 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: mon.c@-1(synchronizing).mds e1 new map 2024-09-13T14:06:19.192 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: mon.c@-1(synchronizing).mds e1 print_map 2024-09-13T14:06:19.192 INFO:journalctl@ceph.mon.c.smithi102.stdout: e1 2024-09-13T14:06:19.192 INFO:journalctl@ceph.mon.c.smithi102.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-13T14:06:19.192 INFO:journalctl@ceph.mon.c.smithi102.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-13T14:06:19.192 INFO:journalctl@ceph.mon.c.smithi102.stdout: legacy client fscid: -1 2024-09-13T14:06:19.192 INFO:journalctl@ceph.mon.c.smithi102.stdout: 2024-09-13T14:06:19.192 INFO:journalctl@ceph.mon.c.smithi102.stdout: No filesystems configured 2024-09-13T14:06:19.192 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-13T14:06:19.193 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-13T14:06:19.193 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-13T14:06:19.193 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-13T14:06:19.193 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-13T14:06:19.193 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-13T14:06:19.193 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-13T14:06:19.193 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-13T14:06:19.193 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-13T14:06:19.193 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-13T14:06:19.193 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:19.193 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi102:172.21.15.102=a;smithi102:[v2:172.21.15.102:3301,v1:172.21.15.102:6790]=c;smithi195:172.21.15.195=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:06:19.193 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: Saving service mon spec with placement smithi102:172.21.15.102=a;smithi102:[v2:172.21.15.102:3301,v1:172.21.15.102:6790]=c;smithi195:172.21.15.195=b;count:3 2024-09-13T14:06:19.193 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:19.193 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:06:19.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:19.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:06:19.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:19.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:06:19.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:19.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: Deploying daemon mon.b on smithi195 2024-09-13T14:06:19.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:19.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:06:19.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: mon.a calling monitor election 2024-09-13T14:06:19.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:06:19.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:06:19.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:19.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:06:19.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: mon.b calling monitor election 2024-09-13T14:06:19.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:06:19.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:19.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:06:19.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:06:19.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-13T14:06:19.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: monmap e2: 2 mons at {a=[v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0],b=[v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-13T14:06:19.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: fsmap 2024-09-13T14:06:19.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: osdmap e4: 0 total, 0 up, 0 in 2024-09-13T14:06:19.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: mgrmap e13: y(active, since 37s) 2024-09-13T14:06:19.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: overall HEALTH_OK 2024-09-13T14:06:19.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:19.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:19.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:19.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:06:19.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:19.196 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: Deploying daemon mon.c on smithi102 2024-09-13T14:06:19.196 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:19.196 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:06:19.196 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: from='client.? 172.21.15.195:0/250342961' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T14:06:19.196 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:19.196 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 ceph-mon[40543]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-13T14:06:19.196 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 bash[40505]: 6c1fd3099a0575ca34c1bd64c87285020d46fce4f537cf4f32c697cef9e2888b 2024-09-13T14:06:19.196 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:18 smithi102 systemd[1]: Started Ceph mon.c for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:06:24.162 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:06:24.163 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":3,"fsid":"08cee502-71d9-11ef-bceb-c7b262605968","modified":"2024-09-13T14:06:18.744278Z","created":"2024-09-13T14:04:45.172288Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:3300","nonce":0},{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:3300","nonce":0},{"type":"v1","addr":"172.21.15.195:6789","nonce":0}]},"addr":"172.21.15.195:6789/0","public_addr":"172.21.15.195:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:3301","nonce":0},{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-09-13T14:06:24.163 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 3 2024-09-13T14:06:24.163 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:23 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:06:24.163 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:23 smithi195 ceph-mon[33205]: mon.a calling monitor election 2024-09-13T14:06:24.163 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:23 smithi195 ceph-mon[33205]: mon.b calling monitor election 2024-09-13T14:06:24.164 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:23 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:06:24.164 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:23 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:06:24.164 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:23 smithi195 ceph-mon[33205]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:24.164 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:23 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:06:24.164 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:23 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:06:24.164 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:23 smithi195 ceph-mon[33205]: mon.c calling monitor election 2024-09-13T14:06:24.164 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:23 smithi195 ceph-mon[33205]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:24.164 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:23 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:06:24.164 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:23 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:06:24.164 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:23 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:06:24.164 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:23 smithi195 ceph-mon[33205]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T14:06:24.164 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:23 smithi195 ceph-mon[33205]: monmap e3: 3 mons at {a=[v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0],b=[v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0],c=[v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-13T14:06:24.164 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:23 smithi195 ceph-mon[33205]: fsmap 2024-09-13T14:06:24.165 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:23 smithi195 ceph-mon[33205]: osdmap e4: 0 total, 0 up, 0 in 2024-09-13T14:06:24.165 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:23 smithi195 ceph-mon[33205]: mgrmap e13: y(active, since 46s) 2024-09-13T14:06:24.165 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:23 smithi195 ceph-mon[33205]: overall HEALTH_OK 2024-09-13T14:06:24.165 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:23 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:24.165 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:23 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:24.165 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:23 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:24.165 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:23 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:24.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:23 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:06:24.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:23 smithi102 ceph-mon[29355]: mon.a calling monitor election 2024-09-13T14:06:24.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:23 smithi102 ceph-mon[29355]: mon.b calling monitor election 2024-09-13T14:06:24.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:23 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:06:24.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:23 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:06:24.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:23 smithi102 ceph-mon[29355]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:24.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:23 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:06:24.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:23 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:06:24.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:23 smithi102 ceph-mon[29355]: mon.c calling monitor election 2024-09-13T14:06:24.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:23 smithi102 ceph-mon[29355]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:24.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:23 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:06:24.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:23 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:06:24.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:23 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:06:24.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:23 smithi102 ceph-mon[29355]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T14:06:24.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:23 smithi102 ceph-mon[29355]: monmap e3: 3 mons at {a=[v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0],b=[v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0],c=[v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-13T14:06:24.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:23 smithi102 ceph-mon[29355]: fsmap 2024-09-13T14:06:24.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:23 smithi102 ceph-mon[29355]: osdmap e4: 0 total, 0 up, 0 in 2024-09-13T14:06:24.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:23 smithi102 ceph-mon[29355]: mgrmap e13: y(active, since 46s) 2024-09-13T14:06:24.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:23 smithi102 ceph-mon[29355]: overall HEALTH_OK 2024-09-13T14:06:24.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:23 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:24.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:23 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:24.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:23 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:24.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:23 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:24.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:23 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:06:24.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:23 smithi102 ceph-mon[40543]: mon.a calling monitor election 2024-09-13T14:06:24.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:23 smithi102 ceph-mon[40543]: mon.b calling monitor election 2024-09-13T14:06:24.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:23 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:06:24.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:23 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:06:24.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:23 smithi102 ceph-mon[40543]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:24.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:23 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:06:24.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:23 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:06:24.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:23 smithi102 ceph-mon[40543]: mon.c calling monitor election 2024-09-13T14:06:24.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:23 smithi102 ceph-mon[40543]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:24.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:23 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:06:24.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:23 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:06:24.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:23 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:06:24.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:23 smithi102 ceph-mon[40543]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T14:06:24.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:23 smithi102 ceph-mon[40543]: monmap e3: 3 mons at {a=[v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0],b=[v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0],c=[v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-13T14:06:24.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:23 smithi102 ceph-mon[40543]: fsmap 2024-09-13T14:06:24.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:23 smithi102 ceph-mon[40543]: osdmap e4: 0 total, 0 up, 0 in 2024-09-13T14:06:24.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:23 smithi102 ceph-mon[40543]: mgrmap e13: y(active, since 46s) 2024-09-13T14:06:24.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:23 smithi102 ceph-mon[40543]: overall HEALTH_OK 2024-09-13T14:06:24.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:23 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:24.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:23 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:24.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:23 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:24.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:23 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:24.701 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-13T14:06:24.701 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph config generate-minimal-conf 2024-09-13T14:06:24.909 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:06:25.030 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:24 smithi102 ceph-mon[29355]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:25.030 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:24 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:06:25.030 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:24 smithi102 ceph-mon[29355]: from='client.? 172.21.15.195:0/1336253723' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T14:06:25.030 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:24 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:06:25.030 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:24 smithi102 ceph-mon[40543]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:25.030 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:24 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:06:25.030 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:24 smithi102 ceph-mon[40543]: from='client.? 172.21.15.195:0/1336253723' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T14:06:25.030 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:24 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:06:25.090 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:24 smithi195 ceph-mon[33205]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:25.090 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:24 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:06:25.090 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:24 smithi195 ceph-mon[33205]: from='client.? 172.21.15.195:0/1336253723' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T14:06:25.090 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:24 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:06:26.028 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:25 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:26.028 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:25 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:26.028 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:25 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:26.028 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:25 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:26.028 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:25 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:26.028 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:25 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:06:26.028 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:25 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:26.028 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:25 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:26.029 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:25 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:26.029 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:25 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:26.029 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:25 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:26.029 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:25 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:06:26.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:25 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:26.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:25 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:26.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:25 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:26.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:25 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:26.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:25 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:26.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:25 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:06:26.634 INFO:teuthology.orchestra.run.smithi102.stdout:# minimal ceph.conf for 08cee502-71d9-11ef-bceb-c7b262605968 2024-09-13T14:06:26.634 INFO:teuthology.orchestra.run.smithi102.stdout:[global] 2024-09-13T14:06:26.634 INFO:teuthology.orchestra.run.smithi102.stdout: fsid = 08cee502-71d9-11ef-bceb-c7b262605968 2024-09-13T14:06:26.635 INFO:teuthology.orchestra.run.smithi102.stdout: mon_host = [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] 2024-09-13T14:06:27.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:26 smithi102 ceph-mon[29355]: Updating smithi102:/etc/ceph/ceph.conf 2024-09-13T14:06:27.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:26 smithi102 ceph-mon[29355]: Updating smithi195:/etc/ceph/ceph.conf 2024-09-13T14:06:27.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:26 smithi102 ceph-mon[29355]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:27.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:26 smithi102 ceph-mon[29355]: Updating smithi102:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:06:27.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:26 smithi102 ceph-mon[29355]: Updating smithi195:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:06:27.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:26 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:27.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:26 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:27.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:26 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:27.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:26 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:27.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:26 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:27.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:26 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:27.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:26 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:27.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:26 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:27.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:26 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:27.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:26 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:06:27.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:26 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:06:27.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:26 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:27.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:26 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/3246048647' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:27.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:26 smithi102 ceph-mon[40543]: Updating smithi102:/etc/ceph/ceph.conf 2024-09-13T14:06:27.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:26 smithi102 ceph-mon[40543]: Updating smithi195:/etc/ceph/ceph.conf 2024-09-13T14:06:27.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:26 smithi102 ceph-mon[40543]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:27.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:26 smithi102 ceph-mon[40543]: Updating smithi102:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:06:27.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:26 smithi102 ceph-mon[40543]: Updating smithi195:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:06:27.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:26 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:27.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:26 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:27.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:26 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:27.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:26 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:27.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:26 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:27.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:26 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:27.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:26 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:27.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:26 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:27.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:26 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:27.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:26 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:06:27.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:26 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:06:27.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:26 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:27.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:26 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/3246048647' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:27.231 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-13T14:06:27.231 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-09-13T14:06:27.231 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/ceph/ceph.conf 2024-09-13T14:06:27.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:26 smithi195 ceph-mon[33205]: Updating smithi102:/etc/ceph/ceph.conf 2024-09-13T14:06:27.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:26 smithi195 ceph-mon[33205]: Updating smithi195:/etc/ceph/ceph.conf 2024-09-13T14:06:27.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:26 smithi195 ceph-mon[33205]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:27.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:26 smithi195 ceph-mon[33205]: Updating smithi102:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:06:27.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:26 smithi195 ceph-mon[33205]: Updating smithi195:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:06:27.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:26 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:27.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:26 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:27.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:26 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:27.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:26 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:27.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:26 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:27.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:26 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:27.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:26 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:27.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:26 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:27.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:26 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:27.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:26 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:06:27.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:26 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:06:27.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:26 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:27.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:26 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/3246048647' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:27.257 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-09-13T14:06:27.257 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:06:27.322 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-09-13T14:06:27.323 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/ceph/ceph.conf 2024-09-13T14:06:27.356 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-09-13T14:06:27.356 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:06:27.427 INFO:tasks.cephadm:Adding mgr.y on smithi102 2024-09-13T14:06:27.427 INFO:tasks.cephadm:Adding mgr.x on smithi195 2024-09-13T14:06:27.427 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph orch apply mgr '2;smithi102=y;smithi195=x' 2024-09-13T14:06:27.663 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:06:27.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:27 smithi195 ceph-mon[33205]: Reconfiguring mon.a (unknown last config time)... 2024-09-13T14:06:27.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:27 smithi195 ceph-mon[33205]: Reconfiguring daemon mon.a on smithi102 2024-09-13T14:06:27.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:27 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:28.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:27 smithi102 ceph-mon[29355]: Reconfiguring mon.a (unknown last config time)... 2024-09-13T14:06:28.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:27 smithi102 ceph-mon[29355]: Reconfiguring daemon mon.a on smithi102 2024-09-13T14:06:28.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:27 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:28.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:27 smithi102 ceph-mon[40543]: Reconfiguring mon.a (unknown last config time)... 2024-09-13T14:06:28.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:27 smithi102 ceph-mon[40543]: Reconfiguring daemon mon.a on smithi102 2024-09-13T14:06:28.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:27 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:29.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:28 smithi102 ceph-mon[29355]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:29.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:28 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:29.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:28 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:29.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:28 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:06:29.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:28 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:06:29.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:28 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:29.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:28 smithi102 ceph-mon[40543]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:29.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:28 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:29.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:28 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:29.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:28 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:06:29.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:28 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:06:29.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:28 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:29.245 INFO:teuthology.orchestra.run.smithi195.stdout:Scheduled mgr update... 2024-09-13T14:06:29.246 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:28 smithi195 ceph-mon[33205]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:29.246 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:28 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:29.246 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:28 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:29.246 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:28 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:06:29.246 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:28 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:06:29.246 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:28 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:29.846 DEBUG:teuthology.orchestra.run.smithi195:mgr.x> sudo journalctl -f -n 0 -u ceph-08cee502-71d9-11ef-bceb-c7b262605968@mgr.x.service 2024-09-13T14:06:29.848 DEBUG:tasks.cephadm:set 0 configs 2024-09-13T14:06:29.848 INFO:tasks.cephadm:Deploying OSDs... 2024-09-13T14:06:29.848 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-09-13T14:06:29.848 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/scratch_devs of=/dev/stdout 2024-09-13T14:06:29.865 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-13T14:06:29.865 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_1 2024-09-13T14:06:29.920 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-13T14:06:29.921 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T14:06:29.921 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 982 Links: 1 2024-09-13T14:06:29.921 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T14:06:29.921 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T14:06:29.921 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-09-13 14:06:18.189572198 +0000 2024-09-13T14:06:29.921 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-09-13 14:01:33.816665783 +0000 2024-09-13T14:06:29.921 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-09-13 14:01:33.816665783 +0000 2024-09-13T14:06:29.921 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-09-13 14:01:33.816665783 +0000 2024-09-13T14:06:29.921 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-13T14:06:29.985 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-09-13T14:06:29.985 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-09-13T14:06:29.985 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000213591 s, 2.4 MB/s 2024-09-13T14:06:29.987 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-13T14:06:30.044 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_2 2024-09-13T14:06:30.095 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:29 smithi195 ceph-mon[33205]: Reconfiguring mon.c (monmap changed)... 2024-09-13T14:06:30.095 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:29 smithi195 ceph-mon[33205]: Reconfiguring daemon mon.c on smithi102 2024-09-13T14:06:30.095 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:29 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:30.095 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:29 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:30.095 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:29 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:30.095 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:29 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:06:30.096 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:29 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:06:30.096 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:29 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:30.099 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-13T14:06:30.100 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T14:06:30.100 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 974 Links: 1 2024-09-13T14:06:30.100 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T14:06:30.100 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T14:06:30.100 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-09-13 14:06:18.190572176 +0000 2024-09-13T14:06:30.100 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-09-13 14:01:33.813665935 +0000 2024-09-13T14:06:30.100 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-09-13 14:01:33.813665935 +0000 2024-09-13T14:06:30.100 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-09-13 14:01:33.813665935 +0000 2024-09-13T14:06:30.100 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-13T14:06:30.161 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:29 smithi102 ceph-mon[29355]: Reconfiguring mon.c (monmap changed)... 2024-09-13T14:06:30.161 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:29 smithi102 ceph-mon[29355]: Reconfiguring daemon mon.c on smithi102 2024-09-13T14:06:30.161 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:29 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:30.161 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:29 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:30.161 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:29 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:30.161 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:29 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:06:30.161 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:29 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:06:30.162 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:29 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:30.162 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:29 smithi102 ceph-mon[40543]: Reconfiguring mon.c (monmap changed)... 2024-09-13T14:06:30.162 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:29 smithi102 ceph-mon[40543]: Reconfiguring daemon mon.c on smithi102 2024-09-13T14:06:30.162 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:29 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:30.162 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:29 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:30.162 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:29 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:30.162 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:29 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:06:30.162 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:29 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:06:30.162 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:29 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:30.163 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-09-13T14:06:30.163 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-09-13T14:06:30.163 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000177916 s, 2.9 MB/s 2024-09-13T14:06:30.164 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-13T14:06:30.221 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_3 2024-09-13T14:06:30.277 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-13T14:06:30.277 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T14:06:30.277 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 976 Links: 1 2024-09-13T14:06:30.277 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T14:06:30.277 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T14:06:30.277 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-09-13 14:06:18.190572176 +0000 2024-09-13T14:06:30.278 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-09-13 14:01:33.814665884 +0000 2024-09-13T14:06:30.278 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-09-13 14:01:33.814665884 +0000 2024-09-13T14:06:30.278 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-09-13 14:01:33.814665884 +0000 2024-09-13T14:06:30.278 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-13T14:06:30.341 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-09-13T14:06:30.341 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-09-13T14:06:30.341 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000178732 s, 2.9 MB/s 2024-09-13T14:06:30.342 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-13T14:06:30.398 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_4 2024-09-13T14:06:30.453 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-13T14:06:30.453 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T14:06:30.453 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 965 Links: 1 2024-09-13T14:06:30.453 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T14:06:30.453 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T14:06:30.453 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-09-13 14:06:18.190572176 +0000 2024-09-13T14:06:30.454 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-09-13 14:01:33.812665985 +0000 2024-09-13T14:06:30.454 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-09-13 14:01:33.812665985 +0000 2024-09-13T14:06:30.454 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-09-13 14:01:33.812665985 +0000 2024-09-13T14:06:30.454 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-13T14:06:30.517 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-09-13T14:06:30.517 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-09-13T14:06:30.517 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000251616 s, 2.0 MB/s 2024-09-13T14:06:30.518 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-13T14:06:30.575 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-09-13T14:06:30.575 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/scratch_devs of=/dev/stdout 2024-09-13T14:06:30.591 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-13T14:06:30.591 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_1 2024-09-13T14:06:30.647 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-13T14:06:30.647 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T14:06:30.647 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 961 Links: 1 2024-09-13T14:06:30.648 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T14:06:30.648 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T14:06:30.648 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2024-09-13 14:06:09.603918786 +0000 2024-09-13T14:06:30.648 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2024-09-13 14:01:34.201607554 +0000 2024-09-13T14:06:30.648 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2024-09-13 14:01:34.201607554 +0000 2024-09-13T14:06:30.648 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: 2024-09-13 14:01:34.201607554 +0000 2024-09-13T14:06:30.648 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-13T14:06:30.713 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2024-09-13T14:06:30.714 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2024-09-13T14:06:30.714 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.000111834 s, 4.6 MB/s 2024-09-13T14:06:30.715 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-13T14:06:30.771 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_2 2024-09-13T14:06:30.827 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-13T14:06:30.828 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T14:06:30.828 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 970 Links: 1 2024-09-13T14:06:30.828 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T14:06:30.828 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T14:06:30.828 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2024-09-13 14:06:09.604918784 +0000 2024-09-13T14:06:30.828 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2024-09-13 14:01:34.202607610 +0000 2024-09-13T14:06:30.828 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2024-09-13 14:01:34.202607610 +0000 2024-09-13T14:06:30.828 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: 2024-09-13 14:01:34.202607610 +0000 2024-09-13T14:06:30.828 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-13T14:06:30.891 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2024-09-13T14:06:30.892 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2024-09-13T14:06:30.892 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.00011981 s, 4.3 MB/s 2024-09-13T14:06:30.893 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-13T14:06:30.949 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_3 2024-09-13T14:06:30.953 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:30 smithi195 ceph-mon[33205]: from='client.14202 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi102=y;smithi195=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:06:30.953 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:30 smithi195 ceph-mon[33205]: Saving service mgr spec with placement smithi102=y;smithi195=x;count:2 2024-09-13T14:06:30.953 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:30 smithi195 ceph-mon[33205]: Reconfiguring mon.b (monmap changed)... 2024-09-13T14:06:30.953 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:30 smithi195 ceph-mon[33205]: Reconfiguring daemon mon.b on smithi195 2024-09-13T14:06:30.953 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:30 smithi195 ceph-mon[33205]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:30.968 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-13T14:06:30.968 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T14:06:30.968 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 950 Links: 1 2024-09-13T14:06:30.968 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T14:06:30.968 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T14:06:30.968 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2024-09-13 14:06:09.604918784 +0000 2024-09-13T14:06:30.968 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2024-09-13 14:01:34.200607498 +0000 2024-09-13T14:06:30.968 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2024-09-13 14:01:34.200607498 +0000 2024-09-13T14:06:30.969 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: 2024-09-13 14:01:34.200607498 +0000 2024-09-13T14:06:30.969 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-13T14:06:31.031 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2024-09-13T14:06:31.031 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2024-09-13T14:06:31.031 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 9.4244e-05 s, 5.4 MB/s 2024-09-13T14:06:31.032 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-13T14:06:31.088 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_4 2024-09-13T14:06:31.144 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-13T14:06:31.145 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T14:06:31.145 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 957 Links: 1 2024-09-13T14:06:31.145 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T14:06:31.145 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T14:06:31.145 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2024-09-13 14:06:09.604918784 +0000 2024-09-13T14:06:31.145 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2024-09-13 14:01:34.201607554 +0000 2024-09-13T14:06:31.145 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2024-09-13 14:01:34.201607554 +0000 2024-09-13T14:06:31.145 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: 2024-09-13 14:01:34.201607554 +0000 2024-09-13T14:06:31.146 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-13T14:06:31.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:30 smithi102 ceph-mon[29355]: from='client.14202 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi102=y;smithi195=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:06:31.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:30 smithi102 ceph-mon[29355]: Saving service mgr spec with placement smithi102=y;smithi195=x;count:2 2024-09-13T14:06:31.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:30 smithi102 ceph-mon[29355]: Reconfiguring mon.b (monmap changed)... 2024-09-13T14:06:31.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:30 smithi102 ceph-mon[29355]: Reconfiguring daemon mon.b on smithi195 2024-09-13T14:06:31.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:30 smithi102 ceph-mon[29355]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:31.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:30 smithi102 ceph-mon[40543]: from='client.14202 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi102=y;smithi195=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:06:31.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:30 smithi102 ceph-mon[40543]: Saving service mgr spec with placement smithi102=y;smithi195=x;count:2 2024-09-13T14:06:31.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:30 smithi102 ceph-mon[40543]: Reconfiguring mon.b (monmap changed)... 2024-09-13T14:06:31.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:30 smithi102 ceph-mon[40543]: Reconfiguring daemon mon.b on smithi195 2024-09-13T14:06:31.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:30 smithi102 ceph-mon[40543]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:31.212 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2024-09-13T14:06:31.212 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2024-09-13T14:06:31.212 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 8.8218e-05 s, 5.8 MB/s 2024-09-13T14:06:31.214 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-13T14:06:31.270 INFO:tasks.cephadm:Deploying osd.0 on smithi102 with /dev/vg_nvme/lv_4... 2024-09-13T14:06:31.271 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-13T14:06:31.460 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:06:32.163 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:31 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:32.163 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:31 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:32.163 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:31 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:06:32.163 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:31 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:32.163 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:31 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:06:32.163 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:31 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:32.163 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:31 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:06:32.163 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:31 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-13T14:06:32.163 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:31 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:06:32.164 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:31 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:32.164 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:31 smithi102 ceph-mon[29355]: Deploying daemon mgr.x on smithi195 2024-09-13T14:06:32.164 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:31 smithi102 ceph-mon[29355]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:32.164 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:31 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:32.164 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:31 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:32.164 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:31 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:06:32.164 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:31 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:32.164 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:31 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:06:32.164 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:31 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:32.164 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:31 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:06:32.164 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:31 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-13T14:06:32.165 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:31 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:06:32.165 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:31 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:32.165 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:31 smithi102 ceph-mon[40543]: Deploying daemon mgr.x on smithi195 2024-09-13T14:06:32.165 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:31 smithi102 ceph-mon[40543]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:32.326 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:31 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:32.326 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:31 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:32.326 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:31 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:06:32.326 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:31 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:32.326 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:31 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:06:32.326 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:31 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:32.326 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:31 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:06:32.326 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:31 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-13T14:06:32.327 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:31 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:06:32.327 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:31 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:32.327 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:31 smithi195 ceph-mon[33205]: Deploying daemon mgr.x on smithi195 2024-09-13T14:06:32.327 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:31 smithi195 ceph-mon[33205]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:32.997 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:06:32 smithi195 systemd[1]: Starting Ceph mgr.x for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T14:06:33.272 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:06:33 smithi195 podman[35249]: 2024-09-13 14:06:32.907669901 +0000 UTC m=+0.015629227 image pull c129f5a2fd4eccce3e6eec43830eff1929af98bc48f3b850cf144ce5b326e8b5 quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11 2024-09-13T14:06:33.272 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:06:33 smithi195 podman[35249]: 2024-09-13 14:06:33.027615541 +0000 UTC m=+0.135574920 container create 4a459ca2934735dbf2db60e427dbcf98d6cc9998ba1ec43b73c6a5023d1e2c8b (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=reef-2c01458, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-09-13T14:06:33.272 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:06:33 smithi195 podman[35249]: 2024-09-13 14:06:33.271653081 +0000 UTC m=+0.379612397 container init 4a459ca2934735dbf2db60e427dbcf98d6cc9998ba1ec43b73c6a5023d1e2c8b (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x, RELEASE=reef-2c01458, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-13T14:06:33.616 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:06:33 smithi195 podman[35249]: 2024-09-13 14:06:33.276652537 +0000 UTC m=+0.384611854 container start 4a459ca2934735dbf2db60e427dbcf98d6cc9998ba1ec43b73c6a5023d1e2c8b (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=reef-2c01458, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0) 2024-09-13T14:06:33.616 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:06:33 smithi195 ceph-mgr[35287]: -- 172.21.15.195:0/1175966022 <== mon.1 v2:172.21.15.195:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555e9a408d00 con 0x555e9a3dd400 2024-09-13T14:06:33.616 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:06:33 smithi195 bash[35249]: 4a459ca2934735dbf2db60e427dbcf98d6cc9998ba1ec43b73c6a5023d1e2c8b 2024-09-13T14:06:33.617 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:06:33 smithi195 systemd[1]: Started Ceph mgr.x for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:06:33.617 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:06:33 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:06:33.394+0000 7fcbffb3e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-13T14:06:33.617 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:06:33 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:06:33.458+0000 7fcbffb3e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-13T14:06:33.617 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:06:33 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:06:33.615+0000 7fcbffb3e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-13T14:06:33.776 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:06:33.794 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi102:vg_nvme/lv_4 2024-09-13T14:06:33.986 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:06:33.992 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:06:33 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:06:33.682+0000 7fcbffb3e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-13T14:06:33.992 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:06:33 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:06:33.850+0000 7fcbffb3e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:06:33.992 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:06:33 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:06:33.991+0000 7fcbffb3e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-13T14:06:34.569 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:06:34 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:06:34.313+0000 7fcbffb3e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-13T14:06:34.569 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:06:34 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:06:34.373+0000 7fcbffb3e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-13T14:06:34.569 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:34 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:34.569 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:34 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:34.570 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:34 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:34.570 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:34 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:34.570 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:34 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:06:34.570 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:34 smithi195 ceph-mon[33205]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:34.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:34 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:34.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:34 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:34.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:34 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:34.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:34 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:34.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:34 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:06:34.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:34 smithi102 ceph-mon[29355]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:34.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:34 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:34.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:34 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:34.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:34 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:34.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:34 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:34.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:34 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:06:34.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:34 smithi102 ceph-mon[40543]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:34.997 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:06:34 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:06:34.568+0000 7fcbffb3e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-13T14:06:35.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:35 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:35.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:35 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:35.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:35 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:35.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:35 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:06:35.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:35 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:35.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:35 smithi102 ceph-mon[29355]: Reconfiguring mgr.y (unknown last config time)... 2024-09-13T14:06:35.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:35 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:06:35.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:35 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:06:35.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:35 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:35.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:35 smithi102 ceph-mon[29355]: Reconfiguring daemon mgr.y on smithi102 2024-09-13T14:06:35.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:35 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:35.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:35 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:35.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:35 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:35.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:35 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:06:35.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:35 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:35.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:35 smithi102 ceph-mon[40543]: Reconfiguring mgr.y (unknown last config time)... 2024-09-13T14:06:35.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:35 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:06:35.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:35 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:06:35.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:35 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:35.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:35 smithi102 ceph-mon[40543]: Reconfiguring daemon mgr.y on smithi102 2024-09-13T14:06:35.747 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:06:35 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:06:35.438+0000 7fcbffb3e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-13T14:06:35.747 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:06:35 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:06:35.503+0000 7fcbffb3e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-13T14:06:35.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:35 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:35.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:35 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:35.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:35 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:35.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:35 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:06:35.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:35 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:35.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:35 smithi195 ceph-mon[33205]: Reconfiguring mgr.y (unknown last config time)... 2024-09-13T14:06:35.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:35 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:06:35.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:35 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:06:35.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:35 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:35.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:35 smithi195 ceph-mon[33205]: Reconfiguring daemon mgr.y on smithi102 2024-09-13T14:06:36.482 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:06:36 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:06:36.220+0000 7fcbffb3e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-13T14:06:36.482 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:06:36 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:06:36.338+0000 7fcbffb3e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-13T14:06:36.482 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:06:36 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:06:36.480+0000 7fcbffb3e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:06:36.747 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:06:36 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:06:36.543+0000 7fcbffb3e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-13T14:06:36.747 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:06:36 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:06:36.690+0000 7fcbffb3e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-13T14:06:36.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:36 smithi195 ceph-mon[33205]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:36.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:36 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:36.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:36 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:36.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:36 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:06:36.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:36 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:36.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:36 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:06:36.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:36 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:36.892 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:36 smithi102 ceph-mon[29355]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:36.892 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:36 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:36.893 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:36 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:36.893 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:36 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:06:36.893 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:36 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:36.893 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:36 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:06:36.893 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:36 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:36.893 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:36 smithi102 ceph-mon[40543]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:36.893 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:36 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:36.893 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:36 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:36.893 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:36 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:06:36.893 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:36 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:36.894 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:36 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:06:36.894 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:36 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:37.036 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:06:36 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:06:36.758+0000 7fcbffb3e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-13T14:06:37.036 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:06:36 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:06:36.817+0000 7fcbffb3e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-13T14:06:37.037 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:06:36 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:06:36.972+0000 7fcbffb3e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-13T14:06:37.037 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:06:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:06:37.035+0000 7fcbffb3e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-13T14:06:37.296 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:06:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:06:37.100+0000 7fcbffb3e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-13T14:06:37.297 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:06:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:06:37.180+0000 7fcbffb3e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-13T14:06:37.297 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:06:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:06:37.237+0000 7fcbffb3e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-13T14:06:37.297 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:06:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:06:37.295+0000 7fcbffb3e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-13T14:06:37.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:37 smithi195 ceph-mon[33205]: from='client.24113 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:06:37.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:37 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:06:37.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:37 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:06:37.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:37 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:37.754 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:37 smithi102 ceph-mon[29355]: from='client.24113 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:06:37.754 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:37 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:06:37.755 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:37 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:06:37.755 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:37 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:37.755 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:37 smithi102 ceph-mon[40543]: from='client.24113 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:06:37.755 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:37 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:06:37.755 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:37 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:06:37.755 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:37 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:38.497 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:06:38 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:06:38.069+0000 7fcbffb3e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-13T14:06:38.931 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:38 smithi102 ceph-mon[29355]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:38.932 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:38 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:38.932 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:38 smithi102 ceph-mon[29355]: Standby manager daemon x started 2024-09-13T14:06:38.932 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:38 smithi102 ceph-mon[29355]: from='mgr.? 172.21.15.195:0/2455161433' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:06:38.932 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:38 smithi102 ceph-mon[29355]: from='mgr.? 172.21.15.195:0/2455161433' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:06:38.932 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:38 smithi102 ceph-mon[29355]: from='mgr.? 172.21.15.195:0/2455161433' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:06:38.932 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:38 smithi102 ceph-mon[29355]: from='mgr.? 172.21.15.195:0/2455161433' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:06:38.932 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:38 smithi102 ceph-mon[40543]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:38.932 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:38 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:38.932 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:38 smithi102 ceph-mon[40543]: Standby manager daemon x started 2024-09-13T14:06:38.932 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:38 smithi102 ceph-mon[40543]: from='mgr.? 172.21.15.195:0/2455161433' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:06:38.933 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:38 smithi102 ceph-mon[40543]: from='mgr.? 172.21.15.195:0/2455161433' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:06:38.933 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:38 smithi102 ceph-mon[40543]: from='mgr.? 172.21.15.195:0/2455161433' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:06:38.933 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:38 smithi102 ceph-mon[40543]: from='mgr.? 172.21.15.195:0/2455161433' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:06:38.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:38 smithi195 ceph-mon[33205]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:38.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:38 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:38.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:38 smithi195 ceph-mon[33205]: Standby manager daemon x started 2024-09-13T14:06:38.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:38 smithi195 ceph-mon[33205]: from='mgr.? 172.21.15.195:0/2455161433' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:06:38.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:38 smithi195 ceph-mon[33205]: from='mgr.? 172.21.15.195:0/2455161433' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:06:38.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:38 smithi195 ceph-mon[33205]: from='mgr.? 172.21.15.195:0/2455161433' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:06:38.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:38 smithi195 ceph-mon[33205]: from='mgr.? 172.21.15.195:0/2455161433' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:06:39.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:39 smithi195 ceph-mon[33205]: mgrmap e14: y(active, since 61s), standbys: x 2024-09-13T14:06:39.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:39 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:06:39.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:39 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/3071601874' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b43b49e0-1cf0-47b8-aa85-37108f321220"}]: dispatch 2024-09-13T14:06:39.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:39 smithi195 ceph-mon[33205]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b43b49e0-1cf0-47b8-aa85-37108f321220"}]: dispatch 2024-09-13T14:06:39.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:39 smithi195 ceph-mon[33205]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b43b49e0-1cf0-47b8-aa85-37108f321220"}]': finished 2024-09-13T14:06:39.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:39 smithi195 ceph-mon[33205]: osdmap e5: 1 total, 0 up, 1 in 2024-09-13T14:06:39.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:39 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:06:39.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:39 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/199037376' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:06:40.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:39 smithi102 ceph-mon[29355]: mgrmap e14: y(active, since 61s), standbys: x 2024-09-13T14:06:40.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:39 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:06:40.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:39 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/3071601874' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b43b49e0-1cf0-47b8-aa85-37108f321220"}]: dispatch 2024-09-13T14:06:40.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:39 smithi102 ceph-mon[29355]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b43b49e0-1cf0-47b8-aa85-37108f321220"}]: dispatch 2024-09-13T14:06:40.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:39 smithi102 ceph-mon[29355]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b43b49e0-1cf0-47b8-aa85-37108f321220"}]': finished 2024-09-13T14:06:40.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:39 smithi102 ceph-mon[29355]: osdmap e5: 1 total, 0 up, 1 in 2024-09-13T14:06:40.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:39 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:06:40.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:39 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/199037376' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:06:40.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:39 smithi102 ceph-mon[40543]: mgrmap e14: y(active, since 61s), standbys: x 2024-09-13T14:06:40.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:39 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:06:40.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:39 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/3071601874' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b43b49e0-1cf0-47b8-aa85-37108f321220"}]: dispatch 2024-09-13T14:06:40.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:39 smithi102 ceph-mon[40543]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b43b49e0-1cf0-47b8-aa85-37108f321220"}]: dispatch 2024-09-13T14:06:40.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:39 smithi102 ceph-mon[40543]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b43b49e0-1cf0-47b8-aa85-37108f321220"}]': finished 2024-09-13T14:06:40.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:39 smithi102 ceph-mon[40543]: osdmap e5: 1 total, 0 up, 1 in 2024-09-13T14:06:40.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:39 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:06:40.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:39 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/199037376' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:06:40.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:40 smithi195 ceph-mon[33205]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:41.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:40 smithi102 ceph-mon[40543]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:41.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:40 smithi102 ceph-mon[29355]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:42.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:42 smithi195 ceph-mon[33205]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:43.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:42 smithi102 ceph-mon[29355]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:43.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:42 smithi102 ceph-mon[40543]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:44.983 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:44 smithi102 ceph-mon[40543]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:44.984 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:44 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-13T14:06:44.984 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:44 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:44.984 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:44 smithi102 ceph-mon[29355]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:44.985 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:44 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-13T14:06:44.985 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:44 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:44.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:44 smithi195 ceph-mon[33205]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:44.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:44 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-13T14:06:44.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:44 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:45.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:45 smithi195 ceph-mon[33205]: Deploying daemon osd.0 on smithi102 2024-09-13T14:06:46.103 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:45 smithi102 ceph-mon[29355]: Deploying daemon osd.0 on smithi102 2024-09-13T14:06:46.104 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:45 smithi102 ceph-mon[40543]: Deploying daemon osd.0 on smithi102 2024-09-13T14:06:46.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:46 smithi195 ceph-mon[33205]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:47.122 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:46 smithi102 ceph-mon[29355]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:47.123 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:46 smithi102 ceph-mon[40543]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:48.875 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:48 smithi102 ceph-mon[29355]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:48.875 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:48 smithi102 ceph-mon[40543]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:48.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:48 smithi195 ceph-mon[33205]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:49.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:49 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:06:49.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:49 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:49.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:49 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:50.056 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:49 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:06:50.056 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:49 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:50.056 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:49 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:50.056 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:49 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:06:50.056 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:49 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:50.056 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:49 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:50.996 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:50 smithi102 ceph-mon[29355]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:50.996 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:50 smithi102 ceph-mon[40543]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:50.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:50 smithi195 ceph-mon[33205]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:52.416 INFO:teuthology.orchestra.run.smithi102.stdout:Created osd(s) 0 on host 'smithi102' 2024-09-13T14:06:52.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:52 smithi195 ceph-mon[33205]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:52.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:52 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:52.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:52 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:52.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:52 smithi195 ceph-mon[33205]: from='osd.0 [v2:172.21.15.102:6802/466204285,v1:172.21.15.102:6803/466204285]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-13T14:06:53.015 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:52 smithi102 ceph-mon[40543]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:53.015 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:52 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:53.015 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:52 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:53.015 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:52 smithi102 ceph-mon[40543]: from='osd.0 [v2:172.21.15.102:6802/466204285,v1:172.21.15.102:6803/466204285]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-13T14:06:53.015 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:52 smithi102 ceph-mon[29355]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:53.015 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:52 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:53.015 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:52 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:53.016 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:52 smithi102 ceph-mon[29355]: from='osd.0 [v2:172.21.15.102:6802/466204285,v1:172.21.15.102:6803/466204285]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-13T14:06:53.041 DEBUG:teuthology.orchestra.run.smithi102:osd.0> sudo journalctl -f -n 0 -u ceph-08cee502-71d9-11ef-bceb-c7b262605968@osd.0.service 2024-09-13T14:06:53.043 INFO:tasks.cephadm:Deploying osd.1 on smithi102 with /dev/vg_nvme/lv_3... 2024-09-13T14:06:53.044 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-13T14:06:53.283 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:06:54.322 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:54 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:54.336 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:54 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:54.336 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:54 smithi102 ceph-mon[40543]: from='osd.0 [v2:172.21.15.102:6802/466204285,v1:172.21.15.102:6803/466204285]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-13T14:06:54.336 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:54 smithi102 ceph-mon[40543]: osdmap e6: 1 total, 0 up, 1 in 2024-09-13T14:06:54.337 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:54 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:06:54.337 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:54 smithi102 ceph-mon[40543]: from='osd.0 [v2:172.21.15.102:6802/466204285,v1:172.21.15.102:6803/466204285]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-09-13T14:06:54.337 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:54 smithi102 ceph-mon[40543]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:54.337 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:54 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:54.337 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:54 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:54.337 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:54 smithi102 ceph-mon[29355]: from='osd.0 [v2:172.21.15.102:6802/466204285,v1:172.21.15.102:6803/466204285]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-13T14:06:54.337 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:54 smithi102 ceph-mon[29355]: osdmap e6: 1 total, 0 up, 1 in 2024-09-13T14:06:54.337 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:54 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:06:54.338 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:54 smithi102 ceph-mon[29355]: from='osd.0 [v2:172.21.15.102:6802/466204285,v1:172.21.15.102:6803/466204285]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-09-13T14:06:54.338 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:54 smithi102 ceph-mon[29355]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:54.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:54 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:54.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:54 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:54.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:54 smithi195 ceph-mon[33205]: from='osd.0 [v2:172.21.15.102:6802/466204285,v1:172.21.15.102:6803/466204285]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-13T14:06:54.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:54 smithi195 ceph-mon[33205]: osdmap e6: 1 total, 0 up, 1 in 2024-09-13T14:06:54.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:54 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:06:54.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:54 smithi195 ceph-mon[33205]: from='osd.0 [v2:172.21.15.102:6802/466204285,v1:172.21.15.102:6803/466204285]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-09-13T14:06:54.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:54 smithi195 ceph-mon[33205]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:55.168 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:06:54 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0[44777]: 2024-09-13T14:06:54.839+0000 7ffbacec9640 -1 osd.0 0 waiting for initial osdmap 2024-09-13T14:06:55.168 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:06:54 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0[44777]: 2024-09-13T14:06:54.842+0000 7ffba4cea640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T14:06:55.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:55 smithi102 ceph-mon[29355]: from='osd.0 [v2:172.21.15.102:6802/466204285,v1:172.21.15.102:6803/466204285]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-09-13T14:06:55.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:55 smithi102 ceph-mon[29355]: osdmap e7: 1 total, 0 up, 1 in 2024-09-13T14:06:55.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:55 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:06:55.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:55 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:06:55.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:55 smithi102 ceph-mon[29355]: from='osd.0 [v2:172.21.15.102:6802/466204285,v1:172.21.15.102:6803/466204285]' entity='osd.0' 2024-09-13T14:06:55.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:55 smithi102 ceph-mon[40543]: from='osd.0 [v2:172.21.15.102:6802/466204285,v1:172.21.15.102:6803/466204285]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-09-13T14:06:55.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:55 smithi102 ceph-mon[40543]: osdmap e7: 1 total, 0 up, 1 in 2024-09-13T14:06:55.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:55 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:06:55.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:55 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:06:55.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:55 smithi102 ceph-mon[40543]: from='osd.0 [v2:172.21.15.102:6802/466204285,v1:172.21.15.102:6803/466204285]' entity='osd.0' 2024-09-13T14:06:55.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:55 smithi195 ceph-mon[33205]: from='osd.0 [v2:172.21.15.102:6802/466204285,v1:172.21.15.102:6803/466204285]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-09-13T14:06:55.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:55 smithi195 ceph-mon[33205]: osdmap e7: 1 total, 0 up, 1 in 2024-09-13T14:06:55.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:55 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:06:55.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:55 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:06:55.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:55 smithi195 ceph-mon[33205]: from='osd.0 [v2:172.21.15.102:6802/466204285,v1:172.21.15.102:6803/466204285]' entity='osd.0' 2024-09-13T14:06:56.719 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:56 smithi102 ceph-mon[40543]: purged_snaps scrub starts 2024-09-13T14:06:56.720 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:56 smithi102 ceph-mon[40543]: purged_snaps scrub ok 2024-09-13T14:06:56.720 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:56 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:06:56.720 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:56 smithi102 ceph-mon[40543]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:56.720 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:56 smithi102 ceph-mon[40543]: osd.0 [v2:172.21.15.102:6802/466204285,v1:172.21.15.102:6803/466204285] boot 2024-09-13T14:06:56.720 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:56 smithi102 ceph-mon[40543]: osdmap e8: 1 total, 1 up, 1 in 2024-09-13T14:06:56.720 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:56 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:06:56.720 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:56 smithi102 ceph-mon[29355]: purged_snaps scrub starts 2024-09-13T14:06:56.720 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:56 smithi102 ceph-mon[29355]: purged_snaps scrub ok 2024-09-13T14:06:56.721 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:56 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:06:56.721 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:56 smithi102 ceph-mon[29355]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:56.721 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:56 smithi102 ceph-mon[29355]: osd.0 [v2:172.21.15.102:6802/466204285,v1:172.21.15.102:6803/466204285] boot 2024-09-13T14:06:56.721 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:56 smithi102 ceph-mon[29355]: osdmap e8: 1 total, 1 up, 1 in 2024-09-13T14:06:56.721 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:56 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:06:56.722 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:06:56.740 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi102:vg_nvme/lv_3 2024-09-13T14:06:56.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:56 smithi195 ceph-mon[33205]: purged_snaps scrub starts 2024-09-13T14:06:56.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:56 smithi195 ceph-mon[33205]: purged_snaps scrub ok 2024-09-13T14:06:56.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:56 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:06:56.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:56 smithi195 ceph-mon[33205]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:06:56.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:56 smithi195 ceph-mon[33205]: osd.0 [v2:172.21.15.102:6802/466204285,v1:172.21.15.102:6803/466204285] boot 2024-09-13T14:06:56.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:56 smithi195 ceph-mon[33205]: osdmap e8: 1 total, 1 up, 1 in 2024-09-13T14:06:56.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:56 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:06:57.290 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:06:58.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:58 smithi195 ceph-mon[33205]: osdmap e9: 1 total, 1 up, 1 in 2024-09-13T14:06:58.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:58 smithi195 ceph-mon[33205]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:06:58.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:58 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:58.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:58 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:58.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:58 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:06:58.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:58 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:58.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:58 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:58.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:58 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:06:58.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:58 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:58.767 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:58 smithi102 ceph-mon[29355]: osdmap e9: 1 total, 1 up, 1 in 2024-09-13T14:06:58.768 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:58 smithi102 ceph-mon[29355]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:06:58.768 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:58 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:58.768 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:58 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:58.768 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:58 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:06:58.768 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:58 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:58.768 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:58 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:58.768 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:58 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:06:58.768 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:58 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:58.768 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:58 smithi102 ceph-mon[40543]: osdmap e9: 1 total, 1 up, 1 in 2024-09-13T14:06:58.768 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:58 smithi102 ceph-mon[40543]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:06:58.769 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:58 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:58.769 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:58 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:58.769 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:58 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:06:58.769 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:58 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:58.769 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:58 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:58.769 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:58 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:06:58.769 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:58 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:59.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:59 smithi195 ceph-mon[33205]: Detected new or changed devices on smithi102 2024-09-13T14:06:59.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:59 smithi195 ceph-mon[33205]: Adjusting osd_memory_target on smithi102 to 16064M 2024-09-13T14:06:59.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:59 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:06:59.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:59 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:59.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:59 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:06:59.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:59 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:59.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:59 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:06:59.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:59 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:06:59.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:06:59 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:59.772 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:59 smithi102 ceph-mon[29355]: Detected new or changed devices on smithi102 2024-09-13T14:06:59.772 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:59 smithi102 ceph-mon[29355]: Adjusting osd_memory_target on smithi102 to 16064M 2024-09-13T14:06:59.772 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:59 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:06:59.772 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:59 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:59.772 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:59 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:06:59.772 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:59 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:59.772 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:59 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:06:59.772 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:59 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:06:59.772 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:06:59 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:59.773 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:59 smithi102 ceph-mon[40543]: Detected new or changed devices on smithi102 2024-09-13T14:06:59.773 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:59 smithi102 ceph-mon[40543]: Adjusting osd_memory_target on smithi102 to 16064M 2024-09-13T14:06:59.773 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:59 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:06:59.773 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:59 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:06:59.773 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:59 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:06:59.773 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:59 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:06:59.773 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:59 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:06:59.773 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:59 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:06:59.773 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:06:59 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:07:00.727 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:00 smithi102 ceph-mon[29355]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:07:00.727 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:00 smithi102 ceph-mon[29355]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:07:00.728 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:00 smithi102 ceph-mon[40543]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:07:00.728 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:00 smithi102 ceph-mon[40543]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:07:00.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:00 smithi195 ceph-mon[33205]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:07:00.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:00 smithi195 ceph-mon[33205]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:07:01.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:01 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/2395447196' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "27e10d5e-c471-4132-9c73-83c5708c3b08"}]: dispatch 2024-09-13T14:07:01.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:01 smithi102 ceph-mon[40543]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "27e10d5e-c471-4132-9c73-83c5708c3b08"}]: dispatch 2024-09-13T14:07:01.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:01 smithi102 ceph-mon[40543]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "27e10d5e-c471-4132-9c73-83c5708c3b08"}]': finished 2024-09-13T14:07:01.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:01 smithi102 ceph-mon[40543]: osdmap e10: 2 total, 1 up, 2 in 2024-09-13T14:07:01.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:01 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:07:01.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:01 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/2395447196' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "27e10d5e-c471-4132-9c73-83c5708c3b08"}]: dispatch 2024-09-13T14:07:01.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:01 smithi102 ceph-mon[29355]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "27e10d5e-c471-4132-9c73-83c5708c3b08"}]: dispatch 2024-09-13T14:07:01.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:01 smithi102 ceph-mon[29355]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "27e10d5e-c471-4132-9c73-83c5708c3b08"}]': finished 2024-09-13T14:07:01.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:01 smithi102 ceph-mon[29355]: osdmap e10: 2 total, 1 up, 2 in 2024-09-13T14:07:01.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:01 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:07:01.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:01 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/2395447196' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "27e10d5e-c471-4132-9c73-83c5708c3b08"}]: dispatch 2024-09-13T14:07:01.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:01 smithi195 ceph-mon[33205]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "27e10d5e-c471-4132-9c73-83c5708c3b08"}]: dispatch 2024-09-13T14:07:01.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:01 smithi195 ceph-mon[33205]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "27e10d5e-c471-4132-9c73-83c5708c3b08"}]': finished 2024-09-13T14:07:01.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:01 smithi195 ceph-mon[33205]: osdmap e10: 2 total, 1 up, 2 in 2024-09-13T14:07:01.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:01 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:07:02.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:02 smithi102 ceph-mon[29355]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:07:02.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:02 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/3273720722' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:07:02.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:02 smithi102 ceph-mon[40543]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:07:02.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:02 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/3273720722' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:07:02.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:02 smithi195 ceph-mon[33205]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:07:02.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:02 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/3273720722' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:07:04.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:04 smithi102 ceph-mon[29355]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:07:04.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:04 smithi102 ceph-mon[40543]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:07:04.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:04 smithi195 ceph-mon[33205]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:07:06.873 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:06 smithi102 ceph-mon[40543]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:07:06.873 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:06 smithi102 ceph-mon[29355]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:07:06.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:06 smithi195 ceph-mon[33205]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:07:07.985 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:07 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-13T14:07:07.985 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:07 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:07:07.985 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:07 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-13T14:07:07.985 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:07 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:07:07.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:07 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-13T14:07:07.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:07 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:07:08.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:08 smithi102 ceph-mon[29355]: Deploying daemon osd.1 on smithi102 2024-09-13T14:07:08.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:08 smithi102 ceph-mon[29355]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:07:08.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:08 smithi102 ceph-mon[40543]: Deploying daemon osd.1 on smithi102 2024-09-13T14:07:08.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:08 smithi102 ceph-mon[40543]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:07:08.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:08 smithi195 ceph-mon[33205]: Deploying daemon osd.1 on smithi102 2024-09-13T14:07:08.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:08 smithi195 ceph-mon[33205]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:07:10.808 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:10 smithi102 ceph-mon[29355]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:07:10.809 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:10 smithi102 ceph-mon[40543]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:07:10.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:10 smithi195 ceph-mon[33205]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:07:12.776 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:12 smithi102 ceph-mon[29355]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:07:12.777 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:12 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:07:12.777 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:12 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:12.777 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:12 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:12.777 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:12 smithi102 ceph-mon[40543]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:07:12.777 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:12 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:07:12.777 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:12 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:12.777 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:12 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:12.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:12 smithi195 ceph-mon[33205]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:07:12.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:12 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:07:12.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:12 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:12.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:12 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:14.866 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:14 smithi102 ceph-mon[29355]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:07:14.866 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:14 smithi102 ceph-mon[40543]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:07:14.900 INFO:teuthology.orchestra.run.smithi102.stdout:Created osd(s) 1 on host 'smithi102' 2024-09-13T14:07:14.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:14 smithi195 ceph-mon[33205]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:07:15.499 DEBUG:teuthology.orchestra.run.smithi102:osd.1> sudo journalctl -f -n 0 -u ceph-08cee502-71d9-11ef-bceb-c7b262605968@osd.1.service 2024-09-13T14:07:15.501 INFO:tasks.cephadm:Deploying osd.2 on smithi102 with /dev/vg_nvme/lv_2... 2024-09-13T14:07:15.501 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-13T14:07:15.748 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:07:16.165 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:15 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:16.165 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:15 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:16.165 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:15 smithi102 ceph-mon[29355]: from='osd.1 [v2:172.21.15.102:6810/1293422793,v1:172.21.15.102:6811/1293422793]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-13T14:07:16.165 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:15 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:16.165 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:15 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:16.165 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:15 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:16.166 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:15 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:16.166 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:15 smithi102 ceph-mon[40543]: from='osd.1 [v2:172.21.15.102:6810/1293422793,v1:172.21.15.102:6811/1293422793]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-13T14:07:16.166 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:15 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:16.166 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:15 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:16.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:15 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:16.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:15 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:16.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:15 smithi195 ceph-mon[33205]: from='osd.1 [v2:172.21.15.102:6810/1293422793,v1:172.21.15.102:6811/1293422793]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-13T14:07:16.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:15 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:16.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:15 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:17.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:16 smithi102 ceph-mon[29355]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:07:17.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:16 smithi102 ceph-mon[29355]: from='osd.1 [v2:172.21.15.102:6810/1293422793,v1:172.21.15.102:6811/1293422793]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-13T14:07:17.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:16 smithi102 ceph-mon[29355]: osdmap e11: 2 total, 1 up, 2 in 2024-09-13T14:07:17.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:16 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:07:17.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:16 smithi102 ceph-mon[29355]: from='osd.1 [v2:172.21.15.102:6810/1293422793,v1:172.21.15.102:6811/1293422793]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-09-13T14:07:17.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:16 smithi102 ceph-mon[40543]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:07:17.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:16 smithi102 ceph-mon[40543]: from='osd.1 [v2:172.21.15.102:6810/1293422793,v1:172.21.15.102:6811/1293422793]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-13T14:07:17.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:16 smithi102 ceph-mon[40543]: osdmap e11: 2 total, 1 up, 2 in 2024-09-13T14:07:17.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:16 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:07:17.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:16 smithi102 ceph-mon[40543]: from='osd.1 [v2:172.21.15.102:6810/1293422793,v1:172.21.15.102:6811/1293422793]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-09-13T14:07:17.170 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:07:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1[51038]: 2024-09-13T14:07:16.905+0000 7f31edbc4640 -1 osd.1 0 waiting for initial osdmap 2024-09-13T14:07:17.170 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:07:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1[51038]: 2024-09-13T14:07:16.909+0000 7f31e99ed640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T14:07:17.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:16 smithi195 ceph-mon[33205]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:07:17.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:16 smithi195 ceph-mon[33205]: from='osd.1 [v2:172.21.15.102:6810/1293422793,v1:172.21.15.102:6811/1293422793]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-13T14:07:17.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:16 smithi195 ceph-mon[33205]: osdmap e11: 2 total, 1 up, 2 in 2024-09-13T14:07:17.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:16 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:07:17.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:16 smithi195 ceph-mon[33205]: from='osd.1 [v2:172.21.15.102:6810/1293422793,v1:172.21.15.102:6811/1293422793]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-09-13T14:07:18.045 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:17 smithi102 ceph-mon[29355]: from='osd.1 [v2:172.21.15.102:6810/1293422793,v1:172.21.15.102:6811/1293422793]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-09-13T14:07:18.046 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:17 smithi102 ceph-mon[29355]: osdmap e12: 2 total, 1 up, 2 in 2024-09-13T14:07:18.046 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:17 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:07:18.046 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:17 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:07:18.046 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:17 smithi102 ceph-mon[29355]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:07:18.046 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:17 smithi102 ceph-mon[40543]: from='osd.1 [v2:172.21.15.102:6810/1293422793,v1:172.21.15.102:6811/1293422793]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-09-13T14:07:18.046 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:17 smithi102 ceph-mon[40543]: osdmap e12: 2 total, 1 up, 2 in 2024-09-13T14:07:18.046 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:17 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:07:18.046 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:17 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:07:18.047 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:17 smithi102 ceph-mon[40543]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:07:18.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:17 smithi195 ceph-mon[33205]: from='osd.1 [v2:172.21.15.102:6810/1293422793,v1:172.21.15.102:6811/1293422793]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-09-13T14:07:18.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:17 smithi195 ceph-mon[33205]: osdmap e12: 2 total, 1 up, 2 in 2024-09-13T14:07:18.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:17 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:07:18.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:17 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:07:18.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:17 smithi195 ceph-mon[33205]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:07:18.886 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:07:18.910 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi102:vg_nvme/lv_2 2024-09-13T14:07:19.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:18 smithi102 ceph-mon[29355]: purged_snaps scrub starts 2024-09-13T14:07:19.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:18 smithi102 ceph-mon[29355]: purged_snaps scrub ok 2024-09-13T14:07:19.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:18 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:07:19.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:18 smithi102 ceph-mon[29355]: osd.1 [v2:172.21.15.102:6810/1293422793,v1:172.21.15.102:6811/1293422793] boot 2024-09-13T14:07:19.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:18 smithi102 ceph-mon[29355]: osdmap e13: 2 total, 2 up, 2 in 2024-09-13T14:07:19.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:18 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:07:19.151 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:18 smithi102 ceph-mon[40543]: purged_snaps scrub starts 2024-09-13T14:07:19.151 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:18 smithi102 ceph-mon[40543]: purged_snaps scrub ok 2024-09-13T14:07:19.151 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:18 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:07:19.151 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:18 smithi102 ceph-mon[40543]: osd.1 [v2:172.21.15.102:6810/1293422793,v1:172.21.15.102:6811/1293422793] boot 2024-09-13T14:07:19.151 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:18 smithi102 ceph-mon[40543]: osdmap e13: 2 total, 2 up, 2 in 2024-09-13T14:07:19.151 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:18 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:07:19.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:18 smithi195 ceph-mon[33205]: purged_snaps scrub starts 2024-09-13T14:07:19.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:18 smithi195 ceph-mon[33205]: purged_snaps scrub ok 2024-09-13T14:07:19.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:18 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:07:19.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:18 smithi195 ceph-mon[33205]: osd.1 [v2:172.21.15.102:6810/1293422793,v1:172.21.15.102:6811/1293422793] boot 2024-09-13T14:07:19.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:18 smithi195 ceph-mon[33205]: osdmap e13: 2 total, 2 up, 2 in 2024-09-13T14:07:19.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:18 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:07:19.577 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:07:19.965 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:19 smithi102 ceph-mon[29355]: osdmap e14: 2 total, 2 up, 2 in 2024-09-13T14:07:19.965 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:19 smithi102 ceph-mon[29355]: pgmap v55: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:07:19.965 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:19 smithi102 ceph-mon[40543]: osdmap e14: 2 total, 2 up, 2 in 2024-09-13T14:07:19.965 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:19 smithi102 ceph-mon[40543]: pgmap v55: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:07:20.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:19 smithi195 ceph-mon[33205]: osdmap e14: 2 total, 2 up, 2 in 2024-09-13T14:07:20.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:19 smithi195 ceph-mon[33205]: pgmap v55: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:07:22.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:21 smithi102 ceph-mon[29355]: Detected new or changed devices on smithi102 2024-09-13T14:07:22.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:21 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:22.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:21 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:22.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:21 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:07:22.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:21 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:07:22.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:21 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:22.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:21 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:07:22.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:21 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:07:22.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:21 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:22.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:21 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:07:22.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:21 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:07:22.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:21 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:07:22.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:21 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:22.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:21 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:07:22.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:21 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:07:22.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:21 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:07:22.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:21 smithi102 ceph-mon[40543]: Detected new or changed devices on smithi102 2024-09-13T14:07:22.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:21 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:22.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:21 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:22.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:21 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:07:22.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:21 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:07:22.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:21 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:22.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:21 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:07:22.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:21 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:07:22.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:21 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:22.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:21 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:07:22.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:21 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:07:22.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:21 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:07:22.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:21 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:22.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:21 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:07:22.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:21 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:07:22.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:21 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:07:22.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:21 smithi195 ceph-mon[33205]: Detected new or changed devices on smithi102 2024-09-13T14:07:22.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:21 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:22.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:21 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:22.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:21 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:07:22.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:21 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:07:22.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:21 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:22.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:21 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:07:22.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:21 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:07:22.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:21 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:22.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:21 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:07:22.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:21 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:07:22.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:21 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:07:22.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:21 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:22.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:21 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:07:22.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:21 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:07:22.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:21 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:07:23.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:22 smithi102 ceph-mon[29355]: Adjusting osd_memory_target on smithi102 to 8032M 2024-09-13T14:07:23.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:22 smithi102 ceph-mon[29355]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:07:23.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:22 smithi102 ceph-mon[29355]: from='client.14265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:07:23.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:22 smithi102 ceph-mon[40543]: Adjusting osd_memory_target on smithi102 to 8032M 2024-09-13T14:07:23.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:22 smithi102 ceph-mon[40543]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:07:23.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:22 smithi102 ceph-mon[40543]: from='client.14265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:07:23.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:22 smithi195 ceph-mon[33205]: Adjusting osd_memory_target on smithi102 to 8032M 2024-09-13T14:07:23.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:22 smithi195 ceph-mon[33205]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:07:23.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:22 smithi195 ceph-mon[33205]: from='client.14265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:07:24.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:23 smithi102 ceph-mon[40543]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:07:24.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:23 smithi102 ceph-mon[29355]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:07:24.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:23 smithi195 ceph-mon[33205]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:07:25.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:24 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/1674746212' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40a94b2b-2c24-4164-91be-0816a1acf3d6"}]: dispatch 2024-09-13T14:07:25.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:24 smithi102 ceph-mon[29355]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40a94b2b-2c24-4164-91be-0816a1acf3d6"}]: dispatch 2024-09-13T14:07:25.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:24 smithi102 ceph-mon[29355]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "40a94b2b-2c24-4164-91be-0816a1acf3d6"}]': finished 2024-09-13T14:07:25.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:24 smithi102 ceph-mon[29355]: osdmap e15: 3 total, 2 up, 3 in 2024-09-13T14:07:25.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:24 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:07:25.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:24 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/1424609222' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:07:25.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:24 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/1674746212' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40a94b2b-2c24-4164-91be-0816a1acf3d6"}]: dispatch 2024-09-13T14:07:25.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:24 smithi102 ceph-mon[40543]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40a94b2b-2c24-4164-91be-0816a1acf3d6"}]: dispatch 2024-09-13T14:07:25.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:24 smithi102 ceph-mon[40543]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "40a94b2b-2c24-4164-91be-0816a1acf3d6"}]': finished 2024-09-13T14:07:25.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:24 smithi102 ceph-mon[40543]: osdmap e15: 3 total, 2 up, 3 in 2024-09-13T14:07:25.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:24 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:07:25.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:24 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/1424609222' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:07:25.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:24 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/1674746212' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40a94b2b-2c24-4164-91be-0816a1acf3d6"}]: dispatch 2024-09-13T14:07:25.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:24 smithi195 ceph-mon[33205]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40a94b2b-2c24-4164-91be-0816a1acf3d6"}]: dispatch 2024-09-13T14:07:25.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:24 smithi195 ceph-mon[33205]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "40a94b2b-2c24-4164-91be-0816a1acf3d6"}]': finished 2024-09-13T14:07:25.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:24 smithi195 ceph-mon[33205]: osdmap e15: 3 total, 2 up, 3 in 2024-09-13T14:07:25.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:24 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:07:25.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:24 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/1424609222' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:07:26.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:25 smithi195 ceph-mon[33205]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:07:26.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:25 smithi102 ceph-mon[29355]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:07:26.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:25 smithi102 ceph-mon[40543]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:07:28.959 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:28 smithi102 ceph-mon[29355]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:07:28.959 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:28 smithi102 ceph-mon[40543]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:07:28.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:28 smithi195 ceph-mon[33205]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:07:30.878 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:30 smithi102 ceph-mon[40543]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:07:30.878 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:30 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-13T14:07:30.878 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:30 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:07:30.878 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:30 smithi102 ceph-mon[40543]: Deploying daemon osd.2 on smithi102 2024-09-13T14:07:30.878 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:30 smithi102 ceph-mon[29355]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:07:30.878 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:30 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-13T14:07:30.878 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:30 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:07:30.878 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:30 smithi102 ceph-mon[29355]: Deploying daemon osd.2 on smithi102 2024-09-13T14:07:30.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:30 smithi195 ceph-mon[33205]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:07:30.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:30 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-13T14:07:30.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:30 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:07:30.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:30 smithi195 ceph-mon[33205]: Deploying daemon osd.2 on smithi102 2024-09-13T14:07:32.981 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:32 smithi102 ceph-mon[29355]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:07:32.981 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:32 smithi102 ceph-mon[40543]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:07:32.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:32 smithi195 ceph-mon[33205]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:07:34.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:34 smithi102 ceph-mon[29355]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:07:34.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:34 smithi102 ceph-mon[40543]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:07:34.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:34 smithi195 ceph-mon[33205]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:07:35.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:35 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:07:35.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:35 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:35.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:35 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:36.112 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:35 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:07:36.112 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:35 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:36.112 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:35 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:36.112 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:35 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:07:36.112 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:35 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:36.113 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:35 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:36.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:36 smithi102 ceph-mon[29355]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:07:36.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:36 smithi102 ceph-mon[40543]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:07:37.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:36 smithi195 ceph-mon[33205]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:07:37.918 INFO:teuthology.orchestra.run.smithi102.stdout:Created osd(s) 2 on host 'smithi102' 2024-09-13T14:07:38.480 DEBUG:teuthology.orchestra.run.smithi102:osd.2> sudo journalctl -f -n 0 -u ceph-08cee502-71d9-11ef-bceb-c7b262605968@osd.2.service 2024-09-13T14:07:38.483 INFO:tasks.cephadm:Deploying osd.3 on smithi102 with /dev/vg_nvme/lv_1... 2024-09-13T14:07:38.483 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-13T14:07:38.734 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:07:39.088 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:38 smithi102 ceph-mon[29355]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:07:39.089 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:38 smithi102 ceph-mon[29355]: from='osd.2 [v2:172.21.15.102:6818/360502781,v1:172.21.15.102:6819/360502781]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-13T14:07:39.089 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:38 smithi102 ceph-mon[29355]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-13T14:07:39.089 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:38 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:39.089 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:38 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:39.089 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:38 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:39.089 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:38 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:39.090 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:38 smithi102 ceph-mon[40543]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:07:39.090 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:38 smithi102 ceph-mon[40543]: from='osd.2 [v2:172.21.15.102:6818/360502781,v1:172.21.15.102:6819/360502781]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-13T14:07:39.090 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:38 smithi102 ceph-mon[40543]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-13T14:07:39.090 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:38 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:39.090 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:38 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:39.090 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:38 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:39.090 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:38 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:39.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:38 smithi195 ceph-mon[33205]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:07:39.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:38 smithi195 ceph-mon[33205]: from='osd.2 [v2:172.21.15.102:6818/360502781,v1:172.21.15.102:6819/360502781]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-13T14:07:39.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:38 smithi195 ceph-mon[33205]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-13T14:07:39.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:38 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:39.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:38 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:39.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:38 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:39.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:38 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:40.102 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:39 smithi102 ceph-mon[29355]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-13T14:07:40.102 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:39 smithi102 ceph-mon[29355]: osdmap e16: 3 total, 2 up, 3 in 2024-09-13T14:07:40.102 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:39 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:07:40.102 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:39 smithi102 ceph-mon[29355]: from='osd.2 [v2:172.21.15.102:6818/360502781,v1:172.21.15.102:6819/360502781]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-09-13T14:07:40.102 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:39 smithi102 ceph-mon[29355]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-09-13T14:07:40.103 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:39 smithi102 ceph-mon[40543]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-13T14:07:40.103 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:39 smithi102 ceph-mon[40543]: osdmap e16: 3 total, 2 up, 3 in 2024-09-13T14:07:40.103 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:39 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:07:40.103 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:39 smithi102 ceph-mon[40543]: from='osd.2 [v2:172.21.15.102:6818/360502781,v1:172.21.15.102:6819/360502781]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-09-13T14:07:40.103 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:39 smithi102 ceph-mon[40543]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-09-13T14:07:40.103 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:07:39 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2[56992]: 2024-09-13T14:07:39.783+0000 7f739c3ec640 -1 osd.2 0 waiting for initial osdmap 2024-09-13T14:07:40.103 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:07:39 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2[56992]: 2024-09-13T14:07:39.786+0000 7f7398215640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T14:07:40.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:39 smithi195 ceph-mon[33205]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-13T14:07:40.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:39 smithi195 ceph-mon[33205]: osdmap e16: 3 total, 2 up, 3 in 2024-09-13T14:07:40.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:39 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:07:40.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:39 smithi195 ceph-mon[33205]: from='osd.2 [v2:172.21.15.102:6818/360502781,v1:172.21.15.102:6819/360502781]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-09-13T14:07:40.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:39 smithi195 ceph-mon[33205]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-09-13T14:07:40.831 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:40 smithi102 ceph-mon[29355]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:07:40.831 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:40 smithi102 ceph-mon[29355]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-09-13T14:07:40.831 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:40 smithi102 ceph-mon[29355]: osdmap e17: 3 total, 2 up, 3 in 2024-09-13T14:07:40.831 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:40 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:07:40.832 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:40 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:07:40.832 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:40 smithi102 ceph-mon[40543]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:07:40.832 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:40 smithi102 ceph-mon[40543]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-09-13T14:07:40.832 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:40 smithi102 ceph-mon[40543]: osdmap e17: 3 total, 2 up, 3 in 2024-09-13T14:07:40.832 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:40 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:07:40.832 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:40 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:07:41.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:40 smithi195 ceph-mon[33205]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:07:41.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:40 smithi195 ceph-mon[33205]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-09-13T14:07:41.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:40 smithi195 ceph-mon[33205]: osdmap e17: 3 total, 2 up, 3 in 2024-09-13T14:07:41.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:40 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:07:41.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:40 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:07:42.121 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:41 smithi102 ceph-mon[29355]: purged_snaps scrub starts 2024-09-13T14:07:42.121 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:41 smithi102 ceph-mon[29355]: purged_snaps scrub ok 2024-09-13T14:07:42.122 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:41 smithi102 ceph-mon[29355]: osd.2 [v2:172.21.15.102:6818/360502781,v1:172.21.15.102:6819/360502781] boot 2024-09-13T14:07:42.122 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:41 smithi102 ceph-mon[29355]: osdmap e18: 3 total, 3 up, 3 in 2024-09-13T14:07:42.122 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:41 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:07:42.122 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:41 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-13T14:07:42.122 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:41 smithi102 ceph-mon[40543]: purged_snaps scrub starts 2024-09-13T14:07:42.122 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:41 smithi102 ceph-mon[40543]: purged_snaps scrub ok 2024-09-13T14:07:42.122 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:41 smithi102 ceph-mon[40543]: osd.2 [v2:172.21.15.102:6818/360502781,v1:172.21.15.102:6819/360502781] boot 2024-09-13T14:07:42.122 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:41 smithi102 ceph-mon[40543]: osdmap e18: 3 total, 3 up, 3 in 2024-09-13T14:07:42.122 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:41 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:07:42.122 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:41 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-13T14:07:42.124 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:07:42.142 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi102:vg_nvme/lv_1 2024-09-13T14:07:42.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:41 smithi195 ceph-mon[33205]: purged_snaps scrub starts 2024-09-13T14:07:42.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:41 smithi195 ceph-mon[33205]: purged_snaps scrub ok 2024-09-13T14:07:42.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:41 smithi195 ceph-mon[33205]: osd.2 [v2:172.21.15.102:6818/360502781,v1:172.21.15.102:6819/360502781] boot 2024-09-13T14:07:42.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:41 smithi195 ceph-mon[33205]: osdmap e18: 3 total, 3 up, 3 in 2024-09-13T14:07:42.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:41 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:07:42.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:41 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-13T14:07:42.624 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:07:43.045 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:42 smithi102 ceph-mon[29355]: pgmap v70: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:07:43.045 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:42 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-13T14:07:43.046 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:42 smithi102 ceph-mon[29355]: osdmap e19: 3 total, 3 up, 3 in 2024-09-13T14:07:43.046 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:42 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T14:07:43.046 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:42 smithi102 ceph-mon[40543]: pgmap v70: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:07:43.046 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:42 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-13T14:07:43.046 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:42 smithi102 ceph-mon[40543]: osdmap e19: 3 total, 3 up, 3 in 2024-09-13T14:07:43.046 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:42 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T14:07:43.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:42 smithi195 ceph-mon[33205]: pgmap v70: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:07:43.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:42 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-13T14:07:43.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:42 smithi195 ceph-mon[33205]: osdmap e19: 3 total, 3 up, 3 in 2024-09-13T14:07:43.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:42 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T14:07:43.418 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:07:43 smithi102 sudo[60812]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-13T14:07:43.418 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:07:43 smithi102 sudo[60812]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T14:07:43.418 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:07:43 smithi102 sudo[60812]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T14:07:43.418 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:07:43 smithi102 sudo[60812]: pam_unix(sudo:session): session closed for user root 2024-09-13T14:07:43.805 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:07:43 smithi102 sudo[60823]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-13T14:07:43.805 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:07:43 smithi102 sudo[60823]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T14:07:43.805 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:07:43 smithi102 sudo[60823]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T14:07:43.805 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:07:43 smithi102 sudo[60823]: pam_unix(sudo:session): session closed for user root 2024-09-13T14:07:44.072 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:07:43 smithi102 sudo[60834]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-13T14:07:44.072 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:07:43 smithi102 sudo[60834]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T14:07:44.072 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:07:43 smithi102 sudo[60834]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T14:07:44.072 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:07:43 smithi102 sudo[60834]: pam_unix(sudo:session): session closed for user root 2024-09-13T14:07:44.073 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:43 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:07:44.073 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:43 smithi102 ceph-mon[29355]: osdmap e20: 3 total, 3 up, 3 in 2024-09-13T14:07:44.073 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:43 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:44.073 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:43 smithi102 ceph-mon[29355]: pgmap v73: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:07:44.073 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:43 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:07:44.073 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:43 smithi102 ceph-mon[40543]: osdmap e20: 3 total, 3 up, 3 in 2024-09-13T14:07:44.073 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:43 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:44.073 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:43 smithi102 ceph-mon[40543]: pgmap v73: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:07:44.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:43 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:07:44.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:43 smithi195 ceph-mon[33205]: osdmap e20: 3 total, 3 up, 3 in 2024-09-13T14:07:44.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:43 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:44.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:43 smithi195 ceph-mon[33205]: pgmap v73: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:07:44.326 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:07:44 smithi102 sudo[60868]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-13T14:07:44.326 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:07:44 smithi102 sudo[60868]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T14:07:44.327 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:07:44 smithi102 sudo[60868]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T14:07:44.327 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:07:44 smithi102 sudo[60868]: pam_unix(sudo:session): session closed for user root 2024-09-13T14:07:44.599 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:07:44 smithi102 sudo[60897]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-13T14:07:44.599 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:07:44 smithi102 sudo[60897]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T14:07:44.599 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:07:44 smithi102 sudo[60897]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T14:07:44.599 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:07:44 smithi102 sudo[60897]: pam_unix(sudo:session): session closed for user root 2024-09-13T14:07:44.863 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:07:44 smithi102 sudo[60973]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-13T14:07:44.863 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:07:44 smithi102 sudo[60973]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T14:07:44.863 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:07:44 smithi102 sudo[60973]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T14:07:44.863 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:07:44 smithi102 sudo[60973]: pam_unix(sudo:session): session closed for user root 2024-09-13T14:07:44.864 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:44 smithi102 sudo[60990]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-13T14:07:44.864 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:44 smithi102 sudo[60990]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T14:07:44.864 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:44 smithi102 sudo[60990]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T14:07:44.864 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:44 smithi102 sudo[60990]: pam_unix(sudo:session): session closed for user root 2024-09-13T14:07:45.113 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:45 smithi102 sudo[61035]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-13T14:07:45.365 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:45 smithi102 ceph-mon[40543]: osdmap e21: 3 total, 3 up, 3 in 2024-09-13T14:07:45.365 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:45 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:07:45.365 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:45 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:07:45.366 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:45 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:07:45.366 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:45 smithi102 ceph-mon[40543]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-13T14:07:45.366 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:45 smithi102 ceph-mon[40543]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T14:07:45.366 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:45 smithi102 sudo[61035]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T14:07:45.366 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:45 smithi102 sudo[61035]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T14:07:45.366 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:45 smithi102 sudo[61035]: pam_unix(sudo:session): session closed for user root 2024-09-13T14:07:45.366 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:45 smithi102 ceph-mon[29355]: osdmap e21: 3 total, 3 up, 3 in 2024-09-13T14:07:45.366 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:45 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:07:45.366 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:45 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:07:45.366 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:45 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:07:45.366 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:45 smithi102 ceph-mon[29355]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-13T14:07:45.366 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:45 smithi102 ceph-mon[29355]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T14:07:45.378 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:45 smithi195 ceph-mon[33205]: osdmap e21: 3 total, 3 up, 3 in 2024-09-13T14:07:45.378 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:45 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:07:45.378 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:45 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:07:45.378 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:45 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:07:45.378 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:45 smithi195 ceph-mon[33205]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-13T14:07:45.378 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:45 smithi195 ceph-mon[33205]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T14:07:45.378 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:45 smithi195 sudo[35876]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-13T14:07:45.378 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:45 smithi195 sudo[35876]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T14:07:45.378 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:45 smithi195 sudo[35876]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T14:07:45.378 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:45 smithi195 sudo[35876]: pam_unix(sudo:session): session closed for user root 2024-09-13T14:07:45.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:45 smithi195 sudo[35879]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-13T14:07:45.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:45 smithi195 sudo[35879]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T14:07:45.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:45 smithi195 sudo[35879]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T14:07:45.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:45 smithi195 sudo[35879]: pam_unix(sudo:session): session closed for user root 2024-09-13T14:07:46.080 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:45 smithi102 sudo[61114]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-13T14:07:46.080 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:45 smithi102 sudo[61114]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T14:07:46.081 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:45 smithi102 sudo[61114]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T14:07:46.081 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:45 smithi102 sudo[61114]: pam_unix(sudo:session): session closed for user root 2024-09-13T14:07:46.344 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[29355]: from='client.14298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:07:46.344 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[29355]: Detected new or changed devices on smithi102 2024-09-13T14:07:46.344 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[29355]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T14:07:46.344 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:07:46.344 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:07:46.344 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:07:46.344 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[29355]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T14:07:46.344 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:07:46.344 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:07:46.345 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:07:46.345 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:46.345 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[29355]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T14:07:46.345 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:46.345 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:07:46.345 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:07:46.345 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:07:46.345 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[29355]: Adjusting osd_memory_target on smithi102 to 5354M 2024-09-13T14:07:46.345 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:46.345 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:07:46.345 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:07:46.345 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:46.346 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:46.346 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:07:46.346 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:07:46.346 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:07:46.346 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[29355]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:07:46.346 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:07:46.346 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:07:46.346 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:07:46.346 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:46 smithi102 sudo[61139]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-13T14:07:46.347 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:46 smithi102 sudo[61139]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T14:07:46.347 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:46 smithi102 sudo[61139]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T14:07:46.347 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:46 smithi102 sudo[61139]: pam_unix(sudo:session): session closed for user root 2024-09-13T14:07:46.347 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[40543]: from='client.14298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:07:46.347 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[40543]: Detected new or changed devices on smithi102 2024-09-13T14:07:46.347 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[40543]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T14:07:46.347 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:07:46.347 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:07:46.347 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:07:46.347 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[40543]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T14:07:46.347 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:07:46.347 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:07:46.347 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:07:46.348 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:46.348 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[40543]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T14:07:46.348 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:46.348 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:07:46.348 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:07:46.348 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:07:46.348 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[40543]: Adjusting osd_memory_target on smithi102 to 5354M 2024-09-13T14:07:46.348 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:46.348 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:07:46.348 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:07:46.348 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:46.348 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:46.349 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:07:46.349 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:07:46.349 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:07:46.349 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[40543]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:07:46.349 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:07:46.349 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:07:46.349 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:46 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:07:46.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:46 smithi195 ceph-mon[33205]: from='client.14298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:07:46.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:46 smithi195 ceph-mon[33205]: Detected new or changed devices on smithi102 2024-09-13T14:07:46.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:46 smithi195 ceph-mon[33205]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T14:07:46.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:46 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:07:46.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:46 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:07:46.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:46 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:07:46.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:46 smithi195 ceph-mon[33205]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T14:07:46.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:46 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:07:46.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:46 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:07:46.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:46 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:07:46.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:46 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:46.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:46 smithi195 ceph-mon[33205]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T14:07:46.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:46 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:46.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:46 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:07:46.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:46 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:07:46.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:46 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:07:46.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:46 smithi195 ceph-mon[33205]: Adjusting osd_memory_target on smithi102 to 5354M 2024-09-13T14:07:46.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:46 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:46.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:46 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:07:46.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:46 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:07:46.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:46 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:46.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:46 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:46.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:46 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:07:46.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:46 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:07:46.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:46 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:07:46.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:46 smithi195 ceph-mon[33205]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:07:46.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:46 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:07:46.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:46 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:07:46.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:46 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:07:47.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:47 smithi102 ceph-mon[29355]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T14:07:47.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:47 smithi102 ceph-mon[29355]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T14:07:47.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:47 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:47.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:47 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/3637941633' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2ed85faf-6610-4bcc-a09d-4a6f93f5c6a0"}]: dispatch 2024-09-13T14:07:47.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:47 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/3637941633' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2ed85faf-6610-4bcc-a09d-4a6f93f5c6a0"}]': finished 2024-09-13T14:07:47.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:47 smithi102 ceph-mon[29355]: osdmap e22: 4 total, 3 up, 4 in 2024-09-13T14:07:47.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:47 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:07:47.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:47 smithi102 ceph-mon[40543]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T14:07:47.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:47 smithi102 ceph-mon[40543]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T14:07:47.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:47 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:47.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:47 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/3637941633' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2ed85faf-6610-4bcc-a09d-4a6f93f5c6a0"}]: dispatch 2024-09-13T14:07:47.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:47 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/3637941633' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2ed85faf-6610-4bcc-a09d-4a6f93f5c6a0"}]': finished 2024-09-13T14:07:47.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:47 smithi102 ceph-mon[40543]: osdmap e22: 4 total, 3 up, 4 in 2024-09-13T14:07:47.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:47 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:07:47.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:47 smithi195 ceph-mon[33205]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T14:07:47.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:47 smithi195 ceph-mon[33205]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T14:07:47.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:47 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:47.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:47 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/3637941633' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2ed85faf-6610-4bcc-a09d-4a6f93f5c6a0"}]: dispatch 2024-09-13T14:07:47.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:47 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/3637941633' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2ed85faf-6610-4bcc-a09d-4a6f93f5c6a0"}]': finished 2024-09-13T14:07:47.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:47 smithi195 ceph-mon[33205]: osdmap e22: 4 total, 3 up, 4 in 2024-09-13T14:07:47.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:47 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:07:48.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:48 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/1864496046' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:07:48.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:48 smithi102 ceph-mon[40543]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:07:48.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:48 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/1864496046' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:07:48.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:48 smithi102 ceph-mon[29355]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:07:48.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:48 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/1864496046' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:07:48.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:48 smithi195 ceph-mon[33205]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:07:50.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:50 smithi102 ceph-mon[29355]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.1 KiB/s rd, 137 KiB/s wr, 9 op/s 2024-09-13T14:07:50.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:50 smithi102 ceph-mon[40543]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.1 KiB/s rd, 137 KiB/s wr, 9 op/s 2024-09-13T14:07:50.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:50 smithi195 ceph-mon[33205]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.1 KiB/s rd, 137 KiB/s wr, 9 op/s 2024-09-13T14:07:52.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:52 smithi195 ceph-mon[33205]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 2.3 KiB/s rd, 102 KiB/s wr, 7 op/s 2024-09-13T14:07:53.004 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:52 smithi102 ceph-mon[40543]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 2.3 KiB/s rd, 102 KiB/s wr, 7 op/s 2024-09-13T14:07:53.004 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:52 smithi102 ceph-mon[29355]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 2.3 KiB/s rd, 102 KiB/s wr, 7 op/s 2024-09-13T14:07:53.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:53 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-13T14:07:53.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:53 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:07:54.035 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:53 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-13T14:07:54.035 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:53 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:07:54.035 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:53 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-13T14:07:54.035 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:53 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:07:54.872 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:54 smithi102 ceph-mon[29355]: Deploying daemon osd.3 on smithi102 2024-09-13T14:07:54.872 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:54 smithi102 ceph-mon[29355]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.9 KiB/s rd, 81 KiB/s wr, 5 op/s 2024-09-13T14:07:54.872 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:54 smithi102 ceph-mon[40543]: Deploying daemon osd.3 on smithi102 2024-09-13T14:07:54.872 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:54 smithi102 ceph-mon[40543]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.9 KiB/s rd, 81 KiB/s wr, 5 op/s 2024-09-13T14:07:54.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:54 smithi195 ceph-mon[33205]: Deploying daemon osd.3 on smithi102 2024-09-13T14:07:54.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:54 smithi195 ceph-mon[33205]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.9 KiB/s rd, 81 KiB/s wr, 5 op/s 2024-09-13T14:07:56.824 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:56 smithi102 ceph-mon[40543]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-09-13T14:07:56.824 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:56 smithi102 ceph-mon[29355]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-09-13T14:07:56.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:56 smithi195 ceph-mon[33205]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-09-13T14:07:58.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:58 smithi102 ceph-mon[40543]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 74 KiB/s wr, 5 op/s 2024-09-13T14:07:58.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:58 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:07:58.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:58 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:58.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:07:58 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:58.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:58 smithi102 ceph-mon[29355]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 74 KiB/s wr, 5 op/s 2024-09-13T14:07:58.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:58 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:07:58.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:58 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:58.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:07:58 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:58.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:58 smithi195 ceph-mon[33205]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 74 KiB/s wr, 5 op/s 2024-09-13T14:07:58.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:58 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:07:58.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:58 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:07:58.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:07:58 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:00.816 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:00 smithi102 ceph-mon[29355]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-09-13T14:08:00.817 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:00 smithi102 ceph-mon[40543]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-09-13T14:08:00.845 INFO:teuthology.orchestra.run.smithi102.stdout:Created osd(s) 3 on host 'smithi102' 2024-09-13T14:08:00.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:00 smithi195 ceph-mon[33205]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-09-13T14:08:01.357 DEBUG:teuthology.orchestra.run.smithi102:osd.3> sudo journalctl -f -n 0 -u ceph-08cee502-71d9-11ef-bceb-c7b262605968@osd.3.service 2024-09-13T14:08:01.359 INFO:tasks.cephadm:Deploying osd.4 on smithi195 with /dev/vg_nvme/lv_4... 2024-09-13T14:08:01.359 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-13T14:08:01.565 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:08:01.937 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:01 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:01.937 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:01 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:01.937 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:01 smithi195 ceph-mon[33205]: from='osd.3 [v2:172.21.15.102:6826/3709228546,v1:172.21.15.102:6827/3709228546]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-13T14:08:01.937 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:01 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:01.937 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:01 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:02.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:01 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:02.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:01 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:02.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:01 smithi102 ceph-mon[40543]: from='osd.3 [v2:172.21.15.102:6826/3709228546,v1:172.21.15.102:6827/3709228546]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-13T14:08:02.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:01 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:02.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:01 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:02.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:01 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:02.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:01 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:02.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:01 smithi102 ceph-mon[29355]: from='osd.3 [v2:172.21.15.102:6826/3709228546,v1:172.21.15.102:6827/3709228546]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-13T14:08:02.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:01 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:02.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:01 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:03.100 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:02 smithi102 ceph-mon[29355]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:08:03.100 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:02 smithi102 ceph-mon[29355]: from='osd.3 [v2:172.21.15.102:6826/3709228546,v1:172.21.15.102:6827/3709228546]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-13T14:08:03.100 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:02 smithi102 ceph-mon[29355]: osdmap e23: 4 total, 3 up, 4 in 2024-09-13T14:08:03.101 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:02 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:08:03.101 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:02 smithi102 ceph-mon[29355]: from='osd.3 [v2:172.21.15.102:6826/3709228546,v1:172.21.15.102:6827/3709228546]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-09-13T14:08:03.101 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:02 smithi102 ceph-mon[40543]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:08:03.101 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:02 smithi102 ceph-mon[40543]: from='osd.3 [v2:172.21.15.102:6826/3709228546,v1:172.21.15.102:6827/3709228546]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-13T14:08:03.101 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:02 smithi102 ceph-mon[40543]: osdmap e23: 4 total, 3 up, 4 in 2024-09-13T14:08:03.101 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:02 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:08:03.101 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:02 smithi102 ceph-mon[40543]: from='osd.3 [v2:172.21.15.102:6826/3709228546,v1:172.21.15.102:6827/3709228546]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-09-13T14:08:03.235 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:02 smithi195 ceph-mon[33205]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:08:03.235 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:02 smithi195 ceph-mon[33205]: from='osd.3 [v2:172.21.15.102:6826/3709228546,v1:172.21.15.102:6827/3709228546]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-13T14:08:03.235 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:02 smithi195 ceph-mon[33205]: osdmap e23: 4 total, 3 up, 4 in 2024-09-13T14:08:03.235 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:02 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:08:03.235 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:02 smithi195 ceph-mon[33205]: from='osd.3 [v2:172.21.15.102:6826/3709228546,v1:172.21.15.102:6827/3709228546]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-09-13T14:08:03.404 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:08:03 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3[63044]: 2024-09-13T14:08:03.282+0000 7fcf04a05640 -1 osd.3 0 waiting for initial osdmap 2024-09-13T14:08:03.404 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:08:03 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3[63044]: 2024-09-13T14:08:03.286+0000 7fcefc826640 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T14:08:03.896 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:08:03.913 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:03 smithi195 ceph-mon[33205]: from='osd.3 [v2:172.21.15.102:6826/3709228546,v1:172.21.15.102:6827/3709228546]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-09-13T14:08:03.913 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:03 smithi195 ceph-mon[33205]: osdmap e24: 4 total, 3 up, 4 in 2024-09-13T14:08:03.913 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:03 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:08:03.913 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:03 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:08:03.913 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:03 smithi195 ceph-mon[33205]: from='osd.3 [v2:172.21.15.102:6826/3709228546,v1:172.21.15.102:6827/3709228546]' entity='osd.3' 2024-09-13T14:08:03.913 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:03 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:08:03.914 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi195:vg_nvme/lv_4 2024-09-13T14:08:04.108 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:08:04.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:03 smithi102 ceph-mon[29355]: from='osd.3 [v2:172.21.15.102:6826/3709228546,v1:172.21.15.102:6827/3709228546]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-09-13T14:08:04.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:03 smithi102 ceph-mon[29355]: osdmap e24: 4 total, 3 up, 4 in 2024-09-13T14:08:04.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:03 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:08:04.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:03 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:08:04.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:03 smithi102 ceph-mon[29355]: from='osd.3 [v2:172.21.15.102:6826/3709228546,v1:172.21.15.102:6827/3709228546]' entity='osd.3' 2024-09-13T14:08:04.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:03 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:08:04.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:03 smithi102 ceph-mon[40543]: from='osd.3 [v2:172.21.15.102:6826/3709228546,v1:172.21.15.102:6827/3709228546]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-09-13T14:08:04.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:03 smithi102 ceph-mon[40543]: osdmap e24: 4 total, 3 up, 4 in 2024-09-13T14:08:04.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:03 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:08:04.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:03 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:08:04.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:03 smithi102 ceph-mon[40543]: from='osd.3 [v2:172.21.15.102:6826/3709228546,v1:172.21.15.102:6827/3709228546]' entity='osd.3' 2024-09-13T14:08:04.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:03 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:08:05.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:04 smithi102 ceph-mon[40543]: purged_snaps scrub starts 2024-09-13T14:08:05.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:04 smithi102 ceph-mon[40543]: purged_snaps scrub ok 2024-09-13T14:08:05.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:04 smithi102 ceph-mon[40543]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:08:05.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:04 smithi102 ceph-mon[40543]: osd.3 [v2:172.21.15.102:6826/3709228546,v1:172.21.15.102:6827/3709228546] boot 2024-09-13T14:08:05.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:04 smithi102 ceph-mon[40543]: osdmap e25: 4 total, 4 up, 4 in 2024-09-13T14:08:05.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:04 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:08:05.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:04 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:05.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:04 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:05.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:04 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:08:05.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:04 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:08:05.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:04 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:08:05.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:04 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:08:05.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:04 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:05.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:04 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:08:05.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:04 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:08:05.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:04 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:05.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:04 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:08:05.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:04 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:08:05.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:04 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:08:05.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:04 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:05.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:04 smithi102 ceph-mon[29355]: purged_snaps scrub starts 2024-09-13T14:08:05.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:04 smithi102 ceph-mon[29355]: purged_snaps scrub ok 2024-09-13T14:08:05.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:04 smithi102 ceph-mon[29355]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:08:05.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:04 smithi102 ceph-mon[29355]: osd.3 [v2:172.21.15.102:6826/3709228546,v1:172.21.15.102:6827/3709228546] boot 2024-09-13T14:08:05.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:04 smithi102 ceph-mon[29355]: osdmap e25: 4 total, 4 up, 4 in 2024-09-13T14:08:05.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:04 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:08:05.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:04 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:05.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:04 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:05.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:04 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:08:05.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:04 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:08:05.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:04 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:08:05.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:04 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:08:05.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:04 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:05.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:04 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:08:05.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:04 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:08:05.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:04 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:05.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:04 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:08:05.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:04 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:08:05.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:04 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:08:05.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:04 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:05.224 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:04 smithi195 ceph-mon[33205]: purged_snaps scrub starts 2024-09-13T14:08:05.224 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:04 smithi195 ceph-mon[33205]: purged_snaps scrub ok 2024-09-13T14:08:05.224 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:04 smithi195 ceph-mon[33205]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:08:05.224 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:04 smithi195 ceph-mon[33205]: osd.3 [v2:172.21.15.102:6826/3709228546,v1:172.21.15.102:6827/3709228546] boot 2024-09-13T14:08:05.224 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:04 smithi195 ceph-mon[33205]: osdmap e25: 4 total, 4 up, 4 in 2024-09-13T14:08:05.225 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:04 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:08:05.225 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:04 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:05.225 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:04 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:05.225 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:04 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:08:05.225 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:04 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:08:05.225 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:04 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:08:05.225 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:04 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:08:05.225 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:04 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:05.225 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:04 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:08:05.225 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:04 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:08:05.225 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:04 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:05.226 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:04 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:08:05.226 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:04 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:08:05.226 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:04 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:08:05.226 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:04 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:06.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:05 smithi102 ceph-mon[29355]: Detected new or changed devices on smithi102 2024-09-13T14:08:06.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:05 smithi102 ceph-mon[29355]: Adjusting osd_memory_target on smithi102 to 4016M 2024-09-13T14:08:06.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:05 smithi102 ceph-mon[40543]: Detected new or changed devices on smithi102 2024-09-13T14:08:06.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:05 smithi102 ceph-mon[40543]: Adjusting osd_memory_target on smithi102 to 4016M 2024-09-13T14:08:06.228 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:05 smithi195 ceph-mon[33205]: Detected new or changed devices on smithi102 2024-09-13T14:08:06.229 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:05 smithi195 ceph-mon[33205]: Adjusting osd_memory_target on smithi102 to 4016M 2024-09-13T14:08:07.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:06 smithi102 ceph-mon[29355]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:08:07.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:06 smithi102 ceph-mon[29355]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:08:07.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:06 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:08:07.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:06 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:08:07.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:06 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:08:07.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:06 smithi102 ceph-mon[29355]: osdmap e26: 4 total, 4 up, 4 in 2024-09-13T14:08:07.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:06 smithi102 ceph-mon[40543]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:08:07.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:06 smithi102 ceph-mon[40543]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:08:07.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:06 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:08:07.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:06 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:08:07.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:06 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:08:07.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:06 smithi102 ceph-mon[40543]: osdmap e26: 4 total, 4 up, 4 in 2024-09-13T14:08:07.174 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:06 smithi195 ceph-mon[33205]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:08:07.174 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:06 smithi195 ceph-mon[33205]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:08:07.174 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:06 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:08:07.174 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:06 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:08:07.175 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:06 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:08:07.175 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:06 smithi195 ceph-mon[33205]: osdmap e26: 4 total, 4 up, 4 in 2024-09-13T14:08:08.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:08 smithi195 ceph-mon[33205]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:08:08.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:08 smithi195 ceph-mon[33205]: from='client.? 172.21.15.195:0/4260345920' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8a45f350-fb3e-4049-bcc4-532d3e20c238"}]: dispatch 2024-09-13T14:08:08.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:08 smithi195 ceph-mon[33205]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8a45f350-fb3e-4049-bcc4-532d3e20c238"}]: dispatch 2024-09-13T14:08:08.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:08 smithi195 ceph-mon[33205]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8a45f350-fb3e-4049-bcc4-532d3e20c238"}]': finished 2024-09-13T14:08:08.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:08 smithi195 ceph-mon[33205]: osdmap e27: 5 total, 4 up, 5 in 2024-09-13T14:08:08.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:08 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:08:08.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:08 smithi195 ceph-mon[33205]: from='client.? 172.21.15.195:0/2807305829' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:08:09.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:08 smithi102 ceph-mon[40543]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:08:09.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:08 smithi102 ceph-mon[40543]: from='client.? 172.21.15.195:0/4260345920' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8a45f350-fb3e-4049-bcc4-532d3e20c238"}]: dispatch 2024-09-13T14:08:09.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:08 smithi102 ceph-mon[40543]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8a45f350-fb3e-4049-bcc4-532d3e20c238"}]: dispatch 2024-09-13T14:08:09.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:08 smithi102 ceph-mon[40543]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8a45f350-fb3e-4049-bcc4-532d3e20c238"}]': finished 2024-09-13T14:08:09.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:08 smithi102 ceph-mon[40543]: osdmap e27: 5 total, 4 up, 5 in 2024-09-13T14:08:09.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:08 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:08:09.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:08 smithi102 ceph-mon[40543]: from='client.? 172.21.15.195:0/2807305829' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:08:09.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:08 smithi102 ceph-mon[29355]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:08:09.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:08 smithi102 ceph-mon[29355]: from='client.? 172.21.15.195:0/4260345920' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8a45f350-fb3e-4049-bcc4-532d3e20c238"}]: dispatch 2024-09-13T14:08:09.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:08 smithi102 ceph-mon[29355]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8a45f350-fb3e-4049-bcc4-532d3e20c238"}]: dispatch 2024-09-13T14:08:09.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:08 smithi102 ceph-mon[29355]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8a45f350-fb3e-4049-bcc4-532d3e20c238"}]': finished 2024-09-13T14:08:09.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:08 smithi102 ceph-mon[29355]: osdmap e27: 5 total, 4 up, 5 in 2024-09-13T14:08:09.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:08 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:08:09.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:08 smithi102 ceph-mon[29355]: from='client.? 172.21.15.195:0/2807305829' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:08:11.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:10 smithi102 ceph-mon[29355]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:08:11.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:10 smithi102 ceph-mon[40543]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:08:11.182 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:10 smithi195 ceph-mon[33205]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:08:13.146 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:12 smithi195 ceph-mon[33205]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:08:13.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:12 smithi102 ceph-mon[40543]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:08:13.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:12 smithi102 ceph-mon[29355]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:08:14.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:14 smithi195 ceph-mon[33205]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:08:14.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:14 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-13T14:08:14.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:14 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:08:15.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:14 smithi102 ceph-mon[29355]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:08:15.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:14 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-13T14:08:15.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:14 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:08:15.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:14 smithi102 ceph-mon[40543]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:08:15.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:14 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-13T14:08:15.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:14 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:08:15.968 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:15 smithi195 ceph-mon[33205]: Deploying daemon osd.4 on smithi195 2024-09-13T14:08:16.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:15 smithi102 ceph-mon[29355]: Deploying daemon osd.4 on smithi195 2024-09-13T14:08:16.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:15 smithi102 ceph-mon[40543]: Deploying daemon osd.4 on smithi195 2024-09-13T14:08:17.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:16 smithi102 ceph-mon[29355]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:08:17.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:16 smithi102 ceph-mon[40543]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:08:17.219 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:16 smithi195 ceph-mon[33205]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:08:19.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:18 smithi102 ceph-mon[29355]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:08:19.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:18 smithi102 ceph-mon[40543]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:08:19.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:18 smithi195 ceph-mon[33205]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:08:20.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:19 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:08:20.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:19 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:20.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:19 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:20.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:19 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:08:20.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:19 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:20.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:19 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:20.209 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:19 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:08:20.209 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:19 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:20.209 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:19 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:21.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:20 smithi102 ceph-mon[29355]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:08:21.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:20 smithi102 ceph-mon[40543]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:08:21.222 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:20 smithi195 ceph-mon[33205]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:08:21.775 INFO:teuthology.orchestra.run.smithi195.stdout:Created osd(s) 4 on host 'smithi195' 2024-09-13T14:08:22.413 DEBUG:teuthology.orchestra.run.smithi195:osd.4> sudo journalctl -f -n 0 -u ceph-08cee502-71d9-11ef-bceb-c7b262605968@osd.4.service 2024-09-13T14:08:22.415 INFO:tasks.cephadm:Deploying osd.5 on smithi195 with /dev/vg_nvme/lv_3... 2024-09-13T14:08:22.415 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-13T14:08:22.650 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:08:23.026 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:22 smithi195 ceph-mon[33205]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:08:23.026 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:22 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:23.026 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:22 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:23.026 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:22 smithi195 ceph-mon[33205]: from='osd.4 [v2:172.21.15.195:6800/2821651815,v1:172.21.15.195:6801/2821651815]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-13T14:08:23.026 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:22 smithi195 ceph-mon[33205]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-13T14:08:23.027 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:22 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:23.027 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:22 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:23.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:22 smithi102 ceph-mon[29355]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:08:23.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:22 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:23.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:22 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:23.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:22 smithi102 ceph-mon[29355]: from='osd.4 [v2:172.21.15.195:6800/2821651815,v1:172.21.15.195:6801/2821651815]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-13T14:08:23.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:22 smithi102 ceph-mon[29355]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-13T14:08:23.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:22 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:23.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:22 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:23.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:22 smithi102 ceph-mon[40543]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:08:23.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:22 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:23.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:22 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:23.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:22 smithi102 ceph-mon[40543]: from='osd.4 [v2:172.21.15.195:6800/2821651815,v1:172.21.15.195:6801/2821651815]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-13T14:08:23.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:22 smithi102 ceph-mon[40543]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-13T14:08:23.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:22 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:23.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:22 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:23.972 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:23 smithi195 ceph-mon[33205]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-13T14:08:23.972 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:23 smithi195 ceph-mon[33205]: osdmap e28: 5 total, 4 up, 5 in 2024-09-13T14:08:23.972 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:23 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:08:23.972 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:23 smithi195 ceph-mon[33205]: from='osd.4 [v2:172.21.15.195:6800/2821651815,v1:172.21.15.195:6801/2821651815]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-09-13T14:08:23.972 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:23 smithi195 ceph-mon[33205]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-09-13T14:08:23.972 INFO:journalctl@ceph.osd.4.smithi195.stdout:Sep 13 14:08:23 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-4[38505]: 2024-09-13T14:08:23.920+0000 7fb686a7c640 -1 osd.4 0 waiting for initial osdmap 2024-09-13T14:08:23.973 INFO:journalctl@ceph.osd.4.smithi195.stdout:Sep 13 14:08:23 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-4[38505]: 2024-09-13T14:08:23.923+0000 7fb6828a5640 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T14:08:24.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:23 smithi102 ceph-mon[29355]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-13T14:08:24.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:23 smithi102 ceph-mon[29355]: osdmap e28: 5 total, 4 up, 5 in 2024-09-13T14:08:24.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:23 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:08:24.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:23 smithi102 ceph-mon[29355]: from='osd.4 [v2:172.21.15.195:6800/2821651815,v1:172.21.15.195:6801/2821651815]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-09-13T14:08:24.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:23 smithi102 ceph-mon[29355]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-09-13T14:08:24.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:23 smithi102 ceph-mon[40543]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-13T14:08:24.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:23 smithi102 ceph-mon[40543]: osdmap e28: 5 total, 4 up, 5 in 2024-09-13T14:08:24.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:23 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:08:24.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:23 smithi102 ceph-mon[40543]: from='osd.4 [v2:172.21.15.195:6800/2821651815,v1:172.21.15.195:6801/2821651815]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-09-13T14:08:24.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:23 smithi102 ceph-mon[40543]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-09-13T14:08:25.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:24 smithi195 ceph-mon[33205]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:08:25.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:24 smithi195 ceph-mon[33205]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-09-13T14:08:25.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:24 smithi195 ceph-mon[33205]: osdmap e29: 5 total, 4 up, 5 in 2024-09-13T14:08:25.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:24 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:08:25.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:24 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:08:25.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:24 smithi195 ceph-mon[33205]: osd.4 [v2:172.21.15.195:6800/2821651815,v1:172.21.15.195:6801/2821651815] boot 2024-09-13T14:08:25.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:24 smithi195 ceph-mon[33205]: osdmap e30: 5 total, 5 up, 5 in 2024-09-13T14:08:25.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:24 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:08:25.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:24 smithi102 ceph-mon[29355]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:08:25.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:24 smithi102 ceph-mon[29355]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-09-13T14:08:25.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:24 smithi102 ceph-mon[29355]: osdmap e29: 5 total, 4 up, 5 in 2024-09-13T14:08:25.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:24 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:08:25.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:24 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:08:25.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:24 smithi102 ceph-mon[29355]: osd.4 [v2:172.21.15.195:6800/2821651815,v1:172.21.15.195:6801/2821651815] boot 2024-09-13T14:08:25.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:24 smithi102 ceph-mon[29355]: osdmap e30: 5 total, 5 up, 5 in 2024-09-13T14:08:25.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:24 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:08:25.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:24 smithi102 ceph-mon[40543]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:08:25.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:24 smithi102 ceph-mon[40543]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-09-13T14:08:25.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:24 smithi102 ceph-mon[40543]: osdmap e29: 5 total, 4 up, 5 in 2024-09-13T14:08:25.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:24 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:08:25.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:24 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:08:25.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:24 smithi102 ceph-mon[40543]: osd.4 [v2:172.21.15.195:6800/2821651815,v1:172.21.15.195:6801/2821651815] boot 2024-09-13T14:08:25.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:24 smithi102 ceph-mon[40543]: osdmap e30: 5 total, 5 up, 5 in 2024-09-13T14:08:25.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:24 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:08:26.014 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:08:26.036 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi195:vg_nvme/lv_3 2024-09-13T14:08:26.217 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:25 smithi195 ceph-mon[33205]: purged_snaps scrub starts 2024-09-13T14:08:26.217 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:25 smithi195 ceph-mon[33205]: purged_snaps scrub ok 2024-09-13T14:08:26.217 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:25 smithi195 ceph-mon[33205]: pgmap v104: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T14:08:26.217 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:25 smithi195 ceph-mon[33205]: osdmap e31: 5 total, 5 up, 5 in 2024-09-13T14:08:26.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:25 smithi102 ceph-mon[29355]: purged_snaps scrub starts 2024-09-13T14:08:26.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:25 smithi102 ceph-mon[29355]: purged_snaps scrub ok 2024-09-13T14:08:26.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:25 smithi102 ceph-mon[29355]: pgmap v104: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T14:08:26.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:25 smithi102 ceph-mon[29355]: osdmap e31: 5 total, 5 up, 5 in 2024-09-13T14:08:26.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:25 smithi102 ceph-mon[40543]: purged_snaps scrub starts 2024-09-13T14:08:26.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:25 smithi102 ceph-mon[40543]: purged_snaps scrub ok 2024-09-13T14:08:26.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:25 smithi102 ceph-mon[40543]: pgmap v104: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T14:08:26.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:25 smithi102 ceph-mon[40543]: osdmap e31: 5 total, 5 up, 5 in 2024-09-13T14:08:26.602 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:08:28.160 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:27 smithi195 ceph-mon[33205]: osdmap e32: 5 total, 5 up, 5 in 2024-09-13T14:08:28.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:27 smithi102 ceph-mon[29355]: osdmap e32: 5 total, 5 up, 5 in 2024-09-13T14:08:28.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:27 smithi102 ceph-mon[40543]: osdmap e32: 5 total, 5 up, 5 in 2024-09-13T14:08:29.119 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:28 smithi195 ceph-mon[33205]: pgmap v107: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T14:08:29.119 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:28 smithi195 ceph-mon[33205]: osdmap e33: 5 total, 5 up, 5 in 2024-09-13T14:08:29.119 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:28 smithi195 ceph-mon[33205]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:08:29.119 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:28 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:08:29.119 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:28 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:08:29.119 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:28 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:08:29.119 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:28 smithi195 ceph-mon[33205]: Detected new or changed devices on smithi195 2024-09-13T14:08:29.119 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:28 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:29.120 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:28 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:29.120 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:28 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:08:29.120 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:28 smithi195 ceph-mon[33205]: Adjusting osd_memory_target on smithi195 to 17088M 2024-09-13T14:08:29.120 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:28 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:29.120 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:28 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:08:29.120 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:28 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:08:29.120 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:28 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:29.120 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:28 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:08:29.120 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:28 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:08:29.121 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:28 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:08:29.121 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:28 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:29.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:28 smithi102 ceph-mon[29355]: pgmap v107: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T14:08:29.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:28 smithi102 ceph-mon[29355]: osdmap e33: 5 total, 5 up, 5 in 2024-09-13T14:08:29.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:28 smithi102 ceph-mon[29355]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:08:29.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:28 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:08:29.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:28 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:08:29.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:28 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:08:29.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:28 smithi102 ceph-mon[29355]: Detected new or changed devices on smithi195 2024-09-13T14:08:29.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:28 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:29.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:28 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:29.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:28 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:08:29.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:28 smithi102 ceph-mon[29355]: Adjusting osd_memory_target on smithi195 to 17088M 2024-09-13T14:08:29.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:28 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:29.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:28 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:08:29.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:28 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:08:29.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:28 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:29.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:28 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:08:29.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:28 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:08:29.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:28 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:08:29.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:28 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:29.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:28 smithi102 ceph-mon[40543]: pgmap v107: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T14:08:29.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:28 smithi102 ceph-mon[40543]: osdmap e33: 5 total, 5 up, 5 in 2024-09-13T14:08:29.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:28 smithi102 ceph-mon[40543]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:08:29.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:28 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:08:29.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:28 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:08:29.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:28 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:08:29.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:28 smithi102 ceph-mon[40543]: Detected new or changed devices on smithi195 2024-09-13T14:08:29.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:28 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:29.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:28 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:29.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:28 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:08:29.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:28 smithi102 ceph-mon[40543]: Adjusting osd_memory_target on smithi195 to 17088M 2024-09-13T14:08:29.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:28 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:29.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:28 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:08:29.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:28 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:08:29.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:28 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:29.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:28 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:08:29.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:28 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:08:29.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:28 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:08:29.422 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:28 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:30.016 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:29 smithi195 ceph-mon[33205]: osdmap e34: 5 total, 5 up, 5 in 2024-09-13T14:08:30.016 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:29 smithi195 ceph-mon[33205]: pgmap v110: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-13T14:08:30.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:29 smithi102 ceph-mon[29355]: osdmap e34: 5 total, 5 up, 5 in 2024-09-13T14:08:30.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:29 smithi102 ceph-mon[29355]: pgmap v110: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-13T14:08:30.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:29 smithi102 ceph-mon[40543]: osdmap e34: 5 total, 5 up, 5 in 2024-09-13T14:08:30.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:29 smithi102 ceph-mon[40543]: pgmap v110: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-13T14:08:31.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:30 smithi195 ceph-mon[33205]: from='client.? 172.21.15.195:0/3096342641' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "363cf638-db26-4c86-b2be-e075fd09ac16"}]: dispatch 2024-09-13T14:08:31.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:30 smithi195 ceph-mon[33205]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "363cf638-db26-4c86-b2be-e075fd09ac16"}]: dispatch 2024-09-13T14:08:31.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:30 smithi195 ceph-mon[33205]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "363cf638-db26-4c86-b2be-e075fd09ac16"}]': finished 2024-09-13T14:08:31.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:30 smithi195 ceph-mon[33205]: osdmap e35: 6 total, 5 up, 6 in 2024-09-13T14:08:31.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:30 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:08:31.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:30 smithi102 ceph-mon[40543]: from='client.? 172.21.15.195:0/3096342641' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "363cf638-db26-4c86-b2be-e075fd09ac16"}]: dispatch 2024-09-13T14:08:31.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:30 smithi102 ceph-mon[40543]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "363cf638-db26-4c86-b2be-e075fd09ac16"}]: dispatch 2024-09-13T14:08:31.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:30 smithi102 ceph-mon[40543]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "363cf638-db26-4c86-b2be-e075fd09ac16"}]': finished 2024-09-13T14:08:31.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:30 smithi102 ceph-mon[40543]: osdmap e35: 6 total, 5 up, 6 in 2024-09-13T14:08:31.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:30 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:08:31.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:30 smithi102 ceph-mon[29355]: from='client.? 172.21.15.195:0/3096342641' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "363cf638-db26-4c86-b2be-e075fd09ac16"}]: dispatch 2024-09-13T14:08:31.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:30 smithi102 ceph-mon[29355]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "363cf638-db26-4c86-b2be-e075fd09ac16"}]: dispatch 2024-09-13T14:08:31.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:30 smithi102 ceph-mon[29355]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "363cf638-db26-4c86-b2be-e075fd09ac16"}]': finished 2024-09-13T14:08:31.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:30 smithi102 ceph-mon[29355]: osdmap e35: 6 total, 5 up, 6 in 2024-09-13T14:08:31.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:30 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:08:32.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:31 smithi195 ceph-mon[33205]: from='client.? 172.21.15.195:0/3787534676' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:08:32.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:31 smithi195 ceph-mon[33205]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 121 KiB/s, 0 objects/s recovering 2024-09-13T14:08:32.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:31 smithi102 ceph-mon[29355]: from='client.? 172.21.15.195:0/3787534676' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:08:32.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:31 smithi102 ceph-mon[29355]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 121 KiB/s, 0 objects/s recovering 2024-09-13T14:08:32.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:31 smithi102 ceph-mon[40543]: from='client.? 172.21.15.195:0/3787534676' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:08:32.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:31 smithi102 ceph-mon[40543]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 121 KiB/s, 0 objects/s recovering 2024-09-13T14:08:34.977 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:34 smithi195 ceph-mon[33205]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-13T14:08:35.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:34 smithi102 ceph-mon[29355]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-13T14:08:35.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:34 smithi102 ceph-mon[40543]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-13T14:08:36.804 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:36 smithi195 ceph-mon[33205]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-09-13T14:08:37.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:36 smithi102 ceph-mon[29355]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-09-13T14:08:37.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:36 smithi102 ceph-mon[40543]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-09-13T14:08:37.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:37 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-13T14:08:37.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:37 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:08:37.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:37 smithi195 ceph-mon[33205]: Deploying daemon osd.5 on smithi195 2024-09-13T14:08:38.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:37 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-13T14:08:38.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:37 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:08:38.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:37 smithi102 ceph-mon[29355]: Deploying daemon osd.5 on smithi195 2024-09-13T14:08:38.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:37 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-13T14:08:38.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:37 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:08:38.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:37 smithi102 ceph-mon[40543]: Deploying daemon osd.5 on smithi195 2024-09-13T14:08:38.967 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:38 smithi195 ceph-mon[33205]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-09-13T14:08:39.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:38 smithi102 ceph-mon[29355]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-09-13T14:08:39.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:38 smithi102 ceph-mon[40543]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-09-13T14:08:40.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:40 smithi195 ceph-mon[33205]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T14:08:41.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:40 smithi102 ceph-mon[29355]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T14:08:41.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:40 smithi102 ceph-mon[40543]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T14:08:41.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:41 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:08:41.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:41 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:41.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:41 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:42.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:41 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:08:42.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:41 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:42.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:41 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:42.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:41 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:08:42.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:41 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:42.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:41 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:42.910 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:42 smithi195 ceph-mon[33205]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T14:08:43.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:42 smithi102 ceph-mon[40543]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T14:08:43.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:42 smithi102 ceph-mon[29355]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T14:08:44.920 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:44 smithi195 ceph-mon[33205]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T14:08:44.920 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:44 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:44.921 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:44 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:44.921 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:44 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:08:44.921 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:44 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:08:44.921 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:44 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:44.921 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:44 smithi195 ceph-mon[33205]: from='osd.5 [v2:172.21.15.195:6808/3074139237,v1:172.21.15.195:6809/3074139237]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-13T14:08:44.921 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:44 smithi195 ceph-mon[33205]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-13T14:08:44.956 INFO:teuthology.orchestra.run.smithi195.stdout:Created osd(s) 5 on host 'smithi195' 2024-09-13T14:08:45.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:44 smithi102 ceph-mon[40543]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T14:08:45.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:44 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:45.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:44 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:45.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:44 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:08:45.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:44 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:08:45.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:44 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:45.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:44 smithi102 ceph-mon[40543]: from='osd.5 [v2:172.21.15.195:6808/3074139237,v1:172.21.15.195:6809/3074139237]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-13T14:08:45.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:44 smithi102 ceph-mon[40543]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-13T14:08:45.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:44 smithi102 ceph-mon[29355]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T14:08:45.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:44 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:45.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:44 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:45.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:44 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:08:45.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:44 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:08:45.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:44 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:45.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:44 smithi102 ceph-mon[29355]: from='osd.5 [v2:172.21.15.195:6808/3074139237,v1:172.21.15.195:6809/3074139237]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-13T14:08:45.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:44 smithi102 ceph-mon[29355]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-13T14:08:45.582 DEBUG:teuthology.orchestra.run.smithi195:osd.5> sudo journalctl -f -n 0 -u ceph-08cee502-71d9-11ef-bceb-c7b262605968@osd.5.service 2024-09-13T14:08:45.585 INFO:tasks.cephadm:Deploying osd.6 on smithi195 with /dev/vg_nvme/lv_2... 2024-09-13T14:08:45.585 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-13T14:08:45.844 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:45 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:08:45.844 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:45 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:45.844 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:45 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:45.845 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:45 smithi195 ceph-mon[33205]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-13T14:08:45.845 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:45 smithi195 ceph-mon[33205]: osdmap e36: 6 total, 5 up, 6 in 2024-09-13T14:08:45.845 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:45 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:08:45.845 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:45 smithi195 ceph-mon[33205]: from='osd.5 [v2:172.21.15.195:6808/3074139237,v1:172.21.15.195:6809/3074139237]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-09-13T14:08:45.845 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:45 smithi195 ceph-mon[33205]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-09-13T14:08:46.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:45 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:08:46.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:45 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:46.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:45 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:46.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:45 smithi102 ceph-mon[29355]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-13T14:08:46.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:45 smithi102 ceph-mon[29355]: osdmap e36: 6 total, 5 up, 6 in 2024-09-13T14:08:46.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:45 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:08:46.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:45 smithi102 ceph-mon[29355]: from='osd.5 [v2:172.21.15.195:6808/3074139237,v1:172.21.15.195:6809/3074139237]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-09-13T14:08:46.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:45 smithi102 ceph-mon[29355]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-09-13T14:08:46.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:45 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:08:46.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:45 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:46.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:45 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:46.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:45 smithi102 ceph-mon[40543]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-13T14:08:46.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:45 smithi102 ceph-mon[40543]: osdmap e36: 6 total, 5 up, 6 in 2024-09-13T14:08:46.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:45 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:08:46.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:45 smithi102 ceph-mon[40543]: from='osd.5 [v2:172.21.15.195:6808/3074139237,v1:172.21.15.195:6809/3074139237]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-09-13T14:08:46.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:45 smithi102 ceph-mon[40543]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-09-13T14:08:46.285 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:08:46.745 INFO:journalctl@ceph.osd.5.smithi195.stdout:Sep 13 14:08:46 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-5[44425]: 2024-09-13T14:08:46.440+0000 7fa74a762640 -1 osd.5 0 waiting for initial osdmap 2024-09-13T14:08:46.746 INFO:journalctl@ceph.osd.5.smithi195.stdout:Sep 13 14:08:46 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-5[44425]: 2024-09-13T14:08:46.444+0000 7fa742583640 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T14:08:46.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:46 smithi195 ceph-mon[33205]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T14:08:46.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:46 smithi195 ceph-mon[33205]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-09-13T14:08:46.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:46 smithi195 ceph-mon[33205]: osdmap e37: 6 total, 5 up, 6 in 2024-09-13T14:08:46.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:46 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:08:46.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:46 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:08:46.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:46 smithi195 ceph-mon[33205]: from='osd.5 ' entity='osd.5' 2024-09-13T14:08:47.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:46 smithi102 ceph-mon[29355]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T14:08:47.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:46 smithi102 ceph-mon[29355]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-09-13T14:08:47.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:46 smithi102 ceph-mon[29355]: osdmap e37: 6 total, 5 up, 6 in 2024-09-13T14:08:47.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:46 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:08:47.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:46 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:08:47.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:46 smithi102 ceph-mon[29355]: from='osd.5 ' entity='osd.5' 2024-09-13T14:08:47.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:46 smithi102 ceph-mon[40543]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T14:08:47.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:46 smithi102 ceph-mon[40543]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-09-13T14:08:47.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:46 smithi102 ceph-mon[40543]: osdmap e37: 6 total, 5 up, 6 in 2024-09-13T14:08:47.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:46 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:08:47.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:46 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:08:47.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:46 smithi102 ceph-mon[40543]: from='osd.5 ' entity='osd.5' 2024-09-13T14:08:48.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:48 smithi195 ceph-mon[33205]: purged_snaps scrub starts 2024-09-13T14:08:48.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:48 smithi195 ceph-mon[33205]: purged_snaps scrub ok 2024-09-13T14:08:48.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:48 smithi195 ceph-mon[33205]: osd.5 [v2:172.21.15.195:6808/3074139237,v1:172.21.15.195:6809/3074139237] boot 2024-09-13T14:08:48.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:48 smithi195 ceph-mon[33205]: osdmap e38: 6 total, 6 up, 6 in 2024-09-13T14:08:48.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:48 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:08:48.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:48 smithi195 ceph-mon[33205]: pgmap v123: 1 pgs: 1 unknown; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-13T14:08:48.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:48 smithi195 ceph-mon[33205]: osdmap e39: 6 total, 6 up, 6 in 2024-09-13T14:08:48.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:48 smithi102 ceph-mon[29355]: purged_snaps scrub starts 2024-09-13T14:08:48.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:48 smithi102 ceph-mon[29355]: purged_snaps scrub ok 2024-09-13T14:08:48.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:48 smithi102 ceph-mon[29355]: osd.5 [v2:172.21.15.195:6808/3074139237,v1:172.21.15.195:6809/3074139237] boot 2024-09-13T14:08:48.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:48 smithi102 ceph-mon[29355]: osdmap e38: 6 total, 6 up, 6 in 2024-09-13T14:08:48.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:48 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:08:48.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:48 smithi102 ceph-mon[29355]: pgmap v123: 1 pgs: 1 unknown; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-13T14:08:48.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:48 smithi102 ceph-mon[29355]: osdmap e39: 6 total, 6 up, 6 in 2024-09-13T14:08:48.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:48 smithi102 ceph-mon[40543]: purged_snaps scrub starts 2024-09-13T14:08:48.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:48 smithi102 ceph-mon[40543]: purged_snaps scrub ok 2024-09-13T14:08:48.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:48 smithi102 ceph-mon[40543]: osd.5 [v2:172.21.15.195:6808/3074139237,v1:172.21.15.195:6809/3074139237] boot 2024-09-13T14:08:48.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:48 smithi102 ceph-mon[40543]: osdmap e38: 6 total, 6 up, 6 in 2024-09-13T14:08:48.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:48 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:08:48.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:48 smithi102 ceph-mon[40543]: pgmap v123: 1 pgs: 1 unknown; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-13T14:08:48.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:48 smithi102 ceph-mon[40543]: osdmap e39: 6 total, 6 up, 6 in 2024-09-13T14:08:50.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:50 smithi102 ceph-mon[29355]: osdmap e40: 6 total, 6 up, 6 in 2024-09-13T14:08:50.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:50 smithi102 ceph-mon[29355]: Detected new or changed devices on smithi195 2024-09-13T14:08:50.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:50 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:50.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:50 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:50.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:50 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:08:50.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:50 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:08:50.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:50 smithi102 ceph-mon[29355]: Adjusting osd_memory_target on smithi195 to 8544M 2024-09-13T14:08:50.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:50 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:50.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:50 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:08:50.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:50 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:08:50.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:50 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:50.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:50 smithi102 ceph-mon[29355]: pgmap v126: 1 pgs: 1 unknown; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-13T14:08:50.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:50 smithi102 ceph-mon[40543]: osdmap e40: 6 total, 6 up, 6 in 2024-09-13T14:08:50.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:50 smithi102 ceph-mon[40543]: Detected new or changed devices on smithi195 2024-09-13T14:08:50.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:50 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:50.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:50 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:50.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:50 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:08:50.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:50 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:08:50.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:50 smithi102 ceph-mon[40543]: Adjusting osd_memory_target on smithi195 to 8544M 2024-09-13T14:08:50.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:50 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:50.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:50 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:08:50.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:50 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:08:50.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:50 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:50.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:50 smithi102 ceph-mon[40543]: pgmap v126: 1 pgs: 1 unknown; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-13T14:08:50.451 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:50 smithi195 ceph-mon[33205]: osdmap e40: 6 total, 6 up, 6 in 2024-09-13T14:08:50.451 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:50 smithi195 ceph-mon[33205]: Detected new or changed devices on smithi195 2024-09-13T14:08:50.451 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:50 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:50.451 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:50 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:50.452 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:50 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:08:50.452 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:50 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:08:50.452 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:50 smithi195 ceph-mon[33205]: Adjusting osd_memory_target on smithi195 to 8544M 2024-09-13T14:08:50.452 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:50 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:50.452 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:50 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:08:50.452 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:50 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:08:50.452 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:50 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:08:50.452 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:50 smithi195 ceph-mon[33205]: pgmap v126: 1 pgs: 1 unknown; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-13T14:08:50.722 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:08:50.740 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi195:vg_nvme/lv_2 2024-09-13T14:08:50.934 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:08:51.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:51 smithi102 ceph-mon[29355]: osdmap e41: 6 total, 6 up, 6 in 2024-09-13T14:08:51.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:51 smithi102 ceph-mon[40543]: osdmap e41: 6 total, 6 up, 6 in 2024-09-13T14:08:51.708 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:51 smithi195 ceph-mon[33205]: osdmap e41: 6 total, 6 up, 6 in 2024-09-13T14:08:52.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:52 smithi102 ceph-mon[29355]: osdmap e42: 6 total, 6 up, 6 in 2024-09-13T14:08:52.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:52 smithi102 ceph-mon[29355]: pgmap v129: 1 pgs: 1 unknown; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-13T14:08:52.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:52 smithi102 ceph-mon[40543]: osdmap e42: 6 total, 6 up, 6 in 2024-09-13T14:08:52.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:52 smithi102 ceph-mon[40543]: pgmap v129: 1 pgs: 1 unknown; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-13T14:08:52.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:52 smithi195 ceph-mon[33205]: osdmap e42: 6 total, 6 up, 6 in 2024-09-13T14:08:52.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:52 smithi195 ceph-mon[33205]: pgmap v129: 1 pgs: 1 unknown; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-13T14:08:53.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:53 smithi102 ceph-mon[29355]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:08:53.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:53 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:08:53.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:53 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:08:53.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:53 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:08:53.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:53 smithi102 ceph-mon[40543]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:08:53.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:53 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:08:53.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:53 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:08:53.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:53 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:08:53.708 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:53 smithi195 ceph-mon[33205]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:08:53.709 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:53 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:08:53.709 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:53 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:08:53.709 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:53 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:08:54.655 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:54 smithi195 ceph-mon[33205]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-09-13T14:08:54.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:54 smithi102 ceph-mon[29355]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-09-13T14:08:54.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:54 smithi102 ceph-mon[40543]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-09-13T14:08:55.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:55 smithi195 ceph-mon[33205]: from='client.? 172.21.15.195:0/840203244' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f37416e8-9c25-4860-a32f-0d1c404a8bc9"}]: dispatch 2024-09-13T14:08:55.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:55 smithi195 ceph-mon[33205]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f37416e8-9c25-4860-a32f-0d1c404a8bc9"}]: dispatch 2024-09-13T14:08:55.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:55 smithi195 ceph-mon[33205]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f37416e8-9c25-4860-a32f-0d1c404a8bc9"}]': finished 2024-09-13T14:08:55.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:55 smithi195 ceph-mon[33205]: osdmap e43: 7 total, 6 up, 7 in 2024-09-13T14:08:55.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:55 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:08:55.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:55 smithi102 ceph-mon[40543]: from='client.? 172.21.15.195:0/840203244' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f37416e8-9c25-4860-a32f-0d1c404a8bc9"}]: dispatch 2024-09-13T14:08:55.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:55 smithi102 ceph-mon[40543]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f37416e8-9c25-4860-a32f-0d1c404a8bc9"}]: dispatch 2024-09-13T14:08:55.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:55 smithi102 ceph-mon[40543]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f37416e8-9c25-4860-a32f-0d1c404a8bc9"}]': finished 2024-09-13T14:08:55.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:55 smithi102 ceph-mon[40543]: osdmap e43: 7 total, 6 up, 7 in 2024-09-13T14:08:55.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:55 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:08:55.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:55 smithi102 ceph-mon[29355]: from='client.? 172.21.15.195:0/840203244' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f37416e8-9c25-4860-a32f-0d1c404a8bc9"}]: dispatch 2024-09-13T14:08:55.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:55 smithi102 ceph-mon[29355]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f37416e8-9c25-4860-a32f-0d1c404a8bc9"}]: dispatch 2024-09-13T14:08:55.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:55 smithi102 ceph-mon[29355]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f37416e8-9c25-4860-a32f-0d1c404a8bc9"}]': finished 2024-09-13T14:08:55.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:55 smithi102 ceph-mon[29355]: osdmap e43: 7 total, 6 up, 7 in 2024-09-13T14:08:55.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:55 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:08:56.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:56 smithi102 ceph-mon[40543]: from='client.? 172.21.15.195:0/1820206401' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:08:56.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:56 smithi102 ceph-mon[40543]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-13T14:08:56.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:56 smithi102 ceph-mon[29355]: from='client.? 172.21.15.195:0/1820206401' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:08:56.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:56 smithi102 ceph-mon[29355]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-13T14:08:56.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:56 smithi195 ceph-mon[33205]: from='client.? 172.21.15.195:0/1820206401' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:08:56.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:56 smithi195 ceph-mon[33205]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-13T14:08:58.905 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:08:58 smithi195 ceph-mon[33205]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-09-13T14:08:59.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:08:58 smithi102 ceph-mon[29355]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-09-13T14:08:59.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:08:58 smithi102 ceph-mon[40543]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-09-13T14:09:00.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:00 smithi195 ceph-mon[33205]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-09-13T14:09:01.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:00 smithi102 ceph-mon[29355]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-09-13T14:09:01.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:00 smithi102 ceph-mon[40543]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-09-13T14:09:01.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:01 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-13T14:09:01.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:01 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:09:02.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:01 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-13T14:09:02.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:01 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:09:02.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:01 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-13T14:09:02.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:01 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:09:02.996 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:02 smithi195 ceph-mon[33205]: Deploying daemon osd.6 on smithi195 2024-09-13T14:09:02.996 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:02 smithi195 ceph-mon[33205]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T14:09:03.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:02 smithi102 ceph-mon[29355]: Deploying daemon osd.6 on smithi195 2024-09-13T14:09:03.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:02 smithi102 ceph-mon[29355]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T14:09:03.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:02 smithi102 ceph-mon[40543]: Deploying daemon osd.6 on smithi195 2024-09-13T14:09:03.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:02 smithi102 ceph-mon[40543]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T14:09:04.893 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:04 smithi195 ceph-mon[33205]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-13T14:09:05.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:04 smithi102 ceph-mon[29355]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-13T14:09:05.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:04 smithi102 ceph-mon[40543]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-13T14:09:06.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:06 smithi195 ceph-mon[33205]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-13T14:09:06.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:06 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:09:06.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:06 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:06.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:06 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:07.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:06 smithi102 ceph-mon[29355]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-13T14:09:07.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:06 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:09:07.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:06 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:07.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:06 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:07.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:06 smithi102 ceph-mon[40543]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-13T14:09:07.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:06 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:09:07.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:06 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:07.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:06 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:08.811 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:08 smithi195 ceph-mon[33205]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-13T14:09:08.811 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:08 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:08.811 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:08 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:08.811 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:08 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:09:08.811 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:08 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:09:08.811 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:08 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:09.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:08 smithi102 ceph-mon[40543]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-13T14:09:09.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:08 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:09.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:08 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:09.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:08 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:09:09.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:08 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:09:09.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:08 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:09.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:08 smithi102 ceph-mon[29355]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-13T14:09:09.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:08 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:09.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:08 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:09.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:08 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:09:09.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:08 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:09:09.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:08 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:09.408 INFO:teuthology.orchestra.run.smithi195.stdout:Created osd(s) 6 on host 'smithi195' 2024-09-13T14:09:09.969 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:09 smithi195 ceph-mon[33205]: from='osd.6 [v2:172.21.15.195:6816/4026627594,v1:172.21.15.195:6817/4026627594]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-13T14:09:09.970 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:09 smithi195 ceph-mon[33205]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-13T14:09:09.970 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:09 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:09:09.970 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:09 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:09.970 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:09 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:09.990 DEBUG:teuthology.orchestra.run.smithi195:osd.6> sudo journalctl -f -n 0 -u ceph-08cee502-71d9-11ef-bceb-c7b262605968@osd.6.service 2024-09-13T14:09:09.992 INFO:tasks.cephadm:Deploying osd.7 on smithi195 with /dev/vg_nvme/lv_1... 2024-09-13T14:09:09.993 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-13T14:09:10.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:09 smithi102 ceph-mon[29355]: from='osd.6 [v2:172.21.15.195:6816/4026627594,v1:172.21.15.195:6817/4026627594]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-13T14:09:10.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:09 smithi102 ceph-mon[29355]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-13T14:09:10.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:09 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:09:10.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:09 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:10.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:09 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:10.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:09 smithi102 ceph-mon[40543]: from='osd.6 [v2:172.21.15.195:6816/4026627594,v1:172.21.15.195:6817/4026627594]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-13T14:09:10.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:09 smithi102 ceph-mon[40543]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-13T14:09:10.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:09 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:09:10.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:09 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:10.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:09 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:10.942 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:10 smithi195 ceph-mon[33205]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-13T14:09:10.943 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:10 smithi195 ceph-mon[33205]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-13T14:09:10.943 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:10 smithi195 ceph-mon[33205]: osdmap e44: 7 total, 6 up, 7 in 2024-09-13T14:09:10.943 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:10 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:09:10.943 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:10 smithi195 ceph-mon[33205]: from='osd.6 [v2:172.21.15.195:6816/4026627594,v1:172.21.15.195:6817/4026627594]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-09-13T14:09:10.943 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:10 smithi195 ceph-mon[33205]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-09-13T14:09:11.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:10 smithi102 ceph-mon[29355]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-13T14:09:11.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:10 smithi102 ceph-mon[29355]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-13T14:09:11.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:10 smithi102 ceph-mon[29355]: osdmap e44: 7 total, 6 up, 7 in 2024-09-13T14:09:11.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:10 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:09:11.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:10 smithi102 ceph-mon[29355]: from='osd.6 [v2:172.21.15.195:6816/4026627594,v1:172.21.15.195:6817/4026627594]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-09-13T14:09:11.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:10 smithi102 ceph-mon[29355]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-09-13T14:09:11.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:10 smithi102 ceph-mon[40543]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-13T14:09:11.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:10 smithi102 ceph-mon[40543]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-13T14:09:11.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:10 smithi102 ceph-mon[40543]: osdmap e44: 7 total, 6 up, 7 in 2024-09-13T14:09:11.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:10 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:09:11.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:10 smithi102 ceph-mon[40543]: from='osd.6 [v2:172.21.15.195:6816/4026627594,v1:172.21.15.195:6817/4026627594]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-09-13T14:09:11.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:10 smithi102 ceph-mon[40543]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-09-13T14:09:11.247 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 14:09:11 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[50503]: 2024-09-13T14:09:11.002+0000 7fb39e22b640 -1 osd.6 0 waiting for initial osdmap 2024-09-13T14:09:11.247 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 14:09:11 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[50503]: 2024-09-13T14:09:11.010+0000 7fb395835640 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T14:09:11.289 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:09:12.032 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:11 smithi195 ceph-mon[33205]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-09-13T14:09:12.032 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:11 smithi195 ceph-mon[33205]: osdmap e45: 7 total, 6 up, 7 in 2024-09-13T14:09:12.032 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:11 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:09:12.032 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:11 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:09:12.032 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:11 smithi195 ceph-mon[33205]: from='osd.6 ' entity='osd.6' 2024-09-13T14:09:12.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:11 smithi102 ceph-mon[40543]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-09-13T14:09:12.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:11 smithi102 ceph-mon[40543]: osdmap e45: 7 total, 6 up, 7 in 2024-09-13T14:09:12.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:11 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:09:12.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:11 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:09:12.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:11 smithi102 ceph-mon[40543]: from='osd.6 ' entity='osd.6' 2024-09-13T14:09:12.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:11 smithi102 ceph-mon[29355]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-09-13T14:09:12.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:11 smithi102 ceph-mon[29355]: osdmap e45: 7 total, 6 up, 7 in 2024-09-13T14:09:12.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:11 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:09:12.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:11 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:09:12.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:11 smithi102 ceph-mon[29355]: from='osd.6 ' entity='osd.6' 2024-09-13T14:09:13.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:12 smithi102 ceph-mon[29355]: purged_snaps scrub starts 2024-09-13T14:09:13.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:12 smithi102 ceph-mon[29355]: purged_snaps scrub ok 2024-09-13T14:09:13.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:12 smithi102 ceph-mon[29355]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-13T14:09:13.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:12 smithi102 ceph-mon[29355]: osd.6 [v2:172.21.15.195:6816/4026627594,v1:172.21.15.195:6817/4026627594] boot 2024-09-13T14:09:13.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:12 smithi102 ceph-mon[29355]: osdmap e46: 7 total, 7 up, 7 in 2024-09-13T14:09:13.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:12 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:09:13.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:12 smithi102 ceph-mon[29355]: osdmap e47: 7 total, 7 up, 7 in 2024-09-13T14:09:13.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:12 smithi102 ceph-mon[40543]: purged_snaps scrub starts 2024-09-13T14:09:13.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:12 smithi102 ceph-mon[40543]: purged_snaps scrub ok 2024-09-13T14:09:13.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:12 smithi102 ceph-mon[40543]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-13T14:09:13.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:12 smithi102 ceph-mon[40543]: osd.6 [v2:172.21.15.195:6816/4026627594,v1:172.21.15.195:6817/4026627594] boot 2024-09-13T14:09:13.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:12 smithi102 ceph-mon[40543]: osdmap e46: 7 total, 7 up, 7 in 2024-09-13T14:09:13.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:12 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:09:13.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:12 smithi102 ceph-mon[40543]: osdmap e47: 7 total, 7 up, 7 in 2024-09-13T14:09:13.173 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:12 smithi195 ceph-mon[33205]: purged_snaps scrub starts 2024-09-13T14:09:13.173 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:12 smithi195 ceph-mon[33205]: purged_snaps scrub ok 2024-09-13T14:09:13.173 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:12 smithi195 ceph-mon[33205]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-13T14:09:13.173 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:12 smithi195 ceph-mon[33205]: osd.6 [v2:172.21.15.195:6816/4026627594,v1:172.21.15.195:6817/4026627594] boot 2024-09-13T14:09:13.173 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:12 smithi195 ceph-mon[33205]: osdmap e46: 7 total, 7 up, 7 in 2024-09-13T14:09:13.173 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:12 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:09:13.174 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:12 smithi195 ceph-mon[33205]: osdmap e47: 7 total, 7 up, 7 in 2024-09-13T14:09:14.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:14 smithi195 ceph-mon[33205]: Detected new or changed devices on smithi195 2024-09-13T14:09:14.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:14 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:14.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:14 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:14.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:14 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:09:14.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:14 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:09:14.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:14 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:09:14.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:14 smithi195 ceph-mon[33205]: Adjusting osd_memory_target on smithi195 to 5696M 2024-09-13T14:09:14.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:14 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:14.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:14 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:09:14.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:14 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:09:14.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:14 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:14.750 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:14 smithi195 ceph-mon[33205]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-13T14:09:14.750 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:14 smithi195 ceph-mon[33205]: osdmap e48: 7 total, 7 up, 7 in 2024-09-13T14:09:14.859 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-13T14:09:14.879 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi195:vg_nvme/lv_1 2024-09-13T14:09:14.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:14 smithi102 ceph-mon[29355]: Detected new or changed devices on smithi195 2024-09-13T14:09:14.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:14 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:14.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:14 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:14.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:14 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:09:14.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:14 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:09:14.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:14 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:09:14.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:14 smithi102 ceph-mon[29355]: Adjusting osd_memory_target on smithi195 to 5696M 2024-09-13T14:09:14.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:14 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:14.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:14 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:09:14.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:14 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:09:14.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:14 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:14.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:14 smithi102 ceph-mon[29355]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-13T14:09:14.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:14 smithi102 ceph-mon[29355]: osdmap e48: 7 total, 7 up, 7 in 2024-09-13T14:09:14.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:14 smithi102 ceph-mon[40543]: Detected new or changed devices on smithi195 2024-09-13T14:09:14.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:14 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:14.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:14 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:14.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:14 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:09:14.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:14 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:09:14.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:14 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:09:14.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:14 smithi102 ceph-mon[40543]: Adjusting osd_memory_target on smithi195 to 5696M 2024-09-13T14:09:14.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:14 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:14.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:14 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:09:14.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:14 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:09:14.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:14 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:14.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:14 smithi102 ceph-mon[40543]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-13T14:09:14.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:14 smithi102 ceph-mon[40543]: osdmap e48: 7 total, 7 up, 7 in 2024-09-13T14:09:15.076 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:09:15.963 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:15 smithi195 ceph-mon[33205]: osdmap e49: 7 total, 7 up, 7 in 2024-09-13T14:09:16.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:15 smithi102 ceph-mon[29355]: osdmap e49: 7 total, 7 up, 7 in 2024-09-13T14:09:16.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:15 smithi102 ceph-mon[40543]: osdmap e49: 7 total, 7 up, 7 in 2024-09-13T14:09:17.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:16 smithi102 ceph-mon[29355]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-13T14:09:17.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:16 smithi102 ceph-mon[29355]: osdmap e50: 7 total, 7 up, 7 in 2024-09-13T14:09:17.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:16 smithi102 ceph-mon[40543]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-13T14:09:17.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:16 smithi102 ceph-mon[40543]: osdmap e50: 7 total, 7 up, 7 in 2024-09-13T14:09:17.236 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:16 smithi195 ceph-mon[33205]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-13T14:09:17.237 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:16 smithi195 ceph-mon[33205]: osdmap e50: 7 total, 7 up, 7 in 2024-09-13T14:09:18.069 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:17 smithi102 ceph-mon[29355]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:09:18.070 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:17 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:09:18.070 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:17 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:09:18.070 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:17 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:09:18.070 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:17 smithi102 ceph-mon[40543]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:09:18.070 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:17 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:09:18.070 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:17 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:09:18.070 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:17 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:09:18.230 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:17 smithi195 ceph-mon[33205]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:09:18.230 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:17 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:09:18.230 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:17 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:09:18.230 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:17 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:09:19.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:18 smithi102 ceph-mon[29355]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-09-13T14:09:19.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:18 smithi102 ceph-mon[40543]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-09-13T14:09:19.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:18 smithi195 ceph-mon[33205]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-09-13T14:09:20.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:19 smithi102 ceph-mon[29355]: from='client.? 172.21.15.195:0/157200516' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9a384a4d-fddb-4fb3-843a-77c19cca0368"}]: dispatch 2024-09-13T14:09:20.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:19 smithi102 ceph-mon[29355]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9a384a4d-fddb-4fb3-843a-77c19cca0368"}]: dispatch 2024-09-13T14:09:20.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:19 smithi102 ceph-mon[29355]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9a384a4d-fddb-4fb3-843a-77c19cca0368"}]': finished 2024-09-13T14:09:20.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:19 smithi102 ceph-mon[29355]: osdmap e51: 8 total, 7 up, 8 in 2024-09-13T14:09:20.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:19 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:09:20.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:19 smithi102 ceph-mon[40543]: from='client.? 172.21.15.195:0/157200516' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9a384a4d-fddb-4fb3-843a-77c19cca0368"}]: dispatch 2024-09-13T14:09:20.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:19 smithi102 ceph-mon[40543]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9a384a4d-fddb-4fb3-843a-77c19cca0368"}]: dispatch 2024-09-13T14:09:20.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:19 smithi102 ceph-mon[40543]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9a384a4d-fddb-4fb3-843a-77c19cca0368"}]': finished 2024-09-13T14:09:20.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:19 smithi102 ceph-mon[40543]: osdmap e51: 8 total, 7 up, 8 in 2024-09-13T14:09:20.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:19 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:09:20.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:19 smithi195 ceph-mon[33205]: from='client.? 172.21.15.195:0/157200516' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9a384a4d-fddb-4fb3-843a-77c19cca0368"}]: dispatch 2024-09-13T14:09:20.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:19 smithi195 ceph-mon[33205]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9a384a4d-fddb-4fb3-843a-77c19cca0368"}]: dispatch 2024-09-13T14:09:20.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:19 smithi195 ceph-mon[33205]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9a384a4d-fddb-4fb3-843a-77c19cca0368"}]': finished 2024-09-13T14:09:20.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:19 smithi195 ceph-mon[33205]: osdmap e51: 8 total, 7 up, 8 in 2024-09-13T14:09:20.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:19 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:09:21.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:20 smithi102 ceph-mon[29355]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-09-13T14:09:21.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:20 smithi102 ceph-mon[29355]: from='client.? 172.21.15.195:0/650891742' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:09:21.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:20 smithi102 ceph-mon[40543]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-09-13T14:09:21.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:20 smithi102 ceph-mon[40543]: from='client.? 172.21.15.195:0/650891742' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:09:21.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:20 smithi195 ceph-mon[33205]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-09-13T14:09:21.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:20 smithi195 ceph-mon[33205]: from='client.? 172.21.15.195:0/650891742' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:09:22.992 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:22 smithi195 ceph-mon[33205]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-09-13T14:09:23.253 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:22 smithi102 ceph-mon[29355]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-09-13T14:09:23.253 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:22 smithi102 ceph-mon[40543]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-09-13T14:09:24.944 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:24 smithi195 ceph-mon[33205]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-13T14:09:25.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:24 smithi102 ceph-mon[29355]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-13T14:09:25.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:24 smithi102 ceph-mon[40543]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-13T14:09:25.855 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:25 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-13T14:09:25.856 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:25 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:09:26.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:25 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-13T14:09:26.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:25 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:09:26.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:25 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-13T14:09:26.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:25 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:09:26.904 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:26 smithi195 ceph-mon[33205]: Deploying daemon osd.7 on smithi195 2024-09-13T14:09:26.904 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:26 smithi195 ceph-mon[33205]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T14:09:27.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:26 smithi102 ceph-mon[29355]: Deploying daemon osd.7 on smithi195 2024-09-13T14:09:27.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:26 smithi102 ceph-mon[29355]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T14:09:27.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:26 smithi102 ceph-mon[40543]: Deploying daemon osd.7 on smithi195 2024-09-13T14:09:27.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:26 smithi102 ceph-mon[40543]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T14:09:29.084 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:28 smithi195 ceph-mon[33205]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T14:09:29.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:28 smithi102 ceph-mon[29355]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T14:09:29.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:28 smithi102 ceph-mon[40543]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T14:09:30.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:30 smithi195 ceph-mon[33205]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T14:09:30.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:30 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:09:30.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:30 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:30.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:30 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:31.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:30 smithi102 ceph-mon[29355]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T14:09:31.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:30 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:09:31.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:30 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:31.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:30 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:31.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:30 smithi102 ceph-mon[40543]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T14:09:31.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:30 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:09:31.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:30 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:31.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:30 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:33.069 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:32 smithi195 ceph-mon[33205]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T14:09:33.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:32 smithi102 ceph-mon[29355]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T14:09:33.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:32 smithi102 ceph-mon[40543]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T14:09:33.402 INFO:teuthology.orchestra.run.smithi195.stdout:Created osd(s) 7 on host 'smithi195' 2024-09-13T14:09:34.028 DEBUG:teuthology.orchestra.run.smithi195:osd.7> sudo journalctl -f -n 0 -u ceph-08cee502-71d9-11ef-bceb-c7b262605968@osd.7.service 2024-09-13T14:09:34.031 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-09-13T14:09:34.031 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-13T14:09:34.245 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:09:34.652 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:34 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:34.652 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:34 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:34.652 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:34 smithi102 ceph-mon[29355]: from='osd.7 [v2:172.21.15.195:6824/2059403787,v1:172.21.15.195:6825/2059403787]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-13T14:09:34.652 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:34 smithi102 ceph-mon[29355]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-13T14:09:34.652 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:34 smithi102 ceph-mon[29355]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T14:09:34.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:34 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:34.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:34 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:34.653 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:34 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:34.653 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:34 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:34.653 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:34 smithi102 ceph-mon[40543]: from='osd.7 [v2:172.21.15.195:6824/2059403787,v1:172.21.15.195:6825/2059403787]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-13T14:09:34.653 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:34 smithi102 ceph-mon[40543]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-13T14:09:34.653 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:34 smithi102 ceph-mon[40543]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T14:09:34.653 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:34 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:34.653 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:34 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:34.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:34 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:34.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:34 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:34.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:34 smithi195 ceph-mon[33205]: from='osd.7 [v2:172.21.15.195:6824/2059403787,v1:172.21.15.195:6825/2059403787]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-13T14:09:34.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:34 smithi195 ceph-mon[33205]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-13T14:09:34.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:34 smithi195 ceph-mon[33205]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T14:09:34.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:34 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:34.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:34 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:35.588 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:35 smithi195 ceph-mon[33205]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-13T14:09:35.589 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:35 smithi195 ceph-mon[33205]: osdmap e52: 8 total, 7 up, 8 in 2024-09-13T14:09:35.589 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:35 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:09:35.589 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:35 smithi195 ceph-mon[33205]: from='osd.7 [v2:172.21.15.195:6824/2059403787,v1:172.21.15.195:6825/2059403787]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-09-13T14:09:35.589 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:35 smithi195 ceph-mon[33205]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-09-13T14:09:35.589 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 14:09:35 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T14:09:35.415+0000 7f6a07c97640 -1 osd.7 0 waiting for initial osdmap 2024-09-13T14:09:35.589 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 14:09:35 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T14:09:35.422+0000 7f6a03ac0640 -1 osd.7 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T14:09:35.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:35 smithi102 ceph-mon[40543]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-13T14:09:35.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:35 smithi102 ceph-mon[40543]: osdmap e52: 8 total, 7 up, 8 in 2024-09-13T14:09:35.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:35 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:09:35.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:35 smithi102 ceph-mon[40543]: from='osd.7 [v2:172.21.15.195:6824/2059403787,v1:172.21.15.195:6825/2059403787]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-09-13T14:09:35.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:35 smithi102 ceph-mon[40543]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-09-13T14:09:35.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:35 smithi102 ceph-mon[29355]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-13T14:09:35.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:35 smithi102 ceph-mon[29355]: osdmap e52: 8 total, 7 up, 8 in 2024-09-13T14:09:35.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:35 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:09:35.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:35 smithi102 ceph-mon[29355]: from='osd.7 [v2:172.21.15.195:6824/2059403787,v1:172.21.15.195:6825/2059403787]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-09-13T14:09:35.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:35 smithi102 ceph-mon[29355]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-09-13T14:09:36.094 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:09:36.663 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:36 smithi102 ceph-mon[29355]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-09-13T14:09:36.663 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:36 smithi102 ceph-mon[29355]: osdmap e53: 8 total, 7 up, 8 in 2024-09-13T14:09:36.663 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:36 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:09:36.663 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:36 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:09:36.663 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:36 smithi102 ceph-mon[29355]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T14:09:36.663 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:36 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/945809509' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-13T14:09:36.664 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:36 smithi102 ceph-mon[40543]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-09-13T14:09:36.664 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:36 smithi102 ceph-mon[40543]: osdmap e53: 8 total, 7 up, 8 in 2024-09-13T14:09:36.664 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:36 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:09:36.664 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:36 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:09:36.665 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:36 smithi102 ceph-mon[40543]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T14:09:36.665 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:36 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/945809509' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-13T14:09:36.683 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":53,"num_osds":8,"num_up_osds":7,"osd_up_since":1726236551,"num_in_osds":8,"osd_in_since":1726236559,"num_remapped_pgs":0} 2024-09-13T14:09:36.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:36 smithi195 ceph-mon[33205]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-09-13T14:09:36.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:36 smithi195 ceph-mon[33205]: osdmap e53: 8 total, 7 up, 8 in 2024-09-13T14:09:36.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:36 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:09:36.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:36 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:09:36.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:36 smithi195 ceph-mon[33205]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-13T14:09:36.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:36 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/945809509' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-13T14:09:37.685 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-13T14:09:37.708 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:37 smithi102 ceph-mon[29355]: purged_snaps scrub starts 2024-09-13T14:09:37.708 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:37 smithi102 ceph-mon[29355]: purged_snaps scrub ok 2024-09-13T14:09:37.708 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:37 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:09:37.708 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:37 smithi102 ceph-mon[29355]: osd.7 [v2:172.21.15.195:6824/2059403787,v1:172.21.15.195:6825/2059403787] boot 2024-09-13T14:09:37.708 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:37 smithi102 ceph-mon[29355]: osdmap e54: 8 total, 8 up, 8 in 2024-09-13T14:09:37.708 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:37 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:09:37.708 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:37 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:37.708 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:37 smithi102 ceph-mon[40543]: purged_snaps scrub starts 2024-09-13T14:09:37.709 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:37 smithi102 ceph-mon[40543]: purged_snaps scrub ok 2024-09-13T14:09:37.709 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:37 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:09:37.709 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:37 smithi102 ceph-mon[40543]: osd.7 [v2:172.21.15.195:6824/2059403787,v1:172.21.15.195:6825/2059403787] boot 2024-09-13T14:09:37.709 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:37 smithi102 ceph-mon[40543]: osdmap e54: 8 total, 8 up, 8 in 2024-09-13T14:09:37.709 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:37 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:09:37.709 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:37 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:37.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:37 smithi195 ceph-mon[33205]: purged_snaps scrub starts 2024-09-13T14:09:37.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:37 smithi195 ceph-mon[33205]: purged_snaps scrub ok 2024-09-13T14:09:37.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:37 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:09:37.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:37 smithi195 ceph-mon[33205]: osd.7 [v2:172.21.15.195:6824/2059403787,v1:172.21.15.195:6825/2059403787] boot 2024-09-13T14:09:37.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:37 smithi195 ceph-mon[33205]: osdmap e54: 8 total, 8 up, 8 in 2024-09-13T14:09:37.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:37 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:09:37.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:37 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:37.879 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:09:38.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:38 smithi102 ceph-mon[29355]: Detected new or changed devices on smithi195 2024-09-13T14:09:38.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:38 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:38.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:38 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:09:38.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:38 smithi102 ceph-mon[29355]: osdmap e55: 8 total, 8 up, 8 in 2024-09-13T14:09:38.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:38 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:09:38.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:38 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:09:38.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:38 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:09:38.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:38 smithi102 ceph-mon[29355]: Adjusting osd_memory_target on smithi195 to 4272M 2024-09-13T14:09:38.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:38 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:38.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:38 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:09:38.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:38 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:09:38.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:38 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:38.671 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:38 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:09:38.671 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:38 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:09:38.671 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:38 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:09:38.671 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:38 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:38.671 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:38 smithi102 ceph-mon[29355]: pgmap v165: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:09:38.671 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:38 smithi102 ceph-mon[29355]: osdmap e56: 8 total, 8 up, 8 in 2024-09-13T14:09:38.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:38 smithi102 ceph-mon[40543]: Detected new or changed devices on smithi195 2024-09-13T14:09:38.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:38 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:38.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:38 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:09:38.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:38 smithi102 ceph-mon[40543]: osdmap e55: 8 total, 8 up, 8 in 2024-09-13T14:09:38.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:38 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:09:38.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:38 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:09:38.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:38 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:09:38.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:38 smithi102 ceph-mon[40543]: Adjusting osd_memory_target on smithi195 to 4272M 2024-09-13T14:09:38.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:38 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:38.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:38 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:09:38.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:38 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:09:38.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:38 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:38.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:38 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:09:38.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:38 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:09:38.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:38 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:09:38.673 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:38 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:38.673 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:38 smithi102 ceph-mon[40543]: pgmap v165: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:09:38.673 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:38 smithi102 ceph-mon[40543]: osdmap e56: 8 total, 8 up, 8 in 2024-09-13T14:09:38.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:38 smithi195 ceph-mon[33205]: Detected new or changed devices on smithi195 2024-09-13T14:09:38.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:38 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:38.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:38 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:09:38.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:38 smithi195 ceph-mon[33205]: osdmap e55: 8 total, 8 up, 8 in 2024-09-13T14:09:38.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:38 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:09:38.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:38 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:09:38.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:38 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:09:38.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:38 smithi195 ceph-mon[33205]: Adjusting osd_memory_target on smithi195 to 4272M 2024-09-13T14:09:38.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:38 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:38.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:38 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:09:38.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:38 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:09:38.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:38 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:38.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:38 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:09:38.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:38 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:09:38.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:38 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:09:38.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:38 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:38.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:38 smithi195 ceph-mon[33205]: pgmap v165: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:09:38.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:38 smithi195 ceph-mon[33205]: osdmap e56: 8 total, 8 up, 8 in 2024-09-13T14:09:39.642 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:09:40.316 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":57,"num_osds":8,"num_up_osds":8,"osd_up_since":1726236576,"num_in_osds":8,"osd_in_since":1726236559,"num_remapped_pgs":0} 2024-09-13T14:09:40.317 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-13T14:09:40.558 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:09:40.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:40 smithi102 ceph-mon[29355]: osdmap e57: 8 total, 8 up, 8 in 2024-09-13T14:09:40.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:40 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/3545823193' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-13T14:09:40.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:40 smithi102 ceph-mon[29355]: pgmap v168: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:09:40.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:40 smithi102 ceph-mon[40543]: osdmap e57: 8 total, 8 up, 8 in 2024-09-13T14:09:40.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:40 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/3545823193' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-13T14:09:40.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:40 smithi102 ceph-mon[40543]: pgmap v168: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:09:40.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:40 smithi195 ceph-mon[33205]: osdmap e57: 8 total, 8 up, 8 in 2024-09-13T14:09:40.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:40 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/3545823193' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-13T14:09:40.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:40 smithi195 ceph-mon[33205]: pgmap v168: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:09:41.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:41 smithi102 ceph-mon[29355]: osdmap e58: 8 total, 8 up, 8 in 2024-09-13T14:09:41.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:41 smithi102 ceph-mon[40543]: osdmap e58: 8 total, 8 up, 8 in 2024-09-13T14:09:41.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:41 smithi195 ceph-mon[33205]: osdmap e58: 8 total, 8 up, 8 in 2024-09-13T14:09:42.391 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:09:42.392 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":58,"fsid":"08cee502-71d9-11ef-bceb-c7b262605968","created":"2024-09-13T14:04:48.179879+0000","modified":"2024-09-13T14:09:40.427951+0000","last_up_change":"2024-09-13T14:09:36.412493+0000","last_in_change":"2024-09-13T14:09:19.207877+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T14:07:41.927240+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b43b49e0-1cf0-47b8-aa85-37108f321220","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":466204285},{"type":"v1","addr":"172.21.15.102:6803","nonce":466204285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":466204285},{"type":"v1","addr":"172.21.15.102:6805","nonce":466204285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":466204285},{"type":"v1","addr":"172.21.15.102:6809","nonce":466204285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":466204285},{"type":"v1","addr":"172.21.15.102:6807","nonce":466204285}]},"public_addr":"172.21.15.102:6803/466204285","cluster_addr":"172.21.15.102:6805/466204285","heartbeat_back_addr":"172.21.15.102:6809/466204285","heartbeat_front_addr":"172.21.15.102:6807/466204285","state":["exists","up"]},{"osd":1,"uuid":"27e10d5e-c471-4132-9c73-83c5708c3b08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":1293422793},{"type":"v1","addr":"172.21.15.102:6811","nonce":1293422793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":1293422793},{"type":"v1","addr":"172.21.15.102:6813","nonce":1293422793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1293422793},{"type":"v1","addr":"172.21.15.102:6817","nonce":1293422793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":1293422793},{"type":"v1","addr":"172.21.15.102:6815","nonce":1293422793}]},"public_addr":"172.21.15.102:6811/1293422793","cluster_addr":"172.21.15.102:6813/1293422793","heartbeat_back_addr":"172.21.15.102:6817/1293422793","heartbeat_front_addr":"172.21.15.102:6815/1293422793","state":["exists","up"]},{"osd":2,"uuid":"40a94b2b-2c24-4164-91be-0816a1acf3d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":360502781},{"type":"v1","addr":"172.21.15.102:6819","nonce":360502781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":360502781},{"type":"v1","addr":"172.21.15.102:6821","nonce":360502781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":360502781},{"type":"v1","addr":"172.21.15.102:6825","nonce":360502781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":360502781},{"type":"v1","addr":"172.21.15.102:6823","nonce":360502781}]},"public_addr":"172.21.15.102:6819/360502781","cluster_addr":"172.21.15.102:6821/360502781","heartbeat_back_addr":"172.21.15.102:6825/360502781","heartbeat_front_addr":"172.21.15.102:6823/360502781","state":["exists","up"]},{"osd":3,"uuid":"2ed85faf-6610-4bcc-a09d-4a6f93f5c6a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":3709228546},{"type":"v1","addr":"172.21.15.102:6827","nonce":3709228546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":3709228546},{"type":"v1","addr":"172.21.15.102:6829","nonce":3709228546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3709228546},{"type":"v1","addr":"172.21.15.102:6833","nonce":3709228546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":3709228546},{"type":"v1","addr":"172.21.15.102:6831","nonce":3709228546}]},"public_addr":"172.21.15.102:6827/3709228546","cluster_addr":"172.21.15.102:6829/3709228546","heartbeat_back_addr":"172.21.15.102:6833/3709228546","heartbeat_front_addr":"172.21.15.102:6831/3709228546","state":["exists","up"]},{"osd":4,"uuid":"8a45f350-fb3e-4049-bcc4-532d3e20c238","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":2821651815},{"type":"v1","addr":"172.21.15.195:6801","nonce":2821651815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":2821651815},{"type":"v1","addr":"172.21.15.195:6803","nonce":2821651815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":2821651815},{"type":"v1","addr":"172.21.15.195:6807","nonce":2821651815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":2821651815},{"type":"v1","addr":"172.21.15.195:6805","nonce":2821651815}]},"public_addr":"172.21.15.195:6801/2821651815","cluster_addr":"172.21.15.195:6803/2821651815","heartbeat_back_addr":"172.21.15.195:6807/2821651815","heartbeat_front_addr":"172.21.15.195:6805/2821651815","state":["exists","up"]},{"osd":5,"uuid":"363cf638-db26-4c86-b2be-e075fd09ac16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":3074139237},{"type":"v1","addr":"172.21.15.195:6809","nonce":3074139237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":3074139237},{"type":"v1","addr":"172.21.15.195:6811","nonce":3074139237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":3074139237},{"type":"v1","addr":"172.21.15.195:6815","nonce":3074139237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":3074139237},{"type":"v1","addr":"172.21.15.195:6813","nonce":3074139237}]},"public_addr":"172.21.15.195:6809/3074139237","cluster_addr":"172.21.15.195:6811/3074139237","heartbeat_back_addr":"172.21.15.195:6815/3074139237","heartbeat_front_addr":"172.21.15.195:6813/3074139237","state":["exists","up"]},{"osd":6,"uuid":"f37416e8-9c25-4860-a32f-0d1c404a8bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":4026627594},{"type":"v1","addr":"172.21.15.195:6817","nonce":4026627594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":4026627594},{"type":"v1","addr":"172.21.15.195:6819","nonce":4026627594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":4026627594},{"type":"v1","addr":"172.21.15.195:6823","nonce":4026627594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":4026627594},{"type":"v1","addr":"172.21.15.195:6821","nonce":4026627594}]},"public_addr":"172.21.15.195:6817/4026627594","cluster_addr":"172.21.15.195:6819/4026627594","heartbeat_back_addr":"172.21.15.195:6823/4026627594","heartbeat_front_addr":"172.21.15.195:6821/4026627594","state":["exists","up"]},{"osd":7,"uuid":"9a384a4d-fddb-4fb3-843a-77c19cca0368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2059403787},{"type":"v1","addr":"172.21.15.195:6825","nonce":2059403787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2059403787},{"type":"v1","addr":"172.21.15.195:6827","nonce":2059403787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2059403787},{"type":"v1","addr":"172.21.15.195:6831","nonce":2059403787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2059403787},{"type":"v1","addr":"172.21.15.195:6829","nonce":2059403787}]},"public_addr":"172.21.15.195:6825/2059403787","cluster_addr":"172.21.15.195:6827/2059403787","heartbeat_back_addr":"172.21.15.195:6831/2059403787","heartbeat_front_addr":"172.21.15.195:6829/2059403787","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:06:53.502746+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:07:16.251669+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:07:38.848272+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:08:02.013035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:08:23.043482+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:08:45.653159+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:09:10.187018+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:09:34.393753+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/1939570054":"2024-09-14T14:05:37.634624+0000","172.21.15.102:0/3399546099":"2024-09-14T14:05:37.634624+0000","172.21.15.102:6801/3480998245":"2024-09-14T14:05:37.634624+0000","172.21.15.102:6800/3480998245":"2024-09-14T14:05:37.634624+0000","172.21.15.102:0/3189088157":"2024-09-14T14:05:15.286479+0000","172.21.15.102:0/1574792592":"2024-09-14T14:05:15.286479+0000","172.21.15.102:6801/2761043787":"2024-09-14T14:05:15.286479+0000","172.21.15.102:0/2444394840":"2024-09-14T14:05:37.634624+0000","172.21.15.102:6800/2761043787":"2024-09-14T14:05:15.286479+0000","172.21.15.102:0/1582001159":"2024-09-14T14:05:15.286479+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T14:09:42.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:42 smithi102 ceph-mon[29355]: pgmap v170: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:09:42.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:42 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/1669164487' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:09:42.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:42 smithi102 ceph-mon[40543]: pgmap v170: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:09:42.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:42 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/1669164487' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:09:42.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:42 smithi195 ceph-mon[33205]: pgmap v170: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:09:42.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:42 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/1669164487' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:09:43.093 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T14:07:41.927240+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T14:09:43.093 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-09-13T14:09:43.292 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:09:44.980 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:44 smithi102 ceph-mon[29355]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-13T14:09:44.981 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:44 smithi102 ceph-mon[40543]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-13T14:09:44.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:44 smithi195 ceph-mon[33205]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-13T14:09:45.126 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-09-13T14:09:45.824 INFO:tasks.cephadm:Adding prometheus.a on smithi195 2024-09-13T14:09:45.825 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph orch apply prometheus '1;smithi195=a' 2024-09-13T14:09:45.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:45 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/1749039868' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-13T14:09:46.036 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:09:46.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:45 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/1749039868' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-13T14:09:46.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:45 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/1749039868' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-13T14:09:46.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:46 smithi195 ceph-mon[33205]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-09-13T14:09:47.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:46 smithi102 ceph-mon[29355]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-09-13T14:09:47.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:46 smithi102 ceph-mon[40543]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-09-13T14:09:47.880 INFO:teuthology.orchestra.run.smithi195.stdout:Scheduled prometheus update... 2024-09-13T14:09:48.672 DEBUG:teuthology.orchestra.run.smithi195:prometheus.a> sudo journalctl -f -n 0 -u ceph-08cee502-71d9-11ef-bceb-c7b262605968@prometheus.a.service 2024-09-13T14:09:48.675 INFO:tasks.cephadm:Adding node-exporter.a on smithi102 2024-09-13T14:09:48.675 INFO:tasks.cephadm:Adding node-exporter.b on smithi195 2024-09-13T14:09:48.676 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph orch apply node-exporter '2;smithi102=a;smithi195=b' 2024-09-13T14:09:48.946 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:09:49.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:48 smithi102 ceph-mon[29355]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-13T14:09:49.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:48 smithi102 ceph-mon[29355]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi195=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:09:49.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:48 smithi102 ceph-mon[29355]: Saving service prometheus spec with placement smithi195=a;count:1 2024-09-13T14:09:49.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:48 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:49.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:48 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:09:49.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:48 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:09:49.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:48 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:09:49.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:48 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:49.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:48 smithi102 ceph-mon[40543]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-13T14:09:49.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:48 smithi102 ceph-mon[40543]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi195=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:09:49.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:48 smithi102 ceph-mon[40543]: Saving service prometheus spec with placement smithi195=a;count:1 2024-09-13T14:09:49.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:48 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:49.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:48 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:09:49.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:48 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:09:49.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:48 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:09:49.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:48 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:49.183 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:48 smithi195 ceph-mon[33205]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-13T14:09:49.183 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:48 smithi195 ceph-mon[33205]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi195=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:09:49.183 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:48 smithi195 ceph-mon[33205]: Saving service prometheus spec with placement smithi195=a;count:1 2024-09-13T14:09:49.183 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:48 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:49.183 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:48 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:09:49.183 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:48 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:09:49.183 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:48 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:09:49.184 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:48 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:50.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:49 smithi102 ceph-mon[29355]: Deploying daemon prometheus.a on smithi195 2024-09-13T14:09:50.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:49 smithi102 ceph-mon[40543]: Deploying daemon prometheus.a on smithi195 2024-09-13T14:09:50.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:49 smithi195 ceph-mon[33205]: Deploying daemon prometheus.a on smithi195 2024-09-13T14:09:50.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:50 smithi195 ceph-mon[33205]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T14:09:51.158 INFO:teuthology.orchestra.run.smithi195.stdout:Scheduled node-exporter update... 2024-09-13T14:09:51.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:50 smithi102 ceph-mon[29355]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T14:09:51.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:50 smithi102 ceph-mon[40543]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T14:09:52.207 DEBUG:teuthology.orchestra.run.smithi102:node-exporter.a> sudo journalctl -f -n 0 -u ceph-08cee502-71d9-11ef-bceb-c7b262605968@node-exporter.a.service 2024-09-13T14:09:52.210 DEBUG:teuthology.orchestra.run.smithi195:node-exporter.b> sudo journalctl -f -n 0 -u ceph-08cee502-71d9-11ef-bceb-c7b262605968@node-exporter.b.service 2024-09-13T14:09:52.213 INFO:tasks.cephadm:Adding alertmanager.a on smithi102 2024-09-13T14:09:52.213 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph orch apply alertmanager '1;smithi102=a' 2024-09-13T14:09:52.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:52 smithi102 ceph-mon[29355]: from='client.24295 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi102=a;smithi195=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:09:52.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:52 smithi102 ceph-mon[29355]: Saving service node-exporter spec with placement smithi102=a;smithi195=b;count:2 2024-09-13T14:09:52.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:52 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:52.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:52 smithi102 ceph-mon[29355]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-09-13T14:09:52.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:52 smithi102 ceph-mon[40543]: from='client.24295 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi102=a;smithi195=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:09:52.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:52 smithi102 ceph-mon[40543]: Saving service node-exporter spec with placement smithi102=a;smithi195=b;count:2 2024-09-13T14:09:52.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:52 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:52.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:52 smithi102 ceph-mon[40543]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-09-13T14:09:52.470 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:09:52.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:52 smithi195 ceph-mon[33205]: from='client.24295 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi102=a;smithi195=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:09:52.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:52 smithi195 ceph-mon[33205]: Saving service node-exporter spec with placement smithi102=a;smithi195=b;count:2 2024-09-13T14:09:52.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:52 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:52.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:52 smithi195 ceph-mon[33205]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-09-13T14:09:54.994 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:54 smithi195 ceph-mon[33205]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-13T14:09:55.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:54 smithi102 ceph-mon[29355]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-13T14:09:55.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:54 smithi102 ceph-mon[40543]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-13T14:09:55.205 INFO:teuthology.orchestra.run.smithi195.stdout:Scheduled alertmanager update... 2024-09-13T14:09:56.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:56 smithi195 ceph-mon[33205]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi102=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:09:56.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:56 smithi195 ceph-mon[33205]: Saving service alertmanager spec with placement smithi102=a;count:1 2024-09-13T14:09:56.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:56 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:56.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:56 smithi195 ceph-mon[33205]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:09:56.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:56 smithi102 ceph-mon[40543]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi102=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:09:56.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:56 smithi102 ceph-mon[40543]: Saving service alertmanager spec with placement smithi102=a;count:1 2024-09-13T14:09:56.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:56 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:56.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:56 smithi102 ceph-mon[40543]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:09:56.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:56 smithi102 ceph-mon[29355]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi102=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:09:56.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:56 smithi102 ceph-mon[29355]: Saving service alertmanager spec with placement smithi102=a;count:1 2024-09-13T14:09:56.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:56 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:09:56.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:56 smithi102 ceph-mon[29355]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:09:56.726 DEBUG:teuthology.orchestra.run.smithi102:alertmanager.a> sudo journalctl -f -n 0 -u ceph-08cee502-71d9-11ef-bceb-c7b262605968@alertmanager.a.service 2024-09-13T14:09:56.729 INFO:tasks.cephadm:Adding grafana.a on smithi195 2024-09-13T14:09:56.729 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph orch apply grafana '1;smithi195=a' 2024-09-13T14:09:57.233 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:09:58.868 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:09:58 smithi195 ceph-mon[33205]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:09:59.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:09:58 smithi102 ceph-mon[29355]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:09:59.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:09:58 smithi102 ceph-mon[40543]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:00.532 INFO:teuthology.orchestra.run.smithi195.stdout:Scheduled grafana update... 2024-09-13T14:10:00.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:00 smithi195 ceph-mon[33205]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:00.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:00 smithi195 ceph-mon[33205]: overall HEALTH_OK 2024-09-13T14:10:00.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:00 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:10:01.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:00 smithi102 ceph-mon[29355]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:01.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:00 smithi102 ceph-mon[29355]: overall HEALTH_OK 2024-09-13T14:10:01.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:00 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:10:01.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:00 smithi102 ceph-mon[40543]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:01.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:00 smithi102 ceph-mon[40543]: overall HEALTH_OK 2024-09-13T14:10:01.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:00 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:10:01.434 DEBUG:teuthology.orchestra.run.smithi195:grafana.a> sudo journalctl -f -n 0 -u ceph-08cee502-71d9-11ef-bceb-c7b262605968@grafana.a.service 2024-09-13T14:10:01.436 INFO:tasks.cephadm:Setting up client nodes... 2024-09-13T14:10:01.437 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-13T14:10:01.651 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:10:01.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:01 smithi195 ceph-mon[33205]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi195=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:10:01.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:01 smithi195 ceph-mon[33205]: Saving service grafana spec with placement smithi195=a;count:1 2024-09-13T14:10:02.055 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:01 smithi102 ceph-mon[29355]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi195=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:10:02.055 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:01 smithi102 ceph-mon[29355]: Saving service grafana spec with placement smithi195=a;count:1 2024-09-13T14:10:02.055 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:01 smithi102 ceph-mon[40543]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi195=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:10:02.055 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:01 smithi102 ceph-mon[40543]: Saving service grafana spec with placement smithi195=a;count:1 2024-09-13T14:10:02.876 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:02 smithi102 ceph-mon[29355]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:02.876 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:02 smithi102 ceph-mon[40543]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:02.924 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:02 smithi195 ceph-mon[33205]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:03.186 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:10:03 smithi195 podman[62258]: 2024-09-13 14:10:03.161363094 +0000 UTC m=+0.350708130 container start 579ec0adef1f5d31626c4aba1da83e108b7365a1f287c3bc5a01b56ec2f746e6 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:10:03.498 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:10:03 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[62273]: ts=2024-09-13T14:10:03.186Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-13T14:10:03.498 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:10:03 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[62273]: ts=2024-09-13T14:10:03.186Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-13T14:10:03.498 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:10:03 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[62273]: ts=2024-09-13T14:10:03.186Z caller=main.go:570 level=info host_details="(Linux 5.14.0-505.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024 x86_64 smithi195 (none))" 2024-09-13T14:10:03.498 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:10:03 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[62273]: ts=2024-09-13T14:10:03.186Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-13T14:10:03.498 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:10:03 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[62273]: ts=2024-09-13T14:10:03.186Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-13T14:10:03.498 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:10:03 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[62273]: ts=2024-09-13T14:10:03.187Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-13T14:10:03.498 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:10:03 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[62273]: ts=2024-09-13T14:10:03.187Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-13T14:10:03.499 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:10:03 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[62273]: ts=2024-09-13T14:10:03.188Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-13T14:10:03.499 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:10:03 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[62273]: ts=2024-09-13T14:10:03.188Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-13T14:10:03.499 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:10:03 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[62273]: ts=2024-09-13T14:10:03.191Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-13T14:10:03.499 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:10:03 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[62273]: ts=2024-09-13T14:10:03.191Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.685µs 2024-09-13T14:10:03.499 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:10:03 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[62273]: ts=2024-09-13T14:10:03.191Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-13T14:10:03.499 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:10:03 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[62273]: ts=2024-09-13T14:10:03.191Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-09-13T14:10:03.499 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:10:03 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[62273]: ts=2024-09-13T14:10:03.191Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=17.477µs wal_replay_duration=298.608µs wbl_replay_duration=113ns total_replay_duration=333.423µs 2024-09-13T14:10:03.499 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:10:03 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[62273]: ts=2024-09-13T14:10:03.192Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-13T14:10:03.499 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:10:03 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[62273]: ts=2024-09-13T14:10:03.192Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-13T14:10:03.499 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:10:03 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[62273]: ts=2024-09-13T14:10:03.192Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-13T14:10:03.499 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:10:03 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[62273]: ts=2024-09-13T14:10:03.212Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.381521ms db_storage=719ns remote_storage=1.026µs web_handler=295ns query_engine=564ns scrape=396.094µs scrape_sd=74.144µs notify=745ns notify_sd=1.32µs rules=19.63303ms tracing=5.767µs 2024-09-13T14:10:03.499 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:10:03 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[62273]: ts=2024-09-13T14:10:03.212Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-13T14:10:03.500 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:10:03 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[62273]: ts=2024-09-13T14:10:03.212Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-13T14:10:03.500 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:10:03 smithi195 bash[62258]: 579ec0adef1f5d31626c4aba1da83e108b7365a1f287c3bc5a01b56ec2f746e6 2024-09-13T14:10:03.500 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:10:03 smithi195 systemd[1]: Started Ceph prometheus.a for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:10:03.630 INFO:teuthology.orchestra.run.smithi102.stdout:[client.0] 2024-09-13T14:10:03.630 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQC7R+Rm3AxEJRAApMrPT76e52JsKeXtslqtag== 2024-09-13T14:10:04.264 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-09-13T14:10:04.264 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-13T14:10:04.264 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-13T14:10:04.301 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-13T14:10:04.523 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:10:04.528 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:04 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:10:04.528 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:04 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:10:04.528 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:04 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:10:04.529 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:04 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-13T14:10:04.529 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:04 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/3159316243' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T14:10:04.529 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:04 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/3159316243' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-13T14:10:04.529 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:04 smithi102 ceph-mon[29355]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:04.529 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:04 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: ignoring --setuser ceph since I am not root 2024-09-13T14:10:04.529 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:04 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: ignoring --setgroup ceph since I am not root 2024-09-13T14:10:04.529 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:04 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:10:04.470+0000 7f0640920040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-13T14:10:04.529 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:04 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:10:04.526+0000 7f0640920040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-13T14:10:04.530 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:04 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:10:04.530 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:04 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:10:04.530 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:04 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:10:04.530 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:04 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-13T14:10:04.530 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:04 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/3159316243' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T14:10:04.530 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:04 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/3159316243' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-13T14:10:04.530 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:04 smithi102 ceph-mon[40543]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:04.645 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:04 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:10:04.645 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:04 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:10:04.645 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:04 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' 2024-09-13T14:10:04.645 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:04 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-13T14:10:04.645 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:04 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/3159316243' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T14:10:04.646 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:04 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/3159316243' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-13T14:10:04.646 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:04 smithi195 ceph-mon[33205]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:04.646 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:10:04 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: ignoring --setuser ceph since I am not root 2024-09-13T14:10:04.646 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:10:04 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: ignoring --setgroup ceph since I am not root 2024-09-13T14:10:04.646 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:10:04 smithi195 ceph-mgr[35287]: -- 172.21.15.195:0/2092065562 <== mon.2 v2:172.21.15.102:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558a3b5fcd00 con 0x558a3b5d1000 2024-09-13T14:10:04.646 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:10:04 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:10:04.485+0000 7f738f3a7040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-13T14:10:04.647 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:10:04 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:10:04.548+0000 7f738f3a7040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-13T14:10:04.882 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:04 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:10:04.673+0000 7f0640920040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-13T14:10:04.882 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:04 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:10:04.738+0000 7f0640920040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-13T14:10:04.882 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:04 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:10:04.880+0000 7f0640920040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:10:04.917 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:10:04 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:10:04.697+0000 7f738f3a7040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-13T14:10:04.917 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:10:04 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:10:04.764+0000 7f738f3a7040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-13T14:10:04.917 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:10:04 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:10:04.915+0000 7f738f3a7040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:10:05.168 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:05 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:10:05.020+0000 7f0640920040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-13T14:10:05.247 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:10:05 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:10:05.057+0000 7f738f3a7040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-13T14:10:05.600 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:05 smithi195 ceph-mon[33205]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-13T14:10:05.601 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:05 smithi195 ceph-mon[33205]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-13T14:10:05.601 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:10:05 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:10:05.360+0000 7f738f3a7040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-13T14:10:05.601 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:10:05 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:10:05.419+0000 7f738f3a7040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-13T14:10:05.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:05 smithi102 ceph-mon[29355]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-13T14:10:05.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:05 smithi102 ceph-mon[29355]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-13T14:10:05.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:05 smithi102 ceph-mon[40543]: from='mgr.14148 172.21.15.102:0/2923699288' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-13T14:10:05.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:05 smithi102 ceph-mon[40543]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-13T14:10:05.669 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:05 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:10:05.325+0000 7f0640920040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-13T14:10:05.669 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:05 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:10:05.385+0000 7f0640920040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-13T14:10:05.669 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:05 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:10:05.564+0000 7f0640920040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-13T14:10:05.997 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:10:05 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:10:05.599+0000 7f738f3a7040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-13T14:10:06.856 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:10:06 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:10:06.511+0000 7f738f3a7040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-13T14:10:06.857 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:10:06 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:10:06.578+0000 7f738f3a7040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-13T14:10:06.918 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:06 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:10:06.419+0000 7f0640920040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-13T14:10:06.918 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:06 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:10:06.484+0000 7f0640920040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-13T14:10:07.415 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:10:07 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:10:07.297+0000 7f738f3a7040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-13T14:10:07.431 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:07 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:10:07.178+0000 7f0640920040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-13T14:10:07.431 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:07 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:10:07.291+0000 7f0640920040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-13T14:10:07.669 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:10:07 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:10:07.414+0000 7f738f3a7040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-13T14:10:07.669 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:10:07 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:10:07.560+0000 7f738f3a7040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:10:07.669 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:10:07 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:10:07.627+0000 7f738f3a7040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-13T14:10:07.703 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:07 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:10:07.430+0000 7f0640920040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:10:07.703 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:07 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:10:07.490+0000 7f0640920040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-13T14:10:07.703 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:07 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:10:07.634+0000 7f0640920040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-13T14:10:07.703 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:07 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:10:07.701+0000 7f0640920040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-13T14:10:07.902 INFO:teuthology.orchestra.run.smithi195.stdout:[client.1] 2024-09-13T14:10:07.902 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQC/R+RmipqUNRAAhfZgcWKWEMcnhZNXFYo3iQ== 2024-09-13T14:10:07.951 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:10:07 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:10:07.778+0000 7f738f3a7040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-13T14:10:07.952 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:10:07 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:10:07.847+0000 7f738f3a7040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-13T14:10:07.952 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:10:07 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:10:07.908+0000 7f738f3a7040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-13T14:10:07.980 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:07 smithi102 ceph-mon[29355]: from='client.? 172.21.15.195:0/2655821801' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T14:10:07.980 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:07 smithi102 ceph-mon[29355]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T14:10:07.980 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:07 smithi102 ceph-mon[29355]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-13T14:10:07.980 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:07 smithi102 ceph-mon[40543]: from='client.? 172.21.15.195:0/2655821801' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T14:10:07.981 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:07 smithi102 ceph-mon[40543]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T14:10:07.981 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:07 smithi102 ceph-mon[40543]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-13T14:10:07.981 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:07 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:10:07.760+0000 7f0640920040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-13T14:10:07.981 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:07 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:10:07.914+0000 7f0640920040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-13T14:10:07.981 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:07 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:10:07.979+0000 7f0640920040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-13T14:10:08.202 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:10:08 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:10:08.067+0000 7f738f3a7040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-13T14:10:08.202 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:10:08 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:10:08.132+0000 7f738f3a7040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-13T14:10:08.202 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:10:08 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:10:08.200+0000 7f738f3a7040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-13T14:10:08.202 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:07 smithi195 ceph-mon[33205]: from='client.? 172.21.15.195:0/2655821801' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T14:10:08.202 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:07 smithi195 ceph-mon[33205]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T14:10:08.202 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:07 smithi195 ceph-mon[33205]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-13T14:10:08.243 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:08 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:10:08.044+0000 7f0640920040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-13T14:10:08.243 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:08 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:10:08.126+0000 7f0640920040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-13T14:10:08.243 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:08 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:10:08.184+0000 7f0640920040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-13T14:10:08.243 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:08 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:10:08.241+0000 7f0640920040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-13T14:10:08.497 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:10:08 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:10:08.283+0000 7f738f3a7040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-13T14:10:08.497 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:10:08 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:10:08.341+0000 7f738f3a7040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-13T14:10:08.497 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:10:08 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:10:08.400+0000 7f738f3a7040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-13T14:10:08.544 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-09-13T14:10:08.544 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-09-13T14:10:08.545 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-13T14:10:08.583 INFO:tasks.cephadm:Creating RBD pool 2024-09-13T14:10:08.584 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd pool create rbd 8 2024-09-13T14:10:08.783 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:10:09.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-mon[29355]: Active manager daemon y restarted 2024-09-13T14:10:09.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-mon[29355]: Activating manager daemon y 2024-09-13T14:10:09.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-mon[29355]: osdmap e59: 8 total, 8 up, 8 in 2024-09-13T14:10:09.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-mon[29355]: mgrmap e17: y(active, starting, since 0.0123699s), standbys: x 2024-09-13T14:10:09.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:10:09.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:10:09.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:10:09.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:10:09.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:10:09.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:10:09.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:10:09.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:10:09.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:10:09.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:10:09.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:10:09.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:10:09.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:10:09.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:10:09.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:10:09.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:10:09.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-mon[29355]: Manager daemon y is now available 2024-09-13T14:10:09.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:10:09.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-mon[40543]: Active manager daemon y restarted 2024-09-13T14:10:09.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-mon[40543]: Activating manager daemon y 2024-09-13T14:10:09.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-mon[40543]: osdmap e59: 8 total, 8 up, 8 in 2024-09-13T14:10:09.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-mon[40543]: mgrmap e17: y(active, starting, since 0.0123699s), standbys: x 2024-09-13T14:10:09.172 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:10:09.172 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:10:09.172 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:10:09.172 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:10:09.172 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:10:09.172 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:10:09.172 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:10:09.172 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:10:09.172 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:10:09.172 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:10:09.173 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:10:09.173 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:10:09.173 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:10:09.173 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:10:09.173 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:10:09.173 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:10:09.173 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-mon[40543]: Manager daemon y is now available 2024-09-13T14:10:09.173 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:10:09.173 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:10:09.035+0000 7f0640920040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-13T14:10:09.173 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: [13/Sep/2024:14:10:09] ENGINE Bus STARTING 2024-09-13T14:10:09.174 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: CherryPy Checker: 2024-09-13T14:10:09.174 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: The Application mounted at '' has an empty config. 2024-09-13T14:10:09.174 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:10:09.406 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:10:09 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:10:09.193+0000 7f738f3a7040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-13T14:10:09.406 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:10:09 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: [13/Sep/2024:14:10:09] ENGINE Bus STARTING 2024-09-13T14:10:09.406 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:10:09 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: CherryPy Checker: 2024-09-13T14:10:09.406 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:10:09 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: The Application mounted at '' has an empty config. 2024-09-13T14:10:09.406 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:10:09 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: 2024-09-13T14:10:09.406 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:10:09 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: [13/Sep/2024:14:10:09] ENGINE Serving on http://:::9283 2024-09-13T14:10:09.406 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:10:09 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[35264]: [13/Sep/2024:14:10:09] ENGINE Bus STARTED 2024-09-13T14:10:09.406 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:09 smithi195 ceph-mon[33205]: Active manager daemon y restarted 2024-09-13T14:10:09.407 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:09 smithi195 ceph-mon[33205]: Activating manager daemon y 2024-09-13T14:10:09.407 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:09 smithi195 ceph-mon[33205]: osdmap e59: 8 total, 8 up, 8 in 2024-09-13T14:10:09.407 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:09 smithi195 ceph-mon[33205]: mgrmap e17: y(active, starting, since 0.0123699s), standbys: x 2024-09-13T14:10:09.407 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:09 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:10:09.407 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:09 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:10:09.407 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:09 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:10:09.407 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:09 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:10:09.407 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:09 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:10:09.407 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:09 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:10:09.407 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:09 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:10:09.407 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:09 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:10:09.407 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:09 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:10:09.407 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:09 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:10:09.407 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:09 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:10:09.408 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:09 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:10:09.408 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:09 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:10:09.408 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:09 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:10:09.408 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:09 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:10:09.408 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:09 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:10:09.408 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:09 smithi195 ceph-mon[33205]: Manager daemon y is now available 2024-09-13T14:10:09.408 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:09 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:10:09.666 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: [13/Sep/2024:14:10:09] ENGINE Serving on http://:::9283 2024-09-13T14:10:09.666 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:09 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: [13/Sep/2024:14:10:09] ENGINE Bus STARTED 2024-09-13T14:10:10.344 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:10 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:10.344 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:10 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:10:10.344 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:10 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:10:10.345 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:10 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:10:10.345 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:10 smithi102 ceph-mon[29355]: Standby manager daemon x restarted 2024-09-13T14:10:10.345 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:10 smithi102 ceph-mon[29355]: Standby manager daemon x started 2024-09-13T14:10:10.345 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:10 smithi102 ceph-mon[29355]: from='mgr.? 172.21.15.195:0/1634937924' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:10:10.345 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:10 smithi102 ceph-mon[29355]: from='mgr.? 172.21.15.195:0/1634937924' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:10:10.345 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:10 smithi102 ceph-mon[29355]: from='mgr.? 172.21.15.195:0/1634937924' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:10:10.345 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:10 smithi102 ceph-mon[29355]: from='mgr.? 172.21.15.195:0/1634937924' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:10:10.345 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:10 smithi102 ceph-mon[29355]: mgrmap e18: y(active, since 1.01528s), standbys: x 2024-09-13T14:10:10.345 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:10 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:10.345 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:10 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:10:10.346 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:10 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:10:10.346 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:10 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:10:10.346 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:10 smithi102 ceph-mon[40543]: Standby manager daemon x restarted 2024-09-13T14:10:10.346 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:10 smithi102 ceph-mon[40543]: Standby manager daemon x started 2024-09-13T14:10:10.346 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:10 smithi102 ceph-mon[40543]: from='mgr.? 172.21.15.195:0/1634937924' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:10:10.346 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:10 smithi102 ceph-mon[40543]: from='mgr.? 172.21.15.195:0/1634937924' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:10:10.346 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:10 smithi102 ceph-mon[40543]: from='mgr.? 172.21.15.195:0/1634937924' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:10:10.346 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:10 smithi102 ceph-mon[40543]: from='mgr.? 172.21.15.195:0/1634937924' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:10:10.346 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:10 smithi102 ceph-mon[40543]: mgrmap e18: y(active, since 1.01528s), standbys: x 2024-09-13T14:10:10.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:10 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:10.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:10 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:10:10.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:10 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:10:10.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:10 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:10:10.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:10 smithi195 ceph-mon[33205]: Standby manager daemon x restarted 2024-09-13T14:10:10.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:10 smithi195 ceph-mon[33205]: Standby manager daemon x started 2024-09-13T14:10:10.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:10 smithi195 ceph-mon[33205]: from='mgr.? 172.21.15.195:0/1634937924' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:10:10.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:10 smithi195 ceph-mon[33205]: from='mgr.? 172.21.15.195:0/1634937924' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:10:10.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:10 smithi195 ceph-mon[33205]: from='mgr.? 172.21.15.195:0/1634937924' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:10:10.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:10 smithi195 ceph-mon[33205]: from='mgr.? 172.21.15.195:0/1634937924' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:10:10.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:10 smithi195 ceph-mon[33205]: mgrmap e18: y(active, since 1.01528s), standbys: x 2024-09-13T14:10:10.669 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:10 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:10:10.404+0000 7f059e69a640 -1 log_channel(cephadm) log [ERR] : [13/Sep/2024:14:10:10] ENGINE Error in HTTPServer.serve 2024-09-13T14:10:10.669 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:10 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: Traceback (most recent call last): 2024-09-13T14:10:10.669 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:10 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:10:10.670 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:10 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: self._connections.run(self.expiration_interval) 2024-09-13T14:10:10.670 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:10 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:10:10.670 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:10 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: self._run(expiration_interval) 2024-09-13T14:10:10.670 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:10 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:10:10.670 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:10 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:10:10.671 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:10 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:10:10.671 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:10 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:10:10.671 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:10 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:10:10.671 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:10 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: s = self.context.wrap_socket( 2024-09-13T14:10:10.671 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:10 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:10:10.671 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:10 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: return self.sslsocket_class._create( 2024-09-13T14:10:10.672 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:10 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:10:10.672 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:10 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: self.do_handshake() 2024-09-13T14:10:10.672 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:10 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:10:10.672 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:10 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: self._sslobj.do_handshake() 2024-09-13T14:10:10.672 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:10 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:10:10.673 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:10 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:10:11.113 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'rbd' created 2024-09-13T14:10:11.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:11 smithi102 ceph-mon[40543]: [13/Sep/2024:14:10:10] ENGINE Bus STARTING 2024-09-13T14:10:11.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:11 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/2785739831' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-09-13T14:10:11.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:11 smithi102 ceph-mon[29355]: [13/Sep/2024:14:10:10] ENGINE Bus STARTING 2024-09-13T14:10:11.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:11 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/2785739831' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-09-13T14:10:11.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:11 smithi195 ceph-mon[33205]: [13/Sep/2024:14:10:10] ENGINE Bus STARTING 2024-09-13T14:10:11.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:11 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/2785739831' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-09-13T14:10:12.044 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- rbd pool init rbd 2024-09-13T14:10:12.267 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:10:12.380 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:12 smithi102 ceph-mon[29355]: [13/Sep/2024:14:10:10] ENGINE Serving on https://172.21.15.102:7150 2024-09-13T14:10:12.380 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:12 smithi102 ceph-mon[29355]: [13/Sep/2024:14:10:10] ENGINE Error in HTTPServer.serve 2024-09-13T14:10:12.380 INFO:journalctl@ceph.mon.a.smithi102.stdout: Traceback (most recent call last): 2024-09-13T14:10:12.380 INFO:journalctl@ceph.mon.a.smithi102.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:10:12.381 INFO:journalctl@ceph.mon.a.smithi102.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:10:12.381 INFO:journalctl@ceph.mon.a.smithi102.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:10:12.381 INFO:journalctl@ceph.mon.a.smithi102.stdout: self._run(expiration_interval) 2024-09-13T14:10:12.381 INFO:journalctl@ceph.mon.a.smithi102.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:10:12.381 INFO:journalctl@ceph.mon.a.smithi102.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:10:12.381 INFO:journalctl@ceph.mon.a.smithi102.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:10:12.381 INFO:journalctl@ceph.mon.a.smithi102.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:10:12.381 INFO:journalctl@ceph.mon.a.smithi102.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:10:12.381 INFO:journalctl@ceph.mon.a.smithi102.stdout: s = self.context.wrap_socket( 2024-09-13T14:10:12.381 INFO:journalctl@ceph.mon.a.smithi102.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:10:12.381 INFO:journalctl@ceph.mon.a.smithi102.stdout: return self.sslsocket_class._create( 2024-09-13T14:10:12.381 INFO:journalctl@ceph.mon.a.smithi102.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:10:12.381 INFO:journalctl@ceph.mon.a.smithi102.stdout: self.do_handshake() 2024-09-13T14:10:12.381 INFO:journalctl@ceph.mon.a.smithi102.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:10:12.382 INFO:journalctl@ceph.mon.a.smithi102.stdout: self._sslobj.do_handshake() 2024-09-13T14:10:12.382 INFO:journalctl@ceph.mon.a.smithi102.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:10:12.382 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:12 smithi102 ceph-mon[29355]: [13/Sep/2024:14:10:10] ENGINE Serving on http://172.21.15.102:8765 2024-09-13T14:10:12.382 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:12 smithi102 ceph-mon[29355]: [13/Sep/2024:14:10:10] ENGINE Bus STARTED 2024-09-13T14:10:12.382 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:12 smithi102 ceph-mon[29355]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:12.382 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:12 smithi102 ceph-mon[29355]: mgrmap e19: y(active, since 2s), standbys: x 2024-09-13T14:10:12.382 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:12 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/2785739831' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]': finished 2024-09-13T14:10:12.382 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:12 smithi102 ceph-mon[29355]: osdmap e60: 8 total, 8 up, 8 in 2024-09-13T14:10:12.382 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:12 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:12.382 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:12 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:12.382 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:12 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:12.382 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:12 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:12.382 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:12 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:10:12.383 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:12 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:10:12.383 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:12 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:10:12.383 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:12 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:10:12.383 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:12 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:12.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:12 smithi102 ceph-mon[40543]: [13/Sep/2024:14:10:10] ENGINE Serving on https://172.21.15.102:7150 2024-09-13T14:10:12.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:12 smithi102 ceph-mon[40543]: [13/Sep/2024:14:10:10] ENGINE Error in HTTPServer.serve 2024-09-13T14:10:12.383 INFO:journalctl@ceph.mon.c.smithi102.stdout: Traceback (most recent call last): 2024-09-13T14:10:12.383 INFO:journalctl@ceph.mon.c.smithi102.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:10:12.383 INFO:journalctl@ceph.mon.c.smithi102.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:10:12.383 INFO:journalctl@ceph.mon.c.smithi102.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:10:12.383 INFO:journalctl@ceph.mon.c.smithi102.stdout: self._run(expiration_interval) 2024-09-13T14:10:12.383 INFO:journalctl@ceph.mon.c.smithi102.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:10:12.384 INFO:journalctl@ceph.mon.c.smithi102.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:10:12.384 INFO:journalctl@ceph.mon.c.smithi102.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:10:12.384 INFO:journalctl@ceph.mon.c.smithi102.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:10:12.384 INFO:journalctl@ceph.mon.c.smithi102.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:10:12.384 INFO:journalctl@ceph.mon.c.smithi102.stdout: s = self.context.wrap_socket( 2024-09-13T14:10:12.384 INFO:journalctl@ceph.mon.c.smithi102.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:10:12.384 INFO:journalctl@ceph.mon.c.smithi102.stdout: return self.sslsocket_class._create( 2024-09-13T14:10:12.384 INFO:journalctl@ceph.mon.c.smithi102.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:10:12.384 INFO:journalctl@ceph.mon.c.smithi102.stdout: self.do_handshake() 2024-09-13T14:10:12.384 INFO:journalctl@ceph.mon.c.smithi102.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:10:12.384 INFO:journalctl@ceph.mon.c.smithi102.stdout: self._sslobj.do_handshake() 2024-09-13T14:10:12.384 INFO:journalctl@ceph.mon.c.smithi102.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:10:12.384 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:12 smithi102 ceph-mon[40543]: [13/Sep/2024:14:10:10] ENGINE Serving on http://172.21.15.102:8765 2024-09-13T14:10:12.385 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:12 smithi102 ceph-mon[40543]: [13/Sep/2024:14:10:10] ENGINE Bus STARTED 2024-09-13T14:10:12.385 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:12 smithi102 ceph-mon[40543]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:12.385 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:12 smithi102 ceph-mon[40543]: mgrmap e19: y(active, since 2s), standbys: x 2024-09-13T14:10:12.385 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:12 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/2785739831' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]': finished 2024-09-13T14:10:12.385 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:12 smithi102 ceph-mon[40543]: osdmap e60: 8 total, 8 up, 8 in 2024-09-13T14:10:12.385 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:12 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:12.385 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:12 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:12.385 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:12 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:12.385 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:12 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:12.385 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:12 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:10:12.385 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:12 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:10:12.385 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:12 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:10:12.385 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:12 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:10:12.385 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:12 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:12.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:12 smithi195 ceph-mon[33205]: [13/Sep/2024:14:10:10] ENGINE Serving on https://172.21.15.102:7150 2024-09-13T14:10:12.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:12 smithi195 ceph-mon[33205]: [13/Sep/2024:14:10:10] ENGINE Error in HTTPServer.serve 2024-09-13T14:10:12.498 INFO:journalctl@ceph.mon.b.smithi195.stdout: Traceback (most recent call last): 2024-09-13T14:10:12.498 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:10:12.498 INFO:journalctl@ceph.mon.b.smithi195.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:10:12.498 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:10:12.498 INFO:journalctl@ceph.mon.b.smithi195.stdout: self._run(expiration_interval) 2024-09-13T14:10:12.498 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:10:12.498 INFO:journalctl@ceph.mon.b.smithi195.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:10:12.498 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:10:12.498 INFO:journalctl@ceph.mon.b.smithi195.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:10:12.498 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:10:12.498 INFO:journalctl@ceph.mon.b.smithi195.stdout: s = self.context.wrap_socket( 2024-09-13T14:10:12.499 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:10:12.499 INFO:journalctl@ceph.mon.b.smithi195.stdout: return self.sslsocket_class._create( 2024-09-13T14:10:12.499 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:10:12.499 INFO:journalctl@ceph.mon.b.smithi195.stdout: self.do_handshake() 2024-09-13T14:10:12.499 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:10:12.499 INFO:journalctl@ceph.mon.b.smithi195.stdout: self._sslobj.do_handshake() 2024-09-13T14:10:12.499 INFO:journalctl@ceph.mon.b.smithi195.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:10:12.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:12 smithi195 ceph-mon[33205]: [13/Sep/2024:14:10:10] ENGINE Serving on http://172.21.15.102:8765 2024-09-13T14:10:12.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:12 smithi195 ceph-mon[33205]: [13/Sep/2024:14:10:10] ENGINE Bus STARTED 2024-09-13T14:10:12.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:12 smithi195 ceph-mon[33205]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:12.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:12 smithi195 ceph-mon[33205]: mgrmap e19: y(active, since 2s), standbys: x 2024-09-13T14:10:12.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:12 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/2785739831' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]': finished 2024-09-13T14:10:12.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:12 smithi195 ceph-mon[33205]: osdmap e60: 8 total, 8 up, 8 in 2024-09-13T14:10:12.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:12 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:12.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:12 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:12.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:12 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:12.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:12 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:12.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:12 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:10:12.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:12 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:10:12.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:12 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:10:12.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:12 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:10:12.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:12 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:13.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:13 smithi102 ceph-mon[29355]: Adjusting osd_memory_target on smithi195 to 4016M 2024-09-13T14:10:13.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:13 smithi102 ceph-mon[29355]: osdmap e61: 8 total, 8 up, 8 in 2024-09-13T14:10:13.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:13 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:13.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:13 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:13.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:13 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:13.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:13 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:13.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:13 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:10:13.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:13 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:10:13.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:13 smithi102 ceph-mon[40543]: Adjusting osd_memory_target on smithi195 to 4016M 2024-09-13T14:10:13.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:13 smithi102 ceph-mon[40543]: osdmap e61: 8 total, 8 up, 8 in 2024-09-13T14:10:13.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:13 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:13.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:13 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:13.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:13 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:13.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:13 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:13.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:13 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:10:13.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:13 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:10:13.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:13 smithi195 ceph-mon[33205]: Adjusting osd_memory_target on smithi195 to 4016M 2024-09-13T14:10:13.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:13 smithi195 ceph-mon[33205]: osdmap e61: 8 total, 8 up, 8 in 2024-09-13T14:10:13.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:13 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:13.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:13 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:13.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:13 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:13.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:13 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:13.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:13 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:10:13.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:13 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:10:14.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:14 smithi102 ceph-mon[29355]: Updating smithi102:/etc/ceph/ceph.conf 2024-09-13T14:10:14.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:14 smithi102 ceph-mon[29355]: Updating smithi195:/etc/ceph/ceph.conf 2024-09-13T14:10:14.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:14 smithi102 ceph-mon[29355]: pgmap v7: 9 pgs: 8 unknown, 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:14.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:14 smithi102 ceph-mon[29355]: osdmap e62: 8 total, 8 up, 8 in 2024-09-13T14:10:14.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:14 smithi102 ceph-mon[29355]: Updating smithi102:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:10:14.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:14 smithi102 ceph-mon[29355]: Updating smithi195:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:10:14.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:14 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:14.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:14 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:14.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:14 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:14.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:14 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:14.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:14 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:14.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:14 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/1993608799' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-13T14:10:14.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:14 smithi102 ceph-mon[29355]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-13T14:10:14.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:14 smithi102 ceph-mon[29355]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:10:14.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:14 smithi102 ceph-mon[29355]: mgrmap e20: y(active, since 4s), standbys: x 2024-09-13T14:10:14.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:14 smithi102 ceph-mon[40543]: Updating smithi102:/etc/ceph/ceph.conf 2024-09-13T14:10:14.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:14 smithi102 ceph-mon[40543]: Updating smithi195:/etc/ceph/ceph.conf 2024-09-13T14:10:14.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:14 smithi102 ceph-mon[40543]: pgmap v7: 9 pgs: 8 unknown, 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:14.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:14 smithi102 ceph-mon[40543]: osdmap e62: 8 total, 8 up, 8 in 2024-09-13T14:10:14.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:14 smithi102 ceph-mon[40543]: Updating smithi102:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:10:14.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:14 smithi102 ceph-mon[40543]: Updating smithi195:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:10:14.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:14 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:14.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:14 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:14.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:14 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:14.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:14 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:14.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:14 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:14.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:14 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/1993608799' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-13T14:10:14.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:14 smithi102 ceph-mon[40543]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-13T14:10:14.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:14 smithi102 ceph-mon[40543]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:10:14.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:14 smithi102 ceph-mon[40543]: mgrmap e20: y(active, since 4s), standbys: x 2024-09-13T14:10:14.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:14 smithi195 ceph-mon[33205]: Updating smithi102:/etc/ceph/ceph.conf 2024-09-13T14:10:14.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:14 smithi195 ceph-mon[33205]: Updating smithi195:/etc/ceph/ceph.conf 2024-09-13T14:10:14.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:14 smithi195 ceph-mon[33205]: pgmap v7: 9 pgs: 8 unknown, 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:14.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:14 smithi195 ceph-mon[33205]: osdmap e62: 8 total, 8 up, 8 in 2024-09-13T14:10:14.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:14 smithi195 ceph-mon[33205]: Updating smithi102:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:10:14.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:14 smithi195 ceph-mon[33205]: Updating smithi195:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:10:14.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:14 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:14.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:14 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:14.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:14 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:14.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:14 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:14.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:14 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:14.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:14 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/1993608799' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-13T14:10:14.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:14 smithi195 ceph-mon[33205]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-13T14:10:14.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:14 smithi195 ceph-mon[33205]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:10:14.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:14 smithi195 ceph-mon[33205]: mgrmap e20: y(active, since 4s), standbys: x 2024-09-13T14:10:15.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:15 smithi102 ceph-mon[29355]: Updating smithi102:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:10:15.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:15 smithi102 ceph-mon[29355]: Updating smithi195:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:10:15.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:15 smithi102 ceph-mon[29355]: Updating smithi102:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:10:15.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:15 smithi102 ceph-mon[29355]: Updating smithi195:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:10:15.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:15 smithi102 ceph-mon[29355]: Deploying daemon alertmanager.a on smithi102 2024-09-13T14:10:15.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:15 smithi102 ceph-mon[29355]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]': finished 2024-09-13T14:10:15.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:15 smithi102 ceph-mon[29355]: osdmap e63: 8 total, 8 up, 8 in 2024-09-13T14:10:15.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:15 smithi102 ceph-mon[40543]: Updating smithi102:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:10:15.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:15 smithi102 ceph-mon[40543]: Updating smithi195:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:10:15.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:15 smithi102 ceph-mon[40543]: Updating smithi102:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:10:15.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:15 smithi102 ceph-mon[40543]: Updating smithi195:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:10:15.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:15 smithi102 ceph-mon[40543]: Deploying daemon alertmanager.a on smithi102 2024-09-13T14:10:15.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:15 smithi102 ceph-mon[40543]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]': finished 2024-09-13T14:10:15.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:15 smithi102 ceph-mon[40543]: osdmap e63: 8 total, 8 up, 8 in 2024-09-13T14:10:15.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:15 smithi195 ceph-mon[33205]: Updating smithi102:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:10:15.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:15 smithi195 ceph-mon[33205]: Updating smithi195:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:10:15.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:15 smithi195 ceph-mon[33205]: Updating smithi102:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:10:15.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:15 smithi195 ceph-mon[33205]: Updating smithi195:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:10:15.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:15 smithi195 ceph-mon[33205]: Deploying daemon alertmanager.a on smithi102 2024-09-13T14:10:15.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:15 smithi195 ceph-mon[33205]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]': finished 2024-09-13T14:10:15.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:15 smithi195 ceph-mon[33205]: osdmap e63: 8 total, 8 up, 8 in 2024-09-13T14:10:16.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:16 smithi102 ceph-mon[29355]: pgmap v10: 9 pgs: 8 unknown, 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:16.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:16 smithi102 ceph-mon[29355]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:10:16.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:16 smithi102 ceph-mon[29355]: Cluster is now healthy 2024-09-13T14:10:16.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:16 smithi102 ceph-mon[29355]: osdmap e64: 8 total, 8 up, 8 in 2024-09-13T14:10:16.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:16 smithi102 ceph-mon[40543]: pgmap v10: 9 pgs: 8 unknown, 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:16.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:16 smithi102 ceph-mon[40543]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:10:16.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:16 smithi102 ceph-mon[40543]: Cluster is now healthy 2024-09-13T14:10:16.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:16 smithi102 ceph-mon[40543]: osdmap e64: 8 total, 8 up, 8 in 2024-09-13T14:10:16.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:16 smithi195 ceph-mon[33205]: pgmap v10: 9 pgs: 8 unknown, 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:16.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:16 smithi195 ceph-mon[33205]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:10:16.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:16 smithi195 ceph-mon[33205]: Cluster is now healthy 2024-09-13T14:10:16.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:16 smithi195 ceph-mon[33205]: osdmap e64: 8 total, 8 up, 8 in 2024-09-13T14:10:17.669 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:17 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:10:17] "GET /metrics HTTP/1.1" 200 23685 "" "Prometheus/2.43.0" 2024-09-13T14:10:18.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:17 smithi102 ceph-mon[29355]: osdmap e65: 8 total, 8 up, 8 in 2024-09-13T14:10:18.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:17 smithi102 ceph-mon[29355]: pgmap v13: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 28 op/s 2024-09-13T14:10:18.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:17 smithi102 ceph-mon[40543]: osdmap e65: 8 total, 8 up, 8 in 2024-09-13T14:10:18.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:17 smithi102 ceph-mon[40543]: pgmap v13: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 28 op/s 2024-09-13T14:10:18.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:17 smithi195 ceph-mon[33205]: osdmap e65: 8 total, 8 up, 8 in 2024-09-13T14:10:18.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:17 smithi195 ceph-mon[33205]: pgmap v13: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 28 op/s 2024-09-13T14:10:18.328 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph config log 1 --format=json 2024-09-13T14:10:19.551 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:10:20.288 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:20 smithi102 ceph-mon[29355]: pgmap v14: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 19 op/s 2024-09-13T14:10:20.288 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:20 smithi102 ceph-mon[40543]: pgmap v14: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 19 op/s 2024-09-13T14:10:20.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:20 smithi195 ceph-mon[33205]: pgmap v14: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 19 op/s 2024-09-13T14:10:22.267 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:22 smithi102 ceph-mon[29355]: pgmap v15: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 332 B/s wr, 19 op/s 2024-09-13T14:10:22.268 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:22 smithi102 ceph-mon[40543]: pgmap v15: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 332 B/s wr, 19 op/s 2024-09-13T14:10:22.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:22 smithi195 ceph-mon[33205]: pgmap v15: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 332 B/s wr, 19 op/s 2024-09-13T14:10:23.168 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:10:23 smithi102 podman[70817]: 2024-09-13 14:10:23.133127496 +0000 UTC m=+0.434435723 container create 2523493d7218c308898e1befbc333be659b4324ddb3809a8b70da261b96401b6 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:10:23.762 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:23 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: [13/Sep/2024:14:10:23] ENGINE Bus STOPPING 2024-09-13T14:10:23.763 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:10:23 smithi102 podman[70817]: 2024-09-13 14:10:23.452709128 +0000 UTC m=+0.754017312 container init 2523493d7218c308898e1befbc333be659b4324ddb3809a8b70da261b96401b6 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:10:23.763 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:10:23 smithi102 podman[70817]: 2024-09-13 14:10:23.457490656 +0000 UTC m=+0.758798847 container start 2523493d7218c308898e1befbc333be659b4324ddb3809a8b70da261b96401b6 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:10:23.763 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:10:23 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a[70855]: ts=2024-09-13T14:10:23.469Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-09-13T14:10:23.763 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:10:23 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a[70855]: ts=2024-09-13T14:10:23.469Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-09-13T14:10:23.763 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:10:23 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a[70855]: ts=2024-09-13T14:10:23.470Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.102 port=9094 2024-09-13T14:10:23.763 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:10:23 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a[70855]: ts=2024-09-13T14:10:23.471Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-13T14:10:23.763 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:10:23 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a[70855]: ts=2024-09-13T14:10:23.494Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-13T14:10:23.763 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:10:23 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a[70855]: ts=2024-09-13T14:10:23.494Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-13T14:10:23.763 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:10:23 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a[70855]: ts=2024-09-13T14:10:23.497Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-09-13T14:10:23.763 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:10:23 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a[70855]: ts=2024-09-13T14:10:23.497Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-13T14:10:23.763 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:10:23 smithi102 bash[70817]: 2523493d7218c308898e1befbc333be659b4324ddb3809a8b70da261b96401b6 2024-09-13T14:10:23.763 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:10:23 smithi102 systemd[1]: Started Ceph alertmanager.a for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:10:23.937 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:10:24.028 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:23 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: [13/Sep/2024:14:10:23] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-13T14:10:24.028 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:23 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: [13/Sep/2024:14:10:23] ENGINE Bus STOPPED 2024-09-13T14:10:24.029 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:23 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: [13/Sep/2024:14:10:23] ENGINE Bus STARTING 2024-09-13T14:10:24.029 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:23 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: [13/Sep/2024:14:10:23] ENGINE Serving on http://:::9283 2024-09-13T14:10:24.029 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:23 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: [13/Sep/2024:14:10:23] ENGINE Bus STARTED 2024-09-13T14:10:24.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:24 smithi102 ceph-mon[40543]: osdmap e66: 8 total, 8 up, 8 in 2024-09-13T14:10:24.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:24 smithi102 ceph-mon[40543]: pgmap v17: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 286 B/s wr, 0 op/s 2024-09-13T14:10:24.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:24 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:24.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:24 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:24.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:24 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:24.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:24 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:24.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:24 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:24.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:24 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:24.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:24 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-13T14:10:24.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:24 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:24.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:24 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/3924799082' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-09-13T14:10:24.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:24 smithi102 ceph-mon[29355]: osdmap e66: 8 total, 8 up, 8 in 2024-09-13T14:10:24.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:24 smithi102 ceph-mon[29355]: pgmap v17: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 286 B/s wr, 0 op/s 2024-09-13T14:10:24.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:24 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:24.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:24 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:24.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:24 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:24.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:24 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:24.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:24 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:24.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:24 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:24.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:24 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-13T14:10:24.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:24 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:24.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:24 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/3924799082' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-09-13T14:10:24.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:24 smithi195 ceph-mon[33205]: osdmap e66: 8 total, 8 up, 8 in 2024-09-13T14:10:24.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:24 smithi195 ceph-mon[33205]: pgmap v17: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 286 B/s wr, 0 op/s 2024-09-13T14:10:24.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:24 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:24.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:24 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:24.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:24 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:24.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:24 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:24.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:24 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:24.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:24 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:24.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:24 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-13T14:10:24.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:24 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:24.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:24 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/3924799082' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-09-13T14:10:24.570 INFO:teuthology.orchestra.run.smithi102.stdout:[{"version":23,"timestamp":"2024-09-13T14:10:23.667581+0000","name":"","changes":[{"name":"mgr/mgr/dashboard/GRAFANA_API_SSL_VERIFY","new_value":"false"}]}] 2024-09-13T14:10:24.571 INFO:tasks.ceph_manager:config epoch is 23 2024-09-13T14:10:24.571 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-13T14:10:24.571 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-13T14:10:24.572 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph mgr dump --format=json 2024-09-13T14:10:24.767 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:10:25.145 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:25 smithi102 ceph-mon[29355]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-13T14:10:25.145 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:25 smithi102 ceph-mon[29355]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-13T14:10:25.145 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:25 smithi102 ceph-mon[29355]: Deploying daemon grafana.a on smithi195 2024-09-13T14:10:25.145 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:25 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:25.145 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:25 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:10:25.146 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:25 smithi102 ceph-mon[40543]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-13T14:10:25.146 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:25 smithi102 ceph-mon[40543]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-13T14:10:25.146 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:25 smithi102 ceph-mon[40543]: Deploying daemon grafana.a on smithi195 2024-09-13T14:10:25.146 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:25 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:25.146 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:25 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:10:25.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:25 smithi195 ceph-mon[33205]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-13T14:10:25.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:25 smithi195 ceph-mon[33205]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-13T14:10:25.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:25 smithi195 ceph-mon[33205]: Deploying daemon grafana.a on smithi195 2024-09-13T14:10:25.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:25 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:25.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:25 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:10:25.918 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:10:25 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a[70855]: ts=2024-09-13T14:10:25.471Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000652234s 2024-09-13T14:10:26.300 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:26 smithi102 ceph-mon[29355]: pgmap v18: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 251 B/s wr, 0 op/s 2024-09-13T14:10:26.301 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:26 smithi102 ceph-mon[40543]: pgmap v18: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 251 B/s wr, 0 op/s 2024-09-13T14:10:26.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:26 smithi195 ceph-mon[33205]: pgmap v18: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 251 B/s wr, 0 op/s 2024-09-13T14:10:26.724 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:10:27.381 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:27 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/2348289121' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-13T14:10:27.381 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:27 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/2348289121' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-13T14:10:27.438 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":20,"flags":0,"active_gid":14421,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":1522897266},{"type":"v1","addr":"172.21.15.102:6801","nonce":1522897266}]},"active_addr":"172.21.15.102:6801/1522897266","active_change":"2024-09-13T14:10:09.037804+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":24319,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.102:8443/","prometheus":"http://172.21.15.102:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":59,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":2083488703}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":2489373272}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":1430433627}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":1792178465}]}]} 2024-09-13T14:10:27.443 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-13T14:10:27.443 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-13T14:10:27.444 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-13T14:10:27.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:27 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/2348289121' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-13T14:10:27.643 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:10:27.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:27 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:10:27] "GET /metrics HTTP/1.1" 200 28524 "" "Prometheus/2.43.0" 2024-09-13T14:10:28.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:28 smithi102 ceph-mon[40543]: pgmap v19: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 204 B/s wr, 0 op/s 2024-09-13T14:10:28.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:28 smithi102 ceph-mon[29355]: pgmap v19: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 204 B/s wr, 0 op/s 2024-09-13T14:10:28.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:28 smithi195 ceph-mon[33205]: pgmap v19: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 204 B/s wr, 0 op/s 2024-09-13T14:10:29.573 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:10:29.573 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":66,"fsid":"08cee502-71d9-11ef-bceb-c7b262605968","created":"2024-09-13T14:04:48.179879+0000","modified":"2024-09-13T14:10:23.021848+0000","last_up_change":"2024-09-13T14:09:36.412493+0000","last_in_change":"2024-09-13T14:09:19.207877+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T14:07:41.927240+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T14:10:10.981300+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":65,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b43b49e0-1cf0-47b8-aa85-37108f321220","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":466204285},{"type":"v1","addr":"172.21.15.102:6803","nonce":466204285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":466204285},{"type":"v1","addr":"172.21.15.102:6805","nonce":466204285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":466204285},{"type":"v1","addr":"172.21.15.102:6809","nonce":466204285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":466204285},{"type":"v1","addr":"172.21.15.102:6807","nonce":466204285}]},"public_addr":"172.21.15.102:6803/466204285","cluster_addr":"172.21.15.102:6805/466204285","heartbeat_back_addr":"172.21.15.102:6809/466204285","heartbeat_front_addr":"172.21.15.102:6807/466204285","state":["exists","up"]},{"osd":1,"uuid":"27e10d5e-c471-4132-9c73-83c5708c3b08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":1293422793},{"type":"v1","addr":"172.21.15.102:6811","nonce":1293422793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":1293422793},{"type":"v1","addr":"172.21.15.102:6813","nonce":1293422793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1293422793},{"type":"v1","addr":"172.21.15.102:6817","nonce":1293422793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":1293422793},{"type":"v1","addr":"172.21.15.102:6815","nonce":1293422793}]},"public_addr":"172.21.15.102:6811/1293422793","cluster_addr":"172.21.15.102:6813/1293422793","heartbeat_back_addr":"172.21.15.102:6817/1293422793","heartbeat_front_addr":"172.21.15.102:6815/1293422793","state":["exists","up"]},{"osd":2,"uuid":"40a94b2b-2c24-4164-91be-0816a1acf3d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":360502781},{"type":"v1","addr":"172.21.15.102:6819","nonce":360502781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":360502781},{"type":"v1","addr":"172.21.15.102:6821","nonce":360502781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":360502781},{"type":"v1","addr":"172.21.15.102:6825","nonce":360502781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":360502781},{"type":"v1","addr":"172.21.15.102:6823","nonce":360502781}]},"public_addr":"172.21.15.102:6819/360502781","cluster_addr":"172.21.15.102:6821/360502781","heartbeat_back_addr":"172.21.15.102:6825/360502781","heartbeat_front_addr":"172.21.15.102:6823/360502781","state":["exists","up"]},{"osd":3,"uuid":"2ed85faf-6610-4bcc-a09d-4a6f93f5c6a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":3709228546},{"type":"v1","addr":"172.21.15.102:6827","nonce":3709228546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":3709228546},{"type":"v1","addr":"172.21.15.102:6829","nonce":3709228546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3709228546},{"type":"v1","addr":"172.21.15.102:6833","nonce":3709228546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":3709228546},{"type":"v1","addr":"172.21.15.102:6831","nonce":3709228546}]},"public_addr":"172.21.15.102:6827/3709228546","cluster_addr":"172.21.15.102:6829/3709228546","heartbeat_back_addr":"172.21.15.102:6833/3709228546","heartbeat_front_addr":"172.21.15.102:6831/3709228546","state":["exists","up"]},{"osd":4,"uuid":"8a45f350-fb3e-4049-bcc4-532d3e20c238","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":2821651815},{"type":"v1","addr":"172.21.15.195:6801","nonce":2821651815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":2821651815},{"type":"v1","addr":"172.21.15.195:6803","nonce":2821651815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":2821651815},{"type":"v1","addr":"172.21.15.195:6807","nonce":2821651815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":2821651815},{"type":"v1","addr":"172.21.15.195:6805","nonce":2821651815}]},"public_addr":"172.21.15.195:6801/2821651815","cluster_addr":"172.21.15.195:6803/2821651815","heartbeat_back_addr":"172.21.15.195:6807/2821651815","heartbeat_front_addr":"172.21.15.195:6805/2821651815","state":["exists","up"]},{"osd":5,"uuid":"363cf638-db26-4c86-b2be-e075fd09ac16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":3074139237},{"type":"v1","addr":"172.21.15.195:6809","nonce":3074139237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":3074139237},{"type":"v1","addr":"172.21.15.195:6811","nonce":3074139237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":3074139237},{"type":"v1","addr":"172.21.15.195:6815","nonce":3074139237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":3074139237},{"type":"v1","addr":"172.21.15.195:6813","nonce":3074139237}]},"public_addr":"172.21.15.195:6809/3074139237","cluster_addr":"172.21.15.195:6811/3074139237","heartbeat_back_addr":"172.21.15.195:6815/3074139237","heartbeat_front_addr":"172.21.15.195:6813/3074139237","state":["exists","up"]},{"osd":6,"uuid":"f37416e8-9c25-4860-a32f-0d1c404a8bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":4026627594},{"type":"v1","addr":"172.21.15.195:6817","nonce":4026627594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":4026627594},{"type":"v1","addr":"172.21.15.195:6819","nonce":4026627594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":4026627594},{"type":"v1","addr":"172.21.15.195:6823","nonce":4026627594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":4026627594},{"type":"v1","addr":"172.21.15.195:6821","nonce":4026627594}]},"public_addr":"172.21.15.195:6817/4026627594","cluster_addr":"172.21.15.195:6819/4026627594","heartbeat_back_addr":"172.21.15.195:6823/4026627594","heartbeat_front_addr":"172.21.15.195:6821/4026627594","state":["exists","up"]},{"osd":7,"uuid":"9a384a4d-fddb-4fb3-843a-77c19cca0368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2059403787},{"type":"v1","addr":"172.21.15.195:6825","nonce":2059403787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2059403787},{"type":"v1","addr":"172.21.15.195:6827","nonce":2059403787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2059403787},{"type":"v1","addr":"172.21.15.195:6831","nonce":2059403787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2059403787},{"type":"v1","addr":"172.21.15.195:6829","nonce":2059403787}]},"public_addr":"172.21.15.195:6825/2059403787","cluster_addr":"172.21.15.195:6827/2059403787","heartbeat_back_addr":"172.21.15.195:6831/2059403787","heartbeat_front_addr":"172.21.15.195:6829/2059403787","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:06:53.502746+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:07:16.251669+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:07:38.848272+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:08:02.013035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:08:23.043482+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:08:45.653159+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:09:10.187018+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:09:34.393753+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/1765904351":"2024-09-14T14:10:09.037657+0000","172.21.15.102:0/2416257718":"2024-09-14T14:10:09.037657+0000","172.21.15.102:0/1582001159":"2024-09-14T14:05:15.286479+0000","172.21.15.102:0/2308924552":"2024-09-14T14:10:09.037657+0000","172.21.15.102:6800/2761043787":"2024-09-14T14:05:15.286479+0000","172.21.15.102:6801/2761043787":"2024-09-14T14:05:15.286479+0000","172.21.15.102:0/2444394840":"2024-09-14T14:05:37.634624+0000","172.21.15.102:0/1574792592":"2024-09-14T14:05:15.286479+0000","172.21.15.102:0/3189088157":"2024-09-14T14:05:15.286479+0000","172.21.15.102:6800/3480998245":"2024-09-14T14:05:37.634624+0000","172.21.15.102:6801/3480998245":"2024-09-14T14:05:37.634624+0000","172.21.15.102:0/1160237252":"2024-09-14T14:10:09.037657+0000","172.21.15.102:0/3399546099":"2024-09-14T14:05:37.634624+0000","172.21.15.102:0/1939570054":"2024-09-14T14:05:37.634624+0000","172.21.15.102:6800/63514412":"2024-09-14T14:10:09.037657+0000","172.21.15.102:6801/63514412":"2024-09-14T14:10:09.037657+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T14:10:30.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:30 smithi102 ceph-mon[29355]: pgmap v20: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 204 B/s wr, 0 op/s 2024-09-13T14:10:30.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:30 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/2056907805' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:10:30.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:30 smithi102 ceph-mon[40543]: pgmap v20: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 204 B/s wr, 0 op/s 2024-09-13T14:10:30.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:30 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/2056907805' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:10:30.235 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-13T14:10:30.236 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-13T14:10:30.431 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:10:30.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:30 smithi195 ceph-mon[33205]: pgmap v20: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 204 B/s wr, 0 op/s 2024-09-13T14:10:30.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:30 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/2056907805' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:10:32.303 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:10:32.304 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":66,"fsid":"08cee502-71d9-11ef-bceb-c7b262605968","created":"2024-09-13T14:04:48.179879+0000","modified":"2024-09-13T14:10:23.021848+0000","last_up_change":"2024-09-13T14:09:36.412493+0000","last_in_change":"2024-09-13T14:09:19.207877+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T14:07:41.927240+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T14:10:10.981300+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":65,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b43b49e0-1cf0-47b8-aa85-37108f321220","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":466204285},{"type":"v1","addr":"172.21.15.102:6803","nonce":466204285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":466204285},{"type":"v1","addr":"172.21.15.102:6805","nonce":466204285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":466204285},{"type":"v1","addr":"172.21.15.102:6809","nonce":466204285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":466204285},{"type":"v1","addr":"172.21.15.102:6807","nonce":466204285}]},"public_addr":"172.21.15.102:6803/466204285","cluster_addr":"172.21.15.102:6805/466204285","heartbeat_back_addr":"172.21.15.102:6809/466204285","heartbeat_front_addr":"172.21.15.102:6807/466204285","state":["exists","up"]},{"osd":1,"uuid":"27e10d5e-c471-4132-9c73-83c5708c3b08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":1293422793},{"type":"v1","addr":"172.21.15.102:6811","nonce":1293422793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":1293422793},{"type":"v1","addr":"172.21.15.102:6813","nonce":1293422793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1293422793},{"type":"v1","addr":"172.21.15.102:6817","nonce":1293422793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":1293422793},{"type":"v1","addr":"172.21.15.102:6815","nonce":1293422793}]},"public_addr":"172.21.15.102:6811/1293422793","cluster_addr":"172.21.15.102:6813/1293422793","heartbeat_back_addr":"172.21.15.102:6817/1293422793","heartbeat_front_addr":"172.21.15.102:6815/1293422793","state":["exists","up"]},{"osd":2,"uuid":"40a94b2b-2c24-4164-91be-0816a1acf3d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":360502781},{"type":"v1","addr":"172.21.15.102:6819","nonce":360502781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":360502781},{"type":"v1","addr":"172.21.15.102:6821","nonce":360502781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":360502781},{"type":"v1","addr":"172.21.15.102:6825","nonce":360502781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":360502781},{"type":"v1","addr":"172.21.15.102:6823","nonce":360502781}]},"public_addr":"172.21.15.102:6819/360502781","cluster_addr":"172.21.15.102:6821/360502781","heartbeat_back_addr":"172.21.15.102:6825/360502781","heartbeat_front_addr":"172.21.15.102:6823/360502781","state":["exists","up"]},{"osd":3,"uuid":"2ed85faf-6610-4bcc-a09d-4a6f93f5c6a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":3709228546},{"type":"v1","addr":"172.21.15.102:6827","nonce":3709228546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":3709228546},{"type":"v1","addr":"172.21.15.102:6829","nonce":3709228546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3709228546},{"type":"v1","addr":"172.21.15.102:6833","nonce":3709228546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":3709228546},{"type":"v1","addr":"172.21.15.102:6831","nonce":3709228546}]},"public_addr":"172.21.15.102:6827/3709228546","cluster_addr":"172.21.15.102:6829/3709228546","heartbeat_back_addr":"172.21.15.102:6833/3709228546","heartbeat_front_addr":"172.21.15.102:6831/3709228546","state":["exists","up"]},{"osd":4,"uuid":"8a45f350-fb3e-4049-bcc4-532d3e20c238","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":2821651815},{"type":"v1","addr":"172.21.15.195:6801","nonce":2821651815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":2821651815},{"type":"v1","addr":"172.21.15.195:6803","nonce":2821651815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":2821651815},{"type":"v1","addr":"172.21.15.195:6807","nonce":2821651815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":2821651815},{"type":"v1","addr":"172.21.15.195:6805","nonce":2821651815}]},"public_addr":"172.21.15.195:6801/2821651815","cluster_addr":"172.21.15.195:6803/2821651815","heartbeat_back_addr":"172.21.15.195:6807/2821651815","heartbeat_front_addr":"172.21.15.195:6805/2821651815","state":["exists","up"]},{"osd":5,"uuid":"363cf638-db26-4c86-b2be-e075fd09ac16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":3074139237},{"type":"v1","addr":"172.21.15.195:6809","nonce":3074139237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":3074139237},{"type":"v1","addr":"172.21.15.195:6811","nonce":3074139237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":3074139237},{"type":"v1","addr":"172.21.15.195:6815","nonce":3074139237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":3074139237},{"type":"v1","addr":"172.21.15.195:6813","nonce":3074139237}]},"public_addr":"172.21.15.195:6809/3074139237","cluster_addr":"172.21.15.195:6811/3074139237","heartbeat_back_addr":"172.21.15.195:6815/3074139237","heartbeat_front_addr":"172.21.15.195:6813/3074139237","state":["exists","up"]},{"osd":6,"uuid":"f37416e8-9c25-4860-a32f-0d1c404a8bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":4026627594},{"type":"v1","addr":"172.21.15.195:6817","nonce":4026627594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":4026627594},{"type":"v1","addr":"172.21.15.195:6819","nonce":4026627594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":4026627594},{"type":"v1","addr":"172.21.15.195:6823","nonce":4026627594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":4026627594},{"type":"v1","addr":"172.21.15.195:6821","nonce":4026627594}]},"public_addr":"172.21.15.195:6817/4026627594","cluster_addr":"172.21.15.195:6819/4026627594","heartbeat_back_addr":"172.21.15.195:6823/4026627594","heartbeat_front_addr":"172.21.15.195:6821/4026627594","state":["exists","up"]},{"osd":7,"uuid":"9a384a4d-fddb-4fb3-843a-77c19cca0368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2059403787},{"type":"v1","addr":"172.21.15.195:6825","nonce":2059403787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2059403787},{"type":"v1","addr":"172.21.15.195:6827","nonce":2059403787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2059403787},{"type":"v1","addr":"172.21.15.195:6831","nonce":2059403787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2059403787},{"type":"v1","addr":"172.21.15.195:6829","nonce":2059403787}]},"public_addr":"172.21.15.195:6825/2059403787","cluster_addr":"172.21.15.195:6827/2059403787","heartbeat_back_addr":"172.21.15.195:6831/2059403787","heartbeat_front_addr":"172.21.15.195:6829/2059403787","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:06:53.502746+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:07:16.251669+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:07:38.848272+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:08:02.013035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:08:23.043482+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:08:45.653159+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:09:10.187018+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:09:34.393753+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/1765904351":"2024-09-14T14:10:09.037657+0000","172.21.15.102:0/2416257718":"2024-09-14T14:10:09.037657+0000","172.21.15.102:0/1582001159":"2024-09-14T14:05:15.286479+0000","172.21.15.102:0/2308924552":"2024-09-14T14:10:09.037657+0000","172.21.15.102:6800/2761043787":"2024-09-14T14:05:15.286479+0000","172.21.15.102:6801/2761043787":"2024-09-14T14:05:15.286479+0000","172.21.15.102:0/2444394840":"2024-09-14T14:05:37.634624+0000","172.21.15.102:0/1574792592":"2024-09-14T14:05:15.286479+0000","172.21.15.102:0/3189088157":"2024-09-14T14:05:15.286479+0000","172.21.15.102:6800/3480998245":"2024-09-14T14:05:37.634624+0000","172.21.15.102:6801/3480998245":"2024-09-14T14:05:37.634624+0000","172.21.15.102:0/1160237252":"2024-09-14T14:10:09.037657+0000","172.21.15.102:0/3399546099":"2024-09-14T14:05:37.634624+0000","172.21.15.102:0/1939570054":"2024-09-14T14:05:37.634624+0000","172.21.15.102:6800/63514412":"2024-09-14T14:10:09.037657+0000","172.21.15.102:6801/63514412":"2024-09-14T14:10:09.037657+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T14:10:32.304 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:32 smithi102 ceph-mon[29355]: pgmap v21: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:32.304 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:32 smithi102 ceph-mon[40543]: pgmap v21: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:32.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:32 smithi195 ceph-mon[33205]: pgmap v21: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:32.910 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-13T14:10:32.911 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-13T14:10:32.911 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-13T14:10:32.911 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-13T14:10:32.912 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-13T14:10:32.912 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-13T14:10:32.913 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-13T14:10:32.913 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-13T14:10:33.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:33 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/2941723552' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:10:33.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:33 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/2941723552' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:10:33.295 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:10:33.295 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:10:33.298 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:10:33.299 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:10:33.300 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:10:33.301 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:10:33.301 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:10:33.312 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:10:33.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:33 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/2941723552' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:10:33.794 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:10:33 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a[70855]: ts=2024-09-13T14:10:33.474Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003105292s 2024-09-13T14:10:34.143 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:34 smithi195 ceph-mon[33205]: pgmap v22: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:34.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:34 smithi102 ceph-mon[29355]: pgmap v22: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:34.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:34 smithi102 ceph-mon[40543]: pgmap v22: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:36.301 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:36 smithi195 ceph-mon[33205]: pgmap v23: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:36.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:36 smithi102 ceph-mon[29355]: pgmap v23: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:36.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:36 smithi102 ceph-mon[40543]: pgmap v23: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:36.695 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:36 smithi195 systemd[1]: Starting Ceph grafana.a for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T14:10:36.967 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:36 smithi195 podman[64356]: 2024-09-13 14:10:36.596443445 +0000 UTC m=+0.014856229 image pull 954c08fa618887fe9a6fe572ae541197c77e625b05f4193df7898f5ac649ada9 quay.io/ceph/ceph-grafana:9.4.7 2024-09-13T14:10:36.967 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:36 smithi195 podman[64356]: 2024-09-13 14:10:36.730829552 +0000 UTC m=+0.149242332 container create 4e2eb92e9f89a83fd3a03922dbb9edd4178a5e653a183111cb0aafacd52d7ad5 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.32.0, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.expose-services=, release=239.1651231664, io.openshift.tags=base rhel8, description=Ceph Grafana Container, maintainer=Paul Cuzner , vendor=Red Hat, Inc., architecture=x86_64, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, name=ubi8, build-date=2022-04-29T11:48:11.055897, version=8.5, com.redhat.component=ubi8-container) 2024-09-13T14:10:36.967 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:36 smithi195 podman[64356]: 2024-09-13 14:10:36.966492179 +0000 UTC m=+0.384904967 container init 4e2eb92e9f89a83fd3a03922dbb9edd4178a5e653a183111cb0aafacd52d7ad5 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a, com.redhat.component=ubi8-container, release=239.1651231664, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8, io.buildah.version=1.32.0, maintainer=Paul Cuzner , com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5, architecture=x86_64, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.openshift.expose-services=, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, description=Ceph Grafana Container, vcs-type=git, vendor=Red Hat, Inc., build-date=2022-04-29T11:48:11.055897) 2024-09-13T14:10:37.218 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:36 smithi195 podman[64356]: 2024-09-13 14:10:36.971588859 +0000 UTC m=+0.390001641 container start 4e2eb92e9f89a83fd3a03922dbb9edd4178a5e653a183111cb0aafacd52d7ad5 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a, maintainer=Paul Cuzner , io.openshift.expose-services=, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vendor=Red Hat, Inc., version=8.5, io.openshift.tags=base rhel8, vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.32.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8, architecture=x86_64, release=239.1651231664, io.k8s.display-name=Red Hat Universal Base Image 8, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, description=Ceph Grafana Container, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-04-29T11:48:11.055897) 2024-09-13T14:10:37.218 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 bash[64356]: 4e2eb92e9f89a83fd3a03922dbb9edd4178a5e653a183111cb0aafacd52d7ad5 2024-09-13T14:10:37.218 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 systemd[1]: Started Ceph grafana.a for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:10:37.218 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=settings t=2024-09-13T14:10:37.12551327Z level=info msg="Starting Grafana" version=9.4.7 commit=4add91f03d branch=HEAD compiled=2023-03-16T23:56:52Z 2024-09-13T14:10:37.218 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=settings t=2024-09-13T14:10:37.125708015Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-09-13T14:10:37.218 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=settings t=2024-09-13T14:10:37.125725734Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-09-13T14:10:37.219 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=settings t=2024-09-13T14:10:37.125732557Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-09-13T14:10:37.219 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=settings t=2024-09-13T14:10:37.125738465Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-13T14:10:37.219 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=settings t=2024-09-13T14:10:37.125744256Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-13T14:10:37.219 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=settings t=2024-09-13T14:10:37.125749814Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-13T14:10:37.219 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=settings t=2024-09-13T14:10:37.125756907Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-13T14:10:37.219 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=settings t=2024-09-13T14:10:37.125763055Z level=info msg="Path Home" path=/usr/share/grafana 2024-09-13T14:10:37.219 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=settings t=2024-09-13T14:10:37.125769112Z level=info msg="Path Data" path=/var/lib/grafana 2024-09-13T14:10:37.220 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=settings t=2024-09-13T14:10:37.125774414Z level=info msg="Path Logs" path=/var/log/grafana 2024-09-13T14:10:37.220 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=settings t=2024-09-13T14:10:37.125779806Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-09-13T14:10:37.220 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=settings t=2024-09-13T14:10:37.125786746Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-09-13T14:10:37.220 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=settings t=2024-09-13T14:10:37.125792674Z level=info msg="App mode production" 2024-09-13T14:10:37.220 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=sqlstore t=2024-09-13T14:10:37.125841774Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-09-13T14:10:37.220 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=sqlstore t=2024-09-13T14:10:37.125857721Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-09-13T14:10:37.220 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.126694777Z level=info msg="Starting DB migrations" 2024-09-13T14:10:37.220 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.12676718Z level=info msg="Executing migration" id="create migration_log table" 2024-09-13T14:10:37.220 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.127266177Z level=info msg="Executing migration" id="create user table" 2024-09-13T14:10:37.220 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.127656169Z level=info msg="Executing migration" id="add unique index user.login" 2024-09-13T14:10:37.220 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.128088966Z level=info msg="Executing migration" id="add unique index user.email" 2024-09-13T14:10:37.220 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.128550977Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-09-13T14:10:37.221 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.128935828Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-09-13T14:10:37.221 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.129309939Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-09-13T14:10:37.221 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.129936891Z level=info msg="Executing migration" id="create user table v2" 2024-09-13T14:10:37.221 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.130313301Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-09-13T14:10:37.221 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.130629805Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-09-13T14:10:37.221 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.131026982Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-09-13T14:10:37.221 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.131354224Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-09-13T14:10:37.221 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.131645511Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-09-13T14:10:37.221 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.132210947Z level=info msg="Executing migration" id="Update user table charset" 2024-09-13T14:10:37.221 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.132556216Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-09-13T14:10:37.221 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.132910667Z level=info msg="Executing migration" id="Add missing user data" 2024-09-13T14:10:37.221 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.133245287Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-09-13T14:10:37.222 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.133590873Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-09-13T14:10:37.222 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.133939198Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-09-13T14:10:37.222 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.13436862Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-09-13T14:10:37.222 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.135441678Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-09-13T14:10:37.222 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.13590108Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-09-13T14:10:37.222 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.136290756Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-09-13T14:10:37.222 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.136655249Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-09-13T14:10:37.222 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.137016711Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-09-13T14:10:37.222 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.137377002Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-09-13T14:10:37.222 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.137596589Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-09-13T14:10:37.222 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.137891865Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-09-13T14:10:37.222 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.138270248Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-09-13T14:10:37.222 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.138575577Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-09-13T14:10:37.223 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.138871987Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-09-13T14:10:37.223 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.139492208Z level=info msg="Executing migration" id="create temp_user v2" 2024-09-13T14:10:37.223 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.139842677Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-09-13T14:10:37.223 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.140197066Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-09-13T14:10:37.224 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.140547859Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-09-13T14:10:37.224 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.14085363Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-09-13T14:10:37.224 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.141227864Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-09-13T14:10:37.224 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.141518787Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-09-13T14:10:37.224 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.141782635Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-09-13T14:10:37.224 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.142079462Z level=info msg="Executing migration" id="create star table" 2024-09-13T14:10:37.224 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.142401355Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-09-13T14:10:37.224 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.142775846Z level=info msg="Executing migration" id="create org table v1" 2024-09-13T14:10:37.224 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.143141909Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-09-13T14:10:37.224 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.143492693Z level=info msg="Executing migration" id="create org_user table v1" 2024-09-13T14:10:37.224 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.143820485Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-09-13T14:10:37.224 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.144178763Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-09-13T14:10:37.225 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.144535246Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-09-13T14:10:37.225 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.14487286Z level=info msg="Executing migration" id="Update org table charset" 2024-09-13T14:10:37.225 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.145140135Z level=info msg="Executing migration" id="Update org_user table charset" 2024-09-13T14:10:37.225 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.14535689Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-09-13T14:10:37.225 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.145592433Z level=info msg="Executing migration" id="create dashboard table" 2024-09-13T14:10:37.225 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.145941329Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-09-13T14:10:37.225 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.146321676Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-09-13T14:10:37.225 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.146698704Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-09-13T14:10:37.225 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.147044648Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-09-13T14:10:37.225 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.14740051Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-09-13T14:10:37.225 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.147704075Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-09-13T14:10:37.225 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.148498381Z level=info msg="Executing migration" id="create dashboard v2" 2024-09-13T14:10:37.225 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.148839495Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-09-13T14:10:37.226 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.149217162Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-09-13T14:10:37.226 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.149630805Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-09-13T14:10:37.226 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.149879855Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-09-13T14:10:37.226 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.150251935Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-09-13T14:10:37.226 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.150473875Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-09-13T14:10:37.226 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.150874073Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-09-13T14:10:37.226 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.151369198Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-09-13T14:10:37.226 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.151815715Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-09-13T14:10:37.226 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.152225526Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-09-13T14:10:37.226 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.152726314Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-09-13T14:10:37.226 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.153161423Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-09-13T14:10:37.227 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.153557323Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-09-13T14:10:37.227 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.153789246Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-09-13T14:10:37.227 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.154060821Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-09-13T14:10:37.227 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.154622211Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-09-13T14:10:37.227 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.155104601Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-09-13T14:10:37.227 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.155617795Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-09-13T14:10:37.227 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.156101769Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-09-13T14:10:37.227 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.15641277Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-09-13T14:10:37.227 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.15681646Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-09-13T14:10:37.227 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.157224368Z level=info msg="Executing migration" id="Update dashboard title length" 2024-09-13T14:10:37.227 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.157518934Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-09-13T14:10:37.227 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.157869912Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-09-13T14:10:37.227 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.158265519Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-09-13T14:10:37.228 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.15916334Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-09-13T14:10:37.228 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.159565297Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-09-13T14:10:37.228 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.159986656Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-09-13T14:10:37.228 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.160393486Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-09-13T14:10:37.228 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.16065293Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-09-13T14:10:37.228 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.160991434Z level=info msg="Executing migration" id="Add check_sum column" 2024-09-13T14:10:37.228 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.161517715Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-09-13T14:10:37.228 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.161862751Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-09-13T14:10:37.228 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.162174873Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-09-13T14:10:37.228 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.162419648Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-09-13T14:10:37.228 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.162823335Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-09-13T14:10:37.228 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.163396843Z level=info msg="Executing migration" id="create data_source table" 2024-09-13T14:10:37.228 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.163823136Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-09-13T14:10:37.229 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.164277108Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-09-13T14:10:37.229 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.164681778Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-09-13T14:10:37.229 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.165075012Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-09-13T14:10:37.229 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.165474145Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-09-13T14:10:37.229 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.166519487Z level=info msg="Executing migration" id="create data_source table v2" 2024-09-13T14:10:37.229 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.166929648Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-09-13T14:10:37.229 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.167320306Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-09-13T14:10:37.229 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.167703964Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-09-13T14:10:37.229 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.168076561Z level=info msg="Executing migration" id="Add column with_credentials" 2024-09-13T14:10:37.229 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.16859021Z level=info msg="Executing migration" id="Add secure json data column" 2024-09-13T14:10:37.229 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.169139532Z level=info msg="Executing migration" id="Update data_source table charset" 2024-09-13T14:10:37.229 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.169434747Z level=info msg="Executing migration" id="Update initial version to 1" 2024-09-13T14:10:37.229 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.169687882Z level=info msg="Executing migration" id="Add read_only data column" 2024-09-13T14:10:37.230 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.170206568Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-09-13T14:10:37.230 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.170517742Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-09-13T14:10:37.230 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.170773767Z level=info msg="Executing migration" id="Add uid column" 2024-09-13T14:10:37.230 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.17131399Z level=info msg="Executing migration" id="Update uid value" 2024-09-13T14:10:37.230 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.171625359Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-09-13T14:10:37.230 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.171988128Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-09-13T14:10:37.230 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.17241144Z level=info msg="Executing migration" id="create api_key table" 2024-09-13T14:10:37.230 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.172805301Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-09-13T14:10:37.230 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.173209527Z level=info msg="Executing migration" id="add index api_key.key" 2024-09-13T14:10:37.230 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.173604398Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-09-13T14:10:37.230 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.174049042Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-09-13T14:10:37.230 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.174454907Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-09-13T14:10:37.230 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.174791635Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-09-13T14:10:37.230 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.175172668Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-09-13T14:10:37.230 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.17617324Z level=info msg="Executing migration" id="create api_key table v2" 2024-09-13T14:10:37.231 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.176599576Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-09-13T14:10:37.231 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.176989359Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-09-13T14:10:37.231 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.177411573Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-09-13T14:10:37.231 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.177817017Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-09-13T14:10:37.231 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.178172098Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-09-13T14:10:37.232 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.178546121Z level=info msg="Executing migration" id="Update api_key table charset" 2024-09-13T14:10:37.232 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.178775264Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-09-13T14:10:37.232 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.179312356Z level=info msg="Executing migration" id="Add service account foreign key" 2024-09-13T14:10:37.232 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.179845297Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-09-13T14:10:37.232 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.180152961Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-09-13T14:10:37.232 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.18068329Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-09-13T14:10:37.232 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.181267452Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-09-13T14:10:37.232 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.18165309Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-09-13T14:10:37.233 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.18196467Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-09-13T14:10:37.233 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.182374324Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-09-13T14:10:37.233 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.182774446Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-09-13T14:10:37.233 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.183192983Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-09-13T14:10:37.233 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.183589192Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-09-13T14:10:37.233 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.183829384Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-09-13T14:10:37.233 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.184131106Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-09-13T14:10:37.233 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.184707771Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-09-13T14:10:37.233 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.18534434Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-09-13T14:10:37.233 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.185650428Z level=info msg="Executing migration" id="create quota table v1" 2024-09-13T14:10:37.233 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.186026345Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-09-13T14:10:37.233 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.186446818Z level=info msg="Executing migration" id="Update quota table charset" 2024-09-13T14:10:37.234 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.186689024Z level=info msg="Executing migration" id="create plugin_setting table" 2024-09-13T14:10:37.234 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.187087513Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-09-13T14:10:37.234 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.187503024Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-09-13T14:10:37.234 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.188090996Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-09-13T14:10:37.234 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.188378995Z level=info msg="Executing migration" id="create session table" 2024-09-13T14:10:37.234 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.188801805Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-09-13T14:10:37.234 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.189131256Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-09-13T14:10:37.234 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.189433405Z level=info msg="Executing migration" id="create playlist table v2" 2024-09-13T14:10:37.234 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.189815799Z level=info msg="Executing migration" id="create playlist item table v2" 2024-09-13T14:10:37.234 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.190199972Z level=info msg="Executing migration" id="Update playlist table charset" 2024-09-13T14:10:37.234 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.190490963Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-09-13T14:10:37.234 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.190720367Z level=info msg="Executing migration" id="drop preferences table v2" 2024-09-13T14:10:37.234 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.190964256Z level=info msg="Executing migration" id="drop preferences table v3" 2024-09-13T14:10:37.235 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.191262717Z level=info msg="Executing migration" id="create preferences table v3" 2024-09-13T14:10:37.235 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.19164243Z level=info msg="Executing migration" id="Update preferences table charset" 2024-09-13T14:10:37.235 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.19188899Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-09-13T14:10:37.235 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.192518204Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-09-13T14:10:37.235 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.192823953Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-09-13T14:10:37.235 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.193439412Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-09-13T14:10:37.235 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.194091802Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-09-13T14:10:37.235 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.194393362Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-09-13T14:10:37.235 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.194804493Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-09-13T14:10:37.235 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.195255679Z level=info msg="Executing migration" id="create alert table v1" 2024-09-13T14:10:37.235 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.195699292Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-09-13T14:10:37.235 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.19615233Z level=info msg="Executing migration" id="add index alert state" 2024-09-13T14:10:37.235 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.196560144Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-09-13T14:10:37.236 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.196974936Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-09-13T14:10:37.236 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.197365998Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-09-13T14:10:37.236 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.197776322Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-09-13T14:10:37.236 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.198183072Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-09-13T14:10:37.236 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.199539298Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-09-13T14:10:37.236 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.199896771Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-09-13T14:10:37.236 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.200300608Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-09-13T14:10:37.236 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.200562394Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-09-13T14:10:37.236 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.200863464Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-09-13T14:10:37.236 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.201246092Z level=info msg="Executing migration" id="Add column is_default" 2024-09-13T14:10:37.236 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.201813805Z level=info msg="Executing migration" id="Add column frequency" 2024-09-13T14:10:37.236 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.20246109Z level=info msg="Executing migration" id="Add column send_reminder" 2024-09-13T14:10:37.236 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.203073628Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-09-13T14:10:37.236 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.203670143Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-09-13T14:10:37.237 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.204144837Z level=info msg="Executing migration" id="Update alert table charset" 2024-09-13T14:10:37.237 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.204480198Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-09-13T14:10:37.237 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.204710706Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-09-13T14:10:37.237 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.205081392Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-09-13T14:10:37.237 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.205485711Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-09-13T14:10:37.237 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.20580185Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-09-13T14:10:37.237 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.206168451Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-09-13T14:10:37.237 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.206548181Z level=info msg="Executing migration" id="Add for to alert table" 2024-09-13T14:10:37.237 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.207172003Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-09-13T14:10:37.237 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.207757209Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-09-13T14:10:37.237 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.208042543Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-09-13T14:10:37.237 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.208426789Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-09-13T14:10:37.237 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.208800325Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-09-13T14:10:37.238 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.209388154Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-09-13T14:10:37.238 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.209661734Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-09-13T14:10:37.238 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.210045453Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-09-13T14:10:37.238 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.210413353Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-09-13T14:10:37.238 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.210646286Z level=info msg="Executing migration" id="create annotation table v5" 2024-09-13T14:10:37.238 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.211049834Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-09-13T14:10:37.238 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.211423748Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-09-13T14:10:37.238 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.211796278Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-09-13T14:10:37.238 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.212190643Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-09-13T14:10:37.238 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.212593632Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-09-13T14:10:37.238 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.21299169Z level=info msg="Executing migration" id="Update annotation table charset" 2024-09-13T14:10:37.239 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.213252349Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-09-13T14:10:37.239 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.213838358Z level=info msg="Executing migration" id="Drop category_id index" 2024-09-13T14:10:37.239 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.214204095Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-09-13T14:10:37.239 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.214817325Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-09-13T14:10:37.239 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.215211425Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-09-13T14:10:37.239 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.215584478Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-09-13T14:10:37.239 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.215933805Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-09-13T14:10:37.239 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.217481947Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-09-13T14:10:37.239 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.217945556Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-09-13T14:10:37.239 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.218481327Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-09-13T14:10:37.239 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.218855779Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-09-13T14:10:37.239 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.219303993Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-09-13T14:10:37.239 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.219673784Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-09-13T14:10:37.239 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.22067297Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-09-13T14:10:37.240 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.221633909Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-09-13T14:10:37.240 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.22221637Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-09-13T14:10:37.240 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.222781898Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-09-13T14:10:37.240 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.223167679Z level=info msg="Executing migration" id="Add epoch_end column" 2024-09-13T14:10:37.240 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.224212337Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-09-13T14:10:37.240 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.224803765Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-09-13T14:10:37.240 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.225198796Z level=info msg="Executing migration" id="Move region to single row" 2024-09-13T14:10:37.240 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.225645151Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-09-13T14:10:37.240 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.226192924Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-09-13T14:10:37.240 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.226807329Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-09-13T14:10:37.240 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.227369304Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-09-13T14:10:37.240 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.227956297Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-09-13T14:10:37.240 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.228488938Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-09-13T14:10:37.503 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.22892101Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-09-13T14:10:37.504 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.229209014Z level=info msg="Executing migration" id="create test_data table" 2024-09-13T14:10:37.504 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.229594053Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-09-13T14:10:37.504 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.22995692Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-09-13T14:10:37.504 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.230341162Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-09-13T14:10:37.504 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.230727527Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-09-13T14:10:37.504 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.230985127Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-09-13T14:10:37.504 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.231268708Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-09-13T14:10:37.504 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.231495191Z level=info msg="Executing migration" id="create team table" 2024-09-13T14:10:37.505 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.231833938Z level=info msg="Executing migration" id="add index team.org_id" 2024-09-13T14:10:37.505 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.232448957Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-09-13T14:10:37.505 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.23283168Z level=info msg="Executing migration" id="create team member table" 2024-09-13T14:10:37.505 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.233184278Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-09-13T14:10:37.505 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.233557036Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-09-13T14:10:37.505 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.233942408Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-09-13T14:10:37.505 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.23432329Z level=info msg="Executing migration" id="Add column email to team table" 2024-09-13T14:10:37.505 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.234941201Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-09-13T14:10:37.505 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.235585358Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-09-13T14:10:37.505 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.236291815Z level=info msg="Executing migration" id="create dashboard acl table" 2024-09-13T14:10:37.505 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.236643615Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-09-13T14:10:37.505 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.237047248Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-09-13T14:10:37.506 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.237453175Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-09-13T14:10:37.506 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.237853267Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-09-13T14:10:37.506 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.23832694Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-09-13T14:10:37.506 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.238723817Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-09-13T14:10:37.506 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.23910815Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-09-13T14:10:37.506 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.239499067Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-09-13T14:10:37.506 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.239827702Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-09-13T14:10:37.507 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.240147935Z level=info msg="Executing migration" id="create tag table" 2024-09-13T14:10:37.507 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.240481437Z level=info msg="Executing migration" id="add index tag.key_value" 2024-09-13T14:10:37.507 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.240869629Z level=info msg="Executing migration" id="create login attempt table" 2024-09-13T14:10:37.507 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.241226073Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-09-13T14:10:37.507 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.241622118Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-09-13T14:10:37.507 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.241961403Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-09-13T14:10:37.507 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.243782412Z level=info msg="Executing migration" id="create login_attempt v2" 2024-09-13T14:10:37.507 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.244139639Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-09-13T14:10:37.508 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.244545865Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-09-13T14:10:37.508 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.244790353Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-09-13T14:10:37.508 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.245117657Z level=info msg="Executing migration" id="create user auth table" 2024-09-13T14:10:37.508 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.245457551Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-09-13T14:10:37.508 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.245851493Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-09-13T14:10:37.508 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.246142108Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-09-13T14:10:37.508 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.246850815Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-09-13T14:10:37.508 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.247565142Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-09-13T14:10:37.508 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.248260545Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-09-13T14:10:37.508 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.248963384Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-09-13T14:10:37.508 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.249349069Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-09-13T14:10:37.508 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.250038015Z level=info msg="Executing migration" id="create server_lock table" 2024-09-13T14:10:37.508 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.250382337Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-09-13T14:10:37.509 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.250784672Z level=info msg="Executing migration" id="create user auth token table" 2024-09-13T14:10:37.509 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.251186037Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-09-13T14:10:37.509 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.251562827Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-09-13T14:10:37.509 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.251939701Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-09-13T14:10:37.509 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.252314466Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-09-13T14:10:37.509 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.252992683Z level=info msg="Executing migration" id="create cache_data table" 2024-09-13T14:10:37.509 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.253353309Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-09-13T14:10:37.509 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.253729368Z level=info msg="Executing migration" id="create short_url table v1" 2024-09-13T14:10:37.509 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.254111386Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-09-13T14:10:37.509 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.254484228Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-09-13T14:10:37.509 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.25470822Z level=info msg="Executing migration" id="delete alert_definition table" 2024-09-13T14:10:37.509 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.254941346Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-09-13T14:10:37.509 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.255356379Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-09-13T14:10:37.510 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.255783264Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-09-13T14:10:37.510 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.256207131Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-09-13T14:10:37.510 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.256491272Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-09-13T14:10:37.510 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.256828046Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-09-13T14:10:37.510 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.257206136Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-09-13T14:10:37.510 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.25760516Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-09-13T14:10:37.510 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.257984131Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-09-13T14:10:37.510 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.258737245Z level=info msg="Executing migration" id="drop alert_definition table" 2024-09-13T14:10:37.510 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.259138578Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-09-13T14:10:37.510 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.259429769Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-09-13T14:10:37.510 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.259756193Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-09-13T14:10:37.510 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.260151041Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-09-13T14:10:37.510 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.260548409Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-09-13T14:10:37.510 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.260772461Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-09-13T14:10:37.511 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.261194901Z level=info msg="Executing migration" id="create alert_instance table" 2024-09-13T14:10:37.511 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.261587395Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-09-13T14:10:37.511 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.261945174Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-09-13T14:10:37.511 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.262347946Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-09-13T14:10:37.511 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.263117237Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-09-13T14:10:37.511 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.263509339Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-09-13T14:10:37.511 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.263840329Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-09-13T14:10:37.511 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.267160769Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-09-13T14:10:37.511 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.270513964Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-09-13T14:10:37.511 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.270932966Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-09-13T14:10:37.511 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.271333512Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-09-13T14:10:37.511 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.272099196Z level=info msg="Executing migration" id="create alert_rule table" 2024-09-13T14:10:37.512 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.272491783Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-09-13T14:10:37.512 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.272873809Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-09-13T14:10:37.512 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.273273632Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-09-13T14:10:37.512 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.273643098Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-09-13T14:10:37.512 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.273870245Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-09-13T14:10:37.512 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.274636513Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-09-13T14:10:37.512 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.275422586Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-09-13T14:10:37.512 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.27621193Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-09-13T14:10:37.512 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.276581915Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-09-13T14:10:37.512 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.276939059Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-09-13T14:10:37.512 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.277717148Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-09-13T14:10:37.512 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.278547713Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-09-13T14:10:37.512 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.27896527Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-09-13T14:10:37.513 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.27974754Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-09-13T14:10:37.513 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.280554787Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-09-13T14:10:37.513 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.28085969Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-09-13T14:10:37.513 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.281304852Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-09-13T14:10:37.513 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.281711506Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-09-13T14:10:37.513 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.282126062Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-09-13T14:10:37.513 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.282413107Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-09-13T14:10:37.513 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.283241066Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-09-13T14:10:37.513 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.284061893Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-09-13T14:10:37.513 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.284890706Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-09-13T14:10:37.513 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.28570077Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-09-13T14:10:37.513 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.286509952Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-09-13T14:10:37.513 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.286768424Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-09-13T14:10:37.514 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.287122125Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-09-13T14:10:37.514 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.287905667Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-09-13T14:10:37.514 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.288184873Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-09-13T14:10:37.514 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.289024258Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-09-13T14:10:37.514 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.289422274Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-09-13T14:10:37.514 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.290215237Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-09-13T14:10:37.514 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.290566Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-09-13T14:10:37.515 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.290949803Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-09-13T14:10:37.515 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.291797861Z level=info msg="Executing migration" id="create provenance_type table" 2024-09-13T14:10:37.515 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.292163154Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-09-13T14:10:37.515 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.29256626Z level=info msg="Executing migration" id="create alert_image table" 2024-09-13T14:10:37.515 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.292909101Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-09-13T14:10:37.515 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.293314763Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-09-13T14:10:37.515 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.293553236Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-09-13T14:10:37.515 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.293908622Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-09-13T14:10:37.515 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.294283228Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-09-13T14:10:37.515 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.294372865Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-09-13T14:10:37.515 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.294577441Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-09-13T14:10:37.515 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.294854682Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-09-13T14:10:37.515 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.295243879Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-09-13T14:10:37.515 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.295353425Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-09-13T14:10:37.516 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.295461171Z level=info msg="alerts found to migrate" alerts=0 2024-09-13T14:10:37.516 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.295802741Z level=info msg="Executing migration" id="create library_element table v1" 2024-09-13T14:10:37.516 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.296205892Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-09-13T14:10:37.516 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.29660189Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-09-13T14:10:37.516 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.296984968Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-09-13T14:10:37.516 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.297376812Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-09-13T14:10:37.516 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.2977833Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-09-13T14:10:37.516 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.29803781Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-09-13T14:10:37.516 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.298368611Z level=info msg="Executing migration" id="create data_keys table" 2024-09-13T14:10:37.516 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.298728383Z level=info msg="Executing migration" id="create secrets table" 2024-09-13T14:10:37.516 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.29911814Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-09-13T14:10:37.516 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.30338673Z level=info msg="Executing migration" id="add name column into data_keys" 2024-09-13T14:10:37.517 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.304268144Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-09-13T14:10:37.517 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.304534053Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-09-13T14:10:37.517 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.308699668Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-09-13T14:10:37.517 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.312902536Z level=info msg="Executing migration" id="create kv_store table v1" 2024-09-13T14:10:37.517 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.313289354Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-09-13T14:10:37.517 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.313678836Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-09-13T14:10:37.517 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.313948248Z level=info msg="Executing migration" id="create permission table" 2024-09-13T14:10:37.517 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.314297459Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-09-13T14:10:37.517 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.314676715Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-09-13T14:10:37.517 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.315072982Z level=info msg="Executing migration" id="create role table" 2024-09-13T14:10:37.517 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.315438971Z level=info msg="Executing migration" id="add column display_name" 2024-09-13T14:10:37.517 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.316348484Z level=info msg="Executing migration" id="add column group_name" 2024-09-13T14:10:37.517 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.317253993Z level=info msg="Executing migration" id="add index role.org_id" 2024-09-13T14:10:37.518 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.317641311Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-09-13T14:10:37.518 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.318029964Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-09-13T14:10:37.518 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.318417985Z level=info msg="Executing migration" id="create team role table" 2024-09-13T14:10:37.518 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.318757055Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-09-13T14:10:37.518 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.319148228Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-09-13T14:10:37.518 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.319555318Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-09-13T14:10:37.518 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.319941353Z level=info msg="Executing migration" id="create user role table" 2024-09-13T14:10:37.518 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.320323735Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-09-13T14:10:37.518 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.32070514Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-09-13T14:10:37.518 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.321106943Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-09-13T14:10:37.518 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.321498372Z level=info msg="Executing migration" id="create builtin role table" 2024-09-13T14:10:37.518 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.321867247Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-09-13T14:10:37.519 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.322258962Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-09-13T14:10:37.519 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.322637798Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-09-13T14:10:37.519 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.323547763Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-09-13T14:10:37.519 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.323996708Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-09-13T14:10:37.519 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.324394179Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-09-13T14:10:37.519 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.32473922Z level=info msg="Executing migration" id="add unique index role.uid" 2024-09-13T14:10:37.519 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.325161883Z level=info msg="Executing migration" id="create seed assignment table" 2024-09-13T14:10:37.519 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.325501519Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-09-13T14:10:37.519 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.325887751Z level=info msg="Executing migration" id="add column hidden to role table" 2024-09-13T14:10:37.519 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.326806415Z level=info msg="Executing migration" id="create query_history table v1" 2024-09-13T14:10:37.519 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.327175793Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-09-13T14:10:37.519 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.32756422Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-09-13T14:10:37.519 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.327786808Z level=info msg="Executing migration" id="teams permissions migration" 2024-09-13T14:10:37.519 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.328160144Z level=info msg="Executing migration" id="dashboard permissions" 2024-09-13T14:10:37.520 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.328559333Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-09-13T14:10:37.520 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.32884952Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-09-13T14:10:37.520 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.329147641Z level=info msg="Executing migration" id="alerting notification permissions" 2024-09-13T14:10:37.520 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.329452611Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-09-13T14:10:37.520 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.329792231Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-09-13T14:10:37.520 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.330190182Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-09-13T14:10:37.520 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.331120682Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-09-13T14:10:37.520 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.33138126Z level=info msg="Executing migration" id="create correlation table v1" 2024-09-13T14:10:37.520 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.331744416Z level=info msg="Executing migration" id="add index correlations.uid" 2024-09-13T14:10:37.520 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.332144222Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-09-13T14:10:37.520 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.332534995Z level=info msg="Executing migration" id="add correlation config column" 2024-09-13T14:10:37.520 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.333557028Z level=info msg="Executing migration" id="create entity_events table" 2024-09-13T14:10:37.521 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.333949946Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-09-13T14:10:37.521 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.334323927Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-13T14:10:37.521 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.334407021Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-13T14:10:37.521 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.334648617Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-13T14:10:37.521 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.334721273Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-13T14:10:37.521 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.334927486Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-09-13T14:10:37.521 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.335317231Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-09-13T14:10:37.521 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.335658348Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-09-13T14:10:37.521 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.336061272Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-13T14:10:37.521 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.336456104Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-09-13T14:10:37.522 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.336794208Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-13T14:10:37.522 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.337187146Z level=info msg="Executing migration" id="Drop public config table" 2024-09-13T14:10:37.522 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.337566912Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-09-13T14:10:37.522 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.337922717Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-09-13T14:10:37.523 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.338316649Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-13T14:10:37.523 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.338699589Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-09-13T14:10:37.523 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.339097221Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-09-13T14:10:37.523 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.342054199Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-09-13T14:10:37.523 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.34306016Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-09-13T14:10:37.523 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.344065363Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-09-13T14:10:37.523 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.344356032Z level=info msg="Executing migration" id="add share column" 2024-09-13T14:10:37.523 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.345318723Z level=info msg="Executing migration" id="create default alerting folders" 2024-09-13T14:10:37.523 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.345731761Z level=info msg="Executing migration" id="create file table" 2024-09-13T14:10:37.523 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.346150213Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-09-13T14:10:37.523 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.34656492Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-09-13T14:10:37.523 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.347031334Z level=info msg="Executing migration" id="create file_meta table" 2024-09-13T14:10:37.523 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.34757435Z level=info msg="Executing migration" id="file table idx: path key" 2024-09-13T14:10:37.523 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.348022106Z level=info msg="Executing migration" id="set path collation in file table" 2024-09-13T14:10:37.524 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.348285458Z level=info msg="Executing migration" id="managed permissions migration" 2024-09-13T14:10:37.524 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.348574566Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-09-13T14:10:37.524 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.348817015Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-09-13T14:10:37.524 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.349209363Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-09-13T14:10:37.524 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.350187841Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-09-13T14:10:37.524 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.350482029Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-09-13T14:10:37.524 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.350892784Z level=info msg="Executing migration" id="update group index for alert rules" 2024-09-13T14:10:37.524 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.351249106Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-09-13T14:10:37.524 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.351497401Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-09-13T14:10:37.524 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.351786688Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-09-13T14:10:37.524 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.352762737Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-09-13T14:10:37.524 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.353751565Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-09-13T14:10:37.524 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.354144845Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-09-13T14:10:37.525 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.3640099Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-09-13T14:10:37.525 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.364438087Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-09-13T14:10:37.525 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.364849061Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-09-13T14:10:37.525 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.368292458Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-09-13T14:10:37.525 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.368574518Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-09-13T14:10:37.525 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.368843038Z level=info msg="Executing migration" id="create folder table" 2024-09-13T14:10:37.525 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.369199039Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-09-13T14:10:37.525 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.369593104Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-09-13T14:10:37.525 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.369986742Z level=info msg="Executing migration" id="Update folder title length" 2024-09-13T14:10:37.525 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.370238478Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-09-13T14:10:37.525 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=migrator t=2024-09-13T14:10:37.370612853Z level=info msg="migrations completed" performed=485 skipped=0 duration=243.864315ms 2024-09-13T14:10:37.525 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=sqlstore t=2024-09-13T14:10:37.370987067Z level=info msg="Created default organization" 2024-09-13T14:10:37.525 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=plugin.loader t=2024-09-13T14:10:37.426721565Z level=info msg="Plugin registered" pluginID=input 2024-09-13T14:10:37.525 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=plugin.loader t=2024-09-13T14:10:37.453945597Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-09-13T14:10:37.526 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=plugin.loader t=2024-09-13T14:10:37.453985068Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-09-13T14:10:37.526 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=secrets t=2024-09-13T14:10:37.454017865Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-09-13T14:10:37.526 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=query_data t=2024-09-13T14:10:37.455735932Z level=info msg="Query Service initialization" 2024-09-13T14:10:37.526 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=live.push_http t=2024-09-13T14:10:37.458196427Z level=info msg="Live Push Gateway initialization" 2024-09-13T14:10:37.628 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:37 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:10:37] "GET /metrics HTTP/1.1" 200 28524 "" "Prometheus/2.43.0" 2024-09-13T14:10:38.367 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:38 smithi195 ceph-mon[33205]: pgmap v24: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:38.367 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:38 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:38.367 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:38 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:38.367 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:38 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:38.368 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:38 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:38.368 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:38 smithi195 ceph-mon[33205]: Deploying daemon node-exporter.a on smithi102 2024-09-13T14:10:38.368 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:38 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=infra.usagestats.collector t=2024-09-13T14:10:38.166055615Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-09-13T14:10:38.368 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:38 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=provisioning.datasources t=2024-09-13T14:10:38.220405064Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-09-13T14:10:38.368 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:38 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=provisioning.datasources t=2024-09-13T14:10:38.252249123Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-09-13T14:10:38.368 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:38 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=provisioning.alerting t=2024-09-13T14:10:38.273664215Z level=info msg="starting to provision alerting" 2024-09-13T14:10:38.368 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:38 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=provisioning.alerting t=2024-09-13T14:10:38.273701409Z level=info msg="finished to provision alerting" 2024-09-13T14:10:38.368 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:38 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=ngalert.state.manager t=2024-09-13T14:10:38.273857326Z level=info msg="Warming state cache for startup" 2024-09-13T14:10:38.369 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:38 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=grafanaStorageLogger t=2024-09-13T14:10:38.273871892Z level=info msg="storage starting" 2024-09-13T14:10:38.369 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:38 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=http.server t=2024-09-13T14:10:38.275574771Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-09-13T14:10:38.369 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:38 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=sqlstore.transactions t=2024-09-13T14:10:38.284855453Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-13T14:10:38.369 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:38 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=ngalert.state.manager t=2024-09-13T14:10:38.2850876Z level=info msg="State cache has been initialized" states=0 duration=11.229514ms 2024-09-13T14:10:38.369 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:38 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=ticker t=2024-09-13T14:10:38.285212812Z level=info msg=starting first_tick=2024-09-13T14:10:40Z 2024-09-13T14:10:38.369 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:38 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=ngalert.multiorg.alertmanager t=2024-09-13T14:10:38.285386153Z level=info msg="starting MultiOrg Alertmanager" 2024-09-13T14:10:38.369 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:38 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=sqlstore.transactions t=2024-09-13T14:10:38.32702748Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-13T14:10:38.369 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:10:38 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[64371]: logger=sqlstore.transactions t=2024-09-13T14:10:38.367267586Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-13T14:10:38.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:38 smithi102 ceph-mon[40543]: pgmap v24: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:38.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:38 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:38.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:38 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:38.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:38 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:38.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:38 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:38.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:38 smithi102 ceph-mon[40543]: Deploying daemon node-exporter.a on smithi102 2024-09-13T14:10:38.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:38 smithi102 ceph-mon[29355]: pgmap v24: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:38.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:38 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:38.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:38 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:38.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:38 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:38.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:38 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:38.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:38 smithi102 ceph-mon[29355]: Deploying daemon node-exporter.a on smithi102 2024-09-13T14:10:40.367 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:40 smithi102 ceph-mon[40543]: pgmap v25: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:40.367 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:40 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:40.368 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:40 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:10:40.368 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:40 smithi102 ceph-mon[29355]: pgmap v25: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:40.368 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:40 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:40.368 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:40 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:10:40.505 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:40 smithi195 ceph-mon[33205]: pgmap v25: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:40.505 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:40 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:40.505 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:40 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:10:40.918 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:40 smithi102 systemd[1]: Starting Ceph node-exporter.a for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T14:10:41.474 INFO:teuthology.orchestra.run.smithi102.stdout:107374182434 2024-09-13T14:10:41.475 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-13T14:10:42.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:42 smithi102 ceph-mon[29355]: pgmap v26: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:42.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:42 smithi102 ceph-mon[40543]: pgmap v26: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:42.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:42 smithi195 ceph-mon[33205]: pgmap v26: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:44.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:44 smithi102 ceph-mon[29355]: pgmap v27: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:44.400 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:44 smithi102 ceph-mon[40543]: pgmap v27: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:44.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:44 smithi195 ceph-mon[33205]: pgmap v27: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:44.660 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:44 smithi102 bash[73044]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-09-13T14:10:44.918 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:44 smithi102 bash[73044]: Getting image source signatures 2024-09-13T14:10:44.918 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:44 smithi102 bash[73044]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-09-13T14:10:44.918 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:44 smithi102 bash[73044]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-09-13T14:10:44.919 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:44 smithi102 bash[73044]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-09-13T14:10:46.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:46 smithi102 ceph-mon[29355]: pgmap v28: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:46.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:46 smithi102 ceph-mon[40543]: pgmap v28: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:46.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:46 smithi195 ceph-mon[33205]: pgmap v28: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:46.771 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:46 smithi102 bash[73044]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-09-13T14:10:46.772 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:46 smithi102 bash[73044]: Writing manifest to image destination 2024-09-13T14:10:47.603 INFO:teuthology.orchestra.run.smithi102.stdout:197568495636 2024-09-13T14:10:47.604 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-13T14:10:47.630 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:47 smithi102 podman[73044]: 2024-09-13 14:10:47.330617012 +0000 UTC m=+3.111817674 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-13T14:10:47.630 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:47 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:10:47] "GET /metrics HTTP/1.1" 200 28517 "" "Prometheus/2.43.0" 2024-09-13T14:10:48.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:48 smithi102 ceph-mon[29355]: pgmap v29: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:48.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:48 smithi102 ceph-mon[40543]: pgmap v29: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:48.406 INFO:teuthology.orchestra.run.smithi102.stdout:77309411367 2024-09-13T14:10:48.407 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-13T14:10:48.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:48 smithi195 ceph-mon[33205]: pgmap v29: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:49.168 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:48 smithi102 podman[73044]: 2024-09-13 14:10:48.861058892 +0000 UTC m=+4.642259553 container create 5538bdf011a365b78667629853a501e81070d1747dac9607be84d5fa976330f9 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T14:10:50.257 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-mon[29355]: pgmap v30: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:50.257 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-mon[40543]: pgmap v30: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:50.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:50 smithi195 ceph-mon[33205]: pgmap v30: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:50.646 INFO:teuthology.orchestra.run.smithi102.stdout:34359738417 2024-09-13T14:10:50.646 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-13T14:10:50.917 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 podman[73044]: 2024-09-13 14:10:50.816929075 +0000 UTC m=+6.598129748 container init 5538bdf011a365b78667629853a501e81070d1747dac9607be84d5fa976330f9 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T14:10:50.918 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 podman[73044]: 2024-09-13 14:10:50.823556132 +0000 UTC m=+6.604756803 container start 5538bdf011a365b78667629853a501e81070d1747dac9607be84d5fa976330f9 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T14:10:50.918 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.828Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-13T14:10:50.918 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.828Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-13T14:10:50.918 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.829Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-13T14:10:50.918 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.829Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-13T14:10:50.918 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.829Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-13T14:10:50.918 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.829Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-13T14:10:50.918 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.830Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-13T14:10:50.919 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.830Z caller=node_exporter.go:117 level=info collector=arp 2024-09-13T14:10:50.919 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.830Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-13T14:10:50.919 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.830Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-13T14:10:50.919 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.830Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-13T14:10:50.919 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.830Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-13T14:10:50.919 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.830Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-13T14:10:50.919 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.830Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-13T14:10:50.919 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.830Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-13T14:10:50.919 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.830Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-13T14:10:50.920 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.830Z caller=node_exporter.go:117 level=info collector=edac 2024-09-13T14:10:50.920 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.830Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-13T14:10:50.920 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.830Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-13T14:10:50.920 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.830Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-13T14:10:50.920 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.830Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-13T14:10:50.920 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.830Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-13T14:10:50.920 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.830Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-13T14:10:50.920 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.830Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-13T14:10:50.920 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.830Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-13T14:10:50.920 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.830Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-13T14:10:50.921 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.830Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-13T14:10:50.921 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.830Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-13T14:10:50.921 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.830Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-13T14:10:50.921 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.830Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-13T14:10:50.921 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.830Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-13T14:10:50.921 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.830Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-13T14:10:50.921 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.830Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-13T14:10:50.921 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.830Z caller=node_exporter.go:117 level=info collector=os 2024-09-13T14:10:50.921 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.830Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-13T14:10:50.921 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.830Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-13T14:10:50.921 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.830Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-13T14:10:50.922 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.830Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-13T14:10:50.922 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.830Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-13T14:10:50.922 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.830Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-13T14:10:50.922 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.830Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-13T14:10:50.922 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.830Z caller=node_exporter.go:117 level=info collector=stat 2024-09-13T14:10:50.922 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.830Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-13T14:10:50.923 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.830Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-13T14:10:50.923 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.830Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-13T14:10:50.923 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.830Z caller=node_exporter.go:117 level=info collector=time 2024-09-13T14:10:50.923 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.830Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-13T14:10:50.923 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.830Z caller=node_exporter.go:117 level=info collector=uname 2024-09-13T14:10:50.923 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.830Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-13T14:10:50.923 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.830Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-13T14:10:50.923 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.830Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-13T14:10:50.923 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.831Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-13T14:10:50.924 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[73463]: ts=2024-09-13T14:10:50.831Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-13T14:10:50.924 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 bash[73044]: 5538bdf011a365b78667629853a501e81070d1747dac9607be84d5fa976330f9 2024-09-13T14:10:51.168 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:10:50 smithi102 systemd[1]: Started Ceph node-exporter.a for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:10:51.882 INFO:teuthology.orchestra.run.smithi102.stdout:231928234001 2024-09-13T14:10:51.882 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-13T14:10:51.981 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:51 smithi195 bash[64653]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-09-13T14:10:52.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:51 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:52.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:51 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:52.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:51 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:52.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:51 smithi102 ceph-mon[29355]: Deploying daemon node-exporter.b on smithi195 2024-09-13T14:10:52.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:51 smithi102 ceph-mon[29355]: pgmap v31: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:52.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:51 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:52.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:51 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:52.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:51 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:52.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:51 smithi102 ceph-mon[40543]: Deploying daemon node-exporter.b on smithi195 2024-09-13T14:10:52.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:51 smithi102 ceph-mon[40543]: pgmap v31: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:52.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:51 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:52.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:51 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:52.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:51 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:52.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:51 smithi195 ceph-mon[33205]: Deploying daemon node-exporter.b on smithi195 2024-09-13T14:10:52.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:51 smithi195 ceph-mon[33205]: pgmap v31: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:52.391 INFO:teuthology.orchestra.run.smithi102.stdout:55834574891 2024-09-13T14:10:52.391 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-13T14:10:52.747 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:52 smithi195 bash[64653]: Getting image source signatures 2024-09-13T14:10:52.747 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:52 smithi195 bash[64653]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-09-13T14:10:52.747 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:52 smithi195 bash[64653]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-09-13T14:10:52.747 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:52 smithi195 bash[64653]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-09-13T14:10:53.379 INFO:teuthology.orchestra.run.smithi102.stdout:128849018911 2024-09-13T14:10:53.380 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-13T14:10:53.806 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:10:53.837 INFO:teuthology.orchestra.run.smithi102.stdout:163208757273 2024-09-13T14:10:53.837 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-13T14:10:53.997 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:53 smithi195 bash[64653]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-09-13T14:10:53.997 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:53 smithi195 bash[64653]: Writing manifest to image destination 2024-09-13T14:10:54.079 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:10:54.086 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:10:54.345 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:54 smithi102 ceph-mon[29355]: pgmap v32: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:54.345 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:54 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:10:54.346 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:54 smithi102 ceph-mon[40543]: pgmap v32: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:54.346 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:54 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:10:54.380 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-mon[33205]: pgmap v32: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:54.380 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:10:54.381 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 podman[64653]: 2024-09-13 14:10:54.063132862 +0000 UTC m=+2.262166719 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-13T14:10:54.381 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 podman[64653]: 2024-09-13 14:10:54.153325749 +0000 UTC m=+2.352359649 container create 138397d5e1298120d1adb47e3c5644b25f1a0470fdee6bb1538874167f71f490 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T14:10:54.748 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 podman[64653]: 2024-09-13 14:10:54.380297097 +0000 UTC m=+2.579330965 container init 138397d5e1298120d1adb47e3c5644b25f1a0470fdee6bb1538874167f71f490 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T14:10:54.748 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 podman[64653]: 2024-09-13 14:10:54.385611644 +0000 UTC m=+2.584645511 container start 138397d5e1298120d1adb47e3c5644b25f1a0470fdee6bb1538874167f71f490 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T14:10:54.748 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.389Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-13T14:10:54.748 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.389Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-13T14:10:54.748 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.389Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-13T14:10:54.749 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.389Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-13T14:10:54.749 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.390Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-13T14:10:54.749 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.390Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-13T14:10:54.749 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.390Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-13T14:10:54.749 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.390Z caller=node_exporter.go:117 level=info collector=arp 2024-09-13T14:10:54.749 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.390Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-13T14:10:54.749 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.390Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-13T14:10:54.749 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.390Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-13T14:10:54.749 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.390Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-13T14:10:54.749 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.390Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-13T14:10:54.749 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.390Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-13T14:10:54.750 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.390Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-13T14:10:54.750 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.390Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-13T14:10:54.750 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.390Z caller=node_exporter.go:117 level=info collector=edac 2024-09-13T14:10:54.750 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.390Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-13T14:10:54.751 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.390Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-13T14:10:54.751 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.390Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-13T14:10:54.751 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.390Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-13T14:10:54.751 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.390Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-13T14:10:54.751 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.390Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-13T14:10:54.751 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.390Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-13T14:10:54.751 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.390Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-13T14:10:54.751 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.390Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-13T14:10:54.751 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.390Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-13T14:10:54.751 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.390Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-13T14:10:54.751 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.390Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-13T14:10:54.751 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.390Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-13T14:10:54.752 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.390Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-13T14:10:54.752 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.390Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-13T14:10:54.752 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.390Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-13T14:10:54.752 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.390Z caller=node_exporter.go:117 level=info collector=os 2024-09-13T14:10:54.752 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.390Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-13T14:10:54.752 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.390Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-13T14:10:54.752 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.390Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-13T14:10:54.752 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.390Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-13T14:10:54.752 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.390Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-13T14:10:54.753 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.390Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-13T14:10:54.753 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.390Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-13T14:10:54.753 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.390Z caller=node_exporter.go:117 level=info collector=stat 2024-09-13T14:10:54.753 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.390Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-13T14:10:54.753 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.390Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-13T14:10:54.753 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.390Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-13T14:10:54.753 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.390Z caller=node_exporter.go:117 level=info collector=time 2024-09-13T14:10:54.753 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.390Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-13T14:10:54.753 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.390Z caller=node_exporter.go:117 level=info collector=uname 2024-09-13T14:10:54.753 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.390Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-13T14:10:54.753 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.390Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-13T14:10:54.753 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.390Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-13T14:10:54.753 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.391Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-13T14:10:54.753 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[64729]: ts=2024-09-13T14:10:54.391Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-13T14:10:54.754 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 bash[64653]: 138397d5e1298120d1adb47e3c5644b25f1a0470fdee6bb1538874167f71f490 2024-09-13T14:10:54.754 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:10:54 smithi195 systemd[1]: Started Ceph node-exporter.b for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:10:55.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:55 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:55.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:55 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:55.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:55 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:55.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:55 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:55.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:55 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:55.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:55 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:10:55.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:55 smithi102 ceph-mon[29355]: pgmap v33: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:55.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:55 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:55.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:55 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:55.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:55 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:55.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:55 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:55.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:55 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:55.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:55 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:10:55.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:55 smithi102 ceph-mon[40543]: pgmap v33: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:55.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:55 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:55.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:55 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:55.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:55 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:55.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:55 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:55.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:55 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:55.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:55 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:10:55.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:55 smithi195 ceph-mon[33205]: pgmap v33: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:56.453 INFO:teuthology.orchestra.run.smithi102.stdout:77309411369 2024-09-13T14:10:56.851 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:56 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/906964523' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-13T14:10:56.852 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:56 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/906964523' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-13T14:10:56.892 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:56 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/906964523' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-13T14:10:57.395 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:10:57.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:57 smithi102 ceph-mon[40543]: pgmap v34: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:57.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:57 smithi102 ceph-mon[29355]: pgmap v34: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:57.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:10:57 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:10:57] "GET /metrics HTTP/1.1" 200 28514 "" "Prometheus/2.43.0" 2024-09-13T14:10:57.769 INFO:teuthology.orchestra.run.smithi102.stdout:34359738419 2024-09-13T14:10:57.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:57 smithi195 ceph-mon[33205]: pgmap v34: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:10:58.085 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:10:58.085 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:10:58.109 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411367 got 77309411369 for osd.2 2024-09-13T14:10:58.110 DEBUG:teuthology.parallel:result is None 2024-09-13T14:10:58.301 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:10:58.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:58 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/2120597747' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-13T14:10:58.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:58 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:58.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:10:58 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:58.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:58 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/2120597747' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-13T14:10:58.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:58 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:58.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:10:58 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:58.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:58 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/2120597747' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-13T14:10:58.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:58 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:58.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:10:58 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:10:59.057 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738417 got 34359738419 for osd.0 2024-09-13T14:10:59.057 DEBUG:teuthology.parallel:result is None 2024-09-13T14:11:00.406 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:00 smithi102 ceph-mon[29355]: pgmap v35: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:00.406 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:00 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:00.407 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:00 smithi102 ceph-mon[40543]: pgmap v35: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:00.407 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:00 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:00.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:00 smithi195 ceph-mon[33205]: pgmap v35: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:00.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:00 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:01.281 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:11:02.372 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:02 smithi102 ceph-mon[29355]: pgmap v36: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:02.372 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:02 smithi102 ceph-mon[40543]: pgmap v36: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:02.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:02 smithi195 ceph-mon[33205]: pgmap v36: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:02.736 INFO:teuthology.orchestra.run.smithi102.stdout:107374182438 2024-09-13T14:11:03.336 INFO:teuthology.orchestra.run.smithi102.stdout:197568495640 2024-09-13T14:11:03.336 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:03 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/345908783' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-13T14:11:03.337 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:03 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/345908783' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-13T14:11:03.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:03 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/345908783' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-13T14:11:03.710 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182434 got 107374182438 for osd.3 2024-09-13T14:11:03.710 DEBUG:teuthology.parallel:result is None 2024-09-13T14:11:04.293 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:04 smithi102 ceph-mon[29355]: pgmap v37: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:04.293 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:04 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/1400661946' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-13T14:11:04.293 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:04 smithi102 ceph-mon[40543]: pgmap v37: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:04.293 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:04 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/1400661946' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-13T14:11:04.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:04 smithi195 ceph-mon[33205]: pgmap v37: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:04.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:04 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/1400661946' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-13T14:11:05.609 INFO:teuthology.orchestra.run.smithi102.stdout:163208757278 2024-09-13T14:11:06.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:06 smithi102 ceph-mon[40543]: pgmap v38: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:06.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:06 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/4242517689' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-13T14:11:06.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:06 smithi102 ceph-mon[29355]: pgmap v38: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:06.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:06 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/4242517689' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-13T14:11:06.235 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757273 got 163208757278 for osd.5 2024-09-13T14:11:06.235 DEBUG:teuthology.parallel:result is None 2024-09-13T14:11:06.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:06 smithi195 ceph-mon[33205]: pgmap v38: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:06.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:06 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/4242517689' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-13T14:11:07.219 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495636 got 197568495640 for osd.6 2024-09-13T14:11:07.219 DEBUG:teuthology.parallel:result is None 2024-09-13T14:11:07.553 INFO:teuthology.orchestra.run.smithi102.stdout:231928234005 2024-09-13T14:11:07.554 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:11:07 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:11:07] "GET /metrics HTTP/1.1" 200 28514 "" "Prometheus/2.43.0" 2024-09-13T14:11:08.245 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:08 smithi102 ceph-mon[40543]: pgmap v39: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:08.245 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:08 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/2400552475' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-13T14:11:08.245 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:08 smithi102 ceph-mon[29355]: pgmap v39: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:08.246 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:08 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/2400552475' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-13T14:11:08.265 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234001 got 231928234005 for osd.7 2024-09-13T14:11:08.265 DEBUG:teuthology.parallel:result is None 2024-09-13T14:11:08.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:08 smithi195 ceph-mon[33205]: pgmap v39: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:08.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:08 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/2400552475' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-13T14:11:08.920 INFO:teuthology.orchestra.run.smithi102.stdout:55834574896 2024-09-13T14:11:09.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:09 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/3502680132' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-13T14:11:09.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:09 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:11:09.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:09 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/3502680132' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-13T14:11:09.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:09 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:11:09.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:09 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/3502680132' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-13T14:11:09.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:09 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:11:09.652 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574891 got 55834574896 for osd.1 2024-09-13T14:11:09.652 DEBUG:teuthology.parallel:result is None 2024-09-13T14:11:10.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:10 smithi102 ceph-mon[40543]: pgmap v40: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:10.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:10 smithi102 ceph-mon[29355]: pgmap v40: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:10.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:10 smithi195 ceph-mon[33205]: pgmap v40: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:11.270 INFO:teuthology.orchestra.run.smithi102.stdout:128849018915 2024-09-13T14:11:11.866 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018911 got 128849018915 for osd.4 2024-09-13T14:11:11.866 DEBUG:teuthology.parallel:result is None 2024-09-13T14:11:11.866 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-13T14:11:11.866 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:11:12.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:12 smithi102 ceph-mon[29355]: pgmap v41: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:12.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:12 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/3493594476' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-13T14:11:12.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:12 smithi102 ceph-mon[40543]: pgmap v41: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:12.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:12 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/3493594476' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-13T14:11:12.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:12 smithi195 ceph-mon[33205]: pgmap v41: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:12.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:12 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/3493594476' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-13T14:11:12.709 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:11:14.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:14 smithi102 ceph-mon[29355]: pgmap v42: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:14.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:14 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:14.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:14 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:14.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:14 smithi102 ceph-mon[40543]: pgmap v42: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:14.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:14 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:14.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:14 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:14.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:14 smithi195 ceph-mon[33205]: pgmap v42: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:14.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:14 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:14.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:14 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:15.119 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:11:15.120 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:11:15.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:15 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:11:15.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:15 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:11:15.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:15 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:15.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:15 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:11:15.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:15 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:11:15.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:15 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:15.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:15 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:11:15.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:15 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:11:15.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:15 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:16.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:16 smithi102 ceph-mon[29355]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-13T14:11:16.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:16 smithi102 ceph-mon[29355]: Reconfiguring daemon alertmanager.a on smithi102 2024-09-13T14:11:16.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:16 smithi102 ceph-mon[29355]: pgmap v43: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:16.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:16 smithi102 ceph-mon[29355]: from='client.24485 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:11:16.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:16 smithi102 ceph-mon[40543]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-13T14:11:16.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:16 smithi102 ceph-mon[40543]: Reconfiguring daemon alertmanager.a on smithi102 2024-09-13T14:11:16.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:16 smithi102 ceph-mon[40543]: pgmap v43: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:16.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:16 smithi102 ceph-mon[40543]: from='client.24485 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:11:16.334 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":43,"stamp":"2024-09-13T14:11:15.064706+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":181,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131,"ondisk_log_size":131,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":231328,"kb_used_data":6980,"kb_used_omap":12,"kb_used_meta":224243,"kb_avail":749729888,"statfs":{"total":767960285184,"available":767723405312,"internally_reserved":0,"allocated":7147520,"data_stored":4034424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":229625401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001987"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:10:23.599212+0000","last_change":"2024-09-13T14:10:16.905852+0000","last_active":"2024-09-13T14:10:23.599212+0000","last_peered":"2024-09-13T14:10:23.599212+0000","last_clean":"2024-09-13T14:10:23.599212+0000","last_became_active":"2024-09-13T14:10:12.122815+0000","last_became_peered":"2024-09-13T14:10:12.122815+0000","last_unstale":"2024-09-13T14:10:23.599212+0000","last_undegraded":"2024-09-13T14:10:23.599212+0000","last_fullsized":"2024-09-13T14:10:23.599212+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:59:17.830936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034784200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:10:23.598405+0000","last_change":"2024-09-13T14:10:16.905444+0000","last_active":"2024-09-13T14:10:23.598405+0000","last_peered":"2024-09-13T14:10:23.598405+0000","last_clean":"2024-09-13T14:10:23.598405+0000","last_became_active":"2024-09-13T14:10:12.878714+0000","last_became_peered":"2024-09-13T14:10:12.878714+0000","last_unstale":"2024-09-13T14:10:23.598405+0000","last_undegraded":"2024-09-13T14:10:23.598405+0000","last_fullsized":"2024-09-13T14:10:23.598405+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:23:02.772611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000229423,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:10:23.599220+0000","last_change":"2024-09-13T14:10:17.227584+0000","last_active":"2024-09-13T14:10:23.599220+0000","last_peered":"2024-09-13T14:10:23.599220+0000","last_clean":"2024-09-13T14:10:23.599220+0000","last_became_active":"2024-09-13T14:10:12.878424+0000","last_became_peered":"2024-09-13T14:10:12.878424+0000","last_unstale":"2024-09-13T14:10:23.599220+0000","last_undegraded":"2024-09-13T14:10:23.599220+0000","last_fullsized":"2024-09-13T14:10:23.599220+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:00:10.824840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032561200000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:10:23.598420+0000","last_change":"2024-09-13T14:10:16.905490+0000","last_active":"2024-09-13T14:10:23.598420+0000","last_peered":"2024-09-13T14:10:23.598420+0000","last_clean":"2024-09-13T14:10:23.598420+0000","last_became_active":"2024-09-13T14:10:12.122768+0000","last_became_peered":"2024-09-13T14:10:12.122768+0000","last_unstale":"2024-09-13T14:10:23.598420+0000","last_undegraded":"2024-09-13T14:10:23.598420+0000","last_fullsized":"2024-09-13T14:10:23.598420+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:55:21.633350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026843400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":22,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:10:23.599294+0000","last_change":"2024-09-13T14:10:16.904932+0000","last_active":"2024-09-13T14:10:23.599294+0000","last_peered":"2024-09-13T14:10:23.599294+0000","last_clean":"2024-09-13T14:10:23.599294+0000","last_became_active":"2024-09-13T14:10:12.121802+0000","last_became_peered":"2024-09-13T14:10:12.121802+0000","last_unstale":"2024-09-13T14:10:23.599294+0000","last_undegraded":"2024-09-13T14:10:23.599294+0000","last_fullsized":"2024-09-13T14:10:23.599294+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:11:49.860408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088127500000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:10:23.027119+0000","last_change":"2024-09-13T14:10:16.901072+0000","last_active":"2024-09-13T14:10:23.027119+0000","last_peered":"2024-09-13T14:10:23.027119+0000","last_clean":"2024-09-13T14:10:23.027119+0000","last_became_active":"2024-09-13T14:10:13.084740+0000","last_became_peered":"2024-09-13T14:10:13.084740+0000","last_unstale":"2024-09-13T14:10:23.027119+0000","last_undegraded":"2024-09-13T14:10:23.027119+0000","last_fullsized":"2024-09-13T14:10:23.027119+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:06:12.555142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028414100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:10:23.599211+0000","last_change":"2024-09-13T14:10:17.227534+0000","last_active":"2024-09-13T14:10:23.599211+0000","last_peered":"2024-09-13T14:10:23.599211+0000","last_clean":"2024-09-13T14:10:23.599211+0000","last_became_active":"2024-09-13T14:10:12.123842+0000","last_became_peered":"2024-09-13T14:10:12.123842+0000","last_unstale":"2024-09-13T14:10:23.599211+0000","last_undegraded":"2024-09-13T14:10:23.599211+0000","last_fullsized":"2024-09-13T14:10:23.599211+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:36:09.588482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031558699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"63'1","reported_seq":21,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:10:23.599175+0000","last_change":"2024-09-13T14:10:16.904388+0000","last_active":"2024-09-13T14:10:23.599175+0000","last_peered":"2024-09-13T14:10:23.599175+0000","last_clean":"2024-09-13T14:10:23.599175+0000","last_became_active":"2024-09-13T14:10:12.122143+0000","last_became_peered":"2024-09-13T14:10:12.122143+0000","last_unstale":"2024-09-13T14:10:23.599175+0000","last_undegraded":"2024-09-13T14:10:23.599175+0000","last_fullsized":"2024-09-13T14:10:23.599175+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:11.434245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034973600000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"60'128","reported_seq":262,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-13T14:10:17.227211+0000","last_change":"2024-09-13T14:09:40.436318+0000","last_active":"2024-09-13T14:10:17.227211+0000","last_peered":"2024-09-13T14:10:17.227211+0000","last_clean":"2024-09-13T14:10:17.227211+0000","last_became_active":"2024-09-13T14:09:40.436016+0000","last_became_peered":"2024-09-13T14:09:40.436016+0000","last_unstale":"2024-09-13T14:10:17.227211+0000","last_undegraded":"2024-09-13T14:10:17.227211+0000","last_fullsized":"2024-09-13T14:10:17.227211+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:07:41.927740+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:16.214871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234007,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28252,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716900,"statfs":{"total":95995035648,"available":95966105600,"internally_reserved":0,"allocated":1261568,"data_stored":873130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:10:36 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.58999999999999997}]},{"osd":1,"last update":"Fri Sep 13 14:10:36 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.46700000000000003}]},{"osd":2,"last update":"Fri Sep 13 14:10:36 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.47799999999999998}]},{"osd":3,"last update":"Fri Sep 13 14:10:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.52800000000000002}]},{"osd":4,"last update":"Fri Sep 13 14:10:36 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.433},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.55900000000000005}]},{"osd":5,"last update":"Fri Sep 13 14:10:36 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.504}]},{"osd":6,"last update":"Fri Sep 13 14:10:36 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.44900000000000001}]}]},{"osd":6,"up_from":46,"seq":197568495643,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32352,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712800,"statfs":{"total":95995035648,"available":95961907200,"internally_reserved":0,"allocated":1265664,"data_stored":873207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:10:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.60099999999999998}]},{"osd":1,"last update":"Fri Sep 13 14:10:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.65000000000000002}]},{"osd":2,"last update":"Fri Sep 13 14:10:15 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.66400000000000003}]},{"osd":3,"last update":"Fri Sep 13 14:10:15 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.51600000000000001}]},{"osd":4,"last update":"Fri Sep 13 14:10:15 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.497}]},{"osd":5,"last update":"Fri Sep 13 14:10:15 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.62,"5min":0.62,"15min":0.62},"last":0.432},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.57899999999999996}]},{"osd":7,"last update":"Fri Sep 13 14:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.69299999999999995}]}]},{"osd":5,"up_from":38,"seq":163208757280,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31772,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713380,"statfs":{"total":95995035648,"available":95962501120,"internally_reserved":0,"allocated":671744,"data_stored":282781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:10:49 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.52900000000000003,"15min":0.53300000000000003},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.33900000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.67500000000000004,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.63700000000000001}]},{"osd":1,"last update":"Fri Sep 13 14:10:49 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.39900000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.65900000000000003,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.504,"5min":0.52100000000000002,"15min":0.52400000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.59099999999999997}]},{"osd":2,"last update":"Fri Sep 13 14:10:49 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.54100000000000004,"15min":0.54500000000000004},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.499,"5min":0.52000000000000002,"15min":0.52400000000000002},"min":{"1min":0.32600000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.69299999999999995,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.496}]},{"osd":3,"last update":"Fri Sep 13 14:10:49 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.50700000000000001,"15min":0.50900000000000001},"min":{"1min":0.35199999999999998,"5min":0.312,"15min":0.312},"max":{"1min":0.76600000000000001,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.54100000000000004,"15min":0.54400000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.61799999999999999}]},{"osd":4,"last update":"Fri Sep 13 14:10:49 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.39800000000000002,"15min":0.39600000000000002},"min":{"1min":0.27600000000000002,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.61199999999999999,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.373},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.41299999999999998,"15min":0.41699999999999998},"min":{"1min":0.30499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.58399999999999996,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.40000000000000002}]},{"osd":6,"last update":"Fri Sep 13 14:10:14 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.46400000000000002}]},{"osd":7,"last update":"Fri Sep 13 14:10:38 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.54900000000000004}]}]},{"osd":4,"up_from":30,"seq":128849018916,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27672,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717480,"statfs":{"total":95995035648,"available":95966699520,"internally_reserved":0,"allocated":667648,"data_stored":282820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:10:27 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.57499999999999996,"15min":0.59399999999999997},"min":{"1min":0.35299999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.59299999999999997,"5min":1.258,"15min":1.258},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.56699999999999995,"15min":0.58099999999999996},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.76000000000000001,"5min":0.997,"15min":0.997},"last":0.57199999999999995}]},{"osd":1,"last update":"Fri Sep 13 14:10:27 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.57299999999999995,"15min":0.58899999999999997},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68799999999999994,"5min":1.117,"15min":1.117},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.56399999999999995,"15min":0.57799999999999996},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.745,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":0.60999999999999999}]},{"osd":2,"last update":"Fri Sep 13 14:10:27 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.56399999999999995,"15min":0.57999999999999996},"min":{"1min":0.38800000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.58899999999999997,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.57499999999999996,"15min":0.59099999999999997},"min":{"1min":0.375,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.61199999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.53100000000000003}]},{"osd":3,"last update":"Fri Sep 13 14:10:27 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.58899999999999997,"15min":0.60299999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.63500000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.58399999999999996,"15min":0.59999999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.65700000000000003,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.627}]},{"osd":5,"last update":"Fri Sep 13 14:10:53 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.35799999999999998,"15min":0.35399999999999998},"min":{"1min":0.28999999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.376},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.47399999999999998,"15min":0.46700000000000003},"min":{"1min":0.34899999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.48199999999999998}]},{"osd":6,"last update":"Fri Sep 13 14:10:16 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.46200000000000002}]},{"osd":7,"last update":"Fri Sep 13 14:10:36 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.42399999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182441,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717476,"statfs":{"total":95995035648,"available":95966695424,"internally_reserved":0,"allocated":671744,"data_stored":283093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:11:12 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.35199999999999998,"15min":0.35399999999999998},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.379,"15min":0.38100000000000001},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.55700000000000005,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.35699999999999998}]},{"osd":1,"last update":"Fri Sep 13 14:11:12 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.39300000000000002,"15min":0.39600000000000002},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.38200000000000001,"15min":0.38500000000000001},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.54400000000000004,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.25700000000000001}]},{"osd":2,"last update":"Fri Sep 13 14:11:12 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.31,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.40100000000000002,"15min":0.40000000000000002},"min":{"1min":0.27800000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.377}]},{"osd":4,"last update":"Fri Sep 13 14:10:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.51700000000000002,"15min":0.51200000000000001},"min":{"1min":0.379,"5min":0.373,"15min":0.373},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.52500000000000002,"15min":0.52100000000000002},"min":{"1min":0.39200000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.72399999999999998}]},{"osd":5,"last update":"Fri Sep 13 14:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.56599999999999995,"15min":0.56399999999999995},"min":{"1min":0.40100000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.53000000000000003,"15min":0.52300000000000002},"min":{"1min":0.379,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.53600000000000003}]},{"osd":6,"last update":"Fri Sep 13 14:10:13 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.61899999999999999}]},{"osd":7,"last update":"Fri Sep 13 14:10:38 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.69499999999999995}]}]},{"osd":2,"up_from":18,"seq":77309411373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27672,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717480,"statfs":{"total":95995035648,"available":95966699520,"internally_reserved":0,"allocated":671744,"data_stored":283093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:10:44 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.377,"15min":0.37},"min":{"1min":0.224,"5min":0.22,"15min":0.22},"max":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.39400000000000002,"15min":0.38900000000000001},"min":{"1min":0.255,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.35399999999999998}]},{"osd":1,"last update":"Fri Sep 13 14:10:44 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.376,"15min":0.36799999999999999},"min":{"1min":0.24099999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.39900000000000002},"min":{"1min":0.27000000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.34599999999999997}]},{"osd":3,"last update":"Fri Sep 13 14:11:10 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.46400000000000002,"15min":0.47699999999999998},"min":{"1min":0.318,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.56200000000000006,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.436,"15min":0.45400000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.48299999999999998,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.30099999999999999}]},{"osd":4,"last update":"Fri Sep 13 14:10:28 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.56899999999999995,"15min":0.57799999999999996},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.61099999999999999,"15min":0.62},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.57799999999999996}]},{"osd":5,"last update":"Fri Sep 13 14:10:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.63900000000000001,"15min":0.64800000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.80800000000000005,"5min":1.2,"15min":1.2},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.57899999999999996,"15min":0.58499999999999996},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.70999999999999996,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.752}]},{"osd":6,"last update":"Fri Sep 13 14:11:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.56100000000000005,"15min":0.55900000000000005},"min":{"1min":0.40200000000000002,"5min":0.378,"15min":0.378},"max":{"1min":0.751,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.57199999999999995,"15min":0.56599999999999995},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.73699999999999999}]},{"osd":7,"last update":"Fri Sep 13 14:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.59299999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717476,"statfs":{"total":95995035648,"available":95966695424,"internally_reserved":0,"allocated":671744,"data_stored":282839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:10:21 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.32000000000000001,"15min":0.29799999999999999},"min":{"1min":0.24199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.33600000000000002,"15min":0.314},"min":{"1min":0.23100000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"last":0.28599999999999998}]},{"osd":2,"last update":"Fri Sep 13 14:10:52 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.35999999999999999,"15min":0.34999999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.53300000000000003,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.314},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.35999999999999999,"15min":0.35399999999999998},"min":{"1min":0.253,"5min":0.224,"15min":0.224},"max":{"1min":0.50900000000000001,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.30099999999999999}]},{"osd":3,"last update":"Fri Sep 13 14:11:08 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.40100000000000002,"15min":0.40300000000000002},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.51700000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.25},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.42699999999999999,"15min":0.42899999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.53100000000000003,"5min":0.621,"15min":0.621},"last":0.34200000000000003}]},{"osd":4,"last update":"Fri Sep 13 14:10:31 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55300000000000005,"15min":0.55200000000000005},"min":{"1min":0.41499999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.72899999999999998,"5min":1.133,"15min":1.133},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.61199999999999999,"15min":0.626},"min":{"1min":0.34499999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.71899999999999997,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.379}]},{"osd":5,"last update":"Fri Sep 13 14:10:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.54700000000000004,"15min":0.54500000000000004},"min":{"1min":0.39700000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.48399999999999999}]},{"osd":6,"last update":"Fri Sep 13 14:10:13 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.47599999999999998}]},{"osd":7,"last update":"Fri Sep 13 14:10:38 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.498}]}]},{"osd":0,"up_from":8,"seq":34359738423,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28256,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716896,"statfs":{"total":95995035648,"available":95966101504,"internally_reserved":0,"allocated":1265664,"data_stored":873461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 14:10:24 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.34499999999999997,"15min":0.32900000000000001},"min":{"1min":0.27400000000000002,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.55500000000000005,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.34999999999999998,"15min":0.33200000000000002},"min":{"1min":0.24199999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.54800000000000004,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.22}]},{"osd":2,"last update":"Fri Sep 13 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.379,"15min":0.376},"min":{"1min":0.25900000000000001,"5min":0.192,"15min":0.192},"max":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"last":0.23300000000000001},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.39000000000000001,"15min":0.39400000000000002},"min":{"1min":0.26700000000000002,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.434,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.255}]},{"osd":3,"last update":"Fri Sep 13 14:11:10 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.40899999999999997,"15min":0.42099999999999999},"min":{"1min":0.23599999999999999,"5min":0.224,"15min":0.224},"max":{"1min":0.48199999999999998,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.45100000000000001,"15min":0.46200000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.48599999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.28899999999999998}]},{"osd":4,"last update":"Fri Sep 13 14:10:28 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.57199999999999995,"15min":0.58199999999999996},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.67200000000000004,"5min":1.044,"15min":1.044},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.57299999999999995,"15min":0.58499999999999996},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.73799999999999999,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.46700000000000003}]},{"osd":5,"last update":"Fri Sep 13 14:10:51 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.60399999999999998,"15min":0.62},"min":{"1min":0.34699999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.63600000000000001,"5min":1.109,"15min":1.109},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.58999999999999997,"15min":0.60899999999999999},"min":{"1min":0.36899999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.57199999999999995,"5min":1.151,"15min":1.151},"last":0.53600000000000003}]},{"osd":6,"last update":"Fri Sep 13 14:10:14 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.73699999999999999}]},{"osd":7,"last update":"Fri Sep 13 14:10:38 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.71199999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:11:16.336 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:11:16.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:16 smithi195 ceph-mon[33205]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-13T14:11:16.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:16 smithi195 ceph-mon[33205]: Reconfiguring daemon alertmanager.a on smithi102 2024-09-13T14:11:16.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:16 smithi195 ceph-mon[33205]: pgmap v43: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:16.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:16 smithi195 ceph-mon[33205]: from='client.24485 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:11:17.615 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:11:17 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:11:17] "GET /metrics HTTP/1.1" 200 28517 "" "Prometheus/2.43.0" 2024-09-13T14:11:18.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:18 smithi102 ceph-mon[29355]: pgmap v44: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:18.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:18 smithi102 ceph-mon[40543]: pgmap v44: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:18.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:18 smithi195 ceph-mon[33205]: pgmap v44: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:18.680 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:11:18.803 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:11:18 smithi102 systemd[1]: Stopping Ceph alertmanager.a for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T14:11:19.385 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:11:19 smithi102 podman[76252]: 2024-09-13 14:11:19.228455529 +0000 UTC m=+0.556779207 container died 335ad2d472c366e068cd0a66fab4781f803241fbf9a4adb8a541e3f52af2e537 (image=quay.ceph.io/ceph-ci/ceph:reef, name=quirky_haslett, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=reef-2c01458, org.label-schema.build-date=20240731) 2024-09-13T14:11:19.668 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:11:19 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a[70855]: ts=2024-09-13T14:11:19.525Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-13T14:11:19.668 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:11:19 smithi102 podman[76334]: 2024-09-13 14:11:19.545929657 +0000 UTC m=+0.114261494 container died 2523493d7218c308898e1befbc333be659b4324ddb3809a8b70da261b96401b6 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:11:20.136 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:11:19 smithi102 podman[76334]: 2024-09-13 14:11:19.931939958 +0000 UTC m=+0.500271796 container cleanup 2523493d7218c308898e1befbc333be659b4324ddb3809a8b70da261b96401b6 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:11:20.136 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:11:19 smithi102 bash[76334]: ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a 2024-09-13T14:11:20.337 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:20 smithi102 ceph-mon[29355]: pgmap v45: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:20.337 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:20 smithi102 ceph-mon[40543]: pgmap v45: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:20.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:20 smithi195 ceph-mon[33205]: pgmap v45: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:20.668 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:11:20 smithi102 podman[76354]: 2024-09-13 14:11:20.560906343 +0000 UTC m=+1.014127148 container remove 2523493d7218c308898e1befbc333be659b4324ddb3809a8b70da261b96401b6 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:11:20.668 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:11:20 smithi102 podman[76354]: 2024-09-13 14:11:20.636552092 +0000 UTC m=+1.089772894 volume remove c56e33581853b295a0e4e682b9513ae62958c67ea676d5126ca3699ec0b205e8 2024-09-13T14:11:21.337 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:11:21.337 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:11:21 smithi102 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-13T14:11:21.337 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:11:21 smithi102 systemd[1]: Stopped Ceph alertmanager.a for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:11:21.338 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:11:21 smithi102 systemd[1]: Starting Ceph alertmanager.a for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T14:11:21.338 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:11:21.668 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:11:21 smithi102 podman[76536]: 2024-09-13 14:11:21.390894502 +0000 UTC m=+0.106426446 volume create 1ba02c42abefdda13346ea52261d27ba9fd5d635197a130cb9f519e2168d7e39 2024-09-13T14:11:21.668 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:11:21 smithi102 podman[76536]: 2024-09-13 14:11:21.300179916 +0000 UTC m=+0.015711829 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-09-13T14:11:21.668 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:11:21 smithi102 podman[76536]: 2024-09-13 14:11:21.4994726 +0000 UTC m=+0.215004561 container create fadcd26e53958ed2f8777d8578736b52125048671ff18d7d2eef9704163acd19 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:11:22.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:22 smithi102 ceph-mon[29355]: pgmap v46: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:22.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:22 smithi102 ceph-mon[29355]: from='client.14598 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:11:22.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:22 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:22.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:22 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:22.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:22 smithi102 ceph-mon[40543]: pgmap v46: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:22.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:22 smithi102 ceph-mon[40543]: from='client.14598 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:11:22.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:22 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:22.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:22 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:22.419 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:11:21 smithi102 podman[76536]: 2024-09-13 14:11:21.98664213 +0000 UTC m=+0.702174029 container init fadcd26e53958ed2f8777d8578736b52125048671ff18d7d2eef9704163acd19 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:11:22.420 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:11:21 smithi102 podman[76536]: 2024-09-13 14:11:21.991589157 +0000 UTC m=+0.707121066 container start fadcd26e53958ed2f8777d8578736b52125048671ff18d7d2eef9704163acd19 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:11:22.420 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:11:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a[76563]: ts=2024-09-13T14:11:22.003Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-09-13T14:11:22.420 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:11:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a[76563]: ts=2024-09-13T14:11:22.003Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-09-13T14:11:22.420 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:11:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a[76563]: ts=2024-09-13T14:11:22.005Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.102 port=9094 2024-09-13T14:11:22.420 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:11:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a[76563]: ts=2024-09-13T14:11:22.005Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-13T14:11:22.420 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:11:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a[76563]: ts=2024-09-13T14:11:22.028Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-13T14:11:22.420 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:11:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a[76563]: ts=2024-09-13T14:11:22.029Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-13T14:11:22.420 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:11:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a[76563]: ts=2024-09-13T14:11:22.031Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-09-13T14:11:22.421 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:11:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a[76563]: ts=2024-09-13T14:11:22.031Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-13T14:11:22.421 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:11:22 smithi102 bash[76536]: fadcd26e53958ed2f8777d8578736b52125048671ff18d7d2eef9704163acd19 2024-09-13T14:11:22.421 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:11:22 smithi102 systemd[1]: Started Ceph alertmanager.a for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:11:22.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:22 smithi195 ceph-mon[33205]: pgmap v46: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:22.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:22 smithi195 ceph-mon[33205]: from='client.14598 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:11:22.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:22 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:22.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:22 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:22.652 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":46,"stamp":"2024-09-13T14:11:21.065979+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":181,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131,"ondisk_log_size":131,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":231328,"kb_used_data":6980,"kb_used_omap":12,"kb_used_meta":224243,"kb_avail":749729888,"statfs":{"total":767960285184,"available":767723405312,"internally_reserved":0,"allocated":7147520,"data_stored":4034424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":229625401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002322"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:10:23.599212+0000","last_change":"2024-09-13T14:10:16.905852+0000","last_active":"2024-09-13T14:10:23.599212+0000","last_peered":"2024-09-13T14:10:23.599212+0000","last_clean":"2024-09-13T14:10:23.599212+0000","last_became_active":"2024-09-13T14:10:12.122815+0000","last_became_peered":"2024-09-13T14:10:12.122815+0000","last_unstale":"2024-09-13T14:10:23.599212+0000","last_undegraded":"2024-09-13T14:10:23.599212+0000","last_fullsized":"2024-09-13T14:10:23.599212+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:59:17.830936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034784200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:10:23.598405+0000","last_change":"2024-09-13T14:10:16.905444+0000","last_active":"2024-09-13T14:10:23.598405+0000","last_peered":"2024-09-13T14:10:23.598405+0000","last_clean":"2024-09-13T14:10:23.598405+0000","last_became_active":"2024-09-13T14:10:12.878714+0000","last_became_peered":"2024-09-13T14:10:12.878714+0000","last_unstale":"2024-09-13T14:10:23.598405+0000","last_undegraded":"2024-09-13T14:10:23.598405+0000","last_fullsized":"2024-09-13T14:10:23.598405+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:23:02.772611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000229423,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:10:23.599220+0000","last_change":"2024-09-13T14:10:17.227584+0000","last_active":"2024-09-13T14:10:23.599220+0000","last_peered":"2024-09-13T14:10:23.599220+0000","last_clean":"2024-09-13T14:10:23.599220+0000","last_became_active":"2024-09-13T14:10:12.878424+0000","last_became_peered":"2024-09-13T14:10:12.878424+0000","last_unstale":"2024-09-13T14:10:23.599220+0000","last_undegraded":"2024-09-13T14:10:23.599220+0000","last_fullsized":"2024-09-13T14:10:23.599220+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:00:10.824840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032561200000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:10:23.598420+0000","last_change":"2024-09-13T14:10:16.905490+0000","last_active":"2024-09-13T14:10:23.598420+0000","last_peered":"2024-09-13T14:10:23.598420+0000","last_clean":"2024-09-13T14:10:23.598420+0000","last_became_active":"2024-09-13T14:10:12.122768+0000","last_became_peered":"2024-09-13T14:10:12.122768+0000","last_unstale":"2024-09-13T14:10:23.598420+0000","last_undegraded":"2024-09-13T14:10:23.598420+0000","last_fullsized":"2024-09-13T14:10:23.598420+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:55:21.633350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026843400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":22,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:10:23.599294+0000","last_change":"2024-09-13T14:10:16.904932+0000","last_active":"2024-09-13T14:10:23.599294+0000","last_peered":"2024-09-13T14:10:23.599294+0000","last_clean":"2024-09-13T14:10:23.599294+0000","last_became_active":"2024-09-13T14:10:12.121802+0000","last_became_peered":"2024-09-13T14:10:12.121802+0000","last_unstale":"2024-09-13T14:10:23.599294+0000","last_undegraded":"2024-09-13T14:10:23.599294+0000","last_fullsized":"2024-09-13T14:10:23.599294+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:11:49.860408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088127500000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:10:23.027119+0000","last_change":"2024-09-13T14:10:16.901072+0000","last_active":"2024-09-13T14:10:23.027119+0000","last_peered":"2024-09-13T14:10:23.027119+0000","last_clean":"2024-09-13T14:10:23.027119+0000","last_became_active":"2024-09-13T14:10:13.084740+0000","last_became_peered":"2024-09-13T14:10:13.084740+0000","last_unstale":"2024-09-13T14:10:23.027119+0000","last_undegraded":"2024-09-13T14:10:23.027119+0000","last_fullsized":"2024-09-13T14:10:23.027119+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:06:12.555142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028414100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:10:23.599211+0000","last_change":"2024-09-13T14:10:17.227534+0000","last_active":"2024-09-13T14:10:23.599211+0000","last_peered":"2024-09-13T14:10:23.599211+0000","last_clean":"2024-09-13T14:10:23.599211+0000","last_became_active":"2024-09-13T14:10:12.123842+0000","last_became_peered":"2024-09-13T14:10:12.123842+0000","last_unstale":"2024-09-13T14:10:23.599211+0000","last_undegraded":"2024-09-13T14:10:23.599211+0000","last_fullsized":"2024-09-13T14:10:23.599211+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:36:09.588482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031558699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"63'1","reported_seq":21,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:10:23.599175+0000","last_change":"2024-09-13T14:10:16.904388+0000","last_active":"2024-09-13T14:10:23.599175+0000","last_peered":"2024-09-13T14:10:23.599175+0000","last_clean":"2024-09-13T14:10:23.599175+0000","last_became_active":"2024-09-13T14:10:12.122143+0000","last_became_peered":"2024-09-13T14:10:12.122143+0000","last_unstale":"2024-09-13T14:10:23.599175+0000","last_undegraded":"2024-09-13T14:10:23.599175+0000","last_fullsized":"2024-09-13T14:10:23.599175+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:11.434245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034973600000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"60'128","reported_seq":262,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-13T14:10:17.227211+0000","last_change":"2024-09-13T14:09:40.436318+0000","last_active":"2024-09-13T14:10:17.227211+0000","last_peered":"2024-09-13T14:10:17.227211+0000","last_clean":"2024-09-13T14:10:17.227211+0000","last_became_active":"2024-09-13T14:09:40.436016+0000","last_became_peered":"2024-09-13T14:09:40.436016+0000","last_unstale":"2024-09-13T14:10:17.227211+0000","last_undegraded":"2024-09-13T14:10:17.227211+0000","last_fullsized":"2024-09-13T14:10:17.227211+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:07:41.927740+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:16.214871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234008,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28252,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716900,"statfs":{"total":95995035648,"available":95966105600,"internally_reserved":0,"allocated":1261568,"data_stored":873130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:10:36 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.52900000000000003}]},{"osd":1,"last update":"Fri Sep 13 14:10:36 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.50800000000000001}]},{"osd":2,"last update":"Fri Sep 13 14:10:36 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.495},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.47799999999999998}]},{"osd":3,"last update":"Fri Sep 13 14:10:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.54900000000000004}]},{"osd":4,"last update":"Fri Sep 13 14:10:36 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.42399999999999999}]},{"osd":5,"last update":"Fri Sep 13 14:10:36 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.36499999999999999}]},{"osd":6,"last update":"Fri Sep 13 14:10:36 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.33600000000000002}]}]},{"osd":6,"up_from":46,"seq":197568495644,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32352,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712800,"statfs":{"total":95995035648,"available":95961907200,"internally_reserved":0,"allocated":1265664,"data_stored":873207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:10:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.27500000000000002}]},{"osd":1,"last update":"Fri Sep 13 14:10:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.35499999999999998}]},{"osd":2,"last update":"Fri Sep 13 14:10:15 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.34300000000000003}]},{"osd":3,"last update":"Fri Sep 13 14:10:15 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.438},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.47699999999999998}]},{"osd":4,"last update":"Fri Sep 13 14:10:15 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.502},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.55800000000000005}]},{"osd":5,"last update":"Fri Sep 13 14:10:15 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.62,"5min":0.62,"15min":0.62},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.49099999999999999}]},{"osd":7,"last update":"Fri Sep 13 14:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.45800000000000002}]}]},{"osd":5,"up_from":38,"seq":163208757281,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31772,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713380,"statfs":{"total":95995035648,"available":95962501120,"internally_reserved":0,"allocated":671744,"data_stored":282781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:10:49 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.52900000000000003,"15min":0.53300000000000003},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.33900000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.67500000000000004,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.61599999999999999}]},{"osd":1,"last update":"Fri Sep 13 14:10:49 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.39900000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.65900000000000003,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.504,"5min":0.52100000000000002,"15min":0.52400000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.54800000000000004}]},{"osd":2,"last update":"Fri Sep 13 14:10:49 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.54100000000000004,"15min":0.54500000000000004},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.499,"5min":0.52000000000000002,"15min":0.52400000000000002},"min":{"1min":0.32600000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.69299999999999995,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.57999999999999996}]},{"osd":3,"last update":"Fri Sep 13 14:10:49 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.50700000000000001,"15min":0.50900000000000001},"min":{"1min":0.35199999999999998,"5min":0.312,"15min":0.312},"max":{"1min":0.76600000000000001,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.54100000000000004,"15min":0.54400000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.56299999999999994}]},{"osd":4,"last update":"Fri Sep 13 14:10:49 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.39800000000000002,"15min":0.39600000000000002},"min":{"1min":0.27600000000000002,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.61199999999999999,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.41299999999999998,"15min":0.41699999999999998},"min":{"1min":0.30499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.58399999999999996,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.39800000000000002}]},{"osd":6,"last update":"Fri Sep 13 14:11:16 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.47899999999999998,"15min":0.48099999999999998},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.69999999999999996,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.495,"5min":0.47899999999999998,"15min":0.47599999999999998},"min":{"1min":0.33400000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.68700000000000006,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.42499999999999999}]},{"osd":7,"last update":"Fri Sep 13 14:10:38 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.438}]}]},{"osd":4,"up_from":30,"seq":128849018917,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27672,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717480,"statfs":{"total":95995035648,"available":95966699520,"internally_reserved":0,"allocated":667648,"data_stored":282820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:10:27 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.57499999999999996,"15min":0.59399999999999997},"min":{"1min":0.35299999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.59299999999999997,"5min":1.258,"15min":1.258},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.56699999999999995,"15min":0.58099999999999996},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.76000000000000001,"5min":0.997,"15min":0.997},"last":0.50700000000000001}]},{"osd":1,"last update":"Fri Sep 13 14:10:27 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.57299999999999995,"15min":0.58899999999999997},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68799999999999994,"5min":1.117,"15min":1.117},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.56399999999999995,"15min":0.57799999999999996},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.745,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":0.53500000000000003}]},{"osd":2,"last update":"Fri Sep 13 14:10:27 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.56399999999999995,"15min":0.57999999999999996},"min":{"1min":0.38800000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.58899999999999997,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.57499999999999996,"15min":0.59099999999999997},"min":{"1min":0.375,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.61199999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.64300000000000002}]},{"osd":3,"last update":"Fri Sep 13 14:10:27 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.58899999999999997,"15min":0.60299999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.63500000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.58399999999999996,"15min":0.59999999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.65700000000000003,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.58399999999999996}]},{"osd":5,"last update":"Fri Sep 13 14:10:53 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.35799999999999998,"15min":0.35399999999999998},"min":{"1min":0.28999999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.47399999999999998,"15min":0.46700000000000003},"min":{"1min":0.34899999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.66500000000000004}]},{"osd":6,"last update":"Fri Sep 13 14:11:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.433,"15min":0.42699999999999999},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.41699999999999998,"15min":0.40699999999999997},"min":{"1min":0.30299999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.59899999999999998}]},{"osd":7,"last update":"Fri Sep 13 14:10:36 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.63},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.55900000000000005}]}]},{"osd":3,"up_from":25,"seq":107374182442,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717476,"statfs":{"total":95995035648,"available":95966695424,"internally_reserved":0,"allocated":671744,"data_stored":283093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:11:12 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.35199999999999998,"15min":0.35399999999999998},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.245},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.379,"15min":0.38100000000000001},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.55700000000000005,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.372}]},{"osd":1,"last update":"Fri Sep 13 14:11:12 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.39300000000000002,"15min":0.39600000000000002},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.307},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.38200000000000001,"15min":0.38500000000000001},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.54400000000000004,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.52000000000000002}]},{"osd":2,"last update":"Fri Sep 13 14:11:12 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.31,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.40100000000000002,"15min":0.40000000000000002},"min":{"1min":0.27800000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.33400000000000002}]},{"osd":4,"last update":"Fri Sep 13 14:10:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.51700000000000002,"15min":0.51200000000000001},"min":{"1min":0.379,"5min":0.373,"15min":0.373},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.52500000000000002,"15min":0.52100000000000002},"min":{"1min":0.39200000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.35399999999999998}]},{"osd":5,"last update":"Fri Sep 13 14:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.56599999999999995,"15min":0.56399999999999995},"min":{"1min":0.40100000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.53000000000000003,"15min":0.52300000000000002},"min":{"1min":0.379,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.47699999999999998}]},{"osd":6,"last update":"Fri Sep 13 14:11:17 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60399999999999998,"15min":0.60299999999999998},"min":{"1min":0.42599999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.60699999999999998,"15min":0.61399999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.72099999999999997,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.53600000000000003}]},{"osd":7,"last update":"Fri Sep 13 14:10:38 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.42399999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411374,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27672,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717480,"statfs":{"total":95995035648,"available":95966699520,"internally_reserved":0,"allocated":671744,"data_stored":283093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:10:44 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.377,"15min":0.37},"min":{"1min":0.224,"5min":0.22,"15min":0.22},"max":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.39400000000000002,"15min":0.38900000000000001},"min":{"1min":0.255,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.317}]},{"osd":1,"last update":"Fri Sep 13 14:10:44 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.376,"15min":0.36799999999999999},"min":{"1min":0.24099999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.39900000000000002},"min":{"1min":0.27000000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.34000000000000002}]},{"osd":3,"last update":"Fri Sep 13 14:11:10 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.46400000000000002,"15min":0.47699999999999998},"min":{"1min":0.318,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.56200000000000006,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.31},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.436,"15min":0.45400000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.48299999999999998,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.35999999999999999}]},{"osd":4,"last update":"Fri Sep 13 14:10:28 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.56899999999999995,"15min":0.57799999999999996},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.61099999999999999,"15min":0.62},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.372}]},{"osd":5,"last update":"Fri Sep 13 14:10:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.63900000000000001,"15min":0.64800000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.80800000000000005,"5min":1.2,"15min":1.2},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.57899999999999996,"15min":0.58499999999999996},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.70999999999999996,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.45200000000000001}]},{"osd":6,"last update":"Fri Sep 13 14:11:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.56100000000000005,"15min":0.55900000000000005},"min":{"1min":0.40200000000000002,"5min":0.378,"15min":0.378},"max":{"1min":0.751,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.57199999999999995,"15min":0.56599999999999995},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.46600000000000003}]},{"osd":7,"last update":"Fri Sep 13 14:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.44}]}]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717476,"statfs":{"total":95995035648,"available":95966695424,"internally_reserved":0,"allocated":671744,"data_stored":282839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:10:21 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.32000000000000001,"15min":0.29799999999999999},"min":{"1min":0.24199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.33600000000000002,"15min":0.314},"min":{"1min":0.23100000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"last":0.41499999999999998}]},{"osd":2,"last update":"Fri Sep 13 14:10:52 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.35999999999999999,"15min":0.34999999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.53300000000000003,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.35999999999999999,"15min":0.35399999999999998},"min":{"1min":0.253,"5min":0.224,"15min":0.224},"max":{"1min":0.50900000000000001,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.42999999999999999}]},{"osd":3,"last update":"Fri Sep 13 14:11:08 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.40100000000000002,"15min":0.40300000000000002},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.51700000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.42699999999999999,"15min":0.42899999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.53100000000000003,"5min":0.621,"15min":0.621},"last":0.53000000000000003}]},{"osd":4,"last update":"Fri Sep 13 14:10:31 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55300000000000005,"15min":0.55200000000000005},"min":{"1min":0.41499999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.72899999999999998,"5min":1.133,"15min":1.133},"last":0.753},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.61199999999999999,"15min":0.626},"min":{"1min":0.34499999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.71899999999999997,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.70899999999999996}]},{"osd":5,"last update":"Fri Sep 13 14:10:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.54700000000000004,"15min":0.54500000000000004},"min":{"1min":0.39700000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.77300000000000002}]},{"osd":6,"last update":"Fri Sep 13 14:11:17 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.57999999999999996,"15min":0.57299999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.625,"5min":0.60999999999999999,"15min":0.60799999999999998},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.78700000000000003}]},{"osd":7,"last update":"Fri Sep 13 14:10:38 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.76400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738424,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28256,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716896,"statfs":{"total":95995035648,"available":95966101504,"internally_reserved":0,"allocated":1265664,"data_stored":873461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 14:10:24 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.34499999999999997,"15min":0.32900000000000001},"min":{"1min":0.27400000000000002,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.55500000000000005,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.34999999999999998,"15min":0.33200000000000002},"min":{"1min":0.24199999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.54800000000000004,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.38200000000000001}]},{"osd":2,"last update":"Fri Sep 13 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.379,"15min":0.376},"min":{"1min":0.25900000000000001,"5min":0.192,"15min":0.192},"max":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.39000000000000001,"15min":0.39400000000000002},"min":{"1min":0.26700000000000002,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.434,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.34399999999999997}]},{"osd":3,"last update":"Fri Sep 13 14:11:10 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.40899999999999997,"15min":0.42099999999999999},"min":{"1min":0.23599999999999999,"5min":0.224,"15min":0.224},"max":{"1min":0.48199999999999998,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.45100000000000001,"15min":0.46200000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.48599999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.45400000000000001}]},{"osd":4,"last update":"Fri Sep 13 14:10:28 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.57199999999999995,"15min":0.58199999999999996},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.67200000000000004,"5min":1.044,"15min":1.044},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.57299999999999995,"15min":0.58499999999999996},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.73799999999999999,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.56699999999999995}]},{"osd":5,"last update":"Fri Sep 13 14:10:51 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.60399999999999998,"15min":0.62},"min":{"1min":0.34699999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.63600000000000001,"5min":1.109,"15min":1.109},"last":0.755},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.58999999999999997,"15min":0.60899999999999999},"min":{"1min":0.36899999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.57199999999999995,"5min":1.151,"15min":1.151},"last":0.59299999999999997}]},{"osd":6,"last update":"Fri Sep 13 14:11:15 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.56899999999999995,"15min":0.57399999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.72799999999999998,"5min":0.879,"15min":0.879},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.58699999999999997,"15min":0.58899999999999997},"min":{"1min":0.42699999999999999,"5min":0.379,"15min":0.379},"max":{"1min":0.77500000000000002,"5min":0.87,"15min":0.87},"last":0.73099999999999998}]},{"osd":7,"last update":"Fri Sep 13 14:10:38 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.77000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:11:22.655 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-13T14:11:22.655 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-13T14:11:22.655 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-13T14:11:22.656 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph health --format=json 2024-09-13T14:11:22.857 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:11:23.243 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:23 smithi102 ceph-mon[29355]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-13T14:11:23.244 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:23 smithi102 ceph-mon[29355]: Reconfiguring daemon prometheus.a on smithi195 2024-09-13T14:11:23.244 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:23 smithi102 ceph-mon[40543]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-13T14:11:23.244 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:23 smithi102 ceph-mon[40543]: Reconfiguring daemon prometheus.a on smithi195 2024-09-13T14:11:23.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:23 smithi195 ceph-mon[33205]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-13T14:11:23.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:23 smithi195 ceph-mon[33205]: Reconfiguring daemon prometheus.a on smithi195 2024-09-13T14:11:24.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:24 smithi195 ceph-mon[33205]: pgmap v47: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:24.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:24 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:11:24.319 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:11:24 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a[76563]: ts=2024-09-13T14:11:24.006Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000309843s 2024-09-13T14:11:24.320 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:24 smithi102 ceph-mon[29355]: pgmap v47: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:24.320 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:24 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:11:24.320 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:24 smithi102 ceph-mon[40543]: pgmap v47: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:24.320 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:24 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:11:24.777 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:11:24.777 INFO:teuthology.orchestra.run.smithi102.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-13T14:11:25.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:25 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/2748824300' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-13T14:11:25.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:25 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/2748824300' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-13T14:11:25.445 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:25 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/2748824300' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-13T14:11:25.536 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-13T14:11:25.537 INFO:tasks.cephadm:Setup complete, yielding 2024-09-13T14:11:25.537 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-13T14:11:25.548 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi102.front.sepia.ceph.com 2024-09-13T14:11:25.548 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- bash -c 'ceph fs volume create foo' 2024-09-13T14:11:25.744 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:11:25.747 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:11:25 smithi195 systemd[1]: Stopping Ceph prometheus.a for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T14:11:26.138 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:11:25 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[62273]: ts=2024-09-13T14:11:25.859Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-13T14:11:26.139 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:11:25 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[62273]: ts=2024-09-13T14:11:25.860Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-13T14:11:26.139 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:11:25 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[62273]: ts=2024-09-13T14:11:25.860Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-13T14:11:26.139 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:11:25 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[62273]: ts=2024-09-13T14:11:25.860Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-13T14:11:26.139 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:11:25 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[62273]: ts=2024-09-13T14:11:25.860Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-13T14:11:26.139 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:11:25 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[62273]: ts=2024-09-13T14:11:25.860Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-13T14:11:26.139 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:11:25 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[62273]: ts=2024-09-13T14:11:25.860Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-13T14:11:26.140 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:11:25 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[62273]: ts=2024-09-13T14:11:25.860Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-13T14:11:26.140 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:11:25 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[62273]: ts=2024-09-13T14:11:25.860Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-13T14:11:26.140 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:11:25 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[62273]: ts=2024-09-13T14:11:25.860Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-13T14:11:26.140 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:11:25 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[62273]: ts=2024-09-13T14:11:25.860Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-13T14:11:26.140 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:11:25 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[62273]: ts=2024-09-13T14:11:25.860Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-13T14:11:26.141 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:11:25 smithi195 podman[65612]: 2024-09-13 14:11:25.870456143 +0000 UTC m=+0.122819576 container died 579ec0adef1f5d31626c4aba1da83e108b7365a1f287c3bc5a01b56ec2f746e6 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:11:26.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:26 smithi102 ceph-mon[29355]: pgmap v48: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:26.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:26 smithi102 ceph-mon[40543]: pgmap v48: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:26.450 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:26 smithi195 ceph-mon[33205]: pgmap v48: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:26.451 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:11:26 smithi195 podman[65612]: 2024-09-13 14:11:26.232603312 +0000 UTC m=+0.484966753 container cleanup 579ec0adef1f5d31626c4aba1da83e108b7365a1f287c3bc5a01b56ec2f746e6 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:11:26.451 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:11:26 smithi195 bash[65612]: ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a 2024-09-13T14:11:26.747 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:11:26 smithi195 podman[65630]: 2024-09-13 14:11:26.450352465 +0000 UTC m=+0.582553025 container remove 579ec0adef1f5d31626c4aba1da83e108b7365a1f287c3bc5a01b56ec2f746e6 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:11:26.747 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:11:26 smithi195 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-09-13T14:11:26.747 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:11:26 smithi195 systemd[1]: Stopped Ceph prometheus.a for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:11:26.747 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:11:26 smithi195 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@prometheus.a.service: Consumed 1.157s CPU time. 2024-09-13T14:11:26.747 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:11:26 smithi195 systemd[1]: Starting Ceph prometheus.a for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T14:11:27.041 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:11:26 smithi195 podman[65728]: 2024-09-13 14:11:26.692813628 +0000 UTC m=+0.016784925 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-09-13T14:11:27.042 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:11:26 smithi195 podman[65728]: 2024-09-13 14:11:26.826515991 +0000 UTC m=+0.150487277 container create f526a9818eaef5ea1580a25b02d632e12b559d580f08bc314f719a0d9c698330 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:11:27.042 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:11:27 smithi195 podman[65728]: 2024-09-13 14:11:27.036621289 +0000 UTC m=+0.360592577 container init f526a9818eaef5ea1580a25b02d632e12b559d580f08bc314f719a0d9c698330 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:11:27.237 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:11:27 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: [13/Sep/2024:14:11:27] ENGINE Bus STOPPING 2024-09-13T14:11:27.497 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:11:27 smithi195 podman[65728]: 2024-09-13 14:11:27.041090493 +0000 UTC m=+0.365061780 container start f526a9818eaef5ea1580a25b02d632e12b559d580f08bc314f719a0d9c698330 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:11:27.498 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:11:27 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[65743]: ts=2024-09-13T14:11:27.064Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-13T14:11:27.498 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:11:27 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[65743]: ts=2024-09-13T14:11:27.064Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-13T14:11:27.498 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:11:27 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[65743]: ts=2024-09-13T14:11:27.064Z caller=main.go:570 level=info host_details="(Linux 5.14.0-505.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024 x86_64 smithi195 (none))" 2024-09-13T14:11:27.498 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:11:27 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[65743]: ts=2024-09-13T14:11:27.064Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-13T14:11:27.498 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:11:27 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[65743]: ts=2024-09-13T14:11:27.064Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-13T14:11:27.498 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:11:27 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[65743]: ts=2024-09-13T14:11:27.066Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-13T14:11:27.498 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:11:27 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[65743]: ts=2024-09-13T14:11:27.066Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-13T14:11:27.498 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:11:27 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[65743]: ts=2024-09-13T14:11:27.067Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-13T14:11:27.498 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:11:27 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[65743]: ts=2024-09-13T14:11:27.067Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-13T14:11:27.498 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:11:27 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[65743]: ts=2024-09-13T14:11:27.069Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-13T14:11:27.498 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:11:27 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[65743]: ts=2024-09-13T14:11:27.069Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.911µs 2024-09-13T14:11:27.499 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:11:27 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[65743]: ts=2024-09-13T14:11:27.069Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-13T14:11:27.499 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:11:27 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[65743]: ts=2024-09-13T14:11:27.070Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-09-13T14:11:27.499 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:11:27 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[65743]: ts=2024-09-13T14:11:27.071Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-09-13T14:11:27.499 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:11:27 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[65743]: ts=2024-09-13T14:11:27.071Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=18.997µs wal_replay_duration=1.465843ms wbl_replay_duration=135ns total_replay_duration=1.503941ms 2024-09-13T14:11:27.499 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:11:27 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[65743]: ts=2024-09-13T14:11:27.071Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-13T14:11:27.499 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:11:27 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[65743]: ts=2024-09-13T14:11:27.071Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-13T14:11:27.499 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:11:27 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[65743]: ts=2024-09-13T14:11:27.071Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-13T14:11:27.499 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:11:27 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[65743]: ts=2024-09-13T14:11:27.091Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=19.974955ms db_storage=780ns remote_storage=1.009µs web_handler=350ns query_engine=525ns scrape=555.987µs scrape_sd=89.621µs notify=9.597µs notify_sd=10.225µs rules=18.985752ms tracing=6.321µs 2024-09-13T14:11:27.499 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:11:27 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[65743]: ts=2024-09-13T14:11:27.091Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-13T14:11:27.499 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:11:27 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[65743]: ts=2024-09-13T14:11:27.091Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-13T14:11:27.499 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:11:27 smithi195 bash[65728]: f526a9818eaef5ea1580a25b02d632e12b559d580f08bc314f719a0d9c698330 2024-09-13T14:11:27.499 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:11:27 smithi195 systemd[1]: Started Ceph prometheus.a for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:11:27.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:11:27 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: [13/Sep/2024:14:11:27] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-13T14:11:27.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:11:27 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: [13/Sep/2024:14:11:27] ENGINE Bus STOPPED 2024-09-13T14:11:27.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:11:27 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: [13/Sep/2024:14:11:27] ENGINE Bus STARTING 2024-09-13T14:11:27.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:11:27 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: [13/Sep/2024:14:11:27] ENGINE Serving on http://:::9283 2024-09-13T14:11:27.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:11:27 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: [13/Sep/2024:14:11:27] ENGINE Bus STARTED 2024-09-13T14:11:27.669 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:11:27 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: [13/Sep/2024:14:11:27] ENGINE Bus STOPPING 2024-09-13T14:11:28.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:28 smithi102 ceph-mon[29355]: pgmap v49: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:28.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:28 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:28.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:28 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:28.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:28 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T14:11:28.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:28 smithi102 ceph-mon[29355]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T14:11:28.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:28 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi102.front.sepia.ceph.com:9093"}]: dispatch 2024-09-13T14:11:28.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:28 smithi102 ceph-mon[29355]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi102.front.sepia.ceph.com:9093"}]: dispatch 2024-09-13T14:11:28.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:28 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:28.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:28 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:11:28.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:28 smithi102 ceph-mon[29355]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:11:28.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:28 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi195.front.sepia.ceph.com:9095"}]: dispatch 2024-09-13T14:11:28.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:28 smithi102 ceph-mon[29355]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi195.front.sepia.ceph.com:9095"}]: dispatch 2024-09-13T14:11:28.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:28 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:28.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:28 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T14:11:28.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:28 smithi102 ceph-mon[29355]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T14:11:28.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:28 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi195.front.sepia.ceph.com:3000"}]: dispatch 2024-09-13T14:11:28.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:28 smithi102 ceph-mon[29355]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi195.front.sepia.ceph.com:3000"}]: dispatch 2024-09-13T14:11:28.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:28 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:28.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:28 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:11:28.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:28 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-09-13T14:11:28.170 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:11:27 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: [13/Sep/2024:14:11:27] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-13T14:11:28.170 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:11:27 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: [13/Sep/2024:14:11:27] ENGINE Bus STOPPED 2024-09-13T14:11:28.170 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:11:27 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: [13/Sep/2024:14:11:27] ENGINE Bus STARTING 2024-09-13T14:11:28.170 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:11:28 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: [13/Sep/2024:14:11:28] ENGINE Serving on http://:::9283 2024-09-13T14:11:28.170 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:11:28 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: [13/Sep/2024:14:11:28] ENGINE Bus STARTED 2024-09-13T14:11:28.170 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:11:28 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: [13/Sep/2024:14:11:28] ENGINE Bus STOPPING 2024-09-13T14:11:28.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:28 smithi102 ceph-mon[40543]: pgmap v49: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:28.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:28 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:28.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:28 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:28.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:28 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T14:11:28.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:28 smithi102 ceph-mon[40543]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T14:11:28.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:28 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi102.front.sepia.ceph.com:9093"}]: dispatch 2024-09-13T14:11:28.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:28 smithi102 ceph-mon[40543]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi102.front.sepia.ceph.com:9093"}]: dispatch 2024-09-13T14:11:28.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:28 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:28.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:28 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:11:28.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:28 smithi102 ceph-mon[40543]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:11:28.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:28 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi195.front.sepia.ceph.com:9095"}]: dispatch 2024-09-13T14:11:28.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:28 smithi102 ceph-mon[40543]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi195.front.sepia.ceph.com:9095"}]: dispatch 2024-09-13T14:11:28.172 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:28 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:28.172 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:28 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T14:11:28.172 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:28 smithi102 ceph-mon[40543]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T14:11:28.172 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:28 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi195.front.sepia.ceph.com:3000"}]: dispatch 2024-09-13T14:11:28.172 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:28 smithi102 ceph-mon[40543]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi195.front.sepia.ceph.com:3000"}]: dispatch 2024-09-13T14:11:28.172 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:28 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:28.172 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:28 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:11:28.172 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:28 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-09-13T14:11:28.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:28 smithi195 ceph-mon[33205]: pgmap v49: 9 pgs: 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:28.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:28 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:28.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:28 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:28.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:28 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T14:11:28.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:28 smithi195 ceph-mon[33205]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T14:11:28.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:28 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi102.front.sepia.ceph.com:9093"}]: dispatch 2024-09-13T14:11:28.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:28 smithi195 ceph-mon[33205]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi102.front.sepia.ceph.com:9093"}]: dispatch 2024-09-13T14:11:28.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:28 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:28.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:28 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:11:28.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:28 smithi195 ceph-mon[33205]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:11:28.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:28 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi195.front.sepia.ceph.com:9095"}]: dispatch 2024-09-13T14:11:28.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:28 smithi195 ceph-mon[33205]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi195.front.sepia.ceph.com:9095"}]: dispatch 2024-09-13T14:11:28.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:28 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:28.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:28 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T14:11:28.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:28 smithi195 ceph-mon[33205]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T14:11:28.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:28 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi195.front.sepia.ceph.com:3000"}]: dispatch 2024-09-13T14:11:28.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:28 smithi195 ceph-mon[33205]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi195.front.sepia.ceph.com:3000"}]: dispatch 2024-09-13T14:11:28.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:28 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:28.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:28 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:11:28.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:28 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-09-13T14:11:28.918 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:11:28 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: [13/Sep/2024:14:11:28] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-13T14:11:28.918 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:11:28 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: [13/Sep/2024:14:11:28] ENGINE Bus STOPPED 2024-09-13T14:11:28.918 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:11:28 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: [13/Sep/2024:14:11:28] ENGINE Bus STARTING 2024-09-13T14:11:28.918 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:11:28 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: [13/Sep/2024:14:11:28] ENGINE Serving on http://:::9283 2024-09-13T14:11:28.918 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:11:28 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: [13/Sep/2024:14:11:28] ENGINE Bus STARTED 2024-09-13T14:11:29.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:29 smithi102 ceph-mon[40543]: from='client.14610 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foo", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:11:29.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:29 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]': finished 2024-09-13T14:11:29.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:29 smithi102 ceph-mon[40543]: osdmap e67: 8 total, 8 up, 8 in 2024-09-13T14:11:29.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:29 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-09-13T14:11:29.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:29 smithi102 ceph-mon[29355]: from='client.14610 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foo", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:11:29.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:29 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]': finished 2024-09-13T14:11:29.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:29 smithi102 ceph-mon[29355]: osdmap e67: 8 total, 8 up, 8 in 2024-09-13T14:11:29.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:29 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-09-13T14:11:29.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:29 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-a[29332]: 2024-09-13T14:11:29.231+0000 7f813a16e640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T14:11:29.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:29 smithi195 ceph-mon[33205]: from='client.14610 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foo", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:11:29.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:29 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]': finished 2024-09-13T14:11:29.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:29 smithi195 ceph-mon[33205]: osdmap e67: 8 total, 8 up, 8 in 2024-09-13T14:11:29.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:29 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-09-13T14:11:30.222 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-09-13T14:11:30.302 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:30 smithi195 ceph-mon[33205]: pgmap v51: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:30.302 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:30 smithi195 ceph-mon[33205]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:11:30.302 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:30 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]': finished 2024-09-13T14:11:30.302 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:30 smithi195 ceph-mon[33205]: osdmap e68: 8 total, 8 up, 8 in 2024-09-13T14:11:30.302 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:30 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-09-13T14:11:30.303 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:30 smithi195 ceph-mon[33205]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T14:11:30.303 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:30 smithi195 ceph-mon[33205]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-13T14:11:30.303 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:30 smithi195 ceph-mon[33205]: osdmap e69: 8 total, 8 up, 8 in 2024-09-13T14:11:30.303 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:30 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd='[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]': finished 2024-09-13T14:11:30.303 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:30 smithi195 ceph-mon[33205]: fsmap foo:0 2024-09-13T14:11:30.303 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:30 smithi195 ceph-mon[33205]: Saving service mds.foo spec with placement count:2 2024-09-13T14:11:30.303 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:30 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:30.451 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:30 smithi102 ceph-mon[29355]: pgmap v51: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:30.452 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:30 smithi102 ceph-mon[29355]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:11:30.452 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:30 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]': finished 2024-09-13T14:11:30.452 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:30 smithi102 ceph-mon[29355]: osdmap e68: 8 total, 8 up, 8 in 2024-09-13T14:11:30.452 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:30 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-09-13T14:11:30.452 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:30 smithi102 ceph-mon[29355]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T14:11:30.452 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:30 smithi102 ceph-mon[29355]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-13T14:11:30.452 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:30 smithi102 ceph-mon[29355]: osdmap e69: 8 total, 8 up, 8 in 2024-09-13T14:11:30.452 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:30 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd='[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]': finished 2024-09-13T14:11:30.452 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:30 smithi102 ceph-mon[29355]: fsmap foo:0 2024-09-13T14:11:30.452 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:30 smithi102 ceph-mon[29355]: Saving service mds.foo spec with placement count:2 2024-09-13T14:11:30.452 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:30 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:30.453 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:30 smithi102 ceph-mon[40543]: pgmap v51: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:30.453 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:30 smithi102 ceph-mon[40543]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:11:30.453 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:30 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]': finished 2024-09-13T14:11:30.453 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:30 smithi102 ceph-mon[40543]: osdmap e68: 8 total, 8 up, 8 in 2024-09-13T14:11:30.453 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:30 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-09-13T14:11:30.453 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:30 smithi102 ceph-mon[40543]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T14:11:30.453 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:30 smithi102 ceph-mon[40543]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-13T14:11:30.453 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:30 smithi102 ceph-mon[40543]: osdmap e69: 8 total, 8 up, 8 in 2024-09-13T14:11:30.453 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:30 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd='[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]': finished 2024-09-13T14:11:30.453 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:30 smithi102 ceph-mon[40543]: fsmap foo:0 2024-09-13T14:11:30.453 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:30 smithi102 ceph-mon[40543]: Saving service mds.foo spec with placement count:2 2024-09-13T14:11:30.454 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:30 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:30.944 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:11:31.336 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:31 smithi102 ceph-mon[29355]: osdmap e70: 8 total, 8 up, 8 in 2024-09-13T14:11:31.336 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:31 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:31.336 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:31 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:31.336 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:31 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:31.336 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:31 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:31.336 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:31 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:11:31.337 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:31 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:11:31.337 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:31 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:31.337 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:31 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi102.njetxr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-13T14:11:31.337 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:31 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi102.njetxr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-13T14:11:31.337 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:31 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:11:31.337 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:31 smithi102 ceph-mon[40543]: osdmap e70: 8 total, 8 up, 8 in 2024-09-13T14:11:31.337 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:31 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:31.337 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:31 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:31.337 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:31 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:31.337 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:31 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:31.337 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:31 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:11:31.338 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:31 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:11:31.338 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:31 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:31.338 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:31 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi102.njetxr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-13T14:11:31.338 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:31 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi102.njetxr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-13T14:11:31.338 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:31 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:11:31.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:31 smithi195 ceph-mon[33205]: osdmap e70: 8 total, 8 up, 8 in 2024-09-13T14:11:31.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:31 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:31.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:31 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:31.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:31 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:31.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:31 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:31.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:31 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:11:31.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:31 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:11:31.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:31 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:31.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:31 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi102.njetxr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-13T14:11:31.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:31 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi102.njetxr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-13T14:11:31.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:31 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:11:32.142 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:11:32 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a[76563]: ts=2024-09-13T14:11:32.009Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003167282s 2024-09-13T14:11:32.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:32 smithi102 ceph-mon[29355]: Deploying daemon mds.foo.smithi102.njetxr on smithi102 2024-09-13T14:11:32.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:32 smithi102 ceph-mon[29355]: pgmap v55: 73 pgs: 31 creating+peering, 42 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:32.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:32 smithi102 ceph-mon[29355]: osdmap e71: 8 total, 8 up, 8 in 2024-09-13T14:11:32.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:32 smithi102 ceph-mon[29355]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:11:32.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:32 smithi102 ceph-mon[40543]: Deploying daemon mds.foo.smithi102.njetxr on smithi102 2024-09-13T14:11:32.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:32 smithi102 ceph-mon[40543]: pgmap v55: 73 pgs: 31 creating+peering, 42 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:32.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:32 smithi102 ceph-mon[40543]: osdmap e71: 8 total, 8 up, 8 in 2024-09-13T14:11:32.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:32 smithi102 ceph-mon[40543]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:11:32.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:32 smithi195 ceph-mon[33205]: Deploying daemon mds.foo.smithi102.njetxr on smithi102 2024-09-13T14:11:32.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:32 smithi195 ceph-mon[33205]: pgmap v55: 73 pgs: 31 creating+peering, 42 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:32.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:32 smithi195 ceph-mon[33205]: osdmap e71: 8 total, 8 up, 8 in 2024-09-13T14:11:32.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:32 smithi195 ceph-mon[33205]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:11:34.197 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-09-13T14:11:34.644 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:34 smithi102 ceph-mon[29355]: pgmap v57: 73 pgs: 31 creating+peering, 42 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:34.645 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:34 smithi102 ceph-mon[40543]: pgmap v57: 73 pgs: 31 creating+peering, 42 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:34.729 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:11:34.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:34 smithi195 ceph-mon[33205]: pgmap v57: 73 pgs: 31 creating+peering, 42 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:36.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:36 smithi102 ceph-mon[29355]: pgmap v58: 73 pgs: 73 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:36.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:36 smithi102 ceph-mon[40543]: pgmap v58: 73 pgs: 73 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:36.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:36 smithi195 ceph-mon[33205]: pgmap v58: 73 pgs: 73 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:37.510 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:11:37 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:11:37] "GET /metrics HTTP/1.1" 200 28517 "" "Prometheus/2.43.0" 2024-09-13T14:11:37.906 INFO:teuthology.run_tasks:Running task ceph.healthy... 2024-09-13T14:11:37.916 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-13T14:11:37.916 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-13T14:11:37.916 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph mgr dump --format=json 2024-09-13T14:11:38.115 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:11:38.436 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:38 smithi195 ceph-mon[33205]: pgmap v59: 73 pgs: 73 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:38.436 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:38 smithi195 ceph-mon[33205]: mds.? [v2:172.21.15.102:6834/54188465,v1:172.21.15.102:6835/54188465] up:boot 2024-09-13T14:11:38.436 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:38 smithi195 ceph-mon[33205]: daemon mds.foo.smithi102.njetxr assigned to filesystem foo as rank 0 (now has 1 ranks) 2024-09-13T14:11:38.437 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:38 smithi195 ceph-mon[33205]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-13T14:11:38.437 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:38 smithi195 ceph-mon[33205]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-13T14:11:38.437 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:38 smithi195 ceph-mon[33205]: Cluster is now healthy 2024-09-13T14:11:38.437 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:38 smithi195 ceph-mon[33205]: fsmap foo:0 1 up:standby 2024-09-13T14:11:38.437 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:38 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi102.njetxr"}]: dispatch 2024-09-13T14:11:38.437 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:38 smithi195 ceph-mon[33205]: fsmap foo:1 {0=foo.smithi102.njetxr=up:creating} 2024-09-13T14:11:38.438 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:38 smithi195 ceph-mon[33205]: daemon mds.foo.smithi102.njetxr is now active in filesystem foo as rank 0 2024-09-13T14:11:38.438 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:38 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:38.438 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:38 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:38.438 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:38 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:38.438 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:38 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi195.yqzrxi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-13T14:11:38.438 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:38 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi195.yqzrxi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-13T14:11:38.439 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:38 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:11:38.439 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:38 smithi195 ceph-mon[33205]: Deploying daemon mds.foo.smithi195.yqzrxi on smithi195 2024-09-13T14:11:38.496 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:38 smithi102 ceph-mon[40543]: pgmap v59: 73 pgs: 73 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:38.496 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:38 smithi102 ceph-mon[40543]: mds.? [v2:172.21.15.102:6834/54188465,v1:172.21.15.102:6835/54188465] up:boot 2024-09-13T14:11:38.497 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:38 smithi102 ceph-mon[40543]: daemon mds.foo.smithi102.njetxr assigned to filesystem foo as rank 0 (now has 1 ranks) 2024-09-13T14:11:38.497 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:38 smithi102 ceph-mon[40543]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-13T14:11:38.497 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:38 smithi102 ceph-mon[40543]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-13T14:11:38.497 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:38 smithi102 ceph-mon[40543]: Cluster is now healthy 2024-09-13T14:11:38.497 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:38 smithi102 ceph-mon[40543]: fsmap foo:0 1 up:standby 2024-09-13T14:11:38.497 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:38 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi102.njetxr"}]: dispatch 2024-09-13T14:11:38.497 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:38 smithi102 ceph-mon[40543]: fsmap foo:1 {0=foo.smithi102.njetxr=up:creating} 2024-09-13T14:11:38.497 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:38 smithi102 ceph-mon[40543]: daemon mds.foo.smithi102.njetxr is now active in filesystem foo as rank 0 2024-09-13T14:11:38.497 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:38 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:38.497 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:38 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:38.497 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:38 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:38.498 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:38 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi195.yqzrxi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-13T14:11:38.498 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:38 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi195.yqzrxi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-13T14:11:38.498 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:38 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:11:38.498 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:38 smithi102 ceph-mon[40543]: Deploying daemon mds.foo.smithi195.yqzrxi on smithi195 2024-09-13T14:11:38.498 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:38 smithi102 ceph-mon[29355]: pgmap v59: 73 pgs: 73 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:38.498 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:38 smithi102 ceph-mon[29355]: mds.? [v2:172.21.15.102:6834/54188465,v1:172.21.15.102:6835/54188465] up:boot 2024-09-13T14:11:38.498 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:38 smithi102 ceph-mon[29355]: daemon mds.foo.smithi102.njetxr assigned to filesystem foo as rank 0 (now has 1 ranks) 2024-09-13T14:11:38.498 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:38 smithi102 ceph-mon[29355]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-13T14:11:38.498 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:38 smithi102 ceph-mon[29355]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-13T14:11:38.498 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:38 smithi102 ceph-mon[29355]: Cluster is now healthy 2024-09-13T14:11:38.499 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:38 smithi102 ceph-mon[29355]: fsmap foo:0 1 up:standby 2024-09-13T14:11:38.499 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:38 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi102.njetxr"}]: dispatch 2024-09-13T14:11:38.499 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:38 smithi102 ceph-mon[29355]: fsmap foo:1 {0=foo.smithi102.njetxr=up:creating} 2024-09-13T14:11:38.499 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:38 smithi102 ceph-mon[29355]: daemon mds.foo.smithi102.njetxr is now active in filesystem foo as rank 0 2024-09-13T14:11:38.499 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:38 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:38.499 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:38 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:38.499 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:38 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:38.500 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:38 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi195.yqzrxi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-13T14:11:38.500 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:38 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi195.yqzrxi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-13T14:11:38.500 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:38 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:11:38.500 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:38 smithi102 ceph-mon[29355]: Deploying daemon mds.foo.smithi195.yqzrxi on smithi195 2024-09-13T14:11:39.376 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:39 smithi102 ceph-mon[29355]: mds.? [v2:172.21.15.102:6834/54188465,v1:172.21.15.102:6835/54188465] up:active 2024-09-13T14:11:39.376 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:39 smithi102 ceph-mon[29355]: fsmap foo:1 {0=foo.smithi102.njetxr=up:active} 2024-09-13T14:11:39.377 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:39 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:11:39.377 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:39 smithi102 ceph-mon[40543]: mds.? [v2:172.21.15.102:6834/54188465,v1:172.21.15.102:6835/54188465] up:active 2024-09-13T14:11:39.377 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:39 smithi102 ceph-mon[40543]: fsmap foo:1 {0=foo.smithi102.njetxr=up:active} 2024-09-13T14:11:39.377 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:39 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:11:39.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:39 smithi195 ceph-mon[33205]: mds.? [v2:172.21.15.102:6834/54188465,v1:172.21.15.102:6835/54188465] up:active 2024-09-13T14:11:39.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:39 smithi195 ceph-mon[33205]: fsmap foo:1 {0=foo.smithi102.njetxr=up:active} 2024-09-13T14:11:39.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:39 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:11:40.049 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:11:40.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:40 smithi102 ceph-mon[29355]: pgmap v60: 73 pgs: 73 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:40.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:40 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:40.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:40 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:40.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:40 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:40.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:40 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:40.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:40 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:40.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:40 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:11:40.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:40 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/3618120189' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-13T14:11:40.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:40 smithi102 ceph-mon[40543]: pgmap v60: 73 pgs: 73 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:40.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:40 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:40.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:40 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:40.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:40 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:40.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:40 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:40.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:40 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:40.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:40 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:11:40.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:40 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/3618120189' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-13T14:11:40.610 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:40 smithi195 ceph-mon[33205]: pgmap v60: 73 pgs: 73 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:40.610 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:40 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:40.610 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:40 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:40.611 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:40 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:40.611 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:40 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:40.611 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:40 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:40.611 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:40 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:11:40.611 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:40 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/3618120189' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-13T14:11:40.759 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":20,"flags":0,"active_gid":14421,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":1522897266},{"type":"v1","addr":"172.21.15.102:6801","nonce":1522897266}]},"active_addr":"172.21.15.102:6801/1522897266","active_change":"2024-09-13T14:10:09.037804+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":24319,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.102:8443/","prometheus":"http://172.21.15.102:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":59,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":2083488703}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":2489373272}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":1430433627}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":1792178465}]}]} 2024-09-13T14:11:40.763 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-13T14:11:40.763 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-13T14:11:40.764 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-13T14:11:41.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:41 smithi102 ceph-mon[29355]: mds.? [v2:172.21.15.195:6832/2017468914,v1:172.21.15.195:6833/2017468914] up:boot 2024-09-13T14:11:41.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:41 smithi102 ceph-mon[29355]: fsmap foo:1 {0=foo.smithi102.njetxr=up:active} 1 up:standby 2024-09-13T14:11:41.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:41 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi195.yqzrxi"}]: dispatch 2024-09-13T14:11:41.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:41 smithi102 ceph-mon[29355]: fsmap foo:1 {0=foo.smithi102.njetxr=up:active} 1 up:standby 2024-09-13T14:11:41.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:41 smithi102 ceph-mon[40543]: mds.? [v2:172.21.15.195:6832/2017468914,v1:172.21.15.195:6833/2017468914] up:boot 2024-09-13T14:11:41.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:41 smithi102 ceph-mon[40543]: fsmap foo:1 {0=foo.smithi102.njetxr=up:active} 1 up:standby 2024-09-13T14:11:41.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:41 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi195.yqzrxi"}]: dispatch 2024-09-13T14:11:41.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:41 smithi102 ceph-mon[40543]: fsmap foo:1 {0=foo.smithi102.njetxr=up:active} 1 up:standby 2024-09-13T14:11:41.525 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:11:41.572 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:41 smithi195 ceph-mon[33205]: mds.? [v2:172.21.15.195:6832/2017468914,v1:172.21.15.195:6833/2017468914] up:boot 2024-09-13T14:11:41.572 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:41 smithi195 ceph-mon[33205]: fsmap foo:1 {0=foo.smithi102.njetxr=up:active} 1 up:standby 2024-09-13T14:11:41.572 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:41 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi195.yqzrxi"}]: dispatch 2024-09-13T14:11:41.572 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:41 smithi195 ceph-mon[33205]: fsmap foo:1 {0=foo.smithi102.njetxr=up:active} 1 up:standby 2024-09-13T14:11:42.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:42 smithi102 ceph-mon[29355]: pgmap v61: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s wr, 4 op/s 2024-09-13T14:11:42.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:42 smithi102 ceph-mon[29355]: mds.? [v2:172.21.15.102:6834/54188465,v1:172.21.15.102:6835/54188465] up:active 2024-09-13T14:11:42.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:42 smithi102 ceph-mon[29355]: fsmap foo:1 {0=foo.smithi102.njetxr=up:active} 1 up:standby 2024-09-13T14:11:42.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:42 smithi102 ceph-mon[40543]: pgmap v61: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s wr, 4 op/s 2024-09-13T14:11:42.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:42 smithi102 ceph-mon[40543]: mds.? [v2:172.21.15.102:6834/54188465,v1:172.21.15.102:6835/54188465] up:active 2024-09-13T14:11:42.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:42 smithi102 ceph-mon[40543]: fsmap foo:1 {0=foo.smithi102.njetxr=up:active} 1 up:standby 2024-09-13T14:11:42.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:42 smithi195 ceph-mon[33205]: pgmap v61: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s wr, 4 op/s 2024-09-13T14:11:42.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:42 smithi195 ceph-mon[33205]: mds.? [v2:172.21.15.102:6834/54188465,v1:172.21.15.102:6835/54188465] up:active 2024-09-13T14:11:42.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:42 smithi195 ceph-mon[33205]: fsmap foo:1 {0=foo.smithi102.njetxr=up:active} 1 up:standby 2024-09-13T14:11:43.508 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:11:43.508 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":71,"fsid":"08cee502-71d9-11ef-bceb-c7b262605968","created":"2024-09-13T14:04:48.179879+0000","modified":"2024-09-13T14:11:31.240493+0000","last_up_change":"2024-09-13T14:09:36.412493+0000","last_in_change":"2024-09-13T14:09:19.207877+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T14:07:41.927240+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T14:10:10.981300+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":65,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-13T14:11:27.935818+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-13T14:11:28.454262+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b43b49e0-1cf0-47b8-aa85-37108f321220","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":466204285},{"type":"v1","addr":"172.21.15.102:6803","nonce":466204285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":466204285},{"type":"v1","addr":"172.21.15.102:6805","nonce":466204285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":466204285},{"type":"v1","addr":"172.21.15.102:6809","nonce":466204285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":466204285},{"type":"v1","addr":"172.21.15.102:6807","nonce":466204285}]},"public_addr":"172.21.15.102:6803/466204285","cluster_addr":"172.21.15.102:6805/466204285","heartbeat_back_addr":"172.21.15.102:6809/466204285","heartbeat_front_addr":"172.21.15.102:6807/466204285","state":["exists","up"]},{"osd":1,"uuid":"27e10d5e-c471-4132-9c73-83c5708c3b08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":1293422793},{"type":"v1","addr":"172.21.15.102:6811","nonce":1293422793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":1293422793},{"type":"v1","addr":"172.21.15.102:6813","nonce":1293422793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1293422793},{"type":"v1","addr":"172.21.15.102:6817","nonce":1293422793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":1293422793},{"type":"v1","addr":"172.21.15.102:6815","nonce":1293422793}]},"public_addr":"172.21.15.102:6811/1293422793","cluster_addr":"172.21.15.102:6813/1293422793","heartbeat_back_addr":"172.21.15.102:6817/1293422793","heartbeat_front_addr":"172.21.15.102:6815/1293422793","state":["exists","up"]},{"osd":2,"uuid":"40a94b2b-2c24-4164-91be-0816a1acf3d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":360502781},{"type":"v1","addr":"172.21.15.102:6819","nonce":360502781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":360502781},{"type":"v1","addr":"172.21.15.102:6821","nonce":360502781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":360502781},{"type":"v1","addr":"172.21.15.102:6825","nonce":360502781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":360502781},{"type":"v1","addr":"172.21.15.102:6823","nonce":360502781}]},"public_addr":"172.21.15.102:6819/360502781","cluster_addr":"172.21.15.102:6821/360502781","heartbeat_back_addr":"172.21.15.102:6825/360502781","heartbeat_front_addr":"172.21.15.102:6823/360502781","state":["exists","up"]},{"osd":3,"uuid":"2ed85faf-6610-4bcc-a09d-4a6f93f5c6a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":3709228546},{"type":"v1","addr":"172.21.15.102:6827","nonce":3709228546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":3709228546},{"type":"v1","addr":"172.21.15.102:6829","nonce":3709228546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3709228546},{"type":"v1","addr":"172.21.15.102:6833","nonce":3709228546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":3709228546},{"type":"v1","addr":"172.21.15.102:6831","nonce":3709228546}]},"public_addr":"172.21.15.102:6827/3709228546","cluster_addr":"172.21.15.102:6829/3709228546","heartbeat_back_addr":"172.21.15.102:6833/3709228546","heartbeat_front_addr":"172.21.15.102:6831/3709228546","state":["exists","up"]},{"osd":4,"uuid":"8a45f350-fb3e-4049-bcc4-532d3e20c238","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":2821651815},{"type":"v1","addr":"172.21.15.195:6801","nonce":2821651815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":2821651815},{"type":"v1","addr":"172.21.15.195:6803","nonce":2821651815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":2821651815},{"type":"v1","addr":"172.21.15.195:6807","nonce":2821651815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":2821651815},{"type":"v1","addr":"172.21.15.195:6805","nonce":2821651815}]},"public_addr":"172.21.15.195:6801/2821651815","cluster_addr":"172.21.15.195:6803/2821651815","heartbeat_back_addr":"172.21.15.195:6807/2821651815","heartbeat_front_addr":"172.21.15.195:6805/2821651815","state":["exists","up"]},{"osd":5,"uuid":"363cf638-db26-4c86-b2be-e075fd09ac16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":3074139237},{"type":"v1","addr":"172.21.15.195:6809","nonce":3074139237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":3074139237},{"type":"v1","addr":"172.21.15.195:6811","nonce":3074139237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":3074139237},{"type":"v1","addr":"172.21.15.195:6815","nonce":3074139237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":3074139237},{"type":"v1","addr":"172.21.15.195:6813","nonce":3074139237}]},"public_addr":"172.21.15.195:6809/3074139237","cluster_addr":"172.21.15.195:6811/3074139237","heartbeat_back_addr":"172.21.15.195:6815/3074139237","heartbeat_front_addr":"172.21.15.195:6813/3074139237","state":["exists","up"]},{"osd":6,"uuid":"f37416e8-9c25-4860-a32f-0d1c404a8bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":4026627594},{"type":"v1","addr":"172.21.15.195:6817","nonce":4026627594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":4026627594},{"type":"v1","addr":"172.21.15.195:6819","nonce":4026627594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":4026627594},{"type":"v1","addr":"172.21.15.195:6823","nonce":4026627594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":4026627594},{"type":"v1","addr":"172.21.15.195:6821","nonce":4026627594}]},"public_addr":"172.21.15.195:6817/4026627594","cluster_addr":"172.21.15.195:6819/4026627594","heartbeat_back_addr":"172.21.15.195:6823/4026627594","heartbeat_front_addr":"172.21.15.195:6821/4026627594","state":["exists","up"]},{"osd":7,"uuid":"9a384a4d-fddb-4fb3-843a-77c19cca0368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2059403787},{"type":"v1","addr":"172.21.15.195:6825","nonce":2059403787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2059403787},{"type":"v1","addr":"172.21.15.195:6827","nonce":2059403787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2059403787},{"type":"v1","addr":"172.21.15.195:6831","nonce":2059403787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2059403787},{"type":"v1","addr":"172.21.15.195:6829","nonce":2059403787}]},"public_addr":"172.21.15.195:6825/2059403787","cluster_addr":"172.21.15.195:6827/2059403787","heartbeat_back_addr":"172.21.15.195:6831/2059403787","heartbeat_front_addr":"172.21.15.195:6829/2059403787","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:06:53.502746+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:07:16.251669+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:07:38.848272+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:08:02.013035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:08:23.043482+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:08:45.653159+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:09:10.187018+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:09:34.393753+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/1765904351":"2024-09-14T14:10:09.037657+0000","172.21.15.102:0/2416257718":"2024-09-14T14:10:09.037657+0000","172.21.15.102:0/1582001159":"2024-09-14T14:05:15.286479+0000","172.21.15.102:0/2308924552":"2024-09-14T14:10:09.037657+0000","172.21.15.102:6800/2761043787":"2024-09-14T14:05:15.286479+0000","172.21.15.102:6801/2761043787":"2024-09-14T14:05:15.286479+0000","172.21.15.102:0/2444394840":"2024-09-14T14:05:37.634624+0000","172.21.15.102:0/1574792592":"2024-09-14T14:05:15.286479+0000","172.21.15.102:0/3189088157":"2024-09-14T14:05:15.286479+0000","172.21.15.102:6800/3480998245":"2024-09-14T14:05:37.634624+0000","172.21.15.102:6801/3480998245":"2024-09-14T14:05:37.634624+0000","172.21.15.102:0/1160237252":"2024-09-14T14:10:09.037657+0000","172.21.15.102:0/3399546099":"2024-09-14T14:05:37.634624+0000","172.21.15.102:0/1939570054":"2024-09-14T14:05:37.634624+0000","172.21.15.102:6800/63514412":"2024-09-14T14:10:09.037657+0000","172.21.15.102:6801/63514412":"2024-09-14T14:10:09.037657+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T14:11:44.507 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-13T14:11:44.508 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-13T14:11:44.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:44 smithi102 ceph-mon[29355]: pgmap v62: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-13T14:11:44.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:44 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:44.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:44 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:44.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:44 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/3878408140' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:11:44.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:44 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:44.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:44 smithi102 ceph-mon[40543]: pgmap v62: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-13T14:11:44.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:44 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:44.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:44 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:44.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:44 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/3878408140' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:11:44.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:44 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:44.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:44 smithi195 ceph-mon[33205]: pgmap v62: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-13T14:11:44.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:44 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:44.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:44 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:44.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:44 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/3878408140' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:11:44.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:44 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:44.897 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:11:45.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:45 smithi102 ceph-mon[29355]: mds.? [v2:172.21.15.195:6832/2017468914,v1:172.21.15.195:6833/2017468914] up:standby 2024-09-13T14:11:45.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:45 smithi102 ceph-mon[29355]: fsmap foo:1 {0=foo.smithi102.njetxr=up:active} 1 up:standby 2024-09-13T14:11:45.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:45 smithi102 ceph-mon[29355]: pgmap v63: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-13T14:11:45.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:45 smithi102 ceph-mon[40543]: mds.? [v2:172.21.15.195:6832/2017468914,v1:172.21.15.195:6833/2017468914] up:standby 2024-09-13T14:11:45.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:45 smithi102 ceph-mon[40543]: fsmap foo:1 {0=foo.smithi102.njetxr=up:active} 1 up:standby 2024-09-13T14:11:45.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:45 smithi102 ceph-mon[40543]: pgmap v63: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-13T14:11:45.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:45 smithi195 ceph-mon[33205]: mds.? [v2:172.21.15.195:6832/2017468914,v1:172.21.15.195:6833/2017468914] up:standby 2024-09-13T14:11:45.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:45 smithi195 ceph-mon[33205]: fsmap foo:1 {0=foo.smithi102.njetxr=up:active} 1 up:standby 2024-09-13T14:11:45.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:45 smithi195 ceph-mon[33205]: pgmap v63: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-13T14:11:47.446 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:11:47.446 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":71,"fsid":"08cee502-71d9-11ef-bceb-c7b262605968","created":"2024-09-13T14:04:48.179879+0000","modified":"2024-09-13T14:11:31.240493+0000","last_up_change":"2024-09-13T14:09:36.412493+0000","last_in_change":"2024-09-13T14:09:19.207877+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T14:07:41.927240+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T14:10:10.981300+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":65,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-13T14:11:27.935818+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-13T14:11:28.454262+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b43b49e0-1cf0-47b8-aa85-37108f321220","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":466204285},{"type":"v1","addr":"172.21.15.102:6803","nonce":466204285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":466204285},{"type":"v1","addr":"172.21.15.102:6805","nonce":466204285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":466204285},{"type":"v1","addr":"172.21.15.102:6809","nonce":466204285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":466204285},{"type":"v1","addr":"172.21.15.102:6807","nonce":466204285}]},"public_addr":"172.21.15.102:6803/466204285","cluster_addr":"172.21.15.102:6805/466204285","heartbeat_back_addr":"172.21.15.102:6809/466204285","heartbeat_front_addr":"172.21.15.102:6807/466204285","state":["exists","up"]},{"osd":1,"uuid":"27e10d5e-c471-4132-9c73-83c5708c3b08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":1293422793},{"type":"v1","addr":"172.21.15.102:6811","nonce":1293422793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":1293422793},{"type":"v1","addr":"172.21.15.102:6813","nonce":1293422793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1293422793},{"type":"v1","addr":"172.21.15.102:6817","nonce":1293422793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":1293422793},{"type":"v1","addr":"172.21.15.102:6815","nonce":1293422793}]},"public_addr":"172.21.15.102:6811/1293422793","cluster_addr":"172.21.15.102:6813/1293422793","heartbeat_back_addr":"172.21.15.102:6817/1293422793","heartbeat_front_addr":"172.21.15.102:6815/1293422793","state":["exists","up"]},{"osd":2,"uuid":"40a94b2b-2c24-4164-91be-0816a1acf3d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":360502781},{"type":"v1","addr":"172.21.15.102:6819","nonce":360502781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":360502781},{"type":"v1","addr":"172.21.15.102:6821","nonce":360502781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":360502781},{"type":"v1","addr":"172.21.15.102:6825","nonce":360502781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":360502781},{"type":"v1","addr":"172.21.15.102:6823","nonce":360502781}]},"public_addr":"172.21.15.102:6819/360502781","cluster_addr":"172.21.15.102:6821/360502781","heartbeat_back_addr":"172.21.15.102:6825/360502781","heartbeat_front_addr":"172.21.15.102:6823/360502781","state":["exists","up"]},{"osd":3,"uuid":"2ed85faf-6610-4bcc-a09d-4a6f93f5c6a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":3709228546},{"type":"v1","addr":"172.21.15.102:6827","nonce":3709228546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":3709228546},{"type":"v1","addr":"172.21.15.102:6829","nonce":3709228546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3709228546},{"type":"v1","addr":"172.21.15.102:6833","nonce":3709228546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":3709228546},{"type":"v1","addr":"172.21.15.102:6831","nonce":3709228546}]},"public_addr":"172.21.15.102:6827/3709228546","cluster_addr":"172.21.15.102:6829/3709228546","heartbeat_back_addr":"172.21.15.102:6833/3709228546","heartbeat_front_addr":"172.21.15.102:6831/3709228546","state":["exists","up"]},{"osd":4,"uuid":"8a45f350-fb3e-4049-bcc4-532d3e20c238","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":2821651815},{"type":"v1","addr":"172.21.15.195:6801","nonce":2821651815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":2821651815},{"type":"v1","addr":"172.21.15.195:6803","nonce":2821651815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":2821651815},{"type":"v1","addr":"172.21.15.195:6807","nonce":2821651815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":2821651815},{"type":"v1","addr":"172.21.15.195:6805","nonce":2821651815}]},"public_addr":"172.21.15.195:6801/2821651815","cluster_addr":"172.21.15.195:6803/2821651815","heartbeat_back_addr":"172.21.15.195:6807/2821651815","heartbeat_front_addr":"172.21.15.195:6805/2821651815","state":["exists","up"]},{"osd":5,"uuid":"363cf638-db26-4c86-b2be-e075fd09ac16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":3074139237},{"type":"v1","addr":"172.21.15.195:6809","nonce":3074139237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":3074139237},{"type":"v1","addr":"172.21.15.195:6811","nonce":3074139237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":3074139237},{"type":"v1","addr":"172.21.15.195:6815","nonce":3074139237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":3074139237},{"type":"v1","addr":"172.21.15.195:6813","nonce":3074139237}]},"public_addr":"172.21.15.195:6809/3074139237","cluster_addr":"172.21.15.195:6811/3074139237","heartbeat_back_addr":"172.21.15.195:6815/3074139237","heartbeat_front_addr":"172.21.15.195:6813/3074139237","state":["exists","up"]},{"osd":6,"uuid":"f37416e8-9c25-4860-a32f-0d1c404a8bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":4026627594},{"type":"v1","addr":"172.21.15.195:6817","nonce":4026627594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":4026627594},{"type":"v1","addr":"172.21.15.195:6819","nonce":4026627594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":4026627594},{"type":"v1","addr":"172.21.15.195:6823","nonce":4026627594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":4026627594},{"type":"v1","addr":"172.21.15.195:6821","nonce":4026627594}]},"public_addr":"172.21.15.195:6817/4026627594","cluster_addr":"172.21.15.195:6819/4026627594","heartbeat_back_addr":"172.21.15.195:6823/4026627594","heartbeat_front_addr":"172.21.15.195:6821/4026627594","state":["exists","up"]},{"osd":7,"uuid":"9a384a4d-fddb-4fb3-843a-77c19cca0368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2059403787},{"type":"v1","addr":"172.21.15.195:6825","nonce":2059403787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2059403787},{"type":"v1","addr":"172.21.15.195:6827","nonce":2059403787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2059403787},{"type":"v1","addr":"172.21.15.195:6831","nonce":2059403787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2059403787},{"type":"v1","addr":"172.21.15.195:6829","nonce":2059403787}]},"public_addr":"172.21.15.195:6825/2059403787","cluster_addr":"172.21.15.195:6827/2059403787","heartbeat_back_addr":"172.21.15.195:6831/2059403787","heartbeat_front_addr":"172.21.15.195:6829/2059403787","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:06:53.502746+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:07:16.251669+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:07:38.848272+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:08:02.013035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:08:23.043482+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:08:45.653159+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:09:10.187018+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:09:34.393753+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/1765904351":"2024-09-14T14:10:09.037657+0000","172.21.15.102:0/2416257718":"2024-09-14T14:10:09.037657+0000","172.21.15.102:0/1582001159":"2024-09-14T14:05:15.286479+0000","172.21.15.102:0/2308924552":"2024-09-14T14:10:09.037657+0000","172.21.15.102:6800/2761043787":"2024-09-14T14:05:15.286479+0000","172.21.15.102:6801/2761043787":"2024-09-14T14:05:15.286479+0000","172.21.15.102:0/2444394840":"2024-09-14T14:05:37.634624+0000","172.21.15.102:0/1574792592":"2024-09-14T14:05:15.286479+0000","172.21.15.102:0/3189088157":"2024-09-14T14:05:15.286479+0000","172.21.15.102:6800/3480998245":"2024-09-14T14:05:37.634624+0000","172.21.15.102:6801/3480998245":"2024-09-14T14:05:37.634624+0000","172.21.15.102:0/1160237252":"2024-09-14T14:10:09.037657+0000","172.21.15.102:0/3399546099":"2024-09-14T14:05:37.634624+0000","172.21.15.102:0/1939570054":"2024-09-14T14:05:37.634624+0000","172.21.15.102:6800/63514412":"2024-09-14T14:10:09.037657+0000","172.21.15.102:6801/63514412":"2024-09-14T14:10:09.037657+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T14:11:47.447 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:11:47 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:11:47] "GET /metrics HTTP/1.1" 200 33425 "" "Prometheus/2.43.0" 2024-09-13T14:11:47.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:47 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:47.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:47 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:47.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:47 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:11:47.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:47 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:11:47.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:47 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:47.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:47 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:11:47.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:47 smithi195 ceph-mon[33205]: pgmap v64: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-13T14:11:47.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:47 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/3167638371' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:11:48.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:47 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:48.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:47 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:48.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:47 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:11:48.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:47 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:11:48.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:47 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:48.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:47 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:11:48.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:47 smithi102 ceph-mon[29355]: pgmap v64: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-13T14:11:48.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:47 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/3167638371' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:11:48.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:47 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:48.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:47 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:48.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:47 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:11:48.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:47 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:11:48.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:47 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:48.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:47 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:11:48.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:47 smithi102 ceph-mon[40543]: pgmap v64: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-13T14:11:48.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:47 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/3167638371' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:11:48.380 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-13T14:11:48.381 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-13T14:11:48.381 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-13T14:11:48.381 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-13T14:11:48.382 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-13T14:11:48.382 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-13T14:11:48.383 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-13T14:11:48.383 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-13T14:11:49.053 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:11:49.053 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:11:49.116 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:11:49.118 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:11:49.360 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:11:49.366 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:11:50.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:50 smithi102 ceph-mon[40543]: pgmap v65: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-13T14:11:50.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:50 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:50.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:50 smithi102 ceph-mon[29355]: pgmap v65: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-13T14:11:50.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:50 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:50.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:50 smithi195 ceph-mon[33205]: pgmap v65: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-13T14:11:50.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:50 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:51.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:51 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:51.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:51 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:51.411 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:51 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:51.411 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:51 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:51.411 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:51 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:51.411 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:51 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:11:52.490 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:52 smithi102 ceph-mon[29355]: pgmap v66: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-13T14:11:52.490 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:52 smithi102 ceph-mon[40543]: pgmap v66: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-13T14:11:52.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:52 smithi195 ceph-mon[33205]: pgmap v66: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-13T14:11:54.375 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:54 smithi102 ceph-mon[29355]: pgmap v67: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-13T14:11:54.376 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:54 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:11:54.376 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:54 smithi102 ceph-mon[40543]: pgmap v67: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-13T14:11:54.376 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:54 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:11:54.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:54 smithi195 ceph-mon[33205]: pgmap v67: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-13T14:11:54.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:54 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:11:56.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:56 smithi102 ceph-mon[29355]: pgmap v68: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-13T14:11:56.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:56 smithi102 ceph-mon[40543]: pgmap v68: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-13T14:11:56.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:56 smithi195 ceph-mon[33205]: pgmap v68: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-13T14:11:56.712 INFO:teuthology.orchestra.run.smithi102.stdout:77309411381 2024-09-13T14:11:56.713 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-13T14:11:57.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:11:57 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:11:57] "GET /metrics HTTP/1.1" 200 33712 "" "Prometheus/2.43.0" 2024-09-13T14:11:57.921 INFO:teuthology.orchestra.run.smithi102.stdout:197568495652 2024-09-13T14:11:57.921 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-13T14:11:58.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:11:58 smithi195 ceph-mon[33205]: pgmap v69: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:58.564 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:11:58 smithi102 ceph-mon[29355]: pgmap v69: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:58.564 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:11:58 smithi102 ceph-mon[40543]: pgmap v69: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:11:59.056 INFO:teuthology.orchestra.run.smithi102.stdout:107374182449 2024-09-13T14:11:59.057 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-13T14:11:59.497 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:11:59.497 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:12:00.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:00 smithi102 ceph-mon[29355]: pgmap v70: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:00.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:00 smithi102 ceph-mon[40543]: pgmap v70: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:00.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:00 smithi195 ceph-mon[33205]: pgmap v70: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:02.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:02 smithi102 ceph-mon[29355]: pgmap v71: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:02.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:02 smithi102 ceph-mon[40543]: pgmap v71: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:02.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:02 smithi195 ceph-mon[33205]: pgmap v71: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:02.523 INFO:teuthology.orchestra.run.smithi102.stdout:128849018926 2024-09-13T14:12:02.524 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-13T14:12:04.269 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:12:04.392 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:04 smithi102 ceph-mon[29355]: pgmap v72: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:04.393 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:04 smithi102 ceph-mon[40543]: pgmap v72: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:04.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:04 smithi195 ceph-mon[33205]: pgmap v72: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:05.908 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:05 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:12:05.908 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:05 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:12:05.908 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:05 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:12:05.908 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:05 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:12:05.908 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:05 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:12:05.909 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:05 smithi102 ceph-mon[29355]: pgmap v73: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:05.909 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:05 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:12:05.909 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:05 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:12:05.909 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:05 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:12:05.909 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:05 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:12:05.909 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:05 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:12:05.909 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:05 smithi102 ceph-mon[40543]: pgmap v73: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:05.932 INFO:teuthology.orchestra.run.smithi102.stdout:231928234017 2024-09-13T14:12:05.932 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-13T14:12:05.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:05 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:12:05.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:05 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:12:05.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:05 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:12:05.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:05 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:12:05.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:05 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:12:05.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:05 smithi195 ceph-mon[33205]: pgmap v73: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:06.643 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:12:07.244 INFO:teuthology.orchestra.run.smithi102.stdout:163208757291 2024-09-13T14:12:07.244 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-13T14:12:07.483 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:12:07 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:12:07] "GET /metrics HTTP/1.1" 200 33712 "" "Prometheus/2.43.0" 2024-09-13T14:12:08.157 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:08 smithi102 ceph-mon[29355]: pgmap v74: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:08.157 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:08 smithi102 ceph-mon[40543]: pgmap v74: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:08.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:08 smithi195 ceph-mon[33205]: pgmap v74: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:08.951 INFO:teuthology.orchestra.run.smithi102.stdout:77309411384 2024-09-13T14:12:09.078 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:12:09.378 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:09 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/2313948528' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-13T14:12:09.378 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:09 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-09-13T14:12:09.378 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:09 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-09-13T14:12:09.378 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:09 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-09-13T14:12:09.378 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:09 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:12:09.378 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:09 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/2313948528' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-13T14:12:09.378 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:09 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-09-13T14:12:09.379 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:09 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-09-13T14:12:09.379 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:09 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-09-13T14:12:09.379 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:09 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:12:09.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:09 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/2313948528' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-13T14:12:09.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:09 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-09-13T14:12:09.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:09 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-09-13T14:12:09.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:09 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-09-13T14:12:09.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:09 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:12:09.791 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411381 got 77309411384 for osd.2 2024-09-13T14:12:09.791 DEBUG:teuthology.parallel:result is None 2024-09-13T14:12:10.337 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:12:10.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:10 smithi102 ceph-mon[29355]: pgmap v75: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:10.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:10 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-09-13T14:12:10.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:10 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-09-13T14:12:10.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:10 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-09-13T14:12:10.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:10 smithi102 ceph-mon[29355]: osdmap e72: 8 total, 8 up, 8 in 2024-09-13T14:12:10.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:10 smithi102 ceph-mon[40543]: pgmap v75: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:10.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:10 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-09-13T14:12:10.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:10 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-09-13T14:12:10.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:10 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-09-13T14:12:10.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:10 smithi102 ceph-mon[40543]: osdmap e72: 8 total, 8 up, 8 in 2024-09-13T14:12:10.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:10 smithi195 ceph-mon[33205]: pgmap v75: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:10.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:10 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-09-13T14:12:10.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:10 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-09-13T14:12:10.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:10 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-09-13T14:12:10.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:10 smithi195 ceph-mon[33205]: osdmap e72: 8 total, 8 up, 8 in 2024-09-13T14:12:10.782 INFO:teuthology.orchestra.run.smithi102.stdout:55834574909 2024-09-13T14:12:10.783 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-13T14:12:11.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:11 smithi102 ceph-mon[29355]: osdmap e73: 8 total, 8 up, 8 in 2024-09-13T14:12:11.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:11 smithi102 ceph-mon[40543]: osdmap e73: 8 total, 8 up, 8 in 2024-09-13T14:12:11.478 INFO:teuthology.orchestra.run.smithi102.stdout:34359738434 2024-09-13T14:12:11.478 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-13T14:12:11.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:11 smithi195 ceph-mon[33205]: osdmap e73: 8 total, 8 up, 8 in 2024-09-13T14:12:12.236 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:12 smithi102 ceph-mon[29355]: pgmap v78: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:12.237 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:12 smithi102 ceph-mon[40543]: pgmap v78: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:12.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:12 smithi195 ceph-mon[33205]: pgmap v78: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:12.526 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:12:14.369 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:14 smithi102 ceph-mon[29355]: pgmap v79: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:14.369 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:14 smithi102 ceph-mon[40543]: pgmap v79: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:14.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:14 smithi195 ceph-mon[33205]: pgmap v79: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:15.447 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:12:15.616 INFO:teuthology.orchestra.run.smithi102.stdout:197568495656 2024-09-13T14:12:16.153 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:12:16.155 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:12:16.271 INFO:teuthology.orchestra.run.smithi102.stdout:128849018929 2024-09-13T14:12:16.272 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:16 smithi102 ceph-mon[29355]: pgmap v80: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:16.272 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:16 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/2465692371' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-13T14:12:16.272 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:16 smithi102 ceph-mon[40543]: pgmap v80: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:16.272 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:16 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/2465692371' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-13T14:12:16.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:16 smithi195 ceph-mon[33205]: pgmap v80: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:16.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:16 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/2465692371' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-13T14:12:17.142 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495652 got 197568495656 for osd.6 2024-09-13T14:12:17.143 DEBUG:teuthology.parallel:result is None 2024-09-13T14:12:17.372 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:17 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/2729170807' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-13T14:12:17.372 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:17 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/2729170807' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-13T14:12:17.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:17 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/2729170807' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-13T14:12:17.663 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:12:17 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:12:17] "GET /metrics HTTP/1.1" 200 33711 "" "Prometheus/2.43.0" 2024-09-13T14:12:18.311 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:18 smithi102 ceph-mon[29355]: pgmap v81: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:18.311 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:18 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/3925488356' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-13T14:12:18.311 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:18 smithi102 ceph-mon[40543]: pgmap v81: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:18.311 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:18 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/3925488356' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-13T14:12:18.437 INFO:teuthology.orchestra.run.smithi102.stdout:107374182454 2024-09-13T14:12:18.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:18 smithi195 ceph-mon[33205]: pgmap v81: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:18.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:18 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/3925488356' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-13T14:12:19.075 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182449 got 107374182454 for osd.3 2024-09-13T14:12:19.075 DEBUG:teuthology.parallel:result is None 2024-09-13T14:12:20.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:20 smithi102 ceph-mon[40543]: pgmap v82: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:20.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:20 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/2837434621' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-13T14:12:20.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:20 smithi102 ceph-mon[29355]: pgmap v82: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:20.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:20 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/2837434621' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-13T14:12:20.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:20 smithi195 ceph-mon[33205]: pgmap v82: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:20.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:20 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/2837434621' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-13T14:12:20.932 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018926 got 128849018929 for osd.4 2024-09-13T14:12:20.933 DEBUG:teuthology.parallel:result is None 2024-09-13T14:12:22.184 INFO:teuthology.orchestra.run.smithi102.stdout:163208757293 2024-09-13T14:12:22.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:22 smithi102 ceph-mon[40543]: pgmap v83: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:22.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:22 smithi102 ceph-mon[29355]: pgmap v83: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:22.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:22 smithi195 ceph-mon[33205]: pgmap v83: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:23.555 INFO:teuthology.orchestra.run.smithi102.stdout:34359738437 2024-09-13T14:12:23.805 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757291 got 163208757293 for osd.5 2024-09-13T14:12:23.805 DEBUG:teuthology.parallel:result is None 2024-09-13T14:12:24.380 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:24 smithi102 ceph-mon[29355]: pgmap v84: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:24.380 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:24 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/3984425839' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-13T14:12:24.380 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:24 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:12:24.381 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:24 smithi102 ceph-mon[40543]: pgmap v84: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:24.381 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:24 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/3984425839' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-13T14:12:24.381 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:24 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:12:24.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:24 smithi195 ceph-mon[33205]: pgmap v84: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:24.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:24 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/3984425839' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-13T14:12:24.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:24 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:12:24.865 INFO:teuthology.orchestra.run.smithi102.stdout:55834574912 2024-09-13T14:12:25.079 INFO:teuthology.orchestra.run.smithi102.stdout:231928234021 2024-09-13T14:12:25.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:25 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/3314163285' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-13T14:12:25.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:25 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/601056001' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-13T14:12:25.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:25 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/3314163285' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-13T14:12:25.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:25 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/601056001' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-13T14:12:25.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:25 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/3314163285' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-13T14:12:25.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:25 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/601056001' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-13T14:12:25.761 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574909 got 55834574912 for osd.1 2024-09-13T14:12:25.762 DEBUG:teuthology.parallel:result is None 2024-09-13T14:12:26.173 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738434 got 34359738437 for osd.0 2024-09-13T14:12:26.173 DEBUG:teuthology.parallel:result is None 2024-09-13T14:12:26.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:26 smithi102 ceph-mon[29355]: pgmap v85: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:26.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:26 smithi102 ceph-mon[40543]: pgmap v85: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:26.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:26 smithi195 ceph-mon[33205]: pgmap v85: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:26.681 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234017 got 231928234021 for osd.7 2024-09-13T14:12:26.681 DEBUG:teuthology.parallel:result is None 2024-09-13T14:12:26.681 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-13T14:12:26.682 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:12:26.883 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:12:27.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:12:27 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:12:27] "GET /metrics HTTP/1.1" 200 33719 "" "Prometheus/2.43.0" 2024-09-13T14:12:28.330 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:28 smithi102 ceph-mon[29355]: pgmap v86: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:28.330 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:28 smithi102 ceph-mon[40543]: pgmap v86: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:28.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:28 smithi195 ceph-mon[33205]: pgmap v86: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:28.722 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:12:28.723 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:12:29.486 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":86,"stamp":"2024-09-13T14:12:27.079231+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592749,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":226,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":170,"ondisk_log_size":170,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":232124,"kb_used_data":7748,"kb_used_omap":12,"kb_used_meta":224243,"kb_avail":749729092,"statfs":{"total":767960285184,"available":767722590208,"internally_reserved":0,"allocated":7933952,"data_stored":4445598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":229625401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002344"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.138372+0000","last_change":"2024-09-13T14:11:30.265609+0000","last_active":"2024-09-13T14:12:09.138372+0000","last_peered":"2024-09-13T14:12:09.138372+0000","last_clean":"2024-09-13T14:12:09.138372+0000","last_became_active":"2024-09-13T14:11:30.264979+0000","last_became_peered":"2024-09-13T14:11:30.264979+0000","last_unstale":"2024-09-13T14:12:09.138372+0000","last_undegraded":"2024-09-13T14:12:09.138372+0000","last_fullsized":"2024-09-13T14:12:09.138372+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:41:13.293676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":17,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.216975+0000","last_change":"2024-09-13T14:11:29.239362+0000","last_active":"2024-09-13T14:12:09.216975+0000","last_peered":"2024-09-13T14:12:09.216975+0000","last_clean":"2024-09-13T14:12:09.216975+0000","last_became_active":"2024-09-13T14:11:29.239104+0000","last_became_peered":"2024-09-13T14:11:29.239104+0000","last_unstale":"2024-09-13T14:12:09.216975+0000","last_undegraded":"2024-09-13T14:12:09.216975+0000","last_fullsized":"2024-09-13T14:12:09.216975+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:58:21.541518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.138181+0000","last_change":"2024-09-13T14:11:30.257032+0000","last_active":"2024-09-13T14:12:09.138181+0000","last_peered":"2024-09-13T14:12:09.138181+0000","last_clean":"2024-09-13T14:12:09.138181+0000","last_became_active":"2024-09-13T14:11:30.256788+0000","last_became_peered":"2024-09-13T14:11:30.256788+0000","last_unstale":"2024-09-13T14:12:09.138181+0000","last_undegraded":"2024-09-13T14:12:09.138181+0000","last_fullsized":"2024-09-13T14:12:09.138181+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:44.640255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.135423+0000","last_change":"2024-09-13T14:11:29.243383+0000","last_active":"2024-09-13T14:12:09.135423+0000","last_peered":"2024-09-13T14:12:09.135423+0000","last_clean":"2024-09-13T14:12:09.135423+0000","last_became_active":"2024-09-13T14:11:29.243110+0000","last_became_peered":"2024-09-13T14:11:29.243110+0000","last_unstale":"2024-09-13T14:12:09.135423+0000","last_undegraded":"2024-09-13T14:12:09.135423+0000","last_fullsized":"2024-09-13T14:12:09.135423+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:22:07.804736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.135897+0000","last_change":"2024-09-13T14:11:30.259449+0000","last_active":"2024-09-13T14:12:09.135897+0000","last_peered":"2024-09-13T14:12:09.135897+0000","last_clean":"2024-09-13T14:12:09.135897+0000","last_became_active":"2024-09-13T14:11:30.259152+0000","last_became_peered":"2024-09-13T14:11:30.259152+0000","last_unstale":"2024-09-13T14:12:09.135897+0000","last_undegraded":"2024-09-13T14:12:09.135897+0000","last_fullsized":"2024-09-13T14:12:09.135897+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:27.277752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.216825+0000","last_change":"2024-09-13T14:11:29.244452+0000","last_active":"2024-09-13T14:12:09.216825+0000","last_peered":"2024-09-13T14:12:09.216825+0000","last_clean":"2024-09-13T14:12:09.216825+0000","last_became_active":"2024-09-13T14:11:29.244102+0000","last_became_peered":"2024-09-13T14:11:29.244102+0000","last_unstale":"2024-09-13T14:12:09.216825+0000","last_undegraded":"2024-09-13T14:12:09.216825+0000","last_fullsized":"2024-09-13T14:12:09.216825+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:13:19.165534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.138625+0000","last_change":"2024-09-13T14:11:30.264079+0000","last_active":"2024-09-13T14:12:09.138625+0000","last_peered":"2024-09-13T14:12:09.138625+0000","last_clean":"2024-09-13T14:12:09.138625+0000","last_became_active":"2024-09-13T14:11:30.263788+0000","last_became_peered":"2024-09-13T14:11:30.263788+0000","last_unstale":"2024-09-13T14:12:09.138625+0000","last_undegraded":"2024-09-13T14:12:09.138625+0000","last_fullsized":"2024-09-13T14:12:09.138625+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:59:50.910992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":23,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.216298+0000","last_change":"2024-09-13T14:11:29.238290+0000","last_active":"2024-09-13T14:12:09.216298+0000","last_peered":"2024-09-13T14:12:09.216298+0000","last_clean":"2024-09-13T14:12:09.216298+0000","last_became_active":"2024-09-13T14:11:29.238025+0000","last_became_peered":"2024-09-13T14:11:29.238025+0000","last_unstale":"2024-09-13T14:12:09.216298+0000","last_undegraded":"2024-09-13T14:12:09.216298+0000","last_fullsized":"2024-09-13T14:12:09.216298+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:28:41.499531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.217217+0000","last_change":"2024-09-13T14:11:29.246164+0000","last_active":"2024-09-13T14:12:09.217217+0000","last_peered":"2024-09-13T14:12:09.217217+0000","last_clean":"2024-09-13T14:12:09.217217+0000","last_became_active":"2024-09-13T14:11:29.245872+0000","last_became_peered":"2024-09-13T14:11:29.245872+0000","last_unstale":"2024-09-13T14:12:09.217217+0000","last_undegraded":"2024-09-13T14:12:09.217217+0000","last_fullsized":"2024-09-13T14:12:09.217217+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:43.848638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.137919+0000","last_change":"2024-09-13T14:11:30.254534+0000","last_active":"2024-09-13T14:12:09.137919+0000","last_peered":"2024-09-13T14:12:09.137919+0000","last_clean":"2024-09-13T14:12:09.137919+0000","last_became_active":"2024-09-13T14:11:30.253959+0000","last_became_peered":"2024-09-13T14:11:30.253959+0000","last_unstale":"2024-09-13T14:12:09.137919+0000","last_undegraded":"2024-09-13T14:12:09.137919+0000","last_fullsized":"2024-09-13T14:12:09.137919+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:09:54.025853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":17,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.217202+0000","last_change":"2024-09-13T14:11:29.237544+0000","last_active":"2024-09-13T14:12:09.217202+0000","last_peered":"2024-09-13T14:12:09.217202+0000","last_clean":"2024-09-13T14:12:09.217202+0000","last_became_active":"2024-09-13T14:11:29.237286+0000","last_became_peered":"2024-09-13T14:11:29.237286+0000","last_unstale":"2024-09-13T14:12:09.217202+0000","last_undegraded":"2024-09-13T14:12:09.217202+0000","last_fullsized":"2024-09-13T14:12:09.217202+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:58:41.036243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.135378+0000","last_change":"2024-09-13T14:11:30.258335+0000","last_active":"2024-09-13T14:12:09.135378+0000","last_peered":"2024-09-13T14:12:09.135378+0000","last_clean":"2024-09-13T14:12:09.135378+0000","last_became_active":"2024-09-13T14:11:30.258018+0000","last_became_peered":"2024-09-13T14:11:30.258018+0000","last_unstale":"2024-09-13T14:12:09.135378+0000","last_undegraded":"2024-09-13T14:12:09.135378+0000","last_fullsized":"2024-09-13T14:12:09.135378+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:29:53.058845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.217146+0000","last_change":"2024-09-13T14:11:29.232897+0000","last_active":"2024-09-13T14:12:09.217146+0000","last_peered":"2024-09-13T14:12:09.217146+0000","last_clean":"2024-09-13T14:12:09.217146+0000","last_became_active":"2024-09-13T14:11:29.232637+0000","last_became_peered":"2024-09-13T14:11:29.232637+0000","last_unstale":"2024-09-13T14:12:09.217146+0000","last_undegraded":"2024-09-13T14:12:09.217146+0000","last_fullsized":"2024-09-13T14:12:09.217146+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:31:05.856046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.216244+0000","last_change":"2024-09-13T14:11:30.262248+0000","last_active":"2024-09-13T14:12:09.216244+0000","last_peered":"2024-09-13T14:12:09.216244+0000","last_clean":"2024-09-13T14:12:09.216244+0000","last_became_active":"2024-09-13T14:11:30.261979+0000","last_became_peered":"2024-09-13T14:11:30.261979+0000","last_unstale":"2024-09-13T14:12:09.216244+0000","last_undegraded":"2024-09-13T14:12:09.216244+0000","last_fullsized":"2024-09-13T14:12:09.216244+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:08:17.918647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.138289+0000","last_change":"2024-09-13T14:11:29.230320+0000","last_active":"2024-09-13T14:12:09.138289+0000","last_peered":"2024-09-13T14:12:09.138289+0000","last_clean":"2024-09-13T14:12:09.138289+0000","last_became_active":"2024-09-13T14:11:29.230052+0000","last_became_peered":"2024-09-13T14:11:29.230052+0000","last_unstale":"2024-09-13T14:12:09.138289+0000","last_undegraded":"2024-09-13T14:12:09.138289+0000","last_fullsized":"2024-09-13T14:12:09.138289+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:55:00.373696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.135482+0000","last_change":"2024-09-13T14:11:30.252480+0000","last_active":"2024-09-13T14:12:09.135482+0000","last_peered":"2024-09-13T14:12:09.135482+0000","last_clean":"2024-09-13T14:12:09.135482+0000","last_became_active":"2024-09-13T14:11:30.251906+0000","last_became_peered":"2024-09-13T14:11:30.251906+0000","last_unstale":"2024-09-13T14:12:09.135482+0000","last_undegraded":"2024-09-13T14:12:09.135482+0000","last_fullsized":"2024-09-13T14:12:09.135482+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:33.996378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":17,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.138153+0000","last_change":"2024-09-13T14:11:29.241031+0000","last_active":"2024-09-13T14:12:09.138153+0000","last_peered":"2024-09-13T14:12:09.138153+0000","last_clean":"2024-09-13T14:12:09.138153+0000","last_became_active":"2024-09-13T14:11:29.240745+0000","last_became_peered":"2024-09-13T14:11:29.240745+0000","last_unstale":"2024-09-13T14:12:09.138153+0000","last_undegraded":"2024-09-13T14:12:09.138153+0000","last_fullsized":"2024-09-13T14:12:09.138153+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:51:07.762351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.135708+0000","last_change":"2024-09-13T14:11:30.258763+0000","last_active":"2024-09-13T14:12:09.135708+0000","last_peered":"2024-09-13T14:12:09.135708+0000","last_clean":"2024-09-13T14:12:09.135708+0000","last_became_active":"2024-09-13T14:11:30.258457+0000","last_became_peered":"2024-09-13T14:11:30.258457+0000","last_unstale":"2024-09-13T14:12:09.135708+0000","last_undegraded":"2024-09-13T14:12:09.135708+0000","last_fullsized":"2024-09-13T14:12:09.135708+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:46:11.993109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.216442+0000","last_change":"2024-09-13T14:11:29.243477+0000","last_active":"2024-09-13T14:12:09.216442+0000","last_peered":"2024-09-13T14:12:09.216442+0000","last_clean":"2024-09-13T14:12:09.216442+0000","last_became_active":"2024-09-13T14:11:29.243194+0000","last_became_peered":"2024-09-13T14:11:29.243194+0000","last_unstale":"2024-09-13T14:12:09.216442+0000","last_undegraded":"2024-09-13T14:12:09.216442+0000","last_fullsized":"2024-09-13T14:12:09.216442+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:21:56.794102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.135858+0000","last_change":"2024-09-13T14:11:30.252706+0000","last_active":"2024-09-13T14:12:09.135858+0000","last_peered":"2024-09-13T14:12:09.135858+0000","last_clean":"2024-09-13T14:12:09.135858+0000","last_became_active":"2024-09-13T14:11:30.252454+0000","last_became_peered":"2024-09-13T14:11:30.252454+0000","last_unstale":"2024-09-13T14:12:09.135858+0000","last_undegraded":"2024-09-13T14:12:09.135858+0000","last_fullsized":"2024-09-13T14:12:09.135858+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:54:08.612195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.135603+0000","last_change":"2024-09-13T14:11:29.240775+0000","last_active":"2024-09-13T14:12:09.135603+0000","last_peered":"2024-09-13T14:12:09.135603+0000","last_clean":"2024-09-13T14:12:09.135603+0000","last_became_active":"2024-09-13T14:11:29.240532+0000","last_became_peered":"2024-09-13T14:11:29.240532+0000","last_unstale":"2024-09-13T14:12:09.135603+0000","last_undegraded":"2024-09-13T14:12:09.135603+0000","last_fullsized":"2024-09-13T14:12:09.135603+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:19:24.828838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.136038+0000","last_change":"2024-09-13T14:11:30.258802+0000","last_active":"2024-09-13T14:12:09.136038+0000","last_peered":"2024-09-13T14:12:09.136038+0000","last_clean":"2024-09-13T14:12:09.136038+0000","last_became_active":"2024-09-13T14:11:30.258508+0000","last_became_peered":"2024-09-13T14:11:30.258508+0000","last_unstale":"2024-09-13T14:12:09.136038+0000","last_undegraded":"2024-09-13T14:12:09.136038+0000","last_fullsized":"2024-09-13T14:12:09.136038+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:06:22.071716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.138003+0000","last_change":"2024-09-13T14:11:29.235406+0000","last_active":"2024-09-13T14:12:09.138003+0000","last_peered":"2024-09-13T14:12:09.138003+0000","last_clean":"2024-09-13T14:12:09.138003+0000","last_became_active":"2024-09-13T14:11:29.235132+0000","last_became_peered":"2024-09-13T14:11:29.235132+0000","last_unstale":"2024-09-13T14:12:09.138003+0000","last_undegraded":"2024-09-13T14:12:09.138003+0000","last_fullsized":"2024-09-13T14:12:09.138003+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:21:21.035202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T14:12:10.460243+0000","last_change":"2024-09-13T14:12:10.460243+0000","last_active":"2024-09-13T14:12:10.460243+0000","last_peered":"2024-09-13T14:12:10.460243+0000","last_clean":"2024-09-13T14:12:10.460243+0000","last_became_active":"2024-09-13T14:12:10.459775+0000","last_became_peered":"2024-09-13T14:12:10.459775+0000","last_unstale":"2024-09-13T14:12:10.460243+0000","last_undegraded":"2024-09-13T14:12:10.460243+0000","last_fullsized":"2024-09-13T14:12:10.460243+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:57:47.924314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"60'128","reported_seq":270,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.216711+0000","last_change":"2024-09-13T14:09:40.436318+0000","last_active":"2024-09-13T14:12:09.216711+0000","last_peered":"2024-09-13T14:12:09.216711+0000","last_clean":"2024-09-13T14:12:09.216711+0000","last_became_active":"2024-09-13T14:09:40.436016+0000","last_became_peered":"2024-09-13T14:09:40.436016+0000","last_unstale":"2024-09-13T14:12:09.216711+0000","last_undegraded":"2024-09-13T14:12:09.216711+0000","last_fullsized":"2024-09-13T14:12:09.216711+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:07:41.927740+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:16.214871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"63'1","reported_seq":29,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T14:12:10.139023+0000","last_change":"2024-09-13T14:10:16.904388+0000","last_active":"2024-09-13T14:12:10.139023+0000","last_peered":"2024-09-13T14:12:10.139023+0000","last_clean":"2024-09-13T14:12:10.139023+0000","last_became_active":"2024-09-13T14:10:12.122143+0000","last_became_peered":"2024-09-13T14:10:12.122143+0000","last_unstale":"2024-09-13T14:12:10.139023+0000","last_undegraded":"2024-09-13T14:12:10.139023+0000","last_fullsized":"2024-09-13T14:12:10.139023+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:11.434245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034973600000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.138338+0000","last_change":"2024-09-13T14:11:29.232107+0000","last_active":"2024-09-13T14:12:09.138338+0000","last_peered":"2024-09-13T14:12:09.138338+0000","last_clean":"2024-09-13T14:12:09.138338+0000","last_became_active":"2024-09-13T14:11:29.231842+0000","last_became_peered":"2024-09-13T14:11:29.231842+0000","last_unstale":"2024-09-13T14:12:09.138338+0000","last_undegraded":"2024-09-13T14:12:09.138338+0000","last_fullsized":"2024-09-13T14:12:09.138338+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:58:26.728667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.216574+0000","last_change":"2024-09-13T14:11:30.253749+0000","last_active":"2024-09-13T14:12:09.216574+0000","last_peered":"2024-09-13T14:12:09.216574+0000","last_clean":"2024-09-13T14:12:09.216574+0000","last_became_active":"2024-09-13T14:11:30.253499+0000","last_became_peered":"2024-09-13T14:11:30.253499+0000","last_unstale":"2024-09-13T14:12:09.216574+0000","last_undegraded":"2024-09-13T14:12:09.216574+0000","last_fullsized":"2024-09-13T14:12:09.216574+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:02:00.987335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":28,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T14:12:10.571224+0000","last_change":"2024-09-13T14:10:17.227534+0000","last_active":"2024-09-13T14:12:10.571224+0000","last_peered":"2024-09-13T14:12:10.571224+0000","last_clean":"2024-09-13T14:12:10.571224+0000","last_became_active":"2024-09-13T14:10:12.123842+0000","last_became_peered":"2024-09-13T14:10:12.123842+0000","last_unstale":"2024-09-13T14:12:10.571224+0000","last_undegraded":"2024-09-13T14:12:10.571224+0000","last_fullsized":"2024-09-13T14:12:10.571224+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:36:09.588482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031558699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.135339+0000","last_change":"2024-09-13T14:11:29.247139+0000","last_active":"2024-09-13T14:12:09.135339+0000","last_peered":"2024-09-13T14:12:09.135339+0000","last_clean":"2024-09-13T14:12:09.135339+0000","last_became_active":"2024-09-13T14:11:29.246049+0000","last_became_peered":"2024-09-13T14:11:29.246049+0000","last_unstale":"2024-09-13T14:12:09.135339+0000","last_undegraded":"2024-09-13T14:12:09.135339+0000","last_fullsized":"2024-09-13T14:12:09.135339+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:14:45.965962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.135802+0000","last_change":"2024-09-13T14:11:30.262905+0000","last_active":"2024-09-13T14:12:09.135802+0000","last_peered":"2024-09-13T14:12:09.135802+0000","last_clean":"2024-09-13T14:12:09.135802+0000","last_became_active":"2024-09-13T14:11:30.262608+0000","last_became_peered":"2024-09-13T14:11:30.262608+0000","last_unstale":"2024-09-13T14:12:09.135802+0000","last_undegraded":"2024-09-13T14:12:09.135802+0000","last_fullsized":"2024-09-13T14:12:09.135802+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:05:29.455679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":28,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T14:12:10.136743+0000","last_change":"2024-09-13T14:10:16.901072+0000","last_active":"2024-09-13T14:12:10.136743+0000","last_peered":"2024-09-13T14:12:10.136743+0000","last_clean":"2024-09-13T14:12:10.136743+0000","last_became_active":"2024-09-13T14:10:13.084740+0000","last_became_peered":"2024-09-13T14:10:13.084740+0000","last_unstale":"2024-09-13T14:12:10.136743+0000","last_undegraded":"2024-09-13T14:12:10.136743+0000","last_fullsized":"2024-09-13T14:12:10.136743+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:06:12.555142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028414100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.139326+0000","last_change":"2024-09-13T14:11:29.230135+0000","last_active":"2024-09-13T14:12:09.139326+0000","last_peered":"2024-09-13T14:12:09.139326+0000","last_clean":"2024-09-13T14:12:09.139326+0000","last_became_active":"2024-09-13T14:11:29.229468+0000","last_became_peered":"2024-09-13T14:11:29.229468+0000","last_unstale":"2024-09-13T14:12:09.139326+0000","last_undegraded":"2024-09-13T14:12:09.139326+0000","last_fullsized":"2024-09-13T14:12:09.139326+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:03:25.995900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.140300+0000","last_change":"2024-09-13T14:11:30.258451+0000","last_active":"2024-09-13T14:12:09.140300+0000","last_peered":"2024-09-13T14:12:09.140300+0000","last_clean":"2024-09-13T14:12:09.140300+0000","last_became_active":"2024-09-13T14:11:30.258149+0000","last_became_peered":"2024-09-13T14:11:30.258149+0000","last_unstale":"2024-09-13T14:12:09.140300+0000","last_undegraded":"2024-09-13T14:12:09.140300+0000","last_fullsized":"2024-09-13T14:12:09.140300+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:45:52.895402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":30,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T14:12:10.139162+0000","last_change":"2024-09-13T14:10:16.904932+0000","last_active":"2024-09-13T14:12:10.139162+0000","last_peered":"2024-09-13T14:12:10.139162+0000","last_clean":"2024-09-13T14:12:10.139162+0000","last_became_active":"2024-09-13T14:10:12.121802+0000","last_became_peered":"2024-09-13T14:10:12.121802+0000","last_unstale":"2024-09-13T14:12:10.139162+0000","last_undegraded":"2024-09-13T14:12:10.139162+0000","last_fullsized":"2024-09-13T14:12:10.139162+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:11:49.860408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088127500000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.135865+0000","last_change":"2024-09-13T14:11:29.245898+0000","last_active":"2024-09-13T14:12:09.135865+0000","last_peered":"2024-09-13T14:12:09.135865+0000","last_clean":"2024-09-13T14:12:09.135865+0000","last_became_active":"2024-09-13T14:11:29.245610+0000","last_became_peered":"2024-09-13T14:11:29.245610+0000","last_unstale":"2024-09-13T14:12:09.135865+0000","last_undegraded":"2024-09-13T14:12:09.135865+0000","last_fullsized":"2024-09-13T14:12:09.135865+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:05:59.228651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T14:12:10.140569+0000","last_change":"2024-09-13T14:12:10.140569+0000","last_active":"2024-09-13T14:12:10.140569+0000","last_peered":"2024-09-13T14:12:10.140569+0000","last_clean":"2024-09-13T14:12:10.140569+0000","last_became_active":"2024-09-13T14:12:10.140244+0000","last_became_peered":"2024-09-13T14:12:10.140244+0000","last_unstale":"2024-09-13T14:12:10.140569+0000","last_undegraded":"2024-09-13T14:12:10.140569+0000","last_fullsized":"2024-09-13T14:12:10.140569+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:42:52.706369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.138278+0000","last_change":"2024-09-13T14:11:29.231269+0000","last_active":"2024-09-13T14:12:09.138278+0000","last_peered":"2024-09-13T14:12:09.138278+0000","last_clean":"2024-09-13T14:12:09.138278+0000","last_became_active":"2024-09-13T14:11:29.230745+0000","last_became_peered":"2024-09-13T14:11:29.230745+0000","last_unstale":"2024-09-13T14:12:09.138278+0000","last_undegraded":"2024-09-13T14:12:09.138278+0000","last_fullsized":"2024-09-13T14:12:09.138278+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:56:34.412003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":28,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T14:12:10.137183+0000","last_change":"2024-09-13T14:10:16.905490+0000","last_active":"2024-09-13T14:12:10.137183+0000","last_peered":"2024-09-13T14:12:10.137183+0000","last_clean":"2024-09-13T14:12:10.137183+0000","last_became_active":"2024-09-13T14:10:12.122768+0000","last_became_peered":"2024-09-13T14:10:12.122768+0000","last_unstale":"2024-09-13T14:12:10.137183+0000","last_undegraded":"2024-09-13T14:12:10.137183+0000","last_fullsized":"2024-09-13T14:12:10.137183+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:55:21.633350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026843400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.139783+0000","last_change":"2024-09-13T14:11:30.259951+0000","last_active":"2024-09-13T14:12:09.139783+0000","last_peered":"2024-09-13T14:12:09.139783+0000","last_clean":"2024-09-13T14:12:09.139783+0000","last_became_active":"2024-09-13T14:11:30.259669+0000","last_became_peered":"2024-09-13T14:11:30.259669+0000","last_unstale":"2024-09-13T14:12:09.139783+0000","last_undegraded":"2024-09-13T14:12:09.139783+0000","last_fullsized":"2024-09-13T14:12:09.139783+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:16:15.483951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.139724+0000","last_change":"2024-09-13T14:11:29.227786+0000","last_active":"2024-09-13T14:12:09.139724+0000","last_peered":"2024-09-13T14:12:09.139724+0000","last_clean":"2024-09-13T14:12:09.139724+0000","last_became_active":"2024-09-13T14:11:29.227448+0000","last_became_peered":"2024-09-13T14:11:29.227448+0000","last_unstale":"2024-09-13T14:12:09.139724+0000","last_undegraded":"2024-09-13T14:12:09.139724+0000","last_fullsized":"2024-09-13T14:12:09.139724+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:51:21.426595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":28,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T14:12:10.571765+0000","last_change":"2024-09-13T14:10:17.227584+0000","last_active":"2024-09-13T14:12:10.571765+0000","last_peered":"2024-09-13T14:12:10.571765+0000","last_clean":"2024-09-13T14:12:10.571765+0000","last_became_active":"2024-09-13T14:10:12.878424+0000","last_became_peered":"2024-09-13T14:10:12.878424+0000","last_unstale":"2024-09-13T14:12:10.571765+0000","last_undegraded":"2024-09-13T14:12:10.571765+0000","last_fullsized":"2024-09-13T14:12:10.571765+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:00:10.824840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032561200000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.135514+0000","last_change":"2024-09-13T14:11:30.258255+0000","last_active":"2024-09-13T14:12:09.135514+0000","last_peered":"2024-09-13T14:12:09.135514+0000","last_clean":"2024-09-13T14:12:09.135514+0000","last_became_active":"2024-09-13T14:11:30.257957+0000","last_became_peered":"2024-09-13T14:11:30.257957+0000","last_unstale":"2024-09-13T14:12:09.135514+0000","last_undegraded":"2024-09-13T14:12:09.135514+0000","last_fullsized":"2024-09-13T14:12:09.135514+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:56:52.739848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":17,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.138513+0000","last_change":"2024-09-13T14:11:29.245356+0000","last_active":"2024-09-13T14:12:09.138513+0000","last_peered":"2024-09-13T14:12:09.138513+0000","last_clean":"2024-09-13T14:12:09.138513+0000","last_became_active":"2024-09-13T14:11:29.245093+0000","last_became_peered":"2024-09-13T14:11:29.245093+0000","last_unstale":"2024-09-13T14:12:09.138513+0000","last_undegraded":"2024-09-13T14:12:09.138513+0000","last_fullsized":"2024-09-13T14:12:09.138513+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:37.819148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":28,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T14:12:10.137274+0000","last_change":"2024-09-13T14:10:16.905444+0000","last_active":"2024-09-13T14:12:10.137274+0000","last_peered":"2024-09-13T14:12:10.137274+0000","last_clean":"2024-09-13T14:12:10.137274+0000","last_became_active":"2024-09-13T14:10:12.878714+0000","last_became_peered":"2024-09-13T14:10:12.878714+0000","last_unstale":"2024-09-13T14:12:10.137274+0000","last_undegraded":"2024-09-13T14:12:10.137274+0000","last_fullsized":"2024-09-13T14:12:10.137274+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:23:02.772611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000229423,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.138569+0000","last_change":"2024-09-13T14:11:30.266038+0000","last_active":"2024-09-13T14:12:09.138569+0000","last_peered":"2024-09-13T14:12:09.138569+0000","last_clean":"2024-09-13T14:12:09.138569+0000","last_became_active":"2024-09-13T14:11:30.265742+0000","last_became_peered":"2024-09-13T14:11:30.265742+0000","last_unstale":"2024-09-13T14:12:09.138569+0000","last_undegraded":"2024-09-13T14:12:09.138569+0000","last_fullsized":"2024-09-13T14:12:09.138569+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:58:01.809837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.135929+0000","last_change":"2024-09-13T14:11:29.248383+0000","last_active":"2024-09-13T14:12:09.135929+0000","last_peered":"2024-09-13T14:12:09.135929+0000","last_clean":"2024-09-13T14:12:09.135929+0000","last_became_active":"2024-09-13T14:11:29.247999+0000","last_became_peered":"2024-09-13T14:11:29.247999+0000","last_unstale":"2024-09-13T14:12:09.135929+0000","last_undegraded":"2024-09-13T14:12:09.135929+0000","last_fullsized":"2024-09-13T14:12:09.135929+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:20:21.763210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":28,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T14:12:10.137607+0000","last_change":"2024-09-13T14:10:16.905852+0000","last_active":"2024-09-13T14:12:10.137607+0000","last_peered":"2024-09-13T14:12:10.137607+0000","last_clean":"2024-09-13T14:12:10.137607+0000","last_became_active":"2024-09-13T14:10:12.122815+0000","last_became_peered":"2024-09-13T14:10:12.122815+0000","last_unstale":"2024-09-13T14:12:10.137607+0000","last_undegraded":"2024-09-13T14:12:10.137607+0000","last_fullsized":"2024-09-13T14:12:10.137607+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:59:17.830936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034784200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.135490+0000","last_change":"2024-09-13T14:11:30.256034+0000","last_active":"2024-09-13T14:12:09.135490+0000","last_peered":"2024-09-13T14:12:09.135490+0000","last_clean":"2024-09-13T14:12:09.135490+0000","last_became_active":"2024-09-13T14:11:30.255727+0000","last_became_peered":"2024-09-13T14:11:30.255727+0000","last_unstale":"2024-09-13T14:12:09.135490+0000","last_undegraded":"2024-09-13T14:12:09.135490+0000","last_fullsized":"2024-09-13T14:12:09.135490+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:21:35.817014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.135716+0000","last_change":"2024-09-13T14:11:30.253539+0000","last_active":"2024-09-13T14:12:09.135716+0000","last_peered":"2024-09-13T14:12:09.135716+0000","last_clean":"2024-09-13T14:12:09.135716+0000","last_became_active":"2024-09-13T14:11:30.253282+0000","last_became_peered":"2024-09-13T14:11:30.253282+0000","last_unstale":"2024-09-13T14:12:09.135716+0000","last_undegraded":"2024-09-13T14:12:09.135716+0000","last_fullsized":"2024-09-13T14:12:09.135716+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:29:58.721512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"71'6","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.135826+0000","last_change":"2024-09-13T14:11:29.237013+0000","last_active":"2024-09-13T14:12:09.135826+0000","last_peered":"2024-09-13T14:12:09.135826+0000","last_clean":"2024-09-13T14:12:09.135826+0000","last_became_active":"2024-09-13T14:11:29.236643+0000","last_became_peered":"2024-09-13T14:11:29.236643+0000","last_unstale":"2024-09-13T14:12:09.135826+0000","last_undegraded":"2024-09-13T14:12:09.135826+0000","last_fullsized":"2024-09-13T14:12:09.135826+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:33:58.438693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.139562+0000","last_change":"2024-09-13T14:11:30.257330+0000","last_active":"2024-09-13T14:12:09.139562+0000","last_peered":"2024-09-13T14:12:09.139562+0000","last_clean":"2024-09-13T14:12:09.139562+0000","last_became_active":"2024-09-13T14:11:30.257017+0000","last_became_peered":"2024-09-13T14:11:30.257017+0000","last_unstale":"2024-09-13T14:12:09.139562+0000","last_undegraded":"2024-09-13T14:12:09.139562+0000","last_fullsized":"2024-09-13T14:12:09.139562+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:43:55.648230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":18,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.216942+0000","last_change":"2024-09-13T14:11:29.237584+0000","last_active":"2024-09-13T14:12:09.216942+0000","last_peered":"2024-09-13T14:12:09.216942+0000","last_clean":"2024-09-13T14:12:09.216942+0000","last_became_active":"2024-09-13T14:11:29.237314+0000","last_became_peered":"2024-09-13T14:11:29.237314+0000","last_unstale":"2024-09-13T14:12:09.216942+0000","last_undegraded":"2024-09-13T14:12:09.216942+0000","last_fullsized":"2024-09-13T14:12:09.216942+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:50:37.627401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.138073+0000","last_change":"2024-09-13T14:11:29.229513+0000","last_active":"2024-09-13T14:12:09.138073+0000","last_peered":"2024-09-13T14:12:09.138073+0000","last_clean":"2024-09-13T14:12:09.138073+0000","last_became_active":"2024-09-13T14:11:29.228815+0000","last_became_peered":"2024-09-13T14:11:29.228815+0000","last_unstale":"2024-09-13T14:12:09.138073+0000","last_undegraded":"2024-09-13T14:12:09.138073+0000","last_fullsized":"2024-09-13T14:12:09.138073+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:43:23.900655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.137847+0000","last_change":"2024-09-13T14:11:30.265539+0000","last_active":"2024-09-13T14:12:09.137847+0000","last_peered":"2024-09-13T14:12:09.137847+0000","last_clean":"2024-09-13T14:12:09.137847+0000","last_became_active":"2024-09-13T14:11:30.264954+0000","last_became_peered":"2024-09-13T14:11:30.264954+0000","last_unstale":"2024-09-13T14:12:09.137847+0000","last_undegraded":"2024-09-13T14:12:09.137847+0000","last_fullsized":"2024-09-13T14:12:09.137847+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:27:39.218811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.135836+0000","last_change":"2024-09-13T14:11:29.247957+0000","last_active":"2024-09-13T14:12:09.135836+0000","last_peered":"2024-09-13T14:12:09.135836+0000","last_clean":"2024-09-13T14:12:09.135836+0000","last_became_active":"2024-09-13T14:11:29.247135+0000","last_became_peered":"2024-09-13T14:11:29.247135+0000","last_unstale":"2024-09-13T14:12:09.135836+0000","last_undegraded":"2024-09-13T14:12:09.135836+0000","last_fullsized":"2024-09-13T14:12:09.135836+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:42:34.941145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.137813+0000","last_change":"2024-09-13T14:11:30.264735+0000","last_active":"2024-09-13T14:12:09.137813+0000","last_peered":"2024-09-13T14:12:09.137813+0000","last_clean":"2024-09-13T14:12:09.137813+0000","last_became_active":"2024-09-13T14:11:30.264369+0000","last_became_peered":"2024-09-13T14:11:30.264369+0000","last_unstale":"2024-09-13T14:12:09.137813+0000","last_undegraded":"2024-09-13T14:12:09.137813+0000","last_fullsized":"2024-09-13T14:12:09.137813+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:32:11.058695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.216633+0000","last_change":"2024-09-13T14:11:30.260356+0000","last_active":"2024-09-13T14:12:09.216633+0000","last_peered":"2024-09-13T14:12:09.216633+0000","last_clean":"2024-09-13T14:12:09.216633+0000","last_became_active":"2024-09-13T14:11:30.260089+0000","last_became_peered":"2024-09-13T14:11:30.260089+0000","last_unstale":"2024-09-13T14:12:09.216633+0000","last_undegraded":"2024-09-13T14:12:09.216633+0000","last_fullsized":"2024-09-13T14:12:09.216633+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:21:42.191411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.137930+0000","last_change":"2024-09-13T14:11:29.239413+0000","last_active":"2024-09-13T14:12:09.137930+0000","last_peered":"2024-09-13T14:12:09.137930+0000","last_clean":"2024-09-13T14:12:09.137930+0000","last_became_active":"2024-09-13T14:11:29.239160+0000","last_became_peered":"2024-09-13T14:11:29.239160+0000","last_unstale":"2024-09-13T14:12:09.137930+0000","last_undegraded":"2024-09-13T14:12:09.137930+0000","last_fullsized":"2024-09-13T14:12:09.137930+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:33:31.411058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.135426+0000","last_change":"2024-09-13T14:11:30.260094+0000","last_active":"2024-09-13T14:12:09.135426+0000","last_peered":"2024-09-13T14:12:09.135426+0000","last_clean":"2024-09-13T14:12:09.135426+0000","last_became_active":"2024-09-13T14:11:30.259825+0000","last_became_peered":"2024-09-13T14:11:30.259825+0000","last_unstale":"2024-09-13T14:12:09.135426+0000","last_undegraded":"2024-09-13T14:12:09.135426+0000","last_fullsized":"2024-09-13T14:12:09.135426+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:13:03.900066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.139421+0000","last_change":"2024-09-13T14:11:29.245452+0000","last_active":"2024-09-13T14:12:09.139421+0000","last_peered":"2024-09-13T14:12:09.139421+0000","last_clean":"2024-09-13T14:12:09.139421+0000","last_became_active":"2024-09-13T14:11:29.245124+0000","last_became_peered":"2024-09-13T14:11:29.245124+0000","last_unstale":"2024-09-13T14:12:09.139421+0000","last_undegraded":"2024-09-13T14:12:09.139421+0000","last_fullsized":"2024-09-13T14:12:09.139421+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:17:21.475412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":23,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T14:12:10.460399+0000","last_change":"2024-09-13T14:12:10.460399+0000","last_active":"2024-09-13T14:12:10.460399+0000","last_peered":"2024-09-13T14:12:10.460399+0000","last_clean":"2024-09-13T14:12:10.460399+0000","last_became_active":"2024-09-13T14:12:10.460037+0000","last_became_peered":"2024-09-13T14:12:10.460037+0000","last_unstale":"2024-09-13T14:12:10.460399+0000","last_undegraded":"2024-09-13T14:12:10.460399+0000","last_fullsized":"2024-09-13T14:12:10.460399+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:52:47.992230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.135624+0000","last_change":"2024-09-13T14:11:29.241053+0000","last_active":"2024-09-13T14:12:09.135624+0000","last_peered":"2024-09-13T14:12:09.135624+0000","last_clean":"2024-09-13T14:12:09.135624+0000","last_became_active":"2024-09-13T14:11:29.240812+0000","last_became_peered":"2024-09-13T14:11:29.240812+0000","last_unstale":"2024-09-13T14:12:09.135624+0000","last_undegraded":"2024-09-13T14:12:09.135624+0000","last_fullsized":"2024-09-13T14:12:09.135624+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:07:48.614371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.135742+0000","last_change":"2024-09-13T14:11:29.436934+0000","last_active":"2024-09-13T14:12:09.135742+0000","last_peered":"2024-09-13T14:12:09.135742+0000","last_clean":"2024-09-13T14:12:09.135742+0000","last_became_active":"2024-09-13T14:11:29.436588+0000","last_became_peered":"2024-09-13T14:11:29.436588+0000","last_unstale":"2024-09-13T14:12:09.135742+0000","last_undegraded":"2024-09-13T14:12:09.135742+0000","last_fullsized":"2024-09-13T14:12:09.135742+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:30:49.599505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.135594+0000","last_change":"2024-09-13T14:11:29.247872+0000","last_active":"2024-09-13T14:12:09.135594+0000","last_peered":"2024-09-13T14:12:09.135594+0000","last_clean":"2024-09-13T14:12:09.135594+0000","last_became_active":"2024-09-13T14:11:29.247195+0000","last_became_peered":"2024-09-13T14:11:29.247195+0000","last_unstale":"2024-09-13T14:12:09.135594+0000","last_undegraded":"2024-09-13T14:12:09.135594+0000","last_fullsized":"2024-09-13T14:12:09.135594+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:27:53.785083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.135643+0000","last_change":"2024-09-13T14:11:30.260805+0000","last_active":"2024-09-13T14:12:09.135643+0000","last_peered":"2024-09-13T14:12:09.135643+0000","last_clean":"2024-09-13T14:12:09.135643+0000","last_became_active":"2024-09-13T14:11:30.259503+0000","last_became_peered":"2024-09-13T14:11:30.259503+0000","last_unstale":"2024-09-13T14:12:09.135643+0000","last_undegraded":"2024-09-13T14:12:09.135643+0000","last_fullsized":"2024-09-13T14:12:09.135643+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:59:42.757834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":17,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.138244+0000","last_change":"2024-09-13T14:11:29.240160+0000","last_active":"2024-09-13T14:12:09.138244+0000","last_peered":"2024-09-13T14:12:09.138244+0000","last_clean":"2024-09-13T14:12:09.138244+0000","last_became_active":"2024-09-13T14:11:29.239900+0000","last_became_peered":"2024-09-13T14:11:29.239900+0000","last_unstale":"2024-09-13T14:12:09.138244+0000","last_undegraded":"2024-09-13T14:12:09.138244+0000","last_fullsized":"2024-09-13T14:12:09.138244+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:12:35.557474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.135487+0000","last_change":"2024-09-13T14:11:30.259223+0000","last_active":"2024-09-13T14:12:09.135487+0000","last_peered":"2024-09-13T14:12:09.135487+0000","last_clean":"2024-09-13T14:12:09.135487+0000","last_became_active":"2024-09-13T14:11:30.258345+0000","last_became_peered":"2024-09-13T14:11:30.258345+0000","last_unstale":"2024-09-13T14:12:09.135487+0000","last_undegraded":"2024-09-13T14:12:09.135487+0000","last_fullsized":"2024-09-13T14:12:09.135487+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:06.225118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":17,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.138227+0000","last_change":"2024-09-13T14:11:29.240351+0000","last_active":"2024-09-13T14:12:09.138227+0000","last_peered":"2024-09-13T14:12:09.138227+0000","last_clean":"2024-09-13T14:12:09.138227+0000","last_became_active":"2024-09-13T14:11:29.240060+0000","last_became_peered":"2024-09-13T14:11:29.240060+0000","last_unstale":"2024-09-13T14:12:09.138227+0000","last_undegraded":"2024-09-13T14:12:09.138227+0000","last_fullsized":"2024-09-13T14:12:09.138227+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:34:36.126368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.137961+0000","last_change":"2024-09-13T14:11:30.263317+0000","last_active":"2024-09-13T14:12:09.137961+0000","last_peered":"2024-09-13T14:12:09.137961+0000","last_clean":"2024-09-13T14:12:09.137961+0000","last_became_active":"2024-09-13T14:11:30.263054+0000","last_became_peered":"2024-09-13T14:11:30.263054+0000","last_unstale":"2024-09-13T14:12:09.137961+0000","last_undegraded":"2024-09-13T14:12:09.137961+0000","last_fullsized":"2024-09-13T14:12:09.137961+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:04.263430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.138156+0000","last_change":"2024-09-13T14:11:29.232949+0000","last_active":"2024-09-13T14:12:09.138156+0000","last_peered":"2024-09-13T14:12:09.138156+0000","last_clean":"2024-09-13T14:12:09.138156+0000","last_became_active":"2024-09-13T14:11:29.232668+0000","last_became_peered":"2024-09-13T14:11:29.232668+0000","last_unstale":"2024-09-13T14:12:09.138156+0000","last_undegraded":"2024-09-13T14:12:09.138156+0000","last_fullsized":"2024-09-13T14:12:09.138156+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:35:06.935233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.135377+0000","last_change":"2024-09-13T14:11:30.252635+0000","last_active":"2024-09-13T14:12:09.135377+0000","last_peered":"2024-09-13T14:12:09.135377+0000","last_clean":"2024-09-13T14:12:09.135377+0000","last_became_active":"2024-09-13T14:11:30.252379+0000","last_became_peered":"2024-09-13T14:11:30.252379+0000","last_unstale":"2024-09-13T14:12:09.135377+0000","last_undegraded":"2024-09-13T14:12:09.135377+0000","last_fullsized":"2024-09-13T14:12:09.135377+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:33:09.887379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":19,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.135843+0000","last_change":"2024-09-13T14:11:29.247512+0000","last_active":"2024-09-13T14:12:09.135843+0000","last_peered":"2024-09-13T14:12:09.135843+0000","last_clean":"2024-09-13T14:12:09.135843+0000","last_became_active":"2024-09-13T14:11:29.247065+0000","last_became_peered":"2024-09-13T14:11:29.247065+0000","last_unstale":"2024-09-13T14:12:09.135843+0000","last_undegraded":"2024-09-13T14:12:09.135843+0000","last_fullsized":"2024-09-13T14:12:09.135843+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:17:27.667480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":7086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234022,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28364,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716788,"statfs":{"total":95995035648,"available":95965990912,"internally_reserved":0,"allocated":1376256,"data_stored":925421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:11:40 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.54300000000000004,"15min":0.54800000000000004},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.76800000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.502,"15min":0.496},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.753,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":1.0760000000000001}]},{"osd":1,"last update":"Fri Sep 13 14:11:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.50700000000000001,"15min":0.501},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.76200000000000001,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52800000000000002,"15min":0.52900000000000003},"min":{"1min":0.40999999999999998,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.77700000000000002,"5min":0.871,"15min":0.871},"last":0.60499999999999998}]},{"osd":2,"last update":"Fri Sep 13 14:11:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.52500000000000002,"15min":0.52100000000000002},"min":{"1min":0.38100000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.68999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.38400000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.71299999999999997,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.47199999999999998}]},{"osd":3,"last update":"Fri Sep 13 14:11:40 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.55500000000000005,"15min":0.55800000000000005},"min":{"1min":0.36799999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.72099999999999997,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.53600000000000003,"15min":0.53500000000000003},"min":{"1min":0.40999999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.73299999999999998,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.66000000000000003}]},{"osd":4,"last update":"Fri Sep 13 14:11:40 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.42899999999999999,"15min":0.42299999999999999},"min":{"1min":0.30099999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.73099999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.57899999999999996}]},{"osd":5,"last update":"Fri Sep 13 14:11:40 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.64100000000000001,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.32700000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.78800000000000003,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.67600000000000005}]},{"osd":6,"last update":"Fri Sep 13 14:11:40 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.52100000000000002,"15min":0.51900000000000002},"min":{"1min":0.42299999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.67700000000000005,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.33600000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.69399999999999995}]}]},{"osd":6,"up_from":46,"seq":197568495658,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32456,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712696,"statfs":{"total":95995035648,"available":95961800704,"internally_reserved":0,"allocated":1368064,"data_stored":925352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:12:22 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52400000000000002},"min":{"1min":0.38500000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.69099999999999995,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.52800000000000002,"15min":0.52600000000000002},"min":{"1min":0.27900000000000003,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.67200000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.627}]},{"osd":1,"last update":"Fri Sep 13 14:12:22 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.35299999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.67300000000000004,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53900000000000003,"15min":0.54100000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.67600000000000005,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.59099999999999997}]},{"osd":2,"last update":"Fri Sep 13 14:12:22 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.53900000000000003,"15min":0.54100000000000004},"min":{"1min":0.38100000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.71099999999999997,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53500000000000003,"15min":0.53800000000000003},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.64700000000000002,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.57999999999999996}]},{"osd":3,"last update":"Fri Sep 13 14:12:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.63800000000000001,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.54100000000000004,"15min":0.53500000000000003},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.70499999999999996,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.68200000000000005}]},{"osd":4,"last update":"Fri Sep 13 14:12:22 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.43099999999999999,"15min":0.41099999999999998},"min":{"1min":0.34699999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.434,"15min":0.41399999999999998},"min":{"1min":0.33600000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.40999999999999998}]},{"osd":5,"last update":"Fri Sep 13 14:12:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.44400000000000001,"15min":0.42999999999999999},"min":{"1min":0.35099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.67400000000000004,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.47599999999999998,"15min":0.45400000000000001},"min":{"1min":0.41399999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.46200000000000002}]},{"osd":7,"last update":"Fri Sep 13 14:11:40 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.46700000000000003,"15min":0.46500000000000002},"min":{"1min":0.37,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.53200000000000003,"15min":0.52800000000000002},"min":{"1min":0.39100000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.69299999999999995,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.441}]}]},{"osd":5,"up_from":38,"seq":163208757295,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31868,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713284,"statfs":{"total":95995035648,"available":95962402816,"internally_reserved":0,"allocated":765952,"data_stored":333944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51900000000000002,"15min":0.52900000000000003},"min":{"1min":0.38100000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.623,"5min":1.216,"15min":1.216},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.49299999999999999,"15min":0.495},"min":{"1min":0.33200000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.63700000000000001,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.53000000000000003}]},{"osd":1,"last update":"Fri Sep 13 14:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.50700000000000001,"15min":0.50900000000000001},"min":{"1min":0.39700000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.63800000000000001,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.50800000000000001,"15min":0.52000000000000002},"min":{"1min":0.35699999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.59099999999999997,"5min":1.133,"15min":1.133},"last":0.51700000000000002}]},{"osd":2,"last update":"Fri Sep 13 14:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.52900000000000003,"15min":0.54100000000000004},"min":{"1min":0.38500000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.60199999999999998,"5min":1.026,"15min":1.026},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.51900000000000002,"15min":0.52300000000000002},"min":{"1min":0.35799999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.68600000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.629}]},{"osd":3,"last update":"Fri Sep 13 14:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.5,"15min":0.50700000000000001},"min":{"1min":0.39800000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.62,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.502,"5min":0.53200000000000003,"15min":0.54100000000000004},"min":{"1min":0.38,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.64400000000000002,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.46800000000000003}]},{"osd":4,"last update":"Fri Sep 13 14:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.40899999999999997,"15min":0.39900000000000002},"min":{"1min":0.313,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.73299999999999998,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.442,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.31,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65500000000000003,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.44600000000000001}]},{"osd":6,"last update":"Fri Sep 13 14:12:17 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.47299999999999998,"15min":0.47899999999999998},"min":{"1min":0.30599999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.77300000000000002,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.48499999999999999,"15min":0.47799999999999998},"min":{"1min":0.29299999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.69699999999999995,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.54400000000000004}]},{"osd":7,"last update":"Fri Sep 13 14:11:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.48899999999999999,"15min":0.48399999999999999},"min":{"1min":0.33100000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.70599999999999996,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.499,"15min":0.49199999999999999},"min":{"1min":0.36199999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.47599999999999998}]}]},{"osd":4,"up_from":30,"seq":128849018931,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27784,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717368,"statfs":{"total":95995035648,"available":95966584832,"internally_reserved":0,"allocated":778240,"data_stored":335041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:11:32 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.55300000000000005,"15min":0.58699999999999997},"min":{"1min":0.374,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.65600000000000003,"5min":1.258,"15min":1.258},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.502,"5min":0.54900000000000004,"15min":0.57499999999999996},"min":{"1min":0.36099999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.67400000000000004,"5min":0.997,"15min":0.997},"last":0.68999999999999995}]},{"osd":1,"last update":"Fri Sep 13 14:11:32 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.55300000000000005,"15min":0.58199999999999996},"min":{"1min":0.38500000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.71699999999999997,"5min":1.117,"15min":1.117},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55100000000000005,"15min":0.57399999999999995},"min":{"1min":0.41499999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.70399999999999996,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":0.63100000000000001}]},{"osd":2,"last update":"Fri Sep 13 14:11:32 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.54600000000000004,"15min":0.57399999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.64500000000000002,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.56100000000000005,"15min":0.58699999999999997},"min":{"1min":0.40600000000000003,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.73099999999999998,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.64500000000000002}]},{"osd":3,"last update":"Fri Sep 13 14:11:32 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.57599999999999996,"15min":0.59899999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.76100000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.56699999999999995,"15min":0.59499999999999997},"min":{"1min":0.42599999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.628,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.61799999999999999}]},{"osd":5,"last update":"Fri Sep 13 14:11:54 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.376,"15min":0.35999999999999999},"min":{"1min":0.30299999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.63900000000000001,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.49099999999999999,"15min":0.47299999999999998},"min":{"1min":0.39600000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.68999999999999995,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.60499999999999998}]},{"osd":6,"last update":"Fri Sep 13 14:12:19 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.44500000000000001,"15min":0.43099999999999999},"min":{"1min":0.36799999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.65400000000000003,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.42799999999999999,"15min":0.40999999999999998},"min":{"1min":0.30599999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.64100000000000001,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.40100000000000002}]},{"osd":7,"last update":"Fri Sep 13 14:11:37 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.47399999999999998,"15min":0.46800000000000003},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.45500000000000002,"15min":0.45200000000000001},"min":{"1min":0.34799999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.41999999999999998}]}]},{"osd":3,"up_from":25,"seq":107374182456,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27764,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717388,"statfs":{"total":95995035648,"available":95966605312,"internally_reserved":0,"allocated":761856,"data_stored":333694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:12:16 2024","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.34999999999999998,"15min":0.35299999999999998},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.505,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.375,"15min":0.38},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.56999999999999995,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.433}]},{"osd":1,"last update":"Fri Sep 13 14:12:16 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.38800000000000001,"15min":0.39400000000000002},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.54100000000000004,"5min":0.747,"15min":0.747},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.378,"15min":0.38400000000000001},"min":{"1min":0.26400000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.38500000000000001}]},{"osd":2,"last update":"Fri Sep 13 14:12:16 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.39700000000000002,"15min":0.39900000000000002},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.371,"5min":0.39500000000000002,"15min":0.39800000000000002},"min":{"1min":0.26200000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.504,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.50700000000000001}]},{"osd":4,"last update":"Fri Sep 13 14:11:35 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52300000000000002,"15min":0.51400000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.73599999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.53900000000000003,"15min":0.52600000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.55100000000000005}]},{"osd":5,"last update":"Fri Sep 13 14:11:51 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56699999999999995,"15min":0.56499999999999995},"min":{"1min":0.39500000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.75900000000000001,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.626},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.54100000000000004,"15min":0.52700000000000002},"min":{"1min":0.45000000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.57299999999999995}]},{"osd":6,"last update":"Fri Sep 13 14:12:18 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.60599999999999998,"15min":0.60399999999999998},"min":{"1min":0.433,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.84899999999999998,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60499999999999998,"15min":0.61299999999999999},"min":{"1min":0.42199999999999999,"5min":0.37,"15min":0.37},"max":{"1min":0.76200000000000001,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.64800000000000002}]},{"osd":7,"last update":"Fri Sep 13 14:11:41 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.59999999999999998,"15min":0.59599999999999997},"min":{"1min":0.45900000000000002,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.625,"5min":0.60599999999999998,"15min":0.60299999999999998},"min":{"1min":0.42399999999999999,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.72099999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411388,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27772,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717380,"statfs":{"total":95995035648,"available":95966597120,"internally_reserved":0,"allocated":765952,"data_stored":334220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:11:49 2024","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.23899999999999999,"5min":0.22,"15min":0.22},"max":{"1min":0.46400000000000002,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.38400000000000001,"15min":0.38600000000000001},"min":{"1min":0.27700000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.44800000000000001,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Sep 13 14:11:49 2024","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.36699999999999999,"15min":0.36499999999999999},"min":{"1min":0.24299999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.44500000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.40500000000000003,"15min":0.39600000000000002},"min":{"1min":0.26000000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.47999999999999998,"5min":0.745,"15min":0.745},"last":0.54600000000000004}]},{"osd":3,"last update":"Fri Sep 13 14:12:13 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.439,"15min":0.46899999999999997},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.501,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.41099999999999998,"15min":0.44600000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.46400000000000002,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.48299999999999998}]},{"osd":4,"last update":"Fri Sep 13 14:11:29 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.55900000000000005,"15min":0.57499999999999996},"min":{"1min":0.39400000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.72399999999999998,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.59199999999999997,"15min":0.61399999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.68500000000000005,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.57299999999999995}]},{"osd":5,"last update":"Fri Sep 13 14:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.621,"15min":0.64200000000000002},"min":{"1min":0.39000000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.89300000000000002,"5min":1.2,"15min":1.2},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57099999999999995,"15min":0.58199999999999996},"min":{"1min":0.41599999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.79000000000000004,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.70599999999999996}]},{"osd":6,"last update":"Fri Sep 13 14:12:13 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56299999999999994,"15min":0.55900000000000005},"min":{"1min":0.38700000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.57699999999999996,"15min":0.56799999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.65800000000000003}]},{"osd":7,"last update":"Fri Sep 13 14:11:41 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.54800000000000004,"15min":0.54100000000000004},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.626},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.81100000000000005,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.67000000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574913,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27764,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717388,"statfs":{"total":95995035648,"available":95966605312,"internally_reserved":0,"allocated":757760,"data_stored":333350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:12:24 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.34200000000000003,"15min":0.30599999999999999},"min":{"1min":0.222,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.40000000000000002,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.35299999999999998,"15min":0.32000000000000001},"min":{"1min":0.21199999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.49299999999999999,"5min":0.54600000000000004,"15min":0.54600000000000004},"last":0.372}]},{"osd":2,"last update":"Fri Sep 13 14:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.36199999999999999,"15min":0.34999999999999998},"min":{"1min":0.222,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.53600000000000003,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.35799999999999998,"15min":0.35299999999999998},"min":{"1min":0.24099999999999999,"5min":0.224,"15min":0.224},"max":{"1min":0.44800000000000001,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.40600000000000003}]},{"osd":3,"last update":"Fri Sep 13 14:12:10 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.38400000000000001,"15min":0.39700000000000002},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.47599999999999998,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.372,"5min":0.41599999999999998,"15min":0.42599999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.53000000000000003,"5min":0.621,"15min":0.621},"last":0.433}]},{"osd":4,"last update":"Fri Sep 13 14:11:35 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.55900000000000005,"15min":0.55400000000000005},"min":{"1min":0.35899999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.89800000000000002,"5min":1.133,"15min":1.133},"last":0.748},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.60199999999999998,"15min":0.623},"min":{"1min":0.36499999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.876,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.73899999999999999}]},{"osd":5,"last update":"Fri Sep 13 14:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.54900000000000004,"15min":0.54600000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.56599999999999995,"15min":0.56100000000000005},"min":{"1min":0.377,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.72699999999999998}]},{"osd":6,"last update":"Fri Sep 13 14:12:19 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.39000000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.70899999999999996,"5min":1.044,"15min":1.044},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.59899999999999998,"15min":0.60399999999999998},"min":{"1min":0.41299999999999998,"5min":0.379,"15min":0.379},"max":{"1min":0.71999999999999997,"5min":1.02,"15min":1.02},"last":0.68799999999999994}]},{"osd":7,"last update":"Fri Sep 13 14:11:41 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.56599999999999995,"15min":0.56299999999999994},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.57799999999999996,"15min":0.57599999999999996},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.63}]}]},{"osd":0,"up_from":8,"seq":34359738438,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28352,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716800,"statfs":{"total":95995035648,"available":95966003200,"internally_reserved":0,"allocated":1359872,"data_stored":924576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 14:11:28 2024","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.34599999999999997,"15min":0.33000000000000002},"min":{"1min":0.245,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.46400000000000002,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.34899999999999998,"15min":0.33200000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.496,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.35499999999999998}]},{"osd":2,"last update":"Fri Sep 13 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.374,"15min":0.374},"min":{"1min":0.23300000000000001,"5min":0.192,"15min":0.192},"max":{"1min":0.47499999999999998,"5min":0.54300000000000004,"15min":0.54300000000000004},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.379,"15min":0.39100000000000001},"min":{"1min":0.255,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.46400000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.315}]},{"osd":3,"last update":"Fri Sep 13 14:12:11 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.40699999999999997,"15min":0.42099999999999999},"min":{"1min":0.28999999999999998,"5min":0.224,"15min":0.224},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.45000000000000001,"15min":0.46200000000000002},"min":{"1min":0.30199999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.33100000000000002}]},{"osd":4,"last update":"Fri Sep 13 14:11:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.56100000000000005,"15min":0.57799999999999996},"min":{"1min":0.38900000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.72299999999999998,"5min":1.044,"15min":1.044},"last":0.501},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.56000000000000005,"15min":0.58099999999999996},"min":{"1min":0.35299999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.71099999999999997,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.55700000000000005}]},{"osd":5,"last update":"Fri Sep 13 14:11:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59699999999999998,"15min":0.61699999999999999},"min":{"1min":0.499,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.755,"5min":1.109,"15min":1.109},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58399999999999996,"15min":0.60699999999999998},"min":{"1min":0.41099999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.84599999999999997,"5min":1.151,"15min":1.151},"last":0.53200000000000003}]},{"osd":6,"last update":"Fri Sep 13 14:12:18 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.57899999999999996,"15min":0.57699999999999996},"min":{"1min":0.504,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.82799999999999996,"5min":0.879,"15min":0.879},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.59899999999999998,"15min":0.59299999999999997},"min":{"1min":0.496,"5min":0.379,"15min":0.379},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.56799999999999995}]},{"osd":7,"last update":"Fri Sep 13 14:11:43 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.53200000000000003,"15min":0.52600000000000002},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.56999999999999995,"15min":0.56599999999999995},"min":{"1min":0.42399999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.54200000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:12:29.491 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:12:29.691 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:12:30.398 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:30 smithi102 ceph-mon[40543]: from='client.24647 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:12:30.398 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:30 smithi102 ceph-mon[40543]: pgmap v87: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:30.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:30 smithi102 ceph-mon[29355]: from='client.24647 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:12:30.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:30 smithi102 ceph-mon[29355]: pgmap v87: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:30.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:30 smithi195 ceph-mon[33205]: from='client.24647 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:12:30.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:30 smithi195 ceph-mon[33205]: pgmap v87: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:31.590 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:12:31.592 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:12:32.316 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":88,"stamp":"2024-09-13T14:12:31.079841+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592749,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":226,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":170,"ondisk_log_size":170,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":232124,"kb_used_data":7748,"kb_used_omap":12,"kb_used_meta":224243,"kb_avail":749729092,"statfs":{"total":767960285184,"available":767722590208,"internally_reserved":0,"allocated":7933952,"data_stored":4445598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":229625401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002154"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.138372+0000","last_change":"2024-09-13T14:11:30.265609+0000","last_active":"2024-09-13T14:12:09.138372+0000","last_peered":"2024-09-13T14:12:09.138372+0000","last_clean":"2024-09-13T14:12:09.138372+0000","last_became_active":"2024-09-13T14:11:30.264979+0000","last_became_peered":"2024-09-13T14:11:30.264979+0000","last_unstale":"2024-09-13T14:12:09.138372+0000","last_undegraded":"2024-09-13T14:12:09.138372+0000","last_fullsized":"2024-09-13T14:12:09.138372+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:41:13.293676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":17,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.216975+0000","last_change":"2024-09-13T14:11:29.239362+0000","last_active":"2024-09-13T14:12:09.216975+0000","last_peered":"2024-09-13T14:12:09.216975+0000","last_clean":"2024-09-13T14:12:09.216975+0000","last_became_active":"2024-09-13T14:11:29.239104+0000","last_became_peered":"2024-09-13T14:11:29.239104+0000","last_unstale":"2024-09-13T14:12:09.216975+0000","last_undegraded":"2024-09-13T14:12:09.216975+0000","last_fullsized":"2024-09-13T14:12:09.216975+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:58:21.541518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.138181+0000","last_change":"2024-09-13T14:11:30.257032+0000","last_active":"2024-09-13T14:12:09.138181+0000","last_peered":"2024-09-13T14:12:09.138181+0000","last_clean":"2024-09-13T14:12:09.138181+0000","last_became_active":"2024-09-13T14:11:30.256788+0000","last_became_peered":"2024-09-13T14:11:30.256788+0000","last_unstale":"2024-09-13T14:12:09.138181+0000","last_undegraded":"2024-09-13T14:12:09.138181+0000","last_fullsized":"2024-09-13T14:12:09.138181+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:44.640255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.135423+0000","last_change":"2024-09-13T14:11:29.243383+0000","last_active":"2024-09-13T14:12:09.135423+0000","last_peered":"2024-09-13T14:12:09.135423+0000","last_clean":"2024-09-13T14:12:09.135423+0000","last_became_active":"2024-09-13T14:11:29.243110+0000","last_became_peered":"2024-09-13T14:11:29.243110+0000","last_unstale":"2024-09-13T14:12:09.135423+0000","last_undegraded":"2024-09-13T14:12:09.135423+0000","last_fullsized":"2024-09-13T14:12:09.135423+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:22:07.804736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.135897+0000","last_change":"2024-09-13T14:11:30.259449+0000","last_active":"2024-09-13T14:12:09.135897+0000","last_peered":"2024-09-13T14:12:09.135897+0000","last_clean":"2024-09-13T14:12:09.135897+0000","last_became_active":"2024-09-13T14:11:30.259152+0000","last_became_peered":"2024-09-13T14:11:30.259152+0000","last_unstale":"2024-09-13T14:12:09.135897+0000","last_undegraded":"2024-09-13T14:12:09.135897+0000","last_fullsized":"2024-09-13T14:12:09.135897+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:27.277752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.216825+0000","last_change":"2024-09-13T14:11:29.244452+0000","last_active":"2024-09-13T14:12:09.216825+0000","last_peered":"2024-09-13T14:12:09.216825+0000","last_clean":"2024-09-13T14:12:09.216825+0000","last_became_active":"2024-09-13T14:11:29.244102+0000","last_became_peered":"2024-09-13T14:11:29.244102+0000","last_unstale":"2024-09-13T14:12:09.216825+0000","last_undegraded":"2024-09-13T14:12:09.216825+0000","last_fullsized":"2024-09-13T14:12:09.216825+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:13:19.165534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.138625+0000","last_change":"2024-09-13T14:11:30.264079+0000","last_active":"2024-09-13T14:12:09.138625+0000","last_peered":"2024-09-13T14:12:09.138625+0000","last_clean":"2024-09-13T14:12:09.138625+0000","last_became_active":"2024-09-13T14:11:30.263788+0000","last_became_peered":"2024-09-13T14:11:30.263788+0000","last_unstale":"2024-09-13T14:12:09.138625+0000","last_undegraded":"2024-09-13T14:12:09.138625+0000","last_fullsized":"2024-09-13T14:12:09.138625+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:59:50.910992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":23,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.216298+0000","last_change":"2024-09-13T14:11:29.238290+0000","last_active":"2024-09-13T14:12:09.216298+0000","last_peered":"2024-09-13T14:12:09.216298+0000","last_clean":"2024-09-13T14:12:09.216298+0000","last_became_active":"2024-09-13T14:11:29.238025+0000","last_became_peered":"2024-09-13T14:11:29.238025+0000","last_unstale":"2024-09-13T14:12:09.216298+0000","last_undegraded":"2024-09-13T14:12:09.216298+0000","last_fullsized":"2024-09-13T14:12:09.216298+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:28:41.499531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.217217+0000","last_change":"2024-09-13T14:11:29.246164+0000","last_active":"2024-09-13T14:12:09.217217+0000","last_peered":"2024-09-13T14:12:09.217217+0000","last_clean":"2024-09-13T14:12:09.217217+0000","last_became_active":"2024-09-13T14:11:29.245872+0000","last_became_peered":"2024-09-13T14:11:29.245872+0000","last_unstale":"2024-09-13T14:12:09.217217+0000","last_undegraded":"2024-09-13T14:12:09.217217+0000","last_fullsized":"2024-09-13T14:12:09.217217+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:43.848638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.137919+0000","last_change":"2024-09-13T14:11:30.254534+0000","last_active":"2024-09-13T14:12:09.137919+0000","last_peered":"2024-09-13T14:12:09.137919+0000","last_clean":"2024-09-13T14:12:09.137919+0000","last_became_active":"2024-09-13T14:11:30.253959+0000","last_became_peered":"2024-09-13T14:11:30.253959+0000","last_unstale":"2024-09-13T14:12:09.137919+0000","last_undegraded":"2024-09-13T14:12:09.137919+0000","last_fullsized":"2024-09-13T14:12:09.137919+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:09:54.025853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":17,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.217202+0000","last_change":"2024-09-13T14:11:29.237544+0000","last_active":"2024-09-13T14:12:09.217202+0000","last_peered":"2024-09-13T14:12:09.217202+0000","last_clean":"2024-09-13T14:12:09.217202+0000","last_became_active":"2024-09-13T14:11:29.237286+0000","last_became_peered":"2024-09-13T14:11:29.237286+0000","last_unstale":"2024-09-13T14:12:09.217202+0000","last_undegraded":"2024-09-13T14:12:09.217202+0000","last_fullsized":"2024-09-13T14:12:09.217202+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:58:41.036243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.135378+0000","last_change":"2024-09-13T14:11:30.258335+0000","last_active":"2024-09-13T14:12:09.135378+0000","last_peered":"2024-09-13T14:12:09.135378+0000","last_clean":"2024-09-13T14:12:09.135378+0000","last_became_active":"2024-09-13T14:11:30.258018+0000","last_became_peered":"2024-09-13T14:11:30.258018+0000","last_unstale":"2024-09-13T14:12:09.135378+0000","last_undegraded":"2024-09-13T14:12:09.135378+0000","last_fullsized":"2024-09-13T14:12:09.135378+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:29:53.058845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.217146+0000","last_change":"2024-09-13T14:11:29.232897+0000","last_active":"2024-09-13T14:12:09.217146+0000","last_peered":"2024-09-13T14:12:09.217146+0000","last_clean":"2024-09-13T14:12:09.217146+0000","last_became_active":"2024-09-13T14:11:29.232637+0000","last_became_peered":"2024-09-13T14:11:29.232637+0000","last_unstale":"2024-09-13T14:12:09.217146+0000","last_undegraded":"2024-09-13T14:12:09.217146+0000","last_fullsized":"2024-09-13T14:12:09.217146+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:31:05.856046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.216244+0000","last_change":"2024-09-13T14:11:30.262248+0000","last_active":"2024-09-13T14:12:09.216244+0000","last_peered":"2024-09-13T14:12:09.216244+0000","last_clean":"2024-09-13T14:12:09.216244+0000","last_became_active":"2024-09-13T14:11:30.261979+0000","last_became_peered":"2024-09-13T14:11:30.261979+0000","last_unstale":"2024-09-13T14:12:09.216244+0000","last_undegraded":"2024-09-13T14:12:09.216244+0000","last_fullsized":"2024-09-13T14:12:09.216244+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:08:17.918647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.138289+0000","last_change":"2024-09-13T14:11:29.230320+0000","last_active":"2024-09-13T14:12:09.138289+0000","last_peered":"2024-09-13T14:12:09.138289+0000","last_clean":"2024-09-13T14:12:09.138289+0000","last_became_active":"2024-09-13T14:11:29.230052+0000","last_became_peered":"2024-09-13T14:11:29.230052+0000","last_unstale":"2024-09-13T14:12:09.138289+0000","last_undegraded":"2024-09-13T14:12:09.138289+0000","last_fullsized":"2024-09-13T14:12:09.138289+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:55:00.373696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.135482+0000","last_change":"2024-09-13T14:11:30.252480+0000","last_active":"2024-09-13T14:12:09.135482+0000","last_peered":"2024-09-13T14:12:09.135482+0000","last_clean":"2024-09-13T14:12:09.135482+0000","last_became_active":"2024-09-13T14:11:30.251906+0000","last_became_peered":"2024-09-13T14:11:30.251906+0000","last_unstale":"2024-09-13T14:12:09.135482+0000","last_undegraded":"2024-09-13T14:12:09.135482+0000","last_fullsized":"2024-09-13T14:12:09.135482+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:33.996378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":17,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.138153+0000","last_change":"2024-09-13T14:11:29.241031+0000","last_active":"2024-09-13T14:12:09.138153+0000","last_peered":"2024-09-13T14:12:09.138153+0000","last_clean":"2024-09-13T14:12:09.138153+0000","last_became_active":"2024-09-13T14:11:29.240745+0000","last_became_peered":"2024-09-13T14:11:29.240745+0000","last_unstale":"2024-09-13T14:12:09.138153+0000","last_undegraded":"2024-09-13T14:12:09.138153+0000","last_fullsized":"2024-09-13T14:12:09.138153+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:51:07.762351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.135708+0000","last_change":"2024-09-13T14:11:30.258763+0000","last_active":"2024-09-13T14:12:09.135708+0000","last_peered":"2024-09-13T14:12:09.135708+0000","last_clean":"2024-09-13T14:12:09.135708+0000","last_became_active":"2024-09-13T14:11:30.258457+0000","last_became_peered":"2024-09-13T14:11:30.258457+0000","last_unstale":"2024-09-13T14:12:09.135708+0000","last_undegraded":"2024-09-13T14:12:09.135708+0000","last_fullsized":"2024-09-13T14:12:09.135708+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:46:11.993109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.216442+0000","last_change":"2024-09-13T14:11:29.243477+0000","last_active":"2024-09-13T14:12:09.216442+0000","last_peered":"2024-09-13T14:12:09.216442+0000","last_clean":"2024-09-13T14:12:09.216442+0000","last_became_active":"2024-09-13T14:11:29.243194+0000","last_became_peered":"2024-09-13T14:11:29.243194+0000","last_unstale":"2024-09-13T14:12:09.216442+0000","last_undegraded":"2024-09-13T14:12:09.216442+0000","last_fullsized":"2024-09-13T14:12:09.216442+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:21:56.794102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.135858+0000","last_change":"2024-09-13T14:11:30.252706+0000","last_active":"2024-09-13T14:12:09.135858+0000","last_peered":"2024-09-13T14:12:09.135858+0000","last_clean":"2024-09-13T14:12:09.135858+0000","last_became_active":"2024-09-13T14:11:30.252454+0000","last_became_peered":"2024-09-13T14:11:30.252454+0000","last_unstale":"2024-09-13T14:12:09.135858+0000","last_undegraded":"2024-09-13T14:12:09.135858+0000","last_fullsized":"2024-09-13T14:12:09.135858+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:54:08.612195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.135603+0000","last_change":"2024-09-13T14:11:29.240775+0000","last_active":"2024-09-13T14:12:09.135603+0000","last_peered":"2024-09-13T14:12:09.135603+0000","last_clean":"2024-09-13T14:12:09.135603+0000","last_became_active":"2024-09-13T14:11:29.240532+0000","last_became_peered":"2024-09-13T14:11:29.240532+0000","last_unstale":"2024-09-13T14:12:09.135603+0000","last_undegraded":"2024-09-13T14:12:09.135603+0000","last_fullsized":"2024-09-13T14:12:09.135603+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:19:24.828838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.136038+0000","last_change":"2024-09-13T14:11:30.258802+0000","last_active":"2024-09-13T14:12:09.136038+0000","last_peered":"2024-09-13T14:12:09.136038+0000","last_clean":"2024-09-13T14:12:09.136038+0000","last_became_active":"2024-09-13T14:11:30.258508+0000","last_became_peered":"2024-09-13T14:11:30.258508+0000","last_unstale":"2024-09-13T14:12:09.136038+0000","last_undegraded":"2024-09-13T14:12:09.136038+0000","last_fullsized":"2024-09-13T14:12:09.136038+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:06:22.071716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.138003+0000","last_change":"2024-09-13T14:11:29.235406+0000","last_active":"2024-09-13T14:12:09.138003+0000","last_peered":"2024-09-13T14:12:09.138003+0000","last_clean":"2024-09-13T14:12:09.138003+0000","last_became_active":"2024-09-13T14:11:29.235132+0000","last_became_peered":"2024-09-13T14:11:29.235132+0000","last_unstale":"2024-09-13T14:12:09.138003+0000","last_undegraded":"2024-09-13T14:12:09.138003+0000","last_fullsized":"2024-09-13T14:12:09.138003+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:21:21.035202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T14:12:10.460243+0000","last_change":"2024-09-13T14:12:10.460243+0000","last_active":"2024-09-13T14:12:10.460243+0000","last_peered":"2024-09-13T14:12:10.460243+0000","last_clean":"2024-09-13T14:12:10.460243+0000","last_became_active":"2024-09-13T14:12:10.459775+0000","last_became_peered":"2024-09-13T14:12:10.459775+0000","last_unstale":"2024-09-13T14:12:10.460243+0000","last_undegraded":"2024-09-13T14:12:10.460243+0000","last_fullsized":"2024-09-13T14:12:10.460243+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:57:47.924314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"60'128","reported_seq":270,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.216711+0000","last_change":"2024-09-13T14:09:40.436318+0000","last_active":"2024-09-13T14:12:09.216711+0000","last_peered":"2024-09-13T14:12:09.216711+0000","last_clean":"2024-09-13T14:12:09.216711+0000","last_became_active":"2024-09-13T14:09:40.436016+0000","last_became_peered":"2024-09-13T14:09:40.436016+0000","last_unstale":"2024-09-13T14:12:09.216711+0000","last_undegraded":"2024-09-13T14:12:09.216711+0000","last_fullsized":"2024-09-13T14:12:09.216711+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:07:41.927740+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:16.214871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"63'1","reported_seq":29,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T14:12:10.139023+0000","last_change":"2024-09-13T14:10:16.904388+0000","last_active":"2024-09-13T14:12:10.139023+0000","last_peered":"2024-09-13T14:12:10.139023+0000","last_clean":"2024-09-13T14:12:10.139023+0000","last_became_active":"2024-09-13T14:10:12.122143+0000","last_became_peered":"2024-09-13T14:10:12.122143+0000","last_unstale":"2024-09-13T14:12:10.139023+0000","last_undegraded":"2024-09-13T14:12:10.139023+0000","last_fullsized":"2024-09-13T14:12:10.139023+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:11.434245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034973600000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.138338+0000","last_change":"2024-09-13T14:11:29.232107+0000","last_active":"2024-09-13T14:12:09.138338+0000","last_peered":"2024-09-13T14:12:09.138338+0000","last_clean":"2024-09-13T14:12:09.138338+0000","last_became_active":"2024-09-13T14:11:29.231842+0000","last_became_peered":"2024-09-13T14:11:29.231842+0000","last_unstale":"2024-09-13T14:12:09.138338+0000","last_undegraded":"2024-09-13T14:12:09.138338+0000","last_fullsized":"2024-09-13T14:12:09.138338+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:58:26.728667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.216574+0000","last_change":"2024-09-13T14:11:30.253749+0000","last_active":"2024-09-13T14:12:09.216574+0000","last_peered":"2024-09-13T14:12:09.216574+0000","last_clean":"2024-09-13T14:12:09.216574+0000","last_became_active":"2024-09-13T14:11:30.253499+0000","last_became_peered":"2024-09-13T14:11:30.253499+0000","last_unstale":"2024-09-13T14:12:09.216574+0000","last_undegraded":"2024-09-13T14:12:09.216574+0000","last_fullsized":"2024-09-13T14:12:09.216574+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:02:00.987335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":28,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T14:12:10.571224+0000","last_change":"2024-09-13T14:10:17.227534+0000","last_active":"2024-09-13T14:12:10.571224+0000","last_peered":"2024-09-13T14:12:10.571224+0000","last_clean":"2024-09-13T14:12:10.571224+0000","last_became_active":"2024-09-13T14:10:12.123842+0000","last_became_peered":"2024-09-13T14:10:12.123842+0000","last_unstale":"2024-09-13T14:12:10.571224+0000","last_undegraded":"2024-09-13T14:12:10.571224+0000","last_fullsized":"2024-09-13T14:12:10.571224+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:36:09.588482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031558699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.135339+0000","last_change":"2024-09-13T14:11:29.247139+0000","last_active":"2024-09-13T14:12:09.135339+0000","last_peered":"2024-09-13T14:12:09.135339+0000","last_clean":"2024-09-13T14:12:09.135339+0000","last_became_active":"2024-09-13T14:11:29.246049+0000","last_became_peered":"2024-09-13T14:11:29.246049+0000","last_unstale":"2024-09-13T14:12:09.135339+0000","last_undegraded":"2024-09-13T14:12:09.135339+0000","last_fullsized":"2024-09-13T14:12:09.135339+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:14:45.965962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.135802+0000","last_change":"2024-09-13T14:11:30.262905+0000","last_active":"2024-09-13T14:12:09.135802+0000","last_peered":"2024-09-13T14:12:09.135802+0000","last_clean":"2024-09-13T14:12:09.135802+0000","last_became_active":"2024-09-13T14:11:30.262608+0000","last_became_peered":"2024-09-13T14:11:30.262608+0000","last_unstale":"2024-09-13T14:12:09.135802+0000","last_undegraded":"2024-09-13T14:12:09.135802+0000","last_fullsized":"2024-09-13T14:12:09.135802+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:05:29.455679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":28,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T14:12:10.136743+0000","last_change":"2024-09-13T14:10:16.901072+0000","last_active":"2024-09-13T14:12:10.136743+0000","last_peered":"2024-09-13T14:12:10.136743+0000","last_clean":"2024-09-13T14:12:10.136743+0000","last_became_active":"2024-09-13T14:10:13.084740+0000","last_became_peered":"2024-09-13T14:10:13.084740+0000","last_unstale":"2024-09-13T14:12:10.136743+0000","last_undegraded":"2024-09-13T14:12:10.136743+0000","last_fullsized":"2024-09-13T14:12:10.136743+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:06:12.555142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028414100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.139326+0000","last_change":"2024-09-13T14:11:29.230135+0000","last_active":"2024-09-13T14:12:09.139326+0000","last_peered":"2024-09-13T14:12:09.139326+0000","last_clean":"2024-09-13T14:12:09.139326+0000","last_became_active":"2024-09-13T14:11:29.229468+0000","last_became_peered":"2024-09-13T14:11:29.229468+0000","last_unstale":"2024-09-13T14:12:09.139326+0000","last_undegraded":"2024-09-13T14:12:09.139326+0000","last_fullsized":"2024-09-13T14:12:09.139326+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:03:25.995900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.140300+0000","last_change":"2024-09-13T14:11:30.258451+0000","last_active":"2024-09-13T14:12:09.140300+0000","last_peered":"2024-09-13T14:12:09.140300+0000","last_clean":"2024-09-13T14:12:09.140300+0000","last_became_active":"2024-09-13T14:11:30.258149+0000","last_became_peered":"2024-09-13T14:11:30.258149+0000","last_unstale":"2024-09-13T14:12:09.140300+0000","last_undegraded":"2024-09-13T14:12:09.140300+0000","last_fullsized":"2024-09-13T14:12:09.140300+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:45:52.895402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":30,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T14:12:10.139162+0000","last_change":"2024-09-13T14:10:16.904932+0000","last_active":"2024-09-13T14:12:10.139162+0000","last_peered":"2024-09-13T14:12:10.139162+0000","last_clean":"2024-09-13T14:12:10.139162+0000","last_became_active":"2024-09-13T14:10:12.121802+0000","last_became_peered":"2024-09-13T14:10:12.121802+0000","last_unstale":"2024-09-13T14:12:10.139162+0000","last_undegraded":"2024-09-13T14:12:10.139162+0000","last_fullsized":"2024-09-13T14:12:10.139162+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:11:49.860408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088127500000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.135865+0000","last_change":"2024-09-13T14:11:29.245898+0000","last_active":"2024-09-13T14:12:09.135865+0000","last_peered":"2024-09-13T14:12:09.135865+0000","last_clean":"2024-09-13T14:12:09.135865+0000","last_became_active":"2024-09-13T14:11:29.245610+0000","last_became_peered":"2024-09-13T14:11:29.245610+0000","last_unstale":"2024-09-13T14:12:09.135865+0000","last_undegraded":"2024-09-13T14:12:09.135865+0000","last_fullsized":"2024-09-13T14:12:09.135865+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:05:59.228651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T14:12:10.140569+0000","last_change":"2024-09-13T14:12:10.140569+0000","last_active":"2024-09-13T14:12:10.140569+0000","last_peered":"2024-09-13T14:12:10.140569+0000","last_clean":"2024-09-13T14:12:10.140569+0000","last_became_active":"2024-09-13T14:12:10.140244+0000","last_became_peered":"2024-09-13T14:12:10.140244+0000","last_unstale":"2024-09-13T14:12:10.140569+0000","last_undegraded":"2024-09-13T14:12:10.140569+0000","last_fullsized":"2024-09-13T14:12:10.140569+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:42:52.706369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.138278+0000","last_change":"2024-09-13T14:11:29.231269+0000","last_active":"2024-09-13T14:12:09.138278+0000","last_peered":"2024-09-13T14:12:09.138278+0000","last_clean":"2024-09-13T14:12:09.138278+0000","last_became_active":"2024-09-13T14:11:29.230745+0000","last_became_peered":"2024-09-13T14:11:29.230745+0000","last_unstale":"2024-09-13T14:12:09.138278+0000","last_undegraded":"2024-09-13T14:12:09.138278+0000","last_fullsized":"2024-09-13T14:12:09.138278+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:56:34.412003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":28,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T14:12:10.137183+0000","last_change":"2024-09-13T14:10:16.905490+0000","last_active":"2024-09-13T14:12:10.137183+0000","last_peered":"2024-09-13T14:12:10.137183+0000","last_clean":"2024-09-13T14:12:10.137183+0000","last_became_active":"2024-09-13T14:10:12.122768+0000","last_became_peered":"2024-09-13T14:10:12.122768+0000","last_unstale":"2024-09-13T14:12:10.137183+0000","last_undegraded":"2024-09-13T14:12:10.137183+0000","last_fullsized":"2024-09-13T14:12:10.137183+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:55:21.633350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026843400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.139783+0000","last_change":"2024-09-13T14:11:30.259951+0000","last_active":"2024-09-13T14:12:09.139783+0000","last_peered":"2024-09-13T14:12:09.139783+0000","last_clean":"2024-09-13T14:12:09.139783+0000","last_became_active":"2024-09-13T14:11:30.259669+0000","last_became_peered":"2024-09-13T14:11:30.259669+0000","last_unstale":"2024-09-13T14:12:09.139783+0000","last_undegraded":"2024-09-13T14:12:09.139783+0000","last_fullsized":"2024-09-13T14:12:09.139783+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:16:15.483951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.139724+0000","last_change":"2024-09-13T14:11:29.227786+0000","last_active":"2024-09-13T14:12:09.139724+0000","last_peered":"2024-09-13T14:12:09.139724+0000","last_clean":"2024-09-13T14:12:09.139724+0000","last_became_active":"2024-09-13T14:11:29.227448+0000","last_became_peered":"2024-09-13T14:11:29.227448+0000","last_unstale":"2024-09-13T14:12:09.139724+0000","last_undegraded":"2024-09-13T14:12:09.139724+0000","last_fullsized":"2024-09-13T14:12:09.139724+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:51:21.426595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":28,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T14:12:10.571765+0000","last_change":"2024-09-13T14:10:17.227584+0000","last_active":"2024-09-13T14:12:10.571765+0000","last_peered":"2024-09-13T14:12:10.571765+0000","last_clean":"2024-09-13T14:12:10.571765+0000","last_became_active":"2024-09-13T14:10:12.878424+0000","last_became_peered":"2024-09-13T14:10:12.878424+0000","last_unstale":"2024-09-13T14:12:10.571765+0000","last_undegraded":"2024-09-13T14:12:10.571765+0000","last_fullsized":"2024-09-13T14:12:10.571765+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:00:10.824840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032561200000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.135514+0000","last_change":"2024-09-13T14:11:30.258255+0000","last_active":"2024-09-13T14:12:09.135514+0000","last_peered":"2024-09-13T14:12:09.135514+0000","last_clean":"2024-09-13T14:12:09.135514+0000","last_became_active":"2024-09-13T14:11:30.257957+0000","last_became_peered":"2024-09-13T14:11:30.257957+0000","last_unstale":"2024-09-13T14:12:09.135514+0000","last_undegraded":"2024-09-13T14:12:09.135514+0000","last_fullsized":"2024-09-13T14:12:09.135514+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:56:52.739848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":17,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.138513+0000","last_change":"2024-09-13T14:11:29.245356+0000","last_active":"2024-09-13T14:12:09.138513+0000","last_peered":"2024-09-13T14:12:09.138513+0000","last_clean":"2024-09-13T14:12:09.138513+0000","last_became_active":"2024-09-13T14:11:29.245093+0000","last_became_peered":"2024-09-13T14:11:29.245093+0000","last_unstale":"2024-09-13T14:12:09.138513+0000","last_undegraded":"2024-09-13T14:12:09.138513+0000","last_fullsized":"2024-09-13T14:12:09.138513+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:37.819148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":28,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T14:12:10.137274+0000","last_change":"2024-09-13T14:10:16.905444+0000","last_active":"2024-09-13T14:12:10.137274+0000","last_peered":"2024-09-13T14:12:10.137274+0000","last_clean":"2024-09-13T14:12:10.137274+0000","last_became_active":"2024-09-13T14:10:12.878714+0000","last_became_peered":"2024-09-13T14:10:12.878714+0000","last_unstale":"2024-09-13T14:12:10.137274+0000","last_undegraded":"2024-09-13T14:12:10.137274+0000","last_fullsized":"2024-09-13T14:12:10.137274+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:23:02.772611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000229423,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.138569+0000","last_change":"2024-09-13T14:11:30.266038+0000","last_active":"2024-09-13T14:12:09.138569+0000","last_peered":"2024-09-13T14:12:09.138569+0000","last_clean":"2024-09-13T14:12:09.138569+0000","last_became_active":"2024-09-13T14:11:30.265742+0000","last_became_peered":"2024-09-13T14:11:30.265742+0000","last_unstale":"2024-09-13T14:12:09.138569+0000","last_undegraded":"2024-09-13T14:12:09.138569+0000","last_fullsized":"2024-09-13T14:12:09.138569+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:58:01.809837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.135929+0000","last_change":"2024-09-13T14:11:29.248383+0000","last_active":"2024-09-13T14:12:09.135929+0000","last_peered":"2024-09-13T14:12:09.135929+0000","last_clean":"2024-09-13T14:12:09.135929+0000","last_became_active":"2024-09-13T14:11:29.247999+0000","last_became_peered":"2024-09-13T14:11:29.247999+0000","last_unstale":"2024-09-13T14:12:09.135929+0000","last_undegraded":"2024-09-13T14:12:09.135929+0000","last_fullsized":"2024-09-13T14:12:09.135929+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:20:21.763210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":28,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T14:12:10.137607+0000","last_change":"2024-09-13T14:10:16.905852+0000","last_active":"2024-09-13T14:12:10.137607+0000","last_peered":"2024-09-13T14:12:10.137607+0000","last_clean":"2024-09-13T14:12:10.137607+0000","last_became_active":"2024-09-13T14:10:12.122815+0000","last_became_peered":"2024-09-13T14:10:12.122815+0000","last_unstale":"2024-09-13T14:12:10.137607+0000","last_undegraded":"2024-09-13T14:12:10.137607+0000","last_fullsized":"2024-09-13T14:12:10.137607+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:59:17.830936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034784200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.135490+0000","last_change":"2024-09-13T14:11:30.256034+0000","last_active":"2024-09-13T14:12:09.135490+0000","last_peered":"2024-09-13T14:12:09.135490+0000","last_clean":"2024-09-13T14:12:09.135490+0000","last_became_active":"2024-09-13T14:11:30.255727+0000","last_became_peered":"2024-09-13T14:11:30.255727+0000","last_unstale":"2024-09-13T14:12:09.135490+0000","last_undegraded":"2024-09-13T14:12:09.135490+0000","last_fullsized":"2024-09-13T14:12:09.135490+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:21:35.817014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.135716+0000","last_change":"2024-09-13T14:11:30.253539+0000","last_active":"2024-09-13T14:12:09.135716+0000","last_peered":"2024-09-13T14:12:09.135716+0000","last_clean":"2024-09-13T14:12:09.135716+0000","last_became_active":"2024-09-13T14:11:30.253282+0000","last_became_peered":"2024-09-13T14:11:30.253282+0000","last_unstale":"2024-09-13T14:12:09.135716+0000","last_undegraded":"2024-09-13T14:12:09.135716+0000","last_fullsized":"2024-09-13T14:12:09.135716+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:29:58.721512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"71'6","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.135826+0000","last_change":"2024-09-13T14:11:29.237013+0000","last_active":"2024-09-13T14:12:09.135826+0000","last_peered":"2024-09-13T14:12:09.135826+0000","last_clean":"2024-09-13T14:12:09.135826+0000","last_became_active":"2024-09-13T14:11:29.236643+0000","last_became_peered":"2024-09-13T14:11:29.236643+0000","last_unstale":"2024-09-13T14:12:09.135826+0000","last_undegraded":"2024-09-13T14:12:09.135826+0000","last_fullsized":"2024-09-13T14:12:09.135826+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:33:58.438693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.139562+0000","last_change":"2024-09-13T14:11:30.257330+0000","last_active":"2024-09-13T14:12:09.139562+0000","last_peered":"2024-09-13T14:12:09.139562+0000","last_clean":"2024-09-13T14:12:09.139562+0000","last_became_active":"2024-09-13T14:11:30.257017+0000","last_became_peered":"2024-09-13T14:11:30.257017+0000","last_unstale":"2024-09-13T14:12:09.139562+0000","last_undegraded":"2024-09-13T14:12:09.139562+0000","last_fullsized":"2024-09-13T14:12:09.139562+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:43:55.648230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":18,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.216942+0000","last_change":"2024-09-13T14:11:29.237584+0000","last_active":"2024-09-13T14:12:09.216942+0000","last_peered":"2024-09-13T14:12:09.216942+0000","last_clean":"2024-09-13T14:12:09.216942+0000","last_became_active":"2024-09-13T14:11:29.237314+0000","last_became_peered":"2024-09-13T14:11:29.237314+0000","last_unstale":"2024-09-13T14:12:09.216942+0000","last_undegraded":"2024-09-13T14:12:09.216942+0000","last_fullsized":"2024-09-13T14:12:09.216942+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:50:37.627401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.138073+0000","last_change":"2024-09-13T14:11:29.229513+0000","last_active":"2024-09-13T14:12:09.138073+0000","last_peered":"2024-09-13T14:12:09.138073+0000","last_clean":"2024-09-13T14:12:09.138073+0000","last_became_active":"2024-09-13T14:11:29.228815+0000","last_became_peered":"2024-09-13T14:11:29.228815+0000","last_unstale":"2024-09-13T14:12:09.138073+0000","last_undegraded":"2024-09-13T14:12:09.138073+0000","last_fullsized":"2024-09-13T14:12:09.138073+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:43:23.900655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.137847+0000","last_change":"2024-09-13T14:11:30.265539+0000","last_active":"2024-09-13T14:12:09.137847+0000","last_peered":"2024-09-13T14:12:09.137847+0000","last_clean":"2024-09-13T14:12:09.137847+0000","last_became_active":"2024-09-13T14:11:30.264954+0000","last_became_peered":"2024-09-13T14:11:30.264954+0000","last_unstale":"2024-09-13T14:12:09.137847+0000","last_undegraded":"2024-09-13T14:12:09.137847+0000","last_fullsized":"2024-09-13T14:12:09.137847+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:27:39.218811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.135836+0000","last_change":"2024-09-13T14:11:29.247957+0000","last_active":"2024-09-13T14:12:09.135836+0000","last_peered":"2024-09-13T14:12:09.135836+0000","last_clean":"2024-09-13T14:12:09.135836+0000","last_became_active":"2024-09-13T14:11:29.247135+0000","last_became_peered":"2024-09-13T14:11:29.247135+0000","last_unstale":"2024-09-13T14:12:09.135836+0000","last_undegraded":"2024-09-13T14:12:09.135836+0000","last_fullsized":"2024-09-13T14:12:09.135836+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:42:34.941145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.137813+0000","last_change":"2024-09-13T14:11:30.264735+0000","last_active":"2024-09-13T14:12:09.137813+0000","last_peered":"2024-09-13T14:12:09.137813+0000","last_clean":"2024-09-13T14:12:09.137813+0000","last_became_active":"2024-09-13T14:11:30.264369+0000","last_became_peered":"2024-09-13T14:11:30.264369+0000","last_unstale":"2024-09-13T14:12:09.137813+0000","last_undegraded":"2024-09-13T14:12:09.137813+0000","last_fullsized":"2024-09-13T14:12:09.137813+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:32:11.058695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.216633+0000","last_change":"2024-09-13T14:11:30.260356+0000","last_active":"2024-09-13T14:12:09.216633+0000","last_peered":"2024-09-13T14:12:09.216633+0000","last_clean":"2024-09-13T14:12:09.216633+0000","last_became_active":"2024-09-13T14:11:30.260089+0000","last_became_peered":"2024-09-13T14:11:30.260089+0000","last_unstale":"2024-09-13T14:12:09.216633+0000","last_undegraded":"2024-09-13T14:12:09.216633+0000","last_fullsized":"2024-09-13T14:12:09.216633+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:21:42.191411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.137930+0000","last_change":"2024-09-13T14:11:29.239413+0000","last_active":"2024-09-13T14:12:09.137930+0000","last_peered":"2024-09-13T14:12:09.137930+0000","last_clean":"2024-09-13T14:12:09.137930+0000","last_became_active":"2024-09-13T14:11:29.239160+0000","last_became_peered":"2024-09-13T14:11:29.239160+0000","last_unstale":"2024-09-13T14:12:09.137930+0000","last_undegraded":"2024-09-13T14:12:09.137930+0000","last_fullsized":"2024-09-13T14:12:09.137930+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:33:31.411058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.135426+0000","last_change":"2024-09-13T14:11:30.260094+0000","last_active":"2024-09-13T14:12:09.135426+0000","last_peered":"2024-09-13T14:12:09.135426+0000","last_clean":"2024-09-13T14:12:09.135426+0000","last_became_active":"2024-09-13T14:11:30.259825+0000","last_became_peered":"2024-09-13T14:11:30.259825+0000","last_unstale":"2024-09-13T14:12:09.135426+0000","last_undegraded":"2024-09-13T14:12:09.135426+0000","last_fullsized":"2024-09-13T14:12:09.135426+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:13:03.900066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.139421+0000","last_change":"2024-09-13T14:11:29.245452+0000","last_active":"2024-09-13T14:12:09.139421+0000","last_peered":"2024-09-13T14:12:09.139421+0000","last_clean":"2024-09-13T14:12:09.139421+0000","last_became_active":"2024-09-13T14:11:29.245124+0000","last_became_peered":"2024-09-13T14:11:29.245124+0000","last_unstale":"2024-09-13T14:12:09.139421+0000","last_undegraded":"2024-09-13T14:12:09.139421+0000","last_fullsized":"2024-09-13T14:12:09.139421+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:17:21.475412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":23,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T14:12:10.460399+0000","last_change":"2024-09-13T14:12:10.460399+0000","last_active":"2024-09-13T14:12:10.460399+0000","last_peered":"2024-09-13T14:12:10.460399+0000","last_clean":"2024-09-13T14:12:10.460399+0000","last_became_active":"2024-09-13T14:12:10.460037+0000","last_became_peered":"2024-09-13T14:12:10.460037+0000","last_unstale":"2024-09-13T14:12:10.460399+0000","last_undegraded":"2024-09-13T14:12:10.460399+0000","last_fullsized":"2024-09-13T14:12:10.460399+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:52:47.992230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.135624+0000","last_change":"2024-09-13T14:11:29.241053+0000","last_active":"2024-09-13T14:12:09.135624+0000","last_peered":"2024-09-13T14:12:09.135624+0000","last_clean":"2024-09-13T14:12:09.135624+0000","last_became_active":"2024-09-13T14:11:29.240812+0000","last_became_peered":"2024-09-13T14:11:29.240812+0000","last_unstale":"2024-09-13T14:12:09.135624+0000","last_undegraded":"2024-09-13T14:12:09.135624+0000","last_fullsized":"2024-09-13T14:12:09.135624+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:07:48.614371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.135742+0000","last_change":"2024-09-13T14:11:29.436934+0000","last_active":"2024-09-13T14:12:09.135742+0000","last_peered":"2024-09-13T14:12:09.135742+0000","last_clean":"2024-09-13T14:12:09.135742+0000","last_became_active":"2024-09-13T14:11:29.436588+0000","last_became_peered":"2024-09-13T14:11:29.436588+0000","last_unstale":"2024-09-13T14:12:09.135742+0000","last_undegraded":"2024-09-13T14:12:09.135742+0000","last_fullsized":"2024-09-13T14:12:09.135742+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:30:49.599505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.135594+0000","last_change":"2024-09-13T14:11:29.247872+0000","last_active":"2024-09-13T14:12:09.135594+0000","last_peered":"2024-09-13T14:12:09.135594+0000","last_clean":"2024-09-13T14:12:09.135594+0000","last_became_active":"2024-09-13T14:11:29.247195+0000","last_became_peered":"2024-09-13T14:11:29.247195+0000","last_unstale":"2024-09-13T14:12:09.135594+0000","last_undegraded":"2024-09-13T14:12:09.135594+0000","last_fullsized":"2024-09-13T14:12:09.135594+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:27:53.785083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.135643+0000","last_change":"2024-09-13T14:11:30.260805+0000","last_active":"2024-09-13T14:12:09.135643+0000","last_peered":"2024-09-13T14:12:09.135643+0000","last_clean":"2024-09-13T14:12:09.135643+0000","last_became_active":"2024-09-13T14:11:30.259503+0000","last_became_peered":"2024-09-13T14:11:30.259503+0000","last_unstale":"2024-09-13T14:12:09.135643+0000","last_undegraded":"2024-09-13T14:12:09.135643+0000","last_fullsized":"2024-09-13T14:12:09.135643+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:59:42.757834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":17,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.138244+0000","last_change":"2024-09-13T14:11:29.240160+0000","last_active":"2024-09-13T14:12:09.138244+0000","last_peered":"2024-09-13T14:12:09.138244+0000","last_clean":"2024-09-13T14:12:09.138244+0000","last_became_active":"2024-09-13T14:11:29.239900+0000","last_became_peered":"2024-09-13T14:11:29.239900+0000","last_unstale":"2024-09-13T14:12:09.138244+0000","last_undegraded":"2024-09-13T14:12:09.138244+0000","last_fullsized":"2024-09-13T14:12:09.138244+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:12:35.557474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.135487+0000","last_change":"2024-09-13T14:11:30.259223+0000","last_active":"2024-09-13T14:12:09.135487+0000","last_peered":"2024-09-13T14:12:09.135487+0000","last_clean":"2024-09-13T14:12:09.135487+0000","last_became_active":"2024-09-13T14:11:30.258345+0000","last_became_peered":"2024-09-13T14:11:30.258345+0000","last_unstale":"2024-09-13T14:12:09.135487+0000","last_undegraded":"2024-09-13T14:12:09.135487+0000","last_fullsized":"2024-09-13T14:12:09.135487+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:06.225118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":17,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.138227+0000","last_change":"2024-09-13T14:11:29.240351+0000","last_active":"2024-09-13T14:12:09.138227+0000","last_peered":"2024-09-13T14:12:09.138227+0000","last_clean":"2024-09-13T14:12:09.138227+0000","last_became_active":"2024-09-13T14:11:29.240060+0000","last_became_peered":"2024-09-13T14:11:29.240060+0000","last_unstale":"2024-09-13T14:12:09.138227+0000","last_undegraded":"2024-09-13T14:12:09.138227+0000","last_fullsized":"2024-09-13T14:12:09.138227+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:34:36.126368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.137961+0000","last_change":"2024-09-13T14:11:30.263317+0000","last_active":"2024-09-13T14:12:09.137961+0000","last_peered":"2024-09-13T14:12:09.137961+0000","last_clean":"2024-09-13T14:12:09.137961+0000","last_became_active":"2024-09-13T14:11:30.263054+0000","last_became_peered":"2024-09-13T14:11:30.263054+0000","last_unstale":"2024-09-13T14:12:09.137961+0000","last_undegraded":"2024-09-13T14:12:09.137961+0000","last_fullsized":"2024-09-13T14:12:09.137961+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:04.263430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.138156+0000","last_change":"2024-09-13T14:11:29.232949+0000","last_active":"2024-09-13T14:12:09.138156+0000","last_peered":"2024-09-13T14:12:09.138156+0000","last_clean":"2024-09-13T14:12:09.138156+0000","last_became_active":"2024-09-13T14:11:29.232668+0000","last_became_peered":"2024-09-13T14:11:29.232668+0000","last_unstale":"2024-09-13T14:12:09.138156+0000","last_undegraded":"2024-09-13T14:12:09.138156+0000","last_fullsized":"2024-09-13T14:12:09.138156+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:35:06.935233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.135377+0000","last_change":"2024-09-13T14:11:30.252635+0000","last_active":"2024-09-13T14:12:09.135377+0000","last_peered":"2024-09-13T14:12:09.135377+0000","last_clean":"2024-09-13T14:12:09.135377+0000","last_became_active":"2024-09-13T14:11:30.252379+0000","last_became_peered":"2024-09-13T14:11:30.252379+0000","last_unstale":"2024-09-13T14:12:09.135377+0000","last_undegraded":"2024-09-13T14:12:09.135377+0000","last_fullsized":"2024-09-13T14:12:09.135377+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:33:09.887379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":19,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:12:09.135843+0000","last_change":"2024-09-13T14:11:29.247512+0000","last_active":"2024-09-13T14:12:09.135843+0000","last_peered":"2024-09-13T14:12:09.135843+0000","last_clean":"2024-09-13T14:12:09.135843+0000","last_became_active":"2024-09-13T14:11:29.247065+0000","last_became_peered":"2024-09-13T14:11:29.247065+0000","last_unstale":"2024-09-13T14:12:09.135843+0000","last_undegraded":"2024-09-13T14:12:09.135843+0000","last_fullsized":"2024-09-13T14:12:09.135843+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:17:27.667480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":7086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234023,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28364,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716788,"statfs":{"total":95995035648,"available":95965990912,"internally_reserved":0,"allocated":1376256,"data_stored":925421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:11:40 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.54300000000000004,"15min":0.54800000000000004},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.76800000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.502,"15min":0.496},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.753,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.65800000000000003}]},{"osd":1,"last update":"Fri Sep 13 14:11:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.50700000000000001,"15min":0.501},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.76200000000000001,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52800000000000002,"15min":0.52900000000000003},"min":{"1min":0.40999999999999998,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.77700000000000002,"5min":0.871,"15min":0.871},"last":0.621}]},{"osd":2,"last update":"Fri Sep 13 14:11:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.52500000000000002,"15min":0.52100000000000002},"min":{"1min":0.38100000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.68999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.38400000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.71299999999999997,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.57799999999999996}]},{"osd":3,"last update":"Fri Sep 13 14:11:40 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.55500000000000005,"15min":0.55800000000000005},"min":{"1min":0.36799999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.72099999999999997,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.53600000000000003,"15min":0.53500000000000003},"min":{"1min":0.40999999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.73299999999999998,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.60799999999999998}]},{"osd":4,"last update":"Fri Sep 13 14:11:40 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.42899999999999999,"15min":0.42299999999999999},"min":{"1min":0.30099999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.73099999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.46000000000000002}]},{"osd":5,"last update":"Fri Sep 13 14:11:40 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.64100000000000001,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.32700000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.78800000000000003,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.54300000000000004}]},{"osd":6,"last update":"Fri Sep 13 14:11:40 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.52100000000000002,"15min":0.51900000000000002},"min":{"1min":0.42299999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.67700000000000005,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.33600000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.52500000000000002}]}]},{"osd":6,"up_from":46,"seq":197568495659,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32456,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712696,"statfs":{"total":95995035648,"available":95961800704,"internally_reserved":0,"allocated":1368064,"data_stored":925352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:12:22 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52400000000000002},"min":{"1min":0.38500000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.69099999999999995,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.52800000000000002,"15min":0.52600000000000002},"min":{"1min":0.27900000000000003,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.67200000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.45000000000000001}]},{"osd":1,"last update":"Fri Sep 13 14:12:22 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.35299999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.67300000000000004,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53900000000000003,"15min":0.54100000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.67600000000000005,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.60499999999999998}]},{"osd":2,"last update":"Fri Sep 13 14:12:22 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.53900000000000003,"15min":0.54100000000000004},"min":{"1min":0.38100000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.71099999999999997,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.501},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53500000000000003,"15min":0.53800000000000003},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.64700000000000002,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.54600000000000004}]},{"osd":3,"last update":"Fri Sep 13 14:12:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.63800000000000001,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.54100000000000004,"15min":0.53500000000000003},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.70499999999999996,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.53100000000000003}]},{"osd":4,"last update":"Fri Sep 13 14:12:22 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.43099999999999999,"15min":0.41099999999999998},"min":{"1min":0.34699999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.434,"15min":0.41399999999999998},"min":{"1min":0.33600000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.42899999999999999}]},{"osd":5,"last update":"Fri Sep 13 14:12:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.44400000000000001,"15min":0.42999999999999999},"min":{"1min":0.35099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.67400000000000004,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.47599999999999998,"15min":0.45400000000000001},"min":{"1min":0.41399999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.56499999999999995}]},{"osd":7,"last update":"Fri Sep 13 14:11:40 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.46700000000000003,"15min":0.46500000000000002},"min":{"1min":0.37,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.53200000000000003,"15min":0.52800000000000002},"min":{"1min":0.39100000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.69299999999999995,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.58999999999999997}]}]},{"osd":5,"up_from":38,"seq":163208757296,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31868,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713284,"statfs":{"total":95995035648,"available":95962402816,"internally_reserved":0,"allocated":765952,"data_stored":333944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51900000000000002,"15min":0.52900000000000003},"min":{"1min":0.38100000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.623,"5min":1.216,"15min":1.216},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.49299999999999999,"15min":0.495},"min":{"1min":0.33200000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.63700000000000001,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.64400000000000002}]},{"osd":1,"last update":"Fri Sep 13 14:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.50700000000000001,"15min":0.50900000000000001},"min":{"1min":0.39700000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.63800000000000001,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.50800000000000001,"15min":0.52000000000000002},"min":{"1min":0.35699999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.59099999999999997,"5min":1.133,"15min":1.133},"last":0.53900000000000003}]},{"osd":2,"last update":"Fri Sep 13 14:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.52900000000000003,"15min":0.54100000000000004},"min":{"1min":0.38500000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.60199999999999998,"5min":1.026,"15min":1.026},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.51900000000000002,"15min":0.52300000000000002},"min":{"1min":0.35799999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.68600000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.57099999999999995}]},{"osd":3,"last update":"Fri Sep 13 14:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.5,"15min":0.50700000000000001},"min":{"1min":0.39800000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.62,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.497},{"interface":"front","average":{"1min":0.502,"5min":0.53200000000000003,"15min":0.54100000000000004},"min":{"1min":0.38,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.64400000000000002,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.50900000000000001}]},{"osd":4,"last update":"Fri Sep 13 14:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.40899999999999997,"15min":0.39900000000000002},"min":{"1min":0.313,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.73299999999999998,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.442,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.31,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65500000000000003,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.42599999999999999}]},{"osd":6,"last update":"Fri Sep 13 14:12:17 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.47299999999999998,"15min":0.47899999999999998},"min":{"1min":0.30599999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.77300000000000002,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.48499999999999999,"15min":0.47799999999999998},"min":{"1min":0.29299999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.69699999999999995,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.55600000000000005}]},{"osd":7,"last update":"Fri Sep 13 14:11:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.48899999999999999,"15min":0.48399999999999999},"min":{"1min":0.33100000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.70599999999999996,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.499,"15min":0.49199999999999999},"min":{"1min":0.36199999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.61099999999999999}]}]},{"osd":4,"up_from":30,"seq":128849018932,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27784,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717368,"statfs":{"total":95995035648,"available":95966584832,"internally_reserved":0,"allocated":778240,"data_stored":335041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:11:32 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.55300000000000005,"15min":0.58699999999999997},"min":{"1min":0.374,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.65600000000000003,"5min":1.258,"15min":1.258},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.502,"5min":0.54900000000000004,"15min":0.57499999999999996},"min":{"1min":0.36099999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.67400000000000004,"5min":0.997,"15min":0.997},"last":0.50900000000000001}]},{"osd":1,"last update":"Fri Sep 13 14:11:32 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.55300000000000005,"15min":0.58199999999999996},"min":{"1min":0.38500000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.71699999999999997,"5min":1.117,"15min":1.117},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55100000000000005,"15min":0.57399999999999995},"min":{"1min":0.41499999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.70399999999999996,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":0.53300000000000003}]},{"osd":2,"last update":"Fri Sep 13 14:11:32 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.54600000000000004,"15min":0.57399999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.64500000000000002,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.56100000000000005,"15min":0.58699999999999997},"min":{"1min":0.40600000000000003,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.73099999999999998,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.55700000000000005}]},{"osd":3,"last update":"Fri Sep 13 14:11:32 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.57599999999999996,"15min":0.59899999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.76100000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.56699999999999995,"15min":0.59499999999999997},"min":{"1min":0.42599999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.628,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.63700000000000001}]},{"osd":5,"last update":"Fri Sep 13 14:11:54 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.376,"15min":0.35999999999999999},"min":{"1min":0.30299999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.63900000000000001,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.49099999999999999,"15min":0.47299999999999998},"min":{"1min":0.39600000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.68999999999999995,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.68500000000000005}]},{"osd":6,"last update":"Fri Sep 13 14:12:19 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.44500000000000001,"15min":0.43099999999999999},"min":{"1min":0.36799999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.65400000000000003,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.42799999999999999,"15min":0.40999999999999998},"min":{"1min":0.30599999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.64100000000000001,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.48399999999999999}]},{"osd":7,"last update":"Fri Sep 13 14:11:37 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.47399999999999998,"15min":0.46800000000000003},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.625},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.45500000000000002,"15min":0.45200000000000001},"min":{"1min":0.34799999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.69899999999999995}]}]},{"osd":3,"up_from":25,"seq":107374182457,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27764,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717388,"statfs":{"total":95995035648,"available":95966605312,"internally_reserved":0,"allocated":761856,"data_stored":333694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:12:16 2024","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.34999999999999998,"15min":0.35299999999999998},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.505,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.375,"15min":0.38},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.56999999999999995,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.27200000000000002}]},{"osd":1,"last update":"Fri Sep 13 14:12:16 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.38800000000000001,"15min":0.39400000000000002},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.54100000000000004,"5min":0.747,"15min":0.747},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.378,"15min":0.38400000000000001},"min":{"1min":0.26400000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.253}]},{"osd":2,"last update":"Fri Sep 13 14:12:16 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.39700000000000002,"15min":0.39900000000000002},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.371,"5min":0.39500000000000002,"15min":0.39800000000000002},"min":{"1min":0.26200000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.504,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.26300000000000001}]},{"osd":4,"last update":"Fri Sep 13 14:11:35 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52300000000000002,"15min":0.51400000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.73599999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.53900000000000003,"15min":0.52600000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.56399999999999995}]},{"osd":5,"last update":"Fri Sep 13 14:11:51 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56699999999999995,"15min":0.56499999999999995},"min":{"1min":0.39500000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.75900000000000001,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.54100000000000004,"15min":0.52700000000000002},"min":{"1min":0.45000000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.48299999999999998}]},{"osd":6,"last update":"Fri Sep 13 14:12:18 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.60599999999999998,"15min":0.60399999999999998},"min":{"1min":0.433,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.84899999999999998,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60499999999999998,"15min":0.61299999999999999},"min":{"1min":0.42199999999999999,"5min":0.37,"15min":0.37},"max":{"1min":0.76200000000000001,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.44}]},{"osd":7,"last update":"Fri Sep 13 14:11:41 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.59999999999999998,"15min":0.59599999999999997},"min":{"1min":0.45900000000000002,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.625,"5min":0.60599999999999998,"15min":0.60299999999999998},"min":{"1min":0.42399999999999999,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.54200000000000004}]}]},{"osd":2,"up_from":18,"seq":77309411389,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27772,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717380,"statfs":{"total":95995035648,"available":95966597120,"internally_reserved":0,"allocated":765952,"data_stored":334220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:11:49 2024","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.23899999999999999,"5min":0.22,"15min":0.22},"max":{"1min":0.46400000000000002,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.38400000000000001,"15min":0.38600000000000001},"min":{"1min":0.27700000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.44800000000000001,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.34100000000000003}]},{"osd":1,"last update":"Fri Sep 13 14:11:49 2024","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.36699999999999999,"15min":0.36499999999999999},"min":{"1min":0.24299999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.44500000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.318},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.40500000000000003,"15min":0.39600000000000002},"min":{"1min":0.26000000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.47999999999999998,"5min":0.745,"15min":0.745},"last":0.35999999999999999}]},{"osd":3,"last update":"Fri Sep 13 14:12:13 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.439,"15min":0.46899999999999997},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.501,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.41099999999999998,"15min":0.44600000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.46400000000000002,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.375}]},{"osd":4,"last update":"Fri Sep 13 14:12:30 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.55800000000000005,"15min":0.57399999999999995},"min":{"1min":0.39600000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57899999999999996,"15min":0.60899999999999999},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.47999999999999998}]},{"osd":5,"last update":"Fri Sep 13 14:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.621,"15min":0.64200000000000002},"min":{"1min":0.39000000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.89300000000000002,"5min":1.2,"15min":1.2},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57099999999999995,"15min":0.58199999999999996},"min":{"1min":0.41599999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.79000000000000004,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.59599999999999997}]},{"osd":6,"last update":"Fri Sep 13 14:12:13 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56299999999999994,"15min":0.55900000000000005},"min":{"1min":0.38700000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.57699999999999996,"15min":0.56799999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.54600000000000004}]},{"osd":7,"last update":"Fri Sep 13 14:11:41 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.54800000000000004,"15min":0.54100000000000004},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.81100000000000005,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.495}]}]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27764,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717388,"statfs":{"total":95995035648,"available":95966605312,"internally_reserved":0,"allocated":757760,"data_stored":333350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:12:24 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.34200000000000003,"15min":0.30599999999999999},"min":{"1min":0.222,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.40000000000000002,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.249},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.35299999999999998,"15min":0.32000000000000001},"min":{"1min":0.21199999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.49299999999999999,"5min":0.54600000000000004,"15min":0.54600000000000004},"last":0.34300000000000003}]},{"osd":2,"last update":"Fri Sep 13 14:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.36199999999999999,"15min":0.34999999999999998},"min":{"1min":0.222,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.53600000000000003,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.35799999999999998,"15min":0.35299999999999998},"min":{"1min":0.24099999999999999,"5min":0.224,"15min":0.224},"max":{"1min":0.44800000000000001,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.31900000000000001}]},{"osd":3,"last update":"Fri Sep 13 14:12:10 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.38400000000000001,"15min":0.39700000000000002},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.47599999999999998,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.372,"5min":0.41599999999999998,"15min":0.42599999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.53000000000000003,"5min":0.621,"15min":0.621},"last":0.38800000000000001}]},{"osd":4,"last update":"Fri Sep 13 14:11:35 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.55900000000000005,"15min":0.55400000000000005},"min":{"1min":0.35899999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.89800000000000002,"5min":1.133,"15min":1.133},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.60199999999999998,"15min":0.623},"min":{"1min":0.36499999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.876,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.58099999999999996}]},{"osd":5,"last update":"Fri Sep 13 14:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.54900000000000004,"15min":0.54600000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.56599999999999995,"15min":0.56100000000000005},"min":{"1min":0.377,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.56999999999999995}]},{"osd":6,"last update":"Fri Sep 13 14:12:19 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.39000000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.70899999999999996,"5min":1.044,"15min":1.044},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.59899999999999998,"15min":0.60399999999999998},"min":{"1min":0.41299999999999998,"5min":0.379,"15min":0.379},"max":{"1min":0.71999999999999997,"5min":1.02,"15min":1.02},"last":0.56000000000000005}]},{"osd":7,"last update":"Fri Sep 13 14:11:41 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.56599999999999995,"15min":0.56299999999999994},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.57799999999999996,"15min":0.57599999999999996},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.61599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738439,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28352,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716800,"statfs":{"total":95995035648,"available":95966003200,"internally_reserved":0,"allocated":1359872,"data_stored":924576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 14:11:28 2024","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.34599999999999997,"15min":0.33000000000000002},"min":{"1min":0.245,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.46400000000000002,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.34899999999999998,"15min":0.33200000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.496,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.27300000000000002}]},{"osd":2,"last update":"Fri Sep 13 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.374,"15min":0.374},"min":{"1min":0.23300000000000001,"5min":0.192,"15min":0.192},"max":{"1min":0.47499999999999998,"5min":0.54300000000000004,"15min":0.54300000000000004},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.379,"15min":0.39100000000000001},"min":{"1min":0.255,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.46400000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.32700000000000001}]},{"osd":3,"last update":"Fri Sep 13 14:12:11 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.40699999999999997,"15min":0.42099999999999999},"min":{"1min":0.28999999999999998,"5min":0.224,"15min":0.224},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.45000000000000001,"15min":0.46200000000000002},"min":{"1min":0.30199999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.35099999999999998}]},{"osd":4,"last update":"Fri Sep 13 14:11:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.56100000000000005,"15min":0.57799999999999996},"min":{"1min":0.38900000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.72299999999999998,"5min":1.044,"15min":1.044},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.56000000000000005,"15min":0.58099999999999996},"min":{"1min":0.35299999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.71099999999999997,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.54300000000000004}]},{"osd":5,"last update":"Fri Sep 13 14:11:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59699999999999998,"15min":0.61699999999999999},"min":{"1min":0.499,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.755,"5min":1.109,"15min":1.109},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58399999999999996,"15min":0.60699999999999998},"min":{"1min":0.41099999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.84599999999999997,"5min":1.151,"15min":1.151},"last":0.47199999999999998}]},{"osd":6,"last update":"Fri Sep 13 14:12:18 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.57899999999999996,"15min":0.57699999999999996},"min":{"1min":0.504,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.82799999999999996,"5min":0.879,"15min":0.879},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.59899999999999998,"15min":0.59299999999999997},"min":{"1min":0.496,"5min":0.379,"15min":0.379},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.435}]},{"osd":7,"last update":"Fri Sep 13 14:11:43 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.53200000000000003,"15min":0.52600000000000002},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.56999999999999995,"15min":0.56599999999999995},"min":{"1min":0.42399999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.48299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:12:32.321 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-13T14:12:32.321 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-13T14:12:32.321 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-13T14:12:32.321 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph health --format=json 2024-09-13T14:12:32.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:32 smithi195 ceph-mon[33205]: pgmap v88: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:32.518 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:12:32.639 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:32 smithi102 ceph-mon[29355]: pgmap v88: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:32.639 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:32 smithi102 ceph-mon[40543]: pgmap v88: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:33.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:33 smithi195 ceph-mon[33205]: from='client.24653 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:12:33.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:33 smithi102 ceph-mon[29355]: from='client.24653 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:12:33.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:33 smithi102 ceph-mon[40543]: from='client.24653 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:12:34.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:34 smithi102 ceph-mon[29355]: pgmap v89: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:34.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:34 smithi102 ceph-mon[40543]: pgmap v89: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:34.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:34 smithi195 ceph-mon[33205]: pgmap v89: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:34.501 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:12:34.501 INFO:teuthology.orchestra.run.smithi102.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-13T14:12:35.167 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-13T14:12:35.167 INFO:teuthology.run_tasks:Running task print... 2024-09-13T14:12:35.176 INFO:teuthology.task.print:**** upgrading first half of cluster, with stress **** 2024-09-13T14:12:35.176 INFO:teuthology.run_tasks:Running task parallel... 2024-09-13T14:12:35.214 INFO:teuthology.task.parallel:starting parallel... 2024-09-13T14:12:35.214 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-09-13T14:12:35.215 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-09-13T14:12:35.220 INFO:tasks.workunit:Pulling workunits from ref reef 2024-09-13T14:12:35.221 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-13T14:12:35.221 DEBUG:teuthology.orchestra.run.smithi102:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-13T14:12:35.221 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-09-13T14:12:35.222 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-09-13T14:12:35.222 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi102.front.sepia.ceph.com 2024-09-13T14:12:35.222 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'ceph config set mgr mgr/cephadm/daemon_cache_timeout 60' 2024-09-13T14:12:35.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:12:35.239 INFO:teuthology.orchestra.run.smithi102.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-13T14:12:35.240 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-13T14:12:35.294 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-13T14:12:35.295 DEBUG:teuthology.orchestra.run.smithi102:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-13T14:12:35.350 INFO:tasks.workunit:timeout=3h 2024-09-13T14:12:35.350 INFO:tasks.workunit:cleanup=True 2024-09-13T14:12:35.351 DEBUG:teuthology.orchestra.run.smithi102:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone --depth 1 --branch reef https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 2024-09-13T14:12:35.407 INFO:tasks.workunit.client.0.smithi102.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-13T14:12:35.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:35 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/230994688' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-13T14:12:35.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:35 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/230994688' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-13T14:12:35.426 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:12:35.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:35 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/230994688' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-13T14:12:36.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:36 smithi195 ceph-mon[33205]: pgmap v90: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:36.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:36 smithi102 ceph-mon[40543]: pgmap v90: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:36.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:36 smithi102 ceph-mon[29355]: pgmap v90: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:37.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:12:37 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:12:37] "GET /metrics HTTP/1.1" 200 33719 "" "Prometheus/2.43.0" 2024-09-13T14:12:37.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:12:37 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: [13/Sep/2024:14:12:37] ENGINE Bus STOPPING 2024-09-13T14:12:38.168 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:12:37 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: [13/Sep/2024:14:12:37] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-13T14:12:38.168 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:12:37 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: [13/Sep/2024:14:12:37] ENGINE Bus STOPPED 2024-09-13T14:12:38.168 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:12:37 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: [13/Sep/2024:14:12:37] ENGINE Bus STARTING 2024-09-13T14:12:38.168 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:12:37 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: [13/Sep/2024:14:12:37] ENGINE Serving on http://:::9283 2024-09-13T14:12:38.168 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:12:37 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: [13/Sep/2024:14:12:37] ENGINE Bus STARTED 2024-09-13T14:12:38.283 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'ceph config set global log_to_journald false --force' 2024-09-13T14:12:38.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:38 smithi195 ceph-mon[33205]: pgmap v91: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:38.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:38 smithi195 ceph-mon[33205]: from='client.? ' entity='client.admin' 2024-09-13T14:12:38.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:38 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:12:38.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:38 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:12:38.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:38 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:12:38.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:38 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:12:38.523 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:12:38.645 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:38 smithi102 ceph-mon[40543]: pgmap v91: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:38.645 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:38 smithi102 ceph-mon[40543]: from='client.? ' entity='client.admin' 2024-09-13T14:12:38.645 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:38 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:12:38.645 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:38 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:12:38.645 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:38 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:12:38.645 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:38 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:12:38.646 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:38 smithi102 ceph-mon[29355]: pgmap v91: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:38.646 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:38 smithi102 ceph-mon[29355]: from='client.? ' entity='client.admin' 2024-09-13T14:12:38.646 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:38 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:12:38.646 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:38 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:12:38.646 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:38 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:12:38.646 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:38 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:12:39.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:39 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:12:39.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:39 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:12:39.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:39 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:12:40.379 DEBUG:teuthology.orchestra.run.smithi102:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-13T14:12:40.454 INFO:tasks.workunit.client.0.smithi102.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-13T14:12:40.456 INFO:tasks.workunit.client.0.smithi102.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-13T14:12:40.456 INFO:tasks.workunit.client.0.smithi102.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-13T14:12:40.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:40 smithi195 ceph-mon[33205]: pgmap v92: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:40.510 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:40 smithi102 ceph-mon[29355]: pgmap v92: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:40.511 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:40 smithi102 ceph-mon[40543]: pgmap v92: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:42.088 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'echo wait for mgr daemons to upgrade' 2024-09-13T14:12:42.264 INFO:tasks.workunit.client.0.smithi102.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-13T14:12:42.299 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:12:42.340 INFO:tasks.workunit.client.0.smithi102.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-13T14:12:42.379 INFO:tasks.workunit.client.0.smithi102.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-13T14:12:42.381 INFO:tasks.workunit.client.0.smithi102.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-13T14:12:42.381 INFO:tasks.workunit.client.0.smithi102.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-13T14:12:42.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:42 smithi102 ceph-mon[29355]: pgmap v93: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:42.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:42 smithi102 ceph-mon[40543]: pgmap v93: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:42.431 INFO:tasks.workunit.client.0.smithi102.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-13T14:12:42.434 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-09-13T14:12:42.434 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-13T14:12:42.450 INFO:tasks.workunit:Running workunits matching rbd/import_export.sh on client.0... 2024-09-13T14:12:42.451 INFO:tasks.workunit:Running workunit rbd/import_export.sh... 2024-09-13T14:12:42.451 DEBUG:teuthology.orchestra.run.smithi102:workunit test rbd/import_export.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=reef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_CREATE_ARGS=--new-format adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/import_export.sh 2024-09-13T14:12:42.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:42 smithi195 ceph-mon[33205]: pgmap v93: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:42.518 INFO:tasks.workunit.client.0.smithi102.stderr:+ export RBD_FORCE_ALLOW_V1=1 2024-09-13T14:12:42.518 INFO:tasks.workunit.client.0.smithi102.stderr:+ RBD_FORCE_ALLOW_V1=1 2024-09-13T14:12:42.518 INFO:tasks.workunit.client.0.smithi102.stderr:+ TMPDIR=/tmp/rbd_import_export_84613 2024-09-13T14:12:42.518 INFO:tasks.workunit.client.0.smithi102.stderr:+ rm -rf /tmp/rbd_import_export_84613 2024-09-13T14:12:42.519 INFO:tasks.workunit.client.0.smithi102.stderr:+ mkdir /tmp/rbd_import_export_84613 2024-09-13T14:12:42.520 INFO:tasks.workunit.client.0.smithi102.stderr:+ trap 'rm -rf /tmp/rbd_import_export_84613' INT TERM EXIT 2024-09-13T14:12:42.520 INFO:tasks.workunit.client.0.smithi102.stderr:+ mkdir foo.84613 2024-09-13T14:12:42.520 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd import foo.84613 foo.dir 2024-09-13T14:12:42.624 INFO:tasks.workunit.client.0.smithi102.stderr:rbd: cannot import a directory 2024-09-13T14:12:42.624 INFO:tasks.workunit.client.0.smithi102.stderr: Importing image: 0% complete...failed. 2024-09-13T14:12:42.624 INFO:tasks.workunit.client.0.smithi102.stderr:rbd: import failed: (21) Is a directory 2024-09-13T14:12:42.629 INFO:tasks.workunit.client.0.smithi102.stderr:+ true 2024-09-13T14:12:42.629 INFO:tasks.workunit.client.0.smithi102.stderr:+ rmdir foo.84613 2024-09-13T14:12:42.638 INFO:tasks.workunit.client.0.smithi102.stderr:+ dd if=/bin/sh of=/tmp/rbd_import_export_84613/img bs=1k count=1 seek=10 2024-09-13T14:12:42.639 INFO:tasks.workunit.client.0.smithi102.stderr:1+0 records in 2024-09-13T14:12:42.639 INFO:tasks.workunit.client.0.smithi102.stderr:1+0 records out 2024-09-13T14:12:42.639 INFO:tasks.workunit.client.0.smithi102.stderr:1024 bytes (1.0 kB, 1.0 KiB) copied, 4.6093e-05 s, 22.2 MB/s 2024-09-13T14:12:42.639 INFO:tasks.workunit.client.0.smithi102.stderr:+ dd if=/bin/dd of=/tmp/rbd_import_export_84613/img bs=1k count=10 seek=100 2024-09-13T14:12:42.640 INFO:tasks.workunit.client.0.smithi102.stderr:10+0 records in 2024-09-13T14:12:42.640 INFO:tasks.workunit.client.0.smithi102.stderr:10+0 records out 2024-09-13T14:12:42.640 INFO:tasks.workunit.client.0.smithi102.stderr:10240 bytes (10 kB, 10 KiB) copied, 5.9367e-05 s, 172 MB/s 2024-09-13T14:12:42.640 INFO:tasks.workunit.client.0.smithi102.stderr:+ dd if=/bin/rm of=/tmp/rbd_import_export_84613/img bs=1k count=100 seek=1000 2024-09-13T14:12:42.641 INFO:tasks.workunit.client.0.smithi102.stderr:60+1 records in 2024-09-13T14:12:42.641 INFO:tasks.workunit.client.0.smithi102.stderr:60+1 records out 2024-09-13T14:12:42.641 INFO:tasks.workunit.client.0.smithi102.stderr:61640 bytes (62 kB, 60 KiB) copied, 0.00014876 s, 414 MB/s 2024-09-13T14:12:42.641 INFO:tasks.workunit.client.0.smithi102.stderr:+ dd if=/bin/ls of=/tmp/rbd_import_export_84613/img bs=1k seek=10000 2024-09-13T14:12:42.642 INFO:tasks.workunit.client.0.smithi102.stderr:137+1 records in 2024-09-13T14:12:42.642 INFO:tasks.workunit.client.0.smithi102.stderr:137+1 records out 2024-09-13T14:12:42.642 INFO:tasks.workunit.client.0.smithi102.stderr:140928 bytes (141 kB, 138 KiB) copied, 0.000476365 s, 296 MB/s 2024-09-13T14:12:42.642 INFO:tasks.workunit.client.0.smithi102.stderr:+ dd if=/bin/ln of=/tmp/rbd_import_export_84613/img bs=1k seek=100000 2024-09-13T14:12:42.643 INFO:tasks.workunit.client.0.smithi102.stderr:60+1 records in 2024-09-13T14:12:42.643 INFO:tasks.workunit.client.0.smithi102.stderr:60+1 records out 2024-09-13T14:12:42.643 INFO:tasks.workunit.client.0.smithi102.stderr:61592 bytes (62 kB, 60 KiB) copied, 0.000154867 s, 398 MB/s 2024-09-13T14:12:42.643 INFO:tasks.workunit.client.0.smithi102.stderr:+ dd if=/bin/grep of=/tmp/rbd_import_export_84613/img bs=1k seek=1000000 2024-09-13T14:12:42.644 INFO:tasks.workunit.client.0.smithi102.stderr:154+1 records in 2024-09-13T14:12:42.644 INFO:tasks.workunit.client.0.smithi102.stderr:154+1 records out 2024-09-13T14:12:42.644 INFO:tasks.workunit.client.0.smithi102.stderr:158176 bytes (158 kB, 154 KiB) copied, 0.00034393 s, 460 MB/s 2024-09-13T14:12:42.644 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd rm testimg 2024-09-13T14:12:42.684 INFO:tasks.workunit.client.0.smithi102.stderr: Removing image: 0% complete...failed. 2024-09-13T14:12:42.684 INFO:tasks.workunit.client.0.smithi102.stderr:rbd: delete error: (2) No such file or directory 2024-09-13T14:12:42.687 INFO:tasks.workunit.client.0.smithi102.stderr:+ true 2024-09-13T14:12:42.687 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd import --new-format /tmp/rbd_import_export_84613/img testimg 2024-09-13T14:12:43.202 INFO:tasks.workunit.client.0.smithi102.stderr: Importing image: 1% complete... Importing image: 2% complete... Importing image: 3% complete... Importing image: 4% complete... Importing image: 5% complete... Importing image: 6% complete... Importing image: 7% complete... Importing image: 8% complete... Importing image: 9% complete... Importing image: 10% complete... Importing image: 11% complete... Importing image: 12% complete... Importing image: 13% complete... Importing image: 14% complete... Importing image: 15% complete... Importing image: 16% complete... Importing image: 17% complete... Importing image: 18% complete... Importing image: 19% complete... Importing image: 20% complete... Importing image: 21% complete... Importing image: 22% complete... Importing image: 23% complete... Importing image: 24% complete... Importing image: 25% complete... Importing image: 26% complete... Importing image: 27% complete... Importing image: 28% complete... Importing image: 29% complete... Importing image: 30% complete... Importing image: 31% complete... Importing image: 32% complete... Importing image: 33% complete... Importing image: 34% complete... Importing image: 35% complete... Importing image: 36% complete... Importing image: 37% complete... Importing image: 38% complete... Importing image: 39% complete... Importing image: 40% complete... Importing image: 41% complete... Importing image: 42% complete... Importing image: 43% complete... Importing image: 44% complete... Importing image: 45% complete... Importing image: 46% complete... Importing image: 47% complete... Importing image: 48% complete... Importing image: 49% complete... Importing image: 50% complete... Importing image: 51% complete... Importing image: 52% complete... Importing image: 53% complete... Importing image: 54% complete... Importing image: 55% complete... Importing image: 56% complete... Importing image: 57% complete... Importing image: 58% complete... Importing image: 59% complete... Importing image: 60% complete... Importing image: 61% complete... Importing image: 62% complete... Importing image: 63% complete... Importing image: 64% complete... Importing image: 65% complete... Importing image: 66% complete... Importing image: 67% complete... Importing image: 68% complete... Importing image: 69% complete... Importing image: 70% complete... Importing image: 71% complete... Importing image: 72% complete... Importing image: 73% complete... Importing image: 74% complete... Importing image: 75% complete... Importing image: 76% complete... Importing image: 77% complete... Importing image: 78% complete... Importing image: 79% complete... Importing image: 80% complete... Importing image: 81% complete... Importing image: 82% complete... Importing image: 83% complete... Importing image: 84% complete... Importing image: 85% complete... Importing image: 86% complete... Importing image: 87% complete... Importing image: 88% complete... Importing image: 89% complete... Importing image: 90% complete... Importing image: 91% complete... Importing image: 92% complete... Importing image: 93% complete... Importing image: 94% complete... Importing image: 95% complete... Importing image: 96% complete... Importing image: 97% complete... Importing image: 98% complete... Importing image: 99% complete... Importing image: 100% complete...done. 2024-09-13T14:12:43.202 INFO:tasks.workunit.client.0.smithi102.stderr:rbd: --new-format is deprecated, use --image-format 2 2024-09-13T14:12:43.205 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd export testimg /tmp/rbd_import_export_84613/img2 2024-09-13T14:12:43.416 INFO:tasks.workunit.client.0.smithi102.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T14:12:43.421 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd export testimg - 2024-09-13T14:12:44.052 INFO:tasks.workunit.client.0.smithi102.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T14:12:44.055 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd rm testimg 2024-09-13T14:12:44.143 INFO:tasks.workunit.client.0.smithi102.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-13T14:12:44.146 INFO:tasks.workunit.client.0.smithi102.stderr:+ cmp /tmp/rbd_import_export_84613/img /tmp/rbd_import_export_84613/img2 2024-09-13T14:12:44.269 INFO:teuthology.orchestra.run.smithi102.stdout:wait for mgr daemons to upgrade 2024-09-13T14:12:44.270 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:44 smithi102 ceph-mon[29355]: pgmap v94: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:44.270 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:44 smithi102 ceph-mon[40543]: pgmap v94: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:44.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:44 smithi195 ceph-mon[33205]: pgmap v94: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:12:44.697 INFO:tasks.workunit.client.0.smithi102.stderr:+ cmp /tmp/rbd_import_export_84613/img /tmp/rbd_import_export_84613/img3 2024-09-13T14:12:44.999 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr' 2024-09-13T14:12:45.100 INFO:tasks.workunit.client.0.smithi102.stderr:+ rm /tmp/rbd_import_export_84613/img2 /tmp/rbd_import_export_84613/img3 2024-09-13T14:12:45.250 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:12:45.281 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd import --new-format - testimg 2024-09-13T14:12:45.649 INFO:tasks.workunit.client.0.smithi102.stderr: Importing image: 100% complete...done. 2024-09-13T14:12:45.650 INFO:tasks.workunit.client.0.smithi102.stderr:rbd: --new-format is deprecated, use --image-format 2 2024-09-13T14:12:45.653 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd export testimg /tmp/rbd_import_export_84613/img2 2024-09-13T14:12:45.855 INFO:tasks.workunit.client.0.smithi102.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T14:12:45.860 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd export testimg - 2024-09-13T14:12:46.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:46 smithi102 ceph-mon[29355]: pgmap v95: 73 pgs: 73 active+clean; 579 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 38 KiB/s wr, 25 op/s 2024-09-13T14:12:46.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:46 smithi102 ceph-mon[40543]: pgmap v95: 73 pgs: 73 active+clean; 579 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 38 KiB/s wr, 25 op/s 2024-09-13T14:12:46.484 INFO:tasks.workunit.client.0.smithi102.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T14:12:46.488 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd rm testimg 2024-09-13T14:12:46.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:46 smithi195 ceph-mon[33205]: pgmap v95: 73 pgs: 73 active+clean; 579 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 38 KiB/s wr, 25 op/s 2024-09-13T14:12:46.575 INFO:tasks.workunit.client.0.smithi102.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-13T14:12:46.579 INFO:tasks.workunit.client.0.smithi102.stderr:+ cmp /tmp/rbd_import_export_84613/img /tmp/rbd_import_export_84613/img2 2024-09-13T14:12:47.132 INFO:tasks.workunit.client.0.smithi102.stderr:+ cmp /tmp/rbd_import_export_84613/img /tmp/rbd_import_export_84613/img3 2024-09-13T14:12:47.418 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:12:47 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:12:47] "GET /metrics HTTP/1.1" 200 33720 "" "Prometheus/2.43.0" 2024-09-13T14:12:47.549 INFO:tasks.workunit.client.0.smithi102.stderr:+ rm /tmp/rbd_import_export_84613/img /tmp/rbd_import_export_84613/img2 /tmp/rbd_import_export_84613/img3 2024-09-13T14:12:47.774 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd help export 2024-09-13T14:12:47.775 INFO:tasks.workunit.client.0.smithi102.stderr:+ grep -q export-format 2024-09-13T14:12:47.798 INFO:tasks.workunit.client.0.smithi102.stderr:+ dd if=/bin/dd of=/tmp/rbd_import_export_84613/img bs=1k count=10 seek=100 2024-09-13T14:12:47.799 INFO:tasks.workunit.client.0.smithi102.stderr:10+0 records in 2024-09-13T14:12:47.799 INFO:tasks.workunit.client.0.smithi102.stderr:10+0 records out 2024-09-13T14:12:47.799 INFO:tasks.workunit.client.0.smithi102.stderr:10240 bytes (10 kB, 10 KiB) copied, 6.186e-05 s, 166 MB/s 2024-09-13T14:12:47.799 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd import --new-format /tmp/rbd_import_export_84613/img testimg 2024-09-13T14:12:47.874 INFO:tasks.workunit.client.0.smithi102.stderr: Importing image: 100% complete...done. 2024-09-13T14:12:47.875 INFO:tasks.workunit.client.0.smithi102.stderr:rbd: --new-format is deprecated, use --image-format 2 2024-09-13T14:12:47.878 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd snap create testimg@snap 2024-09-13T14:12:48.232 INFO:tasks.workunit.client.0.smithi102.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-13T14:12:48.238 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd image-meta set testimg key1 value1 2024-09-13T14:12:48.293 INFO:tasks.workunit.client.0.smithi102.stderr:++ rbd image-meta list testimg 2024-09-13T14:12:48.339 INFO:tasks.workunit.client.0.smithi102.stderr:+ IMAGEMETA_BEFORE='There is 1 metadatum on this image: 2024-09-13T14:12:48.339 INFO:tasks.workunit.client.0.smithi102.stderr: 2024-09-13T14:12:48.339 INFO:tasks.workunit.client.0.smithi102.stderr:Key Value 2024-09-13T14:12:48.339 INFO:tasks.workunit.client.0.smithi102.stderr:key1 value1' 2024-09-13T14:12:48.339 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd export --export-format 2 testimg /tmp/rbd_import_export_84613/img_v2 2024-09-13T14:12:48.383 INFO:tasks.workunit.client.0.smithi102.stderr: Exporting image: 100% complete... Exporting image: 100% complete...done. 2024-09-13T14:12:48.386 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd import --export-format 2 /tmp/rbd_import_export_84613/img_v2 testimg_import 2024-09-13T14:12:48.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:48 smithi195 ceph-mon[33205]: pgmap v96: 73 pgs: 73 active+clean; 579 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 38 KiB/s wr, 25 op/s 2024-09-13T14:12:48.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:48 smithi195 ceph-mon[33205]: from='client.24737 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:12:48.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:48 smithi102 ceph-mon[29355]: pgmap v96: 73 pgs: 73 active+clean; 579 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 38 KiB/s wr, 25 op/s 2024-09-13T14:12:48.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:48 smithi102 ceph-mon[29355]: from='client.24737 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:12:48.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:48 smithi102 ceph-mon[40543]: pgmap v96: 73 pgs: 73 active+clean; 579 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 38 KiB/s wr, 25 op/s 2024-09-13T14:12:48.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:48 smithi102 ceph-mon[40543]: from='client.24737 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:12:49.249 INFO:tasks.workunit.client.0.smithi102.stderr: Importing image: 50% complete... Importing image: 100% complete... Importing image: 100% complete...done. 2024-09-13T14:12:49.253 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd info testimg_import 2024-09-13T14:12:49.291 INFO:tasks.workunit.client.0.smithi102.stdout:rbd image 'testimg_import': 2024-09-13T14:12:49.291 INFO:tasks.workunit.client.0.smithi102.stdout: size 110 KiB in 1 objects 2024-09-13T14:12:49.292 INFO:tasks.workunit.client.0.smithi102.stdout: order 22 (4 MiB objects) 2024-09-13T14:12:49.292 INFO:tasks.workunit.client.0.smithi102.stdout: snapshot_count: 1 2024-09-13T14:12:49.292 INFO:tasks.workunit.client.0.smithi102.stdout: id: 3a2079754d25 2024-09-13T14:12:49.292 INFO:tasks.workunit.client.0.smithi102.stdout: block_name_prefix: rbd_data.3a2079754d25 2024-09-13T14:12:49.292 INFO:tasks.workunit.client.0.smithi102.stdout: format: 2 2024-09-13T14:12:49.292 INFO:tasks.workunit.client.0.smithi102.stdout: features: layering, exclusive-lock, object-map, fast-diff, deep-flatten 2024-09-13T14:12:49.293 INFO:tasks.workunit.client.0.smithi102.stdout: op_features: 2024-09-13T14:12:49.293 INFO:tasks.workunit.client.0.smithi102.stdout: flags: 2024-09-13T14:12:49.293 INFO:tasks.workunit.client.0.smithi102.stdout: create_timestamp: Fri Sep 13 14:12:48 2024 2024-09-13T14:12:49.293 INFO:tasks.workunit.client.0.smithi102.stdout: access_timestamp: Fri Sep 13 14:12:48 2024 2024-09-13T14:12:49.293 INFO:tasks.workunit.client.0.smithi102.stdout: modify_timestamp: Fri Sep 13 14:12:48 2024 2024-09-13T14:12:49.294 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd info testimg_import@snap 2024-09-13T14:12:49.334 INFO:tasks.workunit.client.0.smithi102.stdout:rbd image 'testimg_import': 2024-09-13T14:12:49.334 INFO:tasks.workunit.client.0.smithi102.stdout: size 110 KiB in 1 objects 2024-09-13T14:12:49.334 INFO:tasks.workunit.client.0.smithi102.stdout: order 22 (4 MiB objects) 2024-09-13T14:12:49.334 INFO:tasks.workunit.client.0.smithi102.stdout: snapshot_count: 1 2024-09-13T14:12:49.334 INFO:tasks.workunit.client.0.smithi102.stdout: id: 3a2079754d25 2024-09-13T14:12:49.334 INFO:tasks.workunit.client.0.smithi102.stdout: block_name_prefix: rbd_data.3a2079754d25 2024-09-13T14:12:49.334 INFO:tasks.workunit.client.0.smithi102.stdout: format: 2 2024-09-13T14:12:49.334 INFO:tasks.workunit.client.0.smithi102.stdout: features: layering, exclusive-lock, object-map, fast-diff, deep-flatten 2024-09-13T14:12:49.334 INFO:tasks.workunit.client.0.smithi102.stdout: op_features: 2024-09-13T14:12:49.334 INFO:tasks.workunit.client.0.smithi102.stdout: flags: 2024-09-13T14:12:49.334 INFO:tasks.workunit.client.0.smithi102.stdout: create_timestamp: Fri Sep 13 14:12:48 2024 2024-09-13T14:12:49.335 INFO:tasks.workunit.client.0.smithi102.stdout: access_timestamp: Fri Sep 13 14:12:48 2024 2024-09-13T14:12:49.335 INFO:tasks.workunit.client.0.smithi102.stdout: modify_timestamp: Fri Sep 13 14:12:48 2024 2024-09-13T14:12:49.335 INFO:tasks.workunit.client.0.smithi102.stdout: protected: False 2024-09-13T14:12:49.338 INFO:tasks.workunit.client.0.smithi102.stderr:++ rbd image-meta list testimg_import 2024-09-13T14:12:49.386 INFO:tasks.workunit.client.0.smithi102.stderr:+ IMAGEMETA_AFTER='There is 1 metadatum on this image: 2024-09-13T14:12:49.386 INFO:tasks.workunit.client.0.smithi102.stderr: 2024-09-13T14:12:49.386 INFO:tasks.workunit.client.0.smithi102.stderr:Key Value 2024-09-13T14:12:49.387 INFO:tasks.workunit.client.0.smithi102.stderr:key1 value1' 2024-09-13T14:12:49.387 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 'There is 1 metadatum on this image: 2024-09-13T14:12:49.387 INFO:tasks.workunit.client.0.smithi102.stderr: 2024-09-13T14:12:49.387 INFO:tasks.workunit.client.0.smithi102.stderr:Key Value 2024-09-13T14:12:49.387 INFO:tasks.workunit.client.0.smithi102.stderr:key1 value1' = 'There is 1 metadatum on this image: 2024-09-13T14:12:49.387 INFO:tasks.workunit.client.0.smithi102.stderr: 2024-09-13T14:12:49.387 INFO:tasks.workunit.client.0.smithi102.stderr:Key Value 2024-09-13T14:12:49.387 INFO:tasks.workunit.client.0.smithi102.stderr:key1 value1' ']' 2024-09-13T14:12:49.387 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd export testimg_import /tmp/rbd_import_export_84613/img_import 2024-09-13T14:12:49.430 INFO:tasks.workunit.client.0.smithi102.stderr: Exporting image: 100% complete...done. 2024-09-13T14:12:49.433 INFO:tasks.workunit.client.0.smithi102.stderr:+ compare_files_and_ondisk_sizes /tmp/rbd_import_export_84613/img /tmp/rbd_import_export_84613/img_import 2024-09-13T14:12:49.434 INFO:tasks.workunit.client.0.smithi102.stderr:+ cmp -l /tmp/rbd_import_export_84613/img /tmp/rbd_import_export_84613/img_import 2024-09-13T14:12:49.434 INFO:tasks.workunit.client.0.smithi102.stderr:++ stat /tmp/rbd_import_export_84613/img --format %b 2024-09-13T14:12:49.436 INFO:tasks.workunit.client.0.smithi102.stderr:+ origsize=24 2024-09-13T14:12:49.436 INFO:tasks.workunit.client.0.smithi102.stderr:++ stat /tmp/rbd_import_export_84613/img_import --format %b 2024-09-13T14:12:49.437 INFO:tasks.workunit.client.0.smithi102.stderr:+ exportsize=224 2024-09-13T14:12:49.437 INFO:tasks.workunit.client.0.smithi102.stderr:+ difference=200 2024-09-13T14:12:49.437 INFO:tasks.workunit.client.0.smithi102.stderr:+ difference=200 2024-09-13T14:12:49.437 INFO:tasks.workunit.client.0.smithi102.stderr:+ test 200 -ge 0 -a 200 -lt 4096 2024-09-13T14:12:49.437 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd export testimg@snap /tmp/rbd_import_export_84613/img_snap 2024-09-13T14:12:49.481 INFO:tasks.workunit.client.0.smithi102.stderr: Exporting image: 100% complete...done. 2024-09-13T14:12:49.484 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd export testimg_import@snap /tmp/rbd_import_export_84613/img_snap_import 2024-09-13T14:12:49.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:49 smithi195 ceph-mon[33205]: osdmap e74: 8 total, 8 up, 8 in 2024-09-13T14:12:49.529 INFO:tasks.workunit.client.0.smithi102.stderr: Exporting image: 100% complete...done. 2024-09-13T14:12:49.532 INFO:tasks.workunit.client.0.smithi102.stderr:+ compare_files_and_ondisk_sizes /tmp/rbd_import_export_84613/img_snap /tmp/rbd_import_export_84613/img_snap_import 2024-09-13T14:12:49.532 INFO:tasks.workunit.client.0.smithi102.stderr:+ cmp -l /tmp/rbd_import_export_84613/img_snap /tmp/rbd_import_export_84613/img_snap_import 2024-09-13T14:12:49.533 INFO:tasks.workunit.client.0.smithi102.stderr:++ stat /tmp/rbd_import_export_84613/img_snap --format %b 2024-09-13T14:12:49.534 INFO:tasks.workunit.client.0.smithi102.stderr:+ origsize=224 2024-09-13T14:12:49.535 INFO:tasks.workunit.client.0.smithi102.stderr:++ stat /tmp/rbd_import_export_84613/img_snap_import --format %b 2024-09-13T14:12:49.536 INFO:tasks.workunit.client.0.smithi102.stderr:+ exportsize=224 2024-09-13T14:12:49.536 INFO:tasks.workunit.client.0.smithi102.stderr:+ difference=0 2024-09-13T14:12:49.536 INFO:tasks.workunit.client.0.smithi102.stderr:+ difference=0 2024-09-13T14:12:49.536 INFO:tasks.workunit.client.0.smithi102.stderr:+ test 0 -ge 0 -a 0 -lt 4096 2024-09-13T14:12:49.536 INFO:tasks.workunit.client.0.smithi102.stderr:+ rm /tmp/rbd_import_export_84613/img_v2 2024-09-13T14:12:49.537 INFO:tasks.workunit.client.0.smithi102.stderr:+ rm /tmp/rbd_import_export_84613/img_import 2024-09-13T14:12:49.537 INFO:tasks.workunit.client.0.smithi102.stderr:+ rm /tmp/rbd_import_export_84613/img_snap 2024-09-13T14:12:49.538 INFO:tasks.workunit.client.0.smithi102.stderr:+ rm /tmp/rbd_import_export_84613/img_snap_import 2024-09-13T14:12:49.539 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd snap rm testimg_import@snap 2024-09-13T14:12:49.617 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:49 smithi102 ceph-mon[29355]: osdmap e74: 8 total, 8 up, 8 in 2024-09-13T14:12:49.617 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:49 smithi102 ceph-mon[40543]: osdmap e74: 8 total, 8 up, 8 in 2024-09-13T14:12:50.244 INFO:tasks.workunit.client.0.smithi102.stderr: Removing snap: 100% complete...done. 2024-09-13T14:12:50.251 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd remove testimg_import 2024-09-13T14:12:50.351 INFO:tasks.workunit.client.0.smithi102.stderr: Removing image: 100% complete...done. 2024-09-13T14:12:50.354 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd snap rm testimg@snap 2024-09-13T14:12:50.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:50 smithi195 ceph-mon[33205]: pgmap v98: 73 pgs: 73 active+clean; 579 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 46 KiB/s wr, 30 op/s 2024-09-13T14:12:50.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:50 smithi195 ceph-mon[33205]: osdmap e75: 8 total, 8 up, 8 in 2024-09-13T14:12:50.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:50 smithi102 ceph-mon[29355]: pgmap v98: 73 pgs: 73 active+clean; 579 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 46 KiB/s wr, 30 op/s 2024-09-13T14:12:50.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:50 smithi102 ceph-mon[29355]: osdmap e75: 8 total, 8 up, 8 in 2024-09-13T14:12:50.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:50 smithi102 ceph-mon[40543]: pgmap v98: 73 pgs: 73 active+clean; 579 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 46 KiB/s wr, 30 op/s 2024-09-13T14:12:50.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:50 smithi102 ceph-mon[40543]: osdmap e75: 8 total, 8 up, 8 in 2024-09-13T14:12:51.252 INFO:tasks.workunit.client.0.smithi102.stderr: Removing snap: 100% complete...done. 2024-09-13T14:12:51.259 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd rm testimg 2024-09-13T14:12:51.352 INFO:tasks.workunit.client.0.smithi102.stderr: Removing image: 100% complete...done. 2024-09-13T14:12:51.355 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd import --order 20 /tmp/rbd_import_export_84613/img testimg 2024-09-13T14:12:51.417 INFO:tasks.workunit.client.0.smithi102.stderr: Importing image: 100% complete...done. 2024-09-13T14:12:51.417 INFO:tasks.workunit.client.0.smithi102.stderr:rbd: --order is deprecated, use --object-size 2024-09-13T14:12:51.419 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd export --export-format 2 testimg /tmp/rbd_import_export_84613/img_v2 2024-09-13T14:12:51.462 INFO:tasks.workunit.client.0.smithi102.stderr: Exporting image: 100% complete... Exporting image: 100% complete...done. 2024-09-13T14:12:51.465 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd import --export-format 2 /tmp/rbd_import_export_84613/img_v2 testimg_import 2024-09-13T14:12:51.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:51 smithi195 ceph-mon[33205]: osdmap e76: 8 total, 8 up, 8 in 2024-09-13T14:12:51.528 INFO:tasks.workunit.client.0.smithi102.stderr: Importing image: 100% complete... Importing image: 100% complete...done. 2024-09-13T14:12:51.532 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd info testimg_import 2024-09-13T14:12:51.532 INFO:tasks.workunit.client.0.smithi102.stderr:+ grep order 2024-09-13T14:12:51.532 INFO:tasks.workunit.client.0.smithi102.stderr:+ awk '{print $2}' 2024-09-13T14:12:51.532 INFO:tasks.workunit.client.0.smithi102.stderr:+ grep 20 2024-09-13T14:12:51.574 INFO:tasks.workunit.client.0.smithi102.stdout:20 2024-09-13T14:12:51.574 INFO:tasks.workunit.client.0.smithi102.stderr:+ rm /tmp/rbd_import_export_84613/img_v2 2024-09-13T14:12:51.575 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd remove testimg_import 2024-09-13T14:12:51.661 INFO:tasks.workunit.client.0.smithi102.stderr: Removing image: 100% complete...done. 2024-09-13T14:12:51.664 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd remove testimg 2024-09-13T14:12:51.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:51 smithi102 ceph-mon[29355]: osdmap e76: 8 total, 8 up, 8 in 2024-09-13T14:12:51.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:51 smithi102 ceph-mon[40543]: osdmap e76: 8 total, 8 up, 8 in 2024-09-13T14:12:51.754 INFO:tasks.workunit.client.0.smithi102.stderr: Removing image: 100% complete...done. 2024-09-13T14:12:51.758 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd import --image-feature layering /tmp/rbd_import_export_84613/img testimg 2024-09-13T14:12:51.809 INFO:tasks.workunit.client.0.smithi102.stderr: Importing image: 100% complete...done. 2024-09-13T14:12:51.813 INFO:tasks.workunit.client.0.smithi102.stderr:++ rbd info testimg 2024-09-13T14:12:51.814 INFO:tasks.workunit.client.0.smithi102.stderr:++ grep features 2024-09-13T14:12:51.855 INFO:tasks.workunit.client.0.smithi102.stderr:+ FEATURES_BEFORE=' features: layering 2024-09-13T14:12:51.855 INFO:tasks.workunit.client.0.smithi102.stderr: op_features: ' 2024-09-13T14:12:51.855 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd export --export-format 2 testimg /tmp/rbd_import_export_84613/img_v2 2024-09-13T14:12:51.897 INFO:tasks.workunit.client.0.smithi102.stderr: Exporting image: 100% complete... Exporting image: 100% complete...done. 2024-09-13T14:12:51.900 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd import --export-format 2 /tmp/rbd_import_export_84613/img_v2 testimg_import 2024-09-13T14:12:51.958 INFO:tasks.workunit.client.0.smithi102.stderr: Importing image: 100% complete... Importing image: 100% complete...done. 2024-09-13T14:12:51.962 INFO:tasks.workunit.client.0.smithi102.stderr:++ rbd info testimg_import 2024-09-13T14:12:51.962 INFO:tasks.workunit.client.0.smithi102.stderr:++ grep features 2024-09-13T14:12:52.003 INFO:tasks.workunit.client.0.smithi102.stderr:+ FEATURES_AFTER=' features: layering 2024-09-13T14:12:52.004 INFO:tasks.workunit.client.0.smithi102.stderr: op_features: ' 2024-09-13T14:12:52.004 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' ' features: layering 2024-09-13T14:12:52.004 INFO:tasks.workunit.client.0.smithi102.stderr: op_features: ' '!=' ' features: layering 2024-09-13T14:12:52.004 INFO:tasks.workunit.client.0.smithi102.stderr: op_features: ' ']' 2024-09-13T14:12:52.004 INFO:tasks.workunit.client.0.smithi102.stderr:+ rm /tmp/rbd_import_export_84613/img_v2 2024-09-13T14:12:52.005 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd remove testimg_import 2024-09-13T14:12:52.074 INFO:tasks.workunit.client.0.smithi102.stderr: Removing image: 100% complete...done. 2024-09-13T14:12:52.077 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd remove testimg 2024-09-13T14:12:52.152 INFO:tasks.workunit.client.0.smithi102.stderr: Removing image: 100% complete...done. 2024-09-13T14:12:52.155 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd import --stripe-count 1000 --stripe-unit 4096 /tmp/rbd_import_export_84613/img testimg 2024-09-13T14:12:52.220 INFO:tasks.workunit.client.0.smithi102.stderr: Importing image: 100% complete...done. 2024-09-13T14:12:52.223 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd export --export-format 2 testimg /tmp/rbd_import_export_84613/img_v2 2024-09-13T14:12:52.268 INFO:tasks.workunit.client.0.smithi102.stderr: Exporting image: 100% complete... Exporting image: 100% complete...done. 2024-09-13T14:12:52.271 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd import --export-format 2 /tmp/rbd_import_export_84613/img_v2 testimg_import 2024-09-13T14:12:52.336 INFO:tasks.workunit.client.0.smithi102.stderr: Importing image: 100% complete... Importing image: 100% complete...done. 2024-09-13T14:12:52.340 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd info testimg_import 2024-09-13T14:12:52.340 INFO:tasks.workunit.client.0.smithi102.stderr:+ grep 'stripe unit' 2024-09-13T14:12:52.340 INFO:tasks.workunit.client.0.smithi102.stderr:+ grep -Ei '(4 KiB|4096)' 2024-09-13T14:12:52.382 INFO:tasks.workunit.client.0.smithi102.stdout: stripe unit: 4 KiB 2024-09-13T14:12:52.383 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd info testimg_import 2024-09-13T14:12:52.383 INFO:tasks.workunit.client.0.smithi102.stderr:+ grep 'stripe count' 2024-09-13T14:12:52.383 INFO:tasks.workunit.client.0.smithi102.stderr:+ awk '{print $3}' 2024-09-13T14:12:52.383 INFO:tasks.workunit.client.0.smithi102.stderr:+ grep 1000 2024-09-13T14:12:52.422 INFO:tasks.workunit.client.0.smithi102.stdout:1000 2024-09-13T14:12:52.422 INFO:tasks.workunit.client.0.smithi102.stderr:+ rm /tmp/rbd_import_export_84613/img_v2 2024-09-13T14:12:52.422 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd remove testimg_import 2024-09-13T14:12:52.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:52 smithi195 ceph-mon[33205]: pgmap v101: 73 pgs: 73 active+clean; 799 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 294 KiB/s rd, 87 KiB/s wr, 192 op/s 2024-09-13T14:12:52.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:52 smithi195 ceph-mon[33205]: osdmap e77: 8 total, 8 up, 8 in 2024-09-13T14:12:52.512 INFO:tasks.workunit.client.0.smithi102.stderr: Removing image: 3% complete... Removing image: 7% complete... Removing image: 10% complete... Removing image: 14% complete... Removing image: 17% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 32% complete... Removing image: 35% complete... Removing image: 39% complete... Removing image: 42% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 57% complete... Removing image: 60% complete... Removing image: 64% complete... Removing image: 67% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 82% complete... Removing image: 85% complete... Removing image: 89% complete... Removing image: 92% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T14:12:52.515 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd remove testimg 2024-09-13T14:12:52.606 INFO:tasks.workunit.client.0.smithi102.stderr: Removing image: 3% complete... Removing image: 7% complete... Removing image: 10% complete... Removing image: 14% complete... Removing image: 17% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 32% complete... Removing image: 35% complete... Removing image: 39% complete... Removing image: 42% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 57% complete... Removing image: 60% complete... Removing image: 64% complete... Removing image: 67% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 82% complete... Removing image: 85% complete... Removing image: 89% complete... Removing image: 92% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T14:12:52.609 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd import --image-format=2 /tmp/rbd_import_export_84613/img testimg 2024-09-13T14:12:52.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:52 smithi102 ceph-mon[29355]: pgmap v101: 73 pgs: 73 active+clean; 799 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 294 KiB/s rd, 87 KiB/s wr, 192 op/s 2024-09-13T14:12:52.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:52 smithi102 ceph-mon[29355]: osdmap e77: 8 total, 8 up, 8 in 2024-09-13T14:12:52.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:52 smithi102 ceph-mon[40543]: pgmap v101: 73 pgs: 73 active+clean; 799 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 294 KiB/s rd, 87 KiB/s wr, 192 op/s 2024-09-13T14:12:52.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:52 smithi102 ceph-mon[40543]: osdmap e77: 8 total, 8 up, 8 in 2024-09-13T14:12:52.669 INFO:tasks.workunit.client.0.smithi102.stderr: Importing image: 100% complete...done. 2024-09-13T14:12:52.671 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd snap create testimg@snap1 2024-09-13T14:12:53.263 INFO:tasks.workunit.client.0.smithi102.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-13T14:12:53.269 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd snap create testimg@snap2 2024-09-13T14:12:54.267 INFO:tasks.workunit.client.0.smithi102.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-13T14:12:54.273 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd snap protect testimg@snap2 2024-09-13T14:12:54.320 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd export --export-format 2 testimg /tmp/rbd_import_export_84613/snap_protect 2024-09-13T14:12:54.365 INFO:tasks.workunit.client.0.smithi102.stderr: Exporting image: 33% complete... Exporting image: 100% complete... Exporting image: 100% complete...done. 2024-09-13T14:12:54.369 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd import --export-format 2 /tmp/rbd_import_export_84613/snap_protect testimg_import 2024-09-13T14:12:54.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:54 smithi102 ceph-mon[29355]: pgmap v103: 73 pgs: 73 active+clean; 799 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 363 KiB/s rd, 108 KiB/s wr, 236 op/s 2024-09-13T14:12:54.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:54 smithi102 ceph-mon[29355]: osdmap e78: 8 total, 8 up, 8 in 2024-09-13T14:12:54.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:54 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:12:54.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:54 smithi102 ceph-mon[40543]: pgmap v103: 73 pgs: 73 active+clean; 799 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 363 KiB/s rd, 108 KiB/s wr, 236 op/s 2024-09-13T14:12:54.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:54 smithi102 ceph-mon[40543]: osdmap e78: 8 total, 8 up, 8 in 2024-09-13T14:12:54.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:54 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:12:54.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:54 smithi195 ceph-mon[33205]: pgmap v103: 73 pgs: 73 active+clean; 799 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 363 KiB/s rd, 108 KiB/s wr, 236 op/s 2024-09-13T14:12:54.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:54 smithi195 ceph-mon[33205]: osdmap e78: 8 total, 8 up, 8 in 2024-09-13T14:12:54.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:54 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:12:55.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:55 smithi102 ceph-mon[29355]: osdmap e79: 8 total, 8 up, 8 in 2024-09-13T14:12:55.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:55 smithi102 ceph-mon[40543]: osdmap e79: 8 total, 8 up, 8 in 2024-09-13T14:12:55.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:55 smithi195 ceph-mon[33205]: osdmap e79: 8 total, 8 up, 8 in 2024-09-13T14:12:56.285 INFO:tasks.workunit.client.0.smithi102.stderr: Importing image: 33% complete... Importing image: 66% complete... Importing image: 100% complete... Importing image: 100% complete...done. 2024-09-13T14:12:56.288 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd info testimg_import@snap1 2024-09-13T14:12:56.288 INFO:tasks.workunit.client.0.smithi102.stderr:+ grep 'protected: False' 2024-09-13T14:12:56.332 INFO:tasks.workunit.client.0.smithi102.stdout: protected: False 2024-09-13T14:12:56.332 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd info testimg_import@snap2 2024-09-13T14:12:56.332 INFO:tasks.workunit.client.0.smithi102.stderr:+ grep 'protected: True' 2024-09-13T14:12:56.373 INFO:tasks.workunit.client.0.smithi102.stdout: protected: True 2024-09-13T14:12:56.373 INFO:tasks.workunit.client.0.smithi102.stderr:+ rm /tmp/rbd_import_export_84613/snap_protect 2024-09-13T14:12:56.374 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd snap unprotect testimg@snap2 2024-09-13T14:12:56.425 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd snap unprotect testimg_import@snap2 2024-09-13T14:12:56.473 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd snap purge testimg 2024-09-13T14:12:56.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:56 smithi102 ceph-mon[29355]: pgmap v106: 73 pgs: 73 active+clean; 799 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 345 KiB/s rd, 42 KiB/s wr, 464 op/s 2024-09-13T14:12:56.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:56 smithi102 ceph-mon[29355]: osdmap e80: 8 total, 8 up, 8 in 2024-09-13T14:12:56.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:56 smithi102 ceph-mon[40543]: pgmap v106: 73 pgs: 73 active+clean; 799 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 345 KiB/s rd, 42 KiB/s wr, 464 op/s 2024-09-13T14:12:56.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:56 smithi102 ceph-mon[40543]: osdmap e80: 8 total, 8 up, 8 in 2024-09-13T14:12:56.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:56 smithi195 ceph-mon[33205]: pgmap v106: 73 pgs: 73 active+clean; 799 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 345 KiB/s rd, 42 KiB/s wr, 464 op/s 2024-09-13T14:12:56.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:56 smithi195 ceph-mon[33205]: osdmap e80: 8 total, 8 up, 8 in 2024-09-13T14:12:57.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:57 smithi102 ceph-mon[29355]: osdmap e81: 8 total, 8 up, 8 in 2024-09-13T14:12:57.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:57 smithi102 ceph-mon[40543]: osdmap e81: 8 total, 8 up, 8 in 2024-09-13T14:12:57.669 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:12:57 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:12:57] "GET /metrics HTTP/1.1" 200 33741 "" "Prometheus/2.43.0" 2024-09-13T14:12:57.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:57 smithi195 ceph-mon[33205]: osdmap e81: 8 total, 8 up, 8 in 2024-09-13T14:12:58.062 INFO:tasks.workunit.client.0.smithi102.stderr: Removing all snapshots: 50% complete... Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-09-13T14:12:58.070 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd snap purge testimg_import 2024-09-13T14:12:58.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:58 smithi102 ceph-mon[29355]: pgmap v109: 73 pgs: 73 active+clean; 799 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 418 KiB/s rd, 51 KiB/s wr, 562 op/s 2024-09-13T14:12:58.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:58 smithi102 ceph-mon[29355]: osdmap e82: 8 total, 8 up, 8 in 2024-09-13T14:12:58.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:12:58 smithi102 ceph-mon[29355]: osdmap e83: 8 total, 8 up, 8 in 2024-09-13T14:12:58.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:58 smithi102 ceph-mon[40543]: pgmap v109: 73 pgs: 73 active+clean; 799 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 418 KiB/s rd, 51 KiB/s wr, 562 op/s 2024-09-13T14:12:58.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:58 smithi102 ceph-mon[40543]: osdmap e82: 8 total, 8 up, 8 in 2024-09-13T14:12:58.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:12:58 smithi102 ceph-mon[40543]: osdmap e83: 8 total, 8 up, 8 in 2024-09-13T14:12:58.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:58 smithi195 ceph-mon[33205]: pgmap v109: 73 pgs: 73 active+clean; 799 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 418 KiB/s rd, 51 KiB/s wr, 562 op/s 2024-09-13T14:12:58.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:58 smithi195 ceph-mon[33205]: osdmap e82: 8 total, 8 up, 8 in 2024-09-13T14:12:58.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:12:58 smithi195 ceph-mon[33205]: osdmap e83: 8 total, 8 up, 8 in 2024-09-13T14:13:00.066 INFO:tasks.workunit.client.0.smithi102.stderr: Removing all snapshots: 50% complete... Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-09-13T14:13:00.074 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd remove testimg 2024-09-13T14:13:00.178 INFO:tasks.workunit.client.0.smithi102.stderr: Removing image: 100% complete...done. 2024-09-13T14:13:00.182 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd remove testimg_import 2024-09-13T14:13:00.274 INFO:tasks.workunit.client.0.smithi102.stderr: Removing image: 100% complete...done. 2024-09-13T14:13:00.277 INFO:tasks.workunit.client.0.smithi102.stderr:+ tiered=0 2024-09-13T14:13:00.277 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph osd dump 2024-09-13T14:13:00.277 INFO:tasks.workunit.client.0.smithi102.stderr:+ grep '^pool' 2024-09-13T14:13:00.278 INFO:tasks.workunit.client.0.smithi102.stderr:+ grep ''\''rbd'\''' 2024-09-13T14:13:00.278 INFO:tasks.workunit.client.0.smithi102.stderr:+ grep tier 2024-09-13T14:13:00.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:00 smithi102 ceph-mon[29355]: osdmap e84: 8 total, 8 up, 8 in 2024-09-13T14:13:00.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:00 smithi102 ceph-mon[29355]: pgmap v113: 73 pgs: 73 active+clean; 799 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:00.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:00 smithi102 ceph-mon[40543]: osdmap e84: 8 total, 8 up, 8 in 2024-09-13T14:13:00.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:00 smithi102 ceph-mon[40543]: pgmap v113: 73 pgs: 73 active+clean; 799 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:00.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:00 smithi195 ceph-mon[33205]: osdmap e84: 8 total, 8 up, 8 in 2024-09-13T14:13:00.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:00 smithi195 ceph-mon[33205]: pgmap v113: 73 pgs: 73 active+clean; 799 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:00.676 INFO:tasks.workunit.client.0.smithi102.stderr:+ dd if=/dev/urandom bs=1M seek=1 count=1 of=/tmp/rbd_import_export_84613/sparse1 2024-09-13T14:13:00.680 INFO:tasks.workunit.client.0.smithi102.stderr:1+0 records in 2024-09-13T14:13:00.680 INFO:tasks.workunit.client.0.smithi102.stderr:1+0 records out 2024-09-13T14:13:00.680 INFO:tasks.workunit.client.0.smithi102.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00365223 s, 287 MB/s 2024-09-13T14:13:00.680 INFO:tasks.workunit.client.0.smithi102.stderr:+ dd if=/dev/urandom bs=1M count=1 of=/tmp/rbd_import_export_84613/sparse2 2024-09-13T14:13:00.684 INFO:tasks.workunit.client.0.smithi102.stderr:1+0 records in 2024-09-13T14:13:00.684 INFO:tasks.workunit.client.0.smithi102.stderr:1+0 records out 2024-09-13T14:13:00.684 INFO:tasks.workunit.client.0.smithi102.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00335594 s, 312 MB/s 2024-09-13T14:13:00.685 INFO:tasks.workunit.client.0.smithi102.stderr:+ truncate /tmp/rbd_import_export_84613/sparse2 -s 2M 2024-09-13T14:13:00.693 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd rm sparse1 2024-09-13T14:13:00.742 INFO:tasks.workunit.client.0.smithi102.stderr: Removing image: 0% complete...failed. 2024-09-13T14:13:00.742 INFO:tasks.workunit.client.0.smithi102.stderr:rbd: delete error: (2) No such file or directory 2024-09-13T14:13:00.747 INFO:tasks.workunit.client.0.smithi102.stderr:+ true 2024-09-13T14:13:00.747 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd import --new-format --order 20 /tmp/rbd_import_export_84613/sparse1 2024-09-13T14:13:00.816 INFO:tasks.workunit.client.0.smithi102.stderr: Importing image: 50% complete... Importing image: 100% complete...done. 2024-09-13T14:13:00.817 INFO:tasks.workunit.client.0.smithi102.stderr:rbd: --new-format is deprecated, use --image-format 2 2024-09-13T14:13:00.817 INFO:tasks.workunit.client.0.smithi102.stderr:rbd: --order is deprecated, use --object-size 2024-09-13T14:13:00.820 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd ls -l 2024-09-13T14:13:00.820 INFO:tasks.workunit.client.0.smithi102.stderr:+ grep sparse1 2024-09-13T14:13:00.820 INFO:tasks.workunit.client.0.smithi102.stderr:+ grep -Ei '(2 MiB|2048k)' 2024-09-13T14:13:00.862 INFO:tasks.workunit.client.0.smithi102.stdout:sparse1 2 MiB 2 2024-09-13T14:13:00.863 INFO:tasks.workunit.client.0.smithi102.stderr:++ objects sparse1 2024-09-13T14:13:00.863 INFO:tasks.workunit.client.0.smithi102.stderr:++ image=sparse1 2024-09-13T14:13:00.863 INFO:tasks.workunit.client.0.smithi102.stderr:+++ rbd info sparse1 2024-09-13T14:13:00.863 INFO:tasks.workunit.client.0.smithi102.stderr:+++ grep block_name_prefix 2024-09-13T14:13:00.863 INFO:tasks.workunit.client.0.smithi102.stderr:+++ awk '{print $NF;}' 2024-09-13T14:13:00.904 INFO:tasks.workunit.client.0.smithi102.stderr:++ prefix=rbd_data.3b3aecc03a2e 2024-09-13T14:13:00.905 INFO:tasks.workunit.client.0.smithi102.stderr:+++ grep rbd_data.3b3aecc03a2e 2024-09-13T14:13:00.905 INFO:tasks.workunit.client.0.smithi102.stderr:++++ get_image_data_pool sparse1 2024-09-13T14:13:00.905 INFO:tasks.workunit.client.0.smithi102.stderr:++++ image=sparse1 2024-09-13T14:13:00.906 INFO:tasks.workunit.client.0.smithi102.stderr:+++ sed -e 's/rbd_data.3b3aecc03a2e\.//' -e 's/^0*\([0-9a-f]\)/\1/' 2024-09-13T14:13:00.906 INFO:tasks.workunit.client.0.smithi102.stderr:+++ sort -u 2024-09-13T14:13:00.906 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ rbd info sparse1 2024-09-13T14:13:00.906 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ grep 'data_pool: ' 2024-09-13T14:13:00.906 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ awk -F: '{ print $NF }' 2024-09-13T14:13:00.949 INFO:tasks.workunit.client.0.smithi102.stderr:++++ data_pool= 2024-09-13T14:13:00.949 INFO:tasks.workunit.client.0.smithi102.stderr:++++ '[' -z ']' 2024-09-13T14:13:00.949 INFO:tasks.workunit.client.0.smithi102.stderr:++++ data_pool=rbd 2024-09-13T14:13:00.949 INFO:tasks.workunit.client.0.smithi102.stderr:++++ echo rbd 2024-09-13T14:13:00.949 INFO:tasks.workunit.client.0.smithi102.stderr:+++ rados ls -p rbd 2024-09-13T14:13:00.989 INFO:tasks.workunit.client.0.smithi102.stderr:++ objects=1 2024-09-13T14:13:00.989 INFO:tasks.workunit.client.0.smithi102.stderr:++ echo 1 2024-09-13T14:13:00.989 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 0 -eq 1 -o 1 = 1 ']' 2024-09-13T14:13:00.990 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd export sparse1 /tmp/rbd_import_export_84613/sparse1.out 2024-09-13T14:13:01.031 INFO:tasks.workunit.client.0.smithi102.stderr: Exporting image: 50% complete... Exporting image: 100% complete...done. 2024-09-13T14:13:01.035 INFO:tasks.workunit.client.0.smithi102.stderr:+ compare_files_and_ondisk_sizes /tmp/rbd_import_export_84613/sparse1 /tmp/rbd_import_export_84613/sparse1.out 2024-09-13T14:13:01.035 INFO:tasks.workunit.client.0.smithi102.stderr:+ cmp -l /tmp/rbd_import_export_84613/sparse1 /tmp/rbd_import_export_84613/sparse1.out 2024-09-13T14:13:01.037 INFO:tasks.workunit.client.0.smithi102.stderr:++ stat /tmp/rbd_import_export_84613/sparse1 --format %b 2024-09-13T14:13:01.038 INFO:tasks.workunit.client.0.smithi102.stderr:+ origsize=2048 2024-09-13T14:13:01.038 INFO:tasks.workunit.client.0.smithi102.stderr:++ stat /tmp/rbd_import_export_84613/sparse1.out --format %b 2024-09-13T14:13:01.040 INFO:tasks.workunit.client.0.smithi102.stderr:+ exportsize=2048 2024-09-13T14:13:01.040 INFO:tasks.workunit.client.0.smithi102.stderr:+ difference=0 2024-09-13T14:13:01.040 INFO:tasks.workunit.client.0.smithi102.stderr:+ difference=0 2024-09-13T14:13:01.040 INFO:tasks.workunit.client.0.smithi102.stderr:+ test 0 -ge 0 -a 0 -lt 4096 2024-09-13T14:13:01.040 INFO:tasks.workunit.client.0.smithi102.stderr:+ rm /tmp/rbd_import_export_84613/sparse1.out 2024-09-13T14:13:01.040 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd rm sparse1 2024-09-13T14:13:01.136 INFO:tasks.workunit.client.0.smithi102.stderr: Removing image: 50% complete... Removing image: 100% complete...done. 2024-09-13T14:13:01.139 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd rm sparse2 2024-09-13T14:13:01.182 INFO:tasks.workunit.client.0.smithi102.stderr: Removing image: 0% complete...failed. 2024-09-13T14:13:01.182 INFO:tasks.workunit.client.0.smithi102.stderr:rbd: delete error: (2) No such file or directory 2024-09-13T14:13:01.185 INFO:tasks.workunit.client.0.smithi102.stderr:+ true 2024-09-13T14:13:01.185 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd import --new-format --order 20 /tmp/rbd_import_export_84613/sparse2 2024-09-13T14:13:01.252 INFO:tasks.workunit.client.0.smithi102.stderr: Importing image: 50% complete... Importing image: 100% complete...done. 2024-09-13T14:13:01.252 INFO:tasks.workunit.client.0.smithi102.stderr:rbd: --new-format is deprecated, use --image-format 2 2024-09-13T14:13:01.253 INFO:tasks.workunit.client.0.smithi102.stderr:rbd: --order is deprecated, use --object-size 2024-09-13T14:13:01.256 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd ls -l 2024-09-13T14:13:01.256 INFO:tasks.workunit.client.0.smithi102.stderr:+ grep sparse2 2024-09-13T14:13:01.256 INFO:tasks.workunit.client.0.smithi102.stderr:+ grep -Ei '(2 MiB|2048k)' 2024-09-13T14:13:01.299 INFO:tasks.workunit.client.0.smithi102.stdout:sparse2 2 MiB 2 2024-09-13T14:13:01.300 INFO:tasks.workunit.client.0.smithi102.stderr:++ objects sparse2 2024-09-13T14:13:01.300 INFO:tasks.workunit.client.0.smithi102.stderr:++ image=sparse2 2024-09-13T14:13:01.300 INFO:tasks.workunit.client.0.smithi102.stderr:+++ rbd info sparse2 2024-09-13T14:13:01.300 INFO:tasks.workunit.client.0.smithi102.stderr:+++ grep block_name_prefix 2024-09-13T14:13:01.300 INFO:tasks.workunit.client.0.smithi102.stderr:+++ awk '{print $NF;}' 2024-09-13T14:13:01.342 INFO:tasks.workunit.client.0.smithi102.stderr:++ prefix=rbd_data.621290058696 2024-09-13T14:13:01.343 INFO:tasks.workunit.client.0.smithi102.stderr:+++ grep rbd_data.621290058696 2024-09-13T14:13:01.343 INFO:tasks.workunit.client.0.smithi102.stderr:+++ sed -e 's/rbd_data.621290058696\.//' -e 's/^0*\([0-9a-f]\)/\1/' 2024-09-13T14:13:01.343 INFO:tasks.workunit.client.0.smithi102.stderr:++++ get_image_data_pool sparse2 2024-09-13T14:13:01.343 INFO:tasks.workunit.client.0.smithi102.stderr:++++ image=sparse2 2024-09-13T14:13:01.343 INFO:tasks.workunit.client.0.smithi102.stderr:+++ sort -u 2024-09-13T14:13:01.344 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ rbd info sparse2 2024-09-13T14:13:01.344 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ grep 'data_pool: ' 2024-09-13T14:13:01.344 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ awk -F: '{ print $NF }' 2024-09-13T14:13:01.388 INFO:tasks.workunit.client.0.smithi102.stderr:++++ data_pool= 2024-09-13T14:13:01.388 INFO:tasks.workunit.client.0.smithi102.stderr:++++ '[' -z ']' 2024-09-13T14:13:01.388 INFO:tasks.workunit.client.0.smithi102.stderr:++++ data_pool=rbd 2024-09-13T14:13:01.388 INFO:tasks.workunit.client.0.smithi102.stderr:++++ echo rbd 2024-09-13T14:13:01.388 INFO:tasks.workunit.client.0.smithi102.stderr:+++ rados ls -p rbd 2024-09-13T14:13:01.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:01 smithi102 ceph-mon[29355]: osdmap e85: 8 total, 8 up, 8 in 2024-09-13T14:13:01.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:01 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/1089600312' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-13T14:13:01.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:01 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/1089600312' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-13T14:13:01.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:01 smithi102 ceph-mon[40543]: osdmap e85: 8 total, 8 up, 8 in 2024-09-13T14:13:01.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:01 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/1089600312' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-13T14:13:01.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:01 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/1089600312' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-13T14:13:01.426 INFO:tasks.workunit.client.0.smithi102.stderr:++ objects=0 2024-09-13T14:13:01.426 INFO:tasks.workunit.client.0.smithi102.stderr:++ echo 0 2024-09-13T14:13:01.426 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 0 -eq 1 -o 0 = 0 ']' 2024-09-13T14:13:01.426 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd export sparse2 /tmp/rbd_import_export_84613/sparse2.out 2024-09-13T14:13:01.469 INFO:tasks.workunit.client.0.smithi102.stderr: Exporting image: 50% complete... Exporting image: 100% complete...done. 2024-09-13T14:13:01.473 INFO:tasks.workunit.client.0.smithi102.stderr:+ compare_files_and_ondisk_sizes /tmp/rbd_import_export_84613/sparse2 /tmp/rbd_import_export_84613/sparse2.out 2024-09-13T14:13:01.473 INFO:tasks.workunit.client.0.smithi102.stderr:+ cmp -l /tmp/rbd_import_export_84613/sparse2 /tmp/rbd_import_export_84613/sparse2.out 2024-09-13T14:13:01.475 INFO:tasks.workunit.client.0.smithi102.stderr:++ stat /tmp/rbd_import_export_84613/sparse2 --format %b 2024-09-13T14:13:01.476 INFO:tasks.workunit.client.0.smithi102.stderr:+ origsize=2048 2024-09-13T14:13:01.476 INFO:tasks.workunit.client.0.smithi102.stderr:++ stat /tmp/rbd_import_export_84613/sparse2.out --format %b 2024-09-13T14:13:01.477 INFO:tasks.workunit.client.0.smithi102.stderr:+ exportsize=2048 2024-09-13T14:13:01.477 INFO:tasks.workunit.client.0.smithi102.stderr:+ difference=0 2024-09-13T14:13:01.477 INFO:tasks.workunit.client.0.smithi102.stderr:+ difference=0 2024-09-13T14:13:01.477 INFO:tasks.workunit.client.0.smithi102.stderr:+ test 0 -ge 0 -a 0 -lt 4096 2024-09-13T14:13:01.477 INFO:tasks.workunit.client.0.smithi102.stderr:+ rm /tmp/rbd_import_export_84613/sparse2.out 2024-09-13T14:13:01.478 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd rm sparse2 2024-09-13T14:13:01.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:01 smithi195 ceph-mon[33205]: osdmap e85: 8 total, 8 up, 8 in 2024-09-13T14:13:01.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:01 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/1089600312' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-13T14:13:01.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:01 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/1089600312' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-13T14:13:01.569 INFO:tasks.workunit.client.0.smithi102.stderr: Removing image: 50% complete... Removing image: 100% complete...done. 2024-09-13T14:13:01.572 INFO:tasks.workunit.client.0.smithi102.stderr:+ truncate /tmp/rbd_import_export_84613/sparse1 -s 10M 2024-09-13T14:13:01.573 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd import --new-format --order 20 - sparse1 2024-09-13T14:13:01.663 INFO:tasks.workunit.client.0.smithi102.stderr: Importing image: 100% complete...done. 2024-09-13T14:13:01.664 INFO:tasks.workunit.client.0.smithi102.stderr:rbd: --new-format is deprecated, use --image-format 2 2024-09-13T14:13:01.664 INFO:tasks.workunit.client.0.smithi102.stderr:rbd: --order is deprecated, use --object-size 2024-09-13T14:13:01.667 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd ls -l 2024-09-13T14:13:01.667 INFO:tasks.workunit.client.0.smithi102.stderr:+ grep sparse1 2024-09-13T14:13:01.667 INFO:tasks.workunit.client.0.smithi102.stderr:+ grep -Ei '(10 MiB|10240k)' 2024-09-13T14:13:01.708 INFO:tasks.workunit.client.0.smithi102.stdout:sparse1 10 MiB 2 2024-09-13T14:13:01.709 INFO:tasks.workunit.client.0.smithi102.stderr:++ objects sparse1 2024-09-13T14:13:01.709 INFO:tasks.workunit.client.0.smithi102.stderr:++ image=sparse1 2024-09-13T14:13:01.709 INFO:tasks.workunit.client.0.smithi102.stderr:+++ rbd info sparse1 2024-09-13T14:13:01.709 INFO:tasks.workunit.client.0.smithi102.stderr:+++ grep block_name_prefix 2024-09-13T14:13:01.709 INFO:tasks.workunit.client.0.smithi102.stderr:+++ awk '{print $NF;}' 2024-09-13T14:13:01.752 INFO:tasks.workunit.client.0.smithi102.stderr:++ prefix=rbd_data.3b9a8178a70a 2024-09-13T14:13:01.753 INFO:tasks.workunit.client.0.smithi102.stderr:+++ grep rbd_data.3b9a8178a70a 2024-09-13T14:13:01.753 INFO:tasks.workunit.client.0.smithi102.stderr:+++ sed -e 's/rbd_data.3b9a8178a70a\.//' -e 's/^0*\([0-9a-f]\)/\1/' 2024-09-13T14:13:01.753 INFO:tasks.workunit.client.0.smithi102.stderr:++++ get_image_data_pool sparse1 2024-09-13T14:13:01.753 INFO:tasks.workunit.client.0.smithi102.stderr:++++ image=sparse1 2024-09-13T14:13:01.753 INFO:tasks.workunit.client.0.smithi102.stderr:+++ sort -u 2024-09-13T14:13:01.753 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ rbd info sparse1 2024-09-13T14:13:01.753 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ grep 'data_pool: ' 2024-09-13T14:13:01.754 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ awk -F: '{ print $NF }' 2024-09-13T14:13:01.794 INFO:tasks.workunit.client.0.smithi102.stderr:++++ data_pool= 2024-09-13T14:13:01.794 INFO:tasks.workunit.client.0.smithi102.stderr:++++ '[' -z ']' 2024-09-13T14:13:01.794 INFO:tasks.workunit.client.0.smithi102.stderr:++++ data_pool=rbd 2024-09-13T14:13:01.795 INFO:tasks.workunit.client.0.smithi102.stderr:++++ echo rbd 2024-09-13T14:13:01.795 INFO:tasks.workunit.client.0.smithi102.stderr:+++ rados ls -p rbd 2024-09-13T14:13:01.832 INFO:tasks.workunit.client.0.smithi102.stderr:++ objects=1 2024-09-13T14:13:01.832 INFO:tasks.workunit.client.0.smithi102.stderr:++ echo 1 2024-09-13T14:13:01.832 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 0 -eq 1 -o 1 = 1 ']' 2024-09-13T14:13:01.832 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd export sparse1 /tmp/rbd_import_export_84613/sparse1.out 2024-09-13T14:13:01.874 INFO:tasks.workunit.client.0.smithi102.stderr: Exporting image: 10% complete... Exporting image: 20% complete... Exporting image: 30% complete... Exporting image: 40% complete... Exporting image: 50% complete... Exporting image: 60% complete... Exporting image: 70% complete... Exporting image: 80% complete... Exporting image: 90% complete... Exporting image: 100% complete...done. 2024-09-13T14:13:01.879 INFO:tasks.workunit.client.0.smithi102.stderr:+ compare_files_and_ondisk_sizes /tmp/rbd_import_export_84613/sparse1 /tmp/rbd_import_export_84613/sparse1.out 2024-09-13T14:13:01.879 INFO:tasks.workunit.client.0.smithi102.stderr:+ cmp -l /tmp/rbd_import_export_84613/sparse1 /tmp/rbd_import_export_84613/sparse1.out 2024-09-13T14:13:01.885 INFO:tasks.workunit.client.0.smithi102.stderr:++ stat /tmp/rbd_import_export_84613/sparse1 --format %b 2024-09-13T14:13:01.886 INFO:tasks.workunit.client.0.smithi102.stderr:+ origsize=2048 2024-09-13T14:13:01.887 INFO:tasks.workunit.client.0.smithi102.stderr:++ stat /tmp/rbd_import_export_84613/sparse1.out --format %b 2024-09-13T14:13:01.888 INFO:tasks.workunit.client.0.smithi102.stderr:+ exportsize=2048 2024-09-13T14:13:01.888 INFO:tasks.workunit.client.0.smithi102.stderr:+ difference=0 2024-09-13T14:13:01.888 INFO:tasks.workunit.client.0.smithi102.stderr:+ difference=0 2024-09-13T14:13:01.888 INFO:tasks.workunit.client.0.smithi102.stderr:+ test 0 -ge 0 -a 0 -lt 4096 2024-09-13T14:13:01.888 INFO:tasks.workunit.client.0.smithi102.stderr:+ rm /tmp/rbd_import_export_84613/sparse1.out 2024-09-13T14:13:01.889 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd rm sparse1 2024-09-13T14:13:01.979 INFO:tasks.workunit.client.0.smithi102.stderr: Removing image: 10% complete... Removing image: 20% complete... Removing image: 30% complete... Removing image: 40% complete... Removing image: 50% complete... Removing image: 60% complete... Removing image: 70% complete... Removing image: 80% complete... Removing image: 90% complete... Removing image: 100% complete...done. 2024-09-13T14:13:01.982 INFO:tasks.workunit.client.0.smithi102.stderr:+ dd if=/dev/urandom bs=2M count=1 of=/tmp/rbd_import_export_84613/sparse2 oflag=append conv=notrunc 2024-09-13T14:13:01.990 INFO:tasks.workunit.client.0.smithi102.stderr:1+0 records in 2024-09-13T14:13:01.990 INFO:tasks.workunit.client.0.smithi102.stderr:1+0 records out 2024-09-13T14:13:01.990 INFO:tasks.workunit.client.0.smithi102.stderr:2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.00658386 s, 319 MB/s 2024-09-13T14:13:01.990 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd import --new-format --order 20 - sparse2 2024-09-13T14:13:02.079 INFO:tasks.workunit.client.0.smithi102.stderr: Importing image: 100% complete...done. 2024-09-13T14:13:02.079 INFO:tasks.workunit.client.0.smithi102.stderr:rbd: --new-format is deprecated, use --image-format 2 2024-09-13T14:13:02.079 INFO:tasks.workunit.client.0.smithi102.stderr:rbd: --order is deprecated, use --object-size 2024-09-13T14:13:02.082 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd ls -l 2024-09-13T14:13:02.083 INFO:tasks.workunit.client.0.smithi102.stderr:+ grep sparse2 2024-09-13T14:13:02.083 INFO:tasks.workunit.client.0.smithi102.stderr:+ grep -Ei '(4 MiB|4096k)' 2024-09-13T14:13:02.123 INFO:tasks.workunit.client.0.smithi102.stdout:sparse2 4 MiB 2 2024-09-13T14:13:02.124 INFO:tasks.workunit.client.0.smithi102.stderr:++ objects sparse2 2024-09-13T14:13:02.124 INFO:tasks.workunit.client.0.smithi102.stderr:++ image=sparse2 2024-09-13T14:13:02.124 INFO:tasks.workunit.client.0.smithi102.stderr:+++ rbd info sparse2 2024-09-13T14:13:02.124 INFO:tasks.workunit.client.0.smithi102.stderr:+++ grep block_name_prefix 2024-09-13T14:13:02.124 INFO:tasks.workunit.client.0.smithi102.stderr:+++ awk '{print $NF;}' 2024-09-13T14:13:02.167 INFO:tasks.workunit.client.0.smithi102.stderr:++ prefix=rbd_data.626c8a85f588 2024-09-13T14:13:02.168 INFO:tasks.workunit.client.0.smithi102.stderr:+++ grep rbd_data.626c8a85f588 2024-09-13T14:13:02.168 INFO:tasks.workunit.client.0.smithi102.stderr:++++ get_image_data_pool sparse2 2024-09-13T14:13:02.168 INFO:tasks.workunit.client.0.smithi102.stderr:+++ sed -e 's/rbd_data.626c8a85f588\.//' -e 's/^0*\([0-9a-f]\)/\1/' 2024-09-13T14:13:02.168 INFO:tasks.workunit.client.0.smithi102.stderr:++++ image=sparse2 2024-09-13T14:13:02.168 INFO:tasks.workunit.client.0.smithi102.stderr:+++ sort -u 2024-09-13T14:13:02.168 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ rbd info sparse2 2024-09-13T14:13:02.168 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ grep 'data_pool: ' 2024-09-13T14:13:02.169 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ awk -F: '{ print $NF }' 2024-09-13T14:13:02.211 INFO:tasks.workunit.client.0.smithi102.stderr:++++ data_pool= 2024-09-13T14:13:02.211 INFO:tasks.workunit.client.0.smithi102.stderr:++++ '[' -z ']' 2024-09-13T14:13:02.211 INFO:tasks.workunit.client.0.smithi102.stderr:++++ data_pool=rbd 2024-09-13T14:13:02.211 INFO:tasks.workunit.client.0.smithi102.stderr:++++ echo rbd 2024-09-13T14:13:02.211 INFO:tasks.workunit.client.0.smithi102.stderr:+++ rados ls -p rbd 2024-09-13T14:13:02.250 INFO:tasks.workunit.client.0.smithi102.stderr:++ objects='0 2024-09-13T14:13:02.250 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-09-13T14:13:02.250 INFO:tasks.workunit.client.0.smithi102.stderr:3' 2024-09-13T14:13:02.250 INFO:tasks.workunit.client.0.smithi102.stderr:++ echo 0 2 3 2024-09-13T14:13:02.250 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 0 -eq 1 -o '0 2 3' = '0 2 3' ']' 2024-09-13T14:13:02.250 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd export sparse2 /tmp/rbd_import_export_84613/sparse2.out 2024-09-13T14:13:02.294 INFO:tasks.workunit.client.0.smithi102.stderr: Exporting image: 25% complete... Exporting image: 50% complete... Exporting image: 75% complete... Exporting image: 100% complete...done. 2024-09-13T14:13:02.298 INFO:tasks.workunit.client.0.smithi102.stderr:+ compare_files_and_ondisk_sizes /tmp/rbd_import_export_84613/sparse2 /tmp/rbd_import_export_84613/sparse2.out 2024-09-13T14:13:02.298 INFO:tasks.workunit.client.0.smithi102.stderr:+ cmp -l /tmp/rbd_import_export_84613/sparse2 /tmp/rbd_import_export_84613/sparse2.out 2024-09-13T14:13:02.301 INFO:tasks.workunit.client.0.smithi102.stderr:++ stat /tmp/rbd_import_export_84613/sparse2 --format %b 2024-09-13T14:13:02.302 INFO:tasks.workunit.client.0.smithi102.stderr:+ origsize=6144 2024-09-13T14:13:02.302 INFO:tasks.workunit.client.0.smithi102.stderr:++ stat /tmp/rbd_import_export_84613/sparse2.out --format %b 2024-09-13T14:13:02.303 INFO:tasks.workunit.client.0.smithi102.stderr:+ exportsize=6144 2024-09-13T14:13:02.303 INFO:tasks.workunit.client.0.smithi102.stderr:+ difference=0 2024-09-13T14:13:02.303 INFO:tasks.workunit.client.0.smithi102.stderr:+ difference=0 2024-09-13T14:13:02.303 INFO:tasks.workunit.client.0.smithi102.stderr:+ test 0 -ge 0 -a 0 -lt 4096 2024-09-13T14:13:02.304 INFO:tasks.workunit.client.0.smithi102.stderr:+ rm /tmp/rbd_import_export_84613/sparse2.out 2024-09-13T14:13:02.304 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd rm sparse2 2024-09-13T14:13:02.393 INFO:tasks.workunit.client.0.smithi102.stderr: Removing image: 25% complete... Removing image: 50% complete... Removing image: 75% complete... Removing image: 100% complete...done. 2024-09-13T14:13:02.396 INFO:tasks.workunit.client.0.smithi102.stderr:+ echo 'partially-sparse file imports to partially-sparse image' 2024-09-13T14:13:02.396 INFO:tasks.workunit.client.0.smithi102.stdout:partially-sparse file imports to partially-sparse image 2024-09-13T14:13:02.396 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd import --new-format --order 20 /tmp/rbd_import_export_84613/sparse1 sparse 2024-09-13T14:13:02.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:02 smithi102 ceph-mon[40543]: pgmap v115: 73 pgs: 73 active+clean; 799 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 101 KiB/s rd, 8.0 KiB/s wr, 137 op/s 2024-09-13T14:13:02.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:02 smithi102 ceph-mon[29355]: pgmap v115: 73 pgs: 73 active+clean; 799 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 101 KiB/s rd, 8.0 KiB/s wr, 137 op/s 2024-09-13T14:13:02.462 INFO:tasks.workunit.client.0.smithi102.stderr: Importing image: 10% complete... Importing image: 20% complete... Importing image: 30% complete... Importing image: 40% complete... Importing image: 50% complete... Importing image: 60% complete... Importing image: 70% complete... Importing image: 80% complete... Importing image: 90% complete... Importing image: 100% complete...done. 2024-09-13T14:13:02.463 INFO:tasks.workunit.client.0.smithi102.stderr:rbd: --new-format is deprecated, use --image-format 2 2024-09-13T14:13:02.463 INFO:tasks.workunit.client.0.smithi102.stderr:rbd: --order is deprecated, use --object-size 2024-09-13T14:13:02.466 INFO:tasks.workunit.client.0.smithi102.stderr:++ objects sparse 2024-09-13T14:13:02.466 INFO:tasks.workunit.client.0.smithi102.stderr:++ image=sparse 2024-09-13T14:13:02.467 INFO:tasks.workunit.client.0.smithi102.stderr:+++ rbd info sparse 2024-09-13T14:13:02.467 INFO:tasks.workunit.client.0.smithi102.stderr:+++ grep block_name_prefix 2024-09-13T14:13:02.467 INFO:tasks.workunit.client.0.smithi102.stderr:+++ awk '{print $NF;}' 2024-09-13T14:13:02.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:02 smithi195 ceph-mon[33205]: pgmap v115: 73 pgs: 73 active+clean; 799 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 101 KiB/s rd, 8.0 KiB/s wr, 137 op/s 2024-09-13T14:13:02.508 INFO:tasks.workunit.client.0.smithi102.stderr:++ prefix=rbd_data.3bf4d26f8374 2024-09-13T14:13:02.509 INFO:tasks.workunit.client.0.smithi102.stderr:+++ grep rbd_data.3bf4d26f8374 2024-09-13T14:13:02.509 INFO:tasks.workunit.client.0.smithi102.stderr:+++ sed -e 's/rbd_data.3bf4d26f8374\.//' -e 's/^0*\([0-9a-f]\)/\1/' 2024-09-13T14:13:02.509 INFO:tasks.workunit.client.0.smithi102.stderr:++++ get_image_data_pool sparse 2024-09-13T14:13:02.509 INFO:tasks.workunit.client.0.smithi102.stderr:++++ image=sparse 2024-09-13T14:13:02.510 INFO:tasks.workunit.client.0.smithi102.stderr:+++ sort -u 2024-09-13T14:13:02.510 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ rbd info sparse 2024-09-13T14:13:02.510 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ grep 'data_pool: ' 2024-09-13T14:13:02.510 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ awk -F: '{ print $NF }' 2024-09-13T14:13:02.553 INFO:tasks.workunit.client.0.smithi102.stderr:++++ data_pool= 2024-09-13T14:13:02.554 INFO:tasks.workunit.client.0.smithi102.stderr:++++ '[' -z ']' 2024-09-13T14:13:02.554 INFO:tasks.workunit.client.0.smithi102.stderr:++++ data_pool=rbd 2024-09-13T14:13:02.554 INFO:tasks.workunit.client.0.smithi102.stderr:++++ echo rbd 2024-09-13T14:13:02.554 INFO:tasks.workunit.client.0.smithi102.stderr:+++ rados ls -p rbd 2024-09-13T14:13:02.592 INFO:tasks.workunit.client.0.smithi102.stderr:++ objects=1 2024-09-13T14:13:02.592 INFO:tasks.workunit.client.0.smithi102.stderr:++ echo 1 2024-09-13T14:13:02.592 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 0 -eq 1 -o 1 = 1 ']' 2024-09-13T14:13:02.592 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd rm sparse 2024-09-13T14:13:02.684 INFO:tasks.workunit.client.0.smithi102.stderr: Removing image: 10% complete... Removing image: 20% complete... Removing image: 30% complete... Removing image: 40% complete... Removing image: 50% complete... Removing image: 60% complete... Removing image: 70% complete... Removing image: 80% complete... Removing image: 90% complete... Removing image: 100% complete...done. 2024-09-13T14:13:02.687 INFO:tasks.workunit.client.0.smithi102.stdout:zeros import through stdin to sparse image 2024-09-13T14:13:02.688 INFO:tasks.workunit.client.0.smithi102.stderr:+ echo 'zeros import through stdin to sparse image' 2024-09-13T14:13:02.688 INFO:tasks.workunit.client.0.smithi102.stderr:+ dd if=/dev/zero bs=1M count=4 2024-09-13T14:13:02.688 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd import --new-format - sparse 2024-09-13T14:13:02.742 INFO:tasks.workunit.client.0.smithi102.stderr:4+0 records in 2024-09-13T14:13:02.742 INFO:tasks.workunit.client.0.smithi102.stderr:4+0 records out 2024-09-13T14:13:02.742 INFO:tasks.workunit.client.0.smithi102.stderr:4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.0536978 s, 78.1 MB/s 2024-09-13T14:13:02.751 INFO:tasks.workunit.client.0.smithi102.stderr: Importing image: 100% complete...done. 2024-09-13T14:13:02.751 INFO:tasks.workunit.client.0.smithi102.stderr:rbd: --new-format is deprecated, use --image-format 2 2024-09-13T14:13:02.755 INFO:tasks.workunit.client.0.smithi102.stderr:++ objects sparse 2024-09-13T14:13:02.755 INFO:tasks.workunit.client.0.smithi102.stderr:++ image=sparse 2024-09-13T14:13:02.755 INFO:tasks.workunit.client.0.smithi102.stderr:+++ rbd info sparse 2024-09-13T14:13:02.755 INFO:tasks.workunit.client.0.smithi102.stderr:+++ grep block_name_prefix 2024-09-13T14:13:02.755 INFO:tasks.workunit.client.0.smithi102.stderr:+++ awk '{print $NF;}' 2024-09-13T14:13:02.798 INFO:tasks.workunit.client.0.smithi102.stderr:++ prefix=rbd_data.3c15acfd453d 2024-09-13T14:13:02.799 INFO:tasks.workunit.client.0.smithi102.stderr:+++ grep rbd_data.3c15acfd453d 2024-09-13T14:13:02.799 INFO:tasks.workunit.client.0.smithi102.stderr:++++ get_image_data_pool sparse 2024-09-13T14:13:02.799 INFO:tasks.workunit.client.0.smithi102.stderr:++++ image=sparse 2024-09-13T14:13:02.799 INFO:tasks.workunit.client.0.smithi102.stderr:+++ sed -e 's/rbd_data.3c15acfd453d\.//' -e 's/^0*\([0-9a-f]\)/\1/' 2024-09-13T14:13:02.799 INFO:tasks.workunit.client.0.smithi102.stderr:+++ sort -u 2024-09-13T14:13:02.799 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ rbd info sparse 2024-09-13T14:13:02.800 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ awk -F: '{ print $NF }' 2024-09-13T14:13:02.800 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ grep 'data_pool: ' 2024-09-13T14:13:02.841 INFO:tasks.workunit.client.0.smithi102.stderr:++++ data_pool= 2024-09-13T14:13:02.841 INFO:tasks.workunit.client.0.smithi102.stderr:++++ '[' -z ']' 2024-09-13T14:13:02.841 INFO:tasks.workunit.client.0.smithi102.stderr:++++ data_pool=rbd 2024-09-13T14:13:02.841 INFO:tasks.workunit.client.0.smithi102.stderr:++++ echo rbd 2024-09-13T14:13:02.841 INFO:tasks.workunit.client.0.smithi102.stderr:+++ rados ls -p rbd 2024-09-13T14:13:02.879 INFO:tasks.workunit.client.0.smithi102.stderr:++ objects= 2024-09-13T14:13:02.879 INFO:tasks.workunit.client.0.smithi102.stderr:++ echo 2024-09-13T14:13:02.879 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 0 -eq 1 -o '' = '' ']' 2024-09-13T14:13:02.879 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd rm sparse 2024-09-13T14:13:02.963 INFO:tasks.workunit.client.0.smithi102.stderr: Removing image: 100% complete...done. 2024-09-13T14:13:02.966 INFO:tasks.workunit.client.0.smithi102.stdout:zeros export to sparse file 2024-09-13T14:13:02.967 INFO:tasks.workunit.client.0.smithi102.stderr:+ echo 'zeros export to sparse file' 2024-09-13T14:13:02.967 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd create --new-format sparse --size 4 2024-09-13T14:13:03.011 INFO:tasks.workunit.client.0.smithi102.stderr:rbd: --new-format is deprecated, use --image-format 2 2024-09-13T14:13:03.014 INFO:tasks.workunit.client.0.smithi102.stderr:++ rbd info sparse 2024-09-13T14:13:03.014 INFO:tasks.workunit.client.0.smithi102.stderr:++ grep block_name_prefix 2024-09-13T14:13:03.014 INFO:tasks.workunit.client.0.smithi102.stderr:++ awk '{print $NF;}' 2024-09-13T14:13:03.063 INFO:tasks.workunit.client.0.smithi102.stderr:+ prefix=rbd_data.62dbffc533e9 2024-09-13T14:13:03.063 INFO:tasks.workunit.client.0.smithi102.stderr:+ dd if=/dev/zero bs=4M count=1 2024-09-13T14:13:03.063 INFO:tasks.workunit.client.0.smithi102.stderr:++ get_image_data_pool sparse 2024-09-13T14:13:03.063 INFO:tasks.workunit.client.0.smithi102.stderr:++ image=sparse 2024-09-13T14:13:03.064 INFO:tasks.workunit.client.0.smithi102.stderr:+++ rbd info sparse 2024-09-13T14:13:03.064 INFO:tasks.workunit.client.0.smithi102.stderr:+++ grep 'data_pool: ' 2024-09-13T14:13:03.064 INFO:tasks.workunit.client.0.smithi102.stderr:+++ awk -F: '{ print $NF }' 2024-09-13T14:13:03.110 INFO:tasks.workunit.client.0.smithi102.stderr:++ data_pool= 2024-09-13T14:13:03.110 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -z ']' 2024-09-13T14:13:03.110 INFO:tasks.workunit.client.0.smithi102.stderr:++ data_pool=rbd 2024-09-13T14:13:03.110 INFO:tasks.workunit.client.0.smithi102.stderr:++ echo rbd 2024-09-13T14:13:03.110 INFO:tasks.workunit.client.0.smithi102.stderr:+ rados -p rbd put rbd_data.62dbffc533e9.000000000000 - 2024-09-13T14:13:03.145 INFO:tasks.workunit.client.0.smithi102.stderr:1+0 records in 2024-09-13T14:13:03.145 INFO:tasks.workunit.client.0.smithi102.stderr:1+0 records out 2024-09-13T14:13:03.145 INFO:tasks.workunit.client.0.smithi102.stderr:4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.0818736 s, 51.2 MB/s 2024-09-13T14:13:03.165 INFO:tasks.workunit.client.0.smithi102.stderr:++ objects sparse 2024-09-13T14:13:03.165 INFO:tasks.workunit.client.0.smithi102.stderr:++ image=sparse 2024-09-13T14:13:03.165 INFO:tasks.workunit.client.0.smithi102.stderr:+++ rbd info sparse 2024-09-13T14:13:03.165 INFO:tasks.workunit.client.0.smithi102.stderr:+++ grep block_name_prefix 2024-09-13T14:13:03.165 INFO:tasks.workunit.client.0.smithi102.stderr:+++ awk '{print $NF;}' 2024-09-13T14:13:03.207 INFO:tasks.workunit.client.0.smithi102.stderr:++ prefix=rbd_data.62dbffc533e9 2024-09-13T14:13:03.208 INFO:tasks.workunit.client.0.smithi102.stderr:+++ grep rbd_data.62dbffc533e9 2024-09-13T14:13:03.208 INFO:tasks.workunit.client.0.smithi102.stderr:++++ get_image_data_pool sparse 2024-09-13T14:13:03.208 INFO:tasks.workunit.client.0.smithi102.stderr:++++ image=sparse 2024-09-13T14:13:03.208 INFO:tasks.workunit.client.0.smithi102.stderr:+++ sed -e 's/rbd_data.62dbffc533e9\.//' -e 's/^0*\([0-9a-f]\)/\1/' 2024-09-13T14:13:03.208 INFO:tasks.workunit.client.0.smithi102.stderr:+++ sort -u 2024-09-13T14:13:03.208 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ rbd info sparse 2024-09-13T14:13:03.208 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ grep 'data_pool: ' 2024-09-13T14:13:03.208 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ awk -F: '{ print $NF }' 2024-09-13T14:13:03.251 INFO:tasks.workunit.client.0.smithi102.stderr:++++ data_pool= 2024-09-13T14:13:03.251 INFO:tasks.workunit.client.0.smithi102.stderr:++++ '[' -z ']' 2024-09-13T14:13:03.251 INFO:tasks.workunit.client.0.smithi102.stderr:++++ data_pool=rbd 2024-09-13T14:13:03.251 INFO:tasks.workunit.client.0.smithi102.stderr:++++ echo rbd 2024-09-13T14:13:03.251 INFO:tasks.workunit.client.0.smithi102.stderr:+++ rados ls -p rbd 2024-09-13T14:13:03.289 INFO:tasks.workunit.client.0.smithi102.stderr:++ objects=0 2024-09-13T14:13:03.289 INFO:tasks.workunit.client.0.smithi102.stderr:++ echo 0 2024-09-13T14:13:03.289 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 0 -eq 1 -o 0 = 0 ']' 2024-09-13T14:13:03.290 INFO:tasks.workunit.client.0.smithi102.stderr:+ rm /tmp/rbd_import_export_84613/sparse 2024-09-13T14:13:03.290 INFO:tasks.workunit.client.0.smithi102.stderr:rm: cannot remove '/tmp/rbd_import_export_84613/sparse': No such file or directory 2024-09-13T14:13:03.290 INFO:tasks.workunit.client.0.smithi102.stderr:+ true 2024-09-13T14:13:03.290 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd export sparse /tmp/rbd_import_export_84613/sparse 2024-09-13T14:13:03.331 INFO:tasks.workunit.client.0.smithi102.stderr: Exporting image: 100% complete...done. 2024-09-13T14:13:03.334 INFO:tasks.workunit.client.0.smithi102.stderr:++ stat /tmp/rbd_import_export_84613/sparse --format=%b 2024-09-13T14:13:03.336 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 0 = 0 ']' 2024-09-13T14:13:03.336 INFO:tasks.workunit.client.0.smithi102.stderr:+ rbd rm sparse 2024-09-13T14:13:03.424 INFO:tasks.workunit.client.0.smithi102.stderr: Removing image: 100% complete...done. 2024-09-13T14:13:03.426 INFO:tasks.workunit.client.0.smithi102.stderr:+ rm /tmp/rbd_import_export_84613/sparse /tmp/rbd_import_export_84613/sparse1 /tmp/rbd_import_export_84613/sparse2 /tmp/rbd_import_export_84613/sparse3 2024-09-13T14:13:03.429 INFO:tasks.workunit.client.0.smithi102.stderr:rm: cannot remove '/tmp/rbd_import_export_84613/sparse3': No such file or directory 2024-09-13T14:13:03.429 INFO:tasks.workunit.client.0.smithi102.stderr:+ true 2024-09-13T14:13:03.429 INFO:tasks.workunit.client.0.smithi102.stderr:+ echo OK 2024-09-13T14:13:03.429 INFO:tasks.workunit.client.0.smithi102.stdout:OK 2024-09-13T14:13:03.430 INFO:tasks.workunit.client.0.smithi102.stderr:+ rm -rf /tmp/rbd_import_export_84613 2024-09-13T14:13:03.430 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-13T14:13:03.430 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-13T14:13:03.459 INFO:tasks.workunit:Stopping ['rbd/import_export.sh'] on client.0... 2024-09-13T14:13:03.459 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-13T14:13:03.712 DEBUG:teuthology.parallel:result is None 2024-09-13T14:13:03.713 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-13T14:13:03.738 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-13T14:13:03.738 DEBUG:teuthology.orchestra.run.smithi102:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-13T14:13:03.794 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-13T14:13:03.794 INFO:teuthology.task.sequential:In sequential, running task print... 2024-09-13T14:13:03.794 INFO:teuthology.task.print:**** done rbd/import_export.sh 5-workload 2024-09-13T14:13:03.794 DEBUG:teuthology.parallel:result is None 2024-09-13T14:13:04.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:04 smithi102 ceph-mon[29355]: osdmap e86: 8 total, 8 up, 8 in 2024-09-13T14:13:04.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:04 smithi102 ceph-mon[29355]: pgmap v117: 73 pgs: 73 active+clean; 799 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 6.4 KiB/s wr, 109 op/s 2024-09-13T14:13:04.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:04 smithi102 ceph-mon[40543]: osdmap e86: 8 total, 8 up, 8 in 2024-09-13T14:13:04.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:04 smithi102 ceph-mon[40543]: pgmap v117: 73 pgs: 73 active+clean; 799 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 6.4 KiB/s wr, 109 op/s 2024-09-13T14:13:04.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:04 smithi195 ceph-mon[33205]: osdmap e86: 8 total, 8 up, 8 in 2024-09-13T14:13:04.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:04 smithi195 ceph-mon[33205]: pgmap v117: 73 pgs: 73 active+clean; 799 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 6.4 KiB/s wr, 109 op/s 2024-09-13T14:13:06.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:06 smithi102 ceph-mon[29355]: pgmap v118: 73 pgs: 73 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 519 op/s 2024-09-13T14:13:06.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:06 smithi102 ceph-mon[40543]: pgmap v118: 73 pgs: 73 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 519 op/s 2024-09-13T14:13:06.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:06 smithi195 ceph-mon[33205]: pgmap v118: 73 pgs: 73 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 519 op/s 2024-09-13T14:13:07.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:13:07 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:13:07] "GET /metrics HTTP/1.1" 200 33741 "" "Prometheus/2.43.0" 2024-09-13T14:13:08.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:08 smithi102 ceph-mon[40543]: pgmap v119: 73 pgs: 73 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.0 MiB/s rd, 1.4 MiB/s wr, 391 op/s 2024-09-13T14:13:08.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:08 smithi102 ceph-mon[40543]: osdmap e87: 8 total, 8 up, 8 in 2024-09-13T14:13:08.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:08 smithi102 ceph-mon[29355]: pgmap v119: 73 pgs: 73 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.0 MiB/s rd, 1.4 MiB/s wr, 391 op/s 2024-09-13T14:13:08.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:08 smithi102 ceph-mon[29355]: osdmap e87: 8 total, 8 up, 8 in 2024-09-13T14:13:08.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:08 smithi195 ceph-mon[33205]: pgmap v119: 73 pgs: 73 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.0 MiB/s rd, 1.4 MiB/s wr, 391 op/s 2024-09-13T14:13:08.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:08 smithi195 ceph-mon[33205]: osdmap e87: 8 total, 8 up, 8 in 2024-09-13T14:13:09.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:09 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:13:09.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:09 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:13:09.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:09 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:13:10.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:10 smithi102 ceph-mon[29355]: pgmap v121: 73 pgs: 73 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1010 KiB/s rd, 1.4 MiB/s wr, 322 op/s 2024-09-13T14:13:10.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:10 smithi102 ceph-mon[40543]: pgmap v121: 73 pgs: 73 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1010 KiB/s rd, 1.4 MiB/s wr, 322 op/s 2024-09-13T14:13:10.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:10 smithi195 ceph-mon[33205]: pgmap v121: 73 pgs: 73 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1010 KiB/s rd, 1.4 MiB/s wr, 322 op/s 2024-09-13T14:13:12.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:12 smithi102 ceph-mon[29355]: pgmap v122: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 1006 KiB/s rd, 1.4 MiB/s wr, 321 op/s 2024-09-13T14:13:12.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:12 smithi102 ceph-mon[40543]: pgmap v122: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 1006 KiB/s rd, 1.4 MiB/s wr, 321 op/s 2024-09-13T14:13:12.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:12 smithi195 ceph-mon[33205]: pgmap v122: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 1006 KiB/s rd, 1.4 MiB/s wr, 321 op/s 2024-09-13T14:13:14.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:14 smithi102 ceph-mon[29355]: pgmap v123: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 808 KiB/s rd, 1.1 MiB/s wr, 258 op/s 2024-09-13T14:13:14.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:14 smithi102 ceph-mon[40543]: pgmap v123: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 808 KiB/s rd, 1.1 MiB/s wr, 258 op/s 2024-09-13T14:13:14.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:14 smithi195 ceph-mon[33205]: pgmap v123: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 808 KiB/s rd, 1.1 MiB/s wr, 258 op/s 2024-09-13T14:13:16.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:16 smithi102 ceph-mon[29355]: pgmap v124: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:16.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:16 smithi102 ceph-mon[40543]: pgmap v124: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:16.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:16 smithi195 ceph-mon[33205]: pgmap v124: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:17.421 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:13:17 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:13:17] "GET /metrics HTTP/1.1" 200 33752 "" "Prometheus/2.43.0" 2024-09-13T14:13:18.068 INFO:teuthology.orchestra.run.smithi102.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:13:18.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:18 smithi102 ceph-mon[29355]: pgmap v125: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:18.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:18 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:13:18.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:18 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:13:18.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:18 smithi102 ceph-mon[40543]: pgmap v125: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:18.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:18 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:13:18.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:18 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:13:18.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:18 smithi195 ceph-mon[33205]: pgmap v125: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:18.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:18 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:13:18.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:18 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:13:18.816 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-09-13T14:13:19.353 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:13:19.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:19 smithi102 ceph-mon[29355]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:13:19.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:19 smithi102 ceph-mon[40543]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:13:19.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:19 smithi195 ceph-mon[33205]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:13:20.332 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:20 smithi102 ceph-mon[29355]: pgmap v126: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:20.332 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:20 smithi102 ceph-mon[40543]: pgmap v126: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:20.449 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:20 smithi195 ceph-mon[33205]: pgmap v126: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:21.490 INFO:teuthology.orchestra.run.smithi102.stdout:true 2024-09-13T14:13:22.036 INFO:teuthology.orchestra.run.smithi102.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-13T14:13:22.036 INFO:teuthology.orchestra.run.smithi102.stdout:alertmanager.a smithi102 *:9093,9094 running (2m) 77s ago 2m 16.7M - 0.25.0 c8568f914cd2 fadcd26e5395 2024-09-13T14:13:22.036 INFO:teuthology.orchestra.run.smithi102.stdout:grafana.a smithi195 *:3000 running (2m) 0s ago 2m 86.8M - 9.4.7 954c08fa6188 4e2eb92e9f89 2024-09-13T14:13:22.037 INFO:teuthology.orchestra.run.smithi102.stdout:mds.foo.smithi102.njetxr smithi102 running (105s) 77s ago 104s 16.5M - 18.2.4-843-g2c01458a c129f5a2fd4e c9211cc725ed 2024-09-13T14:13:22.037 INFO:teuthology.orchestra.run.smithi102.stdout:mds.foo.smithi195.yqzrxi smithi195 running (102s) 0s ago 102s 15.6M - 18.2.4-843-g2c01458a c129f5a2fd4e eea37a7230fd 2024-09-13T14:13:22.037 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.x smithi195 *:8443,8765 running (6m) 0s ago 6m 439M - 18.2.4-843-g2c01458a c129f5a2fd4e 4a459ca29347 2024-09-13T14:13:22.037 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.y smithi102 *:9283,8765 running (8m) 77s ago 8m 482M - 18.2.4-843-g2c01458a c129f5a2fd4e 4b4d7bb68ef4 2024-09-13T14:13:22.037 INFO:teuthology.orchestra.run.smithi102.stdout:mon.a smithi102 running (8m) 77s ago 8m 45.4M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e 22c604c4fd02 2024-09-13T14:13:22.037 INFO:teuthology.orchestra.run.smithi102.stdout:mon.b smithi195 running (7m) 0s ago 7m 39.7M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e 608ec4847676 2024-09-13T14:13:22.037 INFO:teuthology.orchestra.run.smithi102.stdout:mon.c smithi102 running (7m) 77s ago 7m 36.6M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e 6c1fd3099a05 2024-09-13T14:13:22.037 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.a smithi102 *:9100 running (2m) 77s ago 2m 15.4M - 1.5.0 0da6a335fe13 5538bdf011a3 2024-09-13T14:13:22.037 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.b smithi195 *:9100 running (2m) 0s ago 2m 8274k - 1.5.0 0da6a335fe13 138397d5e129 2024-09-13T14:13:22.037 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0 smithi102 running (6m) 77s ago 6m 51.7M 4016M 18.2.4-843-g2c01458a c129f5a2fd4e b97482fb8cf9 2024-09-13T14:13:22.037 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1 smithi102 running (6m) 77s ago 6m 41.8M 4016M 18.2.4-843-g2c01458a c129f5a2fd4e 8a67356be7a2 2024-09-13T14:13:22.037 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2 smithi102 running (5m) 77s ago 5m 41.8M 4016M 18.2.4-843-g2c01458a c129f5a2fd4e 1800986ac838 2024-09-13T14:13:22.038 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3 smithi102 running (5m) 77s ago 5m 50.2M 4016M 18.2.4-843-g2c01458a c129f5a2fd4e 968246da9da2 2024-09-13T14:13:22.038 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4 smithi195 running (5m) 0s ago 5m 47.0M 4016M 18.2.4-843-g2c01458a c129f5a2fd4e 515307701d73 2024-09-13T14:13:22.038 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5 smithi195 running (4m) 0s ago 4m 51.2M 4016M 18.2.4-843-g2c01458a c129f5a2fd4e 176ba2b3bd59 2024-09-13T14:13:22.038 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6 smithi195 running (4m) 0s ago 4m 53.6M 4016M 18.2.4-843-g2c01458a c129f5a2fd4e 9cd070a6dd3d 2024-09-13T14:13:22.038 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7 smithi195 running (3m) 0s ago 3m 50.8M 4016M 18.2.4-843-g2c01458a c129f5a2fd4e a6b698c659ca 2024-09-13T14:13:22.038 INFO:teuthology.orchestra.run.smithi102.stdout:prometheus.a smithi195 *:9095 running (115s) 0s ago 3m 42.7M - 2.43.0 a07b618ecd1d f526a9818eae 2024-09-13T14:13:22.371 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:13:22.371 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": { 2024-09-13T14:13:22.371 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 3 2024-09-13T14:13:22.371 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:13:22.371 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": { 2024-09-13T14:13:22.371 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 2 2024-09-13T14:13:22.371 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:13:22.372 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": { 2024-09-13T14:13:22.372 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 8 2024-09-13T14:13:22.372 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:13:22.372 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": { 2024-09-13T14:13:22.372 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 2 2024-09-13T14:13:22.372 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:13:22.372 INFO:teuthology.orchestra.run.smithi102.stdout: "overall": { 2024-09-13T14:13:22.372 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 15 2024-09-13T14:13:22.372 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:13:22.372 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:13:22.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:22 smithi102 ceph-mon[29355]: pgmap v127: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:22.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:22 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:13:22.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:22 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:13:22.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:22 smithi102 ceph-mon[40543]: pgmap v127: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:22.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:22 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:13:22.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:22 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:13:22.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:22 smithi195 ceph-mon[33205]: pgmap v127: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:22.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:22 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:13:22.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:22 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:13:22.651 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:13:22.651 INFO:teuthology.orchestra.run.smithi102.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9", 2024-09-13T14:13:22.651 INFO:teuthology.orchestra.run.smithi102.stdout: "in_progress": true, 2024-09-13T14:13:22.651 INFO:teuthology.orchestra.run.smithi102.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-09-13T14:13:22.651 INFO:teuthology.orchestra.run.smithi102.stdout: "services_complete": [], 2024-09-13T14:13:22.651 INFO:teuthology.orchestra.run.smithi102.stdout: "progress": "", 2024-09-13T14:13:22.651 INFO:teuthology.orchestra.run.smithi102.stdout: "message": "", 2024-09-13T14:13:22.651 INFO:teuthology.orchestra.run.smithi102.stdout: "is_paused": false 2024-09-13T14:13:22.651 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:13:23.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:23 smithi102 ceph-mon[29355]: from='client.25367 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:13:23.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:23 smithi102 ceph-mon[29355]: from='client.15480 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:13:23.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:23 smithi102 ceph-mon[29355]: from='client.15486 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:13:23.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:23 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/3141435906' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:13:23.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:23 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:13:23.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:23 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:13:23.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:23 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:13:23.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:23 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:13:23.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:23 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:13:23.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:23 smithi102 ceph-mon[40543]: from='client.25367 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:13:23.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:23 smithi102 ceph-mon[40543]: from='client.15480 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:13:23.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:23 smithi102 ceph-mon[40543]: from='client.15486 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:13:23.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:23 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/3141435906' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:13:23.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:23 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:13:23.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:23 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:13:23.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:23 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:13:23.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:23 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:13:23.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:23 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:13:23.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:23 smithi195 ceph-mon[33205]: from='client.25367 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:13:23.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:23 smithi195 ceph-mon[33205]: from='client.15480 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:13:23.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:23 smithi195 ceph-mon[33205]: from='client.15486 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:13:23.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:23 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/3141435906' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:13:23.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:23 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:13:23.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:23 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:13:23.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:23 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:13:23.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:23 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:13:23.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:23 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:13:24.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:24 smithi102 ceph-mon[40543]: from='client.25391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:13:24.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:24 smithi102 ceph-mon[40543]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:13:24.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:24 smithi102 ceph-mon[40543]: pgmap v128: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:24.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:24 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:13:24.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:24 smithi102 ceph-mon[29355]: from='client.25391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:13:24.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:24 smithi102 ceph-mon[29355]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:13:24.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:24 smithi102 ceph-mon[29355]: pgmap v128: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:24.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:24 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:13:24.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:24 smithi195 ceph-mon[33205]: from='client.25391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:13:24.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:24 smithi195 ceph-mon[33205]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:13:24.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:24 smithi195 ceph-mon[33205]: pgmap v128: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:24.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:24 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:13:25.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:25 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:13:25.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:25 smithi195 ceph-mon[33205]: Upgrade: Target is version 19.3.0-4697-gd58f7306 (unknown) 2024-09-13T14:13:25.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:25 smithi195 ceph-mon[33205]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, digests ['quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f'] 2024-09-13T14:13:25.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:25 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:13:25.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:25 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:13:25.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:25 smithi195 ceph-mon[33205]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-13T14:13:25.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:25 smithi195 ceph-mon[33205]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f on smithi195 2024-09-13T14:13:25.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:25 smithi195 ceph-mon[33205]: pgmap v129: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:25.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:25 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:13:25.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:25 smithi102 ceph-mon[29355]: Upgrade: Target is version 19.3.0-4697-gd58f7306 (unknown) 2024-09-13T14:13:25.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:25 smithi102 ceph-mon[29355]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, digests ['quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f'] 2024-09-13T14:13:25.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:25 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:13:25.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:25 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:13:25.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:25 smithi102 ceph-mon[29355]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-13T14:13:25.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:25 smithi102 ceph-mon[29355]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f on smithi195 2024-09-13T14:13:25.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:25 smithi102 ceph-mon[29355]: pgmap v129: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:25.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:25 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:13:25.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:25 smithi102 ceph-mon[40543]: Upgrade: Target is version 19.3.0-4697-gd58f7306 (unknown) 2024-09-13T14:13:25.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:25 smithi102 ceph-mon[40543]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, digests ['quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f'] 2024-09-13T14:13:25.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:25 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:13:25.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:25 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:13:25.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:25 smithi102 ceph-mon[40543]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-13T14:13:25.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:25 smithi102 ceph-mon[40543]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f on smithi195 2024-09-13T14:13:25.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:25 smithi102 ceph-mon[40543]: pgmap v129: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:27.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:13:27 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:13:27] "GET /metrics HTTP/1.1" 200 33749 "" "Prometheus/2.43.0" 2024-09-13T14:13:28.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:28 smithi102 ceph-mon[40543]: pgmap v130: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:28.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:28 smithi102 ceph-mon[29355]: pgmap v130: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:28.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:28 smithi195 ceph-mon[33205]: pgmap v130: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:30.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:30 smithi102 ceph-mon[29355]: pgmap v131: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:30.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:30 smithi102 ceph-mon[40543]: pgmap v131: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:30.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:30 smithi195 ceph-mon[33205]: pgmap v131: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:32.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:32 smithi102 ceph-mon[29355]: pgmap v132: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:32.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:32 smithi102 ceph-mon[40543]: pgmap v132: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:32.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:32 smithi195 ceph-mon[33205]: pgmap v132: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:34.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:34 smithi102 ceph-mon[29355]: pgmap v133: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:34.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:34 smithi102 ceph-mon[40543]: pgmap v133: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:34.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:34 smithi195 ceph-mon[33205]: pgmap v133: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:36.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:36 smithi102 ceph-mon[29355]: pgmap v134: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:36.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:36 smithi102 ceph-mon[40543]: pgmap v134: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:36.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:36 smithi195 ceph-mon[33205]: pgmap v134: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:37.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:13:37 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:13:37] "GET /metrics HTTP/1.1" 200 33749 "" "Prometheus/2.43.0" 2024-09-13T14:13:38.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:38 smithi102 ceph-mon[29355]: pgmap v135: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:38.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:38 smithi102 ceph-mon[40543]: pgmap v135: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:38.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:38 smithi195 ceph-mon[33205]: pgmap v135: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:39.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:39 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:13:39.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:39 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:13:39.501 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:39 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:13:40.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:40 smithi102 ceph-mon[29355]: pgmap v136: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:40.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:40 smithi102 ceph-mon[40543]: pgmap v136: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:40.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:40 smithi195 ceph-mon[33205]: pgmap v136: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:42.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:42 smithi102 ceph-mon[29355]: pgmap v137: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:42.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:42 smithi102 ceph-mon[40543]: pgmap v137: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:42.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:42 smithi195 ceph-mon[33205]: pgmap v137: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:44.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:44 smithi195 ceph-mon[33205]: pgmap v138: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:44.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:44 smithi102 ceph-mon[29355]: pgmap v138: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:44.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:44 smithi102 ceph-mon[40543]: pgmap v138: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:46.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:46 smithi195 ceph-mon[33205]: pgmap v139: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:46.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:46 smithi102 ceph-mon[29355]: pgmap v139: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:46.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:46 smithi102 ceph-mon[40543]: pgmap v139: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:47.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:13:47 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:13:47] "GET /metrics HTTP/1.1" 200 33750 "" "Prometheus/2.43.0" 2024-09-13T14:13:48.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:48 smithi195 ceph-mon[33205]: pgmap v140: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:48.569 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:48 smithi102 ceph-mon[29355]: pgmap v140: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:48.569 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:48 smithi102 ceph-mon[40543]: pgmap v140: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:50.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:50 smithi195 ceph-mon[33205]: pgmap v141: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:50.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:50 smithi102 ceph-mon[29355]: pgmap v141: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:50.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:50 smithi102 ceph-mon[40543]: pgmap v141: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:52.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:52 smithi195 ceph-mon[33205]: pgmap v142: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:52.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:52 smithi102 ceph-mon[29355]: pgmap v142: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:52.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:52 smithi102 ceph-mon[40543]: pgmap v142: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:52.941 INFO:teuthology.orchestra.run.smithi102.stdout:true 2024-09-13T14:13:53.476 INFO:teuthology.orchestra.run.smithi102.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-13T14:13:53.476 INFO:teuthology.orchestra.run.smithi102.stdout:alertmanager.a smithi102 *:9093,9094 running (2m) 30s ago 3m 27.8M - 0.25.0 c8568f914cd2 fadcd26e5395 2024-09-13T14:13:53.476 INFO:teuthology.orchestra.run.smithi102.stdout:grafana.a smithi195 *:3000 running (3m) 31s ago 3m 86.8M - 9.4.7 954c08fa6188 4e2eb92e9f89 2024-09-13T14:13:53.476 INFO:teuthology.orchestra.run.smithi102.stdout:mds.foo.smithi102.njetxr smithi102 running (2m) 30s ago 2m 17.3M - 18.2.4-843-g2c01458a c129f5a2fd4e c9211cc725ed 2024-09-13T14:13:53.476 INFO:teuthology.orchestra.run.smithi102.stdout:mds.foo.smithi195.yqzrxi smithi195 running (2m) 31s ago 2m 15.6M - 18.2.4-843-g2c01458a c129f5a2fd4e eea37a7230fd 2024-09-13T14:13:53.476 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.x smithi195 *:8443,8765 running (7m) 31s ago 7m 439M - 18.2.4-843-g2c01458a c129f5a2fd4e 4a459ca29347 2024-09-13T14:13:53.476 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.y smithi102 *:9283,8765 running (8m) 30s ago 8m 488M - 18.2.4-843-g2c01458a c129f5a2fd4e 4b4d7bb68ef4 2024-09-13T14:13:53.477 INFO:teuthology.orchestra.run.smithi102.stdout:mon.a smithi102 running (9m) 30s ago 9m 47.7M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e 22c604c4fd02 2024-09-13T14:13:53.477 INFO:teuthology.orchestra.run.smithi102.stdout:mon.b smithi195 running (7m) 31s ago 7m 39.7M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e 608ec4847676 2024-09-13T14:13:53.477 INFO:teuthology.orchestra.run.smithi102.stdout:mon.c smithi102 running (7m) 30s ago 7m 39.1M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e 6c1fd3099a05 2024-09-13T14:13:53.477 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.a smithi102 *:9100 running (3m) 30s ago 3m 17.2M - 1.5.0 0da6a335fe13 5538bdf011a3 2024-09-13T14:13:53.477 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.b smithi195 *:9100 running (2m) 31s ago 2m 8274k - 1.5.0 0da6a335fe13 138397d5e129 2024-09-13T14:13:53.477 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0 smithi102 running (7m) 30s ago 7m 58.4M 4016M 18.2.4-843-g2c01458a c129f5a2fd4e b97482fb8cf9 2024-09-13T14:13:53.477 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1 smithi102 running (6m) 30s ago 6m 49.9M 4016M 18.2.4-843-g2c01458a c129f5a2fd4e 8a67356be7a2 2024-09-13T14:13:53.477 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2 smithi102 running (6m) 30s ago 6m 47.7M 4016M 18.2.4-843-g2c01458a c129f5a2fd4e 1800986ac838 2024-09-13T14:13:53.477 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3 smithi102 running (5m) 30s ago 5m 54.7M 4016M 18.2.4-843-g2c01458a c129f5a2fd4e 968246da9da2 2024-09-13T14:13:53.477 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4 smithi195 running (5m) 31s ago 5m 47.0M 4016M 18.2.4-843-g2c01458a c129f5a2fd4e 515307701d73 2024-09-13T14:13:53.477 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5 smithi195 running (5m) 31s ago 5m 51.2M 4016M 18.2.4-843-g2c01458a c129f5a2fd4e 176ba2b3bd59 2024-09-13T14:13:53.477 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6 smithi195 running (4m) 31s ago 4m 53.6M 4016M 18.2.4-843-g2c01458a c129f5a2fd4e 9cd070a6dd3d 2024-09-13T14:13:53.477 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7 smithi195 running (4m) 31s ago 4m 50.8M 4016M 18.2.4-843-g2c01458a c129f5a2fd4e a6b698c659ca 2024-09-13T14:13:53.478 INFO:teuthology.orchestra.run.smithi102.stdout:prometheus.a smithi195 *:9095 running (2m) 31s ago 3m 42.7M - 2.43.0 a07b618ecd1d f526a9818eae 2024-09-13T14:13:53.793 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:13:53.793 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": { 2024-09-13T14:13:53.793 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 3 2024-09-13T14:13:53.793 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:13:53.793 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": { 2024-09-13T14:13:53.794 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 2 2024-09-13T14:13:53.794 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:13:53.794 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": { 2024-09-13T14:13:53.794 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 8 2024-09-13T14:13:53.794 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:13:53.794 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": { 2024-09-13T14:13:53.794 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 2 2024-09-13T14:13:53.794 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:13:53.794 INFO:teuthology.orchestra.run.smithi102.stdout: "overall": { 2024-09-13T14:13:53.794 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 15 2024-09-13T14:13:53.794 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:13:53.794 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:13:54.063 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:13:54.063 INFO:teuthology.orchestra.run.smithi102.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f", 2024-09-13T14:13:54.063 INFO:teuthology.orchestra.run.smithi102.stdout: "in_progress": true, 2024-09-13T14:13:54.063 INFO:teuthology.orchestra.run.smithi102.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-09-13T14:13:54.063 INFO:teuthology.orchestra.run.smithi102.stdout: "services_complete": [], 2024-09-13T14:13:54.063 INFO:teuthology.orchestra.run.smithi102.stdout: "progress": "0/2 daemons upgraded", 2024-09-13T14:13:54.063 INFO:teuthology.orchestra.run.smithi102.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f image on host smithi195", 2024-09-13T14:13:54.063 INFO:teuthology.orchestra.run.smithi102.stdout: "is_paused": false 2024-09-13T14:13:54.063 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:13:54.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:54 smithi102 ceph-mon[29355]: from='client.15504 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:13:54.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:54 smithi102 ceph-mon[29355]: pgmap v143: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:54.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:54 smithi102 ceph-mon[29355]: from='client.15510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:13:54.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:54 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/2548578797' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:13:54.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:54 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:13:54.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:54 smithi102 ceph-mon[40543]: from='client.15504 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:13:54.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:54 smithi102 ceph-mon[40543]: pgmap v143: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:54.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:54 smithi102 ceph-mon[40543]: from='client.15510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:13:54.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:54 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/2548578797' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:13:54.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:54 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:13:54.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:54 smithi195 ceph-mon[33205]: from='client.15504 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:13:54.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:54 smithi195 ceph-mon[33205]: pgmap v143: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:54.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:54 smithi195 ceph-mon[33205]: from='client.15510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:13:54.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:54 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/2548578797' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:13:54.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:54 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:13:55.369 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:55 smithi195 ceph-mon[33205]: from='client.25409 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:13:55.369 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:55 smithi195 ceph-mon[33205]: from='client.25421 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:13:55.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:55 smithi102 ceph-mon[29355]: from='client.25409 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:13:55.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:55 smithi102 ceph-mon[29355]: from='client.25421 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:13:55.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:55 smithi102 ceph-mon[40543]: from='client.25409 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:13:55.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:55 smithi102 ceph-mon[40543]: from='client.25421 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:13:56.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:56 smithi195 ceph-mon[33205]: pgmap v144: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:56.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:56 smithi102 ceph-mon[29355]: pgmap v144: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:56.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:56 smithi102 ceph-mon[40543]: pgmap v144: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:57.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:13:57 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:13:57] "GET /metrics HTTP/1.1" 200 33744 "" "Prometheus/2.43.0" 2024-09-13T14:13:57.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:57 smithi102 ceph-mon[40543]: Upgrade: Updating mgr.x 2024-09-13T14:13:57.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:57 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:13:57.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:57 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:13:57.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:57 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:13:57.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:57 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:13:57.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:57 smithi102 ceph-mon[40543]: Deploying daemon mgr.x on smithi195 2024-09-13T14:13:57.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:57 smithi102 ceph-mon[29355]: Upgrade: Updating mgr.x 2024-09-13T14:13:57.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:57 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:13:57.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:57 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:13:57.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:57 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:13:57.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:57 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:13:57.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:57 smithi102 ceph-mon[29355]: Deploying daemon mgr.x on smithi195 2024-09-13T14:13:57.685 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:57 smithi195 ceph-mon[33205]: Upgrade: Updating mgr.x 2024-09-13T14:13:57.686 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:57 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:13:57.686 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:57 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:13:57.686 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:57 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:13:57.686 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:57 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:13:57.686 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:57 smithi195 ceph-mon[33205]: Deploying daemon mgr.x on smithi195 2024-09-13T14:13:58.247 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:13:57 smithi195 systemd[1]: Stopping Ceph mgr.x for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T14:13:58.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:13:58 smithi195 ceph-mon[33205]: pgmap v145: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:58.497 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:13:58 smithi195 podman[69147]: 2024-09-13 14:13:58.246296361 +0000 UTC m=+0.124297345 container died 4a459ca2934735dbf2db60e427dbcf98d6cc9998ba1ec43b73c6a5023d1e2c8b (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x, RELEASE=reef-2c01458, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-13T14:13:58.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:13:58 smithi102 ceph-mon[29355]: pgmap v145: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:58.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:13:58 smithi102 ceph-mon[40543]: pgmap v145: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:13:58.883 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:13:58 smithi195 podman[69147]: 2024-09-13 14:13:58.647994121 +0000 UTC m=+0.525995090 container cleanup 4a459ca2934735dbf2db60e427dbcf98d6cc9998ba1ec43b73c6a5023d1e2c8b (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-2c01458) 2024-09-13T14:13:58.883 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:13:58 smithi195 bash[69147]: ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x 2024-09-13T14:13:59.231 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:13:58 smithi195 podman[69164]: 2024-09-13 14:13:58.882887984 +0000 UTC m=+0.628075260 container remove 4a459ca2934735dbf2db60e427dbcf98d6cc9998ba1ec43b73c6a5023d1e2c8b (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=reef-2c01458, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-09-13T14:13:59.231 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:13:58 smithi195 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-09-13T14:13:59.231 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:13:59 smithi195 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-09-13T14:13:59.231 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:13:59 smithi195 systemd[1]: Stopped Ceph mgr.x for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:13:59.232 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:13:59 smithi195 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@mgr.x.service: Consumed 10.924s CPU time. 2024-09-13T14:13:59.497 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:13:59 smithi195 systemd[1]: Starting Ceph mgr.x for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T14:13:59.497 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:13:59 smithi195 podman[69298]: 2024-09-13 14:13:59.344667963 +0000 UTC m=+0.016068035 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:13:59.498 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:13:59 smithi195 podman[69298]: 2024-09-13 14:13:59.467211153 +0000 UTC m=+0.138611217 container create 2d56352abb141fcda4a606dbb4971233f7350801edb35cb1c88112c18a4b174d (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-aclamk-cbt-combined-d58f730, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T14:13:59.997 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:13:59 smithi195 podman[69298]: 2024-09-13 14:13:59.728029129 +0000 UTC m=+0.399429198 container init 2d56352abb141fcda4a606dbb4971233f7350801edb35cb1c88112c18a4b174d (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x, ceph=True, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-13T14:13:59.997 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:13:59 smithi195 podman[69298]: 2024-09-13 14:13:59.733403497 +0000 UTC m=+0.404803557 container start 2d56352abb141fcda4a606dbb4971233f7350801edb35cb1c88112c18a4b174d (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-13T14:13:59.997 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:13:59 smithi195 ceph-mgr[69336]: -- 172.21.15.195:0/4153809016 <== mon.1 v2:172.21.15.195:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ee3f4f50e0 con 0x55ee3f4d2800 2024-09-13T14:13:59.997 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:13:59 smithi195 bash[69298]: 2d56352abb141fcda4a606dbb4971233f7350801edb35cb1c88112c18a4b174d 2024-09-13T14:13:59.998 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:13:59 smithi195 systemd[1]: Started Ceph mgr.x for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:14:00.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:00 smithi102 ceph-mon[29355]: pgmap v146: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:00.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:00 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:14:00.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:00 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:14:00.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:00 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:14:00.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:00 smithi102 ceph-mon[40543]: pgmap v146: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:00.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:00 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:14:00.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:00 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:14:00.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:00 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:14:00.677 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:00 smithi195 ceph-mon[33205]: pgmap v146: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:00.677 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:00 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:14:00.677 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:00 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:14:00.678 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:00 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:14:02.657 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:02 smithi195 ceph-mon[33205]: pgmap v147: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:02.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:02 smithi102 ceph-mon[29355]: pgmap v147: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:02.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:02 smithi102 ceph-mon[40543]: pgmap v147: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:04.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:04 smithi102 ceph-mon[29355]: pgmap v148: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:04.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:04 smithi102 ceph-mon[40543]: pgmap v148: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:04.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:04 smithi195 ceph-mon[33205]: pgmap v148: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:05.333 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:05 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: [13/Sep/2024:14:14:05] ENGINE Bus STARTING 2024-09-13T14:14:05.333 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:05 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: CherryPy Checker: 2024-09-13T14:14:05.333 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:05 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: The Application mounted at '' has an empty config. 2024-09-13T14:14:05.333 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:05 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: 2024-09-13T14:14:05.333 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:05 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: [13/Sep/2024:14:14:05] ENGINE Serving on http://:::9283 2024-09-13T14:14:05.333 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:05 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: [13/Sep/2024:14:14:05] ENGINE Bus STARTED 2024-09-13T14:14:05.593 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:05 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:14:05.593 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:05 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:14:05.593 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:05 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:14:05.594 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:05 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:14:05.594 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:05 smithi195 ceph-mon[33205]: Standby manager daemon x restarted 2024-09-13T14:14:05.594 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:05 smithi195 ceph-mon[33205]: Standby manager daemon x started 2024-09-13T14:14:05.594 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:05 smithi195 ceph-mon[33205]: from='mgr.? 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:14:05.594 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:05 smithi195 ceph-mon[33205]: from='mgr.? 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:14:05.594 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:05 smithi195 ceph-mon[33205]: from='mgr.? 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:14:05.594 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:05 smithi195 ceph-mon[33205]: from='mgr.? 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:14:05.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:05 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:14:05.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:05 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:14:05.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:05 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:14:05.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:05 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:14:05.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:05 smithi102 ceph-mon[29355]: Standby manager daemon x restarted 2024-09-13T14:14:05.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:05 smithi102 ceph-mon[29355]: Standby manager daemon x started 2024-09-13T14:14:05.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:05 smithi102 ceph-mon[29355]: from='mgr.? 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:14:05.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:05 smithi102 ceph-mon[29355]: from='mgr.? 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:14:05.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:05 smithi102 ceph-mon[29355]: from='mgr.? 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:14:05.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:05 smithi102 ceph-mon[29355]: from='mgr.? 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:14:05.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:05 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:14:05.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:05 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:14:05.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:05 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:14:05.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:05 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:14:05.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:05 smithi102 ceph-mon[40543]: Standby manager daemon x restarted 2024-09-13T14:14:05.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:05 smithi102 ceph-mon[40543]: Standby manager daemon x started 2024-09-13T14:14:05.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:05 smithi102 ceph-mon[40543]: from='mgr.? 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:14:05.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:05 smithi102 ceph-mon[40543]: from='mgr.? 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:14:05.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:05 smithi102 ceph-mon[40543]: from='mgr.? 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:14:05.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:05 smithi102 ceph-mon[40543]: from='mgr.? 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:14:06.581 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:06 smithi195 ceph-mon[33205]: pgmap v149: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:06.581 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:06 smithi195 ceph-mon[33205]: mgrmap e21: y(active, since 3m), standbys: x 2024-09-13T14:14:06.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:06 smithi102 ceph-mon[29355]: pgmap v149: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:06.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:06 smithi102 ceph-mon[29355]: mgrmap e21: y(active, since 3m), standbys: x 2024-09-13T14:14:06.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:06 smithi102 ceph-mon[40543]: pgmap v149: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:06.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:06 smithi102 ceph-mon[40543]: mgrmap e21: y(active, since 3m), standbys: x 2024-09-13T14:14:07.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:07 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:14:07] "GET /metrics HTTP/1.1" 200 33744 "" "Prometheus/2.43.0" 2024-09-13T14:14:08.176 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:08 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: [13/Sep/2024:14:14:08] ENGINE Bus STOPPING 2024-09-13T14:14:08.397 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: ignoring --setuser ceph since I am not root 2024-09-13T14:14:08.397 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: ignoring --setgroup ceph since I am not root 2024-09-13T14:14:08.397 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:14:08.195+0000 7f6c310c3040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-13T14:14:08.397 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:14:08.251+0000 7f6c310c3040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-13T14:14:08.497 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:08 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: [13/Sep/2024:14:14:08] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-13T14:14:08.497 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:08 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: [13/Sep/2024:14:14:08] ENGINE Bus STOPPED 2024-09-13T14:14:08.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:08 smithi195 ceph-mon[33205]: pgmap v150: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:08.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:08 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:14:08.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:08 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:14:08.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:08 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:08.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:08 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:14:08.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:08 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:14:08.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:08 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:14:08.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:08 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-13T14:14:08.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:08 smithi195 ceph-mon[33205]: Activating manager daemon x 2024-09-13T14:14:08.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:08 smithi195 ceph-mon[33205]: osdmap e88: 8 total, 8 up, 8 in 2024-09-13T14:14:08.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:08 smithi195 ceph-mon[33205]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-13T14:14:08.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:08 smithi195 ceph-mon[33205]: mgrmap e22: x(active, starting, since 0.0094668s) 2024-09-13T14:14:08.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:08 smithi195 ceph-mon[33205]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:14:08.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:08 smithi195 ceph-mon[33205]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:14:08.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:08 smithi195 ceph-mon[33205]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:14:08.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:08 smithi195 ceph-mon[33205]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi195.yqzrxi"}]: dispatch 2024-09-13T14:14:08.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:08 smithi195 ceph-mon[33205]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi102.njetxr"}]: dispatch 2024-09-13T14:14:08.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:08 smithi195 ceph-mon[33205]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:14:08.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:08 smithi195 ceph-mon[33205]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:14:08.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:08 smithi195 ceph-mon[33205]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:14:08.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:08 smithi195 ceph-mon[33205]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:14:08.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:08 smithi195 ceph-mon[33205]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:14:08.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:08 smithi195 ceph-mon[33205]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:14:08.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:08 smithi195 ceph-mon[33205]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:14:08.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:08 smithi195 ceph-mon[33205]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:14:08.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:08 smithi195 ceph-mon[33205]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:14:08.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:08 smithi195 ceph-mon[33205]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:14:08.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:08 smithi195 ceph-mon[33205]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:14:08.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:08 smithi195 ceph-mon[33205]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:14:08.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:08 smithi195 ceph-mon[33205]: Manager daemon x is now available 2024-09-13T14:14:08.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:14:08.395+0000 7f6c310c3040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-13T14:14:08.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:14:08.462+0000 7f6c310c3040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-13T14:14:08.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:14:08.603+0000 7f6c310c3040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:14:08.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[40543]: pgmap v150: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:08.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:14:08.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:14:08.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:08.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:14:08.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:14:08.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:14:08.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-13T14:14:08.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[40543]: Activating manager daemon x 2024-09-13T14:14:08.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[40543]: osdmap e88: 8 total, 8 up, 8 in 2024-09-13T14:14:08.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[40543]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-13T14:14:08.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[40543]: mgrmap e22: x(active, starting, since 0.0094668s) 2024-09-13T14:14:08.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[40543]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:14:08.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[40543]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:14:08.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[40543]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:14:08.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[40543]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi195.yqzrxi"}]: dispatch 2024-09-13T14:14:08.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[40543]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi102.njetxr"}]: dispatch 2024-09-13T14:14:08.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[40543]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:14:08.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[40543]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:14:08.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[40543]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:14:08.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[40543]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:14:08.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[40543]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:14:08.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[40543]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:14:08.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[40543]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:14:08.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[40543]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:14:08.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[40543]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:14:08.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[40543]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:14:08.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[40543]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:14:08.672 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[29355]: pgmap v150: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:08.672 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:14:08.672 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:14:08.672 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:08.672 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:14:08.672 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' 2024-09-13T14:14:08.672 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:14:08.672 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-13T14:14:08.672 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[29355]: Activating manager daemon x 2024-09-13T14:14:08.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[29355]: osdmap e88: 8 total, 8 up, 8 in 2024-09-13T14:14:08.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[29355]: from='mgr.14421 172.21.15.102:0/1351487470' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-13T14:14:08.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[29355]: mgrmap e22: x(active, starting, since 0.0094668s) 2024-09-13T14:14:08.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[29355]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:14:08.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[29355]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:14:08.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[29355]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:14:08.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[29355]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi195.yqzrxi"}]: dispatch 2024-09-13T14:14:08.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[29355]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi102.njetxr"}]: dispatch 2024-09-13T14:14:08.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[29355]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:14:08.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[29355]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:14:08.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[29355]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:14:08.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[29355]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:14:08.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[29355]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:14:08.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[29355]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:14:08.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[29355]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:14:08.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[29355]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:14:08.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[29355]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:14:08.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[29355]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:14:08.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[29355]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:14:08.674 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[40543]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:14:08.674 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[40543]: Manager daemon x is now available 2024-09-13T14:14:08.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[29355]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:14:08.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-mon[29355]: Manager daemon x is now available 2024-09-13T14:14:09.002 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:08 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:14:08.741+0000 7f6c310c3040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-13T14:14:09.224 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:08 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: 2024-09-13T14:14:08.916+0000 7f22312f6640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:14:09.225 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:08 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: 2024-09-13T14:14:08.916+0000 7f22312f6640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:14:09.225 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:08 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: 2024-09-13T14:14:08.916+0000 7f22312f6640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:14:09.225 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:08 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: 2024-09-13T14:14:08.916+0000 7f22312f6640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:14:09.225 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:08 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: 2024-09-13T14:14:08.916+0000 7f22312f6640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:14:09.225 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:08 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: 2024-09-13T14:14:08.916+0000 7f222daef640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:14:09.225 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:08 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: 2024-09-13T14:14:08.916+0000 7f222daef640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:14:09.225 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:08 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: 2024-09-13T14:14:08.916+0000 7f222daef640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:14:09.225 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:08 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: 2024-09-13T14:14:08.916+0000 7f222daef640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:14:09.225 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:08 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: 2024-09-13T14:14:08.916+0000 7f222daef640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:14:09.225 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:09 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: [13/Sep/2024:14:14:09] ENGINE Bus STARTING 2024-09-13T14:14:09.225 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:09 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: [13/Sep/2024:14:14:09] ENGINE Serving on http://:::9283 2024-09-13T14:14:09.225 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:09 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: [13/Sep/2024:14:14:09] ENGINE Bus STARTED 2024-09-13T14:14:09.308 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:09 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:14:09.054+0000 7f6c310c3040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-13T14:14:09.308 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:09 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:14:09.120+0000 7f6c310c3040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-13T14:14:09.308 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:09 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:14:09.306+0000 7f6c310c3040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-13T14:14:10.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:09 smithi102 ceph-mon[40543]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:10.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:09 smithi102 ceph-mon[40543]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:10.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:09 smithi102 ceph-mon[40543]: Checking for cert/key for grafana.a 2024-09-13T14:14:10.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:09 smithi102 ceph-mon[40543]: Migrating grafana.a cert to cert store 2024-09-13T14:14:10.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:09 smithi102 ceph-mon[40543]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:10.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:09 smithi102 ceph-mon[40543]: Migrating grafana.a key to cert store 2024-09-13T14:14:10.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:09 smithi102 ceph-mon[40543]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:10.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:09 smithi102 ceph-mon[40543]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:10.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:09 smithi102 ceph-mon[40543]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:10.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:09 smithi102 ceph-mon[40543]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:14:10.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:09 smithi102 ceph-mon[40543]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:14:10.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:09 smithi102 ceph-mon[40543]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:14:10.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:09 smithi102 ceph-mon[40543]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:14:10.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:09 smithi102 ceph-mon[40543]: from='mgr.25045 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:14:10.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:09 smithi102 ceph-mon[40543]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-13T14:14:10.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:09 smithi102 ceph-mon[40543]: from='mgr.25045 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-13T14:14:10.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:09 smithi102 ceph-mon[40543]: mgrmap e23: x(active, since 1.01238s) 2024-09-13T14:14:10.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:09 smithi102 ceph-mon[40543]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:14:10.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:09 smithi102 ceph-mon[40543]: pgmap v3: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:10.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:09 smithi102 ceph-mon[40543]: Deploying cephadm binary to smithi102 2024-09-13T14:14:10.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:09 smithi102 ceph-mon[40543]: Deploying cephadm binary to smithi195 2024-09-13T14:14:10.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:09 smithi102 ceph-mon[29355]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:10.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:09 smithi102 ceph-mon[29355]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:10.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:09 smithi102 ceph-mon[29355]: Checking for cert/key for grafana.a 2024-09-13T14:14:10.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:09 smithi102 ceph-mon[29355]: Migrating grafana.a cert to cert store 2024-09-13T14:14:10.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:09 smithi102 ceph-mon[29355]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:10.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:09 smithi102 ceph-mon[29355]: Migrating grafana.a key to cert store 2024-09-13T14:14:10.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:09 smithi102 ceph-mon[29355]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:10.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:09 smithi102 ceph-mon[29355]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:10.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:09 smithi102 ceph-mon[29355]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:10.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:09 smithi102 ceph-mon[29355]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:14:10.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:09 smithi102 ceph-mon[29355]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:14:10.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:09 smithi102 ceph-mon[29355]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:14:10.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:09 smithi102 ceph-mon[29355]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:14:10.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:09 smithi102 ceph-mon[29355]: from='mgr.25045 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:14:10.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:09 smithi102 ceph-mon[29355]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-13T14:14:10.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:09 smithi102 ceph-mon[29355]: from='mgr.25045 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-13T14:14:10.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:09 smithi102 ceph-mon[29355]: mgrmap e23: x(active, since 1.01238s) 2024-09-13T14:14:10.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:09 smithi102 ceph-mon[29355]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:14:10.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:09 smithi102 ceph-mon[29355]: pgmap v3: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:10.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:09 smithi102 ceph-mon[29355]: Deploying cephadm binary to smithi102 2024-09-13T14:14:10.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:09 smithi102 ceph-mon[29355]: Deploying cephadm binary to smithi195 2024-09-13T14:14:10.248 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:10 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: 2024-09-13T14:14:10.070+0000 7f220e837640 -1 log_channel(cephadm) log [ERR] : [13/Sep/2024:14:14:10] ENGINE Error in HTTPServer.serve 2024-09-13T14:14:10.248 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:10 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: Traceback (most recent call last): 2024-09-13T14:14:10.248 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:10 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:14:10.249 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:10 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: self._connections.run(self.expiration_interval) 2024-09-13T14:14:10.249 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:10 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:14:10.249 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:10 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: self._run(expiration_interval) 2024-09-13T14:14:10.249 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:10 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:14:10.249 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:10 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:14:10.249 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:10 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:14:10.249 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:10 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:14:10.249 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:10 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:14:10.249 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:10 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: s = self.context.wrap_socket( 2024-09-13T14:14:10.249 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:10 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:14:10.249 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:10 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: return self.sslsocket_class._create( 2024-09-13T14:14:10.249 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:10 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:14:10.249 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:10 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: self.do_handshake() 2024-09-13T14:14:10.250 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:10 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:14:10.250 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:10 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: self._sslobj.do_handshake() 2024-09-13T14:14:10.250 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:10 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:14:10.250 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:10 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: 2024-09-13T14:14:10.250 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:09 smithi195 ceph-mon[33205]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:10.250 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:09 smithi195 ceph-mon[33205]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:10.250 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:09 smithi195 ceph-mon[33205]: Checking for cert/key for grafana.a 2024-09-13T14:14:10.250 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:09 smithi195 ceph-mon[33205]: Migrating grafana.a cert to cert store 2024-09-13T14:14:10.250 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:09 smithi195 ceph-mon[33205]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:10.250 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:09 smithi195 ceph-mon[33205]: Migrating grafana.a key to cert store 2024-09-13T14:14:10.250 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:09 smithi195 ceph-mon[33205]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:10.250 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:09 smithi195 ceph-mon[33205]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:10.251 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:09 smithi195 ceph-mon[33205]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:10.251 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:09 smithi195 ceph-mon[33205]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:14:10.251 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:09 smithi195 ceph-mon[33205]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:14:10.251 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:09 smithi195 ceph-mon[33205]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:14:10.251 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:09 smithi195 ceph-mon[33205]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:14:10.251 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:09 smithi195 ceph-mon[33205]: from='mgr.25045 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:14:10.251 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:09 smithi195 ceph-mon[33205]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-13T14:14:10.251 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:09 smithi195 ceph-mon[33205]: from='mgr.25045 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-13T14:14:10.251 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:09 smithi195 ceph-mon[33205]: mgrmap e23: x(active, since 1.01238s) 2024-09-13T14:14:10.251 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:09 smithi195 ceph-mon[33205]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:14:10.251 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:09 smithi195 ceph-mon[33205]: pgmap v3: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:10.251 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:09 smithi195 ceph-mon[33205]: Deploying cephadm binary to smithi102 2024-09-13T14:14:10.251 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:09 smithi195 ceph-mon[33205]: Deploying cephadm binary to smithi195 2024-09-13T14:14:10.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:10 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:14:10.187+0000 7f6c310c3040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-13T14:14:10.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:10 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:14:10.251+0000 7f6c310c3040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-13T14:14:11.237 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:10 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:14:10.958+0000 7f6c310c3040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-13T14:14:11.237 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:11 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:14:11.079+0000 7f6c310c3040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-13T14:14:11.238 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:11 smithi102 ceph-mon[29355]: [13/Sep/2024:14:14:09] ENGINE Bus STARTING 2024-09-13T14:14:11.238 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:11 smithi102 ceph-mon[29355]: [13/Sep/2024:14:14:09] ENGINE Serving on http://172.21.15.195:8765 2024-09-13T14:14:11.238 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:11 smithi102 ceph-mon[29355]: pgmap v4: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:11.238 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:11 smithi102 ceph-mon[29355]: mgrmap e24: x(active, since 2s) 2024-09-13T14:14:11.238 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:11 smithi102 ceph-mon[29355]: [13/Sep/2024:14:14:10] ENGINE Serving on https://172.21.15.195:7150 2024-09-13T14:14:11.238 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:11 smithi102 ceph-mon[29355]: [13/Sep/2024:14:14:10] ENGINE Bus STARTED 2024-09-13T14:14:11.238 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:11 smithi102 ceph-mon[29355]: [13/Sep/2024:14:14:10] ENGINE Error in HTTPServer.serve 2024-09-13T14:14:11.238 INFO:journalctl@ceph.mon.a.smithi102.stdout: Traceback (most recent call last): 2024-09-13T14:14:11.238 INFO:journalctl@ceph.mon.a.smithi102.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:14:11.238 INFO:journalctl@ceph.mon.a.smithi102.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:14:11.238 INFO:journalctl@ceph.mon.a.smithi102.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:14:11.239 INFO:journalctl@ceph.mon.a.smithi102.stdout: self._run(expiration_interval) 2024-09-13T14:14:11.239 INFO:journalctl@ceph.mon.a.smithi102.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:14:11.239 INFO:journalctl@ceph.mon.a.smithi102.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:14:11.239 INFO:journalctl@ceph.mon.a.smithi102.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:14:11.239 INFO:journalctl@ceph.mon.a.smithi102.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:14:11.239 INFO:journalctl@ceph.mon.a.smithi102.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:14:11.239 INFO:journalctl@ceph.mon.a.smithi102.stdout: s = self.context.wrap_socket( 2024-09-13T14:14:11.239 INFO:journalctl@ceph.mon.a.smithi102.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:14:11.239 INFO:journalctl@ceph.mon.a.smithi102.stdout: return self.sslsocket_class._create( 2024-09-13T14:14:11.239 INFO:journalctl@ceph.mon.a.smithi102.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:14:11.239 INFO:journalctl@ceph.mon.a.smithi102.stdout: self.do_handshake() 2024-09-13T14:14:11.239 INFO:journalctl@ceph.mon.a.smithi102.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:14:11.239 INFO:journalctl@ceph.mon.a.smithi102.stdout: self._sslobj.do_handshake() 2024-09-13T14:14:11.239 INFO:journalctl@ceph.mon.a.smithi102.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:14:11.240 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:11 smithi102 ceph-mon[40543]: [13/Sep/2024:14:14:09] ENGINE Bus STARTING 2024-09-13T14:14:11.240 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:11 smithi102 ceph-mon[40543]: [13/Sep/2024:14:14:09] ENGINE Serving on http://172.21.15.195:8765 2024-09-13T14:14:11.240 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:11 smithi102 ceph-mon[40543]: pgmap v4: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:11.240 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:11 smithi102 ceph-mon[40543]: mgrmap e24: x(active, since 2s) 2024-09-13T14:14:11.240 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:11 smithi102 ceph-mon[40543]: [13/Sep/2024:14:14:10] ENGINE Serving on https://172.21.15.195:7150 2024-09-13T14:14:11.240 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:11 smithi102 ceph-mon[40543]: [13/Sep/2024:14:14:10] ENGINE Bus STARTED 2024-09-13T14:14:11.240 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:11 smithi102 ceph-mon[40543]: [13/Sep/2024:14:14:10] ENGINE Error in HTTPServer.serve 2024-09-13T14:14:11.240 INFO:journalctl@ceph.mon.c.smithi102.stdout: Traceback (most recent call last): 2024-09-13T14:14:11.240 INFO:journalctl@ceph.mon.c.smithi102.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:14:11.240 INFO:journalctl@ceph.mon.c.smithi102.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:14:11.240 INFO:journalctl@ceph.mon.c.smithi102.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:14:11.241 INFO:journalctl@ceph.mon.c.smithi102.stdout: self._run(expiration_interval) 2024-09-13T14:14:11.241 INFO:journalctl@ceph.mon.c.smithi102.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:14:11.241 INFO:journalctl@ceph.mon.c.smithi102.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:14:11.241 INFO:journalctl@ceph.mon.c.smithi102.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:14:11.241 INFO:journalctl@ceph.mon.c.smithi102.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:14:11.241 INFO:journalctl@ceph.mon.c.smithi102.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:14:11.241 INFO:journalctl@ceph.mon.c.smithi102.stdout: s = self.context.wrap_socket( 2024-09-13T14:14:11.241 INFO:journalctl@ceph.mon.c.smithi102.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:14:11.241 INFO:journalctl@ceph.mon.c.smithi102.stdout: return self.sslsocket_class._create( 2024-09-13T14:14:11.241 INFO:journalctl@ceph.mon.c.smithi102.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:14:11.241 INFO:journalctl@ceph.mon.c.smithi102.stdout: self.do_handshake() 2024-09-13T14:14:11.241 INFO:journalctl@ceph.mon.c.smithi102.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:14:11.241 INFO:journalctl@ceph.mon.c.smithi102.stdout: self._sslobj.do_handshake() 2024-09-13T14:14:11.242 INFO:journalctl@ceph.mon.c.smithi102.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:14:11.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:11 smithi195 ceph-mon[33205]: [13/Sep/2024:14:14:09] ENGINE Bus STARTING 2024-09-13T14:14:11.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:11 smithi195 ceph-mon[33205]: [13/Sep/2024:14:14:09] ENGINE Serving on http://172.21.15.195:8765 2024-09-13T14:14:11.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:11 smithi195 ceph-mon[33205]: pgmap v4: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:11.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:11 smithi195 ceph-mon[33205]: mgrmap e24: x(active, since 2s) 2024-09-13T14:14:11.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:11 smithi195 ceph-mon[33205]: [13/Sep/2024:14:14:10] ENGINE Serving on https://172.21.15.195:7150 2024-09-13T14:14:11.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:11 smithi195 ceph-mon[33205]: [13/Sep/2024:14:14:10] ENGINE Bus STARTED 2024-09-13T14:14:11.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:11 smithi195 ceph-mon[33205]: [13/Sep/2024:14:14:10] ENGINE Error in HTTPServer.serve 2024-09-13T14:14:11.498 INFO:journalctl@ceph.mon.b.smithi195.stdout: Traceback (most recent call last): 2024-09-13T14:14:11.498 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:14:11.498 INFO:journalctl@ceph.mon.b.smithi195.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:14:11.498 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:14:11.498 INFO:journalctl@ceph.mon.b.smithi195.stdout: self._run(expiration_interval) 2024-09-13T14:14:11.498 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:14:11.498 INFO:journalctl@ceph.mon.b.smithi195.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:14:11.498 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:14:11.498 INFO:journalctl@ceph.mon.b.smithi195.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:14:11.498 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:14:11.498 INFO:journalctl@ceph.mon.b.smithi195.stdout: s = self.context.wrap_socket( 2024-09-13T14:14:11.498 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:14:11.499 INFO:journalctl@ceph.mon.b.smithi195.stdout: return self.sslsocket_class._create( 2024-09-13T14:14:11.499 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:14:11.499 INFO:journalctl@ceph.mon.b.smithi195.stdout: self.do_handshake() 2024-09-13T14:14:11.499 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:14:11.499 INFO:journalctl@ceph.mon.b.smithi195.stdout: self._sslobj.do_handshake() 2024-09-13T14:14:11.499 INFO:journalctl@ceph.mon.b.smithi195.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:14:11.528 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:11 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:14:11.236+0000 7f6c310c3040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:14:11.528 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:11 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:14:11.309+0000 7f6c310c3040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-13T14:14:11.528 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:11 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:14:11.457+0000 7f6c310c3040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-13T14:14:11.528 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:11 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:14:11.526+0000 7f6c310c3040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-13T14:14:11.880 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:11 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:14:11.589+0000 7f6c310c3040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-13T14:14:11.881 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:11 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:14:11.754+0000 7f6c310c3040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-13T14:14:11.881 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:11 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:14:11.821+0000 7f6c310c3040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-13T14:14:12.168 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:11 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:14:11.887+0000 7f6c310c3040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-13T14:14:12.168 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:11 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:14:11.969+0000 7f6c310c3040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-13T14:14:12.168 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:12 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:14:12.027+0000 7f6c310c3040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-13T14:14:12.168 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:12 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:14:12.088+0000 7f6c310c3040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-13T14:14:13.027 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:12 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:14:12.918+0000 7f6c310c3040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-13T14:14:13.027 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:12 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: [13/Sep/2024:14:14:12] ENGINE Bus STARTING 2024-09-13T14:14:13.027 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:12 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: CherryPy Checker: 2024-09-13T14:14:13.027 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:12 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: The Application mounted at '' has an empty config. 2024-09-13T14:14:13.027 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:12 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: 2024-09-13T14:14:13.027 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:13 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: [13/Sep/2024:14:14:13] ENGINE Serving on http://:::9283 2024-09-13T14:14:13.027 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:13 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: [13/Sep/2024:14:14:13] ENGINE Bus STARTED 2024-09-13T14:14:13.408 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:13 smithi195 ceph-mon[33205]: pgmap v5: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:13.408 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:13 smithi195 ceph-mon[33205]: mgrmap e25: x(active, since 4s) 2024-09-13T14:14:13.408 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:13 smithi195 ceph-mon[33205]: Standby manager daemon y started 2024-09-13T14:14:13.408 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:13 smithi195 ceph-mon[33205]: from='mgr.? 172.21.15.102:0/3651418625' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-13T14:14:13.408 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:13 smithi195 ceph-mon[33205]: from='mgr.? 172.21.15.102:0/3651418625' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:14:13.409 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:13 smithi195 ceph-mon[33205]: from='mgr.? 172.21.15.102:0/3651418625' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-13T14:14:13.409 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:13 smithi195 ceph-mon[33205]: from='mgr.? 172.21.15.102:0/3651418625' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:14:13.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:13 smithi102 ceph-mon[29355]: pgmap v5: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:13.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:13 smithi102 ceph-mon[29355]: mgrmap e25: x(active, since 4s) 2024-09-13T14:14:13.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:13 smithi102 ceph-mon[29355]: Standby manager daemon y started 2024-09-13T14:14:13.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:13 smithi102 ceph-mon[29355]: from='mgr.? 172.21.15.102:0/3651418625' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-13T14:14:13.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:13 smithi102 ceph-mon[29355]: from='mgr.? 172.21.15.102:0/3651418625' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:14:13.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:13 smithi102 ceph-mon[29355]: from='mgr.? 172.21.15.102:0/3651418625' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-13T14:14:13.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:13 smithi102 ceph-mon[29355]: from='mgr.? 172.21.15.102:0/3651418625' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:14:13.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:13 smithi102 ceph-mon[40543]: pgmap v5: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:13.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:13 smithi102 ceph-mon[40543]: mgrmap e25: x(active, since 4s) 2024-09-13T14:14:13.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:13 smithi102 ceph-mon[40543]: Standby manager daemon y started 2024-09-13T14:14:13.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:13 smithi102 ceph-mon[40543]: from='mgr.? 172.21.15.102:0/3651418625' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-13T14:14:13.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:13 smithi102 ceph-mon[40543]: from='mgr.? 172.21.15.102:0/3651418625' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:14:13.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:13 smithi102 ceph-mon[40543]: from='mgr.? 172.21.15.102:0/3651418625' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-13T14:14:13.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:13 smithi102 ceph-mon[40543]: from='mgr.? 172.21.15.102:0/3651418625' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:14:14.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:14 smithi102 ceph-mon[40543]: mgrmap e26: x(active, since 5s), standbys: y 2024-09-13T14:14:14.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:14 smithi102 ceph-mon[40543]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:14:14.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:14 smithi102 ceph-mon[40543]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:14.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:14 smithi102 ceph-mon[40543]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:14.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:14 smithi102 ceph-mon[29355]: mgrmap e26: x(active, since 5s), standbys: y 2024-09-13T14:14:14.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:14 smithi102 ceph-mon[29355]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:14:14.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:14 smithi102 ceph-mon[29355]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:14.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:14 smithi102 ceph-mon[29355]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:14.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:14 smithi195 ceph-mon[33205]: mgrmap e26: x(active, since 5s), standbys: y 2024-09-13T14:14:14.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:14 smithi195 ceph-mon[33205]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:14:14.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:14 smithi195 ceph-mon[33205]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:14.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:14 smithi195 ceph-mon[33205]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:15.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:15 smithi102 ceph-mon[40543]: pgmap v6: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:15.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:15 smithi102 ceph-mon[40543]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:15.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:15 smithi102 ceph-mon[40543]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:15.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:15 smithi102 ceph-mon[40543]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:15.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:15 smithi102 ceph-mon[40543]: from='mgr.25045 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:15.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:15 smithi102 ceph-mon[40543]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:15.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:15 smithi102 ceph-mon[40543]: from='mgr.25045 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:15.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:15 smithi102 ceph-mon[40543]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:15.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:15 smithi102 ceph-mon[40543]: from='mgr.25045 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:15.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:15 smithi102 ceph-mon[40543]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:15.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:15 smithi102 ceph-mon[40543]: from='mgr.25045 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:15.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:15 smithi102 ceph-mon[40543]: Adjusting osd_memory_target on smithi102 to 2480M 2024-09-13T14:14:15.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:15 smithi102 ceph-mon[40543]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:15.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:15 smithi102 ceph-mon[40543]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:15.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:15 smithi102 ceph-mon[40543]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:15.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:15 smithi102 ceph-mon[29355]: pgmap v6: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:15.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:15 smithi102 ceph-mon[29355]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:15.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:15 smithi102 ceph-mon[29355]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:15.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:15 smithi102 ceph-mon[29355]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:15.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:15 smithi102 ceph-mon[29355]: from='mgr.25045 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:15.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:15 smithi102 ceph-mon[29355]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:15.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:15 smithi102 ceph-mon[29355]: from='mgr.25045 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:15.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:15 smithi102 ceph-mon[29355]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:15.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:15 smithi102 ceph-mon[29355]: from='mgr.25045 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:15.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:15 smithi102 ceph-mon[29355]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:15.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:15 smithi102 ceph-mon[29355]: from='mgr.25045 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:15.921 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:15 smithi102 ceph-mon[29355]: Adjusting osd_memory_target on smithi102 to 2480M 2024-09-13T14:14:15.921 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:15 smithi102 ceph-mon[29355]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:15.921 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:15 smithi102 ceph-mon[29355]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:15.921 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:15 smithi102 ceph-mon[29355]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:15.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:15 smithi195 ceph-mon[33205]: pgmap v6: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:15.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:15 smithi195 ceph-mon[33205]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:15.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:15 smithi195 ceph-mon[33205]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:15.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:15 smithi195 ceph-mon[33205]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:15.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:15 smithi195 ceph-mon[33205]: from='mgr.25045 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:15.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:15 smithi195 ceph-mon[33205]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:15.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:15 smithi195 ceph-mon[33205]: from='mgr.25045 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:15.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:15 smithi195 ceph-mon[33205]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:15.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:15 smithi195 ceph-mon[33205]: from='mgr.25045 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:15.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:15 smithi195 ceph-mon[33205]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:15.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:15 smithi195 ceph-mon[33205]: from='mgr.25045 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:15.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:15 smithi195 ceph-mon[33205]: Adjusting osd_memory_target on smithi102 to 2480M 2024-09-13T14:14:15.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:15 smithi195 ceph-mon[33205]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:15.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:15 smithi195 ceph-mon[33205]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:15.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:15 smithi195 ceph-mon[33205]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:17.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:16 smithi102 ceph-mon[29355]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:17.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:16 smithi102 ceph-mon[29355]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:17.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:16 smithi102 ceph-mon[29355]: pgmap v7: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-13T14:14:17.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:16 smithi102 ceph-mon[40543]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:17.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:16 smithi102 ceph-mon[40543]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:17.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:16 smithi102 ceph-mon[40543]: pgmap v7: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-13T14:14:17.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:16 smithi195 ceph-mon[33205]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:17.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:16 smithi195 ceph-mon[33205]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:17.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:16 smithi195 ceph-mon[33205]: pgmap v7: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-13T14:14:17.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:17 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:14:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:14:19.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:19 smithi195 ceph-mon[33205]: pgmap v8: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-13T14:14:19.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:19 smithi102 ceph-mon[29355]: pgmap v8: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-13T14:14:19.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:19 smithi102 ceph-mon[40543]: pgmap v8: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-13T14:14:21.176 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:21 smithi195 ceph-mon[33205]: pgmap v9: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-13T14:14:21.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:21 smithi102 ceph-mon[29355]: pgmap v9: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-13T14:14:21.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:21 smithi102 ceph-mon[40543]: pgmap v9: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-13T14:14:23.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:23 smithi102 ceph-mon[29355]: pgmap v10: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-13T14:14:23.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:23 smithi102 ceph-mon[40543]: pgmap v10: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-13T14:14:23.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:23 smithi195 ceph-mon[33205]: pgmap v10: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-13T14:14:24.370 INFO:teuthology.orchestra.run.smithi102.stdout:true 2024-09-13T14:14:24.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:24 smithi102 ceph-mon[29355]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:14:24.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:24 smithi102 ceph-mon[40543]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:14:24.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:24 smithi195 ceph-mon[33205]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:14:24.924 INFO:teuthology.orchestra.run.smithi102.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-13T14:14:24.924 INFO:teuthology.orchestra.run.smithi102.stdout:alertmanager.a smithi102 *:9093,9094 running (3m) 11s ago 4m 27.8M - 0.25.0 c8568f914cd2 fadcd26e5395 2024-09-13T14:14:24.924 INFO:teuthology.orchestra.run.smithi102.stdout:grafana.a smithi195 *:3000 running (3m) 9s ago 3m 87.4M - 9.4.7 954c08fa6188 4e2eb92e9f89 2024-09-13T14:14:24.924 INFO:teuthology.orchestra.run.smithi102.stdout:mds.foo.smithi102.njetxr smithi102 running (2m) 11s ago 2m 18.1M - 18.2.4-843-g2c01458a c129f5a2fd4e c9211cc725ed 2024-09-13T14:14:24.924 INFO:teuthology.orchestra.run.smithi102.stdout:mds.foo.smithi195.yqzrxi smithi195 running (2m) 9s ago 2m 16.4M - 18.2.4-843-g2c01458a c129f5a2fd4e eea37a7230fd 2024-09-13T14:14:24.924 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.x smithi195 *:8443,9283,8765 running (25s) 9s ago 7m 496M - 19.3.0-4697-gd58f7306 fec07016a2b0 2d56352abb14 2024-09-13T14:14:24.924 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.y smithi102 *:9283,8765 running (9m) 11s ago 9m 299M - 18.2.4-843-g2c01458a c129f5a2fd4e 4b4d7bb68ef4 2024-09-13T14:14:24.924 INFO:teuthology.orchestra.run.smithi102.stdout:mon.a smithi102 running (9m) 11s ago 9m 50.4M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e 22c604c4fd02 2024-09-13T14:14:24.924 INFO:teuthology.orchestra.run.smithi102.stdout:mon.b smithi195 running (8m) 9s ago 8m 41.2M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e 608ec4847676 2024-09-13T14:14:24.924 INFO:teuthology.orchestra.run.smithi102.stdout:mon.c smithi102 running (8m) 11s ago 8m 39.7M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e 6c1fd3099a05 2024-09-13T14:14:24.925 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.a smithi102 *:9100 running (3m) 11s ago 3m 17.1M - 1.5.0 0da6a335fe13 5538bdf011a3 2024-09-13T14:14:24.925 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.b smithi195 *:9100 running (3m) 9s ago 3m 15.8M - 1.5.0 0da6a335fe13 138397d5e129 2024-09-13T14:14:24.925 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0 smithi102 running (7m) 11s ago 7m 59.0M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e b97482fb8cf9 2024-09-13T14:14:24.925 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1 smithi102 running (7m) 11s ago 7m 50.2M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 8a67356be7a2 2024-09-13T14:14:24.925 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2 smithi102 running (6m) 11s ago 6m 48.4M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 1800986ac838 2024-09-13T14:14:24.925 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3 smithi102 running (6m) 11s ago 6m 55.4M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 968246da9da2 2024-09-13T14:14:24.925 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4 smithi195 running (6m) 9s ago 6m 47.2M 4016M 18.2.4-843-g2c01458a c129f5a2fd4e 515307701d73 2024-09-13T14:14:24.925 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5 smithi195 running (5m) 9s ago 5m 51.6M 4016M 18.2.4-843-g2c01458a c129f5a2fd4e 176ba2b3bd59 2024-09-13T14:14:24.925 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6 smithi195 running (5m) 9s ago 5m 54.0M 4016M 18.2.4-843-g2c01458a c129f5a2fd4e 9cd070a6dd3d 2024-09-13T14:14:24.925 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7 smithi195 running (4m) 9s ago 4m 51.3M 4016M 18.2.4-843-g2c01458a c129f5a2fd4e a6b698c659ca 2024-09-13T14:14:24.925 INFO:teuthology.orchestra.run.smithi102.stdout:prometheus.a smithi195 *:9095 running (2m) 9s ago 4m 47.4M - 2.43.0 a07b618ecd1d f526a9818eae 2024-09-13T14:14:25.270 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:14:25.270 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": { 2024-09-13T14:14:25.270 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 3 2024-09-13T14:14:25.270 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:14:25.271 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": { 2024-09-13T14:14:25.271 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 1, 2024-09-13T14:14:25.271 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 1 2024-09-13T14:14:25.271 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:14:25.271 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": { 2024-09-13T14:14:25.271 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 8 2024-09-13T14:14:25.271 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:14:25.271 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": { 2024-09-13T14:14:25.271 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 2 2024-09-13T14:14:25.271 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:14:25.271 INFO:teuthology.orchestra.run.smithi102.stdout: "overall": { 2024-09-13T14:14:25.271 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 14, 2024-09-13T14:14:25.271 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 1 2024-09-13T14:14:25.272 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:14:25.272 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:14:25.272 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:25 smithi102 ceph-mon[29355]: pgmap v11: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-13T14:14:25.272 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:25 smithi102 ceph-mon[29355]: from='client.25466 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:14:25.272 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:25 smithi102 ceph-mon[29355]: from='client.25472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:14:25.272 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:25 smithi102 ceph-mon[40543]: pgmap v11: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-13T14:14:25.272 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:25 smithi102 ceph-mon[40543]: from='client.25466 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:14:25.272 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:25 smithi102 ceph-mon[40543]: from='client.25472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:14:25.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:25 smithi195 ceph-mon[33205]: pgmap v11: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-13T14:14:25.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:25 smithi195 ceph-mon[33205]: from='client.25466 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:14:25.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:25 smithi195 ceph-mon[33205]: from='client.25472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:14:25.567 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:14:25.567 INFO:teuthology.orchestra.run.smithi102.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f", 2024-09-13T14:14:25.567 INFO:teuthology.orchestra.run.smithi102.stdout: "in_progress": true, 2024-09-13T14:14:25.567 INFO:teuthology.orchestra.run.smithi102.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-09-13T14:14:25.567 INFO:teuthology.orchestra.run.smithi102.stdout: "services_complete": [], 2024-09-13T14:14:25.567 INFO:teuthology.orchestra.run.smithi102.stdout: "progress": "1/2 daemons upgraded", 2024-09-13T14:14:25.567 INFO:teuthology.orchestra.run.smithi102.stdout: "message": "", 2024-09-13T14:14:25.567 INFO:teuthology.orchestra.run.smithi102.stdout: "is_paused": false 2024-09-13T14:14:25.567 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:14:26.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:26 smithi102 ceph-mon[40543]: from='client.25478 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:14:26.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:26 smithi102 ceph-mon[40543]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:26.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:26 smithi102 ceph-mon[40543]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:26.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:26 smithi102 ceph-mon[40543]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:26.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:26 smithi102 ceph-mon[40543]: from='mgr.25045 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:26.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:26 smithi102 ceph-mon[40543]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:26.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:26 smithi102 ceph-mon[40543]: from='mgr.25045 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:26.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:26 smithi102 ceph-mon[40543]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:26.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:26 smithi102 ceph-mon[40543]: from='mgr.25045 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:26.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:26 smithi102 ceph-mon[40543]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:26.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:26 smithi102 ceph-mon[40543]: from='mgr.25045 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:26.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:26 smithi102 ceph-mon[40543]: Adjusting osd_memory_target on smithi195 to 2480M 2024-09-13T14:14:26.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:26 smithi102 ceph-mon[40543]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:26.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:26 smithi102 ceph-mon[40543]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:26.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:26 smithi102 ceph-mon[40543]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:14:26.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:26 smithi102 ceph-mon[40543]: Updating smithi102:/etc/ceph/ceph.conf 2024-09-13T14:14:26.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:26 smithi102 ceph-mon[40543]: Updating smithi195:/etc/ceph/ceph.conf 2024-09-13T14:14:26.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:26 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/3308548429' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:14:26.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:26 smithi102 ceph-mon[40543]: Updating smithi102:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:14:26.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:26 smithi102 ceph-mon[40543]: Updating smithi195:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:14:26.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:26 smithi102 ceph-mon[40543]: from='client.15600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:14:26.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:26 smithi102 ceph-mon[40543]: Updating smithi102:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:14:26.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:26 smithi102 ceph-mon[40543]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:26.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:26 smithi102 ceph-mon[40543]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:26.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:26 smithi102 ceph-mon[29355]: from='client.25478 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:14:26.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:26 smithi102 ceph-mon[29355]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:26.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:26 smithi102 ceph-mon[29355]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:26.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:26 smithi102 ceph-mon[29355]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:26.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:26 smithi102 ceph-mon[29355]: from='mgr.25045 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:26.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:26 smithi102 ceph-mon[29355]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:26.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:26 smithi102 ceph-mon[29355]: from='mgr.25045 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:26.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:26 smithi102 ceph-mon[29355]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:26.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:26 smithi102 ceph-mon[29355]: from='mgr.25045 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:26.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:26 smithi102 ceph-mon[29355]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:26.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:26 smithi102 ceph-mon[29355]: from='mgr.25045 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:26.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:26 smithi102 ceph-mon[29355]: Adjusting osd_memory_target on smithi195 to 2480M 2024-09-13T14:14:26.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:26 smithi102 ceph-mon[29355]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:26.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:26 smithi102 ceph-mon[29355]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:26.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:26 smithi102 ceph-mon[29355]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:14:26.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:26 smithi102 ceph-mon[29355]: Updating smithi102:/etc/ceph/ceph.conf 2024-09-13T14:14:26.422 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:26 smithi102 ceph-mon[29355]: Updating smithi195:/etc/ceph/ceph.conf 2024-09-13T14:14:26.422 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:26 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/3308548429' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:14:26.422 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:26 smithi102 ceph-mon[29355]: Updating smithi102:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:14:26.422 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:26 smithi102 ceph-mon[29355]: Updating smithi195:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:14:26.422 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:26 smithi102 ceph-mon[29355]: from='client.15600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:14:26.422 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:26 smithi102 ceph-mon[29355]: Updating smithi102:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:14:26.422 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:26 smithi102 ceph-mon[29355]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:26.422 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:26 smithi102 ceph-mon[29355]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:26.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:26 smithi195 ceph-mon[33205]: from='client.25478 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:14:26.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:26 smithi195 ceph-mon[33205]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:26.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:26 smithi195 ceph-mon[33205]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:26.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:26 smithi195 ceph-mon[33205]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:26.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:26 smithi195 ceph-mon[33205]: from='mgr.25045 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:26.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:26 smithi195 ceph-mon[33205]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:26.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:26 smithi195 ceph-mon[33205]: from='mgr.25045 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:26.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:26 smithi195 ceph-mon[33205]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:26.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:26 smithi195 ceph-mon[33205]: from='mgr.25045 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:26.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:26 smithi195 ceph-mon[33205]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:26.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:26 smithi195 ceph-mon[33205]: from='mgr.25045 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:14:26.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:26 smithi195 ceph-mon[33205]: Adjusting osd_memory_target on smithi195 to 2480M 2024-09-13T14:14:26.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:26 smithi195 ceph-mon[33205]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:26.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:26 smithi195 ceph-mon[33205]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:26.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:26 smithi195 ceph-mon[33205]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:14:26.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:26 smithi195 ceph-mon[33205]: Updating smithi102:/etc/ceph/ceph.conf 2024-09-13T14:14:26.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:26 smithi195 ceph-mon[33205]: Updating smithi195:/etc/ceph/ceph.conf 2024-09-13T14:14:26.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:26 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/3308548429' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:14:26.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:26 smithi195 ceph-mon[33205]: Updating smithi102:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:14:26.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:26 smithi195 ceph-mon[33205]: Updating smithi195:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:14:26.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:26 smithi195 ceph-mon[33205]: from='client.15600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:14:26.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:26 smithi195 ceph-mon[33205]: Updating smithi102:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:14:26.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:26 smithi195 ceph-mon[33205]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:26.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:26 smithi195 ceph-mon[33205]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:27.247 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:27 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[65743]: ts=2024-09-13T14:14:27.076Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.102:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.102:8765: connect: connection refused" 2024-09-13T14:14:27.247 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:27 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[65743]: ts=2024-09-13T14:14:27.078Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.102:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.102:8765: connect: connection refused" 2024-09-13T14:14:27.248 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:27 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[65743]: ts=2024-09-13T14:14:27.078Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.102:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.102:8765: connect: connection refused" 2024-09-13T14:14:27.248 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:27 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[65743]: ts=2024-09-13T14:14:27.078Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.102:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.102:8765: connect: connection refused" 2024-09-13T14:14:27.248 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:27 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[65743]: ts=2024-09-13T14:14:27.078Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.102:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.102:8765: connect: connection refused" 2024-09-13T14:14:27.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:27 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[29604]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:14:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:14:27.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:27 smithi102 ceph-mon[29355]: Updating smithi195:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:14:27.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:27 smithi102 ceph-mon[29355]: Updating smithi102:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:14:27.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:27 smithi102 ceph-mon[29355]: Updating smithi195:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:14:27.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:27 smithi102 ceph-mon[29355]: pgmap v12: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-13T14:14:27.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:27 smithi102 ceph-mon[29355]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:27.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:27 smithi102 ceph-mon[29355]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:27.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:27 smithi102 ceph-mon[29355]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:27.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:27 smithi102 ceph-mon[29355]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-13T14:14:27.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:27 smithi102 ceph-mon[29355]: Reconfiguring daemon prometheus.a on smithi195 2024-09-13T14:14:27.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:27 smithi102 ceph-mon[40543]: Updating smithi195:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:14:27.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:27 smithi102 ceph-mon[40543]: Updating smithi102:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:14:27.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:27 smithi102 ceph-mon[40543]: Updating smithi195:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:14:27.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:27 smithi102 ceph-mon[40543]: pgmap v12: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-13T14:14:27.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:27 smithi102 ceph-mon[40543]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:27.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:27 smithi102 ceph-mon[40543]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:27.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:27 smithi102 ceph-mon[40543]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:27.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:27 smithi102 ceph-mon[40543]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-13T14:14:27.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:27 smithi102 ceph-mon[40543]: Reconfiguring daemon prometheus.a on smithi195 2024-09-13T14:14:27.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:27 smithi195 ceph-mon[33205]: Updating smithi195:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:14:27.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:27 smithi195 ceph-mon[33205]: Updating smithi102:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:14:27.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:27 smithi195 ceph-mon[33205]: Updating smithi195:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:14:27.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:27 smithi195 ceph-mon[33205]: pgmap v12: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-13T14:14:27.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:27 smithi195 ceph-mon[33205]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:27.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:27 smithi195 ceph-mon[33205]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:27.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:27 smithi195 ceph-mon[33205]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:27.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:27 smithi195 ceph-mon[33205]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-13T14:14:27.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:27 smithi195 ceph-mon[33205]: Reconfiguring daemon prometheus.a on smithi195 2024-09-13T14:14:29.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:29 smithi102 ceph-mon[29355]: pgmap v13: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:29.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:29 smithi102 ceph-mon[40543]: pgmap v13: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:29.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:29 smithi195 ceph-mon[33205]: pgmap v13: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:31.648 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:31 smithi195 ceph-mon[33205]: pgmap v14: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:31.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:31 smithi102 ceph-mon[29355]: pgmap v14: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:31.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:31 smithi102 ceph-mon[40543]: pgmap v14: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:33.497 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:33 smithi195 systemd[1]: Stopping Ceph prometheus.a for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T14:14:33.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:33 smithi195 ceph-mon[33205]: pgmap v15: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:33.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:33 smithi102 ceph-mon[29355]: pgmap v15: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:33.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:33 smithi102 ceph-mon[40543]: pgmap v15: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:33.912 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:33 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[65743]: ts=2024-09-13T14:14:33.587Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-13T14:14:33.912 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:33 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[65743]: ts=2024-09-13T14:14:33.587Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-13T14:14:33.912 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:33 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[65743]: ts=2024-09-13T14:14:33.587Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-13T14:14:33.912 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:33 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[65743]: ts=2024-09-13T14:14:33.587Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-13T14:14:33.912 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:33 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[65743]: ts=2024-09-13T14:14:33.587Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-13T14:14:33.913 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:33 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[65743]: ts=2024-09-13T14:14:33.587Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-13T14:14:33.913 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:33 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[65743]: ts=2024-09-13T14:14:33.587Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-13T14:14:33.913 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:33 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[65743]: ts=2024-09-13T14:14:33.587Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-13T14:14:33.913 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:33 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[65743]: ts=2024-09-13T14:14:33.588Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-13T14:14:33.913 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:33 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[65743]: ts=2024-09-13T14:14:33.588Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-13T14:14:33.913 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:33 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[65743]: ts=2024-09-13T14:14:33.588Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-13T14:14:33.913 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:33 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[65743]: ts=2024-09-13T14:14:33.588Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-13T14:14:33.913 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:33 smithi195 podman[78065]: 2024-09-13 14:14:33.59788795 +0000 UTC m=+0.127645437 container died f526a9818eaef5ea1580a25b02d632e12b559d580f08bc314f719a0d9c698330 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:14:34.247 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:34 smithi195 podman[78065]: 2024-09-13 14:14:34.078079945 +0000 UTC m=+0.607837431 container cleanup f526a9818eaef5ea1580a25b02d632e12b559d580f08bc314f719a0d9c698330 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:14:34.247 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:34 smithi195 bash[78065]: ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a 2024-09-13T14:14:34.705 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:34 smithi195 podman[78082]: 2024-09-13 14:14:34.363487605 +0000 UTC m=+0.767907844 container remove f526a9818eaef5ea1580a25b02d632e12b559d580f08bc314f719a0d9c698330 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:14:34.705 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:34 smithi195 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-09-13T14:14:34.705 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:34 smithi195 systemd[1]: Stopped Ceph prometheus.a for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:14:34.705 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:34 smithi195 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@prometheus.a.service: Consumed 1.780s CPU time. 2024-09-13T14:14:34.705 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:34 smithi195 systemd[1]: Starting Ceph prometheus.a for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T14:14:34.971 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:34 smithi195 podman[78181]: 2024-09-13 14:14:34.608063956 +0000 UTC m=+0.016720138 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-09-13T14:14:34.971 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:34 smithi195 podman[78181]: 2024-09-13 14:14:34.723466639 +0000 UTC m=+0.132122808 container create 82c978d3e907067efe6a97a191137300f8cb862bc67e30000c1fbe2ce33b9cbc (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:14:34.972 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:34 smithi195 podman[78181]: 2024-09-13 14:14:34.942382571 +0000 UTC m=+0.351038741 container init 82c978d3e907067efe6a97a191137300f8cb862bc67e30000c1fbe2ce33b9cbc (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:14:34.972 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:34 smithi195 podman[78181]: 2024-09-13 14:14:34.94738677 +0000 UTC m=+0.356042939 container start 82c978d3e907067efe6a97a191137300f8cb862bc67e30000c1fbe2ce33b9cbc (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:14:35.248 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:34 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[78196]: ts=2024-09-13T14:14:34.971Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-13T14:14:35.248 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:34 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[78196]: ts=2024-09-13T14:14:34.971Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-13T14:14:35.248 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:34 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[78196]: ts=2024-09-13T14:14:34.971Z caller=main.go:570 level=info host_details="(Linux 5.14.0-505.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024 x86_64 smithi195 (none))" 2024-09-13T14:14:35.248 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:34 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[78196]: ts=2024-09-13T14:14:34.971Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-13T14:14:35.248 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:34 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[78196]: ts=2024-09-13T14:14:34.971Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-13T14:14:35.248 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:34 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[78196]: ts=2024-09-13T14:14:34.972Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-13T14:14:35.248 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:34 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[78196]: ts=2024-09-13T14:14:34.973Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-13T14:14:35.248 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:34 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[78196]: ts=2024-09-13T14:14:34.975Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-13T14:14:35.249 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:34 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[78196]: ts=2024-09-13T14:14:34.975Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-13T14:14:35.249 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:34 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[78196]: ts=2024-09-13T14:14:34.976Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-13T14:14:35.249 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:34 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[78196]: ts=2024-09-13T14:14:34.976Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.237µs 2024-09-13T14:14:35.249 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:34 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[78196]: ts=2024-09-13T14:14:34.976Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-13T14:14:35.249 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:34 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[78196]: ts=2024-09-13T14:14:34.978Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2024-09-13T14:14:35.249 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:34 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[78196]: ts=2024-09-13T14:14:34.984Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2024-09-13T14:14:35.249 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:34 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[78196]: ts=2024-09-13T14:14:34.984Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2024-09-13T14:14:35.249 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:34 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[78196]: ts=2024-09-13T14:14:34.984Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=18.545µs wal_replay_duration=8.393002ms wbl_replay_duration=143ns total_replay_duration=8.43092ms 2024-09-13T14:14:35.249 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:34 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[78196]: ts=2024-09-13T14:14:34.986Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-13T14:14:35.249 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:34 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[78196]: ts=2024-09-13T14:14:34.986Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-13T14:14:35.249 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:34 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[78196]: ts=2024-09-13T14:14:34.986Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-13T14:14:35.249 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:35 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[78196]: ts=2024-09-13T14:14:35.007Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=21.488737ms db_storage=793ns remote_storage=1.169µs web_handler=456ns query_engine=589ns scrape=395.605µs scrape_sd=98.018µs notify=14.323µs notify_sd=40.54µs rules=20.599021ms tracing=5.643µs 2024-09-13T14:14:35.250 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:35 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[78196]: ts=2024-09-13T14:14:35.007Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-13T14:14:35.250 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:35 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[78196]: ts=2024-09-13T14:14:35.007Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-13T14:14:35.250 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:35 smithi195 bash[78181]: 82c978d3e907067efe6a97a191137300f8cb862bc67e30000c1fbe2ce33b9cbc 2024-09-13T14:14:35.250 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:14:35 smithi195 systemd[1]: Started Ceph prometheus.a for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:14:35.631 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:35 smithi102 ceph-mon[29355]: pgmap v16: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:35.631 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:35 smithi102 ceph-mon[29355]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:35.632 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:35 smithi102 ceph-mon[29355]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:35.632 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:35 smithi102 ceph-mon[29355]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:14:35.632 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:35 smithi102 ceph-mon[29355]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:14:35.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:35 smithi102 ceph-mon[40543]: pgmap v16: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:35.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:35 smithi102 ceph-mon[40543]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:35.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:35 smithi102 ceph-mon[40543]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:35.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:35 smithi102 ceph-mon[40543]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:14:35.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:35 smithi102 ceph-mon[40543]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:14:35.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:35 smithi195 ceph-mon[33205]: pgmap v16: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:35.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:35 smithi195 ceph-mon[33205]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:35.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:35 smithi195 ceph-mon[33205]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:35.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:35 smithi195 ceph-mon[33205]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:14:35.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:35 smithi195 ceph-mon[33205]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:14:36.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:36 smithi102 ceph-mon[29355]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:14:36.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:36 smithi102 ceph-mon[29355]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-13T14:14:36.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:36 smithi102 ceph-mon[29355]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-13T14:14:36.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:36 smithi102 ceph-mon[40543]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:14:36.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:36 smithi102 ceph-mon[40543]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-13T14:14:36.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:36 smithi102 ceph-mon[40543]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-13T14:14:36.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:36 smithi195 ceph-mon[33205]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:14:36.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:36 smithi195 ceph-mon[33205]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-13T14:14:36.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:36 smithi195 ceph-mon[33205]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-13T14:14:38.102 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:37 smithi102 ceph-mon[29355]: pgmap v17: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:38.102 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:37 smithi102 ceph-mon[29355]: Upgrade: Updating mgr.y 2024-09-13T14:14:38.103 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:37 smithi102 ceph-mon[29355]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:38.103 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:37 smithi102 ceph-mon[29355]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:14:38.103 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:37 smithi102 ceph-mon[29355]: from='mgr.25045 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:14:38.103 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:37 smithi102 ceph-mon[29355]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:14:38.103 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:37 smithi102 ceph-mon[29355]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:38.103 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:37 smithi102 ceph-mon[29355]: Deploying daemon mgr.y on smithi102 2024-09-13T14:14:38.103 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:37 smithi102 ceph-mon[40543]: pgmap v17: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:38.103 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:37 smithi102 ceph-mon[40543]: Upgrade: Updating mgr.y 2024-09-13T14:14:38.103 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:37 smithi102 ceph-mon[40543]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:38.103 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:37 smithi102 ceph-mon[40543]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:14:38.104 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:37 smithi102 ceph-mon[40543]: from='mgr.25045 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:14:38.104 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:37 smithi102 ceph-mon[40543]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:14:38.104 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:37 smithi102 ceph-mon[40543]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:38.104 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:37 smithi102 ceph-mon[40543]: Deploying daemon mgr.y on smithi102 2024-09-13T14:14:38.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:37 smithi195 ceph-mon[33205]: pgmap v17: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:38.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:37 smithi195 ceph-mon[33205]: Upgrade: Updating mgr.y 2024-09-13T14:14:38.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:37 smithi195 ceph-mon[33205]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:38.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:37 smithi195 ceph-mon[33205]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:14:38.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:37 smithi195 ceph-mon[33205]: from='mgr.25045 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:14:38.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:37 smithi195 ceph-mon[33205]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:14:38.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:37 smithi195 ceph-mon[33205]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:38.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:37 smithi195 ceph-mon[33205]: Deploying daemon mgr.y on smithi102 2024-09-13T14:14:38.386 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:38 smithi102 systemd[1]: Stopping Ceph mgr.y for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T14:14:38.941 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:38 smithi102 podman[91054]: 2024-09-13 14:14:38.681153683 +0000 UTC m=+0.150532359 container died 4b4d7bb68ef4dce6228f7ba6f16f5a2394e0c611e5b7786871b19463ab19b689 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y, org.label-schema.license=GPLv2, RELEASE=reef-2c01458, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T14:14:39.342 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:39 smithi102 podman[91054]: 2024-09-13 14:14:39.115015174 +0000 UTC m=+0.584393847 container cleanup 4b4d7bb68ef4dce6228f7ba6f16f5a2394e0c611e5b7786871b19463ab19b689 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y, RELEASE=reef-2c01458, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-09-13T14:14:39.342 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:39 smithi102 bash[91054]: ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y 2024-09-13T14:14:39.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:39 smithi102 podman[91071]: 2024-09-13 14:14:39.341957017 +0000 UTC m=+0.661311282 container remove 4b4d7bb68ef4dce6228f7ba6f16f5a2394e0c611e5b7786871b19463ab19b689 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=reef-2c01458) 2024-09-13T14:14:39.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:39 smithi102 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-09-13T14:14:39.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:39 smithi102 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@mgr.y.service: Failed with result 'exit-code'. 2024-09-13T14:14:39.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:39 smithi102 systemd[1]: Stopped Ceph mgr.y for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:14:39.669 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:39 smithi102 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@mgr.y.service: Consumed 41.380s CPU time. 2024-09-13T14:14:40.099 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:39 smithi102 ceph-mon[40543]: pgmap v18: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:40.099 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:39 smithi102 ceph-mon[40543]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:14:40.099 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:39 smithi102 ceph-mon[29355]: pgmap v18: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:40.100 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:39 smithi102 ceph-mon[29355]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:14:40.100 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:39 smithi102 systemd[1]: Starting Ceph mgr.y for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T14:14:40.100 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:39 smithi102 podman[91204]: 2024-09-13 14:14:39.984766242 +0000 UTC m=+0.111971121 container create 6b12b28dcde1bdb8dfb811817c1ccf703d65c60eccbbeca5b14d62d4020c8ae6 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-13T14:14:40.100 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:39 smithi102 podman[91204]: 2024-09-13 14:14:39.888472477 +0000 UTC m=+0.015677356 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:14:40.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:39 smithi195 ceph-mon[33205]: pgmap v18: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:40.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:39 smithi195 ceph-mon[33205]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:14:40.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:40 smithi102 podman[91204]: 2024-09-13 14:14:40.19572008 +0000 UTC m=+0.322924967 container init 6b12b28dcde1bdb8dfb811817c1ccf703d65c60eccbbeca5b14d62d4020c8ae6 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-aclamk-cbt-combined-d58f730, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2) 2024-09-13T14:14:40.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:40 smithi102 podman[91204]: 2024-09-13 14:14:40.200454217 +0000 UTC m=+0.327659096 container start 6b12b28dcde1bdb8dfb811817c1ccf703d65c60eccbbeca5b14d62d4020c8ae6 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731) 2024-09-13T14:14:40.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:40 smithi102 ceph-mgr[91242]: -- 172.21.15.102:0/1811228703 <== mon.2 v2:172.21.15.102:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56279bf190e0 con 0x56279bef7400 2024-09-13T14:14:40.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:40 smithi102 bash[91204]: 6b12b28dcde1bdb8dfb811817c1ccf703d65c60eccbbeca5b14d62d4020c8ae6 2024-09-13T14:14:40.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:40 smithi102 systemd[1]: Started Ceph mgr.y for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:14:41.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:41 smithi102 ceph-mon[29355]: pgmap v19: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:41.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:41 smithi102 ceph-mon[29355]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:41.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:41 smithi102 ceph-mon[29355]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:41.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:41 smithi102 ceph-mon[29355]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:14:41.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:41 smithi102 ceph-mon[40543]: pgmap v19: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:41.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:41 smithi102 ceph-mon[40543]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:41.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:41 smithi102 ceph-mon[40543]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:41.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:41 smithi102 ceph-mon[40543]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:14:41.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:41 smithi195 ceph-mon[33205]: pgmap v19: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:41.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:41 smithi195 ceph-mon[33205]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:41.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:41 smithi195 ceph-mon[33205]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:41.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:41 smithi195 ceph-mon[33205]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:14:43.630 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:43 smithi102 ceph-mon[29355]: pgmap v20: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:43.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:43 smithi102 ceph-mon[40543]: pgmap v20: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:43.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:43 smithi195 ceph-mon[33205]: pgmap v20: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:45.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:44 smithi195 ceph-mon[33205]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:45.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:44 smithi195 ceph-mon[33205]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:45.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:44 smithi195 ceph-mon[33205]: pgmap v21: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:45.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:44 smithi195 ceph-mon[33205]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:45.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:44 smithi195 ceph-mon[33205]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:45.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:44 smithi195 ceph-mon[33205]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:45.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:44 smithi195 ceph-mon[33205]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:45.262 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:44 smithi102 ceph-mon[29355]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:45.262 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:44 smithi102 ceph-mon[29355]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:45.262 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:44 smithi102 ceph-mon[29355]: pgmap v21: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:45.262 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:44 smithi102 ceph-mon[29355]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:45.262 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:44 smithi102 ceph-mon[29355]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:45.262 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:44 smithi102 ceph-mon[29355]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:45.262 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:44 smithi102 ceph-mon[29355]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:45.263 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:44 smithi102 ceph-mon[40543]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:45.263 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:44 smithi102 ceph-mon[40543]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:45.263 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:44 smithi102 ceph-mon[40543]: pgmap v21: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:45.263 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:44 smithi102 ceph-mon[40543]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:45.263 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:44 smithi102 ceph-mon[40543]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:45.263 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:44 smithi102 ceph-mon[40543]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:45.263 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:44 smithi102 ceph-mon[40543]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:45.512 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:45 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: [13/Sep/2024:14:14:45] ENGINE Bus STARTING 2024-09-13T14:14:45.512 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:45 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: CherryPy Checker: 2024-09-13T14:14:45.513 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:45 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: The Application mounted at '' has an empty config. 2024-09-13T14:14:45.513 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:45 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: 2024-09-13T14:14:45.879 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:45 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: [13/Sep/2024:14:14:45] ENGINE Serving on http://:::9283 2024-09-13T14:14:45.879 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:45 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: [13/Sep/2024:14:14:45] ENGINE Bus STARTED 2024-09-13T14:14:46.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:46 smithi102 ceph-mon[29355]: Standby manager daemon y restarted 2024-09-13T14:14:46.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:46 smithi102 ceph-mon[29355]: Standby manager daemon y started 2024-09-13T14:14:46.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:46 smithi102 ceph-mon[29355]: from='mgr.? 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-13T14:14:46.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:46 smithi102 ceph-mon[29355]: from='mgr.? 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:14:46.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:46 smithi102 ceph-mon[29355]: from='mgr.? 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-13T14:14:46.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:46 smithi102 ceph-mon[29355]: from='mgr.? 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:14:46.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:46 smithi102 ceph-mon[40543]: Standby manager daemon y restarted 2024-09-13T14:14:46.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:46 smithi102 ceph-mon[40543]: Standby manager daemon y started 2024-09-13T14:14:46.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:46 smithi102 ceph-mon[40543]: from='mgr.? 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-13T14:14:46.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:46 smithi102 ceph-mon[40543]: from='mgr.? 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:14:46.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:46 smithi102 ceph-mon[40543]: from='mgr.? 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-13T14:14:46.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:46 smithi102 ceph-mon[40543]: from='mgr.? 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:14:46.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:46 smithi195 ceph-mon[33205]: Standby manager daemon y restarted 2024-09-13T14:14:46.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:46 smithi195 ceph-mon[33205]: Standby manager daemon y started 2024-09-13T14:14:46.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:46 smithi195 ceph-mon[33205]: from='mgr.? 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-13T14:14:46.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:46 smithi195 ceph-mon[33205]: from='mgr.? 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:14:46.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:46 smithi195 ceph-mon[33205]: from='mgr.? 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-13T14:14:46.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:46 smithi195 ceph-mon[33205]: from='mgr.? 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:14:47.286 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:47 smithi102 ceph-mon[29355]: mgrmap e27: x(active, since 37s), standbys: y 2024-09-13T14:14:47.287 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:47 smithi102 ceph-mon[29355]: pgmap v22: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:47.287 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:47 smithi102 ceph-mon[40543]: mgrmap e27: x(active, since 37s), standbys: y 2024-09-13T14:14:47.287 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:47 smithi102 ceph-mon[40543]: pgmap v22: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:47.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:47 smithi195 ceph-mon[33205]: mgrmap e27: x(active, since 37s), standbys: y 2024-09-13T14:14:47.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:47 smithi195 ceph-mon[33205]: pgmap v22: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:48.168 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: [13/Sep/2024:14:14:48] ENGINE Bus STOPPING 2024-09-13T14:14:48.168 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: [13/Sep/2024:14:14:48] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-13T14:14:48.168 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: [13/Sep/2024:14:14:48] ENGINE Bus STOPPED 2024-09-13T14:14:48.398 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:48 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: 2024-09-13T14:14:48.066+0000 7f22bd1b5640 -1 mgr handle_mgr_map I was active but no longer am 2024-09-13T14:14:48.398 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:48 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: ignoring --setuser ceph since I am not root 2024-09-13T14:14:48.399 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:48 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: ignoring --setgroup ceph since I am not root 2024-09-13T14:14:48.399 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:48 smithi195 ceph-mgr[69336]: -- 172.21.15.195:0/1490349056 <== mon.1 v2:172.21.15.195:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b51484b0e0 con 0x55b514829400 2024-09-13T14:14:48.918 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: 2024-09-13T14:14:48.515+0000 7fe7c36b7640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:14:48.918 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: 2024-09-13T14:14:48.515+0000 7fe7c36b7640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:14:48.918 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: 2024-09-13T14:14:48.515+0000 7fe7c36b7640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:14:48.919 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: 2024-09-13T14:14:48.515+0000 7fe7c36b7640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:14:48.919 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: 2024-09-13T14:14:48.515+0000 7fe7c36b7640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:14:48.919 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: 2024-09-13T14:14:48.515+0000 7fe7c06b1640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:14:48.919 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: 2024-09-13T14:14:48.515+0000 7fe7c06b1640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:14:48.919 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: 2024-09-13T14:14:48.515+0000 7fe7c06b1640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:14:48.919 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: 2024-09-13T14:14:48.515+0000 7fe7c06b1640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:14:48.919 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: 2024-09-13T14:14:48.515+0000 7fe7c06b1640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:14:49.186 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:49 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: [13/Sep/2024:14:14:49] ENGINE Bus STARTING 2024-09-13T14:14:49.186 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:49 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: [13/Sep/2024:14:14:49] ENGINE Serving on http://:::9283 2024-09-13T14:14:49.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[40543]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:49.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[40543]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:49.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[40543]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:49.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[40543]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:14:49.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[40543]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:49.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[40543]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:14:49.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[40543]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-13T14:14:49.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[40543]: from='mgr.25045 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-13T14:14:49.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[40543]: Activating manager daemon y 2024-09-13T14:14:49.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[40543]: osdmap e89: 8 total, 8 up, 8 in 2024-09-13T14:14:49.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[40543]: from='mgr.25045 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-13T14:14:49.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[40543]: mgrmap e28: y(active, starting, since 0.00938932s) 2024-09-13T14:14:49.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:14:49.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:14:49.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:14:49.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi195.yqzrxi"}]: dispatch 2024-09-13T14:14:49.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi102.njetxr"}]: dispatch 2024-09-13T14:14:49.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:14:49.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:14:49.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:14:49.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:14:49.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:14:49.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:14:49.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:14:49.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:14:49.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:14:49.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:14:49.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:14:49.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:14:49.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[40543]: Manager daemon y is now available 2024-09-13T14:14:49.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[29355]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:49.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[29355]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:49.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[29355]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:49.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[29355]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:14:49.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[29355]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:49.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[29355]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:14:49.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[29355]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-13T14:14:49.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[29355]: from='mgr.25045 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-13T14:14:49.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[29355]: Activating manager daemon y 2024-09-13T14:14:49.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[29355]: osdmap e89: 8 total, 8 up, 8 in 2024-09-13T14:14:49.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[29355]: from='mgr.25045 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-13T14:14:49.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[29355]: mgrmap e28: y(active, starting, since 0.00938932s) 2024-09-13T14:14:49.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:14:49.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:14:49.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:14:49.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi195.yqzrxi"}]: dispatch 2024-09-13T14:14:49.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi102.njetxr"}]: dispatch 2024-09-13T14:14:49.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:14:49.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:14:49.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:14:49.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:14:49.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:14:49.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:14:49.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:14:49.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:14:49.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:14:49.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:14:49.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:14:49.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:14:49.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[29355]: Manager daemon y is now available 2024-09-13T14:14:49.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:14:49.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:14:49.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:14:49.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:14:49.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:14:49.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:14:49.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:14:49.192 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:14:49.192 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:14:49.192 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:14:49.192 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:14:49.192 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:14:49.192 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:14:49.192 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:48 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:14:49.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:48 smithi195 ceph-mon[33205]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:49.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:48 smithi195 ceph-mon[33205]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:49.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:48 smithi195 ceph-mon[33205]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:49.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:48 smithi195 ceph-mon[33205]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:14:49.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:48 smithi195 ceph-mon[33205]: from='mgr.25045 ' entity='mgr.x' 2024-09-13T14:14:49.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:48 smithi195 ceph-mon[33205]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:14:49.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:48 smithi195 ceph-mon[33205]: from='mgr.25045 172.21.15.195:0/1434100798' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-13T14:14:49.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:48 smithi195 ceph-mon[33205]: from='mgr.25045 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-13T14:14:49.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:48 smithi195 ceph-mon[33205]: Activating manager daemon y 2024-09-13T14:14:49.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:48 smithi195 ceph-mon[33205]: osdmap e89: 8 total, 8 up, 8 in 2024-09-13T14:14:49.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:48 smithi195 ceph-mon[33205]: from='mgr.25045 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-13T14:14:49.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:48 smithi195 ceph-mon[33205]: mgrmap e28: y(active, starting, since 0.00938932s) 2024-09-13T14:14:49.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:48 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:14:49.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:48 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:14:49.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:48 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:14:49.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:48 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi195.yqzrxi"}]: dispatch 2024-09-13T14:14:49.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:48 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi102.njetxr"}]: dispatch 2024-09-13T14:14:49.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:48 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:14:49.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:48 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:14:49.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:48 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:14:49.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:48 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:14:49.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:48 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:14:49.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:48 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:14:49.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:48 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:14:49.250 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:48 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:14:49.250 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:48 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:14:49.250 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:48 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:14:49.250 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:48 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:14:49.250 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:48 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:14:49.250 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:48 smithi195 ceph-mon[33205]: Manager daemon y is now available 2024-09-13T14:14:49.250 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:48 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:14:49.250 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:48 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:14:49.250 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:48 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:14:49.250 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:48 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:14:49.250 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:48 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:14:49.250 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:48 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:14:49.250 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:48 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:14:49.474 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:49 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: [13/Sep/2024:14:14:49] ENGINE Bus STARTED 2024-09-13T14:14:49.785 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:49 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: 2024-09-13T14:14:49.783+0000 7fe7a13f8640 -1 log_channel(cephadm) log [ERR] : [13/Sep/2024:14:14:49] ENGINE Error in HTTPServer.serve 2024-09-13T14:14:49.785 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:49 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: Traceback (most recent call last): 2024-09-13T14:14:49.785 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:49 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:14:49.785 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:49 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: self._connections.run(self.expiration_interval) 2024-09-13T14:14:49.785 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:49 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:14:50.072 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:49 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: self._run(expiration_interval) 2024-09-13T14:14:50.072 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:49 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:14:50.072 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:49 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:14:50.072 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:49 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:14:50.072 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:49 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:14:50.072 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:49 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:14:50.072 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:49 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: s = self.context.wrap_socket( 2024-09-13T14:14:50.073 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:49 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:14:50.073 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:49 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: return self.sslsocket_class._create( 2024-09-13T14:14:50.073 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:49 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:14:50.073 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:49 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: self.do_handshake() 2024-09-13T14:14:50.073 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:49 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:14:50.073 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:49 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: self._sslobj.do_handshake() 2024-09-13T14:14:50.073 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:49 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:14:50.073 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:14:49 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: 2024-09-13T14:14:50.358 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:50 smithi195 ceph-mon[33205]: mgrmap e29: y(active, since 1.01405s) 2024-09-13T14:14:50.358 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:50 smithi195 ceph-mon[33205]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:14:50.358 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:50 smithi195 ceph-mon[33205]: pgmap v3: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:50.371 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:50 smithi102 ceph-mon[29355]: mgrmap e29: y(active, since 1.01405s) 2024-09-13T14:14:50.371 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:50 smithi102 ceph-mon[29355]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:14:50.371 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:50 smithi102 ceph-mon[29355]: pgmap v3: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:50.372 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:50 smithi102 ceph-mon[40543]: mgrmap e29: y(active, since 1.01405s) 2024-09-13T14:14:50.372 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:50 smithi102 ceph-mon[40543]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:14:50.372 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:50 smithi102 ceph-mon[40543]: pgmap v3: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:51.349 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:51 smithi102 ceph-mon[40543]: [13/Sep/2024:14:14:49] ENGINE Bus STARTING 2024-09-13T14:14:51.349 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:51 smithi102 ceph-mon[40543]: [13/Sep/2024:14:14:49] ENGINE Serving on http://172.21.15.102:8765 2024-09-13T14:14:51.349 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:51 smithi102 ceph-mon[40543]: [13/Sep/2024:14:14:49] ENGINE Serving on https://172.21.15.102:7150 2024-09-13T14:14:51.349 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:51 smithi102 ceph-mon[40543]: [13/Sep/2024:14:14:49] ENGINE Bus STARTED 2024-09-13T14:14:51.350 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:51 smithi102 ceph-mon[40543]: [13/Sep/2024:14:14:49] ENGINE Error in HTTPServer.serve 2024-09-13T14:14:51.350 INFO:journalctl@ceph.mon.c.smithi102.stdout: Traceback (most recent call last): 2024-09-13T14:14:51.350 INFO:journalctl@ceph.mon.c.smithi102.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:14:51.350 INFO:journalctl@ceph.mon.c.smithi102.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:14:51.350 INFO:journalctl@ceph.mon.c.smithi102.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:14:51.350 INFO:journalctl@ceph.mon.c.smithi102.stdout: self._run(expiration_interval) 2024-09-13T14:14:51.350 INFO:journalctl@ceph.mon.c.smithi102.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:14:51.350 INFO:journalctl@ceph.mon.c.smithi102.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:14:51.350 INFO:journalctl@ceph.mon.c.smithi102.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:14:51.350 INFO:journalctl@ceph.mon.c.smithi102.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:14:51.350 INFO:journalctl@ceph.mon.c.smithi102.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:14:51.350 INFO:journalctl@ceph.mon.c.smithi102.stdout: s = self.context.wrap_socket( 2024-09-13T14:14:51.350 INFO:journalctl@ceph.mon.c.smithi102.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:14:51.350 INFO:journalctl@ceph.mon.c.smithi102.stdout: return self.sslsocket_class._create( 2024-09-13T14:14:51.351 INFO:journalctl@ceph.mon.c.smithi102.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:14:51.351 INFO:journalctl@ceph.mon.c.smithi102.stdout: self.do_handshake() 2024-09-13T14:14:51.351 INFO:journalctl@ceph.mon.c.smithi102.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:14:51.351 INFO:journalctl@ceph.mon.c.smithi102.stdout: self._sslobj.do_handshake() 2024-09-13T14:14:51.351 INFO:journalctl@ceph.mon.c.smithi102.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:14:51.351 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:51 smithi102 ceph-mon[40543]: mgrmap e30: y(active, since 2s) 2024-09-13T14:14:51.351 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:51 smithi102 ceph-mon[40543]: pgmap v4: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:51.351 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:51 smithi102 ceph-mon[29355]: [13/Sep/2024:14:14:49] ENGINE Bus STARTING 2024-09-13T14:14:51.351 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:51 smithi102 ceph-mon[29355]: [13/Sep/2024:14:14:49] ENGINE Serving on http://172.21.15.102:8765 2024-09-13T14:14:51.351 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:51 smithi102 ceph-mon[29355]: [13/Sep/2024:14:14:49] ENGINE Serving on https://172.21.15.102:7150 2024-09-13T14:14:51.351 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:51 smithi102 ceph-mon[29355]: [13/Sep/2024:14:14:49] ENGINE Bus STARTED 2024-09-13T14:14:51.351 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:51 smithi102 ceph-mon[29355]: [13/Sep/2024:14:14:49] ENGINE Error in HTTPServer.serve 2024-09-13T14:14:51.352 INFO:journalctl@ceph.mon.a.smithi102.stdout: Traceback (most recent call last): 2024-09-13T14:14:51.352 INFO:journalctl@ceph.mon.a.smithi102.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:14:51.352 INFO:journalctl@ceph.mon.a.smithi102.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:14:51.352 INFO:journalctl@ceph.mon.a.smithi102.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:14:51.352 INFO:journalctl@ceph.mon.a.smithi102.stdout: self._run(expiration_interval) 2024-09-13T14:14:51.352 INFO:journalctl@ceph.mon.a.smithi102.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:14:51.352 INFO:journalctl@ceph.mon.a.smithi102.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:14:51.352 INFO:journalctl@ceph.mon.a.smithi102.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:14:51.352 INFO:journalctl@ceph.mon.a.smithi102.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:14:51.352 INFO:journalctl@ceph.mon.a.smithi102.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:14:51.352 INFO:journalctl@ceph.mon.a.smithi102.stdout: s = self.context.wrap_socket( 2024-09-13T14:14:51.352 INFO:journalctl@ceph.mon.a.smithi102.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:14:51.352 INFO:journalctl@ceph.mon.a.smithi102.stdout: return self.sslsocket_class._create( 2024-09-13T14:14:51.352 INFO:journalctl@ceph.mon.a.smithi102.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:14:51.353 INFO:journalctl@ceph.mon.a.smithi102.stdout: self.do_handshake() 2024-09-13T14:14:51.353 INFO:journalctl@ceph.mon.a.smithi102.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:14:51.353 INFO:journalctl@ceph.mon.a.smithi102.stdout: self._sslobj.do_handshake() 2024-09-13T14:14:51.353 INFO:journalctl@ceph.mon.a.smithi102.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:14:51.353 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:51 smithi102 ceph-mon[29355]: mgrmap e30: y(active, since 2s) 2024-09-13T14:14:51.353 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:51 smithi102 ceph-mon[29355]: pgmap v4: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:51.367 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:51 smithi195 ceph-mon[33205]: [13/Sep/2024:14:14:49] ENGINE Bus STARTING 2024-09-13T14:14:51.368 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:51 smithi195 ceph-mon[33205]: [13/Sep/2024:14:14:49] ENGINE Serving on http://172.21.15.102:8765 2024-09-13T14:14:51.368 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:51 smithi195 ceph-mon[33205]: [13/Sep/2024:14:14:49] ENGINE Serving on https://172.21.15.102:7150 2024-09-13T14:14:51.368 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:51 smithi195 ceph-mon[33205]: [13/Sep/2024:14:14:49] ENGINE Bus STARTED 2024-09-13T14:14:51.368 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:51 smithi195 ceph-mon[33205]: [13/Sep/2024:14:14:49] ENGINE Error in HTTPServer.serve 2024-09-13T14:14:51.368 INFO:journalctl@ceph.mon.b.smithi195.stdout: Traceback (most recent call last): 2024-09-13T14:14:51.368 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:14:51.368 INFO:journalctl@ceph.mon.b.smithi195.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:14:51.368 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:14:51.368 INFO:journalctl@ceph.mon.b.smithi195.stdout: self._run(expiration_interval) 2024-09-13T14:14:51.368 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:14:51.368 INFO:journalctl@ceph.mon.b.smithi195.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:14:51.368 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:14:51.369 INFO:journalctl@ceph.mon.b.smithi195.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:14:51.369 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:14:51.369 INFO:journalctl@ceph.mon.b.smithi195.stdout: s = self.context.wrap_socket( 2024-09-13T14:14:51.369 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:14:51.369 INFO:journalctl@ceph.mon.b.smithi195.stdout: return self.sslsocket_class._create( 2024-09-13T14:14:51.369 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:14:51.369 INFO:journalctl@ceph.mon.b.smithi195.stdout: self.do_handshake() 2024-09-13T14:14:51.369 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:14:51.369 INFO:journalctl@ceph.mon.b.smithi195.stdout: self._sslobj.do_handshake() 2024-09-13T14:14:51.369 INFO:journalctl@ceph.mon.b.smithi195.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:14:51.369 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:51 smithi195 ceph-mon[33205]: mgrmap e30: y(active, since 2s) 2024-09-13T14:14:51.369 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:51 smithi195 ceph-mon[33205]: pgmap v4: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:53.376 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:53 smithi195 ceph-mon[33205]: pgmap v5: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:53.376 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:53 smithi195 ceph-mon[33205]: mgrmap e31: y(active, since 4s) 2024-09-13T14:14:53.376 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:53 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:14:53.376 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:53 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:14:53.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:53 smithi102 ceph-mon[29355]: pgmap v5: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:53.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:53 smithi102 ceph-mon[29355]: mgrmap e31: y(active, since 4s) 2024-09-13T14:14:53.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:53 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:14:53.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:53 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:14:53.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:53 smithi102 ceph-mon[40543]: pgmap v5: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:53.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:53 smithi102 ceph-mon[40543]: mgrmap e31: y(active, since 4s) 2024-09-13T14:14:53.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:53 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:14:53.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:53 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:14:53.747 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:53 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: [13/Sep/2024:14:14:53] ENGINE Bus STARTING 2024-09-13T14:14:53.747 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:53 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: CherryPy Checker: 2024-09-13T14:14:53.747 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:53 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: The Application mounted at '' has an empty config. 2024-09-13T14:14:53.747 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:53 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: 2024-09-13T14:14:53.747 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:53 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: [13/Sep/2024:14:14:53] ENGINE Serving on http://:::9283 2024-09-13T14:14:53.747 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:53 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: [13/Sep/2024:14:14:53] ENGINE Bus STARTED 2024-09-13T14:14:54.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:54 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:14:54.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:54 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:14:54.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:54 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:14:54.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:54 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:14:54.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:54 smithi102 ceph-mon[29355]: Standby manager daemon x started 2024-09-13T14:14:54.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:54 smithi102 ceph-mon[29355]: from='mgr.? 172.21.15.195:0/3954581952' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:14:54.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:54 smithi102 ceph-mon[29355]: from='mgr.? 172.21.15.195:0/3954581952' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:14:54.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:54 smithi102 ceph-mon[29355]: from='mgr.? 172.21.15.195:0/3954581952' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:14:54.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:54 smithi102 ceph-mon[29355]: from='mgr.? 172.21.15.195:0/3954581952' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:14:54.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:54 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:14:54.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:54 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:14:54.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:54 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:14:54.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:54 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:14:54.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:54 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:14:54.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:54 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:14:54.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:54 smithi102 ceph-mon[40543]: Standby manager daemon x started 2024-09-13T14:14:54.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:54 smithi102 ceph-mon[40543]: from='mgr.? 172.21.15.195:0/3954581952' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:14:54.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:54 smithi102 ceph-mon[40543]: from='mgr.? 172.21.15.195:0/3954581952' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:14:54.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:54 smithi102 ceph-mon[40543]: from='mgr.? 172.21.15.195:0/3954581952' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:14:54.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:54 smithi102 ceph-mon[40543]: from='mgr.? 172.21.15.195:0/3954581952' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:14:54.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:54 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:14:54.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:54 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:14:54.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:54 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:14:54.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:54 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:14:54.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:54 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:14:54.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:54 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:14:54.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:54 smithi195 ceph-mon[33205]: Standby manager daemon x started 2024-09-13T14:14:54.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:54 smithi195 ceph-mon[33205]: from='mgr.? 172.21.15.195:0/3954581952' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:14:54.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:54 smithi195 ceph-mon[33205]: from='mgr.? 172.21.15.195:0/3954581952' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:14:54.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:54 smithi195 ceph-mon[33205]: from='mgr.? 172.21.15.195:0/3954581952' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:14:54.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:54 smithi195 ceph-mon[33205]: from='mgr.? 172.21.15.195:0/3954581952' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:14:54.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:54 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:14:54.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:54 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:14:55.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:55 smithi102 ceph-mon[29355]: pgmap v6: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:55.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:55 smithi102 ceph-mon[29355]: mgrmap e32: y(active, since 6s), standbys: x 2024-09-13T14:14:55.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:55 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:14:55.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:55 smithi102 ceph-mon[40543]: pgmap v6: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:55.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:55 smithi102 ceph-mon[40543]: mgrmap e32: y(active, since 6s), standbys: x 2024-09-13T14:14:55.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:55 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:14:55.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:55 smithi195 ceph-mon[33205]: pgmap v6: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:14:55.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:55 smithi195 ceph-mon[33205]: mgrmap e32: y(active, since 6s), standbys: x 2024-09-13T14:14:55.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:55 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:14:55.869 INFO:teuthology.orchestra.run.smithi102.stdout:true 2024-09-13T14:14:56.443 INFO:teuthology.orchestra.run.smithi102.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-13T14:14:56.443 INFO:teuthology.orchestra.run.smithi102.stdout:alertmanager.a smithi102 *:9093,9094 running (3m) 3s ago 4m 27.8M - 0.25.0 c8568f914cd2 fadcd26e5395 2024-09-13T14:14:56.443 INFO:teuthology.orchestra.run.smithi102.stdout:grafana.a smithi195 *:3000 running (4m) 3s ago 4m 87.9M - 9.4.7 954c08fa6188 4e2eb92e9f89 2024-09-13T14:14:56.444 INFO:teuthology.orchestra.run.smithi102.stdout:mds.foo.smithi102.njetxr smithi102 running (3m) 3s ago 3m 18.9M - 18.2.4-843-g2c01458a c129f5a2fd4e c9211cc725ed 2024-09-13T14:14:56.444 INFO:teuthology.orchestra.run.smithi102.stdout:mds.foo.smithi195.yqzrxi smithi195 running (3m) 3s ago 3m 16.7M - 18.2.4-843-g2c01458a c129f5a2fd4e eea37a7230fd 2024-09-13T14:14:56.444 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.x smithi195 *:8443,9283,8765 running (57s) 3s ago 8m 110M - 19.3.0-4697-gd58f7306 fec07016a2b0 2d56352abb14 2024-09-13T14:14:56.444 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.y smithi102 *:8443,9283,8765 running (16s) 3s ago 10m 486M - 19.3.0-4697-gd58f7306 fec07016a2b0 6b12b28dcde1 2024-09-13T14:14:56.444 INFO:teuthology.orchestra.run.smithi102.stdout:mon.a smithi102 running (10m) 3s ago 10m 53.9M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e 22c604c4fd02 2024-09-13T14:14:56.444 INFO:teuthology.orchestra.run.smithi102.stdout:mon.b smithi195 running (8m) 3s ago 8m 46.0M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e 608ec4847676 2024-09-13T14:14:56.444 INFO:teuthology.orchestra.run.smithi102.stdout:mon.c smithi102 running (8m) 3s ago 8m 43.1M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e 6c1fd3099a05 2024-09-13T14:14:56.444 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.a smithi102 *:9100 running (4m) 3s ago 4m 17.2M - 1.5.0 0da6a335fe13 5538bdf011a3 2024-09-13T14:14:56.444 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.b smithi195 *:9100 running (4m) 3s ago 4m 15.1M - 1.5.0 0da6a335fe13 138397d5e129 2024-09-13T14:14:56.444 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0 smithi102 running (8m) 3s ago 8m 59.1M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e b97482fb8cf9 2024-09-13T14:14:56.444 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1 smithi102 running (7m) 3s ago 7m 50.6M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 8a67356be7a2 2024-09-13T14:14:56.444 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2 smithi102 running (7m) 3s ago 7m 48.7M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 1800986ac838 2024-09-13T14:14:56.444 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3 smithi102 running (6m) 3s ago 6m 55.7M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 968246da9da2 2024-09-13T14:14:56.444 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4 smithi195 running (6m) 3s ago 6m 47.5M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 515307701d73 2024-09-13T14:14:56.445 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5 smithi195 running (6m) 3s ago 6m 51.9M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 176ba2b3bd59 2024-09-13T14:14:56.445 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6 smithi195 running (5m) 3s ago 5m 54.4M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 9cd070a6dd3d 2024-09-13T14:14:56.445 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7 smithi195 running (5m) 3s ago 5m 51.5M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e a6b698c659ca 2024-09-13T14:14:56.445 INFO:teuthology.orchestra.run.smithi102.stdout:prometheus.a smithi195 *:9095 running (21s) 3s ago 4m 36.1M - 2.43.0 a07b618ecd1d 82c978d3e907 2024-09-13T14:14:56.793 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:14:56.793 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": { 2024-09-13T14:14:56.793 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 3 2024-09-13T14:14:56.793 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:14:56.793 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": { 2024-09-13T14:14:56.793 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 2 2024-09-13T14:14:56.793 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:14:56.793 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": { 2024-09-13T14:14:56.794 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 8 2024-09-13T14:14:56.794 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:14:56.794 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": { 2024-09-13T14:14:56.794 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 2 2024-09-13T14:14:56.794 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:14:56.794 INFO:teuthology.orchestra.run.smithi102.stdout: "overall": { 2024-09-13T14:14:56.794 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 13, 2024-09-13T14:14:56.794 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 2 2024-09-13T14:14:56.794 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:14:56.794 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:14:57.098 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:14:57.098 INFO:teuthology.orchestra.run.smithi102.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f", 2024-09-13T14:14:57.098 INFO:teuthology.orchestra.run.smithi102.stdout: "in_progress": true, 2024-09-13T14:14:57.098 INFO:teuthology.orchestra.run.smithi102.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-09-13T14:14:57.098 INFO:teuthology.orchestra.run.smithi102.stdout: "services_complete": [ 2024-09-13T14:14:57.098 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr" 2024-09-13T14:14:57.098 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-09-13T14:14:57.099 INFO:teuthology.orchestra.run.smithi102.stdout: "progress": "2/2 daemons upgraded", 2024-09-13T14:14:57.099 INFO:teuthology.orchestra.run.smithi102.stdout: "message": "", 2024-09-13T14:14:57.099 INFO:teuthology.orchestra.run.smithi102.stdout: "is_paused": false 2024-09-13T14:14:57.099 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:14:57.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:57 smithi102 ceph-mon[40543]: from='client.25544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:14:57.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:57 smithi102 ceph-mon[40543]: pgmap v7: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-13T14:14:57.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:57 smithi102 ceph-mon[40543]: from='client.15654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:14:57.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:57 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:14:57.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:57 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:14:57.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:57 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:57.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:57 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:14:57.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:57 smithi102 ceph-mon[40543]: Updating smithi102:/etc/ceph/ceph.conf 2024-09-13T14:14:57.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:57 smithi102 ceph-mon[40543]: Updating smithi195:/etc/ceph/ceph.conf 2024-09-13T14:14:57.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:57 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/2954315127' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:14:57.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:57 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:14:57.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:57 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:14:57.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:57 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:14:57.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:57 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:14:57.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:57 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:14:57.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:57 smithi102 ceph-mon[29355]: from='client.25544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:14:57.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:57 smithi102 ceph-mon[29355]: pgmap v7: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-13T14:14:57.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:57 smithi102 ceph-mon[29355]: from='client.15654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:14:57.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:57 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:14:57.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:57 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:14:57.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:57 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:57.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:57 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:14:57.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:57 smithi102 ceph-mon[29355]: Updating smithi102:/etc/ceph/ceph.conf 2024-09-13T14:14:57.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:57 smithi102 ceph-mon[29355]: Updating smithi195:/etc/ceph/ceph.conf 2024-09-13T14:14:57.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:57 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/2954315127' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:14:57.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:57 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:14:57.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:57 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:14:57.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:57 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:14:57.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:57 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:14:57.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:57 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:14:57.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:57 smithi195 ceph-mon[33205]: from='client.25544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:14:57.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:57 smithi195 ceph-mon[33205]: pgmap v7: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-13T14:14:57.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:57 smithi195 ceph-mon[33205]: from='client.15654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:14:57.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:57 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:14:57.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:57 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:14:57.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:57 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:14:57.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:57 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:14:57.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:57 smithi195 ceph-mon[33205]: Updating smithi102:/etc/ceph/ceph.conf 2024-09-13T14:14:57.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:57 smithi195 ceph-mon[33205]: Updating smithi195:/etc/ceph/ceph.conf 2024-09-13T14:14:57.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:57 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/2954315127' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:14:57.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:57 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:14:57.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:57 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:14:57.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:57 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:14:57.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:57 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:14:57.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:57 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:14:58.299 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:58 smithi195 ceph-mon[33205]: Updating smithi102:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:14:58.300 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:58 smithi195 ceph-mon[33205]: Updating smithi195:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:14:58.300 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:58 smithi195 ceph-mon[33205]: from='client.15660 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:14:58.300 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:58 smithi195 ceph-mon[33205]: Updating smithi102:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:14:58.300 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:58 smithi195 ceph-mon[33205]: Updating smithi195:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:14:58.300 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:58 smithi195 ceph-mon[33205]: Updating smithi102:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:14:58.300 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:58 smithi195 ceph-mon[33205]: Updating smithi195:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:14:58.300 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:58 smithi195 ceph-mon[33205]: from='client.15672 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:14:58.300 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:58 smithi195 ceph-mon[33205]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-13T14:14:58.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:58 smithi102 ceph-mon[29355]: Updating smithi102:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:14:58.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:58 smithi102 ceph-mon[29355]: Updating smithi195:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:14:58.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:58 smithi102 ceph-mon[29355]: from='client.15660 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:14:58.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:58 smithi102 ceph-mon[29355]: Updating smithi102:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:14:58.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:58 smithi102 ceph-mon[29355]: Updating smithi195:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:14:58.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:58 smithi102 ceph-mon[29355]: Updating smithi102:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:14:58.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:58 smithi102 ceph-mon[29355]: Updating smithi195:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:14:58.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:58 smithi102 ceph-mon[29355]: from='client.15672 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:14:58.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:58 smithi102 ceph-mon[29355]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-13T14:14:58.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:58 smithi102 ceph-mon[40543]: Updating smithi102:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:14:58.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:58 smithi102 ceph-mon[40543]: Updating smithi195:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:14:58.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:58 smithi102 ceph-mon[40543]: from='client.15660 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:14:58.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:58 smithi102 ceph-mon[40543]: Updating smithi102:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:14:58.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:58 smithi102 ceph-mon[40543]: Updating smithi195:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:14:58.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:58 smithi102 ceph-mon[40543]: Updating smithi102:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:14:58.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:58 smithi102 ceph-mon[40543]: Updating smithi195:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:14:58.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:58 smithi102 ceph-mon[40543]: from='client.15672 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:14:58.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:58 smithi102 ceph-mon[40543]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-13T14:14:59.290 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:14:59 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[69313]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:14:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:14:59.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:59 smithi195 ceph-mon[33205]: Reconfiguring daemon prometheus.a on smithi195 2024-09-13T14:14:59.290 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:14:59 smithi195 ceph-mon[33205]: pgmap v8: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-13T14:14:59.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:59 smithi102 ceph-mon[29355]: Reconfiguring daemon prometheus.a on smithi195 2024-09-13T14:14:59.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:14:59 smithi102 ceph-mon[29355]: pgmap v8: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-13T14:14:59.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:59 smithi102 ceph-mon[40543]: Reconfiguring daemon prometheus.a on smithi195 2024-09-13T14:14:59.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:14:59 smithi102 ceph-mon[40543]: pgmap v8: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-13T14:15:00.985 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:00 smithi195 systemd[1]: Stopping Ceph prometheus.a for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T14:15:01.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:01 smithi195 ceph-mon[33205]: pgmap v9: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-13T14:15:01.248 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:00 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[78196]: ts=2024-09-13T14:15:00.984Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-13T14:15:01.248 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:00 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[78196]: ts=2024-09-13T14:15:00.984Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-13T14:15:01.248 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:00 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[78196]: ts=2024-09-13T14:15:00.984Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-13T14:15:01.248 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:00 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[78196]: ts=2024-09-13T14:15:00.984Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-13T14:15:01.249 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:00 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[78196]: ts=2024-09-13T14:15:00.984Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-13T14:15:01.249 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:00 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[78196]: ts=2024-09-13T14:15:00.984Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-13T14:15:01.249 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:00 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[78196]: ts=2024-09-13T14:15:00.984Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-13T14:15:01.249 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:00 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[78196]: ts=2024-09-13T14:15:00.984Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-13T14:15:01.249 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:00 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[78196]: ts=2024-09-13T14:15:00.985Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-13T14:15:01.250 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:00 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[78196]: ts=2024-09-13T14:15:00.985Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-13T14:15:01.250 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:00 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[78196]: ts=2024-09-13T14:15:00.985Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-13T14:15:01.250 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:00 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[78196]: ts=2024-09-13T14:15:00.985Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-13T14:15:01.250 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:00 smithi195 podman[80818]: 2024-09-13 14:15:00.995396886 +0000 UTC m=+0.120179493 container died 82c978d3e907067efe6a97a191137300f8cb862bc67e30000c1fbe2ce33b9cbc (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:15:01.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:01 smithi102 ceph-mon[29355]: pgmap v9: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-13T14:15:01.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:01 smithi102 ceph-mon[40543]: pgmap v9: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-13T14:15:01.734 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:01 smithi195 podman[80818]: 2024-09-13 14:15:01.465881805 +0000 UTC m=+0.590664415 container cleanup 82c978d3e907067efe6a97a191137300f8cb862bc67e30000c1fbe2ce33b9cbc (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:15:01.734 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:01 smithi195 bash[80818]: ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a 2024-09-13T14:15:01.997 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:01 smithi195 podman[80835]: 2024-09-13 14:15:01.733829492 +0000 UTC m=+0.741096386 container remove 82c978d3e907067efe6a97a191137300f8cb862bc67e30000c1fbe2ce33b9cbc (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:15:01.997 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:01 smithi195 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-09-13T14:15:01.997 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:01 smithi195 systemd[1]: Stopped Ceph prometheus.a for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:15:01.998 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:01 smithi195 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@prometheus.a.service: Consumed 1.094s CPU time. 2024-09-13T14:15:01.998 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:01 smithi195 systemd[1]: Starting Ceph prometheus.a for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T14:15:02.358 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:02 smithi195 podman[80936]: 2024-09-13 14:15:02.008542025 +0000 UTC m=+0.016781538 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-09-13T14:15:02.358 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:02 smithi195 podman[80936]: 2024-09-13 14:15:02.134929908 +0000 UTC m=+0.143169414 container create c6714032dc017193cc193f8257c4909e40db1e8b477ddac883f02ba31a0cb5a7 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:15:02.359 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:02 smithi195 podman[80936]: 2024-09-13 14:15:02.35362248 +0000 UTC m=+0.361861989 container init c6714032dc017193cc193f8257c4909e40db1e8b477ddac883f02ba31a0cb5a7 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:15:02.747 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:02 smithi195 podman[80936]: 2024-09-13 14:15:02.358413417 +0000 UTC m=+0.366652918 container start c6714032dc017193cc193f8257c4909e40db1e8b477ddac883f02ba31a0cb5a7 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:15:02.748 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:02 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[80950]: ts=2024-09-13T14:15:02.383Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-13T14:15:02.748 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:02 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[80950]: ts=2024-09-13T14:15:02.383Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-13T14:15:02.748 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:02 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[80950]: ts=2024-09-13T14:15:02.383Z caller=main.go:570 level=info host_details="(Linux 5.14.0-505.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024 x86_64 smithi195 (none))" 2024-09-13T14:15:02.748 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:02 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[80950]: ts=2024-09-13T14:15:02.383Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-13T14:15:02.748 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:02 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[80950]: ts=2024-09-13T14:15:02.383Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-13T14:15:02.748 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:02 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[80950]: ts=2024-09-13T14:15:02.384Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-13T14:15:02.748 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:02 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[80950]: ts=2024-09-13T14:15:02.385Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-13T14:15:02.748 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:02 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[80950]: ts=2024-09-13T14:15:02.385Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-13T14:15:02.748 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:02 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[80950]: ts=2024-09-13T14:15:02.385Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-13T14:15:02.748 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:02 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[80950]: ts=2024-09-13T14:15:02.388Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-13T14:15:02.749 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:02 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[80950]: ts=2024-09-13T14:15:02.388Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.703µs 2024-09-13T14:15:02.749 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:02 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[80950]: ts=2024-09-13T14:15:02.388Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-13T14:15:02.749 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:02 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[80950]: ts=2024-09-13T14:15:02.389Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3 2024-09-13T14:15:02.749 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:02 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[80950]: ts=2024-09-13T14:15:02.395Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3 2024-09-13T14:15:02.749 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:02 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[80950]: ts=2024-09-13T14:15:02.397Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3 2024-09-13T14:15:02.749 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:02 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[80950]: ts=2024-09-13T14:15:02.398Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3 2024-09-13T14:15:02.749 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:02 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[80950]: ts=2024-09-13T14:15:02.398Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=19.816µs wal_replay_duration=9.826552ms wbl_replay_duration=133ns total_replay_duration=9.864583ms 2024-09-13T14:15:02.749 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:02 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[80950]: ts=2024-09-13T14:15:02.399Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-13T14:15:02.749 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:02 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[80950]: ts=2024-09-13T14:15:02.399Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-13T14:15:02.749 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:02 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[80950]: ts=2024-09-13T14:15:02.399Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-13T14:15:02.749 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:02 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[80950]: ts=2024-09-13T14:15:02.419Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=19.350071ms db_storage=777ns remote_storage=985ns web_handler=418ns query_engine=586ns scrape=382.453µs scrape_sd=94.728µs notify=17.386µs notify_sd=20.64µs rules=18.494824ms tracing=5.298µs 2024-09-13T14:15:02.749 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:02 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[80950]: ts=2024-09-13T14:15:02.419Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-13T14:15:02.749 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:02 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[80950]: ts=2024-09-13T14:15:02.419Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-13T14:15:02.750 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:02 smithi195 bash[80936]: c6714032dc017193cc193f8257c4909e40db1e8b477ddac883f02ba31a0cb5a7 2024-09-13T14:15:02.750 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:02 smithi195 systemd[1]: Started Ceph prometheus.a for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:15:03.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:03 smithi195 ceph-mon[33205]: pgmap v10: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-13T14:15:03.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:03 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:03.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:03 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:03.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:03 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:15:03.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:03 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:15:03.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:03 smithi102 ceph-mon[40543]: pgmap v10: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-13T14:15:03.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:03 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:03.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:03 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:03.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:03 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:15:03.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:03 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:15:03.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:03 smithi102 ceph-mon[29355]: pgmap v10: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-13T14:15:03.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:03 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:03.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:03 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:03.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:03 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:15:03.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:03 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:15:04.488 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:04 smithi195 ceph-mon[33205]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:15:04.488 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:04 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:15:04.488 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:04 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:15:04.489 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:04 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:15:04.489 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:04 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:15:04.489 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:04 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:15:04.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:04 smithi102 ceph-mon[29355]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:15:04.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:04 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:15:04.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:04 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:15:04.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:04 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:15:04.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:04 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:15:04.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:04 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:15:04.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:04 smithi102 ceph-mon[40543]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:15:04.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:04 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:15:04.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:04 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:15:04.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:04 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:15:04.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:04 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:15:04.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:04 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:15:05.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:05 smithi195 ceph-mon[33205]: Upgrade: Updating mgr.x 2024-09-13T14:15:05.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:05 smithi195 ceph-mon[33205]: Deploying daemon mgr.x on smithi195 2024-09-13T14:15:05.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:05 smithi195 ceph-mon[33205]: pgmap v11: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-13T14:15:05.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:05 smithi102 ceph-mon[29355]: Upgrade: Updating mgr.x 2024-09-13T14:15:05.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:05 smithi102 ceph-mon[29355]: Deploying daemon mgr.x on smithi195 2024-09-13T14:15:05.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:05 smithi102 ceph-mon[29355]: pgmap v11: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-13T14:15:05.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:05 smithi102 ceph-mon[40543]: Upgrade: Updating mgr.x 2024-09-13T14:15:05.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:05 smithi102 ceph-mon[40543]: Deploying daemon mgr.x on smithi195 2024-09-13T14:15:05.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:05 smithi102 ceph-mon[40543]: pgmap v11: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-13T14:15:05.871 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:15:05 smithi195 systemd[1]: Stopping Ceph mgr.x for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T14:15:06.247 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:15:05 smithi195 podman[81337]: 2024-09-13 14:15:05.87070483 +0000 UTC m=+0.150624117 container died 2d56352abb141fcda4a606dbb4971233f7350801edb35cb1c88112c18a4b174d (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-09-13T14:15:06.648 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:15:06 smithi195 podman[81337]: 2024-09-13 14:15:06.387809411 +0000 UTC m=+0.667728699 container cleanup 2d56352abb141fcda4a606dbb4971233f7350801edb35cb1c88112c18a4b174d (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-09-13T14:15:06.648 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:15:06 smithi195 bash[81337]: ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x 2024-09-13T14:15:06.997 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:15:06 smithi195 podman[81355]: 2024-09-13 14:15:06.647698071 +0000 UTC m=+0.779601677 container remove 2d56352abb141fcda4a606dbb4971233f7350801edb35cb1c88112c18a4b174d (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-aclamk-cbt-combined-d58f730, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T14:15:06.997 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:15:06 smithi195 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-09-13T14:15:06.997 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:15:06 smithi195 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-09-13T14:15:06.997 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:15:06 smithi195 systemd[1]: Stopped Ceph mgr.x for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:15:06.997 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:15:06 smithi195 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@mgr.x.service: Consumed 17.163s CPU time. 2024-09-13T14:15:07.413 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:07 smithi195 ceph-mon[33205]: pgmap v12: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-13T14:15:07.413 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:15:07 smithi195 systemd[1]: Starting Ceph mgr.x for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T14:15:07.413 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:15:07 smithi195 podman[81489]: 2024-09-13 14:15:07.180006123 +0000 UTC m=+0.015943114 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:15:07.413 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:15:07 smithi195 podman[81489]: 2024-09-13 14:15:07.315798142 +0000 UTC m=+0.151735167 container create 4a23b8f0ea8b5b747dff1959e85fcf3c6fcb4a43aa3c8f4ddecab77a898b6d4f (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-aclamk-cbt-combined-d58f730, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-09-13T14:15:07.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:07 smithi102 ceph-mon[29355]: pgmap v12: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-13T14:15:07.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:07 smithi102 ceph-mon[40543]: pgmap v12: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-13T14:15:07.997 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:15:07 smithi195 podman[81489]: 2024-09-13 14:15:07.518125307 +0000 UTC m=+0.354062287 container init 4a23b8f0ea8b5b747dff1959e85fcf3c6fcb4a43aa3c8f4ddecab77a898b6d4f (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T14:15:07.997 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:15:07 smithi195 podman[81489]: 2024-09-13 14:15:07.523332513 +0000 UTC m=+0.359269505 container start 4a23b8f0ea8b5b747dff1959e85fcf3c6fcb4a43aa3c8f4ddecab77a898b6d4f (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x, RELEASE=wip-aclamk-cbt-combined-d58f730, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-09-13T14:15:07.997 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:15:07 smithi195 ceph-mgr[81527]: -- 172.21.15.195:0/3287972479 <== mon.1 v2:172.21.15.195:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5641efd5f0e0 con 0x5641efd3d400 2024-09-13T14:15:07.998 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:15:07 smithi195 bash[81489]: 4a23b8f0ea8b5b747dff1959e85fcf3c6fcb4a43aa3c8f4ddecab77a898b6d4f 2024-09-13T14:15:07.998 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:15:07 smithi195 systemd[1]: Started Ceph mgr.x for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:15:08.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:08 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:08.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:08 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:08.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:08 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:15:08.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:08 smithi195 ceph-mon[33205]: pgmap v13: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:09.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:08 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:09.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:08 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:09.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:08 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:15:09.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:08 smithi102 ceph-mon[29355]: pgmap v13: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:09.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:08 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:09.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:08 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:09.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:08 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:15:09.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:08 smithi102 ceph-mon[40543]: pgmap v13: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:11.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:11 smithi102 ceph-mon[29355]: pgmap v14: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:11.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:11 smithi102 ceph-mon[40543]: pgmap v14: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:11.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:11 smithi195 ceph-mon[33205]: pgmap v14: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:13.137 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:15:12 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[81504]: [13/Sep/2024:14:15:12] ENGINE Bus STARTING 2024-09-13T14:15:13.137 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:15:12 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[81504]: CherryPy Checker: 2024-09-13T14:15:13.137 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:15:12 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[81504]: The Application mounted at '' has an empty config. 2024-09-13T14:15:13.138 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:15:12 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[81504]: 2024-09-13T14:15:13.138 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:15:12 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[81504]: [13/Sep/2024:14:15:12] ENGINE Serving on http://:::9283 2024-09-13T14:15:13.138 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:15:12 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[81504]: [13/Sep/2024:14:15:12] ENGINE Bus STARTED 2024-09-13T14:15:13.138 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:13 smithi195 ceph-mon[33205]: pgmap v15: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:13.138 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:13 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:13.138 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:13 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:13.138 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:13 smithi195 ceph-mon[33205]: Standby manager daemon x restarted 2024-09-13T14:15:13.138 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:13 smithi195 ceph-mon[33205]: Standby manager daemon x started 2024-09-13T14:15:13.138 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:13 smithi195 ceph-mon[33205]: from='mgr.? 172.21.15.195:0/110678262' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:15:13.139 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:13 smithi195 ceph-mon[33205]: from='mgr.? 172.21.15.195:0/110678262' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:15:13.139 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:13 smithi195 ceph-mon[33205]: from='mgr.? 172.21.15.195:0/110678262' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:15:13.139 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:13 smithi195 ceph-mon[33205]: from='mgr.? 172.21.15.195:0/110678262' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:15:13.139 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:13 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:13.139 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:13 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:13.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:13 smithi102 ceph-mon[29355]: pgmap v15: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:13.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:13 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:13.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:13 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:13.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:13 smithi102 ceph-mon[29355]: Standby manager daemon x restarted 2024-09-13T14:15:13.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:13 smithi102 ceph-mon[29355]: Standby manager daemon x started 2024-09-13T14:15:13.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:13 smithi102 ceph-mon[29355]: from='mgr.? 172.21.15.195:0/110678262' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:15:13.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:13 smithi102 ceph-mon[29355]: from='mgr.? 172.21.15.195:0/110678262' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:15:13.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:13 smithi102 ceph-mon[29355]: from='mgr.? 172.21.15.195:0/110678262' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:15:13.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:13 smithi102 ceph-mon[29355]: from='mgr.? 172.21.15.195:0/110678262' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:15:13.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:13 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:13.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:13 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:13.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:13 smithi102 ceph-mon[40543]: pgmap v15: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:13.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:13 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:13.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:13 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:13.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:13 smithi102 ceph-mon[40543]: Standby manager daemon x restarted 2024-09-13T14:15:13.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:13 smithi102 ceph-mon[40543]: Standby manager daemon x started 2024-09-13T14:15:13.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:13 smithi102 ceph-mon[40543]: from='mgr.? 172.21.15.195:0/110678262' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:15:13.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:13 smithi102 ceph-mon[40543]: from='mgr.? 172.21.15.195:0/110678262' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:15:13.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:13 smithi102 ceph-mon[40543]: from='mgr.? 172.21.15.195:0/110678262' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:15:13.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:13 smithi102 ceph-mon[40543]: from='mgr.? 172.21.15.195:0/110678262' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:15:13.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:13 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:13.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:13 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:14.582 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:14 smithi195 ceph-mon[33205]: mgrmap e33: y(active, since 25s), standbys: x 2024-09-13T14:15:14.582 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:14 smithi195 ceph-mon[33205]: pgmap v16: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:14.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:14 smithi102 ceph-mon[29355]: mgrmap e33: y(active, since 25s), standbys: x 2024-09-13T14:15:14.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:14 smithi102 ceph-mon[29355]: pgmap v16: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:14.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:14 smithi102 ceph-mon[40543]: mgrmap e33: y(active, since 25s), standbys: x 2024-09-13T14:15:14.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:14 smithi102 ceph-mon[40543]: pgmap v16: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:17.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:17.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:17.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:15:17.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:15:17.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:17.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[29355]: pgmap v17: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:17.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:15:17.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:17.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[29355]: Upgrade: Setting container_image for all mgr 2024-09-13T14:15:17.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:17.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-13T14:15:17.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-13T14:15:17.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-13T14:15:17.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-13T14:15:17.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-13T14:15:17.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-09-13T14:15:17.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:17.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[29355]: Upgrade: Setting container_image for all crash 2024-09-13T14:15:17.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:17.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:17.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[29355]: Upgrade: Setting container_image for all rgw 2024-09-13T14:15:17.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:17.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:17.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[29355]: Upgrade: Setting container_image for all rbd-mirror 2024-09-13T14:15:17.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:17.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:17.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[29355]: Upgrade: Setting container_image for all cephfs-mirror 2024-09-13T14:15:17.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:17.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:17.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:17.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:17.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:17.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:17.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:17.172 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:17.172 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:17.172 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:15:17.172 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:15:17.172 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:17.172 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[40543]: pgmap v17: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:17.173 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:15:17.173 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:17.173 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[40543]: Upgrade: Setting container_image for all mgr 2024-09-13T14:15:17.173 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:17.173 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-13T14:15:17.173 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-13T14:15:17.173 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-13T14:15:17.173 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-13T14:15:17.173 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-13T14:15:17.173 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-09-13T14:15:17.173 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:17.173 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[40543]: Upgrade: Setting container_image for all crash 2024-09-13T14:15:17.173 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:17.173 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:17.174 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[40543]: Upgrade: Setting container_image for all rgw 2024-09-13T14:15:17.174 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:17.174 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:17.174 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[40543]: Upgrade: Setting container_image for all rbd-mirror 2024-09-13T14:15:17.174 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:17.174 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:17.174 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[40543]: Upgrade: Setting container_image for all cephfs-mirror 2024-09-13T14:15:17.174 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:17.174 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:17.174 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:17.174 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:17.174 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:17.174 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:17.175 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:17.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:17.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:17.175 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:17.175 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:17.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:17 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:17.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:17 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:17.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:17 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:15:17.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:17 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:15:17.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:17 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:17.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:17 smithi195 ceph-mon[33205]: pgmap v17: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:17.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:17 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:15:17.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:17 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:17.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:17 smithi195 ceph-mon[33205]: Upgrade: Setting container_image for all mgr 2024-09-13T14:15:17.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:17 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:17.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:17 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-13T14:15:17.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:17 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-13T14:15:17.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:17 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-13T14:15:17.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:17 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-13T14:15:17.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:17 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-13T14:15:17.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:17 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-09-13T14:15:17.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:17 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:17.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:17 smithi195 ceph-mon[33205]: Upgrade: Setting container_image for all crash 2024-09-13T14:15:17.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:17 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:17.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:17 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:17.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:17 smithi195 ceph-mon[33205]: Upgrade: Setting container_image for all rgw 2024-09-13T14:15:17.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:17 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:17.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:17 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:17.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:17 smithi195 ceph-mon[33205]: Upgrade: Setting container_image for all rbd-mirror 2024-09-13T14:15:17.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:17 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:17.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:17 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:17.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:17 smithi195 ceph-mon[33205]: Upgrade: Setting container_image for all cephfs-mirror 2024-09-13T14:15:17.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:17 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:17.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:17 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:17.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:17 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:17.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:17 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:17.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:17 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:17.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:17 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:17.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:17 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:17.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:17 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:17.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:17 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:17.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:15:17 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:15:17] "GET /metrics HTTP/1.1" 200 33822 "" "Prometheus/2.43.0" 2024-09-13T14:15:18.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:18 smithi102 ceph-mon[40543]: Upgrade: Setting container_image for all ceph-exporter 2024-09-13T14:15:18.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:18 smithi102 ceph-mon[40543]: Upgrade: Setting container_image for all iscsi 2024-09-13T14:15:18.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:18 smithi102 ceph-mon[40543]: Upgrade: Setting container_image for all nfs 2024-09-13T14:15:18.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:18 smithi102 ceph-mon[40543]: Upgrade: Setting container_image for all nvmeof 2024-09-13T14:15:18.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:18 smithi102 ceph-mon[29355]: Upgrade: Setting container_image for all ceph-exporter 2024-09-13T14:15:18.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:18 smithi102 ceph-mon[29355]: Upgrade: Setting container_image for all iscsi 2024-09-13T14:15:18.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:18 smithi102 ceph-mon[29355]: Upgrade: Setting container_image for all nfs 2024-09-13T14:15:18.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:18 smithi102 ceph-mon[29355]: Upgrade: Setting container_image for all nvmeof 2024-09-13T14:15:18.398 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:18 smithi195 ceph-mon[33205]: Upgrade: Setting container_image for all ceph-exporter 2024-09-13T14:15:18.398 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:18 smithi195 ceph-mon[33205]: Upgrade: Setting container_image for all iscsi 2024-09-13T14:15:18.399 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:18 smithi195 ceph-mon[33205]: Upgrade: Setting container_image for all nfs 2024-09-13T14:15:18.399 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:18 smithi195 ceph-mon[33205]: Upgrade: Setting container_image for all nvmeof 2024-09-13T14:15:18.657 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:18 smithi102 systemd[1]: Stopping Ceph node-exporter.a for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T14:15:18.918 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:18 smithi102 podman[100014]: 2024-09-13 14:15:18.656925279 +0000 UTC m=+0.129686692 container died 5538bdf011a365b78667629853a501e81070d1747dac9607be84d5fa976330f9 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T14:15:19.414 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:19 smithi195 ceph-mon[33205]: Upgrade: Updating node-exporter.a (1/2) 2024-09-13T14:15:19.415 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:19 smithi195 ceph-mon[33205]: Deploying daemon node-exporter.a on smithi102 2024-09-13T14:15:19.415 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:19 smithi195 ceph-mon[33205]: pgmap v18: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:19.415 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:19 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:15:19.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:19 smithi102 ceph-mon[29355]: Upgrade: Updating node-exporter.a (1/2) 2024-09-13T14:15:19.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:19 smithi102 ceph-mon[29355]: Deploying daemon node-exporter.a on smithi102 2024-09-13T14:15:19.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:19 smithi102 ceph-mon[29355]: pgmap v18: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:19.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:19 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:15:19.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:19 smithi102 ceph-mon[40543]: Upgrade: Updating node-exporter.a (1/2) 2024-09-13T14:15:19.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:19 smithi102 ceph-mon[40543]: Deploying daemon node-exporter.a on smithi102 2024-09-13T14:15:19.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:19 smithi102 ceph-mon[40543]: pgmap v18: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:19.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:19 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:15:19.419 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:19 smithi102 podman[100014]: 2024-09-13 14:15:19.145002932 +0000 UTC m=+0.617764348 container cleanup 5538bdf011a365b78667629853a501e81070d1747dac9607be84d5fa976330f9 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T14:15:19.419 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:19 smithi102 bash[100014]: ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a 2024-09-13T14:15:19.786 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:19 smithi102 podman[100032]: 2024-09-13 14:15:19.443262754 +0000 UTC m=+0.790195153 container remove 5538bdf011a365b78667629853a501e81070d1747dac9607be84d5fa976330f9 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T14:15:19.786 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:19 smithi102 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-13T14:15:19.786 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:19 smithi102 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-13T14:15:19.786 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:19 smithi102 systemd[1]: Stopped Ceph node-exporter.a for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:15:19.786 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:19 smithi102 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@node-exporter.a.service: Consumed 2.305s CPU time. 2024-09-13T14:15:20.168 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:19 smithi102 systemd[1]: Starting Ceph node-exporter.a for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T14:15:20.168 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:19 smithi102 bash[100163]: Trying to pull quay.io/prometheus/node-exporter:v1.7.0... 2024-09-13T14:15:20.668 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:20 smithi102 bash[100163]: Getting image source signatures 2024-09-13T14:15:20.668 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:20 smithi102 bash[100163]: Copying blob sha256:2abcce694348cd2c949c0e98a7400ebdfd8341021bcf6b541bc72033ce982510 2024-09-13T14:15:20.668 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:20 smithi102 bash[100163]: Copying blob sha256:455fd88e5221bc1e278ef2d059cd70e4df99a24e5af050ede621534276f6cf9a 2024-09-13T14:15:20.668 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:20 smithi102 bash[100163]: Copying blob sha256:324153f2810a9927fcce320af9e4e291e0b6e805cbdd1f338386c756b9defa24 2024-09-13T14:15:21.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:21 smithi102 ceph-mon[29355]: pgmap v19: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:21.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:21 smithi102 ceph-mon[40543]: pgmap v19: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:21.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:21 smithi195 ceph-mon[33205]: pgmap v19: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:22.089 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:21 smithi102 bash[100163]: Copying config sha256:72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e 2024-09-13T14:15:22.089 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:21 smithi102 bash[100163]: Writing manifest to image destination 2024-09-13T14:15:22.413 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 podman[100163]: 2024-09-13 14:15:22.200306945 +0000 UTC m=+2.321663185 image pull 72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e quay.io/prometheus/node-exporter:v1.7.0 2024-09-13T14:15:22.413 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 podman[100163]: 2024-09-13 14:15:22.307617057 +0000 UTC m=+2.428973297 container create 0c92d0b39468100a77b6682d8279999de190faa4885fa01d6000970e4a5aedbd (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T14:15:22.918 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 podman[100163]: 2024-09-13 14:15:22.526198231 +0000 UTC m=+2.647554478 container init 0c92d0b39468100a77b6682d8279999de190faa4885fa01d6000970e4a5aedbd (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T14:15:22.919 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 podman[100163]: 2024-09-13 14:15:22.531455091 +0000 UTC m=+2.652811331 container start 0c92d0b39468100a77b6682d8279999de190faa4885fa01d6000970e4a5aedbd (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T14:15:22.919 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.534Z caller=node_exporter.go:192 level=info msg="Starting node_exporter" version="(version=1.7.0, branch=HEAD, revision=7333465abf9efba81876303bb57e6fadb946041b)" 2024-09-13T14:15:22.919 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.534Z caller=node_exporter.go:193 level=info msg="Build context" build_context="(go=go1.21.4, platform=linux/amd64, user=root@35918982f6d8, date=20231112-23:53:35, tags=netgo osusergo static_build)" 2024-09-13T14:15:22.919 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.534Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-13T14:15:22.919 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.534Z caller=diskstats_linux.go:265 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-13T14:15:22.919 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.535Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-13T14:15:22.919 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.535Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-13T14:15:22.919 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.535Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-13T14:15:22.919 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.535Z caller=node_exporter.go:117 level=info collector=arp 2024-09-13T14:15:22.919 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.535Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-13T14:15:22.919 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.535Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-13T14:15:22.919 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.535Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-13T14:15:22.920 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.535Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-13T14:15:22.920 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.535Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-13T14:15:22.920 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.535Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-13T14:15:22.920 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.535Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-13T14:15:22.920 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.535Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-13T14:15:22.920 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.535Z caller=node_exporter.go:117 level=info collector=edac 2024-09-13T14:15:22.921 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.535Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-13T14:15:22.921 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.535Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-13T14:15:22.921 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.535Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-13T14:15:22.921 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.535Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-13T14:15:22.921 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.535Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-13T14:15:22.921 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.535Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-13T14:15:22.921 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.535Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-13T14:15:22.921 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.535Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-13T14:15:22.921 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.535Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-13T14:15:22.921 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.535Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-13T14:15:22.921 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.535Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-13T14:15:22.921 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.535Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-13T14:15:22.921 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.535Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-13T14:15:22.922 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.535Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-13T14:15:22.922 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.535Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-13T14:15:22.922 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.535Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-13T14:15:22.922 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.535Z caller=node_exporter.go:117 level=info collector=os 2024-09-13T14:15:22.922 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.535Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-13T14:15:22.922 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.535Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-13T14:15:22.922 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.535Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-13T14:15:22.922 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.535Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-13T14:15:22.922 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.535Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-13T14:15:22.922 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.535Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-13T14:15:22.922 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.535Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-13T14:15:22.922 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.535Z caller=node_exporter.go:117 level=info collector=stat 2024-09-13T14:15:22.922 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.535Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-13T14:15:22.923 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.535Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-13T14:15:22.923 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.535Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-13T14:15:22.923 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.535Z caller=node_exporter.go:117 level=info collector=time 2024-09-13T14:15:22.923 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.535Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-13T14:15:22.923 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.535Z caller=node_exporter.go:117 level=info collector=uname 2024-09-13T14:15:22.923 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.535Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-13T14:15:22.923 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.535Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-13T14:15:22.923 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.535Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-13T14:15:22.923 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.536Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9100 2024-09-13T14:15:22.923 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a[100248]: ts=2024-09-13T14:15:22.536Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-13T14:15:22.923 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 bash[100163]: 0c92d0b39468100a77b6682d8279999de190faa4885fa01d6000970e4a5aedbd 2024-09-13T14:15:22.923 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 14:15:22 smithi102 systemd[1]: Started Ceph node-exporter.a for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:15:23.406 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:23 smithi195 ceph-mon[33205]: pgmap v20: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:23.406 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:23 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:23.406 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:23 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:23.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:23 smithi102 ceph-mon[29355]: pgmap v20: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:23.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:23 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:23.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:23 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:23.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:23 smithi102 ceph-mon[40543]: pgmap v20: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:23.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:23 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:23.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:23 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:24.747 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:24 smithi195 systemd[1]: Stopping Ceph node-exporter.b for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T14:15:25.107 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:24 smithi195 podman[85181]: 2024-09-13 14:15:24.85105533 +0000 UTC m=+0.131147041 container died 138397d5e1298120d1adb47e3c5644b25f1a0470fdee6bb1538874167f71f490 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T14:15:25.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:25 smithi102 ceph-mon[29355]: Upgrade: Updating node-exporter.b (2/2) 2024-09-13T14:15:25.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:25 smithi102 ceph-mon[29355]: Deploying daemon node-exporter.b on smithi195 2024-09-13T14:15:25.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:25 smithi102 ceph-mon[29355]: pgmap v21: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:25.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:25 smithi102 ceph-mon[40543]: Upgrade: Updating node-exporter.b (2/2) 2024-09-13T14:15:25.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:25 smithi102 ceph-mon[40543]: Deploying daemon node-exporter.b on smithi195 2024-09-13T14:15:25.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:25 smithi102 ceph-mon[40543]: pgmap v21: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:25.489 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:25 smithi195 podman[85181]: 2024-09-13 14:15:25.205004669 +0000 UTC m=+0.485096547 container cleanup 138397d5e1298120d1adb47e3c5644b25f1a0470fdee6bb1538874167f71f490 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T14:15:25.490 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:25 smithi195 bash[85181]: ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b 2024-09-13T14:15:25.490 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:25 smithi195 ceph-mon[33205]: Upgrade: Updating node-exporter.b (2/2) 2024-09-13T14:15:25.490 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:25 smithi195 ceph-mon[33205]: Deploying daemon node-exporter.b on smithi195 2024-09-13T14:15:25.490 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:25 smithi195 ceph-mon[33205]: pgmap v21: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:25.747 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:25 smithi195 podman[85200]: 2024-09-13 14:15:25.489542073 +0000 UTC m=+0.642222317 container remove 138397d5e1298120d1adb47e3c5644b25f1a0470fdee6bb1538874167f71f490 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T14:15:25.747 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:25 smithi195 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-13T14:15:25.747 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:25 smithi195 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-13T14:15:25.747 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:25 smithi195 systemd[1]: Stopped Ceph node-exporter.b for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:15:25.748 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:25 smithi195 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@node-exporter.b.service: Consumed 2.338s CPU time. 2024-09-13T14:15:26.247 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:25 smithi195 systemd[1]: Starting Ceph node-exporter.b for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T14:15:26.247 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:26 smithi195 bash[85336]: Trying to pull quay.io/prometheus/node-exporter:v1.7.0... 2024-09-13T14:15:26.747 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:26 smithi195 bash[85336]: Getting image source signatures 2024-09-13T14:15:26.747 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:26 smithi195 bash[85336]: Copying blob sha256:455fd88e5221bc1e278ef2d059cd70e4df99a24e5af050ede621534276f6cf9a 2024-09-13T14:15:26.747 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:26 smithi195 bash[85336]: Copying blob sha256:2abcce694348cd2c949c0e98a7400ebdfd8341021bcf6b541bc72033ce982510 2024-09-13T14:15:26.747 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:26 smithi195 bash[85336]: Copying blob sha256:324153f2810a9927fcce320af9e4e291e0b6e805cbdd1f338386c756b9defa24 2024-09-13T14:15:27.406 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:27 smithi102 ceph-mon[40543]: pgmap v22: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:27.406 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:27 smithi102 ceph-mon[29355]: pgmap v22: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:27.407 INFO:teuthology.orchestra.run.smithi102.stdout:true 2024-09-13T14:15:27.407 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:15:27 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:15:27] "GET /metrics HTTP/1.1" 200 33824 "" "Prometheus/2.43.0" 2024-09-13T14:15:27.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:27 smithi195 ceph-mon[33205]: pgmap v22: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:27.956 INFO:teuthology.orchestra.run.smithi102.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-13T14:15:27.956 INFO:teuthology.orchestra.run.smithi102.stdout:alertmanager.a smithi102 *:9093,9094 running (4m) 35s ago 5m 27.8M - 0.25.0 c8568f914cd2 fadcd26e5395 2024-09-13T14:15:27.956 INFO:teuthology.orchestra.run.smithi102.stdout:grafana.a smithi195 *:3000 running (4m) 15s ago 4m 85.0M - 9.4.7 954c08fa6188 4e2eb92e9f89 2024-09-13T14:15:27.956 INFO:teuthology.orchestra.run.smithi102.stdout:mds.foo.smithi102.njetxr smithi102 running (3m) 35s ago 3m 18.9M - 18.2.4-843-g2c01458a c129f5a2fd4e c9211cc725ed 2024-09-13T14:15:27.956 INFO:teuthology.orchestra.run.smithi102.stdout:mds.foo.smithi195.yqzrxi smithi195 running (3m) 15s ago 3m 16.9M - 18.2.4-843-g2c01458a c129f5a2fd4e eea37a7230fd 2024-09-13T14:15:27.956 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.x smithi195 *:8443,9283,8765 running (20s) 15s ago 8m 69.7M - 19.3.0-4697-gd58f7306 fec07016a2b0 4a23b8f0ea8b 2024-09-13T14:15:27.956 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.y smithi102 *:8443,9283,8765 running (48s) 35s ago 10m 486M - 19.3.0-4697-gd58f7306 fec07016a2b0 6b12b28dcde1 2024-09-13T14:15:27.956 INFO:teuthology.orchestra.run.smithi102.stdout:mon.a smithi102 running (10m) 35s ago 10m 53.9M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e 22c604c4fd02 2024-09-13T14:15:27.956 INFO:teuthology.orchestra.run.smithi102.stdout:mon.b smithi195 running (9m) 15s ago 9m 46.5M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e 608ec4847676 2024-09-13T14:15:27.957 INFO:teuthology.orchestra.run.smithi102.stdout:mon.c smithi102 running (9m) 35s ago 9m 43.1M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e 6c1fd3099a05 2024-09-13T14:15:27.957 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.a smithi102 *:9100 starting - - - - 2024-09-13T14:15:27.957 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.b smithi195 *:9100 running (4m) 15s ago 4m 14.7M - 1.5.0 0da6a335fe13 138397d5e129 2024-09-13T14:15:27.957 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0 smithi102 running (8m) 35s ago 8m 59.1M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e b97482fb8cf9 2024-09-13T14:15:27.957 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1 smithi102 running (8m) 35s ago 8m 50.6M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 8a67356be7a2 2024-09-13T14:15:27.957 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2 smithi102 running (7m) 35s ago 7m 48.7M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 1800986ac838 2024-09-13T14:15:27.957 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3 smithi102 running (7m) 35s ago 7m 55.7M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 968246da9da2 2024-09-13T14:15:27.957 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4 smithi195 running (7m) 15s ago 7m 47.5M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 515307701d73 2024-09-13T14:15:27.957 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5 smithi195 running (6m) 15s ago 6m 51.9M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 176ba2b3bd59 2024-09-13T14:15:27.957 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6 smithi195 running (6m) 15s ago 6m 54.4M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 9cd070a6dd3d 2024-09-13T14:15:27.957 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7 smithi195 running (5m) 15s ago 5m 51.5M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e a6b698c659ca 2024-09-13T14:15:27.957 INFO:teuthology.orchestra.run.smithi102.stdout:prometheus.a smithi195 *:9095 running (25s) 15s ago 5m 33.6M - 2.43.0 a07b618ecd1d c6714032dc01 2024-09-13T14:15:28.247 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:27 smithi195 bash[85336]: Copying config sha256:72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e 2024-09-13T14:15:28.247 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 bash[85336]: Writing manifest to image destination 2024-09-13T14:15:28.288 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:15:28.288 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": { 2024-09-13T14:15:28.288 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 3 2024-09-13T14:15:28.288 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:15:28.288 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": { 2024-09-13T14:15:28.288 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 2 2024-09-13T14:15:28.288 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:15:28.288 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": { 2024-09-13T14:15:28.288 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 8 2024-09-13T14:15:28.288 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:15:28.288 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": { 2024-09-13T14:15:28.289 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 2 2024-09-13T14:15:28.289 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:15:28.289 INFO:teuthology.orchestra.run.smithi102.stdout: "overall": { 2024-09-13T14:15:28.289 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 13, 2024-09-13T14:15:28.289 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 2 2024-09-13T14:15:28.289 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:15:28.289 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:15:28.571 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:15:28.571 INFO:teuthology.orchestra.run.smithi102.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f", 2024-09-13T14:15:28.571 INFO:teuthology.orchestra.run.smithi102.stdout: "in_progress": true, 2024-09-13T14:15:28.571 INFO:teuthology.orchestra.run.smithi102.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-09-13T14:15:28.571 INFO:teuthology.orchestra.run.smithi102.stdout: "services_complete": [ 2024-09-13T14:15:28.571 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr" 2024-09-13T14:15:28.571 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-09-13T14:15:28.572 INFO:teuthology.orchestra.run.smithi102.stdout: "progress": "2/2 daemons upgraded", 2024-09-13T14:15:28.572 INFO:teuthology.orchestra.run.smithi102.stdout: "message": "Currently upgrading node-exporter daemons", 2024-09-13T14:15:28.572 INFO:teuthology.orchestra.run.smithi102.stdout: "is_paused": false 2024-09-13T14:15:28.572 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:15:28.735 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 podman[85336]: 2024-09-13 14:15:28.351974321 +0000 UTC m=+2.422928742 image pull 72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e quay.io/prometheus/node-exporter:v1.7.0 2024-09-13T14:15:28.735 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 podman[85336]: 2024-09-13 14:15:28.475231819 +0000 UTC m=+2.546186236 container create fde867a7dbf91652db7c868363d7d2cc2593716918a95075e2ea9cebc614ab0f (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T14:15:29.149 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 podman[85336]: 2024-09-13 14:15:28.852834101 +0000 UTC m=+2.923788523 container init fde867a7dbf91652db7c868363d7d2cc2593716918a95075e2ea9cebc614ab0f (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T14:15:29.149 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 podman[85336]: 2024-09-13 14:15:28.858287065 +0000 UTC m=+2.929241484 container start fde867a7dbf91652db7c868363d7d2cc2593716918a95075e2ea9cebc614ab0f (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T14:15:29.149 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.862Z caller=node_exporter.go:192 level=info msg="Starting node_exporter" version="(version=1.7.0, branch=HEAD, revision=7333465abf9efba81876303bb57e6fadb946041b)" 2024-09-13T14:15:29.149 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.862Z caller=node_exporter.go:193 level=info msg="Build context" build_context="(go=go1.21.4, platform=linux/amd64, user=root@35918982f6d8, date=20231112-23:53:35, tags=netgo osusergo static_build)" 2024-09-13T14:15:29.150 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.863Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-13T14:15:29.150 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.863Z caller=diskstats_linux.go:265 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-13T14:15:29.150 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.863Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-13T14:15:29.150 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.863Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-13T14:15:29.150 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.863Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-13T14:15:29.150 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.863Z caller=node_exporter.go:117 level=info collector=arp 2024-09-13T14:15:29.150 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.863Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-13T14:15:29.150 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.863Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-13T14:15:29.150 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.863Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-13T14:15:29.150 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.863Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-13T14:15:29.150 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.863Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-13T14:15:29.150 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.863Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-13T14:15:29.151 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.863Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-13T14:15:29.151 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.863Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-13T14:15:29.151 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.863Z caller=node_exporter.go:117 level=info collector=edac 2024-09-13T14:15:29.151 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.863Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-13T14:15:29.151 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.863Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-13T14:15:29.151 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.863Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-13T14:15:29.151 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.863Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-13T14:15:29.152 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.863Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-13T14:15:29.152 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.863Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-13T14:15:29.152 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.863Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-13T14:15:29.152 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.863Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-13T14:15:29.152 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.863Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-13T14:15:29.152 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.863Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-13T14:15:29.152 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.863Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-13T14:15:29.152 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.863Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-13T14:15:29.152 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.863Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-13T14:15:29.152 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.863Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-13T14:15:29.152 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.863Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-13T14:15:29.152 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.863Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-13T14:15:29.153 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.863Z caller=node_exporter.go:117 level=info collector=os 2024-09-13T14:15:29.153 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.863Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-13T14:15:29.153 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.863Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-13T14:15:29.153 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.863Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-13T14:15:29.153 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.863Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-13T14:15:29.153 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.863Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-13T14:15:29.153 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.863Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-13T14:15:29.153 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.863Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-13T14:15:29.153 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.863Z caller=node_exporter.go:117 level=info collector=stat 2024-09-13T14:15:29.153 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.863Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-13T14:15:29.153 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.863Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-13T14:15:29.153 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.863Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-13T14:15:29.153 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.863Z caller=node_exporter.go:117 level=info collector=time 2024-09-13T14:15:29.154 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.863Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-13T14:15:29.154 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.863Z caller=node_exporter.go:117 level=info collector=uname 2024-09-13T14:15:29.154 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.863Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-13T14:15:29.154 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.863Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-13T14:15:29.154 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.863Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-13T14:15:29.154 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.864Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9100 2024-09-13T14:15:29.154 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b[85410]: ts=2024-09-13T14:15:28.864Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-13T14:15:29.154 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 bash[85336]: fde867a7dbf91652db7c868363d7d2cc2593716918a95075e2ea9cebc614ab0f 2024-09-13T14:15:29.154 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 14:15:28 smithi195 systemd[1]: Started Ceph node-exporter.b for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:15:29.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:29 smithi102 ceph-mon[29355]: from='client.15684 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:15:29.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:29 smithi102 ceph-mon[29355]: from='client.15690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:15:29.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:29 smithi102 ceph-mon[29355]: from='client.15696 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:15:29.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:29 smithi102 ceph-mon[29355]: pgmap v23: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:29.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:29 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/1687965634' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:29.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:29 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:29.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:29 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:29.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:29 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:15:29.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:29 smithi102 ceph-mon[40543]: from='client.15684 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:15:29.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:29 smithi102 ceph-mon[40543]: from='client.15690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:15:29.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:29 smithi102 ceph-mon[40543]: from='client.15696 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:15:29.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:29 smithi102 ceph-mon[40543]: pgmap v23: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:29.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:29 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/1687965634' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:29.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:29 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:29.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:29 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:29.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:29 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:15:29.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:29 smithi195 ceph-mon[33205]: from='client.15684 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:15:29.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:29 smithi195 ceph-mon[33205]: from='client.15690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:15:29.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:29 smithi195 ceph-mon[33205]: from='client.15696 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:15:29.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:29 smithi195 ceph-mon[33205]: pgmap v23: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:29.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:29 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/1687965634' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:29.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:29 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:29.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:29 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:29.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:29 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:15:30.290 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:30 smithi102 ceph-mon[29355]: from='client.25601 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:15:30.290 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:30 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:30.290 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:30 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:30.290 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:30 smithi102 ceph-mon[40543]: from='client.25601 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:15:30.291 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:30 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:30.291 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:30 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:30.392 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:30 smithi195 ceph-mon[33205]: from='client.25601 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:15:30.392 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:30 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:30.392 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:30 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:31.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:31 smithi102 ceph-mon[29355]: pgmap v24: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:31.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:31 smithi102 ceph-mon[40543]: pgmap v24: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:31.480 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:31 smithi195 ceph-mon[33205]: pgmap v24: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:33.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:33 smithi102 ceph-mon[29355]: pgmap v25: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:33.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:33 smithi102 ceph-mon[40543]: pgmap v25: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:33.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:33 smithi195 ceph-mon[33205]: pgmap v25: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:34.413 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:34 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:34.413 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:34 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:34.413 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:34 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:34.414 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:34 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:34.414 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:34 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:15:34.414 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:34 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:34.414 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:34 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:34.414 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:34 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:34.414 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:34 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:34.415 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:34 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:34.415 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:34 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:34.415 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:34 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:34.415 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:34 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:34.415 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:34 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:15:34.415 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:34 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:34.415 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:34 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:34.415 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:34 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:34.416 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:34 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:34.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:34 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:34.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:34 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:34.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:34 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:34.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:34 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:34.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:34 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:15:34.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:34 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:34.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:34 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:34.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:34 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:34.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:34 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:35.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:35 smithi195 ceph-mon[33205]: pgmap v26: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:35.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:35 smithi102 ceph-mon[29355]: pgmap v26: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:35.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:35 smithi102 ceph-mon[40543]: pgmap v26: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:37.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:37 smithi102 ceph-mon[29355]: pgmap v27: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:37.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:37 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:37.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:37 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:37.419 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:15:37 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:15:37] "GET /metrics HTTP/1.1" 200 33822 "" "Prometheus/2.43.0" 2024-09-13T14:15:37.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:37 smithi102 ceph-mon[40543]: pgmap v27: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:37.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:37 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:37.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:37 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:37.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:37 smithi195 ceph-mon[33205]: pgmap v27: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:37.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:37 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:37.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:37 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:38.388 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:38 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:38.389 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:38 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:38.389 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:38 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:15:38.389 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:38 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:15:38.389 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:38 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:38.389 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:38 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:15:38.389 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:38 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:38.389 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:38 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:38.389 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:38 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:38.389 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:38 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:38.389 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:38 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:38.389 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:38 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:38.389 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:38 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:38.390 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:38 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:38.390 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:38 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:38.390 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:38 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:38.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:38 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:38.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:38 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:38.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:38 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:15:38.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:38 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:15:38.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:38 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:38.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:38 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:15:38.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:38 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:38.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:38 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:38.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:38 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:38.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:38 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:38.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:38 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:38.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:38 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:38.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:38 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:38.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:38 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:38.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:38 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:38.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:38 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:38.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:38 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:38.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:38 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:38.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:38 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:15:38.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:38 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:15:38.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:38 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:38.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:38 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:15:38.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:38 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:38.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:38 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:38.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:38 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:38.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:38 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:38.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:38 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:38.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:38 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:38.671 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:38 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:38.671 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:38 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:38.671 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:38 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:38.671 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:38 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:15:39.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:39 smithi195 ceph-mon[33205]: pgmap v28: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:39.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:39 smithi102 ceph-mon[40543]: pgmap v28: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:39.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:39 smithi102 ceph-mon[29355]: pgmap v28: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:40.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:40 smithi195 ceph-mon[33205]: Upgrade: Updating prometheus.a 2024-09-13T14:15:40.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:40 smithi195 ceph-mon[33205]: Deploying daemon prometheus.a on smithi195 2024-09-13T14:15:40.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:40 smithi102 ceph-mon[40543]: Upgrade: Updating prometheus.a 2024-09-13T14:15:40.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:40 smithi102 ceph-mon[40543]: Deploying daemon prometheus.a on smithi195 2024-09-13T14:15:40.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:40 smithi102 ceph-mon[29355]: Upgrade: Updating prometheus.a 2024-09-13T14:15:40.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:40 smithi102 ceph-mon[29355]: Deploying daemon prometheus.a on smithi195 2024-09-13T14:15:41.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:41 smithi195 ceph-mon[33205]: pgmap v29: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:41.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:41 smithi102 ceph-mon[40543]: pgmap v29: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:41.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:41 smithi102 ceph-mon[29355]: pgmap v29: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:43.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:43 smithi195 ceph-mon[33205]: pgmap v30: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:43.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:43 smithi102 ceph-mon[40543]: pgmap v30: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:43.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:43 smithi102 ceph-mon[29355]: pgmap v30: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:44.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:44 smithi195 ceph-mon[33205]: pgmap v31: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:44.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:44 smithi102 ceph-mon[29355]: pgmap v31: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:44.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:44 smithi102 ceph-mon[40543]: pgmap v31: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:47.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:47 smithi195 ceph-mon[33205]: pgmap v32: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:47.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:47 smithi102 ceph-mon[29355]: pgmap v32: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:47.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:47 smithi102 ceph-mon[40543]: pgmap v32: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:47.419 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:15:47 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:15:47] "GET /metrics HTTP/1.1" 200 33822 "" "Prometheus/2.43.0" 2024-09-13T14:15:49.379 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:49 smithi195 ceph-mon[33205]: pgmap v33: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:49.379 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:49 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:15:49.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:49 smithi102 ceph-mon[29355]: pgmap v33: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:49.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:49 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:15:49.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:49 smithi102 ceph-mon[40543]: pgmap v33: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:49.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:49 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:15:49.652 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:49 smithi195 systemd[1]: Stopping Ceph prometheus.a for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T14:15:49.951 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:49 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[80950]: ts=2024-09-13T14:15:49.651Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-13T14:15:49.951 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:49 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[80950]: ts=2024-09-13T14:15:49.651Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-13T14:15:49.951 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:49 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[80950]: ts=2024-09-13T14:15:49.651Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-13T14:15:49.951 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:49 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[80950]: ts=2024-09-13T14:15:49.651Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-13T14:15:49.951 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:49 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[80950]: ts=2024-09-13T14:15:49.651Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-13T14:15:49.951 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:49 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[80950]: ts=2024-09-13T14:15:49.651Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-13T14:15:49.951 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:49 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[80950]: ts=2024-09-13T14:15:49.651Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-13T14:15:49.951 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:49 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[80950]: ts=2024-09-13T14:15:49.651Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-13T14:15:49.951 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:49 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[80950]: ts=2024-09-13T14:15:49.651Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-13T14:15:49.951 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:49 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[80950]: ts=2024-09-13T14:15:49.652Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-13T14:15:49.951 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:49 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[80950]: ts=2024-09-13T14:15:49.652Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-13T14:15:49.952 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:49 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[80950]: ts=2024-09-13T14:15:49.652Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-13T14:15:49.952 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:49 smithi195 podman[89361]: 2024-09-13 14:15:49.661796983 +0000 UTC m=+0.137127149 container died c6714032dc017193cc193f8257c4909e40db1e8b477ddac883f02ba31a0cb5a7 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:15:50.242 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:50 smithi195 podman[89361]: 2024-09-13 14:15:50.057514166 +0000 UTC m=+0.532844329 container cleanup c6714032dc017193cc193f8257c4909e40db1e8b477ddac883f02ba31a0cb5a7 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:15:50.242 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:50 smithi195 bash[89361]: ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a 2024-09-13T14:15:50.497 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:50 smithi195 podman[89380]: 2024-09-13 14:15:50.242075807 +0000 UTC m=+0.582269536 container remove c6714032dc017193cc193f8257c4909e40db1e8b477ddac883f02ba31a0cb5a7 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:15:50.497 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:50 smithi195 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-09-13T14:15:50.497 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:50 smithi195 systemd[1]: Stopped Ceph prometheus.a for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:15:50.497 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:50 smithi195 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@prometheus.a.service: Consumed 1.149s CPU time. 2024-09-13T14:15:50.907 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:50 smithi195 systemd[1]: Starting Ceph prometheus.a for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T14:15:51.159 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:51 smithi195 ceph-mon[33205]: pgmap v34: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:51.159 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:50 smithi195 podman[89507]: 2024-09-13 14:15:50.810723998 +0000 UTC m=+0.023316984 image pull 1d3b7f56885b6dd623f1785be963aa9c195f86bc256ea454e8d02a7980b79c53 quay.io/prometheus/prometheus:v2.51.0 2024-09-13T14:15:51.159 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:50 smithi195 podman[89507]: 2024-09-13 14:15:50.930725747 +0000 UTC m=+0.143318729 container create d131e53f1ef9319ff3efc21e6bfc7a7e1bc630b1a34402d7507836f08a03326c (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:15:51.159 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:51 smithi195 podman[89507]: 2024-09-13 14:15:51.153991386 +0000 UTC m=+0.366584371 container init d131e53f1ef9319ff3efc21e6bfc7a7e1bc630b1a34402d7507836f08a03326c (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:15:51.159 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:51 smithi195 podman[89507]: 2024-09-13 14:15:51.158897945 +0000 UTC m=+0.371490927 container start d131e53f1ef9319ff3efc21e6bfc7a7e1bc630b1a34402d7507836f08a03326c (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:15:51.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:51 smithi102 ceph-mon[29355]: pgmap v34: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:51.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:51 smithi102 ceph-mon[40543]: pgmap v34: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:51.497 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:51 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[89530]: ts=2024-09-13T14:15:51.178Z caller=main.go:617 level=info msg="Starting Prometheus Server" mode=server version="(version=2.51.0, branch=HEAD, revision=c05c15512acb675e3f6cd662a6727854e93fc024)" 2024-09-13T14:15:51.498 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:51 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[89530]: ts=2024-09-13T14:15:51.178Z caller=main.go:622 level=info build_context="(go=go1.22.1, platform=linux/amd64, user=root@b5723e458358, date=20240319-10:54:45, tags=netgo,builtinassets,stringlabels)" 2024-09-13T14:15:51.498 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:51 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[89530]: ts=2024-09-13T14:15:51.178Z caller=main.go:623 level=info host_details="(Linux 5.14.0-505.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024 x86_64 smithi195 (none))" 2024-09-13T14:15:51.498 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:51 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[89530]: ts=2024-09-13T14:15:51.178Z caller=main.go:624 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-13T14:15:51.498 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:51 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[89530]: ts=2024-09-13T14:15:51.178Z caller=main.go:625 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-13T14:15:51.498 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:51 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[89530]: ts=2024-09-13T14:15:51.180Z caller=web.go:568 level=info component=web msg="Start listening for connections" address=:9095 2024-09-13T14:15:51.498 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:51 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[89530]: ts=2024-09-13T14:15:51.180Z caller=main.go:1129 level=info msg="Starting TSDB ..." 2024-09-13T14:15:51.498 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:51 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[89530]: ts=2024-09-13T14:15:51.183Z caller=tls_config.go:313 level=info component=web msg="Listening on" address=[::]:9095 2024-09-13T14:15:51.498 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:51 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[89530]: ts=2024-09-13T14:15:51.183Z caller=tls_config.go:316 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-13T14:15:51.498 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:51 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[89530]: ts=2024-09-13T14:15:51.184Z caller=head.go:616 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-13T14:15:51.498 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:51 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[89530]: ts=2024-09-13T14:15:51.184Z caller=head.go:698 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.544µs 2024-09-13T14:15:51.498 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:51 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[89530]: ts=2024-09-13T14:15:51.184Z caller=head.go:706 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-13T14:15:51.499 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:51 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[89530]: ts=2024-09-13T14:15:51.185Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=4 2024-09-13T14:15:51.499 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:51 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[89530]: ts=2024-09-13T14:15:51.192Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=4 2024-09-13T14:15:51.499 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:51 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[89530]: ts=2024-09-13T14:15:51.192Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=4 2024-09-13T14:15:51.499 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:51 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[89530]: ts=2024-09-13T14:15:51.193Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=4 2024-09-13T14:15:51.499 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:51 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[89530]: ts=2024-09-13T14:15:51.193Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=4 maxSegment=4 2024-09-13T14:15:51.499 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:51 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[89530]: ts=2024-09-13T14:15:51.193Z caller=head.go:815 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=26.551µs wal_replay_duration=9.40319ms wbl_replay_duration=113ns total_replay_duration=9.464107ms 2024-09-13T14:15:51.499 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:51 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[89530]: ts=2024-09-13T14:15:51.196Z caller=main.go:1150 level=info fs_type=XFS_SUPER_MAGIC 2024-09-13T14:15:51.499 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:51 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[89530]: ts=2024-09-13T14:15:51.196Z caller=main.go:1153 level=info msg="TSDB started" 2024-09-13T14:15:51.499 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:51 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[89530]: ts=2024-09-13T14:15:51.196Z caller=main.go:1335 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-13T14:15:51.499 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:51 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[89530]: ts=2024-09-13T14:15:51.212Z caller=main.go:1372 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=16.704674ms db_storage=840ns remote_storage=1.234µs web_handler=341ns query_engine=752ns scrape=423.614µs scrape_sd=102.337µs notify=28.096µs notify_sd=23.59µs rules=15.798745ms tracing=6.384µs 2024-09-13T14:15:51.499 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:51 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[89530]: ts=2024-09-13T14:15:51.212Z caller=main.go:1114 level=info msg="Server is ready to receive web requests." 2024-09-13T14:15:51.499 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:51 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[89530]: ts=2024-09-13T14:15:51.212Z caller=manager.go:163 level=info component="rule manager" msg="Starting rule manager..." 2024-09-13T14:15:51.499 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:51 smithi195 bash[89507]: d131e53f1ef9319ff3efc21e6bfc7a7e1bc630b1a34402d7507836f08a03326c 2024-09-13T14:15:51.500 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 14:15:51 smithi195 systemd[1]: Started Ceph prometheus.a for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:15:52.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:52 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:52.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:52 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:52.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:52 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:15:52.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:52 smithi195 ceph-mon[33205]: pgmap v35: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:52.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:52 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:52.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:52 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:52.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:52 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:15:52.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:52 smithi102 ceph-mon[29355]: pgmap v35: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:52.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:52 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:52.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:52 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:52.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:52 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:15:52.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:52 smithi102 ceph-mon[40543]: pgmap v35: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:55.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:55 smithi102 ceph-mon[29355]: pgmap v36: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:55.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:55 smithi102 ceph-mon[40543]: pgmap v36: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:55.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:55 smithi195 ceph-mon[33205]: pgmap v36: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:56.717 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:15:56 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:15:56] "GET /metrics HTTP/1.1" 200 33822 "" "Prometheus/2.51.0" 2024-09-13T14:15:56.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:56 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:56.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:56 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:56.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:56 smithi195 ceph-mon[33205]: pgmap v37: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:56.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:56 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:56.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:56 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:57.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:56 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:57.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:56 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:57.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:56 smithi102 ceph-mon[29355]: pgmap v37: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:57.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:56 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:57.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:56 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:57.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:56 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:57.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:56 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:57.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:56 smithi102 ceph-mon[40543]: pgmap v37: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:57.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:56 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:57.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:56 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:15:58.876 INFO:teuthology.orchestra.run.smithi102.stdout:true 2024-09-13T14:15:59.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:15:59 smithi102 ceph-mon[29355]: pgmap v38: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:59.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:15:59 smithi102 ceph-mon[40543]: pgmap v38: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:59.429 INFO:teuthology.orchestra.run.smithi102.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-13T14:15:59.429 INFO:teuthology.orchestra.run.smithi102.stdout:alertmanager.a smithi102 *:9093,9094 running (4m) 26s ago 5m 26.3M - 0.25.0 c8568f914cd2 fadcd26e5395 2024-09-13T14:15:59.429 INFO:teuthology.orchestra.run.smithi102.stdout:grafana.a smithi195 *:3000 running (5m) 3s ago 5m 85.5M - 9.4.7 954c08fa6188 4e2eb92e9f89 2024-09-13T14:15:59.429 INFO:teuthology.orchestra.run.smithi102.stdout:mds.foo.smithi102.njetxr smithi102 running (4m) 26s ago 4m 19.3M - 18.2.4-843-g2c01458a c129f5a2fd4e c9211cc725ed 2024-09-13T14:15:59.429 INFO:teuthology.orchestra.run.smithi102.stdout:mds.foo.smithi195.yqzrxi smithi195 running (4m) 3s ago 4m 17.1M - 18.2.4-843-g2c01458a c129f5a2fd4e eea37a7230fd 2024-09-13T14:15:59.429 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.x smithi195 *:8443,9283,8765 running (52s) 3s ago 9m 453M - 19.3.0-4697-gd58f7306 fec07016a2b0 4a23b8f0ea8b 2024-09-13T14:15:59.430 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.y smithi102 *:8443,9283,8765 running (79s) 26s ago 11m 506M - 19.3.0-4697-gd58f7306 fec07016a2b0 6b12b28dcde1 2024-09-13T14:15:59.430 INFO:teuthology.orchestra.run.smithi102.stdout:mon.a smithi102 running (11m) 26s ago 11m 54.7M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e 22c604c4fd02 2024-09-13T14:15:59.430 INFO:teuthology.orchestra.run.smithi102.stdout:mon.b smithi195 running (9m) 3s ago 9m 47.9M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e 608ec4847676 2024-09-13T14:15:59.430 INFO:teuthology.orchestra.run.smithi102.stdout:mon.c smithi102 running (9m) 26s ago 9m 45.7M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e 6c1fd3099a05 2024-09-13T14:15:59.430 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.a smithi102 *:9100 running (37s) 26s ago 5m 7868k - 1.7.0 72c9c2088986 0c92d0b39468 2024-09-13T14:15:59.430 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.b smithi195 *:9100 running (31s) 3s ago 5m 9395k - 1.7.0 72c9c2088986 fde867a7dbf9 2024-09-13T14:15:59.430 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0 smithi102 running (9m) 26s ago 9m 59.5M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e b97482fb8cf9 2024-09-13T14:15:59.430 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1 smithi102 running (8m) 26s ago 8m 51.0M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 8a67356be7a2 2024-09-13T14:15:59.430 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2 smithi102 running (8m) 26s ago 8m 48.9M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 1800986ac838 2024-09-13T14:15:59.430 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3 smithi102 running (8m) 26s ago 8m 55.9M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 968246da9da2 2024-09-13T14:15:59.430 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4 smithi195 running (7m) 3s ago 7m 47.8M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 515307701d73 2024-09-13T14:15:59.430 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5 smithi195 running (7m) 3s ago 7m 52.2M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 176ba2b3bd59 2024-09-13T14:15:59.430 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6 smithi195 running (6m) 3s ago 6m 54.7M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 9cd070a6dd3d 2024-09-13T14:15:59.431 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7 smithi195 running (6m) 3s ago 6m 51.9M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e a6b698c659ca 2024-09-13T14:15:59.431 INFO:teuthology.orchestra.run.smithi102.stdout:prometheus.a smithi195 *:9095 running (8s) 3s ago 5m 33.0M - 2.51.0 1d3b7f56885b d131e53f1ef9 2024-09-13T14:15:59.449 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:15:59 smithi195 ceph-mon[33205]: pgmap v38: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:15:59.802 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:15:59.803 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": { 2024-09-13T14:15:59.803 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 3 2024-09-13T14:15:59.803 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:15:59.803 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": { 2024-09-13T14:15:59.803 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 2 2024-09-13T14:15:59.803 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:15:59.803 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": { 2024-09-13T14:15:59.803 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 8 2024-09-13T14:15:59.803 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:15:59.803 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": { 2024-09-13T14:15:59.803 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 2 2024-09-13T14:15:59.803 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:15:59.803 INFO:teuthology.orchestra.run.smithi102.stdout: "overall": { 2024-09-13T14:15:59.803 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 13, 2024-09-13T14:15:59.804 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 2 2024-09-13T14:15:59.804 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:15:59.804 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:16:00.101 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:16:00.102 INFO:teuthology.orchestra.run.smithi102.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f", 2024-09-13T14:16:00.102 INFO:teuthology.orchestra.run.smithi102.stdout: "in_progress": true, 2024-09-13T14:16:00.102 INFO:teuthology.orchestra.run.smithi102.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-09-13T14:16:00.102 INFO:teuthology.orchestra.run.smithi102.stdout: "services_complete": [ 2024-09-13T14:16:00.102 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr" 2024-09-13T14:16:00.102 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-09-13T14:16:00.102 INFO:teuthology.orchestra.run.smithi102.stdout: "progress": "2/2 daemons upgraded", 2024-09-13T14:16:00.102 INFO:teuthology.orchestra.run.smithi102.stdout: "message": "Currently upgrading alertmanager daemons", 2024-09-13T14:16:00.102 INFO:teuthology.orchestra.run.smithi102.stdout: "is_paused": false 2024-09-13T14:16:00.102 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:16:00.342 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:00 smithi102 ceph-mon[29355]: from='client.25607 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:16:00.342 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:00 smithi102 ceph-mon[29355]: from='client.15720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:16:00.342 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:00 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:00.342 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:00 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:00.342 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:00 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:00.342 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:00 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:16:00.342 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:00 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:00.343 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:00 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:16:00.343 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:00 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:00.343 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:00 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:00.343 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:00 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:00.343 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:00 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:00.343 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:00 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:00.343 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:00 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:00.343 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:00 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:00.343 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:00 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:00.343 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:00 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:00.343 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:00 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:00.343 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:00 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:00.343 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:00 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:00.343 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:00 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/1072625418' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:00.344 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:00 smithi102 ceph-mon[40543]: from='client.25607 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:16:00.344 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:00 smithi102 ceph-mon[40543]: from='client.15720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:16:00.344 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:00 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:00.344 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:00 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:00.344 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:00 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:00.344 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:00 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:16:00.344 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:00 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:00.344 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:00 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:16:00.344 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:00 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:00.344 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:00 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:00.344 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:00 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:00.344 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:00 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:00.345 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:00 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:00.345 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:00 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:00.345 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:00 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:00.345 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:00 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:00.345 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:00 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:00.345 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:00 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:00.345 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:00 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:00.345 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:00 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:00.345 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:00 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/1072625418' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:00.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:00 smithi195 ceph-mon[33205]: from='client.25607 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:16:00.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:00 smithi195 ceph-mon[33205]: from='client.15720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:16:00.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:00 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:00.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:00 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:00.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:00 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:00.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:00 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:16:00.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:00 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:00.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:00 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:16:00.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:00 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:00.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:00 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:00.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:00 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:00.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:00 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:00.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:00 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:00.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:00 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:00.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:00 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:00.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:00 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:00.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:00 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:00.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:00 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:00.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:00 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:00.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:00 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:00.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:00 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/1072625418' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:01.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:01 smithi102 ceph-mon[40543]: from='client.25619 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:16:01.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:01 smithi102 ceph-mon[40543]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:16:01.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:01 smithi102 ceph-mon[40543]: pgmap v39: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:01.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:01 smithi102 ceph-mon[40543]: from='client.15738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:16:01.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:01 smithi102 ceph-mon[29355]: from='client.25619 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:16:01.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:01 smithi102 ceph-mon[29355]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:16:01.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:01 smithi102 ceph-mon[29355]: pgmap v39: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:01.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:01 smithi102 ceph-mon[29355]: from='client.15738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:16:01.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:01 smithi195 ceph-mon[33205]: from='client.25619 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:16:01.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:01 smithi195 ceph-mon[33205]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:16:01.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:01 smithi195 ceph-mon[33205]: pgmap v39: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:01.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:01 smithi195 ceph-mon[33205]: from='client.15738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:16:02.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:02 smithi102 ceph-mon[29355]: Upgrade: Updating alertmanager.a 2024-09-13T14:16:02.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:02 smithi102 ceph-mon[29355]: Deploying daemon alertmanager.a on smithi102 2024-09-13T14:16:02.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:02 smithi102 ceph-mon[40543]: Upgrade: Updating alertmanager.a 2024-09-13T14:16:02.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:02 smithi102 ceph-mon[40543]: Deploying daemon alertmanager.a on smithi102 2024-09-13T14:16:02.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:02 smithi195 ceph-mon[33205]: Upgrade: Updating alertmanager.a 2024-09-13T14:16:02.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:02 smithi195 ceph-mon[33205]: Deploying daemon alertmanager.a on smithi102 2024-09-13T14:16:03.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:03 smithi102 ceph-mon[29355]: pgmap v40: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:03.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:03 smithi102 ceph-mon[40543]: pgmap v40: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:03.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:03 smithi195 ceph-mon[33205]: pgmap v40: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:04.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:04 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:16:04.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:04 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:16:04.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:04 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:16:05.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:05 smithi102 ceph-mon[29355]: pgmap v41: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:05.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:05 smithi102 ceph-mon[40543]: pgmap v41: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:05.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:05 smithi195 ceph-mon[33205]: pgmap v41: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:06.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:16:06 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:16:06] "GET /metrics HTTP/1.1" 200 33821 "" "Prometheus/2.51.0" 2024-09-13T14:16:07.335 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:07 smithi102 ceph-mon[29355]: pgmap v42: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:07.335 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:07 smithi102 ceph-mon[40543]: pgmap v42: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:07.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:07 smithi195 ceph-mon[33205]: pgmap v42: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:09.073 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:16:08 smithi102 systemd[1]: Stopping Ceph alertmanager.a for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T14:16:09.073 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:16:09 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a[76563]: ts=2024-09-13T14:16:09.040Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-13T14:16:09.324 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:16:09 smithi102 podman[104428]: 2024-09-13 14:16:09.072387553 +0000 UTC m=+0.121168978 container died fadcd26e53958ed2f8777d8578736b52125048671ff18d7d2eef9704163acd19 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:16:09.325 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:09 smithi102 ceph-mon[40543]: pgmap v43: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:09.325 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:09 smithi102 ceph-mon[29355]: pgmap v43: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:09.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:09 smithi195 ceph-mon[33205]: pgmap v43: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:09.668 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:16:09 smithi102 podman[104428]: 2024-09-13 14:16:09.514510962 +0000 UTC m=+0.563292381 container cleanup fadcd26e53958ed2f8777d8578736b52125048671ff18d7d2eef9704163acd19 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:16:09.668 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:16:09 smithi102 bash[104428]: ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a 2024-09-13T14:16:10.070 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:16:09 smithi102 podman[104447]: 2024-09-13 14:16:09.724679059 +0000 UTC m=+0.645010797 container remove fadcd26e53958ed2f8777d8578736b52125048671ff18d7d2eef9704163acd19 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:16:10.070 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:16:09 smithi102 podman[104447]: 2024-09-13 14:16:09.800205464 +0000 UTC m=+0.720537200 volume remove 1ba02c42abefdda13346ea52261d27ba9fd5d635197a130cb9f519e2168d7e39 2024-09-13T14:16:10.418 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:16:10 smithi102 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-13T14:16:10.418 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:16:10 smithi102 systemd[1]: Stopped Ceph alertmanager.a for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:16:10.418 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:16:10 smithi102 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@alertmanager.a.service: Consumed 1.314s CPU time. 2024-09-13T14:16:10.418 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:16:10 smithi102 systemd[1]: Starting Ceph alertmanager.a for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T14:16:10.820 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:16:10 smithi102 podman[104581]: 2024-09-13 14:16:10.398278659 +0000 UTC m=+0.015920585 image pull 11f11916f8cdfb17c1da74b20deb2075d778357b4d60474d82453303f8d461f5 quay.io/prometheus/alertmanager:v0.27.0 2024-09-13T14:16:10.820 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:16:10 smithi102 podman[104581]: 2024-09-13 14:16:10.518829402 +0000 UTC m=+0.136471370 volume create 28d455061ff69238c1c4f13fd6ba611ba16997265aa3b0324407e86cdafb6198 2024-09-13T14:16:10.820 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:16:10 smithi102 podman[104581]: 2024-09-13 14:16:10.610913253 +0000 UTC m=+0.228555179 container create 0322b99ec767f5517953f4f2878ec0867ac99f60ce85590fb48052a82d694c02 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:16:11.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:11 smithi102 ceph-mon[29355]: pgmap v44: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:11.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:11 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:11.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:11 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:11.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:11 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:11.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:11 smithi102 ceph-mon[40543]: pgmap v44: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:11.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:11 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:11.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:11 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:11.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:11 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:11.419 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:16:10 smithi102 podman[104581]: 2024-09-13 14:16:10.921847618 +0000 UTC m=+0.539489549 container init 0322b99ec767f5517953f4f2878ec0867ac99f60ce85590fb48052a82d694c02 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:16:11.419 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:16:10 smithi102 podman[104581]: 2024-09-13 14:16:10.926904166 +0000 UTC m=+0.544546093 container start 0322b99ec767f5517953f4f2878ec0867ac99f60ce85590fb48052a82d694c02 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:16:11.419 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:16:10 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a[104597]: ts=2024-09-13T14:16:10.937Z caller=main.go:181 level=info msg="Starting Alertmanager" version="(version=0.27.0, branch=HEAD, revision=0aa3c2aad14cff039931923ab16b26b7481783b5)" 2024-09-13T14:16:11.419 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:16:10 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a[104597]: ts=2024-09-13T14:16:10.937Z caller=main.go:182 level=info build_context="(go=go1.21.7, platform=linux/amd64, user=root@22cd11f671e9, date=20240228-11:51:20, tags=netgo)" 2024-09-13T14:16:11.420 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:16:10 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a[104597]: ts=2024-09-13T14:16:10.938Z caller=cluster.go:186 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.102 port=9094 2024-09-13T14:16:11.420 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:16:10 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a[104597]: ts=2024-09-13T14:16:10.939Z caller=cluster.go:683 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-13T14:16:11.420 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:16:10 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a[104597]: ts=2024-09-13T14:16:10.960Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-13T14:16:11.420 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:16:10 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a[104597]: ts=2024-09-13T14:16:10.960Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-13T14:16:11.420 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:16:10 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a[104597]: ts=2024-09-13T14:16:10.962Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9093 2024-09-13T14:16:11.420 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:16:10 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a[104597]: ts=2024-09-13T14:16:10.962Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-13T14:16:11.420 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:16:11 smithi102 bash[104581]: 0322b99ec767f5517953f4f2878ec0867ac99f60ce85590fb48052a82d694c02 2024-09-13T14:16:11.420 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:16:11 smithi102 systemd[1]: Started Ceph alertmanager.a for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:16:11.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:11 smithi195 ceph-mon[33205]: pgmap v44: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:11.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:11 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:11.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:11 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:11.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:11 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:12.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:12 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:12.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:12 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:12.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:12 smithi102 ceph-mon[29355]: pgmap v45: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:12.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:12 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:12.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:12 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:12.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:12 smithi102 ceph-mon[40543]: pgmap v45: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:12.740 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:12 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:12.740 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:12 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:12.741 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:12 smithi195 ceph-mon[33205]: pgmap v45: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:13.418 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:16:12 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a[104597]: ts=2024-09-13T14:16:12.940Z caller=cluster.go:708 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000717308s 2024-09-13T14:16:15.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:15 smithi102 ceph-mon[29355]: pgmap v46: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:15.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:15 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:15.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:15 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:15.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:15 smithi102 ceph-mon[40543]: pgmap v46: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:15.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:15 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:15.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:15 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:15.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:15 smithi195 ceph-mon[33205]: pgmap v46: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:15.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:15 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:15.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:15 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:16.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:16:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:16:16] "GET /metrics HTTP/1.1" 200 33821 "" "Prometheus/2.51.0" 2024-09-13T14:16:16.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:16 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:16.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:16 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:16.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:16 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:16.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:16 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:16.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:16 smithi102 ceph-mon[40543]: pgmap v47: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:16.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:16 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:16.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:16 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:16.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:16 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:16.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:16 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:16.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:16 smithi102 ceph-mon[29355]: pgmap v47: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:16.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:16 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:16.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:16 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:16.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:16 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:16.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:16 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:16.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:16 smithi195 ceph-mon[33205]: pgmap v47: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:19.362 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:19 smithi195 ceph-mon[33205]: pgmap v48: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:19.363 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:19 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:16:19.363 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:19 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:19.363 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:19 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:19.363 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:19 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:19.363 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:19 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:16:19.363 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:19 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:19.363 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:19 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T14:16:19.363 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:19 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:19.363 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:19 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:19.363 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:19 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:19.363 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:19 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:19.363 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:19 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:19.363 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:19 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:19.364 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:19 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:19.364 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:19 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:19.364 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:19 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:19.364 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:19 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:19.364 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:19 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:19.364 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:19 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:19.364 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:19 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:19.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:19 smithi102 ceph-mon[29355]: pgmap v48: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:19.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:19 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:16:19.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:19 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:19.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:19 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:19.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:19 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:19.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:19 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:16:19.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:19 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:19.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:19 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T14:16:19.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:19 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:19.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:19 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:19.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:19 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:19.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:19 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:19.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:19 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:19.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:19 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:19.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:19 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:19.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:19 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:19.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:19 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:19.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:19 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:19.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:19 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:19.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:19 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:19.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:19 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:19.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:19 smithi102 ceph-mon[40543]: pgmap v48: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:19.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:19 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:16:19.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:19 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:19.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:19 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:19.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:19 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:19.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:19 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:16:19.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:19 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:19.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:19 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T14:16:19.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:19 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:19.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:19 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:19.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:19 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:19.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:19 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:19.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:19 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:19.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:19 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:19.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:19 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:19.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:19 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:19.422 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:19 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:19.422 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:19 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:19.422 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:19 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:19.422 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:19 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:19.422 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:19 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:20.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:20 smithi195 ceph-mon[33205]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T14:16:20.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:20 smithi102 ceph-mon[40543]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T14:16:20.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:20 smithi102 ceph-mon[29355]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T14:16:21.157 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 14:16:20 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a[104597]: ts=2024-09-13T14:16:20.943Z caller=cluster.go:700 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003612357s 2024-09-13T14:16:21.350 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:21 smithi195 ceph-mon[33205]: Upgrade: Updating grafana.a 2024-09-13T14:16:21.350 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:21 smithi195 ceph-mon[33205]: Deploying daemon grafana.a on smithi195 2024-09-13T14:16:21.350 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:21 smithi195 ceph-mon[33205]: pgmap v49: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:16:21.350 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:21 smithi195 ceph-mon[33205]: mgrmap e34: y(active, since 92s), standbys: x 2024-09-13T14:16:21.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:21 smithi102 ceph-mon[40543]: Upgrade: Updating grafana.a 2024-09-13T14:16:21.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:21 smithi102 ceph-mon[40543]: Deploying daemon grafana.a on smithi195 2024-09-13T14:16:21.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:21 smithi102 ceph-mon[40543]: pgmap v49: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:16:21.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:21 smithi102 ceph-mon[40543]: mgrmap e34: y(active, since 92s), standbys: x 2024-09-13T14:16:21.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:21 smithi102 ceph-mon[29355]: Upgrade: Updating grafana.a 2024-09-13T14:16:21.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:21 smithi102 ceph-mon[29355]: Deploying daemon grafana.a on smithi195 2024-09-13T14:16:21.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:21 smithi102 ceph-mon[29355]: pgmap v49: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:16:21.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:21 smithi102 ceph-mon[29355]: mgrmap e34: y(active, since 92s), standbys: x 2024-09-13T14:16:23.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:23 smithi102 ceph-mon[29355]: pgmap v50: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:16:23.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:23 smithi102 ceph-mon[40543]: pgmap v50: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:16:23.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:23 smithi195 ceph-mon[33205]: pgmap v50: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:16:25.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:25 smithi102 ceph-mon[29355]: pgmap v51: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:16:25.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:25 smithi102 ceph-mon[40543]: pgmap v51: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:16:25.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:25 smithi195 ceph-mon[33205]: pgmap v51: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:16:26.918 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:16:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:16:26] "GET /metrics HTTP/1.1" 200 33818 "" "Prometheus/2.51.0" 2024-09-13T14:16:27.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:27 smithi102 ceph-mon[29355]: pgmap v52: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:16:27.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:27 smithi102 ceph-mon[40543]: pgmap v52: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:16:27.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:27 smithi195 ceph-mon[33205]: pgmap v52: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:16:29.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:29 smithi195 ceph-mon[33205]: pgmap v53: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:16:29.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:29 smithi102 ceph-mon[29355]: pgmap v53: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:16:29.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:29 smithi102 ceph-mon[40543]: pgmap v53: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:16:30.385 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:30 smithi195 systemd[1]: Stopping Ceph grafana.a for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T14:16:30.404 INFO:teuthology.orchestra.run.smithi102.stdout:true 2024-09-13T14:16:30.730 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:30 smithi195 podman[94200]: 2024-09-13 14:16:30.384613265 +0000 UTC m=+0.119053725 container died 4e2eb92e9f89a83fd3a03922dbb9edd4178a5e653a183111cb0aafacd52d7ad5 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, build-date=2022-04-29T11:48:11.055897, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, release=239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, distribution-scope=public, io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, maintainer=Paul Cuzner , version=8.5, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.32.0, name=ubi8, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, description=Ceph Grafana Container, io.openshift.expose-services=, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-09-13T14:16:30.952 INFO:teuthology.orchestra.run.smithi102.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-13T14:16:30.952 INFO:teuthology.orchestra.run.smithi102.stdout:alertmanager.a smithi102 *:9093,9094 running (20s) 16s ago 6m 16.4M - 0.27.0 11f11916f8cd 0322b99ec767 2024-09-13T14:16:30.952 INFO:teuthology.orchestra.run.smithi102.stdout:grafana.a smithi195 *:3000 running (5m) 15s ago 5m 85.6M - 9.4.7 954c08fa6188 4e2eb92e9f89 2024-09-13T14:16:30.953 INFO:teuthology.orchestra.run.smithi102.stdout:mds.foo.smithi102.njetxr smithi102 running (4m) 16s ago 4m 19.6M - 18.2.4-843-g2c01458a c129f5a2fd4e c9211cc725ed 2024-09-13T14:16:30.953 INFO:teuthology.orchestra.run.smithi102.stdout:mds.foo.smithi195.yqzrxi smithi195 running (4m) 15s ago 4m 17.2M - 18.2.4-843-g2c01458a c129f5a2fd4e eea37a7230fd 2024-09-13T14:16:30.953 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.x smithi195 *:8443,9283,8765 running (83s) 15s ago 9m 453M - 19.3.0-4697-gd58f7306 fec07016a2b0 4a23b8f0ea8b 2024-09-13T14:16:30.953 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.y smithi102 *:8443,9283,8765 running (111s) 16s ago 11m 509M - 19.3.0-4697-gd58f7306 fec07016a2b0 6b12b28dcde1 2024-09-13T14:16:30.953 INFO:teuthology.orchestra.run.smithi102.stdout:mon.a smithi102 running (11m) 16s ago 11m 53.6M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e 22c604c4fd02 2024-09-13T14:16:30.953 INFO:teuthology.orchestra.run.smithi102.stdout:mon.b smithi195 running (10m) 15s ago 10m 44.9M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e 608ec4847676 2024-09-13T14:16:30.953 INFO:teuthology.orchestra.run.smithi102.stdout:mon.c smithi102 running (10m) 16s ago 10m 43.2M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e 6c1fd3099a05 2024-09-13T14:16:30.953 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.a smithi102 *:9100 running (68s) 16s ago 5m 9592k - 1.7.0 72c9c2088986 0c92d0b39468 2024-09-13T14:16:30.954 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.b smithi195 *:9100 running (62s) 15s ago 5m 9411k - 1.7.0 72c9c2088986 fde867a7dbf9 2024-09-13T14:16:30.954 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0 smithi102 running (9m) 16s ago 9m 59.7M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e b97482fb8cf9 2024-09-13T14:16:30.954 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1 smithi102 running (9m) 16s ago 9m 51.3M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 8a67356be7a2 2024-09-13T14:16:30.954 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2 smithi102 running (8m) 16s ago 8m 49.2M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 1800986ac838 2024-09-13T14:16:30.954 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3 smithi102 running (8m) 16s ago 8m 56.1M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 968246da9da2 2024-09-13T14:16:30.954 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4 smithi195 running (8m) 15s ago 8m 47.9M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 515307701d73 2024-09-13T14:16:30.954 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5 smithi195 running (7m) 15s ago 7m 52.2M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 176ba2b3bd59 2024-09-13T14:16:30.954 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6 smithi195 running (7m) 15s ago 7m 54.8M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 9cd070a6dd3d 2024-09-13T14:16:30.954 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7 smithi195 running (7m) 15s ago 7m 52.0M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e a6b698c659ca 2024-09-13T14:16:30.954 INFO:teuthology.orchestra.run.smithi102.stdout:prometheus.a smithi195 *:9095 running (40s) 15s ago 6m 37.5M - 2.51.0 1d3b7f56885b d131e53f1ef9 2024-09-13T14:16:30.997 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:30 smithi195 podman[94200]: 2024-09-13 14:16:30.730062857 +0000 UTC m=+0.464503294 container cleanup 4e2eb92e9f89a83fd3a03922dbb9edd4178a5e653a183111cb0aafacd52d7ad5 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, architecture=x86_64, com.redhat.component=ubi8-container, name=ubi8, version=8.5, description=Ceph Grafana Container, vcs-type=git, vendor=Red Hat, Inc., release=239.1651231664, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-04-29T11:48:11.055897, io.buildah.version=1.32.0, io.openshift.expose-services=, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration) 2024-09-13T14:16:30.997 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:30 smithi195 bash[94200]: ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a 2024-09-13T14:16:31.287 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:16:31.287 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": { 2024-09-13T14:16:31.287 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 3 2024-09-13T14:16:31.287 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:16:31.287 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": { 2024-09-13T14:16:31.288 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 2 2024-09-13T14:16:31.288 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:16:31.288 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": { 2024-09-13T14:16:31.288 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 8 2024-09-13T14:16:31.288 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:16:31.288 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": { 2024-09-13T14:16:31.288 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 2 2024-09-13T14:16:31.288 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:16:31.288 INFO:teuthology.orchestra.run.smithi102.stdout: "overall": { 2024-09-13T14:16:31.288 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 13, 2024-09-13T14:16:31.288 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 2 2024-09-13T14:16:31.288 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:16:31.289 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:16:31.289 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:31 smithi102 ceph-mon[29355]: pgmap v54: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:16:31.289 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:31 smithi102 ceph-mon[40543]: pgmap v54: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:16:31.359 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-mon[33205]: pgmap v54: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:16:31.360 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:30 smithi195 podman[94218]: 2024-09-13 14:16:30.998098025 +0000 UTC m=+0.612122158 container remove 4e2eb92e9f89a83fd3a03922dbb9edd4178a5e653a183111cb0aafacd52d7ad5 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , vendor=Red Hat, Inc., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, vcs-type=git, build-date=2022-04-29T11:48:11.055897, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, architecture=x86_64, io.buildah.version=1.32.0, io.openshift.tags=base rhel8, version=8.5, release=239.1651231664, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, distribution-scope=public, description=Ceph Grafana Container) 2024-09-13T14:16:31.360 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-09-13T14:16:31.360 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-09-13T14:16:31.360 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 systemd[1]: Stopped Ceph grafana.a for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:16:31.361 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@grafana.a.service: Consumed 7.534s CPU time. 2024-09-13T14:16:31.361 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 systemd[1]: Starting Ceph grafana.a for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T14:16:31.572 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:16:31.572 INFO:teuthology.orchestra.run.smithi102.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f", 2024-09-13T14:16:31.572 INFO:teuthology.orchestra.run.smithi102.stdout: "in_progress": true, 2024-09-13T14:16:31.572 INFO:teuthology.orchestra.run.smithi102.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-09-13T14:16:31.572 INFO:teuthology.orchestra.run.smithi102.stdout: "services_complete": [ 2024-09-13T14:16:31.572 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr" 2024-09-13T14:16:31.572 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-09-13T14:16:31.572 INFO:teuthology.orchestra.run.smithi102.stdout: "progress": "2/2 daemons upgraded", 2024-09-13T14:16:31.573 INFO:teuthology.orchestra.run.smithi102.stdout: "message": "Currently upgrading grafana daemons", 2024-09-13T14:16:31.573 INFO:teuthology.orchestra.run.smithi102.stdout: "is_paused": false 2024-09-13T14:16:31.573 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:16:31.706 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 podman[94351]: 2024-09-13 14:16:31.471777789 +0000 UTC m=+0.017201385 image pull c8b91775d855b99270fc5d22f3c6737e8cca01ef4c25c8b0362295e0746fa39b quay.io/ceph/grafana:10.4.0 2024-09-13T14:16:31.706 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 podman[94351]: 2024-09-13 14:16:31.582607177 +0000 UTC m=+0.128030774 container create 152e89bad2c83187ec1732fb75be2866fe0baed6be2b2e6353d87e2c4d70a1f2 (image=quay.io/ceph/grafana:10.4.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-09-13T14:16:32.053 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 podman[94351]: 2024-09-13 14:16:31.801565829 +0000 UTC m=+0.346989419 container init 152e89bad2c83187ec1732fb75be2866fe0baed6be2b2e6353d87e2c4d70a1f2 (image=quay.io/ceph/grafana:10.4.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-09-13T14:16:32.053 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 podman[94351]: 2024-09-13 14:16:31.806462972 +0000 UTC m=+0.351886562 container start 152e89bad2c83187ec1732fb75be2866fe0baed6be2b2e6353d87e2c4d70a1f2 (image=quay.io/ceph/grafana:10.4.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-09-13T14:16:32.053 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 bash[94351]: 152e89bad2c83187ec1732fb75be2866fe0baed6be2b2e6353d87e2c4d70a1f2 2024-09-13T14:16:32.053 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 systemd[1]: Started Ceph grafana.a for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:16:32.053 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=settings t=2024-09-13T14:16:31.928669088Z level=info msg="Starting Grafana" version=10.4.0 commit=03f502a94d17f7dc4e6c34acdf8428aedd986e4c branch=HEAD compiled=2024-09-13T14:16:31Z 2024-09-13T14:16:32.053 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=settings t=2024-09-13T14:16:31.928904532Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-09-13T14:16:32.053 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=settings t=2024-09-13T14:16:31.928912522Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-09-13T14:16:32.053 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=settings t=2024-09-13T14:16:31.928921536Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana" 2024-09-13T14:16:32.053 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=settings t=2024-09-13T14:16:31.928924501Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana" 2024-09-13T14:16:32.053 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=settings t=2024-09-13T14:16:31.928926952Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins" 2024-09-13T14:16:32.054 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=settings t=2024-09-13T14:16:31.928929793Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning" 2024-09-13T14:16:32.054 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=settings t=2024-09-13T14:16:31.928932271Z level=info msg="Config overridden from command line" arg="default.log.mode=console" 2024-09-13T14:16:32.054 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=settings t=2024-09-13T14:16:31.928935005Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-13T14:16:32.054 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=settings t=2024-09-13T14:16:31.928937673Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-13T14:16:32.054 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=settings t=2024-09-13T14:16:31.928940125Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-13T14:16:32.054 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=settings t=2024-09-13T14:16:31.928942765Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-13T14:16:32.054 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=settings t=2024-09-13T14:16:31.928945344Z level=info msg=Target target=[all] 2024-09-13T14:16:32.054 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=settings t=2024-09-13T14:16:31.928951765Z level=info msg="Path Home" path=/usr/share/grafana 2024-09-13T14:16:32.054 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=settings t=2024-09-13T14:16:31.928954293Z level=info msg="Path Data" path=/var/lib/grafana 2024-09-13T14:16:32.054 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=settings t=2024-09-13T14:16:31.928956671Z level=info msg="Path Logs" path=/var/log/grafana 2024-09-13T14:16:32.054 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=settings t=2024-09-13T14:16:31.928958904Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-09-13T14:16:32.054 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=settings t=2024-09-13T14:16:31.928961243Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-09-13T14:16:32.055 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=settings t=2024-09-13T14:16:31.928963685Z level=info msg="App mode production" 2024-09-13T14:16:32.055 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=sqlstore t=2024-09-13T14:16:31.929193289Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-09-13T14:16:32.055 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=sqlstore t=2024-09-13T14:16:31.929206821Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-09-13T14:16:32.055 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:31.929635872Z level=info msg="Starting DB migrations" 2024-09-13T14:16:32.055 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:31.943313869Z level=info msg="Executing migration" id="Add uid column to user" 2024-09-13T14:16:32.055 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:31.948970314Z level=info msg="Migration successfully executed" id="Add uid column to user" duration=5.652305ms 2024-09-13T14:16:32.055 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:31.949513518Z level=info msg="Executing migration" id="Update uid column values for users" 2024-09-13T14:16:32.055 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:31.94969081Z level=info msg="Migration successfully executed" id="Update uid column values for users" duration=177.967µs 2024-09-13T14:16:32.055 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:31.950089828Z level=info msg="Executing migration" id="Add unique index user_uid" 2024-09-13T14:16:32.055 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:31.950872783Z level=info msg="Migration successfully executed" id="Add unique index user_uid" duration=782.631µs 2024-09-13T14:16:32.055 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:31.951389091Z level=info msg="Executing migration" id="Add playlist column created_at" 2024-09-13T14:16:32.055 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:31.957693148Z level=info msg="Migration successfully executed" id="Add playlist column created_at" duration=6.301966ms 2024-09-13T14:16:32.055 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:31.958305923Z level=info msg="Executing migration" id="Add playlist column updated_at" 2024-09-13T14:16:32.055 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:31.96495745Z level=info msg="Migration successfully executed" id="Add playlist column updated_at" duration=6.648188ms 2024-09-13T14:16:32.055 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:31.965424212Z level=info msg="Executing migration" id="Add column uid in team" 2024-09-13T14:16:32.056 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:31.970543956Z level=info msg="Migration successfully executed" id="Add column uid in team" duration=5.118557ms 2024-09-13T14:16:32.056 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:31.97087431Z level=info msg="Executing migration" id="Update uid column values in team" 2024-09-13T14:16:32.057 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:31.971005298Z level=info msg="Migration successfully executed" id="Update uid column values in team" duration=132.336µs 2024-09-13T14:16:32.057 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:31.971343465Z level=info msg="Executing migration" id="Add unique index team_org_id_uid" 2024-09-13T14:16:32.057 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:31.972099619Z level=info msg="Migration successfully executed" id="Add unique index team_org_id_uid" duration=755.548µs 2024-09-13T14:16:32.057 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:31.972479784Z level=info msg="Executing migration" id="add index user_auth_token.revoked_at" 2024-09-13T14:16:32.057 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:31.973179263Z level=info msg="Migration successfully executed" id="add index user_auth_token.revoked_at" duration=696.885µs 2024-09-13T14:16:32.057 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:31.973606755Z level=info msg="Executing migration" id="add result_fingerprint column to alert_instance" 2024-09-13T14:16:32.057 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:31.979718289Z level=info msg="Migration successfully executed" id="add result_fingerprint column to alert_instance" duration=6.110233ms 2024-09-13T14:16:32.058 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:31.980178122Z level=info msg="Executing migration" id="add last_applied column to alert_configuration_history" 2024-09-13T14:16:32.058 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:31.985645634Z level=info msg="Migration successfully executed" id="add last_applied column to alert_configuration_history" duration=5.46669ms 2024-09-13T14:16:32.058 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:31.985987876Z level=info msg="Executing migration" id="alter library_element model to mediumtext" 2024-09-13T14:16:32.058 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:31.986028077Z level=info msg="Migration successfully executed" id="alter library_element model to mediumtext" duration=40.07µs 2024-09-13T14:16:32.058 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:31.986532501Z level=info msg="Executing migration" id="permission kind migration" 2024-09-13T14:16:32.058 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:31.99155494Z level=info msg="Migration successfully executed" id="permission kind migration" duration=5.020882ms 2024-09-13T14:16:32.058 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:31.991900526Z level=info msg="Executing migration" id="permission attribute migration" 2024-09-13T14:16:32.058 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:31.996990449Z level=info msg="Migration successfully executed" id="permission attribute migration" duration=5.088953ms 2024-09-13T14:16:32.058 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:31.997327724Z level=info msg="Executing migration" id="permission identifier migration" 2024-09-13T14:16:32.058 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.002738323Z level=info msg="Migration successfully executed" id="permission identifier migration" duration=5.409673ms 2024-09-13T14:16:32.059 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.003079812Z level=info msg="Executing migration" id="add permission identifier index" 2024-09-13T14:16:32.059 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.003890433Z level=info msg="Migration successfully executed" id="add permission identifier index" duration=810.196µs 2024-09-13T14:16:32.059 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.004231208Z level=info msg="Executing migration" id="add permission action scope role_id index" 2024-09-13T14:16:32.059 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.005018164Z level=info msg="Migration successfully executed" id="add permission action scope role_id index" duration=786.417µs 2024-09-13T14:16:32.059 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.005327082Z level=info msg="Executing migration" id="remove permission role_id action scope index" 2024-09-13T14:16:32.059 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.006052029Z level=info msg="Migration successfully executed" id="remove permission role_id action scope index" duration=724.875µs 2024-09-13T14:16:32.059 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.006356838Z level=info msg="Executing migration" id="rbac disabled migrator" 2024-09-13T14:16:32.059 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.006379512Z level=info msg="Migration successfully executed" id="rbac disabled migrator" duration=22.704µs 2024-09-13T14:16:32.059 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.006721676Z level=info msg="Executing migration" id="drop index IDX_correlation_uid - v1" 2024-09-13T14:16:32.059 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.007455832Z level=info msg="Migration successfully executed" id="drop index IDX_correlation_uid - v1" duration=733.937µs 2024-09-13T14:16:32.059 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.007774563Z level=info msg="Executing migration" id="drop index IDX_correlation_source_uid - v1" 2024-09-13T14:16:32.059 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.008473011Z level=info msg="Migration successfully executed" id="drop index IDX_correlation_source_uid - v1" duration=698.723µs 2024-09-13T14:16:32.060 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.008775429Z level=info msg="Executing migration" id="Rename table correlation to correlation_tmp_qwerty - v1" 2024-09-13T14:16:32.060 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.024283411Z level=info msg="Migration successfully executed" id="Rename table correlation to correlation_tmp_qwerty - v1" duration=15.505112ms 2024-09-13T14:16:32.060 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.024710464Z level=info msg="Executing migration" id="create correlation v2" 2024-09-13T14:16:32.060 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.025445533Z level=info msg="Migration successfully executed" id="create correlation v2" duration=734.53µs 2024-09-13T14:16:32.060 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.02576218Z level=info msg="Executing migration" id="create index IDX_correlation_uid - v2" 2024-09-13T14:16:32.060 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.02647556Z level=info msg="Migration successfully executed" id="create index IDX_correlation_uid - v2" duration=713.075µs 2024-09-13T14:16:32.060 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.026777103Z level=info msg="Executing migration" id="create index IDX_correlation_source_uid - v2" 2024-09-13T14:16:32.060 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.027478751Z level=info msg="Migration successfully executed" id="create index IDX_correlation_source_uid - v2" duration=701.356µs 2024-09-13T14:16:32.060 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.027801836Z level=info msg="Executing migration" id="create index IDX_correlation_org_id - v2" 2024-09-13T14:16:32.060 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.028487826Z level=info msg="Migration successfully executed" id="create index IDX_correlation_org_id - v2" duration=684.945µs 2024-09-13T14:16:32.060 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.028796619Z level=info msg="Executing migration" id="copy correlation v1 to v2" 2024-09-13T14:16:32.060 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.028991682Z level=info msg="Migration successfully executed" id="copy correlation v1 to v2" duration=194.988µs 2024-09-13T14:16:32.061 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.029331914Z level=info msg="Executing migration" id="drop correlation_tmp_qwerty" 2024-09-13T14:16:32.061 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.029895807Z level=info msg="Migration successfully executed" id="drop correlation_tmp_qwerty" duration=563.489µs 2024-09-13T14:16:32.061 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.03020987Z level=info msg="Executing migration" id="add provisioning column" 2024-09-13T14:16:32.061 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.035426978Z level=info msg="Migration successfully executed" id="add provisioning column" duration=5.216754ms 2024-09-13T14:16:32.061 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.035785987Z level=info msg="Executing migration" id="backfill empty share column fields with default of public" 2024-09-13T14:16:32.061 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.035935187Z level=info msg="Migration successfully executed" id="backfill empty share column fields with default of public" duration=147.993µs 2024-09-13T14:16:32.061 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.03628646Z level=info msg="Executing migration" id="migrate contents column to mediumblob for MySQL" 2024-09-13T14:16:32.061 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.036324072Z level=info msg="Migration successfully executed" id="migrate contents column to mediumblob for MySQL" duration=39.078µs 2024-09-13T14:16:32.061 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.036698642Z level=info msg="Executing migration" id="add origin column to seed_assignment" 2024-09-13T14:16:32.061 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.042366762Z level=info msg="Migration successfully executed" id="add origin column to seed_assignment" duration=5.666657ms 2024-09-13T14:16:32.061 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.042725902Z level=info msg="Executing migration" id="add origin to plugin seed_assignment" 2024-09-13T14:16:32.061 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.042943666Z level=info msg="Migration successfully executed" id="add origin to plugin seed_assignment" duration=216.659µs 2024-09-13T14:16:32.062 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.043312171Z level=info msg="Executing migration" id="prevent seeding OnCall access" 2024-09-13T14:16:32.062 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.043421998Z level=info msg="Migration successfully executed" id="prevent seeding OnCall access" duration=109.814µs 2024-09-13T14:16:32.062 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.043740106Z level=info msg="Executing migration" id="managed folder permissions library panel actions migration" 2024-09-13T14:16:32.062 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.044682107Z level=info msg="Migration successfully executed" id="managed folder permissions library panel actions migration" duration=941.23µs 2024-09-13T14:16:32.062 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.045005787Z level=info msg="Executing migration" id="Remove unique index for folder.title and folder.parent_uid" 2024-09-13T14:16:32.062 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.045700785Z level=info msg="Migration successfully executed" id="Remove unique index for folder.title and folder.parent_uid" duration=694.376µs 2024-09-13T14:16:32.062 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.046021972Z level=info msg="Executing migration" id="Add unique index for title, parent_uid, and org_id" 2024-09-13T14:16:32.062 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.046783105Z level=info msg="Migration successfully executed" id="Add unique index for title, parent_uid, and org_id" duration=760.784µs 2024-09-13T14:16:32.062 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.047112384Z level=info msg="Executing migration" id="Sync dashboard and folder table" 2024-09-13T14:16:32.062 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.047488451Z level=info msg="Migration successfully executed" id="Sync dashboard and folder table" duration=375.83µs 2024-09-13T14:16:32.062 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.047787127Z level=info msg="Executing migration" id="Remove ghost folders from the folder table" 2024-09-13T14:16:32.062 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.047986585Z level=info msg="Migration successfully executed" id="Remove ghost folders from the folder table" duration=199.361µs 2024-09-13T14:16:32.062 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.04832809Z level=info msg="Executing migration" id="Remove unique index UQE_folder_uid_org_id" 2024-09-13T14:16:32.063 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.049011828Z level=info msg="Migration successfully executed" id="Remove unique index UQE_folder_uid_org_id" duration=683.696µs 2024-09-13T14:16:32.063 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.0493138Z level=info msg="Executing migration" id="Add unique index UQE_folder_org_id_uid" 2024-09-13T14:16:32.063 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.050022476Z level=info msg="Migration successfully executed" id="Add unique index UQE_folder_org_id_uid" duration=708.718µs 2024-09-13T14:16:32.063 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.050323494Z level=info msg="Executing migration" id="Remove unique index UQE_folder_title_parent_uid_org_id" 2024-09-13T14:16:32.063 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.051008407Z level=info msg="Migration successfully executed" id="Remove unique index UQE_folder_title_parent_uid_org_id" duration=684.758µs 2024-09-13T14:16:32.063 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.051313944Z level=info msg="Executing migration" id="Add unique index UQE_folder_org_id_parent_uid_title" 2024-09-13T14:16:32.063 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.052039787Z level=info msg="Migration successfully executed" id="Add unique index UQE_folder_org_id_parent_uid_title" duration=725.371µs 2024-09-13T14:16:32.063 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.052389911Z level=info msg="Executing migration" id="Remove index IDX_folder_parent_uid_org_id" 2024-09-13T14:16:32.063 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.053252669Z level=info msg="Migration successfully executed" id="Remove index IDX_folder_parent_uid_org_id" duration=862.039µs 2024-09-13T14:16:32.063 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.05357772Z level=info msg="Executing migration" id="create anon_device table" 2024-09-13T14:16:32.063 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.054195853Z level=info msg="Migration successfully executed" id="create anon_device table" duration=618.464µs 2024-09-13T14:16:32.063 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.054516458Z level=info msg="Executing migration" id="add unique index anon_device.device_id" 2024-09-13T14:16:32.386 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-mon[33205]: from='client.15744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:16:32.387 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-mon[33205]: from='client.15750 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:16:32.387 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-mon[33205]: from='client.15756 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:16:32.387 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/1964312405' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:32.387 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:32.387 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:32.387 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:32.387 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.055277154Z level=info msg="Migration successfully executed" id="add unique index anon_device.device_id" duration=760.277µs 2024-09-13T14:16:32.388 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.055576358Z level=info msg="Executing migration" id="add index anon_device.updated_at" 2024-09-13T14:16:32.388 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.056259176Z level=info msg="Migration successfully executed" id="add index anon_device.updated_at" duration=682.547µs 2024-09-13T14:16:32.388 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.056574354Z level=info msg="Executing migration" id="create signing_key table" 2024-09-13T14:16:32.388 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.0571477Z level=info msg="Migration successfully executed" id="create signing_key table" duration=572.991µs 2024-09-13T14:16:32.388 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.05745678Z level=info msg="Executing migration" id="add unique index signing_key.key_id" 2024-09-13T14:16:32.388 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.058142172Z level=info msg="Migration successfully executed" id="add unique index signing_key.key_id" duration=685.156µs 2024-09-13T14:16:32.388 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.058451666Z level=info msg="Executing migration" id="set legacy alert migration status in kvstore" 2024-09-13T14:16:32.388 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.059136381Z level=info msg="Migration successfully executed" id="set legacy alert migration status in kvstore" duration=683.112µs 2024-09-13T14:16:32.388 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.059439696Z level=info msg="Executing migration" id="migrate record of created folders during legacy migration to kvstore" 2024-09-13T14:16:32.388 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.059854808Z level=info msg="Migration successfully executed" id="migrate record of created folders during legacy migration to kvstore" duration=414.964µs 2024-09-13T14:16:32.389 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.060221757Z level=info msg="Executing migration" id="Add folder_uid for dashboard" 2024-09-13T14:16:32.389 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.065374909Z level=info msg="Migration successfully executed" id="Add folder_uid for dashboard" duration=5.15227ms 2024-09-13T14:16:32.389 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.065747837Z level=info msg="Executing migration" id="Populate dashboard folder_uid column" 2024-09-13T14:16:32.389 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.067446248Z level=info msg="Migration successfully executed" id="Populate dashboard folder_uid column" duration=1.699082ms 2024-09-13T14:16:32.389 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.068013585Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_folder_uid_title" 2024-09-13T14:16:32.389 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.068941398Z level=info msg="Migration successfully executed" id="Add unique index for dashboard_org_id_folder_uid_title" duration=926.277µs 2024-09-13T14:16:32.389 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.069264406Z level=info msg="Executing migration" id="Delete unique index for dashboard_org_id_folder_id_title" 2024-09-13T14:16:32.389 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.069978173Z level=info msg="Migration successfully executed" id="Delete unique index for dashboard_org_id_folder_id_title" duration=711.686µs 2024-09-13T14:16:32.389 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.070282675Z level=info msg="Executing migration" id="Delete unique index for dashboard_org_id_folder_uid_title" 2024-09-13T14:16:32.389 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.070994905Z level=info msg="Migration successfully executed" id="Delete unique index for dashboard_org_id_folder_uid_title" duration=712.702µs 2024-09-13T14:16:32.389 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.071303248Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_folder_uid_title_is_folder" 2024-09-13T14:16:32.389 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.072145434Z level=info msg="Migration successfully executed" id="Add unique index for dashboard_org_id_folder_uid_title_is_folder" duration=841.667µs 2024-09-13T14:16:32.390 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.072486143Z level=info msg="Executing migration" id="Restore index for dashboard_org_id_folder_id_title" 2024-09-13T14:16:32.390 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.073281491Z level=info msg="Migration successfully executed" id="Restore index for dashboard_org_id_folder_id_title" duration=795.458µs 2024-09-13T14:16:32.390 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.073610034Z level=info msg="Executing migration" id="create sso_setting table" 2024-09-13T14:16:32.390 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.074364708Z level=info msg="Migration successfully executed" id="create sso_setting table" duration=754.488µs 2024-09-13T14:16:32.390 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.074716553Z level=info msg="Executing migration" id="copy kvstore migration status to each org" 2024-09-13T14:16:32.390 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.075387321Z level=info msg="Migration successfully executed" id="copy kvstore migration status to each org" duration=670.761µs 2024-09-13T14:16:32.390 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.075681709Z level=info msg="Executing migration" id="add back entry for orgid=0 migrated status" 2024-09-13T14:16:32.390 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.076089606Z level=info msg="Migration successfully executed" id="add back entry for orgid=0 migrated status" duration=407.692µs 2024-09-13T14:16:32.390 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.076380326Z level=info msg="Executing migration" id="alter kv_store.value to longtext" 2024-09-13T14:16:32.390 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.07641847Z level=info msg="Migration successfully executed" id="alter kv_store.value to longtext" duration=38.475µs 2024-09-13T14:16:32.390 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.076781435Z level=info msg="Executing migration" id="add notification_settings column to alert_rule table" 2024-09-13T14:16:32.391 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.082124254Z level=info msg="Migration successfully executed" id="add notification_settings column to alert_rule table" duration=5.342163ms 2024-09-13T14:16:32.391 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.082450421Z level=info msg="Executing migration" id="add notification_settings column to alert_rule_version table" 2024-09-13T14:16:32.391 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.087778183Z level=info msg="Migration successfully executed" id="add notification_settings column to alert_rule_version table" duration=5.326676ms 2024-09-13T14:16:32.391 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.088137656Z level=info msg="Executing migration" id="removing scope from alert.instances:read action migration" 2024-09-13T14:16:32.391 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.088360671Z level=info msg="Migration successfully executed" id="removing scope from alert.instances:read action migration" duration=223.244µs 2024-09-13T14:16:32.392 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=migrator t=2024-09-13T14:16:32.088764815Z level=info msg="migrations completed" performed=65 skipped=482 duration=145.524798ms 2024-09-13T14:16:32.392 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=sqlstore t=2024-09-13T14:16:32.089453395Z level=info msg="Created default organization" 2024-09-13T14:16:32.392 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=secrets t=2024-09-13T14:16:32.091548731Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-09-13T14:16:32.392 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=plugin.store t=2024-09-13T14:16:32.106149339Z level=info msg="Loading plugins..." 2024-09-13T14:16:32.392 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=local.finder t=2024-09-13T14:16:32.148031359Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled 2024-09-13T14:16:32.392 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=plugin.store t=2024-09-13T14:16:32.148049494Z level=info msg="Plugins loaded" count=55 duration=41.900876ms 2024-09-13T14:16:32.392 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=query_data t=2024-09-13T14:16:32.150523563Z level=info msg="Query Service initialization" 2024-09-13T14:16:32.392 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=live.push_http t=2024-09-13T14:16:32.152945566Z level=info msg="Live Push Gateway initialization" 2024-09-13T14:16:32.392 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=ngalert.migration t=2024-09-13T14:16:32.155615468Z level=info msg=Starting 2024-09-13T14:16:32.393 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=ngalert.state.manager t=2024-09-13T14:16:32.161386729Z level=info msg="Running in alternative execution of Error/NoData mode" 2024-09-13T14:16:32.393 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=infra.usagestats.collector t=2024-09-13T14:16:32.162672118Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-09-13T14:16:32.393 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=provisioning.datasources t=2024-09-13T14:16:32.165069031Z level=info msg="deleted datasource based on configuration" name=Dashboard1 2024-09-13T14:16:32.393 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=provisioning.datasources t=2024-09-13T14:16:32.165370916Z level=info msg="inserting datasource from configuration" name=Dashboard1 uid=P43CA22E17D0F9596 2024-09-13T14:16:32.393 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=provisioning.alerting t=2024-09-13T14:16:32.196201571Z level=info msg="starting to provision alerting" 2024-09-13T14:16:32.393 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=provisioning.alerting t=2024-09-13T14:16:32.196219538Z level=info msg="finished to provision alerting" 2024-09-13T14:16:32.393 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=ngalert.state.manager t=2024-09-13T14:16:32.197003664Z level=info msg="Warming state cache for startup" 2024-09-13T14:16:32.393 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=ngalert.multiorg.alertmanager t=2024-09-13T14:16:32.197127774Z level=info msg="Starting MultiOrg Alertmanager" 2024-09-13T14:16:32.393 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=ngalert.state.manager t=2024-09-13T14:16:32.1974061Z level=info msg="State cache has been initialized" states=0 duration=401.258µs 2024-09-13T14:16:32.393 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=ngalert.scheduler t=2024-09-13T14:16:32.197472476Z level=info msg="Starting scheduler" tickInterval=10s maxAttempts=1 2024-09-13T14:16:32.393 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=ticker t=2024-09-13T14:16:32.197565187Z level=info msg=starting first_tick=2024-09-13T14:16:40Z 2024-09-13T14:16:32.393 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=grafanaStorageLogger t=2024-09-13T14:16:32.197598404Z level=info msg="Storage starting" 2024-09-13T14:16:32.394 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=http.server t=2024-09-13T14:16:32.199131839Z level=info msg="HTTP Server TLS settings" MinTLSVersion=TLS1.2 configuredciphers=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA 2024-09-13T14:16:32.394 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=http.server t=2024-09-13T14:16:32.199502651Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-09-13T14:16:32.394 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=provisioning.dashboard t=2024-09-13T14:16:32.216094752Z level=info msg="starting to provision dashboards" 2024-09-13T14:16:32.394 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=grafana.update.checker t=2024-09-13T14:16:32.275250618Z level=info msg="Update check succeeded" duration=78.211535ms 2024-09-13T14:16:32.394 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=plugins.update.checker t=2024-09-13T14:16:32.27744262Z level=info msg="Update check succeeded" duration=80.844993ms 2024-09-13T14:16:32.394 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=provisioning.dashboard t=2024-09-13T14:16:32.386410168Z level=info msg="finished to provision dashboards" 2024-09-13T14:16:32.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:32 smithi102 ceph-mon[29355]: from='client.15744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:16:32.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:32 smithi102 ceph-mon[29355]: from='client.15750 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:16:32.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:32 smithi102 ceph-mon[29355]: from='client.15756 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:16:32.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:32 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/1964312405' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:32.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:32 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:32.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:32 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:32.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:32 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:32.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:32 smithi102 ceph-mon[40543]: from='client.15744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:16:32.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:32 smithi102 ceph-mon[40543]: from='client.15750 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:16:32.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:32 smithi102 ceph-mon[40543]: from='client.15756 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:16:32.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:32 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/1964312405' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:32.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:32 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:32.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:32 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:32.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:32 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:32.997 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=grafana-apiserver t=2024-09-13T14:16:32.820158623Z level=info msg="Adding GroupVersion playlist.grafana.app v0alpha1 to ResourceManager" 2024-09-13T14:16:32.997 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:16:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=grafana-apiserver t=2024-09-13T14:16:32.820899952Z level=info msg="Adding GroupVersion featuretoggle.grafana.app v0alpha1 to ResourceManager" 2024-09-13T14:16:33.425 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:33 smithi195 ceph-mon[33205]: from='client.25661 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:16:33.425 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:33 smithi195 ceph-mon[33205]: pgmap v55: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:33.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:33 smithi102 ceph-mon[29355]: from='client.25661 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:16:33.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:33 smithi102 ceph-mon[29355]: pgmap v55: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:33.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:33 smithi102 ceph-mon[40543]: from='client.25661 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:16:33.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:33 smithi102 ceph-mon[40543]: pgmap v55: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:34.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:34 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:16:34.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:34 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:16:34.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:34 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:16:35.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:35 smithi195 ceph-mon[33205]: pgmap v56: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:35.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:35 smithi102 ceph-mon[29355]: pgmap v56: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:35.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:35 smithi102 ceph-mon[40543]: pgmap v56: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:36.918 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:16:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:16:36] "GET /metrics HTTP/1.1" 200 33815 "" "Prometheus/2.51.0" 2024-09-13T14:16:37.431 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:37 smithi195 ceph-mon[33205]: pgmap v57: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:37.431 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:37 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:37.431 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:37 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:37.432 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:37 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:37.432 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:37 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:37.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:37 smithi102 ceph-mon[29355]: pgmap v57: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:37.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:37 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:37.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:37 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:37.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:37 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:37.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:37 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:37.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:37 smithi102 ceph-mon[40543]: pgmap v57: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:37.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:37 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:37.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:37 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:37.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:37 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:37.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:37 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:39.451 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:39 smithi195 ceph-mon[33205]: pgmap v58: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:39.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:39 smithi102 ceph-mon[29355]: pgmap v58: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:39.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:39 smithi102 ceph-mon[40543]: pgmap v58: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:41.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: pgmap v59: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:41.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:41.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:41.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:41.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:16:41.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:41.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T14:16:41.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:41.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:41.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:41.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:41.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:41.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:41.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:41.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:41.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:41.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:41.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:41.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:41.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:41.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:41.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:41.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:41.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:41.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-09-13T14:16:41.501 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-09-13T14:16:41.501 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-13T14:16:41.501 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:16:41.501 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:16:41.501 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-13T14:16:41.501 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-09-13T14:16:41.501 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-09-13T14:16:41.501 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-09-13T14:16:41.501 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-09-13T14:16:41.501 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-09-13T14:16:41.501 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-09-13T14:16:41.501 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-09-13T14:16:41.501 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-09-13T14:16:41.502 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-09-13T14:16:41.502 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-09-13T14:16:41.502 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-09-13T14:16:41.502 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-09-13T14:16:41.502 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-09-13T14:16:41.502 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:16:41.502 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:16:41.502 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-09-13T14:16:41.502 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-09-13T14:16:41.502 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-09-13T14:16:41.502 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-09-13T14:16:41.502 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-09-13T14:16:41.502 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-09-13T14:16:41.502 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-09-13T14:16:41.503 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-09-13T14:16:41.503 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-09-13T14:16:41.503 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-09-13T14:16:41.503 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-09-13T14:16:41.503 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-13T14:16:41.503 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:16:41.503 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:16:41.503 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:16:41.503 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:16:41.503 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:16:41.503 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:16:41.504 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:16:41.504 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:16:41.504 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:16:41.504 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:16:41.504 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:16:41.504 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:16:41.504 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-09-13T14:16:41.504 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-09-13T14:16:41.504 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-13T14:16:41.504 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:41.504 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:41.504 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:16:41.504 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:41.505 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:41.505 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:41.505 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:16:41.505 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:41 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:41.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: pgmap v59: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:41.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:41.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:41.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:41.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:16:41.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:41.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T14:16:41.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:41.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:41.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:41.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:41.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:41.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:41.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:41.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:41.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:41.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:41.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:41.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:41.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:41.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:41.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:41.671 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:41.671 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:41.671 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-09-13T14:16:41.671 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-09-13T14:16:41.671 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-13T14:16:41.671 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:16:41.671 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:16:41.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: pgmap v59: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:41.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:41.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:41.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:41.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:16:41.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:41.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T14:16:41.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:41.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:41.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:41.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:41.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:41.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:41.673 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:41.673 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:41.673 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:41.673 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:41.673 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:41.673 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:41.673 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:41.673 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:41.673 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:41.673 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:16:41.673 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:41.673 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-09-13T14:16:41.674 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-09-13T14:16:41.674 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-13T14:16:41.674 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:16:41.674 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:16:41.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-13T14:16:41.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-09-13T14:16:41.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-09-13T14:16:41.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-09-13T14:16:41.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-09-13T14:16:41.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-09-13T14:16:41.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-09-13T14:16:41.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-09-13T14:16:41.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-09-13T14:16:41.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-09-13T14:16:41.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-09-13T14:16:41.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-09-13T14:16:41.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-09-13T14:16:41.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-09-13T14:16:41.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:16:41.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:16:41.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-09-13T14:16:41.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-09-13T14:16:41.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-09-13T14:16:41.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-09-13T14:16:41.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-09-13T14:16:41.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-09-13T14:16:41.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-09-13T14:16:41.676 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-13T14:16:41.676 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-09-13T14:16:41.677 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-09-13T14:16:41.677 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-09-13T14:16:41.677 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-09-13T14:16:41.677 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-09-13T14:16:41.677 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-09-13T14:16:41.677 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-09-13T14:16:41.677 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-09-13T14:16:41.677 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-09-13T14:16:41.677 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-09-13T14:16:41.677 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-09-13T14:16:41.677 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-09-13T14:16:41.677 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-09-13T14:16:41.677 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:16:41.678 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:16:41.678 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-09-13T14:16:41.678 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-09-13T14:16:41.678 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-09-13T14:16:41.678 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-09-13T14:16:41.678 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-09-13T14:16:41.678 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-09-13T14:16:41.678 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-09-13T14:16:41.678 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-09-13T14:16:41.678 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-09-13T14:16:41.679 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-09-13T14:16:41.679 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-09-13T14:16:41.679 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-13T14:16:41.679 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:16:41.679 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:16:41.679 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:16:41.679 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:16:41.679 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:16:41.679 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:16:41.679 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:16:41.679 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:16:41.679 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:16:41.679 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:16:41.680 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:16:41.680 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:16:41.680 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-09-13T14:16:41.680 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-09-13T14:16:41.680 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-13T14:16:41.680 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:41.680 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:41.680 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:16:41.680 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:41.680 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:41.680 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-09-13T14:16:41.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-09-13T14:16:41.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-09-13T14:16:41.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-09-13T14:16:41.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-13T14:16:41.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:16:41.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:16:41.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:16:41.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:16:41.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:16:41.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:16:41.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:16:41.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:16:41.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:16:41.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:16:41.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:16:41.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:16:41.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-09-13T14:16:41.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-09-13T14:16:41.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-13T14:16:41.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:41.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:41.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:16:41.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:41.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:16:41.683 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:41.683 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:16:41.683 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:41.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:16:41.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:16:41.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:41 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:42.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:42 smithi195 ceph-mon[33205]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T14:16:42.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:42 smithi195 ceph-mon[33205]: Upgrade: Finalizing container_image settings 2024-09-13T14:16:42.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:42 smithi195 ceph-mon[33205]: Upgrade: Complete! 2024-09-13T14:16:42.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:42 smithi102 ceph-mon[29355]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T14:16:42.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:42 smithi102 ceph-mon[29355]: Upgrade: Finalizing container_image settings 2024-09-13T14:16:42.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:42 smithi102 ceph-mon[29355]: Upgrade: Complete! 2024-09-13T14:16:42.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:42 smithi102 ceph-mon[40543]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T14:16:42.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:42 smithi102 ceph-mon[40543]: Upgrade: Finalizing container_image settings 2024-09-13T14:16:42.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:42 smithi102 ceph-mon[40543]: Upgrade: Complete! 2024-09-13T14:16:43.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:43 smithi195 ceph-mon[33205]: pgmap v60: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:43.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:43 smithi102 ceph-mon[40543]: pgmap v60: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:43.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:43 smithi102 ceph-mon[29355]: pgmap v60: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:44.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:44 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:44.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:44 smithi195 ceph-mon[33205]: pgmap v61: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:44.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:44 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:44.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:44 smithi102 ceph-mon[40543]: pgmap v61: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:44.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:44 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:16:44.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:44 smithi102 ceph-mon[29355]: pgmap v61: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:46.918 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:16:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:16:46] "GET /metrics HTTP/1.1" 200 33815 "" "Prometheus/2.51.0" 2024-09-13T14:16:47.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:47 smithi102 ceph-mon[40543]: pgmap v62: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:47.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:47 smithi102 ceph-mon[29355]: pgmap v62: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:47.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:47 smithi195 ceph-mon[33205]: pgmap v62: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:49.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:49 smithi102 ceph-mon[29355]: pgmap v63: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:49.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:49 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:16:49.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:49 smithi102 ceph-mon[40543]: pgmap v63: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:49.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:49 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:16:49.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:49 smithi195 ceph-mon[33205]: pgmap v63: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:49.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:49 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:16:51.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:51 smithi102 ceph-mon[29355]: pgmap v64: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:51.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:51 smithi102 ceph-mon[40543]: pgmap v64: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:51.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:51 smithi195 ceph-mon[33205]: pgmap v64: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:53.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:53 smithi102 ceph-mon[29355]: pgmap v65: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:53.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:53 smithi102 ceph-mon[40543]: pgmap v65: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:53.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:53 smithi195 ceph-mon[33205]: pgmap v65: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:55.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:55 smithi102 ceph-mon[29355]: pgmap v66: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:55.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:55 smithi102 ceph-mon[40543]: pgmap v66: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:55.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:55 smithi195 ceph-mon[33205]: pgmap v66: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:56.918 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:16:56 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:16:56] "GET /metrics HTTP/1.1" 200 33821 "" "Prometheus/2.51.0" 2024-09-13T14:16:57.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:57 smithi195 ceph-mon[33205]: pgmap v67: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:57.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:57 smithi102 ceph-mon[29355]: pgmap v67: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:57.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:57 smithi102 ceph-mon[40543]: pgmap v67: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:59.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:16:59 smithi195 ceph-mon[33205]: pgmap v68: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:59.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:16:59 smithi102 ceph-mon[29355]: pgmap v68: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:16:59.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:16:59 smithi102 ceph-mon[40543]: pgmap v68: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:01.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:01 smithi195 ceph-mon[33205]: pgmap v69: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:01.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:01 smithi102 ceph-mon[29355]: pgmap v69: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:01.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:01 smithi102 ceph-mon[40543]: pgmap v69: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:02.812 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'echo wait for minority of mons to upgrade' 2024-09-13T14:17:03.014 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:17:03.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:03 smithi102 ceph-mon[29355]: from='client.25667 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:03.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:03 smithi102 ceph-mon[29355]: pgmap v70: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:03.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:03 smithi102 ceph-mon[40543]: from='client.25667 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:03.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:03 smithi102 ceph-mon[40543]: pgmap v70: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:03.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:03 smithi195 ceph-mon[33205]: from='client.25667 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:03.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:03 smithi195 ceph-mon[33205]: pgmap v70: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:04.448 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:04 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:17:04.448 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:04 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:17:04.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:04 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:17:04.657 INFO:teuthology.orchestra.run.smithi102.stdout:wait for minority of mons to upgrade 2024-09-13T14:17:05.345 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:05 smithi102 ceph-mon[29355]: pgmap v71: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:05.346 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:05 smithi102 ceph-mon[40543]: pgmap v71: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:05.368 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --limit 1' 2024-09-13T14:17:05.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:05 smithi195 ceph-mon[33205]: pgmap v71: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:05.567 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:17:06.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:06 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:17:06] "GET /metrics HTTP/1.1" 200 33829 "" "Prometheus/2.51.0" 2024-09-13T14:17:07.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:07 smithi102 ceph-mon[29355]: pgmap v72: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:07.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:07 smithi102 ceph-mon[40543]: pgmap v72: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:07.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:07 smithi195 ceph-mon[33205]: pgmap v72: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:09.384 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:09 smithi102 ceph-mon[29355]: from='client.15780 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9", "daemon_types": "mon", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:09.385 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:09 smithi102 ceph-mon[29355]: pgmap v73: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:09.385 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:09 smithi102 ceph-mon[40543]: from='client.15780 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9", "daemon_types": "mon", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:09.385 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:09 smithi102 ceph-mon[40543]: pgmap v73: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:09.419 INFO:teuthology.orchestra.run.smithi102.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:17:09.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:09 smithi195 ceph-mon[33205]: from='client.15780 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9", "daemon_types": "mon", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:09.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:09 smithi195 ceph-mon[33205]: pgmap v73: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:10.111 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-09-13T14:17:10.317 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:17:10.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:10 smithi102 ceph-mon[29355]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:17:10.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:10 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:17:10.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:10 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:17:10.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:10 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:10.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:10 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:17:10.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:10 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:17:10.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:10 smithi102 ceph-mon[29355]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:17:10.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:10 smithi102 ceph-mon[29355]: pgmap v74: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:10.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:10 smithi102 ceph-mon[40543]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:17:10.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:10 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:17:10.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:10 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:17:10.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:10 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:10.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:10 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:17:10.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:10 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:17:10.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:10 smithi102 ceph-mon[40543]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:17:10.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:10 smithi102 ceph-mon[40543]: pgmap v74: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:10.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:10 smithi195 ceph-mon[33205]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:17:10.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:10 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:17:10.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:10 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:17:10.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:10 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:10.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:10 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:17:10.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:10 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:17:10.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:10 smithi195 ceph-mon[33205]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:17:10.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:10 smithi195 ceph-mon[33205]: pgmap v74: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:13.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:13 smithi102 ceph-mon[29355]: pgmap v75: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:13.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:13 smithi102 ceph-mon[40543]: pgmap v75: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:13.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:13 smithi195 ceph-mon[33205]: pgmap v75: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:13.544 INFO:teuthology.orchestra.run.smithi102.stdout:true 2024-09-13T14:17:13.544 INFO:teuthology.orchestra.run.smithi102.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-13T14:17:13.544 INFO:teuthology.orchestra.run.smithi102.stdout:alertmanager.a smithi102 *:9093,9094 running (62s) 58s ago 6m 16.4M - 0.27.0 11f11916f8cd 0322b99ec767 2024-09-13T14:17:13.545 INFO:teuthology.orchestra.run.smithi102.stdout:grafana.a smithi195 *:3000 running (41s) 36s ago 6m 53.8M - c8b91775d855 152e89bad2c8 2024-09-13T14:17:13.545 INFO:teuthology.orchestra.run.smithi102.stdout:mds.foo.smithi102.njetxr smithi102 running (5m) 58s ago 5m 19.6M - 18.2.4-843-g2c01458a c129f5a2fd4e c9211cc725ed 2024-09-13T14:17:13.545 INFO:teuthology.orchestra.run.smithi102.stdout:mds.foo.smithi195.yqzrxi smithi195 running (5m) 36s ago 5m 17.3M - 18.2.4-843-g2c01458a c129f5a2fd4e eea37a7230fd 2024-09-13T14:17:13.545 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.x smithi195 *:8443,9283,8765 running (2m) 36s ago 10m 453M - 19.3.0-4697-gd58f7306 fec07016a2b0 4a23b8f0ea8b 2024-09-13T14:17:13.545 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.y smithi102 *:8443,9283,8765 running (2m) 58s ago 12m 509M - 19.3.0-4697-gd58f7306 fec07016a2b0 6b12b28dcde1 2024-09-13T14:17:13.545 INFO:teuthology.orchestra.run.smithi102.stdout:mon.a smithi102 running (12m) 58s ago 12m 53.6M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e 22c604c4fd02 2024-09-13T14:17:13.545 INFO:teuthology.orchestra.run.smithi102.stdout:mon.b smithi195 running (11m) 36s ago 11m 45.2M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e 608ec4847676 2024-09-13T14:17:13.545 INFO:teuthology.orchestra.run.smithi102.stdout:mon.c smithi102 running (10m) 58s ago 10m 43.2M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e 6c1fd3099a05 2024-09-13T14:17:13.545 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.a smithi102 *:9100 running (111s) 58s ago 6m 9592k - 1.7.0 72c9c2088986 0c92d0b39468 2024-09-13T14:17:13.545 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.b smithi195 *:9100 running (104s) 36s ago 6m 10.9M - 1.7.0 72c9c2088986 fde867a7dbf9 2024-09-13T14:17:13.545 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0 smithi102 running (10m) 58s ago 10m 59.7M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e b97482fb8cf9 2024-09-13T14:17:13.546 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1 smithi102 running (10m) 58s ago 10m 51.3M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 8a67356be7a2 2024-09-13T14:17:13.546 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2 smithi102 running (9m) 58s ago 9m 49.2M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 1800986ac838 2024-09-13T14:17:13.546 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3 smithi102 running (9m) 58s ago 9m 56.1M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 968246da9da2 2024-09-13T14:17:13.546 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4 smithi195 running (8m) 36s ago 8m 48.1M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 515307701d73 2024-09-13T14:17:13.546 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5 smithi195 running (8m) 36s ago 8m 52.4M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 176ba2b3bd59 2024-09-13T14:17:13.546 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6 smithi195 running (8m) 36s ago 8m 55.0M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 9cd070a6dd3d 2024-09-13T14:17:13.546 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7 smithi195 running (7m) 36s ago 7m 52.2M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e a6b698c659ca 2024-09-13T14:17:13.546 INFO:teuthology.orchestra.run.smithi102.stdout:prometheus.a smithi195 *:9095 running (82s) 36s ago 7m 37.6M - 2.51.0 1d3b7f56885b d131e53f1ef9 2024-09-13T14:17:13.716 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:17:13.716 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": { 2024-09-13T14:17:13.716 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 3 2024-09-13T14:17:13.716 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:17:13.716 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": { 2024-09-13T14:17:13.716 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 2 2024-09-13T14:17:13.716 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:17:13.716 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": { 2024-09-13T14:17:13.716 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 8 2024-09-13T14:17:13.716 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:17:13.717 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": { 2024-09-13T14:17:13.717 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 2 2024-09-13T14:17:13.717 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:17:13.717 INFO:teuthology.orchestra.run.smithi102.stdout: "overall": { 2024-09-13T14:17:13.717 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 13, 2024-09-13T14:17:13.717 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 2 2024-09-13T14:17:13.717 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:17:13.717 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:17:14.006 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:17:14.006 INFO:teuthology.orchestra.run.smithi102.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f", 2024-09-13T14:17:14.006 INFO:teuthology.orchestra.run.smithi102.stdout: "in_progress": true, 2024-09-13T14:17:14.006 INFO:teuthology.orchestra.run.smithi102.stdout: "which": "Upgrading daemons of type(s) mon. Upgrade limited to 1 daemons (1 remaining).", 2024-09-13T14:17:14.006 INFO:teuthology.orchestra.run.smithi102.stdout: "services_complete": [], 2024-09-13T14:17:14.006 INFO:teuthology.orchestra.run.smithi102.stdout: "progress": "0/3 daemons upgraded", 2024-09-13T14:17:14.006 INFO:teuthology.orchestra.run.smithi102.stdout: "message": "Currently upgrading mon daemons", 2024-09-13T14:17:14.006 INFO:teuthology.orchestra.run.smithi102.stdout: "is_paused": false 2024-09-13T14:17:14.006 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:17:14.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:14 smithi102 ceph-mon[29355]: from='client.15786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:14.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:14 smithi102 ceph-mon[29355]: from='client.15792 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:14.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:14 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:17:14.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:14 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:17:14.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:14 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:17:14.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:14 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:17:14.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:14 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-13T14:17:14.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:14 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-13T14:17:14.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:14 smithi102 ceph-mon[29355]: from='client.? 172.21.15.102:0/1831029246' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:17:14.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:14 smithi102 ceph-mon[40543]: from='client.15786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:14.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:14 smithi102 ceph-mon[40543]: from='client.15792 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:14.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:14 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:17:14.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:14 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:17:14.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:14 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:17:14.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:14 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:17:14.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:14 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-13T14:17:14.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:14 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-13T14:17:14.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:14 smithi102 ceph-mon[40543]: from='client.? 172.21.15.102:0/1831029246' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:17:14.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:14 smithi195 ceph-mon[33205]: from='client.15786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:14.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:14 smithi195 ceph-mon[33205]: from='client.15792 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:14.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:14 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:17:14.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:14 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:17:14.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:14 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:17:14.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:14 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:17:14.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:14 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-13T14:17:14.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:14 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-13T14:17:14.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:14 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/1831029246' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:17:15.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:15 smithi102 ceph-mon[29355]: from='client.25691 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:15.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:15 smithi102 ceph-mon[29355]: Upgrade: Target is version 19.3.0-4697-gd58f7306 (squid) 2024-09-13T14:17:15.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:15 smithi102 ceph-mon[29355]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, digests ['quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f'] 2024-09-13T14:17:15.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:15 smithi102 ceph-mon[29355]: Upgrade: Setting container_image for all mgr 2024-09-13T14:17:15.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:15 smithi102 ceph-mon[29355]: Upgrade: It appears safe to stop mon.a 2024-09-13T14:17:15.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:15 smithi102 ceph-mon[29355]: from='client.25703 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:15.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:15 smithi102 ceph-mon[29355]: pgmap v76: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:15.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:15 smithi102 ceph-mon[29355]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:17:15.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:15 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:17:15.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:15 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:17:15.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:15 smithi102 ceph-mon[29355]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:15.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:15 smithi102 ceph-mon[40543]: from='client.25691 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:15.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:15 smithi102 ceph-mon[40543]: Upgrade: Target is version 19.3.0-4697-gd58f7306 (squid) 2024-09-13T14:17:15.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:15 smithi102 ceph-mon[40543]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, digests ['quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f'] 2024-09-13T14:17:15.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:15 smithi102 ceph-mon[40543]: Upgrade: Setting container_image for all mgr 2024-09-13T14:17:15.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:15 smithi102 ceph-mon[40543]: Upgrade: It appears safe to stop mon.a 2024-09-13T14:17:15.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:15 smithi102 ceph-mon[40543]: from='client.25703 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:15.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:15 smithi102 ceph-mon[40543]: pgmap v76: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:15.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:15 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:17:15.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:15 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:17:15.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:15 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:17:15.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:15 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:15.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:15 smithi195 ceph-mon[33205]: from='client.25691 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:15.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:15 smithi195 ceph-mon[33205]: Upgrade: Target is version 19.3.0-4697-gd58f7306 (squid) 2024-09-13T14:17:15.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:15 smithi195 ceph-mon[33205]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, digests ['quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f'] 2024-09-13T14:17:15.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:15 smithi195 ceph-mon[33205]: Upgrade: Setting container_image for all mgr 2024-09-13T14:17:15.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:15 smithi195 ceph-mon[33205]: Upgrade: It appears safe to stop mon.a 2024-09-13T14:17:15.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:15 smithi195 ceph-mon[33205]: from='client.25703 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:15.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:15 smithi195 ceph-mon[33205]: pgmap v76: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:15.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:15 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='mgr.y' 2024-09-13T14:17:15.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:15 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:17:15.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:15 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:17:15.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:15 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:16.361 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:16 smithi102 ceph-mon[40543]: Upgrade: Updating mon.a 2024-09-13T14:17:16.361 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:16 smithi102 ceph-mon[40543]: Deploying daemon mon.a on smithi102 2024-09-13T14:17:16.361 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:16 smithi102 ceph-mon[29355]: Upgrade: Updating mon.a 2024-09-13T14:17:16.361 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:16 smithi102 ceph-mon[29355]: Deploying daemon mon.a on smithi102 2024-09-13T14:17:16.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:16 smithi195 ceph-mon[33205]: Upgrade: Updating mon.a 2024-09-13T14:17:16.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:16 smithi195 ceph-mon[33205]: Deploying daemon mon.a on smithi102 2024-09-13T14:17:16.641 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:17:16] "GET /metrics HTTP/1.1" 200 33829 "" "Prometheus/2.51.0" 2024-09-13T14:17:16.909 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:16 smithi102 systemd[1]: Stopping Ceph mon.a for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T14:17:17.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-a[29332]: 2024-09-13T14:17:16.908+0000 7f813f979640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T14:17:17.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-a[29332]: 2024-09-13T14:17:16.908+0000 7f813f979640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-13T14:17:17.442 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:17 smithi102 podman[109991]: 2024-09-13 14:17:17.173792553 +0000 UTC m=+0.388125178 container died 22c604c4fd0269e547915a69d7a65a74f374b0b93816f828974e294fcd2efdd4 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-a, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, RELEASE=reef-2c01458, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-13T14:17:17.767 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:17 smithi102 podman[109991]: 2024-09-13 14:17:17.58217705 +0000 UTC m=+0.796509666 container cleanup 22c604c4fd0269e547915a69d7a65a74f374b0b93816f828974e294fcd2efdd4 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=reef-2c01458, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-13T14:17:17.767 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:17 smithi102 bash[109991]: ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-a 2024-09-13T14:17:18.103 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:17 smithi102 podman[110008]: 2024-09-13 14:17:17.767095186 +0000 UTC m=+0.591268541 container remove 22c604c4fd0269e547915a69d7a65a74f374b0b93816f828974e294fcd2efdd4 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-a, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=reef-2c01458) 2024-09-13T14:17:18.103 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:17 smithi102 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@mon.a.service: Deactivated successfully. 2024-09-13T14:17:18.103 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:17 smithi102 systemd[1]: Stopped Ceph mon.a for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:17:18.103 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:17 smithi102 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@mon.a.service: Consumed 12.774s CPU time. 2024-09-13T14:17:18.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 systemd[1]: Starting Ceph mon.a for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T14:17:18.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 podman[110144]: 2024-09-13 14:17:18.213620511 +0000 UTC m=+0.015706540 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:17:18.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 podman[110144]: 2024-09-13 14:17:18.343818631 +0000 UTC m=+0.145904652 container create f3234b7bbdde9ed3592a602cc8a3bf41419784f4a968be100e922d6736358f59 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-a, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-13T14:17:18.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 podman[110144]: 2024-09-13 14:17:18.57149323 +0000 UTC m=+0.373579249 container init f3234b7bbdde9ed3592a602cc8a3bf41419784f4a968be100e922d6736358f59 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-a, org.label-schema.schema-version=1.0, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T14:17:18.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 podman[110144]: 2024-09-13 14:17:18.576487258 +0000 UTC m=+0.378573288 container start f3234b7bbdde9ed3592a602cc8a3bf41419784f4a968be100e922d6736358f59 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-a, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731) 2024-09-13T14:17:18.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: set uid:gid to 167:167 (ceph:ceph) 2024-09-13T14:17:18.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev), process ceph-mon, pid 7 2024-09-13T14:17:18.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: pidfile_write: ignore empty --pid-file 2024-09-13T14:17:18.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: load: jerasure load: lrc 2024-09-13T14:17:18.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: RocksDB version: 7.9.2 2024-09-13T14:17:18.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Git sha 0 2024-09-13T14:17:18.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Compile date 2024-09-12 20:46:14 2024-09-13T14:17:18.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: DB SUMMARY 2024-09-13T14:17:18.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: DB Session ID: T9PNQNXM6CPVUAB5671P 2024-09-13T14:17:18.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: CURRENT file: CURRENT 2024-09-13T14:17:18.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: IDENTITY file: IDENTITY 2024-09-13T14:17:18.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: MANIFEST file: MANIFEST-000015 size: 1519 Bytes 2024-09-13T14:17:18.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000032.sst 2024-09-13T14:17:18.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000030.log size: 1419536 ; 2024-09-13T14:17:18.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.error_if_exists: 0 2024-09-13T14:17:18.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.create_if_missing: 0 2024-09-13T14:17:18.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.paranoid_checks: 1 2024-09-13T14:17:18.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-13T14:17:18.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-13T14:17:18.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-13T14:17:18.921 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.env: 0x557e71df8160 2024-09-13T14:17:18.921 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.fs: PosixFileSystem 2024-09-13T14:17:18.921 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.info_log: 0x557e72fb5880 2024-09-13T14:17:18.921 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.max_file_opening_threads: 16 2024-09-13T14:17:18.921 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.statistics: (nil) 2024-09-13T14:17:18.921 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.use_fsync: 0 2024-09-13T14:17:18.921 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.max_log_file_size: 0 2024-09-13T14:17:18.921 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-13T14:17:18.921 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-13T14:17:18.921 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.keep_log_file_num: 1000 2024-09-13T14:17:18.921 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.recycle_log_file_num: 0 2024-09-13T14:17:18.921 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.allow_fallocate: 1 2024-09-13T14:17:18.922 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.allow_mmap_reads: 0 2024-09-13T14:17:18.922 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.allow_mmap_writes: 0 2024-09-13T14:17:18.922 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.use_direct_reads: 0 2024-09-13T14:17:18.922 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-13T14:17:18.922 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.create_missing_column_families: 0 2024-09-13T14:17:18.922 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.db_log_dir: 2024-09-13T14:17:18.922 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.wal_dir: 2024-09-13T14:17:18.922 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-13T14:17:18.922 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-13T14:17:18.922 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-13T14:17:18.923 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-13T14:17:18.923 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-13T14:17:18.923 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-13T14:17:18.923 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.advise_random_on_open: 1 2024-09-13T14:17:18.923 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.db_write_buffer_size: 0 2024-09-13T14:17:18.923 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.write_buffer_manager: 0x557e72fb8960 2024-09-13T14:17:18.923 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-13T14:17:18.923 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-13T14:17:18.923 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-13T14:17:18.923 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.rate_limiter: (nil) 2024-09-13T14:17:18.923 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-13T14:17:18.923 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.wal_recovery_mode: 2 2024-09-13T14:17:18.923 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.enable_thread_tracking: 0 2024-09-13T14:17:18.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.enable_pipelined_write: 0 2024-09-13T14:17:18.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.unordered_write: 0 2024-09-13T14:17:18.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-13T14:17:18.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-13T14:17:18.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-13T14:17:18.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-13T14:17:18.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.row_cache: None 2024-09-13T14:17:18.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.wal_filter: None 2024-09-13T14:17:18.925 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-13T14:17:18.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.allow_ingest_behind: 0 2024-09-13T14:17:18.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.two_write_queues: 0 2024-09-13T14:17:18.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.manual_wal_flush: 0 2024-09-13T14:17:18.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.wal_compression: 0 2024-09-13T14:17:18.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.atomic_flush: 0 2024-09-13T14:17:18.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-13T14:17:18.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-13T14:17:18.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-13T14:17:18.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.log_readahead_size: 0 2024-09-13T14:17:18.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-13T14:17:18.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.best_efforts_recovery: 0 2024-09-13T14:17:18.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-13T14:17:18.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-13T14:17:18.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.allow_data_in_errors: 0 2024-09-13T14:17:18.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.db_host_id: __hostname__ 2024-09-13T14:17:18.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-13T14:17:18.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.max_background_jobs: 2 2024-09-13T14:17:18.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.max_background_compactions: -1 2024-09-13T14:17:18.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.max_subcompactions: 1 2024-09-13T14:17:18.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-13T14:17:18.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-13T14:17:18.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-13T14:17:18.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.max_total_wal_size: 0 2024-09-13T14:17:18.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-13T14:17:18.928 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-13T14:17:18.928 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-13T14:17:18.928 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-13T14:17:18.928 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.max_open_files: -1 2024-09-13T14:17:18.928 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.bytes_per_sync: 0 2024-09-13T14:17:18.928 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-13T14:17:18.928 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-13T14:17:18.928 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.compaction_readahead_size: 0 2024-09-13T14:17:18.928 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.max_background_flushes: -1 2024-09-13T14:17:18.928 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Compression algorithms supported: 2024-09-13T14:17:18.928 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: kZSTD supported: 0 2024-09-13T14:17:18.928 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: kXpressCompression supported: 0 2024-09-13T14:17:18.928 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: kBZip2Compression supported: 0 2024-09-13T14:17:18.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-13T14:17:18.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: kLZ4Compression supported: 1 2024-09-13T14:17:18.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: kZlibCompression supported: 1 2024-09-13T14:17:18.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: kLZ4HCCompression supported: 1 2024-09-13T14:17:18.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: kSnappyCompression supported: 1 2024-09-13T14:17:18.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-13T14:17:18.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-13T14:17:18.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 2024-09-13T14:17:18.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-13T14:17:18.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-13T14:17:18.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.merge_operator: 2024-09-13T14:17:18.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.compaction_filter: None 2024-09-13T14:17:18.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.compaction_filter_factory: None 2024-09-13T14:17:18.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.sst_partitioner_factory: None 2024-09-13T14:17:18.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-13T14:17:18.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-13T14:17:18.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x557e72fb54e0) 2024-09-13T14:17:18.930 INFO:journalctl@ceph.mon.a.smithi102.stdout: cache_index_and_filter_blocks: 1 2024-09-13T14:17:18.930 INFO:journalctl@ceph.mon.a.smithi102.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-13T14:17:18.930 INFO:journalctl@ceph.mon.a.smithi102.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-13T14:17:18.930 INFO:journalctl@ceph.mon.a.smithi102.stdout: pin_top_level_index_and_filter: 1 2024-09-13T14:17:18.930 INFO:journalctl@ceph.mon.a.smithi102.stdout: index_type: 0 2024-09-13T14:17:18.930 INFO:journalctl@ceph.mon.a.smithi102.stdout: data_block_index_type: 0 2024-09-13T14:17:18.930 INFO:journalctl@ceph.mon.a.smithi102.stdout: index_shortening: 1 2024-09-13T14:17:18.930 INFO:journalctl@ceph.mon.a.smithi102.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-13T14:17:18.930 INFO:journalctl@ceph.mon.a.smithi102.stdout: checksum: 4 2024-09-13T14:17:18.931 INFO:journalctl@ceph.mon.a.smithi102.stdout: no_block_cache: 0 2024-09-13T14:17:18.931 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_cache: 0x557e72fa8850 2024-09-13T14:17:18.931 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_cache_name: BinnedLRUCache 2024-09-13T14:17:18.931 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_cache_options: 2024-09-13T14:17:18.931 INFO:journalctl@ceph.mon.a.smithi102.stdout: capacity : 536870912 2024-09-13T14:17:18.931 INFO:journalctl@ceph.mon.a.smithi102.stdout: num_shard_bits : 4 2024-09-13T14:17:18.931 INFO:journalctl@ceph.mon.a.smithi102.stdout: strict_capacity_limit : 0 2024-09-13T14:17:18.931 INFO:journalctl@ceph.mon.a.smithi102.stdout: high_pri_pool_ratio: 0.000 2024-09-13T14:17:18.931 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_cache_compressed: (nil) 2024-09-13T14:17:18.931 INFO:journalctl@ceph.mon.a.smithi102.stdout: persistent_cache: (nil) 2024-09-13T14:17:18.931 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_size: 4096 2024-09-13T14:17:18.931 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_size_deviation: 10 2024-09-13T14:17:18.931 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_restart_interval: 16 2024-09-13T14:17:18.932 INFO:journalctl@ceph.mon.a.smithi102.stdout: index_block_restart_interval: 1 2024-09-13T14:17:18.932 INFO:journalctl@ceph.mon.a.smithi102.stdout: metadata_block_size: 4096 2024-09-13T14:17:18.932 INFO:journalctl@ceph.mon.a.smithi102.stdout: partition_filters: 0 2024-09-13T14:17:18.932 INFO:journalctl@ceph.mon.a.smithi102.stdout: use_delta_encoding: 1 2024-09-13T14:17:18.932 INFO:journalctl@ceph.mon.a.smithi102.stdout: filter_policy: bloomfilter 2024-09-13T14:17:18.932 INFO:journalctl@ceph.mon.a.smithi102.stdout: whole_key_filtering: 1 2024-09-13T14:17:18.932 INFO:journalctl@ceph.mon.a.smithi102.stdout: verify_compression: 0 2024-09-13T14:17:18.932 INFO:journalctl@ceph.mon.a.smithi102.stdout: read_amp_bytes_per_bit: 0 2024-09-13T14:17:18.932 INFO:journalctl@ceph.mon.a.smithi102.stdout: format_version: 5 2024-09-13T14:17:18.932 INFO:journalctl@ceph.mon.a.smithi102.stdout: enable_index_compression: 1 2024-09-13T14:17:18.932 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_align: 0 2024-09-13T14:17:18.932 INFO:journalctl@ceph.mon.a.smithi102.stdout: max_auto_readahead_size: 262144 2024-09-13T14:17:18.932 INFO:journalctl@ceph.mon.a.smithi102.stdout: prepopulate_block_cache: 0 2024-09-13T14:17:18.933 INFO:journalctl@ceph.mon.a.smithi102.stdout: initial_auto_readahead_size: 8192 2024-09-13T14:17:18.933 INFO:journalctl@ceph.mon.a.smithi102.stdout: num_file_reads_for_auto_readahead: 2 2024-09-13T14:17:18.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.write_buffer_size: 33554432 2024-09-13T14:17:18.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.max_write_buffer_number: 2 2024-09-13T14:17:18.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.compression: NoCompression 2024-09-13T14:17:18.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.bottommost_compression: Disabled 2024-09-13T14:17:18.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.prefix_extractor: nullptr 2024-09-13T14:17:18.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-13T14:17:18.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.num_levels: 7 2024-09-13T14:17:18.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-13T14:17:18.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-13T14:17:18.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-13T14:17:18.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-13T14:17:18.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-13T14:17:18.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-13T14:17:18.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-13T14:17:18.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-13T14:17:18.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-13T14:17:18.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-13T14:17:18.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-13T14:17:18.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-13T14:17:18.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-13T14:17:18.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.compression_opts.level: 32767 2024-09-13T14:17:18.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.compression_opts.strategy: 0 2024-09-13T14:17:18.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-13T14:17:18.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-13T14:17:18.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-13T14:17:18.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-13T14:17:18.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.compression_opts.enabled: false 2024-09-13T14:17:18.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-13T14:17:18.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-13T14:17:18.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-13T14:17:18.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-13T14:17:18.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.target_file_size_base: 67108864 2024-09-13T14:17:18.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-13T14:17:18.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-13T14:17:18.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-13T14:17:18.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-13T14:17:18.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-13T14:17:18.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-13T14:17:18.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-13T14:17:18.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-13T14:17:18.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-13T14:17:18.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-13T14:17:18.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-13T14:17:18.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-13T14:17:18.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-13T14:17:18.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-13T14:17:18.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.arena_block_size: 1048576 2024-09-13T14:17:18.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-13T14:17:18.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-13T14:17:18.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.disable_auto_compactions: 0 2024-09-13T14:17:18.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-13T14:17:18.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-13T14:17:18.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-13T14:17:18.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-13T14:17:18.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-13T14:17:18.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-13T14:17:18.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-13T14:17:18.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-13T14:17:18.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-13T14:17:18.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-13T14:17:18.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-13T14:17:18.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.inplace_update_support: 0 2024-09-13T14:17:18.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-13T14:17:18.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-13T14:17:18.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-13T14:17:18.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-13T14:17:18.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.bloom_locality: 0 2024-09-13T14:17:18.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.max_successive_merges: 0 2024-09-13T14:17:18.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-13T14:17:18.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.paranoid_file_checks: 0 2024-09-13T14:17:18.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.force_consistency_checks: 1 2024-09-13T14:17:18.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.report_bg_io_stats: 0 2024-09-13T14:17:18.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.ttl: 2592000 2024-09-13T14:17:18.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-13T14:17:18.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-13T14:17:18.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-13T14:17:18.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.enable_blob_files: false 2024-09-13T14:17:18.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.min_blob_size: 0 2024-09-13T14:17:18.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.blob_file_size: 268435456 2024-09-13T14:17:18.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-13T14:17:18.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-13T14:17:18.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-13T14:17:18.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-13T14:17:18.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-13T14:17:18.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.blob_file_starting_level: 0 2024-09-13T14:17:18.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-13T14:17:18.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-13T14:17:18.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 34, last_sequence is 12339, log_number is 30,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 30 2024-09-13T14:17:18.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 30 2024-09-13T14:17:18.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 02b0d92a-6f0f-4483-8913-1505901ad274 2024-09-13T14:17:18.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726237038602518, "job": 1, "event": "recovery_started", "wal_files": [30]} 2024-09-13T14:17:18.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #30 mode 2 2024-09-13T14:17:18.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726237038607668, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 35, "file_size": 1303352, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12340, "largest_seqno": 13481, "table_properties": {"data_size": 1297707, "index_size": 3196, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1413, "raw_key_size": 13475, "raw_average_key_size": 24, "raw_value_size": 1286082, "raw_average_value_size": 2351, "num_data_blocks": 145, "num_entries": 547, "num_filter_entries": 547, "num_deletions": 8, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726237038, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "02b0d92a-6f0f-4483-8913-1505901ad274", "db_session_id": "T9PNQNXM6CPVUAB5671P", "orig_file_number": 35, "seqno_to_time_mapping": "N/A"}} 2024-09-13T14:17:18.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726237038607774, "job": 1, "event": "recovery_finished"} 2024-09-13T14:17:18.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: [db/version_set.cc:5047] Creating manifest 37 2024-09-13T14:17:18.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-13T14:17:18.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000030.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-13T14:17:18.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x557e72fd8e00 2024-09-13T14:17:18.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: DB pointer 0x557e730d8000 2024-09-13T14:17:18.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-13T14:17:18.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-13T14:17:18.940 INFO:journalctl@ceph.mon.a.smithi102.stdout: ** DB Stats ** 2024-09-13T14:17:18.940 INFO:journalctl@ceph.mon.a.smithi102.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T14:17:18.941 INFO:journalctl@ceph.mon.a.smithi102.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-13T14:17:18.941 INFO:journalctl@ceph.mon.a.smithi102.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-13T14:17:18.941 INFO:journalctl@ceph.mon.a.smithi102.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T14:17:18.941 INFO:journalctl@ceph.mon.a.smithi102.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-13T14:17:18.941 INFO:journalctl@ceph.mon.a.smithi102.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-13T14:17:18.941 INFO:journalctl@ceph.mon.a.smithi102.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T14:17:18.941 INFO:journalctl@ceph.mon.a.smithi102.stdout: 2024-09-13T14:17:18.941 INFO:journalctl@ceph.mon.a.smithi102.stdout: ** Compaction Stats [default] ** 2024-09-13T14:17:18.941 INFO:journalctl@ceph.mon.a.smithi102.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-13T14:17:18.941 INFO:journalctl@ceph.mon.a.smithi102.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-13T14:17:18.941 INFO:journalctl@ceph.mon.a.smithi102.stdout: L0 1/0 1.24 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 455.0 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-09-13T14:17:18.941 INFO:journalctl@ceph.mon.a.smithi102.stdout: L6 1/0 7.96 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-13T14:17:18.942 INFO:journalctl@ceph.mon.a.smithi102.stdout: Sum 2/0 9.20 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 455.0 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-09-13T14:17:18.942 INFO:journalctl@ceph.mon.a.smithi102.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 455.0 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-09-13T14:17:18.942 INFO:journalctl@ceph.mon.a.smithi102.stdout: 2024-09-13T14:17:18.942 INFO:journalctl@ceph.mon.a.smithi102.stdout: ** Compaction Stats [default] ** 2024-09-13T14:17:18.942 INFO:journalctl@ceph.mon.a.smithi102.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-13T14:17:18.942 INFO:journalctl@ceph.mon.a.smithi102.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-13T14:17:18.942 INFO:journalctl@ceph.mon.a.smithi102.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 455.0 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-09-13T14:17:18.942 INFO:journalctl@ceph.mon.a.smithi102.stdout: 2024-09-13T14:17:18.942 INFO:journalctl@ceph.mon.a.smithi102.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-13T14:17:18.942 INFO:journalctl@ceph.mon.a.smithi102.stdout: 2024-09-13T14:17:18.942 INFO:journalctl@ceph.mon.a.smithi102.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T14:17:18.942 INFO:journalctl@ceph.mon.a.smithi102.stdout: Flush(GB): cumulative 0.001, interval 0.001 2024-09-13T14:17:18.943 INFO:journalctl@ceph.mon.a.smithi102.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-13T14:17:18.943 INFO:journalctl@ceph.mon.a.smithi102.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-13T14:17:18.943 INFO:journalctl@ceph.mon.a.smithi102.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-13T14:17:18.943 INFO:journalctl@ceph.mon.a.smithi102.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-13T14:17:18.943 INFO:journalctl@ceph.mon.a.smithi102.stdout: Cumulative compaction: 0.00 GB write, 165.09 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T14:17:18.943 INFO:journalctl@ceph.mon.a.smithi102.stdout: Interval compaction: 0.00 GB write, 165.09 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T14:17:18.943 INFO:journalctl@ceph.mon.a.smithi102.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-13T14:17:18.943 INFO:journalctl@ceph.mon.a.smithi102.stdout: Block cache BinnedLRUCache@0x557e72fa8850#7 capacity: 512.00 MB usage: 4.86 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-09-13T14:17:18.943 INFO:journalctl@ceph.mon.a.smithi102.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.53 KB,0.000292063%) IndexBlock(1,3.33 KB,0.000634789%) Misc(1,0.00 KB,0%) 2024-09-13T14:17:18.943 INFO:journalctl@ceph.mon.a.smithi102.stdout: 2024-09-13T14:17:18.943 INFO:journalctl@ceph.mon.a.smithi102.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-13T14:17:18.943 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: starting mon.a rank 0 at public addrs [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] at bind addrs [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 08cee502-71d9-11ef-bceb-c7b262605968 2024-09-13T14:17:18.943 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: mon.a@-1(???) e3 preinit fsid 08cee502-71d9-11ef-bceb-c7b262605968 2024-09-13T14:17:18.944 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: mon.a@-1(???).mds e9 new map 2024-09-13T14:17:18.944 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: mon.a@-1(???).mds e9 print_map 2024-09-13T14:17:18.944 INFO:journalctl@ceph.mon.a.smithi102.stdout: e9 2024-09-13T14:17:18.944 INFO:journalctl@ceph.mon.a.smithi102.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-09-13T14:17:18.944 INFO:journalctl@ceph.mon.a.smithi102.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-13T14:17:18.944 INFO:journalctl@ceph.mon.a.smithi102.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-13T14:17:18.944 INFO:journalctl@ceph.mon.a.smithi102.stdout: legacy client fscid: 1 2024-09-13T14:17:18.944 INFO:journalctl@ceph.mon.a.smithi102.stdout: 2024-09-13T14:17:18.944 INFO:journalctl@ceph.mon.a.smithi102.stdout: Filesystem 'foo' (1) 2024-09-13T14:17:18.944 INFO:journalctl@ceph.mon.a.smithi102.stdout: fs_name foo 2024-09-13T14:17:18.944 INFO:journalctl@ceph.mon.a.smithi102.stdout: epoch 8 2024-09-13T14:17:18.944 INFO:journalctl@ceph.mon.a.smithi102.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-09-13T14:17:18.944 INFO:journalctl@ceph.mon.a.smithi102.stdout: created 2024-09-13T14:11:29.232738+0000 2024-09-13T14:17:18.944 INFO:journalctl@ceph.mon.a.smithi102.stdout: modified 2024-09-13T14:11:41.325298+0000 2024-09-13T14:17:18.945 INFO:journalctl@ceph.mon.a.smithi102.stdout: tableserver 0 2024-09-13T14:17:18.945 INFO:journalctl@ceph.mon.a.smithi102.stdout: root 0 2024-09-13T14:17:18.945 INFO:journalctl@ceph.mon.a.smithi102.stdout: session_timeout 60 2024-09-13T14:17:18.945 INFO:journalctl@ceph.mon.a.smithi102.stdout: session_autoclose 300 2024-09-13T14:17:18.945 INFO:journalctl@ceph.mon.a.smithi102.stdout: max_file_size 1099511627776 2024-09-13T14:17:18.945 INFO:journalctl@ceph.mon.a.smithi102.stdout: max_xattr_size 65536 2024-09-13T14:17:18.945 INFO:journalctl@ceph.mon.a.smithi102.stdout: required_client_features {} 2024-09-13T14:17:18.945 INFO:journalctl@ceph.mon.a.smithi102.stdout: last_failure 0 2024-09-13T14:17:18.945 INFO:journalctl@ceph.mon.a.smithi102.stdout: last_failure_osd_epoch 0 2024-09-13T14:17:18.945 INFO:journalctl@ceph.mon.a.smithi102.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-13T14:17:18.945 INFO:journalctl@ceph.mon.a.smithi102.stdout: max_mds 1 2024-09-13T14:17:18.945 INFO:journalctl@ceph.mon.a.smithi102.stdout: in 0 2024-09-13T14:17:18.945 INFO:journalctl@ceph.mon.a.smithi102.stdout: up {0=24521} 2024-09-13T14:17:18.945 INFO:journalctl@ceph.mon.a.smithi102.stdout: failed 2024-09-13T14:17:18.946 INFO:journalctl@ceph.mon.a.smithi102.stdout: damaged 2024-09-13T14:17:18.946 INFO:journalctl@ceph.mon.a.smithi102.stdout: stopped 2024-09-13T14:17:18.946 INFO:journalctl@ceph.mon.a.smithi102.stdout: data_pools [4] 2024-09-13T14:17:18.946 INFO:journalctl@ceph.mon.a.smithi102.stdout: metadata_pool 3 2024-09-13T14:17:18.946 INFO:journalctl@ceph.mon.a.smithi102.stdout: inline_data disabled 2024-09-13T14:17:18.946 INFO:journalctl@ceph.mon.a.smithi102.stdout: balancer 2024-09-13T14:17:18.946 INFO:journalctl@ceph.mon.a.smithi102.stdout: bal_rank_mask -1 2024-09-13T14:17:18.946 INFO:journalctl@ceph.mon.a.smithi102.stdout: standby_count_wanted 1 2024-09-13T14:17:18.946 INFO:journalctl@ceph.mon.a.smithi102.stdout: qdb_cluster leader: 0 members: 2024-09-13T14:17:18.946 INFO:journalctl@ceph.mon.a.smithi102.stdout: [mds.foo.smithi102.njetxr{0:24521} state up:active seq 3 join_fscid=1 addr [v2:172.21.15.102:6834/54188465,v1:172.21.15.102:6835/54188465] compat {c=[1],r=[1],i=[7ff]}] 2024-09-13T14:17:18.946 INFO:journalctl@ceph.mon.a.smithi102.stdout: 2024-09-13T14:17:18.947 INFO:journalctl@ceph.mon.a.smithi102.stdout: 2024-09-13T14:17:18.947 INFO:journalctl@ceph.mon.a.smithi102.stdout: Standby daemons: 2024-09-13T14:17:18.947 INFO:journalctl@ceph.mon.a.smithi102.stdout: 2024-09-13T14:17:18.947 INFO:journalctl@ceph.mon.a.smithi102.stdout: [mds.foo.smithi195.yqzrxi{-1:24442} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.195:6832/2017468914,v1:172.21.15.195:6833/2017468914] compat {c=[1],r=[1],i=[7ff]}] 2024-09-13T14:17:18.947 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: mon.a@-1(???).osd e89 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-13T14:17:18.947 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: mon.a@-1(???).osd e89 crush map has features 432629239337189376, adjusting msgr requires 2024-09-13T14:17:18.947 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: mon.a@-1(???).osd e89 crush map has features 432629239337189376, adjusting msgr requires 2024-09-13T14:17:18.947 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: mon.a@-1(???).osd e89 crush map has features 432629239337189376, adjusting msgr requires 2024-09-13T14:17:18.947 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 ceph-mon[110183]: mon.a@-1(???).paxosservice(auth 1..19) refresh upgraded, format 0 -> 3 2024-09-13T14:17:18.947 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 bash[110144]: f3234b7bbdde9ed3592a602cc8a3bf41419784f4a968be100e922d6736358f59 2024-09-13T14:17:18.947 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:18 smithi102 systemd[1]: Started Ceph mon.a for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:17:20.418 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:20 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ignoring --setuser ceph since I am not root 2024-09-13T14:17:20.419 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:20 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ignoring --setgroup ceph since I am not root 2024-09-13T14:17:20.419 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:20 smithi102 ceph-mgr[91242]: -- 172.21.15.102:0/4104315107 <== mon.0 v2:172.21.15.102:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55855499b0e0 con 0x558554979000 2024-09-13T14:17:20.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:20 smithi102 ceph-mon[110183]: pgmap v78: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:20.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:20 smithi102 ceph-mon[110183]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:17:20.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:20 smithi102 ceph-mon[110183]: mon.a calling monitor election 2024-09-13T14:17:20.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:20 smithi102 ceph-mon[110183]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T14:17:20.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:20 smithi102 ceph-mon[110183]: monmap epoch 3 2024-09-13T14:17:20.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:20 smithi102 ceph-mon[110183]: fsid 08cee502-71d9-11ef-bceb-c7b262605968 2024-09-13T14:17:20.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:20 smithi102 ceph-mon[110183]: last_changed 2024-09-13T14:06:18.744278+0000 2024-09-13T14:17:20.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:20 smithi102 ceph-mon[110183]: created 2024-09-13T14:04:45.172288+0000 2024-09-13T14:17:20.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:20 smithi102 ceph-mon[110183]: min_mon_release 18 (reef) 2024-09-13T14:17:20.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:20 smithi102 ceph-mon[110183]: election_strategy: 1 2024-09-13T14:17:20.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:20 smithi102 ceph-mon[110183]: 0: [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon.a 2024-09-13T14:17:20.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:20 smithi102 ceph-mon[110183]: 1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2024-09-13T14:17:20.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:20 smithi102 ceph-mon[110183]: 2: [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] mon.c 2024-09-13T14:17:20.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:20 smithi102 ceph-mon[110183]: fsmap foo:1 {0=foo.smithi102.njetxr=up:active} 1 up:standby 2024-09-13T14:17:20.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:20 smithi102 ceph-mon[110183]: osdmap e89: 8 total, 8 up, 8 in 2024-09-13T14:17:20.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:20 smithi102 ceph-mon[110183]: mgrmap e34: y(active, since 2m), standbys: x 2024-09-13T14:17:20.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:20 smithi102 ceph-mon[110183]: overall HEALTH_OK 2024-09-13T14:17:20.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:20 smithi102 ceph-mon[110183]: from='mgr.25496 ' entity='' 2024-09-13T14:17:20.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:20 smithi102 ceph-mon[110183]: mgrmap e35: y(active, since 2m), standbys: x 2024-09-13T14:17:20.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:20 smithi102 ceph-mon[40543]: pgmap v78: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:20.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:20 smithi102 ceph-mon[40543]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:17:20.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:20 smithi102 ceph-mon[40543]: mon.a calling monitor election 2024-09-13T14:17:20.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:20 smithi102 ceph-mon[40543]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T14:17:20.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:20 smithi102 ceph-mon[40543]: monmap epoch 3 2024-09-13T14:17:20.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:20 smithi102 ceph-mon[40543]: fsid 08cee502-71d9-11ef-bceb-c7b262605968 2024-09-13T14:17:20.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:20 smithi102 ceph-mon[40543]: last_changed 2024-09-13T14:06:18.744278+0000 2024-09-13T14:17:20.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:20 smithi102 ceph-mon[40543]: created 2024-09-13T14:04:45.172288+0000 2024-09-13T14:17:20.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:20 smithi102 ceph-mon[40543]: min_mon_release 18 (reef) 2024-09-13T14:17:20.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:20 smithi102 ceph-mon[40543]: election_strategy: 1 2024-09-13T14:17:20.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:20 smithi102 ceph-mon[40543]: 0: [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon.a 2024-09-13T14:17:20.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:20 smithi102 ceph-mon[40543]: 1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2024-09-13T14:17:20.422 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:20 smithi102 ceph-mon[40543]: 2: [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] mon.c 2024-09-13T14:17:20.422 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:20 smithi102 ceph-mon[40543]: fsmap foo:1 {0=foo.smithi102.njetxr=up:active} 1 up:standby 2024-09-13T14:17:20.422 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:20 smithi102 ceph-mon[40543]: osdmap e89: 8 total, 8 up, 8 in 2024-09-13T14:17:20.422 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:20 smithi102 ceph-mon[40543]: mgrmap e34: y(active, since 2m), standbys: x 2024-09-13T14:17:20.422 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:20 smithi102 ceph-mon[40543]: overall HEALTH_OK 2024-09-13T14:17:20.422 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:20 smithi102 ceph-mon[40543]: from='mgr.25496 ' entity='' 2024-09-13T14:17:20.422 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:20 smithi102 ceph-mon[40543]: mgrmap e35: y(active, since 2m), standbys: x 2024-09-13T14:17:20.497 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:17:20 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[81504]: ignoring --setuser ceph since I am not root 2024-09-13T14:17:20.497 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:17:20 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[81504]: ignoring --setgroup ceph since I am not root 2024-09-13T14:17:20.497 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:17:20 smithi195 ceph-mgr[81527]: -- 172.21.15.195:0/558773831 <== mon.1 v2:172.21.15.195:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564c81b610e0 con 0x564c81b3f400 2024-09-13T14:17:20.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:20 smithi195 ceph-mon[33205]: pgmap v78: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:20.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:20 smithi195 ceph-mon[33205]: from='mgr.25496 172.21.15.102:0/832254289' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:17:20.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:20 smithi195 ceph-mon[33205]: mon.a calling monitor election 2024-09-13T14:17:20.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:20 smithi195 ceph-mon[33205]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T14:17:20.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:20 smithi195 ceph-mon[33205]: monmap epoch 3 2024-09-13T14:17:20.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:20 smithi195 ceph-mon[33205]: fsid 08cee502-71d9-11ef-bceb-c7b262605968 2024-09-13T14:17:20.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:20 smithi195 ceph-mon[33205]: last_changed 2024-09-13T14:06:18.744278+0000 2024-09-13T14:17:20.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:20 smithi195 ceph-mon[33205]: created 2024-09-13T14:04:45.172288+0000 2024-09-13T14:17:20.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:20 smithi195 ceph-mon[33205]: min_mon_release 18 (reef) 2024-09-13T14:17:20.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:20 smithi195 ceph-mon[33205]: election_strategy: 1 2024-09-13T14:17:20.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:20 smithi195 ceph-mon[33205]: 0: [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon.a 2024-09-13T14:17:20.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:20 smithi195 ceph-mon[33205]: 1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2024-09-13T14:17:20.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:20 smithi195 ceph-mon[33205]: 2: [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] mon.c 2024-09-13T14:17:20.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:20 smithi195 ceph-mon[33205]: fsmap foo:1 {0=foo.smithi102.njetxr=up:active} 1 up:standby 2024-09-13T14:17:20.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:20 smithi195 ceph-mon[33205]: osdmap e89: 8 total, 8 up, 8 in 2024-09-13T14:17:20.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:20 smithi195 ceph-mon[33205]: mgrmap e34: y(active, since 2m), standbys: x 2024-09-13T14:17:20.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:20 smithi195 ceph-mon[33205]: overall HEALTH_OK 2024-09-13T14:17:20.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:20 smithi195 ceph-mon[33205]: from='mgr.25496 ' entity='' 2024-09-13T14:17:20.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:20 smithi195 ceph-mon[33205]: mgrmap e35: y(active, since 2m), standbys: x 2024-09-13T14:17:25.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:25 smithi102 ceph-mon[110183]: Standby manager daemon x restarted 2024-09-13T14:17:25.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:25 smithi102 ceph-mon[110183]: Standby manager daemon x started 2024-09-13T14:17:25.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:25 smithi102 ceph-mon[110183]: from='mgr.? 172.21.15.195:0/1763820670' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:17:25.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:25 smithi102 ceph-mon[110183]: from='mgr.? 172.21.15.195:0/1763820670' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:17:25.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:25 smithi102 ceph-mon[110183]: from='mgr.? 172.21.15.195:0/1763820670' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:17:25.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:25 smithi102 ceph-mon[110183]: from='mgr.? 172.21.15.195:0/1763820670' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:17:25.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:25 smithi102 ceph-mon[40543]: Standby manager daemon x restarted 2024-09-13T14:17:25.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:25 smithi102 ceph-mon[40543]: Standby manager daemon x started 2024-09-13T14:17:25.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:25 smithi102 ceph-mon[40543]: from='mgr.? 172.21.15.195:0/1763820670' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:17:25.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:25 smithi102 ceph-mon[40543]: from='mgr.? 172.21.15.195:0/1763820670' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:17:25.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:25 smithi102 ceph-mon[40543]: from='mgr.? 172.21.15.195:0/1763820670' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:17:25.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:25 smithi102 ceph-mon[40543]: from='mgr.? 172.21.15.195:0/1763820670' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:17:25.497 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:17:25 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[81504]: [13/Sep/2024:14:17:25] ENGINE Bus STARTING 2024-09-13T14:17:25.497 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:17:25 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[81504]: CherryPy Checker: 2024-09-13T14:17:25.497 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:17:25 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[81504]: The Application mounted at '' has an empty config. 2024-09-13T14:17:25.498 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:17:25 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[81504]: 2024-09-13T14:17:25.498 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:17:25 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[81504]: [13/Sep/2024:14:17:25] ENGINE Serving on http://:::9283 2024-09-13T14:17:25.498 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 14:17:25 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x[81504]: [13/Sep/2024:14:17:25] ENGINE Bus STARTED 2024-09-13T14:17:25.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:25 smithi195 ceph-mon[33205]: Standby manager daemon x restarted 2024-09-13T14:17:25.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:25 smithi195 ceph-mon[33205]: Standby manager daemon x started 2024-09-13T14:17:25.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:25 smithi195 ceph-mon[33205]: from='mgr.? 172.21.15.195:0/1763820670' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:17:25.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:25 smithi195 ceph-mon[33205]: from='mgr.? 172.21.15.195:0/1763820670' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:17:25.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:25 smithi195 ceph-mon[33205]: from='mgr.? 172.21.15.195:0/1763820670' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:17:25.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:25 smithi195 ceph-mon[33205]: from='mgr.? 172.21.15.195:0/1763820670' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:17:25.918 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:25 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: 2024-09-13T14:17:25.445+0000 7f0c60b4c640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:17:25.918 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:25 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: 2024-09-13T14:17:25.446+0000 7f0c60b4c640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:17:25.918 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:25 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: 2024-09-13T14:17:25.446+0000 7f0c60b4c640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:17:25.918 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:25 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: 2024-09-13T14:17:25.446+0000 7f0c60b4c640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:17:25.918 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:25 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: 2024-09-13T14:17:25.446+0000 7f0c60b4c640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:17:25.919 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:25 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: 2024-09-13T14:17:25.446+0000 7f0c64353640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:17:25.919 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:25 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: 2024-09-13T14:17:25.446+0000 7f0c64353640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:17:25.919 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:25 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: 2024-09-13T14:17:25.446+0000 7f0c64353640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:17:25.919 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:25 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: 2024-09-13T14:17:25.446+0000 7f0c64353640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:17:25.919 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:25 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: 2024-09-13T14:17:25.446+0000 7f0c64353640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:17:26.077 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:26 smithi195 ceph-mon[33205]: mgrmap e36: y(active, since 2m), standbys: x 2024-09-13T14:17:26.078 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:26 smithi195 ceph-mon[33205]: Active manager daemon y restarted 2024-09-13T14:17:26.078 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:26 smithi195 ceph-mon[33205]: Activating manager daemon y 2024-09-13T14:17:26.078 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:26 smithi195 ceph-mon[33205]: osdmap e90: 8 total, 8 up, 8 in 2024-09-13T14:17:26.078 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:26 smithi195 ceph-mon[33205]: mgrmap e37: y(active, starting, since 0.0069586s), standbys: x 2024-09-13T14:17:26.078 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:26 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:17:26.078 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:26 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:17:26.078 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:26 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:17:26.078 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:26 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi195.yqzrxi"}]: dispatch 2024-09-13T14:17:26.078 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:26 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi102.njetxr"}]: dispatch 2024-09-13T14:17:26.078 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:26 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:17:26.078 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:26 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:17:26.078 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:26 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:17:26.079 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:26 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:17:26.079 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:26 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:17:26.079 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:26 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:17:26.079 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:26 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:17:26.079 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:26 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:17:26.079 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:26 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:17:26.079 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:26 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:17:26.079 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:26 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:17:26.079 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:26 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:17:26.079 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:26 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:17:26.079 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:26 smithi195 ceph-mon[33205]: Manager daemon y is now available 2024-09-13T14:17:26.079 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:26 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:17:26.080 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:26 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:17:26.080 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:26 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:17:26.080 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:26 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:17:26.080 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:26 smithi195 ceph-mon[33205]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-13T14:17:26.080 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:26 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:17:26.080 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:26 smithi195 ceph-mon[33205]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-13T14:17:26.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[110183]: mgrmap e36: y(active, since 2m), standbys: x 2024-09-13T14:17:26.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[110183]: Active manager daemon y restarted 2024-09-13T14:17:26.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[110183]: Activating manager daemon y 2024-09-13T14:17:26.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[110183]: osdmap e90: 8 total, 8 up, 8 in 2024-09-13T14:17:26.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[110183]: mgrmap e37: y(active, starting, since 0.0069586s), standbys: x 2024-09-13T14:17:26.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:17:26.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:17:26.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:17:26.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi195.yqzrxi"}]: dispatch 2024-09-13T14:17:26.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi102.njetxr"}]: dispatch 2024-09-13T14:17:26.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:17:26.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:17:26.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:17:26.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:17:26.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:17:26.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:17:26.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:17:26.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:17:26.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:17:26.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:17:26.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:17:26.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:17:26.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:17:26.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[110183]: Manager daemon y is now available 2024-09-13T14:17:26.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:17:26.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:17:26.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:17:26.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:17:26.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[110183]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-13T14:17:26.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:17:26.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[110183]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-13T14:17:26.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[40543]: mgrmap e36: y(active, since 2m), standbys: x 2024-09-13T14:17:26.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[40543]: Active manager daemon y restarted 2024-09-13T14:17:26.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[40543]: Activating manager daemon y 2024-09-13T14:17:26.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[40543]: osdmap e90: 8 total, 8 up, 8 in 2024-09-13T14:17:26.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[40543]: mgrmap e37: y(active, starting, since 0.0069586s), standbys: x 2024-09-13T14:17:26.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[40543]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:17:26.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[40543]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:17:26.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[40543]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:17:26.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[40543]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi195.yqzrxi"}]: dispatch 2024-09-13T14:17:26.422 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[40543]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi102.njetxr"}]: dispatch 2024-09-13T14:17:26.422 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[40543]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:17:26.422 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[40543]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:17:26.422 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[40543]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:17:26.422 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[40543]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:17:26.422 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[40543]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:17:26.422 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[40543]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:17:26.422 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[40543]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:17:26.422 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[40543]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:17:26.422 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[40543]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:17:26.422 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[40543]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:17:26.422 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[40543]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:17:26.422 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[40543]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:17:26.422 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[40543]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:17:26.423 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[40543]: Manager daemon y is now available 2024-09-13T14:17:26.423 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[40543]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:17:26.423 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[40543]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:17:26.423 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[40543]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:17:26.423 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[40543]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:17:26.423 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[40543]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-13T14:17:26.423 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[40543]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:17:26.423 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-mon[40543]: from='mgr.34107 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-13T14:17:26.423 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: 2024-09-13T14:17:26.110+0000 7f0c92e7e640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-13T14:17:26.423 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: [13/Sep/2024:14:17:26] ENGINE Bus STARTING 2024-09-13T14:17:26.423 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: CherryPy Checker: 2024-09-13T14:17:26.424 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: The Application mounted at '' has an empty config. 2024-09-13T14:17:26.424 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: 2024-09-13T14:17:26.424 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: [13/Sep/2024:14:17:26] ENGINE Serving on http://:::9283 2024-09-13T14:17:26.424 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: [13/Sep/2024:14:17:26] ENGINE Bus STARTED 2024-09-13T14:17:26.424 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: 2024-09-13T14:17:26.260+0000 7f0c41153640 -1 log_channel(cephadm) log [ERR] : [13/Sep/2024:14:17:26] ENGINE Error in HTTPServer.serve 2024-09-13T14:17:26.424 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: Traceback (most recent call last): 2024-09-13T14:17:26.424 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:17:26.424 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: self._connections.run(self.expiration_interval) 2024-09-13T14:17:26.424 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:17:26.424 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: self._run(expiration_interval) 2024-09-13T14:17:26.424 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:17:26.424 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:17:26.424 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:17:26.425 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:17:26.425 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:17:26.425 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: s = self.context.wrap_socket( 2024-09-13T14:17:26.425 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:17:26.425 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: return self.sslsocket_class._create( 2024-09-13T14:17:26.425 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:17:26.425 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: self.do_handshake() 2024-09-13T14:17:26.425 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:17:26.425 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: self._sslobj.do_handshake() 2024-09-13T14:17:26.425 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:17:26.425 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: 2024-09-13T14:17:26.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:17:26] "GET /metrics HTTP/1.1" 200 31138 "" "Prometheus/2.51.0" 2024-09-13T14:17:27.380 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:27 smithi102 ceph-mon[110183]: [13/Sep/2024:14:17:26] ENGINE Bus STARTING 2024-09-13T14:17:27.380 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:27 smithi102 ceph-mon[110183]: mgrmap e38: y(active, since 1.00848s), standbys: x 2024-09-13T14:17:27.380 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:27 smithi102 ceph-mon[40543]: [13/Sep/2024:14:17:26] ENGINE Bus STARTING 2024-09-13T14:17:27.380 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:27 smithi102 ceph-mon[40543]: mgrmap e38: y(active, since 1.00848s), standbys: x 2024-09-13T14:17:27.381 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:27 smithi195 ceph-mon[33205]: [13/Sep/2024:14:17:26] ENGINE Bus STARTING 2024-09-13T14:17:27.381 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:27 smithi195 ceph-mon[33205]: mgrmap e38: y(active, since 1.00848s), standbys: x 2024-09-13T14:17:28.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:28 smithi102 ceph-mon[40543]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:17:28.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:28 smithi102 ceph-mon[40543]: pgmap v3: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:28.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:28 smithi102 ceph-mon[40543]: [13/Sep/2024:14:17:26] ENGINE Serving on http://172.21.15.102:8765 2024-09-13T14:17:28.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:28 smithi102 ceph-mon[40543]: [13/Sep/2024:14:17:26] ENGINE Serving on https://172.21.15.102:7150 2024-09-13T14:17:28.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:28 smithi102 ceph-mon[40543]: [13/Sep/2024:14:17:26] ENGINE Bus STARTED 2024-09-13T14:17:28.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:28 smithi102 ceph-mon[40543]: [13/Sep/2024:14:17:26] ENGINE Error in HTTPServer.serve 2024-09-13T14:17:28.419 INFO:journalctl@ceph.mon.c.smithi102.stdout: Traceback (most recent call last): 2024-09-13T14:17:28.419 INFO:journalctl@ceph.mon.c.smithi102.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:17:28.419 INFO:journalctl@ceph.mon.c.smithi102.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:17:28.419 INFO:journalctl@ceph.mon.c.smithi102.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:17:28.419 INFO:journalctl@ceph.mon.c.smithi102.stdout: self._run(expiration_interval) 2024-09-13T14:17:28.419 INFO:journalctl@ceph.mon.c.smithi102.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:17:28.419 INFO:journalctl@ceph.mon.c.smithi102.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:17:28.419 INFO:journalctl@ceph.mon.c.smithi102.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:17:28.419 INFO:journalctl@ceph.mon.c.smithi102.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:17:28.419 INFO:journalctl@ceph.mon.c.smithi102.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:17:28.419 INFO:journalctl@ceph.mon.c.smithi102.stdout: s = self.context.wrap_socket( 2024-09-13T14:17:28.419 INFO:journalctl@ceph.mon.c.smithi102.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:17:28.420 INFO:journalctl@ceph.mon.c.smithi102.stdout: return self.sslsocket_class._create( 2024-09-13T14:17:28.420 INFO:journalctl@ceph.mon.c.smithi102.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:17:28.420 INFO:journalctl@ceph.mon.c.smithi102.stdout: self.do_handshake() 2024-09-13T14:17:28.420 INFO:journalctl@ceph.mon.c.smithi102.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:17:28.420 INFO:journalctl@ceph.mon.c.smithi102.stdout: self._sslobj.do_handshake() 2024-09-13T14:17:28.420 INFO:journalctl@ceph.mon.c.smithi102.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:17:28.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:28 smithi102 ceph-mon[40543]: pgmap v4: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:28.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:28 smithi102 ceph-mon[40543]: mgrmap e39: y(active, since 2s), standbys: x 2024-09-13T14:17:28.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:28 smithi102 ceph-mon[110183]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:17:28.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:28 smithi102 ceph-mon[110183]: pgmap v3: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:28.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:28 smithi102 ceph-mon[110183]: [13/Sep/2024:14:17:26] ENGINE Serving on http://172.21.15.102:8765 2024-09-13T14:17:28.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:28 smithi102 ceph-mon[110183]: [13/Sep/2024:14:17:26] ENGINE Serving on https://172.21.15.102:7150 2024-09-13T14:17:28.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:28 smithi102 ceph-mon[110183]: [13/Sep/2024:14:17:26] ENGINE Bus STARTED 2024-09-13T14:17:28.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:28 smithi102 ceph-mon[110183]: [13/Sep/2024:14:17:26] ENGINE Error in HTTPServer.serve 2024-09-13T14:17:28.421 INFO:journalctl@ceph.mon.a.smithi102.stdout: Traceback (most recent call last): 2024-09-13T14:17:28.421 INFO:journalctl@ceph.mon.a.smithi102.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:17:28.421 INFO:journalctl@ceph.mon.a.smithi102.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:17:28.421 INFO:journalctl@ceph.mon.a.smithi102.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:17:28.421 INFO:journalctl@ceph.mon.a.smithi102.stdout: self._run(expiration_interval) 2024-09-13T14:17:28.421 INFO:journalctl@ceph.mon.a.smithi102.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:17:28.421 INFO:journalctl@ceph.mon.a.smithi102.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:17:28.421 INFO:journalctl@ceph.mon.a.smithi102.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:17:28.421 INFO:journalctl@ceph.mon.a.smithi102.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:17:28.421 INFO:journalctl@ceph.mon.a.smithi102.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:17:28.422 INFO:journalctl@ceph.mon.a.smithi102.stdout: s = self.context.wrap_socket( 2024-09-13T14:17:28.422 INFO:journalctl@ceph.mon.a.smithi102.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:17:28.422 INFO:journalctl@ceph.mon.a.smithi102.stdout: return self.sslsocket_class._create( 2024-09-13T14:17:28.422 INFO:journalctl@ceph.mon.a.smithi102.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:17:28.422 INFO:journalctl@ceph.mon.a.smithi102.stdout: self.do_handshake() 2024-09-13T14:17:28.422 INFO:journalctl@ceph.mon.a.smithi102.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:17:28.422 INFO:journalctl@ceph.mon.a.smithi102.stdout: self._sslobj.do_handshake() 2024-09-13T14:17:28.422 INFO:journalctl@ceph.mon.a.smithi102.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:17:28.422 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:28 smithi102 ceph-mon[110183]: pgmap v4: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:28.422 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:28 smithi102 ceph-mon[110183]: mgrmap e39: y(active, since 2s), standbys: x 2024-09-13T14:17:28.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:28 smithi195 ceph-mon[33205]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:17:28.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:28 smithi195 ceph-mon[33205]: pgmap v3: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:28.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:28 smithi195 ceph-mon[33205]: [13/Sep/2024:14:17:26] ENGINE Serving on http://172.21.15.102:8765 2024-09-13T14:17:28.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:28 smithi195 ceph-mon[33205]: [13/Sep/2024:14:17:26] ENGINE Serving on https://172.21.15.102:7150 2024-09-13T14:17:28.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:28 smithi195 ceph-mon[33205]: [13/Sep/2024:14:17:26] ENGINE Bus STARTED 2024-09-13T14:17:28.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:28 smithi195 ceph-mon[33205]: [13/Sep/2024:14:17:26] ENGINE Error in HTTPServer.serve 2024-09-13T14:17:28.498 INFO:journalctl@ceph.mon.b.smithi195.stdout: Traceback (most recent call last): 2024-09-13T14:17:28.498 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:17:28.498 INFO:journalctl@ceph.mon.b.smithi195.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:17:28.498 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:17:28.498 INFO:journalctl@ceph.mon.b.smithi195.stdout: self._run(expiration_interval) 2024-09-13T14:17:28.498 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:17:28.498 INFO:journalctl@ceph.mon.b.smithi195.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:17:28.498 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:17:28.498 INFO:journalctl@ceph.mon.b.smithi195.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:17:28.498 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:17:28.498 INFO:journalctl@ceph.mon.b.smithi195.stdout: s = self.context.wrap_socket( 2024-09-13T14:17:28.498 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:17:28.499 INFO:journalctl@ceph.mon.b.smithi195.stdout: return self.sslsocket_class._create( 2024-09-13T14:17:28.499 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:17:28.499 INFO:journalctl@ceph.mon.b.smithi195.stdout: self.do_handshake() 2024-09-13T14:17:28.499 INFO:journalctl@ceph.mon.b.smithi195.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:17:28.499 INFO:journalctl@ceph.mon.b.smithi195.stdout: self._sslobj.do_handshake() 2024-09-13T14:17:28.499 INFO:journalctl@ceph.mon.b.smithi195.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:17:28.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:28 smithi195 ceph-mon[33205]: pgmap v4: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:28.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:28 smithi195 ceph-mon[33205]: mgrmap e39: y(active, since 2s), standbys: x 2024-09-13T14:17:30.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:30 smithi102 ceph-mon[40543]: pgmap v5: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:30.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:30 smithi102 ceph-mon[40543]: mgrmap e40: y(active, since 4s), standbys: x 2024-09-13T14:17:30.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:30 smithi102 ceph-mon[40543]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:30.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:30 smithi102 ceph-mon[40543]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:30.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:30 smithi102 ceph-mon[40543]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:30.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:30 smithi102 ceph-mon[40543]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:30.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:30 smithi102 ceph-mon[40543]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:30.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:30 smithi102 ceph-mon[40543]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:30.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:30 smithi102 ceph-mon[110183]: pgmap v5: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:30.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:30 smithi102 ceph-mon[110183]: mgrmap e40: y(active, since 4s), standbys: x 2024-09-13T14:17:30.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:30 smithi102 ceph-mon[110183]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:30.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:30 smithi102 ceph-mon[110183]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:30.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:30 smithi102 ceph-mon[110183]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:30.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:30 smithi102 ceph-mon[110183]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:30.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:30 smithi102 ceph-mon[110183]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:30.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:30 smithi102 ceph-mon[110183]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:30.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:30 smithi195 ceph-mon[33205]: pgmap v5: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:30.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:30 smithi195 ceph-mon[33205]: mgrmap e40: y(active, since 4s), standbys: x 2024-09-13T14:17:30.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:30 smithi195 ceph-mon[33205]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:30.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:30 smithi195 ceph-mon[33205]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:30.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:30 smithi195 ceph-mon[33205]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:30.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:30 smithi195 ceph-mon[33205]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:30.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:30 smithi195 ceph-mon[33205]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:30.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:30 smithi195 ceph-mon[33205]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:31.838 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:31 smithi102 ceph-mon[110183]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:31.838 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:31 smithi102 ceph-mon[110183]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:31.839 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:31 smithi102 ceph-mon[40543]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:31.839 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:31 smithi102 ceph-mon[40543]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:31.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:31 smithi195 ceph-mon[33205]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:31.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:31 smithi195 ceph-mon[33205]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:32.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:32 smithi102 ceph-mon[110183]: pgmap v6: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:32.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:32 smithi102 ceph-mon[40543]: pgmap v6: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:32.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:32 smithi195 ceph-mon[33205]: pgmap v6: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:34.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:34 smithi195 ceph-mon[33205]: pgmap v7: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-13T14:17:34.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:34 smithi195 ceph-mon[33205]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:34.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:34 smithi195 ceph-mon[33205]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:34.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:34 smithi195 ceph-mon[33205]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:34.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:34 smithi195 ceph-mon[33205]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:34.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:34 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:34.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:34 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:17:34.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:34 smithi195 ceph-mon[33205]: Updating smithi102:/etc/ceph/ceph.conf 2024-09-13T14:17:34.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:34 smithi195 ceph-mon[33205]: Updating smithi195:/etc/ceph/ceph.conf 2024-09-13T14:17:34.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:34 smithi195 ceph-mon[33205]: Updating smithi195:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:17:34.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:34 smithi195 ceph-mon[33205]: Updating smithi102:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:17:34.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:34 smithi102 ceph-mon[110183]: pgmap v7: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-13T14:17:34.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:34 smithi102 ceph-mon[110183]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:34.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:34 smithi102 ceph-mon[110183]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:34.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:34 smithi102 ceph-mon[110183]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:34.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:34 smithi102 ceph-mon[110183]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:34.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:34 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:34.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:34 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:17:34.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:34 smithi102 ceph-mon[110183]: Updating smithi102:/etc/ceph/ceph.conf 2024-09-13T14:17:34.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:34 smithi102 ceph-mon[110183]: Updating smithi195:/etc/ceph/ceph.conf 2024-09-13T14:17:34.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:34 smithi102 ceph-mon[110183]: Updating smithi195:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:17:34.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:34 smithi102 ceph-mon[110183]: Updating smithi102:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:17:34.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:34 smithi102 ceph-mon[40543]: pgmap v7: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-13T14:17:34.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:34 smithi102 ceph-mon[40543]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:34.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:34 smithi102 ceph-mon[40543]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:34.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:34 smithi102 ceph-mon[40543]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:34.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:34 smithi102 ceph-mon[40543]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:34.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:34 smithi102 ceph-mon[40543]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:34.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:34 smithi102 ceph-mon[40543]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:17:34.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:34 smithi102 ceph-mon[40543]: Updating smithi102:/etc/ceph/ceph.conf 2024-09-13T14:17:34.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:34 smithi102 ceph-mon[40543]: Updating smithi195:/etc/ceph/ceph.conf 2024-09-13T14:17:34.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:34 smithi102 ceph-mon[40543]: Updating smithi195:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:17:34.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:34 smithi102 ceph-mon[40543]: Updating smithi102:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:17:35.882 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:35 smithi102 ceph-mon[110183]: Updating smithi195:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:17:35.882 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:35 smithi102 ceph-mon[110183]: Updating smithi102:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:17:35.882 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:35 smithi102 ceph-mon[110183]: Updating smithi195:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:17:35.882 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:35 smithi102 ceph-mon[110183]: Updating smithi102:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:17:35.882 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:35 smithi102 ceph-mon[110183]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:35.882 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:35 smithi102 ceph-mon[110183]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:35.882 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:35 smithi102 ceph-mon[110183]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:35.882 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:35 smithi102 ceph-mon[110183]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:35.882 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:35 smithi102 ceph-mon[110183]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:35.883 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:35 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:17:35.883 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:35 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:17:35.883 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:35 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-13T14:17:35.883 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:35 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-09-13T14:17:35.883 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:35 smithi102 ceph-mon[110183]: Upgrade: It appears safe to stop mon.c 2024-09-13T14:17:35.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:35 smithi102 ceph-mon[40543]: Updating smithi195:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:17:35.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:35 smithi102 ceph-mon[40543]: Updating smithi102:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:17:35.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:35 smithi102 ceph-mon[40543]: Updating smithi195:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:17:35.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:35 smithi102 ceph-mon[40543]: Updating smithi102:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:17:35.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:35 smithi102 ceph-mon[40543]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:35.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:35 smithi102 ceph-mon[40543]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:35.884 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:35 smithi102 ceph-mon[40543]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:35.884 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:35 smithi102 ceph-mon[40543]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:35.884 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:35 smithi102 ceph-mon[40543]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:35.884 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:35 smithi102 ceph-mon[40543]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:17:35.884 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:35 smithi102 ceph-mon[40543]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:17:35.884 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:35 smithi102 ceph-mon[40543]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-13T14:17:35.884 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:35 smithi102 ceph-mon[40543]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-09-13T14:17:35.884 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:35 smithi102 ceph-mon[40543]: Upgrade: It appears safe to stop mon.c 2024-09-13T14:17:35.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:35 smithi195 ceph-mon[33205]: Updating smithi195:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:17:35.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:35 smithi195 ceph-mon[33205]: Updating smithi102:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:17:35.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:35 smithi195 ceph-mon[33205]: Updating smithi195:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:17:35.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:35 smithi195 ceph-mon[33205]: Updating smithi102:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:17:35.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:35 smithi195 ceph-mon[33205]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:35.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:35 smithi195 ceph-mon[33205]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:35.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:35 smithi195 ceph-mon[33205]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:35.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:35 smithi195 ceph-mon[33205]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:35.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:35 smithi195 ceph-mon[33205]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:35.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:35 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:17:35.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:35 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:17:35.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:35 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-13T14:17:35.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:35 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-09-13T14:17:35.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:35 smithi195 ceph-mon[33205]: Upgrade: It appears safe to stop mon.c 2024-09-13T14:17:36.607 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:17:36] "GET /metrics HTTP/1.1" 200 31138 "" "Prometheus/2.51.0" 2024-09-13T14:17:36.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:36 smithi102 ceph-mon[110183]: pgmap v8: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-13T14:17:36.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:36 smithi102 ceph-mon[110183]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:36.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:36 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:17:36.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:36 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:17:36.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:36 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:36.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:36 smithi102 ceph-mon[40543]: pgmap v8: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-13T14:17:36.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:36 smithi102 ceph-mon[40543]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:36.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:36 smithi102 ceph-mon[40543]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:17:36.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:36 smithi102 ceph-mon[40543]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:17:36.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:36 smithi102 ceph-mon[40543]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:36.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:36 smithi195 ceph-mon[33205]: pgmap v8: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-13T14:17:36.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:36 smithi195 ceph-mon[33205]: from='mgr.34107 ' entity='mgr.y' 2024-09-13T14:17:36.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:36 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:17:36.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:36 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:17:36.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:36 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:17:37.979 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:37 smithi102 ceph-mon[110183]: Upgrade: Updating mon.c 2024-09-13T14:17:37.979 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:37 smithi102 ceph-mon[110183]: Deploying daemon mon.c on smithi102 2024-09-13T14:17:37.979 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:37 smithi102 ceph-mon[40543]: Upgrade: Updating mon.c 2024-09-13T14:17:37.980 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:37 smithi102 ceph-mon[40543]: Deploying daemon mon.c on smithi102 2024-09-13T14:17:38.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:37 smithi195 ceph-mon[33205]: Upgrade: Updating mon.c 2024-09-13T14:17:38.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:37 smithi195 ceph-mon[33205]: Deploying daemon mon.c on smithi102 2024-09-13T14:17:38.264 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:37 smithi102 systemd[1]: Stopping Ceph mon.c for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T14:17:38.551 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:38 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-c[40520]: 2024-09-13T14:17:38.261+0000 7fe1b407c640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T14:17:38.551 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:38 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-c[40520]: 2024-09-13T14:17:38.261+0000 7fe1b407c640 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-09-13T14:17:38.551 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:38 smithi102 podman[115070]: 2024-09-13 14:17:38.314179602 +0000 UTC m=+0.192615012 container died 6c1fd3099a0575ca34c1bd64c87285020d46fce4f537cf4f32c697cef9e2888b (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-c, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2, RELEASE=reef-2c01458, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T14:17:38.551 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:38 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: [13/Sep/2024:14:17:38] ENGINE Bus STOPPING 2024-09-13T14:17:38.551 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:38 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: [13/Sep/2024:14:17:38] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-13T14:17:38.552 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:38 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: [13/Sep/2024:14:17:38] ENGINE Bus STOPPED 2024-09-13T14:17:38.552 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:38 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: [13/Sep/2024:14:17:38] ENGINE Bus STARTING 2024-09-13T14:17:38.552 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:38 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: [13/Sep/2024:14:17:38] ENGINE Serving on http://:::9283 2024-09-13T14:17:38.552 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:38 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: [13/Sep/2024:14:17:38] ENGINE Bus STARTED 2024-09-13T14:17:38.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:38 smithi102 podman[115070]: 2024-09-13 14:17:38.703213981 +0000 UTC m=+0.581649390 container cleanup 6c1fd3099a0575ca34c1bd64c87285020d46fce4f537cf4f32c697cef9e2888b (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-c, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=reef-2c01458, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T14:17:38.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:38 smithi102 bash[115070]: ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-c 2024-09-13T14:17:39.280 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:38 smithi102 podman[115087]: 2024-09-13 14:17:38.929805523 +0000 UTC m=+0.610791122 container remove 6c1fd3099a0575ca34c1bd64c87285020d46fce4f537cf4f32c697cef9e2888b (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-c, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, RELEASE=reef-2c01458, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0) 2024-09-13T14:17:39.280 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@mon.c.service: Deactivated successfully. 2024-09-13T14:17:39.280 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 systemd[1]: Stopped Ceph mon.c for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:17:39.280 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@mon.c.service: Consumed 7.332s CPU time. 2024-09-13T14:17:39.603 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 systemd[1]: Starting Ceph mon.c for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T14:17:39.603 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 podman[115231]: 2024-09-13 14:17:39.389630455 +0000 UTC m=+0.016062428 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:17:39.603 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 podman[115231]: 2024-09-13 14:17:39.489141791 +0000 UTC m=+0.115573749 container create dba6b3d55296917b6bbfcb26dc0d57810fa907f423fa552fca517d4a49931029 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-c, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.license=GPLv2) 2024-09-13T14:17:40.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 podman[115231]: 2024-09-13 14:17:39.700350014 +0000 UTC m=+0.326781979 container init dba6b3d55296917b6bbfcb26dc0d57810fa907f423fa552fca517d4a49931029 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-c, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=) 2024-09-13T14:17:40.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 podman[115231]: 2024-09-13 14:17:39.705238086 +0000 UTC m=+0.331670044 container start dba6b3d55296917b6bbfcb26dc0d57810fa907f423fa552fca517d4a49931029 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-c, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-09-13T14:17:40.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: set uid:gid to 167:167 (ceph:ceph) 2024-09-13T14:17:40.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev), process ceph-mon, pid 7 2024-09-13T14:17:40.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: pidfile_write: ignore empty --pid-file 2024-09-13T14:17:40.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: load: jerasure load: lrc 2024-09-13T14:17:40.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: RocksDB version: 7.9.2 2024-09-13T14:17:40.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Git sha 0 2024-09-13T14:17:40.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Compile date 2024-09-12 20:46:14 2024-09-13T14:17:40.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: DB SUMMARY 2024-09-13T14:17:40.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: DB Session ID: M04PSOI7JQ3V3OCKBZGL 2024-09-13T14:17:40.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: CURRENT file: CURRENT 2024-09-13T14:17:40.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: IDENTITY file: IDENTITY 2024-09-13T14:17:40.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: MANIFEST file: MANIFEST-000010 size: 1412 Bytes 2024-09-13T14:17:40.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000027.sst 2024-09-13T14:17:40.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000025.log size: 5222545 ; 2024-09-13T14:17:40.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.error_if_exists: 0 2024-09-13T14:17:40.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.create_if_missing: 0 2024-09-13T14:17:40.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.paranoid_checks: 1 2024-09-13T14:17:40.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-13T14:17:40.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-13T14:17:40.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-13T14:17:40.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.env: 0x5580eed1c160 2024-09-13T14:17:40.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.fs: PosixFileSystem 2024-09-13T14:17:40.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.info_log: 0x5580ef935880 2024-09-13T14:17:40.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.max_file_opening_threads: 16 2024-09-13T14:17:40.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.statistics: (nil) 2024-09-13T14:17:40.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.use_fsync: 0 2024-09-13T14:17:40.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.max_log_file_size: 0 2024-09-13T14:17:40.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-13T14:17:40.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-13T14:17:40.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.keep_log_file_num: 1000 2024-09-13T14:17:40.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.recycle_log_file_num: 0 2024-09-13T14:17:40.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.allow_fallocate: 1 2024-09-13T14:17:40.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.allow_mmap_reads: 0 2024-09-13T14:17:40.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.allow_mmap_writes: 0 2024-09-13T14:17:40.172 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.use_direct_reads: 0 2024-09-13T14:17:40.172 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-13T14:17:40.172 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.create_missing_column_families: 0 2024-09-13T14:17:40.172 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.db_log_dir: 2024-09-13T14:17:40.172 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.wal_dir: 2024-09-13T14:17:40.172 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-13T14:17:40.172 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-13T14:17:40.172 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-13T14:17:40.172 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-13T14:17:40.172 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-13T14:17:40.172 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-13T14:17:40.172 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.advise_random_on_open: 1 2024-09-13T14:17:40.172 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.db_write_buffer_size: 0 2024-09-13T14:17:40.173 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.write_buffer_manager: 0x5580ef938960 2024-09-13T14:17:40.173 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-13T14:17:40.173 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-13T14:17:40.173 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-13T14:17:40.173 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.rate_limiter: (nil) 2024-09-13T14:17:40.173 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-13T14:17:40.173 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.wal_recovery_mode: 2 2024-09-13T14:17:40.173 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.enable_thread_tracking: 0 2024-09-13T14:17:40.173 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.enable_pipelined_write: 0 2024-09-13T14:17:40.173 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.unordered_write: 0 2024-09-13T14:17:40.173 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-13T14:17:40.173 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-13T14:17:40.173 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-13T14:17:40.173 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-13T14:17:40.174 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.row_cache: None 2024-09-13T14:17:40.174 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.wal_filter: None 2024-09-13T14:17:40.175 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-13T14:17:40.175 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.allow_ingest_behind: 0 2024-09-13T14:17:40.175 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.two_write_queues: 0 2024-09-13T14:17:40.175 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.manual_wal_flush: 0 2024-09-13T14:17:40.175 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.wal_compression: 0 2024-09-13T14:17:40.175 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.atomic_flush: 0 2024-09-13T14:17:40.175 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-13T14:17:40.175 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-13T14:17:40.176 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-13T14:17:40.176 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.log_readahead_size: 0 2024-09-13T14:17:40.176 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-13T14:17:40.176 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.best_efforts_recovery: 0 2024-09-13T14:17:40.176 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-13T14:17:40.176 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-13T14:17:40.176 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.allow_data_in_errors: 0 2024-09-13T14:17:40.176 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.db_host_id: __hostname__ 2024-09-13T14:17:40.176 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-13T14:17:40.176 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.max_background_jobs: 2 2024-09-13T14:17:40.176 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.max_background_compactions: -1 2024-09-13T14:17:40.176 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.max_subcompactions: 1 2024-09-13T14:17:40.176 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-13T14:17:40.177 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-13T14:17:40.177 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-13T14:17:40.177 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.max_total_wal_size: 0 2024-09-13T14:17:40.177 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-13T14:17:40.177 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-13T14:17:40.177 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-13T14:17:40.177 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-13T14:17:40.177 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.max_open_files: -1 2024-09-13T14:17:40.177 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.bytes_per_sync: 0 2024-09-13T14:17:40.177 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-13T14:17:40.177 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-13T14:17:40.177 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.compaction_readahead_size: 0 2024-09-13T14:17:40.177 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.max_background_flushes: -1 2024-09-13T14:17:40.178 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Compression algorithms supported: 2024-09-13T14:17:40.178 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: kZSTD supported: 0 2024-09-13T14:17:40.178 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: kXpressCompression supported: 0 2024-09-13T14:17:40.178 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: kBZip2Compression supported: 0 2024-09-13T14:17:40.178 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-13T14:17:40.178 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: kLZ4Compression supported: 1 2024-09-13T14:17:40.178 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: kZlibCompression supported: 1 2024-09-13T14:17:40.178 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: kLZ4HCCompression supported: 1 2024-09-13T14:17:40.178 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: kSnappyCompression supported: 1 2024-09-13T14:17:40.178 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-13T14:17:40.178 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-13T14:17:40.178 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000010 2024-09-13T14:17:40.178 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-13T14:17:40.178 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-13T14:17:40.179 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.merge_operator: 2024-09-13T14:17:40.179 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.compaction_filter: None 2024-09-13T14:17:40.179 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.compaction_filter_factory: None 2024-09-13T14:17:40.179 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.sst_partitioner_factory: None 2024-09-13T14:17:40.179 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-13T14:17:40.179 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-13T14:17:40.179 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5580ef9354e0) 2024-09-13T14:17:40.179 INFO:journalctl@ceph.mon.c.smithi102.stdout: cache_index_and_filter_blocks: 1 2024-09-13T14:17:40.179 INFO:journalctl@ceph.mon.c.smithi102.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-13T14:17:40.179 INFO:journalctl@ceph.mon.c.smithi102.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-13T14:17:40.179 INFO:journalctl@ceph.mon.c.smithi102.stdout: pin_top_level_index_and_filter: 1 2024-09-13T14:17:40.179 INFO:journalctl@ceph.mon.c.smithi102.stdout: index_type: 0 2024-09-13T14:17:40.179 INFO:journalctl@ceph.mon.c.smithi102.stdout: data_block_index_type: 0 2024-09-13T14:17:40.180 INFO:journalctl@ceph.mon.c.smithi102.stdout: index_shortening: 1 2024-09-13T14:17:40.180 INFO:journalctl@ceph.mon.c.smithi102.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-13T14:17:40.180 INFO:journalctl@ceph.mon.c.smithi102.stdout: checksum: 4 2024-09-13T14:17:40.180 INFO:journalctl@ceph.mon.c.smithi102.stdout: no_block_cache: 0 2024-09-13T14:17:40.180 INFO:journalctl@ceph.mon.c.smithi102.stdout: block_cache: 0x5580ef928850 2024-09-13T14:17:40.180 INFO:journalctl@ceph.mon.c.smithi102.stdout: block_cache_name: BinnedLRUCache 2024-09-13T14:17:40.180 INFO:journalctl@ceph.mon.c.smithi102.stdout: block_cache_options: 2024-09-13T14:17:40.180 INFO:journalctl@ceph.mon.c.smithi102.stdout: capacity : 536870912 2024-09-13T14:17:40.180 INFO:journalctl@ceph.mon.c.smithi102.stdout: num_shard_bits : 4 2024-09-13T14:17:40.180 INFO:journalctl@ceph.mon.c.smithi102.stdout: strict_capacity_limit : 0 2024-09-13T14:17:40.180 INFO:journalctl@ceph.mon.c.smithi102.stdout: high_pri_pool_ratio: 0.000 2024-09-13T14:17:40.180 INFO:journalctl@ceph.mon.c.smithi102.stdout: block_cache_compressed: (nil) 2024-09-13T14:17:40.180 INFO:journalctl@ceph.mon.c.smithi102.stdout: persistent_cache: (nil) 2024-09-13T14:17:40.181 INFO:journalctl@ceph.mon.c.smithi102.stdout: block_size: 4096 2024-09-13T14:17:40.181 INFO:journalctl@ceph.mon.c.smithi102.stdout: block_size_deviation: 10 2024-09-13T14:17:40.181 INFO:journalctl@ceph.mon.c.smithi102.stdout: block_restart_interval: 16 2024-09-13T14:17:40.181 INFO:journalctl@ceph.mon.c.smithi102.stdout: index_block_restart_interval: 1 2024-09-13T14:17:40.181 INFO:journalctl@ceph.mon.c.smithi102.stdout: metadata_block_size: 4096 2024-09-13T14:17:40.181 INFO:journalctl@ceph.mon.c.smithi102.stdout: partition_filters: 0 2024-09-13T14:17:40.181 INFO:journalctl@ceph.mon.c.smithi102.stdout: use_delta_encoding: 1 2024-09-13T14:17:40.181 INFO:journalctl@ceph.mon.c.smithi102.stdout: filter_policy: bloomfilter 2024-09-13T14:17:40.181 INFO:journalctl@ceph.mon.c.smithi102.stdout: whole_key_filtering: 1 2024-09-13T14:17:40.181 INFO:journalctl@ceph.mon.c.smithi102.stdout: verify_compression: 0 2024-09-13T14:17:40.181 INFO:journalctl@ceph.mon.c.smithi102.stdout: read_amp_bytes_per_bit: 0 2024-09-13T14:17:40.181 INFO:journalctl@ceph.mon.c.smithi102.stdout: format_version: 5 2024-09-13T14:17:40.182 INFO:journalctl@ceph.mon.c.smithi102.stdout: enable_index_compression: 1 2024-09-13T14:17:40.182 INFO:journalctl@ceph.mon.c.smithi102.stdout: block_align: 0 2024-09-13T14:17:40.182 INFO:journalctl@ceph.mon.c.smithi102.stdout: max_auto_readahead_size: 262144 2024-09-13T14:17:40.182 INFO:journalctl@ceph.mon.c.smithi102.stdout: prepopulate_block_cache: 0 2024-09-13T14:17:40.182 INFO:journalctl@ceph.mon.c.smithi102.stdout: initial_auto_readahead_size: 8192 2024-09-13T14:17:40.182 INFO:journalctl@ceph.mon.c.smithi102.stdout: num_file_reads_for_auto_readahead: 2 2024-09-13T14:17:40.182 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.write_buffer_size: 33554432 2024-09-13T14:17:40.182 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.max_write_buffer_number: 2 2024-09-13T14:17:40.182 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.compression: NoCompression 2024-09-13T14:17:40.182 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.bottommost_compression: Disabled 2024-09-13T14:17:40.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.prefix_extractor: nullptr 2024-09-13T14:17:40.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-13T14:17:40.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.num_levels: 7 2024-09-13T14:17:40.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-13T14:17:40.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-13T14:17:40.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-13T14:17:40.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-13T14:17:40.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-13T14:17:40.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-13T14:17:40.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-13T14:17:40.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-13T14:17:40.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-13T14:17:40.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-13T14:17:40.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-13T14:17:40.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-13T14:17:40.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-13T14:17:40.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.compression_opts.level: 32767 2024-09-13T14:17:40.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.compression_opts.strategy: 0 2024-09-13T14:17:40.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-13T14:17:40.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-13T14:17:40.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-13T14:17:40.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-13T14:17:40.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.compression_opts.enabled: false 2024-09-13T14:17:40.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-13T14:17:40.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-13T14:17:40.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-13T14:17:40.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-13T14:17:40.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.target_file_size_base: 67108864 2024-09-13T14:17:40.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-13T14:17:40.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-13T14:17:40.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-13T14:17:40.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-13T14:17:40.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-13T14:17:40.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-13T14:17:40.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-13T14:17:40.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-13T14:17:40.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-13T14:17:40.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-13T14:17:40.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-13T14:17:40.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-13T14:17:40.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-13T14:17:40.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-13T14:17:40.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.arena_block_size: 1048576 2024-09-13T14:17:40.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-13T14:17:40.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-13T14:17:40.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.disable_auto_compactions: 0 2024-09-13T14:17:40.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-13T14:17:40.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-13T14:17:40.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-13T14:17:40.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-13T14:17:40.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-13T14:17:40.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-13T14:17:40.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-13T14:17:40.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-13T14:17:40.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-13T14:17:40.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-13T14:17:40.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-13T14:17:40.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.inplace_update_support: 0 2024-09-13T14:17:40.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-13T14:17:40.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-13T14:17:40.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-13T14:17:40.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-13T14:17:40.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.bloom_locality: 0 2024-09-13T14:17:40.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.max_successive_merges: 0 2024-09-13T14:17:40.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-13T14:17:40.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.paranoid_file_checks: 0 2024-09-13T14:17:40.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.force_consistency_checks: 1 2024-09-13T14:17:40.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.report_bg_io_stats: 0 2024-09-13T14:17:40.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.ttl: 2592000 2024-09-13T14:17:40.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-13T14:17:40.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-13T14:17:40.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-13T14:17:40.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.enable_blob_files: false 2024-09-13T14:17:40.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.min_blob_size: 0 2024-09-13T14:17:40.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.blob_file_size: 268435456 2024-09-13T14:17:40.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-13T14:17:40.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-13T14:17:40.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-13T14:17:40.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-13T14:17:40.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-13T14:17:40.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.blob_file_starting_level: 0 2024-09-13T14:17:40.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-13T14:17:40.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-13T14:17:40.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 29, last_sequence is 12958, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 25 2024-09-13T14:17:40.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-09-13T14:17:40.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 83a210d1-0432-49a0-a7ce-d40c676b3e42 2024-09-13T14:17:40.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726237059731991, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-09-13T14:17:40.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-09-13T14:17:40.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726237059745385, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 3261966, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12963, "largest_seqno": 14706, "table_properties": {"data_size": 3254224, "index_size": 4776, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1925, "raw_key_size": 18562, "raw_average_key_size": 24, "raw_value_size": 3237906, "raw_average_value_size": 4238, "num_data_blocks": 217, "num_entries": 764, "num_filter_entries": 764, "num_deletions": 8, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726237059, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "83a210d1-0432-49a0-a7ce-d40c676b3e42", "db_session_id": "M04PSOI7JQ3V3OCKBZGL", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-09-13T14:17:40.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726237059745549, "job": 1, "event": "recovery_finished"} 2024-09-13T14:17:40.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-09-13T14:17:40.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-13T14:17:40.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-13T14:17:40.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5580ef958e00 2024-09-13T14:17:40.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: DB pointer 0x5580efa58000 2024-09-13T14:17:40.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-13T14:17:40.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-13T14:17:40.190 INFO:journalctl@ceph.mon.c.smithi102.stdout: ** DB Stats ** 2024-09-13T14:17:40.190 INFO:journalctl@ceph.mon.c.smithi102.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T14:17:40.190 INFO:journalctl@ceph.mon.c.smithi102.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-13T14:17:40.190 INFO:journalctl@ceph.mon.c.smithi102.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-13T14:17:40.190 INFO:journalctl@ceph.mon.c.smithi102.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T14:17:40.190 INFO:journalctl@ceph.mon.c.smithi102.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-13T14:17:40.190 INFO:journalctl@ceph.mon.c.smithi102.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-13T14:17:40.190 INFO:journalctl@ceph.mon.c.smithi102.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T14:17:40.190 INFO:journalctl@ceph.mon.c.smithi102.stdout: 2024-09-13T14:17:40.190 INFO:journalctl@ceph.mon.c.smithi102.stdout: ** Compaction Stats [default] ** 2024-09-13T14:17:40.191 INFO:journalctl@ceph.mon.c.smithi102.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-13T14:17:40.191 INFO:journalctl@ceph.mon.c.smithi102.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-13T14:17:40.191 INFO:journalctl@ceph.mon.c.smithi102.stdout: L0 1/0 3.11 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 704.5 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-09-13T14:17:40.191 INFO:journalctl@ceph.mon.c.smithi102.stdout: L6 1/0 7.96 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-13T14:17:40.191 INFO:journalctl@ceph.mon.c.smithi102.stdout: Sum 2/0 11.07 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 704.5 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-09-13T14:17:40.191 INFO:journalctl@ceph.mon.c.smithi102.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 704.5 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-09-13T14:17:40.191 INFO:journalctl@ceph.mon.c.smithi102.stdout: 2024-09-13T14:17:40.191 INFO:journalctl@ceph.mon.c.smithi102.stdout: ** Compaction Stats [default] ** 2024-09-13T14:17:40.191 INFO:journalctl@ceph.mon.c.smithi102.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-13T14:17:40.191 INFO:journalctl@ceph.mon.c.smithi102.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-13T14:17:40.191 INFO:journalctl@ceph.mon.c.smithi102.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 704.5 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-09-13T14:17:40.191 INFO:journalctl@ceph.mon.c.smithi102.stdout: 2024-09-13T14:17:40.191 INFO:journalctl@ceph.mon.c.smithi102.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-13T14:17:40.191 INFO:journalctl@ceph.mon.c.smithi102.stdout: 2024-09-13T14:17:40.192 INFO:journalctl@ceph.mon.c.smithi102.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T14:17:40.192 INFO:journalctl@ceph.mon.c.smithi102.stdout: Flush(GB): cumulative 0.003, interval 0.003 2024-09-13T14:17:40.192 INFO:journalctl@ceph.mon.c.smithi102.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-13T14:17:40.192 INFO:journalctl@ceph.mon.c.smithi102.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-13T14:17:40.192 INFO:journalctl@ceph.mon.c.smithi102.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-13T14:17:40.192 INFO:journalctl@ceph.mon.c.smithi102.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-13T14:17:40.192 INFO:journalctl@ceph.mon.c.smithi102.stdout: Cumulative compaction: 0.00 GB write, 188.07 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T14:17:40.192 INFO:journalctl@ceph.mon.c.smithi102.stdout: Interval compaction: 0.00 GB write, 188.07 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T14:17:40.192 INFO:journalctl@ceph.mon.c.smithi102.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-13T14:17:40.192 INFO:journalctl@ceph.mon.c.smithi102.stdout: Block cache BinnedLRUCache@0x5580ef928850#7 capacity: 512.00 MB usage: 7.11 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-09-13T14:17:40.192 INFO:journalctl@ceph.mon.c.smithi102.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,2.03 KB,0.00038743%) IndexBlock(1,5.08 KB,0.000968575%) Misc(1,0.00 KB,0%) 2024-09-13T14:17:40.192 INFO:journalctl@ceph.mon.c.smithi102.stdout: 2024-09-13T14:17:40.193 INFO:journalctl@ceph.mon.c.smithi102.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-13T14:17:40.193 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: starting mon.c rank 2 at public addrs [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] at bind addrs [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 08cee502-71d9-11ef-bceb-c7b262605968 2024-09-13T14:17:40.193 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: mon.c@-1(???) e3 preinit fsid 08cee502-71d9-11ef-bceb-c7b262605968 2024-09-13T14:17:40.193 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: mon.c@-1(???).mds e9 new map 2024-09-13T14:17:40.193 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: mon.c@-1(???).mds e9 print_map 2024-09-13T14:17:40.193 INFO:journalctl@ceph.mon.c.smithi102.stdout: e9 2024-09-13T14:17:40.193 INFO:journalctl@ceph.mon.c.smithi102.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-09-13T14:17:40.193 INFO:journalctl@ceph.mon.c.smithi102.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-13T14:17:40.193 INFO:journalctl@ceph.mon.c.smithi102.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-13T14:17:40.193 INFO:journalctl@ceph.mon.c.smithi102.stdout: legacy client fscid: 1 2024-09-13T14:17:40.193 INFO:journalctl@ceph.mon.c.smithi102.stdout: 2024-09-13T14:17:40.193 INFO:journalctl@ceph.mon.c.smithi102.stdout: Filesystem 'foo' (1) 2024-09-13T14:17:40.193 INFO:journalctl@ceph.mon.c.smithi102.stdout: fs_name foo 2024-09-13T14:17:40.193 INFO:journalctl@ceph.mon.c.smithi102.stdout: epoch 8 2024-09-13T14:17:40.194 INFO:journalctl@ceph.mon.c.smithi102.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-09-13T14:17:40.194 INFO:journalctl@ceph.mon.c.smithi102.stdout: created 2024-09-13T14:11:29.232738+0000 2024-09-13T14:17:40.194 INFO:journalctl@ceph.mon.c.smithi102.stdout: modified 2024-09-13T14:11:41.325298+0000 2024-09-13T14:17:40.194 INFO:journalctl@ceph.mon.c.smithi102.stdout: tableserver 0 2024-09-13T14:17:40.194 INFO:journalctl@ceph.mon.c.smithi102.stdout: root 0 2024-09-13T14:17:40.194 INFO:journalctl@ceph.mon.c.smithi102.stdout: session_timeout 60 2024-09-13T14:17:40.194 INFO:journalctl@ceph.mon.c.smithi102.stdout: session_autoclose 300 2024-09-13T14:17:40.194 INFO:journalctl@ceph.mon.c.smithi102.stdout: max_file_size 1099511627776 2024-09-13T14:17:40.194 INFO:journalctl@ceph.mon.c.smithi102.stdout: max_xattr_size 65536 2024-09-13T14:17:40.194 INFO:journalctl@ceph.mon.c.smithi102.stdout: required_client_features {} 2024-09-13T14:17:40.194 INFO:journalctl@ceph.mon.c.smithi102.stdout: last_failure 0 2024-09-13T14:17:40.194 INFO:journalctl@ceph.mon.c.smithi102.stdout: last_failure_osd_epoch 0 2024-09-13T14:17:40.194 INFO:journalctl@ceph.mon.c.smithi102.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-13T14:17:40.194 INFO:journalctl@ceph.mon.c.smithi102.stdout: max_mds 1 2024-09-13T14:17:40.194 INFO:journalctl@ceph.mon.c.smithi102.stdout: in 0 2024-09-13T14:17:40.195 INFO:journalctl@ceph.mon.c.smithi102.stdout: up {0=24521} 2024-09-13T14:17:40.195 INFO:journalctl@ceph.mon.c.smithi102.stdout: failed 2024-09-13T14:17:40.195 INFO:journalctl@ceph.mon.c.smithi102.stdout: damaged 2024-09-13T14:17:40.195 INFO:journalctl@ceph.mon.c.smithi102.stdout: stopped 2024-09-13T14:17:40.195 INFO:journalctl@ceph.mon.c.smithi102.stdout: data_pools [4] 2024-09-13T14:17:40.195 INFO:journalctl@ceph.mon.c.smithi102.stdout: metadata_pool 3 2024-09-13T14:17:40.195 INFO:journalctl@ceph.mon.c.smithi102.stdout: inline_data disabled 2024-09-13T14:17:40.195 INFO:journalctl@ceph.mon.c.smithi102.stdout: balancer 2024-09-13T14:17:40.195 INFO:journalctl@ceph.mon.c.smithi102.stdout: bal_rank_mask -1 2024-09-13T14:17:40.195 INFO:journalctl@ceph.mon.c.smithi102.stdout: standby_count_wanted 1 2024-09-13T14:17:40.195 INFO:journalctl@ceph.mon.c.smithi102.stdout: qdb_cluster leader: 0 members: 2024-09-13T14:17:40.195 INFO:journalctl@ceph.mon.c.smithi102.stdout: [mds.foo.smithi102.njetxr{0:24521} state up:active seq 3 join_fscid=1 addr [v2:172.21.15.102:6834/54188465,v1:172.21.15.102:6835/54188465] compat {c=[1],r=[1],i=[7ff]}] 2024-09-13T14:17:40.195 INFO:journalctl@ceph.mon.c.smithi102.stdout: 2024-09-13T14:17:40.196 INFO:journalctl@ceph.mon.c.smithi102.stdout: 2024-09-13T14:17:40.196 INFO:journalctl@ceph.mon.c.smithi102.stdout: Standby daemons: 2024-09-13T14:17:40.196 INFO:journalctl@ceph.mon.c.smithi102.stdout: 2024-09-13T14:17:40.196 INFO:journalctl@ceph.mon.c.smithi102.stdout: [mds.foo.smithi195.yqzrxi{-1:24442} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.195:6832/2017468914,v1:172.21.15.195:6833/2017468914] compat {c=[1],r=[1],i=[7ff]}] 2024-09-13T14:17:40.196 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: mon.c@-1(???).osd e90 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-13T14:17:40.196 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: mon.c@-1(???).osd e90 crush map has features 432629239337189376, adjusting msgr requires 2024-09-13T14:17:40.196 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: mon.c@-1(???).osd e90 crush map has features 432629239337189376, adjusting msgr requires 2024-09-13T14:17:40.196 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: mon.c@-1(???).osd e90 crush map has features 432629239337189376, adjusting msgr requires 2024-09-13T14:17:40.196 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 ceph-mon[115269]: mon.c@-1(???).paxosservice(auth 1..20) refresh upgraded, format 0 -> 3 2024-09-13T14:17:40.196 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 bash[115231]: dba6b3d55296917b6bbfcb26dc0d57810fa907f423fa552fca517d4a49931029 2024-09-13T14:17:40.196 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:39 smithi102 systemd[1]: Started Ceph mon.c for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:17:45.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:44 smithi195 ceph-mon[33205]: pgmap v9: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-13T14:17:45.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:44 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-13T14:17:45.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:44 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T14:17:45.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:44 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T14:17:45.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:44 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-13T14:17:45.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:44 smithi102 ceph-mon[110183]: pgmap v9: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-13T14:17:45.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:44 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-13T14:17:45.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:44 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T14:17:45.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:44 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T14:17:45.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:44 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-13T14:17:45.724 INFO:teuthology.orchestra.run.smithi102.stdout:true 2024-09-13T14:17:46.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:45 smithi195 ceph-mon[33205]: mon.c calling monitor election 2024-09-13T14:17:46.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:45 smithi195 ceph-mon[33205]: mon.b calling monitor election 2024-09-13T14:17:46.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:45 smithi195 ceph-mon[33205]: mon.a calling monitor election 2024-09-13T14:17:46.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:45 smithi195 ceph-mon[33205]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T14:17:46.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:45 smithi195 ceph-mon[33205]: monmap epoch 3 2024-09-13T14:17:46.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:45 smithi195 ceph-mon[33205]: fsid 08cee502-71d9-11ef-bceb-c7b262605968 2024-09-13T14:17:46.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:45 smithi195 ceph-mon[33205]: last_changed 2024-09-13T14:06:18.744278+0000 2024-09-13T14:17:46.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:45 smithi195 ceph-mon[33205]: created 2024-09-13T14:04:45.172288+0000 2024-09-13T14:17:46.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:45 smithi195 ceph-mon[33205]: min_mon_release 18 (reef) 2024-09-13T14:17:46.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:45 smithi195 ceph-mon[33205]: election_strategy: 1 2024-09-13T14:17:46.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:45 smithi195 ceph-mon[33205]: 0: [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon.a 2024-09-13T14:17:46.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:45 smithi195 ceph-mon[33205]: 1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2024-09-13T14:17:46.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:45 smithi195 ceph-mon[33205]: 2: [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] mon.c 2024-09-13T14:17:46.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:45 smithi195 ceph-mon[33205]: fsmap foo:1 {0=foo.smithi102.njetxr=up:active} 1 up:standby 2024-09-13T14:17:46.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:45 smithi195 ceph-mon[33205]: osdmap e90: 8 total, 8 up, 8 in 2024-09-13T14:17:46.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:45 smithi195 ceph-mon[33205]: mgrmap e40: y(active, since 20s), standbys: x 2024-09-13T14:17:46.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:45 smithi195 ceph-mon[33205]: overall HEALTH_OK 2024-09-13T14:17:46.337 INFO:teuthology.orchestra.run.smithi102.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-13T14:17:46.337 INFO:teuthology.orchestra.run.smithi102.stdout:alertmanager.a smithi102 *:9093,9094 running (95s) 16s ago 7m 19.2M - 0.27.0 11f11916f8cd 0322b99ec767 2024-09-13T14:17:46.337 INFO:teuthology.orchestra.run.smithi102.stdout:grafana.a smithi195 *:3000 running (74s) 16s ago 7m 79.3M - c8b91775d855 152e89bad2c8 2024-09-13T14:17:46.337 INFO:teuthology.orchestra.run.smithi102.stdout:mds.foo.smithi102.njetxr smithi102 running (6m) 16s ago 6m 20.2M - 18.2.4-843-g2c01458a c129f5a2fd4e c9211cc725ed 2024-09-13T14:17:46.337 INFO:teuthology.orchestra.run.smithi102.stdout:mds.foo.smithi195.yqzrxi smithi195 running (6m) 16s ago 6m 17.7M - 18.2.4-843-g2c01458a c129f5a2fd4e eea37a7230fd 2024-09-13T14:17:46.337 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.x smithi195 *:8443,9283,8765 running (2m) 16s ago 11m 451M - 19.3.0-4697-gd58f7306 fec07016a2b0 4a23b8f0ea8b 2024-09-13T14:17:46.337 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.y smithi102 *:8443,9283,8765 running (3m) 16s ago 12m 486M - 19.3.0-4697-gd58f7306 fec07016a2b0 6b12b28dcde1 2024-09-13T14:17:46.338 INFO:teuthology.orchestra.run.smithi102.stdout:mon.a smithi102 running (28s) 16s ago 12m 28.3M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 f3234b7bbdde 2024-09-13T14:17:46.338 INFO:teuthology.orchestra.run.smithi102.stdout:mon.b smithi195 running (11m) 16s ago 11m 49.4M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e 608ec4847676 2024-09-13T14:17:46.338 INFO:teuthology.orchestra.run.smithi102.stdout:mon.c smithi102 starting - - - 2048M 2024-09-13T14:17:46.338 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.a smithi102 *:9100 running (2m) 16s ago 6m 12.4M - 1.7.0 72c9c2088986 0c92d0b39468 2024-09-13T14:17:46.338 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.b smithi195 *:9100 running (2m) 16s ago 6m 10.4M - 1.7.0 72c9c2088986 fde867a7dbf9 2024-09-13T14:17:46.338 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0 smithi102 running (10m) 16s ago 10m 60.2M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e b97482fb8cf9 2024-09-13T14:17:46.338 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1 smithi102 running (10m) 16s ago 10m 51.7M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 8a67356be7a2 2024-09-13T14:17:46.338 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2 smithi102 running (10m) 16s ago 10m 49.7M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 1800986ac838 2024-09-13T14:17:46.338 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3 smithi102 running (9m) 16s ago 9m 56.4M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 968246da9da2 2024-09-13T14:17:46.338 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4 smithi195 running (9m) 16s ago 9m 48.6M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 515307701d73 2024-09-13T14:17:46.338 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5 smithi195 running (9m) 16s ago 9m 52.8M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 176ba2b3bd59 2024-09-13T14:17:46.338 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6 smithi195 running (8m) 16s ago 8m 55.4M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 9cd070a6dd3d 2024-09-13T14:17:46.338 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7 smithi195 running (8m) 16s ago 8m 52.6M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e a6b698c659ca 2024-09-13T14:17:46.338 INFO:teuthology.orchestra.run.smithi102.stdout:prometheus.a smithi195 *:9095 running (115s) 16s ago 7m 41.6M - 2.51.0 1d3b7f56885b d131e53f1ef9 2024-09-13T14:17:46.339 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:45 smithi102 ceph-mon[115269]: pgmap v9: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-13T14:17:46.339 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:45 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-13T14:17:46.339 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:45 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T14:17:46.339 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:45 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T14:17:46.339 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:45 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-13T14:17:46.339 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:45 smithi102 ceph-mon[115269]: mon.c calling monitor election 2024-09-13T14:17:46.339 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:45 smithi102 ceph-mon[115269]: mon.b calling monitor election 2024-09-13T14:17:46.339 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:45 smithi102 ceph-mon[115269]: mon.a calling monitor election 2024-09-13T14:17:46.339 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:45 smithi102 ceph-mon[115269]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T14:17:46.339 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:45 smithi102 ceph-mon[115269]: monmap epoch 3 2024-09-13T14:17:46.340 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:45 smithi102 ceph-mon[115269]: fsid 08cee502-71d9-11ef-bceb-c7b262605968 2024-09-13T14:17:46.340 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:45 smithi102 ceph-mon[115269]: last_changed 2024-09-13T14:06:18.744278+0000 2024-09-13T14:17:46.340 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:45 smithi102 ceph-mon[115269]: created 2024-09-13T14:04:45.172288+0000 2024-09-13T14:17:46.340 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:45 smithi102 ceph-mon[115269]: min_mon_release 18 (reef) 2024-09-13T14:17:46.340 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:45 smithi102 ceph-mon[115269]: election_strategy: 1 2024-09-13T14:17:46.340 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:45 smithi102 ceph-mon[115269]: 0: [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon.a 2024-09-13T14:17:46.340 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:45 smithi102 ceph-mon[115269]: 1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2024-09-13T14:17:46.340 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:45 smithi102 ceph-mon[115269]: 2: [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] mon.c 2024-09-13T14:17:46.340 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:45 smithi102 ceph-mon[115269]: fsmap foo:1 {0=foo.smithi102.njetxr=up:active} 1 up:standby 2024-09-13T14:17:46.340 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:45 smithi102 ceph-mon[115269]: osdmap e90: 8 total, 8 up, 8 in 2024-09-13T14:17:46.340 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:45 smithi102 ceph-mon[115269]: mgrmap e40: y(active, since 20s), standbys: x 2024-09-13T14:17:46.340 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:45 smithi102 ceph-mon[115269]: overall HEALTH_OK 2024-09-13T14:17:46.341 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:45 smithi102 ceph-mon[110183]: mon.c calling monitor election 2024-09-13T14:17:46.341 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:45 smithi102 ceph-mon[110183]: mon.b calling monitor election 2024-09-13T14:17:46.341 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:45 smithi102 ceph-mon[110183]: mon.a calling monitor election 2024-09-13T14:17:46.341 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:45 smithi102 ceph-mon[110183]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T14:17:46.341 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:45 smithi102 ceph-mon[110183]: monmap epoch 3 2024-09-13T14:17:46.341 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:45 smithi102 ceph-mon[110183]: fsid 08cee502-71d9-11ef-bceb-c7b262605968 2024-09-13T14:17:46.341 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:45 smithi102 ceph-mon[110183]: last_changed 2024-09-13T14:06:18.744278+0000 2024-09-13T14:17:46.341 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:45 smithi102 ceph-mon[110183]: created 2024-09-13T14:04:45.172288+0000 2024-09-13T14:17:46.341 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:45 smithi102 ceph-mon[110183]: min_mon_release 18 (reef) 2024-09-13T14:17:46.341 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:45 smithi102 ceph-mon[110183]: election_strategy: 1 2024-09-13T14:17:46.341 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:45 smithi102 ceph-mon[110183]: 0: [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon.a 2024-09-13T14:17:46.341 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:45 smithi102 ceph-mon[110183]: 1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2024-09-13T14:17:46.341 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:45 smithi102 ceph-mon[110183]: 2: [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] mon.c 2024-09-13T14:17:46.342 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:45 smithi102 ceph-mon[110183]: fsmap foo:1 {0=foo.smithi102.njetxr=up:active} 1 up:standby 2024-09-13T14:17:46.342 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:45 smithi102 ceph-mon[110183]: osdmap e90: 8 total, 8 up, 8 in 2024-09-13T14:17:46.342 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:45 smithi102 ceph-mon[110183]: mgrmap e40: y(active, since 20s), standbys: x 2024-09-13T14:17:46.342 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:45 smithi102 ceph-mon[110183]: overall HEALTH_OK 2024-09-13T14:17:46.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:17:46] "GET /metrics HTTP/1.1" 200 33819 "" "Prometheus/2.51.0" 2024-09-13T14:17:46.690 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:17:46.690 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": { 2024-09-13T14:17:46.690 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 1, 2024-09-13T14:17:46.691 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 2 2024-09-13T14:17:46.691 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:17:46.691 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": { 2024-09-13T14:17:46.691 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 2 2024-09-13T14:17:46.691 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:17:46.691 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": { 2024-09-13T14:17:46.691 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 8 2024-09-13T14:17:46.691 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:17:46.691 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": { 2024-09-13T14:17:46.691 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 2 2024-09-13T14:17:46.691 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:17:46.691 INFO:teuthology.orchestra.run.smithi102.stdout: "overall": { 2024-09-13T14:17:46.691 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 11, 2024-09-13T14:17:46.692 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 4 2024-09-13T14:17:46.692 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:17:46.692 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:17:46.979 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:17:46.979 INFO:teuthology.orchestra.run.smithi102.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f", 2024-09-13T14:17:46.979 INFO:teuthology.orchestra.run.smithi102.stdout: "in_progress": true, 2024-09-13T14:17:46.979 INFO:teuthology.orchestra.run.smithi102.stdout: "which": "Upgrading daemons of type(s) mon. Upgrade limited to 1 daemons (0 remaining).", 2024-09-13T14:17:46.979 INFO:teuthology.orchestra.run.smithi102.stdout: "services_complete": [], 2024-09-13T14:17:46.980 INFO:teuthology.orchestra.run.smithi102.stdout: "progress": "1/3 daemons upgraded", 2024-09-13T14:17:46.980 INFO:teuthology.orchestra.run.smithi102.stdout: "message": "Currently upgrading mon daemons", 2024-09-13T14:17:46.980 INFO:teuthology.orchestra.run.smithi102.stdout: "is_paused": false 2024-09-13T14:17:46.980 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:17:47.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:46 smithi195 ceph-mon[33205]: pgmap v13: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:47.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:46 smithi195 ceph-mon[33205]: from='client.34149 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:47.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:46 smithi195 ceph-mon[33205]: from='client.34155 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:47.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:46 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/3157726238' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:17:47.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:46 smithi102 ceph-mon[110183]: pgmap v13: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:47.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:46 smithi102 ceph-mon[110183]: from='client.34149 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:47.404 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:46 smithi102 ceph-mon[110183]: from='client.34155 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:47.404 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:46 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3157726238' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:17:47.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:46 smithi102 ceph-mon[115269]: pgmap v13: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:47.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:46 smithi102 ceph-mon[115269]: from='client.34149 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:47.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:46 smithi102 ceph-mon[115269]: from='client.34155 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:47.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:46 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3157726238' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:17:48.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:47 smithi102 ceph-mon[110183]: from='client.34161 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:48.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:47 smithi102 ceph-mon[110183]: from='client.25321 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:48.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:48 smithi102 ceph-mon[115269]: from='client.34161 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:48.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:48 smithi102 ceph-mon[115269]: from='client.25321 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:48.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:47 smithi195 ceph-mon[33205]: from='client.34161 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:48.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:47 smithi195 ceph-mon[33205]: from='client.25321 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:17:49.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:48 smithi102 ceph-mon[110183]: pgmap v14: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:49.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:48 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:17:49.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:48 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:17:49.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:48 smithi102 ceph-mon[115269]: pgmap v14: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:49.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:48 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:17:49.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:48 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:17:49.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:48 smithi195 ceph-mon[33205]: pgmap v14: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:49.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:49 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:17:49.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:49 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:17:49.804 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:49 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: 2024-09-13T14:17:49.756+0000 7f0c92e7e640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-13T14:17:50.398 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:50 smithi195 ceph-mon[33205]: pgmap v15: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:50.399 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:50 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:17:50.399 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:50 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:17:50.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:50 smithi102 ceph-mon[110183]: pgmap v15: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:50.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:50 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:17:50.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:50 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:17:50.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:50 smithi102 ceph-mon[115269]: pgmap v15: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:50.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:50 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:17:50.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:50 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:17:52.325 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:52 smithi102 ceph-mon[110183]: pgmap v16: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:52.325 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:52 smithi102 ceph-mon[115269]: pgmap v16: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:52.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:52 smithi195 ceph-mon[33205]: pgmap v16: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:53.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:17:53.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:17:53.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:17:53.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:17:53.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:17:53.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:17:53.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:17:53.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:17:53.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: Upgrade: Setting container_image for all nvmeof 2024-09-13T14:17:53.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:17:53.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:17:53.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: Upgrade: Setting container_image for all node-exporter 2024-09-13T14:17:53.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:17:53.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-13T14:17:53.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-09-13T14:17:53.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-13T14:17:53.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-09-13T14:17:53.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:17:53.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: Upgrade: Setting container_image for all prometheus 2024-09-13T14:17:53.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-13T14:17:53.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-13T14:17:53.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:17:53.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: Upgrade: Setting container_image for all alertmanager 2024-09-13T14:17:53.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-13T14:17:53.671 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-13T14:17:53.671 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:17:53.671 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: Upgrade: Setting container_image for all grafana 2024-09-13T14:17:53.671 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-13T14:17:53.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:17:53.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:17:53.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:17:53.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:17:53.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:17:53.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:17:53.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:17:53.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:17:53.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: Upgrade: Setting container_image for all nvmeof 2024-09-13T14:17:53.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:17:53.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:17:53.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: Upgrade: Setting container_image for all node-exporter 2024-09-13T14:17:53.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:17:53.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-13T14:17:53.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-09-13T14:17:53.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-13T14:17:53.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-09-13T14:17:53.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:17:53.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: Upgrade: Setting container_image for all prometheus 2024-09-13T14:17:53.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-13T14:17:53.673 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-13T14:17:53.673 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:17:53.673 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: Upgrade: Setting container_image for all alertmanager 2024-09-13T14:17:53.673 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-13T14:17:53.673 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-13T14:17:53.673 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:17:53.673 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: Upgrade: Setting container_image for all grafana 2024-09-13T14:17:53.673 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-13T14:17:53.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-13T14:17:53.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:17:53.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: Upgrade: Setting container_image for all loki 2024-09-13T14:17:53.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-13T14:17:53.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-13T14:17:53.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:17:53.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: Upgrade: Setting container_image for all promtail 2024-09-13T14:17:53.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-13T14:17:53.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-13T14:17:53.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: Upgrade: Finalizing container_image settings 2024-09-13T14:17:53.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr"} : dispatch 2024-09-13T14:17:53.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-13T14:17:53.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:17:53.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-13T14:17:53.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.crash"} : dispatch 2024-09-13T14:17:53.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd"} : dispatch 2024-09-13T14:17:53.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mds"} : dispatch 2024-09-13T14:17:53.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw"} : dispatch 2024-09-13T14:17:53.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"} : dispatch 2024-09-13T14:17:53.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:17:53.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"} : dispatch 2024-09-13T14:17:53.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.iscsi"} : dispatch 2024-09-13T14:17:53.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nfs"} : dispatch 2024-09-13T14:17:53.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"} : dispatch 2024-09-13T14:17:53.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-13T14:17:53.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:17:53.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:17:53.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:17:53.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:17:53.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:17:53.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:17:53.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: Upgrade: Complete! 2024-09-13T14:17:53.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"} : dispatch 2024-09-13T14:17:53.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-13T14:17:53.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:17:53.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:17:53.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:17:53.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:17:53.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:17:53.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:17:53.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:17:53.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:17:53.677 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-13T14:17:53.677 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:17:53.677 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: Upgrade: Setting container_image for all loki 2024-09-13T14:17:53.677 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-13T14:17:53.677 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-13T14:17:53.677 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:17:53.677 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: Upgrade: Setting container_image for all promtail 2024-09-13T14:17:53.678 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-13T14:17:53.678 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-13T14:17:53.678 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: Upgrade: Finalizing container_image settings 2024-09-13T14:17:53.678 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr"} : dispatch 2024-09-13T14:17:53.678 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-13T14:17:53.678 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:17:53.678 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-13T14:17:53.678 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.crash"} : dispatch 2024-09-13T14:17:53.678 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd"} : dispatch 2024-09-13T14:17:53.678 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mds"} : dispatch 2024-09-13T14:17:53.678 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw"} : dispatch 2024-09-13T14:17:53.679 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"} : dispatch 2024-09-13T14:17:53.679 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:17:53.679 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"} : dispatch 2024-09-13T14:17:53.679 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.iscsi"} : dispatch 2024-09-13T14:17:53.679 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nfs"} : dispatch 2024-09-13T14:17:53.679 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"} : dispatch 2024-09-13T14:17:53.679 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-13T14:17:53.679 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:17:53.679 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:17:53.679 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:17:53.679 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:17:53.680 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:17:53.680 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:17:53.680 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: Upgrade: Complete! 2024-09-13T14:17:53.680 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"} : dispatch 2024-09-13T14:17:53.680 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-13T14:17:53.680 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:17:53.680 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:17:53.680 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:17:53.680 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:17:53.680 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:17:53.680 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:17:53.680 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:17:53.680 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:17:53.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:17:53.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:17:53.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:17:53.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:17:53.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:17:53.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:17:53.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:17:53.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:17:53.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: Upgrade: Setting container_image for all nvmeof 2024-09-13T14:17:53.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:17:53.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:17:53.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: Upgrade: Setting container_image for all node-exporter 2024-09-13T14:17:53.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:17:53.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-13T14:17:53.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-09-13T14:17:53.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-13T14:17:53.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-09-13T14:17:53.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:17:53.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: Upgrade: Setting container_image for all prometheus 2024-09-13T14:17:53.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-13T14:17:53.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-13T14:17:53.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:17:53.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: Upgrade: Setting container_image for all alertmanager 2024-09-13T14:17:53.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-13T14:17:53.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-13T14:17:53.750 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:17:53.750 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: Upgrade: Setting container_image for all grafana 2024-09-13T14:17:53.750 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-13T14:17:53.750 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-13T14:17:53.750 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:17:53.750 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: Upgrade: Setting container_image for all loki 2024-09-13T14:17:53.750 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-13T14:17:53.750 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-13T14:17:53.750 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:17:53.750 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: Upgrade: Setting container_image for all promtail 2024-09-13T14:17:53.750 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-13T14:17:53.750 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-13T14:17:53.750 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: Upgrade: Finalizing container_image settings 2024-09-13T14:17:53.750 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr"} : dispatch 2024-09-13T14:17:53.751 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-13T14:17:53.751 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:17:53.751 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-13T14:17:53.751 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.crash"} : dispatch 2024-09-13T14:17:53.751 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd"} : dispatch 2024-09-13T14:17:53.751 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mds"} : dispatch 2024-09-13T14:17:53.751 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw"} : dispatch 2024-09-13T14:17:53.751 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"} : dispatch 2024-09-13T14:17:53.751 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:17:53.751 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"} : dispatch 2024-09-13T14:17:53.751 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.iscsi"} : dispatch 2024-09-13T14:17:53.751 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nfs"} : dispatch 2024-09-13T14:17:53.751 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"} : dispatch 2024-09-13T14:17:53.751 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-13T14:17:53.752 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:17:53.752 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:17:53.752 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:17:53.752 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:17:53.752 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:17:53.752 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:17:53.752 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: Upgrade: Complete! 2024-09-13T14:17:53.752 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"} : dispatch 2024-09-13T14:17:53.752 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-13T14:17:53.752 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:17:53.752 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:17:53.752 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:17:53.752 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:17:53.752 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:17:53.752 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:17:53.753 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:17:53.753 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:17:54.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:54 smithi102 ceph-mon[110183]: pgmap v17: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:54.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:54 smithi102 ceph-mon[115269]: pgmap v17: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:54.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:54 smithi195 ceph-mon[33205]: pgmap v17: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:55.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:55 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:17:55.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:55 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:17:55.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:55 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:17:55.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:55 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:17:55.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:55 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:17:55.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:55 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:17:56.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:56 smithi102 ceph-mon[110183]: pgmap v18: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:56.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:56 smithi102 ceph-mon[115269]: pgmap v18: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:56.919 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:17:56 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:17:56] "GET /metrics HTTP/1.1" 200 33819 "" "Prometheus/2.51.0" 2024-09-13T14:17:56.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:56 smithi195 ceph-mon[33205]: pgmap v18: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:58.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:17:58 smithi102 ceph-mon[110183]: pgmap v19: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:58.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:17:58 smithi102 ceph-mon[115269]: pgmap v19: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:17:58.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:17:58 smithi195 ceph-mon[33205]: pgmap v19: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:00.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:18:00 smithi102 ceph-mon[110183]: pgmap v20: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:00.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:18:00 smithi102 ceph-mon[115269]: pgmap v20: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:00.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:18:00 smithi195 ceph-mon[33205]: pgmap v20: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:02.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:18:02 smithi102 ceph-mon[115269]: pgmap v21: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:02.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:18:02 smithi102 ceph-mon[110183]: pgmap v21: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:02.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:18:02 smithi195 ceph-mon[33205]: pgmap v21: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:03.497 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:18:03 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=infra.usagestats t=2024-09-13T14:18:03.202822284Z level=info msg="Usage stats are ready to report" 2024-09-13T14:18:04.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:18:04 smithi102 ceph-mon[115269]: pgmap v22: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:04.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:18:04 smithi102 ceph-mon[110183]: pgmap v22: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:04.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:18:04 smithi195 ceph-mon[33205]: pgmap v22: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:06.918 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:18:06 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:18:06] "GET /metrics HTTP/1.1" 200 33819 "" "Prometheus/2.51.0" 2024-09-13T14:18:06.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:18:06 smithi102 ceph-mon[115269]: pgmap v23: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:06.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:18:06 smithi102 ceph-mon[110183]: pgmap v23: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:06.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:18:06 smithi195 ceph-mon[33205]: pgmap v23: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:08.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:18:08 smithi195 ceph-mon[33205]: pgmap v24: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:09.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:18:08 smithi102 ceph-mon[110183]: pgmap v24: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:09.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:18:08 smithi102 ceph-mon[115269]: pgmap v24: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:10.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:18:10 smithi195 ceph-mon[33205]: pgmap v25: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:10.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:18:10 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:18:11.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:18:10 smithi102 ceph-mon[110183]: pgmap v25: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:11.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:18:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:18:11.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:18:10 smithi102 ceph-mon[115269]: pgmap v25: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:11.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:18:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:18:12.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:18:12 smithi195 ceph-mon[33205]: pgmap v26: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:13.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:18:12 smithi102 ceph-mon[110183]: pgmap v26: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:13.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:18:12 smithi102 ceph-mon[115269]: pgmap v26: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:14.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:18:14 smithi195 ceph-mon[33205]: pgmap v27: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:15.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:18:14 smithi102 ceph-mon[110183]: pgmap v27: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:15.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:18:14 smithi102 ceph-mon[115269]: pgmap v27: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:16.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:18:16 smithi102 ceph-mon[110183]: pgmap v28: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:16.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:18:16 smithi102 ceph-mon[115269]: pgmap v28: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:16.919 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:18:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:18:16] "GET /metrics HTTP/1.1" 200 33813 "" "Prometheus/2.51.0" 2024-09-13T14:18:16.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:18:16 smithi195 ceph-mon[33205]: pgmap v28: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:17.968 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'sleep 60' 2024-09-13T14:18:18.170 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:18:18.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:18:18 smithi102 ceph-mon[115269]: pgmap v29: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:18.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:18:18 smithi102 ceph-mon[115269]: from='client.34179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:18:18.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:18:18 smithi102 ceph-mon[110183]: pgmap v29: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:18.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:18:18 smithi102 ceph-mon[110183]: from='client.34179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:18:18.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:18:18 smithi195 ceph-mon[33205]: pgmap v29: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:18.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:18:18 smithi195 ceph-mon[33205]: from='client.34179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:18:20.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:18:20 smithi195 ceph-mon[33205]: pgmap v30: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:21.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:18:20 smithi102 ceph-mon[115269]: pgmap v30: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:21.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:18:20 smithi102 ceph-mon[110183]: pgmap v30: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:22.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:18:22 smithi195 ceph-mon[33205]: pgmap v31: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:23.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:18:22 smithi102 ceph-mon[110183]: pgmap v31: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:23.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:18:22 smithi102 ceph-mon[115269]: pgmap v31: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:24.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:18:24 smithi195 ceph-mon[33205]: pgmap v32: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:25.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:18:24 smithi102 ceph-mon[110183]: pgmap v32: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:25.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:18:24 smithi102 ceph-mon[115269]: pgmap v32: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:25.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:18:25 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:18:26.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:18:25 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:18:26.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:18:25 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:18:26.702 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:18:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:18:26] "GET /metrics HTTP/1.1" 200 33817 "" "Prometheus/2.51.0" 2024-09-13T14:18:26.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:18:26 smithi195 ceph-mon[33205]: pgmap v33: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:27.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:18:26 smithi102 ceph-mon[110183]: pgmap v33: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:27.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:18:26 smithi102 ceph-mon[115269]: pgmap v33: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:28.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:18:28 smithi195 ceph-mon[33205]: pgmap v34: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:29.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:18:28 smithi102 ceph-mon[110183]: pgmap v34: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:29.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:18:28 smithi102 ceph-mon[115269]: pgmap v34: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:30.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:18:30 smithi195 ceph-mon[33205]: pgmap v35: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:31.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:18:30 smithi102 ceph-mon[110183]: pgmap v35: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:31.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:18:30 smithi102 ceph-mon[115269]: pgmap v35: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:32.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:18:32 smithi195 ceph-mon[33205]: pgmap v36: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:33.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:18:32 smithi102 ceph-mon[110183]: pgmap v36: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:33.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:18:32 smithi102 ceph-mon[115269]: pgmap v36: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:34.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:18:34 smithi195 ceph-mon[33205]: pgmap v37: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:35.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:18:34 smithi102 ceph-mon[110183]: pgmap v37: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:35.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:18:34 smithi102 ceph-mon[115269]: pgmap v37: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:36.722 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:18:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:18:36] "GET /metrics HTTP/1.1" 200 33817 "" "Prometheus/2.51.0" 2024-09-13T14:18:36.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:18:36 smithi195 ceph-mon[33205]: pgmap v38: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:37.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:18:36 smithi102 ceph-mon[110183]: pgmap v38: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:37.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:18:36 smithi102 ceph-mon[115269]: pgmap v38: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:38.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:18:38 smithi195 ceph-mon[33205]: pgmap v39: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:39.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:18:38 smithi102 ceph-mon[110183]: pgmap v39: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:39.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:18:38 smithi102 ceph-mon[115269]: pgmap v39: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:40.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:18:40 smithi195 ceph-mon[33205]: pgmap v40: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:40.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:18:40 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:18:41.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:18:40 smithi102 ceph-mon[110183]: pgmap v40: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:41.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:18:40 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:18:41.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:18:40 smithi102 ceph-mon[115269]: pgmap v40: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:41.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:18:40 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:18:42.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:18:42 smithi195 ceph-mon[33205]: pgmap v41: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:43.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:18:42 smithi102 ceph-mon[110183]: pgmap v41: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:43.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:18:42 smithi102 ceph-mon[115269]: pgmap v41: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:44.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:18:44 smithi195 ceph-mon[33205]: pgmap v42: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:45.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:18:44 smithi102 ceph-mon[110183]: pgmap v42: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:45.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:18:44 smithi102 ceph-mon[115269]: pgmap v42: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:46.738 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:18:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:18:46] "GET /metrics HTTP/1.1" 200 33817 "" "Prometheus/2.51.0" 2024-09-13T14:18:46.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:18:46 smithi195 ceph-mon[33205]: pgmap v43: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:47.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:18:46 smithi102 ceph-mon[110183]: pgmap v43: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:47.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:18:46 smithi102 ceph-mon[115269]: pgmap v43: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:48.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:18:48 smithi195 ceph-mon[33205]: pgmap v44: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:49.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:18:48 smithi102 ceph-mon[110183]: pgmap v44: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:49.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:18:48 smithi102 ceph-mon[115269]: pgmap v44: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:50.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:18:50 smithi195 ceph-mon[33205]: pgmap v45: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:51.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:18:50 smithi102 ceph-mon[110183]: pgmap v45: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:51.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:18:50 smithi102 ceph-mon[115269]: pgmap v45: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:53.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:18:52 smithi102 ceph-mon[110183]: pgmap v46: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:53.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:18:52 smithi102 ceph-mon[115269]: pgmap v46: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:53.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:18:52 smithi195 ceph-mon[33205]: pgmap v46: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:53.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:18:53 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:18:53.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:18:53 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:18:54.195 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:18:53 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:18:55.074 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:18:54 smithi102 ceph-mon[110183]: pgmap v47: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:55.074 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:18:54 smithi102 ceph-mon[115269]: pgmap v47: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:55.083 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:18:54 smithi195 ceph-mon[33205]: pgmap v47: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:56.078 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:18:55 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:18:56.078 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:18:55 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:18:56.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:18:55 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:18:56.758 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:18:56 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:18:56] "GET /metrics HTTP/1.1" 200 33814 "" "Prometheus/2.51.0" 2024-09-13T14:18:57.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:18:56 smithi102 ceph-mon[115269]: pgmap v48: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:57.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:18:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:18:57.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:18:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:18:57.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:18:56 smithi102 ceph-mon[110183]: pgmap v48: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:57.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:18:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:18:57.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:18:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:18:57.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:18:56 smithi195 ceph-mon[33205]: pgmap v48: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:18:57.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:18:56 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:18:57.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:18:56 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:18:58.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:18:57 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:18:58.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:18:57 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:18:58.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:18:57 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:18:58.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:18:57 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:18:58.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:18:57 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:18:58.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:18:57 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:18:58.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:18:57 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:18:58.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:18:57 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:18:58.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:18:57 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:18:58.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:18:57 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:18:58.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:18:57 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:18:58.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:18:57 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:18:58.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:18:57 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:18:58.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:18:57 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:18:58.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:18:57 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:18:59.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:18:58 smithi195 ceph-mon[33205]: pgmap v49: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:18:59.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:18:58 smithi195 ceph-mon[33205]: mgrmap e41: y(active, since 92s), standbys: x 2024-09-13T14:18:59.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:18:58 smithi102 ceph-mon[110183]: pgmap v49: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:18:59.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:18:58 smithi102 ceph-mon[110183]: mgrmap e41: y(active, since 92s), standbys: x 2024-09-13T14:18:59.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:18:58 smithi102 ceph-mon[115269]: pgmap v49: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:18:59.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:18:58 smithi102 ceph-mon[115269]: mgrmap e41: y(active, since 92s), standbys: x 2024-09-13T14:19:01.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:00 smithi195 ceph-mon[33205]: pgmap v50: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:19:01.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:00 smithi102 ceph-mon[110183]: pgmap v50: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:19:01.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:00 smithi102 ceph-mon[115269]: pgmap v50: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:19:03.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:02 smithi195 ceph-mon[33205]: pgmap v51: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:19:03.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:02 smithi102 ceph-mon[115269]: pgmap v51: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:19:03.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:02 smithi102 ceph-mon[110183]: pgmap v51: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:19:05.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:04 smithi195 ceph-mon[33205]: pgmap v52: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:19:05.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:04 smithi102 ceph-mon[110183]: pgmap v52: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:19:05.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:04 smithi102 ceph-mon[115269]: pgmap v52: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:19:06.918 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:19:06 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:19:06] "GET /metrics HTTP/1.1" 200 33814 "" "Prometheus/2.51.0" 2024-09-13T14:19:07.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:06 smithi195 ceph-mon[33205]: pgmap v53: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:19:07.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:06 smithi102 ceph-mon[110183]: pgmap v53: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:19:07.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:06 smithi102 ceph-mon[115269]: pgmap v53: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:19:09.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:08 smithi195 ceph-mon[33205]: pgmap v54: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:19:09.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:08 smithi102 ceph-mon[110183]: pgmap v54: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:19:09.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:08 smithi102 ceph-mon[115269]: pgmap v54: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:19:11.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:10 smithi195 ceph-mon[33205]: pgmap v55: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:11.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:10 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:19:11.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:10 smithi102 ceph-mon[115269]: pgmap v55: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:11.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:19:11.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:10 smithi102 ceph-mon[110183]: pgmap v55: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:11.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:19:13.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:12 smithi195 ceph-mon[33205]: pgmap v56: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:13.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:12 smithi102 ceph-mon[110183]: pgmap v56: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:13.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:12 smithi102 ceph-mon[115269]: pgmap v56: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:15.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:14 smithi195 ceph-mon[33205]: pgmap v57: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:15.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:14 smithi102 ceph-mon[110183]: pgmap v57: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:15.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:14 smithi102 ceph-mon[115269]: pgmap v57: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:16.918 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:19:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:19:16] "GET /metrics HTTP/1.1" 200 33814 "" "Prometheus/2.51.0" 2024-09-13T14:19:17.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:16 smithi195 ceph-mon[33205]: pgmap v58: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:17.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:16 smithi102 ceph-mon[110183]: pgmap v58: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:17.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:16 smithi102 ceph-mon[115269]: pgmap v58: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:19.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:18 smithi195 ceph-mon[33205]: pgmap v59: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:19.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:18 smithi102 ceph-mon[110183]: pgmap v59: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:19.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:18 smithi102 ceph-mon[115269]: pgmap v59: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:20.557 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'echo wait for majority of mons to upgrade' 2024-09-13T14:19:20.761 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:19:21.145 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:20 smithi102 ceph-mon[110183]: pgmap v60: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:21.146 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:20 smithi102 ceph-mon[115269]: pgmap v60: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:21.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:20 smithi195 ceph-mon[33205]: pgmap v60: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:22.428 INFO:teuthology.orchestra.run.smithi102.stdout:wait for majority of mons to upgrade 2024-09-13T14:19:23.010 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --limit 1' 2024-09-13T14:19:23.208 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:19:23.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:22 smithi195 ceph-mon[33205]: pgmap v61: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:23.323 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:22 smithi102 ceph-mon[110183]: pgmap v61: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:23.324 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:22 smithi102 ceph-mon[115269]: pgmap v61: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:25.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:24 smithi102 ceph-mon[110183]: pgmap v62: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:25.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:24 smithi102 ceph-mon[115269]: pgmap v62: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:25.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:24 smithi195 ceph-mon[33205]: pgmap v62: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:26.109 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:25 smithi102 ceph-mon[110183]: from='client.44108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9", "daemon_types": "mon", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:19:26.109 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:25 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:19:26.110 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:25 smithi102 ceph-mon[115269]: from='client.44108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9", "daemon_types": "mon", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:19:26.110 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:25 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:19:26.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:25 smithi195 ceph-mon[33205]: from='client.44108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9", "daemon_types": "mon", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:19:26.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:25 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:19:26.653 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:19:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:19:26] "GET /metrics HTTP/1.1" 200 33814 "" "Prometheus/2.51.0" 2024-09-13T14:19:26.936 INFO:teuthology.orchestra.run.smithi102.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:19:27.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:26 smithi102 ceph-mon[110183]: pgmap v63: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:27.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:27.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:19:27.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:19:27.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:19:27.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:27.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:26 smithi102 ceph-mon[115269]: pgmap v63: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:27.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:27.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:19:27.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:19:27.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:19:27.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:27.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:26 smithi195 ceph-mon[33205]: pgmap v63: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:27.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:26 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:27.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:26 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:19:27.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:26 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:19:27.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:26 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:19:27.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:26 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:27.624 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-09-13T14:19:27.849 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:19:28.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:27 smithi195 ceph-mon[33205]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:19:28.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:27 smithi195 ceph-mon[33205]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:19:28.249 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:27 smithi102 ceph-mon[110183]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:19:28.249 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:27 smithi102 ceph-mon[110183]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:19:28.249 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:27 smithi102 ceph-mon[115269]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:19:28.249 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:27 smithi102 ceph-mon[115269]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:19:29.150 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:28 smithi102 ceph-mon[115269]: pgmap v64: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:29.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:28 smithi102 ceph-mon[110183]: pgmap v64: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:29.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:28 smithi195 ceph-mon[33205]: pgmap v64: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:30.062 INFO:teuthology.orchestra.run.smithi102.stdout:true 2024-09-13T14:19:30.631 INFO:teuthology.orchestra.run.smithi102.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-13T14:19:30.631 INFO:teuthology.orchestra.run.smithi102.stdout:alertmanager.a smithi102 *:9093,9094 running (3m) 34s ago 9m 19.2M - 0.27.0 11f11916f8cd 0322b99ec767 2024-09-13T14:19:30.631 INFO:teuthology.orchestra.run.smithi102.stdout:grafana.a smithi195 *:3000 running (2m) 33s ago 8m 84.4M - c8b91775d855 152e89bad2c8 2024-09-13T14:19:30.631 INFO:teuthology.orchestra.run.smithi102.stdout:mds.foo.smithi102.njetxr smithi102 running (7m) 34s ago 7m 21.2M - 18.2.4-843-g2c01458a c129f5a2fd4e c9211cc725ed 2024-09-13T14:19:30.631 INFO:teuthology.orchestra.run.smithi102.stdout:mds.foo.smithi195.yqzrxi smithi195 running (7m) 33s ago 7m 18.1M - 18.2.4-843-g2c01458a c129f5a2fd4e eea37a7230fd 2024-09-13T14:19:30.631 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.x smithi195 *:8443,9283,8765 running (4m) 33s ago 12m 453M - 19.3.0-4697-gd58f7306 fec07016a2b0 4a23b8f0ea8b 2024-09-13T14:19:30.631 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.y smithi102 *:8443,9283,8765 running (4m) 34s ago 14m 507M - 19.3.0-4697-gd58f7306 fec07016a2b0 6b12b28dcde1 2024-09-13T14:19:30.632 INFO:teuthology.orchestra.run.smithi102.stdout:mon.a smithi102 running (2m) 34s ago 14m 41.7M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 f3234b7bbdde 2024-09-13T14:19:30.632 INFO:teuthology.orchestra.run.smithi102.stdout:mon.b smithi195 running (13m) 33s ago 13m 54.1M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e 608ec4847676 2024-09-13T14:19:30.632 INFO:teuthology.orchestra.run.smithi102.stdout:mon.c smithi102 running (111s) 34s ago 13m 31.7M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 dba6b3d55296 2024-09-13T14:19:30.632 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.a smithi102 *:9100 running (4m) 34s ago 8m 11.7M - 1.7.0 72c9c2088986 0c92d0b39468 2024-09-13T14:19:30.632 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.b smithi195 *:9100 running (4m) 33s ago 8m 10.7M - 1.7.0 72c9c2088986 fde867a7dbf9 2024-09-13T14:19:30.632 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0 smithi102 running (12m) 34s ago 12m 60.7M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e b97482fb8cf9 2024-09-13T14:19:30.632 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1 smithi102 running (12m) 34s ago 12m 52.2M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 8a67356be7a2 2024-09-13T14:19:30.632 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2 smithi102 running (11m) 34s ago 11m 50.3M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 1800986ac838 2024-09-13T14:19:30.632 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3 smithi102 running (11m) 34s ago 11m 56.7M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 968246da9da2 2024-09-13T14:19:30.632 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4 smithi195 running (11m) 33s ago 11m 49.1M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 515307701d73 2024-09-13T14:19:30.632 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5 smithi195 running (10m) 33s ago 10m 53.2M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 176ba2b3bd59 2024-09-13T14:19:30.632 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6 smithi195 running (10m) 33s ago 10m 55.0M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 9cd070a6dd3d 2024-09-13T14:19:30.632 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7 smithi195 running (10m) 33s ago 10m 53.0M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e a6b698c659ca 2024-09-13T14:19:30.633 INFO:teuthology.orchestra.run.smithi102.stdout:prometheus.a smithi195 *:9095 running (3m) 33s ago 9m 43.7M - 2.51.0 1d3b7f56885b d131e53f1ef9 2024-09-13T14:19:30.976 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:19:30.976 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": { 2024-09-13T14:19:30.976 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 1, 2024-09-13T14:19:30.976 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 2 2024-09-13T14:19:30.976 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:19:30.976 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": { 2024-09-13T14:19:30.976 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 2 2024-09-13T14:19:30.976 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:19:30.977 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": { 2024-09-13T14:19:30.977 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 8 2024-09-13T14:19:30.977 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:19:30.977 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": { 2024-09-13T14:19:30.977 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 2 2024-09-13T14:19:30.977 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:19:30.977 INFO:teuthology.orchestra.run.smithi102.stdout: "overall": { 2024-09-13T14:19:30.977 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 11, 2024-09-13T14:19:30.977 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 4 2024-09-13T14:19:30.977 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:19:30.977 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:19:31.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:30 smithi195 ceph-mon[33205]: pgmap v65: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:31.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:30 smithi195 ceph-mon[33205]: from='client.34191 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:19:31.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:30 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:31.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:30 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:19:31.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:30 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:19:31.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:30 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:31.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:30 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-09-13T14:19:31.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:30 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["b"]} : dispatch 2024-09-13T14:19:31.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:30 smithi195 ceph-mon[33205]: from='client.? 172.21.15.102:0/836216583' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:19:31.266 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:30 smithi102 ceph-mon[110183]: pgmap v65: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:31.266 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:30 smithi102 ceph-mon[110183]: from='client.34191 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:19:31.266 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:30 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:31.266 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:30 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:19:31.266 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:30 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:19:31.266 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:30 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:31.266 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:30 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-09-13T14:19:31.266 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:30 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["b"]} : dispatch 2024-09-13T14:19:31.266 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:30 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/836216583' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:19:31.267 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:30 smithi102 ceph-mon[115269]: pgmap v65: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:31.267 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:30 smithi102 ceph-mon[115269]: from='client.34191 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:19:31.267 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:30 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:31.267 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:30 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:19:31.267 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:30 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:19:31.267 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:30 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:31.267 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:30 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-09-13T14:19:31.267 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:30 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["b"]} : dispatch 2024-09-13T14:19:31.267 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:30 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/836216583' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:19:31.267 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:19:31.268 INFO:teuthology.orchestra.run.smithi102.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f", 2024-09-13T14:19:31.268 INFO:teuthology.orchestra.run.smithi102.stdout: "in_progress": true, 2024-09-13T14:19:31.268 INFO:teuthology.orchestra.run.smithi102.stdout: "which": "Upgrading daemons of type(s) mon. Upgrade limited to 1 daemons (1 remaining).", 2024-09-13T14:19:31.268 INFO:teuthology.orchestra.run.smithi102.stdout: "services_complete": [], 2024-09-13T14:19:31.268 INFO:teuthology.orchestra.run.smithi102.stdout: "progress": "2/3 daemons upgraded", 2024-09-13T14:19:31.268 INFO:teuthology.orchestra.run.smithi102.stdout: "message": "Currently upgrading mon daemons", 2024-09-13T14:19:31.268 INFO:teuthology.orchestra.run.smithi102.stdout: "is_paused": false 2024-09-13T14:19:31.268 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:19:32.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:31 smithi102 ceph-mon[115269]: from='client.44120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:19:32.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:31 smithi102 ceph-mon[115269]: from='client.34203 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:19:32.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:31 smithi102 ceph-mon[115269]: Upgrade: Target is version 19.3.0-4697-gd58f7306 (squid) 2024-09-13T14:19:32.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:31 smithi102 ceph-mon[115269]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, digests ['quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f'] 2024-09-13T14:19:32.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:31 smithi102 ceph-mon[115269]: Upgrade: Setting container_image for all mgr 2024-09-13T14:19:32.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:31 smithi102 ceph-mon[115269]: Upgrade: It appears safe to stop mon.b 2024-09-13T14:19:32.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:31 smithi102 ceph-mon[110183]: from='client.44120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:19:32.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:31 smithi102 ceph-mon[110183]: from='client.34203 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:19:32.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:31 smithi102 ceph-mon[110183]: Upgrade: Target is version 19.3.0-4697-gd58f7306 (squid) 2024-09-13T14:19:32.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:31 smithi102 ceph-mon[110183]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, digests ['quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f'] 2024-09-13T14:19:32.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:31 smithi102 ceph-mon[110183]: Upgrade: Setting container_image for all mgr 2024-09-13T14:19:32.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:31 smithi102 ceph-mon[110183]: Upgrade: It appears safe to stop mon.b 2024-09-13T14:19:32.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:31 smithi195 ceph-mon[33205]: from='client.44120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:19:32.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:31 smithi195 ceph-mon[33205]: from='client.34203 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:19:32.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:31 smithi195 ceph-mon[33205]: Upgrade: Target is version 19.3.0-4697-gd58f7306 (squid) 2024-09-13T14:19:32.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:31 smithi195 ceph-mon[33205]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, digests ['quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f'] 2024-09-13T14:19:32.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:31 smithi195 ceph-mon[33205]: Upgrade: Setting container_image for all mgr 2024-09-13T14:19:32.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:31 smithi195 ceph-mon[33205]: Upgrade: It appears safe to stop mon.b 2024-09-13T14:19:33.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:32 smithi195 ceph-mon[33205]: pgmap v66: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:33.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:32 smithi195 ceph-mon[33205]: from='client.44138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:19:33.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:32 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:33.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:32 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:19:33.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:32 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:19:33.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:32 smithi195 ceph-mon[33205]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:19:33.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:32 smithi102 ceph-mon[110183]: pgmap v66: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:33.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:32 smithi102 ceph-mon[110183]: from='client.44138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:19:33.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:32 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:33.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:32 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:19:33.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:32 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:19:33.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:32 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:19:33.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:32 smithi102 ceph-mon[115269]: pgmap v66: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:33.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:32 smithi102 ceph-mon[115269]: from='client.44138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:19:33.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:32 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:33.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:32 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:19:33.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:32 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:19:33.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:32 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:19:34.237 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:33 smithi195 systemd[1]: Stopping Ceph mon.b for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T14:19:34.237 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:34 smithi195 ceph-mon[33205]: Upgrade: Updating mon.b 2024-09-13T14:19:34.237 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:34 smithi195 ceph-mon[33205]: Deploying daemon mon.b on smithi195 2024-09-13T14:19:34.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:34 smithi102 ceph-mon[110183]: Upgrade: Updating mon.b 2024-09-13T14:19:34.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:34 smithi102 ceph-mon[110183]: Deploying daemon mon.b on smithi195 2024-09-13T14:19:34.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:34 smithi102 ceph-mon[115269]: Upgrade: Updating mon.b 2024-09-13T14:19:34.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:34 smithi102 ceph-mon[115269]: Deploying daemon mon.b on smithi195 2024-09-13T14:19:34.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:34 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-b[33169]: 2024-09-13T14:19:34.234+0000 7fb732d54640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T14:19:34.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:34 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-b[33169]: 2024-09-13T14:19:34.234+0000 7fb732d54640 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-09-13T14:19:34.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:34 smithi195 podman[104074]: 2024-09-13 14:19:34.286563961 +0000 UTC m=+0.154879052 container died 608ec48476768eb0992ff1fc126d853f5e0de5e1f6d707edd7032b6c35e4afb6 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-b, GIT_BRANCH=HEAD, RELEASE=reef-2c01458, GIT_CLEAN=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T14:19:34.901 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:34 smithi195 podman[104074]: 2024-09-13 14:19:34.708323963 +0000 UTC m=+0.576639042 container cleanup 608ec48476768eb0992ff1fc126d853f5e0de5e1f6d707edd7032b6c35e4afb6 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=reef-2c01458, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-13T14:19:34.901 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:34 smithi195 bash[104074]: ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-b 2024-09-13T14:19:35.153 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:34 smithi195 podman[104091]: 2024-09-13 14:19:34.901147945 +0000 UTC m=+0.614185208 container remove 608ec48476768eb0992ff1fc126d853f5e0de5e1f6d707edd7032b6c35e4afb6 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-2c01458, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-13T14:19:35.153 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@mon.b.service: Deactivated successfully. 2024-09-13T14:19:35.153 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 systemd[1]: Stopped Ceph mon.b for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:19:35.153 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@mon.b.service: Consumed 9.645s CPU time. 2024-09-13T14:19:35.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 systemd[1]: Starting Ceph mon.b for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T14:19:35.795 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 podman[104224]: 2024-09-13 14:19:35.446401643 +0000 UTC m=+0.015682543 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:19:35.795 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 podman[104224]: 2024-09-13 14:19:35.569692101 +0000 UTC m=+0.138973002 container create 9921a2aa81682b9156cd45bc658f893cec490637ebe670d36db4a003bce58ba6 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, io.buildah.version=1.37.2) 2024-09-13T14:19:35.795 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 podman[104224]: 2024-09-13 14:19:35.763702712 +0000 UTC m=+0.332983612 container init 9921a2aa81682b9156cd45bc658f893cec490637ebe670d36db4a003bce58ba6 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-b, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-aclamk-cbt-combined-d58f730, CEPH_POINT_RELEASE=) 2024-09-13T14:19:35.795 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 podman[104224]: 2024-09-13 14:19:35.768295558 +0000 UTC m=+0.337576459 container start 9921a2aa81682b9156cd45bc658f893cec490637ebe670d36db4a003bce58ba6 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=) 2024-09-13T14:19:35.795 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: set uid:gid to 167:167 (ceph:ceph) 2024-09-13T14:19:35.795 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev), process ceph-mon, pid 7 2024-09-13T14:19:35.795 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: pidfile_write: ignore empty --pid-file 2024-09-13T14:19:35.796 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: load: jerasure load: lrc 2024-09-13T14:19:35.796 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: RocksDB version: 7.9.2 2024-09-13T14:19:35.796 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Git sha 0 2024-09-13T14:19:35.796 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Compile date 2024-09-12 20:46:14 2024-09-13T14:19:35.796 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: DB SUMMARY 2024-09-13T14:19:35.796 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: DB Session ID: OTOKXEZ9S3BNPF3N6QLT 2024-09-13T14:19:35.796 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: CURRENT file: CURRENT 2024-09-13T14:19:35.796 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: IDENTITY file: IDENTITY 2024-09-13T14:19:35.796 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: MANIFEST file: MANIFEST-000010 size: 1412 Bytes 2024-09-13T14:19:35.796 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 1, files: 000027.sst 2024-09-13T14:19:35.796 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000025.log size: 7326547 ; 2024-09-13T14:19:35.797 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.error_if_exists: 0 2024-09-13T14:19:35.797 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.create_if_missing: 0 2024-09-13T14:19:35.797 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.paranoid_checks: 1 2024-09-13T14:19:35.797 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-13T14:19:35.797 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-13T14:19:35.797 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-13T14:19:35.797 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.env: 0x56548a1ae160 2024-09-13T14:19:35.797 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.fs: PosixFileSystem 2024-09-13T14:19:35.798 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.info_log: 0x56548ba39880 2024-09-13T14:19:35.798 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.max_file_opening_threads: 16 2024-09-13T14:19:35.798 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.statistics: (nil) 2024-09-13T14:19:35.798 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.use_fsync: 0 2024-09-13T14:19:35.798 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.max_log_file_size: 0 2024-09-13T14:19:35.798 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-13T14:19:35.798 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-13T14:19:35.798 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.keep_log_file_num: 1000 2024-09-13T14:19:35.798 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.recycle_log_file_num: 0 2024-09-13T14:19:35.798 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.allow_fallocate: 1 2024-09-13T14:19:35.798 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.allow_mmap_reads: 0 2024-09-13T14:19:35.798 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.allow_mmap_writes: 0 2024-09-13T14:19:35.798 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.use_direct_reads: 0 2024-09-13T14:19:35.798 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-13T14:19:35.799 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.create_missing_column_families: 0 2024-09-13T14:19:35.799 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.db_log_dir: 2024-09-13T14:19:35.799 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.wal_dir: 2024-09-13T14:19:35.799 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-13T14:19:35.799 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-13T14:19:35.799 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-13T14:19:35.799 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-13T14:19:35.799 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-13T14:19:35.799 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-13T14:19:35.799 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.advise_random_on_open: 1 2024-09-13T14:19:35.799 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.db_write_buffer_size: 0 2024-09-13T14:19:35.799 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.write_buffer_manager: 0x56548ba3c960 2024-09-13T14:19:35.799 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-13T14:19:35.800 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-13T14:19:35.800 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-13T14:19:35.800 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.rate_limiter: (nil) 2024-09-13T14:19:35.800 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-13T14:19:35.800 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.wal_recovery_mode: 2 2024-09-13T14:19:35.800 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.enable_thread_tracking: 0 2024-09-13T14:19:35.800 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.enable_pipelined_write: 0 2024-09-13T14:19:35.800 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.unordered_write: 0 2024-09-13T14:19:35.800 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-13T14:19:35.800 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-13T14:19:35.800 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-13T14:19:35.800 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-13T14:19:35.800 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.row_cache: None 2024-09-13T14:19:35.800 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.wal_filter: None 2024-09-13T14:19:35.801 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-13T14:19:35.801 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.allow_ingest_behind: 0 2024-09-13T14:19:35.801 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.two_write_queues: 0 2024-09-13T14:19:35.801 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.manual_wal_flush: 0 2024-09-13T14:19:35.801 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.wal_compression: 0 2024-09-13T14:19:35.801 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.atomic_flush: 0 2024-09-13T14:19:35.801 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-13T14:19:35.801 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-13T14:19:35.801 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-13T14:19:35.801 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.log_readahead_size: 0 2024-09-13T14:19:35.801 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-13T14:19:35.801 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.best_efforts_recovery: 0 2024-09-13T14:19:35.801 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-13T14:19:35.802 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-13T14:19:35.802 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.allow_data_in_errors: 0 2024-09-13T14:19:35.802 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.db_host_id: __hostname__ 2024-09-13T14:19:35.802 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-13T14:19:35.802 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.max_background_jobs: 2 2024-09-13T14:19:35.802 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.max_background_compactions: -1 2024-09-13T14:19:35.802 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.max_subcompactions: 1 2024-09-13T14:19:35.802 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-13T14:19:35.802 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-13T14:19:35.802 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-13T14:19:35.802 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.max_total_wal_size: 0 2024-09-13T14:19:35.802 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-13T14:19:35.802 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-13T14:19:35.802 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-13T14:19:35.802 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-13T14:19:35.803 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.max_open_files: -1 2024-09-13T14:19:35.803 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.bytes_per_sync: 0 2024-09-13T14:19:35.803 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-13T14:19:35.803 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-13T14:19:35.803 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.compaction_readahead_size: 0 2024-09-13T14:19:35.803 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.max_background_flushes: -1 2024-09-13T14:19:35.803 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Compression algorithms supported: 2024-09-13T14:19:35.803 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: kZSTD supported: 0 2024-09-13T14:19:35.803 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: kXpressCompression supported: 0 2024-09-13T14:19:35.803 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: kBZip2Compression supported: 0 2024-09-13T14:19:35.803 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-13T14:19:35.803 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: kLZ4Compression supported: 1 2024-09-13T14:19:35.803 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: kZlibCompression supported: 1 2024-09-13T14:19:35.803 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: kLZ4HCCompression supported: 1 2024-09-13T14:19:35.803 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: kSnappyCompression supported: 1 2024-09-13T14:19:35.804 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-13T14:19:35.804 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-13T14:19:35.804 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000010 2024-09-13T14:19:35.804 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-13T14:19:35.804 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-13T14:19:35.804 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.merge_operator: 2024-09-13T14:19:35.804 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.compaction_filter: None 2024-09-13T14:19:35.804 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.compaction_filter_factory: None 2024-09-13T14:19:35.804 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.sst_partitioner_factory: None 2024-09-13T14:19:35.804 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-13T14:19:35.804 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-13T14:19:35.804 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56548ba394e0) 2024-09-13T14:19:35.804 INFO:journalctl@ceph.mon.b.smithi195.stdout: cache_index_and_filter_blocks: 1 2024-09-13T14:19:35.804 INFO:journalctl@ceph.mon.b.smithi195.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-13T14:19:35.804 INFO:journalctl@ceph.mon.b.smithi195.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-13T14:19:35.805 INFO:journalctl@ceph.mon.b.smithi195.stdout: pin_top_level_index_and_filter: 1 2024-09-13T14:19:35.805 INFO:journalctl@ceph.mon.b.smithi195.stdout: index_type: 0 2024-09-13T14:19:35.805 INFO:journalctl@ceph.mon.b.smithi195.stdout: data_block_index_type: 0 2024-09-13T14:19:35.805 INFO:journalctl@ceph.mon.b.smithi195.stdout: index_shortening: 1 2024-09-13T14:19:35.805 INFO:journalctl@ceph.mon.b.smithi195.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-13T14:19:35.805 INFO:journalctl@ceph.mon.b.smithi195.stdout: checksum: 4 2024-09-13T14:19:35.805 INFO:journalctl@ceph.mon.b.smithi195.stdout: no_block_cache: 0 2024-09-13T14:19:35.805 INFO:journalctl@ceph.mon.b.smithi195.stdout: block_cache: 0x56548ba2c850 2024-09-13T14:19:35.805 INFO:journalctl@ceph.mon.b.smithi195.stdout: block_cache_name: BinnedLRUCache 2024-09-13T14:19:35.805 INFO:journalctl@ceph.mon.b.smithi195.stdout: block_cache_options: 2024-09-13T14:19:35.805 INFO:journalctl@ceph.mon.b.smithi195.stdout: capacity : 536870912 2024-09-13T14:19:35.805 INFO:journalctl@ceph.mon.b.smithi195.stdout: num_shard_bits : 4 2024-09-13T14:19:35.805 INFO:journalctl@ceph.mon.b.smithi195.stdout: strict_capacity_limit : 0 2024-09-13T14:19:35.805 INFO:journalctl@ceph.mon.b.smithi195.stdout: high_pri_pool_ratio: 0.000 2024-09-13T14:19:35.805 INFO:journalctl@ceph.mon.b.smithi195.stdout: block_cache_compressed: (nil) 2024-09-13T14:19:35.806 INFO:journalctl@ceph.mon.b.smithi195.stdout: persistent_cache: (nil) 2024-09-13T14:19:35.806 INFO:journalctl@ceph.mon.b.smithi195.stdout: block_size: 4096 2024-09-13T14:19:35.806 INFO:journalctl@ceph.mon.b.smithi195.stdout: block_size_deviation: 10 2024-09-13T14:19:35.806 INFO:journalctl@ceph.mon.b.smithi195.stdout: block_restart_interval: 16 2024-09-13T14:19:35.806 INFO:journalctl@ceph.mon.b.smithi195.stdout: index_block_restart_interval: 1 2024-09-13T14:19:35.806 INFO:journalctl@ceph.mon.b.smithi195.stdout: metadata_block_size: 4096 2024-09-13T14:19:35.806 INFO:journalctl@ceph.mon.b.smithi195.stdout: partition_filters: 0 2024-09-13T14:19:35.807 INFO:journalctl@ceph.mon.b.smithi195.stdout: use_delta_encoding: 1 2024-09-13T14:19:35.807 INFO:journalctl@ceph.mon.b.smithi195.stdout: filter_policy: bloomfilter 2024-09-13T14:19:35.807 INFO:journalctl@ceph.mon.b.smithi195.stdout: whole_key_filtering: 1 2024-09-13T14:19:35.807 INFO:journalctl@ceph.mon.b.smithi195.stdout: verify_compression: 0 2024-09-13T14:19:35.807 INFO:journalctl@ceph.mon.b.smithi195.stdout: read_amp_bytes_per_bit: 0 2024-09-13T14:19:35.807 INFO:journalctl@ceph.mon.b.smithi195.stdout: format_version: 5 2024-09-13T14:19:35.807 INFO:journalctl@ceph.mon.b.smithi195.stdout: enable_index_compression: 1 2024-09-13T14:19:35.807 INFO:journalctl@ceph.mon.b.smithi195.stdout: block_align: 0 2024-09-13T14:19:35.807 INFO:journalctl@ceph.mon.b.smithi195.stdout: max_auto_readahead_size: 262144 2024-09-13T14:19:35.807 INFO:journalctl@ceph.mon.b.smithi195.stdout: prepopulate_block_cache: 0 2024-09-13T14:19:35.807 INFO:journalctl@ceph.mon.b.smithi195.stdout: initial_auto_readahead_size: 8192 2024-09-13T14:19:35.807 INFO:journalctl@ceph.mon.b.smithi195.stdout: num_file_reads_for_auto_readahead: 2 2024-09-13T14:19:35.807 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.write_buffer_size: 33554432 2024-09-13T14:19:35.807 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.max_write_buffer_number: 2 2024-09-13T14:19:35.807 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.compression: NoCompression 2024-09-13T14:19:35.808 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.bottommost_compression: Disabled 2024-09-13T14:19:35.808 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.prefix_extractor: nullptr 2024-09-13T14:19:35.808 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-13T14:19:35.808 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.num_levels: 7 2024-09-13T14:19:35.808 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-13T14:19:35.808 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-13T14:19:35.808 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-13T14:19:35.808 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-13T14:19:35.808 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-13T14:19:35.808 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-13T14:19:35.808 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-13T14:19:35.808 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-13T14:19:35.808 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-13T14:19:35.809 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-13T14:19:35.809 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-13T14:19:35.809 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-13T14:19:35.809 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-13T14:19:35.809 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.compression_opts.level: 32767 2024-09-13T14:19:35.809 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.compression_opts.strategy: 0 2024-09-13T14:19:35.809 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-13T14:19:35.809 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-13T14:19:35.809 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-13T14:19:35.809 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-13T14:19:35.809 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.compression_opts.enabled: false 2024-09-13T14:19:35.809 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-13T14:19:35.809 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-13T14:19:35.809 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-13T14:19:35.809 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-13T14:19:35.810 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.target_file_size_base: 67108864 2024-09-13T14:19:35.810 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-13T14:19:35.810 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-13T14:19:35.810 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-13T14:19:35.810 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-13T14:19:35.810 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-13T14:19:35.810 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-13T14:19:35.810 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-13T14:19:35.810 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-13T14:19:35.810 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-13T14:19:35.810 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-13T14:19:35.810 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-13T14:19:35.810 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-13T14:19:35.810 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-13T14:19:35.811 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-13T14:19:35.811 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.arena_block_size: 1048576 2024-09-13T14:19:35.811 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-13T14:19:35.811 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-13T14:19:35.811 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.disable_auto_compactions: 0 2024-09-13T14:19:35.811 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-13T14:19:35.811 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-13T14:19:35.811 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-13T14:19:35.811 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-13T14:19:35.811 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-13T14:19:35.811 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-13T14:19:35.811 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-13T14:19:35.811 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-13T14:19:35.811 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-13T14:19:35.811 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-13T14:19:35.812 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-13T14:19:35.812 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.inplace_update_support: 0 2024-09-13T14:19:35.812 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-13T14:19:35.812 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-13T14:19:35.812 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-13T14:19:35.812 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-13T14:19:35.812 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.bloom_locality: 0 2024-09-13T14:19:35.812 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.max_successive_merges: 0 2024-09-13T14:19:35.812 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-13T14:19:35.812 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.paranoid_file_checks: 0 2024-09-13T14:19:35.812 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.force_consistency_checks: 1 2024-09-13T14:19:35.812 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.report_bg_io_stats: 0 2024-09-13T14:19:35.812 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.ttl: 2592000 2024-09-13T14:19:35.812 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-13T14:19:35.813 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-13T14:19:35.813 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-13T14:19:35.813 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.enable_blob_files: false 2024-09-13T14:19:35.813 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.min_blob_size: 0 2024-09-13T14:19:35.813 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.blob_file_size: 268435456 2024-09-13T14:19:35.813 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-13T14:19:35.813 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-13T14:19:35.813 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-13T14:19:35.813 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-13T14:19:35.813 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-13T14:19:35.813 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.blob_file_starting_level: 0 2024-09-13T14:19:35.813 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-13T14:19:35.813 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-13T14:19:35.813 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 29, last_sequence is 13041, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 25 2024-09-13T14:19:35.814 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-09-13T14:19:35.814 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: e1378262-404a-4896-b800-2ede53023ed0 2024-09-13T14:19:35.814 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726237175794933, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-09-13T14:19:35.814 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-09-13T14:19:36.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726237175813641, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 4551991, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 13046, "largest_seqno": 16500, "table_properties": {"data_size": 4538690, "index_size": 8350, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 3909, "raw_key_size": 34903, "raw_average_key_size": 22, "raw_value_size": 4508144, "raw_average_value_size": 2910, "num_data_blocks": 375, "num_entries": 1549, "num_filter_entries": 1549, "num_deletions": 261, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726237175, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "e1378262-404a-4896-b800-2ede53023ed0", "db_session_id": "OTOKXEZ9S3BNPF3N6QLT", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-09-13T14:19:36.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726237175813766, "job": 1, "event": "recovery_finished"} 2024-09-13T14:19:36.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-09-13T14:19:36.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-13T14:19:36.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-13T14:19:36.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56548ba5ce00 2024-09-13T14:19:36.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: DB pointer 0x56548bb5c000 2024-09-13T14:19:36.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-13T14:19:36.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-13T14:19:36.248 INFO:journalctl@ceph.mon.b.smithi195.stdout: ** DB Stats ** 2024-09-13T14:19:36.248 INFO:journalctl@ceph.mon.b.smithi195.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T14:19:36.249 INFO:journalctl@ceph.mon.b.smithi195.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-13T14:19:36.249 INFO:journalctl@ceph.mon.b.smithi195.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-13T14:19:36.249 INFO:journalctl@ceph.mon.b.smithi195.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T14:19:36.249 INFO:journalctl@ceph.mon.b.smithi195.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-13T14:19:36.249 INFO:journalctl@ceph.mon.b.smithi195.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-13T14:19:36.249 INFO:journalctl@ceph.mon.b.smithi195.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T14:19:36.249 INFO:journalctl@ceph.mon.b.smithi195.stdout: 2024-09-13T14:19:36.249 INFO:journalctl@ceph.mon.b.smithi195.stdout: ** Compaction Stats [default] ** 2024-09-13T14:19:36.249 INFO:journalctl@ceph.mon.b.smithi195.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-13T14:19:36.249 INFO:journalctl@ceph.mon.b.smithi195.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-13T14:19:36.249 INFO:journalctl@ceph.mon.b.smithi195.stdout: L0 1/0 4.34 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 641.2 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-09-13T14:19:36.249 INFO:journalctl@ceph.mon.b.smithi195.stdout: L6 1/0 7.96 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-13T14:19:36.249 INFO:journalctl@ceph.mon.b.smithi195.stdout: Sum 2/0 12.30 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 641.2 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-09-13T14:19:36.250 INFO:journalctl@ceph.mon.b.smithi195.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 641.2 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-09-13T14:19:36.250 INFO:journalctl@ceph.mon.b.smithi195.stdout: 2024-09-13T14:19:36.250 INFO:journalctl@ceph.mon.b.smithi195.stdout: ** Compaction Stats [default] ** 2024-09-13T14:19:36.250 INFO:journalctl@ceph.mon.b.smithi195.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-13T14:19:36.250 INFO:journalctl@ceph.mon.b.smithi195.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-13T14:19:36.250 INFO:journalctl@ceph.mon.b.smithi195.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 641.2 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-09-13T14:19:36.250 INFO:journalctl@ceph.mon.b.smithi195.stdout: 2024-09-13T14:19:36.250 INFO:journalctl@ceph.mon.b.smithi195.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-13T14:19:36.250 INFO:journalctl@ceph.mon.b.smithi195.stdout: 2024-09-13T14:19:36.250 INFO:journalctl@ceph.mon.b.smithi195.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T14:19:36.250 INFO:journalctl@ceph.mon.b.smithi195.stdout: Flush(GB): cumulative 0.004, interval 0.004 2024-09-13T14:19:36.250 INFO:journalctl@ceph.mon.b.smithi195.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-13T14:19:36.250 INFO:journalctl@ceph.mon.b.smithi195.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-13T14:19:36.250 INFO:journalctl@ceph.mon.b.smithi195.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-13T14:19:36.250 INFO:journalctl@ceph.mon.b.smithi195.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-13T14:19:36.250 INFO:journalctl@ceph.mon.b.smithi195.stdout: Cumulative compaction: 0.00 GB write, 200.08 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T14:19:36.251 INFO:journalctl@ceph.mon.b.smithi195.stdout: Interval compaction: 0.00 GB write, 200.08 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T14:19:36.251 INFO:journalctl@ceph.mon.b.smithi195.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-13T14:19:36.251 INFO:journalctl@ceph.mon.b.smithi195.stdout: Block cache BinnedLRUCache@0x56548ba2c850#7 capacity: 512.00 MB usage: 13.11 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-09-13T14:19:36.251 INFO:journalctl@ceph.mon.b.smithi195.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,4.03 KB,0.0007689%) IndexBlock(1,9.08 KB,0.00173151%) Misc(1,0.00 KB,0%) 2024-09-13T14:19:36.251 INFO:journalctl@ceph.mon.b.smithi195.stdout: 2024-09-13T14:19:36.251 INFO:journalctl@ceph.mon.b.smithi195.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-13T14:19:36.251 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: starting mon.b rank 1 at public addrs [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] at bind addrs [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 08cee502-71d9-11ef-bceb-c7b262605968 2024-09-13T14:19:36.251 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: mon.b@-1(???) e3 preinit fsid 08cee502-71d9-11ef-bceb-c7b262605968 2024-09-13T14:19:36.251 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: mon.b@-1(???).mds e9 new map 2024-09-13T14:19:36.251 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: mon.b@-1(???).mds e9 print_map 2024-09-13T14:19:36.251 INFO:journalctl@ceph.mon.b.smithi195.stdout: e9 2024-09-13T14:19:36.251 INFO:journalctl@ceph.mon.b.smithi195.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-09-13T14:19:36.251 INFO:journalctl@ceph.mon.b.smithi195.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-13T14:19:36.251 INFO:journalctl@ceph.mon.b.smithi195.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-13T14:19:36.251 INFO:journalctl@ceph.mon.b.smithi195.stdout: legacy client fscid: 1 2024-09-13T14:19:36.252 INFO:journalctl@ceph.mon.b.smithi195.stdout: 2024-09-13T14:19:36.252 INFO:journalctl@ceph.mon.b.smithi195.stdout: Filesystem 'foo' (1) 2024-09-13T14:19:36.252 INFO:journalctl@ceph.mon.b.smithi195.stdout: fs_name foo 2024-09-13T14:19:36.252 INFO:journalctl@ceph.mon.b.smithi195.stdout: epoch 8 2024-09-13T14:19:36.252 INFO:journalctl@ceph.mon.b.smithi195.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-09-13T14:19:36.252 INFO:journalctl@ceph.mon.b.smithi195.stdout: created 2024-09-13T14:11:29.232738+0000 2024-09-13T14:19:36.252 INFO:journalctl@ceph.mon.b.smithi195.stdout: modified 2024-09-13T14:11:41.325298+0000 2024-09-13T14:19:36.252 INFO:journalctl@ceph.mon.b.smithi195.stdout: tableserver 0 2024-09-13T14:19:36.252 INFO:journalctl@ceph.mon.b.smithi195.stdout: root 0 2024-09-13T14:19:36.252 INFO:journalctl@ceph.mon.b.smithi195.stdout: session_timeout 60 2024-09-13T14:19:36.252 INFO:journalctl@ceph.mon.b.smithi195.stdout: session_autoclose 300 2024-09-13T14:19:36.252 INFO:journalctl@ceph.mon.b.smithi195.stdout: max_file_size 1099511627776 2024-09-13T14:19:36.252 INFO:journalctl@ceph.mon.b.smithi195.stdout: max_xattr_size 65536 2024-09-13T14:19:36.252 INFO:journalctl@ceph.mon.b.smithi195.stdout: required_client_features {} 2024-09-13T14:19:36.253 INFO:journalctl@ceph.mon.b.smithi195.stdout: last_failure 0 2024-09-13T14:19:36.253 INFO:journalctl@ceph.mon.b.smithi195.stdout: last_failure_osd_epoch 0 2024-09-13T14:19:36.253 INFO:journalctl@ceph.mon.b.smithi195.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-13T14:19:36.253 INFO:journalctl@ceph.mon.b.smithi195.stdout: max_mds 1 2024-09-13T14:19:36.253 INFO:journalctl@ceph.mon.b.smithi195.stdout: in 0 2024-09-13T14:19:36.253 INFO:journalctl@ceph.mon.b.smithi195.stdout: up {0=24521} 2024-09-13T14:19:36.253 INFO:journalctl@ceph.mon.b.smithi195.stdout: failed 2024-09-13T14:19:36.253 INFO:journalctl@ceph.mon.b.smithi195.stdout: damaged 2024-09-13T14:19:36.253 INFO:journalctl@ceph.mon.b.smithi195.stdout: stopped 2024-09-13T14:19:36.253 INFO:journalctl@ceph.mon.b.smithi195.stdout: data_pools [4] 2024-09-13T14:19:36.253 INFO:journalctl@ceph.mon.b.smithi195.stdout: metadata_pool 3 2024-09-13T14:19:36.253 INFO:journalctl@ceph.mon.b.smithi195.stdout: inline_data disabled 2024-09-13T14:19:36.253 INFO:journalctl@ceph.mon.b.smithi195.stdout: balancer 2024-09-13T14:19:36.254 INFO:journalctl@ceph.mon.b.smithi195.stdout: bal_rank_mask -1 2024-09-13T14:19:36.254 INFO:journalctl@ceph.mon.b.smithi195.stdout: standby_count_wanted 1 2024-09-13T14:19:36.254 INFO:journalctl@ceph.mon.b.smithi195.stdout: qdb_cluster leader: 0 members: 2024-09-13T14:19:36.254 INFO:journalctl@ceph.mon.b.smithi195.stdout: [mds.foo.smithi102.njetxr{0:24521} state up:active seq 3 join_fscid=1 addr [v2:172.21.15.102:6834/54188465,v1:172.21.15.102:6835/54188465] compat {c=[1],r=[1],i=[7ff]}] 2024-09-13T14:19:36.254 INFO:journalctl@ceph.mon.b.smithi195.stdout: 2024-09-13T14:19:36.254 INFO:journalctl@ceph.mon.b.smithi195.stdout: 2024-09-13T14:19:36.254 INFO:journalctl@ceph.mon.b.smithi195.stdout: Standby daemons: 2024-09-13T14:19:36.254 INFO:journalctl@ceph.mon.b.smithi195.stdout: 2024-09-13T14:19:36.254 INFO:journalctl@ceph.mon.b.smithi195.stdout: [mds.foo.smithi195.yqzrxi{-1:24442} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.195:6832/2017468914,v1:172.21.15.195:6833/2017468914] compat {c=[1],r=[1],i=[7ff]}] 2024-09-13T14:19:36.254 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: mon.b@-1(???).osd e90 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-13T14:19:36.254 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: mon.b@-1(???).osd e90 crush map has features 432629239337189376, adjusting msgr requires 2024-09-13T14:19:36.254 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: mon.b@-1(???).osd e90 crush map has features 432629239337189376, adjusting msgr requires 2024-09-13T14:19:36.254 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: mon.b@-1(???).osd e90 crush map has features 432629239337189376, adjusting msgr requires 2024-09-13T14:19:36.254 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 ceph-mon[104262]: mon.b@-1(???).paxosservice(auth 1..21) refresh upgraded, format 0 -> 3 2024-09-13T14:19:36.254 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 bash[104224]: 9921a2aa81682b9156cd45bc658f893cec490637ebe670d36db4a003bce58ba6 2024-09-13T14:19:36.254 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:35 smithi195 systemd[1]: Started Ceph mon.b for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:19:36.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:36 smithi195 ceph-mon[104262]: pgmap v67: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:36.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:36 smithi102 ceph-mon[110183]: pgmap v67: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:36.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:36 smithi102 ceph-mon[115269]: pgmap v67: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:36.919 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:19:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:19:36] "GET /metrics HTTP/1.1" 200 33814 "" "Prometheus/2.51.0" 2024-09-13T14:19:37.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:37 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-13T14:19:37.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:37 smithi195 ceph-mon[104262]: mon.b calling monitor election 2024-09-13T14:19:37.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:37 smithi195 ceph-mon[104262]: mon.c calling monitor election 2024-09-13T14:19:37.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:37 smithi195 ceph-mon[104262]: mon.a calling monitor election 2024-09-13T14:19:37.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:37 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T14:19:37.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:37 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T14:19:37.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:37 smithi195 ceph-mon[104262]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T14:19:37.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:37 smithi195 ceph-mon[104262]: monmap epoch 4 2024-09-13T14:19:37.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:37 smithi195 ceph-mon[104262]: fsid 08cee502-71d9-11ef-bceb-c7b262605968 2024-09-13T14:19:37.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:37 smithi195 ceph-mon[104262]: last_changed 2024-09-13T14:19:36.425251+0000 2024-09-13T14:19:37.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:37 smithi195 ceph-mon[104262]: created 2024-09-13T14:04:45.172288+0000 2024-09-13T14:19:37.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:37 smithi195 ceph-mon[104262]: min_mon_release 19 (squid) 2024-09-13T14:19:37.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:37 smithi195 ceph-mon[104262]: election_strategy: 1 2024-09-13T14:19:37.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:37 smithi195 ceph-mon[104262]: 0: [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon.a 2024-09-13T14:19:37.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:37 smithi195 ceph-mon[104262]: 1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2024-09-13T14:19:37.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:37 smithi195 ceph-mon[104262]: 2: [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] mon.c 2024-09-13T14:19:37.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:37 smithi195 ceph-mon[104262]: fsmap foo:1 {0=foo.smithi102.njetxr=up:active} 1 up:standby 2024-09-13T14:19:37.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:37 smithi195 ceph-mon[104262]: osdmap e90: 8 total, 8 up, 8 in 2024-09-13T14:19:37.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:37 smithi195 ceph-mon[104262]: mgrmap e41: y(active, since 2m), standbys: x 2024-09-13T14:19:37.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:37 smithi195 ceph-mon[104262]: overall HEALTH_OK 2024-09-13T14:19:37.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:37 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:37.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:37 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:37.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:37 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:37.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:37 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:19:37.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:37 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-13T14:19:37.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:37 smithi102 ceph-mon[110183]: mon.b calling monitor election 2024-09-13T14:19:37.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:37 smithi102 ceph-mon[110183]: mon.c calling monitor election 2024-09-13T14:19:37.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:37 smithi102 ceph-mon[110183]: mon.a calling monitor election 2024-09-13T14:19:37.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:37 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T14:19:37.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:37 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T14:19:37.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:37 smithi102 ceph-mon[110183]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T14:19:37.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:37 smithi102 ceph-mon[110183]: monmap epoch 4 2024-09-13T14:19:37.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:37 smithi102 ceph-mon[110183]: fsid 08cee502-71d9-11ef-bceb-c7b262605968 2024-09-13T14:19:37.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:37 smithi102 ceph-mon[110183]: last_changed 2024-09-13T14:19:36.425251+0000 2024-09-13T14:19:37.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:37 smithi102 ceph-mon[110183]: created 2024-09-13T14:04:45.172288+0000 2024-09-13T14:19:37.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:37 smithi102 ceph-mon[110183]: min_mon_release 19 (squid) 2024-09-13T14:19:37.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:37 smithi102 ceph-mon[110183]: election_strategy: 1 2024-09-13T14:19:37.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:37 smithi102 ceph-mon[110183]: 0: [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon.a 2024-09-13T14:19:37.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:37 smithi102 ceph-mon[110183]: 1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2024-09-13T14:19:37.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:37 smithi102 ceph-mon[110183]: 2: [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] mon.c 2024-09-13T14:19:37.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:37 smithi102 ceph-mon[110183]: fsmap foo:1 {0=foo.smithi102.njetxr=up:active} 1 up:standby 2024-09-13T14:19:37.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:37 smithi102 ceph-mon[110183]: osdmap e90: 8 total, 8 up, 8 in 2024-09-13T14:19:37.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:37 smithi102 ceph-mon[110183]: mgrmap e41: y(active, since 2m), standbys: x 2024-09-13T14:19:37.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:37 smithi102 ceph-mon[110183]: overall HEALTH_OK 2024-09-13T14:19:37.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:37 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:37.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:37 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:37.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:37 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:37.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:37 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:19:37.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:37 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-13T14:19:37.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:37 smithi102 ceph-mon[115269]: mon.b calling monitor election 2024-09-13T14:19:37.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:37 smithi102 ceph-mon[115269]: mon.c calling monitor election 2024-09-13T14:19:37.921 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:37 smithi102 ceph-mon[115269]: mon.a calling monitor election 2024-09-13T14:19:37.921 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:37 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T14:19:37.921 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:37 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T14:19:37.921 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:37 smithi102 ceph-mon[115269]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T14:19:37.921 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:37 smithi102 ceph-mon[115269]: monmap epoch 4 2024-09-13T14:19:37.921 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:37 smithi102 ceph-mon[115269]: fsid 08cee502-71d9-11ef-bceb-c7b262605968 2024-09-13T14:19:37.921 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:37 smithi102 ceph-mon[115269]: last_changed 2024-09-13T14:19:36.425251+0000 2024-09-13T14:19:37.921 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:37 smithi102 ceph-mon[115269]: created 2024-09-13T14:04:45.172288+0000 2024-09-13T14:19:37.921 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:37 smithi102 ceph-mon[115269]: min_mon_release 19 (squid) 2024-09-13T14:19:37.921 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:37 smithi102 ceph-mon[115269]: election_strategy: 1 2024-09-13T14:19:37.921 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:37 smithi102 ceph-mon[115269]: 0: [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon.a 2024-09-13T14:19:37.921 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:37 smithi102 ceph-mon[115269]: 1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2024-09-13T14:19:37.921 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:37 smithi102 ceph-mon[115269]: 2: [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] mon.c 2024-09-13T14:19:37.921 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:37 smithi102 ceph-mon[115269]: fsmap foo:1 {0=foo.smithi102.njetxr=up:active} 1 up:standby 2024-09-13T14:19:37.921 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:37 smithi102 ceph-mon[115269]: osdmap e90: 8 total, 8 up, 8 in 2024-09-13T14:19:37.921 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:37 smithi102 ceph-mon[115269]: mgrmap e41: y(active, since 2m), standbys: x 2024-09-13T14:19:37.922 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:37 smithi102 ceph-mon[115269]: overall HEALTH_OK 2024-09-13T14:19:37.922 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:37 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:37.922 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:37 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:37.922 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:37 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:37.922 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:37 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:19:38.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:38 smithi195 ceph-mon[104262]: pgmap v69: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:38.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:38 smithi102 ceph-mon[110183]: pgmap v69: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:38.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:38 smithi102 ceph-mon[115269]: pgmap v69: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:40.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:40 smithi195 ceph-mon[104262]: pgmap v70: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:40.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:40 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:19:40.825 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:40 smithi102 ceph-mon[110183]: pgmap v70: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:40.825 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:40 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:19:40.825 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:40 smithi102 ceph-mon[115269]: pgmap v70: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:40.825 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:40 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:19:41.168 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:19:40 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: 2024-09-13T14:19:40.824+0000 7f0c92e7e640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-13T14:19:41.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:41 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:41.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:41 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:41.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:41 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:41.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:41 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:41.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:41 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:41.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:41 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:41.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:41 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:41.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:41 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:41.919 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:41 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:41.919 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:41 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:41.919 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:41 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:41.919 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:41 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:42.938 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:42 smithi195 ceph-mon[104262]: pgmap v71: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:43.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:42 smithi102 ceph-mon[110183]: pgmap v71: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:43.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:42 smithi102 ceph-mon[115269]: pgmap v71: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:45.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:44 smithi102 ceph-mon[110183]: pgmap v72: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:45.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:44 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:45.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:44 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:45.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:44 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:19:45.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:44 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:19:45.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:44 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:45.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:44 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:19:45.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:44 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:19:45.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:44 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:19:45.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:44 smithi102 ceph-mon[115269]: pgmap v72: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:45.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:44 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:45.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:44 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:45.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:44 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:19:45.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:44 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:19:45.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:44 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:45.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:44 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:19:45.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:44 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:19:45.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:44 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:19:45.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:44 smithi195 ceph-mon[104262]: pgmap v72: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:45.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:44 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:45.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:44 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:45.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:44 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:19:45.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:44 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:19:45.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:44 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:45.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:44 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:19:45.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:44 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:19:45.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:44 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:19:45.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:45 smithi102 ceph-mon[110183]: Reconfiguring mon.a (monmap changed)... 2024-09-13T14:19:45.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:45 smithi102 ceph-mon[110183]: Reconfiguring daemon mon.a on smithi102 2024-09-13T14:19:45.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:45 smithi102 ceph-mon[115269]: Reconfiguring mon.a (monmap changed)... 2024-09-13T14:19:45.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:45 smithi102 ceph-mon[115269]: Reconfiguring daemon mon.a on smithi102 2024-09-13T14:19:46.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:45 smithi195 ceph-mon[104262]: Reconfiguring mon.a (monmap changed)... 2024-09-13T14:19:46.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:45 smithi195 ceph-mon[104262]: Reconfiguring daemon mon.a on smithi102 2024-09-13T14:19:46.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:19:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:19:46] "GET /metrics HTTP/1.1" 200 33754 "" "Prometheus/2.51.0" 2024-09-13T14:19:47.327 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:47 smithi102 ceph-mon[115269]: pgmap v73: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:47.327 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:47 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:47.327 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:47 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:47.328 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:47 smithi102 ceph-mon[115269]: Reconfiguring mgr.y (monmap changed)... 2024-09-13T14:19:47.328 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:47 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-13T14:19:47.328 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:47 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-13T14:19:47.328 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:47 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:19:47.328 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:47 smithi102 ceph-mon[115269]: Reconfiguring daemon mgr.y on smithi102 2024-09-13T14:19:47.328 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:47 smithi102 ceph-mon[110183]: pgmap v73: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:47.328 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:47 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:47.328 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:47 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:47.328 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:47 smithi102 ceph-mon[110183]: Reconfiguring mgr.y (monmap changed)... 2024-09-13T14:19:47.328 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:47 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-13T14:19:47.328 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:47 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-13T14:19:47.329 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:47 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:19:47.329 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:47 smithi102 ceph-mon[110183]: Reconfiguring daemon mgr.y on smithi102 2024-09-13T14:19:47.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:47 smithi195 ceph-mon[104262]: pgmap v73: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:47.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:47 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:47.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:47 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:47.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:47 smithi195 ceph-mon[104262]: Reconfiguring mgr.y (monmap changed)... 2024-09-13T14:19:47.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:47 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-13T14:19:47.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:47 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-13T14:19:47.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:47 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:19:47.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:47 smithi195 ceph-mon[104262]: Reconfiguring daemon mgr.y on smithi102 2024-09-13T14:19:48.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:48 smithi102 ceph-mon[110183]: pgmap v74: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:48.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:48 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:48.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:48 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:48.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:48 smithi102 ceph-mon[110183]: Reconfiguring mon.c (monmap changed)... 2024-09-13T14:19:48.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:48 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:19:48.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:48 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:19:48.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:48 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:19:48.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:48 smithi102 ceph-mon[110183]: Reconfiguring daemon mon.c on smithi102 2024-09-13T14:19:48.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:48 smithi102 ceph-mon[115269]: pgmap v74: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:48.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:48 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:48.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:48 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:48.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:48 smithi102 ceph-mon[115269]: Reconfiguring mon.c (monmap changed)... 2024-09-13T14:19:48.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:48 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:19:48.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:48 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:19:48.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:48 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:19:48.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:48 smithi102 ceph-mon[115269]: Reconfiguring daemon mon.c on smithi102 2024-09-13T14:19:48.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:48 smithi195 ceph-mon[104262]: pgmap v74: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:48.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:48 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:48.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:48 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:48.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:48 smithi195 ceph-mon[104262]: Reconfiguring mon.c (monmap changed)... 2024-09-13T14:19:48.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:48 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:19:48.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:48 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:19:48.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:48 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:19:48.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:48 smithi195 ceph-mon[104262]: Reconfiguring daemon mon.c on smithi102 2024-09-13T14:19:50.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:49 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:50.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:49 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:50.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:49 smithi102 ceph-mon[110183]: Reconfiguring osd.0 (monmap changed)... 2024-09-13T14:19:50.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:49 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-13T14:19:50.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:49 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:19:50.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:49 smithi102 ceph-mon[110183]: Reconfiguring daemon osd.0 on smithi102 2024-09-13T14:19:50.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:49 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:50.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:49 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:50.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:49 smithi102 ceph-mon[115269]: Reconfiguring osd.0 (monmap changed)... 2024-09-13T14:19:50.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:49 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-13T14:19:50.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:49 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:19:50.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:49 smithi102 ceph-mon[115269]: Reconfiguring daemon osd.0 on smithi102 2024-09-13T14:19:50.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:49 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:50.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:49 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:50.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:49 smithi195 ceph-mon[104262]: Reconfiguring osd.0 (monmap changed)... 2024-09-13T14:19:50.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:49 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-13T14:19:50.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:49 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:19:50.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:49 smithi195 ceph-mon[104262]: Reconfiguring daemon osd.0 on smithi102 2024-09-13T14:19:50.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:50 smithi102 ceph-mon[115269]: pgmap v75: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:50.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:50 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:50.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:50 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:50.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:50 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-13T14:19:50.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:50 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:19:50.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:50 smithi102 ceph-mon[110183]: pgmap v75: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:50.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:50 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:50.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:50 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:50.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:50 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-13T14:19:50.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:50 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:19:51.149 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:50 smithi195 ceph-mon[104262]: pgmap v75: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:51.149 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:50 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:51.149 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:50 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:51.149 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:50 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-13T14:19:51.149 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:50 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:19:52.083 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:51 smithi102 ceph-mon[110183]: Reconfiguring osd.1 (monmap changed)... 2024-09-13T14:19:52.083 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:51 smithi102 ceph-mon[110183]: Reconfiguring daemon osd.1 on smithi102 2024-09-13T14:19:52.083 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:51 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:52.083 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:51 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:52.083 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:51 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-13T14:19:52.084 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:51 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:19:52.084 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:51 smithi102 ceph-mon[115269]: Reconfiguring osd.1 (monmap changed)... 2024-09-13T14:19:52.084 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:51 smithi102 ceph-mon[115269]: Reconfiguring daemon osd.1 on smithi102 2024-09-13T14:19:52.084 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:51 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:52.084 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:51 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:52.084 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:51 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-13T14:19:52.084 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:51 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:19:52.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:51 smithi195 ceph-mon[104262]: Reconfiguring osd.1 (monmap changed)... 2024-09-13T14:19:52.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:51 smithi195 ceph-mon[104262]: Reconfiguring daemon osd.1 on smithi102 2024-09-13T14:19:52.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:51 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:52.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:51 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:52.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:51 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-13T14:19:52.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:51 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:19:53.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:52 smithi102 ceph-mon[115269]: pgmap v76: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:53.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:52 smithi102 ceph-mon[115269]: Reconfiguring osd.2 (monmap changed)... 2024-09-13T14:19:53.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:52 smithi102 ceph-mon[115269]: Reconfiguring daemon osd.2 on smithi102 2024-09-13T14:19:53.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:52 smithi102 ceph-mon[110183]: pgmap v76: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:53.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:52 smithi102 ceph-mon[110183]: Reconfiguring osd.2 (monmap changed)... 2024-09-13T14:19:53.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:52 smithi102 ceph-mon[110183]: Reconfiguring daemon osd.2 on smithi102 2024-09-13T14:19:53.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:52 smithi195 ceph-mon[104262]: pgmap v76: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:53.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:52 smithi195 ceph-mon[104262]: Reconfiguring osd.2 (monmap changed)... 2024-09-13T14:19:53.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:52 smithi195 ceph-mon[104262]: Reconfiguring daemon osd.2 on smithi102 2024-09-13T14:19:54.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:54 smithi102 ceph-mon[110183]: pgmap v77: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:54.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:54 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:54.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:54 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:54.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:54 smithi102 ceph-mon[110183]: Reconfiguring osd.3 (monmap changed)... 2024-09-13T14:19:54.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:54 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-13T14:19:54.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:54 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:19:54.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:54 smithi102 ceph-mon[110183]: Reconfiguring daemon osd.3 on smithi102 2024-09-13T14:19:54.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:54 smithi102 ceph-mon[115269]: pgmap v77: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:54.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:54 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:54.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:54 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:54.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:54 smithi102 ceph-mon[115269]: Reconfiguring osd.3 (monmap changed)... 2024-09-13T14:19:54.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:54 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-13T14:19:54.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:54 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:19:54.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:54 smithi102 ceph-mon[115269]: Reconfiguring daemon osd.3 on smithi102 2024-09-13T14:19:54.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:54 smithi195 ceph-mon[104262]: pgmap v77: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:54.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:54 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:54.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:54 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:54.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:54 smithi195 ceph-mon[104262]: Reconfiguring osd.3 (monmap changed)... 2024-09-13T14:19:54.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:54 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-13T14:19:54.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:54 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:19:54.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:54 smithi195 ceph-mon[104262]: Reconfiguring daemon osd.3 on smithi102 2024-09-13T14:19:56.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:55 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:56.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:55 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:56.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:55 smithi102 ceph-mon[115269]: Reconfiguring mds.foo.smithi102.njetxr (monmap changed)... 2024-09-13T14:19:56.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:55 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi102.njetxr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-13T14:19:56.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:55 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:19:56.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:55 smithi102 ceph-mon[115269]: Reconfiguring daemon mds.foo.smithi102.njetxr on smithi102 2024-09-13T14:19:56.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:55 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:19:56.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:55 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:56.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:55 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:56.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:55 smithi102 ceph-mon[110183]: Reconfiguring mds.foo.smithi102.njetxr (monmap changed)... 2024-09-13T14:19:56.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:55 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi102.njetxr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-13T14:19:56.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:55 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:19:56.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:55 smithi102 ceph-mon[110183]: Reconfiguring daemon mds.foo.smithi102.njetxr on smithi102 2024-09-13T14:19:56.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:55 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:19:56.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:55 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:56.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:55 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:56.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:55 smithi195 ceph-mon[104262]: Reconfiguring mds.foo.smithi102.njetxr (monmap changed)... 2024-09-13T14:19:56.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:55 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi102.njetxr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-13T14:19:56.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:55 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:19:56.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:55 smithi195 ceph-mon[104262]: Reconfiguring daemon mds.foo.smithi102.njetxr on smithi102 2024-09-13T14:19:56.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:55 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:19:56.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:19:56 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:19:56] "GET /metrics HTTP/1.1" 200 33821 "" "Prometheus/2.51.0" 2024-09-13T14:19:56.900 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:56 smithi195 ceph-mon[104262]: pgmap v78: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:56.900 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:56.900 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:56.900 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:19:56.900 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:19:56.900 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:19:57.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:56 smithi102 ceph-mon[110183]: pgmap v78: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:57.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:57.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:57.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:19:57.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:19:57.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:19:57.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:56 smithi102 ceph-mon[115269]: pgmap v78: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:57.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:57.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:57.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:19:57.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:19:57.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:19:58.067 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:57 smithi195 ceph-mon[104262]: Reconfiguring mon.b (monmap changed)... 2024-09-13T14:19:58.067 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:57 smithi195 ceph-mon[104262]: Reconfiguring daemon mon.b on smithi195 2024-09-13T14:19:58.067 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:57 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:58.068 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:57 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:58.068 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:57 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-13T14:19:58.068 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:57 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-13T14:19:58.068 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:57 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:19:58.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:57 smithi102 ceph-mon[110183]: Reconfiguring mon.b (monmap changed)... 2024-09-13T14:19:58.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:57 smithi102 ceph-mon[110183]: Reconfiguring daemon mon.b on smithi195 2024-09-13T14:19:58.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:57 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:58.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:57 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:58.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:57 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-13T14:19:58.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:57 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-13T14:19:58.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:57 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:19:58.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:57 smithi102 ceph-mon[115269]: Reconfiguring mon.b (monmap changed)... 2024-09-13T14:19:58.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:57 smithi102 ceph-mon[115269]: Reconfiguring daemon mon.b on smithi195 2024-09-13T14:19:58.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:57 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:58.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:57 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:19:58.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:57 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-13T14:19:58.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:57 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-13T14:19:58.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:57 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:19:58.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:58 smithi195 ceph-mon[104262]: pgmap v79: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:58.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:58 smithi195 ceph-mon[104262]: Reconfiguring mgr.x (monmap changed)... 2024-09-13T14:19:58.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:19:58 smithi195 ceph-mon[104262]: Reconfiguring daemon mgr.x on smithi195 2024-09-13T14:19:59.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:58 smithi102 ceph-mon[110183]: pgmap v79: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:59.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:58 smithi102 ceph-mon[110183]: Reconfiguring mgr.x (monmap changed)... 2024-09-13T14:19:59.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:19:58 smithi102 ceph-mon[110183]: Reconfiguring daemon mgr.x on smithi195 2024-09-13T14:19:59.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:58 smithi102 ceph-mon[115269]: pgmap v79: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:19:59.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:58 smithi102 ceph-mon[115269]: Reconfiguring mgr.x (monmap changed)... 2024-09-13T14:19:59.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:19:58 smithi102 ceph-mon[115269]: Reconfiguring daemon mgr.x on smithi195 2024-09-13T14:20:00.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:00 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:00.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:00 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:00.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:00 smithi102 ceph-mon[110183]: Reconfiguring osd.4 (monmap changed)... 2024-09-13T14:20:00.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:00 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-13T14:20:00.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:00 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:20:00.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:00 smithi102 ceph-mon[110183]: Reconfiguring daemon osd.4 on smithi195 2024-09-13T14:20:00.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:00 smithi102 ceph-mon[110183]: overall HEALTH_OK 2024-09-13T14:20:00.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:00 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:00.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:00 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:00.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:00 smithi102 ceph-mon[115269]: Reconfiguring osd.4 (monmap changed)... 2024-09-13T14:20:00.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:00 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-13T14:20:00.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:00 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:20:00.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:00 smithi102 ceph-mon[115269]: Reconfiguring daemon osd.4 on smithi195 2024-09-13T14:20:00.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:00 smithi102 ceph-mon[115269]: overall HEALTH_OK 2024-09-13T14:20:00.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:00 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:00.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:00 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:00.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:00 smithi195 ceph-mon[104262]: Reconfiguring osd.4 (monmap changed)... 2024-09-13T14:20:00.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:00 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-13T14:20:00.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:00 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:20:00.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:00 smithi195 ceph-mon[104262]: Reconfiguring daemon osd.4 on smithi195 2024-09-13T14:20:00.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:00 smithi195 ceph-mon[104262]: overall HEALTH_OK 2024-09-13T14:20:01.314 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:01 smithi195 ceph-mon[104262]: pgmap v80: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:01.314 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:01 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:01.314 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:01 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:01.314 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:01 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-13T14:20:01.314 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:01 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:20:01.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:01 smithi102 ceph-mon[110183]: pgmap v80: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:01.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:01 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:01.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:01 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:01.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:01 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-13T14:20:01.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:01 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:20:01.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:01 smithi102 ceph-mon[115269]: pgmap v80: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:01.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:01 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:01.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:01 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:01.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:01 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-13T14:20:01.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:01 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:20:01.574 INFO:teuthology.orchestra.run.smithi102.stdout:true 2024-09-13T14:20:02.130 INFO:teuthology.orchestra.run.smithi102.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-13T14:20:02.130 INFO:teuthology.orchestra.run.smithi102.stdout:alertmanager.a smithi102 *:9093,9094 running (3m) 65s ago 9m 19.2M - 0.27.0 11f11916f8cd 0322b99ec767 2024-09-13T14:20:02.131 INFO:teuthology.orchestra.run.smithi102.stdout:grafana.a smithi195 *:3000 running (3m) 21s ago 9m 84.4M - c8b91775d855 152e89bad2c8 2024-09-13T14:20:02.131 INFO:teuthology.orchestra.run.smithi102.stdout:mds.foo.smithi102.njetxr smithi102 running (8m) 65s ago 8m 21.2M - 18.2.4-843-g2c01458a c129f5a2fd4e c9211cc725ed 2024-09-13T14:20:02.131 INFO:teuthology.orchestra.run.smithi102.stdout:mds.foo.smithi195.yqzrxi smithi195 running (8m) 21s ago 8m 18.4M - 18.2.4-843-g2c01458a c129f5a2fd4e eea37a7230fd 2024-09-13T14:20:02.131 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.x smithi195 *:8443,9283,8765 running (4m) 21s ago 13m 454M - 19.3.0-4697-gd58f7306 fec07016a2b0 4a23b8f0ea8b 2024-09-13T14:20:02.131 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.y smithi102 *:8443,9283,8765 running (5m) 65s ago 15m 507M - 19.3.0-4697-gd58f7306 fec07016a2b0 6b12b28dcde1 2024-09-13T14:20:02.131 INFO:teuthology.orchestra.run.smithi102.stdout:mon.a smithi102 running (2m) 65s ago 15m 41.7M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 f3234b7bbdde 2024-09-13T14:20:02.131 INFO:teuthology.orchestra.run.smithi102.stdout:mon.b smithi195 running (26s) 21s ago 13m 28.3M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 9921a2aa8168 2024-09-13T14:20:02.131 INFO:teuthology.orchestra.run.smithi102.stdout:mon.c smithi102 running (2m) 65s ago 13m 31.7M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 dba6b3d55296 2024-09-13T14:20:02.131 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.a smithi102 *:9100 running (4m) 65s ago 9m 11.7M - 1.7.0 72c9c2088986 0c92d0b39468 2024-09-13T14:20:02.131 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.b smithi195 *:9100 running (4m) 21s ago 9m 11.0M - 1.7.0 72c9c2088986 fde867a7dbf9 2024-09-13T14:20:02.131 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0 smithi102 running (13m) 65s ago 13m 60.7M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e b97482fb8cf9 2024-09-13T14:20:02.131 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1 smithi102 running (12m) 65s ago 12m 52.2M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 8a67356be7a2 2024-09-13T14:20:02.131 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2 smithi102 running (12m) 65s ago 12m 50.3M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 1800986ac838 2024-09-13T14:20:02.131 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3 smithi102 running (12m) 65s ago 12m 56.7M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 968246da9da2 2024-09-13T14:20:02.131 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4 smithi195 running (11m) 21s ago 11m 49.7M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 515307701d73 2024-09-13T14:20:02.132 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5 smithi195 running (11m) 21s ago 11m 53.6M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 176ba2b3bd59 2024-09-13T14:20:02.132 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6 smithi195 running (10m) 21s ago 10m 55.5M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 9cd070a6dd3d 2024-09-13T14:20:02.132 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7 smithi195 running (10m) 21s ago 10m 53.6M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e a6b698c659ca 2024-09-13T14:20:02.132 INFO:teuthology.orchestra.run.smithi102.stdout:prometheus.a smithi195 *:9095 running (4m) 21s ago 9m 43.8M - 2.51.0 1d3b7f56885b d131e53f1ef9 2024-09-13T14:20:02.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:02 smithi102 ceph-mon[110183]: Reconfiguring osd.5 (monmap changed)... 2024-09-13T14:20:02.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:02 smithi102 ceph-mon[110183]: Reconfiguring daemon osd.5 on smithi195 2024-09-13T14:20:02.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:02 smithi102 ceph-mon[115269]: Reconfiguring osd.5 (monmap changed)... 2024-09-13T14:20:02.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:02 smithi102 ceph-mon[115269]: Reconfiguring daemon osd.5 on smithi195 2024-09-13T14:20:02.458 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:02 smithi195 ceph-mon[104262]: Reconfiguring osd.5 (monmap changed)... 2024-09-13T14:20:02.458 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:02 smithi195 ceph-mon[104262]: Reconfiguring daemon osd.5 on smithi195 2024-09-13T14:20:02.483 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:20:02.483 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": { 2024-09-13T14:20:02.483 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 3 2024-09-13T14:20:02.483 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:20:02.483 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": { 2024-09-13T14:20:02.483 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 2 2024-09-13T14:20:02.483 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:20:02.483 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": { 2024-09-13T14:20:02.483 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 8 2024-09-13T14:20:02.484 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:20:02.484 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": { 2024-09-13T14:20:02.484 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 2 2024-09-13T14:20:02.484 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:20:02.484 INFO:teuthology.orchestra.run.smithi102.stdout: "overall": { 2024-09-13T14:20:02.484 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 10, 2024-09-13T14:20:02.484 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 5 2024-09-13T14:20:02.484 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:20:02.484 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:20:02.766 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:20:02.766 INFO:teuthology.orchestra.run.smithi102.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f", 2024-09-13T14:20:02.766 INFO:teuthology.orchestra.run.smithi102.stdout: "in_progress": true, 2024-09-13T14:20:02.766 INFO:teuthology.orchestra.run.smithi102.stdout: "which": "Upgrading daemons of type(s) mon. Upgrade limited to 1 daemons (0 remaining).", 2024-09-13T14:20:02.766 INFO:teuthology.orchestra.run.smithi102.stdout: "services_complete": [ 2024-09-13T14:20:02.766 INFO:teuthology.orchestra.run.smithi102.stdout: "mon" 2024-09-13T14:20:02.766 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-09-13T14:20:02.766 INFO:teuthology.orchestra.run.smithi102.stdout: "progress": "3/3 daemons upgraded", 2024-09-13T14:20:02.766 INFO:teuthology.orchestra.run.smithi102.stdout: "message": "Currently upgrading mon daemons", 2024-09-13T14:20:02.767 INFO:teuthology.orchestra.run.smithi102.stdout: "is_paused": false 2024-09-13T14:20:02.767 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:20:03.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:03 smithi102 ceph-mon[110183]: pgmap v81: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:03.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:03 smithi102 ceph-mon[110183]: from='client.34221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:20:03.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:03 smithi102 ceph-mon[110183]: from='client.34227 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:20:03.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:03 smithi102 ceph-mon[110183]: from='client.34233 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:20:03.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:03 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:03.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:03 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:03.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:03 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-13T14:20:03.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:03 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:20:03.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:03 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/549650221' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:20:03.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:03 smithi102 ceph-mon[115269]: pgmap v81: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:03.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:03 smithi102 ceph-mon[115269]: from='client.34221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:20:03.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:03 smithi102 ceph-mon[115269]: from='client.34227 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:20:03.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:03 smithi102 ceph-mon[115269]: from='client.34233 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:20:03.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:03 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:03.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:03 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:03.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:03 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-13T14:20:03.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:03 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:20:03.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:03 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/549650221' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:20:03.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:03 smithi195 ceph-mon[104262]: pgmap v81: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:03.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:03 smithi195 ceph-mon[104262]: from='client.34221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:20:03.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:03 smithi195 ceph-mon[104262]: from='client.34227 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:20:03.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:03 smithi195 ceph-mon[104262]: from='client.34233 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:20:03.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:03 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:03.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:03 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:03.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:03 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-13T14:20:03.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:03 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:20:03.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:03 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/549650221' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:20:04.345 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:04 smithi195 ceph-mon[104262]: Reconfiguring osd.6 (monmap changed)... 2024-09-13T14:20:04.345 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:04 smithi195 ceph-mon[104262]: Reconfiguring daemon osd.6 on smithi195 2024-09-13T14:20:04.345 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:04 smithi195 ceph-mon[104262]: from='client.34245 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:20:04.345 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:04 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:04.345 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:04 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:04.346 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:04 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-13T14:20:04.346 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:04 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:20:04.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:04 smithi102 ceph-mon[115269]: Reconfiguring osd.6 (monmap changed)... 2024-09-13T14:20:04.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:04 smithi102 ceph-mon[115269]: Reconfiguring daemon osd.6 on smithi195 2024-09-13T14:20:04.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:04 smithi102 ceph-mon[115269]: from='client.34245 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:20:04.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:04 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:04.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:04 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:04.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:04 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-13T14:20:04.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:04 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:20:04.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:04 smithi102 ceph-mon[110183]: Reconfiguring osd.6 (monmap changed)... 2024-09-13T14:20:04.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:04 smithi102 ceph-mon[110183]: Reconfiguring daemon osd.6 on smithi195 2024-09-13T14:20:04.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:04 smithi102 ceph-mon[110183]: from='client.34245 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:20:04.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:04 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:04.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:04 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:04.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:04 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-13T14:20:04.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:04 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:20:05.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:05 smithi102 ceph-mon[115269]: pgmap v82: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:05.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:05 smithi102 ceph-mon[115269]: Reconfiguring osd.7 (monmap changed)... 2024-09-13T14:20:05.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:05 smithi102 ceph-mon[115269]: Reconfiguring daemon osd.7 on smithi195 2024-09-13T14:20:05.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:05 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:05.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:05 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:05.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:05 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi195.yqzrxi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-13T14:20:05.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:05 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:20:05.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:05 smithi102 ceph-mon[110183]: pgmap v82: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:05.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:05 smithi102 ceph-mon[110183]: Reconfiguring osd.7 (monmap changed)... 2024-09-13T14:20:05.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:05 smithi102 ceph-mon[110183]: Reconfiguring daemon osd.7 on smithi195 2024-09-13T14:20:05.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:05 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:05.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:05 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:05.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:05 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi195.yqzrxi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-13T14:20:05.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:05 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:20:05.471 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:05 smithi195 ceph-mon[104262]: pgmap v82: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:05.471 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:05 smithi195 ceph-mon[104262]: Reconfiguring osd.7 (monmap changed)... 2024-09-13T14:20:05.471 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:05 smithi195 ceph-mon[104262]: Reconfiguring daemon osd.7 on smithi195 2024-09-13T14:20:05.471 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:05 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:05.471 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:05 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:05.471 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:05 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi195.yqzrxi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-13T14:20:05.471 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:05 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:20:06.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:06 smithi102 ceph-mon[115269]: Reconfiguring mds.foo.smithi195.yqzrxi (monmap changed)... 2024-09-13T14:20:06.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:06 smithi102 ceph-mon[115269]: Reconfiguring daemon mds.foo.smithi195.yqzrxi on smithi195 2024-09-13T14:20:06.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:06 smithi102 ceph-mon[110183]: Reconfiguring mds.foo.smithi195.yqzrxi (monmap changed)... 2024-09-13T14:20:06.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:06 smithi102 ceph-mon[110183]: Reconfiguring daemon mds.foo.smithi195.yqzrxi on smithi195 2024-09-13T14:20:06.447 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:06 smithi195 ceph-mon[104262]: Reconfiguring mds.foo.smithi195.yqzrxi (monmap changed)... 2024-09-13T14:20:06.448 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:06 smithi195 ceph-mon[104262]: Reconfiguring daemon mds.foo.smithi195.yqzrxi on smithi195 2024-09-13T14:20:06.918 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:20:06 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:20:06] "GET /metrics HTTP/1.1" 200 33821 "" "Prometheus/2.51.0" 2024-09-13T14:20:07.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: pgmap v83: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:07.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:07.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:07.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:20:07.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:20:07.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:20:07.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: Upgrade: Setting container_image for all nvmeof 2024-09-13T14:20:07.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:07.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:20:07.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: Upgrade: Setting container_image for all node-exporter 2024-09-13T14:20:07.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:07.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-13T14:20:07.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-09-13T14:20:07.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:20:07.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: Upgrade: Setting container_image for all prometheus 2024-09-13T14:20:07.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-13T14:20:07.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:20:07.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: Upgrade: Setting container_image for all alertmanager 2024-09-13T14:20:07.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-13T14:20:07.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:20:07.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: Upgrade: Setting container_image for all grafana 2024-09-13T14:20:07.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-13T14:20:07.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:20:07.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: Upgrade: Setting container_image for all loki 2024-09-13T14:20:07.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-13T14:20:07.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:20:07.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: Upgrade: Setting container_image for all promtail 2024-09-13T14:20:07.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-13T14:20:07.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: Upgrade: Finalizing container_image settings 2024-09-13T14:20:07.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr"} : dispatch 2024-09-13T14:20:07.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: pgmap v83: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:07.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:07.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:07.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:20:07.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:20:07.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:20:07.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: Upgrade: Setting container_image for all nvmeof 2024-09-13T14:20:07.422 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:07.422 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:20:07.422 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: Upgrade: Setting container_image for all node-exporter 2024-09-13T14:20:07.422 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:07.422 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-13T14:20:07.422 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-09-13T14:20:07.422 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:20:07.422 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: Upgrade: Setting container_image for all prometheus 2024-09-13T14:20:07.422 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-13T14:20:07.422 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:20:07.422 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: Upgrade: Setting container_image for all alertmanager 2024-09-13T14:20:07.422 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-13T14:20:07.422 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:20:07.422 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: Upgrade: Setting container_image for all grafana 2024-09-13T14:20:07.422 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-13T14:20:07.422 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:20:07.423 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: Upgrade: Setting container_image for all loki 2024-09-13T14:20:07.423 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-13T14:20:07.423 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:20:07.423 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: Upgrade: Setting container_image for all promtail 2024-09-13T14:20:07.423 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-13T14:20:07.423 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: Upgrade: Finalizing container_image settings 2024-09-13T14:20:07.423 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr"} : dispatch 2024-09-13T14:20:07.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-13T14:20:07.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:20:07.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-13T14:20:07.424 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.crash"} : dispatch 2024-09-13T14:20:07.424 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd"} : dispatch 2024-09-13T14:20:07.424 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mds"} : dispatch 2024-09-13T14:20:07.424 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw"} : dispatch 2024-09-13T14:20:07.424 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"} : dispatch 2024-09-13T14:20:07.424 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:20:07.424 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"} : dispatch 2024-09-13T14:20:07.424 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.iscsi"} : dispatch 2024-09-13T14:20:07.424 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nfs"} : dispatch 2024-09-13T14:20:07.424 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"} : dispatch 2024-09-13T14:20:07.424 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-13T14:20:07.424 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:20:07.424 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:20:07.424 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:20:07.425 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:20:07.425 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:20:07.425 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:20:07.425 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: Upgrade: Complete! 2024-09-13T14:20:07.425 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"} : dispatch 2024-09-13T14:20:07.425 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-13T14:20:07.425 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:20:07.425 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-13T14:20:07.425 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:20:07.425 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-13T14:20:07.426 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.crash"} : dispatch 2024-09-13T14:20:07.426 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd"} : dispatch 2024-09-13T14:20:07.426 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mds"} : dispatch 2024-09-13T14:20:07.426 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw"} : dispatch 2024-09-13T14:20:07.426 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"} : dispatch 2024-09-13T14:20:07.426 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:20:07.426 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"} : dispatch 2024-09-13T14:20:07.426 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.iscsi"} : dispatch 2024-09-13T14:20:07.426 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nfs"} : dispatch 2024-09-13T14:20:07.426 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"} : dispatch 2024-09-13T14:20:07.426 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-13T14:20:07.426 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:20:07.426 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:20:07.427 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:20:07.427 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:20:07.427 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:20:07.427 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:20:07.427 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: Upgrade: Complete! 2024-09-13T14:20:07.427 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"} : dispatch 2024-09-13T14:20:07.427 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-13T14:20:07.427 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:07 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:20:07.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: pgmap v83: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:07.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:07.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:07.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:20:07.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:20:07.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:20:07.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: Upgrade: Setting container_image for all nvmeof 2024-09-13T14:20:07.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:07.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:20:07.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: Upgrade: Setting container_image for all node-exporter 2024-09-13T14:20:07.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:07.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-13T14:20:07.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-09-13T14:20:07.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:20:07.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: Upgrade: Setting container_image for all prometheus 2024-09-13T14:20:07.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-13T14:20:07.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:20:07.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: Upgrade: Setting container_image for all alertmanager 2024-09-13T14:20:07.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-13T14:20:07.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:20:07.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: Upgrade: Setting container_image for all grafana 2024-09-13T14:20:07.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-13T14:20:07.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:20:07.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: Upgrade: Setting container_image for all loki 2024-09-13T14:20:07.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-13T14:20:07.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:20:07.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: Upgrade: Setting container_image for all promtail 2024-09-13T14:20:07.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-13T14:20:07.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: Upgrade: Finalizing container_image settings 2024-09-13T14:20:07.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr"} : dispatch 2024-09-13T14:20:07.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-13T14:20:07.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:20:07.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-13T14:20:07.501 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.crash"} : dispatch 2024-09-13T14:20:07.501 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd"} : dispatch 2024-09-13T14:20:07.501 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mds"} : dispatch 2024-09-13T14:20:07.501 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw"} : dispatch 2024-09-13T14:20:07.501 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"} : dispatch 2024-09-13T14:20:07.501 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:20:07.501 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"} : dispatch 2024-09-13T14:20:07.501 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.iscsi"} : dispatch 2024-09-13T14:20:07.501 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nfs"} : dispatch 2024-09-13T14:20:07.501 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"} : dispatch 2024-09-13T14:20:07.501 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-13T14:20:07.501 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:20:07.501 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:20:07.502 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:20:07.502 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:20:07.502 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:20:07.502 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:20:07.502 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: Upgrade: Complete! 2024-09-13T14:20:07.502 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"} : dispatch 2024-09-13T14:20:07.502 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-13T14:20:07.502 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:07 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:20:08.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:08 smithi102 ceph-mon[110183]: pgmap v84: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:08.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:08 smithi102 ceph-mon[115269]: pgmap v84: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:08.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:08 smithi195 ceph-mon[104262]: pgmap v84: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:11.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:11 smithi102 ceph-mon[110183]: pgmap v85: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:11.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:11 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:11.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:11 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:11.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:11 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:20:11.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:11 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:11.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:11 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:20:11.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:11 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:20:11.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:11 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:11.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:11 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:20:11.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:11 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:20:11.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:11 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:20:11.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:11 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:11.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:11 smithi102 ceph-mon[115269]: pgmap v85: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:11.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:11 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:11.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:11 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:11.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:11 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:20:11.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:11 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:11.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:11 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:20:11.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:11 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:20:11.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:11 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:11.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:11 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:20:11.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:11 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:20:11.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:11 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:20:11.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:11 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:11.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:11 smithi195 ceph-mon[104262]: pgmap v85: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:11.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:11 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:11.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:11 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:11.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:11 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:20:11.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:11 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:11.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:11 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:20:11.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:11 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:20:11.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:11 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:11.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:11 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:20:11.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:11 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:20:11.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:11 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:20:11.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:11 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:20:13.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:13 smithi102 ceph-mon[110183]: pgmap v86: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:13.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:13 smithi102 ceph-mon[115269]: pgmap v86: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:13.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:13 smithi195 ceph-mon[104262]: pgmap v86: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:15.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:15 smithi102 ceph-mon[110183]: pgmap v87: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:15.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:15 smithi102 ceph-mon[115269]: pgmap v87: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:15.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:15 smithi195 ceph-mon[104262]: pgmap v87: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:16.918 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:20:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:20:16] "GET /metrics HTTP/1.1" 200 33753 "" "Prometheus/2.51.0" 2024-09-13T14:20:17.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:17 smithi102 ceph-mon[110183]: pgmap v88: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:17.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:17 smithi102 ceph-mon[115269]: pgmap v88: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:17.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:17 smithi195 ceph-mon[104262]: pgmap v88: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:19.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:19 smithi102 ceph-mon[110183]: pgmap v89: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:19.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:19 smithi102 ceph-mon[115269]: pgmap v89: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:19.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:19 smithi195 ceph-mon[104262]: pgmap v89: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:21.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:21 smithi102 ceph-mon[110183]: pgmap v90: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:21.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:21 smithi102 ceph-mon[115269]: pgmap v90: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:21.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:21 smithi195 ceph-mon[104262]: pgmap v90: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:23.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:23 smithi102 ceph-mon[110183]: pgmap v91: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:23.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:23 smithi102 ceph-mon[115269]: pgmap v91: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:23.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:23 smithi195 ceph-mon[104262]: pgmap v91: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:25.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:25 smithi102 ceph-mon[110183]: pgmap v92: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:25.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:25 smithi102 ceph-mon[115269]: pgmap v92: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:25.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:25 smithi195 ceph-mon[104262]: pgmap v92: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:26.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:20:26.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:20:26.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:20:26.918 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:20:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:20:26] "GET /metrics HTTP/1.1" 200 33822 "" "Prometheus/2.51.0" 2024-09-13T14:20:27.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:27 smithi102 ceph-mon[110183]: pgmap v93: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:27.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:27 smithi102 ceph-mon[115269]: pgmap v93: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:27.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:27 smithi195 ceph-mon[104262]: pgmap v93: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:29.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:29 smithi102 ceph-mon[110183]: pgmap v94: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:29.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:29 smithi102 ceph-mon[115269]: pgmap v94: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:29.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:29 smithi195 ceph-mon[104262]: pgmap v94: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:31.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:31 smithi102 ceph-mon[110183]: pgmap v95: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:31.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:31 smithi102 ceph-mon[115269]: pgmap v95: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:31.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:31 smithi195 ceph-mon[104262]: pgmap v95: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:33.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:33 smithi102 ceph-mon[115269]: pgmap v96: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:33.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:33 smithi102 ceph-mon[110183]: pgmap v96: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:33.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:33 smithi195 ceph-mon[104262]: pgmap v96: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:33.806 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'sleep 60' 2024-09-13T14:20:34.005 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:20:34.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:34 smithi102 ceph-mon[115269]: from='client.34251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:20:34.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:34 smithi102 ceph-mon[110183]: from='client.34251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:20:34.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:34 smithi195 ceph-mon[104262]: from='client.34251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:20:35.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:35 smithi102 ceph-mon[110183]: pgmap v97: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:35.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:35 smithi102 ceph-mon[115269]: pgmap v97: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:35.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:35 smithi195 ceph-mon[104262]: pgmap v97: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:36.918 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:20:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:20:36] "GET /metrics HTTP/1.1" 200 33822 "" "Prometheus/2.51.0" 2024-09-13T14:20:37.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:37 smithi102 ceph-mon[110183]: pgmap v98: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:37.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:37 smithi102 ceph-mon[115269]: pgmap v98: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:37.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:37 smithi195 ceph-mon[104262]: pgmap v98: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:39.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:39 smithi102 ceph-mon[110183]: pgmap v99: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:39.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:39 smithi102 ceph-mon[115269]: pgmap v99: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:39.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:39 smithi195 ceph-mon[104262]: pgmap v99: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:41.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:41 smithi102 ceph-mon[110183]: pgmap v100: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:41.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:41 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:20:41.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:41 smithi102 ceph-mon[115269]: pgmap v100: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:41.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:41 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:20:41.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:41 smithi195 ceph-mon[104262]: pgmap v100: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:41.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:41 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:20:42.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:42 smithi195 ceph-mon[104262]: pgmap v101: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:42.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:42 smithi102 ceph-mon[110183]: pgmap v101: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:42.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:42 smithi102 ceph-mon[115269]: pgmap v101: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:44.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:44 smithi195 ceph-mon[104262]: pgmap v102: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:44.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:44 smithi102 ceph-mon[110183]: pgmap v102: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:44.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:44 smithi102 ceph-mon[115269]: pgmap v102: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:46.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:46 smithi195 ceph-mon[104262]: pgmap v103: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:46.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:46 smithi102 ceph-mon[110183]: pgmap v103: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:46.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:46 smithi102 ceph-mon[115269]: pgmap v103: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:46.669 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:20:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:20:46] "GET /metrics HTTP/1.1" 200 33821 "" "Prometheus/2.51.0" 2024-09-13T14:20:48.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:48 smithi195 ceph-mon[104262]: pgmap v104: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:48.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:48 smithi102 ceph-mon[110183]: pgmap v104: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:48.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:48 smithi102 ceph-mon[115269]: pgmap v104: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:50.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:50 smithi195 ceph-mon[104262]: pgmap v105: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:50.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:50 smithi102 ceph-mon[110183]: pgmap v105: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:50.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:50 smithi102 ceph-mon[115269]: pgmap v105: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:52.470 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:52 smithi102 ceph-mon[110183]: pgmap v106: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:52.471 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:52 smithi102 ceph-mon[115269]: pgmap v106: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:52.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:52 smithi195 ceph-mon[104262]: pgmap v106: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:54.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:54 smithi102 ceph-mon[110183]: pgmap v107: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:54.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:54 smithi102 ceph-mon[115269]: pgmap v107: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:54.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:54 smithi195 ceph-mon[104262]: pgmap v107: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:55.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:55 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:20:55.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:55 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:20:55.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:55 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:20:56.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:56 smithi102 ceph-mon[110183]: pgmap v108: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:56.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:56 smithi102 ceph-mon[115269]: pgmap v108: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:56.919 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:20:56 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:20:56] "GET /metrics HTTP/1.1" 200 33821 "" "Prometheus/2.51.0" 2024-09-13T14:20:56.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:56 smithi195 ceph-mon[104262]: pgmap v108: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:58.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:20:58 smithi102 ceph-mon[115269]: pgmap v109: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:58.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:20:58 smithi102 ceph-mon[110183]: pgmap v109: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:20:58.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:20:58 smithi195 ceph-mon[104262]: pgmap v109: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:00.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:00 smithi195 ceph-mon[104262]: pgmap v110: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:01.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:00 smithi102 ceph-mon[115269]: pgmap v110: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:01.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:00 smithi102 ceph-mon[110183]: pgmap v110: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:02.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:02 smithi195 ceph-mon[104262]: pgmap v111: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:03.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:02 smithi102 ceph-mon[110183]: pgmap v111: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:03.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:02 smithi102 ceph-mon[115269]: pgmap v111: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:04.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:04 smithi195 ceph-mon[104262]: pgmap v112: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:05.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:04 smithi102 ceph-mon[115269]: pgmap v112: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:05.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:04 smithi102 ceph-mon[110183]: pgmap v112: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:06.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:06 smithi102 ceph-mon[115269]: pgmap v113: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:06.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:06 smithi102 ceph-mon[110183]: pgmap v113: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:06.919 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:21:06 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:21:06] "GET /metrics HTTP/1.1" 200 33821 "" "Prometheus/2.51.0" 2024-09-13T14:21:06.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:06 smithi195 ceph-mon[104262]: pgmap v113: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:08.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:08 smithi195 ceph-mon[104262]: pgmap v114: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:09.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:08 smithi102 ceph-mon[110183]: pgmap v114: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:09.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:08 smithi102 ceph-mon[115269]: pgmap v114: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:10.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:10 smithi195 ceph-mon[104262]: pgmap v115: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:10.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:21:10.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:21:11.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:10 smithi102 ceph-mon[110183]: pgmap v115: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:11.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:21:11.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:21:11.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:10 smithi102 ceph-mon[115269]: pgmap v115: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:11.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:21:11.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:21:12.963 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:12 smithi102 ceph-mon[110183]: pgmap v116: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:12.964 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:12 smithi102 ceph-mon[115269]: pgmap v116: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:12.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:12 smithi195 ceph-mon[104262]: pgmap v116: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:15.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:15 smithi102 ceph-mon[110183]: pgmap v117: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:15.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:15 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:15.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:15 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:15.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:15 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:15.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:15 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:15.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:15 smithi102 ceph-mon[115269]: pgmap v117: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:15.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:15 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:15.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:15 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:15.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:15 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:15.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:15 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:15.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:15 smithi195 ceph-mon[104262]: pgmap v117: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:15.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:15 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:15.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:15 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:15.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:15 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:15.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:15 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:16.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:16 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:21:16.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:16 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:21:16.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:16 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:16.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:16 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:21:16.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:16 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:21:16.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:16 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:16.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:16 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:21:16.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:16 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:21:16.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:16 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:16.918 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:21:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:21:16] "GET /metrics HTTP/1.1" 200 33821 "" "Prometheus/2.51.0" 2024-09-13T14:21:17.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:17 smithi102 ceph-mon[115269]: pgmap v118: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:17.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:17 smithi102 ceph-mon[110183]: pgmap v118: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:17.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:17 smithi195 ceph-mon[104262]: pgmap v118: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:19.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:19 smithi102 ceph-mon[110183]: pgmap v119: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:19.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:19 smithi102 ceph-mon[115269]: pgmap v119: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:19.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:19 smithi195 ceph-mon[104262]: pgmap v119: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:21.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:21 smithi102 ceph-mon[110183]: pgmap v120: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:21.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:21 smithi102 ceph-mon[115269]: pgmap v120: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:21.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:21 smithi195 ceph-mon[104262]: pgmap v120: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:23.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:23 smithi102 ceph-mon[110183]: pgmap v121: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:23.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:23 smithi102 ceph-mon[115269]: pgmap v121: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:23.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:23 smithi195 ceph-mon[104262]: pgmap v121: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:25.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:25 smithi102 ceph-mon[110183]: pgmap v122: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:25.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:25 smithi102 ceph-mon[115269]: pgmap v122: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:25.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:25 smithi195 ceph-mon[104262]: pgmap v122: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:26.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:21:26.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:21:26.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:21:26.918 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:21:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:21:26] "GET /metrics HTTP/1.1" 200 33820 "" "Prometheus/2.51.0" 2024-09-13T14:21:27.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:27 smithi102 ceph-mon[110183]: pgmap v123: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:27.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:27 smithi102 ceph-mon[115269]: pgmap v123: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:27.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:27 smithi195 ceph-mon[104262]: pgmap v123: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:29.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:29 smithi102 ceph-mon[110183]: pgmap v124: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:29.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:29 smithi102 ceph-mon[115269]: pgmap v124: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:29.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:29 smithi195 ceph-mon[104262]: pgmap v124: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:31.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:31 smithi102 ceph-mon[110183]: pgmap v125: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:31.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:31 smithi102 ceph-mon[115269]: pgmap v125: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:31.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:31 smithi195 ceph-mon[104262]: pgmap v125: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:32.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:32 smithi195 ceph-mon[104262]: pgmap v126: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:32.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:32 smithi102 ceph-mon[110183]: pgmap v126: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:32.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:32 smithi102 ceph-mon[115269]: pgmap v126: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:34.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:34 smithi195 ceph-mon[104262]: pgmap v127: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:34.508 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:34 smithi102 ceph-mon[110183]: pgmap v127: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:34.508 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:34 smithi102 ceph-mon[115269]: pgmap v127: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:36.356 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:36 smithi102 ceph-mon[115269]: pgmap v128: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:36.356 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:36 smithi102 ceph-mon[110183]: pgmap v128: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:36.384 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'echo wait for all mons to upgrade' 2024-09-13T14:21:36.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:36 smithi195 ceph-mon[104262]: pgmap v128: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:36.591 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:21:36.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:21:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:21:36] "GET /metrics HTTP/1.1" 200 33820 "" "Prometheus/2.51.0" 2024-09-13T14:21:38.158 INFO:teuthology.orchestra.run.smithi102.stdout:wait for all mons to upgrade 2024-09-13T14:21:38.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:38 smithi102 ceph-mon[110183]: pgmap v129: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:38.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:38 smithi102 ceph-mon[115269]: pgmap v129: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:38.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:38 smithi195 ceph-mon[104262]: pgmap v129: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:38.792 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon' 2024-09-13T14:21:38.989 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:21:40.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:40 smithi102 ceph-mon[110183]: pgmap v130: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:40.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:40 smithi102 ceph-mon[115269]: pgmap v130: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:40.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:40 smithi195 ceph-mon[104262]: pgmap v130: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:41.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:41 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:21:41.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:41 smithi195 ceph-mon[104262]: from='client.44180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9", "daemon_types": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:41.610 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:41 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:21:41.610 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:41 smithi102 ceph-mon[115269]: from='client.44180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9", "daemon_types": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:41.610 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:41 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:21:41.610 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:41 smithi102 ceph-mon[110183]: from='client.44180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9", "daemon_types": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:42.457 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:42 smithi102 ceph-mon[110183]: pgmap v131: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:42.458 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:42 smithi102 ceph-mon[115269]: pgmap v131: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:42.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:42 smithi195 ceph-mon[104262]: pgmap v131: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:42.724 INFO:teuthology.orchestra.run.smithi102.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:21:43.416 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-09-13T14:21:43.623 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:21:43.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:43 smithi195 ceph-mon[104262]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:21:43.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:43 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:43.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:43 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:21:43.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:43 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:21:43.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:43 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:21:43.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:43 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:43.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:43 smithi195 ceph-mon[104262]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:21:44.008 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:43 smithi102 ceph-mon[110183]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:21:44.008 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:43 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:44.008 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:43 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:21:44.008 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:43 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:21:44.008 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:43 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:21:44.008 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:43 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:44.008 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:43 smithi102 ceph-mon[110183]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:21:44.009 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:43 smithi102 ceph-mon[115269]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:21:44.009 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:43 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:44.009 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:43 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:21:44.009 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:43 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:21:44.009 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:43 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:21:44.009 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:43 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:44.009 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:43 smithi102 ceph-mon[115269]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:21:44.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:44 smithi195 ceph-mon[104262]: pgmap v132: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:45.063 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:44 smithi102 ceph-mon[110183]: pgmap v132: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:45.063 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:44 smithi102 ceph-mon[115269]: pgmap v132: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:46.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:21:46] "GET /metrics HTTP/1.1" 200 33823 "" "Prometheus/2.51.0" 2024-09-13T14:21:46.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: pgmap v133: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:46.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:46.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:21:46.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:21:46.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:46.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:21:46.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:46.999 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:21:46.999 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:46.999 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:21:46.999 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:46.999 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:21:47.000 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:47.000 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:21:47.000 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:21:47.000 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:47.000 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:21:47.000 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:47.000 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:21:47.000 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:47.000 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:21:47.000 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:47.001 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:21:47.001 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:21:47.001 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:21:47.001 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:21:47.001 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:21:47.001 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:21:47.001 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr"} : dispatch 2024-09-13T14:21:47.001 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-13T14:21:47.001 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:21:47.001 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-13T14:21:47.001 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.crash"} : dispatch 2024-09-13T14:21:47.001 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-09-13T14:21:47.002 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd"} : dispatch 2024-09-13T14:21:47.002 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mds"} : dispatch 2024-09-13T14:21:47.002 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw"} : dispatch 2024-09-13T14:21:47.002 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-09-13T14:21:47.002 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"} : dispatch 2024-09-13T14:21:47.002 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-09-13T14:21:47.002 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:21:47.002 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"} : dispatch 2024-09-13T14:21:47.002 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-09-13T14:21:47.002 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.iscsi"} : dispatch 2024-09-13T14:21:47.002 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-09-13T14:21:47.002 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nfs"} : dispatch 2024-09-13T14:21:47.002 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-09-13T14:21:47.003 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"} : dispatch 2024-09-13T14:21:47.003 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-13T14:21:47.003 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:21:47.003 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:21:47.003 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:21:47.003 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:21:47.003 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:21:47.003 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:21:47.003 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"} : dispatch 2024-09-13T14:21:47.003 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-13T14:21:47.003 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:21:47.003 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:21:47.003 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:21:47.003 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:47.008 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: pgmap v133: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:47.008 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:47.008 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:21:47.009 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:21:47.009 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:47.009 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:21:47.009 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:47.009 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:21:47.009 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:47.009 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:21:47.009 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:47.009 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:21:47.009 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:47.009 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:21:47.009 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:21:47.010 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:47.010 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:21:47.010 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:47.010 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:21:47.010 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:47.010 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:21:47.010 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:47.010 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:21:47.010 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:21:47.010 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:21:47.010 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:21:47.010 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:21:47.010 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:21:47.010 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr"} : dispatch 2024-09-13T14:21:47.011 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-13T14:21:47.011 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: pgmap v133: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:47.011 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:47.011 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:21:47.011 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:21:47.011 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:47.011 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:21:47.011 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:47.012 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:21:47.012 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:47.012 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:21:47.012 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:47.012 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:21:47.012 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:47.012 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:21:47.012 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:21:47.012 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:47.012 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:21:47.013 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:47.013 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:21:47.013 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:47.013 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:21:47.013 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:47.013 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:21:47.013 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:21:47.013 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:21:47.013 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:21:47.013 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:21:47.013 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:21:47.013 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr"} : dispatch 2024-09-13T14:21:47.014 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-13T14:21:47.014 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:21:47.014 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-13T14:21:47.014 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.crash"} : dispatch 2024-09-13T14:21:47.014 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-09-13T14:21:47.014 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd"} : dispatch 2024-09-13T14:21:47.014 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mds"} : dispatch 2024-09-13T14:21:47.014 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw"} : dispatch 2024-09-13T14:21:47.014 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-09-13T14:21:47.015 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"} : dispatch 2024-09-13T14:21:47.015 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-09-13T14:21:47.015 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:21:47.015 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"} : dispatch 2024-09-13T14:21:47.015 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-09-13T14:21:47.015 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.iscsi"} : dispatch 2024-09-13T14:21:47.015 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-09-13T14:21:47.015 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nfs"} : dispatch 2024-09-13T14:21:47.015 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-09-13T14:21:47.015 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"} : dispatch 2024-09-13T14:21:47.016 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-13T14:21:47.016 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:21:47.016 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:21:47.016 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:21:47.016 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:21:47.016 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-13T14:21:47.016 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.crash"} : dispatch 2024-09-13T14:21:47.016 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-09-13T14:21:47.016 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd"} : dispatch 2024-09-13T14:21:47.017 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mds"} : dispatch 2024-09-13T14:21:47.017 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw"} : dispatch 2024-09-13T14:21:47.017 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-09-13T14:21:47.017 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"} : dispatch 2024-09-13T14:21:47.017 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-09-13T14:21:47.017 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:21:47.017 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"} : dispatch 2024-09-13T14:21:47.017 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-09-13T14:21:47.017 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.iscsi"} : dispatch 2024-09-13T14:21:47.017 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-09-13T14:21:47.017 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nfs"} : dispatch 2024-09-13T14:21:47.017 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-09-13T14:21:47.017 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"} : dispatch 2024-09-13T14:21:47.017 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-13T14:21:47.018 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:21:47.018 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:21:47.018 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:21:47.018 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:21:47.018 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:21:47.018 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:21:47.018 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"} : dispatch 2024-09-13T14:21:47.018 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-13T14:21:47.018 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:21:47.018 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:21:47.018 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:21:47.018 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:47.019 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:21:47.019 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:21:47.019 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:21:47.019 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"} : dispatch 2024-09-13T14:21:47.019 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-13T14:21:47.019 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:21:47.019 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:21:47.019 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:21:47.019 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:47.366 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'sleep 60' 2024-09-13T14:21:47.564 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:21:47.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:47 smithi102 ceph-mon[110183]: Upgrade: Target is version 19.3.0-4697-gd58f7306 (squid) 2024-09-13T14:21:47.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:47 smithi102 ceph-mon[110183]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, digests ['quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f'] 2024-09-13T14:21:47.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:47 smithi102 ceph-mon[110183]: Upgrade: Setting container_image for all mgr 2024-09-13T14:21:47.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:47 smithi102 ceph-mon[110183]: Upgrade: Setting container_image for all mon 2024-09-13T14:21:47.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:47 smithi102 ceph-mon[110183]: Upgrade: Setting container_image for all crash 2024-09-13T14:21:47.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:47 smithi102 ceph-mon[110183]: Upgrade: Setting container_image for all rgw 2024-09-13T14:21:47.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:47 smithi102 ceph-mon[110183]: Upgrade: Setting container_image for all rbd-mirror 2024-09-13T14:21:47.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:47 smithi102 ceph-mon[110183]: Upgrade: Setting container_image for all cephfs-mirror 2024-09-13T14:21:47.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:47 smithi102 ceph-mon[110183]: Upgrade: Setting container_image for all ceph-exporter 2024-09-13T14:21:47.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:47 smithi102 ceph-mon[110183]: Upgrade: Setting container_image for all iscsi 2024-09-13T14:21:47.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:47 smithi102 ceph-mon[110183]: Upgrade: Setting container_image for all nfs 2024-09-13T14:21:47.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:47 smithi102 ceph-mon[110183]: Upgrade: Setting container_image for all nvmeof 2024-09-13T14:21:47.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:47 smithi102 ceph-mon[110183]: Upgrade: Setting container_image for all node-exporter 2024-09-13T14:21:47.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:47 smithi102 ceph-mon[110183]: Upgrade: Setting container_image for all prometheus 2024-09-13T14:21:47.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:47 smithi102 ceph-mon[110183]: Upgrade: Setting container_image for all alertmanager 2024-09-13T14:21:47.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:47 smithi102 ceph-mon[110183]: Upgrade: Setting container_image for all grafana 2024-09-13T14:21:47.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:47 smithi102 ceph-mon[110183]: Upgrade: Setting container_image for all loki 2024-09-13T14:21:47.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:47 smithi102 ceph-mon[110183]: Upgrade: Setting container_image for all promtail 2024-09-13T14:21:47.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:47 smithi102 ceph-mon[110183]: Upgrade: Finalizing container_image settings 2024-09-13T14:21:47.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:47 smithi102 ceph-mon[110183]: Upgrade: Complete! 2024-09-13T14:21:47.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:47 smithi102 ceph-mon[110183]: from='client.34263 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:47.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:47 smithi102 ceph-mon[115269]: Upgrade: Target is version 19.3.0-4697-gd58f7306 (squid) 2024-09-13T14:21:47.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:47 smithi102 ceph-mon[115269]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, digests ['quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f'] 2024-09-13T14:21:47.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:47 smithi102 ceph-mon[115269]: Upgrade: Setting container_image for all mgr 2024-09-13T14:21:47.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:47 smithi102 ceph-mon[115269]: Upgrade: Setting container_image for all mon 2024-09-13T14:21:47.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:47 smithi102 ceph-mon[115269]: Upgrade: Setting container_image for all crash 2024-09-13T14:21:47.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:47 smithi102 ceph-mon[115269]: Upgrade: Setting container_image for all rgw 2024-09-13T14:21:47.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:47 smithi102 ceph-mon[115269]: Upgrade: Setting container_image for all rbd-mirror 2024-09-13T14:21:47.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:47 smithi102 ceph-mon[115269]: Upgrade: Setting container_image for all cephfs-mirror 2024-09-13T14:21:47.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:47 smithi102 ceph-mon[115269]: Upgrade: Setting container_image for all ceph-exporter 2024-09-13T14:21:47.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:47 smithi102 ceph-mon[115269]: Upgrade: Setting container_image for all iscsi 2024-09-13T14:21:47.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:47 smithi102 ceph-mon[115269]: Upgrade: Setting container_image for all nfs 2024-09-13T14:21:47.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:47 smithi102 ceph-mon[115269]: Upgrade: Setting container_image for all nvmeof 2024-09-13T14:21:47.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:47 smithi102 ceph-mon[115269]: Upgrade: Setting container_image for all node-exporter 2024-09-13T14:21:47.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:47 smithi102 ceph-mon[115269]: Upgrade: Setting container_image for all prometheus 2024-09-13T14:21:47.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:47 smithi102 ceph-mon[115269]: Upgrade: Setting container_image for all alertmanager 2024-09-13T14:21:47.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:47 smithi102 ceph-mon[115269]: Upgrade: Setting container_image for all grafana 2024-09-13T14:21:47.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:47 smithi102 ceph-mon[115269]: Upgrade: Setting container_image for all loki 2024-09-13T14:21:47.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:47 smithi102 ceph-mon[115269]: Upgrade: Setting container_image for all promtail 2024-09-13T14:21:47.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:47 smithi102 ceph-mon[115269]: Upgrade: Finalizing container_image settings 2024-09-13T14:21:47.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:47 smithi102 ceph-mon[115269]: Upgrade: Complete! 2024-09-13T14:21:47.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:47 smithi102 ceph-mon[115269]: from='client.34263 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:47.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:47 smithi195 ceph-mon[104262]: Upgrade: Target is version 19.3.0-4697-gd58f7306 (squid) 2024-09-13T14:21:47.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:47 smithi195 ceph-mon[104262]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, digests ['quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f'] 2024-09-13T14:21:47.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:47 smithi195 ceph-mon[104262]: Upgrade: Setting container_image for all mgr 2024-09-13T14:21:47.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:47 smithi195 ceph-mon[104262]: Upgrade: Setting container_image for all mon 2024-09-13T14:21:47.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:47 smithi195 ceph-mon[104262]: Upgrade: Setting container_image for all crash 2024-09-13T14:21:47.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:47 smithi195 ceph-mon[104262]: Upgrade: Setting container_image for all rgw 2024-09-13T14:21:47.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:47 smithi195 ceph-mon[104262]: Upgrade: Setting container_image for all rbd-mirror 2024-09-13T14:21:47.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:47 smithi195 ceph-mon[104262]: Upgrade: Setting container_image for all cephfs-mirror 2024-09-13T14:21:47.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:47 smithi195 ceph-mon[104262]: Upgrade: Setting container_image for all ceph-exporter 2024-09-13T14:21:47.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:47 smithi195 ceph-mon[104262]: Upgrade: Setting container_image for all iscsi 2024-09-13T14:21:47.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:47 smithi195 ceph-mon[104262]: Upgrade: Setting container_image for all nfs 2024-09-13T14:21:47.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:47 smithi195 ceph-mon[104262]: Upgrade: Setting container_image for all nvmeof 2024-09-13T14:21:47.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:47 smithi195 ceph-mon[104262]: Upgrade: Setting container_image for all node-exporter 2024-09-13T14:21:47.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:47 smithi195 ceph-mon[104262]: Upgrade: Setting container_image for all prometheus 2024-09-13T14:21:47.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:47 smithi195 ceph-mon[104262]: Upgrade: Setting container_image for all alertmanager 2024-09-13T14:21:47.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:47 smithi195 ceph-mon[104262]: Upgrade: Setting container_image for all grafana 2024-09-13T14:21:47.999 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:47 smithi195 ceph-mon[104262]: Upgrade: Setting container_image for all loki 2024-09-13T14:21:47.999 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:47 smithi195 ceph-mon[104262]: Upgrade: Setting container_image for all promtail 2024-09-13T14:21:47.999 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:47 smithi195 ceph-mon[104262]: Upgrade: Finalizing container_image settings 2024-09-13T14:21:47.999 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:47 smithi195 ceph-mon[104262]: Upgrade: Complete! 2024-09-13T14:21:47.999 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:47 smithi195 ceph-mon[104262]: from='client.34263 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:21:48.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:48 smithi102 ceph-mon[110183]: pgmap v134: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:48.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:48 smithi102 ceph-mon[115269]: pgmap v134: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:48.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:48 smithi195 ceph-mon[104262]: pgmap v134: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:50.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:50 smithi195 ceph-mon[104262]: pgmap v135: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:50.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:50 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:51.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:50 smithi102 ceph-mon[115269]: pgmap v135: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:51.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:50 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:51.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:50 smithi102 ceph-mon[110183]: pgmap v135: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:51.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:50 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:21:53.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:52 smithi102 ceph-mon[110183]: pgmap v136: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:53.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:52 smithi102 ceph-mon[115269]: pgmap v136: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:53.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:52 smithi195 ceph-mon[104262]: pgmap v136: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:55.049 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:54 smithi195 ceph-mon[104262]: pgmap v137: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:55.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:54 smithi102 ceph-mon[110183]: pgmap v137: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:55.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:54 smithi102 ceph-mon[115269]: pgmap v137: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:56.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:55 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:21:56.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:55 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:21:56.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:55 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:21:56.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:21:56 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:21:56] "GET /metrics HTTP/1.1" 200 33819 "" "Prometheus/2.51.0" 2024-09-13T14:21:57.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:56 smithi102 ceph-mon[110183]: pgmap v138: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:57.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:56 smithi102 ceph-mon[115269]: pgmap v138: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:57.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:56 smithi195 ceph-mon[104262]: pgmap v138: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:59.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:21:58 smithi102 ceph-mon[115269]: pgmap v139: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:59.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:21:58 smithi102 ceph-mon[110183]: pgmap v139: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:21:59.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:21:58 smithi195 ceph-mon[104262]: pgmap v139: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:01.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:22:00 smithi102 ceph-mon[110183]: pgmap v140: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:01.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:22:00 smithi102 ceph-mon[115269]: pgmap v140: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:01.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:22:00 smithi195 ceph-mon[104262]: pgmap v140: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:03.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:22:02 smithi102 ceph-mon[110183]: pgmap v141: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:03.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:22:02 smithi102 ceph-mon[115269]: pgmap v141: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:03.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:22:02 smithi195 ceph-mon[104262]: pgmap v141: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:05.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:22:04 smithi102 ceph-mon[110183]: pgmap v142: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:05.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:22:04 smithi102 ceph-mon[115269]: pgmap v142: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:05.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:22:04 smithi195 ceph-mon[104262]: pgmap v142: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:06.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:22:06 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:22:06] "GET /metrics HTTP/1.1" 200 33819 "" "Prometheus/2.51.0" 2024-09-13T14:22:07.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:22:06 smithi102 ceph-mon[115269]: pgmap v143: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:07.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:22:06 smithi102 ceph-mon[110183]: pgmap v143: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:07.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:22:06 smithi195 ceph-mon[104262]: pgmap v143: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:09.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:22:08 smithi102 ceph-mon[110183]: pgmap v144: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:09.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:22:08 smithi102 ceph-mon[115269]: pgmap v144: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:09.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:22:08 smithi195 ceph-mon[104262]: pgmap v144: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:11.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:22:10 smithi102 ceph-mon[110183]: pgmap v145: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:11.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:22:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:22:11.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:22:10 smithi102 ceph-mon[115269]: pgmap v145: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:11.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:22:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:22:11.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:22:10 smithi195 ceph-mon[104262]: pgmap v145: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:11.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:22:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:22:13.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:22:12 smithi102 ceph-mon[110183]: pgmap v146: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:13.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:22:12 smithi102 ceph-mon[115269]: pgmap v146: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:13.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:22:12 smithi195 ceph-mon[104262]: pgmap v146: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:15.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:22:14 smithi102 ceph-mon[110183]: pgmap v147: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:15.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:22:14 smithi102 ceph-mon[115269]: pgmap v147: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:15.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:22:14 smithi195 ceph-mon[104262]: pgmap v147: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:16.790 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:22:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:22:16] "GET /metrics HTTP/1.1" 200 33818 "" "Prometheus/2.51.0" 2024-09-13T14:22:17.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:22:16 smithi102 ceph-mon[110183]: pgmap v148: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:17.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:22:16 smithi102 ceph-mon[115269]: pgmap v148: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:17.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:22:16 smithi195 ceph-mon[104262]: pgmap v148: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:19.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:22:18 smithi102 ceph-mon[110183]: pgmap v149: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:19.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:22:18 smithi102 ceph-mon[115269]: pgmap v149: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:19.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:22:18 smithi195 ceph-mon[104262]: pgmap v149: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:21.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:22:20 smithi102 ceph-mon[110183]: pgmap v150: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:21.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:22:20 smithi102 ceph-mon[115269]: pgmap v150: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:21.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:22:20 smithi195 ceph-mon[104262]: pgmap v150: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:23.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:22:22 smithi102 ceph-mon[110183]: pgmap v151: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:23.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:22:22 smithi102 ceph-mon[115269]: pgmap v151: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:23.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:22:22 smithi195 ceph-mon[104262]: pgmap v151: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:25.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:22:24 smithi102 ceph-mon[110183]: pgmap v152: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:25.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:22:24 smithi102 ceph-mon[115269]: pgmap v152: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:25.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:22:24 smithi195 ceph-mon[104262]: pgmap v152: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:26.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:22:25 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:22:26.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:22:25 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:22:26.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:22:25 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:22:26.808 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:22:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:22:26] "GET /metrics HTTP/1.1" 200 33823 "" "Prometheus/2.51.0" 2024-09-13T14:22:27.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:22:26 smithi102 ceph-mon[110183]: pgmap v153: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:27.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:22:26 smithi102 ceph-mon[115269]: pgmap v153: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:27.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:22:26 smithi195 ceph-mon[104262]: pgmap v153: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:29.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:22:28 smithi102 ceph-mon[110183]: pgmap v154: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:29.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:22:28 smithi102 ceph-mon[115269]: pgmap v154: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:29.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:22:28 smithi195 ceph-mon[104262]: pgmap v154: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:31.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:22:30 smithi102 ceph-mon[110183]: pgmap v155: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:31.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:22:30 smithi102 ceph-mon[115269]: pgmap v155: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:31.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:22:30 smithi195 ceph-mon[104262]: pgmap v155: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:33.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:22:32 smithi102 ceph-mon[110183]: pgmap v156: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:33.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:22:32 smithi102 ceph-mon[115269]: pgmap v156: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:33.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:22:32 smithi195 ceph-mon[104262]: pgmap v156: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:35.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:22:34 smithi102 ceph-mon[110183]: pgmap v157: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:35.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:22:34 smithi102 ceph-mon[115269]: pgmap v157: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:35.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:22:34 smithi195 ceph-mon[104262]: pgmap v157: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:36.824 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:22:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:22:36] "GET /metrics HTTP/1.1" 200 33823 "" "Prometheus/2.51.0" 2024-09-13T14:22:37.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:22:36 smithi102 ceph-mon[110183]: pgmap v158: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:37.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:22:36 smithi102 ceph-mon[115269]: pgmap v158: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:37.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:22:36 smithi195 ceph-mon[104262]: pgmap v158: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:39.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:22:38 smithi102 ceph-mon[110183]: pgmap v159: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:39.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:22:38 smithi102 ceph-mon[115269]: pgmap v159: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:39.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:22:38 smithi195 ceph-mon[104262]: pgmap v159: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:41.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:22:40 smithi102 ceph-mon[110183]: pgmap v160: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:41.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:22:40 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:22:41.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:22:40 smithi102 ceph-mon[115269]: pgmap v160: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:41.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:22:40 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:22:41.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:22:40 smithi195 ceph-mon[104262]: pgmap v160: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:41.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:22:40 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:22:43.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:22:42 smithi102 ceph-mon[110183]: pgmap v161: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:43.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:22:42 smithi102 ceph-mon[115269]: pgmap v161: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:43.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:22:42 smithi195 ceph-mon[104262]: pgmap v161: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:45.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:22:44 smithi102 ceph-mon[110183]: pgmap v162: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:45.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:22:44 smithi102 ceph-mon[115269]: pgmap v162: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:45.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:22:44 smithi195 ceph-mon[104262]: pgmap v162: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:46.839 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:22:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:22:46] "GET /metrics HTTP/1.1" 200 33811 "" "Prometheus/2.51.0" 2024-09-13T14:22:47.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:22:46 smithi102 ceph-mon[110183]: pgmap v163: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:47.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:22:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:22:47.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:22:46 smithi102 ceph-mon[115269]: pgmap v163: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:47.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:22:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:22:47.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:22:46 smithi195 ceph-mon[104262]: pgmap v163: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:47.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:22:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:22:49.100 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:22:48 smithi102 ceph-mon[110183]: pgmap v164: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:49.101 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:22:48 smithi102 ceph-mon[115269]: pgmap v164: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:49.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:22:48 smithi195 ceph-mon[104262]: pgmap v164: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:50.183 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'echo wait for half of osds to upgrade' 2024-09-13T14:22:50.844 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:22:50 smithi102 ceph-mon[110183]: pgmap v165: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:50.845 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:22:50 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:22:50.845 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:22:50 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:22:50.931 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:22:51.102 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:22:50 smithi102 ceph-mon[115269]: pgmap v165: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:51.102 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:22:50 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:22:51.102 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:22:50 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:22:51.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:22:50 smithi195 ceph-mon[104262]: pgmap v165: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:51.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:22:50 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:22:51.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:22:50 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:22:52.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:22:51 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:22:52.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:22:51 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:22:52.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:22:51 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:22:52.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:22:51 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:22:52.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:22:51 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:22:52.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:22:51 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:22:52.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:22:51 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:22:52.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:22:51 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:22:52.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:22:51 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:22:52.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:22:51 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:22:52.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:22:51 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:22:52.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:22:51 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:22:52.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:22:51 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:22:52.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:22:51 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:22:52.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:22:51 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:22:52.697 INFO:teuthology.orchestra.run.smithi102.stdout:wait for half of osds to upgrade 2024-09-13T14:22:53.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:22:52 smithi102 ceph-mon[110183]: pgmap v166: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:53.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:22:52 smithi102 ceph-mon[115269]: pgmap v166: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:53.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:22:52 smithi195 ceph-mon[104262]: pgmap v166: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:53.321 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types osd --limit 4' 2024-09-13T14:22:53.519 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:22:55.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:22:54 smithi102 ceph-mon[110183]: pgmap v167: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:55.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:22:54 smithi102 ceph-mon[115269]: pgmap v167: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:55.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:22:54 smithi195 ceph-mon[104262]: pgmap v167: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:56.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:22:55 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:22:56.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:22:55 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:22:56.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:22:55 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:22:56.571 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:22:56 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:22:56] "GET /metrics HTTP/1.1" 200 33818 "" "Prometheus/2.51.0" 2024-09-13T14:22:57.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:22:56 smithi102 ceph-mon[115269]: pgmap v168: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:57.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:22:56 smithi102 ceph-mon[115269]: from='client.44192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9", "daemon_types": "osd", "limit": 4, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:57.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:22:56 smithi102 ceph-mon[110183]: pgmap v168: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:57.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:22:56 smithi102 ceph-mon[110183]: from='client.44192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9", "daemon_types": "osd", "limit": 4, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:57.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:22:56 smithi195 ceph-mon[104262]: pgmap v168: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:57.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:22:56 smithi195 ceph-mon[104262]: from='client.44192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9", "daemon_types": "osd", "limit": 4, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:57.282 INFO:teuthology.orchestra.run.smithi102.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:22:57.896 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-09-13T14:22:58.117 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:22:58.617 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:22:58 smithi102 ceph-mon[115269]: pgmap v169: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:58.617 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:22:58 smithi102 ceph-mon[115269]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:22:58.617 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:22:58 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:22:58.617 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:22:58 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:22:58.617 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:22:58 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:22:58.617 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:22:58 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:22:58.617 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:22:58 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:22:58.617 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:22:58 smithi102 ceph-mon[115269]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:22:58.618 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:22:58 smithi102 ceph-mon[110183]: pgmap v169: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:58.618 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:22:58 smithi102 ceph-mon[110183]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:22:58.618 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:22:58 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:22:58.618 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:22:58 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:22:58.618 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:22:58 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:22:58.618 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:22:58 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:22:58.618 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:22:58 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:22:58.618 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:22:58 smithi102 ceph-mon[110183]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:22:58.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:22:58 smithi195 ceph-mon[104262]: pgmap v169: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:22:58.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:22:58 smithi195 ceph-mon[104262]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:22:58.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:22:58 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:22:58.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:22:58 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:22:58.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:22:58 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:22:58.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:22:58 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:22:58.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:22:58 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:22:58.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:22:58 smithi195 ceph-mon[104262]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:22:59.979 INFO:teuthology.orchestra.run.smithi102.stdout:true 2024-09-13T14:23:00.558 INFO:teuthology.orchestra.run.smithi102.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-13T14:23:00.558 INFO:teuthology.orchestra.run.smithi102.stdout:alertmanager.a smithi102 *:9093,9094 running (6m) 9s ago 12m 19.2M - 0.27.0 11f11916f8cd 0322b99ec767 2024-09-13T14:23:00.558 INFO:teuthology.orchestra.run.smithi102.stdout:grafana.a smithi195 *:3000 running (6m) 9s ago 12m 84.1M - c8b91775d855 152e89bad2c8 2024-09-13T14:23:00.558 INFO:teuthology.orchestra.run.smithi102.stdout:mds.foo.smithi102.njetxr smithi102 running (11m) 9s ago 11m 23.0M - 18.2.4-843-g2c01458a c129f5a2fd4e c9211cc725ed 2024-09-13T14:23:00.558 INFO:teuthology.orchestra.run.smithi102.stdout:mds.foo.smithi195.yqzrxi smithi195 running (11m) 9s ago 11m 19.1M - 18.2.4-843-g2c01458a c129f5a2fd4e eea37a7230fd 2024-09-13T14:23:00.558 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.x smithi195 *:8443,9283,8765 running (7m) 9s ago 16m 456M - 19.3.0-4697-gd58f7306 fec07016a2b0 4a23b8f0ea8b 2024-09-13T14:23:00.558 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.y smithi102 *:8443,9283,8765 running (8m) 9s ago 18m 517M - 19.3.0-4697-gd58f7306 fec07016a2b0 6b12b28dcde1 2024-09-13T14:23:00.558 INFO:teuthology.orchestra.run.smithi102.stdout:mon.a smithi102 running (5m) 9s ago 18m 51.8M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 f3234b7bbdde 2024-09-13T14:23:00.559 INFO:teuthology.orchestra.run.smithi102.stdout:mon.b smithi195 running (3m) 9s ago 16m 42.9M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 9921a2aa8168 2024-09-13T14:23:00.559 INFO:teuthology.orchestra.run.smithi102.stdout:mon.c smithi102 running (5m) 9s ago 16m 43.5M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 dba6b3d55296 2024-09-13T14:23:00.559 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.a smithi102 *:9100 running (7m) 9s ago 12m 12.0M - 1.7.0 72c9c2088986 0c92d0b39468 2024-09-13T14:23:00.559 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.b smithi195 *:9100 running (7m) 9s ago 12m 11.2M - 1.7.0 72c9c2088986 fde867a7dbf9 2024-09-13T14:23:00.559 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0 smithi102 running (16m) 9s ago 16m 61.2M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e b97482fb8cf9 2024-09-13T14:23:00.559 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1 smithi102 running (15m) 9s ago 15m 52.8M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 8a67356be7a2 2024-09-13T14:23:00.559 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2 smithi102 running (15m) 9s ago 15m 50.9M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 1800986ac838 2024-09-13T14:23:00.559 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3 smithi102 running (15m) 9s ago 15m 57.6M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 968246da9da2 2024-09-13T14:23:00.559 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4 smithi195 running (14m) 9s ago 14m 50.4M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 515307701d73 2024-09-13T14:23:00.559 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5 smithi195 running (14m) 9s ago 14m 54.1M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 176ba2b3bd59 2024-09-13T14:23:00.559 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6 smithi195 running (13m) 9s ago 13m 56.0M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 9cd070a6dd3d 2024-09-13T14:23:00.559 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7 smithi195 running (13m) 9s ago 13m 54.4M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e a6b698c659ca 2024-09-13T14:23:00.559 INFO:teuthology.orchestra.run.smithi102.stdout:prometheus.a smithi195 *:9095 running (7m) 9s ago 12m 48.2M - 2.51.0 1d3b7f56885b d131e53f1ef9 2024-09-13T14:23:00.560 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:00 smithi102 ceph-mon[110183]: pgmap v170: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:00.560 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:00 smithi102 ceph-mon[110183]: from='client.44198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:00.560 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:00 smithi102 ceph-mon[115269]: pgmap v170: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:00.560 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:00 smithi102 ceph-mon[115269]: from='client.44198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:00.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:00 smithi195 ceph-mon[104262]: pgmap v170: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:00.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:00 smithi195 ceph-mon[104262]: from='client.44198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:00.896 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:23:00.896 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": { 2024-09-13T14:23:00.897 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 3 2024-09-13T14:23:00.897 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:23:00.897 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": { 2024-09-13T14:23:00.897 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 2 2024-09-13T14:23:00.897 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:23:00.897 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": { 2024-09-13T14:23:00.897 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 8 2024-09-13T14:23:00.897 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:23:00.897 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": { 2024-09-13T14:23:00.897 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 2 2024-09-13T14:23:00.897 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:23:00.897 INFO:teuthology.orchestra.run.smithi102.stdout: "overall": { 2024-09-13T14:23:00.897 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 10, 2024-09-13T14:23:00.898 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 5 2024-09-13T14:23:00.898 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:23:00.898 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:23:01.203 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:23:01.204 INFO:teuthology.orchestra.run.smithi102.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f", 2024-09-13T14:23:01.204 INFO:teuthology.orchestra.run.smithi102.stdout: "in_progress": true, 2024-09-13T14:23:01.204 INFO:teuthology.orchestra.run.smithi102.stdout: "which": "Upgrading daemons of type(s) osd. Upgrade limited to 4 daemons (4 remaining).", 2024-09-13T14:23:01.204 INFO:teuthology.orchestra.run.smithi102.stdout: "services_complete": [], 2024-09-13T14:23:01.204 INFO:teuthology.orchestra.run.smithi102.stdout: "progress": "0/8 daemons upgraded", 2024-09-13T14:23:01.204 INFO:teuthology.orchestra.run.smithi102.stdout: "message": "Currently upgrading osd daemons", 2024-09-13T14:23:01.204 INFO:teuthology.orchestra.run.smithi102.stdout: "is_paused": false 2024-09-13T14:23:01.204 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:23:01.465 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:01 smithi102 ceph-mon[110183]: from='client.34281 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:01.465 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:01 smithi102 ceph-mon[110183]: from='client.34287 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:01.465 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:01 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/610120736' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:23:01.465 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:01 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:01.465 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:01 smithi102 ceph-mon[110183]: Upgrade: Target is version 19.3.0-4697-gd58f7306 (squid) 2024-09-13T14:23:01.466 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:01 smithi102 ceph-mon[110183]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, digests ['quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f'] 2024-09-13T14:23:01.466 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:01 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:23:01.466 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:01 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:23:01.466 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:01 smithi102 ceph-mon[110183]: Upgrade: Setting container_image for all mgr 2024-09-13T14:23:01.466 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:01 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:01.466 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:01 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:23:01.466 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:01 smithi102 ceph-mon[110183]: Upgrade: Setting container_image for all mon 2024-09-13T14:23:01.466 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:01 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:01.466 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:01 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:23:01.466 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:01 smithi102 ceph-mon[110183]: Upgrade: Setting container_image for all crash 2024-09-13T14:23:01.466 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:01 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:01.466 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:01 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16} : dispatch 2024-09-13T14:23:01.466 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:01 smithi102 ceph-mon[110183]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-13T14:23:01.467 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:01 smithi102 ceph-mon[110183]: Upgrade: osd.0 is safe to restart 2024-09-13T14:23:01.467 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:01 smithi102 ceph-mon[115269]: from='client.34281 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:01.467 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:01 smithi102 ceph-mon[115269]: from='client.34287 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:01.467 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:01 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/610120736' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:23:01.467 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:01 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:01.467 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:01 smithi102 ceph-mon[115269]: Upgrade: Target is version 19.3.0-4697-gd58f7306 (squid) 2024-09-13T14:23:01.467 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:01 smithi102 ceph-mon[115269]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, digests ['quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f'] 2024-09-13T14:23:01.467 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:01 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:23:01.467 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:01 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:23:01.467 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:01 smithi102 ceph-mon[115269]: Upgrade: Setting container_image for all mgr 2024-09-13T14:23:01.467 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:01 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:01.467 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:01 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:23:01.468 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:01 smithi102 ceph-mon[115269]: Upgrade: Setting container_image for all mon 2024-09-13T14:23:01.468 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:01 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:01.468 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:01 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:23:01.468 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:01 smithi102 ceph-mon[115269]: Upgrade: Setting container_image for all crash 2024-09-13T14:23:01.468 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:01 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:01.468 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:01 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16} : dispatch 2024-09-13T14:23:01.468 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:01 smithi102 ceph-mon[115269]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-13T14:23:01.468 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:01 smithi102 ceph-mon[115269]: Upgrade: osd.0 is safe to restart 2024-09-13T14:23:01.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:01 smithi195 ceph-mon[104262]: from='client.34281 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:01.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:01 smithi195 ceph-mon[104262]: from='client.34287 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:01.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:01 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/610120736' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:23:01.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:01 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:01.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:01 smithi195 ceph-mon[104262]: Upgrade: Target is version 19.3.0-4697-gd58f7306 (squid) 2024-09-13T14:23:01.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:01 smithi195 ceph-mon[104262]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, digests ['quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f'] 2024-09-13T14:23:01.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:01 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:23:01.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:01 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:23:01.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:01 smithi195 ceph-mon[104262]: Upgrade: Setting container_image for all mgr 2024-09-13T14:23:01.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:01 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:01.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:01 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:23:01.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:01 smithi195 ceph-mon[104262]: Upgrade: Setting container_image for all mon 2024-09-13T14:23:01.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:01 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:01.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:01 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:23:01.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:01 smithi195 ceph-mon[104262]: Upgrade: Setting container_image for all crash 2024-09-13T14:23:01.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:01 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:01.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:01 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16} : dispatch 2024-09-13T14:23:01.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:01 smithi195 ceph-mon[104262]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-13T14:23:01.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:01 smithi195 ceph-mon[104262]: Upgrade: osd.0 is safe to restart 2024-09-13T14:23:03.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:02 smithi102 ceph-mon[110183]: pgmap v171: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:03.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:02 smithi102 ceph-mon[110183]: from='client.34299 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:03.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:02 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:03.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:02 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-13T14:23:03.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:02 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:23:03.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:02 smithi102 ceph-mon[115269]: pgmap v171: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:03.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:02 smithi102 ceph-mon[115269]: from='client.34299 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:03.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:02 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:03.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:02 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-13T14:23:03.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:02 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:23:03.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:02 smithi195 ceph-mon[104262]: pgmap v171: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:03.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:02 smithi195 ceph-mon[104262]: from='client.34299 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:03.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:02 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:03.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:02 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-13T14:23:03.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:02 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:23:04.090 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:03 smithi102 ceph-mon[110183]: Upgrade: Updating osd.0 2024-09-13T14:23:04.090 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:03 smithi102 ceph-mon[110183]: Deploying daemon osd.0 on smithi102 2024-09-13T14:23:04.090 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:03 smithi102 ceph-mon[115269]: Upgrade: Updating osd.0 2024-09-13T14:23:04.091 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:03 smithi102 ceph-mon[115269]: Deploying daemon osd.0 on smithi102 2024-09-13T14:23:04.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:03 smithi195 ceph-mon[104262]: Upgrade: Updating osd.0 2024-09-13T14:23:04.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:03 smithi195 ceph-mon[104262]: Deploying daemon osd.0 on smithi102 2024-09-13T14:23:05.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:04 smithi195 ceph-mon[104262]: pgmap v172: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:05.392 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:04 smithi102 ceph-mon[110183]: pgmap v172: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:05.392 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:04 smithi102 ceph-mon[115269]: pgmap v172: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:05.393 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:05 smithi102 systemd[1]: Stopping Ceph osd.0 for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T14:23:05.668 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:05 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0[44777]: 2024-09-13T14:23:05.389+0000 7ffbadee1640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T14:23:05.668 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:05 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0[44777]: 2024-09-13T14:23:05.389+0000 7ffbadee1640 -1 osd.0 90 *** Got signal Terminated *** 2024-09-13T14:23:05.668 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:05 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0[44777]: 2024-09-13T14:23:05.389+0000 7ffbadee1640 -1 osd.0 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T14:23:06.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:05 smithi195 ceph-mon[104262]: osd.0 marked itself down and dead 2024-09-13T14:23:06.331 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:05 smithi102 ceph-mon[110183]: osd.0 marked itself down and dead 2024-09-13T14:23:06.331 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:05 smithi102 ceph-mon[115269]: osd.0 marked itself down and dead 2024-09-13T14:23:06.632 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:23:06 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:23:06] "GET /metrics HTTP/1.1" 200 33818 "" "Prometheus/2.51.0" 2024-09-13T14:23:06.633 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:06 smithi102 podman[127018]: 2024-09-13 14:23:06.330387644 +0000 UTC m=+1.022381581 container died b97482fb8cf96809369d8aa0899190626d03ad5151afccb23bd64eca15387b81 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, RELEASE=reef-2c01458, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-13T14:23:06.918 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:06 smithi102 podman[127018]: 2024-09-13 14:23:06.731465931 +0000 UTC m=+1.423459857 container cleanup b97482fb8cf96809369d8aa0899190626d03ad5151afccb23bd64eca15387b81 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=reef-2c01458) 2024-09-13T14:23:06.918 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:06 smithi102 bash[127018]: ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0 2024-09-13T14:23:07.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:06 smithi195 ceph-mon[104262]: pgmap v173: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:07.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:06 smithi195 ceph-mon[104262]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:23:07.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:06 smithi195 ceph-mon[104262]: osdmap e91: 8 total, 7 up, 8 in 2024-09-13T14:23:07.272 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:06 smithi102 ceph-mon[110183]: pgmap v173: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:07.272 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:06 smithi102 ceph-mon[110183]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:23:07.272 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:06 smithi102 ceph-mon[110183]: osdmap e91: 8 total, 7 up, 8 in 2024-09-13T14:23:07.272 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:06 smithi102 ceph-mon[115269]: pgmap v173: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:07.272 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:06 smithi102 ceph-mon[115269]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:23:07.273 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:06 smithi102 ceph-mon[115269]: osdmap e91: 8 total, 7 up, 8 in 2024-09-13T14:23:07.273 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:06 smithi102 podman[127038]: 2024-09-13 14:23:06.951382879 +0000 UTC m=+0.615645701 container remove b97482fb8cf96809369d8aa0899190626d03ad5151afccb23bd64eca15387b81 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, RELEASE=reef-2c01458, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2) 2024-09-13T14:23:07.553 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:07 smithi102 podman[127135]: 2024-09-13 14:23:07.173781313 +0000 UTC m=+0.015870049 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:23:07.553 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:07 smithi102 podman[127135]: 2024-09-13 14:23:07.309009907 +0000 UTC m=+0.151098643 container create a73ea7d87b56ef79f258c637fbcc679af99c3747a5155d665577c23fc743b4b4 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True) 2024-09-13T14:23:07.918 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:07 smithi102 podman[127135]: 2024-09-13 14:23:07.553129629 +0000 UTC m=+0.395218368 container init a73ea7d87b56ef79f258c637fbcc679af99c3747a5155d665577c23fc743b4b4 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=) 2024-09-13T14:23:07.918 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:07 smithi102 podman[127135]: 2024-09-13 14:23:07.558134615 +0000 UTC m=+0.400223352 container start a73ea7d87b56ef79f258c637fbcc679af99c3747a5155d665577c23fc743b4b4 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-13T14:23:07.918 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:07 smithi102 podman[127135]: 2024-09-13 14:23:07.644785407 +0000 UTC m=+0.486874148 container attach a73ea7d87b56ef79f258c637fbcc679af99c3747a5155d665577c23fc743b4b4 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-13T14:23:07.918 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:07 smithi102 podman[127135]: 2024-09-13 14:23:07.741347906 +0000 UTC m=+0.583436644 container died a73ea7d87b56ef79f258c637fbcc679af99c3747a5155d665577c23fc743b4b4 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-13T14:23:08.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:07 smithi195 ceph-mon[104262]: osdmap e92: 8 total, 7 up, 8 in 2024-09-13T14:23:08.324 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:07 smithi102 ceph-mon[110183]: osdmap e92: 8 total, 7 up, 8 in 2024-09-13T14:23:08.324 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:07 smithi102 ceph-mon[115269]: osdmap e92: 8 total, 7 up, 8 in 2024-09-13T14:23:08.668 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:08 smithi102 podman[127178]: 2024-09-13 14:23:08.323730813 +0000 UTC m=+0.572056010 container remove a73ea7d87b56ef79f258c637fbcc679af99c3747a5155d665577c23fc743b4b4 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T14:23:08.668 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:08 smithi102 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@osd.0.service: Deactivated successfully. 2024-09-13T14:23:08.668 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:08 smithi102 systemd[1]: Stopped Ceph osd.0 for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:23:08.668 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:08 smithi102 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@osd.0.service: Consumed 6.654s CPU time. 2024-09-13T14:23:08.669 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:08 smithi102 systemd[1]: Starting Ceph osd.0 for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T14:23:09.086 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:08 smithi102 ceph-mon[110183]: pgmap v176: 73 pgs: 26 peering, 47 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:09.086 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:08 smithi102 ceph-mon[110183]: Health check failed: Reduced data availability: 9 pgs inactive, 18 pgs peering (PG_AVAILABILITY) 2024-09-13T14:23:09.086 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:08 smithi102 ceph-mon[115269]: pgmap v176: 73 pgs: 26 peering, 47 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:09.086 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:08 smithi102 ceph-mon[115269]: Health check failed: Reduced data availability: 9 pgs inactive, 18 pgs peering (PG_AVAILABILITY) 2024-09-13T14:23:09.087 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:08 smithi102 podman[127281]: 2024-09-13 14:23:08.656126389 +0000 UTC m=+0.015805005 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:23:09.087 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:08 smithi102 podman[127281]: 2024-09-13 14:23:08.783244374 +0000 UTC m=+0.142923056 container create c4f5ff5aead60aa9c6d37571c7915f95adc6ffacfdaa5a954e44c50ebc2dccf0 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0-activate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_CLEAN=True) 2024-09-13T14:23:09.087 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:08 smithi102 podman[127281]: 2024-09-13 14:23:08.985872024 +0000 UTC m=+0.345550643 container init c4f5ff5aead60aa9c6d37571c7915f95adc6ffacfdaa5a954e44c50ebc2dccf0 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0-activate, RELEASE=wip-aclamk-cbt-combined-d58f730, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-09-13T14:23:09.087 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:08 smithi102 podman[127281]: 2024-09-13 14:23:08.990704079 +0000 UTC m=+0.350382695 container start c4f5ff5aead60aa9c6d37571c7915f95adc6ffacfdaa5a954e44c50ebc2dccf0 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0-activate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-13T14:23:09.087 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:09 smithi102 podman[127281]: 2024-09-13 14:23:09.085693126 +0000 UTC m=+0.445371761 container attach c4f5ff5aead60aa9c6d37571c7915f95adc6ffacfdaa5a954e44c50ebc2dccf0 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0-activate, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T14:23:09.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:08 smithi195 ceph-mon[104262]: pgmap v176: 73 pgs: 26 peering, 47 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:09.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:08 smithi195 ceph-mon[104262]: Health check failed: Reduced data availability: 9 pgs inactive, 18 pgs peering (PG_AVAILABILITY) 2024-09-13T14:23:09.419 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:09 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0-activate[127296]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:23:09.419 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:09 smithi102 bash[127281]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:23:09.419 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:09 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0-activate[127296]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:23:09.419 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:09 smithi102 bash[127281]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:23:09.419 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:09 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0-activate[127296]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-09-13T14:23:09.419 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:09 smithi102 bash[127281]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-09-13T14:23:09.419 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:09 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0-activate[127296]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:23:09.419 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:09 smithi102 bash[127281]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:23:09.420 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:09 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0-activate[127296]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:23:09.420 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:09 smithi102 bash[127281]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:23:09.420 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:09 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0-activate[127296]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-13T14:23:09.420 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:09 smithi102 bash[127281]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-13T14:23:09.420 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:09 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0-activate[127296]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-0 --no-mon-config 2024-09-13T14:23:09.420 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:09 smithi102 bash[127281]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-0 --no-mon-config 2024-09-13T14:23:09.857 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:09 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0-activate[127296]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-0/block 2024-09-13T14:23:09.857 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:09 smithi102 bash[127281]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-0/block 2024-09-13T14:23:09.857 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:09 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0-activate[127296]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block 2024-09-13T14:23:09.857 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:09 smithi102 bash[127281]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block 2024-09-13T14:23:09.857 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:09 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0-activate[127296]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-13T14:23:09.858 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:09 smithi102 bash[127281]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-13T14:23:09.858 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:09 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0-activate[127296]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-13T14:23:09.858 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:09 smithi102 bash[127281]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-13T14:23:09.858 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:09 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0-activate[127296]: --> ceph-volume lvm activate successful for osd ID: 0 2024-09-13T14:23:09.858 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:09 smithi102 bash[127281]: --> ceph-volume lvm activate successful for osd ID: 0 2024-09-13T14:23:09.858 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:09 smithi102 podman[127281]: 2024-09-13 14:23:09.535557494 +0000 UTC m=+0.895236122 container died c4f5ff5aead60aa9c6d37571c7915f95adc6ffacfdaa5a954e44c50ebc2dccf0 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0-activate, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-13T14:23:10.418 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:10 smithi102 podman[127421]: 2024-09-13 14:23:10.156693263 +0000 UTC m=+0.612092272 container remove c4f5ff5aead60aa9c6d37571c7915f95adc6ffacfdaa5a954e44c50ebc2dccf0 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0-activate, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-13T14:23:10.418 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:10 smithi102 podman[127486]: 2024-09-13 14:23:10.267840248 +0000 UTC m=+0.015717490 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:23:10.418 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:10 smithi102 podman[127486]: 2024-09-13 14:23:10.38265082 +0000 UTC m=+0.130528057 container create d5c895f22b2a5ac04c6f0c37c5b546b2e36fd140969231235a504a9bfb77edd7 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-aclamk-cbt-combined-d58f730, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T14:23:10.918 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:10 smithi102 podman[127486]: 2024-09-13 14:23:10.62682224 +0000 UTC m=+0.374699478 container init d5c895f22b2a5ac04c6f0c37c5b546b2e36fd140969231235a504a9bfb77edd7 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=wip-aclamk-cbt-combined-d58f730, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T14:23:10.918 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:10 smithi102 podman[127486]: 2024-09-13 14:23:10.631452957 +0000 UTC m=+0.379330195 container start d5c895f22b2a5ac04c6f0c37c5b546b2e36fd140969231235a504a9bfb77edd7 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T14:23:10.918 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:10 smithi102 ceph-osd[127524]: -- 172.21.15.102:0/2508024972 <== mon.0 v2:172.21.15.102:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55945ef2da40 con 0x55945fcfa400 2024-09-13T14:23:10.918 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:10 smithi102 bash[127486]: d5c895f22b2a5ac04c6f0c37c5b546b2e36fd140969231235a504a9bfb77edd7 2024-09-13T14:23:10.918 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:10 smithi102 systemd[1]: Started Ceph osd.0 for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:23:11.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:10 smithi195 ceph-mon[104262]: pgmap v177: 73 pgs: 26 peering, 47 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:11.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:11.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:23:11.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:11.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:11.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:11.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:23:11.396 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:10 smithi102 ceph-mon[110183]: pgmap v177: 73 pgs: 26 peering, 47 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:11.397 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:11.397 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:23:11.397 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:11.397 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:11.397 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:11.397 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:23:11.397 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:10 smithi102 ceph-mon[115269]: pgmap v177: 73 pgs: 26 peering, 47 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:11.397 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:11.397 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:23:11.398 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:11.398 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:11.398 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:11.398 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:23:12.267 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:12 smithi102 ceph-mon[110183]: Health check failed: Degraded data redundancy: 13/84 objects degraded (15.476%), 8 pgs degraded (PG_DEGRADED) 2024-09-13T14:23:12.267 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:12 smithi102 ceph-mon[110183]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs inactive, 18 pgs peering) 2024-09-13T14:23:12.267 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:12 smithi102 ceph-mon[115269]: Health check failed: Degraded data redundancy: 13/84 objects degraded (15.476%), 8 pgs degraded (PG_DEGRADED) 2024-09-13T14:23:12.267 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:12 smithi102 ceph-mon[115269]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs inactive, 18 pgs peering) 2024-09-13T14:23:12.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:12 smithi195 ceph-mon[104262]: Health check failed: Degraded data redundancy: 13/84 objects degraded (15.476%), 8 pgs degraded (PG_DEGRADED) 2024-09-13T14:23:12.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:12 smithi195 ceph-mon[104262]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs inactive, 18 pgs peering) 2024-09-13T14:23:12.668 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:12 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0[127501]: 2024-09-13T14:23:12.265+0000 7fe9a6eab640 -1 Falling back to public interface 2024-09-13T14:23:13.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:13 smithi102 ceph-mon[110183]: pgmap v178: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-09-13T14:23:13.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:13 smithi102 ceph-mon[115269]: pgmap v178: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-09-13T14:23:13.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:13 smithi195 ceph-mon[104262]: pgmap v178: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-09-13T14:23:15.319 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:15 smithi102 ceph-mon[110183]: pgmap v179: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-09-13T14:23:15.320 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:15 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:15.320 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:15 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:15.320 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:15 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:15.320 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:15 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:15.320 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:15 smithi102 ceph-mon[115269]: pgmap v179: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-09-13T14:23:15.320 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:15 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:15.320 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:15 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:15.320 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:15 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:15.320 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:15 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:15.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:15 smithi195 ceph-mon[104262]: pgmap v179: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-09-13T14:23:15.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:15 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:15.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:15 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:15.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:15 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:15.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:15 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:15.918 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:15 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0[127501]: 2024-09-13T14:23:15.806+0000 7fe9a6eab640 -1 osd.0 90 log_to_monitors true 2024-09-13T14:23:16.284 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:16 smithi102 ceph-mon[110183]: from='osd.0 [v2:172.21.15.102:6802/3285732759,v1:172.21.15.102:6803/3285732759]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-13T14:23:16.285 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:16 smithi102 ceph-mon[115269]: from='osd.0 [v2:172.21.15.102:6802/3285732759,v1:172.21.15.102:6803/3285732759]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-13T14:23:16.285 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:23:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0[127501]: 2024-09-13T14:23:16.038+0000 7fe99ec55640 -1 osd.0 90 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T14:23:16.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:16 smithi195 ceph-mon[104262]: from='osd.0 [v2:172.21.15.102:6802/3285732759,v1:172.21.15.102:6803/3285732759]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-13T14:23:16.575 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:23:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:23:16] "GET /metrics HTTP/1.1" 200 33956 "" "Prometheus/2.51.0" 2024-09-13T14:23:17.292 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:17 smithi102 ceph-mon[110183]: pgmap v180: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-09-13T14:23:17.293 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:17 smithi102 ceph-mon[110183]: from='osd.0 [v2:172.21.15.102:6802/3285732759,v1:172.21.15.102:6803/3285732759]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-13T14:23:17.293 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:17 smithi102 ceph-mon[110183]: osdmap e93: 8 total, 7 up, 8 in 2024-09-13T14:23:17.293 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:17 smithi102 ceph-mon[110183]: from='osd.0 [v2:172.21.15.102:6802/3285732759,v1:172.21.15.102:6803/3285732759]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]} : dispatch 2024-09-13T14:23:17.293 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:17 smithi102 ceph-mon[115269]: pgmap v180: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-09-13T14:23:17.294 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:17 smithi102 ceph-mon[115269]: from='osd.0 [v2:172.21.15.102:6802/3285732759,v1:172.21.15.102:6803/3285732759]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-13T14:23:17.294 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:17 smithi102 ceph-mon[115269]: osdmap e93: 8 total, 7 up, 8 in 2024-09-13T14:23:17.294 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:17 smithi102 ceph-mon[115269]: from='osd.0 [v2:172.21.15.102:6802/3285732759,v1:172.21.15.102:6803/3285732759]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]} : dispatch 2024-09-13T14:23:17.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:17 smithi195 ceph-mon[104262]: pgmap v180: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-09-13T14:23:17.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:17 smithi195 ceph-mon[104262]: from='osd.0 [v2:172.21.15.102:6802/3285732759,v1:172.21.15.102:6803/3285732759]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-13T14:23:17.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:17 smithi195 ceph-mon[104262]: osdmap e93: 8 total, 7 up, 8 in 2024-09-13T14:23:17.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:17 smithi195 ceph-mon[104262]: from='osd.0 [v2:172.21.15.102:6802/3285732759,v1:172.21.15.102:6803/3285732759]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]} : dispatch 2024-09-13T14:23:18.418 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:23:18 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: 2024-09-13T14:23:18.189+0000 7f0c92e7e640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-09-13T14:23:18.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:18 smithi102 ceph-mon[110183]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:23:18.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:18 smithi102 ceph-mon[110183]: osd.0 [v2:172.21.15.102:6802/3285732759,v1:172.21.15.102:6803/3285732759] boot 2024-09-13T14:23:18.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:18 smithi102 ceph-mon[110183]: osdmap e94: 8 total, 8 up, 8 in 2024-09-13T14:23:18.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:18 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-13T14:23:18.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:18 smithi102 ceph-mon[110183]: osdmap e95: 8 total, 8 up, 8 in 2024-09-13T14:23:18.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:18 smithi102 ceph-mon[115269]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:23:18.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:18 smithi102 ceph-mon[115269]: osd.0 [v2:172.21.15.102:6802/3285732759,v1:172.21.15.102:6803/3285732759] boot 2024-09-13T14:23:18.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:18 smithi102 ceph-mon[115269]: osdmap e94: 8 total, 8 up, 8 in 2024-09-13T14:23:18.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:18 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-13T14:23:18.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:18 smithi102 ceph-mon[115269]: osdmap e95: 8 total, 8 up, 8 in 2024-09-13T14:23:18.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:18 smithi195 ceph-mon[104262]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:23:18.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:18 smithi195 ceph-mon[104262]: osd.0 [v2:172.21.15.102:6802/3285732759,v1:172.21.15.102:6803/3285732759] boot 2024-09-13T14:23:18.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:18 smithi195 ceph-mon[104262]: osdmap e94: 8 total, 8 up, 8 in 2024-09-13T14:23:18.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:18 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-13T14:23:18.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:18 smithi195 ceph-mon[104262]: osdmap e95: 8 total, 8 up, 8 in 2024-09-13T14:23:19.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:19 smithi102 ceph-mon[110183]: pgmap v183: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 253 MiB used, 715 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-09-13T14:23:19.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:19 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:19.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:19 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:19.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:19 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:23:19.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:19 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:23:19.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:19 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:19.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:19 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:23:19.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:19 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:23:19.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:19 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:23:19.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:19 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:23:19.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:19 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-13T14:23:19.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:19 smithi102 ceph-mon[115269]: pgmap v183: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 253 MiB used, 715 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-09-13T14:23:19.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:19 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:19.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:19 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:19.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:19 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:23:19.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:19 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:23:19.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:19 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:19.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:19 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:23:19.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:19 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:23:19.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:19 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:23:19.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:19 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:23:19.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:19 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-13T14:23:19.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:19 smithi195 ceph-mon[104262]: pgmap v183: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 253 MiB used, 715 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-09-13T14:23:19.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:19 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:19.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:19 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:19.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:19 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:23:19.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:19 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:23:19.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:19 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:19.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:19 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:23:19.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:19 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:23:19.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:19 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:23:19.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:19 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:23:19.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:19 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-13T14:23:20.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:20 smithi102 ceph-mon[115269]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-13T14:23:20.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:20 smithi102 ceph-mon[115269]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-09-13T14:23:20.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:20 smithi102 ceph-mon[110183]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-13T14:23:20.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:20 smithi102 ceph-mon[110183]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-09-13T14:23:20.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:20 smithi195 ceph-mon[104262]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-13T14:23:20.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:20 smithi195 ceph-mon[104262]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-09-13T14:23:21.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:21 smithi102 ceph-mon[110183]: pgmap v185: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 253 MiB used, 715 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-09-13T14:23:21.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:21 smithi102 ceph-mon[115269]: pgmap v185: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 253 MiB used, 715 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-09-13T14:23:21.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:21 smithi195 ceph-mon[104262]: pgmap v185: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 253 MiB used, 715 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-09-13T14:23:22.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:22 smithi102 ceph-mon[110183]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 13/84 objects degraded (15.476%), 8 pgs degraded) 2024-09-13T14:23:22.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:22 smithi102 ceph-mon[110183]: Cluster is now healthy 2024-09-13T14:23:22.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:22 smithi102 ceph-mon[115269]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 13/84 objects degraded (15.476%), 8 pgs degraded) 2024-09-13T14:23:22.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:22 smithi102 ceph-mon[115269]: Cluster is now healthy 2024-09-13T14:23:22.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:22 smithi195 ceph-mon[104262]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 13/84 objects degraded (15.476%), 8 pgs degraded) 2024-09-13T14:23:22.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:22 smithi195 ceph-mon[104262]: Cluster is now healthy 2024-09-13T14:23:23.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:23 smithi102 ceph-mon[110183]: pgmap v186: 73 pgs: 73 active+clean; 4.6 MiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:23.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:23 smithi102 ceph-mon[115269]: pgmap v186: 73 pgs: 73 active+clean; 4.6 MiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:23.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:23 smithi195 ceph-mon[104262]: pgmap v186: 73 pgs: 73 active+clean; 4.6 MiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:25.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:25 smithi102 ceph-mon[110183]: pgmap v187: 73 pgs: 73 active+clean; 4.6 MiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:25.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:25 smithi102 ceph-mon[115269]: pgmap v187: 73 pgs: 73 active+clean; 4.6 MiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:25.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:25 smithi195 ceph-mon[104262]: pgmap v187: 73 pgs: 73 active+clean; 4.6 MiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:26.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:26 smithi102 ceph-mon[115269]: pgmap v188: 73 pgs: 73 active+clean; 4.6 MiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:26.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:26.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:23:26.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:26 smithi102 ceph-mon[110183]: pgmap v188: 73 pgs: 73 active+clean; 4.6 MiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:26.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:26.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:23:26.670 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:23:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:23:26] "GET /metrics HTTP/1.1" 200 33954 "" "Prometheus/2.51.0" 2024-09-13T14:23:26.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:26 smithi195 ceph-mon[104262]: pgmap v188: 73 pgs: 73 active+clean; 4.6 MiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:26.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:26.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:23:28.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:28 smithi102 ceph-mon[110183]: pgmap v189: 73 pgs: 73 active+clean; 4.6 MiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:28.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:28 smithi102 ceph-mon[115269]: pgmap v189: 73 pgs: 73 active+clean; 4.6 MiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:28.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:28 smithi195 ceph-mon[104262]: pgmap v189: 73 pgs: 73 active+clean; 4.6 MiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:30.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:30 smithi102 ceph-mon[110183]: pgmap v190: 73 pgs: 73 active+clean; 4.6 MiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:30.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:30 smithi102 ceph-mon[115269]: pgmap v190: 73 pgs: 73 active+clean; 4.6 MiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:30.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:30 smithi195 ceph-mon[104262]: pgmap v190: 73 pgs: 73 active+clean; 4.6 MiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:31.518 INFO:teuthology.orchestra.run.smithi102.stdout:true 2024-09-13T14:23:31.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:31 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:31.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:31 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:31.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:31 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:32.073 INFO:teuthology.orchestra.run.smithi102.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-13T14:23:32.073 INFO:teuthology.orchestra.run.smithi102.stdout:alertmanager.a smithi102 *:9093,9094 running (7m) 17s ago 13m 19.2M - 0.27.0 11f11916f8cd 0322b99ec767 2024-09-13T14:23:32.073 INFO:teuthology.orchestra.run.smithi102.stdout:grafana.a smithi195 *:3000 running (7m) 41s ago 12m 84.1M - c8b91775d855 152e89bad2c8 2024-09-13T14:23:32.073 INFO:teuthology.orchestra.run.smithi102.stdout:mds.foo.smithi102.njetxr smithi102 running (11m) 17s ago 11m 23.2M - 18.2.4-843-g2c01458a c129f5a2fd4e c9211cc725ed 2024-09-13T14:23:32.073 INFO:teuthology.orchestra.run.smithi102.stdout:mds.foo.smithi195.yqzrxi smithi195 running (11m) 41s ago 11m 19.1M - 18.2.4-843-g2c01458a c129f5a2fd4e eea37a7230fd 2024-09-13T14:23:32.073 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.x smithi195 *:8443,9283,8765 running (8m) 41s ago 16m 456M - 19.3.0-4697-gd58f7306 fec07016a2b0 4a23b8f0ea8b 2024-09-13T14:23:32.073 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.y smithi102 *:8443,9283,8765 running (8m) 17s ago 18m 517M - 19.3.0-4697-gd58f7306 fec07016a2b0 6b12b28dcde1 2024-09-13T14:23:32.073 INFO:teuthology.orchestra.run.smithi102.stdout:mon.a smithi102 running (6m) 17s ago 18m 52.7M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 f3234b7bbdde 2024-09-13T14:23:32.073 INFO:teuthology.orchestra.run.smithi102.stdout:mon.b smithi195 running (3m) 41s ago 17m 42.9M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 9921a2aa8168 2024-09-13T14:23:32.073 INFO:teuthology.orchestra.run.smithi102.stdout:mon.c smithi102 running (5m) 17s ago 17m 40.9M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 dba6b3d55296 2024-09-13T14:23:32.073 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.a smithi102 *:9100 running (8m) 17s ago 12m 12.3M - 1.7.0 72c9c2088986 0c92d0b39468 2024-09-13T14:23:32.073 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.b smithi195 *:9100 running (8m) 41s ago 12m 11.2M - 1.7.0 72c9c2088986 fde867a7dbf9 2024-09-13T14:23:32.074 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0 smithi102 running (21s) 17s ago 16m 12.2M 2480M 19.3.0-4697-gd58f7306 fec07016a2b0 d5c895f22b2a 2024-09-13T14:23:32.074 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1 smithi102 running (16m) 17s ago 16m 53.1M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 8a67356be7a2 2024-09-13T14:23:32.074 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2 smithi102 running (15m) 17s ago 15m 51.0M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 1800986ac838 2024-09-13T14:23:32.074 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3 smithi102 running (15m) 17s ago 15m 58.0M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 968246da9da2 2024-09-13T14:23:32.074 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4 smithi195 running (15m) 41s ago 15m 50.4M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 515307701d73 2024-09-13T14:23:32.074 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5 smithi195 running (14m) 41s ago 14m 54.1M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 176ba2b3bd59 2024-09-13T14:23:32.074 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6 smithi195 running (14m) 41s ago 14m 56.0M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 9cd070a6dd3d 2024-09-13T14:23:32.074 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7 smithi195 running (14m) 41s ago 14m 54.4M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e a6b698c659ca 2024-09-13T14:23:32.074 INFO:teuthology.orchestra.run.smithi102.stdout:prometheus.a smithi195 *:9095 running (7m) 41s ago 13m 48.2M - 2.51.0 1d3b7f56885b d131e53f1ef9 2024-09-13T14:23:32.403 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:23:32.403 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": { 2024-09-13T14:23:32.404 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 3 2024-09-13T14:23:32.404 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:23:32.404 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": { 2024-09-13T14:23:32.404 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 2 2024-09-13T14:23:32.404 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:23:32.404 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": { 2024-09-13T14:23:32.404 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 7, 2024-09-13T14:23:32.404 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 1 2024-09-13T14:23:32.404 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:23:32.404 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": { 2024-09-13T14:23:32.404 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 2 2024-09-13T14:23:32.404 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:23:32.404 INFO:teuthology.orchestra.run.smithi102.stdout: "overall": { 2024-09-13T14:23:32.405 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 9, 2024-09-13T14:23:32.405 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 6 2024-09-13T14:23:32.405 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:23:32.405 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:23:32.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:32 smithi102 ceph-mon[115269]: pgmap v191: 73 pgs: 73 active+clean; 4.6 MiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:32.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:32 smithi102 ceph-mon[115269]: from='client.34311 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:32.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:32 smithi102 ceph-mon[115269]: from='client.34317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:32.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:32 smithi102 ceph-mon[115269]: from='client.44246 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:32.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:32 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2453097448' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:23:32.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:32 smithi102 ceph-mon[110183]: pgmap v191: 73 pgs: 73 active+clean; 4.6 MiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:32.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:32 smithi102 ceph-mon[110183]: from='client.34311 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:32.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:32 smithi102 ceph-mon[110183]: from='client.34317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:32.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:32 smithi102 ceph-mon[110183]: from='client.44246 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:32.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:32 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2453097448' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:23:32.690 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:23:32.690 INFO:teuthology.orchestra.run.smithi102.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f", 2024-09-13T14:23:32.690 INFO:teuthology.orchestra.run.smithi102.stdout: "in_progress": true, 2024-09-13T14:23:32.690 INFO:teuthology.orchestra.run.smithi102.stdout: "which": "Upgrading daemons of type(s) osd. Upgrade limited to 4 daemons (3 remaining).", 2024-09-13T14:23:32.690 INFO:teuthology.orchestra.run.smithi102.stdout: "services_complete": [], 2024-09-13T14:23:32.690 INFO:teuthology.orchestra.run.smithi102.stdout: "progress": "1/8 daemons upgraded", 2024-09-13T14:23:32.690 INFO:teuthology.orchestra.run.smithi102.stdout: "message": "Currently upgrading osd daemons", 2024-09-13T14:23:32.691 INFO:teuthology.orchestra.run.smithi102.stdout: "is_paused": false 2024-09-13T14:23:32.691 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:23:32.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:32 smithi195 ceph-mon[104262]: pgmap v191: 73 pgs: 73 active+clean; 4.6 MiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:32.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:32 smithi195 ceph-mon[104262]: from='client.34311 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:32.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:32 smithi195 ceph-mon[104262]: from='client.34317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:32.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:32 smithi195 ceph-mon[104262]: from='client.44246 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:32.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:32 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2453097448' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:23:33.722 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:33 smithi102 ceph-mon[110183]: from='client.34335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:33.722 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:33 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-13T14:23:33.723 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:33 smithi102 ceph-mon[115269]: from='client.34335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:33.723 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:33 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-13T14:23:33.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:33 smithi195 ceph-mon[104262]: from='client.34335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:33.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:33 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-13T14:23:34.575 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:34 smithi102 ceph-mon[110183]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-13T14:23:34.576 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:34 smithi102 ceph-mon[110183]: Upgrade: osd.1 is safe to restart 2024-09-13T14:23:34.576 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:34 smithi102 ceph-mon[110183]: pgmap v192: 73 pgs: 73 active+clean; 4.6 MiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:34.576 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:34 smithi102 ceph-mon[115269]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-13T14:23:34.576 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:34 smithi102 ceph-mon[115269]: Upgrade: osd.1 is safe to restart 2024-09-13T14:23:34.576 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:34 smithi102 ceph-mon[115269]: pgmap v192: 73 pgs: 73 active+clean; 4.6 MiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:34.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:34 smithi195 ceph-mon[104262]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-13T14:23:34.750 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:34 smithi195 ceph-mon[104262]: Upgrade: osd.1 is safe to restart 2024-09-13T14:23:34.750 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:34 smithi195 ceph-mon[104262]: pgmap v192: 73 pgs: 73 active+clean; 4.6 MiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:35.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:35 smithi102 ceph-mon[110183]: Upgrade: Updating osd.1 2024-09-13T14:23:35.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:35 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:35.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:35 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-13T14:23:35.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:35 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:23:35.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:35 smithi102 ceph-mon[110183]: Deploying daemon osd.1 on smithi102 2024-09-13T14:23:35.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:35 smithi102 ceph-mon[115269]: Upgrade: Updating osd.1 2024-09-13T14:23:35.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:35 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:35.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:35 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-13T14:23:35.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:35 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:23:35.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:35 smithi102 ceph-mon[115269]: Deploying daemon osd.1 on smithi102 2024-09-13T14:23:35.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:35 smithi195 ceph-mon[104262]: Upgrade: Updating osd.1 2024-09-13T14:23:35.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:35 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:35.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:35 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-13T14:23:35.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:35 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:23:35.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:35 smithi195 ceph-mon[104262]: Deploying daemon osd.1 on smithi102 2024-09-13T14:23:36.530 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:23:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:23:36] "GET /metrics HTTP/1.1" 200 33954 "" "Prometheus/2.51.0" 2024-09-13T14:23:36.881 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:36 smithi102 ceph-mon[110183]: pgmap v193: 73 pgs: 73 active+clean; 4.6 MiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:36.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:36 smithi102 ceph-mon[115269]: pgmap v193: 73 pgs: 73 active+clean; 4.6 MiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:36.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:36 smithi195 ceph-mon[104262]: pgmap v193: 73 pgs: 73 active+clean; 4.6 MiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:37.789 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:37 smithi102 systemd[1]: Stopping Ceph osd.1 for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T14:23:38.168 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:37 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1[51038]: 2024-09-13T14:23:37.787+0000 7f31eebdc640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T14:23:38.168 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:37 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1[51038]: 2024-09-13T14:23:37.787+0000 7f31eebdc640 -1 osd.1 95 *** Got signal Terminated *** 2024-09-13T14:23:38.168 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:37 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1[51038]: 2024-09-13T14:23:37.787+0000 7f31eebdc640 -1 osd.1 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T14:23:38.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:38 smithi102 ceph-mon[110183]: pgmap v194: 73 pgs: 73 active+clean; 4.6 MiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:38.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:38 smithi102 ceph-mon[110183]: osd.1 marked itself down and dead 2024-09-13T14:23:38.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:38 smithi102 ceph-mon[115269]: pgmap v194: 73 pgs: 73 active+clean; 4.6 MiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:38.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:38 smithi102 ceph-mon[115269]: osd.1 marked itself down and dead 2024-09-13T14:23:38.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:38 smithi195 ceph-mon[104262]: pgmap v194: 73 pgs: 73 active+clean; 4.6 MiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:38.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:38 smithi195 ceph-mon[104262]: osd.1 marked itself down and dead 2024-09-13T14:23:39.279 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:39 smithi102 podman[132078]: 2024-09-13 14:23:39.014097218 +0000 UTC m=+1.320371410 container died 8a67356be7a2094f006609c0037c3ebab0c83d3a2f07629d01a259baf680781a (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=reef-2c01458, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-13T14:23:39.626 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:39 smithi102 podman[132078]: 2024-09-13 14:23:39.377555489 +0000 UTC m=+1.683829681 container cleanup 8a67356be7a2094f006609c0037c3ebab0c83d3a2f07629d01a259baf680781a (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, RELEASE=reef-2c01458, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-09-13T14:23:39.626 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:39 smithi102 bash[132078]: ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1 2024-09-13T14:23:39.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:39 smithi102 ceph-mon[110183]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:23:39.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:39 smithi102 ceph-mon[110183]: osdmap e96: 8 total, 7 up, 8 in 2024-09-13T14:23:39.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:39 smithi102 ceph-mon[115269]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:23:39.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:39 smithi102 ceph-mon[115269]: osdmap e96: 8 total, 7 up, 8 in 2024-09-13T14:23:39.919 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:39 smithi102 podman[132099]: 2024-09-13 14:23:39.704727682 +0000 UTC m=+0.687419305 container remove 8a67356be7a2094f006609c0037c3ebab0c83d3a2f07629d01a259baf680781a (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, RELEASE=reef-2c01458, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-09-13T14:23:39.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:39 smithi195 ceph-mon[104262]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:23:39.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:39 smithi195 ceph-mon[104262]: osdmap e96: 8 total, 7 up, 8 in 2024-09-13T14:23:40.367 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:40 smithi102 podman[132194]: 2024-09-13 14:23:39.947950213 +0000 UTC m=+0.015630986 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:23:40.367 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:40 smithi102 podman[132194]: 2024-09-13 14:23:40.072445617 +0000 UTC m=+0.140126423 container create b18c6ca86eda352cd3321781bbda7b0d4d967dd90f6be9c4d9bb436a242d29ad (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1-deactivate, RELEASE=wip-aclamk-cbt-combined-d58f730, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2) 2024-09-13T14:23:40.367 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:40 smithi102 podman[132194]: 2024-09-13 14:23:40.283202725 +0000 UTC m=+0.350883489 container init b18c6ca86eda352cd3321781bbda7b0d4d967dd90f6be9c4d9bb436a242d29ad (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-13T14:23:40.367 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:40 smithi102 podman[132194]: 2024-09-13 14:23:40.288182115 +0000 UTC m=+0.355862877 container start b18c6ca86eda352cd3321781bbda7b0d4d967dd90f6be9c4d9bb436a242d29ad (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-13T14:23:40.367 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:40 smithi102 podman[132194]: 2024-09-13 14:23:40.366375579 +0000 UTC m=+0.434056352 container attach b18c6ca86eda352cd3321781bbda7b0d4d967dd90f6be9c4d9bb436a242d29ad (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1-deactivate, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-aclamk-cbt-combined-d58f730, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T14:23:40.644 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:40 smithi102 podman[132194]: 2024-09-13 14:23:40.473548921 +0000 UTC m=+0.541229694 container died b18c6ca86eda352cd3321781bbda7b0d4d967dd90f6be9c4d9bb436a242d29ad (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-aclamk-cbt-combined-d58f730, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-13T14:23:40.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:40 smithi102 ceph-mon[110183]: pgmap v196: 73 pgs: 8 stale+active+clean, 65 active+clean; 4.6 MiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:40.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:40 smithi102 ceph-mon[110183]: osdmap e97: 8 total, 7 up, 8 in 2024-09-13T14:23:40.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:40 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:40.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:40 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:23:40.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:40 smithi102 ceph-mon[115269]: pgmap v196: 73 pgs: 8 stale+active+clean, 65 active+clean; 4.6 MiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:40.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:40 smithi102 ceph-mon[115269]: osdmap e97: 8 total, 7 up, 8 in 2024-09-13T14:23:40.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:40 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:40.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:40 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:23:40.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:40 smithi195 ceph-mon[104262]: pgmap v196: 73 pgs: 8 stale+active+clean, 65 active+clean; 4.6 MiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:40.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:40 smithi195 ceph-mon[104262]: osdmap e97: 8 total, 7 up, 8 in 2024-09-13T14:23:40.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:40 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:40.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:40 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:23:41.418 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:41 smithi102 podman[132237]: 2024-09-13 14:23:41.062195181 +0000 UTC m=+0.582835866 container remove b18c6ca86eda352cd3321781bbda7b0d4d967dd90f6be9c4d9bb436a242d29ad (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T14:23:41.418 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:41 smithi102 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@osd.1.service: Deactivated successfully. 2024-09-13T14:23:41.418 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:41 smithi102 systemd[1]: Stopped Ceph osd.1 for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:23:41.418 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:41 smithi102 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@osd.1.service: Consumed 7.302s CPU time. 2024-09-13T14:23:41.419 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:41 smithi102 systemd[1]: Starting Ceph osd.1 for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T14:23:41.817 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:41 smithi102 ceph-mon[110183]: Health check failed: Degraded data redundancy: 6/84 objects degraded (7.143%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:23:41.817 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:41 smithi102 ceph-mon[115269]: Health check failed: Degraded data redundancy: 6/84 objects degraded (7.143%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:23:41.817 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:41 smithi102 podman[132341]: 2024-09-13 14:23:41.410231231 +0000 UTC m=+0.016068759 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:23:41.817 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:41 smithi102 podman[132341]: 2024-09-13 14:23:41.538297183 +0000 UTC m=+0.144134697 container create 79e1a9bc19bc32c2139c012a8a3bbbefa9180876b513bf04f69addfca19cbb9d (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1-activate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-13T14:23:41.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:41 smithi195 ceph-mon[104262]: Health check failed: Degraded data redundancy: 6/84 objects degraded (7.143%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:23:42.169 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:41 smithi102 podman[132341]: 2024-09-13 14:23:41.81664477 +0000 UTC m=+0.422482284 container init 79e1a9bc19bc32c2139c012a8a3bbbefa9180876b513bf04f69addfca19cbb9d (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1-activate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-13T14:23:42.169 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:41 smithi102 podman[132341]: 2024-09-13 14:23:41.821532148 +0000 UTC m=+0.427369679 container start 79e1a9bc19bc32c2139c012a8a3bbbefa9180876b513bf04f69addfca19cbb9d (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1-activate, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-aclamk-cbt-combined-d58f730, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-13T14:23:42.169 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:41 smithi102 podman[132341]: 2024-09-13 14:23:41.924700485 +0000 UTC m=+0.530538019 container attach 79e1a9bc19bc32c2139c012a8a3bbbefa9180876b513bf04f69addfca19cbb9d (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1-activate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True) 2024-09-13T14:23:42.169 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:41 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1-activate[132356]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:23:42.169 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:41 smithi102 bash[132341]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:23:42.169 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:41 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1-activate[132356]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:23:42.169 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:41 smithi102 bash[132341]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:23:42.169 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:41 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1-activate[132356]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-09-13T14:23:42.169 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:41 smithi102 bash[132341]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-09-13T14:23:42.169 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:41 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1-activate[132356]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:23:42.170 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:41 smithi102 bash[132341]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:23:42.170 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:41 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1-activate[132356]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:23:42.170 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:41 smithi102 bash[132341]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:23:42.170 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:42 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1-activate[132356]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-13T14:23:42.170 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:42 smithi102 bash[132341]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-13T14:23:42.170 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:42 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1-activate[132356]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-1 --no-mon-config 2024-09-13T14:23:42.170 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:42 smithi102 bash[132341]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-1 --no-mon-config 2024-09-13T14:23:42.668 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:42 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1-activate[132356]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-1/block 2024-09-13T14:23:42.668 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:42 smithi102 bash[132341]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-1/block 2024-09-13T14:23:42.669 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:42 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1-activate[132356]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-1/block 2024-09-13T14:23:42.669 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:42 smithi102 bash[132341]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-1/block 2024-09-13T14:23:42.669 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:42 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1-activate[132356]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-13T14:23:42.669 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:42 smithi102 bash[132341]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-13T14:23:42.669 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:42 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1-activate[132356]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-13T14:23:42.669 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:42 smithi102 bash[132341]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-13T14:23:42.669 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:42 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1-activate[132356]: --> ceph-volume lvm activate successful for osd ID: 1 2024-09-13T14:23:42.669 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:42 smithi102 bash[132341]: --> ceph-volume lvm activate successful for osd ID: 1 2024-09-13T14:23:42.669 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:42 smithi102 podman[132341]: 2024-09-13 14:23:42.378876798 +0000 UTC m=+0.984714331 container died 79e1a9bc19bc32c2139c012a8a3bbbefa9180876b513bf04f69addfca19cbb9d (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1-activate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-aclamk-cbt-combined-d58f730, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-13T14:23:42.962 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:42 smithi102 ceph-mon[110183]: pgmap v198: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 253 MiB used, 715 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-09-13T14:23:42.963 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:42 smithi102 ceph-mon[115269]: pgmap v198: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 253 MiB used, 715 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-09-13T14:23:42.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:42 smithi195 ceph-mon[104262]: pgmap v198: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 253 MiB used, 715 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-09-13T14:23:43.307 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:42 smithi102 podman[132492]: 2024-09-13 14:23:42.962090896 +0000 UTC m=+0.572968804 container remove 79e1a9bc19bc32c2139c012a8a3bbbefa9180876b513bf04f69addfca19cbb9d (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1-activate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, RELEASE=wip-aclamk-cbt-combined-d58f730, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-13T14:23:43.307 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:43 smithi102 podman[132558]: 2024-09-13 14:23:43.069947286 +0000 UTC m=+0.015793357 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:23:43.307 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:43 smithi102 podman[132558]: 2024-09-13 14:23:43.179764282 +0000 UTC m=+0.125610389 container create 5d264b086bc0b2c6d454c11958068b5a7787de1c28bb3b0aae62e9d74c35fab9 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-09-13T14:23:43.668 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:43 smithi102 podman[132558]: 2024-09-13 14:23:43.382271945 +0000 UTC m=+0.328118011 container init 5d264b086bc0b2c6d454c11958068b5a7787de1c28bb3b0aae62e9d74c35fab9 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-13T14:23:43.668 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:43 smithi102 podman[132558]: 2024-09-13 14:23:43.387126065 +0000 UTC m=+0.332972130 container start 5d264b086bc0b2c6d454c11958068b5a7787de1c28bb3b0aae62e9d74c35fab9 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-09-13T14:23:43.669 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:43 smithi102 ceph-osd[132596]: -- 172.21.15.102:0/108421126 <== mon.2 v2:172.21.15.102:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d864849a40 con 0x55d8649ec000 2024-09-13T14:23:43.669 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:43 smithi102 bash[132558]: 5d264b086bc0b2c6d454c11958068b5a7787de1c28bb3b0aae62e9d74c35fab9 2024-09-13T14:23:43.669 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:43 smithi102 systemd[1]: Started Ceph osd.1 for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:23:44.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:44 smithi102 ceph-mon[110183]: pgmap v199: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 253 MiB used, 715 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-09-13T14:23:44.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:44 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:44.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:44 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:44.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:44 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:44.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:44 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:23:44.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:44 smithi102 ceph-mon[115269]: pgmap v199: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 253 MiB used, 715 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-09-13T14:23:44.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:44 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:44.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:44 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:44.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:44 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:44.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:44 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:23:44.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:44 smithi195 ceph-mon[104262]: pgmap v199: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 253 MiB used, 715 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-09-13T14:23:44.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:44 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:44.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:44 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:44.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:44 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:44.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:44 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:23:45.290 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:45 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1[132573]: 2024-09-13T14:23:45.037+0000 7f1d38af6640 -1 Falling back to public interface 2024-09-13T14:23:46.806 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:46 smithi102 ceph-mon[110183]: pgmap v200: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 253 MiB used, 715 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-09-13T14:23:46.806 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:46 smithi102 ceph-mon[115269]: pgmap v200: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 253 MiB used, 715 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-09-13T14:23:46.807 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:23:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:23:46] "GET /metrics HTTP/1.1" 200 33953 "" "Prometheus/2.51.0" 2024-09-13T14:23:46.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:46 smithi195 ceph-mon[104262]: pgmap v200: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 253 MiB used, 715 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-09-13T14:23:48.418 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:48 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1[132573]: 2024-09-13T14:23:48.352+0000 7f1d38af6640 -1 osd.1 95 log_to_monitors true 2024-09-13T14:23:48.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:48 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:48.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:48 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:48.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:48 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:48.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:48 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:48.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:48 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:48.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:48 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:48.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:48 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:48.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:48 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:48.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:48 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:48.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:48 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:48.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:48 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:48.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:48 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:49.337 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:49 smithi102 ceph-mon[110183]: pgmap v201: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 253 MiB used, 715 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-09-13T14:23:49.337 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:49 smithi102 ceph-mon[110183]: from='osd.1 [v2:172.21.15.102:6810/2240964461,v1:172.21.15.102:6811/2240964461]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-13T14:23:49.337 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:49 smithi102 ceph-mon[110183]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-13T14:23:49.337 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:49 smithi102 ceph-mon[115269]: pgmap v201: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 253 MiB used, 715 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-09-13T14:23:49.338 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:49 smithi102 ceph-mon[115269]: from='osd.1 [v2:172.21.15.102:6810/2240964461,v1:172.21.15.102:6811/2240964461]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-13T14:23:49.338 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:49 smithi102 ceph-mon[115269]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-13T14:23:49.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:49 smithi195 ceph-mon[104262]: pgmap v201: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 253 MiB used, 715 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-09-13T14:23:49.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:49 smithi195 ceph-mon[104262]: from='osd.1 [v2:172.21.15.102:6810/2240964461,v1:172.21.15.102:6811/2240964461]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-13T14:23:49.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:49 smithi195 ceph-mon[104262]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-13T14:23:49.918 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:23:49 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1[132573]: 2024-09-13T14:23:49.596+0000 7f1d2c898640 -1 osd.1 95 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T14:23:50.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:50 smithi102 ceph-mon[110183]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-13T14:23:50.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:50 smithi102 ceph-mon[110183]: osdmap e98: 8 total, 7 up, 8 in 2024-09-13T14:23:50.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:50 smithi102 ceph-mon[110183]: from='osd.1 [v2:172.21.15.102:6810/2240964461,v1:172.21.15.102:6811/2240964461]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]} : dispatch 2024-09-13T14:23:50.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:50 smithi102 ceph-mon[110183]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]} : dispatch 2024-09-13T14:23:50.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:50 smithi102 ceph-mon[110183]: from='osd.1 ' entity='osd.1' 2024-09-13T14:23:50.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:50 smithi102 ceph-mon[115269]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-13T14:23:50.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:50 smithi102 ceph-mon[115269]: osdmap e98: 8 total, 7 up, 8 in 2024-09-13T14:23:50.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:50 smithi102 ceph-mon[115269]: from='osd.1 [v2:172.21.15.102:6810/2240964461,v1:172.21.15.102:6811/2240964461]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]} : dispatch 2024-09-13T14:23:50.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:50 smithi102 ceph-mon[115269]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]} : dispatch 2024-09-13T14:23:50.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:50 smithi102 ceph-mon[115269]: from='osd.1 ' entity='osd.1' 2024-09-13T14:23:50.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:50 smithi195 ceph-mon[104262]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-13T14:23:50.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:50 smithi195 ceph-mon[104262]: osdmap e98: 8 total, 7 up, 8 in 2024-09-13T14:23:50.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:50 smithi195 ceph-mon[104262]: from='osd.1 [v2:172.21.15.102:6810/2240964461,v1:172.21.15.102:6811/2240964461]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]} : dispatch 2024-09-13T14:23:50.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:50 smithi195 ceph-mon[104262]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]} : dispatch 2024-09-13T14:23:50.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:50 smithi195 ceph-mon[104262]: from='osd.1 ' entity='osd.1' 2024-09-13T14:23:51.154 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:23:50 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: 2024-09-13T14:23:50.966+0000 7f0c92e7e640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-09-13T14:23:51.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:51 smithi102 ceph-mon[110183]: pgmap v203: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 250 MiB used, 715 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-09-13T14:23:51.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:51 smithi102 ceph-mon[110183]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:23:51.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:51 smithi102 ceph-mon[110183]: osd.1 [v2:172.21.15.102:6810/2240964461,v1:172.21.15.102:6811/2240964461] boot 2024-09-13T14:23:51.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:51 smithi102 ceph-mon[110183]: osdmap e99: 8 total, 8 up, 8 in 2024-09-13T14:23:51.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:51 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-13T14:23:51.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:51 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:51.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:51 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:51.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:51 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:23:51.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:51 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:23:51.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:51 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:51.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:51 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:23:51.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:51 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:23:51.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:51 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:23:51.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:51 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:23:51.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:51 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:23:51.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:51 smithi102 ceph-mon[115269]: pgmap v203: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 250 MiB used, 715 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-09-13T14:23:51.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:51 smithi102 ceph-mon[115269]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:23:51.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:51 smithi102 ceph-mon[115269]: osd.1 [v2:172.21.15.102:6810/2240964461,v1:172.21.15.102:6811/2240964461] boot 2024-09-13T14:23:51.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:51 smithi102 ceph-mon[115269]: osdmap e99: 8 total, 8 up, 8 in 2024-09-13T14:23:51.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:51 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-13T14:23:51.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:51 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:51.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:51 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:51.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:51 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:23:51.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:51 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:23:51.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:51 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:51.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:51 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:23:51.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:51 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:23:51.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:51 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:23:51.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:51 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:23:51.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:51 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:23:51.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:51 smithi195 ceph-mon[104262]: pgmap v203: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 250 MiB used, 715 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-09-13T14:23:51.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:51 smithi195 ceph-mon[104262]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:23:51.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:51 smithi195 ceph-mon[104262]: osd.1 [v2:172.21.15.102:6810/2240964461,v1:172.21.15.102:6811/2240964461] boot 2024-09-13T14:23:51.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:51 smithi195 ceph-mon[104262]: osdmap e99: 8 total, 8 up, 8 in 2024-09-13T14:23:51.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:51 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-13T14:23:51.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:51 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:51.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:51 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:51.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:51 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:23:51.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:51 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:23:51.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:51 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:51.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:51 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:23:51.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:51 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:23:51.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:51 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:23:51.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:51 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:23:51.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:51 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:23:52.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:52 smithi102 ceph-mon[110183]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:23:52.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:52 smithi102 ceph-mon[110183]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-09-13T14:23:52.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:52 smithi102 ceph-mon[110183]: osdmap e100: 8 total, 8 up, 8 in 2024-09-13T14:23:52.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:52 smithi102 ceph-mon[115269]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:23:52.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:52 smithi102 ceph-mon[115269]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-09-13T14:23:52.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:52 smithi102 ceph-mon[115269]: osdmap e100: 8 total, 8 up, 8 in 2024-09-13T14:23:52.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:52 smithi195 ceph-mon[104262]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:23:52.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:52 smithi195 ceph-mon[104262]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-09-13T14:23:52.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:52 smithi195 ceph-mon[104262]: osdmap e100: 8 total, 8 up, 8 in 2024-09-13T14:23:53.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:53 smithi102 ceph-mon[115269]: pgmap v205: 73 pgs: 24 peering, 7 active+undersized, 1 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1/84 objects degraded (1.190%) 2024-09-13T14:23:53.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:53 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 1/84 objects degraded (1.190%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:23:53.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:53 smithi102 ceph-mon[110183]: pgmap v205: 73 pgs: 24 peering, 7 active+undersized, 1 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1/84 objects degraded (1.190%) 2024-09-13T14:23:53.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:53 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 1/84 objects degraded (1.190%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:23:53.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:53 smithi195 ceph-mon[104262]: pgmap v205: 73 pgs: 24 peering, 7 active+undersized, 1 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1/84 objects degraded (1.190%) 2024-09-13T14:23:53.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:53 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 1/84 objects degraded (1.190%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:23:55.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:55 smithi102 ceph-mon[115269]: pgmap v207: 73 pgs: 24 peering, 7 active+undersized, 1 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1/84 objects degraded (1.190%) 2024-09-13T14:23:55.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:55 smithi102 ceph-mon[110183]: pgmap v207: 73 pgs: 24 peering, 7 active+undersized, 1 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1/84 objects degraded (1.190%) 2024-09-13T14:23:55.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:55 smithi195 ceph-mon[104262]: pgmap v207: 73 pgs: 24 peering, 7 active+undersized, 1 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1/84 objects degraded (1.190%) 2024-09-13T14:23:56.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:56 smithi102 ceph-mon[110183]: pgmap v208: 73 pgs: 24 peering, 49 active+clean; 4.6 MiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:56.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:56.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:23:56.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:56 smithi102 ceph-mon[110183]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/84 objects degraded (1.190%), 1 pg degraded) 2024-09-13T14:23:56.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:56 smithi102 ceph-mon[110183]: Cluster is now healthy 2024-09-13T14:23:56.669 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:23:56 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:23:56] "GET /metrics HTTP/1.1" 200 34015 "" "Prometheus/2.51.0" 2024-09-13T14:23:56.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:56 smithi102 ceph-mon[115269]: pgmap v208: 73 pgs: 24 peering, 49 active+clean; 4.6 MiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:56.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:56.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:23:56.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:56 smithi102 ceph-mon[115269]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/84 objects degraded (1.190%), 1 pg degraded) 2024-09-13T14:23:56.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:56 smithi102 ceph-mon[115269]: Cluster is now healthy 2024-09-13T14:23:56.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:56 smithi195 ceph-mon[104262]: pgmap v208: 73 pgs: 24 peering, 49 active+clean; 4.6 MiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:56.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:23:56.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:23:56.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:56 smithi195 ceph-mon[104262]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/84 objects degraded (1.190%), 1 pg degraded) 2024-09-13T14:23:56.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:56 smithi195 ceph-mon[104262]: Cluster is now healthy 2024-09-13T14:23:58.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:23:58 smithi102 ceph-mon[110183]: pgmap v209: 73 pgs: 73 active+clean; 4.6 MiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:58.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:23:58 smithi102 ceph-mon[115269]: pgmap v209: 73 pgs: 73 active+clean; 4.6 MiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:23:58.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:23:58 smithi195 ceph-mon[104262]: pgmap v209: 73 pgs: 73 active+clean; 4.6 MiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:00.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:00 smithi102 ceph-mon[110183]: pgmap v210: 73 pgs: 73 active+clean; 4.6 MiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:00.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:00 smithi102 ceph-mon[115269]: pgmap v210: 73 pgs: 73 active+clean; 4.6 MiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:00.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:00 smithi195 ceph-mon[104262]: pgmap v210: 73 pgs: 73 active+clean; 4.6 MiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:02.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:02 smithi102 ceph-mon[110183]: pgmap v211: 73 pgs: 73 active+clean; 4.6 MiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:02.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:02 smithi102 ceph-mon[115269]: pgmap v211: 73 pgs: 73 active+clean; 4.6 MiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:02.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:02 smithi195 ceph-mon[104262]: pgmap v211: 73 pgs: 73 active+clean; 4.6 MiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:02.998 INFO:teuthology.orchestra.run.smithi102.stdout:true 2024-09-13T14:24:03.558 INFO:teuthology.orchestra.run.smithi102.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-13T14:24:03.558 INFO:teuthology.orchestra.run.smithi102.stdout:alertmanager.a smithi102 *:9093,9094 running (7m) 16s ago 13m 19.2M - 0.27.0 11f11916f8cd 0322b99ec767 2024-09-13T14:24:03.558 INFO:teuthology.orchestra.run.smithi102.stdout:grafana.a smithi195 *:3000 running (7m) 72s ago 13m 84.1M - c8b91775d855 152e89bad2c8 2024-09-13T14:24:03.558 INFO:teuthology.orchestra.run.smithi102.stdout:mds.foo.smithi102.njetxr smithi102 running (12m) 16s ago 12m 23.5M - 18.2.4-843-g2c01458a c129f5a2fd4e c9211cc725ed 2024-09-13T14:24:03.558 INFO:teuthology.orchestra.run.smithi102.stdout:mds.foo.smithi195.yqzrxi smithi195 running (12m) 72s ago 12m 19.1M - 18.2.4-843-g2c01458a c129f5a2fd4e eea37a7230fd 2024-09-13T14:24:03.558 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.x smithi195 *:8443,9283,8765 running (8m) 72s ago 17m 456M - 19.3.0-4697-gd58f7306 fec07016a2b0 4a23b8f0ea8b 2024-09-13T14:24:03.558 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.y smithi102 *:8443,9283,8765 running (9m) 16s ago 19m 518M - 19.3.0-4697-gd58f7306 fec07016a2b0 6b12b28dcde1 2024-09-13T14:24:03.559 INFO:teuthology.orchestra.run.smithi102.stdout:mon.a smithi102 running (6m) 16s ago 19m 53.1M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 f3234b7bbdde 2024-09-13T14:24:03.559 INFO:teuthology.orchestra.run.smithi102.stdout:mon.b smithi195 running (4m) 72s ago 17m 42.9M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 9921a2aa8168 2024-09-13T14:24:03.559 INFO:teuthology.orchestra.run.smithi102.stdout:mon.c smithi102 running (6m) 16s ago 17m 41.6M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 dba6b3d55296 2024-09-13T14:24:03.559 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.a smithi102 *:9100 running (8m) 16s ago 13m 12.1M - 1.7.0 72c9c2088986 0c92d0b39468 2024-09-13T14:24:03.559 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.b smithi195 *:9100 running (8m) 72s ago 13m 11.2M - 1.7.0 72c9c2088986 fde867a7dbf9 2024-09-13T14:24:03.559 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0 smithi102 running (53s) 16s ago 17m 41.0M 2480M 19.3.0-4697-gd58f7306 fec07016a2b0 d5c895f22b2a 2024-09-13T14:24:03.559 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1 smithi102 running (20s) 16s ago 16m 12.0M 2480M 19.3.0-4697-gd58f7306 fec07016a2b0 5d264b086bc0 2024-09-13T14:24:03.559 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2 smithi102 running (16m) 16s ago 16m 51.4M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 1800986ac838 2024-09-13T14:24:03.559 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3 smithi102 running (16m) 16s ago 16m 57.8M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 968246da9da2 2024-09-13T14:24:03.559 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4 smithi195 running (15m) 72s ago 15m 50.4M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 515307701d73 2024-09-13T14:24:03.559 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5 smithi195 running (15m) 72s ago 15m 54.1M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 176ba2b3bd59 2024-09-13T14:24:03.559 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6 smithi195 running (14m) 72s ago 14m 56.0M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 9cd070a6dd3d 2024-09-13T14:24:03.559 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7 smithi195 running (14m) 72s ago 14m 54.4M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e a6b698c659ca 2024-09-13T14:24:03.559 INFO:teuthology.orchestra.run.smithi102.stdout:prometheus.a smithi195 *:9095 running (8m) 72s ago 14m 48.2M - 2.51.0 1d3b7f56885b d131e53f1ef9 2024-09-13T14:24:03.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:03 smithi195 ceph-mon[104262]: from='client.44270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:03.892 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:24:03.892 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": { 2024-09-13T14:24:03.892 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 3 2024-09-13T14:24:03.893 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:24:03.893 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": { 2024-09-13T14:24:03.893 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 2 2024-09-13T14:24:03.893 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:24:03.893 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": { 2024-09-13T14:24:03.893 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 6, 2024-09-13T14:24:03.893 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 2 2024-09-13T14:24:03.893 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:24:03.893 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": { 2024-09-13T14:24:03.893 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 2 2024-09-13T14:24:03.893 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:24:03.893 INFO:teuthology.orchestra.run.smithi102.stdout: "overall": { 2024-09-13T14:24:03.893 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 8, 2024-09-13T14:24:03.894 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 7 2024-09-13T14:24:03.894 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:24:03.894 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:24:03.894 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:03 smithi102 ceph-mon[115269]: from='client.44270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:03.894 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:03 smithi102 ceph-mon[110183]: from='client.44270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:04.178 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:24:04.178 INFO:teuthology.orchestra.run.smithi102.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f", 2024-09-13T14:24:04.179 INFO:teuthology.orchestra.run.smithi102.stdout: "in_progress": true, 2024-09-13T14:24:04.179 INFO:teuthology.orchestra.run.smithi102.stdout: "which": "Upgrading daemons of type(s) osd. Upgrade limited to 4 daemons (2 remaining).", 2024-09-13T14:24:04.179 INFO:teuthology.orchestra.run.smithi102.stdout: "services_complete": [], 2024-09-13T14:24:04.179 INFO:teuthology.orchestra.run.smithi102.stdout: "progress": "2/8 daemons upgraded", 2024-09-13T14:24:04.179 INFO:teuthology.orchestra.run.smithi102.stdout: "message": "Currently upgrading osd daemons", 2024-09-13T14:24:04.179 INFO:teuthology.orchestra.run.smithi102.stdout: "is_paused": false 2024-09-13T14:24:04.179 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:24:04.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:04 smithi102 ceph-mon[110183]: pgmap v212: 73 pgs: 73 active+clean; 4.6 MiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:04.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:04 smithi102 ceph-mon[110183]: from='client.34353 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:04.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:04 smithi102 ceph-mon[110183]: from='client.34359 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:04.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:04 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1677991750' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:24:04.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:04 smithi102 ceph-mon[115269]: pgmap v212: 73 pgs: 73 active+clean; 4.6 MiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:04.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:04 smithi102 ceph-mon[115269]: from='client.34353 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:04.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:04 smithi102 ceph-mon[115269]: from='client.34359 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:04.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:04 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1677991750' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:24:04.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:04 smithi195 ceph-mon[104262]: pgmap v212: 73 pgs: 73 active+clean; 4.6 MiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:04.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:04 smithi195 ceph-mon[104262]: from='client.34353 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:04.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:04 smithi195 ceph-mon[104262]: from='client.34359 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:04.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:04 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1677991750' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:24:05.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:05 smithi195 ceph-mon[104262]: from='client.34371 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:05.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:05 smithi102 ceph-mon[110183]: from='client.34371 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:05.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:05 smithi102 ceph-mon[115269]: from='client.34371 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:06.627 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:06 smithi102 ceph-mon[115269]: pgmap v213: 73 pgs: 73 active+clean; 4.6 MiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:06.627 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:06 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:06.627 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:06 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:24:06.627 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:06 smithi102 ceph-mon[115269]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:24:06.627 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:06 smithi102 ceph-mon[115269]: Upgrade: osd.2 is safe to restart 2024-09-13T14:24:06.627 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:06 smithi102 ceph-mon[110183]: pgmap v213: 73 pgs: 73 active+clean; 4.6 MiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:06.627 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:06 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:06.627 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:06 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:24:06.628 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:06 smithi102 ceph-mon[110183]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:24:06.628 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:06 smithi102 ceph-mon[110183]: Upgrade: osd.2 is safe to restart 2024-09-13T14:24:06.628 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:24:06 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:24:06] "GET /metrics HTTP/1.1" 200 34015 "" "Prometheus/2.51.0" 2024-09-13T14:24:06.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:06 smithi195 ceph-mon[104262]: pgmap v213: 73 pgs: 73 active+clean; 4.6 MiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:06.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:06 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:06.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:06 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:24:06.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:06 smithi195 ceph-mon[104262]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:24:06.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:06 smithi195 ceph-mon[104262]: Upgrade: osd.2 is safe to restart 2024-09-13T14:24:08.596 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:08 smithi102 ceph-mon[110183]: pgmap v214: 73 pgs: 73 active+clean; 4.6 MiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:08.596 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:08 smithi102 ceph-mon[110183]: Upgrade: Updating osd.2 2024-09-13T14:24:08.596 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:08 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:08.596 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:08 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-13T14:24:08.596 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:08 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:24:08.597 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:08 smithi102 ceph-mon[110183]: Deploying daemon osd.2 on smithi102 2024-09-13T14:24:08.597 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:08 smithi102 ceph-mon[115269]: pgmap v214: 73 pgs: 73 active+clean; 4.6 MiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:08.597 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:08 smithi102 ceph-mon[115269]: Upgrade: Updating osd.2 2024-09-13T14:24:08.597 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:08 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:08.597 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:08 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-13T14:24:08.597 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:08 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:24:08.597 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:08 smithi102 ceph-mon[115269]: Deploying daemon osd.2 on smithi102 2024-09-13T14:24:08.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:08 smithi195 ceph-mon[104262]: pgmap v214: 73 pgs: 73 active+clean; 4.6 MiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:08.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:08 smithi195 ceph-mon[104262]: Upgrade: Updating osd.2 2024-09-13T14:24:08.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:08 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:08.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:08 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-13T14:24:08.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:08 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:24:08.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:08 smithi195 ceph-mon[104262]: Deploying daemon osd.2 on smithi102 2024-09-13T14:24:10.404 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:10 smithi102 systemd[1]: Stopping Ceph osd.2 for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T14:24:10.668 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:10 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2[56992]: 2024-09-13T14:24:10.402+0000 7f739d404640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T14:24:10.668 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:10 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2[56992]: 2024-09-13T14:24:10.402+0000 7f739d404640 -1 osd.2 100 *** Got signal Terminated *** 2024-09-13T14:24:10.668 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:10 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2[56992]: 2024-09-13T14:24:10.402+0000 7f739d404640 -1 osd.2 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T14:24:10.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:10 smithi195 ceph-mon[104262]: pgmap v215: 73 pgs: 73 active+clean; 4.6 MiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:10.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:10 smithi195 ceph-mon[104262]: osd.2 marked itself down and dead 2024-09-13T14:24:10.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:10.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:24:11.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:10 smithi102 ceph-mon[115269]: pgmap v215: 73 pgs: 73 active+clean; 4.6 MiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:11.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:10 smithi102 ceph-mon[115269]: osd.2 marked itself down and dead 2024-09-13T14:24:11.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:11.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:24:11.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:10 smithi102 ceph-mon[110183]: pgmap v215: 73 pgs: 73 active+clean; 4.6 MiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:11.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:10 smithi102 ceph-mon[110183]: osd.2 marked itself down and dead 2024-09-13T14:24:11.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:11.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:24:11.511 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:11 smithi102 podman[136898]: 2024-09-13 14:24:11.189460068 +0000 UTC m=+0.894483577 container died 1800986ac838b607df8c67b63e26cc6825b8595b43e005fdda10ca9151ef4ca7 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2, org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=reef-2c01458, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-09-13T14:24:11.828 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:11 smithi102 ceph-mon[110183]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:24:11.828 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:11 smithi102 ceph-mon[110183]: osdmap e101: 8 total, 7 up, 8 in 2024-09-13T14:24:11.829 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:11 smithi102 ceph-mon[110183]: osdmap e102: 8 total, 7 up, 8 in 2024-09-13T14:24:11.829 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:11 smithi102 ceph-mon[115269]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:24:11.829 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:11 smithi102 ceph-mon[115269]: osdmap e101: 8 total, 7 up, 8 in 2024-09-13T14:24:11.829 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:11 smithi102 ceph-mon[115269]: osdmap e102: 8 total, 7 up, 8 in 2024-09-13T14:24:11.829 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:11 smithi102 podman[136898]: 2024-09-13 14:24:11.60127579 +0000 UTC m=+1.306299292 container cleanup 1800986ac838b607df8c67b63e26cc6825b8595b43e005fdda10ca9151ef4ca7 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=reef-2c01458, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T14:24:11.829 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:11 smithi102 bash[136898]: ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2 2024-09-13T14:24:11.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:11 smithi195 ceph-mon[104262]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:24:11.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:11 smithi195 ceph-mon[104262]: osdmap e101: 8 total, 7 up, 8 in 2024-09-13T14:24:11.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:11 smithi195 ceph-mon[104262]: osdmap e102: 8 total, 7 up, 8 in 2024-09-13T14:24:12.168 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:11 smithi102 podman[136920]: 2024-09-13 14:24:11.827943891 +0000 UTC m=+0.641526444 container remove 1800986ac838b607df8c67b63e26cc6825b8595b43e005fdda10ca9151ef4ca7 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=reef-2c01458, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-13T14:24:12.590 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:12 smithi102 podman[137014]: 2024-09-13 14:24:12.158911874 +0000 UTC m=+0.016357650 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:24:12.590 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:12 smithi102 podman[137014]: 2024-09-13 14:24:12.279021757 +0000 UTC m=+0.136467520 container create 966cce4ba4dd08fd0140405b6c37d0f4f0e2bbe9474e073b53f90d6588d1c90c (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2-deactivate, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-aclamk-cbt-combined-d58f730, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731) 2024-09-13T14:24:12.590 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:12 smithi102 podman[137014]: 2024-09-13 14:24:12.498679746 +0000 UTC m=+0.356125514 container init 966cce4ba4dd08fd0140405b6c37d0f4f0e2bbe9474e073b53f90d6588d1c90c (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2-deactivate, RELEASE=wip-aclamk-cbt-combined-d58f730, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-13T14:24:12.590 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:12 smithi102 podman[137014]: 2024-09-13 14:24:12.503981673 +0000 UTC m=+0.361427438 container start 966cce4ba4dd08fd0140405b6c37d0f4f0e2bbe9474e073b53f90d6588d1c90c (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=wip-aclamk-cbt-combined-d58f730, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T14:24:12.590 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:12 smithi102 podman[137014]: 2024-09-13 14:24:12.58980472 +0000 UTC m=+0.447250492 container attach 966cce4ba4dd08fd0140405b6c37d0f4f0e2bbe9474e073b53f90d6588d1c90c (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-13T14:24:12.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:12 smithi102 ceph-mon[110183]: pgmap v217: 73 pgs: 15 peering, 1 stale+active+clean, 57 active+clean; 4.6 MiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:12.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:12 smithi102 ceph-mon[110183]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-13T14:24:12.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:12 smithi102 ceph-mon[115269]: pgmap v217: 73 pgs: 15 peering, 1 stale+active+clean, 57 active+clean; 4.6 MiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:12.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:12 smithi102 ceph-mon[115269]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-13T14:24:12.919 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:12 smithi102 podman[137014]: 2024-09-13 14:24:12.697289067 +0000 UTC m=+0.554734845 container died 966cce4ba4dd08fd0140405b6c37d0f4f0e2bbe9474e073b53f90d6588d1c90c (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2-deactivate, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-09-13T14:24:12.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:12 smithi195 ceph-mon[104262]: pgmap v217: 73 pgs: 15 peering, 1 stale+active+clean, 57 active+clean; 4.6 MiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:12.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:12 smithi195 ceph-mon[104262]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-13T14:24:13.669 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:13 smithi102 podman[137057]: 2024-09-13 14:24:13.260179899 +0000 UTC m=+0.556762923 container remove 966cce4ba4dd08fd0140405b6c37d0f4f0e2bbe9474e073b53f90d6588d1c90c (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=wip-aclamk-cbt-combined-d58f730, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-09-13T14:24:13.669 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:13 smithi102 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@osd.2.service: Deactivated successfully. 2024-09-13T14:24:13.669 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:13 smithi102 systemd[1]: Stopped Ceph osd.2 for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:24:13.669 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:13 smithi102 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@osd.2.service: Consumed 6.128s CPU time. 2024-09-13T14:24:13.669 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:13 smithi102 systemd[1]: Starting Ceph osd.2 for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T14:24:13.975 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:13 smithi102 podman[137161]: 2024-09-13 14:24:13.609242567 +0000 UTC m=+0.015932285 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:24:13.976 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:13 smithi102 podman[137161]: 2024-09-13 14:24:13.719909051 +0000 UTC m=+0.126598761 container create 59239d2bdce966f10632da044f1996bcc9aa29d1b6141ef78ca8e8bed3103146 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2-activate, org.label-schema.schema-version=1.0, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T14:24:14.418 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:13 smithi102 podman[137161]: 2024-09-13 14:24:13.975563468 +0000 UTC m=+0.382253172 container init 59239d2bdce966f10632da044f1996bcc9aa29d1b6141ef78ca8e8bed3103146 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2-activate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-13T14:24:14.419 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:13 smithi102 podman[137161]: 2024-09-13 14:24:13.98044985 +0000 UTC m=+0.387139555 container start 59239d2bdce966f10632da044f1996bcc9aa29d1b6141ef78ca8e8bed3103146 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=) 2024-09-13T14:24:14.419 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:14 smithi102 podman[137161]: 2024-09-13 14:24:14.038924605 +0000 UTC m=+0.445614326 container attach 59239d2bdce966f10632da044f1996bcc9aa29d1b6141ef78ca8e8bed3103146 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2-activate, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-13T14:24:14.419 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:14 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2-activate[137177]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:24:14.419 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:14 smithi102 bash[137161]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:24:14.419 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:14 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2-activate[137177]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:24:14.419 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:14 smithi102 bash[137161]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:24:14.419 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:14 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2-activate[137177]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-09-13T14:24:14.419 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:14 smithi102 bash[137161]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-09-13T14:24:14.419 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:14 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2-activate[137177]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:24:14.419 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:14 smithi102 bash[137161]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:24:14.419 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:14 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2-activate[137177]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:24:14.420 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:14 smithi102 bash[137161]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:24:14.420 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:14 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2-activate[137177]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-13T14:24:14.420 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:14 smithi102 bash[137161]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-13T14:24:14.420 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:14 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2-activate[137177]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_2 --path /var/lib/ceph/osd/ceph-2 --no-mon-config 2024-09-13T14:24:14.420 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:14 smithi102 bash[137161]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_2 --path /var/lib/ceph/osd/ceph-2 --no-mon-config 2024-09-13T14:24:14.835 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:14 smithi102 ceph-mon[110183]: pgmap v219: 73 pgs: 15 peering, 1 stale+active+clean, 57 active+clean; 4.6 MiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:14.835 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:14 smithi102 ceph-mon[115269]: pgmap v219: 73 pgs: 15 peering, 1 stale+active+clean, 57 active+clean; 4.6 MiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:14.836 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:14 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2-activate[137177]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-2/block 2024-09-13T14:24:14.836 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:14 smithi102 bash[137161]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-2/block 2024-09-13T14:24:14.836 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:14 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2-activate[137177]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block 2024-09-13T14:24:14.836 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:14 smithi102 bash[137161]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block 2024-09-13T14:24:14.836 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:14 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2-activate[137177]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-13T14:24:14.836 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:14 smithi102 bash[137161]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-13T14:24:14.836 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:14 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2-activate[137177]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-13T14:24:14.836 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:14 smithi102 bash[137161]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-13T14:24:14.836 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:14 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2-activate[137177]: --> ceph-volume lvm activate successful for osd ID: 2 2024-09-13T14:24:14.837 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:14 smithi102 bash[137161]: --> ceph-volume lvm activate successful for osd ID: 2 2024-09-13T14:24:14.837 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:14 smithi102 podman[137161]: 2024-09-13 14:24:14.525666099 +0000 UTC m=+0.932355821 container died 59239d2bdce966f10632da044f1996bcc9aa29d1b6141ef78ca8e8bed3103146 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2) 2024-09-13T14:24:14.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:14 smithi195 ceph-mon[104262]: pgmap v219: 73 pgs: 15 peering, 1 stale+active+clean, 57 active+clean; 4.6 MiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:15.418 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:15 smithi102 podman[137303]: 2024-09-13 14:24:15.117841233 +0000 UTC m=+0.585857564 container remove 59239d2bdce966f10632da044f1996bcc9aa29d1b6141ef78ca8e8bed3103146 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2-activate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-aclamk-cbt-combined-d58f730, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731) 2024-09-13T14:24:15.418 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:15 smithi102 podman[137369]: 2024-09-13 14:24:15.227895604 +0000 UTC m=+0.015926411 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:24:15.418 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:15 smithi102 podman[137369]: 2024-09-13 14:24:15.335480773 +0000 UTC m=+0.123511567 container create fba29968c6b9ca62e1714f20028939d268fd8f4d5be7ada86b12d86fec5c73d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-13T14:24:15.918 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:15 smithi102 podman[137369]: 2024-09-13 14:24:15.56332025 +0000 UTC m=+0.351351045 container init fba29968c6b9ca62e1714f20028939d268fd8f4d5be7ada86b12d86fec5c73d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-09-13T14:24:15.918 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:15 smithi102 podman[137369]: 2024-09-13 14:24:15.568467472 +0000 UTC m=+0.356498280 container start fba29968c6b9ca62e1714f20028939d268fd8f4d5be7ada86b12d86fec5c73d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-aclamk-cbt-combined-d58f730) 2024-09-13T14:24:15.918 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:15 smithi102 ceph-osd[137407]: -- 172.21.15.102:0/274875893 <== mon.0 v2:172.21.15.102:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55db21c8da40 con 0x55db22a5a400 2024-09-13T14:24:15.918 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:15 smithi102 bash[137369]: fba29968c6b9ca62e1714f20028939d268fd8f4d5be7ada86b12d86fec5c73d9 2024-09-13T14:24:15.918 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:15 smithi102 systemd[1]: Started Ceph osd.2 for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:24:16.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:24:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:24:16] "GET /metrics HTTP/1.1" 200 34019 "" "Prometheus/2.51.0" 2024-09-13T14:24:16.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:16 smithi195 ceph-mon[104262]: pgmap v220: 73 pgs: 4 active+undersized, 15 peering, 54 active+clean; 4.6 MiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:16.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:16 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:16.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:16 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:16.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:16 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:16.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:16 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:24:17.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:16 smithi102 ceph-mon[110183]: pgmap v220: 73 pgs: 4 active+undersized, 15 peering, 54 active+clean; 4.6 MiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:17.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:16 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:17.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:16 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:17.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:16 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:17.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:16 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:24:17.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:16 smithi102 ceph-mon[115269]: pgmap v220: 73 pgs: 4 active+undersized, 15 peering, 54 active+clean; 4.6 MiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:17.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:16 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:17.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:16 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:17.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:16 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:17.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:16 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:24:17.487 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:17 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2[137384]: 2024-09-13T14:24:17.210+0000 7fbcfb571640 -1 Falling back to public interface 2024-09-13T14:24:18.047 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:17 smithi195 ceph-mon[104262]: Health check failed: Degraded data redundancy: 5/84 objects degraded (5.952%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:24:18.047 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:17 smithi195 ceph-mon[104262]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-13T14:24:18.090 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:17 smithi102 ceph-mon[110183]: Health check failed: Degraded data redundancy: 5/84 objects degraded (5.952%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:24:18.090 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:17 smithi102 ceph-mon[110183]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-13T14:24:18.090 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:17 smithi102 ceph-mon[115269]: Health check failed: Degraded data redundancy: 5/84 objects degraded (5.952%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:24:18.090 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:17 smithi102 ceph-mon[115269]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-13T14:24:19.036 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:18 smithi102 ceph-mon[110183]: pgmap v221: 73 pgs: 16 active+undersized, 3 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 254 MiB used, 715 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-13T14:24:19.036 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:18 smithi102 ceph-mon[115269]: pgmap v221: 73 pgs: 16 active+undersized, 3 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 254 MiB used, 715 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-13T14:24:19.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:18 smithi195 ceph-mon[104262]: pgmap v221: 73 pgs: 16 active+undersized, 3 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 254 MiB used, 715 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-13T14:24:20.599 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:20 smithi102 ceph-mon[115269]: pgmap v222: 73 pgs: 16 active+undersized, 3 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 254 MiB used, 715 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-13T14:24:20.600 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:20 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:20.600 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:20 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:20.600 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:20 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:20.600 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:20 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:20.600 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:20 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:20.600 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:20 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:20.600 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:20 smithi102 ceph-mon[110183]: pgmap v222: 73 pgs: 16 active+undersized, 3 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 254 MiB used, 715 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-13T14:24:20.600 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:20 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:20.600 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:20 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:20.600 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:20 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:20.601 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:20 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:20.601 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:20 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:20.601 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:20 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:20.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:20 smithi195 ceph-mon[104262]: pgmap v222: 73 pgs: 16 active+undersized, 3 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 254 MiB used, 715 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-13T14:24:20.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:20 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:20.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:20 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:20.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:20 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:20.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:20 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:20.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:20 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:20.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:20 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:20.918 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:20 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2[137384]: 2024-09-13T14:24:20.773+0000 7fbcfb571640 -1 osd.2 100 log_to_monitors true 2024-09-13T14:24:21.604 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:21 smithi102 ceph-mon[110183]: from='osd.2 [v2:172.21.15.102:6818/4066184234,v1:172.21.15.102:6819/4066184234]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-13T14:24:21.604 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:21 smithi102 ceph-mon[110183]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-13T14:24:21.604 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:21 smithi102 ceph-mon[115269]: from='osd.2 [v2:172.21.15.102:6818/4066184234,v1:172.21.15.102:6819/4066184234]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-13T14:24:21.604 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:21 smithi102 ceph-mon[115269]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-13T14:24:21.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:21 smithi195 ceph-mon[104262]: from='osd.2 [v2:172.21.15.102:6818/4066184234,v1:172.21.15.102:6819/4066184234]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-13T14:24:21.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:21 smithi195 ceph-mon[104262]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-13T14:24:21.918 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 14:24:21 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2[137384]: 2024-09-13T14:24:21.766+0000 7fbcef313640 -1 osd.2 100 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T14:24:22.612 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:22 smithi102 ceph-mon[110183]: pgmap v223: 73 pgs: 16 active+undersized, 3 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 254 MiB used, 715 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-13T14:24:22.612 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:22 smithi102 ceph-mon[110183]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-13T14:24:22.612 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:22 smithi102 ceph-mon[110183]: from='osd.2 [v2:172.21.15.102:6818/4066184234,v1:172.21.15.102:6819/4066184234]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]} : dispatch 2024-09-13T14:24:22.612 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:22 smithi102 ceph-mon[110183]: osdmap e103: 8 total, 7 up, 8 in 2024-09-13T14:24:22.612 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:22 smithi102 ceph-mon[110183]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]} : dispatch 2024-09-13T14:24:22.612 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:22 smithi102 ceph-mon[110183]: from='osd.2 ' entity='osd.2' 2024-09-13T14:24:22.613 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:22 smithi102 ceph-mon[115269]: pgmap v223: 73 pgs: 16 active+undersized, 3 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 254 MiB used, 715 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-13T14:24:22.613 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:22 smithi102 ceph-mon[115269]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-13T14:24:22.613 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:22 smithi102 ceph-mon[115269]: from='osd.2 [v2:172.21.15.102:6818/4066184234,v1:172.21.15.102:6819/4066184234]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]} : dispatch 2024-09-13T14:24:22.613 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:22 smithi102 ceph-mon[115269]: osdmap e103: 8 total, 7 up, 8 in 2024-09-13T14:24:22.613 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:22 smithi102 ceph-mon[115269]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]} : dispatch 2024-09-13T14:24:22.613 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:22 smithi102 ceph-mon[115269]: from='osd.2 ' entity='osd.2' 2024-09-13T14:24:22.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:22 smithi195 ceph-mon[104262]: pgmap v223: 73 pgs: 16 active+undersized, 3 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 254 MiB used, 715 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-13T14:24:22.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:22 smithi195 ceph-mon[104262]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-13T14:24:22.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:22 smithi195 ceph-mon[104262]: from='osd.2 [v2:172.21.15.102:6818/4066184234,v1:172.21.15.102:6819/4066184234]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]} : dispatch 2024-09-13T14:24:22.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:22 smithi195 ceph-mon[104262]: osdmap e103: 8 total, 7 up, 8 in 2024-09-13T14:24:22.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:22 smithi195 ceph-mon[104262]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]} : dispatch 2024-09-13T14:24:22.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:22 smithi195 ceph-mon[104262]: from='osd.2 ' entity='osd.2' 2024-09-13T14:24:23.329 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:24:23 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: 2024-09-13T14:24:23.178+0000 7f0c92e7e640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-09-13T14:24:23.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:23 smithi102 ceph-mon[110183]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:24:23.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:23 smithi102 ceph-mon[110183]: osd.2 [v2:172.21.15.102:6818/4066184234,v1:172.21.15.102:6819/4066184234] boot 2024-09-13T14:24:23.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:23 smithi102 ceph-mon[110183]: osdmap e104: 8 total, 8 up, 8 in 2024-09-13T14:24:23.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:23 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-13T14:24:23.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:23 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:23.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:23 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:23.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:23 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:24:23.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:23 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:24:23.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:23 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:23.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:23 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:24:23.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:23 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:24:23.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:23 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:24:23.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:23 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:24:23.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:23 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-13T14:24:23.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:23 smithi102 ceph-mon[115269]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:24:23.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:23 smithi102 ceph-mon[115269]: osd.2 [v2:172.21.15.102:6818/4066184234,v1:172.21.15.102:6819/4066184234] boot 2024-09-13T14:24:23.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:23 smithi102 ceph-mon[115269]: osdmap e104: 8 total, 8 up, 8 in 2024-09-13T14:24:23.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:23 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-13T14:24:23.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:23 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:23.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:23 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:23.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:23 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:24:23.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:23 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:24:23.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:23 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:23.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:23 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:24:23.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:23 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:24:23.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:23 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:24:23.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:23 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:24:23.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:23 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-13T14:24:23.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:23 smithi195 ceph-mon[104262]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:24:23.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:23 smithi195 ceph-mon[104262]: osd.2 [v2:172.21.15.102:6818/4066184234,v1:172.21.15.102:6819/4066184234] boot 2024-09-13T14:24:23.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:23 smithi195 ceph-mon[104262]: osdmap e104: 8 total, 8 up, 8 in 2024-09-13T14:24:23.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:23 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-13T14:24:23.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:23 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:23.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:23 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:23.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:23 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:24:23.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:23 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:24:23.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:23 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:23.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:23 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:24:23.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:23 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:24:23.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:23 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:24:23.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:23 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:24:23.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:23 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-13T14:24:25.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:24 smithi102 ceph-mon[115269]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-13T14:24:25.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:24 smithi102 ceph-mon[115269]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-09-13T14:24:25.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:24 smithi102 ceph-mon[115269]: pgmap v226: 73 pgs: 16 active+undersized, 3 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 251 MiB used, 715 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-13T14:24:25.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:24 smithi102 ceph-mon[115269]: osdmap e105: 8 total, 8 up, 8 in 2024-09-13T14:24:25.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:24 smithi102 ceph-mon[110183]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-13T14:24:25.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:24 smithi102 ceph-mon[110183]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-09-13T14:24:25.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:24 smithi102 ceph-mon[110183]: pgmap v226: 73 pgs: 16 active+undersized, 3 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 251 MiB used, 715 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-13T14:24:25.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:24 smithi102 ceph-mon[110183]: osdmap e105: 8 total, 8 up, 8 in 2024-09-13T14:24:25.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:24 smithi195 ceph-mon[104262]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-13T14:24:25.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:24 smithi195 ceph-mon[104262]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-09-13T14:24:25.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:24 smithi195 ceph-mon[104262]: pgmap v226: 73 pgs: 16 active+undersized, 3 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 251 MiB used, 715 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-13T14:24:25.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:24 smithi195 ceph-mon[104262]: osdmap e105: 8 total, 8 up, 8 in 2024-09-13T14:24:26.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:26 smithi102 ceph-mon[110183]: pgmap v228: 73 pgs: 13 active+undersized, 3 active+undersized+degraded, 57 active+clean; 4.6 MiB data, 251 MiB used, 715 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-13T14:24:26.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:26.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:24:26.669 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:24:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:24:26] "GET /metrics HTTP/1.1" 200 33868 "" "Prometheus/2.51.0" 2024-09-13T14:24:26.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:26 smithi102 ceph-mon[115269]: pgmap v228: 73 pgs: 13 active+undersized, 3 active+undersized+degraded, 57 active+clean; 4.6 MiB data, 251 MiB used, 715 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-13T14:24:26.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:26.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:24:26.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:26 smithi195 ceph-mon[104262]: pgmap v228: 73 pgs: 13 active+undersized, 3 active+undersized+degraded, 57 active+clean; 4.6 MiB data, 251 MiB used, 715 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-13T14:24:26.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:26.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:24:27.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:27 smithi195 ceph-mon[104262]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 5/84 objects degraded (5.952%), 3 pgs degraded) 2024-09-13T14:24:27.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:27 smithi195 ceph-mon[104262]: Cluster is now healthy 2024-09-13T14:24:27.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:27 smithi102 ceph-mon[110183]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 5/84 objects degraded (5.952%), 3 pgs degraded) 2024-09-13T14:24:27.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:27 smithi102 ceph-mon[110183]: Cluster is now healthy 2024-09-13T14:24:27.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:27 smithi102 ceph-mon[115269]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 5/84 objects degraded (5.952%), 3 pgs degraded) 2024-09-13T14:24:27.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:27 smithi102 ceph-mon[115269]: Cluster is now healthy 2024-09-13T14:24:28.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:28 smithi195 ceph-mon[104262]: pgmap v229: 73 pgs: 73 active+clean; 4.6 MiB data, 251 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:28.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:28 smithi102 ceph-mon[110183]: pgmap v229: 73 pgs: 73 active+clean; 4.6 MiB data, 251 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:28.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:28 smithi102 ceph-mon[115269]: pgmap v229: 73 pgs: 73 active+clean; 4.6 MiB data, 251 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:30.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:30 smithi195 ceph-mon[104262]: pgmap v230: 73 pgs: 73 active+clean; 4.6 MiB data, 251 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:30.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:30 smithi102 ceph-mon[110183]: pgmap v230: 73 pgs: 73 active+clean; 4.6 MiB data, 251 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:30.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:30 smithi102 ceph-mon[115269]: pgmap v230: 73 pgs: 73 active+clean; 4.6 MiB data, 251 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:31.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:31 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:31.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:31 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:31.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:31 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:32.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:32 smithi195 ceph-mon[104262]: pgmap v231: 73 pgs: 73 active+clean; 4.6 MiB data, 251 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:32.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:32 smithi102 ceph-mon[110183]: pgmap v231: 73 pgs: 73 active+clean; 4.6 MiB data, 251 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:32.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:32 smithi102 ceph-mon[115269]: pgmap v231: 73 pgs: 73 active+clean; 4.6 MiB data, 251 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:34.490 INFO:teuthology.orchestra.run.smithi102.stdout:true 2024-09-13T14:24:34.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:34 smithi195 ceph-mon[104262]: pgmap v232: 73 pgs: 73 active+clean; 4.6 MiB data, 251 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:34.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:34 smithi102 ceph-mon[110183]: pgmap v232: 73 pgs: 73 active+clean; 4.6 MiB data, 251 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:34.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:34 smithi102 ceph-mon[115269]: pgmap v232: 73 pgs: 73 active+clean; 4.6 MiB data, 251 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:35.044 INFO:teuthology.orchestra.run.smithi102.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-13T14:24:35.044 INFO:teuthology.orchestra.run.smithi102.stdout:alertmanager.a smithi102 *:9093,9094 running (8m) 15s ago 14m 19.2M - 0.27.0 11f11916f8cd 0322b99ec767 2024-09-13T14:24:35.045 INFO:teuthology.orchestra.run.smithi102.stdout:grafana.a smithi195 *:3000 running (8m) 15s ago 13m 84.3M - c8b91775d855 152e89bad2c8 2024-09-13T14:24:35.045 INFO:teuthology.orchestra.run.smithi102.stdout:mds.foo.smithi102.njetxr smithi102 running (12m) 15s ago 12m 23.7M - 18.2.4-843-g2c01458a c129f5a2fd4e c9211cc725ed 2024-09-13T14:24:35.045 INFO:teuthology.orchestra.run.smithi102.stdout:mds.foo.smithi195.yqzrxi smithi195 running (12m) 15s ago 12m 19.5M - 18.2.4-843-g2c01458a c129f5a2fd4e eea37a7230fd 2024-09-13T14:24:35.045 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.x smithi195 *:8443,9283,8765 running (9m) 15s ago 18m 456M - 19.3.0-4697-gd58f7306 fec07016a2b0 4a23b8f0ea8b 2024-09-13T14:24:35.045 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.y smithi102 *:8443,9283,8765 running (9m) 15s ago 19m 518M - 19.3.0-4697-gd58f7306 fec07016a2b0 6b12b28dcde1 2024-09-13T14:24:35.046 INFO:teuthology.orchestra.run.smithi102.stdout:mon.a smithi102 running (7m) 15s ago 19m 56.9M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 f3234b7bbdde 2024-09-13T14:24:35.046 INFO:teuthology.orchestra.run.smithi102.stdout:mon.b smithi195 running (4m) 15s ago 18m 45.0M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 9921a2aa8168 2024-09-13T14:24:35.046 INFO:teuthology.orchestra.run.smithi102.stdout:mon.c smithi102 running (6m) 15s ago 18m 45.0M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 dba6b3d55296 2024-09-13T14:24:35.046 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.a smithi102 *:9100 running (9m) 15s ago 13m 12.4M - 1.7.0 72c9c2088986 0c92d0b39468 2024-09-13T14:24:35.046 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.b smithi195 *:9100 running (9m) 15s ago 13m 10.8M - 1.7.0 72c9c2088986 fde867a7dbf9 2024-09-13T14:24:35.046 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0 smithi102 running (84s) 15s ago 17m 41.9M 2480M 19.3.0-4697-gd58f7306 fec07016a2b0 d5c895f22b2a 2024-09-13T14:24:35.047 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1 smithi102 running (51s) 15s ago 17m 48.4M 2480M 19.3.0-4697-gd58f7306 fec07016a2b0 5d264b086bc0 2024-09-13T14:24:35.047 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2 smithi102 running (19s) 15s ago 17m 11.7M 2480M 19.3.0-4697-gd58f7306 fec07016a2b0 fba29968c6b9 2024-09-13T14:24:35.047 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3 smithi102 running (16m) 15s ago 16m 58.9M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 968246da9da2 2024-09-13T14:24:35.047 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4 smithi195 running (16m) 15s ago 16m 51.5M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 515307701d73 2024-09-13T14:24:35.047 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5 smithi195 running (15m) 15s ago 15m 55.6M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 176ba2b3bd59 2024-09-13T14:24:35.048 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6 smithi195 running (15m) 15s ago 15m 58.3M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 9cd070a6dd3d 2024-09-13T14:24:35.048 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7 smithi195 running (15m) 15s ago 15m 54.8M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e a6b698c659ca 2024-09-13T14:24:35.048 INFO:teuthology.orchestra.run.smithi102.stdout:prometheus.a smithi195 *:9095 running (8m) 15s ago 14m 48.2M - 2.51.0 1d3b7f56885b d131e53f1ef9 2024-09-13T14:24:35.377 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:24:35.377 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": { 2024-09-13T14:24:35.377 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 3 2024-09-13T14:24:35.377 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:24:35.377 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": { 2024-09-13T14:24:35.377 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 2 2024-09-13T14:24:35.377 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:24:35.377 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": { 2024-09-13T14:24:35.377 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 5, 2024-09-13T14:24:35.377 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 3 2024-09-13T14:24:35.378 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:24:35.378 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": { 2024-09-13T14:24:35.378 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 2 2024-09-13T14:24:35.378 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:24:35.378 INFO:teuthology.orchestra.run.smithi102.stdout: "overall": { 2024-09-13T14:24:35.378 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 7, 2024-09-13T14:24:35.378 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 8 2024-09-13T14:24:35.378 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:24:35.378 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:24:35.666 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:24:35.667 INFO:teuthology.orchestra.run.smithi102.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f", 2024-09-13T14:24:35.667 INFO:teuthology.orchestra.run.smithi102.stdout: "in_progress": true, 2024-09-13T14:24:35.667 INFO:teuthology.orchestra.run.smithi102.stdout: "which": "Upgrading daemons of type(s) osd. Upgrade limited to 4 daemons (1 remaining).", 2024-09-13T14:24:35.667 INFO:teuthology.orchestra.run.smithi102.stdout: "services_complete": [], 2024-09-13T14:24:35.667 INFO:teuthology.orchestra.run.smithi102.stdout: "progress": "3/8 daemons upgraded", 2024-09-13T14:24:35.667 INFO:teuthology.orchestra.run.smithi102.stdout: "message": "Currently upgrading osd daemons", 2024-09-13T14:24:35.667 INFO:teuthology.orchestra.run.smithi102.stdout: "is_paused": false 2024-09-13T14:24:35.667 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:24:35.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:35 smithi102 ceph-mon[110183]: from='client.44306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:35.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:35 smithi102 ceph-mon[110183]: from='client.34389 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:35.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:35 smithi102 ceph-mon[110183]: from='client.34395 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:35.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:35 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3320988765' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:24:35.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:35 smithi102 ceph-mon[115269]: from='client.44306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:35.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:35 smithi102 ceph-mon[115269]: from='client.34389 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:35.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:35 smithi102 ceph-mon[115269]: from='client.34395 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:35.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:35 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3320988765' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:24:35.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:35 smithi195 ceph-mon[104262]: from='client.44306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:35.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:35 smithi195 ceph-mon[104262]: from='client.34389 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:35.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:35 smithi195 ceph-mon[104262]: from='client.34395 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:35.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:35 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3320988765' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:24:36.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:36 smithi102 ceph-mon[115269]: pgmap v233: 73 pgs: 73 active+clean; 4.6 MiB data, 251 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:36.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:36 smithi102 ceph-mon[115269]: from='client.44330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:36.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:36 smithi102 ceph-mon[110183]: pgmap v233: 73 pgs: 73 active+clean; 4.6 MiB data, 251 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:36.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:36 smithi102 ceph-mon[110183]: from='client.44330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:36.919 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:24:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:24:36] "GET /metrics HTTP/1.1" 200 33868 "" "Prometheus/2.51.0" 2024-09-13T14:24:36.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:36 smithi195 ceph-mon[104262]: pgmap v233: 73 pgs: 73 active+clean; 4.6 MiB data, 251 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:36.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:36 smithi195 ceph-mon[104262]: from='client.44330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:38.832 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:38 smithi102 ceph-mon[110183]: pgmap v234: 73 pgs: 73 active+clean; 4.6 MiB data, 251 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:38.832 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:38 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-13T14:24:38.833 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:38 smithi102 ceph-mon[115269]: pgmap v234: 73 pgs: 73 active+clean; 4.6 MiB data, 251 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:38.833 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:38 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-13T14:24:38.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:38 smithi195 ceph-mon[104262]: pgmap v234: 73 pgs: 73 active+clean; 4.6 MiB data, 251 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:38.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:38 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-13T14:24:39.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:39 smithi102 ceph-mon[110183]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-13T14:24:39.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:39 smithi102 ceph-mon[110183]: Upgrade: osd.3 is safe to restart 2024-09-13T14:24:39.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:39 smithi102 ceph-mon[115269]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-13T14:24:39.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:39 smithi102 ceph-mon[115269]: Upgrade: osd.3 is safe to restart 2024-09-13T14:24:39.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:39 smithi195 ceph-mon[104262]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-13T14:24:39.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:39 smithi195 ceph-mon[104262]: Upgrade: osd.3 is safe to restart 2024-09-13T14:24:40.869 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:40 smithi102 ceph-mon[110183]: pgmap v235: 73 pgs: 73 active+clean; 4.6 MiB data, 251 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:40.869 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:40 smithi102 ceph-mon[110183]: Upgrade: Updating osd.3 2024-09-13T14:24:40.869 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:40 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:40.869 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:40 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-13T14:24:40.869 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:40 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:24:40.869 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:40 smithi102 ceph-mon[110183]: Deploying daemon osd.3 on smithi102 2024-09-13T14:24:40.869 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:40 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:40.870 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:40 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:24:40.870 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:40 smithi102 ceph-mon[115269]: pgmap v235: 73 pgs: 73 active+clean; 4.6 MiB data, 251 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:40.870 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:40 smithi102 ceph-mon[115269]: Upgrade: Updating osd.3 2024-09-13T14:24:40.870 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:40 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:40.870 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:40 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-13T14:24:40.870 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:40 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:24:40.870 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:40 smithi102 ceph-mon[115269]: Deploying daemon osd.3 on smithi102 2024-09-13T14:24:40.870 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:40 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:40.870 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:40 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:24:40.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:40 smithi195 ceph-mon[104262]: pgmap v235: 73 pgs: 73 active+clean; 4.6 MiB data, 251 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:40.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:40 smithi195 ceph-mon[104262]: Upgrade: Updating osd.3 2024-09-13T14:24:40.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:40 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:40.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:40 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-13T14:24:40.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:40 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:24:40.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:40 smithi195 ceph-mon[104262]: Deploying daemon osd.3 on smithi102 2024-09-13T14:24:40.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:40 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:40.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:40 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:24:42.592 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:42 smithi102 systemd[1]: Stopping Ceph osd.3 for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T14:24:42.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:42 smithi102 ceph-mon[110183]: pgmap v236: 73 pgs: 73 active+clean; 4.6 MiB data, 251 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:42.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:42 smithi102 ceph-mon[110183]: osd.3 marked itself down and dead 2024-09-13T14:24:42.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:42 smithi102 ceph-mon[115269]: pgmap v236: 73 pgs: 73 active+clean; 4.6 MiB data, 251 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:42.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:42 smithi102 ceph-mon[115269]: osd.3 marked itself down and dead 2024-09-13T14:24:42.919 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:42 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3[63044]: 2024-09-13T14:24:42.590+0000 7fcf05a1d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T14:24:42.919 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:42 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3[63044]: 2024-09-13T14:24:42.590+0000 7fcf05a1d640 -1 osd.3 105 *** Got signal Terminated *** 2024-09-13T14:24:42.919 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:42 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3[63044]: 2024-09-13T14:24:42.590+0000 7fcf05a1d640 -1 osd.3 105 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T14:24:42.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:42 smithi195 ceph-mon[104262]: pgmap v236: 73 pgs: 73 active+clean; 4.6 MiB data, 251 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:42.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:42 smithi195 ceph-mon[104262]: osd.3 marked itself down and dead 2024-09-13T14:24:43.418 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:43 smithi102 podman[141713]: 2024-09-13 14:24:43.081311901 +0000 UTC m=+0.603773205 container died 968246da9da23567a4a1435f8bfe70b0420e8ccc25bd4fe91b2a121cb3716cab (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=reef-2c01458, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-13T14:24:43.699 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:43 smithi102 ceph-mon[110183]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:24:43.700 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:43 smithi102 ceph-mon[110183]: osdmap e106: 8 total, 7 up, 8 in 2024-09-13T14:24:43.700 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:43 smithi102 ceph-mon[115269]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:24:43.700 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:43 smithi102 ceph-mon[115269]: osdmap e106: 8 total, 7 up, 8 in 2024-09-13T14:24:43.700 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:43 smithi102 podman[141713]: 2024-09-13 14:24:43.480972346 +0000 UTC m=+1.003433642 container cleanup 968246da9da23567a4a1435f8bfe70b0420e8ccc25bd4fe91b2a121cb3716cab (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-2c01458, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-09-13T14:24:43.700 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:43 smithi102 bash[141713]: ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3 2024-09-13T14:24:43.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:43 smithi195 ceph-mon[104262]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:24:43.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:43 smithi195 ceph-mon[104262]: osdmap e106: 8 total, 7 up, 8 in 2024-09-13T14:24:44.108 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:43 smithi102 podman[141733]: 2024-09-13 14:24:43.699217162 +0000 UTC m=+0.620275692 container remove 968246da9da23567a4a1435f8bfe70b0420e8ccc25bd4fe91b2a121cb3716cab (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, RELEASE=reef-2c01458, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-13T14:24:44.395 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:44 smithi102 podman[141826]: 2024-09-13 14:24:44.01052442 +0000 UTC m=+0.015922435 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:24:44.395 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:44 smithi102 podman[141826]: 2024-09-13 14:24:44.11689356 +0000 UTC m=+0.122291611 container create 76bc7235b1cb049b5d977a85e25edd32d9ce4d82dfac8c2c6d8ec7fa8df804fd (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3-deactivate, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-13T14:24:44.395 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:44 smithi102 podman[141826]: 2024-09-13 14:24:44.319611059 +0000 UTC m=+0.325009069 container init 76bc7235b1cb049b5d977a85e25edd32d9ce4d82dfac8c2c6d8ec7fa8df804fd (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2) 2024-09-13T14:24:44.395 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:44 smithi102 podman[141826]: 2024-09-13 14:24:44.32413405 +0000 UTC m=+0.329532064 container start 76bc7235b1cb049b5d977a85e25edd32d9ce4d82dfac8c2c6d8ec7fa8df804fd (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3-deactivate, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T14:24:44.396 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:44 smithi102 podman[141826]: 2024-09-13 14:24:44.394576178 +0000 UTC m=+0.399974197 container attach 76bc7235b1cb049b5d977a85e25edd32d9ce4d82dfac8c2c6d8ec7fa8df804fd (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-13T14:24:44.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:44 smithi102 ceph-mon[110183]: pgmap v238: 73 pgs: 13 stale+active+clean, 60 active+clean; 4.6 MiB data, 251 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:44.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:44 smithi102 ceph-mon[110183]: osdmap e107: 8 total, 7 up, 8 in 2024-09-13T14:24:44.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:44 smithi102 ceph-mon[115269]: pgmap v238: 73 pgs: 13 stale+active+clean, 60 active+clean; 4.6 MiB data, 251 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:44.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:44 smithi102 ceph-mon[115269]: osdmap e107: 8 total, 7 up, 8 in 2024-09-13T14:24:44.669 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:44 smithi102 podman[141826]: 2024-09-13 14:24:44.510915623 +0000 UTC m=+0.516313631 container died 76bc7235b1cb049b5d977a85e25edd32d9ce4d82dfac8c2c6d8ec7fa8df804fd (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, RELEASE=wip-aclamk-cbt-combined-d58f730, ceph=True, GIT_CLEAN=True) 2024-09-13T14:24:44.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:44 smithi195 ceph-mon[104262]: pgmap v238: 73 pgs: 13 stale+active+clean, 60 active+clean; 4.6 MiB data, 251 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:44.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:44 smithi195 ceph-mon[104262]: osdmap e107: 8 total, 7 up, 8 in 2024-09-13T14:24:45.059 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:45 smithi102 podman[141869]: 2024-09-13 14:24:45.039983669 +0000 UTC m=+0.519934761 container remove 76bc7235b1cb049b5d977a85e25edd32d9ce4d82dfac8c2c6d8ec7fa8df804fd (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-09-13T14:24:45.418 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:45 smithi102 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@osd.3.service: Deactivated successfully. 2024-09-13T14:24:45.418 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:45 smithi102 systemd[1]: Stopped Ceph osd.3 for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:24:45.418 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:45 smithi102 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@osd.3.service: Consumed 6.122s CPU time. 2024-09-13T14:24:45.419 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:45 smithi102 systemd[1]: Starting Ceph osd.3 for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T14:24:45.778 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:45 smithi102 podman[141971]: 2024-09-13 14:24:45.465948228 +0000 UTC m=+0.100782169 container create eaa3805f6600a9d73aada6f7c6e1ed7968922319bc85a534bf02f87af83fae84 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3-activate, RELEASE=wip-aclamk-cbt-combined-d58f730, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T14:24:45.778 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:45 smithi102 podman[141971]: 2024-09-13 14:24:45.381079846 +0000 UTC m=+0.015913794 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:24:45.778 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:45 smithi102 podman[141971]: 2024-09-13 14:24:45.69436954 +0000 UTC m=+0.329203487 container init eaa3805f6600a9d73aada6f7c6e1ed7968922319bc85a534bf02f87af83fae84 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3-activate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=wip-aclamk-cbt-combined-d58f730, maintainer=Guillaume Abrioux ) 2024-09-13T14:24:45.778 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:45 smithi102 podman[141971]: 2024-09-13 14:24:45.699734241 +0000 UTC m=+0.334568186 container start eaa3805f6600a9d73aada6f7c6e1ed7968922319bc85a534bf02f87af83fae84 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3-activate, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-13T14:24:45.778 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:45 smithi102 podman[141971]: 2024-09-13 14:24:45.777206972 +0000 UTC m=+0.412040929 container attach eaa3805f6600a9d73aada6f7c6e1ed7968922319bc85a534bf02f87af83fae84 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3-activate, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-aclamk-cbt-combined-d58f730) 2024-09-13T14:24:46.168 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:45 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3-activate[141988]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:24:46.169 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:45 smithi102 bash[141971]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:24:46.169 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:45 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3-activate[141988]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:24:46.169 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:45 smithi102 bash[141971]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:24:46.169 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:45 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3-activate[141988]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-09-13T14:24:46.169 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:45 smithi102 bash[141971]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-09-13T14:24:46.169 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:45 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3-activate[141988]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:24:46.169 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:45 smithi102 bash[141971]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:24:46.169 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:45 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3-activate[141988]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:24:46.169 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:45 smithi102 bash[141971]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:24:46.169 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:45 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3-activate[141988]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-13T14:24:46.169 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:45 smithi102 bash[141971]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-13T14:24:46.169 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:45 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3-activate[141988]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_1 --path /var/lib/ceph/osd/ceph-3 --no-mon-config 2024-09-13T14:24:46.170 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:45 smithi102 bash[141971]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_1 --path /var/lib/ceph/osd/ceph-3 --no-mon-config 2024-09-13T14:24:46.573 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:24:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:24:46] "GET /metrics HTTP/1.1" 200 34012 "" "Prometheus/2.51.0" 2024-09-13T14:24:46.573 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3-activate[141988]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3/block 2024-09-13T14:24:46.573 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:46 smithi102 bash[141971]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3/block 2024-09-13T14:24:46.573 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3-activate[141988]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-3/block 2024-09-13T14:24:46.573 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:46 smithi102 bash[141971]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-3/block 2024-09-13T14:24:46.573 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3-activate[141988]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-13T14:24:46.573 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:46 smithi102 bash[141971]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-13T14:24:46.574 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3-activate[141988]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-13T14:24:46.574 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:46 smithi102 bash[141971]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-13T14:24:46.574 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3-activate[141988]: --> ceph-volume lvm activate successful for osd ID: 3 2024-09-13T14:24:46.574 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:46 smithi102 bash[141971]: --> ceph-volume lvm activate successful for osd ID: 3 2024-09-13T14:24:46.574 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:46 smithi102 podman[141971]: 2024-09-13 14:24:46.244914763 +0000 UTC m=+0.879748713 container died eaa3805f6600a9d73aada6f7c6e1ed7968922319bc85a534bf02f87af83fae84 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3-activate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-aclamk-cbt-combined-d58f730, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-13T14:24:46.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:46 smithi102 ceph-mon[110183]: pgmap v240: 73 pgs: 4 active+undersized, 10 stale+active+clean, 59 active+clean; 4.6 MiB data, 251 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:46.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:46 smithi102 ceph-mon[115269]: pgmap v240: 73 pgs: 4 active+undersized, 10 stale+active+clean, 59 active+clean; 4.6 MiB data, 251 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:46.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:46 smithi195 ceph-mon[104262]: pgmap v240: 73 pgs: 4 active+undersized, 10 stale+active+clean, 59 active+clean; 4.6 MiB data, 251 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:24:47.418 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:46 smithi102 podman[142129]: 2024-09-13 14:24:46.948263211 +0000 UTC m=+0.697152226 container remove eaa3805f6600a9d73aada6f7c6e1ed7968922319bc85a534bf02f87af83fae84 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3-activate, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-aclamk-cbt-combined-d58f730, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-13T14:24:47.418 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:47 smithi102 podman[142193]: 2024-09-13 14:24:47.056062547 +0000 UTC m=+0.015415001 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:24:47.419 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:47 smithi102 podman[142193]: 2024-09-13 14:24:47.16583036 +0000 UTC m=+0.125182860 container create cb2596701036e089f0c6d8e67a427d5a298858b68305fa58f72e164db262dea8 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True) 2024-09-13T14:24:47.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:47 smithi102 ceph-mon[115269]: Health check failed: Degraded data redundancy: 4/84 objects degraded (4.762%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:24:47.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:47 smithi102 ceph-mon[110183]: Health check failed: Degraded data redundancy: 4/84 objects degraded (4.762%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:24:47.919 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:47 smithi102 podman[142193]: 2024-09-13 14:24:47.560138155 +0000 UTC m=+0.519490600 container init cb2596701036e089f0c6d8e67a427d5a298858b68305fa58f72e164db262dea8 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-09-13T14:24:47.919 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:47 smithi102 podman[142193]: 2024-09-13 14:24:47.565418437 +0000 UTC m=+0.524770889 container start cb2596701036e089f0c6d8e67a427d5a298858b68305fa58f72e164db262dea8 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux ) 2024-09-13T14:24:47.919 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:47 smithi102 ceph-osd[142231]: -- 172.21.15.102:0/3298160313 <== mon.0 v2:172.21.15.102:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bd4e1fba40 con 0x55bd4e1d9c00 2024-09-13T14:24:47.919 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:47 smithi102 bash[142193]: cb2596701036e089f0c6d8e67a427d5a298858b68305fa58f72e164db262dea8 2024-09-13T14:24:47.919 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:47 smithi102 systemd[1]: Started Ceph osd.3 for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:24:47.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:47 smithi195 ceph-mon[104262]: Health check failed: Degraded data redundancy: 4/84 objects degraded (4.762%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:24:48.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:48 smithi195 ceph-mon[104262]: pgmap v241: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 251 MiB used, 715 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-09-13T14:24:48.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:48 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:48.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:48 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:48.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:48 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:48.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:48 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:24:49.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:48 smithi102 ceph-mon[115269]: pgmap v241: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 251 MiB used, 715 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-09-13T14:24:49.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:48 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:49.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:48 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:49.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:48 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:49.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:48 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:24:49.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:48 smithi102 ceph-mon[110183]: pgmap v241: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 251 MiB used, 715 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-09-13T14:24:49.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:48 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:49.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:48 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:49.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:48 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:49.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:48 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:24:49.668 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:49 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3[142208]: 2024-09-13T14:24:49.241+0000 7fd1de59b640 -1 Falling back to public interface 2024-09-13T14:24:50.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:50 smithi195 ceph-mon[104262]: pgmap v242: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 251 MiB used, 715 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-09-13T14:24:51.034 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:50 smithi102 ceph-mon[110183]: pgmap v242: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 251 MiB used, 715 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-09-13T14:24:51.034 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:50 smithi102 ceph-mon[115269]: pgmap v242: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 251 MiB used, 715 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-09-13T14:24:52.581 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:52 smithi102 ceph-mon[110183]: pgmap v243: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 251 MiB used, 715 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-09-13T14:24:52.581 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:52 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:52.581 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:52 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:52.581 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:52 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:52.581 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:52 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:52.581 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:52 smithi102 ceph-mon[115269]: pgmap v243: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 251 MiB used, 715 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-09-13T14:24:52.582 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:52 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:52.582 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:52 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:52.582 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:52 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:52.582 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:52 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:52.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:52 smithi195 ceph-mon[104262]: pgmap v243: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 251 MiB used, 715 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-09-13T14:24:52.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:52 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:52.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:52 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:52.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:52 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:52.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:52 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:52.918 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:52 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3[142208]: 2024-09-13T14:24:52.772+0000 7fd1de59b640 -1 osd.3 105 log_to_monitors true 2024-09-13T14:24:53.290 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:53 smithi102 ceph-mon[110183]: from='osd.3 [v2:172.21.15.102:6826/1858757656,v1:172.21.15.102:6827/1858757656]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-13T14:24:53.290 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:53 smithi102 ceph-mon[110183]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-13T14:24:53.290 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:53 smithi102 ceph-mon[115269]: from='osd.3 [v2:172.21.15.102:6826/1858757656,v1:172.21.15.102:6827/1858757656]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-13T14:24:53.290 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:53 smithi102 ceph-mon[115269]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-13T14:24:53.545 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 14:24:53 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3[142208]: 2024-09-13T14:24:53.289+0000 7fd1d233d640 -1 osd.3 105 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T14:24:53.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:53 smithi195 ceph-mon[104262]: from='osd.3 [v2:172.21.15.102:6826/1858757656,v1:172.21.15.102:6827/1858757656]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-13T14:24:53.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:53 smithi195 ceph-mon[104262]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-13T14:24:54.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:54 smithi102 ceph-mon[110183]: pgmap v244: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 251 MiB used, 715 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-09-13T14:24:54.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:54 smithi102 ceph-mon[110183]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-13T14:24:54.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:54 smithi102 ceph-mon[110183]: osdmap e108: 8 total, 7 up, 8 in 2024-09-13T14:24:54.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:54 smithi102 ceph-mon[110183]: from='osd.3 [v2:172.21.15.102:6826/1858757656,v1:172.21.15.102:6827/1858757656]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]} : dispatch 2024-09-13T14:24:54.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:54 smithi102 ceph-mon[110183]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]} : dispatch 2024-09-13T14:24:54.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:54 smithi102 ceph-mon[115269]: pgmap v244: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 251 MiB used, 715 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-09-13T14:24:54.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:54 smithi102 ceph-mon[115269]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-13T14:24:54.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:54 smithi102 ceph-mon[115269]: osdmap e108: 8 total, 7 up, 8 in 2024-09-13T14:24:54.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:54 smithi102 ceph-mon[115269]: from='osd.3 [v2:172.21.15.102:6826/1858757656,v1:172.21.15.102:6827/1858757656]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]} : dispatch 2024-09-13T14:24:54.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:54 smithi102 ceph-mon[115269]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]} : dispatch 2024-09-13T14:24:54.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:54 smithi195 ceph-mon[104262]: pgmap v244: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 251 MiB used, 715 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-09-13T14:24:54.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:54 smithi195 ceph-mon[104262]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-13T14:24:54.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:54 smithi195 ceph-mon[104262]: osdmap e108: 8 total, 7 up, 8 in 2024-09-13T14:24:54.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:54 smithi195 ceph-mon[104262]: from='osd.3 [v2:172.21.15.102:6826/1858757656,v1:172.21.15.102:6827/1858757656]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]} : dispatch 2024-09-13T14:24:54.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:54 smithi195 ceph-mon[104262]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]} : dispatch 2024-09-13T14:24:55.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:55 smithi102 ceph-mon[110183]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:24:55.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:55 smithi102 ceph-mon[110183]: osd.3 [v2:172.21.15.102:6826/1858757656,v1:172.21.15.102:6827/1858757656] boot 2024-09-13T14:24:55.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:55 smithi102 ceph-mon[110183]: osdmap e109: 8 total, 8 up, 8 in 2024-09-13T14:24:55.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:55 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-13T14:24:55.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:55 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:55.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:55 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:55.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:55 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:24:55.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:55 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:24:55.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:55 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:55.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:55 smithi102 ceph-mon[110183]: osdmap e110: 8 total, 8 up, 8 in 2024-09-13T14:24:55.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:55 smithi102 ceph-mon[115269]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:24:55.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:55 smithi102 ceph-mon[115269]: osd.3 [v2:172.21.15.102:6826/1858757656,v1:172.21.15.102:6827/1858757656] boot 2024-09-13T14:24:55.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:55 smithi102 ceph-mon[115269]: osdmap e109: 8 total, 8 up, 8 in 2024-09-13T14:24:55.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:55 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-13T14:24:55.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:55 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:55.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:55 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:55.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:55 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:24:55.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:55 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:24:55.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:55 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:55.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:55 smithi102 ceph-mon[115269]: osdmap e110: 8 total, 8 up, 8 in 2024-09-13T14:24:55.648 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:55 smithi195 ceph-mon[104262]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:24:55.648 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:55 smithi195 ceph-mon[104262]: osd.3 [v2:172.21.15.102:6826/1858757656,v1:172.21.15.102:6827/1858757656] boot 2024-09-13T14:24:55.648 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:55 smithi195 ceph-mon[104262]: osdmap e109: 8 total, 8 up, 8 in 2024-09-13T14:24:55.648 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:55 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-13T14:24:55.649 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:55 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:55.649 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:55 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:55.649 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:55 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:24:55.649 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:55 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:24:55.649 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:55 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:55.649 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:55 smithi195 ceph-mon[104262]: osdmap e110: 8 total, 8 up, 8 in 2024-09-13T14:24:56.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:24:56] "GET /metrics HTTP/1.1" 200 33870 "" "Prometheus/2.51.0" 2024-09-13T14:24:56.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[115269]: pgmap v247: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-09-13T14:24:56.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:24:56.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:24:56.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:24:56.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[115269]: Upgrade: Setting container_image for all nvmeof 2024-09-13T14:24:56.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:56.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:24:56.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:24:56.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:24:56.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:24:56.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:24:56.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:24:56.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[115269]: Upgrade: Finalizing container_image settings 2024-09-13T14:24:56.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr"} : dispatch 2024-09-13T14:24:56.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-13T14:24:56.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:24:56.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-13T14:24:56.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.crash"} : dispatch 2024-09-13T14:24:56.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-09-13T14:24:56.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd"} : dispatch 2024-09-13T14:24:56.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mds"} : dispatch 2024-09-13T14:24:56.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw"} : dispatch 2024-09-13T14:24:56.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"} : dispatch 2024-09-13T14:24:56.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:24:56.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"} : dispatch 2024-09-13T14:24:56.671 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[110183]: pgmap v247: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-09-13T14:24:56.671 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:24:56.671 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:24:56.671 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:24:56.671 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[110183]: Upgrade: Setting container_image for all nvmeof 2024-09-13T14:24:56.671 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:56.671 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:24:56.672 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:24:56.672 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:24:56.672 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:24:56.672 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:24:56.672 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:24:56.672 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[110183]: Upgrade: Finalizing container_image settings 2024-09-13T14:24:56.672 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr"} : dispatch 2024-09-13T14:24:56.672 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-13T14:24:56.672 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:24:56.672 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-13T14:24:56.672 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.crash"} : dispatch 2024-09-13T14:24:56.672 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-09-13T14:24:56.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd"} : dispatch 2024-09-13T14:24:56.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mds"} : dispatch 2024-09-13T14:24:56.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw"} : dispatch 2024-09-13T14:24:56.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"} : dispatch 2024-09-13T14:24:56.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:24:56.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"} : dispatch 2024-09-13T14:24:56.673 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.iscsi"} : dispatch 2024-09-13T14:24:56.673 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nfs"} : dispatch 2024-09-13T14:24:56.673 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"} : dispatch 2024-09-13T14:24:56.673 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-13T14:24:56.674 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:24:56.674 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:24:56.674 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:24:56.674 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:24:56.674 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:24:56.674 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:24:56.674 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[115269]: Upgrade: Complete! 2024-09-13T14:24:56.674 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"} : dispatch 2024-09-13T14:24:56.674 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-13T14:24:56.674 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:24:56.674 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:24:56.674 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:24:56.674 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:56.675 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:56.675 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:24:56.675 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:24:56.675 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:24:56.675 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:24:56.675 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:56.675 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:56.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.iscsi"} : dispatch 2024-09-13T14:24:56.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nfs"} : dispatch 2024-09-13T14:24:56.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"} : dispatch 2024-09-13T14:24:56.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-13T14:24:56.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:24:56.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:24:56.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:24:56.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:24:56.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:24:56.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:24:56.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[110183]: Upgrade: Complete! 2024-09-13T14:24:56.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"} : dispatch 2024-09-13T14:24:56.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-13T14:24:56.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:24:56.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:24:56.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:24:56.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:56.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:56.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:24:56.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:24:56.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:24:56.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:24:56.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:56.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:56.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:56 smithi195 ceph-mon[104262]: pgmap v247: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-09-13T14:24:56.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:24:56.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:24:56.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:24:56.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:56 smithi195 ceph-mon[104262]: Upgrade: Setting container_image for all nvmeof 2024-09-13T14:24:56.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:56.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:24:56.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:24:56.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:24:56.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:24:56.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:24:56.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:24:56.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:56 smithi195 ceph-mon[104262]: Upgrade: Finalizing container_image settings 2024-09-13T14:24:56.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr"} : dispatch 2024-09-13T14:24:56.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-13T14:24:56.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:24:56.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-13T14:24:56.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.crash"} : dispatch 2024-09-13T14:24:56.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-09-13T14:24:56.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd"} : dispatch 2024-09-13T14:24:56.750 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mds"} : dispatch 2024-09-13T14:24:56.750 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw"} : dispatch 2024-09-13T14:24:56.750 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"} : dispatch 2024-09-13T14:24:56.750 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:24:56.750 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"} : dispatch 2024-09-13T14:24:56.750 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.iscsi"} : dispatch 2024-09-13T14:24:56.750 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nfs"} : dispatch 2024-09-13T14:24:56.750 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"} : dispatch 2024-09-13T14:24:56.750 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-13T14:24:56.750 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:24:56.750 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:24:56.750 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:24:56.750 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:24:56.751 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:24:56.751 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:24:56.751 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:56 smithi195 ceph-mon[104262]: Upgrade: Complete! 2024-09-13T14:24:56.751 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"} : dispatch 2024-09-13T14:24:56.751 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-13T14:24:56.751 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:24:56.751 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:24:56.751 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:24:56.751 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:56.751 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:56.751 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:24:56.751 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:24:56.751 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:24:56.751 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:24:56.752 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:56.752 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:24:57.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:57 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 2/84 objects degraded (2.381%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:24:57.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:57 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 2/84 objects degraded (2.381%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:24:57.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:57 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 2/84 objects degraded (2.381%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:24:58.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:58 smithi102 ceph-mon[110183]: pgmap v249: 73 pgs: 12 active+undersized, 2 active+undersized+degraded, 59 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail; 2/84 objects degraded (2.381%) 2024-09-13T14:24:58.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:58 smithi102 ceph-mon[115269]: pgmap v249: 73 pgs: 12 active+undersized, 2 active+undersized+degraded, 59 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail; 2/84 objects degraded (2.381%) 2024-09-13T14:24:58.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:58 smithi195 ceph-mon[104262]: pgmap v249: 73 pgs: 12 active+undersized, 2 active+undersized+degraded, 59 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail; 2/84 objects degraded (2.381%) 2024-09-13T14:24:59.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:59 smithi102 ceph-mon[115269]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/84 objects degraded (2.381%), 2 pgs degraded) 2024-09-13T14:24:59.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:24:59 smithi102 ceph-mon[115269]: Cluster is now healthy 2024-09-13T14:24:59.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:59 smithi102 ceph-mon[110183]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/84 objects degraded (2.381%), 2 pgs degraded) 2024-09-13T14:24:59.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:24:59 smithi102 ceph-mon[110183]: Cluster is now healthy 2024-09-13T14:24:59.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:59 smithi195 ceph-mon[104262]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/84 objects degraded (2.381%), 2 pgs degraded) 2024-09-13T14:24:59.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:24:59 smithi195 ceph-mon[104262]: Cluster is now healthy 2024-09-13T14:25:00.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:00 smithi102 ceph-mon[110183]: pgmap v250: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:00.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:00 smithi102 ceph-mon[115269]: pgmap v250: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:00.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:00 smithi195 ceph-mon[104262]: pgmap v250: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:02.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:02 smithi102 ceph-mon[110183]: pgmap v251: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:02.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:02 smithi102 ceph-mon[115269]: pgmap v251: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:02.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:02 smithi195 ceph-mon[104262]: pgmap v251: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:04.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:04 smithi102 ceph-mon[110183]: pgmap v252: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:04.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:04 smithi102 ceph-mon[115269]: pgmap v252: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:04.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:04 smithi195 ceph-mon[104262]: pgmap v252: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:06.490 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:25:06 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:25:06] "GET /metrics HTTP/1.1" 200 33870 "" "Prometheus/2.51.0" 2024-09-13T14:25:06.611 DEBUG:teuthology.parallel:result is None 2024-09-13T14:25:06.611 INFO:teuthology.run_tasks:Running task print... 2024-09-13T14:25:06.621 INFO:teuthology.task.print:**** done upgrading first half of cluster **** 2024-09-13T14:25:06.621 INFO:teuthology.run_tasks:Running task ceph.healthy... 2024-09-13T14:25:06.655 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-13T14:25:06.655 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-13T14:25:06.655 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph mgr dump --format=json 2024-09-13T14:25:06.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:06 smithi195 ceph-mon[104262]: pgmap v253: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:06.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:06 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:25:06.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:06 smithi195 ceph-mon[104262]: from='client.34419 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:25:06.856 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:25:06.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:06 smithi102 ceph-mon[110183]: pgmap v253: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:06.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:06 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:25:06.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:06 smithi102 ceph-mon[110183]: from='client.34419 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:25:06.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:06 smithi102 ceph-mon[115269]: pgmap v253: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:06.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:06 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:25:06.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:06 smithi102 ceph-mon[115269]: from='client.34419 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:25:08.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:08 smithi195 ceph-mon[104262]: pgmap v254: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:08.785 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:25:08.786 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:08 smithi102 ceph-mon[110183]: pgmap v254: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:08.786 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:08 smithi102 ceph-mon[115269]: pgmap v254: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:09.417 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":41,"flags":0,"active_gid":34107,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":3745799088},{"type":"v1","addr":"172.21.15.102:6801","nonce":3745799088}]},"active_addr":"172.21.15.102:6801/3745799088","active_change":"2024-09-13T14:17:25.102348+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[{"gid":25270,"name":"x","mgr_features":4540719139924082687,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.102:8443/","prometheus":"http://172.21.15.102:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":90,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":2558404676}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":2188424410}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":1601121167}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":1875985575}]}]} 2024-09-13T14:25:09.422 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-13T14:25:09.423 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-13T14:25:09.423 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-13T14:25:09.623 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:25:09.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:09 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/746170162' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-13T14:25:09.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:09 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/746170162' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-13T14:25:09.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:09 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/746170162' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-13T14:25:10.898 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:10 smithi102 ceph-mon[115269]: pgmap v255: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:10.898 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:25:10.898 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:25:10.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:10 smithi102 ceph-mon[110183]: pgmap v255: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:10.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:25:10.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:25:10.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:10 smithi195 ceph-mon[104262]: pgmap v255: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:10.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:25:10.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:25:11.405 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:25:11.405 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":110,"fsid":"08cee502-71d9-11ef-bceb-c7b262605968","created":"2024-09-13T14:04:48.179879+0000","modified":"2024-09-13T14:24:55.267939+0000","last_up_change":"2024-09-13T14:24:54.266582+0000","last_in_change":"2024-09-13T14:09:19.207877+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":22,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T14:07:41.927240+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T14:10:10.981300+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":85,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-13T14:11:27.935818+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-13T14:11:28.454262+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b43b49e0-1cf0-47b8-aa85-37108f321220","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":90,"up_from":94,"up_thru":109,"down_at":91,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":3285732759},{"type":"v1","addr":"172.21.15.102:6803","nonce":3285732759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":3285732759},{"type":"v1","addr":"172.21.15.102:6805","nonce":3285732759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3285732759},{"type":"v1","addr":"172.21.15.102:6809","nonce":3285732759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":3285732759},{"type":"v1","addr":"172.21.15.102:6807","nonce":3285732759}]},"public_addr":"172.21.15.102:6803/3285732759","cluster_addr":"172.21.15.102:6805/3285732759","heartbeat_back_addr":"172.21.15.102:6809/3285732759","heartbeat_front_addr":"172.21.15.102:6807/3285732759","state":["exists","up"]},{"osd":1,"uuid":"27e10d5e-c471-4132-9c73-83c5708c3b08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":95,"up_from":99,"up_thru":109,"down_at":96,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":2240964461},{"type":"v1","addr":"172.21.15.102:6811","nonce":2240964461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":2240964461},{"type":"v1","addr":"172.21.15.102:6813","nonce":2240964461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":2240964461},{"type":"v1","addr":"172.21.15.102:6817","nonce":2240964461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":2240964461},{"type":"v1","addr":"172.21.15.102:6815","nonce":2240964461}]},"public_addr":"172.21.15.102:6811/2240964461","cluster_addr":"172.21.15.102:6813/2240964461","heartbeat_back_addr":"172.21.15.102:6817/2240964461","heartbeat_front_addr":"172.21.15.102:6815/2240964461","state":["exists","up"]},{"osd":2,"uuid":"40a94b2b-2c24-4164-91be-0816a1acf3d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":100,"up_from":104,"up_thru":109,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":4066184234},{"type":"v1","addr":"172.21.15.102:6819","nonce":4066184234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":4066184234},{"type":"v1","addr":"172.21.15.102:6821","nonce":4066184234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":4066184234},{"type":"v1","addr":"172.21.15.102:6825","nonce":4066184234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":4066184234},{"type":"v1","addr":"172.21.15.102:6823","nonce":4066184234}]},"public_addr":"172.21.15.102:6819/4066184234","cluster_addr":"172.21.15.102:6821/4066184234","heartbeat_back_addr":"172.21.15.102:6825/4066184234","heartbeat_front_addr":"172.21.15.102:6823/4066184234","state":["exists","up"]},{"osd":3,"uuid":"2ed85faf-6610-4bcc-a09d-4a6f93f5c6a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":25,"last_clean_end":105,"up_from":109,"up_thru":109,"down_at":106,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1858757656},{"type":"v1","addr":"172.21.15.102:6827","nonce":1858757656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":1858757656},{"type":"v1","addr":"172.21.15.102:6829","nonce":1858757656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":1858757656},{"type":"v1","addr":"172.21.15.102:6833","nonce":1858757656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":1858757656},{"type":"v1","addr":"172.21.15.102:6831","nonce":1858757656}]},"public_addr":"172.21.15.102:6827/1858757656","cluster_addr":"172.21.15.102:6829/1858757656","heartbeat_back_addr":"172.21.15.102:6833/1858757656","heartbeat_front_addr":"172.21.15.102:6831/1858757656","state":["exists","up"]},{"osd":4,"uuid":"8a45f350-fb3e-4049-bcc4-532d3e20c238","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":2821651815},{"type":"v1","addr":"172.21.15.195:6801","nonce":2821651815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":2821651815},{"type":"v1","addr":"172.21.15.195:6803","nonce":2821651815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":2821651815},{"type":"v1","addr":"172.21.15.195:6807","nonce":2821651815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":2821651815},{"type":"v1","addr":"172.21.15.195:6805","nonce":2821651815}]},"public_addr":"172.21.15.195:6801/2821651815","cluster_addr":"172.21.15.195:6803/2821651815","heartbeat_back_addr":"172.21.15.195:6807/2821651815","heartbeat_front_addr":"172.21.15.195:6805/2821651815","state":["exists","up"]},{"osd":5,"uuid":"363cf638-db26-4c86-b2be-e075fd09ac16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":3074139237},{"type":"v1","addr":"172.21.15.195:6809","nonce":3074139237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":3074139237},{"type":"v1","addr":"172.21.15.195:6811","nonce":3074139237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":3074139237},{"type":"v1","addr":"172.21.15.195:6815","nonce":3074139237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":3074139237},{"type":"v1","addr":"172.21.15.195:6813","nonce":3074139237}]},"public_addr":"172.21.15.195:6809/3074139237","cluster_addr":"172.21.15.195:6811/3074139237","heartbeat_back_addr":"172.21.15.195:6815/3074139237","heartbeat_front_addr":"172.21.15.195:6813/3074139237","state":["exists","up"]},{"osd":6,"uuid":"f37416e8-9c25-4860-a32f-0d1c404a8bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":4026627594},{"type":"v1","addr":"172.21.15.195:6817","nonce":4026627594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":4026627594},{"type":"v1","addr":"172.21.15.195:6819","nonce":4026627594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":4026627594},{"type":"v1","addr":"172.21.15.195:6823","nonce":4026627594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":4026627594},{"type":"v1","addr":"172.21.15.195:6821","nonce":4026627594}]},"public_addr":"172.21.15.195:6817/4026627594","cluster_addr":"172.21.15.195:6819/4026627594","heartbeat_back_addr":"172.21.15.195:6823/4026627594","heartbeat_front_addr":"172.21.15.195:6821/4026627594","state":["exists","up"]},{"osd":7,"uuid":"9a384a4d-fddb-4fb3-843a-77c19cca0368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2059403787},{"type":"v1","addr":"172.21.15.195:6825","nonce":2059403787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2059403787},{"type":"v1","addr":"172.21.15.195:6827","nonce":2059403787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2059403787},{"type":"v1","addr":"172.21.15.195:6831","nonce":2059403787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2059403787},{"type":"v1","addr":"172.21.15.195:6829","nonce":2059403787}]},"public_addr":"172.21.15.195:6825/2059403787","cluster_addr":"172.21.15.195:6827/2059403787","heartbeat_back_addr":"172.21.15.195:6831/2059403787","heartbeat_front_addr":"172.21.15.195:6829/2059403787","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:06:53.502746+0000","dead_epoch":90},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:07:16.251669+0000","dead_epoch":95},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:07:38.848272+0000","dead_epoch":100},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:08:02.013035+0000","dead_epoch":105},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:08:23.043482+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:08:45.653159+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:09:10.187018+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:09:34.393753+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/3514811128":"2024-09-14T14:17:25.102165+0000","172.21.15.102:6801/3856547061":"2024-09-14T14:17:25.102165+0000","172.21.15.102:6800/3856547061":"2024-09-14T14:17:25.102165+0000","172.21.15.102:0/3399546099":"2024-09-14T14:05:37.634624+0000","172.21.15.102:0/2489373272":"2024-09-14T14:14:08.048329+0000","172.21.15.195:6835/883376241":"2024-09-14T14:14:48.059120+0000","172.21.15.102:6801/2761043787":"2024-09-14T14:05:15.286479+0000","172.21.15.102:0/1574792592":"2024-09-14T14:05:15.286479+0000","172.21.15.102:6801/63514412":"2024-09-14T14:10:09.037657+0000","172.21.15.102:6800/63514412":"2024-09-14T14:10:09.037657+0000","172.21.15.102:0/2083488703":"2024-09-14T14:14:08.048329+0000","172.21.15.102:6800/2761043787":"2024-09-14T14:05:15.286479+0000","172.21.15.102:6801/3480998245":"2024-09-14T14:05:37.634624+0000","172.21.15.195:6834/883376241":"2024-09-14T14:14:48.059120+0000","172.21.15.102:0/1792178465":"2024-09-14T14:14:08.048329+0000","172.21.15.102:0/1430433627":"2024-09-14T14:14:08.048329+0000","172.21.15.195:0/2524944732":"2024-09-14T14:14:48.059120+0000","172.21.15.102:0/2975343136":"2024-09-14T14:17:25.102165+0000","172.21.15.195:0/3629133224":"2024-09-14T14:14:48.059120+0000","172.21.15.102:0/1160237252":"2024-09-14T14:10:09.037657+0000","172.21.15.102:0/120875803":"2024-09-14T14:17:25.102165+0000","172.21.15.102:0/1582001159":"2024-09-14T14:05:15.286479+0000","172.21.15.195:0/1568198689":"2024-09-14T14:14:48.059120+0000","172.21.15.102:6800/3480998245":"2024-09-14T14:05:37.634624+0000","172.21.15.195:0/4292730726":"2024-09-14T14:14:48.059120+0000","172.21.15.102:6800/1522897266":"2024-09-14T14:14:08.048329+0000","172.21.15.102:0/1939570054":"2024-09-14T14:05:37.634624+0000","172.21.15.102:0/1765904351":"2024-09-14T14:10:09.037657+0000","172.21.15.102:6801/1522897266":"2024-09-14T14:14:08.048329+0000","172.21.15.102:0/3189088157":"2024-09-14T14:05:15.286479+0000","172.21.15.195:0/495121375":"2024-09-14T14:14:48.059120+0000","172.21.15.102:0/2444394840":"2024-09-14T14:05:37.634624+0000","172.21.15.195:0/1729595567":"2024-09-14T14:14:48.059120+0000","172.21.15.102:0/2308924552":"2024-09-14T14:10:09.037657+0000","172.21.15.102:0/2416257718":"2024-09-14T14:10:09.037657+0000","172.21.15.102:0/2872102326":"2024-09-14T14:17:25.102165+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T14:25:11.773 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:11 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2302272147' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T14:25:11.773 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:11 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2302272147' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T14:25:11.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:11 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2302272147' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T14:25:12.083 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-13T14:25:12.083 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-13T14:25:12.281 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:25:12.665 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:12 smithi102 ceph-mon[110183]: pgmap v256: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:12.665 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:12 smithi102 ceph-mon[115269]: pgmap v256: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:12.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:12 smithi195 ceph-mon[104262]: pgmap v256: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:14.070 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:25:14.070 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":110,"fsid":"08cee502-71d9-11ef-bceb-c7b262605968","created":"2024-09-13T14:04:48.179879+0000","modified":"2024-09-13T14:24:55.267939+0000","last_up_change":"2024-09-13T14:24:54.266582+0000","last_in_change":"2024-09-13T14:09:19.207877+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":22,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T14:07:41.927240+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T14:10:10.981300+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":85,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-13T14:11:27.935818+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-13T14:11:28.454262+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b43b49e0-1cf0-47b8-aa85-37108f321220","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":90,"up_from":94,"up_thru":109,"down_at":91,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":3285732759},{"type":"v1","addr":"172.21.15.102:6803","nonce":3285732759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":3285732759},{"type":"v1","addr":"172.21.15.102:6805","nonce":3285732759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3285732759},{"type":"v1","addr":"172.21.15.102:6809","nonce":3285732759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":3285732759},{"type":"v1","addr":"172.21.15.102:6807","nonce":3285732759}]},"public_addr":"172.21.15.102:6803/3285732759","cluster_addr":"172.21.15.102:6805/3285732759","heartbeat_back_addr":"172.21.15.102:6809/3285732759","heartbeat_front_addr":"172.21.15.102:6807/3285732759","state":["exists","up"]},{"osd":1,"uuid":"27e10d5e-c471-4132-9c73-83c5708c3b08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":95,"up_from":99,"up_thru":109,"down_at":96,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":2240964461},{"type":"v1","addr":"172.21.15.102:6811","nonce":2240964461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":2240964461},{"type":"v1","addr":"172.21.15.102:6813","nonce":2240964461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":2240964461},{"type":"v1","addr":"172.21.15.102:6817","nonce":2240964461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":2240964461},{"type":"v1","addr":"172.21.15.102:6815","nonce":2240964461}]},"public_addr":"172.21.15.102:6811/2240964461","cluster_addr":"172.21.15.102:6813/2240964461","heartbeat_back_addr":"172.21.15.102:6817/2240964461","heartbeat_front_addr":"172.21.15.102:6815/2240964461","state":["exists","up"]},{"osd":2,"uuid":"40a94b2b-2c24-4164-91be-0816a1acf3d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":100,"up_from":104,"up_thru":109,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":4066184234},{"type":"v1","addr":"172.21.15.102:6819","nonce":4066184234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":4066184234},{"type":"v1","addr":"172.21.15.102:6821","nonce":4066184234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":4066184234},{"type":"v1","addr":"172.21.15.102:6825","nonce":4066184234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":4066184234},{"type":"v1","addr":"172.21.15.102:6823","nonce":4066184234}]},"public_addr":"172.21.15.102:6819/4066184234","cluster_addr":"172.21.15.102:6821/4066184234","heartbeat_back_addr":"172.21.15.102:6825/4066184234","heartbeat_front_addr":"172.21.15.102:6823/4066184234","state":["exists","up"]},{"osd":3,"uuid":"2ed85faf-6610-4bcc-a09d-4a6f93f5c6a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":25,"last_clean_end":105,"up_from":109,"up_thru":109,"down_at":106,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1858757656},{"type":"v1","addr":"172.21.15.102:6827","nonce":1858757656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":1858757656},{"type":"v1","addr":"172.21.15.102:6829","nonce":1858757656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":1858757656},{"type":"v1","addr":"172.21.15.102:6833","nonce":1858757656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":1858757656},{"type":"v1","addr":"172.21.15.102:6831","nonce":1858757656}]},"public_addr":"172.21.15.102:6827/1858757656","cluster_addr":"172.21.15.102:6829/1858757656","heartbeat_back_addr":"172.21.15.102:6833/1858757656","heartbeat_front_addr":"172.21.15.102:6831/1858757656","state":["exists","up"]},{"osd":4,"uuid":"8a45f350-fb3e-4049-bcc4-532d3e20c238","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":2821651815},{"type":"v1","addr":"172.21.15.195:6801","nonce":2821651815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":2821651815},{"type":"v1","addr":"172.21.15.195:6803","nonce":2821651815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":2821651815},{"type":"v1","addr":"172.21.15.195:6807","nonce":2821651815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":2821651815},{"type":"v1","addr":"172.21.15.195:6805","nonce":2821651815}]},"public_addr":"172.21.15.195:6801/2821651815","cluster_addr":"172.21.15.195:6803/2821651815","heartbeat_back_addr":"172.21.15.195:6807/2821651815","heartbeat_front_addr":"172.21.15.195:6805/2821651815","state":["exists","up"]},{"osd":5,"uuid":"363cf638-db26-4c86-b2be-e075fd09ac16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":3074139237},{"type":"v1","addr":"172.21.15.195:6809","nonce":3074139237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":3074139237},{"type":"v1","addr":"172.21.15.195:6811","nonce":3074139237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":3074139237},{"type":"v1","addr":"172.21.15.195:6815","nonce":3074139237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":3074139237},{"type":"v1","addr":"172.21.15.195:6813","nonce":3074139237}]},"public_addr":"172.21.15.195:6809/3074139237","cluster_addr":"172.21.15.195:6811/3074139237","heartbeat_back_addr":"172.21.15.195:6815/3074139237","heartbeat_front_addr":"172.21.15.195:6813/3074139237","state":["exists","up"]},{"osd":6,"uuid":"f37416e8-9c25-4860-a32f-0d1c404a8bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":4026627594},{"type":"v1","addr":"172.21.15.195:6817","nonce":4026627594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":4026627594},{"type":"v1","addr":"172.21.15.195:6819","nonce":4026627594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":4026627594},{"type":"v1","addr":"172.21.15.195:6823","nonce":4026627594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":4026627594},{"type":"v1","addr":"172.21.15.195:6821","nonce":4026627594}]},"public_addr":"172.21.15.195:6817/4026627594","cluster_addr":"172.21.15.195:6819/4026627594","heartbeat_back_addr":"172.21.15.195:6823/4026627594","heartbeat_front_addr":"172.21.15.195:6821/4026627594","state":["exists","up"]},{"osd":7,"uuid":"9a384a4d-fddb-4fb3-843a-77c19cca0368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2059403787},{"type":"v1","addr":"172.21.15.195:6825","nonce":2059403787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2059403787},{"type":"v1","addr":"172.21.15.195:6827","nonce":2059403787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2059403787},{"type":"v1","addr":"172.21.15.195:6831","nonce":2059403787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2059403787},{"type":"v1","addr":"172.21.15.195:6829","nonce":2059403787}]},"public_addr":"172.21.15.195:6825/2059403787","cluster_addr":"172.21.15.195:6827/2059403787","heartbeat_back_addr":"172.21.15.195:6831/2059403787","heartbeat_front_addr":"172.21.15.195:6829/2059403787","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:06:53.502746+0000","dead_epoch":90},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:07:16.251669+0000","dead_epoch":95},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:07:38.848272+0000","dead_epoch":100},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:08:02.013035+0000","dead_epoch":105},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:08:23.043482+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:08:45.653159+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:09:10.187018+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:09:34.393753+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/3514811128":"2024-09-14T14:17:25.102165+0000","172.21.15.102:6801/3856547061":"2024-09-14T14:17:25.102165+0000","172.21.15.102:6800/3856547061":"2024-09-14T14:17:25.102165+0000","172.21.15.102:0/2872102326":"2024-09-14T14:17:25.102165+0000","172.21.15.102:0/3399546099":"2024-09-14T14:05:37.634624+0000","172.21.15.102:0/1430433627":"2024-09-14T14:14:08.048329+0000","172.21.15.102:0/2489373272":"2024-09-14T14:14:08.048329+0000","172.21.15.195:6835/883376241":"2024-09-14T14:14:48.059120+0000","172.21.15.102:6801/2761043787":"2024-09-14T14:05:15.286479+0000","172.21.15.102:0/1574792592":"2024-09-14T14:05:15.286479+0000","172.21.15.102:6801/63514412":"2024-09-14T14:10:09.037657+0000","172.21.15.102:6800/63514412":"2024-09-14T14:10:09.037657+0000","172.21.15.102:0/2083488703":"2024-09-14T14:14:08.048329+0000","172.21.15.102:0/2975343136":"2024-09-14T14:17:25.102165+0000","172.21.15.195:0/2524944732":"2024-09-14T14:14:48.059120+0000","172.21.15.102:6800/2761043787":"2024-09-14T14:05:15.286479+0000","172.21.15.102:6801/3480998245":"2024-09-14T14:05:37.634624+0000","172.21.15.102:6800/3480998245":"2024-09-14T14:05:37.634624+0000","172.21.15.195:0/1568198689":"2024-09-14T14:14:48.059120+0000","172.21.15.195:6834/883376241":"2024-09-14T14:14:48.059120+0000","172.21.15.102:0/1792178465":"2024-09-14T14:14:08.048329+0000","172.21.15.102:0/120875803":"2024-09-14T14:17:25.102165+0000","172.21.15.102:0/1160237252":"2024-09-14T14:10:09.037657+0000","172.21.15.102:0/1582001159":"2024-09-14T14:05:15.286479+0000","172.21.15.195:0/4292730726":"2024-09-14T14:14:48.059120+0000","172.21.15.102:6800/1522897266":"2024-09-14T14:14:08.048329+0000","172.21.15.102:0/1939570054":"2024-09-14T14:05:37.634624+0000","172.21.15.102:0/3189088157":"2024-09-14T14:05:15.286479+0000","172.21.15.195:0/3629133224":"2024-09-14T14:14:48.059120+0000","172.21.15.102:0/2308924552":"2024-09-14T14:10:09.037657+0000","172.21.15.102:0/2416257718":"2024-09-14T14:10:09.037657+0000","172.21.15.102:6801/1522897266":"2024-09-14T14:14:08.048329+0000","172.21.15.102:0/1765904351":"2024-09-14T14:10:09.037657+0000","172.21.15.102:0/2444394840":"2024-09-14T14:05:37.634624+0000","172.21.15.195:0/1729595567":"2024-09-14T14:14:48.059120+0000","172.21.15.195:0/495121375":"2024-09-14T14:14:48.059120+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T14:25:14.768 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:14 smithi102 ceph-mon[110183]: pgmap v257: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:14.769 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:14 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/114906544' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T14:25:14.769 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:14 smithi102 ceph-mon[115269]: pgmap v257: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:14.769 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:14 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/114906544' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T14:25:14.794 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-13T14:25:14.794 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-13T14:25:14.794 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-13T14:25:14.795 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-13T14:25:14.795 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-13T14:25:14.796 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-13T14:25:14.796 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-13T14:25:14.796 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-13T14:25:14.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:14 smithi195 ceph-mon[104262]: pgmap v257: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:14.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:14 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/114906544' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T14:25:15.222 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:25:15.227 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:25:15.227 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:25:15.229 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:25:15.232 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:25:15.233 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:25:15.233 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:25:15.234 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:25:16.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:16 smithi102 ceph-mon[110183]: pgmap v258: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:16.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:16 smithi102 ceph-mon[115269]: pgmap v258: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:16.669 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:25:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:25:16] "GET /metrics HTTP/1.1" 200 34011 "" "Prometheus/2.51.0" 2024-09-13T14:25:16.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:16 smithi195 ceph-mon[104262]: pgmap v258: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:18.820 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:18 smithi102 ceph-mon[110183]: pgmap v259: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:18.821 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:18 smithi102 ceph-mon[115269]: pgmap v259: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:18.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:18 smithi195 ceph-mon[104262]: pgmap v259: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:20.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:20 smithi102 ceph-mon[110183]: pgmap v260: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:20.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:20 smithi102 ceph-mon[115269]: pgmap v260: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:20.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:20 smithi195 ceph-mon[104262]: pgmap v260: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:22.817 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:22 smithi102 ceph-mon[110183]: pgmap v261: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:22.817 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:22 smithi102 ceph-mon[115269]: pgmap v261: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:22.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:22 smithi195 ceph-mon[104262]: pgmap v261: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:24.808 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:24 smithi102 ceph-mon[110183]: pgmap v262: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:24.809 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:24 smithi102 ceph-mon[115269]: pgmap v262: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:24.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:24 smithi195 ceph-mon[104262]: pgmap v262: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:25.208 INFO:teuthology.orchestra.run.smithi102.stdout:163208757451 2024-09-13T14:25:25.208 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-13T14:25:25.809 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:25 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:25:25.810 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:25 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:25:25.898 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:25 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:25:26.726 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:26 smithi102 ceph-mon[110183]: pgmap v263: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:26.726 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:26 smithi102 ceph-mon[115269]: pgmap v263: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:26.726 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:25:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:25:26] "GET /metrics HTTP/1.1" 200 34015 "" "Prometheus/2.51.0" 2024-09-13T14:25:26.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:26 smithi195 ceph-mon[104262]: pgmap v263: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:27.038 INFO:teuthology.orchestra.run.smithi102.stdout:425201762325 2024-09-13T14:25:27.039 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-13T14:25:28.046 INFO:teuthology.orchestra.run.smithi102.stdout:468151435272 2024-09-13T14:25:28.047 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-13T14:25:28.498 INFO:teuthology.orchestra.run.smithi102.stdout:197568495814 2024-09-13T14:25:28.498 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-13T14:25:28.780 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:28 smithi102 ceph-mon[110183]: pgmap v264: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:28.780 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:28 smithi102 ceph-mon[115269]: pgmap v264: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:28.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:28 smithi195 ceph-mon[104262]: pgmap v264: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:29.641 INFO:teuthology.orchestra.run.smithi102.stdout:128849019087 2024-09-13T14:25:29.641 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-13T14:25:30.594 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:30 smithi102 ceph-mon[110183]: pgmap v265: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:30.594 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:30 smithi102 ceph-mon[115269]: pgmap v265: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:30.876 INFO:teuthology.orchestra.run.smithi102.stdout:231928234179 2024-09-13T14:25:30.877 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-13T14:25:30.903 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:25:30.904 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:25:30.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:30 smithi195 ceph-mon[104262]: pgmap v265: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:31.303 INFO:teuthology.orchestra.run.smithi102.stdout:446676598799 2024-09-13T14:25:31.303 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-13T14:25:32.322 INFO:teuthology.orchestra.run.smithi102.stdout:403726925852 2024-09-13T14:25:32.323 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-13T14:25:32.586 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:25:32.589 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:32 smithi102 ceph-mon[110183]: pgmap v266: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:32.589 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:32 smithi102 ceph-mon[115269]: pgmap v266: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:32.613 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:25:32.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:32 smithi195 ceph-mon[104262]: pgmap v266: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:34.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:34 smithi102 ceph-mon[110183]: pgmap v267: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:34.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:34 smithi102 ceph-mon[115269]: pgmap v267: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:34.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:34 smithi195 ceph-mon[104262]: pgmap v267: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:35.533 INFO:teuthology.orchestra.run.smithi102.stdout:425201762326 2024-09-13T14:25:35.773 INFO:teuthology.orchestra.run.smithi102.stdout:163208757453 2024-09-13T14:25:35.774 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:35 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2759381323' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-13T14:25:35.774 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:35 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2759381323' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-13T14:25:35.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:35 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2759381323' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-13T14:25:36.547 INFO:tasks.cephadm.ceph_manager.ceph:need seq 425201762325 got 425201762326 for osd.1 2024-09-13T14:25:36.548 DEBUG:teuthology.parallel:result is None 2024-09-13T14:25:36.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:36 smithi102 ceph-mon[115269]: pgmap v268: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:36.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:36 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2723189911' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-13T14:25:36.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:36 smithi102 ceph-mon[110183]: pgmap v268: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:36.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:36 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2723189911' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-13T14:25:36.919 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:25:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:25:36] "GET /metrics HTTP/1.1" 200 34015 "" "Prometheus/2.51.0" 2024-09-13T14:25:36.979 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757451 got 163208757453 for osd.5 2024-09-13T14:25:36.979 DEBUG:teuthology.parallel:result is None 2024-09-13T14:25:36.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:36 smithi195 ceph-mon[104262]: pgmap v268: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:36.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:36 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2723189911' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-13T14:25:37.310 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:25:37.311 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:25:37.311 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:25:38.860 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:38 smithi102 ceph-mon[110183]: pgmap v269: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:38.861 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:38 smithi102 ceph-mon[115269]: pgmap v269: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:38.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:38 smithi195 ceph-mon[104262]: pgmap v269: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:39.012 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:25:39.134 INFO:teuthology.orchestra.run.smithi102.stdout:197568495817 2024-09-13T14:25:39.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:39 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/550712813' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-13T14:25:39.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:39 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/550712813' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-13T14:25:39.955 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495814 got 197568495817 for osd.6 2024-09-13T14:25:39.955 DEBUG:teuthology.parallel:result is None 2024-09-13T14:25:39.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:39 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/550712813' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-13T14:25:40.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:40 smithi102 ceph-mon[110183]: pgmap v270: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:40.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:40 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:25:40.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:40 smithi102 ceph-mon[115269]: pgmap v270: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:40.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:40 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:25:40.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:40 smithi195 ceph-mon[104262]: pgmap v270: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:40.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:40 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:25:42.837 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:42 smithi102 ceph-mon[110183]: pgmap v271: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:42.837 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:42 smithi102 ceph-mon[115269]: pgmap v271: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:42.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:42 smithi195 ceph-mon[104262]: pgmap v271: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:43.218 INFO:teuthology.orchestra.run.smithi102.stdout:231928234182 2024-09-13T14:25:43.797 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:43 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/4087588077' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-13T14:25:43.797 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:43 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/4087588077' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-13T14:25:43.982 INFO:teuthology.orchestra.run.smithi102.stdout:128849019091 2024-09-13T14:25:43.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:43 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/4087588077' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-13T14:25:44.548 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849019087 got 128849019091 for osd.4 2024-09-13T14:25:44.548 DEBUG:teuthology.parallel:result is None 2024-09-13T14:25:44.880 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:44 smithi102 ceph-mon[110183]: pgmap v272: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:44.880 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:44 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/857315587' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-13T14:25:44.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:44 smithi102 ceph-mon[115269]: pgmap v272: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:44.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:44 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/857315587' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-13T14:25:44.941 INFO:teuthology.orchestra.run.smithi102.stdout:446676598802 2024-09-13T14:25:44.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:44 smithi195 ceph-mon[104262]: pgmap v272: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:44.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:44 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/857315587' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-13T14:25:45.160 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234179 got 231928234182 for osd.7 2024-09-13T14:25:45.160 DEBUG:teuthology.parallel:result is None 2024-09-13T14:25:45.718 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:45 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/153561063' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-13T14:25:45.718 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:45 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/153561063' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-13T14:25:45.739 INFO:tasks.cephadm.ceph_manager.ceph:need seq 446676598799 got 446676598802 for osd.2 2024-09-13T14:25:45.740 DEBUG:teuthology.parallel:result is None 2024-09-13T14:25:45.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:45 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/153561063' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-13T14:25:46.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:46 smithi102 ceph-mon[110183]: pgmap v273: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:46.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:46 smithi102 ceph-mon[115269]: pgmap v273: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:46.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:25:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:25:46] "GET /metrics HTTP/1.1" 200 34014 "" "Prometheus/2.51.0" 2024-09-13T14:25:46.851 INFO:teuthology.orchestra.run.smithi102.stdout:468151435276 2024-09-13T14:25:46.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:46 smithi195 ceph-mon[104262]: pgmap v273: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:47.409 INFO:tasks.cephadm.ceph_manager.ceph:need seq 468151435272 got 468151435276 for osd.3 2024-09-13T14:25:47.409 DEBUG:teuthology.parallel:result is None 2024-09-13T14:25:47.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:47 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1601871921' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-13T14:25:47.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:47 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1601871921' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-13T14:25:47.751 INFO:teuthology.orchestra.run.smithi102.stdout:403726925856 2024-09-13T14:25:47.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:47 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1601871921' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-13T14:25:48.341 INFO:tasks.cephadm.ceph_manager.ceph:need seq 403726925852 got 403726925856 for osd.0 2024-09-13T14:25:48.342 DEBUG:teuthology.parallel:result is None 2024-09-13T14:25:48.342 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-13T14:25:48.342 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:25:48.542 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:25:48.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:48 smithi102 ceph-mon[115269]: pgmap v274: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:48.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:48 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/294806588' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-13T14:25:48.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:48 smithi102 ceph-mon[110183]: pgmap v274: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:48.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:48 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/294806588' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-13T14:25:48.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:48 smithi195 ceph-mon[104262]: pgmap v274: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:48.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:48 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/294806588' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-13T14:25:50.531 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:25:50.531 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:25:50.883 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:50 smithi102 ceph-mon[110183]: pgmap v275: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:50.884 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:50 smithi102 ceph-mon[115269]: pgmap v275: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:50.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:50 smithi195 ceph-mon[104262]: pgmap v275: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:51.189 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":275,"stamp":"2024-09-13T14:25:49.233102+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790245,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":6390,"num_read_kb":13845,"num_write":1123,"num_write_kb":15041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1182,"ondisk_log_size":1182,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":254108,"kb_used_data":23588,"kb_used_omap":35,"kb_used_meta":230364,"kb_avail":749707108,"statfs":{"total":767960285184,"available":767700078592,"internally_reserved":0,"allocated":24154112,"data_stored":19392558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36053,"internal_metadata":235893547},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002364"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":74,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.290727+0000","last_change":"2024-09-13T14:24:55.290644+0000","last_active":"2024-09-13T14:24:55.290727+0000","last_peered":"2024-09-13T14:24:55.290727+0000","last_clean":"2024-09-13T14:24:55.290727+0000","last_became_active":"2024-09-13T14:24:55.290222+0000","last_became_peered":"2024-09-13T14:24:55.290222+0000","last_unstale":"2024-09-13T14:24:55.290727+0000","last_undegraded":"2024-09-13T14:24:55.290727+0000","last_fullsized":"2024-09-13T14:24:55.290727+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:38:41.205619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":75,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.274419+0000","last_change":"2024-09-13T14:24:23.773702+0000","last_active":"2024-09-13T14:24:55.274419+0000","last_peered":"2024-09-13T14:24:55.274419+0000","last_clean":"2024-09-13T14:24:55.274419+0000","last_became_active":"2024-09-13T14:24:23.773290+0000","last_became_peered":"2024-09-13T14:24:23.773290+0000","last_unstale":"2024-09-13T14:24:55.274419+0000","last_undegraded":"2024-09-13T14:24:55.274419+0000","last_fullsized":"2024-09-13T14:24:55.274419+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:20.820970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":71,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.282407+0000","last_change":"2024-09-13T14:24:55.282407+0000","last_active":"2024-09-13T14:24:55.282407+0000","last_peered":"2024-09-13T14:24:55.282407+0000","last_clean":"2024-09-13T14:24:55.282407+0000","last_became_active":"2024-09-13T14:24:55.282083+0000","last_became_peered":"2024-09-13T14:24:55.282083+0000","last_unstale":"2024-09-13T14:24:55.282407+0000","last_undegraded":"2024-09-13T14:24:55.282407+0000","last_fullsized":"2024-09-13T14:24:55.282407+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:23:10.525707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":65,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.284816+0000","last_change":"2024-09-13T14:24:55.284433+0000","last_active":"2024-09-13T14:24:55.284816+0000","last_peered":"2024-09-13T14:24:55.284816+0000","last_clean":"2024-09-13T14:24:55.284816+0000","last_became_active":"2024-09-13T14:24:55.284143+0000","last_became_peered":"2024-09-13T14:24:55.284143+0000","last_unstale":"2024-09-13T14:24:55.284816+0000","last_undegraded":"2024-09-13T14:24:55.284816+0000","last_fullsized":"2024-09-13T14:24:55.284816+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:28.107405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":67,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.282846+0000","last_change":"2024-09-13T14:24:55.282416+0000","last_active":"2024-09-13T14:24:55.282846+0000","last_peered":"2024-09-13T14:24:55.282846+0000","last_clean":"2024-09-13T14:24:55.282846+0000","last_became_active":"2024-09-13T14:24:55.282045+0000","last_became_peered":"2024-09-13T14:24:55.282045+0000","last_unstale":"2024-09-13T14:24:55.282846+0000","last_undegraded":"2024-09-13T14:24:55.282846+0000","last_fullsized":"2024-09-13T14:24:55.282846+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:58.692897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":58,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.274055+0000","last_change":"2024-09-13T14:11:29.244452+0000","last_active":"2024-09-13T14:24:55.274055+0000","last_peered":"2024-09-13T14:24:55.274055+0000","last_clean":"2024-09-13T14:24:55.274055+0000","last_became_active":"2024-09-13T14:11:29.244102+0000","last_became_peered":"2024-09-13T14:11:29.244102+0000","last_unstale":"2024-09-13T14:24:55.274055+0000","last_undegraded":"2024-09-13T14:24:55.274055+0000","last_fullsized":"2024-09-13T14:24:55.274055+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:13:19.165534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":74,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.289698+0000","last_change":"2024-09-13T14:24:55.288871+0000","last_active":"2024-09-13T14:24:55.289698+0000","last_peered":"2024-09-13T14:24:55.289698+0000","last_clean":"2024-09-13T14:24:55.289698+0000","last_became_active":"2024-09-13T14:24:55.288506+0000","last_became_peered":"2024-09-13T14:24:55.288506+0000","last_unstale":"2024-09-13T14:24:55.289698+0000","last_undegraded":"2024-09-13T14:24:55.289698+0000","last_fullsized":"2024-09-13T14:24:55.289698+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:44.796657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":81,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.283649+0000","last_change":"2024-09-13T14:23:18.039061+0000","last_active":"2024-09-13T14:24:55.283649+0000","last_peered":"2024-09-13T14:24:55.283649+0000","last_clean":"2024-09-13T14:24:55.283649+0000","last_became_active":"2024-09-13T14:23:18.038661+0000","last_became_peered":"2024-09-13T14:23:18.038661+0000","last_unstale":"2024-09-13T14:24:55.283649+0000","last_undegraded":"2024-09-13T14:24:55.283649+0000","last_fullsized":"2024-09-13T14:24:55.283649+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:05:29.109844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":74,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.275099+0000","last_change":"2024-09-13T14:23:18.038597+0000","last_active":"2024-09-13T14:24:55.275099+0000","last_peered":"2024-09-13T14:24:55.275099+0000","last_clean":"2024-09-13T14:24:55.275099+0000","last_became_active":"2024-09-13T14:23:18.038240+0000","last_became_peered":"2024-09-13T14:23:18.038240+0000","last_unstale":"2024-09-13T14:24:55.275099+0000","last_undegraded":"2024-09-13T14:24:55.275099+0000","last_fullsized":"2024-09-13T14:24:55.275099+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:27:51.842195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":57,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.269670+0000","last_change":"2024-09-13T14:11:30.254534+0000","last_active":"2024-09-13T14:24:54.269670+0000","last_peered":"2024-09-13T14:24:54.269670+0000","last_clean":"2024-09-13T14:24:54.269670+0000","last_became_active":"2024-09-13T14:11:30.253959+0000","last_became_peered":"2024-09-13T14:11:30.253959+0000","last_unstale":"2024-09-13T14:24:54.269670+0000","last_undegraded":"2024-09-13T14:24:54.269670+0000","last_fullsized":"2024-09-13T14:24:54.269670+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:09:54.025853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":75,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.274762+0000","last_change":"2024-09-13T14:23:51.612105+0000","last_active":"2024-09-13T14:24:55.274762+0000","last_peered":"2024-09-13T14:24:55.274762+0000","last_clean":"2024-09-13T14:24:55.274762+0000","last_became_active":"2024-09-13T14:23:51.611740+0000","last_became_peered":"2024-09-13T14:23:51.611740+0000","last_unstale":"2024-09-13T14:24:55.274762+0000","last_undegraded":"2024-09-13T14:24:55.274762+0000","last_fullsized":"2024-09-13T14:24:55.274762+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:26:23.262096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":86,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.288950+0000","last_change":"2024-09-13T14:24:55.288950+0000","last_active":"2024-09-13T14:24:55.288950+0000","last_peered":"2024-09-13T14:24:55.288950+0000","last_clean":"2024-09-13T14:24:55.288950+0000","last_became_active":"2024-09-13T14:24:55.288671+0000","last_became_peered":"2024-09-13T14:24:55.288671+0000","last_unstale":"2024-09-13T14:24:55.288950+0000","last_undegraded":"2024-09-13T14:24:55.288950+0000","last_fullsized":"2024-09-13T14:24:55.288950+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:14:53.298390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":59,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.275139+0000","last_change":"2024-09-13T14:11:29.232897+0000","last_active":"2024-09-13T14:24:55.275139+0000","last_peered":"2024-09-13T14:24:55.275139+0000","last_clean":"2024-09-13T14:24:55.275139+0000","last_became_active":"2024-09-13T14:11:29.232637+0000","last_became_peered":"2024-09-13T14:11:29.232637+0000","last_unstale":"2024-09-13T14:24:55.275139+0000","last_undegraded":"2024-09-13T14:24:55.275139+0000","last_fullsized":"2024-09-13T14:24:55.275139+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:31:05.856046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":72,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.276706+0000","last_change":"2024-09-13T14:23:51.609167+0000","last_active":"2024-09-13T14:24:54.276706+0000","last_peered":"2024-09-13T14:24:54.276706+0000","last_clean":"2024-09-13T14:24:54.276706+0000","last_became_active":"2024-09-13T14:23:51.608811+0000","last_became_peered":"2024-09-13T14:23:51.608811+0000","last_unstale":"2024-09-13T14:24:54.276706+0000","last_undegraded":"2024-09-13T14:24:54.276706+0000","last_fullsized":"2024-09-13T14:24:54.276706+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:40.506763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":73,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.291355+0000","last_change":"2024-09-13T14:24:55.290798+0000","last_active":"2024-09-13T14:24:55.291355+0000","last_peered":"2024-09-13T14:24:55.291355+0000","last_clean":"2024-09-13T14:24:55.291355+0000","last_became_active":"2024-09-13T14:24:55.290510+0000","last_became_peered":"2024-09-13T14:24:55.290510+0000","last_unstale":"2024-09-13T14:24:55.291355+0000","last_undegraded":"2024-09-13T14:24:55.291355+0000","last_fullsized":"2024-09-13T14:24:55.291355+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:19:03.129132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":67,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.271982+0000","last_change":"2024-09-13T14:23:51.609082+0000","last_active":"2024-09-13T14:24:55.271982+0000","last_peered":"2024-09-13T14:24:55.271982+0000","last_clean":"2024-09-13T14:24:55.271982+0000","last_became_active":"2024-09-13T14:23:51.608799+0000","last_became_peered":"2024-09-13T14:23:51.608799+0000","last_unstale":"2024-09-13T14:24:55.271982+0000","last_undegraded":"2024-09-13T14:24:55.271982+0000","last_fullsized":"2024-09-13T14:24:55.271982+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:43:24.721453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":72,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.284590+0000","last_change":"2024-09-13T14:24:55.284499+0000","last_active":"2024-09-13T14:24:55.284590+0000","last_peered":"2024-09-13T14:24:55.284590+0000","last_clean":"2024-09-13T14:24:55.284590+0000","last_became_active":"2024-09-13T14:24:55.284096+0000","last_became_peered":"2024-09-13T14:24:55.284096+0000","last_unstale":"2024-09-13T14:24:55.284590+0000","last_undegraded":"2024-09-13T14:24:55.284590+0000","last_fullsized":"2024-09-13T14:24:55.284590+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:02:06.836099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":71,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.291592+0000","last_change":"2024-09-13T14:24:55.291371+0000","last_active":"2024-09-13T14:24:55.291592+0000","last_peered":"2024-09-13T14:24:55.291592+0000","last_clean":"2024-09-13T14:24:55.291592+0000","last_became_active":"2024-09-13T14:24:55.291047+0000","last_became_peered":"2024-09-13T14:24:55.291047+0000","last_unstale":"2024-09-13T14:24:55.291592+0000","last_undegraded":"2024-09-13T14:24:55.291592+0000","last_fullsized":"2024-09-13T14:24:55.291592+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:43.912412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":73,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.276868+0000","last_change":"2024-09-13T14:23:51.606589+0000","last_active":"2024-09-13T14:24:54.276868+0000","last_peered":"2024-09-13T14:24:54.276868+0000","last_clean":"2024-09-13T14:24:54.276868+0000","last_became_active":"2024-09-13T14:23:51.606244+0000","last_became_peered":"2024-09-13T14:23:51.606244+0000","last_unstale":"2024-09-13T14:24:54.276868+0000","last_undegraded":"2024-09-13T14:24:54.276868+0000","last_fullsized":"2024-09-13T14:24:54.276868+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:08.776526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":86,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.275537+0000","last_change":"2024-09-13T14:24:23.777891+0000","last_active":"2024-09-13T14:24:55.275537+0000","last_peered":"2024-09-13T14:24:55.275537+0000","last_clean":"2024-09-13T14:24:55.275537+0000","last_became_active":"2024-09-13T14:24:23.777578+0000","last_became_peered":"2024-09-13T14:24:23.777578+0000","last_unstale":"2024-09-13T14:24:55.275537+0000","last_undegraded":"2024-09-13T14:24:55.275537+0000","last_fullsized":"2024-09-13T14:24:55.275537+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:56:47.441579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":72,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.270580+0000","last_change":"2024-09-13T14:24:23.777970+0000","last_active":"2024-09-13T14:24:54.270580+0000","last_peered":"2024-09-13T14:24:54.270580+0000","last_clean":"2024-09-13T14:24:54.270580+0000","last_became_active":"2024-09-13T14:24:23.777650+0000","last_became_peered":"2024-09-13T14:24:23.777650+0000","last_unstale":"2024-09-13T14:24:54.270580+0000","last_undegraded":"2024-09-13T14:24:54.270580+0000","last_fullsized":"2024-09-13T14:24:54.270580+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:50:00.549905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":71,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.274746+0000","last_change":"2024-09-13T14:23:51.610572+0000","last_active":"2024-09-13T14:24:55.274746+0000","last_peered":"2024-09-13T14:24:55.274746+0000","last_clean":"2024-09-13T14:24:55.274746+0000","last_became_active":"2024-09-13T14:23:51.610273+0000","last_became_peered":"2024-09-13T14:23:51.610273+0000","last_unstale":"2024-09-13T14:24:55.274746+0000","last_undegraded":"2024-09-13T14:24:55.274746+0000","last_fullsized":"2024-09-13T14:24:55.274746+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:17:10.574417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":75,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.281133+0000","last_change":"2024-09-13T14:24:55.281042+0000","last_active":"2024-09-13T14:24:55.281133+0000","last_peered":"2024-09-13T14:24:55.281133+0000","last_clean":"2024-09-13T14:24:55.281133+0000","last_became_active":"2024-09-13T14:24:55.280741+0000","last_became_peered":"2024-09-13T14:24:55.280741+0000","last_unstale":"2024-09-13T14:24:55.281133+0000","last_undegraded":"2024-09-13T14:24:55.281133+0000","last_fullsized":"2024-09-13T14:24:55.281133+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:45:40.787103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":46,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.289331+0000","last_change":"2024-09-13T14:24:55.289245+0000","last_active":"2024-09-13T14:24:55.289331+0000","last_peered":"2024-09-13T14:24:55.289331+0000","last_clean":"2024-09-13T14:24:55.289331+0000","last_became_active":"2024-09-13T14:24:55.288881+0000","last_became_peered":"2024-09-13T14:24:55.288881+0000","last_unstale":"2024-09-13T14:24:55.289331+0000","last_undegraded":"2024-09-13T14:24:55.289331+0000","last_fullsized":"2024-09-13T14:24:55.289331+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:14:12.194309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":403,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.275493+0000","last_change":"2024-09-13T14:23:18.032973+0000","last_active":"2024-09-13T14:24:54.275493+0000","last_peered":"2024-09-13T14:24:54.275493+0000","last_clean":"2024-09-13T14:24:54.275493+0000","last_became_active":"2024-09-13T14:23:18.032579+0000","last_became_peered":"2024-09-13T14:23:18.032579+0000","last_unstale":"2024-09-13T14:24:54.275493+0000","last_undegraded":"2024-09-13T14:24:54.275493+0000","last_fullsized":"2024-09-13T14:24:54.275493+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:07:41.927740+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:17.109445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":400,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.270769+0000","last_change":"2024-09-13T14:24:23.776868+0000","last_active":"2024-09-13T14:24:54.270769+0000","last_peered":"2024-09-13T14:24:54.270769+0000","last_clean":"2024-09-13T14:24:54.270769+0000","last_became_active":"2024-09-13T14:24:23.776306+0000","last_became_peered":"2024-09-13T14:24:23.776306+0000","last_unstale":"2024-09-13T14:24:54.270769+0000","last_undegraded":"2024-09-13T14:24:54.270769+0000","last_fullsized":"2024-09-13T14:24:54.270769+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:28:41.805918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":23,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.288138+0000","last_change":"2024-09-13T14:24:55.288058+0000","last_active":"2024-09-13T14:24:55.288138+0000","last_peered":"2024-09-13T14:24:55.288138+0000","last_clean":"2024-09-13T14:24:55.288138+0000","last_became_active":"2024-09-13T14:24:55.287601+0000","last_became_peered":"2024-09-13T14:24:55.287601+0000","last_unstale":"2024-09-13T14:24:55.288138+0000","last_undegraded":"2024-09-13T14:24:55.288138+0000","last_fullsized":"2024-09-13T14:24:55.288138+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:56:34.622696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":57,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.277982+0000","last_change":"2024-09-13T14:11:30.253749+0000","last_active":"2024-09-13T14:24:54.277982+0000","last_peered":"2024-09-13T14:24:54.277982+0000","last_clean":"2024-09-13T14:24:54.277982+0000","last_became_active":"2024-09-13T14:11:30.253499+0000","last_became_peered":"2024-09-13T14:11:30.253499+0000","last_unstale":"2024-09-13T14:24:54.277982+0000","last_undegraded":"2024-09-13T14:24:54.277982+0000","last_fullsized":"2024-09-13T14:24:54.277982+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:02:00.987335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":289,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.275098+0000","last_change":"2024-09-13T14:23:51.608681+0000","last_active":"2024-09-13T14:24:54.275098+0000","last_peered":"2024-09-13T14:24:54.275098+0000","last_clean":"2024-09-13T14:24:54.275098+0000","last_became_active":"2024-09-13T14:23:51.608223+0000","last_became_peered":"2024-09-13T14:23:51.608223+0000","last_unstale":"2024-09-13T14:24:54.275098+0000","last_undegraded":"2024-09-13T14:24:54.275098+0000","last_fullsized":"2024-09-13T14:24:54.275098+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:13:15.930987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.286928+0000","last_change":"2024-09-13T14:24:55.286576+0000","last_active":"2024-09-13T14:24:55.286928+0000","last_peered":"2024-09-13T14:24:55.286928+0000","last_clean":"2024-09-13T14:24:55.286928+0000","last_became_active":"2024-09-13T14:24:55.286284+0000","last_became_peered":"2024-09-13T14:24:55.286284+0000","last_unstale":"2024-09-13T14:24:55.286928+0000","last_undegraded":"2024-09-13T14:24:55.286928+0000","last_fullsized":"2024-09-13T14:24:55.286928+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:35.589736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":78,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.271821+0000","last_change":"2024-09-13T14:24:23.776557+0000","last_active":"2024-09-13T14:24:55.271821+0000","last_peered":"2024-09-13T14:24:55.271821+0000","last_clean":"2024-09-13T14:24:55.271821+0000","last_became_active":"2024-09-13T14:24:23.775204+0000","last_became_peered":"2024-09-13T14:24:23.775204+0000","last_unstale":"2024-09-13T14:24:55.271821+0000","last_undegraded":"2024-09-13T14:24:55.271821+0000","last_fullsized":"2024-09-13T14:24:55.271821+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:18:21.095561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":494,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.285398+0000","last_change":"2024-09-13T14:24:55.285288+0000","last_active":"2024-09-13T14:24:55.285398+0000","last_peered":"2024-09-13T14:24:55.285398+0000","last_clean":"2024-09-13T14:24:55.285398+0000","last_became_active":"2024-09-13T14:24:55.284733+0000","last_became_peered":"2024-09-13T14:24:55.284733+0000","last_unstale":"2024-09-13T14:24:55.285398+0000","last_undegraded":"2024-09-13T14:24:55.285398+0000","last_fullsized":"2024-09-13T14:24:55.285398+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:51:50.420796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":55,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.276363+0000","last_change":"2024-09-13T14:24:23.779725+0000","last_active":"2024-09-13T14:24:55.276363+0000","last_peered":"2024-09-13T14:24:55.276363+0000","last_clean":"2024-09-13T14:24:55.276363+0000","last_became_active":"2024-09-13T14:24:23.779418+0000","last_became_peered":"2024-09-13T14:24:23.779418+0000","last_unstale":"2024-09-13T14:24:55.276363+0000","last_undegraded":"2024-09-13T14:24:55.276363+0000","last_fullsized":"2024-09-13T14:24:55.276363+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:48:08.470127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":89,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.276842+0000","last_change":"2024-09-13T14:23:51.608576+0000","last_active":"2024-09-13T14:24:55.276842+0000","last_peered":"2024-09-13T14:24:55.276842+0000","last_clean":"2024-09-13T14:24:55.276842+0000","last_became_active":"2024-09-13T14:23:51.607956+0000","last_became_peered":"2024-09-13T14:23:51.607956+0000","last_unstale":"2024-09-13T14:24:55.276842+0000","last_undegraded":"2024-09-13T14:24:55.276842+0000","last_fullsized":"2024-09-13T14:24:55.276842+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:53.399552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":829,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.272095+0000","last_change":"2024-09-13T14:23:51.606335+0000","last_active":"2024-09-13T14:24:54.272095+0000","last_peered":"2024-09-13T14:24:54.272095+0000","last_clean":"2024-09-13T14:24:54.272095+0000","last_became_active":"2024-09-13T14:23:51.605905+0000","last_became_peered":"2024-09-13T14:23:51.605905+0000","last_unstale":"2024-09-13T14:24:54.272095+0000","last_undegraded":"2024-09-13T14:24:54.272095+0000","last_fullsized":"2024-09-13T14:24:54.272095+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:22:42.470188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":72,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.272867+0000","last_change":"2024-09-13T14:23:18.042891+0000","last_active":"2024-09-13T14:24:54.272867+0000","last_peered":"2024-09-13T14:24:54.272867+0000","last_clean":"2024-09-13T14:24:54.272867+0000","last_became_active":"2024-09-13T14:23:18.042562+0000","last_became_peered":"2024-09-13T14:23:18.042562+0000","last_unstale":"2024-09-13T14:24:54.272867+0000","last_undegraded":"2024-09-13T14:24:54.272867+0000","last_fullsized":"2024-09-13T14:24:54.272867+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:30:42.996661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":84,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.291080+0000","last_change":"2024-09-13T14:24:55.291080+0000","last_active":"2024-09-13T14:24:55.291080+0000","last_peered":"2024-09-13T14:24:55.291080+0000","last_clean":"2024-09-13T14:24:55.291080+0000","last_became_active":"2024-09-13T14:24:55.290560+0000","last_became_peered":"2024-09-13T14:24:55.290560+0000","last_unstale":"2024-09-13T14:24:55.291080+0000","last_undegraded":"2024-09-13T14:24:55.291080+0000","last_fullsized":"2024-09-13T14:24:55.291080+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:16.041382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":88,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.282619+0000","last_change":"2024-09-13T14:24:55.282619+0000","last_active":"2024-09-13T14:24:55.282619+0000","last_peered":"2024-09-13T14:24:55.282619+0000","last_clean":"2024-09-13T14:24:55.282619+0000","last_became_active":"2024-09-13T14:24:55.282110+0000","last_became_peered":"2024-09-13T14:24:55.282110+0000","last_unstale":"2024-09-13T14:24:55.282619+0000","last_undegraded":"2024-09-13T14:24:55.282619+0000","last_fullsized":"2024-09-13T14:24:55.282619+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:33:19.425344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":777,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.277056+0000","last_change":"2024-09-13T14:23:51.610302+0000","last_active":"2024-09-13T14:24:55.277056+0000","last_peered":"2024-09-13T14:24:55.277056+0000","last_clean":"2024-09-13T14:24:55.277056+0000","last_became_active":"2024-09-13T14:23:51.607653+0000","last_became_peered":"2024-09-13T14:23:51.607653+0000","last_unstale":"2024-09-13T14:24:55.277056+0000","last_undegraded":"2024-09-13T14:24:55.277056+0000","last_fullsized":"2024-09-13T14:24:55.277056+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:03:52.131651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":45,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.275826+0000","last_change":"2024-09-13T14:23:51.610321+0000","last_active":"2024-09-13T14:24:55.275826+0000","last_peered":"2024-09-13T14:24:55.275826+0000","last_clean":"2024-09-13T14:24:55.275826+0000","last_became_active":"2024-09-13T14:23:51.607733+0000","last_became_peered":"2024-09-13T14:23:51.607733+0000","last_unstale":"2024-09-13T14:24:55.275826+0000","last_undegraded":"2024-09-13T14:24:55.275826+0000","last_fullsized":"2024-09-13T14:24:55.275826+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:58:05.246186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":54,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.276963+0000","last_change":"2024-09-13T14:24:23.779678+0000","last_active":"2024-09-13T14:24:55.276963+0000","last_peered":"2024-09-13T14:24:55.276963+0000","last_clean":"2024-09-13T14:24:55.276963+0000","last_became_active":"2024-09-13T14:24:23.779343+0000","last_became_peered":"2024-09-13T14:24:23.779343+0000","last_unstale":"2024-09-13T14:24:55.276963+0000","last_undegraded":"2024-09-13T14:24:55.276963+0000","last_fullsized":"2024-09-13T14:24:55.276963+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:14:41.230951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":595,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.275237+0000","last_change":"2024-09-13T14:23:18.038847+0000","last_active":"2024-09-13T14:24:55.275237+0000","last_peered":"2024-09-13T14:24:55.275237+0000","last_clean":"2024-09-13T14:24:55.275237+0000","last_became_active":"2024-09-13T14:23:18.038188+0000","last_became_peered":"2024-09-13T14:23:18.038188+0000","last_unstale":"2024-09-13T14:24:55.275237+0000","last_undegraded":"2024-09-13T14:24:55.275237+0000","last_fullsized":"2024-09-13T14:24:55.275237+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:30:37.213496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":56,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.271836+0000","last_change":"2024-09-13T14:23:18.034427+0000","last_active":"2024-09-13T14:24:55.271836+0000","last_peered":"2024-09-13T14:24:55.271836+0000","last_clean":"2024-09-13T14:24:55.271836+0000","last_became_active":"2024-09-13T14:23:18.034092+0000","last_became_peered":"2024-09-13T14:23:18.034092+0000","last_unstale":"2024-09-13T14:24:55.271836+0000","last_undegraded":"2024-09-13T14:24:55.271836+0000","last_fullsized":"2024-09-13T14:24:55.271836+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:21.755115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":72,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.290792+0000","last_change":"2024-09-13T14:24:55.290711+0000","last_active":"2024-09-13T14:24:55.290792+0000","last_peered":"2024-09-13T14:24:55.290792+0000","last_clean":"2024-09-13T14:24:55.290792+0000","last_became_active":"2024-09-13T14:24:55.288549+0000","last_became_peered":"2024-09-13T14:24:55.288549+0000","last_unstale":"2024-09-13T14:24:55.290792+0000","last_undegraded":"2024-09-13T14:24:55.290792+0000","last_fullsized":"2024-09-13T14:24:55.290792+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:55:21.763931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":502,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.276663+0000","last_change":"2024-09-13T14:23:51.606481+0000","last_active":"2024-09-13T14:24:55.276663+0000","last_peered":"2024-09-13T14:24:55.276663+0000","last_clean":"2024-09-13T14:24:55.276663+0000","last_became_active":"2024-09-13T14:23:51.605783+0000","last_became_peered":"2024-09-13T14:23:51.605783+0000","last_unstale":"2024-09-13T14:24:55.276663+0000","last_undegraded":"2024-09-13T14:24:55.276663+0000","last_fullsized":"2024-09-13T14:24:55.276663+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:09:01.265713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":69,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.281723+0000","last_change":"2024-09-13T14:24:55.281593+0000","last_active":"2024-09-13T14:24:55.281723+0000","last_peered":"2024-09-13T14:24:55.281723+0000","last_clean":"2024-09-13T14:24:55.281723+0000","last_became_active":"2024-09-13T14:24:55.280805+0000","last_became_peered":"2024-09-13T14:24:55.280805+0000","last_unstale":"2024-09-13T14:24:55.281723+0000","last_undegraded":"2024-09-13T14:24:55.281723+0000","last_fullsized":"2024-09-13T14:24:55.281723+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:14.806713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":70,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.271380+0000","last_change":"2024-09-13T14:23:51.609769+0000","last_active":"2024-09-13T14:24:55.271380+0000","last_peered":"2024-09-13T14:24:55.271380+0000","last_clean":"2024-09-13T14:24:55.271380+0000","last_became_active":"2024-09-13T14:23:51.609343+0000","last_became_peered":"2024-09-13T14:23:51.609343+0000","last_unstale":"2024-09-13T14:24:55.271380+0000","last_undegraded":"2024-09-13T14:24:55.271380+0000","last_fullsized":"2024-09-13T14:24:55.271380+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:06:58.544473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":514,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.282977+0000","last_change":"2024-09-13T14:24:23.775768+0000","last_active":"2024-09-13T14:24:55.282977+0000","last_peered":"2024-09-13T14:24:55.282977+0000","last_clean":"2024-09-13T14:24:55.282977+0000","last_became_active":"2024-09-13T14:24:23.775261+0000","last_became_peered":"2024-09-13T14:24:23.775261+0000","last_unstale":"2024-09-13T14:24:55.282977+0000","last_undegraded":"2024-09-13T14:24:55.282977+0000","last_fullsized":"2024-09-13T14:24:55.282977+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:30:05.340603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":56,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.270378+0000","last_change":"2024-09-13T14:11:30.256034+0000","last_active":"2024-09-13T14:24:54.270378+0000","last_peered":"2024-09-13T14:24:54.270378+0000","last_clean":"2024-09-13T14:24:54.270378+0000","last_became_active":"2024-09-13T14:11:30.255727+0000","last_became_peered":"2024-09-13T14:11:30.255727+0000","last_unstale":"2024-09-13T14:24:54.270378+0000","last_undegraded":"2024-09-13T14:24:54.270378+0000","last_fullsized":"2024-09-13T14:24:54.270378+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:21:35.817014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":71,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.274287+0000","last_change":"2024-09-13T14:23:51.606397+0000","last_active":"2024-09-13T14:24:55.274287+0000","last_peered":"2024-09-13T14:24:55.274287+0000","last_clean":"2024-09-13T14:24:55.274287+0000","last_became_active":"2024-09-13T14:23:51.605792+0000","last_became_peered":"2024-09-13T14:23:51.605792+0000","last_unstale":"2024-09-13T14:24:55.274287+0000","last_undegraded":"2024-09-13T14:24:55.274287+0000","last_fullsized":"2024-09-13T14:24:55.274287+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:42:44.617374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":80,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.275351+0000","last_change":"2024-09-13T14:11:29.237013+0000","last_active":"2024-09-13T14:24:55.275351+0000","last_peered":"2024-09-13T14:24:55.275351+0000","last_clean":"2024-09-13T14:24:55.275351+0000","last_became_active":"2024-09-13T14:11:29.236643+0000","last_became_peered":"2024-09-13T14:11:29.236643+0000","last_unstale":"2024-09-13T14:24:55.275351+0000","last_undegraded":"2024-09-13T14:24:55.275351+0000","last_fullsized":"2024-09-13T14:24:55.275351+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:33:58.438693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":56,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.275681+0000","last_change":"2024-09-13T14:24:23.779556+0000","last_active":"2024-09-13T14:24:55.275681+0000","last_peered":"2024-09-13T14:24:55.275681+0000","last_clean":"2024-09-13T14:24:55.275681+0000","last_became_active":"2024-09-13T14:24:23.779234+0000","last_became_peered":"2024-09-13T14:24:23.779234+0000","last_unstale":"2024-09-13T14:24:55.275681+0000","last_undegraded":"2024-09-13T14:24:55.275681+0000","last_fullsized":"2024-09-13T14:24:55.275681+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:40.733288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":75,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.287275+0000","last_change":"2024-09-13T14:24:55.287275+0000","last_active":"2024-09-13T14:24:55.287275+0000","last_peered":"2024-09-13T14:24:55.287275+0000","last_clean":"2024-09-13T14:24:55.287275+0000","last_became_active":"2024-09-13T14:24:55.286919+0000","last_became_peered":"2024-09-13T14:24:55.286919+0000","last_unstale":"2024-09-13T14:24:55.287275+0000","last_undegraded":"2024-09-13T14:24:55.287275+0000","last_fullsized":"2024-09-13T14:24:55.287275+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:10.790711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":72,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.269937+0000","last_change":"2024-09-13T14:23:51.610941+0000","last_active":"2024-09-13T14:24:54.269937+0000","last_peered":"2024-09-13T14:24:54.269937+0000","last_clean":"2024-09-13T14:24:54.269937+0000","last_became_active":"2024-09-13T14:23:51.610587+0000","last_became_peered":"2024-09-13T14:23:51.610587+0000","last_unstale":"2024-09-13T14:24:54.269937+0000","last_undegraded":"2024-09-13T14:24:54.269937+0000","last_fullsized":"2024-09-13T14:24:54.269937+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:48:01.436607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.288101+0000","last_change":"2024-09-13T14:24:55.287509+0000","last_active":"2024-09-13T14:24:55.288101+0000","last_peered":"2024-09-13T14:24:55.288101+0000","last_clean":"2024-09-13T14:24:55.288101+0000","last_became_active":"2024-09-13T14:24:55.287193+0000","last_became_peered":"2024-09-13T14:24:55.287193+0000","last_unstale":"2024-09-13T14:24:55.288101+0000","last_undegraded":"2024-09-13T14:24:55.288101+0000","last_fullsized":"2024-09-13T14:24:55.288101+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:14:10.603307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.289393+0000","last_change":"2024-09-13T14:24:55.289055+0000","last_active":"2024-09-13T14:24:55.289393+0000","last_peered":"2024-09-13T14:24:55.289393+0000","last_clean":"2024-09-13T14:24:55.289393+0000","last_became_active":"2024-09-13T14:24:55.288762+0000","last_became_peered":"2024-09-13T14:24:55.288762+0000","last_unstale":"2024-09-13T14:24:55.289393+0000","last_undegraded":"2024-09-13T14:24:55.289393+0000","last_fullsized":"2024-09-13T14:24:55.289393+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:37:26.639877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":74,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.287280+0000","last_change":"2024-09-13T14:24:55.287211+0000","last_active":"2024-09-13T14:24:55.287280+0000","last_peered":"2024-09-13T14:24:55.287280+0000","last_clean":"2024-09-13T14:24:55.287280+0000","last_became_active":"2024-09-13T14:24:55.286301+0000","last_became_peered":"2024-09-13T14:24:55.286301+0000","last_unstale":"2024-09-13T14:24:55.287280+0000","last_undegraded":"2024-09-13T14:24:55.287280+0000","last_fullsized":"2024-09-13T14:24:55.287280+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:32:41.099046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":72,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.276450+0000","last_change":"2024-09-13T14:23:51.606509+0000","last_active":"2024-09-13T14:24:54.276450+0000","last_peered":"2024-09-13T14:24:54.276450+0000","last_clean":"2024-09-13T14:24:54.276450+0000","last_became_active":"2024-09-13T14:23:51.606195+0000","last_became_peered":"2024-09-13T14:23:51.606195+0000","last_unstale":"2024-09-13T14:24:54.276450+0000","last_undegraded":"2024-09-13T14:24:54.276450+0000","last_fullsized":"2024-09-13T14:24:54.276450+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:29:49.729550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":90,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.280528+0000","last_change":"2024-09-13T14:24:55.280438+0000","last_active":"2024-09-13T14:24:55.280528+0000","last_peered":"2024-09-13T14:24:55.280528+0000","last_clean":"2024-09-13T14:24:55.280528+0000","last_became_active":"2024-09-13T14:24:55.279359+0000","last_became_peered":"2024-09-13T14:24:55.279359+0000","last_unstale":"2024-09-13T14:24:55.280528+0000","last_undegraded":"2024-09-13T14:24:55.280528+0000","last_fullsized":"2024-09-13T14:24:55.280528+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:12:48.432924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":67,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.284867+0000","last_change":"2024-09-13T14:24:55.283070+0000","last_active":"2024-09-13T14:24:55.284867+0000","last_peered":"2024-09-13T14:24:55.284867+0000","last_clean":"2024-09-13T14:24:55.284867+0000","last_became_active":"2024-09-13T14:24:55.282591+0000","last_became_peered":"2024-09-13T14:24:55.282591+0000","last_unstale":"2024-09-13T14:24:55.284867+0000","last_undegraded":"2024-09-13T14:24:55.284867+0000","last_fullsized":"2024-09-13T14:24:55.284867+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:35:21.896294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":55,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.289919+0000","last_change":"2024-09-13T14:24:55.289828+0000","last_active":"2024-09-13T14:24:55.289919+0000","last_peered":"2024-09-13T14:24:55.289919+0000","last_clean":"2024-09-13T14:24:55.289919+0000","last_became_active":"2024-09-13T14:24:55.289525+0000","last_became_peered":"2024-09-13T14:24:55.289525+0000","last_unstale":"2024-09-13T14:24:55.289919+0000","last_undegraded":"2024-09-13T14:24:55.289919+0000","last_fullsized":"2024-09-13T14:24:55.289919+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:25:39.041539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":87,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.284789+0000","last_change":"2024-09-13T14:24:55.284018+0000","last_active":"2024-09-13T14:24:55.284789+0000","last_peered":"2024-09-13T14:24:55.284789+0000","last_clean":"2024-09-13T14:24:55.284789+0000","last_became_active":"2024-09-13T14:24:55.283571+0000","last_became_peered":"2024-09-13T14:24:55.283571+0000","last_unstale":"2024-09-13T14:24:55.284789+0000","last_undegraded":"2024-09-13T14:24:55.284789+0000","last_fullsized":"2024-09-13T14:24:55.284789+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:48.890356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":87,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.270277+0000","last_change":"2024-09-13T14:24:23.778031+0000","last_active":"2024-09-13T14:24:54.270277+0000","last_peered":"2024-09-13T14:24:54.270277+0000","last_clean":"2024-09-13T14:24:54.270277+0000","last_became_active":"2024-09-13T14:24:23.777665+0000","last_became_peered":"2024-09-13T14:24:23.777665+0000","last_unstale":"2024-09-13T14:24:54.270277+0000","last_undegraded":"2024-09-13T14:24:54.270277+0000","last_fullsized":"2024-09-13T14:24:54.270277+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:27:54.790142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":94,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.290937+0000","last_change":"2024-09-13T14:24:55.290839+0000","last_active":"2024-09-13T14:24:55.290937+0000","last_peered":"2024-09-13T14:24:55.290937+0000","last_clean":"2024-09-13T14:24:55.290937+0000","last_became_active":"2024-09-13T14:24:55.290348+0000","last_became_peered":"2024-09-13T14:24:55.290348+0000","last_unstale":"2024-09-13T14:24:55.290937+0000","last_undegraded":"2024-09-13T14:24:55.290937+0000","last_fullsized":"2024-09-13T14:24:55.290937+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:11:38.467219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":59,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.271734+0000","last_change":"2024-09-13T14:23:18.035101+0000","last_active":"2024-09-13T14:24:55.271734+0000","last_peered":"2024-09-13T14:24:55.271734+0000","last_clean":"2024-09-13T14:24:55.271734+0000","last_became_active":"2024-09-13T14:23:18.034724+0000","last_became_peered":"2024-09-13T14:23:18.034724+0000","last_unstale":"2024-09-13T14:24:55.271734+0000","last_undegraded":"2024-09-13T14:24:55.271734+0000","last_fullsized":"2024-09-13T14:24:55.271734+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:39.358144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":85,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.282346+0000","last_change":"2024-09-13T14:24:55.282346+0000","last_active":"2024-09-13T14:24:55.282346+0000","last_peered":"2024-09-13T14:24:55.282346+0000","last_clean":"2024-09-13T14:24:55.282346+0000","last_became_active":"2024-09-13T14:24:55.281988+0000","last_became_peered":"2024-09-13T14:24:55.281988+0000","last_unstale":"2024-09-13T14:24:55.282346+0000","last_undegraded":"2024-09-13T14:24:55.282346+0000","last_fullsized":"2024-09-13T14:24:55.282346+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:05:35.737441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":74,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.270907+0000","last_change":"2024-09-13T14:23:51.612637+0000","last_active":"2024-09-13T14:24:54.270907+0000","last_peered":"2024-09-13T14:24:54.270907+0000","last_clean":"2024-09-13T14:24:54.270907+0000","last_became_active":"2024-09-13T14:23:51.612175+0000","last_became_peered":"2024-09-13T14:23:51.612175+0000","last_unstale":"2024-09-13T14:24:54.270907+0000","last_undegraded":"2024-09-13T14:24:54.270907+0000","last_fullsized":"2024-09-13T14:24:54.270907+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:52:30.299317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":85,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.285384+0000","last_change":"2024-09-13T14:24:55.285384+0000","last_active":"2024-09-13T14:24:55.285384+0000","last_peered":"2024-09-13T14:24:55.285384+0000","last_clean":"2024-09-13T14:24:55.285384+0000","last_became_active":"2024-09-13T14:24:55.285081+0000","last_became_peered":"2024-09-13T14:24:55.285081+0000","last_unstale":"2024-09-13T14:24:55.285384+0000","last_undegraded":"2024-09-13T14:24:55.285384+0000","last_fullsized":"2024-09-13T14:24:55.285384+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:47:07.946120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":59,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.272758+0000","last_change":"2024-09-13T14:11:29.240351+0000","last_active":"2024-09-13T14:24:54.272758+0000","last_peered":"2024-09-13T14:24:54.272758+0000","last_clean":"2024-09-13T14:24:54.272758+0000","last_became_active":"2024-09-13T14:11:29.240060+0000","last_became_peered":"2024-09-13T14:11:29.240060+0000","last_unstale":"2024-09-13T14:24:54.272758+0000","last_undegraded":"2024-09-13T14:24:54.272758+0000","last_fullsized":"2024-09-13T14:24:54.272758+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:34:36.126368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":93,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.285566+0000","last_change":"2024-09-13T14:24:55.285488+0000","last_active":"2024-09-13T14:24:55.285566+0000","last_peered":"2024-09-13T14:24:55.285566+0000","last_clean":"2024-09-13T14:24:55.285566+0000","last_became_active":"2024-09-13T14:24:55.283433+0000","last_became_peered":"2024-09-13T14:24:55.283433+0000","last_unstale":"2024-09-13T14:24:55.285566+0000","last_undegraded":"2024-09-13T14:24:55.285566+0000","last_fullsized":"2024-09-13T14:24:55.285566+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:21:11.486340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":81,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.287819+0000","last_change":"2024-09-13T14:24:55.287498+0000","last_active":"2024-09-13T14:24:55.287819+0000","last_peered":"2024-09-13T14:24:55.287819+0000","last_clean":"2024-09-13T14:24:55.287819+0000","last_became_active":"2024-09-13T14:24:55.285957+0000","last_became_peered":"2024-09-13T14:24:55.285957+0000","last_unstale":"2024-09-13T14:24:55.287819+0000","last_undegraded":"2024-09-13T14:24:55.287819+0000","last_fullsized":"2024-09-13T14:24:55.287819+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:21:50.821805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":70,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.269687+0000","last_change":"2024-09-13T14:23:51.609501+0000","last_active":"2024-09-13T14:24:54.269687+0000","last_peered":"2024-09-13T14:24:54.269687+0000","last_clean":"2024-09-13T14:24:54.269687+0000","last_became_active":"2024-09-13T14:23:51.609135+0000","last_became_peered":"2024-09-13T14:23:51.609135+0000","last_unstale":"2024-09-13T14:24:54.269687+0000","last_undegraded":"2024-09-13T14:24:54.269687+0000","last_fullsized":"2024-09-13T14:24:54.269687+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:21:25.318030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":70,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.271702+0000","last_change":"2024-09-13T14:24:23.775769+0000","last_active":"2024-09-13T14:24:55.271702+0000","last_peered":"2024-09-13T14:24:55.271702+0000","last_clean":"2024-09-13T14:24:55.271702+0000","last_became_active":"2024-09-13T14:24:23.775409+0000","last_became_peered":"2024-09-13T14:24:23.775409+0000","last_unstale":"2024-09-13T14:24:55.271702+0000","last_undegraded":"2024-09-13T14:24:55.271702+0000","last_fullsized":"2024-09-13T14:24:55.271702+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:20:09.966476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18272,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234183,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32912,"kb_used_data":1788,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712240,"statfs":{"total":95995035648,"available":95961333760,"internally_reserved":0,"allocated":1830912,"data_stored":1222916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495819,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36992,"kb_used_data":5876,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93708160,"statfs":{"total":95995035648,"available":95957155840,"internally_reserved":0,"allocated":6017024,"data_stored":5417151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757456,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36404,"kb_used_data":5288,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93708748,"statfs":{"total":95995035648,"available":95957757952,"internally_reserved":0,"allocated":5414912,"data_stored":4822551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019092,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28224,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716928,"statfs":{"total":95995035648,"available":95966134272,"internally_reserved":0,"allocated":1232896,"data_stored":632536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435277,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28728,"kb_used_data":1188,"kb_used_omap":5,"kb_used_meta":27514,"kb_avail":93716424,"statfs":{"total":95995035648,"available":95965618176,"internally_reserved":0,"allocated":1216512,"data_stored":628009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":28174462},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":104,"seq":446676598803,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28732,"kb_used_data":1192,"kb_used_omap":7,"kb_used_meta":27512,"kb_avail":93716420,"statfs":{"total":95995035648,"available":95965614080,"internally_reserved":0,"allocated":1220608,"data_stored":628535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8010,"internal_metadata":28172470},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":99,"seq":425201762330,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32812,"kb_used_data":5280,"kb_used_omap":7,"kb_used_meta":27512,"kb_avail":93712340,"statfs":{"total":95995035648,"available":95961436160,"internally_reserved":0,"allocated":5406720,"data_stored":4821969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7759,"internal_metadata":28172721},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726925856,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":29304,"kb_used_data":1772,"kb_used_omap":7,"kb_used_meta":27512,"kb_avail":93715848,"statfs":{"total":95995035648,"available":95965028352,"internally_reserved":0,"allocated":1814528,"data_stored":1218891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":28172584},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:25:51.193 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:25:51.394 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:25:51.767 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:51 smithi102 ceph-mon[110183]: from='client.44459 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:25:51.767 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:51 smithi102 ceph-mon[115269]: from='client.44459 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:25:51.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:51 smithi195 ceph-mon[104262]: from='client.44459 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:25:52.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:52 smithi102 ceph-mon[110183]: pgmap v276: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:52.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:52 smithi102 ceph-mon[115269]: pgmap v276: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:52.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:52 smithi195 ceph-mon[104262]: pgmap v276: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:53.126 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:25:53.127 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:25:53.748 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:53 smithi102 ceph-mon[115269]: from='client.44465 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:25:53.754 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:53 smithi102 ceph-mon[110183]: from='client.44465 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:25:53.769 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":276,"stamp":"2024-09-13T14:25:51.233448+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790245,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":6390,"num_read_kb":13845,"num_write":1123,"num_write_kb":15041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1182,"ondisk_log_size":1182,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":254108,"kb_used_data":23588,"kb_used_omap":35,"kb_used_meta":230364,"kb_avail":749707108,"statfs":{"total":767960285184,"available":767700078592,"internally_reserved":0,"allocated":24154112,"data_stored":19392558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36053,"internal_metadata":235893547},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002157"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":74,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.290727+0000","last_change":"2024-09-13T14:24:55.290644+0000","last_active":"2024-09-13T14:24:55.290727+0000","last_peered":"2024-09-13T14:24:55.290727+0000","last_clean":"2024-09-13T14:24:55.290727+0000","last_became_active":"2024-09-13T14:24:55.290222+0000","last_became_peered":"2024-09-13T14:24:55.290222+0000","last_unstale":"2024-09-13T14:24:55.290727+0000","last_undegraded":"2024-09-13T14:24:55.290727+0000","last_fullsized":"2024-09-13T14:24:55.290727+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:38:41.205619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":75,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.274419+0000","last_change":"2024-09-13T14:24:23.773702+0000","last_active":"2024-09-13T14:24:55.274419+0000","last_peered":"2024-09-13T14:24:55.274419+0000","last_clean":"2024-09-13T14:24:55.274419+0000","last_became_active":"2024-09-13T14:24:23.773290+0000","last_became_peered":"2024-09-13T14:24:23.773290+0000","last_unstale":"2024-09-13T14:24:55.274419+0000","last_undegraded":"2024-09-13T14:24:55.274419+0000","last_fullsized":"2024-09-13T14:24:55.274419+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:20.820970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":71,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.282407+0000","last_change":"2024-09-13T14:24:55.282407+0000","last_active":"2024-09-13T14:24:55.282407+0000","last_peered":"2024-09-13T14:24:55.282407+0000","last_clean":"2024-09-13T14:24:55.282407+0000","last_became_active":"2024-09-13T14:24:55.282083+0000","last_became_peered":"2024-09-13T14:24:55.282083+0000","last_unstale":"2024-09-13T14:24:55.282407+0000","last_undegraded":"2024-09-13T14:24:55.282407+0000","last_fullsized":"2024-09-13T14:24:55.282407+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:23:10.525707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":65,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.284816+0000","last_change":"2024-09-13T14:24:55.284433+0000","last_active":"2024-09-13T14:24:55.284816+0000","last_peered":"2024-09-13T14:24:55.284816+0000","last_clean":"2024-09-13T14:24:55.284816+0000","last_became_active":"2024-09-13T14:24:55.284143+0000","last_became_peered":"2024-09-13T14:24:55.284143+0000","last_unstale":"2024-09-13T14:24:55.284816+0000","last_undegraded":"2024-09-13T14:24:55.284816+0000","last_fullsized":"2024-09-13T14:24:55.284816+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:28.107405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":67,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.282846+0000","last_change":"2024-09-13T14:24:55.282416+0000","last_active":"2024-09-13T14:24:55.282846+0000","last_peered":"2024-09-13T14:24:55.282846+0000","last_clean":"2024-09-13T14:24:55.282846+0000","last_became_active":"2024-09-13T14:24:55.282045+0000","last_became_peered":"2024-09-13T14:24:55.282045+0000","last_unstale":"2024-09-13T14:24:55.282846+0000","last_undegraded":"2024-09-13T14:24:55.282846+0000","last_fullsized":"2024-09-13T14:24:55.282846+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:58.692897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":58,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.274055+0000","last_change":"2024-09-13T14:11:29.244452+0000","last_active":"2024-09-13T14:24:55.274055+0000","last_peered":"2024-09-13T14:24:55.274055+0000","last_clean":"2024-09-13T14:24:55.274055+0000","last_became_active":"2024-09-13T14:11:29.244102+0000","last_became_peered":"2024-09-13T14:11:29.244102+0000","last_unstale":"2024-09-13T14:24:55.274055+0000","last_undegraded":"2024-09-13T14:24:55.274055+0000","last_fullsized":"2024-09-13T14:24:55.274055+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:13:19.165534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":74,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.289698+0000","last_change":"2024-09-13T14:24:55.288871+0000","last_active":"2024-09-13T14:24:55.289698+0000","last_peered":"2024-09-13T14:24:55.289698+0000","last_clean":"2024-09-13T14:24:55.289698+0000","last_became_active":"2024-09-13T14:24:55.288506+0000","last_became_peered":"2024-09-13T14:24:55.288506+0000","last_unstale":"2024-09-13T14:24:55.289698+0000","last_undegraded":"2024-09-13T14:24:55.289698+0000","last_fullsized":"2024-09-13T14:24:55.289698+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:44.796657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":81,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.283649+0000","last_change":"2024-09-13T14:23:18.039061+0000","last_active":"2024-09-13T14:24:55.283649+0000","last_peered":"2024-09-13T14:24:55.283649+0000","last_clean":"2024-09-13T14:24:55.283649+0000","last_became_active":"2024-09-13T14:23:18.038661+0000","last_became_peered":"2024-09-13T14:23:18.038661+0000","last_unstale":"2024-09-13T14:24:55.283649+0000","last_undegraded":"2024-09-13T14:24:55.283649+0000","last_fullsized":"2024-09-13T14:24:55.283649+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:05:29.109844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":74,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.275099+0000","last_change":"2024-09-13T14:23:18.038597+0000","last_active":"2024-09-13T14:24:55.275099+0000","last_peered":"2024-09-13T14:24:55.275099+0000","last_clean":"2024-09-13T14:24:55.275099+0000","last_became_active":"2024-09-13T14:23:18.038240+0000","last_became_peered":"2024-09-13T14:23:18.038240+0000","last_unstale":"2024-09-13T14:24:55.275099+0000","last_undegraded":"2024-09-13T14:24:55.275099+0000","last_fullsized":"2024-09-13T14:24:55.275099+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:27:51.842195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":57,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.269670+0000","last_change":"2024-09-13T14:11:30.254534+0000","last_active":"2024-09-13T14:24:54.269670+0000","last_peered":"2024-09-13T14:24:54.269670+0000","last_clean":"2024-09-13T14:24:54.269670+0000","last_became_active":"2024-09-13T14:11:30.253959+0000","last_became_peered":"2024-09-13T14:11:30.253959+0000","last_unstale":"2024-09-13T14:24:54.269670+0000","last_undegraded":"2024-09-13T14:24:54.269670+0000","last_fullsized":"2024-09-13T14:24:54.269670+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:09:54.025853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":75,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.274762+0000","last_change":"2024-09-13T14:23:51.612105+0000","last_active":"2024-09-13T14:24:55.274762+0000","last_peered":"2024-09-13T14:24:55.274762+0000","last_clean":"2024-09-13T14:24:55.274762+0000","last_became_active":"2024-09-13T14:23:51.611740+0000","last_became_peered":"2024-09-13T14:23:51.611740+0000","last_unstale":"2024-09-13T14:24:55.274762+0000","last_undegraded":"2024-09-13T14:24:55.274762+0000","last_fullsized":"2024-09-13T14:24:55.274762+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:26:23.262096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":86,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.288950+0000","last_change":"2024-09-13T14:24:55.288950+0000","last_active":"2024-09-13T14:24:55.288950+0000","last_peered":"2024-09-13T14:24:55.288950+0000","last_clean":"2024-09-13T14:24:55.288950+0000","last_became_active":"2024-09-13T14:24:55.288671+0000","last_became_peered":"2024-09-13T14:24:55.288671+0000","last_unstale":"2024-09-13T14:24:55.288950+0000","last_undegraded":"2024-09-13T14:24:55.288950+0000","last_fullsized":"2024-09-13T14:24:55.288950+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:14:53.298390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":59,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.275139+0000","last_change":"2024-09-13T14:11:29.232897+0000","last_active":"2024-09-13T14:24:55.275139+0000","last_peered":"2024-09-13T14:24:55.275139+0000","last_clean":"2024-09-13T14:24:55.275139+0000","last_became_active":"2024-09-13T14:11:29.232637+0000","last_became_peered":"2024-09-13T14:11:29.232637+0000","last_unstale":"2024-09-13T14:24:55.275139+0000","last_undegraded":"2024-09-13T14:24:55.275139+0000","last_fullsized":"2024-09-13T14:24:55.275139+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:31:05.856046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":72,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.276706+0000","last_change":"2024-09-13T14:23:51.609167+0000","last_active":"2024-09-13T14:24:54.276706+0000","last_peered":"2024-09-13T14:24:54.276706+0000","last_clean":"2024-09-13T14:24:54.276706+0000","last_became_active":"2024-09-13T14:23:51.608811+0000","last_became_peered":"2024-09-13T14:23:51.608811+0000","last_unstale":"2024-09-13T14:24:54.276706+0000","last_undegraded":"2024-09-13T14:24:54.276706+0000","last_fullsized":"2024-09-13T14:24:54.276706+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:40.506763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":73,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.291355+0000","last_change":"2024-09-13T14:24:55.290798+0000","last_active":"2024-09-13T14:24:55.291355+0000","last_peered":"2024-09-13T14:24:55.291355+0000","last_clean":"2024-09-13T14:24:55.291355+0000","last_became_active":"2024-09-13T14:24:55.290510+0000","last_became_peered":"2024-09-13T14:24:55.290510+0000","last_unstale":"2024-09-13T14:24:55.291355+0000","last_undegraded":"2024-09-13T14:24:55.291355+0000","last_fullsized":"2024-09-13T14:24:55.291355+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:19:03.129132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":67,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.271982+0000","last_change":"2024-09-13T14:23:51.609082+0000","last_active":"2024-09-13T14:24:55.271982+0000","last_peered":"2024-09-13T14:24:55.271982+0000","last_clean":"2024-09-13T14:24:55.271982+0000","last_became_active":"2024-09-13T14:23:51.608799+0000","last_became_peered":"2024-09-13T14:23:51.608799+0000","last_unstale":"2024-09-13T14:24:55.271982+0000","last_undegraded":"2024-09-13T14:24:55.271982+0000","last_fullsized":"2024-09-13T14:24:55.271982+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:43:24.721453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":72,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.284590+0000","last_change":"2024-09-13T14:24:55.284499+0000","last_active":"2024-09-13T14:24:55.284590+0000","last_peered":"2024-09-13T14:24:55.284590+0000","last_clean":"2024-09-13T14:24:55.284590+0000","last_became_active":"2024-09-13T14:24:55.284096+0000","last_became_peered":"2024-09-13T14:24:55.284096+0000","last_unstale":"2024-09-13T14:24:55.284590+0000","last_undegraded":"2024-09-13T14:24:55.284590+0000","last_fullsized":"2024-09-13T14:24:55.284590+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:02:06.836099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":71,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.291592+0000","last_change":"2024-09-13T14:24:55.291371+0000","last_active":"2024-09-13T14:24:55.291592+0000","last_peered":"2024-09-13T14:24:55.291592+0000","last_clean":"2024-09-13T14:24:55.291592+0000","last_became_active":"2024-09-13T14:24:55.291047+0000","last_became_peered":"2024-09-13T14:24:55.291047+0000","last_unstale":"2024-09-13T14:24:55.291592+0000","last_undegraded":"2024-09-13T14:24:55.291592+0000","last_fullsized":"2024-09-13T14:24:55.291592+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:43.912412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":73,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.276868+0000","last_change":"2024-09-13T14:23:51.606589+0000","last_active":"2024-09-13T14:24:54.276868+0000","last_peered":"2024-09-13T14:24:54.276868+0000","last_clean":"2024-09-13T14:24:54.276868+0000","last_became_active":"2024-09-13T14:23:51.606244+0000","last_became_peered":"2024-09-13T14:23:51.606244+0000","last_unstale":"2024-09-13T14:24:54.276868+0000","last_undegraded":"2024-09-13T14:24:54.276868+0000","last_fullsized":"2024-09-13T14:24:54.276868+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:08.776526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":86,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.275537+0000","last_change":"2024-09-13T14:24:23.777891+0000","last_active":"2024-09-13T14:24:55.275537+0000","last_peered":"2024-09-13T14:24:55.275537+0000","last_clean":"2024-09-13T14:24:55.275537+0000","last_became_active":"2024-09-13T14:24:23.777578+0000","last_became_peered":"2024-09-13T14:24:23.777578+0000","last_unstale":"2024-09-13T14:24:55.275537+0000","last_undegraded":"2024-09-13T14:24:55.275537+0000","last_fullsized":"2024-09-13T14:24:55.275537+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:56:47.441579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":72,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.270580+0000","last_change":"2024-09-13T14:24:23.777970+0000","last_active":"2024-09-13T14:24:54.270580+0000","last_peered":"2024-09-13T14:24:54.270580+0000","last_clean":"2024-09-13T14:24:54.270580+0000","last_became_active":"2024-09-13T14:24:23.777650+0000","last_became_peered":"2024-09-13T14:24:23.777650+0000","last_unstale":"2024-09-13T14:24:54.270580+0000","last_undegraded":"2024-09-13T14:24:54.270580+0000","last_fullsized":"2024-09-13T14:24:54.270580+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:50:00.549905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":71,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.274746+0000","last_change":"2024-09-13T14:23:51.610572+0000","last_active":"2024-09-13T14:24:55.274746+0000","last_peered":"2024-09-13T14:24:55.274746+0000","last_clean":"2024-09-13T14:24:55.274746+0000","last_became_active":"2024-09-13T14:23:51.610273+0000","last_became_peered":"2024-09-13T14:23:51.610273+0000","last_unstale":"2024-09-13T14:24:55.274746+0000","last_undegraded":"2024-09-13T14:24:55.274746+0000","last_fullsized":"2024-09-13T14:24:55.274746+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:17:10.574417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":75,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.281133+0000","last_change":"2024-09-13T14:24:55.281042+0000","last_active":"2024-09-13T14:24:55.281133+0000","last_peered":"2024-09-13T14:24:55.281133+0000","last_clean":"2024-09-13T14:24:55.281133+0000","last_became_active":"2024-09-13T14:24:55.280741+0000","last_became_peered":"2024-09-13T14:24:55.280741+0000","last_unstale":"2024-09-13T14:24:55.281133+0000","last_undegraded":"2024-09-13T14:24:55.281133+0000","last_fullsized":"2024-09-13T14:24:55.281133+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:45:40.787103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":46,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.289331+0000","last_change":"2024-09-13T14:24:55.289245+0000","last_active":"2024-09-13T14:24:55.289331+0000","last_peered":"2024-09-13T14:24:55.289331+0000","last_clean":"2024-09-13T14:24:55.289331+0000","last_became_active":"2024-09-13T14:24:55.288881+0000","last_became_peered":"2024-09-13T14:24:55.288881+0000","last_unstale":"2024-09-13T14:24:55.289331+0000","last_undegraded":"2024-09-13T14:24:55.289331+0000","last_fullsized":"2024-09-13T14:24:55.289331+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:14:12.194309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":404,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.274086+0000","last_change":"2024-09-13T14:23:18.032973+0000","last_active":"2024-09-13T14:24:55.274086+0000","last_peered":"2024-09-13T14:24:55.274086+0000","last_clean":"2024-09-13T14:24:55.274086+0000","last_became_active":"2024-09-13T14:23:18.032579+0000","last_became_peered":"2024-09-13T14:23:18.032579+0000","last_unstale":"2024-09-13T14:24:55.274086+0000","last_undegraded":"2024-09-13T14:24:55.274086+0000","last_fullsized":"2024-09-13T14:24:55.274086+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:07:41.927740+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:17.109445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":400,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.270769+0000","last_change":"2024-09-13T14:24:23.776868+0000","last_active":"2024-09-13T14:24:54.270769+0000","last_peered":"2024-09-13T14:24:54.270769+0000","last_clean":"2024-09-13T14:24:54.270769+0000","last_became_active":"2024-09-13T14:24:23.776306+0000","last_became_peered":"2024-09-13T14:24:23.776306+0000","last_unstale":"2024-09-13T14:24:54.270769+0000","last_undegraded":"2024-09-13T14:24:54.270769+0000","last_fullsized":"2024-09-13T14:24:54.270769+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:28:41.805918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":23,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.288138+0000","last_change":"2024-09-13T14:24:55.288058+0000","last_active":"2024-09-13T14:24:55.288138+0000","last_peered":"2024-09-13T14:24:55.288138+0000","last_clean":"2024-09-13T14:24:55.288138+0000","last_became_active":"2024-09-13T14:24:55.287601+0000","last_became_peered":"2024-09-13T14:24:55.287601+0000","last_unstale":"2024-09-13T14:24:55.288138+0000","last_undegraded":"2024-09-13T14:24:55.288138+0000","last_fullsized":"2024-09-13T14:24:55.288138+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:56:34.622696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":57,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.277982+0000","last_change":"2024-09-13T14:11:30.253749+0000","last_active":"2024-09-13T14:24:54.277982+0000","last_peered":"2024-09-13T14:24:54.277982+0000","last_clean":"2024-09-13T14:24:54.277982+0000","last_became_active":"2024-09-13T14:11:30.253499+0000","last_became_peered":"2024-09-13T14:11:30.253499+0000","last_unstale":"2024-09-13T14:24:54.277982+0000","last_undegraded":"2024-09-13T14:24:54.277982+0000","last_fullsized":"2024-09-13T14:24:54.277982+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:02:00.987335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":289,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.275098+0000","last_change":"2024-09-13T14:23:51.608681+0000","last_active":"2024-09-13T14:24:54.275098+0000","last_peered":"2024-09-13T14:24:54.275098+0000","last_clean":"2024-09-13T14:24:54.275098+0000","last_became_active":"2024-09-13T14:23:51.608223+0000","last_became_peered":"2024-09-13T14:23:51.608223+0000","last_unstale":"2024-09-13T14:24:54.275098+0000","last_undegraded":"2024-09-13T14:24:54.275098+0000","last_fullsized":"2024-09-13T14:24:54.275098+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:13:15.930987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.286928+0000","last_change":"2024-09-13T14:24:55.286576+0000","last_active":"2024-09-13T14:24:55.286928+0000","last_peered":"2024-09-13T14:24:55.286928+0000","last_clean":"2024-09-13T14:24:55.286928+0000","last_became_active":"2024-09-13T14:24:55.286284+0000","last_became_peered":"2024-09-13T14:24:55.286284+0000","last_unstale":"2024-09-13T14:24:55.286928+0000","last_undegraded":"2024-09-13T14:24:55.286928+0000","last_fullsized":"2024-09-13T14:24:55.286928+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:35.589736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":78,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.271821+0000","last_change":"2024-09-13T14:24:23.776557+0000","last_active":"2024-09-13T14:24:55.271821+0000","last_peered":"2024-09-13T14:24:55.271821+0000","last_clean":"2024-09-13T14:24:55.271821+0000","last_became_active":"2024-09-13T14:24:23.775204+0000","last_became_peered":"2024-09-13T14:24:23.775204+0000","last_unstale":"2024-09-13T14:24:55.271821+0000","last_undegraded":"2024-09-13T14:24:55.271821+0000","last_fullsized":"2024-09-13T14:24:55.271821+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:18:21.095561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":494,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.285398+0000","last_change":"2024-09-13T14:24:55.285288+0000","last_active":"2024-09-13T14:24:55.285398+0000","last_peered":"2024-09-13T14:24:55.285398+0000","last_clean":"2024-09-13T14:24:55.285398+0000","last_became_active":"2024-09-13T14:24:55.284733+0000","last_became_peered":"2024-09-13T14:24:55.284733+0000","last_unstale":"2024-09-13T14:24:55.285398+0000","last_undegraded":"2024-09-13T14:24:55.285398+0000","last_fullsized":"2024-09-13T14:24:55.285398+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:51:50.420796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":55,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.276363+0000","last_change":"2024-09-13T14:24:23.779725+0000","last_active":"2024-09-13T14:24:55.276363+0000","last_peered":"2024-09-13T14:24:55.276363+0000","last_clean":"2024-09-13T14:24:55.276363+0000","last_became_active":"2024-09-13T14:24:23.779418+0000","last_became_peered":"2024-09-13T14:24:23.779418+0000","last_unstale":"2024-09-13T14:24:55.276363+0000","last_undegraded":"2024-09-13T14:24:55.276363+0000","last_fullsized":"2024-09-13T14:24:55.276363+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:48:08.470127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":89,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.276842+0000","last_change":"2024-09-13T14:23:51.608576+0000","last_active":"2024-09-13T14:24:55.276842+0000","last_peered":"2024-09-13T14:24:55.276842+0000","last_clean":"2024-09-13T14:24:55.276842+0000","last_became_active":"2024-09-13T14:23:51.607956+0000","last_became_peered":"2024-09-13T14:23:51.607956+0000","last_unstale":"2024-09-13T14:24:55.276842+0000","last_undegraded":"2024-09-13T14:24:55.276842+0000","last_fullsized":"2024-09-13T14:24:55.276842+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:53.399552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":829,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.272095+0000","last_change":"2024-09-13T14:23:51.606335+0000","last_active":"2024-09-13T14:24:54.272095+0000","last_peered":"2024-09-13T14:24:54.272095+0000","last_clean":"2024-09-13T14:24:54.272095+0000","last_became_active":"2024-09-13T14:23:51.605905+0000","last_became_peered":"2024-09-13T14:23:51.605905+0000","last_unstale":"2024-09-13T14:24:54.272095+0000","last_undegraded":"2024-09-13T14:24:54.272095+0000","last_fullsized":"2024-09-13T14:24:54.272095+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:22:42.470188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":72,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.272867+0000","last_change":"2024-09-13T14:23:18.042891+0000","last_active":"2024-09-13T14:24:54.272867+0000","last_peered":"2024-09-13T14:24:54.272867+0000","last_clean":"2024-09-13T14:24:54.272867+0000","last_became_active":"2024-09-13T14:23:18.042562+0000","last_became_peered":"2024-09-13T14:23:18.042562+0000","last_unstale":"2024-09-13T14:24:54.272867+0000","last_undegraded":"2024-09-13T14:24:54.272867+0000","last_fullsized":"2024-09-13T14:24:54.272867+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:30:42.996661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":84,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.291080+0000","last_change":"2024-09-13T14:24:55.291080+0000","last_active":"2024-09-13T14:24:55.291080+0000","last_peered":"2024-09-13T14:24:55.291080+0000","last_clean":"2024-09-13T14:24:55.291080+0000","last_became_active":"2024-09-13T14:24:55.290560+0000","last_became_peered":"2024-09-13T14:24:55.290560+0000","last_unstale":"2024-09-13T14:24:55.291080+0000","last_undegraded":"2024-09-13T14:24:55.291080+0000","last_fullsized":"2024-09-13T14:24:55.291080+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:16.041382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":88,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.282619+0000","last_change":"2024-09-13T14:24:55.282619+0000","last_active":"2024-09-13T14:24:55.282619+0000","last_peered":"2024-09-13T14:24:55.282619+0000","last_clean":"2024-09-13T14:24:55.282619+0000","last_became_active":"2024-09-13T14:24:55.282110+0000","last_became_peered":"2024-09-13T14:24:55.282110+0000","last_unstale":"2024-09-13T14:24:55.282619+0000","last_undegraded":"2024-09-13T14:24:55.282619+0000","last_fullsized":"2024-09-13T14:24:55.282619+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:33:19.425344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":777,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.277056+0000","last_change":"2024-09-13T14:23:51.610302+0000","last_active":"2024-09-13T14:24:55.277056+0000","last_peered":"2024-09-13T14:24:55.277056+0000","last_clean":"2024-09-13T14:24:55.277056+0000","last_became_active":"2024-09-13T14:23:51.607653+0000","last_became_peered":"2024-09-13T14:23:51.607653+0000","last_unstale":"2024-09-13T14:24:55.277056+0000","last_undegraded":"2024-09-13T14:24:55.277056+0000","last_fullsized":"2024-09-13T14:24:55.277056+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:03:52.131651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":45,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.275826+0000","last_change":"2024-09-13T14:23:51.610321+0000","last_active":"2024-09-13T14:24:55.275826+0000","last_peered":"2024-09-13T14:24:55.275826+0000","last_clean":"2024-09-13T14:24:55.275826+0000","last_became_active":"2024-09-13T14:23:51.607733+0000","last_became_peered":"2024-09-13T14:23:51.607733+0000","last_unstale":"2024-09-13T14:24:55.275826+0000","last_undegraded":"2024-09-13T14:24:55.275826+0000","last_fullsized":"2024-09-13T14:24:55.275826+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:58:05.246186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":54,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.276963+0000","last_change":"2024-09-13T14:24:23.779678+0000","last_active":"2024-09-13T14:24:55.276963+0000","last_peered":"2024-09-13T14:24:55.276963+0000","last_clean":"2024-09-13T14:24:55.276963+0000","last_became_active":"2024-09-13T14:24:23.779343+0000","last_became_peered":"2024-09-13T14:24:23.779343+0000","last_unstale":"2024-09-13T14:24:55.276963+0000","last_undegraded":"2024-09-13T14:24:55.276963+0000","last_fullsized":"2024-09-13T14:24:55.276963+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:14:41.230951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":595,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.275237+0000","last_change":"2024-09-13T14:23:18.038847+0000","last_active":"2024-09-13T14:24:55.275237+0000","last_peered":"2024-09-13T14:24:55.275237+0000","last_clean":"2024-09-13T14:24:55.275237+0000","last_became_active":"2024-09-13T14:23:18.038188+0000","last_became_peered":"2024-09-13T14:23:18.038188+0000","last_unstale":"2024-09-13T14:24:55.275237+0000","last_undegraded":"2024-09-13T14:24:55.275237+0000","last_fullsized":"2024-09-13T14:24:55.275237+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:30:37.213496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":56,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.271836+0000","last_change":"2024-09-13T14:23:18.034427+0000","last_active":"2024-09-13T14:24:55.271836+0000","last_peered":"2024-09-13T14:24:55.271836+0000","last_clean":"2024-09-13T14:24:55.271836+0000","last_became_active":"2024-09-13T14:23:18.034092+0000","last_became_peered":"2024-09-13T14:23:18.034092+0000","last_unstale":"2024-09-13T14:24:55.271836+0000","last_undegraded":"2024-09-13T14:24:55.271836+0000","last_fullsized":"2024-09-13T14:24:55.271836+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:21.755115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":72,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.290792+0000","last_change":"2024-09-13T14:24:55.290711+0000","last_active":"2024-09-13T14:24:55.290792+0000","last_peered":"2024-09-13T14:24:55.290792+0000","last_clean":"2024-09-13T14:24:55.290792+0000","last_became_active":"2024-09-13T14:24:55.288549+0000","last_became_peered":"2024-09-13T14:24:55.288549+0000","last_unstale":"2024-09-13T14:24:55.290792+0000","last_undegraded":"2024-09-13T14:24:55.290792+0000","last_fullsized":"2024-09-13T14:24:55.290792+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:55:21.763931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":502,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.276663+0000","last_change":"2024-09-13T14:23:51.606481+0000","last_active":"2024-09-13T14:24:55.276663+0000","last_peered":"2024-09-13T14:24:55.276663+0000","last_clean":"2024-09-13T14:24:55.276663+0000","last_became_active":"2024-09-13T14:23:51.605783+0000","last_became_peered":"2024-09-13T14:23:51.605783+0000","last_unstale":"2024-09-13T14:24:55.276663+0000","last_undegraded":"2024-09-13T14:24:55.276663+0000","last_fullsized":"2024-09-13T14:24:55.276663+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:09:01.265713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":69,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.281723+0000","last_change":"2024-09-13T14:24:55.281593+0000","last_active":"2024-09-13T14:24:55.281723+0000","last_peered":"2024-09-13T14:24:55.281723+0000","last_clean":"2024-09-13T14:24:55.281723+0000","last_became_active":"2024-09-13T14:24:55.280805+0000","last_became_peered":"2024-09-13T14:24:55.280805+0000","last_unstale":"2024-09-13T14:24:55.281723+0000","last_undegraded":"2024-09-13T14:24:55.281723+0000","last_fullsized":"2024-09-13T14:24:55.281723+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:14.806713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":70,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.271380+0000","last_change":"2024-09-13T14:23:51.609769+0000","last_active":"2024-09-13T14:24:55.271380+0000","last_peered":"2024-09-13T14:24:55.271380+0000","last_clean":"2024-09-13T14:24:55.271380+0000","last_became_active":"2024-09-13T14:23:51.609343+0000","last_became_peered":"2024-09-13T14:23:51.609343+0000","last_unstale":"2024-09-13T14:24:55.271380+0000","last_undegraded":"2024-09-13T14:24:55.271380+0000","last_fullsized":"2024-09-13T14:24:55.271380+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:06:58.544473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":514,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.282977+0000","last_change":"2024-09-13T14:24:23.775768+0000","last_active":"2024-09-13T14:24:55.282977+0000","last_peered":"2024-09-13T14:24:55.282977+0000","last_clean":"2024-09-13T14:24:55.282977+0000","last_became_active":"2024-09-13T14:24:23.775261+0000","last_became_peered":"2024-09-13T14:24:23.775261+0000","last_unstale":"2024-09-13T14:24:55.282977+0000","last_undegraded":"2024-09-13T14:24:55.282977+0000","last_fullsized":"2024-09-13T14:24:55.282977+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:30:05.340603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":56,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.270378+0000","last_change":"2024-09-13T14:11:30.256034+0000","last_active":"2024-09-13T14:24:54.270378+0000","last_peered":"2024-09-13T14:24:54.270378+0000","last_clean":"2024-09-13T14:24:54.270378+0000","last_became_active":"2024-09-13T14:11:30.255727+0000","last_became_peered":"2024-09-13T14:11:30.255727+0000","last_unstale":"2024-09-13T14:24:54.270378+0000","last_undegraded":"2024-09-13T14:24:54.270378+0000","last_fullsized":"2024-09-13T14:24:54.270378+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:21:35.817014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":71,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.274287+0000","last_change":"2024-09-13T14:23:51.606397+0000","last_active":"2024-09-13T14:24:55.274287+0000","last_peered":"2024-09-13T14:24:55.274287+0000","last_clean":"2024-09-13T14:24:55.274287+0000","last_became_active":"2024-09-13T14:23:51.605792+0000","last_became_peered":"2024-09-13T14:23:51.605792+0000","last_unstale":"2024-09-13T14:24:55.274287+0000","last_undegraded":"2024-09-13T14:24:55.274287+0000","last_fullsized":"2024-09-13T14:24:55.274287+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:42:44.617374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":80,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.275351+0000","last_change":"2024-09-13T14:11:29.237013+0000","last_active":"2024-09-13T14:24:55.275351+0000","last_peered":"2024-09-13T14:24:55.275351+0000","last_clean":"2024-09-13T14:24:55.275351+0000","last_became_active":"2024-09-13T14:11:29.236643+0000","last_became_peered":"2024-09-13T14:11:29.236643+0000","last_unstale":"2024-09-13T14:24:55.275351+0000","last_undegraded":"2024-09-13T14:24:55.275351+0000","last_fullsized":"2024-09-13T14:24:55.275351+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:33:58.438693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":56,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.275681+0000","last_change":"2024-09-13T14:24:23.779556+0000","last_active":"2024-09-13T14:24:55.275681+0000","last_peered":"2024-09-13T14:24:55.275681+0000","last_clean":"2024-09-13T14:24:55.275681+0000","last_became_active":"2024-09-13T14:24:23.779234+0000","last_became_peered":"2024-09-13T14:24:23.779234+0000","last_unstale":"2024-09-13T14:24:55.275681+0000","last_undegraded":"2024-09-13T14:24:55.275681+0000","last_fullsized":"2024-09-13T14:24:55.275681+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:40.733288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":75,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.287275+0000","last_change":"2024-09-13T14:24:55.287275+0000","last_active":"2024-09-13T14:24:55.287275+0000","last_peered":"2024-09-13T14:24:55.287275+0000","last_clean":"2024-09-13T14:24:55.287275+0000","last_became_active":"2024-09-13T14:24:55.286919+0000","last_became_peered":"2024-09-13T14:24:55.286919+0000","last_unstale":"2024-09-13T14:24:55.287275+0000","last_undegraded":"2024-09-13T14:24:55.287275+0000","last_fullsized":"2024-09-13T14:24:55.287275+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:10.790711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":72,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.269937+0000","last_change":"2024-09-13T14:23:51.610941+0000","last_active":"2024-09-13T14:24:54.269937+0000","last_peered":"2024-09-13T14:24:54.269937+0000","last_clean":"2024-09-13T14:24:54.269937+0000","last_became_active":"2024-09-13T14:23:51.610587+0000","last_became_peered":"2024-09-13T14:23:51.610587+0000","last_unstale":"2024-09-13T14:24:54.269937+0000","last_undegraded":"2024-09-13T14:24:54.269937+0000","last_fullsized":"2024-09-13T14:24:54.269937+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:48:01.436607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.288101+0000","last_change":"2024-09-13T14:24:55.287509+0000","last_active":"2024-09-13T14:24:55.288101+0000","last_peered":"2024-09-13T14:24:55.288101+0000","last_clean":"2024-09-13T14:24:55.288101+0000","last_became_active":"2024-09-13T14:24:55.287193+0000","last_became_peered":"2024-09-13T14:24:55.287193+0000","last_unstale":"2024-09-13T14:24:55.288101+0000","last_undegraded":"2024-09-13T14:24:55.288101+0000","last_fullsized":"2024-09-13T14:24:55.288101+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:14:10.603307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.289393+0000","last_change":"2024-09-13T14:24:55.289055+0000","last_active":"2024-09-13T14:24:55.289393+0000","last_peered":"2024-09-13T14:24:55.289393+0000","last_clean":"2024-09-13T14:24:55.289393+0000","last_became_active":"2024-09-13T14:24:55.288762+0000","last_became_peered":"2024-09-13T14:24:55.288762+0000","last_unstale":"2024-09-13T14:24:55.289393+0000","last_undegraded":"2024-09-13T14:24:55.289393+0000","last_fullsized":"2024-09-13T14:24:55.289393+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:37:26.639877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":74,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.287280+0000","last_change":"2024-09-13T14:24:55.287211+0000","last_active":"2024-09-13T14:24:55.287280+0000","last_peered":"2024-09-13T14:24:55.287280+0000","last_clean":"2024-09-13T14:24:55.287280+0000","last_became_active":"2024-09-13T14:24:55.286301+0000","last_became_peered":"2024-09-13T14:24:55.286301+0000","last_unstale":"2024-09-13T14:24:55.287280+0000","last_undegraded":"2024-09-13T14:24:55.287280+0000","last_fullsized":"2024-09-13T14:24:55.287280+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:32:41.099046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":72,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.276450+0000","last_change":"2024-09-13T14:23:51.606509+0000","last_active":"2024-09-13T14:24:54.276450+0000","last_peered":"2024-09-13T14:24:54.276450+0000","last_clean":"2024-09-13T14:24:54.276450+0000","last_became_active":"2024-09-13T14:23:51.606195+0000","last_became_peered":"2024-09-13T14:23:51.606195+0000","last_unstale":"2024-09-13T14:24:54.276450+0000","last_undegraded":"2024-09-13T14:24:54.276450+0000","last_fullsized":"2024-09-13T14:24:54.276450+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:29:49.729550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":90,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.280528+0000","last_change":"2024-09-13T14:24:55.280438+0000","last_active":"2024-09-13T14:24:55.280528+0000","last_peered":"2024-09-13T14:24:55.280528+0000","last_clean":"2024-09-13T14:24:55.280528+0000","last_became_active":"2024-09-13T14:24:55.279359+0000","last_became_peered":"2024-09-13T14:24:55.279359+0000","last_unstale":"2024-09-13T14:24:55.280528+0000","last_undegraded":"2024-09-13T14:24:55.280528+0000","last_fullsized":"2024-09-13T14:24:55.280528+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:12:48.432924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":67,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.284867+0000","last_change":"2024-09-13T14:24:55.283070+0000","last_active":"2024-09-13T14:24:55.284867+0000","last_peered":"2024-09-13T14:24:55.284867+0000","last_clean":"2024-09-13T14:24:55.284867+0000","last_became_active":"2024-09-13T14:24:55.282591+0000","last_became_peered":"2024-09-13T14:24:55.282591+0000","last_unstale":"2024-09-13T14:24:55.284867+0000","last_undegraded":"2024-09-13T14:24:55.284867+0000","last_fullsized":"2024-09-13T14:24:55.284867+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:35:21.896294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":55,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.289919+0000","last_change":"2024-09-13T14:24:55.289828+0000","last_active":"2024-09-13T14:24:55.289919+0000","last_peered":"2024-09-13T14:24:55.289919+0000","last_clean":"2024-09-13T14:24:55.289919+0000","last_became_active":"2024-09-13T14:24:55.289525+0000","last_became_peered":"2024-09-13T14:24:55.289525+0000","last_unstale":"2024-09-13T14:24:55.289919+0000","last_undegraded":"2024-09-13T14:24:55.289919+0000","last_fullsized":"2024-09-13T14:24:55.289919+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:25:39.041539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":87,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.284789+0000","last_change":"2024-09-13T14:24:55.284018+0000","last_active":"2024-09-13T14:24:55.284789+0000","last_peered":"2024-09-13T14:24:55.284789+0000","last_clean":"2024-09-13T14:24:55.284789+0000","last_became_active":"2024-09-13T14:24:55.283571+0000","last_became_peered":"2024-09-13T14:24:55.283571+0000","last_unstale":"2024-09-13T14:24:55.284789+0000","last_undegraded":"2024-09-13T14:24:55.284789+0000","last_fullsized":"2024-09-13T14:24:55.284789+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:48.890356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":87,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.270277+0000","last_change":"2024-09-13T14:24:23.778031+0000","last_active":"2024-09-13T14:24:54.270277+0000","last_peered":"2024-09-13T14:24:54.270277+0000","last_clean":"2024-09-13T14:24:54.270277+0000","last_became_active":"2024-09-13T14:24:23.777665+0000","last_became_peered":"2024-09-13T14:24:23.777665+0000","last_unstale":"2024-09-13T14:24:54.270277+0000","last_undegraded":"2024-09-13T14:24:54.270277+0000","last_fullsized":"2024-09-13T14:24:54.270277+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:27:54.790142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":94,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.290937+0000","last_change":"2024-09-13T14:24:55.290839+0000","last_active":"2024-09-13T14:24:55.290937+0000","last_peered":"2024-09-13T14:24:55.290937+0000","last_clean":"2024-09-13T14:24:55.290937+0000","last_became_active":"2024-09-13T14:24:55.290348+0000","last_became_peered":"2024-09-13T14:24:55.290348+0000","last_unstale":"2024-09-13T14:24:55.290937+0000","last_undegraded":"2024-09-13T14:24:55.290937+0000","last_fullsized":"2024-09-13T14:24:55.290937+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:11:38.467219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":59,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.271734+0000","last_change":"2024-09-13T14:23:18.035101+0000","last_active":"2024-09-13T14:24:55.271734+0000","last_peered":"2024-09-13T14:24:55.271734+0000","last_clean":"2024-09-13T14:24:55.271734+0000","last_became_active":"2024-09-13T14:23:18.034724+0000","last_became_peered":"2024-09-13T14:23:18.034724+0000","last_unstale":"2024-09-13T14:24:55.271734+0000","last_undegraded":"2024-09-13T14:24:55.271734+0000","last_fullsized":"2024-09-13T14:24:55.271734+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:39.358144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":85,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.282346+0000","last_change":"2024-09-13T14:24:55.282346+0000","last_active":"2024-09-13T14:24:55.282346+0000","last_peered":"2024-09-13T14:24:55.282346+0000","last_clean":"2024-09-13T14:24:55.282346+0000","last_became_active":"2024-09-13T14:24:55.281988+0000","last_became_peered":"2024-09-13T14:24:55.281988+0000","last_unstale":"2024-09-13T14:24:55.282346+0000","last_undegraded":"2024-09-13T14:24:55.282346+0000","last_fullsized":"2024-09-13T14:24:55.282346+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:05:35.737441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":74,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.270907+0000","last_change":"2024-09-13T14:23:51.612637+0000","last_active":"2024-09-13T14:24:54.270907+0000","last_peered":"2024-09-13T14:24:54.270907+0000","last_clean":"2024-09-13T14:24:54.270907+0000","last_became_active":"2024-09-13T14:23:51.612175+0000","last_became_peered":"2024-09-13T14:23:51.612175+0000","last_unstale":"2024-09-13T14:24:54.270907+0000","last_undegraded":"2024-09-13T14:24:54.270907+0000","last_fullsized":"2024-09-13T14:24:54.270907+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:52:30.299317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":85,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.285384+0000","last_change":"2024-09-13T14:24:55.285384+0000","last_active":"2024-09-13T14:24:55.285384+0000","last_peered":"2024-09-13T14:24:55.285384+0000","last_clean":"2024-09-13T14:24:55.285384+0000","last_became_active":"2024-09-13T14:24:55.285081+0000","last_became_peered":"2024-09-13T14:24:55.285081+0000","last_unstale":"2024-09-13T14:24:55.285384+0000","last_undegraded":"2024-09-13T14:24:55.285384+0000","last_fullsized":"2024-09-13T14:24:55.285384+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:47:07.946120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":59,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.272758+0000","last_change":"2024-09-13T14:11:29.240351+0000","last_active":"2024-09-13T14:24:54.272758+0000","last_peered":"2024-09-13T14:24:54.272758+0000","last_clean":"2024-09-13T14:24:54.272758+0000","last_became_active":"2024-09-13T14:11:29.240060+0000","last_became_peered":"2024-09-13T14:11:29.240060+0000","last_unstale":"2024-09-13T14:24:54.272758+0000","last_undegraded":"2024-09-13T14:24:54.272758+0000","last_fullsized":"2024-09-13T14:24:54.272758+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:34:36.126368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":93,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.285566+0000","last_change":"2024-09-13T14:24:55.285488+0000","last_active":"2024-09-13T14:24:55.285566+0000","last_peered":"2024-09-13T14:24:55.285566+0000","last_clean":"2024-09-13T14:24:55.285566+0000","last_became_active":"2024-09-13T14:24:55.283433+0000","last_became_peered":"2024-09-13T14:24:55.283433+0000","last_unstale":"2024-09-13T14:24:55.285566+0000","last_undegraded":"2024-09-13T14:24:55.285566+0000","last_fullsized":"2024-09-13T14:24:55.285566+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:21:11.486340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":81,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.287819+0000","last_change":"2024-09-13T14:24:55.287498+0000","last_active":"2024-09-13T14:24:55.287819+0000","last_peered":"2024-09-13T14:24:55.287819+0000","last_clean":"2024-09-13T14:24:55.287819+0000","last_became_active":"2024-09-13T14:24:55.285957+0000","last_became_peered":"2024-09-13T14:24:55.285957+0000","last_unstale":"2024-09-13T14:24:55.287819+0000","last_undegraded":"2024-09-13T14:24:55.287819+0000","last_fullsized":"2024-09-13T14:24:55.287819+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:21:50.821805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":70,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.269687+0000","last_change":"2024-09-13T14:23:51.609501+0000","last_active":"2024-09-13T14:24:54.269687+0000","last_peered":"2024-09-13T14:24:54.269687+0000","last_clean":"2024-09-13T14:24:54.269687+0000","last_became_active":"2024-09-13T14:23:51.609135+0000","last_became_peered":"2024-09-13T14:23:51.609135+0000","last_unstale":"2024-09-13T14:24:54.269687+0000","last_undegraded":"2024-09-13T14:24:54.269687+0000","last_fullsized":"2024-09-13T14:24:54.269687+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:21:25.318030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":70,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.271702+0000","last_change":"2024-09-13T14:24:23.775769+0000","last_active":"2024-09-13T14:24:55.271702+0000","last_peered":"2024-09-13T14:24:55.271702+0000","last_clean":"2024-09-13T14:24:55.271702+0000","last_became_active":"2024-09-13T14:24:23.775409+0000","last_became_peered":"2024-09-13T14:24:23.775409+0000","last_unstale":"2024-09-13T14:24:55.271702+0000","last_undegraded":"2024-09-13T14:24:55.271702+0000","last_fullsized":"2024-09-13T14:24:55.271702+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:20:09.966476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18272,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234184,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32912,"kb_used_data":1788,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712240,"statfs":{"total":95995035648,"available":95961333760,"internally_reserved":0,"allocated":1830912,"data_stored":1222916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495820,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36992,"kb_used_data":5876,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93708160,"statfs":{"total":95995035648,"available":95957155840,"internally_reserved":0,"allocated":6017024,"data_stored":5417151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757457,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36404,"kb_used_data":5288,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93708748,"statfs":{"total":95995035648,"available":95957757952,"internally_reserved":0,"allocated":5414912,"data_stored":4822551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019093,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28224,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716928,"statfs":{"total":95995035648,"available":95966134272,"internally_reserved":0,"allocated":1232896,"data_stored":632536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435277,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28728,"kb_used_data":1188,"kb_used_omap":5,"kb_used_meta":27514,"kb_avail":93716424,"statfs":{"total":95995035648,"available":95965618176,"internally_reserved":0,"allocated":1216512,"data_stored":628009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":28174462},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":104,"seq":446676598803,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28732,"kb_used_data":1192,"kb_used_omap":7,"kb_used_meta":27512,"kb_avail":93716420,"statfs":{"total":95995035648,"available":95965614080,"internally_reserved":0,"allocated":1220608,"data_stored":628535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8010,"internal_metadata":28172470},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":99,"seq":425201762330,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32812,"kb_used_data":5280,"kb_used_omap":7,"kb_used_meta":27512,"kb_avail":93712340,"statfs":{"total":95995035648,"available":95961436160,"internally_reserved":0,"allocated":5406720,"data_stored":4821969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7759,"internal_metadata":28172721},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726925857,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":29304,"kb_used_data":1772,"kb_used_omap":7,"kb_used_meta":27512,"kb_avail":93715848,"statfs":{"total":95995035648,"available":95965028352,"internally_reserved":0,"allocated":1814528,"data_stored":1218891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":28172584},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:25:53.773 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-13T14:25:53.773 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-13T14:25:53.773 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-13T14:25:53.774 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph health --format=json 2024-09-13T14:25:53.970 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:25:53.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:53 smithi195 ceph-mon[104262]: from='client.44465 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:25:54.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:54 smithi102 ceph-mon[110183]: pgmap v277: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:54.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:54 smithi102 ceph-mon[115269]: pgmap v277: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:54.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:54 smithi195 ceph-mon[104262]: pgmap v277: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:55.733 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:25:55.733 INFO:teuthology.orchestra.run.smithi102.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-13T14:25:55.733 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:55 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:25:55.733 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:55 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:25:55.734 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:55 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:25:55.734 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:55 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:25:55.898 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:55 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:25:55.898 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:55 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:25:56.401 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-13T14:25:56.401 INFO:teuthology.run_tasks:Running task print... 2024-09-13T14:25:56.413 INFO:teuthology.task.print:**** applying stress + thrashing to mixed-version cluster **** 2024-09-13T14:25:56.413 INFO:teuthology.run_tasks:Running task parallel... 2024-09-13T14:25:56.446 INFO:teuthology.task.parallel:starting parallel... 2024-09-13T14:25:56.447 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-09-13T14:25:56.447 INFO:teuthology.task.sequential:In sequential, running task thrashosds... 2024-09-13T14:25:56.449 INFO:tasks.thrashosds:config is {'aggressive_pg_num_changes': False, 'chance_force_recovery': 0, 'chance_pgnum_grow': 1, 'chance_pgpnum_fix': 1, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'chance_thrash_pg_upmap_items': 0, 'disable_objectstore_tool_tests': True, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-13T14:25:56.450 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-aclamk-cbt-combined'}, 'ceph': {'conf': {'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'create_rbd_pool': True, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'mons down', 'mon down', 'MON_DOWN', 'out of quorum', 'PG_AVAILABILITY'], 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-aclamk-cbt-combined', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}} 2024-09-13T14:25:56.450 INFO:tasks.thrashosds:config is {'aggressive_pg_num_changes': False, 'chance_force_recovery': 0, 'chance_pgnum_grow': 1, 'chance_pgpnum_fix': 1, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'chance_thrash_pg_upmap_items': 0, 'disable_objectstore_tool_tests': True, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-13T14:25:56.450 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-13T14:25:56.450 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-13T14:25:56.450 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:25:56.616 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:25:56 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:25:56] "GET /metrics HTTP/1.1" 200 34014 "" "Prometheus/2.51.0" 2024-09-13T14:25:56.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:56 smithi102 ceph-mon[110183]: pgmap v278: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:56.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:56 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3222043534' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-13T14:25:56.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:25:56.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:25:56.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:56 smithi102 ceph-mon[115269]: pgmap v278: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:56.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:56 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3222043534' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-13T14:25:56.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:25:56.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:25:56.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:56 smithi195 ceph-mon[104262]: pgmap v278: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:56.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:56 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3222043534' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-13T14:25:56.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:25:56.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:25:57.199 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:25:58.897 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:25:58.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:58 smithi102 ceph-mon[110183]: pgmap v279: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:58.898 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:58 smithi102 ceph-mon[115269]: pgmap v279: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:58.899 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:25:58.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:58 smithi195 ceph-mon[104262]: pgmap v279: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:25:59.562 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":279,"stamp":"2024-09-13T14:25:57.234793+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790245,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":6390,"num_read_kb":13845,"num_write":1123,"num_write_kb":15041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1182,"ondisk_log_size":1182,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":254108,"kb_used_data":23588,"kb_used_omap":35,"kb_used_meta":230364,"kb_avail":749707108,"statfs":{"total":767960285184,"available":767700078592,"internally_reserved":0,"allocated":24154112,"data_stored":19392558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36053,"internal_metadata":235893547},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002432"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":74,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.290727+0000","last_change":"2024-09-13T14:24:55.290644+0000","last_active":"2024-09-13T14:24:55.290727+0000","last_peered":"2024-09-13T14:24:55.290727+0000","last_clean":"2024-09-13T14:24:55.290727+0000","last_became_active":"2024-09-13T14:24:55.290222+0000","last_became_peered":"2024-09-13T14:24:55.290222+0000","last_unstale":"2024-09-13T14:24:55.290727+0000","last_undegraded":"2024-09-13T14:24:55.290727+0000","last_fullsized":"2024-09-13T14:24:55.290727+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:38:41.205619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":75,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.274419+0000","last_change":"2024-09-13T14:24:23.773702+0000","last_active":"2024-09-13T14:24:55.274419+0000","last_peered":"2024-09-13T14:24:55.274419+0000","last_clean":"2024-09-13T14:24:55.274419+0000","last_became_active":"2024-09-13T14:24:23.773290+0000","last_became_peered":"2024-09-13T14:24:23.773290+0000","last_unstale":"2024-09-13T14:24:55.274419+0000","last_undegraded":"2024-09-13T14:24:55.274419+0000","last_fullsized":"2024-09-13T14:24:55.274419+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:20.820970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":72,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.283071+0000","last_change":"2024-09-13T14:24:55.282407+0000","last_active":"2024-09-13T14:24:55.283071+0000","last_peered":"2024-09-13T14:24:55.283071+0000","last_clean":"2024-09-13T14:24:55.283071+0000","last_became_active":"2024-09-13T14:24:55.282083+0000","last_became_peered":"2024-09-13T14:24:55.282083+0000","last_unstale":"2024-09-13T14:24:55.283071+0000","last_undegraded":"2024-09-13T14:24:55.283071+0000","last_fullsized":"2024-09-13T14:24:55.283071+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:23:10.525707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":65,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.284816+0000","last_change":"2024-09-13T14:24:55.284433+0000","last_active":"2024-09-13T14:24:55.284816+0000","last_peered":"2024-09-13T14:24:55.284816+0000","last_clean":"2024-09-13T14:24:55.284816+0000","last_became_active":"2024-09-13T14:24:55.284143+0000","last_became_peered":"2024-09-13T14:24:55.284143+0000","last_unstale":"2024-09-13T14:24:55.284816+0000","last_undegraded":"2024-09-13T14:24:55.284816+0000","last_fullsized":"2024-09-13T14:24:55.284816+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:28.107405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":67,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.282846+0000","last_change":"2024-09-13T14:24:55.282416+0000","last_active":"2024-09-13T14:24:55.282846+0000","last_peered":"2024-09-13T14:24:55.282846+0000","last_clean":"2024-09-13T14:24:55.282846+0000","last_became_active":"2024-09-13T14:24:55.282045+0000","last_became_peered":"2024-09-13T14:24:55.282045+0000","last_unstale":"2024-09-13T14:24:55.282846+0000","last_undegraded":"2024-09-13T14:24:55.282846+0000","last_fullsized":"2024-09-13T14:24:55.282846+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:58.692897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":58,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.274055+0000","last_change":"2024-09-13T14:11:29.244452+0000","last_active":"2024-09-13T14:24:55.274055+0000","last_peered":"2024-09-13T14:24:55.274055+0000","last_clean":"2024-09-13T14:24:55.274055+0000","last_became_active":"2024-09-13T14:11:29.244102+0000","last_became_peered":"2024-09-13T14:11:29.244102+0000","last_unstale":"2024-09-13T14:24:55.274055+0000","last_undegraded":"2024-09-13T14:24:55.274055+0000","last_fullsized":"2024-09-13T14:24:55.274055+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:13:19.165534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":74,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.289698+0000","last_change":"2024-09-13T14:24:55.288871+0000","last_active":"2024-09-13T14:24:55.289698+0000","last_peered":"2024-09-13T14:24:55.289698+0000","last_clean":"2024-09-13T14:24:55.289698+0000","last_became_active":"2024-09-13T14:24:55.288506+0000","last_became_peered":"2024-09-13T14:24:55.288506+0000","last_unstale":"2024-09-13T14:24:55.289698+0000","last_undegraded":"2024-09-13T14:24:55.289698+0000","last_fullsized":"2024-09-13T14:24:55.289698+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:44.796657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":81,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.283649+0000","last_change":"2024-09-13T14:23:18.039061+0000","last_active":"2024-09-13T14:24:55.283649+0000","last_peered":"2024-09-13T14:24:55.283649+0000","last_clean":"2024-09-13T14:24:55.283649+0000","last_became_active":"2024-09-13T14:23:18.038661+0000","last_became_peered":"2024-09-13T14:23:18.038661+0000","last_unstale":"2024-09-13T14:24:55.283649+0000","last_undegraded":"2024-09-13T14:24:55.283649+0000","last_fullsized":"2024-09-13T14:24:55.283649+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:05:29.109844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":74,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.275099+0000","last_change":"2024-09-13T14:23:18.038597+0000","last_active":"2024-09-13T14:24:55.275099+0000","last_peered":"2024-09-13T14:24:55.275099+0000","last_clean":"2024-09-13T14:24:55.275099+0000","last_became_active":"2024-09-13T14:23:18.038240+0000","last_became_peered":"2024-09-13T14:23:18.038240+0000","last_unstale":"2024-09-13T14:24:55.275099+0000","last_undegraded":"2024-09-13T14:24:55.275099+0000","last_fullsized":"2024-09-13T14:24:55.275099+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:27:51.842195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":57,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.269670+0000","last_change":"2024-09-13T14:11:30.254534+0000","last_active":"2024-09-13T14:24:54.269670+0000","last_peered":"2024-09-13T14:24:54.269670+0000","last_clean":"2024-09-13T14:24:54.269670+0000","last_became_active":"2024-09-13T14:11:30.253959+0000","last_became_peered":"2024-09-13T14:11:30.253959+0000","last_unstale":"2024-09-13T14:24:54.269670+0000","last_undegraded":"2024-09-13T14:24:54.269670+0000","last_fullsized":"2024-09-13T14:24:54.269670+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:09:54.025853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":75,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.274762+0000","last_change":"2024-09-13T14:23:51.612105+0000","last_active":"2024-09-13T14:24:55.274762+0000","last_peered":"2024-09-13T14:24:55.274762+0000","last_clean":"2024-09-13T14:24:55.274762+0000","last_became_active":"2024-09-13T14:23:51.611740+0000","last_became_peered":"2024-09-13T14:23:51.611740+0000","last_unstale":"2024-09-13T14:24:55.274762+0000","last_undegraded":"2024-09-13T14:24:55.274762+0000","last_fullsized":"2024-09-13T14:24:55.274762+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:26:23.262096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":86,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.288950+0000","last_change":"2024-09-13T14:24:55.288950+0000","last_active":"2024-09-13T14:24:55.288950+0000","last_peered":"2024-09-13T14:24:55.288950+0000","last_clean":"2024-09-13T14:24:55.288950+0000","last_became_active":"2024-09-13T14:24:55.288671+0000","last_became_peered":"2024-09-13T14:24:55.288671+0000","last_unstale":"2024-09-13T14:24:55.288950+0000","last_undegraded":"2024-09-13T14:24:55.288950+0000","last_fullsized":"2024-09-13T14:24:55.288950+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:14:53.298390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":59,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.275139+0000","last_change":"2024-09-13T14:11:29.232897+0000","last_active":"2024-09-13T14:24:55.275139+0000","last_peered":"2024-09-13T14:24:55.275139+0000","last_clean":"2024-09-13T14:24:55.275139+0000","last_became_active":"2024-09-13T14:11:29.232637+0000","last_became_peered":"2024-09-13T14:11:29.232637+0000","last_unstale":"2024-09-13T14:24:55.275139+0000","last_undegraded":"2024-09-13T14:24:55.275139+0000","last_fullsized":"2024-09-13T14:24:55.275139+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:31:05.856046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":72,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.276706+0000","last_change":"2024-09-13T14:23:51.609167+0000","last_active":"2024-09-13T14:24:54.276706+0000","last_peered":"2024-09-13T14:24:54.276706+0000","last_clean":"2024-09-13T14:24:54.276706+0000","last_became_active":"2024-09-13T14:23:51.608811+0000","last_became_peered":"2024-09-13T14:23:51.608811+0000","last_unstale":"2024-09-13T14:24:54.276706+0000","last_undegraded":"2024-09-13T14:24:54.276706+0000","last_fullsized":"2024-09-13T14:24:54.276706+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:40.506763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":73,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.291355+0000","last_change":"2024-09-13T14:24:55.290798+0000","last_active":"2024-09-13T14:24:55.291355+0000","last_peered":"2024-09-13T14:24:55.291355+0000","last_clean":"2024-09-13T14:24:55.291355+0000","last_became_active":"2024-09-13T14:24:55.290510+0000","last_became_peered":"2024-09-13T14:24:55.290510+0000","last_unstale":"2024-09-13T14:24:55.291355+0000","last_undegraded":"2024-09-13T14:24:55.291355+0000","last_fullsized":"2024-09-13T14:24:55.291355+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:19:03.129132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":67,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.271982+0000","last_change":"2024-09-13T14:23:51.609082+0000","last_active":"2024-09-13T14:24:55.271982+0000","last_peered":"2024-09-13T14:24:55.271982+0000","last_clean":"2024-09-13T14:24:55.271982+0000","last_became_active":"2024-09-13T14:23:51.608799+0000","last_became_peered":"2024-09-13T14:23:51.608799+0000","last_unstale":"2024-09-13T14:24:55.271982+0000","last_undegraded":"2024-09-13T14:24:55.271982+0000","last_fullsized":"2024-09-13T14:24:55.271982+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:43:24.721453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":72,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.284590+0000","last_change":"2024-09-13T14:24:55.284499+0000","last_active":"2024-09-13T14:24:55.284590+0000","last_peered":"2024-09-13T14:24:55.284590+0000","last_clean":"2024-09-13T14:24:55.284590+0000","last_became_active":"2024-09-13T14:24:55.284096+0000","last_became_peered":"2024-09-13T14:24:55.284096+0000","last_unstale":"2024-09-13T14:24:55.284590+0000","last_undegraded":"2024-09-13T14:24:55.284590+0000","last_fullsized":"2024-09-13T14:24:55.284590+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:02:06.836099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":71,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.291592+0000","last_change":"2024-09-13T14:24:55.291371+0000","last_active":"2024-09-13T14:24:55.291592+0000","last_peered":"2024-09-13T14:24:55.291592+0000","last_clean":"2024-09-13T14:24:55.291592+0000","last_became_active":"2024-09-13T14:24:55.291047+0000","last_became_peered":"2024-09-13T14:24:55.291047+0000","last_unstale":"2024-09-13T14:24:55.291592+0000","last_undegraded":"2024-09-13T14:24:55.291592+0000","last_fullsized":"2024-09-13T14:24:55.291592+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:43.912412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":73,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.276868+0000","last_change":"2024-09-13T14:23:51.606589+0000","last_active":"2024-09-13T14:24:54.276868+0000","last_peered":"2024-09-13T14:24:54.276868+0000","last_clean":"2024-09-13T14:24:54.276868+0000","last_became_active":"2024-09-13T14:23:51.606244+0000","last_became_peered":"2024-09-13T14:23:51.606244+0000","last_unstale":"2024-09-13T14:24:54.276868+0000","last_undegraded":"2024-09-13T14:24:54.276868+0000","last_fullsized":"2024-09-13T14:24:54.276868+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:08.776526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":86,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.275537+0000","last_change":"2024-09-13T14:24:23.777891+0000","last_active":"2024-09-13T14:24:55.275537+0000","last_peered":"2024-09-13T14:24:55.275537+0000","last_clean":"2024-09-13T14:24:55.275537+0000","last_became_active":"2024-09-13T14:24:23.777578+0000","last_became_peered":"2024-09-13T14:24:23.777578+0000","last_unstale":"2024-09-13T14:24:55.275537+0000","last_undegraded":"2024-09-13T14:24:55.275537+0000","last_fullsized":"2024-09-13T14:24:55.275537+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:56:47.441579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":72,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.270580+0000","last_change":"2024-09-13T14:24:23.777970+0000","last_active":"2024-09-13T14:24:54.270580+0000","last_peered":"2024-09-13T14:24:54.270580+0000","last_clean":"2024-09-13T14:24:54.270580+0000","last_became_active":"2024-09-13T14:24:23.777650+0000","last_became_peered":"2024-09-13T14:24:23.777650+0000","last_unstale":"2024-09-13T14:24:54.270580+0000","last_undegraded":"2024-09-13T14:24:54.270580+0000","last_fullsized":"2024-09-13T14:24:54.270580+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:50:00.549905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":71,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.274746+0000","last_change":"2024-09-13T14:23:51.610572+0000","last_active":"2024-09-13T14:24:55.274746+0000","last_peered":"2024-09-13T14:24:55.274746+0000","last_clean":"2024-09-13T14:24:55.274746+0000","last_became_active":"2024-09-13T14:23:51.610273+0000","last_became_peered":"2024-09-13T14:23:51.610273+0000","last_unstale":"2024-09-13T14:24:55.274746+0000","last_undegraded":"2024-09-13T14:24:55.274746+0000","last_fullsized":"2024-09-13T14:24:55.274746+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:17:10.574417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":75,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.281133+0000","last_change":"2024-09-13T14:24:55.281042+0000","last_active":"2024-09-13T14:24:55.281133+0000","last_peered":"2024-09-13T14:24:55.281133+0000","last_clean":"2024-09-13T14:24:55.281133+0000","last_became_active":"2024-09-13T14:24:55.280741+0000","last_became_peered":"2024-09-13T14:24:55.280741+0000","last_unstale":"2024-09-13T14:24:55.281133+0000","last_undegraded":"2024-09-13T14:24:55.281133+0000","last_fullsized":"2024-09-13T14:24:55.281133+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:45:40.787103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":46,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.289331+0000","last_change":"2024-09-13T14:24:55.289245+0000","last_active":"2024-09-13T14:24:55.289331+0000","last_peered":"2024-09-13T14:24:55.289331+0000","last_clean":"2024-09-13T14:24:55.289331+0000","last_became_active":"2024-09-13T14:24:55.288881+0000","last_became_peered":"2024-09-13T14:24:55.288881+0000","last_unstale":"2024-09-13T14:24:55.289331+0000","last_undegraded":"2024-09-13T14:24:55.289331+0000","last_fullsized":"2024-09-13T14:24:55.289331+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:14:12.194309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":404,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.274086+0000","last_change":"2024-09-13T14:23:18.032973+0000","last_active":"2024-09-13T14:24:55.274086+0000","last_peered":"2024-09-13T14:24:55.274086+0000","last_clean":"2024-09-13T14:24:55.274086+0000","last_became_active":"2024-09-13T14:23:18.032579+0000","last_became_peered":"2024-09-13T14:23:18.032579+0000","last_unstale":"2024-09-13T14:24:55.274086+0000","last_undegraded":"2024-09-13T14:24:55.274086+0000","last_fullsized":"2024-09-13T14:24:55.274086+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:07:41.927740+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:17.109445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":400,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.270769+0000","last_change":"2024-09-13T14:24:23.776868+0000","last_active":"2024-09-13T14:24:54.270769+0000","last_peered":"2024-09-13T14:24:54.270769+0000","last_clean":"2024-09-13T14:24:54.270769+0000","last_became_active":"2024-09-13T14:24:23.776306+0000","last_became_peered":"2024-09-13T14:24:23.776306+0000","last_unstale":"2024-09-13T14:24:54.270769+0000","last_undegraded":"2024-09-13T14:24:54.270769+0000","last_fullsized":"2024-09-13T14:24:54.270769+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:28:41.805918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":23,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.288138+0000","last_change":"2024-09-13T14:24:55.288058+0000","last_active":"2024-09-13T14:24:55.288138+0000","last_peered":"2024-09-13T14:24:55.288138+0000","last_clean":"2024-09-13T14:24:55.288138+0000","last_became_active":"2024-09-13T14:24:55.287601+0000","last_became_peered":"2024-09-13T14:24:55.287601+0000","last_unstale":"2024-09-13T14:24:55.288138+0000","last_undegraded":"2024-09-13T14:24:55.288138+0000","last_fullsized":"2024-09-13T14:24:55.288138+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:56:34.622696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":57,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.277982+0000","last_change":"2024-09-13T14:11:30.253749+0000","last_active":"2024-09-13T14:24:54.277982+0000","last_peered":"2024-09-13T14:24:54.277982+0000","last_clean":"2024-09-13T14:24:54.277982+0000","last_became_active":"2024-09-13T14:11:30.253499+0000","last_became_peered":"2024-09-13T14:11:30.253499+0000","last_unstale":"2024-09-13T14:24:54.277982+0000","last_undegraded":"2024-09-13T14:24:54.277982+0000","last_fullsized":"2024-09-13T14:24:54.277982+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:02:00.987335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":289,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.275098+0000","last_change":"2024-09-13T14:23:51.608681+0000","last_active":"2024-09-13T14:24:54.275098+0000","last_peered":"2024-09-13T14:24:54.275098+0000","last_clean":"2024-09-13T14:24:54.275098+0000","last_became_active":"2024-09-13T14:23:51.608223+0000","last_became_peered":"2024-09-13T14:23:51.608223+0000","last_unstale":"2024-09-13T14:24:54.275098+0000","last_undegraded":"2024-09-13T14:24:54.275098+0000","last_fullsized":"2024-09-13T14:24:54.275098+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:13:15.930987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.286928+0000","last_change":"2024-09-13T14:24:55.286576+0000","last_active":"2024-09-13T14:24:55.286928+0000","last_peered":"2024-09-13T14:24:55.286928+0000","last_clean":"2024-09-13T14:24:55.286928+0000","last_became_active":"2024-09-13T14:24:55.286284+0000","last_became_peered":"2024-09-13T14:24:55.286284+0000","last_unstale":"2024-09-13T14:24:55.286928+0000","last_undegraded":"2024-09-13T14:24:55.286928+0000","last_fullsized":"2024-09-13T14:24:55.286928+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:35.589736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":78,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.271821+0000","last_change":"2024-09-13T14:24:23.776557+0000","last_active":"2024-09-13T14:24:55.271821+0000","last_peered":"2024-09-13T14:24:55.271821+0000","last_clean":"2024-09-13T14:24:55.271821+0000","last_became_active":"2024-09-13T14:24:23.775204+0000","last_became_peered":"2024-09-13T14:24:23.775204+0000","last_unstale":"2024-09-13T14:24:55.271821+0000","last_undegraded":"2024-09-13T14:24:55.271821+0000","last_fullsized":"2024-09-13T14:24:55.271821+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:18:21.095561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":494,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.285398+0000","last_change":"2024-09-13T14:24:55.285288+0000","last_active":"2024-09-13T14:24:55.285398+0000","last_peered":"2024-09-13T14:24:55.285398+0000","last_clean":"2024-09-13T14:24:55.285398+0000","last_became_active":"2024-09-13T14:24:55.284733+0000","last_became_peered":"2024-09-13T14:24:55.284733+0000","last_unstale":"2024-09-13T14:24:55.285398+0000","last_undegraded":"2024-09-13T14:24:55.285398+0000","last_fullsized":"2024-09-13T14:24:55.285398+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:51:50.420796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":55,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.276363+0000","last_change":"2024-09-13T14:24:23.779725+0000","last_active":"2024-09-13T14:24:55.276363+0000","last_peered":"2024-09-13T14:24:55.276363+0000","last_clean":"2024-09-13T14:24:55.276363+0000","last_became_active":"2024-09-13T14:24:23.779418+0000","last_became_peered":"2024-09-13T14:24:23.779418+0000","last_unstale":"2024-09-13T14:24:55.276363+0000","last_undegraded":"2024-09-13T14:24:55.276363+0000","last_fullsized":"2024-09-13T14:24:55.276363+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:48:08.470127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":89,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.276842+0000","last_change":"2024-09-13T14:23:51.608576+0000","last_active":"2024-09-13T14:24:55.276842+0000","last_peered":"2024-09-13T14:24:55.276842+0000","last_clean":"2024-09-13T14:24:55.276842+0000","last_became_active":"2024-09-13T14:23:51.607956+0000","last_became_peered":"2024-09-13T14:23:51.607956+0000","last_unstale":"2024-09-13T14:24:55.276842+0000","last_undegraded":"2024-09-13T14:24:55.276842+0000","last_fullsized":"2024-09-13T14:24:55.276842+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:53.399552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":829,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.272095+0000","last_change":"2024-09-13T14:23:51.606335+0000","last_active":"2024-09-13T14:24:54.272095+0000","last_peered":"2024-09-13T14:24:54.272095+0000","last_clean":"2024-09-13T14:24:54.272095+0000","last_became_active":"2024-09-13T14:23:51.605905+0000","last_became_peered":"2024-09-13T14:23:51.605905+0000","last_unstale":"2024-09-13T14:24:54.272095+0000","last_undegraded":"2024-09-13T14:24:54.272095+0000","last_fullsized":"2024-09-13T14:24:54.272095+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:22:42.470188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":72,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.272867+0000","last_change":"2024-09-13T14:23:18.042891+0000","last_active":"2024-09-13T14:24:54.272867+0000","last_peered":"2024-09-13T14:24:54.272867+0000","last_clean":"2024-09-13T14:24:54.272867+0000","last_became_active":"2024-09-13T14:23:18.042562+0000","last_became_peered":"2024-09-13T14:23:18.042562+0000","last_unstale":"2024-09-13T14:24:54.272867+0000","last_undegraded":"2024-09-13T14:24:54.272867+0000","last_fullsized":"2024-09-13T14:24:54.272867+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:30:42.996661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":84,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.291080+0000","last_change":"2024-09-13T14:24:55.291080+0000","last_active":"2024-09-13T14:24:55.291080+0000","last_peered":"2024-09-13T14:24:55.291080+0000","last_clean":"2024-09-13T14:24:55.291080+0000","last_became_active":"2024-09-13T14:24:55.290560+0000","last_became_peered":"2024-09-13T14:24:55.290560+0000","last_unstale":"2024-09-13T14:24:55.291080+0000","last_undegraded":"2024-09-13T14:24:55.291080+0000","last_fullsized":"2024-09-13T14:24:55.291080+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:16.041382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":88,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.282619+0000","last_change":"2024-09-13T14:24:55.282619+0000","last_active":"2024-09-13T14:24:55.282619+0000","last_peered":"2024-09-13T14:24:55.282619+0000","last_clean":"2024-09-13T14:24:55.282619+0000","last_became_active":"2024-09-13T14:24:55.282110+0000","last_became_peered":"2024-09-13T14:24:55.282110+0000","last_unstale":"2024-09-13T14:24:55.282619+0000","last_undegraded":"2024-09-13T14:24:55.282619+0000","last_fullsized":"2024-09-13T14:24:55.282619+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:33:19.425344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":778,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.277098+0000","last_change":"2024-09-13T14:23:51.610302+0000","last_active":"2024-09-13T14:24:55.277098+0000","last_peered":"2024-09-13T14:24:55.277098+0000","last_clean":"2024-09-13T14:24:55.277098+0000","last_became_active":"2024-09-13T14:23:51.607653+0000","last_became_peered":"2024-09-13T14:23:51.607653+0000","last_unstale":"2024-09-13T14:24:55.277098+0000","last_undegraded":"2024-09-13T14:24:55.277098+0000","last_fullsized":"2024-09-13T14:24:55.277098+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:03:52.131651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":46,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.275869+0000","last_change":"2024-09-13T14:23:51.610321+0000","last_active":"2024-09-13T14:24:55.275869+0000","last_peered":"2024-09-13T14:24:55.275869+0000","last_clean":"2024-09-13T14:24:55.275869+0000","last_became_active":"2024-09-13T14:23:51.607733+0000","last_became_peered":"2024-09-13T14:23:51.607733+0000","last_unstale":"2024-09-13T14:24:55.275869+0000","last_undegraded":"2024-09-13T14:24:55.275869+0000","last_fullsized":"2024-09-13T14:24:55.275869+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:58:05.246186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":55,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.277007+0000","last_change":"2024-09-13T14:24:23.779678+0000","last_active":"2024-09-13T14:24:55.277007+0000","last_peered":"2024-09-13T14:24:55.277007+0000","last_clean":"2024-09-13T14:24:55.277007+0000","last_became_active":"2024-09-13T14:24:23.779343+0000","last_became_peered":"2024-09-13T14:24:23.779343+0000","last_unstale":"2024-09-13T14:24:55.277007+0000","last_undegraded":"2024-09-13T14:24:55.277007+0000","last_fullsized":"2024-09-13T14:24:55.277007+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:14:41.230951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":595,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.275237+0000","last_change":"2024-09-13T14:23:18.038847+0000","last_active":"2024-09-13T14:24:55.275237+0000","last_peered":"2024-09-13T14:24:55.275237+0000","last_clean":"2024-09-13T14:24:55.275237+0000","last_became_active":"2024-09-13T14:23:18.038188+0000","last_became_peered":"2024-09-13T14:23:18.038188+0000","last_unstale":"2024-09-13T14:24:55.275237+0000","last_undegraded":"2024-09-13T14:24:55.275237+0000","last_fullsized":"2024-09-13T14:24:55.275237+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:30:37.213496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":56,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.271836+0000","last_change":"2024-09-13T14:23:18.034427+0000","last_active":"2024-09-13T14:24:55.271836+0000","last_peered":"2024-09-13T14:24:55.271836+0000","last_clean":"2024-09-13T14:24:55.271836+0000","last_became_active":"2024-09-13T14:23:18.034092+0000","last_became_peered":"2024-09-13T14:23:18.034092+0000","last_unstale":"2024-09-13T14:24:55.271836+0000","last_undegraded":"2024-09-13T14:24:55.271836+0000","last_fullsized":"2024-09-13T14:24:55.271836+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:21.755115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":72,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.290792+0000","last_change":"2024-09-13T14:24:55.290711+0000","last_active":"2024-09-13T14:24:55.290792+0000","last_peered":"2024-09-13T14:24:55.290792+0000","last_clean":"2024-09-13T14:24:55.290792+0000","last_became_active":"2024-09-13T14:24:55.288549+0000","last_became_peered":"2024-09-13T14:24:55.288549+0000","last_unstale":"2024-09-13T14:24:55.290792+0000","last_undegraded":"2024-09-13T14:24:55.290792+0000","last_fullsized":"2024-09-13T14:24:55.290792+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:55:21.763931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":502,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.276663+0000","last_change":"2024-09-13T14:23:51.606481+0000","last_active":"2024-09-13T14:24:55.276663+0000","last_peered":"2024-09-13T14:24:55.276663+0000","last_clean":"2024-09-13T14:24:55.276663+0000","last_became_active":"2024-09-13T14:23:51.605783+0000","last_became_peered":"2024-09-13T14:23:51.605783+0000","last_unstale":"2024-09-13T14:24:55.276663+0000","last_undegraded":"2024-09-13T14:24:55.276663+0000","last_fullsized":"2024-09-13T14:24:55.276663+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:09:01.265713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":69,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.281723+0000","last_change":"2024-09-13T14:24:55.281593+0000","last_active":"2024-09-13T14:24:55.281723+0000","last_peered":"2024-09-13T14:24:55.281723+0000","last_clean":"2024-09-13T14:24:55.281723+0000","last_became_active":"2024-09-13T14:24:55.280805+0000","last_became_peered":"2024-09-13T14:24:55.280805+0000","last_unstale":"2024-09-13T14:24:55.281723+0000","last_undegraded":"2024-09-13T14:24:55.281723+0000","last_fullsized":"2024-09-13T14:24:55.281723+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:14.806713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":70,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.271380+0000","last_change":"2024-09-13T14:23:51.609769+0000","last_active":"2024-09-13T14:24:55.271380+0000","last_peered":"2024-09-13T14:24:55.271380+0000","last_clean":"2024-09-13T14:24:55.271380+0000","last_became_active":"2024-09-13T14:23:51.609343+0000","last_became_peered":"2024-09-13T14:23:51.609343+0000","last_unstale":"2024-09-13T14:24:55.271380+0000","last_undegraded":"2024-09-13T14:24:55.271380+0000","last_fullsized":"2024-09-13T14:24:55.271380+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:06:58.544473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":514,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.282977+0000","last_change":"2024-09-13T14:24:23.775768+0000","last_active":"2024-09-13T14:24:55.282977+0000","last_peered":"2024-09-13T14:24:55.282977+0000","last_clean":"2024-09-13T14:24:55.282977+0000","last_became_active":"2024-09-13T14:24:23.775261+0000","last_became_peered":"2024-09-13T14:24:23.775261+0000","last_unstale":"2024-09-13T14:24:55.282977+0000","last_undegraded":"2024-09-13T14:24:55.282977+0000","last_fullsized":"2024-09-13T14:24:55.282977+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:30:05.340603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":56,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.270378+0000","last_change":"2024-09-13T14:11:30.256034+0000","last_active":"2024-09-13T14:24:54.270378+0000","last_peered":"2024-09-13T14:24:54.270378+0000","last_clean":"2024-09-13T14:24:54.270378+0000","last_became_active":"2024-09-13T14:11:30.255727+0000","last_became_peered":"2024-09-13T14:11:30.255727+0000","last_unstale":"2024-09-13T14:24:54.270378+0000","last_undegraded":"2024-09-13T14:24:54.270378+0000","last_fullsized":"2024-09-13T14:24:54.270378+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:21:35.817014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":71,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.274287+0000","last_change":"2024-09-13T14:23:51.606397+0000","last_active":"2024-09-13T14:24:55.274287+0000","last_peered":"2024-09-13T14:24:55.274287+0000","last_clean":"2024-09-13T14:24:55.274287+0000","last_became_active":"2024-09-13T14:23:51.605792+0000","last_became_peered":"2024-09-13T14:23:51.605792+0000","last_unstale":"2024-09-13T14:24:55.274287+0000","last_undegraded":"2024-09-13T14:24:55.274287+0000","last_fullsized":"2024-09-13T14:24:55.274287+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:42:44.617374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":80,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.275351+0000","last_change":"2024-09-13T14:11:29.237013+0000","last_active":"2024-09-13T14:24:55.275351+0000","last_peered":"2024-09-13T14:24:55.275351+0000","last_clean":"2024-09-13T14:24:55.275351+0000","last_became_active":"2024-09-13T14:11:29.236643+0000","last_became_peered":"2024-09-13T14:11:29.236643+0000","last_unstale":"2024-09-13T14:24:55.275351+0000","last_undegraded":"2024-09-13T14:24:55.275351+0000","last_fullsized":"2024-09-13T14:24:55.275351+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:33:58.438693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":56,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.275681+0000","last_change":"2024-09-13T14:24:23.779556+0000","last_active":"2024-09-13T14:24:55.275681+0000","last_peered":"2024-09-13T14:24:55.275681+0000","last_clean":"2024-09-13T14:24:55.275681+0000","last_became_active":"2024-09-13T14:24:23.779234+0000","last_became_peered":"2024-09-13T14:24:23.779234+0000","last_unstale":"2024-09-13T14:24:55.275681+0000","last_undegraded":"2024-09-13T14:24:55.275681+0000","last_fullsized":"2024-09-13T14:24:55.275681+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:40.733288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":76,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.287585+0000","last_change":"2024-09-13T14:24:55.287275+0000","last_active":"2024-09-13T14:24:55.287585+0000","last_peered":"2024-09-13T14:24:55.287585+0000","last_clean":"2024-09-13T14:24:55.287585+0000","last_became_active":"2024-09-13T14:24:55.286919+0000","last_became_peered":"2024-09-13T14:24:55.286919+0000","last_unstale":"2024-09-13T14:24:55.287585+0000","last_undegraded":"2024-09-13T14:24:55.287585+0000","last_fullsized":"2024-09-13T14:24:55.287585+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:10.790711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":73,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.272716+0000","last_change":"2024-09-13T14:23:51.610941+0000","last_active":"2024-09-13T14:24:55.272716+0000","last_peered":"2024-09-13T14:24:55.272716+0000","last_clean":"2024-09-13T14:24:55.272716+0000","last_became_active":"2024-09-13T14:23:51.610587+0000","last_became_peered":"2024-09-13T14:23:51.610587+0000","last_unstale":"2024-09-13T14:24:55.272716+0000","last_undegraded":"2024-09-13T14:24:55.272716+0000","last_fullsized":"2024-09-13T14:24:55.272716+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:48:01.436607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.288101+0000","last_change":"2024-09-13T14:24:55.287509+0000","last_active":"2024-09-13T14:24:55.288101+0000","last_peered":"2024-09-13T14:24:55.288101+0000","last_clean":"2024-09-13T14:24:55.288101+0000","last_became_active":"2024-09-13T14:24:55.287193+0000","last_became_peered":"2024-09-13T14:24:55.287193+0000","last_unstale":"2024-09-13T14:24:55.288101+0000","last_undegraded":"2024-09-13T14:24:55.288101+0000","last_fullsized":"2024-09-13T14:24:55.288101+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:14:10.603307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.289393+0000","last_change":"2024-09-13T14:24:55.289055+0000","last_active":"2024-09-13T14:24:55.289393+0000","last_peered":"2024-09-13T14:24:55.289393+0000","last_clean":"2024-09-13T14:24:55.289393+0000","last_became_active":"2024-09-13T14:24:55.288762+0000","last_became_peered":"2024-09-13T14:24:55.288762+0000","last_unstale":"2024-09-13T14:24:55.289393+0000","last_undegraded":"2024-09-13T14:24:55.289393+0000","last_fullsized":"2024-09-13T14:24:55.289393+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:37:26.639877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":74,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.287280+0000","last_change":"2024-09-13T14:24:55.287211+0000","last_active":"2024-09-13T14:24:55.287280+0000","last_peered":"2024-09-13T14:24:55.287280+0000","last_clean":"2024-09-13T14:24:55.287280+0000","last_became_active":"2024-09-13T14:24:55.286301+0000","last_became_peered":"2024-09-13T14:24:55.286301+0000","last_unstale":"2024-09-13T14:24:55.287280+0000","last_undegraded":"2024-09-13T14:24:55.287280+0000","last_fullsized":"2024-09-13T14:24:55.287280+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:32:41.099046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":72,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.276450+0000","last_change":"2024-09-13T14:23:51.606509+0000","last_active":"2024-09-13T14:24:54.276450+0000","last_peered":"2024-09-13T14:24:54.276450+0000","last_clean":"2024-09-13T14:24:54.276450+0000","last_became_active":"2024-09-13T14:23:51.606195+0000","last_became_peered":"2024-09-13T14:23:51.606195+0000","last_unstale":"2024-09-13T14:24:54.276450+0000","last_undegraded":"2024-09-13T14:24:54.276450+0000","last_fullsized":"2024-09-13T14:24:54.276450+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:29:49.729550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":90,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.280528+0000","last_change":"2024-09-13T14:24:55.280438+0000","last_active":"2024-09-13T14:24:55.280528+0000","last_peered":"2024-09-13T14:24:55.280528+0000","last_clean":"2024-09-13T14:24:55.280528+0000","last_became_active":"2024-09-13T14:24:55.279359+0000","last_became_peered":"2024-09-13T14:24:55.279359+0000","last_unstale":"2024-09-13T14:24:55.280528+0000","last_undegraded":"2024-09-13T14:24:55.280528+0000","last_fullsized":"2024-09-13T14:24:55.280528+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:12:48.432924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":67,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.284867+0000","last_change":"2024-09-13T14:24:55.283070+0000","last_active":"2024-09-13T14:24:55.284867+0000","last_peered":"2024-09-13T14:24:55.284867+0000","last_clean":"2024-09-13T14:24:55.284867+0000","last_became_active":"2024-09-13T14:24:55.282591+0000","last_became_peered":"2024-09-13T14:24:55.282591+0000","last_unstale":"2024-09-13T14:24:55.284867+0000","last_undegraded":"2024-09-13T14:24:55.284867+0000","last_fullsized":"2024-09-13T14:24:55.284867+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:35:21.896294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":55,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.289919+0000","last_change":"2024-09-13T14:24:55.289828+0000","last_active":"2024-09-13T14:24:55.289919+0000","last_peered":"2024-09-13T14:24:55.289919+0000","last_clean":"2024-09-13T14:24:55.289919+0000","last_became_active":"2024-09-13T14:24:55.289525+0000","last_became_peered":"2024-09-13T14:24:55.289525+0000","last_unstale":"2024-09-13T14:24:55.289919+0000","last_undegraded":"2024-09-13T14:24:55.289919+0000","last_fullsized":"2024-09-13T14:24:55.289919+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:25:39.041539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":87,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.284789+0000","last_change":"2024-09-13T14:24:55.284018+0000","last_active":"2024-09-13T14:24:55.284789+0000","last_peered":"2024-09-13T14:24:55.284789+0000","last_clean":"2024-09-13T14:24:55.284789+0000","last_became_active":"2024-09-13T14:24:55.283571+0000","last_became_peered":"2024-09-13T14:24:55.283571+0000","last_unstale":"2024-09-13T14:24:55.284789+0000","last_undegraded":"2024-09-13T14:24:55.284789+0000","last_fullsized":"2024-09-13T14:24:55.284789+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:48.890356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":87,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.270277+0000","last_change":"2024-09-13T14:24:23.778031+0000","last_active":"2024-09-13T14:24:54.270277+0000","last_peered":"2024-09-13T14:24:54.270277+0000","last_clean":"2024-09-13T14:24:54.270277+0000","last_became_active":"2024-09-13T14:24:23.777665+0000","last_became_peered":"2024-09-13T14:24:23.777665+0000","last_unstale":"2024-09-13T14:24:54.270277+0000","last_undegraded":"2024-09-13T14:24:54.270277+0000","last_fullsized":"2024-09-13T14:24:54.270277+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:27:54.790142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":94,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.290937+0000","last_change":"2024-09-13T14:24:55.290839+0000","last_active":"2024-09-13T14:24:55.290937+0000","last_peered":"2024-09-13T14:24:55.290937+0000","last_clean":"2024-09-13T14:24:55.290937+0000","last_became_active":"2024-09-13T14:24:55.290348+0000","last_became_peered":"2024-09-13T14:24:55.290348+0000","last_unstale":"2024-09-13T14:24:55.290937+0000","last_undegraded":"2024-09-13T14:24:55.290937+0000","last_fullsized":"2024-09-13T14:24:55.290937+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:11:38.467219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":59,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.271734+0000","last_change":"2024-09-13T14:23:18.035101+0000","last_active":"2024-09-13T14:24:55.271734+0000","last_peered":"2024-09-13T14:24:55.271734+0000","last_clean":"2024-09-13T14:24:55.271734+0000","last_became_active":"2024-09-13T14:23:18.034724+0000","last_became_peered":"2024-09-13T14:23:18.034724+0000","last_unstale":"2024-09-13T14:24:55.271734+0000","last_undegraded":"2024-09-13T14:24:55.271734+0000","last_fullsized":"2024-09-13T14:24:55.271734+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:39.358144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":85,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.282346+0000","last_change":"2024-09-13T14:24:55.282346+0000","last_active":"2024-09-13T14:24:55.282346+0000","last_peered":"2024-09-13T14:24:55.282346+0000","last_clean":"2024-09-13T14:24:55.282346+0000","last_became_active":"2024-09-13T14:24:55.281988+0000","last_became_peered":"2024-09-13T14:24:55.281988+0000","last_unstale":"2024-09-13T14:24:55.282346+0000","last_undegraded":"2024-09-13T14:24:55.282346+0000","last_fullsized":"2024-09-13T14:24:55.282346+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:05:35.737441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":74,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.270907+0000","last_change":"2024-09-13T14:23:51.612637+0000","last_active":"2024-09-13T14:24:54.270907+0000","last_peered":"2024-09-13T14:24:54.270907+0000","last_clean":"2024-09-13T14:24:54.270907+0000","last_became_active":"2024-09-13T14:23:51.612175+0000","last_became_peered":"2024-09-13T14:23:51.612175+0000","last_unstale":"2024-09-13T14:24:54.270907+0000","last_undegraded":"2024-09-13T14:24:54.270907+0000","last_fullsized":"2024-09-13T14:24:54.270907+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:52:30.299317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":85,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.285384+0000","last_change":"2024-09-13T14:24:55.285384+0000","last_active":"2024-09-13T14:24:55.285384+0000","last_peered":"2024-09-13T14:24:55.285384+0000","last_clean":"2024-09-13T14:24:55.285384+0000","last_became_active":"2024-09-13T14:24:55.285081+0000","last_became_peered":"2024-09-13T14:24:55.285081+0000","last_unstale":"2024-09-13T14:24:55.285384+0000","last_undegraded":"2024-09-13T14:24:55.285384+0000","last_fullsized":"2024-09-13T14:24:55.285384+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:47:07.946120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":59,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.272758+0000","last_change":"2024-09-13T14:11:29.240351+0000","last_active":"2024-09-13T14:24:54.272758+0000","last_peered":"2024-09-13T14:24:54.272758+0000","last_clean":"2024-09-13T14:24:54.272758+0000","last_became_active":"2024-09-13T14:11:29.240060+0000","last_became_peered":"2024-09-13T14:11:29.240060+0000","last_unstale":"2024-09-13T14:24:54.272758+0000","last_undegraded":"2024-09-13T14:24:54.272758+0000","last_fullsized":"2024-09-13T14:24:54.272758+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:34:36.126368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":93,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.285566+0000","last_change":"2024-09-13T14:24:55.285488+0000","last_active":"2024-09-13T14:24:55.285566+0000","last_peered":"2024-09-13T14:24:55.285566+0000","last_clean":"2024-09-13T14:24:55.285566+0000","last_became_active":"2024-09-13T14:24:55.283433+0000","last_became_peered":"2024-09-13T14:24:55.283433+0000","last_unstale":"2024-09-13T14:24:55.285566+0000","last_undegraded":"2024-09-13T14:24:55.285566+0000","last_fullsized":"2024-09-13T14:24:55.285566+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:21:11.486340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":81,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.287819+0000","last_change":"2024-09-13T14:24:55.287498+0000","last_active":"2024-09-13T14:24:55.287819+0000","last_peered":"2024-09-13T14:24:55.287819+0000","last_clean":"2024-09-13T14:24:55.287819+0000","last_became_active":"2024-09-13T14:24:55.285957+0000","last_became_peered":"2024-09-13T14:24:55.285957+0000","last_unstale":"2024-09-13T14:24:55.287819+0000","last_undegraded":"2024-09-13T14:24:55.287819+0000","last_fullsized":"2024-09-13T14:24:55.287819+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:21:50.821805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":70,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.269687+0000","last_change":"2024-09-13T14:23:51.609501+0000","last_active":"2024-09-13T14:24:54.269687+0000","last_peered":"2024-09-13T14:24:54.269687+0000","last_clean":"2024-09-13T14:24:54.269687+0000","last_became_active":"2024-09-13T14:23:51.609135+0000","last_became_peered":"2024-09-13T14:23:51.609135+0000","last_unstale":"2024-09-13T14:24:54.269687+0000","last_undegraded":"2024-09-13T14:24:54.269687+0000","last_fullsized":"2024-09-13T14:24:54.269687+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:21:25.318030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":70,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.271702+0000","last_change":"2024-09-13T14:24:23.775769+0000","last_active":"2024-09-13T14:24:55.271702+0000","last_peered":"2024-09-13T14:24:55.271702+0000","last_clean":"2024-09-13T14:24:55.271702+0000","last_became_active":"2024-09-13T14:24:23.775409+0000","last_became_peered":"2024-09-13T14:24:23.775409+0000","last_unstale":"2024-09-13T14:24:55.271702+0000","last_undegraded":"2024-09-13T14:24:55.271702+0000","last_fullsized":"2024-09-13T14:24:55.271702+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:20:09.966476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18272,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234185,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32912,"kb_used_data":1788,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712240,"statfs":{"total":95995035648,"available":95961333760,"internally_reserved":0,"allocated":1830912,"data_stored":1222916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495821,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36992,"kb_used_data":5876,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93708160,"statfs":{"total":95995035648,"available":95957155840,"internally_reserved":0,"allocated":6017024,"data_stored":5417151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757458,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36404,"kb_used_data":5288,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93708748,"statfs":{"total":95995035648,"available":95957757952,"internally_reserved":0,"allocated":5414912,"data_stored":4822551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019094,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28224,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716928,"statfs":{"total":95995035648,"available":95966134272,"internally_reserved":0,"allocated":1232896,"data_stored":632536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435278,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28728,"kb_used_data":1188,"kb_used_omap":5,"kb_used_meta":27514,"kb_avail":93716424,"statfs":{"total":95995035648,"available":95965618176,"internally_reserved":0,"allocated":1216512,"data_stored":628009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":28174462},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":104,"seq":446676598805,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28732,"kb_used_data":1192,"kb_used_omap":7,"kb_used_meta":27512,"kb_avail":93716420,"statfs":{"total":95995035648,"available":95965614080,"internally_reserved":0,"allocated":1220608,"data_stored":628535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8010,"internal_metadata":28172470},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":99,"seq":425201762331,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32812,"kb_used_data":5280,"kb_used_omap":7,"kb_used_meta":27512,"kb_avail":93712340,"statfs":{"total":95995035648,"available":95961436160,"internally_reserved":0,"allocated":5406720,"data_stored":4821969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7759,"internal_metadata":28172721},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726925858,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":29304,"kb_used_data":1772,"kb_used_omap":7,"kb_used_meta":27512,"kb_avail":93715848,"statfs":{"total":95995035648,"available":95965028352,"internally_reserved":0,"allocated":1814528,"data_stored":1218891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":28172584},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:25:59.566 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:25:59.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:25:59 smithi102 ceph-mon[110183]: from='client.44477 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:25:59.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:25:59 smithi102 ceph-mon[115269]: from='client.44477 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:25:59.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:25:59 smithi195 ceph-mon[104262]: from='client.44477 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:26:00.351 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:26:01.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:00 smithi102 ceph-mon[115269]: pgmap v280: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:26:01.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:00 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:26:01.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:00 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:26:01.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:00 smithi102 ceph-mon[110183]: pgmap v280: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:26:01.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:00 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:26:01.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:00 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:26:01.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:00 smithi195 ceph-mon[104262]: pgmap v280: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:26:01.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:00 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:26:01.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:00 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:26:02.376 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:26:02.377 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:26:02.997 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:02 smithi102 ceph-mon[110183]: pgmap v281: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:26:02.997 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:02 smithi102 ceph-mon[115269]: pgmap v281: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:26:03.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:02 smithi195 ceph-mon[104262]: pgmap v281: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:26:03.310 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":281,"stamp":"2024-09-13T14:26:01.235804+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790245,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":6390,"num_read_kb":13845,"num_write":1123,"num_write_kb":15041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1182,"ondisk_log_size":1182,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":254108,"kb_used_data":23588,"kb_used_omap":35,"kb_used_meta":230364,"kb_avail":749707108,"statfs":{"total":767960285184,"available":767700078592,"internally_reserved":0,"allocated":24154112,"data_stored":19392558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36053,"internal_metadata":235893547},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002701"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":74,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.290727+0000","last_change":"2024-09-13T14:24:55.290644+0000","last_active":"2024-09-13T14:24:55.290727+0000","last_peered":"2024-09-13T14:24:55.290727+0000","last_clean":"2024-09-13T14:24:55.290727+0000","last_became_active":"2024-09-13T14:24:55.290222+0000","last_became_peered":"2024-09-13T14:24:55.290222+0000","last_unstale":"2024-09-13T14:24:55.290727+0000","last_undegraded":"2024-09-13T14:24:55.290727+0000","last_fullsized":"2024-09-13T14:24:55.290727+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:38:41.205619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":75,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.274419+0000","last_change":"2024-09-13T14:24:23.773702+0000","last_active":"2024-09-13T14:24:55.274419+0000","last_peered":"2024-09-13T14:24:55.274419+0000","last_clean":"2024-09-13T14:24:55.274419+0000","last_became_active":"2024-09-13T14:24:23.773290+0000","last_became_peered":"2024-09-13T14:24:23.773290+0000","last_unstale":"2024-09-13T14:24:55.274419+0000","last_undegraded":"2024-09-13T14:24:55.274419+0000","last_fullsized":"2024-09-13T14:24:55.274419+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:20.820970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":72,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.283071+0000","last_change":"2024-09-13T14:24:55.282407+0000","last_active":"2024-09-13T14:24:55.283071+0000","last_peered":"2024-09-13T14:24:55.283071+0000","last_clean":"2024-09-13T14:24:55.283071+0000","last_became_active":"2024-09-13T14:24:55.282083+0000","last_became_peered":"2024-09-13T14:24:55.282083+0000","last_unstale":"2024-09-13T14:24:55.283071+0000","last_undegraded":"2024-09-13T14:24:55.283071+0000","last_fullsized":"2024-09-13T14:24:55.283071+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:23:10.525707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":65,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.284816+0000","last_change":"2024-09-13T14:24:55.284433+0000","last_active":"2024-09-13T14:24:55.284816+0000","last_peered":"2024-09-13T14:24:55.284816+0000","last_clean":"2024-09-13T14:24:55.284816+0000","last_became_active":"2024-09-13T14:24:55.284143+0000","last_became_peered":"2024-09-13T14:24:55.284143+0000","last_unstale":"2024-09-13T14:24:55.284816+0000","last_undegraded":"2024-09-13T14:24:55.284816+0000","last_fullsized":"2024-09-13T14:24:55.284816+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:28.107405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":67,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.282846+0000","last_change":"2024-09-13T14:24:55.282416+0000","last_active":"2024-09-13T14:24:55.282846+0000","last_peered":"2024-09-13T14:24:55.282846+0000","last_clean":"2024-09-13T14:24:55.282846+0000","last_became_active":"2024-09-13T14:24:55.282045+0000","last_became_peered":"2024-09-13T14:24:55.282045+0000","last_unstale":"2024-09-13T14:24:55.282846+0000","last_undegraded":"2024-09-13T14:24:55.282846+0000","last_fullsized":"2024-09-13T14:24:55.282846+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:58.692897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":58,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.274055+0000","last_change":"2024-09-13T14:11:29.244452+0000","last_active":"2024-09-13T14:24:55.274055+0000","last_peered":"2024-09-13T14:24:55.274055+0000","last_clean":"2024-09-13T14:24:55.274055+0000","last_became_active":"2024-09-13T14:11:29.244102+0000","last_became_peered":"2024-09-13T14:11:29.244102+0000","last_unstale":"2024-09-13T14:24:55.274055+0000","last_undegraded":"2024-09-13T14:24:55.274055+0000","last_fullsized":"2024-09-13T14:24:55.274055+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:13:19.165534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":74,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.289698+0000","last_change":"2024-09-13T14:24:55.288871+0000","last_active":"2024-09-13T14:24:55.289698+0000","last_peered":"2024-09-13T14:24:55.289698+0000","last_clean":"2024-09-13T14:24:55.289698+0000","last_became_active":"2024-09-13T14:24:55.288506+0000","last_became_peered":"2024-09-13T14:24:55.288506+0000","last_unstale":"2024-09-13T14:24:55.289698+0000","last_undegraded":"2024-09-13T14:24:55.289698+0000","last_fullsized":"2024-09-13T14:24:55.289698+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:44.796657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":81,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.283649+0000","last_change":"2024-09-13T14:23:18.039061+0000","last_active":"2024-09-13T14:24:55.283649+0000","last_peered":"2024-09-13T14:24:55.283649+0000","last_clean":"2024-09-13T14:24:55.283649+0000","last_became_active":"2024-09-13T14:23:18.038661+0000","last_became_peered":"2024-09-13T14:23:18.038661+0000","last_unstale":"2024-09-13T14:24:55.283649+0000","last_undegraded":"2024-09-13T14:24:55.283649+0000","last_fullsized":"2024-09-13T14:24:55.283649+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:05:29.109844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":74,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.275099+0000","last_change":"2024-09-13T14:23:18.038597+0000","last_active":"2024-09-13T14:24:55.275099+0000","last_peered":"2024-09-13T14:24:55.275099+0000","last_clean":"2024-09-13T14:24:55.275099+0000","last_became_active":"2024-09-13T14:23:18.038240+0000","last_became_peered":"2024-09-13T14:23:18.038240+0000","last_unstale":"2024-09-13T14:24:55.275099+0000","last_undegraded":"2024-09-13T14:24:55.275099+0000","last_fullsized":"2024-09-13T14:24:55.275099+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:27:51.842195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":57,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.269670+0000","last_change":"2024-09-13T14:11:30.254534+0000","last_active":"2024-09-13T14:24:54.269670+0000","last_peered":"2024-09-13T14:24:54.269670+0000","last_clean":"2024-09-13T14:24:54.269670+0000","last_became_active":"2024-09-13T14:11:30.253959+0000","last_became_peered":"2024-09-13T14:11:30.253959+0000","last_unstale":"2024-09-13T14:24:54.269670+0000","last_undegraded":"2024-09-13T14:24:54.269670+0000","last_fullsized":"2024-09-13T14:24:54.269670+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:09:54.025853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":75,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.274762+0000","last_change":"2024-09-13T14:23:51.612105+0000","last_active":"2024-09-13T14:24:55.274762+0000","last_peered":"2024-09-13T14:24:55.274762+0000","last_clean":"2024-09-13T14:24:55.274762+0000","last_became_active":"2024-09-13T14:23:51.611740+0000","last_became_peered":"2024-09-13T14:23:51.611740+0000","last_unstale":"2024-09-13T14:24:55.274762+0000","last_undegraded":"2024-09-13T14:24:55.274762+0000","last_fullsized":"2024-09-13T14:24:55.274762+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:26:23.262096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":86,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.288950+0000","last_change":"2024-09-13T14:24:55.288950+0000","last_active":"2024-09-13T14:24:55.288950+0000","last_peered":"2024-09-13T14:24:55.288950+0000","last_clean":"2024-09-13T14:24:55.288950+0000","last_became_active":"2024-09-13T14:24:55.288671+0000","last_became_peered":"2024-09-13T14:24:55.288671+0000","last_unstale":"2024-09-13T14:24:55.288950+0000","last_undegraded":"2024-09-13T14:24:55.288950+0000","last_fullsized":"2024-09-13T14:24:55.288950+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:14:53.298390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":59,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.275139+0000","last_change":"2024-09-13T14:11:29.232897+0000","last_active":"2024-09-13T14:24:55.275139+0000","last_peered":"2024-09-13T14:24:55.275139+0000","last_clean":"2024-09-13T14:24:55.275139+0000","last_became_active":"2024-09-13T14:11:29.232637+0000","last_became_peered":"2024-09-13T14:11:29.232637+0000","last_unstale":"2024-09-13T14:24:55.275139+0000","last_undegraded":"2024-09-13T14:24:55.275139+0000","last_fullsized":"2024-09-13T14:24:55.275139+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:31:05.856046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":72,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.276706+0000","last_change":"2024-09-13T14:23:51.609167+0000","last_active":"2024-09-13T14:24:54.276706+0000","last_peered":"2024-09-13T14:24:54.276706+0000","last_clean":"2024-09-13T14:24:54.276706+0000","last_became_active":"2024-09-13T14:23:51.608811+0000","last_became_peered":"2024-09-13T14:23:51.608811+0000","last_unstale":"2024-09-13T14:24:54.276706+0000","last_undegraded":"2024-09-13T14:24:54.276706+0000","last_fullsized":"2024-09-13T14:24:54.276706+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:40.506763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":73,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.291355+0000","last_change":"2024-09-13T14:24:55.290798+0000","last_active":"2024-09-13T14:24:55.291355+0000","last_peered":"2024-09-13T14:24:55.291355+0000","last_clean":"2024-09-13T14:24:55.291355+0000","last_became_active":"2024-09-13T14:24:55.290510+0000","last_became_peered":"2024-09-13T14:24:55.290510+0000","last_unstale":"2024-09-13T14:24:55.291355+0000","last_undegraded":"2024-09-13T14:24:55.291355+0000","last_fullsized":"2024-09-13T14:24:55.291355+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:19:03.129132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":67,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.271982+0000","last_change":"2024-09-13T14:23:51.609082+0000","last_active":"2024-09-13T14:24:55.271982+0000","last_peered":"2024-09-13T14:24:55.271982+0000","last_clean":"2024-09-13T14:24:55.271982+0000","last_became_active":"2024-09-13T14:23:51.608799+0000","last_became_peered":"2024-09-13T14:23:51.608799+0000","last_unstale":"2024-09-13T14:24:55.271982+0000","last_undegraded":"2024-09-13T14:24:55.271982+0000","last_fullsized":"2024-09-13T14:24:55.271982+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:43:24.721453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":72,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.284590+0000","last_change":"2024-09-13T14:24:55.284499+0000","last_active":"2024-09-13T14:24:55.284590+0000","last_peered":"2024-09-13T14:24:55.284590+0000","last_clean":"2024-09-13T14:24:55.284590+0000","last_became_active":"2024-09-13T14:24:55.284096+0000","last_became_peered":"2024-09-13T14:24:55.284096+0000","last_unstale":"2024-09-13T14:24:55.284590+0000","last_undegraded":"2024-09-13T14:24:55.284590+0000","last_fullsized":"2024-09-13T14:24:55.284590+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:02:06.836099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":71,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.291592+0000","last_change":"2024-09-13T14:24:55.291371+0000","last_active":"2024-09-13T14:24:55.291592+0000","last_peered":"2024-09-13T14:24:55.291592+0000","last_clean":"2024-09-13T14:24:55.291592+0000","last_became_active":"2024-09-13T14:24:55.291047+0000","last_became_peered":"2024-09-13T14:24:55.291047+0000","last_unstale":"2024-09-13T14:24:55.291592+0000","last_undegraded":"2024-09-13T14:24:55.291592+0000","last_fullsized":"2024-09-13T14:24:55.291592+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:43.912412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":73,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.276868+0000","last_change":"2024-09-13T14:23:51.606589+0000","last_active":"2024-09-13T14:24:54.276868+0000","last_peered":"2024-09-13T14:24:54.276868+0000","last_clean":"2024-09-13T14:24:54.276868+0000","last_became_active":"2024-09-13T14:23:51.606244+0000","last_became_peered":"2024-09-13T14:23:51.606244+0000","last_unstale":"2024-09-13T14:24:54.276868+0000","last_undegraded":"2024-09-13T14:24:54.276868+0000","last_fullsized":"2024-09-13T14:24:54.276868+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:08.776526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":86,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.275537+0000","last_change":"2024-09-13T14:24:23.777891+0000","last_active":"2024-09-13T14:24:55.275537+0000","last_peered":"2024-09-13T14:24:55.275537+0000","last_clean":"2024-09-13T14:24:55.275537+0000","last_became_active":"2024-09-13T14:24:23.777578+0000","last_became_peered":"2024-09-13T14:24:23.777578+0000","last_unstale":"2024-09-13T14:24:55.275537+0000","last_undegraded":"2024-09-13T14:24:55.275537+0000","last_fullsized":"2024-09-13T14:24:55.275537+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:56:47.441579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":72,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.270580+0000","last_change":"2024-09-13T14:24:23.777970+0000","last_active":"2024-09-13T14:24:54.270580+0000","last_peered":"2024-09-13T14:24:54.270580+0000","last_clean":"2024-09-13T14:24:54.270580+0000","last_became_active":"2024-09-13T14:24:23.777650+0000","last_became_peered":"2024-09-13T14:24:23.777650+0000","last_unstale":"2024-09-13T14:24:54.270580+0000","last_undegraded":"2024-09-13T14:24:54.270580+0000","last_fullsized":"2024-09-13T14:24:54.270580+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:50:00.549905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":71,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.274746+0000","last_change":"2024-09-13T14:23:51.610572+0000","last_active":"2024-09-13T14:24:55.274746+0000","last_peered":"2024-09-13T14:24:55.274746+0000","last_clean":"2024-09-13T14:24:55.274746+0000","last_became_active":"2024-09-13T14:23:51.610273+0000","last_became_peered":"2024-09-13T14:23:51.610273+0000","last_unstale":"2024-09-13T14:24:55.274746+0000","last_undegraded":"2024-09-13T14:24:55.274746+0000","last_fullsized":"2024-09-13T14:24:55.274746+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:17:10.574417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":75,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.281133+0000","last_change":"2024-09-13T14:24:55.281042+0000","last_active":"2024-09-13T14:24:55.281133+0000","last_peered":"2024-09-13T14:24:55.281133+0000","last_clean":"2024-09-13T14:24:55.281133+0000","last_became_active":"2024-09-13T14:24:55.280741+0000","last_became_peered":"2024-09-13T14:24:55.280741+0000","last_unstale":"2024-09-13T14:24:55.281133+0000","last_undegraded":"2024-09-13T14:24:55.281133+0000","last_fullsized":"2024-09-13T14:24:55.281133+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:45:40.787103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":46,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.289331+0000","last_change":"2024-09-13T14:24:55.289245+0000","last_active":"2024-09-13T14:24:55.289331+0000","last_peered":"2024-09-13T14:24:55.289331+0000","last_clean":"2024-09-13T14:24:55.289331+0000","last_became_active":"2024-09-13T14:24:55.288881+0000","last_became_peered":"2024-09-13T14:24:55.288881+0000","last_unstale":"2024-09-13T14:24:55.289331+0000","last_undegraded":"2024-09-13T14:24:55.289331+0000","last_fullsized":"2024-09-13T14:24:55.289331+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:14:12.194309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":404,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.274086+0000","last_change":"2024-09-13T14:23:18.032973+0000","last_active":"2024-09-13T14:24:55.274086+0000","last_peered":"2024-09-13T14:24:55.274086+0000","last_clean":"2024-09-13T14:24:55.274086+0000","last_became_active":"2024-09-13T14:23:18.032579+0000","last_became_peered":"2024-09-13T14:23:18.032579+0000","last_unstale":"2024-09-13T14:24:55.274086+0000","last_undegraded":"2024-09-13T14:24:55.274086+0000","last_fullsized":"2024-09-13T14:24:55.274086+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:07:41.927740+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:17.109445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":400,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.270769+0000","last_change":"2024-09-13T14:24:23.776868+0000","last_active":"2024-09-13T14:24:54.270769+0000","last_peered":"2024-09-13T14:24:54.270769+0000","last_clean":"2024-09-13T14:24:54.270769+0000","last_became_active":"2024-09-13T14:24:23.776306+0000","last_became_peered":"2024-09-13T14:24:23.776306+0000","last_unstale":"2024-09-13T14:24:54.270769+0000","last_undegraded":"2024-09-13T14:24:54.270769+0000","last_fullsized":"2024-09-13T14:24:54.270769+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:28:41.805918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":23,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.288138+0000","last_change":"2024-09-13T14:24:55.288058+0000","last_active":"2024-09-13T14:24:55.288138+0000","last_peered":"2024-09-13T14:24:55.288138+0000","last_clean":"2024-09-13T14:24:55.288138+0000","last_became_active":"2024-09-13T14:24:55.287601+0000","last_became_peered":"2024-09-13T14:24:55.287601+0000","last_unstale":"2024-09-13T14:24:55.288138+0000","last_undegraded":"2024-09-13T14:24:55.288138+0000","last_fullsized":"2024-09-13T14:24:55.288138+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:56:34.622696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":57,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.277982+0000","last_change":"2024-09-13T14:11:30.253749+0000","last_active":"2024-09-13T14:24:54.277982+0000","last_peered":"2024-09-13T14:24:54.277982+0000","last_clean":"2024-09-13T14:24:54.277982+0000","last_became_active":"2024-09-13T14:11:30.253499+0000","last_became_peered":"2024-09-13T14:11:30.253499+0000","last_unstale":"2024-09-13T14:24:54.277982+0000","last_undegraded":"2024-09-13T14:24:54.277982+0000","last_fullsized":"2024-09-13T14:24:54.277982+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:02:00.987335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":289,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.275098+0000","last_change":"2024-09-13T14:23:51.608681+0000","last_active":"2024-09-13T14:24:54.275098+0000","last_peered":"2024-09-13T14:24:54.275098+0000","last_clean":"2024-09-13T14:24:54.275098+0000","last_became_active":"2024-09-13T14:23:51.608223+0000","last_became_peered":"2024-09-13T14:23:51.608223+0000","last_unstale":"2024-09-13T14:24:54.275098+0000","last_undegraded":"2024-09-13T14:24:54.275098+0000","last_fullsized":"2024-09-13T14:24:54.275098+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:13:15.930987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.286928+0000","last_change":"2024-09-13T14:24:55.286576+0000","last_active":"2024-09-13T14:24:55.286928+0000","last_peered":"2024-09-13T14:24:55.286928+0000","last_clean":"2024-09-13T14:24:55.286928+0000","last_became_active":"2024-09-13T14:24:55.286284+0000","last_became_peered":"2024-09-13T14:24:55.286284+0000","last_unstale":"2024-09-13T14:24:55.286928+0000","last_undegraded":"2024-09-13T14:24:55.286928+0000","last_fullsized":"2024-09-13T14:24:55.286928+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:35.589736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":78,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.271821+0000","last_change":"2024-09-13T14:24:23.776557+0000","last_active":"2024-09-13T14:24:55.271821+0000","last_peered":"2024-09-13T14:24:55.271821+0000","last_clean":"2024-09-13T14:24:55.271821+0000","last_became_active":"2024-09-13T14:24:23.775204+0000","last_became_peered":"2024-09-13T14:24:23.775204+0000","last_unstale":"2024-09-13T14:24:55.271821+0000","last_undegraded":"2024-09-13T14:24:55.271821+0000","last_fullsized":"2024-09-13T14:24:55.271821+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:18:21.095561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":494,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.285398+0000","last_change":"2024-09-13T14:24:55.285288+0000","last_active":"2024-09-13T14:24:55.285398+0000","last_peered":"2024-09-13T14:24:55.285398+0000","last_clean":"2024-09-13T14:24:55.285398+0000","last_became_active":"2024-09-13T14:24:55.284733+0000","last_became_peered":"2024-09-13T14:24:55.284733+0000","last_unstale":"2024-09-13T14:24:55.285398+0000","last_undegraded":"2024-09-13T14:24:55.285398+0000","last_fullsized":"2024-09-13T14:24:55.285398+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:51:50.420796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":55,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.276363+0000","last_change":"2024-09-13T14:24:23.779725+0000","last_active":"2024-09-13T14:24:55.276363+0000","last_peered":"2024-09-13T14:24:55.276363+0000","last_clean":"2024-09-13T14:24:55.276363+0000","last_became_active":"2024-09-13T14:24:23.779418+0000","last_became_peered":"2024-09-13T14:24:23.779418+0000","last_unstale":"2024-09-13T14:24:55.276363+0000","last_undegraded":"2024-09-13T14:24:55.276363+0000","last_fullsized":"2024-09-13T14:24:55.276363+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:48:08.470127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":90,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.276884+0000","last_change":"2024-09-13T14:23:51.608576+0000","last_active":"2024-09-13T14:24:55.276884+0000","last_peered":"2024-09-13T14:24:55.276884+0000","last_clean":"2024-09-13T14:24:55.276884+0000","last_became_active":"2024-09-13T14:23:51.607956+0000","last_became_peered":"2024-09-13T14:23:51.607956+0000","last_unstale":"2024-09-13T14:24:55.276884+0000","last_undegraded":"2024-09-13T14:24:55.276884+0000","last_fullsized":"2024-09-13T14:24:55.276884+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:53.399552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":829,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.272095+0000","last_change":"2024-09-13T14:23:51.606335+0000","last_active":"2024-09-13T14:24:54.272095+0000","last_peered":"2024-09-13T14:24:54.272095+0000","last_clean":"2024-09-13T14:24:54.272095+0000","last_became_active":"2024-09-13T14:23:51.605905+0000","last_became_peered":"2024-09-13T14:23:51.605905+0000","last_unstale":"2024-09-13T14:24:54.272095+0000","last_undegraded":"2024-09-13T14:24:54.272095+0000","last_fullsized":"2024-09-13T14:24:54.272095+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:22:42.470188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":72,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.272867+0000","last_change":"2024-09-13T14:23:18.042891+0000","last_active":"2024-09-13T14:24:54.272867+0000","last_peered":"2024-09-13T14:24:54.272867+0000","last_clean":"2024-09-13T14:24:54.272867+0000","last_became_active":"2024-09-13T14:23:18.042562+0000","last_became_peered":"2024-09-13T14:23:18.042562+0000","last_unstale":"2024-09-13T14:24:54.272867+0000","last_undegraded":"2024-09-13T14:24:54.272867+0000","last_fullsized":"2024-09-13T14:24:54.272867+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:30:42.996661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":84,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.291080+0000","last_change":"2024-09-13T14:24:55.291080+0000","last_active":"2024-09-13T14:24:55.291080+0000","last_peered":"2024-09-13T14:24:55.291080+0000","last_clean":"2024-09-13T14:24:55.291080+0000","last_became_active":"2024-09-13T14:24:55.290560+0000","last_became_peered":"2024-09-13T14:24:55.290560+0000","last_unstale":"2024-09-13T14:24:55.291080+0000","last_undegraded":"2024-09-13T14:24:55.291080+0000","last_fullsized":"2024-09-13T14:24:55.291080+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:16.041382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":88,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.282619+0000","last_change":"2024-09-13T14:24:55.282619+0000","last_active":"2024-09-13T14:24:55.282619+0000","last_peered":"2024-09-13T14:24:55.282619+0000","last_clean":"2024-09-13T14:24:55.282619+0000","last_became_active":"2024-09-13T14:24:55.282110+0000","last_became_peered":"2024-09-13T14:24:55.282110+0000","last_unstale":"2024-09-13T14:24:55.282619+0000","last_undegraded":"2024-09-13T14:24:55.282619+0000","last_fullsized":"2024-09-13T14:24:55.282619+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:33:19.425344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":778,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.277098+0000","last_change":"2024-09-13T14:23:51.610302+0000","last_active":"2024-09-13T14:24:55.277098+0000","last_peered":"2024-09-13T14:24:55.277098+0000","last_clean":"2024-09-13T14:24:55.277098+0000","last_became_active":"2024-09-13T14:23:51.607653+0000","last_became_peered":"2024-09-13T14:23:51.607653+0000","last_unstale":"2024-09-13T14:24:55.277098+0000","last_undegraded":"2024-09-13T14:24:55.277098+0000","last_fullsized":"2024-09-13T14:24:55.277098+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:03:52.131651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":46,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.275869+0000","last_change":"2024-09-13T14:23:51.610321+0000","last_active":"2024-09-13T14:24:55.275869+0000","last_peered":"2024-09-13T14:24:55.275869+0000","last_clean":"2024-09-13T14:24:55.275869+0000","last_became_active":"2024-09-13T14:23:51.607733+0000","last_became_peered":"2024-09-13T14:23:51.607733+0000","last_unstale":"2024-09-13T14:24:55.275869+0000","last_undegraded":"2024-09-13T14:24:55.275869+0000","last_fullsized":"2024-09-13T14:24:55.275869+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:58:05.246186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":55,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.277007+0000","last_change":"2024-09-13T14:24:23.779678+0000","last_active":"2024-09-13T14:24:55.277007+0000","last_peered":"2024-09-13T14:24:55.277007+0000","last_clean":"2024-09-13T14:24:55.277007+0000","last_became_active":"2024-09-13T14:24:23.779343+0000","last_became_peered":"2024-09-13T14:24:23.779343+0000","last_unstale":"2024-09-13T14:24:55.277007+0000","last_undegraded":"2024-09-13T14:24:55.277007+0000","last_fullsized":"2024-09-13T14:24:55.277007+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:14:41.230951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":595,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.275237+0000","last_change":"2024-09-13T14:23:18.038847+0000","last_active":"2024-09-13T14:24:55.275237+0000","last_peered":"2024-09-13T14:24:55.275237+0000","last_clean":"2024-09-13T14:24:55.275237+0000","last_became_active":"2024-09-13T14:23:18.038188+0000","last_became_peered":"2024-09-13T14:23:18.038188+0000","last_unstale":"2024-09-13T14:24:55.275237+0000","last_undegraded":"2024-09-13T14:24:55.275237+0000","last_fullsized":"2024-09-13T14:24:55.275237+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:30:37.213496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":56,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.271836+0000","last_change":"2024-09-13T14:23:18.034427+0000","last_active":"2024-09-13T14:24:55.271836+0000","last_peered":"2024-09-13T14:24:55.271836+0000","last_clean":"2024-09-13T14:24:55.271836+0000","last_became_active":"2024-09-13T14:23:18.034092+0000","last_became_peered":"2024-09-13T14:23:18.034092+0000","last_unstale":"2024-09-13T14:24:55.271836+0000","last_undegraded":"2024-09-13T14:24:55.271836+0000","last_fullsized":"2024-09-13T14:24:55.271836+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:21.755115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":72,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.290792+0000","last_change":"2024-09-13T14:24:55.290711+0000","last_active":"2024-09-13T14:24:55.290792+0000","last_peered":"2024-09-13T14:24:55.290792+0000","last_clean":"2024-09-13T14:24:55.290792+0000","last_became_active":"2024-09-13T14:24:55.288549+0000","last_became_peered":"2024-09-13T14:24:55.288549+0000","last_unstale":"2024-09-13T14:24:55.290792+0000","last_undegraded":"2024-09-13T14:24:55.290792+0000","last_fullsized":"2024-09-13T14:24:55.290792+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:55:21.763931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":503,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.276718+0000","last_change":"2024-09-13T14:23:51.606481+0000","last_active":"2024-09-13T14:24:55.276718+0000","last_peered":"2024-09-13T14:24:55.276718+0000","last_clean":"2024-09-13T14:24:55.276718+0000","last_became_active":"2024-09-13T14:23:51.605783+0000","last_became_peered":"2024-09-13T14:23:51.605783+0000","last_unstale":"2024-09-13T14:24:55.276718+0000","last_undegraded":"2024-09-13T14:24:55.276718+0000","last_fullsized":"2024-09-13T14:24:55.276718+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:09:01.265713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":69,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.281723+0000","last_change":"2024-09-13T14:24:55.281593+0000","last_active":"2024-09-13T14:24:55.281723+0000","last_peered":"2024-09-13T14:24:55.281723+0000","last_clean":"2024-09-13T14:24:55.281723+0000","last_became_active":"2024-09-13T14:24:55.280805+0000","last_became_peered":"2024-09-13T14:24:55.280805+0000","last_unstale":"2024-09-13T14:24:55.281723+0000","last_undegraded":"2024-09-13T14:24:55.281723+0000","last_fullsized":"2024-09-13T14:24:55.281723+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:14.806713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":70,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.271380+0000","last_change":"2024-09-13T14:23:51.609769+0000","last_active":"2024-09-13T14:24:55.271380+0000","last_peered":"2024-09-13T14:24:55.271380+0000","last_clean":"2024-09-13T14:24:55.271380+0000","last_became_active":"2024-09-13T14:23:51.609343+0000","last_became_peered":"2024-09-13T14:23:51.609343+0000","last_unstale":"2024-09-13T14:24:55.271380+0000","last_undegraded":"2024-09-13T14:24:55.271380+0000","last_fullsized":"2024-09-13T14:24:55.271380+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:06:58.544473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":514,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.282977+0000","last_change":"2024-09-13T14:24:23.775768+0000","last_active":"2024-09-13T14:24:55.282977+0000","last_peered":"2024-09-13T14:24:55.282977+0000","last_clean":"2024-09-13T14:24:55.282977+0000","last_became_active":"2024-09-13T14:24:23.775261+0000","last_became_peered":"2024-09-13T14:24:23.775261+0000","last_unstale":"2024-09-13T14:24:55.282977+0000","last_undegraded":"2024-09-13T14:24:55.282977+0000","last_fullsized":"2024-09-13T14:24:55.282977+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:30:05.340603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":57,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.274468+0000","last_change":"2024-09-13T14:11:30.256034+0000","last_active":"2024-09-13T14:24:55.274468+0000","last_peered":"2024-09-13T14:24:55.274468+0000","last_clean":"2024-09-13T14:24:55.274468+0000","last_became_active":"2024-09-13T14:11:30.255727+0000","last_became_peered":"2024-09-13T14:11:30.255727+0000","last_unstale":"2024-09-13T14:24:55.274468+0000","last_undegraded":"2024-09-13T14:24:55.274468+0000","last_fullsized":"2024-09-13T14:24:55.274468+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:21:35.817014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":71,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.274287+0000","last_change":"2024-09-13T14:23:51.606397+0000","last_active":"2024-09-13T14:24:55.274287+0000","last_peered":"2024-09-13T14:24:55.274287+0000","last_clean":"2024-09-13T14:24:55.274287+0000","last_became_active":"2024-09-13T14:23:51.605792+0000","last_became_peered":"2024-09-13T14:23:51.605792+0000","last_unstale":"2024-09-13T14:24:55.274287+0000","last_undegraded":"2024-09-13T14:24:55.274287+0000","last_fullsized":"2024-09-13T14:24:55.274287+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:42:44.617374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":80,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.275351+0000","last_change":"2024-09-13T14:11:29.237013+0000","last_active":"2024-09-13T14:24:55.275351+0000","last_peered":"2024-09-13T14:24:55.275351+0000","last_clean":"2024-09-13T14:24:55.275351+0000","last_became_active":"2024-09-13T14:11:29.236643+0000","last_became_peered":"2024-09-13T14:11:29.236643+0000","last_unstale":"2024-09-13T14:24:55.275351+0000","last_undegraded":"2024-09-13T14:24:55.275351+0000","last_fullsized":"2024-09-13T14:24:55.275351+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:33:58.438693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":56,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.275681+0000","last_change":"2024-09-13T14:24:23.779556+0000","last_active":"2024-09-13T14:24:55.275681+0000","last_peered":"2024-09-13T14:24:55.275681+0000","last_clean":"2024-09-13T14:24:55.275681+0000","last_became_active":"2024-09-13T14:24:23.779234+0000","last_became_peered":"2024-09-13T14:24:23.779234+0000","last_unstale":"2024-09-13T14:24:55.275681+0000","last_undegraded":"2024-09-13T14:24:55.275681+0000","last_fullsized":"2024-09-13T14:24:55.275681+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:40.733288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":76,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.287585+0000","last_change":"2024-09-13T14:24:55.287275+0000","last_active":"2024-09-13T14:24:55.287585+0000","last_peered":"2024-09-13T14:24:55.287585+0000","last_clean":"2024-09-13T14:24:55.287585+0000","last_became_active":"2024-09-13T14:24:55.286919+0000","last_became_peered":"2024-09-13T14:24:55.286919+0000","last_unstale":"2024-09-13T14:24:55.287585+0000","last_undegraded":"2024-09-13T14:24:55.287585+0000","last_fullsized":"2024-09-13T14:24:55.287585+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:10.790711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":73,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.272716+0000","last_change":"2024-09-13T14:23:51.610941+0000","last_active":"2024-09-13T14:24:55.272716+0000","last_peered":"2024-09-13T14:24:55.272716+0000","last_clean":"2024-09-13T14:24:55.272716+0000","last_became_active":"2024-09-13T14:23:51.610587+0000","last_became_peered":"2024-09-13T14:23:51.610587+0000","last_unstale":"2024-09-13T14:24:55.272716+0000","last_undegraded":"2024-09-13T14:24:55.272716+0000","last_fullsized":"2024-09-13T14:24:55.272716+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:48:01.436607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.288101+0000","last_change":"2024-09-13T14:24:55.287509+0000","last_active":"2024-09-13T14:24:55.288101+0000","last_peered":"2024-09-13T14:24:55.288101+0000","last_clean":"2024-09-13T14:24:55.288101+0000","last_became_active":"2024-09-13T14:24:55.287193+0000","last_became_peered":"2024-09-13T14:24:55.287193+0000","last_unstale":"2024-09-13T14:24:55.288101+0000","last_undegraded":"2024-09-13T14:24:55.288101+0000","last_fullsized":"2024-09-13T14:24:55.288101+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:14:10.603307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.289393+0000","last_change":"2024-09-13T14:24:55.289055+0000","last_active":"2024-09-13T14:24:55.289393+0000","last_peered":"2024-09-13T14:24:55.289393+0000","last_clean":"2024-09-13T14:24:55.289393+0000","last_became_active":"2024-09-13T14:24:55.288762+0000","last_became_peered":"2024-09-13T14:24:55.288762+0000","last_unstale":"2024-09-13T14:24:55.289393+0000","last_undegraded":"2024-09-13T14:24:55.289393+0000","last_fullsized":"2024-09-13T14:24:55.289393+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:37:26.639877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":74,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.287280+0000","last_change":"2024-09-13T14:24:55.287211+0000","last_active":"2024-09-13T14:24:55.287280+0000","last_peered":"2024-09-13T14:24:55.287280+0000","last_clean":"2024-09-13T14:24:55.287280+0000","last_became_active":"2024-09-13T14:24:55.286301+0000","last_became_peered":"2024-09-13T14:24:55.286301+0000","last_unstale":"2024-09-13T14:24:55.287280+0000","last_undegraded":"2024-09-13T14:24:55.287280+0000","last_fullsized":"2024-09-13T14:24:55.287280+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:32:41.099046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":73,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.282888+0000","last_change":"2024-09-13T14:23:51.606509+0000","last_active":"2024-09-13T14:24:55.282888+0000","last_peered":"2024-09-13T14:24:55.282888+0000","last_clean":"2024-09-13T14:24:55.282888+0000","last_became_active":"2024-09-13T14:23:51.606195+0000","last_became_peered":"2024-09-13T14:23:51.606195+0000","last_unstale":"2024-09-13T14:24:55.282888+0000","last_undegraded":"2024-09-13T14:24:55.282888+0000","last_fullsized":"2024-09-13T14:24:55.282888+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:29:49.729550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":90,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.280528+0000","last_change":"2024-09-13T14:24:55.280438+0000","last_active":"2024-09-13T14:24:55.280528+0000","last_peered":"2024-09-13T14:24:55.280528+0000","last_clean":"2024-09-13T14:24:55.280528+0000","last_became_active":"2024-09-13T14:24:55.279359+0000","last_became_peered":"2024-09-13T14:24:55.279359+0000","last_unstale":"2024-09-13T14:24:55.280528+0000","last_undegraded":"2024-09-13T14:24:55.280528+0000","last_fullsized":"2024-09-13T14:24:55.280528+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:12:48.432924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":67,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.284867+0000","last_change":"2024-09-13T14:24:55.283070+0000","last_active":"2024-09-13T14:24:55.284867+0000","last_peered":"2024-09-13T14:24:55.284867+0000","last_clean":"2024-09-13T14:24:55.284867+0000","last_became_active":"2024-09-13T14:24:55.282591+0000","last_became_peered":"2024-09-13T14:24:55.282591+0000","last_unstale":"2024-09-13T14:24:55.284867+0000","last_undegraded":"2024-09-13T14:24:55.284867+0000","last_fullsized":"2024-09-13T14:24:55.284867+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:35:21.896294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":55,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.289919+0000","last_change":"2024-09-13T14:24:55.289828+0000","last_active":"2024-09-13T14:24:55.289919+0000","last_peered":"2024-09-13T14:24:55.289919+0000","last_clean":"2024-09-13T14:24:55.289919+0000","last_became_active":"2024-09-13T14:24:55.289525+0000","last_became_peered":"2024-09-13T14:24:55.289525+0000","last_unstale":"2024-09-13T14:24:55.289919+0000","last_undegraded":"2024-09-13T14:24:55.289919+0000","last_fullsized":"2024-09-13T14:24:55.289919+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:25:39.041539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":87,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.284789+0000","last_change":"2024-09-13T14:24:55.284018+0000","last_active":"2024-09-13T14:24:55.284789+0000","last_peered":"2024-09-13T14:24:55.284789+0000","last_clean":"2024-09-13T14:24:55.284789+0000","last_became_active":"2024-09-13T14:24:55.283571+0000","last_became_peered":"2024-09-13T14:24:55.283571+0000","last_unstale":"2024-09-13T14:24:55.284789+0000","last_undegraded":"2024-09-13T14:24:55.284789+0000","last_fullsized":"2024-09-13T14:24:55.284789+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:48.890356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":87,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.270277+0000","last_change":"2024-09-13T14:24:23.778031+0000","last_active":"2024-09-13T14:24:54.270277+0000","last_peered":"2024-09-13T14:24:54.270277+0000","last_clean":"2024-09-13T14:24:54.270277+0000","last_became_active":"2024-09-13T14:24:23.777665+0000","last_became_peered":"2024-09-13T14:24:23.777665+0000","last_unstale":"2024-09-13T14:24:54.270277+0000","last_undegraded":"2024-09-13T14:24:54.270277+0000","last_fullsized":"2024-09-13T14:24:54.270277+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:27:54.790142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":94,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.290937+0000","last_change":"2024-09-13T14:24:55.290839+0000","last_active":"2024-09-13T14:24:55.290937+0000","last_peered":"2024-09-13T14:24:55.290937+0000","last_clean":"2024-09-13T14:24:55.290937+0000","last_became_active":"2024-09-13T14:24:55.290348+0000","last_became_peered":"2024-09-13T14:24:55.290348+0000","last_unstale":"2024-09-13T14:24:55.290937+0000","last_undegraded":"2024-09-13T14:24:55.290937+0000","last_fullsized":"2024-09-13T14:24:55.290937+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:11:38.467219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":59,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.271734+0000","last_change":"2024-09-13T14:23:18.035101+0000","last_active":"2024-09-13T14:24:55.271734+0000","last_peered":"2024-09-13T14:24:55.271734+0000","last_clean":"2024-09-13T14:24:55.271734+0000","last_became_active":"2024-09-13T14:23:18.034724+0000","last_became_peered":"2024-09-13T14:23:18.034724+0000","last_unstale":"2024-09-13T14:24:55.271734+0000","last_undegraded":"2024-09-13T14:24:55.271734+0000","last_fullsized":"2024-09-13T14:24:55.271734+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:39.358144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":85,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.282346+0000","last_change":"2024-09-13T14:24:55.282346+0000","last_active":"2024-09-13T14:24:55.282346+0000","last_peered":"2024-09-13T14:24:55.282346+0000","last_clean":"2024-09-13T14:24:55.282346+0000","last_became_active":"2024-09-13T14:24:55.281988+0000","last_became_peered":"2024-09-13T14:24:55.281988+0000","last_unstale":"2024-09-13T14:24:55.282346+0000","last_undegraded":"2024-09-13T14:24:55.282346+0000","last_fullsized":"2024-09-13T14:24:55.282346+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:05:35.737441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":74,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.270907+0000","last_change":"2024-09-13T14:23:51.612637+0000","last_active":"2024-09-13T14:24:54.270907+0000","last_peered":"2024-09-13T14:24:54.270907+0000","last_clean":"2024-09-13T14:24:54.270907+0000","last_became_active":"2024-09-13T14:23:51.612175+0000","last_became_peered":"2024-09-13T14:23:51.612175+0000","last_unstale":"2024-09-13T14:24:54.270907+0000","last_undegraded":"2024-09-13T14:24:54.270907+0000","last_fullsized":"2024-09-13T14:24:54.270907+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:52:30.299317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":85,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.285384+0000","last_change":"2024-09-13T14:24:55.285384+0000","last_active":"2024-09-13T14:24:55.285384+0000","last_peered":"2024-09-13T14:24:55.285384+0000","last_clean":"2024-09-13T14:24:55.285384+0000","last_became_active":"2024-09-13T14:24:55.285081+0000","last_became_peered":"2024-09-13T14:24:55.285081+0000","last_unstale":"2024-09-13T14:24:55.285384+0000","last_undegraded":"2024-09-13T14:24:55.285384+0000","last_fullsized":"2024-09-13T14:24:55.285384+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:47:07.946120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":59,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:24:54.272758+0000","last_change":"2024-09-13T14:11:29.240351+0000","last_active":"2024-09-13T14:24:54.272758+0000","last_peered":"2024-09-13T14:24:54.272758+0000","last_clean":"2024-09-13T14:24:54.272758+0000","last_became_active":"2024-09-13T14:11:29.240060+0000","last_became_peered":"2024-09-13T14:11:29.240060+0000","last_unstale":"2024-09-13T14:24:54.272758+0000","last_undegraded":"2024-09-13T14:24:54.272758+0000","last_fullsized":"2024-09-13T14:24:54.272758+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:34:36.126368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":93,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.285566+0000","last_change":"2024-09-13T14:24:55.285488+0000","last_active":"2024-09-13T14:24:55.285566+0000","last_peered":"2024-09-13T14:24:55.285566+0000","last_clean":"2024-09-13T14:24:55.285566+0000","last_became_active":"2024-09-13T14:24:55.283433+0000","last_became_peered":"2024-09-13T14:24:55.283433+0000","last_unstale":"2024-09-13T14:24:55.285566+0000","last_undegraded":"2024-09-13T14:24:55.285566+0000","last_fullsized":"2024-09-13T14:24:55.285566+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:21:11.486340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":81,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.287819+0000","last_change":"2024-09-13T14:24:55.287498+0000","last_active":"2024-09-13T14:24:55.287819+0000","last_peered":"2024-09-13T14:24:55.287819+0000","last_clean":"2024-09-13T14:24:55.287819+0000","last_became_active":"2024-09-13T14:24:55.285957+0000","last_became_peered":"2024-09-13T14:24:55.285957+0000","last_unstale":"2024-09-13T14:24:55.287819+0000","last_undegraded":"2024-09-13T14:24:55.287819+0000","last_fullsized":"2024-09-13T14:24:55.287819+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:21:50.821805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":71,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.274377+0000","last_change":"2024-09-13T14:23:51.609501+0000","last_active":"2024-09-13T14:24:55.274377+0000","last_peered":"2024-09-13T14:24:55.274377+0000","last_clean":"2024-09-13T14:24:55.274377+0000","last_became_active":"2024-09-13T14:23:51.609135+0000","last_became_peered":"2024-09-13T14:23:51.609135+0000","last_unstale":"2024-09-13T14:24:55.274377+0000","last_undegraded":"2024-09-13T14:24:55.274377+0000","last_fullsized":"2024-09-13T14:24:55.274377+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:21:25.318030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":70,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T14:24:55.271702+0000","last_change":"2024-09-13T14:24:23.775769+0000","last_active":"2024-09-13T14:24:55.271702+0000","last_peered":"2024-09-13T14:24:55.271702+0000","last_clean":"2024-09-13T14:24:55.271702+0000","last_became_active":"2024-09-13T14:24:23.775409+0000","last_became_peered":"2024-09-13T14:24:23.775409+0000","last_unstale":"2024-09-13T14:24:55.271702+0000","last_undegraded":"2024-09-13T14:24:55.271702+0000","last_fullsized":"2024-09-13T14:24:55.271702+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:20:09.966476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18272,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234186,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32912,"kb_used_data":1788,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712240,"statfs":{"total":95995035648,"available":95961333760,"internally_reserved":0,"allocated":1830912,"data_stored":1222916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495822,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36992,"kb_used_data":5876,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93708160,"statfs":{"total":95995035648,"available":95957155840,"internally_reserved":0,"allocated":6017024,"data_stored":5417151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757459,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36404,"kb_used_data":5288,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93708748,"statfs":{"total":95995035648,"available":95957757952,"internally_reserved":0,"allocated":5414912,"data_stored":4822551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019095,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28224,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716928,"statfs":{"total":95995035648,"available":95966134272,"internally_reserved":0,"allocated":1232896,"data_stored":632536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435279,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28728,"kb_used_data":1188,"kb_used_omap":5,"kb_used_meta":27514,"kb_avail":93716424,"statfs":{"total":95995035648,"available":95965618176,"internally_reserved":0,"allocated":1216512,"data_stored":628009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":28174462},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":104,"seq":446676598805,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28732,"kb_used_data":1192,"kb_used_omap":7,"kb_used_meta":27512,"kb_avail":93716420,"statfs":{"total":95995035648,"available":95965614080,"internally_reserved":0,"allocated":1220608,"data_stored":628535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8010,"internal_metadata":28172470},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":99,"seq":425201762332,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32812,"kb_used_data":5280,"kb_used_omap":7,"kb_used_meta":27512,"kb_avail":93712340,"statfs":{"total":95995035648,"available":95961436160,"internally_reserved":0,"allocated":5406720,"data_stored":4821969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7759,"internal_metadata":28172721},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726925859,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":29304,"kb_used_data":1772,"kb_used_omap":7,"kb_used_meta":27512,"kb_avail":93715848,"statfs":{"total":95995035648,"available":95965028352,"internally_reserved":0,"allocated":1814528,"data_stored":1218891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":28172584},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:26:03.314 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-13T14:26:03.314 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd dump 2024-09-13T14:26:03.741 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:26:04.087 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:03 smithi102 ceph-mon[110183]: from='client.44483 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:26:04.087 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:03 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:26:04.087 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:03 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:26:04.087 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:03 smithi102 ceph-mon[115269]: from='client.44483 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:26:04.088 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:03 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:26:04.088 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:03 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:26:04.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:03 smithi195 ceph-mon[104262]: from='client.44483 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:26:04.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:03 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:26:04.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:03 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:26:04.833 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:04 smithi102 ceph-mon[110183]: pgmap v282: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:26:04.833 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:04 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:26:04.833 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:04 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:26:04.833 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:04 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:26:04.833 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:04 smithi102 ceph-mon[115269]: pgmap v282: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:26:04.834 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:04 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:26:04.834 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:04 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:26:04.834 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:04 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:26:05.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:04 smithi195 ceph-mon[104262]: pgmap v282: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:26:05.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:04 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:26:05.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:04 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:26:05.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:04 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:26:05.479 INFO:teuthology.orchestra.run.smithi102.stdout:epoch 110 2024-09-13T14:26:05.479 INFO:teuthology.orchestra.run.smithi102.stdout:fsid 08cee502-71d9-11ef-bceb-c7b262605968 2024-09-13T14:26:05.479 INFO:teuthology.orchestra.run.smithi102.stdout:created 2024-09-13T14:04:48.179879+0000 2024-09-13T14:26:05.479 INFO:teuthology.orchestra.run.smithi102.stdout:modified 2024-09-13T14:24:55.267939+0000 2024-09-13T14:26:05.479 INFO:teuthology.orchestra.run.smithi102.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-13T14:26:05.480 INFO:teuthology.orchestra.run.smithi102.stdout:crush_version 22 2024-09-13T14:26:05.480 INFO:teuthology.orchestra.run.smithi102.stdout:full_ratio 0.95 2024-09-13T14:26:05.480 INFO:teuthology.orchestra.run.smithi102.stdout:backfillfull_ratio 0.9 2024-09-13T14:26:05.480 INFO:teuthology.orchestra.run.smithi102.stdout:nearfull_ratio 0.85 2024-09-13T14:26:05.480 INFO:teuthology.orchestra.run.smithi102.stdout:require_min_compat_client luminous 2024-09-13T14:26:05.480 INFO:teuthology.orchestra.run.smithi102.stdout:min_compat_client luminous 2024-09-13T14:26:05.480 INFO:teuthology.orchestra.run.smithi102.stdout:require_osd_release reef 2024-09-13T14:26:05.480 INFO:teuthology.orchestra.run.smithi102.stdout:stretch_mode_enabled false 2024-09-13T14:26:05.480 INFO:teuthology.orchestra.run.smithi102.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-09-13T14:26:05.480 INFO:teuthology.orchestra.run.smithi102.stdout:pool 2 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 85 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.00 2024-09-13T14:26:05.480 INFO:teuthology.orchestra.run.smithi102.stdout:pool 3 'cephfs.foo.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 70 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.50 2024-09-13T14:26:05.480 INFO:teuthology.orchestra.run.smithi102.stdout:pool 4 'cephfs.foo.data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 71 flags hashpspool,bulk stripe_width 0 application cephfs read_balance_score 2.25 2024-09-13T14:26:05.481 INFO:teuthology.orchestra.run.smithi102.stdout:max_osd 8 2024-09-13T14:26:05.481 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0 up in weight 1 up_from 94 up_thru 109 down_at 91 last_clean_interval [8,90) [v2:172.21.15.102:6802/3285732759,v1:172.21.15.102:6803/3285732759] [v2:172.21.15.102:6804/3285732759,v1:172.21.15.102:6805/3285732759] exists,up b43b49e0-1cf0-47b8-aa85-37108f321220 2024-09-13T14:26:05.481 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1 up in weight 1 up_from 99 up_thru 109 down_at 96 last_clean_interval [13,95) [v2:172.21.15.102:6810/2240964461,v1:172.21.15.102:6811/2240964461] [v2:172.21.15.102:6812/2240964461,v1:172.21.15.102:6813/2240964461] exists,up 27e10d5e-c471-4132-9c73-83c5708c3b08 2024-09-13T14:26:05.481 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2 up in weight 1 up_from 104 up_thru 109 down_at 101 last_clean_interval [18,100) [v2:172.21.15.102:6818/4066184234,v1:172.21.15.102:6819/4066184234] [v2:172.21.15.102:6820/4066184234,v1:172.21.15.102:6821/4066184234] exists,up 40a94b2b-2c24-4164-91be-0816a1acf3d6 2024-09-13T14:26:05.481 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3 up in weight 1 up_from 109 up_thru 109 down_at 106 last_clean_interval [25,105) [v2:172.21.15.102:6826/1858757656,v1:172.21.15.102:6827/1858757656] [v2:172.21.15.102:6828/1858757656,v1:172.21.15.102:6829/1858757656] exists,up 2ed85faf-6610-4bcc-a09d-4a6f93f5c6a0 2024-09-13T14:26:05.481 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4 up in weight 1 up_from 30 up_thru 109 down_at 0 last_clean_interval [0,0) [v2:172.21.15.195:6800/2821651815,v1:172.21.15.195:6801/2821651815] [v2:172.21.15.195:6802/2821651815,v1:172.21.15.195:6803/2821651815] exists,up 8a45f350-fb3e-4049-bcc4-532d3e20c238 2024-09-13T14:26:05.481 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5 up in weight 1 up_from 38 up_thru 109 down_at 0 last_clean_interval [0,0) [v2:172.21.15.195:6808/3074139237,v1:172.21.15.195:6809/3074139237] [v2:172.21.15.195:6810/3074139237,v1:172.21.15.195:6811/3074139237] exists,up 363cf638-db26-4c86-b2be-e075fd09ac16 2024-09-13T14:26:05.481 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6 up in weight 1 up_from 46 up_thru 106 down_at 0 last_clean_interval [0,0) [v2:172.21.15.195:6816/4026627594,v1:172.21.15.195:6817/4026627594] [v2:172.21.15.195:6818/4026627594,v1:172.21.15.195:6819/4026627594] exists,up f37416e8-9c25-4860-a32f-0d1c404a8bc9 2024-09-13T14:26:05.481 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7 up in weight 1 up_from 54 up_thru 109 down_at 0 last_clean_interval [0,0) [v2:172.21.15.195:6824/2059403787,v1:172.21.15.195:6825/2059403787] [v2:172.21.15.195:6826/2059403787,v1:172.21.15.195:6827/2059403787] exists,up 9a384a4d-fddb-4fb3-843a-77c19cca0368 2024-09-13T14:26:05.481 INFO:teuthology.orchestra.run.smithi102.stdout:pg_upmap_items 4.4 [1,5] 2024-09-13T14:26:05.481 INFO:teuthology.orchestra.run.smithi102.stdout:pg_upmap_items 4.f [1,2] 2024-09-13T14:26:05.481 INFO:teuthology.orchestra.run.smithi102.stdout:pg_upmap_items 4.1d [1,2] 2024-09-13T14:26:05.482 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.102:0/3514811128 expires 2024-09-14T14:17:25.102165+0000 2024-09-13T14:26:05.482 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.102:6801/3856547061 expires 2024-09-14T14:17:25.102165+0000 2024-09-13T14:26:05.482 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.102:6800/3856547061 expires 2024-09-14T14:17:25.102165+0000 2024-09-13T14:26:05.482 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.102:0/2872102326 expires 2024-09-14T14:17:25.102165+0000 2024-09-13T14:26:05.482 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.102:0/3399546099 expires 2024-09-14T14:05:37.634624+0000 2024-09-13T14:26:05.482 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.102:0/1430433627 expires 2024-09-14T14:14:08.048329+0000 2024-09-13T14:26:05.482 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.102:0/2489373272 expires 2024-09-14T14:14:08.048329+0000 2024-09-13T14:26:05.482 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.195:6835/883376241 expires 2024-09-14T14:14:48.059120+0000 2024-09-13T14:26:05.482 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.102:6801/2761043787 expires 2024-09-14T14:05:15.286479+0000 2024-09-13T14:26:05.482 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.102:0/1574792592 expires 2024-09-14T14:05:15.286479+0000 2024-09-13T14:26:05.482 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.102:6801/63514412 expires 2024-09-14T14:10:09.037657+0000 2024-09-13T14:26:05.482 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.102:6800/63514412 expires 2024-09-14T14:10:09.037657+0000 2024-09-13T14:26:05.482 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.102:0/2083488703 expires 2024-09-14T14:14:08.048329+0000 2024-09-13T14:26:05.483 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.102:0/2975343136 expires 2024-09-14T14:17:25.102165+0000 2024-09-13T14:26:05.483 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.195:0/2524944732 expires 2024-09-14T14:14:48.059120+0000 2024-09-13T14:26:05.483 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.102:6800/2761043787 expires 2024-09-14T14:05:15.286479+0000 2024-09-13T14:26:05.483 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.102:6801/3480998245 expires 2024-09-14T14:05:37.634624+0000 2024-09-13T14:26:05.483 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.102:6800/3480998245 expires 2024-09-14T14:05:37.634624+0000 2024-09-13T14:26:05.483 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.195:0/1568198689 expires 2024-09-14T14:14:48.059120+0000 2024-09-13T14:26:05.483 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.195:6834/883376241 expires 2024-09-14T14:14:48.059120+0000 2024-09-13T14:26:05.483 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.102:0/1792178465 expires 2024-09-14T14:14:08.048329+0000 2024-09-13T14:26:05.483 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.102:0/120875803 expires 2024-09-14T14:17:25.102165+0000 2024-09-13T14:26:05.483 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.102:0/1160237252 expires 2024-09-14T14:10:09.037657+0000 2024-09-13T14:26:05.483 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.102:0/1582001159 expires 2024-09-14T14:05:15.286479+0000 2024-09-13T14:26:05.483 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.195:0/4292730726 expires 2024-09-14T14:14:48.059120+0000 2024-09-13T14:26:05.484 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.102:6800/1522897266 expires 2024-09-14T14:14:08.048329+0000 2024-09-13T14:26:05.484 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.102:0/1939570054 expires 2024-09-14T14:05:37.634624+0000 2024-09-13T14:26:05.484 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.102:0/3189088157 expires 2024-09-14T14:05:15.286479+0000 2024-09-13T14:26:05.484 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.195:0/3629133224 expires 2024-09-14T14:14:48.059120+0000 2024-09-13T14:26:05.484 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.102:0/2308924552 expires 2024-09-14T14:10:09.037657+0000 2024-09-13T14:26:05.484 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.102:0/2416257718 expires 2024-09-14T14:10:09.037657+0000 2024-09-13T14:26:05.484 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.102:6801/1522897266 expires 2024-09-14T14:14:08.048329+0000 2024-09-13T14:26:05.484 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.102:0/1765904351 expires 2024-09-14T14:10:09.037657+0000 2024-09-13T14:26:05.484 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.102:0/2444394840 expires 2024-09-14T14:05:37.634624+0000 2024-09-13T14:26:05.484 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.195:0/1729595567 expires 2024-09-14T14:14:48.059120+0000 2024-09-13T14:26:05.484 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.195:0/495121375 expires 2024-09-14T14:14:48.059120+0000 2024-09-13T14:26:06.097 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:05 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2752881602' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-13T14:26:06.097 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:05 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2752881602' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-13T14:26:06.118 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 94 up_thru 109 down_at 91 last_clean_interval [8,90) [v2:172.21.15.102:6802/3285732759,v1:172.21.15.102:6803/3285732759] [v2:172.21.15.102:6804/3285732759,v1:172.21.15.102:6805/3285732759] exists,up b43b49e0-1cf0-47b8-aa85-37108f321220', 'osd.1 up in weight 1 up_from 99 up_thru 109 down_at 96 last_clean_interval [13,95) [v2:172.21.15.102:6810/2240964461,v1:172.21.15.102:6811/2240964461] [v2:172.21.15.102:6812/2240964461,v1:172.21.15.102:6813/2240964461] exists,up 27e10d5e-c471-4132-9c73-83c5708c3b08', 'osd.2 up in weight 1 up_from 104 up_thru 109 down_at 101 last_clean_interval [18,100) [v2:172.21.15.102:6818/4066184234,v1:172.21.15.102:6819/4066184234] [v2:172.21.15.102:6820/4066184234,v1:172.21.15.102:6821/4066184234] exists,up 40a94b2b-2c24-4164-91be-0816a1acf3d6', 'osd.3 up in weight 1 up_from 109 up_thru 109 down_at 106 last_clean_interval [25,105) [v2:172.21.15.102:6826/1858757656,v1:172.21.15.102:6827/1858757656] [v2:172.21.15.102:6828/1858757656,v1:172.21.15.102:6829/1858757656] exists,up 2ed85faf-6610-4bcc-a09d-4a6f93f5c6a0', 'osd.4 up in weight 1 up_from 30 up_thru 109 down_at 0 last_clean_interval [0,0) [v2:172.21.15.195:6800/2821651815,v1:172.21.15.195:6801/2821651815] [v2:172.21.15.195:6802/2821651815,v1:172.21.15.195:6803/2821651815] exists,up 8a45f350-fb3e-4049-bcc4-532d3e20c238', 'osd.5 up in weight 1 up_from 38 up_thru 109 down_at 0 last_clean_interval [0,0) [v2:172.21.15.195:6808/3074139237,v1:172.21.15.195:6809/3074139237] [v2:172.21.15.195:6810/3074139237,v1:172.21.15.195:6811/3074139237] exists,up 363cf638-db26-4c86-b2be-e075fd09ac16', 'osd.6 up in weight 1 up_from 46 up_thru 106 down_at 0 last_clean_interval [0,0) [v2:172.21.15.195:6816/4026627594,v1:172.21.15.195:6817/4026627594] [v2:172.21.15.195:6818/4026627594,v1:172.21.15.195:6819/4026627594] exists,up f37416e8-9c25-4860-a32f-0d1c404a8bc9', 'osd.7 up in weight 1 up_from 54 up_thru 109 down_at 0 last_clean_interval [0,0) [v2:172.21.15.195:6824/2059403787,v1:172.21.15.195:6825/2059403787] [v2:172.21.15.195:6826/2059403787,v1:172.21.15.195:6827/2059403787] exists,up 9a384a4d-fddb-4fb3-843a-77c19cca0368'] 2024-09-13T14:26:06.118 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon mon.a config show 2024-09-13T14:26:06.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:05 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2752881602' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-13T14:26:06.315 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:26:06.696 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:26:06 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:26:06] "GET /metrics HTTP/1.1" 200 34014 "" "Prometheus/2.51.0" 2024-09-13T14:26:07.158 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:06 smithi102 ceph-mon[110183]: pgmap v283: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:26:07.159 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:06 smithi102 ceph-mon[115269]: pgmap v283: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:26:07.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:06 smithi195 ceph-mon[104262]: pgmap v283: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:26:07.913 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:26:07.913 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "mon.a", 2024-09-13T14:26:07.913 INFO:teuthology.orchestra.run.smithi102.stdout: "cluster": "ceph", 2024-09-13T14:26:07.913 INFO:teuthology.orchestra.run.smithi102.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-13T14:26:07.913 INFO:teuthology.orchestra.run.smithi102.stdout: "admin_socket_mode": "", 2024-09-13T14:26:07.914 INFO:teuthology.orchestra.run.smithi102.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-09-13T14:26:07.914 INFO:teuthology.orchestra.run.smithi102.stdout: "auth_client_required": "cephx, none", 2024-09-13T14:26:07.914 INFO:teuthology.orchestra.run.smithi102.stdout: "auth_cluster_required": "cephx", 2024-09-13T14:26:07.914 INFO:teuthology.orchestra.run.smithi102.stdout: "auth_debug": "true", 2024-09-13T14:26:07.914 INFO:teuthology.orchestra.run.smithi102.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-13T14:26:07.914 INFO:teuthology.orchestra.run.smithi102.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-13T14:26:07.914 INFO:teuthology.orchestra.run.smithi102.stdout: "auth_service_required": "cephx", 2024-09-13T14:26:07.914 INFO:teuthology.orchestra.run.smithi102.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-13T14:26:07.914 INFO:teuthology.orchestra.run.smithi102.stdout: "auth_supported": "", 2024-09-13T14:26:07.914 INFO:teuthology.orchestra.run.smithi102.stdout: "bdev_aio": "true", 2024-09-13T14:26:07.914 INFO:teuthology.orchestra.run.smithi102.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-13T14:26:07.914 INFO:teuthology.orchestra.run.smithi102.stdout: "bdev_aio_poll_ms": "250", 2024-09-13T14:26:07.915 INFO:teuthology.orchestra.run.smithi102.stdout: "bdev_aio_reap_max": "16", 2024-09-13T14:26:07.915 INFO:teuthology.orchestra.run.smithi102.stdout: "bdev_async_discard_threads": "0", 2024-09-13T14:26:07.915 INFO:teuthology.orchestra.run.smithi102.stdout: "bdev_block_size": "4096", 2024-09-13T14:26:07.915 INFO:teuthology.orchestra.run.smithi102.stdout: "bdev_debug_aio": "false", 2024-09-13T14:26:07.915 INFO:teuthology.orchestra.run.smithi102.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-13T14:26:07.915 INFO:teuthology.orchestra.run.smithi102.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-13T14:26:07.915 INFO:teuthology.orchestra.run.smithi102.stdout: "bdev_debug_inflight_ios": "false", 2024-09-13T14:26:07.915 INFO:teuthology.orchestra.run.smithi102.stdout: "bdev_enable_discard": "false", 2024-09-13T14:26:07.915 INFO:teuthology.orchestra.run.smithi102.stdout: "bdev_flock_retry": "3", 2024-09-13T14:26:07.915 INFO:teuthology.orchestra.run.smithi102.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-13T14:26:07.915 INFO:teuthology.orchestra.run.smithi102.stdout: "bdev_inject_crash": "0", 2024-09-13T14:26:07.915 INFO:teuthology.orchestra.run.smithi102.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-13T14:26:07.915 INFO:teuthology.orchestra.run.smithi102.stdout: "bdev_ioring": "false", 2024-09-13T14:26:07.916 INFO:teuthology.orchestra.run.smithi102.stdout: "bdev_ioring_hipri": "false", 2024-09-13T14:26:07.916 INFO:teuthology.orchestra.run.smithi102.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-13T14:26:07.916 INFO:teuthology.orchestra.run.smithi102.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-13T14:26:07.916 INFO:teuthology.orchestra.run.smithi102.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-13T14:26:07.916 INFO:teuthology.orchestra.run.smithi102.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-13T14:26:07.916 INFO:teuthology.orchestra.run.smithi102.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2024-09-13T14:26:07.916 INFO:teuthology.orchestra.run.smithi102.stdout: "bdev_stalled_read_warn_threshold": "1", 2024-09-13T14:26:07.916 INFO:teuthology.orchestra.run.smithi102.stdout: "bdev_type": "", 2024-09-13T14:26:07.916 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_alloc_size": "1048576", 2024-09-13T14:26:07.916 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_allocator": "hybrid", 2024-09-13T14:26:07.916 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_buffered_io": "true", 2024-09-13T14:26:07.916 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_check_for_zeros": "false", 2024-09-13T14:26:07.916 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-13T14:26:07.916 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-13T14:26:07.917 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_compact_log_sync": "false", 2024-09-13T14:26:07.917 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-13T14:26:07.917 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-13T14:26:07.917 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-13T14:26:07.917 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-13T14:26:07.917 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_max_log_runway": "4194304", 2024-09-13T14:26:07.917 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_max_prefetch": "1048576", 2024-09-13T14:26:07.917 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_min_flush_size": "524288", 2024-09-13T14:26:07.917 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_min_log_runway": "1048576", 2024-09-13T14:26:07.917 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_replay_recovery": "false", 2024-09-13T14:26:07.917 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-13T14:26:07.917 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-13T14:26:07.917 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_sync_write": "false", 2024-09-13T14:26:07.918 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-13T14:26:07.918 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-13T14:26:07.918 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-13T14:26:07.918 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_allocation_from_file": "true", 2024-09-13T14:26:07.918 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_allocator": "hybrid", 2024-09-13T14:26:07.918 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_async_db_compaction": "true", 2024-09-13T14:26:07.918 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-13T14:26:07.918 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-13T14:26:07.918 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-13T14:26:07.918 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-13T14:26:07.918 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_bdev_label_multi": "true", 2024-09-13T14:26:07.918 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2024-09-13T14:26:07.919 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_bdev_label_require_all": "true", 2024-09-13T14:26:07.919 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-13T14:26:07.919 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-13T14:26:07.919 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-13T14:26:07.919 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_block_create": "true", 2024-09-13T14:26:07.919 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_block_db_create": "false", 2024-09-13T14:26:07.919 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_block_db_path": "", 2024-09-13T14:26:07.919 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_block_db_size": "0", 2024-09-13T14:26:07.919 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_block_path": "", 2024-09-13T14:26:07.919 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_block_preallocate_file": "false", 2024-09-13T14:26:07.919 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_block_size": "107374182400", 2024-09-13T14:26:07.919 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_block_wal_create": "false", 2024-09-13T14:26:07.919 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_block_wal_path": "", 2024-09-13T14:26:07.920 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_block_wal_size": "100663296", 2024-09-13T14:26:07.920 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_bluefs": "true", 2024-09-13T14:26:07.920 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-13T14:26:07.920 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-13T14:26:07.920 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-13T14:26:07.920 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_btree2_alloc_weight_factor": "2.000000", 2024-09-13T14:26:07.920 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-13T14:26:07.920 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-13T14:26:07.920 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-13T14:26:07.920 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-13T14:26:07.920 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-13T14:26:07.920 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_autotune": "true", 2024-09-13T14:26:07.920 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-13T14:26:07.921 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-13T14:26:07.921 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-13T14:26:07.921 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-13T14:26:07.921 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_size": "0", 2024-09-13T14:26:07.921 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-13T14:26:07.921 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-13T14:26:07.921 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-13T14:26:07.921 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-13T14:26:07.921 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_type": "2q", 2024-09-13T14:26:07.921 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-13T14:26:07.921 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_clone_cow": "true", 2024-09-13T14:26:07.921 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-13T14:26:07.921 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-13T14:26:07.922 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-13T14:26:07.922 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-13T14:26:07.922 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-13T14:26:07.922 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-13T14:26:07.922 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-13T14:26:07.922 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_compression_mode": "none", 2024-09-13T14:26:07.922 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-13T14:26:07.922 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_csum_type": "crc32c", 2024-09-13T14:26:07.922 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-09-13T14:26:07.923 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-13T14:26:07.923 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_debug_freelist": "false", 2024-09-13T14:26:07.923 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-13T14:26:07.923 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-13T14:26:07.923 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-13T14:26:07.923 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-13T14:26:07.923 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-13T14:26:07.923 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-13T14:26:07.923 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-13T14:26:07.923 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-13T14:26:07.923 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-13T14:26:07.923 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-13T14:26:07.923 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-13T14:26:07.924 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-13T14:26:07.924 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_debug_small_allocations": "0", 2024-09-13T14:26:07.924 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-13T14:26:07.924 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_default_buffered_read": "true", 2024-09-13T14:26:07.924 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_default_buffered_write": "false", 2024-09-13T14:26:07.924 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-13T14:26:07.924 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-13T14:26:07.924 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-13T14:26:07.924 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_elastic_shared_blobs": "true", 2024-09-13T14:26:07.924 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-13T14:26:07.924 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-13T14:26:07.924 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-13T14:26:07.924 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-13T14:26:07.924 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-13T14:26:07.925 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_fail_eio": "false", 2024-09-13T14:26:07.925 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-13T14:26:07.925 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-13T14:26:07.925 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-13T14:26:07.925 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-13T14:26:07.925 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-13T14:26:07.925 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-13T14:26:07.925 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_fsck_on_mount": "false", 2024-09-13T14:26:07.925 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-13T14:26:07.925 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_fsck_on_umount": "false", 2024-09-13T14:26:07.925 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-13T14:26:07.925 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-13T14:26:07.925 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-13T14:26:07.925 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-13T14:26:07.926 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-13T14:26:07.926 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-13T14:26:07.926 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-13T14:26:07.926 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-13T14:26:07.926 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_ignore_data_csum": "false", 2024-09-13T14:26:07.926 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-13T14:26:07.926 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-13T14:26:07.926 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-13T14:26:07.926 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-13T14:26:07.926 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_log_op_age": "5.000000", 2024-09-13T14:26:07.926 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_max_alloc_size": "0", 2024-09-13T14:26:07.926 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_max_blob_size": "0", 2024-09-13T14:26:07.927 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-13T14:26:07.927 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-13T14:26:07.927 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-13T14:26:07.927 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_max_deferred_txc": "32", 2024-09-13T14:26:07.927 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_min_alloc_size": "0", 2024-09-13T14:26:07.927 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-13T14:26:07.927 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-13T14:26:07.927 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_nid_prealloc": "1024", 2024-09-13T14:26:07.927 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-13T14:26:07.927 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-13T14:26:07.928 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-13T14:26:07.928 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-13T14:26:07.928 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_retry_disk_reads": "3", 2024-09-13T14:26:07.928 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_rocksdb_cf": "true", 2024-09-13T14:26:07.928 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-09-13T14:26:07.928 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-09-13T14:26:07.929 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-13T14:26:07.929 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2024-09-13T14:26:07.929 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_slow_ops_warn_threshold": "1", 2024-09-13T14:26:07.929 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-13T14:26:07.929 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-13T14:26:07.929 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-13T14:26:07.929 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_spdk_mem": "512", 2024-09-13T14:26:07.929 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-13T14:26:07.929 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-13T14:26:07.929 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-13T14:26:07.929 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-13T14:26:07.929 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-13T14:26:07.929 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-13T14:26:07.930 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-13T14:26:07.930 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_tracing": "false", 2024-09-13T14:26:07.930 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-13T14:26:07.930 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-13T14:26:07.930 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-13T14:26:07.930 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-13T14:26:07.930 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-13T14:26:07.930 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-13T14:26:07.930 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-13T14:26:07.930 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-13T14:26:07.930 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-13T14:26:07.930 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_write_v2": "false", 2024-09-13T14:26:07.930 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_write_v2_random": "false", 2024-09-13T14:26:07.930 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_zero_block_detection": "false", 2024-09-13T14:26:07.931 INFO:teuthology.orchestra.run.smithi102.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-13T14:26:07.931 INFO:teuthology.orchestra.run.smithi102.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-13T14:26:07.931 INFO:teuthology.orchestra.run.smithi102.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-13T14:26:07.931 INFO:teuthology.orchestra.run.smithi102.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-13T14:26:07.931 INFO:teuthology.orchestra.run.smithi102.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-13T14:26:07.931 INFO:teuthology.orchestra.run.smithi102.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-13T14:26:07.931 INFO:teuthology.orchestra.run.smithi102.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-13T14:26:07.931 INFO:teuthology.orchestra.run.smithi102.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-09-13T14:26:07.931 INFO:teuthology.orchestra.run.smithi102.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-13T14:26:07.931 INFO:teuthology.orchestra.run.smithi102.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-13T14:26:07.931 INFO:teuthology.orchestra.run.smithi102.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-13T14:26:07.931 INFO:teuthology.orchestra.run.smithi102.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-13T14:26:07.931 INFO:teuthology.orchestra.run.smithi102.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-13T14:26:07.932 INFO:teuthology.orchestra.run.smithi102.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-13T14:26:07.932 INFO:teuthology.orchestra.run.smithi102.stdout: "cephx_cluster_require_signatures": "false", 2024-09-13T14:26:07.932 INFO:teuthology.orchestra.run.smithi102.stdout: "cephx_cluster_require_version": "2", 2024-09-13T14:26:07.932 INFO:teuthology.orchestra.run.smithi102.stdout: "cephx_require_signatures": "false", 2024-09-13T14:26:07.932 INFO:teuthology.orchestra.run.smithi102.stdout: "cephx_require_version": "2", 2024-09-13T14:26:07.932 INFO:teuthology.orchestra.run.smithi102.stdout: "cephx_service_require_signatures": "false", 2024-09-13T14:26:07.932 INFO:teuthology.orchestra.run.smithi102.stdout: "cephx_service_require_version": "2", 2024-09-13T14:26:07.932 INFO:teuthology.orchestra.run.smithi102.stdout: "cephx_sign_messages": "true", 2024-09-13T14:26:07.932 INFO:teuthology.orchestra.run.smithi102.stdout: "chdir": "", 2024-09-13T14:26:07.933 INFO:teuthology.orchestra.run.smithi102.stdout: "client_acl_type": "", 2024-09-13T14:26:07.933 INFO:teuthology.orchestra.run.smithi102.stdout: "client_asio_thread_count": "2", 2024-09-13T14:26:07.933 INFO:teuthology.orchestra.run.smithi102.stdout: "client_cache_mid": "0.750000", 2024-09-13T14:26:07.933 INFO:teuthology.orchestra.run.smithi102.stdout: "client_cache_size": "16384", 2024-09-13T14:26:07.933 INFO:teuthology.orchestra.run.smithi102.stdout: "client_caps_release_delay": "5", 2024-09-13T14:26:07.933 INFO:teuthology.orchestra.run.smithi102.stdout: "client_check_pool_perm": "true", 2024-09-13T14:26:07.933 INFO:teuthology.orchestra.run.smithi102.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-13T14:26:07.933 INFO:teuthology.orchestra.run.smithi102.stdout: "client_debug_force_sync_read": "false", 2024-09-13T14:26:07.933 INFO:teuthology.orchestra.run.smithi102.stdout: "client_debug_getattr_caps": "false", 2024-09-13T14:26:07.933 INFO:teuthology.orchestra.run.smithi102.stdout: "client_debug_inject_features": "", 2024-09-13T14:26:07.933 INFO:teuthology.orchestra.run.smithi102.stdout: "client_debug_inject_tick_delay": "0", 2024-09-13T14:26:07.933 INFO:teuthology.orchestra.run.smithi102.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-13T14:26:07.933 INFO:teuthology.orchestra.run.smithi102.stdout: "client_die_on_failed_remount": "false", 2024-09-13T14:26:07.934 INFO:teuthology.orchestra.run.smithi102.stdout: "client_dirsize_rbytes": "true", 2024-09-13T14:26:07.934 INFO:teuthology.orchestra.run.smithi102.stdout: "client_force_lazyio": "false", 2024-09-13T14:26:07.934 INFO:teuthology.orchestra.run.smithi102.stdout: "client_fs": "", 2024-09-13T14:26:07.934 INFO:teuthology.orchestra.run.smithi102.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-13T14:26:07.934 INFO:teuthology.orchestra.run.smithi102.stdout: "client_inject_release_failure": "false", 2024-09-13T14:26:07.934 INFO:teuthology.orchestra.run.smithi102.stdout: "client_max_inline_size": "4096", 2024-09-13T14:26:07.934 INFO:teuthology.orchestra.run.smithi102.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-13T14:26:07.934 INFO:teuthology.orchestra.run.smithi102.stdout: "client_mds_namespace": "", 2024-09-13T14:26:07.934 INFO:teuthology.orchestra.run.smithi102.stdout: "client_metadata": "", 2024-09-13T14:26:07.934 INFO:teuthology.orchestra.run.smithi102.stdout: "client_mount_gid": "-1", 2024-09-13T14:26:07.934 INFO:teuthology.orchestra.run.smithi102.stdout: "client_mount_timeout": "300", 2024-09-13T14:26:07.934 INFO:teuthology.orchestra.run.smithi102.stdout: "client_mount_uid": "-1", 2024-09-13T14:26:07.934 INFO:teuthology.orchestra.run.smithi102.stdout: "client_mountpoint": "/", 2024-09-13T14:26:07.934 INFO:teuthology.orchestra.run.smithi102.stdout: "client_notify_timeout": "10", 2024-09-13T14:26:07.935 INFO:teuthology.orchestra.run.smithi102.stdout: "client_oc": "true", 2024-09-13T14:26:07.935 INFO:teuthology.orchestra.run.smithi102.stdout: "client_oc_max_dirty": "104857600", 2024-09-13T14:26:07.935 INFO:teuthology.orchestra.run.smithi102.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-13T14:26:07.935 INFO:teuthology.orchestra.run.smithi102.stdout: "client_oc_max_objects": "1000", 2024-09-13T14:26:07.935 INFO:teuthology.orchestra.run.smithi102.stdout: "client_oc_size": "209715200", 2024-09-13T14:26:07.935 INFO:teuthology.orchestra.run.smithi102.stdout: "client_oc_target_dirty": "8388608", 2024-09-13T14:26:07.935 INFO:teuthology.orchestra.run.smithi102.stdout: "client_permissions": "true", 2024-09-13T14:26:07.935 INFO:teuthology.orchestra.run.smithi102.stdout: "client_quota": "true", 2024-09-13T14:26:07.935 INFO:teuthology.orchestra.run.smithi102.stdout: "client_quota_df": "true", 2024-09-13T14:26:07.935 INFO:teuthology.orchestra.run.smithi102.stdout: "client_readahead_max_bytes": "0", 2024-09-13T14:26:07.935 INFO:teuthology.orchestra.run.smithi102.stdout: "client_readahead_max_periods": "4", 2024-09-13T14:26:07.935 INFO:teuthology.orchestra.run.smithi102.stdout: "client_readahead_min": "131072", 2024-09-13T14:26:07.935 INFO:teuthology.orchestra.run.smithi102.stdout: "client_reconnect_stale": "false", 2024-09-13T14:26:07.936 INFO:teuthology.orchestra.run.smithi102.stdout: "client_shutdown_timeout": "30", 2024-09-13T14:26:07.936 INFO:teuthology.orchestra.run.smithi102.stdout: "client_snapdir": ".snap", 2024-09-13T14:26:07.936 INFO:teuthology.orchestra.run.smithi102.stdout: "client_tick_interval": "1", 2024-09-13T14:26:07.936 INFO:teuthology.orchestra.run.smithi102.stdout: "client_trace": "", 2024-09-13T14:26:07.936 INFO:teuthology.orchestra.run.smithi102.stdout: "client_try_dentry_invalidate": "false", 2024-09-13T14:26:07.936 INFO:teuthology.orchestra.run.smithi102.stdout: "client_use_faked_inos": "false", 2024-09-13T14:26:07.936 INFO:teuthology.orchestra.run.smithi102.stdout: "client_use_random_mds": "false", 2024-09-13T14:26:07.936 INFO:teuthology.orchestra.run.smithi102.stdout: "clog_to_graylog": "false", 2024-09-13T14:26:07.937 INFO:teuthology.orchestra.run.smithi102.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-13T14:26:07.937 INFO:teuthology.orchestra.run.smithi102.stdout: "clog_to_graylog_port": "12201", 2024-09-13T14:26:07.937 INFO:teuthology.orchestra.run.smithi102.stdout: "clog_to_monitors": "default=true", 2024-09-13T14:26:07.937 INFO:teuthology.orchestra.run.smithi102.stdout: "clog_to_syslog": "false", 2024-09-13T14:26:07.937 INFO:teuthology.orchestra.run.smithi102.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-13T14:26:07.937 INFO:teuthology.orchestra.run.smithi102.stdout: "clog_to_syslog_level": "info", 2024-09-13T14:26:07.937 INFO:teuthology.orchestra.run.smithi102.stdout: "cluster_addr": "-", 2024-09-13T14:26:07.937 INFO:teuthology.orchestra.run.smithi102.stdout: "cluster_network": "", 2024-09-13T14:26:07.937 INFO:teuthology.orchestra.run.smithi102.stdout: "cluster_network_interface": "", 2024-09-13T14:26:07.937 INFO:teuthology.orchestra.run.smithi102.stdout: "compressor_zlib_isal": "false", 2024-09-13T14:26:07.937 INFO:teuthology.orchestra.run.smithi102.stdout: "compressor_zlib_level": "5", 2024-09-13T14:26:07.938 INFO:teuthology.orchestra.run.smithi102.stdout: "compressor_zlib_winsize": "-15", 2024-09-13T14:26:07.938 INFO:teuthology.orchestra.run.smithi102.stdout: "compressor_zstd_level": "1", 2024-09-13T14:26:07.938 INFO:teuthology.orchestra.run.smithi102.stdout: "container_image": "quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f", 2024-09-13T14:26:07.938 INFO:teuthology.orchestra.run.smithi102.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-13T14:26:07.938 INFO:teuthology.orchestra.run.smithi102.stdout: "crimson_alien_op_num_threads": "6", 2024-09-13T14:26:07.938 INFO:teuthology.orchestra.run.smithi102.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-13T14:26:07.938 INFO:teuthology.orchestra.run.smithi102.stdout: "crimson_osd_obc_lru_size": "512", 2024-09-13T14:26:07.938 INFO:teuthology.orchestra.run.smithi102.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-13T14:26:07.938 INFO:teuthology.orchestra.run.smithi102.stdout: "crimson_osd_stat_interval": "0", 2024-09-13T14:26:07.939 INFO:teuthology.orchestra.run.smithi102.stdout: "crimson_seastar_cpu_cores": "", 2024-09-13T14:26:07.939 INFO:teuthology.orchestra.run.smithi102.stdout: "crimson_seastar_num_threads": "0", 2024-09-13T14:26:07.939 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location": "", 2024-09-13T14:26:07.939 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location_hook": "", 2024-09-13T14:26:07.939 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_location_hook_timeout": "10", 2024-09-13T14:26:07.939 INFO:teuthology.orchestra.run.smithi102.stdout: "daemonize": "false", 2024-09-13T14:26:07.939 INFO:teuthology.orchestra.run.smithi102.stdout: "daos_pool": "tank", 2024-09-13T14:26:07.939 INFO:teuthology.orchestra.run.smithi102.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-09-13T14:26:07.939 INFO:teuthology.orchestra.run.smithi102.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-09-13T14:26:07.939 INFO:teuthology.orchestra.run.smithi102.stdout: "dbstore_db_name_prefix": "dbstore", 2024-09-13T14:26:07.939 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_alienstore": "0/5", 2024-09-13T14:26:07.939 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_allow_any_pool_priority": "false", 2024-09-13T14:26:07.939 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_asok": "1/5", 2024-09-13T14:26:07.939 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_asok_assert_abort": "false", 2024-09-13T14:26:07.940 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_asserts_on_shutdown": "true", 2024-09-13T14:26:07.940 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_auth": "1/5", 2024-09-13T14:26:07.940 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_bdev": "1/3", 2024-09-13T14:26:07.940 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_bluefs": "1/5", 2024-09-13T14:26:07.940 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_bluestore": "1/5", 2024-09-13T14:26:07.940 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_buffer": "0/1", 2024-09-13T14:26:07.940 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_ceph_dedup": "0/5", 2024-09-13T14:26:07.940 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_ceph_exporter": "1/5", 2024-09-13T14:26:07.940 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_cephfs_mirror": "0/5", 2024-09-13T14:26:07.940 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_cephsqlite": "0/5", 2024-09-13T14:26:07.940 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_client": "0/5", 2024-09-13T14:26:07.940 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_compressor": "1/5", 2024-09-13T14:26:07.940 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_context": "0/1", 2024-09-13T14:26:07.941 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_crimson_interrupt": "0/5", 2024-09-13T14:26:07.941 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_crush": "1/1", 2024-09-13T14:26:07.941 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_crypto": "1/5", 2024-09-13T14:26:07.941 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_cyanstore": "0/5", 2024-09-13T14:26:07.941 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_deliberately_leak_memory": "false", 2024-09-13T14:26:07.941 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_disable_randomized_ping": "false", 2024-09-13T14:26:07.941 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_dpdk": "1/5", 2024-09-13T14:26:07.941 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_eventtrace": "1/5", 2024-09-13T14:26:07.941 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_filer": "0/1", 2024-09-13T14:26:07.941 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_filestore": "1/3", 2024-09-13T14:26:07.941 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_finisher": "1/1", 2024-09-13T14:26:07.941 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_fuse": "1/5", 2024-09-13T14:26:07.941 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_heartbeat_testing_span": "0", 2024-09-13T14:26:07.941 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_heartbeatmap": "1/5", 2024-09-13T14:26:07.942 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-13T14:26:07.942 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_javaclient": "1/5", 2024-09-13T14:26:07.942 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_journal": "1/3", 2024-09-13T14:26:07.942 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_journaler": "0/5", 2024-09-13T14:26:07.942 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_kstore": "1/5", 2024-09-13T14:26:07.942 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_lockdep": "0/1", 2024-09-13T14:26:07.943 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_mclock": "1/5", 2024-09-13T14:26:07.943 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_mds": "1/5", 2024-09-13T14:26:07.943 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_mds_balancer": "1/5", 2024-09-13T14:26:07.943 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_mds_locker": "1/5", 2024-09-13T14:26:07.943 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_mds_log": "1/5", 2024-09-13T14:26:07.943 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_mds_log_expire": "1/5", 2024-09-13T14:26:07.943 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_mds_migrator": "1/5", 2024-09-13T14:26:07.943 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_mds_quiesce": "3/5", 2024-09-13T14:26:07.943 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_memstore": "1/5", 2024-09-13T14:26:07.944 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_mgr": "2/5", 2024-09-13T14:26:07.944 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_mgrc": "1/5", 2024-09-13T14:26:07.944 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_mon": "20/20", 2024-09-13T14:26:07.944 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_monc": "0/10", 2024-09-13T14:26:07.944 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_ms": "1/1", 2024-09-13T14:26:07.944 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_none": "0/5", 2024-09-13T14:26:07.944 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_objclass": "0/5", 2024-09-13T14:26:07.944 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_objectcacher": "0/5", 2024-09-13T14:26:07.944 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_objecter": "0/1", 2024-09-13T14:26:07.944 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_optracker": "0/5", 2024-09-13T14:26:07.944 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_osd": "1/5", 2024-09-13T14:26:07.944 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_paxos": "20/20", 2024-09-13T14:26:07.944 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_perfcounter": "1/5", 2024-09-13T14:26:07.944 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_prioritycache": "1/5", 2024-09-13T14:26:07.945 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_rados": "0/5", 2024-09-13T14:26:07.945 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_rbd": "0/5", 2024-09-13T14:26:07.945 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_rbd_mirror": "0/5", 2024-09-13T14:26:07.945 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_rbd_pwl": "0/5", 2024-09-13T14:26:07.945 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_rbd_replay": "0/5", 2024-09-13T14:26:07.945 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_refs": "0/0", 2024-09-13T14:26:07.945 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_reserver": "1/1", 2024-09-13T14:26:07.945 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_rgw": "1/5", 2024-09-13T14:26:07.945 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_rgw_access": "1/5", 2024-09-13T14:26:07.945 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_rgw_datacache": "1/5", 2024-09-13T14:26:07.945 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_rgw_dbstore": "1/5", 2024-09-13T14:26:07.945 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_rgw_flight": "1/5", 2024-09-13T14:26:07.945 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_rgw_lifecycle": "1/5", 2024-09-13T14:26:07.947 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_rgw_notification": "1/5", 2024-09-13T14:26:07.947 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_rgw_sync": "1/5", 2024-09-13T14:26:07.947 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_rocksdb": "4/5", 2024-09-13T14:26:07.947 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_seastore": "0/5", 2024-09-13T14:26:07.947 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_seastore_backref": "0/5", 2024-09-13T14:26:07.947 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_seastore_cache": "0/5", 2024-09-13T14:26:07.947 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_seastore_cleaner": "0/5", 2024-09-13T14:26:07.947 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_seastore_device": "0/5", 2024-09-13T14:26:07.948 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_seastore_epm": "0/5", 2024-09-13T14:26:07.948 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-09-13T14:26:07.948 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_seastore_journal": "0/5", 2024-09-13T14:26:07.948 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_seastore_lba": "0/5", 2024-09-13T14:26:07.948 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_seastore_odata": "0/5", 2024-09-13T14:26:07.948 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_seastore_omap": "0/5", 2024-09-13T14:26:07.948 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_seastore_onode": "0/5", 2024-09-13T14:26:07.948 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_seastore_t": "0/5", 2024-09-13T14:26:07.948 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_seastore_tm": "0/5", 2024-09-13T14:26:07.948 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_striper": "0/1", 2024-09-13T14:26:07.948 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_test": "0/5", 2024-09-13T14:26:07.948 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_throttle": "1/1", 2024-09-13T14:26:07.948 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_timer": "0/1", 2024-09-13T14:26:07.948 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_tp": "0/5", 2024-09-13T14:26:07.949 INFO:teuthology.orchestra.run.smithi102.stdout: "debug_trace": "1/5", 2024-09-13T14:26:07.949 INFO:teuthology.orchestra.run.smithi102.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-13T14:26:07.949 INFO:teuthology.orchestra.run.smithi102.stdout: "device_failure_prediction_mode": "none", 2024-09-13T14:26:07.949 INFO:teuthology.orchestra.run.smithi102.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-13T14:26:07.949 INFO:teuthology.orchestra.run.smithi102.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-09-13T14:26:07.949 INFO:teuthology.orchestra.run.smithi102.stdout: "err_to_graylog": "false", 2024-09-13T14:26:07.949 INFO:teuthology.orchestra.run.smithi102.stdout: "err_to_journald": "false", 2024-09-13T14:26:07.949 INFO:teuthology.orchestra.run.smithi102.stdout: "err_to_stderr": "true", 2024-09-13T14:26:07.949 INFO:teuthology.orchestra.run.smithi102.stdout: "err_to_syslog": "false", 2024-09-13T14:26:07.949 INFO:teuthology.orchestra.run.smithi102.stdout: "event_tracing": "false", 2024-09-13T14:26:07.949 INFO:teuthology.orchestra.run.smithi102.stdout: "exporter_addr": "0.0.0.0", 2024-09-13T14:26:07.949 INFO:teuthology.orchestra.run.smithi102.stdout: "exporter_cert_file": "", 2024-09-13T14:26:07.949 INFO:teuthology.orchestra.run.smithi102.stdout: "exporter_http_port": "9926", 2024-09-13T14:26:07.950 INFO:teuthology.orchestra.run.smithi102.stdout: "exporter_key_file": "", 2024-09-13T14:26:07.950 INFO:teuthology.orchestra.run.smithi102.stdout: "exporter_prio_limit": "5", 2024-09-13T14:26:07.950 INFO:teuthology.orchestra.run.smithi102.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-13T14:26:07.950 INFO:teuthology.orchestra.run.smithi102.stdout: "exporter_sort_metrics": "true", 2024-09-13T14:26:07.950 INFO:teuthology.orchestra.run.smithi102.stdout: "exporter_stats_period": "5", 2024-09-13T14:26:07.950 INFO:teuthology.orchestra.run.smithi102.stdout: "fake_statfs_for_testing": "0", 2024-09-13T14:26:07.950 INFO:teuthology.orchestra.run.smithi102.stdout: "fatal_signal_handlers": "true", 2024-09-13T14:26:07.950 INFO:teuthology.orchestra.run.smithi102.stdout: "filer_max_purge_ops": "10", 2024-09-13T14:26:07.950 INFO:teuthology.orchestra.run.smithi102.stdout: "filer_max_truncate_ops": "128", 2024-09-13T14:26:07.950 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_apply_finisher_threads": "1", 2024-09-13T14:26:07.950 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_blackhole": "false", 2024-09-13T14:26:07.950 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_btrfs_clone_range": "true", 2024-09-13T14:26:07.950 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_btrfs_snap": "true", 2024-09-13T14:26:07.950 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_caller_concurrency": "10", 2024-09-13T14:26:07.951 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_collect_device_partition_information": "true", 2024-09-13T14:26:07.951 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_commit_timeout": "600.000000", 2024-09-13T14:26:07.951 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_debug_inject_read_err": "false", 2024-09-13T14:26:07.951 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_debug_omap_check": "false", 2024-09-13T14:26:07.951 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_debug_verify_split": "false", 2024-09-13T14:26:07.951 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_dump_file": "", 2024-09-13T14:26:07.951 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-13T14:26:07.951 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-13T14:26:07.951 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_fadvise": "true", 2024-09-13T14:26:07.951 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_fail_eio": "true", 2024-09-13T14:26:07.951 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_fd_cache_shards": "16", 2024-09-13T14:26:07.951 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_fd_cache_size": "128", 2024-09-13T14:26:07.951 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_fiemap": "false", 2024-09-13T14:26:07.951 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_fiemap_threshold": "4096", 2024-09-13T14:26:07.952 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-13T14:26:07.952 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-13T14:26:07.952 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_inject_stall": "0", 2024-09-13T14:26:07.952 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_journal_parallel": "false", 2024-09-13T14:26:07.952 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_journal_trailing": "false", 2024-09-13T14:26:07.952 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_journal_writeahead": "false", 2024-09-13T14:26:07.952 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_kill_at": "0", 2024-09-13T14:26:07.952 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-13T14:26:07.952 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-13T14:26:07.952 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-13T14:26:07.952 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-13T14:26:07.952 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-13T14:26:07.952 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_max_inline_xattrs": "0", 2024-09-13T14:26:07.952 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-13T14:26:07.953 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-13T14:26:07.953 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-13T14:26:07.953 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-13T14:26:07.953 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_max_xattr_value_size": "0", 2024-09-13T14:26:07.953 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-13T14:26:07.953 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-13T14:26:07.953 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-13T14:26:07.953 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_merge_threshold": "-10", 2024-09-13T14:26:07.953 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-13T14:26:07.953 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_odsync_write": "false", 2024-09-13T14:26:07.953 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_omap_backend": "rocksdb", 2024-09-13T14:26:07.953 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_omap_backend_path": "", 2024-09-13T14:26:07.953 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-13T14:26:07.953 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-13T14:26:07.954 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-13T14:26:07.954 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_op_thread_timeout": "60", 2024-09-13T14:26:07.954 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_op_threads": "2", 2024-09-13T14:26:07.954 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_punch_hole": "false", 2024-09-13T14:26:07.954 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-13T14:26:07.954 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-13T14:26:07.954 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-13T14:26:07.954 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-13T14:26:07.954 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-13T14:26:07.954 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-13T14:26:07.954 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-13T14:26:07.954 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-13T14:26:07.954 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-13T14:26:07.954 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_queue_max_ops": "50", 2024-09-13T14:26:07.955 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-13T14:26:07.955 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_seek_data_hole": "false", 2024-09-13T14:26:07.955 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_sloppy_crc": "false", 2024-09-13T14:26:07.955 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-13T14:26:07.955 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_splice": "false", 2024-09-13T14:26:07.955 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_split_multiple": "2", 2024-09-13T14:26:07.955 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_split_rand_factor": "20", 2024-09-13T14:26:07.955 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_update_to": "1000", 2024-09-13T14:26:07.955 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-13T14:26:07.955 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-13T14:26:07.955 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-13T14:26:07.955 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-13T14:26:07.955 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-13T14:26:07.955 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-13T14:26:07.956 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_wbthrottle_enable": "true", 2024-09-13T14:26:07.956 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-13T14:26:07.956 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-13T14:26:07.956 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-13T14:26:07.956 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-13T14:26:07.956 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-13T14:26:07.956 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-13T14:26:07.956 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_xfs_extsize": "false", 2024-09-13T14:26:07.956 INFO:teuthology.orchestra.run.smithi102.stdout: "filestore_zfs_snap": "false", 2024-09-13T14:26:07.956 INFO:teuthology.orchestra.run.smithi102.stdout: "fio_dir": "/tmp/fio", 2024-09-13T14:26:07.956 INFO:teuthology.orchestra.run.smithi102.stdout: "fsid": "08cee502-71d9-11ef-bceb-c7b262605968", 2024-09-13T14:26:07.956 INFO:teuthology.orchestra.run.smithi102.stdout: "fuse_allow_other": "true", 2024-09-13T14:26:07.957 INFO:teuthology.orchestra.run.smithi102.stdout: "fuse_atomic_o_trunc": "true", 2024-09-13T14:26:07.957 INFO:teuthology.orchestra.run.smithi102.stdout: "fuse_big_writes": "true", 2024-09-13T14:26:07.957 INFO:teuthology.orchestra.run.smithi102.stdout: "fuse_debug": "false", 2024-09-13T14:26:07.957 INFO:teuthology.orchestra.run.smithi102.stdout: "fuse_default_permissions": "false", 2024-09-13T14:26:07.957 INFO:teuthology.orchestra.run.smithi102.stdout: "fuse_disable_pagecache": "false", 2024-09-13T14:26:07.957 INFO:teuthology.orchestra.run.smithi102.stdout: "fuse_max_write": "0", 2024-09-13T14:26:07.957 INFO:teuthology.orchestra.run.smithi102.stdout: "fuse_multithreaded": "true", 2024-09-13T14:26:07.957 INFO:teuthology.orchestra.run.smithi102.stdout: "fuse_require_active_mds": "true", 2024-09-13T14:26:07.957 INFO:teuthology.orchestra.run.smithi102.stdout: "fuse_set_user_groups": "true", 2024-09-13T14:26:07.957 INFO:teuthology.orchestra.run.smithi102.stdout: "fuse_splice_move": "true", 2024-09-13T14:26:07.957 INFO:teuthology.orchestra.run.smithi102.stdout: "fuse_splice_read": "true", 2024-09-13T14:26:07.957 INFO:teuthology.orchestra.run.smithi102.stdout: "fuse_splice_write": "true", 2024-09-13T14:26:07.957 INFO:teuthology.orchestra.run.smithi102.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-13T14:26:07.957 INFO:teuthology.orchestra.run.smithi102.stdout: "fuse_use_invalidate_cb": "true", 2024-09-13T14:26:07.957 INFO:teuthology.orchestra.run.smithi102.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-13T14:26:07.958 INFO:teuthology.orchestra.run.smithi102.stdout: "gss_target_name": "ceph", 2024-09-13T14:26:07.958 INFO:teuthology.orchestra.run.smithi102.stdout: "heartbeat_file": "", 2024-09-13T14:26:07.958 INFO:teuthology.orchestra.run.smithi102.stdout: "heartbeat_inject_failure": "0", 2024-09-13T14:26:07.958 INFO:teuthology.orchestra.run.smithi102.stdout: "heartbeat_interval": "5", 2024-09-13T14:26:07.958 INFO:teuthology.orchestra.run.smithi102.stdout: "host": "smithi102", 2024-09-13T14:26:07.958 INFO:teuthology.orchestra.run.smithi102.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-13T14:26:07.958 INFO:teuthology.orchestra.run.smithi102.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-13T14:26:07.958 INFO:teuthology.orchestra.run.smithi102.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-13T14:26:07.958 INFO:teuthology.orchestra.run.smithi102.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-13T14:26:07.958 INFO:teuthology.orchestra.run.smithi102.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-13T14:26:07.958 INFO:teuthology.orchestra.run.smithi102.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-13T14:26:07.958 INFO:teuthology.orchestra.run.smithi102.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-13T14:26:07.958 INFO:teuthology.orchestra.run.smithi102.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-13T14:26:07.958 INFO:teuthology.orchestra.run.smithi102.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-13T14:26:07.959 INFO:teuthology.orchestra.run.smithi102.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-13T14:26:07.959 INFO:teuthology.orchestra.run.smithi102.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-13T14:26:07.959 INFO:teuthology.orchestra.run.smithi102.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-13T14:26:07.959 INFO:teuthology.orchestra.run.smithi102.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-13T14:26:07.959 INFO:teuthology.orchestra.run.smithi102.stdout: "inject_early_sigterm": "false", 2024-09-13T14:26:07.959 INFO:teuthology.orchestra.run.smithi102.stdout: "jaeger_agent_port": "6799", 2024-09-13T14:26:07.959 INFO:teuthology.orchestra.run.smithi102.stdout: "jaeger_tracing_enable": "false", 2024-09-13T14:26:07.959 INFO:teuthology.orchestra.run.smithi102.stdout: "journal_aio": "true", 2024-09-13T14:26:07.959 INFO:teuthology.orchestra.run.smithi102.stdout: "journal_align_min_size": "65536", 2024-09-13T14:26:07.959 INFO:teuthology.orchestra.run.smithi102.stdout: "journal_block_align": "true", 2024-09-13T14:26:07.959 INFO:teuthology.orchestra.run.smithi102.stdout: "journal_block_size": "4096", 2024-09-13T14:26:07.959 INFO:teuthology.orchestra.run.smithi102.stdout: "journal_dio": "true", 2024-09-13T14:26:07.959 INFO:teuthology.orchestra.run.smithi102.stdout: "journal_discard": "false", 2024-09-13T14:26:07.959 INFO:teuthology.orchestra.run.smithi102.stdout: "journal_force_aio": "false", 2024-09-13T14:26:07.960 INFO:teuthology.orchestra.run.smithi102.stdout: "journal_ignore_corruption": "false", 2024-09-13T14:26:07.960 INFO:teuthology.orchestra.run.smithi102.stdout: "journal_max_write_bytes": "10485760", 2024-09-13T14:26:07.960 INFO:teuthology.orchestra.run.smithi102.stdout: "journal_max_write_entries": "100", 2024-09-13T14:26:07.960 INFO:teuthology.orchestra.run.smithi102.stdout: "journal_replay_from": "0", 2024-09-13T14:26:07.960 INFO:teuthology.orchestra.run.smithi102.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-13T14:26:07.960 INFO:teuthology.orchestra.run.smithi102.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-13T14:26:07.960 INFO:teuthology.orchestra.run.smithi102.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-13T14:26:07.960 INFO:teuthology.orchestra.run.smithi102.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-13T14:26:07.960 INFO:teuthology.orchestra.run.smithi102.stdout: "journal_write_header_frequency": "0", 2024-09-13T14:26:07.960 INFO:teuthology.orchestra.run.smithi102.stdout: "journal_zero_on_create": "false", 2024-09-13T14:26:07.960 INFO:teuthology.orchestra.run.smithi102.stdout: "journaler_prefetch_periods": "10", 2024-09-13T14:26:07.960 INFO:teuthology.orchestra.run.smithi102.stdout: "journaler_prezero_periods": "5", 2024-09-13T14:26:07.960 INFO:teuthology.orchestra.run.smithi102.stdout: "journaler_write_head_interval": "15", 2024-09-13T14:26:07.960 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "", 2024-09-13T14:26:07.961 INFO:teuthology.orchestra.run.smithi102.stdout: "keyfile": "", 2024-09-13T14:26:07.961 INFO:teuthology.orchestra.run.smithi102.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-13T14:26:07.961 INFO:teuthology.orchestra.run.smithi102.stdout: "kstore_backend": "rocksdb", 2024-09-13T14:26:07.961 INFO:teuthology.orchestra.run.smithi102.stdout: "kstore_default_stripe_size": "65536", 2024-09-13T14:26:07.961 INFO:teuthology.orchestra.run.smithi102.stdout: "kstore_fsck_on_mount": "false", 2024-09-13T14:26:07.961 INFO:teuthology.orchestra.run.smithi102.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-13T14:26:07.961 INFO:teuthology.orchestra.run.smithi102.stdout: "kstore_max_bytes": "67108864", 2024-09-13T14:26:07.961 INFO:teuthology.orchestra.run.smithi102.stdout: "kstore_max_ops": "512", 2024-09-13T14:26:07.961 INFO:teuthology.orchestra.run.smithi102.stdout: "kstore_nid_prealloc": "1024", 2024-09-13T14:26:07.961 INFO:teuthology.orchestra.run.smithi102.stdout: "kstore_onode_map_size": "1024", 2024-09-13T14:26:07.961 INFO:teuthology.orchestra.run.smithi102.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-13T14:26:07.961 INFO:teuthology.orchestra.run.smithi102.stdout: "kstore_sync_submit_transaction": "false", 2024-09-13T14:26:07.961 INFO:teuthology.orchestra.run.smithi102.stdout: "kstore_sync_transaction": "false", 2024-09-13T14:26:07.962 INFO:teuthology.orchestra.run.smithi102.stdout: "librados_thread_count": "2", 2024-09-13T14:26:07.962 INFO:teuthology.orchestra.run.smithi102.stdout: "lockdep": "false", 2024-09-13T14:26:07.962 INFO:teuthology.orchestra.run.smithi102.stdout: "lockdep_force_backtrace": "false", 2024-09-13T14:26:07.962 INFO:teuthology.orchestra.run.smithi102.stdout: "log_coarse_timestamps": "true", 2024-09-13T14:26:07.962 INFO:teuthology.orchestra.run.smithi102.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-13T14:26:07.962 INFO:teuthology.orchestra.run.smithi102.stdout: "log_flush_on_exit": "false", 2024-09-13T14:26:07.962 INFO:teuthology.orchestra.run.smithi102.stdout: "log_graylog_host": "127.0.0.1", 2024-09-13T14:26:07.962 INFO:teuthology.orchestra.run.smithi102.stdout: "log_graylog_port": "12201", 2024-09-13T14:26:07.962 INFO:teuthology.orchestra.run.smithi102.stdout: "log_max_new": "1000", 2024-09-13T14:26:07.962 INFO:teuthology.orchestra.run.smithi102.stdout: "log_max_recent": "10000", 2024-09-13T14:26:07.962 INFO:teuthology.orchestra.run.smithi102.stdout: "log_stderr_prefix": "", 2024-09-13T14:26:07.962 INFO:teuthology.orchestra.run.smithi102.stdout: "log_stop_at_utilization": "0.970000", 2024-09-13T14:26:07.962 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": "true", 2024-09-13T14:26:07.963 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_graylog": "false", 2024-09-13T14:26:07.963 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_journald": "false", 2024-09-13T14:26:07.963 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_stderr": "false", 2024-09-13T14:26:07.963 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_syslog": "false", 2024-09-13T14:26:07.963 INFO:teuthology.orchestra.run.smithi102.stdout: "max_rotating_auth_attempts": "10", 2024-09-13T14:26:07.963 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-13T14:26:07.963 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_action_on_write_error": "1", 2024-09-13T14:26:07.963 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_alternate_name_max": "8192", 2024-09-13T14:26:07.963 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_asio_thread_count": "2", 2024-09-13T14:26:07.964 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_bal_export_pin": "true", 2024-09-13T14:26:07.964 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_bal_fragment_dirs": "true", 2024-09-13T14:26:07.964 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-13T14:26:07.964 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_bal_fragment_interval": "5", 2024-09-13T14:26:07.964 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-13T14:26:07.964 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-13T14:26:07.964 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_bal_interval": "10", 2024-09-13T14:26:07.965 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_bal_max": "-1", 2024-09-13T14:26:07.965 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_bal_max_until": "-1", 2024-09-13T14:26:07.965 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_bal_merge_size": "50", 2024-09-13T14:26:07.965 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_bal_midchunk": "0.300000", 2024-09-13T14:26:07.965 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-13T14:26:07.965 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_bal_min_start": "0.200000", 2024-09-13T14:26:07.965 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_bal_minchunk": "0.001000", 2024-09-13T14:26:07.965 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_bal_mode": "0", 2024-09-13T14:26:07.965 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_bal_need_max": "1.200000", 2024-09-13T14:26:07.965 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_bal_need_min": "0.800000", 2024-09-13T14:26:07.965 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_bal_overload_epochs": "2", 2024-09-13T14:26:07.965 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-13T14:26:07.965 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-13T14:26:07.966 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_bal_split_bits": "3", 2024-09-13T14:26:07.966 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-13T14:26:07.966 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_bal_split_size": "10000", 2024-09-13T14:26:07.966 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-13T14:26:07.966 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_bal_target_decay": "10.000000", 2024-09-13T14:26:07.966 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-13T14:26:07.966 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_beacon_grace": "15.000000", 2024-09-13T14:26:07.966 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_beacon_interval": "4.000000", 2024-09-13T14:26:07.966 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-13T14:26:07.966 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-13T14:26:07.966 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_cache_mid": "0.700000", 2024-09-13T14:26:07.966 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-09-13T14:26:07.966 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_cache_quiesce_delay": "0", 2024-09-13T14:26:07.966 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_cache_quiesce_sleep": "200", 2024-09-13T14:26:07.967 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_cache_quiesce_splitauth": "true", 2024-09-13T14:26:07.967 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_cache_quiesce_threshold": "524288", 2024-09-13T14:26:07.967 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_cache_release_free_interval": "10", 2024-09-13T14:26:07.967 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_cache_reservation": "0.050000", 2024-09-13T14:26:07.967 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-13T14:26:07.967 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_cache_trim_interval": "1", 2024-09-13T14:26:07.967 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_cache_trim_threshold": "262144", 2024-09-13T14:26:07.967 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-13T14:26:07.967 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-13T14:26:07.967 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-13T14:26:07.967 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_client_prealloc_inos": "1000", 2024-09-13T14:26:07.967 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-13T14:26:07.967 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_connect_bootstrapping": "false", 2024-09-13T14:26:07.968 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_damage_table_max_entries": "10000", 2024-09-13T14:26:07.968 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-13T14:26:07.968 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_debug_auth_pins": "false", 2024-09-13T14:26:07.968 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_debug_frag": "false", 2024-09-13T14:26:07.968 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_debug_scatterstat": "false", 2024-09-13T14:26:07.968 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_debug_subtrees": "false", 2024-09-13T14:26:07.968 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_decay_halflife": "5.000000", 2024-09-13T14:26:07.968 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_default_dir_hash": "2", 2024-09-13T14:26:07.968 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_defer_session_stale": "true", 2024-09-13T14:26:07.968 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_deny_all_reconnect": "false", 2024-09-13T14:26:07.968 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_dir_keys_per_op": "16384", 2024-09-13T14:26:07.968 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_dir_max_commit_size": "10", 2024-09-13T14:26:07.968 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_dir_max_entries": "0", 2024-09-13T14:26:07.969 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_dir_prefetch": "true", 2024-09-13T14:26:07.969 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-13T14:26:07.969 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-13T14:26:07.969 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_dump_cache_on_map": "false", 2024-09-13T14:26:07.969 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-13T14:26:07.969 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-13T14:26:07.969 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_early_reply": "true", 2024-09-13T14:26:07.969 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_enable_op_tracker": "true", 2024-09-13T14:26:07.969 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_enforce_unique_name": "true", 2024-09-13T14:26:07.969 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-13T14:26:07.969 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-13T14:26:07.969 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_export_ephemeral_random": "true", 2024-09-13T14:26:07.969 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-13T14:26:07.969 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-09-13T14:26:07.970 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-13T14:26:07.970 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-13T14:26:07.970 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-09-13T14:26:07.970 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-13T14:26:07.970 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-13T14:26:07.970 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-13T14:26:07.970 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_health_summarize_threshold": "10", 2024-09-13T14:26:07.970 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-13T14:26:07.970 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-13T14:26:07.970 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_inject_health_dummy": "false", 2024-09-13T14:26:07.970 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-13T14:26:07.970 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_inject_migrator_session_race": "false", 2024-09-13T14:26:07.970 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-13T14:26:07.970 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-13T14:26:07.971 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-13T14:26:07.971 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_join_fs": "", 2024-09-13T14:26:07.971 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_journal_format": "1", 2024-09-13T14:26:07.971 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-09-13T14:26:07.971 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_kill_create_at": "0", 2024-09-13T14:26:07.971 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_kill_dirfrag_at": "0", 2024-09-13T14:26:07.971 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_kill_export_at": "0", 2024-09-13T14:26:07.971 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_kill_import_at": "0", 2024-09-13T14:26:07.971 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_kill_journal_at": "0", 2024-09-13T14:26:07.971 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_kill_journal_expire_at": "0", 2024-09-13T14:26:07.971 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_kill_journal_replay_at": "0", 2024-09-13T14:26:07.971 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_kill_link_at": "0", 2024-09-13T14:26:07.971 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_kill_mdstable_at": "0", 2024-09-13T14:26:07.972 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_kill_openc_at": "0", 2024-09-13T14:26:07.972 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_kill_rename_at": "0", 2024-09-13T14:26:07.972 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_kill_shutdown_at": "0", 2024-09-13T14:26:07.972 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_log_event_large_threshold": "524288", 2024-09-13T14:26:07.972 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_log_events_per_segment": "1024", 2024-09-13T14:26:07.972 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_log_major_segment_event_ratio": "12", 2024-09-13T14:26:07.972 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_log_max_events": "-1", 2024-09-13T14:26:07.972 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_log_max_segments": "128", 2024-09-13T14:26:07.972 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_log_pause": "false", 2024-09-13T14:26:07.972 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_log_segment_size": "0", 2024-09-13T14:26:07.972 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-13T14:26:07.972 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_log_skip_unbounded_events": "false", 2024-09-13T14:26:07.972 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-09-13T14:26:07.973 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_log_trim_threshold": "128", 2024-09-13T14:26:07.973 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-09-13T14:26:07.973 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_log_warn_factor": "2.000000", 2024-09-13T14:26:07.973 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_max_caps_per_client": "1048576", 2024-09-13T14:26:07.973 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_max_completed_flushes": "100000", 2024-09-13T14:26:07.973 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_max_completed_requests": "100000", 2024-09-13T14:26:07.973 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_max_export_size": "20971520", 2024-09-13T14:26:07.973 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_max_file_recover": "32", 2024-09-13T14:26:07.973 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_max_purge_files": "64", 2024-09-13T14:26:07.973 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_max_purge_ops": "8192", 2024-09-13T14:26:07.973 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-13T14:26:07.973 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-13T14:26:07.973 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_max_snaps_per_dir": "100", 2024-09-13T14:26:07.973 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_metrics_update_interval": "2", 2024-09-13T14:26:07.974 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_min_caps_per_client": "100", 2024-09-13T14:26:07.974 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_min_caps_working_set": "10000", 2024-09-13T14:26:07.974 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-13T14:26:07.974 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_numa_node": "-1", 2024-09-13T14:26:07.974 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-13T14:26:07.974 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_op_complaint_time": "30.000000", 2024-09-13T14:26:07.974 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_op_history_duration": "600", 2024-09-13T14:26:07.974 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_op_history_size": "20", 2024-09-13T14:26:07.974 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_op_history_slow_op_size": "20", 2024-09-13T14:26:07.974 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-13T14:26:07.974 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_op_log_threshold": "5", 2024-09-13T14:26:07.974 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_ping_grace": "15", 2024-09-13T14:26:07.974 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_ping_interval": "5", 2024-09-13T14:26:07.974 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-13T14:26:07.974 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-13T14:26:07.975 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_recall_max_caps": "30000", 2024-09-13T14:26:07.975 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-13T14:26:07.975 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-13T14:26:07.975 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-13T14:26:07.975 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_recall_warning_threshold": "262144", 2024-09-13T14:26:07.975 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-13T14:26:07.975 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_replay_interval": "1.000000", 2024-09-13T14:26:07.975 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-13T14:26:07.975 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-13T14:26:07.975 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_root_ino_gid": "0", 2024-09-13T14:26:07.975 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_root_ino_uid": "0", 2024-09-13T14:26:07.975 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-13T14:26:07.975 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_server_dispatch_client_request_delay": "0", 2024-09-13T14:26:07.976 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2024-09-13T14:26:07.976 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-13T14:26:07.976 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-13T14:26:07.976 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-13T14:26:07.976 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-13T14:26:07.976 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-13T14:26:07.976 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-13T14:26:07.976 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-13T14:26:07.976 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-13T14:26:07.976 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-13T14:26:07.976 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_shutdown_check": "0", 2024-09-13T14:26:07.976 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_skip_ino": "0", 2024-09-13T14:26:07.976 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-13T14:26:07.976 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_snap_max_uid": "4294967294", 2024-09-13T14:26:07.977 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_snap_min_uid": "0", 2024-09-13T14:26:07.977 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_snap_rstat": "false", 2024-09-13T14:26:07.977 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_standby_replay_damaged": "false", 2024-09-13T14:26:07.977 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_symlink_recovery": "true", 2024-09-13T14:26:07.977 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-13T14:26:07.977 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_thrash_exports": "0", 2024-09-13T14:26:07.977 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_thrash_fragments": "0", 2024-09-13T14:26:07.977 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_tick_interval": "5.000000", 2024-09-13T14:26:07.977 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_valgrind_exit": "false", 2024-09-13T14:26:07.977 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_verify_backtrace": "1", 2024-09-13T14:26:07.977 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_verify_scatter": "false", 2024-09-13T14:26:07.977 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-13T14:26:07.977 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_wipe_sessions": "false", 2024-09-13T14:26:07.978 INFO:teuthology.orchestra.run.smithi102.stdout: "mempool_debug": "false", 2024-09-13T14:26:07.978 INFO:teuthology.orchestra.run.smithi102.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-13T14:26:07.978 INFO:teuthology.orchestra.run.smithi102.stdout: "memstore_device_bytes": "1073741824", 2024-09-13T14:26:07.978 INFO:teuthology.orchestra.run.smithi102.stdout: "memstore_page_set": "false", 2024-09-13T14:26:07.978 INFO:teuthology.orchestra.run.smithi102.stdout: "memstore_page_size": "65536", 2024-09-13T14:26:07.978 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr_client_bytes": "134217728", 2024-09-13T14:26:07.978 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr_client_messages": "512", 2024-09-13T14:26:07.978 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-13T14:26:07.978 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-13T14:26:07.978 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-13T14:26:07.978 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-13T14:26:07.978 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr_disabled_modules": "", 2024-09-13T14:26:07.978 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr_enable_op_tracker": "true", 2024-09-13T14:26:07.978 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-13T14:26:07.979 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr_max_pg_creating": "1024", 2024-09-13T14:26:07.979 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr_max_pg_num_change": "128", 2024-09-13T14:26:07.979 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr_mds_bytes": "134217728", 2024-09-13T14:26:07.979 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr_mds_messages": "128", 2024-09-13T14:26:07.979 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-13T14:26:07.979 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr_mon_bytes": "134217728", 2024-09-13T14:26:07.979 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr_mon_messages": "128", 2024-09-13T14:26:07.979 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr_num_op_tracker_shard": "32", 2024-09-13T14:26:07.979 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr_op_complaint_time": "30.000000", 2024-09-13T14:26:07.979 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr_op_history_duration": "600", 2024-09-13T14:26:07.979 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr_op_history_size": "20", 2024-09-13T14:26:07.979 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr_op_history_slow_op_size": "20", 2024-09-13T14:26:07.979 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-09-13T14:26:07.979 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr_op_log_threshold": "5", 2024-09-13T14:26:07.980 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr_osd_bytes": "536870912", 2024-09-13T14:26:07.980 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr_osd_messages": "8192", 2024-09-13T14:26:07.980 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr_pool": "true", 2024-09-13T14:26:07.980 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-13T14:26:07.980 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr_standby_modules": "true", 2024-09-13T14:26:07.980 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr_stats_period": "5", 2024-09-13T14:26:07.980 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr_stats_threshold": "5", 2024-09-13T14:26:07.980 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr_tick_period": "2", 2024-09-13T14:26:07.980 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-13T14:26:07.980 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-13T14:26:07.980 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_allow_pool_delete": "true", 2024-09-13T14:26:07.980 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_allow_pool_size_one": "false", 2024-09-13T14:26:07.980 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_auth_validate_all_caps": "true", 2024-09-13T14:26:07.980 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-13T14:26:07.981 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-13T14:26:07.981 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_client_bytes": "104857600", 2024-09-13T14:26:07.981 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_client_directed_command_retry": "2", 2024-09-13T14:26:07.981 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-13T14:26:07.981 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-13T14:26:07.981 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-13T14:26:07.981 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-13T14:26:07.981 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_client_hunt_parallel": "3", 2024-09-13T14:26:07.981 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_client_log_interval": "1.000000", 2024-09-13T14:26:07.981 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-13T14:26:07.981 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_client_ping_interval": "10.000000", 2024-09-13T14:26:07.981 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-13T14:26:07.981 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_client_target_rank": "-1", 2024-09-13T14:26:07.982 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-13T14:26:07.982 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-13T14:26:07.982 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-13T14:26:07.982 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_cluster_log_level": "debug", 2024-09-13T14:26:07.982 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_cluster_log_to_file": "true", 2024-09-13T14:26:07.982 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-13T14:26:07.982 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-13T14:26:07.982 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-13T14:26:07.982 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_cluster_log_to_journald": "true", 2024-09-13T14:26:07.982 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-13T14:26:07.982 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-13T14:26:07.982 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-13T14:26:07.982 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_compact_on_bootstrap": "false", 2024-09-13T14:26:07.982 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_compact_on_start": "false", 2024-09-13T14:26:07.983 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_compact_on_trim": "true", 2024-09-13T14:26:07.983 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-13T14:26:07.983 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-13T14:26:07.983 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-13T14:26:07.983 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_cpu_threads": "4", 2024-09-13T14:26:07.983 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_crush_min_required_version": "hammer", 2024-09-13T14:26:07.983 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_daemon_bytes": "419430400", 2024-09-13T14:26:07.983 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-13T14:26:07.983 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_data_avail_crit": "5", 2024-09-13T14:26:07.983 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_data_avail_warn": "5", 2024-09-13T14:26:07.983 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_data_size_warn": "16106127360", 2024-09-13T14:26:07.983 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-13T14:26:07.983 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-13T14:26:07.983 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_debug_dump_json": "false", 2024-09-13T14:26:07.984 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-13T14:26:07.984 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_debug_dump_transactions": "false", 2024-09-13T14:26:07.984 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_debug_extra_checks": "false", 2024-09-13T14:26:07.984 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-13T14:26:07.984 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-13T14:26:07.984 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_debug_no_require_reef": "false", 2024-09-13T14:26:07.984 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_debug_no_require_squid": "false", 2024-09-13T14:26:07.984 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-13T14:26:07.984 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-13T14:26:07.984 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-13T14:26:07.984 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_down_mkfs_grace": "60", 2024-09-13T14:26:07.984 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_down_uptime_grace": "60", 2024-09-13T14:26:07.984 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_election_default_strategy": "1", 2024-09-13T14:26:07.984 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_election_timeout": "5.000000", 2024-09-13T14:26:07.985 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-13T14:26:07.985 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_elector_ping_divisor": "2", 2024-09-13T14:26:07.985 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-13T14:26:07.985 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_enable_op_tracker": "true", 2024-09-13T14:26:07.985 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_fake_pool_delete": "false", 2024-09-13T14:26:07.985 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_force_quorum_join": "false", 2024-09-13T14:26:07.985 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_fsmap_prune_threshold": "300", 2024-09-13T14:26:07.985 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_globalid_prealloc": "10000", 2024-09-13T14:26:07.985 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_health_detail_to_clog": "true", 2024-09-13T14:26:07.985 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_health_log_update_period": "5", 2024-09-13T14:26:07.985 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_health_max_detail": "50", 2024-09-13T14:26:07.985 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_health_to_clog": "true", 2024-09-13T14:26:07.985 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_health_to_clog_interval": "600", 2024-09-13T14:26:07.985 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-13T14:26:07.986 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_host": "[v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0]", 2024-09-13T14:26:07.986 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_host_override": "", 2024-09-13T14:26:07.986 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_initial_members": "", 2024-09-13T14:26:07.986 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-13T14:26:07.986 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-13T14:26:07.986 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-13T14:26:07.986 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-13T14:26:07.986 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-13T14:26:07.986 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_lease": "5.000000", 2024-09-13T14:26:07.986 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-13T14:26:07.986 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-13T14:26:07.986 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_log_full_interval": "50", 2024-09-13T14:26:07.986 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_log_max": "10000", 2024-09-13T14:26:07.986 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_log_max_summary": "50", 2024-09-13T14:26:07.987 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-13T14:26:07.987 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_max_log_epochs": "500", 2024-09-13T14:26:07.987 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-13T14:26:07.987 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-13T14:26:07.987 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_max_nvmeof_epochs": "500", 2024-09-13T14:26:07.987 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_max_osd": "10000", 2024-09-13T14:26:07.987 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_max_pg_per_osd": "10000", 2024-09-13T14:26:07.987 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_max_pool_pg_num": "65536", 2024-09-13T14:26:07.987 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-13T14:26:07.987 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-13T14:26:07.987 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_mds_force_trim_to": "0", 2024-09-13T14:26:07.987 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_mds_skip_sanity": "true", 2024-09-13T14:26:07.987 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_memory_autotune": "true", 2024-09-13T14:26:07.987 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_memory_target": "2147483648", 2024-09-13T14:26:07.987 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_mgr_beacon_grace": "30", 2024-09-13T14:26:07.988 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-13T14:26:07.988 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_mgr_digest_period": "5", 2024-09-13T14:26:07.988 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_mgr_inactive_grace": "60", 2024-09-13T14:26:07.988 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-13T14:26:07.988 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-13T14:26:07.988 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_min_osdmap_epochs": "500", 2024-09-13T14:26:07.988 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_nvmeofgw_beacon_grace": "10", 2024-09-13T14:26:07.988 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_nvmeofgw_set_group_id_retry": "1000", 2024-09-13T14:26:07.988 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_op_complaint_time": "30", 2024-09-13T14:26:07.988 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_op_history_duration": "600", 2024-09-13T14:26:07.988 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_op_history_size": "20", 2024-09-13T14:26:07.988 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_op_history_slow_op_size": "20", 2024-09-13T14:26:07.988 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-13T14:26:07.988 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_op_log_threshold": "5", 2024-09-13T14:26:07.989 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-13T14:26:07.989 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-13T14:26:07.989 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-13T14:26:07.989 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_osd_auto_mark_in": "false", 2024-09-13T14:26:07.989 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-13T14:26:07.989 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-13T14:26:07.989 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-13T14:26:07.989 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_osd_cache_size": "500", 2024-09-13T14:26:07.989 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-13T14:26:07.989 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-13T14:26:07.989 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-13T14:26:07.989 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_osd_down_out_interval": "600", 2024-09-13T14:26:07.989 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-13T14:26:07.989 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-13T14:26:07.990 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_osd_force_trim_to": "0", 2024-09-13T14:26:07.990 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-13T14:26:07.990 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-13T14:26:07.990 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-13T14:26:07.990 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-13T14:26:07.990 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-13T14:26:07.990 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-13T14:26:07.990 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-13T14:26:07.990 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_osd_min_down_reporters": "2", 2024-09-13T14:26:07.990 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-13T14:26:07.990 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-13T14:26:07.990 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-13T14:26:07.990 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-13T14:26:07.991 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-13T14:26:07.991 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-13T14:26:07.991 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_osd_report_timeout": "900", 2024-09-13T14:26:07.991 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-13T14:26:07.991 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-13T14:26:07.991 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-13T14:26:07.991 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-13T14:26:07.991 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-13T14:26:07.991 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_osdmap_full_prune_interval": "10", 2024-09-13T14:26:07.991 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_osdmap_full_prune_min": "10000", 2024-09-13T14:26:07.991 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-09-13T14:26:07.991 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-13T14:26:07.991 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_pg_stuck_threshold": "60", 2024-09-13T14:26:07.991 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-13T14:26:07.992 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-13T14:26:07.992 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-13T14:26:07.992 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-13T14:26:07.992 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-13T14:26:07.992 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-13T14:26:07.992 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_probe_timeout": "2.000000", 2024-09-13T14:26:07.992 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_reweight_max_change": "0.050000", 2024-09-13T14:26:07.992 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_reweight_max_osds": "4", 2024-09-13T14:26:07.992 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-13T14:26:07.992 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-13T14:26:07.992 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-13T14:26:07.992 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-13T14:26:07.992 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-13T14:26:07.992 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_scrub_interval": "86400", 2024-09-13T14:26:07.993 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_scrub_max_keys": "100", 2024-09-13T14:26:07.993 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_scrub_timeout": "300", 2024-09-13T14:26:07.993 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_session_timeout": "300", 2024-09-13T14:26:07.993 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_smart_report_timeout": "5", 2024-09-13T14:26:07.993 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_stat_smooth_intervals": "6", 2024-09-13T14:26:07.993 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-13T14:26:07.993 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_stretch_pool_min_size": "2", 2024-09-13T14:26:07.993 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_stretch_pool_size": "4", 2024-09-13T14:26:07.993 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-13T14:26:07.993 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-13T14:26:07.993 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_sync_debug": "false", 2024-09-13T14:26:07.993 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-13T14:26:07.993 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-13T14:26:07.993 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_sync_provider_kill_at": "0", 2024-09-13T14:26:07.994 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_sync_requester_kill_at": "0", 2024-09-13T14:26:07.994 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_sync_timeout": "60.000000", 2024-09-13T14:26:07.994 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_target_pg_per_osd": "100", 2024-09-13T14:26:07.994 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_tick_interval": "5", 2024-09-13T14:26:07.994 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_timecheck_interval": "300.000000", 2024-09-13T14:26:07.994 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-13T14:26:07.994 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_use_min_delay_socket": "false", 2024-09-13T14:26:07.994 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_warn_older_version_delay": "604800", 2024-09-13T14:26:07.994 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-13T14:26:07.994 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-13T14:26:07.994 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-13T14:26:07.994 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_warn_on_filestore_osds": "true", 2024-09-13T14:26:07.994 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-13T14:26:07.994 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-13T14:26:07.995 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-13T14:26:07.995 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_warn_on_misplaced": "false", 2024-09-13T14:26:07.995 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-13T14:26:07.995 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_warn_on_older_version": "true", 2024-09-13T14:26:07.995 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-13T14:26:07.995 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_warn_on_pool_no_app": "true", 2024-09-13T14:26:07.995 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-09-13T14:26:07.995 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-13T14:26:07.995 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-13T14:26:07.995 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-13T14:26:07.995 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-13T14:26:07.995 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-13T14:26:07.995 INFO:teuthology.orchestra.run.smithi102.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-13T14:26:07.995 INFO:teuthology.orchestra.run.smithi102.stdout: "monmap": "", 2024-09-13T14:26:07.996 INFO:teuthology.orchestra.run.smithi102.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-13T14:26:07.996 INFO:teuthology.orchestra.run.smithi102.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-09-13T14:26:07.996 INFO:teuthology.orchestra.run.smithi102.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-09-13T14:26:07.996 INFO:teuthology.orchestra.run.smithi102.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-13T14:26:07.996 INFO:teuthology.orchestra.run.smithi102.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-09-13T14:26:07.996 INFO:teuthology.orchestra.run.smithi102.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-09-13T14:26:07.996 INFO:teuthology.orchestra.run.smithi102.stdout: "motr_tracing_enabled": "false", 2024-09-13T14:26:07.996 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_async_op_threads": "3", 2024-09-13T14:26:07.996 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-13T14:26:07.996 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_async_rdma_cm": "false", 2024-09-13T14:26:07.996 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_async_rdma_device_name": "", 2024-09-13T14:26:07.996 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_async_rdma_dscp": "96", 2024-09-13T14:26:07.996 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-13T14:26:07.996 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-13T14:26:07.997 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_async_rdma_local_gid": "", 2024-09-13T14:26:07.997 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-13T14:26:07.997 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_async_rdma_port_num": "1", 2024-09-13T14:26:07.997 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-13T14:26:07.997 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-13T14:26:07.997 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-13T14:26:07.997 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-13T14:26:07.997 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_async_rdma_sl": "3", 2024-09-13T14:26:07.997 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_async_rdma_support_srq": "true", 2024-09-13T14:26:07.997 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_async_rdma_type": "ib", 2024-09-13T14:26:07.997 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_async_reap_threshold": "5", 2024-09-13T14:26:07.997 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_bind_before_connect": "false", 2024-09-13T14:26:07.997 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_bind_ipv4": "true", 2024-09-13T14:26:07.997 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_bind_ipv6": "false", 2024-09-13T14:26:07.998 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_bind_msgr1": "true", 2024-09-13T14:26:07.998 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_bind_msgr2": "true", 2024-09-13T14:26:07.998 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_bind_port_max": "7568", 2024-09-13T14:26:07.998 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_bind_port_min": "6800", 2024-09-13T14:26:07.998 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-13T14:26:07.998 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_bind_retry_count": "3", 2024-09-13T14:26:07.998 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_bind_retry_delay": "5", 2024-09-13T14:26:07.998 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_blackhole_client": "false", 2024-09-13T14:26:07.998 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_blackhole_mds": "false", 2024-09-13T14:26:07.998 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_blackhole_mgr": "false", 2024-09-13T14:26:07.998 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_blackhole_mon": "false", 2024-09-13T14:26:07.998 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_blackhole_osd": "false", 2024-09-13T14:26:07.998 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_client_mode": "crc secure", 2024-09-13T14:26:07.999 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_client_throttle_retry_time_interval": "5000", 2024-09-13T14:26:07.999 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_cluster_mode": "crc secure", 2024-09-13T14:26:07.999 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_cluster_type": "", 2024-09-13T14:26:07.999 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_compress_secure": "false", 2024-09-13T14:26:07.999 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_connection_idle_timeout": "900", 2024-09-13T14:26:07.999 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_connection_ready_timeout": "10", 2024-09-13T14:26:07.999 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_crc_data": "true", 2024-09-13T14:26:07.999 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_crc_header": "true", 2024-09-13T14:26:07.999 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_die_on_bad_msg": "false", 2024-09-13T14:26:07.999 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_die_on_bug": "true", 2024-09-13T14:26:07.999 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_die_on_old_message": "true", 2024-09-13T14:26:07.999 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_die_on_skipped_message": "false", 2024-09-13T14:26:07.999 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-13T14:26:07.999 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-13T14:26:08.000 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_dpdk_coremask": "0xF", 2024-09-13T14:26:08.000 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-13T14:26:08.000 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-13T14:26:08.000 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_dpdk_enable_tso": "true", 2024-09-13T14:26:08.000 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-13T14:26:08.000 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-13T14:26:08.000 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_dpdk_hugepages": "", 2024-09-13T14:26:08.000 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-13T14:26:08.000 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-13T14:26:08.000 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_dpdk_lro": "true", 2024-09-13T14:26:08.000 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_dpdk_memory_channel": "4", 2024-09-13T14:26:08.000 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-13T14:26:08.000 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_dpdk_pmd": "", 2024-09-13T14:26:08.000 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_dpdk_port_id": "0", 2024-09-13T14:26:08.001 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-13T14:26:08.001 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-13T14:26:08.001 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_dump_on_send": "false", 2024-09-13T14:26:08.001 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_initial_backoff": "0.200000", 2024-09-13T14:26:08.001 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_inject_delay_max": "1.000000", 2024-09-13T14:26:08.001 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_inject_delay_probability": "0.000000", 2024-09-13T14:26:08.001 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_inject_delay_type": "", 2024-09-13T14:26:08.001 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_inject_internal_delays": "0.000000", 2024-09-13T14:26:08.001 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_inject_network_congestion": "0", 2024-09-13T14:26:08.001 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_inject_socket_failures": "0", 2024-09-13T14:26:08.001 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_learn_addr_from_peer": "true", 2024-09-13T14:26:08.001 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_max_accept_failures": "4", 2024-09-13T14:26:08.001 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_max_backoff": "15.000000", 2024-09-13T14:26:08.001 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_mon_client_mode": "secure crc", 2024-09-13T14:26:08.002 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-13T14:26:08.002 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_mon_service_mode": "secure crc", 2024-09-13T14:26:08.002 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_osd_compress_min_size": "1024", 2024-09-13T14:26:08.002 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_osd_compress_mode": "none", 2024-09-13T14:26:08.002 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_osd_compression_algorithm": "snappy", 2024-09-13T14:26:08.002 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-13T14:26:08.002 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_pq_min_cost": "65536", 2024-09-13T14:26:08.002 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_public_type": "", 2024-09-13T14:26:08.002 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_service_mode": "crc secure", 2024-09-13T14:26:08.002 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_tcp_listen_backlog": "512", 2024-09-13T14:26:08.002 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_tcp_nodelay": "true", 2024-09-13T14:26:08.002 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-13T14:26:08.002 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_tcp_rcvbuf": "0", 2024-09-13T14:26:08.002 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_time_events_min_wait_interval": "1000", 2024-09-13T14:26:08.003 INFO:teuthology.orchestra.run.smithi102.stdout: "ms_type": "async+posix", 2024-09-13T14:26:08.003 INFO:teuthology.orchestra.run.smithi102.stdout: "no_config_file": "false", 2024-09-13T14:26:08.003 INFO:teuthology.orchestra.run.smithi102.stdout: "nvmeof_mon_client_disconnect_panic": "100", 2024-09-13T14:26:08.003 INFO:teuthology.orchestra.run.smithi102.stdout: "nvmeof_mon_client_tick_period": "2", 2024-09-13T14:26:08.003 INFO:teuthology.orchestra.run.smithi102.stdout: "objecter_completion_locks_per_session": "32", 2024-09-13T14:26:08.003 INFO:teuthology.orchestra.run.smithi102.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-13T14:26:08.003 INFO:teuthology.orchestra.run.smithi102.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-13T14:26:08.003 INFO:teuthology.orchestra.run.smithi102.stdout: "objecter_inflight_ops": "1024", 2024-09-13T14:26:08.003 INFO:teuthology.orchestra.run.smithi102.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-13T14:26:08.003 INFO:teuthology.orchestra.run.smithi102.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-13T14:26:08.003 INFO:teuthology.orchestra.run.smithi102.stdout: "objecter_tick_interval": "5.000000", 2024-09-13T14:26:08.003 INFO:teuthology.orchestra.run.smithi102.stdout: "objecter_timeout": "10.000000", 2024-09-13T14:26:08.003 INFO:teuthology.orchestra.run.smithi102.stdout: "objectstore_blackhole": "false", 2024-09-13T14:26:08.003 INFO:teuthology.orchestra.run.smithi102.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-09-13T14:26:08.003 INFO:teuthology.orchestra.run.smithi102.stdout: "openssl_engine_opts": "", 2024-09-13T14:26:08.004 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_agent_delay_time": "5.000000", 2024-09-13T14:26:08.004 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_agent_hist_halflife": "1000", 2024-09-13T14:26:08.004 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_agent_max_low_ops": "2", 2024-09-13T14:26:08.004 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_agent_max_ops": "4", 2024-09-13T14:26:08.004 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-13T14:26:08.004 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-13T14:26:08.004 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_agent_slop": "0.020000", 2024-09-13T14:26:08.004 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-13T14:26:08.004 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-13T14:26:08.004 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_asio_thread_count": "2", 2024-09-13T14:26:08.004 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_async_recovery_min_cost": "100", 2024-09-13T14:26:08.004 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-13T14:26:08.004 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-13T14:26:08.004 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_backfill_scan_max": "512", 2024-09-13T14:26:08.005 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_backfill_scan_min": "64", 2024-09-13T14:26:08.005 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_backoff_on_degraded": "false", 2024-09-13T14:26:08.005 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_backoff_on_peering": "false", 2024-09-13T14:26:08.005 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_backoff_on_unfound": "true", 2024-09-13T14:26:08.005 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_beacon_report_interval": "300", 2024-09-13T14:26:08.005 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_bench_duration": "30", 2024-09-13T14:26:08.005 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-13T14:26:08.005 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_bench_max_block_size": "67108864", 2024-09-13T14:26:08.005 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-13T14:26:08.005 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_blkin_trace_all": "false", 2024-09-13T14:26:08.005 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_blocked_scrub_grace_period": "120", 2024-09-13T14:26:08.005 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-13T14:26:08.005 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-09-13T14:26:08.006 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-13T14:26:08.006 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_check_for_log_corruption": "false", 2024-09-13T14:26:08.006 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-13T14:26:08.006 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-13T14:26:08.006 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-09-13T14:26:08.006 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-13T14:26:08.006 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_class_update_on_start": "true", 2024-09-13T14:26:08.006 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_client_message_cap": "256", 2024-09-13T14:26:08.006 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_client_message_size_cap": "524288000", 2024-09-13T14:26:08.006 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_client_op_priority": "63", 2024-09-13T14:26:08.006 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_client_watch_timeout": "30", 2024-09-13T14:26:08.006 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_command_max_records": "256", 2024-09-13T14:26:08.006 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_compact_on_start": "false", 2024-09-13T14:26:08.006 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-13T14:26:08.006 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_crush_chooseleaf_type": "1", 2024-09-13T14:26:08.007 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-13T14:26:08.007 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_crush_update_on_start": "true", 2024-09-13T14:26:08.007 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_crush_update_weight_set": "true", 2024-09-13T14:26:08.007 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-13T14:26:08.007 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-13T14:26:08.007 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-13T14:26:08.007 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-13T14:26:08.007 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-13T14:26:08.007 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_debug_feed_pullee": "-1", 2024-09-13T14:26:08.007 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-13T14:26:08.007 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-13T14:26:08.007 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-13T14:26:08.007 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_debug_misdirected_ops": "false", 2024-09-13T14:26:08.007 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_debug_no_acting_change": "false", 2024-09-13T14:26:08.008 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_debug_no_purge_strays": "false", 2024-09-13T14:26:08.008 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_debug_op_order": "false", 2024-09-13T14:26:08.008 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-13T14:26:08.008 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-13T14:26:08.008 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-13T14:26:08.008 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-13T14:26:08.008 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_debug_shutdown": "false", 2024-09-13T14:26:08.008 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-13T14:26:08.008 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-13T14:26:08.008 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_debug_trim_objects": "false", 2024-09-13T14:26:08.008 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-13T14:26:08.008 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-13T14:26:08.008 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_debug_verify_snaps": "false", 2024-09-13T14:26:08.008 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-13T14:26:08.009 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-13T14:26:08.009 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_deep_scrub_keys": "1024", 2024-09-13T14:26:08.009 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-13T14:26:08.009 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-13T14:26:08.009 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-13T14:26:08.009 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_deep_scrub_stride": "524288", 2024-09-13T14:26:08.009 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-13T14:26:08.009 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_default_data_pool_replay_window": "45", 2024-09-13T14:26:08.009 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_default_notify_timeout": "30", 2024-09-13T14:26:08.009 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_delete_sleep": "0.000000", 2024-09-13T14:26:08.009 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-13T14:26:08.009 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-13T14:26:08.009 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-13T14:26:08.010 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_discard_disconnected_ops": "true", 2024-09-13T14:26:08.010 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_ec_partial_reads": "true", 2024-09-13T14:26:08.010 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_enable_op_tracker": "true", 2024-09-13T14:26:08.010 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-13T14:26:08.010 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_extblkdev_plugins": "vdo", 2024-09-13T14:26:08.010 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-13T14:26:08.010 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-13T14:26:08.010 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_fast_info": "true", 2024-09-13T14:26:08.010 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_fast_shutdown": "true", 2024-09-13T14:26:08.010 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-13T14:26:08.010 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-13T14:26:08.010 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-13T14:26:08.010 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-13T14:26:08.010 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-13T14:26:08.010 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_function_tracing": "false", 2024-09-13T14:26:08.011 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_heartbeat_grace": "20", 2024-09-13T14:26:08.011 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_heartbeat_interval": "6", 2024-09-13T14:26:08.011 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-13T14:26:08.011 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_heartbeat_min_peers": "10", 2024-09-13T14:26:08.011 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_heartbeat_min_size": "2000", 2024-09-13T14:26:08.011 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_heartbeat_stale": "600", 2024-09-13T14:26:08.011 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-13T14:26:08.011 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_hit_set_max_size": "100000", 2024-09-13T14:26:08.011 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_hit_set_min_size": "1000", 2024-09-13T14:26:08.011 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-13T14:26:08.011 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-13T14:26:08.011 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-13T14:26:08.011 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-13T14:26:08.012 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-13T14:26:08.012 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-13T14:26:08.012 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_journal_size": "5120", 2024-09-13T14:26:08.012 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_kill_backfill_at": "0", 2024-09-13T14:26:08.012 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-13T14:26:08.012 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_map_cache_size": "50", 2024-09-13T14:26:08.012 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_map_dedup": "true", 2024-09-13T14:26:08.012 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_map_message_max": "40", 2024-09-13T14:26:08.012 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-13T14:26:08.012 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_map_share_max_epochs": "40", 2024-09-13T14:26:08.012 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_max_attr_name_len": "100", 2024-09-13T14:26:08.012 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_max_attr_size": "0", 2024-09-13T14:26:08.012 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_max_backfills": "1", 2024-09-13T14:26:08.012 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_max_markdown_count": "5", 2024-09-13T14:26:08.013 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_max_markdown_period": "600", 2024-09-13T14:26:08.013 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_max_object_name_len": "2048", 2024-09-13T14:26:08.013 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_max_object_namespace_len": "256", 2024-09-13T14:26:08.013 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_max_object_size": "134217728", 2024-09-13T14:26:08.013 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-13T14:26:08.013 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-13T14:26:08.013 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_max_pg_blocked_by": "16", 2024-09-13T14:26:08.013 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_max_pg_log_entries": "10000", 2024-09-13T14:26:08.013 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-13T14:26:08.013 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_max_pgls": "1024", 2024-09-13T14:26:08.013 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_max_push_cost": "8388608", 2024-09-13T14:26:08.013 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_max_push_objects": "10", 2024-09-13T14:26:08.013 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_max_scrubs": "3", 2024-09-13T14:26:08.014 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-13T14:26:08.014 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_max_trimming_pgs": "2", 2024-09-13T14:26:08.014 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_max_write_op_reply_len": "64", 2024-09-13T14:26:08.014 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_max_write_size": "90", 2024-09-13T14:26:08.014 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-13T14:26:08.014 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-13T14:26:08.014 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-13T14:26:08.014 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-13T14:26:08.014 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-13T14:26:08.014 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-13T14:26:08.014 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-13T14:26:08.014 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-13T14:26:08.014 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_mclock_profile": "balanced", 2024-09-13T14:26:08.014 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-13T14:26:08.014 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-13T14:26:08.015 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-13T14:26:08.015 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-13T14:26:08.015 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-13T14:26:08.015 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-13T14:26:08.015 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-13T14:26:08.015 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-13T14:26:08.015 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-13T14:26:08.015 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-13T14:26:08.015 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-13T14:26:08.015 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_memory_base": "805306368", 2024-09-13T14:26:08.015 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_memory_cache_min": "134217728", 2024-09-13T14:26:08.015 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-13T14:26:08.015 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-13T14:26:08.016 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_memory_target": "4294967296", 2024-09-13T14:26:08.016 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_memory_target_autotune": "false", 2024-09-13T14:26:08.016 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-13T14:26:08.016 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_min_pg_log_entries": "250", 2024-09-13T14:26:08.016 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_min_recovery_priority": "0", 2024-09-13T14:26:08.016 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-13T14:26:08.016 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-13T14:26:08.016 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_mon_report_interval": "5", 2024-09-13T14:26:08.016 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-13T14:26:08.016 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-13T14:26:08.016 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_num_cache_shards": "32", 2024-09-13T14:26:08.016 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_num_op_tracker_shard": "32", 2024-09-13T14:26:08.016 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_numa_auto_affinity": "true", 2024-09-13T14:26:08.017 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_numa_node": "-1", 2024-09-13T14:26:08.017 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_numa_prefer_iface": "true", 2024-09-13T14:26:08.017 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-09-13T14:26:08.017 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_objecter_finishers": "1", 2024-09-13T14:26:08.017 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_objectstore": "bluestore", 2024-09-13T14:26:08.017 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_objectstore_fuse": "false", 2024-09-13T14:26:08.017 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_objectstore_ideal_list_max": "64", 2024-09-13T14:26:08.017 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_objectstore_tracing": "false", 2024-09-13T14:26:08.017 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_op_complaint_time": "30.000000", 2024-09-13T14:26:08.017 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_op_history_duration": "600", 2024-09-13T14:26:08.017 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_op_history_size": "20", 2024-09-13T14:26:08.017 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_op_history_slow_op_size": "20", 2024-09-13T14:26:08.017 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-13T14:26:08.017 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_op_log_threshold": "5", 2024-09-13T14:26:08.018 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_op_num_shards": "0", 2024-09-13T14:26:08.018 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_op_num_shards_hdd": "5", 2024-09-13T14:26:08.018 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_op_num_shards_ssd": "8", 2024-09-13T14:26:08.018 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-13T14:26:08.018 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-13T14:26:08.018 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-13T14:26:08.018 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-13T14:26:08.018 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_op_pq_min_cost": "65536", 2024-09-13T14:26:08.018 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-13T14:26:08.018 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_op_queue_cut_off": "high", 2024-09-13T14:26:08.018 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-13T14:26:08.018 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_op_thread_timeout": "15", 2024-09-13T14:26:08.018 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_open_classes_on_start": "true", 2024-09-13T14:26:08.018 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_os_flags": "0", 2024-09-13T14:26:08.019 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_peering_op_priority": "255", 2024-09-13T14:26:08.019 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pg_delete_cost": "1048576", 2024-09-13T14:26:08.019 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pg_delete_priority": "5", 2024-09-13T14:26:08.019 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-13T14:26:08.019 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-13T14:26:08.019 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-13T14:26:08.019 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pg_log_trim_max": "10000", 2024-09-13T14:26:08.019 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pg_log_trim_min": "100", 2024-09-13T14:26:08.019 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-13T14:26:08.019 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-13T14:26:08.019 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-09-13T14:26:08.019 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-09-13T14:26:08.019 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-13T14:26:08.019 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-13T14:26:08.020 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-13T14:26:08.020 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-13T14:26:08.020 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-13T14:26:08.020 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-13T14:26:08.020 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pool_default_crimson": "false", 2024-09-13T14:26:08.020 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-13T14:26:08.020 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-13T14:26:08.020 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-13T14:26:08.020 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-13T14:26:08.020 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-13T14:26:08.020 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-13T14:26:08.020 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-13T14:26:08.020 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-13T14:26:08.020 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pool_default_flags": "0", 2024-09-13T14:26:08.021 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-13T14:26:08.021 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pool_default_min_size": "0", 2024-09-13T14:26:08.021 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-13T14:26:08.021 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pool_default_pg_num": "32", 2024-09-13T14:26:08.021 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pool_default_pgp_num": "0", 2024-09-13T14:26:08.021 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-13T14:26:08.021 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pool_default_read_ratio": "70", 2024-09-13T14:26:08.021 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pool_default_size": "3", 2024-09-13T14:26:08.021 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pool_default_type": "replicated", 2024-09-13T14:26:08.021 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-13T14:26:08.021 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-13T14:26:08.021 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_push_per_object_cost": "1000", 2024-09-13T14:26:08.021 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-13T14:26:08.021 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_recover_clone_overlap": "true", 2024-09-13T14:26:08.022 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-13T14:26:08.022 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_recovery_cost": "20971520", 2024-09-13T14:26:08.022 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-13T14:26:08.022 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_recovery_max_active": "0", 2024-09-13T14:26:08.022 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-13T14:26:08.022 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-13T14:26:08.022 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-13T14:26:08.022 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-13T14:26:08.022 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_recovery_max_single_start": "1", 2024-09-13T14:26:08.022 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_recovery_op_priority": "3", 2024-09-13T14:26:08.022 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-13T14:26:08.022 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_recovery_priority": "5", 2024-09-13T14:26:08.022 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-13T14:26:08.022 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_recovery_sleep": "0.000000", 2024-09-13T14:26:08.023 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-13T14:26:08.023 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-13T14:26:08.023 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-13T14:26:08.023 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_repair_during_recovery": "false", 2024-09-13T14:26:08.023 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_requested_scrub_priority": "120", 2024-09-13T14:26:08.023 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-13T14:26:08.023 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-13T14:26:08.023 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_scrub_auto_repair": "false", 2024-09-13T14:26:08.023 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-13T14:26:08.023 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-13T14:26:08.023 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_scrub_begin_hour": "0", 2024-09-13T14:26:08.023 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_scrub_begin_week_day": "0", 2024-09-13T14:26:08.023 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_scrub_chunk_max": "25", 2024-09-13T14:26:08.023 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_scrub_chunk_min": "5", 2024-09-13T14:26:08.023 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_scrub_cost": "52428800", 2024-09-13T14:26:08.024 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-09-13T14:26:08.024 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_scrub_during_recovery": "false", 2024-09-13T14:26:08.024 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_scrub_end_hour": "0", 2024-09-13T14:26:08.024 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_scrub_end_week_day": "0", 2024-09-13T14:26:08.024 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_scrub_event_cost": "4096", 2024-09-13T14:26:08.024 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-13T14:26:08.024 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-13T14:26:08.024 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_scrub_invalid_stats": "true", 2024-09-13T14:26:08.024 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-13T14:26:08.024 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-13T14:26:08.024 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_scrub_max_preemptions": "5", 2024-09-13T14:26:08.024 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-13T14:26:08.024 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_scrub_priority": "5", 2024-09-13T14:26:08.024 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_scrub_sleep": "0.000000", 2024-09-13T14:26:08.025 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-09-13T14:26:08.025 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-09-13T14:26:08.025 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-13T14:26:08.025 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-09-13T14:26:08.025 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_skip_data_digest": "false", 2024-09-13T14:26:08.025 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_smart_report_timeout": "5", 2024-09-13T14:26:08.025 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_snap_trim_cost": "1048576", 2024-09-13T14:26:08.025 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_snap_trim_priority": "5", 2024-09-13T14:26:08.025 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-13T14:26:08.025 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-13T14:26:08.025 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-13T14:26:08.025 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-13T14:26:08.025 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-09-13T14:26:08.025 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_stats_update_period_scrubbing": "15", 2024-09-13T14:26:08.026 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-13T14:26:08.026 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_target_transaction_size": "30", 2024-09-13T14:26:08.026 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-13T14:26:08.026 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-13T14:26:08.026 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-13T14:26:08.026 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-13T14:26:08.026 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-13T14:26:08.026 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-13T14:26:08.026 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-13T14:26:08.026 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-13T14:26:08.026 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-13T14:26:08.026 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-13T14:26:08.026 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_tracing": "false", 2024-09-13T14:26:08.026 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_use_stale_snap": "false", 2024-09-13T14:26:08.026 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-13T14:26:08.027 INFO:teuthology.orchestra.run.smithi102.stdout: "osdc_blkin_trace_all": "false", 2024-09-13T14:26:08.027 INFO:teuthology.orchestra.run.smithi102.stdout: "paxos_kill_at": "0", 2024-09-13T14:26:08.027 INFO:teuthology.orchestra.run.smithi102.stdout: "paxos_max_join_drift": "10", 2024-09-13T14:26:08.027 INFO:teuthology.orchestra.run.smithi102.stdout: "paxos_min": "500", 2024-09-13T14:26:08.027 INFO:teuthology.orchestra.run.smithi102.stdout: "paxos_min_wait": "0.050000", 2024-09-13T14:26:08.027 INFO:teuthology.orchestra.run.smithi102.stdout: "paxos_propose_interval": "1.000000", 2024-09-13T14:26:08.027 INFO:teuthology.orchestra.run.smithi102.stdout: "paxos_service_trim_max": "500", 2024-09-13T14:26:08.027 INFO:teuthology.orchestra.run.smithi102.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-13T14:26:08.027 INFO:teuthology.orchestra.run.smithi102.stdout: "paxos_service_trim_min": "250", 2024-09-13T14:26:08.027 INFO:teuthology.orchestra.run.smithi102.stdout: "paxos_stash_full_interval": "25", 2024-09-13T14:26:08.027 INFO:teuthology.orchestra.run.smithi102.stdout: "paxos_trim_max": "500", 2024-09-13T14:26:08.027 INFO:teuthology.orchestra.run.smithi102.stdout: "paxos_trim_min": "250", 2024-09-13T14:26:08.027 INFO:teuthology.orchestra.run.smithi102.stdout: "perf": "true", 2024-09-13T14:26:08.027 INFO:teuthology.orchestra.run.smithi102.stdout: "pid_file": "", 2024-09-13T14:26:08.028 INFO:teuthology.orchestra.run.smithi102.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-13T14:26:08.028 INFO:teuthology.orchestra.run.smithi102.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-09-13T14:26:08.028 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addr": "-", 2024-09-13T14:26:08.028 INFO:teuthology.orchestra.run.smithi102.stdout: "public_addrv": "", 2024-09-13T14:26:08.028 INFO:teuthology.orchestra.run.smithi102.stdout: "public_bind_addr": "-", 2024-09-13T14:26:08.028 INFO:teuthology.orchestra.run.smithi102.stdout: "public_network": "172.21.0.0/20", 2024-09-13T14:26:08.028 INFO:teuthology.orchestra.run.smithi102.stdout: "public_network_interface": "", 2024-09-13T14:26:08.028 INFO:teuthology.orchestra.run.smithi102.stdout: "qat_compressor_busy_polling": "false", 2024-09-13T14:26:08.028 INFO:teuthology.orchestra.run.smithi102.stdout: "qat_compressor_enabled": "false", 2024-09-13T14:26:08.028 INFO:teuthology.orchestra.run.smithi102.stdout: "qat_compressor_session_max_number": "256", 2024-09-13T14:26:08.028 INFO:teuthology.orchestra.run.smithi102.stdout: "rados_mon_op_timeout": "0", 2024-09-13T14:26:08.028 INFO:teuthology.orchestra.run.smithi102.stdout: "rados_osd_op_timeout": "0", 2024-09-13T14:26:08.028 INFO:teuthology.orchestra.run.smithi102.stdout: "rados_tracing": "false", 2024-09-13T14:26:08.028 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_atime_update_interval": "60", 2024-09-13T14:26:08.029 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-13T14:26:08.029 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_balance_parent_reads": "false", 2024-09-13T14:26:08.029 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_balance_snap_reads": "false", 2024-09-13T14:26:08.029 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_blkin_trace_all": "false", 2024-09-13T14:26:08.029 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-13T14:26:08.029 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-13T14:26:08.029 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_cache": "true", 2024-09-13T14:26:08.029 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-13T14:26:08.029 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-13T14:26:08.029 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-13T14:26:08.029 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-13T14:26:08.029 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_cache_policy": "writearound", 2024-09-13T14:26:08.029 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_cache_size": "33554432", 2024-09-13T14:26:08.029 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-13T14:26:08.030 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-13T14:26:08.030 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_clone_copy_on_read": "false", 2024-09-13T14:26:08.030 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_compression_hint": "none", 2024-09-13T14:26:08.030 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_concurrent_management_ops": "10", 2024-09-13T14:26:08.030 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-13T14:26:08.030 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_default_clone_format": "auto", 2024-09-13T14:26:08.030 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_default_data_pool": "", 2024-09-13T14:26:08.030 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_default_features": "61", 2024-09-13T14:26:08.030 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_default_format": "2", 2024-09-13T14:26:08.030 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_default_map_options": "", 2024-09-13T14:26:08.030 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_default_order": "22", 2024-09-13T14:26:08.030 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_default_pool": "rbd", 2024-09-13T14:26:08.030 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-13T14:26:08.030 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_default_stripe_count": "0", 2024-09-13T14:26:08.031 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_default_stripe_unit": "0", 2024-09-13T14:26:08.031 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-13T14:26:08.031 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-13T14:26:08.031 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-13T14:26:08.031 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_enable_alloc_hint": "true", 2024-09-13T14:26:08.031 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-13T14:26:08.031 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_io_scheduler": "simple", 2024-09-13T14:26:08.031 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-13T14:26:08.031 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-13T14:26:08.031 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-13T14:26:08.031 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-13T14:26:08.031 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-13T14:26:08.031 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-13T14:26:08.031 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-13T14:26:08.032 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-13T14:26:08.032 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-13T14:26:08.032 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_journal_order": "24", 2024-09-13T14:26:08.032 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_journal_pool": "", 2024-09-13T14:26:08.032 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_journal_splay_width": "4", 2024-09-13T14:26:08.032 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_localize_parent_reads": "false", 2024-09-13T14:26:08.032 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_localize_snap_reads": "false", 2024-09-13T14:26:08.032 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-13T14:26:08.032 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-13T14:26:08.032 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-13T14:26:08.032 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-13T14:26:08.032 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-13T14:26:08.032 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-13T14:26:08.033 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-13T14:26:08.033 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-13T14:26:08.033 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-13T14:26:08.033 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-13T14:26:08.033 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-13T14:26:08.033 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-13T14:26:08.033 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-13T14:26:08.033 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-13T14:26:08.033 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-13T14:26:08.033 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-13T14:26:08.033 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-13T14:26:08.033 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-13T14:26:08.033 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-13T14:26:08.033 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-13T14:26:08.034 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-13T14:26:08.034 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-13T14:26:08.034 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-13T14:26:08.034 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-13T14:26:08.034 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-13T14:26:08.034 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-13T14:26:08.034 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-13T14:26:08.034 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-13T14:26:08.034 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-13T14:26:08.034 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-13T14:26:08.034 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-13T14:26:08.034 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_mtime_update_interval": "60", 2024-09-13T14:26:08.034 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_non_blocking_aio": "true", 2024-09-13T14:26:08.034 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_op_thread_timeout": "60", 2024-09-13T14:26:08.035 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_op_threads": "1", 2024-09-13T14:26:08.035 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_parent_cache_enabled": "false", 2024-09-13T14:26:08.035 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-13T14:26:08.035 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-13T14:26:08.035 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-13T14:26:08.035 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_plugins": "", 2024-09-13T14:26:08.035 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_qos_bps_burst": "0", 2024-09-13T14:26:08.035 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-13T14:26:08.035 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_qos_bps_limit": "0", 2024-09-13T14:26:08.035 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_qos_exclude_ops": "0", 2024-09-13T14:26:08.035 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_qos_iops_burst": "0", 2024-09-13T14:26:08.035 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-13T14:26:08.035 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_qos_iops_limit": "0", 2024-09-13T14:26:08.036 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-13T14:26:08.036 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-13T14:26:08.036 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-13T14:26:08.036 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-13T14:26:08.036 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-13T14:26:08.036 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-13T14:26:08.036 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-13T14:26:08.036 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-13T14:26:08.036 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-13T14:26:08.036 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-13T14:26:08.036 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-13T14:26:08.036 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-13T14:26:08.036 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-13T14:26:08.036 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-13T14:26:08.037 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_read_from_replica_policy": "default", 2024-09-13T14:26:08.037 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-13T14:26:08.037 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-13T14:26:08.037 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-13T14:26:08.037 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-13T14:26:08.037 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_skip_partial_discard": "true", 2024-09-13T14:26:08.037 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-13T14:26:08.037 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_tracing": "false", 2024-09-13T14:26:08.037 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_validate_names": "true", 2024-09-13T14:26:08.037 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_validate_pool": "true", 2024-09-13T14:26:08.037 INFO:teuthology.orchestra.run.smithi102.stdout: "restapi_base_url": "", 2024-09-13T14:26:08.037 INFO:teuthology.orchestra.run.smithi102.stdout: "restapi_log_level": "", 2024-09-13T14:26:08.037 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-09-13T14:26:08.038 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_account_default_quota_max_size": "-1", 2024-09-13T14:26:08.038 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_acl_grants_max_num": "100", 2024-09-13T14:26:08.038 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_admin_entry": "admin", 2024-09-13T14:26:08.038 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-09-13T14:26:08.038 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_asio_assert_yielding": "false", 2024-09-13T14:26:08.038 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_backend_store": "rados", 2024-09-13T14:26:08.038 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_barbican_url": "", 2024-09-13T14:26:08.038 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_beast_enable_async": "true", 2024-09-13T14:26:08.038 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_bucket_counters_cache": "false", 2024-09-13T14:26:08.038 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-09-13T14:26:08.038 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-13T14:26:08.038 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-13T14:26:08.038 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-13T14:26:08.039 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-09-13T14:26:08.039 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-13T14:26:08.039 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-13T14:26:08.039 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-09-13T14:26:08.039 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_cache_enabled": "true", 2024-09-13T14:26:08.039 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_cache_expiry_interval": "900", 2024-09-13T14:26:08.039 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_cache_lru_size": "25000", 2024-09-13T14:26:08.039 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_config_store": "rados", 2024-09-13T14:26:08.039 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_content_length_compat": "false", 2024-09-13T14:26:08.039 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_copy_obj_progress": "true", 2024-09-13T14:26:08.039 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-13T14:26:08.039 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_cors_rules_max_num": "100", 2024-09-13T14:26:08.039 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_cross_domain_policy": "", 2024-09-13T14:26:08.039 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-13T14:26:08.040 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_crypt_kmip_addr": "", 2024-09-13T14:26:08.040 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-13T14:26:08.040 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-13T14:26:08.040 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-13T14:26:08.040 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-13T14:26:08.040 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_crypt_kmip_password": "", 2024-09-13T14:26:08.040 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-13T14:26:08.040 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_crypt_kmip_username": "", 2024-09-13T14:26:08.040 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_crypt_require_ssl": "true", 2024-09-13T14:26:08.040 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-13T14:26:08.040 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-13T14:26:08.040 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-13T14:26:08.040 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-13T14:26:08.041 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-13T14:26:08.041 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-13T14:26:08.041 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-13T14:26:08.041 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-13T14:26:08.041 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-13T14:26:08.041 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-13T14:26:08.041 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-13T14:26:08.041 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-13T14:26:08.041 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-13T14:26:08.041 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-13T14:26:08.041 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-13T14:26:08.041 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_crypt_vault_addr": "", 2024-09-13T14:26:08.041 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_crypt_vault_auth": "token", 2024-09-13T14:26:08.041 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_crypt_vault_namespace": "", 2024-09-13T14:26:08.042 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_crypt_vault_prefix": "", 2024-09-13T14:26:08.042 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-13T14:26:08.042 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-13T14:26:08.042 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-13T14:26:08.042 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-13T14:26:08.042 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_crypt_vault_token_file": "", 2024-09-13T14:26:08.042 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-13T14:26:08.042 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_curl_buffersize": "524288", 2024-09-13T14:26:08.042 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-13T14:26:08.042 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_curl_low_speed_time": "300", 2024-09-13T14:26:08.042 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_curl_tcp_keepalive": "0", 2024-09-13T14:26:08.042 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-13T14:26:08.042 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-13T14:26:08.042 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-13T14:26:08.043 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-13T14:26:08.043 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-13T14:26:08.043 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-13T14:26:08.043 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-13T14:26:08.043 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-13T14:26:08.043 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-13T14:26:08.043 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_d4n_address": "127.0.0.1:6379", 2024-09-13T14:26:08.043 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_d4n_l1_datacache_address": "127.0.0.1:6379", 2024-09-13T14:26:08.043 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-09-13T14:26:08.043 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-09-13T14:26:08.043 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-09-13T14:26:08.043 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_d4n_l1_fadvise": "4", 2024-09-13T14:26:08.043 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-09-13T14:26:08.044 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-09-13T14:26:08.044 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-13T14:26:08.044 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_data_log_changes_size": "1000", 2024-09-13T14:26:08.044 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_data_log_num_shards": "128", 2024-09-13T14:26:08.044 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-13T14:26:08.044 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_data_log_window": "30", 2024-09-13T14:26:08.044 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_data_notify_interval_msec": "0", 2024-09-13T14:26:08.044 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_data_sync_poll_interval": "20", 2024-09-13T14:26:08.044 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_data_sync_spawn_window": "20", 2024-09-13T14:26:08.044 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-13T14:26:08.044 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-13T14:26:08.044 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-13T14:26:08.044 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-13T14:26:08.044 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-13T14:26:08.045 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-13T14:26:08.045 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-13T14:26:08.045 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-13T14:26:08.045 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-13T14:26:08.045 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-13T14:26:08.045 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_disable_s3select": "false", 2024-09-13T14:26:08.045 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-13T14:26:08.045 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-13T14:26:08.045 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-13T14:26:08.045 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-13T14:26:08.045 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-13T14:26:08.045 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-13T14:26:08.045 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-13T14:26:08.045 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-13T14:26:08.045 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-13T14:26:08.046 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-13T14:26:08.046 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-13T14:26:08.046 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-13T14:26:08.046 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_dns_name": "", 2024-09-13T14:26:08.046 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_dns_s3website_name": "", 2024-09-13T14:26:08.046 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_dynamic_resharding": "true", 2024-09-13T14:26:08.046 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_dynamic_resharding_may_reduce": "true", 2024-09-13T14:26:08.046 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_dynamic_resharding_reduction_wait": "120", 2024-09-13T14:26:08.047 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-13T14:26:08.047 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_enable_gc_threads": "true", 2024-09-13T14:26:08.047 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_enable_lc_threads": "true", 2024-09-13T14:26:08.047 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_enable_mdsearch": "true", 2024-09-13T14:26:08.047 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_enable_ops_log": "false", 2024-09-13T14:26:08.047 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_enable_quota_threads": "true", 2024-09-13T14:26:08.047 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_enable_static_website": "false", 2024-09-13T14:26:08.047 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_enable_usage_log": "false", 2024-09-13T14:26:08.047 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_enforce_swift_acls": "true", 2024-09-13T14:26:08.048 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_exit_timeout_secs": "120", 2024-09-13T14:26:08.048 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_expose_bucket": "false", 2024-09-13T14:26:08.048 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_extended_http_attrs": "", 2024-09-13T14:26:08.048 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_filter": "none", 2024-09-13T14:26:08.048 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-13T14:26:08.048 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_frontends": "beast port=7480", 2024-09-13T14:26:08.048 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-13T14:26:08.048 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_gc_max_deferred": "50", 2024-09-13T14:26:08.048 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-13T14:26:08.048 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_gc_max_objs": "32", 2024-09-13T14:26:08.048 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-13T14:26:08.048 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-13T14:26:08.048 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-13T14:26:08.048 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-13T14:26:08.049 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_gc_processor_period": "3600", 2024-09-13T14:26:08.049 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-13T14:26:08.049 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-13T14:26:08.049 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-13T14:26:08.049 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-13T14:26:08.049 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_init_timeout": "300", 2024-09-13T14:26:08.049 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-13T14:26:08.049 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-09-13T14:26:08.049 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_kafka_connection_idle": "300", 2024-09-13T14:26:08.049 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-13T14:26:08.049 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_kafka_sleep_timeout": "10", 2024-09-13T14:26:08.049 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-13T14:26:08.049 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-09-13T14:26:08.049 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-13T14:26:08.050 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_keystone_admin_domain": "", 2024-09-13T14:26:08.050 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_keystone_admin_password": "", 2024-09-13T14:26:08.050 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_keystone_admin_password_path": "", 2024-09-13T14:26:08.050 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_keystone_admin_project": "", 2024-09-13T14:26:08.050 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_keystone_admin_tenant": "", 2024-09-13T14:26:08.050 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_keystone_admin_token": "", 2024-09-13T14:26:08.050 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_keystone_admin_token_path": "", 2024-09-13T14:26:08.050 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_keystone_admin_user": "", 2024-09-13T14:26:08.050 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_keystone_barbican_domain": "", 2024-09-13T14:26:08.050 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_keystone_barbican_password": "", 2024-09-13T14:26:08.050 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_keystone_barbican_project": "", 2024-09-13T14:26:08.050 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-13T14:26:08.050 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_keystone_barbican_user": "", 2024-09-13T14:26:08.050 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-13T14:26:08.051 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-13T14:26:08.051 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-13T14:26:08.051 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-13T14:26:08.051 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-13T14:26:08.051 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_keystone_url": "", 2024-09-13T14:26:08.051 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-13T14:26:08.051 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_lc_debug_interval": "-1", 2024-09-13T14:26:08.051 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_lc_lock_max_time": "90", 2024-09-13T14:26:08.051 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_lc_max_objs": "32", 2024-09-13T14:26:08.051 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_lc_max_rules": "1000", 2024-09-13T14:26:08.051 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_lc_max_worker": "3", 2024-09-13T14:26:08.051 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-13T14:26:08.051 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_lc_thread_delay": "0", 2024-09-13T14:26:08.051 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-13T14:26:08.052 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_ldap_dnattr": "uid", 2024-09-13T14:26:08.052 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-13T14:26:08.052 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_ldap_searchfilter": "", 2024-09-13T14:26:08.052 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-13T14:26:08.052 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_ldap_uri": "", 2024-09-13T14:26:08.052 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_lfuda_sync_frequency": "60", 2024-09-13T14:26:08.052 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-13T14:26:08.052 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-13T14:26:08.052 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-13T14:26:08.052 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_log_http_headers": "", 2024-09-13T14:26:08.052 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-13T14:26:08.052 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-13T14:26:08.052 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_log_object_name_utc": "false", 2024-09-13T14:26:08.052 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-09-13T14:26:08.053 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-09-13T14:26:08.053 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_max_attr_name_len": "0", 2024-09-13T14:26:08.053 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_max_attr_size": "0", 2024-09-13T14:26:08.053 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-13T14:26:08.053 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_max_chunk_size": "4194304", 2024-09-13T14:26:08.053 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-13T14:26:08.053 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-09-13T14:26:08.053 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-13T14:26:08.053 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_max_listing_results": "1000", 2024-09-13T14:26:08.053 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_max_notify_retries": "10", 2024-09-13T14:26:08.053 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-13T14:26:08.053 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_max_put_param_size": "1048576", 2024-09-13T14:26:08.053 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_max_put_size": "5368709120", 2024-09-13T14:26:08.053 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_max_slo_entries": "1000", 2024-09-13T14:26:08.054 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_md_log_max_shards": "64", 2024-09-13T14:26:08.054 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-13T14:26:08.054 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_meta_sync_poll_interval": "20", 2024-09-13T14:26:08.054 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_meta_sync_spawn_window": "20", 2024-09-13T14:26:08.054 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-13T14:26:08.054 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_mp_lock_max_time": "600", 2024-09-13T14:26:08.054 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-13T14:26:08.054 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-13T14:26:08.054 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-13T14:26:08.054 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-13T14:26:08.054 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-13T14:26:08.054 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-09-13T14:26:08.054 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-13T14:26:08.054 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-13T14:26:08.055 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_nfs_max_gc": "300", 2024-09-13T14:26:08.055 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-13T14:26:08.055 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-13T14:26:08.055 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-13T14:26:08.055 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-13T14:26:08.055 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-13T14:26:08.055 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-13T14:26:08.055 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-13T14:26:08.055 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_num_async_rados_threads": "32", 2024-09-13T14:26:08.055 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_num_control_oids": "8", 2024-09-13T14:26:08.055 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_numa_node": "-1", 2024-09-13T14:26:08.055 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-13T14:26:08.055 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-13T14:26:08.055 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_objexp_chunk_size": "100", 2024-09-13T14:26:08.056 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_objexp_gc_interval": "600", 2024-09-13T14:26:08.056 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-13T14:26:08.056 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-13T14:26:08.056 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-13T14:26:08.056 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_op_thread_timeout": "600", 2024-09-13T14:26:08.056 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_op_tracing": "false", 2024-09-13T14:26:08.056 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_opa_token": "", 2024-09-13T14:26:08.056 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_opa_url": "", 2024-09-13T14:26:08.056 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_opa_verify_ssl": "true", 2024-09-13T14:26:08.056 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-13T14:26:08.056 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-09-13T14:26:08.056 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_ops_log_rados": "false", 2024-09-13T14:26:08.056 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_ops_log_socket_path": "", 2024-09-13T14:26:08.056 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-13T14:26:08.057 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-09-13T14:26:08.057 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-13T14:26:08.057 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_period_push_interval": "2.000000", 2024-09-13T14:26:08.057 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-13T14:26:08.057 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-13T14:26:08.057 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-09-13T14:26:08.057 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-09-13T14:26:08.057 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_posix_cache_lanes": "3", 2024-09-13T14:26:08.057 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-09-13T14:26:08.057 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_posix_cache_max_buckets": "100", 2024-09-13T14:26:08.057 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_posix_cache_partitions": "3", 2024-09-13T14:26:08.057 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-09-13T14:26:08.057 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_print_continue": "true", 2024-09-13T14:26:08.057 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-13T14:26:08.058 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-13T14:26:08.058 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-13T14:26:08.058 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-13T14:26:08.058 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-13T14:26:08.058 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_rados_tracing": "false", 2024-09-13T14:26:08.058 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_realm": "", 2024-09-13T14:26:08.058 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_realm_id": "", 2024-09-13T14:26:08.058 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-13T14:26:08.058 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_region": "", 2024-09-13T14:26:08.058 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-13T14:26:08.058 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-13T14:26:08.058 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-13T14:26:08.058 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-13T14:26:08.058 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_request_uri": "", 2024-09-13T14:26:08.058 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_reshard_batch_size": "64", 2024-09-13T14:26:08.059 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-13T14:26:08.059 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_reshard_debug_interval": "-1", 2024-09-13T14:26:08.059 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_reshard_max_aio": "128", 2024-09-13T14:26:08.059 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_reshard_num_logs": "16", 2024-09-13T14:26:08.059 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_reshard_thread_interval": "600", 2024-09-13T14:26:08.059 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_resolve_cname": "false", 2024-09-13T14:26:08.059 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-13T14:26:08.059 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_run_sync_thread": "true", 2024-09-13T14:26:08.059 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-13T14:26:08.059 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-13T14:26:08.059 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-13T14:26:08.059 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-13T14:26:08.059 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-13T14:26:08.059 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-13T14:26:08.060 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-13T14:26:08.060 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-13T14:26:08.060 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-13T14:26:08.060 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_scheduler_type": "throttler", 2024-09-13T14:26:08.060 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_script_uri": "", 2024-09-13T14:26:08.060 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_service_provider_name": "", 2024-09-13T14:26:08.060 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-13T14:26:08.060 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_sts_client_id": "", 2024-09-13T14:26:08.060 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_sts_client_secret": "", 2024-09-13T14:26:08.060 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_sts_entry": "sts", 2024-09-13T14:26:08.060 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_sts_key": "sts", 2024-09-13T14:26:08.060 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-13T14:26:08.060 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_sts_min_session_duration": "900", 2024-09-13T14:26:08.060 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_sts_token_introspection_url": "", 2024-09-13T14:26:08.061 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_swift_account_in_url": "false", 2024-09-13T14:26:08.061 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_swift_auth_entry": "auth", 2024-09-13T14:26:08.061 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_swift_auth_url": "", 2024-09-13T14:26:08.061 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_swift_custom_header": "", 2024-09-13T14:26:08.061 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-13T14:26:08.061 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_swift_need_stats": "true", 2024-09-13T14:26:08.061 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_swift_tenant_name": "", 2024-09-13T14:26:08.061 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_swift_token_expiration": "86400", 2024-09-13T14:26:08.061 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_swift_url": "", 2024-09-13T14:26:08.061 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_swift_url_prefix": "swift", 2024-09-13T14:26:08.061 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-13T14:26:08.061 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-09-13T14:26:08.061 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-13T14:26:08.061 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_sync_lease_period": "120", 2024-09-13T14:26:08.062 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-13T14:26:08.062 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-13T14:26:08.062 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-13T14:26:08.062 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-13T14:26:08.062 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-13T14:26:08.062 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-13T14:26:08.062 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-13T14:26:08.062 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-13T14:26:08.062 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-13T14:26:08.062 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_thread_pool_size": "512", 2024-09-13T14:26:08.062 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_topic_persistency_max_retries": "0", 2024-09-13T14:26:08.062 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-09-13T14:26:08.062 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-09-13T14:26:08.062 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_topic_require_publish_policy": "false", 2024-09-13T14:26:08.063 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_torrent_comment": "", 2024-09-13T14:26:08.063 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_torrent_createby": "", 2024-09-13T14:26:08.063 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_torrent_encoding": "", 2024-09-13T14:26:08.063 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_torrent_flag": "false", 2024-09-13T14:26:08.063 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_torrent_max_size": "5368709120", 2024-09-13T14:26:08.063 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_torrent_origin": "", 2024-09-13T14:26:08.063 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-13T14:26:08.063 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_torrent_tracker": "", 2024-09-13T14:26:08.063 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_trust_forwarded_https": "false", 2024-09-13T14:26:08.063 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-13T14:26:08.063 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-13T14:26:08.063 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_usage_max_shards": "32", 2024-09-13T14:26:08.063 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_usage_max_user_shards": "1", 2024-09-13T14:26:08.063 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_use_opa_authz": "false", 2024-09-13T14:26:08.064 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_user_counters_cache": "false", 2024-09-13T14:26:08.064 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_user_counters_cache_size": "10000", 2024-09-13T14:26:08.064 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-13T14:26:08.064 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-13T14:26:08.064 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_user_max_buckets": "1000", 2024-09-13T14:26:08.064 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_user_policies_max_num": "100", 2024-09-13T14:26:08.064 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-13T14:26:08.064 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-13T14:26:08.064 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-13T14:26:08.064 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-13T14:26:08.064 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_user_unique_email": "true", 2024-09-13T14:26:08.064 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_verify_ssl": "true", 2024-09-13T14:26:08.064 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-13T14:26:08.064 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_zone": "", 2024-09-13T14:26:08.065 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_zone_id": "", 2024-09-13T14:26:08.065 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-13T14:26:08.065 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_zonegroup": "", 2024-09-13T14:26:08.065 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_zonegroup_id": "", 2024-09-13T14:26:08.065 INFO:teuthology.orchestra.run.smithi102.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-13T14:26:08.065 INFO:teuthology.orchestra.run.smithi102.stdout: "rgwlc_auto_session_clear": "true", 2024-09-13T14:26:08.065 INFO:teuthology.orchestra.run.smithi102.stdout: "rgwlc_skip_bucket_step": "false", 2024-09-13T14:26:08.065 INFO:teuthology.orchestra.run.smithi102.stdout: "rocksdb_block_size": "4096", 2024-09-13T14:26:08.065 INFO:teuthology.orchestra.run.smithi102.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-13T14:26:08.065 INFO:teuthology.orchestra.run.smithi102.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-13T14:26:08.065 INFO:teuthology.orchestra.run.smithi102.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-13T14:26:08.065 INFO:teuthology.orchestra.run.smithi102.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-13T14:26:08.065 INFO:teuthology.orchestra.run.smithi102.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-13T14:26:08.065 INFO:teuthology.orchestra.run.smithi102.stdout: "rocksdb_cache_size": "536870912", 2024-09-13T14:26:08.065 INFO:teuthology.orchestra.run.smithi102.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-13T14:26:08.066 INFO:teuthology.orchestra.run.smithi102.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-13T14:26:08.066 INFO:teuthology.orchestra.run.smithi102.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-13T14:26:08.066 INFO:teuthology.orchestra.run.smithi102.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-13T14:26:08.066 INFO:teuthology.orchestra.run.smithi102.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-13T14:26:08.066 INFO:teuthology.orchestra.run.smithi102.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-13T14:26:08.066 INFO:teuthology.orchestra.run.smithi102.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-13T14:26:08.066 INFO:teuthology.orchestra.run.smithi102.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-13T14:26:08.066 INFO:teuthology.orchestra.run.smithi102.stdout: "rocksdb_index_type": "binary_search", 2024-09-13T14:26:08.066 INFO:teuthology.orchestra.run.smithi102.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-13T14:26:08.066 INFO:teuthology.orchestra.run.smithi102.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-13T14:26:08.066 INFO:teuthology.orchestra.run.smithi102.stdout: "rocksdb_partition_filters": "false", 2024-09-13T14:26:08.066 INFO:teuthology.orchestra.run.smithi102.stdout: "rocksdb_perf": "false", 2024-09-13T14:26:08.066 INFO:teuthology.orchestra.run.smithi102.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-13T14:26:08.066 INFO:teuthology.orchestra.run.smithi102.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-13T14:26:08.067 INFO:teuthology.orchestra.run.smithi102.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-13T14:26:08.067 INFO:teuthology.orchestra.run.smithi102.stdout: "run_dir": "/var/run/ceph", 2024-09-13T14:26:08.067 INFO:teuthology.orchestra.run.smithi102.stdout: "seastore_block_create": "true", 2024-09-13T14:26:08.067 INFO:teuthology.orchestra.run.smithi102.stdout: "seastore_cache_lru_size": "67108864", 2024-09-13T14:26:08.067 INFO:teuthology.orchestra.run.smithi102.stdout: "seastore_cbjournal_size": "5368709120", 2024-09-13T14:26:08.067 INFO:teuthology.orchestra.run.smithi102.stdout: "seastore_data_delta_based_overwrite": "0", 2024-09-13T14:26:08.067 INFO:teuthology.orchestra.run.smithi102.stdout: "seastore_default_max_object_size": "16777216", 2024-09-13T14:26:08.067 INFO:teuthology.orchestra.run.smithi102.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-13T14:26:08.067 INFO:teuthology.orchestra.run.smithi102.stdout: "seastore_device_size": "53687091200", 2024-09-13T14:26:08.067 INFO:teuthology.orchestra.run.smithi102.stdout: "seastore_disable_end_to_end_data_protection": "true", 2024-09-13T14:26:08.067 INFO:teuthology.orchestra.run.smithi102.stdout: "seastore_full_integrity_check": "false", 2024-09-13T14:26:08.067 INFO:teuthology.orchestra.run.smithi102.stdout: "seastore_journal_batch_capacity": "16", 2024-09-13T14:26:08.067 INFO:teuthology.orchestra.run.smithi102.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-13T14:26:08.067 INFO:teuthology.orchestra.run.smithi102.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-13T14:26:08.068 INFO:teuthology.orchestra.run.smithi102.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-13T14:26:08.068 INFO:teuthology.orchestra.run.smithi102.stdout: "seastore_main_device_type": "SSD", 2024-09-13T14:26:08.068 INFO:teuthology.orchestra.run.smithi102.stdout: "seastore_max_concurrent_transactions": "8", 2024-09-13T14:26:08.068 INFO:teuthology.orchestra.run.smithi102.stdout: "seastore_max_data_allocation_size": "32768", 2024-09-13T14:26:08.068 INFO:teuthology.orchestra.run.smithi102.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-09-13T14:26:08.068 INFO:teuthology.orchestra.run.smithi102.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-09-13T14:26:08.068 INFO:teuthology.orchestra.run.smithi102.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-09-13T14:26:08.068 INFO:teuthology.orchestra.run.smithi102.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-09-13T14:26:08.068 INFO:teuthology.orchestra.run.smithi102.stdout: "seastore_segment_size": "67108864", 2024-09-13T14:26:08.068 INFO:teuthology.orchestra.run.smithi102.stdout: "set_keepcaps": "false", 2024-09-13T14:26:08.068 INFO:teuthology.orchestra.run.smithi102.stdout: "setgroup": "ceph", 2024-09-13T14:26:08.068 INFO:teuthology.orchestra.run.smithi102.stdout: "setuser": "ceph", 2024-09-13T14:26:08.068 INFO:teuthology.orchestra.run.smithi102.stdout: "setuser_match_path": "", 2024-09-13T14:26:08.068 INFO:teuthology.orchestra.run.smithi102.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-13T14:26:08.069 INFO:teuthology.orchestra.run.smithi102.stdout: "thp": "false", 2024-09-13T14:26:08.069 INFO:teuthology.orchestra.run.smithi102.stdout: "threadpool_default_timeout": "60", 2024-09-13T14:26:08.069 INFO:teuthology.orchestra.run.smithi102.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-13T14:26:08.069 INFO:teuthology.orchestra.run.smithi102.stdout: "throttler_perf_counter": "true", 2024-09-13T14:26:08.069 INFO:teuthology.orchestra.run.smithi102.stdout: "tmp_dir": "/tmp", 2024-09-13T14:26:08.069 INFO:teuthology.orchestra.run.smithi102.stdout: "tmp_file_template": "/tmp/ceph-mon.a.XXXXXX", 2024-09-13T14:26:08.069 INFO:teuthology.orchestra.run.smithi102.stdout: "uadk_compressor_enabled": "false", 2024-09-13T14:26:08.069 INFO:teuthology.orchestra.run.smithi102.stdout: "uadk_wd_sync_ctx_num": "2" 2024-09-13T14:26:08.069 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:26:08.773 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-13T14:26:08.970 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:26:09.091 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:08 smithi102 ceph-mon[110183]: pgmap v284: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:26:09.092 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:08 smithi102 ceph-mon[115269]: pgmap v284: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:26:09.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:08 smithi195 ceph-mon[104262]: pgmap v284: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:26:10.611 INFO:teuthology.orchestra.run.smithi102.stdout:mon.a: {} 2024-09-13T14:26:10.612 INFO:teuthology.orchestra.run.smithi102.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T14:26:10.614 INFO:teuthology.orchestra.run.smithi102.stdout:mon.b: {} 2024-09-13T14:26:10.614 INFO:teuthology.orchestra.run.smithi102.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T14:26:10.622 INFO:teuthology.orchestra.run.smithi102.stdout:mon.c: {} 2024-09-13T14:26:10.622 INFO:teuthology.orchestra.run.smithi102.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T14:26:11.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:10 smithi102 ceph-mon[110183]: pgmap v285: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:26:11.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:26:11.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:10 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1274709025' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-13T14:26:11.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:10 smithi102 ceph-mon[115269]: pgmap v285: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:26:11.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:26:11.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:10 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1274709025' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-13T14:26:11.217 INFO:tasks.thrashosds.thrasher:Unable to test ceph-objectstore-tool, not available on all OSD nodes 2024-09-13T14:26:11.217 INFO:teuthology.task.sequential:In sequential, running task full_sequential... 2024-09-13T14:26:11.218 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-13T14:26:11.220 INFO:tasks.radosbench:Beginning radosbench... 2024-09-13T14:26:11.220 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-09-13T14:26:11.221 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_0 16 2024-09-13T14:26:11.221 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-13T14:26:11.221 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:26:11.221 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:26:11.221 INFO:tasks.thrashosds.thrasher:Removing osd 1, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:26:11.222 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd out 1 2024-09-13T14:26:11.222 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-13T14:26:11.222 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:26:11.222 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-13T14:26:11.223 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:26:11.223 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-13T14:26:11.223 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T14:26:11.223 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-13T14:26:11.224 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:26:11.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:10 smithi195 ceph-mon[104262]: pgmap v285: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:26:11.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:26:11.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:10 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1274709025' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-13T14:26:11.251 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:11.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:11.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:26:11.356 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:26:11.391 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:11.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:11.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:26:11.466 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:26:11.477 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:26:11.477 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:26:11.493 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:26:11.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:11.524 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:11.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:26:11.626 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:26:11.652 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:11.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:11.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:26:11.754 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:26:11.754 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:26:11.755 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:11.780 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:11.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:11.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:11.882 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:26:11.906 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:11.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:11.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:26:12.008 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:26:12.031 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:12.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:12.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:26:12.132 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:26:12.153 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:12.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:12.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:26:12.255 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:26:12.281 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:12.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:12.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:26:12.383 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:26:12.404 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:12.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:12.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:26:12.506 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:26:12.528 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:12.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:12.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:26:12.630 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:26:12.651 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:12.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:12.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:26:12.753 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:26:12.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:12.775 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:12.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:26:12.876 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:26:12.899 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:12.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:12.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:26:13.001 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:26:13.026 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:13.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:13.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:26:13.099 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:12 smithi102 ceph-mon[110183]: pgmap v286: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:26:13.099 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:12 smithi102 ceph-mon[115269]: pgmap v286: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:26:13.127 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:26:13.148 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:13.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:13.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:26:13.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:12 smithi195 ceph-mon[104262]: pgmap v286: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:26:13.250 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:26:13.276 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:13.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:13.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:26:13.378 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:13.403 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:13.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:13.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:13.505 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:13.531 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:13.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:13.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:13.633 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:26:13.654 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:13.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:13.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:26:13.756 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:26:13.782 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:13.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:13.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:26:13.884 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:26:13.906 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:13.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:13.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:26:14.010 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:26:14.031 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:14.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:14.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:26:14.133 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:26:14.155 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:14.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:14.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:26:14.257 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:26:14.278 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:14.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:14.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:26:14.380 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:26:14.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:14.402 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:14.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:26:14.503 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:26:14.529 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:14.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:14.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:26:14.634 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:14.660 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:14.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:14.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:14.763 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:26:14.789 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:14.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:14.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:26:14.798 INFO:teuthology.orchestra.run.smithi102.stderr:marked out osd.1. 2024-09-13T14:26:14.891 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:26:14.915 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:14.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:14.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:26:15.013 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:14 smithi102 ceph-mon[115269]: pgmap v287: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:26:15.013 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:14 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1603885129' entity='client.admin' cmd={"prefix": "osd out", "ids": ["1"]} : dispatch 2024-09-13T14:26:15.013 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:14 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["1"]} : dispatch 2024-09-13T14:26:15.013 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:14 smithi102 ceph-mon[115269]: Client client.admin marked osd.1 out, while it was still marked up 2024-09-13T14:26:15.013 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:14 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-a[110159]: 2024-09-13T14:26:14.755+0000 7fa3e7f42640 -1 mon.a@0(leader).osd e110 definitely_dead 0 2024-09-13T14:26:15.013 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:14 smithi102 ceph-mon[110183]: pgmap v287: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:26:15.013 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:14 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1603885129' entity='client.admin' cmd={"prefix": "osd out", "ids": ["1"]} : dispatch 2024-09-13T14:26:15.013 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:14 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["1"]} : dispatch 2024-09-13T14:26:15.013 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:14 smithi102 ceph-mon[110183]: Client client.admin marked osd.1 out, while it was still marked up 2024-09-13T14:26:15.017 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:26:15.041 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:15.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:15.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:26:15.143 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:26:15.166 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:15.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:15.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:26:15.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:14 smithi195 ceph-mon[104262]: pgmap v287: 73 pgs: 73 active+clean; 4.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:26:15.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:14 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1603885129' entity='client.admin' cmd={"prefix": "osd out", "ids": ["1"]} : dispatch 2024-09-13T14:26:15.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:14 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["1"]} : dispatch 2024-09-13T14:26:15.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:14 smithi195 ceph-mon[104262]: Client client.admin marked osd.1 out, while it was still marked up 2024-09-13T14:26:15.269 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:26:15.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:15.294 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:15.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:26:15.395 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:26:15.418 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:15.418 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:26:15.419 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:26:15.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:15.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:26:15.435 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:26:15.436 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:26:15.452 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:26:15.452 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:26:15.468 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:26:15.468 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-09-13T14:26:15.485 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:26:15.485 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:26:15.501 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:26:15.501 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:26:15.518 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:26:15.518 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:26:15.520 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:26:15.534 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:26:15.534 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:26:15.542 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:15.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:15.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:26:15.644 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:26:15.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:15.670 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:15.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:26:15.771 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:26:15.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:15.793 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:15.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:26:15.894 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:26:15.915 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:15.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:15.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:26:16.017 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:26:16.038 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:16.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:16.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:26:16.140 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:26:16.154 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:15 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-09-13T14:26:16.154 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:15 smithi102 ceph-mon[110183]: osdmap e111: 8 total, 8 up, 7 in 2024-09-13T14:26:16.154 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:15 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/585592392' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:26:16.154 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:15 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-09-13T14:26:16.154 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:15 smithi102 ceph-mon[115269]: osdmap e111: 8 total, 8 up, 7 in 2024-09-13T14:26:16.154 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:15 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/585592392' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:26:16.162 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:16.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:16.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:26:16.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:15 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-09-13T14:26:16.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:15 smithi195 ceph-mon[104262]: osdmap e111: 8 total, 8 up, 7 in 2024-09-13T14:26:16.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:15 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/585592392' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:26:16.264 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:26:16.288 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:16.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:16.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:26:16.390 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:16.414 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:16.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:16.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:16.516 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:26:16.537 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:16.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:16.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:26:16.639 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:26:16.664 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:16.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:16.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:26:16.766 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:16.772 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:26:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:26:16] "GET /metrics HTTP/1.1" 200 34016 "" "Prometheus/2.51.0" 2024-09-13T14:26:16.791 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:16.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:16.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:16.893 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:26:16.914 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:16.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:16.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:26:17.016 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:26:17.037 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:17.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:17.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:26:17.139 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:26:17.160 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:17.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:17.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:26:17.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:16 smithi102 ceph-mon[115269]: pgmap v289: 73 pgs: 73 active+clean; 4.6 MiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-09-13T14:26:17.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:16 smithi102 ceph-mon[115269]: osdmap e112: 8 total, 8 up, 7 in 2024-09-13T14:26:17.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:16 smithi102 ceph-mon[110183]: pgmap v289: 73 pgs: 73 active+clean; 4.6 MiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-09-13T14:26:17.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:16 smithi102 ceph-mon[110183]: osdmap e112: 8 total, 8 up, 7 in 2024-09-13T14:26:17.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:16 smithi195 ceph-mon[104262]: pgmap v289: 73 pgs: 73 active+clean; 4.6 MiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-09-13T14:26:17.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:16 smithi195 ceph-mon[104262]: osdmap e112: 8 total, 8 up, 7 in 2024-09-13T14:26:17.262 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:26:17.283 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:17.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:17.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:26:17.385 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:17.412 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:17.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:17.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:17.513 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:26:17.538 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:17.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:17.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:26:17.639 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:26:17.661 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:17.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:17.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:26:17.763 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:26:17.784 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:17.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:17.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:26:17.886 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:26:17.912 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:17.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:17.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:26:18.014 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:26:18.035 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:18.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:18.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:26:18.098 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:17 smithi102 ceph-mon[110183]: osdmap e113: 8 total, 8 up, 7 in 2024-09-13T14:26:18.099 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:17 smithi102 ceph-mon[115269]: osdmap e113: 8 total, 8 up, 7 in 2024-09-13T14:26:18.137 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:26:18.163 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:18.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:18.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:26:18.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:17 smithi195 ceph-mon[104262]: osdmap e113: 8 total, 8 up, 7 in 2024-09-13T14:26:18.265 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:26:18.287 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:18.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:18.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:26:18.388 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:26:18.414 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:18.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:18.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:26:18.516 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:26:18.541 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:18.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:18.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:26:18.642 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:26:18.667 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:18.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:18.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:26:18.769 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:26:18.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:18.793 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:18.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:26:18.830 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:18 smithi102 ceph-mon[115269]: pgmap v292: 73 pgs: 3 remapped+peering, 70 active+clean; 4.6 MiB data, 216 MiB used, 626 GiB / 626 GiB avail; 0 B/s, 1 objects/s recovering 2024-09-13T14:26:18.831 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:18 smithi102 ceph-mon[115269]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-09-13T14:26:18.831 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:18 smithi102 ceph-mon[115269]: osdmap e114: 8 total, 8 up, 7 in 2024-09-13T14:26:18.894 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:18.920 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:18.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:18.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:19.027 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:26:19.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:19.054 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:19.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:26:19.124 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:18 smithi102 ceph-mon[110183]: pgmap v292: 73 pgs: 3 remapped+peering, 70 active+clean; 4.6 MiB data, 216 MiB used, 626 GiB / 626 GiB avail; 0 B/s, 1 objects/s recovering 2024-09-13T14:26:19.124 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:18 smithi102 ceph-mon[110183]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-09-13T14:26:19.124 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:18 smithi102 ceph-mon[110183]: osdmap e114: 8 total, 8 up, 7 in 2024-09-13T14:26:19.155 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:26:19.180 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:19.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:19.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:26:19.232 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:26:19.232 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:26:19.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:18 smithi195 ceph-mon[104262]: pgmap v292: 73 pgs: 3 remapped+peering, 70 active+clean; 4.6 MiB data, 216 MiB used, 626 GiB / 626 GiB avail; 0 B/s, 1 objects/s recovering 2024-09-13T14:26:19.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:18 smithi195 ceph-mon[104262]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-09-13T14:26:19.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:18 smithi195 ceph-mon[104262]: osdmap e114: 8 total, 8 up, 7 in 2024-09-13T14:26:19.283 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:26:19.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:19.305 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:19.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:26:19.406 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:26:19.427 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:19.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:19.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:26:19.529 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:26:19.550 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:19.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:19.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:26:19.652 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:26:19.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:19.674 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:19.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:26:19.775 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:26:19.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:19.797 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:19.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:26:19.852 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T14:26:19.899 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:26:19.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:19.922 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:19.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:26:20.023 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:26:20.045 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:20.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:20.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:26:20.106 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:19 smithi102 ceph-mon[115269]: osdmap e115: 8 total, 8 up, 7 in 2024-09-13T14:26:20.106 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:19 smithi102 ceph-mon[110183]: osdmap e115: 8 total, 8 up, 7 in 2024-09-13T14:26:20.119 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:26:20.147 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:26:20.174 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:20.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:20.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:26:20.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:19 smithi195 ceph-mon[104262]: osdmap e115: 8 total, 8 up, 7 in 2024-09-13T14:26:20.276 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:26:20.301 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:20.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:20.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:26:20.403 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:26:20.424 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:20.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:20.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:26:20.526 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:26:20.546 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:20.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:20.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:26:20.648 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:26:20.674 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:20.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:20.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:26:20.776 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:26:20.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:20.799 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:20.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:26:20.840 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'unique_pool_0' created 2024-09-13T14:26:20.900 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:26:20.922 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:20.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:20.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:26:21.024 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:26:21.045 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:21.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:21.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:26:21.147 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:26:21.163 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:20 smithi102 ceph-mon[110183]: pgmap v295: 73 pgs: 3 remapped+peering, 70 active+clean; 4.6 MiB data, 216 MiB used, 626 GiB / 626 GiB avail; 0 B/s, 2 objects/s recovering 2024-09-13T14:26:21.163 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:20 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3311058670' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-13T14:26:21.164 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:20 smithi102 ceph-mon[115269]: pgmap v295: 73 pgs: 3 remapped+peering, 70 active+clean; 4.6 MiB data, 216 MiB used, 626 GiB / 626 GiB avail; 0 B/s, 2 objects/s recovering 2024-09-13T14:26:21.164 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:20 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3311058670' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-13T14:26:21.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:21.173 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:21.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:26:21.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:20 smithi195 ceph-mon[104262]: pgmap v295: 73 pgs: 3 remapped+peering, 70 active+clean; 4.6 MiB data, 216 MiB used, 626 GiB / 626 GiB avail; 0 B/s, 2 objects/s recovering 2024-09-13T14:26:21.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:20 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3311058670' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-13T14:26:21.275 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:26:21.297 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:21.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:21.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:26:21.398 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:26:21.425 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:21.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:21.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:26:21.502 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-13T14:26:21.526 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:26:21.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:21.553 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:21.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:26:21.653 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:26:21.679 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:21.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:21.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:26:21.781 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:26:21.794 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:26:21.812 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:21.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:21.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:26:21.818 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:26:21.853 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:21 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3311058670' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-13T14:26:21.853 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:21 smithi102 ceph-mon[115269]: osdmap e116: 8 total, 8 up, 7 in 2024-09-13T14:26:21.915 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:21.940 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:21.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:21.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:22.042 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:22.065 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:22.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:22.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:22.167 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:22.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:21 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3311058670' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-13T14:26:22.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:21 smithi102 ceph-mon[110183]: osdmap e116: 8 total, 8 up, 7 in 2024-09-13T14:26:22.192 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:22.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:22.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:22.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:21 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3311058670' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-13T14:26:22.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:21 smithi195 ceph-mon[104262]: osdmap e116: 8 total, 8 up, 7 in 2024-09-13T14:26:22.281 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:26:22.281 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:26:22.294 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:22.318 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:22.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:22.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:22.420 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:26:22.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:22.444 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:22.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:26:22.545 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:26:22.568 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:22.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:22.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:26:22.670 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:26:22.692 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:22.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:22.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:26:22.694 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:26:22.794 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:26:22.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:22.816 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:22.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:26:22.858 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is set 2024-09-13T14:26:22.917 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:26:22.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:22.940 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:22.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:26:23.041 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:26:23.067 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:23.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:23.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:26:23.092 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:22 smithi102 ceph-mon[110183]: pgmap v297: 89 pgs: 10 creating+peering, 6 unknown, 73 active+clean; 4.6 MiB data, 220 MiB used, 626 GiB / 626 GiB avail; 924 KiB/s, 7 objects/s recovering 2024-09-13T14:26:23.092 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:22 smithi102 ceph-mon[110183]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:26:23.092 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:22 smithi102 ceph-mon[110183]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-09-13T14:26:23.092 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:22 smithi102 ceph-mon[110183]: osdmap e117: 8 total, 8 up, 7 in 2024-09-13T14:26:23.092 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:22 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3983877394' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:26:23.092 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:22 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:26:23.093 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:22 smithi102 ceph-mon[115269]: pgmap v297: 89 pgs: 10 creating+peering, 6 unknown, 73 active+clean; 4.6 MiB data, 220 MiB used, 626 GiB / 626 GiB avail; 924 KiB/s, 7 objects/s recovering 2024-09-13T14:26:23.093 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:22 smithi102 ceph-mon[115269]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:26:23.093 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:22 smithi102 ceph-mon[115269]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-09-13T14:26:23.093 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:22 smithi102 ceph-mon[115269]: osdmap e117: 8 total, 8 up, 7 in 2024-09-13T14:26:23.093 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:22 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3983877394' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:26:23.093 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:22 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:26:23.169 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:26:23.190 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:23.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:23.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:26:23.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:22 smithi195 ceph-mon[104262]: pgmap v297: 89 pgs: 10 creating+peering, 6 unknown, 73 active+clean; 4.6 MiB data, 220 MiB used, 626 GiB / 626 GiB avail; 924 KiB/s, 7 objects/s recovering 2024-09-13T14:26:23.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:22 smithi195 ceph-mon[104262]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:26:23.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:22 smithi195 ceph-mon[104262]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-09-13T14:26:23.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:22 smithi195 ceph-mon[104262]: osdmap e117: 8 total, 8 up, 7 in 2024-09-13T14:26:23.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:22 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3983877394' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:26:23.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:22 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:26:23.292 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:23.316 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:23.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:23.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:23.418 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:26:23.443 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:23.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:23.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:26:23.545 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:26:23.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:23.567 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:23.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:26:23.668 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:23.693 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:23.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:23.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:23.794 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:26:23.820 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:23.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:23.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:26:23.922 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:26:23.945 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:23.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:23.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:26:23.995 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:23 smithi102 ceph-mon[110183]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:26:24.044 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:23 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:26:24.044 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:23 smithi102 ceph-mon[110183]: osdmap e118: 8 total, 8 up, 7 in 2024-09-13T14:26:24.044 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:23 smithi102 ceph-mon[115269]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:26:24.044 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:23 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:26:24.044 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:23 smithi102 ceph-mon[115269]: osdmap e118: 8 total, 8 up, 7 in 2024-09-13T14:26:24.047 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:26:24.070 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:24.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:24.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:26:24.171 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:26:24.195 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:24.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:24.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:26:24.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:23 smithi195 ceph-mon[104262]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:26:24.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:23 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:26:24.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:23 smithi195 ceph-mon[104262]: osdmap e118: 8 total, 8 up, 7 in 2024-09-13T14:26:24.297 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:26:24.318 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:24.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:24.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:26:24.420 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:26:24.441 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:24.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:24.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:26:24.543 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:26:24.564 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:24.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:24.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:26:24.666 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:26:24.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:24.689 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:24.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:26:24.790 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:26:24.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:24.813 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:24.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:26:24.914 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:26:24.940 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:24.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:24.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:26:25.043 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:26:25.068 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:25.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:25.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:26:25.148 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:24 smithi195 ceph-mon[104262]: pgmap v300: 89 pgs: 15 creating+peering, 1 unknown, 73 active+clean; 4.6 MiB data, 221 MiB used, 626 GiB / 626 GiB avail; 927 KiB/s, 7 objects/s recovering 2024-09-13T14:26:25.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:24 smithi102 ceph-mon[115269]: pgmap v300: 89 pgs: 15 creating+peering, 1 unknown, 73 active+clean; 4.6 MiB data, 221 MiB used, 626 GiB / 626 GiB avail; 927 KiB/s, 7 objects/s recovering 2024-09-13T14:26:25.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:24 smithi102 ceph-mon[110183]: pgmap v300: 89 pgs: 15 creating+peering, 1 unknown, 73 active+clean; 4.6 MiB data, 221 MiB used, 626 GiB / 626 GiB avail; 927 KiB/s, 7 objects/s recovering 2024-09-13T14:26:25.170 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:26:25.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:25.196 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:25.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:26:25.298 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:26:25.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:25.320 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:25.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:26:25.421 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:26:25.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:25.445 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:25.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:26:25.533 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:26:25.533 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:26:25.547 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:26:25.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:25.569 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:25.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:26:25.673 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:26:25.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:25.694 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:25.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:26:25.709 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:26:25.797 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:26:25.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:25.820 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:25.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:26:25.921 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:26:25.943 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:25.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:25.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:26:26.046 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:26.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:26.068 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:26.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:26.125 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T14:26:26.169 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:26:26.191 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:26.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:26.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:26:26.293 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:26:26.314 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:26.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:26.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:26:26.416 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:26:26.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:26.439 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:26.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:26:26.540 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:26:26.562 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:26.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:26.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:26:26.664 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:26:26.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:26 smithi102 ceph-mon[110183]: pgmap v301: 89 pgs: 15 creating+peering, 74 active+clean; 4.6 MiB data, 221 MiB used, 626 GiB / 626 GiB avail; 683 KiB/s, 5 objects/s recovering 2024-09-13T14:26:26.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:26:26.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:26:26.669 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:26:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:26:26] "GET /metrics HTTP/1.1" 200 36455 "" "Prometheus/2.51.0" 2024-09-13T14:26:26.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:26 smithi102 ceph-mon[115269]: pgmap v301: 89 pgs: 15 creating+peering, 74 active+clean; 4.6 MiB data, 221 MiB used, 626 GiB / 626 GiB avail; 683 KiB/s, 5 objects/s recovering 2024-09-13T14:26:26.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:26:26.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:26:26.686 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:26.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:26.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:26:26.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:26 smithi195 ceph-mon[104262]: pgmap v301: 89 pgs: 15 creating+peering, 74 active+clean; 4.6 MiB data, 221 MiB used, 626 GiB / 626 GiB avail; 683 KiB/s, 5 objects/s recovering 2024-09-13T14:26:26.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:26:26.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:26:26.788 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:26:26.809 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:26.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:26.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:26:26.894 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:26:26.911 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:26:26.932 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:26.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:26.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:26:27.034 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:26:27.056 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:27.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:27.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:26:27.158 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:26:27.183 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:27.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:27.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:26:27.285 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:27.308 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:27.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:27.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:27.410 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:26:27.431 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:27.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:27.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:26:27.519 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:26:27.520 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:26:27.520 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:26:27.520 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:26:27.533 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:26:27.534 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:26:27.535 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:26:27.553 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:26:27.553 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:26:27.556 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:27.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:27.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:26:27.569 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:26:27.569 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:26:27.586 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:26:27.586 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:26:27.602 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:26:27.602 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:26:27.618 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:26:27.618 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:26:27.659 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:26:27.681 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:27.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:27.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:26:27.782 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:26:27.805 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:27.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:27.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:26:27.907 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:26:27.930 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:27.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:27.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:26:28.032 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:26:28.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:28.055 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:28.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:26:28.156 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:26:28.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:28.180 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:28.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:26:28.281 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:26:28.304 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:28.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:28.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:26:28.406 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:28.430 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:28.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:28.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:28.492 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:28 smithi102 ceph-mon[115269]: pgmap v302: 89 pgs: 89 active+clean; 4.6 MiB data, 221 MiB used, 626 GiB / 626 GiB avail; 640 KiB/s, 5 objects/s recovering 2024-09-13T14:26:28.492 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:28 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/292160104' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:26:28.492 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:28 smithi102 ceph-mon[110183]: pgmap v302: 89 pgs: 89 active+clean; 4.6 MiB data, 221 MiB used, 626 GiB / 626 GiB avail; 640 KiB/s, 5 objects/s recovering 2024-09-13T14:26:28.492 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:28 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/292160104' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:26:28.532 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:26:28.554 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:28.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:28.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:26:28.656 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:26:28.677 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:28.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:28.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:26:28.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:28 smithi195 ceph-mon[104262]: pgmap v302: 89 pgs: 89 active+clean; 4.6 MiB data, 221 MiB used, 626 GiB / 626 GiB avail; 640 KiB/s, 5 objects/s recovering 2024-09-13T14:26:28.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:28 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/292160104' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:26:28.780 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:26:28.806 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:28.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:28.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:26:28.907 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:26:28.933 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:28.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:28.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:26:29.035 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:29.061 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:29.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:29.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:29.164 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:26:29.165 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:26:29.190 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:29.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:29.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:26:29.292 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:26:29.319 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:29.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:29.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:26:29.420 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:29.423 INFO:teuthology.orchestra.run.smithi102.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-13T14:26:29.445 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:29.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:29.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:29.547 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:26:29.573 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:29.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:29.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:26:29.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:29 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2923398537' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:26:29.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:29 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:26:29.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:29 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2923398537' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:26:29.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:29 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:26:29.675 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:26:29.696 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:29.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:29.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:26:29.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:29 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2923398537' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:26:29.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:29 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:26:29.798 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:26:29.823 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:29.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:29.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:26:29.925 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:26:29.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:29.951 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:29.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:26:30.052 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:26:30.074 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:30.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:30.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:26:30.176 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:30.202 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:30.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:30.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:30.304 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:26:30.330 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:30.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:30.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:26:30.432 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:26:30.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:30.458 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:30.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:26:30.494 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:26:30.559 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:26:30.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:30.582 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:30.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:26:30.683 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:26:30.708 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:30.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:30.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:26:30.719 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:30 smithi102 ceph-mon[110183]: pgmap v303: 89 pgs: 89 active+clean; 4.6 MiB data, 221 MiB used, 626 GiB / 626 GiB avail 2024-09-13T14:26:30.719 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:30 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:26:30.720 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:30 smithi102 ceph-mon[110183]: osdmap e119: 8 total, 8 up, 7 in 2024-09-13T14:26:30.720 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:30 smithi102 ceph-mon[115269]: pgmap v303: 89 pgs: 89 active+clean; 4.6 MiB data, 221 MiB used, 626 GiB / 626 GiB avail 2024-09-13T14:26:30.720 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:30 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:26:30.720 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:30 smithi102 ceph-mon[115269]: osdmap e119: 8 total, 8 up, 7 in 2024-09-13T14:26:30.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:30 smithi195 ceph-mon[104262]: pgmap v303: 89 pgs: 89 active+clean; 4.6 MiB data, 221 MiB used, 626 GiB / 626 GiB avail 2024-09-13T14:26:30.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:30 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:26:30.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:30 smithi195 ceph-mon[104262]: osdmap e119: 8 total, 8 up, 7 in 2024-09-13T14:26:30.810 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:26:30.836 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:30.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:30.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:26:30.937 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:26:30.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:30.961 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:30.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:26:31.062 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:26:31.063 DEBUG:teuthology.orchestra.run.smithi102:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-09-13T14:26:31.064 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_0 2024-09-13T14:26:31.064 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:26:31.068 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-13T14:26:31.068 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-13T14:26:31.087 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:31.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:31.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:26:31.117 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:hints = 1 2024-09-13T14:26:31.117 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-13T14:26:31.117 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Object prefix: benchmark_data_smithi102_155989 2024-09-13T14:26:31.118 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:26:31.118 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 0 0 0 0 0 0 - 0 2024-09-13T14:26:31.189 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:26:31.235 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:31.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:31.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:26:31.337 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:31.372 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:31.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:31.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:31.474 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:26:31.511 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:31.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:31.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:26:31.603 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:31 smithi102 ceph-mon[110183]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:26:31.603 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:31 smithi102 ceph-mon[115269]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:26:31.614 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:26:31.651 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:31.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:31.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:26:31.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:31 smithi195 ceph-mon[104262]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:26:31.752 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:26:31.790 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:31.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:31.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:26:31.892 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:26:31.937 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:31.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:31.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:26:31.948 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:26:31.949 INFO:teuthology.orchestra.run.smithi102.stdout: "size": 20, 2024-09-13T14:26:31.949 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 600, 2024-09-13T14:26:31.949 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [ 2024-09-13T14:26:31.949 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.949 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:47 5.3 5:cfce67ca:::benchmark_data_smithi102_155989_object46:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T14:26:31.949 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:26:31.132443+0000", 2024-09-13T14:26:31.949 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.81082095099999996, 2024-09-13T14:26:31.949 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.006185744, 2024-09-13T14:26:31.949 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:26:31.949 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:26:31.949 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:26:31.950 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:26:31.950 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:26:31.950 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:26:31.950 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 47 2024-09-13T14:26:31.950 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.950 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:26:31.950 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.950 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:26:31.950 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.132443+0000", 2024-09-13T14:26:31.950 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:26:31.950 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.950 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.950 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:26:31.950 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.132443+0000", 2024-09-13T14:26:31.951 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:26:31.951 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.951 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.951 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:26:31.951 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.132445+0000", 2024-09-13T14:26:31.951 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.714e-06 2024-09-13T14:26:31.951 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.951 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.951 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:26:31.951 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.132482+0000", 2024-09-13T14:26:31.951 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.6622999999999998e-05 2024-09-13T14:26:31.951 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.951 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.952 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:26:31.952 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.132494+0000", 2024-09-13T14:26:31.952 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2041e-05 2024-09-13T14:26:31.952 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.952 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.952 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:26:31.952 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.132501+0000", 2024-09-13T14:26:31.952 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.1960000000000002e-06 2024-09-13T14:26:31.952 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.952 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.952 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:26:31.952 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.132553+0000", 2024-09-13T14:26:31.952 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.2420999999999998e-05 2024-09-13T14:26:31.953 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.953 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.953 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:26:31.953 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.132656+0000", 2024-09-13T14:26:31.953 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000102099 2024-09-13T14:26:31.953 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.953 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.953 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:26:31.953 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.132782+0000", 2024-09-13T14:26:31.953 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000126124 2024-09-13T14:26:31.953 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.953 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.953 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:26:31.954 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.133169+0000", 2024-09-13T14:26:31.954 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00038772500000000001 2024-09-13T14:26:31.954 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.954 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.954 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:26:31.954 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.137803+0000", 2024-09-13T14:26:31.954 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.004633257 2024-09-13T14:26:31.954 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.954 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.954 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:26:31.954 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.138541+0000", 2024-09-13T14:26:31.954 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00073863100000000001 2024-09-13T14:26:31.954 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.955 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.955 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:26:31.955 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.138590+0000", 2024-09-13T14:26:31.955 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.9186e-05 2024-09-13T14:26:31.955 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.955 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.955 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:26:31.955 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.138629+0000", 2024-09-13T14:26:31.955 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.8726999999999997e-05 2024-09-13T14:26:31.955 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:26:31.955 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:26:31.955 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:26:31.955 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.956 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.956 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:216 5.3 5:cd14792c:::benchmark_data_smithi102_155989_object215:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T14:26:31.956 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:26:31.179525+0000", 2024-09-13T14:26:31.956 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.76373980900000005, 2024-09-13T14:26:31.956 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0058373560000000001, 2024-09-13T14:26:31.956 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:26:31.956 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:26:31.956 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:26:31.956 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:26:31.956 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:26:31.956 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:26:31.956 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 216 2024-09-13T14:26:31.956 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.957 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:26:31.957 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.957 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:26:31.957 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.179525+0000", 2024-09-13T14:26:31.957 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:26:31.957 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.957 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.957 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:26:31.957 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.179525+0000", 2024-09-13T14:26:31.957 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:26:31.957 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.957 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.957 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:26:31.958 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.179527+0000", 2024-09-13T14:26:31.958 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.886e-06 2024-09-13T14:26:31.958 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.958 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.958 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:26:31.958 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.179553+0000", 2024-09-13T14:26:31.958 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.6276999999999999e-05 2024-09-13T14:26:31.958 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.958 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.958 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:26:31.958 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.179576+0000", 2024-09-13T14:26:31.958 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.3275e-05 2024-09-13T14:26:31.958 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.958 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.959 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:26:31.959 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.179584+0000", 2024-09-13T14:26:31.959 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.9240000000000007e-06 2024-09-13T14:26:31.959 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.959 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.959 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:26:31.959 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.179637+0000", 2024-09-13T14:26:31.959 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.2967000000000003e-05 2024-09-13T14:26:31.959 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.959 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.959 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:26:31.959 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.179762+0000", 2024-09-13T14:26:31.959 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00012491299999999999 2024-09-13T14:26:31.959 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.959 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.960 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:26:31.960 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.179969+0000", 2024-09-13T14:26:31.960 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00020729400000000001 2024-09-13T14:26:31.960 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.960 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.960 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:26:31.960 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.181274+0000", 2024-09-13T14:26:31.960 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.001304342 2024-09-13T14:26:31.960 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.960 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.960 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:26:31.960 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.181307+0000", 2024-09-13T14:26:31.960 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.3250999999999997e-05 2024-09-13T14:26:31.961 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.961 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.961 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:26:31.961 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.185272+0000", 2024-09-13T14:26:31.961 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.003965433 2024-09-13T14:26:31.961 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.961 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.961 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:26:31.961 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.185324+0000", 2024-09-13T14:26:31.961 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.1428999999999998e-05 2024-09-13T14:26:31.961 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.961 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.961 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:26:31.961 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.185362+0000", 2024-09-13T14:26:31.961 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.8365000000000003e-05 2024-09-13T14:26:31.962 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:26:31.962 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:26:31.962 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:26:31.962 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.962 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.962 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:218 5.3 5:cfb04aab:::benchmark_data_smithi102_155989_object217:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T14:26:31.962 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:26:31.179800+0000", 2024-09-13T14:26:31.962 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.76346460999999999, 2024-09-13T14:26:31.962 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0057549519999999998, 2024-09-13T14:26:31.962 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:26:31.962 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:26:31.962 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:26:31.962 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:26:31.963 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:26:31.963 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:26:31.963 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 218 2024-09-13T14:26:31.963 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.963 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:26:31.963 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.963 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:26:31.963 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.179800+0000", 2024-09-13T14:26:31.963 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:26:31.963 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.963 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.963 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:26:31.963 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.179800+0000", 2024-09-13T14:26:31.964 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:26:31.964 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.964 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.964 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:26:31.964 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.179801+0000", 2024-09-13T14:26:31.964 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.5000000000000001e-07 2024-09-13T14:26:31.964 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.964 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.964 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:26:31.964 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.179828+0000", 2024-09-13T14:26:31.964 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.7347000000000001e-05 2024-09-13T14:26:31.964 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.964 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.964 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:26:31.965 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.179839+0000", 2024-09-13T14:26:31.965 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1347e-05 2024-09-13T14:26:31.965 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.965 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.965 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:26:31.965 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.179846+0000", 2024-09-13T14:26:31.965 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.8739999999999997e-06 2024-09-13T14:26:31.965 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.965 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.965 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:26:31.965 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.180231+0000", 2024-09-13T14:26:31.965 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000384378 2024-09-13T14:26:31.965 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.966 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.966 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:26:31.966 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.180343+0000", 2024-09-13T14:26:31.966 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00011233799999999999 2024-09-13T14:26:31.966 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.966 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.966 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:26:31.966 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.180447+0000", 2024-09-13T14:26:31.966 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000103688 2024-09-13T14:26:31.966 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.966 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.966 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:26:31.966 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.181338+0000", 2024-09-13T14:26:31.966 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00089101399999999998 2024-09-13T14:26:31.967 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.967 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.967 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:26:31.967 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.182667+0000", 2024-09-13T14:26:31.967 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.001329025 2024-09-13T14:26:31.967 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.967 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.967 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:26:31.967 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.185461+0000", 2024-09-13T14:26:31.967 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0027937880000000002 2024-09-13T14:26:31.967 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.967 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.967 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:26:31.967 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.185515+0000", 2024-09-13T14:26:31.968 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.4718e-05 2024-09-13T14:26:31.968 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.968 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.968 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:26:31.968 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.185555+0000", 2024-09-13T14:26:31.968 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.9484999999999999e-05 2024-09-13T14:26:31.968 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:26:31.968 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:26:31.968 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:26:31.968 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.968 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.968 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:219 5.3 5:c175b12e:::benchmark_data_smithi102_155989_object218:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T14:26:31.968 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:26:31.179977+0000", 2024-09-13T14:26:31.969 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.76328710899999996, 2024-09-13T14:26:31.969 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0057190840000000001, 2024-09-13T14:26:31.969 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:26:31.969 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:26:31.969 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:26:31.969 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:26:31.969 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:26:31.969 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:26:31.969 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 219 2024-09-13T14:26:31.969 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.969 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:26:31.969 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.969 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:26:31.969 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.179977+0000", 2024-09-13T14:26:31.970 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:26:31.970 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.970 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.970 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:26:31.970 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.179977+0000", 2024-09-13T14:26:31.970 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:26:31.970 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.970 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.970 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:26:31.970 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.179978+0000", 2024-09-13T14:26:31.970 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.3399999999999998e-07 2024-09-13T14:26:31.970 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.970 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.971 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:26:31.971 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.179997+0000", 2024-09-13T14:26:31.971 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.8512000000000002e-05 2024-09-13T14:26:31.971 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.971 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.971 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:26:31.971 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.180006+0000", 2024-09-13T14:26:31.971 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.825e-06 2024-09-13T14:26:31.971 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.971 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.971 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:26:31.971 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.180014+0000", 2024-09-13T14:26:31.971 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.0029999999999994e-06 2024-09-13T14:26:31.971 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.972 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.972 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:26:31.972 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.180836+0000", 2024-09-13T14:26:31.972 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00082118000000000002 2024-09-13T14:26:31.972 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.972 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.972 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:26:31.972 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.180944+0000", 2024-09-13T14:26:31.972 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000108162 2024-09-13T14:26:31.972 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.972 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.972 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:26:31.972 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.181068+0000", 2024-09-13T14:26:31.972 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00012368899999999999 2024-09-13T14:26:31.973 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.973 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.973 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:26:31.973 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.182727+0000", 2024-09-13T14:26:31.973 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.001659153 2024-09-13T14:26:31.973 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.973 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.973 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:26:31.973 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.183587+0000", 2024-09-13T14:26:31.973 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00085999399999999999 2024-09-13T14:26:31.973 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.973 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.973 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:26:31.974 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.185591+0000", 2024-09-13T14:26:31.974 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0020047730000000001 2024-09-13T14:26:31.974 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.974 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.974 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:26:31.974 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.185636+0000", 2024-09-13T14:26:31.974 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.4589999999999998e-05 2024-09-13T14:26:31.974 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.974 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.974 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:26:31.974 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.185696+0000", 2024-09-13T14:26:31.974 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.0368999999999997e-05 2024-09-13T14:26:31.974 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:26:31.974 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:26:31.974 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:26:31.975 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.975 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.975 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:277 5.5 5:af9c441c:::benchmark_data_smithi102_155989_object276:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T14:26:31.975 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:26:31.197217+0000", 2024-09-13T14:26:31.975 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.74604715600000004, 2024-09-13T14:26:31.975 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0061113850000000004, 2024-09-13T14:26:31.975 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:26:31.975 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:26:31.975 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:26:31.975 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:26:31.975 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:26:31.975 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:26:31.975 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 277 2024-09-13T14:26:31.976 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.976 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:26:31.976 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.976 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:26:31.976 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.197217+0000", 2024-09-13T14:26:31.976 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:26:31.976 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.976 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.976 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:26:31.976 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.197217+0000", 2024-09-13T14:26:31.976 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:26:31.976 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.976 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.976 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:26:31.977 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.197219+0000", 2024-09-13T14:26:31.978 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.361e-06 2024-09-13T14:26:31.978 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.978 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.978 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:26:31.978 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.197240+0000", 2024-09-13T14:26:31.978 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.1223999999999999e-05 2024-09-13T14:26:31.978 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.978 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.978 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:26:31.978 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.197250+0000", 2024-09-13T14:26:31.978 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0535e-05 2024-09-13T14:26:31.978 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.978 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.979 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:26:31.979 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.197257+0000", 2024-09-13T14:26:31.979 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.7349999999999999e-06 2024-09-13T14:26:31.979 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.979 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.979 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:26:31.979 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.197578+0000", 2024-09-13T14:26:31.979 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00032122800000000001 2024-09-13T14:26:31.979 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.979 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.979 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:26:31.979 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.197703+0000", 2024-09-13T14:26:31.979 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00012491100000000001 2024-09-13T14:26:31.979 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.980 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.980 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:26:31.980 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.197861+0000", 2024-09-13T14:26:31.980 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00015757400000000001 2024-09-13T14:26:31.980 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.980 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.980 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:26:31.980 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.198304+0000", 2024-09-13T14:26:31.980 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00044304499999999998 2024-09-13T14:26:31.980 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.980 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.980 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:26:31.980 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.200667+0000", 2024-09-13T14:26:31.981 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00236273 2024-09-13T14:26:31.981 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.981 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.981 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:26:31.981 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.203220+0000", 2024-09-13T14:26:31.981 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0025533859999999999 2024-09-13T14:26:31.981 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.981 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.981 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:26:31.981 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.203284+0000", 2024-09-13T14:26:31.981 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.3793000000000004e-05 2024-09-13T14:26:31.981 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.981 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.981 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:26:31.982 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.203329+0000", 2024-09-13T14:26:31.982 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.4863000000000001e-05 2024-09-13T14:26:31.982 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:26:31.982 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:26:31.982 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:26:31.982 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.982 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.982 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:390 5.5 5:a51a7a21:::benchmark_data_smithi102_155989_object389:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T14:26:31.982 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:26:31.229172+0000", 2024-09-13T14:26:31.982 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.714092269, 2024-09-13T14:26:31.982 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0068347829999999997, 2024-09-13T14:26:31.982 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:26:31.982 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:26:31.983 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:26:31.983 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:26:31.983 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:26:31.983 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:26:31.983 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 390 2024-09-13T14:26:31.983 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.983 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:26:31.983 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.984 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:26:31.984 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.229172+0000", 2024-09-13T14:26:31.984 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:26:31.984 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.984 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.984 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:26:31.984 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.229172+0000", 2024-09-13T14:26:31.984 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:26:31.984 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.984 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.984 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:26:31.984 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.229173+0000", 2024-09-13T14:26:31.984 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.1299999999999999e-07 2024-09-13T14:26:31.984 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.985 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.985 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:26:31.985 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.229192+0000", 2024-09-13T14:26:31.985 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.9366000000000001e-05 2024-09-13T14:26:31.985 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.985 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.985 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:26:31.985 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.229202+0000", 2024-09-13T14:26:31.985 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.3409999999999992e-06 2024-09-13T14:26:31.985 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.985 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.985 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:26:31.985 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.229209+0000", 2024-09-13T14:26:31.985 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.5900000000000002e-06 2024-09-13T14:26:31.986 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.986 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.986 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:26:31.986 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.229866+0000", 2024-09-13T14:26:31.986 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00065677499999999996 2024-09-13T14:26:31.986 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.986 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.986 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:26:31.986 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.229971+0000", 2024-09-13T14:26:31.986 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00010501999999999999 2024-09-13T14:26:31.986 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.986 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.986 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:26:31.986 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.230140+0000", 2024-09-13T14:26:31.987 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00016911500000000001 2024-09-13T14:26:31.987 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.987 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.987 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:26:31.987 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.231220+0000", 2024-09-13T14:26:31.987 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0010798209999999999 2024-09-13T14:26:31.987 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.987 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.987 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:26:31.987 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.233716+0000", 2024-09-13T14:26:31.987 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0024955910000000001 2024-09-13T14:26:31.987 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.987 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.988 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:26:31.988 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.235907+0000", 2024-09-13T14:26:31.988 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0021916539999999999 2024-09-13T14:26:31.988 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.988 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.988 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:26:31.988 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.235964+0000", 2024-09-13T14:26:31.988 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.6734000000000003e-05 2024-09-13T14:26:31.988 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.988 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.988 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:26:31.988 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.236007+0000", 2024-09-13T14:26:31.988 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.2963000000000002e-05 2024-09-13T14:26:31.988 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:26:31.989 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:26:31.989 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:26:31.989 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.989 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.989 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:1494 5.5 5:aac17679:::benchmark_data_smithi102_155989_object1493:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T14:26:31.989 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:26:31.489062+0000", 2024-09-13T14:26:31.989 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.45420234300000001, 2024-09-13T14:26:31.989 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0057615950000000004, 2024-09-13T14:26:31.989 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:26:31.989 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:26:31.989 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:26:31.989 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:26:31.989 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:26:31.989 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:26:31.990 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 1494 2024-09-13T14:26:31.990 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.990 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:26:31.990 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.990 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:26:31.990 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.489062+0000", 2024-09-13T14:26:31.990 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:26:31.990 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.990 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.990 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:26:31.990 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.489062+0000", 2024-09-13T14:26:31.990 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:26:31.990 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.990 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.991 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:26:31.991 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.489063+0000", 2024-09-13T14:26:31.991 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.44e-07 2024-09-13T14:26:31.991 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.991 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.991 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:26:31.991 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.489080+0000", 2024-09-13T14:26:31.991 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.7244999999999998e-05 2024-09-13T14:26:31.991 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.991 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.991 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:26:31.991 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.489094+0000", 2024-09-13T14:26:31.991 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.379e-05 2024-09-13T14:26:31.991 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.992 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.992 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:26:31.992 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.489101+0000", 2024-09-13T14:26:31.992 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.4619999999999999e-06 2024-09-13T14:26:31.992 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.992 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.992 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:26:31.992 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.489181+0000", 2024-09-13T14:26:31.992 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.9271999999999996e-05 2024-09-13T14:26:31.992 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.992 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.992 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:26:31.993 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.489910+0000", 2024-09-13T14:26:31.993 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00072987900000000001 2024-09-13T14:26:31.993 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.993 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.993 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:26:31.993 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.490042+0000", 2024-09-13T14:26:31.993 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00013204100000000001 2024-09-13T14:26:31.993 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.993 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.993 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:26:31.993 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.490665+0000", 2024-09-13T14:26:31.993 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00062269099999999998 2024-09-13T14:26:31.993 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.994 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.994 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:26:31.994 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.491275+0000", 2024-09-13T14:26:31.994 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00061024500000000004 2024-09-13T14:26:31.994 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.994 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.994 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:26:31.994 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.494727+0000", 2024-09-13T14:26:31.994 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0034516989999999999 2024-09-13T14:26:31.994 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.994 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.994 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:26:31.994 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.494786+0000", 2024-09-13T14:26:31.994 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.8808999999999999e-05 2024-09-13T14:26:31.995 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.995 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.995 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:26:31.995 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.494824+0000", 2024-09-13T14:26:31.995 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.7818000000000003e-05 2024-09-13T14:26:31.995 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:26:31.995 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:26:31.995 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:26:31.995 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.995 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.995 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:2059 5.3 5:c17f7602:::benchmark_data_smithi102_155989_object2058:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T14:26:31.995 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:26:31.619845+0000", 2024-09-13T14:26:31.996 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.32341936100000002, 2024-09-13T14:26:31.996 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0054171519999999997, 2024-09-13T14:26:31.996 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:26:31.996 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:26:31.996 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:26:31.996 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:26:31.996 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:26:31.996 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:26:31.996 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 2059 2024-09-13T14:26:31.996 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.996 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:26:31.996 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.996 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:26:31.996 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.619845+0000", 2024-09-13T14:26:31.997 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:26:31.997 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.997 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.997 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:26:31.997 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.619845+0000", 2024-09-13T14:26:31.997 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:26:31.997 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.997 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.997 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:26:31.997 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.619846+0000", 2024-09-13T14:26:31.997 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.2e-07 2024-09-13T14:26:31.997 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.997 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.998 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:26:31.998 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.619869+0000", 2024-09-13T14:26:31.998 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.3734e-05 2024-09-13T14:26:31.998 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.998 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.998 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:26:31.998 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.619877+0000", 2024-09-13T14:26:31.998 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.0290000000000005e-06 2024-09-13T14:26:31.998 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.998 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.998 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:26:31.998 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.619883+0000", 2024-09-13T14:26:31.998 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.1830000000000002e-06 2024-09-13T14:26:31.999 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.999 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.999 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:26:31.999 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.620608+0000", 2024-09-13T14:26:31.999 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00072562899999999999 2024-09-13T14:26:31.999 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.999 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.999 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:26:31.999 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.620715+0000", 2024-09-13T14:26:31.999 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000106517 2024-09-13T14:26:31.999 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:31.999 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:31.999 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:26:31.999 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.620841+0000", 2024-09-13T14:26:32.000 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000125857 2024-09-13T14:26:32.000 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.000 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.000 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:26:32.000 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.621574+0000", 2024-09-13T14:26:32.000 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00073324299999999996 2024-09-13T14:26:32.000 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.000 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.000 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:26:32.000 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.622443+0000", 2024-09-13T14:26:32.000 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00086958100000000002 2024-09-13T14:26:32.000 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.001 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.001 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:26:32.001 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.625184+0000", 2024-09-13T14:26:32.001 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.002740892 2024-09-13T14:26:32.001 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.001 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.001 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:26:32.001 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.625229+0000", 2024-09-13T14:26:32.001 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.4725000000000003e-05 2024-09-13T14:26:32.001 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.001 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.001 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:26:32.001 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.625262+0000", 2024-09-13T14:26:32.002 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.3241999999999998e-05 2024-09-13T14:26:32.002 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:26:32.002 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:26:32.002 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:26:32.002 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.002 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.002 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:2201 5.3 5:c7d56c68:::benchmark_data_smithi102_155989_object2200:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T14:26:32.002 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:26:31.654580+0000", 2024-09-13T14:26:32.002 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.28868401599999999, 2024-09-13T14:26:32.002 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0089130149999999998, 2024-09-13T14:26:32.002 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:26:32.002 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:26:32.002 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:26:32.002 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:26:32.003 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:26:32.003 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:26:32.003 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 2201 2024-09-13T14:26:32.003 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.003 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:26:32.003 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.003 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:26:32.003 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.654580+0000", 2024-09-13T14:26:32.003 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:26:32.003 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.003 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.003 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:26:32.003 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.654580+0000", 2024-09-13T14:26:32.003 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:26:32.004 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.004 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.004 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:26:32.004 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.654582+0000", 2024-09-13T14:26:32.004 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4309999999999999e-06 2024-09-13T14:26:32.004 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.004 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.004 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:26:32.004 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.654605+0000", 2024-09-13T14:26:32.004 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.2702e-05 2024-09-13T14:26:32.004 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.004 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.004 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:26:32.004 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.654616+0000", 2024-09-13T14:26:32.005 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0978000000000001e-05 2024-09-13T14:26:32.005 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.008 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.008 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:26:32.008 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.654623+0000", 2024-09-13T14:26:32.008 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.1189999999999999e-06 2024-09-13T14:26:32.008 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.008 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.008 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:26:32.008 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.654941+0000", 2024-09-13T14:26:32.008 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00031857499999999998 2024-09-13T14:26:32.008 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.008 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.009 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:26:32.009 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.655089+0000", 2024-09-13T14:26:32.009 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00014731499999999999 2024-09-13T14:26:32.009 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.009 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.009 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:26:32.009 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.655203+0000", 2024-09-13T14:26:32.009 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000114632 2024-09-13T14:26:32.009 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.009 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.009 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:26:32.009 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.656530+0000", 2024-09-13T14:26:32.009 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.001326799 2024-09-13T14:26:32.010 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.010 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.010 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:26:32.010 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.656844+0000", 2024-09-13T14:26:32.010 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000313839 2024-09-13T14:26:32.010 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.010 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.010 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:26:32.010 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.663414+0000", 2024-09-13T14:26:32.010 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0065697890000000004 2024-09-13T14:26:32.010 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.010 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.010 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:26:32.011 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.663457+0000", 2024-09-13T14:26:32.011 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.3380000000000001e-05 2024-09-13T14:26:32.011 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.011 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.011 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:26:32.011 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.663493+0000", 2024-09-13T14:26:32.011 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.6455999999999998e-05 2024-09-13T14:26:32.011 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:26:32.011 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:26:32.011 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:26:32.011 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.011 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.011 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.44546.0:2199 5.2 e119/116 5:4476ad34:::benchmark_data_smithi102_155989_object2198:head v 119'128, mlcod=119'126)", 2024-09-13T14:26:32.012 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:26:31.654933+0000", 2024-09-13T14:26:32.012 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.28833175500000002, 2024-09-13T14:26:32.012 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0083924220000000001, 2024-09-13T14:26:32.012 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:26:32.012 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:26:32.012 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:26:32.012 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:26:32.012 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.012 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:26:32.012 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.654933+0000", 2024-09-13T14:26:32.012 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:26:32.012 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.012 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.013 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:26:32.013 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.654933+0000", 2024-09-13T14:26:32.013 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:26:32.013 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.013 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.013 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:26:32.013 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.654933+0000", 2024-09-13T14:26:32.013 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.46e-07 2024-09-13T14:26:32.013 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.013 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.013 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:26:32.013 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.654964+0000", 2024-09-13T14:26:32.013 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.1254000000000003e-05 2024-09-13T14:26:32.013 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.014 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.014 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:26:32.014 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.654972+0000", 2024-09-13T14:26:32.014 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.3749999999999997e-06 2024-09-13T14:26:32.014 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.014 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.014 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:26:32.014 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.654977+0000", 2024-09-13T14:26:32.014 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.4299999999999997e-06 2024-09-13T14:26:32.014 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.014 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.014 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:26:32.014 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.655032+0000", 2024-09-13T14:26:32.015 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.4475999999999998e-05 2024-09-13T14:26:32.015 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.015 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.015 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:26:32.015 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.655050+0000", 2024-09-13T14:26:32.015 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.8053000000000001e-05 2024-09-13T14:26:32.015 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.015 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.015 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:26:32.015 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.663284+0000", 2024-09-13T14:26:32.015 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.008233878 2024-09-13T14:26:32.015 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.015 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.015 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:26:32.016 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.663325+0000", 2024-09-13T14:26:32.016 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.1510000000000001e-05 2024-09-13T14:26:32.016 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:26:32.016 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:26:32.016 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:26:32.016 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.016 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.016 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.44546.0:2203 5.b e119/116 5:d5a5767a:::benchmark_data_smithi102_155989_object2202:head v 119'124, mlcod=119'122)", 2024-09-13T14:26:32.016 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:26:31.655479+0000", 2024-09-13T14:26:32.016 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.28778525999999999, 2024-09-13T14:26:32.016 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0059109000000000002, 2024-09-13T14:26:32.017 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:26:32.017 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:26:32.017 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:26:32.017 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:26:32.017 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.017 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:26:32.017 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.655479+0000", 2024-09-13T14:26:32.017 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:26:32.017 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.017 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.017 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:26:32.017 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.655479+0000", 2024-09-13T14:26:32.018 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:26:32.018 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.018 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.018 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:26:32.018 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.655480+0000", 2024-09-13T14:26:32.018 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.71e-07 2024-09-13T14:26:32.018 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.018 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.018 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:26:32.018 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.655511+0000", 2024-09-13T14:26:32.018 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.0706e-05 2024-09-13T14:26:32.018 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.018 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.018 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:26:32.019 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.655520+0000", 2024-09-13T14:26:32.019 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.0829999999999993e-06 2024-09-13T14:26:32.019 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.019 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.019 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:26:32.019 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.655526+0000", 2024-09-13T14:26:32.019 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.8980000000000001e-06 2024-09-13T14:26:32.019 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.019 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.019 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:26:32.019 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.655573+0000", 2024-09-13T14:26:32.019 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.7299999999999998e-05 2024-09-13T14:26:32.019 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.019 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.020 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:26:32.020 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.655594+0000", 2024-09-13T14:26:32.020 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.0883999999999999e-05 2024-09-13T14:26:32.020 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.020 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.020 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:26:32.020 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.661337+0000", 2024-09-13T14:26:32.020 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0057432189999999999 2024-09-13T14:26:32.020 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.020 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.020 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:26:32.020 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.661390+0000", 2024-09-13T14:26:32.020 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.2939000000000003e-05 2024-09-13T14:26:32.021 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:26:32.021 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:26:32.021 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:26:32.021 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.021 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.021 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:2209 5.5 5:af747015:::benchmark_data_smithi102_155989_object2208:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T14:26:32.021 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:26:31.655905+0000", 2024-09-13T14:26:32.021 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.287359912, 2024-09-13T14:26:32.021 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0074675139999999997, 2024-09-13T14:26:32.021 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:26:32.021 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:26:32.021 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:26:32.021 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:26:32.021 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:26:32.022 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:26:32.022 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 2209 2024-09-13T14:26:32.022 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.022 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:26:32.022 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.022 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:26:32.022 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.655905+0000", 2024-09-13T14:26:32.022 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:26:32.022 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.022 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.022 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:26:32.023 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.655905+0000", 2024-09-13T14:26:32.023 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:26:32.023 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.023 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.023 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:26:32.023 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.655906+0000", 2024-09-13T14:26:32.023 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1090000000000001e-06 2024-09-13T14:26:32.023 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.023 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.023 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:26:32.023 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.655927+0000", 2024-09-13T14:26:32.023 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.1090999999999998e-05 2024-09-13T14:26:32.024 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.025 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.025 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:26:32.025 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.655938+0000", 2024-09-13T14:26:32.025 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1161999999999999e-05 2024-09-13T14:26:32.025 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.025 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.025 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:26:32.025 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.655945+0000", 2024-09-13T14:26:32.025 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.9120000000000001e-06 2024-09-13T14:26:32.025 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.025 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.025 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:26:32.025 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.656001+0000", 2024-09-13T14:26:32.026 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.6344000000000001e-05 2024-09-13T14:26:32.026 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.026 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.026 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:26:32.026 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.656130+0000", 2024-09-13T14:26:32.026 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00012914 2024-09-13T14:26:32.026 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.026 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.026 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:26:32.026 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.656285+0000", 2024-09-13T14:26:32.026 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00015455300000000001 2024-09-13T14:26:32.026 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.026 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.026 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:26:32.027 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.657575+0000", 2024-09-13T14:26:32.027 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0012896959999999999 2024-09-13T14:26:32.027 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.027 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.027 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:26:32.027 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.658609+0000", 2024-09-13T14:26:32.027 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0010342319999999999 2024-09-13T14:26:32.027 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.027 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.027 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:26:32.027 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.663271+0000", 2024-09-13T14:26:32.027 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.004662696 2024-09-13T14:26:32.027 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.028 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.028 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:26:32.028 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.663325+0000", 2024-09-13T14:26:32.028 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.3862e-05 2024-09-13T14:26:32.028 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.028 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.028 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:26:32.028 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.663372+0000", 2024-09-13T14:26:32.028 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.6717e-05 2024-09-13T14:26:32.028 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:26:32.028 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:26:32.028 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:26:32.028 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.028 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.029 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.44546.0:2207 5.7 e119/116 5:eddd8243:::benchmark_data_smithi102_155989_object2206:head v 119'168, mlcod=119'166)", 2024-09-13T14:26:32.029 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:26:31.656147+0000", 2024-09-13T14:26:32.029 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.287117018, 2024-09-13T14:26:32.029 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0072209520000000001, 2024-09-13T14:26:32.029 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:26:32.029 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:26:32.029 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:26:32.029 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:26:32.029 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.030 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:26:32.030 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.656147+0000", 2024-09-13T14:26:32.030 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:26:32.030 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.030 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.030 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:26:32.030 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.656147+0000", 2024-09-13T14:26:32.030 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:26:32.030 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.030 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.030 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:26:32.030 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.656148+0000", 2024-09-13T14:26:32.030 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.06e-07 2024-09-13T14:26:32.030 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.031 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.031 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:26:32.031 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.656224+0000", 2024-09-13T14:26:32.031 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.6234000000000002e-05 2024-09-13T14:26:32.031 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.031 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.031 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:26:32.031 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.656233+0000", 2024-09-13T14:26:32.031 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.2109999999999998e-06 2024-09-13T14:26:32.031 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.031 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.031 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:26:32.031 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.656238+0000", 2024-09-13T14:26:32.032 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.502e-06 2024-09-13T14:26:32.032 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.032 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.032 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:26:32.032 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.656585+0000", 2024-09-13T14:26:32.032 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00034738400000000001 2024-09-13T14:26:32.032 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.032 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.032 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:26:32.032 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.656597+0000", 2024-09-13T14:26:32.032 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1968e-05 2024-09-13T14:26:32.032 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.032 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.032 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:26:32.033 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.663337+0000", 2024-09-13T14:26:32.033 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0067395500000000004 2024-09-13T14:26:32.033 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.033 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.033 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:26:32.033 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.663368+0000", 2024-09-13T14:26:32.033 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.1396999999999998e-05 2024-09-13T14:26:32.033 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:26:32.033 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:26:32.033 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:26:32.033 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.033 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.033 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:2214 5.5 5:a5888b13:::benchmark_data_smithi102_155989_object2213:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T14:26:32.033 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:26:31.656796+0000", 2024-09-13T14:26:32.034 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.28646847600000003, 2024-09-13T14:26:32.034 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0066570780000000003, 2024-09-13T14:26:32.034 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:26:32.034 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:26:32.034 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:26:32.034 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:26:32.034 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:26:32.034 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:26:32.034 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 2214 2024-09-13T14:26:32.034 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.034 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:26:32.034 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.034 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:26:32.034 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.656796+0000", 2024-09-13T14:26:32.035 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:26:32.035 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.035 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.035 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:26:32.035 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.656796+0000", 2024-09-13T14:26:32.035 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:26:32.035 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.035 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.035 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:26:32.035 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.656797+0000", 2024-09-13T14:26:32.035 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0690000000000001e-06 2024-09-13T14:26:32.035 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.035 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.036 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:26:32.036 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.656823+0000", 2024-09-13T14:26:32.036 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.5618000000000001e-05 2024-09-13T14:26:32.036 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.036 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.036 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:26:32.036 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.656836+0000", 2024-09-13T14:26:32.036 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2894000000000001e-05 2024-09-13T14:26:32.036 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.036 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.036 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:26:32.036 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.656843+0000", 2024-09-13T14:26:32.036 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.323e-06 2024-09-13T14:26:32.036 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.037 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.037 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:26:32.037 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.656898+0000", 2024-09-13T14:26:32.037 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.5553999999999997e-05 2024-09-13T14:26:32.037 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.037 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.037 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:26:32.037 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.657009+0000", 2024-09-13T14:26:32.037 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000110512 2024-09-13T14:26:32.037 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.037 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.037 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:26:32.037 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.657145+0000", 2024-09-13T14:26:32.037 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000135728 2024-09-13T14:26:32.038 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.038 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.038 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:26:32.038 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.658201+0000", 2024-09-13T14:26:32.038 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0010564400000000001 2024-09-13T14:26:32.038 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.038 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.038 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:26:32.038 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.658725+0000", 2024-09-13T14:26:32.038 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00052423900000000002 2024-09-13T14:26:32.038 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.038 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.038 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:26:32.038 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.663381+0000", 2024-09-13T14:26:32.039 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0046558600000000004 2024-09-13T14:26:32.039 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.039 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.039 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:26:32.039 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.663421+0000", 2024-09-13T14:26:32.039 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.9900000000000001e-05 2024-09-13T14:26:32.039 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.039 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.039 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:26:32.039 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.663453+0000", 2024-09-13T14:26:32.039 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.1940999999999998e-05 2024-09-13T14:26:32.039 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:26:32.039 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:26:32.040 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:26:32.040 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.040 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.040 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.44546.0:2204 5.e e119/116 5:7c61af62:::benchmark_data_smithi102_155989_object2203:head v 119'121, mlcod=119'119)", 2024-09-13T14:26:32.040 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:26:31.657700+0000", 2024-09-13T14:26:32.040 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.28556448699999998, 2024-09-13T14:26:32.040 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0055815819999999999, 2024-09-13T14:26:32.040 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:26:32.040 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:26:32.040 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:26:32.040 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:26:32.040 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.040 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:26:32.041 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.657700+0000", 2024-09-13T14:26:32.041 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:26:32.041 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.041 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.041 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:26:32.041 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.657700+0000", 2024-09-13T14:26:32.041 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:26:32.041 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.041 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.041 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:26:32.041 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.657701+0000", 2024-09-13T14:26:32.041 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.9999999999999997e-07 2024-09-13T14:26:32.041 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.041 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.042 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:26:32.042 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.657764+0000", 2024-09-13T14:26:32.042 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.3021999999999994e-05 2024-09-13T14:26:32.042 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.042 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.042 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:26:32.042 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.657775+0000", 2024-09-13T14:26:32.042 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1126e-05 2024-09-13T14:26:32.042 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.042 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.042 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:26:32.042 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.657779+0000", 2024-09-13T14:26:32.042 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.6249999999999998e-06 2024-09-13T14:26:32.043 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.043 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.043 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:26:32.043 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.657831+0000", 2024-09-13T14:26:32.043 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.2074999999999999e-05 2024-09-13T14:26:32.043 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.043 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.043 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:26:32.043 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.657862+0000", 2024-09-13T14:26:32.043 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.0383e-05 2024-09-13T14:26:32.043 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.043 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.043 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:26:32.043 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.663224+0000", 2024-09-13T14:26:32.044 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0053620339999999999 2024-09-13T14:26:32.044 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.044 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.044 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:26:32.044 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.663282+0000", 2024-09-13T14:26:32.044 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.7617000000000001e-05 2024-09-13T14:26:32.044 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:26:32.044 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:26:32.044 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:26:32.044 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.044 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.044 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.44546.0:2208 5.e e119/116 5:7bdf1087:::benchmark_data_smithi102_155989_object2207:head v 119'122, mlcod=119'119)", 2024-09-13T14:26:32.044 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:26:31.657792+0000", 2024-09-13T14:26:32.044 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.28547262099999998, 2024-09-13T14:26:32.045 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0058586289999999997, 2024-09-13T14:26:32.045 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:26:32.045 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:26:32.045 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:26:32.045 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:26:32.045 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.045 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:26:32.045 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.657792+0000", 2024-09-13T14:26:32.045 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:26:32.045 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.045 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.045 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:26:32.046 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.657792+0000", 2024-09-13T14:26:32.046 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:26:32.046 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.046 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.046 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:26:32.046 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.657792+0000", 2024-09-13T14:26:32.046 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.4400000000000001e-07 2024-09-13T14:26:32.046 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.046 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.046 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:26:32.046 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.657839+0000", 2024-09-13T14:26:32.046 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.6978000000000003e-05 2024-09-13T14:26:32.046 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.046 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.047 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:26:32.047 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.657847+0000", 2024-09-13T14:26:32.047 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.7619999999999995e-06 2024-09-13T14:26:32.047 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.047 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.047 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:26:32.047 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.657862+0000", 2024-09-13T14:26:32.047 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4661000000000001e-05 2024-09-13T14:26:32.047 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.047 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.047 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:26:32.047 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.658111+0000", 2024-09-13T14:26:32.047 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00024947199999999999 2024-09-13T14:26:32.048 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.048 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.048 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:26:32.048 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.658130+0000", 2024-09-13T14:26:32.048 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.8556999999999999e-05 2024-09-13T14:26:32.048 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.048 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.048 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:26:32.048 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.663600+0000", 2024-09-13T14:26:32.048 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0054699270000000003 2024-09-13T14:26:32.048 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.048 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.048 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:26:32.049 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.663650+0000", 2024-09-13T14:26:32.049 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.0927999999999998e-05 2024-09-13T14:26:32.049 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:26:32.049 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:26:32.049 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:26:32.049 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.049 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.049 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.44546.0:2213 5.c e119/116 5:3bdfe1b2:::benchmark_data_smithi102_155989_object2212:head v 119'149, mlcod=119'147)", 2024-09-13T14:26:32.049 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:26:31.657874+0000", 2024-09-13T14:26:32.049 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.28538994699999998, 2024-09-13T14:26:32.049 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0055987620000000002, 2024-09-13T14:26:32.049 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:26:32.049 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:26:32.049 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:26:32.050 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:26:32.050 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.050 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:26:32.050 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.657874+0000", 2024-09-13T14:26:32.050 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:26:32.050 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.050 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.050 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:26:32.050 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.657874+0000", 2024-09-13T14:26:32.050 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:26:32.050 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.050 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.050 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:26:32.051 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.657875+0000", 2024-09-13T14:26:32.051 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.65e-07 2024-09-13T14:26:32.051 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.051 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.051 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:26:32.051 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.657908+0000", 2024-09-13T14:26:32.051 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.3049000000000002e-05 2024-09-13T14:26:32.051 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.051 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.051 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:26:32.051 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.657915+0000", 2024-09-13T14:26:32.051 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.742e-06 2024-09-13T14:26:32.051 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.051 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.052 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:26:32.052 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.657920+0000", 2024-09-13T14:26:32.052 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.1830000000000002e-06 2024-09-13T14:26:32.052 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.052 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.052 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:26:32.052 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.657966+0000", 2024-09-13T14:26:32.052 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.6524999999999999e-05 2024-09-13T14:26:32.052 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.052 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.052 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:26:32.052 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.657989+0000", 2024-09-13T14:26:32.052 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.2189e-05 2024-09-13T14:26:32.053 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.053 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.053 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:26:32.053 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.663430+0000", 2024-09-13T14:26:32.053 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0054412380000000002 2024-09-13T14:26:32.053 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.053 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.053 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:26:32.053 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.663473+0000", 2024-09-13T14:26:32.053 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.3470999999999998e-05 2024-09-13T14:26:32.053 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:26:32.053 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:26:32.053 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:26:32.054 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.054 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.054 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.44546.0:3134 5.0 e119/116 5:05d64f6b:::benchmark_data_smithi102_155989_object3133:head v 119'171, mlcod=119'169)", 2024-09-13T14:26:32.054 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:26:31.887682+0000", 2024-09-13T14:26:32.054 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.055582906000000001, 2024-09-13T14:26:32.054 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0076814420000000001, 2024-09-13T14:26:32.054 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:26:32.054 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:26:32.054 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:26:32.054 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:26:32.054 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.054 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:26:32.054 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.887682+0000", 2024-09-13T14:26:32.054 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:26:32.054 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.055 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.055 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:26:32.055 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.887682+0000", 2024-09-13T14:26:32.055 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:26:32.055 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.055 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.055 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:26:32.055 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.887682+0000", 2024-09-13T14:26:32.055 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.0299999999999997e-07 2024-09-13T14:26:32.055 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.055 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.055 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:26:32.055 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.887706+0000", 2024-09-13T14:26:32.055 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.3638999999999998e-05 2024-09-13T14:26:32.056 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.056 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.056 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:26:32.056 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.887722+0000", 2024-09-13T14:26:32.056 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.5940999999999999e-05 2024-09-13T14:26:32.056 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.056 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.056 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:26:32.056 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.887730+0000", 2024-09-13T14:26:32.056 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.6229999999999997e-06 2024-09-13T14:26:32.056 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.056 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.056 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:26:32.057 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.889507+0000", 2024-09-13T14:26:32.057 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0017773750000000001 2024-09-13T14:26:32.057 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.057 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.057 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:26:32.057 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.889532+0000", 2024-09-13T14:26:32.057 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.4879e-05 2024-09-13T14:26:32.057 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.057 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.057 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:26:32.057 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.895318+0000", 2024-09-13T14:26:32.057 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0057858010000000001 2024-09-13T14:26:32.057 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.057 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.057 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:26:32.058 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.895363+0000", 2024-09-13T14:26:32.058 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.5281000000000002e-05 2024-09-13T14:26:32.058 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:26:32.058 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:26:32.058 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:26:32.058 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.058 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.058 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.44546.0:3133 5.8 e119/116 5:10b0400b:::benchmark_data_smithi102_155989_object3132:head v 119'201, mlcod=119'199)", 2024-09-13T14:26:32.058 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:26:31.889571+0000", 2024-09-13T14:26:32.058 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.053693025999999998, 2024-09-13T14:26:32.058 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0058189920000000003, 2024-09-13T14:26:32.058 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:26:32.058 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:26:32.059 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:26:32.059 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:26:32.059 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.059 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:26:32.059 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.889571+0000", 2024-09-13T14:26:32.059 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:26:32.059 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.059 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.059 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:26:32.059 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.889571+0000", 2024-09-13T14:26:32.059 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:26:32.059 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.059 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.059 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:26:32.060 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.889572+0000", 2024-09-13T14:26:32.060 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.9899999999999999e-07 2024-09-13T14:26:32.060 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.060 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.060 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:26:32.060 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.889627+0000", 2024-09-13T14:26:32.060 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.4373000000000003e-05 2024-09-13T14:26:32.060 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.060 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.060 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:26:32.060 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.889643+0000", 2024-09-13T14:26:32.060 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.6296000000000001e-05 2024-09-13T14:26:32.060 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.060 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.061 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:26:32.061 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.889650+0000", 2024-09-13T14:26:32.061 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.8970000000000002e-06 2024-09-13T14:26:32.061 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.061 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.061 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:26:32.061 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.890069+0000", 2024-09-13T14:26:32.061 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000419634 2024-09-13T14:26:32.061 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.061 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.061 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:26:32.061 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.890111+0000", 2024-09-13T14:26:32.061 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.1421000000000002e-05 2024-09-13T14:26:32.062 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.062 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.062 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:26:32.062 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.895366+0000", 2024-09-13T14:26:32.062 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0052548170000000002 2024-09-13T14:26:32.062 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.062 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.062 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:26:32.062 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.895390+0000", 2024-09-13T14:26:32.062 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.4754999999999999e-05 2024-09-13T14:26:32.062 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:26:32.062 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:26:32.062 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:26:32.063 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.063 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.063 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:3195 5.3 5:cc4bbf53:::benchmark_data_smithi102_155989_object3194:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T14:26:32.063 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:26:31.910539+0000", 2024-09-13T14:26:32.063 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.032725298999999999, 2024-09-13T14:26:32.063 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0057673769999999997, 2024-09-13T14:26:32.063 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:26:32.063 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:26:32.063 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:26:32.063 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:26:32.063 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:26:32.063 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:26:32.063 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 3195 2024-09-13T14:26:32.063 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.064 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:26:32.064 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.064 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:26:32.064 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.910539+0000", 2024-09-13T14:26:32.064 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:26:32.064 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.064 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.064 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:26:32.064 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.910539+0000", 2024-09-13T14:26:32.064 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:26:32.064 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.064 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.064 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:26:32.064 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.910540+0000", 2024-09-13T14:26:32.065 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.2099999999999997e-07 2024-09-13T14:26:32.065 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.065 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.065 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:26:32.065 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.910559+0000", 2024-09-13T14:26:32.065 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.9137999999999999e-05 2024-09-13T14:26:32.065 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.065 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.065 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:26:32.065 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.910566+0000", 2024-09-13T14:26:32.065 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.486e-06 2024-09-13T14:26:32.065 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.065 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.065 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:26:32.066 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.910572+0000", 2024-09-13T14:26:32.066 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.5230000000000003e-06 2024-09-13T14:26:32.066 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.066 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.066 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:26:32.066 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.911615+0000", 2024-09-13T14:26:32.066 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0010435609999999999 2024-09-13T14:26:32.066 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.066 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.066 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:26:32.066 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.911744+0000", 2024-09-13T14:26:32.066 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00012862600000000001 2024-09-13T14:26:32.066 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.067 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.067 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:26:32.067 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.911898+0000", 2024-09-13T14:26:32.067 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00015360800000000001 2024-09-13T14:26:32.067 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.067 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.067 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:26:32.067 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.913108+0000", 2024-09-13T14:26:32.067 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.001210348 2024-09-13T14:26:32.067 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.067 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.067 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:26:32.067 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.913292+0000", 2024-09-13T14:26:32.067 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00018455199999999999 2024-09-13T14:26:32.068 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.068 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.068 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:26:32.068 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.916193+0000", 2024-09-13T14:26:32.068 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.002900967 2024-09-13T14:26:32.068 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.068 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.068 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:26:32.068 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.916251+0000", 2024-09-13T14:26:32.068 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.7249e-05 2024-09-13T14:26:32.068 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:26:32.068 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:26:32.068 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:26:32.069 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:26:31.916307+0000", 2024-09-13T14:26:32.069 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.5797999999999997e-05 2024-09-13T14:26:32.069 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:26:32.069 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:26:32.069 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:26:32.069 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:26:32.069 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:26:32.069 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:26:32.069 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:26:32.104 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:32.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:32.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:26:32.118 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 1 16 3967 3951 246.924 246.938 0.00439071 0.00403858 2024-09-13T14:26:32.205 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:26:32.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:32.239 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:32.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:26:32.340 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:32.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:32.378 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:32.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:32.475 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:26:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=sqlstore.transactions t=2024-09-13T14:26:32.208164239Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-13T14:26:32.475 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:26:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=cleanup t=2024-09-13T14:26:32.213977856Z level=info msg="Completed cleanup jobs" duration=16.43129ms 2024-09-13T14:26:32.475 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:26:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=sqlstore.transactions t=2024-09-13T14:26:32.219848456Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-09-13T14:26:32.476 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:26:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=grafana.update.checker t=2024-09-13T14:26:32.336459326Z level=info msg="Update check succeeded" duration=60.387287ms 2024-09-13T14:26:32.476 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:26:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=plugins.update.checker t=2024-09-13T14:26:32.338633457Z level=info msg="Update check succeeded" duration=60.252499ms 2024-09-13T14:26:32.479 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:26:32.512 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:32.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:32.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:26:32.614 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:26:32.650 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:32.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:32.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:26:32.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:32 smithi195 ceph-mon[104262]: pgmap v305: 89 pgs: 89 active+clean; 12 MiB data, 221 MiB used, 626 GiB / 626 GiB avail; 931 KiB/s wr, 14 op/s 2024-09-13T14:26:32.751 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:32.787 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:32.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:32.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:32.889 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:26:32.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:32 smithi102 ceph-mon[110183]: pgmap v305: 89 pgs: 89 active+clean; 12 MiB data, 221 MiB used, 626 GiB / 626 GiB avail; 931 KiB/s wr, 14 op/s 2024-09-13T14:26:32.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:32 smithi102 ceph-mon[115269]: pgmap v305: 89 pgs: 89 active+clean; 12 MiB data, 221 MiB used, 626 GiB / 626 GiB avail; 931 KiB/s wr, 14 op/s 2024-09-13T14:26:32.925 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:32.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:32.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:26:33.026 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:26:33.062 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:33.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:33.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:26:33.118 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 2 16 7907 7891 246.577 246.25 0.00440109 0.00404886 2024-09-13T14:26:33.162 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T14:26:33.163 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:33.198 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:33.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:33.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:33.300 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:33.336 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:33.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:33.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:33.409 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:26:33.409 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:26:33.438 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:26:33.476 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:33.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:33.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:26:33.578 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:26:33.617 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:33.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:33.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:26:33.720 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:26:33.758 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:33.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:33.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:26:33.860 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:33.896 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:33.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:33.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:33.997 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:26:34.035 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:34.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:34.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:26:34.118 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 3 16 11335 11319 235.795 214.25 0.00523751 0.00423482 2024-09-13T14:26:34.137 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:26:34.172 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:34.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:34.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:26:34.274 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:26:34.306 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:34.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:34.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:26:34.408 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:26:34.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:34.441 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:34.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:26:34.491 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is set 2024-09-13T14:26:34.543 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:26:34.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:34.578 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:34.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:26:34.680 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:26:34.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:34.713 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:34.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:26:34.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:34 smithi195 ceph-mon[104262]: pgmap v306: 89 pgs: 89 active+clean; 33 MiB data, 239 MiB used, 626 GiB / 626 GiB avail; 2.8 MiB/s wr, 45 op/s 2024-09-13T14:26:34.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:34 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3434249894' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:26:34.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:34 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:26:34.814 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:26:34.849 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:34.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:34.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:26:34.855 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:34 smithi102 ceph-mon[110183]: pgmap v306: 89 pgs: 89 active+clean; 33 MiB data, 239 MiB used, 626 GiB / 626 GiB avail; 2.8 MiB/s wr, 45 op/s 2024-09-13T14:26:34.855 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:34 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3434249894' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:26:34.855 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:34 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:26:34.855 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:34 smithi102 ceph-mon[115269]: pgmap v306: 89 pgs: 89 active+clean; 33 MiB data, 239 MiB used, 626 GiB / 626 GiB avail; 2.8 MiB/s wr, 45 op/s 2024-09-13T14:26:34.855 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:34 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3434249894' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:26:34.855 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:34 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:26:34.951 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:34.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:34.986 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:34.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:35.087 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:35.118 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 4 16 14744 14728 230.107 213.062 0.00574411 0.00433968 2024-09-13T14:26:35.123 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:35.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:35.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:35.225 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:26:35.255 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:35.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:35.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:26:35.357 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:26:35.387 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:35.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:35.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:26:35.488 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:26:35.521 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:35.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:35.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:26:35.623 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:35.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:35.658 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:35.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:35.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:35 smithi102 ceph-mon[110183]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:26:35.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:35 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:26:35.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:35 smithi102 ceph-mon[110183]: osdmap e120: 8 total, 8 up, 7 in 2024-09-13T14:26:35.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:35 smithi102 ceph-mon[115269]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:26:35.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:35 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:26:35.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:35 smithi102 ceph-mon[115269]: osdmap e120: 8 total, 8 up, 7 in 2024-09-13T14:26:35.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:35 smithi195 ceph-mon[104262]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:26:35.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:35 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:26:35.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:35 smithi195 ceph-mon[104262]: osdmap e120: 8 total, 8 up, 7 in 2024-09-13T14:26:35.760 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:26:35.797 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:35.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:35.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:26:35.899 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:26:35.931 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:35.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:35.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:26:36.033 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:26:36.071 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:36.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:36.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:26:36.118 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 5 16 17888 17872 223.382 196.5 0.00299934 0.0044714 2024-09-13T14:26:36.173 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:26:36.197 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:26:36.208 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:36.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:36.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:26:36.310 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:26:36.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:36.343 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:36.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:26:36.444 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:26:36.473 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:36.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:36.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:26:36.493 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:26:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:26:36] "GET /metrics HTTP/1.1" 200 36455 "" "Prometheus/2.51.0" 2024-09-13T14:26:36.576 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:26:36.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:36.610 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:36.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:26:36.711 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:26:36.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:36 smithi195 ceph-mon[104262]: pgmap v308: 89 pgs: 89 active+clean; 67 MiB data, 330 MiB used, 625 GiB / 626 GiB avail; 7.8 MiB/s wr, 124 op/s 2024-09-13T14:26:36.749 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:36.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:36.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:26:36.819 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:36 smithi102 ceph-mon[115269]: pgmap v308: 89 pgs: 89 active+clean; 67 MiB data, 330 MiB used, 625 GiB / 626 GiB avail; 7.8 MiB/s wr, 124 op/s 2024-09-13T14:26:36.819 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:36 smithi102 ceph-mon[110183]: pgmap v308: 89 pgs: 89 active+clean; 67 MiB data, 330 MiB used, 625 GiB / 626 GiB avail; 7.8 MiB/s wr, 124 op/s 2024-09-13T14:26:36.851 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:26:36.885 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:36.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:36.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:26:36.987 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:37.022 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:37.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:37.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:37.118 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 6 16 20783 20767 216.306 180.938 0.00627205 0.00461698 2024-09-13T14:26:37.124 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:26:37.158 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:37.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:37.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:26:37.260 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:37.296 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:37.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:37.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:37.390 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:26:37.391 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:26:37.398 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:26:37.412 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:26:37.412 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:26:37.433 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:37.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:37.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:26:37.446 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:26:37.446 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:26:37.468 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:26:37.469 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:26:37.478 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:26:37.479 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:26:37.518 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:26:37.519 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:26:37.535 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:26:37.540 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:26:37.540 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:26:37.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:37.569 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:37.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:26:37.569 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:26:37.569 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:26:37.598 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:26:37.598 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:26:37.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:37 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1405672976' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:26:37.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:37 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1405672976' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:26:37.670 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:26:37.692 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:37.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:37.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:26:37.794 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:26:37.827 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:37.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:37.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:26:37.929 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:26:37.962 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:37.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:37.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:26:37.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:37 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1405672976' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:26:38.063 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:26:38.102 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:38.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:38.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:26:38.118 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 7 16 23153 23137 206.564 148.125 0.00539306 0.00483586 2024-09-13T14:26:38.204 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:26:38.236 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:38.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:38.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:26:38.338 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:26:38.374 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:38.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:38.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:26:38.476 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:26:38.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:38.509 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:38.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:26:38.610 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:26:38.646 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:38.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:38.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:26:38.747 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:26:38.782 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:38.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:38.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:26:38.818 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:38 smithi102 ceph-mon[110183]: pgmap v309: 89 pgs: 89 active+clean; 1.1 GiB data, 3.3 GiB used, 623 GiB / 626 GiB avail; 138 MiB/s wr, 2.20k op/s 2024-09-13T14:26:38.818 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:38 smithi102 ceph-mon[110183]: from='client.34635 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:26:38.818 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:38 smithi102 ceph-mon[115269]: pgmap v309: 89 pgs: 89 active+clean; 1.1 GiB data, 3.3 GiB used, 623 GiB / 626 GiB avail; 138 MiB/s wr, 2.20k op/s 2024-09-13T14:26:38.818 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:38 smithi102 ceph-mon[115269]: from='client.34635 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:26:38.884 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:26:38.917 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:38.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:38.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:26:38.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:38 smithi195 ceph-mon[104262]: pgmap v309: 89 pgs: 89 active+clean; 1.1 GiB data, 3.3 GiB used, 623 GiB / 626 GiB avail; 138 MiB/s wr, 2.20k op/s 2024-09-13T14:26:38.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:38 smithi195 ceph-mon[104262]: from='client.34635 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:26:39.019 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:39.055 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:39.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:39.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:39.118 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 8 16 25930 25914 202.436 173.562 0.00839873 0.00493462 2024-09-13T14:26:39.157 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:26:39.188 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:39.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:39.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:26:39.291 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:26:39.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:39.322 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:39.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:26:39.423 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:26:39.459 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:39.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:39.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:26:39.561 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:26:39.591 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:39.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:39.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:26:39.693 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:26:39.722 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:39.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:39.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:26:39.824 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:26:39.856 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:39.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:39.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:26:39.958 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:39.996 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:39.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:39.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:40.098 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:26:40.118 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 9 16 28456 28440 197.483 157.875 0.00450052 0.00503577 2024-09-13T14:26:40.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:40.120 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:40.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:26:40.221 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:26:40.254 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:40.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:40.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:26:40.356 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:26:40.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:40.393 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:40.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:26:40.494 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:26:40.519 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:40.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:40.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:26:40.621 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:26:40.646 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:40.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:40.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:26:40.748 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:26:40.775 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:40.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:40.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:26:40.876 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:26:40.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:40 smithi102 ceph-mon[110183]: pgmap v310: 89 pgs: 89 active+clean; 1.1 GiB data, 3.5 GiB used, 622 GiB / 626 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-09-13T14:26:40.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:40 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:26:40.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:40 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:26:40.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:40 smithi102 ceph-mon[115269]: pgmap v310: 89 pgs: 89 active+clean; 1.1 GiB data, 3.5 GiB used, 622 GiB / 626 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-09-13T14:26:40.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:40 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:26:40.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:40 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:26:40.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:40.920 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:40.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:26:40.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:40 smithi195 ceph-mon[104262]: pgmap v310: 89 pgs: 89 active+clean; 1.1 GiB data, 3.5 GiB used, 622 GiB / 626 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-09-13T14:26:40.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:40 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:26:40.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:40 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:26:41.021 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:26:41.055 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:41.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:41.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:26:41.119 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 10 16 30583 30567 191.028 132.938 0.00418477 0.00522995 2024-09-13T14:26:41.157 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:26:41.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:41.195 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:41.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:26:41.296 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:26:41.330 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:41.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:41.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:26:41.433 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:26:41.466 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:41.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:41.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:26:41.568 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:26:41.589 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:41.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:41.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:26:41.690 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:26:41.729 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:41.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:41.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:26:41.831 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:26:41.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:41.861 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:41.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:26:41.962 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:26:42.002 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:42.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:42.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:26:42.104 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:42.119 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 11 16 32693 32677 185.649 131.875 0.0121767 0.00538179 2024-09-13T14:26:42.140 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:42.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:42.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:42.223 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":309,"stamp":"2024-09-13T14:26:37.244072+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1160320997,"num_objects":17660,"num_object_clones":0,"num_object_copies":52980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17660,"num_whiteouts":0,"num_read":6390,"num_read_kb":13845,"num_write":18755,"num_write_kb":1143489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":5965427,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18814,"ondisk_log_size":18814,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":3443916,"kb_used_data":3212100,"kb_used_omap":27,"kb_used_meta":231716,"kb_avail":652772148,"statfs":{"total":671965249536,"available":668438679552,"internally_reserved":0,"allocated":3289190400,"data_stored":3284727066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28294,"internal_metadata":237277562},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[10,4,4],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1155530752,"num_objects":17632,"num_object_clones":0,"num_object_copies":52896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17632,"num_write_kb":1128448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002328"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":99,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.965745+0000","last_change":"2024-09-13T14:26:15.823152+0000","last_active":"2024-09-13T14:26:29.965745+0000","last_peered":"2024-09-13T14:26:29.965745+0000","last_clean":"2024-09-13T14:26:29.965745+0000","last_became_active":"2024-09-13T14:26:15.822686+0000","last_became_peered":"2024-09-13T14:26:15.822686+0000","last_unstale":"2024-09-13T14:26:29.965745+0000","last_undegraded":"2024-09-13T14:26:29.965745+0000","last_fullsized":"2024-09-13T14:26:29.965745+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:18:50.765530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":85,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.477027+0000","last_change":"2024-09-13T14:24:23.773702+0000","last_active":"2024-09-13T14:26:29.477027+0000","last_peered":"2024-09-13T14:26:29.477027+0000","last_clean":"2024-09-13T14:26:29.477027+0000","last_became_active":"2024-09-13T14:24:23.773290+0000","last_became_peered":"2024-09-13T14:24:23.773290+0000","last_unstale":"2024-09-13T14:26:29.477027+0000","last_undegraded":"2024-09-13T14:26:29.477027+0000","last_fullsized":"2024-09-13T14:26:29.477027+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:20.820970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":82,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.965688+0000","last_change":"2024-09-13T14:24:55.282407+0000","last_active":"2024-09-13T14:26:29.965688+0000","last_peered":"2024-09-13T14:26:29.965688+0000","last_clean":"2024-09-13T14:26:29.965688+0000","last_became_active":"2024-09-13T14:24:55.282083+0000","last_became_peered":"2024-09-13T14:24:55.282083+0000","last_unstale":"2024-09-13T14:26:29.965688+0000","last_undegraded":"2024-09-13T14:26:29.965688+0000","last_fullsized":"2024-09-13T14:26:29.965688+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:23:10.525707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":85,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T14:26:34.495522+0000","last_change":"2024-09-13T14:24:55.284433+0000","last_active":"2024-09-13T14:26:34.495522+0000","last_peered":"2024-09-13T14:26:34.495522+0000","last_clean":"2024-09-13T14:26:34.495522+0000","last_became_active":"2024-09-13T14:24:55.284143+0000","last_became_peered":"2024-09-13T14:24:55.284143+0000","last_unstale":"2024-09-13T14:26:34.495522+0000","last_undegraded":"2024-09-13T14:26:34.495522+0000","last_fullsized":"2024-09-13T14:26:34.495522+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:28.107405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":87,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T14:26:34.496343+0000","last_change":"2024-09-13T14:24:55.282416+0000","last_active":"2024-09-13T14:26:34.496343+0000","last_peered":"2024-09-13T14:26:34.496343+0000","last_clean":"2024-09-13T14:26:34.496343+0000","last_became_active":"2024-09-13T14:24:55.282045+0000","last_became_peered":"2024-09-13T14:24:55.282045+0000","last_unstale":"2024-09-13T14:26:34.496343+0000","last_undegraded":"2024-09-13T14:26:34.496343+0000","last_fullsized":"2024-09-13T14:26:34.496343+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:58.692897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":68,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.476987+0000","last_change":"2024-09-13T14:11:29.244452+0000","last_active":"2024-09-13T14:26:29.476987+0000","last_peered":"2024-09-13T14:26:29.476987+0000","last_clean":"2024-09-13T14:26:29.476987+0000","last_became_active":"2024-09-13T14:11:29.244102+0000","last_became_peered":"2024-09-13T14:11:29.244102+0000","last_unstale":"2024-09-13T14:26:29.476987+0000","last_undegraded":"2024-09-13T14:26:29.476987+0000","last_fullsized":"2024-09-13T14:26:29.476987+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:13:19.165534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":99,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.965920+0000","last_change":"2024-09-13T14:26:15.822031+0000","last_active":"2024-09-13T14:26:29.965920+0000","last_peered":"2024-09-13T14:26:29.965920+0000","last_clean":"2024-09-13T14:26:29.965920+0000","last_became_active":"2024-09-13T14:26:15.821699+0000","last_became_peered":"2024-09-13T14:26:15.821699+0000","last_unstale":"2024-09-13T14:26:29.965920+0000","last_undegraded":"2024-09-13T14:26:29.965920+0000","last_fullsized":"2024-09-13T14:26:29.965920+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:53:53.561285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":91,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.629768+0000","last_change":"2024-09-13T14:23:18.039061+0000","last_active":"2024-09-13T14:26:29.629768+0000","last_peered":"2024-09-13T14:26:29.629768+0000","last_clean":"2024-09-13T14:26:29.629768+0000","last_became_active":"2024-09-13T14:23:18.038661+0000","last_became_peered":"2024-09-13T14:23:18.038661+0000","last_unstale":"2024-09-13T14:26:29.629768+0000","last_undegraded":"2024-09-13T14:26:29.629768+0000","last_fullsized":"2024-09-13T14:26:29.629768+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:05:29.109844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.9","version":"120'1150","reported_seq":1162,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T14:26:36.169907+0000","last_change":"2024-09-13T14:26:21.865664+0000","last_active":"2024-09-13T14:26:36.169907+0000","last_peered":"2024-09-13T14:26:36.169907+0000","last_clean":"2024-09-13T14:26:36.169907+0000","last_became_active":"2024-09-13T14:26:21.865396+0000","last_became_peered":"2024-09-13T14:26:21.865396+0000","last_unstale":"2024-09-13T14:26:36.169907+0000","last_undegraded":"2024-09-13T14:26:36.169907+0000","last_fullsized":"2024-09-13T14:26:36.169907+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":1150,"log_dups_size":0,"ondisk_log_size":1150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:54:35.654265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75366400,"num_objects":1150,"num_object_clones":0,"num_object_copies":3450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1150,"num_write_kb":73600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":84,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.477458+0000","last_change":"2024-09-13T14:23:18.038597+0000","last_active":"2024-09-13T14:26:29.477458+0000","last_peered":"2024-09-13T14:26:29.477458+0000","last_clean":"2024-09-13T14:26:29.477458+0000","last_became_active":"2024-09-13T14:23:18.038240+0000","last_became_peered":"2024-09-13T14:23:18.038240+0000","last_unstale":"2024-09-13T14:26:29.477458+0000","last_undegraded":"2024-09-13T14:26:29.477458+0000","last_fullsized":"2024-09-13T14:26:29.477458+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:27:51.842195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":67,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.965155+0000","last_change":"2024-09-13T14:11:30.254534+0000","last_active":"2024-09-13T14:26:29.965155+0000","last_peered":"2024-09-13T14:26:29.965155+0000","last_clean":"2024-09-13T14:26:29.965155+0000","last_became_active":"2024-09-13T14:11:30.253959+0000","last_became_peered":"2024-09-13T14:11:30.253959+0000","last_unstale":"2024-09-13T14:26:29.965155+0000","last_undegraded":"2024-09-13T14:26:29.965155+0000","last_fullsized":"2024-09-13T14:26:29.965155+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:09:54.025853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"120'1197","reported_seq":1216,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T14:26:36.387527+0000","last_change":"2024-09-13T14:26:21.862194+0000","last_active":"2024-09-13T14:26:36.387527+0000","last_peered":"2024-09-13T14:26:36.387527+0000","last_clean":"2024-09-13T14:26:36.387527+0000","last_became_active":"2024-09-13T14:26:21.861896+0000","last_became_peered":"2024-09-13T14:26:21.861896+0000","last_unstale":"2024-09-13T14:26:36.387527+0000","last_undegraded":"2024-09-13T14:26:36.387527+0000","last_fullsized":"2024-09-13T14:26:36.387527+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":1197,"log_dups_size":0,"ondisk_log_size":1197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:06:16.702795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78446592,"num_objects":1197,"num_object_clones":0,"num_object_copies":3591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1197,"num_write_kb":76608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":98,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.477455+0000","last_change":"2024-09-13T14:26:15.828529+0000","last_active":"2024-09-13T14:26:29.477455+0000","last_peered":"2024-09-13T14:26:29.477455+0000","last_clean":"2024-09-13T14:26:29.477455+0000","last_became_active":"2024-09-13T14:26:15.820486+0000","last_became_peered":"2024-09-13T14:26:15.820486+0000","last_unstale":"2024-09-13T14:26:29.477455+0000","last_undegraded":"2024-09-13T14:26:29.477455+0000","last_fullsized":"2024-09-13T14:26:29.477455+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:11:46.283172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":105,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.422088+0000","last_change":"2024-09-13T14:26:15.824632+0000","last_active":"2024-09-13T14:26:29.422088+0000","last_peered":"2024-09-13T14:26:29.422088+0000","last_clean":"2024-09-13T14:26:29.422088+0000","last_became_active":"2024-09-13T14:26:15.824330+0000","last_became_peered":"2024-09-13T14:26:15.824330+0000","last_unstale":"2024-09-13T14:26:29.422088+0000","last_undegraded":"2024-09-13T14:26:29.422088+0000","last_fullsized":"2024-09-13T14:26:29.422088+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:25.288116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"120'1181","reported_seq":1200,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T14:26:36.382405+0000","last_change":"2024-09-13T14:26:21.862430+0000","last_active":"2024-09-13T14:26:36.382405+0000","last_peered":"2024-09-13T14:26:36.382405+0000","last_clean":"2024-09-13T14:26:36.382405+0000","last_became_active":"2024-09-13T14:26:21.861921+0000","last_became_peered":"2024-09-13T14:26:21.861921+0000","last_unstale":"2024-09-13T14:26:36.382405+0000","last_undegraded":"2024-09-13T14:26:36.382405+0000","last_fullsized":"2024-09-13T14:26:36.382405+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":1181,"log_dups_size":0,"ondisk_log_size":1181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:08:47.216775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77398016,"num_objects":1181,"num_object_clones":0,"num_object_copies":3543,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1181,"num_write_kb":75584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":69,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.477340+0000","last_change":"2024-09-13T14:11:29.232897+0000","last_active":"2024-09-13T14:26:29.477340+0000","last_peered":"2024-09-13T14:26:29.477340+0000","last_clean":"2024-09-13T14:26:29.477340+0000","last_became_active":"2024-09-13T14:11:29.232637+0000","last_became_peered":"2024-09-13T14:11:29.232637+0000","last_unstale":"2024-09-13T14:26:29.477340+0000","last_undegraded":"2024-09-13T14:26:29.477340+0000","last_fullsized":"2024-09-13T14:26:29.477340+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:31:05.856046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":91,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.630001+0000","last_change":"2024-09-13T14:26:15.822452+0000","last_active":"2024-09-13T14:26:29.630001+0000","last_peered":"2024-09-13T14:26:29.630001+0000","last_clean":"2024-09-13T14:26:29.630001+0000","last_became_active":"2024-09-13T14:26:15.822111+0000","last_became_peered":"2024-09-13T14:26:15.822111+0000","last_unstale":"2024-09-13T14:26:29.630001+0000","last_undegraded":"2024-09-13T14:26:29.630001+0000","last_fullsized":"2024-09-13T14:26:29.630001+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:32:53.078767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"120'1199","reported_seq":1218,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T14:26:36.361944+0000","last_change":"2024-09-13T14:26:21.865623+0000","last_active":"2024-09-13T14:26:36.361944+0000","last_peered":"2024-09-13T14:26:36.361944+0000","last_clean":"2024-09-13T14:26:36.361944+0000","last_became_active":"2024-09-13T14:26:21.865252+0000","last_became_peered":"2024-09-13T14:26:21.865252+0000","last_unstale":"2024-09-13T14:26:36.361944+0000","last_undegraded":"2024-09-13T14:26:36.361944+0000","last_fullsized":"2024-09-13T14:26:36.361944+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":1199,"log_dups_size":0,"ondisk_log_size":1199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:55:20.893985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78577664,"num_objects":1199,"num_object_clones":0,"num_object_copies":3597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1199,"num_write_kb":76736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":83,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.965616+0000","last_change":"2024-09-13T14:24:55.290798+0000","last_active":"2024-09-13T14:26:29.965616+0000","last_peered":"2024-09-13T14:26:29.965616+0000","last_clean":"2024-09-13T14:26:29.965616+0000","last_became_active":"2024-09-13T14:24:55.290510+0000","last_became_peered":"2024-09-13T14:24:55.290510+0000","last_unstale":"2024-09-13T14:26:29.965616+0000","last_undegraded":"2024-09-13T14:26:29.965616+0000","last_fullsized":"2024-09-13T14:26:29.965616+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:19:03.129132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":94,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T14:26:34.496489+0000","last_change":"2024-09-13T14:26:15.823316+0000","last_active":"2024-09-13T14:26:34.496489+0000","last_peered":"2024-09-13T14:26:34.496489+0000","last_clean":"2024-09-13T14:26:34.496489+0000","last_became_active":"2024-09-13T14:26:15.822980+0000","last_became_peered":"2024-09-13T14:26:15.822980+0000","last_unstale":"2024-09-13T14:26:34.496489+0000","last_undegraded":"2024-09-13T14:26:34.496489+0000","last_fullsized":"2024-09-13T14:26:34.496489+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:33:47.420548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"120'1155","reported_seq":1174,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T14:26:36.378435+0000","last_change":"2024-09-13T14:26:21.863411+0000","last_active":"2024-09-13T14:26:36.378435+0000","last_peered":"2024-09-13T14:26:36.378435+0000","last_clean":"2024-09-13T14:26:36.378435+0000","last_became_active":"2024-09-13T14:26:21.863116+0000","last_became_peered":"2024-09-13T14:26:21.863116+0000","last_unstale":"2024-09-13T14:26:36.378435+0000","last_undegraded":"2024-09-13T14:26:36.378435+0000","last_fullsized":"2024-09-13T14:26:36.378435+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":1155,"log_dups_size":0,"ondisk_log_size":1155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:15:32.062951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75694080,"num_objects":1155,"num_object_clones":0,"num_object_copies":3465,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1155,"num_write_kb":73920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":90,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.965789+0000","last_change":"2024-09-13T14:24:55.284499+0000","last_active":"2024-09-13T14:26:29.965789+0000","last_peered":"2024-09-13T14:26:29.965789+0000","last_clean":"2024-09-13T14:26:29.965789+0000","last_became_active":"2024-09-13T14:24:55.284096+0000","last_became_peered":"2024-09-13T14:24:55.284096+0000","last_unstale":"2024-09-13T14:26:29.965789+0000","last_undegraded":"2024-09-13T14:26:29.965789+0000","last_fullsized":"2024-09-13T14:26:29.965789+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:02:06.836099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":81,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.422291+0000","last_change":"2024-09-13T14:24:55.291371+0000","last_active":"2024-09-13T14:26:29.422291+0000","last_peered":"2024-09-13T14:26:29.422291+0000","last_clean":"2024-09-13T14:26:29.422291+0000","last_became_active":"2024-09-13T14:24:55.291047+0000","last_became_peered":"2024-09-13T14:24:55.291047+0000","last_unstale":"2024-09-13T14:26:29.422291+0000","last_undegraded":"2024-09-13T14:26:29.422291+0000","last_fullsized":"2024-09-13T14:26:29.422291+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:43.912412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"120'1102","reported_seq":1115,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T14:26:36.174701+0000","last_change":"2024-09-13T14:26:21.864062+0000","last_active":"2024-09-13T14:26:36.174701+0000","last_peered":"2024-09-13T14:26:36.174701+0000","last_clean":"2024-09-13T14:26:36.174701+0000","last_became_active":"2024-09-13T14:26:21.863721+0000","last_became_peered":"2024-09-13T14:26:21.863721+0000","last_unstale":"2024-09-13T14:26:36.174701+0000","last_undegraded":"2024-09-13T14:26:36.174701+0000","last_fullsized":"2024-09-13T14:26:36.174701+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":1102,"log_dups_size":0,"ondisk_log_size":1102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:15:42.744309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72220672,"num_objects":1102,"num_object_clones":0,"num_object_copies":3306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1102,"num_write_kb":70528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":92,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.630125+0000","last_change":"2024-09-13T14:26:15.827333+0000","last_active":"2024-09-13T14:26:29.630125+0000","last_peered":"2024-09-13T14:26:29.630125+0000","last_clean":"2024-09-13T14:26:29.630125+0000","last_became_active":"2024-09-13T14:26:15.826945+0000","last_became_peered":"2024-09-13T14:26:15.826945+0000","last_unstale":"2024-09-13T14:26:29.630125+0000","last_undegraded":"2024-09-13T14:26:29.630125+0000","last_fullsized":"2024-09-13T14:26:29.630125+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:35:24.870052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":105,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.422498+0000","last_change":"2024-09-13T14:26:15.827814+0000","last_active":"2024-09-13T14:26:29.422498+0000","last_peered":"2024-09-13T14:26:29.422498+0000","last_clean":"2024-09-13T14:26:29.422498+0000","last_became_active":"2024-09-13T14:26:15.827502+0000","last_became_peered":"2024-09-13T14:26:15.827502+0000","last_unstale":"2024-09-13T14:26:29.422498+0000","last_undegraded":"2024-09-13T14:26:29.422498+0000","last_fullsized":"2024-09-13T14:26:29.422498+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:24:22.901236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"120'1106","reported_seq":1117,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T14:26:36.187720+0000","last_change":"2024-09-13T14:26:21.867125+0000","last_active":"2024-09-13T14:26:36.187720+0000","last_peered":"2024-09-13T14:26:36.187720+0000","last_clean":"2024-09-13T14:26:36.187720+0000","last_became_active":"2024-09-13T14:26:21.866815+0000","last_became_peered":"2024-09-13T14:26:21.866815+0000","last_unstale":"2024-09-13T14:26:36.187720+0000","last_undegraded":"2024-09-13T14:26:36.187720+0000","last_fullsized":"2024-09-13T14:26:36.187720+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":1106,"log_dups_size":0,"ondisk_log_size":1106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:44.218255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72482816,"num_objects":1106,"num_object_clones":0,"num_object_copies":3318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1106,"num_write_kb":70784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":82,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.422321+0000","last_change":"2024-09-13T14:24:23.777970+0000","last_active":"2024-09-13T14:26:29.422321+0000","last_peered":"2024-09-13T14:26:29.422321+0000","last_clean":"2024-09-13T14:26:29.422321+0000","last_became_active":"2024-09-13T14:24:23.777650+0000","last_became_peered":"2024-09-13T14:24:23.777650+0000","last_unstale":"2024-09-13T14:26:29.422321+0000","last_undegraded":"2024-09-13T14:26:29.422321+0000","last_fullsized":"2024-09-13T14:26:29.422321+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:50:00.549905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":90,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.422577+0000","last_change":"2024-09-13T14:26:15.825304+0000","last_active":"2024-09-13T14:26:29.422577+0000","last_peered":"2024-09-13T14:26:29.422577+0000","last_clean":"2024-09-13T14:26:29.422577+0000","last_became_active":"2024-09-13T14:26:15.825018+0000","last_became_peered":"2024-09-13T14:26:15.825018+0000","last_unstale":"2024-09-13T14:26:29.422577+0000","last_undegraded":"2024-09-13T14:26:29.422577+0000","last_fullsized":"2024-09-13T14:26:29.422577+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:22:59.408928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"120'1087","reported_seq":1100,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T14:26:36.165756+0000","last_change":"2024-09-13T14:26:21.865199+0000","last_active":"2024-09-13T14:26:36.165756+0000","last_peered":"2024-09-13T14:26:36.165756+0000","last_clean":"2024-09-13T14:26:36.165756+0000","last_became_active":"2024-09-13T14:26:21.864903+0000","last_became_peered":"2024-09-13T14:26:21.864903+0000","last_unstale":"2024-09-13T14:26:36.165756+0000","last_undegraded":"2024-09-13T14:26:36.165756+0000","last_fullsized":"2024-09-13T14:26:36.165756+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":1087,"log_dups_size":0,"ondisk_log_size":1087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:25.014330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71237632,"num_objects":1087,"num_object_clones":0,"num_object_copies":3261,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1087,"num_write_kb":69568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":100,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.966151+0000","last_change":"2024-09-13T14:26:15.823898+0000","last_active":"2024-09-13T14:26:29.966151+0000","last_peered":"2024-09-13T14:26:29.966151+0000","last_clean":"2024-09-13T14:26:29.966151+0000","last_became_active":"2024-09-13T14:26:15.823125+0000","last_became_peered":"2024-09-13T14:26:15.823125+0000","last_unstale":"2024-09-13T14:26:29.966151+0000","last_undegraded":"2024-09-13T14:26:29.966151+0000","last_fullsized":"2024-09-13T14:26:29.966151+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:51:27.949528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":63,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.966058+0000","last_change":"2024-09-13T14:26:15.823198+0000","last_active":"2024-09-13T14:26:29.966058+0000","last_peered":"2024-09-13T14:26:29.966058+0000","last_clean":"2024-09-13T14:26:29.966058+0000","last_became_active":"2024-09-13T14:26:15.822717+0000","last_became_peered":"2024-09-13T14:26:15.822717+0000","last_unstale":"2024-09-13T14:26:29.966058+0000","last_undegraded":"2024-09-13T14:26:29.966058+0000","last_fullsized":"2024-09-13T14:26:29.966058+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:22.471622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"120'1164","reported_seq":1177,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T14:26:36.173779+0000","last_change":"2024-09-13T14:26:21.863524+0000","last_active":"2024-09-13T14:26:36.173779+0000","last_peered":"2024-09-13T14:26:36.173779+0000","last_clean":"2024-09-13T14:26:36.173779+0000","last_became_active":"2024-09-13T14:26:21.863152+0000","last_became_peered":"2024-09-13T14:26:21.863152+0000","last_unstale":"2024-09-13T14:26:36.173779+0000","last_undegraded":"2024-09-13T14:26:36.173779+0000","last_fullsized":"2024-09-13T14:26:36.173779+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":1164,"log_dups_size":0,"ondisk_log_size":1164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:10:48.419866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76283904,"num_objects":1164,"num_object_clones":0,"num_object_copies":3492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1164,"num_write_kb":74496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":414,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.476937+0000","last_change":"2024-09-13T14:23:18.032973+0000","last_active":"2024-09-13T14:26:29.476937+0000","last_peered":"2024-09-13T14:26:29.476937+0000","last_clean":"2024-09-13T14:26:29.476937+0000","last_became_active":"2024-09-13T14:23:18.032579+0000","last_became_peered":"2024-09-13T14:23:18.032579+0000","last_unstale":"2024-09-13T14:26:29.476937+0000","last_undegraded":"2024-09-13T14:26:29.476937+0000","last_fullsized":"2024-09-13T14:26:29.476937+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:07:41.927740+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:17.109445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":410,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.965562+0000","last_change":"2024-09-13T14:24:23.776868+0000","last_active":"2024-09-13T14:26:29.965562+0000","last_peered":"2024-09-13T14:26:29.965562+0000","last_clean":"2024-09-13T14:26:29.965562+0000","last_became_active":"2024-09-13T14:24:23.776306+0000","last_became_peered":"2024-09-13T14:24:23.776306+0000","last_unstale":"2024-09-13T14:26:29.965562+0000","last_undegraded":"2024-09-13T14:26:29.965562+0000","last_fullsized":"2024-09-13T14:26:29.965562+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:28:41.805918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":41,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.965808+0000","last_change":"2024-09-13T14:24:55.288058+0000","last_active":"2024-09-13T14:26:29.965808+0000","last_peered":"2024-09-13T14:26:29.965808+0000","last_clean":"2024-09-13T14:26:29.965808+0000","last_became_active":"2024-09-13T14:24:55.287601+0000","last_became_peered":"2024-09-13T14:24:55.287601+0000","last_unstale":"2024-09-13T14:26:29.965808+0000","last_undegraded":"2024-09-13T14:26:29.965808+0000","last_fullsized":"2024-09-13T14:26:29.965808+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:56:34.622696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":68,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.630301+0000","last_change":"2024-09-13T14:11:30.253749+0000","last_active":"2024-09-13T14:26:29.630301+0000","last_peered":"2024-09-13T14:26:29.630301+0000","last_clean":"2024-09-13T14:26:29.630301+0000","last_became_active":"2024-09-13T14:11:30.253499+0000","last_became_peered":"2024-09-13T14:11:30.253499+0000","last_unstale":"2024-09-13T14:26:29.630301+0000","last_undegraded":"2024-09-13T14:26:29.630301+0000","last_fullsized":"2024-09-13T14:26:29.630301+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:02:00.987335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"120'1183","reported_seq":1196,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T14:26:36.168762+0000","last_change":"2024-09-13T14:26:21.865805+0000","last_active":"2024-09-13T14:26:36.168762+0000","last_peered":"2024-09-13T14:26:36.168762+0000","last_clean":"2024-09-13T14:26:36.168762+0000","last_became_active":"2024-09-13T14:26:21.865288+0000","last_became_peered":"2024-09-13T14:26:21.865288+0000","last_unstale":"2024-09-13T14:26:36.168762+0000","last_undegraded":"2024-09-13T14:26:36.168762+0000","last_fullsized":"2024-09-13T14:26:36.168762+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":1183,"log_dups_size":0,"ondisk_log_size":1183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:46.510065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77529088,"num_objects":1183,"num_object_clones":0,"num_object_copies":3549,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1183,"num_write_kb":75712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":329,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.476849+0000","last_change":"2024-09-13T14:26:15.835178+0000","last_active":"2024-09-13T14:26:29.476849+0000","last_peered":"2024-09-13T14:26:29.476849+0000","last_clean":"2024-09-13T14:26:29.476849+0000","last_became_active":"2024-09-13T14:26:15.820283+0000","last_became_peered":"2024-09-13T14:26:15.820283+0000","last_unstale":"2024-09-13T14:26:29.476849+0000","last_undegraded":"2024-09-13T14:26:29.476849+0000","last_fullsized":"2024-09-13T14:26:29.476849+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:30.340038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":86,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T14:26:34.494579+0000","last_change":"2024-09-13T14:24:55.286576+0000","last_active":"2024-09-13T14:26:34.494579+0000","last_peered":"2024-09-13T14:26:34.494579+0000","last_clean":"2024-09-13T14:26:34.494579+0000","last_became_active":"2024-09-13T14:24:55.286284+0000","last_became_peered":"2024-09-13T14:24:55.286284+0000","last_unstale":"2024-09-13T14:26:34.494579+0000","last_undegraded":"2024-09-13T14:26:34.494579+0000","last_fullsized":"2024-09-13T14:26:34.494579+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:35.589736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":105,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T14:26:34.496592+0000","last_change":"2024-09-13T14:26:15.818402+0000","last_active":"2024-09-13T14:26:34.496592+0000","last_peered":"2024-09-13T14:26:34.496592+0000","last_clean":"2024-09-13T14:26:34.496592+0000","last_became_active":"2024-09-13T14:26:15.818097+0000","last_became_peered":"2024-09-13T14:26:15.818097+0000","last_unstale":"2024-09-13T14:26:34.496592+0000","last_undegraded":"2024-09-13T14:26:34.496592+0000","last_fullsized":"2024-09-13T14:26:34.496592+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:14:39.559258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"120'1155","reported_seq":1172,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T14:26:36.385861+0000","last_change":"2024-09-13T14:26:21.865091+0000","last_active":"2024-09-13T14:26:36.385861+0000","last_peered":"2024-09-13T14:26:36.385861+0000","last_clean":"2024-09-13T14:26:36.385861+0000","last_became_active":"2024-09-13T14:26:21.864703+0000","last_became_peered":"2024-09-13T14:26:21.864703+0000","last_unstale":"2024-09-13T14:26:36.385861+0000","last_undegraded":"2024-09-13T14:26:36.385861+0000","last_fullsized":"2024-09-13T14:26:36.385861+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":1155,"log_dups_size":0,"ondisk_log_size":1155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:50:53.132665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75694080,"num_objects":1155,"num_object_clones":0,"num_object_copies":3465,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1155,"num_write_kb":73920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":515,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T14:26:34.491945+0000","last_change":"2024-09-13T14:24:55.285288+0000","last_active":"2024-09-13T14:26:34.491945+0000","last_peered":"2024-09-13T14:26:34.491945+0000","last_clean":"2024-09-13T14:26:34.491945+0000","last_became_active":"2024-09-13T14:24:55.284733+0000","last_became_peered":"2024-09-13T14:24:55.284733+0000","last_unstale":"2024-09-13T14:26:34.491945+0000","last_undegraded":"2024-09-13T14:26:34.491945+0000","last_fullsized":"2024-09-13T14:26:34.491945+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:51:50.420796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":63,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T14:26:34.492321+0000","last_change":"2024-09-13T14:26:15.820929+0000","last_active":"2024-09-13T14:26:34.492321+0000","last_peered":"2024-09-13T14:26:34.492321+0000","last_clean":"2024-09-13T14:26:34.492321+0000","last_became_active":"2024-09-13T14:26:15.820419+0000","last_became_peered":"2024-09-13T14:26:15.820419+0000","last_unstale":"2024-09-13T14:26:34.492321+0000","last_undegraded":"2024-09-13T14:26:34.492321+0000","last_fullsized":"2024-09-13T14:26:34.492321+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:50:10.066207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":79,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.965813+0000","last_change":"2024-09-13T14:26:15.817732+0000","last_active":"2024-09-13T14:26:29.965813+0000","last_peered":"2024-09-13T14:26:29.965813+0000","last_clean":"2024-09-13T14:26:29.965813+0000","last_became_active":"2024-09-13T14:26:15.817403+0000","last_became_peered":"2024-09-13T14:26:15.817403+0000","last_unstale":"2024-09-13T14:26:29.965813+0000","last_undegraded":"2024-09-13T14:26:29.965813+0000","last_fullsized":"2024-09-13T14:26:29.965813+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:56:39.961300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"120'1080","reported_seq":1099,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T14:26:36.043648+0000","last_change":"2024-09-13T14:26:21.866279+0000","last_active":"2024-09-13T14:26:36.043648+0000","last_peered":"2024-09-13T14:26:36.043648+0000","last_clean":"2024-09-13T14:26:36.043648+0000","last_became_active":"2024-09-13T14:26:21.864793+0000","last_became_peered":"2024-09-13T14:26:21.864793+0000","last_unstale":"2024-09-13T14:26:36.043648+0000","last_undegraded":"2024-09-13T14:26:36.043648+0000","last_fullsized":"2024-09-13T14:26:36.043648+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":1080,"log_dups_size":0,"ondisk_log_size":1080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:15:15.895316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70778880,"num_objects":1080,"num_object_clones":0,"num_object_copies":3240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1080,"num_write_kb":69120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":878,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.965380+0000","last_change":"2024-09-13T14:26:18.829131+0000","last_active":"2024-09-13T14:26:29.965380+0000","last_peered":"2024-09-13T14:26:29.965380+0000","last_clean":"2024-09-13T14:26:29.965380+0000","last_became_active":"2024-09-13T14:26:18.828619+0000","last_became_peered":"2024-09-13T14:26:18.828619+0000","last_unstale":"2024-09-13T14:26:29.965380+0000","last_undegraded":"2024-09-13T14:26:29.965380+0000","last_fullsized":"2024-09-13T14:26:29.965380+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:24:37.073618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":83,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.422420+0000","last_change":"2024-09-13T14:23:18.042891+0000","last_active":"2024-09-13T14:26:29.422420+0000","last_peered":"2024-09-13T14:26:29.422420+0000","last_clean":"2024-09-13T14:26:29.422420+0000","last_became_active":"2024-09-13T14:23:18.042562+0000","last_became_peered":"2024-09-13T14:23:18.042562+0000","last_unstale":"2024-09-13T14:26:29.422420+0000","last_undegraded":"2024-09-13T14:26:29.422420+0000","last_fullsized":"2024-09-13T14:26:29.422420+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:30:42.996661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":104,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T14:26:34.492866+0000","last_change":"2024-09-13T14:26:15.819418+0000","last_active":"2024-09-13T14:26:34.492866+0000","last_peered":"2024-09-13T14:26:34.492866+0000","last_clean":"2024-09-13T14:26:34.492866+0000","last_became_active":"2024-09-13T14:26:15.819099+0000","last_became_peered":"2024-09-13T14:26:15.819099+0000","last_unstale":"2024-09-13T14:26:34.492866+0000","last_undegraded":"2024-09-13T14:26:34.492866+0000","last_fullsized":"2024-09-13T14:26:34.492866+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:39:34.432561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"120'1059","reported_seq":1078,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T14:26:36.040084+0000","last_change":"2024-09-13T14:26:21.862954+0000","last_active":"2024-09-13T14:26:36.040084+0000","last_peered":"2024-09-13T14:26:36.040084+0000","last_clean":"2024-09-13T14:26:36.040084+0000","last_became_active":"2024-09-13T14:26:21.862650+0000","last_became_peered":"2024-09-13T14:26:21.862650+0000","last_unstale":"2024-09-13T14:26:36.040084+0000","last_undegraded":"2024-09-13T14:26:36.040084+0000","last_fullsized":"2024-09-13T14:26:36.040084+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":1059,"log_dups_size":0,"ondisk_log_size":1059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:29:23.398662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69402624,"num_objects":1059,"num_object_clones":0,"num_object_copies":3177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1059,"num_write_kb":67776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":99,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.965808+0000","last_change":"2024-09-13T14:24:55.282619+0000","last_active":"2024-09-13T14:26:29.965808+0000","last_peered":"2024-09-13T14:26:29.965808+0000","last_clean":"2024-09-13T14:26:29.965808+0000","last_became_active":"2024-09-13T14:24:55.282110+0000","last_became_peered":"2024-09-13T14:24:55.282110+0000","last_unstale":"2024-09-13T14:26:29.965808+0000","last_undegraded":"2024-09-13T14:26:29.965808+0000","last_fullsized":"2024-09-13T14:26:29.965808+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:33:19.425344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":780,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T14:26:34.496253+0000","last_change":"2024-09-13T14:26:18.829803+0000","last_active":"2024-09-13T14:26:34.496253+0000","last_peered":"2024-09-13T14:26:34.496253+0000","last_clean":"2024-09-13T14:26:34.496253+0000","last_became_active":"2024-09-13T14:26:18.829006+0000","last_became_peered":"2024-09-13T14:26:18.829006+0000","last_unstale":"2024-09-13T14:26:34.496253+0000","last_undegraded":"2024-09-13T14:26:34.496253+0000","last_fullsized":"2024-09-13T14:26:34.496253+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:58.424977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":35,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.965219+0000","last_change":"2024-09-13T14:26:15.819032+0000","last_active":"2024-09-13T14:26:29.965219+0000","last_peered":"2024-09-13T14:26:29.965219+0000","last_clean":"2024-09-13T14:26:29.965219+0000","last_became_active":"2024-09-13T14:26:15.818675+0000","last_became_peered":"2024-09-13T14:26:15.818675+0000","last_unstale":"2024-09-13T14:26:29.965219+0000","last_undegraded":"2024-09-13T14:26:29.965219+0000","last_fullsized":"2024-09-13T14:26:29.965219+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:37:13.628256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"120'1151","reported_seq":1162,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T14:26:36.174125+0000","last_change":"2024-09-13T14:26:21.863343+0000","last_active":"2024-09-13T14:26:36.174125+0000","last_peered":"2024-09-13T14:26:36.174125+0000","last_clean":"2024-09-13T14:26:36.174125+0000","last_became_active":"2024-09-13T14:26:21.863057+0000","last_became_peered":"2024-09-13T14:26:21.863057+0000","last_unstale":"2024-09-13T14:26:36.174125+0000","last_undegraded":"2024-09-13T14:26:36.174125+0000","last_fullsized":"2024-09-13T14:26:36.174125+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":1151,"log_dups_size":0,"ondisk_log_size":1151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:38:25.794314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75431936,"num_objects":1151,"num_object_clones":0,"num_object_copies":3453,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1151,"num_write_kb":73664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":63,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T14:26:34.492767+0000","last_change":"2024-09-13T14:26:15.817437+0000","last_active":"2024-09-13T14:26:34.492767+0000","last_peered":"2024-09-13T14:26:34.492767+0000","last_clean":"2024-09-13T14:26:34.492767+0000","last_became_active":"2024-09-13T14:26:15.817105+0000","last_became_peered":"2024-09-13T14:26:15.817105+0000","last_unstale":"2024-09-13T14:26:34.492767+0000","last_undegraded":"2024-09-13T14:26:34.492767+0000","last_fullsized":"2024-09-13T14:26:34.492767+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:40.879808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":605,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.477377+0000","last_change":"2024-09-13T14:23:18.038847+0000","last_active":"2024-09-13T14:26:29.477377+0000","last_peered":"2024-09-13T14:26:29.477377+0000","last_clean":"2024-09-13T14:26:29.477377+0000","last_became_active":"2024-09-13T14:23:18.038188+0000","last_became_peered":"2024-09-13T14:23:18.038188+0000","last_unstale":"2024-09-13T14:26:29.477377+0000","last_undegraded":"2024-09-13T14:26:29.477377+0000","last_fullsized":"2024-09-13T14:26:29.477377+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:30:37.213496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":76,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T14:26:34.496402+0000","last_change":"2024-09-13T14:23:18.034427+0000","last_active":"2024-09-13T14:26:34.496402+0000","last_peered":"2024-09-13T14:26:34.496402+0000","last_clean":"2024-09-13T14:26:34.496402+0000","last_became_active":"2024-09-13T14:23:18.034092+0000","last_became_peered":"2024-09-13T14:23:18.034092+0000","last_unstale":"2024-09-13T14:26:34.496402+0000","last_undegraded":"2024-09-13T14:26:34.496402+0000","last_fullsized":"2024-09-13T14:26:34.496402+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:21.755115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"120'1121","reported_seq":1134,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T14:26:36.171488+0000","last_change":"2024-09-13T14:26:21.867237+0000","last_active":"2024-09-13T14:26:36.171488+0000","last_peered":"2024-09-13T14:26:36.171488+0000","last_clean":"2024-09-13T14:26:36.171488+0000","last_became_active":"2024-09-13T14:26:21.866931+0000","last_became_peered":"2024-09-13T14:26:21.866931+0000","last_unstale":"2024-09-13T14:26:36.171488+0000","last_undegraded":"2024-09-13T14:26:36.171488+0000","last_fullsized":"2024-09-13T14:26:36.171488+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":1121,"log_dups_size":0,"ondisk_log_size":1121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:52:57.569446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73465856,"num_objects":1121,"num_object_clones":0,"num_object_copies":3363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1121,"num_write_kb":71744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":90,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.966358+0000","last_change":"2024-09-13T14:24:55.290711+0000","last_active":"2024-09-13T14:26:29.966358+0000","last_peered":"2024-09-13T14:26:29.966358+0000","last_clean":"2024-09-13T14:26:29.966358+0000","last_became_active":"2024-09-13T14:24:55.288549+0000","last_became_peered":"2024-09-13T14:24:55.288549+0000","last_unstale":"2024-09-13T14:26:29.966358+0000","last_undegraded":"2024-09-13T14:26:29.966358+0000","last_fullsized":"2024-09-13T14:26:29.966358+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:55:21.763931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":519,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.630268+0000","last_change":"2024-09-13T14:26:18.827876+0000","last_active":"2024-09-13T14:26:29.630268+0000","last_peered":"2024-09-13T14:26:29.630268+0000","last_clean":"2024-09-13T14:26:29.630268+0000","last_became_active":"2024-09-13T14:26:18.827317+0000","last_became_peered":"2024-09-13T14:26:18.827317+0000","last_unstale":"2024-09-13T14:26:29.630268+0000","last_undegraded":"2024-09-13T14:26:29.630268+0000","last_fullsized":"2024-09-13T14:26:29.630268+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:31:48.611286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":87,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.966000+0000","last_change":"2024-09-13T14:24:55.281593+0000","last_active":"2024-09-13T14:26:29.966000+0000","last_peered":"2024-09-13T14:26:29.966000+0000","last_clean":"2024-09-13T14:26:29.966000+0000","last_became_active":"2024-09-13T14:24:55.280805+0000","last_became_peered":"2024-09-13T14:24:55.280805+0000","last_unstale":"2024-09-13T14:26:29.966000+0000","last_undegraded":"2024-09-13T14:26:29.966000+0000","last_fullsized":"2024-09-13T14:26:29.966000+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:14.806713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"119'542","reported_seq":559,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:33.285091+0000","last_change":"2024-09-13T14:26:21.864819+0000","last_active":"2024-09-13T14:26:33.285091+0000","last_peered":"2024-09-13T14:26:33.285091+0000","last_clean":"2024-09-13T14:26:33.285091+0000","last_became_active":"2024-09-13T14:26:21.864511+0000","last_became_peered":"2024-09-13T14:26:21.864511+0000","last_unstale":"2024-09-13T14:26:33.285091+0000","last_undegraded":"2024-09-13T14:26:33.285091+0000","last_fullsized":"2024-09-13T14:26:33.285091+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":542,"log_dups_size":0,"ondisk_log_size":542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:03:20.168808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35520512,"num_objects":542,"num_object_clones":0,"num_object_copies":1626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":542,"num_write_kb":34688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":101,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T14:26:34.496236+0000","last_change":"2024-09-13T14:26:15.830309+0000","last_active":"2024-09-13T14:26:34.496236+0000","last_peered":"2024-09-13T14:26:34.496236+0000","last_clean":"2024-09-13T14:26:34.496236+0000","last_became_active":"2024-09-13T14:26:15.823041+0000","last_became_peered":"2024-09-13T14:26:15.823041+0000","last_unstale":"2024-09-13T14:26:34.496236+0000","last_undegraded":"2024-09-13T14:26:34.496236+0000","last_fullsized":"2024-09-13T14:26:34.496236+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:23:08.158576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":524,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.630328+0000","last_change":"2024-09-13T14:24:23.775768+0000","last_active":"2024-09-13T14:26:29.630328+0000","last_peered":"2024-09-13T14:26:29.630328+0000","last_clean":"2024-09-13T14:26:29.630328+0000","last_became_active":"2024-09-13T14:24:23.775261+0000","last_became_peered":"2024-09-13T14:24:23.775261+0000","last_unstale":"2024-09-13T14:26:29.630328+0000","last_undegraded":"2024-09-13T14:26:29.630328+0000","last_fullsized":"2024-09-13T14:26:29.630328+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:30:05.340603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":67,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.422291+0000","last_change":"2024-09-13T14:11:30.256034+0000","last_active":"2024-09-13T14:26:29.422291+0000","last_peered":"2024-09-13T14:26:29.422291+0000","last_clean":"2024-09-13T14:26:29.422291+0000","last_became_active":"2024-09-13T14:11:30.255727+0000","last_became_peered":"2024-09-13T14:11:30.255727+0000","last_unstale":"2024-09-13T14:26:29.422291+0000","last_undegraded":"2024-09-13T14:26:29.422291+0000","last_fullsized":"2024-09-13T14:26:29.422291+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:21:35.817014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":90,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.422198+0000","last_change":"2024-09-13T14:26:15.825390+0000","last_active":"2024-09-13T14:26:29.422198+0000","last_peered":"2024-09-13T14:26:29.422198+0000","last_clean":"2024-09-13T14:26:29.422198+0000","last_became_active":"2024-09-13T14:26:15.825084+0000","last_became_peered":"2024-09-13T14:26:15.825084+0000","last_unstale":"2024-09-13T14:26:29.422198+0000","last_undegraded":"2024-09-13T14:26:29.422198+0000","last_fullsized":"2024-09-13T14:26:29.422198+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:44.680451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":90,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.422486+0000","last_change":"2024-09-13T14:11:29.237013+0000","last_active":"2024-09-13T14:26:29.422486+0000","last_peered":"2024-09-13T14:26:29.422486+0000","last_clean":"2024-09-13T14:26:29.422486+0000","last_became_active":"2024-09-13T14:11:29.236643+0000","last_became_peered":"2024-09-13T14:11:29.236643+0000","last_unstale":"2024-09-13T14:26:29.422486+0000","last_undegraded":"2024-09-13T14:26:29.422486+0000","last_fullsized":"2024-09-13T14:26:29.422486+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:33:58.438693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":56,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.629966+0000","last_change":"2024-09-13T14:26:15.826481+0000","last_active":"2024-09-13T14:26:29.629966+0000","last_peered":"2024-09-13T14:26:29.629966+0000","last_clean":"2024-09-13T14:26:29.629966+0000","last_became_active":"2024-09-13T14:26:15.826172+0000","last_became_peered":"2024-09-13T14:26:15.826172+0000","last_unstale":"2024-09-13T14:26:29.629966+0000","last_undegraded":"2024-09-13T14:26:29.629966+0000","last_fullsized":"2024-09-13T14:26:29.629966+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:54:47.170649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":86,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.477226+0000","last_change":"2024-09-13T14:24:55.287275+0000","last_active":"2024-09-13T14:26:29.477226+0000","last_peered":"2024-09-13T14:26:29.477226+0000","last_clean":"2024-09-13T14:26:29.477226+0000","last_became_active":"2024-09-13T14:24:55.286919+0000","last_became_peered":"2024-09-13T14:24:55.286919+0000","last_unstale":"2024-09-13T14:26:29.477226+0000","last_undegraded":"2024-09-13T14:26:29.477226+0000","last_fullsized":"2024-09-13T14:26:29.477226+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:10.790711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":92,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.965698+0000","last_change":"2024-09-13T14:26:15.819037+0000","last_active":"2024-09-13T14:26:29.965698+0000","last_peered":"2024-09-13T14:26:29.965698+0000","last_clean":"2024-09-13T14:26:29.965698+0000","last_became_active":"2024-09-13T14:26:15.818721+0000","last_became_peered":"2024-09-13T14:26:15.818721+0000","last_unstale":"2024-09-13T14:26:29.965698+0000","last_undegraded":"2024-09-13T14:26:29.965698+0000","last_fullsized":"2024-09-13T14:26:29.965698+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:44:06.419136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":43,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.965453+0000","last_change":"2024-09-13T14:24:55.287509+0000","last_active":"2024-09-13T14:26:29.965453+0000","last_peered":"2024-09-13T14:26:29.965453+0000","last_clean":"2024-09-13T14:26:29.965453+0000","last_became_active":"2024-09-13T14:24:55.287193+0000","last_became_peered":"2024-09-13T14:24:55.287193+0000","last_unstale":"2024-09-13T14:26:29.965453+0000","last_undegraded":"2024-09-13T14:26:29.965453+0000","last_fullsized":"2024-09-13T14:26:29.965453+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:14:10.603307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":86,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T14:26:34.496614+0000","last_change":"2024-09-13T14:24:55.289055+0000","last_active":"2024-09-13T14:26:34.496614+0000","last_peered":"2024-09-13T14:26:34.496614+0000","last_clean":"2024-09-13T14:26:34.496614+0000","last_became_active":"2024-09-13T14:24:55.288762+0000","last_became_peered":"2024-09-13T14:24:55.288762+0000","last_unstale":"2024-09-13T14:26:34.496614+0000","last_undegraded":"2024-09-13T14:26:34.496614+0000","last_fullsized":"2024-09-13T14:26:34.496614+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:37:26.639877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":99,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.965907+0000","last_change":"2024-09-13T14:26:15.824540+0000","last_active":"2024-09-13T14:26:29.965907+0000","last_peered":"2024-09-13T14:26:29.965907+0000","last_clean":"2024-09-13T14:26:29.965907+0000","last_became_active":"2024-09-13T14:26:15.824118+0000","last_became_peered":"2024-09-13T14:26:15.824118+0000","last_unstale":"2024-09-13T14:26:29.965907+0000","last_undegraded":"2024-09-13T14:26:29.965907+0000","last_fullsized":"2024-09-13T14:26:29.965907+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:56.677401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":92,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.630228+0000","last_change":"2024-09-13T14:26:15.823892+0000","last_active":"2024-09-13T14:26:29.630228+0000","last_peered":"2024-09-13T14:26:29.630228+0000","last_clean":"2024-09-13T14:26:29.630228+0000","last_became_active":"2024-09-13T14:26:15.823582+0000","last_became_peered":"2024-09-13T14:26:15.823582+0000","last_unstale":"2024-09-13T14:26:29.630228+0000","last_undegraded":"2024-09-13T14:26:29.630228+0000","last_fullsized":"2024-09-13T14:26:29.630228+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:53:09.918997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":115,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.966071+0000","last_change":"2024-09-13T14:26:15.819137+0000","last_active":"2024-09-13T14:26:29.966071+0000","last_peered":"2024-09-13T14:26:29.966071+0000","last_clean":"2024-09-13T14:26:29.966071+0000","last_became_active":"2024-09-13T14:26:15.818197+0000","last_became_peered":"2024-09-13T14:26:15.818197+0000","last_unstale":"2024-09-13T14:26:29.966071+0000","last_undegraded":"2024-09-13T14:26:29.966071+0000","last_fullsized":"2024-09-13T14:26:29.966071+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:14:06.336957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":87,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T14:26:34.496491+0000","last_change":"2024-09-13T14:24:55.283070+0000","last_active":"2024-09-13T14:26:34.496491+0000","last_peered":"2024-09-13T14:26:34.496491+0000","last_clean":"2024-09-13T14:26:34.496491+0000","last_became_active":"2024-09-13T14:24:55.282591+0000","last_became_peered":"2024-09-13T14:24:55.282591+0000","last_unstale":"2024-09-13T14:26:34.496491+0000","last_undegraded":"2024-09-13T14:26:34.496491+0000","last_fullsized":"2024-09-13T14:26:34.496491+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:35:21.896294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":72,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.965663+0000","last_change":"2024-09-13T14:26:15.823524+0000","last_active":"2024-09-13T14:26:29.965663+0000","last_peered":"2024-09-13T14:26:29.965663+0000","last_clean":"2024-09-13T14:26:29.965663+0000","last_became_active":"2024-09-13T14:26:15.823162+0000","last_became_peered":"2024-09-13T14:26:15.823162+0000","last_unstale":"2024-09-13T14:26:29.965663+0000","last_undegraded":"2024-09-13T14:26:29.965663+0000","last_fullsized":"2024-09-13T14:26:29.965663+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:17:13.664812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":112,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.965707+0000","last_change":"2024-09-13T14:26:15.823703+0000","last_active":"2024-09-13T14:26:29.965707+0000","last_peered":"2024-09-13T14:26:29.965707+0000","last_clean":"2024-09-13T14:26:29.965707+0000","last_became_active":"2024-09-13T14:26:15.823249+0000","last_became_peered":"2024-09-13T14:26:15.823249+0000","last_unstale":"2024-09-13T14:26:29.965707+0000","last_undegraded":"2024-09-13T14:26:29.965707+0000","last_fullsized":"2024-09-13T14:26:29.965707+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:42:40.209150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":107,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.422219+0000","last_change":"2024-09-13T14:26:15.823770+0000","last_active":"2024-09-13T14:26:29.422219+0000","last_peered":"2024-09-13T14:26:29.422219+0000","last_clean":"2024-09-13T14:26:29.422219+0000","last_became_active":"2024-09-13T14:26:15.823107+0000","last_became_peered":"2024-09-13T14:26:15.823107+0000","last_unstale":"2024-09-13T14:26:29.422219+0000","last_undegraded":"2024-09-13T14:26:29.422219+0000","last_fullsized":"2024-09-13T14:26:29.422219+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:15:19.495259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":121,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T14:26:34.492827+0000","last_change":"2024-09-13T14:26:15.819040+0000","last_active":"2024-09-13T14:26:34.492827+0000","last_peered":"2024-09-13T14:26:34.492827+0000","last_clean":"2024-09-13T14:26:34.492827+0000","last_became_active":"2024-09-13T14:26:15.818640+0000","last_became_peered":"2024-09-13T14:26:15.818640+0000","last_unstale":"2024-09-13T14:26:34.492827+0000","last_undegraded":"2024-09-13T14:26:34.492827+0000","last_fullsized":"2024-09-13T14:26:34.492827+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:25:29.711965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":79,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T14:26:34.496195+0000","last_change":"2024-09-13T14:23:18.035101+0000","last_active":"2024-09-13T14:26:34.496195+0000","last_peered":"2024-09-13T14:26:34.496195+0000","last_clean":"2024-09-13T14:26:34.496195+0000","last_became_active":"2024-09-13T14:23:18.034724+0000","last_became_peered":"2024-09-13T14:23:18.034724+0000","last_unstale":"2024-09-13T14:26:34.496195+0000","last_undegraded":"2024-09-13T14:26:34.496195+0000","last_fullsized":"2024-09-13T14:26:34.496195+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:39.358144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":105,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.422170+0000","last_change":"2024-09-13T14:26:15.823522+0000","last_active":"2024-09-13T14:26:29.422170+0000","last_peered":"2024-09-13T14:26:29.422170+0000","last_clean":"2024-09-13T14:26:29.422170+0000","last_became_active":"2024-09-13T14:26:15.822386+0000","last_became_peered":"2024-09-13T14:26:15.822386+0000","last_unstale":"2024-09-13T14:26:29.422170+0000","last_undegraded":"2024-09-13T14:26:29.422170+0000","last_fullsized":"2024-09-13T14:26:29.422170+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:06:43.599926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":98,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:31.180963+0000","last_change":"2024-09-13T14:26:15.826283+0000","last_active":"2024-09-13T14:26:31.180963+0000","last_peered":"2024-09-13T14:26:31.180963+0000","last_clean":"2024-09-13T14:26:31.180963+0000","last_became_active":"2024-09-13T14:26:15.820774+0000","last_became_peered":"2024-09-13T14:26:15.820774+0000","last_unstale":"2024-09-13T14:26:31.180963+0000","last_undegraded":"2024-09-13T14:26:31.180963+0000","last_fullsized":"2024-09-13T14:26:31.180963+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:28:39.849420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":96,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.422144+0000","last_change":"2024-09-13T14:24:55.285384+0000","last_active":"2024-09-13T14:26:29.422144+0000","last_peered":"2024-09-13T14:26:29.422144+0000","last_clean":"2024-09-13T14:26:29.422144+0000","last_became_active":"2024-09-13T14:24:55.285081+0000","last_became_peered":"2024-09-13T14:24:55.285081+0000","last_unstale":"2024-09-13T14:26:29.422144+0000","last_undegraded":"2024-09-13T14:26:29.422144+0000","last_fullsized":"2024-09-13T14:26:29.422144+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:47:07.946120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":70,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:31.180999+0000","last_change":"2024-09-13T14:11:29.240351+0000","last_active":"2024-09-13T14:26:31.180999+0000","last_peered":"2024-09-13T14:26:31.180999+0000","last_clean":"2024-09-13T14:26:31.180999+0000","last_became_active":"2024-09-13T14:11:29.240060+0000","last_became_peered":"2024-09-13T14:11:29.240060+0000","last_unstale":"2024-09-13T14:26:31.180999+0000","last_undegraded":"2024-09-13T14:26:31.180999+0000","last_fullsized":"2024-09-13T14:26:31.180999+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:34:36.126368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":111,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.966146+0000","last_change":"2024-09-13T14:24:55.285488+0000","last_active":"2024-09-13T14:26:29.966146+0000","last_peered":"2024-09-13T14:26:29.966146+0000","last_clean":"2024-09-13T14:26:29.966146+0000","last_became_active":"2024-09-13T14:24:55.283433+0000","last_became_peered":"2024-09-13T14:24:55.283433+0000","last_unstale":"2024-09-13T14:26:29.966146+0000","last_undegraded":"2024-09-13T14:26:29.966146+0000","last_fullsized":"2024-09-13T14:26:29.966146+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:21:11.486340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":99,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.966584+0000","last_change":"2024-09-13T14:24:55.287498+0000","last_active":"2024-09-13T14:26:29.966584+0000","last_peered":"2024-09-13T14:26:29.966584+0000","last_clean":"2024-09-13T14:26:29.966584+0000","last_became_active":"2024-09-13T14:24:55.285957+0000","last_became_peered":"2024-09-13T14:24:55.285957+0000","last_unstale":"2024-09-13T14:26:29.966584+0000","last_undegraded":"2024-09-13T14:26:29.966584+0000","last_fullsized":"2024-09-13T14:26:29.966584+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:21:50.821805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":90,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-13T14:26:29.421982+0000","last_change":"2024-09-13T14:26:15.824075+0000","last_active":"2024-09-13T14:26:29.421982+0000","last_peered":"2024-09-13T14:26:29.421982+0000","last_clean":"2024-09-13T14:26:29.421982+0000","last_became_active":"2024-09-13T14:26:15.823762+0000","last_became_peered":"2024-09-13T14:26:15.823762+0000","last_unstale":"2024-09-13T14:26:29.421982+0000","last_undegraded":"2024-09-13T14:26:29.421982+0000","last_fullsized":"2024-09-13T14:26:29.421982+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:26:51.088388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":90,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-13T14:26:34.496480+0000","last_change":"2024-09-13T14:24:23.775769+0000","last_active":"2024-09-13T14:26:34.496480+0000","last_peered":"2024-09-13T14:26:34.496480+0000","last_clean":"2024-09-13T14:26:34.496480+0000","last_became_active":"2024-09-13T14:24:23.775409+0000","last_became_peered":"2024-09-13T14:24:23.775409+0000","last_unstale":"2024-09-13T14:26:34.496480+0000","last_undegraded":"2024-09-13T14:26:34.496480+0000","last_fullsized":"2024-09-13T14:26:34.496480+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:20:09.966476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":1155530752,"num_objects":17632,"num_object_clones":0,"num_object_copies":52896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17632,"num_write_kb":1128448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3442212864,"data_stored":3442212864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17632,"ondisk_log_size":17632,"up":48,"acting":48,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18272,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16793600,"data_stored":16777292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234193,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":363900,"kb_used_data":332788,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93381252,"statfs":{"total":95995035648,"available":95622402048,"internally_reserved":0,"allocated":340774912,"data_stored":340125973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495829,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":412280,"kb_used_data":377068,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":93332872,"statfs":{"total":95995035648,"available":95572860928,"internally_reserved":0,"allocated":386117632,"data_stored":385476816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757466,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":749484,"kb_used_data":718368,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":92995668,"statfs":{"total":95995035648,"available":95227564032,"internally_reserved":0,"allocated":735608832,"data_stored":734975528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019102,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":653624,"kb_used_data":614316,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":93091528,"statfs":{"total":95995035648,"available":95325724672,"internally_reserved":0,"allocated":629059584,"data_stored":628418281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435286,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":122712,"kb_used_data":91024,"kb_used_omap":5,"kb_used_meta":31674,"kb_avail":93622440,"statfs":{"total":95995035648,"available":95869378560,"internally_reserved":0,"allocated":93208576,"data_stored":92583277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":32434302},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":104,"seq":446676598813,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":475240,"kb_used_data":439392,"kb_used_omap":7,"kb_used_meta":35832,"kb_avail":93269912,"statfs":{"total":95995035648,"available":95508389888,"internally_reserved":0,"allocated":449937408,"data_stored":449304392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8010,"internal_metadata":36692150},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":425201762339,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726925866,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":666676,"kb_used_data":639144,"kb_used_omap":7,"kb_used_meta":27512,"kb_avail":93078476,"statfs":{"total":95995035648,"available":95312359424,"internally_reserved":0,"allocated":654483456,"data_stored":653842799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":28172584},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":650641408,"data_stored":650641408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":465043456,"data_stored":465043456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":108396544,"data_stored":108396544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":664010752,"data_stored":664010752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":743374848,"data_stored":743374848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":438894592,"data_stored":438894592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":371851264,"data_stored":371851264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:26:42.227 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_0 pgp_num 16 2024-09-13T14:26:42.242 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:26:42.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:42.313 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:42.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:26:42.414 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:26:42.448 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:42.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:42.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:26:42.550 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:26:42.584 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:42.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:42.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:26:42.686 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:26:42.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:42.724 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:42.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:26:42.809 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:42 smithi102 ceph-mon[110183]: pgmap v311: 89 pgs: 89 active+clean; 1.6 GiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 165 MiB/s wr, 2.63k op/s 2024-09-13T14:26:42.809 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:42 smithi102 ceph-mon[115269]: pgmap v311: 89 pgs: 89 active+clean; 1.6 GiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 165 MiB/s wr, 2.63k op/s 2024-09-13T14:26:42.825 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:26:42.863 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:42.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:42.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:26:42.964 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:26:42.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:42 smithi195 ceph-mon[104262]: pgmap v311: 89 pgs: 89 active+clean; 1.6 GiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 165 MiB/s wr, 2.63k op/s 2024-09-13T14:26:43.003 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:43.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:43.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:26:43.105 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:26:43.119 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 12 16 35020 35004 182.294 145.438 0.00521842 0.00548059 2024-09-13T14:26:43.139 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:43.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:43.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:26:43.241 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:26:43.274 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:43.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:43.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:26:43.376 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:26:43.409 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:43.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:43.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:26:43.512 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:26:43.533 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:43.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:43.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:26:43.635 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:26:43.665 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:43.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:43.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:26:43.767 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:43.801 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:43.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:43.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:43.903 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:43.939 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:43.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:43.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:44.016 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:26:44.041 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:26:44.079 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:44.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:44.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:26:44.119 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 13 16 37234 37218 178.914 138.375 0.011653 0.00558381 2024-09-13T14:26:44.182 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:26:44.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:44.243 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:44.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:26:44.344 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:26:44.378 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:44.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:44.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:26:44.479 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:26:44.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:44.514 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:44.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:26:44.616 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:26:44.646 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:44.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:44.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:26:44.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:44 smithi102 ceph-mon[110183]: pgmap v312: 89 pgs: 89 active+clean; 1.9 GiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 187 MiB/s wr, 2.99k op/s 2024-09-13T14:26:44.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:44 smithi102 ceph-mon[115269]: pgmap v312: 89 pgs: 89 active+clean; 1.9 GiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 187 MiB/s wr, 2.99k op/s 2024-09-13T14:26:44.748 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:26:44.783 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:44.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:44.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:26:44.845 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:26:44.885 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:26:44.924 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:44.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:44.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:26:44.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:44 smithi195 ceph-mon[104262]: pgmap v312: 89 pgs: 89 active+clean; 1.9 GiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 187 MiB/s wr, 2.99k op/s 2024-09-13T14:26:45.026 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:26:45.067 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:45.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:45.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:26:45.120 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 14 16 39235 39219 175.058 125.062 0.00462913 0.0057074 2024-09-13T14:26:45.169 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:26:45.199 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:45.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:45.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:26:45.301 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:26:45.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:45.323 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:45.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:26:45.424 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:26:45.454 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:45.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:45.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:26:45.512 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:26:45.556 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:26:45.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:45.590 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:45.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:26:45.691 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:26:45.726 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:45.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:45.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:26:45.809 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:26:45.809 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:26:45.828 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:26:45.862 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:45.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:45.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:26:45.963 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:26:46.001 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:46.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:46.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:26:46.103 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:26:46.120 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 15 16 41277 41261 171.895 127.625 0.00486156 0.005813 2024-09-13T14:26:46.135 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:46.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:46.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:26:46.237 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:26:46.269 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:26:46.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:46.279 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:46.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:26:46.380 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:26:46.408 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:46.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:46.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:26:46.510 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:26:46.542 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:46.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:46.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:26:46.644 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:26:46.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:46 smithi102 ceph-mon[110183]: pgmap v313: 89 pgs: 89 active+clean; 1.9 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 175 MiB/s wr, 2.80k op/s 2024-09-13T14:26:46.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:46 smithi102 ceph-mon[115269]: pgmap v313: 89 pgs: 89 active+clean; 1.9 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 175 MiB/s wr, 2.80k op/s 2024-09-13T14:26:46.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:26:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:26:46] "GET /metrics HTTP/1.1" 200 36499 "" "Prometheus/2.51.0" 2024-09-13T14:26:46.682 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:46.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:46.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:26:46.784 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:26:46.794 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T14:26:46.816 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:46.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:46.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:26:46.919 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:26:46.960 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:46.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:46.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:26:46.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:46 smithi195 ceph-mon[104262]: pgmap v313: 89 pgs: 89 active+clean; 1.9 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 175 MiB/s wr, 2.80k op/s 2024-09-13T14:26:47.062 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:26:47.095 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:47.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:47.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:26:47.120 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 16 16 43350 43334 169.249 129.562 0.0129216 0.00590414 2024-09-13T14:26:47.197 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:26:47.236 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:47.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:47.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:26:47.288 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:26:47.339 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:26:47.368 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:47.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:47.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:26:47.470 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:26:47.502 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:47.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:47.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:26:47.604 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:26:47.643 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:47.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:47.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:26:47.744 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:26:47.773 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:47.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:47.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:26:47.875 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:26:47.920 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:47.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:47.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:26:48.022 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:26:48.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:48.063 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:48.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:26:48.120 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 17 16 45301 45285 166.465 121.938 0.00731623 0.00600265 2024-09-13T14:26:48.164 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:26:48.205 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:48.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:48.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:26:48.307 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:48.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:48.348 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:48.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:48.449 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:26:48.490 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:48.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:48.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:26:48.592 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:48.629 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:48.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:48.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:48.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:48 smithi102 ceph-mon[110183]: pgmap v314: 89 pgs: 89 active+clean; 2.5 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 209 MiB/s wr, 3.34k op/s 2024-09-13T14:26:48.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:48 smithi102 ceph-mon[115269]: pgmap v314: 89 pgs: 89 active+clean; 2.5 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 209 MiB/s wr, 3.34k op/s 2024-09-13T14:26:48.731 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:26:48.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:48.764 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:48.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:26:48.866 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:26:48.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:48.902 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:48.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:26:48.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:48 smithi195 ceph-mon[104262]: pgmap v314: 89 pgs: 89 active+clean; 2.5 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 209 MiB/s wr, 3.34k op/s 2024-09-13T14:26:49.004 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:26:49.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:49.045 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:49.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:26:49.120 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 18 16 47234 47218 163.929 120.812 0.00437507 0.00609497 2024-09-13T14:26:49.147 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:26:49.183 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:49.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:49.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:26:49.285 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:26:49.320 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:49.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:49.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:26:49.422 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:26:49.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:49.455 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:49.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:26:49.469 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:26:49.557 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:26:49.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:49.599 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:49.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:26:49.701 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:26:49.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:49.736 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:49.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:26:49.837 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:26:49.866 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:49.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:49.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:26:49.968 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:26:50.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:50.004 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:50.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:26:50.105 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:26:50.120 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 19 16 49173 49157 161.679 121.188 0.0101321 0.00618012 2024-09-13T14:26:50.147 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:50.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:50.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:26:50.249 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:26:50.290 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:50.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:50.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:26:50.392 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:50.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:50.431 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:50.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:50.532 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:26:50.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:50.572 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:50.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:26:50.673 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:26:50.706 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:50.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:50.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:26:50.808 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:50.846 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:50.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:50.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:50.948 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:26:50.988 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:50.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:50.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:26:50.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:50 smithi195 ceph-mon[104262]: pgmap v315: 89 pgs: 89 active+clean; 2.6 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 126 MiB/s wr, 2.01k op/s 2024-09-13T14:26:51.024 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:50 smithi102 ceph-mon[115269]: pgmap v315: 89 pgs: 89 active+clean; 2.6 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 126 MiB/s wr, 2.01k op/s 2024-09-13T14:26:51.024 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:50 smithi102 ceph-mon[110183]: pgmap v315: 89 pgs: 89 active+clean; 2.6 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 126 MiB/s wr, 2.01k op/s 2024-09-13T14:26:51.090 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:26:51.120 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:2024-09-13T14:26:51.120866+0000 min lat: 0.00124054 max lat: 0.118191 avg lat: 0.00627337 2024-09-13T14:26:51.121 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:26:51.121 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 20 16 50997 50981 159.294 114 0.00732108 0.00627337 2024-09-13T14:26:51.125 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:51.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:51.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:26:51.227 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:51.267 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:51.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:51.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:51.368 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:51.404 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:51.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:51.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:51.506 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:26:51.537 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:51.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:51.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:26:51.639 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:51.678 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:51.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:51.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:51.745 INFO:teuthology.orchestra.run.smithi102.stderr:set pool 5 pgp_num to 16 2024-09-13T14:26:51.780 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:26:51.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:51.817 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:51.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:26:51.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:51 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2978580207' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"} : dispatch 2024-09-13T14:26:51.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:51 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"} : dispatch 2024-09-13T14:26:51.918 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:26:51.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:51 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2978580207' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"} : dispatch 2024-09-13T14:26:51.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:51 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"} : dispatch 2024-09-13T14:26:51.957 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:51.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:51.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:26:51.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:51 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2978580207' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"} : dispatch 2024-09-13T14:26:51.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:51 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"} : dispatch 2024-09-13T14:26:52.059 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:26:52.096 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:52.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:52.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:26:52.121 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 21 16 52798 52782 157.069 112.562 0.00858185 0.00636204 2024-09-13T14:26:52.198 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:26:52.206 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:26:52.206 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:26:52.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:52.229 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:52.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:26:52.235 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:26:52.235 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:26:52.268 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:26:52.268 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:26:52.290 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:26:52.290 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:26:52.318 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:26:52.318 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:26:52.330 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:26:52.346 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:26:52.347 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:26:52.366 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:52.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:52.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:26:52.384 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:26:52.384 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:26:52.414 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:26:52.414 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:26:52.468 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:26:52.509 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:52.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:52.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:26:52.610 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:26:52.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:52.650 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:52.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:26:52.751 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:26:52.784 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:52.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:52.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:26:52.885 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:26:52.914 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:52.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:52.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:26:52.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:52 smithi195 ceph-mon[104262]: pgmap v316: 89 pgs: 89 active+clean; 2.9 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 153 MiB/s wr, 2.45k op/s 2024-09-13T14:26:52.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:52 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]': finished 2024-09-13T14:26:52.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:52 smithi195 ceph-mon[104262]: osdmap e121: 8 total, 8 up, 7 in 2024-09-13T14:26:52.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:52 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1566045913' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:26:53.016 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:26:53.046 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:53.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:53.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:26:53.121 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 22 16 54561 54545 154.937 110.188 0.00576744 0.00644975 2024-09-13T14:26:53.134 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:52 smithi102 ceph-mon[110183]: pgmap v316: 89 pgs: 89 active+clean; 2.9 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 153 MiB/s wr, 2.45k op/s 2024-09-13T14:26:53.134 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:52 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]': finished 2024-09-13T14:26:53.134 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:52 smithi102 ceph-mon[110183]: osdmap e121: 8 total, 8 up, 7 in 2024-09-13T14:26:53.134 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:52 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1566045913' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:26:53.135 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:52 smithi102 ceph-mon[115269]: pgmap v316: 89 pgs: 89 active+clean; 2.9 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 153 MiB/s wr, 2.45k op/s 2024-09-13T14:26:53.135 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:52 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]': finished 2024-09-13T14:26:53.135 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:52 smithi102 ceph-mon[115269]: osdmap e121: 8 total, 8 up, 7 in 2024-09-13T14:26:53.135 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:52 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1566045913' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:26:53.148 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:53.185 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:53.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:53.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:53.288 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:26:53.319 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:53.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:53.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:26:53.421 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:26:53.456 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:53.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:53.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:26:53.558 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:26:53.600 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:53.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:53.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:26:53.702 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:26:53.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:53.734 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:53.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:26:53.752 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is unset 2024-09-13T14:26:53.835 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:53.873 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:53.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:53.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:53.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:53 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/304163930' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:26:53.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:53 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/304163930' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:26:53.975 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:26:54.010 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:54.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:54.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:26:54.112 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:26:54.121 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 23 16 56259 56243 152.815 106.125 0.0107165 0.00653863 2024-09-13T14:26:54.140 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:54.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:54.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:26:54.242 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:54.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:53 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/304163930' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:26:54.275 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:54.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:54.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:54.377 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:26:54.405 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:54.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:54.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:26:54.507 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:26:54.536 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:54.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:54.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:26:54.638 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:54.673 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:54.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:54.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:54.790 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:26:54.826 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:54.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:54.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:26:54.928 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:26:54.956 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:54.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:54.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:26:54.986 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:54 smithi102 ceph-mon[110183]: pgmap v318: 89 pgs: 89 active+clean; 3.1 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 128 MiB/s wr, 2.05k op/s 2024-09-13T14:26:54.986 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:54 smithi102 ceph-mon[110183]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:26:54.986 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:54 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/304163930' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:26:54.986 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:54 smithi102 ceph-mon[110183]: osdmap e122: 8 total, 8 up, 7 in 2024-09-13T14:26:54.987 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:54 smithi102 ceph-mon[115269]: pgmap v318: 89 pgs: 89 active+clean; 3.1 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 128 MiB/s wr, 2.05k op/s 2024-09-13T14:26:54.987 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:54 smithi102 ceph-mon[115269]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:26:54.987 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:54 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/304163930' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:26:54.987 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:54 smithi102 ceph-mon[115269]: osdmap e122: 8 total, 8 up, 7 in 2024-09-13T14:26:55.059 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:26:55.088 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:55.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:55.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:26:55.121 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 24 16 57958 57942 150.871 106.188 0.0056287 0.00662357 2024-09-13T14:26:55.148 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:54 smithi195 ceph-mon[104262]: pgmap v318: 89 pgs: 89 active+clean; 3.1 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 128 MiB/s wr, 2.05k op/s 2024-09-13T14:26:55.149 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:54 smithi195 ceph-mon[104262]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:26:55.149 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:54 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/304163930' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:26:55.149 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:54 smithi195 ceph-mon[104262]: osdmap e122: 8 total, 8 up, 7 in 2024-09-13T14:26:55.170 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:26:55.190 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:55.226 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:55.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:55.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:55.310 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:26:55.310 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [], 2024-09-13T14:26:55.310 INFO:teuthology.orchestra.run.smithi102.stdout: "complaint_time": 30, 2024-09-13T14:26:55.310 INFO:teuthology.orchestra.run.smithi102.stdout: "num_blocked_ops": 0 2024-09-13T14:26:55.310 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:26:55.328 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:55.363 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:55.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:55.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:55.465 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:26:55.498 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:55.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:55.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:26:55.600 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:26:55.633 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:55.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:55.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:26:55.735 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:26:55.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:55.773 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:55.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:26:55.875 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:26:55.910 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:55.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:55.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:26:56.012 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:56.034 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:55 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:26:56.034 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:55 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:26:56.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:56.051 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:56.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:56.121 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 25 16 59582 59566 148.896 101.5 0.0125267 0.00671138 2024-09-13T14:26:56.151 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:56.180 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:56.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:56.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:56.183 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T14:26:56.234 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:26:56.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:55 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:26:56.281 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:26:56.332 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:56.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:56.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:26:56.434 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:26:56.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:56.471 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:56.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:26:56.572 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:26:56.605 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:56.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:56.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:26:56.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:26:56 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:26:56] "GET /metrics HTTP/1.1" 200 36512 "" "Prometheus/2.51.0" 2024-09-13T14:26:56.707 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:26:56.710 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:26:56.751 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:56.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:56.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:26:56.853 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:26:56.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:56.891 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:56.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:26:56.992 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:26:57.027 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:57.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:57.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:26:57.096 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:26:57.121 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 26 16 61136 61120 146.905 97.125 0.00704902 0.00680263 2024-09-13T14:26:57.129 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:26:57.165 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:57.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:57.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:26:57.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:56 smithi102 ceph-mon[115269]: pgmap v320: 89 pgs: 89 active+clean; 3.1 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-09-13T14:26:57.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:56 smithi102 ceph-mon[110183]: pgmap v320: 89 pgs: 89 active+clean; 3.1 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-09-13T14:26:57.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:56 smithi195 ceph-mon[104262]: pgmap v320: 89 pgs: 89 active+clean; 3.1 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-09-13T14:26:57.266 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:26:57.299 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:57.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:57.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:26:57.401 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:26:57.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:57.436 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:57.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:26:57.537 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:26:57.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:57.564 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:57.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:26:57.665 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:26:57.708 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:57.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:57.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:26:57.811 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:26:57.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:57.840 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:57.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:26:57.932 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:26:57.941 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:26:57.986 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:57.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:57.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:26:58.089 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:26:58.118 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:58.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:58.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:26:58.121 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 27 16 62725 62709 145.142 99.3125 0.00538923 0.00688501 2024-09-13T14:26:58.220 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:26:58.260 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:58.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:58.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:26:58.362 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:26:58.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:58.399 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:58.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:26:58.500 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:26:58.540 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:58.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:58.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:26:58.616 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:26:58.616 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:26:58.616 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [0, 2, 3, 4, 5, 6, 7] 2024-09-13T14:26:58.616 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd out 2 2024-09-13T14:26:58.642 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:58.675 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:58.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:58.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:58.777 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:58.815 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:58.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:58.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:58.917 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:26:58.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:58 smithi102 ceph-mon[110183]: pgmap v321: 89 pgs: 89 active+clean; 3.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 138 MiB/s wr, 2.21k op/s 2024-09-13T14:26:58.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:58 smithi102 ceph-mon[115269]: pgmap v321: 89 pgs: 89 active+clean; 3.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 138 MiB/s wr, 2.21k op/s 2024-09-13T14:26:58.957 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:58.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:58.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:26:59.059 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:26:59.097 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:59.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:59.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:26:59.121 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 28 16 64132 64116 143.099 87.9375 0.0103001 0.00698364 2024-09-13T14:26:59.199 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:26:59.236 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:59.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:59.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:26:59.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:58 smithi195 ceph-mon[104262]: pgmap v321: 89 pgs: 89 active+clean; 3.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 138 MiB/s wr, 2.21k op/s 2024-09-13T14:26:59.337 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:26:59.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:59.381 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:59.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:26:59.483 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:26:59.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:59.516 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:59.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:26:59.617 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:26:59.659 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:59.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:59.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:26:59.760 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:26:59.800 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:26:59.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:59.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:26:59.902 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:26:59.931 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:26:59.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:26:59.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:27:00.033 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:00.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:00.070 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:00.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:00.121 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 29 16 65649 65633 141.433 94.8125 0.011072 0.00706561 2024-09-13T14:27:00.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:26:59 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/469914188' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:27:00.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:26:59 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/469914188' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:27:00.171 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:00.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:00.212 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:00.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:00.214 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:27:00.214 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:27:00.214 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:27:00.214 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:27:00.214 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:27:00.215 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:27:00.215 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:27:00.215 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:27:00.215 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:27:00.215 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:27:00.215 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:27:00.215 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:27:00.215 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:27:00.216 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:27:00.216 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:27:00.216 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:27:00.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:26:59 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/469914188' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:27:00.313 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:27:00.343 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:00.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:00.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:27:00.444 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:00.481 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:00.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:00.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:00.583 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:00.618 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:00.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:00.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:00.720 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:00.753 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:00.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:00.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:00.856 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:00.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:00.885 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:00.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:00.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:00 smithi102 ceph-mon[110183]: pgmap v322: 89 pgs: 89 active+clean; 3.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-09-13T14:27:00.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:00 smithi102 ceph-mon[115269]: pgmap v322: 89 pgs: 89 active+clean; 3.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-09-13T14:27:00.986 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:27:01.014 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:01.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:01.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:27:01.116 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:01.121 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 30 16 67117 67101 139.777 91.75 0.00291499 0.00714628 2024-09-13T14:27:01.137 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:01.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:01.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:01.239 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:01.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:00 smithi195 ceph-mon[104262]: pgmap v322: 89 pgs: 89 active+clean; 3.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-09-13T14:27:01.275 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:01.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:01.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:01.377 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:01.409 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:01.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:01.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:01.511 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:01.552 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:01.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:01.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:01.654 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:01.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:01.693 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:01.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:01.794 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:27:01.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:01.825 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:01.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:27:01.926 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:27:01.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:01.956 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:01.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:27:02.057 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:02.092 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:02.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:02.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:02.121 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 31 16 68508 68492 138.072 86.9375 0.00615888 0.00723794 2024-09-13T14:27:02.194 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:27:02.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:02.216 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:02.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:27:02.317 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:02.346 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:02.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:02.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:02.448 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:27:02.480 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:02.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:02.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:27:02.582 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:27:02.607 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:02.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:02.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:27:02.709 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:02.743 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:02.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:02.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:02.845 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:02.886 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:02.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:02.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:02.988 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:27:03.015 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:03.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:03.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:27:03.117 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:03.123 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 32 16 69910 69894 136.495 87.625 0.00587425 0.00732191 2024-09-13T14:27:03.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:03.150 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:03.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:03.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:02 smithi102 ceph-mon[110183]: pgmap v323: 89 pgs: 89 active+clean; 4.0 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-09-13T14:27:03.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:02 smithi102 ceph-mon[115269]: pgmap v323: 89 pgs: 89 active+clean; 4.0 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-09-13T14:27:03.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:02 smithi195 ceph-mon[104262]: pgmap v323: 89 pgs: 89 active+clean; 4.0 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-09-13T14:27:03.251 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:03.272 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:03.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:03.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:03.374 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:03.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:03.410 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:03.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:03.511 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:27:03.544 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:03.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:03.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:27:03.646 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:03.687 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:03.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:03.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:03.789 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:27:03.818 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:03.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:03.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:27:03.920 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:27:03.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:03.956 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:03.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:27:04.057 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:04.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:04.086 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:04.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:04.122 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 33 16 71255 71239 134.906 84.0625 0.0116029 0.00740774 2024-09-13T14:27:04.187 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:27:04.215 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:04.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:04.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:27:04.316 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:04.367 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:04.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:04.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:04.469 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:04.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:04.503 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:04.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:04.604 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:04.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:04.659 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:04.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:04.760 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:04.790 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:04.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:04.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:04.892 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:04.941 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:04.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:04.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:05.043 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:05.052 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:04 smithi102 ceph-mon[110183]: pgmap v324: 89 pgs: 89 active+clean; 4.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-09-13T14:27:05.052 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:04 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:27:05.052 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:04 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:27:05.052 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:04 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:27:05.052 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:04 smithi102 ceph-mon[115269]: pgmap v324: 89 pgs: 89 active+clean; 4.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-09-13T14:27:05.053 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:04 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:27:05.053 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:04 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:27:05.053 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:04 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:27:05.071 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:05.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:05.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:05.122 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 34 16 72500 72484 133.227 77.8125 0.01328 0.00750073 2024-09-13T14:27:05.172 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:05.246 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:05.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:04 smithi195 ceph-mon[104262]: pgmap v324: 89 pgs: 89 active+clean; 4.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-09-13T14:27:05.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:04 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:27:05.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:04 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:27:05.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:04 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:27:05.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:05.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:05.349 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:27:05.373 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:05.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:05.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:27:05.475 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:05.513 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:05.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:05.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:05.615 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:05.643 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:05.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:05.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:05.745 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:05.794 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:05.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:05.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:05.896 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:05.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:05.940 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:05.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:06.041 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:27:06.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:06.074 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:06.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:27:06.122 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 35 16 73649 73633 131.472 71.8125 0.00425739 0.0075985 2024-09-13T14:27:06.176 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:06.225 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:06.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:06.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:06.327 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:06.352 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:06.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:06.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:06.453 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:06.489 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:06.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:06.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:06.590 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:06.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:06.644 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:06.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:06.698 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:27:06.699 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:27:06.699 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:27:06 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:27:06] "GET /metrics HTTP/1.1" 200 36512 "" "Prometheus/2.51.0" 2024-09-13T14:27:06.746 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:27:06.772 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:06.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:06.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:27:06.874 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:06.915 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:06.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:06.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:06.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:06 smithi195 ceph-mon[104262]: pgmap v325: 89 pgs: 89 active+clean; 4.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-13T14:27:07.017 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:07.025 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:06 smithi102 ceph-mon[110183]: pgmap v325: 89 pgs: 89 active+clean; 4.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-13T14:27:07.025 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:06 smithi102 ceph-mon[115269]: pgmap v325: 89 pgs: 89 active+clean; 4.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-13T14:27:07.053 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:07.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:07.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:07.122 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 36 16 74825 74809 129.862 73.5 0.0145606 0.00769611 2024-09-13T14:27:07.155 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:07.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:07.199 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:07.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:07.300 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:07.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:07.345 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:07.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:07.446 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:27:07.482 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:07.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:07.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:27:07.488 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:27:07.585 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:27:07.613 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:07.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:07.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:27:07.716 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:07.752 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:07.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:07.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:07.853 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:07.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:07.884 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:07.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:07.941 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T14:27:07.985 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:08.039 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:08.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:08.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:08.122 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 37 16 76075 76059 128.463 78.125 0.0168439 0.0077796 2024-09-13T14:27:08.142 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:08.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:08.174 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:08.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:08.275 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:08.314 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:08.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:08.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:08.416 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:08.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:08.463 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:08.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:08.565 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:08.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:08.597 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:08.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:08.698 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:08.725 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:08.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:08.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:08.805 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is unset 2024-09-13T14:27:08.827 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:08.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:08.872 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:08.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:08.973 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:09.008 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:09.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:09.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:09.028 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:27:09.079 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:08 smithi102 ceph-mon[110183]: pgmap v326: 89 pgs: 89 active+clean; 4.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-09-13T14:27:09.080 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:08 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/244498333' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:27:09.080 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:08 smithi102 ceph-mon[115269]: pgmap v326: 89 pgs: 89 active+clean; 4.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-09-13T14:27:09.080 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:08 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/244498333' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:27:09.110 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:27:09.116 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:08 smithi195 ceph-mon[104262]: pgmap v326: 89 pgs: 89 active+clean; 4.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-09-13T14:27:09.116 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:08 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/244498333' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:27:09.123 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 38 16 77326 77310 127.14 78.1875 0.0142709 0.00786069 2024-09-13T14:27:09.148 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:09.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:09.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:27:09.252 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:09.288 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:09.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:09.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:09.390 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:09.423 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:09.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:09.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:09.525 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:27:09.552 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:09.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:09.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:27:09.653 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:09.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:09.693 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:09.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:09.794 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:09.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:09.830 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:09.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:09.931 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:09.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:09.970 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:09.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:10.049 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:09 smithi102 ceph-mon[110183]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:27:10.049 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:09 smithi102 ceph-mon[110183]: Cluster is now healthy 2024-09-13T14:27:10.049 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:09 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/244498333' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:27:10.049 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:09 smithi102 ceph-mon[110183]: osdmap e123: 8 total, 8 up, 7 in 2024-09-13T14:27:10.049 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:09 smithi102 ceph-mon[115269]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:27:10.049 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:09 smithi102 ceph-mon[115269]: Cluster is now healthy 2024-09-13T14:27:10.050 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:09 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/244498333' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:27:10.050 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:09 smithi102 ceph-mon[115269]: osdmap e123: 8 total, 8 up, 7 in 2024-09-13T14:27:10.070 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:10.098 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:09 smithi195 ceph-mon[104262]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:27:10.098 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:09 smithi195 ceph-mon[104262]: Cluster is now healthy 2024-09-13T14:27:10.098 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:09 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/244498333' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:27:10.098 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:09 smithi195 ceph-mon[104262]: osdmap e123: 8 total, 8 up, 7 in 2024-09-13T14:27:10.122 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 39 16 78605 78589 125.93 79.9375 0.0126496 0.00793569 2024-09-13T14:27:10.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:10.123 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:10.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:10.224 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:10.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:10.260 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:10.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:10.361 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:27:10.414 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:10.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:10.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:27:10.516 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:10.545 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:10.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:10.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:10.631 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:27:10.647 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:10.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:10.695 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:10.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:10.796 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:10.828 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:10.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:10.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:10.877 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:10 smithi102 ceph-mon[110183]: pgmap v328: 89 pgs: 89 active+clean; 4.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 87 MiB/s wr, 1.40k op/s 2024-09-13T14:27:10.877 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:27:10.878 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:27:10.878 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:27:10.878 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:27:10.878 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:10 smithi102 ceph-mon[115269]: pgmap v328: 89 pgs: 89 active+clean; 4.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 87 MiB/s wr, 1.40k op/s 2024-09-13T14:27:10.878 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:27:10.878 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:27:10.878 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:27:10.878 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:27:10.930 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:10.979 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:10.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:10.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:11.082 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:27:11.112 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:11.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:11.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:27:11.122 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:2024-09-13T14:27:11.122733+0000 min lat: 0.00124054 max lat: 0.146591 avg lat: 0.00801542 2024-09-13T14:27:11.122 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:27:11.122 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 40 16 79815 79799 124.672 75.625 0.00797113 0.00801542 2024-09-13T14:27:11.170 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:27:11.214 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:11.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:10 smithi195 ceph-mon[104262]: pgmap v328: 89 pgs: 89 active+clean; 4.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 87 MiB/s wr, 1.40k op/s 2024-09-13T14:27:11.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:27:11.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:27:11.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:27:11.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:27:11.268 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:11.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:11.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:11.370 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:11.412 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:11.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:11.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:11.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:11 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-a[110159]: 2024-09-13T14:27:11.230+0000 7fa3e7f42640 -1 mon.a@0(leader).osd e123 definitely_dead 0 2024-09-13T14:27:11.514 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:11.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:11.552 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:11.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:11.653 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:11.691 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:11.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:11.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:11.793 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:11.825 INFO:teuthology.orchestra.run.smithi102.stderr:marked out osd.2. 2024-09-13T14:27:11.835 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:11.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:11.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:11.937 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:27:11.959 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:11.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:11.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:27:12.061 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:12.083 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:12.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:12.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:12.122 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 41 16 80745 80729 123.049 58.125 0.00279041 0.00806627 2024-09-13T14:27:12.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:11 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1576647407' entity='client.admin' cmd={"prefix": "osd out", "ids": ["2"]} : dispatch 2024-09-13T14:27:12.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:11 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["2"]} : dispatch 2024-09-13T14:27:12.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:11 smithi102 ceph-mon[110183]: Client client.admin marked osd.2 out, while it was still marked up 2024-09-13T14:27:12.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:11 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1576647407' entity='client.admin' cmd={"prefix": "osd out", "ids": ["2"]} : dispatch 2024-09-13T14:27:12.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:11 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["2"]} : dispatch 2024-09-13T14:27:12.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:11 smithi102 ceph-mon[115269]: Client client.admin marked osd.2 out, while it was still marked up 2024-09-13T14:27:12.185 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:12.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:12.208 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:12.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:12.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:11 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1576647407' entity='client.admin' cmd={"prefix": "osd out", "ids": ["2"]} : dispatch 2024-09-13T14:27:12.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:11 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["2"]} : dispatch 2024-09-13T14:27:12.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:11 smithi195 ceph-mon[104262]: Client client.admin marked osd.2 out, while it was still marked up 2024-09-13T14:27:12.309 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:27:12.330 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:12.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:12.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:27:12.400 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:27:12.432 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:12.458 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:12.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:12.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:12.560 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:27:12.581 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:12.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:12.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:27:12.683 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:12.705 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:12.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:12.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:12.806 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:12.841 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:12.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:12.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:12.943 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:12.968 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:12.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:12.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:13.070 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:13.095 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:13.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:13.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:13.102 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:12 smithi102 ceph-mon[110183]: pgmap v329: 89 pgs: 89 active+clean; 4.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 82 MiB/s wr, 1.30k op/s 2024-09-13T14:27:13.103 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:12 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-09-13T14:27:13.103 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:12 smithi102 ceph-mon[110183]: osdmap e124: 8 total, 8 up, 6 in 2024-09-13T14:27:13.103 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:12 smithi102 ceph-mon[115269]: pgmap v329: 89 pgs: 89 active+clean; 4.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 82 MiB/s wr, 1.30k op/s 2024-09-13T14:27:13.103 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:12 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-09-13T14:27:13.103 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:12 smithi102 ceph-mon[115269]: osdmap e124: 8 total, 8 up, 6 in 2024-09-13T14:27:13.122 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 42 16 80745 80729 120.119 0 - 0.00806627 2024-09-13T14:27:13.197 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:13.221 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:13.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:13.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:13.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:12 smithi195 ceph-mon[104262]: pgmap v329: 89 pgs: 89 active+clean; 4.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 82 MiB/s wr, 1.30k op/s 2024-09-13T14:27:13.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:12 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-09-13T14:27:13.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:12 smithi195 ceph-mon[104262]: osdmap e124: 8 total, 8 up, 6 in 2024-09-13T14:27:13.322 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:13.346 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:13.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:13.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:13.449 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:13.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:13.472 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:13.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:13.573 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:13.596 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:13.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:13.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:13.698 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:13.720 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:13.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:13.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:13.822 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:13.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:13.858 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:13.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:13.959 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:13.982 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:13.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:13.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:14.084 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:14.108 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:14.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:14.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:14.123 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 43 16 80826 80810 117.443 2.53125 0.0164751 0.00851014 2024-09-13T14:27:14.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:13 smithi102 ceph-mon[110183]: osdmap e125: 8 total, 8 up, 6 in 2024-09-13T14:27:14.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:13 smithi102 ceph-mon[110183]: osdmap e126: 8 total, 8 up, 6 in 2024-09-13T14:27:14.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:13 smithi102 ceph-mon[115269]: osdmap e125: 8 total, 8 up, 6 in 2024-09-13T14:27:14.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:13 smithi102 ceph-mon[115269]: osdmap e126: 8 total, 8 up, 6 in 2024-09-13T14:27:14.210 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:27:14.235 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:14.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:14.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:27:14.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:13 smithi195 ceph-mon[104262]: osdmap e125: 8 total, 8 up, 6 in 2024-09-13T14:27:14.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:13 smithi195 ceph-mon[104262]: osdmap e126: 8 total, 8 up, 6 in 2024-09-13T14:27:14.336 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:14.378 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:14.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:14.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:14.479 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:14.506 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:14.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:14.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:14.609 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:27:14.640 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:27:14.664 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:14.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:14.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:27:14.766 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:14.807 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:14.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:14.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:14.882 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:14 smithi102 ceph-mon[110183]: pgmap v332: 89 pgs: 89 active+clean; 4.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-09-13T14:27:14.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:14 smithi102 ceph-mon[115269]: pgmap v332: 89 pgs: 89 active+clean; 4.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-09-13T14:27:14.909 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:14.943 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:14.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:14.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:15.045 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:27:15.073 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:15.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:15.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:27:15.123 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 44 16 81752 81736 116.089 57.875 0.00787359 0.00860954 2024-09-13T14:27:15.175 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:27:15.359 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:14 smithi195 ceph-mon[104262]: pgmap v332: 89 pgs: 89 active+clean; 4.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-09-13T14:27:15.388 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:27:15.388 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:27:15.391 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:15.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:15.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:27:15.493 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:15.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:15.530 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:15.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:15.605 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:27:15.605 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:27:15.626 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:27:15.626 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:27:15.631 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:27:15.651 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:27:15.651 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:27:15.660 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:15.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:15.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:27:15.675 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:27:15.675 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:27:15.718 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:27:15.718 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:27:15.744 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:27:15.744 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:27:15.762 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:15.775 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:27:15.776 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:27:15.790 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:15.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:15.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:15.810 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:27:15.811 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:27:15.892 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:15.928 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:15.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:15.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:16.030 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:16.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:16.075 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:16.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:16.123 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 45 16 82650 82634 114.757 56.125 0.0127708 0.00870921 2024-09-13T14:27:16.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:15 smithi102 ceph-mon[110183]: osdmap e127: 8 total, 8 up, 6 in 2024-09-13T14:27:16.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:15 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3824234143' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:27:16.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:15 smithi102 ceph-mon[115269]: osdmap e127: 8 total, 8 up, 6 in 2024-09-13T14:27:16.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:15 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3824234143' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:27:16.176 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:27:16.204 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:16.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:16.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:27:16.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:15 smithi195 ceph-mon[104262]: osdmap e127: 8 total, 8 up, 6 in 2024-09-13T14:27:16.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:15 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3824234143' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:27:16.306 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:16.315 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T14:27:16.332 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:16.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:16.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:16.435 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:27:16.465 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:16.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:16.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:27:16.567 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:16.593 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:16.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:16.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:16.604 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:27:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:27:16] "GET /metrics HTTP/1.1" 200 36513 "" "Prometheus/2.51.0" 2024-09-13T14:27:16.695 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:16.720 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:16.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:16.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:16.822 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:27:16.847 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:16.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:16.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:27:16.949 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:16.980 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:16.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:16.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:17.084 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:17.123 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 46 16 83538 83522 113.468 55.5 0.0199942 0.0088067 2024-09-13T14:27:17.126 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:17.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:17.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:17.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:16 smithi102 ceph-mon[110183]: pgmap v335: 89 pgs: 1 remapped+peering, 88 active+clean; 4.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 36 MiB/s wr, 571 op/s 2024-09-13T14:27:17.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:16 smithi102 ceph-mon[110183]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-13T14:27:17.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:16 smithi102 ceph-mon[110183]: osdmap e128: 8 total, 8 up, 6 in 2024-09-13T14:27:17.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:16 smithi102 ceph-mon[115269]: pgmap v335: 89 pgs: 1 remapped+peering, 88 active+clean; 4.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 36 MiB/s wr, 571 op/s 2024-09-13T14:27:17.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:16 smithi102 ceph-mon[115269]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-13T14:27:17.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:16 smithi102 ceph-mon[115269]: osdmap e128: 8 total, 8 up, 6 in 2024-09-13T14:27:17.228 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:17.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:16 smithi195 ceph-mon[104262]: pgmap v335: 89 pgs: 1 remapped+peering, 88 active+clean; 4.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 36 MiB/s wr, 571 op/s 2024-09-13T14:27:17.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:16 smithi195 ceph-mon[104262]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-13T14:27:17.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:16 smithi195 ceph-mon[104262]: osdmap e128: 8 total, 8 up, 6 in 2024-09-13T14:27:17.269 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:17.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:17.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:17.370 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:17.409 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:17.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:17.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:17.511 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:27:17.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:17.544 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:17.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:27:17.645 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:17.690 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:17.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:17.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:17.792 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:17.834 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:17.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:17.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:17.936 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:17.961 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:17.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:17.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:18.064 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:27:18.091 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:18.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:18.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:27:18.123 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 47 16 84421 84405 112.228 55.1875 0.0113181 0.00890457 2024-09-13T14:27:18.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:17 smithi102 ceph-mon[115269]: Health check failed: Degraded data redundancy: 25440/247830 objects degraded (10.265%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:27:18.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:17 smithi102 ceph-mon[110183]: Health check failed: Degraded data redundancy: 25440/247830 objects degraded (10.265%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:27:18.193 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:18.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:18.222 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:18.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:18.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:17 smithi195 ceph-mon[104262]: Health check failed: Degraded data redundancy: 25440/247830 objects degraded (10.265%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:27:18.322 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:18.362 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:18.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:18.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:18.464 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:18.501 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:18.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:18.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:18.603 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:18.633 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:18.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:18.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:18.735 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:18.769 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:18.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:18.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:18.842 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 7] out_osds: [1, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:27:18.842 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:27:18.842 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-09-13T14:27:18.842 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T14:27:18.843 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T14:27:18.843 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-09-13T14:27:18.871 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:18.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:18.951 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:18.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:19.053 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:19.087 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:18 smithi102 ceph-mon[115269]: pgmap v337: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 81 active+clean; 5.0 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 40 MiB/s wr, 642 op/s; 25440/247830 objects degraded (10.265%); 4809/247830 objects misplaced (1.940%); 1.2 MiB/s, 2 keys/s, 23 objects/s recovering 2024-09-13T14:27:19.088 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:18 smithi102 ceph-mon[110183]: pgmap v337: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 81 active+clean; 5.0 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 40 MiB/s wr, 642 op/s; 25440/247830 objects degraded (10.265%); 4809/247830 objects misplaced (1.940%); 1.2 MiB/s, 2 keys/s, 23 objects/s recovering 2024-09-13T14:27:19.106 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:19.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:19.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:19.123 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 48 16 85274 85258 111.001 53.3125 0.0149436 0.00900365 2024-09-13T14:27:19.208 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:19.240 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:27:19.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:18 smithi195 ceph-mon[104262]: pgmap v337: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 81 active+clean; 5.0 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 40 MiB/s wr, 642 op/s; 25440/247830 objects degraded (10.265%); 4809/247830 objects misplaced (1.940%); 1.2 MiB/s, 2 keys/s, 23 objects/s recovering 2024-09-13T14:27:19.268 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:19.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:19.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:19.371 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:19.400 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:19.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:19.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:19.503 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:19.563 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:19.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:19.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:19.667 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:19.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:19.705 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:19.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:19.806 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:19.833 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:19.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:19.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:19.846 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:27:19.934 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:19.975 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:19.976 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:19 smithi102 ceph-mon[110183]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-13T14:27:19.976 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:19 smithi102 ceph-mon[115269]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-13T14:27:19.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:19.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:19.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:19 smithi195 ceph-mon[104262]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-13T14:27:20.005 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:27:20.078 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:20.119 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:20.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:20.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:20.123 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 49 16 86106 86090 109.797 52 0.0218587 0.00910168 2024-09-13T14:27:20.221 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:20.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:20.270 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:20.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:20.371 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:20.417 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:20.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:20.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:20.519 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:20.566 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:20.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:20.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:20.669 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:20.712 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:20.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:20.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:20.814 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:20.839 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:20.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:20.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:20.941 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:20.975 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is set 2024-09-13T14:27:20.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:20.982 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:20.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:20.993 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:20 smithi102 ceph-mon[110183]: pgmap v338: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.1 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 32 MiB/s wr, 514 op/s; 25440/248532 objects degraded (10.236%); 4809/248532 objects misplaced (1.935%); 896 KiB/s, 1 keys/s, 19 objects/s recovering 2024-09-13T14:27:20.993 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:20 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/762563882' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:27:20.993 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:20 smithi102 ceph-mon[115269]: pgmap v338: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.1 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 32 MiB/s wr, 514 op/s; 25440/248532 objects degraded (10.236%); 4809/248532 objects misplaced (1.935%); 896 KiB/s, 1 keys/s, 19 objects/s recovering 2024-09-13T14:27:20.993 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:20 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/762563882' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:27:21.083 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:21.111 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:21.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:21.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:21.123 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 50 16 86883 86867 108.572 48.5625 0.0130102 0.00920521 2024-09-13T14:27:21.214 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:21.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:20 smithi195 ceph-mon[104262]: pgmap v338: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.1 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 32 MiB/s wr, 514 op/s; 25440/248532 objects degraded (10.236%); 4809/248532 objects misplaced (1.935%); 896 KiB/s, 1 keys/s, 19 objects/s recovering 2024-09-13T14:27:21.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:20 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/762563882' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:27:21.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:21.255 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:21.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:21.356 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:21.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:21.410 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:21.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:21.511 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:21.553 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:21.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:21.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:21.654 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:21.682 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:21.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:21.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:21.785 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:27:21.811 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:21.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:21.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:27:21.912 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:27:21.938 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:21.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:21.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:27:22.040 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:22.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:22.072 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:22.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:22.123 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 51 16 87724 87708 107.474 52.5625 0.0247557 0.00929975 2024-09-13T14:27:22.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:21 smithi102 ceph-mon[115269]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:27:22.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:21 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/762563882' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:27:22.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:21 smithi102 ceph-mon[115269]: osdmap e129: 8 total, 8 up, 6 in 2024-09-13T14:27:22.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:21 smithi102 ceph-mon[110183]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:27:22.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:21 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/762563882' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:27:22.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:21 smithi102 ceph-mon[110183]: osdmap e129: 8 total, 8 up, 6 in 2024-09-13T14:27:22.173 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:22.217 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:22.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:22.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:22.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:21 smithi195 ceph-mon[104262]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:27:22.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:21 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/762563882' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:27:22.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:21 smithi195 ceph-mon[104262]: osdmap e129: 8 total, 8 up, 6 in 2024-09-13T14:27:22.319 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:27:22.351 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:22.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:22.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:27:22.453 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:22.477 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:22.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:22.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:22.566 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:27:22.579 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:22.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:22.605 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:22.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:22.706 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:27:22.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:22.738 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:22.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:27:22.839 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:27:22.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:22.878 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:22.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:27:22.979 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:23.006 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:23.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:23.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:23.108 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:27:23.124 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 52 16 88484 88468 106.319 47.5 0.0170535 0.00940036 2024-09-13T14:27:23.137 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:23.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:23.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:27:23.190 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:27:23.191 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [], 2024-09-13T14:27:23.191 INFO:teuthology.orchestra.run.smithi102.stdout: "complaint_time": 30, 2024-09-13T14:27:23.191 INFO:teuthology.orchestra.run.smithi102.stdout: "num_blocked_ops": 0 2024-09-13T14:27:23.191 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:27:23.191 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:23 smithi195 ceph-mon[104262]: pgmap v340: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.2 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 48 MiB/s wr, 770 op/s; 25440/255006 objects degraded (9.976%); 4608/255006 objects misplaced (1.807%); 2.8 MiB/s, 1 keys/s, 49 objects/s recovering 2024-09-13T14:27:23.191 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:23 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:27:23.192 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:23 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:27:23.192 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:23 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:23.192 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:23 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:27:23.192 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:23 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:27:23.239 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:23.264 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:23.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:23.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:23.358 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:23 smithi102 ceph-mon[110183]: pgmap v340: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.2 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 48 MiB/s wr, 770 op/s; 25440/255006 objects degraded (9.976%); 4608/255006 objects misplaced (1.807%); 2.8 MiB/s, 1 keys/s, 49 objects/s recovering 2024-09-13T14:27:23.358 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:23 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:27:23.359 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:23 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:27:23.359 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:23 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:23.359 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:23 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:27:23.359 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:23 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:27:23.359 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:27:23.359 INFO:teuthology.orchestra.run.smithi195.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-13T14:27:23.359 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:27:23.359 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:23 smithi102 ceph-mon[115269]: pgmap v340: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.2 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 48 MiB/s wr, 770 op/s; 25440/255006 objects degraded (9.976%); 4608/255006 objects misplaced (1.807%); 2.8 MiB/s, 1 keys/s, 49 objects/s recovering 2024-09-13T14:27:23.359 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:23 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:27:23.360 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:23 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:27:23.360 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:23 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:27:23.360 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:23 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:27:23.360 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:23 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:27:23.366 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:27:23.395 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:23.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:23.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:27:23.497 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:23.539 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:23.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:23.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:23.640 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:27:23.665 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:23.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:23.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:27:23.756 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:27:23.766 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:27:23.824 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:23.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:23.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:27:23.926 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:23.957 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:23.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:23.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:24.059 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:24.088 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:24.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:24.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:24.124 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 53 16 89322 89306 105.301 52.375 0.0198851 0.00949104 2024-09-13T14:27:24.192 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:24.229 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:24.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:24.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:24.331 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:24.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:24.355 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:24.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:24.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:24 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 25440/255006 objects degraded (9.976%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:27:24.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:24 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 25440/255006 objects degraded (9.976%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:27:24.456 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:24.487 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:24.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:24.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:24.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:24 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 25440/255006 objects degraded (9.976%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:27:24.589 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:27:24.616 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:24.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:24.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:27:24.718 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:24.743 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:24.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:24.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:24.845 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:24.888 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:24.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:24.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:24.989 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:25.019 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:25.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:25.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:25.121 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:25.124 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 54 16 90160 90144 104.321 52.375 0.0331075 0.00957966 2024-09-13T14:27:25.152 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:25.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:25.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:25.254 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:25.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:25.282 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:25.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:25.383 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:25.398 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:25 smithi195 ceph-mon[104262]: pgmap v341: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.3 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 53 MiB/s wr, 845 op/s; 25440/260481 objects degraded (9.767%); 4608/260481 objects misplaced (1.769%); 2.2 MiB/s, 1 keys/s, 39 objects/s recovering 2024-09-13T14:27:25.415 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:25.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:25.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:25.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:25 smithi102 ceph-mon[110183]: pgmap v341: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.3 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 53 MiB/s wr, 845 op/s; 25440/260481 objects degraded (9.767%); 4608/260481 objects misplaced (1.769%); 2.2 MiB/s, 1 keys/s, 39 objects/s recovering 2024-09-13T14:27:25.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:25 smithi102 ceph-mon[115269]: pgmap v341: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.3 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 53 MiB/s wr, 845 op/s; 25440/260481 objects degraded (9.767%); 4608/260481 objects misplaced (1.769%); 2.2 MiB/s, 1 keys/s, 39 objects/s recovering 2024-09-13T14:27:25.517 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:25.561 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:25.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:25.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:25.664 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:25.707 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:25.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:25.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:25.809 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:25.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:25.848 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:25.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:25.949 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:25.975 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:25.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:25.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:26.077 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:27:26.105 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:26.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:26.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:27:26.124 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 55 16 90828 90812 103.184 41.75 0.011388 0.00968413 2024-09-13T14:27:26.207 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:26.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:26.250 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:26.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:26.352 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:26.377 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:26.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:26.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:26.479 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:26.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:26 smithi195 ceph-mon[104262]: pgmap v342: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.3 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 47 MiB/s wr, 745 op/s; 25440/261234 objects degraded (9.738%); 4608/261234 objects misplaced (1.764%); 1.9 MiB/s, 1 keys/s, 33 objects/s recovering 2024-09-13T14:27:26.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:27:26.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:27:26.525 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:26.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:26.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:26.627 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:26.662 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:26.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:26.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:26.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:27:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:27:26] "GET /metrics HTTP/1.1" 200 36542 "" "Prometheus/2.51.0" 2024-09-13T14:27:26.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:26 smithi102 ceph-mon[115269]: pgmap v342: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.3 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 47 MiB/s wr, 745 op/s; 25440/261234 objects degraded (9.738%); 4608/261234 objects misplaced (1.764%); 1.9 MiB/s, 1 keys/s, 33 objects/s recovering 2024-09-13T14:27:26.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:27:26.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:27:26.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:26 smithi102 ceph-mon[110183]: pgmap v342: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.3 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 47 MiB/s wr, 745 op/s; 25440/261234 objects degraded (9.738%); 4608/261234 objects misplaced (1.764%); 1.9 MiB/s, 1 keys/s, 33 objects/s recovering 2024-09-13T14:27:26.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:27:26.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:27:26.764 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:26.788 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:26.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:26.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:26.891 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:26.924 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:26.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:26.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:27.026 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:27.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:27.064 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:27.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:27.124 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 56 16 91511 91495 102.103 42.6875 0.017439 0.00978881 2024-09-13T14:27:27.165 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:27.192 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:27.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:27.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:27.294 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:27:27.318 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:27.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:27.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:27:27.420 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:27.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:27.460 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:27.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:27.561 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:27.588 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:27.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:27.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:27.689 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:27:27.714 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:27.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:27.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:27:27.763 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T14:27:27.817 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:27.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:27.860 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:27.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:27.961 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:27:27.988 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:27.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:27.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:27:28.090 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:28.124 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 57 16 92271 92255 101.145 47.5 0.0188743 0.00988109 2024-09-13T14:27:28.130 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:28.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:28.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:28.232 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:27:28.267 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:28.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:28.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:27:28.282 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:27:28.370 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:27:28.396 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:28.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:28.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:27:28.498 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:28.544 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:28.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:28.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:28.646 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:28.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:28 smithi102 ceph-mon[110183]: pgmap v343: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.5 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 51 MiB/s wr, 815 op/s; 25440/272289 objects degraded (9.343%); 4417/272289 objects misplaced (1.622%); 2.4 MiB/s, 40 objects/s recovering 2024-09-13T14:27:28.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:28 smithi102 ceph-mon[115269]: pgmap v343: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.5 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 51 MiB/s wr, 815 op/s; 25440/272289 objects degraded (9.343%); 4417/272289 objects misplaced (1.622%); 2.4 MiB/s, 40 objects/s recovering 2024-09-13T14:27:28.685 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:28.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:28.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:28.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:28 smithi195 ceph-mon[104262]: pgmap v343: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.5 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 51 MiB/s wr, 815 op/s; 25440/272289 objects degraded (9.343%); 4417/272289 objects misplaced (1.622%); 2.4 MiB/s, 40 objects/s recovering 2024-09-13T14:27:28.787 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:28.822 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:28.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:28.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:28.924 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:28.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:28.963 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:28.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:29.064 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:29.096 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:29.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:29.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:29.116 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 7] out_osds: [1, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:27:29.116 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:27:29.116 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-09-13T14:27:29.117 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.0 reset_purged_snaps_last 2024-09-13T14:27:29.124 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 58 16 93116 93100 100.312 52.8125 0.0231093 0.00996345 2024-09-13T14:27:29.199 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:27:29.229 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:29.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:29.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:27:29.331 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:29.359 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:29.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:29.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:29.462 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:29.497 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:29.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:29.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:29.580 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:29 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 25440/272289 objects degraded (9.343%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:27:29.580 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:29 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 25440/272289 objects degraded (9.343%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:27:29.598 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:29.642 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:29.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:29.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:29.744 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:29.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:29 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 25440/272289 objects degraded (9.343%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:27:29.771 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:29.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:29.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:29.873 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:29.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:29.916 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:29.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:29.916 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:27:29.917 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:27:29.917 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:27:29.917 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:27:29.917 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:27:29.917 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:27:29.917 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:27:29.917 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:27:29.917 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:27:29.917 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:27:29.917 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:27:29.918 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:27:29.918 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:27:29.918 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:27:29.920 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:27:29.920 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:27:30.016 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:27:30.042 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:30.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:30.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:27:30.125 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 59 16 93947 93931 99.4919 51.9375 0.0272853 0.0100449 2024-09-13T14:27:30.144 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:30.170 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:30.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:30.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:30.272 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:30.316 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:30.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:30.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:30.418 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:27:30.450 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:30.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:30.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:27:30.521 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:30 smithi102 ceph-mon[110183]: pgmap v344: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.6 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 51 MiB/s wr, 817 op/s; 25440/273063 objects degraded (9.317%); 4417/273063 objects misplaced (1.618%); 2.4 MiB/s, 39 objects/s recovering 2024-09-13T14:27:30.521 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:30 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/319623392' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:27:30.521 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:30 smithi102 ceph-mon[115269]: pgmap v344: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.6 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 51 MiB/s wr, 817 op/s; 25440/273063 objects degraded (9.317%); 4417/273063 objects misplaced (1.618%); 2.4 MiB/s, 39 objects/s recovering 2024-09-13T14:27:30.521 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:30 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/319623392' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:27:30.553 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:30.586 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:30.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:30.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:30.688 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:30.722 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:30.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:30.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:30.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:30 smithi195 ceph-mon[104262]: pgmap v344: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.6 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 51 MiB/s wr, 817 op/s; 25440/273063 objects degraded (9.317%); 4417/273063 objects misplaced (1.618%); 2.4 MiB/s, 39 objects/s recovering 2024-09-13T14:27:30.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:30 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/319623392' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:27:30.824 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:30.865 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:30.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:30.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:30.967 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:31.006 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:27:31.016 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:31.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:31.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:31.118 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:31.124 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:2024-09-13T14:27:31.124954+0000 min lat: 0.00124054 max lat: 2.25079 avg lat: 0.0101259 2024-09-13T14:27:31.125 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:27:31.125 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 60 16 94779 94763 98.7003 52 0.0118442 0.0101259 2024-09-13T14:27:31.144 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:31.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:31.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:31.246 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:31.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:31.288 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:31.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:31.389 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:27:31.417 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:31.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:31.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:27:31.519 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:31.556 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:31.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:31.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:31.658 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:31.701 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:31.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:31.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:31.803 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:31.836 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:31.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:31.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:31.938 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:31.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:31.981 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:31.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:32.082 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:32.110 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:32.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:32.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:32.124 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 61 16 95614 95598 97.9378 52.1875 0.0188006 0.0102049 2024-09-13T14:27:32.213 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:32.239 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:32.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:32.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:32.340 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:32.366 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:32.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:32.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:32.469 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:32.509 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:32.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:32.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:32.611 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:32.646 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:32.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:32.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:32.737 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:32 smithi102 ceph-mon[115269]: pgmap v345: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.6 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 57 MiB/s wr, 911 op/s; 25440/276660 objects degraded (9.195%); 4216/276660 objects misplaced (1.524%); 3.6 MiB/s, 57 objects/s recovering 2024-09-13T14:27:32.737 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:32 smithi102 ceph-mon[110183]: pgmap v345: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.6 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 57 MiB/s wr, 911 op/s; 25440/276660 objects degraded (9.195%); 4216/276660 objects misplaced (1.524%); 3.6 MiB/s, 57 objects/s recovering 2024-09-13T14:27:32.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:32 smithi195 ceph-mon[104262]: pgmap v345: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.6 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 57 MiB/s wr, 911 op/s; 25440/276660 objects degraded (9.195%); 4216/276660 objects misplaced (1.524%); 3.6 MiB/s, 57 objects/s recovering 2024-09-13T14:27:32.748 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:32.785 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:32.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:32.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:32.887 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:32.918 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:32.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:32.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:32.987 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:27:33.020 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:33.041 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:27:33.061 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:33.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:33.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:33.126 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 62 16 96395 96379 97.1427 48.8125 0.00757246 0.0102878 2024-09-13T14:27:33.162 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:27:33.197 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:33.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:33.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:27:33.299 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:33.327 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:33.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:33.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:33.429 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:27:33.448 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is set 2024-09-13T14:27:33.463 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:33.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:33.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:27:33.565 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:33.605 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:33.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:33.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:33.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:33 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3638329888' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:27:33.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:33 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3638329888' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:27:33.707 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:33.746 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:33.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:33.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:33.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:33 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3638329888' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:27:33.849 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:33.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:33.897 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:33.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:33.998 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:34.037 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:34.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:34.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:34.126 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 63 16 97202 97186 96.4013 50.4375 0.0104383 0.0103676 2024-09-13T14:27:34.139 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:34.173 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:34.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:34.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:34.275 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:34.315 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:34.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:34.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:34.366 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:27:34.366 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:27:34.417 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:34.456 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:34.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:34.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:34.558 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:34.594 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:34.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:34.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:34.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:34 smithi102 ceph-mon[110183]: pgmap v346: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.8 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 51 MiB/s wr, 811 op/s; 25440/284235 objects degraded (8.950%); 4216/284235 objects misplaced (1.483%); 2.0 MiB/s, 32 objects/s recovering 2024-09-13T14:27:34.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:34 smithi102 ceph-mon[110183]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:27:34.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:34 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3638329888' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:27:34.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:34 smithi102 ceph-mon[110183]: osdmap e130: 8 total, 8 up, 6 in 2024-09-13T14:27:34.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:34 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 25440/284235 objects degraded (8.950%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:27:34.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:34 smithi102 ceph-mon[115269]: pgmap v346: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.8 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 51 MiB/s wr, 811 op/s; 25440/284235 objects degraded (8.950%); 4216/284235 objects misplaced (1.483%); 2.0 MiB/s, 32 objects/s recovering 2024-09-13T14:27:34.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:34 smithi102 ceph-mon[115269]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:27:34.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:34 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3638329888' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:27:34.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:34 smithi102 ceph-mon[115269]: osdmap e130: 8 total, 8 up, 6 in 2024-09-13T14:27:34.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:34 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 25440/284235 objects degraded (8.950%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:27:34.696 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:34.737 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:34.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:34.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:34.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:34 smithi195 ceph-mon[104262]: pgmap v346: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.8 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 51 MiB/s wr, 811 op/s; 25440/284235 objects degraded (8.950%); 4216/284235 objects misplaced (1.483%); 2.0 MiB/s, 32 objects/s recovering 2024-09-13T14:27:34.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:34 smithi195 ceph-mon[104262]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:27:34.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:34 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3638329888' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:27:34.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:34 smithi195 ceph-mon[104262]: osdmap e130: 8 total, 8 up, 6 in 2024-09-13T14:27:34.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:34 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 25440/284235 objects degraded (8.950%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:27:34.840 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:27:34.866 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:34.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:34.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:27:34.969 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:27:35.000 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:35.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:35.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:27:35.103 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:35.126 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 64 16 98031 98015 95.7045 51.8125 0.0262848 0.0104425 2024-09-13T14:27:35.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:35.145 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:35.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:35.246 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:27:35.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:35.275 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:35.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:27:35.376 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:35.414 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:35.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:35.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:35.440 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:27:35.516 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:35.552 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:35.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:35.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:35.654 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:27:35.682 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:35.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:35.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:27:35.784 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:35.818 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:35.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:35.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:35.920 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:35.924 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T14:27:35.955 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:35.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:35.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:36.057 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:36.097 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:36.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:36.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:36.127 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 65 16 98849 98833 95.0187 51.125 0.0108126 0.0105189 2024-09-13T14:27:36.158 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:27:36.198 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:36.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:36.237 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:36.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:36.338 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:27:36.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:36.367 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:36.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:27:36.468 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:36.497 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:36.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:36.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:36.599 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:36.624 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:36.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:36.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:36.726 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:27:36.745 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:36 smithi102 ceph-mon[110183]: pgmap v348: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.8 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 50 MiB/s wr, 793 op/s; 25440/285066 objects degraded (8.924%); 4216/285066 objects misplaced (1.479%); 2.4 MiB/s, 39 objects/s recovering 2024-09-13T14:27:36.745 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:36 smithi102 ceph-mon[115269]: pgmap v348: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.8 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 50 MiB/s wr, 793 op/s; 25440/285066 objects degraded (8.924%); 4216/285066 objects misplaced (1.479%); 2.4 MiB/s, 39 objects/s recovering 2024-09-13T14:27:36.746 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:27:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:27:36] "GET /metrics HTTP/1.1" 200 36542 "" "Prometheus/2.51.0" 2024-09-13T14:27:36.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:36 smithi195 ceph-mon[104262]: pgmap v348: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.8 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 50 MiB/s wr, 793 op/s; 25440/285066 objects degraded (8.924%); 4216/285066 objects misplaced (1.479%); 2.4 MiB/s, 39 objects/s recovering 2024-09-13T14:27:36.760 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:36.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:36.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:27:36.862 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:36.898 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:36.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:36.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:37.000 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:37.024 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:37.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:37.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:37.126 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:27:37.127 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 66 16 99653 99637 94.3404 50.25 0.022206 0.0105946 2024-09-13T14:27:37.156 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:37.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:37.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:27:37.258 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:37.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:37.301 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:37.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:37.402 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:37.432 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:37.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:37.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:37.535 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:37.585 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:37.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:37.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:37.687 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:27:37.717 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:37.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:37.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:27:37.751 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:27:37.819 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:37.856 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:37.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:37.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:37.958 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:27:37.984 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:37.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:37.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:27:38.086 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:38.127 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 67 16 100439 100423 93.6655 49.125 0.0389805 0.0106706 2024-09-13T14:27:38.134 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:38.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:38.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:38.231 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:27:38.236 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:38.257 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.3 reset_purged_snaps_last 2024-09-13T14:27:38.272 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:38.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:38.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:38.374 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:38.410 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:38.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:38.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:38.486 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:38 smithi102 ceph-mon[110183]: pgmap v349: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.0 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 50 MiB/s wr, 804 op/s; 25440/296454 objects degraded (8.581%); 4002/296454 objects misplaced (1.350%); 2.6 MiB/s, 41 objects/s recovering 2024-09-13T14:27:38.486 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:38 smithi102 ceph-mon[115269]: pgmap v349: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.0 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 50 MiB/s wr, 804 op/s; 25440/296454 objects degraded (8.581%); 4002/296454 objects misplaced (1.350%); 2.6 MiB/s, 41 objects/s recovering 2024-09-13T14:27:38.512 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:27:38.549 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:38.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:38.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:27:38.651 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:38.686 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:38.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:38.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:38.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:38 smithi195 ceph-mon[104262]: pgmap v349: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.0 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 50 MiB/s wr, 804 op/s; 25440/296454 objects degraded (8.581%); 4002/296454 objects misplaced (1.350%); 2.6 MiB/s, 41 objects/s recovering 2024-09-13T14:27:38.788 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:38.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:38.827 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:38.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:38.928 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:38.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:38.969 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:38.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:39.070 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:39.106 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:39.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:39.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:39.127 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 68 16 101248 101232 93.0316 50.5625 0.0233338 0.010744 2024-09-13T14:27:39.207 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:39.250 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:39.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:39.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:39.352 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:39.389 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:39.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:39.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:39.491 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:39.532 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:39.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:39.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:39.634 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:27:39.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:39.662 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:39.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:27:39.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:39 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 25440/296454 objects degraded (8.581%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:27:39.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:39 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 25440/296454 objects degraded (8.581%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:27:39.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:39 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 25440/296454 objects degraded (8.581%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:27:39.762 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:27:39.792 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:39.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:39.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:27:39.894 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:39.924 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:39.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:39.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:40.026 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:40.070 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:40.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:40.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:40.127 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 69 16 102055 102039 92.4143 50.4375 0.0156923 0.0108155 2024-09-13T14:27:40.171 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:40.212 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:40.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:40.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:40.314 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:40.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:40.358 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:40.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:40.459 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:40.499 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:40.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:40.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:40.601 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:27:40.630 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:40.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:40.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:27:40.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:40 smithi102 ceph-mon[110183]: pgmap v350: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.0 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 50 MiB/s wr, 804 op/s; 25440/297210 objects degraded (8.560%); 4002/297210 objects misplaced (1.347%); 2.6 MiB/s, 41 objects/s recovering 2024-09-13T14:27:40.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:40 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:27:40.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:40 smithi102 ceph-mon[115269]: pgmap v350: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.0 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 50 MiB/s wr, 804 op/s; 25440/297210 objects degraded (8.560%); 4002/297210 objects misplaced (1.347%); 2.6 MiB/s, 41 objects/s recovering 2024-09-13T14:27:40.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:40 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:27:40.731 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:40.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:40 smithi195 ceph-mon[104262]: pgmap v350: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.0 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 50 MiB/s wr, 804 op/s; 25440/297210 objects degraded (8.560%); 4002/297210 objects misplaced (1.347%); 2.6 MiB/s, 41 objects/s recovering 2024-09-13T14:27:40.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:40 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:27:40.770 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:40.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:40.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:40.872 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:27:40.900 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:40.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:40.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:27:41.002 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:41.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:41.029 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:41.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:41.127 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 70 16 102862 102846 91.8146 50.4375 0.0208793 0.010886 2024-09-13T14:27:41.130 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:41.157 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:41.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:41.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:41.259 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:41.277 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:27:41.277 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:27:41.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:41.295 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:41.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:41.396 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:41.439 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:41.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:41.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:41.541 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:27:41.565 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:41.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:41.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:27:41.667 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:41.699 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:41.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:41.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:41.721 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:27:41.768 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:27:41.768 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:27:41.800 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:27:41.801 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:27:41.801 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:41.827 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:27:41.827 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:27:41.835 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:41.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:41.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:41.852 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:27:41.852 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:27:41.876 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:27:41.876 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:27:41.905 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:27:41.905 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:27:41.934 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:27:41.934 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:27:41.937 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:27:41.960 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:27:41.960 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:27:41.964 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:41.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:41.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:27:42.066 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:42.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:42.106 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:42.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:42.127 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 71 16 103650 103634 91.215 49.25 0.0105894 0.0109566 2024-09-13T14:27:42.207 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:42.243 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:42.244 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T14:27:42.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:42.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:42.346 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:42.377 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:42.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:42.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:42.470 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:42 smithi102 ceph-mon[110183]: pgmap v351: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.1 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 50 MiB/s wr, 805 op/s; 25440/300846 objects degraded (8.456%); 3779/300846 objects misplaced (1.256%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:27:42.470 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:42 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1620001244' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:27:42.479 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:42.523 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:42.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:42.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:42.625 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:42.662 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:42.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:42.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:42.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:42 smithi195 ceph-mon[104262]: pgmap v351: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.1 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 50 MiB/s wr, 805 op/s; 25440/300846 objects degraded (8.456%); 3779/300846 objects misplaced (1.256%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:27:42.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:42 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1620001244' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:27:42.764 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:42.808 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:42.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:42.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:42.836 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:42 smithi102 ceph-mon[115269]: pgmap v351: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.1 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 50 MiB/s wr, 805 op/s; 25440/300846 objects degraded (8.456%); 3779/300846 objects misplaced (1.256%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:27:42.836 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:42 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1620001244' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:27:42.910 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:42.953 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:42.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:42.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:43.055 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:43.083 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:43.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:43.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:43.127 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 72 16 104448 104432 90.6409 49.875 0.00991574 0.0110273 2024-09-13T14:27:43.185 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:43.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:43.213 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:43.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:43.314 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:43.357 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:43.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:43.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:43.458 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:27:43.492 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:43.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:43.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:27:43.594 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:27:43.621 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:43.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:43.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:27:43.723 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:43.768 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:43.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:43.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:43.870 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:43.903 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:43.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:43.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:44.005 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:44.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:44.047 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:44.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:44.127 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 73 16 105235 105219 90.073 49.1875 0.0170596 0.0110964 2024-09-13T14:27:44.148 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:44.189 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:44.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:44.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:44.211 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:27:44.293 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:44.320 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:44.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:44.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:44.422 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:44.458 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:44.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:44.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:44.488 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is unset 2024-09-13T14:27:44.559 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:44.590 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:44.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:44.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:44.624 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:44 smithi102 ceph-mon[110183]: pgmap v352: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.3 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 51 MiB/s wr, 811 op/s; 25440/308589 objects degraded (8.244%); 3779/308589 objects misplaced (1.225%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:27:44.624 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:44 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/4000475988' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:27:44.624 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:44 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:27:44.624 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:44 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 25440/308589 objects degraded (8.244%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:27:44.624 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:44 smithi102 ceph-mon[115269]: pgmap v352: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.3 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 51 MiB/s wr, 811 op/s; 25440/308589 objects degraded (8.244%); 3779/308589 objects misplaced (1.225%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:27:44.624 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:44 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/4000475988' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:27:44.624 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:44 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:27:44.625 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:44 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 25440/308589 objects degraded (8.244%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:27:44.692 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:44.724 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:44.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:44.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:44.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:44 smithi195 ceph-mon[104262]: pgmap v352: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.3 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 51 MiB/s wr, 811 op/s; 25440/308589 objects degraded (8.244%); 3779/308589 objects misplaced (1.225%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:27:44.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:44 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/4000475988' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:27:44.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:44 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:27:44.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:44 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 25440/308589 objects degraded (8.244%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:27:44.826 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:44.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:44.866 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:44.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:44.967 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:45.010 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:45.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:45.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:45.111 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:45.127 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 74 16 106011 105995 89.5111 48.5 0.00549654 0.0111652 2024-09-13T14:27:45.140 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:45.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:45.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:45.242 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:45.285 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:45.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:45.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:45.387 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:45.431 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:45.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:45.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:45.533 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:45.568 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:45.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:45.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:45.670 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:45.714 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:45.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:45.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:45.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:45 smithi195 ceph-mon[104262]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:27:45.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:45 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:27:45.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:45 smithi195 ceph-mon[104262]: osdmap e131: 8 total, 8 up, 6 in 2024-09-13T14:27:45.761 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:45 smithi102 ceph-mon[110183]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:27:45.761 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:45 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:27:45.761 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:45 smithi102 ceph-mon[110183]: osdmap e131: 8 total, 8 up, 6 in 2024-09-13T14:27:45.761 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:45 smithi102 ceph-mon[115269]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:27:45.761 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:45 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:27:45.761 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:45 smithi102 ceph-mon[115269]: osdmap e131: 8 total, 8 up, 6 in 2024-09-13T14:27:45.816 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:27:45.844 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:45.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:45.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:27:45.895 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:27:45.947 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:45.987 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:45.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:45.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:46.089 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:46.127 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 75 16 106782 106766 88.9601 48.1875 0.0261042 0.0112349 2024-09-13T14:27:46.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:46.166 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:46.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:46.267 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:46.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:46.311 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:46.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:46.412 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:46.441 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:46.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:46.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:46.543 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:27:46.566 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:46.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:46.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:27:46.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:46 smithi102 ceph-mon[110183]: pgmap v354: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.3 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 51 MiB/s wr, 808 op/s; 25440/309318 objects degraded (8.225%); 3779/309318 objects misplaced (1.222%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:27:46.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:46 smithi102 ceph-mon[115269]: pgmap v354: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.3 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 51 MiB/s wr, 808 op/s; 25440/309318 objects degraded (8.225%); 3779/309318 objects misplaced (1.222%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:27:46.669 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:27:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:27:46] "GET /metrics HTTP/1.1" 200 36541 "" "Prometheus/2.51.0" 2024-09-13T14:27:46.669 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:46.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:46.695 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:46.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:46.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:46 smithi195 ceph-mon[104262]: pgmap v354: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.3 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 51 MiB/s wr, 808 op/s; 25440/309318 objects degraded (8.225%); 3779/309318 objects misplaced (1.222%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:27:46.796 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:27:46.822 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:46.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:46.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:27:46.924 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:46.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:46.963 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:46.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:47.064 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:47.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:47.110 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:47.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:47.128 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 76 16 107560 107544 88.4293 48.625 0.0256977 0.0113038 2024-09-13T14:27:47.211 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:47.258 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:47.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:47.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:47.360 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:47.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:47.401 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:47.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:47.502 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:47.513 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:27:47.536 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:47.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:47.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:47.638 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:47.670 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:47.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:47.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:47.701 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:27:47.757 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:27:47.757 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [], 2024-09-13T14:27:47.757 INFO:teuthology.orchestra.run.smithi102.stdout: "complaint_time": 30, 2024-09-13T14:27:47.757 INFO:teuthology.orchestra.run.smithi102.stdout: "num_blocked_ops": 0 2024-09-13T14:27:47.757 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:27:47.772 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:47.808 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:47.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:47.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:47.910 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:47.936 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:47.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:47.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:48.038 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:48.071 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:48.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:48.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:48.128 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 77 16 108334 108318 87.9091 48.375 0.0370185 0.0113702 2024-09-13T14:27:48.172 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:48.196 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:48.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:48.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:48.298 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:48.344 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:48.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:48.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:48.408 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.4 reset_purged_snaps_last 2024-09-13T14:27:48.449 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:48.488 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:48.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:48.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:48.590 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:48.624 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:48.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:48.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:48.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:48 smithi102 ceph-mon[110183]: pgmap v355: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.5 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 50 MiB/s wr, 795 op/s; 25440/320340 objects degraded (7.942%); 3567/320340 objects misplaced (1.114%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:27:48.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:48 smithi102 ceph-mon[115269]: pgmap v355: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.5 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 50 MiB/s wr, 795 op/s; 25440/320340 objects degraded (7.942%); 3567/320340 objects misplaced (1.114%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:27:48.726 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:48.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:48 smithi195 ceph-mon[104262]: pgmap v355: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.5 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 50 MiB/s wr, 795 op/s; 25440/320340 objects degraded (7.942%); 3567/320340 objects misplaced (1.114%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:27:48.767 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:48.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:48.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:48.869 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:27:48.895 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:48.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:48.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:27:48.997 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:49.034 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:49.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:49.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:49.091 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T14:27:49.128 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 78 16 109109 109093 87.403 48.4375 0.033356 0.0114356 2024-09-13T14:27:49.136 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:49.170 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:49.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:49.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:49.272 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:49.308 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:49.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:49.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:49.410 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:27:49.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:49.440 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:49.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:27:49.541 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:49.584 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:49.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:49.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:49.603 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:49 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 25440/320340 objects degraded (7.942%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:27:49.604 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:49 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 25440/320340 objects degraded (7.942%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:27:49.686 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:49.720 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:49.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:49.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:49.822 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:49.867 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:49.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:49.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:49.970 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:49.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:49 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 25440/320340 objects degraded (7.942%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:27:50.007 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:50.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:50.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:50.109 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:50.128 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 79 16 109875 109859 86.9026 47.875 0.00519443 0.0115016 2024-09-13T14:27:50.157 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:50.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:50.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:50.260 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:50.298 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:50.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:50.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:50.387 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:27:50.387 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:27:50.388 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:27:50.399 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:50.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:50.438 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:50.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:50.539 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:50.574 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:50.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:50.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:50.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:50 smithi102 ceph-mon[110183]: pgmap v356: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.5 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 50 MiB/s wr, 793 op/s; 25440/321033 objects degraded (7.924%); 3567/321033 objects misplaced (1.111%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:27:50.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:50 smithi102 ceph-mon[115269]: pgmap v356: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.5 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 50 MiB/s wr, 793 op/s; 25440/321033 objects degraded (7.924%); 3567/321033 objects misplaced (1.111%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:27:50.676 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:50.709 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:50.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:50.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:50.811 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:50.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:50.841 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:50.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:50.942 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:50.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:50.979 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:50.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:50.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:50 smithi195 ceph-mon[104262]: pgmap v356: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.5 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 50 MiB/s wr, 793 op/s; 25440/321033 objects degraded (7.924%); 3567/321033 objects misplaced (1.111%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:27:51.080 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:51.113 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:51.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:51.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:51.128 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:2024-09-13T14:27:51.128432+0000 min lat: 0.00124054 max lat: 2.25079 avg lat: 0.0115667 2024-09-13T14:27:51.128 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:27:51.128 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 80 16 110632 110616 86.4077 47.3125 0.0598973 0.0115667 2024-09-13T14:27:51.215 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:51.259 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:51.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:51.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:51.361 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:51.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:51.389 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:51.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:51.490 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:51.519 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:51.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:51.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:51.621 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:51.667 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:51.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:51.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:51.769 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:51.801 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:51.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:51.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:51.903 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:51.955 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:51.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:51.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:52.057 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:52.097 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:52.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:52.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:52.128 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 81 16 111388 111372 85.9242 47.25 0.0118855 0.0116326 2024-09-13T14:27:52.200 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:27:52.225 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:52.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:52.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:27:52.327 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:52.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:52.371 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:52.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:52.472 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:52.515 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:52.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:52.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:52.617 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:52.642 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:52.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:52.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:52.744 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:27:52.769 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:52.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:52.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:27:52.821 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:52 smithi102 ceph-mon[110183]: pgmap v357: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.6 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 49 MiB/s wr, 791 op/s; 25440/324609 objects degraded (7.837%); 3347/324609 objects misplaced (1.031%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:27:52.821 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:52 smithi102 ceph-mon[115269]: pgmap v357: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.6 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 49 MiB/s wr, 791 op/s; 25440/324609 objects degraded (7.837%); 3347/324609 objects misplaced (1.031%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:27:52.870 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:52.911 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:52.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:52.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:52.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:52 smithi195 ceph-mon[104262]: pgmap v357: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.6 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 49 MiB/s wr, 791 op/s; 25440/324609 objects degraded (7.837%); 3347/324609 objects misplaced (1.031%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:27:53.012 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:53.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:53.049 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:53.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:53.128 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 82 16 112165 112149 85.4686 48.5625 0.0320038 0.0116952 2024-09-13T14:27:53.151 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:53.178 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:53.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:53.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:53.280 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:53.327 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:53.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:53.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:53.429 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:27:53.455 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:53.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:53.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:27:53.557 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:53.601 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:53.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:53.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:53.703 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:53.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:53.750 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:53.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:53.850 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:53.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:53.893 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:53.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:53.994 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:54.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:54.030 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:54.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:54.129 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 83 16 112899 112883 84.9915 45.875 0.0146837 0.0117587 2024-09-13T14:27:54.131 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:54.152 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:54.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:54.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:54.254 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:54.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:54.293 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:54.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:54.394 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:54.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:54.416 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:54.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:54.517 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:54.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:54.559 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:54.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:54.661 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:54.686 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:54.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:54.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:54.788 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:27:54.812 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:54.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:54.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:27:54.913 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:54.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:54 smithi102 ceph-mon[110183]: pgmap v358: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 49 MiB/s wr, 777 op/s; 25440/331917 objects degraded (7.665%); 3347/331917 objects misplaced (1.008%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:27:54.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:54 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 25440/331917 objects degraded (7.665%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:27:54.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:54 smithi102 ceph-mon[115269]: pgmap v358: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 49 MiB/s wr, 777 op/s; 25440/331917 objects degraded (7.665%); 3347/331917 objects misplaced (1.008%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:27:54.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:54 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 25440/331917 objects degraded (7.665%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:27:54.941 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:54.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:54.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:54.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:54 smithi195 ceph-mon[104262]: pgmap v358: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 49 MiB/s wr, 777 op/s; 25440/331917 objects degraded (7.665%); 3347/331917 objects misplaced (1.008%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:27:54.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:54 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 25440/331917 objects degraded (7.665%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:27:55.043 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:55.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:55.086 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:55.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:55.128 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 84 16 113586 113570 84.4908 42.9375 0.0215136 0.011831 2024-09-13T14:27:55.187 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:55.234 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:55.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:55.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:55.336 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:55.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:55.359 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:55.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:55.461 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:55.504 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:55.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:55.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:55.606 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:55.634 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:55.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:55.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:55.736 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:55.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:55.766 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:55.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:55.819 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:55 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:27:55.819 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:55 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:27:55.867 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:55.898 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:55 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:27:55.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:55.908 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:55.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:56.009 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:56.040 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:56.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:56.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:56.129 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 85 16 114291 114275 84.0145 44.0625 0.0136864 0.0118973 2024-09-13T14:27:56.142 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:27:56.173 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:56.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:56.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:27:56.275 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:56.309 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:56.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:56.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:56.411 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:56.437 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:56.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:56.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:56.540 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:56.574 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:56.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:56.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:56.676 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:56.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:56.718 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:56.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:56.819 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:27:56.852 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:56.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:56.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:27:56.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:56 smithi102 ceph-mon[110183]: pgmap v359: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 45 MiB/s wr, 720 op/s; 25440/332637 objects degraded (7.648%); 3347/332637 objects misplaced (1.006%); 2.5 MiB/s, 40 objects/s recovering 2024-09-13T14:27:56.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:56 smithi102 ceph-mon[115269]: pgmap v359: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 45 MiB/s wr, 720 op/s; 25440/332637 objects degraded (7.648%); 3347/332637 objects misplaced (1.006%); 2.5 MiB/s, 40 objects/s recovering 2024-09-13T14:27:56.919 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:27:56 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:27:56] "GET /metrics HTTP/1.1" 200 36538 "" "Prometheus/2.51.0" 2024-09-13T14:27:56.953 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:27:56.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:56.982 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:56.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:27:56.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:56 smithi195 ceph-mon[104262]: pgmap v359: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 45 MiB/s wr, 720 op/s; 25440/332637 objects degraded (7.648%); 3347/332637 objects misplaced (1.006%); 2.5 MiB/s, 40 objects/s recovering 2024-09-13T14:27:57.083 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:27:57.110 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:57.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:57.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:27:57.129 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 86 16 115052 115036 83.5906 47.5625 0.0229919 0.0119572 2024-09-13T14:27:57.212 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:57.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:57.260 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:57.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:57.361 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:57.403 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:57.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:57.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:57.505 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:57.550 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:57.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:57.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:57.652 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:57.689 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:57.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:57.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:57.790 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:27:57.817 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:57.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:57.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:27:57.918 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:57.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:57.969 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:57.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:58.070 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:58.093 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:58.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:58.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:58.130 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 87 16 115769 115753 83.1448 44.8125 0.0336401 0.0120218 2024-09-13T14:27:58.195 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:58.237 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:58.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:58.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:58.339 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:58.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:58.376 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:58.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:58.477 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:27:58.517 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:58.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:58.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:27:58.619 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:58.665 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:58.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:58.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:58.767 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:58.800 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:58.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:58.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:58.902 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:27:58.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:58 smithi102 ceph-mon[110183]: pgmap v360: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 7.0 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 58 MiB/s wr, 934 op/s; 25440/342984 objects degraded (7.417%); 3123/342984 objects misplaced (0.911%); 3.4 MiB/s, 54 objects/s recovering 2024-09-13T14:27:58.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:58 smithi102 ceph-mon[115269]: pgmap v360: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 7.0 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 58 MiB/s wr, 934 op/s; 25440/342984 objects degraded (7.417%); 3123/342984 objects misplaced (0.911%); 3.4 MiB/s, 54 objects/s recovering 2024-09-13T14:27:58.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:58.944 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:58.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:27:58.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:58 smithi195 ceph-mon[104262]: pgmap v360: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 7.0 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 58 MiB/s wr, 934 op/s; 25440/342984 objects degraded (7.417%); 3123/342984 objects misplaced (0.911%); 3.4 MiB/s, 54 objects/s recovering 2024-09-13T14:27:59.046 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:27:59.072 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:59.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:59.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:27:59.129 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 88 16 116496 116480 82.7163 45.4375 0.0180734 0.0120843 2024-09-13T14:27:59.174 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:27:59.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:59.200 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:59.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:27:59.301 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:27:59.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:59.330 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:59.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:27:59.431 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:27:59.451 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:59.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:59.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:27:59.553 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:27:59.593 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:27:59.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:59.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:27:59.695 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:27:59.722 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:59.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:59.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:27:59.824 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:27:59.849 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:59.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:59.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:27:59.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:27:59 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 25440/342984 objects degraded (7.417%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:27:59.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:27:59 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 25440/342984 objects degraded (7.417%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:27:59.952 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:27:59.978 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:27:59.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:27:59.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:27:59.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:27:59 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 25440/342984 objects degraded (7.417%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:28:00.079 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:00.108 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:00.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:00.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:00.130 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 89 16 117191 117175 82.275 43.4375 0.0110704 0.0121493 2024-09-13T14:28:00.212 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:00.239 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:00.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:00.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:00.342 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:00.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:00.385 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:00.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:00.486 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:28:00.508 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:00.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:00.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:28:00.610 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:00.637 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:00.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:00.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:00.739 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:00.781 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:00.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:00.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:00.882 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:00.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:00 smithi102 ceph-mon[110183]: pgmap v361: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 7.0 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 40 MiB/s wr, 647 op/s; 25440/343656 objects degraded (7.403%); 3123/343656 objects misplaced (0.909%); 2.3 MiB/s, 36 objects/s recovering 2024-09-13T14:28:00.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:00 smithi102 ceph-mon[115269]: pgmap v361: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 7.0 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 40 MiB/s wr, 647 op/s; 25440/343656 objects degraded (7.403%); 3123/343656 objects misplaced (0.909%); 2.3 MiB/s, 36 objects/s recovering 2024-09-13T14:28:00.927 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:00.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:00.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:00.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:00 smithi195 ceph-mon[104262]: pgmap v361: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 7.0 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 40 MiB/s wr, 647 op/s; 25440/343656 objects degraded (7.403%); 3123/343656 objects misplaced (0.909%); 2.3 MiB/s, 36 objects/s recovering 2024-09-13T14:28:01.029 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:01.072 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:01.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:01.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:01.130 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Total time run: 90.0099 2024-09-13T14:28:01.130 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Total writes made: 117915 2024-09-13T14:28:01.130 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Write size: 65536 2024-09-13T14:28:01.130 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Object size: 65536 2024-09-13T14:28:01.130 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Bandwidth (MB/sec): 81.8764 2024-09-13T14:28:01.130 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Stddev Bandwidth: 50.2118 2024-09-13T14:28:01.130 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Max bandwidth (MB/sec): 246.938 2024-09-13T14:28:01.130 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Min bandwidth (MB/sec): 0 2024-09-13T14:28:01.130 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Average IOPS: 1310 2024-09-13T14:28:01.131 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Stddev IOPS: 803.399 2024-09-13T14:28:01.131 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Max IOPS: 3951 2024-09-13T14:28:01.131 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Min IOPS: 0 2024-09-13T14:28:01.131 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Average Latency(s): 0.0122094 2024-09-13T14:28:01.131 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Stddev Latency(s): 0.0274711 2024-09-13T14:28:01.131 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Max latency(s): 2.25079 2024-09-13T14:28:01.131 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Min latency(s): 0.00124054 2024-09-13T14:28:01.139 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Cleaning up (deleting benchmark objects) 2024-09-13T14:28:01.174 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:01.209 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:01.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:01.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:01.312 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:01.356 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:01.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:01.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:01.458 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:28:01.502 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:01.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:01.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:28:01.604 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:28:01.630 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:01.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:01.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:28:01.731 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:01.757 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:01.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:01.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:01.859 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:01.899 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:01.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:01.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:01.996 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:28:01.996 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:28:02.002 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:28:02.018 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:28:02.018 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:28:02.039 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:28:02.040 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:28:02.043 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:02.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:02.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:28:02.063 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:28:02.063 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:28:02.088 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:28:02.089 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:28:02.122 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:28:02.122 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:28:02.145 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:02.155 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:28:02.156 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:28:02.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:02.181 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:02.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:02.185 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:28:02.185 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:28:02.283 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:28:02.319 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:02.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:02.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:28:02.421 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:02.447 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:02.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:02.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:02.549 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:02.575 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:02.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:02.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:02.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:02 smithi102 ceph-mon[110183]: pgmap v362: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 7.1 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 45 MiB/s wr, 720 op/s; 25440/346896 objects degraded (7.334%); 2884/346896 objects misplaced (0.831%); 3.6 MiB/s, 56 objects/s recovering 2024-09-13T14:28:02.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:02 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/443018367' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:28:02.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:02 smithi102 ceph-mon[115269]: pgmap v362: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 7.1 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 45 MiB/s wr, 720 op/s; 25440/346896 objects degraded (7.334%); 2884/346896 objects misplaced (0.831%); 3.6 MiB/s, 56 objects/s recovering 2024-09-13T14:28:02.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:02 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/443018367' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:28:02.677 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:02.715 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:02.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:02.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:02.817 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:28:02.861 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:02.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:02.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:28:02.963 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:02.987 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:02.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:02.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:02.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:02 smithi195 ceph-mon[104262]: pgmap v362: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 7.1 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 45 MiB/s wr, 720 op/s; 25440/346896 objects degraded (7.334%); 2884/346896 objects misplaced (0.831%); 3.6 MiB/s, 56 objects/s recovering 2024-09-13T14:28:02.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:02 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/443018367' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:28:03.089 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:03.113 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:03.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:03.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:03.215 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:03.255 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:03.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:03.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:03.357 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:03.397 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:03.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:03.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:03.499 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:03.540 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:03.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:03.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:03.642 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:28:03.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:03.689 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:03.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:28:03.790 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:03.823 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:03.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:03.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:03.873 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:28:03.874 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:28:03.925 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:03.954 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:03.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:03.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:04.055 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:04.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:04.106 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:04.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:04.207 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:04.254 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:04.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:04.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:04.355 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:04.393 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:04.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:04.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:04.495 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:04.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:04.521 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:04.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:04.622 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:04.647 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:04.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:04.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:04.748 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:04.777 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T14:28:04.786 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:04.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:04.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:04.827 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:04 smithi195 ceph-mon[104262]: pgmap v363: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 7.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 85 B/s rd, 50 MiB/s wr, 813 op/s; 25440/353007 objects degraded (7.207%); 2884/353007 objects misplaced (0.817%); 2.4 MiB/s, 38 objects/s recovering 2024-09-13T14:28:04.827 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:04 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 25440/353007 objects degraded (7.207%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:28:04.888 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:28:04.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:04 smithi102 ceph-mon[110183]: pgmap v363: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 7.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 85 B/s rd, 50 MiB/s wr, 813 op/s; 25440/353007 objects degraded (7.207%); 2884/353007 objects misplaced (0.817%); 2.4 MiB/s, 38 objects/s recovering 2024-09-13T14:28:04.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:04 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 25440/353007 objects degraded (7.207%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:28:04.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:04 smithi102 ceph-mon[115269]: pgmap v363: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 7.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 85 B/s rd, 50 MiB/s wr, 813 op/s; 25440/353007 objects degraded (7.207%); 2884/353007 objects misplaced (0.817%); 2.4 MiB/s, 38 objects/s recovering 2024-09-13T14:28:04.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:04 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 25440/353007 objects degraded (7.207%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:28:04.932 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:04.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:04.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:28:05.026 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:28:05.034 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:05.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:05.100 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:05.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:05.154 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:28:05.201 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:05.247 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:05.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:05.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:05.348 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:05.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:05.398 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:05.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:05.499 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:05.543 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:05.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:05.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:05.557 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is unset 2024-09-13T14:28:05.576 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:05 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2544149869' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:28:05.577 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:05 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2544149869' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:28:05.645 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:05.663 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.5 reset_purged_snaps_last 2024-09-13T14:28:05.680 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:05.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:05.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:05.782 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:05.828 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:05.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:05.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:05.931 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:28:05.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:05.967 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:05.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:28:05.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:05 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2544149869' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:28:06.069 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:06.110 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:06.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:06.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:06.213 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:06.244 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:06.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:06.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:06.346 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:06.373 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:06.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:06.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:06.476 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:06.508 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:06.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:06.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:06.511 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:28:06.610 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:28:06.635 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:06.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:06.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:28:06.684 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:28:06.736 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:06.768 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:06 smithi195 ceph-mon[104262]: pgmap v364: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 7.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 85 B/s rd, 38 MiB/s wr, 628 op/s; 25440/353100 objects degraded (7.205%); 2884/353100 objects misplaced (0.817%); 2.4 MiB/s, 38 objects/s recovering 2024-09-13T14:28:06.768 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:06 smithi195 ceph-mon[104262]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:28:06.768 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:06 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2544149869' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:28:06.768 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:06 smithi195 ceph-mon[104262]: osdmap e132: 8 total, 8 up, 6 in 2024-09-13T14:28:06.788 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:06.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:06.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:06.890 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:06.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:06 smithi102 ceph-mon[110183]: pgmap v364: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 7.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 85 B/s rd, 38 MiB/s wr, 628 op/s; 25440/353100 objects degraded (7.205%); 2884/353100 objects misplaced (0.817%); 2.4 MiB/s, 38 objects/s recovering 2024-09-13T14:28:06.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:06 smithi102 ceph-mon[110183]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:28:06.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:06 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2544149869' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:28:06.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:06 smithi102 ceph-mon[110183]: osdmap e132: 8 total, 8 up, 6 in 2024-09-13T14:28:06.919 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:28:06 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:28:06] "GET /metrics HTTP/1.1" 200 36538 "" "Prometheus/2.51.0" 2024-09-13T14:28:06.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:06 smithi102 ceph-mon[115269]: pgmap v364: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 7.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 85 B/s rd, 38 MiB/s wr, 628 op/s; 25440/353100 objects degraded (7.205%); 2884/353100 objects misplaced (0.817%); 2.4 MiB/s, 38 objects/s recovering 2024-09-13T14:28:06.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:06 smithi102 ceph-mon[115269]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:28:06.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:06 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2544149869' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:28:06.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:06 smithi102 ceph-mon[115269]: osdmap e132: 8 total, 8 up, 6 in 2024-09-13T14:28:06.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:06.932 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:06.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:07.033 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:07.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:07.071 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:07.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:07.172 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:07.212 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:07.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:07.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:07.262 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:28:07.262 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:28:07.314 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:07.340 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:07.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:07.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:07.442 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:07.468 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:07.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:07.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:07.570 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:28:07.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:07.603 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:07.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:28:07.705 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:07.735 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:07.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:07.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:07.837 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:07.871 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:07.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:07.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:07.973 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:28:07.982 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T14:28:08.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:08.000 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:08.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:28:08.101 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:28:08.154 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:08.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:08.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:28:08.256 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:08.281 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:08.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:08.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:08.339 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:28:08.383 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:28:08.407 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:08.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:08.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:28:08.502 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:28:08.509 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:28:08.574 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:08.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:08.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:28:08.676 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:08.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:08.721 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:08.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:08.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:08 smithi195 ceph-mon[104262]: pgmap v366: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 7.0 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 102 B/s rd, 23 MiB/s wr, 753 op/s; 25440/342078 objects degraded (7.437%); 2637/342078 objects misplaced (0.771%); 3.0 MiB/s, 48 objects/s recovering 2024-09-13T14:28:08.822 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:08.849 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:08.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:08.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:08.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:08 smithi102 ceph-mon[110183]: pgmap v366: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 7.0 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 102 B/s rd, 23 MiB/s wr, 753 op/s; 25440/342078 objects degraded (7.437%); 2637/342078 objects misplaced (0.771%); 3.0 MiB/s, 48 objects/s recovering 2024-09-13T14:28:08.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:08 smithi102 ceph-mon[115269]: pgmap v366: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 7.0 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 102 B/s rd, 23 MiB/s wr, 753 op/s; 25440/342078 objects degraded (7.437%); 2637/342078 objects misplaced (0.771%); 3.0 MiB/s, 48 objects/s recovering 2024-09-13T14:28:08.951 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:08.976 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:08.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:08.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:09.078 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:09.104 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:09.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:09.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:09.206 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:09.243 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:09.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:09.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:09.345 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:09.371 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:09.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:09.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:09.473 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:28:09.507 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:09.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:09.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:28:09.609 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:09.652 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:09.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:09.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:09.754 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:28:09.799 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:09.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:09.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:28:09.901 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:09.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:09 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 25440/342078 objects degraded (7.437%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:28:09.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:09 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 25440/342078 objects degraded (7.437%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:28:09.942 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:09.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:09.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:09.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:09 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 25440/342078 objects degraded (7.437%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:28:10.045 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:10.069 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:10.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:10.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:10.171 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:10.213 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:10.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:10.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:10.315 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:10.346 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:10.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:10.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:10.448 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:10.474 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:10.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:10.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:10.576 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:10.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:10.626 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:10.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:10.656 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:10 smithi102 ceph-mon[110183]: pgmap v367: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.9 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 102 B/s rd, 21 MiB/s wr, 754 op/s; 25440/341385 objects degraded (7.452%); 2637/341385 objects misplaced (0.772%); 3.0 MiB/s, 48 objects/s recovering 2024-09-13T14:28:10.656 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:28:10.656 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:28:10.657 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:10 smithi102 ceph-mon[115269]: pgmap v367: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.9 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 102 B/s rd, 21 MiB/s wr, 754 op/s; 25440/341385 objects degraded (7.452%); 2637/341385 objects misplaced (0.772%); 3.0 MiB/s, 48 objects/s recovering 2024-09-13T14:28:10.657 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:28:10.657 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:28:10.716 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:10 smithi195 ceph-mon[104262]: pgmap v367: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.9 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 102 B/s rd, 21 MiB/s wr, 754 op/s; 25440/341385 objects degraded (7.452%); 2637/341385 objects misplaced (0.772%); 3.0 MiB/s, 48 objects/s recovering 2024-09-13T14:28:10.717 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:28:10.717 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:28:10.728 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:10.773 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:10.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:10.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:10.855 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:28:10.875 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:10.888 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:28:10.889 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:28:10.902 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:10.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:10.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:10.910 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:28:10.910 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:28:10.928 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:28:10.929 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:28:10.949 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:28:10.949 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:28:10.973 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:28:10.974 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:28:11.004 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:11.016 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:28:11.017 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:28:11.031 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:11.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:11.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:11.049 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:28:11.049 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:28:11.065 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:28:11.065 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-13T14:28:11.065 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-09-13T14:28:11.065 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-09-13T14:28:11.065 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:28:11.083 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:28:11.083 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:28:11.133 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:11.174 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:11.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:11.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:11.276 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:11.303 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:11.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:11.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:11.405 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:11.452 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:11.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:11.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:11.554 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:11.582 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:11.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:11.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:11.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:11 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3839234775' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:28:11.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:11 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3839234775' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:28:11.684 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:11.721 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:11.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:11.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:11.830 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:11.871 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:11.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:11.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:11.973 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:11.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:11 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3839234775' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:28:12.015 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:12.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:12.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:12.116 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:12.140 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T14:28:12.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:12.164 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:12.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:12.265 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:28:12.292 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:12.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:12.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:28:12.394 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:12.418 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.6 reset_purged_snaps_last 2024-09-13T14:28:12.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:12.457 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:12.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:12.559 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:28:12.563 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:28:12.585 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:12.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:12.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:28:12.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:12 smithi102 ceph-mon[115269]: pgmap v368: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.9 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 102 B/s rd, 14 MiB/s wr, 760 op/s; 25440/337848 objects degraded (7.530%); 2391/337848 objects misplaced (0.708%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T14:28:12.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:12 smithi102 ceph-mon[110183]: pgmap v368: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.9 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 102 B/s rd, 14 MiB/s wr, 760 op/s; 25440/337848 objects degraded (7.530%); 2391/337848 objects misplaced (0.708%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T14:28:12.688 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:12.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:12.734 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:12.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:12.835 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:12.862 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:12.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:12.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:12.894 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:12 smithi195 ceph-mon[104262]: pgmap v368: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.9 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 102 B/s rd, 14 MiB/s wr, 760 op/s; 25440/337848 objects degraded (7.530%); 2391/337848 objects misplaced (0.708%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T14:28:12.964 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:12.991 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:12.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:12.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:13.093 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:28:13.149 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:13.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:13.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:28:13.251 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:28:13.288 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:13.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:13.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:28:13.390 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:28:13.413 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:13.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:13.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:28:13.515 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:13.541 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:13.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:13.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:13.643 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:13.671 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:13.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:13.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:13.771 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:28:13.773 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:13.797 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:13.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:13.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:13.899 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:13.926 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:13.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:13.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:14.034 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:14.076 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:14.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:14.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:14.178 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:14.204 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:14.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:14.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:14.306 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:14.347 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:14.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:14.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:14.449 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:14.480 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:14.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:14.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:14.582 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:14.592 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:28:14.592 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-09-13T14:28:14.592 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-09-13T14:28:14.592 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-13T14:28:14.592 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.593 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:87399 5.e 5:792e1e51:::benchmark_data_smithi102_155989_object87398:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:28:14.593 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:21.705932+0000", 2024-09-13T14:28:14.593 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 52.882769371000002, 2024-09-13T14:28:14.593 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.103644367, 2024-09-13T14:28:14.593 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:14.593 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:14.593 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:14.593 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:14.593 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:14.593 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 87399 2024-09-13T14:28:14.593 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.594 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:14.594 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.594 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:14.594 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:21.705932+0000", 2024-09-13T14:28:14.594 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:14.594 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.594 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.594 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:14.594 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:21.705932+0000", 2024-09-13T14:28:14.594 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:14.595 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.595 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.595 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:14.595 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:21.705930+0000", 2024-09-13T14:28:14.595 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:28:14.595 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.595 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.595 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:14.595 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:21.705974+0000", 2024-09-13T14:28:14.595 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3590000000000001e-05 2024-09-13T14:28:14.595 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.595 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.596 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:14.596 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:21.705987+0000", 2024-09-13T14:28:14.596 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2801e-05 2024-09-13T14:28:14.596 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.596 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.596 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:14.596 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:21.705992+0000", 2024-09-13T14:28:14.596 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7119999999999997e-06 2024-09-13T14:28:14.596 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.596 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.597 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:14.597 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:21.769289+0000", 2024-09-13T14:28:14.597 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.063296963999999997 2024-09-13T14:28:14.597 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.597 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.597 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:14.597 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:21.769430+0000", 2024-09-13T14:28:14.597 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014089300000000001 2024-09-13T14:28:14.597 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.597 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.597 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:28:14.598 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:21.769593+0000", 2024-09-13T14:28:14.598 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000163077 2024-09-13T14:28:14.598 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.598 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.598 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:14.598 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:21.778996+0000", 2024-09-13T14:28:14.598 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0094022570000000007 2024-09-13T14:28:14.598 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.598 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.599 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:14.599 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:21.785978+0000", 2024-09-13T14:28:14.599 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.006982009 2024-09-13T14:28:14.599 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.599 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.599 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:14.599 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:21.809496+0000", 2024-09-13T14:28:14.599 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.023518125000000001 2024-09-13T14:28:14.599 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.599 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.600 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:14.600 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:21.809540+0000", 2024-09-13T14:28:14.600 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4058999999999999e-05 2024-09-13T14:28:14.600 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.600 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.600 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:14.600 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:21.809576+0000", 2024-09-13T14:28:14.601 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6535999999999997e-05 2024-09-13T14:28:14.601 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:14.601 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:14.601 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:14.601 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.601 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.601 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:87402 5.e 5:7f436efa:::benchmark_data_smithi102_155989_object87401:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:28:14.602 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:21.707746+0000", 2024-09-13T14:28:14.602 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 52.880954811999999, 2024-09-13T14:28:14.602 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.101971374, 2024-09-13T14:28:14.602 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:14.602 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:14.602 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:14.602 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:14.602 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:14.602 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 87402 2024-09-13T14:28:14.602 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.602 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:14.603 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.603 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:14.603 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:21.707746+0000", 2024-09-13T14:28:14.603 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:14.603 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.603 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.603 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:14.603 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:21.707746+0000", 2024-09-13T14:28:14.603 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:14.603 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.604 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.604 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:14.604 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:21.707746+0000", 2024-09-13T14:28:14.604 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:28:14.604 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.604 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.604 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:14.604 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:21.707775+0000", 2024-09-13T14:28:14.604 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8725e-05 2024-09-13T14:28:14.604 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.605 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.605 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:14.605 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:21.707781+0000", 2024-09-13T14:28:14.605 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4960000000000001e-06 2024-09-13T14:28:14.605 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.605 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.605 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:14.605 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:21.707786+0000", 2024-09-13T14:28:14.605 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5549999999999996e-06 2024-09-13T14:28:14.605 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.605 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.606 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:14.606 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:21.772455+0000", 2024-09-13T14:28:14.606 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.064669218000000001 2024-09-13T14:28:14.606 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.606 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.606 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:14.606 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:21.772579+0000", 2024-09-13T14:28:14.606 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012424699999999999 2024-09-13T14:28:14.606 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.606 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.606 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:28:14.607 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:21.772714+0000", 2024-09-13T14:28:14.607 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000134812 2024-09-13T14:28:14.607 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.607 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.607 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:14.607 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:21.779024+0000", 2024-09-13T14:28:14.607 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0063102330000000002 2024-09-13T14:28:14.607 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.607 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.607 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:14.607 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:21.785985+0000", 2024-09-13T14:28:14.608 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0069607879999999999 2024-09-13T14:28:14.608 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.608 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.608 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:14.608 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:21.809625+0000", 2024-09-13T14:28:14.608 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.023640352999999999 2024-09-13T14:28:14.608 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.608 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.608 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:14.608 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:21.809679+0000", 2024-09-13T14:28:14.608 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3340000000000001e-05 2024-09-13T14:28:14.608 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.609 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.609 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:14.609 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:21.809718+0000", 2024-09-13T14:28:14.609 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9076999999999999e-05 2024-09-13T14:28:14.609 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:14.609 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:14.609 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:14.609 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.609 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.610 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.44546.0:88000 5.9 e129/116 5:903d079f:::benchmark_data_smithi102_155989_object87999:head v 129'5409, mlcod=129'5405)", 2024-09-13T14:28:14.610 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:22.487055+0000", 2024-09-13T14:28:14.610 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 52.101646004000003, 2024-09-13T14:28:14.610 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.107327372, 2024-09-13T14:28:14.610 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:14.610 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:14.610 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:14.611 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.611 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:14.611 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.487055+0000", 2024-09-13T14:28:14.611 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:14.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:14.612 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.613 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.613 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:14.613 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.487055+0000", 2024-09-13T14:28:14.613 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:14.613 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.613 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.613 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:14.613 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.487055+0000", 2024-09-13T14:28:14.613 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:28:14.613 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.613 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.614 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:14.614 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.487082+0000", 2024-09-13T14:28:14.614 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7127e-05 2024-09-13T14:28:14.614 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.614 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.614 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:14.614 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.487087+0000", 2024-09-13T14:28:14.614 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4580000000000001e-06 2024-09-13T14:28:14.614 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.614 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.614 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:14.615 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.487093+0000", 2024-09-13T14:28:14.615 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.502e-06 2024-09-13T14:28:14.615 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.615 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.615 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:14.615 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.491265+0000", 2024-09-13T14:28:14.615 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0041719900000000004 2024-09-13T14:28:14.615 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.615 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.615 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:14.615 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.491307+0000", 2024-09-13T14:28:14.616 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1870999999999999e-05 2024-09-13T14:28:14.616 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.616 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.616 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:14.616 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.503687+0000", 2024-09-13T14:28:14.616 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012380276000000001 2024-09-13T14:28:14.617 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.617 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.617 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:14.617 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.594383+0000", 2024-09-13T14:28:14.617 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.090695551999999999 2024-09-13T14:28:14.617 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:14.617 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:14.617 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:14.617 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.617 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.617 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:88020 5.e 5:7278f105:::benchmark_data_smithi102_155989_object88019:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:28:14.618 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:22.488136+0000", 2024-09-13T14:28:14.618 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 52.100565490000001, 2024-09-13T14:28:14.618 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.116470484, 2024-09-13T14:28:14.618 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:14.618 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:14.618 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:14.618 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:14.618 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:14.618 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 88020 2024-09-13T14:28:14.618 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.619 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:14.619 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.619 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:14.619 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.488136+0000", 2024-09-13T14:28:14.619 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:14.619 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.619 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.619 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:14.619 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.488136+0000", 2024-09-13T14:28:14.619 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:14.619 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.619 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.620 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:14.620 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.488134+0000", 2024-09-13T14:28:14.620 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:28:14.620 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.620 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.620 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:14.620 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.488190+0000", 2024-09-13T14:28:14.620 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5541999999999998e-05 2024-09-13T14:28:14.620 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.620 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.621 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:14.621 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.488200+0000", 2024-09-13T14:28:14.621 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0465999999999999e-05 2024-09-13T14:28:14.621 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.621 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.621 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:14.621 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.488207+0000", 2024-09-13T14:28:14.621 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.601e-06 2024-09-13T14:28:14.621 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.621 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.621 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:14.621 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.490502+0000", 2024-09-13T14:28:14.622 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002294904 2024-09-13T14:28:14.622 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.622 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.622 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:14.622 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.490634+0000", 2024-09-13T14:28:14.622 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000131867 2024-09-13T14:28:14.622 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.622 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.622 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:28:14.622 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.490768+0000", 2024-09-13T14:28:14.622 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013445800000000001 2024-09-13T14:28:14.623 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.623 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.623 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:14.623 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.500446+0000", 2024-09-13T14:28:14.623 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0096782120000000003 2024-09-13T14:28:14.623 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.623 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.623 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:14.623 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.503247+0000", 2024-09-13T14:28:14.623 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0028011910000000002 2024-09-13T14:28:14.623 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.624 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.624 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:14.624 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.604320+0000", 2024-09-13T14:28:14.624 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.101072963 2024-09-13T14:28:14.624 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.624 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.624 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:14.624 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.604497+0000", 2024-09-13T14:28:14.624 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00017705999999999999 2024-09-13T14:28:14.624 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.624 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.624 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:14.625 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.604606+0000", 2024-09-13T14:28:14.625 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010888200000000001 2024-09-13T14:28:14.625 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:14.625 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:14.625 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:14.625 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.625 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.625 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:88021 5.e 5:70ef7763:::benchmark_data_smithi102_155989_object88020:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:28:14.625 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:22.488214+0000", 2024-09-13T14:28:14.625 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 52.100487033999997, 2024-09-13T14:28:14.625 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.116624456, 2024-09-13T14:28:14.625 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:14.626 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:14.626 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:14.626 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:14.626 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:14.626 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 88021 2024-09-13T14:28:14.626 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.626 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:14.626 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.626 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:14.626 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.488214+0000", 2024-09-13T14:28:14.626 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:14.626 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.627 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.627 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:14.627 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.488214+0000", 2024-09-13T14:28:14.627 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:14.627 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.627 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.627 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:14.627 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.488214+0000", 2024-09-13T14:28:14.627 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:28:14.627 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.627 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.627 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:14.628 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.488278+0000", 2024-09-13T14:28:14.628 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3907999999999999e-05 2024-09-13T14:28:14.628 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.628 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.628 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:14.628 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.488285+0000", 2024-09-13T14:28:14.628 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2640000000000002e-06 2024-09-13T14:28:14.628 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.628 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.628 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:14.628 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.488290+0000", 2024-09-13T14:28:14.629 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4519999999999996e-06 2024-09-13T14:28:14.629 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.629 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.629 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:14.629 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.493941+0000", 2024-09-13T14:28:14.629 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0056507969999999999 2024-09-13T14:28:14.629 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.629 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.629 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:14.629 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.494056+0000", 2024-09-13T14:28:14.629 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011510099999999999 2024-09-13T14:28:14.629 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.630 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.630 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:28:14.630 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.494177+0000", 2024-09-13T14:28:14.630 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012020500000000001 2024-09-13T14:28:14.630 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.630 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.630 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:14.630 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.500482+0000", 2024-09-13T14:28:14.630 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0063050329999999998 2024-09-13T14:28:14.630 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.630 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.631 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:14.631 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.594207+0000", 2024-09-13T14:28:14.631 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.093725369000000003 2024-09-13T14:28:14.631 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.631 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.631 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:14.631 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.604627+0000", 2024-09-13T14:28:14.631 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010419964 2024-09-13T14:28:14.631 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.631 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.631 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:14.631 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.604736+0000", 2024-09-13T14:28:14.632 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0001094 2024-09-13T14:28:14.632 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.632 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.632 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:14.632 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.604839+0000", 2024-09-13T14:28:14.632 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000102435 2024-09-13T14:28:14.632 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:14.632 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:14.632 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:14.632 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.632 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.633 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.44546.0:88019 5.f e129/116 5:f5db7c40:::benchmark_data_smithi102_155989_object88018:head v 129'5437, mlcod=129'5433)", 2024-09-13T14:28:14.633 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:22.489059+0000", 2024-09-13T14:28:14.633 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 52.099641859000002, 2024-09-13T14:28:14.633 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.10924980500000001, 2024-09-13T14:28:14.633 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:14.633 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:14.633 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:14.633 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.633 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:14.633 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.489059+0000", 2024-09-13T14:28:14.633 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:14.634 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.634 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.634 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:14.634 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.489059+0000", 2024-09-13T14:28:14.634 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:14.634 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.634 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.634 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:14.634 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.489059+0000", 2024-09-13T14:28:14.634 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:28:14.634 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.634 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.635 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:14.635 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.489096+0000", 2024-09-13T14:28:14.635 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7122999999999997e-05 2024-09-13T14:28:14.635 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.635 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.635 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:14.635 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.489107+0000", 2024-09-13T14:28:14.635 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1549000000000001e-05 2024-09-13T14:28:14.635 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.635 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.635 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:14.636 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.489114+0000", 2024-09-13T14:28:14.636 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5719999999999999e-06 2024-09-13T14:28:14.636 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.636 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.636 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:14.636 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.491354+0000", 2024-09-13T14:28:14.636 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002240662 2024-09-13T14:28:14.636 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.636 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.636 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:14.636 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.491371+0000", 2024-09-13T14:28:14.637 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6639000000000001e-05 2024-09-13T14:28:14.637 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.637 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.637 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:14.637 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.598210+0000", 2024-09-13T14:28:14.637 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.106838579 2024-09-13T14:28:14.637 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.637 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.637 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:14.637 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.598309+0000", 2024-09-13T14:28:14.637 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.9557999999999998e-05 2024-09-13T14:28:14.638 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:14.638 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:14.638 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:14.638 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.638 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.638 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.44546.0:88025 5.8 e129/125 5:11d6cef9:::benchmark_data_smithi102_155989_object88024:head v 129'5693, mlcod=0'0)", 2024-09-13T14:28:14.638 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:22.491691+0000", 2024-09-13T14:28:14.638 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 52.09700995, 2024-09-13T14:28:14.638 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.10257414400000001, 2024-09-13T14:28:14.638 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:14.638 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:14.638 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:14.639 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.639 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:14.639 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.491691+0000", 2024-09-13T14:28:14.639 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:14.639 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.639 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.639 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:14.639 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.491691+0000", 2024-09-13T14:28:14.639 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:14.639 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.639 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.640 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:14.640 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.491690+0000", 2024-09-13T14:28:14.640 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:28:14.640 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.640 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.640 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:14.640 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.491814+0000", 2024-09-13T14:28:14.641 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000123585 2024-09-13T14:28:14.641 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.641 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.641 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:14.641 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.491824+0000", 2024-09-13T14:28:14.641 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6749999999999997e-06 2024-09-13T14:28:14.641 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.641 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.641 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:14.641 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.491830+0000", 2024-09-13T14:28:14.642 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3069999999999999e-06 2024-09-13T14:28:14.642 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.642 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.642 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:14.642 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.491884+0000", 2024-09-13T14:28:14.642 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3751000000000001e-05 2024-09-13T14:28:14.642 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.642 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.642 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:14.642 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.491905+0000", 2024-09-13T14:28:14.642 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1027999999999999e-05 2024-09-13T14:28:14.642 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.643 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.643 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:14.643 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.594230+0000", 2024-09-13T14:28:14.643 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.102325316 2024-09-13T14:28:14.643 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.643 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.643 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:14.643 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.594265+0000", 2024-09-13T14:28:14.643 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5500999999999998e-05 2024-09-13T14:28:14.643 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:14.643 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:14.643 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:14.643 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.644 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.644 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.44546.0:88023 5.f e129/116 5:f4de59b3:::benchmark_data_smithi102_155989_object88022:head v 129'5438, mlcod=129'5434)", 2024-09-13T14:28:14.644 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:22.494759+0000", 2024-09-13T14:28:14.644 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 52.093942470999998, 2024-09-13T14:28:14.644 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.103604424, 2024-09-13T14:28:14.644 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:14.644 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:14.644 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:14.644 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.644 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:14.644 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.494759+0000", 2024-09-13T14:28:14.644 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:14.645 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.645 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.645 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:14.645 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.494759+0000", 2024-09-13T14:28:14.645 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:14.645 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.645 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.645 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:14.645 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.494758+0000", 2024-09-13T14:28:14.645 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:28:14.645 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.645 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.646 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:14.646 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.494787+0000", 2024-09-13T14:28:14.646 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9122e-05 2024-09-13T14:28:14.646 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.646 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.646 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:14.646 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.494796+0000", 2024-09-13T14:28:14.646 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.3409999999999992e-06 2024-09-13T14:28:14.646 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.646 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.646 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:14.647 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.494802+0000", 2024-09-13T14:28:14.647 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7790000000000001e-06 2024-09-13T14:28:14.647 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.647 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.647 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:14.647 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.494885+0000", 2024-09-13T14:28:14.647 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3349000000000004e-05 2024-09-13T14:28:14.647 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.647 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.647 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:14.647 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.495150+0000", 2024-09-13T14:28:14.648 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00026479100000000001 2024-09-13T14:28:14.648 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.648 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.648 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:14.648 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.598313+0000", 2024-09-13T14:28:14.648 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.103162645 2024-09-13T14:28:14.648 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.648 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.648 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:14.648 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.598363+0000", 2024-09-13T14:28:14.648 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0414999999999998e-05 2024-09-13T14:28:14.649 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:14.649 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:14.649 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:14.649 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.649 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.649 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.44546.0:88026 5.6 e129/125 5:690e1d48:::benchmark_data_smithi102_155989_object88025:head v 129'5446, mlcod=0'0)", 2024-09-13T14:28:14.649 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:22.495761+0000", 2024-09-13T14:28:14.649 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 52.092940405, 2024-09-13T14:28:14.649 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.124147801, 2024-09-13T14:28:14.649 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:14.649 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:14.650 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:14.650 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.650 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:14.650 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.495761+0000", 2024-09-13T14:28:14.650 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:14.650 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.650 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.651 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:14.651 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.495761+0000", 2024-09-13T14:28:14.651 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:14.651 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.651 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.651 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:14.651 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.495760+0000", 2024-09-13T14:28:14.651 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:28:14.651 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.651 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.651 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:14.651 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.495782+0000", 2024-09-13T14:28:14.652 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1478999999999999e-05 2024-09-13T14:28:14.652 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.652 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.652 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:14.652 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.495792+0000", 2024-09-13T14:28:14.652 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0538e-05 2024-09-13T14:28:14.652 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.652 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.652 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:14.652 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.495798+0000", 2024-09-13T14:28:14.652 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0689999999999998e-06 2024-09-13T14:28:14.653 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.653 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.653 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:14.653 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.500510+0000", 2024-09-13T14:28:14.653 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0047120640000000002 2024-09-13T14:28:14.653 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.653 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.653 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:14.653 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.500532+0000", 2024-09-13T14:28:14.653 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1551000000000001e-05 2024-09-13T14:28:14.653 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.654 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.654 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:14.654 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.619795+0000", 2024-09-13T14:28:14.654 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.11926363600000001 2024-09-13T14:28:14.654 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.654 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.654 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:14.654 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.619909+0000", 2024-09-13T14:28:14.654 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011324600000000001 2024-09-13T14:28:14.654 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:14.654 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:14.655 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:14.655 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.655 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.655 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:88030 5.1 5:8d0d7fa6:::benchmark_data_smithi102_155989_object88029:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:28:14.655 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:22.498834+0000", 2024-09-13T14:28:14.656 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 52.089867525000003, 2024-09-13T14:28:14.656 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.106449656, 2024-09-13T14:28:14.656 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:14.656 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:14.656 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:14.656 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:14.656 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:14.656 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 88030 2024-09-13T14:28:14.656 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.656 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:14.656 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.657 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:14.657 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.498834+0000", 2024-09-13T14:28:14.657 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:14.657 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.657 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.657 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:14.657 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.498834+0000", 2024-09-13T14:28:14.657 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:14.657 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.657 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.657 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:14.657 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.498832+0000", 2024-09-13T14:28:14.658 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:28:14.658 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.658 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.658 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:14.658 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.498870+0000", 2024-09-13T14:28:14.658 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7934e-05 2024-09-13T14:28:14.658 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.658 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.658 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:14.658 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.498879+0000", 2024-09-13T14:28:14.658 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.5100000000000004e-06 2024-09-13T14:28:14.659 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.659 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.659 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:14.659 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.498885+0000", 2024-09-13T14:28:14.659 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8660000000000002e-06 2024-09-13T14:28:14.659 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.659 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.659 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:14.659 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.500108+0000", 2024-09-13T14:28:14.659 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001222676 2024-09-13T14:28:14.659 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.659 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.660 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:14.660 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.500234+0000", 2024-09-13T14:28:14.660 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012565800000000001 2024-09-13T14:28:14.660 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.660 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.660 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:28:14.660 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.500389+0000", 2024-09-13T14:28:14.660 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000155012 2024-09-13T14:28:14.660 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.660 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.660 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:14.660 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.596416+0000", 2024-09-13T14:28:14.661 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.096027869000000002 2024-09-13T14:28:14.661 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.661 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.661 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:14.661 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.596448+0000", 2024-09-13T14:28:14.661 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.1535999999999998e-05 2024-09-13T14:28:14.661 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.661 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.661 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:14.661 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.605208+0000", 2024-09-13T14:28:14.661 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0087603290000000007 2024-09-13T14:28:14.661 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.662 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.662 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:14.662 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.605250+0000", 2024-09-13T14:28:14.662 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1349e-05 2024-09-13T14:28:14.662 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.662 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.662 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:14.662 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.605283+0000", 2024-09-13T14:28:14.662 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.3744000000000001e-05 2024-09-13T14:28:14.662 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:14.662 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:14.662 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:14.663 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.663 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.663 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:90271 5.c 5:314a6f09:::benchmark_data_smithi102_155989_object90270:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:28:14.663 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:25.252927+0000", 2024-09-13T14:28:14.663 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 49.335774100000002, 2024-09-13T14:28:14.663 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.103091371, 2024-09-13T14:28:14.663 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:14.663 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:14.663 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:14.663 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:14.663 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:14.663 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 90271 2024-09-13T14:28:14.664 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.664 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:14.664 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.664 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:14.664 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.252927+0000", 2024-09-13T14:28:14.664 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:14.664 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.664 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.664 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:14.664 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.252927+0000", 2024-09-13T14:28:14.664 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:14.665 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:14.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:14.666 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.666 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.667 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:14.667 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.252926+0000", 2024-09-13T14:28:14.667 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:28:14.667 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.667 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.667 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:14.667 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.253013+0000", 2024-09-13T14:28:14.667 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7138999999999994e-05 2024-09-13T14:28:14.667 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.667 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.667 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:14.667 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.253023+0000", 2024-09-13T14:28:14.668 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0513000000000001e-05 2024-09-13T14:28:14.668 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.668 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.668 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:14.668 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.253030+0000", 2024-09-13T14:28:14.668 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4590000000000003e-06 2024-09-13T14:28:14.668 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.668 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.668 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:14.668 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.253070+0000", 2024-09-13T14:28:14.668 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0210000000000003e-05 2024-09-13T14:28:14.669 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.669 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.669 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:14.669 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.253200+0000", 2024-09-13T14:28:14.669 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012980000000000001 2024-09-13T14:28:14.669 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.669 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.669 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,6", 2024-09-13T14:28:14.669 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.253323+0000", 2024-09-13T14:28:14.669 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012299199999999999 2024-09-13T14:28:14.669 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.669 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.669 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:14.670 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.258848+0000", 2024-09-13T14:28:14.670 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0055248140000000003 2024-09-13T14:28:14.670 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.670 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.670 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:14.670 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.259935+0000", 2024-09-13T14:28:14.670 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0010874700000000001 2024-09-13T14:28:14.670 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.670 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.670 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:14.670 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.355940+0000", 2024-09-13T14:28:14.670 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.096005251999999999 2024-09-13T14:28:14.670 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.671 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.671 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:14.671 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.355982+0000", 2024-09-13T14:28:14.671 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2145999999999999e-05 2024-09-13T14:28:14.671 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.671 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.671 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:14.671 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.356019+0000", 2024-09-13T14:28:14.671 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6146000000000003e-05 2024-09-13T14:28:14.671 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:14.671 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:14.671 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:14.671 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.672 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.672 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:91654 5.a 5:5299faa7:::benchmark_data_smithi102_155989_object91653:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:28:14.672 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:27.316609+0000", 2024-09-13T14:28:14.672 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 47.272092284999999, 2024-09-13T14:28:14.672 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.117235135, 2024-09-13T14:28:14.672 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:14.672 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:14.672 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:14.672 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:14.672 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:14.672 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 91654 2024-09-13T14:28:14.673 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.673 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:14.673 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.673 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:14.673 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.316609+0000", 2024-09-13T14:28:14.673 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:14.673 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.673 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.673 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:14.673 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.316609+0000", 2024-09-13T14:28:14.674 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:14.674 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.674 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.674 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:14.674 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.316607+0000", 2024-09-13T14:28:14.674 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:28:14.674 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.674 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.674 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:14.674 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.316644+0000", 2024-09-13T14:28:14.674 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6854999999999999e-05 2024-09-13T14:28:14.675 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.675 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.675 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:14.675 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.316660+0000", 2024-09-13T14:28:14.675 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5599000000000001e-05 2024-09-13T14:28:14.675 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.675 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.675 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:14.675 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.316667+0000", 2024-09-13T14:28:14.675 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4279999999999999e-06 2024-09-13T14:28:14.675 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.675 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.676 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:14.676 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.324526+0000", 2024-09-13T14:28:14.676 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0078587329999999997 2024-09-13T14:28:14.676 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.676 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.676 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:14.676 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.324681+0000", 2024-09-13T14:28:14.676 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015515599999999999 2024-09-13T14:28:14.676 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.676 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.676 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,3", 2024-09-13T14:28:14.677 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.324860+0000", 2024-09-13T14:28:14.677 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00017919999999999999 2024-09-13T14:28:14.677 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.677 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.677 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:14.677 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.343677+0000", 2024-09-13T14:28:14.677 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018816833000000002 2024-09-13T14:28:14.677 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.677 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.677 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:14.677 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.343746+0000", 2024-09-13T14:28:14.677 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8774000000000003e-05 2024-09-13T14:28:14.678 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.678 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.678 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:14.678 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.425055+0000", 2024-09-13T14:28:14.678 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.081308742000000003 2024-09-13T14:28:14.678 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.678 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.678 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:14.678 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.425187+0000", 2024-09-13T14:28:14.678 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013233299999999999 2024-09-13T14:28:14.679 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.679 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.679 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:14.679 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.433844+0000", 2024-09-13T14:28:14.679 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0086572899999999998 2024-09-13T14:28:14.679 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:14.679 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:14.679 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:14.679 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.679 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.679 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:91656 5.a 5:508f8413:::benchmark_data_smithi102_155989_object91655:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:28:14.680 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:27.316731+0000", 2024-09-13T14:28:14.680 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 47.271970625999998, 2024-09-13T14:28:14.680 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.12772304500000001, 2024-09-13T14:28:14.680 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:14.680 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:14.680 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:14.680 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:14.680 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:14.680 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 91656 2024-09-13T14:28:14.680 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.680 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:14.680 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.681 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:14.681 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.316731+0000", 2024-09-13T14:28:14.681 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:14.681 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.681 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.681 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:14.681 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.316731+0000", 2024-09-13T14:28:14.681 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:14.681 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.681 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.681 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:14.682 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.316730+0000", 2024-09-13T14:28:14.682 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:28:14.682 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.682 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.682 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:14.682 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.316764+0000", 2024-09-13T14:28:14.682 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.3442000000000003e-05 2024-09-13T14:28:14.682 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.682 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.682 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:14.682 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.316773+0000", 2024-09-13T14:28:14.682 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.2089999999999994e-06 2024-09-13T14:28:14.683 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.683 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.683 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:14.683 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.316778+0000", 2024-09-13T14:28:14.683 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0590000000000002e-06 2024-09-13T14:28:14.683 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.683 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.683 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:14.683 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.331802+0000", 2024-09-13T14:28:14.683 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.015023642 2024-09-13T14:28:14.683 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.684 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.684 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:14.684 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.331948+0000", 2024-09-13T14:28:14.684 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000146482 2024-09-13T14:28:14.684 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.684 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.684 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,3", 2024-09-13T14:28:14.684 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.332100+0000", 2024-09-13T14:28:14.684 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015182699999999999 2024-09-13T14:28:14.684 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.684 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.685 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:14.685 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.343857+0000", 2024-09-13T14:28:14.685 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.01175733 2024-09-13T14:28:14.685 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.685 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.685 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:14.685 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.343889+0000", 2024-09-13T14:28:14.685 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.1683000000000001e-05 2024-09-13T14:28:14.685 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.685 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.685 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:14.685 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.438087+0000", 2024-09-13T14:28:14.686 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.094197914999999993 2024-09-13T14:28:14.686 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.686 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.686 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:14.686 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.438139+0000", 2024-09-13T14:28:14.686 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2346000000000003e-05 2024-09-13T14:28:14.686 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.686 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.686 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:14.686 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.444454+0000", 2024-09-13T14:28:14.686 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0063145229999999998 2024-09-13T14:28:14.686 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:14.687 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:14.687 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:14.687 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.687 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.687 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:91657 5.a 5:51ff3ce2:::benchmark_data_smithi102_155989_object91656:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:28:14.687 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:27.316895+0000", 2024-09-13T14:28:14.687 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 47.271806515000002, 2024-09-13T14:28:14.687 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.13008430200000001, 2024-09-13T14:28:14.687 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:14.687 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:14.687 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:14.687 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:14.688 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:14.688 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 91657 2024-09-13T14:28:14.688 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.688 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:14.688 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.688 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:14.688 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.316895+0000", 2024-09-13T14:28:14.689 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:14.689 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.689 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.689 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:14.689 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.316895+0000", 2024-09-13T14:28:14.689 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:14.689 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.689 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.689 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:14.689 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.316894+0000", 2024-09-13T14:28:14.689 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:28:14.689 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.690 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.690 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:14.690 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.316964+0000", 2024-09-13T14:28:14.690 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.0160000000000006e-05 2024-09-13T14:28:14.690 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.690 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.690 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:14.690 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.316984+0000", 2024-09-13T14:28:14.690 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9992000000000001e-05 2024-09-13T14:28:14.690 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.691 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.691 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:14.691 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.316990+0000", 2024-09-13T14:28:14.691 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7649999999999999e-06 2024-09-13T14:28:14.691 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.691 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.691 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:14.691 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.335386+0000", 2024-09-13T14:28:14.691 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018395575000000001 2024-09-13T14:28:14.691 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.691 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.691 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:14.692 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.335511+0000", 2024-09-13T14:28:14.692 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012501499999999999 2024-09-13T14:28:14.692 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.692 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.692 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,3", 2024-09-13T14:28:14.692 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.335664+0000", 2024-09-13T14:28:14.692 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015311200000000001 2024-09-13T14:28:14.692 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.692 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.692 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:14.692 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.343907+0000", 2024-09-13T14:28:14.693 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0082433409999999995 2024-09-13T14:28:14.693 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.693 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.693 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:14.693 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.343936+0000", 2024-09-13T14:28:14.693 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9009e-05 2024-09-13T14:28:14.693 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.693 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.693 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:14.693 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.444517+0000", 2024-09-13T14:28:14.693 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.100580424 2024-09-13T14:28:14.693 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.694 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.694 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:14.694 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.444619+0000", 2024-09-13T14:28:14.694 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010208399999999999 2024-09-13T14:28:14.694 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.694 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.694 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:14.694 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.446979+0000", 2024-09-13T14:28:14.694 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0023604229999999999 2024-09-13T14:28:14.694 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:14.694 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:14.694 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:14.695 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.695 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.695 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:91665 5.a 5:5e4f4c4f:::benchmark_data_smithi102_155989_object91664:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:28:14.695 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:27.326817+0000", 2024-09-13T14:28:14.695 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 47.261884516999999, 2024-09-13T14:28:14.695 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.122295468, 2024-09-13T14:28:14.695 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:14.695 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:14.695 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:14.695 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:14.695 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:14.695 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 91665 2024-09-13T14:28:14.696 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.696 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:14.696 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.696 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:14.696 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.326817+0000", 2024-09-13T14:28:14.696 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:14.696 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.696 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.696 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:14.696 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.326817+0000", 2024-09-13T14:28:14.696 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:14.696 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.697 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.697 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:14.697 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.326815+0000", 2024-09-13T14:28:14.697 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:28:14.697 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.697 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.697 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:14.697 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.326860+0000", 2024-09-13T14:28:14.697 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4332000000000002e-05 2024-09-13T14:28:14.697 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.697 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.697 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:14.698 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.326868+0000", 2024-09-13T14:28:14.698 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3459999999999994e-06 2024-09-13T14:28:14.698 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.698 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.698 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:14.698 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.326873+0000", 2024-09-13T14:28:14.698 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4349999999999996e-06 2024-09-13T14:28:14.698 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.698 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.698 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:14.698 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.343951+0000", 2024-09-13T14:28:14.698 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017078142000000001 2024-09-13T14:28:14.699 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.699 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.699 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:14.699 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.344068+0000", 2024-09-13T14:28:14.699 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011694800000000001 2024-09-13T14:28:14.699 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.699 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.699 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,3", 2024-09-13T14:28:14.699 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.344179+0000", 2024-09-13T14:28:14.699 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000111062 2024-09-13T14:28:14.700 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.700 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.700 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:14.700 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.346456+0000", 2024-09-13T14:28:14.700 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002277004 2024-09-13T14:28:14.700 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.700 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.700 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:14.700 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.346503+0000", 2024-09-13T14:28:14.700 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6257000000000001e-05 2024-09-13T14:28:14.700 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.700 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.701 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:14.701 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.447273+0000", 2024-09-13T14:28:14.701 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.10077000799999999 2024-09-13T14:28:14.701 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.701 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.701 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:14.701 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.447325+0000", 2024-09-13T14:28:14.702 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2488000000000003e-05 2024-09-13T14:28:14.702 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.702 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.702 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:14.702 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.449112+0000", 2024-09-13T14:28:14.702 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001787039 2024-09-13T14:28:14.702 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:14.702 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:14.702 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:14.702 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.702 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.702 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:113752 5.a 5:52d0af89:::benchmark_data_smithi102_155989_object113751:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:28:14.703 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:55.347828+0000", 2024-09-13T14:28:14.703 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 19.240873173000001, 2024-09-13T14:28:14.703 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.10004697799999999, 2024-09-13T14:28:14.703 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:14.703 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:14.703 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:14.703 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:14.703 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:14.703 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113752 2024-09-13T14:28:14.703 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.703 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:14.703 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.704 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:14.704 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.347828+0000", 2024-09-13T14:28:14.704 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:14.704 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.704 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.704 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:14.704 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.347828+0000", 2024-09-13T14:28:14.704 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:14.704 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.704 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.704 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:14.705 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.347826+0000", 2024-09-13T14:28:14.705 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:28:14.705 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.705 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.705 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:14.705 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.347896+0000", 2024-09-13T14:28:14.705 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9460000000000002e-05 2024-09-13T14:28:14.705 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.705 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.705 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:14.706 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.347908+0000", 2024-09-13T14:28:14.706 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2496e-05 2024-09-13T14:28:14.706 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.706 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.706 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:14.706 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.347921+0000", 2024-09-13T14:28:14.706 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2818e-05 2024-09-13T14:28:14.706 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.706 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.706 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:14.706 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.347980+0000", 2024-09-13T14:28:14.706 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8659999999999997e-05 2024-09-13T14:28:14.707 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.707 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.707 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:14.707 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.348136+0000", 2024-09-13T14:28:14.707 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015640200000000001 2024-09-13T14:28:14.707 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.707 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.707 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,3", 2024-09-13T14:28:14.707 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.348314+0000", 2024-09-13T14:28:14.707 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000178226 2024-09-13T14:28:14.707 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.708 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.708 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:14.708 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.352548+0000", 2024-09-13T14:28:14.708 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0042339250000000004 2024-09-13T14:28:14.708 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.708 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.708 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:14.708 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.352556+0000", 2024-09-13T14:28:14.708 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3129999999999999e-06 2024-09-13T14:28:14.708 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.708 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.709 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:14.709 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.445798+0000", 2024-09-13T14:28:14.709 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.093241287000000006 2024-09-13T14:28:14.709 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.709 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.709 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:14.709 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.445853+0000", 2024-09-13T14:28:14.709 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5433000000000003e-05 2024-09-13T14:28:14.709 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.709 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.709 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:14.710 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.447875+0000", 2024-09-13T14:28:14.710 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0020219840000000001 2024-09-13T14:28:14.710 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:14.710 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:14.710 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:14.710 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.710 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.710 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:116687 5.a 5:50c2b639:::benchmark_data_smithi102_155989_object116686:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:28:14.710 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:59.385059+0000", 2024-09-13T14:28:14.710 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 15.203642780999999, 2024-09-13T14:28:14.710 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.108871435, 2024-09-13T14:28:14.710 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:14.711 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:14.711 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:14.711 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:14.711 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:14.711 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 116687 2024-09-13T14:28:14.711 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.711 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:14.711 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.711 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:14.711 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.385059+0000", 2024-09-13T14:28:14.711 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:14.712 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.712 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.712 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:14.712 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.385059+0000", 2024-09-13T14:28:14.712 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:14.712 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.712 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.712 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:14.712 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.385057+0000", 2024-09-13T14:28:14.712 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:28:14.712 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.713 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.713 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:14.713 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.385097+0000", 2024-09-13T14:28:14.713 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9626000000000003e-05 2024-09-13T14:28:14.713 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.713 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.713 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:14.713 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.385108+0000", 2024-09-13T14:28:14.713 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1215e-05 2024-09-13T14:28:14.713 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.713 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.713 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:14.714 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.385115+0000", 2024-09-13T14:28:14.714 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9149999999999999e-06 2024-09-13T14:28:14.714 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.714 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.714 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:14.714 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.385154+0000", 2024-09-13T14:28:14.714 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8748000000000002e-05 2024-09-13T14:28:14.714 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.714 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.714 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:14.714 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.385279+0000", 2024-09-13T14:28:14.714 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012510900000000001 2024-09-13T14:28:14.715 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.715 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.715 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,3", 2024-09-13T14:28:14.715 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.385449+0000", 2024-09-13T14:28:14.715 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00017060799999999999 2024-09-13T14:28:14.715 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.715 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.715 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:14.715 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.389085+0000", 2024-09-13T14:28:14.716 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0036353219999999999 2024-09-13T14:28:14.716 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.716 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.716 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:14.716 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.390922+0000", 2024-09-13T14:28:14.716 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001836868 2024-09-13T14:28:14.716 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.716 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.716 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:14.716 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.491886+0000", 2024-09-13T14:28:14.717 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.100964282 2024-09-13T14:28:14.717 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.717 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.717 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:14.717 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.491970+0000", 2024-09-13T14:28:14.717 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.4016000000000003e-05 2024-09-13T14:28:14.717 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.717 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.717 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:14.717 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.493930+0000", 2024-09-13T14:28:14.717 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0019601169999999999 2024-09-13T14:28:14.718 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:14.718 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:14.718 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:14.718 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.718 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.718 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:116690 5.e 5:76ab6c22:::benchmark_data_smithi102_155989_object116689:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:28:14.718 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:59.389852+0000", 2024-09-13T14:28:14.718 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 15.198849453999999, 2024-09-13T14:28:14.718 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.102158811, 2024-09-13T14:28:14.719 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:14.719 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:14.719 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:14.719 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:14.719 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:14.719 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 116690 2024-09-13T14:28:14.719 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.719 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:14.719 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.719 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:14.719 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.389852+0000", 2024-09-13T14:28:14.720 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:14.720 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.720 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.720 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:14.720 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.389852+0000", 2024-09-13T14:28:14.720 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:14.720 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.720 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.720 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:14.720 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.389850+0000", 2024-09-13T14:28:14.720 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:28:14.721 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.721 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.721 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:14.721 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.389879+0000", 2024-09-13T14:28:14.721 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8554000000000001e-05 2024-09-13T14:28:14.721 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.721 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.721 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:14.721 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.389889+0000", 2024-09-13T14:28:14.721 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.01e-05 2024-09-13T14:28:14.721 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.721 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.722 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:14.722 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.389895+0000", 2024-09-13T14:28:14.722 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3260000000000001e-06 2024-09-13T14:28:14.722 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.722 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.722 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:14.722 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.391505+0000", 2024-09-13T14:28:14.722 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001609596 2024-09-13T14:28:14.722 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.722 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.722 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:14.723 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.391624+0000", 2024-09-13T14:28:14.723 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011888 2024-09-13T14:28:14.723 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.723 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.723 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:28:14.723 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.391778+0000", 2024-09-13T14:28:14.723 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000154427 2024-09-13T14:28:14.723 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.723 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.723 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:14.723 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.395582+0000", 2024-09-13T14:28:14.723 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0038038579999999998 2024-09-13T14:28:14.724 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.724 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.724 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:14.724 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.406664+0000", 2024-09-13T14:28:14.724 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011081706 2024-09-13T14:28:14.724 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.724 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.724 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:14.724 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.491904+0000", 2024-09-13T14:28:14.724 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.085239696000000004 2024-09-13T14:28:14.724 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.725 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.725 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:14.725 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.491973+0000", 2024-09-13T14:28:14.725 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9752000000000006e-05 2024-09-13T14:28:14.725 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.725 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.725 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:14.725 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.492011+0000", 2024-09-13T14:28:14.725 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7373000000000003e-05 2024-09-13T14:28:14.725 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:14.725 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:14.726 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:14.726 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.726 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.726 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:127067 5.e 5:7b064861:::benchmark_data_smithi102_155989_object9149:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-13T14:28:14.726 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:28:13.122115+0000", 2024-09-13T14:28:14.726 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 1.466586497, 2024-09-13T14:28:14.726 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.10508351, 2024-09-13T14:28:14.726 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:14.726 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:14.726 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:14.726 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:14.727 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:14.727 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 127067 2024-09-13T14:28:14.727 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.727 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:14.727 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.727 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:14.727 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.122115+0000", 2024-09-13T14:28:14.727 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:14.727 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.727 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.727 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:14.728 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.122115+0000", 2024-09-13T14:28:14.728 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:14.728 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.728 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.728 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:14.728 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.122113+0000", 2024-09-13T14:28:14.728 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:28:14.728 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.728 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.728 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:14.728 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.122118+0000", 2024-09-13T14:28:14.729 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.527e-06 2024-09-13T14:28:14.729 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.729 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.729 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:14.729 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.122126+0000", 2024-09-13T14:28:14.729 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9920000000000007e-06 2024-09-13T14:28:14.729 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.729 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.729 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:14.729 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.122131+0000", 2024-09-13T14:28:14.729 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4689999999999996e-06 2024-09-13T14:28:14.729 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.730 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.730 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:14.730 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.193233+0000", 2024-09-13T14:28:14.730 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.071101628 2024-09-13T14:28:14.730 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.730 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.730 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:14.730 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.193344+0000", 2024-09-13T14:28:14.730 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011135 2024-09-13T14:28:14.730 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.730 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.731 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:28:14.731 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.193448+0000", 2024-09-13T14:28:14.731 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010391300000000001 2024-09-13T14:28:14.731 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.731 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.731 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:14.731 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.208740+0000", 2024-09-13T14:28:14.731 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.015292065000000001 2024-09-13T14:28:14.731 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.731 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.731 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:14.732 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.216147+0000", 2024-09-13T14:28:14.732 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0074065329999999999 2024-09-13T14:28:14.732 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.732 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.732 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:14.732 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.227124+0000", 2024-09-13T14:28:14.732 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010977557000000001 2024-09-13T14:28:14.732 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.732 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.733 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:14.733 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.227166+0000", 2024-09-13T14:28:14.733 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1820000000000003e-05 2024-09-13T14:28:14.733 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.733 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.733 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:14.733 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.227198+0000", 2024-09-13T14:28:14.733 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2033000000000003e-05 2024-09-13T14:28:14.733 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:14.733 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:14.733 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:14.734 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.734 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.734 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:127599 5.a 5:5ca2f1c2:::benchmark_data_smithi102_155989_object9681:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-13T14:28:14.734 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:28:13.842069+0000", 2024-09-13T14:28:14.734 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.74663221499999999, 2024-09-13T14:28:14.734 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.100501036, 2024-09-13T14:28:14.734 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:14.734 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:14.734 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:14.734 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:14.735 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:14.735 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 127599 2024-09-13T14:28:14.735 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.735 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:14.735 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.735 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:14.735 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.842069+0000", 2024-09-13T14:28:14.735 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:14.735 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.735 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.735 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:14.736 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.842069+0000", 2024-09-13T14:28:14.736 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:14.736 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.736 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.736 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:14.736 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.842068+0000", 2024-09-13T14:28:14.736 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:28:14.736 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.736 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.736 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:14.736 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.842073+0000", 2024-09-13T14:28:14.737 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4299999999999997e-06 2024-09-13T14:28:14.737 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.737 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.737 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:14.737 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.842086+0000", 2024-09-13T14:28:14.737 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.257e-05 2024-09-13T14:28:14.737 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.737 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.737 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:14.737 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.842092+0000", 2024-09-13T14:28:14.737 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.939e-06 2024-09-13T14:28:14.738 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.738 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.738 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:14.738 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.846327+0000", 2024-09-13T14:28:14.738 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0042348159999999998 2024-09-13T14:28:14.738 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.738 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.738 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:14.738 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.846491+0000", 2024-09-13T14:28:14.739 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000163888 2024-09-13T14:28:14.739 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.739 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.739 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,3", 2024-09-13T14:28:14.739 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.846664+0000", 2024-09-13T14:28:14.739 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00017240699999999999 2024-09-13T14:28:14.739 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.739 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.739 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:14.739 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.851096+0000", 2024-09-13T14:28:14.740 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0044323100000000001 2024-09-13T14:28:14.740 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.740 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.740 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:14.740 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.851128+0000", 2024-09-13T14:28:14.740 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2413999999999998e-05 2024-09-13T14:28:14.740 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.740 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.740 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:14.740 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.935712+0000", 2024-09-13T14:28:14.740 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.084584118999999999 2024-09-13T14:28:14.741 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.741 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.741 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:14.741 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.935799+0000", 2024-09-13T14:28:14.741 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.6786000000000006e-05 2024-09-13T14:28:14.741 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:14.741 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:14.741 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:14.742 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.942570+0000", 2024-09-13T14:28:14.742 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0067709320000000003 2024-09-13T14:28:14.742 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:14.742 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:14.742 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:14.742 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:14.742 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:14.742 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:28:14.766 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:14.783 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:14 smithi102 ceph-mon[110183]: pgmap v369: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.7 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 806 KiB/s wr, 763 op/s; 25440/330846 objects degraded (7.689%); 2391/330846 objects misplaced (0.723%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T14:28:14.783 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:14 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 25440/330846 objects degraded (7.689%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:28:14.783 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:14 smithi102 ceph-mon[115269]: pgmap v369: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.7 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 806 KiB/s wr, 763 op/s; 25440/330846 objects degraded (7.689%); 2391/330846 objects misplaced (0.723%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T14:28:14.783 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:14 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 25440/330846 objects degraded (7.689%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:28:14.791 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:28:14.799 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:14.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:14.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:14.845 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:28:14.900 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:14.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:14.943 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:14.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:14.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:14 smithi195 ceph-mon[104262]: pgmap v369: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.7 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 806 KiB/s wr, 763 op/s; 25440/330846 objects degraded (7.689%); 2391/330846 objects misplaced (0.723%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T14:28:14.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:14 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 25440/330846 objects degraded (7.689%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:28:15.045 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:15.078 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:15.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:15.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:15.180 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:15.205 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:15.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:15.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:15.307 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:28:15.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:15.338 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:15.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:28:15.440 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:15.481 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:15.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:15.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:15.582 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:15.610 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:15.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:15.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:15.711 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:15.739 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:15.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:15.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:15.841 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:28:15.865 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:15.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:15.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:28:15.966 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:16.011 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:16.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:16.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:16.113 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:16.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:16.146 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:16.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:16.247 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:16.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:16.293 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:16.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:16.396 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:16.433 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:16.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:16.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:16.535 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:28:16.577 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:16.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:16.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:28:16.679 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:28:16.725 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:16.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:16.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:28:16.826 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:28:16.851 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:16 smithi195 ceph-mon[104262]: pgmap v370: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.7 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 768 op/s; 25440/330048 objects degraded (7.708%); 2391/330048 objects misplaced (0.724%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T14:28:16.853 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:16.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:16.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:28:16.879 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T14:28:16.893 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:16 smithi102 ceph-mon[110183]: pgmap v370: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.7 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 768 op/s; 25440/330048 objects degraded (7.708%); 2391/330048 objects misplaced (0.724%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T14:28:16.893 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:16 smithi102 ceph-mon[115269]: pgmap v370: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.7 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 768 op/s; 25440/330048 objects degraded (7.708%); 2391/330048 objects misplaced (0.724%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T14:28:16.894 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:28:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:28:16] "GET /metrics HTTP/1.1" 200 36547 "" "Prometheus/2.51.0" 2024-09-13T14:28:16.954 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:17.015 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:17.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:17.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:17.118 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:17.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:17.142 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:17.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:17.244 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:28:17.272 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:17.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:17.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:28:17.280 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:28:17.374 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:17.401 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:17.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:17.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:17.503 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:17.539 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:17.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:17.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:17.604 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:28:17.604 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:28:17.605 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is set 2024-09-13T14:28:17.641 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:28:17.641 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:28:17.642 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:28:17.670 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:28:17.670 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:28:17.678 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:17.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:17.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:28:17.690 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:28:17.690 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:28:17.720 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:28:17.721 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:28:17.721 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:17 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1808459448' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:28:17.721 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:17 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3649695231' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:28:17.721 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:17 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1808459448' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:28:17.721 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:17 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3649695231' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:28:17.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:17 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1808459448' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:28:17.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:17 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3649695231' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:28:17.755 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:28:17.755 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:28:17.781 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:28:17.781 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:28:17.782 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:28:17.807 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:17.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:17.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:28:17.817 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:28:17.817 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:28:17.909 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:17.939 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:17.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:17.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:18.041 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:18.075 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:18.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:18.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:18.177 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:18.201 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:18.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:18.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:18.303 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:18.327 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:18.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:18.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:18.429 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:18.457 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:18.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:18.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:18.559 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:18.584 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:18.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:18.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:18.686 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:18.712 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:18.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:18.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:18.815 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:18.840 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:18.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:18.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:18.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:18 smithi102 ceph-mon[110183]: pgmap v371: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 951 op/s; 25440/319644 objects degraded (7.959%); 2118/319644 objects misplaced (0.663%); 4.1 MiB/s, 65 objects/s recovering 2024-09-13T14:28:18.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:18 smithi102 ceph-mon[110183]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:28:18.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:18 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1808459448' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:28:18.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:18 smithi102 ceph-mon[110183]: osdmap e133: 8 total, 8 up, 6 in 2024-09-13T14:28:18.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:18 smithi102 ceph-mon[115269]: pgmap v371: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 951 op/s; 25440/319644 objects degraded (7.959%); 2118/319644 objects misplaced (0.663%); 4.1 MiB/s, 65 objects/s recovering 2024-09-13T14:28:18.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:18 smithi102 ceph-mon[115269]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:28:18.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:18 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1808459448' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:28:18.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:18 smithi102 ceph-mon[115269]: osdmap e133: 8 total, 8 up, 6 in 2024-09-13T14:28:18.942 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:18.978 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:18.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:18.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:18.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:18 smithi195 ceph-mon[104262]: pgmap v371: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 951 op/s; 25440/319644 objects degraded (7.959%); 2118/319644 objects misplaced (0.663%); 4.1 MiB/s, 65 objects/s recovering 2024-09-13T14:28:18.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:18 smithi195 ceph-mon[104262]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:28:18.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:18 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1808459448' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:28:18.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:18 smithi195 ceph-mon[104262]: osdmap e133: 8 total, 8 up, 6 in 2024-09-13T14:28:19.080 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:28:19.117 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:19.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:19.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:28:19.219 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:19.246 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:19.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:19.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:19.348 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:28:19.372 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:19.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:19.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:28:19.474 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:19.519 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:19.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:19.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:19.621 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:19.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:19.649 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:19.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:19.673 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:19 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 25440/319644 objects degraded (7.959%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:28:19.751 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:19.773 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:19.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:19.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:19.833 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:28:19.833 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:28:19.876 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:19.901 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:19.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:19.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:19.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:19 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 25440/319644 objects degraded (7.959%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:28:19.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:19 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 25440/319644 objects degraded (7.959%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:28:20.003 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:20.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:20.027 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:20.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:20.128 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:20.153 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:20.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:20.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:20.255 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:20.294 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:20.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:20.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:20.396 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:20.432 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:20.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:20.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:20.534 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:28:20.551 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T14:28:20.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:20.585 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:20.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:28:20.646 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:28:20.687 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:20.738 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:20.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:20.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:20.839 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:20.863 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:20.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:20.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:20.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:20 smithi102 ceph-mon[110183]: pgmap v373: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 744 op/s; 25440/319038 objects degraded (7.974%); 2118/319038 objects misplaced (0.664%); 3.2 MiB/s, 51 objects/s recovering 2024-09-13T14:28:20.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:20 smithi102 ceph-mon[115269]: pgmap v373: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 744 op/s; 25440/319038 objects degraded (7.974%); 2118/319038 objects misplaced (0.664%); 3.2 MiB/s, 51 objects/s recovering 2024-09-13T14:28:20.939 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:28:20.965 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:20.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:20 smithi195 ceph-mon[104262]: pgmap v373: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 744 op/s; 25440/319038 objects degraded (7.974%); 2118/319038 objects misplaced (0.664%); 3.2 MiB/s, 51 objects/s recovering 2024-09-13T14:28:21.013 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:21.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:21.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:21.043 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.7 reset_purged_snaps_last 2024-09-13T14:28:21.115 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:21.145 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:28:21.148 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:21.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:21.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:21.249 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:21.289 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:21.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:21.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:21.391 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:21.433 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:21.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:21.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:21.526 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:28:21.535 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:21.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:21.580 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:21.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:21.681 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:21.732 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:21.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:21.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:21.834 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:21.860 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:21.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:21.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:21.903 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:28:21.961 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:21.988 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:21.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:21.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:22.090 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:22.131 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:22.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:22.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:22.236 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:28:22.285 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:22.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:22.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:28:22.387 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:22.415 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:22.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:22.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:22.517 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:22.562 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:22.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:22.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:22.663 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:22.709 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:22.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:22.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:22.812 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:22.813 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:22 smithi195 ceph-mon[104262]: pgmap v374: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.4 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 743 op/s; 25440/315528 objects degraded (8.063%); 1822/315528 objects misplaced (0.577%); 3.6 MiB/s, 56 objects/s recovering 2024-09-13T14:28:22.837 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:22.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:22.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:22.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:22 smithi102 ceph-mon[110183]: pgmap v374: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.4 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 743 op/s; 25440/315528 objects degraded (8.063%); 1822/315528 objects misplaced (0.577%); 3.6 MiB/s, 56 objects/s recovering 2024-09-13T14:28:22.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:22 smithi102 ceph-mon[115269]: pgmap v374: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.4 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 743 op/s; 25440/315528 objects degraded (8.063%); 1822/315528 objects misplaced (0.577%); 3.6 MiB/s, 56 objects/s recovering 2024-09-13T14:28:22.940 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:22.972 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:22.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:22.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:22.982 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:28:22.983 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:28:23.074 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:23.099 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:23.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:23.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:23.201 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:23.265 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:23.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:23.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:23.366 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:23.423 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:23.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:23.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:23.445 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:28:23.524 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:23.554 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:28:23.554 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:28:23.554 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:28:23.554 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:28:23.554 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:28:23.554 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:28:23.554 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:28:23.555 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:28:23.555 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:28:23.555 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:28:23.555 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:28:23.555 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:28:23.555 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:28:23.555 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:28:23.555 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:28:23.555 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:28:23.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:23.566 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:23.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:23.667 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:23.696 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T14:28:23.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:23.720 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:23.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:23.821 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:23.846 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:23.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:23.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:23.900 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:23 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:28:23.900 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:23 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3471286926' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:28:23.901 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:23 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:28:23.901 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:23 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3471286926' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:28:23.948 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:28:23.971 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:23.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:23.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:28:23.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:23 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:28:23.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:23 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3471286926' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:28:24.073 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:28:24.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:24.133 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:24.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:28:24.234 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:28:24.258 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:24.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:24.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:28:24.360 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:24.396 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:24.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:24.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:24.498 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:24.523 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:24.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:24.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:24.625 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:24.650 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:24.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:24.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:24.716 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:24 smithi102 ceph-mon[110183]: pgmap v375: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.3 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 737 op/s; 25440/308715 objects degraded (8.241%); 1822/308715 objects misplaced (0.590%); 3.6 MiB/s, 56 objects/s recovering 2024-09-13T14:28:24.716 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:24 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 25440/308715 objects degraded (8.241%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:28:24.716 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:24 smithi102 ceph-mon[115269]: pgmap v375: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.3 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 737 op/s; 25440/308715 objects degraded (8.241%); 1822/308715 objects misplaced (0.590%); 3.6 MiB/s, 56 objects/s recovering 2024-09-13T14:28:24.717 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:24 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 25440/308715 objects degraded (8.241%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:28:24.753 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:24.754 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:28:24.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:24.810 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:24.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:24.912 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:24.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:24.950 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:24.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:24.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:24 smithi195 ceph-mon[104262]: pgmap v375: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.3 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 737 op/s; 25440/308715 objects degraded (8.241%); 1822/308715 objects misplaced (0.590%); 3.6 MiB/s, 56 objects/s recovering 2024-09-13T14:28:24.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:24 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 25440/308715 objects degraded (8.241%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:28:25.051 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:25.078 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:25.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:25.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:25.180 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:25.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:25.236 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:25.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:25.337 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:25.364 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:25.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:25.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:25.466 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:28:25.490 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:25.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:25.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:28:25.592 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:25.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:25.634 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:25.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:25.735 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:25.762 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:25.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:25.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:25.864 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:25.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:25.897 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:25.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:25.998 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:26.023 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:26.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:26.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:26.125 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:28:26.152 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:26.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:26.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:28:26.254 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:26.300 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:26.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:26.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:26.402 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:26.433 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:26.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:26.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:26.534 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:26.570 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:26.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:26.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:26.578 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:26 smithi102 ceph-mon[110183]: pgmap v376: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.3 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 733 op/s; 25440/308040 objects degraded (8.259%); 1822/308040 objects misplaced (0.591%); 3.6 MiB/s, 56 objects/s recovering 2024-09-13T14:28:26.578 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:28:26.578 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:28:26.578 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:26 smithi102 ceph-mon[115269]: pgmap v376: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.3 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 733 op/s; 25440/308040 objects degraded (8.259%); 1822/308040 objects misplaced (0.591%); 3.6 MiB/s, 56 objects/s recovering 2024-09-13T14:28:26.578 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:28:26.578 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:28:26.579 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:28:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:28:26] "GET /metrics HTTP/1.1" 200 36545 "" "Prometheus/2.51.0" 2024-09-13T14:28:26.672 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:26.702 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:26.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:26.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:26.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:26 smithi195 ceph-mon[104262]: pgmap v376: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.3 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 733 op/s; 25440/308040 objects degraded (8.259%); 1822/308040 objects misplaced (0.591%); 3.6 MiB/s, 56 objects/s recovering 2024-09-13T14:28:26.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:28:26.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:28:26.804 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:26.830 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:26.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:26.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:26.932 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:26.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:26.979 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:26.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:27.081 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:28:27.125 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:27.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:27.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:28:27.227 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:27.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:27.276 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:27.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:27.377 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:27.401 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:27.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:27.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:27.503 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:27.553 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:27.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:27.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:27.655 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:28:27.700 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:27.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:27.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:28:27.802 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:27.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:27.861 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:27.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:27.955 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:28:27.955 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-09-13T14:28:27.955 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-09-13T14:28:27.955 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-13T14:28:27.955 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.955 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:86656 5.7 5:e1f5eeb2:::benchmark_data_smithi102_155989_object86655:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-09-13T14:28:27.956 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:20.763678+0000", 2024-09-13T14:28:27.956 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 67.185825989999998, 2024-09-13T14:28:27.956 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.086803327, 2024-09-13T14:28:27.956 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:27.956 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:27.956 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:27.956 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:27.956 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:27.956 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 86656 2024-09-13T14:28:27.956 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.956 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:27.956 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.957 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:27.957 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:20.763678+0000", 2024-09-13T14:28:27.957 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:27.957 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.957 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.957 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:27.957 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:20.763678+0000", 2024-09-13T14:28:27.957 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:27.957 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.957 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.957 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:27.957 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:20.763676+0000", 2024-09-13T14:28:27.958 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-09-13T14:28:27.958 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.958 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.958 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:27.958 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:20.763727+0000", 2024-09-13T14:28:27.958 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0815000000000001e-05 2024-09-13T14:28:27.958 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.958 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.958 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:27.958 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:20.763743+0000", 2024-09-13T14:28:27.958 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6235999999999998e-05 2024-09-13T14:28:27.958 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.958 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.958 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:27.958 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:20.763752+0000", 2024-09-13T14:28:27.959 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1500000000000005e-06 2024-09-13T14:28:27.959 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.959 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.959 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:27.959 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:20.764030+0000", 2024-09-13T14:28:27.959 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00027786800000000002 2024-09-13T14:28:27.959 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.959 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.959 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:27.959 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:20.764148+0000", 2024-09-13T14:28:27.959 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000118569 2024-09-13T14:28:27.959 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.959 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.959 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,4", 2024-09-13T14:28:27.960 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:20.767055+0000", 2024-09-13T14:28:27.960 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0029061239999999999 2024-09-13T14:28:27.960 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.960 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.960 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:27.960 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:20.772679+0000", 2024-09-13T14:28:27.960 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0056241440000000002 2024-09-13T14:28:27.960 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.960 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.960 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:27.960 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:20.773157+0000", 2024-09-13T14:28:27.960 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00047866599999999999 2024-09-13T14:28:27.960 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.961 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.961 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:27.961 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:20.850354+0000", 2024-09-13T14:28:27.961 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.077196244999999997 2024-09-13T14:28:27.961 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.961 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.961 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:27.961 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:20.850442+0000", 2024-09-13T14:28:27.961 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8183999999999995e-05 2024-09-13T14:28:27.961 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.961 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.961 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:27.961 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:20.850481+0000", 2024-09-13T14:28:27.962 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9576000000000002e-05 2024-09-13T14:28:27.962 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:27.962 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:27.962 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:27.962 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.962 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.962 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:86658 5.6 5:61a62d2a:::benchmark_data_smithi102_155989_object86657:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-09-13T14:28:27.962 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:20.763772+0000", 2024-09-13T14:28:27.962 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 67.185732474999995, 2024-09-13T14:28:27.962 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.090023387999999996, 2024-09-13T14:28:27.962 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:27.962 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:27.962 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:27.962 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:27.962 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:27.962 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 86658 2024-09-13T14:28:27.963 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.963 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:27.963 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.963 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:27.963 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:20.763772+0000", 2024-09-13T14:28:27.963 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:27.963 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.963 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.963 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:27.963 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:20.763772+0000", 2024-09-13T14:28:27.963 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:27.963 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.963 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.963 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:27.963 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:20.763771+0000", 2024-09-13T14:28:27.964 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:28:27.964 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.964 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.964 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:27.964 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:20.763809+0000", 2024-09-13T14:28:27.964 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8164999999999998e-05 2024-09-13T14:28:27.964 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.964 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.964 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:27.964 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:20.763817+0000", 2024-09-13T14:28:27.964 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0949999999999996e-06 2024-09-13T14:28:27.964 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.964 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.964 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:27.964 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:20.763823+0000", 2024-09-13T14:28:27.965 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3140000000000003e-06 2024-09-13T14:28:27.965 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.965 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.965 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:27.965 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:20.763869+0000", 2024-09-13T14:28:27.965 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5909999999999999e-05 2024-09-13T14:28:27.965 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.965 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.965 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:27.965 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:20.764015+0000", 2024-09-13T14:28:27.965 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014596300000000001 2024-09-13T14:28:27.965 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.965 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.965 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 4,7", 2024-09-13T14:28:27.966 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:20.764178+0000", 2024-09-13T14:28:27.966 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016339699999999999 2024-09-13T14:28:27.966 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.966 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.966 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:27.966 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:20.770797+0000", 2024-09-13T14:28:27.966 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0066188499999999999 2024-09-13T14:28:27.966 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.966 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.966 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:27.966 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:20.771821+0000", 2024-09-13T14:28:27.966 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001024175 2024-09-13T14:28:27.966 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.966 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.967 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:27.967 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:20.851063+0000", 2024-09-13T14:28:27.967 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.079242424000000006 2024-09-13T14:28:27.967 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.967 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.967 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:27.967 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:20.851146+0000", 2024-09-13T14:28:27.967 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2231000000000006e-05 2024-09-13T14:28:27.967 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.967 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.967 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:27.967 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:20.853795+0000", 2024-09-13T14:28:27.967 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0026494209999999999 2024-09-13T14:28:27.968 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:27.968 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:27.968 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:27.968 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.968 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.968 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:88019 5.f 5:f5db7c40:::benchmark_data_smithi102_155989_object88018:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:28:27.968 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:22.487876+0000", 2024-09-13T14:28:27.968 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 65.461627710000002, 2024-09-13T14:28:27.968 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.112006882, 2024-09-13T14:28:27.968 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:27.968 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:27.968 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:27.968 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:27.968 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:27.969 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 88019 2024-09-13T14:28:27.969 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.969 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:27.969 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.969 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:27.969 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.487876+0000", 2024-09-13T14:28:27.969 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:27.969 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.969 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.969 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:27.969 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.487876+0000", 2024-09-13T14:28:27.969 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:27.969 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.970 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.970 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:27.970 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.487874+0000", 2024-09-13T14:28:27.970 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-09-13T14:28:27.970 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.970 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.970 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:27.970 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.487909+0000", 2024-09-13T14:28:27.970 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4734000000000003e-05 2024-09-13T14:28:27.970 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.970 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.970 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:27.970 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.487939+0000", 2024-09-13T14:28:27.971 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9903000000000002e-05 2024-09-13T14:28:27.971 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.971 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.971 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:27.971 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.487947+0000", 2024-09-13T14:28:27.971 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.6200000000000005e-06 2024-09-13T14:28:27.971 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.971 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.971 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:27.971 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.488093+0000", 2024-09-13T14:28:27.971 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014596999999999999 2024-09-13T14:28:27.971 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.971 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.971 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:27.972 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.488406+0000", 2024-09-13T14:28:27.972 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00031254200000000002 2024-09-13T14:28:27.972 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.972 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.972 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:28:27.972 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.488795+0000", 2024-09-13T14:28:27.972 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00038913899999999999 2024-09-13T14:28:27.972 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.972 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.972 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:27.972 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.495561+0000", 2024-09-13T14:28:27.972 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0067662540000000002 2024-09-13T14:28:27.972 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.973 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.973 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:27.973 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.495569+0000", 2024-09-13T14:28:27.973 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8679999999999999e-06 2024-09-13T14:28:27.973 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.973 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.973 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:27.973 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.599763+0000", 2024-09-13T14:28:27.973 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.10419413700000001 2024-09-13T14:28:27.973 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.973 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.973 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:27.973 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.599836+0000", 2024-09-13T14:28:27.973 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3108999999999994e-05 2024-09-13T14:28:27.974 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.974 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.974 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:27.974 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.599883+0000", 2024-09-13T14:28:27.974 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6984999999999998e-05 2024-09-13T14:28:27.974 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:27.974 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:27.974 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:27.974 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.974 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.974 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:88023 5.f 5:f4de59b3:::benchmark_data_smithi102_155989_object88022:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:28:27.974 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:22.490563+0000", 2024-09-13T14:28:27.974 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 65.458940795999993, 2024-09-13T14:28:27.975 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.109678688, 2024-09-13T14:28:27.975 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:27.975 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:27.975 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:27.975 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:27.975 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:27.975 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 88023 2024-09-13T14:28:27.975 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.975 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:27.975 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.975 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:27.975 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.490563+0000", 2024-09-13T14:28:27.975 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:27.976 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.976 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.976 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:27.976 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.490563+0000", 2024-09-13T14:28:27.976 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:27.976 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.976 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.976 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:27.976 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.490561+0000", 2024-09-13T14:28:27.976 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-09-13T14:28:27.976 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.976 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.976 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:27.977 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.490619+0000", 2024-09-13T14:28:27.977 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8108e-05 2024-09-13T14:28:27.977 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.977 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.977 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:27.977 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.490644+0000", 2024-09-13T14:28:27.977 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4502e-05 2024-09-13T14:28:27.977 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.977 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.977 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:27.977 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.490651+0000", 2024-09-13T14:28:27.977 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2400000000000001e-06 2024-09-13T14:28:27.977 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.977 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.978 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:27.978 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.492450+0000", 2024-09-13T14:28:27.978 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001799337 2024-09-13T14:28:27.978 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.978 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.978 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:27.978 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.492586+0000", 2024-09-13T14:28:27.978 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000135513 2024-09-13T14:28:27.978 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.978 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.978 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:28:27.978 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.492718+0000", 2024-09-13T14:28:27.978 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000132311 2024-09-13T14:28:27.978 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.979 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.979 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:27.979 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.495749+0000", 2024-09-13T14:28:27.979 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0030307609999999999 2024-09-13T14:28:27.979 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.979 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.979 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:27.979 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.499092+0000", 2024-09-13T14:28:27.979 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0033432280000000002 2024-09-13T14:28:27.979 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.979 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.979 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:27.979 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.600151+0000", 2024-09-13T14:28:27.979 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.10105908399999999 2024-09-13T14:28:27.980 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.980 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.980 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:27.980 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.600204+0000", 2024-09-13T14:28:27.980 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2466000000000002e-05 2024-09-13T14:28:27.980 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.980 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.980 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:27.980 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.600242+0000", 2024-09-13T14:28:27.980 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8334999999999998e-05 2024-09-13T14:28:27.980 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:27.980 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:27.980 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:27.980 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.981 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.981 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:88026 5.6 5:690e1d48:::benchmark_data_smithi102_155989_object88025:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:28:27.981 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:22.492451+0000", 2024-09-13T14:28:27.981 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 65.457053505999994, 2024-09-13T14:28:27.981 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.12953968099999999, 2024-09-13T14:28:27.981 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:27.981 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:27.981 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:27.981 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:27.981 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:27.981 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 88026 2024-09-13T14:28:27.981 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.981 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:27.981 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.982 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:27.982 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.492451+0000", 2024-09-13T14:28:27.982 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:27.982 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.982 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.982 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:27.982 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.492451+0000", 2024-09-13T14:28:27.982 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:27.982 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.982 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.982 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:27.982 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:27.983 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.492449+0000", 2024-09-13T14:28:27.983 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:28:27.983 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.984 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.984 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:27.984 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.492480+0000", 2024-09-13T14:28:27.984 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0494999999999999e-05 2024-09-13T14:28:27.984 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.984 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.984 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:27.984 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.492488+0000", 2024-09-13T14:28:27.984 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.704e-06 2024-09-13T14:28:27.984 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.984 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.984 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:27.984 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.492494+0000", 2024-09-13T14:28:27.985 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.9100000000000002e-06 2024-09-13T14:28:27.985 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.985 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.985 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:27.985 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.494939+0000", 2024-09-13T14:28:27.985 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0024447840000000002 2024-09-13T14:28:27.985 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.985 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.985 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:27.985 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.495079+0000", 2024-09-13T14:28:27.985 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013958500000000001 2024-09-13T14:28:27.985 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.985 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.986 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 4,7", 2024-09-13T14:28:27.986 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.495219+0000", 2024-09-13T14:28:27.986 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014044400000000001 2024-09-13T14:28:27.986 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.986 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.986 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:27.986 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.502484+0000", 2024-09-13T14:28:27.986 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0072652560000000003 2024-09-13T14:28:27.986 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.986 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.986 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:27.986 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.502687+0000", 2024-09-13T14:28:27.986 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00020273399999999999 2024-09-13T14:28:27.986 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.986 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.987 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:27.987 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.620053+0000", 2024-09-13T14:28:27.987 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.117365497 2024-09-13T14:28:27.987 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.987 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.987 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:27.987 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.620153+0000", 2024-09-13T14:28:27.987 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010079899999999999 2024-09-13T14:28:27.987 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.987 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.987 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:27.987 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.621990+0000", 2024-09-13T14:28:27.987 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0018368919999999999 2024-09-13T14:28:27.988 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:27.988 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:27.988 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:27.988 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.988 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.988 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:88033 5.7 5:ecf99ba3:::benchmark_data_smithi102_155989_object88032:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:28:27.988 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:22.504492+0000", 2024-09-13T14:28:27.988 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 65.445012152999993, 2024-09-13T14:28:27.988 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.095869687999999995, 2024-09-13T14:28:27.988 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:27.988 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:27.988 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:27.988 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:27.989 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:27.989 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 88033 2024-09-13T14:28:27.989 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.989 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:27.989 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.989 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:27.989 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.504492+0000", 2024-09-13T14:28:27.989 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:27.989 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.989 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.989 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:27.989 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.504492+0000", 2024-09-13T14:28:27.989 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:27.989 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.990 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.990 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:27.990 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.504490+0000", 2024-09-13T14:28:27.990 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:28:27.990 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.990 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.990 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:27.990 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.504533+0000", 2024-09-13T14:28:27.990 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3103999999999999e-05 2024-09-13T14:28:27.990 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.990 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.990 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:27.990 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.504553+0000", 2024-09-13T14:28:27.990 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9419000000000002e-05 2024-09-13T14:28:27.991 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.991 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.991 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:27.991 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.504561+0000", 2024-09-13T14:28:27.991 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1769999999999998e-06 2024-09-13T14:28:27.991 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.991 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.991 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:27.991 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.505224+0000", 2024-09-13T14:28:27.991 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00066350700000000003 2024-09-13T14:28:27.991 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.991 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.991 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:27.991 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.505370+0000", 2024-09-13T14:28:27.992 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000145883 2024-09-13T14:28:27.992 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.992 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.992 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,4", 2024-09-13T14:28:27.992 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.505507+0000", 2024-09-13T14:28:27.992 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013690499999999999 2024-09-13T14:28:27.992 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.992 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.992 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:27.992 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.508325+0000", 2024-09-13T14:28:27.992 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0028180190000000002 2024-09-13T14:28:27.992 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.992 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.992 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:27.993 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.508681+0000", 2024-09-13T14:28:27.993 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00035594199999999999 2024-09-13T14:28:27.993 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.993 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.993 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:27.993 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.599943+0000", 2024-09-13T14:28:27.993 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.091262006000000007 2024-09-13T14:28:27.993 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.993 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.993 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:27.993 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.600324+0000", 2024-09-13T14:28:27.993 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00038048900000000002 2024-09-13T14:28:27.993 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.993 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.994 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:27.994 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.600362+0000", 2024-09-13T14:28:27.994 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8043000000000002e-05 2024-09-13T14:28:27.994 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:27.994 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:27.994 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:27.994 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.994 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.994 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:88037 5.6 5:64aee5a5:::benchmark_data_smithi102_155989_object88036:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:28:27.994 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:22.509816+0000", 2024-09-13T14:28:27.994 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 65.439687802999998, 2024-09-13T14:28:27.994 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.115979078, 2024-09-13T14:28:27.994 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:27.995 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:27.995 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:27.995 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:27.995 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:27.995 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 88037 2024-09-13T14:28:27.995 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.995 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:27.995 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.995 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:27.995 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.509816+0000", 2024-09-13T14:28:27.995 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:27.995 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.995 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.995 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:27.996 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.509816+0000", 2024-09-13T14:28:27.996 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:27.996 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.996 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.996 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:27.996 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.509815+0000", 2024-09-13T14:28:27.996 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:28:27.996 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.996 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.996 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:27.996 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.509866+0000", 2024-09-13T14:28:27.996 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0902000000000002e-05 2024-09-13T14:28:27.996 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.997 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.997 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:27.997 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.509875+0000", 2024-09-13T14:28:27.997 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.3260000000000002e-06 2024-09-13T14:28:27.997 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.997 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.997 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:27.997 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.509880+0000", 2024-09-13T14:28:27.997 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.391e-06 2024-09-13T14:28:27.997 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.997 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.997 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:27.997 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.509922+0000", 2024-09-13T14:28:27.997 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2447999999999997e-05 2024-09-13T14:28:27.998 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.998 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.998 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:27.998 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.510054+0000", 2024-09-13T14:28:27.998 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000131452 2024-09-13T14:28:27.998 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.998 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.998 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 4,7", 2024-09-13T14:28:27.998 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.510186+0000", 2024-09-13T14:28:27.998 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000132473 2024-09-13T14:28:27.998 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.998 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.998 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:27.998 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.512743+0000", 2024-09-13T14:28:27.999 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0025567160000000001 2024-09-13T14:28:27.999 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.999 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.999 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:27.999 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.513038+0000", 2024-09-13T14:28:27.999 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00029513400000000001 2024-09-13T14:28:27.999 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.999 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:27.999 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:27.999 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.623346+0000", 2024-09-13T14:28:27.999 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.110307751 2024-09-13T14:28:27.999 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:27.999 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.000 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:28.000 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.623509+0000", 2024-09-13T14:28:28.000 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016350300000000001 2024-09-13T14:28:28.000 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.000 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.000 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:28.000 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.625795+0000", 2024-09-13T14:28:28.000 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0022861460000000002 2024-09-13T14:28:28.000 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:28.000 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:28.000 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:28.000 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.000 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.000 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:91082 5.6 5:6b27ebdc:::benchmark_data_smithi102_155989_object91081:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:28:28.001 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:26.508623+0000", 2024-09-13T14:28:28.001 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 61.440881437999998, 2024-09-13T14:28:28.001 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.086224054999999994, 2024-09-13T14:28:28.001 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:28.001 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:28.001 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:28.001 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:28.001 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:28.001 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 91082 2024-09-13T14:28:28.001 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.001 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:28.001 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.001 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:28.002 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:26.508623+0000", 2024-09-13T14:28:28.002 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:28.002 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.002 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.002 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:28.002 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:26.508623+0000", 2024-09-13T14:28:28.002 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:28.002 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.002 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.002 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:28.002 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:26.508622+0000", 2024-09-13T14:28:28.002 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:28:28.002 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.003 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.003 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:28.003 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:26.508657+0000", 2024-09-13T14:28:28.003 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4581e-05 2024-09-13T14:28:28.003 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.003 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.003 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:28.003 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:26.508663+0000", 2024-09-13T14:28:28.003 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5359999999999998e-06 2024-09-13T14:28:28.003 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.003 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.003 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:28.003 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:26.508668+0000", 2024-09-13T14:28:28.003 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3000000000000003e-06 2024-09-13T14:28:28.004 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.004 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.004 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:28.004 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:26.525654+0000", 2024-09-13T14:28:28.004 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016986211000000001 2024-09-13T14:28:28.004 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.004 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.004 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:28.004 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:26.525744+0000", 2024-09-13T14:28:28.004 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.0427999999999995e-05 2024-09-13T14:28:28.004 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.004 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.005 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 4,7", 2024-09-13T14:28:28.005 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:26.525848+0000", 2024-09-13T14:28:28.005 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000103819 2024-09-13T14:28:28.005 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.005 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.005 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:28.005 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:26.532089+0000", 2024-09-13T14:28:28.005 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0062404519999999996 2024-09-13T14:28:28.005 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.005 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.005 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:28.005 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:26.541844+0000", 2024-09-13T14:28:28.005 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0097551540000000003 2024-09-13T14:28:28.005 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.006 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.006 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:28.006 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:26.590652+0000", 2024-09-13T14:28:28.006 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.048808076999999998 2024-09-13T14:28:28.006 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.006 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.006 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:28.006 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:26.590743+0000", 2024-09-13T14:28:28.006 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.0812999999999999e-05 2024-09-13T14:28:28.006 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.006 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.006 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:28.006 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:26.594847+0000", 2024-09-13T14:28:28.007 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0041041510000000003 2024-09-13T14:28:28.007 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:28.007 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:28.007 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:28.007 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.007 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.007 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:91667 5.7 5:e8d5d3ed:::benchmark_data_smithi102_155989_object91666:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:28:28.007 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:27.328056+0000", 2024-09-13T14:28:28.007 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 60.621448373, 2024-09-13T14:28:28.007 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.088713318999999999, 2024-09-13T14:28:28.007 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:28.007 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:28.008 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:28.008 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:28.008 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:28.008 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 91667 2024-09-13T14:28:28.008 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.008 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:28.008 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.008 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:28.008 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.328056+0000", 2024-09-13T14:28:28.008 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:28.008 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.008 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.008 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:28.009 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.328056+0000", 2024-09-13T14:28:28.009 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:28.009 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.009 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.009 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:28.009 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.328054+0000", 2024-09-13T14:28:28.009 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:28:28.009 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.009 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.009 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:28.009 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.328184+0000", 2024-09-13T14:28:28.009 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013020800000000001 2024-09-13T14:28:28.009 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.009 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.010 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:28.010 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.328198+0000", 2024-09-13T14:28:28.010 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3448999999999999e-05 2024-09-13T14:28:28.010 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.010 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.010 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:28.010 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.328215+0000", 2024-09-13T14:28:28.010 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7289999999999999e-05 2024-09-13T14:28:28.010 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.010 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.010 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:28.010 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.328322+0000", 2024-09-13T14:28:28.010 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000106536 2024-09-13T14:28:28.010 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.011 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.011 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:28.011 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.328470+0000", 2024-09-13T14:28:28.011 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014837800000000001 2024-09-13T14:28:28.011 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.011 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.011 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,4", 2024-09-13T14:28:28.011 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.328605+0000", 2024-09-13T14:28:28.011 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013538800000000001 2024-09-13T14:28:28.011 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.011 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.011 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:28.011 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.332732+0000", 2024-09-13T14:28:28.012 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.004126523 2024-09-13T14:28:28.012 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.012 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.012 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:28.012 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.333341+0000", 2024-09-13T14:28:28.012 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00060955600000000003 2024-09-13T14:28:28.012 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.012 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.012 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:28.012 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.416663+0000", 2024-09-13T14:28:28.012 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.083321602999999994 2024-09-13T14:28:28.012 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.012 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.013 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:28.013 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.416722+0000", 2024-09-13T14:28:28.013 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.9271999999999998e-05 2024-09-13T14:28:28.013 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.013 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.013 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:28.013 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.416769+0000", 2024-09-13T14:28:28.013 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6706999999999999e-05 2024-09-13T14:28:28.013 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:28.013 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:28.013 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:28.013 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.013 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.014 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:113965 5.6 5:6bc7c9a8:::benchmark_data_smithi102_155989_object113964:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:28:28.014 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:55.701689+0000", 2024-09-13T14:28:28.014 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 32.247814806999997, 2024-09-13T14:28:28.014 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.085767901999999993, 2024-09-13T14:28:28.014 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:28.014 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:28.014 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:28.014 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:28.014 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:28.014 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113965 2024-09-13T14:28:28.014 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.014 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:28.014 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.015 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:28.015 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.701689+0000", 2024-09-13T14:28:28.015 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:28.015 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.015 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.015 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:28.015 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.701689+0000", 2024-09-13T14:28:28.015 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:28.015 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.015 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.015 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:28.015 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.701688+0000", 2024-09-13T14:28:28.016 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:28:28.016 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.016 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.016 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:28.016 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.701743+0000", 2024-09-13T14:28:28.016 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5108000000000002e-05 2024-09-13T14:28:28.016 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.016 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.016 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:28.016 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.701756+0000", 2024-09-13T14:28:28.016 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3013999999999999e-05 2024-09-13T14:28:28.016 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.016 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.017 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:28.017 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.701763+0000", 2024-09-13T14:28:28.017 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7399999999999998e-06 2024-09-13T14:28:28.017 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.017 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.017 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:28.017 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.706689+0000", 2024-09-13T14:28:28.017 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0049261699999999997 2024-09-13T14:28:28.017 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.017 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.017 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:28.017 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.706858+0000", 2024-09-13T14:28:28.017 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000169109 2024-09-13T14:28:28.018 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.018 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.018 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 4,7", 2024-09-13T14:28:28.018 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.707249+0000", 2024-09-13T14:28:28.018 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00039047499999999999 2024-09-13T14:28:28.018 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.018 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.018 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:28.018 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.712377+0000", 2024-09-13T14:28:28.018 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.005127983 2024-09-13T14:28:28.018 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.018 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.018 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:28.018 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.719793+0000", 2024-09-13T14:28:28.019 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0074162189999999999 2024-09-13T14:28:28.019 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.019 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.019 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:28.019 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.785754+0000", 2024-09-13T14:28:28.019 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.065961346000000004 2024-09-13T14:28:28.019 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.019 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.019 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:28.019 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.785846+0000", 2024-09-13T14:28:28.019 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1927000000000003e-05 2024-09-13T14:28:28.019 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.019 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.019 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:28.020 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.787457+0000", 2024-09-13T14:28:28.020 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0016107809999999999 2024-09-13T14:28:28.020 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:28.020 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:28.020 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:28.020 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.020 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.020 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:115228 5.6 5:69ce3cc2:::benchmark_data_smithi102_155989_object115227:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:28:28.020 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:57.376547+0000", 2024-09-13T14:28:28.020 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 30.572957428999999, 2024-09-13T14:28:28.020 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.089655372999999997, 2024-09-13T14:28:28.020 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:28.020 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:28.021 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:28.021 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:28.021 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:28.021 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 115228 2024-09-13T14:28:28.021 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.021 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:28.021 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.021 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:28.021 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:57.376547+0000", 2024-09-13T14:28:28.021 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:28.021 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.021 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.021 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:28.021 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:57.376547+0000", 2024-09-13T14:28:28.021 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:28.022 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.022 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.022 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:28.022 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:57.376543+0000", 2024-09-13T14:28:28.022 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999962 2024-09-13T14:28:28.022 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.022 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.022 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:28.022 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:57.376586+0000", 2024-09-13T14:28:28.022 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3130999999999998e-05 2024-09-13T14:28:28.022 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.022 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.022 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:28.022 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:57.376601+0000", 2024-09-13T14:28:28.023 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4758e-05 2024-09-13T14:28:28.023 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.023 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.023 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:28.023 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:57.376608+0000", 2024-09-13T14:28:28.023 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4070000000000004e-06 2024-09-13T14:28:28.023 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.023 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.023 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:28.023 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:57.376662+0000", 2024-09-13T14:28:28.023 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4073000000000002e-05 2024-09-13T14:28:28.023 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.023 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.023 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:28.024 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:57.376844+0000", 2024-09-13T14:28:28.024 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000181767 2024-09-13T14:28:28.024 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.024 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.024 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 4,7", 2024-09-13T14:28:28.024 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:57.377010+0000", 2024-09-13T14:28:28.024 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000166596 2024-09-13T14:28:28.024 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.024 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.024 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:28.024 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:57.387668+0000", 2024-09-13T14:28:28.024 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010657257 2024-09-13T14:28:28.024 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.025 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.025 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:28.025 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:57.387706+0000", 2024-09-13T14:28:28.025 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7879000000000001e-05 2024-09-13T14:28:28.025 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.025 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.025 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:28.025 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:57.464165+0000", 2024-09-13T14:28:28.027 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.076459898999999998 2024-09-13T14:28:28.027 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.027 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.027 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:28.027 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:57.464211+0000", 2024-09-13T14:28:28.027 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5457999999999997e-05 2024-09-13T14:28:28.028 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.028 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.028 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:28.028 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:57.466202+0000", 2024-09-13T14:28:28.028 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0019911759999999999 2024-09-13T14:28:28.028 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:28.028 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:28.028 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:28.028 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.028 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.028 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:115244 5.6 5:66259d9c:::benchmark_data_smithi102_155989_object115243:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:28:28.029 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:57.392285+0000", 2024-09-13T14:28:28.029 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 30.55721896, 2024-09-13T14:28:28.029 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.087043112000000006, 2024-09-13T14:28:28.029 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:28.029 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:28.029 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:28.029 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:28.029 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:28.029 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 115244 2024-09-13T14:28:28.029 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.029 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:28.029 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.029 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:28.030 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:57.392285+0000", 2024-09-13T14:28:28.030 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:28.030 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.030 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.030 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:28.030 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:57.392285+0000", 2024-09-13T14:28:28.030 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:28.030 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.030 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.030 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:28.030 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:57.392284+0000", 2024-09-13T14:28:28.030 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:28:28.030 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.030 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.031 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:28.031 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:57.392320+0000", 2024-09-13T14:28:28.031 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6269000000000001e-05 2024-09-13T14:28:28.031 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.031 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.031 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:28.031 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:57.392333+0000", 2024-09-13T14:28:28.031 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2658e-05 2024-09-13T14:28:28.031 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.031 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.031 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:28.031 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:57.392339+0000", 2024-09-13T14:28:28.031 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8440000000000003e-06 2024-09-13T14:28:28.031 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.032 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.032 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:28.032 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:57.394575+0000", 2024-09-13T14:28:28.032 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0022362020000000001 2024-09-13T14:28:28.032 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.032 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.032 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:28.032 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:57.394749+0000", 2024-09-13T14:28:28.032 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00017389599999999999 2024-09-13T14:28:28.032 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.032 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.033 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 4,7", 2024-09-13T14:28:28.033 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:57.394932+0000", 2024-09-13T14:28:28.033 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000183172 2024-09-13T14:28:28.033 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.033 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.033 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:28.033 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:57.405053+0000", 2024-09-13T14:28:28.033 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010121009 2024-09-13T14:28:28.033 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.033 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.033 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:28.034 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:57.451474+0000", 2024-09-13T14:28:28.034 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.046420701000000002 2024-09-13T14:28:28.034 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.034 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.034 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:28.034 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:57.473692+0000", 2024-09-13T14:28:28.034 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.02221822 2024-09-13T14:28:28.034 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.034 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.034 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:28.034 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:57.473746+0000", 2024-09-13T14:28:28.034 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4497999999999999e-05 2024-09-13T14:28:28.034 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.035 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.035 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:28.035 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:57.479328+0000", 2024-09-13T14:28:28.035 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0055819650000000004 2024-09-13T14:28:28.035 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:28.035 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:28.035 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:28.035 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.035 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.035 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:116702 5.7 5:ec26bcd9:::benchmark_data_smithi102_155989_object116701:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:28:28.036 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:59.405902+0000", 2024-09-13T14:28:28.036 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 28.543601839000001, 2024-09-13T14:28:28.036 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.086690046000000007, 2024-09-13T14:28:28.036 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:28.036 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:28.036 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:28.036 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:28.036 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:28.036 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 116702 2024-09-13T14:28:28.036 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.036 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:28.036 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.037 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:28.037 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.405902+0000", 2024-09-13T14:28:28.037 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:28.037 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.037 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.037 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:28.037 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.405902+0000", 2024-09-13T14:28:28.037 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:28.037 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.037 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.037 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:28.038 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.405897+0000", 2024-09-13T14:28:28.038 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999952 2024-09-13T14:28:28.038 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.038 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.038 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:28.038 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.405952+0000", 2024-09-13T14:28:28.038 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4654999999999998e-05 2024-09-13T14:28:28.039 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.039 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.039 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:28.039 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.405966+0000", 2024-09-13T14:28:28.039 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3864e-05 2024-09-13T14:28:28.040 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.040 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.040 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:28.040 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.405974+0000", 2024-09-13T14:28:28.040 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1380000000000007e-06 2024-09-13T14:28:28.041 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.041 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.041 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:28.041 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.406991+0000", 2024-09-13T14:28:28.041 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0010166800000000001 2024-09-13T14:28:28.041 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.042 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.042 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:28.042 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.407098+0000", 2024-09-13T14:28:28.042 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000107593 2024-09-13T14:28:28.042 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.042 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.042 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,4", 2024-09-13T14:28:28.042 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.407219+0000", 2024-09-13T14:28:28.042 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000120543 2024-09-13T14:28:28.042 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.042 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.042 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:28.042 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.411178+0000", 2024-09-13T14:28:28.043 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0039592799999999999 2024-09-13T14:28:28.043 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.043 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.043 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:28.043 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.411648+0000", 2024-09-13T14:28:28.043 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000469536 2024-09-13T14:28:28.043 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.043 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.043 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:28.043 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.492485+0000", 2024-09-13T14:28:28.043 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.080837120999999998 2024-09-13T14:28:28.043 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.044 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.044 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:28.044 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.492538+0000", 2024-09-13T14:28:28.044 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2728000000000001e-05 2024-09-13T14:28:28.044 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.044 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.044 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:28.044 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.492592+0000", 2024-09-13T14:28:28.044 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4728999999999997e-05 2024-09-13T14:28:28.044 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:28.044 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:28.045 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:28.045 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.045 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.045 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:127079 5.6 5:6f5a457a:::benchmark_data_smithi102_155989_object9161:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-13T14:28:28.045 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:28:13.138204+0000", 2024-09-13T14:28:28.045 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 14.811299624, 2024-09-13T14:28:28.046 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.087514321000000006, 2024-09-13T14:28:28.046 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:28.046 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:28.046 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:28.046 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:28.046 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:28.046 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 127079 2024-09-13T14:28:28.046 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.047 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:28.047 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.047 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:28.047 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.138204+0000", 2024-09-13T14:28:28.047 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:28.047 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.047 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.048 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:28.048 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.138204+0000", 2024-09-13T14:28:28.048 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:28.048 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.048 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.048 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:28.048 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.138204+0000", 2024-09-13T14:28:28.048 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:28:28.049 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.049 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.049 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:28.049 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.138209+0000", 2024-09-13T14:28:28.049 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1839999999999998e-06 2024-09-13T14:28:28.049 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.049 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.049 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:28.049 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.138219+0000", 2024-09-13T14:28:28.049 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0767999999999999e-05 2024-09-13T14:28:28.050 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.050 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.050 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:28.050 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.138226+0000", 2024-09-13T14:28:28.050 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7719999999999997e-06 2024-09-13T14:28:28.050 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.050 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.050 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:28.050 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.149684+0000", 2024-09-13T14:28:28.050 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011457943 2024-09-13T14:28:28.051 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.051 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.051 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:28.051 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.149839+0000", 2024-09-13T14:28:28.051 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000154544 2024-09-13T14:28:28.051 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.052 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.052 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 4,7", 2024-09-13T14:28:28.052 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.150001+0000", 2024-09-13T14:28:28.052 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016251500000000001 2024-09-13T14:28:28.052 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.052 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.052 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:28.052 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.159340+0000", 2024-09-13T14:28:28.052 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0093392430000000005 2024-09-13T14:28:28.053 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.053 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.053 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:28.053 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.161580+0000", 2024-09-13T14:28:28.053 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0022396920000000002 2024-09-13T14:28:28.053 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.053 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.053 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:28.053 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.223966+0000", 2024-09-13T14:28:28.053 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.062386034999999999 2024-09-13T14:28:28.054 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.054 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.054 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:28.054 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.224062+0000", 2024-09-13T14:28:28.054 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.5303000000000006e-05 2024-09-13T14:28:28.054 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.054 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.054 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:28.054 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.225719+0000", 2024-09-13T14:28:28.054 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001657293 2024-09-13T14:28:28.054 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:28.054 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:28.054 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:28.055 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.055 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.055 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:129230 5.6 5:66517ca0:::benchmark_data_smithi102_155989_object11312:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-13T14:28:28.055 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:28:16.050473+0000", 2024-09-13T14:28:28.055 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 11.899031281999999, 2024-09-13T14:28:28.055 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.114388369, 2024-09-13T14:28:28.055 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:28.055 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:28.055 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:28.055 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:28.056 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:28.056 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 129230 2024-09-13T14:28:28.056 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.056 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:28.056 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.056 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:28.056 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.050473+0000", 2024-09-13T14:28:28.056 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:28.056 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.056 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.056 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:28.056 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.050473+0000", 2024-09-13T14:28:28.057 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:28.057 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.057 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.057 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:28.057 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.050471+0000", 2024-09-13T14:28:28.057 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:28:28.057 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.057 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.057 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:28.058 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.050478+0000", 2024-09-13T14:28:28.058 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3999999999999997e-06 2024-09-13T14:28:28.058 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.058 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.058 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:28.058 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.050492+0000", 2024-09-13T14:28:28.058 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4786e-05 2024-09-13T14:28:28.060 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.060 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.060 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:28.060 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.050500+0000", 2024-09-13T14:28:28.060 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6699999999999994e-06 2024-09-13T14:28:28.060 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.060 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.060 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:28.060 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.050615+0000", 2024-09-13T14:28:28.060 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000114876 2024-09-13T14:28:28.061 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.061 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.061 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:28.061 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.050962+0000", 2024-09-13T14:28:28.061 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00034689999999999998 2024-09-13T14:28:28.061 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.061 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.061 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 4,7", 2024-09-13T14:28:28.061 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.051342+0000", 2024-09-13T14:28:28.061 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00037975999999999998 2024-09-13T14:28:28.061 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.061 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.061 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:28.062 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.056287+0000", 2024-09-13T14:28:28.062 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0049449749999999999 2024-09-13T14:28:28.062 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.062 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.062 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:28.062 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.070382+0000", 2024-09-13T14:28:28.062 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014095524 2024-09-13T14:28:28.062 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.062 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.062 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:28.062 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.163313+0000", 2024-09-13T14:28:28.062 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.092930615999999994 2024-09-13T14:28:28.062 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.062 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.063 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:28.063 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.163368+0000", 2024-09-13T14:28:28.063 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5072999999999999e-05 2024-09-13T14:28:28.063 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.063 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.063 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:28.063 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.164861+0000", 2024-09-13T14:28:28.063 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001493381 2024-09-13T14:28:28.063 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:28.063 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:28.063 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:28.063 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.063 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.063 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:129237 5.6 5:652651e6:::benchmark_data_smithi102_155989_object11319:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-13T14:28:28.063 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:28:16.060587+0000", 2024-09-13T14:28:28.064 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 11.888917445000001, 2024-09-13T14:28:28.064 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.10567101600000001, 2024-09-13T14:28:28.064 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:28.064 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:28.064 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:28.064 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:28.064 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:28.064 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 129237 2024-09-13T14:28:28.064 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.064 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:28.065 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.065 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:28.065 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.060587+0000", 2024-09-13T14:28:28.065 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:28.065 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.065 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.065 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:28.065 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.060587+0000", 2024-09-13T14:28:28.065 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:28.065 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.065 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.065 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:28.066 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.060585+0000", 2024-09-13T14:28:28.066 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:28:28.066 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.066 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.066 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:28.066 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.060592+0000", 2024-09-13T14:28:28.066 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.55e-06 2024-09-13T14:28:28.066 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.066 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.066 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:28.066 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.060603+0000", 2024-09-13T14:28:28.066 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1827999999999999e-05 2024-09-13T14:28:28.066 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.067 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.067 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:28.067 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.060610+0000", 2024-09-13T14:28:28.067 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3670000000000002e-06 2024-09-13T14:28:28.067 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.067 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.067 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:28.067 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.064136+0000", 2024-09-13T14:28:28.067 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0035264850000000002 2024-09-13T14:28:28.068 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.068 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.068 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:28.068 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.064280+0000", 2024-09-13T14:28:28.068 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014394899999999999 2024-09-13T14:28:28.068 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.068 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.068 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 4,7", 2024-09-13T14:28:28.068 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.064418+0000", 2024-09-13T14:28:28.068 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013817899999999999 2024-09-13T14:28:28.068 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.068 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.069 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:28.069 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.068555+0000", 2024-09-13T14:28:28.069 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0041362669999999999 2024-09-13T14:28:28.069 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.069 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.069 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:28.069 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.070413+0000", 2024-09-13T14:28:28.069 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001858878 2024-09-13T14:28:28.069 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.070 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.070 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:28.070 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.164902+0000", 2024-09-13T14:28:28.070 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.094488614999999998 2024-09-13T14:28:28.070 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.070 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.070 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:28.070 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.164965+0000", 2024-09-13T14:28:28.070 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3299999999999994e-05 2024-09-13T14:28:28.070 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.070 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.070 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:28.070 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.166258+0000", 2024-09-13T14:28:28.071 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001292287 2024-09-13T14:28:28.071 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:28.071 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:28.071 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:28.071 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.071 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.071 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:129257 5.6 5:69bb1e7f:::benchmark_data_smithi102_155989_object11339:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-13T14:28:28.071 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:28:16.083136+0000", 2024-09-13T14:28:28.071 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 11.866367835, 2024-09-13T14:28:28.071 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.091234220000000005, 2024-09-13T14:28:28.071 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:28.071 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:28.071 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:28.072 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:28.072 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:28.072 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 129257 2024-09-13T14:28:28.072 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.072 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:28.072 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.072 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:28.072 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.083136+0000", 2024-09-13T14:28:28.072 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:28.072 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.072 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.072 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:28.072 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.083136+0000", 2024-09-13T14:28:28.073 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:28.073 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.073 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.073 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:28.073 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.083135+0000", 2024-09-13T14:28:28.073 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:28:28.073 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.073 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.073 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:28.073 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.083140+0000", 2024-09-13T14:28:28.073 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.814e-06 2024-09-13T14:28:28.073 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.073 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.074 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:28.074 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.083156+0000", 2024-09-13T14:28:28.074 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5886999999999999e-05 2024-09-13T14:28:28.074 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.074 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.074 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:28.074 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.083162+0000", 2024-09-13T14:28:28.074 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7120000000000003e-06 2024-09-13T14:28:28.074 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.074 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.075 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:28.075 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.083273+0000", 2024-09-13T14:28:28.075 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000110272 2024-09-13T14:28:28.075 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.075 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.075 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:28.075 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.083425+0000", 2024-09-13T14:28:28.075 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015257899999999999 2024-09-13T14:28:28.075 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.075 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.075 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 4,7", 2024-09-13T14:28:28.075 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.083580+0000", 2024-09-13T14:28:28.075 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000155091 2024-09-13T14:28:28.075 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.076 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.076 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:28.076 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.090636+0000", 2024-09-13T14:28:28.076 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0070557129999999999 2024-09-13T14:28:28.076 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.076 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.076 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:28.076 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.099880+0000", 2024-09-13T14:28:28.076 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0092442679999999999 2024-09-13T14:28:28.076 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.077 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.077 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:28.077 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.172570+0000", 2024-09-13T14:28:28.077 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.072689826999999999 2024-09-13T14:28:28.077 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.077 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.077 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:28.077 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.172660+0000", 2024-09-13T14:28:28.077 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.9443000000000004e-05 2024-09-13T14:28:28.077 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.077 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.077 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:28.077 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.174370+0000", 2024-09-13T14:28:28.078 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001710961 2024-09-13T14:28:28.078 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:28.078 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:28.078 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:28.078 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.078 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.078 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:129793 5.6 5:6765a86e:::benchmark_data_smithi102_155989_object11875:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-13T14:28:28.078 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:28:16.842703+0000", 2024-09-13T14:28:28.078 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 11.106801001000001, 2024-09-13T14:28:28.078 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.086888147999999998, 2024-09-13T14:28:28.078 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:28.078 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:28.079 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:28.079 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:28.079 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:28.079 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 129793 2024-09-13T14:28:28.079 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.079 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:28.079 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.079 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:28.079 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.842703+0000", 2024-09-13T14:28:28.079 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:28.079 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.079 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.080 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:28.080 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.842703+0000", 2024-09-13T14:28:28.080 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:28.080 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.080 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.080 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:28.080 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.842701+0000", 2024-09-13T14:28:28.080 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:28:28.080 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.080 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.080 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:28.081 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.842714+0000", 2024-09-13T14:28:28.081 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2344000000000001e-05 2024-09-13T14:28:28.081 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.081 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.081 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:28.081 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.842731+0000", 2024-09-13T14:28:28.081 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7581e-05 2024-09-13T14:28:28.081 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.081 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.082 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:28.083 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.842739+0000", 2024-09-13T14:28:28.083 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8480000000000001e-06 2024-09-13T14:28:28.083 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.083 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.083 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:28.083 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.850134+0000", 2024-09-13T14:28:28.083 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0073946330000000003 2024-09-13T14:28:28.083 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.083 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.083 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:28.083 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.850295+0000", 2024-09-13T14:28:28.083 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016169000000000001 2024-09-13T14:28:28.084 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.084 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.084 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 4,7", 2024-09-13T14:28:28.084 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.850454+0000", 2024-09-13T14:28:28.084 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015910699999999999 2024-09-13T14:28:28.084 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.084 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.084 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:28.084 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.860663+0000", 2024-09-13T14:28:28.084 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010208873 2024-09-13T14:28:28.085 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.085 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.085 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:28.085 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.878702+0000", 2024-09-13T14:28:28.085 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018038348999999999 2024-09-13T14:28:28.085 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.085 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.085 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:28.085 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.927836+0000", 2024-09-13T14:28:28.086 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.049134323000000001 2024-09-13T14:28:28.086 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.086 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.086 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:28.086 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.927871+0000", 2024-09-13T14:28:28.086 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4801000000000001e-05 2024-09-13T14:28:28.086 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.086 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.086 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:28.086 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.929591+0000", 2024-09-13T14:28:28.087 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001720428 2024-09-13T14:28:28.087 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:28.087 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:28.087 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:28.087 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.087 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.087 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:129795 5.6 5:6de50133:::benchmark_data_smithi102_155989_object11877:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-13T14:28:28.087 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:28:16.844029+0000", 2024-09-13T14:28:28.088 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 11.105474925999999, 2024-09-13T14:28:28.088 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.087258843000000003, 2024-09-13T14:28:28.088 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:28.088 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:28.088 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:28.088 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:28.088 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:28.088 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 129795 2024-09-13T14:28:28.089 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.089 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:28.089 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.089 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:28.089 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.844029+0000", 2024-09-13T14:28:28.089 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:28.089 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.089 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.089 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:28.090 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.844029+0000", 2024-09-13T14:28:28.090 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:28.090 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.090 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.090 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:28.090 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.844028+0000", 2024-09-13T14:28:28.090 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:28:28.090 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.091 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.091 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:28.091 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.844033+0000", 2024-09-13T14:28:28.091 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5300000000000004e-06 2024-09-13T14:28:28.091 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.091 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.091 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:28.091 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.844046+0000", 2024-09-13T14:28:28.091 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2724e-05 2024-09-13T14:28:28.092 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.092 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.092 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:28.092 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.844053+0000", 2024-09-13T14:28:28.092 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.0960000000000003e-06 2024-09-13T14:28:28.092 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.092 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.092 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:28.093 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.854827+0000", 2024-09-13T14:28:28.093 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010774149 2024-09-13T14:28:28.093 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.093 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.093 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:28.093 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.855087+0000", 2024-09-13T14:28:28.093 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00025952899999999999 2024-09-13T14:28:28.093 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.093 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.094 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 4,7", 2024-09-13T14:28:28.094 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.855321+0000", 2024-09-13T14:28:28.094 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000233952 2024-09-13T14:28:28.094 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.094 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.094 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:28.094 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.872943+0000", 2024-09-13T14:28:28.094 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017622695000000001 2024-09-13T14:28:28.095 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.095 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.095 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:28.095 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.891532+0000", 2024-09-13T14:28:28.095 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018588562999999999 2024-09-13T14:28:28.095 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.095 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.095 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:28.095 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.929757+0000", 2024-09-13T14:28:28.096 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.038224969999999997 2024-09-13T14:28:28.096 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.096 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.096 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:28.096 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.929808+0000", 2024-09-13T14:28:28.096 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1038999999999997e-05 2024-09-13T14:28:28.096 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.096 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.097 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:28.097 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:16.931288+0000", 2024-09-13T14:28:28.097 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001480021 2024-09-13T14:28:28.097 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:28.097 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:28.097 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:28.097 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.097 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.098 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:137384 5.6 5:67c0ca92:::benchmark_data_smithi102_155989_object19466:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-09-13T14:28:28.098 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:28:27.370983+0000", 2024-09-13T14:28:28.098 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.57852072799999998, 2024-09-13T14:28:28.098 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.085928550000000006, 2024-09-13T14:28:28.098 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:28.098 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:28.098 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:28.098 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:28.098 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:28.099 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 137384 2024-09-13T14:28:28.099 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.099 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:28.099 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.099 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:28.099 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:27.370983+0000", 2024-09-13T14:28:28.099 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:28.099 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.100 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.100 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:28.100 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:27.370983+0000", 2024-09-13T14:28:28.100 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:28.100 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.100 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.100 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:28.100 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:27.370982+0000", 2024-09-13T14:28:28.101 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:28:28.101 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.101 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.101 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:28.101 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:27.370988+0000", 2024-09-13T14:28:28.101 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1290000000000001e-06 2024-09-13T14:28:28.101 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.101 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.101 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:28.101 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:27.370998+0000", 2024-09-13T14:28:28.101 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0438999999999999e-05 2024-09-13T14:28:28.101 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.102 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.102 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:28.102 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:27.371005+0000", 2024-09-13T14:28:28.102 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9720000000000003e-06 2024-09-13T14:28:28.102 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.102 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.102 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:28.102 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:27.372828+0000", 2024-09-13T14:28:28.102 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0018224249999999999 2024-09-13T14:28:28.102 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.102 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.102 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:28.102 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:27.372988+0000", 2024-09-13T14:28:28.103 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000160304 2024-09-13T14:28:28.103 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.103 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.103 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 4,7", 2024-09-13T14:28:28.103 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:27.373139+0000", 2024-09-13T14:28:28.103 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015102799999999999 2024-09-13T14:28:28.103 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.103 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.103 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:28.103 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:27.387330+0000", 2024-09-13T14:28:28.103 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014191272 2024-09-13T14:28:28.103 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.104 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.104 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:28.104 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:27.392282+0000", 2024-09-13T14:28:28.104 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0049514509999999999 2024-09-13T14:28:28.104 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.104 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.104 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:28.104 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:27.455136+0000", 2024-09-13T14:28:28.104 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.062854570999999998 2024-09-13T14:28:28.104 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.105 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.105 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:28.105 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:27.455307+0000", 2024-09-13T14:28:28.105 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000170963 2024-09-13T14:28:28.105 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:28.105 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:28.105 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:28.105 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:27.456912+0000", 2024-09-13T14:28:28.105 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0016044830000000001 2024-09-13T14:28:28.105 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:28.105 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:28.105 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:28.105 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:28.106 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:28.106 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:28:28.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:28.107 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:28.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:28.208 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:28:28.233 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:28.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:28.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:28:28.335 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:28.343 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:28:28.359 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:28.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:28.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:28.433 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is set 2024-09-13T14:28:28.461 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:28.497 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:28.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:28.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:28.599 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:28.626 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:28.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:28.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:28.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:28 smithi102 ceph-mon[115269]: pgmap v377: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.1 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 723 op/s; 25440/297945 objects degraded (8.538%); 1526/297945 objects misplaced (0.512%); 3.7 MiB/s, 59 objects/s recovering 2024-09-13T14:28:28.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:28 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3656276890' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:28:28.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:28 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:28:28.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:28 smithi102 ceph-mon[110183]: pgmap v377: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.1 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 723 op/s; 25440/297945 objects degraded (8.538%); 1526/297945 objects misplaced (0.512%); 3.7 MiB/s, 59 objects/s recovering 2024-09-13T14:28:28.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:28 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3656276890' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:28:28.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:28 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:28:28.728 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:28:28.742 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:28 smithi195 ceph-mon[104262]: pgmap v377: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.1 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 723 op/s; 25440/297945 objects degraded (8.538%); 1526/297945 objects misplaced (0.512%); 3.7 MiB/s, 59 objects/s recovering 2024-09-13T14:28:28.743 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:28 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3656276890' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:28:28.743 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:28 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:28:28.788 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:28.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:28.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:28:28.808 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:28:28.890 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:28.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:28.950 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:28.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:29.051 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:29.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:29.094 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:29.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:29.195 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:29.239 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:29.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:29.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:29.266 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T14:28:29.341 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:29.395 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:29.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:29.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:29.497 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:28:29.515 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:29 smithi102 ceph-mon[110183]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:28:29.515 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:29 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:28:29.515 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:29 smithi102 ceph-mon[110183]: osdmap e134: 8 total, 8 up, 6 in 2024-09-13T14:28:29.515 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:29 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 25440/297945 objects degraded (8.538%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:28:29.515 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:29 smithi102 ceph-mon[115269]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:28:29.515 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:29 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:28:29.516 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:29 smithi102 ceph-mon[115269]: osdmap e134: 8 total, 8 up, 6 in 2024-09-13T14:28:29.516 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:29 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 25440/297945 objects degraded (8.538%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:28:29.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:29.548 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:29.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:28:29.649 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:29.677 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:29.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:29.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:29.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:29 smithi195 ceph-mon[104262]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:28:29.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:29 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:28:29.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:29 smithi195 ceph-mon[104262]: osdmap e134: 8 total, 8 up, 6 in 2024-09-13T14:28:29.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:29 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 25440/297945 objects degraded (8.538%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:28:29.779 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:29.840 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:29.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:29.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:29.881 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:28:29.943 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:29.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:29.982 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:29.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:30.083 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:28:30.109 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:30.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:30.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:28:30.211 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:30.233 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:30.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:30.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:30.335 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:28:30.359 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:30.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:30.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:28:30.461 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:28:30.490 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:30.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:30.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:28:30.592 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:30.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:30.629 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:30.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:30.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:30 smithi102 ceph-mon[110183]: pgmap v379: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.1 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 724 op/s; 25440/297300 objects degraded (8.557%); 1526/297300 objects misplaced (0.513%); 3.7 MiB/s, 59 objects/s recovering 2024-09-13T14:28:30.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:30 smithi102 ceph-mon[115269]: pgmap v379: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.1 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 724 op/s; 25440/297300 objects degraded (8.557%); 1526/297300 objects misplaced (0.513%); 3.7 MiB/s, 59 objects/s recovering 2024-09-13T14:28:30.730 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:30.746 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:30 smithi195 ceph-mon[104262]: pgmap v379: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.1 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 724 op/s; 25440/297300 objects degraded (8.557%); 1526/297300 objects misplaced (0.513%); 3.7 MiB/s, 59 objects/s recovering 2024-09-13T14:28:30.803 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:30.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:30.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:30.905 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:30.933 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:30.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:30.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:31.035 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:28:31.045 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:28:31.045 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:28:31.068 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:28:31.068 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:28:31.087 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:28:31.087 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:28:31.089 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:31.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:31.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:28:31.113 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:28:31.113 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:28:31.143 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:28:31.143 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:28:31.167 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:28:31.167 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:28:31.190 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:28:31.191 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:28:31.191 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:31.220 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:28:31.220 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:28:31.233 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:31.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:31.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:31.246 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:28:31.335 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:28:31.371 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:31.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:31.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:28:31.474 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:31.512 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:31.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:31.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:31.614 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:31.637 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:31.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:31.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:31.645 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:31 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1481465245' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:28:31.645 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:31 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1481465245' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:28:31.740 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:28:31.746 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:31 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1481465245' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:28:31.796 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:31.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:31.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:28:31.901 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:31.927 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:31.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:31.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:32.029 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:32.072 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:32.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:32.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:32.174 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:28:32.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:32.270 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:32.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:28:32.271 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 7] out_osds: [1, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:28:32.271 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:28:32.271 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-09-13T14:28:32.271 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T14:28:32.271 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T14:28:32.272 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-09-13T14:28:32.372 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:32.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:32.430 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:32.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:32.445 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:28:32.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:32 smithi195 ceph-mon[104262]: pgmap v380: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.0 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 628 op/s; 25440/296658 objects degraded (8.576%); 1526/296658 objects misplaced (0.514%); 1.8 MiB/s, 29 objects/s recovering 2024-09-13T14:28:32.532 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:32.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:32.558 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:32.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:32.659 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:28:32.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:32 smithi102 ceph-mon[110183]: pgmap v380: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.0 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 628 op/s; 25440/296658 objects degraded (8.576%); 1526/296658 objects misplaced (0.514%); 1.8 MiB/s, 29 objects/s recovering 2024-09-13T14:28:32.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:32 smithi102 ceph-mon[115269]: pgmap v380: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.0 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 628 op/s; 25440/296658 objects degraded (8.576%); 1526/296658 objects misplaced (0.514%); 1.8 MiB/s, 29 objects/s recovering 2024-09-13T14:28:32.705 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:32.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:32.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:28:32.807 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:32.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:32.852 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:32.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:32.953 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:32.994 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:32.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:32.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:33.097 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:33.122 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:33.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:33.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:33.224 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:33.249 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:33.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:33.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:33.351 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:33.374 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:33.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:33.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:33.476 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:33.506 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:33.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:33.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:33.608 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:28:33.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:33.643 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:33.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:28:33.744 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:33.769 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:33.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:33.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:33.871 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:33.909 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:33.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:33.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:34.011 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:28:34.046 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:34.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:34.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:28:34.148 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:34.194 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:34.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:34.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:34.260 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:28:34.260 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:28:34.297 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:34.330 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:34.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:34.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:34.405 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:28:34.433 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:34.483 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:34.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:34.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:34.584 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:28:34.632 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:34.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:34.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:28:34.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:34 smithi102 ceph-mon[110183]: pgmap v381: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 696 op/s; 25440/287826 objects degraded (8.839%); 1247/287826 objects misplaced (0.433%); 3.6 MiB/s, 57 objects/s recovering 2024-09-13T14:28:34.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:34 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 25440/287826 objects degraded (8.839%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:28:34.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:34 smithi102 ceph-mon[115269]: pgmap v381: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 696 op/s; 25440/287826 objects degraded (8.839%); 1247/287826 objects misplaced (0.433%); 3.6 MiB/s, 57 objects/s recovering 2024-09-13T14:28:34.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:34 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 25440/287826 objects degraded (8.839%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:28:34.734 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:28:34.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:34 smithi195 ceph-mon[104262]: pgmap v381: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 696 op/s; 25440/287826 objects degraded (8.839%); 1247/287826 objects misplaced (0.433%); 3.6 MiB/s, 57 objects/s recovering 2024-09-13T14:28:34.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:34 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 25440/287826 objects degraded (8.839%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:28:34.776 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:34.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:34.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:28:34.878 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:28:34.906 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:34.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:34.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:28:35.008 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:28:35.041 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:35.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:35.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:28:35.142 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:35.168 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:35.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:35.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:35.259 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:28:35.269 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:35.279 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T14:28:35.292 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:35.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:35.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:35.394 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:35.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:35.422 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:35.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:35.523 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:35.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:35.547 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:35.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:35.648 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:35.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:35.695 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:35.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:35.796 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:35.844 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:35.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:35.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:35.946 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:28:35.989 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:35.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:35.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:28:36.091 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:28:36.114 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:36.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:36.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:28:36.139 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:28:36.216 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:36.262 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:36.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:36.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:36.336 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:28:36.365 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:36.393 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:36.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:36.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:36.461 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is unset 2024-09-13T14:28:36.496 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:36.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:36.535 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:36.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:36.604 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:36 smithi195 ceph-mon[104262]: pgmap v382: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.8 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 695 op/s; 25440/287166 objects degraded (8.859%); 1247/287166 objects misplaced (0.434%); 3.6 MiB/s, 57 objects/s recovering 2024-09-13T14:28:36.604 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:36 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:28:36.605 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:36 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:28:36.605 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:36 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/4182555359' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:28:36.605 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:36 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:28:36.605 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:36 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:28:36.634 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:36 smithi102 ceph-mon[110183]: pgmap v382: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.8 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 695 op/s; 25440/287166 objects degraded (8.859%); 1247/287166 objects misplaced (0.434%); 3.6 MiB/s, 57 objects/s recovering 2024-09-13T14:28:36.634 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:36 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:28:36.634 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:36 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:28:36.634 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:36 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/4182555359' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:28:36.635 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:36 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:28:36.635 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:36 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:28:36.635 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:36 smithi102 ceph-mon[115269]: pgmap v382: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.8 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 695 op/s; 25440/287166 objects degraded (8.859%); 1247/287166 objects misplaced (0.434%); 3.6 MiB/s, 57 objects/s recovering 2024-09-13T14:28:36.635 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:36 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:28:36.635 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:36 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:28:36.635 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:36 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/4182555359' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:28:36.635 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:36 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:28:36.635 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:36 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:28:36.636 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:28:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:28:36] "GET /metrics HTTP/1.1" 200 36545 "" "Prometheus/2.51.0" 2024-09-13T14:28:36.636 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:36.695 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:36.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:36.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:36.797 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:36.823 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:36.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:36.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:36.926 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:28:36.969 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:36.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:36.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:28:37.071 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:37.101 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:37.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:37.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:37.203 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:37.252 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:37.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:37.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:37.354 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:37.399 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:37.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:37.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:37.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:37 smithi195 ceph-mon[104262]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:28:37.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:37 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/4182555359' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:28:37.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:37 smithi195 ceph-mon[104262]: osdmap e135: 8 total, 8 up, 6 in 2024-09-13T14:28:37.502 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:37.540 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:37.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:37.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:37.642 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:28:37.650 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:37 smithi102 ceph-mon[110183]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:28:37.650 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:37 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/4182555359' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:28:37.650 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:37 smithi102 ceph-mon[110183]: osdmap e135: 8 total, 8 up, 6 in 2024-09-13T14:28:37.650 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:37 smithi102 ceph-mon[115269]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:28:37.650 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:37 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/4182555359' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:28:37.650 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:37 smithi102 ceph-mon[115269]: osdmap e135: 8 total, 8 up, 6 in 2024-09-13T14:28:37.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:37.681 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:37.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:28:37.782 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:37.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:37.817 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:37.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:37.918 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:37.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:37.967 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:37.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:38.068 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:38.091 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:38.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:38.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:38.193 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:38.215 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:38.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:38.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:38.317 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:38.361 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:38.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:38.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:38.463 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:38.491 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:38.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:38.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:38.593 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:38.598 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:38 smithi102 ceph-mon[110183]: pgmap v384: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.6 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 764 op/s; 25440/276993 objects degraded (9.184%); 917/276993 objects misplaced (0.331%); 4.3 MiB/s, 68 objects/s recovering 2024-09-13T14:28:38.598 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:38 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:28:38.598 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:38 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:28:38.598 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:38 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:28:38.598 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:38 smithi102 ceph-mon[115269]: pgmap v384: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.6 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 764 op/s; 25440/276993 objects degraded (9.184%); 917/276993 objects misplaced (0.331%); 4.3 MiB/s, 68 objects/s recovering 2024-09-13T14:28:38.598 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:38 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:28:38.599 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:38 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:28:38.599 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:38 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:28:38.628 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:38.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:38.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:38.643 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:38 smithi195 ceph-mon[104262]: pgmap v384: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.6 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 764 op/s; 25440/276993 objects degraded (9.184%); 917/276993 objects misplaced (0.331%); 4.3 MiB/s, 68 objects/s recovering 2024-09-13T14:28:38.644 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:38 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:28:38.644 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:38 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:28:38.644 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:38 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:28:38.730 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:38.756 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:38.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:38.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:38.859 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:28:38.876 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:28:38.877 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:28:38.885 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:38.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:38.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:28:38.896 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:28:38.896 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:28:38.919 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:28:38.919 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:28:38.940 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:28:38.940 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:28:38.964 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:28:38.964 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:28:38.984 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:28:38.985 INFO:teuthology.orchestra.run.smithi195.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-13T14:28:38.985 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:28:38.987 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:38.999 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:28:39.000 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:28:39.019 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:39.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:39.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:39.022 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:28:39.022 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:28:39.057 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:28:39.057 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:28:39.120 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:28:39.154 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:39.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:39.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:28:39.256 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:39.282 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:39.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:39.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:39.384 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:39.392 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:28:39.409 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:39.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:39.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:39.511 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:39.551 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:39.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:39.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:39.653 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:39.691 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:39.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:39.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:39.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:39 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 25440/276993 objects degraded (9.184%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:28:39.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:39 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/170776347' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:28:39.793 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:39.818 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:39.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:39.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:39.873 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:39 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 25440/276993 objects degraded (9.184%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:28:39.874 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:39 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/170776347' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:28:39.874 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:39 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 25440/276993 objects degraded (9.184%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:28:39.874 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:39 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/170776347' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:28:39.920 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:28:39.959 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:39.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:39.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:28:39.990 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:28:40.062 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:40.087 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:40.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:40.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:40.189 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:40.221 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:40.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:40.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:40.323 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:40.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:40.371 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:40.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:40.472 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:40.511 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:40.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:40.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:40.614 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:40.639 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:40.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:40.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:40.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:40 smithi102 ceph-mon[110183]: pgmap v385: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.6 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 701 op/s; 25440/276255 objects degraded (9.209%); 917/276255 objects misplaced (0.332%); 3.8 MiB/s, 60 objects/s recovering 2024-09-13T14:28:40.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:40 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:28:40.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:40 smithi102 ceph-mon[115269]: pgmap v385: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.6 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 701 op/s; 25440/276255 objects degraded (9.209%); 917/276255 objects misplaced (0.332%); 3.8 MiB/s, 60 objects/s recovering 2024-09-13T14:28:40.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:40 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:28:40.693 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:40 smithi195 ceph-mon[104262]: pgmap v385: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.6 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 701 op/s; 25440/276255 objects degraded (9.209%); 917/276255 objects misplaced (0.332%); 3.8 MiB/s, 60 objects/s recovering 2024-09-13T14:28:40.693 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:40 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:28:40.741 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:40.764 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:40.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:40.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:40.866 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:40.890 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:40.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:40.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:40.992 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:41.018 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:41.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:41.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:41.120 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:41.145 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:41.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:41.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:41.247 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:41.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:41.302 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:41.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:41.403 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:41.427 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:41.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:41.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:41.529 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:41.549 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:41.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:41.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:41.651 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:41.672 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:28:41.673 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-13T14:28:41.673 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-09-13T14:28:41.673 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-09-13T14:28:41.673 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:28:41.691 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:41.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:41.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:41.793 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:41.832 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:41.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:41.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:41.902 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:28:41.934 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:41.980 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:41.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:41.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:42.085 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:42.123 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:42.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:42.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:42.225 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:42.251 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:42.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:42.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:42.353 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:42.375 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:42.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:42.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:42.389 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T14:28:42.477 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:42.500 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:42.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:42.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:42.602 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:42.625 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:42.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:42.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:42.727 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:42.750 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:42 smithi195 ceph-mon[104262]: pgmap v386: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 702 op/s; 25440/275586 objects degraded (9.231%); 917/275586 objects misplaced (0.333%); 3.8 MiB/s, 60 objects/s recovering 2024-09-13T14:28:42.762 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:28:42.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:42.796 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:42.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:42.897 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:42.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:42 smithi102 ceph-mon[110183]: pgmap v386: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 702 op/s; 25440/275586 objects degraded (9.231%); 917/275586 objects misplaced (0.333%); 3.8 MiB/s, 60 objects/s recovering 2024-09-13T14:28:42.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:42 smithi102 ceph-mon[115269]: pgmap v386: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 702 op/s; 25440/275586 objects degraded (9.231%); 917/275586 objects misplaced (0.333%); 3.8 MiB/s, 60 objects/s recovering 2024-09-13T14:28:42.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:42.945 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:42.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:43.046 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:43.070 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:43.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:43.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:43.172 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:43.204 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:43.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:43.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:43.306 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:43.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:43.356 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:43.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:43.457 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:43.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:43.487 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:43.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:43.588 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:43.611 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:43.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:43.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:43.713 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:43.752 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:43.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:43.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:43.854 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:43.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:43.890 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:43.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:43.991 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:28:44.015 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:44.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:44.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:28:44.117 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:28:44.139 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:44.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:44.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:28:44.240 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:44.268 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:44.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:44.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:44.370 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:44.415 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:44.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:44.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:44.517 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:44.541 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:44.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:44.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:44.611 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:44 smithi195 ceph-mon[104262]: pgmap v387: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.4 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 721 op/s; 25440/266172 objects degraded (9.558%); 537/266172 objects misplaced (0.202%); 4.4 MiB/s, 70 objects/s recovering 2024-09-13T14:28:44.611 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:44 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 25440/266172 objects degraded (9.558%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:28:44.643 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:28:44.666 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:44.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:44.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:28:44.767 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:44.791 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:44.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:44.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:44.853 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:28:44.853 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-09-13T14:28:44.854 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-09-13T14:28:44.854 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-13T14:28:44.854 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.854 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:90451 5.2 5:42ca4429:::benchmark_data_smithi102_155989_object90450:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:28:44.854 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:25.550923+0000", 2024-09-13T14:28:44.854 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 79.221238733000007, 2024-09-13T14:28:44.854 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.064851735999999993, 2024-09-13T14:28:44.854 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:44.854 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:44.854 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:44.854 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:44.855 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:44.855 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 90451 2024-09-13T14:28:44.855 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.855 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:44.855 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.855 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:44.855 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.550923+0000", 2024-09-13T14:28:44.855 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:44.855 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.855 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.855 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:44.855 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.550923+0000", 2024-09-13T14:28:44.855 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:44.856 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.856 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.856 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:44.856 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.550922+0000", 2024-09-13T14:28:44.856 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:28:44.856 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.856 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.856 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:44.857 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.550963+0000", 2024-09-13T14:28:44.857 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1820000000000003e-05 2024-09-13T14:28:44.861 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.861 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.861 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:44.861 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.550977+0000", 2024-09-13T14:28:44.861 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3896999999999999e-05 2024-09-13T14:28:44.861 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.861 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.861 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:44.862 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.550986+0000", 2024-09-13T14:28:44.862 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2930000000000001e-06 2024-09-13T14:28:44.862 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.862 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.862 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:44.863 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.551199+0000", 2024-09-13T14:28:44.863 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00021318699999999999 2024-09-13T14:28:44.863 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.863 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.863 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:44.863 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.551662+0000", 2024-09-13T14:28:44.863 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00046375099999999999 2024-09-13T14:28:44.863 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.863 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.863 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:28:44.863 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.552302+0000", 2024-09-13T14:28:44.863 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00063985400000000003 2024-09-13T14:28:44.864 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.864 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.864 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:44.864 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.558678+0000", 2024-09-13T14:28:44.864 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0063757960000000004 2024-09-13T14:28:44.864 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.864 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.864 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:44.864 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.558722+0000", 2024-09-13T14:28:44.864 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3628000000000003e-05 2024-09-13T14:28:44.864 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.865 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.865 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:44.865 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.615639+0000", 2024-09-13T14:28:44.865 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.056917275000000003 2024-09-13T14:28:44.865 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.865 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.865 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:44.865 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.615737+0000", 2024-09-13T14:28:44.865 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.7643e-05 2024-09-13T14:28:44.865 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.865 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.865 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:44.866 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.615775+0000", 2024-09-13T14:28:44.866 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8481e-05 2024-09-13T14:28:44.866 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:44.866 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:44.866 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:44.866 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.866 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.866 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:90452 5.2 5:4ad05d41:::benchmark_data_smithi102_155989_object90451:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:28:44.866 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:25.551004+0000", 2024-09-13T14:28:44.866 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 79.221158404999997, 2024-09-13T14:28:44.866 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.064965966999999999, 2024-09-13T14:28:44.866 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:44.866 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:44.867 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:44.867 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:44.867 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:44.867 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 90452 2024-09-13T14:28:44.867 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.867 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:44.867 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.867 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:44.867 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.551004+0000", 2024-09-13T14:28:44.867 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:44.867 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.867 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.868 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:44.868 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.551004+0000", 2024-09-13T14:28:44.868 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:44.868 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.868 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.868 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:44.868 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.551003+0000", 2024-09-13T14:28:44.868 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:28:44.868 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.868 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.868 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:44.868 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.551036+0000", 2024-09-13T14:28:44.868 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.3015999999999997e-05 2024-09-13T14:28:44.869 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.869 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.869 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:44.869 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.551045+0000", 2024-09-13T14:28:44.869 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5590000000000006e-06 2024-09-13T14:28:44.869 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.869 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.869 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:44.869 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.551051+0000", 2024-09-13T14:28:44.869 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7620000000000001e-06 2024-09-13T14:28:44.869 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.869 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.870 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:44.870 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.555809+0000", 2024-09-13T14:28:44.870 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0047581209999999997 2024-09-13T14:28:44.870 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.870 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.870 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:44.870 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.556100+0000", 2024-09-13T14:28:44.870 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00029170500000000001 2024-09-13T14:28:44.870 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.870 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.870 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:28:44.870 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.556321+0000", 2024-09-13T14:28:44.870 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00022086100000000001 2024-09-13T14:28:44.870 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.871 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.871 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:44.871 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.558857+0000", 2024-09-13T14:28:44.871 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002536235 2024-09-13T14:28:44.871 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.871 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.871 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:44.871 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.559052+0000", 2024-09-13T14:28:44.871 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00019489400000000001 2024-09-13T14:28:44.871 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.871 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.871 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:44.871 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.615815+0000", 2024-09-13T14:28:44.872 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.056762827000000002 2024-09-13T14:28:44.872 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.872 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.872 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:44.872 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.615937+0000", 2024-09-13T14:28:44.872 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000121463 2024-09-13T14:28:44.872 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.872 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.872 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:44.872 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.615970+0000", 2024-09-13T14:28:44.872 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.3068000000000003e-05 2024-09-13T14:28:44.872 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:44.873 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:44.873 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:44.873 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.873 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.873 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:91318 5.2 5:466ae2df:::benchmark_data_smithi102_155989_object91317:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:28:44.873 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:26.827818+0000", 2024-09-13T14:28:44.873 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 77.944344611999995, 2024-09-13T14:28:44.873 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.070213572000000002, 2024-09-13T14:28:44.873 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:44.873 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:44.873 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:44.873 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:44.874 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:44.874 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 91318 2024-09-13T14:28:44.874 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.874 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:44.874 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.874 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:44.874 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:26.827818+0000", 2024-09-13T14:28:44.874 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:44.874 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.874 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.874 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:44.874 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:26.827818+0000", 2024-09-13T14:28:44.874 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:44.875 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.875 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.875 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:44.875 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:26.827815+0000", 2024-09-13T14:28:44.875 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-09-13T14:28:44.875 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.875 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.875 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:44.875 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:26.827851+0000", 2024-09-13T14:28:44.875 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5633999999999998e-05 2024-09-13T14:28:44.875 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.875 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.876 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:44.876 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:26.827871+0000", 2024-09-13T14:28:44.876 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9695e-05 2024-09-13T14:28:44.876 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.876 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.876 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:44.876 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:26.827879+0000", 2024-09-13T14:28:44.876 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2109999999999998e-06 2024-09-13T14:28:44.876 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.876 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.876 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:44.876 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:26.827935+0000", 2024-09-13T14:28:44.876 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.6637e-05 2024-09-13T14:28:44.877 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.877 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.877 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:44.877 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:26.828059+0000", 2024-09-13T14:28:44.877 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000123597 2024-09-13T14:28:44.877 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.877 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.877 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:28:44.877 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:26.828180+0000", 2024-09-13T14:28:44.877 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012057000000000001 2024-09-13T14:28:44.877 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.877 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.877 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:44.878 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:26.831282+0000", 2024-09-13T14:28:44.878 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0031025919999999999 2024-09-13T14:28:44.878 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.878 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.878 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:44.878 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:26.835513+0000", 2024-09-13T14:28:44.878 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0042309899999999996 2024-09-13T14:28:44.878 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.878 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.878 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:44.878 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:26.897950+0000", 2024-09-13T14:28:44.878 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.062437126000000003 2024-09-13T14:28:44.879 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.879 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.879 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:44.879 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:26.897993+0000", 2024-09-13T14:28:44.879 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2873000000000001e-05 2024-09-13T14:28:44.879 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.879 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.879 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:44.879 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:26.898031+0000", 2024-09-13T14:28:44.879 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7968e-05 2024-09-13T14:28:44.879 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:44.879 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:44.879 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:44.880 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.880 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.880 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:91674 5.2 5:445bdc5a:::benchmark_data_smithi102_155989_object91673:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:28:44.880 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:27.336850+0000", 2024-09-13T14:28:44.880 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 77.435312132999996, 2024-09-13T14:28:44.880 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.098296165000000005, 2024-09-13T14:28:44.880 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:44.880 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:44.880 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:44.880 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:44.880 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:44.880 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 91674 2024-09-13T14:28:44.880 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.881 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:44.881 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.881 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:44.881 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.336850+0000", 2024-09-13T14:28:44.881 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:44.881 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.881 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.881 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:44.881 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.336850+0000", 2024-09-13T14:28:44.881 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:44.881 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.881 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.881 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:44.882 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.336848+0000", 2024-09-13T14:28:44.882 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-09-13T14:28:44.882 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.882 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.882 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:44.882 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.336885+0000", 2024-09-13T14:28:44.882 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7684e-05 2024-09-13T14:28:44.882 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.882 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.882 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:44.882 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.336903+0000", 2024-09-13T14:28:44.882 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7453999999999999e-05 2024-09-13T14:28:44.882 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.883 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.883 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:44.883 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.336912+0000", 2024-09-13T14:28:44.883 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1619999999999997e-06 2024-09-13T14:28:44.883 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.883 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.883 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:44.883 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.336977+0000", 2024-09-13T14:28:44.883 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5368000000000002e-05 2024-09-13T14:28:44.883 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.883 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.884 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:44.884 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.337115+0000", 2024-09-13T14:28:44.884 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013742499999999999 2024-09-13T14:28:44.884 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.884 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.884 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:28:44.884 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.337279+0000", 2024-09-13T14:28:44.884 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016446000000000001 2024-09-13T14:28:44.884 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.884 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.884 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:44.884 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.339928+0000", 2024-09-13T14:28:44.884 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0026487149999999998 2024-09-13T14:28:44.884 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.885 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.885 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:44.885 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.341023+0000", 2024-09-13T14:28:44.885 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0010955419999999999 2024-09-13T14:28:44.885 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.885 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.885 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:44.885 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.434976+0000", 2024-09-13T14:28:44.885 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.093952941999999998 2024-09-13T14:28:44.885 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.885 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.885 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:44.885 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.435106+0000", 2024-09-13T14:28:44.886 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012945900000000001 2024-09-13T14:28:44.886 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.886 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.886 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:44.886 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.435146+0000", 2024-09-13T14:28:44.886 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0410000000000001e-05 2024-09-13T14:28:44.886 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:44.886 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:44.886 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:44.886 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.886 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.886 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:112891 5.2 5:447b5a5e:::benchmark_data_smithi102_155989_object112890:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:28:44.886 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:54.112549+0000", 2024-09-13T14:28:44.886 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 50.659613573999998, 2024-09-13T14:28:44.887 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.080517594999999997, 2024-09-13T14:28:44.887 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:44.887 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:44.887 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:44.887 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:44.887 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:44.887 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 112891 2024-09-13T14:28:44.887 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.887 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:44.887 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.887 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:44.887 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.112549+0000", 2024-09-13T14:28:44.887 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:44.887 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.887 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.888 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:44.888 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.112549+0000", 2024-09-13T14:28:44.888 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:44.888 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.888 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.888 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:44.888 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.112547+0000", 2024-09-13T14:28:44.888 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:28:44.888 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.888 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.888 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:44.888 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.112582+0000", 2024-09-13T14:28:44.889 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.523e-05 2024-09-13T14:28:44.889 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.889 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.889 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:44.889 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.112594+0000", 2024-09-13T14:28:44.889 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1946999999999999e-05 2024-09-13T14:28:44.889 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.889 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.889 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:44.889 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.112602+0000", 2024-09-13T14:28:44.889 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8550000000000002e-06 2024-09-13T14:28:44.889 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.889 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.890 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:44.890 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.112966+0000", 2024-09-13T14:28:44.890 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00036412400000000001 2024-09-13T14:28:44.890 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.890 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.890 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:44.890 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.113125+0000", 2024-09-13T14:28:44.890 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015927399999999999 2024-09-13T14:28:44.890 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.890 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.890 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:28:44.890 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.113369+0000", 2024-09-13T14:28:44.891 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000244066 2024-09-13T14:28:44.891 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.891 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.891 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:44.891 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.122780+0000", 2024-09-13T14:28:44.891 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0094102500000000002 2024-09-13T14:28:44.891 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.891 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.891 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:44.891 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.127254+0000", 2024-09-13T14:28:44.891 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0044746509999999996 2024-09-13T14:28:44.891 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.891 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.892 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:44.892 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.192967+0000", 2024-09-13T14:28:44.892 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.065712830999999999 2024-09-13T14:28:44.892 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.892 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.892 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:44.892 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.193024+0000", 2024-09-13T14:28:44.892 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.6928000000000001e-05 2024-09-13T14:28:44.892 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.892 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.892 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:44.892 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.193066+0000", 2024-09-13T14:28:44.892 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2138999999999998e-05 2024-09-13T14:28:44.893 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:44.893 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:44.893 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:44.893 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.893 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.893 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:112898 5.2 5:419311ca:::benchmark_data_smithi102_155989_object112897:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:28:44.893 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:54.115476+0000", 2024-09-13T14:28:44.893 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 50.656685678999999, 2024-09-13T14:28:44.893 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.086851202000000002, 2024-09-13T14:28:44.893 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:44.893 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:44.893 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:44.893 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:44.893 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:44.894 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 112898 2024-09-13T14:28:44.894 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.894 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:44.894 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.894 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:44.894 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.115476+0000", 2024-09-13T14:28:44.894 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:44.894 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.894 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.894 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:44.894 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.115476+0000", 2024-09-13T14:28:44.894 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:44.894 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.894 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.894 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:44.895 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.115474+0000", 2024-09-13T14:28:44.895 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:28:44.895 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.895 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.895 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:44.895 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.115530+0000", 2024-09-13T14:28:44.895 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5500000000000001e-05 2024-09-13T14:28:44.895 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.895 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.895 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:44.895 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.115549+0000", 2024-09-13T14:28:44.895 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8627999999999999e-05 2024-09-13T14:28:44.895 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.895 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.895 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:44.896 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.115558+0000", 2024-09-13T14:28:44.896 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.3360000000000001e-06 2024-09-13T14:28:44.896 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.896 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.896 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:44.896 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.118101+0000", 2024-09-13T14:28:44.896 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0025430069999999999 2024-09-13T14:28:44.896 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.896 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.896 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:44.896 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.118389+0000", 2024-09-13T14:28:44.896 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00028812 2024-09-13T14:28:44.896 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.897 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.897 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:28:44.897 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.118789+0000", 2024-09-13T14:28:44.897 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00040019699999999998 2024-09-13T14:28:44.897 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.897 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.897 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:44.897 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.122889+0000", 2024-09-13T14:28:44.897 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.004100228 2024-09-13T14:28:44.897 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.897 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.897 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:44.897 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.127383+0000", 2024-09-13T14:28:44.898 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0044937550000000003 2024-09-13T14:28:44.898 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.898 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.898 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:44.898 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.202199+0000", 2024-09-13T14:28:44.898 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.074815581000000006 2024-09-13T14:28:44.898 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.898 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.898 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:44.898 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.202288+0000", 2024-09-13T14:28:44.898 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.9388000000000005e-05 2024-09-13T14:28:44.898 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.898 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.898 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:44.899 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.202328+0000", 2024-09-13T14:28:44.899 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9468999999999998e-05 2024-09-13T14:28:44.899 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:44.899 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:44.899 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:44.899 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.899 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.899 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:113023 5.2 5:41246a78:::benchmark_data_smithi102_155989_object113022:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:28:44.899 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:54.349397+0000", 2024-09-13T14:28:44.899 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 50.422765492000003, 2024-09-13T14:28:44.899 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.079303431999999993, 2024-09-13T14:28:44.899 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:44.899 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:44.900 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:44.900 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:44.900 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:44.900 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113023 2024-09-13T14:28:44.900 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.900 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:44.900 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.900 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:44.900 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.349397+0000", 2024-09-13T14:28:44.900 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:44.900 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.900 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.900 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:44.900 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.349397+0000", 2024-09-13T14:28:44.900 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:44.901 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.901 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.901 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:44.901 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.349395+0000", 2024-09-13T14:28:44.901 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:28:44.901 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.901 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.901 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:44.901 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.349432+0000", 2024-09-13T14:28:44.901 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.718e-05 2024-09-13T14:28:44.901 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.901 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.901 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:44.901 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.349443+0000", 2024-09-13T14:28:44.902 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0729e-05 2024-09-13T14:28:44.902 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.902 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.902 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:44.902 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.349449+0000", 2024-09-13T14:28:44.902 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.263e-06 2024-09-13T14:28:44.902 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.902 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.902 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:44.902 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.353718+0000", 2024-09-13T14:28:44.903 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0042686989999999999 2024-09-13T14:28:44.903 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.903 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.903 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:44.903 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.353864+0000", 2024-09-13T14:28:44.903 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000146351 2024-09-13T14:28:44.903 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.903 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.903 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:28:44.903 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.354049+0000", 2024-09-13T14:28:44.903 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000184957 2024-09-13T14:28:44.903 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.904 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.904 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:44.904 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.358117+0000", 2024-09-13T14:28:44.904 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0040676239999999997 2024-09-13T14:28:44.904 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.904 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.904 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:44.904 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.358813+0000", 2024-09-13T14:28:44.904 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00069628699999999997 2024-09-13T14:28:44.904 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.904 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.904 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:44.904 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.428625+0000", 2024-09-13T14:28:44.905 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.069811467000000002 2024-09-13T14:28:44.905 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.905 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.905 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:44.905 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.428667+0000", 2024-09-13T14:28:44.905 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2179999999999999e-05 2024-09-13T14:28:44.905 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.905 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.905 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:44.905 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.428700+0000", 2024-09-13T14:28:44.905 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.3170999999999998e-05 2024-09-13T14:28:44.905 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:44.905 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:44.906 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:44.906 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.906 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.906 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:113037 5.2 5:461d0743:::benchmark_data_smithi102_155989_object113036:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:28:44.906 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:54.361936+0000", 2024-09-13T14:28:44.906 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 50.410225818000001, 2024-09-13T14:28:44.906 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.076134389999999996, 2024-09-13T14:28:44.906 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:44.906 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:44.906 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:44.906 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:44.906 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:44.906 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113037 2024-09-13T14:28:44.907 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.907 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:44.907 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.907 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:44.907 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.361936+0000", 2024-09-13T14:28:44.907 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:44.907 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.907 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.907 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:44.907 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.361936+0000", 2024-09-13T14:28:44.907 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:44.907 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.908 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.908 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:44.908 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.361934+0000", 2024-09-13T14:28:44.908 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-09-13T14:28:44.908 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.908 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.908 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:44.908 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.361971+0000", 2024-09-13T14:28:44.908 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6687000000000003e-05 2024-09-13T14:28:44.908 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.908 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.908 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:44.908 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.361985+0000", 2024-09-13T14:28:44.908 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4361e-05 2024-09-13T14:28:44.909 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.909 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.909 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:44.909 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.361993+0000", 2024-09-13T14:28:44.909 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4699999999999996e-06 2024-09-13T14:28:44.909 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.909 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.909 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:44.909 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.362044+0000", 2024-09-13T14:28:44.909 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1131000000000002e-05 2024-09-13T14:28:44.909 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.909 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.909 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:44.909 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.362166+0000", 2024-09-13T14:28:44.910 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012267499999999999 2024-09-13T14:28:44.910 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.910 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.910 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:28:44.910 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.362298+0000", 2024-09-13T14:28:44.910 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013130100000000001 2024-09-13T14:28:44.910 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.910 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.910 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:44.910 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.366490+0000", 2024-09-13T14:28:44.910 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0041923739999999996 2024-09-13T14:28:44.910 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.910 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.911 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:44.911 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.368290+0000", 2024-09-13T14:28:44.911 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001800409 2024-09-13T14:28:44.911 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.911 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.911 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:44.911 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.437859+0000", 2024-09-13T14:28:44.911 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.069568481000000001 2024-09-13T14:28:44.911 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.911 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.911 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:44.911 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.437946+0000", 2024-09-13T14:28:44.912 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.6784000000000001e-05 2024-09-13T14:28:44.912 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.912 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.912 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:44.912 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.438071+0000", 2024-09-13T14:28:44.912 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012500200000000001 2024-09-13T14:28:44.912 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:44.912 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:44.912 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:44.913 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.913 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.913 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:113039 5.2 5:40939e83:::benchmark_data_smithi102_155989_object113038:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:28:44.913 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:54.370277+0000", 2024-09-13T14:28:44.913 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 50.401884854999999, 2024-09-13T14:28:44.913 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.068259345999999999, 2024-09-13T14:28:44.913 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:44.913 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:44.913 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:44.913 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:44.913 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:44.913 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113039 2024-09-13T14:28:44.913 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.913 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:44.914 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.914 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:44.914 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.370277+0000", 2024-09-13T14:28:44.914 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:44.914 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.914 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.914 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:44.914 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.370277+0000", 2024-09-13T14:28:44.914 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:44.914 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.914 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.914 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:44.914 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.370276+0000", 2024-09-13T14:28:44.914 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:28:44.915 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.915 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.915 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:44.915 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.370319+0000", 2024-09-13T14:28:44.915 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3170999999999997e-05 2024-09-13T14:28:44.915 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.915 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.915 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:44.915 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.370332+0000", 2024-09-13T14:28:44.915 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2994e-05 2024-09-13T14:28:44.915 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.915 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.915 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:44.916 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.370338+0000", 2024-09-13T14:28:44.916 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5199999999999997e-06 2024-09-13T14:28:44.916 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.916 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.916 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:44.916 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.370375+0000", 2024-09-13T14:28:44.916 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7472000000000003e-05 2024-09-13T14:28:44.916 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.916 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.916 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:44.916 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.370481+0000", 2024-09-13T14:28:44.916 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010548 2024-09-13T14:28:44.916 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.917 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.917 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:28:44.917 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.370596+0000", 2024-09-13T14:28:44.917 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011521799999999999 2024-09-13T14:28:44.917 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.917 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.917 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:44.917 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.374003+0000", 2024-09-13T14:28:44.917 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003407584 2024-09-13T14:28:44.917 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.917 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.917 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:44.917 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.374176+0000", 2024-09-13T14:28:44.917 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00017315899999999999 2024-09-13T14:28:44.918 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.918 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.918 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:44.918 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.438213+0000", 2024-09-13T14:28:44.918 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.064036478999999993 2024-09-13T14:28:44.918 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.918 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.918 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:44.918 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.438386+0000", 2024-09-13T14:28:44.918 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000172988 2024-09-13T14:28:44.918 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.918 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.918 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:44.918 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.438537+0000", 2024-09-13T14:28:44.919 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000150688 2024-09-13T14:28:44.919 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:44.919 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:44.919 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:44.919 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.919 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.919 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:113753 5.2 5:4a3b37a0:::benchmark_data_smithi102_155989_object113752:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:28:44.919 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:55.352637+0000", 2024-09-13T14:28:44.919 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 49.419524766999999, 2024-09-13T14:28:44.919 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.082186414999999999, 2024-09-13T14:28:44.919 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:44.919 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:44.920 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:44.920 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:44.920 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:44.920 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113753 2024-09-13T14:28:44.920 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.920 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:44.920 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.920 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:44.920 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.352637+0000", 2024-09-13T14:28:44.920 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:44.920 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.920 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.920 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:44.921 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.352637+0000", 2024-09-13T14:28:44.921 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:44.921 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.921 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.921 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:44.921 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.352636+0000", 2024-09-13T14:28:44.921 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:28:44.921 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.921 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.921 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:44.921 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.352729+0000", 2024-09-13T14:28:44.921 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.2750999999999993e-05 2024-09-13T14:28:44.921 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.922 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.922 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:44.922 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.352742+0000", 2024-09-13T14:28:44.922 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3874e-05 2024-09-13T14:28:44.922 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.922 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.922 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:44.922 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.352750+0000", 2024-09-13T14:28:44.922 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2509999999999997e-06 2024-09-13T14:28:44.922 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.922 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.922 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:44.922 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.352831+0000", 2024-09-13T14:28:44.923 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1213999999999999e-05 2024-09-13T14:28:44.923 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.923 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.923 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:44.923 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.352983+0000", 2024-09-13T14:28:44.923 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015199300000000001 2024-09-13T14:28:44.923 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.923 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.923 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:28:44.923 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.353127+0000", 2024-09-13T14:28:44.923 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014379099999999999 2024-09-13T14:28:44.923 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.924 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.924 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:44.924 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.358485+0000", 2024-09-13T14:28:44.924 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0053579409999999997 2024-09-13T14:28:44.924 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.924 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.924 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:44.924 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.362585+0000", 2024-09-13T14:28:44.924 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0041000840000000004 2024-09-13T14:28:44.924 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.924 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.924 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:44.924 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.434711+0000", 2024-09-13T14:28:44.925 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.072126032000000007 2024-09-13T14:28:44.925 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.925 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.925 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:44.925 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.434782+0000", 2024-09-13T14:28:44.925 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.0838000000000002e-05 2024-09-13T14:28:44.925 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.925 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.925 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:44.925 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.434824+0000", 2024-09-13T14:28:44.925 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2150000000000001e-05 2024-09-13T14:28:44.925 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:44.925 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:44.926 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:44.926 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.926 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.926 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:113758 5.2 5:45b8ba06:::benchmark_data_smithi102_155989_object113757:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:28:44.926 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:55.358413+0000", 2024-09-13T14:28:44.926 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 49.413749105999997, 2024-09-13T14:28:44.926 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.076541652000000002, 2024-09-13T14:28:44.926 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:44.926 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:44.926 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:44.926 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:44.926 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:44.926 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113758 2024-09-13T14:28:44.927 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.927 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:44.927 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.927 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:44.927 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.358413+0000", 2024-09-13T14:28:44.927 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:44.927 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.927 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.927 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:44.927 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.358413+0000", 2024-09-13T14:28:44.927 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:44.927 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.927 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.928 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:44.928 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.358411+0000", 2024-09-13T14:28:44.928 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:28:44.928 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.928 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.928 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:44.928 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.358448+0000", 2024-09-13T14:28:44.928 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6878000000000002e-05 2024-09-13T14:28:44.928 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.928 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.928 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:44.928 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.358526+0000", 2024-09-13T14:28:44.928 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7841000000000002e-05 2024-09-13T14:28:44.929 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.929 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.929 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:44.929 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.358536+0000", 2024-09-13T14:28:44.929 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4890000000000001e-06 2024-09-13T14:28:44.929 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.929 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.929 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:44.929 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.358600+0000", 2024-09-13T14:28:44.929 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4548000000000006e-05 2024-09-13T14:28:44.929 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.929 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.930 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:44.930 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.358947+0000", 2024-09-13T14:28:44.930 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00034709999999999998 2024-09-13T14:28:44.930 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.930 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.930 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:28:44.930 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.359208+0000", 2024-09-13T14:28:44.930 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00026032699999999998 2024-09-13T14:28:44.930 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.930 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.930 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:44.930 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.362687+0000", 2024-09-13T14:28:44.930 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0034797500000000002 2024-09-13T14:28:44.931 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.931 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.931 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:44.931 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.362738+0000", 2024-09-13T14:28:44.931 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0664000000000001e-05 2024-09-13T14:28:44.931 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.931 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.931 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:44.931 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.434855+0000", 2024-09-13T14:28:44.931 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.072117280000000006 2024-09-13T14:28:44.931 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.931 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.931 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:44.932 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.434927+0000", 2024-09-13T14:28:44.932 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1328999999999994e-05 2024-09-13T14:28:44.932 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.932 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.932 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:44.932 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.434955+0000", 2024-09-13T14:28:44.932 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8130999999999999e-05 2024-09-13T14:28:44.932 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:44.932 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:44.932 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:44.932 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.932 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.933 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:116701 5.2 5:4ad2b1b2:::benchmark_data_smithi102_155989_object116700:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:28:44.933 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:59.402062+0000", 2024-09-13T14:28:44.933 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 45.370100088999997, 2024-09-13T14:28:44.933 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.096687686999999994, 2024-09-13T14:28:44.933 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:44.933 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:44.933 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:44.933 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:44.933 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:44.933 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 116701 2024-09-13T14:28:44.933 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.933 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:44.934 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.934 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:44.934 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.402062+0000", 2024-09-13T14:28:44.934 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:44.934 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.934 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.934 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:44.934 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.402062+0000", 2024-09-13T14:28:44.934 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:44.934 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.934 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.934 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:44.935 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.402060+0000", 2024-09-13T14:28:44.935 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:28:44.935 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.935 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.935 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:44.935 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.402109+0000", 2024-09-13T14:28:44.935 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8319000000000003e-05 2024-09-13T14:28:44.935 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.935 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.935 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:44.935 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.402120+0000", 2024-09-13T14:28:44.935 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1993e-05 2024-09-13T14:28:44.935 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.936 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.936 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:44.936 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.402129+0000", 2024-09-13T14:28:44.936 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0639999999999994e-06 2024-09-13T14:28:44.936 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.936 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.936 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:44.936 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.402196+0000", 2024-09-13T14:28:44.936 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7712000000000005e-05 2024-09-13T14:28:44.936 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.936 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.936 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:44.937 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.402312+0000", 2024-09-13T14:28:44.937 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000115351 2024-09-13T14:28:44.937 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.937 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.937 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:28:44.937 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.402444+0000", 2024-09-13T14:28:44.937 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000132781 2024-09-13T14:28:44.937 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.937 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.937 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:44.937 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.406257+0000", 2024-09-13T14:28:44.937 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0038126079999999999 2024-09-13T14:28:44.938 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.938 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.938 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:44.938 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.414144+0000", 2024-09-13T14:28:44.938 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0078872130000000006 2024-09-13T14:28:44.938 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.938 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.938 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:44.938 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.498654+0000", 2024-09-13T14:28:44.938 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.084509976000000001 2024-09-13T14:28:44.938 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.938 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.939 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:44.939 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.498712+0000", 2024-09-13T14:28:44.939 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8037e-05 2024-09-13T14:28:44.939 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.939 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.939 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:44.939 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.498750+0000", 2024-09-13T14:28:44.939 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7497999999999999e-05 2024-09-13T14:28:44.939 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:44.939 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:44.939 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:44.939 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.940 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.940 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:116703 5.2 5:4688eb25:::benchmark_data_smithi102_155989_object116702:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:28:44.940 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:59.406689+0000", 2024-09-13T14:28:44.940 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 45.365473412, 2024-09-13T14:28:44.940 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.092431676000000004, 2024-09-13T14:28:44.940 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:44.940 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:44.940 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:44.940 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:44.940 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:44.940 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 116703 2024-09-13T14:28:44.941 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.941 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:44.941 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.941 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:44.941 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.406689+0000", 2024-09-13T14:28:44.941 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:44.941 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.941 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.941 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:44.941 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.406689+0000", 2024-09-13T14:28:44.941 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:44.941 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.941 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.942 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:44.942 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.406687+0000", 2024-09-13T14:28:44.942 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:28:44.942 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.942 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.942 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:44.942 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.406730+0000", 2024-09-13T14:28:44.942 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2775000000000003e-05 2024-09-13T14:28:44.942 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.942 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.942 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:44.942 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.406746+0000", 2024-09-13T14:28:44.942 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5985999999999999e-05 2024-09-13T14:28:44.942 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.943 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.943 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:44.943 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.406754+0000", 2024-09-13T14:28:44.943 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9680000000000006e-06 2024-09-13T14:28:44.943 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.943 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.943 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:44.943 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.406794+0000", 2024-09-13T14:28:44.943 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.036e-05 2024-09-13T14:28:44.943 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.943 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.943 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:44.943 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.406924+0000", 2024-09-13T14:28:44.944 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013029900000000001 2024-09-13T14:28:44.944 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.944 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.944 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:28:44.944 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.407062+0000", 2024-09-13T14:28:44.944 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013760399999999999 2024-09-13T14:28:44.944 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.944 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.944 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:44.944 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.410866+0000", 2024-09-13T14:28:44.944 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0038043339999999999 2024-09-13T14:28:44.944 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.945 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.945 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:44.945 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.414960+0000", 2024-09-13T14:28:44.945 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0040940200000000003 2024-09-13T14:28:44.945 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.945 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.945 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:44.945 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.499038+0000", 2024-09-13T14:28:44.945 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.084077951999999997 2024-09-13T14:28:44.945 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.945 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.945 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:44.945 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.499085+0000", 2024-09-13T14:28:44.945 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6841000000000001e-05 2024-09-13T14:28:44.946 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.946 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.946 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:44.946 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.499120+0000", 2024-09-13T14:28:44.946 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5348000000000001e-05 2024-09-13T14:28:44.946 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:44.946 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:44.946 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:44.946 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.946 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.946 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:117467 5.2 5:4a8dc47a:::benchmark_data_smithi102_155989_object117466:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:28:44.946 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:28:00.495953+0000", 2024-09-13T14:28:44.947 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 44.276209420999997, 2024-09-13T14:28:44.947 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.092397426000000005, 2024-09-13T14:28:44.947 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:44.947 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:44.947 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:44.947 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:44.947 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:44.947 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 117467 2024-09-13T14:28:44.947 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.947 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:44.947 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.947 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:44.947 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.495953+0000", 2024-09-13T14:28:44.948 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:44.948 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.948 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.948 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:44.948 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.495953+0000", 2024-09-13T14:28:44.948 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:44.948 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.948 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.948 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:44.948 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.495950+0000", 2024-09-13T14:28:44.948 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-09-13T14:28:44.948 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.948 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.948 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:44.949 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.495993+0000", 2024-09-13T14:28:44.949 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3019000000000003e-05 2024-09-13T14:28:44.949 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.949 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.949 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:44.949 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.496013+0000", 2024-09-13T14:28:44.949 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9602999999999998e-05 2024-09-13T14:28:44.949 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.949 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.949 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:44.949 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.496023+0000", 2024-09-13T14:28:44.949 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4390000000000006e-06 2024-09-13T14:28:44.949 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.950 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.950 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:44.950 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.498242+0000", 2024-09-13T14:28:44.950 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0022199720000000002 2024-09-13T14:28:44.950 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.950 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.950 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:44.950 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.498572+0000", 2024-09-13T14:28:44.950 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00032968500000000003 2024-09-13T14:28:44.950 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.950 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.950 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:28:44.950 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.498936+0000", 2024-09-13T14:28:44.950 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000364185 2024-09-13T14:28:44.950 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.951 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.951 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:44.951 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.509682+0000", 2024-09-13T14:28:44.951 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010745879 2024-09-13T14:28:44.951 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.951 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.951 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:44.951 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.510286+0000", 2024-09-13T14:28:44.951 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00060406800000000003 2024-09-13T14:28:44.951 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.951 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.951 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:44.951 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.588289+0000", 2024-09-13T14:28:44.951 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.078002929999999998 2024-09-13T14:28:44.952 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.952 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.952 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:44.952 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.588324+0000", 2024-09-13T14:28:44.952 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4928999999999997e-05 2024-09-13T14:28:44.952 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.952 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.952 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:44.952 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.588350+0000", 2024-09-13T14:28:44.952 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5978000000000001e-05 2024-09-13T14:28:44.952 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:44.952 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:44.952 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:44.952 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.953 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.953 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:117471 5.2 5:419b8c44:::benchmark_data_smithi102_155989_object117470:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:28:44.953 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:28:00.498101+0000", 2024-09-13T14:28:44.953 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 44.274060861000002, 2024-09-13T14:28:44.953 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.090334384000000004, 2024-09-13T14:28:44.953 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:44.953 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:44.953 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:44.953 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:44.953 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:44.953 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 117471 2024-09-13T14:28:44.953 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.953 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:44.954 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.954 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:44.954 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.498101+0000", 2024-09-13T14:28:44.954 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:44.954 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.954 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.954 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:44.954 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.498101+0000", 2024-09-13T14:28:44.954 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:44.954 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.954 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.954 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:44.954 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.498100+0000", 2024-09-13T14:28:44.955 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:28:44.955 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.955 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.955 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:44.955 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.498129+0000", 2024-09-13T14:28:44.955 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9184e-05 2024-09-13T14:28:44.955 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.955 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.955 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:44.955 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.498145+0000", 2024-09-13T14:28:44.955 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6005e-05 2024-09-13T14:28:44.955 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.955 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.956 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:44.956 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.498152+0000", 2024-09-13T14:28:44.956 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2799999999999998e-06 2024-09-13T14:28:44.956 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.956 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.956 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:44.956 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.502688+0000", 2024-09-13T14:28:44.956 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0045362429999999997 2024-09-13T14:28:44.956 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.956 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.956 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:44.956 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.502829+0000", 2024-09-13T14:28:44.956 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000140564 2024-09-13T14:28:44.957 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.957 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.957 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:28:44.957 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.502991+0000", 2024-09-13T14:28:44.957 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000161519 2024-09-13T14:28:44.957 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.957 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.957 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:44.957 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.510054+0000", 2024-09-13T14:28:44.957 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0070637959999999998 2024-09-13T14:28:44.957 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.957 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.957 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:44.958 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.510438+0000", 2024-09-13T14:28:44.958 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00038332399999999999 2024-09-13T14:28:44.958 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.958 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.958 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:44.958 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.588381+0000", 2024-09-13T14:28:44.958 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.077943296999999995 2024-09-13T14:28:44.958 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.958 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.958 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:44.958 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.588413+0000", 2024-09-13T14:28:44.958 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2011999999999998e-05 2024-09-13T14:28:44.959 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.959 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.959 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:44.959 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.588436+0000", 2024-09-13T14:28:44.959 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2699000000000001e-05 2024-09-13T14:28:44.959 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:44.959 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:44.959 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:44.959 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.959 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.959 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:127406 5.2 5:401974e1:::benchmark_data_smithi102_155989_object9488:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-13T14:28:44.959 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:28:13.570743+0000", 2024-09-13T14:28:44.959 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 31.201418746000002, 2024-09-13T14:28:44.960 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.069567083000000002, 2024-09-13T14:28:44.960 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:44.960 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:44.960 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:44.960 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:44.960 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:44.960 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 127406 2024-09-13T14:28:44.960 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.960 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:44.960 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.960 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:44.960 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.570743+0000", 2024-09-13T14:28:44.961 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:44.961 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.961 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.961 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:44.961 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.570743+0000", 2024-09-13T14:28:44.961 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:44.961 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.961 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.961 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:44.961 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.570742+0000", 2024-09-13T14:28:44.961 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:28:44.961 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.961 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.962 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:44.962 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.570748+0000", 2024-09-13T14:28:44.962 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0580000000000003e-06 2024-09-13T14:28:44.962 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.962 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.962 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:44.962 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.570760+0000", 2024-09-13T14:28:44.962 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1389000000000001e-05 2024-09-13T14:28:44.962 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.962 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.962 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:44.963 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.570766+0000", 2024-09-13T14:28:44.963 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3230000000000003e-06 2024-09-13T14:28:44.963 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.963 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.963 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:44.963 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.570829+0000", 2024-09-13T14:28:44.963 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3225000000000005e-05 2024-09-13T14:28:44.963 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.963 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.964 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:44.964 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.570936+0000", 2024-09-13T14:28:44.964 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010673 2024-09-13T14:28:44.964 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.964 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.964 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:28:44.964 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.571033+0000", 2024-09-13T14:28:44.964 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.7323000000000003e-05 2024-09-13T14:28:44.964 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.964 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.964 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:44.964 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.575126+0000", 2024-09-13T14:28:44.964 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0040930280000000003 2024-09-13T14:28:44.965 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.965 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.965 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:44.965 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.582607+0000", 2024-09-13T14:28:44.965 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0074805100000000001 2024-09-13T14:28:44.965 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.965 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.965 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:44.965 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.640211+0000", 2024-09-13T14:28:44.965 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.057604732999999998 2024-09-13T14:28:44.965 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.965 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.965 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:44.966 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.640260+0000", 2024-09-13T14:28:44.966 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8822000000000001e-05 2024-09-13T14:28:44.966 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.966 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.966 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:44.966 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.640310+0000", 2024-09-13T14:28:44.966 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0235000000000003e-05 2024-09-13T14:28:44.966 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:44.966 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:44.966 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:44.966 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.966 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.967 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:127410 5.2 5:408e4dea:::benchmark_data_smithi102_155989_object9492:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-13T14:28:44.967 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:28:13.576204+0000", 2024-09-13T14:28:44.967 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 31.195958227999999, 2024-09-13T14:28:44.967 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.064496429999999993, 2024-09-13T14:28:44.967 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:44.967 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:44.967 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:44.967 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:44.967 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:44.967 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 127410 2024-09-13T14:28:44.967 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.967 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:44.967 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.968 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:44.968 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.576204+0000", 2024-09-13T14:28:44.968 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:44.968 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.968 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.968 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:44.968 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.576204+0000", 2024-09-13T14:28:44.968 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:44.968 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.968 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.968 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:44.968 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.576202+0000", 2024-09-13T14:28:44.968 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:28:44.968 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.969 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.969 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:44.969 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.576209+0000", 2024-09-13T14:28:44.969 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8000000000000001e-06 2024-09-13T14:28:44.969 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.969 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.969 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:44.969 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.576224+0000", 2024-09-13T14:28:44.969 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5003e-05 2024-09-13T14:28:44.969 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.969 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.969 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:44.969 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.576231+0000", 2024-09-13T14:28:44.970 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9689999999999997e-06 2024-09-13T14:28:44.970 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.970 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.970 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:44.970 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.576276+0000", 2024-09-13T14:28:44.970 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4848999999999997e-05 2024-09-13T14:28:44.970 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.970 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.970 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:44.970 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.576485+0000", 2024-09-13T14:28:44.970 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000209521 2024-09-13T14:28:44.970 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.971 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.971 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:28:44.971 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.576619+0000", 2024-09-13T14:28:44.971 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013378500000000001 2024-09-13T14:28:44.971 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.971 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.971 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:44.971 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.579687+0000", 2024-09-13T14:28:44.971 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0030682270000000002 2024-09-13T14:28:44.971 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.971 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.971 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:44.971 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.582798+0000", 2024-09-13T14:28:44.972 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0031111110000000002 2024-09-13T14:28:44.972 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.972 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.972 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:44.972 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.640591+0000", 2024-09-13T14:28:44.972 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.057792625 2024-09-13T14:28:44.972 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.972 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.972 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:44.972 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.640663+0000", 2024-09-13T14:28:44.972 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1896000000000004e-05 2024-09-13T14:28:44.972 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.973 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.973 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:44.973 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.640700+0000", 2024-09-13T14:28:44.973 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7518999999999998e-05 2024-09-13T14:28:44.973 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:44.973 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:44.973 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:44.973 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.973 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.973 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:127604 5.2 5:4b8edd25:::benchmark_data_smithi102_155989_object9686:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-13T14:28:44.973 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:28:13.850196+0000", 2024-09-13T14:28:44.973 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 30.921965839999999, 2024-09-13T14:28:44.973 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.085607760000000005, 2024-09-13T14:28:44.973 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:44.974 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:44.974 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:44.974 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:44.974 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:44.974 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 127604 2024-09-13T14:28:44.974 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.974 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:44.974 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.974 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:44.974 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.850196+0000", 2024-09-13T14:28:44.974 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:44.974 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.974 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.974 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:44.975 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.850196+0000", 2024-09-13T14:28:44.975 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:44.975 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.975 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.975 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:44.975 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.850194+0000", 2024-09-13T14:28:44.975 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-09-13T14:28:44.975 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.975 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.975 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:44.975 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.850202+0000", 2024-09-13T14:28:44.975 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7600000000000002e-06 2024-09-13T14:28:44.975 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.976 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.976 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:44.976 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.850219+0000", 2024-09-13T14:28:44.976 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6752000000000002e-05 2024-09-13T14:28:44.976 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.976 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.976 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:44.976 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.850227+0000", 2024-09-13T14:28:44.976 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0369999999999995e-06 2024-09-13T14:28:44.976 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.976 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.976 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:44.976 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.850289+0000", 2024-09-13T14:28:44.977 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2848000000000005e-05 2024-09-13T14:28:44.977 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.977 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.977 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:44.977 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.850589+0000", 2024-09-13T14:28:44.977 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00029958200000000001 2024-09-13T14:28:44.977 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.977 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.977 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:28:44.977 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.850929+0000", 2024-09-13T14:28:44.977 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00034012799999999998 2024-09-13T14:28:44.977 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.977 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.978 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:44.978 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.860869+0000", 2024-09-13T14:28:44.978 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0099398339999999998 2024-09-13T14:28:44.978 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.978 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.978 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:44.978 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.861006+0000", 2024-09-13T14:28:44.978 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000136592 2024-09-13T14:28:44.978 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.978 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.978 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:44.978 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.935335+0000", 2024-09-13T14:28:44.978 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.074329002000000005 2024-09-13T14:28:44.979 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.979 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.979 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:44.979 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.935672+0000", 2024-09-13T14:28:44.979 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000337141 2024-09-13T14:28:44.979 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.979 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.979 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:44.979 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.935804+0000", 2024-09-13T14:28:44.979 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013234100000000001 2024-09-13T14:28:44.979 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:44.979 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:44.979 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:44.980 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.980 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.980 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:127607 5.2 5:4a72a91f:::benchmark_data_smithi102_155989_object9689:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-13T14:28:44.980 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:28:13.856547+0000", 2024-09-13T14:28:44.980 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 30.915614905999998, 2024-09-13T14:28:44.980 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.079705397999999997, 2024-09-13T14:28:44.980 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:44.980 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:44.980 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:44.980 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:44.980 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:44.980 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 127607 2024-09-13T14:28:44.980 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.980 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:44.981 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.981 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:44.981 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.856547+0000", 2024-09-13T14:28:44.981 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:44.981 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.981 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.981 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:44.981 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.856547+0000", 2024-09-13T14:28:44.981 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:44.981 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.981 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.981 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:44.981 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.856545+0000", 2024-09-13T14:28:44.982 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-09-13T14:28:44.982 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.982 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.982 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:44.982 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.856557+0000", 2024-09-13T14:28:44.982 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2177999999999999e-05 2024-09-13T14:28:44.982 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.982 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.982 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:44.982 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.856575+0000", 2024-09-13T14:28:44.982 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8236999999999998e-05 2024-09-13T14:28:44.982 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.982 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.982 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:44.983 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.856583+0000", 2024-09-13T14:28:44.983 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5109999999999997e-06 2024-09-13T14:28:44.983 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.983 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.983 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:44.983 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.856675+0000", 2024-09-13T14:28:44.983 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.2089999999999994e-05 2024-09-13T14:28:44.983 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.983 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.983 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:44.983 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.856974+0000", 2024-09-13T14:28:44.983 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00029847400000000002 2024-09-13T14:28:44.983 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.984 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.984 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:28:44.984 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.857308+0000", 2024-09-13T14:28:44.984 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00033488799999999998 2024-09-13T14:28:44.984 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.984 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.984 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:44.984 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.861095+0000", 2024-09-13T14:28:44.984 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003786106 2024-09-13T14:28:44.984 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.984 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.984 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:44.984 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.861255+0000", 2024-09-13T14:28:44.984 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016012200000000001 2024-09-13T14:28:44.985 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.985 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.985 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:44.985 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.935963+0000", 2024-09-13T14:28:44.985 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.074708732 2024-09-13T14:28:44.985 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.985 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.985 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:44.985 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.936117+0000", 2024-09-13T14:28:44.985 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015398000000000001 2024-09-13T14:28:44.985 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.985 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.985 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:44.986 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.936253+0000", 2024-09-13T14:28:44.986 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013526799999999999 2024-09-13T14:28:44.986 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:44.986 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:44.986 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:44.986 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.986 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.986 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:148497 5.2 5:44c2546f:::benchmark_data_smithi102_155989_object30579:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T14:28:44.986 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:28:43.190575+0000", 2024-09-13T14:28:44.986 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 1.581587224, 2024-09-13T14:28:44.986 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.076364062999999996, 2024-09-13T14:28:44.986 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:44.986 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:28:44.986 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:44.987 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:44.987 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:44.987 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 148497 2024-09-13T14:28:44.987 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.987 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:44.987 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.987 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:44.987 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.190575+0000", 2024-09-13T14:28:44.987 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:44.987 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.987 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.987 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:44.987 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.190575+0000", 2024-09-13T14:28:44.987 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:44.988 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.988 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.988 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:44.988 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.190574+0000", 2024-09-13T14:28:44.988 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:28:44.988 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.988 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.988 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:44.988 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.190581+0000", 2024-09-13T14:28:44.988 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.0419999999999997e-06 2024-09-13T14:28:44.988 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.988 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.988 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:44.989 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.190594+0000", 2024-09-13T14:28:44.989 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3421000000000001e-05 2024-09-13T14:28:44.989 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.989 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.989 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:44.989 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.190600+0000", 2024-09-13T14:28:44.989 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3180000000000003e-06 2024-09-13T14:28:44.989 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.989 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.989 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:44.989 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.190638+0000", 2024-09-13T14:28:44.989 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7667999999999999e-05 2024-09-13T14:28:44.990 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.990 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.990 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:44.990 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.190715+0000", 2024-09-13T14:28:44.990 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7311000000000006e-05 2024-09-13T14:28:44.990 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.990 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.990 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:28:44.990 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.190784+0000", 2024-09-13T14:28:44.990 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9082e-05 2024-09-13T14:28:44.990 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.990 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.990 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:44.991 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.195660+0000", 2024-09-13T14:28:44.991 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0048754749999999998 2024-09-13T14:28:44.991 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.991 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.991 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:44.991 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.196096+0000", 2024-09-13T14:28:44.991 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00043614599999999999 2024-09-13T14:28:44.991 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.991 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.991 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:28:44.991 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.266845+0000", 2024-09-13T14:28:44.991 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.070748855999999999 2024-09-13T14:28:44.991 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.992 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.992 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:28:44.992 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.266896+0000", 2024-09-13T14:28:44.992 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1106000000000002e-05 2024-09-13T14:28:44.992 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:44.992 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:44.992 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:28:44.992 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.266939+0000", 2024-09-13T14:28:44.992 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2933999999999999e-05 2024-09-13T14:28:44.992 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:44.992 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:44.992 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:44.992 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:44.993 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:44.993 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:28:44.993 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:44.994 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:44 smithi102 ceph-mon[110183]: pgmap v387: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.4 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 721 op/s; 25440/266172 objects degraded (9.558%); 537/266172 objects misplaced (0.202%); 4.4 MiB/s, 70 objects/s recovering 2024-09-13T14:28:44.994 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:44 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 25440/266172 objects degraded (9.558%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:28:44.994 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:44 smithi102 ceph-mon[115269]: pgmap v387: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.4 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 721 op/s; 25440/266172 objects degraded (9.558%); 537/266172 objects misplaced (0.202%); 4.4 MiB/s, 70 objects/s recovering 2024-09-13T14:28:44.995 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:44 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 25440/266172 objects degraded (9.558%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:28:45.017 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:45.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:45.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:45.119 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:45.161 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:45.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:45.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:45.263 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:45.307 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:45.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:45.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:45.410 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:45.434 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:45.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:45.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:45.535 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:45.561 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:45.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:45.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:45.663 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:45.705 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:28:45.706 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:45.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:45.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:45.743 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T14:28:45.808 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:28:45.832 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:45.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:45.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:28:45.934 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:45.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:45.993 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:45.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:46.094 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:46.153 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:46.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:46.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:46.193 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:28:46.255 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:28:46.285 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:46.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:46.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:28:46.316 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 7] out_osds: [1, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:28:46.316 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-13T14:28:46.387 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:46.435 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:46.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:46.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:46.537 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:46.565 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:46.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:46.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:46.667 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:46.694 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:46.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:46.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:46.698 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:46 smithi195 ceph-mon[104262]: pgmap v388: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.4 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 721 op/s; 25440/265512 objects degraded (9.581%); 537/265512 objects misplaced (0.202%); 4.4 MiB/s, 70 objects/s recovering 2024-09-13T14:28:46.788 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:46 smithi102 ceph-mon[110183]: pgmap v388: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.4 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 721 op/s; 25440/265512 objects degraded (9.581%); 537/265512 objects misplaced (0.202%); 4.4 MiB/s, 70 objects/s recovering 2024-09-13T14:28:46.788 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:46 smithi102 ceph-mon[115269]: pgmap v388: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.4 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 721 op/s; 25440/265512 objects degraded (9.581%); 537/265512 objects misplaced (0.202%); 4.4 MiB/s, 70 objects/s recovering 2024-09-13T14:28:46.788 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:28:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:28:46] "GET /metrics HTTP/1.1" 200 36541 "" "Prometheus/2.51.0" 2024-09-13T14:28:46.797 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:46.821 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:46.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:46.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:46.923 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:28:46.950 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:46.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:46.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:28:47.052 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:47.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:47.085 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:47.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:47.186 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:47.226 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:47.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:47.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:47.241 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:28:47.327 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:28:47.353 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:47.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:47.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:28:47.454 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:47.479 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:47.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:47.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:47.483 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:47 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1122981046' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:28:47.502 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is unset 2024-09-13T14:28:47.581 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:47.604 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:47.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:47.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:47.643 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:47 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1122981046' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:28:47.643 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:47 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:28:47.643 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:47 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1122981046' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:28:47.643 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:47 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:28:47.706 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:47.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:47 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:28:47.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:47.749 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:47.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:47.850 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:28:47.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:47.894 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:47.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:28:47.995 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:48.046 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:48.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:48.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:48.148 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:48.173 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:48.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:48.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:48.275 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:48.299 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:48.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:48.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:48.400 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:28:48.454 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:48.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:48.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:28:48.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:48 smithi195 ceph-mon[104262]: pgmap v389: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 864 op/s; 25440/255711 objects degraded (9.949%); 161/255711 objects misplaced (0.063%); 4.4 MiB/s, 69 objects/s recovering 2024-09-13T14:28:48.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:48 smithi195 ceph-mon[104262]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:28:48.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:48 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:28:48.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:48 smithi195 ceph-mon[104262]: osdmap e136: 8 total, 8 up, 6 in 2024-09-13T14:28:48.558 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:28:48.582 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:48.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:48.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:28:48.684 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:48.702 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:48 smithi102 ceph-mon[110183]: pgmap v389: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 864 op/s; 25440/255711 objects degraded (9.949%); 161/255711 objects misplaced (0.063%); 4.4 MiB/s, 69 objects/s recovering 2024-09-13T14:28:48.702 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:48 smithi102 ceph-mon[110183]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:28:48.702 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:48 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:28:48.702 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:48 smithi102 ceph-mon[110183]: osdmap e136: 8 total, 8 up, 6 in 2024-09-13T14:28:48.703 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:48 smithi102 ceph-mon[115269]: pgmap v389: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 864 op/s; 25440/255711 objects degraded (9.949%); 161/255711 objects misplaced (0.063%); 4.4 MiB/s, 69 objects/s recovering 2024-09-13T14:28:48.703 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:48 smithi102 ceph-mon[115269]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:28:48.703 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:48 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:28:48.703 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:48 smithi102 ceph-mon[115269]: osdmap e136: 8 total, 8 up, 6 in 2024-09-13T14:28:48.724 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:48.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:48.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:48.752 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:28:48.752 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-13T14:28:48.752 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:48.752 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:152379 5.d 5:bca7061b:::benchmark_data_smithi102_155989_object34461:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-13T14:28:48.753 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:28:48.736858+0000", 2024-09-13T14:28:48.753 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.01268938, 2024-09-13T14:28:48.753 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012723052, 2024-09-13T14:28:48.753 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:48.753 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:28:48.753 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:48.753 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:48.753 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:48.753 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 152379 2024-09-13T14:28:48.753 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:48.753 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:48.753 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:48.753 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:48.754 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:48.736858+0000", 2024-09-13T14:28:48.754 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:48.754 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:48.754 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:48.754 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:48.754 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:48.736858+0000", 2024-09-13T14:28:48.754 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:48.754 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:48.754 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:48.754 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:48.754 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:48.736857+0000", 2024-09-13T14:28:48.754 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:28:48.754 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:48.754 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:48.755 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:48.755 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:48.736863+0000", 2024-09-13T14:28:48.755 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8200000000000002e-06 2024-09-13T14:28:48.755 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:48.755 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:48.755 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:48.755 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:48.736872+0000", 2024-09-13T14:28:48.755 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.2410000000000001e-06 2024-09-13T14:28:48.756 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:48.756 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:48.756 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:48.756 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:48.736878+0000", 2024-09-13T14:28:48.756 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2610000000000001e-06 2024-09-13T14:28:48.756 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:48.756 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:48.756 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:48.756 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:48.739514+0000", 2024-09-13T14:28:48.756 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0026359389999999999 2024-09-13T14:28:48.756 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:48.756 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:48.756 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:48.757 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:48.739605+0000", 2024-09-13T14:28:48.757 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1656000000000005e-05 2024-09-13T14:28:48.757 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:48.757 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:48.757 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:28:48.757 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:48.739741+0000", 2024-09-13T14:28:48.757 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013617999999999999 2024-09-13T14:28:48.757 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:48.757 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:48.757 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:28:48.757 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:48.745672+0000", 2024-09-13T14:28:48.757 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0059307400000000003 2024-09-13T14:28:48.757 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:48.758 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:48.758 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:48.758 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:48.758 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:48.758 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop_reply(client.44546.0:152379 5.d e137/125)", 2024-09-13T14:28:48.758 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:28:48.745680+0000", 2024-09-13T14:28:48.758 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.0038675390000000001, 2024-09-13T14:28:48.758 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0039870299999999999, 2024-09-13T14:28:48.758 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:48.758 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "queued for pg", 2024-09-13T14:28:48.758 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:48.758 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:48.758 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:48.759 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:48.745680+0000", 2024-09-13T14:28:48.759 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:48.759 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:48.759 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:48.759 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:48.759 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:48.745680+0000", 2024-09-13T14:28:48.759 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:48.759 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:48.759 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:48.759 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:48.759 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:48.745679+0000", 2024-09-13T14:28:48.759 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:28:48.759 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:48.760 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:48.760 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:48.760 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:48.745685+0000", 2024-09-13T14:28:48.760 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7119999999999997e-06 2024-09-13T14:28:48.760 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:48.760 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:48.760 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:48.760 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:48.745697+0000", 2024-09-13T14:28:48.760 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2089000000000001e-05 2024-09-13T14:28:48.760 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:48.760 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:48.760 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:48.761 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:48.745704+0000", 2024-09-13T14:28:48.761 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1690000000000003e-06 2024-09-13T14:28:48.761 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:48.761 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:48.761 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:48.761 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:48.761 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:48.761 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:152385 5.d 5.5c461e1d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-13T14:28:48.761 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:28:48.745795+0000", 2024-09-13T14:28:48.761 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.0037526289999999999, 2024-09-13T14:28:48.761 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003924327, 2024-09-13T14:28:48.761 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:48.762 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "queued for pg", 2024-09-13T14:28:48.762 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:28:48.762 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:28:48.762 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:28:48.762 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 152385 2024-09-13T14:28:48.762 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:48.762 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:48.762 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:48.762 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:48.762 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:48.745795+0000", 2024-09-13T14:28:48.762 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:48.762 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:48.762 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:48.763 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:48.763 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:48.745795+0000", 2024-09-13T14:28:48.763 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:48.763 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:48.763 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:48.763 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:48.763 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:48.745794+0000", 2024-09-13T14:28:48.763 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:28:48.763 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:48.763 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:48.763 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:48.763 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:48.745799+0000", 2024-09-13T14:28:48.763 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.6890000000000001e-06 2024-09-13T14:28:48.764 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:48.764 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:48.764 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:48.764 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:48.745813+0000", 2024-09-13T14:28:48.764 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.343e-05 2024-09-13T14:28:48.764 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:48.764 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:48.764 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:48.764 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:48.745819+0000", 2024-09-13T14:28:48.764 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.156e-06 2024-09-13T14:28:48.764 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:48.764 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:48.765 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:48.765 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:48.765 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:48.765 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop_reply(client.44546.0:152379 5.d e137/125)", 2024-09-13T14:28:48.765 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:28:48.746117+0000", 2024-09-13T14:28:48.765 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.0034303350000000001, 2024-09-13T14:28:48.765 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0036512509999999999, 2024-09-13T14:28:48.765 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:48.765 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "queued for pg", 2024-09-13T14:28:48.765 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:48.765 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:48.765 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:48.766 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:48.746117+0000", 2024-09-13T14:28:48.766 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:48.766 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:48.766 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:48.766 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:48.766 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:48.746117+0000", 2024-09-13T14:28:48.766 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:48.766 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:48.766 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:48.766 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:48.766 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:48.746116+0000", 2024-09-13T14:28:48.766 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:28:48.766 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:48.766 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:48.767 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:48.767 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:48.746123+0000", 2024-09-13T14:28:48.767 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1739999999999997e-06 2024-09-13T14:28:48.767 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:48.767 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:48.767 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:48.767 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:48.746136+0000", 2024-09-13T14:28:48.767 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3582e-05 2024-09-13T14:28:48.767 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:48.767 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:48.767 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:48.767 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:48.746143+0000", 2024-09-13T14:28:48.768 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9190000000000002e-06 2024-09-13T14:28:48.768 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:48.768 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:48.768 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:48.768 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:48.768 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:48.768 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.44546.0:152384 5.6 e137/125 5:6650145e:::benchmark_data_smithi102_155989_object34466:head v 137'9528, mlcod=120'2143)", 2024-09-13T14:28:48.768 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:28:48.746257+0000", 2024-09-13T14:28:48.768 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.0032910410000000002, 2024-09-13T14:28:48.768 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003558833, 2024-09-13T14:28:48.768 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:28:48.769 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T14:28:48.769 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:28:48.769 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:48.769 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:28:48.769 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:48.746257+0000", 2024-09-13T14:28:48.769 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:48.769 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:48.770 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:48.770 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:28:48.770 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:48.746257+0000", 2024-09-13T14:28:48.770 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:28:48.770 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:48.770 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:48.770 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:28:48.770 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:48.746256+0000", 2024-09-13T14:28:48.770 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:28:48.770 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:48.770 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:48.770 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:28:48.771 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:48.746262+0000", 2024-09-13T14:28:48.771 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8690000000000001e-06 2024-09-13T14:28:48.771 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:48.771 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:48.771 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:28:48.771 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:48.746274+0000", 2024-09-13T14:28:48.771 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2553e-05 2024-09-13T14:28:48.771 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:48.771 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:48.771 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:28:48.771 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:48.746283+0000", 2024-09-13T14:28:48.771 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.0100000000000001e-06 2024-09-13T14:28:48.771 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:48.772 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:48.772 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:28:48.772 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:48.746346+0000", 2024-09-13T14:28:48.772 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2152999999999999e-05 2024-09-13T14:28:48.772 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:28:48.772 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:28:48.772 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:28:48.772 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:48.746371+0000", 2024-09-13T14:28:48.772 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5902999999999999e-05 2024-09-13T14:28:48.772 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:48.772 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:28:48.772 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:48.772 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:28:48.772 INFO:teuthology.orchestra.run.smithi195.stdout: ], 2024-09-13T14:28:48.773 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 5 2024-09-13T14:28:48.773 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:28:48.826 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:28:48.857 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:48.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:48.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:28:48.959 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:48.986 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:48.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:48.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:49.088 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:49.110 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:49.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:49.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:49.149 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:28:49.149 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:28:49.166 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:28:49.166 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:28:49.182 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:28:49.183 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:28:49.198 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:28:49.199 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:28:49.212 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:49.215 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:28:49.215 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:28:49.231 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:28:49.232 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:28:49.236 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:49.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:49.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:49.248 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:28:49.248 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:28:49.264 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:28:49.264 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:28:49.338 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:49.360 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:49.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:49.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:49.462 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:49.484 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:49.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:49.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:49.586 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:49.614 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:49.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:49.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:49.716 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:49.764 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:49.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:49.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:49.866 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:49.891 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:49.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:49.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:49.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:49 smithi102 ceph-mon[110183]: osdmap e137: 8 total, 8 up, 6 in 2024-09-13T14:28:49.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:49 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 25440/255711 objects degraded (9.949%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:28:49.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:49 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1582030989' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:28:49.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:49 smithi102 ceph-mon[115269]: osdmap e137: 8 total, 8 up, 6 in 2024-09-13T14:28:49.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:49 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 25440/255711 objects degraded (9.949%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:28:49.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:49 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1582030989' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:28:49.993 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:49.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:49 smithi195 ceph-mon[104262]: osdmap e137: 8 total, 8 up, 6 in 2024-09-13T14:28:49.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:49 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 25440/255711 objects degraded (9.949%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:28:49.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:49 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1582030989' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:28:50.036 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:50.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:50.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:50.138 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:50.162 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:50.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:50.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:50.264 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:28:50.303 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:50.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:50.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:28:50.405 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:50.441 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T14:28:50.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:50.443 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:50.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:50.471 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:28:50.545 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:50.611 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:50.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:50.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:50.713 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:28:50.737 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:50.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:50.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:28:50.752 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:50 smithi195 ceph-mon[104262]: pgmap v392: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 856 op/s; 25440/255027 objects degraded (9.975%); 161/255027 objects misplaced (0.063%); 5.9 MiB/s, 94 objects/s recovering 2024-09-13T14:28:50.753 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:50 smithi195 ceph-mon[104262]: osdmap e138: 8 total, 8 up, 6 in 2024-09-13T14:28:50.839 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:50.863 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:50.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:50.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:50.871 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:28:50.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:50 smithi102 ceph-mon[110183]: pgmap v392: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 856 op/s; 25440/255027 objects degraded (9.975%); 161/255027 objects misplaced (0.063%); 5.9 MiB/s, 94 objects/s recovering 2024-09-13T14:28:50.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:50 smithi102 ceph-mon[110183]: osdmap e138: 8 total, 8 up, 6 in 2024-09-13T14:28:50.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:50 smithi102 ceph-mon[115269]: pgmap v392: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 856 op/s; 25440/255027 objects degraded (9.975%); 161/255027 objects misplaced (0.063%); 5.9 MiB/s, 94 objects/s recovering 2024-09-13T14:28:50.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:50 smithi102 ceph-mon[115269]: osdmap e138: 8 total, 8 up, 6 in 2024-09-13T14:28:50.965 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:50.990 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:50.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:50.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:51.092 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:51.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:51.135 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:51.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:51.236 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:51.283 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:51.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:51.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:51.385 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:51.413 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:51.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:51.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:51.515 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:28:51.539 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:51.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:51.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:28:51.641 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:51.678 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:51.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:51.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:51.780 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:28:51.811 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:51.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:51.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:28:51.913 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:28:51.948 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:51.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:51.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:28:51.959 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:28:52.050 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:52.085 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:52.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:52.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:52.187 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:28:52.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:52.229 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:52.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:28:52.331 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:52.380 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:52.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:52.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:52.482 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:28:52.511 INFO:teuthology.orchestra.run.smithi102.stderr:reweighted osd.0 to 1 (10000) 2024-09-13T14:28:52.526 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:52.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:52.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:28:52.628 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:28:52.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:52 smithi102 ceph-mon[110183]: pgmap v394: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 582 op/s; 25440/255027 objects degraded (9.975%); 161/255027 objects misplaced (0.063%); 3.9 MiB/s, 62 objects/s recovering 2024-09-13T14:28:52.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:52 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3680983488' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:28:52.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:52 smithi102 ceph-mon[115269]: pgmap v394: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 582 op/s; 25440/255027 objects degraded (9.975%); 161/255027 objects misplaced (0.063%); 3.9 MiB/s, 62 objects/s recovering 2024-09-13T14:28:52.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:52 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3680983488' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:28:52.677 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:52.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:52.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:28:52.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:52 smithi195 ceph-mon[104262]: pgmap v394: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 582 op/s; 25440/255027 objects degraded (9.975%); 161/255027 objects misplaced (0.063%); 3.9 MiB/s, 62 objects/s recovering 2024-09-13T14:28:52.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:52 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3680983488' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:28:52.847 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:52.892 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:52.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:52.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:52.994 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:53.017 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:53.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:53.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:53.120 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:53.148 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:28:53.148 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:28:53.157 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:53.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:53.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:53.221 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:28:53.260 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:53.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:53.296 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:53.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:53.318 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-13T14:28:53.397 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:53.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:53.438 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:53.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:53.539 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:53.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:53.579 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:53.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:53.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:53 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3680983488' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T14:28:53.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:53 smithi102 ceph-mon[110183]: osdmap e139: 8 total, 8 up, 6 in 2024-09-13T14:28:53.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:53 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3680983488' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T14:28:53.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:53 smithi102 ceph-mon[115269]: osdmap e139: 8 total, 8 up, 6 in 2024-09-13T14:28:53.680 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:53.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:53.726 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:53.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:53.827 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:53.855 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:53 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3680983488' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T14:28:53.856 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:53 smithi195 ceph-mon[104262]: osdmap e139: 8 total, 8 up, 6 in 2024-09-13T14:28:53.865 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:53.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:53.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:53.895 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T14:28:53.967 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:53.994 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:53.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:53.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:54.096 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:54.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:54.139 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:54.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:54.240 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:54.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:54.294 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:54.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:54.337 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:28:54.391 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:28:54.392 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:28:54.395 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:28:54.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:54.434 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:54.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:28:54.535 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:54.561 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:54.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:54.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:54.663 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:54.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:54 smithi102 ceph-mon[110183]: pgmap v396: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.0 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 535 op/s; 19663/246414 objects degraded (7.980%); 5539/246414 objects misplaced (2.248%); 4.3 MiB/s, 68 objects/s recovering 2024-09-13T14:28:54.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:54 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 19663/246414 objects degraded (7.980%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:28:54.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:54 smithi102 ceph-mon[115269]: pgmap v396: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.0 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 535 op/s; 19663/246414 objects degraded (7.980%); 5539/246414 objects misplaced (2.248%); 4.3 MiB/s, 68 objects/s recovering 2024-09-13T14:28:54.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:54 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 19663/246414 objects degraded (7.980%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:28:54.688 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:54.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:54.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:54.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:54 smithi195 ceph-mon[104262]: pgmap v396: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.0 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 535 op/s; 19663/246414 objects degraded (7.980%); 5539/246414 objects misplaced (2.248%); 4.3 MiB/s, 68 objects/s recovering 2024-09-13T14:28:54.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:54 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 19663/246414 objects degraded (7.980%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:28:54.790 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:54.826 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:54.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:54.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:54.928 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:54.961 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:54.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:54.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:55.063 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:55.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:55.094 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:55.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:55.195 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:55.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:55.240 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:55.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:55.341 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:55.377 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:55.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:55.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:55.479 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:55.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:55.511 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:55.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:55.537 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is set 2024-09-13T14:28:55.612 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:55.637 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:55.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:55.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:55.739 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:55.763 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:55.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:55.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:55.865 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:28:55.888 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:55.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:55.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:28:55.921 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:55 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1057410713' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:28:55.922 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:55 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:28:55.922 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:55 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:28:55.922 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:55 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1057410713' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:28:55.922 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:55 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:28:55.922 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:55 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:28:55.991 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:55.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:55 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1057410713' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:28:55.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:55 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:28:55.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:55 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:28:56.015 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:56.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:56.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:56.117 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:56.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:56.164 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:56.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:56.265 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:28:56.303 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:56.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:56.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:28:56.405 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:56.448 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:56.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:56.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:56.538 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:28:56 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:28:56] "GET /metrics HTTP/1.1" 200 36540 "" "Prometheus/2.51.0" 2024-09-13T14:28:56.550 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:56.572 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:56.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:56.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:56.674 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:28:56.697 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:56.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:56.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:28:56.800 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:28:56.823 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:56.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:56.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:28:56.857 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:56 smithi195 ceph-mon[104262]: pgmap v397: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.0 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 451 op/s; 19663/245826 objects degraded (7.999%); 5539/245826 objects misplaced (2.253%); 3.7 MiB/s, 58 objects/s recovering 2024-09-13T14:28:56.857 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:56 smithi195 ceph-mon[104262]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:28:56.857 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:56 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1057410713' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:28:56.857 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:56 smithi195 ceph-mon[104262]: osdmap e140: 8 total, 8 up, 6 in 2024-09-13T14:28:56.887 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:56 smithi102 ceph-mon[110183]: pgmap v397: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.0 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 451 op/s; 19663/245826 objects degraded (7.999%); 5539/245826 objects misplaced (2.253%); 3.7 MiB/s, 58 objects/s recovering 2024-09-13T14:28:56.887 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:56 smithi102 ceph-mon[110183]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:28:56.887 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:56 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1057410713' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:28:56.887 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:56 smithi102 ceph-mon[110183]: osdmap e140: 8 total, 8 up, 6 in 2024-09-13T14:28:56.888 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:56 smithi102 ceph-mon[115269]: pgmap v397: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.0 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 451 op/s; 19663/245826 objects degraded (7.999%); 5539/245826 objects misplaced (2.253%); 3.7 MiB/s, 58 objects/s recovering 2024-09-13T14:28:56.888 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:56 smithi102 ceph-mon[115269]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:28:56.888 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:56 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1057410713' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:28:56.888 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:56 smithi102 ceph-mon[115269]: osdmap e140: 8 total, 8 up, 6 in 2024-09-13T14:28:56.925 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:56.949 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:56.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:56.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:57.051 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:57.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:57.102 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:57.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:57.203 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:57.248 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:57.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:57.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:57.350 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:57.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:57.378 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:57.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:57.478 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:57.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:57.524 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:57.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:57.626 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:57.652 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:57.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:57.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:57.756 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:28:57.779 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:57.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:57.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:28:57.829 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:28:57.830 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:28:57.881 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:28:57.924 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:57.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:57.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:28:58.025 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:58.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:58.070 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:58.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:58.171 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:28:58.197 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:58.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:58.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:28:58.299 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:28:58.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:58.348 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:58.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:28:58.449 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:28:58.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:58.479 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:58.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:28:58.552 INFO:teuthology.orchestra.run.smithi102.stderr:reweighted osd.3 to 1 (10000) 2024-09-13T14:28:58.566 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T14:28:58.581 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:58.594 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:28:58.595 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:28:58.627 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:28:58.627 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:28:58.631 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:58.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:58.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:58.656 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:28:58.657 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:28:58.685 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:28:58.685 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:28:58.709 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:28:58.710 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:28:58.736 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:58.757 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:28:58.757 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:28:58.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:58.781 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:58.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:58.788 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:28:58.788 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:28:58.819 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:28:58.819 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:28:58.820 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:58 smithi102 ceph-mon[110183]: pgmap v399: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.8 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 802 op/s; 19817/236271 objects degraded (8.387%); 5135/236271 objects misplaced (2.173%); 5.2 MiB/s, 83 objects/s recovering 2024-09-13T14:28:58.820 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:58 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1941342171' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:28:58.820 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:58 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:28:58.820 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:58 smithi102 ceph-mon[115269]: pgmap v399: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.8 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 802 op/s; 19817/236271 objects degraded (8.387%); 5135/236271 objects misplaced (2.173%); 5.2 MiB/s, 83 objects/s recovering 2024-09-13T14:28:58.820 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:58 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1941342171' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:28:58.820 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:58 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:28:58.882 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:58.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:58.922 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:58.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:58.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:58 smithi195 ceph-mon[104262]: pgmap v399: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.8 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 802 op/s; 19817/236271 objects degraded (8.387%); 5135/236271 objects misplaced (2.173%); 5.2 MiB/s, 83 objects/s recovering 2024-09-13T14:28:58.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:58 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1941342171' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:28:58.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:58 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:28:59.023 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:59.064 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:59.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:59.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:59.166 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:28:59.188 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:59.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:59.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:28:59.290 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:28:59.330 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:59.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:59.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:28:59.432 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:28:59.473 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:59.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:59.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:28:59.575 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:28:59.598 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:59 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T14:28:59.598 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:59 smithi102 ceph-mon[110183]: osdmap e141: 8 total, 8 up, 6 in 2024-09-13T14:28:59.598 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:59 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1265147552' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:28:59.598 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:28:59 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 19817/236271 objects degraded (8.387%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:28:59.598 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:59 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T14:28:59.598 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:59 smithi102 ceph-mon[115269]: osdmap e141: 8 total, 8 up, 6 in 2024-09-13T14:28:59.598 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:59 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1265147552' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:28:59.598 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:28:59 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 19817/236271 objects degraded (8.387%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:28:59.599 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:59.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:59.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:28:59.701 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:28:59.726 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:28:59.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:59.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:28:59.828 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:28:59.861 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:28:59.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:28:59.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:28:59.963 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:00.003 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:59 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T14:29:00.003 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:59 smithi195 ceph-mon[104262]: osdmap e141: 8 total, 8 up, 6 in 2024-09-13T14:29:00.003 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:59 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1265147552' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:29:00.003 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:28:59 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 19817/236271 objects degraded (8.387%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:29:00.004 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:00.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:00.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:00.010 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:29:00.075 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-13T14:29:00.110 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:00.129 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:29:00.143 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:00.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:00.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:00.245 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:00.270 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:00.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:00.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:00.372 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:00.410 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:00.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:00.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:00.512 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:00.536 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:00.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:00.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:00.638 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:00.665 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:00.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:00.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:00.767 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:00.802 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:00.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:00.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:00.904 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:00.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:00 smithi102 ceph-mon[110183]: pgmap v401: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.8 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 528 op/s; 19817/235656 objects degraded (8.409%); 5135/235656 objects misplaced (2.179%); 2.3 MiB/s, 36 objects/s recovering 2024-09-13T14:29:00.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:00 smithi102 ceph-mon[115269]: pgmap v401: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.8 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 528 op/s; 19817/235656 objects degraded (8.409%); 5135/235656 objects misplaced (2.179%); 2.3 MiB/s, 36 objects/s recovering 2024-09-13T14:29:00.932 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:29:00.933 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:29:00.943 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:00.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:00.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:00.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:00 smithi195 ceph-mon[104262]: pgmap v401: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.8 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 528 op/s; 19817/235656 objects degraded (8.409%); 5135/235656 objects misplaced (2.179%); 2.3 MiB/s, 36 objects/s recovering 2024-09-13T14:29:01.045 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:01.070 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:01.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:01.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:01.172 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:01.194 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:01.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:01.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:01.296 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:29:01.336 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:01.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:01.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:29:01.438 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:29:01.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:01.482 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:01.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:29:01.583 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:29:01.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:01.610 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:01.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:29:01.625 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:29:01.711 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:01.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:01.743 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:01.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:01.844 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:01.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:01.887 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:01.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:01.988 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:02.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:02.032 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:02.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:02.133 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:02.173 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:02.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:02.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:02.275 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:02.317 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:02.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:02.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:02.419 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:02.447 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:02.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:02.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:02.549 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:02.589 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:02.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:02.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:02.691 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:29:02.715 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:02.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:02.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:29:02.817 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:29:02.841 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:02.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:02.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:29:02.843 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:02 smithi102 ceph-mon[110183]: pgmap v402: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.8 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 448 op/s; 19817/235656 objects degraded (8.409%); 5135/235656 objects misplaced (2.179%); 2.0 MiB/s, 31 objects/s recovering 2024-09-13T14:29:02.843 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:02 smithi102 ceph-mon[115269]: pgmap v402: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.8 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 448 op/s; 19817/235656 objects degraded (8.409%); 5135/235656 objects misplaced (2.179%); 2.0 MiB/s, 31 objects/s recovering 2024-09-13T14:29:02.944 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:02.970 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:02.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:02.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:02.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:02 smithi195 ceph-mon[104262]: pgmap v402: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.8 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 448 op/s; 19817/235656 objects degraded (8.409%); 5135/235656 objects misplaced (2.179%); 2.0 MiB/s, 31 objects/s recovering 2024-09-13T14:29:03.072 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:03.097 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:03.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:03.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:03.198 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:03.244 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:03.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:03.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:03.346 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:03.369 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:03.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:03.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:03.471 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:03.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:03.512 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:03.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:03.613 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:03.645 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:03.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:03.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:03.652 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:29:03.748 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:03.770 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:03.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:03.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:03.871 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:03.900 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:03.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:03.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:04.002 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:04.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:04.039 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:04.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:04.140 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:04.167 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:04.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:04.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:04.268 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:29:04.293 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:04.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:04.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:29:04.394 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:04.421 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:04.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:04.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:04.524 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:29:04.548 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:04.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:04.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:29:04.650 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:29:04.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:04 smithi102 ceph-mon[110183]: pgmap v403: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.6 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 816 op/s; 19992/226227 objects degraded (8.837%); 4715/226227 objects misplaced (2.084%); 3.9 MiB/s, 61 objects/s recovering 2024-09-13T14:29:04.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:04 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 19992/226227 objects degraded (8.837%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:29:04.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:04 smithi102 ceph-mon[115269]: pgmap v403: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.6 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 816 op/s; 19992/226227 objects degraded (8.837%); 4715/226227 objects misplaced (2.084%); 3.9 MiB/s, 61 objects/s recovering 2024-09-13T14:29:04.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:04 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 19992/226227 objects degraded (8.837%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:29:04.691 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:04.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:04.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:29:04.793 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:04.821 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:04.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:04.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:04.923 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:04.932 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:29:04.933 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [ 2024-09-13T14:29:04.933 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:04.933 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.44546.0:163098 5.2 e141/116 5:4906aa2f:::benchmark_data_smithi102_155989_object45180:head v 141'10227, mlcod=141'10224)", 2024-09-13T14:29:04.933 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:29:04.925721+0000", 2024-09-13T14:29:04.934 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.0024610729999999998, 2024-09-13T14:29:04.934 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0024870679999999998, 2024-09-13T14:29:04.934 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:29:04.934 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:29:04.934 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "started", 2024-09-13T14:29:04.934 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:29:04.934 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:04.934 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:29:04.934 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:29:04.925721+0000", 2024-09-13T14:29:04.934 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:29:04.934 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:04.934 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:04.935 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:29:04.935 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:29:04.925721+0000", 2024-09-13T14:29:04.935 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:29:04.935 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:04.935 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:04.935 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:29:04.935 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:29:04.925721+0000", 2024-09-13T14:29:04.935 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.89e-07 2024-09-13T14:29:04.935 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:04.935 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:04.935 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:29:04.935 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:29:04.925727+0000", 2024-09-13T14:29:04.936 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.5759999999999996e-06 2024-09-13T14:29:04.936 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:04.936 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:04.936 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:29:04.936 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:29:04.925736+0000", 2024-09-13T14:29:04.936 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.9320000000000002e-06 2024-09-13T14:29:04.936 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:04.936 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:04.936 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:29:04.936 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:29:04.925742+0000", 2024-09-13T14:29:04.936 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.3600000000000001e-06 2024-09-13T14:29:04.936 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:04.937 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:04.937 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:29:04.937 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:29:04.925785+0000", 2024-09-13T14:29:04.937 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.2939999999999999e-05 2024-09-13T14:29:04.937 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:04.937 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:04.937 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:29:04.937 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:29:04.925798+0000", 2024-09-13T14:29:04.937 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2587e-05 2024-09-13T14:29:04.937 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:29:04.937 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:29:04.937 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:29:04.937 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:04.937 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:04.938 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.44546.0:163100 5.4 e141/125 5:2cc3ac39:::benchmark_data_smithi102_155989_object45182:head v 141'10372, mlcod=0'0)", 2024-09-13T14:29:04.938 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:29:04.926674+0000", 2024-09-13T14:29:04.938 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.001507396, 2024-09-13T14:29:04.938 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.001583969, 2024-09-13T14:29:04.938 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:29:04.938 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:29:04.938 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "started", 2024-09-13T14:29:04.938 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:29:04.938 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:04.938 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:29:04.938 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:29:04.926674+0000", 2024-09-13T14:29:04.938 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:29:04.938 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:04.938 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:04.939 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:29:04.939 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:29:04.926674+0000", 2024-09-13T14:29:04.939 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:29:04.939 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:04.939 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:04.939 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:29:04.939 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:29:04.926675+0000", 2024-09-13T14:29:04.939 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.3099999999999998e-07 2024-09-13T14:29:04.939 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:04.939 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:04.939 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:29:04.939 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:29:04.926680+0000", 2024-09-13T14:29:04.939 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.8659999999999997e-06 2024-09-13T14:29:04.939 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:04.940 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:04.940 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:29:04.940 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:29:04.926688+0000", 2024-09-13T14:29:04.940 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.1040000000000007e-06 2024-09-13T14:29:04.940 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:04.940 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:04.940 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:29:04.940 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:29:04.926692+0000", 2024-09-13T14:29:04.940 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.5129999999999998e-06 2024-09-13T14:29:04.940 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:04.940 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:04.940 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:29:04.940 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:29:04.926742+0000", 2024-09-13T14:29:04.940 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.9944999999999997e-05 2024-09-13T14:29:04.941 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:04.941 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:04.941 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:29:04.941 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:29:04.926766+0000", 2024-09-13T14:29:04.941 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.355e-05 2024-09-13T14:29:04.941 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:29:04.941 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:29:04.941 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:29:04.941 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:29:04.941 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-09-13T14:29:04.941 INFO:teuthology.orchestra.run.smithi102.stdout: "num_ops": 2 2024-09-13T14:29:04.941 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:29:04.951 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:04.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:04.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:04.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:04 smithi195 ceph-mon[104262]: pgmap v403: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.6 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 816 op/s; 19992/226227 objects degraded (8.837%); 4715/226227 objects misplaced (2.084%); 3.9 MiB/s, 61 objects/s recovering 2024-09-13T14:29:04.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:04 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 19992/226227 objects degraded (8.837%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:29:05.053 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:05.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:05.096 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:05.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:05.198 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:29:05.235 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:05.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:05.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:29:05.337 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:05.376 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:05.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:05.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:05.478 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:05.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:05.508 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:05.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:05.609 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:05.652 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:05.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:05.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:05.754 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:29:05.793 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:05.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:05.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:29:05.895 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:29:05.918 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:05.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:05.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:29:06.020 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:06.041 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T14:29:06.046 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:06.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:06.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:06.148 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:06.190 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:06.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:06.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:06.292 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:06.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:06.330 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:06.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:06.431 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:06.456 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:06.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:06.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:06.558 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:06.567 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:29:06 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:29:06] "GET /metrics HTTP/1.1" 200 36540 "" "Prometheus/2.51.0" 2024-09-13T14:29:06.574 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is set 2024-09-13T14:29:06.574 INFO:teuthology.orchestra.run.smithi102.stderr:reweighted osd.4 to 1 (10000) 2024-09-13T14:29:06.587 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:06.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:06.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:06.688 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:06.730 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:06.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:06.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:06.832 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:06.880 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:06.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:06.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:06.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:06 smithi102 ceph-mon[110183]: pgmap v404: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.6 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 694 op/s; 19992/225504 objects degraded (8.865%); 4715/225504 objects misplaced (2.091%); 3.2 MiB/s, 50 objects/s recovering 2024-09-13T14:29:06.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:06 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/4282843177' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:29:06.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:06 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1104501595' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:29:06.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:06 smithi102 ceph-mon[115269]: pgmap v404: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.6 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 694 op/s; 19992/225504 objects degraded (8.865%); 4715/225504 objects misplaced (2.091%); 3.2 MiB/s, 50 objects/s recovering 2024-09-13T14:29:06.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:06 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/4282843177' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:29:06.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:06 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1104501595' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:29:06.982 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:06.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:06 smithi195 ceph-mon[104262]: pgmap v404: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.6 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 694 op/s; 19992/225504 objects degraded (8.865%); 4715/225504 objects misplaced (2.091%); 3.2 MiB/s, 50 objects/s recovering 2024-09-13T14:29:06.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:06 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/4282843177' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:29:06.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:06 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1104501595' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:29:07.021 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:07.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:07.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:07.124 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:07.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:07.165 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:07.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:07.267 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:07.309 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:07.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:07.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:07.411 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:07.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:07.455 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:07.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:07.557 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:29:07.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:07.586 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:07.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:29:07.687 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:07.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:07.725 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:07.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:07.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:07 smithi195 ceph-mon[104262]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:29:07.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:07 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/4282843177' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:29:07.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:07 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1104501595' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T14:29:07.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:07 smithi195 ceph-mon[104262]: osdmap e142: 8 total, 8 up, 6 in 2024-09-13T14:29:07.826 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:07.863 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:07.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:07.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:07.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:07 smithi102 ceph-mon[110183]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:29:07.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:07 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/4282843177' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:29:07.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:07 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1104501595' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T14:29:07.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:07 smithi102 ceph-mon[110183]: osdmap e142: 8 total, 8 up, 6 in 2024-09-13T14:29:07.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:07 smithi102 ceph-mon[115269]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:29:07.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:07 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/4282843177' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:29:07.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:07 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1104501595' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T14:29:07.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:07 smithi102 ceph-mon[115269]: osdmap e142: 8 total, 8 up, 6 in 2024-09-13T14:29:07.965 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:08.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:08.015 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:08.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:08.116 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:08.141 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:08.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:08.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:08.243 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:08.281 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:08.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:08.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:08.382 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:08.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:08.405 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:08.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:08.506 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:08.530 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:08.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:08.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:08.632 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:08.677 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:08.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:08.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:08.754 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-13T14:29:08.778 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:08.826 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:08.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:08.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:08.889 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:08 smithi102 ceph-mon[110183]: pgmap v406: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.4 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 782 op/s; 20186/215139 objects degraded (9.383%); 4262/215139 objects misplaced (1.981%); 3.6 MiB/s, 57 objects/s recovering 2024-09-13T14:29:08.889 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:08 smithi102 ceph-mon[115269]: pgmap v406: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.4 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 782 op/s; 20186/215139 objects degraded (9.383%); 4262/215139 objects misplaced (1.981%); 3.6 MiB/s, 57 objects/s recovering 2024-09-13T14:29:08.927 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:29:08.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:08.955 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:08.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:29:08.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:08 smithi195 ceph-mon[104262]: pgmap v406: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.4 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 782 op/s; 20186/215139 objects degraded (9.383%); 4262/215139 objects misplaced (1.981%); 3.6 MiB/s, 57 objects/s recovering 2024-09-13T14:29:09.026 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:29:09.026 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:29:09.056 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:09.068 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:29:09.068 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:29:09.069 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:29:09.105 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:29:09.105 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:29:09.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:09.132 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:09.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:09.144 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:29:09.144 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:29:09.174 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:29:09.174 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:29:09.204 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:29:09.204 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:29:09.228 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:29:09.228 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:29:09.234 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:09.255 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:29:09.255 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:29:09.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:09.265 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:09.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:09.366 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:09.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:09.404 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:09.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:09.506 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:09.540 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:09.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:09.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:09.643 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:09.669 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:09.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:09.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:09.771 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:09.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:09.798 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:09.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:09.900 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:29:09.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:09 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 20186/215139 objects degraded (9.383%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:29:09.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:09 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2924235062' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:29:09.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:09 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 20186/215139 objects degraded (9.383%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:29:09.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:09 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2924235062' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:29:09.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:09.941 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:09.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:29:09.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:09 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 20186/215139 objects degraded (9.383%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:29:09.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:09 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2924235062' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:29:10.042 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:10.046 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:29:10.083 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:10.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:10.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:10.169 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:29:10.185 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:10.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:10.215 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:10.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:10.316 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:10.353 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:10.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:10.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:10.454 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:29:10.481 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:10.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:10.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:29:10.583 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:10.627 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:10.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:10.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:10.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:10 smithi102 ceph-mon[110183]: pgmap v407: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.4 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 705 op/s; 20186/214497 objects degraded (9.411%); 4262/214497 objects misplaced (1.987%); 3.1 MiB/s, 50 objects/s recovering 2024-09-13T14:29:10.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:29:10.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:29:10.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:10 smithi102 ceph-mon[115269]: pgmap v407: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.4 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 705 op/s; 20186/214497 objects degraded (9.411%); 4262/214497 objects misplaced (1.987%); 3.1 MiB/s, 50 objects/s recovering 2024-09-13T14:29:10.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:29:10.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:29:10.729 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:10.755 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:10.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:10.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:10.857 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:10.888 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:10.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:10.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:10.989 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:10.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:10 smithi195 ceph-mon[104262]: pgmap v407: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.4 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 705 op/s; 20186/214497 objects degraded (9.411%); 4262/214497 objects misplaced (1.987%); 3.1 MiB/s, 50 objects/s recovering 2024-09-13T14:29:10.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:29:10.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:29:11.021 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:11.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:11.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:11.123 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:29:11.147 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:11.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:11.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:29:11.249 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:11.286 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:11.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:11.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:11.388 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:11.411 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:11.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:11.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:11.513 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:11.548 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:11.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:11.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:11.650 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:11.673 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:11.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:11.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:11.775 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:29:11.814 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:11.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:11.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:29:11.896 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:29:11.916 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:29:11.956 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:11.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:11.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:29:12.059 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:29:12.084 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:12.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:12.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:29:12.181 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:29:12.186 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:29:12.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:12.250 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:12.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:29:12.351 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:12.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:12.392 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:12.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:12.494 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:29:12.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:12.522 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:12.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:29:12.623 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:12.649 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:12.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:12.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:12.752 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:12.779 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:12.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:12.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:12.872 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:12 smithi102 ceph-mon[110183]: pgmap v408: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.4 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 705 op/s; 20186/214497 objects degraded (9.411%); 4262/214497 objects misplaced (1.987%); 3.1 MiB/s, 50 objects/s recovering 2024-09-13T14:29:12.873 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:12 smithi102 ceph-mon[115269]: pgmap v408: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.4 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 705 op/s; 20186/214497 objects degraded (9.411%); 4262/214497 objects misplaced (1.987%); 3.1 MiB/s, 50 objects/s recovering 2024-09-13T14:29:12.881 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:12.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:12.919 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:12.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:12.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:12 smithi195 ceph-mon[104262]: pgmap v408: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.4 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 705 op/s; 20186/214497 objects degraded (9.411%); 4262/214497 objects misplaced (1.987%); 3.1 MiB/s, 50 objects/s recovering 2024-09-13T14:29:13.021 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:13.053 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:13.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:13.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:13.155 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:13.190 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:13.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:13.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:13.205 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:29:13.206 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:29:13.292 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:13.316 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:13.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:13.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:13.418 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:13.443 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:13.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:13.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:13.545 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:13.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:13.580 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:13.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:13.680 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:13.707 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:13.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:13.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:13.809 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:29:13.833 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:13.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:13.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:29:13.878 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T14:29:13.935 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:29:13.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:13.970 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:13.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:29:14.071 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:29:14.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:14.096 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:14.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:29:14.197 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:14.244 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:14.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:14.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:14.348 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:14.372 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:14.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:14.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:14.474 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:14.516 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:14.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:14.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:14.618 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:14.661 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:14.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:14.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:14.763 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:14.802 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:14.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:14.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:14.804 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:14 smithi102 ceph-mon[115269]: pgmap v409: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.2 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 728 op/s; 20359/204372 objects degraded (9.962%); 3822/204372 objects misplaced (1.870%); 3.3 MiB/s, 52 objects/s recovering 2024-09-13T14:29:14.804 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:14 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 20359/204372 objects degraded (9.962%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:29:14.805 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:14 smithi102 ceph-mon[110183]: pgmap v409: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.2 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 728 op/s; 20359/204372 objects degraded (9.962%); 3822/204372 objects misplaced (1.870%); 3.3 MiB/s, 52 objects/s recovering 2024-09-13T14:29:14.805 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:14 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 20359/204372 objects degraded (9.962%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:29:14.905 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:14.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:14.947 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:14.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:14.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:14 smithi195 ceph-mon[104262]: pgmap v409: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.2 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 728 op/s; 20359/204372 objects degraded (9.962%); 3822/204372 objects misplaced (1.870%); 3.3 MiB/s, 52 objects/s recovering 2024-09-13T14:29:14.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:14 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 20359/204372 objects degraded (9.962%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:29:15.048 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:15.084 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:15.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:15.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:15.186 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:15.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:15.210 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:15.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:15.277 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:29:15.281 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:29:15.312 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:15.343 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:15.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:15.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:15.445 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:15.468 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:15.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:15.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:15.571 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:29:15.601 INFO:teuthology.orchestra.run.smithi102.stderr:reweighted osd.5 to 1 (10000) 2024-09-13T14:29:15.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:15.616 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:15.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:29:15.717 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:15.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:15.762 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:15.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:15.864 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:15.902 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:15.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:15.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:15.909 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:15 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3282809659' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:29:15.909 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:15 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3282809659' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:29:15.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:15 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3282809659' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:29:16.004 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:16.028 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:16.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:16.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:16.130 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:16.168 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:16.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:16.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:16.270 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:16.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:16.296 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:16.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:16.397 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:16.433 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:16.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:16.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:16.535 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:29:16.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:16.567 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:16.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:29:16.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:16 smithi102 ceph-mon[110183]: pgmap v410: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.1 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 727 op/s; 20359/203664 objects degraded (9.996%); 3822/203664 objects misplaced (1.877%); 3.3 MiB/s, 52 objects/s recovering 2024-09-13T14:29:16.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:16 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3282809659' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T14:29:16.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:16 smithi102 ceph-mon[110183]: osdmap e143: 8 total, 8 up, 6 in 2024-09-13T14:29:16.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:16 smithi102 ceph-mon[115269]: pgmap v410: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.1 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 727 op/s; 20359/203664 objects degraded (9.996%); 3822/203664 objects misplaced (1.877%); 3.3 MiB/s, 52 objects/s recovering 2024-09-13T14:29:16.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:16 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3282809659' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T14:29:16.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:16 smithi102 ceph-mon[115269]: osdmap e143: 8 total, 8 up, 6 in 2024-09-13T14:29:16.669 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:29:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:29:16] "GET /metrics HTTP/1.1" 200 36543 "" "Prometheus/2.51.0" 2024-09-13T14:29:16.669 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:16.693 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:16.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:16.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:16.795 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:16.832 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:16.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:16.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:16.935 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:16.961 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:16.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:16.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:16.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:16 smithi195 ceph-mon[104262]: pgmap v410: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.1 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 727 op/s; 20359/203664 objects degraded (9.996%); 3822/203664 objects misplaced (1.877%); 3.3 MiB/s, 52 objects/s recovering 2024-09-13T14:29:16.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:16 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3282809659' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T14:29:16.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:16 smithi195 ceph-mon[104262]: osdmap e143: 8 total, 8 up, 6 in 2024-09-13T14:29:17.063 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:17.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:17.112 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:17.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:17.213 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:29:17.239 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:17.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:17.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:29:17.341 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:17.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:17.381 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:17.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:17.482 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:17.506 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:17.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:17.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:17.608 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:17.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:17.655 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:17.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:17.756 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:17.781 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:17.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:17.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:17.883 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:17.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:17.938 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:17.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:18.039 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:18.065 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:18.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:18.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:18.167 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:18.201 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:18.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:18.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:18.303 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:29:18.326 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:18.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:18.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:29:18.428 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:18.452 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-13T14:29:18.469 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:18.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:18.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:18.573 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:18.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:18.606 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:18.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:18.624 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is unset 2024-09-13T14:29:18.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:18 smithi102 ceph-mon[115269]: pgmap v412: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.9 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 724 op/s; 20564/193389 objects degraded (10.633%); 3350/193389 objects misplaced (1.732%); 3.3 MiB/s, 53 objects/s recovering 2024-09-13T14:29:18.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:18 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3673861913' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:29:18.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:18 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:29:18.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:18 smithi102 ceph-mon[110183]: pgmap v412: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.9 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 724 op/s; 20564/193389 objects degraded (10.633%); 3350/193389 objects misplaced (1.732%); 3.3 MiB/s, 53 objects/s recovering 2024-09-13T14:29:18.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:18 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3673861913' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:29:18.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:18 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:29:18.706 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:18.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:18.762 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:18.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:18.865 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:18.893 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:18.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:18.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:18.995 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:18.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:18 smithi195 ceph-mon[104262]: pgmap v412: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.9 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 724 op/s; 20564/193389 objects degraded (10.633%); 3350/193389 objects misplaced (1.732%); 3.3 MiB/s, 53 objects/s recovering 2024-09-13T14:29:18.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:18 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3673861913' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:29:18.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:18 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:29:19.035 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:19.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:19.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:19.137 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:29:19.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:19.174 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:19.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:29:19.275 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:19.314 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:19.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:19.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:19.416 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:19.444 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:19.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:19.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:19.545 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:19.581 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:19.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:19.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:19.683 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:19.720 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:19.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:19.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:19.822 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:19.854 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:19.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:19.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:19.885 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:19 smithi102 ceph-mon[110183]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:29:19.885 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:19 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:29:19.885 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:19 smithi102 ceph-mon[110183]: osdmap e144: 8 total, 8 up, 6 in 2024-09-13T14:29:19.885 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:19 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 20564/193389 objects degraded (10.633%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:29:19.885 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:19 smithi102 ceph-mon[115269]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:29:19.886 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:19 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:29:19.886 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:19 smithi102 ceph-mon[115269]: osdmap e144: 8 total, 8 up, 6 in 2024-09-13T14:29:19.886 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:19 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 20564/193389 objects degraded (10.633%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:29:19.955 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:19.979 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:19.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:19.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:19.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:19 smithi195 ceph-mon[104262]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:29:19.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:19 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:29:19.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:19 smithi195 ceph-mon[104262]: osdmap e144: 8 total, 8 up, 6 in 2024-09-13T14:29:19.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:19 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 20564/193389 objects degraded (10.633%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:29:20.081 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:20.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:20.128 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:20.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:20.229 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:20.253 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:20.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:20.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:20.357 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:20.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:20.400 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:20.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:20.501 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:29:20.546 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:20.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:20.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:29:20.649 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:20.650 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:29:20.687 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:29:20.687 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:29:20.689 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:20.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:20.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:20.711 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:29:20.711 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:29:20.733 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:29:20.734 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:29:20.753 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:29:20.753 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:29:20.780 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:29:20.780 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:29:20.791 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:20.812 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:29:20.812 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:29:20.819 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:20.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:20.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:20.849 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:29:20.849 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:29:20.873 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:29:20.873 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:29:20.885 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:20 smithi102 ceph-mon[110183]: pgmap v414: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.9 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 907 op/s; 20564/192717 objects degraded (10.671%); 3350/192717 objects misplaced (1.738%); 4.2 MiB/s, 66 objects/s recovering 2024-09-13T14:29:20.886 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:20 smithi102 ceph-mon[115269]: pgmap v414: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.9 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 907 op/s; 20564/192717 objects degraded (10.671%); 3350/192717 objects misplaced (1.738%); 4.2 MiB/s, 66 objects/s recovering 2024-09-13T14:29:20.921 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:20.947 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:20.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:20.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:20.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:20 smithi195 ceph-mon[104262]: pgmap v414: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.9 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 907 op/s; 20564/192717 objects degraded (10.671%); 3350/192717 objects misplaced (1.738%); 4.2 MiB/s, 66 objects/s recovering 2024-09-13T14:29:21.049 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:21.090 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:21.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:21.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:21.192 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:21.242 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:21.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:21.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:21.344 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:21.372 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:21.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:21.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:21.473 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:29:21.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:21.511 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:21.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:29:21.612 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:29:21.645 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:21.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:21.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:29:21.655 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:29:21.747 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:21.775 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:21.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:21.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:21.778 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:21 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2143522644' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:29:21.778 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:21 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2143522644' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:29:21.879 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:21.908 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:21.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:21.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:21.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:21 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2143522644' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:29:22.010 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:22.011 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:29:22.011 INFO:teuthology.orchestra.run.smithi102.stdout: "size": 20, 2024-09-13T14:29:22.011 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 600, 2024-09-13T14:29:22.011 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [ 2024-09-13T14:29:22.011 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.011 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:88025 5.8 5:11d6cef9:::benchmark_data_smithi102_155989_object88024:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:29:22.011 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:27:22.491284+0000", 2024-09-13T14:29:22.011 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 119.514301948, 2024-09-13T14:29:22.012 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.104851342, 2024-09-13T14:29:22.012 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:29:22.012 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:29:22.012 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:29:22.012 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:29:22.012 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:29:22.012 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:29:22.012 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 88025 2024-09-13T14:29:22.012 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.012 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:29:22.012 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.012 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:29:22.012 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:22.491284+0000", 2024-09-13T14:29:22.013 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:29:22.013 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.013 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.013 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:29:22.013 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:22.491284+0000", 2024-09-13T14:29:22.013 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:29:22.013 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.013 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.013 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:29:22.013 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:22.491285+0000", 2024-09-13T14:29:22.013 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.51e-07 2024-09-13T14:29:22.013 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.013 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.013 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:29:22.013 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:22.491301+0000", 2024-09-13T14:29:22.014 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.677e-05 2024-09-13T14:29:22.014 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.014 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.014 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:29:22.014 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:22.491308+0000", 2024-09-13T14:29:22.014 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.7449999999999998e-06 2024-09-13T14:29:22.014 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.014 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.014 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:29:22.015 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:22.491312+0000", 2024-09-13T14:29:22.015 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.7249999999999999e-06 2024-09-13T14:29:22.015 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.015 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.015 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:29:22.015 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:22.491350+0000", 2024-09-13T14:29:22.015 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.8198999999999998e-05 2024-09-13T14:29:22.015 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.015 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.015 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:29:22.015 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:22.491465+0000", 2024-09-13T14:29:22.015 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000114572 2024-09-13T14:29:22.015 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.016 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.016 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,5", 2024-09-13T14:29:22.016 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:22.491570+0000", 2024-09-13T14:29:22.016 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000105499 2024-09-13T14:29:22.016 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.016 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.016 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:29:22.016 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:22.494338+0000", 2024-09-13T14:29:22.016 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0027675400000000002 2024-09-13T14:29:22.016 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.016 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.016 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:29:22.016 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:22.495782+0000", 2024-09-13T14:29:22.017 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0014445669999999999 2024-09-13T14:29:22.017 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.017 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.017 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:29:22.017 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:22.594577+0000", 2024-09-13T14:29:22.017 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.098794655999999995 2024-09-13T14:29:22.017 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.017 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.017 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:29:22.017 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:22.594636+0000", 2024-09-13T14:29:22.017 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.9261000000000001e-05 2024-09-13T14:29:22.017 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.017 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.017 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:29:22.018 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:22.596135+0000", 2024-09-13T14:29:22.018 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0014992569999999999 2024-09-13T14:29:22.018 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:29:22.018 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:29:22.018 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:29:22.018 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.018 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.018 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:90264 5.b 5:d62e4012:::benchmark_data_smithi102_155989_object90263:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:29:22.018 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:27:25.244092+0000", 2024-09-13T14:29:22.018 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 116.76149366999999, 2024-09-13T14:29:22.018 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.107510888, 2024-09-13T14:29:22.018 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:29:22.018 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:29:22.018 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:29:22.019 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:29:22.019 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:29:22.019 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:29:22.019 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 90264 2024-09-13T14:29:22.019 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.019 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:29:22.019 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.019 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:29:22.019 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.244092+0000", 2024-09-13T14:29:22.019 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:29:22.019 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.019 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.019 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:29:22.019 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.244092+0000", 2024-09-13T14:29:22.020 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:29:22.020 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.020 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.020 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:29:22.020 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.244093+0000", 2024-09-13T14:29:22.020 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.2500000000000004e-07 2024-09-13T14:29:22.020 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.020 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.020 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:29:22.020 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.244117+0000", 2024-09-13T14:29:22.020 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.3901e-05 2024-09-13T14:29:22.020 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.020 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.020 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:29:22.021 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.244128+0000", 2024-09-13T14:29:22.021 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0857e-05 2024-09-13T14:29:22.021 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.021 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.021 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:29:22.021 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.244134+0000", 2024-09-13T14:29:22.021 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.4559999999999997e-06 2024-09-13T14:29:22.021 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.021 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.021 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:29:22.021 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.244186+0000", 2024-09-13T14:29:22.021 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.1283999999999999e-05 2024-09-13T14:29:22.021 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.021 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.021 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:29:22.022 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.244313+0000", 2024-09-13T14:29:22.022 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000127868 2024-09-13T14:29:22.022 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.022 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.022 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 5,7", 2024-09-13T14:29:22.022 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.244458+0000", 2024-09-13T14:29:22.022 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000144893 2024-09-13T14:29:22.022 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.022 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.022 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:29:22.022 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.247474+0000", 2024-09-13T14:29:22.022 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0030161060000000002 2024-09-13T14:29:22.022 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.022 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.023 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:29:22.023 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.255216+0000", 2024-09-13T14:29:22.023 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0077416250000000002 2024-09-13T14:29:22.023 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.023 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.023 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:29:22.023 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.350031+0000", 2024-09-13T14:29:22.023 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.094814988000000003 2024-09-13T14:29:22.023 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.023 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.023 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:29:22.023 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.350103+0000", 2024-09-13T14:29:22.023 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.1499000000000001e-05 2024-09-13T14:29:22.023 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.024 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.024 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:29:22.024 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.351603+0000", 2024-09-13T14:29:22.024 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0015005859999999999 2024-09-13T14:29:22.024 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:29:22.024 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:29:22.024 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:29:22.024 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.024 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.024 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.44546.0:90240 5.4 e129/125 5:2cbe6f91:::benchmark_data_smithi102_155989_object90239:head v 129'5726, mlcod=0'0)", 2024-09-13T14:29:22.025 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:27:25.244455+0000", 2024-09-13T14:29:22.025 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 116.761131021, 2024-09-13T14:29:22.025 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.108220684, 2024-09-13T14:29:22.025 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:29:22.025 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:29:22.025 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:29:22.025 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:29:22.025 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.025 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:29:22.025 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.244455+0000", 2024-09-13T14:29:22.025 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:29:22.025 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.025 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.025 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:29:22.026 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.244455+0000", 2024-09-13T14:29:22.026 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:29:22.026 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.026 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.026 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:29:22.026 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.244456+0000", 2024-09-13T14:29:22.026 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.2699999999999999e-07 2024-09-13T14:29:22.026 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.026 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.026 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:29:22.026 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.244519+0000", 2024-09-13T14:29:22.026 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.3781999999999993e-05 2024-09-13T14:29:22.026 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.026 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.027 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:29:22.027 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.244529+0000", 2024-09-13T14:29:22.027 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.8640000000000008e-06 2024-09-13T14:29:22.027 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.027 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.027 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:29:22.027 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.244535+0000", 2024-09-13T14:29:22.027 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.57e-06 2024-09-13T14:29:22.027 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.027 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.027 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:29:22.027 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.244624+0000", 2024-09-13T14:29:22.027 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.9716999999999995e-05 2024-09-13T14:29:22.027 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.028 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.028 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:29:22.028 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.244649+0000", 2024-09-13T14:29:22.028 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.4513999999999999e-05 2024-09-13T14:29:22.028 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.028 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.028 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:29:22.028 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.352624+0000", 2024-09-13T14:29:22.028 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.107975212 2024-09-13T14:29:22.028 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.028 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.028 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:29:22.028 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.352676+0000", 2024-09-13T14:29:22.028 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.1397999999999998e-05 2024-09-13T14:29:22.029 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:29:22.029 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:29:22.029 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:29:22.029 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.029 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.029 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.44546.0:90243 5.4 e129/125 5:2eed8f7a:::benchmark_data_smithi102_155989_object90242:head v 129'5727, mlcod=0'0)", 2024-09-13T14:29:22.029 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:27:25.248059+0000", 2024-09-13T14:29:22.029 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 116.75752698399999, 2024-09-13T14:29:22.029 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.104650215, 2024-09-13T14:29:22.029 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:29:22.029 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:29:22.029 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:29:22.029 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:29:22.029 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.030 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:29:22.030 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.248059+0000", 2024-09-13T14:29:22.030 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:29:22.030 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.030 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.030 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:29:22.030 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.248059+0000", 2024-09-13T14:29:22.030 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:29:22.030 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.030 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.030 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:29:22.030 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.248060+0000", 2024-09-13T14:29:22.030 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.92e-07 2024-09-13T14:29:22.030 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.031 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.031 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:29:22.031 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.248112+0000", 2024-09-13T14:29:22.031 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.2417000000000003e-05 2024-09-13T14:29:22.031 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.031 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.031 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:29:22.031 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.248123+0000", 2024-09-13T14:29:22.031 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0696000000000001e-05 2024-09-13T14:29:22.031 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.031 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.031 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:29:22.031 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.248132+0000", 2024-09-13T14:29:22.031 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.9029999999999993e-06 2024-09-13T14:29:22.032 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.032 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.032 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:29:22.032 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.249687+0000", 2024-09-13T14:29:22.032 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0015555829999999999 2024-09-13T14:29:22.032 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.032 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.032 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:29:22.032 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.249705+0000", 2024-09-13T14:29:22.032 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.7206000000000001e-05 2024-09-13T14:29:22.032 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.033 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.033 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:29:22.033 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.352681+0000", 2024-09-13T14:29:22.033 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.102976849 2024-09-13T14:29:22.033 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.033 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.033 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:29:22.033 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.352709+0000", 2024-09-13T14:29:22.033 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.7769000000000001e-05 2024-09-13T14:29:22.033 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:29:22.033 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:29:22.033 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:29:22.033 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.033 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.034 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.44546.0:90251 5.4 e129/125 5:22275ccb:::benchmark_data_smithi102_155989_object90250:head v 129'5728, mlcod=0'0)", 2024-09-13T14:29:22.034 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:27:25.251471+0000", 2024-09-13T14:29:22.034 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 116.75411447499999, 2024-09-13T14:29:22.034 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.111321792, 2024-09-13T14:29:22.034 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:29:22.034 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:29:22.034 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:29:22.034 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:29:22.034 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.034 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:29:22.035 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.251471+0000", 2024-09-13T14:29:22.035 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:29:22.035 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.035 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.035 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:29:22.035 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.251471+0000", 2024-09-13T14:29:22.035 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:29:22.035 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.035 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.035 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:29:22.035 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.251472+0000", 2024-09-13T14:29:22.035 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.2099999999999996e-07 2024-09-13T14:29:22.035 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.035 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.036 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:29:22.036 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.251533+0000", 2024-09-13T14:29:22.036 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.0692000000000001e-05 2024-09-13T14:29:22.036 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.036 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.036 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:29:22.036 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.251540+0000", 2024-09-13T14:29:22.036 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.5689999999999999e-06 2024-09-13T14:29:22.036 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.036 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.036 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:29:22.036 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.251545+0000", 2024-09-13T14:29:22.036 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.2189999999999998e-06 2024-09-13T14:29:22.036 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.037 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.037 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:29:22.037 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.255944+0000", 2024-09-13T14:29:22.037 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0043997180000000004 2024-09-13T14:29:22.037 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.037 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.037 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:29:22.037 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.255961+0000", 2024-09-13T14:29:22.037 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.6956000000000002e-05 2024-09-13T14:29:22.037 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.037 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.037 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:29:22.037 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.362738+0000", 2024-09-13T14:29:22.038 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.106777187 2024-09-13T14:29:22.038 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.038 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.038 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:29:22.038 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.362793+0000", 2024-09-13T14:29:22.038 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.4830000000000002e-05 2024-09-13T14:29:22.038 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:29:22.038 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:29:22.038 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:29:22.038 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.038 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.038 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:90268 5.b 5:d7934069:::benchmark_data_smithi102_155989_object90267:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:29:22.038 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:27:25.252276+0000", 2024-09-13T14:29:22.038 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 116.753310185, 2024-09-13T14:29:22.038 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.10246933900000001, 2024-09-13T14:29:22.039 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:29:22.039 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:29:22.039 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:29:22.039 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:29:22.039 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:29:22.039 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:29:22.039 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 90268 2024-09-13T14:29:22.039 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.039 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:29:22.039 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.039 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:29:22.039 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.252276+0000", 2024-09-13T14:29:22.039 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:29:22.039 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.040 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.040 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:29:22.040 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.252276+0000", 2024-09-13T14:29:22.040 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:29:22.040 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.040 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.040 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:29:22.040 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.252276+0000", 2024-09-13T14:29:22.040 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.4799999999999998e-07 2024-09-13T14:29:22.040 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.040 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.040 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:29:22.040 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.252298+0000", 2024-09-13T14:29:22.040 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.1676e-05 2024-09-13T14:29:22.041 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.041 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.041 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:29:22.041 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.252306+0000", 2024-09-13T14:29:22.041 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.9240000000000007e-06 2024-09-13T14:29:22.041 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.041 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.041 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:29:22.041 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.252312+0000", 2024-09-13T14:29:22.041 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.4150000000000004e-06 2024-09-13T14:29:22.041 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.041 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.041 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:29:22.041 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.255265+0000", 2024-09-13T14:29:22.042 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0029528179999999998 2024-09-13T14:29:22.042 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.042 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.042 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:29:22.042 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.255361+0000", 2024-09-13T14:29:22.042 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.6138999999999996e-05 2024-09-13T14:29:22.042 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.042 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.042 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 5,7", 2024-09-13T14:29:22.042 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.255454+0000", 2024-09-13T14:29:22.042 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.2447000000000004e-05 2024-09-13T14:29:22.042 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.042 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.042 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:29:22.042 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.349945+0000", 2024-09-13T14:29:22.043 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.094491358999999997 2024-09-13T14:29:22.043 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.043 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.043 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:29:22.043 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.351685+0000", 2024-09-13T14:29:22.043 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00173974 2024-09-13T14:29:22.043 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.043 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.043 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:29:22.043 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.353316+0000", 2024-09-13T14:29:22.043 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0016307979999999999 2024-09-13T14:29:22.043 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.043 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.043 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:29:22.044 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.353368+0000", 2024-09-13T14:29:22.044 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.2160000000000002e-05 2024-09-13T14:29:22.044 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.044 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.044 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:29:22.044 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.354745+0000", 2024-09-13T14:29:22.044 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.001377315 2024-09-13T14:29:22.044 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:29:22.044 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:29:22.044 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:29:22.044 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.044 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.044 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.44546.0:90269 5.4 e129/125 5:2fd94343:::benchmark_data_smithi102_155989_object90268:head v 129'5729, mlcod=0'0)", 2024-09-13T14:29:22.045 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:27:25.259532+0000", 2024-09-13T14:29:22.045 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 116.746053782, 2024-09-13T14:29:22.045 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.10329374600000001, 2024-09-13T14:29:22.045 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:29:22.045 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:29:22.045 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:29:22.045 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:29:22.045 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.045 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:29:22.045 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.259532+0000", 2024-09-13T14:29:22.045 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:29:22.046 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.046 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.046 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:29:22.046 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.259532+0000", 2024-09-13T14:29:22.046 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:29:22.046 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.046 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.046 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:29:22.046 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.259533+0000", 2024-09-13T14:29:22.046 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.3600000000000003e-07 2024-09-13T14:29:22.046 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.046 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.046 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:29:22.046 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.259586+0000", 2024-09-13T14:29:22.047 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.2938e-05 2024-09-13T14:29:22.047 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.047 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.047 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:29:22.047 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.345896+0000", 2024-09-13T14:29:22.047 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.086309773000000006 2024-09-13T14:29:22.047 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.047 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.047 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:29:22.047 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.345911+0000", 2024-09-13T14:29:22.047 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.5500999999999999e-05 2024-09-13T14:29:22.047 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.047 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.047 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:29:22.047 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.345972+0000", 2024-09-13T14:29:22.048 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.0921000000000002e-05 2024-09-13T14:29:22.048 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.048 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.048 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:29:22.048 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.346004+0000", 2024-09-13T14:29:22.048 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.2066000000000001e-05 2024-09-13T14:29:22.048 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.048 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.048 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:29:22.048 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.362798+0000", 2024-09-13T14:29:22.048 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.016794420000000001 2024-09-13T14:29:22.048 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.048 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.048 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:29:22.049 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.362826+0000", 2024-09-13T14:29:22.049 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.7390999999999999e-05 2024-09-13T14:29:22.049 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:29:22.049 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:29:22.049 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:29:22.049 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.049 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.049 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:112866 5.b 5:daf8674e:::benchmark_data_smithi102_155989_object112865:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:29:22.049 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:27:54.076365+0000", 2024-09-13T14:29:22.049 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 87.929221390999999, 2024-09-13T14:29:22.049 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.12113017600000001, 2024-09-13T14:29:22.049 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:29:22.049 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:29:22.049 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:29:22.050 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:29:22.050 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:29:22.050 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:29:22.050 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 112866 2024-09-13T14:29:22.050 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.050 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:29:22.050 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.050 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:29:22.050 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.076365+0000", 2024-09-13T14:29:22.050 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:29:22.050 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.050 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.050 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:29:22.050 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.076365+0000", 2024-09-13T14:29:22.050 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:29:22.051 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.051 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.051 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:29:22.051 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.076365+0000", 2024-09-13T14:29:22.051 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.7199999999999999e-07 2024-09-13T14:29:22.051 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.051 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.051 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:29:22.051 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.076385+0000", 2024-09-13T14:29:22.051 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.9913999999999999e-05 2024-09-13T14:29:22.051 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.051 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.051 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:29:22.051 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.076391+0000", 2024-09-13T14:29:22.052 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.3049999999999998e-06 2024-09-13T14:29:22.052 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.052 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.052 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:29:22.052 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.076396+0000", 2024-09-13T14:29:22.052 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.3830000000000002e-06 2024-09-13T14:29:22.052 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.052 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.052 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:29:22.052 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.077577+0000", 2024-09-13T14:29:22.052 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0011813310000000001 2024-09-13T14:29:22.052 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.052 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.052 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:29:22.053 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.077715+0000", 2024-09-13T14:29:22.053 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00013823000000000001 2024-09-13T14:29:22.053 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.053 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.053 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 5,7", 2024-09-13T14:29:22.053 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.077886+0000", 2024-09-13T14:29:22.053 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000170874 2024-09-13T14:29:22.053 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.053 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.054 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:29:22.054 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.086317+0000", 2024-09-13T14:29:22.054 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0084313819999999994 2024-09-13T14:29:22.055 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.055 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.055 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:29:22.055 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.112248+0000", 2024-09-13T14:29:22.055 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.025930796999999998 2024-09-13T14:29:22.055 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.055 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.055 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:29:22.055 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.195948+0000", 2024-09-13T14:29:22.055 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.083699457000000005 2024-09-13T14:29:22.055 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.055 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.055 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:29:22.055 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.196021+0000", 2024-09-13T14:29:22.056 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.3787000000000004e-05 2024-09-13T14:29:22.056 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.056 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.056 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:29:22.056 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.197495+0000", 2024-09-13T14:29:22.056 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0014732440000000001 2024-09-13T14:29:22.056 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:29:22.056 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:29:22.056 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:29:22.056 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.056 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.056 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:112869 5.b 5:d70321bf:::benchmark_data_smithi102_155989_object112868:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:29:22.056 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:27:54.079135+0000", 2024-09-13T14:29:22.056 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 87.926450858999999, 2024-09-13T14:29:22.057 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.11983437399999999, 2024-09-13T14:29:22.057 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:29:22.057 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:29:22.057 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:29:22.057 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:29:22.057 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:29:22.057 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:29:22.057 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 112869 2024-09-13T14:29:22.057 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.057 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:29:22.057 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.057 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:29:22.057 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.079135+0000", 2024-09-13T14:29:22.057 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:29:22.058 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.058 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.058 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:29:22.058 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.079135+0000", 2024-09-13T14:29:22.058 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:29:22.058 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.058 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.058 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:29:22.058 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.079137+0000", 2024-09-13T14:29:22.058 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.725e-06 2024-09-13T14:29:22.058 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.058 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.058 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:29:22.058 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.079159+0000", 2024-09-13T14:29:22.059 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.2356000000000001e-05 2024-09-13T14:29:22.059 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.059 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.059 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:29:22.059 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.079172+0000", 2024-09-13T14:29:22.059 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2587e-05 2024-09-13T14:29:22.059 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.059 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.059 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:29:22.059 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.079180+0000", 2024-09-13T14:29:22.059 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.2439999999999993e-06 2024-09-13T14:29:22.059 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.059 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.059 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:29:22.060 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.082932+0000", 2024-09-13T14:29:22.060 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.003752235 2024-09-13T14:29:22.060 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.060 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.060 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:29:22.060 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.083077+0000", 2024-09-13T14:29:22.060 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00014442599999999999 2024-09-13T14:29:22.060 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.060 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.060 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 5,7", 2024-09-13T14:29:22.060 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.083236+0000", 2024-09-13T14:29:22.060 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000159247 2024-09-13T14:29:22.060 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.060 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.061 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:29:22.061 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.086405+0000", 2024-09-13T14:29:22.061 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0031694420000000002 2024-09-13T14:29:22.061 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.061 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.061 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:29:22.061 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.112288+0000", 2024-09-13T14:29:22.061 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.025882496000000001 2024-09-13T14:29:22.061 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.061 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.061 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:29:22.061 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.197578+0000", 2024-09-13T14:29:22.061 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.085290445000000006 2024-09-13T14:29:22.061 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.062 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.062 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:29:22.062 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.197653+0000", 2024-09-13T14:29:22.062 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.4931999999999993e-05 2024-09-13T14:29:22.062 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.062 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.062 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:29:22.062 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.198969+0000", 2024-09-13T14:29:22.062 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0013162390000000001 2024-09-13T14:29:22.062 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:29:22.062 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:29:22.062 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:29:22.062 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.062 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.063 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:112877 5.b 5:d625fa27:::benchmark_data_smithi102_155989_object112876:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:29:22.063 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:27:54.087314+0000", 2024-09-13T14:29:22.063 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 87.918271688999994, 2024-09-13T14:29:22.063 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.115652147, 2024-09-13T14:29:22.063 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:29:22.063 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:29:22.063 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:29:22.063 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:29:22.063 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:29:22.063 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:29:22.063 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 112877 2024-09-13T14:29:22.063 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.063 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:29:22.063 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.063 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:29:22.064 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.087314+0000", 2024-09-13T14:29:22.064 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:29:22.064 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.064 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.064 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:29:22.064 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.087314+0000", 2024-09-13T14:29:22.064 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:29:22.064 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.064 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.064 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:29:22.064 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.087315+0000", 2024-09-13T14:29:22.064 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.9500000000000002e-07 2024-09-13T14:29:22.064 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.065 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.065 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:29:22.065 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.087333+0000", 2024-09-13T14:29:22.065 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.8349999999999999e-05 2024-09-13T14:29:22.065 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.065 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.065 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:29:22.065 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.087342+0000", 2024-09-13T14:29:22.065 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.4770000000000003e-06 2024-09-13T14:29:22.065 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.065 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.065 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:29:22.065 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.087346+0000", 2024-09-13T14:29:22.065 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.6999999999999999e-06 2024-09-13T14:29:22.066 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.066 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.066 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:29:22.066 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.087392+0000", 2024-09-13T14:29:22.066 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.5998000000000002e-05 2024-09-13T14:29:22.066 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.066 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.066 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:29:22.066 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.087529+0000", 2024-09-13T14:29:22.066 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000136868 2024-09-13T14:29:22.066 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.066 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.066 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 5,7", 2024-09-13T14:29:22.066 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.087704+0000", 2024-09-13T14:29:22.066 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00017465699999999999 2024-09-13T14:29:22.067 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.067 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.067 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:29:22.067 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.090517+0000", 2024-09-13T14:29:22.067 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0028129180000000002 2024-09-13T14:29:22.067 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.067 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.067 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:29:22.067 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.112319+0000", 2024-09-13T14:29:22.067 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.021802027000000002 2024-09-13T14:29:22.067 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.067 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.067 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:29:22.068 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.201157+0000", 2024-09-13T14:29:22.068 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.088838557999999998 2024-09-13T14:29:22.068 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.068 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.068 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:29:22.068 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.201216+0000", 2024-09-13T14:29:22.068 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.8804000000000001e-05 2024-09-13T14:29:22.068 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.068 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.068 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:29:22.068 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.202966+0000", 2024-09-13T14:29:22.068 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.001750195 2024-09-13T14:29:22.068 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:29:22.069 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:29:22.069 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:29:22.069 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.069 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.069 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:112879 5.b 5:d8d74457:::benchmark_data_smithi102_155989_object112878:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:29:22.069 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:27:54.093629+0000", 2024-09-13T14:29:22.069 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 87.911956989999993, 2024-09-13T14:29:22.069 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.110882463, 2024-09-13T14:29:22.069 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:29:22.069 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:29:22.069 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:29:22.069 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:29:22.069 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:29:22.069 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:29:22.070 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 112879 2024-09-13T14:29:22.070 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.070 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:29:22.070 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.070 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:29:22.070 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.093629+0000", 2024-09-13T14:29:22.070 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:29:22.070 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.070 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.070 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:29:22.070 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.093629+0000", 2024-09-13T14:29:22.070 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:29:22.070 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.070 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.071 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:29:22.071 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.093630+0000", 2024-09-13T14:29:22.071 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1489999999999999e-06 2024-09-13T14:29:22.071 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.071 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.071 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:29:22.071 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.093648+0000", 2024-09-13T14:29:22.071 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.8099999999999999e-05 2024-09-13T14:29:22.071 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.071 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.071 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:29:22.071 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.093658+0000", 2024-09-13T14:29:22.071 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.5719999999999992e-06 2024-09-13T14:29:22.071 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.072 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.072 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:29:22.072 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.093663+0000", 2024-09-13T14:29:22.072 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.882e-06 2024-09-13T14:29:22.072 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.072 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.072 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:29:22.072 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.093732+0000", 2024-09-13T14:29:22.072 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.8986000000000007e-05 2024-09-13T14:29:22.072 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.072 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.072 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:29:22.072 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.093840+0000", 2024-09-13T14:29:22.072 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000108448 2024-09-13T14:29:22.072 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.073 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.073 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 5,7", 2024-09-13T14:29:22.073 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.093960+0000", 2024-09-13T14:29:22.073 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000119764 2024-09-13T14:29:22.073 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.073 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.073 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:29:22.073 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.099419+0000", 2024-09-13T14:29:22.073 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0054587689999999996 2024-09-13T14:29:22.073 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.073 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.073 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:29:22.073 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.115925+0000", 2024-09-13T14:29:22.073 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.016506460000000001 2024-09-13T14:29:22.074 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.074 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.074 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:29:22.074 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.203105+0000", 2024-09-13T14:29:22.074 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.087180013000000001 2024-09-13T14:29:22.074 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.074 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.074 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:29:22.074 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.203177+0000", 2024-09-13T14:29:22.074 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.2250999999999997e-05 2024-09-13T14:29:22.074 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.074 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.074 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:29:22.074 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.204511+0000", 2024-09-13T14:29:22.075 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.001334069 2024-09-13T14:29:22.075 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:29:22.075 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:29:22.075 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:29:22.075 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.075 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.075 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:112880 5.b 5:d2a66a20:::benchmark_data_smithi102_155989_object112879:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:29:22.075 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:27:54.093777+0000", 2024-09-13T14:29:22.075 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 87.911808801000006, 2024-09-13T14:29:22.075 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.11214436699999999, 2024-09-13T14:29:22.075 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:29:22.075 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:29:22.075 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:29:22.075 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:29:22.076 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:29:22.076 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:29:22.076 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 112880 2024-09-13T14:29:22.076 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.076 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:29:22.076 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.076 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:29:22.076 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.093777+0000", 2024-09-13T14:29:22.076 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:29:22.076 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.076 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.076 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:29:22.076 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.093777+0000", 2024-09-13T14:29:22.076 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:29:22.077 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.077 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.077 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:29:22.077 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.093778+0000", 2024-09-13T14:29:22.077 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.5199999999999996e-07 2024-09-13T14:29:22.077 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.077 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.077 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:29:22.077 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.093797+0000", 2024-09-13T14:29:22.077 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.8828999999999999e-05 2024-09-13T14:29:22.077 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.077 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.077 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:29:22.077 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.093803+0000", 2024-09-13T14:29:22.077 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.2179999999999996e-06 2024-09-13T14:29:22.078 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.078 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.078 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:29:22.078 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.093807+0000", 2024-09-13T14:29:22.078 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.788e-06 2024-09-13T14:29:22.078 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.078 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.078 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:29:22.078 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.096849+0000", 2024-09-13T14:29:22.078 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0030418849999999998 2024-09-13T14:29:22.078 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.078 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.078 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:29:22.078 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.096972+0000", 2024-09-13T14:29:22.079 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00012337399999999999 2024-09-13T14:29:22.079 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.079 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.079 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 5,7", 2024-09-13T14:29:22.079 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.097086+0000", 2024-09-13T14:29:22.079 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000114218 2024-09-13T14:29:22.079 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.079 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.079 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:29:22.079 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.099560+0000", 2024-09-13T14:29:22.079 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0024740449999999998 2024-09-13T14:29:22.079 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.079 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.079 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:29:22.079 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.192059+0000", 2024-09-13T14:29:22.080 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.092498884000000003 2024-09-13T14:29:22.080 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.080 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.080 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:29:22.080 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.204583+0000", 2024-09-13T14:29:22.080 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.012524186 2024-09-13T14:29:22.080 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.080 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.080 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:29:22.080 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.204649+0000", 2024-09-13T14:29:22.080 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.5692999999999996e-05 2024-09-13T14:29:22.080 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.080 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.080 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:29:22.081 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.205921+0000", 2024-09-13T14:29:22.081 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.001272495 2024-09-13T14:29:22.081 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:29:22.081 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:29:22.081 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:29:22.081 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.081 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.081 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:116679 5.8 5:190cce24:::benchmark_data_smithi102_155989_object116678:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:29:22.081 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:27:59.371873+0000", 2024-09-13T14:29:22.081 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 82.633712845999995, 2024-09-13T14:29:22.081 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.121846316, 2024-09-13T14:29:22.081 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:29:22.081 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:29:22.082 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:29:22.082 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:29:22.082 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:29:22.082 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:29:22.082 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 116679 2024-09-13T14:29:22.082 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.082 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:29:22.082 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.082 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:29:22.082 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.371873+0000", 2024-09-13T14:29:22.082 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:29:22.082 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.082 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.082 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:29:22.082 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.371873+0000", 2024-09-13T14:29:22.083 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:29:22.083 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.083 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.083 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:29:22.083 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.371875+0000", 2024-09-13T14:29:22.083 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.567e-06 2024-09-13T14:29:22.083 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.083 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.083 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:29:22.083 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.371899+0000", 2024-09-13T14:29:22.083 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.4326000000000001e-05 2024-09-13T14:29:22.083 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.083 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.083 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:29:22.084 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.371914+0000", 2024-09-13T14:29:22.084 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4865000000000001e-05 2024-09-13T14:29:22.084 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.084 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.084 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:29:22.084 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.371922+0000", 2024-09-13T14:29:22.084 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.3850000000000002e-06 2024-09-13T14:29:22.084 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.084 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.084 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:29:22.084 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.371989+0000", 2024-09-13T14:29:22.084 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.7244999999999997e-05 2024-09-13T14:29:22.084 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.084 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.085 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:29:22.085 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.372167+0000", 2024-09-13T14:29:22.085 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000177861 2024-09-13T14:29:22.085 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.085 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.085 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,5", 2024-09-13T14:29:22.085 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.372358+0000", 2024-09-13T14:29:22.085 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00019059099999999999 2024-09-13T14:29:22.085 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.085 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.085 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:29:22.085 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.375821+0000", 2024-09-13T14:29:22.085 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0034634380000000001 2024-09-13T14:29:22.085 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.086 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.086 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:29:22.086 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.382391+0000", 2024-09-13T14:29:22.086 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0065692340000000002 2024-09-13T14:29:22.086 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.086 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.086 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:29:22.086 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.491965+0000", 2024-09-13T14:29:22.086 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.10957447200000001 2024-09-13T14:29:22.086 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.086 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.086 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:29:22.086 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.492009+0000", 2024-09-13T14:29:22.086 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.4187999999999998e-05 2024-09-13T14:29:22.086 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.087 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.087 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:29:22.087 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.493719+0000", 2024-09-13T14:29:22.087 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.001710144 2024-09-13T14:29:22.087 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:29:22.087 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:29:22.087 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:29:22.087 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.087 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.087 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:116682 5.8 5:11aa58c2:::benchmark_data_smithi102_155989_object116681:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:29:22.087 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:27:59.379192+0000", 2024-09-13T14:29:22.087 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 82.62639369, 2024-09-13T14:29:22.087 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.11629210199999999, 2024-09-13T14:29:22.087 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:29:22.088 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:29:22.088 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:29:22.088 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:29:22.088 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:29:22.088 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:29:22.088 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 116682 2024-09-13T14:29:22.088 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.088 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:29:22.088 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.088 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:29:22.088 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.379192+0000", 2024-09-13T14:29:22.088 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:29:22.088 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.088 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.089 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:29:22.089 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.379192+0000", 2024-09-13T14:29:22.089 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:29:22.089 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.089 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.089 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:29:22.089 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.379194+0000", 2024-09-13T14:29:22.089 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.5600000000000001e-06 2024-09-13T14:29:22.089 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.089 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.089 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:29:22.089 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.379220+0000", 2024-09-13T14:29:22.089 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.5782000000000002e-05 2024-09-13T14:29:22.089 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.090 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.090 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:29:22.090 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.379233+0000", 2024-09-13T14:29:22.090 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3799e-05 2024-09-13T14:29:22.090 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.090 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.090 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:29:22.090 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.379241+0000", 2024-09-13T14:29:22.090 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.3880000000000002e-06 2024-09-13T14:29:22.090 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.090 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.090 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:29:22.090 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.379290+0000", 2024-09-13T14:29:22.090 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.8825000000000001e-05 2024-09-13T14:29:22.090 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.091 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.091 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:29:22.091 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.379420+0000", 2024-09-13T14:29:22.091 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00013070900000000001 2024-09-13T14:29:22.091 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.091 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.091 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,5", 2024-09-13T14:29:22.091 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.379601+0000", 2024-09-13T14:29:22.091 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00018065999999999999 2024-09-13T14:29:22.091 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.091 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.091 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:29:22.091 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.382467+0000", 2024-09-13T14:29:22.091 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0028658009999999999 2024-09-13T14:29:22.092 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.092 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.092 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:29:22.092 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.385031+0000", 2024-09-13T14:29:22.092 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0025645640000000001 2024-09-13T14:29:22.092 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.092 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.092 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:29:22.092 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.493786+0000", 2024-09-13T14:29:22.092 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.108754976 2024-09-13T14:29:22.092 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.092 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.092 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:29:22.092 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.493858+0000", 2024-09-13T14:29:22.092 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.2057000000000005e-05 2024-09-13T14:29:22.093 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.093 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.093 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:29:22.093 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.495484+0000", 2024-09-13T14:29:22.093 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0016259810000000001 2024-09-13T14:29:22.093 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:29:22.093 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:29:22.093 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:29:22.093 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.093 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.093 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:116688 5.8 5:11d64788:::benchmark_data_smithi102_155989_object116687:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:29:22.093 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:27:59.385784+0000", 2024-09-13T14:29:22.093 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 82.619801691000006, 2024-09-13T14:29:22.093 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.111592976, 2024-09-13T14:29:22.094 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:29:22.094 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:29:22.094 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:29:22.094 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:29:22.094 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:29:22.094 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:29:22.094 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 116688 2024-09-13T14:29:22.094 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.094 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:29:22.094 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.094 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:29:22.094 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.385784+0000", 2024-09-13T14:29:22.094 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:29:22.094 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.095 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.095 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:29:22.095 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.385784+0000", 2024-09-13T14:29:22.095 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:29:22.095 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.095 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.095 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:29:22.095 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.385785+0000", 2024-09-13T14:29:22.095 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1829999999999999e-06 2024-09-13T14:29:22.095 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.095 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.095 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:29:22.095 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.385804+0000", 2024-09-13T14:29:22.095 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.8485999999999999e-05 2024-09-13T14:29:22.095 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.096 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.096 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:29:22.096 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.385813+0000", 2024-09-13T14:29:22.096 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.8689999999999993e-06 2024-09-13T14:29:22.096 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.096 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.096 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:29:22.096 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.385818+0000", 2024-09-13T14:29:22.096 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.8219999999999997e-06 2024-09-13T14:29:22.096 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.096 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.096 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:29:22.096 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.385875+0000", 2024-09-13T14:29:22.096 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.7768e-05 2024-09-13T14:29:22.097 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.097 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.097 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:29:22.097 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.386024+0000", 2024-09-13T14:29:22.097 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00014836999999999999 2024-09-13T14:29:22.097 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.097 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.097 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,5", 2024-09-13T14:29:22.097 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.386192+0000", 2024-09-13T14:29:22.097 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00016833500000000001 2024-09-13T14:29:22.097 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.097 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.097 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:29:22.097 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.389274+0000", 2024-09-13T14:29:22.098 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0030815410000000001 2024-09-13T14:29:22.098 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.098 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.098 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:29:22.098 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.488242+0000", 2024-09-13T14:29:22.098 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.098968889000000004 2024-09-13T14:29:22.098 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.098 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.098 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:29:22.099 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.495539+0000", 2024-09-13T14:29:22.099 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0072964249999999996 2024-09-13T14:29:22.100 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.100 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.100 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:29:22.100 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.495645+0000", 2024-09-13T14:29:22.100 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000106155 2024-09-13T14:29:22.100 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.100 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.100 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:29:22.100 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.497377+0000", 2024-09-13T14:29:22.100 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0017321330000000001 2024-09-13T14:29:22.100 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:29:22.100 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:29:22.100 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:29:22.100 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.101 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.101 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:116689 5.3 5:cf540b53:::benchmark_data_smithi102_155989_object116688:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:29:22.101 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:27:59.389614+0000", 2024-09-13T14:29:22.101 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 82.615971690999999, 2024-09-13T14:29:22.101 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.10243406300000001, 2024-09-13T14:29:22.101 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:29:22.101 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:29:22.101 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:29:22.101 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:29:22.101 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:29:22.101 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:29:22.101 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 116689 2024-09-13T14:29:22.101 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.101 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:29:22.102 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.102 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:29:22.102 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.389614+0000", 2024-09-13T14:29:22.102 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:29:22.102 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.102 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.102 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:29:22.102 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.389614+0000", 2024-09-13T14:29:22.102 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:29:22.102 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.102 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.102 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:29:22.102 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.389616+0000", 2024-09-13T14:29:22.102 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.2699999999999999e-06 2024-09-13T14:29:22.103 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.103 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.103 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:29:22.103 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.389641+0000", 2024-09-13T14:29:22.103 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.4964999999999999e-05 2024-09-13T14:29:22.103 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.103 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.103 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:29:22.103 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.389651+0000", 2024-09-13T14:29:22.103 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.8600000000000005e-06 2024-09-13T14:29:22.103 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.103 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.103 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:29:22.103 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.389658+0000", 2024-09-13T14:29:22.104 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.3210000000000001e-06 2024-09-13T14:29:22.104 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.104 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.104 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:29:22.104 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.389693+0000", 2024-09-13T14:29:22.104 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.5485999999999999e-05 2024-09-13T14:29:22.104 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.104 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.104 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:29:22.104 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.389779+0000", 2024-09-13T14:29:22.104 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.6112000000000005e-05 2024-09-13T14:29:22.104 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.104 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.104 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:29:22.105 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.389883+0000", 2024-09-13T14:29:22.105 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00010373400000000001 2024-09-13T14:29:22.105 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.105 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.105 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:29:22.105 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.392587+0000", 2024-09-13T14:29:22.105 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0027037799999999998 2024-09-13T14:29:22.105 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.105 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.105 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:29:22.105 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.488185+0000", 2024-09-13T14:29:22.105 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.095598163999999999 2024-09-13T14:29:22.105 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.105 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.105 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:29:22.106 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.491795+0000", 2024-09-13T14:29:22.106 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0036105460000000001 2024-09-13T14:29:22.106 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.106 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.106 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:29:22.106 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.491997+0000", 2024-09-13T14:29:22.106 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00020154799999999999 2024-09-13T14:29:22.106 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.106 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.106 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:29:22.106 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.492048+0000", 2024-09-13T14:29:22.106 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.1276999999999997e-05 2024-09-13T14:29:22.106 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:29:22.106 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:29:22.107 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:29:22.107 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.107 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.107 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:116692 5.8 5:18e6722a:::benchmark_data_smithi102_155989_object116691:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:29:22.107 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:27:59.391197+0000", 2024-09-13T14:29:22.107 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 82.614388796, 2024-09-13T14:29:22.107 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.107969913, 2024-09-13T14:29:22.107 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:29:22.107 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:29:22.107 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:29:22.107 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:29:22.107 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:29:22.107 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:29:22.107 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 116692 2024-09-13T14:29:22.108 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.108 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:29:22.108 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.108 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:29:22.108 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.391197+0000", 2024-09-13T14:29:22.108 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:29:22.108 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.108 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.108 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:29:22.108 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.391197+0000", 2024-09-13T14:29:22.108 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:29:22.108 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.109 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.109 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:29:22.109 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.391198+0000", 2024-09-13T14:29:22.109 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.079e-06 2024-09-13T14:29:22.109 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.109 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.109 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:29:22.109 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.391219+0000", 2024-09-13T14:29:22.109 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.105e-05 2024-09-13T14:29:22.109 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.109 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.109 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:29:22.110 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.391231+0000", 2024-09-13T14:29:22.110 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1965e-05 2024-09-13T14:29:22.110 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.110 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.110 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:29:22.110 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.391239+0000", 2024-09-13T14:29:22.110 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.3180000000000001e-06 2024-09-13T14:29:22.110 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.110 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.110 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:29:22.110 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.391283+0000", 2024-09-13T14:29:22.110 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.4617000000000003e-05 2024-09-13T14:29:22.110 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.110 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.111 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:29:22.111 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.391416+0000", 2024-09-13T14:29:22.111 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00013308699999999999 2024-09-13T14:29:22.111 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.111 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.111 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,5", 2024-09-13T14:29:22.111 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.391537+0000", 2024-09-13T14:29:22.111 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000120347 2024-09-13T14:29:22.111 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.111 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.111 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:29:22.111 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.488049+0000", 2024-09-13T14:29:22.111 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.096512403999999996 2024-09-13T14:29:22.111 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.112 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.112 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:29:22.112 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.488285+0000", 2024-09-13T14:29:22.112 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00023624599999999999 2024-09-13T14:29:22.112 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.112 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.112 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:29:22.112 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.497452+0000", 2024-09-13T14:29:22.112 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0091672590000000005 2024-09-13T14:29:22.112 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.112 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.112 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:29:22.112 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.497518+0000", 2024-09-13T14:29:22.113 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.5140000000000003e-05 2024-09-13T14:29:22.113 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.113 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.113 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:29:22.113 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.499167+0000", 2024-09-13T14:29:22.113 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.001649401 2024-09-13T14:29:22.113 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:29:22.113 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:29:22.113 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:29:22.113 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.113 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.113 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.44546.0:116696 5.0 e131/116 5:0dbc8dea:::benchmark_data_smithi102_155989_object116695:head v 131'7377, mlcod=131'7375)", 2024-09-13T14:29:22.113 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:27:59.396027+0000", 2024-09-13T14:29:22.113 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 82.609559352000005, 2024-09-13T14:29:22.114 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.103207778, 2024-09-13T14:29:22.114 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:29:22.114 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:29:22.114 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:29:22.114 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:29:22.114 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.114 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:29:22.114 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.396027+0000", 2024-09-13T14:29:22.114 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:29:22.114 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.114 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.114 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:29:22.114 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.396027+0000", 2024-09-13T14:29:22.114 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:29:22.115 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.115 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.115 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:29:22.115 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.396027+0000", 2024-09-13T14:29:22.115 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.1200000000000001e-07 2024-09-13T14:29:22.115 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.115 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.115 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:29:22.115 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.396047+0000", 2024-09-13T14:29:22.115 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.9490000000000001e-05 2024-09-13T14:29:22.115 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.115 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.115 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:29:22.115 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.488024+0000", 2024-09-13T14:29:22.116 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.091977391000000006 2024-09-13T14:29:22.116 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.116 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.116 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:29:22.116 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.488034+0000", 2024-09-13T14:29:22.116 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.9639999999999998e-06 2024-09-13T14:29:22.116 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.116 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.116 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:29:22.116 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.488083+0000", 2024-09-13T14:29:22.116 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.8575999999999997e-05 2024-09-13T14:29:22.116 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.116 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.117 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:29:22.117 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.488105+0000", 2024-09-13T14:29:22.117 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.1827999999999998e-05 2024-09-13T14:29:22.117 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.117 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.117 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:29:22.117 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.499179+0000", 2024-09-13T14:29:22.117 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.011073796 2024-09-13T14:29:22.117 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.117 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.117 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:29:22.117 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.499234+0000", 2024-09-13T14:29:22.117 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.5821e-05 2024-09-13T14:29:22.117 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:29:22.118 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:29:22.118 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:29:22.118 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.118 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.118 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:150972 5.8 5:140af814:::benchmark_data_smithi102_155989_object33054:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T14:29:22.118 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:28:46.725951+0000", 2024-09-13T14:29:22.118 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 35.279634725999998, 2024-09-13T14:29:22.118 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.10818009100000001, 2024-09-13T14:29:22.118 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:29:22.118 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:29:22.118 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:29:22.118 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:29:22.118 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:29:22.118 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:29:22.119 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 150972 2024-09-13T14:29:22.119 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.119 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:29:22.119 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.119 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:29:22.119 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.725951+0000", 2024-09-13T14:29:22.119 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:29:22.119 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.119 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.119 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:29:22.119 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.725951+0000", 2024-09-13T14:29:22.119 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:29:22.119 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.119 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.120 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:29:22.120 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.725952+0000", 2024-09-13T14:29:22.120 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.3900000000000004e-07 2024-09-13T14:29:22.120 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.120 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.120 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:29:22.120 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.725956+0000", 2024-09-13T14:29:22.120 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.126e-06 2024-09-13T14:29:22.120 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.120 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.120 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:29:22.120 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.725967+0000", 2024-09-13T14:29:22.120 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0413e-05 2024-09-13T14:29:22.120 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.121 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.121 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:29:22.121 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.725971+0000", 2024-09-13T14:29:22.121 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.826e-06 2024-09-13T14:29:22.121 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.121 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.121 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:29:22.121 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.727890+0000", 2024-09-13T14:29:22.121 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.001918283 2024-09-13T14:29:22.121 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.121 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.121 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:29:22.121 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.728037+0000", 2024-09-13T14:29:22.121 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00014736099999999999 2024-09-13T14:29:22.122 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.122 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.122 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,5", 2024-09-13T14:29:22.122 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.728192+0000", 2024-09-13T14:29:22.122 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00015504200000000001 2024-09-13T14:29:22.122 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.122 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.122 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:29:22.122 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.732463+0000", 2024-09-13T14:29:22.122 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0042705449999999997 2024-09-13T14:29:22.122 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.122 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.122 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:29:22.122 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.732485+0000", 2024-09-13T14:29:22.122 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.2257000000000001e-05 2024-09-13T14:29:22.123 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.123 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.123 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:29:22.124 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.832399+0000", 2024-09-13T14:29:22.124 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.099914441000000007 2024-09-13T14:29:22.124 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.124 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.124 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:29:22.124 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.832585+0000", 2024-09-13T14:29:22.124 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00018537600000000001 2024-09-13T14:29:22.124 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.125 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.125 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:29:22.125 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.834131+0000", 2024-09-13T14:29:22.125 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0015465819999999999 2024-09-13T14:29:22.125 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:29:22.125 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:29:22.125 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:29:22.125 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.125 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.125 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:150980 5.8 5:1a046fd0:::benchmark_data_smithi102_155989_object33062:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T14:29:22.125 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:28:46.733592+0000", 2024-09-13T14:29:22.125 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 35.271994006, 2024-09-13T14:29:22.125 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.102963253, 2024-09-13T14:29:22.125 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:29:22.126 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:29:22.126 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:29:22.126 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:29:22.126 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:29:22.126 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:29:22.126 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 150980 2024-09-13T14:29:22.126 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.126 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:29:22.126 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.126 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:29:22.126 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.733592+0000", 2024-09-13T14:29:22.126 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:29:22.126 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.126 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.127 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:29:22.127 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.733592+0000", 2024-09-13T14:29:22.127 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:29:22.127 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.127 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.127 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:29:22.127 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.733592+0000", 2024-09-13T14:29:22.127 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.8100000000000003e-07 2024-09-13T14:29:22.127 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.127 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.127 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:29:22.127 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.733595+0000", 2024-09-13T14:29:22.127 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.6410000000000002e-06 2024-09-13T14:29:22.127 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.128 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.128 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:29:22.128 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.733605+0000", 2024-09-13T14:29:22.128 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.3789999999999995e-06 2024-09-13T14:29:22.128 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.128 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.128 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:29:22.128 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.733609+0000", 2024-09-13T14:29:22.128 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.8810000000000004e-06 2024-09-13T14:29:22.128 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.128 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.128 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:29:22.128 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.736636+0000", 2024-09-13T14:29:22.129 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0030264689999999999 2024-09-13T14:29:22.129 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.129 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.129 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:29:22.129 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.736798+0000", 2024-09-13T14:29:22.129 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000162287 2024-09-13T14:29:22.129 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.129 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.129 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,5", 2024-09-13T14:29:22.129 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.736971+0000", 2024-09-13T14:29:22.129 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000173098 2024-09-13T14:29:22.129 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.129 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.129 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:29:22.130 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.739877+0000", 2024-09-13T14:29:22.130 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.002905318 2024-09-13T14:29:22.130 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.130 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.130 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:29:22.130 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.743786+0000", 2024-09-13T14:29:22.130 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.003909839 2024-09-13T14:29:22.130 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.130 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.130 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:29:22.130 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.835482+0000", 2024-09-13T14:29:22.130 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.091695862000000003 2024-09-13T14:29:22.130 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.130 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.130 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:29:22.131 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.835542+0000", 2024-09-13T14:29:22.131 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.9478999999999998e-05 2024-09-13T14:29:22.131 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:29:22.131 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:29:22.131 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:29:22.131 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.836555+0000", 2024-09-13T14:29:22.131 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.001013419 2024-09-13T14:29:22.131 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:29:22.131 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:29:22.131 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:29:22.131 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:29:22.131 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:29:22.131 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:29:22.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:22.132 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:22.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:22.234 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:29:22.277 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:22.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:22.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:29:22.379 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:22.405 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:22.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:22.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:22.507 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:22.551 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:22.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:22.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:22.652 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:22.696 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:22.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:22.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:22.797 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:22.820 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:22.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:22.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:22.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:22 smithi102 ceph-mon[110183]: pgmap v415: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.9 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 485 op/s; 20564/192717 objects degraded (10.671%); 3350/192717 objects misplaced (1.738%); 2.1 MiB/s, 33 objects/s recovering 2024-09-13T14:29:22.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:22 smithi102 ceph-mon[115269]: pgmap v415: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.9 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 485 op/s; 20564/192717 objects degraded (10.671%); 3350/192717 objects misplaced (1.738%); 2.1 MiB/s, 33 objects/s recovering 2024-09-13T14:29:22.923 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:22.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:22.949 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:22.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:22.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:22 smithi195 ceph-mon[104262]: pgmap v415: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.9 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 485 op/s; 20564/192717 objects degraded (10.671%); 3350/192717 objects misplaced (1.738%); 2.1 MiB/s, 33 objects/s recovering 2024-09-13T14:29:23.050 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:23.078 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:23.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:23.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:23.180 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:23.223 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:23.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:23.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:23.324 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:29:23.348 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:23.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:23.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:29:23.450 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:23.538 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:29:23.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:23.545 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:23.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:23.646 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:23.682 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:23.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:23.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:23.784 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:23.825 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:23.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:23.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:23.927 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:23.965 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:23.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:23.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:24.067 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:24.112 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:24.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:24.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:24.214 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:24.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:24.251 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:24.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:24.352 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:24.378 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:24.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:24.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:24.448 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:29:24.480 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:24.503 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:24.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:24.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:24.606 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:24.649 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:24.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:24.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:24.751 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:24.776 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:24.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:24.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:24.878 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:29:24.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:24 smithi102 ceph-mon[110183]: pgmap v416: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.7 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 879 op/s; 20759/182556 objects degraded (11.371%); 2876/182556 objects misplaced (1.575%); 4.3 MiB/s, 68 objects/s recovering 2024-09-13T14:29:24.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:24 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 20759/182556 objects degraded (11.371%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:29:24.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:24 smithi102 ceph-mon[115269]: pgmap v416: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.7 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 879 op/s; 20759/182556 objects degraded (11.371%); 2876/182556 objects misplaced (1.575%); 4.3 MiB/s, 68 objects/s recovering 2024-09-13T14:29:24.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:24 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 20759/182556 objects degraded (11.371%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:29:24.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:24.920 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:24.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:29:24.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:24 smithi195 ceph-mon[104262]: pgmap v416: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.7 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 879 op/s; 20759/182556 objects degraded (11.371%); 2876/182556 objects misplaced (1.575%); 4.3 MiB/s, 68 objects/s recovering 2024-09-13T14:29:24.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:24 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 20759/182556 objects degraded (11.371%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:29:25.021 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:25.058 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:25.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:25.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:25.160 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:29:25.182 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:25.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:25.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:29:25.284 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:25.318 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:25.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:25.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:25.420 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:25.459 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:25.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:25.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:25.561 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:29:25.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:25.606 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:25.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:29:25.707 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:25.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:25.751 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:25.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:25.852 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:25.856 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T14:29:25.891 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:25.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:25.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:25.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:25 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:29:25.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:25 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:29:25.993 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:25.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:25 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:29:26.028 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:26.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:26.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:26.130 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:26.172 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:26.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:26.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:26.274 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:26.303 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:26.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:26.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:26.405 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:29:26.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:26.451 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:26.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:29:26.550 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:29:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:29:26] "GET /metrics HTTP/1.1" 200 36537 "" "Prometheus/2.51.0" 2024-09-13T14:29:26.553 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:26.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:26.604 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:26.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:26.705 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:26.741 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:26.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:26.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:26.812 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:26 smithi102 ceph-mon[110183]: pgmap v417: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.7 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 747 op/s; 20759/181911 objects degraded (11.412%); 2876/181911 objects misplaced (1.581%); 3.5 MiB/s, 56 objects/s recovering 2024-09-13T14:29:26.812 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:26 smithi102 ceph-mon[115269]: pgmap v417: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.7 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 747 op/s; 20759/181911 objects degraded (11.412%); 2876/181911 objects misplaced (1.581%); 3.5 MiB/s, 56 objects/s recovering 2024-09-13T14:29:26.843 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:26.870 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:26.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:26.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:26.877 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:26 smithi195 ceph-mon[104262]: pgmap v417: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.7 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 747 op/s; 20759/181911 objects degraded (11.412%); 2876/181911 objects misplaced (1.581%); 3.5 MiB/s, 56 objects/s recovering 2024-09-13T14:29:26.972 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:26.995 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:26.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:26.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:27.097 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:27.120 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:27.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:27.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:27.222 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:27.260 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:27.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:27.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:27.362 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:27.385 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:27.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:27.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:27.396 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:29:27.397 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:29:27.487 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:27.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:27.530 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:27.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:27.631 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:27.668 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:27.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:27.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:27.770 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:27.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:27.813 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:27.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:27.914 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:27.952 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:27.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:27.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:28.055 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:28.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:28.092 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:28.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:28.193 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:28.218 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:28.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:28.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:28.320 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:29:28.346 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:28.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:28.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:29:28.448 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:28.483 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:28.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:28.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:28.585 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:28.627 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:28.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:28.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:28.677 INFO:teuthology.orchestra.run.smithi102.stderr:reweighted osd.6 to 1 (10000) 2024-09-13T14:29:28.729 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:29:28.773 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:28 smithi102 ceph-mon[110183]: pgmap v418: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.5 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 715 op/s; 20899/171933 objects degraded (12.155%); 2454/171933 objects misplaced (1.427%); 3.5 MiB/s, 56 objects/s recovering 2024-09-13T14:29:28.773 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:28 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2374317408' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:29:28.773 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:28 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:29:28.773 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:28 smithi102 ceph-mon[115269]: pgmap v418: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.5 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 715 op/s; 20899/171933 objects degraded (12.155%); 2454/171933 objects misplaced (1.427%); 3.5 MiB/s, 56 objects/s recovering 2024-09-13T14:29:28.774 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:28 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2374317408' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:29:28.774 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:28 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:29:28.776 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:28.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:28.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:29:28.878 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:28.901 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:28.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:28.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:28.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:28 smithi195 ceph-mon[104262]: pgmap v418: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.5 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 715 op/s; 20899/171933 objects degraded (12.155%); 2454/171933 objects misplaced (1.427%); 3.5 MiB/s, 56 objects/s recovering 2024-09-13T14:29:28.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:28 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2374317408' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:29:28.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:28 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:29:29.003 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:29.045 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:29.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:29.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:29.147 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:29.173 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:29.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:29.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:29.274 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:29.315 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:29.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:29.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:29.417 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:29:29.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:29.453 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:29.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:29:29.554 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:29.591 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:29.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:29.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:29.693 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:29.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:29.731 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:29.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:29.832 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:29.857 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:29.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:29.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:29.959 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:29.983 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:29.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:29.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:29.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:29 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T14:29:29.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:29 smithi195 ceph-mon[104262]: osdmap e145: 8 total, 8 up, 6 in 2024-09-13T14:29:29.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:29 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 20899/171933 objects degraded (12.155%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:29:30.078 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:29 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T14:29:30.078 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:29 smithi102 ceph-mon[110183]: osdmap e145: 8 total, 8 up, 6 in 2024-09-13T14:29:30.078 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:29 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 20899/171933 objects degraded (12.155%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:29:30.079 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:29 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T14:29:30.079 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:29 smithi102 ceph-mon[115269]: osdmap e145: 8 total, 8 up, 6 in 2024-09-13T14:29:30.079 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:29 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 20899/171933 objects degraded (12.155%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:29:30.084 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:30.109 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:30.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:30.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:30.211 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:30.255 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:30.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:30.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:30.357 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:30.385 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:30.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:30.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:30.487 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:30.527 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:30.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:30.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:30.629 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:30.665 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:30.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:30.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:30.693 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is unset 2024-09-13T14:29:30.766 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:30.792 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:30.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:30.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:30.894 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:30.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:30 smithi102 ceph-mon[110183]: pgmap v420: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.5 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 715 op/s; 20899/171249 objects degraded (12.204%); 2454/171249 objects misplaced (1.433%); 3.5 MiB/s, 56 objects/s recovering 2024-09-13T14:29:30.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:30 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2438263907' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:29:30.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:30 smithi102 ceph-mon[115269]: pgmap v420: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.5 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 715 op/s; 20899/171249 objects degraded (12.204%); 2454/171249 objects misplaced (1.433%); 3.5 MiB/s, 56 objects/s recovering 2024-09-13T14:29:30.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:30 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2438263907' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:29:30.937 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:30.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:30.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:30.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:30 smithi195 ceph-mon[104262]: pgmap v420: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.5 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 715 op/s; 20899/171249 objects degraded (12.204%); 2454/171249 objects misplaced (1.433%); 3.5 MiB/s, 56 objects/s recovering 2024-09-13T14:29:30.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:30 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2438263907' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:29:31.039 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:29:31.080 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:31.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:31.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:29:31.182 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:29:31.210 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:31.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:31.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:29:31.312 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:31.348 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:31.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:31.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:31.451 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:31.494 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:31.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:31.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:31.587 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:29:31.588 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:29:31.588 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:29:31.588 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:29:31.588 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:29:31.588 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:29:31.588 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:29:31.588 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:29:31.588 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:29:31.589 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:29:31.589 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:29:31.589 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:29:31.589 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:29:31.589 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:29:31.589 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:29:31.589 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:29:31.596 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:29:31.625 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:31.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:31.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:29:31.727 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:29:31.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:31.772 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:31.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:29:31.873 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:31.911 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:31.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:31.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:31.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:31 smithi102 ceph-mon[115269]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:29:31.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:31 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2438263907' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:29:31.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:31 smithi102 ceph-mon[115269]: osdmap e146: 8 total, 8 up, 6 in 2024-09-13T14:29:31.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:31 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/625364827' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:29:31.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:31 smithi102 ceph-mon[110183]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:29:31.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:31 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2438263907' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:29:31.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:31 smithi102 ceph-mon[110183]: osdmap e146: 8 total, 8 up, 6 in 2024-09-13T14:29:31.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:31 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/625364827' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:29:31.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:31 smithi195 ceph-mon[104262]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:29:31.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:31 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2438263907' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:29:31.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:31 smithi195 ceph-mon[104262]: osdmap e146: 8 total, 8 up, 6 in 2024-09-13T14:29:31.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:31 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/625364827' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:29:32.013 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:32.037 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:32.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:32.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:32.139 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:32.166 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:32.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:32.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:32.268 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:32.294 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:32.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:32.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:32.398 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:32.420 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:32.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:32.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:32.521 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:32.564 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:32.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:32.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:32.616 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-13T14:29:32.666 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:29:32.691 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:32.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:32.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:29:32.793 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:32.817 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:32.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:32.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:32.847 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:32 smithi102 ceph-mon[110183]: pgmap v422: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.5 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 470 op/s; 20899/171249 objects degraded (12.204%); 2454/171249 objects misplaced (1.433%); 2.2 MiB/s, 35 objects/s recovering 2024-09-13T14:29:32.847 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:32 smithi102 ceph-mon[115269]: pgmap v422: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.5 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 470 op/s; 20899/171249 objects degraded (12.204%); 2454/171249 objects misplaced (1.433%); 2.2 MiB/s, 35 objects/s recovering 2024-09-13T14:29:32.919 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:32.944 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:32.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:32.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:32.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:32 smithi195 ceph-mon[104262]: pgmap v422: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.5 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 470 op/s; 20899/171249 objects degraded (12.204%); 2454/171249 objects misplaced (1.433%); 2.2 MiB/s, 35 objects/s recovering 2024-09-13T14:29:33.046 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:33.083 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:33.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:33.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:33.185 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:33.210 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:33.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:33.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:33.312 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:29:33.356 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:33.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:33.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:29:33.458 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:33.481 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:33.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:33.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:33.583 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:33.600 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:29:33.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:33.615 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:33.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:33.716 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:33.741 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:33.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:33.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:33.842 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:33.876 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:33.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:33.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:33.978 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:34.000 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:34.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:34.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:34.102 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:34.129 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:34.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:34.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:34.231 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:34.254 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:34.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:34.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:34.302 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:29:34.357 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:34.398 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:34.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:34.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:34.500 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:34.528 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:34.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:34.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:34.630 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:34.657 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:34.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:34.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:34.759 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:34.785 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:34.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:34.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:34.887 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:34.892 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:34 smithi102 ceph-mon[110183]: pgmap v423: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.3 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 852 op/s; 21031/161439 objects degraded (13.027%); 2031/161439 objects misplaced (1.258%); 4.5 MiB/s, 71 objects/s recovering 2024-09-13T14:29:34.892 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:34 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 21031/161439 objects degraded (13.027%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:29:34.893 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:34 smithi102 ceph-mon[115269]: pgmap v423: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.3 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 852 op/s; 21031/161439 objects degraded (13.027%); 2031/161439 objects misplaced (1.258%); 4.5 MiB/s, 71 objects/s recovering 2024-09-13T14:29:34.893 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:34 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 21031/161439 objects degraded (13.027%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:29:34.898 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:29:34.898 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:29:34.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:34.933 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:34.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:34.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:34 smithi195 ceph-mon[104262]: pgmap v423: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.3 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 852 op/s; 21031/161439 objects degraded (13.027%); 2031/161439 objects misplaced (1.258%); 4.5 MiB/s, 71 objects/s recovering 2024-09-13T14:29:34.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:34 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 21031/161439 objects degraded (13.027%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:29:35.035 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:29:35.060 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:35.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:35.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:29:35.162 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:29:35.187 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:35.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:35.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:29:35.289 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:35.315 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:35.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:35.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:35.416 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:29:35.441 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:35.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:35.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:29:35.543 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:29:35.546 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:29:35.573 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:35.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:35.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:29:35.601 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T14:29:35.675 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:35.717 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:35.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:35.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:35.819 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:35.845 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:35.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:35.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:35.947 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:29:35.989 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:35.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:35.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:29:36.091 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:36.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:36.134 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:36.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:36.235 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:36.258 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:36.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:36.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:36.360 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:36.375 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:29:36.398 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:36.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:36.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:36.500 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:36.525 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:36.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:36.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:36.573 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:29:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:29:36] "GET /metrics HTTP/1.1" 200 36537 "" "Prometheus/2.51.0" 2024-09-13T14:29:36.627 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:29:36.651 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:36.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:36.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:29:36.753 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:29:36.790 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:36.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:36.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:29:36.892 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:36.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:36 smithi102 ceph-mon[110183]: pgmap v424: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.3 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 466 op/s; 21031/160740 objects degraded (13.084%); 2031/160740 objects misplaced (1.264%); 2.3 MiB/s, 36 objects/s recovering 2024-09-13T14:29:36.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:36 smithi102 ceph-mon[115269]: pgmap v424: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.3 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 466 op/s; 21031/160740 objects degraded (13.084%); 2031/160740 objects misplaced (1.264%); 2.3 MiB/s, 36 objects/s recovering 2024-09-13T14:29:36.921 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:36.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:36.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:36.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:36 smithi195 ceph-mon[104262]: pgmap v424: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.3 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 466 op/s; 21031/160740 objects degraded (13.084%); 2031/160740 objects misplaced (1.264%); 2.3 MiB/s, 36 objects/s recovering 2024-09-13T14:29:37.023 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:37.045 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:37.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:37.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:37.119 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:29:37.147 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:37.189 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:37.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:37.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:37.290 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:37.319 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:37.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:37.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:37.421 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:37.460 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:37.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:37.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:37.461 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:29:37.562 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:37.601 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:37.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:37.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:37.703 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:29:37.724 INFO:teuthology.orchestra.run.smithi102.stderr:reweighted osd.7 to 1 (10000) 2024-09-13T14:29:37.736 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:37.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:37.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:29:37.838 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:37.862 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:37.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:37.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:37.957 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:37 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1146984450' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T14:29:37.957 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:37 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T14:29:37.957 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:37 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1146984450' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T14:29:37.957 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:37 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T14:29:37.964 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:37.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:37 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1146984450' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T14:29:37.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:37 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T14:29:38.011 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:38.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:38.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:38.113 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:38.187 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:38.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:38.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:38.289 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:38.316 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:38.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:38.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:38.417 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:29:38.445 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:38.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:38.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:29:38.547 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:38.569 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:38.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:38.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:38.671 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:38.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:38.718 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:38.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:38.819 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:29:38.846 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:38.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:38.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:29:38.948 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:38.972 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:38.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:38.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:38.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:38 smithi195 ceph-mon[104262]: pgmap v425: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.1 GiB data, 9.8 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 797 op/s; 21153/150606 objects degraded (14.045%); 1595/150606 objects misplaced (1.059%); 4.4 MiB/s, 70 objects/s recovering 2024-09-13T14:29:38.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:38 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T14:29:38.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:38 smithi195 ceph-mon[104262]: osdmap e147: 8 total, 8 up, 6 in 2024-09-13T14:29:38.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:38 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:29:39.074 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:39.082 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:38 smithi102 ceph-mon[115269]: pgmap v425: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.1 GiB data, 9.8 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 797 op/s; 21153/150606 objects degraded (14.045%); 1595/150606 objects misplaced (1.059%); 4.4 MiB/s, 70 objects/s recovering 2024-09-13T14:29:39.083 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:38 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T14:29:39.083 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:38 smithi102 ceph-mon[115269]: osdmap e147: 8 total, 8 up, 6 in 2024-09-13T14:29:39.083 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:38 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:29:39.083 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:38 smithi102 ceph-mon[110183]: pgmap v425: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.1 GiB data, 9.8 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 797 op/s; 21153/150606 objects degraded (14.045%); 1595/150606 objects misplaced (1.059%); 4.4 MiB/s, 70 objects/s recovering 2024-09-13T14:29:39.083 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:38 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T14:29:39.083 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:38 smithi102 ceph-mon[110183]: osdmap e147: 8 total, 8 up, 6 in 2024-09-13T14:29:39.084 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:38 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:29:39.106 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:39.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:39.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:39.209 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:29:39.231 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:39.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:39.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:29:39.333 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:39.375 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:39.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:39.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:39.477 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:39.524 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:39.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:39.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:39.626 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:39.649 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:39.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:39.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:39.751 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:39.777 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:39.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:39.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:39.857 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-13T14:29:39.858 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:29:39.879 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:39.919 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:39.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:39.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:39.991 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:39 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 21153/150606 objects degraded (14.045%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:29:39.991 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:39 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 21153/150606 objects degraded (14.045%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:29:40.000 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:39 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 21153/150606 objects degraded (14.045%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:29:40.021 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:40.043 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:40.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:40.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:40.145 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:40.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:40.209 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:40.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:40.310 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:40.333 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:40.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:40.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:40.435 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:29:40.468 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:40.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:40.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:29:40.570 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:29:40.593 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:40.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:40.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:29:40.695 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:40.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:40.738 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:40.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:40.839 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:29:40.874 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:40.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:40.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:29:40.976 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:40.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:40 smithi195 ceph-mon[104262]: pgmap v427: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.1 GiB data, 9.8 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 827 op/s; 21153/149889 objects degraded (14.112%); 1595/149889 objects misplaced (1.064%); 4.4 MiB/s, 70 objects/s recovering 2024-09-13T14:29:40.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:40 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:29:40.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:40 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:29:41.014 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:41.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:41.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:41.027 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:40 smithi102 ceph-mon[110183]: pgmap v427: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.1 GiB data, 9.8 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 827 op/s; 21153/149889 objects degraded (14.112%); 1595/149889 objects misplaced (1.064%); 4.4 MiB/s, 70 objects/s recovering 2024-09-13T14:29:41.027 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:40 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:29:41.027 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:40 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:29:41.027 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:40 smithi102 ceph-mon[115269]: pgmap v427: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.1 GiB data, 9.8 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 827 op/s; 21153/149889 objects degraded (14.112%); 1595/149889 objects misplaced (1.064%); 4.4 MiB/s, 70 objects/s recovering 2024-09-13T14:29:41.027 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:40 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:29:41.027 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:40 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:29:41.116 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:29:41.139 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:41.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:41.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:29:41.241 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:29:41.264 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:41.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:41.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:29:41.365 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:41.393 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:41.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:41.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:41.495 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:29:41.532 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:41.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:41.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:29:41.634 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:41.657 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:41.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:41.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:41.758 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:41.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:41.791 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:41.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:41.892 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:29:41.927 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:41.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:41.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:29:42.029 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:42.068 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:42.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:42.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:42.170 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:42.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:42.196 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:42.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:42.297 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:42.322 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:42.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:42.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:42.424 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:29:42.470 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:42.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:42.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:29:42.572 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:42.614 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:42.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:42.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:42.716 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:29:42.748 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:42.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:42.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:29:42.758 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:29:42.758 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:29:42.850 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:42.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:42 smithi102 ceph-mon[110183]: pgmap v428: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.1 GiB data, 9.8 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 711 op/s; 21153/149889 objects degraded (14.112%); 1595/149889 objects misplaced (1.064%); 3.8 MiB/s, 60 objects/s recovering 2024-09-13T14:29:42.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:42 smithi102 ceph-mon[115269]: pgmap v428: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.1 GiB data, 9.8 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 711 op/s; 21153/149889 objects degraded (14.112%); 1595/149889 objects misplaced (1.064%); 3.8 MiB/s, 60 objects/s recovering 2024-09-13T14:29:42.919 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:42.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:42.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:42.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:42 smithi195 ceph-mon[104262]: pgmap v428: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.1 GiB data, 9.8 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 711 op/s; 21153/149889 objects degraded (14.112%); 1595/149889 objects misplaced (1.064%); 3.8 MiB/s, 60 objects/s recovering 2024-09-13T14:29:43.021 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:43.047 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:43.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:43.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:43.149 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:29:43.188 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:43.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:43.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:29:43.290 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:43.334 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:43.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:43.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:43.435 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:43.460 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:43.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:43.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:43.561 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:43.626 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:43.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:43.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:43.728 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:43.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:43.754 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:43.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:43.855 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:43.904 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:43.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:43.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:43.999 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:43 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2086644492' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:29:44.000 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:43 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2086644492' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:29:44.006 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:44.029 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T14:29:44.032 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:43 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2086644492' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:29:44.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:44.068 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:44.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:44.116 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:29:44.117 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:29:44.117 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:29:44.117 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:29:44.117 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:29:44.117 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:29:44.117 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:29:44.117 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:29:44.117 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:29:44.117 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:29:44.117 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:29:44.118 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:29:44.118 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:29:44.118 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:29:44.118 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:29:44.118 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:29:44.169 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:44.197 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:44.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:44.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:44.299 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:44.323 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:44.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:44.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:44.424 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:44.462 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:44.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:44.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:44.564 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:44.590 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:44.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:44.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:44.692 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:44.715 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:44.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:44.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:44.816 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:44.862 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:44.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:44.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:44.963 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:45.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:45.000 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:45.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:45.075 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:44 smithi195 ceph-mon[104262]: pgmap v429: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.9 GiB data, 9.3 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 703 op/s; 21284/140334 objects degraded (15.167%); 1168/140334 objects misplaced (0.832%); 3.8 MiB/s, 60 objects/s recovering 2024-09-13T14:29:45.075 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:44 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 21284/140334 objects degraded (15.167%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:29:45.102 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:45.125 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:45.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:45.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:45.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:44 smithi102 ceph-mon[115269]: pgmap v429: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.9 GiB data, 9.3 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 703 op/s; 21284/140334 objects degraded (15.167%); 1168/140334 objects misplaced (0.832%); 3.8 MiB/s, 60 objects/s recovering 2024-09-13T14:29:45.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:44 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 21284/140334 objects degraded (15.167%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:29:45.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:44 smithi102 ceph-mon[110183]: pgmap v429: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.9 GiB data, 9.3 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 703 op/s; 21284/140334 objects degraded (15.167%); 1168/140334 objects misplaced (0.832%); 3.8 MiB/s, 60 objects/s recovering 2024-09-13T14:29:45.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:44 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 21284/140334 objects degraded (15.167%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:29:45.227 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:45.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:45.273 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:45.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:45.374 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:45.401 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:45.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:45.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:45.503 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:45.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:45.543 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:45.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:45.554 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:29:45.578 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:29:45.644 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:45.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:45.670 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:45.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:45.771 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:45.793 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:45.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:45.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:45.895 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:45.921 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:45.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:45.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:46.023 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:29:46.050 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:46.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:46.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:29:46.153 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:46.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:46.186 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:46.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:46.287 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:46.310 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:46.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:46.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:46.412 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:46.437 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:46.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:46.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:46.539 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:46.578 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:46.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:46.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:46.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:29:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:29:46] "GET /metrics HTTP/1.1" 200 36543 "" "Prometheus/2.51.0" 2024-09-13T14:29:46.681 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:29:46.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:46.707 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:46.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:29:46.766 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is set 2024-09-13T14:29:46.808 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:46.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:46.855 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:46.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:46.956 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:29:47.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:47.022 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:47.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:29:47.123 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:47.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:46 smithi102 ceph-mon[110183]: pgmap v430: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.8 GiB data, 9.3 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 700 op/s; 21284/139731 objects degraded (15.232%); 1168/139731 objects misplaced (0.836%); 3.8 MiB/s, 60 objects/s recovering 2024-09-13T14:29:47.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:46 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/4042300241' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:29:47.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:29:47.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:46 smithi102 ceph-mon[115269]: pgmap v430: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.8 GiB data, 9.3 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 700 op/s; 21284/139731 objects degraded (15.232%); 1168/139731 objects misplaced (0.836%); 3.8 MiB/s, 60 objects/s recovering 2024-09-13T14:29:47.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:46 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/4042300241' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:29:47.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:29:47.182 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:47.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:47.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:47.218 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:29:47.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:46 smithi195 ceph-mon[104262]: pgmap v430: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.8 GiB data, 9.3 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 700 op/s; 21284/139731 objects degraded (15.232%); 1168/139731 objects misplaced (0.836%); 3.8 MiB/s, 60 objects/s recovering 2024-09-13T14:29:47.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:46 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/4042300241' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:29:47.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:29:47.286 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:47.317 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:47.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:47.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:47.419 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:29:47.443 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:47.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:47.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:29:47.545 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:47.586 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:47.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:47.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:47.687 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:47.714 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:47.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:47.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:47.816 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:29:47.854 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:47.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:47.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:29:47.880 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:47 smithi102 ceph-mon[110183]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:29:47.880 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:47 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:29:47.880 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:47 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/4042300241' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:29:47.880 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:47 smithi102 ceph-mon[110183]: osdmap e148: 8 total, 8 up, 6 in 2024-09-13T14:29:47.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:47 smithi102 ceph-mon[115269]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:29:47.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:47 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:29:47.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:47 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/4042300241' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:29:47.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:47 smithi102 ceph-mon[115269]: osdmap e148: 8 total, 8 up, 6 in 2024-09-13T14:29:47.955 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:47.979 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:47.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:47.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:48.081 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:29:48.119 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:48.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:48.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:29:48.221 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:29:48.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:47 smithi195 ceph-mon[104262]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:29:48.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:47 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:29:48.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:47 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/4042300241' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:29:48.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:47 smithi195 ceph-mon[104262]: osdmap e148: 8 total, 8 up, 6 in 2024-09-13T14:29:48.266 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:48.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:48.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:29:48.368 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:29:48.392 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:48.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:48.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:29:48.494 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:48.519 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:48.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:48.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:48.621 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:48.645 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:48.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:48.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:48.747 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:48.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:48.788 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:48.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:48.889 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:29:48.917 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:48.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:48.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:29:48.996 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:48 smithi102 ceph-mon[110183]: pgmap v432: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.6 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 714 op/s; 21404/130071 objects degraded (16.456%); 733/130071 objects misplaced (0.564%); 4.0 MiB/s, 63 objects/s recovering 2024-09-13T14:29:48.996 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:48 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 21404/130071 objects degraded (16.456%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:29:48.997 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:48 smithi102 ceph-mon[115269]: pgmap v432: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.6 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 714 op/s; 21404/130071 objects degraded (16.456%); 733/130071 objects misplaced (0.564%); 4.0 MiB/s, 63 objects/s recovering 2024-09-13T14:29:48.997 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:48 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 21404/130071 objects degraded (16.456%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:29:49.019 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:49.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:49.064 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:49.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:49.165 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:49.204 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:49.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:49.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:49.249 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:29:49.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:48 smithi195 ceph-mon[104262]: pgmap v432: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.6 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 714 op/s; 21404/130071 objects degraded (16.456%); 733/130071 objects misplaced (0.564%); 4.0 MiB/s, 63 objects/s recovering 2024-09-13T14:29:49.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:48 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 21404/130071 objects degraded (16.456%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:29:49.307 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:49.332 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:49.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:49.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:49.434 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:49.478 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:49.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:49.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:49.580 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:49.608 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:49.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:49.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:49.638 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:29:49.711 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:49.741 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:49.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:49.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:49.845 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:29:49.883 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:49.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:49.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:29:49.985 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:50.012 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:50.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:50.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:50.114 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:29:50.139 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:50.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:50.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:29:50.240 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:29:50.278 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:50.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:50.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:29:50.324 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:29:50.326 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:29:50.337 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:29:50.337 INFO:teuthology.orchestra.run.smithi102.stdout: "size": 20, 2024-09-13T14:29:50.337 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 600, 2024-09-13T14:29:50.337 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [] 2024-09-13T14:29:50.337 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:29:50.380 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:50.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:50.424 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:50.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:50.525 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:50.559 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:50.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:50.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:50.661 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:29:50.687 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:50.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:50.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:29:50.789 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:29:50.822 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:50.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:50.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:29:50.910 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:50 smithi102 ceph-mon[110183]: pgmap v433: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.6 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 681 op/s; 21404/129441 objects degraded (16.536%); 733/129441 objects misplaced (0.566%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:29:50.910 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:50 smithi102 ceph-mon[115269]: pgmap v433: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.6 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 681 op/s; 21404/129441 objects degraded (16.536%); 733/129441 objects misplaced (0.566%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:29:50.924 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:50.964 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:50.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:50.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:51.065 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:51.089 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:51.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:51.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:51.191 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:29:51.217 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:51.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:51.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:29:51.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:50 smithi195 ceph-mon[104262]: pgmap v433: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.6 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 681 op/s; 21404/129441 objects degraded (16.536%); 733/129441 objects misplaced (0.566%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:29:51.264 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":433,"stamp":"2024-09-13T14:29:49.304823+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2830571517,"num_objects":43147,"num_object_clones":0,"num_object_copies":129441,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21404,"num_objects_misplaced":733,"num_objects_unfound":0,"num_objects_dirty":43147,"num_whiteouts":0,"num_read":6391,"num_read_kb":13846,"num_write":193836,"num_write_kb":7561602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8280,"num_bytes_recovered":543099099,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109695,"ondisk_log_size":109695,"up":267,"acting":261,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":9125912,"kb_used_data":8261992,"kb_used_omap":19,"kb_used_meta":863852,"kb_avail":553345000,"statfs":{"total":575970213888,"available":566625280000,"internally_reserved":0,"allocated":8460279808,"data_stored":8455785524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20284,"internal_metadata":884584644},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-446693376,"num_objects":-6816,"num_object_clones":0,"num_object_copies":-20448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":251,"num_objects_misplaced":-862,"num_objects_unfound":0,"num_objects_dirty":-6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":611,"num_bytes_recovered":40042496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002590"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":166,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.781711+0000","last_change":"2024-09-13T14:27:12.839984+0000","last_active":"2024-09-13T14:29:46.781711+0000","last_peered":"2024-09-13T14:29:46.781711+0000","last_clean":"2024-09-13T14:29:46.781711+0000","last_became_active":"2024-09-13T14:27:12.839653+0000","last_became_peered":"2024-09-13T14:27:12.839653+0000","last_unstale":"2024-09-13T14:29:46.781711+0000","last_undegraded":"2024-09-13T14:29:46.781711+0000","last_fullsized":"2024-09-13T14:29:46.781711+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:14:13.672830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":129,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.731859+0000","last_change":"2024-09-13T14:27:12.846760+0000","last_active":"2024-09-13T14:29:37.731859+0000","last_peered":"2024-09-13T14:29:37.731859+0000","last_clean":"2024-09-13T14:29:37.731859+0000","last_became_active":"2024-09-13T14:27:12.838041+0000","last_became_peered":"2024-09-13T14:27:12.838041+0000","last_unstale":"2024-09-13T14:29:37.731859+0000","last_undegraded":"2024-09-13T14:29:37.731859+0000","last_fullsized":"2024-09-13T14:29:37.731859+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:22.402859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":110,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T14:29:30.716398+0000","last_change":"2024-09-13T14:24:55.282407+0000","last_active":"2024-09-13T14:29:30.716398+0000","last_peered":"2024-09-13T14:29:30.716398+0000","last_clean":"2024-09-13T14:29:30.716398+0000","last_became_active":"2024-09-13T14:24:55.282083+0000","last_became_peered":"2024-09-13T14:24:55.282083+0000","last_unstale":"2024-09-13T14:29:30.716398+0000","last_undegraded":"2024-09-13T14:29:30.716398+0000","last_fullsized":"2024-09-13T14:29:30.716398+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:23:10.525707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":140,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.745250+0000","last_change":"2024-09-13T14:24:55.284433+0000","last_active":"2024-09-13T14:29:37.745250+0000","last_peered":"2024-09-13T14:29:37.745250+0000","last_clean":"2024-09-13T14:29:37.745250+0000","last_became_active":"2024-09-13T14:24:55.284143+0000","last_became_peered":"2024-09-13T14:24:55.284143+0000","last_unstale":"2024-09-13T14:29:37.745250+0000","last_undegraded":"2024-09-13T14:29:37.745250+0000","last_fullsized":"2024-09-13T14:29:37.745250+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:28.107405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":142,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.733684+0000","last_change":"2024-09-13T14:24:55.282416+0000","last_active":"2024-09-13T14:29:37.733684+0000","last_peered":"2024-09-13T14:29:37.733684+0000","last_clean":"2024-09-13T14:29:37.733684+0000","last_became_active":"2024-09-13T14:24:55.282045+0000","last_became_peered":"2024-09-13T14:24:55.282045+0000","last_unstale":"2024-09-13T14:29:37.733684+0000","last_undegraded":"2024-09-13T14:29:37.733684+0000","last_fullsized":"2024-09-13T14:29:37.733684+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:58.692897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":97,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.732517+0000","last_change":"2024-09-13T14:11:29.244452+0000","last_active":"2024-09-13T14:29:37.732517+0000","last_peered":"2024-09-13T14:29:37.732517+0000","last_clean":"2024-09-13T14:29:37.732517+0000","last_became_active":"2024-09-13T14:11:29.244102+0000","last_became_peered":"2024-09-13T14:11:29.244102+0000","last_unstale":"2024-09-13T14:29:37.732517+0000","last_undegraded":"2024-09-13T14:29:37.732517+0000","last_fullsized":"2024-09-13T14:29:37.732517+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:13:19.165534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":166,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.782361+0000","last_change":"2024-09-13T14:27:12.834300+0000","last_active":"2024-09-13T14:29:46.782361+0000","last_peered":"2024-09-13T14:29:46.782361+0000","last_clean":"2024-09-13T14:29:46.782361+0000","last_became_active":"2024-09-13T14:27:12.834046+0000","last_became_peered":"2024-09-13T14:27:12.834046+0000","last_unstale":"2024-09-13T14:29:46.782361+0000","last_undegraded":"2024-09-13T14:29:46.782361+0000","last_fullsized":"2024-09-13T14:29:46.782361+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:46:58.906862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":119,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T14:29:30.708234+0000","last_change":"2024-09-13T14:23:18.039061+0000","last_active":"2024-09-13T14:29:30.708234+0000","last_peered":"2024-09-13T14:29:30.708234+0000","last_clean":"2024-09-13T14:29:30.708234+0000","last_became_active":"2024-09-13T14:23:18.038661+0000","last_became_peered":"2024-09-13T14:23:18.038661+0000","last_unstale":"2024-09-13T14:29:30.708234+0000","last_undegraded":"2024-09-13T14:29:30.708234+0000","last_fullsized":"2024-09-13T14:29:30.708234+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:05:29.109844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.9","version":"148'11861","reported_seq":11902,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.793337+0000","last_change":"2024-09-13T14:26:21.865664+0000","last_active":"2024-09-13T14:29:46.793337+0000","last_peered":"2024-09-13T14:29:46.793337+0000","last_clean":"2024-09-13T14:29:46.793337+0000","last_became_active":"2024-09-13T14:26:21.865396+0000","last_became_peered":"2024-09-13T14:26:21.865396+0000","last_unstale":"2024-09-13T14:29:46.793337+0000","last_undegraded":"2024-09-13T14:29:46.793337+0000","last_fullsized":"2024-09-13T14:29:46.793337+0000","mapping_epoch":116,"log_start":"128'5100","ondisk_log_start":"128'5100","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6761,"log_dups_size":0,"ondisk_log_size":6761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:54:35.654265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174391296,"num_objects":2661,"num_object_clones":0,"num_object_copies":7983,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11861,"num_write_kb":464704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":114,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.732198+0000","last_change":"2024-09-13T14:23:18.038597+0000","last_active":"2024-09-13T14:29:37.732198+0000","last_peered":"2024-09-13T14:29:37.732198+0000","last_clean":"2024-09-13T14:29:37.732198+0000","last_became_active":"2024-09-13T14:23:18.038240+0000","last_became_peered":"2024-09-13T14:23:18.038240+0000","last_unstale":"2024-09-13T14:29:37.732198+0000","last_undegraded":"2024-09-13T14:29:37.732198+0000","last_fullsized":"2024-09-13T14:29:37.732198+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:27:51.842195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":96,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T14:29:30.715178+0000","last_change":"2024-09-13T14:11:30.254534+0000","last_active":"2024-09-13T14:29:30.715178+0000","last_peered":"2024-09-13T14:29:30.715178+0000","last_clean":"2024-09-13T14:29:30.715178+0000","last_became_active":"2024-09-13T14:11:30.253959+0000","last_became_peered":"2024-09-13T14:11:30.253959+0000","last_unstale":"2024-09-13T14:29:30.715178+0000","last_undegraded":"2024-09-13T14:29:30.715178+0000","last_fullsized":"2024-09-13T14:29:30.715178+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:09:54.025853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"147'12319","reported_seq":12396,"reported_epoch":147,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:29:46.362246+0000","last_change":"2024-09-13T14:27:14.009733+0000","last_active":"2024-09-13T14:29:46.362246+0000","last_peered":"2024-09-13T14:29:46.362246+0000","last_clean":"2024-09-13T14:27:11.786427+0000","last_became_active":"2024-09-13T14:27:13.994962+0000","last_became_peered":"2024-09-13T14:27:13.994962+0000","last_unstale":"2024-09-13T14:29:46.362246+0000","last_undegraded":"2024-09-13T14:27:13.854850+0000","last_fullsized":"2024-09-13T14:27:13.824319+0000","mapping_epoch":125,"log_start":"129'6000","ondisk_log_start":"129'6000","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6319,"log_dups_size":0,"ondisk_log_size":6319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179372032,"num_objects":2737,"num_object_clones":0,"num_object_copies":8211,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5198,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12319,"num_write_kb":481792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5],"avail_no_missing":["0","2","5"],"object_location_counts":[{"shards":"0,2,5","objects":407},{"shards":"0,5","objects":-2461}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":128,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.732357+0000","last_change":"2024-09-13T14:26:15.828529+0000","last_active":"2024-09-13T14:29:37.732357+0000","last_peered":"2024-09-13T14:29:37.732357+0000","last_clean":"2024-09-13T14:29:37.732357+0000","last_became_active":"2024-09-13T14:26:15.820486+0000","last_became_peered":"2024-09-13T14:26:15.820486+0000","last_unstale":"2024-09-13T14:29:37.732357+0000","last_undegraded":"2024-09-13T14:29:37.732357+0000","last_fullsized":"2024-09-13T14:29:37.732357+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:11:46.283172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":135,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.742537+0000","last_change":"2024-09-13T14:26:15.824632+0000","last_active":"2024-09-13T14:29:37.742537+0000","last_peered":"2024-09-13T14:29:37.742537+0000","last_clean":"2024-09-13T14:29:37.742537+0000","last_became_active":"2024-09-13T14:26:15.824330+0000","last_became_peered":"2024-09-13T14:26:15.824330+0000","last_unstale":"2024-09-13T14:29:37.742537+0000","last_undegraded":"2024-09-13T14:29:37.742537+0000","last_fullsized":"2024-09-13T14:29:37.742537+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:25.288116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"147'11713","reported_seq":21586,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.373153+0000","last_change":"2024-09-13T14:28:49.532961+0000","last_active":"2024-09-13T14:29:46.373153+0000","last_peered":"2024-09-13T14:29:46.373153+0000","last_clean":"2024-09-13T14:29:46.373153+0000","last_became_active":"2024-09-13T14:28:49.511508+0000","last_became_peered":"2024-09-13T14:28:49.511508+0000","last_unstale":"2024-09-13T14:29:46.373153+0000","last_undegraded":"2024-09-13T14:29:46.373153+0000","last_fullsized":"2024-09-13T14:29:46.373153+0000","mapping_epoch":137,"log_start":"129'5400","ondisk_log_start":"129'5400","created":116,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6313,"log_dups_size":0,"ondisk_log_size":6313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:42:44.821074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176488448,"num_objects":2693,"num_object_clones":0,"num_object_copies":8079,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":460992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4893,"num_bytes_recovered":320667648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":99,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.733431+0000","last_change":"2024-09-13T14:11:29.232897+0000","last_active":"2024-09-13T14:29:37.733431+0000","last_peered":"2024-09-13T14:29:37.733431+0000","last_clean":"2024-09-13T14:29:37.733431+0000","last_became_active":"2024-09-13T14:11:29.232637+0000","last_became_peered":"2024-09-13T14:11:29.232637+0000","last_unstale":"2024-09-13T14:29:37.733431+0000","last_undegraded":"2024-09-13T14:29:37.733431+0000","last_fullsized":"2024-09-13T14:29:37.733431+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:31:05.856046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":120,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T14:29:30.706685+0000","last_change":"2024-09-13T14:26:15.822452+0000","last_active":"2024-09-13T14:29:30.706685+0000","last_peered":"2024-09-13T14:29:30.706685+0000","last_clean":"2024-09-13T14:29:30.706685+0000","last_became_active":"2024-09-13T14:26:15.822111+0000","last_became_peered":"2024-09-13T14:26:15.822111+0000","last_unstale":"2024-09-13T14:29:30.706685+0000","last_undegraded":"2024-09-13T14:29:30.706685+0000","last_fullsized":"2024-09-13T14:29:30.706685+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:32:53.078767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"148'12182","reported_seq":12240,"reported_epoch":148,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:29:46.870461+0000","last_change":"2024-09-13T14:27:14.040788+0000","last_active":"2024-09-13T14:29:46.870461+0000","last_peered":"2024-09-13T14:29:46.870461+0000","last_clean":"2024-09-13T14:27:11.793438+0000","last_became_active":"2024-09-13T14:27:14.032063+0000","last_became_peered":"2024-09-13T14:27:14.032063+0000","last_unstale":"2024-09-13T14:29:46.870461+0000","last_undegraded":"2024-09-13T14:27:13.861208+0000","last_fullsized":"2024-09-13T14:27:13.827412+0000","mapping_epoch":125,"log_start":"129'5700","ondisk_log_start":"129'5700","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6482,"log_dups_size":0,"ondisk_log_size":6482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:07:10.560903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180617216,"num_objects":2756,"num_object_clones":0,"num_object_copies":8268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12182,"num_write_kb":478016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3],"avail_no_missing":["4","2","3"],"object_location_counts":[{"shards":"2,3,4","objects":365},{"shards":"3,4","objects":-2322}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":112,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T14:29:30.715449+0000","last_change":"2024-09-13T14:24:55.290798+0000","last_active":"2024-09-13T14:29:30.715449+0000","last_peered":"2024-09-13T14:29:30.715449+0000","last_clean":"2024-09-13T14:29:30.715449+0000","last_became_active":"2024-09-13T14:24:55.290510+0000","last_became_peered":"2024-09-13T14:24:55.290510+0000","last_unstale":"2024-09-13T14:29:30.715449+0000","last_undegraded":"2024-09-13T14:29:30.715449+0000","last_fullsized":"2024-09-13T14:29:30.715449+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:19:03.129132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":149,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.735344+0000","last_change":"2024-09-13T14:26:15.823316+0000","last_active":"2024-09-13T14:29:37.735344+0000","last_peered":"2024-09-13T14:29:37.735344+0000","last_clean":"2024-09-13T14:29:37.735344+0000","last_became_active":"2024-09-13T14:26:15.822980+0000","last_became_peered":"2024-09-13T14:26:15.822980+0000","last_unstale":"2024-09-13T14:29:37.735344+0000","last_undegraded":"2024-09-13T14:29:37.735344+0000","last_fullsized":"2024-09-13T14:29:37.735344+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:33:47.420548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"148'12105","reported_seq":16126,"reported_epoch":148,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T14:29:46.838252+0000","last_change":"2024-09-13T14:28:48.293087+0000","last_active":"2024-09-13T14:29:46.838252+0000","last_peered":"2024-09-13T14:29:46.838252+0000","last_clean":"2024-09-13T14:27:11.806319+0000","last_became_active":"2024-09-13T14:27:14.027828+0000","last_became_peered":"2024-09-13T14:27:14.027828+0000","last_unstale":"2024-09-13T14:29:46.838252+0000","last_undegraded":"2024-09-13T14:27:13.862259+0000","last_fullsized":"2024-09-13T14:27:13.823132+0000","mapping_epoch":125,"log_start":"128'5400","ondisk_log_start":"128'5400","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6705,"log_dups_size":0,"ondisk_log_size":6705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:40:06.560106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177602584,"num_objects":2711,"num_object_clones":0,"num_object_copies":8133,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2721,"num_objects_misplaced":356,"num_objects_unfound":0,"num_objects_dirty":2711,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12105,"num_write_kb":474049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1980,"num_bytes_recovered":129761280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5],"avail_no_missing":["7","2","5"],"object_location_counts":[{"shards":"2,5,7","objects":356},{"shards":"5,7","objects":-366}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":150,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.781479+0000","last_change":"2024-09-13T14:24:55.284499+0000","last_active":"2024-09-13T14:29:46.781479+0000","last_peered":"2024-09-13T14:29:46.781479+0000","last_clean":"2024-09-13T14:29:46.781479+0000","last_became_active":"2024-09-13T14:24:55.284096+0000","last_became_peered":"2024-09-13T14:24:55.284096+0000","last_unstale":"2024-09-13T14:29:46.781479+0000","last_undegraded":"2024-09-13T14:29:46.781479+0000","last_fullsized":"2024-09-13T14:29:46.781479+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:02:06.836099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":111,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.740963+0000","last_change":"2024-09-13T14:24:55.291371+0000","last_active":"2024-09-13T14:29:37.740963+0000","last_peered":"2024-09-13T14:29:37.740963+0000","last_clean":"2024-09-13T14:29:37.740963+0000","last_became_active":"2024-09-13T14:24:55.291047+0000","last_became_peered":"2024-09-13T14:24:55.291047+0000","last_unstale":"2024-09-13T14:29:37.740963+0000","last_undegraded":"2024-09-13T14:29:37.740963+0000","last_fullsized":"2024-09-13T14:29:37.740963+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:43.912412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"148'11948","reported_seq":11990,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.859782+0000","last_change":"2024-09-13T14:26:21.864062+0000","last_active":"2024-09-13T14:29:46.859782+0000","last_peered":"2024-09-13T14:29:46.859782+0000","last_clean":"2024-09-13T14:29:46.859782+0000","last_became_active":"2024-09-13T14:26:21.863721+0000","last_became_peered":"2024-09-13T14:26:21.863721+0000","last_unstale":"2024-09-13T14:29:46.859782+0000","last_undegraded":"2024-09-13T14:29:46.859782+0000","last_fullsized":"2024-09-13T14:29:46.859782+0000","mapping_epoch":116,"log_start":"129'5500","ondisk_log_start":"129'5500","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6448,"log_dups_size":0,"ondisk_log_size":6448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:15:42.744309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169607168,"num_objects":2588,"num_object_clones":0,"num_object_copies":7764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11948,"num_write_kb":465152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":121,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T14:29:30.706825+0000","last_change":"2024-09-13T14:26:15.827333+0000","last_active":"2024-09-13T14:29:30.706825+0000","last_peered":"2024-09-13T14:29:30.706825+0000","last_clean":"2024-09-13T14:29:30.706825+0000","last_became_active":"2024-09-13T14:26:15.826945+0000","last_became_peered":"2024-09-13T14:26:15.826945+0000","last_unstale":"2024-09-13T14:29:30.706825+0000","last_undegraded":"2024-09-13T14:29:30.706825+0000","last_fullsized":"2024-09-13T14:29:30.706825+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:35:24.870052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":144,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.741255+0000","last_change":"2024-09-13T14:27:12.842604+0000","last_active":"2024-09-13T14:29:37.741255+0000","last_peered":"2024-09-13T14:29:37.741255+0000","last_clean":"2024-09-13T14:29:37.741255+0000","last_became_active":"2024-09-13T14:27:12.841643+0000","last_became_peered":"2024-09-13T14:27:12.841643+0000","last_unstale":"2024-09-13T14:29:37.741255+0000","last_undegraded":"2024-09-13T14:29:37.741255+0000","last_fullsized":"2024-09-13T14:29:37.741255+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:34:34.751790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"147'12002","reported_seq":12043,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.742841+0000","last_change":"2024-09-13T14:26:21.867125+0000","last_active":"2024-09-13T14:29:46.742841+0000","last_peered":"2024-09-13T14:29:46.742841+0000","last_clean":"2024-09-13T14:29:46.742841+0000","last_became_active":"2024-09-13T14:26:21.866815+0000","last_became_peered":"2024-09-13T14:26:21.866815+0000","last_unstale":"2024-09-13T14:29:46.742841+0000","last_undegraded":"2024-09-13T14:29:46.742841+0000","last_fullsized":"2024-09-13T14:29:46.742841+0000","mapping_epoch":116,"log_start":"122'4100","ondisk_log_start":"122'4100","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":7902,"log_dups_size":0,"ondisk_log_size":7902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:44.218255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177471488,"num_objects":2708,"num_object_clones":0,"num_object_copies":8124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12002,"num_write_kb":470720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":121,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.742781+0000","last_change":"2024-09-13T14:27:12.842271+0000","last_active":"2024-09-13T14:29:37.742781+0000","last_peered":"2024-09-13T14:29:37.742781+0000","last_clean":"2024-09-13T14:29:37.742781+0000","last_became_active":"2024-09-13T14:27:12.841479+0000","last_became_peered":"2024-09-13T14:27:12.841479+0000","last_unstale":"2024-09-13T14:29:37.742781+0000","last_undegraded":"2024-09-13T14:29:37.742781+0000","last_fullsized":"2024-09-13T14:29:37.742781+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:30:22.746575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":120,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.741301+0000","last_change":"2024-09-13T14:26:15.825304+0000","last_active":"2024-09-13T14:29:37.741301+0000","last_peered":"2024-09-13T14:29:37.741301+0000","last_clean":"2024-09-13T14:29:37.741301+0000","last_became_active":"2024-09-13T14:26:15.825018+0000","last_became_peered":"2024-09-13T14:26:15.825018+0000","last_unstale":"2024-09-13T14:29:37.741301+0000","last_undegraded":"2024-09-13T14:29:37.741301+0000","last_fullsized":"2024-09-13T14:29:37.741301+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:22:59.408928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"148'12069","reported_seq":12108,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.838693+0000","last_change":"2024-09-13T14:26:21.865199+0000","last_active":"2024-09-13T14:29:46.838693+0000","last_peered":"2024-09-13T14:29:46.838693+0000","last_clean":"2024-09-13T14:29:46.838693+0000","last_became_active":"2024-09-13T14:26:21.864903+0000","last_became_peered":"2024-09-13T14:26:21.864903+0000","last_unstale":"2024-09-13T14:29:46.838693+0000","last_undegraded":"2024-09-13T14:29:46.838693+0000","last_fullsized":"2024-09-13T14:29:46.838693+0000","mapping_epoch":116,"log_start":"129'5600","ondisk_log_start":"129'5600","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6469,"log_dups_size":0,"ondisk_log_size":6469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:25.014330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176095232,"num_objects":2687,"num_object_clones":0,"num_object_copies":8061,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12069,"num_write_kb":472192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":160,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.780528+0000","last_change":"2024-09-13T14:26:15.823898+0000","last_active":"2024-09-13T14:29:46.780528+0000","last_peered":"2024-09-13T14:29:46.780528+0000","last_clean":"2024-09-13T14:29:46.780528+0000","last_became_active":"2024-09-13T14:26:15.823125+0000","last_became_peered":"2024-09-13T14:26:15.823125+0000","last_unstale":"2024-09-13T14:29:46.780528+0000","last_undegraded":"2024-09-13T14:29:46.780528+0000","last_fullsized":"2024-09-13T14:29:46.780528+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:51:27.949528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":122,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.782497+0000","last_change":"2024-09-13T14:26:15.823198+0000","last_active":"2024-09-13T14:29:46.782497+0000","last_peered":"2024-09-13T14:29:46.782497+0000","last_clean":"2024-09-13T14:29:46.782497+0000","last_became_active":"2024-09-13T14:26:15.822717+0000","last_became_peered":"2024-09-13T14:26:15.822717+0000","last_unstale":"2024-09-13T14:29:46.782497+0000","last_undegraded":"2024-09-13T14:29:46.782497+0000","last_fullsized":"2024-09-13T14:29:46.782497+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:22.471622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"147'12173","reported_seq":12219,"reported_epoch":147,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:29:46.765623+0000","last_change":"2024-09-13T14:27:14.052640+0000","last_active":"2024-09-13T14:29:46.765623+0000","last_peered":"2024-09-13T14:29:46.765623+0000","last_clean":"2024-09-13T14:27:11.828545+0000","last_became_active":"2024-09-13T14:27:14.042591+0000","last_became_peered":"2024-09-13T14:27:14.042591+0000","last_unstale":"2024-09-13T14:29:46.765623+0000","last_undegraded":"2024-09-13T14:27:13.862613+0000","last_fullsized":"2024-09-13T14:27:13.823368+0000","mapping_epoch":125,"log_start":"128'5500","ondisk_log_start":"128'5500","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6673,"log_dups_size":0,"ondisk_log_size":6673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:17:38.266600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175964160,"num_objects":2685,"num_object_clones":0,"num_object_copies":8055,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12173,"num_write_kb":475456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5],"avail_no_missing":["7","2","5"],"object_location_counts":[{"shards":"2,5,7","objects":378},{"shards":"5,7","objects":-2437}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":443,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.731494+0000","last_change":"2024-09-13T14:23:18.032973+0000","last_active":"2024-09-13T14:29:37.731494+0000","last_peered":"2024-09-13T14:29:37.731494+0000","last_clean":"2024-09-13T14:29:37.731494+0000","last_became_active":"2024-09-13T14:23:18.032579+0000","last_became_peered":"2024-09-13T14:23:18.032579+0000","last_unstale":"2024-09-13T14:29:37.731494+0000","last_undegraded":"2024-09-13T14:29:37.731494+0000","last_fullsized":"2024-09-13T14:29:37.731494+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:07:41.927740+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:17.109445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":476,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T14:29:30.714936+0000","last_change":"2024-09-13T14:27:15.865400+0000","last_active":"2024-09-13T14:29:30.714936+0000","last_peered":"2024-09-13T14:29:30.714936+0000","last_clean":"2024-09-13T14:29:30.714936+0000","last_became_active":"2024-09-13T14:27:15.864835+0000","last_became_peered":"2024-09-13T14:27:15.864835+0000","last_unstale":"2024-09-13T14:29:30.714936+0000","last_undegraded":"2024-09-13T14:29:30.714936+0000","last_fullsized":"2024-09-13T14:29:30.714936+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:27:10.928041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":101,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.783467+0000","last_change":"2024-09-13T14:24:55.288058+0000","last_active":"2024-09-13T14:29:46.783467+0000","last_peered":"2024-09-13T14:29:46.783467+0000","last_clean":"2024-09-13T14:29:46.783467+0000","last_became_active":"2024-09-13T14:24:55.287601+0000","last_became_peered":"2024-09-13T14:24:55.287601+0000","last_unstale":"2024-09-13T14:29:46.783467+0000","last_undegraded":"2024-09-13T14:29:46.783467+0000","last_fullsized":"2024-09-13T14:29:46.783467+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:56:34.622696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":96,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T14:29:30.711070+0000","last_change":"2024-09-13T14:11:30.253749+0000","last_active":"2024-09-13T14:29:30.711070+0000","last_peered":"2024-09-13T14:29:30.711070+0000","last_clean":"2024-09-13T14:29:30.711070+0000","last_became_active":"2024-09-13T14:11:30.253499+0000","last_became_peered":"2024-09-13T14:11:30.253499+0000","last_unstale":"2024-09-13T14:29:30.711070+0000","last_undegraded":"2024-09-13T14:29:30.711070+0000","last_fullsized":"2024-09-13T14:29:30.711070+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:02:00.987335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"147'12226","reported_seq":12267,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.689339+0000","last_change":"2024-09-13T14:26:21.865805+0000","last_active":"2024-09-13T14:29:46.689339+0000","last_peered":"2024-09-13T14:29:46.689339+0000","last_clean":"2024-09-13T14:29:46.689339+0000","last_became_active":"2024-09-13T14:26:21.865288+0000","last_became_peered":"2024-09-13T14:26:21.865288+0000","last_unstale":"2024-09-13T14:29:46.689339+0000","last_undegraded":"2024-09-13T14:29:46.689339+0000","last_fullsized":"2024-09-13T14:29:46.689339+0000","mapping_epoch":116,"log_start":"122'4300","ondisk_log_start":"122'4300","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":7926,"log_dups_size":0,"ondisk_log_size":7926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:46.510065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180486144,"num_objects":2754,"num_object_clones":0,"num_object_copies":8262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12226,"num_write_kb":479360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":358,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.731361+0000","last_change":"2024-09-13T14:26:15.835178+0000","last_active":"2024-09-13T14:29:37.731361+0000","last_peered":"2024-09-13T14:29:37.731361+0000","last_clean":"2024-09-13T14:29:37.731361+0000","last_became_active":"2024-09-13T14:26:15.820283+0000","last_became_peered":"2024-09-13T14:26:15.820283+0000","last_unstale":"2024-09-13T14:29:37.731361+0000","last_undegraded":"2024-09-13T14:29:37.731361+0000","last_fullsized":"2024-09-13T14:29:37.731361+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:30.340038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":142,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.733277+0000","last_change":"2024-09-13T14:24:55.286576+0000","last_active":"2024-09-13T14:29:37.733277+0000","last_peered":"2024-09-13T14:29:37.733277+0000","last_clean":"2024-09-13T14:29:37.733277+0000","last_became_active":"2024-09-13T14:24:55.286284+0000","last_became_peered":"2024-09-13T14:24:55.286284+0000","last_unstale":"2024-09-13T14:29:37.733277+0000","last_undegraded":"2024-09-13T14:29:37.733277+0000","last_fullsized":"2024-09-13T14:29:37.733277+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:35.589736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":168,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.733982+0000","last_change":"2024-09-13T14:27:12.836204+0000","last_active":"2024-09-13T14:29:37.733982+0000","last_peered":"2024-09-13T14:29:37.733982+0000","last_clean":"2024-09-13T14:29:37.733982+0000","last_became_active":"2024-09-13T14:27:12.835900+0000","last_became_peered":"2024-09-13T14:27:12.835900+0000","last_unstale":"2024-09-13T14:29:37.733982+0000","last_undegraded":"2024-09-13T14:29:37.733982+0000","last_fullsized":"2024-09-13T14:29:37.733982+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:36:52.121730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"147'11978","reported_seq":14678,"reported_epoch":147,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T14:29:46.770910+0000","last_change":"2024-09-13T14:28:48.300429+0000","last_active":"2024-09-13T14:29:46.770910+0000","last_peered":"2024-09-13T14:29:46.770910+0000","last_clean":"2024-09-13T14:27:11.802376+0000","last_became_active":"2024-09-13T14:27:13.985759+0000","last_became_peered":"2024-09-13T14:27:13.985759+0000","last_unstale":"2024-09-13T14:29:46.770910+0000","last_undegraded":"2024-09-13T14:27:13.852391+0000","last_fullsized":"2024-09-13T14:27:13.824524+0000","mapping_epoch":125,"log_start":"122'4000","ondisk_log_start":"122'4000","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":7978,"log_dups_size":0,"ondisk_log_size":7978,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:58:21.117245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182583296,"num_objects":2786,"num_object_clones":0,"num_object_copies":8358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3273,"num_objects_misplaced":377,"num_objects_unfound":0,"num_objects_dirty":2786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11978,"num_write_kb":472448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1323,"num_bytes_recovered":86704128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7],"avail_no_missing":["5","2","7"],"object_location_counts":[{"shards":"2,5,7","objects":377},{"shards":"5,7","objects":-864}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":524,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.780506+0000","last_change":"2024-09-13T14:27:15.863966+0000","last_active":"2024-09-13T14:29:46.780506+0000","last_peered":"2024-09-13T14:29:46.780506+0000","last_clean":"2024-09-13T14:29:46.780506+0000","last_became_active":"2024-09-13T14:27:15.863124+0000","last_became_peered":"2024-09-13T14:27:15.863124+0000","last_unstale":"2024-09-13T14:29:46.780506+0000","last_undegraded":"2024-09-13T14:29:46.780506+0000","last_fullsized":"2024-09-13T14:29:46.780506+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:35:02.715013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":71,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T14:29:30.708350+0000","last_change":"2024-09-13T14:27:12.829079+0000","last_active":"2024-09-13T14:29:30.708350+0000","last_peered":"2024-09-13T14:29:30.708350+0000","last_clean":"2024-09-13T14:29:30.708350+0000","last_became_active":"2024-09-13T14:27:12.828619+0000","last_became_peered":"2024-09-13T14:27:12.828619+0000","last_unstale":"2024-09-13T14:29:30.708350+0000","last_undegraded":"2024-09-13T14:29:30.708350+0000","last_fullsized":"2024-09-13T14:29:30.708350+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:39:08.260304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":107,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T14:29:30.717546+0000","last_change":"2024-09-13T14:26:15.817732+0000","last_active":"2024-09-13T14:29:30.717546+0000","last_peered":"2024-09-13T14:29:30.717546+0000","last_clean":"2024-09-13T14:29:30.717546+0000","last_became_active":"2024-09-13T14:26:15.817403+0000","last_became_peered":"2024-09-13T14:26:15.817403+0000","last_unstale":"2024-09-13T14:29:30.717546+0000","last_undegraded":"2024-09-13T14:29:30.717546+0000","last_fullsized":"2024-09-13T14:29:30.717546+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:56:39.961300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"147'11914","reported_seq":11989,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.391989+0000","last_change":"2024-09-13T14:26:21.866279+0000","last_active":"2024-09-13T14:29:46.391989+0000","last_peered":"2024-09-13T14:29:46.391989+0000","last_clean":"2024-09-13T14:29:46.391989+0000","last_became_active":"2024-09-13T14:26:21.864793+0000","last_became_peered":"2024-09-13T14:26:21.864793+0000","last_unstale":"2024-09-13T14:29:46.391989+0000","last_undegraded":"2024-09-13T14:29:46.391989+0000","last_fullsized":"2024-09-13T14:29:46.391989+0000","mapping_epoch":116,"log_start":"129'5600","ondisk_log_start":"129'5600","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6314,"log_dups_size":0,"ondisk_log_size":6314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:48:28.309201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170262528,"num_objects":2598,"num_object_clones":0,"num_object_copies":7794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11914,"num_write_kb":464384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":907,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T14:29:30.715031+0000","last_change":"2024-09-13T14:26:18.829131+0000","last_active":"2024-09-13T14:29:30.715031+0000","last_peered":"2024-09-13T14:29:30.715031+0000","last_clean":"2024-09-13T14:29:30.715031+0000","last_became_active":"2024-09-13T14:26:18.828619+0000","last_became_peered":"2024-09-13T14:26:18.828619+0000","last_unstale":"2024-09-13T14:29:30.715031+0000","last_undegraded":"2024-09-13T14:29:30.715031+0000","last_fullsized":"2024-09-13T14:29:30.715031+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:24:37.073618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":112,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.740502+0000","last_change":"2024-09-13T14:23:18.042891+0000","last_active":"2024-09-13T14:29:37.740502+0000","last_peered":"2024-09-13T14:29:37.740502+0000","last_clean":"2024-09-13T14:29:37.740502+0000","last_became_active":"2024-09-13T14:23:18.042562+0000","last_became_peered":"2024-09-13T14:23:18.042562+0000","last_unstale":"2024-09-13T14:29:37.740502+0000","last_undegraded":"2024-09-13T14:29:37.740502+0000","last_fullsized":"2024-09-13T14:29:37.740502+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:30:42.996661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":137,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.781635+0000","last_change":"2024-09-13T14:27:12.832666+0000","last_active":"2024-09-13T14:29:46.781635+0000","last_peered":"2024-09-13T14:29:46.781635+0000","last_clean":"2024-09-13T14:29:46.781635+0000","last_became_active":"2024-09-13T14:27:12.832332+0000","last_became_peered":"2024-09-13T14:27:12.832332+0000","last_unstale":"2024-09-13T14:29:46.781635+0000","last_undegraded":"2024-09-13T14:29:46.781635+0000","last_fullsized":"2024-09-13T14:29:46.781635+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:07:17.724156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"147'11862","reported_seq":11937,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.345018+0000","last_change":"2024-09-13T14:26:21.862954+0000","last_active":"2024-09-13T14:29:46.345018+0000","last_peered":"2024-09-13T14:29:46.345018+0000","last_clean":"2024-09-13T14:29:46.345018+0000","last_became_active":"2024-09-13T14:26:21.862650+0000","last_became_peered":"2024-09-13T14:26:21.862650+0000","last_unstale":"2024-09-13T14:29:46.345018+0000","last_undegraded":"2024-09-13T14:29:46.345018+0000","last_fullsized":"2024-09-13T14:29:46.345018+0000","mapping_epoch":116,"log_start":"129'5600","ondisk_log_start":"129'5600","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6262,"log_dups_size":0,"ondisk_log_size":6262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:46:50.507774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173146112,"num_objects":2642,"num_object_clones":0,"num_object_copies":7926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11862,"num_write_kb":464128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":136,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T14:29:30.716470+0000","last_change":"2024-09-13T14:27:12.843038+0000","last_active":"2024-09-13T14:29:30.716470+0000","last_peered":"2024-09-13T14:29:30.716470+0000","last_clean":"2024-09-13T14:29:30.716470+0000","last_became_active":"2024-09-13T14:27:12.842663+0000","last_became_peered":"2024-09-13T14:27:12.842663+0000","last_unstale":"2024-09-13T14:29:30.716470+0000","last_undegraded":"2024-09-13T14:29:30.716470+0000","last_fullsized":"2024-09-13T14:29:30.716470+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:23:38.880811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":838,"reported_epoch":147,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:29:37.739871+0000","last_change":"2024-09-13T14:27:14.029991+0000","last_active":"2024-09-13T14:29:37.739871+0000","last_peered":"2024-09-13T14:29:37.739871+0000","last_clean":"2024-09-13T14:27:08.815347+0000","last_became_active":"2024-09-13T14:27:13.888039+0000","last_became_peered":"2024-09-13T14:27:13.888039+0000","last_unstale":"2024-09-13T14:29:37.739871+0000","last_undegraded":"2024-09-13T14:27:13.826593+0000","last_fullsized":"2024-09-13T14:27:13.824237+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7],"avail_no_missing":["0","2","7"],"object_location_counts":[{"shards":"0,2,7","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":63,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T14:29:30.714853+0000","last_change":"2024-09-13T14:26:15.819032+0000","last_active":"2024-09-13T14:29:30.714853+0000","last_peered":"2024-09-13T14:29:30.714853+0000","last_clean":"2024-09-13T14:29:30.714853+0000","last_became_active":"2024-09-13T14:26:15.818675+0000","last_became_peered":"2024-09-13T14:26:15.818675+0000","last_unstale":"2024-09-13T14:29:30.714853+0000","last_undegraded":"2024-09-13T14:29:30.714853+0000","last_fullsized":"2024-09-13T14:29:30.714853+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:37:13.628256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"147'12012","reported_seq":12053,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.272430+0000","last_change":"2024-09-13T14:26:21.863343+0000","last_active":"2024-09-13T14:29:46.272430+0000","last_peered":"2024-09-13T14:29:46.272430+0000","last_clean":"2024-09-13T14:29:46.272430+0000","last_became_active":"2024-09-13T14:26:21.863057+0000","last_became_peered":"2024-09-13T14:26:21.863057+0000","last_unstale":"2024-09-13T14:29:46.272430+0000","last_undegraded":"2024-09-13T14:29:46.272430+0000","last_fullsized":"2024-09-13T14:29:46.272430+0000","mapping_epoch":116,"log_start":"128'5300","ondisk_log_start":"128'5300","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6712,"log_dups_size":0,"ondisk_log_size":6712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:38:25.794314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178913280,"num_objects":2730,"num_object_clones":0,"num_object_copies":8190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12012,"num_write_kb":471744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":71,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T14:29:30.715377+0000","last_change":"2024-09-13T14:27:12.843268+0000","last_active":"2024-09-13T14:29:30.715377+0000","last_peered":"2024-09-13T14:29:30.715377+0000","last_clean":"2024-09-13T14:29:30.715377+0000","last_became_active":"2024-09-13T14:27:12.842867+0000","last_became_peered":"2024-09-13T14:27:12.842867+0000","last_unstale":"2024-09-13T14:29:30.715377+0000","last_undegraded":"2024-09-13T14:29:30.715377+0000","last_fullsized":"2024-09-13T14:29:30.715377+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:03:59.575866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":635,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.733505+0000","last_change":"2024-09-13T14:23:18.038847+0000","last_active":"2024-09-13T14:29:37.733505+0000","last_peered":"2024-09-13T14:29:37.733505+0000","last_clean":"2024-09-13T14:29:37.733505+0000","last_became_active":"2024-09-13T14:23:18.038188+0000","last_became_peered":"2024-09-13T14:23:18.038188+0000","last_unstale":"2024-09-13T14:29:37.733505+0000","last_undegraded":"2024-09-13T14:29:37.733505+0000","last_fullsized":"2024-09-13T14:29:37.733505+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:30:37.213496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":132,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.735273+0000","last_change":"2024-09-13T14:23:18.034427+0000","last_active":"2024-09-13T14:29:37.735273+0000","last_peered":"2024-09-13T14:29:37.735273+0000","last_clean":"2024-09-13T14:29:37.735273+0000","last_became_active":"2024-09-13T14:23:18.034092+0000","last_became_peered":"2024-09-13T14:23:18.034092+0000","last_unstale":"2024-09-13T14:29:37.735273+0000","last_undegraded":"2024-09-13T14:29:37.735273+0000","last_fullsized":"2024-09-13T14:29:37.735273+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:21.755115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"148'12100","reported_seq":12142,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.897347+0000","last_change":"2024-09-13T14:26:21.867237+0000","last_active":"2024-09-13T14:29:46.897347+0000","last_peered":"2024-09-13T14:29:46.897347+0000","last_clean":"2024-09-13T14:29:46.897347+0000","last_became_active":"2024-09-13T14:26:21.866931+0000","last_became_peered":"2024-09-13T14:26:21.866931+0000","last_unstale":"2024-09-13T14:29:46.897347+0000","last_undegraded":"2024-09-13T14:29:46.897347+0000","last_fullsized":"2024-09-13T14:29:46.897347+0000","mapping_epoch":116,"log_start":"129'5700","ondisk_log_start":"129'5700","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6400,"log_dups_size":0,"ondisk_log_size":6400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:52:57.569446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177864704,"num_objects":2714,"num_object_clones":0,"num_object_copies":8142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12100,"num_write_kb":474048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":150,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.782612+0000","last_change":"2024-09-13T14:24:55.290711+0000","last_active":"2024-09-13T14:29:46.782612+0000","last_peered":"2024-09-13T14:29:46.782612+0000","last_clean":"2024-09-13T14:29:46.782612+0000","last_became_active":"2024-09-13T14:24:55.288549+0000","last_became_peered":"2024-09-13T14:24:55.288549+0000","last_unstale":"2024-09-13T14:29:46.782612+0000","last_undegraded":"2024-09-13T14:29:46.782612+0000","last_fullsized":"2024-09-13T14:29:46.782612+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:55:21.763931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":547,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T14:29:30.706812+0000","last_change":"2024-09-13T14:26:18.827876+0000","last_active":"2024-09-13T14:29:30.706812+0000","last_peered":"2024-09-13T14:29:30.706812+0000","last_clean":"2024-09-13T14:29:30.706812+0000","last_became_active":"2024-09-13T14:26:18.827317+0000","last_became_peered":"2024-09-13T14:26:18.827317+0000","last_unstale":"2024-09-13T14:29:30.706812+0000","last_undegraded":"2024-09-13T14:29:30.706812+0000","last_fullsized":"2024-09-13T14:29:30.706812+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:31:48.611286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":147,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.780373+0000","last_change":"2024-09-13T14:24:55.281593+0000","last_active":"2024-09-13T14:29:46.780373+0000","last_peered":"2024-09-13T14:29:46.780373+0000","last_clean":"2024-09-13T14:29:46.780373+0000","last_became_active":"2024-09-13T14:24:55.280805+0000","last_became_peered":"2024-09-13T14:24:55.280805+0000","last_unstale":"2024-09-13T14:29:46.780373+0000","last_undegraded":"2024-09-13T14:29:46.780373+0000","last_fullsized":"2024-09-13T14:29:46.780373+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:14.806713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"148'12249","reported_seq":12326,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:48.195296+0000","last_change":"2024-09-13T14:26:21.864819+0000","last_active":"2024-09-13T14:29:48.195296+0000","last_peered":"2024-09-13T14:29:48.195296+0000","last_clean":"2024-09-13T14:29:48.195296+0000","last_became_active":"2024-09-13T14:26:21.864511+0000","last_became_peered":"2024-09-13T14:26:21.864511+0000","last_unstale":"2024-09-13T14:29:48.195296+0000","last_undegraded":"2024-09-13T14:29:48.195296+0000","last_fullsized":"2024-09-13T14:29:48.195296+0000","mapping_epoch":116,"log_start":"128'5400","ondisk_log_start":"128'5400","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6849,"log_dups_size":0,"ondisk_log_size":6849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:11:52.262311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174915584,"num_objects":2669,"num_object_clones":0,"num_object_copies":8007,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12249,"num_write_kb":477376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":157,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.733622+0000","last_change":"2024-09-13T14:26:15.830309+0000","last_active":"2024-09-13T14:29:37.733622+0000","last_peered":"2024-09-13T14:29:37.733622+0000","last_clean":"2024-09-13T14:29:37.733622+0000","last_became_active":"2024-09-13T14:26:15.823041+0000","last_became_peered":"2024-09-13T14:26:15.823041+0000","last_unstale":"2024-09-13T14:29:37.733622+0000","last_undegraded":"2024-09-13T14:29:37.733622+0000","last_fullsized":"2024-09-13T14:29:37.733622+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:23:08.158576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":578,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.727450+0000","last_change":"2024-09-13T14:27:12.848354+0000","last_active":"2024-09-13T14:29:37.727450+0000","last_peered":"2024-09-13T14:29:37.727450+0000","last_clean":"2024-09-13T14:29:37.727450+0000","last_became_active":"2024-09-13T14:27:12.832182+0000","last_became_peered":"2024-09-13T14:27:12.832182+0000","last_unstale":"2024-09-13T14:29:37.727450+0000","last_undegraded":"2024-09-13T14:29:37.727450+0000","last_fullsized":"2024-09-13T14:29:37.727450+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:46:42.064509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":96,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.742671+0000","last_change":"2024-09-13T14:11:30.256034+0000","last_active":"2024-09-13T14:29:37.742671+0000","last_peered":"2024-09-13T14:29:37.742671+0000","last_clean":"2024-09-13T14:29:37.742671+0000","last_became_active":"2024-09-13T14:11:30.255727+0000","last_became_peered":"2024-09-13T14:11:30.255727+0000","last_unstale":"2024-09-13T14:29:37.742671+0000","last_undegraded":"2024-09-13T14:29:37.742671+0000","last_fullsized":"2024-09-13T14:29:37.742671+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:21:35.817014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":119,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.740323+0000","last_change":"2024-09-13T14:26:15.825390+0000","last_active":"2024-09-13T14:29:37.740323+0000","last_peered":"2024-09-13T14:29:37.740323+0000","last_clean":"2024-09-13T14:29:37.740323+0000","last_became_active":"2024-09-13T14:26:15.825084+0000","last_became_peered":"2024-09-13T14:26:15.825084+0000","last_unstale":"2024-09-13T14:29:37.740323+0000","last_undegraded":"2024-09-13T14:29:37.740323+0000","last_fullsized":"2024-09-13T14:29:37.740323+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:44.680451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":119,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.741045+0000","last_change":"2024-09-13T14:11:29.237013+0000","last_active":"2024-09-13T14:29:37.741045+0000","last_peered":"2024-09-13T14:29:37.741045+0000","last_clean":"2024-09-13T14:29:37.741045+0000","last_became_active":"2024-09-13T14:11:29.236643+0000","last_became_peered":"2024-09-13T14:11:29.236643+0000","last_unstale":"2024-09-13T14:29:37.741045+0000","last_undegraded":"2024-09-13T14:29:37.741045+0000","last_fullsized":"2024-09-13T14:29:37.741045+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:33:58.438693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":93,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T14:29:30.706755+0000","last_change":"2024-09-13T14:27:12.843904+0000","last_active":"2024-09-13T14:29:30.706755+0000","last_peered":"2024-09-13T14:29:30.706755+0000","last_clean":"2024-09-13T14:29:30.706755+0000","last_became_active":"2024-09-13T14:27:12.842566+0000","last_became_peered":"2024-09-13T14:27:12.842566+0000","last_unstale":"2024-09-13T14:29:30.706755+0000","last_undegraded":"2024-09-13T14:29:30.706755+0000","last_fullsized":"2024-09-13T14:29:30.706755+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:11:46.253003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":115,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.733307+0000","last_change":"2024-09-13T14:24:55.287275+0000","last_active":"2024-09-13T14:29:37.733307+0000","last_peered":"2024-09-13T14:29:37.733307+0000","last_clean":"2024-09-13T14:29:37.733307+0000","last_became_active":"2024-09-13T14:24:55.286919+0000","last_became_peered":"2024-09-13T14:24:55.286919+0000","last_unstale":"2024-09-13T14:29:37.733307+0000","last_undegraded":"2024-09-13T14:29:37.733307+0000","last_fullsized":"2024-09-13T14:29:37.733307+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:10.790711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":120,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T14:29:30.718667+0000","last_change":"2024-09-13T14:26:15.819037+0000","last_active":"2024-09-13T14:29:30.718667+0000","last_peered":"2024-09-13T14:29:30.718667+0000","last_clean":"2024-09-13T14:29:30.718667+0000","last_became_active":"2024-09-13T14:26:15.818721+0000","last_became_peered":"2024-09-13T14:26:15.818721+0000","last_unstale":"2024-09-13T14:29:30.718667+0000","last_undegraded":"2024-09-13T14:29:30.718667+0000","last_fullsized":"2024-09-13T14:29:30.718667+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:44:06.419136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":103,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.780233+0000","last_change":"2024-09-13T14:24:55.287509+0000","last_active":"2024-09-13T14:29:46.780233+0000","last_peered":"2024-09-13T14:29:46.780233+0000","last_clean":"2024-09-13T14:29:46.780233+0000","last_became_active":"2024-09-13T14:24:55.287193+0000","last_became_peered":"2024-09-13T14:24:55.287193+0000","last_unstale":"2024-09-13T14:29:46.780233+0000","last_undegraded":"2024-09-13T14:29:46.780233+0000","last_fullsized":"2024-09-13T14:29:46.780233+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:14:10.603307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":141,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.733697+0000","last_change":"2024-09-13T14:24:55.289055+0000","last_active":"2024-09-13T14:29:37.733697+0000","last_peered":"2024-09-13T14:29:37.733697+0000","last_clean":"2024-09-13T14:29:37.733697+0000","last_became_active":"2024-09-13T14:24:55.288762+0000","last_became_peered":"2024-09-13T14:24:55.288762+0000","last_unstale":"2024-09-13T14:29:37.733697+0000","last_undegraded":"2024-09-13T14:29:37.733697+0000","last_fullsized":"2024-09-13T14:29:37.733697+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:37:26.639877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":159,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.780297+0000","last_change":"2024-09-13T14:26:15.824540+0000","last_active":"2024-09-13T14:29:46.780297+0000","last_peered":"2024-09-13T14:29:46.780297+0000","last_clean":"2024-09-13T14:29:46.780297+0000","last_became_active":"2024-09-13T14:26:15.824118+0000","last_became_peered":"2024-09-13T14:26:15.824118+0000","last_unstale":"2024-09-13T14:29:46.780297+0000","last_undegraded":"2024-09-13T14:29:46.780297+0000","last_fullsized":"2024-09-13T14:29:46.780297+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:56.677401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":120,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T14:29:30.710239+0000","last_change":"2024-09-13T14:26:15.823892+0000","last_active":"2024-09-13T14:29:30.710239+0000","last_peered":"2024-09-13T14:29:30.710239+0000","last_clean":"2024-09-13T14:29:30.710239+0000","last_became_active":"2024-09-13T14:26:15.823582+0000","last_became_peered":"2024-09-13T14:26:15.823582+0000","last_unstale":"2024-09-13T14:29:30.710239+0000","last_undegraded":"2024-09-13T14:29:30.710239+0000","last_fullsized":"2024-09-13T14:29:30.710239+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:53:09.918997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":182,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.780447+0000","last_change":"2024-09-13T14:27:12.830642+0000","last_active":"2024-09-13T14:29:46.780447+0000","last_peered":"2024-09-13T14:29:46.780447+0000","last_clean":"2024-09-13T14:29:46.780447+0000","last_became_active":"2024-09-13T14:27:12.830026+0000","last_became_peered":"2024-09-13T14:27:12.830026+0000","last_unstale":"2024-09-13T14:29:46.780447+0000","last_undegraded":"2024-09-13T14:29:46.780447+0000","last_fullsized":"2024-09-13T14:29:46.780447+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:12:31.783552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":143,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.733848+0000","last_change":"2024-09-13T14:24:55.283070+0000","last_active":"2024-09-13T14:29:37.733848+0000","last_peered":"2024-09-13T14:29:37.733848+0000","last_clean":"2024-09-13T14:29:37.733848+0000","last_became_active":"2024-09-13T14:24:55.282591+0000","last_became_peered":"2024-09-13T14:24:55.282591+0000","last_unstale":"2024-09-13T14:29:37.733848+0000","last_undegraded":"2024-09-13T14:29:37.733848+0000","last_fullsized":"2024-09-13T14:29:37.733848+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:35:21.896294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":131,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.780420+0000","last_change":"2024-09-13T14:26:15.823524+0000","last_active":"2024-09-13T14:29:46.780420+0000","last_peered":"2024-09-13T14:29:46.780420+0000","last_clean":"2024-09-13T14:29:46.780420+0000","last_became_active":"2024-09-13T14:26:15.823162+0000","last_became_peered":"2024-09-13T14:26:15.823162+0000","last_unstale":"2024-09-13T14:29:46.780420+0000","last_undegraded":"2024-09-13T14:29:46.780420+0000","last_fullsized":"2024-09-13T14:29:46.780420+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:17:13.664812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":171,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.781646+0000","last_change":"2024-09-13T14:26:15.823703+0000","last_active":"2024-09-13T14:29:46.781646+0000","last_peered":"2024-09-13T14:29:46.781646+0000","last_clean":"2024-09-13T14:29:46.781646+0000","last_became_active":"2024-09-13T14:26:15.823249+0000","last_became_peered":"2024-09-13T14:26:15.823249+0000","last_unstale":"2024-09-13T14:29:46.781646+0000","last_undegraded":"2024-09-13T14:29:46.781646+0000","last_fullsized":"2024-09-13T14:29:46.781646+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:42:40.209150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":145,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.743258+0000","last_change":"2024-09-13T14:27:12.841881+0000","last_active":"2024-09-13T14:29:37.743258+0000","last_peered":"2024-09-13T14:29:37.743258+0000","last_clean":"2024-09-13T14:29:37.743258+0000","last_became_active":"2024-09-13T14:27:12.841247+0000","last_became_peered":"2024-09-13T14:27:12.841247+0000","last_unstale":"2024-09-13T14:29:37.743258+0000","last_undegraded":"2024-09-13T14:29:37.743258+0000","last_fullsized":"2024-09-13T14:29:37.743258+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:59:16.937848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":155,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.781789+0000","last_change":"2024-09-13T14:27:12.838453+0000","last_active":"2024-09-13T14:29:46.781789+0000","last_peered":"2024-09-13T14:29:46.781789+0000","last_clean":"2024-09-13T14:29:46.781789+0000","last_became_active":"2024-09-13T14:27:12.838124+0000","last_became_peered":"2024-09-13T14:27:12.838124+0000","last_unstale":"2024-09-13T14:29:46.781789+0000","last_undegraded":"2024-09-13T14:29:46.781789+0000","last_fullsized":"2024-09-13T14:29:46.781789+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:24.259191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":134,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.735192+0000","last_change":"2024-09-13T14:23:18.035101+0000","last_active":"2024-09-13T14:29:37.735192+0000","last_peered":"2024-09-13T14:29:37.735192+0000","last_clean":"2024-09-13T14:29:37.735192+0000","last_became_active":"2024-09-13T14:23:18.034724+0000","last_became_peered":"2024-09-13T14:23:18.034724+0000","last_unstale":"2024-09-13T14:29:37.735192+0000","last_undegraded":"2024-09-13T14:29:37.735192+0000","last_fullsized":"2024-09-13T14:29:37.735192+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:39.358144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":143,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.740258+0000","last_change":"2024-09-13T14:27:12.834931+0000","last_active":"2024-09-13T14:29:37.740258+0000","last_peered":"2024-09-13T14:29:37.740258+0000","last_clean":"2024-09-13T14:29:37.740258+0000","last_became_active":"2024-09-13T14:27:12.834572+0000","last_became_peered":"2024-09-13T14:27:12.834572+0000","last_unstale":"2024-09-13T14:29:37.740258+0000","last_undegraded":"2024-09-13T14:29:37.740258+0000","last_fullsized":"2024-09-13T14:29:37.740258+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:54:21.155689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":126,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T14:29:30.715406+0000","last_change":"2024-09-13T14:26:15.826283+0000","last_active":"2024-09-13T14:29:30.715406+0000","last_peered":"2024-09-13T14:29:30.715406+0000","last_clean":"2024-09-13T14:29:30.715406+0000","last_became_active":"2024-09-13T14:26:15.820774+0000","last_became_peered":"2024-09-13T14:26:15.820774+0000","last_unstale":"2024-09-13T14:29:30.715406+0000","last_undegraded":"2024-09-13T14:29:30.715406+0000","last_fullsized":"2024-09-13T14:29:30.715406+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:28:39.849420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":125,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.743132+0000","last_change":"2024-09-13T14:24:55.285384+0000","last_active":"2024-09-13T14:29:37.743132+0000","last_peered":"2024-09-13T14:29:37.743132+0000","last_clean":"2024-09-13T14:29:37.743132+0000","last_became_active":"2024-09-13T14:24:55.285081+0000","last_became_peered":"2024-09-13T14:24:55.285081+0000","last_unstale":"2024-09-13T14:29:37.743132+0000","last_undegraded":"2024-09-13T14:29:37.743132+0000","last_fullsized":"2024-09-13T14:29:37.743132+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:47:07.946120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":98,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-13T14:29:30.716490+0000","last_change":"2024-09-13T14:11:29.240351+0000","last_active":"2024-09-13T14:29:30.716490+0000","last_peered":"2024-09-13T14:29:30.716490+0000","last_clean":"2024-09-13T14:29:30.716490+0000","last_became_active":"2024-09-13T14:11:29.240060+0000","last_became_peered":"2024-09-13T14:11:29.240060+0000","last_unstale":"2024-09-13T14:29:30.716490+0000","last_undegraded":"2024-09-13T14:29:30.716490+0000","last_fullsized":"2024-09-13T14:29:30.716490+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:34:36.126368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":178,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.780592+0000","last_change":"2024-09-13T14:27:12.839943+0000","last_active":"2024-09-13T14:29:46.780592+0000","last_peered":"2024-09-13T14:29:46.780592+0000","last_clean":"2024-09-13T14:29:46.780592+0000","last_became_active":"2024-09-13T14:27:12.839534+0000","last_became_peered":"2024-09-13T14:27:12.839534+0000","last_unstale":"2024-09-13T14:29:46.780592+0000","last_undegraded":"2024-09-13T14:29:46.780592+0000","last_fullsized":"2024-09-13T14:29:46.780592+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:34:56.677019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":166,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.782184+0000","last_change":"2024-09-13T14:27:12.836567+0000","last_active":"2024-09-13T14:29:46.782184+0000","last_peered":"2024-09-13T14:29:46.782184+0000","last_clean":"2024-09-13T14:29:46.782184+0000","last_became_active":"2024-09-13T14:27:12.836271+0000","last_became_peered":"2024-09-13T14:27:12.836271+0000","last_unstale":"2024-09-13T14:29:46.782184+0000","last_undegraded":"2024-09-13T14:29:46.782184+0000","last_fullsized":"2024-09-13T14:29:46.782184+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:25:32.910438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":119,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.754793+0000","last_change":"2024-09-13T14:26:15.824075+0000","last_active":"2024-09-13T14:29:37.754793+0000","last_peered":"2024-09-13T14:29:37.754793+0000","last_clean":"2024-09-13T14:29:37.754793+0000","last_became_active":"2024-09-13T14:26:15.823762+0000","last_became_peered":"2024-09-13T14:26:15.823762+0000","last_unstale":"2024-09-13T14:29:37.754793+0000","last_undegraded":"2024-09-13T14:29:37.754793+0000","last_fullsized":"2024-09-13T14:29:37.754793+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:26:51.088388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":159,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.733597+0000","last_change":"2024-09-13T14:27:12.844201+0000","last_active":"2024-09-13T14:29:37.733597+0000","last_peered":"2024-09-13T14:29:37.733597+0000","last_clean":"2024-09-13T14:29:37.733597+0000","last_became_active":"2024-09-13T14:27:12.834075+0000","last_became_peered":"2024-09-13T14:27:12.834075+0000","last_unstale":"2024-09-13T14:29:37.733597+0000","last_undegraded":"2024-09-13T14:29:37.733597+0000","last_fullsized":"2024-09-13T14:29:37.733597+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:06.657030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":2825781272,"num_objects":43119,"num_object_clones":0,"num_object_copies":129357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21392,"num_objects_misplaced":733,"num_objects_unfound":0,"num_objects_dirty":43119,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":192713,"num_write_kb":7546561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8196,"num_bytes_recovered":537133056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10021777408,"data_stored":10021765192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108513,"ondisk_log_size":108513,"up":48,"acting":43,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":616,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":17278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18272,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16793600,"data_stored":16777292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":23,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234231,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1177860,"kb_used_data":1052412,"kb_used_omap":1,"kb_used_meta":125438,"kb_avail":92567292,"statfs":{"total":95995035648,"available":94788907008,"internally_reserved":0,"allocated":1077669888,"data_stored":1076910868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":128448971},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495867,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1142016,"kb_used_data":1038712,"kb_used_omap":1,"kb_used_meta":103294,"kb_avail":92603136,"statfs":{"total":95995035648,"available":94825611264,"internally_reserved":0,"allocated":1063641088,"data_stored":1062886687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":105773511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757504,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1932912,"kb_used_data":1757416,"kb_used_omap":1,"kb_used_meta":175486,"kb_avail":91812240,"statfs":{"total":95995035648,"available":94015733760,"internally_reserved":0,"allocated":1799593984,"data_stored":1798850599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":179698118},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019140,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2033336,"kb_used_data":1861936,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":91711816,"statfs":{"total":95995035648,"available":93912899584,"internally_reserved":0,"allocated":1906622464,"data_stored":1905875152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":175503814},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435325,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":598004,"kb_used_data":521900,"kb_used_omap":5,"kb_used_meta":76090,"kb_avail":93147148,"statfs":{"total":95995035648,"available":95382679552,"internally_reserved":0,"allocated":534425600,"data_stored":533686172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":77916286},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":446676598851,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":425201762378,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726925904,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2241784,"kb_used_data":2029616,"kb_used_omap":7,"kb_used_meta":212152,"kb_avail":91503368,"statfs":{"total":95995035648,"available":93699448832,"internally_reserved":0,"allocated":2078326784,"data_stored":2077576046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":217243944},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2050166784,"data_stored":2050162712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1666449408,"data_stored":1666449408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":524353536,"data_stored":524353536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1894252544,"data_stored":1894252544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1773801472,"data_stored":1773797400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1048903680,"data_stored":1048903680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1063849984,"data_stored":1063845912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:29:51.272 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:29:51.319 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:51.347 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:51.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:51.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:51.449 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:51.474 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:51.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:51.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:51.576 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:51.601 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:51.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:51.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:51.703 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:51.707 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T14:29:51.748 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:51.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:51.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:51.850 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:51.881 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:51.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:51.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:51.953 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:51 smithi102 ceph-mon[110183]: from='client.44834 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:29:51.958 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:51 smithi102 ceph-mon[115269]: from='client.44834 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:29:51.983 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:52.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:52.023 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:52.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:52.124 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:52.157 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:52.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:52.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:52.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:51 smithi195 ceph-mon[104262]: from='client.44834 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:29:52.259 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:52.297 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:52.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:52.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:52.399 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:52.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:52.443 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:52.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:52.544 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:52.579 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:52.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:52.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:52.681 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:52.707 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:52.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:52.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:52.810 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:52.835 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:52.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:52.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:52.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:52 smithi102 ceph-mon[110183]: pgmap v434: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.6 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 681 op/s; 21404/129441 objects degraded (16.536%); 733/129441 objects misplaced (0.566%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:29:52.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:52 smithi102 ceph-mon[115269]: pgmap v434: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.6 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 681 op/s; 21404/129441 objects degraded (16.536%); 733/129441 objects misplaced (0.566%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:29:52.938 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:52.962 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:52.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:52.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:53.055 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:29:53.055 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:29:53.064 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:53.114 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:53.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:53.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:53.216 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:29:53.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:52 smithi195 ceph-mon[104262]: pgmap v434: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.6 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 681 op/s; 21404/129441 objects degraded (16.536%); 733/129441 objects misplaced (0.566%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:29:53.255 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:53.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:53.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:29:53.357 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:53.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:53.396 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:53.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:53.497 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:53.535 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:53.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:53.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:53.637 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:53.683 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:53.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:53.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:53.785 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:53.826 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:53.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:53.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:53.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:53 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 21404/129441 objects degraded (16.536%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:29:53.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:53 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 21404/129441 objects degraded (16.536%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:29:53.928 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:53.955 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:53.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:53.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:54.057 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:29:54.082 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:54.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:54.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:29:54.184 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:54.226 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:54.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:54.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:54.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:53 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 21404/129441 objects degraded (16.536%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:29:54.329 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:54.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:54.360 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:54.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:54.461 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:54.487 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:54.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:54.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:54.589 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:54.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:54.629 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:54.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:54.730 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:54.754 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:54.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:54.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:54.856 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:54.889 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:54.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:54.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:54.917 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:54 smithi102 ceph-mon[110183]: pgmap v435: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.4 GiB data, 8.2 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 703 op/s; 21527/119235 objects degraded (18.054%); 278/119235 objects misplaced (0.233%); 4.0 MiB/s, 64 objects/s recovering 2024-09-13T14:29:54.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:54 smithi102 ceph-mon[115269]: pgmap v435: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.4 GiB data, 8.2 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 703 op/s; 21527/119235 objects degraded (18.054%); 278/119235 objects misplaced (0.233%); 4.0 MiB/s, 64 objects/s recovering 2024-09-13T14:29:54.991 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:55.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:55.022 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:55.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:55.123 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:55.149 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:54 smithi195 ceph-mon[104262]: pgmap v435: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.4 GiB data, 8.2 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 703 op/s; 21527/119235 objects degraded (18.054%); 278/119235 objects misplaced (0.233%); 4.0 MiB/s, 64 objects/s recovering 2024-09-13T14:29:55.164 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:55.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:55.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:55.266 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:55.292 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:55.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:55.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:55.394 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:55.437 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:55.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:55.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:55.539 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:55.574 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:55.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:55.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:55.675 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:55.713 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:55.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:55.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:55.815 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:55.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:55.858 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:55.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:55.901 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:55 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/967294163' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:29:55.901 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:55 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:29:55.901 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:55 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:29:55.934 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:29:55.935 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:29:55.935 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:29:55.935 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:29:55.935 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:29:55.935 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:29:55.935 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:29:55.935 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:29:55.935 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:29:55.935 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:29:55.935 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:29:55.935 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:29:55.936 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:29:55.936 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:29:55.936 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:29:55.936 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:29:55.959 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:55.982 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:55.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:55.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:56.081 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:55 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/967294163' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:29:56.081 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:55 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:29:56.081 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:55 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:29:56.084 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:29:56.086 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:29:56.086 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:55 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/967294163' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:29:56.086 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:55 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:29:56.086 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:55 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:29:56.087 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:56.129 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:56.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:56.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:56.231 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:29:56.260 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:56.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:56.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:29:56.362 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:29:56.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:56.391 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:56.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:29:56.492 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:56.526 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:56.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:56.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:56.630 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:29:56.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:56.677 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:56.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:29:56.693 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:29:56 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:29:56] "GET /metrics HTTP/1.1" 200 36530 "" "Prometheus/2.51.0" 2024-09-13T14:29:56.778 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:29:56.791 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:29:56.810 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:56.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:56.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:29:56.912 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:56.951 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:56.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:56.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:57.053 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:57.092 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:57.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:57.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:57.100 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:56 smithi195 ceph-mon[104262]: pgmap v436: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.4 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 709 op/s; 21527/118446 objects degraded (18.175%); 278/118446 objects misplaced (0.235%); 4.0 MiB/s, 64 objects/s recovering 2024-09-13T14:29:57.100 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:56 smithi195 ceph-mon[104262]: from='client.34905 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:29:57.120 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:56 smithi102 ceph-mon[110183]: pgmap v436: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.4 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 709 op/s; 21527/118446 objects degraded (18.175%); 278/118446 objects misplaced (0.235%); 4.0 MiB/s, 64 objects/s recovering 2024-09-13T14:29:57.120 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:56 smithi102 ceph-mon[110183]: from='client.34905 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:29:57.121 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:56 smithi102 ceph-mon[115269]: pgmap v436: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.4 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 709 op/s; 21527/118446 objects degraded (18.175%); 278/118446 objects misplaced (0.235%); 4.0 MiB/s, 64 objects/s recovering 2024-09-13T14:29:57.121 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:56 smithi102 ceph-mon[115269]: from='client.34905 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:29:57.194 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:29:57.229 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:57.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:57.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:29:57.331 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:57.370 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:57.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:57.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:57.473 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:57.506 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:57.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:57.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:57.608 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:29:57.635 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:57.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:57.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:29:57.737 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:57.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:57.765 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:57.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:57.866 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:57.892 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:57.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:57.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:57.994 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:29:58.040 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:58.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:58.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:29:58.142 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:29:58.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:58.169 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:58.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:29:58.270 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:29:58.302 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:58.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:58.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:29:58.404 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:58.443 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:58.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:58.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:58.545 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:58.589 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:58.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:58.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:29:58.692 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:58.718 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:58.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:58.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:58.720 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":436,"stamp":"2024-09-13T14:29:55.306350+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2590382077,"num_objects":39482,"num_object_clones":0,"num_object_copies":118446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21527,"num_objects_misplaced":278,"num_objects_unfound":0,"num_objects_dirty":39482,"num_whiteouts":0,"num_read":6391,"num_read_kb":13846,"num_write":197501,"num_write_kb":7561602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8612,"num_bytes_recovered":564857051,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109760,"ondisk_log_size":109760,"up":267,"acting":261,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":8518356,"kb_used_data":7642020,"kb_used_omap":19,"kb_used_meta":875884,"kb_avail":553952556,"statfs":{"total":575970213888,"available":567247417344,"internally_reserved":0,"allocated":7825428480,"data_stored":7820914331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20284,"internal_metadata":896905412},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,4,8,6,6,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-464977920,"num_objects":-7095,"num_object_clones":0,"num_object_copies":-21285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":243,"num_objects_misplaced":-890,"num_objects_unfound":0,"num_objects_dirty":-7095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7095,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":647,"num_bytes_recovered":42401792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002548"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":166,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.781711+0000","last_change":"2024-09-13T14:27:12.839984+0000","last_active":"2024-09-13T14:29:46.781711+0000","last_peered":"2024-09-13T14:29:46.781711+0000","last_clean":"2024-09-13T14:29:46.781711+0000","last_became_active":"2024-09-13T14:27:12.839653+0000","last_became_peered":"2024-09-13T14:27:12.839653+0000","last_unstale":"2024-09-13T14:29:46.781711+0000","last_undegraded":"2024-09-13T14:29:46.781711+0000","last_fullsized":"2024-09-13T14:29:46.781711+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:14:13.672830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":129,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.731859+0000","last_change":"2024-09-13T14:27:12.846760+0000","last_active":"2024-09-13T14:29:37.731859+0000","last_peered":"2024-09-13T14:29:37.731859+0000","last_clean":"2024-09-13T14:29:37.731859+0000","last_became_active":"2024-09-13T14:27:12.838041+0000","last_became_peered":"2024-09-13T14:27:12.838041+0000","last_unstale":"2024-09-13T14:29:37.731859+0000","last_undegraded":"2024-09-13T14:29:37.731859+0000","last_fullsized":"2024-09-13T14:29:37.731859+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:22.402859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":111,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.735414+0000","last_change":"2024-09-13T14:24:55.282407+0000","last_active":"2024-09-13T14:29:37.735414+0000","last_peered":"2024-09-13T14:29:37.735414+0000","last_clean":"2024-09-13T14:29:37.735414+0000","last_became_active":"2024-09-13T14:24:55.282083+0000","last_became_peered":"2024-09-13T14:24:55.282083+0000","last_unstale":"2024-09-13T14:29:37.735414+0000","last_undegraded":"2024-09-13T14:29:37.735414+0000","last_fullsized":"2024-09-13T14:29:37.735414+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:23:10.525707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":142,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.787029+0000","last_change":"2024-09-13T14:24:55.284433+0000","last_active":"2024-09-13T14:29:46.787029+0000","last_peered":"2024-09-13T14:29:46.787029+0000","last_clean":"2024-09-13T14:29:46.787029+0000","last_became_active":"2024-09-13T14:24:55.284143+0000","last_became_peered":"2024-09-13T14:24:55.284143+0000","last_unstale":"2024-09-13T14:29:46.787029+0000","last_undegraded":"2024-09-13T14:29:46.787029+0000","last_fullsized":"2024-09-13T14:29:46.787029+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:28.107405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":144,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.780947+0000","last_change":"2024-09-13T14:24:55.282416+0000","last_active":"2024-09-13T14:29:46.780947+0000","last_peered":"2024-09-13T14:29:46.780947+0000","last_clean":"2024-09-13T14:29:46.780947+0000","last_became_active":"2024-09-13T14:24:55.282045+0000","last_became_peered":"2024-09-13T14:24:55.282045+0000","last_unstale":"2024-09-13T14:29:46.780947+0000","last_undegraded":"2024-09-13T14:29:46.780947+0000","last_fullsized":"2024-09-13T14:29:46.780947+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:58.692897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":97,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.732517+0000","last_change":"2024-09-13T14:11:29.244452+0000","last_active":"2024-09-13T14:29:37.732517+0000","last_peered":"2024-09-13T14:29:37.732517+0000","last_clean":"2024-09-13T14:29:37.732517+0000","last_became_active":"2024-09-13T14:11:29.244102+0000","last_became_peered":"2024-09-13T14:11:29.244102+0000","last_unstale":"2024-09-13T14:29:37.732517+0000","last_undegraded":"2024-09-13T14:29:37.732517+0000","last_fullsized":"2024-09-13T14:29:37.732517+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:13:19.165534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":166,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.782361+0000","last_change":"2024-09-13T14:27:12.834300+0000","last_active":"2024-09-13T14:29:46.782361+0000","last_peered":"2024-09-13T14:29:46.782361+0000","last_clean":"2024-09-13T14:29:46.782361+0000","last_became_active":"2024-09-13T14:27:12.834046+0000","last_became_peered":"2024-09-13T14:27:12.834046+0000","last_unstale":"2024-09-13T14:29:46.782361+0000","last_undegraded":"2024-09-13T14:29:46.782361+0000","last_fullsized":"2024-09-13T14:29:46.782361+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:46:58.906862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":120,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.726650+0000","last_change":"2024-09-13T14:23:18.039061+0000","last_active":"2024-09-13T14:29:37.726650+0000","last_peered":"2024-09-13T14:29:37.726650+0000","last_clean":"2024-09-13T14:29:37.726650+0000","last_became_active":"2024-09-13T14:23:18.038661+0000","last_became_peered":"2024-09-13T14:23:18.038661+0000","last_unstale":"2024-09-13T14:29:37.726650+0000","last_undegraded":"2024-09-13T14:29:37.726650+0000","last_fullsized":"2024-09-13T14:29:37.726650+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:05:29.109844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.9","version":"148'12088","reported_seq":12130,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:51.793194+0000","last_change":"2024-09-13T14:26:21.865664+0000","last_active":"2024-09-13T14:29:51.793194+0000","last_peered":"2024-09-13T14:29:51.793194+0000","last_clean":"2024-09-13T14:29:51.793194+0000","last_became_active":"2024-09-13T14:26:21.865396+0000","last_became_peered":"2024-09-13T14:26:21.865396+0000","last_unstale":"2024-09-13T14:29:51.793194+0000","last_undegraded":"2024-09-13T14:29:51.793194+0000","last_fullsized":"2024-09-13T14:29:51.793194+0000","mapping_epoch":116,"log_start":"129'5400","ondisk_log_start":"129'5400","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6688,"log_dups_size":0,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:54:35.654265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159514624,"num_objects":2434,"num_object_clones":0,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12088,"num_write_kb":464704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":114,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.732198+0000","last_change":"2024-09-13T14:23:18.038597+0000","last_active":"2024-09-13T14:29:37.732198+0000","last_peered":"2024-09-13T14:29:37.732198+0000","last_clean":"2024-09-13T14:29:37.732198+0000","last_became_active":"2024-09-13T14:23:18.038240+0000","last_became_peered":"2024-09-13T14:23:18.038240+0000","last_unstale":"2024-09-13T14:29:37.732198+0000","last_undegraded":"2024-09-13T14:29:37.732198+0000","last_fullsized":"2024-09-13T14:29:37.732198+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:27:51.842195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":97,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.735272+0000","last_change":"2024-09-13T14:11:30.254534+0000","last_active":"2024-09-13T14:29:37.735272+0000","last_peered":"2024-09-13T14:29:37.735272+0000","last_clean":"2024-09-13T14:29:37.735272+0000","last_became_active":"2024-09-13T14:11:30.253959+0000","last_became_peered":"2024-09-13T14:11:30.253959+0000","last_unstale":"2024-09-13T14:29:37.735272+0000","last_undegraded":"2024-09-13T14:29:37.735272+0000","last_fullsized":"2024-09-13T14:29:37.735272+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:09:54.025853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"148'12546","reported_seq":12624,"reported_epoch":148,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:29:51.346338+0000","last_change":"2024-09-13T14:27:14.009733+0000","last_active":"2024-09-13T14:29:51.346338+0000","last_peered":"2024-09-13T14:29:51.346338+0000","last_clean":"2024-09-13T14:27:11.786427+0000","last_became_active":"2024-09-13T14:27:13.994962+0000","last_became_peered":"2024-09-13T14:27:13.994962+0000","last_unstale":"2024-09-13T14:29:51.346338+0000","last_undegraded":"2024-09-13T14:27:13.854850+0000","last_fullsized":"2024-09-13T14:27:13.824319+0000","mapping_epoch":125,"log_start":"130'6200","ondisk_log_start":"130'6200","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6346,"log_dups_size":0,"ondisk_log_size":6346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164495360,"num_objects":2510,"num_object_clones":0,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5198,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12546,"num_write_kb":481792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5],"avail_no_missing":["0","2","5"],"object_location_counts":[{"shards":"0,2,5","objects":180},{"shards":"0,5","objects":-2688}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":128,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.732357+0000","last_change":"2024-09-13T14:26:15.828529+0000","last_active":"2024-09-13T14:29:37.732357+0000","last_peered":"2024-09-13T14:29:37.732357+0000","last_clean":"2024-09-13T14:29:37.732357+0000","last_became_active":"2024-09-13T14:26:15.820486+0000","last_became_peered":"2024-09-13T14:26:15.820486+0000","last_unstale":"2024-09-13T14:29:37.732357+0000","last_undegraded":"2024-09-13T14:29:37.732357+0000","last_fullsized":"2024-09-13T14:29:37.732357+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:11:46.283172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":135,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.742537+0000","last_change":"2024-09-13T14:26:15.824632+0000","last_active":"2024-09-13T14:29:37.742537+0000","last_peered":"2024-09-13T14:29:37.742537+0000","last_clean":"2024-09-13T14:29:37.742537+0000","last_became_active":"2024-09-13T14:26:15.824330+0000","last_became_peered":"2024-09-13T14:26:15.824330+0000","last_unstale":"2024-09-13T14:29:37.742537+0000","last_undegraded":"2024-09-13T14:29:37.742537+0000","last_fullsized":"2024-09-13T14:29:37.742537+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:25.288116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"148'11940","reported_seq":21815,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:51.377892+0000","last_change":"2024-09-13T14:28:49.532961+0000","last_active":"2024-09-13T14:29:51.377892+0000","last_peered":"2024-09-13T14:29:51.377892+0000","last_clean":"2024-09-13T14:29:51.377892+0000","last_became_active":"2024-09-13T14:28:49.511508+0000","last_became_peered":"2024-09-13T14:28:49.511508+0000","last_unstale":"2024-09-13T14:29:51.377892+0000","last_undegraded":"2024-09-13T14:29:51.377892+0000","last_fullsized":"2024-09-13T14:29:51.377892+0000","mapping_epoch":137,"log_start":"129'5600","ondisk_log_start":"129'5600","created":116,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6340,"log_dups_size":0,"ondisk_log_size":6340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:42:44.821074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161611776,"num_objects":2466,"num_object_clones":0,"num_object_copies":7398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11940,"num_write_kb":460992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4893,"num_bytes_recovered":320667648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":99,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.733431+0000","last_change":"2024-09-13T14:11:29.232897+0000","last_active":"2024-09-13T14:29:37.733431+0000","last_peered":"2024-09-13T14:29:37.733431+0000","last_clean":"2024-09-13T14:29:37.733431+0000","last_became_active":"2024-09-13T14:11:29.232637+0000","last_became_peered":"2024-09-13T14:11:29.232637+0000","last_unstale":"2024-09-13T14:29:37.733431+0000","last_undegraded":"2024-09-13T14:29:37.733431+0000","last_fullsized":"2024-09-13T14:29:37.733431+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:31:05.856046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":121,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.728484+0000","last_change":"2024-09-13T14:26:15.822452+0000","last_active":"2024-09-13T14:29:37.728484+0000","last_peered":"2024-09-13T14:29:37.728484+0000","last_clean":"2024-09-13T14:29:37.728484+0000","last_became_active":"2024-09-13T14:26:15.822111+0000","last_became_peered":"2024-09-13T14:26:15.822111+0000","last_unstale":"2024-09-13T14:29:37.728484+0000","last_undegraded":"2024-09-13T14:29:37.728484+0000","last_fullsized":"2024-09-13T14:29:37.728484+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:32:53.078767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"148'12427","reported_seq":12485,"reported_epoch":148,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:29:51.886511+0000","last_change":"2024-09-13T14:27:14.040788+0000","last_active":"2024-09-13T14:29:51.886511+0000","last_peered":"2024-09-13T14:29:51.886511+0000","last_clean":"2024-09-13T14:27:11.793438+0000","last_became_active":"2024-09-13T14:27:14.032063+0000","last_became_peered":"2024-09-13T14:27:14.032063+0000","last_unstale":"2024-09-13T14:29:51.886511+0000","last_undegraded":"2024-09-13T14:27:13.861208+0000","last_fullsized":"2024-09-13T14:27:13.827412+0000","mapping_epoch":125,"log_start":"129'6000","ondisk_log_start":"129'6000","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6427,"log_dups_size":0,"ondisk_log_size":6427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:07:10.560903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164560896,"num_objects":2511,"num_object_clones":0,"num_object_copies":7533,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12427,"num_write_kb":478016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3],"avail_no_missing":["4","2","3"],"object_location_counts":[{"shards":"2,3,4","objects":120},{"shards":"3,4","objects":-2567}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":113,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.740014+0000","last_change":"2024-09-13T14:24:55.290798+0000","last_active":"2024-09-13T14:29:37.740014+0000","last_peered":"2024-09-13T14:29:37.740014+0000","last_clean":"2024-09-13T14:29:37.740014+0000","last_became_active":"2024-09-13T14:24:55.290510+0000","last_became_peered":"2024-09-13T14:24:55.290510+0000","last_unstale":"2024-09-13T14:29:37.740014+0000","last_undegraded":"2024-09-13T14:29:37.740014+0000","last_fullsized":"2024-09-13T14:29:37.740014+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:19:03.129132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":151,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.781202+0000","last_change":"2024-09-13T14:26:15.823316+0000","last_active":"2024-09-13T14:29:46.781202+0000","last_peered":"2024-09-13T14:29:46.781202+0000","last_clean":"2024-09-13T14:29:46.781202+0000","last_became_active":"2024-09-13T14:26:15.822980+0000","last_became_peered":"2024-09-13T14:26:15.822980+0000","last_unstale":"2024-09-13T14:29:46.781202+0000","last_undegraded":"2024-09-13T14:29:46.781202+0000","last_fullsized":"2024-09-13T14:29:46.781202+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:33:47.420548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"148'12320","reported_seq":16770,"reported_epoch":148,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T14:29:51.844072+0000","last_change":"2024-09-13T14:28:48.293087+0000","last_active":"2024-09-13T14:29:51.844072+0000","last_peered":"2024-09-13T14:29:51.844072+0000","last_clean":"2024-09-13T14:27:11.806319+0000","last_became_active":"2024-09-13T14:27:14.027828+0000","last_became_peered":"2024-09-13T14:27:14.027828+0000","last_unstale":"2024-09-13T14:29:51.844072+0000","last_undegraded":"2024-09-13T14:27:13.862259+0000","last_fullsized":"2024-09-13T14:27:13.823132+0000","mapping_epoch":125,"log_start":"129'5600","ondisk_log_start":"129'5600","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6720,"log_dups_size":0,"ondisk_log_size":6720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:40:06.560106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163512344,"num_objects":2496,"num_object_clones":0,"num_object_copies":7488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2722,"num_objects_misplaced":139,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12320,"num_write_kb":474049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2196,"num_bytes_recovered":143917056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5],"avail_no_missing":["7","2","5"],"object_location_counts":[{"shards":"2,5,7","objects":139},{"shards":"5,7","objects":-365}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":150,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.781479+0000","last_change":"2024-09-13T14:24:55.284499+0000","last_active":"2024-09-13T14:29:46.781479+0000","last_peered":"2024-09-13T14:29:46.781479+0000","last_clean":"2024-09-13T14:29:46.781479+0000","last_became_active":"2024-09-13T14:24:55.284096+0000","last_became_peered":"2024-09-13T14:24:55.284096+0000","last_unstale":"2024-09-13T14:29:46.781479+0000","last_undegraded":"2024-09-13T14:29:46.781479+0000","last_fullsized":"2024-09-13T14:29:46.781479+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:02:06.836099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":111,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.740963+0000","last_change":"2024-09-13T14:24:55.291371+0000","last_active":"2024-09-13T14:29:37.740963+0000","last_peered":"2024-09-13T14:29:37.740963+0000","last_clean":"2024-09-13T14:29:37.740963+0000","last_became_active":"2024-09-13T14:24:55.291047+0000","last_became_peered":"2024-09-13T14:24:55.291047+0000","last_unstale":"2024-09-13T14:29:37.740963+0000","last_undegraded":"2024-09-13T14:29:37.740963+0000","last_fullsized":"2024-09-13T14:29:37.740963+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:43.912412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"148'12162","reported_seq":12204,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:51.833912+0000","last_change":"2024-09-13T14:26:21.864062+0000","last_active":"2024-09-13T14:29:51.833912+0000","last_peered":"2024-09-13T14:29:51.833912+0000","last_clean":"2024-09-13T14:29:51.833912+0000","last_became_active":"2024-09-13T14:26:21.863721+0000","last_became_peered":"2024-09-13T14:26:21.863721+0000","last_unstale":"2024-09-13T14:29:51.833912+0000","last_undegraded":"2024-09-13T14:29:51.833912+0000","last_fullsized":"2024-09-13T14:29:51.833912+0000","mapping_epoch":116,"log_start":"129'5700","ondisk_log_start":"129'5700","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6462,"log_dups_size":0,"ondisk_log_size":6462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:15:42.744309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155582464,"num_objects":2374,"num_object_clones":0,"num_object_copies":7122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12162,"num_write_kb":465152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":122,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.728625+0000","last_change":"2024-09-13T14:26:15.827333+0000","last_active":"2024-09-13T14:29:37.728625+0000","last_peered":"2024-09-13T14:29:37.728625+0000","last_clean":"2024-09-13T14:29:37.728625+0000","last_became_active":"2024-09-13T14:26:15.826945+0000","last_became_peered":"2024-09-13T14:26:15.826945+0000","last_unstale":"2024-09-13T14:29:37.728625+0000","last_undegraded":"2024-09-13T14:29:37.728625+0000","last_fullsized":"2024-09-13T14:29:37.728625+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:35:24.870052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":144,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.741255+0000","last_change":"2024-09-13T14:27:12.842604+0000","last_active":"2024-09-13T14:29:37.741255+0000","last_peered":"2024-09-13T14:29:37.741255+0000","last_clean":"2024-09-13T14:29:37.741255+0000","last_became_active":"2024-09-13T14:27:12.841643+0000","last_became_peered":"2024-09-13T14:27:12.841643+0000","last_unstale":"2024-09-13T14:29:37.741255+0000","last_undegraded":"2024-09-13T14:29:37.741255+0000","last_fullsized":"2024-09-13T14:29:37.741255+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:34:34.751790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"148'12219","reported_seq":12261,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:51.758183+0000","last_change":"2024-09-13T14:26:21.867125+0000","last_active":"2024-09-13T14:29:51.758183+0000","last_peered":"2024-09-13T14:29:51.758183+0000","last_clean":"2024-09-13T14:29:51.758183+0000","last_became_active":"2024-09-13T14:26:21.866815+0000","last_became_peered":"2024-09-13T14:26:21.866815+0000","last_unstale":"2024-09-13T14:29:51.758183+0000","last_undegraded":"2024-09-13T14:29:51.758183+0000","last_fullsized":"2024-09-13T14:29:51.758183+0000","mapping_epoch":116,"log_start":"122'4300","ondisk_log_start":"122'4300","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":7919,"log_dups_size":0,"ondisk_log_size":7919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:44.218255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163250176,"num_objects":2491,"num_object_clones":0,"num_object_copies":7473,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12219,"num_write_kb":470720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":121,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.742781+0000","last_change":"2024-09-13T14:27:12.842271+0000","last_active":"2024-09-13T14:29:37.742781+0000","last_peered":"2024-09-13T14:29:37.742781+0000","last_clean":"2024-09-13T14:29:37.742781+0000","last_became_active":"2024-09-13T14:27:12.841479+0000","last_became_peered":"2024-09-13T14:27:12.841479+0000","last_unstale":"2024-09-13T14:29:37.742781+0000","last_undegraded":"2024-09-13T14:29:37.742781+0000","last_fullsized":"2024-09-13T14:29:37.742781+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:30:22.746575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":120,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.741301+0000","last_change":"2024-09-13T14:26:15.825304+0000","last_active":"2024-09-13T14:29:37.741301+0000","last_peered":"2024-09-13T14:29:37.741301+0000","last_clean":"2024-09-13T14:29:37.741301+0000","last_became_active":"2024-09-13T14:26:15.825018+0000","last_became_peered":"2024-09-13T14:26:15.825018+0000","last_unstale":"2024-09-13T14:29:37.741301+0000","last_undegraded":"2024-09-13T14:29:37.741301+0000","last_fullsized":"2024-09-13T14:29:37.741301+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:22:59.408928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"148'12292","reported_seq":12334,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:51.883732+0000","last_change":"2024-09-13T14:26:21.865199+0000","last_active":"2024-09-13T14:29:51.883732+0000","last_peered":"2024-09-13T14:29:51.883732+0000","last_clean":"2024-09-13T14:29:51.883732+0000","last_became_active":"2024-09-13T14:26:21.864903+0000","last_became_peered":"2024-09-13T14:26:21.864903+0000","last_unstale":"2024-09-13T14:29:51.883732+0000","last_undegraded":"2024-09-13T14:29:51.883732+0000","last_fullsized":"2024-09-13T14:29:51.883732+0000","mapping_epoch":116,"log_start":"129'5900","ondisk_log_start":"129'5900","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6392,"log_dups_size":0,"ondisk_log_size":6392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:25.014330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161480704,"num_objects":2464,"num_object_clones":0,"num_object_copies":7392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12292,"num_write_kb":472192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":160,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.780528+0000","last_change":"2024-09-13T14:26:15.823898+0000","last_active":"2024-09-13T14:29:46.780528+0000","last_peered":"2024-09-13T14:29:46.780528+0000","last_clean":"2024-09-13T14:29:46.780528+0000","last_became_active":"2024-09-13T14:26:15.823125+0000","last_became_peered":"2024-09-13T14:26:15.823125+0000","last_unstale":"2024-09-13T14:29:46.780528+0000","last_undegraded":"2024-09-13T14:29:46.780528+0000","last_fullsized":"2024-09-13T14:29:46.780528+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:51:27.949528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":122,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.782497+0000","last_change":"2024-09-13T14:26:15.823198+0000","last_active":"2024-09-13T14:29:46.782497+0000","last_peered":"2024-09-13T14:29:46.782497+0000","last_clean":"2024-09-13T14:29:46.782497+0000","last_became_active":"2024-09-13T14:26:15.822717+0000","last_became_peered":"2024-09-13T14:26:15.822717+0000","last_unstale":"2024-09-13T14:29:46.782497+0000","last_undegraded":"2024-09-13T14:29:46.782497+0000","last_fullsized":"2024-09-13T14:29:46.782497+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:22.471622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"148'12399","reported_seq":12445,"reported_epoch":148,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:29:51.826071+0000","last_change":"2024-09-13T14:27:14.052640+0000","last_active":"2024-09-13T14:29:51.826071+0000","last_peered":"2024-09-13T14:29:51.826071+0000","last_clean":"2024-09-13T14:27:11.828545+0000","last_became_active":"2024-09-13T14:27:14.042591+0000","last_became_peered":"2024-09-13T14:27:14.042591+0000","last_unstale":"2024-09-13T14:29:51.826071+0000","last_undegraded":"2024-09-13T14:27:13.862613+0000","last_fullsized":"2024-09-13T14:27:13.823368+0000","mapping_epoch":125,"log_start":"129'5700","ondisk_log_start":"129'5700","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6699,"log_dups_size":0,"ondisk_log_size":6699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:17:38.266600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161153024,"num_objects":2459,"num_object_clones":0,"num_object_copies":7377,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12399,"num_write_kb":475456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5],"avail_no_missing":["7","2","5"],"object_location_counts":[{"shards":"2,5,7","objects":152},{"shards":"5,7","objects":-2663}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":444,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.792830+0000","last_change":"2024-09-13T14:23:18.032973+0000","last_active":"2024-09-13T14:29:46.792830+0000","last_peered":"2024-09-13T14:29:46.792830+0000","last_clean":"2024-09-13T14:29:46.792830+0000","last_became_active":"2024-09-13T14:23:18.032579+0000","last_became_peered":"2024-09-13T14:23:18.032579+0000","last_unstale":"2024-09-13T14:29:46.792830+0000","last_undegraded":"2024-09-13T14:29:46.792830+0000","last_fullsized":"2024-09-13T14:29:46.792830+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:07:41.927740+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:17.109445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":477,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.742404+0000","last_change":"2024-09-13T14:27:15.865400+0000","last_active":"2024-09-13T14:29:37.742404+0000","last_peered":"2024-09-13T14:29:37.742404+0000","last_clean":"2024-09-13T14:29:37.742404+0000","last_became_active":"2024-09-13T14:27:15.864835+0000","last_became_peered":"2024-09-13T14:27:15.864835+0000","last_unstale":"2024-09-13T14:29:37.742404+0000","last_undegraded":"2024-09-13T14:29:37.742404+0000","last_fullsized":"2024-09-13T14:29:37.742404+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:27:10.928041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":101,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.783467+0000","last_change":"2024-09-13T14:24:55.288058+0000","last_active":"2024-09-13T14:29:46.783467+0000","last_peered":"2024-09-13T14:29:46.783467+0000","last_clean":"2024-09-13T14:29:46.783467+0000","last_became_active":"2024-09-13T14:24:55.287601+0000","last_became_peered":"2024-09-13T14:24:55.287601+0000","last_unstale":"2024-09-13T14:29:46.783467+0000","last_undegraded":"2024-09-13T14:29:46.783467+0000","last_fullsized":"2024-09-13T14:29:46.783467+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:56:34.622696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":97,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.727861+0000","last_change":"2024-09-13T14:11:30.253749+0000","last_active":"2024-09-13T14:29:37.727861+0000","last_peered":"2024-09-13T14:29:37.727861+0000","last_clean":"2024-09-13T14:29:37.727861+0000","last_became_active":"2024-09-13T14:11:30.253499+0000","last_became_peered":"2024-09-13T14:11:30.253499+0000","last_unstale":"2024-09-13T14:29:37.727861+0000","last_undegraded":"2024-09-13T14:29:37.727861+0000","last_fullsized":"2024-09-13T14:29:37.727861+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:02:00.987335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"148'12460","reported_seq":12502,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:51.758041+0000","last_change":"2024-09-13T14:26:21.865805+0000","last_active":"2024-09-13T14:29:51.758041+0000","last_peered":"2024-09-13T14:29:51.758041+0000","last_clean":"2024-09-13T14:29:51.758041+0000","last_became_active":"2024-09-13T14:26:21.865288+0000","last_became_peered":"2024-09-13T14:26:21.865288+0000","last_unstale":"2024-09-13T14:29:51.758041+0000","last_undegraded":"2024-09-13T14:29:51.758041+0000","last_fullsized":"2024-09-13T14:29:51.758041+0000","mapping_epoch":116,"log_start":"122'4500","ondisk_log_start":"122'4500","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":7960,"log_dups_size":0,"ondisk_log_size":7960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:46.510065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165150720,"num_objects":2520,"num_object_clones":0,"num_object_copies":7560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12460,"num_write_kb":479360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":358,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.731361+0000","last_change":"2024-09-13T14:26:15.835178+0000","last_active":"2024-09-13T14:29:37.731361+0000","last_peered":"2024-09-13T14:29:37.731361+0000","last_clean":"2024-09-13T14:29:37.731361+0000","last_became_active":"2024-09-13T14:26:15.820283+0000","last_became_peered":"2024-09-13T14:26:15.820283+0000","last_unstale":"2024-09-13T14:29:37.731361+0000","last_undegraded":"2024-09-13T14:29:37.731361+0000","last_fullsized":"2024-09-13T14:29:37.731361+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:30.340038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":144,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.780710+0000","last_change":"2024-09-13T14:24:55.286576+0000","last_active":"2024-09-13T14:29:46.780710+0000","last_peered":"2024-09-13T14:29:46.780710+0000","last_clean":"2024-09-13T14:29:46.780710+0000","last_became_active":"2024-09-13T14:24:55.286284+0000","last_became_peered":"2024-09-13T14:24:55.286284+0000","last_unstale":"2024-09-13T14:29:46.780710+0000","last_undegraded":"2024-09-13T14:29:46.780710+0000","last_fullsized":"2024-09-13T14:29:46.780710+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:35.589736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":170,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.781087+0000","last_change":"2024-09-13T14:27:12.836204+0000","last_active":"2024-09-13T14:29:46.781087+0000","last_peered":"2024-09-13T14:29:46.781087+0000","last_clean":"2024-09-13T14:29:46.781087+0000","last_became_active":"2024-09-13T14:27:12.835900+0000","last_became_peered":"2024-09-13T14:27:12.835900+0000","last_unstale":"2024-09-13T14:29:46.781087+0000","last_undegraded":"2024-09-13T14:29:46.781087+0000","last_fullsized":"2024-09-13T14:29:46.781087+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:36:52.121730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"148'12215","reported_seq":15152,"reported_epoch":148,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T14:29:51.764286+0000","last_change":"2024-09-13T14:28:48.300429+0000","last_active":"2024-09-13T14:29:51.764286+0000","last_peered":"2024-09-13T14:29:51.764286+0000","last_clean":"2024-09-13T14:27:11.802376+0000","last_became_active":"2024-09-13T14:27:13.985759+0000","last_became_peered":"2024-09-13T14:27:13.985759+0000","last_unstale":"2024-09-13T14:29:51.764286+0000","last_undegraded":"2024-09-13T14:27:13.852391+0000","last_fullsized":"2024-09-13T14:27:13.824524+0000","mapping_epoch":125,"log_start":"122'4300","ondisk_log_start":"122'4300","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":7915,"log_dups_size":0,"ondisk_log_size":7915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:58:21.117245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167051264,"num_objects":2549,"num_object_clones":0,"num_object_copies":7647,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3395,"num_objects_misplaced":139,"num_objects_unfound":0,"num_objects_dirty":2549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12215,"num_write_kb":472448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1439,"num_bytes_recovered":94306304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7],"avail_no_missing":["5","2","7"],"object_location_counts":[{"shards":"2,5,7","objects":139},{"shards":"5,7","objects":-985}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":524,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.780506+0000","last_change":"2024-09-13T14:27:15.863966+0000","last_active":"2024-09-13T14:29:46.780506+0000","last_peered":"2024-09-13T14:29:46.780506+0000","last_clean":"2024-09-13T14:29:46.780506+0000","last_became_active":"2024-09-13T14:27:15.863124+0000","last_became_peered":"2024-09-13T14:27:15.863124+0000","last_unstale":"2024-09-13T14:29:46.780506+0000","last_undegraded":"2024-09-13T14:29:46.780506+0000","last_fullsized":"2024-09-13T14:29:46.780506+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:35:02.715013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":72,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.726576+0000","last_change":"2024-09-13T14:27:12.829079+0000","last_active":"2024-09-13T14:29:37.726576+0000","last_peered":"2024-09-13T14:29:37.726576+0000","last_clean":"2024-09-13T14:29:37.726576+0000","last_became_active":"2024-09-13T14:27:12.828619+0000","last_became_peered":"2024-09-13T14:27:12.828619+0000","last_unstale":"2024-09-13T14:29:37.726576+0000","last_undegraded":"2024-09-13T14:29:37.726576+0000","last_fullsized":"2024-09-13T14:29:37.726576+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:39:08.260304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":108,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.736592+0000","last_change":"2024-09-13T14:26:15.817732+0000","last_active":"2024-09-13T14:29:37.736592+0000","last_peered":"2024-09-13T14:29:37.736592+0000","last_clean":"2024-09-13T14:29:37.736592+0000","last_became_active":"2024-09-13T14:26:15.817403+0000","last_became_peered":"2024-09-13T14:26:15.817403+0000","last_unstale":"2024-09-13T14:29:37.736592+0000","last_undegraded":"2024-09-13T14:29:37.736592+0000","last_fullsized":"2024-09-13T14:29:37.736592+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:56:39.961300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"148'12119","reported_seq":12196,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:51.395442+0000","last_change":"2024-09-13T14:26:21.866279+0000","last_active":"2024-09-13T14:29:51.395442+0000","last_peered":"2024-09-13T14:29:51.395442+0000","last_clean":"2024-09-13T14:29:51.395442+0000","last_became_active":"2024-09-13T14:26:21.864793+0000","last_became_peered":"2024-09-13T14:26:21.864793+0000","last_unstale":"2024-09-13T14:29:51.395442+0000","last_undegraded":"2024-09-13T14:29:51.395442+0000","last_fullsized":"2024-09-13T14:29:51.395442+0000","mapping_epoch":116,"log_start":"129'5800","ondisk_log_start":"129'5800","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6319,"log_dups_size":0,"ondisk_log_size":6319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:48:28.309201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156827648,"num_objects":2393,"num_object_clones":0,"num_object_copies":7179,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12119,"num_write_kb":464384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":908,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.735316+0000","last_change":"2024-09-13T14:26:18.829131+0000","last_active":"2024-09-13T14:29:37.735316+0000","last_peered":"2024-09-13T14:29:37.735316+0000","last_clean":"2024-09-13T14:29:37.735316+0000","last_became_active":"2024-09-13T14:26:18.828619+0000","last_became_peered":"2024-09-13T14:26:18.828619+0000","last_unstale":"2024-09-13T14:29:37.735316+0000","last_undegraded":"2024-09-13T14:29:37.735316+0000","last_fullsized":"2024-09-13T14:29:37.735316+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:24:37.073618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":112,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.740502+0000","last_change":"2024-09-13T14:23:18.042891+0000","last_active":"2024-09-13T14:29:37.740502+0000","last_peered":"2024-09-13T14:29:37.740502+0000","last_clean":"2024-09-13T14:29:37.740502+0000","last_became_active":"2024-09-13T14:23:18.042562+0000","last_became_peered":"2024-09-13T14:23:18.042562+0000","last_unstale":"2024-09-13T14:29:37.740502+0000","last_undegraded":"2024-09-13T14:29:37.740502+0000","last_fullsized":"2024-09-13T14:29:37.740502+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:30:42.996661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":137,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.781635+0000","last_change":"2024-09-13T14:27:12.832666+0000","last_active":"2024-09-13T14:29:46.781635+0000","last_peered":"2024-09-13T14:29:46.781635+0000","last_clean":"2024-09-13T14:29:46.781635+0000","last_became_active":"2024-09-13T14:27:12.832332+0000","last_became_peered":"2024-09-13T14:27:12.832332+0000","last_unstale":"2024-09-13T14:29:46.781635+0000","last_undegraded":"2024-09-13T14:29:46.781635+0000","last_fullsized":"2024-09-13T14:29:46.781635+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:07:17.724156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"148'12093","reported_seq":12170,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:51.240158+0000","last_change":"2024-09-13T14:26:21.862954+0000","last_active":"2024-09-13T14:29:51.240158+0000","last_peered":"2024-09-13T14:29:51.240158+0000","last_clean":"2024-09-13T14:29:51.240158+0000","last_became_active":"2024-09-13T14:26:21.862650+0000","last_became_peered":"2024-09-13T14:26:21.862650+0000","last_unstale":"2024-09-13T14:29:51.240158+0000","last_undegraded":"2024-09-13T14:29:51.240158+0000","last_fullsized":"2024-09-13T14:29:51.240158+0000","mapping_epoch":116,"log_start":"129'5800","ondisk_log_start":"129'5800","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6293,"log_dups_size":0,"ondisk_log_size":6293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:46:50.507774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158007296,"num_objects":2411,"num_object_clones":0,"num_object_copies":7233,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12093,"num_write_kb":464128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":137,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.735552+0000","last_change":"2024-09-13T14:27:12.843038+0000","last_active":"2024-09-13T14:29:37.735552+0000","last_peered":"2024-09-13T14:29:37.735552+0000","last_clean":"2024-09-13T14:29:37.735552+0000","last_became_active":"2024-09-13T14:27:12.842663+0000","last_became_peered":"2024-09-13T14:27:12.842663+0000","last_unstale":"2024-09-13T14:29:37.735552+0000","last_undegraded":"2024-09-13T14:29:37.735552+0000","last_fullsized":"2024-09-13T14:29:37.735552+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:23:38.880811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":840,"reported_epoch":148,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:29:46.781340+0000","last_change":"2024-09-13T14:27:14.029991+0000","last_active":"2024-09-13T14:29:46.781340+0000","last_peered":"2024-09-13T14:29:46.781340+0000","last_clean":"2024-09-13T14:27:08.815347+0000","last_became_active":"2024-09-13T14:27:13.888039+0000","last_became_peered":"2024-09-13T14:27:13.888039+0000","last_unstale":"2024-09-13T14:29:46.781340+0000","last_undegraded":"2024-09-13T14:27:13.826593+0000","last_fullsized":"2024-09-13T14:27:13.824237+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7],"avail_no_missing":["0","2","7"],"object_location_counts":[{"shards":"0,2,7","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":64,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.735137+0000","last_change":"2024-09-13T14:26:15.819032+0000","last_active":"2024-09-13T14:29:37.735137+0000","last_peered":"2024-09-13T14:29:37.735137+0000","last_clean":"2024-09-13T14:29:37.735137+0000","last_became_active":"2024-09-13T14:26:15.818675+0000","last_became_peered":"2024-09-13T14:26:15.818675+0000","last_unstale":"2024-09-13T14:29:37.735137+0000","last_undegraded":"2024-09-13T14:29:37.735137+0000","last_fullsized":"2024-09-13T14:29:37.735137+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:37:13.628256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"148'12236","reported_seq":12278,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:51.269893+0000","last_change":"2024-09-13T14:26:21.863343+0000","last_active":"2024-09-13T14:29:51.269893+0000","last_peered":"2024-09-13T14:29:51.269893+0000","last_clean":"2024-09-13T14:29:51.269893+0000","last_became_active":"2024-09-13T14:26:21.863057+0000","last_became_peered":"2024-09-13T14:26:21.863057+0000","last_unstale":"2024-09-13T14:29:51.269893+0000","last_undegraded":"2024-09-13T14:29:51.269893+0000","last_fullsized":"2024-09-13T14:29:51.269893+0000","mapping_epoch":116,"log_start":"129'5500","ondisk_log_start":"129'5500","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6736,"log_dups_size":0,"ondisk_log_size":6736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:38:25.794314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164233216,"num_objects":2506,"num_object_clones":0,"num_object_copies":7518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12236,"num_write_kb":471744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":72,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.739600+0000","last_change":"2024-09-13T14:27:12.843268+0000","last_active":"2024-09-13T14:29:37.739600+0000","last_peered":"2024-09-13T14:29:37.739600+0000","last_clean":"2024-09-13T14:29:37.739600+0000","last_became_active":"2024-09-13T14:27:12.842867+0000","last_became_peered":"2024-09-13T14:27:12.842867+0000","last_unstale":"2024-09-13T14:29:37.739600+0000","last_undegraded":"2024-09-13T14:29:37.739600+0000","last_fullsized":"2024-09-13T14:29:37.739600+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:03:59.575866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":635,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.733505+0000","last_change":"2024-09-13T14:23:18.038847+0000","last_active":"2024-09-13T14:29:37.733505+0000","last_peered":"2024-09-13T14:29:37.733505+0000","last_clean":"2024-09-13T14:29:37.733505+0000","last_became_active":"2024-09-13T14:23:18.038188+0000","last_became_peered":"2024-09-13T14:23:18.038188+0000","last_unstale":"2024-09-13T14:29:37.733505+0000","last_undegraded":"2024-09-13T14:29:37.733505+0000","last_fullsized":"2024-09-13T14:29:37.733505+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:30:37.213496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":134,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.781117+0000","last_change":"2024-09-13T14:23:18.034427+0000","last_active":"2024-09-13T14:29:46.781117+0000","last_peered":"2024-09-13T14:29:46.781117+0000","last_clean":"2024-09-13T14:29:46.781117+0000","last_became_active":"2024-09-13T14:23:18.034092+0000","last_became_peered":"2024-09-13T14:23:18.034092+0000","last_unstale":"2024-09-13T14:29:46.781117+0000","last_undegraded":"2024-09-13T14:29:46.781117+0000","last_fullsized":"2024-09-13T14:29:46.781117+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:21.755115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"148'12350","reported_seq":12392,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:51.935039+0000","last_change":"2024-09-13T14:26:21.867237+0000","last_active":"2024-09-13T14:29:51.935039+0000","last_peered":"2024-09-13T14:29:51.935039+0000","last_clean":"2024-09-13T14:29:51.935039+0000","last_became_active":"2024-09-13T14:26:21.866931+0000","last_became_peered":"2024-09-13T14:26:21.866931+0000","last_unstale":"2024-09-13T14:29:51.935039+0000","last_undegraded":"2024-09-13T14:29:51.935039+0000","last_fullsized":"2024-09-13T14:29:51.935039+0000","mapping_epoch":116,"log_start":"129'5900","ondisk_log_start":"129'5900","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6450,"log_dups_size":0,"ondisk_log_size":6450,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:52:57.569446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161480704,"num_objects":2464,"num_object_clones":0,"num_object_copies":7392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12350,"num_write_kb":474048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":150,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.782612+0000","last_change":"2024-09-13T14:24:55.290711+0000","last_active":"2024-09-13T14:29:46.782612+0000","last_peered":"2024-09-13T14:29:46.782612+0000","last_clean":"2024-09-13T14:29:46.782612+0000","last_became_active":"2024-09-13T14:24:55.288549+0000","last_became_peered":"2024-09-13T14:24:55.288549+0000","last_unstale":"2024-09-13T14:29:46.782612+0000","last_undegraded":"2024-09-13T14:29:46.782612+0000","last_fullsized":"2024-09-13T14:29:46.782612+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:55:21.763931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":548,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.726997+0000","last_change":"2024-09-13T14:26:18.827876+0000","last_active":"2024-09-13T14:29:37.726997+0000","last_peered":"2024-09-13T14:29:37.726997+0000","last_clean":"2024-09-13T14:29:37.726997+0000","last_became_active":"2024-09-13T14:26:18.827317+0000","last_became_peered":"2024-09-13T14:26:18.827317+0000","last_unstale":"2024-09-13T14:29:37.726997+0000","last_undegraded":"2024-09-13T14:29:37.726997+0000","last_fullsized":"2024-09-13T14:29:37.726997+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:31:48.611286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":147,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.780373+0000","last_change":"2024-09-13T14:24:55.281593+0000","last_active":"2024-09-13T14:29:46.780373+0000","last_peered":"2024-09-13T14:29:46.780373+0000","last_clean":"2024-09-13T14:29:46.780373+0000","last_became_active":"2024-09-13T14:24:55.280805+0000","last_became_peered":"2024-09-13T14:24:55.280805+0000","last_unstale":"2024-09-13T14:29:46.780373+0000","last_undegraded":"2024-09-13T14:29:46.780373+0000","last_fullsized":"2024-09-13T14:29:46.780373+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:14.806713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"148'12512","reported_seq":12589,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:53.322583+0000","last_change":"2024-09-13T14:26:21.864819+0000","last_active":"2024-09-13T14:29:53.322583+0000","last_peered":"2024-09-13T14:29:53.322583+0000","last_clean":"2024-09-13T14:29:53.322583+0000","last_became_active":"2024-09-13T14:26:21.864511+0000","last_became_peered":"2024-09-13T14:26:21.864511+0000","last_unstale":"2024-09-13T14:29:53.322583+0000","last_undegraded":"2024-09-13T14:29:53.322583+0000","last_fullsized":"2024-09-13T14:29:53.322583+0000","mapping_epoch":116,"log_start":"129'5600","ondisk_log_start":"129'5600","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6912,"log_dups_size":0,"ondisk_log_size":6912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:11:52.262311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":157679616,"num_objects":2406,"num_object_clones":0,"num_object_copies":7218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12512,"num_write_kb":477376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":159,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.780829+0000","last_change":"2024-09-13T14:26:15.830309+0000","last_active":"2024-09-13T14:29:46.780829+0000","last_peered":"2024-09-13T14:29:46.780829+0000","last_clean":"2024-09-13T14:29:46.780829+0000","last_became_active":"2024-09-13T14:26:15.823041+0000","last_became_peered":"2024-09-13T14:26:15.823041+0000","last_unstale":"2024-09-13T14:29:46.780829+0000","last_undegraded":"2024-09-13T14:29:46.780829+0000","last_fullsized":"2024-09-13T14:29:46.780829+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:23:08.158576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":579,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:41.333938+0000","last_change":"2024-09-13T14:27:12.848354+0000","last_active":"2024-09-13T14:29:41.333938+0000","last_peered":"2024-09-13T14:29:41.333938+0000","last_clean":"2024-09-13T14:29:41.333938+0000","last_became_active":"2024-09-13T14:27:12.832182+0000","last_became_peered":"2024-09-13T14:27:12.832182+0000","last_unstale":"2024-09-13T14:29:41.333938+0000","last_undegraded":"2024-09-13T14:29:41.333938+0000","last_fullsized":"2024-09-13T14:29:41.333938+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:46:42.064509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":96,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.742671+0000","last_change":"2024-09-13T14:11:30.256034+0000","last_active":"2024-09-13T14:29:37.742671+0000","last_peered":"2024-09-13T14:29:37.742671+0000","last_clean":"2024-09-13T14:29:37.742671+0000","last_became_active":"2024-09-13T14:11:30.255727+0000","last_became_peered":"2024-09-13T14:11:30.255727+0000","last_unstale":"2024-09-13T14:29:37.742671+0000","last_undegraded":"2024-09-13T14:29:37.742671+0000","last_fullsized":"2024-09-13T14:29:37.742671+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:21:35.817014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":119,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.740323+0000","last_change":"2024-09-13T14:26:15.825390+0000","last_active":"2024-09-13T14:29:37.740323+0000","last_peered":"2024-09-13T14:29:37.740323+0000","last_clean":"2024-09-13T14:29:37.740323+0000","last_became_active":"2024-09-13T14:26:15.825084+0000","last_became_peered":"2024-09-13T14:26:15.825084+0000","last_unstale":"2024-09-13T14:29:37.740323+0000","last_undegraded":"2024-09-13T14:29:37.740323+0000","last_fullsized":"2024-09-13T14:29:37.740323+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:44.680451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":119,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.741045+0000","last_change":"2024-09-13T14:11:29.237013+0000","last_active":"2024-09-13T14:29:37.741045+0000","last_peered":"2024-09-13T14:29:37.741045+0000","last_clean":"2024-09-13T14:29:37.741045+0000","last_became_active":"2024-09-13T14:11:29.236643+0000","last_became_peered":"2024-09-13T14:11:29.236643+0000","last_unstale":"2024-09-13T14:29:37.741045+0000","last_undegraded":"2024-09-13T14:29:37.741045+0000","last_fullsized":"2024-09-13T14:29:37.741045+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:33:58.438693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":94,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.728553+0000","last_change":"2024-09-13T14:27:12.843904+0000","last_active":"2024-09-13T14:29:37.728553+0000","last_peered":"2024-09-13T14:29:37.728553+0000","last_clean":"2024-09-13T14:29:37.728553+0000","last_became_active":"2024-09-13T14:27:12.842566+0000","last_became_peered":"2024-09-13T14:27:12.842566+0000","last_unstale":"2024-09-13T14:29:37.728553+0000","last_undegraded":"2024-09-13T14:29:37.728553+0000","last_fullsized":"2024-09-13T14:29:37.728553+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:11:46.253003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":115,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.733307+0000","last_change":"2024-09-13T14:24:55.287275+0000","last_active":"2024-09-13T14:29:37.733307+0000","last_peered":"2024-09-13T14:29:37.733307+0000","last_clean":"2024-09-13T14:29:37.733307+0000","last_became_active":"2024-09-13T14:24:55.286919+0000","last_became_peered":"2024-09-13T14:24:55.286919+0000","last_unstale":"2024-09-13T14:29:37.733307+0000","last_undegraded":"2024-09-13T14:29:37.733307+0000","last_fullsized":"2024-09-13T14:29:37.733307+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:10.790711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":121,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.741686+0000","last_change":"2024-09-13T14:26:15.819037+0000","last_active":"2024-09-13T14:29:37.741686+0000","last_peered":"2024-09-13T14:29:37.741686+0000","last_clean":"2024-09-13T14:29:37.741686+0000","last_became_active":"2024-09-13T14:26:15.818721+0000","last_became_peered":"2024-09-13T14:26:15.818721+0000","last_unstale":"2024-09-13T14:29:37.741686+0000","last_undegraded":"2024-09-13T14:29:37.741686+0000","last_fullsized":"2024-09-13T14:29:37.741686+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:44:06.419136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":103,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.780233+0000","last_change":"2024-09-13T14:24:55.287509+0000","last_active":"2024-09-13T14:29:46.780233+0000","last_peered":"2024-09-13T14:29:46.780233+0000","last_clean":"2024-09-13T14:29:46.780233+0000","last_became_active":"2024-09-13T14:24:55.287193+0000","last_became_peered":"2024-09-13T14:24:55.287193+0000","last_unstale":"2024-09-13T14:29:46.780233+0000","last_undegraded":"2024-09-13T14:29:46.780233+0000","last_fullsized":"2024-09-13T14:29:46.780233+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:14:10.603307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":143,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.781875+0000","last_change":"2024-09-13T14:24:55.289055+0000","last_active":"2024-09-13T14:29:46.781875+0000","last_peered":"2024-09-13T14:29:46.781875+0000","last_clean":"2024-09-13T14:29:46.781875+0000","last_became_active":"2024-09-13T14:24:55.288762+0000","last_became_peered":"2024-09-13T14:24:55.288762+0000","last_unstale":"2024-09-13T14:29:46.781875+0000","last_undegraded":"2024-09-13T14:29:46.781875+0000","last_fullsized":"2024-09-13T14:29:46.781875+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:37:26.639877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":159,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.780297+0000","last_change":"2024-09-13T14:26:15.824540+0000","last_active":"2024-09-13T14:29:46.780297+0000","last_peered":"2024-09-13T14:29:46.780297+0000","last_clean":"2024-09-13T14:29:46.780297+0000","last_became_active":"2024-09-13T14:26:15.824118+0000","last_became_peered":"2024-09-13T14:26:15.824118+0000","last_unstale":"2024-09-13T14:29:46.780297+0000","last_undegraded":"2024-09-13T14:29:46.780297+0000","last_fullsized":"2024-09-13T14:29:46.780297+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:56.677401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":121,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.727374+0000","last_change":"2024-09-13T14:26:15.823892+0000","last_active":"2024-09-13T14:29:37.727374+0000","last_peered":"2024-09-13T14:29:37.727374+0000","last_clean":"2024-09-13T14:29:37.727374+0000","last_became_active":"2024-09-13T14:26:15.823582+0000","last_became_peered":"2024-09-13T14:26:15.823582+0000","last_unstale":"2024-09-13T14:29:37.727374+0000","last_undegraded":"2024-09-13T14:29:37.727374+0000","last_fullsized":"2024-09-13T14:29:37.727374+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:53:09.918997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":182,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.780447+0000","last_change":"2024-09-13T14:27:12.830642+0000","last_active":"2024-09-13T14:29:46.780447+0000","last_peered":"2024-09-13T14:29:46.780447+0000","last_clean":"2024-09-13T14:29:46.780447+0000","last_became_active":"2024-09-13T14:27:12.830026+0000","last_became_peered":"2024-09-13T14:27:12.830026+0000","last_unstale":"2024-09-13T14:29:46.780447+0000","last_undegraded":"2024-09-13T14:29:46.780447+0000","last_fullsized":"2024-09-13T14:29:46.780447+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:12:31.783552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":145,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.781035+0000","last_change":"2024-09-13T14:24:55.283070+0000","last_active":"2024-09-13T14:29:46.781035+0000","last_peered":"2024-09-13T14:29:46.781035+0000","last_clean":"2024-09-13T14:29:46.781035+0000","last_became_active":"2024-09-13T14:24:55.282591+0000","last_became_peered":"2024-09-13T14:24:55.282591+0000","last_unstale":"2024-09-13T14:29:46.781035+0000","last_undegraded":"2024-09-13T14:29:46.781035+0000","last_fullsized":"2024-09-13T14:29:46.781035+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:35:21.896294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":131,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.780420+0000","last_change":"2024-09-13T14:26:15.823524+0000","last_active":"2024-09-13T14:29:46.780420+0000","last_peered":"2024-09-13T14:29:46.780420+0000","last_clean":"2024-09-13T14:29:46.780420+0000","last_became_active":"2024-09-13T14:26:15.823162+0000","last_became_peered":"2024-09-13T14:26:15.823162+0000","last_unstale":"2024-09-13T14:29:46.780420+0000","last_undegraded":"2024-09-13T14:29:46.780420+0000","last_fullsized":"2024-09-13T14:29:46.780420+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:17:13.664812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":172,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.781691+0000","last_change":"2024-09-13T14:26:15.823703+0000","last_active":"2024-09-13T14:29:46.781691+0000","last_peered":"2024-09-13T14:29:46.781691+0000","last_clean":"2024-09-13T14:29:46.781691+0000","last_became_active":"2024-09-13T14:26:15.823249+0000","last_became_peered":"2024-09-13T14:26:15.823249+0000","last_unstale":"2024-09-13T14:29:46.781691+0000","last_undegraded":"2024-09-13T14:29:46.781691+0000","last_fullsized":"2024-09-13T14:29:46.781691+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:42:40.209150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":145,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.743258+0000","last_change":"2024-09-13T14:27:12.841881+0000","last_active":"2024-09-13T14:29:37.743258+0000","last_peered":"2024-09-13T14:29:37.743258+0000","last_clean":"2024-09-13T14:29:37.743258+0000","last_became_active":"2024-09-13T14:27:12.841247+0000","last_became_peered":"2024-09-13T14:27:12.841247+0000","last_unstale":"2024-09-13T14:29:37.743258+0000","last_undegraded":"2024-09-13T14:29:37.743258+0000","last_fullsized":"2024-09-13T14:29:37.743258+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:59:16.937848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":155,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.781789+0000","last_change":"2024-09-13T14:27:12.838453+0000","last_active":"2024-09-13T14:29:46.781789+0000","last_peered":"2024-09-13T14:29:46.781789+0000","last_clean":"2024-09-13T14:29:46.781789+0000","last_became_active":"2024-09-13T14:27:12.838124+0000","last_became_peered":"2024-09-13T14:27:12.838124+0000","last_unstale":"2024-09-13T14:29:46.781789+0000","last_undegraded":"2024-09-13T14:29:46.781789+0000","last_fullsized":"2024-09-13T14:29:46.781789+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:24.259191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":136,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.781080+0000","last_change":"2024-09-13T14:23:18.035101+0000","last_active":"2024-09-13T14:29:46.781080+0000","last_peered":"2024-09-13T14:29:46.781080+0000","last_clean":"2024-09-13T14:29:46.781080+0000","last_became_active":"2024-09-13T14:23:18.034724+0000","last_became_peered":"2024-09-13T14:23:18.034724+0000","last_unstale":"2024-09-13T14:29:46.781080+0000","last_undegraded":"2024-09-13T14:29:46.781080+0000","last_fullsized":"2024-09-13T14:29:46.781080+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:39.358144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":143,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.740258+0000","last_change":"2024-09-13T14:27:12.834931+0000","last_active":"2024-09-13T14:29:37.740258+0000","last_peered":"2024-09-13T14:29:37.740258+0000","last_clean":"2024-09-13T14:29:37.740258+0000","last_became_active":"2024-09-13T14:27:12.834572+0000","last_became_peered":"2024-09-13T14:27:12.834572+0000","last_unstale":"2024-09-13T14:29:37.740258+0000","last_undegraded":"2024-09-13T14:29:37.740258+0000","last_fullsized":"2024-09-13T14:29:37.740258+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:54:21.155689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":127,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.739940+0000","last_change":"2024-09-13T14:26:15.826283+0000","last_active":"2024-09-13T14:29:37.739940+0000","last_peered":"2024-09-13T14:29:37.739940+0000","last_clean":"2024-09-13T14:29:37.739940+0000","last_became_active":"2024-09-13T14:26:15.820774+0000","last_became_peered":"2024-09-13T14:26:15.820774+0000","last_unstale":"2024-09-13T14:29:37.739940+0000","last_undegraded":"2024-09-13T14:29:37.739940+0000","last_fullsized":"2024-09-13T14:29:37.739940+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:28:39.849420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":125,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.743132+0000","last_change":"2024-09-13T14:24:55.285384+0000","last_active":"2024-09-13T14:29:37.743132+0000","last_peered":"2024-09-13T14:29:37.743132+0000","last_clean":"2024-09-13T14:29:37.743132+0000","last_became_active":"2024-09-13T14:24:55.285081+0000","last_became_peered":"2024-09-13T14:24:55.285081+0000","last_unstale":"2024-09-13T14:29:37.743132+0000","last_undegraded":"2024-09-13T14:29:37.743132+0000","last_fullsized":"2024-09-13T14:29:37.743132+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:47:07.946120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":99,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.735485+0000","last_change":"2024-09-13T14:11:29.240351+0000","last_active":"2024-09-13T14:29:37.735485+0000","last_peered":"2024-09-13T14:29:37.735485+0000","last_clean":"2024-09-13T14:29:37.735485+0000","last_became_active":"2024-09-13T14:11:29.240060+0000","last_became_peered":"2024-09-13T14:11:29.240060+0000","last_unstale":"2024-09-13T14:29:37.735485+0000","last_undegraded":"2024-09-13T14:29:37.735485+0000","last_fullsized":"2024-09-13T14:29:37.735485+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:34:36.126368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":178,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.780592+0000","last_change":"2024-09-13T14:27:12.839943+0000","last_active":"2024-09-13T14:29:46.780592+0000","last_peered":"2024-09-13T14:29:46.780592+0000","last_clean":"2024-09-13T14:29:46.780592+0000","last_became_active":"2024-09-13T14:27:12.839534+0000","last_became_peered":"2024-09-13T14:27:12.839534+0000","last_unstale":"2024-09-13T14:29:46.780592+0000","last_undegraded":"2024-09-13T14:29:46.780592+0000","last_fullsized":"2024-09-13T14:29:46.780592+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:34:56.677019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":166,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.782184+0000","last_change":"2024-09-13T14:27:12.836567+0000","last_active":"2024-09-13T14:29:46.782184+0000","last_peered":"2024-09-13T14:29:46.782184+0000","last_clean":"2024-09-13T14:29:46.782184+0000","last_became_active":"2024-09-13T14:27:12.836271+0000","last_became_peered":"2024-09-13T14:27:12.836271+0000","last_unstale":"2024-09-13T14:29:46.782184+0000","last_undegraded":"2024-09-13T14:29:46.782184+0000","last_fullsized":"2024-09-13T14:29:46.782184+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:25:32.910438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":119,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-13T14:29:37.754793+0000","last_change":"2024-09-13T14:26:15.824075+0000","last_active":"2024-09-13T14:29:37.754793+0000","last_peered":"2024-09-13T14:29:37.754793+0000","last_clean":"2024-09-13T14:29:37.754793+0000","last_became_active":"2024-09-13T14:26:15.823762+0000","last_became_peered":"2024-09-13T14:26:15.823762+0000","last_unstale":"2024-09-13T14:29:37.754793+0000","last_undegraded":"2024-09-13T14:29:37.754793+0000","last_fullsized":"2024-09-13T14:29:37.754793+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:26:51.088388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":161,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-13T14:29:46.781761+0000","last_change":"2024-09-13T14:27:12.844201+0000","last_active":"2024-09-13T14:29:46.781761+0000","last_peered":"2024-09-13T14:29:46.781761+0000","last_clean":"2024-09-13T14:29:46.781761+0000","last_became_active":"2024-09-13T14:27:12.834075+0000","last_became_peered":"2024-09-13T14:27:12.834075+0000","last_unstale":"2024-09-13T14:29:46.781761+0000","last_undegraded":"2024-09-13T14:29:46.781761+0000","last_fullsized":"2024-09-13T14:29:46.781761+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:06.657030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":2585591832,"num_objects":39454,"num_object_clones":0,"num_object_copies":118362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21515,"num_objects_misplaced":278,"num_objects_unfound":0,"num_objects_dirty":39454,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":196378,"num_write_kb":7546561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8528,"num_bytes_recovered":558891008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9377689600,"data_stored":9377677384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108578,"ondisk_log_size":108578,"up":48,"acting":43,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":616,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":17278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18272,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16793600,"data_stored":16777292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":23,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234232,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1100624,"kb_used_data":966984,"kb_used_omap":1,"kb_used_meta":133630,"kb_avail":92644528,"statfs":{"total":95995035648,"available":94867996672,"internally_reserved":0,"allocated":990191616,"data_stored":989428623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":136837579},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495868,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1062540,"kb_used_data":955012,"kb_used_omap":1,"kb_used_meta":107390,"kb_avail":92682612,"statfs":{"total":95995035648,"available":94906994688,"internally_reserved":0,"allocated":977932288,"data_stored":977173914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":109967815},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757505,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1797244,"kb_used_data":1613364,"kb_used_omap":1,"kb_used_meta":183678,"kb_avail":91947908,"statfs":{"total":95995035648,"available":94154657792,"internally_reserved":0,"allocated":1652084736,"data_stored":1651337378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":188086726},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019141,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1910660,"kb_used_data":1731068,"kb_used_omap":1,"kb_used_meta":179582,"kb_avail":91834492,"statfs":{"total":95995035648,"available":94038519808,"internally_reserved":0,"allocated":1772613632,"data_stored":1771862347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":183892422},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,4,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435326,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":537460,"kb_used_data":473836,"kb_used_omap":5,"kb_used_meta":63610,"kb_avail":93207692,"statfs":{"total":95995035648,"available":95444676608,"internally_reserved":0,"allocated":485208064,"data_stored":484468636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":65136766},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":446676598852,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":425201762379,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726925905,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2109828,"kb_used_data":1901756,"kb_used_omap":7,"kb_used_meta":207992,"kb_avail":91635324,"statfs":{"total":95995035648,"available":93834571776,"internally_reserved":0,"allocated":1947398144,"data_stored":1946643433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":212984104},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1917128704,"data_stored":1917124632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1666449408,"data_stored":1666449408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":476053504,"data_stored":476053504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1757216768,"data_stored":1757216768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1625886720,"data_stored":1625882648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":962199552,"data_stored":962199552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":972754944,"data_stored":972750872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:29:58.724 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph status --format=json 2024-09-13T14:29:58.803 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:29:58.819 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:58.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:58.878 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:58.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:58.946 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:58 smithi102 ceph-mon[110183]: pgmap v437: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.2 GiB data, 7.5 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 892 op/s; 21457/108363 objects degraded (19.801%); 5.1 MiB/s, 94 objects/s recovering 2024-09-13T14:29:58.946 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:29:58 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 21457/108363 objects degraded (19.801%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:29:58.947 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:58 smithi102 ceph-mon[115269]: pgmap v437: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.2 GiB data, 7.5 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 892 op/s; 21457/108363 objects degraded (19.801%); 5.1 MiB/s, 94 objects/s recovering 2024-09-13T14:29:58.948 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:29:58 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 21457/108363 objects degraded (19.801%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:29:58.979 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:59.015 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:59.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:59.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:59.116 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:29:59.155 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:59.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:59.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:29:59.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:58 smithi195 ceph-mon[104262]: pgmap v437: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.2 GiB data, 7.5 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 892 op/s; 21457/108363 objects degraded (19.801%); 5.1 MiB/s, 94 objects/s recovering 2024-09-13T14:29:59.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:29:58 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 21457/108363 objects degraded (19.801%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:29:59.258 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:59.284 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:59.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:59.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:59.386 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:59.411 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:59.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:59.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:59.513 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:29:59.556 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:59.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:59.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:29:59.658 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:29:59.683 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:59.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:59.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:29:59.785 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:29:59.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:59.811 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:29:59.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:29:59.911 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:29:59.944 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:29:59.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:59.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:30:00.025 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:30:00.025 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:30:00.045 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:00.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:00.070 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:00.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:00.171 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:00.196 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:00.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:00.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:00.299 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:00.339 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:00.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:00.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:00.441 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:00.465 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:00.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:00.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:00.566 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:00.594 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:00.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:00.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:00.696 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:00.730 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:00.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:00.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:00.832 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:00.867 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:00.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:00.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:00.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:00 smithi102 ceph-mon[110183]: pgmap v438: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.2 GiB data, 7.5 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 622 op/s; 21457/107667 objects degraded (19.929%); 2.8 MiB/s, 56 objects/s recovering 2024-09-13T14:30:00.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:00 smithi102 ceph-mon[110183]: Health detail: HEALTH_WARN noscrub flag(s) set; Degraded data redundancy: 21457/107667 objects degraded (19.929%), 6 pgs degraded, 6 pgs undersized 2024-09-13T14:30:00.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:00 smithi102 ceph-mon[110183]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-09-13T14:30:00.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:00 smithi102 ceph-mon[110183]: [WRN] PG_DEGRADED: Degraded data redundancy: 21457/107667 objects degraded (19.929%), 6 pgs degraded, 6 pgs undersized 2024-09-13T14:30:00.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:00 smithi102 ceph-mon[110183]: pg 2.4 is stuck undersized for 2m, current state active+recovery_wait+undersized+degraded+remapped, last acting [0,7] 2024-09-13T14:30:00.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:00 smithi102 ceph-mon[110183]: pg 5.4 is stuck undersized for 2m, current state active+recovery_wait+undersized+degraded+remapped, last acting [7,5] 2024-09-13T14:30:00.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:00 smithi102 ceph-mon[110183]: pg 5.6 is stuck undersized for 2m, current state active+recovering+undersized+degraded+remapped, last acting [5,7] 2024-09-13T14:30:00.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:00 smithi102 ceph-mon[110183]: pg 5.8 is stuck undersized for 2m, current state active+recovery_wait+undersized+degraded+remapped, last acting [0,5] 2024-09-13T14:30:00.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:00 smithi102 ceph-mon[110183]: pg 5.a is stuck undersized for 2m, current state active+recovery_wait+undersized+degraded+remapped, last acting [4,3] 2024-09-13T14:30:00.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:00 smithi102 ceph-mon[110183]: pg 5.d is stuck undersized for 2m, current state active+recovering+undersized+degraded+remapped, last acting [7,5] 2024-09-13T14:30:00.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:00 smithi102 ceph-mon[115269]: pgmap v438: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.2 GiB data, 7.5 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 622 op/s; 21457/107667 objects degraded (19.929%); 2.8 MiB/s, 56 objects/s recovering 2024-09-13T14:30:00.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:00 smithi102 ceph-mon[115269]: Health detail: HEALTH_WARN noscrub flag(s) set; Degraded data redundancy: 21457/107667 objects degraded (19.929%), 6 pgs degraded, 6 pgs undersized 2024-09-13T14:30:00.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:00 smithi102 ceph-mon[115269]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-09-13T14:30:00.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:00 smithi102 ceph-mon[115269]: [WRN] PG_DEGRADED: Degraded data redundancy: 21457/107667 objects degraded (19.929%), 6 pgs degraded, 6 pgs undersized 2024-09-13T14:30:00.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:00 smithi102 ceph-mon[115269]: pg 2.4 is stuck undersized for 2m, current state active+recovery_wait+undersized+degraded+remapped, last acting [0,7] 2024-09-13T14:30:00.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:00 smithi102 ceph-mon[115269]: pg 5.4 is stuck undersized for 2m, current state active+recovery_wait+undersized+degraded+remapped, last acting [7,5] 2024-09-13T14:30:00.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:00 smithi102 ceph-mon[115269]: pg 5.6 is stuck undersized for 2m, current state active+recovering+undersized+degraded+remapped, last acting [5,7] 2024-09-13T14:30:00.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:00 smithi102 ceph-mon[115269]: pg 5.8 is stuck undersized for 2m, current state active+recovery_wait+undersized+degraded+remapped, last acting [0,5] 2024-09-13T14:30:00.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:00 smithi102 ceph-mon[115269]: pg 5.a is stuck undersized for 2m, current state active+recovery_wait+undersized+degraded+remapped, last acting [4,3] 2024-09-13T14:30:00.921 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:00 smithi102 ceph-mon[115269]: pg 5.d is stuck undersized for 2m, current state active+recovering+undersized+degraded+remapped, last acting [7,5] 2024-09-13T14:30:00.969 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:01.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:01.003 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:01.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:01.104 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:01.130 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:01.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:01.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:01.232 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:01.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:00 smithi195 ceph-mon[104262]: pgmap v438: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.2 GiB data, 7.5 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 622 op/s; 21457/107667 objects degraded (19.929%); 2.8 MiB/s, 56 objects/s recovering 2024-09-13T14:30:01.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:00 smithi195 ceph-mon[104262]: Health detail: HEALTH_WARN noscrub flag(s) set; Degraded data redundancy: 21457/107667 objects degraded (19.929%), 6 pgs degraded, 6 pgs undersized 2024-09-13T14:30:01.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:00 smithi195 ceph-mon[104262]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-09-13T14:30:01.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:00 smithi195 ceph-mon[104262]: [WRN] PG_DEGRADED: Degraded data redundancy: 21457/107667 objects degraded (19.929%), 6 pgs degraded, 6 pgs undersized 2024-09-13T14:30:01.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:00 smithi195 ceph-mon[104262]: pg 2.4 is stuck undersized for 2m, current state active+recovery_wait+undersized+degraded+remapped, last acting [0,7] 2024-09-13T14:30:01.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:00 smithi195 ceph-mon[104262]: pg 5.4 is stuck undersized for 2m, current state active+recovery_wait+undersized+degraded+remapped, last acting [7,5] 2024-09-13T14:30:01.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:00 smithi195 ceph-mon[104262]: pg 5.6 is stuck undersized for 2m, current state active+recovering+undersized+degraded+remapped, last acting [5,7] 2024-09-13T14:30:01.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:00 smithi195 ceph-mon[104262]: pg 5.8 is stuck undersized for 2m, current state active+recovery_wait+undersized+degraded+remapped, last acting [0,5] 2024-09-13T14:30:01.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:00 smithi195 ceph-mon[104262]: pg 5.a is stuck undersized for 2m, current state active+recovery_wait+undersized+degraded+remapped, last acting [4,3] 2024-09-13T14:30:01.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:00 smithi195 ceph-mon[104262]: pg 5.d is stuck undersized for 2m, current state active+recovering+undersized+degraded+remapped, last acting [7,5] 2024-09-13T14:30:01.269 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:01.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:01.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:01.371 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:01.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:01.411 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:01.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:01.512 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:30:01.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:01.566 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:01.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:30:01.667 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:01.695 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:01.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:01.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:01.796 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:01.821 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:01.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:01.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:01.923 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:01.956 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:01.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:01.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:02.013 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T14:30:02.058 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:02.098 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:02.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:02.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:02.200 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:02.226 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:02.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:02.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:02.328 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:02.366 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:02.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:02.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:02.468 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:02.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:02.492 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:02.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:02.593 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:30:02.593 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:02.597 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:30:02.656 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:02.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:02.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:02.758 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:30:02.803 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:02.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:02.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:30:02.905 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:30:02.941 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:02.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:02.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:30:03.043 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:03.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:03.071 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:03.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:03.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:02 smithi102 ceph-mon[110183]: pgmap v439: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.2 GiB data, 7.5 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 604 op/s; 21457/107667 objects degraded (19.929%); 2.8 MiB/s, 56 objects/s recovering 2024-09-13T14:30:03.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:02 smithi102 ceph-mon[115269]: pgmap v439: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.2 GiB data, 7.5 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 604 op/s; 21457/107667 objects degraded (19.929%); 2.8 MiB/s, 56 objects/s recovering 2024-09-13T14:30:03.172 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:03.200 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:03.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:03.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:03.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:02 smithi195 ceph-mon[104262]: pgmap v439: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.2 GiB data, 7.5 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 604 op/s; 21457/107667 objects degraded (19.929%); 2.8 MiB/s, 56 objects/s recovering 2024-09-13T14:30:03.302 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:03.326 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:03.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:03.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:03.428 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:30:03.467 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:03.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:03.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:30:03.569 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:03.615 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:03.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:03.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:03.717 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:03.741 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:03.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:03.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:03.833 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is set 2024-09-13T14:30:03.843 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:03.871 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:03.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:03.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:03.973 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:04.015 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:04.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:04.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:04.117 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:04.140 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:04.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:04.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:04.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:03 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3276941396' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:30:04.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:03 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:30:04.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:03 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 21457/107667 objects degraded (19.929%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:30:04.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:03 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3276941396' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:30:04.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:03 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:30:04.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:03 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 21457/107667 objects degraded (19.929%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:30:04.242 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:04.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:03 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3276941396' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:30:04.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:03 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:30:04.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:03 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 21457/107667 objects degraded (19.929%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:30:04.266 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:04.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:04.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:04.368 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:30:04.404 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:04.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:04.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:30:04.506 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:30:04.553 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:04.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:04.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:30:04.656 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:30:04.703 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:04.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:04.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:30:04.805 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:30:04.851 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:04.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:04.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:30:04.949 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:30:04.953 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:04.986 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:04.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:04.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:05.084 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:04 smithi102 ceph-mon[110183]: pgmap v440: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.0 GiB data, 6.9 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 896 op/s; 21068/97161 objects degraded (21.684%); 2.8 MiB/s, 89 objects/s recovering 2024-09-13T14:30:05.084 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:04 smithi102 ceph-mon[110183]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:30:05.085 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:04 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:30:05.085 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:04 smithi102 ceph-mon[110183]: osdmap e149: 8 total, 8 up, 6 in 2024-09-13T14:30:05.085 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:04 smithi102 ceph-mon[115269]: pgmap v440: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.0 GiB data, 6.9 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 896 op/s; 21068/97161 objects degraded (21.684%); 2.8 MiB/s, 89 objects/s recovering 2024-09-13T14:30:05.085 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:04 smithi102 ceph-mon[115269]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:30:05.085 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:04 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:30:05.085 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:04 smithi102 ceph-mon[115269]: osdmap e149: 8 total, 8 up, 6 in 2024-09-13T14:30:05.088 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:05.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:05.119 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:05.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:05.220 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:05.247 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:05.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:04 smithi195 ceph-mon[104262]: pgmap v440: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.0 GiB data, 6.9 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 896 op/s; 21068/97161 objects degraded (21.684%); 2.8 MiB/s, 89 objects/s recovering 2024-09-13T14:30:05.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:04 smithi195 ceph-mon[104262]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:30:05.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:04 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:30:05.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:04 smithi195 ceph-mon[104262]: osdmap e149: 8 total, 8 up, 6 in 2024-09-13T14:30:05.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:05.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:05.350 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:30:05.387 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:05.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:05.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:30:05.488 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:05.513 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:05.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:05.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:05.614 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:05.658 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:05.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:05.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:05.760 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:05.788 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:05.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:05.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:05.889 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:05.915 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:05.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:05.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:06.017 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:06.054 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:06.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:06.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:06.156 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:06.180 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:06.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:06.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:06.282 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:06.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:06.310 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:06.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:06.411 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:06.437 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:06.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:06.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:06.539 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:06.582 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:06.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:06.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:06.588 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:30:06.683 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:06.705 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:30:06 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:30:06] "GET /metrics HTTP/1.1" 200 36530 "" "Prometheus/2.51.0" 2024-09-13T14:30:06.732 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:06.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:06.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:06.834 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:06.858 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:06.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:06.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:06.960 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:06.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:06 smithi195 ceph-mon[104262]: pgmap v442: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.0 GiB data, 6.9 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 730 op/s; 21068/96519 objects degraded (21.828%); 1.3 MiB/s, 73 objects/s recovering 2024-09-13T14:30:06.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:06 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:30:06.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:06 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:30:06.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:06 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:30:06.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:06 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:30:06.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:06 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:30:07.005 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:07.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:07.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:07.031 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:06 smithi102 ceph-mon[115269]: pgmap v442: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.0 GiB data, 6.9 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 730 op/s; 21068/96519 objects degraded (21.828%); 1.3 MiB/s, 73 objects/s recovering 2024-09-13T14:30:07.031 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:06 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:30:07.031 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:06 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:30:07.031 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:06 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:30:07.031 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:06 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:30:07.031 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:06 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:30:07.031 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:06 smithi102 ceph-mon[110183]: pgmap v442: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.0 GiB data, 6.9 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 730 op/s; 21068/96519 objects degraded (21.828%); 1.3 MiB/s, 73 objects/s recovering 2024-09-13T14:30:07.032 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:06 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:30:07.032 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:06 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:30:07.032 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:06 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:30:07.032 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:06 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:30:07.032 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:06 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:30:07.107 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:07.140 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:07.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:07.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:07.244 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:07.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:07.291 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:07.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:07.392 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:07.427 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:07.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:07.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:07.529 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:30:07.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:07.574 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:07.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:30:07.675 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:07.723 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:07.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:07.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:07.825 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:07.854 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:07.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:07.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:07.956 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:07.987 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:07.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:07.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:07.997 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:30:07.998 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:30:08.020 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:30:08.020 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:30:08.039 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:30:08.039 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:30:08.062 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:30:08.062 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:30:08.089 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:08.091 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:30:08.091 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:30:08.123 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:30:08.124 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:30:08.138 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:08.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:08.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:08.148 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:30:08.148 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:30:08.172 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:30:08.173 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:30:08.239 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:08.262 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:08.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:08.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:08.364 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:08.389 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:08.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:08.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:08.491 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:30:08.531 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:08.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:08.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:30:08.633 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:08.672 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:08.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:08.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:08.751 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:30:08.774 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:08.816 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:08.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:08.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:08.841 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:08 smithi102 ceph-mon[110183]: pgmap v443: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.8 GiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 740 op/s; 20663/86157 objects degraded (23.983%); 0 B/s, 79 objects/s recovering 2024-09-13T14:30:08.842 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:08 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/4193654816' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:30:08.842 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:08 smithi102 ceph-mon[115269]: pgmap v443: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.8 GiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 740 op/s; 20663/86157 objects degraded (23.983%); 0 B/s, 79 objects/s recovering 2024-09-13T14:30:08.842 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:08 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/4193654816' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:30:08.918 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:30:08.966 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:08.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:08.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:30:08.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:08 smithi195 ceph-mon[104262]: pgmap v443: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.8 GiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 740 op/s; 20663/86157 objects degraded (23.983%); 0 B/s, 79 objects/s recovering 2024-09-13T14:30:08.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:08 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/4193654816' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:30:09.068 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:09.092 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:09.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:09.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:09.194 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:09.218 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:09.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:09.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:09.319 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:09.364 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:09.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:09.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:09.466 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:09.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:09.487 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:09.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:09.588 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:09.612 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:09.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:09.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:09.714 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:09.738 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:09.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:09.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:09.840 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:30:09.884 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:09.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:09.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:30:09.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:09 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 20663/86157 objects degraded (23.983%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:30:09.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:09 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 20663/86157 objects degraded (23.983%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:30:09.986 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:30:09.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:09 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 20663/86157 objects degraded (23.983%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:30:10.033 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:10.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:10.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:30:10.135 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:10.159 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:10.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:10.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:10.261 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:10.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:10.300 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:10.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:10.401 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:10.428 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:10.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:10.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:10.530 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:10.558 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:10.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:10.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:10.660 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:10.683 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:10.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:10.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:10.785 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:10.822 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:10.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:10.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:10.923 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:10.947 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:10.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:10.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:10.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:10 smithi195 ceph-mon[104262]: pgmap v444: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.7 GiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 739 op/s; 20663/85467 objects degraded (24.177%); 0 B/s, 79 objects/s recovering 2024-09-13T14:30:10.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:30:11.003 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:10 smithi102 ceph-mon[110183]: pgmap v444: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.7 GiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 739 op/s; 20663/85467 objects degraded (24.177%); 0 B/s, 79 objects/s recovering 2024-09-13T14:30:11.004 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:30:11.004 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:10 smithi102 ceph-mon[115269]: pgmap v444: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.7 GiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 739 op/s; 20663/85467 objects degraded (24.177%); 0 B/s, 79 objects/s recovering 2024-09-13T14:30:11.004 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:30:11.049 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:11.073 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:11.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:11.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:11.175 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:11.199 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:11.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:11.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:11.301 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:30:11.343 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:11.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:11.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:30:11.445 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:30:11.488 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:11.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:11.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:30:11.590 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:11.616 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:11.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:11.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:11.718 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:11.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:11.761 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:11.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:11.862 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:11.893 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:11.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:11.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:11.995 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:12.020 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:12.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:12.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:12.122 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:12.165 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:12.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:12.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:12.184 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:30:12.184 INFO:teuthology.orchestra.run.smithi102.stdout:{"fsid":"08cee502-71d9-11ef-bceb-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 20663/85467 objects degraded (24.177%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":635,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":149,"num_osds":8,"num_up_osds":8,"osd_up_since":1726237494,"num_in_osds":6,"osd_in_since":1726237631,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":83},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":2}],"num_pgs":89,"num_pools":5,"num_objects":28489,"data_bytes":1869944829,"bytes_used":6639681536,"bytes_avail":569330532352,"bytes_total":575970213888,"degraded_objects":20663,"degraded_total":85467,"degraded_ratio":0.24176582774638164,"recovering_objects_per_sec":79,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":794,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":739},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi102.njetxr","status":"up:active","gid":24521}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.102:8443/","prometheus":"http://172.21.15.102:9283/"}},"servicemap":{"epoch":23,"modified":"2024-09-13T14:25:55.234425+0000","services":{}},"progress_events":{}} 2024-09-13T14:30:12.267 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:30:12.307 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:12.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:12.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:30:12.409 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:12.433 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:12.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:12.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:12.535 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:30:12.574 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:12.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:12.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:30:12.676 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:12.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:12.705 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:12.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:12.806 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:12.833 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:12.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:12.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:12.934 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:12.958 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:12 smithi102 ceph-mon[110183]: pgmap v445: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.7 GiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 739 op/s; 20663/85467 objects degraded (24.177%); 0 B/s, 79 objects/s recovering 2024-09-13T14:30:12.958 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:12 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1578566114' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:30:12.958 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:12 smithi102 ceph-mon[115269]: pgmap v445: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.7 GiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 739 op/s; 20663/85467 objects degraded (24.177%); 0 B/s, 79 objects/s recovering 2024-09-13T14:30:12.958 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:12 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1578566114' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:30:12.974 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:12.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:12.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:12.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:12 smithi195 ceph-mon[104262]: pgmap v445: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.7 GiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 739 op/s; 20663/85467 objects degraded (24.177%); 0 B/s, 79 objects/s recovering 2024-09-13T14:30:12.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:12 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1578566114' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:30:13.033 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:30:13.076 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:13.101 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:13.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:13.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:13.203 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:13.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:13.238 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:13.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:13.339 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:13.371 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:13.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:13.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:13.469 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:30:13.469 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [], 2024-09-13T14:30:13.469 INFO:teuthology.orchestra.run.smithi102.stdout: "complaint_time": 30, 2024-09-13T14:30:13.469 INFO:teuthology.orchestra.run.smithi102.stdout: "num_blocked_ops": 0 2024-09-13T14:30:13.469 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:30:13.474 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:13.514 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:13.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:13.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:13.616 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:30:13.654 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:13.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:13.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:30:13.756 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:13.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:13.783 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:13.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:13.884 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:13.915 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:13.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:13.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:14.017 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:14.049 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:14.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:14.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:14.151 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:14.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:14.199 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:14.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:14.300 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:30:14.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:14.346 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:14.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:30:14.447 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:14.473 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:14.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:14.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:14.575 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:14.601 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:14.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:14.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:14.664 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-13T14:30:14.665 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:30:14.703 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:30:14.742 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:14.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:14.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:30:14.765 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is unset 2024-09-13T14:30:14.843 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:14.871 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:14.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:14.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:14.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:14 smithi102 ceph-mon[110183]: pgmap v446: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.5 GiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 737 op/s; 20247/75036 objects degraded (26.983%); 0 B/s, 82 objects/s recovering 2024-09-13T14:30:14.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:14 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 20247/75036 objects degraded (26.983%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:30:14.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:14 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/4227772810' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:30:14.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:14 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:30:14.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:14 smithi102 ceph-mon[115269]: pgmap v446: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.5 GiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 737 op/s; 20247/75036 objects degraded (26.983%); 0 B/s, 82 objects/s recovering 2024-09-13T14:30:14.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:14 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 20247/75036 objects degraded (26.983%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:30:14.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:14 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/4227772810' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:30:14.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:14 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:30:14.972 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:14.998 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:14.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:14.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:15.100 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:15.119 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T14:30:15.129 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:15.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:15.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:15.232 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:15.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:14 smithi195 ceph-mon[104262]: pgmap v446: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.5 GiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 737 op/s; 20247/75036 objects degraded (26.983%); 0 B/s, 82 objects/s recovering 2024-09-13T14:30:15.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:14 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 20247/75036 objects degraded (26.983%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:30:15.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:14 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/4227772810' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:30:15.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:14 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:30:15.258 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:15.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:15.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:15.333 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:30:15.360 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:30:15.399 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:15.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:15.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:30:15.501 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:15.541 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:15.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:15.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:15.642 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:15.666 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:15.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:15.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:15.768 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:15.797 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:15.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:15.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:15.900 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:15.925 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:15.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:15.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:16.027 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:16.051 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:16.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:16.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:16.152 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:16.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:15 smithi102 ceph-mon[110183]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:30:16.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:15 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:30:16.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:15 smithi102 ceph-mon[110183]: osdmap e150: 8 total, 8 up, 6 in 2024-09-13T14:30:16.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:15 smithi102 ceph-mon[115269]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:30:16.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:15 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:30:16.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:15 smithi102 ceph-mon[115269]: osdmap e150: 8 total, 8 up, 6 in 2024-09-13T14:30:16.198 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:16.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:16.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:16.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:15 smithi195 ceph-mon[104262]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:30:16.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:15 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:30:16.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:15 smithi195 ceph-mon[104262]: osdmap e150: 8 total, 8 up, 6 in 2024-09-13T14:30:16.300 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:30:16.335 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:16.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:16.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:30:16.437 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:16.463 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:16.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:16.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:16.565 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:16.586 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:16.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:16.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:16.688 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:16.734 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:16.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:16.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:16.764 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:30:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:30:16] "GET /metrics HTTP/1.1" 200 36519 "" "Prometheus/2.51.0" 2024-09-13T14:30:16.835 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:16.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:16.879 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:16.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:16.980 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:17.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:17.004 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:17.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:17.014 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:16 smithi102 ceph-mon[110183]: pgmap v448: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.5 GiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 742 op/s; 20247/74244 objects degraded (27.271%); 0 B/s, 82 objects/s recovering 2024-09-13T14:30:17.014 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:16 smithi102 ceph-mon[115269]: pgmap v448: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.5 GiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 742 op/s; 20247/74244 objects degraded (27.271%); 0 B/s, 82 objects/s recovering 2024-09-13T14:30:17.105 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:17.131 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:17.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:17.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:17.230 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:30:17.234 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:17.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:16 smithi195 ceph-mon[104262]: pgmap v448: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.5 GiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 742 op/s; 20247/74244 objects degraded (27.271%); 0 B/s, 82 objects/s recovering 2024-09-13T14:30:17.273 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:17.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:17.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:17.376 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:17.401 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:17.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:17.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:17.502 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:17.532 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:17.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:17.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:17.634 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:17.657 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:17.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:17.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:17.758 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:17.783 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:17.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:17.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:17.816 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:30:17.885 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:17.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:17.924 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:17.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:18.025 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:18.047 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:18.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:18.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:18.149 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:18.176 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:18.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:18.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:18.277 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:30:18.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:18.317 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:18.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:30:18.418 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:30:18.470 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:18.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:18.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:30:18.572 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:18.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:18.615 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:18.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:18.716 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:18.740 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:18.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:18.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:18.752 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:30:18.842 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:18.869 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:18.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:18.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:18.970 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:18.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:18.998 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:18.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:19.099 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:30:19.135 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:19.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:19.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:30:19.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:18 smithi102 ceph-mon[110183]: pgmap v449: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.3 GiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 743 op/s; 19805/63855 objects degraded (31.016%); 0 B/s, 85 objects/s recovering 2024-09-13T14:30:19.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:18 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 19805/63855 objects degraded (31.016%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:30:19.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:18 smithi102 ceph-mon[115269]: pgmap v449: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.3 GiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 743 op/s; 19805/63855 objects degraded (31.016%); 0 B/s, 85 objects/s recovering 2024-09-13T14:30:19.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:18 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 19805/63855 objects degraded (31.016%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:30:19.237 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:30:19.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:18 smithi195 ceph-mon[104262]: pgmap v449: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.3 GiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 743 op/s; 19805/63855 objects degraded (31.016%); 0 B/s, 85 objects/s recovering 2024-09-13T14:30:19.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:18 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 19805/63855 objects degraded (31.016%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:30:19.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:19.280 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:19.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:30:19.309 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:30:19.381 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:30:19.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:19.432 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:19.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:30:19.533 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:19.559 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:19.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:19.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:19.662 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:19.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:19.708 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:19.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:19.809 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:19.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:19.849 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:19.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:19.950 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:19.981 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:20.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:20.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:20.049 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:30:20.049 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:30:20.050 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:30:20.050 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:30:20.052 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:30:20.052 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:30:20.057 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:30:20.057 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:30:20.082 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:30:20.082 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:30:20.111 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:30:20.112 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:30:20.135 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:30:20.135 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:30:20.149 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:20.155 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:30:20.156 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:30:20.189 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:20.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:20.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:20.291 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:30:20.327 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:20.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:20.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:30:20.429 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:20.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:20.471 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:20.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:20.572 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:20.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:20.603 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:20.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:20.683 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:30:20.683 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:30:20.704 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:20.728 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:20.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:20.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:20.830 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:20.855 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:20.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:20.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:20.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:20 smithi102 ceph-mon[110183]: pgmap v450: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.3 GiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 745 op/s; 19805/63108 objects degraded (31.383%); 0 B/s, 85 objects/s recovering 2024-09-13T14:30:20.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:20 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/415307579' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:30:20.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:20 smithi102 ceph-mon[115269]: pgmap v450: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.3 GiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 745 op/s; 19805/63108 objects degraded (31.383%); 0 B/s, 85 objects/s recovering 2024-09-13T14:30:20.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:20 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/415307579' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:30:20.957 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:20.998 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:20.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:20.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:21.099 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:21.123 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:21.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:21.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:21.225 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:30:21.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:20 smithi195 ceph-mon[104262]: pgmap v450: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.3 GiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 745 op/s; 19805/63108 objects degraded (31.383%); 0 B/s, 85 objects/s recovering 2024-09-13T14:30:21.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:20 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/415307579' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:30:21.264 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:21.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:21.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:30:21.365 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:21.388 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:21.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:21.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:21.417 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T14:30:21.491 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:21.527 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:21.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:21.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:21.629 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:21.656 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:21.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:21.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:21.758 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:30:21.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:21.805 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:21.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:30:21.906 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:21.932 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:21.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:21.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:22.034 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:22.060 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:22.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:22.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:22.162 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:22.189 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:22.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:22.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:22.291 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:22.320 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:22.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:22.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:22.422 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:22.446 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:22.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:22.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:22.548 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:22.575 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:22.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:22.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:22.677 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:22.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:22.704 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:22.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:22.805 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:22.832 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:22.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:22.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:22.934 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:22.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:22.959 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:22.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:23.061 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:23.110 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:23.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:23.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:23.136 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:22 smithi102 ceph-mon[110183]: pgmap v451: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.3 GiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 745 op/s; 19805/63108 objects degraded (31.383%); 0 B/s, 85 objects/s recovering 2024-09-13T14:30:23.137 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:22 smithi102 ceph-mon[115269]: pgmap v451: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.3 GiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 745 op/s; 19805/63108 objects degraded (31.383%); 0 B/s, 85 objects/s recovering 2024-09-13T14:30:23.212 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:30:23.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:22 smithi195 ceph-mon[104262]: pgmap v451: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.3 GiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 745 op/s; 19805/63108 objects degraded (31.383%); 0 B/s, 85 objects/s recovering 2024-09-13T14:30:23.253 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:23.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:23.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:30:23.355 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:30:23.403 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:23.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:23.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:30:23.505 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:23.529 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:23.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:23.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:23.631 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:30:23.670 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:23.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:23.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:30:23.772 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:23.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:23.799 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:23.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:23.900 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:23.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:23.927 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:23.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:24.029 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:24.066 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:24.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:24.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:24.111 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:23 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 19805/63108 objects degraded (31.383%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:30:24.112 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:23 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 19805/63108 objects degraded (31.383%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:30:24.167 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:24.190 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:24.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:24.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:24.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:23 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 19805/63108 objects degraded (31.383%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:30:24.292 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:24.333 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:24.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:24.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:24.435 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:24.463 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:24.463 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:30:24.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:24.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:24.565 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:24.591 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:24.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:24.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:24.693 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:24.734 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:24.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:24.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:24.836 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:24.861 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:24.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:24.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:24.963 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:30:25.004 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:25.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:25.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:30:25.033 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:24 smithi102 ceph-mon[110183]: pgmap v452: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.1 GiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 756 op/s; 19296/52335 objects degraded (36.870%); 0 B/s, 95 objects/s recovering 2024-09-13T14:30:25.034 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:24 smithi102 ceph-mon[115269]: pgmap v452: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.1 GiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 756 op/s; 19296/52335 objects degraded (36.870%); 0 B/s, 95 objects/s recovering 2024-09-13T14:30:25.106 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:30:25.149 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:24 smithi195 ceph-mon[104262]: pgmap v452: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.1 GiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 756 op/s; 19296/52335 objects degraded (36.870%); 0 B/s, 95 objects/s recovering 2024-09-13T14:30:25.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:25.152 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:25.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:30:25.253 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:25.290 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:25.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:25.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:25.392 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:30:25.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:25.422 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:25.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:30:25.523 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:30:25.566 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:25.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:25.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:30:25.668 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:25.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:25.692 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:25.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:25.793 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:25.836 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:25.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:25.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:25.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:25 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:30:25.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:25 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:30:25.938 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:30:25.952 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:30:25.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:25.982 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:25.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:30:26.083 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:26.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:26.126 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:26.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:26.227 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:26.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:25 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:30:26.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:26.256 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:26.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:26.357 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:26.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:26.393 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:26.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:26.495 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:26.530 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:26.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:26.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:26.608 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:30:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:30:26] "GET /metrics HTTP/1.1" 200 36523 "" "Prometheus/2.51.0" 2024-09-13T14:30:26.633 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:26.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:26.667 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:26.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:26.769 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:26.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:26.796 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:26.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:26.824 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is unset 2024-09-13T14:30:26.875 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:26 smithi102 ceph-mon[110183]: pgmap v453: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.1 GiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 714 op/s; 19296/51612 objects degraded (37.387%); 0 B/s, 90 objects/s recovering 2024-09-13T14:30:26.875 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:26 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/4058736357' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:30:26.875 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:26 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:30:26.875 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:26 smithi102 ceph-mon[115269]: pgmap v453: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.1 GiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 714 op/s; 19296/51612 objects degraded (37.387%); 0 B/s, 90 objects/s recovering 2024-09-13T14:30:26.876 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:26 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/4058736357' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:30:26.876 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:26 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:30:26.897 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:30:26.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:26.932 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:26.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:30:27.033 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:27.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:27.065 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:27.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:27.166 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:27.199 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:27.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:27.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:27.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:26 smithi195 ceph-mon[104262]: pgmap v453: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.1 GiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 714 op/s; 19296/51612 objects degraded (37.387%); 0 B/s, 90 objects/s recovering 2024-09-13T14:30:27.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:26 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/4058736357' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:30:27.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:26 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:30:27.301 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:27.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:27.337 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:27.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:27.438 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:27.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:27.471 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:27.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:27.573 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:27.580 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:30:27.581 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:30:27.608 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:27.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:27.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:27.710 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:27.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:27.742 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:27.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:27.843 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:27.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:27.869 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:27.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:27.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:27 smithi102 ceph-mon[115269]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:30:27.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:27 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:30:27.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:27 smithi102 ceph-mon[115269]: osdmap e151: 8 total, 8 up, 6 in 2024-09-13T14:30:27.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:27 smithi102 ceph-mon[110183]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:30:27.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:27 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:30:27.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:27 smithi102 ceph-mon[110183]: osdmap e151: 8 total, 8 up, 6 in 2024-09-13T14:30:27.970 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:28.011 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:28.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:28.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:28.113 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:30:28.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:28.149 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:28.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:30:28.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:27 smithi195 ceph-mon[104262]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:30:28.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:27 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:30:28.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:27 smithi195 ceph-mon[104262]: osdmap e151: 8 total, 8 up, 6 in 2024-09-13T14:30:28.250 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:28.277 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:28.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:28.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:28.312 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:30:28.313 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:30:28.380 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:28.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:28.415 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:28.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:28.516 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:30:28.556 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:28.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:28.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:30:28.658 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:28.685 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:28.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:28.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:28.787 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:30:28.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:28.823 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:28.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:30:28.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:28 smithi102 ceph-mon[110183]: pgmap v455: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 851 MiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 0 B/s wr, 771 op/s; 18738/40692 objects degraded (46.048%); 0 B/s, 106 objects/s recovering 2024-09-13T14:30:28.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:28 smithi102 ceph-mon[110183]: from='client.44888 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:30:28.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:28 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 18738/40692 objects degraded (46.048%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:30:28.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:28 smithi102 ceph-mon[115269]: pgmap v455: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 851 MiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 0 B/s wr, 771 op/s; 18738/40692 objects degraded (46.048%); 0 B/s, 106 objects/s recovering 2024-09-13T14:30:28.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:28 smithi102 ceph-mon[115269]: from='client.44888 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:30:28.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:28 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 18738/40692 objects degraded (46.048%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:30:28.924 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:28.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:28.957 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:28.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:28.979 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T14:30:28.983 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:30:29.060 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:29.088 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:29.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:29.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:29.190 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:29.214 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:29.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:29.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:29.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:28 smithi195 ceph-mon[104262]: pgmap v455: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 851 MiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 0 B/s wr, 771 op/s; 18738/40692 objects degraded (46.048%); 0 B/s, 106 objects/s recovering 2024-09-13T14:30:29.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:28 smithi195 ceph-mon[104262]: from='client.44888 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:30:29.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:28 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 18738/40692 objects degraded (46.048%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:30:29.315 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:29.338 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:29.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:29.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:29.440 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:30:29.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:29.497 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:29.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:30:29.535 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":455,"stamp":"2024-09-13T14:30:27.314843+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":891820029,"num_objects":13564,"num_object_clones":0,"num_object_copies":40692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18738,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13564,"num_whiteouts":0,"num_read":6391,"num_read_kb":13846,"num_write":223419,"num_write_kb":7561602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11679,"num_bytes_recovered":578357467,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109578,"ondisk_log_size":109578,"up":267,"acting":261,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":3637732,"kb_used_data":2709748,"kb_used_omap":19,"kb_used_meta":927788,"kb_avail":558833180,"statfs":{"total":575970213888,"available":572245176320,"internally_reserved":0,"allocated":2774781952,"data_stored":2770220127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20284,"internal_metadata":950055108},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-506003456,"num_objects":-7721,"num_object_clones":0,"num_object_copies":-23163,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1067,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7721,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1067,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002660"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":170,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.777577+0000","last_change":"2024-09-13T14:27:12.839984+0000","last_active":"2024-09-13T14:30:14.777577+0000","last_peered":"2024-09-13T14:30:14.777577+0000","last_clean":"2024-09-13T14:30:14.777577+0000","last_became_active":"2024-09-13T14:27:12.839653+0000","last_became_peered":"2024-09-13T14:27:12.839653+0000","last_unstale":"2024-09-13T14:30:14.777577+0000","last_undegraded":"2024-09-13T14:30:14.777577+0000","last_fullsized":"2024-09-13T14:30:14.777577+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:14:13.672830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":133,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.773919+0000","last_change":"2024-09-13T14:27:12.846760+0000","last_active":"2024-09-13T14:30:14.773919+0000","last_peered":"2024-09-13T14:30:14.773919+0000","last_clean":"2024-09-13T14:30:14.773919+0000","last_became_active":"2024-09-13T14:27:12.838041+0000","last_became_peered":"2024-09-13T14:27:12.838041+0000","last_unstale":"2024-09-13T14:30:14.773919+0000","last_undegraded":"2024-09-13T14:30:14.773919+0000","last_fullsized":"2024-09-13T14:30:14.773919+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:22.402859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":114,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.781350+0000","last_change":"2024-09-13T14:24:55.282407+0000","last_active":"2024-09-13T14:30:14.781350+0000","last_peered":"2024-09-13T14:30:14.781350+0000","last_clean":"2024-09-13T14:30:14.781350+0000","last_became_active":"2024-09-13T14:24:55.282083+0000","last_became_peered":"2024-09-13T14:24:55.282083+0000","last_unstale":"2024-09-13T14:30:14.781350+0000","last_undegraded":"2024-09-13T14:30:14.781350+0000","last_fullsized":"2024-09-13T14:30:14.781350+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:23:10.525707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":147,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.771348+0000","last_change":"2024-09-13T14:24:55.284433+0000","last_active":"2024-09-13T14:30:14.771348+0000","last_peered":"2024-09-13T14:30:14.771348+0000","last_clean":"2024-09-13T14:30:14.771348+0000","last_became_active":"2024-09-13T14:24:55.284143+0000","last_became_peered":"2024-09-13T14:24:55.284143+0000","last_unstale":"2024-09-13T14:30:14.771348+0000","last_undegraded":"2024-09-13T14:30:14.771348+0000","last_fullsized":"2024-09-13T14:30:14.771348+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:28.107405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":148,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.771548+0000","last_change":"2024-09-13T14:24:55.282416+0000","last_active":"2024-09-13T14:30:14.771548+0000","last_peered":"2024-09-13T14:30:14.771548+0000","last_clean":"2024-09-13T14:30:14.771548+0000","last_became_active":"2024-09-13T14:24:55.282045+0000","last_became_peered":"2024-09-13T14:24:55.282045+0000","last_unstale":"2024-09-13T14:30:14.771548+0000","last_undegraded":"2024-09-13T14:30:14.771548+0000","last_fullsized":"2024-09-13T14:30:14.771548+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:58.692897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":101,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.775146+0000","last_change":"2024-09-13T14:11:29.244452+0000","last_active":"2024-09-13T14:30:14.775146+0000","last_peered":"2024-09-13T14:30:14.775146+0000","last_clean":"2024-09-13T14:30:14.775146+0000","last_became_active":"2024-09-13T14:11:29.244102+0000","last_became_peered":"2024-09-13T14:11:29.244102+0000","last_unstale":"2024-09-13T14:30:14.775146+0000","last_undegraded":"2024-09-13T14:30:14.775146+0000","last_fullsized":"2024-09-13T14:30:14.775146+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:13:19.165534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":170,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.777251+0000","last_change":"2024-09-13T14:27:12.834300+0000","last_active":"2024-09-13T14:30:14.777251+0000","last_peered":"2024-09-13T14:30:14.777251+0000","last_clean":"2024-09-13T14:30:14.777251+0000","last_became_active":"2024-09-13T14:27:12.834046+0000","last_became_peered":"2024-09-13T14:27:12.834046+0000","last_unstale":"2024-09-13T14:30:14.777251+0000","last_undegraded":"2024-09-13T14:30:14.777251+0000","last_fullsized":"2024-09-13T14:30:14.777251+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:46:58.906862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":123,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:30:03.844527+0000","last_change":"2024-09-13T14:23:18.039061+0000","last_active":"2024-09-13T14:30:03.844527+0000","last_peered":"2024-09-13T14:30:03.844527+0000","last_clean":"2024-09-13T14:30:03.844527+0000","last_became_active":"2024-09-13T14:23:18.038661+0000","last_became_peered":"2024-09-13T14:23:18.038661+0000","last_unstale":"2024-09-13T14:30:03.844527+0000","last_undegraded":"2024-09-13T14:30:03.844527+0000","last_fullsized":"2024-09-13T14:30:03.844527+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:05:29.109844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.9","version":"151'13702","reported_seq":13747,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T14:30:26.929117+0000","last_change":"2024-09-13T14:26:21.865664+0000","last_active":"2024-09-13T14:30:26.929117+0000","last_peered":"2024-09-13T14:30:26.929117+0000","last_clean":"2024-09-13T14:30:26.929117+0000","last_became_active":"2024-09-13T14:26:21.865396+0000","last_became_peered":"2024-09-13T14:26:21.865396+0000","last_unstale":"2024-09-13T14:30:26.929117+0000","last_undegraded":"2024-09-13T14:30:26.929117+0000","last_fullsized":"2024-09-13T14:30:26.929117+0000","mapping_epoch":116,"log_start":"131'7000","ondisk_log_start":"131'7000","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6702,"log_dups_size":0,"ondisk_log_size":6702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:54:35.654265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53739520,"num_objects":820,"num_object_clones":0,"num_object_copies":2460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13702,"num_write_kb":464704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":117,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.774233+0000","last_change":"2024-09-13T14:23:18.038597+0000","last_active":"2024-09-13T14:30:14.774233+0000","last_peered":"2024-09-13T14:30:14.774233+0000","last_clean":"2024-09-13T14:30:14.774233+0000","last_became_active":"2024-09-13T14:23:18.038240+0000","last_became_peered":"2024-09-13T14:23:18.038240+0000","last_unstale":"2024-09-13T14:30:14.774233+0000","last_undegraded":"2024-09-13T14:30:14.774233+0000","last_fullsized":"2024-09-13T14:30:14.774233+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:27:51.842195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":100,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.778168+0000","last_change":"2024-09-13T14:11:30.254534+0000","last_active":"2024-09-13T14:30:14.778168+0000","last_peered":"2024-09-13T14:30:14.778168+0000","last_clean":"2024-09-13T14:30:14.778168+0000","last_became_active":"2024-09-13T14:11:30.253959+0000","last_became_peered":"2024-09-13T14:11:30.253959+0000","last_unstale":"2024-09-13T14:30:14.778168+0000","last_undegraded":"2024-09-13T14:30:14.778168+0000","last_fullsized":"2024-09-13T14:30:14.778168+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:09:54.025853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"150'14217","reported_seq":14298,"reported_epoch":150,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:30:26.483283+0000","last_change":"2024-09-13T14:27:14.009733+0000","last_active":"2024-09-13T14:30:26.483283+0000","last_peered":"2024-09-13T14:30:26.483283+0000","last_clean":"2024-09-13T14:27:11.786427+0000","last_became_active":"2024-09-13T14:27:13.994962+0000","last_became_peered":"2024-09-13T14:27:13.994962+0000","last_unstale":"2024-09-13T14:30:26.483283+0000","last_undegraded":"2024-09-13T14:27:13.854850+0000","last_fullsized":"2024-09-13T14:27:13.824319+0000","mapping_epoch":125,"log_start":"132'7900","ondisk_log_start":"132'7900","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6317,"log_dups_size":0,"ondisk_log_size":6317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54984704,"num_objects":839,"num_object_clones":0,"num_object_copies":2517,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5198,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14217,"num_write_kb":481792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5],"avail_no_missing":["0","2","5"],"object_location_counts":[{"shards":"0,5","objects":-4359}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":131,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.778167+0000","last_change":"2024-09-13T14:26:15.828529+0000","last_active":"2024-09-13T14:30:14.778167+0000","last_peered":"2024-09-13T14:30:14.778167+0000","last_clean":"2024-09-13T14:30:14.778167+0000","last_became_active":"2024-09-13T14:26:15.820486+0000","last_became_peered":"2024-09-13T14:26:15.820486+0000","last_unstale":"2024-09-13T14:30:14.778167+0000","last_undegraded":"2024-09-13T14:30:14.778167+0000","last_fullsized":"2024-09-13T14:30:14.778167+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:11:46.283172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":138,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.773016+0000","last_change":"2024-09-13T14:26:15.824632+0000","last_active":"2024-09-13T14:30:14.773016+0000","last_peered":"2024-09-13T14:30:14.773016+0000","last_clean":"2024-09-13T14:30:14.773016+0000","last_became_active":"2024-09-13T14:26:15.824330+0000","last_became_peered":"2024-09-13T14:26:15.824330+0000","last_unstale":"2024-09-13T14:30:14.773016+0000","last_undegraded":"2024-09-13T14:30:14.773016+0000","last_fullsized":"2024-09-13T14:30:14.773016+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:25.288116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"150'13535","reported_seq":23414,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:26.462957+0000","last_change":"2024-09-13T14:28:49.532961+0000","last_active":"2024-09-13T14:30:26.462957+0000","last_peered":"2024-09-13T14:30:26.462957+0000","last_clean":"2024-09-13T14:30:26.462957+0000","last_became_active":"2024-09-13T14:28:49.511508+0000","last_became_peered":"2024-09-13T14:28:49.511508+0000","last_unstale":"2024-09-13T14:30:26.462957+0000","last_undegraded":"2024-09-13T14:30:26.462957+0000","last_fullsized":"2024-09-13T14:30:26.462957+0000","mapping_epoch":137,"log_start":"131'7200","ondisk_log_start":"131'7200","created":116,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6335,"log_dups_size":0,"ondisk_log_size":6335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:42:44.821074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57081856,"num_objects":871,"num_object_clones":0,"num_object_copies":2613,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13535,"num_write_kb":460992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4893,"num_bytes_recovered":320667648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":102,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.780472+0000","last_change":"2024-09-13T14:11:29.232897+0000","last_active":"2024-09-13T14:30:14.780472+0000","last_peered":"2024-09-13T14:30:14.780472+0000","last_clean":"2024-09-13T14:30:14.780472+0000","last_became_active":"2024-09-13T14:11:29.232637+0000","last_became_peered":"2024-09-13T14:11:29.232637+0000","last_unstale":"2024-09-13T14:30:14.780472+0000","last_undegraded":"2024-09-13T14:30:14.780472+0000","last_fullsized":"2024-09-13T14:30:14.780472+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:31:05.856046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":123,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:30:03.844690+0000","last_change":"2024-09-13T14:26:15.822452+0000","last_active":"2024-09-13T14:30:03.844690+0000","last_peered":"2024-09-13T14:30:03.844690+0000","last_clean":"2024-09-13T14:30:03.844690+0000","last_became_active":"2024-09-13T14:26:15.822111+0000","last_became_peered":"2024-09-13T14:26:15.822111+0000","last_unstale":"2024-09-13T14:30:03.844690+0000","last_undegraded":"2024-09-13T14:30:03.844690+0000","last_fullsized":"2024-09-13T14:30:03.844690+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:32:53.078767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"151'14100","reported_seq":14160,"reported_epoch":151,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:30:27.136087+0000","last_change":"2024-09-13T14:27:14.040788+0000","last_active":"2024-09-13T14:30:27.136087+0000","last_peered":"2024-09-13T14:30:27.136087+0000","last_clean":"2024-09-13T14:27:11.793438+0000","last_became_active":"2024-09-13T14:27:14.032063+0000","last_became_peered":"2024-09-13T14:27:14.032063+0000","last_unstale":"2024-09-13T14:30:27.136087+0000","last_undegraded":"2024-09-13T14:27:13.861208+0000","last_fullsized":"2024-09-13T14:27:13.827412+0000","mapping_epoch":125,"log_start":"132'7700","ondisk_log_start":"132'7700","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6400,"log_dups_size":0,"ondisk_log_size":6400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:07:10.560903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54919168,"num_objects":838,"num_object_clones":0,"num_object_copies":2514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14100,"num_write_kb":478016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3],"avail_no_missing":["4","2","3"],"object_location_counts":[{"shards":"3,4","objects":-4240}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":116,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.780823+0000","last_change":"2024-09-13T14:24:55.290798+0000","last_active":"2024-09-13T14:30:14.780823+0000","last_peered":"2024-09-13T14:30:14.780823+0000","last_clean":"2024-09-13T14:30:14.780823+0000","last_became_active":"2024-09-13T14:24:55.290510+0000","last_became_peered":"2024-09-13T14:24:55.290510+0000","last_unstale":"2024-09-13T14:30:14.780823+0000","last_undegraded":"2024-09-13T14:30:14.780823+0000","last_fullsized":"2024-09-13T14:30:14.780823+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:19:03.129132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":156,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.771775+0000","last_change":"2024-09-13T14:26:15.823316+0000","last_active":"2024-09-13T14:30:14.771775+0000","last_peered":"2024-09-13T14:30:14.771775+0000","last_clean":"2024-09-13T14:30:14.771775+0000","last_became_active":"2024-09-13T14:26:15.822980+0000","last_became_peered":"2024-09-13T14:26:15.822980+0000","last_unstale":"2024-09-13T14:30:14.771775+0000","last_undegraded":"2024-09-13T14:30:14.771775+0000","last_fullsized":"2024-09-13T14:30:14.771775+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:33:47.420548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"151'14000","reported_seq":22854,"reported_epoch":151,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T14:30:26.943161+0000","last_change":"2024-09-13T14:28:48.293087+0000","last_active":"2024-09-13T14:30:26.943161+0000","last_peered":"2024-09-13T14:30:26.943161+0000","last_clean":"2024-09-13T14:27:11.806319+0000","last_became_active":"2024-09-13T14:27:14.027828+0000","last_became_peered":"2024-09-13T14:27:14.027828+0000","last_unstale":"2024-09-13T14:30:26.943161+0000","last_undegraded":"2024-09-13T14:27:13.862259+0000","last_fullsized":"2024-09-13T14:27:13.823132+0000","mapping_epoch":125,"log_start":"131'7300","ondisk_log_start":"131'7300","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6700,"log_dups_size":0,"ondisk_log_size":6700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:40:06.560106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53411864,"num_objects":816,"num_object_clones":0,"num_object_copies":2448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":661,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":816,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14000,"num_write_kb":474049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4396,"num_bytes_recovered":152240128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5],"avail_no_missing":["7","2","5"],"object_location_counts":[{"shards":"5,7","objects":155}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":154,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.776216+0000","last_change":"2024-09-13T14:24:55.284499+0000","last_active":"2024-09-13T14:30:14.776216+0000","last_peered":"2024-09-13T14:30:14.776216+0000","last_clean":"2024-09-13T14:30:14.776216+0000","last_became_active":"2024-09-13T14:24:55.284096+0000","last_became_peered":"2024-09-13T14:24:55.284096+0000","last_unstale":"2024-09-13T14:30:14.776216+0000","last_undegraded":"2024-09-13T14:30:14.776216+0000","last_fullsized":"2024-09-13T14:30:14.776216+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:02:06.836099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":114,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.769615+0000","last_change":"2024-09-13T14:24:55.291371+0000","last_active":"2024-09-13T14:30:14.769615+0000","last_peered":"2024-09-13T14:30:14.769615+0000","last_clean":"2024-09-13T14:30:14.769615+0000","last_became_active":"2024-09-13T14:24:55.291047+0000","last_became_peered":"2024-09-13T14:24:55.291047+0000","last_unstale":"2024-09-13T14:30:14.769615+0000","last_undegraded":"2024-09-13T14:30:14.769615+0000","last_fullsized":"2024-09-13T14:30:14.769615+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:43.912412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"151'13755","reported_seq":13800,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T14:30:27.089642+0000","last_change":"2024-09-13T14:26:21.864062+0000","last_active":"2024-09-13T14:30:27.089642+0000","last_peered":"2024-09-13T14:30:27.089642+0000","last_clean":"2024-09-13T14:30:27.089642+0000","last_became_active":"2024-09-13T14:26:21.863721+0000","last_became_peered":"2024-09-13T14:26:21.863721+0000","last_unstale":"2024-09-13T14:30:27.089642+0000","last_undegraded":"2024-09-13T14:30:27.089642+0000","last_fullsized":"2024-09-13T14:30:27.089642+0000","mapping_epoch":116,"log_start":"131'7300","ondisk_log_start":"131'7300","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6455,"log_dups_size":0,"ondisk_log_size":6455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:15:42.744309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51183616,"num_objects":781,"num_object_clones":0,"num_object_copies":2343,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13755,"num_write_kb":465152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":124,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:30:03.844854+0000","last_change":"2024-09-13T14:26:15.827333+0000","last_active":"2024-09-13T14:30:03.844854+0000","last_peered":"2024-09-13T14:30:03.844854+0000","last_clean":"2024-09-13T14:30:03.844854+0000","last_became_active":"2024-09-13T14:26:15.826945+0000","last_became_peered":"2024-09-13T14:26:15.826945+0000","last_unstale":"2024-09-13T14:30:03.844854+0000","last_undegraded":"2024-09-13T14:30:03.844854+0000","last_fullsized":"2024-09-13T14:30:03.844854+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:35:24.870052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":147,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.769368+0000","last_change":"2024-09-13T14:27:12.842604+0000","last_active":"2024-09-13T14:30:14.769368+0000","last_peered":"2024-09-13T14:30:14.769368+0000","last_clean":"2024-09-13T14:30:14.769368+0000","last_became_active":"2024-09-13T14:27:12.841643+0000","last_became_peered":"2024-09-13T14:27:12.841643+0000","last_unstale":"2024-09-13T14:30:14.769368+0000","last_undegraded":"2024-09-13T14:30:14.769368+0000","last_fullsized":"2024-09-13T14:30:14.769368+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:34:34.751790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"151'13831","reported_seq":13876,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T14:30:26.890699+0000","last_change":"2024-09-13T14:26:21.867125+0000","last_active":"2024-09-13T14:30:26.890699+0000","last_peered":"2024-09-13T14:30:26.890699+0000","last_clean":"2024-09-13T14:30:26.890699+0000","last_became_active":"2024-09-13T14:26:21.866815+0000","last_became_peered":"2024-09-13T14:26:21.866815+0000","last_unstale":"2024-09-13T14:30:26.890699+0000","last_undegraded":"2024-09-13T14:30:26.890699+0000","last_fullsized":"2024-09-13T14:30:26.890699+0000","mapping_epoch":116,"log_start":"129'5900","ondisk_log_start":"129'5900","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":7931,"log_dups_size":0,"ondisk_log_size":7931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:44.218255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57606144,"num_objects":879,"num_object_clones":0,"num_object_copies":2637,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13831,"num_write_kb":470720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":124,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.774942+0000","last_change":"2024-09-13T14:27:12.842271+0000","last_active":"2024-09-13T14:30:14.774942+0000","last_peered":"2024-09-13T14:30:14.774942+0000","last_clean":"2024-09-13T14:30:14.774942+0000","last_became_active":"2024-09-13T14:27:12.841479+0000","last_became_peered":"2024-09-13T14:27:12.841479+0000","last_unstale":"2024-09-13T14:30:14.774942+0000","last_undegraded":"2024-09-13T14:30:14.774942+0000","last_fullsized":"2024-09-13T14:30:14.774942+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:30:22.746575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":123,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.771058+0000","last_change":"2024-09-13T14:26:15.825304+0000","last_active":"2024-09-13T14:30:14.771058+0000","last_peered":"2024-09-13T14:30:14.771058+0000","last_clean":"2024-09-13T14:30:14.771058+0000","last_became_active":"2024-09-13T14:26:15.825018+0000","last_became_peered":"2024-09-13T14:26:15.825018+0000","last_unstale":"2024-09-13T14:30:14.771058+0000","last_undegraded":"2024-09-13T14:30:14.771058+0000","last_fullsized":"2024-09-13T14:30:14.771058+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:22:59.408928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"151'13936","reported_seq":13981,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T14:30:27.117173+0000","last_change":"2024-09-13T14:26:21.865199+0000","last_active":"2024-09-13T14:30:27.117173+0000","last_peered":"2024-09-13T14:30:27.117173+0000","last_clean":"2024-09-13T14:30:27.117173+0000","last_became_active":"2024-09-13T14:26:21.864903+0000","last_became_peered":"2024-09-13T14:26:21.864903+0000","last_unstale":"2024-09-13T14:30:27.117173+0000","last_undegraded":"2024-09-13T14:30:27.117173+0000","last_fullsized":"2024-09-13T14:30:27.117173+0000","mapping_epoch":116,"log_start":"131'7500","ondisk_log_start":"131'7500","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6436,"log_dups_size":0,"ondisk_log_size":6436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:25.014330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53739520,"num_objects":820,"num_object_clones":0,"num_object_copies":2460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13936,"num_write_kb":472192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":164,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.776077+0000","last_change":"2024-09-13T14:26:15.823898+0000","last_active":"2024-09-13T14:30:14.776077+0000","last_peered":"2024-09-13T14:30:14.776077+0000","last_clean":"2024-09-13T14:30:14.776077+0000","last_became_active":"2024-09-13T14:26:15.823125+0000","last_became_peered":"2024-09-13T14:26:15.823125+0000","last_unstale":"2024-09-13T14:30:14.776077+0000","last_undegraded":"2024-09-13T14:30:14.776077+0000","last_fullsized":"2024-09-13T14:30:14.776077+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:51:27.949528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":126,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.777381+0000","last_change":"2024-09-13T14:26:15.823198+0000","last_active":"2024-09-13T14:30:14.777381+0000","last_peered":"2024-09-13T14:30:14.777381+0000","last_clean":"2024-09-13T14:30:14.777381+0000","last_became_active":"2024-09-13T14:26:15.822717+0000","last_became_peered":"2024-09-13T14:26:15.822717+0000","last_unstale":"2024-09-13T14:30:14.777381+0000","last_undegraded":"2024-09-13T14:30:14.777381+0000","last_fullsized":"2024-09-13T14:30:14.777381+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:22.471622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"151'14025","reported_seq":14074,"reported_epoch":151,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:30:26.928899+0000","last_change":"2024-09-13T14:27:14.052640+0000","last_active":"2024-09-13T14:30:26.928899+0000","last_peered":"2024-09-13T14:30:26.928899+0000","last_clean":"2024-09-13T14:27:11.828545+0000","last_became_active":"2024-09-13T14:27:14.042591+0000","last_became_peered":"2024-09-13T14:27:14.042591+0000","last_unstale":"2024-09-13T14:30:26.928899+0000","last_undegraded":"2024-09-13T14:27:13.862613+0000","last_fullsized":"2024-09-13T14:27:13.823368+0000","mapping_epoch":125,"log_start":"131'7300","ondisk_log_start":"131'7300","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6725,"log_dups_size":0,"ondisk_log_size":6725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:17:38.266600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54591488,"num_objects":833,"num_object_clones":0,"num_object_copies":2499,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14025,"num_write_kb":475456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5],"avail_no_missing":["7","2","5"],"object_location_counts":[{"shards":"5,7","objects":-4289}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":447,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.774140+0000","last_change":"2024-09-13T14:23:18.032973+0000","last_active":"2024-09-13T14:30:14.774140+0000","last_peered":"2024-09-13T14:30:14.774140+0000","last_clean":"2024-09-13T14:30:14.774140+0000","last_became_active":"2024-09-13T14:23:18.032579+0000","last_became_peered":"2024-09-13T14:23:18.032579+0000","last_unstale":"2024-09-13T14:30:14.774140+0000","last_undegraded":"2024-09-13T14:30:14.774140+0000","last_fullsized":"2024-09-13T14:30:14.774140+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:07:41.927740+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:17.109445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":480,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.780985+0000","last_change":"2024-09-13T14:27:15.865400+0000","last_active":"2024-09-13T14:30:14.780985+0000","last_peered":"2024-09-13T14:30:14.780985+0000","last_clean":"2024-09-13T14:30:14.780985+0000","last_became_active":"2024-09-13T14:27:15.864835+0000","last_became_peered":"2024-09-13T14:27:15.864835+0000","last_unstale":"2024-09-13T14:30:14.780985+0000","last_undegraded":"2024-09-13T14:30:14.780985+0000","last_fullsized":"2024-09-13T14:30:14.780985+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:27:10.928041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":105,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.776120+0000","last_change":"2024-09-13T14:24:55.288058+0000","last_active":"2024-09-13T14:30:14.776120+0000","last_peered":"2024-09-13T14:30:14.776120+0000","last_clean":"2024-09-13T14:30:14.776120+0000","last_became_active":"2024-09-13T14:24:55.287601+0000","last_became_peered":"2024-09-13T14:24:55.287601+0000","last_unstale":"2024-09-13T14:30:14.776120+0000","last_undegraded":"2024-09-13T14:30:14.776120+0000","last_fullsized":"2024-09-13T14:30:14.776120+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:56:34.622696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":100,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.777096+0000","last_change":"2024-09-13T14:11:30.253749+0000","last_active":"2024-09-13T14:30:14.777096+0000","last_peered":"2024-09-13T14:30:14.777096+0000","last_clean":"2024-09-13T14:30:14.777096+0000","last_became_active":"2024-09-13T14:11:30.253499+0000","last_became_peered":"2024-09-13T14:11:30.253499+0000","last_unstale":"2024-09-13T14:30:14.777096+0000","last_undegraded":"2024-09-13T14:30:14.777096+0000","last_fullsized":"2024-09-13T14:30:14.777096+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:02:00.987335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"151'14091","reported_seq":14136,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T14:30:26.877841+0000","last_change":"2024-09-13T14:26:21.865805+0000","last_active":"2024-09-13T14:30:26.877841+0000","last_peered":"2024-09-13T14:30:26.877841+0000","last_clean":"2024-09-13T14:30:26.877841+0000","last_became_active":"2024-09-13T14:26:21.865288+0000","last_became_peered":"2024-09-13T14:26:21.865288+0000","last_unstale":"2024-09-13T14:30:26.877841+0000","last_undegraded":"2024-09-13T14:30:26.877841+0000","last_fullsized":"2024-09-13T14:30:26.877841+0000","mapping_epoch":116,"log_start":"130'6100","ondisk_log_start":"130'6100","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":7991,"log_dups_size":0,"ondisk_log_size":7991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:46.510065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58261504,"num_objects":889,"num_object_clones":0,"num_object_copies":2667,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14091,"num_write_kb":479360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":362,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:21.947815+0000","last_change":"2024-09-13T14:26:15.835178+0000","last_active":"2024-09-13T14:30:21.947815+0000","last_peered":"2024-09-13T14:30:21.947815+0000","last_clean":"2024-09-13T14:30:21.947815+0000","last_became_active":"2024-09-13T14:26:15.820283+0000","last_became_peered":"2024-09-13T14:26:15.820283+0000","last_unstale":"2024-09-13T14:30:21.947815+0000","last_undegraded":"2024-09-13T14:30:21.947815+0000","last_fullsized":"2024-09-13T14:30:21.947815+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:30.340038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":148,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.770842+0000","last_change":"2024-09-13T14:24:55.286576+0000","last_active":"2024-09-13T14:30:14.770842+0000","last_peered":"2024-09-13T14:30:14.770842+0000","last_clean":"2024-09-13T14:30:14.770842+0000","last_became_active":"2024-09-13T14:24:55.286284+0000","last_became_peered":"2024-09-13T14:24:55.286284+0000","last_unstale":"2024-09-13T14:30:14.770842+0000","last_undegraded":"2024-09-13T14:30:14.770842+0000","last_fullsized":"2024-09-13T14:30:14.770842+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:35.589736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":174,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.771783+0000","last_change":"2024-09-13T14:27:12.836204+0000","last_active":"2024-09-13T14:30:14.771783+0000","last_peered":"2024-09-13T14:30:14.771783+0000","last_clean":"2024-09-13T14:30:14.771783+0000","last_became_active":"2024-09-13T14:27:12.835900+0000","last_became_peered":"2024-09-13T14:27:12.835900+0000","last_unstale":"2024-09-13T14:30:14.771783+0000","last_undegraded":"2024-09-13T14:30:14.771783+0000","last_fullsized":"2024-09-13T14:30:14.771783+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:36:52.121730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"151'13912","reported_seq":18586,"reported_epoch":151,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T14:30:26.883389+0000","last_change":"2024-09-13T14:28:48.300429+0000","last_active":"2024-09-13T14:30:26.883389+0000","last_peered":"2024-09-13T14:30:26.883389+0000","last_clean":"2024-09-13T14:27:11.802376+0000","last_became_active":"2024-09-13T14:27:13.985759+0000","last_became_peered":"2024-09-13T14:27:13.985759+0000","last_unstale":"2024-09-13T14:30:26.883389+0000","last_undegraded":"2024-09-13T14:27:13.852391+0000","last_fullsized":"2024-09-13T14:27:13.824524+0000","mapping_epoch":125,"log_start":"129'6000","ondisk_log_start":"129'6000","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":7912,"log_dups_size":0,"ondisk_log_size":7912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:58:21.117245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55836672,"num_objects":852,"num_object_clones":0,"num_object_copies":2556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2667,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13912,"num_write_kb":472448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2306,"num_bytes_recovered":99483648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7],"avail_no_missing":["5","2","7"],"object_location_counts":[{"shards":"5,7","objects":-1815}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":528,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.776587+0000","last_change":"2024-09-13T14:27:15.863966+0000","last_active":"2024-09-13T14:30:14.776587+0000","last_peered":"2024-09-13T14:30:14.776587+0000","last_clean":"2024-09-13T14:30:14.776587+0000","last_became_active":"2024-09-13T14:27:15.863124+0000","last_became_peered":"2024-09-13T14:27:15.863124+0000","last_unstale":"2024-09-13T14:30:14.776587+0000","last_undegraded":"2024-09-13T14:30:14.776587+0000","last_fullsized":"2024-09-13T14:30:14.776587+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:35:02.715013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":74,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:30:03.844455+0000","last_change":"2024-09-13T14:27:12.829079+0000","last_active":"2024-09-13T14:30:03.844455+0000","last_peered":"2024-09-13T14:30:03.844455+0000","last_clean":"2024-09-13T14:30:03.844455+0000","last_became_active":"2024-09-13T14:27:12.828619+0000","last_became_peered":"2024-09-13T14:27:12.828619+0000","last_unstale":"2024-09-13T14:30:03.844455+0000","last_undegraded":"2024-09-13T14:30:03.844455+0000","last_fullsized":"2024-09-13T14:30:03.844455+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:39:08.260304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":111,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.780658+0000","last_change":"2024-09-13T14:26:15.817732+0000","last_active":"2024-09-13T14:30:14.780658+0000","last_peered":"2024-09-13T14:30:14.780658+0000","last_clean":"2024-09-13T14:30:14.780658+0000","last_became_active":"2024-09-13T14:26:15.817403+0000","last_became_peered":"2024-09-13T14:26:15.817403+0000","last_unstale":"2024-09-13T14:30:14.780658+0000","last_undegraded":"2024-09-13T14:30:14.780658+0000","last_fullsized":"2024-09-13T14:30:14.780658+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:56:39.961300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"150'13722","reported_seq":13803,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:26.428000+0000","last_change":"2024-09-13T14:26:21.866279+0000","last_active":"2024-09-13T14:30:26.428000+0000","last_peered":"2024-09-13T14:30:26.428000+0000","last_clean":"2024-09-13T14:30:26.428000+0000","last_became_active":"2024-09-13T14:26:21.864793+0000","last_became_peered":"2024-09-13T14:26:21.864793+0000","last_unstale":"2024-09-13T14:30:26.428000+0000","last_undegraded":"2024-09-13T14:30:26.428000+0000","last_fullsized":"2024-09-13T14:30:26.428000+0000","mapping_epoch":116,"log_start":"131'7400","ondisk_log_start":"131'7400","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6322,"log_dups_size":0,"ondisk_log_size":6322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:48:28.309201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51773440,"num_objects":790,"num_object_clones":0,"num_object_copies":2370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13722,"num_write_kb":464384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":911,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.780153+0000","last_change":"2024-09-13T14:26:18.829131+0000","last_active":"2024-09-13T14:30:14.780153+0000","last_peered":"2024-09-13T14:30:14.780153+0000","last_clean":"2024-09-13T14:30:14.780153+0000","last_became_active":"2024-09-13T14:26:18.828619+0000","last_became_peered":"2024-09-13T14:26:18.828619+0000","last_unstale":"2024-09-13T14:30:14.780153+0000","last_undegraded":"2024-09-13T14:30:14.780153+0000","last_fullsized":"2024-09-13T14:30:14.780153+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:24:37.073618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":116,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:17.065294+0000","last_change":"2024-09-13T14:23:18.042891+0000","last_active":"2024-09-13T14:30:17.065294+0000","last_peered":"2024-09-13T14:30:17.065294+0000","last_clean":"2024-09-13T14:30:17.065294+0000","last_became_active":"2024-09-13T14:23:18.042562+0000","last_became_peered":"2024-09-13T14:23:18.042562+0000","last_unstale":"2024-09-13T14:30:17.065294+0000","last_undegraded":"2024-09-13T14:30:17.065294+0000","last_fullsized":"2024-09-13T14:30:17.065294+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:30:42.996661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":141,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.776395+0000","last_change":"2024-09-13T14:27:12.832666+0000","last_active":"2024-09-13T14:30:14.776395+0000","last_peered":"2024-09-13T14:30:14.776395+0000","last_clean":"2024-09-13T14:30:14.776395+0000","last_became_active":"2024-09-13T14:27:12.832332+0000","last_became_peered":"2024-09-13T14:27:12.832332+0000","last_unstale":"2024-09-13T14:30:14.776395+0000","last_undegraded":"2024-09-13T14:30:14.776395+0000","last_fullsized":"2024-09-13T14:30:14.776395+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:07:17.724156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"150'13657","reported_seq":13738,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:26.435835+0000","last_change":"2024-09-13T14:26:21.862954+0000","last_active":"2024-09-13T14:30:26.435835+0000","last_peered":"2024-09-13T14:30:26.435835+0000","last_clean":"2024-09-13T14:30:26.435835+0000","last_became_active":"2024-09-13T14:26:21.862650+0000","last_became_peered":"2024-09-13T14:26:21.862650+0000","last_unstale":"2024-09-13T14:30:26.435835+0000","last_undegraded":"2024-09-13T14:30:26.435835+0000","last_fullsized":"2024-09-13T14:30:26.435835+0000","mapping_epoch":116,"log_start":"131'7400","ondisk_log_start":"131'7400","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6257,"log_dups_size":0,"ondisk_log_size":6257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:46:50.507774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55508992,"num_objects":847,"num_object_clones":0,"num_object_copies":2541,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13657,"num_write_kb":464128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":141,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:21.879145+0000","last_change":"2024-09-13T14:27:12.843038+0000","last_active":"2024-09-13T14:30:21.879145+0000","last_peered":"2024-09-13T14:30:21.879145+0000","last_clean":"2024-09-13T14:30:21.879145+0000","last_became_active":"2024-09-13T14:27:12.842663+0000","last_became_peered":"2024-09-13T14:27:12.842663+0000","last_unstale":"2024-09-13T14:30:21.879145+0000","last_undegraded":"2024-09-13T14:30:21.879145+0000","last_fullsized":"2024-09-13T14:30:21.879145+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:23:38.880811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":845,"reported_epoch":150,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:30:14.774120+0000","last_change":"2024-09-13T14:27:14.029991+0000","last_active":"2024-09-13T14:30:14.774120+0000","last_peered":"2024-09-13T14:30:14.774120+0000","last_clean":"2024-09-13T14:27:08.815347+0000","last_became_active":"2024-09-13T14:27:13.888039+0000","last_became_peered":"2024-09-13T14:27:13.888039+0000","last_unstale":"2024-09-13T14:30:14.774120+0000","last_undegraded":"2024-09-13T14:27:13.826593+0000","last_fullsized":"2024-09-13T14:27:13.824237+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7],"avail_no_missing":["0","2","7"],"object_location_counts":[{"shards":"0,2,7","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":68,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T14:30:26.831324+0000","last_change":"2024-09-13T14:26:15.819032+0000","last_active":"2024-09-13T14:30:26.831324+0000","last_peered":"2024-09-13T14:30:26.831324+0000","last_clean":"2024-09-13T14:30:26.831324+0000","last_became_active":"2024-09-13T14:26:15.818675+0000","last_became_peered":"2024-09-13T14:26:15.818675+0000","last_unstale":"2024-09-13T14:30:26.831324+0000","last_undegraded":"2024-09-13T14:30:26.831324+0000","last_fullsized":"2024-09-13T14:30:26.831324+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:37:13.628256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"150'13880","reported_seq":13924,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:26.313165+0000","last_change":"2024-09-13T14:26:21.863343+0000","last_active":"2024-09-13T14:30:26.313165+0000","last_peered":"2024-09-13T14:30:26.313165+0000","last_clean":"2024-09-13T14:30:26.313165+0000","last_became_active":"2024-09-13T14:26:21.863057+0000","last_became_peered":"2024-09-13T14:26:21.863057+0000","last_unstale":"2024-09-13T14:30:26.313165+0000","last_undegraded":"2024-09-13T14:30:26.313165+0000","last_fullsized":"2024-09-13T14:30:26.313165+0000","mapping_epoch":116,"log_start":"131'7200","ondisk_log_start":"131'7200","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6680,"log_dups_size":0,"ondisk_log_size":6680,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:38:25.794314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56492032,"num_objects":862,"num_object_clones":0,"num_object_copies":2586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13880,"num_write_kb":471744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":75,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.780661+0000","last_change":"2024-09-13T14:27:12.843268+0000","last_active":"2024-09-13T14:30:14.780661+0000","last_peered":"2024-09-13T14:30:14.780661+0000","last_clean":"2024-09-13T14:30:14.780661+0000","last_became_active":"2024-09-13T14:27:12.842867+0000","last_became_peered":"2024-09-13T14:27:12.842867+0000","last_unstale":"2024-09-13T14:30:14.780661+0000","last_undegraded":"2024-09-13T14:30:14.780661+0000","last_fullsized":"2024-09-13T14:30:14.780661+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:03:59.575866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":638,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.780551+0000","last_change":"2024-09-13T14:23:18.038847+0000","last_active":"2024-09-13T14:30:14.780551+0000","last_peered":"2024-09-13T14:30:14.780551+0000","last_clean":"2024-09-13T14:30:14.780551+0000","last_became_active":"2024-09-13T14:23:18.038188+0000","last_became_peered":"2024-09-13T14:23:18.038188+0000","last_unstale":"2024-09-13T14:30:14.780551+0000","last_undegraded":"2024-09-13T14:30:14.780551+0000","last_fullsized":"2024-09-13T14:30:14.780551+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:30:37.213496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":138,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.771686+0000","last_change":"2024-09-13T14:23:18.034427+0000","last_active":"2024-09-13T14:30:14.771686+0000","last_peered":"2024-09-13T14:30:14.771686+0000","last_clean":"2024-09-13T14:30:14.771686+0000","last_became_active":"2024-09-13T14:23:18.034092+0000","last_became_peered":"2024-09-13T14:23:18.034092+0000","last_unstale":"2024-09-13T14:30:14.771686+0000","last_undegraded":"2024-09-13T14:30:14.771686+0000","last_fullsized":"2024-09-13T14:30:14.771686+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:21.755115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"151'13991","reported_seq":14036,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-13T14:30:27.078453+0000","last_change":"2024-09-13T14:26:21.867237+0000","last_active":"2024-09-13T14:30:27.078453+0000","last_peered":"2024-09-13T14:30:27.078453+0000","last_clean":"2024-09-13T14:30:27.078453+0000","last_became_active":"2024-09-13T14:26:21.866931+0000","last_became_peered":"2024-09-13T14:26:21.866931+0000","last_unstale":"2024-09-13T14:30:27.078453+0000","last_undegraded":"2024-09-13T14:30:27.078453+0000","last_fullsized":"2024-09-13T14:30:27.078453+0000","mapping_epoch":116,"log_start":"131'7600","ondisk_log_start":"131'7600","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6391,"log_dups_size":0,"ondisk_log_size":6391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:52:57.569446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53936128,"num_objects":823,"num_object_clones":0,"num_object_copies":2469,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13991,"num_write_kb":474048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":154,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.777461+0000","last_change":"2024-09-13T14:24:55.290711+0000","last_active":"2024-09-13T14:30:14.777461+0000","last_peered":"2024-09-13T14:30:14.777461+0000","last_clean":"2024-09-13T14:30:14.777461+0000","last_became_active":"2024-09-13T14:24:55.288549+0000","last_became_peered":"2024-09-13T14:24:55.288549+0000","last_unstale":"2024-09-13T14:30:14.777461+0000","last_undegraded":"2024-09-13T14:30:14.777461+0000","last_fullsized":"2024-09-13T14:30:14.777461+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:55:21.763931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":550,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:30:03.845227+0000","last_change":"2024-09-13T14:26:18.827876+0000","last_active":"2024-09-13T14:30:03.845227+0000","last_peered":"2024-09-13T14:30:03.845227+0000","last_clean":"2024-09-13T14:30:03.845227+0000","last_became_active":"2024-09-13T14:26:18.827317+0000","last_became_peered":"2024-09-13T14:26:18.827317+0000","last_unstale":"2024-09-13T14:30:03.845227+0000","last_undegraded":"2024-09-13T14:30:03.845227+0000","last_fullsized":"2024-09-13T14:30:03.845227+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:31:48.611286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":151,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.775974+0000","last_change":"2024-09-13T14:24:55.281593+0000","last_active":"2024-09-13T14:30:14.775974+0000","last_peered":"2024-09-13T14:30:14.775974+0000","last_clean":"2024-09-13T14:30:14.775974+0000","last_became_active":"2024-09-13T14:24:55.280805+0000","last_became_peered":"2024-09-13T14:24:55.280805+0000","last_unstale":"2024-09-13T14:30:14.775974+0000","last_undegraded":"2024-09-13T14:30:14.775974+0000","last_fullsized":"2024-09-13T14:30:14.775974+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:14.806713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"150'13942","reported_seq":14024,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:23.301143+0000","last_change":"2024-09-13T14:26:21.864819+0000","last_active":"2024-09-13T14:30:23.301143+0000","last_peered":"2024-09-13T14:30:23.301143+0000","last_clean":"2024-09-13T14:30:23.301143+0000","last_became_active":"2024-09-13T14:26:21.864511+0000","last_became_peered":"2024-09-13T14:26:21.864511+0000","last_unstale":"2024-09-13T14:30:23.301143+0000","last_undegraded":"2024-09-13T14:30:23.301143+0000","last_fullsized":"2024-09-13T14:30:23.301143+0000","mapping_epoch":116,"log_start":"131'7100","ondisk_log_start":"131'7100","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6842,"log_dups_size":0,"ondisk_log_size":6842,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:11:52.262311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63963136,"num_objects":976,"num_object_clones":0,"num_object_copies":2928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13942,"num_write_kb":477376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":163,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.771451+0000","last_change":"2024-09-13T14:26:15.830309+0000","last_active":"2024-09-13T14:30:14.771451+0000","last_peered":"2024-09-13T14:30:14.771451+0000","last_clean":"2024-09-13T14:30:14.771451+0000","last_became_active":"2024-09-13T14:26:15.823041+0000","last_became_peered":"2024-09-13T14:26:15.823041+0000","last_unstale":"2024-09-13T14:30:14.771451+0000","last_undegraded":"2024-09-13T14:30:14.771451+0000","last_fullsized":"2024-09-13T14:30:14.771451+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:23:08.158576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":581,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:30:03.845564+0000","last_change":"2024-09-13T14:27:12.848354+0000","last_active":"2024-09-13T14:30:03.845564+0000","last_peered":"2024-09-13T14:30:03.845564+0000","last_clean":"2024-09-13T14:30:03.845564+0000","last_became_active":"2024-09-13T14:27:12.832182+0000","last_became_peered":"2024-09-13T14:27:12.832182+0000","last_unstale":"2024-09-13T14:30:03.845564+0000","last_undegraded":"2024-09-13T14:30:03.845564+0000","last_fullsized":"2024-09-13T14:30:03.845564+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:46:42.064509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":100,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.774853+0000","last_change":"2024-09-13T14:11:30.256034+0000","last_active":"2024-09-13T14:30:14.774853+0000","last_peered":"2024-09-13T14:30:14.774853+0000","last_clean":"2024-09-13T14:30:14.774853+0000","last_became_active":"2024-09-13T14:11:30.255727+0000","last_became_peered":"2024-09-13T14:11:30.255727+0000","last_unstale":"2024-09-13T14:30:14.774853+0000","last_undegraded":"2024-09-13T14:30:14.774853+0000","last_fullsized":"2024-09-13T14:30:14.774853+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:21:35.817014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":123,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.770167+0000","last_change":"2024-09-13T14:26:15.825390+0000","last_active":"2024-09-13T14:30:14.770167+0000","last_peered":"2024-09-13T14:30:14.770167+0000","last_clean":"2024-09-13T14:30:14.770167+0000","last_became_active":"2024-09-13T14:26:15.825084+0000","last_became_peered":"2024-09-13T14:26:15.825084+0000","last_unstale":"2024-09-13T14:30:14.770167+0000","last_undegraded":"2024-09-13T14:30:14.770167+0000","last_fullsized":"2024-09-13T14:30:14.770167+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:44.680451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":123,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:17.065332+0000","last_change":"2024-09-13T14:11:29.237013+0000","last_active":"2024-09-13T14:30:17.065332+0000","last_peered":"2024-09-13T14:30:17.065332+0000","last_clean":"2024-09-13T14:30:17.065332+0000","last_became_active":"2024-09-13T14:11:29.236643+0000","last_became_peered":"2024-09-13T14:11:29.236643+0000","last_unstale":"2024-09-13T14:30:17.065332+0000","last_undegraded":"2024-09-13T14:30:17.065332+0000","last_fullsized":"2024-09-13T14:30:17.065332+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:33:58.438693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":96,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:30:03.844778+0000","last_change":"2024-09-13T14:27:12.843904+0000","last_active":"2024-09-13T14:30:03.844778+0000","last_peered":"2024-09-13T14:30:03.844778+0000","last_clean":"2024-09-13T14:30:03.844778+0000","last_became_active":"2024-09-13T14:27:12.842566+0000","last_became_peered":"2024-09-13T14:27:12.842566+0000","last_unstale":"2024-09-13T14:30:03.844778+0000","last_undegraded":"2024-09-13T14:30:03.844778+0000","last_fullsized":"2024-09-13T14:30:03.844778+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:11:46.253003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":118,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.780354+0000","last_change":"2024-09-13T14:24:55.287275+0000","last_active":"2024-09-13T14:30:14.780354+0000","last_peered":"2024-09-13T14:30:14.780354+0000","last_clean":"2024-09-13T14:30:14.780354+0000","last_became_active":"2024-09-13T14:24:55.286919+0000","last_became_peered":"2024-09-13T14:24:55.286919+0000","last_unstale":"2024-09-13T14:30:14.780354+0000","last_undegraded":"2024-09-13T14:30:14.780354+0000","last_fullsized":"2024-09-13T14:30:14.780354+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:10.790711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":124,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.785342+0000","last_change":"2024-09-13T14:26:15.819037+0000","last_active":"2024-09-13T14:30:14.785342+0000","last_peered":"2024-09-13T14:30:14.785342+0000","last_clean":"2024-09-13T14:30:14.785342+0000","last_became_active":"2024-09-13T14:26:15.818721+0000","last_became_peered":"2024-09-13T14:26:15.818721+0000","last_unstale":"2024-09-13T14:30:14.785342+0000","last_undegraded":"2024-09-13T14:30:14.785342+0000","last_fullsized":"2024-09-13T14:30:14.785342+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:44:06.419136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":107,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.775992+0000","last_change":"2024-09-13T14:24:55.287509+0000","last_active":"2024-09-13T14:30:14.775992+0000","last_peered":"2024-09-13T14:30:14.775992+0000","last_clean":"2024-09-13T14:30:14.775992+0000","last_became_active":"2024-09-13T14:24:55.287193+0000","last_became_peered":"2024-09-13T14:24:55.287193+0000","last_unstale":"2024-09-13T14:30:14.775992+0000","last_undegraded":"2024-09-13T14:30:14.775992+0000","last_fullsized":"2024-09-13T14:30:14.775992+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:14:10.603307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":147,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.771598+0000","last_change":"2024-09-13T14:24:55.289055+0000","last_active":"2024-09-13T14:30:14.771598+0000","last_peered":"2024-09-13T14:30:14.771598+0000","last_clean":"2024-09-13T14:30:14.771598+0000","last_became_active":"2024-09-13T14:24:55.288762+0000","last_became_peered":"2024-09-13T14:24:55.288762+0000","last_unstale":"2024-09-13T14:30:14.771598+0000","last_undegraded":"2024-09-13T14:30:14.771598+0000","last_fullsized":"2024-09-13T14:30:14.771598+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:37:26.639877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":163,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.775889+0000","last_change":"2024-09-13T14:26:15.824540+0000","last_active":"2024-09-13T14:30:14.775889+0000","last_peered":"2024-09-13T14:30:14.775889+0000","last_clean":"2024-09-13T14:30:14.775889+0000","last_became_active":"2024-09-13T14:26:15.824118+0000","last_became_peered":"2024-09-13T14:26:15.824118+0000","last_unstale":"2024-09-13T14:30:14.775889+0000","last_undegraded":"2024-09-13T14:30:14.775889+0000","last_fullsized":"2024-09-13T14:30:14.775889+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:56.677401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":124,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-13T14:30:11.348405+0000","last_change":"2024-09-13T14:26:15.823892+0000","last_active":"2024-09-13T14:30:11.348405+0000","last_peered":"2024-09-13T14:30:11.348405+0000","last_clean":"2024-09-13T14:30:11.348405+0000","last_became_active":"2024-09-13T14:26:15.823582+0000","last_became_peered":"2024-09-13T14:26:15.823582+0000","last_unstale":"2024-09-13T14:30:11.348405+0000","last_undegraded":"2024-09-13T14:30:11.348405+0000","last_fullsized":"2024-09-13T14:30:11.348405+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:53:09.918997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":186,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.776042+0000","last_change":"2024-09-13T14:27:12.830642+0000","last_active":"2024-09-13T14:30:14.776042+0000","last_peered":"2024-09-13T14:30:14.776042+0000","last_clean":"2024-09-13T14:30:14.776042+0000","last_became_active":"2024-09-13T14:27:12.830026+0000","last_became_peered":"2024-09-13T14:27:12.830026+0000","last_unstale":"2024-09-13T14:30:14.776042+0000","last_undegraded":"2024-09-13T14:30:14.776042+0000","last_fullsized":"2024-09-13T14:30:14.776042+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:12:31.783552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":149,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.771695+0000","last_change":"2024-09-13T14:24:55.283070+0000","last_active":"2024-09-13T14:30:14.771695+0000","last_peered":"2024-09-13T14:30:14.771695+0000","last_clean":"2024-09-13T14:30:14.771695+0000","last_became_active":"2024-09-13T14:24:55.282591+0000","last_became_peered":"2024-09-13T14:24:55.282591+0000","last_unstale":"2024-09-13T14:30:14.771695+0000","last_undegraded":"2024-09-13T14:30:14.771695+0000","last_fullsized":"2024-09-13T14:30:14.771695+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:35:21.896294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":135,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.776520+0000","last_change":"2024-09-13T14:26:15.823524+0000","last_active":"2024-09-13T14:30:14.776520+0000","last_peered":"2024-09-13T14:30:14.776520+0000","last_clean":"2024-09-13T14:30:14.776520+0000","last_became_active":"2024-09-13T14:26:15.823162+0000","last_became_peered":"2024-09-13T14:26:15.823162+0000","last_unstale":"2024-09-13T14:30:14.776520+0000","last_undegraded":"2024-09-13T14:30:14.776520+0000","last_fullsized":"2024-09-13T14:30:14.776520+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:17:13.664812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":176,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.776605+0000","last_change":"2024-09-13T14:26:15.823703+0000","last_active":"2024-09-13T14:30:14.776605+0000","last_peered":"2024-09-13T14:30:14.776605+0000","last_clean":"2024-09-13T14:30:14.776605+0000","last_became_active":"2024-09-13T14:26:15.823249+0000","last_became_peered":"2024-09-13T14:26:15.823249+0000","last_unstale":"2024-09-13T14:30:14.776605+0000","last_undegraded":"2024-09-13T14:30:14.776605+0000","last_fullsized":"2024-09-13T14:30:14.776605+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:42:40.209150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":148,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.769801+0000","last_change":"2024-09-13T14:27:12.841881+0000","last_active":"2024-09-13T14:30:14.769801+0000","last_peered":"2024-09-13T14:30:14.769801+0000","last_clean":"2024-09-13T14:30:14.769801+0000","last_became_active":"2024-09-13T14:27:12.841247+0000","last_became_peered":"2024-09-13T14:27:12.841247+0000","last_unstale":"2024-09-13T14:30:14.769801+0000","last_undegraded":"2024-09-13T14:30:14.769801+0000","last_fullsized":"2024-09-13T14:30:14.769801+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:59:16.937848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":159,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.777634+0000","last_change":"2024-09-13T14:27:12.838453+0000","last_active":"2024-09-13T14:30:14.777634+0000","last_peered":"2024-09-13T14:30:14.777634+0000","last_clean":"2024-09-13T14:30:14.777634+0000","last_became_active":"2024-09-13T14:27:12.838124+0000","last_became_peered":"2024-09-13T14:27:12.838124+0000","last_unstale":"2024-09-13T14:30:14.777634+0000","last_undegraded":"2024-09-13T14:30:14.777634+0000","last_fullsized":"2024-09-13T14:30:14.777634+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:24.259191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":140,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.771591+0000","last_change":"2024-09-13T14:23:18.035101+0000","last_active":"2024-09-13T14:30:14.771591+0000","last_peered":"2024-09-13T14:30:14.771591+0000","last_clean":"2024-09-13T14:30:14.771591+0000","last_became_active":"2024-09-13T14:23:18.034724+0000","last_became_peered":"2024-09-13T14:23:18.034724+0000","last_unstale":"2024-09-13T14:30:14.771591+0000","last_undegraded":"2024-09-13T14:30:14.771591+0000","last_fullsized":"2024-09-13T14:30:14.771591+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:39.358144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":146,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.770102+0000","last_change":"2024-09-13T14:27:12.834931+0000","last_active":"2024-09-13T14:30:14.770102+0000","last_peered":"2024-09-13T14:30:14.770102+0000","last_clean":"2024-09-13T14:30:14.770102+0000","last_became_active":"2024-09-13T14:27:12.834572+0000","last_became_peered":"2024-09-13T14:27:12.834572+0000","last_unstale":"2024-09-13T14:30:14.770102+0000","last_undegraded":"2024-09-13T14:30:14.770102+0000","last_fullsized":"2024-09-13T14:30:14.770102+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:54:21.155689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":130,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.780747+0000","last_change":"2024-09-13T14:26:15.826283+0000","last_active":"2024-09-13T14:30:14.780747+0000","last_peered":"2024-09-13T14:30:14.780747+0000","last_clean":"2024-09-13T14:30:14.780747+0000","last_became_active":"2024-09-13T14:26:15.820774+0000","last_became_peered":"2024-09-13T14:26:15.820774+0000","last_unstale":"2024-09-13T14:30:14.780747+0000","last_undegraded":"2024-09-13T14:30:14.780747+0000","last_fullsized":"2024-09-13T14:30:14.780747+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:28:39.849420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":128,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.769683+0000","last_change":"2024-09-13T14:24:55.285384+0000","last_active":"2024-09-13T14:30:14.769683+0000","last_peered":"2024-09-13T14:30:14.769683+0000","last_clean":"2024-09-13T14:30:14.769683+0000","last_became_active":"2024-09-13T14:24:55.285081+0000","last_became_peered":"2024-09-13T14:24:55.285081+0000","last_unstale":"2024-09-13T14:30:14.769683+0000","last_undegraded":"2024-09-13T14:30:14.769683+0000","last_fullsized":"2024-09-13T14:30:14.769683+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:47:07.946120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":102,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.781425+0000","last_change":"2024-09-13T14:11:29.240351+0000","last_active":"2024-09-13T14:30:14.781425+0000","last_peered":"2024-09-13T14:30:14.781425+0000","last_clean":"2024-09-13T14:30:14.781425+0000","last_became_active":"2024-09-13T14:11:29.240060+0000","last_became_peered":"2024-09-13T14:11:29.240060+0000","last_unstale":"2024-09-13T14:30:14.781425+0000","last_undegraded":"2024-09-13T14:30:14.781425+0000","last_fullsized":"2024-09-13T14:30:14.781425+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:34:36.126368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":182,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.776617+0000","last_change":"2024-09-13T14:27:12.839943+0000","last_active":"2024-09-13T14:30:14.776617+0000","last_peered":"2024-09-13T14:30:14.776617+0000","last_clean":"2024-09-13T14:30:14.776617+0000","last_became_active":"2024-09-13T14:27:12.839534+0000","last_became_peered":"2024-09-13T14:27:12.839534+0000","last_unstale":"2024-09-13T14:30:14.776617+0000","last_undegraded":"2024-09-13T14:30:14.776617+0000","last_fullsized":"2024-09-13T14:30:14.776617+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:34:56.677019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":170,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.777096+0000","last_change":"2024-09-13T14:27:12.836567+0000","last_active":"2024-09-13T14:30:14.777096+0000","last_peered":"2024-09-13T14:30:14.777096+0000","last_clean":"2024-09-13T14:30:14.777096+0000","last_became_active":"2024-09-13T14:27:12.836271+0000","last_became_peered":"2024-09-13T14:27:12.836271+0000","last_unstale":"2024-09-13T14:30:14.777096+0000","last_undegraded":"2024-09-13T14:30:14.777096+0000","last_fullsized":"2024-09-13T14:30:14.777096+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:25:32.910438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":122,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.781890+0000","last_change":"2024-09-13T14:26:15.824075+0000","last_active":"2024-09-13T14:30:14.781890+0000","last_peered":"2024-09-13T14:30:14.781890+0000","last_clean":"2024-09-13T14:30:14.781890+0000","last_became_active":"2024-09-13T14:26:15.823762+0000","last_became_peered":"2024-09-13T14:26:15.823762+0000","last_unstale":"2024-09-13T14:30:14.781890+0000","last_undegraded":"2024-09-13T14:30:14.781890+0000","last_fullsized":"2024-09-13T14:30:14.781890+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:26:51.088388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":165,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-13T14:30:14.771502+0000","last_change":"2024-09-13T14:27:12.844201+0000","last_active":"2024-09-13T14:30:14.771502+0000","last_peered":"2024-09-13T14:30:14.771502+0000","last_clean":"2024-09-13T14:30:14.771502+0000","last_became_active":"2024-09-13T14:27:12.834075+0000","last_became_peered":"2024-09-13T14:27:12.834075+0000","last_unstale":"2024-09-13T14:30:14.771502+0000","last_undegraded":"2024-09-13T14:30:14.771502+0000","last_fullsized":"2024-09-13T14:30:14.771502+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:06.657030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":887029784,"num_objects":13536,"num_object_clones":0,"num_object_copies":40608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18726,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13536,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":222296,"num_write_kb":7546561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11595,"num_bytes_recovered":572391424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4337446912,"data_stored":4337434696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108396,"ondisk_log_size":108396,"up":48,"acting":43,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":616,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":17278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18272,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16793600,"data_stored":16777292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":23,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234239,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":473384,"kb_used_data":338080,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":93271768,"statfs":{"total":95995035648,"available":95510290432,"internally_reserved":0,"allocated":346193920,"data_stored":345422981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":138541515},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495875,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":451492,"kb_used_data":343964,"kb_used_omap":1,"kb_used_meta":107390,"kb_avail":93293660,"statfs":{"total":95995035648,"available":95532707840,"internally_reserved":0,"allocated":352219136,"data_stored":351452816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":109967815},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757512,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":755348,"kb_used_data":565900,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":92989804,"statfs":{"total":95995035648,"available":95221559296,"internally_reserved":0,"allocated":579481600,"data_stored":578726296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":193984966},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019148,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":793628,"kb_used_data":600084,"kb_used_omap":1,"kb_used_meta":193534,"kb_avail":92951524,"statfs":{"total":95995035648,"available":95182360576,"internally_reserved":0,"allocated":614486016,"data_stored":613726785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":198179270},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435332,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":266956,"kb_used_data":203332,"kb_used_omap":5,"kb_used_meta":63610,"kb_avail":93478196,"statfs":{"total":95995035648,"available":95721672704,"internally_reserved":0,"allocated":208211968,"data_stored":207464594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":65136766},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":446676598859,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":425201762385,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726925912,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":896924,"kb_used_data":658388,"kb_used_omap":7,"kb_used_meta":238520,"kb_avail":92848228,"statfs":{"total":95995035648,"available":95076585472,"internally_reserved":0,"allocated":674189312,"data_stored":673426655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":244244776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":667422720,"data_stored":667418648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1666449408,"data_stored":1666449408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":197001216,"data_stored":197001216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":592052224,"data_stored":592052224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":553521152,"data_stored":553517080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":333971456,"data_stored":333971456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327028736,"data_stored":327024664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:30:29.597 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:29.635 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:29.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:29.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:29.737 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:29.761 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:29.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:29.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:29.863 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:29.893 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:29.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:29.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:29.995 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:30.021 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:30.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:30.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:30.100 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:30:30.124 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:30.170 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:30.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:30.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:30.272 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:30.303 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:30.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:30.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:30.310 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:30:30.405 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:30:30.459 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:30.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:30.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:30:30.561 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:30.606 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:30.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:30.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:30.708 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:30.753 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:30.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:30.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:30.854 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:30.879 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:30.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:30.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:30.981 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:31.017 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:31.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:31.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:31.118 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:31.154 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:31.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:31.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:31.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:30 smithi102 ceph-mon[110183]: pgmap v456: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 836 MiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 0 B/s wr, 769 op/s; 18738/40014 objects degraded (46.829%); 0 B/s, 106 objects/s recovering 2024-09-13T14:30:31.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:30 smithi102 ceph-mon[115269]: pgmap v456: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 836 MiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 0 B/s wr, 769 op/s; 18738/40014 objects degraded (46.829%); 0 B/s, 106 objects/s recovering 2024-09-13T14:30:31.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:30 smithi195 ceph-mon[104262]: pgmap v456: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 836 MiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 0 B/s wr, 769 op/s; 18738/40014 objects degraded (46.829%); 0 B/s, 106 objects/s recovering 2024-09-13T14:30:31.256 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:31.282 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:31.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:31.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:31.384 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:31.410 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:31.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:31.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:31.512 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:31.543 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:31.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:31.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:31.645 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:31.692 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:31.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:31.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:31.794 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:31.821 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:31.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:31.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:31.923 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:30:31.969 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:31.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:31.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:30:32.070 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:32.098 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:32.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:32.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:32.200 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:32.242 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:32.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:32.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:32.344 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:32.370 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:32.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:32.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:32.472 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:32.497 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:32.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:32.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:32.540 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:30:32.599 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:32.629 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:32.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:32.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:32.731 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:32.756 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:32.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:32.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:32.858 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:32.883 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:32.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:32.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:32.984 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:33.010 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:33.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:33.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:33.112 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:33.124 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:32 smithi102 ceph-mon[110183]: pgmap v457: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 836 MiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 0 B/s wr, 769 op/s; 18738/40014 objects degraded (46.829%); 0 B/s, 106 objects/s recovering 2024-09-13T14:30:33.125 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:32 smithi102 ceph-mon[115269]: pgmap v457: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 836 MiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 0 B/s wr, 769 op/s; 18738/40014 objects degraded (46.829%); 0 B/s, 106 objects/s recovering 2024-09-13T14:30:33.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:33.164 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:33.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:33.241 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:30:33.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:32 smithi195 ceph-mon[104262]: pgmap v457: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 836 MiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 0 B/s wr, 769 op/s; 18738/40014 objects degraded (46.829%); 0 B/s, 106 objects/s recovering 2024-09-13T14:30:33.257 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:30:33.257 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:30:33.265 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:33.282 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:30:33.282 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:30:33.289 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:33.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:33.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:33.306 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:30:33.307 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:30:33.331 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:30:33.331 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:30:33.367 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:30:33.367 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:30:33.391 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:33.393 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:30:33.393 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:30:33.422 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:30:33.422 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:30:33.430 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:33.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:33.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:33.456 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:30:33.456 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:30:33.487 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:30:33.487 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [], 2024-09-13T14:30:33.487 INFO:teuthology.orchestra.run.smithi102.stdout: "complaint_time": 30, 2024-09-13T14:30:33.487 INFO:teuthology.orchestra.run.smithi102.stdout: "num_blocked_ops": 0 2024-09-13T14:30:33.487 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:30:33.534 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:30:33.573 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:33.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:33.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:30:33.675 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:33.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:33.717 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:33.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:33.818 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:30:33.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:33.858 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:33.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:30:33.959 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:33.974 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:33 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2875890968' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:30:33.975 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:33 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 18738/40014 objects degraded (46.829%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:30:33.975 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:33 smithi102 ceph-mon[110183]: osdmap e152: 8 total, 8 up, 6 in 2024-09-13T14:30:33.975 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:33 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2875890968' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:30:33.975 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:33 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 18738/40014 objects degraded (46.829%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:30:33.975 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:33 smithi102 ceph-mon[115269]: osdmap e152: 8 total, 8 up, 6 in 2024-09-13T14:30:33.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:33.989 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:33.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:34.090 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:34.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:34.115 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:34.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:34.216 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:30:34.238 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:34.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:34.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:30:34.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:33 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2875890968' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:30:34.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:33 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 18738/40014 objects degraded (46.829%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:30:34.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:33 smithi195 ceph-mon[104262]: osdmap e152: 8 total, 8 up, 6 in 2024-09-13T14:30:34.341 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:34.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:34.364 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:34.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:34.439 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T14:30:34.465 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:34.504 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:34.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:34.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:34.606 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:34.628 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:34.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:34.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:34.730 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:34.751 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:34.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:34.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:34.853 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:34.881 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:34.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:34.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:34.983 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:35.010 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:35.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:35.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:35.112 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:35.137 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:35.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:35.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:35.165 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:34 smithi102 ceph-mon[110183]: pgmap v458: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 606 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 0 B/s wr, 779 op/s; 18085/28935 objects degraded (62.502%); 0 B/s, 121 objects/s recovering 2024-09-13T14:30:35.166 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:34 smithi102 ceph-mon[110183]: osdmap e153: 8 total, 8 up, 6 in 2024-09-13T14:30:35.166 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:34 smithi102 ceph-mon[115269]: pgmap v458: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 606 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 0 B/s wr, 779 op/s; 18085/28935 objects degraded (62.502%); 0 B/s, 121 objects/s recovering 2024-09-13T14:30:35.166 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:34 smithi102 ceph-mon[115269]: osdmap e153: 8 total, 8 up, 6 in 2024-09-13T14:30:35.239 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:35.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:34 smithi195 ceph-mon[104262]: pgmap v458: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 606 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 0 B/s wr, 779 op/s; 18085/28935 objects degraded (62.502%); 0 B/s, 121 objects/s recovering 2024-09-13T14:30:35.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:34 smithi195 ceph-mon[104262]: osdmap e153: 8 total, 8 up, 6 in 2024-09-13T14:30:35.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:35.263 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:35.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:35.364 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:35.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:35.389 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:35.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:35.490 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:35.514 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:35.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:35.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:35.592 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:30:35.616 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:35.643 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:35.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:35.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:35.745 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:35.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:35.770 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:35.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:35.871 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:35.899 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:35.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:35.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:36.000 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:36.041 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:36.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:36.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:36.143 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:30:36.181 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:36.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:36.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:30:36.283 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:36.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:36.309 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:36.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:36.410 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:36.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:36.437 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:36.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:36.539 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:30:36.591 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:36.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:36.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:30:36.693 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:30:36.719 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:30:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:30:36] "GET /metrics HTTP/1.1" 200 36523 "" "Prometheus/2.51.0" 2024-09-13T14:30:36.735 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:36.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:36.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:30:36.837 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:36.891 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:36.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:36.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:36.993 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:37.036 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:37.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:37.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:37.072 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:36 smithi102 ceph-mon[110183]: pgmap v461: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 591 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 0 B/s wr, 519 op/s; 18085/28230 objects degraded (64.063%); 0 B/s, 81 objects/s recovering 2024-09-13T14:30:37.072 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:36 smithi102 ceph-mon[115269]: pgmap v461: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 591 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 0 B/s wr, 519 op/s; 18085/28230 objects degraded (64.063%); 0 B/s, 81 objects/s recovering 2024-09-13T14:30:37.138 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:37.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:37.162 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:37.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:37.188 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:30:37.195 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:30:37.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:36 smithi195 ceph-mon[104262]: pgmap v461: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 591 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 0 B/s wr, 519 op/s; 18085/28230 objects degraded (64.063%); 0 B/s, 81 objects/s recovering 2024-09-13T14:30:37.263 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:37.303 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:37.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:37.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:37.405 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:37.440 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:37.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:37.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:37.542 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:37.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:37.569 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:37.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:37.670 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:37.695 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:37.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:37.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:37.797 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:30:37.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:37.848 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:37.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:30:37.949 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:37.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:37.991 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:37.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:38.092 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:38.135 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:38.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:38.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:38.237 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:38.273 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:38.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:38.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:38.375 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:38.403 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:38.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:38.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:38.504 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:38.551 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:38.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:38.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:38.653 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:30:38.701 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:38.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:38.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:30:38.803 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:38.829 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:38.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:38.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:38.854 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is set 2024-09-13T14:30:38.930 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:38.966 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:38.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:38.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:39.068 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:39.097 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:39.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:39.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:39.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:38 smithi102 ceph-mon[110183]: pgmap v462: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 403 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 0 B/s wr, 866 op/s; 17506/19206 objects degraded (91.149%); 0 B/s, 153 objects/s recovering 2024-09-13T14:30:39.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:38 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/964485666' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:30:39.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:38 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:30:39.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:38 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 17506/19206 objects degraded (91.149%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:30:39.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:38 smithi102 ceph-mon[115269]: pgmap v462: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 403 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 0 B/s wr, 866 op/s; 17506/19206 objects degraded (91.149%); 0 B/s, 153 objects/s recovering 2024-09-13T14:30:39.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:38 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/964485666' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:30:39.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:38 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:30:39.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:38 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 17506/19206 objects degraded (91.149%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:30:39.200 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:39.240 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:39.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:39.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:39.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:38 smithi195 ceph-mon[104262]: pgmap v462: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 403 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 0 B/s wr, 866 op/s; 17506/19206 objects degraded (91.149%); 0 B/s, 153 objects/s recovering 2024-09-13T14:30:39.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:38 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/964485666' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:30:39.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:38 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:30:39.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:38 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 17506/19206 objects degraded (91.149%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:30:39.342 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:39.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:39.380 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:39.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:39.481 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:39.507 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:39.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:39.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:39.591 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:30:39.609 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:39.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:39.634 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:39.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:39.735 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:39.758 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:39.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:39.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:39.860 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:30:39.900 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:39.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:39.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:30:40.002 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:40.025 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:40.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:40.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:40.111 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:39 smithi102 ceph-mon[110183]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:30:40.111 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:39 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:30:40.111 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:39 smithi102 ceph-mon[110183]: osdmap e154: 8 total, 8 up, 6 in 2024-09-13T14:30:40.112 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:39 smithi102 ceph-mon[115269]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:30:40.112 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:39 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:30:40.112 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:39 smithi102 ceph-mon[115269]: osdmap e154: 8 total, 8 up, 6 in 2024-09-13T14:30:40.127 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:30:40.172 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:40.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:40.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:30:40.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:39 smithi195 ceph-mon[104262]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:30:40.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:39 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:30:40.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:39 smithi195 ceph-mon[104262]: osdmap e154: 8 total, 8 up, 6 in 2024-09-13T14:30:40.275 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:40.316 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:40.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:40.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:40.418 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:40.448 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:40.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:40.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:40.601 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:30:40.602 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:30:40.657 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:40.679 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:40.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:40.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:40.781 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:40.802 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:40.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:40.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:40.904 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:30:40.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:40.950 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:40.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:30:41.051 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:41.074 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:41.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:41.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:41.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:40 smithi102 ceph-mon[115269]: pgmap v464: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 391 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 0 B/s wr, 572 op/s; 17506/18618 objects degraded (94.027%); 0 B/s, 96 objects/s recovering 2024-09-13T14:30:41.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:40 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:30:41.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:40 smithi102 ceph-mon[110183]: pgmap v464: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 391 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 0 B/s wr, 572 op/s; 17506/18618 objects degraded (94.027%); 0 B/s, 96 objects/s recovering 2024-09-13T14:30:41.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:40 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:30:41.176 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:41.224 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:41.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:41.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:41.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:40 smithi195 ceph-mon[104262]: pgmap v464: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 391 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 0 B/s wr, 572 op/s; 17506/18618 objects degraded (94.027%); 0 B/s, 96 objects/s recovering 2024-09-13T14:30:41.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:40 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:30:41.326 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:41.351 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:41.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:41.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:41.422 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":464,"stamp":"2024-09-13T14:30:39.318150+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":409606141,"num_objects":6206,"num_object_clones":0,"num_object_copies":18618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17506,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6206,"num_whiteouts":0,"num_read":6391,"num_read_kb":13846,"num_write":230777,"num_write_kb":7561602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12911,"num_bytes_recovered":578357467,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109736,"ondisk_log_size":109736,"up":267,"acting":262,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":2289040,"kb_used_data":1322380,"kb_used_omap":19,"kb_used_meta":966444,"kb_avail":560181872,"statfs":{"total":575970213888,"available":573626236928,"internally_reserved":0,"allocated":1354117120,"data_stored":1349483655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20284,"internal_metadata":989638852},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,2,3,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-225378304,"num_objects":-3439,"num_object_clones":0,"num_object_copies":-10317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-579,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3439,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":579,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001754"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":176,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T14:30:34.828586+0000","last_change":"2024-09-13T14:27:12.839984+0000","last_active":"2024-09-13T14:30:34.828586+0000","last_peered":"2024-09-13T14:30:34.828586+0000","last_clean":"2024-09-13T14:30:34.828586+0000","last_became_active":"2024-09-13T14:27:12.839653+0000","last_became_peered":"2024-09-13T14:27:12.839653+0000","last_unstale":"2024-09-13T14:30:34.828586+0000","last_undegraded":"2024-09-13T14:30:34.828586+0000","last_fullsized":"2024-09-13T14:30:34.828586+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:14:13.672830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":135,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T14:30:33.783182+0000","last_change":"2024-09-13T14:27:12.846760+0000","last_active":"2024-09-13T14:30:33.783182+0000","last_peered":"2024-09-13T14:30:33.783182+0000","last_clean":"2024-09-13T14:30:33.783182+0000","last_became_active":"2024-09-13T14:27:12.838041+0000","last_became_peered":"2024-09-13T14:27:12.838041+0000","last_unstale":"2024-09-13T14:30:33.783182+0000","last_undegraded":"2024-09-13T14:30:33.783182+0000","last_fullsized":"2024-09-13T14:30:33.783182+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:22.402859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":116,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T14:30:33.793169+0000","last_change":"2024-09-13T14:24:55.282407+0000","last_active":"2024-09-13T14:30:33.793169+0000","last_peered":"2024-09-13T14:30:33.793169+0000","last_clean":"2024-09-13T14:30:33.793169+0000","last_became_active":"2024-09-13T14:24:55.282083+0000","last_became_peered":"2024-09-13T14:24:55.282083+0000","last_unstale":"2024-09-13T14:30:33.793169+0000","last_undegraded":"2024-09-13T14:30:33.793169+0000","last_fullsized":"2024-09-13T14:30:33.793169+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:23:10.525707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":153,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T14:30:34.785060+0000","last_change":"2024-09-13T14:24:55.284433+0000","last_active":"2024-09-13T14:30:34.785060+0000","last_peered":"2024-09-13T14:30:34.785060+0000","last_clean":"2024-09-13T14:30:34.785060+0000","last_became_active":"2024-09-13T14:24:55.284143+0000","last_became_peered":"2024-09-13T14:24:55.284143+0000","last_unstale":"2024-09-13T14:30:34.785060+0000","last_undegraded":"2024-09-13T14:30:34.785060+0000","last_fullsized":"2024-09-13T14:30:34.785060+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:28.107405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":155,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T14:30:34.782607+0000","last_change":"2024-09-13T14:24:55.282416+0000","last_active":"2024-09-13T14:30:34.782607+0000","last_peered":"2024-09-13T14:30:34.782607+0000","last_clean":"2024-09-13T14:30:34.782607+0000","last_became_active":"2024-09-13T14:24:55.282045+0000","last_became_peered":"2024-09-13T14:24:55.282045+0000","last_unstale":"2024-09-13T14:30:34.782607+0000","last_undegraded":"2024-09-13T14:30:34.782607+0000","last_fullsized":"2024-09-13T14:30:34.782607+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:58.692897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":103,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T14:30:33.792833+0000","last_change":"2024-09-13T14:11:29.244452+0000","last_active":"2024-09-13T14:30:33.792833+0000","last_peered":"2024-09-13T14:30:33.792833+0000","last_clean":"2024-09-13T14:30:33.792833+0000","last_became_active":"2024-09-13T14:11:29.244102+0000","last_became_peered":"2024-09-13T14:11:29.244102+0000","last_unstale":"2024-09-13T14:30:33.792833+0000","last_undegraded":"2024-09-13T14:30:33.792833+0000","last_fullsized":"2024-09-13T14:30:33.792833+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:13:19.165534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":176,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T14:30:34.829566+0000","last_change":"2024-09-13T14:27:12.834300+0000","last_active":"2024-09-13T14:30:34.829566+0000","last_peered":"2024-09-13T14:30:34.829566+0000","last_clean":"2024-09-13T14:30:34.829566+0000","last_became_active":"2024-09-13T14:27:12.834046+0000","last_became_peered":"2024-09-13T14:27:12.834046+0000","last_unstale":"2024-09-13T14:30:34.829566+0000","last_undegraded":"2024-09-13T14:30:34.829566+0000","last_fullsized":"2024-09-13T14:30:34.829566+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:46:58.906862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":126,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T14:30:33.779205+0000","last_change":"2024-09-13T14:23:18.039061+0000","last_active":"2024-09-13T14:30:33.779205+0000","last_peered":"2024-09-13T14:30:33.779205+0000","last_clean":"2024-09-13T14:30:33.779205+0000","last_became_active":"2024-09-13T14:23:18.038661+0000","last_became_peered":"2024-09-13T14:23:18.038661+0000","last_unstale":"2024-09-13T14:30:33.779205+0000","last_undegraded":"2024-09-13T14:30:33.779205+0000","last_fullsized":"2024-09-13T14:30:33.779205+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:05:29.109844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.9","version":"153'14146","reported_seq":14193,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T14:30:36.921969+0000","last_change":"2024-09-13T14:26:21.865664+0000","last_active":"2024-09-13T14:30:36.921969+0000","last_peered":"2024-09-13T14:30:36.921969+0000","last_clean":"2024-09-13T14:30:36.921969+0000","last_became_active":"2024-09-13T14:26:21.865396+0000","last_became_peered":"2024-09-13T14:26:21.865396+0000","last_unstale":"2024-09-13T14:30:36.921969+0000","last_undegraded":"2024-09-13T14:30:36.921969+0000","last_fullsized":"2024-09-13T14:30:36.921969+0000","mapping_epoch":116,"log_start":"131'7400","ondisk_log_start":"131'7400","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6746,"log_dups_size":0,"ondisk_log_size":6746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:54:35.654265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24641536,"num_objects":376,"num_object_clones":0,"num_object_copies":1128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14146,"num_write_kb":464704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":119,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T14:30:33.785019+0000","last_change":"2024-09-13T14:23:18.038597+0000","last_active":"2024-09-13T14:30:33.785019+0000","last_peered":"2024-09-13T14:30:33.785019+0000","last_clean":"2024-09-13T14:30:33.785019+0000","last_became_active":"2024-09-13T14:23:18.038240+0000","last_became_peered":"2024-09-13T14:23:18.038240+0000","last_unstale":"2024-09-13T14:30:33.785019+0000","last_undegraded":"2024-09-13T14:30:33.785019+0000","last_fullsized":"2024-09-13T14:30:33.785019+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:27:51.842195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":102,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T14:30:33.791064+0000","last_change":"2024-09-13T14:11:30.254534+0000","last_active":"2024-09-13T14:30:33.791064+0000","last_peered":"2024-09-13T14:30:33.791064+0000","last_clean":"2024-09-13T14:30:33.791064+0000","last_became_active":"2024-09-13T14:11:30.253959+0000","last_became_peered":"2024-09-13T14:11:30.253959+0000","last_unstale":"2024-09-13T14:30:33.791064+0000","last_undegraded":"2024-09-13T14:30:33.791064+0000","last_fullsized":"2024-09-13T14:30:33.791064+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:09:54.025853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"153'14660","reported_seq":14749,"reported_epoch":153,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:30:36.492460+0000","last_change":"2024-09-13T14:27:14.009733+0000","last_active":"2024-09-13T14:30:36.492460+0000","last_peered":"2024-09-13T14:30:36.492460+0000","last_clean":"2024-09-13T14:27:11.786427+0000","last_became_active":"2024-09-13T14:27:13.994962+0000","last_became_peered":"2024-09-13T14:27:13.994962+0000","last_unstale":"2024-09-13T14:30:36.492460+0000","last_undegraded":"2024-09-13T14:27:13.854850+0000","last_fullsized":"2024-09-13T14:27:13.824319+0000","mapping_epoch":125,"log_start":"133'8400","ondisk_log_start":"133'8400","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6260,"log_dups_size":0,"ondisk_log_size":6260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25952256,"num_objects":396,"num_object_clones":0,"num_object_copies":1188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5198,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14660,"num_write_kb":481792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5],"avail_no_missing":["0","2","5"],"object_location_counts":[{"shards":"0,5","objects":-4802}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":133,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T14:30:33.786233+0000","last_change":"2024-09-13T14:26:15.828529+0000","last_active":"2024-09-13T14:30:33.786233+0000","last_peered":"2024-09-13T14:30:33.786233+0000","last_clean":"2024-09-13T14:30:33.786233+0000","last_became_active":"2024-09-13T14:26:15.820486+0000","last_became_peered":"2024-09-13T14:26:15.820486+0000","last_unstale":"2024-09-13T14:30:33.786233+0000","last_undegraded":"2024-09-13T14:30:33.786233+0000","last_fullsized":"2024-09-13T14:30:33.786233+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:11:46.283172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":140,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T14:30:33.790404+0000","last_change":"2024-09-13T14:26:15.824632+0000","last_active":"2024-09-13T14:30:33.790404+0000","last_peered":"2024-09-13T14:30:33.790404+0000","last_clean":"2024-09-13T14:30:33.790404+0000","last_became_active":"2024-09-13T14:26:15.824330+0000","last_became_peered":"2024-09-13T14:26:15.824330+0000","last_unstale":"2024-09-13T14:30:33.790404+0000","last_undegraded":"2024-09-13T14:30:33.790404+0000","last_fullsized":"2024-09-13T14:30:33.790404+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:25.288116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"153'13999","reported_seq":23884,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T14:30:36.430674+0000","last_change":"2024-09-13T14:28:49.532961+0000","last_active":"2024-09-13T14:30:36.430674+0000","last_peered":"2024-09-13T14:30:36.430674+0000","last_clean":"2024-09-13T14:30:36.430674+0000","last_became_active":"2024-09-13T14:28:49.511508+0000","last_became_peered":"2024-09-13T14:28:49.511508+0000","last_unstale":"2024-09-13T14:30:36.430674+0000","last_undegraded":"2024-09-13T14:30:36.430674+0000","last_fullsized":"2024-09-13T14:30:36.430674+0000","mapping_epoch":137,"log_start":"132'7700","ondisk_log_start":"132'7700","created":116,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6299,"log_dups_size":0,"ondisk_log_size":6299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:42:44.821074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26673152,"num_objects":407,"num_object_clones":0,"num_object_copies":1221,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13999,"num_write_kb":460992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4893,"num_bytes_recovered":320667648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":104,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T14:30:33.784955+0000","last_change":"2024-09-13T14:11:29.232897+0000","last_active":"2024-09-13T14:30:33.784955+0000","last_peered":"2024-09-13T14:30:33.784955+0000","last_clean":"2024-09-13T14:30:33.784955+0000","last_became_active":"2024-09-13T14:11:29.232637+0000","last_became_peered":"2024-09-13T14:11:29.232637+0000","last_unstale":"2024-09-13T14:30:33.784955+0000","last_undegraded":"2024-09-13T14:30:33.784955+0000","last_fullsized":"2024-09-13T14:30:33.784955+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:31:05.856046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":126,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T14:30:33.782988+0000","last_change":"2024-09-13T14:26:15.822452+0000","last_active":"2024-09-13T14:30:33.782988+0000","last_peered":"2024-09-13T14:30:33.782988+0000","last_clean":"2024-09-13T14:30:33.782988+0000","last_became_active":"2024-09-13T14:26:15.822111+0000","last_became_peered":"2024-09-13T14:26:15.822111+0000","last_unstale":"2024-09-13T14:30:33.782988+0000","last_undegraded":"2024-09-13T14:30:33.782988+0000","last_fullsized":"2024-09-13T14:30:33.782988+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:32:53.078767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"153'14540","reported_seq":15036,"reported_epoch":153,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T14:30:37.148165+0000","last_change":"2024-09-13T14:30:32.988585+0000","last_active":"2024-09-13T14:30:37.148165+0000","last_peered":"2024-09-13T14:30:37.148165+0000","last_clean":"2024-09-13T14:27:11.793438+0000","last_became_active":"2024-09-13T14:27:14.032063+0000","last_became_peered":"2024-09-13T14:27:14.032063+0000","last_unstale":"2024-09-13T14:30:37.148165+0000","last_undegraded":"2024-09-13T14:27:13.861208+0000","last_fullsized":"2024-09-13T14:27:13.827412+0000","mapping_epoch":125,"log_start":"132'8100","ondisk_log_start":"132'8100","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6440,"log_dups_size":0,"ondisk_log_size":6440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:07:10.560903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26083328,"num_objects":398,"num_object_clones":0,"num_object_copies":1194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4862,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14540,"num_write_kb":478016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":216,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3],"avail_no_missing":["4","2","3"],"object_location_counts":[{"shards":"3,4","objects":-4464}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":118,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T14:30:33.791642+0000","last_change":"2024-09-13T14:24:55.290798+0000","last_active":"2024-09-13T14:30:33.791642+0000","last_peered":"2024-09-13T14:30:33.791642+0000","last_clean":"2024-09-13T14:30:33.791642+0000","last_became_active":"2024-09-13T14:24:55.290510+0000","last_became_peered":"2024-09-13T14:24:55.290510+0000","last_unstale":"2024-09-13T14:30:33.791642+0000","last_undegraded":"2024-09-13T14:30:33.791642+0000","last_fullsized":"2024-09-13T14:30:33.791642+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:19:03.129132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":162,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T14:30:34.782639+0000","last_change":"2024-09-13T14:26:15.823316+0000","last_active":"2024-09-13T14:30:34.782639+0000","last_peered":"2024-09-13T14:30:34.782639+0000","last_clean":"2024-09-13T14:30:34.782639+0000","last_became_active":"2024-09-13T14:26:15.822980+0000","last_became_peered":"2024-09-13T14:26:15.822980+0000","last_unstale":"2024-09-13T14:30:34.782639+0000","last_undegraded":"2024-09-13T14:30:34.782639+0000","last_fullsized":"2024-09-13T14:30:34.782639+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:33:47.420548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"153'14442","reported_seq":24628,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T14:30:36.936070+0000","last_change":"2024-09-13T14:30:34.826470+0000","last_active":"2024-09-13T14:30:36.936070+0000","last_peered":"2024-09-13T14:30:36.936070+0000","last_clean":"2024-09-13T14:30:36.936070+0000","last_became_active":"2024-09-13T14:30:34.784024+0000","last_became_peered":"2024-09-13T14:30:34.784024+0000","last_unstale":"2024-09-13T14:30:36.936070+0000","last_undegraded":"2024-09-13T14:30:36.936070+0000","last_fullsized":"2024-09-13T14:30:36.936070+0000","mapping_epoch":152,"log_start":"132'7700","ondisk_log_start":"132'7700","created":116,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6742,"log_dups_size":0,"ondisk_log_size":6742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:44.838006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24444952,"num_objects":374,"num_object_clones":0,"num_object_copies":1122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":374,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14442,"num_write_kb":474049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5057,"num_bytes_recovered":152240128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":160,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T14:30:34.830015+0000","last_change":"2024-09-13T14:24:55.284499+0000","last_active":"2024-09-13T14:30:34.830015+0000","last_peered":"2024-09-13T14:30:34.830015+0000","last_clean":"2024-09-13T14:30:34.830015+0000","last_became_active":"2024-09-13T14:24:55.284096+0000","last_became_peered":"2024-09-13T14:24:55.284096+0000","last_unstale":"2024-09-13T14:30:34.830015+0000","last_undegraded":"2024-09-13T14:30:34.830015+0000","last_fullsized":"2024-09-13T14:30:34.830015+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:02:06.836099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":116,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T14:30:33.791665+0000","last_change":"2024-09-13T14:24:55.291371+0000","last_active":"2024-09-13T14:30:33.791665+0000","last_peered":"2024-09-13T14:30:33.791665+0000","last_clean":"2024-09-13T14:30:33.791665+0000","last_became_active":"2024-09-13T14:24:55.291047+0000","last_became_peered":"2024-09-13T14:24:55.291047+0000","last_unstale":"2024-09-13T14:30:33.791665+0000","last_undegraded":"2024-09-13T14:30:33.791665+0000","last_fullsized":"2024-09-13T14:30:33.791665+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:43.912412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"153'14191","reported_seq":14238,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T14:30:37.181724+0000","last_change":"2024-09-13T14:26:21.864062+0000","last_active":"2024-09-13T14:30:37.181724+0000","last_peered":"2024-09-13T14:30:37.181724+0000","last_clean":"2024-09-13T14:30:37.181724+0000","last_became_active":"2024-09-13T14:26:21.863721+0000","last_became_peered":"2024-09-13T14:26:21.863721+0000","last_unstale":"2024-09-13T14:30:37.181724+0000","last_undegraded":"2024-09-13T14:30:37.181724+0000","last_fullsized":"2024-09-13T14:30:37.181724+0000","mapping_epoch":116,"log_start":"132'7800","ondisk_log_start":"132'7800","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6391,"log_dups_size":0,"ondisk_log_size":6391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:15:42.744309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22609920,"num_objects":345,"num_object_clones":0,"num_object_copies":1035,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14191,"num_write_kb":465152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":127,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T14:30:33.784135+0000","last_change":"2024-09-13T14:26:15.827333+0000","last_active":"2024-09-13T14:30:33.784135+0000","last_peered":"2024-09-13T14:30:33.784135+0000","last_clean":"2024-09-13T14:30:33.784135+0000","last_became_active":"2024-09-13T14:26:15.826945+0000","last_became_peered":"2024-09-13T14:26:15.826945+0000","last_unstale":"2024-09-13T14:30:33.784135+0000","last_undegraded":"2024-09-13T14:30:33.784135+0000","last_fullsized":"2024-09-13T14:30:33.784135+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:35:24.870052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":149,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T14:30:33.790184+0000","last_change":"2024-09-13T14:27:12.842604+0000","last_active":"2024-09-13T14:30:33.790184+0000","last_peered":"2024-09-13T14:30:33.790184+0000","last_clean":"2024-09-13T14:30:33.790184+0000","last_became_active":"2024-09-13T14:27:12.841643+0000","last_became_peered":"2024-09-13T14:27:12.841643+0000","last_unstale":"2024-09-13T14:30:33.790184+0000","last_undegraded":"2024-09-13T14:30:33.790184+0000","last_fullsized":"2024-09-13T14:30:33.790184+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:34:34.751790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"153'14269","reported_seq":14315,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T14:30:36.936729+0000","last_change":"2024-09-13T14:26:21.867125+0000","last_active":"2024-09-13T14:30:36.936729+0000","last_peered":"2024-09-13T14:30:36.936729+0000","last_clean":"2024-09-13T14:30:36.936729+0000","last_became_active":"2024-09-13T14:26:21.866815+0000","last_became_peered":"2024-09-13T14:26:21.866815+0000","last_unstale":"2024-09-13T14:30:36.936729+0000","last_undegraded":"2024-09-13T14:30:36.936729+0000","last_fullsized":"2024-09-13T14:30:36.936729+0000","mapping_epoch":116,"log_start":"130'6300","ondisk_log_start":"130'6300","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":7969,"log_dups_size":0,"ondisk_log_size":7969,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:44.218255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28901376,"num_objects":441,"num_object_clones":0,"num_object_copies":1323,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14269,"num_write_kb":470720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":126,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T14:30:33.790712+0000","last_change":"2024-09-13T14:27:12.842271+0000","last_active":"2024-09-13T14:30:33.790712+0000","last_peered":"2024-09-13T14:30:33.790712+0000","last_clean":"2024-09-13T14:30:33.790712+0000","last_became_active":"2024-09-13T14:27:12.841479+0000","last_became_peered":"2024-09-13T14:27:12.841479+0000","last_unstale":"2024-09-13T14:30:33.790712+0000","last_undegraded":"2024-09-13T14:30:33.790712+0000","last_fullsized":"2024-09-13T14:30:33.790712+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:30:22.746575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":125,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T14:30:33.791353+0000","last_change":"2024-09-13T14:26:15.825304+0000","last_active":"2024-09-13T14:30:33.791353+0000","last_peered":"2024-09-13T14:30:33.791353+0000","last_clean":"2024-09-13T14:30:33.791353+0000","last_became_active":"2024-09-13T14:26:15.825018+0000","last_became_peered":"2024-09-13T14:26:15.825018+0000","last_unstale":"2024-09-13T14:30:33.791353+0000","last_undegraded":"2024-09-13T14:30:33.791353+0000","last_fullsized":"2024-09-13T14:30:33.791353+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:22:59.408928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"153'14397","reported_seq":14444,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T14:30:37.139032+0000","last_change":"2024-09-13T14:26:21.865199+0000","last_active":"2024-09-13T14:30:37.139032+0000","last_peered":"2024-09-13T14:30:37.139032+0000","last_clean":"2024-09-13T14:30:37.139032+0000","last_became_active":"2024-09-13T14:26:21.864903+0000","last_became_peered":"2024-09-13T14:26:21.864903+0000","last_unstale":"2024-09-13T14:30:37.139032+0000","last_undegraded":"2024-09-13T14:30:37.139032+0000","last_fullsized":"2024-09-13T14:30:37.139032+0000","mapping_epoch":116,"log_start":"132'8000","ondisk_log_start":"132'8000","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6397,"log_dups_size":0,"ondisk_log_size":6397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:25.014330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23527424,"num_objects":359,"num_object_clones":0,"num_object_copies":1077,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14397,"num_write_kb":472192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":170,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T14:30:34.828371+0000","last_change":"2024-09-13T14:26:15.823898+0000","last_active":"2024-09-13T14:30:34.828371+0000","last_peered":"2024-09-13T14:30:34.828371+0000","last_clean":"2024-09-13T14:30:34.828371+0000","last_became_active":"2024-09-13T14:26:15.823125+0000","last_became_peered":"2024-09-13T14:26:15.823125+0000","last_unstale":"2024-09-13T14:30:34.828371+0000","last_undegraded":"2024-09-13T14:30:34.828371+0000","last_fullsized":"2024-09-13T14:30:34.828371+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:51:27.949528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":132,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T14:30:34.829689+0000","last_change":"2024-09-13T14:26:15.823198+0000","last_active":"2024-09-13T14:30:34.829689+0000","last_peered":"2024-09-13T14:30:34.829689+0000","last_clean":"2024-09-13T14:30:34.829689+0000","last_became_active":"2024-09-13T14:26:15.822717+0000","last_became_peered":"2024-09-13T14:26:15.822717+0000","last_unstale":"2024-09-13T14:30:34.829689+0000","last_undegraded":"2024-09-13T14:30:34.829689+0000","last_fullsized":"2024-09-13T14:30:34.829689+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:22.471622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"153'14495","reported_seq":14547,"reported_epoch":153,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:30:36.934954+0000","last_change":"2024-09-13T14:27:14.052640+0000","last_active":"2024-09-13T14:30:36.934954+0000","last_peered":"2024-09-13T14:30:36.934954+0000","last_clean":"2024-09-13T14:27:11.828545+0000","last_became_active":"2024-09-13T14:27:14.042591+0000","last_became_peered":"2024-09-13T14:27:14.042591+0000","last_unstale":"2024-09-13T14:30:36.934954+0000","last_undegraded":"2024-09-13T14:27:13.862613+0000","last_fullsized":"2024-09-13T14:27:13.823368+0000","mapping_epoch":125,"log_start":"132'7800","ondisk_log_start":"132'7800","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6695,"log_dups_size":0,"ondisk_log_size":6695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:17:38.266600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23789568,"num_objects":363,"num_object_clones":0,"num_object_copies":1089,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14495,"num_write_kb":475456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5],"avail_no_missing":["7","2","5"],"object_location_counts":[{"shards":"5,7","objects":-4759}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":449,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T14:30:33.782936+0000","last_change":"2024-09-13T14:23:18.032973+0000","last_active":"2024-09-13T14:30:33.782936+0000","last_peered":"2024-09-13T14:30:33.782936+0000","last_clean":"2024-09-13T14:30:33.782936+0000","last_became_active":"2024-09-13T14:23:18.032579+0000","last_became_peered":"2024-09-13T14:23:18.032579+0000","last_unstale":"2024-09-13T14:30:33.782936+0000","last_undegraded":"2024-09-13T14:30:33.782936+0000","last_fullsized":"2024-09-13T14:30:33.782936+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:07:41.927740+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:17.109445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":482,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T14:30:33.791951+0000","last_change":"2024-09-13T14:27:15.865400+0000","last_active":"2024-09-13T14:30:33.791951+0000","last_peered":"2024-09-13T14:30:33.791951+0000","last_clean":"2024-09-13T14:30:33.791951+0000","last_became_active":"2024-09-13T14:27:15.864835+0000","last_became_peered":"2024-09-13T14:27:15.864835+0000","last_unstale":"2024-09-13T14:30:33.791951+0000","last_undegraded":"2024-09-13T14:30:33.791951+0000","last_fullsized":"2024-09-13T14:30:33.791951+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:27:10.928041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":111,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T14:30:34.828521+0000","last_change":"2024-09-13T14:24:55.288058+0000","last_active":"2024-09-13T14:30:34.828521+0000","last_peered":"2024-09-13T14:30:34.828521+0000","last_clean":"2024-09-13T14:30:34.828521+0000","last_became_active":"2024-09-13T14:24:55.287601+0000","last_became_peered":"2024-09-13T14:24:55.287601+0000","last_unstale":"2024-09-13T14:30:34.828521+0000","last_undegraded":"2024-09-13T14:30:34.828521+0000","last_fullsized":"2024-09-13T14:30:34.828521+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:56:34.622696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":103,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T14:30:33.779963+0000","last_change":"2024-09-13T14:11:30.253749+0000","last_active":"2024-09-13T14:30:33.779963+0000","last_peered":"2024-09-13T14:30:33.779963+0000","last_clean":"2024-09-13T14:30:33.779963+0000","last_became_active":"2024-09-13T14:11:30.253499+0000","last_became_peered":"2024-09-13T14:11:30.253499+0000","last_unstale":"2024-09-13T14:30:33.779963+0000","last_undegraded":"2024-09-13T14:30:33.779963+0000","last_fullsized":"2024-09-13T14:30:33.779963+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:02:00.987335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"153'14554","reported_seq":14600,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T14:30:36.905518+0000","last_change":"2024-09-13T14:26:21.865805+0000","last_active":"2024-09-13T14:30:36.905518+0000","last_peered":"2024-09-13T14:30:36.905518+0000","last_clean":"2024-09-13T14:30:36.905518+0000","last_became_active":"2024-09-13T14:26:21.865288+0000","last_became_peered":"2024-09-13T14:26:21.865288+0000","last_unstale":"2024-09-13T14:30:36.905518+0000","last_undegraded":"2024-09-13T14:30:36.905518+0000","last_fullsized":"2024-09-13T14:30:36.905518+0000","mapping_epoch":116,"log_start":"130'6600","ondisk_log_start":"130'6600","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":7954,"log_dups_size":0,"ondisk_log_size":7954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:46.510065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27918336,"num_objects":426,"num_object_clones":0,"num_object_copies":1278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14554,"num_write_kb":479360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":364,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T14:30:33.782840+0000","last_change":"2024-09-13T14:26:15.835178+0000","last_active":"2024-09-13T14:30:33.782840+0000","last_peered":"2024-09-13T14:30:33.782840+0000","last_clean":"2024-09-13T14:30:33.782840+0000","last_became_active":"2024-09-13T14:26:15.820283+0000","last_became_peered":"2024-09-13T14:26:15.820283+0000","last_unstale":"2024-09-13T14:30:33.782840+0000","last_undegraded":"2024-09-13T14:30:33.782840+0000","last_fullsized":"2024-09-13T14:30:33.782840+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:30.340038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":154,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T14:30:34.782203+0000","last_change":"2024-09-13T14:24:55.286576+0000","last_active":"2024-09-13T14:30:34.782203+0000","last_peered":"2024-09-13T14:30:34.782203+0000","last_clean":"2024-09-13T14:30:34.782203+0000","last_became_active":"2024-09-13T14:24:55.286284+0000","last_became_peered":"2024-09-13T14:24:55.286284+0000","last_unstale":"2024-09-13T14:30:34.782203+0000","last_undegraded":"2024-09-13T14:30:34.782203+0000","last_fullsized":"2024-09-13T14:30:34.782203+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:35.589736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":180,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T14:30:34.782875+0000","last_change":"2024-09-13T14:27:12.836204+0000","last_active":"2024-09-13T14:30:34.782875+0000","last_peered":"2024-09-13T14:30:34.782875+0000","last_clean":"2024-09-13T14:30:34.782875+0000","last_became_active":"2024-09-13T14:27:12.835900+0000","last_became_peered":"2024-09-13T14:27:12.835900+0000","last_unstale":"2024-09-13T14:30:34.782875+0000","last_undegraded":"2024-09-13T14:30:34.782875+0000","last_fullsized":"2024-09-13T14:30:34.782875+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:36:52.121730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"153'14364","reported_seq":19750,"reported_epoch":153,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T14:30:36.929575+0000","last_change":"2024-09-13T14:28:48.300429+0000","last_active":"2024-09-13T14:30:36.929575+0000","last_peered":"2024-09-13T14:30:36.929575+0000","last_clean":"2024-09-13T14:27:11.802376+0000","last_became_active":"2024-09-13T14:27:13.985759+0000","last_became_peered":"2024-09-13T14:27:13.985759+0000","last_unstale":"2024-09-13T14:30:36.929575+0000","last_undegraded":"2024-09-13T14:27:13.852391+0000","last_fullsized":"2024-09-13T14:27:13.824524+0000","mapping_epoch":125,"log_start":"130'6400","ondisk_log_start":"130'6400","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":7964,"log_dups_size":0,"ondisk_log_size":7964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:58:21.117245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26214400,"num_objects":400,"num_object_clones":0,"num_object_copies":1200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14364,"num_write_kb":472448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2661,"num_bytes_recovered":99483648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7],"avail_no_missing":["5","2","7"],"object_location_counts":[{"shards":"5,7","objects":-1912}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":534,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T14:30:34.828726+0000","last_change":"2024-09-13T14:27:15.863966+0000","last_active":"2024-09-13T14:30:34.828726+0000","last_peered":"2024-09-13T14:30:34.828726+0000","last_clean":"2024-09-13T14:30:34.828726+0000","last_became_active":"2024-09-13T14:27:15.863124+0000","last_became_peered":"2024-09-13T14:27:15.863124+0000","last_unstale":"2024-09-13T14:30:34.828726+0000","last_undegraded":"2024-09-13T14:30:34.828726+0000","last_fullsized":"2024-09-13T14:30:34.828726+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:35:02.715013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":77,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T14:30:33.779121+0000","last_change":"2024-09-13T14:27:12.829079+0000","last_active":"2024-09-13T14:30:33.779121+0000","last_peered":"2024-09-13T14:30:33.779121+0000","last_clean":"2024-09-13T14:30:33.779121+0000","last_became_active":"2024-09-13T14:27:12.828619+0000","last_became_peered":"2024-09-13T14:27:12.828619+0000","last_unstale":"2024-09-13T14:30:33.779121+0000","last_undegraded":"2024-09-13T14:30:33.779121+0000","last_fullsized":"2024-09-13T14:30:33.779121+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:39:08.260304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":114,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T14:30:34.783308+0000","last_change":"2024-09-13T14:26:15.817732+0000","last_active":"2024-09-13T14:30:34.783308+0000","last_peered":"2024-09-13T14:30:34.783308+0000","last_clean":"2024-09-13T14:30:34.783308+0000","last_became_active":"2024-09-13T14:26:15.817403+0000","last_became_peered":"2024-09-13T14:26:15.817403+0000","last_unstale":"2024-09-13T14:30:34.783308+0000","last_undegraded":"2024-09-13T14:30:34.783308+0000","last_fullsized":"2024-09-13T14:30:34.783308+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:56:39.961300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"153'14142","reported_seq":14230,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T14:30:36.365847+0000","last_change":"2024-09-13T14:26:21.866279+0000","last_active":"2024-09-13T14:30:36.365847+0000","last_peered":"2024-09-13T14:30:36.365847+0000","last_clean":"2024-09-13T14:30:36.365847+0000","last_became_active":"2024-09-13T14:26:21.864793+0000","last_became_peered":"2024-09-13T14:26:21.864793+0000","last_unstale":"2024-09-13T14:30:36.365847+0000","last_undegraded":"2024-09-13T14:30:36.365847+0000","last_fullsized":"2024-09-13T14:30:36.365847+0000","mapping_epoch":116,"log_start":"132'7800","ondisk_log_start":"132'7800","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6342,"log_dups_size":0,"ondisk_log_size":6342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:48:28.309201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24248320,"num_objects":370,"num_object_clones":0,"num_object_copies":1110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14142,"num_write_kb":464384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":913,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T14:30:33.791459+0000","last_change":"2024-09-13T14:26:18.829131+0000","last_active":"2024-09-13T14:30:33.791459+0000","last_peered":"2024-09-13T14:30:33.791459+0000","last_clean":"2024-09-13T14:30:33.791459+0000","last_became_active":"2024-09-13T14:26:18.828619+0000","last_became_peered":"2024-09-13T14:26:18.828619+0000","last_unstale":"2024-09-13T14:30:33.791459+0000","last_undegraded":"2024-09-13T14:30:33.791459+0000","last_fullsized":"2024-09-13T14:30:33.791459+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:24:37.073618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":118,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T14:30:33.789919+0000","last_change":"2024-09-13T14:23:18.042891+0000","last_active":"2024-09-13T14:30:33.789919+0000","last_peered":"2024-09-13T14:30:33.789919+0000","last_clean":"2024-09-13T14:30:33.789919+0000","last_became_active":"2024-09-13T14:23:18.042562+0000","last_became_peered":"2024-09-13T14:23:18.042562+0000","last_unstale":"2024-09-13T14:30:33.789919+0000","last_undegraded":"2024-09-13T14:30:33.789919+0000","last_fullsized":"2024-09-13T14:30:33.789919+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:30:42.996661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":147,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T14:30:34.828501+0000","last_change":"2024-09-13T14:27:12.832666+0000","last_active":"2024-09-13T14:30:34.828501+0000","last_peered":"2024-09-13T14:30:34.828501+0000","last_clean":"2024-09-13T14:30:34.828501+0000","last_became_active":"2024-09-13T14:27:12.832332+0000","last_became_peered":"2024-09-13T14:27:12.832332+0000","last_unstale":"2024-09-13T14:30:34.828501+0000","last_undegraded":"2024-09-13T14:30:34.828501+0000","last_fullsized":"2024-09-13T14:30:34.828501+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:07:17.724156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"153'14101","reported_seq":14189,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T14:30:36.425985+0000","last_change":"2024-09-13T14:26:21.862954+0000","last_active":"2024-09-13T14:30:36.425985+0000","last_peered":"2024-09-13T14:30:36.425985+0000","last_clean":"2024-09-13T14:30:36.425985+0000","last_became_active":"2024-09-13T14:26:21.862650+0000","last_became_peered":"2024-09-13T14:26:21.862650+0000","last_unstale":"2024-09-13T14:30:36.425985+0000","last_undegraded":"2024-09-13T14:30:36.425985+0000","last_fullsized":"2024-09-13T14:30:36.425985+0000","mapping_epoch":116,"log_start":"132'7800","ondisk_log_start":"132'7800","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6301,"log_dups_size":0,"ondisk_log_size":6301,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:46:50.507774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26411008,"num_objects":403,"num_object_clones":0,"num_object_copies":1209,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14101,"num_write_kb":464128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":143,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T14:30:33.793334+0000","last_change":"2024-09-13T14:27:12.843038+0000","last_active":"2024-09-13T14:30:33.793334+0000","last_peered":"2024-09-13T14:30:33.793334+0000","last_clean":"2024-09-13T14:30:33.793334+0000","last_became_active":"2024-09-13T14:27:12.842663+0000","last_became_peered":"2024-09-13T14:27:12.842663+0000","last_unstale":"2024-09-13T14:30:33.793334+0000","last_undegraded":"2024-09-13T14:30:33.793334+0000","last_fullsized":"2024-09-13T14:30:33.793334+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:23:38.880811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":851,"reported_epoch":153,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:30:34.782745+0000","last_change":"2024-09-13T14:27:14.029991+0000","last_active":"2024-09-13T14:30:34.782745+0000","last_peered":"2024-09-13T14:30:34.782745+0000","last_clean":"2024-09-13T14:27:08.815347+0000","last_became_active":"2024-09-13T14:27:13.888039+0000","last_became_peered":"2024-09-13T14:27:13.888039+0000","last_unstale":"2024-09-13T14:30:34.782745+0000","last_undegraded":"2024-09-13T14:27:13.826593+0000","last_fullsized":"2024-09-13T14:27:13.824237+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7],"avail_no_missing":["0","2","7"],"object_location_counts":[{"shards":"0,2,7","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":70,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T14:30:33.791305+0000","last_change":"2024-09-13T14:26:15.819032+0000","last_active":"2024-09-13T14:30:33.791305+0000","last_peered":"2024-09-13T14:30:33.791305+0000","last_clean":"2024-09-13T14:30:33.791305+0000","last_became_active":"2024-09-13T14:26:15.818675+0000","last_became_peered":"2024-09-13T14:26:15.818675+0000","last_unstale":"2024-09-13T14:30:33.791305+0000","last_undegraded":"2024-09-13T14:30:33.791305+0000","last_fullsized":"2024-09-13T14:30:33.791305+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:37:13.628256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"153'14306","reported_seq":14354,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T14:30:36.346161+0000","last_change":"2024-09-13T14:26:21.863343+0000","last_active":"2024-09-13T14:30:36.346161+0000","last_peered":"2024-09-13T14:30:36.346161+0000","last_clean":"2024-09-13T14:30:36.346161+0000","last_became_active":"2024-09-13T14:26:21.863057+0000","last_became_peered":"2024-09-13T14:26:21.863057+0000","last_unstale":"2024-09-13T14:30:36.346161+0000","last_undegraded":"2024-09-13T14:30:36.346161+0000","last_fullsized":"2024-09-13T14:30:36.346161+0000","mapping_epoch":116,"log_start":"132'7600","ondisk_log_start":"132'7600","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6706,"log_dups_size":0,"ondisk_log_size":6706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:38:25.794314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28573696,"num_objects":436,"num_object_clones":0,"num_object_copies":1308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14306,"num_write_kb":471744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":77,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T14:30:33.791570+0000","last_change":"2024-09-13T14:27:12.843268+0000","last_active":"2024-09-13T14:30:33.791570+0000","last_peered":"2024-09-13T14:30:33.791570+0000","last_clean":"2024-09-13T14:30:33.791570+0000","last_became_active":"2024-09-13T14:27:12.842867+0000","last_became_peered":"2024-09-13T14:27:12.842867+0000","last_unstale":"2024-09-13T14:30:33.791570+0000","last_undegraded":"2024-09-13T14:30:33.791570+0000","last_fullsized":"2024-09-13T14:30:33.791570+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:03:59.575866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":640,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T14:30:33.785033+0000","last_change":"2024-09-13T14:23:18.038847+0000","last_active":"2024-09-13T14:30:33.785033+0000","last_peered":"2024-09-13T14:30:33.785033+0000","last_clean":"2024-09-13T14:30:33.785033+0000","last_became_active":"2024-09-13T14:23:18.038188+0000","last_became_peered":"2024-09-13T14:23:18.038188+0000","last_unstale":"2024-09-13T14:30:33.785033+0000","last_undegraded":"2024-09-13T14:30:33.785033+0000","last_fullsized":"2024-09-13T14:30:33.785033+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:30:37.213496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":144,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T14:30:34.782524+0000","last_change":"2024-09-13T14:23:18.034427+0000","last_active":"2024-09-13T14:30:34.782524+0000","last_peered":"2024-09-13T14:30:34.782524+0000","last_clean":"2024-09-13T14:30:34.782524+0000","last_became_active":"2024-09-13T14:23:18.034092+0000","last_became_peered":"2024-09-13T14:23:18.034092+0000","last_unstale":"2024-09-13T14:30:34.782524+0000","last_undegraded":"2024-09-13T14:30:34.782524+0000","last_fullsized":"2024-09-13T14:30:34.782524+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:21.755115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"153'14449","reported_seq":14497,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T14:30:37.122373+0000","last_change":"2024-09-13T14:26:21.867237+0000","last_active":"2024-09-13T14:30:37.122373+0000","last_peered":"2024-09-13T14:30:37.122373+0000","last_clean":"2024-09-13T14:30:37.122373+0000","last_became_active":"2024-09-13T14:26:21.866931+0000","last_became_peered":"2024-09-13T14:26:21.866931+0000","last_unstale":"2024-09-13T14:30:37.122373+0000","last_undegraded":"2024-09-13T14:30:37.122373+0000","last_fullsized":"2024-09-13T14:30:37.122373+0000","mapping_epoch":116,"log_start":"132'8000","ondisk_log_start":"132'8000","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6449,"log_dups_size":0,"ondisk_log_size":6449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:52:57.569446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23920640,"num_objects":365,"num_object_clones":0,"num_object_copies":1095,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14449,"num_write_kb":474048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":160,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T14:30:34.829788+0000","last_change":"2024-09-13T14:24:55.290711+0000","last_active":"2024-09-13T14:30:34.829788+0000","last_peered":"2024-09-13T14:30:34.829788+0000","last_clean":"2024-09-13T14:30:34.829788+0000","last_became_active":"2024-09-13T14:24:55.288549+0000","last_became_peered":"2024-09-13T14:24:55.288549+0000","last_unstale":"2024-09-13T14:30:34.829788+0000","last_undegraded":"2024-09-13T14:30:34.829788+0000","last_fullsized":"2024-09-13T14:30:34.829788+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:55:21.763931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":554,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T14:30:34.833050+0000","last_change":"2024-09-13T14:26:18.827876+0000","last_active":"2024-09-13T14:30:34.833050+0000","last_peered":"2024-09-13T14:30:34.833050+0000","last_clean":"2024-09-13T14:30:34.833050+0000","last_became_active":"2024-09-13T14:26:18.827317+0000","last_became_peered":"2024-09-13T14:26:18.827317+0000","last_unstale":"2024-09-13T14:30:34.833050+0000","last_undegraded":"2024-09-13T14:30:34.833050+0000","last_fullsized":"2024-09-13T14:30:34.833050+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:31:48.611286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":157,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T14:30:34.828261+0000","last_change":"2024-09-13T14:24:55.281593+0000","last_active":"2024-09-13T14:30:34.828261+0000","last_peered":"2024-09-13T14:30:34.828261+0000","last_clean":"2024-09-13T14:30:34.828261+0000","last_became_active":"2024-09-13T14:24:55.280805+0000","last_became_peered":"2024-09-13T14:24:55.280805+0000","last_unstale":"2024-09-13T14:30:34.828261+0000","last_undegraded":"2024-09-13T14:30:34.828261+0000","last_fullsized":"2024-09-13T14:30:34.828261+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:14.806713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"153'14599","reported_seq":14687,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T14:30:38.294315+0000","last_change":"2024-09-13T14:26:21.864819+0000","last_active":"2024-09-13T14:30:38.294315+0000","last_peered":"2024-09-13T14:30:38.294315+0000","last_clean":"2024-09-13T14:30:38.294315+0000","last_became_active":"2024-09-13T14:26:21.864511+0000","last_became_peered":"2024-09-13T14:26:21.864511+0000","last_unstale":"2024-09-13T14:30:38.294315+0000","last_undegraded":"2024-09-13T14:30:38.294315+0000","last_fullsized":"2024-09-13T14:30:38.294315+0000","mapping_epoch":116,"log_start":"132'7700","ondisk_log_start":"132'7700","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:26:20.834032+0000","last_clean_scrub_stamp":"2024-09-13T14:26:20.834032+0000","objects_scrubbed":0,"log_size":6899,"log_dups_size":0,"ondisk_log_size":6899,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:11:52.262311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20905984,"num_objects":319,"num_object_clones":0,"num_object_copies":957,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14599,"num_write_kb":477376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":169,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T14:30:34.782944+0000","last_change":"2024-09-13T14:26:15.830309+0000","last_active":"2024-09-13T14:30:34.782944+0000","last_peered":"2024-09-13T14:30:34.782944+0000","last_clean":"2024-09-13T14:30:34.782944+0000","last_became_active":"2024-09-13T14:26:15.823041+0000","last_became_peered":"2024-09-13T14:26:15.823041+0000","last_unstale":"2024-09-13T14:30:34.782944+0000","last_undegraded":"2024-09-13T14:30:34.782944+0000","last_fullsized":"2024-09-13T14:30:34.782944+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:23:08.158576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":584,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T14:30:33.779851+0000","last_change":"2024-09-13T14:27:12.848354+0000","last_active":"2024-09-13T14:30:33.779851+0000","last_peered":"2024-09-13T14:30:33.779851+0000","last_clean":"2024-09-13T14:30:33.779851+0000","last_became_active":"2024-09-13T14:27:12.832182+0000","last_became_peered":"2024-09-13T14:27:12.832182+0000","last_unstale":"2024-09-13T14:30:33.779851+0000","last_undegraded":"2024-09-13T14:30:33.779851+0000","last_fullsized":"2024-09-13T14:30:33.779851+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:46:42.064509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":102,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T14:30:33.790548+0000","last_change":"2024-09-13T14:11:30.256034+0000","last_active":"2024-09-13T14:30:33.790548+0000","last_peered":"2024-09-13T14:30:33.790548+0000","last_clean":"2024-09-13T14:30:33.790548+0000","last_became_active":"2024-09-13T14:11:30.255727+0000","last_became_peered":"2024-09-13T14:11:30.255727+0000","last_unstale":"2024-09-13T14:30:33.790548+0000","last_undegraded":"2024-09-13T14:30:33.790548+0000","last_fullsized":"2024-09-13T14:30:33.790548+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:21:35.817014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":125,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T14:30:33.789753+0000","last_change":"2024-09-13T14:26:15.825390+0000","last_active":"2024-09-13T14:30:33.789753+0000","last_peered":"2024-09-13T14:30:33.789753+0000","last_clean":"2024-09-13T14:30:33.789753+0000","last_became_active":"2024-09-13T14:26:15.825084+0000","last_became_peered":"2024-09-13T14:26:15.825084+0000","last_unstale":"2024-09-13T14:30:33.789753+0000","last_undegraded":"2024-09-13T14:30:33.789753+0000","last_fullsized":"2024-09-13T14:30:33.789753+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:44.680451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":125,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T14:30:33.791783+0000","last_change":"2024-09-13T14:11:29.237013+0000","last_active":"2024-09-13T14:30:33.791783+0000","last_peered":"2024-09-13T14:30:33.791783+0000","last_clean":"2024-09-13T14:30:33.791783+0000","last_became_active":"2024-09-13T14:11:29.236643+0000","last_became_peered":"2024-09-13T14:11:29.236643+0000","last_unstale":"2024-09-13T14:30:33.791783+0000","last_undegraded":"2024-09-13T14:30:33.791783+0000","last_fullsized":"2024-09-13T14:30:33.791783+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:33:58.438693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":99,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T14:30:33.783052+0000","last_change":"2024-09-13T14:27:12.843904+0000","last_active":"2024-09-13T14:30:33.783052+0000","last_peered":"2024-09-13T14:30:33.783052+0000","last_clean":"2024-09-13T14:30:33.783052+0000","last_became_active":"2024-09-13T14:27:12.842566+0000","last_became_peered":"2024-09-13T14:27:12.842566+0000","last_unstale":"2024-09-13T14:30:33.783052+0000","last_undegraded":"2024-09-13T14:30:33.783052+0000","last_fullsized":"2024-09-13T14:30:33.783052+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:11:46.253003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":120,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T14:30:33.784777+0000","last_change":"2024-09-13T14:24:55.287275+0000","last_active":"2024-09-13T14:30:33.784777+0000","last_peered":"2024-09-13T14:30:33.784777+0000","last_clean":"2024-09-13T14:30:33.784777+0000","last_became_active":"2024-09-13T14:24:55.286919+0000","last_became_peered":"2024-09-13T14:24:55.286919+0000","last_unstale":"2024-09-13T14:30:33.784777+0000","last_undegraded":"2024-09-13T14:30:33.784777+0000","last_fullsized":"2024-09-13T14:30:33.784777+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:10.790711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":126,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T14:30:33.804025+0000","last_change":"2024-09-13T14:26:15.819037+0000","last_active":"2024-09-13T14:30:33.804025+0000","last_peered":"2024-09-13T14:30:33.804025+0000","last_clean":"2024-09-13T14:30:33.804025+0000","last_became_active":"2024-09-13T14:26:15.818721+0000","last_became_peered":"2024-09-13T14:26:15.818721+0000","last_unstale":"2024-09-13T14:30:33.804025+0000","last_undegraded":"2024-09-13T14:30:33.804025+0000","last_fullsized":"2024-09-13T14:30:33.804025+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:44:06.419136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":113,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T14:30:34.828197+0000","last_change":"2024-09-13T14:24:55.287509+0000","last_active":"2024-09-13T14:30:34.828197+0000","last_peered":"2024-09-13T14:30:34.828197+0000","last_clean":"2024-09-13T14:30:34.828197+0000","last_became_active":"2024-09-13T14:24:55.287193+0000","last_became_peered":"2024-09-13T14:24:55.287193+0000","last_unstale":"2024-09-13T14:30:34.828197+0000","last_undegraded":"2024-09-13T14:30:34.828197+0000","last_fullsized":"2024-09-13T14:30:34.828197+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:14:10.603307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":154,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T14:30:34.782959+0000","last_change":"2024-09-13T14:24:55.289055+0000","last_active":"2024-09-13T14:30:34.782959+0000","last_peered":"2024-09-13T14:30:34.782959+0000","last_clean":"2024-09-13T14:30:34.782959+0000","last_became_active":"2024-09-13T14:24:55.288762+0000","last_became_peered":"2024-09-13T14:24:55.288762+0000","last_unstale":"2024-09-13T14:30:34.782959+0000","last_undegraded":"2024-09-13T14:30:34.782959+0000","last_fullsized":"2024-09-13T14:30:34.782959+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:37:26.639877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":169,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T14:30:34.828221+0000","last_change":"2024-09-13T14:26:15.824540+0000","last_active":"2024-09-13T14:30:34.828221+0000","last_peered":"2024-09-13T14:30:34.828221+0000","last_clean":"2024-09-13T14:30:34.828221+0000","last_became_active":"2024-09-13T14:26:15.824118+0000","last_became_peered":"2024-09-13T14:26:15.824118+0000","last_unstale":"2024-09-13T14:30:34.828221+0000","last_undegraded":"2024-09-13T14:30:34.828221+0000","last_fullsized":"2024-09-13T14:30:34.828221+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:56.677401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":127,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T14:30:33.779785+0000","last_change":"2024-09-13T14:26:15.823892+0000","last_active":"2024-09-13T14:30:33.779785+0000","last_peered":"2024-09-13T14:30:33.779785+0000","last_clean":"2024-09-13T14:30:33.779785+0000","last_became_active":"2024-09-13T14:26:15.823582+0000","last_became_peered":"2024-09-13T14:26:15.823582+0000","last_unstale":"2024-09-13T14:30:33.779785+0000","last_undegraded":"2024-09-13T14:30:33.779785+0000","last_fullsized":"2024-09-13T14:30:33.779785+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:53:09.918997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":192,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T14:30:34.828352+0000","last_change":"2024-09-13T14:27:12.830642+0000","last_active":"2024-09-13T14:30:34.828352+0000","last_peered":"2024-09-13T14:30:34.828352+0000","last_clean":"2024-09-13T14:30:34.828352+0000","last_became_active":"2024-09-13T14:27:12.830026+0000","last_became_peered":"2024-09-13T14:27:12.830026+0000","last_unstale":"2024-09-13T14:30:34.828352+0000","last_undegraded":"2024-09-13T14:30:34.828352+0000","last_fullsized":"2024-09-13T14:30:34.828352+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:12:31.783552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":155,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T14:30:34.782784+0000","last_change":"2024-09-13T14:24:55.283070+0000","last_active":"2024-09-13T14:30:34.782784+0000","last_peered":"2024-09-13T14:30:34.782784+0000","last_clean":"2024-09-13T14:30:34.782784+0000","last_became_active":"2024-09-13T14:24:55.282591+0000","last_became_peered":"2024-09-13T14:24:55.282591+0000","last_unstale":"2024-09-13T14:30:34.782784+0000","last_undegraded":"2024-09-13T14:30:34.782784+0000","last_fullsized":"2024-09-13T14:30:34.782784+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:35:21.896294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":141,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T14:30:34.828482+0000","last_change":"2024-09-13T14:26:15.823524+0000","last_active":"2024-09-13T14:30:34.828482+0000","last_peered":"2024-09-13T14:30:34.828482+0000","last_clean":"2024-09-13T14:30:34.828482+0000","last_became_active":"2024-09-13T14:26:15.823162+0000","last_became_peered":"2024-09-13T14:26:15.823162+0000","last_unstale":"2024-09-13T14:30:34.828482+0000","last_undegraded":"2024-09-13T14:30:34.828482+0000","last_fullsized":"2024-09-13T14:30:34.828482+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:17:13.664812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":182,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T14:30:34.828857+0000","last_change":"2024-09-13T14:26:15.823703+0000","last_active":"2024-09-13T14:30:34.828857+0000","last_peered":"2024-09-13T14:30:34.828857+0000","last_clean":"2024-09-13T14:30:34.828857+0000","last_became_active":"2024-09-13T14:26:15.823249+0000","last_became_peered":"2024-09-13T14:26:15.823249+0000","last_unstale":"2024-09-13T14:30:34.828857+0000","last_undegraded":"2024-09-13T14:30:34.828857+0000","last_fullsized":"2024-09-13T14:30:34.828857+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:42:40.209150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":150,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T14:30:33.795705+0000","last_change":"2024-09-13T14:27:12.841881+0000","last_active":"2024-09-13T14:30:33.795705+0000","last_peered":"2024-09-13T14:30:33.795705+0000","last_clean":"2024-09-13T14:30:33.795705+0000","last_became_active":"2024-09-13T14:27:12.841247+0000","last_became_peered":"2024-09-13T14:27:12.841247+0000","last_unstale":"2024-09-13T14:30:33.795705+0000","last_undegraded":"2024-09-13T14:30:33.795705+0000","last_fullsized":"2024-09-13T14:30:33.795705+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:59:16.937848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":165,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T14:30:34.828665+0000","last_change":"2024-09-13T14:27:12.838453+0000","last_active":"2024-09-13T14:30:34.828665+0000","last_peered":"2024-09-13T14:30:34.828665+0000","last_clean":"2024-09-13T14:30:34.828665+0000","last_became_active":"2024-09-13T14:27:12.838124+0000","last_became_peered":"2024-09-13T14:27:12.838124+0000","last_unstale":"2024-09-13T14:30:34.828665+0000","last_undegraded":"2024-09-13T14:30:34.828665+0000","last_fullsized":"2024-09-13T14:30:34.828665+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:24.259191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":146,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T14:30:34.782511+0000","last_change":"2024-09-13T14:23:18.035101+0000","last_active":"2024-09-13T14:30:34.782511+0000","last_peered":"2024-09-13T14:30:34.782511+0000","last_clean":"2024-09-13T14:30:34.782511+0000","last_became_active":"2024-09-13T14:23:18.034724+0000","last_became_peered":"2024-09-13T14:23:18.034724+0000","last_unstale":"2024-09-13T14:30:34.782511+0000","last_undegraded":"2024-09-13T14:30:34.782511+0000","last_fullsized":"2024-09-13T14:30:34.782511+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:39.358144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":148,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T14:30:33.789690+0000","last_change":"2024-09-13T14:27:12.834931+0000","last_active":"2024-09-13T14:30:33.789690+0000","last_peered":"2024-09-13T14:30:33.789690+0000","last_clean":"2024-09-13T14:30:33.789690+0000","last_became_active":"2024-09-13T14:27:12.834572+0000","last_became_peered":"2024-09-13T14:27:12.834572+0000","last_unstale":"2024-09-13T14:30:33.789690+0000","last_undegraded":"2024-09-13T14:30:33.789690+0000","last_fullsized":"2024-09-13T14:30:33.789690+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:54:21.155689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":132,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T14:30:33.791611+0000","last_change":"2024-09-13T14:26:15.826283+0000","last_active":"2024-09-13T14:30:33.791611+0000","last_peered":"2024-09-13T14:30:33.791611+0000","last_clean":"2024-09-13T14:30:33.791611+0000","last_became_active":"2024-09-13T14:26:15.820774+0000","last_became_peered":"2024-09-13T14:26:15.820774+0000","last_unstale":"2024-09-13T14:30:33.791611+0000","last_undegraded":"2024-09-13T14:30:33.791611+0000","last_fullsized":"2024-09-13T14:30:33.791611+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:28:39.849420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":130,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T14:30:33.795486+0000","last_change":"2024-09-13T14:24:55.285384+0000","last_active":"2024-09-13T14:30:33.795486+0000","last_peered":"2024-09-13T14:30:33.795486+0000","last_clean":"2024-09-13T14:30:33.795486+0000","last_became_active":"2024-09-13T14:24:55.285081+0000","last_became_peered":"2024-09-13T14:24:55.285081+0000","last_unstale":"2024-09-13T14:30:33.795486+0000","last_undegraded":"2024-09-13T14:30:33.795486+0000","last_fullsized":"2024-09-13T14:30:33.795486+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:47:07.946120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":104,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T14:30:33.793255+0000","last_change":"2024-09-13T14:11:29.240351+0000","last_active":"2024-09-13T14:30:33.793255+0000","last_peered":"2024-09-13T14:30:33.793255+0000","last_clean":"2024-09-13T14:30:33.793255+0000","last_became_active":"2024-09-13T14:11:29.240060+0000","last_became_peered":"2024-09-13T14:11:29.240060+0000","last_unstale":"2024-09-13T14:30:33.793255+0000","last_undegraded":"2024-09-13T14:30:33.793255+0000","last_fullsized":"2024-09-13T14:30:33.793255+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:34:36.126368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":188,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T14:30:34.828634+0000","last_change":"2024-09-13T14:27:12.839943+0000","last_active":"2024-09-13T14:30:34.828634+0000","last_peered":"2024-09-13T14:30:34.828634+0000","last_clean":"2024-09-13T14:30:34.828634+0000","last_became_active":"2024-09-13T14:27:12.839534+0000","last_became_peered":"2024-09-13T14:27:12.839534+0000","last_unstale":"2024-09-13T14:30:34.828634+0000","last_undegraded":"2024-09-13T14:30:34.828634+0000","last_fullsized":"2024-09-13T14:30:34.828634+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:34:56.677019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":176,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T14:30:34.829413+0000","last_change":"2024-09-13T14:27:12.836567+0000","last_active":"2024-09-13T14:30:34.829413+0000","last_peered":"2024-09-13T14:30:34.829413+0000","last_clean":"2024-09-13T14:30:34.829413+0000","last_became_active":"2024-09-13T14:27:12.836271+0000","last_became_peered":"2024-09-13T14:27:12.836271+0000","last_unstale":"2024-09-13T14:30:34.829413+0000","last_undegraded":"2024-09-13T14:30:34.829413+0000","last_fullsized":"2024-09-13T14:30:34.829413+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:25:32.910438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":124,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T14:30:33.790988+0000","last_change":"2024-09-13T14:26:15.824075+0000","last_active":"2024-09-13T14:30:33.790988+0000","last_peered":"2024-09-13T14:30:33.790988+0000","last_clean":"2024-09-13T14:30:33.790988+0000","last_became_active":"2024-09-13T14:26:15.823762+0000","last_became_peered":"2024-09-13T14:26:15.823762+0000","last_unstale":"2024-09-13T14:30:33.790988+0000","last_undegraded":"2024-09-13T14:30:33.790988+0000","last_fullsized":"2024-09-13T14:30:33.790988+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:26:51.088388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":171,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T14:30:34.782876+0000","last_change":"2024-09-13T14:27:12.844201+0000","last_active":"2024-09-13T14:30:34.782876+0000","last_peered":"2024-09-13T14:30:34.782876+0000","last_clean":"2024-09-13T14:30:34.782876+0000","last_became_active":"2024-09-13T14:27:12.834075+0000","last_became_peered":"2024-09-13T14:27:12.834075+0000","last_unstale":"2024-09-13T14:30:34.782876+0000","last_undegraded":"2024-09-13T14:30:34.782876+0000","last_fullsized":"2024-09-13T14:30:34.782876+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:06.657030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":404815896,"num_objects":6178,"num_object_clones":0,"num_object_copies":18534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17494,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6178,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":229654,"num_write_kb":7546561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12827,"num_bytes_recovered":572391424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2553556992,"data_stored":2553544776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108554,"ondisk_log_size":108554,"up":48,"acting":44,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":616,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":17278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18272,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16793600,"data_stored":16777292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":23,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234241,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309460,"kb_used_data":161860,"kb_used_omap":1,"kb_used_meta":147582,"kb_avail":93435692,"statfs":{"total":95995035648,"available":95678148608,"internally_reserved":0,"allocated":165744640,"data_stored":164961761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":151124427},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495877,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":283148,"kb_used_data":179840,"kb_used_omap":1,"kb_used_meta":103294,"kb_avail":93462004,"statfs":{"total":95995035648,"available":95705092096,"internally_reserved":0,"allocated":184156160,"data_stored":183377900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":105773511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757514,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":473600,"kb_used_data":284016,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":93271552,"statfs":{"total":95995035648,"available":95510069248,"internally_reserved":0,"allocated":290832384,"data_stored":290065140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":193984966},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019150,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":479424,"kb_used_data":281784,"kb_used_omap":1,"kb_used_meta":197630,"kb_avail":93265728,"statfs":{"total":95995035648,"available":95504105472,"internally_reserved":0,"allocated":288546816,"data_stored":287775645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":202373574},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435335,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":140592,"kb_used_data":64488,"kb_used_omap":5,"kb_used_meta":76090,"kb_avail":93604560,"statfs":{"total":95995035648,"available":95851069440,"internally_reserved":0,"allocated":66035712,"data_stored":65276398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":77916286},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":446676598861,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":425201762388,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726925914,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":602816,"kb_used_data":350392,"kb_used_omap":7,"kb_used_meta":252408,"kb_avail":93142336,"statfs":{"total":95995035648,"available":95377752064,"internally_reserved":0,"allocated":358801408,"data_stored":358026811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":258466088},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":316018688,"data_stored":316014616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1335033856,"data_stored":1335033856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":62783488,"data_stored":62783488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":271843328,"data_stored":271843328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":257036288,"data_stored":257032216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":162791424,"data_stored":162791424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":148049920,"data_stored":148045848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:30:41.426 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph status --format=json 2024-09-13T14:30:41.453 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:41.495 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:41.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:41.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:41.596 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:30:41.636 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:41.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:41.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:30:41.738 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:30:41.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:41.761 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:41.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:30:41.862 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:30:41.898 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:41.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:41.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:30:42.000 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:30:42.032 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:42.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:42.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:30:42.134 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:30:42.139 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:30:42.165 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:41 smithi102 ceph-mon[110183]: from='client.34962 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:30:42.165 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:41 smithi102 ceph-mon[115269]: from='client.34962 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:30:42.172 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:42.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:42.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:30:42.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:41 smithi195 ceph-mon[104262]: from='client.34962 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:30:42.275 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:42.302 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:42.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:42.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:42.403 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:30:42.437 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:42.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:42.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:30:42.538 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:42.565 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:42.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:42.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:42.667 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:42.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:42.709 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:42.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:42.810 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:42.860 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:42.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:42.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:42.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:42 smithi102 ceph-mon[110183]: pgmap v465: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 391 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 0 B/s wr, 424 op/s; 17506/18618 objects degraded (94.027%); 0 B/s, 76 objects/s recovering 2024-09-13T14:30:42.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:42 smithi102 ceph-mon[115269]: pgmap v465: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 391 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 0 B/s wr, 424 op/s; 17506/18618 objects degraded (94.027%); 0 B/s, 76 objects/s recovering 2024-09-13T14:30:42.962 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:42.985 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:42.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:42.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:43.087 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:30:43.125 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:43.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:43.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:30:43.226 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:43.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:42 smithi195 ceph-mon[104262]: pgmap v465: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 391 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 0 B/s wr, 424 op/s; 17506/18618 objects degraded (94.027%); 0 B/s, 76 objects/s recovering 2024-09-13T14:30:43.250 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:43.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:43.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:43.352 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:43.375 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:43.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:43.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:43.477 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:30:43.520 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:43.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:43.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:30:43.621 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:30:43.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:43.670 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:43.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:30:43.771 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:43.794 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:43.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:43.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:43.896 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:43.925 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:43.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:43.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:44.027 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:44.051 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:44.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:44.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:44.152 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:44.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:43 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 17506/18618 objects degraded (94.027%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:30:44.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:43 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 17506/18618 objects degraded (94.027%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:30:44.186 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:44.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:44.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:44.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:43 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 17506/18618 objects degraded (94.027%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:30:44.288 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:30:44.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:44.318 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:44.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:30:44.419 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:44.443 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:44.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:44.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:44.545 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:44.568 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:44.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:44.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:44.670 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:44.699 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:44.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:44.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:44.800 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:44.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:44.841 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:44.841 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:44.942 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:44.968 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:44.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:44.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:45.063 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:44 smithi102 ceph-mon[115269]: pgmap v466: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 172 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 784 op/s; 17199/8133 objects degraded (211.472%); 0 B/s, 103 objects/s recovering 2024-09-13T14:30:45.064 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:44 smithi102 ceph-mon[110183]: pgmap v466: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 172 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 784 op/s; 17199/8133 objects degraded (211.472%); 0 B/s, 103 objects/s recovering 2024-09-13T14:30:45.070 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:30:45.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:45.108 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:45.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:30:45.201 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:30:45.201 INFO:teuthology.orchestra.run.smithi102.stdout: "size": 20, 2024-09-13T14:30:45.202 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 600, 2024-09-13T14:30:45.202 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [ 2024-09-13T14:30:45.202 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.202 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:88039 5.0 5:0123a9ea:::benchmark_data_smithi102_155989_object88038:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:30:45.202 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:27:22.518037+0000", 2024-09-13T14:30:45.202 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 202.67962599000001, 2024-09-13T14:30:45.202 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.091216610000000004, 2024-09-13T14:30:45.202 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:30:45.202 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:30:45.202 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:30:45.202 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:30:45.202 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:30:45.203 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:30:45.203 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 88039 2024-09-13T14:30:45.203 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.203 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:30:45.203 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.203 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:30:45.203 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:22.518037+0000", 2024-09-13T14:30:45.203 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:30:45.203 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.203 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.203 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:30:45.203 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:22.518037+0000", 2024-09-13T14:30:45.203 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:30:45.204 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.204 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.204 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:30:45.204 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:22.518040+0000", 2024-09-13T14:30:45.204 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.5289999999999998e-06 2024-09-13T14:30:45.204 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.204 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.204 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:30:45.204 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:22.518069+0000", 2024-09-13T14:30:45.204 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.9774999999999999e-05 2024-09-13T14:30:45.204 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.204 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.205 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:30:45.205 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:22.518096+0000", 2024-09-13T14:30:45.205 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.6741999999999999e-05 2024-09-13T14:30:45.205 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.205 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.205 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:30:45.205 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:22.518109+0000", 2024-09-13T14:30:45.205 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2645e-05 2024-09-13T14:30:45.205 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.205 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.205 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:30:45.205 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:22.518207+0000", 2024-09-13T14:30:45.205 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.8436000000000004e-05 2024-09-13T14:30:45.206 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.206 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.206 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:30:45.206 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:22.518382+0000", 2024-09-13T14:30:45.206 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000175069 2024-09-13T14:30:45.206 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.206 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.206 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 0,4", 2024-09-13T14:30:45.206 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:22.518559+0000", 2024-09-13T14:30:45.206 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000177116 2024-09-13T14:30:45.206 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.206 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.206 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:30:45.207 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:22.521136+0000", 2024-09-13T14:30:45.207 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.002576465 2024-09-13T14:30:45.207 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.207 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.207 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:45.207 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:22.522168+0000", 2024-09-13T14:30:45.207 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0010317460000000001 2024-09-13T14:30:45.207 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.207 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.207 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:45.207 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:22.609136+0000", 2024-09-13T14:30:45.208 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.086968663000000002 2024-09-13T14:30:45.208 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.208 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.208 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:30:45.208 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:22.609198+0000", 2024-09-13T14:30:45.208 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.1928999999999996e-05 2024-09-13T14:30:45.208 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.208 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.208 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:30:45.208 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:22.609254+0000", 2024-09-13T14:30:45.208 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.5495000000000003e-05 2024-09-13T14:30:45.209 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:30:45.209 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:30:45.209 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:30:45.209 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.209 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.209 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:90273 5.0 5:009ab727:::benchmark_data_smithi102_155989_object90272:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:30:45.209 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:27:25.259068+0000", 2024-09-13T14:30:45.209 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 199.93859526899999, 2024-09-13T14:30:45.209 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.094445754000000007, 2024-09-13T14:30:45.209 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:30:45.209 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:30:45.209 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:30:45.210 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:30:45.210 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:30:45.210 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:30:45.210 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 90273 2024-09-13T14:30:45.210 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.210 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:30:45.210 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.210 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:30:45.210 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.259068+0000", 2024-09-13T14:30:45.211 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:30:45.211 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.211 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.211 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:30:45.211 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.259068+0000", 2024-09-13T14:30:45.211 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:30:45.211 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.211 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.211 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:30:45.211 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.259070+0000", 2024-09-13T14:30:45.211 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.159e-06 2024-09-13T14:30:45.211 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.212 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.212 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:30:45.212 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.259102+0000", 2024-09-13T14:30:45.212 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.2020000000000002e-05 2024-09-13T14:30:45.212 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.212 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.212 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:30:45.212 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.259132+0000", 2024-09-13T14:30:45.212 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.0165e-05 2024-09-13T14:30:45.212 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.212 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.212 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:30:45.212 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.259142+0000", 2024-09-13T14:30:45.212 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.027e-05 2024-09-13T14:30:45.213 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.213 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.213 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:30:45.213 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.259203+0000", 2024-09-13T14:30:45.213 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.0939e-05 2024-09-13T14:30:45.213 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.213 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.213 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:30:45.213 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.259323+0000", 2024-09-13T14:30:45.213 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000119541 2024-09-13T14:30:45.213 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.213 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.213 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 0,4", 2024-09-13T14:30:45.214 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.259428+0000", 2024-09-13T14:30:45.214 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00010484000000000001 2024-09-13T14:30:45.214 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.214 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.214 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:30:45.214 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.261553+0000", 2024-09-13T14:30:45.214 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0021256360000000002 2024-09-13T14:30:45.214 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.214 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.214 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:45.214 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.263893+0000", 2024-09-13T14:30:45.214 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0023399610000000002 2024-09-13T14:30:45.214 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.214 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.215 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:45.215 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.353162+0000", 2024-09-13T14:30:45.215 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.089269016000000007 2024-09-13T14:30:45.215 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.215 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.215 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:30:45.215 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.353433+0000", 2024-09-13T14:30:45.215 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00027070399999999998 2024-09-13T14:30:45.215 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.215 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.215 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:30:45.215 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.353514+0000", 2024-09-13T14:30:45.215 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.0502999999999998e-05 2024-09-13T14:30:45.215 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:30:45.216 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:30:45.216 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:30:45.216 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.216 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.216 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:90280 5.0 5:0eab510b:::benchmark_data_smithi102_155989_object90279:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:30:45.216 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:27:25.279410+0000", 2024-09-13T14:30:45.216 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 199.91825352699999, 2024-09-13T14:30:45.216 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.075979005000000002, 2024-09-13T14:30:45.216 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:30:45.216 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:30:45.216 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:30:45.216 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:30:45.217 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:45.218 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:30:45.218 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:30:45.218 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 90280 2024-09-13T14:30:45.218 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.218 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:30:45.218 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.218 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:30:45.219 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.279410+0000", 2024-09-13T14:30:45.219 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:30:45.219 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.219 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.219 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:30:45.219 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.279410+0000", 2024-09-13T14:30:45.219 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:30:45.219 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.219 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.219 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:30:45.219 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.279411+0000", 2024-09-13T14:30:45.219 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.339e-06 2024-09-13T14:30:45.219 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.220 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.220 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:30:45.220 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.279451+0000", 2024-09-13T14:30:45.220 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.0389000000000002e-05 2024-09-13T14:30:45.220 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.220 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.220 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:30:45.220 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.279462+0000", 2024-09-13T14:30:45.220 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0496000000000001e-05 2024-09-13T14:30:45.220 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.220 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.220 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:30:45.221 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.279467+0000", 2024-09-13T14:30:45.221 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.9930000000000003e-06 2024-09-13T14:30:45.221 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.221 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.221 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:30:45.221 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.279515+0000", 2024-09-13T14:30:45.221 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.8123e-05 2024-09-13T14:30:45.221 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.221 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.221 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:30:45.221 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.279605+0000", 2024-09-13T14:30:45.221 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.9704999999999996e-05 2024-09-13T14:30:45.222 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.222 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.222 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 0,4", 2024-09-13T14:30:45.222 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.279690+0000", 2024-09-13T14:30:45.222 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.5718999999999997e-05 2024-09-13T14:30:45.222 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.222 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.222 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:30:45.222 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.281623+0000", 2024-09-13T14:30:45.222 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0019325270000000001 2024-09-13T14:30:45.222 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.222 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.222 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:45.223 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.282593+0000", 2024-09-13T14:30:45.223 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00097022700000000005 2024-09-13T14:30:45.223 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.223 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.223 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:45.223 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.355284+0000", 2024-09-13T14:30:45.223 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.072690736000000006 2024-09-13T14:30:45.223 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.223 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.223 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:30:45.223 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.355344+0000", 2024-09-13T14:30:45.224 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.0207000000000001e-05 2024-09-13T14:30:45.224 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.224 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.224 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:30:45.224 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.355389+0000", 2024-09-13T14:30:45.224 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.4543999999999999e-05 2024-09-13T14:30:45.224 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:30:45.224 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:30:45.224 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:30:45.224 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.224 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.224 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:91672 5.0 5:0740083e:::benchmark_data_smithi102_155989_object91671:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:30:45.224 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:27:27.334269+0000", 2024-09-13T14:30:45.225 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 197.863394032, 2024-09-13T14:30:45.225 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.093371884000000002, 2024-09-13T14:30:45.225 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:30:45.225 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:30:45.225 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:30:45.225 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:30:45.225 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:30:45.225 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:30:45.225 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 91672 2024-09-13T14:30:45.225 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.225 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:30:45.225 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.225 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:30:45.226 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:27.334269+0000", 2024-09-13T14:30:45.226 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:30:45.226 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.226 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.226 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:30:45.226 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:27.334269+0000", 2024-09-13T14:30:45.226 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:30:45.226 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.226 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.226 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:30:45.226 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:27.334270+0000", 2024-09-13T14:30:45.226 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1379999999999999e-06 2024-09-13T14:30:45.227 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.227 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.227 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:30:45.227 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:27.334290+0000", 2024-09-13T14:30:45.228 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.9366999999999999e-05 2024-09-13T14:30:45.228 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.228 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.228 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:30:45.228 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:27.334299+0000", 2024-09-13T14:30:45.228 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.3090000000000001e-06 2024-09-13T14:30:45.229 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.229 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.229 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:30:45.229 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:27.334304+0000", 2024-09-13T14:30:45.229 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.1429999999999997e-06 2024-09-13T14:30:45.229 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.229 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.229 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:30:45.229 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:27.334348+0000", 2024-09-13T14:30:45.229 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.3732000000000001e-05 2024-09-13T14:30:45.230 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.230 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.230 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:30:45.230 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:27.334437+0000", 2024-09-13T14:30:45.230 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.9691999999999994e-05 2024-09-13T14:30:45.230 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.230 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.230 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 0,4", 2024-09-13T14:30:45.230 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:27.334523+0000", 2024-09-13T14:30:45.230 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.5858000000000004e-05 2024-09-13T14:30:45.230 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.230 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.230 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:30:45.230 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:27.336604+0000", 2024-09-13T14:30:45.231 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0020810949999999998 2024-09-13T14:30:45.231 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.231 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.231 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:45.231 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:27.338567+0000", 2024-09-13T14:30:45.231 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0019621080000000002 2024-09-13T14:30:45.231 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.231 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.231 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:45.231 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:27.427553+0000", 2024-09-13T14:30:45.231 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.088986304000000002 2024-09-13T14:30:45.231 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.231 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.232 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:30:45.232 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:27.427593+0000", 2024-09-13T14:30:45.232 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.0061000000000002e-05 2024-09-13T14:30:45.232 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.232 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.232 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:30:45.232 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:27.427641+0000", 2024-09-13T14:30:45.232 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.8077000000000001e-05 2024-09-13T14:30:45.232 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:30:45.232 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:30:45.232 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:30:45.232 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.232 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.233 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:91676 5.0 5:05c6b6a0:::benchmark_data_smithi102_155989_object91675:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:30:45.233 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:27:27.341679+0000", 2024-09-13T14:30:45.233 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 197.85598429500001, 2024-09-13T14:30:45.233 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.08609936, 2024-09-13T14:30:45.233 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:30:45.233 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:30:45.233 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:30:45.233 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:30:45.233 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:30:45.233 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:30:45.233 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 91676 2024-09-13T14:30:45.233 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.234 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:30:45.234 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.234 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:30:45.234 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:27.341679+0000", 2024-09-13T14:30:45.234 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:30:45.234 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.234 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.234 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:30:45.234 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:27.341679+0000", 2024-09-13T14:30:45.234 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:30:45.234 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.234 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.234 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:30:45.235 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:27.341681+0000", 2024-09-13T14:30:45.235 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.857e-06 2024-09-13T14:30:45.235 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.235 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.235 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:30:45.235 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:27.341719+0000", 2024-09-13T14:30:45.235 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.8396000000000003e-05 2024-09-13T14:30:45.235 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.235 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.235 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:30:45.235 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:27.341740+0000", 2024-09-13T14:30:45.235 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.1379999999999999e-05 2024-09-13T14:30:45.235 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.236 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.236 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:30:45.236 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:27.341752+0000", 2024-09-13T14:30:45.236 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1800000000000001e-05 2024-09-13T14:30:45.236 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.236 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.236 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:30:45.236 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:27.341872+0000", 2024-09-13T14:30:45.236 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000120203 2024-09-13T14:30:45.236 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.236 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.236 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:30:45.236 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:27.342019+0000", 2024-09-13T14:30:45.237 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000146582 2024-09-13T14:30:45.237 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.237 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.237 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 0,4", 2024-09-13T14:30:45.237 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:27.342097+0000", 2024-09-13T14:30:45.237 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.7731999999999994e-05 2024-09-13T14:30:45.237 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.237 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.237 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:30:45.237 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:27.343600+0000", 2024-09-13T14:30:45.237 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.001502817 2024-09-13T14:30:45.240 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.240 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.240 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:45.240 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:27.344792+0000", 2024-09-13T14:30:45.240 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.001192577 2024-09-13T14:30:45.240 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.240 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.241 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:45.241 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:27.427681+0000", 2024-09-13T14:30:45.241 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.082888584000000001 2024-09-13T14:30:45.241 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.241 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.241 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:30:45.241 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:27.427721+0000", 2024-09-13T14:30:45.241 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.9913999999999998e-05 2024-09-13T14:30:45.241 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.241 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.242 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:30:45.242 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:27.427778+0000", 2024-09-13T14:30:45.242 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.7518000000000001e-05 2024-09-13T14:30:45.242 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:30:45.242 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:30:45.242 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:30:45.242 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.242 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.242 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:107496 5.0 5:093cfad8:::benchmark_data_smithi102_155989_object107495:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:30:45.242 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:27:47.040453+0000", 2024-09-13T14:30:45.242 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 178.157210386, 2024-09-13T14:30:45.242 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.077949883999999997, 2024-09-13T14:30:45.242 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:30:45.243 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:30:45.243 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:30:45.243 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:30:45.243 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:30:45.243 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:30:45.243 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 107496 2024-09-13T14:30:45.243 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.243 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:30:45.243 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.243 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:30:45.243 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:47.040453+0000", 2024-09-13T14:30:45.243 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:30:45.243 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.244 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.244 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:30:45.244 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:47.040453+0000", 2024-09-13T14:30:45.244 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:30:45.244 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.244 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.244 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:30:45.244 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:47.040455+0000", 2024-09-13T14:30:45.244 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.7910000000000001e-06 2024-09-13T14:30:45.244 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.244 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.244 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:30:45.244 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:47.040478+0000", 2024-09-13T14:30:45.245 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.3567e-05 2024-09-13T14:30:45.245 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.245 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.245 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:30:45.245 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:47.040496+0000", 2024-09-13T14:30:45.245 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.7672999999999998e-05 2024-09-13T14:30:45.245 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.245 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.245 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:30:45.245 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:47.040505+0000", 2024-09-13T14:30:45.245 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.7299999999999994e-06 2024-09-13T14:30:45.245 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.246 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.246 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:30:45.246 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:47.040563+0000", 2024-09-13T14:30:45.246 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.8826999999999998e-05 2024-09-13T14:30:45.246 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.246 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.246 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:30:45.246 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:47.040653+0000", 2024-09-13T14:30:45.246 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.9165000000000004e-05 2024-09-13T14:30:45.246 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.246 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.246 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 0,4", 2024-09-13T14:30:45.246 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:47.040752+0000", 2024-09-13T14:30:45.246 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.9573000000000004e-05 2024-09-13T14:30:45.247 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.247 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.247 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:30:45.247 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:47.043430+0000", 2024-09-13T14:30:45.247 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0026774860000000002 2024-09-13T14:30:45.247 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.247 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.247 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:45.247 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:47.043893+0000", 2024-09-13T14:30:45.247 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00046310900000000001 2024-09-13T14:30:45.247 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.247 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.247 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:45.247 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:47.118316+0000", 2024-09-13T14:30:45.248 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.074423182000000004 2024-09-13T14:30:45.248 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.248 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.248 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:30:45.248 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:47.118358+0000", 2024-09-13T14:30:45.248 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.1865e-05 2024-09-13T14:30:45.248 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.248 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.248 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:30:45.248 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:47.118403+0000", 2024-09-13T14:30:45.248 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.4916000000000002e-05 2024-09-13T14:30:45.248 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:30:45.248 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:30:45.249 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:30:45.249 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.249 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.249 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:113028 5.0 5:04e98fdf:::benchmark_data_smithi102_155989_object113027:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:30:45.249 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:27:54.353765+0000", 2024-09-13T14:30:45.249 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 170.84389795000001, 2024-09-13T14:30:45.249 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.075469989000000001, 2024-09-13T14:30:45.249 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:30:45.249 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:30:45.249 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:30:45.249 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:30:45.249 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:30:45.249 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:30:45.250 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 113028 2024-09-13T14:30:45.250 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.250 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:30:45.250 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.250 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:30:45.250 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.353765+0000", 2024-09-13T14:30:45.250 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:30:45.250 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.250 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.250 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:30:45.250 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.353765+0000", 2024-09-13T14:30:45.250 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:30:45.250 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.250 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.251 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:30:45.251 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.353767+0000", 2024-09-13T14:30:45.251 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.7150000000000001e-06 2024-09-13T14:30:45.251 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.251 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.251 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:30:45.251 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.353792+0000", 2024-09-13T14:30:45.251 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.4715999999999998e-05 2024-09-13T14:30:45.251 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.251 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.251 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:30:45.252 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.353815+0000", 2024-09-13T14:30:45.252 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.3737e-05 2024-09-13T14:30:45.252 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.252 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.252 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:30:45.252 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.353824+0000", 2024-09-13T14:30:45.252 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.9279999999999999e-06 2024-09-13T14:30:45.252 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.252 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.252 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:30:45.252 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.353894+0000", 2024-09-13T14:30:45.252 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.9511999999999994e-05 2024-09-13T14:30:45.253 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.253 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.253 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:30:45.253 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.354006+0000", 2024-09-13T14:30:45.253 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000112687 2024-09-13T14:30:45.253 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.253 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.253 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 0,4", 2024-09-13T14:30:45.253 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.354099+0000", 2024-09-13T14:30:45.253 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.2278e-05 2024-09-13T14:30:45.253 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.253 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.253 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:30:45.253 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.356922+0000", 2024-09-13T14:30:45.254 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0028228039999999999 2024-09-13T14:30:45.254 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.254 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.254 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:45.254 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.367920+0000", 2024-09-13T14:30:45.254 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.010998245 2024-09-13T14:30:45.254 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.254 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.254 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:45.254 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.429001+0000", 2024-09-13T14:30:45.255 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.061080896000000003 2024-09-13T14:30:45.255 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.255 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.255 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:30:45.255 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.429173+0000", 2024-09-13T14:30:45.255 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000171927 2024-09-13T14:30:45.255 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.255 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.255 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:30:45.255 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.429235+0000", 2024-09-13T14:30:45.255 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.2544000000000003e-05 2024-09-13T14:30:45.256 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:30:45.256 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:30:45.256 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:30:45.256 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.256 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.256 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:113033 5.0 5:0376efbc:::benchmark_data_smithi102_155989_object113032:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:30:45.256 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:27:54.358672+0000", 2024-09-13T14:30:45.256 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 170.83899097700001, 2024-09-13T14:30:45.256 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.081457275999999995, 2024-09-13T14:30:45.256 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:30:45.256 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:30:45.256 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:30:45.257 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:30:45.257 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:30:45.257 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:30:45.257 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 113033 2024-09-13T14:30:45.257 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.257 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:30:45.257 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.257 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:30:45.257 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.358672+0000", 2024-09-13T14:30:45.257 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:30:45.257 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.257 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.257 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:30:45.258 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.358672+0000", 2024-09-13T14:30:45.258 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:30:45.258 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.258 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.258 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:30:45.258 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.358674+0000", 2024-09-13T14:30:45.258 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3909999999999999e-06 2024-09-13T14:30:45.258 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.259 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.259 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:30:45.259 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.358694+0000", 2024-09-13T14:30:45.259 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.0409999999999999e-05 2024-09-13T14:30:45.259 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.259 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.259 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:30:45.259 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.358704+0000", 2024-09-13T14:30:45.259 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0067e-05 2024-09-13T14:30:45.259 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.259 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.259 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:30:45.259 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.358710+0000", 2024-09-13T14:30:45.259 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.3609999999999997e-06 2024-09-13T14:30:45.260 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.260 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.260 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:30:45.260 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.358756+0000", 2024-09-13T14:30:45.260 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.5698999999999997e-05 2024-09-13T14:30:45.260 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.260 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.260 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:30:45.260 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.358845+0000", 2024-09-13T14:30:45.260 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.8970999999999999e-05 2024-09-13T14:30:45.260 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.260 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.260 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 0,4", 2024-09-13T14:30:45.261 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.358943+0000", 2024-09-13T14:30:45.261 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.7652999999999995e-05 2024-09-13T14:30:45.261 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.261 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.261 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:30:45.261 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.361162+0000", 2024-09-13T14:30:45.261 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.002219019 2024-09-13T14:30:45.261 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.261 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.261 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:45.262 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.368030+0000", 2024-09-13T14:30:45.262 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0068680929999999996 2024-09-13T14:30:45.262 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.262 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.262 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:45.262 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.440016+0000", 2024-09-13T14:30:45.262 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.071985781999999998 2024-09-13T14:30:45.262 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.262 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.262 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:30:45.262 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.440082+0000", 2024-09-13T14:30:45.262 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.6390000000000006e-05 2024-09-13T14:30:45.262 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.262 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.263 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:30:45.263 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.440129+0000", 2024-09-13T14:30:45.263 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.744e-05 2024-09-13T14:30:45.263 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:30:45.263 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:30:45.263 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:30:45.263 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.263 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.263 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:113379 5.0 5:0f306f3b:::benchmark_data_smithi102_155989_object113378:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:30:45.263 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:27:54.854626+0000", 2024-09-13T14:30:45.263 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 170.343037173, 2024-09-13T14:30:45.263 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.076667983999999995, 2024-09-13T14:30:45.263 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:30:45.264 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:30:45.264 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:30:45.264 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:30:45.264 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:30:45.264 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:30:45.264 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 113379 2024-09-13T14:30:45.264 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.264 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:30:45.264 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.264 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:30:45.264 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.854626+0000", 2024-09-13T14:30:45.264 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:30:45.265 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.265 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.265 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:30:45.265 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.854626+0000", 2024-09-13T14:30:45.265 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:30:45.265 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.265 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.265 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:30:45.265 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.854627+0000", 2024-09-13T14:30:45.265 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0499999999999999e-06 2024-09-13T14:30:45.265 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.265 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.265 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:30:45.265 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.854647+0000", 2024-09-13T14:30:45.266 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.9573e-05 2024-09-13T14:30:45.266 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.266 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.266 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:30:45.266 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.854656+0000", 2024-09-13T14:30:45.266 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.9029999999999999e-06 2024-09-13T14:30:45.267 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.267 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.267 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:30:45.267 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.854661+0000", 2024-09-13T14:30:45.267 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.9540000000000003e-06 2024-09-13T14:30:45.267 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.268 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.268 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:30:45.268 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.854707+0000", 2024-09-13T14:30:45.268 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.5442000000000003e-05 2024-09-13T14:30:45.268 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.268 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.269 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:30:45.270 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.854804+0000", 2024-09-13T14:30:45.271 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.6725999999999995e-05 2024-09-13T14:30:45.271 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.271 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.271 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 0,4", 2024-09-13T14:30:45.271 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.854902+0000", 2024-09-13T14:30:45.271 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.878e-05 2024-09-13T14:30:45.271 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.272 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.272 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:30:45.272 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.857716+0000", 2024-09-13T14:30:45.272 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.002813397 2024-09-13T14:30:45.272 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.272 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.273 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:45.273 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.860310+0000", 2024-09-13T14:30:45.273 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0025940059999999998 2024-09-13T14:30:45.273 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.273 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.273 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:45.274 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.931189+0000", 2024-09-13T14:30:45.274 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.070879243999999994 2024-09-13T14:30:45.274 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.274 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.274 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:30:45.275 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.931247+0000", 2024-09-13T14:30:45.275 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.8390000000000002e-05 2024-09-13T14:30:45.275 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.275 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.275 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:30:45.275 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.931294+0000", 2024-09-13T14:30:45.275 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.6519e-05 2024-09-13T14:30:45.275 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:30:45.275 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:30:45.275 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:30:45.276 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.276 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.276 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:116696 5.0 5:0dbc8dea:::benchmark_data_smithi102_155989_object116695:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:30:45.276 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:27:59.395680+0000", 2024-09-13T14:30:45.276 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 165.801983291, 2024-09-13T14:30:45.276 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.103866395, 2024-09-13T14:30:45.276 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:30:45.276 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:30:45.276 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:30:45.276 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:30:45.276 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:30:45.277 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:30:45.277 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 116696 2024-09-13T14:30:45.277 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.277 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:30:45.277 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.277 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:30:45.277 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.395680+0000", 2024-09-13T14:30:45.277 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:30:45.277 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.277 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.277 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:30:45.277 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.395680+0000", 2024-09-13T14:30:45.277 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:30:45.277 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.278 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.278 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:30:45.278 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.395681+0000", 2024-09-13T14:30:45.278 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3060000000000001e-06 2024-09-13T14:30:45.278 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.278 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.278 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:30:45.278 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.395701+0000", 2024-09-13T14:30:45.278 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.0318000000000001e-05 2024-09-13T14:30:45.278 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.278 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.278 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:30:45.278 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.395712+0000", 2024-09-13T14:30:45.279 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0738e-05 2024-09-13T14:30:45.279 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.279 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.279 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:30:45.279 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.395719+0000", 2024-09-13T14:30:45.279 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.55e-06 2024-09-13T14:30:45.279 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.279 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.279 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:30:45.279 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.395768+0000", 2024-09-13T14:30:45.279 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.8968000000000003e-05 2024-09-13T14:30:45.279 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.279 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.279 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:30:45.280 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.395865+0000", 2024-09-13T14:30:45.280 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.6966999999999995e-05 2024-09-13T14:30:45.280 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.280 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.280 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 0,4", 2024-09-13T14:30:45.280 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.395956+0000", 2024-09-13T14:30:45.280 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.1607000000000006e-05 2024-09-13T14:30:45.280 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.280 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.280 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:30:45.280 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.398349+0000", 2024-09-13T14:30:45.280 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.002392539 2024-09-13T14:30:45.280 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.281 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.281 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:45.281 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.414547+0000", 2024-09-13T14:30:45.281 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.016197774000000002 2024-09-13T14:30:45.281 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.281 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.281 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:45.281 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.499417+0000", 2024-09-13T14:30:45.281 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.084870059999999997 2024-09-13T14:30:45.281 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.281 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.281 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:30:45.281 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.499481+0000", 2024-09-13T14:30:45.281 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.4460000000000003e-05 2024-09-13T14:30:45.282 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.282 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.282 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:30:45.282 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.499546+0000", 2024-09-13T14:30:45.282 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.5108000000000001e-05 2024-09-13T14:30:45.282 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:30:45.282 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:30:45.282 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:30:45.282 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.282 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.282 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:150975 5.0 5:0f1c52b9:::benchmark_data_smithi102_155989_object33057:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T14:30:45.282 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:28:46.729122+0000", 2024-09-13T14:30:45.282 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 118.468540981, 2024-09-13T14:30:45.283 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.10350013299999999, 2024-09-13T14:30:45.283 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:30:45.283 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:30:45.283 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:30:45.283 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:30:45.283 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:30:45.283 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:30:45.283 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 150975 2024-09-13T14:30:45.283 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.283 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:30:45.283 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.283 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:30:45.283 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.729122+0000", 2024-09-13T14:30:45.283 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:30:45.284 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.284 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.284 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:30:45.284 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.729122+0000", 2024-09-13T14:30:45.284 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:30:45.284 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.284 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.284 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:30:45.284 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.729123+0000", 2024-09-13T14:30:45.284 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3060000000000001e-06 2024-09-13T14:30:45.284 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.284 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.284 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:30:45.285 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.729129+0000", 2024-09-13T14:30:45.285 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.6149999999999996e-06 2024-09-13T14:30:45.285 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.285 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.285 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:30:45.285 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.729143+0000", 2024-09-13T14:30:45.285 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3679000000000001e-05 2024-09-13T14:30:45.285 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.285 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.285 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:30:45.285 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.729150+0000", 2024-09-13T14:30:45.285 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.2899999999999997e-06 2024-09-13T14:30:45.285 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.285 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.286 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:30:45.286 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.729202+0000", 2024-09-13T14:30:45.286 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.2175000000000001e-05 2024-09-13T14:30:45.286 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.286 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.286 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:30:45.286 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.729307+0000", 2024-09-13T14:30:45.286 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00010478400000000001 2024-09-13T14:30:45.286 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.286 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.286 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 0,4", 2024-09-13T14:30:45.286 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.729410+0000", 2024-09-13T14:30:45.286 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000102797 2024-09-13T14:30:45.286 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.287 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.287 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:30:45.287 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.732167+0000", 2024-09-13T14:30:45.287 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0027575619999999999 2024-09-13T14:30:45.287 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.287 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.287 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:45.287 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.732675+0000", 2024-09-13T14:30:45.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:44 smithi195 ceph-mon[104262]: pgmap v466: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 172 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 784 op/s; 17199/8133 objects degraded (211.472%); 0 B/s, 103 objects/s recovering 2024-09-13T14:30:45.288 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:30:45.288 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:30:45.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:45.290 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000508104 2024-09-13T14:30:45.290 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.290 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.290 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:45.290 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.832383+0000", 2024-09-13T14:30:45.290 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.099707710000000005 2024-09-13T14:30:45.290 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.290 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.290 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:30:45.291 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.832541+0000", 2024-09-13T14:30:45.291 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00015767999999999999 2024-09-13T14:30:45.291 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.291 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.291 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:30:45.291 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.832622+0000", 2024-09-13T14:30:45.291 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.1431e-05 2024-09-13T14:30:45.291 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:30:45.291 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:30:45.291 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:30:45.291 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.291 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.291 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:150992 5.0 5:032770e9:::benchmark_data_smithi102_155989_object33074:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T14:30:45.291 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:28:46.755737+0000", 2024-09-13T14:30:45.292 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 118.441926588, 2024-09-13T14:30:45.292 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.077101473000000004, 2024-09-13T14:30:45.292 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:30:45.292 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:30:45.292 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:30:45.292 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:30:45.292 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:30:45.292 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:30:45.292 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 150992 2024-09-13T14:30:45.292 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.292 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:30:45.292 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.293 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:30:45.293 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.755737+0000", 2024-09-13T14:30:45.293 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:30:45.293 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.293 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.293 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:30:45.293 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.755737+0000", 2024-09-13T14:30:45.293 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:30:45.293 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.293 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.293 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:30:45.293 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.755739+0000", 2024-09-13T14:30:45.294 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.148e-06 2024-09-13T14:30:45.294 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.294 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.294 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:30:45.294 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.755746+0000", 2024-09-13T14:30:45.294 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.029e-06 2024-09-13T14:30:45.294 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.294 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.294 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:30:45.294 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.755762+0000", 2024-09-13T14:30:45.294 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.6313000000000001e-05 2024-09-13T14:30:45.294 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.294 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.294 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:30:45.295 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.755770+0000", 2024-09-13T14:30:45.295 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.7530000000000001e-06 2024-09-13T14:30:45.295 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.295 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.295 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:30:45.295 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.755822+0000", 2024-09-13T14:30:45.295 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.1755000000000002e-05 2024-09-13T14:30:45.295 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.295 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.295 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:30:45.295 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.755925+0000", 2024-09-13T14:30:45.296 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000103058 2024-09-13T14:30:45.296 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.296 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.296 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 0,4", 2024-09-13T14:30:45.296 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.756010+0000", 2024-09-13T14:30:45.296 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.5822000000000006e-05 2024-09-13T14:30:45.296 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.296 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.296 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:30:45.296 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.759010+0000", 2024-09-13T14:30:45.296 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0029997169999999998 2024-09-13T14:30:45.296 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.297 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.297 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:45.297 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.761801+0000", 2024-09-13T14:30:45.297 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.002791174 2024-09-13T14:30:45.297 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.297 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.297 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:45.297 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.832693+0000", 2024-09-13T14:30:45.297 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.070891286999999997 2024-09-13T14:30:45.297 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.297 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.297 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:30:45.298 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.832752+0000", 2024-09-13T14:30:45.298 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.9519999999999999e-05 2024-09-13T14:30:45.298 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.298 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.298 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:30:45.298 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:28:46.832838+0000", 2024-09-13T14:30:45.298 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.5896999999999994e-05 2024-09-13T14:30:45.298 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:30:45.299 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:30:45.299 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:30:45.299 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.299 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.299 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:181292 5.0 5:08a217ad:::benchmark_data_smithi102_155989_object63374:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-13T14:30:45.299 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:29:30.376886+0000", 2024-09-13T14:30:45.299 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 74.820776660000007, 2024-09-13T14:30:45.299 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.079129755999999996, 2024-09-13T14:30:45.299 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:30:45.299 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:30:45.299 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:30:45.299 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:30:45.300 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:30:45.300 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:30:45.300 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 181292 2024-09-13T14:30:45.300 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.300 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:30:45.300 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.300 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:30:45.300 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:29:30.376886+0000", 2024-09-13T14:30:45.300 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:30:45.300 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.300 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.300 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:30:45.300 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:29:30.376886+0000", 2024-09-13T14:30:45.300 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:30:45.301 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.301 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.301 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:30:45.301 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:29:30.376889+0000", 2024-09-13T14:30:45.301 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.548e-06 2024-09-13T14:30:45.301 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.301 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.301 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:30:45.301 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:29:30.376896+0000", 2024-09-13T14:30:45.302 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.3359999999999997e-06 2024-09-13T14:30:45.302 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.302 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.302 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:30:45.302 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:29:30.376920+0000", 2024-09-13T14:30:45.302 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.3283999999999999e-05 2024-09-13T14:30:45.302 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.302 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.302 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:30:45.302 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:29:30.376930+0000", 2024-09-13T14:30:45.302 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0631999999999999e-05 2024-09-13T14:30:45.302 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.302 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.302 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:30:45.303 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:29:30.376996+0000", 2024-09-13T14:30:45.303 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.5456999999999993e-05 2024-09-13T14:30:45.303 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.303 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.303 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:30:45.303 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:29:30.377147+0000", 2024-09-13T14:30:45.303 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000151013 2024-09-13T14:30:45.303 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.303 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.303 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 0,4", 2024-09-13T14:30:45.303 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:29:30.377284+0000", 2024-09-13T14:30:45.303 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000137081 2024-09-13T14:30:45.303 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.303 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.304 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:30:45.304 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:29:30.381200+0000", 2024-09-13T14:30:45.304 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0039159290000000003 2024-09-13T14:30:45.304 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.304 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.304 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:45.304 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:29:30.385080+0000", 2024-09-13T14:30:45.304 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0038801719999999999 2024-09-13T14:30:45.304 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.304 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.304 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:45.304 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:29:30.455944+0000", 2024-09-13T14:30:45.304 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.070863581999999994 2024-09-13T14:30:45.305 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.305 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.305 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:30:45.305 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:29:30.455982+0000", 2024-09-13T14:30:45.305 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.8161000000000003e-05 2024-09-13T14:30:45.305 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.305 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.305 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:30:45.305 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:29:30.456016+0000", 2024-09-13T14:30:45.305 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.4561000000000003e-05 2024-09-13T14:30:45.305 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:30:45.305 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:30:45.305 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:30:45.306 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.306 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.306 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:181304 5.0 5:06659bf3:::benchmark_data_smithi102_155989_object63386:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-13T14:30:45.306 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:29:30.390800+0000", 2024-09-13T14:30:45.306 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 74.806862987000002, 2024-09-13T14:30:45.306 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.074337142999999994, 2024-09-13T14:30:45.306 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:30:45.306 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:30:45.306 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:30:45.306 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:30:45.306 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:30:45.306 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:30:45.306 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 181304 2024-09-13T14:30:45.307 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.307 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:30:45.307 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.307 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:30:45.307 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:29:30.390800+0000", 2024-09-13T14:30:45.307 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:30:45.307 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.307 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.307 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:30:45.307 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:29:30.390800+0000", 2024-09-13T14:30:45.307 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:30:45.307 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.307 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.307 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:30:45.308 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:29:30.390803+0000", 2024-09-13T14:30:45.308 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.5110000000000002e-06 2024-09-13T14:30:45.308 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.308 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.308 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:30:45.308 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:29:30.390810+0000", 2024-09-13T14:30:45.308 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.4590000000000001e-06 2024-09-13T14:30:45.308 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.309 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.309 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:30:45.309 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:29:30.390828+0000", 2024-09-13T14:30:45.309 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.8328000000000001e-05 2024-09-13T14:30:45.309 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.309 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.309 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:30:45.310 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:29:30.390836+0000", 2024-09-13T14:30:45.310 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.1609999999999997e-06 2024-09-13T14:30:45.310 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.310 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.310 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:30:45.310 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:29:30.390905+0000", 2024-09-13T14:30:45.310 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.9419999999999996e-05 2024-09-13T14:30:45.310 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.311 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.311 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:30:45.311 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:29:30.391039+0000", 2024-09-13T14:30:45.311 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00013423499999999999 2024-09-13T14:30:45.311 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.311 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.311 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 0,4", 2024-09-13T14:30:45.311 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:29:30.391176+0000", 2024-09-13T14:30:45.312 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00013689 2024-09-13T14:30:45.312 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.312 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.312 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:45.312 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:29:30.394725+0000", 2024-09-13T14:30:45.312 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.003549118 2024-09-13T14:30:45.312 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.312 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.313 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:30:45.313 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:29:30.395167+0000", 2024-09-13T14:30:45.313 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000441734 2024-09-13T14:30:45.313 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.313 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.313 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:45.313 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:29:30.465068+0000", 2024-09-13T14:30:45.314 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.069900639000000001 2024-09-13T14:30:45.314 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.314 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.314 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:30:45.314 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:29:30.465104+0000", 2024-09-13T14:30:45.314 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.6584000000000001e-05 2024-09-13T14:30:45.314 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.314 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.315 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:30:45.315 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:29:30.465137+0000", 2024-09-13T14:30:45.315 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.3064000000000001e-05 2024-09-13T14:30:45.315 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:30:45.315 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:30:45.315 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:30:45.315 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.315 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.315 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:214306 5.0 5:09628cd8:::benchmark_data_smithi102_155989_object96388:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-09-13T14:30:45.316 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:30:16.052100+0000", 2024-09-13T14:30:45.316 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 29.145563468999999, 2024-09-13T14:30:45.316 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.075151331000000002, 2024-09-13T14:30:45.316 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:30:45.316 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:30:45.316 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:30:45.316 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:30:45.317 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:30:45.317 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:30:45.317 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 214306 2024-09-13T14:30:45.317 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.317 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:30:45.317 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.317 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:30:45.317 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.052100+0000", 2024-09-13T14:30:45.317 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:30:45.318 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.318 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.318 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:30:45.318 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.052100+0000", 2024-09-13T14:30:45.318 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:30:45.318 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.318 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.318 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:30:45.319 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.052101+0000", 2024-09-13T14:30:45.319 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3e-06 2024-09-13T14:30:45.319 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.319 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.319 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:30:45.319 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.052106+0000", 2024-09-13T14:30:45.319 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.724e-06 2024-09-13T14:30:45.319 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.320 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.320 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:30:45.320 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.052116+0000", 2024-09-13T14:30:45.320 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.8900000000000002e-06 2024-09-13T14:30:45.320 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.320 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.320 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:30:45.320 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.052121+0000", 2024-09-13T14:30:45.321 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.0069999999999997e-06 2024-09-13T14:30:45.321 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.321 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.321 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:30:45.321 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.052162+0000", 2024-09-13T14:30:45.321 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.1661e-05 2024-09-13T14:30:45.321 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.321 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.322 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:30:45.322 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.052242+0000", 2024-09-13T14:30:45.322 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.9566000000000004e-05 2024-09-13T14:30:45.322 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.322 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.322 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 0,4", 2024-09-13T14:30:45.322 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.052315+0000", 2024-09-13T14:30:45.322 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.3090999999999995e-05 2024-09-13T14:30:45.323 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.323 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.323 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:30:45.323 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.054715+0000", 2024-09-13T14:30:45.323 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.002400213 2024-09-13T14:30:45.323 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.323 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.324 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:45.324 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.058311+0000", 2024-09-13T14:30:45.324 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0035962429999999998 2024-09-13T14:30:45.324 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.324 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.324 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:45.324 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.127147+0000", 2024-09-13T14:30:45.324 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.068835239000000006 2024-09-13T14:30:45.325 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.325 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.325 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:30:45.325 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.127200+0000", 2024-09-13T14:30:45.325 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.3817000000000003e-05 2024-09-13T14:30:45.325 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.325 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.325 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:30:45.326 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.127251+0000", 2024-09-13T14:30:45.326 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.058e-05 2024-09-13T14:30:45.326 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:30:45.326 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:30:45.326 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:30:45.326 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.326 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.326 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:214423 5.0 5:07caf530:::benchmark_data_smithi102_155989_object96505:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-09-13T14:30:45.327 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:30:16.263923+0000", 2024-09-13T14:30:45.327 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 28.933740047000001, 2024-09-13T14:30:45.327 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.079656455000000001, 2024-09-13T14:30:45.327 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:30:45.327 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:30:45.328 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:30:45.328 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:30:45.328 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:30:45.328 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:30:45.328 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 214423 2024-09-13T14:30:45.328 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.329 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:30:45.329 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.329 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:30:45.329 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.263923+0000", 2024-09-13T14:30:45.329 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:30:45.329 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.329 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.330 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:30:45.330 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.263923+0000", 2024-09-13T14:30:45.330 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:30:45.330 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.330 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.330 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:30:45.331 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.263925+0000", 2024-09-13T14:30:45.331 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.5069999999999999e-06 2024-09-13T14:30:45.331 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.331 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.331 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:30:45.331 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.263931+0000", 2024-09-13T14:30:45.331 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.2999999999999998e-06 2024-09-13T14:30:45.331 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.332 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.332 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:30:45.332 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.263945+0000", 2024-09-13T14:30:45.332 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3750999999999999e-05 2024-09-13T14:30:45.332 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.332 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.332 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:30:45.332 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.263951+0000", 2024-09-13T14:30:45.333 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.1460000000000001e-06 2024-09-13T14:30:45.333 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.333 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.333 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:30:45.333 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.264010+0000", 2024-09-13T14:30:45.333 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.9392999999999999e-05 2024-09-13T14:30:45.333 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.333 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.334 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:30:45.334 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.264139+0000", 2024-09-13T14:30:45.334 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000128415 2024-09-13T14:30:45.334 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.334 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.334 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 0,4", 2024-09-13T14:30:45.334 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.264251+0000", 2024-09-13T14:30:45.334 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000112126 2024-09-13T14:30:45.335 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.335 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.335 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:30:45.335 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.267277+0000", 2024-09-13T14:30:45.335 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.003026656 2024-09-13T14:30:45.335 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.335 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.335 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:45.336 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.271505+0000", 2024-09-13T14:30:45.336 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00422727 2024-09-13T14:30:45.336 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.336 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.336 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:45.336 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.343514+0000", 2024-09-13T14:30:45.336 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.072008988999999995 2024-09-13T14:30:45.336 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.337 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.337 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:30:45.337 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.343548+0000", 2024-09-13T14:30:45.337 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.4588999999999997e-05 2024-09-13T14:30:45.337 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.337 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.337 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:30:45.338 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.343580+0000", 2024-09-13T14:30:45.338 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.1313000000000003e-05 2024-09-13T14:30:45.338 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:30:45.338 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:30:45.338 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:30:45.338 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.338 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.338 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:214431 5.0 5:02682357:::benchmark_data_smithi102_155989_object96513:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-09-13T14:30:45.339 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:30:16.268153+0000", 2024-09-13T14:30:45.339 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 28.929510178000001, 2024-09-13T14:30:45.339 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.078861790000000001, 2024-09-13T14:30:45.339 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:30:45.339 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:30:45.339 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:30:45.339 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:30:45.339 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:30:45.340 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:30:45.340 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 214431 2024-09-13T14:30:45.340 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.340 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:30:45.340 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.340 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:30:45.340 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.268153+0000", 2024-09-13T14:30:45.340 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:30:45.341 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.341 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.341 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:30:45.341 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.268153+0000", 2024-09-13T14:30:45.341 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:30:45.341 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.341 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.342 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:30:45.342 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.268154+0000", 2024-09-13T14:30:45.342 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.449e-06 2024-09-13T14:30:45.342 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.342 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.342 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:30:45.342 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.268160+0000", 2024-09-13T14:30:45.342 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.5160000000000002e-06 2024-09-13T14:30:45.343 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.343 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.343 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:30:45.343 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.268175+0000", 2024-09-13T14:30:45.343 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4842e-05 2024-09-13T14:30:45.343 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.343 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.343 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:30:45.344 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.268184+0000", 2024-09-13T14:30:45.344 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.8990000000000007e-06 2024-09-13T14:30:45.344 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.344 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.344 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:30:45.344 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.268244+0000", 2024-09-13T14:30:45.344 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.0559000000000001e-05 2024-09-13T14:30:45.344 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.345 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.345 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:30:45.345 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.268330+0000", 2024-09-13T14:30:45.345 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.5512000000000004e-05 2024-09-13T14:30:45.345 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.345 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.345 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 0,4", 2024-09-13T14:30:45.345 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.268407+0000", 2024-09-13T14:30:45.346 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.7262999999999995e-05 2024-09-13T14:30:45.346 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.346 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.346 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:45.346 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.271554+0000", 2024-09-13T14:30:45.346 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00314671 2024-09-13T14:30:45.346 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.346 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.346 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:30:45.347 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.271567+0000", 2024-09-13T14:30:45.347 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3584000000000001e-05 2024-09-13T14:30:45.347 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.347 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.347 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:45.347 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.346935+0000", 2024-09-13T14:30:45.347 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.075367892000000006 2024-09-13T14:30:45.347 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.348 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.348 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:30:45.348 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.346987+0000", 2024-09-13T14:30:45.348 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.2209000000000001e-05 2024-09-13T14:30:45.348 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.348 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.348 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:30:45.348 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.347015+0000", 2024-09-13T14:30:45.349 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.7355000000000002e-05 2024-09-13T14:30:45.349 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:30:45.349 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:30:45.349 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:30:45.349 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.349 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.349 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:214618 5.0 5:09ce4356:::benchmark_data_smithi102_155989_object96700:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-09-13T14:30:45.349 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:30:16.527433+0000", 2024-09-13T14:30:45.350 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 28.670230389, 2024-09-13T14:30:45.350 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.121705457, 2024-09-13T14:30:45.350 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:30:45.350 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:30:45.350 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:30:45.350 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:30:45.350 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:30:45.350 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:30:45.351 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 214618 2024-09-13T14:30:45.351 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.351 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:30:45.351 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.351 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:30:45.351 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.527433+0000", 2024-09-13T14:30:45.351 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:30:45.351 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.351 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.352 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:30:45.352 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.527433+0000", 2024-09-13T14:30:45.352 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:30:45.352 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.352 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.352 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:30:45.352 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.527435+0000", 2024-09-13T14:30:45.352 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.1629999999999999e-06 2024-09-13T14:30:45.353 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.353 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.353 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:30:45.353 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.527444+0000", 2024-09-13T14:30:45.353 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.5909999999999996e-06 2024-09-13T14:30:45.353 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.353 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.354 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:30:45.354 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.527464+0000", 2024-09-13T14:30:45.354 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.0809e-05 2024-09-13T14:30:45.354 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.354 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.354 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:30:45.354 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.527474+0000", 2024-09-13T14:30:45.354 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.3300000000000005e-06 2024-09-13T14:30:45.354 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.355 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.355 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:30:45.355 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.527544+0000", 2024-09-13T14:30:45.355 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.0290999999999995e-05 2024-09-13T14:30:45.355 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.355 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.355 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:30:45.355 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.527657+0000", 2024-09-13T14:30:45.356 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000113307 2024-09-13T14:30:45.356 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.356 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.356 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 0,4", 2024-09-13T14:30:45.356 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.527772+0000", 2024-09-13T14:30:45.356 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00011453199999999999 2024-09-13T14:30:45.356 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.357 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.357 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:30:45.357 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.532754+0000", 2024-09-13T14:30:45.357 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0049820139999999999 2024-09-13T14:30:45.357 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.357 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.357 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:45.357 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.539880+0000", 2024-09-13T14:30:45.358 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0071266810000000002 2024-09-13T14:30:45.358 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.358 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.358 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:45.358 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.649026+0000", 2024-09-13T14:30:45.358 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.109145846 2024-09-13T14:30:45.358 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.359 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.359 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:30:45.359 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.649087+0000", 2024-09-13T14:30:45.359 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.0949999999999998e-05 2024-09-13T14:30:45.359 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.359 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.359 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:30:45.360 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.649138+0000", 2024-09-13T14:30:45.360 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.0943000000000003e-05 2024-09-13T14:30:45.360 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:30:45.360 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:30:45.360 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:30:45.360 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.360 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.361 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:214619 5.0 5:04dd9390:::benchmark_data_smithi102_155989_object96701:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-09-13T14:30:45.361 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:30:16.528076+0000", 2024-09-13T14:30:45.361 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 28.669587168, 2024-09-13T14:30:45.361 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.12119305399999999, 2024-09-13T14:30:45.361 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:30:45.361 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:30:45.361 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:30:45.361 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:30:45.362 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:30:45.362 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:30:45.362 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 214619 2024-09-13T14:30:45.362 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.362 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:30:45.362 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.362 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:30:45.362 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.528076+0000", 2024-09-13T14:30:45.363 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:30:45.363 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.363 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.363 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:30:45.363 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.528076+0000", 2024-09-13T14:30:45.363 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:30:45.363 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.364 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.364 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:30:45.364 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.528077+0000", 2024-09-13T14:30:45.364 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.4900000000000004e-07 2024-09-13T14:30:45.364 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.364 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.364 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:30:45.365 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.528082+0000", 2024-09-13T14:30:45.365 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.5920000000000002e-06 2024-09-13T14:30:45.365 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.365 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.365 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:30:45.365 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.528095+0000", 2024-09-13T14:30:45.365 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3111e-05 2024-09-13T14:30:45.365 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.366 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.366 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:30:45.366 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.528102+0000", 2024-09-13T14:30:45.366 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.2910000000000002e-06 2024-09-13T14:30:45.366 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.366 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.366 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:30:45.366 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.530653+0000", 2024-09-13T14:30:45.367 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.002551233 2024-09-13T14:30:45.367 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.367 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.367 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:30:45.367 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.530734+0000", 2024-09-13T14:30:45.367 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.0493000000000003e-05 2024-09-13T14:30:45.367 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.367 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.368 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 0,4", 2024-09-13T14:30:45.368 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.530808+0000", 2024-09-13T14:30:45.368 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.4238000000000003e-05 2024-09-13T14:30:45.368 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.368 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.368 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:30:45.368 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.532868+0000", 2024-09-13T14:30:45.369 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0020600699999999998 2024-09-13T14:30:45.369 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.369 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.369 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:45.369 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.539923+0000", 2024-09-13T14:30:45.369 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0070546990000000002 2024-09-13T14:30:45.369 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.370 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.370 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:45.370 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.649189+0000", 2024-09-13T14:30:45.370 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.109266555 2024-09-13T14:30:45.370 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.370 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.370 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:30:45.370 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.649227+0000", 2024-09-13T14:30:45.371 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.7327999999999999e-05 2024-09-13T14:30:45.371 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.371 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.371 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:30:45.371 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.649269+0000", 2024-09-13T14:30:45.371 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.2494999999999999e-05 2024-09-13T14:30:45.371 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:30:45.371 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:30:45.372 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:30:45.372 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.372 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.372 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:230317 5.0 5:0e3fced2:::benchmark_data_smithi102_155989_object112399:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-09-13T14:30:45.372 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:30:37.806873+0000", 2024-09-13T14:30:45.372 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 7.3907902239999999, 2024-09-13T14:30:45.372 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.073046994000000004, 2024-09-13T14:30:45.373 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:30:45.373 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:30:45.373 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:30:45.373 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:30:45.373 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:30:45.373 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:30:45.373 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 230317 2024-09-13T14:30:45.374 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.374 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:30:45.374 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.374 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:30:45.374 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:37.806873+0000", 2024-09-13T14:30:45.374 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:30:45.374 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.374 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.375 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:30:45.375 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:37.806873+0000", 2024-09-13T14:30:45.375 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:30:45.375 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.375 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.375 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:30:45.375 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:37.806874+0000", 2024-09-13T14:30:45.375 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.176e-06 2024-09-13T14:30:45.376 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.376 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.376 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:30:45.376 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:37.806880+0000", 2024-09-13T14:30:45.376 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.5820000000000001e-06 2024-09-13T14:30:45.376 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.376 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.377 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:30:45.377 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:37.806892+0000", 2024-09-13T14:30:45.377 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2812e-05 2024-09-13T14:30:45.377 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.377 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.377 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:30:45.377 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:37.806898+0000", 2024-09-13T14:30:45.377 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.0159999999999997e-06 2024-09-13T14:30:45.378 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.378 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.378 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:30:45.378 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:37.806954+0000", 2024-09-13T14:30:45.378 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.5469e-05 2024-09-13T14:30:45.378 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.378 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.379 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:30:45.379 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:37.807066+0000", 2024-09-13T14:30:45.379 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000112391 2024-09-13T14:30:45.379 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.379 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.379 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 0,4", 2024-09-13T14:30:45.379 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:37.807186+0000", 2024-09-13T14:30:45.380 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00011925 2024-09-13T14:30:45.380 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.380 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.380 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:45.380 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:37.810533+0000", 2024-09-13T14:30:45.380 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0033471410000000001 2024-09-13T14:30:45.380 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.380 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.381 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:30:45.381 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:37.810580+0000", 2024-09-13T14:30:45.381 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.7018000000000003e-05 2024-09-13T14:30:45.381 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.381 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.381 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:45.381 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:37.879813+0000", 2024-09-13T14:30:45.382 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.069233164999999999 2024-09-13T14:30:45.382 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.382 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.382 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:30:45.382 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:37.879877+0000", 2024-09-13T14:30:45.382 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.4110999999999997e-05 2024-09-13T14:30:45.382 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:30:45.382 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:30:45.383 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:30:45.383 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:37.879920+0000", 2024-09-13T14:30:45.383 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.2862999999999999e-05 2024-09-13T14:30:45.383 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:30:45.383 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:30:45.383 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:30:45.383 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:30:45.383 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:30:45.383 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:30:45.383 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:45.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:45.385 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Removed 117915 objects 2024-09-13T14:30:45.385 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Clean up completed and total clean up time :164.189 2024-09-13T14:30:45.484 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:45.506 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:45.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:45.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:45.607 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:45.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:45.630 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:45.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:45.731 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:45.752 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:45.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:45.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:45.854 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:45.876 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:45.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:45.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:45.949 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:30:45.949 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:30:45.949 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:30:45.949 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:30:45.950 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:30:45.951 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:30:45.970 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:30:45.970 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:30:45.979 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:45.988 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:30:45.988 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:30:46.001 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:46.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:46.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:46.009 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:30:46.009 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:30:46.029 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:30:46.030 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:30:46.045 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:30:46.046 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:30:46.103 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:46.125 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:46.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:46.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:46.227 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:46.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:46.248 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:46.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:46.349 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:46.370 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:46.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:46.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:46.472 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:46.494 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:46.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:46.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:46.596 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:46.618 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:46.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:46.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:46.720 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:30:46.742 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:46.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:46.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:30:46.844 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:30:46.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:46.869 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:46.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:30:46.875 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:30:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:30:46] "GET /metrics HTTP/1.1" 200 36519 "" "Prometheus/2.51.0" 2024-09-13T14:30:46.970 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:30:46.993 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:46.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:46.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:30:47.094 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:47.116 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:47.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:47.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:47.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:46 smithi102 ceph-mon[110183]: pgmap v467: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 158 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 691 op/s; 17199/7467 objects degraded (230.333%); 0 B/s, 88 objects/s recovering 2024-09-13T14:30:47.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:46 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/766347877' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:30:47.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:46 smithi102 ceph-mon[115269]: pgmap v467: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 158 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 691 op/s; 17199/7467 objects degraded (230.333%); 0 B/s, 88 objects/s recovering 2024-09-13T14:30:47.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:46 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/766347877' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:30:47.218 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:30:47.240 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:47.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:47.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:30:47.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:46 smithi195 ceph-mon[104262]: pgmap v467: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 158 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 691 op/s; 17199/7467 objects degraded (230.333%); 0 B/s, 88 objects/s recovering 2024-09-13T14:30:47.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:46 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/766347877' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:30:47.326 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-09-13T14:30:47.326 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-09-13T14:30:47.342 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:47.391 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:47.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:47.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:47.493 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:30:47.515 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:47.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:47.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:30:47.617 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:47.639 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:47.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:47.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:47.740 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:47.762 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:47.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:47.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:47.864 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:30:47.887 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:47.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:47.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:30:47.988 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:48.009 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:48.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:48.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:48.111 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:30:48.135 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:48.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:48.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:30:48.236 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:48.259 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:48.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:48.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:48.361 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:48.383 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:48.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:48.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:48.485 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:48.506 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:48.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:48.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:48.608 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:48.630 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:48.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:48.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:48.731 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:48.753 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:48.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:48.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:48.836 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T14:30:48.855 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:48.876 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:48.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:48.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:48.978 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:49.003 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:49.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:49.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:49.065 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:48 smithi102 ceph-mon[110183]: pgmap v468: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 11 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 627 op/s; 16518/375 objects degraded (4404.800%); 0 B/s, 98 objects/s recovering 2024-09-13T14:30:49.066 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:48 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 16518/375 objects degraded (4404.800%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:30:49.066 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:48 smithi102 ceph-mon[115269]: pgmap v468: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 11 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 627 op/s; 16518/375 objects degraded (4404.800%); 0 B/s, 98 objects/s recovering 2024-09-13T14:30:49.066 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:48 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 16518/375 objects degraded (4404.800%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:30:49.073 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:30:49.082 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:30:49.105 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:49.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:49.128 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:49.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:49.205 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:48 smithi195 ceph-mon[104262]: pgmap v468: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 11 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 627 op/s; 16518/375 objects degraded (4404.800%); 0 B/s, 98 objects/s recovering 2024-09-13T14:30:49.206 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:48 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 16518/375 objects degraded (4404.800%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:30:49.229 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:30:49.251 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:49.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:49.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:30:49.353 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:49.375 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:49.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:49.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:49.477 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:49.500 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:49.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:49.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:49.602 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:49.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:49.624 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:49.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:49.725 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:49.745 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:49.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:49.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:49.847 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:30:49.869 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:49.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:49.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:30:49.971 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:49.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:49.995 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:49.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:50.096 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:50.117 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:50.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:50.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:50.220 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:50.241 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:50.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:50.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:50.343 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:50.364 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:50.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:50.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:50.466 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:50.487 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:50.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:50.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:50.589 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:50.611 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:50.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:50.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:50.713 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:50.734 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:50.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:50.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:50.783 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:30:50.784 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:30:50.836 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:50.859 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:50.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:50.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:50.961 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:50.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:50.984 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:50.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:51.027 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:50 smithi102 ceph-mon[110183]: pgmap v469: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 MiB data, 1023 MiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 590 op/s; 16518/84 objects degraded (19664.286%); 0 B/s, 94 objects/s recovering 2024-09-13T14:30:51.028 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:50 smithi102 ceph-mon[115269]: pgmap v469: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 MiB data, 1023 MiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 590 op/s; 16518/84 objects degraded (19664.286%); 0 B/s, 94 objects/s recovering 2024-09-13T14:30:51.086 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:30:51.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:51.110 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:51.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:30:51.211 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:51.233 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:51.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:51.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:51.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:50 smithi195 ceph-mon[104262]: pgmap v469: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 MiB data, 1023 MiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 590 op/s; 16518/84 objects degraded (19664.286%); 0 B/s, 94 objects/s recovering 2024-09-13T14:30:51.335 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:51.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:51.362 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:51.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:51.378 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:30:51.378 INFO:teuthology.orchestra.run.smithi102.stdout:{"fsid":"08cee502-71d9-11ef-bceb-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 16518/84 objects degraded (19664.286%), 5 pgs degraded, 5 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":674,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":154,"num_osds":8,"num_up_osds":8,"osd_up_since":1726237494,"num_in_osds":6,"osd_in_since":1726237631,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":84},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":2}],"num_pgs":89,"num_pools":5,"num_objects":28,"data_bytes":4790245,"bytes_used":1072750592,"bytes_avail":574897463296,"bytes_total":575970213888,"degraded_objects":16518,"degraded_total":84,"degraded_ratio":196.64285714285714,"recovering_objects_per_sec":94,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":988,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":590},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi102.njetxr","status":"up:active","gid":24521}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.102:8443/","prometheus":"http://172.21.15.102:9283/"}},"servicemap":{"epoch":23,"modified":"2024-09-13T14:25:55.234425+0000","services":{}},"progress_events":{}} 2024-09-13T14:30:51.463 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:51.484 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:51.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:51.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:51.587 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:51.612 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:51.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:51.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:51.713 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:51.736 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:51.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:51.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:51.838 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:51.861 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:51.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:51.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:51.889 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is set 2024-09-13T14:30:51.963 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:30:51.988 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:51.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:51.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:30:51.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:51 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/894075918' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:30:51.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:51 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2032756227' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:30:51.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:51 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:30:52.091 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:52.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:52.113 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:52.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:52.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:51 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/894075918' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:30:52.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:51 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2032756227' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:30:52.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:51 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:30:52.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:51 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/894075918' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:30:52.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:51 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2032756227' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:30:52.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:51 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:30:52.214 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:52.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:52.236 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:52.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:52.337 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:52.358 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:52.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:52.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:52.460 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:52.485 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:52.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:52.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:52.587 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:52.608 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:52.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:52.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:52.710 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:52.737 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:52.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:52.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:52.838 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:52.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:52.862 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:52.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:52.963 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:52.985 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:52.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:52.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:52.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:52 smithi195 ceph-mon[104262]: pgmap v470: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 MiB data, 1023 MiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 514 op/s; 16518/84 objects degraded (19664.286%); 0 B/s, 82 objects/s recovering 2024-09-13T14:30:52.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:52 smithi195 ceph-mon[104262]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:30:52.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:52 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:30:52.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:52 smithi195 ceph-mon[104262]: osdmap e155: 8 total, 8 up, 6 in 2024-09-13T14:30:53.009 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:30:53.009 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:30:53.087 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:53.111 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:53.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:53.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:53.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:52 smithi102 ceph-mon[110183]: pgmap v470: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 MiB data, 1023 MiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 514 op/s; 16518/84 objects degraded (19664.286%); 0 B/s, 82 objects/s recovering 2024-09-13T14:30:53.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:52 smithi102 ceph-mon[110183]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:30:53.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:52 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:30:53.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:52 smithi102 ceph-mon[110183]: osdmap e155: 8 total, 8 up, 6 in 2024-09-13T14:30:53.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:52 smithi102 ceph-mon[115269]: pgmap v470: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 MiB data, 1023 MiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 514 op/s; 16518/84 objects degraded (19664.286%); 0 B/s, 82 objects/s recovering 2024-09-13T14:30:53.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:52 smithi102 ceph-mon[115269]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:30:53.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:52 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:30:53.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:52 smithi102 ceph-mon[115269]: osdmap e155: 8 total, 8 up, 6 in 2024-09-13T14:30:53.213 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:30:53.237 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:53.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:53.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:30:53.339 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:53.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:53.362 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:53.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:53.463 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:53.484 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:53.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:53.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:53.586 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:53.611 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:53.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:53.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:53.713 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:53.734 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:53.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:53.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:53.836 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:53.839 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T14:30:53.858 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:53.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:53.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:53.960 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:53.982 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:53.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:53.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:54.084 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:54.089 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:30:54.095 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:53 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 16518/84 objects degraded (19664.286%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:30:54.095 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:53 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 16518/84 objects degraded (19664.286%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:30:54.109 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:54.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:54.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:54.211 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:54.213 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:53 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 16518/84 objects degraded (19664.286%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:30:54.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:54.234 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:54.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:54.335 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:54.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:54.357 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:54.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:54.458 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:54.480 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:54.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:54.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:54.534 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-13T14:30:54.535 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:30:54.582 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:30:54.605 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:54.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:54.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:30:54.707 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:54.729 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:54.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:54.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:54.831 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:54.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:54.854 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:54.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:54.955 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:54.978 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:54.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:54.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:55.081 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:55.103 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:55.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:55.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:55.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:54 smithi102 ceph-mon[110183]: pgmap v472: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 MiB data, 1015 MiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 268 op/s; 15057/84 objects degraded (17925.000%); 0 B/s, 214 objects/s recovering 2024-09-13T14:30:55.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:54 smithi102 ceph-mon[115269]: pgmap v472: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 MiB data, 1015 MiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 268 op/s; 15057/84 objects degraded (17925.000%); 0 B/s, 214 objects/s recovering 2024-09-13T14:30:55.205 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:55.227 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:55.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:55.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:55.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:54 smithi195 ceph-mon[104262]: pgmap v472: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 MiB data, 1015 MiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 268 op/s; 15057/84 objects degraded (17925.000%); 0 B/s, 214 objects/s recovering 2024-09-13T14:30:55.329 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:30:55.351 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:55.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:55.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:30:55.453 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:55.475 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:55.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:55.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:55.577 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:55.600 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:55.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:55.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:55.702 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:55.725 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:55.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:55.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:55.827 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:55.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:55.880 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:55.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:55.913 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:30:55.913 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:30:55.981 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:56.004 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:56.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:56.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:56.105 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:30:56.123 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:55 smithi195 ceph-mon[104262]: osdmap e156: 8 total, 8 up, 6 in 2024-09-13T14:30:56.123 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:55 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:30:56.130 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:56.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:56.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:30:56.133 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:55 smithi102 ceph-mon[110183]: osdmap e156: 8 total, 8 up, 6 in 2024-09-13T14:30:56.133 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:55 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:30:56.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:55 smithi102 ceph-mon[115269]: osdmap e156: 8 total, 8 up, 6 in 2024-09-13T14:30:56.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:55 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:30:56.232 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:30:56.257 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:56.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:56.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:30:56.358 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:56.381 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:56.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:56.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:56.483 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:56.508 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:56.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:56.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:56.595 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T14:30:56.610 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:30:56.652 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:30:56 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:30:56] "GET /metrics HTTP/1.1" 200 36513 "" "Prometheus/2.51.0" 2024-09-13T14:30:56.661 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:56.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:56.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:30:56.705 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:30:56.705 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:30:56.733 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:30:56.733 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:30:56.753 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:30:56.760 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:30:56.761 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:30:56.762 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:56.791 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:56.791 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:30:56.791 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:30:56.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:56.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:56.813 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:30:56.813 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:30:56.835 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:30:56.835 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:30:56.839 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:30:56.853 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:30:56.853 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:30:56.881 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:30:56.881 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:30:56.893 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:56.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:56 smithi102 ceph-mon[110183]: pgmap v474: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 MiB data, 1015 MiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 12 op/s; 15057/84 objects degraded (17925.000%); 0 B/s, 182 objects/s recovering 2024-09-13T14:30:56.912 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'unique_pool_0' removed 2024-09-13T14:30:56.920 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:56.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:56.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:56.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:56 smithi195 ceph-mon[104262]: pgmap v474: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 MiB data, 1015 MiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 12 op/s; 15057/84 objects degraded (17925.000%); 0 B/s, 182 objects/s recovering 2024-09-13T14:30:56.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:56 smithi195 ceph-mon[104262]: osdmap e157: 8 total, 8 up, 6 in 2024-09-13T14:30:56.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:56 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1505677773' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:30:56.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:56 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:30:56.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:56 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/675223231' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:30:57.022 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:57.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:57.044 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:57.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:57.145 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:57.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:57.166 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:57.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:57.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:56 smithi102 ceph-mon[110183]: osdmap e157: 8 total, 8 up, 6 in 2024-09-13T14:30:57.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:56 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1505677773' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:30:57.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:56 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:30:57.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:56 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/675223231' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:30:57.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:56 smithi102 ceph-mon[115269]: pgmap v474: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 MiB data, 1015 MiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 12 op/s; 15057/84 objects degraded (17925.000%); 0 B/s, 182 objects/s recovering 2024-09-13T14:30:57.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:56 smithi102 ceph-mon[115269]: osdmap e157: 8 total, 8 up, 6 in 2024-09-13T14:30:57.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:56 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1505677773' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:30:57.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:56 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:30:57.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:56 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/675223231' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:30:57.267 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:30:57.289 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:57.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:57.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:30:57.391 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:30:57.412 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:57.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:57.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:30:57.514 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:30:57.535 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:57.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:57.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:30:57.606 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-13T14:30:57.607 INFO:tasks.radosbench:Beginning radosbench... 2024-09-13T14:30:57.607 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-09-13T14:30:57.607 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_1 16 2024-09-13T14:30:57.637 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:57.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:57.672 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:57.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:57.773 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:57.794 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:57.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:57.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:57.896 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:57.920 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:57.920 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:30:57.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:57.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:58.022 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:58.044 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:58.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:58.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:58.146 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:58.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:57 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:30:58.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:57 smithi102 ceph-mon[110183]: osdmap e158: 8 total, 8 up, 6 in 2024-09-13T14:30:58.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:57 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:30:58.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:57 smithi102 ceph-mon[115269]: osdmap e158: 8 total, 8 up, 6 in 2024-09-13T14:30:58.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:58.169 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:58.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:58.246 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:57 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:30:58.246 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:57 smithi195 ceph-mon[104262]: osdmap e158: 8 total, 8 up, 6 in 2024-09-13T14:30:58.270 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:58.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:58.292 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:58.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:58.393 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:58.414 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:58.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:58.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:58.516 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:30:58.539 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:58.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:58.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:30:58.641 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:30:58.664 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:58.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:58.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:30:58.753 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:30:58.753 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-09-13T14:30:58.753 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-09-13T14:30:58.753 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-13T14:30:58.753 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.754 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:88020 5.e 5:7278f105:::benchmark_data_smithi102_155989_object88019:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:30:58.754 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:22.488136+0000", 2024-09-13T14:30:58.754 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 216.26219780900001, 2024-09-13T14:30:58.754 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.116470484, 2024-09-13T14:30:58.754 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:30:58.754 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:30:58.755 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:30:58.755 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:30:58.755 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:30:58.755 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 88020 2024-09-13T14:30:58.755 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.755 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:30:58.756 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.756 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:30:58.756 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.488136+0000", 2024-09-13T14:30:58.756 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:30:58.756 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.756 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.757 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:30:58.757 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.488136+0000", 2024-09-13T14:30:58.757 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:30:58.758 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.758 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.758 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:30:58.758 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.488134+0000", 2024-09-13T14:30:58.758 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:30:58.758 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.758 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.758 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:30:58.758 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.488190+0000", 2024-09-13T14:30:58.759 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5541999999999998e-05 2024-09-13T14:30:58.759 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.759 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.759 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:30:58.759 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.488200+0000", 2024-09-13T14:30:58.759 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0465999999999999e-05 2024-09-13T14:30:58.759 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.759 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.759 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:30:58.760 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.488207+0000", 2024-09-13T14:30:58.760 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.601e-06 2024-09-13T14:30:58.760 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.760 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.760 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:30:58.760 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.490502+0000", 2024-09-13T14:30:58.760 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002294904 2024-09-13T14:30:58.760 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.760 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.760 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:30:58.760 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.490634+0000", 2024-09-13T14:30:58.760 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000131867 2024-09-13T14:30:58.760 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.761 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.761 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:30:58.761 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.490768+0000", 2024-09-13T14:30:58.761 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013445800000000001 2024-09-13T14:30:58.761 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.761 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.761 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:58.761 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.500446+0000", 2024-09-13T14:30:58.761 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0096782120000000003 2024-09-13T14:30:58.761 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.761 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.761 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:58.762 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.503247+0000", 2024-09-13T14:30:58.762 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0028011910000000002 2024-09-13T14:30:58.762 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.762 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.762 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:30:58.762 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.604320+0000", 2024-09-13T14:30:58.762 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.101072963 2024-09-13T14:30:58.762 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.762 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.762 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:30:58.762 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.604497+0000", 2024-09-13T14:30:58.763 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00017705999999999999 2024-09-13T14:30:58.763 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.763 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.763 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:30:58.763 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.604606+0000", 2024-09-13T14:30:58.763 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010888200000000001 2024-09-13T14:30:58.763 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:30:58.763 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:30:58.763 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:30:58.763 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.763 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.763 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:88021 5.e 5:70ef7763:::benchmark_data_smithi102_155989_object88020:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:30:58.763 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:22.488214+0000", 2024-09-13T14:30:58.764 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 216.262119353, 2024-09-13T14:30:58.764 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.116624456, 2024-09-13T14:30:58.764 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:30:58.764 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:30:58.764 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:30:58.764 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:30:58.764 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:30:58.764 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 88021 2024-09-13T14:30:58.764 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.764 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:30:58.764 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.764 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:30:58.764 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.488214+0000", 2024-09-13T14:30:58.764 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:30:58.765 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.765 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.765 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:30:58.765 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.488214+0000", 2024-09-13T14:30:58.765 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:30:58.765 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.765 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.765 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:30:58.765 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.488214+0000", 2024-09-13T14:30:58.765 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:30:58.765 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.766 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.766 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:30:58.766 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.488278+0000", 2024-09-13T14:30:58.766 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3907999999999999e-05 2024-09-13T14:30:58.766 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.766 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.766 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:30:58.766 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.488285+0000", 2024-09-13T14:30:58.766 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2640000000000002e-06 2024-09-13T14:30:58.766 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.766 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.766 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:30:58.767 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.488290+0000", 2024-09-13T14:30:58.767 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4519999999999996e-06 2024-09-13T14:30:58.767 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.767 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.767 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:30:58.767 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.493941+0000", 2024-09-13T14:30:58.768 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0056507969999999999 2024-09-13T14:30:58.768 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.768 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.768 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:30:58.768 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.494056+0000", 2024-09-13T14:30:58.768 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011510099999999999 2024-09-13T14:30:58.768 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.768 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.769 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:30:58.769 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.494177+0000", 2024-09-13T14:30:58.769 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012020500000000001 2024-09-13T14:30:58.769 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.769 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.769 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:58.769 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.500482+0000", 2024-09-13T14:30:58.769 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0063050329999999998 2024-09-13T14:30:58.770 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.770 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.771 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:58.771 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.594207+0000", 2024-09-13T14:30:58.771 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.093725369000000003 2024-09-13T14:30:58.771 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.771 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.772 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:30:58.772 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.604627+0000", 2024-09-13T14:30:58.772 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010419964 2024-09-13T14:30:58.772 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.772 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.772 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:30:58.772 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.604736+0000", 2024-09-13T14:30:58.773 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0001094 2024-09-13T14:30:58.773 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.773 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.773 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:30:58.773 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.604839+0000", 2024-09-13T14:30:58.773 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000102435 2024-09-13T14:30:58.773 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:30:58.774 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:30:58.774 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:30:58.774 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.774 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.774 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.44546.0:88026 5.6 e129/125 5:690e1d48:::benchmark_data_smithi102_155989_object88025:head v 129'5446, mlcod=0'0)", 2024-09-13T14:30:58.774 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:22.495761+0000", 2024-09-13T14:30:58.774 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 216.25457272400001, 2024-09-13T14:30:58.774 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.124147801, 2024-09-13T14:30:58.775 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:30:58.775 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:30:58.775 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:30:58.775 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.775 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:30:58.775 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.495761+0000", 2024-09-13T14:30:58.776 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:30:58.776 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.776 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.776 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:30:58.776 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.495761+0000", 2024-09-13T14:30:58.776 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:30:58.777 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.777 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.777 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:30:58.777 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.495760+0000", 2024-09-13T14:30:58.777 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:30:58.778 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.778 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.778 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:30:58.778 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.495782+0000", 2024-09-13T14:30:58.778 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1478999999999999e-05 2024-09-13T14:30:58.778 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.778 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.778 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:30:58.778 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.495792+0000", 2024-09-13T14:30:58.778 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0538e-05 2024-09-13T14:30:58.779 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.779 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.779 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:30:58.779 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.495798+0000", 2024-09-13T14:30:58.779 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0689999999999998e-06 2024-09-13T14:30:58.779 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.779 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.779 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:30:58.779 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.500510+0000", 2024-09-13T14:30:58.779 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0047120640000000002 2024-09-13T14:30:58.779 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.779 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.779 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:30:58.780 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.500532+0000", 2024-09-13T14:30:58.780 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1551000000000001e-05 2024-09-13T14:30:58.780 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.780 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.780 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:30:58.781 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.619795+0000", 2024-09-13T14:30:58.781 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.11926363600000001 2024-09-13T14:30:58.781 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.781 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.781 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:30:58.781 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.619909+0000", 2024-09-13T14:30:58.781 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011324600000000001 2024-09-13T14:30:58.781 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:30:58.781 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:30:58.781 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:30:58.781 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.782 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.782 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:91654 5.a 5:5299faa7:::benchmark_data_smithi102_155989_object91653:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:30:58.782 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:27.316609+0000", 2024-09-13T14:30:58.782 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 211.43372460399999, 2024-09-13T14:30:58.783 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.117235135, 2024-09-13T14:30:58.783 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:30:58.783 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:30:58.783 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:30:58.783 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:30:58.783 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:30:58.783 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 91654 2024-09-13T14:30:58.783 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.783 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:30:58.783 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.783 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:30:58.783 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.316609+0000", 2024-09-13T14:30:58.783 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:30:58.784 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.784 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.784 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:30:58.784 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.316609+0000", 2024-09-13T14:30:58.784 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:30:58.784 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.784 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.784 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:30:58.784 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.316607+0000", 2024-09-13T14:30:58.784 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:30:58.784 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.784 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.784 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:30:58.785 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.316644+0000", 2024-09-13T14:30:58.785 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6854999999999999e-05 2024-09-13T14:30:58.785 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.785 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.785 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:30:58.785 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.316660+0000", 2024-09-13T14:30:58.785 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5599000000000001e-05 2024-09-13T14:30:58.785 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.785 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.785 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:30:58.785 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.316667+0000", 2024-09-13T14:30:58.785 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4279999999999999e-06 2024-09-13T14:30:58.785 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.785 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.786 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:30:58.786 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.324526+0000", 2024-09-13T14:30:58.786 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0078587329999999997 2024-09-13T14:30:58.786 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.786 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.786 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:30:58.786 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.324681+0000", 2024-09-13T14:30:58.786 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015515599999999999 2024-09-13T14:30:58.786 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.787 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.787 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,3", 2024-09-13T14:30:58.787 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.324860+0000", 2024-09-13T14:30:58.787 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00017919999999999999 2024-09-13T14:30:58.787 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.787 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.787 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:30:58.787 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.343677+0000", 2024-09-13T14:30:58.787 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018816833000000002 2024-09-13T14:30:58.787 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.787 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.787 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:58.787 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.343746+0000", 2024-09-13T14:30:58.787 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8774000000000003e-05 2024-09-13T14:30:58.788 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.788 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.788 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:58.788 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.425055+0000", 2024-09-13T14:30:58.788 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.081308742000000003 2024-09-13T14:30:58.788 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.788 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.788 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:30:58.788 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.425187+0000", 2024-09-13T14:30:58.789 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013233299999999999 2024-09-13T14:30:58.789 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.789 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.789 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:30:58.789 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.433844+0000", 2024-09-13T14:30:58.789 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0086572899999999998 2024-09-13T14:30:58.789 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:30:58.789 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:30:58.789 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:30:58.790 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.790 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.790 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:91656 5.a 5:508f8413:::benchmark_data_smithi102_155989_object91655:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:30:58.790 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:27.316731+0000", 2024-09-13T14:30:58.790 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 211.43360294499999, 2024-09-13T14:30:58.790 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.12772304500000001, 2024-09-13T14:30:58.790 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:30:58.790 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:30:58.790 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:30:58.790 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:30:58.790 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:30:58.790 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 91656 2024-09-13T14:30:58.790 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.791 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:30:58.791 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.791 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:30:58.791 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.316731+0000", 2024-09-13T14:30:58.791 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:30:58.791 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.791 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.791 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:30:58.791 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.316731+0000", 2024-09-13T14:30:58.791 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:30:58.791 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.791 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.791 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:30:58.791 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.316730+0000", 2024-09-13T14:30:58.792 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:30:58.792 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.792 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.792 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:30:58.792 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.316764+0000", 2024-09-13T14:30:58.792 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.3442000000000003e-05 2024-09-13T14:30:58.792 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.792 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.792 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:30:58.792 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.316773+0000", 2024-09-13T14:30:58.792 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.2089999999999994e-06 2024-09-13T14:30:58.792 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.792 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.792 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:30:58.793 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.316778+0000", 2024-09-13T14:30:58.793 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0590000000000002e-06 2024-09-13T14:30:58.793 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.793 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.793 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:30:58.793 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.331802+0000", 2024-09-13T14:30:58.793 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.015023642 2024-09-13T14:30:58.793 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.793 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.793 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:30:58.793 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.331948+0000", 2024-09-13T14:30:58.793 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000146482 2024-09-13T14:30:58.793 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.793 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.794 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:30:58.795 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,3", 2024-09-13T14:30:58.796 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.332100+0000", 2024-09-13T14:30:58.796 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015182699999999999 2024-09-13T14:30:58.796 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.796 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.796 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:58.796 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.343857+0000", 2024-09-13T14:30:58.796 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.01175733 2024-09-13T14:30:58.796 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.796 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.797 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:30:58.797 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.343889+0000", 2024-09-13T14:30:58.797 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.1683000000000001e-05 2024-09-13T14:30:58.797 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.797 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.797 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:58.797 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.438087+0000", 2024-09-13T14:30:58.797 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.094197914999999993 2024-09-13T14:30:58.797 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.797 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.797 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:30:58.797 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.438139+0000", 2024-09-13T14:30:58.797 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2346000000000003e-05 2024-09-13T14:30:58.797 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.798 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.798 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:30:58.798 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.444454+0000", 2024-09-13T14:30:58.798 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0063145229999999998 2024-09-13T14:30:58.798 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:30:58.798 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:30:58.798 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:30:58.798 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.798 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.798 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:91657 5.a 5:51ff3ce2:::benchmark_data_smithi102_155989_object91656:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:30:58.798 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:27.316895+0000", 2024-09-13T14:30:58.798 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 211.43343883399999, 2024-09-13T14:30:58.798 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.13008430200000001, 2024-09-13T14:30:58.798 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:30:58.799 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:30:58.799 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:30:58.799 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:30:58.799 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:30:58.799 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 91657 2024-09-13T14:30:58.799 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.799 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:30:58.799 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.799 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:30:58.799 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.316895+0000", 2024-09-13T14:30:58.799 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:30:58.799 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.799 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.799 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:30:58.800 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.316895+0000", 2024-09-13T14:30:58.800 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:30:58.800 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.800 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.800 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:30:58.800 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.316894+0000", 2024-09-13T14:30:58.800 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:30:58.800 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.800 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.800 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:30:58.800 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.316964+0000", 2024-09-13T14:30:58.800 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.0160000000000006e-05 2024-09-13T14:30:58.800 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.801 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.801 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:30:58.801 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.316984+0000", 2024-09-13T14:30:58.801 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9992000000000001e-05 2024-09-13T14:30:58.801 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.801 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.801 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:30:58.801 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.316990+0000", 2024-09-13T14:30:58.801 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7649999999999999e-06 2024-09-13T14:30:58.801 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.801 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.801 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:30:58.801 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.335386+0000", 2024-09-13T14:30:58.801 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018395575000000001 2024-09-13T14:30:58.802 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.802 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.802 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:30:58.802 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.335511+0000", 2024-09-13T14:30:58.802 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012501499999999999 2024-09-13T14:30:58.802 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.802 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.802 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,3", 2024-09-13T14:30:58.802 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.335664+0000", 2024-09-13T14:30:58.802 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015311200000000001 2024-09-13T14:30:58.802 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.802 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.802 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:30:58.802 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.343907+0000", 2024-09-13T14:30:58.803 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0082433409999999995 2024-09-13T14:30:58.803 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.803 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.803 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:58.803 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.343936+0000", 2024-09-13T14:30:58.803 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9009e-05 2024-09-13T14:30:58.803 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.803 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.803 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:58.803 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.444517+0000", 2024-09-13T14:30:58.803 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.100580424 2024-09-13T14:30:58.803 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.803 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.803 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:30:58.804 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.444619+0000", 2024-09-13T14:30:58.804 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010208399999999999 2024-09-13T14:30:58.804 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.804 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.804 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:30:58.804 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.446979+0000", 2024-09-13T14:30:58.804 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0023604229999999999 2024-09-13T14:30:58.804 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:30:58.804 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:30:58.804 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:30:58.804 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.804 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.804 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:91665 5.a 5:5e4f4c4f:::benchmark_data_smithi102_155989_object91664:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:30:58.805 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:27.326817+0000", 2024-09-13T14:30:58.805 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 211.423516836, 2024-09-13T14:30:58.805 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.122295468, 2024-09-13T14:30:58.805 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:30:58.805 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:30:58.805 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:30:58.805 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:30:58.805 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:30:58.805 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 91665 2024-09-13T14:30:58.805 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.805 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:30:58.805 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.805 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:30:58.806 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.326817+0000", 2024-09-13T14:30:58.806 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:30:58.806 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.806 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.806 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:30:58.806 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.326817+0000", 2024-09-13T14:30:58.806 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:30:58.806 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.806 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.806 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:30:58.806 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.326815+0000", 2024-09-13T14:30:58.806 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:30:58.806 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.806 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.807 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:30:58.807 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.326860+0000", 2024-09-13T14:30:58.807 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4332000000000002e-05 2024-09-13T14:30:58.807 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.807 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.807 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:30:58.807 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.326868+0000", 2024-09-13T14:30:58.807 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3459999999999994e-06 2024-09-13T14:30:58.807 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.807 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.807 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:30:58.807 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.326873+0000", 2024-09-13T14:30:58.807 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4349999999999996e-06 2024-09-13T14:30:58.807 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.808 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.808 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:30:58.808 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.343951+0000", 2024-09-13T14:30:58.808 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017078142000000001 2024-09-13T14:30:58.808 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.808 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.808 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:30:58.808 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.344068+0000", 2024-09-13T14:30:58.808 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011694800000000001 2024-09-13T14:30:58.808 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.808 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.808 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,3", 2024-09-13T14:30:58.808 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.344179+0000", 2024-09-13T14:30:58.808 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000111062 2024-09-13T14:30:58.809 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.809 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.809 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:58.809 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.346456+0000", 2024-09-13T14:30:58.809 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002277004 2024-09-13T14:30:58.809 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.809 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.809 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:30:58.809 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.346503+0000", 2024-09-13T14:30:58.809 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6257000000000001e-05 2024-09-13T14:30:58.809 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.809 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.809 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:58.810 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.447273+0000", 2024-09-13T14:30:58.810 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.10077000799999999 2024-09-13T14:30:58.810 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.810 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.810 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:30:58.810 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.447325+0000", 2024-09-13T14:30:58.810 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2488000000000003e-05 2024-09-13T14:30:58.810 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.810 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.810 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:30:58.810 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.449112+0000", 2024-09-13T14:30:58.810 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001787039 2024-09-13T14:30:58.810 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:30:58.810 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:30:58.810 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:30:58.811 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.811 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.811 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:148474 5.a 5:598131e0:::benchmark_data_smithi102_155989_object30556:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T14:30:58.811 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:28:43.154494+0000", 2024-09-13T14:30:58.811 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 135.59583955900001, 2024-09-13T14:30:58.811 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.120926199, 2024-09-13T14:30:58.811 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:30:58.811 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:30:58.811 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:30:58.811 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:30:58.811 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:30:58.811 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 148474 2024-09-13T14:30:58.811 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.811 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:30:58.812 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.812 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:30:58.812 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.154494+0000", 2024-09-13T14:30:58.812 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:30:58.812 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.812 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.812 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:30:58.812 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.154494+0000", 2024-09-13T14:30:58.812 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:30:58.812 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.812 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.812 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:30:58.812 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.154492+0000", 2024-09-13T14:30:58.812 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:30:58.813 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.813 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.813 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:30:58.813 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.154499+0000", 2024-09-13T14:30:58.813 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4969999999999998e-06 2024-09-13T14:30:58.813 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.813 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.813 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:30:58.813 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.154515+0000", 2024-09-13T14:30:58.813 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5719e-05 2024-09-13T14:30:58.813 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.813 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.813 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:30:58.813 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.154526+0000", 2024-09-13T14:30:58.814 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1756000000000001e-05 2024-09-13T14:30:58.814 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.814 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.814 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:30:58.814 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.173309+0000", 2024-09-13T14:30:58.814 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018782936 2024-09-13T14:30:58.814 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.814 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.814 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:30:58.814 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.173647+0000", 2024-09-13T14:30:58.814 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00033812600000000001 2024-09-13T14:30:58.814 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.814 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.814 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,3", 2024-09-13T14:30:58.815 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.174041+0000", 2024-09-13T14:30:58.815 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00039339300000000002 2024-09-13T14:30:58.815 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.815 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.815 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:58.815 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.183800+0000", 2024-09-13T14:30:58.815 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0097590379999999994 2024-09-13T14:30:58.815 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.815 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.815 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:30:58.815 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.183808+0000", 2024-09-13T14:30:58.815 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9379999999999992e-06 2024-09-13T14:30:58.815 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.815 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.816 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:58.816 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.273700+0000", 2024-09-13T14:30:58.816 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.089892578000000001 2024-09-13T14:30:58.816 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.816 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.816 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:30:58.816 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.273766+0000", 2024-09-13T14:30:58.816 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5741999999999995e-05 2024-09-13T14:30:58.816 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.816 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.816 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:30:58.816 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.275420+0000", 2024-09-13T14:30:58.816 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0016540960000000001 2024-09-13T14:30:58.816 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:30:58.817 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:30:58.817 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:30:58.817 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.817 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.817 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:148479 5.a 5:51e5171a:::benchmark_data_smithi102_155989_object30561:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T14:30:58.817 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:28:43.161952+0000", 2024-09-13T14:30:58.817 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 135.58838145999999, 2024-09-13T14:30:58.817 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.12857813600000001, 2024-09-13T14:30:58.817 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:30:58.817 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:30:58.817 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:30:58.817 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:30:58.817 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:30:58.817 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 148479 2024-09-13T14:30:58.818 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.818 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:30:58.818 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.818 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:30:58.818 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.161952+0000", 2024-09-13T14:30:58.818 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:30:58.818 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.818 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.818 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:30:58.818 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.161952+0000", 2024-09-13T14:30:58.818 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:30:58.818 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.818 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.818 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:30:58.819 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.161950+0000", 2024-09-13T14:30:58.819 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:30:58.819 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.819 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.819 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:30:58.819 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.161958+0000", 2024-09-13T14:30:58.819 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3050000000000004e-06 2024-09-13T14:30:58.819 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.819 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.819 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:30:58.819 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.161971+0000", 2024-09-13T14:30:58.819 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3526999999999999e-05 2024-09-13T14:30:58.819 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.819 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.820 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:30:58.820 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.161978+0000", 2024-09-13T14:30:58.820 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1729999999999998e-06 2024-09-13T14:30:58.820 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.820 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.820 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:30:58.821 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.178715+0000", 2024-09-13T14:30:58.821 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016737031999999999 2024-09-13T14:30:58.821 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.821 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.821 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:30:58.821 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.178871+0000", 2024-09-13T14:30:58.821 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015592700000000001 2024-09-13T14:30:58.821 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.821 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.821 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,3", 2024-09-13T14:30:58.821 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.179040+0000", 2024-09-13T14:30:58.821 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016913299999999999 2024-09-13T14:30:58.821 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.821 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.822 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:30:58.822 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.183814+0000", 2024-09-13T14:30:58.822 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0047731980000000002 2024-09-13T14:30:58.822 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.822 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.822 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:58.822 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.191883+0000", 2024-09-13T14:30:58.822 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0080691019999999999 2024-09-13T14:30:58.822 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.822 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.822 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:58.822 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.280583+0000", 2024-09-13T14:30:58.822 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.088700627000000004 2024-09-13T14:30:58.822 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.823 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.823 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:30:58.823 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.280634+0000", 2024-09-13T14:30:58.823 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0494000000000001e-05 2024-09-13T14:30:58.823 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.823 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.823 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:30:58.823 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.290530+0000", 2024-09-13T14:30:58.823 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0098965500000000005 2024-09-13T14:30:58.823 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:30:58.823 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:30:58.824 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:30:58.824 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.824 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.824 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:148482 5.a 5:56be860e:::benchmark_data_smithi102_155989_object30564:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T14:30:58.824 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:28:43.168340+0000", 2024-09-13T14:30:58.824 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 135.58199386499999, 2024-09-13T14:30:58.824 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.12385785000000001, 2024-09-13T14:30:58.824 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:30:58.824 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:30:58.824 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:30:58.824 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:30:58.824 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:30:58.824 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 148482 2024-09-13T14:30:58.825 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.825 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:30:58.825 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.825 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:30:58.825 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.168340+0000", 2024-09-13T14:30:58.825 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:30:58.825 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.825 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.825 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:30:58.825 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.168340+0000", 2024-09-13T14:30:58.825 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:30:58.825 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.825 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.826 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:30:58.826 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.168338+0000", 2024-09-13T14:30:58.826 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:30:58.826 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.826 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.826 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:30:58.826 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.168345+0000", 2024-09-13T14:30:58.826 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2200000000000003e-06 2024-09-13T14:30:58.826 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.826 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.826 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:30:58.826 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.168357+0000", 2024-09-13T14:30:58.826 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1708999999999999e-05 2024-09-13T14:30:58.826 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.827 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.827 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:30:58.827 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.168363+0000", 2024-09-13T14:30:58.827 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.584e-06 2024-09-13T14:30:58.827 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.827 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.827 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:30:58.827 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.183836+0000", 2024-09-13T14:30:58.827 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.015473038999999999 2024-09-13T14:30:58.827 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.827 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.827 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:30:58.827 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.183978+0000", 2024-09-13T14:30:58.827 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014142500000000001 2024-09-13T14:30:58.828 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.828 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.828 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,3", 2024-09-13T14:30:58.828 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.184115+0000", 2024-09-13T14:30:58.828 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013713000000000001 2024-09-13T14:30:58.828 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.828 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.828 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:30:58.828 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.191891+0000", 2024-09-13T14:30:58.828 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0077762400000000002 2024-09-13T14:30:58.828 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.828 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.828 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:58.828 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.191945+0000", 2024-09-13T14:30:58.829 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3514000000000003e-05 2024-09-13T14:30:58.829 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.829 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.829 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:58.829 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.290583+0000", 2024-09-13T14:30:58.829 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.098638112 2024-09-13T14:30:58.829 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.829 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.829 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:30:58.829 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.290626+0000", 2024-09-13T14:30:58.829 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3579999999999999e-05 2024-09-13T14:30:58.829 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.829 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.829 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:30:58.830 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.292198+0000", 2024-09-13T14:30:58.830 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00157139 2024-09-13T14:30:58.830 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:30:58.830 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:30:58.830 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:30:58.830 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.830 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.830 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:148489 5.a 5:583543c7:::benchmark_data_smithi102_155989_object30571:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T14:30:58.830 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:28:43.180323+0000", 2024-09-13T14:30:58.830 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 135.570010709, 2024-09-13T14:30:58.830 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.11345999900000001, 2024-09-13T14:30:58.830 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:30:58.830 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:30:58.830 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:30:58.831 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:30:58.831 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:30:58.831 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 148489 2024-09-13T14:30:58.831 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.831 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:30:58.831 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.831 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:30:58.831 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.180323+0000", 2024-09-13T14:30:58.831 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:30:58.831 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.831 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.831 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:30:58.831 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.180323+0000", 2024-09-13T14:30:58.831 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:30:58.832 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.832 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.832 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:30:58.832 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.180321+0000", 2024-09-13T14:30:58.832 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:30:58.832 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.832 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.832 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:30:58.832 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.180329+0000", 2024-09-13T14:30:58.832 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.0400000000000004e-06 2024-09-13T14:30:58.832 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.832 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.832 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:30:58.832 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.180341+0000", 2024-09-13T14:30:58.833 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2754e-05 2024-09-13T14:30:58.833 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.833 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.833 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:30:58.833 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.180349+0000", 2024-09-13T14:30:58.833 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.469e-06 2024-09-13T14:30:58.833 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.833 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.833 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:30:58.833 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.188343+0000", 2024-09-13T14:30:58.833 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0079944730000000002 2024-09-13T14:30:58.833 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.833 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.833 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:30:58.834 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.188511+0000", 2024-09-13T14:30:58.834 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016730799999999999 2024-09-13T14:30:58.834 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.834 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.834 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,3", 2024-09-13T14:30:58.834 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.188670+0000", 2024-09-13T14:30:58.834 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000159393 2024-09-13T14:30:58.834 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.834 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.834 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:30:58.834 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.191897+0000", 2024-09-13T14:30:58.834 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0032270609999999998 2024-09-13T14:30:58.834 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.834 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.835 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:58.835 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.192991+0000", 2024-09-13T14:30:58.835 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00109391 2024-09-13T14:30:58.835 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.835 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.835 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:58.835 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.292241+0000", 2024-09-13T14:30:58.835 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.099249712000000004 2024-09-13T14:30:58.835 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.835 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.835 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:30:58.835 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.292283+0000", 2024-09-13T14:30:58.835 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2750000000000002e-05 2024-09-13T14:30:58.835 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.836 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.836 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:30:58.836 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.293783+0000", 2024-09-13T14:30:58.836 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0014995550000000001 2024-09-13T14:30:58.836 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:30:58.836 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:30:58.836 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:30:58.836 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.836 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.836 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:214611 5.e 5:7939c0fa:::benchmark_data_smithi102_155989_object96693:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-09-13T14:30:58.836 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:16.526407+0000", 2024-09-13T14:30:58.836 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 42.22392619, 2024-09-13T14:30:58.836 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.117210882, 2024-09-13T14:30:58.836 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:30:58.837 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:30:58.837 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:30:58.837 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:30:58.837 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:30:58.837 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 214611 2024-09-13T14:30:58.837 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.837 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:30:58.837 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.837 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:30:58.837 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.526407+0000", 2024-09-13T14:30:58.837 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:30:58.837 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.837 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.837 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:30:58.838 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.526407+0000", 2024-09-13T14:30:58.838 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:30:58.838 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.838 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.838 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:30:58.838 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.526406+0000", 2024-09-13T14:30:58.838 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:30:58.838 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.838 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.838 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:30:58.838 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.526416+0000", 2024-09-13T14:30:58.838 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0209e-05 2024-09-13T14:30:58.838 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.838 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.839 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:30:58.839 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.526430+0000", 2024-09-13T14:30:58.839 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3569e-05 2024-09-13T14:30:58.839 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.839 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.839 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:30:58.839 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.526437+0000", 2024-09-13T14:30:58.839 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8699999999999992e-06 2024-09-13T14:30:58.839 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.839 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.839 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:30:58.839 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.528856+0000", 2024-09-13T14:30:58.839 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0024188590000000002 2024-09-13T14:30:58.839 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.840 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.840 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:30:58.840 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.528981+0000", 2024-09-13T14:30:58.840 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012508300000000001 2024-09-13T14:30:58.840 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.840 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.840 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:30:58.840 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.529094+0000", 2024-09-13T14:30:58.840 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011223200000000001 2024-09-13T14:30:58.840 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.840 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.840 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:30:58.840 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.537171+0000", 2024-09-13T14:30:58.841 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0080773770000000002 2024-09-13T14:30:58.841 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.841 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.841 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:58.841 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.538801+0000", 2024-09-13T14:30:58.841 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0016298529999999999 2024-09-13T14:30:58.841 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.841 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.841 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:58.841 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.643545+0000", 2024-09-13T14:30:58.841 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.10474383600000001 2024-09-13T14:30:58.841 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.841 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.841 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:30:58.842 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.643583+0000", 2024-09-13T14:30:58.842 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8600999999999999e-05 2024-09-13T14:30:58.842 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.842 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.842 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:30:58.842 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.643618+0000", 2024-09-13T14:30:58.842 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5002000000000001e-05 2024-09-13T14:30:58.842 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:30:58.842 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:30:58.842 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:30:58.842 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.842 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.842 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:214614 5.e 5:75de5dda:::benchmark_data_smithi102_155989_object96696:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-09-13T14:30:58.842 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:16.526444+0000", 2024-09-13T14:30:58.843 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 42.223889522, 2024-09-13T14:30:58.843 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.11734820999999999, 2024-09-13T14:30:58.843 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:30:58.843 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:30:58.843 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:30:58.843 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:30:58.843 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:30:58.843 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 214614 2024-09-13T14:30:58.843 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.843 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:30:58.843 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.843 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:30:58.843 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.526444+0000", 2024-09-13T14:30:58.843 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:30:58.844 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.844 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.844 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:30:58.844 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.526444+0000", 2024-09-13T14:30:58.844 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:30:58.844 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.844 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.844 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:30:58.844 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.526444+0000", 2024-09-13T14:30:58.844 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:30:58.844 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.844 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.844 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:30:58.845 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.526447+0000", 2024-09-13T14:30:58.845 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.898e-06 2024-09-13T14:30:58.845 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.845 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.845 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:30:58.845 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.526454+0000", 2024-09-13T14:30:58.845 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9549999999999992e-06 2024-09-13T14:30:58.845 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.845 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.845 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:30:58.845 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.526460+0000", 2024-09-13T14:30:58.845 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.463e-06 2024-09-13T14:30:58.845 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.846 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.846 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:30:58.846 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.532900+0000", 2024-09-13T14:30:58.846 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0064400430000000003 2024-09-13T14:30:58.846 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.846 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.846 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:30:58.846 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.533163+0000", 2024-09-13T14:30:58.846 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00026262900000000001 2024-09-13T14:30:58.846 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.846 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.846 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:30:58.846 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.533269+0000", 2024-09-13T14:30:58.846 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000106754 2024-09-13T14:30:58.847 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.847 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.847 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:30:58.847 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.537256+0000", 2024-09-13T14:30:58.847 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0039869809999999997 2024-09-13T14:30:58.847 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.847 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.847 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:58.847 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.541076+0000", 2024-09-13T14:30:58.847 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003819729 2024-09-13T14:30:58.847 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.847 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.847 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:58.847 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.643661+0000", 2024-09-13T14:30:58.848 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.102584594 2024-09-13T14:30:58.848 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.848 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.848 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:30:58.848 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.643759+0000", 2024-09-13T14:30:58.848 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.8010999999999994e-05 2024-09-13T14:30:58.848 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.848 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.848 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:30:58.848 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.643792+0000", 2024-09-13T14:30:58.848 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.3606999999999999e-05 2024-09-13T14:30:58.848 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:30:58.848 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:30:58.848 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:30:58.849 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.849 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.849 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:214621 5.a 5:588e6e95:::benchmark_data_smithi102_155989_object96703:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-09-13T14:30:58.849 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:16.535646+0000", 2024-09-13T14:30:58.849 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 42.214687709000003, 2024-09-13T14:30:58.849 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.11604697899999999, 2024-09-13T14:30:58.849 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:30:58.849 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:30:58.849 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:30:58.849 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:30:58.849 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:30:58.849 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 214621 2024-09-13T14:30:58.849 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.849 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:30:58.850 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.850 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:30:58.850 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.535646+0000", 2024-09-13T14:30:58.850 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:30:58.850 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.850 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.850 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:30:58.850 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.535646+0000", 2024-09-13T14:30:58.850 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:30:58.850 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.850 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.850 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:30:58.850 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.535644+0000", 2024-09-13T14:30:58.850 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:30:58.851 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.851 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.851 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:30:58.851 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.535649+0000", 2024-09-13T14:30:58.851 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0749999999999997e-06 2024-09-13T14:30:58.851 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.851 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.851 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:30:58.851 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.535666+0000", 2024-09-13T14:30:58.851 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6697999999999998e-05 2024-09-13T14:30:58.851 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.851 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.851 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:30:58.851 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.535673+0000", 2024-09-13T14:30:58.852 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9249999999999998e-06 2024-09-13T14:30:58.852 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.852 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.852 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:30:58.852 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.535749+0000", 2024-09-13T14:30:58.852 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5555000000000004e-05 2024-09-13T14:30:58.852 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.852 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.852 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:30:58.852 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.535876+0000", 2024-09-13T14:30:58.852 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012745700000000001 2024-09-13T14:30:58.852 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.852 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.853 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,3", 2024-09-13T14:30:58.853 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.536016+0000", 2024-09-13T14:30:58.853 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013996899999999999 2024-09-13T14:30:58.853 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.853 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.853 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:58.853 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.542655+0000", 2024-09-13T14:30:58.853 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0066386359999999998 2024-09-13T14:30:58.853 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.853 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.853 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:30:58.853 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.542665+0000", 2024-09-13T14:30:58.853 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0587999999999999e-05 2024-09-13T14:30:58.853 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.853 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.854 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:58.854 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.650568+0000", 2024-09-13T14:30:58.854 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.10790282700000001 2024-09-13T14:30:58.854 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.854 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.854 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:30:58.854 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.650649+0000", 2024-09-13T14:30:58.854 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1095999999999997e-05 2024-09-13T14:30:58.854 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.854 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.854 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:30:58.854 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.651693+0000", 2024-09-13T14:30:58.854 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0010436410000000001 2024-09-13T14:30:58.854 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:30:58.855 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:30:58.855 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:30:58.855 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.855 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.855 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:214623 5.a 5:5bd7a124:::benchmark_data_smithi102_155989_object96705:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-09-13T14:30:58.855 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:16.537367+0000", 2024-09-13T14:30:58.855 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 42.212966661999999, 2024-09-13T14:30:58.855 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.124219291, 2024-09-13T14:30:58.855 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:30:58.855 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:30:58.855 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:30:58.855 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:30:58.855 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:30:58.855 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 214623 2024-09-13T14:30:58.856 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.856 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:30:58.856 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.856 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:30:58.856 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.537367+0000", 2024-09-13T14:30:58.856 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:30:58.856 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.856 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.856 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:30:58.856 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.537367+0000", 2024-09-13T14:30:58.856 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:30:58.856 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.856 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.857 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:30:58.857 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.537366+0000", 2024-09-13T14:30:58.857 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:30:58.857 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.857 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.857 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:30:58.857 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.537371+0000", 2024-09-13T14:30:58.857 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2530000000000004e-06 2024-09-13T14:30:58.857 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.857 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.857 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:30:58.857 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.537382+0000", 2024-09-13T14:30:58.857 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1054e-05 2024-09-13T14:30:58.857 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.858 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.858 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:30:58.858 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.537390+0000", 2024-09-13T14:30:58.858 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.204e-06 2024-09-13T14:30:58.858 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.858 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.858 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:30:58.858 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.539324+0000", 2024-09-13T14:30:58.858 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001934035 2024-09-13T14:30:58.858 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.858 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.858 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:30:58.858 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.539438+0000", 2024-09-13T14:30:58.858 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011466799999999999 2024-09-13T14:30:58.859 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.859 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.859 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,3", 2024-09-13T14:30:58.859 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.539558+0000", 2024-09-13T14:30:58.859 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012010600000000001 2024-09-13T14:30:58.859 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.859 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.859 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:30:58.859 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.542679+0000", 2024-09-13T14:30:58.859 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003120589 2024-09-13T14:30:58.859 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.859 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.859 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:58.859 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.542723+0000", 2024-09-13T14:30:58.860 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4011000000000003e-05 2024-09-13T14:30:58.860 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.860 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.860 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:58.860 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.660444+0000", 2024-09-13T14:30:58.860 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.117720959 2024-09-13T14:30:58.860 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.860 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.860 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:30:58.860 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.660492+0000", 2024-09-13T14:30:58.860 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7783e-05 2024-09-13T14:30:58.860 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.860 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.861 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:30:58.861 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.661586+0000", 2024-09-13T14:30:58.861 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0010945989999999999 2024-09-13T14:30:58.861 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:30:58.861 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:30:58.861 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:30:58.861 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.861 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.861 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:233222 5.a 5:5d43e0eb:::benchmark_data_smithi102_155989_object115304:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-13T14:30:58.861 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:41.665433+0000", 2024-09-13T14:30:58.861 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 17.084900761, 2024-09-13T14:30:58.861 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.17010931900000001, 2024-09-13T14:30:58.861 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:30:58.861 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:30:58.862 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:30:58.862 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:30:58.862 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:30:58.862 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 233222 2024-09-13T14:30:58.862 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.862 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:30:58.862 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.862 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:30:58.862 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.665433+0000", 2024-09-13T14:30:58.862 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:30:58.862 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.862 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.862 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:30:58.862 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.665433+0000", 2024-09-13T14:30:58.862 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:30:58.863 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.863 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.863 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:30:58.863 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.665432+0000", 2024-09-13T14:30:58.863 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:30:58.863 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.863 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.863 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:30:58.863 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.665437+0000", 2024-09-13T14:30:58.863 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9309999999999999e-06 2024-09-13T14:30:58.863 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.863 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.863 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:30:58.863 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.665447+0000", 2024-09-13T14:30:58.864 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0825000000000001e-05 2024-09-13T14:30:58.864 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.864 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.864 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:30:58.864 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.665454+0000", 2024-09-13T14:30:58.864 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6730000000000003e-06 2024-09-13T14:30:58.864 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.864 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.864 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:30:58.864 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.672469+0000", 2024-09-13T14:30:58.864 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0070153790000000004 2024-09-13T14:30:58.864 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.864 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.864 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:30:58.865 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.803548+0000", 2024-09-13T14:30:58.865 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.13107861000000001 2024-09-13T14:30:58.865 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.865 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.865 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,3", 2024-09-13T14:30:58.865 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.803692+0000", 2024-09-13T14:30:58.865 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014347599999999999 2024-09-13T14:30:58.865 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.865 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.865 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:58.865 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.817160+0000", 2024-09-13T14:30:58.865 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.01346893 2024-09-13T14:30:58.865 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.866 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.866 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:58.866 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.817190+0000", 2024-09-13T14:30:58.866 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9842999999999999e-05 2024-09-13T14:30:58.866 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.866 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.866 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:30:58.866 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.834483+0000", 2024-09-13T14:30:58.866 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017293059 2024-09-13T14:30:58.866 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.866 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.866 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:30:58.866 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.834549+0000", 2024-09-13T14:30:58.866 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5988000000000006e-05 2024-09-13T14:30:58.867 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.867 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.867 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:30:58.867 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.835542+0000", 2024-09-13T14:30:58.867 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00099279300000000006 2024-09-13T14:30:58.867 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:30:58.867 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:30:58.867 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:30:58.867 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.867 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.867 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:233224 5.e 5:73838589:::benchmark_data_smithi102_155989_object115306:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-13T14:30:58.867 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:41.667956+0000", 2024-09-13T14:30:58.867 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 17.082377576999999, 2024-09-13T14:30:58.867 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.15649626, 2024-09-13T14:30:58.867 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:30:58.868 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:30:58.868 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:30:58.868 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:30:58.868 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:30:58.868 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 233224 2024-09-13T14:30:58.868 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.868 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:30:58.868 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.868 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:30:58.868 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.667956+0000", 2024-09-13T14:30:58.868 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:30:58.868 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.868 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.869 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:30:58.869 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.667956+0000", 2024-09-13T14:30:58.869 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:30:58.869 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.869 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.869 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:30:58.869 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.667955+0000", 2024-09-13T14:30:58.869 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:30:58.869 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.869 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.869 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:30:58.869 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.667961+0000", 2024-09-13T14:30:58.869 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5010000000000001e-06 2024-09-13T14:30:58.869 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.870 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.870 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:30:58.870 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.667973+0000", 2024-09-13T14:30:58.870 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1996e-05 2024-09-13T14:30:58.870 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.870 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.870 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:30:58.870 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.667981+0000", 2024-09-13T14:30:58.870 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4080000000000001e-06 2024-09-13T14:30:58.870 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.870 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.870 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:30:58.870 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.668039+0000", 2024-09-13T14:30:58.870 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8029000000000003e-05 2024-09-13T14:30:58.871 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.871 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.871 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:30:58.871 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.668149+0000", 2024-09-13T14:30:58.871 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000110889 2024-09-13T14:30:58.871 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.871 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.871 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:30:58.871 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.668277+0000", 2024-09-13T14:30:58.871 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000127541 2024-09-13T14:30:58.871 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.871 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.871 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:58.871 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.672498+0000", 2024-09-13T14:30:58.872 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0042207640000000001 2024-09-13T14:30:58.872 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.872 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.872 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:30:58.872 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.672514+0000", 2024-09-13T14:30:58.872 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6135999999999999e-05 2024-09-13T14:30:58.872 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.872 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.872 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:30:58.872 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.824368+0000", 2024-09-13T14:30:58.872 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.15185453600000001 2024-09-13T14:30:58.872 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.872 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.873 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:30:58.873 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.824416+0000", 2024-09-13T14:30:58.873 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7236e-05 2024-09-13T14:30:58.873 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.873 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.873 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:30:58.873 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.824452+0000", 2024-09-13T14:30:58.873 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6653000000000003e-05 2024-09-13T14:30:58.873 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:30:58.873 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:30:58.873 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:30:58.873 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.873 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.873 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.44546.0:233211 5.f e154/116 5:fa93aecc:::benchmark_data_smithi102_155989_object115293:head v 154'14525, mlcod=154'14520)", 2024-09-13T14:30:58.874 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:41.668158+0000", 2024-09-13T14:30:58.874 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 17.082175127999999, 2024-09-13T14:30:58.874 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.13524302999999999, 2024-09-13T14:30:58.874 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:30:58.874 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:30:58.874 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:30:58.874 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.874 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:30:58.874 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.668158+0000", 2024-09-13T14:30:58.874 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:30:58.874 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.874 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.874 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:30:58.874 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.668158+0000", 2024-09-13T14:30:58.875 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:30:58.875 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.875 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.875 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:30:58.875 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.668158+0000", 2024-09-13T14:30:58.875 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:30:58.875 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.875 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.875 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:30:58.875 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.668162+0000", 2024-09-13T14:30:58.875 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1019999999999999e-06 2024-09-13T14:30:58.875 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.875 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.875 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:30:58.876 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.668168+0000", 2024-09-13T14:30:58.876 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.327e-06 2024-09-13T14:30:58.876 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.876 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.876 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:30:58.876 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.668171+0000", 2024-09-13T14:30:58.876 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8160000000000004e-06 2024-09-13T14:30:58.876 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.876 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.876 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:30:58.876 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.668259+0000", 2024-09-13T14:30:58.876 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7798000000000002e-05 2024-09-13T14:30:58.876 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.876 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.877 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:30:58.877 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.668277+0000", 2024-09-13T14:30:58.877 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7402999999999999e-05 2024-09-13T14:30:58.877 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.877 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.877 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:30:58.877 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.803363+0000", 2024-09-13T14:30:58.877 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.13508611200000001 2024-09-13T14:30:58.877 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.877 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.877 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:30:58.877 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.803402+0000", 2024-09-13T14:30:58.877 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8763999999999997e-05 2024-09-13T14:30:58.878 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:30:58.878 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:30:58.878 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:30:58.878 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.878 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.878 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.44546.0:233213 5.7 e154/116 5:ef469b63:::benchmark_data_smithi102_155989_object115295:head v 154'14788, mlcod=154'14786)", 2024-09-13T14:30:58.878 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:41.668182+0000", 2024-09-13T14:30:58.878 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 17.082151790000001, 2024-09-13T14:30:58.878 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.137443175, 2024-09-13T14:30:58.878 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:30:58.878 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:30:58.878 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:30:58.878 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.878 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:30:58.879 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.668182+0000", 2024-09-13T14:30:58.879 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:30:58.879 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.879 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.879 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:30:58.879 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.668182+0000", 2024-09-13T14:30:58.879 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:30:58.879 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.879 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.879 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:30:58.879 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.668182+0000", 2024-09-13T14:30:58.879 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967296 2024-09-13T14:30:58.879 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.879 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.880 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:30:58.880 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.668185+0000", 2024-09-13T14:30:58.880 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0680000000000001e-06 2024-09-13T14:30:58.880 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.880 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.880 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:30:58.880 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.668190+0000", 2024-09-13T14:30:58.880 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9620000000000001e-06 2024-09-13T14:30:58.880 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.880 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.880 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:30:58.880 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.668193+0000", 2024-09-13T14:30:58.880 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4659999999999999e-06 2024-09-13T14:30:58.881 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.881 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.881 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:30:58.881 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.669012+0000", 2024-09-13T14:30:58.881 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000818335 2024-09-13T14:30:58.881 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.881 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.881 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:30:58.881 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.669030+0000", 2024-09-13T14:30:58.881 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8102999999999999e-05 2024-09-13T14:30:58.881 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.881 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.881 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:30:58.881 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.803406+0000", 2024-09-13T14:30:58.882 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.13437612199999999 2024-09-13T14:30:58.882 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.882 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.882 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:30:58.882 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.805625+0000", 2024-09-13T14:30:58.882 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0022192700000000002 2024-09-13T14:30:58.882 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:30:58.882 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:30:58.882 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:30:58.882 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.883 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.883 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDPGRecoveryDeleteReply(5.a e154,125 5:51031a25:::benchmark_data_smithi102_155989_object5896:head,132'7840)", 2024-09-13T14:30:58.883 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:41.675340+0000", 2024-09-13T14:30:58.883 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 17.074993300999999, 2024-09-13T14:30:58.883 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.141782088, 2024-09-13T14:30:58.883 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:30:58.883 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-13T14:30:58.883 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:30:58.883 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.883 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:30:58.883 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.675340+0000", 2024-09-13T14:30:58.883 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:30:58.883 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.883 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.884 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:30:58.884 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.675340+0000", 2024-09-13T14:30:58.884 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:30:58.884 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.884 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.884 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:30:58.884 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.675338+0000", 2024-09-13T14:30:58.884 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:30:58.884 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.884 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.884 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:30:58.884 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.675350+0000", 2024-09-13T14:30:58.884 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1246999999999999e-05 2024-09-13T14:30:58.885 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.885 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.885 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:30:58.885 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.803436+0000", 2024-09-13T14:30:58.885 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.12808615000000001 2024-09-13T14:30:58.885 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.885 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.885 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:30:58.885 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.803449+0000", 2024-09-13T14:30:58.885 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2764e-05 2024-09-13T14:30:58.885 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.885 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.885 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:30:58.885 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.815507+0000", 2024-09-13T14:30:58.886 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012058615999999999 2024-09-13T14:30:58.886 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:30:58.886 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:30:58.886 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:30:58.886 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.817122+0000", 2024-09-13T14:30:58.886 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0016151550000000001 2024-09-13T14:30:58.886 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:30:58.886 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:30:58.886 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:30:58.886 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:30:58.886 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:30:58.886 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:30:58.907 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:58.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:58.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:30:59.009 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:59.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:59.031 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:59.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:59.132 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:59.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:59.154 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:59.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:59.211 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:58 smithi195 ceph-mon[104262]: pgmap v477: 73 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 4.6 MiB data, 1015 MiB used, 535 GiB / 536 GiB avail; 12/84 objects degraded (14.286%) 2024-09-13T14:30:59.211 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:58 smithi195 ceph-mon[104262]: osdmap e159: 8 total, 8 up, 6 in 2024-09-13T14:30:59.211 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:58 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 12/84 objects degraded (14.286%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:30:59.255 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:30:59.277 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:59.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:59.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:30:59.373 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:58 smithi102 ceph-mon[110183]: pgmap v477: 73 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 4.6 MiB data, 1015 MiB used, 535 GiB / 536 GiB avail; 12/84 objects degraded (14.286%) 2024-09-13T14:30:59.373 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:58 smithi102 ceph-mon[110183]: osdmap e159: 8 total, 8 up, 6 in 2024-09-13T14:30:59.373 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:58 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 12/84 objects degraded (14.286%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:30:59.373 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:58 smithi102 ceph-mon[115269]: pgmap v477: 73 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 4.6 MiB data, 1015 MiB used, 535 GiB / 536 GiB avail; 12/84 objects degraded (14.286%) 2024-09-13T14:30:59.373 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:58 smithi102 ceph-mon[115269]: osdmap e159: 8 total, 8 up, 6 in 2024-09-13T14:30:59.373 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:58 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 12/84 objects degraded (14.286%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:30:59.379 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:30:59.401 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:59.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:59.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:30:59.412 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:30:59.412 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:30:59.503 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:59.523 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:59.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:59.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:59.625 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:30:59.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:59.647 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:59.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:30:59.743 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T14:30:59.747 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:30:59.807 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:30:59.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:59.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:30:59.909 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:30:59.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:30:59.933 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:30:59.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:30:59.943 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:30:59.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:30:59 smithi195 ceph-mon[104262]: osdmap e160: 8 total, 8 up, 6 in 2024-09-13T14:31:00.035 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:00.056 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:00.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:00.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:00.158 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:00.179 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:00.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:00.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:00.281 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:31:00.303 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:00.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:00.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:31:00.353 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:30:59 smithi102 ceph-mon[115269]: osdmap e160: 8 total, 8 up, 6 in 2024-09-13T14:31:00.353 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:30:59 smithi102 ceph-mon[110183]: osdmap e160: 8 total, 8 up, 6 in 2024-09-13T14:31:00.405 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:00.426 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:00.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:00.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:00.528 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:00.538 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:31:00.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:00.551 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:00.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:00.652 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:00.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:00.674 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:00.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:00.776 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:00.797 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:00.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:00.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:00.899 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:00.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:00.921 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:00.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:01.022 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:01.044 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:01.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:01.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:01.146 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:01.167 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:01.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:01.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:01.185 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:00 smithi195 ceph-mon[104262]: pgmap v480: 73 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 4.6 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 12/84 objects degraded (14.286%) 2024-09-13T14:31:01.268 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:00 smithi102 ceph-mon[115269]: pgmap v480: 73 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 4.6 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 12/84 objects degraded (14.286%) 2024-09-13T14:31:01.268 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:00 smithi102 ceph-mon[110183]: pgmap v480: 73 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 4.6 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 12/84 objects degraded (14.286%) 2024-09-13T14:31:01.269 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:01.290 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:01.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:01.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:01.392 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:31:01.402 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:31:01.402 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-13T14:31:01.403 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 0 2024-09-13T14:31:01.403 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:31:01.417 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:01.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:01.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:31:01.519 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:01.541 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:01.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:01.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:01.643 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:01.664 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:01.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:01.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:01.766 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:01.788 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:01.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:01.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:01.890 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:01.912 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:01.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:01.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:01.975 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T14:31:02.014 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:02.035 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:02.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:02.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:02.111 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:31:02.112 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:31:02.137 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:02.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:02.158 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:02.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:02.174 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:31:02.260 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:31:02.281 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:02.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:02.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:31:02.383 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:02.405 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:02.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:02.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:02.507 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:31:02.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:02.529 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:02.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:31:02.630 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:31:02.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:02.653 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:02.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:31:02.754 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:02.778 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:02.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:02.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:02.879 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:31:02.902 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:02.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:02.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:31:03.004 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:03.025 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:03.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:03.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:03.123 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:02 smithi102 ceph-mon[110183]: pgmap v481: 73 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 4.6 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 12/84 objects degraded (14.286%) 2024-09-13T14:31:03.123 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:02 smithi102 ceph-mon[110183]: from='client.44942 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:31:03.123 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:02 smithi102 ceph-mon[115269]: pgmap v481: 73 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 4.6 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 12/84 objects degraded (14.286%) 2024-09-13T14:31:03.124 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:02 smithi102 ceph-mon[115269]: from='client.44942 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:31:03.127 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:31:03.149 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:03.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:03.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:31:03.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:02 smithi195 ceph-mon[104262]: pgmap v481: 73 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 4.6 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 12/84 objects degraded (14.286%) 2024-09-13T14:31:03.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:02 smithi195 ceph-mon[104262]: from='client.44942 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:31:03.251 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:03.273 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:03.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:03.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:03.374 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:03.396 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:03.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:03.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:03.497 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:03.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:03.520 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:03.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:03.621 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:03.642 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:03.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:03.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:03.744 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:03.768 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:03.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:03.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:03.870 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:03.890 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:03.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:03.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:03.908 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:31:03.908 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-13T14:31:03.908 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-09-13T14:31:03.908 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-09-13T14:31:03.908 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:31:03.940 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is unset 2024-09-13T14:31:03.943 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'unique_pool_1' created 2024-09-13T14:31:03.992 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:04.013 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:04.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:04.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:04.115 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:04.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:04.138 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:04.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:04.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:03 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2067356997' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-13T14:31:04.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:03 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-13T14:31:04.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:03 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3941422491' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:31:04.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:03 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2067356997' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-13T14:31:04.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:03 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-13T14:31:04.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:03 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3941422491' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:31:04.213 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":481,"stamp":"2024-09-13T14:31:01.323135+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790245,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":6390,"num_read_kb":13845,"num_write":1123,"num_write_kb":15041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":5966043,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1182,"ondisk_log_size":1182,"up":219,"acting":218,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":1057580,"kb_used_data":24600,"kb_used_omap":19,"kb_used_meta":1032876,"kb_avail":561413332,"statfs":{"total":575970213888,"available":574887251968,"internally_reserved":0,"allocated":25190400,"data_stored":20472399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20284,"internal_metadata":1057665220},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-15045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-235832,"num_write_kb":-7546561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-15276,"num_bytes_recovered":-572391424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.422373"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":188,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:30:57.927693+0000","last_change":"2024-09-13T14:27:12.839984+0000","last_active":"2024-09-13T14:30:57.927693+0000","last_peered":"2024-09-13T14:30:57.927693+0000","last_clean":"2024-09-13T14:30:57.927693+0000","last_became_active":"2024-09-13T14:27:12.839653+0000","last_became_peered":"2024-09-13T14:27:12.839653+0000","last_unstale":"2024-09-13T14:30:57.927693+0000","last_undegraded":"2024-09-13T14:30:57.927693+0000","last_fullsized":"2024-09-13T14:30:57.927693+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:14:13.672830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":139,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:30:54.903546+0000","last_change":"2024-09-13T14:27:12.846760+0000","last_active":"2024-09-13T14:30:54.903546+0000","last_peered":"2024-09-13T14:30:54.903546+0000","last_clean":"2024-09-13T14:30:54.903546+0000","last_became_active":"2024-09-13T14:27:12.838041+0000","last_became_peered":"2024-09-13T14:27:12.838041+0000","last_unstale":"2024-09-13T14:30:54.903546+0000","last_undegraded":"2024-09-13T14:30:54.903546+0000","last_fullsized":"2024-09-13T14:30:54.903546+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:22.402859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":121,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:30:54.898420+0000","last_change":"2024-09-13T14:24:55.282407+0000","last_active":"2024-09-13T14:30:54.898420+0000","last_peered":"2024-09-13T14:30:54.898420+0000","last_clean":"2024-09-13T14:30:54.898420+0000","last_became_active":"2024-09-13T14:24:55.282083+0000","last_became_peered":"2024-09-13T14:24:55.282083+0000","last_unstale":"2024-09-13T14:30:54.898420+0000","last_undegraded":"2024-09-13T14:30:54.898420+0000","last_fullsized":"2024-09-13T14:30:54.898420+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:23:10.525707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":161,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-13T14:30:55.910789+0000","last_change":"2024-09-13T14:24:55.284433+0000","last_active":"2024-09-13T14:30:55.910789+0000","last_peered":"2024-09-13T14:30:55.910789+0000","last_clean":"2024-09-13T14:30:55.910789+0000","last_became_active":"2024-09-13T14:24:55.284143+0000","last_became_peered":"2024-09-13T14:24:55.284143+0000","last_unstale":"2024-09-13T14:30:55.910789+0000","last_undegraded":"2024-09-13T14:30:55.910789+0000","last_fullsized":"2024-09-13T14:30:55.910789+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:28.107405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":163,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-13T14:30:55.908710+0000","last_change":"2024-09-13T14:24:55.282416+0000","last_active":"2024-09-13T14:30:55.908710+0000","last_peered":"2024-09-13T14:30:55.908710+0000","last_clean":"2024-09-13T14:30:55.908710+0000","last_became_active":"2024-09-13T14:24:55.282045+0000","last_became_peered":"2024-09-13T14:24:55.282045+0000","last_unstale":"2024-09-13T14:30:55.908710+0000","last_undegraded":"2024-09-13T14:30:55.908710+0000","last_fullsized":"2024-09-13T14:30:55.908710+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:58.692897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":107,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:30:54.903306+0000","last_change":"2024-09-13T14:11:29.244452+0000","last_active":"2024-09-13T14:30:54.903306+0000","last_peered":"2024-09-13T14:30:54.903306+0000","last_clean":"2024-09-13T14:30:54.903306+0000","last_became_active":"2024-09-13T14:11:29.244102+0000","last_became_peered":"2024-09-13T14:11:29.244102+0000","last_unstale":"2024-09-13T14:30:54.903306+0000","last_undegraded":"2024-09-13T14:30:54.903306+0000","last_fullsized":"2024-09-13T14:30:54.903306+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:13:19.165534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":188,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:30:57.927523+0000","last_change":"2024-09-13T14:27:12.834300+0000","last_active":"2024-09-13T14:30:57.927523+0000","last_peered":"2024-09-13T14:30:57.927523+0000","last_clean":"2024-09-13T14:30:57.927523+0000","last_became_active":"2024-09-13T14:27:12.834046+0000","last_became_peered":"2024-09-13T14:27:12.834046+0000","last_unstale":"2024-09-13T14:30:57.927523+0000","last_undegraded":"2024-09-13T14:30:57.927523+0000","last_fullsized":"2024-09-13T14:30:57.927523+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:46:58.906862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":130,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:30:55.453291+0000","last_change":"2024-09-13T14:23:18.039061+0000","last_active":"2024-09-13T14:30:55.453291+0000","last_peered":"2024-09-13T14:30:55.453291+0000","last_clean":"2024-09-13T14:30:55.453291+0000","last_became_active":"2024-09-13T14:23:18.038661+0000","last_became_peered":"2024-09-13T14:23:18.038661+0000","last_unstale":"2024-09-13T14:30:55.453291+0000","last_undegraded":"2024-09-13T14:30:55.453291+0000","last_fullsized":"2024-09-13T14:30:55.453291+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:05:29.109844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":123,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:30:54.903667+0000","last_change":"2024-09-13T14:23:18.038597+0000","last_active":"2024-09-13T14:30:54.903667+0000","last_peered":"2024-09-13T14:30:54.903667+0000","last_clean":"2024-09-13T14:30:54.903667+0000","last_became_active":"2024-09-13T14:23:18.038240+0000","last_became_peered":"2024-09-13T14:23:18.038240+0000","last_unstale":"2024-09-13T14:30:54.903667+0000","last_undegraded":"2024-09-13T14:30:54.903667+0000","last_fullsized":"2024-09-13T14:30:54.903667+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:27:51.842195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":106,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:30:54.896891+0000","last_change":"2024-09-13T14:11:30.254534+0000","last_active":"2024-09-13T14:30:54.896891+0000","last_peered":"2024-09-13T14:30:54.896891+0000","last_clean":"2024-09-13T14:30:54.896891+0000","last_became_active":"2024-09-13T14:11:30.253959+0000","last_became_peered":"2024-09-13T14:11:30.253959+0000","last_unstale":"2024-09-13T14:30:54.896891+0000","last_undegraded":"2024-09-13T14:30:54.896891+0000","last_fullsized":"2024-09-13T14:30:54.896891+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:09:54.025853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":137,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:30:54.903848+0000","last_change":"2024-09-13T14:26:15.828529+0000","last_active":"2024-09-13T14:30:54.903848+0000","last_peered":"2024-09-13T14:30:54.903848+0000","last_clean":"2024-09-13T14:30:54.903848+0000","last_became_active":"2024-09-13T14:26:15.820486+0000","last_became_peered":"2024-09-13T14:26:15.820486+0000","last_unstale":"2024-09-13T14:30:54.903848+0000","last_undegraded":"2024-09-13T14:30:54.903848+0000","last_fullsized":"2024-09-13T14:30:54.903848+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:11:46.283172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":145,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-13T14:30:55.905783+0000","last_change":"2024-09-13T14:26:15.824632+0000","last_active":"2024-09-13T14:30:55.905783+0000","last_peered":"2024-09-13T14:30:55.905783+0000","last_clean":"2024-09-13T14:30:55.905783+0000","last_became_active":"2024-09-13T14:26:15.824330+0000","last_became_peered":"2024-09-13T14:26:15.824330+0000","last_unstale":"2024-09-13T14:30:55.905783+0000","last_undegraded":"2024-09-13T14:30:55.905783+0000","last_fullsized":"2024-09-13T14:30:55.905783+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:25.288116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":108,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:30:54.903848+0000","last_change":"2024-09-13T14:11:29.232897+0000","last_active":"2024-09-13T14:30:54.903848+0000","last_peered":"2024-09-13T14:30:54.903848+0000","last_clean":"2024-09-13T14:30:54.903848+0000","last_became_active":"2024-09-13T14:11:29.232637+0000","last_became_peered":"2024-09-13T14:11:29.232637+0000","last_unstale":"2024-09-13T14:30:54.903848+0000","last_undegraded":"2024-09-13T14:30:54.903848+0000","last_fullsized":"2024-09-13T14:30:54.903848+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:31:05.856046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":130,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:30:55.453467+0000","last_change":"2024-09-13T14:26:15.822452+0000","last_active":"2024-09-13T14:30:55.453467+0000","last_peered":"2024-09-13T14:30:55.453467+0000","last_clean":"2024-09-13T14:30:55.453467+0000","last_became_active":"2024-09-13T14:26:15.822111+0000","last_became_peered":"2024-09-13T14:26:15.822111+0000","last_unstale":"2024-09-13T14:30:55.453467+0000","last_undegraded":"2024-09-13T14:30:55.453467+0000","last_fullsized":"2024-09-13T14:30:55.453467+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:32:53.078767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":122,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:30:54.898211+0000","last_change":"2024-09-13T14:24:55.290798+0000","last_active":"2024-09-13T14:30:54.898211+0000","last_peered":"2024-09-13T14:30:54.898211+0000","last_clean":"2024-09-13T14:30:54.898211+0000","last_became_active":"2024-09-13T14:24:55.290510+0000","last_became_peered":"2024-09-13T14:24:55.290510+0000","last_unstale":"2024-09-13T14:30:54.898211+0000","last_undegraded":"2024-09-13T14:30:54.898211+0000","last_fullsized":"2024-09-13T14:30:54.898211+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:19:03.129132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":170,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-13T14:30:55.908439+0000","last_change":"2024-09-13T14:26:15.823316+0000","last_active":"2024-09-13T14:30:55.908439+0000","last_peered":"2024-09-13T14:30:55.908439+0000","last_clean":"2024-09-13T14:30:55.908439+0000","last_became_active":"2024-09-13T14:26:15.822980+0000","last_became_peered":"2024-09-13T14:26:15.822980+0000","last_unstale":"2024-09-13T14:30:55.908439+0000","last_undegraded":"2024-09-13T14:30:55.908439+0000","last_fullsized":"2024-09-13T14:30:55.908439+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:33:47.420548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":172,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:30:57.927023+0000","last_change":"2024-09-13T14:24:55.284499+0000","last_active":"2024-09-13T14:30:57.927023+0000","last_peered":"2024-09-13T14:30:57.927023+0000","last_clean":"2024-09-13T14:30:57.927023+0000","last_became_active":"2024-09-13T14:24:55.284096+0000","last_became_peered":"2024-09-13T14:24:55.284096+0000","last_unstale":"2024-09-13T14:30:57.927023+0000","last_undegraded":"2024-09-13T14:30:57.927023+0000","last_fullsized":"2024-09-13T14:30:57.927023+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:02:06.836099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":121,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-13T14:30:55.907238+0000","last_change":"2024-09-13T14:24:55.291371+0000","last_active":"2024-09-13T14:30:55.907238+0000","last_peered":"2024-09-13T14:30:55.907238+0000","last_clean":"2024-09-13T14:30:55.907238+0000","last_became_active":"2024-09-13T14:24:55.291047+0000","last_became_peered":"2024-09-13T14:24:55.291047+0000","last_unstale":"2024-09-13T14:30:55.907238+0000","last_undegraded":"2024-09-13T14:30:55.907238+0000","last_fullsized":"2024-09-13T14:30:55.907238+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:43.912412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":131,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:30:55.453630+0000","last_change":"2024-09-13T14:26:15.827333+0000","last_active":"2024-09-13T14:30:55.453630+0000","last_peered":"2024-09-13T14:30:55.453630+0000","last_clean":"2024-09-13T14:30:55.453630+0000","last_became_active":"2024-09-13T14:26:15.826945+0000","last_became_peered":"2024-09-13T14:26:15.826945+0000","last_unstale":"2024-09-13T14:30:55.453630+0000","last_undegraded":"2024-09-13T14:30:55.453630+0000","last_fullsized":"2024-09-13T14:30:55.453630+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:35:24.870052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":154,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-13T14:30:55.907239+0000","last_change":"2024-09-13T14:27:12.842604+0000","last_active":"2024-09-13T14:30:55.907239+0000","last_peered":"2024-09-13T14:30:55.907239+0000","last_clean":"2024-09-13T14:30:55.907239+0000","last_became_active":"2024-09-13T14:27:12.841643+0000","last_became_peered":"2024-09-13T14:27:12.841643+0000","last_unstale":"2024-09-13T14:30:55.907239+0000","last_undegraded":"2024-09-13T14:30:55.907239+0000","last_fullsized":"2024-09-13T14:30:55.907239+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:34:34.751790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":131,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-13T14:30:55.906051+0000","last_change":"2024-09-13T14:27:12.842271+0000","last_active":"2024-09-13T14:30:55.906051+0000","last_peered":"2024-09-13T14:30:55.906051+0000","last_clean":"2024-09-13T14:30:55.906051+0000","last_became_active":"2024-09-13T14:27:12.841479+0000","last_became_peered":"2024-09-13T14:27:12.841479+0000","last_unstale":"2024-09-13T14:30:55.906051+0000","last_undegraded":"2024-09-13T14:30:55.906051+0000","last_fullsized":"2024-09-13T14:30:55.906051+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:30:22.746575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":130,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-13T14:30:55.907988+0000","last_change":"2024-09-13T14:26:15.825304+0000","last_active":"2024-09-13T14:30:55.907988+0000","last_peered":"2024-09-13T14:30:55.907988+0000","last_clean":"2024-09-13T14:30:55.907988+0000","last_became_active":"2024-09-13T14:26:15.825018+0000","last_became_peered":"2024-09-13T14:26:15.825018+0000","last_unstale":"2024-09-13T14:30:55.907988+0000","last_undegraded":"2024-09-13T14:30:55.907988+0000","last_fullsized":"2024-09-13T14:30:55.907988+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:22:59.408928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":182,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:30:57.926776+0000","last_change":"2024-09-13T14:26:15.823898+0000","last_active":"2024-09-13T14:30:57.926776+0000","last_peered":"2024-09-13T14:30:57.926776+0000","last_clean":"2024-09-13T14:30:57.926776+0000","last_became_active":"2024-09-13T14:26:15.823125+0000","last_became_peered":"2024-09-13T14:26:15.823125+0000","last_unstale":"2024-09-13T14:30:57.926776+0000","last_undegraded":"2024-09-13T14:30:57.926776+0000","last_fullsized":"2024-09-13T14:30:57.926776+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:51:27.949528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":145,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:30:57.927606+0000","last_change":"2024-09-13T14:26:15.823198+0000","last_active":"2024-09-13T14:30:57.927606+0000","last_peered":"2024-09-13T14:30:57.927606+0000","last_clean":"2024-09-13T14:30:57.927606+0000","last_became_active":"2024-09-13T14:26:15.822717+0000","last_became_peered":"2024-09-13T14:26:15.822717+0000","last_unstale":"2024-09-13T14:30:57.927606+0000","last_undegraded":"2024-09-13T14:30:57.927606+0000","last_fullsized":"2024-09-13T14:30:57.927606+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:22.471622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":453,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:30:54.903214+0000","last_change":"2024-09-13T14:23:18.032973+0000","last_active":"2024-09-13T14:30:54.903214+0000","last_peered":"2024-09-13T14:30:54.903214+0000","last_clean":"2024-09-13T14:30:54.903214+0000","last_became_active":"2024-09-13T14:23:18.032579+0000","last_became_peered":"2024-09-13T14:23:18.032579+0000","last_unstale":"2024-09-13T14:30:54.903214+0000","last_undegraded":"2024-09-13T14:30:54.903214+0000","last_fullsized":"2024-09-13T14:30:54.903214+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:07:41.927740+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:17.109445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":486,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:30:54.897802+0000","last_change":"2024-09-13T14:27:15.865400+0000","last_active":"2024-09-13T14:30:54.897802+0000","last_peered":"2024-09-13T14:30:54.897802+0000","last_clean":"2024-09-13T14:30:54.897802+0000","last_became_active":"2024-09-13T14:27:15.864835+0000","last_became_peered":"2024-09-13T14:27:15.864835+0000","last_unstale":"2024-09-13T14:30:54.897802+0000","last_undegraded":"2024-09-13T14:30:54.897802+0000","last_fullsized":"2024-09-13T14:30:54.897802+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:27:10.928041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":123,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:30:57.926804+0000","last_change":"2024-09-13T14:24:55.288058+0000","last_active":"2024-09-13T14:30:57.926804+0000","last_peered":"2024-09-13T14:30:57.926804+0000","last_clean":"2024-09-13T14:30:57.926804+0000","last_became_active":"2024-09-13T14:24:55.287601+0000","last_became_peered":"2024-09-13T14:24:55.287601+0000","last_unstale":"2024-09-13T14:30:57.926804+0000","last_undegraded":"2024-09-13T14:30:57.926804+0000","last_fullsized":"2024-09-13T14:30:57.926804+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:56:34.622696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":107,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:30:55.453815+0000","last_change":"2024-09-13T14:11:30.253749+0000","last_active":"2024-09-13T14:30:55.453815+0000","last_peered":"2024-09-13T14:30:55.453815+0000","last_clean":"2024-09-13T14:30:55.453815+0000","last_became_active":"2024-09-13T14:11:30.253499+0000","last_became_peered":"2024-09-13T14:11:30.253499+0000","last_unstale":"2024-09-13T14:30:55.453815+0000","last_undegraded":"2024-09-13T14:30:55.453815+0000","last_fullsized":"2024-09-13T14:30:55.453815+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:02:00.987335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":368,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:30:54.903320+0000","last_change":"2024-09-13T14:26:15.835178+0000","last_active":"2024-09-13T14:30:54.903320+0000","last_peered":"2024-09-13T14:30:54.903320+0000","last_clean":"2024-09-13T14:30:54.903320+0000","last_became_active":"2024-09-13T14:26:15.820283+0000","last_became_peered":"2024-09-13T14:26:15.820283+0000","last_unstale":"2024-09-13T14:30:54.903320+0000","last_undegraded":"2024-09-13T14:30:54.903320+0000","last_fullsized":"2024-09-13T14:30:54.903320+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:30.340038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":162,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-13T14:30:55.907959+0000","last_change":"2024-09-13T14:24:55.286576+0000","last_active":"2024-09-13T14:30:55.907959+0000","last_peered":"2024-09-13T14:30:55.907959+0000","last_clean":"2024-09-13T14:30:55.907959+0000","last_became_active":"2024-09-13T14:24:55.286284+0000","last_became_peered":"2024-09-13T14:24:55.286284+0000","last_unstale":"2024-09-13T14:30:55.907959+0000","last_undegraded":"2024-09-13T14:30:55.907959+0000","last_fullsized":"2024-09-13T14:30:55.907959+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:35.589736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":188,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-13T14:30:55.908936+0000","last_change":"2024-09-13T14:27:12.836204+0000","last_active":"2024-09-13T14:30:55.908936+0000","last_peered":"2024-09-13T14:30:55.908936+0000","last_clean":"2024-09-13T14:30:55.908936+0000","last_became_active":"2024-09-13T14:27:12.835900+0000","last_became_peered":"2024-09-13T14:27:12.835900+0000","last_unstale":"2024-09-13T14:30:55.908936+0000","last_undegraded":"2024-09-13T14:30:55.908936+0000","last_fullsized":"2024-09-13T14:30:55.908936+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:36:52.121730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":546,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:30:57.927347+0000","last_change":"2024-09-13T14:27:15.863966+0000","last_active":"2024-09-13T14:30:57.927347+0000","last_peered":"2024-09-13T14:30:57.927347+0000","last_clean":"2024-09-13T14:30:57.927347+0000","last_became_active":"2024-09-13T14:27:15.863124+0000","last_became_peered":"2024-09-13T14:27:15.863124+0000","last_unstale":"2024-09-13T14:30:57.927347+0000","last_undegraded":"2024-09-13T14:30:57.927347+0000","last_fullsized":"2024-09-13T14:30:57.927347+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:35:02.715013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":81,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:30:55.453263+0000","last_change":"2024-09-13T14:27:12.829079+0000","last_active":"2024-09-13T14:30:55.453263+0000","last_peered":"2024-09-13T14:30:55.453263+0000","last_clean":"2024-09-13T14:30:55.453263+0000","last_became_active":"2024-09-13T14:27:12.828619+0000","last_became_peered":"2024-09-13T14:27:12.828619+0000","last_unstale":"2024-09-13T14:30:55.453263+0000","last_undegraded":"2024-09-13T14:30:55.453263+0000","last_fullsized":"2024-09-13T14:30:55.453263+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:39:08.260304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":118,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:30:54.898535+0000","last_change":"2024-09-13T14:26:15.817732+0000","last_active":"2024-09-13T14:30:54.898535+0000","last_peered":"2024-09-13T14:30:54.898535+0000","last_clean":"2024-09-13T14:30:54.898535+0000","last_became_active":"2024-09-13T14:26:15.817403+0000","last_became_peered":"2024-09-13T14:26:15.817403+0000","last_unstale":"2024-09-13T14:30:54.898535+0000","last_undegraded":"2024-09-13T14:30:54.898535+0000","last_fullsized":"2024-09-13T14:30:54.898535+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:56:39.961300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":917,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:30:54.897412+0000","last_change":"2024-09-13T14:26:18.829131+0000","last_active":"2024-09-13T14:30:54.897412+0000","last_peered":"2024-09-13T14:30:54.897412+0000","last_clean":"2024-09-13T14:30:54.897412+0000","last_became_active":"2024-09-13T14:26:18.828619+0000","last_became_peered":"2024-09-13T14:26:18.828619+0000","last_unstale":"2024-09-13T14:30:54.897412+0000","last_undegraded":"2024-09-13T14:30:54.897412+0000","last_fullsized":"2024-09-13T14:30:54.897412+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:24:37.073618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":123,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-13T14:30:55.907670+0000","last_change":"2024-09-13T14:23:18.042891+0000","last_active":"2024-09-13T14:30:55.907670+0000","last_peered":"2024-09-13T14:30:55.907670+0000","last_clean":"2024-09-13T14:30:55.907670+0000","last_became_active":"2024-09-13T14:23:18.042562+0000","last_became_peered":"2024-09-13T14:23:18.042562+0000","last_unstale":"2024-09-13T14:30:55.907670+0000","last_undegraded":"2024-09-13T14:30:55.907670+0000","last_fullsized":"2024-09-13T14:30:55.907670+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:30:42.996661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":159,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:30:57.927661+0000","last_change":"2024-09-13T14:27:12.832666+0000","last_active":"2024-09-13T14:30:57.927661+0000","last_peered":"2024-09-13T14:30:57.927661+0000","last_clean":"2024-09-13T14:30:57.927661+0000","last_became_active":"2024-09-13T14:27:12.832332+0000","last_became_peered":"2024-09-13T14:27:12.832332+0000","last_unstale":"2024-09-13T14:30:57.927661+0000","last_undegraded":"2024-09-13T14:30:57.927661+0000","last_fullsized":"2024-09-13T14:30:57.927661+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:07:17.724156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":147,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:30:54.898508+0000","last_change":"2024-09-13T14:27:12.843038+0000","last_active":"2024-09-13T14:30:54.898508+0000","last_peered":"2024-09-13T14:30:54.898508+0000","last_clean":"2024-09-13T14:30:54.898508+0000","last_became_active":"2024-09-13T14:27:12.842663+0000","last_became_peered":"2024-09-13T14:27:12.842663+0000","last_unstale":"2024-09-13T14:30:54.898508+0000","last_undegraded":"2024-09-13T14:30:54.898508+0000","last_fullsized":"2024-09-13T14:30:54.898508+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:23:38.880811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":859,"reported_epoch":157,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:30:55.908479+0000","last_change":"2024-09-13T14:27:14.029991+0000","last_active":"2024-09-13T14:30:55.908479+0000","last_peered":"2024-09-13T14:30:55.908479+0000","last_clean":"2024-09-13T14:27:08.815347+0000","last_became_active":"2024-09-13T14:27:13.888039+0000","last_became_peered":"2024-09-13T14:27:13.888039+0000","last_unstale":"2024-09-13T14:30:55.908479+0000","last_undegraded":"2024-09-13T14:27:13.826593+0000","last_fullsized":"2024-09-13T14:27:13.824237+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7],"avail_no_missing":["0","2","7"],"object_location_counts":[{"shards":"0,2,7","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":74,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:30:54.897210+0000","last_change":"2024-09-13T14:26:15.819032+0000","last_active":"2024-09-13T14:30:54.897210+0000","last_peered":"2024-09-13T14:30:54.897210+0000","last_clean":"2024-09-13T14:30:54.897210+0000","last_became_active":"2024-09-13T14:26:15.818675+0000","last_became_peered":"2024-09-13T14:26:15.818675+0000","last_unstale":"2024-09-13T14:30:54.897210+0000","last_undegraded":"2024-09-13T14:30:54.897210+0000","last_fullsized":"2024-09-13T14:30:54.897210+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:37:13.628256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":81,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:30:54.898043+0000","last_change":"2024-09-13T14:27:12.843268+0000","last_active":"2024-09-13T14:30:54.898043+0000","last_peered":"2024-09-13T14:30:54.898043+0000","last_clean":"2024-09-13T14:30:54.898043+0000","last_became_active":"2024-09-13T14:27:12.842867+0000","last_became_peered":"2024-09-13T14:27:12.842867+0000","last_unstale":"2024-09-13T14:30:54.898043+0000","last_undegraded":"2024-09-13T14:30:54.898043+0000","last_fullsized":"2024-09-13T14:30:54.898043+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:03:59.575866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":644,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:30:54.903931+0000","last_change":"2024-09-13T14:23:18.038847+0000","last_active":"2024-09-13T14:30:54.903931+0000","last_peered":"2024-09-13T14:30:54.903931+0000","last_clean":"2024-09-13T14:30:54.903931+0000","last_became_active":"2024-09-13T14:23:18.038188+0000","last_became_peered":"2024-09-13T14:23:18.038188+0000","last_unstale":"2024-09-13T14:30:54.903931+0000","last_undegraded":"2024-09-13T14:30:54.903931+0000","last_fullsized":"2024-09-13T14:30:54.903931+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:30:37.213496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":152,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-13T14:30:55.908344+0000","last_change":"2024-09-13T14:23:18.034427+0000","last_active":"2024-09-13T14:30:55.908344+0000","last_peered":"2024-09-13T14:30:55.908344+0000","last_clean":"2024-09-13T14:30:55.908344+0000","last_became_active":"2024-09-13T14:23:18.034092+0000","last_became_peered":"2024-09-13T14:23:18.034092+0000","last_unstale":"2024-09-13T14:30:55.908344+0000","last_undegraded":"2024-09-13T14:30:55.908344+0000","last_fullsized":"2024-09-13T14:30:55.908344+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:21.755115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":172,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:30:57.927626+0000","last_change":"2024-09-13T14:24:55.290711+0000","last_active":"2024-09-13T14:30:57.927626+0000","last_peered":"2024-09-13T14:30:57.927626+0000","last_clean":"2024-09-13T14:30:57.927626+0000","last_became_active":"2024-09-13T14:24:55.288549+0000","last_became_peered":"2024-09-13T14:24:55.288549+0000","last_unstale":"2024-09-13T14:30:57.927626+0000","last_undegraded":"2024-09-13T14:30:57.927626+0000","last_fullsized":"2024-09-13T14:30:57.927626+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:55:21.763931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":558,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:30:55.453643+0000","last_change":"2024-09-13T14:26:18.827876+0000","last_active":"2024-09-13T14:30:55.453643+0000","last_peered":"2024-09-13T14:30:55.453643+0000","last_clean":"2024-09-13T14:30:55.453643+0000","last_became_active":"2024-09-13T14:26:18.827317+0000","last_became_peered":"2024-09-13T14:26:18.827317+0000","last_unstale":"2024-09-13T14:30:55.453643+0000","last_undegraded":"2024-09-13T14:30:55.453643+0000","last_fullsized":"2024-09-13T14:30:55.453643+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:31:48.611286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":170,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:30:57.926670+0000","last_change":"2024-09-13T14:24:55.281593+0000","last_active":"2024-09-13T14:30:57.926670+0000","last_peered":"2024-09-13T14:30:57.926670+0000","last_clean":"2024-09-13T14:30:57.926670+0000","last_became_active":"2024-09-13T14:24:55.280805+0000","last_became_peered":"2024-09-13T14:24:55.280805+0000","last_unstale":"2024-09-13T14:30:57.926670+0000","last_undegraded":"2024-09-13T14:30:57.926670+0000","last_fullsized":"2024-09-13T14:30:57.926670+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:14.806713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":177,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-13T14:30:55.908614+0000","last_change":"2024-09-13T14:26:15.830309+0000","last_active":"2024-09-13T14:30:55.908614+0000","last_peered":"2024-09-13T14:30:55.908614+0000","last_clean":"2024-09-13T14:30:55.908614+0000","last_became_active":"2024-09-13T14:26:15.823041+0000","last_became_peered":"2024-09-13T14:26:15.823041+0000","last_unstale":"2024-09-13T14:30:55.908614+0000","last_undegraded":"2024-09-13T14:30:55.908614+0000","last_fullsized":"2024-09-13T14:30:55.908614+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:23:08.158576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":588,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:30:55.453926+0000","last_change":"2024-09-13T14:27:12.848354+0000","last_active":"2024-09-13T14:30:55.453926+0000","last_peered":"2024-09-13T14:30:55.453926+0000","last_clean":"2024-09-13T14:30:55.453926+0000","last_became_active":"2024-09-13T14:27:12.832182+0000","last_became_peered":"2024-09-13T14:27:12.832182+0000","last_unstale":"2024-09-13T14:30:55.453926+0000","last_undegraded":"2024-09-13T14:30:55.453926+0000","last_fullsized":"2024-09-13T14:30:55.453926+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:46:42.064509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":107,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-13T14:30:55.905911+0000","last_change":"2024-09-13T14:11:30.256034+0000","last_active":"2024-09-13T14:30:55.905911+0000","last_peered":"2024-09-13T14:30:55.905911+0000","last_clean":"2024-09-13T14:30:55.905911+0000","last_became_active":"2024-09-13T14:11:30.255727+0000","last_became_peered":"2024-09-13T14:11:30.255727+0000","last_unstale":"2024-09-13T14:30:55.905911+0000","last_undegraded":"2024-09-13T14:30:55.905911+0000","last_fullsized":"2024-09-13T14:30:55.905911+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:21:35.817014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":130,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-13T14:30:55.906621+0000","last_change":"2024-09-13T14:26:15.825390+0000","last_active":"2024-09-13T14:30:55.906621+0000","last_peered":"2024-09-13T14:30:55.906621+0000","last_clean":"2024-09-13T14:30:55.906621+0000","last_became_active":"2024-09-13T14:26:15.825084+0000","last_became_peered":"2024-09-13T14:26:15.825084+0000","last_unstale":"2024-09-13T14:30:55.906621+0000","last_undegraded":"2024-09-13T14:30:55.906621+0000","last_fullsized":"2024-09-13T14:30:55.906621+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:44.680451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":130,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-13T14:30:55.907335+0000","last_change":"2024-09-13T14:11:29.237013+0000","last_active":"2024-09-13T14:30:55.907335+0000","last_peered":"2024-09-13T14:30:55.907335+0000","last_clean":"2024-09-13T14:30:55.907335+0000","last_became_active":"2024-09-13T14:11:29.236643+0000","last_became_peered":"2024-09-13T14:11:29.236643+0000","last_unstale":"2024-09-13T14:30:55.907335+0000","last_undegraded":"2024-09-13T14:30:55.907335+0000","last_fullsized":"2024-09-13T14:30:55.907335+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:33:58.438693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":104,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-13T14:30:55.902392+0000","last_change":"2024-09-13T14:27:12.843904+0000","last_active":"2024-09-13T14:30:55.902392+0000","last_peered":"2024-09-13T14:30:55.902392+0000","last_clean":"2024-09-13T14:30:55.902392+0000","last_became_active":"2024-09-13T14:27:12.842566+0000","last_became_peered":"2024-09-13T14:27:12.842566+0000","last_unstale":"2024-09-13T14:30:55.902392+0000","last_undegraded":"2024-09-13T14:30:55.902392+0000","last_fullsized":"2024-09-13T14:30:55.902392+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:11:46.253003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":125,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:30:54.903722+0000","last_change":"2024-09-13T14:24:55.287275+0000","last_active":"2024-09-13T14:30:54.903722+0000","last_peered":"2024-09-13T14:30:54.903722+0000","last_clean":"2024-09-13T14:30:54.903722+0000","last_became_active":"2024-09-13T14:24:55.286919+0000","last_became_peered":"2024-09-13T14:24:55.286919+0000","last_unstale":"2024-09-13T14:30:54.903722+0000","last_undegraded":"2024-09-13T14:30:54.903722+0000","last_fullsized":"2024-09-13T14:30:54.903722+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:10.790711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":131,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:30:54.902086+0000","last_change":"2024-09-13T14:26:15.819037+0000","last_active":"2024-09-13T14:30:54.902086+0000","last_peered":"2024-09-13T14:30:54.902086+0000","last_clean":"2024-09-13T14:30:54.902086+0000","last_became_active":"2024-09-13T14:26:15.818721+0000","last_became_peered":"2024-09-13T14:26:15.818721+0000","last_unstale":"2024-09-13T14:30:54.902086+0000","last_undegraded":"2024-09-13T14:30:54.902086+0000","last_fullsized":"2024-09-13T14:30:54.902086+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:44:06.419136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":125,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:30:57.927240+0000","last_change":"2024-09-13T14:24:55.287509+0000","last_active":"2024-09-13T14:30:57.927240+0000","last_peered":"2024-09-13T14:30:57.927240+0000","last_clean":"2024-09-13T14:30:57.927240+0000","last_became_active":"2024-09-13T14:24:55.287193+0000","last_became_peered":"2024-09-13T14:24:55.287193+0000","last_unstale":"2024-09-13T14:30:57.927240+0000","last_undegraded":"2024-09-13T14:30:57.927240+0000","last_fullsized":"2024-09-13T14:30:57.927240+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:14:10.603307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":162,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-13T14:30:55.909073+0000","last_change":"2024-09-13T14:24:55.289055+0000","last_active":"2024-09-13T14:30:55.909073+0000","last_peered":"2024-09-13T14:30:55.909073+0000","last_clean":"2024-09-13T14:30:55.909073+0000","last_became_active":"2024-09-13T14:24:55.288762+0000","last_became_peered":"2024-09-13T14:24:55.288762+0000","last_unstale":"2024-09-13T14:30:55.909073+0000","last_undegraded":"2024-09-13T14:30:55.909073+0000","last_fullsized":"2024-09-13T14:30:55.909073+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:37:26.639877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":181,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:30:57.926663+0000","last_change":"2024-09-13T14:26:15.824540+0000","last_active":"2024-09-13T14:30:57.926663+0000","last_peered":"2024-09-13T14:30:57.926663+0000","last_clean":"2024-09-13T14:30:57.926663+0000","last_became_active":"2024-09-13T14:26:15.824118+0000","last_became_peered":"2024-09-13T14:26:15.824118+0000","last_unstale":"2024-09-13T14:30:57.926663+0000","last_undegraded":"2024-09-13T14:30:57.926663+0000","last_fullsized":"2024-09-13T14:30:57.926663+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:56.677401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":131,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:30:55.453860+0000","last_change":"2024-09-13T14:26:15.823892+0000","last_active":"2024-09-13T14:30:55.453860+0000","last_peered":"2024-09-13T14:30:55.453860+0000","last_clean":"2024-09-13T14:30:55.453860+0000","last_became_active":"2024-09-13T14:26:15.823582+0000","last_became_peered":"2024-09-13T14:26:15.823582+0000","last_unstale":"2024-09-13T14:30:55.453860+0000","last_undegraded":"2024-09-13T14:30:55.453860+0000","last_fullsized":"2024-09-13T14:30:55.453860+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:53:09.918997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":204,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:30:57.926759+0000","last_change":"2024-09-13T14:27:12.830642+0000","last_active":"2024-09-13T14:30:57.926759+0000","last_peered":"2024-09-13T14:30:57.926759+0000","last_clean":"2024-09-13T14:30:57.926759+0000","last_became_active":"2024-09-13T14:27:12.830026+0000","last_became_peered":"2024-09-13T14:27:12.830026+0000","last_unstale":"2024-09-13T14:30:57.926759+0000","last_undegraded":"2024-09-13T14:30:57.926759+0000","last_fullsized":"2024-09-13T14:30:57.926759+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:12:31.783552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":163,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-13T14:30:55.908856+0000","last_change":"2024-09-13T14:24:55.283070+0000","last_active":"2024-09-13T14:30:55.908856+0000","last_peered":"2024-09-13T14:30:55.908856+0000","last_clean":"2024-09-13T14:30:55.908856+0000","last_became_active":"2024-09-13T14:24:55.282591+0000","last_became_peered":"2024-09-13T14:24:55.282591+0000","last_unstale":"2024-09-13T14:30:55.908856+0000","last_undegraded":"2024-09-13T14:30:55.908856+0000","last_fullsized":"2024-09-13T14:30:55.908856+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:35:21.896294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":153,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:30:57.927336+0000","last_change":"2024-09-13T14:26:15.823524+0000","last_active":"2024-09-13T14:30:57.927336+0000","last_peered":"2024-09-13T14:30:57.927336+0000","last_clean":"2024-09-13T14:30:57.927336+0000","last_became_active":"2024-09-13T14:26:15.823162+0000","last_became_peered":"2024-09-13T14:26:15.823162+0000","last_unstale":"2024-09-13T14:30:57.927336+0000","last_undegraded":"2024-09-13T14:30:57.927336+0000","last_fullsized":"2024-09-13T14:30:57.927336+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:17:13.664812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":194,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:30:57.926911+0000","last_change":"2024-09-13T14:26:15.823703+0000","last_active":"2024-09-13T14:30:57.926911+0000","last_peered":"2024-09-13T14:30:57.926911+0000","last_clean":"2024-09-13T14:30:57.926911+0000","last_became_active":"2024-09-13T14:26:15.823249+0000","last_became_peered":"2024-09-13T14:26:15.823249+0000","last_unstale":"2024-09-13T14:30:57.926911+0000","last_undegraded":"2024-09-13T14:30:57.926911+0000","last_fullsized":"2024-09-13T14:30:57.926911+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:42:40.209150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":155,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-13T14:30:55.906998+0000","last_change":"2024-09-13T14:27:12.841881+0000","last_active":"2024-09-13T14:30:55.906998+0000","last_peered":"2024-09-13T14:30:55.906998+0000","last_clean":"2024-09-13T14:30:55.906998+0000","last_became_active":"2024-09-13T14:27:12.841247+0000","last_became_peered":"2024-09-13T14:27:12.841247+0000","last_unstale":"2024-09-13T14:30:55.906998+0000","last_undegraded":"2024-09-13T14:30:55.906998+0000","last_fullsized":"2024-09-13T14:30:55.906998+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:59:16.937848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":177,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:30:57.927759+0000","last_change":"2024-09-13T14:27:12.838453+0000","last_active":"2024-09-13T14:30:57.927759+0000","last_peered":"2024-09-13T14:30:57.927759+0000","last_clean":"2024-09-13T14:30:57.927759+0000","last_became_active":"2024-09-13T14:27:12.838124+0000","last_became_peered":"2024-09-13T14:27:12.838124+0000","last_unstale":"2024-09-13T14:30:57.927759+0000","last_undegraded":"2024-09-13T14:30:57.927759+0000","last_fullsized":"2024-09-13T14:30:57.927759+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:24.259191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":155,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-13T14:30:55.908246+0000","last_change":"2024-09-13T14:23:18.035101+0000","last_active":"2024-09-13T14:30:55.908246+0000","last_peered":"2024-09-13T14:30:55.908246+0000","last_clean":"2024-09-13T14:30:55.908246+0000","last_became_active":"2024-09-13T14:23:18.034724+0000","last_became_peered":"2024-09-13T14:23:18.034724+0000","last_unstale":"2024-09-13T14:30:55.908246+0000","last_undegraded":"2024-09-13T14:30:55.908246+0000","last_fullsized":"2024-09-13T14:30:55.908246+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:39.358144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":153,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-13T14:30:55.906552+0000","last_change":"2024-09-13T14:27:12.834931+0000","last_active":"2024-09-13T14:30:55.906552+0000","last_peered":"2024-09-13T14:30:55.906552+0000","last_clean":"2024-09-13T14:30:55.906552+0000","last_became_active":"2024-09-13T14:27:12.834572+0000","last_became_peered":"2024-09-13T14:27:12.834572+0000","last_unstale":"2024-09-13T14:30:55.906552+0000","last_undegraded":"2024-09-13T14:30:55.906552+0000","last_fullsized":"2024-09-13T14:30:55.906552+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:54:21.155689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":136,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:30:54.898132+0000","last_change":"2024-09-13T14:26:15.826283+0000","last_active":"2024-09-13T14:30:54.898132+0000","last_peered":"2024-09-13T14:30:54.898132+0000","last_clean":"2024-09-13T14:30:54.898132+0000","last_became_active":"2024-09-13T14:26:15.820774+0000","last_became_peered":"2024-09-13T14:26:15.820774+0000","last_unstale":"2024-09-13T14:30:54.898132+0000","last_undegraded":"2024-09-13T14:30:54.898132+0000","last_fullsized":"2024-09-13T14:30:54.898132+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:28:39.849420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":135,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-13T14:30:55.906807+0000","last_change":"2024-09-13T14:24:55.285384+0000","last_active":"2024-09-13T14:30:55.906807+0000","last_peered":"2024-09-13T14:30:55.906807+0000","last_clean":"2024-09-13T14:30:55.906807+0000","last_became_active":"2024-09-13T14:24:55.285081+0000","last_became_peered":"2024-09-13T14:24:55.285081+0000","last_unstale":"2024-09-13T14:30:55.906807+0000","last_undegraded":"2024-09-13T14:30:55.906807+0000","last_fullsized":"2024-09-13T14:30:55.906807+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:47:07.946120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":108,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T14:30:54.898485+0000","last_change":"2024-09-13T14:11:29.240351+0000","last_active":"2024-09-13T14:30:54.898485+0000","last_peered":"2024-09-13T14:30:54.898485+0000","last_clean":"2024-09-13T14:30:54.898485+0000","last_became_active":"2024-09-13T14:11:29.240060+0000","last_became_peered":"2024-09-13T14:11:29.240060+0000","last_unstale":"2024-09-13T14:30:54.898485+0000","last_undegraded":"2024-09-13T14:30:54.898485+0000","last_fullsized":"2024-09-13T14:30:54.898485+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:34:36.126368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":200,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:30:57.927429+0000","last_change":"2024-09-13T14:27:12.839943+0000","last_active":"2024-09-13T14:30:57.927429+0000","last_peered":"2024-09-13T14:30:57.927429+0000","last_clean":"2024-09-13T14:30:57.927429+0000","last_became_active":"2024-09-13T14:27:12.839534+0000","last_became_peered":"2024-09-13T14:27:12.839534+0000","last_unstale":"2024-09-13T14:30:57.927429+0000","last_undegraded":"2024-09-13T14:30:57.927429+0000","last_fullsized":"2024-09-13T14:30:57.927429+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:34:56.677019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":189,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:30:57.927525+0000","last_change":"2024-09-13T14:27:12.836567+0000","last_active":"2024-09-13T14:30:57.927525+0000","last_peered":"2024-09-13T14:30:57.927525+0000","last_clean":"2024-09-13T14:30:57.927525+0000","last_became_active":"2024-09-13T14:27:12.836271+0000","last_became_peered":"2024-09-13T14:27:12.836271+0000","last_unstale":"2024-09-13T14:30:57.927525+0000","last_undegraded":"2024-09-13T14:30:57.927525+0000","last_fullsized":"2024-09-13T14:30:57.927525+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:25:32.910438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":130,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-13T14:30:55.906324+0000","last_change":"2024-09-13T14:26:15.824075+0000","last_active":"2024-09-13T14:30:55.906324+0000","last_peered":"2024-09-13T14:30:55.906324+0000","last_clean":"2024-09-13T14:30:55.906324+0000","last_became_active":"2024-09-13T14:26:15.823762+0000","last_became_peered":"2024-09-13T14:26:15.823762+0000","last_unstale":"2024-09-13T14:30:55.906324+0000","last_undegraded":"2024-09-13T14:30:55.906324+0000","last_fullsized":"2024-09-13T14:30:55.906324+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:26:51.088388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":179,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-13T14:30:55.909035+0000","last_change":"2024-09-13T14:27:12.844201+0000","last_active":"2024-09-13T14:30:55.909035+0000","last_peered":"2024-09-13T14:30:55.909035+0000","last_clean":"2024-09-13T14:30:55.909035+0000","last_became_active":"2024-09-13T14:27:12.834075+0000","last_became_peered":"2024-09-13T14:27:12.834075+0000","last_unstale":"2024-09-13T14:30:55.909035+0000","last_undegraded":"2024-09-13T14:30:55.909035+0000","last_fullsized":"2024-09-13T14:30:55.909035+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:06.657030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":616,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":17278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18272,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16793600,"data_stored":16777292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":23,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234245,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":141760,"kb_used_data":2352,"kb_used_omap":1,"kb_used_meta":139390,"kb_avail":93603392,"statfs":{"total":95995035648,"available":95849873408,"internally_reserved":0,"allocated":2408448,"data_stored":1613612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":142735819},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495881,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":135972,"kb_used_data":6424,"kb_used_omap":1,"kb_used_meta":129534,"kb_avail":93609180,"statfs":{"total":95995035648,"available":95855800320,"internally_reserved":0,"allocated":6578176,"data_stored":5791912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":132643271},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757518,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":207592,"kb_used_data":5852,"kb_used_omap":1,"kb_used_meta":201726,"kb_avail":93537560,"statfs":{"total":95995035648,"available":95782461440,"internally_reserved":0,"allocated":5992448,"data_stored":5213247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":206567878},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019154,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":217464,"kb_used_data":1768,"kb_used_omap":1,"kb_used_meta":215678,"kb_avail":93527688,"statfs":{"total":95995035648,"available":95772352512,"internally_reserved":0,"allocated":1810432,"data_stored":1023232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":220854726},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435339,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":95940,"kb_used_data":1780,"kb_used_omap":5,"kb_used_meta":94138,"kb_avail":93649212,"statfs":{"total":95995035648,"available":95896793088,"internally_reserved":0,"allocated":1822720,"data_stored":1034751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":96397438},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":0,"seq":446676598865,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":425201762392,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726925918,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":258852,"kb_used_data":6424,"kb_used_omap":7,"kb_used_meta":252408,"kb_avail":93486300,"statfs":{"total":95995035648,"available":95729971200,"internally_reserved":0,"allocated":6578176,"data_stored":5795645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":258466088},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:31:04.239 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:04.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:03 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2067356997' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-13T14:31:04.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:03 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-13T14:31:04.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:03 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3941422491' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:31:04.260 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:04.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:04.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:04.363 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:04.385 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:04.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:04.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:04.487 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:04.508 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:04.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:04.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:04.610 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:04.636 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:04.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:04.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:04.738 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:04.763 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:04.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:04.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:04.865 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:04.891 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:04.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:04.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:04.993 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:05.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:05.015 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:05.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:05.116 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:05.138 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:05.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:05.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:05.194 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:31:05.240 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:05.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:04 smithi195 ceph-mon[104262]: pgmap v482: 73 pgs: 73 active+clean; 4.6 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 0 B/s, 1 objects/s recovering 2024-09-13T14:31:05.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:04 smithi195 ceph-mon[104262]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 12/84 objects degraded (14.286%), 1 pg degraded, 1 pg undersized) 2024-09-13T14:31:05.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:04 smithi195 ceph-mon[104262]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:31:05.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:04 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-13T14:31:05.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:04 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3941422491' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:31:05.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:04 smithi195 ceph-mon[104262]: osdmap e161: 8 total, 8 up, 6 in 2024-09-13T14:31:05.265 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:05.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:05.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:05.315 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:04 smithi102 ceph-mon[110183]: pgmap v482: 73 pgs: 73 active+clean; 4.6 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 0 B/s, 1 objects/s recovering 2024-09-13T14:31:05.315 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:04 smithi102 ceph-mon[110183]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 12/84 objects degraded (14.286%), 1 pg degraded, 1 pg undersized) 2024-09-13T14:31:05.315 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:04 smithi102 ceph-mon[110183]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:31:05.315 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:04 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-13T14:31:05.315 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:04 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3941422491' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:31:05.316 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:04 smithi102 ceph-mon[110183]: osdmap e161: 8 total, 8 up, 6 in 2024-09-13T14:31:05.316 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:04 smithi102 ceph-mon[115269]: pgmap v482: 73 pgs: 73 active+clean; 4.6 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 0 B/s, 1 objects/s recovering 2024-09-13T14:31:05.316 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:04 smithi102 ceph-mon[115269]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 12/84 objects degraded (14.286%), 1 pg degraded, 1 pg undersized) 2024-09-13T14:31:05.316 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:04 smithi102 ceph-mon[115269]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:31:05.316 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:04 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-13T14:31:05.316 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:04 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3941422491' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:31:05.316 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:04 smithi102 ceph-mon[115269]: osdmap e161: 8 total, 8 up, 6 in 2024-09-13T14:31:05.367 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:05.388 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:05.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:05.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:05.490 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:05.511 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:05.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:05.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:05.614 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:31:05.639 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:05.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:05.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:31:05.737 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-09-13T14:31:05.741 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:05.802 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:05.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:05.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:05.904 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:05.925 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:05.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:05.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:06.010 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:05 smithi102 ceph-mon[110183]: osdmap e162: 8 total, 8 up, 6 in 2024-09-13T14:31:06.010 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:05 smithi102 ceph-mon[115269]: osdmap e162: 8 total, 8 up, 6 in 2024-09-13T14:31:06.027 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:06.049 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:06.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:06.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:06.151 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:06.173 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:06.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:06.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:06.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:05 smithi195 ceph-mon[104262]: osdmap e162: 8 total, 8 up, 6 in 2024-09-13T14:31:06.275 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:06.300 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:06.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:06.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:06.402 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:06.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:06.425 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:06.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:06.448 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:31:06.527 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:06.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:06.549 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:06.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:06.650 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:31:06.672 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:06.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:06.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:31:06.707 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:31:06 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:31:06] "GET /metrics HTTP/1.1" 200 36513 "" "Prometheus/2.51.0" 2024-09-13T14:31:06.774 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:06.796 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:06.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:06.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:06.898 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:06.919 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:06.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:06.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:06.976 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:06 smithi102 ceph-mon[110183]: pgmap v485: 89 pgs: 16 unknown, 73 active+clean; 4.6 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 0 B/s, 1 objects/s recovering 2024-09-13T14:31:06.976 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:06 smithi102 ceph-mon[110183]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:31:06.976 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:06 smithi102 ceph-mon[110183]: osdmap e163: 8 total, 8 up, 6 in 2024-09-13T14:31:06.976 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:06 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:31:06.976 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:06 smithi102 ceph-mon[115269]: pgmap v485: 89 pgs: 16 unknown, 73 active+clean; 4.6 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 0 B/s, 1 objects/s recovering 2024-09-13T14:31:06.976 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:06 smithi102 ceph-mon[115269]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:31:06.977 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:06 smithi102 ceph-mon[115269]: osdmap e163: 8 total, 8 up, 6 in 2024-09-13T14:31:06.977 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:06 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:31:07.001 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:31:07.021 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:07.065 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:07.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:07.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:07.167 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:07.188 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:07.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:07.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:07.218 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:31:07.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:06 smithi195 ceph-mon[104262]: pgmap v485: 89 pgs: 16 unknown, 73 active+clean; 4.6 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 0 B/s, 1 objects/s recovering 2024-09-13T14:31:07.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:06 smithi195 ceph-mon[104262]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:31:07.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:06 smithi195 ceph-mon[104262]: osdmap e163: 8 total, 8 up, 6 in 2024-09-13T14:31:07.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:06 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:31:07.290 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:07.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:07.312 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:07.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:07.413 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:31:07.440 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:07.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:07.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:31:07.541 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:07.562 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:07.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:07.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:07.664 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:07.688 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:07.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:07.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:07.790 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:07.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:07.815 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:07.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:07.916 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:07.937 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:07.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:07.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:08.039 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:08.063 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:08.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:08.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:08.165 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:08.186 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:08.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:08.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:08.288 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:31:08.313 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:08.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:08.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:31:08.415 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:08.436 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:08.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:08.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:08.443 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:31:08.538 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:08.560 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:08.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:08.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:08.661 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:08.682 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:08.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:08.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:08.784 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:08.806 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:08.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:08.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:08.907 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:08.928 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:08.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:08.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:09.030 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:09.043 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T14:31:09.058 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:09.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:09.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:09.160 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:09.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:08 smithi102 ceph-mon[110183]: pgmap v487: 89 pgs: 4 unknown, 85 active+clean; 4.6 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 0 B/s, 1 objects/s recovering 2024-09-13T14:31:09.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:08 smithi102 ceph-mon[115269]: pgmap v487: 89 pgs: 4 unknown, 85 active+clean; 4.6 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 0 B/s, 1 objects/s recovering 2024-09-13T14:31:09.181 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:09.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:09.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:09.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:08 smithi195 ceph-mon[104262]: pgmap v487: 89 pgs: 4 unknown, 85 active+clean; 4.6 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 0 B/s, 1 objects/s recovering 2024-09-13T14:31:09.283 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:09.305 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:09.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:09.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:09.407 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:09.429 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:09.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:09.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:09.530 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:09.552 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:09.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:09.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:09.653 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:31:09.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:09.676 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:09.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:31:09.777 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:31:09.798 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:09.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:09.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:31:09.900 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:31:09.900 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:09.923 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:09.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:09.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:10.024 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:10.045 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:10.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:10.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:10.089 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:31:10.089 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:31:10.107 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:31:10.107 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:31:10.129 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:31:10.129 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:31:10.146 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:31:10.146 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:31:10.147 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:10.164 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:31:10.164 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:31:10.169 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:10.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:10.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:10.181 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:31:10.181 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:31:10.198 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:31:10.198 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:31:10.214 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:31:10.215 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:31:10.271 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:10.293 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:10.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:10.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:10.394 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:10.416 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:10.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:10.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:10.518 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:10.539 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:10.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:10.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:10.641 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:10.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:10.662 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:10.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:10.763 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:10.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:10.785 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:10.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:10.886 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:31:10.908 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:10.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:10.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:31:11.009 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:11.030 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:11.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:11.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:11.058 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:10 smithi195 ceph-mon[104262]: pgmap v488: 89 pgs: 89 active+clean; 4.6 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:31:11.058 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:10 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/685225464' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:31:11.058 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:31:11.059 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:31:11.132 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:11.153 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:11.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:11.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:11.215 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:10 smithi102 ceph-mon[110183]: pgmap v488: 89 pgs: 89 active+clean; 4.6 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:31:11.216 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:10 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/685225464' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:31:11.216 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:31:11.216 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:31:11.216 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:10 smithi102 ceph-mon[115269]: pgmap v488: 89 pgs: 89 active+clean; 4.6 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:31:11.216 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:10 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/685225464' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:31:11.216 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:31:11.216 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:31:11.255 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:11.276 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:11.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:11.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:11.378 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:31:11.400 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:11.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:11.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:31:11.501 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:31:11.523 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:11.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:11.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:31:11.625 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:31:11.646 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:11.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:11.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:31:11.748 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:31:11.769 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:11.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:11.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:31:11.871 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:11.893 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:11.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:11.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:11.961 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is unset 2024-09-13T14:31:11.961 INFO:teuthology.orchestra.run.smithi102.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-09-13T14:31:11.994 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:31:12.015 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:12.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:12.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:31:12.117 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:12.139 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:12.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:12.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:12.240 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:12.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:11 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3634319488' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:31:12.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:11 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/158538181' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:31:12.262 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:12.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:12.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:12.341 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:31:12.341 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:31:12.364 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:12.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:12.386 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:12.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:12.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:11 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3634319488' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:31:12.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:11 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/158538181' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:31:12.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:11 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3634319488' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:31:12.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:11 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/158538181' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:31:12.488 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:12.510 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:12.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:12.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:12.612 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:12.634 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:12.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:12.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:12.736 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:12.758 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:12.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:12.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:12.859 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:12.882 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:12.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:12.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:12.941 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T14:31:12.983 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:13.005 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:13.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:13.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:13.107 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:13.118 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:31:13.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:13.129 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:13.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:13.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:12 smithi102 ceph-mon[110183]: pgmap v489: 89 pgs: 89 active+clean; 4.6 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:31:13.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:12 smithi102 ceph-mon[110183]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:31:13.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:12 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3634319488' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:31:13.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:12 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/158538181' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:31:13.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:12 smithi102 ceph-mon[110183]: osdmap e164: 8 total, 8 up, 6 in 2024-09-13T14:31:13.149 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:12 smithi102 ceph-mon[115269]: pgmap v489: 89 pgs: 89 active+clean; 4.6 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:31:13.149 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:12 smithi102 ceph-mon[115269]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:31:13.149 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:12 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3634319488' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:31:13.150 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:12 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/158538181' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:31:13.150 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:12 smithi102 ceph-mon[115269]: osdmap e164: 8 total, 8 up, 6 in 2024-09-13T14:31:13.230 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:13.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:12 smithi195 ceph-mon[104262]: pgmap v489: 89 pgs: 89 active+clean; 4.6 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:31:13.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:12 smithi195 ceph-mon[104262]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:31:13.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:12 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3634319488' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:31:13.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:12 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/158538181' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:31:13.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:12 smithi195 ceph-mon[104262]: osdmap e164: 8 total, 8 up, 6 in 2024-09-13T14:31:13.253 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:13.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:13.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:13.355 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:13.376 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:13.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:13.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:13.478 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:13.499 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:13.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:13.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:13.601 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:13.628 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:13.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:13.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:13.730 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:13.756 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:13.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:13.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:13.830 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:31:13.857 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:13.868 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:31:13.880 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:13.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:13.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:13.982 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:14.004 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:14.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:14.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:14.105 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:14.117 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:14 smithi102 ceph-mon[110183]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:31:14.117 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:14 smithi102 ceph-mon[110183]: Cluster is now healthy 2024-09-13T14:31:14.117 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:14 smithi102 ceph-mon[115269]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:31:14.117 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:14 smithi102 ceph-mon[115269]: Cluster is now healthy 2024-09-13T14:31:14.129 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:14.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:14.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:14.230 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:14.232 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:31:14.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:14 smithi195 ceph-mon[104262]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:31:14.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:14 smithi195 ceph-mon[104262]: Cluster is now healthy 2024-09-13T14:31:14.251 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:14.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:14.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:14.353 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:31:14.374 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:14.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:14.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:31:14.476 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:14.501 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:14.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:14.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:14.603 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:31:14.626 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:14.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:14.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:31:14.728 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:14.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:14.751 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:14.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:14.841 DEBUG:teuthology.orchestra.run.smithi102:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-09-13T14:31:14.844 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-13T14:31:14.844 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-13T14:31:14.852 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:14.872 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:14.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:14.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:14.893 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:hints = 1 2024-09-13T14:31:14.893 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-13T14:31:14.894 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Object prefix: benchmark_data_smithi102_194590 2024-09-13T14:31:14.894 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:31:14.894 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 0 0 0 0 0 0 - 0 2024-09-13T14:31:14.974 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:15.007 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:15.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:15.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:15.108 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:15.139 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:15.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:15.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:15.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:15 smithi102 ceph-mon[110183]: pgmap v491: 89 pgs: 89 active+clean; 4.6 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:31:15.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:15 smithi102 ceph-mon[115269]: pgmap v491: 89 pgs: 89 active+clean; 4.6 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:31:15.240 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:15.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:15 smithi195 ceph-mon[104262]: pgmap v491: 89 pgs: 89 active+clean; 4.6 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:31:15.253 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:31:15.274 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:15.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:15.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:15.376 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:15.411 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:15.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:15.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:15.513 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:15.545 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:15.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:15.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:15.646 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:15.677 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:15.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:15.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:15.778 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:15.817 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:15.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:15.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:15.894 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 1 16 3934 3918 244.857 244.875 0.00392348 0.00407302 2024-09-13T14:31:15.919 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:15.957 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:15.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:15.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:16.059 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:16.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:16.097 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:16.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:16.197 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:16.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:16.245 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:16.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:16.346 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:16.377 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:16.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:16.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:16.479 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:31:16.521 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:16.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:16.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:31:16.601 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:31:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:31:16] "GET /metrics HTTP/1.1" 200 36447 "" "Prometheus/2.51.0" 2024-09-13T14:31:16.623 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:16.658 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:16.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:16.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:16.759 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:16.807 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:16.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:16.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:16.894 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 2 16 7354 7338 229.295 213.75 0.00455281 0.00435355 2024-09-13T14:31:16.909 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:16.955 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:16.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:16.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:17.055 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:31:17.056 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:31:17.057 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:17.097 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:17.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:17.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:17.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:17 smithi102 ceph-mon[115269]: pgmap v492: 89 pgs: 89 active+clean; 4.6 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:31:17.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:17 smithi102 ceph-mon[110183]: pgmap v492: 89 pgs: 89 active+clean; 4.6 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:31:17.199 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:17.232 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:17.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:17.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:17.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:17 smithi195 ceph-mon[104262]: pgmap v492: 89 pgs: 89 active+clean; 4.6 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-09-13T14:31:17.334 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:17.371 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:17.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:17.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:17.473 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:31:17.510 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:31:17.510 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:31:17.510 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:31:17.510 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:31:17.511 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:17.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:17.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:31:17.514 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:31:17.514 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:31:17.546 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:31:17.546 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:31:17.575 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:31:17.575 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:31:17.603 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:31:17.603 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:31:17.613 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:31:17.634 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:31:17.635 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:31:17.649 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:17.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:17.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:31:17.665 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:31:17.665 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:31:17.729 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T14:31:17.750 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:17.783 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:17.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:17.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:17.870 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:31:17.872 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:31:17.885 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:17.894 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 3 16 10497 10481 218.337 196.438 0.00568389 0.00457045 2024-09-13T14:31:17.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:17.919 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:17.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:18.019 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:18.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:18.069 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:18.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:18.123 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:31:18.144 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:31:18.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:18 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/483511999' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:31:18.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:18 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/483511999' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:31:18.171 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:31:18.216 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:18.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:18.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:31:18.316 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:18 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/483511999' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:31:18.318 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:18.369 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:18.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:18.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:18.471 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:31:18.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:18.509 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:18.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:31:18.610 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:31:18.653 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:18.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:18.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:31:18.755 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:31:18.792 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:18.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:18.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:31:18.893 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:18.895 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 4 16 13251 13235 206.78 172.125 0.00297618 0.00482976 2024-09-13T14:31:18.923 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:18.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:18.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:19.025 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:19.040 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":493,"stamp":"2024-09-13T14:31:17.326592+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":364910565,"num_objects":5523,"num_object_clones":0,"num_object_copies":16569,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5523,"num_whiteouts":0,"num_read":6390,"num_read_kb":13845,"num_write":6618,"num_write_kb":366721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":5966043,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6677,"ondisk_log_size":6677,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":1854372,"kb_used_data":801776,"kb_used_omap":19,"kb_used_meta":1052524,"kb_avail":560616540,"statfs":{"total":575970213888,"available":574071336960,"internally_reserved":0,"allocated":821018624,"data_stored":816105181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20284,"internal_metadata":1077784772},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":360120320,"num_objects":5495,"num_object_clones":0,"num_object_copies":16485,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5495,"num_write_kb":351680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002188"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":198,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:11.965716+0000","last_change":"2024-09-13T14:27:12.839984+0000","last_active":"2024-09-13T14:31:11.965716+0000","last_peered":"2024-09-13T14:31:11.965716+0000","last_clean":"2024-09-13T14:31:11.965716+0000","last_became_active":"2024-09-13T14:27:12.839653+0000","last_became_peered":"2024-09-13T14:27:12.839653+0000","last_unstale":"2024-09-13T14:31:11.965716+0000","last_undegraded":"2024-09-13T14:31:11.965716+0000","last_fullsized":"2024-09-13T14:31:11.965716+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:14:13.672830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":146,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T14:31:06.414629+0000","last_change":"2024-09-13T14:27:12.846760+0000","last_active":"2024-09-13T14:31:06.414629+0000","last_peered":"2024-09-13T14:31:06.414629+0000","last_clean":"2024-09-13T14:31:06.414629+0000","last_became_active":"2024-09-13T14:27:12.838041+0000","last_became_peered":"2024-09-13T14:27:12.838041+0000","last_unstale":"2024-09-13T14:31:06.414629+0000","last_undegraded":"2024-09-13T14:31:06.414629+0000","last_fullsized":"2024-09-13T14:31:06.414629+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:22.402859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":128,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T14:31:06.454080+0000","last_change":"2024-09-13T14:24:55.282407+0000","last_active":"2024-09-13T14:31:06.454080+0000","last_peered":"2024-09-13T14:31:06.454080+0000","last_clean":"2024-09-13T14:31:06.454080+0000","last_became_active":"2024-09-13T14:24:55.282083+0000","last_became_peered":"2024-09-13T14:24:55.282083+0000","last_unstale":"2024-09-13T14:31:06.454080+0000","last_undegraded":"2024-09-13T14:31:06.454080+0000","last_fullsized":"2024-09-13T14:31:06.454080+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:23:10.525707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":175,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:13.015309+0000","last_change":"2024-09-13T14:24:55.284433+0000","last_active":"2024-09-13T14:31:13.015309+0000","last_peered":"2024-09-13T14:31:13.015309+0000","last_clean":"2024-09-13T14:31:13.015309+0000","last_became_active":"2024-09-13T14:24:55.284143+0000","last_became_peered":"2024-09-13T14:24:55.284143+0000","last_unstale":"2024-09-13T14:31:13.015309+0000","last_undegraded":"2024-09-13T14:31:13.015309+0000","last_fullsized":"2024-09-13T14:31:13.015309+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:28.107405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":177,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:13.015491+0000","last_change":"2024-09-13T14:24:55.282416+0000","last_active":"2024-09-13T14:31:13.015491+0000","last_peered":"2024-09-13T14:31:13.015491+0000","last_clean":"2024-09-13T14:31:13.015491+0000","last_became_active":"2024-09-13T14:24:55.282045+0000","last_became_peered":"2024-09-13T14:24:55.282045+0000","last_unstale":"2024-09-13T14:31:13.015491+0000","last_undegraded":"2024-09-13T14:31:13.015491+0000","last_fullsized":"2024-09-13T14:31:13.015491+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:58.692897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":114,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T14:31:06.414371+0000","last_change":"2024-09-13T14:11:29.244452+0000","last_active":"2024-09-13T14:31:06.414371+0000","last_peered":"2024-09-13T14:31:06.414371+0000","last_clean":"2024-09-13T14:31:06.414371+0000","last_became_active":"2024-09-13T14:11:29.244102+0000","last_became_peered":"2024-09-13T14:11:29.244102+0000","last_unstale":"2024-09-13T14:31:06.414371+0000","last_undegraded":"2024-09-13T14:31:06.414371+0000","last_fullsized":"2024-09-13T14:31:06.414371+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:13:19.165534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":198,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:11.966109+0000","last_change":"2024-09-13T14:27:12.834300+0000","last_active":"2024-09-13T14:31:11.966109+0000","last_peered":"2024-09-13T14:31:11.966109+0000","last_clean":"2024-09-13T14:31:11.966109+0000","last_became_active":"2024-09-13T14:27:12.834046+0000","last_became_peered":"2024-09-13T14:27:12.834046+0000","last_unstale":"2024-09-13T14:31:11.966109+0000","last_undegraded":"2024-09-13T14:31:11.966109+0000","last_fullsized":"2024-09-13T14:31:11.966109+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:46:58.906862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":137,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T14:31:05.953187+0000","last_change":"2024-09-13T14:23:18.039061+0000","last_active":"2024-09-13T14:31:05.953187+0000","last_peered":"2024-09-13T14:31:05.953187+0000","last_clean":"2024-09-13T14:31:05.953187+0000","last_became_active":"2024-09-13T14:23:18.038661+0000","last_became_peered":"2024-09-13T14:23:18.038661+0000","last_unstale":"2024-09-13T14:31:05.953187+0000","last_undegraded":"2024-09-13T14:31:05.953187+0000","last_fullsized":"2024-09-13T14:31:05.953187+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:05:29.109844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"164'456","reported_seq":468,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:16.937499+0000","last_change":"2024-09-13T14:31:04.959615+0000","last_active":"2024-09-13T14:31:16.937499+0000","last_peered":"2024-09-13T14:31:16.937499+0000","last_clean":"2024-09-13T14:31:16.937499+0000","last_became_active":"2024-09-13T14:31:04.959265+0000","last_became_peered":"2024-09-13T14:31:04.959265+0000","last_unstale":"2024-09-13T14:31:16.937499+0000","last_undegraded":"2024-09-13T14:31:16.937499+0000","last_fullsized":"2024-09-13T14:31:16.937499+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_clean_scrub_stamp":"2024-09-13T14:31:03.932624+0000","objects_scrubbed":0,"log_size":456,"log_dups_size":0,"ondisk_log_size":456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:44.257407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29884416,"num_objects":456,"num_object_clones":0,"num_object_copies":1368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":456,"num_write_kb":29184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":130,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T14:31:06.461315+0000","last_change":"2024-09-13T14:23:18.038597+0000","last_active":"2024-09-13T14:31:06.461315+0000","last_peered":"2024-09-13T14:31:06.461315+0000","last_clean":"2024-09-13T14:31:06.461315+0000","last_became_active":"2024-09-13T14:23:18.038240+0000","last_became_peered":"2024-09-13T14:23:18.038240+0000","last_unstale":"2024-09-13T14:31:06.461315+0000","last_undegraded":"2024-09-13T14:31:06.461315+0000","last_fullsized":"2024-09-13T14:31:06.461315+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:27:51.842195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":114,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T14:31:06.453576+0000","last_change":"2024-09-13T14:11:30.254534+0000","last_active":"2024-09-13T14:31:06.453576+0000","last_peered":"2024-09-13T14:31:06.453576+0000","last_clean":"2024-09-13T14:31:06.453576+0000","last_became_active":"2024-09-13T14:11:30.253959+0000","last_became_peered":"2024-09-13T14:11:30.253959+0000","last_unstale":"2024-09-13T14:31:06.453576+0000","last_undegraded":"2024-09-13T14:31:06.453576+0000","last_fullsized":"2024-09-13T14:31:06.453576+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:09:54.025853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":17,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:11.964661+0000","last_change":"2024-09-13T14:31:04.957456+0000","last_active":"2024-09-13T14:31:11.964661+0000","last_peered":"2024-09-13T14:31:11.964661+0000","last_clean":"2024-09-13T14:31:11.964661+0000","last_became_active":"2024-09-13T14:31:04.957189+0000","last_became_peered":"2024-09-13T14:31:04.957189+0000","last_unstale":"2024-09-13T14:31:11.964661+0000","last_undegraded":"2024-09-13T14:31:11.964661+0000","last_fullsized":"2024-09-13T14:31:11.964661+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_clean_scrub_stamp":"2024-09-13T14:31:03.932624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:26:09.289712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":144,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T14:31:06.461353+0000","last_change":"2024-09-13T14:26:15.828529+0000","last_active":"2024-09-13T14:31:06.461353+0000","last_peered":"2024-09-13T14:31:06.461353+0000","last_clean":"2024-09-13T14:31:06.461353+0000","last_became_active":"2024-09-13T14:26:15.820486+0000","last_became_peered":"2024-09-13T14:26:15.820486+0000","last_unstale":"2024-09-13T14:31:06.461353+0000","last_undegraded":"2024-09-13T14:31:06.461353+0000","last_fullsized":"2024-09-13T14:31:06.461353+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:11:46.283172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":152,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T14:31:06.803528+0000","last_change":"2024-09-13T14:26:15.824632+0000","last_active":"2024-09-13T14:31:06.803528+0000","last_peered":"2024-09-13T14:31:06.803528+0000","last_clean":"2024-09-13T14:31:06.803528+0000","last_became_active":"2024-09-13T14:26:15.824330+0000","last_became_peered":"2024-09-13T14:26:15.824330+0000","last_unstale":"2024-09-13T14:31:06.803528+0000","last_undegraded":"2024-09-13T14:31:06.803528+0000","last_fullsized":"2024-09-13T14:31:06.803528+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:25.288116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"164'465","reported_seq":477,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:16.993363+0000","last_change":"2024-09-13T14:31:04.953557+0000","last_active":"2024-09-13T14:31:16.993363+0000","last_peered":"2024-09-13T14:31:16.993363+0000","last_clean":"2024-09-13T14:31:16.993363+0000","last_became_active":"2024-09-13T14:31:04.953250+0000","last_became_peered":"2024-09-13T14:31:04.953250+0000","last_unstale":"2024-09-13T14:31:16.993363+0000","last_undegraded":"2024-09-13T14:31:16.993363+0000","last_fullsized":"2024-09-13T14:31:16.993363+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_clean_scrub_stamp":"2024-09-13T14:31:03.932624+0000","objects_scrubbed":0,"log_size":465,"log_dups_size":0,"ondisk_log_size":465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:48.668087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30474240,"num_objects":465,"num_object_clones":0,"num_object_copies":1395,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":465,"num_write_kb":29760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":115,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T14:31:06.461349+0000","last_change":"2024-09-13T14:11:29.232897+0000","last_active":"2024-09-13T14:31:06.461349+0000","last_peered":"2024-09-13T14:31:06.461349+0000","last_clean":"2024-09-13T14:31:06.461349+0000","last_became_active":"2024-09-13T14:11:29.232637+0000","last_became_peered":"2024-09-13T14:11:29.232637+0000","last_unstale":"2024-09-13T14:31:06.461349+0000","last_undegraded":"2024-09-13T14:31:06.461349+0000","last_fullsized":"2024-09-13T14:31:06.461349+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:31:05.856046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":138,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T14:31:06.367470+0000","last_change":"2024-09-13T14:26:15.822452+0000","last_active":"2024-09-13T14:31:06.367470+0000","last_peered":"2024-09-13T14:31:06.367470+0000","last_clean":"2024-09-13T14:31:06.367470+0000","last_became_active":"2024-09-13T14:26:15.822111+0000","last_became_peered":"2024-09-13T14:26:15.822111+0000","last_unstale":"2024-09-13T14:31:06.367470+0000","last_undegraded":"2024-09-13T14:31:06.367470+0000","last_fullsized":"2024-09-13T14:31:06.367470+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:32:53.078767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"164'378","reported_seq":395,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:16.490166+0000","last_change":"2024-09-13T14:31:04.953002+0000","last_active":"2024-09-13T14:31:16.490166+0000","last_peered":"2024-09-13T14:31:16.490166+0000","last_clean":"2024-09-13T14:31:16.490166+0000","last_became_active":"2024-09-13T14:31:04.952667+0000","last_became_peered":"2024-09-13T14:31:04.952667+0000","last_unstale":"2024-09-13T14:31:16.490166+0000","last_undegraded":"2024-09-13T14:31:16.490166+0000","last_fullsized":"2024-09-13T14:31:16.490166+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_clean_scrub_stamp":"2024-09-13T14:31:03.932624+0000","objects_scrubbed":0,"log_size":378,"log_dups_size":0,"ondisk_log_size":378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:22:35.630717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24772608,"num_objects":378,"num_object_clones":0,"num_object_copies":1134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":378,"num_write_kb":24192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":129,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T14:31:06.454184+0000","last_change":"2024-09-13T14:24:55.290798+0000","last_active":"2024-09-13T14:31:06.454184+0000","last_peered":"2024-09-13T14:31:06.454184+0000","last_clean":"2024-09-13T14:31:06.454184+0000","last_became_active":"2024-09-13T14:24:55.290510+0000","last_became_peered":"2024-09-13T14:24:55.290510+0000","last_unstale":"2024-09-13T14:31:06.454184+0000","last_undegraded":"2024-09-13T14:31:06.454184+0000","last_fullsized":"2024-09-13T14:31:06.454184+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:19:03.129132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":184,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:13.015555+0000","last_change":"2024-09-13T14:26:15.823316+0000","last_active":"2024-09-13T14:31:13.015555+0000","last_peered":"2024-09-13T14:31:13.015555+0000","last_clean":"2024-09-13T14:31:13.015555+0000","last_became_active":"2024-09-13T14:26:15.822980+0000","last_became_peered":"2024-09-13T14:26:15.822980+0000","last_unstale":"2024-09-13T14:31:13.015555+0000","last_undegraded":"2024-09-13T14:31:13.015555+0000","last_fullsized":"2024-09-13T14:31:13.015555+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:33:47.420548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"164'530","reported_seq":542,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:17.244632+0000","last_change":"2024-09-13T14:31:04.958431+0000","last_active":"2024-09-13T14:31:17.244632+0000","last_peered":"2024-09-13T14:31:17.244632+0000","last_clean":"2024-09-13T14:31:17.244632+0000","last_became_active":"2024-09-13T14:31:04.958121+0000","last_became_peered":"2024-09-13T14:31:04.958121+0000","last_unstale":"2024-09-13T14:31:17.244632+0000","last_undegraded":"2024-09-13T14:31:17.244632+0000","last_fullsized":"2024-09-13T14:31:17.244632+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_clean_scrub_stamp":"2024-09-13T14:31:03.932624+0000","objects_scrubbed":0,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:09:51.570034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34734080,"num_objects":530,"num_object_clones":0,"num_object_copies":1590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":530,"num_write_kb":33920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":182,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:11.964939+0000","last_change":"2024-09-13T14:24:55.284499+0000","last_active":"2024-09-13T14:31:11.964939+0000","last_peered":"2024-09-13T14:31:11.964939+0000","last_clean":"2024-09-13T14:31:11.964939+0000","last_became_active":"2024-09-13T14:24:55.284096+0000","last_became_peered":"2024-09-13T14:24:55.284096+0000","last_unstale":"2024-09-13T14:31:11.964939+0000","last_undegraded":"2024-09-13T14:31:11.964939+0000","last_fullsized":"2024-09-13T14:31:11.964939+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:02:06.836099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":127,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T14:31:06.803797+0000","last_change":"2024-09-13T14:24:55.291371+0000","last_active":"2024-09-13T14:31:06.803797+0000","last_peered":"2024-09-13T14:31:06.803797+0000","last_clean":"2024-09-13T14:31:06.803797+0000","last_became_active":"2024-09-13T14:24:55.291047+0000","last_became_peered":"2024-09-13T14:24:55.291047+0000","last_unstale":"2024-09-13T14:31:06.803797+0000","last_undegraded":"2024-09-13T14:31:06.803797+0000","last_fullsized":"2024-09-13T14:31:06.803797+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:43.912412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":17,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:11.965936+0000","last_change":"2024-09-13T14:31:04.956319+0000","last_active":"2024-09-13T14:31:11.965936+0000","last_peered":"2024-09-13T14:31:11.965936+0000","last_clean":"2024-09-13T14:31:11.965936+0000","last_became_active":"2024-09-13T14:31:04.956014+0000","last_became_peered":"2024-09-13T14:31:04.956014+0000","last_unstale":"2024-09-13T14:31:11.965936+0000","last_undegraded":"2024-09-13T14:31:11.965936+0000","last_fullsized":"2024-09-13T14:31:11.965936+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_clean_scrub_stamp":"2024-09-13T14:31:03.932624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:51:35.156658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":139,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T14:31:06.367486+0000","last_change":"2024-09-13T14:26:15.827333+0000","last_active":"2024-09-13T14:31:06.367486+0000","last_peered":"2024-09-13T14:31:06.367486+0000","last_clean":"2024-09-13T14:31:06.367486+0000","last_became_active":"2024-09-13T14:26:15.826945+0000","last_became_peered":"2024-09-13T14:26:15.826945+0000","last_unstale":"2024-09-13T14:31:06.367486+0000","last_undegraded":"2024-09-13T14:31:06.367486+0000","last_fullsized":"2024-09-13T14:31:06.367486+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:35:24.870052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":160,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T14:31:06.803881+0000","last_change":"2024-09-13T14:27:12.842604+0000","last_active":"2024-09-13T14:31:06.803881+0000","last_peered":"2024-09-13T14:31:06.803881+0000","last_clean":"2024-09-13T14:31:06.803881+0000","last_became_active":"2024-09-13T14:27:12.841643+0000","last_became_peered":"2024-09-13T14:27:12.841643+0000","last_unstale":"2024-09-13T14:31:06.803881+0000","last_undegraded":"2024-09-13T14:31:06.803881+0000","last_fullsized":"2024-09-13T14:31:06.803881+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:34:34.751790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":17,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:11.965426+0000","last_change":"2024-09-13T14:31:04.957232+0000","last_active":"2024-09-13T14:31:11.965426+0000","last_peered":"2024-09-13T14:31:11.965426+0000","last_clean":"2024-09-13T14:31:11.965426+0000","last_became_active":"2024-09-13T14:31:04.956942+0000","last_became_peered":"2024-09-13T14:31:04.956942+0000","last_unstale":"2024-09-13T14:31:11.965426+0000","last_undegraded":"2024-09-13T14:31:11.965426+0000","last_fullsized":"2024-09-13T14:31:11.965426+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_clean_scrub_stamp":"2024-09-13T14:31:03.932624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:13:50.184110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":138,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T14:31:06.803657+0000","last_change":"2024-09-13T14:27:12.842271+0000","last_active":"2024-09-13T14:31:06.803657+0000","last_peered":"2024-09-13T14:31:06.803657+0000","last_clean":"2024-09-13T14:31:06.803657+0000","last_became_active":"2024-09-13T14:27:12.841479+0000","last_became_peered":"2024-09-13T14:27:12.841479+0000","last_unstale":"2024-09-13T14:31:06.803657+0000","last_undegraded":"2024-09-13T14:31:06.803657+0000","last_fullsized":"2024-09-13T14:31:06.803657+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:30:22.746575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":136,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T14:31:06.865739+0000","last_change":"2024-09-13T14:26:15.825304+0000","last_active":"2024-09-13T14:31:06.865739+0000","last_peered":"2024-09-13T14:31:06.865739+0000","last_clean":"2024-09-13T14:31:06.865739+0000","last_became_active":"2024-09-13T14:26:15.825018+0000","last_became_peered":"2024-09-13T14:26:15.825018+0000","last_unstale":"2024-09-13T14:31:06.865739+0000","last_undegraded":"2024-09-13T14:31:06.865739+0000","last_fullsized":"2024-09-13T14:31:06.865739+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:22:59.408928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"164'496","reported_seq":508,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:16.937496+0000","last_change":"2024-09-13T14:31:04.960524+0000","last_active":"2024-09-13T14:31:16.937496+0000","last_peered":"2024-09-13T14:31:16.937496+0000","last_clean":"2024-09-13T14:31:16.937496+0000","last_became_active":"2024-09-13T14:31:04.960207+0000","last_became_peered":"2024-09-13T14:31:04.960207+0000","last_unstale":"2024-09-13T14:31:16.937496+0000","last_undegraded":"2024-09-13T14:31:16.937496+0000","last_fullsized":"2024-09-13T14:31:16.937496+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_clean_scrub_stamp":"2024-09-13T14:31:03.932624+0000","objects_scrubbed":0,"log_size":496,"log_dups_size":0,"ondisk_log_size":496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:48:58.950478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32505856,"num_objects":496,"num_object_clones":0,"num_object_copies":1488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":496,"num_write_kb":31744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":192,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:11.965054+0000","last_change":"2024-09-13T14:26:15.823898+0000","last_active":"2024-09-13T14:31:11.965054+0000","last_peered":"2024-09-13T14:31:11.965054+0000","last_clean":"2024-09-13T14:31:11.965054+0000","last_became_active":"2024-09-13T14:26:15.823125+0000","last_became_peered":"2024-09-13T14:26:15.823125+0000","last_unstale":"2024-09-13T14:31:11.965054+0000","last_undegraded":"2024-09-13T14:31:11.965054+0000","last_fullsized":"2024-09-13T14:31:11.965054+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:51:27.949528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":155,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:11.966216+0000","last_change":"2024-09-13T14:26:15.823198+0000","last_active":"2024-09-13T14:31:11.966216+0000","last_peered":"2024-09-13T14:31:11.966216+0000","last_clean":"2024-09-13T14:31:11.966216+0000","last_became_active":"2024-09-13T14:26:15.822717+0000","last_became_peered":"2024-09-13T14:26:15.822717+0000","last_unstale":"2024-09-13T14:31:11.966216+0000","last_undegraded":"2024-09-13T14:31:11.966216+0000","last_fullsized":"2024-09-13T14:31:11.966216+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:22.471622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"164'453","reported_seq":465,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:16.933026+0000","last_change":"2024-09-13T14:31:04.959681+0000","last_active":"2024-09-13T14:31:16.933026+0000","last_peered":"2024-09-13T14:31:16.933026+0000","last_clean":"2024-09-13T14:31:16.933026+0000","last_became_active":"2024-09-13T14:31:04.959394+0000","last_became_peered":"2024-09-13T14:31:04.959394+0000","last_unstale":"2024-09-13T14:31:16.933026+0000","last_undegraded":"2024-09-13T14:31:16.933026+0000","last_fullsized":"2024-09-13T14:31:16.933026+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_clean_scrub_stamp":"2024-09-13T14:31:03.932624+0000","objects_scrubbed":0,"log_size":453,"log_dups_size":0,"ondisk_log_size":453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:54:25.539991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29687808,"num_objects":453,"num_object_clones":0,"num_object_copies":1359,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":453,"num_write_kb":28992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":460,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T14:31:06.414535+0000","last_change":"2024-09-13T14:23:18.032973+0000","last_active":"2024-09-13T14:31:06.414535+0000","last_peered":"2024-09-13T14:31:06.414535+0000","last_clean":"2024-09-13T14:31:06.414535+0000","last_became_active":"2024-09-13T14:23:18.032579+0000","last_became_peered":"2024-09-13T14:23:18.032579+0000","last_unstale":"2024-09-13T14:31:06.414535+0000","last_undegraded":"2024-09-13T14:31:06.414535+0000","last_fullsized":"2024-09-13T14:31:06.414535+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:07:41.927740+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:17.109445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":494,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T14:31:06.955024+0000","last_change":"2024-09-13T14:27:15.865400+0000","last_active":"2024-09-13T14:31:06.955024+0000","last_peered":"2024-09-13T14:31:06.955024+0000","last_clean":"2024-09-13T14:31:06.955024+0000","last_became_active":"2024-09-13T14:27:15.864835+0000","last_became_peered":"2024-09-13T14:27:15.864835+0000","last_unstale":"2024-09-13T14:31:06.955024+0000","last_undegraded":"2024-09-13T14:31:06.955024+0000","last_fullsized":"2024-09-13T14:31:06.955024+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:27:10.928041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":134,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:11.964681+0000","last_change":"2024-09-13T14:24:55.288058+0000","last_active":"2024-09-13T14:31:11.964681+0000","last_peered":"2024-09-13T14:31:11.964681+0000","last_clean":"2024-09-13T14:31:11.964681+0000","last_became_active":"2024-09-13T14:24:55.287601+0000","last_became_peered":"2024-09-13T14:24:55.287601+0000","last_unstale":"2024-09-13T14:31:11.964681+0000","last_undegraded":"2024-09-13T14:31:11.964681+0000","last_fullsized":"2024-09-13T14:31:11.964681+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:56:34.622696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":114,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T14:31:05.953871+0000","last_change":"2024-09-13T14:11:30.253749+0000","last_active":"2024-09-13T14:31:05.953871+0000","last_peered":"2024-09-13T14:31:05.953871+0000","last_clean":"2024-09-13T14:31:05.953871+0000","last_became_active":"2024-09-13T14:11:30.253499+0000","last_became_peered":"2024-09-13T14:11:30.253499+0000","last_unstale":"2024-09-13T14:31:05.953871+0000","last_undegraded":"2024-09-13T14:31:05.953871+0000","last_fullsized":"2024-09-13T14:31:05.953871+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:02:00.987335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"164'482","reported_seq":494,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:16.943420+0000","last_change":"2024-09-13T14:31:04.961057+0000","last_active":"2024-09-13T14:31:16.943420+0000","last_peered":"2024-09-13T14:31:16.943420+0000","last_clean":"2024-09-13T14:31:16.943420+0000","last_became_active":"2024-09-13T14:31:04.960799+0000","last_became_peered":"2024-09-13T14:31:04.960799+0000","last_unstale":"2024-09-13T14:31:16.943420+0000","last_undegraded":"2024-09-13T14:31:16.943420+0000","last_fullsized":"2024-09-13T14:31:16.943420+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_clean_scrub_stamp":"2024-09-13T14:31:03.932624+0000","objects_scrubbed":0,"log_size":482,"log_dups_size":0,"ondisk_log_size":482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:57:45.740628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31588352,"num_objects":482,"num_object_clones":0,"num_object_copies":1446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":482,"num_write_kb":30848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":375,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T14:31:06.414224+0000","last_change":"2024-09-13T14:26:15.835178+0000","last_active":"2024-09-13T14:31:06.414224+0000","last_peered":"2024-09-13T14:31:06.414224+0000","last_clean":"2024-09-13T14:31:06.414224+0000","last_became_active":"2024-09-13T14:26:15.820283+0000","last_became_peered":"2024-09-13T14:26:15.820283+0000","last_unstale":"2024-09-13T14:31:06.414224+0000","last_undegraded":"2024-09-13T14:31:06.414224+0000","last_fullsized":"2024-09-13T14:31:06.414224+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:30.340038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":176,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:13.015169+0000","last_change":"2024-09-13T14:24:55.286576+0000","last_active":"2024-09-13T14:31:13.015169+0000","last_peered":"2024-09-13T14:31:13.015169+0000","last_clean":"2024-09-13T14:31:13.015169+0000","last_became_active":"2024-09-13T14:24:55.286284+0000","last_became_peered":"2024-09-13T14:24:55.286284+0000","last_unstale":"2024-09-13T14:31:13.015169+0000","last_undegraded":"2024-09-13T14:31:13.015169+0000","last_fullsized":"2024-09-13T14:31:13.015169+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:35.589736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":202,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:13.015664+0000","last_change":"2024-09-13T14:27:12.836204+0000","last_active":"2024-09-13T14:31:13.015664+0000","last_peered":"2024-09-13T14:31:13.015664+0000","last_clean":"2024-09-13T14:31:13.015664+0000","last_became_active":"2024-09-13T14:27:12.835900+0000","last_became_peered":"2024-09-13T14:27:12.835900+0000","last_unstale":"2024-09-13T14:31:13.015664+0000","last_undegraded":"2024-09-13T14:31:13.015664+0000","last_fullsized":"2024-09-13T14:31:13.015664+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:36:52.121730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"164'488","reported_seq":500,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:16.993524+0000","last_change":"2024-09-13T14:31:04.954224+0000","last_active":"2024-09-13T14:31:16.993524+0000","last_peered":"2024-09-13T14:31:16.993524+0000","last_clean":"2024-09-13T14:31:16.993524+0000","last_became_active":"2024-09-13T14:31:04.953723+0000","last_became_peered":"2024-09-13T14:31:04.953723+0000","last_unstale":"2024-09-13T14:31:16.993524+0000","last_undegraded":"2024-09-13T14:31:16.993524+0000","last_fullsized":"2024-09-13T14:31:16.993524+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_clean_scrub_stamp":"2024-09-13T14:31:03.932624+0000","objects_scrubbed":0,"log_size":488,"log_dups_size":0,"ondisk_log_size":488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:41:04.638415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31981568,"num_objects":488,"num_object_clones":0,"num_object_copies":1464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":488,"num_write_kb":31232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":556,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:11.965165+0000","last_change":"2024-09-13T14:27:15.863966+0000","last_active":"2024-09-13T14:31:11.965165+0000","last_peered":"2024-09-13T14:31:11.965165+0000","last_clean":"2024-09-13T14:31:11.965165+0000","last_became_active":"2024-09-13T14:27:15.863124+0000","last_became_peered":"2024-09-13T14:27:15.863124+0000","last_unstale":"2024-09-13T14:31:11.965165+0000","last_undegraded":"2024-09-13T14:31:11.965165+0000","last_fullsized":"2024-09-13T14:31:11.965165+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:35:02.715013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":89,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:12.317191+0000","last_change":"2024-09-13T14:27:12.829079+0000","last_active":"2024-09-13T14:31:12.317191+0000","last_peered":"2024-09-13T14:31:12.317191+0000","last_clean":"2024-09-13T14:31:12.317191+0000","last_became_active":"2024-09-13T14:27:12.828619+0000","last_became_peered":"2024-09-13T14:27:12.828619+0000","last_unstale":"2024-09-13T14:31:12.317191+0000","last_undegraded":"2024-09-13T14:31:12.317191+0000","last_fullsized":"2024-09-13T14:31:12.317191+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:39:08.260304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":125,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T14:31:06.460884+0000","last_change":"2024-09-13T14:26:15.817732+0000","last_active":"2024-09-13T14:31:06.460884+0000","last_peered":"2024-09-13T14:31:06.460884+0000","last_clean":"2024-09-13T14:31:06.460884+0000","last_became_active":"2024-09-13T14:26:15.817403+0000","last_became_peered":"2024-09-13T14:26:15.817403+0000","last_unstale":"2024-09-13T14:31:06.460884+0000","last_undegraded":"2024-09-13T14:31:06.460884+0000","last_fullsized":"2024-09-13T14:31:06.460884+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:56:39.961300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":17,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:11.965456+0000","last_change":"2024-09-13T14:31:04.953392+0000","last_active":"2024-09-13T14:31:11.965456+0000","last_peered":"2024-09-13T14:31:11.965456+0000","last_clean":"2024-09-13T14:31:11.965456+0000","last_became_active":"2024-09-13T14:31:04.953114+0000","last_became_peered":"2024-09-13T14:31:04.953114+0000","last_unstale":"2024-09-13T14:31:11.965456+0000","last_undegraded":"2024-09-13T14:31:11.965456+0000","last_fullsized":"2024-09-13T14:31:11.965456+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_clean_scrub_stamp":"2024-09-13T14:31:03.932624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:41:13.856511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":925,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T14:31:06.454080+0000","last_change":"2024-09-13T14:26:18.829131+0000","last_active":"2024-09-13T14:31:06.454080+0000","last_peered":"2024-09-13T14:31:06.454080+0000","last_clean":"2024-09-13T14:31:06.454080+0000","last_became_active":"2024-09-13T14:26:18.828619+0000","last_became_peered":"2024-09-13T14:26:18.828619+0000","last_unstale":"2024-09-13T14:31:06.454080+0000","last_undegraded":"2024-09-13T14:31:06.454080+0000","last_fullsized":"2024-09-13T14:31:06.454080+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:24:37.073618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":129,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T14:31:06.803961+0000","last_change":"2024-09-13T14:23:18.042891+0000","last_active":"2024-09-13T14:31:06.803961+0000","last_peered":"2024-09-13T14:31:06.803961+0000","last_clean":"2024-09-13T14:31:06.803961+0000","last_became_active":"2024-09-13T14:23:18.042562+0000","last_became_peered":"2024-09-13T14:23:18.042562+0000","last_unstale":"2024-09-13T14:31:06.803961+0000","last_undegraded":"2024-09-13T14:31:06.803961+0000","last_fullsized":"2024-09-13T14:31:06.803961+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:30:42.996661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":169,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:11.965632+0000","last_change":"2024-09-13T14:27:12.832666+0000","last_active":"2024-09-13T14:31:11.965632+0000","last_peered":"2024-09-13T14:31:11.965632+0000","last_clean":"2024-09-13T14:31:11.965632+0000","last_became_active":"2024-09-13T14:27:12.832332+0000","last_became_peered":"2024-09-13T14:27:12.832332+0000","last_unstale":"2024-09-13T14:31:11.965632+0000","last_undegraded":"2024-09-13T14:31:11.965632+0000","last_fullsized":"2024-09-13T14:31:11.965632+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:07:17.724156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"164'400","reported_seq":417,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:16.508258+0000","last_change":"2024-09-13T14:31:04.956472+0000","last_active":"2024-09-13T14:31:16.508258+0000","last_peered":"2024-09-13T14:31:16.508258+0000","last_clean":"2024-09-13T14:31:16.508258+0000","last_became_active":"2024-09-13T14:31:04.956181+0000","last_became_peered":"2024-09-13T14:31:04.956181+0000","last_unstale":"2024-09-13T14:31:16.508258+0000","last_undegraded":"2024-09-13T14:31:16.508258+0000","last_fullsized":"2024-09-13T14:31:16.508258+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_clean_scrub_stamp":"2024-09-13T14:31:03.932624+0000","objects_scrubbed":0,"log_size":400,"log_dups_size":0,"ondisk_log_size":400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:00:30.194516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26214400,"num_objects":400,"num_object_clones":0,"num_object_copies":1200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":400,"num_write_kb":25600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":154,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T14:31:06.454175+0000","last_change":"2024-09-13T14:27:12.843038+0000","last_active":"2024-09-13T14:31:06.454175+0000","last_peered":"2024-09-13T14:31:06.454175+0000","last_clean":"2024-09-13T14:31:06.454175+0000","last_became_active":"2024-09-13T14:27:12.842663+0000","last_became_peered":"2024-09-13T14:27:12.842663+0000","last_unstale":"2024-09-13T14:31:06.454175+0000","last_undegraded":"2024-09-13T14:31:06.454175+0000","last_fullsized":"2024-09-13T14:31:06.454175+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:23:38.880811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":906,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:13.015628+0000","last_change":"2024-09-13T14:30:58.928302+0000","last_active":"2024-09-13T14:31:13.015628+0000","last_peered":"2024-09-13T14:31:13.015628+0000","last_clean":"2024-09-13T14:31:13.015628+0000","last_became_active":"2024-09-13T14:30:58.927718+0000","last_became_peered":"2024-09-13T14:30:58.927718+0000","last_unstale":"2024-09-13T14:31:13.015628+0000","last_undegraded":"2024-09-13T14:31:13.015628+0000","last_fullsized":"2024-09-13T14:31:13.015628+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:37.323036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":81,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T14:31:06.453854+0000","last_change":"2024-09-13T14:26:15.819032+0000","last_active":"2024-09-13T14:31:06.453854+0000","last_peered":"2024-09-13T14:31:06.453854+0000","last_clean":"2024-09-13T14:31:06.453854+0000","last_became_active":"2024-09-13T14:26:15.818675+0000","last_became_peered":"2024-09-13T14:26:15.818675+0000","last_unstale":"2024-09-13T14:31:06.453854+0000","last_undegraded":"2024-09-13T14:31:06.453854+0000","last_fullsized":"2024-09-13T14:31:06.453854+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:37:13.628256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"164'341","reported_seq":353,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:16.358795+0000","last_change":"2024-09-13T14:31:04.954718+0000","last_active":"2024-09-13T14:31:16.358795+0000","last_peered":"2024-09-13T14:31:16.358795+0000","last_clean":"2024-09-13T14:31:16.358795+0000","last_became_active":"2024-09-13T14:31:04.954428+0000","last_became_peered":"2024-09-13T14:31:04.954428+0000","last_unstale":"2024-09-13T14:31:16.358795+0000","last_undegraded":"2024-09-13T14:31:16.358795+0000","last_fullsized":"2024-09-13T14:31:16.358795+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_clean_scrub_stamp":"2024-09-13T14:31:03.932624+0000","objects_scrubbed":0,"log_size":341,"log_dups_size":0,"ondisk_log_size":341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:45:36.697605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22347776,"num_objects":341,"num_object_clones":0,"num_object_copies":1023,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":341,"num_write_kb":21824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":88,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T14:31:06.453964+0000","last_change":"2024-09-13T14:27:12.843268+0000","last_active":"2024-09-13T14:31:06.453964+0000","last_peered":"2024-09-13T14:31:06.453964+0000","last_clean":"2024-09-13T14:31:06.453964+0000","last_became_active":"2024-09-13T14:27:12.842867+0000","last_became_peered":"2024-09-13T14:27:12.842867+0000","last_unstale":"2024-09-13T14:31:06.453964+0000","last_undegraded":"2024-09-13T14:31:06.453964+0000","last_fullsized":"2024-09-13T14:31:06.453964+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:03:59.575866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":652,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T14:31:12.000843+0000","last_change":"2024-09-13T14:23:18.038847+0000","last_active":"2024-09-13T14:31:12.000843+0000","last_peered":"2024-09-13T14:31:12.000843+0000","last_clean":"2024-09-13T14:31:12.000843+0000","last_became_active":"2024-09-13T14:23:18.038188+0000","last_became_peered":"2024-09-13T14:23:18.038188+0000","last_unstale":"2024-09-13T14:31:12.000843+0000","last_undegraded":"2024-09-13T14:31:12.000843+0000","last_fullsized":"2024-09-13T14:31:12.000843+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:30:37.213496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":166,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:13.015467+0000","last_change":"2024-09-13T14:23:18.034427+0000","last_active":"2024-09-13T14:31:13.015467+0000","last_peered":"2024-09-13T14:31:13.015467+0000","last_clean":"2024-09-13T14:31:13.015467+0000","last_became_active":"2024-09-13T14:23:18.034092+0000","last_became_peered":"2024-09-13T14:23:18.034092+0000","last_unstale":"2024-09-13T14:31:13.015467+0000","last_undegraded":"2024-09-13T14:31:13.015467+0000","last_fullsized":"2024-09-13T14:31:13.015467+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:21.755115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"164'515","reported_seq":525,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:17.246643+0000","last_change":"2024-09-13T14:31:04.957425+0000","last_active":"2024-09-13T14:31:17.246643+0000","last_peered":"2024-09-13T14:31:17.246643+0000","last_clean":"2024-09-13T14:31:17.246643+0000","last_became_active":"2024-09-13T14:31:04.957101+0000","last_became_peered":"2024-09-13T14:31:04.957101+0000","last_unstale":"2024-09-13T14:31:17.246643+0000","last_undegraded":"2024-09-13T14:31:17.246643+0000","last_fullsized":"2024-09-13T14:31:17.246643+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_clean_scrub_stamp":"2024-09-13T14:31:03.932624+0000","objects_scrubbed":0,"log_size":515,"log_dups_size":0,"ondisk_log_size":515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:14:30.110858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33751040,"num_objects":515,"num_object_clones":0,"num_object_copies":1545,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":515,"num_write_kb":32960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":182,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:11.966221+0000","last_change":"2024-09-13T14:24:55.290711+0000","last_active":"2024-09-13T14:31:11.966221+0000","last_peered":"2024-09-13T14:31:11.966221+0000","last_clean":"2024-09-13T14:31:11.966221+0000","last_became_active":"2024-09-13T14:24:55.288549+0000","last_became_peered":"2024-09-13T14:24:55.288549+0000","last_unstale":"2024-09-13T14:31:11.966221+0000","last_undegraded":"2024-09-13T14:31:11.966221+0000","last_fullsized":"2024-09-13T14:31:11.966221+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:55:21.763931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":565,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T14:31:05.953883+0000","last_change":"2024-09-13T14:26:18.827876+0000","last_active":"2024-09-13T14:31:05.953883+0000","last_peered":"2024-09-13T14:31:05.953883+0000","last_clean":"2024-09-13T14:31:05.953883+0000","last_became_active":"2024-09-13T14:26:18.827317+0000","last_became_peered":"2024-09-13T14:26:18.827317+0000","last_unstale":"2024-09-13T14:31:05.953883+0000","last_undegraded":"2024-09-13T14:31:05.953883+0000","last_fullsized":"2024-09-13T14:31:05.953883+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:31:48.611286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":180,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:11.964755+0000","last_change":"2024-09-13T14:24:55.281593+0000","last_active":"2024-09-13T14:31:11.964755+0000","last_peered":"2024-09-13T14:31:11.964755+0000","last_clean":"2024-09-13T14:31:11.964755+0000","last_became_active":"2024-09-13T14:24:55.280805+0000","last_became_peered":"2024-09-13T14:24:55.280805+0000","last_unstale":"2024-09-13T14:31:11.964755+0000","last_undegraded":"2024-09-13T14:31:11.964755+0000","last_fullsized":"2024-09-13T14:31:11.964755+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:14.806713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"164'491","reported_seq":502,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:16.999798+0000","last_change":"2024-09-13T14:31:04.956103+0000","last_active":"2024-09-13T14:31:16.999798+0000","last_peered":"2024-09-13T14:31:16.999798+0000","last_clean":"2024-09-13T14:31:16.999798+0000","last_became_active":"2024-09-13T14:31:04.955773+0000","last_became_peered":"2024-09-13T14:31:04.955773+0000","last_unstale":"2024-09-13T14:31:16.999798+0000","last_undegraded":"2024-09-13T14:31:16.999798+0000","last_fullsized":"2024-09-13T14:31:16.999798+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_clean_scrub_stamp":"2024-09-13T14:31:03.932624+0000","objects_scrubbed":0,"log_size":491,"log_dups_size":0,"ondisk_log_size":491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:36:08.391308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32178176,"num_objects":491,"num_object_clones":0,"num_object_copies":1473,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":491,"num_write_kb":31424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":191,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:13.015355+0000","last_change":"2024-09-13T14:26:15.830309+0000","last_active":"2024-09-13T14:31:13.015355+0000","last_peered":"2024-09-13T14:31:13.015355+0000","last_clean":"2024-09-13T14:31:13.015355+0000","last_became_active":"2024-09-13T14:26:15.823041+0000","last_became_peered":"2024-09-13T14:26:15.823041+0000","last_unstale":"2024-09-13T14:31:13.015355+0000","last_undegraded":"2024-09-13T14:31:13.015355+0000","last_fullsized":"2024-09-13T14:31:13.015355+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:23:08.158576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":595,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T14:31:05.954012+0000","last_change":"2024-09-13T14:27:12.848354+0000","last_active":"2024-09-13T14:31:05.954012+0000","last_peered":"2024-09-13T14:31:05.954012+0000","last_clean":"2024-09-13T14:31:05.954012+0000","last_became_active":"2024-09-13T14:27:12.832182+0000","last_became_peered":"2024-09-13T14:27:12.832182+0000","last_unstale":"2024-09-13T14:31:05.954012+0000","last_undegraded":"2024-09-13T14:31:05.954012+0000","last_fullsized":"2024-09-13T14:31:05.954012+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:46:42.064509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":113,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T14:31:06.803412+0000","last_change":"2024-09-13T14:11:30.256034+0000","last_active":"2024-09-13T14:31:06.803412+0000","last_peered":"2024-09-13T14:31:06.803412+0000","last_clean":"2024-09-13T14:31:06.803412+0000","last_became_active":"2024-09-13T14:11:30.255727+0000","last_became_peered":"2024-09-13T14:11:30.255727+0000","last_unstale":"2024-09-13T14:31:06.803412+0000","last_undegraded":"2024-09-13T14:31:06.803412+0000","last_fullsized":"2024-09-13T14:31:06.803412+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:21:35.817014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":136,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T14:31:06.803800+0000","last_change":"2024-09-13T14:26:15.825390+0000","last_active":"2024-09-13T14:31:06.803800+0000","last_peered":"2024-09-13T14:31:06.803800+0000","last_clean":"2024-09-13T14:31:06.803800+0000","last_became_active":"2024-09-13T14:26:15.825084+0000","last_became_peered":"2024-09-13T14:26:15.825084+0000","last_unstale":"2024-09-13T14:31:06.803800+0000","last_undegraded":"2024-09-13T14:31:06.803800+0000","last_fullsized":"2024-09-13T14:31:06.803800+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:44.680451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":136,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T14:31:06.803859+0000","last_change":"2024-09-13T14:11:29.237013+0000","last_active":"2024-09-13T14:31:06.803859+0000","last_peered":"2024-09-13T14:31:06.803859+0000","last_clean":"2024-09-13T14:31:06.803859+0000","last_became_active":"2024-09-13T14:11:29.236643+0000","last_became_peered":"2024-09-13T14:11:29.236643+0000","last_unstale":"2024-09-13T14:31:06.803859+0000","last_undegraded":"2024-09-13T14:31:06.803859+0000","last_fullsized":"2024-09-13T14:31:06.803859+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:33:58.438693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":111,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T14:31:05.953499+0000","last_change":"2024-09-13T14:27:12.843904+0000","last_active":"2024-09-13T14:31:05.953499+0000","last_peered":"2024-09-13T14:31:05.953499+0000","last_clean":"2024-09-13T14:31:05.953499+0000","last_became_active":"2024-09-13T14:27:12.842566+0000","last_became_peered":"2024-09-13T14:27:12.842566+0000","last_unstale":"2024-09-13T14:31:05.953499+0000","last_undegraded":"2024-09-13T14:31:05.953499+0000","last_fullsized":"2024-09-13T14:31:05.953499+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:11:46.253003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":132,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T14:31:06.414680+0000","last_change":"2024-09-13T14:24:55.287275+0000","last_active":"2024-09-13T14:31:06.414680+0000","last_peered":"2024-09-13T14:31:06.414680+0000","last_clean":"2024-09-13T14:31:06.414680+0000","last_became_active":"2024-09-13T14:24:55.286919+0000","last_became_peered":"2024-09-13T14:24:55.286919+0000","last_unstale":"2024-09-13T14:31:06.414680+0000","last_undegraded":"2024-09-13T14:31:06.414680+0000","last_fullsized":"2024-09-13T14:31:06.414680+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:10.790711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":138,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T14:31:06.454178+0000","last_change":"2024-09-13T14:26:15.819037+0000","last_active":"2024-09-13T14:31:06.454178+0000","last_peered":"2024-09-13T14:31:06.454178+0000","last_clean":"2024-09-13T14:31:06.454178+0000","last_became_active":"2024-09-13T14:26:15.818721+0000","last_became_peered":"2024-09-13T14:26:15.818721+0000","last_unstale":"2024-09-13T14:31:06.454178+0000","last_undegraded":"2024-09-13T14:31:06.454178+0000","last_fullsized":"2024-09-13T14:31:06.454178+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:44:06.419136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":135,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:11.964893+0000","last_change":"2024-09-13T14:24:55.287509+0000","last_active":"2024-09-13T14:31:11.964893+0000","last_peered":"2024-09-13T14:31:11.964893+0000","last_clean":"2024-09-13T14:31:11.964893+0000","last_became_active":"2024-09-13T14:24:55.287193+0000","last_became_peered":"2024-09-13T14:24:55.287193+0000","last_unstale":"2024-09-13T14:31:11.964893+0000","last_undegraded":"2024-09-13T14:31:11.964893+0000","last_fullsized":"2024-09-13T14:31:11.964893+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:14:10.603307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":176,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:13.015552+0000","last_change":"2024-09-13T14:24:55.289055+0000","last_active":"2024-09-13T14:31:13.015552+0000","last_peered":"2024-09-13T14:31:13.015552+0000","last_clean":"2024-09-13T14:31:13.015552+0000","last_became_active":"2024-09-13T14:24:55.288762+0000","last_became_peered":"2024-09-13T14:24:55.288762+0000","last_unstale":"2024-09-13T14:31:13.015552+0000","last_undegraded":"2024-09-13T14:31:13.015552+0000","last_fullsized":"2024-09-13T14:31:13.015552+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:37:26.639877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":191,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:11.964552+0000","last_change":"2024-09-13T14:26:15.824540+0000","last_active":"2024-09-13T14:31:11.964552+0000","last_peered":"2024-09-13T14:31:11.964552+0000","last_clean":"2024-09-13T14:31:11.964552+0000","last_became_active":"2024-09-13T14:26:15.824118+0000","last_became_peered":"2024-09-13T14:26:15.824118+0000","last_unstale":"2024-09-13T14:31:11.964552+0000","last_undegraded":"2024-09-13T14:31:11.964552+0000","last_fullsized":"2024-09-13T14:31:11.964552+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:56.677401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":138,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T14:31:05.953984+0000","last_change":"2024-09-13T14:26:15.823892+0000","last_active":"2024-09-13T14:31:05.953984+0000","last_peered":"2024-09-13T14:31:05.953984+0000","last_clean":"2024-09-13T14:31:05.953984+0000","last_became_active":"2024-09-13T14:26:15.823582+0000","last_became_peered":"2024-09-13T14:26:15.823582+0000","last_unstale":"2024-09-13T14:31:05.953984+0000","last_undegraded":"2024-09-13T14:31:05.953984+0000","last_fullsized":"2024-09-13T14:31:05.953984+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:53:09.918997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":214,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:11.964924+0000","last_change":"2024-09-13T14:27:12.830642+0000","last_active":"2024-09-13T14:31:11.964924+0000","last_peered":"2024-09-13T14:31:11.964924+0000","last_clean":"2024-09-13T14:31:11.964924+0000","last_became_active":"2024-09-13T14:27:12.830026+0000","last_became_peered":"2024-09-13T14:27:12.830026+0000","last_unstale":"2024-09-13T14:31:11.964924+0000","last_undegraded":"2024-09-13T14:31:11.964924+0000","last_fullsized":"2024-09-13T14:31:11.964924+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:12:31.783552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":178,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:13.015691+0000","last_change":"2024-09-13T14:24:55.283070+0000","last_active":"2024-09-13T14:31:13.015691+0000","last_peered":"2024-09-13T14:31:13.015691+0000","last_clean":"2024-09-13T14:31:13.015691+0000","last_became_active":"2024-09-13T14:24:55.282591+0000","last_became_peered":"2024-09-13T14:24:55.282591+0000","last_unstale":"2024-09-13T14:31:13.015691+0000","last_undegraded":"2024-09-13T14:31:13.015691+0000","last_fullsized":"2024-09-13T14:31:13.015691+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:35:21.896294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":164,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:11.965117+0000","last_change":"2024-09-13T14:26:15.823524+0000","last_active":"2024-09-13T14:31:11.965117+0000","last_peered":"2024-09-13T14:31:11.965117+0000","last_clean":"2024-09-13T14:31:11.965117+0000","last_became_active":"2024-09-13T14:26:15.823162+0000","last_became_peered":"2024-09-13T14:26:15.823162+0000","last_unstale":"2024-09-13T14:31:11.965117+0000","last_undegraded":"2024-09-13T14:31:11.965117+0000","last_fullsized":"2024-09-13T14:31:11.965117+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:17:13.664812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":204,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:11.964980+0000","last_change":"2024-09-13T14:26:15.823703+0000","last_active":"2024-09-13T14:31:11.964980+0000","last_peered":"2024-09-13T14:31:11.964980+0000","last_clean":"2024-09-13T14:31:11.964980+0000","last_became_active":"2024-09-13T14:26:15.823249+0000","last_became_peered":"2024-09-13T14:26:15.823249+0000","last_unstale":"2024-09-13T14:31:11.964980+0000","last_undegraded":"2024-09-13T14:31:11.964980+0000","last_fullsized":"2024-09-13T14:31:11.964980+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:42:40.209150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":161,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T14:31:06.803722+0000","last_change":"2024-09-13T14:27:12.841881+0000","last_active":"2024-09-13T14:31:06.803722+0000","last_peered":"2024-09-13T14:31:06.803722+0000","last_clean":"2024-09-13T14:31:06.803722+0000","last_became_active":"2024-09-13T14:27:12.841247+0000","last_became_peered":"2024-09-13T14:27:12.841247+0000","last_unstale":"2024-09-13T14:31:06.803722+0000","last_undegraded":"2024-09-13T14:31:06.803722+0000","last_fullsized":"2024-09-13T14:31:06.803722+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:59:16.937848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":187,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:11.965802+0000","last_change":"2024-09-13T14:27:12.838453+0000","last_active":"2024-09-13T14:31:11.965802+0000","last_peered":"2024-09-13T14:31:11.965802+0000","last_clean":"2024-09-13T14:31:11.965802+0000","last_became_active":"2024-09-13T14:27:12.838124+0000","last_became_peered":"2024-09-13T14:27:12.838124+0000","last_unstale":"2024-09-13T14:31:11.965802+0000","last_undegraded":"2024-09-13T14:31:11.965802+0000","last_fullsized":"2024-09-13T14:31:11.965802+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:24.259191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":169,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:13.015468+0000","last_change":"2024-09-13T14:23:18.035101+0000","last_active":"2024-09-13T14:31:13.015468+0000","last_peered":"2024-09-13T14:31:13.015468+0000","last_clean":"2024-09-13T14:31:13.015468+0000","last_became_active":"2024-09-13T14:23:18.034724+0000","last_became_peered":"2024-09-13T14:23:18.034724+0000","last_unstale":"2024-09-13T14:31:13.015468+0000","last_undegraded":"2024-09-13T14:31:13.015468+0000","last_fullsized":"2024-09-13T14:31:13.015468+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:39.358144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":159,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T14:31:06.803776+0000","last_change":"2024-09-13T14:27:12.834931+0000","last_active":"2024-09-13T14:31:06.803776+0000","last_peered":"2024-09-13T14:31:06.803776+0000","last_clean":"2024-09-13T14:31:06.803776+0000","last_became_active":"2024-09-13T14:27:12.834572+0000","last_became_peered":"2024-09-13T14:27:12.834572+0000","last_unstale":"2024-09-13T14:31:06.803776+0000","last_undegraded":"2024-09-13T14:31:06.803776+0000","last_fullsized":"2024-09-13T14:31:06.803776+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:54:21.155689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":143,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T14:31:06.454037+0000","last_change":"2024-09-13T14:26:15.826283+0000","last_active":"2024-09-13T14:31:06.454037+0000","last_peered":"2024-09-13T14:31:06.454037+0000","last_clean":"2024-09-13T14:31:06.454037+0000","last_became_active":"2024-09-13T14:26:15.820774+0000","last_became_peered":"2024-09-13T14:26:15.820774+0000","last_unstale":"2024-09-13T14:31:06.454037+0000","last_undegraded":"2024-09-13T14:31:06.454037+0000","last_fullsized":"2024-09-13T14:31:06.454037+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:28:39.849420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":141,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T14:31:06.803661+0000","last_change":"2024-09-13T14:24:55.285384+0000","last_active":"2024-09-13T14:31:06.803661+0000","last_peered":"2024-09-13T14:31:06.803661+0000","last_clean":"2024-09-13T14:31:06.803661+0000","last_became_active":"2024-09-13T14:24:55.285081+0000","last_became_peered":"2024-09-13T14:24:55.285081+0000","last_unstale":"2024-09-13T14:31:06.803661+0000","last_undegraded":"2024-09-13T14:31:06.803661+0000","last_fullsized":"2024-09-13T14:31:06.803661+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:47:07.946120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":115,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T14:31:06.454182+0000","last_change":"2024-09-13T14:11:29.240351+0000","last_active":"2024-09-13T14:31:06.454182+0000","last_peered":"2024-09-13T14:31:06.454182+0000","last_clean":"2024-09-13T14:31:06.454182+0000","last_became_active":"2024-09-13T14:11:29.240060+0000","last_became_peered":"2024-09-13T14:11:29.240060+0000","last_unstale":"2024-09-13T14:31:06.454182+0000","last_undegraded":"2024-09-13T14:31:06.454182+0000","last_fullsized":"2024-09-13T14:31:06.454182+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:34:36.126368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":210,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:11.965210+0000","last_change":"2024-09-13T14:27:12.839943+0000","last_active":"2024-09-13T14:31:11.965210+0000","last_peered":"2024-09-13T14:31:11.965210+0000","last_clean":"2024-09-13T14:31:11.965210+0000","last_became_active":"2024-09-13T14:27:12.839534+0000","last_became_peered":"2024-09-13T14:27:12.839534+0000","last_unstale":"2024-09-13T14:31:11.965210+0000","last_undegraded":"2024-09-13T14:31:11.965210+0000","last_fullsized":"2024-09-13T14:31:11.965210+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:34:56.677019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":199,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:11.965703+0000","last_change":"2024-09-13T14:27:12.836567+0000","last_active":"2024-09-13T14:31:11.965703+0000","last_peered":"2024-09-13T14:31:11.965703+0000","last_clean":"2024-09-13T14:31:11.965703+0000","last_became_active":"2024-09-13T14:27:12.836271+0000","last_became_peered":"2024-09-13T14:27:12.836271+0000","last_unstale":"2024-09-13T14:31:11.965703+0000","last_undegraded":"2024-09-13T14:31:11.965703+0000","last_fullsized":"2024-09-13T14:31:11.965703+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:25:32.910438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":136,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T14:31:06.803531+0000","last_change":"2024-09-13T14:26:15.824075+0000","last_active":"2024-09-13T14:31:06.803531+0000","last_peered":"2024-09-13T14:31:06.803531+0000","last_clean":"2024-09-13T14:31:06.803531+0000","last_became_active":"2024-09-13T14:26:15.823762+0000","last_became_peered":"2024-09-13T14:26:15.823762+0000","last_unstale":"2024-09-13T14:31:06.803531+0000","last_undegraded":"2024-09-13T14:31:06.803531+0000","last_fullsized":"2024-09-13T14:31:06.803531+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:26:51.088388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":193,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:13.015049+0000","last_change":"2024-09-13T14:27:12.844201+0000","last_active":"2024-09-13T14:31:13.015049+0000","last_peered":"2024-09-13T14:31:13.015049+0000","last_clean":"2024-09-13T14:31:13.015049+0000","last_became_active":"2024-09-13T14:27:12.834075+0000","last_became_peered":"2024-09-13T14:27:12.834075+0000","last_unstale":"2024-09-13T14:31:13.015049+0000","last_undegraded":"2024-09-13T14:31:13.015049+0000","last_fullsized":"2024-09-13T14:31:13.015049+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:06.657030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":360120320,"num_objects":5495,"num_object_clones":0,"num_object_copies":16485,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5495,"num_write_kb":351680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1037303808,"data_stored":1037303808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5495,"ondisk_log_size":5495,"up":48,"acting":48,"num_store_stats":6},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":616,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":17278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18272,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16793600,"data_stored":16777292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234249,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":347216,"kb_used_data":199624,"kb_used_omap":1,"kb_used_meta":147582,"kb_avail":93397936,"statfs":{"total":95995035648,"available":95639486464,"internally_reserved":0,"allocated":204414976,"data_stored":203587470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":151124427},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495885,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":370704,"kb_used_data":245256,"kb_used_omap":1,"kb_used_meta":125438,"kb_avail":93374448,"statfs":{"total":95995035648,"available":95615434752,"internally_reserved":0,"allocated":251142144,"data_stored":250315185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":128448967},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757522,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":362364,"kb_used_data":132724,"kb_used_omap":1,"kb_used_meta":229630,"kb_avail":93382788,"statfs":{"total":95995035648,"available":95623974912,"internally_reserved":0,"allocated":135909376,"data_stored":135097505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":235141574},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019158,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":351944,"kb_used_data":144448,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":93393208,"statfs":{"total":95995035648,"available":95634644992,"internally_reserved":0,"allocated":147914752,"data_stored":147094882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":212466118},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435342,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":100156,"kb_used_data":1840,"kb_used_omap":5,"kb_used_meta":98298,"kb_avail":93644996,"statfs":{"total":95995035648,"available":95892475904,"internally_reserved":0,"allocated":1884160,"data_stored":1076147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":100657278},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":446676598869,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":425201762395,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726925922,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":321988,"kb_used_data":77884,"kb_used_omap":7,"kb_used_meta":244088,"kb_avail":93423164,"statfs":{"total":95995035648,"available":95665319936,"internally_reserved":0,"allocated":79753216,"data_stored":78933992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":249946408},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":149684224,"data_stored":149684224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":172163072,"data_stored":172163072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":215810048,"data_stored":215810048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":247201792,"data_stored":247201792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":252444672,"data_stored":252444672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:31:19.045 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-13T14:31:19.046 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:31:19.046 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-13T14:31:19.047 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:31:19.055 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:19.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:19.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:19.158 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:19.191 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:19.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:19.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:19.256 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:19 smithi102 ceph-mon[110183]: pgmap v493: 89 pgs: 89 active+clean; 348 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 34 MiB/s wr, 549 op/s 2024-09-13T14:31:19.257 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:19 smithi102 ceph-mon[110183]: from='client.35049 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:31:19.257 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:19 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:31:19.257 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:19 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:31:19.257 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:19 smithi102 ceph-mon[115269]: pgmap v493: 89 pgs: 89 active+clean; 348 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 34 MiB/s wr, 549 op/s 2024-09-13T14:31:19.257 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:19 smithi102 ceph-mon[115269]: from='client.35049 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:31:19.257 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:19 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:31:19.257 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:19 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:31:19.294 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:19.331 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:19.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:19.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:19.433 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:19.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:19.458 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:19.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:19.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:19 smithi195 ceph-mon[104262]: pgmap v493: 89 pgs: 89 active+clean; 348 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 34 MiB/s wr, 549 op/s 2024-09-13T14:31:19.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:19 smithi195 ceph-mon[104262]: from='client.35049 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:31:19.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:19 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:31:19.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:19 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:31:19.559 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:19.590 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:19.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:19.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:19.691 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:19.728 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:19.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:19.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:19.830 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:19.882 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:19.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:19.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:19.894 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 5 16 15879 15863 198.272 164.25 0.00373434 0.00503813 2024-09-13T14:31:19.985 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:31:20.024 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:20.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:20.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:31:20.055 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:31:20.125 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:20.160 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:20.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:20.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:20.261 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:20.303 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:20.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:20.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:20.405 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:31:20.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:20.441 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:20.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:31:20.542 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:20.572 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:20.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:20.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:20.610 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:31:20.610 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-13T14:31:20.610 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-09-13T14:31:20.610 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-09-13T14:31:20.610 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:31:20.620 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:31:20.674 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:20.713 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:20.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:20.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:20.815 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:20.851 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:20.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:20.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:20.895 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 6 16 18560 18544 193.148 167.562 0.00919 0.00517074 2024-09-13T14:31:20.953 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:20.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:20.982 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:20.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:21.084 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:21.115 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:21.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:21.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:21.218 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:21.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:21 smithi195 ceph-mon[104262]: pgmap v494: 89 pgs: 89 active+clean; 530 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 53 MiB/s wr, 841 op/s 2024-09-13T14:31:21.248 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:21.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:21.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:21.350 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:21.354 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T14:31:21.388 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:21.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:21.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:21.405 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:21 smithi102 ceph-mon[110183]: pgmap v494: 89 pgs: 89 active+clean; 530 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 53 MiB/s wr, 841 op/s 2024-09-13T14:31:21.405 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:21 smithi102 ceph-mon[115269]: pgmap v494: 89 pgs: 89 active+clean; 530 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 53 MiB/s wr, 841 op/s 2024-09-13T14:31:21.490 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:21.520 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:21.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:21.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:21.624 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:21.655 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:21.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:21.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:21.754 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:31:21.757 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:21.806 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:21.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:21.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:21.895 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 7 16 21095 21079 188.187 158.438 0.00296676 0.00530851 2024-09-13T14:31:21.908 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:21.936 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:21.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:21.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:22.038 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:31:22.078 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:22.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:22.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:31:22.180 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:22.219 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:22.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:22.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:22.321 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:22.346 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:22.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:22.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:22.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:22 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/340308164' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:31:22.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:22 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/340308164' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:31:22.431 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:22 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/340308164' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:31:22.449 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:22.479 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:22.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:22.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:22.581 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:31:22.617 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:22.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:22.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:31:22.718 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:22.756 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:22.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:22.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:22.858 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:22.887 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:22.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:22.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:22.895 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 8 16 23406 23390 182.717 144.438 0.00824865 0.00546785 2024-09-13T14:31:22.989 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:31:23.027 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:23.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:23.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:31:23.129 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:23.160 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:23.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:23.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:23.223 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:23 smithi195 ceph-mon[104262]: pgmap v495: 89 pgs: 89 active+clean; 530 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 53 MiB/s wr, 841 op/s 2024-09-13T14:31:23.223 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:23 smithi195 ceph-mon[104262]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:31:23.223 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:23 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/340308164' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:31:23.223 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:23 smithi195 ceph-mon[104262]: osdmap e165: 8 total, 8 up, 6 in 2024-09-13T14:31:23.264 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:23.291 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:23.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:23.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:23.393 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:31:23.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:23 smithi102 ceph-mon[110183]: pgmap v495: 89 pgs: 89 active+clean; 530 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 53 MiB/s wr, 841 op/s 2024-09-13T14:31:23.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:23 smithi102 ceph-mon[110183]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:31:23.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:23 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/340308164' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:31:23.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:23 smithi102 ceph-mon[110183]: osdmap e165: 8 total, 8 up, 6 in 2024-09-13T14:31:23.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:23 smithi102 ceph-mon[115269]: pgmap v495: 89 pgs: 89 active+clean; 530 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 53 MiB/s wr, 841 op/s 2024-09-13T14:31:23.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:23 smithi102 ceph-mon[115269]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:31:23.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:23 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/340308164' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:31:23.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:23 smithi102 ceph-mon[115269]: osdmap e165: 8 total, 8 up, 6 in 2024-09-13T14:31:23.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:23.436 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:23.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:31:23.537 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:23.568 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:23.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:23.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:23.671 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:23.699 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:23.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:23.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:23.801 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:31:23.839 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:23.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:23.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:31:23.895 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 9 16 25694 25678 178.303 143 0.0043138 0.00560245 2024-09-13T14:31:23.941 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:31:23.962 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:31:23.962 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-09-13T14:31:23.962 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-09-13T14:31:23.962 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-13T14:31:23.962 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.962 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:90451 5.2 5:42ca4429:::benchmark_data_smithi102_155989_object90450:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:31:23.962 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:25.550923+0000", 2024-09-13T14:31:23.962 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 238.405505011, 2024-09-13T14:31:23.962 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.064851735999999993, 2024-09-13T14:31:23.963 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:31:23.963 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:31:23.963 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:31:23.963 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:31:23.963 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:31:23.963 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 90451 2024-09-13T14:31:23.963 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.963 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:31:23.963 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.963 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:31:23.963 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.550923+0000", 2024-09-13T14:31:23.963 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:23.963 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.963 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.963 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:31:23.964 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.550923+0000", 2024-09-13T14:31:23.964 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:23.964 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.964 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.964 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:31:23.964 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.550922+0000", 2024-09-13T14:31:23.964 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:31:23.964 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.964 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.964 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:31:23.964 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.550963+0000", 2024-09-13T14:31:23.964 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1820000000000003e-05 2024-09-13T14:31:23.964 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.964 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.965 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:31:23.965 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.550977+0000", 2024-09-13T14:31:23.965 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3896999999999999e-05 2024-09-13T14:31:23.965 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.965 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.965 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:31:23.965 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.550986+0000", 2024-09-13T14:31:23.965 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2930000000000001e-06 2024-09-13T14:31:23.965 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.965 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.965 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:23.965 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.551199+0000", 2024-09-13T14:31:23.965 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00021318699999999999 2024-09-13T14:31:23.966 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.966 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.966 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:31:23.966 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.551662+0000", 2024-09-13T14:31:23.966 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00046375099999999999 2024-09-13T14:31:23.966 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.966 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.966 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:31:23.966 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.552302+0000", 2024-09-13T14:31:23.966 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00063985400000000003 2024-09-13T14:31:23.966 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.966 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.966 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:31:23.967 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.558678+0000", 2024-09-13T14:31:23.967 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0063757960000000004 2024-09-13T14:31:23.967 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.967 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.967 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:23.967 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.558722+0000", 2024-09-13T14:31:23.967 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3628000000000003e-05 2024-09-13T14:31:23.967 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.967 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.967 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:23.967 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.615639+0000", 2024-09-13T14:31:23.967 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.056917275000000003 2024-09-13T14:31:23.968 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.968 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.968 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:31:23.968 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.615737+0000", 2024-09-13T14:31:23.968 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.7643e-05 2024-09-13T14:31:23.968 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.968 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.968 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:31:23.968 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.615775+0000", 2024-09-13T14:31:23.968 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8481e-05 2024-09-13T14:31:23.968 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:23.968 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:31:23.969 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:23.969 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.969 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.969 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:90452 5.2 5:4ad05d41:::benchmark_data_smithi102_155989_object90451:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:31:23.969 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:25.551004+0000", 2024-09-13T14:31:23.969 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 238.40542468300001, 2024-09-13T14:31:23.969 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.064965966999999999, 2024-09-13T14:31:23.969 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:31:23.969 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:31:23.969 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:31:23.969 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:31:23.969 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:31:23.969 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 90452 2024-09-13T14:31:23.969 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.970 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:31:23.970 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.970 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:31:23.970 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.551004+0000", 2024-09-13T14:31:23.970 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:23.970 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.970 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.970 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:31:23.970 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.551004+0000", 2024-09-13T14:31:23.970 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:23.970 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.970 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.970 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:31:23.970 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.551003+0000", 2024-09-13T14:31:23.970 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:31:23.971 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.971 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.971 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:31:23.971 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.551036+0000", 2024-09-13T14:31:23.971 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.3015999999999997e-05 2024-09-13T14:31:23.971 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.971 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.971 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:31:23.971 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.551045+0000", 2024-09-13T14:31:23.971 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5590000000000006e-06 2024-09-13T14:31:23.971 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.971 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.971 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:31:23.971 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.551051+0000", 2024-09-13T14:31:23.971 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7620000000000001e-06 2024-09-13T14:31:23.972 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.972 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.972 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:23.972 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.555809+0000", 2024-09-13T14:31:23.972 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0047581209999999997 2024-09-13T14:31:23.972 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.972 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.972 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:31:23.972 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.556100+0000", 2024-09-13T14:31:23.972 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00029170500000000001 2024-09-13T14:31:23.972 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.972 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.972 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:31:23.972 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.556321+0000", 2024-09-13T14:31:23.973 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00022086100000000001 2024-09-13T14:31:23.973 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.973 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.973 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:31:23.973 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.558857+0000", 2024-09-13T14:31:23.973 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002536235 2024-09-13T14:31:23.973 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.973 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.973 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:23.973 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.559052+0000", 2024-09-13T14:31:23.973 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00019489400000000001 2024-09-13T14:31:23.973 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.973 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.974 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:23.974 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.615815+0000", 2024-09-13T14:31:23.974 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.056762827000000002 2024-09-13T14:31:23.974 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.974 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.974 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:31:23.974 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.615937+0000", 2024-09-13T14:31:23.974 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000121463 2024-09-13T14:31:23.974 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.974 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.974 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:31:23.974 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.615970+0000", 2024-09-13T14:31:23.974 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.3068000000000003e-05 2024-09-13T14:31:23.974 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:23.974 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:31:23.975 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:23.975 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.975 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.975 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:91318 5.2 5:466ae2df:::benchmark_data_smithi102_155989_object91317:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:31:23.975 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:26.827818+0000", 2024-09-13T14:31:23.975 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 237.12861089, 2024-09-13T14:31:23.975 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.070213572000000002, 2024-09-13T14:31:23.975 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:31:23.975 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:31:23.975 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:31:23.975 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:31:23.975 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:31:23.975 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 91318 2024-09-13T14:31:23.975 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.975 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:31:23.976 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.976 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:31:23.976 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:26.827818+0000", 2024-09-13T14:31:23.976 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:23.976 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.976 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.976 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:31:23.976 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:26.827818+0000", 2024-09-13T14:31:23.976 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:23.976 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.976 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.976 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:31:23.976 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:26.827815+0000", 2024-09-13T14:31:23.976 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-09-13T14:31:23.976 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.977 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.977 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:31:23.977 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:26.827851+0000", 2024-09-13T14:31:23.977 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5633999999999998e-05 2024-09-13T14:31:23.977 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.977 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.977 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:31:23.977 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:26.827871+0000", 2024-09-13T14:31:23.977 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9695e-05 2024-09-13T14:31:23.977 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.977 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.977 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:31:23.977 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:26.827879+0000", 2024-09-13T14:31:23.977 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2109999999999998e-06 2024-09-13T14:31:23.978 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.978 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.978 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:23.978 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:26.827935+0000", 2024-09-13T14:31:23.978 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.6637e-05 2024-09-13T14:31:23.978 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.978 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.978 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:31:23.978 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:26.828059+0000", 2024-09-13T14:31:23.978 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000123597 2024-09-13T14:31:23.978 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.978 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.978 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:31:23.978 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:26.828180+0000", 2024-09-13T14:31:23.979 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012057000000000001 2024-09-13T14:31:23.979 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.979 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.979 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:31:23.979 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:26.831282+0000", 2024-09-13T14:31:23.979 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0031025919999999999 2024-09-13T14:31:23.979 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.979 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.979 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:23.979 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:26.835513+0000", 2024-09-13T14:31:23.979 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0042309899999999996 2024-09-13T14:31:23.979 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.979 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.980 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:23.980 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:26.897950+0000", 2024-09-13T14:31:23.980 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.062437126000000003 2024-09-13T14:31:23.980 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.980 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.980 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:31:23.980 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:26.897993+0000", 2024-09-13T14:31:23.980 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2873000000000001e-05 2024-09-13T14:31:23.980 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.980 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.980 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:31:23.980 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:26.898031+0000", 2024-09-13T14:31:23.980 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7968e-05 2024-09-13T14:31:23.981 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:23.981 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:31:23.981 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:23.981 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.981 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.981 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:91674 5.2 5:445bdc5a:::benchmark_data_smithi102_155989_object91673:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:31:23.981 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:27.336850+0000", 2024-09-13T14:31:23.981 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 236.61957841099999, 2024-09-13T14:31:23.981 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.098296165000000005, 2024-09-13T14:31:23.981 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:31:23.981 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:31:23.981 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:31:23.981 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:31:23.981 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:31:23.982 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 91674 2024-09-13T14:31:23.982 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.982 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:31:23.982 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.982 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:31:23.982 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.336850+0000", 2024-09-13T14:31:23.982 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:23.982 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.982 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.982 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:31:23.982 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.336850+0000", 2024-09-13T14:31:23.982 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:23.982 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.982 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.982 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:31:23.983 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.336848+0000", 2024-09-13T14:31:23.983 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-09-13T14:31:23.983 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.983 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.983 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:31:23.983 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.336885+0000", 2024-09-13T14:31:23.983 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7684e-05 2024-09-13T14:31:23.983 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.983 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.983 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:31:23.983 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.336903+0000", 2024-09-13T14:31:23.983 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7453999999999999e-05 2024-09-13T14:31:23.983 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.983 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.984 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:31:23.984 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.336912+0000", 2024-09-13T14:31:23.984 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1619999999999997e-06 2024-09-13T14:31:23.984 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.984 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.984 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:23.984 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.336977+0000", 2024-09-13T14:31:23.984 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5368000000000002e-05 2024-09-13T14:31:23.984 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.984 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.984 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:31:23.984 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.337115+0000", 2024-09-13T14:31:23.984 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013742499999999999 2024-09-13T14:31:23.985 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.985 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.985 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:31:23.985 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.337279+0000", 2024-09-13T14:31:23.985 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016446000000000001 2024-09-13T14:31:23.985 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.985 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.985 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:31:23.985 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.339928+0000", 2024-09-13T14:31:23.985 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0026487149999999998 2024-09-13T14:31:23.985 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.985 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.985 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:23.985 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.341023+0000", 2024-09-13T14:31:23.986 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0010955419999999999 2024-09-13T14:31:23.986 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.986 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.986 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:23.986 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.434976+0000", 2024-09-13T14:31:23.986 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.093952941999999998 2024-09-13T14:31:23.986 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.986 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.986 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:31:23.986 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.435106+0000", 2024-09-13T14:31:23.986 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012945900000000001 2024-09-13T14:31:23.986 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.986 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.986 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:31:23.987 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.435146+0000", 2024-09-13T14:31:23.987 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0410000000000001e-05 2024-09-13T14:31:23.987 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:23.987 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:31:23.987 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:23.987 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.987 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.987 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:112891 5.2 5:447b5a5e:::benchmark_data_smithi102_155989_object112890:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:31:23.987 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:54.112549+0000", 2024-09-13T14:31:23.987 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 209.84387985199999, 2024-09-13T14:31:23.987 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.080517594999999997, 2024-09-13T14:31:23.987 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:31:23.987 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:31:23.988 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:31:23.988 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:31:23.988 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:31:23.988 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 112891 2024-09-13T14:31:23.988 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.988 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:31:23.988 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.988 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:31:23.988 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.112549+0000", 2024-09-13T14:31:23.988 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:23.988 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.988 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.989 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:31:23.989 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.112549+0000", 2024-09-13T14:31:23.989 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:23.989 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.989 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.989 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:31:23.989 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.112547+0000", 2024-09-13T14:31:23.990 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:31:23.990 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.990 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.990 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:31:23.990 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.112582+0000", 2024-09-13T14:31:23.990 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.523e-05 2024-09-13T14:31:23.990 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.990 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.990 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:31:23.990 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.112594+0000", 2024-09-13T14:31:23.990 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1946999999999999e-05 2024-09-13T14:31:23.990 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.991 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.991 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:31:23.991 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.112602+0000", 2024-09-13T14:31:23.991 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8550000000000002e-06 2024-09-13T14:31:23.991 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.991 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.991 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:23.991 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.112966+0000", 2024-09-13T14:31:23.991 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00036412400000000001 2024-09-13T14:31:23.991 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.991 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.991 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:31:23.991 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.113125+0000", 2024-09-13T14:31:23.991 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015927399999999999 2024-09-13T14:31:23.992 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.992 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.992 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:31:23.992 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.113369+0000", 2024-09-13T14:31:23.992 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000244066 2024-09-13T14:31:23.992 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.992 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.992 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:31:23.992 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.122780+0000", 2024-09-13T14:31:23.992 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0094102500000000002 2024-09-13T14:31:23.992 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.992 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.992 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:23.992 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.127254+0000", 2024-09-13T14:31:23.993 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0044746509999999996 2024-09-13T14:31:23.993 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.993 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.993 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:23.993 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.192967+0000", 2024-09-13T14:31:23.993 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.065712830999999999 2024-09-13T14:31:23.993 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.993 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.993 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:31:23.993 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.193024+0000", 2024-09-13T14:31:23.993 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.6928000000000001e-05 2024-09-13T14:31:23.993 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.993 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.994 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:31:23.994 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.193066+0000", 2024-09-13T14:31:23.994 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2138999999999998e-05 2024-09-13T14:31:23.994 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:23.994 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:31:23.994 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:23.994 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.994 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.994 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:112898 5.2 5:419311ca:::benchmark_data_smithi102_155989_object112897:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:31:23.994 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:54.115476+0000", 2024-09-13T14:31:23.994 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 209.84095195699999, 2024-09-13T14:31:23.994 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.086851202000000002, 2024-09-13T14:31:23.994 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:31:23.994 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:31:23.994 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:31:23.995 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:31:23.995 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:31:23.995 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 112898 2024-09-13T14:31:23.995 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.995 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:31:23.995 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.995 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:31:23.995 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.115476+0000", 2024-09-13T14:31:23.995 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:23.995 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.995 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.995 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:31:23.995 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.115476+0000", 2024-09-13T14:31:23.995 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:23.995 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.996 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.996 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:31:23.996 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.115474+0000", 2024-09-13T14:31:23.996 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:31:23.996 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.996 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.996 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:31:23.996 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.115530+0000", 2024-09-13T14:31:23.996 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5500000000000001e-05 2024-09-13T14:31:23.996 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.996 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.996 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:31:23.996 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.115549+0000", 2024-09-13T14:31:23.997 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8627999999999999e-05 2024-09-13T14:31:23.997 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.997 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.997 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:31:23.997 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.115558+0000", 2024-09-13T14:31:23.997 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.3360000000000001e-06 2024-09-13T14:31:23.997 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.997 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.997 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:23.997 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.118101+0000", 2024-09-13T14:31:23.997 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0025430069999999999 2024-09-13T14:31:23.997 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.997 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.997 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:31:23.998 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.118389+0000", 2024-09-13T14:31:23.998 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00028812 2024-09-13T14:31:23.998 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.998 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.998 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:31:23.998 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.118789+0000", 2024-09-13T14:31:23.998 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00040019699999999998 2024-09-13T14:31:23.998 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.998 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.998 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:31:23.998 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.122889+0000", 2024-09-13T14:31:23.998 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.004100228 2024-09-13T14:31:23.998 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.998 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.998 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:23.999 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.127383+0000", 2024-09-13T14:31:23.999 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0044937550000000003 2024-09-13T14:31:23.999 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.999 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.999 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:23.999 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.202199+0000", 2024-09-13T14:31:23.999 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.074815581000000006 2024-09-13T14:31:23.999 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:23.999 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:23.999 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:31:23.999 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.202288+0000", 2024-09-13T14:31:23.999 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.9388000000000005e-05 2024-09-13T14:31:23.999 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.000 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.000 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:31:24.000 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.202328+0000", 2024-09-13T14:31:24.000 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9468999999999998e-05 2024-09-13T14:31:24.000 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:24.000 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:31:24.000 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:24.000 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.000 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.000 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:113023 5.2 5:41246a78:::benchmark_data_smithi102_155989_object113022:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:31:24.000 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:54.349397+0000", 2024-09-13T14:31:24.000 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 209.60703176999999, 2024-09-13T14:31:24.000 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.079303431999999993, 2024-09-13T14:31:24.000 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:31:24.001 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:31:24.001 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:31:24.001 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:31:24.001 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:31:24.001 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113023 2024-09-13T14:31:24.001 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.001 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:31:24.001 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.001 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:31:24.001 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.349397+0000", 2024-09-13T14:31:24.001 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:24.001 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.001 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.001 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:31:24.002 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.349397+0000", 2024-09-13T14:31:24.002 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:24.002 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.002 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.002 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:31:24.002 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.349395+0000", 2024-09-13T14:31:24.002 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:31:24.002 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.002 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.002 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:31:24.002 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.349432+0000", 2024-09-13T14:31:24.002 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.718e-05 2024-09-13T14:31:24.002 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.002 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.002 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:31:24.003 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.349443+0000", 2024-09-13T14:31:24.003 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0729e-05 2024-09-13T14:31:24.003 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.003 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.003 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:31:24.003 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.349449+0000", 2024-09-13T14:31:24.003 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.263e-06 2024-09-13T14:31:24.003 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.003 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.003 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:24.003 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.353718+0000", 2024-09-13T14:31:24.003 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0042686989999999999 2024-09-13T14:31:24.003 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.003 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.004 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:31:24.004 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.353864+0000", 2024-09-13T14:31:24.004 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000146351 2024-09-13T14:31:24.004 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.004 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.004 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:31:24.004 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.354049+0000", 2024-09-13T14:31:24.004 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000184957 2024-09-13T14:31:24.004 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.004 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.004 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:31:24.004 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.358117+0000", 2024-09-13T14:31:24.004 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0040676239999999997 2024-09-13T14:31:24.004 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.005 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.005 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:24.005 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.358813+0000", 2024-09-13T14:31:24.005 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00069628699999999997 2024-09-13T14:31:24.005 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.005 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.005 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:24.005 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.428625+0000", 2024-09-13T14:31:24.005 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.069811467000000002 2024-09-13T14:31:24.005 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.005 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.005 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:31:24.005 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.428667+0000", 2024-09-13T14:31:24.005 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2179999999999999e-05 2024-09-13T14:31:24.006 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.006 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.006 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:31:24.006 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.428700+0000", 2024-09-13T14:31:24.006 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.3170999999999998e-05 2024-09-13T14:31:24.006 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:24.006 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:31:24.006 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:24.006 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.006 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.006 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:113037 5.2 5:461d0743:::benchmark_data_smithi102_155989_object113036:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:31:24.006 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:54.361936+0000", 2024-09-13T14:31:24.006 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 209.59449209600001, 2024-09-13T14:31:24.006 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.076134389999999996, 2024-09-13T14:31:24.007 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:31:24.007 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:31:24.007 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:31:24.007 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:31:24.007 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:31:24.007 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113037 2024-09-13T14:31:24.007 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.007 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:31:24.007 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.007 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:31:24.007 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.361936+0000", 2024-09-13T14:31:24.007 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:24.007 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.007 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.007 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:31:24.008 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.361936+0000", 2024-09-13T14:31:24.008 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:24.008 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.008 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.008 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:31:24.008 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.361934+0000", 2024-09-13T14:31:24.008 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-09-13T14:31:24.008 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.008 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.008 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:31:24.008 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.361971+0000", 2024-09-13T14:31:24.008 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6687000000000003e-05 2024-09-13T14:31:24.008 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.008 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.009 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:31:24.009 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.361985+0000", 2024-09-13T14:31:24.009 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4361e-05 2024-09-13T14:31:24.009 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.009 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.009 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:31:24.009 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.361993+0000", 2024-09-13T14:31:24.009 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4699999999999996e-06 2024-09-13T14:31:24.009 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.009 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.009 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:24.009 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.362044+0000", 2024-09-13T14:31:24.009 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1131000000000002e-05 2024-09-13T14:31:24.009 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.010 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.010 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:31:24.010 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.362166+0000", 2024-09-13T14:31:24.010 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012267499999999999 2024-09-13T14:31:24.010 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.010 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.010 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:31:24.010 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.362298+0000", 2024-09-13T14:31:24.010 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013130100000000001 2024-09-13T14:31:24.010 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.010 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.010 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:31:24.010 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.366490+0000", 2024-09-13T14:31:24.010 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0041923739999999996 2024-09-13T14:31:24.010 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.011 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.011 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:24.011 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.368290+0000", 2024-09-13T14:31:24.011 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001800409 2024-09-13T14:31:24.011 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.011 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.011 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:24.011 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.437859+0000", 2024-09-13T14:31:24.011 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.069568481000000001 2024-09-13T14:31:24.011 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.011 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.011 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:31:24.011 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.437946+0000", 2024-09-13T14:31:24.011 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.6784000000000001e-05 2024-09-13T14:31:24.012 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.012 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.012 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:31:24.012 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.438071+0000", 2024-09-13T14:31:24.012 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012500200000000001 2024-09-13T14:31:24.012 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:24.012 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:31:24.012 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:24.012 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.012 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.012 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:113039 5.2 5:40939e83:::benchmark_data_smithi102_155989_object113038:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:31:24.012 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:54.370277+0000", 2024-09-13T14:31:24.012 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 209.58615113299999, 2024-09-13T14:31:24.012 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.068259345999999999, 2024-09-13T14:31:24.012 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:31:24.013 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:31:24.013 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:31:24.013 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:31:24.013 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:31:24.013 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113039 2024-09-13T14:31:24.013 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.013 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:31:24.013 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.013 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:31:24.013 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.370277+0000", 2024-09-13T14:31:24.013 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:24.013 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.013 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.013 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:31:24.013 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.370277+0000", 2024-09-13T14:31:24.014 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:24.014 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.014 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.014 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:31:24.014 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.370276+0000", 2024-09-13T14:31:24.014 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:31:24.014 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.014 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.014 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:31:24.014 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.370319+0000", 2024-09-13T14:31:24.014 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3170999999999997e-05 2024-09-13T14:31:24.014 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.014 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.014 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:31:24.014 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.370332+0000", 2024-09-13T14:31:24.015 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2994e-05 2024-09-13T14:31:24.015 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.015 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.015 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:31:24.015 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.370338+0000", 2024-09-13T14:31:24.015 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5199999999999997e-06 2024-09-13T14:31:24.015 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.015 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.015 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:24.015 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.370375+0000", 2024-09-13T14:31:24.015 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7472000000000003e-05 2024-09-13T14:31:24.015 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.015 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.015 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:31:24.015 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.370481+0000", 2024-09-13T14:31:24.016 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010548 2024-09-13T14:31:24.016 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.016 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.016 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:31:24.016 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.370596+0000", 2024-09-13T14:31:24.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:24.017 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:24.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:31:24.017 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011521799999999999 2024-09-13T14:31:24.018 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.018 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.018 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:31:24.018 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.374003+0000", 2024-09-13T14:31:24.018 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003407584 2024-09-13T14:31:24.018 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.018 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.018 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:24.018 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.374176+0000", 2024-09-13T14:31:24.018 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00017315899999999999 2024-09-13T14:31:24.018 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.018 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.018 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:24.019 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.438213+0000", 2024-09-13T14:31:24.019 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.064036478999999993 2024-09-13T14:31:24.019 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.019 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.019 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:31:24.019 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.438386+0000", 2024-09-13T14:31:24.019 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000172988 2024-09-13T14:31:24.019 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.019 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.019 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:31:24.019 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:54.438537+0000", 2024-09-13T14:31:24.019 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000150688 2024-09-13T14:31:24.019 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:24.020 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:31:24.020 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:24.020 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.020 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.020 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:113753 5.2 5:4a3b37a0:::benchmark_data_smithi102_155989_object113752:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:31:24.020 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:55.352637+0000", 2024-09-13T14:31:24.020 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 208.60379104500001, 2024-09-13T14:31:24.020 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.082186414999999999, 2024-09-13T14:31:24.020 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:31:24.020 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:31:24.020 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:31:24.020 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:31:24.021 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:31:24.021 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113753 2024-09-13T14:31:24.021 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.021 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:31:24.021 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.021 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:31:24.021 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.352637+0000", 2024-09-13T14:31:24.021 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:24.021 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.021 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.021 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:31:24.021 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.352637+0000", 2024-09-13T14:31:24.021 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:24.022 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.022 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.022 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:31:24.022 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.352636+0000", 2024-09-13T14:31:24.022 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:31:24.022 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.022 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.022 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:31:24.022 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.352729+0000", 2024-09-13T14:31:24.022 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.2750999999999993e-05 2024-09-13T14:31:24.022 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.022 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.022 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:31:24.022 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.352742+0000", 2024-09-13T14:31:24.023 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3874e-05 2024-09-13T14:31:24.023 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.023 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.023 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:31:24.023 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.352750+0000", 2024-09-13T14:31:24.023 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2509999999999997e-06 2024-09-13T14:31:24.023 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.023 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.023 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:24.023 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.352831+0000", 2024-09-13T14:31:24.023 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1213999999999999e-05 2024-09-13T14:31:24.023 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.023 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.023 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:31:24.023 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.352983+0000", 2024-09-13T14:31:24.024 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015199300000000001 2024-09-13T14:31:24.024 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.024 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.024 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:31:24.024 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.353127+0000", 2024-09-13T14:31:24.024 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014379099999999999 2024-09-13T14:31:24.024 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.024 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.024 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:31:24.024 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.358485+0000", 2024-09-13T14:31:24.024 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0053579409999999997 2024-09-13T14:31:24.024 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.024 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.024 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:24.024 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.362585+0000", 2024-09-13T14:31:24.025 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0041000840000000004 2024-09-13T14:31:24.025 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.025 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.025 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:24.025 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.434711+0000", 2024-09-13T14:31:24.025 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.072126032000000007 2024-09-13T14:31:24.025 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.025 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.025 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:31:24.025 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.434782+0000", 2024-09-13T14:31:24.025 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.0838000000000002e-05 2024-09-13T14:31:24.025 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.025 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.025 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:31:24.026 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.434824+0000", 2024-09-13T14:31:24.026 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2150000000000001e-05 2024-09-13T14:31:24.026 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:24.026 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:31:24.026 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:24.026 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.026 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.026 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:113758 5.2 5:45b8ba06:::benchmark_data_smithi102_155989_object113757:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:31:24.026 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:55.358413+0000", 2024-09-13T14:31:24.026 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 208.59801538400001, 2024-09-13T14:31:24.026 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.076541652000000002, 2024-09-13T14:31:24.026 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:31:24.026 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:31:24.026 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:31:24.026 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:31:24.027 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:31:24.027 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113758 2024-09-13T14:31:24.027 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.027 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:31:24.027 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.027 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:31:24.027 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.358413+0000", 2024-09-13T14:31:24.027 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:24.027 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.027 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.027 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:31:24.027 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.358413+0000", 2024-09-13T14:31:24.027 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:24.027 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.027 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.028 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:31:24.028 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.358411+0000", 2024-09-13T14:31:24.028 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:31:24.028 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.028 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.028 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:31:24.028 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.358448+0000", 2024-09-13T14:31:24.028 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6878000000000002e-05 2024-09-13T14:31:24.028 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.028 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.028 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:31:24.028 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.358526+0000", 2024-09-13T14:31:24.028 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7841000000000002e-05 2024-09-13T14:31:24.028 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.029 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.029 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:31:24.029 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.358536+0000", 2024-09-13T14:31:24.029 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4890000000000001e-06 2024-09-13T14:31:24.029 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.029 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.029 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:24.029 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.358600+0000", 2024-09-13T14:31:24.029 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4548000000000006e-05 2024-09-13T14:31:24.029 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.029 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.029 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:31:24.029 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.358947+0000", 2024-09-13T14:31:24.029 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00034709999999999998 2024-09-13T14:31:24.030 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.030 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.030 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:31:24.030 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.359208+0000", 2024-09-13T14:31:24.030 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00026032699999999998 2024-09-13T14:31:24.030 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.030 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.030 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:31:24.030 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.362687+0000", 2024-09-13T14:31:24.030 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0034797500000000002 2024-09-13T14:31:24.030 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.030 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.030 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:24.031 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.362738+0000", 2024-09-13T14:31:24.031 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0664000000000001e-05 2024-09-13T14:31:24.031 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.031 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.031 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:24.031 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.434855+0000", 2024-09-13T14:31:24.031 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.072117280000000006 2024-09-13T14:31:24.031 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.031 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.031 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:31:24.031 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.434927+0000", 2024-09-13T14:31:24.031 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1328999999999994e-05 2024-09-13T14:31:24.031 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.031 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.032 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:31:24.032 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:55.434955+0000", 2024-09-13T14:31:24.032 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8130999999999999e-05 2024-09-13T14:31:24.032 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:24.032 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:31:24.032 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:24.032 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.032 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.032 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:116701 5.2 5:4ad2b1b2:::benchmark_data_smithi102_155989_object116700:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:31:24.032 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:59.402062+0000", 2024-09-13T14:31:24.032 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 204.554366367, 2024-09-13T14:31:24.032 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.096687686999999994, 2024-09-13T14:31:24.033 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:31:24.033 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:31:24.033 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:31:24.033 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:31:24.033 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:31:24.033 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 116701 2024-09-13T14:31:24.033 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.033 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:31:24.033 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.033 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:31:24.033 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.402062+0000", 2024-09-13T14:31:24.033 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:24.033 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.034 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.034 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:31:24.034 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.402062+0000", 2024-09-13T14:31:24.034 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:24.034 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.034 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.034 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:31:24.034 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.402060+0000", 2024-09-13T14:31:24.034 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:31:24.034 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.034 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.034 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:31:24.034 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.402109+0000", 2024-09-13T14:31:24.034 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8319000000000003e-05 2024-09-13T14:31:24.035 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.035 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.035 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:31:24.035 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.402120+0000", 2024-09-13T14:31:24.035 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1993e-05 2024-09-13T14:31:24.035 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.035 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.035 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:31:24.035 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.402129+0000", 2024-09-13T14:31:24.035 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0639999999999994e-06 2024-09-13T14:31:24.035 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.035 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.035 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:24.035 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.402196+0000", 2024-09-13T14:31:24.036 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7712000000000005e-05 2024-09-13T14:31:24.036 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.036 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.036 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:31:24.036 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.402312+0000", 2024-09-13T14:31:24.036 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000115351 2024-09-13T14:31:24.036 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.036 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.036 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:31:24.036 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.402444+0000", 2024-09-13T14:31:24.036 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000132781 2024-09-13T14:31:24.036 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.036 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.037 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:31:24.037 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.406257+0000", 2024-09-13T14:31:24.037 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0038126079999999999 2024-09-13T14:31:24.037 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.037 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.037 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:24.037 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.414144+0000", 2024-09-13T14:31:24.037 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0078872130000000006 2024-09-13T14:31:24.037 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.037 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.037 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:24.037 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.498654+0000", 2024-09-13T14:31:24.037 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.084509976000000001 2024-09-13T14:31:24.037 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.038 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.038 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:31:24.038 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.498712+0000", 2024-09-13T14:31:24.038 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8037e-05 2024-09-13T14:31:24.038 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.038 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.038 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:31:24.038 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.498750+0000", 2024-09-13T14:31:24.038 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7497999999999999e-05 2024-09-13T14:31:24.038 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:24.038 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:31:24.038 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:24.038 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.038 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.039 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:116703 5.2 5:4688eb25:::benchmark_data_smithi102_155989_object116702:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:31:24.039 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:59.406689+0000", 2024-09-13T14:31:24.039 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 204.54973969, 2024-09-13T14:31:24.039 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.092431676000000004, 2024-09-13T14:31:24.039 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:31:24.039 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:31:24.039 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:31:24.039 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:31:24.039 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:31:24.039 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 116703 2024-09-13T14:31:24.039 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.039 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:31:24.039 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.039 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:31:24.039 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.406689+0000", 2024-09-13T14:31:24.040 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:24.040 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.040 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.040 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:31:24.040 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.406689+0000", 2024-09-13T14:31:24.040 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:24.040 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.040 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.040 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:31:24.040 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.406687+0000", 2024-09-13T14:31:24.040 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:31:24.040 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.040 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.040 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:31:24.041 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.406730+0000", 2024-09-13T14:31:24.041 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2775000000000003e-05 2024-09-13T14:31:24.041 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.041 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.041 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:31:24.041 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.406746+0000", 2024-09-13T14:31:24.041 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5985999999999999e-05 2024-09-13T14:31:24.041 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.041 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.041 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:31:24.041 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.406754+0000", 2024-09-13T14:31:24.041 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9680000000000006e-06 2024-09-13T14:31:24.041 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.041 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.042 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:24.042 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.406794+0000", 2024-09-13T14:31:24.042 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.036e-05 2024-09-13T14:31:24.042 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.042 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.042 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:31:24.042 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.406924+0000", 2024-09-13T14:31:24.042 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013029900000000001 2024-09-13T14:31:24.042 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.042 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.042 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:31:24.042 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.407062+0000", 2024-09-13T14:31:24.042 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013760399999999999 2024-09-13T14:31:24.043 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.043 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.043 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:31:24.043 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.410866+0000", 2024-09-13T14:31:24.043 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0038043339999999999 2024-09-13T14:31:24.043 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.043 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.043 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:24.043 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.414960+0000", 2024-09-13T14:31:24.043 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0040940200000000003 2024-09-13T14:31:24.043 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.043 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.043 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:24.044 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.499038+0000", 2024-09-13T14:31:24.044 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.084077951999999997 2024-09-13T14:31:24.044 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.044 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.044 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:31:24.044 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.499085+0000", 2024-09-13T14:31:24.044 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6841000000000001e-05 2024-09-13T14:31:24.044 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.044 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.044 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:31:24.044 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:59.499120+0000", 2024-09-13T14:31:24.044 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5348000000000001e-05 2024-09-13T14:31:24.044 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:24.044 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:31:24.045 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:24.045 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.045 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.045 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:117467 5.2 5:4a8dc47a:::benchmark_data_smithi102_155989_object117466:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:31:24.045 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:28:00.495953+0000", 2024-09-13T14:31:24.045 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 203.460475699, 2024-09-13T14:31:24.045 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.092397426000000005, 2024-09-13T14:31:24.045 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:31:24.045 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:31:24.045 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:31:24.045 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:31:24.045 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:31:24.045 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 117467 2024-09-13T14:31:24.045 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.046 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:31:24.046 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.046 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:31:24.046 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.495953+0000", 2024-09-13T14:31:24.046 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:24.046 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.046 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.046 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:31:24.046 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.495953+0000", 2024-09-13T14:31:24.046 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:24.046 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.046 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.046 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:31:24.046 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.495950+0000", 2024-09-13T14:31:24.047 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-09-13T14:31:24.047 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.047 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.047 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:31:24.047 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.495993+0000", 2024-09-13T14:31:24.047 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3019000000000003e-05 2024-09-13T14:31:24.047 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.047 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.047 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:31:24.047 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.496013+0000", 2024-09-13T14:31:24.047 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9602999999999998e-05 2024-09-13T14:31:24.047 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.047 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.047 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:31:24.047 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.496023+0000", 2024-09-13T14:31:24.048 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4390000000000006e-06 2024-09-13T14:31:24.048 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.048 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.048 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:24.048 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.498242+0000", 2024-09-13T14:31:24.048 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0022199720000000002 2024-09-13T14:31:24.048 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.048 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.048 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:31:24.048 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.498572+0000", 2024-09-13T14:31:24.048 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00032968500000000003 2024-09-13T14:31:24.048 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.048 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.049 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:31:24.049 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.498936+0000", 2024-09-13T14:31:24.049 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000364185 2024-09-13T14:31:24.049 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.049 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.049 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:31:24.049 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.509682+0000", 2024-09-13T14:31:24.049 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010745879 2024-09-13T14:31:24.049 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.049 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.049 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:24.049 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.510286+0000", 2024-09-13T14:31:24.049 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00060406800000000003 2024-09-13T14:31:24.050 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.050 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.050 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:24.050 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.588289+0000", 2024-09-13T14:31:24.050 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.078002929999999998 2024-09-13T14:31:24.050 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.050 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.050 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:31:24.050 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.588324+0000", 2024-09-13T14:31:24.050 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4928999999999997e-05 2024-09-13T14:31:24.050 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.050 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.050 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:31:24.050 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.588350+0000", 2024-09-13T14:31:24.051 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5978000000000001e-05 2024-09-13T14:31:24.051 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:24.051 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:31:24.051 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:24.051 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.051 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.051 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:117471 5.2 5:419b8c44:::benchmark_data_smithi102_155989_object117470:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:31:24.051 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:28:00.498101+0000", 2024-09-13T14:31:24.051 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 203.458327139, 2024-09-13T14:31:24.051 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.090334384000000004, 2024-09-13T14:31:24.051 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:31:24.051 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:31:24.052 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:31:24.052 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:31:24.052 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:31:24.052 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 117471 2024-09-13T14:31:24.052 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.052 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:31:24.052 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.052 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:31:24.052 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.498101+0000", 2024-09-13T14:31:24.052 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:24.052 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.052 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.052 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:31:24.052 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.498101+0000", 2024-09-13T14:31:24.053 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:24.053 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.053 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.053 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:31:24.053 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.498100+0000", 2024-09-13T14:31:24.053 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:31:24.053 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.053 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.053 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:31:24.053 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.498129+0000", 2024-09-13T14:31:24.053 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9184e-05 2024-09-13T14:31:24.053 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.053 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.053 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:31:24.054 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.498145+0000", 2024-09-13T14:31:24.054 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6005e-05 2024-09-13T14:31:24.054 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.054 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.054 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:31:24.054 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.498152+0000", 2024-09-13T14:31:24.054 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2799999999999998e-06 2024-09-13T14:31:24.054 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.054 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.054 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:24.054 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.502688+0000", 2024-09-13T14:31:24.054 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0045362429999999997 2024-09-13T14:31:24.054 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.054 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.054 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:31:24.055 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.502829+0000", 2024-09-13T14:31:24.055 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000140564 2024-09-13T14:31:24.055 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.055 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.055 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:31:24.055 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.502991+0000", 2024-09-13T14:31:24.055 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000161519 2024-09-13T14:31:24.055 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.055 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.055 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:31:24.055 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.510054+0000", 2024-09-13T14:31:24.055 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0070637959999999998 2024-09-13T14:31:24.055 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.055 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.055 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:24.056 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.510438+0000", 2024-09-13T14:31:24.056 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00038332399999999999 2024-09-13T14:31:24.056 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.056 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.056 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:24.056 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.588381+0000", 2024-09-13T14:31:24.056 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.077943296999999995 2024-09-13T14:31:24.056 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.056 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.056 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:31:24.056 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.588413+0000", 2024-09-13T14:31:24.056 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2011999999999998e-05 2024-09-13T14:31:24.056 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.057 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.057 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:31:24.057 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:00.588436+0000", 2024-09-13T14:31:24.057 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2699000000000001e-05 2024-09-13T14:31:24.057 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:24.057 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:31:24.057 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:24.057 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.057 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.057 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:127406 5.2 5:401974e1:::benchmark_data_smithi102_155989_object9488:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-13T14:31:24.057 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:28:13.570743+0000", 2024-09-13T14:31:24.057 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 190.385685024, 2024-09-13T14:31:24.057 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.069567083000000002, 2024-09-13T14:31:24.057 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:31:24.058 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:31:24.058 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:31:24.058 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:31:24.058 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:31:24.058 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 127406 2024-09-13T14:31:24.058 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.058 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:31:24.058 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.058 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:31:24.059 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.570743+0000", 2024-09-13T14:31:24.059 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:24.059 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.059 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.059 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:31:24.059 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.570743+0000", 2024-09-13T14:31:24.059 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:24.059 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.059 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.060 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:31:24.060 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.570742+0000", 2024-09-13T14:31:24.060 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:31:24.060 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.060 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.060 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:31:24.060 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.570748+0000", 2024-09-13T14:31:24.060 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0580000000000003e-06 2024-09-13T14:31:24.060 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.060 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.060 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:31:24.060 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.570760+0000", 2024-09-13T14:31:24.061 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1389000000000001e-05 2024-09-13T14:31:24.061 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.061 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.061 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:31:24.061 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.570766+0000", 2024-09-13T14:31:24.061 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3230000000000003e-06 2024-09-13T14:31:24.061 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.061 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.061 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:24.061 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.570829+0000", 2024-09-13T14:31:24.061 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3225000000000005e-05 2024-09-13T14:31:24.061 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.061 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.061 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:31:24.062 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.570936+0000", 2024-09-13T14:31:24.062 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010673 2024-09-13T14:31:24.062 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.062 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.062 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:31:24.062 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.571033+0000", 2024-09-13T14:31:24.062 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.7323000000000003e-05 2024-09-13T14:31:24.062 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.062 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.062 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:31:24.062 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.575126+0000", 2024-09-13T14:31:24.062 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0040930280000000003 2024-09-13T14:31:24.062 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.063 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.063 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:24.063 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.582607+0000", 2024-09-13T14:31:24.063 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0074805100000000001 2024-09-13T14:31:24.063 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.063 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.063 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:24.063 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.640211+0000", 2024-09-13T14:31:24.063 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.057604732999999998 2024-09-13T14:31:24.063 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.063 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.063 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:31:24.063 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.640260+0000", 2024-09-13T14:31:24.063 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8822000000000001e-05 2024-09-13T14:31:24.064 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.064 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.064 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:31:24.064 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.640310+0000", 2024-09-13T14:31:24.064 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0235000000000003e-05 2024-09-13T14:31:24.064 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:24.064 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:31:24.064 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:24.064 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.064 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.064 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:127604 5.2 5:4b8edd25:::benchmark_data_smithi102_155989_object9686:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-13T14:31:24.064 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:28:13.850196+0000", 2024-09-13T14:31:24.064 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 190.10623211800001, 2024-09-13T14:31:24.065 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.085607760000000005, 2024-09-13T14:31:24.065 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:31:24.065 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:31:24.065 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:31:24.065 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:31:24.065 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:31:24.065 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 127604 2024-09-13T14:31:24.065 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.065 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:31:24.065 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.065 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:31:24.065 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.850196+0000", 2024-09-13T14:31:24.065 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:24.066 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.066 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.066 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:31:24.066 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.850196+0000", 2024-09-13T14:31:24.066 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:24.066 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.066 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.066 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:31:24.066 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.850194+0000", 2024-09-13T14:31:24.066 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-09-13T14:31:24.066 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.066 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.066 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:31:24.066 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.850202+0000", 2024-09-13T14:31:24.067 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7600000000000002e-06 2024-09-13T14:31:24.067 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.067 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.067 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:31:24.067 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.850219+0000", 2024-09-13T14:31:24.067 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6752000000000002e-05 2024-09-13T14:31:24.067 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.067 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.067 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:31:24.067 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.850227+0000", 2024-09-13T14:31:24.067 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0369999999999995e-06 2024-09-13T14:31:24.067 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.067 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.067 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:24.068 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.850289+0000", 2024-09-13T14:31:24.068 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2848000000000005e-05 2024-09-13T14:31:24.068 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.068 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.068 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:31:24.068 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.850589+0000", 2024-09-13T14:31:24.068 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00029958200000000001 2024-09-13T14:31:24.068 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.068 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.068 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:31:24.068 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.850929+0000", 2024-09-13T14:31:24.068 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00034012799999999998 2024-09-13T14:31:24.068 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.068 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.069 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:31:24.069 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.860869+0000", 2024-09-13T14:31:24.069 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0099398339999999998 2024-09-13T14:31:24.069 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.069 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.069 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:24.069 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.861006+0000", 2024-09-13T14:31:24.069 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000136592 2024-09-13T14:31:24.069 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.069 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.069 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:24.069 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.935335+0000", 2024-09-13T14:31:24.069 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.074329002000000005 2024-09-13T14:31:24.070 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.070 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.070 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:31:24.070 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.935672+0000", 2024-09-13T14:31:24.070 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000337141 2024-09-13T14:31:24.070 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.070 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.070 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:31:24.070 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.935804+0000", 2024-09-13T14:31:24.070 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013234100000000001 2024-09-13T14:31:24.070 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:24.070 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:31:24.070 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:24.071 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.071 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.071 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:127607 5.2 5:4a72a91f:::benchmark_data_smithi102_155989_object9689:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-13T14:31:24.071 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:28:13.856547+0000", 2024-09-13T14:31:24.071 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 190.099881184, 2024-09-13T14:31:24.071 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.079705397999999997, 2024-09-13T14:31:24.071 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:31:24.071 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:31:24.071 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:31:24.071 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:31:24.071 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:31:24.071 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 127607 2024-09-13T14:31:24.071 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.071 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:31:24.072 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.072 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:31:24.072 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.856547+0000", 2024-09-13T14:31:24.072 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:24.072 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.072 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.072 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:31:24.072 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.856547+0000", 2024-09-13T14:31:24.072 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:24.072 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.072 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.072 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:31:24.072 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.856545+0000", 2024-09-13T14:31:24.072 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-09-13T14:31:24.073 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.073 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.073 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:31:24.073 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.856557+0000", 2024-09-13T14:31:24.073 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2177999999999999e-05 2024-09-13T14:31:24.073 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.073 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.073 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:31:24.073 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.856575+0000", 2024-09-13T14:31:24.073 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8236999999999998e-05 2024-09-13T14:31:24.073 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.073 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.073 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:31:24.073 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.856583+0000", 2024-09-13T14:31:24.074 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5109999999999997e-06 2024-09-13T14:31:24.074 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.074 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.074 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:24.074 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.856675+0000", 2024-09-13T14:31:24.074 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.2089999999999994e-05 2024-09-13T14:31:24.074 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.074 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.074 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:31:24.074 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.856974+0000", 2024-09-13T14:31:24.074 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00029847400000000002 2024-09-13T14:31:24.074 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.076 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.076 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:31:24.076 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.857308+0000", 2024-09-13T14:31:24.076 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00033488799999999998 2024-09-13T14:31:24.076 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.076 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.076 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:31:24.076 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.861095+0000", 2024-09-13T14:31:24.076 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003786106 2024-09-13T14:31:24.076 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.077 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.077 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:24.077 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.861255+0000", 2024-09-13T14:31:24.077 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016012200000000001 2024-09-13T14:31:24.077 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.077 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.077 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:24.077 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.935963+0000", 2024-09-13T14:31:24.077 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.074708732 2024-09-13T14:31:24.077 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.077 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.077 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:31:24.077 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.936117+0000", 2024-09-13T14:31:24.078 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015398000000000001 2024-09-13T14:31:24.078 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.078 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.078 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:31:24.078 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:13.936253+0000", 2024-09-13T14:31:24.078 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013526799999999999 2024-09-13T14:31:24.078 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:24.078 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:31:24.078 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:24.078 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.078 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.078 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:148497 5.2 5:44c2546f:::benchmark_data_smithi102_155989_object30579:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T14:31:24.078 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:28:43.190575+0000", 2024-09-13T14:31:24.078 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 160.765853502, 2024-09-13T14:31:24.079 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.076364062999999996, 2024-09-13T14:31:24.079 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:31:24.079 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:31:24.079 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:31:24.079 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:31:24.079 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:31:24.079 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 148497 2024-09-13T14:31:24.079 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.079 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:31:24.079 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.079 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:31:24.079 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.190575+0000", 2024-09-13T14:31:24.079 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:24.079 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.080 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.080 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:31:24.080 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.190575+0000", 2024-09-13T14:31:24.080 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:24.080 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.080 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.080 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:31:24.080 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.190574+0000", 2024-09-13T14:31:24.080 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:31:24.080 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.080 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.080 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:31:24.080 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.190581+0000", 2024-09-13T14:31:24.080 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.0419999999999997e-06 2024-09-13T14:31:24.081 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.081 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.081 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:31:24.081 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.190594+0000", 2024-09-13T14:31:24.081 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3421000000000001e-05 2024-09-13T14:31:24.081 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.081 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.081 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:31:24.081 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.190600+0000", 2024-09-13T14:31:24.081 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3180000000000003e-06 2024-09-13T14:31:24.081 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.081 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.081 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:24.082 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.190638+0000", 2024-09-13T14:31:24.082 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7667999999999999e-05 2024-09-13T14:31:24.082 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.082 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.082 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:31:24.082 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.190715+0000", 2024-09-13T14:31:24.082 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7311000000000006e-05 2024-09-13T14:31:24.082 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.082 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.082 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:31:24.082 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.190784+0000", 2024-09-13T14:31:24.082 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9082e-05 2024-09-13T14:31:24.082 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.083 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.083 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:31:24.083 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.195660+0000", 2024-09-13T14:31:24.083 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0048754749999999998 2024-09-13T14:31:24.083 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.083 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.083 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:24.083 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.196096+0000", 2024-09-13T14:31:24.083 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00043614599999999999 2024-09-13T14:31:24.083 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.083 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.083 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:24.083 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.266845+0000", 2024-09-13T14:31:24.083 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.070748855999999999 2024-09-13T14:31:24.084 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.084 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.084 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:31:24.084 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.266896+0000", 2024-09-13T14:31:24.084 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1106000000000002e-05 2024-09-13T14:31:24.084 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.084 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.084 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:31:24.084 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.266939+0000", 2024-09-13T14:31:24.084 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2933999999999999e-05 2024-09-13T14:31:24.084 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:24.084 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:31:24.084 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:24.085 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.085 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.085 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:149656 5.2 5:4847ba63:::benchmark_data_smithi102_155989_object31738:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T14:31:24.085 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:28:44.877514+0000", 2024-09-13T14:31:24.085 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 159.07891489100001, 2024-09-13T14:31:24.085 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.066647814, 2024-09-13T14:31:24.085 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:31:24.085 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:31:24.085 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:31:24.085 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:31:24.085 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:31:24.085 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 149656 2024-09-13T14:31:24.085 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.086 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:31:24.086 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.086 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:31:24.086 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:44.877514+0000", 2024-09-13T14:31:24.086 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:24.086 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.086 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.086 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:31:24.086 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:44.877514+0000", 2024-09-13T14:31:24.086 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:24.086 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.086 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.086 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:31:24.086 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:44.877511+0000", 2024-09-13T14:31:24.087 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-09-13T14:31:24.087 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.087 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.087 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:31:24.087 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:44.877521+0000", 2024-09-13T14:31:24.087 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0199e-05 2024-09-13T14:31:24.087 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.087 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.087 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:31:24.087 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:44.877544+0000", 2024-09-13T14:31:24.087 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2575999999999999e-05 2024-09-13T14:31:24.087 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.087 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.087 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:31:24.088 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:44.877553+0000", 2024-09-13T14:31:24.088 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.5740000000000002e-06 2024-09-13T14:31:24.088 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.088 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.088 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:24.088 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:44.877602+0000", 2024-09-13T14:31:24.088 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8154e-05 2024-09-13T14:31:24.088 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.088 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.088 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:31:24.088 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:44.877711+0000", 2024-09-13T14:31:24.088 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000109511 2024-09-13T14:31:24.088 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.089 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.089 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:31:24.089 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:44.877814+0000", 2024-09-13T14:31:24.089 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000102812 2024-09-13T14:31:24.089 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.089 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.089 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:31:24.089 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:44.881312+0000", 2024-09-13T14:31:24.089 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0034982419999999999 2024-09-13T14:31:24.089 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.089 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.089 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:24.089 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:44.881822+0000", 2024-09-13T14:31:24.089 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00051006800000000002 2024-09-13T14:31:24.090 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.090 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.090 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:24.090 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:44.944065+0000", 2024-09-13T14:31:24.090 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.062242810000000003 2024-09-13T14:31:24.090 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.090 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.090 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:31:24.090 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:44.944122+0000", 2024-09-13T14:31:24.090 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7328999999999999e-05 2024-09-13T14:31:24.090 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:24.090 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:24.090 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:31:24.091 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:44.944161+0000", 2024-09-13T14:31:24.091 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8930000000000002e-05 2024-09-13T14:31:24.091 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:24.091 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:31:24.091 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:24.091 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:24.091 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:31:24.091 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:31:24.118 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:24.146 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:24.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:24.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:24.247 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:24.284 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:24.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:24.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:24.386 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:24.424 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:24.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:24.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:24.525 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:24.563 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:24.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:24.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:24.649 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is set 2024-09-13T14:31:24.665 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:24.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:24.702 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:24.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:24.803 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:24.833 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:24.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:24.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:24.895 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 10 16 27970 27954 174.696 142.25 0.00797212 0.0057187 2024-09-13T14:31:24.935 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:24.973 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:24.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:24.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:25.075 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:25.100 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:25.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:25.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:25.202 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:25.224 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:25.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:25.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:25.326 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:25.354 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:25.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:25.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:25.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:25 smithi102 ceph-mon[110183]: pgmap v497: 89 pgs: 89 active+clean; 1.2 GiB data, 4.3 GiB used, 532 GiB / 536 GiB avail; 117 MiB/s wr, 1.88k op/s 2024-09-13T14:31:25.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:25 smithi102 ceph-mon[115269]: pgmap v497: 89 pgs: 89 active+clean; 1.2 GiB data, 4.3 GiB used, 532 GiB / 536 GiB avail; 117 MiB/s wr, 1.88k op/s 2024-09-13T14:31:25.457 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:25.461 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:25 smithi195 ceph-mon[104262]: pgmap v497: 89 pgs: 89 active+clean; 1.2 GiB data, 4.3 GiB used, 532 GiB / 536 GiB avail; 117 MiB/s wr, 1.88k op/s 2024-09-13T14:31:25.484 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:25.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:25.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:25.586 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:31:25.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:25.627 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:25.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:31:25.728 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:25.757 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:25.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:25.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:25.859 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:25.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:25.881 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:25.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:25.895 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 11 16 29493 29477 167.468 95.1875 0.00418069 0.0059253 2024-09-13T14:31:25.982 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:26.016 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:26.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:26.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:26.118 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:26.163 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:26.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:26.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:26.266 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:31:26.303 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:26.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:26.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:31:26.405 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:26.438 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:26.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:26.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:26.540 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:26.570 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:26.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:26.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:26.672 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:26.697 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:26.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:26.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:26.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:26 smithi195 ceph-mon[104262]: pgmap v498: 89 pgs: 89 active+clean; 1.3 GiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 137 MiB/s wr, 2.19k op/s 2024-09-13T14:31:26.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:31:26.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:31:26.799 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:26.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:26.842 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:26.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:26.895 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 12 16 31201 31185 162.407 106.75 0.0088857 0.0061527 2024-09-13T14:31:26.918 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:31:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:31:26] "GET /metrics HTTP/1.1" 200 36506 "" "Prometheus/2.51.0" 2024-09-13T14:31:26.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:26 smithi102 ceph-mon[110183]: pgmap v498: 89 pgs: 89 active+clean; 1.3 GiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 137 MiB/s wr, 2.19k op/s 2024-09-13T14:31:26.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:31:26.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:31:26.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:26 smithi102 ceph-mon[115269]: pgmap v498: 89 pgs: 89 active+clean; 1.3 GiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 137 MiB/s wr, 2.19k op/s 2024-09-13T14:31:26.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:31:26.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:31:26.943 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:26.964 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:26.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:26.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:27.066 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:27.098 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:27.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:27.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:27.200 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:31:27.231 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:27.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:27.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:31:27.333 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:31:27.372 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:27.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:27.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:31:27.467 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T14:31:27.475 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:27.502 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:27.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:27.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:27.605 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:27.625 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:27.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:27.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:27.727 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:27.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:27.757 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:27.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:27.829 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:31:27.858 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:31:27.895 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 13 16 32907 32891 158.116 106.625 0.00707041 0.00631885 2024-09-13T14:31:27.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:27.909 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:27.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:31:28.010 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:28.049 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:28.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:28.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:28.151 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:28.178 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:28.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:28.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:28.280 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:31:28.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:28.318 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:28.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:31:28.325 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:31:28.326 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:31:28.420 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:28.448 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:28.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:28.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:28.523 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:28 smithi102 ceph-mon[110183]: pgmap v499: 89 pgs: 89 active+clean; 1.8 GiB data, 6.4 GiB used, 530 GiB / 536 GiB avail; 150 MiB/s wr, 2.40k op/s 2024-09-13T14:31:28.524 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:28 smithi102 ceph-mon[115269]: pgmap v499: 89 pgs: 89 active+clean; 1.8 GiB data, 6.4 GiB used, 530 GiB / 536 GiB avail; 150 MiB/s wr, 2.40k op/s 2024-09-13T14:31:28.550 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:28.586 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:28.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:28.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:28.688 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:28.715 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:28.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:28.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:28.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:28 smithi195 ceph-mon[104262]: pgmap v499: 89 pgs: 89 active+clean; 1.8 GiB data, 6.4 GiB used, 530 GiB / 536 GiB avail; 150 MiB/s wr, 2.40k op/s 2024-09-13T14:31:28.816 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:28.848 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:28.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:28.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:28.895 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 14 16 34813 34797 155.33 119.125 0.00804456 0.00643295 2024-09-13T14:31:28.950 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:31:28.990 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:28.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:28.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:31:29.092 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:29.129 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:29.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:29.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:29.231 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:31:29.265 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:29.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:29.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:31:29.330 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":499,"stamp":"2024-09-13T14:31:27.328775+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1935087589,"num_objects":29482,"num_object_clones":0,"num_object_copies":88446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29482,"num_whiteouts":0,"num_read":6390,"num_read_kb":13845,"num_write":30577,"num_write_kb":1900097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":5966043,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":30636,"ondisk_log_size":30636,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":6714668,"kb_used_data":5453752,"kb_used_omap":19,"kb_used_meta":1260780,"kb_avail":555756244,"statfs":{"total":575970213888,"available":569094393856,"internally_reserved":0,"allocated":5584642048,"data_stored":5579703723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20284,"internal_metadata":1291038916},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,5,2,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1570177024,"num_objects":23959,"num_object_clones":0,"num_object_copies":71877,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23959,"num_write_kb":1533376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002182"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":200,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T14:31:22.174315+0000","last_change":"2024-09-13T14:27:12.839984+0000","last_active":"2024-09-13T14:31:22.174315+0000","last_peered":"2024-09-13T14:31:22.174315+0000","last_clean":"2024-09-13T14:31:22.174315+0000","last_became_active":"2024-09-13T14:27:12.839653+0000","last_became_peered":"2024-09-13T14:27:12.839653+0000","last_unstale":"2024-09-13T14:31:22.174315+0000","last_undegraded":"2024-09-13T14:31:22.174315+0000","last_fullsized":"2024-09-13T14:31:22.174315+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:14:13.672830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":147,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:13.355185+0000","last_change":"2024-09-13T14:27:12.846760+0000","last_active":"2024-09-13T14:31:13.355185+0000","last_peered":"2024-09-13T14:31:13.355185+0000","last_clean":"2024-09-13T14:31:13.355185+0000","last_became_active":"2024-09-13T14:27:12.838041+0000","last_became_peered":"2024-09-13T14:27:12.838041+0000","last_unstale":"2024-09-13T14:31:13.355185+0000","last_undegraded":"2024-09-13T14:31:13.355185+0000","last_fullsized":"2024-09-13T14:31:13.355185+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:22.402859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":129,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:13.133292+0000","last_change":"2024-09-13T14:24:55.282407+0000","last_active":"2024-09-13T14:31:13.133292+0000","last_peered":"2024-09-13T14:31:13.133292+0000","last_clean":"2024-09-13T14:31:13.133292+0000","last_became_active":"2024-09-13T14:24:55.282083+0000","last_became_peered":"2024-09-13T14:24:55.282083+0000","last_unstale":"2024-09-13T14:31:13.133292+0000","last_undegraded":"2024-09-13T14:31:13.133292+0000","last_fullsized":"2024-09-13T14:31:13.133292+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:23:10.525707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":177,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T14:31:22.171484+0000","last_change":"2024-09-13T14:24:55.284433+0000","last_active":"2024-09-13T14:31:22.171484+0000","last_peered":"2024-09-13T14:31:22.171484+0000","last_clean":"2024-09-13T14:31:22.171484+0000","last_became_active":"2024-09-13T14:24:55.284143+0000","last_became_peered":"2024-09-13T14:24:55.284143+0000","last_unstale":"2024-09-13T14:31:22.171484+0000","last_undegraded":"2024-09-13T14:31:22.171484+0000","last_fullsized":"2024-09-13T14:31:22.171484+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:28.107405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":179,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T14:31:22.171751+0000","last_change":"2024-09-13T14:24:55.282416+0000","last_active":"2024-09-13T14:31:22.171751+0000","last_peered":"2024-09-13T14:31:22.171751+0000","last_clean":"2024-09-13T14:31:22.171751+0000","last_became_active":"2024-09-13T14:24:55.282045+0000","last_became_peered":"2024-09-13T14:24:55.282045+0000","last_unstale":"2024-09-13T14:31:22.171751+0000","last_undegraded":"2024-09-13T14:31:22.171751+0000","last_fullsized":"2024-09-13T14:31:22.171751+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:58.692897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":115,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:13.355131+0000","last_change":"2024-09-13T14:11:29.244452+0000","last_active":"2024-09-13T14:31:13.355131+0000","last_peered":"2024-09-13T14:31:13.355131+0000","last_clean":"2024-09-13T14:31:13.355131+0000","last_became_active":"2024-09-13T14:11:29.244102+0000","last_became_peered":"2024-09-13T14:11:29.244102+0000","last_unstale":"2024-09-13T14:31:13.355131+0000","last_undegraded":"2024-09-13T14:31:13.355131+0000","last_fullsized":"2024-09-13T14:31:13.355131+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:13:19.165534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":200,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T14:31:22.174079+0000","last_change":"2024-09-13T14:27:12.834300+0000","last_active":"2024-09-13T14:31:22.174079+0000","last_peered":"2024-09-13T14:31:22.174079+0000","last_clean":"2024-09-13T14:31:22.174079+0000","last_became_active":"2024-09-13T14:27:12.834046+0000","last_became_peered":"2024-09-13T14:27:12.834046+0000","last_unstale":"2024-09-13T14:31:22.174079+0000","last_undegraded":"2024-09-13T14:31:22.174079+0000","last_fullsized":"2024-09-13T14:31:22.174079+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:46:58.906862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":138,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:12.317333+0000","last_change":"2024-09-13T14:23:18.039061+0000","last_active":"2024-09-13T14:31:12.317333+0000","last_peered":"2024-09-13T14:31:12.317333+0000","last_clean":"2024-09-13T14:31:12.317333+0000","last_became_active":"2024-09-13T14:23:18.038661+0000","last_became_peered":"2024-09-13T14:23:18.038661+0000","last_unstale":"2024-09-13T14:31:12.317333+0000","last_undegraded":"2024-09-13T14:31:12.317333+0000","last_fullsized":"2024-09-13T14:31:12.317333+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:05:29.109844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"165'1997","reported_seq":2010,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T14:31:27.008256+0000","last_change":"2024-09-13T14:31:04.959615+0000","last_active":"2024-09-13T14:31:27.008256+0000","last_peered":"2024-09-13T14:31:27.008256+0000","last_clean":"2024-09-13T14:31:27.008256+0000","last_became_active":"2024-09-13T14:31:04.959265+0000","last_became_peered":"2024-09-13T14:31:04.959265+0000","last_unstale":"2024-09-13T14:31:27.008256+0000","last_undegraded":"2024-09-13T14:31:27.008256+0000","last_fullsized":"2024-09-13T14:31:27.008256+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_clean_scrub_stamp":"2024-09-13T14:31:03.932624+0000","objects_scrubbed":0,"log_size":1997,"log_dups_size":0,"ondisk_log_size":1997,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:44.257407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130875392,"num_objects":1997,"num_object_clones":0,"num_object_copies":5991,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1997,"num_write_kb":127808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":131,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:13.356256+0000","last_change":"2024-09-13T14:23:18.038597+0000","last_active":"2024-09-13T14:31:13.356256+0000","last_peered":"2024-09-13T14:31:13.356256+0000","last_clean":"2024-09-13T14:31:13.356256+0000","last_became_active":"2024-09-13T14:23:18.038240+0000","last_became_peered":"2024-09-13T14:23:18.038240+0000","last_unstale":"2024-09-13T14:31:13.356256+0000","last_undegraded":"2024-09-13T14:31:13.356256+0000","last_fullsized":"2024-09-13T14:31:13.356256+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:27:51.842195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":115,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:13.132821+0000","last_change":"2024-09-13T14:11:30.254534+0000","last_active":"2024-09-13T14:31:13.132821+0000","last_peered":"2024-09-13T14:31:13.132821+0000","last_clean":"2024-09-13T14:31:13.132821+0000","last_became_active":"2024-09-13T14:11:30.253959+0000","last_became_peered":"2024-09-13T14:11:30.253959+0000","last_unstale":"2024-09-13T14:31:13.132821+0000","last_undegraded":"2024-09-13T14:31:13.132821+0000","last_fullsized":"2024-09-13T14:31:13.132821+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:09:54.025853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.b","version":"165'1518","reported_seq":1537,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T14:31:23.337317+0000","last_change":"2024-09-13T14:31:04.957456+0000","last_active":"2024-09-13T14:31:23.337317+0000","last_peered":"2024-09-13T14:31:23.337317+0000","last_clean":"2024-09-13T14:31:23.337317+0000","last_became_active":"2024-09-13T14:31:04.957189+0000","last_became_peered":"2024-09-13T14:31:04.957189+0000","last_unstale":"2024-09-13T14:31:23.337317+0000","last_undegraded":"2024-09-13T14:31:23.337317+0000","last_fullsized":"2024-09-13T14:31:23.337317+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_clean_scrub_stamp":"2024-09-13T14:31:03.932624+0000","objects_scrubbed":0,"log_size":1518,"log_dups_size":0,"ondisk_log_size":1518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:26:09.289712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99483648,"num_objects":1518,"num_object_clones":0,"num_object_copies":4554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1518,"num_write_kb":97152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":145,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:13.356248+0000","last_change":"2024-09-13T14:26:15.828529+0000","last_active":"2024-09-13T14:31:13.356248+0000","last_peered":"2024-09-13T14:31:13.356248+0000","last_clean":"2024-09-13T14:31:13.356248+0000","last_became_active":"2024-09-13T14:26:15.820486+0000","last_became_peered":"2024-09-13T14:26:15.820486+0000","last_unstale":"2024-09-13T14:31:13.356248+0000","last_undegraded":"2024-09-13T14:31:13.356248+0000","last_fullsized":"2024-09-13T14:31:13.356248+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:11:46.283172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":153,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:11.962688+0000","last_change":"2024-09-13T14:26:15.824632+0000","last_active":"2024-09-13T14:31:11.962688+0000","last_peered":"2024-09-13T14:31:11.962688+0000","last_clean":"2024-09-13T14:31:11.962688+0000","last_became_active":"2024-09-13T14:26:15.824330+0000","last_became_peered":"2024-09-13T14:26:15.824330+0000","last_unstale":"2024-09-13T14:31:11.962688+0000","last_undegraded":"2024-09-13T14:31:11.962688+0000","last_fullsized":"2024-09-13T14:31:11.962688+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:25.288116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"165'1957","reported_seq":1970,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T14:31:26.939402+0000","last_change":"2024-09-13T14:31:04.953557+0000","last_active":"2024-09-13T14:31:26.939402+0000","last_peered":"2024-09-13T14:31:26.939402+0000","last_clean":"2024-09-13T14:31:26.939402+0000","last_became_active":"2024-09-13T14:31:04.953250+0000","last_became_peered":"2024-09-13T14:31:04.953250+0000","last_unstale":"2024-09-13T14:31:26.939402+0000","last_undegraded":"2024-09-13T14:31:26.939402+0000","last_fullsized":"2024-09-13T14:31:26.939402+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_clean_scrub_stamp":"2024-09-13T14:31:03.932624+0000","objects_scrubbed":0,"log_size":1957,"log_dups_size":0,"ondisk_log_size":1957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:48.668087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128253952,"num_objects":1957,"num_object_clones":0,"num_object_copies":5871,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1957,"num_write_kb":125248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":116,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:13.355856+0000","last_change":"2024-09-13T14:11:29.232897+0000","last_active":"2024-09-13T14:31:13.355856+0000","last_peered":"2024-09-13T14:31:13.355856+0000","last_clean":"2024-09-13T14:31:13.355856+0000","last_became_active":"2024-09-13T14:11:29.232637+0000","last_became_peered":"2024-09-13T14:11:29.232637+0000","last_unstale":"2024-09-13T14:31:13.355856+0000","last_undegraded":"2024-09-13T14:31:13.355856+0000","last_fullsized":"2024-09-13T14:31:13.355856+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:31:05.856046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":139,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:12.317321+0000","last_change":"2024-09-13T14:26:15.822452+0000","last_active":"2024-09-13T14:31:12.317321+0000","last_peered":"2024-09-13T14:31:12.317321+0000","last_clean":"2024-09-13T14:31:12.317321+0000","last_became_active":"2024-09-13T14:26:15.822111+0000","last_became_peered":"2024-09-13T14:26:15.822111+0000","last_unstale":"2024-09-13T14:31:12.317321+0000","last_undegraded":"2024-09-13T14:31:12.317321+0000","last_fullsized":"2024-09-13T14:31:12.317321+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:32:53.078767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"165'1881","reported_seq":1900,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T14:31:26.501210+0000","last_change":"2024-09-13T14:31:04.953002+0000","last_active":"2024-09-13T14:31:26.501210+0000","last_peered":"2024-09-13T14:31:26.501210+0000","last_clean":"2024-09-13T14:31:26.501210+0000","last_became_active":"2024-09-13T14:31:04.952667+0000","last_became_peered":"2024-09-13T14:31:04.952667+0000","last_unstale":"2024-09-13T14:31:26.501210+0000","last_undegraded":"2024-09-13T14:31:26.501210+0000","last_fullsized":"2024-09-13T14:31:26.501210+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_clean_scrub_stamp":"2024-09-13T14:31:03.932624+0000","objects_scrubbed":0,"log_size":1881,"log_dups_size":0,"ondisk_log_size":1881,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:22:35.630717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123273216,"num_objects":1881,"num_object_clones":0,"num_object_copies":5643,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1881,"num_write_kb":120384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":130,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:13.133258+0000","last_change":"2024-09-13T14:24:55.290798+0000","last_active":"2024-09-13T14:31:13.133258+0000","last_peered":"2024-09-13T14:31:13.133258+0000","last_clean":"2024-09-13T14:31:13.133258+0000","last_became_active":"2024-09-13T14:24:55.290510+0000","last_became_peered":"2024-09-13T14:24:55.290510+0000","last_unstale":"2024-09-13T14:31:13.133258+0000","last_undegraded":"2024-09-13T14:31:13.133258+0000","last_fullsized":"2024-09-13T14:31:13.133258+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:19:03.129132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":186,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T14:31:22.171698+0000","last_change":"2024-09-13T14:26:15.823316+0000","last_active":"2024-09-13T14:31:22.171698+0000","last_peered":"2024-09-13T14:31:22.171698+0000","last_clean":"2024-09-13T14:31:22.171698+0000","last_became_active":"2024-09-13T14:26:15.822980+0000","last_became_peered":"2024-09-13T14:26:15.822980+0000","last_unstale":"2024-09-13T14:31:22.171698+0000","last_undegraded":"2024-09-13T14:31:22.171698+0000","last_fullsized":"2024-09-13T14:31:22.171698+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:33:47.420548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"165'2000","reported_seq":2013,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T14:31:27.211538+0000","last_change":"2024-09-13T14:31:04.958431+0000","last_active":"2024-09-13T14:31:27.211538+0000","last_peered":"2024-09-13T14:31:27.211538+0000","last_clean":"2024-09-13T14:31:27.211538+0000","last_became_active":"2024-09-13T14:31:04.958121+0000","last_became_peered":"2024-09-13T14:31:04.958121+0000","last_unstale":"2024-09-13T14:31:27.211538+0000","last_undegraded":"2024-09-13T14:31:27.211538+0000","last_fullsized":"2024-09-13T14:31:27.211538+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_clean_scrub_stamp":"2024-09-13T14:31:03.932624+0000","objects_scrubbed":0,"log_size":2000,"log_dups_size":0,"ondisk_log_size":2000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:09:51.570034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131072000,"num_objects":2000,"num_object_clones":0,"num_object_copies":6000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2000,"num_write_kb":128000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":184,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T14:31:22.173779+0000","last_change":"2024-09-13T14:24:55.284499+0000","last_active":"2024-09-13T14:31:22.173779+0000","last_peered":"2024-09-13T14:31:22.173779+0000","last_clean":"2024-09-13T14:31:22.173779+0000","last_became_active":"2024-09-13T14:24:55.284096+0000","last_became_peered":"2024-09-13T14:24:55.284096+0000","last_unstale":"2024-09-13T14:31:22.173779+0000","last_undegraded":"2024-09-13T14:31:22.173779+0000","last_fullsized":"2024-09-13T14:31:22.173779+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:02:06.836099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":128,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:11.962531+0000","last_change":"2024-09-13T14:24:55.291371+0000","last_active":"2024-09-13T14:31:11.962531+0000","last_peered":"2024-09-13T14:31:11.962531+0000","last_clean":"2024-09-13T14:31:11.962531+0000","last_became_active":"2024-09-13T14:24:55.291047+0000","last_became_peered":"2024-09-13T14:24:55.291047+0000","last_unstale":"2024-09-13T14:31:11.962531+0000","last_undegraded":"2024-09-13T14:31:11.962531+0000","last_fullsized":"2024-09-13T14:31:11.962531+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:43.912412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"165'1561","reported_seq":1580,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T14:31:23.356262+0000","last_change":"2024-09-13T14:31:04.956319+0000","last_active":"2024-09-13T14:31:23.356262+0000","last_peered":"2024-09-13T14:31:23.356262+0000","last_clean":"2024-09-13T14:31:23.356262+0000","last_became_active":"2024-09-13T14:31:04.956014+0000","last_became_peered":"2024-09-13T14:31:04.956014+0000","last_unstale":"2024-09-13T14:31:23.356262+0000","last_undegraded":"2024-09-13T14:31:23.356262+0000","last_fullsized":"2024-09-13T14:31:23.356262+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_clean_scrub_stamp":"2024-09-13T14:31:03.932624+0000","objects_scrubbed":0,"log_size":1561,"log_dups_size":0,"ondisk_log_size":1561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:51:35.156658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102301696,"num_objects":1561,"num_object_clones":0,"num_object_copies":4683,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1561,"num_write_kb":99904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":140,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:12.317525+0000","last_change":"2024-09-13T14:26:15.827333+0000","last_active":"2024-09-13T14:31:12.317525+0000","last_peered":"2024-09-13T14:31:12.317525+0000","last_clean":"2024-09-13T14:31:12.317525+0000","last_became_active":"2024-09-13T14:26:15.826945+0000","last_became_peered":"2024-09-13T14:26:15.826945+0000","last_unstale":"2024-09-13T14:31:12.317525+0000","last_undegraded":"2024-09-13T14:31:12.317525+0000","last_fullsized":"2024-09-13T14:31:12.317525+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:35:24.870052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":161,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:11.962747+0000","last_change":"2024-09-13T14:27:12.842604+0000","last_active":"2024-09-13T14:31:11.962747+0000","last_peered":"2024-09-13T14:31:11.962747+0000","last_clean":"2024-09-13T14:31:11.962747+0000","last_became_active":"2024-09-13T14:27:12.841643+0000","last_became_peered":"2024-09-13T14:27:12.841643+0000","last_unstale":"2024-09-13T14:31:11.962747+0000","last_undegraded":"2024-09-13T14:31:11.962747+0000","last_fullsized":"2024-09-13T14:31:11.962747+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:34:34.751790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"165'1464","reported_seq":1482,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T14:31:23.344154+0000","last_change":"2024-09-13T14:31:04.957232+0000","last_active":"2024-09-13T14:31:23.344154+0000","last_peered":"2024-09-13T14:31:23.344154+0000","last_clean":"2024-09-13T14:31:23.344154+0000","last_became_active":"2024-09-13T14:31:04.956942+0000","last_became_peered":"2024-09-13T14:31:04.956942+0000","last_unstale":"2024-09-13T14:31:23.344154+0000","last_undegraded":"2024-09-13T14:31:23.344154+0000","last_fullsized":"2024-09-13T14:31:23.344154+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_clean_scrub_stamp":"2024-09-13T14:31:03.932624+0000","objects_scrubbed":0,"log_size":1464,"log_dups_size":0,"ondisk_log_size":1464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:13:50.184110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95944704,"num_objects":1464,"num_object_clones":0,"num_object_copies":4392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1464,"num_write_kb":93696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":139,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:11.962524+0000","last_change":"2024-09-13T14:27:12.842271+0000","last_active":"2024-09-13T14:31:11.962524+0000","last_peered":"2024-09-13T14:31:11.962524+0000","last_clean":"2024-09-13T14:31:11.962524+0000","last_became_active":"2024-09-13T14:27:12.841479+0000","last_became_peered":"2024-09-13T14:27:12.841479+0000","last_unstale":"2024-09-13T14:31:11.962524+0000","last_undegraded":"2024-09-13T14:31:11.962524+0000","last_fullsized":"2024-09-13T14:31:11.962524+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:30:22.746575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":137,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:11.962865+0000","last_change":"2024-09-13T14:26:15.825304+0000","last_active":"2024-09-13T14:31:11.962865+0000","last_peered":"2024-09-13T14:31:11.962865+0000","last_clean":"2024-09-13T14:31:11.962865+0000","last_became_active":"2024-09-13T14:26:15.825018+0000","last_became_peered":"2024-09-13T14:26:15.825018+0000","last_unstale":"2024-09-13T14:31:11.962865+0000","last_undegraded":"2024-09-13T14:31:11.962865+0000","last_fullsized":"2024-09-13T14:31:11.962865+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:22:59.408928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"165'1952","reported_seq":1965,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T14:31:26.935986+0000","last_change":"2024-09-13T14:31:04.960524+0000","last_active":"2024-09-13T14:31:26.935986+0000","last_peered":"2024-09-13T14:31:26.935986+0000","last_clean":"2024-09-13T14:31:26.935986+0000","last_became_active":"2024-09-13T14:31:04.960207+0000","last_became_peered":"2024-09-13T14:31:04.960207+0000","last_unstale":"2024-09-13T14:31:26.935986+0000","last_undegraded":"2024-09-13T14:31:26.935986+0000","last_fullsized":"2024-09-13T14:31:26.935986+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_clean_scrub_stamp":"2024-09-13T14:31:03.932624+0000","objects_scrubbed":0,"log_size":1952,"log_dups_size":0,"ondisk_log_size":1952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:48:58.950478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127926272,"num_objects":1952,"num_object_clones":0,"num_object_copies":5856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1952,"num_write_kb":124928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":195,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T14:31:22.173670+0000","last_change":"2024-09-13T14:26:15.823898+0000","last_active":"2024-09-13T14:31:22.173670+0000","last_peered":"2024-09-13T14:31:22.173670+0000","last_clean":"2024-09-13T14:31:22.173670+0000","last_became_active":"2024-09-13T14:26:15.823125+0000","last_became_peered":"2024-09-13T14:26:15.823125+0000","last_unstale":"2024-09-13T14:31:22.173670+0000","last_undegraded":"2024-09-13T14:31:22.173670+0000","last_fullsized":"2024-09-13T14:31:22.173670+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:51:27.949528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":157,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T14:31:22.174208+0000","last_change":"2024-09-13T14:26:15.823198+0000","last_active":"2024-09-13T14:31:22.174208+0000","last_peered":"2024-09-13T14:31:22.174208+0000","last_clean":"2024-09-13T14:31:22.174208+0000","last_became_active":"2024-09-13T14:26:15.822717+0000","last_became_peered":"2024-09-13T14:26:15.822717+0000","last_unstale":"2024-09-13T14:31:22.174208+0000","last_undegraded":"2024-09-13T14:31:22.174208+0000","last_fullsized":"2024-09-13T14:31:22.174208+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:22.471622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"165'1972","reported_seq":1985,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T14:31:26.890967+0000","last_change":"2024-09-13T14:31:04.959681+0000","last_active":"2024-09-13T14:31:26.890967+0000","last_peered":"2024-09-13T14:31:26.890967+0000","last_clean":"2024-09-13T14:31:26.890967+0000","last_became_active":"2024-09-13T14:31:04.959394+0000","last_became_peered":"2024-09-13T14:31:04.959394+0000","last_unstale":"2024-09-13T14:31:26.890967+0000","last_undegraded":"2024-09-13T14:31:26.890967+0000","last_fullsized":"2024-09-13T14:31:26.890967+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_clean_scrub_stamp":"2024-09-13T14:31:03.932624+0000","objects_scrubbed":0,"log_size":1972,"log_dups_size":0,"ondisk_log_size":1972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:54:25.539991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129236992,"num_objects":1972,"num_object_clones":0,"num_object_copies":5916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1972,"num_write_kb":126208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":461,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:13.355734+0000","last_change":"2024-09-13T14:23:18.032973+0000","last_active":"2024-09-13T14:31:13.355734+0000","last_peered":"2024-09-13T14:31:13.355734+0000","last_clean":"2024-09-13T14:31:13.355734+0000","last_became_active":"2024-09-13T14:23:18.032579+0000","last_became_peered":"2024-09-13T14:23:18.032579+0000","last_unstale":"2024-09-13T14:31:13.355734+0000","last_undegraded":"2024-09-13T14:31:13.355734+0000","last_fullsized":"2024-09-13T14:31:13.355734+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:07:41.927740+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:17.109445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":495,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:13.133172+0000","last_change":"2024-09-13T14:27:15.865400+0000","last_active":"2024-09-13T14:31:13.133172+0000","last_peered":"2024-09-13T14:31:13.133172+0000","last_clean":"2024-09-13T14:31:13.133172+0000","last_became_active":"2024-09-13T14:27:15.864835+0000","last_became_peered":"2024-09-13T14:27:15.864835+0000","last_unstale":"2024-09-13T14:31:13.133172+0000","last_undegraded":"2024-09-13T14:31:13.133172+0000","last_fullsized":"2024-09-13T14:31:13.133172+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:27:10.928041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":136,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T14:31:22.173325+0000","last_change":"2024-09-13T14:24:55.288058+0000","last_active":"2024-09-13T14:31:22.173325+0000","last_peered":"2024-09-13T14:31:22.173325+0000","last_clean":"2024-09-13T14:31:22.173325+0000","last_became_active":"2024-09-13T14:24:55.287601+0000","last_became_peered":"2024-09-13T14:24:55.287601+0000","last_unstale":"2024-09-13T14:31:22.173325+0000","last_undegraded":"2024-09-13T14:31:22.173325+0000","last_fullsized":"2024-09-13T14:31:22.173325+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:56:34.622696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":115,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:12.317792+0000","last_change":"2024-09-13T14:11:30.253749+0000","last_active":"2024-09-13T14:31:12.317792+0000","last_peered":"2024-09-13T14:31:12.317792+0000","last_clean":"2024-09-13T14:31:12.317792+0000","last_became_active":"2024-09-13T14:11:30.253499+0000","last_became_peered":"2024-09-13T14:11:30.253499+0000","last_unstale":"2024-09-13T14:31:12.317792+0000","last_undegraded":"2024-09-13T14:31:12.317792+0000","last_fullsized":"2024-09-13T14:31:12.317792+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:02:00.987335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"165'1927","reported_seq":1940,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T14:31:26.939720+0000","last_change":"2024-09-13T14:31:04.961057+0000","last_active":"2024-09-13T14:31:26.939720+0000","last_peered":"2024-09-13T14:31:26.939720+0000","last_clean":"2024-09-13T14:31:26.939720+0000","last_became_active":"2024-09-13T14:31:04.960799+0000","last_became_peered":"2024-09-13T14:31:04.960799+0000","last_unstale":"2024-09-13T14:31:26.939720+0000","last_undegraded":"2024-09-13T14:31:26.939720+0000","last_fullsized":"2024-09-13T14:31:26.939720+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_clean_scrub_stamp":"2024-09-13T14:31:03.932624+0000","objects_scrubbed":0,"log_size":1927,"log_dups_size":0,"ondisk_log_size":1927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:57:45.740628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":126287872,"num_objects":1927,"num_object_clones":0,"num_object_copies":5781,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1927,"num_write_kb":123328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":376,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:13.355669+0000","last_change":"2024-09-13T14:26:15.835178+0000","last_active":"2024-09-13T14:31:13.355669+0000","last_peered":"2024-09-13T14:31:13.355669+0000","last_clean":"2024-09-13T14:31:13.355669+0000","last_became_active":"2024-09-13T14:26:15.820283+0000","last_became_peered":"2024-09-13T14:26:15.820283+0000","last_unstale":"2024-09-13T14:31:13.355669+0000","last_undegraded":"2024-09-13T14:31:13.355669+0000","last_fullsized":"2024-09-13T14:31:13.355669+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:30.340038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":178,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T14:31:22.171977+0000","last_change":"2024-09-13T14:24:55.286576+0000","last_active":"2024-09-13T14:31:22.171977+0000","last_peered":"2024-09-13T14:31:22.171977+0000","last_clean":"2024-09-13T14:31:22.171977+0000","last_became_active":"2024-09-13T14:24:55.286284+0000","last_became_peered":"2024-09-13T14:24:55.286284+0000","last_unstale":"2024-09-13T14:31:22.171977+0000","last_undegraded":"2024-09-13T14:31:22.171977+0000","last_fullsized":"2024-09-13T14:31:22.171977+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:35.589736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":204,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T14:31:22.171849+0000","last_change":"2024-09-13T14:27:12.836204+0000","last_active":"2024-09-13T14:31:22.171849+0000","last_peered":"2024-09-13T14:31:22.171849+0000","last_clean":"2024-09-13T14:31:22.171849+0000","last_became_active":"2024-09-13T14:27:12.835900+0000","last_became_peered":"2024-09-13T14:27:12.835900+0000","last_unstale":"2024-09-13T14:31:22.171849+0000","last_undegraded":"2024-09-13T14:31:22.171849+0000","last_fullsized":"2024-09-13T14:31:22.171849+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:36:52.121730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"165'1981","reported_seq":1994,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T14:31:26.930429+0000","last_change":"2024-09-13T14:31:04.954224+0000","last_active":"2024-09-13T14:31:26.930429+0000","last_peered":"2024-09-13T14:31:26.930429+0000","last_clean":"2024-09-13T14:31:26.930429+0000","last_became_active":"2024-09-13T14:31:04.953723+0000","last_became_peered":"2024-09-13T14:31:04.953723+0000","last_unstale":"2024-09-13T14:31:26.930429+0000","last_undegraded":"2024-09-13T14:31:26.930429+0000","last_fullsized":"2024-09-13T14:31:26.930429+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_clean_scrub_stamp":"2024-09-13T14:31:03.932624+0000","objects_scrubbed":0,"log_size":1981,"log_dups_size":0,"ondisk_log_size":1981,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:41:04.638415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129826816,"num_objects":1981,"num_object_clones":0,"num_object_copies":5943,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1981,"num_write_kb":126784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":559,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T14:31:22.174019+0000","last_change":"2024-09-13T14:27:15.863966+0000","last_active":"2024-09-13T14:31:22.174019+0000","last_peered":"2024-09-13T14:31:22.174019+0000","last_clean":"2024-09-13T14:31:22.174019+0000","last_became_active":"2024-09-13T14:27:15.863124+0000","last_became_peered":"2024-09-13T14:27:15.863124+0000","last_unstale":"2024-09-13T14:31:22.174019+0000","last_undegraded":"2024-09-13T14:31:22.174019+0000","last_fullsized":"2024-09-13T14:31:22.174019+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:35:02.715013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":90,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:16.368671+0000","last_change":"2024-09-13T14:27:12.829079+0000","last_active":"2024-09-13T14:31:16.368671+0000","last_peered":"2024-09-13T14:31:16.368671+0000","last_clean":"2024-09-13T14:31:16.368671+0000","last_became_active":"2024-09-13T14:27:12.828619+0000","last_became_peered":"2024-09-13T14:27:12.828619+0000","last_unstale":"2024-09-13T14:31:16.368671+0000","last_undegraded":"2024-09-13T14:31:16.368671+0000","last_fullsized":"2024-09-13T14:31:16.368671+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:39:08.260304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":126,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:13.133408+0000","last_change":"2024-09-13T14:26:15.817732+0000","last_active":"2024-09-13T14:31:13.133408+0000","last_peered":"2024-09-13T14:31:13.133408+0000","last_clean":"2024-09-13T14:31:13.133408+0000","last_became_active":"2024-09-13T14:26:15.817403+0000","last_became_peered":"2024-09-13T14:26:15.817403+0000","last_unstale":"2024-09-13T14:31:13.133408+0000","last_undegraded":"2024-09-13T14:31:13.133408+0000","last_fullsized":"2024-09-13T14:31:13.133408+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:56:39.961300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"165'1532","reported_seq":1551,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T14:31:23.327045+0000","last_change":"2024-09-13T14:31:04.953392+0000","last_active":"2024-09-13T14:31:23.327045+0000","last_peered":"2024-09-13T14:31:23.327045+0000","last_clean":"2024-09-13T14:31:23.327045+0000","last_became_active":"2024-09-13T14:31:04.953114+0000","last_became_peered":"2024-09-13T14:31:04.953114+0000","last_unstale":"2024-09-13T14:31:23.327045+0000","last_undegraded":"2024-09-13T14:31:23.327045+0000","last_fullsized":"2024-09-13T14:31:23.327045+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_clean_scrub_stamp":"2024-09-13T14:31:03.932624+0000","objects_scrubbed":0,"log_size":1532,"log_dups_size":0,"ondisk_log_size":1532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:41:13.856511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100401152,"num_objects":1532,"num_object_clones":0,"num_object_copies":4596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1532,"num_write_kb":98048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":926,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:13.133230+0000","last_change":"2024-09-13T14:26:18.829131+0000","last_active":"2024-09-13T14:31:13.133230+0000","last_peered":"2024-09-13T14:31:13.133230+0000","last_clean":"2024-09-13T14:31:13.133230+0000","last_became_active":"2024-09-13T14:26:18.828619+0000","last_became_peered":"2024-09-13T14:26:18.828619+0000","last_unstale":"2024-09-13T14:31:13.133230+0000","last_undegraded":"2024-09-13T14:31:13.133230+0000","last_fullsized":"2024-09-13T14:31:13.133230+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:24:37.073618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":130,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:11.962651+0000","last_change":"2024-09-13T14:23:18.042891+0000","last_active":"2024-09-13T14:31:11.962651+0000","last_peered":"2024-09-13T14:31:11.962651+0000","last_clean":"2024-09-13T14:31:11.962651+0000","last_became_active":"2024-09-13T14:23:18.042562+0000","last_became_peered":"2024-09-13T14:23:18.042562+0000","last_unstale":"2024-09-13T14:31:11.962651+0000","last_undegraded":"2024-09-13T14:31:11.962651+0000","last_fullsized":"2024-09-13T14:31:11.962651+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:30:42.996661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":172,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T14:31:22.174255+0000","last_change":"2024-09-13T14:27:12.832666+0000","last_active":"2024-09-13T14:31:22.174255+0000","last_peered":"2024-09-13T14:31:22.174255+0000","last_clean":"2024-09-13T14:31:22.174255+0000","last_became_active":"2024-09-13T14:27:12.832332+0000","last_became_peered":"2024-09-13T14:27:12.832332+0000","last_unstale":"2024-09-13T14:31:22.174255+0000","last_undegraded":"2024-09-13T14:31:22.174255+0000","last_fullsized":"2024-09-13T14:31:22.174255+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:07:17.724156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"165'1937","reported_seq":1956,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T14:31:26.508439+0000","last_change":"2024-09-13T14:31:04.956472+0000","last_active":"2024-09-13T14:31:26.508439+0000","last_peered":"2024-09-13T14:31:26.508439+0000","last_clean":"2024-09-13T14:31:26.508439+0000","last_became_active":"2024-09-13T14:31:04.956181+0000","last_became_peered":"2024-09-13T14:31:04.956181+0000","last_unstale":"2024-09-13T14:31:26.508439+0000","last_undegraded":"2024-09-13T14:31:26.508439+0000","last_fullsized":"2024-09-13T14:31:26.508439+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_clean_scrub_stamp":"2024-09-13T14:31:03.932624+0000","objects_scrubbed":0,"log_size":1937,"log_dups_size":0,"ondisk_log_size":1937,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:00:30.194516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":126943232,"num_objects":1937,"num_object_clones":0,"num_object_copies":5811,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1937,"num_write_kb":123968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":155,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:13.132808+0000","last_change":"2024-09-13T14:27:12.843038+0000","last_active":"2024-09-13T14:31:13.132808+0000","last_peered":"2024-09-13T14:31:13.132808+0000","last_clean":"2024-09-13T14:31:13.132808+0000","last_became_active":"2024-09-13T14:27:12.842663+0000","last_became_peered":"2024-09-13T14:27:12.842663+0000","last_unstale":"2024-09-13T14:31:13.132808+0000","last_undegraded":"2024-09-13T14:31:13.132808+0000","last_fullsized":"2024-09-13T14:31:13.132808+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:23:38.880811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":908,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T14:31:22.171563+0000","last_change":"2024-09-13T14:30:58.928302+0000","last_active":"2024-09-13T14:31:22.171563+0000","last_peered":"2024-09-13T14:31:22.171563+0000","last_clean":"2024-09-13T14:31:22.171563+0000","last_became_active":"2024-09-13T14:30:58.927718+0000","last_became_peered":"2024-09-13T14:30:58.927718+0000","last_unstale":"2024-09-13T14:31:22.171563+0000","last_undegraded":"2024-09-13T14:31:22.171563+0000","last_fullsized":"2024-09-13T14:31:22.171563+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:37.323036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":82,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:13.132981+0000","last_change":"2024-09-13T14:26:15.819032+0000","last_active":"2024-09-13T14:31:13.132981+0000","last_peered":"2024-09-13T14:31:13.132981+0000","last_clean":"2024-09-13T14:31:13.132981+0000","last_became_active":"2024-09-13T14:26:15.818675+0000","last_became_peered":"2024-09-13T14:26:15.818675+0000","last_unstale":"2024-09-13T14:31:13.132981+0000","last_undegraded":"2024-09-13T14:31:13.132981+0000","last_fullsized":"2024-09-13T14:31:13.132981+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:37:13.628256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"165'1900","reported_seq":1913,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T14:31:26.355843+0000","last_change":"2024-09-13T14:31:04.954718+0000","last_active":"2024-09-13T14:31:26.355843+0000","last_peered":"2024-09-13T14:31:26.355843+0000","last_clean":"2024-09-13T14:31:26.355843+0000","last_became_active":"2024-09-13T14:31:04.954428+0000","last_became_peered":"2024-09-13T14:31:04.954428+0000","last_unstale":"2024-09-13T14:31:26.355843+0000","last_undegraded":"2024-09-13T14:31:26.355843+0000","last_fullsized":"2024-09-13T14:31:26.355843+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_clean_scrub_stamp":"2024-09-13T14:31:03.932624+0000","objects_scrubbed":0,"log_size":1900,"log_dups_size":0,"ondisk_log_size":1900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:45:36.697605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124518400,"num_objects":1900,"num_object_clones":0,"num_object_copies":5700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1900,"num_write_kb":121600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":90,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T14:31:22.172760+0000","last_change":"2024-09-13T14:27:12.843268+0000","last_active":"2024-09-13T14:31:22.172760+0000","last_peered":"2024-09-13T14:31:22.172760+0000","last_clean":"2024-09-13T14:31:22.172760+0000","last_became_active":"2024-09-13T14:27:12.842867+0000","last_became_peered":"2024-09-13T14:27:12.842867+0000","last_unstale":"2024-09-13T14:31:22.172760+0000","last_undegraded":"2024-09-13T14:31:22.172760+0000","last_fullsized":"2024-09-13T14:31:22.172760+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:03:59.575866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":653,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:13.356261+0000","last_change":"2024-09-13T14:23:18.038847+0000","last_active":"2024-09-13T14:31:13.356261+0000","last_peered":"2024-09-13T14:31:13.356261+0000","last_clean":"2024-09-13T14:31:13.356261+0000","last_became_active":"2024-09-13T14:23:18.038188+0000","last_became_peered":"2024-09-13T14:23:18.038188+0000","last_unstale":"2024-09-13T14:31:13.356261+0000","last_undegraded":"2024-09-13T14:31:13.356261+0000","last_fullsized":"2024-09-13T14:31:13.356261+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:30:37.213496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":168,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T14:31:22.171596+0000","last_change":"2024-09-13T14:23:18.034427+0000","last_active":"2024-09-13T14:31:22.171596+0000","last_peered":"2024-09-13T14:31:22.171596+0000","last_clean":"2024-09-13T14:31:22.171596+0000","last_became_active":"2024-09-13T14:23:18.034092+0000","last_became_peered":"2024-09-13T14:23:18.034092+0000","last_unstale":"2024-09-13T14:31:22.171596+0000","last_undegraded":"2024-09-13T14:31:22.171596+0000","last_fullsized":"2024-09-13T14:31:22.171596+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:21.755115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"165'1966","reported_seq":1979,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T14:31:27.243043+0000","last_change":"2024-09-13T14:31:04.957425+0000","last_active":"2024-09-13T14:31:27.243043+0000","last_peered":"2024-09-13T14:31:27.243043+0000","last_clean":"2024-09-13T14:31:27.243043+0000","last_became_active":"2024-09-13T14:31:04.957101+0000","last_became_peered":"2024-09-13T14:31:04.957101+0000","last_unstale":"2024-09-13T14:31:27.243043+0000","last_undegraded":"2024-09-13T14:31:27.243043+0000","last_fullsized":"2024-09-13T14:31:27.243043+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_clean_scrub_stamp":"2024-09-13T14:31:03.932624+0000","objects_scrubbed":0,"log_size":1966,"log_dups_size":0,"ondisk_log_size":1966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:14:30.110858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128843776,"num_objects":1966,"num_object_clones":0,"num_object_copies":5898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1966,"num_write_kb":125824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":185,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T14:31:22.174287+0000","last_change":"2024-09-13T14:24:55.290711+0000","last_active":"2024-09-13T14:31:22.174287+0000","last_peered":"2024-09-13T14:31:22.174287+0000","last_clean":"2024-09-13T14:31:22.174287+0000","last_became_active":"2024-09-13T14:24:55.288549+0000","last_became_peered":"2024-09-13T14:24:55.288549+0000","last_unstale":"2024-09-13T14:31:22.174287+0000","last_undegraded":"2024-09-13T14:31:22.174287+0000","last_fullsized":"2024-09-13T14:31:22.174287+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:55:21.763931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":566,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:12.317655+0000","last_change":"2024-09-13T14:26:18.827876+0000","last_active":"2024-09-13T14:31:12.317655+0000","last_peered":"2024-09-13T14:31:12.317655+0000","last_clean":"2024-09-13T14:31:12.317655+0000","last_became_active":"2024-09-13T14:26:18.827317+0000","last_became_peered":"2024-09-13T14:26:18.827317+0000","last_unstale":"2024-09-13T14:31:12.317655+0000","last_undegraded":"2024-09-13T14:31:12.317655+0000","last_fullsized":"2024-09-13T14:31:12.317655+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:31:48.611286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":182,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T14:31:22.173320+0000","last_change":"2024-09-13T14:24:55.281593+0000","last_active":"2024-09-13T14:31:22.173320+0000","last_peered":"2024-09-13T14:31:22.173320+0000","last_clean":"2024-09-13T14:31:22.173320+0000","last_became_active":"2024-09-13T14:24:55.280805+0000","last_became_peered":"2024-09-13T14:24:55.280805+0000","last_unstale":"2024-09-13T14:31:22.173320+0000","last_undegraded":"2024-09-13T14:31:22.173320+0000","last_fullsized":"2024-09-13T14:31:22.173320+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:14.806713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"165'1909","reported_seq":1922,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T14:31:26.939637+0000","last_change":"2024-09-13T14:31:04.956103+0000","last_active":"2024-09-13T14:31:26.939637+0000","last_peered":"2024-09-13T14:31:26.939637+0000","last_clean":"2024-09-13T14:31:26.939637+0000","last_became_active":"2024-09-13T14:31:04.955773+0000","last_became_peered":"2024-09-13T14:31:04.955773+0000","last_unstale":"2024-09-13T14:31:26.939637+0000","last_undegraded":"2024-09-13T14:31:26.939637+0000","last_fullsized":"2024-09-13T14:31:26.939637+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_clean_scrub_stamp":"2024-09-13T14:31:03.932624+0000","objects_scrubbed":0,"log_size":1909,"log_dups_size":0,"ondisk_log_size":1909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:36:08.391308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125108224,"num_objects":1909,"num_object_clones":0,"num_object_copies":5727,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1909,"num_write_kb":122176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":193,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T14:31:22.171691+0000","last_change":"2024-09-13T14:26:15.830309+0000","last_active":"2024-09-13T14:31:22.171691+0000","last_peered":"2024-09-13T14:31:22.171691+0000","last_clean":"2024-09-13T14:31:22.171691+0000","last_became_active":"2024-09-13T14:26:15.823041+0000","last_became_peered":"2024-09-13T14:26:15.823041+0000","last_unstale":"2024-09-13T14:31:22.171691+0000","last_undegraded":"2024-09-13T14:31:22.171691+0000","last_fullsized":"2024-09-13T14:31:22.171691+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:23:08.158576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":596,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:12.317748+0000","last_change":"2024-09-13T14:27:12.848354+0000","last_active":"2024-09-13T14:31:12.317748+0000","last_peered":"2024-09-13T14:31:12.317748+0000","last_clean":"2024-09-13T14:31:12.317748+0000","last_became_active":"2024-09-13T14:27:12.832182+0000","last_became_peered":"2024-09-13T14:27:12.832182+0000","last_unstale":"2024-09-13T14:31:12.317748+0000","last_undegraded":"2024-09-13T14:31:12.317748+0000","last_fullsized":"2024-09-13T14:31:12.317748+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:46:42.064509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":114,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:11.962391+0000","last_change":"2024-09-13T14:11:30.256034+0000","last_active":"2024-09-13T14:31:11.962391+0000","last_peered":"2024-09-13T14:31:11.962391+0000","last_clean":"2024-09-13T14:31:11.962391+0000","last_became_active":"2024-09-13T14:11:30.255727+0000","last_became_peered":"2024-09-13T14:11:30.255727+0000","last_unstale":"2024-09-13T14:31:11.962391+0000","last_undegraded":"2024-09-13T14:31:11.962391+0000","last_fullsized":"2024-09-13T14:31:11.962391+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:21:35.817014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":137,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:11.962525+0000","last_change":"2024-09-13T14:26:15.825390+0000","last_active":"2024-09-13T14:31:11.962525+0000","last_peered":"2024-09-13T14:31:11.962525+0000","last_clean":"2024-09-13T14:31:11.962525+0000","last_became_active":"2024-09-13T14:26:15.825084+0000","last_became_peered":"2024-09-13T14:26:15.825084+0000","last_unstale":"2024-09-13T14:31:11.962525+0000","last_undegraded":"2024-09-13T14:31:11.962525+0000","last_fullsized":"2024-09-13T14:31:11.962525+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:44.680451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":137,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:11.962614+0000","last_change":"2024-09-13T14:11:29.237013+0000","last_active":"2024-09-13T14:31:11.962614+0000","last_peered":"2024-09-13T14:31:11.962614+0000","last_clean":"2024-09-13T14:31:11.962614+0000","last_became_active":"2024-09-13T14:11:29.236643+0000","last_became_peered":"2024-09-13T14:11:29.236643+0000","last_unstale":"2024-09-13T14:31:11.962614+0000","last_undegraded":"2024-09-13T14:31:11.962614+0000","last_fullsized":"2024-09-13T14:31:11.962614+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:33:58.438693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":112,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:12.317483+0000","last_change":"2024-09-13T14:27:12.843904+0000","last_active":"2024-09-13T14:31:12.317483+0000","last_peered":"2024-09-13T14:31:12.317483+0000","last_clean":"2024-09-13T14:31:12.317483+0000","last_became_active":"2024-09-13T14:27:12.842566+0000","last_became_peered":"2024-09-13T14:27:12.842566+0000","last_unstale":"2024-09-13T14:31:12.317483+0000","last_undegraded":"2024-09-13T14:31:12.317483+0000","last_fullsized":"2024-09-13T14:31:12.317483+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:11:46.253003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":133,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:13.356073+0000","last_change":"2024-09-13T14:24:55.287275+0000","last_active":"2024-09-13T14:31:13.356073+0000","last_peered":"2024-09-13T14:31:13.356073+0000","last_clean":"2024-09-13T14:31:13.356073+0000","last_became_active":"2024-09-13T14:24:55.286919+0000","last_became_peered":"2024-09-13T14:24:55.286919+0000","last_unstale":"2024-09-13T14:31:13.356073+0000","last_undegraded":"2024-09-13T14:31:13.356073+0000","last_fullsized":"2024-09-13T14:31:13.356073+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:10.790711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":139,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:13.133377+0000","last_change":"2024-09-13T14:26:15.819037+0000","last_active":"2024-09-13T14:31:13.133377+0000","last_peered":"2024-09-13T14:31:13.133377+0000","last_clean":"2024-09-13T14:31:13.133377+0000","last_became_active":"2024-09-13T14:26:15.818721+0000","last_became_peered":"2024-09-13T14:26:15.818721+0000","last_unstale":"2024-09-13T14:31:13.133377+0000","last_undegraded":"2024-09-13T14:31:13.133377+0000","last_fullsized":"2024-09-13T14:31:13.133377+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:44:06.419136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":137,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T14:31:22.173458+0000","last_change":"2024-09-13T14:24:55.287509+0000","last_active":"2024-09-13T14:31:22.173458+0000","last_peered":"2024-09-13T14:31:22.173458+0000","last_clean":"2024-09-13T14:31:22.173458+0000","last_became_active":"2024-09-13T14:24:55.287193+0000","last_became_peered":"2024-09-13T14:24:55.287193+0000","last_unstale":"2024-09-13T14:31:22.173458+0000","last_undegraded":"2024-09-13T14:31:22.173458+0000","last_fullsized":"2024-09-13T14:31:22.173458+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:14:10.603307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":178,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T14:31:22.171899+0000","last_change":"2024-09-13T14:24:55.289055+0000","last_active":"2024-09-13T14:31:22.171899+0000","last_peered":"2024-09-13T14:31:22.171899+0000","last_clean":"2024-09-13T14:31:22.171899+0000","last_became_active":"2024-09-13T14:24:55.288762+0000","last_became_peered":"2024-09-13T14:24:55.288762+0000","last_unstale":"2024-09-13T14:31:22.171899+0000","last_undegraded":"2024-09-13T14:31:22.171899+0000","last_fullsized":"2024-09-13T14:31:22.171899+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:37:26.639877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":193,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T14:31:22.173309+0000","last_change":"2024-09-13T14:26:15.824540+0000","last_active":"2024-09-13T14:31:22.173309+0000","last_peered":"2024-09-13T14:31:22.173309+0000","last_clean":"2024-09-13T14:31:22.173309+0000","last_became_active":"2024-09-13T14:26:15.824118+0000","last_became_peered":"2024-09-13T14:26:15.824118+0000","last_unstale":"2024-09-13T14:31:22.173309+0000","last_undegraded":"2024-09-13T14:31:22.173309+0000","last_fullsized":"2024-09-13T14:31:22.173309+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:56.677401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":139,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:12.317709+0000","last_change":"2024-09-13T14:26:15.823892+0000","last_active":"2024-09-13T14:31:12.317709+0000","last_peered":"2024-09-13T14:31:12.317709+0000","last_clean":"2024-09-13T14:31:12.317709+0000","last_became_active":"2024-09-13T14:26:15.823582+0000","last_became_peered":"2024-09-13T14:26:15.823582+0000","last_unstale":"2024-09-13T14:31:12.317709+0000","last_undegraded":"2024-09-13T14:31:12.317709+0000","last_fullsized":"2024-09-13T14:31:12.317709+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:53:09.918997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":216,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T14:31:22.173533+0000","last_change":"2024-09-13T14:27:12.830642+0000","last_active":"2024-09-13T14:31:22.173533+0000","last_peered":"2024-09-13T14:31:22.173533+0000","last_clean":"2024-09-13T14:31:22.173533+0000","last_became_active":"2024-09-13T14:27:12.830026+0000","last_became_peered":"2024-09-13T14:27:12.830026+0000","last_unstale":"2024-09-13T14:31:22.173533+0000","last_undegraded":"2024-09-13T14:31:22.173533+0000","last_fullsized":"2024-09-13T14:31:22.173533+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:12:31.783552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":180,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T14:31:22.172101+0000","last_change":"2024-09-13T14:24:55.283070+0000","last_active":"2024-09-13T14:31:22.172101+0000","last_peered":"2024-09-13T14:31:22.172101+0000","last_clean":"2024-09-13T14:31:22.172101+0000","last_became_active":"2024-09-13T14:24:55.282591+0000","last_became_peered":"2024-09-13T14:24:55.282591+0000","last_unstale":"2024-09-13T14:31:22.172101+0000","last_undegraded":"2024-09-13T14:31:22.172101+0000","last_fullsized":"2024-09-13T14:31:22.172101+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:35:21.896294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":166,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T14:31:22.173876+0000","last_change":"2024-09-13T14:26:15.823524+0000","last_active":"2024-09-13T14:31:22.173876+0000","last_peered":"2024-09-13T14:31:22.173876+0000","last_clean":"2024-09-13T14:31:22.173876+0000","last_became_active":"2024-09-13T14:26:15.823162+0000","last_became_peered":"2024-09-13T14:26:15.823162+0000","last_unstale":"2024-09-13T14:31:22.173876+0000","last_undegraded":"2024-09-13T14:31:22.173876+0000","last_fullsized":"2024-09-13T14:31:22.173876+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:17:13.664812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":206,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T14:31:22.173547+0000","last_change":"2024-09-13T14:26:15.823703+0000","last_active":"2024-09-13T14:31:22.173547+0000","last_peered":"2024-09-13T14:31:22.173547+0000","last_clean":"2024-09-13T14:31:22.173547+0000","last_became_active":"2024-09-13T14:26:15.823249+0000","last_became_peered":"2024-09-13T14:26:15.823249+0000","last_unstale":"2024-09-13T14:31:22.173547+0000","last_undegraded":"2024-09-13T14:31:22.173547+0000","last_fullsized":"2024-09-13T14:31:22.173547+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:42:40.209150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":163,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T14:31:22.175795+0000","last_change":"2024-09-13T14:27:12.841881+0000","last_active":"2024-09-13T14:31:22.175795+0000","last_peered":"2024-09-13T14:31:22.175795+0000","last_clean":"2024-09-13T14:31:22.175795+0000","last_became_active":"2024-09-13T14:27:12.841247+0000","last_became_peered":"2024-09-13T14:27:12.841247+0000","last_unstale":"2024-09-13T14:31:22.175795+0000","last_undegraded":"2024-09-13T14:31:22.175795+0000","last_fullsized":"2024-09-13T14:31:22.175795+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:59:16.937848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":189,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T14:31:22.174448+0000","last_change":"2024-09-13T14:27:12.838453+0000","last_active":"2024-09-13T14:31:22.174448+0000","last_peered":"2024-09-13T14:31:22.174448+0000","last_clean":"2024-09-13T14:31:22.174448+0000","last_became_active":"2024-09-13T14:27:12.838124+0000","last_became_peered":"2024-09-13T14:27:12.838124+0000","last_unstale":"2024-09-13T14:31:22.174448+0000","last_undegraded":"2024-09-13T14:31:22.174448+0000","last_fullsized":"2024-09-13T14:31:22.174448+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:24.259191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":171,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T14:31:22.171593+0000","last_change":"2024-09-13T14:23:18.035101+0000","last_active":"2024-09-13T14:31:22.171593+0000","last_peered":"2024-09-13T14:31:22.171593+0000","last_clean":"2024-09-13T14:31:22.171593+0000","last_became_active":"2024-09-13T14:23:18.034724+0000","last_became_peered":"2024-09-13T14:23:18.034724+0000","last_unstale":"2024-09-13T14:31:22.171593+0000","last_undegraded":"2024-09-13T14:31:22.171593+0000","last_fullsized":"2024-09-13T14:31:22.171593+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:39.358144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":161,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T14:31:22.175838+0000","last_change":"2024-09-13T14:27:12.834931+0000","last_active":"2024-09-13T14:31:22.175838+0000","last_peered":"2024-09-13T14:31:22.175838+0000","last_clean":"2024-09-13T14:31:22.175838+0000","last_became_active":"2024-09-13T14:27:12.834572+0000","last_became_peered":"2024-09-13T14:27:12.834572+0000","last_unstale":"2024-09-13T14:31:22.175838+0000","last_undegraded":"2024-09-13T14:31:22.175838+0000","last_fullsized":"2024-09-13T14:31:22.175838+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:54:21.155689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":144,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:13.133258+0000","last_change":"2024-09-13T14:26:15.826283+0000","last_active":"2024-09-13T14:31:13.133258+0000","last_peered":"2024-09-13T14:31:13.133258+0000","last_clean":"2024-09-13T14:31:13.133258+0000","last_became_active":"2024-09-13T14:26:15.820774+0000","last_became_peered":"2024-09-13T14:26:15.820774+0000","last_unstale":"2024-09-13T14:31:13.133258+0000","last_undegraded":"2024-09-13T14:31:13.133258+0000","last_fullsized":"2024-09-13T14:31:13.133258+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:28:39.849420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":143,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T14:31:22.175653+0000","last_change":"2024-09-13T14:24:55.285384+0000","last_active":"2024-09-13T14:31:22.175653+0000","last_peered":"2024-09-13T14:31:22.175653+0000","last_clean":"2024-09-13T14:31:22.175653+0000","last_became_active":"2024-09-13T14:24:55.285081+0000","last_became_peered":"2024-09-13T14:24:55.285081+0000","last_unstale":"2024-09-13T14:31:22.175653+0000","last_undegraded":"2024-09-13T14:31:22.175653+0000","last_fullsized":"2024-09-13T14:31:22.175653+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:47:07.946120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":116,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:13.133351+0000","last_change":"2024-09-13T14:11:29.240351+0000","last_active":"2024-09-13T14:31:13.133351+0000","last_peered":"2024-09-13T14:31:13.133351+0000","last_clean":"2024-09-13T14:31:13.133351+0000","last_became_active":"2024-09-13T14:11:29.240060+0000","last_became_peered":"2024-09-13T14:11:29.240060+0000","last_unstale":"2024-09-13T14:31:13.133351+0000","last_undegraded":"2024-09-13T14:31:13.133351+0000","last_fullsized":"2024-09-13T14:31:13.133351+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:34:36.126368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":212,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T14:31:22.174081+0000","last_change":"2024-09-13T14:27:12.839943+0000","last_active":"2024-09-13T14:31:22.174081+0000","last_peered":"2024-09-13T14:31:22.174081+0000","last_clean":"2024-09-13T14:31:22.174081+0000","last_became_active":"2024-09-13T14:27:12.839534+0000","last_became_peered":"2024-09-13T14:27:12.839534+0000","last_unstale":"2024-09-13T14:31:22.174081+0000","last_undegraded":"2024-09-13T14:31:22.174081+0000","last_fullsized":"2024-09-13T14:31:22.174081+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:34:56.677019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":201,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T14:31:22.174076+0000","last_change":"2024-09-13T14:27:12.836567+0000","last_active":"2024-09-13T14:31:22.174076+0000","last_peered":"2024-09-13T14:31:22.174076+0000","last_clean":"2024-09-13T14:31:22.174076+0000","last_became_active":"2024-09-13T14:27:12.836271+0000","last_became_peered":"2024-09-13T14:27:12.836271+0000","last_unstale":"2024-09-13T14:31:22.174076+0000","last_undegraded":"2024-09-13T14:31:22.174076+0000","last_fullsized":"2024-09-13T14:31:22.174076+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:25:32.910438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":137,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-13T14:31:11.962354+0000","last_change":"2024-09-13T14:26:15.824075+0000","last_active":"2024-09-13T14:31:11.962354+0000","last_peered":"2024-09-13T14:31:11.962354+0000","last_clean":"2024-09-13T14:31:11.962354+0000","last_became_active":"2024-09-13T14:26:15.823762+0000","last_became_peered":"2024-09-13T14:26:15.823762+0000","last_unstale":"2024-09-13T14:31:11.962354+0000","last_undegraded":"2024-09-13T14:31:11.962354+0000","last_fullsized":"2024-09-13T14:31:11.962354+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:26:51.088388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":195,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-13T14:31:22.171864+0000","last_change":"2024-09-13T14:27:12.844201+0000","last_active":"2024-09-13T14:31:22.171864+0000","last_peered":"2024-09-13T14:31:22.171864+0000","last_clean":"2024-09-13T14:31:22.171864+0000","last_became_active":"2024-09-13T14:27:12.834075+0000","last_became_peered":"2024-09-13T14:27:12.834075+0000","last_unstale":"2024-09-13T14:31:22.171864+0000","last_undegraded":"2024-09-13T14:31:22.171864+0000","last_fullsized":"2024-09-13T14:31:22.171864+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:06.657030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":1930297344,"num_objects":29454,"num_object_clones":0,"num_object_copies":88362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29454,"num_write_kb":1885056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5791285248,"data_stored":5791285248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29454,"ondisk_log_size":29454,"up":48,"acting":48,"num_store_stats":6},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":616,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":17278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18272,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16793600,"data_stored":16777292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234251,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1158044,"kb_used_data":982548,"kb_used_omap":1,"kb_used_meta":175486,"kb_avail":92587108,"statfs":{"total":95995035648,"available":94809198592,"internally_reserved":0,"allocated":1006129152,"data_stored":1005297499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":179698123},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495887,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1460188,"kb_used_data":1324820,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":92284964,"statfs":{"total":95995035648,"available":94499803136,"internally_reserved":0,"allocated":1356615680,"data_stored":1355784574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":138541511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757524,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1109640,"kb_used_data":839808,"kb_used_omap":1,"kb_used_meta":269822,"kb_avail":92635512,"statfs":{"total":95995035648,"available":94858764288,"internally_reserved":0,"allocated":859963392,"data_stored":859147374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":276298182},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019160,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":905428,"kb_used_data":631500,"kb_used_omap":1,"kb_used_meta":273918,"kb_avail":92839724,"statfs":{"total":95995035648,"available":95067877376,"internally_reserved":0,"allocated":646656000,"data_stored":645831983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":280492486},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435344,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1056456,"kb_used_data":970620,"kb_used_omap":5,"kb_used_meta":85818,"kb_avail":92688696,"statfs":{"total":95995035648,"available":94913224704,"internally_reserved":0,"allocated":993914880,"data_stored":993102720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":87877758},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":446676598871,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":425201762397,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726925924,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1024912,"kb_used_data":704456,"kb_used_omap":7,"kb_used_meta":320440,"kb_avail":92720240,"statfs":{"total":95995035648,"available":94945525760,"internally_reserved":0,"allocated":721362944,"data_stored":720539573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":328130856},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":751894528,"data_stored":751894528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1098842112,"data_stored":1098842112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":656146432,"data_stored":656146432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":895352832,"data_stored":895352832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1364393984,"data_stored":1364393984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1024655360,"data_stored":1024655360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:31:29.334 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pgp_num 1 2024-09-13T14:31:29.367 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:29.383 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:31:29.416 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:29.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:29.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:29.518 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:29.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:29.560 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:29.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:29.661 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:29.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:29 smithi102 ceph-mon[110183]: from='client.54727 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:31:29.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:29 smithi102 ceph-mon[115269]: from='client.54727 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:31:29.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:29.700 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:29.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:29.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:29 smithi195 ceph-mon[104262]: from='client.54727 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:31:29.802 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:29.836 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:29.836 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:31:29.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:29.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:29.895 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 15 16 36688 36672 152.786 117.188 0.00180182 0.0065151 2024-09-13T14:31:29.938 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:29.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:29.979 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:29.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:30.080 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:30.108 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:30.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:30.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:30.210 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:30.239 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:30.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:30.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:30.341 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:31:30.381 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:30.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:30.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:31:30.483 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:31:30.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:30.525 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:30.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:31:30.616 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:30 smithi102 ceph-mon[110183]: pgmap v500: 89 pgs: 89 active+clean; 1.9 GiB data, 6.9 GiB used, 530 GiB / 536 GiB avail; 146 MiB/s wr, 2.34k op/s 2024-09-13T14:31:30.617 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:30 smithi102 ceph-mon[115269]: pgmap v500: 89 pgs: 89 active+clean; 1.9 GiB data, 6.9 GiB used, 530 GiB / 536 GiB avail; 146 MiB/s wr, 2.34k op/s 2024-09-13T14:31:30.625 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:30.653 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:30.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:30.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:30.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:30 smithi195 ceph-mon[104262]: pgmap v500: 89 pgs: 89 active+clean; 1.9 GiB data, 6.9 GiB used, 530 GiB / 536 GiB avail; 146 MiB/s wr, 2.34k op/s 2024-09-13T14:31:30.755 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:30.776 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:30.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:30.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:30.878 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:30.895 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 16 16 38506 38490 150.338 113.625 0.0129023 0.00664545 2024-09-13T14:31:30.914 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:30.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:30.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:31.016 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:31.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:31.055 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:31.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:31.156 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:31:31.176 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:31.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:31.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:31:31.278 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:31.305 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:31.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:31.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:31.407 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:31.423 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:31:31.429 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:31:31.441 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:31.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:31.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:31.543 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:31.568 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:31.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:31.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:31.670 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:31.789 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:31.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:31.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:31.891 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:31:31.895 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 17 16 40296 40280 148.075 111.875 0.00986538 0.00674849 2024-09-13T14:31:31.927 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:31.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:31.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:31:32.029 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:32.065 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:32.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:32.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:32.167 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:31:32.205 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:32.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:32.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:31:32.307 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:32.345 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:32.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:32.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:32.447 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:32.480 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:32.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:32.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:32.582 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:31:32.588 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:32 smithi195 ceph-mon[104262]: pgmap v501: 89 pgs: 89 active+clean; 1.9 GiB data, 6.9 GiB used, 530 GiB / 536 GiB avail; 146 MiB/s wr, 2.34k op/s 2024-09-13T14:31:32.621 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:32.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:32.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:31:32.723 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:32.749 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:32 smithi102 ceph-mon[110183]: pgmap v501: 89 pgs: 89 active+clean; 1.9 GiB data, 6.9 GiB used, 530 GiB / 536 GiB avail; 146 MiB/s wr, 2.34k op/s 2024-09-13T14:31:32.749 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:32 smithi102 ceph-mon[115269]: pgmap v501: 89 pgs: 89 active+clean; 1.9 GiB data, 6.9 GiB used, 530 GiB / 536 GiB avail; 146 MiB/s wr, 2.34k op/s 2024-09-13T14:31:32.752 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:32.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:32.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:32.763 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:31:32.763 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-13T14:31:32.763 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.763 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35037.0:41789 6.6 e165/161 6:69dcbf22:::benchmark_data_smithi102_194590_object41788:head v 165'2652, mlcod=165'2650)", 2024-09-13T14:31:32.763 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:31:32.752109+0000", 2024-09-13T14:31:32.763 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.0077381630000000002, 2024-09-13T14:31:32.763 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.007768924, 2024-09-13T14:31:32.763 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:31:32.763 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T14:31:32.764 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:31:32.764 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.764 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:31:32.764 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.752109+0000", 2024-09-13T14:31:32.764 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:32.764 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.764 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.764 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:31:32.764 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.752109+0000", 2024-09-13T14:31:32.764 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:32.764 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.764 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.764 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:31:32.765 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.752108+0000", 2024-09-13T14:31:32.765 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:31:32.765 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.765 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.765 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:31:32.765 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.752144+0000", 2024-09-13T14:31:32.765 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6285999999999998e-05 2024-09-13T14:31:32.765 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.765 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.765 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:31:32.765 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.752156+0000", 2024-09-13T14:31:32.765 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1508000000000001e-05 2024-09-13T14:31:32.765 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.766 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.766 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:31:32.766 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.752163+0000", 2024-09-13T14:31:32.766 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.452e-06 2024-09-13T14:31:32.766 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.766 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.766 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:32.766 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.752322+0000", 2024-09-13T14:31:32.766 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015850700000000001 2024-09-13T14:31:32.766 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.766 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.766 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:31:32.767 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.752340+0000", 2024-09-13T14:31:32.767 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8760999999999999e-05 2024-09-13T14:31:32.767 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:32.767 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:31:32.767 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:32.767 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.767 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.767 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:41793 6.3 6:c29a0633:::benchmark_data_smithi102_194590_object41792:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-09-13T14:31:32.767 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:31:32.753377+0000", 2024-09-13T14:31:32.767 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.0064700599999999997, 2024-09-13T14:31:32.767 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0065852580000000001, 2024-09-13T14:31:32.767 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:31:32.767 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:31:32.768 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:31:32.768 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:31:32.768 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:31:32.768 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 41793 2024-09-13T14:31:32.768 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.768 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:31:32.768 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.768 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:31:32.768 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.753377+0000", 2024-09-13T14:31:32.768 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:32.769 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.769 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.769 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:31:32.769 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.753377+0000", 2024-09-13T14:31:32.769 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:32.769 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.769 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.769 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:31:32.770 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.753375+0000", 2024-09-13T14:31:32.770 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:31:32.770 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.770 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.770 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:31:32.770 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.753437+0000", 2024-09-13T14:31:32.770 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1824999999999998e-05 2024-09-13T14:31:32.770 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.770 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.770 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:31:32.770 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.753462+0000", 2024-09-13T14:31:32.770 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5041e-05 2024-09-13T14:31:32.770 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.771 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.771 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:31:32.771 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.753479+0000", 2024-09-13T14:31:32.771 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7261e-05 2024-09-13T14:31:32.771 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.771 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.771 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:32.771 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.753932+0000", 2024-09-13T14:31:32.771 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00045302800000000001 2024-09-13T14:31:32.771 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.771 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.771 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:31:32.771 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.754236+0000", 2024-09-13T14:31:32.771 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000303827 2024-09-13T14:31:32.772 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.772 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.772 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,6", 2024-09-13T14:31:32.772 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.754365+0000", 2024-09-13T14:31:32.772 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012888600000000001 2024-09-13T14:31:32.772 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.772 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.772 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:32.772 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.758021+0000", 2024-09-13T14:31:32.772 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0036558340000000002 2024-09-13T14:31:32.773 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:32.773 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:31:32.773 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:32.773 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.773 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.773 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:41794 6.5 6:ae853f79:::benchmark_data_smithi102_194590_object41793:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-09-13T14:31:32.773 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:31:32.753494+0000", 2024-09-13T14:31:32.773 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.0063533080000000002, 2024-09-13T14:31:32.773 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0065673270000000004, 2024-09-13T14:31:32.773 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:31:32.773 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:31:32.773 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:31:32.773 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:31:32.774 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:31:32.774 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 41794 2024-09-13T14:31:32.774 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.774 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:31:32.774 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.774 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:31:32.774 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.753494+0000", 2024-09-13T14:31:32.774 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:32.774 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.774 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.774 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:31:32.774 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.753494+0000", 2024-09-13T14:31:32.774 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:32.774 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.775 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.775 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:31:32.775 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.753493+0000", 2024-09-13T14:31:32.775 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:31:32.775 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.775 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.775 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:31:32.775 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.753544+0000", 2024-09-13T14:31:32.775 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1094000000000002e-05 2024-09-13T14:31:32.775 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.775 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.775 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:31:32.775 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.753553+0000", 2024-09-13T14:31:32.775 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.6209999999999993e-06 2024-09-13T14:31:32.776 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.776 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.776 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:31:32.776 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.753559+0000", 2024-09-13T14:31:32.776 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0270000000000001e-06 2024-09-13T14:31:32.776 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.776 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.776 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:32.776 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.753603+0000", 2024-09-13T14:31:32.776 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.46e-05 2024-09-13T14:31:32.776 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.776 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.776 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:31:32.777 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.753728+0000", 2024-09-13T14:31:32.777 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012465800000000001 2024-09-13T14:31:32.777 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.777 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.777 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:31:32.777 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.753864+0000", 2024-09-13T14:31:32.777 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000135971 2024-09-13T14:31:32.777 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.777 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.777 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:32.777 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.757519+0000", 2024-09-13T14:31:32.777 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0036553520000000002 2024-09-13T14:31:32.778 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:32.778 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:31:32.778 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:32.778 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.778 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.778 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:41795 6.8 6:14c183b5:::benchmark_data_smithi102_194590_object41794:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-09-13T14:31:32.778 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:31:32.753575+0000", 2024-09-13T14:31:32.778 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.0062719209999999997, 2024-09-13T14:31:32.778 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0065689809999999998, 2024-09-13T14:31:32.778 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:31:32.778 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:31:32.778 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:31:32.778 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:31:32.778 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:31:32.779 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 41795 2024-09-13T14:31:32.779 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.779 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:31:32.779 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.779 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:31:32.779 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.753575+0000", 2024-09-13T14:31:32.779 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:32.779 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.779 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.779 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:31:32.779 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.753575+0000", 2024-09-13T14:31:32.779 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:32.779 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.779 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.780 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:31:32.780 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.753575+0000", 2024-09-13T14:31:32.780 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:31:32.780 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.780 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.780 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:31:32.780 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.753598+0000", 2024-09-13T14:31:32.780 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3036000000000001e-05 2024-09-13T14:31:32.780 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.780 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.780 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:31:32.780 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.753605+0000", 2024-09-13T14:31:32.780 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1389999999999998e-06 2024-09-13T14:31:32.781 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.781 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.781 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:31:32.781 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.753610+0000", 2024-09-13T14:31:32.781 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7350000000000001e-06 2024-09-13T14:31:32.781 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.781 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.781 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:32.781 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.753701+0000", 2024-09-13T14:31:32.781 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.0223000000000007e-05 2024-09-13T14:31:32.781 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.781 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.781 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:31:32.782 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.753825+0000", 2024-09-13T14:31:32.782 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012458 2024-09-13T14:31:32.782 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.782 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.782 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:31:32.782 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.753953+0000", 2024-09-13T14:31:32.782 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000127453 2024-09-13T14:31:32.782 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.782 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.782 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:32.782 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.758230+0000", 2024-09-13T14:31:32.782 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0042770439999999998 2024-09-13T14:31:32.782 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.782 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.783 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:31:32.783 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.759095+0000", 2024-09-13T14:31:32.783 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00086568599999999995 2024-09-13T14:31:32.783 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.783 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.783 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:32.783 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.760095+0000", 2024-09-13T14:31:32.783 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00099983400000000001 2024-09-13T14:31:32.783 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.783 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.783 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:31:32.783 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.760141+0000", 2024-09-13T14:31:32.783 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5982999999999997e-05 2024-09-13T14:31:32.784 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:32.784 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:31:32.784 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:32.784 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.784 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.784 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35037.0:41797 6.e e165/161 6:7933c730:::benchmark_data_smithi102_194590_object41796:head v 165'2628, mlcod=165'2626)", 2024-09-13T14:31:32.784 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:31:32.757358+0000", 2024-09-13T14:31:32.784 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.0024893129999999999, 2024-09-13T14:31:32.784 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0028824390000000001, 2024-09-13T14:31:32.784 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:31:32.784 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T14:31:32.784 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:31:32.785 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.785 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:31:32.785 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.757358+0000", 2024-09-13T14:31:32.785 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:32.785 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.785 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.785 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:31:32.785 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.757358+0000", 2024-09-13T14:31:32.785 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:32.785 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.785 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.785 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:31:32.786 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.757357+0000", 2024-09-13T14:31:32.786 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:31:32.786 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.786 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.786 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:31:32.786 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.757382+0000", 2024-09-13T14:31:32.786 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5037000000000001e-05 2024-09-13T14:31:32.786 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.786 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.786 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:31:32.786 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.757390+0000", 2024-09-13T14:31:32.786 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0839999999999992e-06 2024-09-13T14:31:32.786 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.786 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.787 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:31:32.787 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.757395+0000", 2024-09-13T14:31:32.787 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8729999999999998e-06 2024-09-13T14:31:32.787 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.787 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.787 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:32.787 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.757452+0000", 2024-09-13T14:31:32.787 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7655000000000003e-05 2024-09-13T14:31:32.787 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.787 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.787 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:31:32.787 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.757472+0000", 2024-09-13T14:31:32.787 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9986000000000001e-05 2024-09-13T14:31:32.787 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:32.788 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:31:32.788 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:32.788 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.788 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.788 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35037.0:41801 6.6 e165/161 6:68c6077f:::benchmark_data_smithi102_194590_object41800:head v 165'2653, mlcod=165'2650)", 2024-09-13T14:31:32.788 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:31:32.758246+0000", 2024-09-13T14:31:32.788 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.001601356, 2024-09-13T14:31:32.788 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0020531770000000002, 2024-09-13T14:31:32.788 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:31:32.788 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T14:31:32.788 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:31:32.788 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.788 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:31:32.788 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.758246+0000", 2024-09-13T14:31:32.789 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:32.789 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.789 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.789 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:31:32.789 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.758246+0000", 2024-09-13T14:31:32.789 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:32.789 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.789 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.789 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:31:32.789 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.758245+0000", 2024-09-13T14:31:32.789 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:31:32.789 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.789 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.790 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:31:32.790 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.758286+0000", 2024-09-13T14:31:32.790 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1507000000000001e-05 2024-09-13T14:31:32.790 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.790 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.790 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:31:32.790 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.758295+0000", 2024-09-13T14:31:32.790 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.9269999999999994e-06 2024-09-13T14:31:32.790 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.790 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.790 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:31:32.790 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.758301+0000", 2024-09-13T14:31:32.790 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5169999999999999e-06 2024-09-13T14:31:32.790 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.791 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.791 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:32.791 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.758840+0000", 2024-09-13T14:31:32.791 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00053939899999999998 2024-09-13T14:31:32.791 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.791 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.791 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:31:32.791 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.758864+0000", 2024-09-13T14:31:32.791 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4138000000000002e-05 2024-09-13T14:31:32.791 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:32.791 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:31:32.791 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:32.791 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.791 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.791 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35037.0:41802 6.6 e165/161 6:67af98d1:::benchmark_data_smithi102_194590_object41801:head v 165'2654, mlcod=165'2650)", 2024-09-13T14:31:32.792 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:31:32.760065+0000", 2024-09-13T14:31:32.792 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 4294967295.9997821, 2024-09-13T14:31:32.792 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00032195100000000001, 2024-09-13T14:31:32.792 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:31:32.792 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T14:31:32.792 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:31:32.792 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.792 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:31:32.792 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.760065+0000", 2024-09-13T14:31:32.792 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:32.792 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.792 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.792 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:31:32.793 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.760065+0000", 2024-09-13T14:31:32.793 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:32.793 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.793 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.793 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:31:32.793 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.760058+0000", 2024-09-13T14:31:32.793 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999924 2024-09-13T14:31:32.793 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.793 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.793 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:31:32.793 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.760105+0000", 2024-09-13T14:31:32.793 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7627000000000003e-05 2024-09-13T14:31:32.793 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.794 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.794 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:31:32.794 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.760116+0000", 2024-09-13T14:31:32.794 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1116e-05 2024-09-13T14:31:32.794 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.794 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.794 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:31:32.794 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.760123+0000", 2024-09-13T14:31:32.794 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2339999999999999e-06 2024-09-13T14:31:32.794 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.794 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.794 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:32.794 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.760278+0000", 2024-09-13T14:31:32.794 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000155928 2024-09-13T14:31:32.795 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:32.795 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:32.795 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:31:32.795 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:31:32.760300+0000", 2024-09-13T14:31:32.795 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1540999999999999e-05 2024-09-13T14:31:32.795 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:32.795 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:31:32.795 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:32.795 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:32.795 INFO:teuthology.orchestra.run.smithi195.stdout: ], 2024-09-13T14:31:32.795 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 7 2024-09-13T14:31:32.795 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:31:32.854 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:32.891 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:32.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:32.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:32.896 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 18 16 42029 42013 145.864 108.312 0.0190796 0.00685089 2024-09-13T14:31:32.992 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:33.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:33.026 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:33.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:33.127 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:33.155 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:33.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:33.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:33.257 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:33.286 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:33.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:33.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:33.388 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:31:33.423 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:33.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:33.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:31:33.525 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:33.555 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:33.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:33.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:33.557 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:31:33.557 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:31:33.583 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:31:33.583 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:31:33.608 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:31:33.608 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:31:33.632 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:31:33.632 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:31:33.658 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:33.671 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:31:33.672 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:31:33.698 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:33.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:33.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:33.702 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:31:33.702 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:31:33.733 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:31:33.733 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:31:33.752 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T14:31:33.760 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:31:33.761 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:31:33.800 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:31:33.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:33.844 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:33.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:31:33.896 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 19 16 43796 43780 143.999 110.438 0.00418233 0.00693924 2024-09-13T14:31:33.945 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:33.972 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:33.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:33.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:34.074 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:31:34.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:34.127 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:34.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:31:34.130 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:31:34.228 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:31:34.273 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:34.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:34.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:31:34.374 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:31:34.415 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:34.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:34.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:31:34.517 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:31:34.555 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:34.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:34.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:31:34.618 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:34 smithi195 ceph-mon[104262]: pgmap v502: 89 pgs: 89 active+clean; 2.4 GiB data, 8.2 GiB used, 528 GiB / 536 GiB avail; 121 MiB/s wr, 1.93k op/s 2024-09-13T14:31:34.618 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:34 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/861816907' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:31:34.618 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:34 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:31:34.618 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:34 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:31:34.657 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:34.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:34 smithi102 ceph-mon[115269]: pgmap v502: 89 pgs: 89 active+clean; 2.4 GiB data, 8.2 GiB used, 528 GiB / 536 GiB avail; 121 MiB/s wr, 1.93k op/s 2024-09-13T14:31:34.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:34 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/861816907' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:31:34.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:34 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:31:34.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:34 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:31:34.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:34 smithi102 ceph-mon[110183]: pgmap v502: 89 pgs: 89 active+clean; 2.4 GiB data, 8.2 GiB used, 528 GiB / 536 GiB avail; 121 MiB/s wr, 1.93k op/s 2024-09-13T14:31:34.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:34 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/861816907' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:31:34.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:34 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:31:34.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:34 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:31:34.695 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:34.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:34.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:34.797 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:34.834 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:34.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:34.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:34.896 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:2024-09-13T14:31:34.896285+0000 min lat: 0.00118498 max lat: 0.106431 avg lat: 0.00703585 2024-09-13T14:31:34.896 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:31:34.896 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 20 16 45469 45453 142.026 104.562 0.00559582 0.00703585 2024-09-13T14:31:34.937 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:34.968 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:34.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:34.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:35.070 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:35.099 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:35.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:35.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:35.201 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:35.235 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:35.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:35.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:35.337 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:35.366 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:35.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:35.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:35.468 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:35.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:35.511 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:35.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:35.611 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:35.641 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:35.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:35.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:35.743 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:35.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:35 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:31:35.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:35 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:31:35.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:35 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:31:35.764 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:35.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:35.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:35.789 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:35 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:31:35.789 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:35 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:31:35.790 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:35 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:31:35.790 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:35 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:31:35.790 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:35 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:31:35.790 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:35 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:31:35.866 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:35.896 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 21 16 47038 47022 139.932 98.0625 0.00499622 0.00714115 2024-09-13T14:31:35.899 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:35.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:35.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:36.001 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:36.027 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:36.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:36.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:36.129 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:31:36.171 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:36.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:36.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:31:36.273 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:36.306 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:36.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:36.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:36.408 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:36.438 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:36.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:36.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:36.540 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:36.571 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:36.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:36.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:36.603 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:31:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:31:36] "GET /metrics HTTP/1.1" 200 36506 "" "Prometheus/2.51.0" 2024-09-13T14:31:36.673 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:36.695 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:36.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:36.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:36.797 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:36.836 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:36.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:36.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:36.896 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 22 16 48469 48453 137.637 89.4375 0.0108386 0.00726053 2024-09-13T14:31:36.938 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:36.965 INFO:teuthology.orchestra.run.smithi102.stderr:set pool 1 pgp_num to 1 2024-09-13T14:31:36.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:36.991 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:36.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:37.092 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:37.129 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:37.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:37.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:37.151 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:36 smithi195 ceph-mon[104262]: pgmap v503: 89 pgs: 89 active+clean; 2.5 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-09-13T14:31:37.151 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:36 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3082507290' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-13T14:31:37.162 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:36 smithi102 ceph-mon[115269]: pgmap v503: 89 pgs: 89 active+clean; 2.5 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-09-13T14:31:37.162 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:36 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3082507290' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-13T14:31:37.163 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:36 smithi102 ceph-mon[110183]: pgmap v503: 89 pgs: 89 active+clean; 2.5 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-09-13T14:31:37.163 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:36 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3082507290' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-13T14:31:37.231 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:37.265 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:37.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:37.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:37.367 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:37.393 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:37.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:37.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:37.495 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:37.536 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:37.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:37.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:37.605 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:31:37.605 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:31:37.639 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:31:37.684 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:37.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:37.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:31:37.788 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:37.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:37.826 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:37.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:37.827 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:31:37.896 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 23 15 50023 50008 135.878 97.1875 0.0108042 0.00735485 2024-09-13T14:31:37.927 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:37.959 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:37.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:37.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:37.991 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is set 2024-09-13T14:31:38.062 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:38.089 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:38.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:38.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:38.191 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:38.216 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:38.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:38.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:38.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:37 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3082507290' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-13T14:31:38.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:37 smithi195 ceph-mon[104262]: osdmap e166: 8 total, 8 up, 6 in 2024-09-13T14:31:38.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:37 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2312495737' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:31:38.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:37 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:31:38.318 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:38.359 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:38.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:38.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:38.362 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T14:31:38.392 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:37 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3082507290' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-13T14:31:38.392 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:37 smithi102 ceph-mon[110183]: osdmap e166: 8 total, 8 up, 6 in 2024-09-13T14:31:38.392 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:37 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2312495737' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:31:38.393 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:37 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:31:38.393 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:37 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3082507290' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-13T14:31:38.393 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:37 smithi102 ceph-mon[115269]: osdmap e166: 8 total, 8 up, 6 in 2024-09-13T14:31:38.393 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:37 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2312495737' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:31:38.393 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:37 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:31:38.461 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:38.487 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:38.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:38.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:38.589 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:31:38.649 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:38.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:38.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:31:38.751 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:31:38.763 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:31:38.807 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:38.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:38.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:31:38.884 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:31:38.896 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 24 16 51565 51549 134.229 96.3125 0.0101517 0.00744493 2024-09-13T14:31:38.909 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:38.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:38.940 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:38.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:39.040 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:39.069 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:38 smithi102 ceph-mon[110183]: pgmap v505: 89 pgs: 89 active+clean; 2.9 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 110 MiB/s wr, 1.76k op/s 2024-09-13T14:31:39.069 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:38 smithi102 ceph-mon[110183]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:31:39.070 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:38 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:31:39.070 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:38 smithi102 ceph-mon[110183]: osdmap e167: 8 total, 8 up, 6 in 2024-09-13T14:31:39.070 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:38 smithi102 ceph-mon[115269]: pgmap v505: 89 pgs: 89 active+clean; 2.9 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 110 MiB/s wr, 1.76k op/s 2024-09-13T14:31:39.070 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:38 smithi102 ceph-mon[115269]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:31:39.070 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:38 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:31:39.070 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:38 smithi102 ceph-mon[115269]: osdmap e167: 8 total, 8 up, 6 in 2024-09-13T14:31:39.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:39.071 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:39.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:39.172 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:39.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:39.210 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:39.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:39.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:38 smithi195 ceph-mon[104262]: pgmap v505: 89 pgs: 89 active+clean; 2.9 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 110 MiB/s wr, 1.76k op/s 2024-09-13T14:31:39.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:38 smithi195 ceph-mon[104262]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:31:39.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:38 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:31:39.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:38 smithi195 ceph-mon[104262]: osdmap e167: 8 total, 8 up, 6 in 2024-09-13T14:31:39.311 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:39.352 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:39.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:39.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:39.454 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:39.496 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:39.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:39.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:39.598 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:39.639 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:39.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:39.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:39.741 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:39.768 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:39.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:39.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:39.870 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:39.896 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 25 16 53167 53151 132.864 100.125 0.0104192 0.00752139 2024-09-13T14:31:39.897 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:39.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:39.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:39.999 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:40.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:40.039 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:40.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:40.140 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:31:40.179 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:40.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:40.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:31:40.282 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:40.317 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:40.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:40.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:40.419 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:40.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:40.471 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:40.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:40.572 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:40.613 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:40.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:40.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:40.715 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:31:40.737 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:31:40.738 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:31:40.746 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:40.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:40.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:31:40.761 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:31:40.761 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:31:40.787 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:31:40.787 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:31:40.812 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:31:40.812 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:31:40.830 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:31:40.834 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:31:40.834 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:31:40.848 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:31:40.862 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:31:40.863 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:31:40.883 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:40.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:40.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:31:40.887 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:31:40.888 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:31:40.896 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 26 16 54687 54671 131.408 95 0.00839206 0.00760483 2024-09-13T14:31:40.918 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:31:40.918 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:31:40.985 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:41.032 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:41.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:41.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:41.129 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:31:41.129 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-09-13T14:31:41.129 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-09-13T14:31:41.129 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-13T14:31:41.129 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.130 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:88000 5.9 5:903d079f:::benchmark_data_smithi102_155989_object87999:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:31:41.130 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:22.445508+0000", 2024-09-13T14:31:41.130 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 258.67856476200001, 2024-09-13T14:31:41.130 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.14914573, 2024-09-13T14:31:41.130 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:31:41.130 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:31:41.130 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:31:41.130 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:31:41.130 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:31:41.130 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 88000 2024-09-13T14:31:41.130 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.130 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:31:41.130 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.131 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:31:41.131 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.445508+0000", 2024-09-13T14:31:41.131 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:41.131 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.131 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.131 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:31:41.131 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.445508+0000", 2024-09-13T14:31:41.131 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:41.131 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.131 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.131 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:31:41.131 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.445507+0000", 2024-09-13T14:31:41.131 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:31:41.131 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.132 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.132 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:31:41.132 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.445534+0000", 2024-09-13T14:31:41.132 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.6614e-05 2024-09-13T14:31:41.132 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.132 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.132 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:31:41.132 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.445540+0000", 2024-09-13T14:31:41.132 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2620000000000004e-06 2024-09-13T14:31:41.132 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.132 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.132 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:31:41.133 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.445545+0000", 2024-09-13T14:31:41.133 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7890000000000002e-06 2024-09-13T14:31:41.133 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.133 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.133 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:41.133 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.462090+0000", 2024-09-13T14:31:41.133 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016544999000000001 2024-09-13T14:31:41.133 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.133 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.133 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:31:41.133 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.462209+0000", 2024-09-13T14:31:41.133 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000119498 2024-09-13T14:31:41.133 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.133 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.134 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:31:41.134 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.462329+0000", 2024-09-13T14:31:41.134 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011931500000000001 2024-09-13T14:31:41.134 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.134 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.134 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:41.134 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.465622+0000", 2024-09-13T14:31:41.134 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0032937090000000001 2024-09-13T14:31:41.134 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.134 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.135 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:31:41.135 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.465707+0000", 2024-09-13T14:31:41.135 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.4243e-05 2024-09-13T14:31:41.135 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.135 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.135 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:41.135 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.594506+0000", 2024-09-13T14:31:41.135 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.12879969999999999 2024-09-13T14:31:41.135 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.135 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.135 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:31:41.135 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.594571+0000", 2024-09-13T14:31:41.135 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4988999999999997e-05 2024-09-13T14:31:41.136 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.136 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.136 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:31:41.136 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.594653+0000", 2024-09-13T14:31:41.136 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2091999999999999e-05 2024-09-13T14:31:41.136 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:41.136 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:31:41.136 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:41.137 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.137 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.137 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:90240 5.4 5:2cbe6f91:::benchmark_data_smithi102_155989_object90239:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:31:41.137 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:25.218503+0000", 2024-09-13T14:31:41.137 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 255.905569778, 2024-09-13T14:31:41.137 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.142181319, 2024-09-13T14:31:41.137 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:31:41.137 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:31:41.137 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:31:41.138 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:31:41.138 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:31:41.138 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 90240 2024-09-13T14:31:41.138 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.138 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:31:41.138 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.138 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:31:41.138 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.218503+0000", 2024-09-13T14:31:41.138 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:41.138 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.138 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.138 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:31:41.138 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.218503+0000", 2024-09-13T14:31:41.138 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:41.139 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.139 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.139 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:31:41.139 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.218501+0000", 2024-09-13T14:31:41.139 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:31:41.139 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.139 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.139 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:31:41.139 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.218561+0000", 2024-09-13T14:31:41.139 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.9478000000000002e-05 2024-09-13T14:31:41.139 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.139 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.139 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:31:41.140 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.218574+0000", 2024-09-13T14:31:41.140 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3285000000000001e-05 2024-09-13T14:31:41.140 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.140 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.140 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:31:41.140 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.218581+0000", 2024-09-13T14:31:41.140 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.0770000000000002e-06 2024-09-13T14:31:41.140 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.140 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.140 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:41.140 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.244084+0000", 2024-09-13T14:31:41.140 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.025503168999999999 2024-09-13T14:31:41.140 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.141 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.141 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:31:41.141 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.244205+0000", 2024-09-13T14:31:41.141 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000120279 2024-09-13T14:31:41.141 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.141 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.141 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:31:41.141 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.244351+0000", 2024-09-13T14:31:41.141 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000146651 2024-09-13T14:31:41.141 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.141 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.141 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:31:41.141 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.258065+0000", 2024-09-13T14:31:41.141 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013714066 2024-09-13T14:31:41.142 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.142 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.142 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:41.142 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.267523+0000", 2024-09-13T14:31:41.142 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0094575969999999999 2024-09-13T14:31:41.142 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.142 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.142 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:41.142 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.357540+0000", 2024-09-13T14:31:41.142 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.090017224000000007 2024-09-13T14:31:41.142 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.142 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.142 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:31:41.143 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.357594+0000", 2024-09-13T14:31:41.143 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3579999999999999e-05 2024-09-13T14:31:41.143 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.143 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.143 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:31:41.143 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:25.360684+0000", 2024-09-13T14:31:41.143 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0030902210000000002 2024-09-13T14:31:41.143 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:41.143 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:31:41.143 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:41.143 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.143 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.143 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:227896 5.d 5:b81995d9:::benchmark_data_smithi102_155989_object109978:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-13T14:31:41.144 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:33.751073+0000", 2024-09-13T14:31:41.144 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 67.372999539999995, 2024-09-13T14:31:41.144 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0751938379999999, 2024-09-13T14:31:41.144 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:31:41.144 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-13T14:31:41.144 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:31:41.144 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:31:41.144 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:31:41.144 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 227896 2024-09-13T14:31:41.144 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.144 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:31:41.144 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.145 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:31:41.145 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.751073+0000", 2024-09-13T14:31:41.145 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:41.145 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.145 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.145 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:31:41.145 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.751073+0000", 2024-09-13T14:31:41.145 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:41.145 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.145 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.145 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:31:41.145 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.751071+0000", 2024-09-13T14:31:41.145 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:31:41.146 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.146 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.146 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:31:41.146 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.751077+0000", 2024-09-13T14:31:41.146 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.818e-06 2024-09-13T14:31:41.146 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.146 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.146 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:31:41.146 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.751089+0000", 2024-09-13T14:31:41.146 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1321000000000001e-05 2024-09-13T14:31:41.146 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.146 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.146 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:31:41.146 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.751095+0000", 2024-09-13T14:31:41.147 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0599999999999996e-06 2024-09-13T14:31:41.147 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.147 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.147 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:41.147 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.758089+0000", 2024-09-13T14:31:41.147 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0069944409999999997 2024-09-13T14:31:41.147 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.147 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.147 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:31:41.147 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.758217+0000", 2024-09-13T14:31:41.147 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012764699999999999 2024-09-13T14:31:41.147 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.147 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.147 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:31:41.148 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.758352+0000", 2024-09-13T14:31:41.148 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013484 2024-09-13T14:31:41.148 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.148 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.148 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:31:41.148 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.762219+0000", 2024-09-13T14:31:41.148 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0038677770000000002 2024-09-13T14:31:41.148 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.148 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.148 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:41.148 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.762712+0000", 2024-09-13T14:31:41.148 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00049232300000000002 2024-09-13T14:31:41.148 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.149 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.149 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:41.149 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.791924+0000", 2024-09-13T14:31:41.149 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.029212175999999999 2024-09-13T14:31:41.149 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.149 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.149 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:31:41.149 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.791937+0000", 2024-09-13T14:31:41.149 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3121e-05 2024-09-13T14:31:41.149 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.149 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.149 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:41.149 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.826186+0000", 2024-09-13T14:31:41.150 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0342485370000001 2024-09-13T14:31:41.150 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.150 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.150 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:31:41.150 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.826267+0000", 2024-09-13T14:31:41.150 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1280999999999996e-05 2024-09-13T14:31:41.150 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:41.150 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:31:41.150 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:41.150 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.150 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.150 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:227905 5.d 5:b4feef4a:::benchmark_data_smithi102_155989_object109987:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-13T14:31:41.150 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:33.762563+0000", 2024-09-13T14:31:41.151 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 67.361509354999995, 2024-09-13T14:31:41.151 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.064023068, 2024-09-13T14:31:41.151 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:31:41.151 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-13T14:31:41.151 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:31:41.151 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:31:41.151 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:31:41.151 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 227905 2024-09-13T14:31:41.151 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.151 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:31:41.151 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.151 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:31:41.151 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.762563+0000", 2024-09-13T14:31:41.151 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:41.152 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.152 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.152 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:31:41.152 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.762563+0000", 2024-09-13T14:31:41.152 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:41.152 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.152 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.152 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:31:41.152 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.762562+0000", 2024-09-13T14:31:41.152 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:31:41.152 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.152 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.152 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:31:41.153 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.762569+0000", 2024-09-13T14:31:41.153 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5870000000000004e-06 2024-09-13T14:31:41.153 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.153 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.153 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:31:41.153 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.762582+0000", 2024-09-13T14:31:41.153 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2828e-05 2024-09-13T14:31:41.153 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.153 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.153 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:31:41.153 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.762590+0000", 2024-09-13T14:31:41.153 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7989999999999994e-06 2024-09-13T14:31:41.153 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.154 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.154 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:41.154 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.762762+0000", 2024-09-13T14:31:41.154 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00017276699999999999 2024-09-13T14:31:41.154 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.154 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.154 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:31:41.154 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.762887+0000", 2024-09-13T14:31:41.154 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012444299999999999 2024-09-13T14:31:41.154 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.154 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.154 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:31:41.154 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.763018+0000", 2024-09-13T14:31:41.155 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013129999999999999 2024-09-13T14:31:41.155 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.155 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.155 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:41.155 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.766939+0000", 2024-09-13T14:31:41.155 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0039210190000000004 2024-09-13T14:31:41.155 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.155 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.155 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:31:41.155 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.775873+0000", 2024-09-13T14:31:41.155 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0089337670000000004 2024-09-13T14:31:41.155 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.155 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.155 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:41.156 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.791965+0000", 2024-09-13T14:31:41.156 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016091520000000002 2024-09-13T14:31:41.156 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.156 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.156 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:31:41.156 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.791973+0000", 2024-09-13T14:31:41.156 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0539999999999995e-06 2024-09-13T14:31:41.156 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.156 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.156 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:41.156 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.826539+0000", 2024-09-13T14:31:41.156 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0345661960000001 2024-09-13T14:31:41.156 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.157 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.157 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:31:41.157 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.826586+0000", 2024-09-13T14:31:41.157 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7432999999999998e-05 2024-09-13T14:31:41.157 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:41.157 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:31:41.157 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:41.157 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.157 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.157 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:227906 5.d 5:b54f7af9:::benchmark_data_smithi102_155989_object109988:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-13T14:31:41.157 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:33.764422+0000", 2024-09-13T14:31:41.157 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 67.359650723000001, 2024-09-13T14:31:41.157 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0622330689999999, 2024-09-13T14:31:41.158 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:31:41.158 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-13T14:31:41.158 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:31:41.158 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:31:41.158 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:31:41.158 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 227906 2024-09-13T14:31:41.158 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.158 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:31:41.158 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.158 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:31:41.158 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.764422+0000", 2024-09-13T14:31:41.158 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:41.158 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.159 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.159 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:31:41.159 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.764422+0000", 2024-09-13T14:31:41.159 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:41.160 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:41.160 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.160 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.160 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:31:41.160 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.764420+0000", 2024-09-13T14:31:41.160 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:31:41.160 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.161 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.161 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:31:41.161 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.764429+0000", 2024-09-13T14:31:41.161 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7690000000000002e-06 2024-09-13T14:31:41.161 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.161 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.161 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:31:41.161 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.764446+0000", 2024-09-13T14:31:41.161 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7343e-05 2024-09-13T14:31:41.161 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.161 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.161 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:31:41.161 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.764454+0000", 2024-09-13T14:31:41.162 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3930000000000002e-06 2024-09-13T14:31:41.162 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.162 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.162 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:41.162 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.766963+0000", 2024-09-13T14:31:41.162 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0025088789999999999 2024-09-13T14:31:41.162 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.162 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.162 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:31:41.162 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.767080+0000", 2024-09-13T14:31:41.162 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011749 2024-09-13T14:31:41.162 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.163 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.163 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:31:41.163 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.767200+0000", 2024-09-13T14:31:41.163 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000120247 2024-09-13T14:31:41.163 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.163 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.163 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:41.163 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.771536+0000", 2024-09-13T14:31:41.163 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0043359749999999997 2024-09-13T14:31:41.163 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.163 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.163 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:31:41.163 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.775881+0000", 2024-09-13T14:31:41.164 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0043450520000000003 2024-09-13T14:31:41.164 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.164 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.164 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:41.164 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.792001+0000", 2024-09-13T14:31:41.164 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016119228999999999 2024-09-13T14:31:41.164 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.164 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.164 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:31:41.164 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.792009+0000", 2024-09-13T14:31:41.164 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2320000000000001e-06 2024-09-13T14:31:41.165 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.165 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.165 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:41.165 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.826613+0000", 2024-09-13T14:31:41.165 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.034603977 2024-09-13T14:31:41.165 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.165 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.165 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:31:41.165 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.826655+0000", 2024-09-13T14:31:41.165 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2023000000000001e-05 2024-09-13T14:31:41.165 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:41.165 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:31:41.165 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:41.165 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.166 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.166 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:227907 5.d 5:be3572e2:::benchmark_data_smithi102_155989_object109989:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-13T14:31:41.166 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:33.768928+0000", 2024-09-13T14:31:41.166 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 67.355144535999997, 2024-09-13T14:31:41.166 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.05785475, 2024-09-13T14:31:41.166 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:31:41.166 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-13T14:31:41.166 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:31:41.166 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:31:41.166 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:31:41.166 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 227907 2024-09-13T14:31:41.166 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.166 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:31:41.166 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.167 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:31:41.167 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.768928+0000", 2024-09-13T14:31:41.167 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:41.167 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.167 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.167 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:31:41.167 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.768928+0000", 2024-09-13T14:31:41.167 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:41.167 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.167 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.167 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:31:41.167 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.768926+0000", 2024-09-13T14:31:41.167 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:31:41.168 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.168 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.168 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:31:41.168 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.768934+0000", 2024-09-13T14:31:41.168 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9389999999999997e-06 2024-09-13T14:31:41.168 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.168 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.168 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:31:41.168 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.768946+0000", 2024-09-13T14:31:41.168 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2251e-05 2024-09-13T14:31:41.168 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.168 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.168 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:31:41.168 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.768954+0000", 2024-09-13T14:31:41.169 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8019999999999992e-06 2024-09-13T14:31:41.169 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.169 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.169 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:41.169 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.771557+0000", 2024-09-13T14:31:41.169 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002602942 2024-09-13T14:31:41.169 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.169 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.169 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:31:41.169 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.771676+0000", 2024-09-13T14:31:41.169 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011906200000000001 2024-09-13T14:31:41.169 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.169 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.170 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:31:41.170 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.771793+0000", 2024-09-13T14:31:41.170 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011686800000000001 2024-09-13T14:31:41.170 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.170 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.170 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:41.170 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.782402+0000", 2024-09-13T14:31:41.170 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010609034999999999 2024-09-13T14:31:41.170 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.170 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.170 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:31:41.170 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.782881+0000", 2024-09-13T14:31:41.170 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00047899999999999999 2024-09-13T14:31:41.171 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.171 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.171 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:41.171 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.792131+0000", 2024-09-13T14:31:41.171 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0092503280000000004 2024-09-13T14:31:41.171 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.171 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.171 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:31:41.171 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.792142+0000", 2024-09-13T14:31:41.171 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0261999999999999e-05 2024-09-13T14:31:41.171 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.171 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.171 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:41.171 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.826703+0000", 2024-09-13T14:31:41.172 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0345612500000001 2024-09-13T14:31:41.172 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.172 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.172 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:31:41.172 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.826783+0000", 2024-09-13T14:31:41.172 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9729999999999997e-05 2024-09-13T14:31:41.172 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:41.172 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:31:41.172 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:41.172 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.172 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.172 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:227932 5.d 5:b1de39af:::benchmark_data_smithi102_155989_object110014:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-09-13T14:31:41.172 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:33.813605+0000", 2024-09-13T14:31:41.173 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 67.310467113000001, 2024-09-13T14:31:41.173 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.028329944, 2024-09-13T14:31:41.173 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:31:41.173 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:31:41.173 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:31:41.173 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:31:41.173 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:31:41.173 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 227932 2024-09-13T14:31:41.173 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.173 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:31:41.173 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.173 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:31:41.173 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.813605+0000", 2024-09-13T14:31:41.173 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:41.174 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.174 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.174 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:31:41.174 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.813605+0000", 2024-09-13T14:31:41.174 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:41.174 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.174 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.174 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:31:41.174 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.813604+0000", 2024-09-13T14:31:41.174 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:31:41.174 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.174 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.174 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:31:41.174 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.813610+0000", 2024-09-13T14:31:41.175 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1709999999999999e-06 2024-09-13T14:31:41.175 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.175 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.175 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:31:41.175 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.813621+0000", 2024-09-13T14:31:41.175 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0560000000000001e-05 2024-09-13T14:31:41.175 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.175 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.175 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:31:41.175 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.813629+0000", 2024-09-13T14:31:41.175 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8490000000000006e-06 2024-09-13T14:31:41.176 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.176 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.176 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:41.176 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.813961+0000", 2024-09-13T14:31:41.176 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00033266799999999999 2024-09-13T14:31:41.176 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.176 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.176 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:31:41.176 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.813971+0000", 2024-09-13T14:31:41.176 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4099999999999997e-06 2024-09-13T14:31:41.176 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.176 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.176 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:41.177 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.826810+0000", 2024-09-13T14:31:41.177 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0128397709999999 2024-09-13T14:31:41.177 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.177 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.177 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:31:41.177 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.826931+0000", 2024-09-13T14:31:41.177 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000120252 2024-09-13T14:31:41.177 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.177 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.177 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:31:41.177 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.827056+0000", 2024-09-13T14:31:41.177 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012539199999999999 2024-09-13T14:31:41.177 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.177 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.178 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:41.178 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.841785+0000", 2024-09-13T14:31:41.178 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014728890999999999 2024-09-13T14:31:41.178 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.178 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.178 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:31:41.178 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.841794+0000", 2024-09-13T14:31:41.178 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8740000000000001e-06 2024-09-13T14:31:41.178 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.178 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.178 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:41.178 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.841838+0000", 2024-09-13T14:31:41.178 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4066999999999997e-05 2024-09-13T14:31:41.179 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.179 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.179 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:31:41.179 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.841887+0000", 2024-09-13T14:31:41.179 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8823000000000003e-05 2024-09-13T14:31:41.179 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.179 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.179 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:31:41.179 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.841935+0000", 2024-09-13T14:31:41.179 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8547000000000001e-05 2024-09-13T14:31:41.179 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:41.179 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:31:41.179 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:41.180 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.180 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.180 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:227960 5.d 5:b3b8d790:::benchmark_data_smithi102_155989_object110042:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-09-13T14:31:41.180 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:33.860733+0000", 2024-09-13T14:31:41.180 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 67.263339849000005, 2024-09-13T14:31:41.180 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.98902412699999998, 2024-09-13T14:31:41.180 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:31:41.180 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:31:41.180 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:31:41.180 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:31:41.180 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:31:41.180 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 227960 2024-09-13T14:31:41.180 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.180 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:31:41.181 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.181 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:31:41.181 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.860733+0000", 2024-09-13T14:31:41.181 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:41.181 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.181 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.181 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:31:41.181 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.860733+0000", 2024-09-13T14:31:41.181 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:41.181 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.181 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.181 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:31:41.181 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.860731+0000", 2024-09-13T14:31:41.182 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:31:41.182 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.182 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.182 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:31:41.182 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.860740+0000", 2024-09-13T14:31:41.182 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.2259999999999994e-06 2024-09-13T14:31:41.182 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.182 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.182 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:31:41.182 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.860751+0000", 2024-09-13T14:31:41.182 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0334e-05 2024-09-13T14:31:41.182 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.182 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.182 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:31:41.182 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.860757+0000", 2024-09-13T14:31:41.183 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0990000000000004e-06 2024-09-13T14:31:41.183 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.183 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.183 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:41.183 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.860935+0000", 2024-09-13T14:31:41.183 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000178701 2024-09-13T14:31:41.183 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.183 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.183 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:31:41.183 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.860960+0000", 2024-09-13T14:31:41.183 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4587999999999999e-05 2024-09-13T14:31:41.183 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.183 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.184 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:41.184 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.830632+0000", 2024-09-13T14:31:41.184 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.96967207399999999 2024-09-13T14:31:41.184 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.184 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.184 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:31:41.184 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.830754+0000", 2024-09-13T14:31:41.184 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012228099999999999 2024-09-13T14:31:41.184 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.184 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.184 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:31:41.184 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.830894+0000", 2024-09-13T14:31:41.184 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013926300000000001 2024-09-13T14:31:41.185 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.185 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.185 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:41.185 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.841985+0000", 2024-09-13T14:31:41.185 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011091247 2024-09-13T14:31:41.185 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.185 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.185 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:31:41.185 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.841996+0000", 2024-09-13T14:31:41.185 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0798999999999999e-05 2024-09-13T14:31:41.185 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.185 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.185 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:41.185 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.849677+0000", 2024-09-13T14:31:41.186 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0076816180000000003 2024-09-13T14:31:41.186 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.186 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.186 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:31:41.186 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.849722+0000", 2024-09-13T14:31:41.186 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4997000000000003e-05 2024-09-13T14:31:41.186 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.186 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.186 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:31:41.186 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.849757+0000", 2024-09-13T14:31:41.186 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4422999999999999e-05 2024-09-13T14:31:41.186 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:41.186 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:31:41.186 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:41.187 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.187 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.187 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:227961 5.d 5:ba86b973:::benchmark_data_smithi102_155989_object110043:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-09-13T14:31:41.187 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:33.860768+0000", 2024-09-13T14:31:41.187 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 67.263304073, 2024-09-13T14:31:41.187 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.98916236300000004, 2024-09-13T14:31:41.187 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:31:41.187 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:31:41.187 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:31:41.187 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:31:41.187 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:31:41.187 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 227961 2024-09-13T14:31:41.187 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.187 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:31:41.188 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.188 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:31:41.188 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.860768+0000", 2024-09-13T14:31:41.188 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:41.188 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.188 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.188 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:31:41.188 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.860768+0000", 2024-09-13T14:31:41.188 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:41.188 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.188 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.188 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:31:41.188 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.860768+0000", 2024-09-13T14:31:41.188 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:31:41.189 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.189 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.189 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:31:41.189 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.860770+0000", 2024-09-13T14:31:41.189 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0810000000000001e-06 2024-09-13T14:31:41.189 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.189 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.189 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:31:41.189 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.860775+0000", 2024-09-13T14:31:41.189 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3299999999999998e-06 2024-09-13T14:31:41.189 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.189 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.189 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:31:41.190 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.860780+0000", 2024-09-13T14:31:41.190 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1370000000000004e-06 2024-09-13T14:31:41.190 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.190 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.190 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:41.190 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.861060+0000", 2024-09-13T14:31:41.190 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00028065399999999997 2024-09-13T14:31:41.190 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.190 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.190 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:31:41.190 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.861082+0000", 2024-09-13T14:31:41.190 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1608e-05 2024-09-13T14:31:41.190 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.191 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.191 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:41.191 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.834509+0000", 2024-09-13T14:31:41.191 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.97342739199999995 2024-09-13T14:31:41.191 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.191 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.191 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:31:41.191 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.834663+0000", 2024-09-13T14:31:41.191 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015392699999999999 2024-09-13T14:31:41.191 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.191 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.191 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:31:41.191 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.834789+0000", 2024-09-13T14:31:41.191 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012604700000000001 2024-09-13T14:31:41.192 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.192 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.192 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:31:41.192 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.842000+0000", 2024-09-13T14:31:41.192 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0072104860000000003 2024-09-13T14:31:41.192 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.192 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.192 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:41.192 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.849800+0000", 2024-09-13T14:31:41.192 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0078000250000000004 2024-09-13T14:31:41.192 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.192 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.192 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:41.192 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.849842+0000", 2024-09-13T14:31:41.193 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1857000000000003e-05 2024-09-13T14:31:41.193 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.193 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.193 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:31:41.193 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.849891+0000", 2024-09-13T14:31:41.193 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8897000000000003e-05 2024-09-13T14:31:41.193 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.193 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.193 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:31:41.193 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.849931+0000", 2024-09-13T14:31:41.193 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0268999999999997e-05 2024-09-13T14:31:41.193 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:41.193 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:31:41.193 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:41.194 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.194 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.194 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:227970 5.d 5:be0539d0:::benchmark_data_smithi102_155989_object110052:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-09-13T14:31:41.194 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:33.872593+0000", 2024-09-13T14:31:41.194 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 67.251479426000003, 2024-09-13T14:31:41.194 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.97752432700000003, 2024-09-13T14:31:41.194 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:31:41.194 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:31:41.194 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:31:41.194 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:31:41.194 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:31:41.194 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 227970 2024-09-13T14:31:41.194 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.195 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:31:41.195 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.195 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:31:41.195 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.872593+0000", 2024-09-13T14:31:41.195 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:41.195 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.195 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.195 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:31:41.195 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.872593+0000", 2024-09-13T14:31:41.195 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:41.195 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.195 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.195 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:31:41.195 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.872592+0000", 2024-09-13T14:31:41.196 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:31:41.196 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.196 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.196 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:31:41.196 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.872597+0000", 2024-09-13T14:31:41.196 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4700000000000004e-06 2024-09-13T14:31:41.196 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.196 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.196 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:31:41.196 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.872605+0000", 2024-09-13T14:31:41.196 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8970000000000008e-06 2024-09-13T14:31:41.196 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.196 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.197 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:31:41.197 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.872611+0000", 2024-09-13T14:31:41.197 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5899999999999996e-06 2024-09-13T14:31:41.197 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.197 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.197 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:41.197 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.872668+0000", 2024-09-13T14:31:41.197 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.6808999999999998e-05 2024-09-13T14:31:41.197 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.197 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.197 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:31:41.197 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.872688+0000", 2024-09-13T14:31:41.198 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9898000000000002e-05 2024-09-13T14:31:41.198 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.198 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.198 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:41.198 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.838196+0000", 2024-09-13T14:31:41.198 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.96550749599999997 2024-09-13T14:31:41.198 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.198 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.198 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:31:41.198 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.838308+0000", 2024-09-13T14:31:41.198 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000112339 2024-09-13T14:31:41.198 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.198 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.198 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:31:41.199 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.838416+0000", 2024-09-13T14:31:41.199 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000108495 2024-09-13T14:31:41.199 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.199 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.199 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:31:41.199 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.849806+0000", 2024-09-13T14:31:41.199 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011389644000000001 2024-09-13T14:31:41.199 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.199 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.199 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:41.199 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.849965+0000", 2024-09-13T14:31:41.199 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000159025 2024-09-13T14:31:41.199 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.200 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.200 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:41.200 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.850005+0000", 2024-09-13T14:31:41.200 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9731000000000003e-05 2024-09-13T14:31:41.200 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.200 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.200 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:31:41.200 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.850073+0000", 2024-09-13T14:31:41.200 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8078999999999997e-05 2024-09-13T14:31:41.200 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.200 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.200 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:31:41.200 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.850117+0000", 2024-09-13T14:31:41.201 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.456e-05 2024-09-13T14:31:41.201 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:41.201 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:31:41.201 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:41.201 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.201 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.201 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:227976 5.d 5:baf0dd50:::benchmark_data_smithi102_155989_object110058:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-09-13T14:31:41.201 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:33.882421+0000", 2024-09-13T14:31:41.201 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 67.241651845000007, 2024-09-13T14:31:41.201 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.96790013699999999, 2024-09-13T14:31:41.201 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:31:41.201 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:31:41.201 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:31:41.201 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:31:41.202 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:31:41.202 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 227976 2024-09-13T14:31:41.202 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.202 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:31:41.202 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.202 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:31:41.202 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.882421+0000", 2024-09-13T14:31:41.202 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:41.202 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.202 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.202 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:31:41.203 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.882421+0000", 2024-09-13T14:31:41.203 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:41.203 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.203 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.203 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:31:41.204 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.882419+0000", 2024-09-13T14:31:41.204 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:31:41.204 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.204 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.205 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:31:41.205 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.882426+0000", 2024-09-13T14:31:41.205 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8889999999999996e-06 2024-09-13T14:31:41.205 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.205 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.205 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:31:41.205 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.882440+0000", 2024-09-13T14:31:41.205 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3852000000000001e-05 2024-09-13T14:31:41.205 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.205 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.205 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:31:41.205 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.882447+0000", 2024-09-13T14:31:41.206 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4540000000000001e-06 2024-09-13T14:31:41.206 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.206 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.206 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:41.206 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.882588+0000", 2024-09-13T14:31:41.206 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000140419 2024-09-13T14:31:41.206 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.206 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.206 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:31:41.206 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.882614+0000", 2024-09-13T14:31:41.206 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.6061999999999999e-05 2024-09-13T14:31:41.206 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.206 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.207 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:41.207 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.842047+0000", 2024-09-13T14:31:41.207 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.95943292300000005 2024-09-13T14:31:41.207 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.207 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.207 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:31:41.207 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.842169+0000", 2024-09-13T14:31:41.207 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012222000000000001 2024-09-13T14:31:41.207 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.207 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.207 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:31:41.207 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.842289+0000", 2024-09-13T14:31:41.207 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012015000000000001 2024-09-13T14:31:41.207 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.208 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.208 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:31:41.208 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.849810+0000", 2024-09-13T14:31:41.208 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0075213420000000003 2024-09-13T14:31:41.208 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.208 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.208 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:41.208 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.850170+0000", 2024-09-13T14:31:41.208 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00035967200000000001 2024-09-13T14:31:41.208 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.208 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.208 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:41.208 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.850229+0000", 2024-09-13T14:31:41.209 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8646000000000001e-05 2024-09-13T14:31:41.209 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.209 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.209 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:31:41.209 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.850291+0000", 2024-09-13T14:31:41.209 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1736000000000007e-05 2024-09-13T14:31:41.209 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.209 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.209 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:31:41.209 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.850321+0000", 2024-09-13T14:31:41.209 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0270999999999999e-05 2024-09-13T14:31:41.209 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:41.209 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:31:41.210 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:41.210 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.210 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.210 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:227980 5.d 5:bdcdc726:::benchmark_data_smithi102_155989_object110062:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-09-13T14:31:41.210 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:33.888564+0000", 2024-09-13T14:31:41.210 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 67.235508342000003, 2024-09-13T14:31:41.210 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.98116541899999998, 2024-09-13T14:31:41.210 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:31:41.210 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:31:41.210 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:31:41.210 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:31:41.210 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:31:41.210 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 227980 2024-09-13T14:31:41.210 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.211 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:31:41.211 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.211 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:31:41.211 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.888564+0000", 2024-09-13T14:31:41.211 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:41.211 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.211 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.211 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:31:41.211 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.888564+0000", 2024-09-13T14:31:41.211 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:41.211 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.211 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.211 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:31:41.211 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.888562+0000", 2024-09-13T14:31:41.212 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:31:41.212 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.212 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.212 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:31:41.212 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.888573+0000", 2024-09-13T14:31:41.212 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0964999999999999e-05 2024-09-13T14:31:41.212 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.212 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.212 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:31:41.212 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.888593+0000", 2024-09-13T14:31:41.212 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9604999999999999e-05 2024-09-13T14:31:41.212 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.212 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.213 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:31:41.213 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.888601+0000", 2024-09-13T14:31:41.213 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2239999999999995e-06 2024-09-13T14:31:41.213 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.213 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.213 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:41.213 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.888661+0000", 2024-09-13T14:31:41.213 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0241000000000001e-05 2024-09-13T14:31:41.213 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.213 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.213 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:31:41.213 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.888674+0000", 2024-09-13T14:31:41.213 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2904e-05 2024-09-13T14:31:41.214 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.214 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.214 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:41.214 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.845769+0000", 2024-09-13T14:31:41.214 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.95709459900000005 2024-09-13T14:31:41.214 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.214 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.214 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:31:41.214 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.845868+0000", 2024-09-13T14:31:41.214 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.8649999999999999e-05 2024-09-13T14:31:41.214 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.214 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.214 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:31:41.215 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.845979+0000", 2024-09-13T14:31:41.215 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011089799999999999 2024-09-13T14:31:41.215 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.215 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.215 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:31:41.215 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.849815+0000", 2024-09-13T14:31:41.215 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0038365529999999999 2024-09-13T14:31:41.215 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.215 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.215 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:41.215 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.858462+0000", 2024-09-13T14:31:41.216 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.008646473 2024-09-13T14:31:41.216 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.216 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.216 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:41.216 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.869652+0000", 2024-09-13T14:31:41.216 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011190054 2024-09-13T14:31:41.216 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.216 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.216 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:31:41.216 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.869694+0000", 2024-09-13T14:31:41.216 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2077999999999999e-05 2024-09-13T14:31:41.216 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.216 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.217 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:31:41.217 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.869730+0000", 2024-09-13T14:31:41.217 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5876e-05 2024-09-13T14:31:41.217 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:41.217 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:31:41.217 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:41.217 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.217 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.217 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:227991 5.d 5:bf4fea11:::benchmark_data_smithi102_155989_object110073:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-09-13T14:31:41.217 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:33.906105+0000", 2024-09-13T14:31:41.217 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 67.217967939000005, 2024-09-13T14:31:41.217 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.97247658400000003, 2024-09-13T14:31:41.217 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:31:41.217 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:31:41.218 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:31:41.218 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:31:41.218 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:31:41.218 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 227991 2024-09-13T14:31:41.218 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.218 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:31:41.218 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.218 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:31:41.218 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.906105+0000", 2024-09-13T14:31:41.218 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:41.218 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.218 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.218 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:31:41.218 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.906105+0000", 2024-09-13T14:31:41.219 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:41.219 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.219 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.219 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:31:41.219 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.906102+0000", 2024-09-13T14:31:41.219 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-09-13T14:31:41.219 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.219 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.219 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:31:41.219 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.906112+0000", 2024-09-13T14:31:41.219 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.3899999999999999e-06 2024-09-13T14:31:41.219 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.219 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.220 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:31:41.220 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.906131+0000", 2024-09-13T14:31:41.220 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9721e-05 2024-09-13T14:31:41.220 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.220 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.220 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:31:41.220 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.906140+0000", 2024-09-13T14:31:41.220 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.9190000000000005e-06 2024-09-13T14:31:41.220 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.220 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.220 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:41.220 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.906199+0000", 2024-09-13T14:31:41.221 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8479e-05 2024-09-13T14:31:41.221 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.221 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.221 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:31:41.221 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.906208+0000", 2024-09-13T14:31:41.221 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1670000000000005e-06 2024-09-13T14:31:41.221 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.221 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.221 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:41.221 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.850361+0000", 2024-09-13T14:31:41.221 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.94415305500000002 2024-09-13T14:31:41.221 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.221 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.221 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:31:41.222 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.850472+0000", 2024-09-13T14:31:41.222 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011134899999999999 2024-09-13T14:31:41.222 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.222 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.222 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:31:41.222 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.850602+0000", 2024-09-13T14:31:41.222 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012967799999999999 2024-09-13T14:31:41.222 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.222 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.222 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:41.222 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.858500+0000", 2024-09-13T14:31:41.222 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0078974240000000001 2024-09-13T14:31:41.222 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.223 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.223 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:31:41.223 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.858507+0000", 2024-09-13T14:31:41.223 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4189999999999996e-06 2024-09-13T14:31:41.223 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.223 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.223 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:41.223 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.878498+0000", 2024-09-13T14:31:41.223 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.019990785 2024-09-13T14:31:41.223 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.223 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.223 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:31:41.223 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.878545+0000", 2024-09-13T14:31:41.224 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7302000000000002e-05 2024-09-13T14:31:41.224 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.224 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.224 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:31:41.224 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.878581+0000", 2024-09-13T14:31:41.224 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6074000000000001e-05 2024-09-13T14:31:41.224 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:41.224 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:31:41.224 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:41.224 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.224 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.224 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:228001 5.d 5:b17ce506:::benchmark_data_smithi102_155989_object110083:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-09-13T14:31:41.224 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:33.921096+0000", 2024-09-13T14:31:41.224 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 67.202976573000001, 2024-09-13T14:31:41.225 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.96293915699999999, 2024-09-13T14:31:41.225 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:31:41.225 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:31:41.225 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:31:41.225 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:31:41.225 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:31:41.225 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 228001 2024-09-13T14:31:41.225 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.225 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:31:41.225 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.225 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:31:41.225 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.921096+0000", 2024-09-13T14:31:41.226 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:41.226 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.226 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.226 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:31:41.226 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.921096+0000", 2024-09-13T14:31:41.226 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:41.226 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.226 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.226 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:31:41.226 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.921095+0000", 2024-09-13T14:31:41.226 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:31:41.226 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.226 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.226 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:31:41.227 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.921101+0000", 2024-09-13T14:31:41.227 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4389999999999997e-06 2024-09-13T14:31:41.227 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.227 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.227 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:31:41.227 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.921114+0000", 2024-09-13T14:31:41.227 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3173000000000001e-05 2024-09-13T14:31:41.227 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.227 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.227 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:31:41.227 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.921121+0000", 2024-09-13T14:31:41.227 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3430000000000001e-06 2024-09-13T14:31:41.227 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.227 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.228 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:41.228 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.921250+0000", 2024-09-13T14:31:41.228 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012937199999999999 2024-09-13T14:31:41.228 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.228 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.228 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:31:41.228 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.921281+0000", 2024-09-13T14:31:41.228 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0942000000000003e-05 2024-09-13T14:31:41.228 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.228 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.228 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:41.228 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.854267+0000", 2024-09-13T14:31:41.229 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.93298647899999998 2024-09-13T14:31:41.229 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.229 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.229 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:31:41.229 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.854367+0000", 2024-09-13T14:31:41.229 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.9473000000000001e-05 2024-09-13T14:31:41.229 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.229 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.229 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:31:41.229 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.854478+0000", 2024-09-13T14:31:41.229 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000110863 2024-09-13T14:31:41.229 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.229 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.229 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:31:41.230 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.858513+0000", 2024-09-13T14:31:41.230 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0040354140000000002 2024-09-13T14:31:41.230 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.230 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.230 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:41.230 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.858617+0000", 2024-09-13T14:31:41.230 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000103667 2024-09-13T14:31:41.230 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.230 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.230 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:41.230 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.883942+0000", 2024-09-13T14:31:41.230 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.025325754999999998 2024-09-13T14:31:41.230 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.231 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.231 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:31:41.231 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.883991+0000", 2024-09-13T14:31:41.231 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8579999999999999e-05 2024-09-13T14:31:41.231 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.231 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.231 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:31:41.231 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.884035+0000", 2024-09-13T14:31:41.231 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4027999999999999e-05 2024-09-13T14:31:41.231 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:41.231 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:31:41.231 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:41.231 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.231 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.232 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:228031 5.d 5:b8c11733:::benchmark_data_smithi102_155989_object110113:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-09-13T14:31:41.232 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:33.981537+0000", 2024-09-13T14:31:41.232 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 67.142535457999998, 2024-09-13T14:31:41.232 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.91229017300000004, 2024-09-13T14:31:41.232 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:31:41.232 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:31:41.232 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:31:41.232 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:31:41.232 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:31:41.232 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 228031 2024-09-13T14:31:41.232 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.232 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:31:41.232 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.232 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:31:41.233 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.981537+0000", 2024-09-13T14:31:41.233 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:41.233 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.233 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.233 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:31:41.233 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.981537+0000", 2024-09-13T14:31:41.233 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:41.233 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.233 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.233 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:31:41.233 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.981535+0000", 2024-09-13T14:31:41.233 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-09-13T14:31:41.233 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.234 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.234 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:31:41.234 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.981545+0000", 2024-09-13T14:31:41.234 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0327e-05 2024-09-13T14:31:41.234 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.234 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.234 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:31:41.234 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.981570+0000", 2024-09-13T14:31:41.234 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4828e-05 2024-09-13T14:31:41.234 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.234 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.234 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:31:41.234 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.981580+0000", 2024-09-13T14:31:41.235 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0069e-05 2024-09-13T14:31:41.235 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.235 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.235 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:41.235 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.981646+0000", 2024-09-13T14:31:41.235 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5489999999999998e-05 2024-09-13T14:31:41.235 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.235 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.235 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:31:41.235 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.981660+0000", 2024-09-13T14:31:41.235 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4430000000000001e-05 2024-09-13T14:31:41.235 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.235 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.235 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:41.236 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.858663+0000", 2024-09-13T14:31:41.236 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.87700307899999996 2024-09-13T14:31:41.236 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.236 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.236 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:31:41.236 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.858778+0000", 2024-09-13T14:31:41.236 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000115408 2024-09-13T14:31:41.236 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.236 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.237 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:31:41.237 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.858894+0000", 2024-09-13T14:31:41.237 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000115122 2024-09-13T14:31:41.237 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.237 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.237 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:31:41.237 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.869617+0000", 2024-09-13T14:31:41.237 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010723425999999999 2024-09-13T14:31:41.237 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.237 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.237 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:41.237 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.869768+0000", 2024-09-13T14:31:41.237 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000150598 2024-09-13T14:31:41.238 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.238 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.238 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:41.238 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.893709+0000", 2024-09-13T14:31:41.238 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.023941535999999999 2024-09-13T14:31:41.238 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.238 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.238 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:31:41.238 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.893783+0000", 2024-09-13T14:31:41.238 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3767e-05 2024-09-13T14:31:41.238 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.238 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.238 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:31:41.238 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.893827+0000", 2024-09-13T14:31:41.238 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4342000000000003e-05 2024-09-13T14:31:41.239 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:41.239 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:31:41.239 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:41.239 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.239 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.239 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:228049 5.d 5:b5b2849e:::benchmark_data_smithi102_155989_object110131:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-09-13T14:31:41.239 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:34.031254+0000", 2024-09-13T14:31:41.239 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 67.092818676999997, 2024-09-13T14:31:41.239 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.871372916, 2024-09-13T14:31:41.239 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:31:41.239 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:31:41.239 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:31:41.239 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:31:41.239 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:31:41.240 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 228049 2024-09-13T14:31:41.240 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.240 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:31:41.240 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.240 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:31:41.240 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.031254+0000", 2024-09-13T14:31:41.240 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:41.240 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.240 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.240 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:31:41.240 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.031254+0000", 2024-09-13T14:31:41.240 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:41.240 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.240 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.241 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:31:41.241 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.031253+0000", 2024-09-13T14:31:41.241 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:31:41.241 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.241 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.241 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:31:41.241 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.031258+0000", 2024-09-13T14:31:41.241 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.6559999999999997e-06 2024-09-13T14:31:41.241 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.241 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.241 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:31:41.241 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.031269+0000", 2024-09-13T14:31:41.241 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0147e-05 2024-09-13T14:31:41.242 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.242 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.242 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:31:41.242 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.031275+0000", 2024-09-13T14:31:41.242 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1800000000000001e-06 2024-09-13T14:31:41.242 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.242 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.242 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:41.242 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.031346+0000", 2024-09-13T14:31:41.242 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1205999999999996e-05 2024-09-13T14:31:41.242 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.242 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.242 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:31:41.243 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.031359+0000", 2024-09-13T14:31:41.243 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2853999999999999e-05 2024-09-13T14:31:41.243 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.243 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.243 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:41.243 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.862342+0000", 2024-09-13T14:31:41.243 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.83098279200000003 2024-09-13T14:31:41.243 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.243 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.243 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:31:41.243 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.862593+0000", 2024-09-13T14:31:41.243 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00025126 2024-09-13T14:31:41.243 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.243 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.243 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:31:41.244 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.862696+0000", 2024-09-13T14:31:41.244 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010274500000000001 2024-09-13T14:31:41.244 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.244 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.244 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:31:41.244 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.869738+0000", 2024-09-13T14:31:41.244 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0070420980000000001 2024-09-13T14:31:41.244 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.244 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.244 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:41.244 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.869796+0000", 2024-09-13T14:31:41.244 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8091000000000003e-05 2024-09-13T14:31:41.244 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.245 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.245 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:41.245 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.902543+0000", 2024-09-13T14:31:41.245 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.03274726 2024-09-13T14:31:41.245 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.245 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.245 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:31:41.245 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.902591+0000", 2024-09-13T14:31:41.245 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7809999999999998e-05 2024-09-13T14:31:41.245 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.245 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.245 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:31:41.245 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.902627+0000", 2024-09-13T14:31:41.245 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5815999999999997e-05 2024-09-13T14:31:41.246 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:41.246 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:31:41.246 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:41.246 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.246 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.246 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:228051 5.d 5:be5f511d:::benchmark_data_smithi102_155989_object110133:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-09-13T14:31:41.246 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:34.037246+0000", 2024-09-13T14:31:41.246 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 67.086826486000007, 2024-09-13T14:31:41.246 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.86547465199999996, 2024-09-13T14:31:41.246 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:31:41.246 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:31:41.246 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:31:41.246 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:31:41.247 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:31:41.247 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 228051 2024-09-13T14:31:41.247 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.247 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:31:41.247 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.247 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:31:41.247 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.037246+0000", 2024-09-13T14:31:41.247 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:41.247 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.247 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.247 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:31:41.247 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.037246+0000", 2024-09-13T14:31:41.247 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:41.248 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.248 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.248 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:31:41.248 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.037245+0000", 2024-09-13T14:31:41.248 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:31:41.248 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.248 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.248 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:31:41.248 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.037251+0000", 2024-09-13T14:31:41.248 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1600000000000003e-06 2024-09-13T14:31:41.248 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.248 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.248 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:31:41.248 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.037263+0000", 2024-09-13T14:31:41.249 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1881e-05 2024-09-13T14:31:41.249 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.249 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.249 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:31:41.249 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.037271+0000", 2024-09-13T14:31:41.249 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3680000000000002e-06 2024-09-13T14:31:41.249 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.249 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.249 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:41.249 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.037322+0000", 2024-09-13T14:31:41.249 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1171000000000002e-05 2024-09-13T14:31:41.249 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.249 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.250 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:31:41.250 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.037333+0000", 2024-09-13T14:31:41.250 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0463e-05 2024-09-13T14:31:41.250 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.250 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.250 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:41.250 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.866056+0000", 2024-09-13T14:31:41.250 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.82872323199999998 2024-09-13T14:31:41.250 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.250 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.250 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:31:41.250 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.866158+0000", 2024-09-13T14:31:41.250 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010219500000000001 2024-09-13T14:31:41.250 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.251 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.251 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:31:41.251 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.866273+0000", 2024-09-13T14:31:41.251 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011437700000000001 2024-09-13T14:31:41.251 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.251 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.251 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:31:41.251 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.873381+0000", 2024-09-13T14:31:41.251 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0071080099999999997 2024-09-13T14:31:41.251 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.251 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.251 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:41.251 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.876997+0000", 2024-09-13T14:31:41.251 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0036160010000000002 2024-09-13T14:31:41.252 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.252 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.252 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:41.252 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.902661+0000", 2024-09-13T14:31:41.252 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.025664585 2024-09-13T14:31:41.252 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.252 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.252 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:31:41.252 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.902692+0000", 2024-09-13T14:31:41.252 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.1047000000000003e-05 2024-09-13T14:31:41.252 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.252 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.252 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:31:41.253 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.902721+0000", 2024-09-13T14:31:41.253 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8388999999999999e-05 2024-09-13T14:31:41.253 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:41.253 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:31:41.253 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:41.253 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.253 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.253 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:228072 5.d 5:bb5a3f8e:::benchmark_data_smithi102_155989_object110154:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-09-13T14:31:41.253 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:34.139716+0000", 2024-09-13T14:31:41.253 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 66.984356321999996, 2024-09-13T14:31:41.253 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.76309005699999999, 2024-09-13T14:31:41.253 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:31:41.253 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:31:41.253 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:31:41.254 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:31:41.254 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:31:41.254 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 228072 2024-09-13T14:31:41.254 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.254 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:31:41.254 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.254 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:31:41.254 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.139716+0000", 2024-09-13T14:31:41.254 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:41.254 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.254 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.254 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:31:41.254 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.139716+0000", 2024-09-13T14:31:41.254 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:41.255 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.255 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.255 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:31:41.255 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.139714+0000", 2024-09-13T14:31:41.255 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-09-13T14:31:41.255 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.255 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.255 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:31:41.255 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.139725+0000", 2024-09-13T14:31:41.255 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0658e-05 2024-09-13T14:31:41.255 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.255 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.255 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:31:41.255 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.139745+0000", 2024-09-13T14:31:41.256 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0801999999999998e-05 2024-09-13T14:31:41.256 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.256 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.256 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:31:41.256 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.139755+0000", 2024-09-13T14:31:41.256 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4690000000000003e-06 2024-09-13T14:31:41.256 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.256 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.256 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:41.256 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.139821+0000", 2024-09-13T14:31:41.256 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5639e-05 2024-09-13T14:31:41.256 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.256 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.256 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:31:41.257 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.139834+0000", 2024-09-13T14:31:41.257 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3393999999999999e-05 2024-09-13T14:31:41.257 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.257 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.257 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:41.257 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.869818+0000", 2024-09-13T14:31:41.257 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.72998423099999998 2024-09-13T14:31:41.257 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.257 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.257 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:31:41.257 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.869919+0000", 2024-09-13T14:31:41.257 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000101116 2024-09-13T14:31:41.257 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.258 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.258 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:31:41.258 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.870020+0000", 2024-09-13T14:31:41.258 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000100689 2024-09-13T14:31:41.258 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.258 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.258 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:31:41.258 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.877004+0000", 2024-09-13T14:31:41.258 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0069837550000000003 2024-09-13T14:31:41.258 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.258 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.258 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:41.258 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.884289+0000", 2024-09-13T14:31:41.258 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0072851160000000003 2024-09-13T14:31:41.259 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.259 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.259 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:41.259 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.902746+0000", 2024-09-13T14:31:41.259 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.01845757 2024-09-13T14:31:41.259 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.259 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.259 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:31:41.259 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.902776+0000", 2024-09-13T14:31:41.259 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9428e-05 2024-09-13T14:31:41.259 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.259 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.259 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:31:41.259 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.902806+0000", 2024-09-13T14:31:41.259 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0366e-05 2024-09-13T14:31:41.260 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:41.260 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:31:41.260 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:41.260 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.260 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.260 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:233219 5.d 5:b100a7f5:::benchmark_data_smithi102_155989_object115301:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-13T14:31:41.260 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:41.662936+0000", 2024-09-13T14:31:41.260 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 59.461136273999998, 2024-09-13T14:31:41.260 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.142031041, 2024-09-13T14:31:41.260 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:31:41.260 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:31:41.260 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:31:41.260 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:31:41.261 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:31:41.261 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 233219 2024-09-13T14:31:41.261 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.261 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:31:41.261 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.261 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:31:41.261 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.662936+0000", 2024-09-13T14:31:41.265 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:40 smithi195 ceph-mon[104262]: pgmap v507: 89 pgs: 89 active+clean; 3.0 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 135 MiB/s wr, 2.16k op/s 2024-09-13T14:31:41.265 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:40 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:31:41.265 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:40 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1793452725' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:31:41.266 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:41.266 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.266 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.266 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:31:41.266 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.662936+0000", 2024-09-13T14:31:41.266 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:41.266 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.266 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.266 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:31:41.266 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.662934+0000", 2024-09-13T14:31:41.267 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:31:41.267 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.267 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.267 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:31:41.267 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.662943+0000", 2024-09-13T14:31:41.267 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.0189999999999995e-06 2024-09-13T14:31:41.267 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.267 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.267 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:31:41.267 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.662961+0000", 2024-09-13T14:31:41.267 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7265000000000002e-05 2024-09-13T14:31:41.267 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.268 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.268 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:31:41.268 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.662970+0000", 2024-09-13T14:31:41.268 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.0280000000000006e-06 2024-09-13T14:31:41.268 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.268 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.268 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:41.268 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.663022+0000", 2024-09-13T14:31:41.268 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2701999999999998e-05 2024-09-13T14:31:41.268 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.268 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.268 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:31:41.269 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.663141+0000", 2024-09-13T14:31:41.269 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000118243 2024-09-13T14:31:41.269 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.269 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.269 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:31:41.269 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.663252+0000", 2024-09-13T14:31:41.269 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011093 2024-09-13T14:31:41.269 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.269 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.269 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:41.269 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.667010+0000", 2024-09-13T14:31:41.269 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0037583209999999998 2024-09-13T14:31:41.270 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.270 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.270 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:31:41.270 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.667016+0000", 2024-09-13T14:31:41.270 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2910000000000004e-06 2024-09-13T14:31:41.270 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.270 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.270 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:41.270 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.804888+0000", 2024-09-13T14:31:41.270 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.13787195799999999 2024-09-13T14:31:41.270 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.270 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.271 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:31:41.271 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.804935+0000", 2024-09-13T14:31:41.271 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6746000000000003e-05 2024-09-13T14:31:41.271 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.271 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.271 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:31:41.271 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.804967+0000", 2024-09-13T14:31:41.271 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2450000000000003e-05 2024-09-13T14:31:41.271 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:41.271 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:31:41.271 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:41.271 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.271 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.272 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:233221 5.4 5:2396de17:::benchmark_data_smithi102_155989_object115303:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-13T14:31:41.272 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:41.664506+0000", 2024-09-13T14:31:41.272 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 59.459566742, 2024-09-13T14:31:41.272 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.141504662, 2024-09-13T14:31:41.272 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:31:41.272 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:31:41.272 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:31:41.272 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:31:41.272 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:31:41.272 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 233221 2024-09-13T14:31:41.272 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.272 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:31:41.272 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.273 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:31:41.273 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.664506+0000", 2024-09-13T14:31:41.273 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:41.273 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.273 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.273 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:31:41.273 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.664506+0000", 2024-09-13T14:31:41.273 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:31:41.273 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.273 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.273 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:31:41.273 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.664504+0000", 2024-09-13T14:31:41.273 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:31:41.274 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.274 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.274 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:31:41.274 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.664511+0000", 2024-09-13T14:31:41.274 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1700000000000002e-06 2024-09-13T14:31:41.274 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.274 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.274 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:31:41.274 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.664524+0000", 2024-09-13T14:31:41.274 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3174e-05 2024-09-13T14:31:41.274 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.274 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.274 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:31:41.275 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.664531+0000", 2024-09-13T14:31:41.275 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6480000000000003e-06 2024-09-13T14:31:41.275 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.275 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.275 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:31:41.275 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.664578+0000", 2024-09-13T14:31:41.275 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6622999999999997e-05 2024-09-13T14:31:41.275 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.275 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.275 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:31:41.275 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.664702+0000", 2024-09-13T14:31:41.275 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012382199999999999 2024-09-13T14:31:41.275 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.275 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.276 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:31:41.276 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.664821+0000", 2024-09-13T14:31:41.276 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000119059 2024-09-13T14:31:41.276 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.276 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.276 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:31:41.276 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.671853+0000", 2024-09-13T14:31:41.276 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0070317189999999996 2024-09-13T14:31:41.276 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.276 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.276 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:41.276 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.672091+0000", 2024-09-13T14:31:41.276 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000238268 2024-09-13T14:31:41.277 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.277 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.277 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:31:41.277 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.803878+0000", 2024-09-13T14:31:41.277 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.131786607 2024-09-13T14:31:41.277 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.277 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.277 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:31:41.277 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.803946+0000", 2024-09-13T14:31:41.277 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7978999999999994e-05 2024-09-13T14:31:41.277 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:31:41.277 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:31:41.277 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:31:41.277 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.806010+0000", 2024-09-13T14:31:41.278 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0020648670000000002 2024-09-13T14:31:41.278 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:41.278 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:31:41.278 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:41.278 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:31:41.278 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:31:41.278 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:31:41.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:41.279 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:41.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:41.330 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:40 smithi102 ceph-mon[110183]: pgmap v507: 89 pgs: 89 active+clean; 3.0 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 135 MiB/s wr, 2.16k op/s 2024-09-13T14:31:41.330 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:40 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:31:41.331 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:40 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1793452725' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:31:41.331 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:40 smithi102 ceph-mon[115269]: pgmap v507: 89 pgs: 89 active+clean; 3.0 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 135 MiB/s wr, 2.16k op/s 2024-09-13T14:31:41.331 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:40 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:31:41.331 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:40 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1793452725' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:31:41.381 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:41.422 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:41.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:41.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:41.524 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:41.552 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:41.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:41.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:41.654 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:31:41.688 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:41.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:41.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:31:41.690 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:31:41.790 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:41.827 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:41.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:41.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:41.897 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 27 16 56137 56121 129.897 90.625 0.00962298 0.00769397 2024-09-13T14:31:41.929 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:41.963 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:41.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:41.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:42.065 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:42.093 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:42.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:42.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:42.195 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:42.222 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:42.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:42.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:42.282 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T14:31:42.324 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:42.355 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:42.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:42.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:42.457 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:42.492 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:42.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:42.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:42.597 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:31:42.638 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:42.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:42.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:31:42.736 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 7] out_osds: [1, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:31:42.736 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:31:42.736 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-13T14:31:42.737 INFO:tasks.thrashosds.thrasher:Setting osd 6 primary_affinity to 1.000000 2024-09-13T14:31:42.737 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 6 1 2024-09-13T14:31:42.740 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:42.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:42.812 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:42.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:42.897 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 28 16 57665 57649 128.669 95.5 0.00990588 0.00776695 2024-09-13T14:31:42.913 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:31:42.950 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:42.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:42.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:31:43.016 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:31:43.052 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:43.075 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:31:43.094 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:43.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:43.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:43.195 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:43.217 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:42 smithi102 ceph-mon[110183]: pgmap v508: 89 pgs: 89 active+clean; 3.0 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-09-13T14:31:43.217 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:42 smithi102 ceph-mon[115269]: pgmap v508: 89 pgs: 89 active+clean; 3.0 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-09-13T14:31:43.224 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:43.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:43.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:43.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:42 smithi195 ceph-mon[104262]: pgmap v508: 89 pgs: 89 active+clean; 3.0 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-09-13T14:31:43.326 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:31:43.363 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:43.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:43.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:31:43.465 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:31:43.503 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:43.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:43.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:31:43.605 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:43.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:43.638 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:43.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:43.679 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:31:43.739 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:43.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:43.774 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:43.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:43.875 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:43.897 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 29 16 59159 59143 127.451 93.375 0.0104119 0.00784105 2024-09-13T14:31:43.911 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:43.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:43.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:44.014 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:44.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:44.047 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:44.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:44.149 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:44.191 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:44.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:44.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:44.293 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:44.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:44.326 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:44.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:44.427 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:44.465 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:44.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:44.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:44.567 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:44.593 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:44.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:44.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:44.695 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:44.722 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:44.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:44.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:44.824 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:44.866 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:44.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:44.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:44.897 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 30 16 60644 60628 126.296 92.8125 0.0110972 0.00791333 2024-09-13T14:31:44.968 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:44.996 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:44.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:44.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:45.007 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is unset 2024-09-13T14:31:45.098 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:31:45.132 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:45.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:45.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:31:45.234 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:45.272 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:45.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:45.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:45.337 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:44 smithi102 ceph-mon[110183]: pgmap v509: 89 pgs: 89 active+clean; 3.4 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 108 MiB/s wr, 1.74k op/s 2024-09-13T14:31:45.337 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:44 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3024585904' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:31:45.338 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:45 smithi102 ceph-mon[115269]: pgmap v509: 89 pgs: 89 active+clean; 3.4 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 108 MiB/s wr, 1.74k op/s 2024-09-13T14:31:45.338 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:45 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3024585904' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:31:45.373 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:45.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:45.415 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:45.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:45.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:45 smithi195 ceph-mon[104262]: pgmap v509: 89 pgs: 89 active+clean; 3.4 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 108 MiB/s wr, 1.74k op/s 2024-09-13T14:31:45.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:45 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3024585904' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:31:45.516 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:45.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:45.550 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:45.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:45.651 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:45.687 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:45.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:45.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:45.788 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:45.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:45.831 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:45.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:45.897 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 31 16 62090 62074 125.137 90.375 0.0123751 0.00798543 2024-09-13T14:31:45.932 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:45.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:45.975 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:45.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:46.076 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:46.103 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:46.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:46.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:46.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:46 smithi102 ceph-mon[110183]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:31:46.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:46 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3024585904' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:31:46.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:46 smithi102 ceph-mon[110183]: osdmap e168: 8 total, 8 up, 6 in 2024-09-13T14:31:46.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:46 smithi102 ceph-mon[115269]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:31:46.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:46 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3024585904' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:31:46.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:46 smithi102 ceph-mon[115269]: osdmap e168: 8 total, 8 up, 6 in 2024-09-13T14:31:46.207 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:31:46.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:46.252 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:46.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:31:46.354 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:46.386 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:46.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:46.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:46.488 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:46.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:46 smithi195 ceph-mon[104262]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:31:46.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:46 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3024585904' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:31:46.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:46 smithi195 ceph-mon[104262]: osdmap e168: 8 total, 8 up, 6 in 2024-09-13T14:31:46.526 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:46.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:46.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:46.628 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:46.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:46.659 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:46.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:46.760 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:46.787 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:46.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:46.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:46.811 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:31:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:31:46] "GET /metrics HTTP/1.1" 200 36513 "" "Prometheus/2.51.0" 2024-09-13T14:31:46.819 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:31:46.888 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:46.897 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 32 16 63539 63523 124.057 90.5625 0.0106421 0.00805548 2024-09-13T14:31:46.917 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:46.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:46.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:47.011 INFO:teuthology.orchestra.run.smithi102.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-13T14:31:47.019 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:47.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:47.053 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:47.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:47.110 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:47 smithi102 ceph-mon[110183]: pgmap v511: 89 pgs: 89 active+clean; 3.5 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-13T14:31:47.110 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:47 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/686589123' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:31:47.110 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:47 smithi102 ceph-mon[115269]: pgmap v511: 89 pgs: 89 active+clean; 3.5 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-13T14:31:47.111 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:47 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/686589123' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:31:47.154 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:47.182 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:47.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:47.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:47.283 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:47.310 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:47.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:47.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:47.412 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:31:47.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:47.449 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:47.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:31:47.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:47 smithi195 ceph-mon[104262]: pgmap v511: 89 pgs: 89 active+clean; 3.5 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-13T14:31:47.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:47 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/686589123' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:31:47.550 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:31:47.589 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:47.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:47.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:31:47.685 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:31:47.691 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:47.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:47.757 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:47.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:47.858 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:47.890 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:47.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:47.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:47.897 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 33 16 64954 64938 122.977 88.4375 0.00676158 0.00812646 2024-09-13T14:31:47.992 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:48.056 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:48.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:48.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:48.158 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:48.186 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:48.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:48.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:48.287 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:48.314 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:48.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:48.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:48.416 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:31:48.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:48 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/686589123' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-13T14:31:48.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:48 smithi102 ceph-mon[110183]: osdmap e169: 8 total, 8 up, 6 in 2024-09-13T14:31:48.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:48 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/686589123' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-13T14:31:48.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:48 smithi102 ceph-mon[115269]: osdmap e169: 8 total, 8 up, 6 in 2024-09-13T14:31:48.453 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:48.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:48.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:31:48.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:48 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/686589123' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-13T14:31:48.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:48 smithi195 ceph-mon[104262]: osdmap e169: 8 total, 8 up, 6 in 2024-09-13T14:31:48.555 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:48.595 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:48.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:48.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:48.697 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:48.722 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:48.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:48.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:48.824 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:48.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:48.853 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:48.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:48.897 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 34 16 66267 66251 121.773 82.0625 0.00997473 0.00820692 2024-09-13T14:31:48.955 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:48.982 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:48.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:48.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:49.084 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:49.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:49.119 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:49.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:49.220 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:49.248 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:49.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:49.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:49.350 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:49.375 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:49.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:49.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:49.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:49 smithi102 ceph-mon[110183]: pgmap v513: 89 pgs: 89 active+clean; 3.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 103 MiB/s wr, 1.64k op/s 2024-09-13T14:31:49.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:49 smithi102 ceph-mon[115269]: pgmap v513: 89 pgs: 89 active+clean; 3.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 103 MiB/s wr, 1.64k op/s 2024-09-13T14:31:49.477 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:31:49.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:49 smithi195 ceph-mon[104262]: pgmap v513: 89 pgs: 89 active+clean; 3.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 103 MiB/s wr, 1.64k op/s 2024-09-13T14:31:49.514 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:49.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:49.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:31:49.616 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:31:49.655 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:49.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:49.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:31:49.757 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:49.783 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:49.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:49.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:49.884 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:49.897 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 35 16 67573 67557 120.626 81.625 0.00542027 0.00828495 2024-09-13T14:31:49.922 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:49.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:49.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:50.024 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:50.053 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:50.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:50.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:50.155 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:50.185 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:50.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:50.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:50.287 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:50.319 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:50.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:50.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:50.421 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:31:50.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:50.458 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:50.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:31:50.559 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:50.584 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:50.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:50.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:50.686 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:31:50.721 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:50.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:50.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:31:50.823 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:50.857 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:50.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:50.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:50.897 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 36 16 68890 68874 119.562 82.3125 0.00911738 0.00835961 2024-09-13T14:31:50.959 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:31:50.999 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:51.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:51.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:31:51.101 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:51.129 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:51.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:51.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:51.231 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:51.258 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:51.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:51.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:51.359 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:51.387 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:51.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:51.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:51.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:51 smithi102 ceph-mon[110183]: pgmap v514: 89 pgs: 89 active+clean; 3.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 117 MiB/s wr, 1.88k op/s 2024-09-13T14:31:51.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:51 smithi102 ceph-mon[115269]: pgmap v514: 89 pgs: 89 active+clean; 3.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 117 MiB/s wr, 1.88k op/s 2024-09-13T14:31:51.489 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:31:51.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:51 smithi195 ceph-mon[104262]: pgmap v514: 89 pgs: 89 active+clean; 3.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 117 MiB/s wr, 1.88k op/s 2024-09-13T14:31:51.513 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:51.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:51.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:31:51.615 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:51.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:51.643 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:51.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:51.744 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:31:51.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:51.781 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:51.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:31:51.882 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:51.897 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 37 16 70162 70146 118.479 79.5 0.00501199 0.00843537 2024-09-13T14:31:51.908 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:51.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:51.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:52.010 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:52.032 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:52.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:52.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:52.134 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:52.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:52.217 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:52.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:52.318 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:52.345 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:52.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:52.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:52.447 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:52.467 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:52.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:52.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:52.569 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:31:52.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:52.604 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:52.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:31:52.705 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:52.726 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:52.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:52.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:52.828 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:52.864 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:52.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:52.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:52.897 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 38 16 71281 71265 117.201 69.9375 0.0153374 0.00852686 2024-09-13T14:31:52.965 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:52.986 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:52.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:52.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:52.989 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 7] out_osds: [1, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:31:52.990 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:31:52.990 INFO:tasks.thrashosds.thrasher:Reweighting osd 7 to 0.5593538375785977 2024-09-13T14:31:52.990 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd reweight 7 0.5593538375785977 2024-09-13T14:31:53.088 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:31:53.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:53.128 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:53.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:31:53.229 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:53.255 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:53.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:53.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:53.357 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:53.393 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:53.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:53.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:53.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:53 smithi102 ceph-mon[110183]: pgmap v515: 89 pgs: 89 active+clean; 3.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 72 MiB/s wr, 1.16k op/s 2024-09-13T14:31:53.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:53 smithi102 ceph-mon[115269]: pgmap v515: 89 pgs: 89 active+clean; 3.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 72 MiB/s wr, 1.16k op/s 2024-09-13T14:31:53.495 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:53.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:53 smithi195 ceph-mon[104262]: pgmap v515: 89 pgs: 89 active+clean; 3.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 72 MiB/s wr, 1.16k op/s 2024-09-13T14:31:53.522 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:53.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:53.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:53.623 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:53.650 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:53.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:53.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:53.752 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:53.796 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:53.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:53.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:53.898 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 39 16 72379 72363 115.956 68.625 0.0131461 0.00861906 2024-09-13T14:31:53.898 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:53.929 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:53.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:53.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:54.031 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:54.059 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:54.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:54.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:54.160 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:54.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:54.199 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:54.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:54.300 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:54.327 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:54.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:54.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:54.430 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:54.456 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:54.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:54.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:54.558 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:54.586 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:54.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:54.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:54.688 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:54.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:54.717 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:54.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:54.818 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:31:54.856 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:54.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:54.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:31:54.898 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:2024-09-13T14:31:54.897945+0000 min lat: 0.00118498 max lat: 0.106431 avg lat: 0.00868277 2024-09-13T14:31:54.898 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:31:54.898 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 40 16 73685 73669 115.097 81.625 0.00995229 0.00868277 2024-09-13T14:31:54.958 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:54.988 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:54.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:54.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:55.090 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:55.116 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:55.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:55.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:55.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:55 smithi102 ceph-mon[110183]: pgmap v516: 89 pgs: 89 active+clean; 4.2 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 92 MiB/s wr, 1.48k op/s 2024-09-13T14:31:55.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:55 smithi102 ceph-mon[115269]: pgmap v516: 89 pgs: 89 active+clean; 4.2 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 92 MiB/s wr, 1.48k op/s 2024-09-13T14:31:55.218 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:31:55.252 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:55.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:55.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:31:55.300 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:55 smithi195 ceph-mon[104262]: pgmap v516: 89 pgs: 89 active+clean; 4.2 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 92 MiB/s wr, 1.48k op/s 2024-09-13T14:31:55.354 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:55.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:55.394 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:55.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:55.477 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:31:55.477 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:31:55.478 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:31:55.478 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:31:55.494 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:55.499 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:31:55.499 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:31:55.534 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:31:55.534 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:31:55.537 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:55.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:55.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:55.569 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:31:55.570 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:31:55.610 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:31:55.610 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:31:55.611 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:31:55.611 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:31:55.639 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:55.651 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:31:55.651 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:31:55.666 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:55.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:55.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:55.685 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:31:55.686 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:31:55.768 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:55.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:55.814 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:55.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:55.898 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 41 16 74837 74821 114.046 72 0.0114469 0.00876366 2024-09-13T14:31:55.915 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:55.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:55.954 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:55.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:56.056 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:56.072 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:31:56.073 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:31:56.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:56.106 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:56.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:56.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:56 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3152184388' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:31:56.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:31:56.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:56 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3152184388' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:31:56.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:31:56.207 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:56.234 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:56.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:56.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:56.336 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:56.363 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:56.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:56.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:56.465 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:56.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:56.494 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:56.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:56.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:56 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3152184388' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:31:56.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:31:56.595 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:56.637 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:56.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:56.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:56.736 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:31:56 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:31:56] "GET /metrics HTTP/1.1" 200 36517 "" "Prometheus/2.51.0" 2024-09-13T14:31:56.739 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:56.767 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:56.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:56.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:56.869 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:56.898 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 42 16 75916 75900 112.936 67.4375 0.0104507 0.00884952 2024-09-13T14:31:56.904 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:56.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:56.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:57.006 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:57.033 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:57.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:57.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:57.135 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:57.172 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:57.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:57.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:57.274 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:57.310 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:57.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:57.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:57.313 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:57 smithi195 ceph-mon[104262]: pgmap v517: 89 pgs: 89 active+clean; 4.3 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-09-13T14:31:57.413 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:57.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:57 smithi102 ceph-mon[110183]: pgmap v517: 89 pgs: 89 active+clean; 4.3 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-09-13T14:31:57.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:57 smithi102 ceph-mon[115269]: pgmap v517: 89 pgs: 89 active+clean; 4.3 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-09-13T14:31:57.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:57.448 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:57.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:57.549 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:57.574 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:57.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:57.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:57.676 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:57.715 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:57.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:57.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:57.817 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:57.855 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:57.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:57.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:57.898 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 43 16 77079 77063 112 72.6875 0.0066769 0.00892421 2024-09-13T14:31:57.957 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:31:57.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:57.993 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:57.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:31:58.094 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:58.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:58.134 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:58.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:58.235 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:31:58.269 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:58.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:58.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:31:58.371 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:31:58.407 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:58.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:58.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:31:58.509 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:31:58.536 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:58.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:58.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:31:58.609 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:31:58.638 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:58.674 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:58.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:58.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:58.776 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:58.806 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:58.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:58.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:58.898 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 44 16 78287 78271 111.17 75.5 0.00990156 0.00899021 2024-09-13T14:31:58.907 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:58.933 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:58.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:58.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:59.035 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:59.063 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:59.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:59.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:59.090 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T14:31:59.165 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:59.201 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:59.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:59.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:59.303 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:59.334 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:59.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:59.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:59.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:31:59 smithi102 ceph-mon[110183]: pgmap v518: 89 pgs: 89 active+clean; 4.6 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 102 MiB/s wr, 1.62k op/s 2024-09-13T14:31:59.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:31:59 smithi102 ceph-mon[115269]: pgmap v518: 89 pgs: 89 active+clean; 4.6 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 102 MiB/s wr, 1.62k op/s 2024-09-13T14:31:59.436 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:31:59.471 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:31:59.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:59.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:31:59.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:31:59 smithi195 ceph-mon[104262]: pgmap v518: 89 pgs: 89 active+clean; 4.6 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 102 MiB/s wr, 1.62k op/s 2024-09-13T14:31:59.573 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:31:59.603 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:59.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:59.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:31:59.705 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:31:59.741 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:59.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:59.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:31:59.842 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:31:59.874 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:31:59.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:31:59.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:31:59.898 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 45 16 79498 79482 110.382 75.6875 0.0128964 0.00905429 2024-09-13T14:31:59.976 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:00.019 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:00.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:00.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:00.071 INFO:teuthology.orchestra.run.smithi102.stderr:reweighted osd.7 to 0.559354 (8f31) 2024-09-13T14:32:00.120 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:32:00.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:00.148 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:00.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:32:00.250 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:00.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:00.271 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:00.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:00.372 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:00.393 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:00.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:00.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:00.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:00 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2364091387' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 0.5593538375785977} : dispatch 2024-09-13T14:32:00.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:00 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 0.5593538375785977} : dispatch 2024-09-13T14:32:00.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:00 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2364091387' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 0.5593538375785977} : dispatch 2024-09-13T14:32:00.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:00 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 0.5593538375785977} : dispatch 2024-09-13T14:32:00.495 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:00.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:00 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2364091387' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 0.5593538375785977} : dispatch 2024-09-13T14:32:00.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:00 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 0.5593538375785977} : dispatch 2024-09-13T14:32:00.515 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:00.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:00.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:00.607 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:32:00.607 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:32:00.617 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:00.640 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:00.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:00.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:00.742 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:32:00.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:00.764 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:00.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:32:00.865 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:32:00.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:00.888 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:00.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:32:00.898 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 46 16 79765 79749 108.345 16.6875 0.0031814 0.00906503 2024-09-13T14:32:00.989 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:01.010 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:01.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:01.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:01.112 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:01.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:01.134 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:01.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:01.235 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:01.256 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:01.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:01.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:01.358 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:32:01.381 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:01.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:01.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:32:01.393 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:01 smithi102 ceph-mon[110183]: pgmap v519: 89 pgs: 89 active+clean; 4.7 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-13T14:32:01.393 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:01 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 0.5593538375785977}]': finished 2024-09-13T14:32:01.393 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:01 smithi102 ceph-mon[110183]: osdmap e170: 8 total, 8 up, 6 in 2024-09-13T14:32:01.393 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:01 smithi102 ceph-mon[115269]: pgmap v519: 89 pgs: 89 active+clean; 4.7 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-13T14:32:01.393 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:01 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 0.5593538375785977}]': finished 2024-09-13T14:32:01.393 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:01 smithi102 ceph-mon[115269]: osdmap e170: 8 total, 8 up, 6 in 2024-09-13T14:32:01.482 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:01.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:01 smithi195 ceph-mon[104262]: pgmap v519: 89 pgs: 89 active+clean; 4.7 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-13T14:32:01.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:01 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 0.5593538375785977}]': finished 2024-09-13T14:32:01.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:01 smithi195 ceph-mon[104262]: osdmap e170: 8 total, 8 up, 6 in 2024-09-13T14:32:01.504 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:01.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:01.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:01.606 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:01.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:01.628 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:01.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:01.729 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:01.750 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:01.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:01.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:01.852 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:32:01.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:01.876 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:01.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:32:01.898 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 47 16 79765 79749 106.039 0 - 0.00906503 2024-09-13T14:32:01.977 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:32:01.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:01.999 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:01.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:32:02.100 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:02.126 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:02.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:02.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:02.228 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:32:02.253 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:02.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:02.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:32:02.283 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:02 smithi102 ceph-mon[110183]: osdmap e171: 8 total, 8 up, 6 in 2024-09-13T14:32:02.283 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:02 smithi102 ceph-mon[115269]: osdmap e171: 8 total, 8 up, 6 in 2024-09-13T14:32:02.355 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:02.378 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:02.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:02.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:02.480 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:02.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:02 smithi195 ceph-mon[104262]: osdmap e171: 8 total, 8 up, 6 in 2024-09-13T14:32:02.505 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:02.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:02.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:02.607 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:32:02.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:02.630 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:02.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:32:02.731 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:02.753 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:02.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:02.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:02.857 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:32:02.879 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:02.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:02.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:32:02.898 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 48 16 79802 79786 103.878 1.15625 0.00841209 0.00922939 2024-09-13T14:32:02.981 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:03.003 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:03.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:03.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:03.105 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:32:03.127 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:03.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:03.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:32:03.229 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:03.251 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:03.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:03.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:03.353 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:03.375 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:03.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:03.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:03.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:03 smithi102 ceph-mon[110183]: pgmap v522: 89 pgs: 89 active+clean; 4.7 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 57 MiB/s wr, 919 op/s 2024-09-13T14:32:03.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:03 smithi102 ceph-mon[110183]: osdmap e172: 8 total, 8 up, 6 in 2024-09-13T14:32:03.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:03 smithi102 ceph-mon[115269]: pgmap v522: 89 pgs: 89 active+clean; 4.7 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 57 MiB/s wr, 919 op/s 2024-09-13T14:32:03.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:03 smithi102 ceph-mon[115269]: osdmap e172: 8 total, 8 up, 6 in 2024-09-13T14:32:03.477 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:32:03.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:03 smithi195 ceph-mon[104262]: pgmap v522: 89 pgs: 89 active+clean; 4.7 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 57 MiB/s wr, 919 op/s 2024-09-13T14:32:03.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:03 smithi195 ceph-mon[104262]: osdmap e172: 8 total, 8 up, 6 in 2024-09-13T14:32:03.501 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:03.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:03.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:32:03.603 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:03.624 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:03.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:03.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:03.726 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:32:03.748 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:03.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:03.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:32:03.850 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:03.873 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:03.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:03.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:03.898 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 49 16 79802 79786 101.759 0 - 0.00922939 2024-09-13T14:32:03.975 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:32:03.999 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:04.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:04.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:32:04.095 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is unset 2024-09-13T14:32:04.102 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:04.128 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:04.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:04.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:04.229 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:32:04.253 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:04.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:04.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:32:04.355 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:04.377 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:04.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:04.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:04.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:04 smithi102 ceph-mon[110183]: osdmap e173: 8 total, 8 up, 6 in 2024-09-13T14:32:04.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:04 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/274278390' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:32:04.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:04 smithi102 ceph-mon[115269]: osdmap e173: 8 total, 8 up, 6 in 2024-09-13T14:32:04.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:04 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/274278390' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:32:04.479 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:32:04.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:04 smithi195 ceph-mon[104262]: osdmap e173: 8 total, 8 up, 6 in 2024-09-13T14:32:04.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:04 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/274278390' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:32:04.502 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:04.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:04.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:32:04.604 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:04.626 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:04.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:04.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:04.728 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:04.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:04.750 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:04.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:04.852 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:04.873 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:04.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:04.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:04.898 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 50 16 79802 79786 99.7234 0 - 0.00922939 2024-09-13T14:32:04.976 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:32:04.997 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:04.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:04.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:32:05.099 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:05.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:05.122 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:05.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:05.223 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:05.245 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:05.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:05.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:05.347 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:32:05.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:05.369 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:05.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:32:05.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:05 smithi102 ceph-mon[110183]: pgmap v525: 89 pgs: 3 remapped+peering, 86 active+clean; 4.8 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 41 MiB/s wr, 656 op/s; 5 B/s, 2 objects/s recovering 2024-09-13T14:32:05.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:05 smithi102 ceph-mon[110183]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-13T14:32:05.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:05 smithi102 ceph-mon[110183]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:32:05.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:05 smithi102 ceph-mon[110183]: Cluster is now healthy 2024-09-13T14:32:05.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:05 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/274278390' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:32:05.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:05 smithi102 ceph-mon[110183]: osdmap e174: 8 total, 8 up, 6 in 2024-09-13T14:32:05.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:05 smithi102 ceph-mon[115269]: pgmap v525: 89 pgs: 3 remapped+peering, 86 active+clean; 4.8 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 41 MiB/s wr, 656 op/s; 5 B/s, 2 objects/s recovering 2024-09-13T14:32:05.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:05 smithi102 ceph-mon[115269]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-13T14:32:05.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:05 smithi102 ceph-mon[115269]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:32:05.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:05 smithi102 ceph-mon[115269]: Cluster is now healthy 2024-09-13T14:32:05.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:05 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/274278390' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:32:05.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:05 smithi102 ceph-mon[115269]: osdmap e174: 8 total, 8 up, 6 in 2024-09-13T14:32:05.471 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:05.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:05.494 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:05.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:05.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:05 smithi195 ceph-mon[104262]: pgmap v525: 89 pgs: 3 remapped+peering, 86 active+clean; 4.8 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 41 MiB/s wr, 656 op/s; 5 B/s, 2 objects/s recovering 2024-09-13T14:32:05.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:05 smithi195 ceph-mon[104262]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-13T14:32:05.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:05 smithi195 ceph-mon[104262]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:32:05.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:05 smithi195 ceph-mon[104262]: Cluster is now healthy 2024-09-13T14:32:05.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:05 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/274278390' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:32:05.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:05 smithi195 ceph-mon[104262]: osdmap e174: 8 total, 8 up, 6 in 2024-09-13T14:32:05.596 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:05.620 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:05.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:05.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:05.630 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:32:05.630 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:32:05.654 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:32:05.654 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:32:05.671 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:32:05.672 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:32:05.722 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:05.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:05.745 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:05.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:05.846 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:05.868 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:05.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:05.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:05.898 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 51 16 79802 79786 97.768 0 - 0.00922939 2024-09-13T14:32:05.969 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:05.991 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:05.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:05.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:06.093 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:06.115 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:06.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:06.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:06.144 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:32:06.144 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [], 2024-09-13T14:32:06.145 INFO:teuthology.orchestra.run.smithi102.stdout: "complaint_time": 30, 2024-09-13T14:32:06.145 INFO:teuthology.orchestra.run.smithi102.stdout: "num_blocked_ops": 0 2024-09-13T14:32:06.145 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:32:06.160 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 7] out_osds: [1, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:32:06.160 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:32:06.160 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-13T14:32:06.161 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.0 'trim stale osdmaps' 2024-09-13T14:32:06.217 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:06.239 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:06.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:06.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:06.340 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:32:06.363 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:06.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:06.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:32:06.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:06 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1500019124' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:32:06.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:06 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1500019124' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:32:06.465 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:06.482 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:06 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1500019124' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:32:06.487 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:06.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:06.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:06.589 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:06.610 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:06.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:06.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:06.712 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:06.735 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:06.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:06.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:06.836 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:06.859 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:06.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:06.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:06.899 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 52 16 79802 79786 95.8879 0 - 0.00922939 2024-09-13T14:32:06.918 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:32:06 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:32:06] "GET /metrics HTTP/1.1" 200 36517 "" "Prometheus/2.51.0" 2024-09-13T14:32:06.961 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:06.965 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:32:06.993 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T14:32:06.993 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:06.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:06.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:07.095 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:32:07.118 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:07.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:07.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:32:07.220 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:32:07.241 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:07.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:07.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:32:07.270 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:07 smithi102 ceph-mon[110183]: pgmap v527: 89 pgs: 3 remapped+peering, 86 active+clean; 4.8 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 39 MiB/s wr, 616 op/s; 5 B/s, 2 objects/s recovering 2024-09-13T14:32:07.270 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:07 smithi102 ceph-mon[115269]: pgmap v527: 89 pgs: 3 remapped+peering, 86 active+clean; 4.8 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 39 MiB/s wr, 616 op/s; 5 B/s, 2 objects/s recovering 2024-09-13T14:32:07.343 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:07.366 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:07.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:07.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:07.383 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:07 smithi195 ceph-mon[104262]: pgmap v527: 89 pgs: 3 remapped+peering, 86 active+clean; 4.8 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 39 MiB/s wr, 616 op/s; 5 B/s, 2 objects/s recovering 2024-09-13T14:32:07.468 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:07.490 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:07.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:07.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:07.547 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:32:07.592 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:07.613 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:07.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:07.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:07.715 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:07.737 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:07.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:07.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:07.839 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:07.862 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:07.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:07.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:07.899 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 53 16 79802 79786 94.0787 0 - 0.00922939 2024-09-13T14:32:07.963 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:07.984 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:07.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:07.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:08.086 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:32:08.109 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:08.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:08.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:32:08.211 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:08.232 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:08.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:08.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:08.257 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:32:08.334 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:08.357 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:08.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:08.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:08.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:08 smithi102 ceph-mon[110183]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-13T14:32:08.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:08 smithi102 ceph-mon[110183]: Cluster is now healthy 2024-09-13T14:32:08.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:08 smithi102 ceph-mon[115269]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-13T14:32:08.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:08 smithi102 ceph-mon[115269]: Cluster is now healthy 2024-09-13T14:32:08.459 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:08.480 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:08.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:08.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:08.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:08 smithi195 ceph-mon[104262]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-13T14:32:08.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:08 smithi195 ceph-mon[104262]: Cluster is now healthy 2024-09-13T14:32:08.582 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:08.604 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:08.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:08.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:08.705 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:08.727 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:08.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:08.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:08.828 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:08.850 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:08.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:08.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:08.899 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 54 16 79802 79786 92.3365 0 - 0.00922939 2024-09-13T14:32:08.948 INFO:teuthology.orchestra.run.smithi102.stderr:no valid command found; 10 closest matches: 2024-09-13T14:32:08.948 INFO:teuthology.orchestra.run.smithi102.stderr:0 2024-09-13T14:32:08.948 INFO:teuthology.orchestra.run.smithi102.stderr:1 2024-09-13T14:32:08.948 INFO:teuthology.orchestra.run.smithi102.stderr:2 2024-09-13T14:32:08.949 INFO:teuthology.orchestra.run.smithi102.stderr:abort 2024-09-13T14:32:08.949 INFO:teuthology.orchestra.run.smithi102.stderr:assert 2024-09-13T14:32:08.949 INFO:teuthology.orchestra.run.smithi102.stderr:bench [] [] [] [] 2024-09-13T14:32:08.949 INFO:teuthology.orchestra.run.smithi102.stderr:bluefs debug_inject_read_zeros 2024-09-13T14:32:08.949 INFO:teuthology.orchestra.run.smithi102.stderr:bluefs files list 2024-09-13T14:32:08.949 INFO:teuthology.orchestra.run.smithi102.stderr:bluefs stats 2024-09-13T14:32:08.949 INFO:teuthology.orchestra.run.smithi102.stderr:bluestore allocator dump block 2024-09-13T14:32:08.949 INFO:teuthology.orchestra.run.smithi102.stderr:Error EINVAL: invalid command 2024-09-13T14:32:08.951 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:32:08.976 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:08.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:08.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:32:09.078 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:32:09.101 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:09.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:09.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:32:09.162 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:09 smithi102 ceph-mon[110183]: pgmap v528: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 4.8 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 28 MiB/s wr, 442 op/s; 4495/237189 objects misplaced (1.895%); 4.6 MiB/s, 75 objects/s recovering 2024-09-13T14:32:09.163 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:09 smithi102 ceph-mon[115269]: pgmap v528: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 4.8 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 28 MiB/s wr, 442 op/s; 4495/237189 objects misplaced (1.895%); 4.6 MiB/s, 75 objects/s recovering 2024-09-13T14:32:09.203 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:32:09.225 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:09.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:09.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:32:09.326 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:32:09.348 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:09.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:09.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:32:09.450 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:09.471 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:09.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:09.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:09.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:09 smithi195 ceph-mon[104262]: pgmap v528: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 4.8 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 28 MiB/s wr, 442 op/s; 4495/237189 objects misplaced (1.895%); 4.6 MiB/s, 75 objects/s recovering 2024-09-13T14:32:09.573 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:32:09.595 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:09.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:09.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:32:09.697 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:32:09.720 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:09.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:09.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:32:09.729 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T14:32:09.729 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T14:32:09.730 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.3 'trim stale osdmaps' 2024-09-13T14:32:09.822 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:32:09.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:09.844 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:09.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:32:09.899 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 55 16 79802 79786 90.6576 0 - 0.00922939 2024-09-13T14:32:09.945 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:32:09.967 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:09.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:09.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:32:10.069 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:10.091 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:10.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:10.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:10.193 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:10.214 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:10.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:10.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:10.316 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:32:10.319 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:32:10.341 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:10.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:10.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:32:10.443 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:10.465 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:10.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:10.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:10.567 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:10.591 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:10.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:10.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:10.693 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:32:10.715 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:10.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:10.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:32:10.817 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:10.840 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:10.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:10.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:10.899 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 56 16 79802 79786 89.0388 0 - 0.00922939 2024-09-13T14:32:10.942 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:10.963 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:10.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:10.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:11.065 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:32:11.087 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:11.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:11.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:32:11.188 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:11.211 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:11.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:11.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:11.312 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:11.334 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:11.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:11.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:11.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:11 smithi102 ceph-mon[110183]: pgmap v529: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 4.8 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 4.2 MiB/s wr, 66 op/s; 4495/237195 objects misplaced (1.895%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:32:11.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:11 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:32:11.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:11 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:32:11.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:11 smithi102 ceph-mon[115269]: pgmap v529: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 4.8 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 4.2 MiB/s wr, 66 op/s; 4495/237195 objects misplaced (1.895%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:32:11.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:11 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:32:11.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:11 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:32:11.436 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:11.458 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:11.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:11.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:11.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:11 smithi195 ceph-mon[104262]: pgmap v529: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 4.8 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 4.2 MiB/s wr, 66 op/s; 4495/237195 objects misplaced (1.895%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:32:11.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:11 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:32:11.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:11 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:32:11.560 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:11.581 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:11.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:11.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:11.683 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:11.705 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:11.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:11.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:11.807 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:11.831 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:11.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:11.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:11.899 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 57 16 79802 79786 87.4767 0 - 0.00922939 2024-09-13T14:32:11.933 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:11.959 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:11.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:11.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:11.992 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:32:11.993 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:32:12.020 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:32:12.020 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:32:12.046 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:32:12.046 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:32:12.061 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:12.070 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:32:12.071 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:32:12.088 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:12.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:12.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:12.091 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:32:12.091 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:32:12.190 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:32:12.225 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:12.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:12.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:32:12.324 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:32:12.324 INFO:teuthology.orchestra.run.smithi102.stdout: "size": 20, 2024-09-13T14:32:12.324 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 600, 2024-09-13T14:32:12.324 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [ 2024-09-13T14:32:12.324 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.325 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.44546.0:90251 5.4 e129/125 5:22275ccb:::benchmark_data_smithi102_155989_object90250:head v 129'5728, mlcod=0'0)", 2024-09-13T14:32:12.325 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:27:25.251471+0000", 2024-09-13T14:32:12.325 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 287.06886771799998, 2024-09-13T14:32:12.325 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.111321792, 2024-09-13T14:32:12.325 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:32:12.325 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:32:12.325 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:32:12.325 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:32:12.325 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.325 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:32:12.325 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.251471+0000", 2024-09-13T14:32:12.325 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:32:12.325 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.326 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.326 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:32:12.326 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.251471+0000", 2024-09-13T14:32:12.326 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:32:12.326 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.326 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.326 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:32:12.326 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.251472+0000", 2024-09-13T14:32:12.326 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.2099999999999996e-07 2024-09-13T14:32:12.326 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.326 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.326 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:32:12.326 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.251533+0000", 2024-09-13T14:32:12.326 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.0692000000000001e-05 2024-09-13T14:32:12.327 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.327 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.327 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:32:12.327 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.251540+0000", 2024-09-13T14:32:12.327 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.5689999999999999e-06 2024-09-13T14:32:12.327 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.327 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.327 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:32:12.327 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.251545+0000", 2024-09-13T14:32:12.327 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.2189999999999998e-06 2024-09-13T14:32:12.327 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.327 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.327 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:32:12.328 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.255944+0000", 2024-09-13T14:32:12.328 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0043997180000000004 2024-09-13T14:32:12.328 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.328 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.328 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:32:12.328 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.255961+0000", 2024-09-13T14:32:12.328 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.6956000000000002e-05 2024-09-13T14:32:12.328 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.328 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.328 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:32:12.328 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.362738+0000", 2024-09-13T14:32:12.328 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.106777187 2024-09-13T14:32:12.328 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.328 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.329 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:32:12.329 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:25.362793+0000", 2024-09-13T14:32:12.329 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.4830000000000002e-05 2024-09-13T14:32:12.329 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:32:12.329 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:32:12.329 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:32:12.329 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.329 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.329 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:112866 5.b 5:daf8674e:::benchmark_data_smithi102_155989_object112865:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:32:12.329 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:27:54.076365+0000", 2024-09-13T14:32:12.329 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 258.24397463399998, 2024-09-13T14:32:12.329 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.12113017600000001, 2024-09-13T14:32:12.330 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:32:12.330 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:32:12.330 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:32:12.330 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:32:12.330 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:32:12.330 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:32:12.330 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 112866 2024-09-13T14:32:12.330 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.330 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:32:12.330 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.330 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:32:12.330 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.076365+0000", 2024-09-13T14:32:12.330 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:32:12.331 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.331 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.331 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:32:12.331 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.076365+0000", 2024-09-13T14:32:12.331 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:32:12.331 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.331 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.331 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:32:12.331 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.076365+0000", 2024-09-13T14:32:12.331 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.7199999999999999e-07 2024-09-13T14:32:12.331 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.331 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.331 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:32:12.332 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.076385+0000", 2024-09-13T14:32:12.332 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.9913999999999999e-05 2024-09-13T14:32:12.332 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.332 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.332 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:32:12.332 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.076391+0000", 2024-09-13T14:32:12.332 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.3049999999999998e-06 2024-09-13T14:32:12.332 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.332 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.332 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:32:12.332 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.076396+0000", 2024-09-13T14:32:12.332 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.3830000000000002e-06 2024-09-13T14:32:12.332 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.333 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.333 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:32:12.333 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.077577+0000", 2024-09-13T14:32:12.333 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0011813310000000001 2024-09-13T14:32:12.333 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.333 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.333 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:32:12.333 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.077715+0000", 2024-09-13T14:32:12.333 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00013823000000000001 2024-09-13T14:32:12.333 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.333 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.333 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 5,7", 2024-09-13T14:32:12.333 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.077886+0000", 2024-09-13T14:32:12.333 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000170874 2024-09-13T14:32:12.334 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.334 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.334 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:32:12.334 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.086317+0000", 2024-09-13T14:32:12.334 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0084313819999999994 2024-09-13T14:32:12.334 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.334 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.334 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:32:12.334 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.112248+0000", 2024-09-13T14:32:12.334 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.025930796999999998 2024-09-13T14:32:12.334 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.334 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.334 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:32:12.334 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.195948+0000", 2024-09-13T14:32:12.334 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.083699457000000005 2024-09-13T14:32:12.335 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.335 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.335 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:32:12.335 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.196021+0000", 2024-09-13T14:32:12.335 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.3787000000000004e-05 2024-09-13T14:32:12.335 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.335 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.335 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:32:12.335 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.197495+0000", 2024-09-13T14:32:12.335 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0014732440000000001 2024-09-13T14:32:12.335 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:32:12.335 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:32:12.335 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:32:12.336 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.336 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.336 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:112869 5.b 5:d70321bf:::benchmark_data_smithi102_155989_object112868:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:32:12.336 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:27:54.079135+0000", 2024-09-13T14:32:12.336 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 258.24120410199998, 2024-09-13T14:32:12.336 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.11983437399999999, 2024-09-13T14:32:12.336 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:32:12.336 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:32:12.336 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:32:12.336 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:32:12.336 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:32:12.336 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:32:12.336 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 112869 2024-09-13T14:32:12.337 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.337 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:32:12.337 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.337 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:32:12.337 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.079135+0000", 2024-09-13T14:32:12.337 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:32:12.337 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.337 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.337 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:32:12.337 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.079135+0000", 2024-09-13T14:32:12.337 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:32:12.337 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.337 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.338 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:32:12.338 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.079137+0000", 2024-09-13T14:32:12.338 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.725e-06 2024-09-13T14:32:12.338 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.338 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.338 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:32:12.338 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.079159+0000", 2024-09-13T14:32:12.338 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.2356000000000001e-05 2024-09-13T14:32:12.338 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.338 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.338 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:32:12.338 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.079172+0000", 2024-09-13T14:32:12.339 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2587e-05 2024-09-13T14:32:12.339 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.339 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.339 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:32:12.339 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.079180+0000", 2024-09-13T14:32:12.339 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.2439999999999993e-06 2024-09-13T14:32:12.339 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.339 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.339 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:32:12.339 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.082932+0000", 2024-09-13T14:32:12.339 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.003752235 2024-09-13T14:32:12.339 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.339 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.339 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:32:12.340 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.083077+0000", 2024-09-13T14:32:12.340 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00014442599999999999 2024-09-13T14:32:12.340 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.340 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.340 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 5,7", 2024-09-13T14:32:12.340 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.083236+0000", 2024-09-13T14:32:12.340 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000159247 2024-09-13T14:32:12.340 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.340 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.340 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:32:12.340 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.086405+0000", 2024-09-13T14:32:12.340 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0031694420000000002 2024-09-13T14:32:12.340 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.341 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.341 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:32:12.341 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.112288+0000", 2024-09-13T14:32:12.341 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.025882496000000001 2024-09-13T14:32:12.341 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.341 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.341 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:32:12.341 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.197578+0000", 2024-09-13T14:32:12.341 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.085290445000000006 2024-09-13T14:32:12.341 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.341 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.341 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:32:12.341 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.197653+0000", 2024-09-13T14:32:12.342 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.4931999999999993e-05 2024-09-13T14:32:12.342 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.342 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.342 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:32:12.342 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.198969+0000", 2024-09-13T14:32:12.342 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0013162390000000001 2024-09-13T14:32:12.342 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:32:12.342 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:32:12.342 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:32:12.342 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.342 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.342 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:112877 5.b 5:d625fa27:::benchmark_data_smithi102_155989_object112876:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:32:12.342 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:27:54.087314+0000", 2024-09-13T14:32:12.342 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 258.23302493199998, 2024-09-13T14:32:12.343 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.115652147, 2024-09-13T14:32:12.343 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:32:12.343 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:32:12.343 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:32:12.343 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:32:12.343 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:32:12.343 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:32:12.343 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 112877 2024-09-13T14:32:12.343 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.343 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:32:12.343 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.343 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:32:12.343 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.087314+0000", 2024-09-13T14:32:12.344 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:32:12.344 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.344 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.344 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:32:12.344 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.087314+0000", 2024-09-13T14:32:12.344 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:32:12.344 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.344 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.344 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:32:12.344 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.087315+0000", 2024-09-13T14:32:12.344 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.9500000000000002e-07 2024-09-13T14:32:12.344 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.344 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.345 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:32:12.345 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.087333+0000", 2024-09-13T14:32:12.345 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.8349999999999999e-05 2024-09-13T14:32:12.345 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.345 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.345 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:32:12.345 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.087342+0000", 2024-09-13T14:32:12.345 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.4770000000000003e-06 2024-09-13T14:32:12.345 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.345 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.345 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:32:12.345 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.087346+0000", 2024-09-13T14:32:12.345 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.6999999999999999e-06 2024-09-13T14:32:12.346 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.346 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.346 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:32:12.346 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.087392+0000", 2024-09-13T14:32:12.346 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.5998000000000002e-05 2024-09-13T14:32:12.346 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.346 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.346 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:32:12.346 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.087529+0000", 2024-09-13T14:32:12.346 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000136868 2024-09-13T14:32:12.346 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.346 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.346 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 5,7", 2024-09-13T14:32:12.347 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.087704+0000", 2024-09-13T14:32:12.347 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00017465699999999999 2024-09-13T14:32:12.347 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.347 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.347 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:32:12.347 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.090517+0000", 2024-09-13T14:32:12.347 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0028129180000000002 2024-09-13T14:32:12.347 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.347 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.347 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:32:12.347 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.112319+0000", 2024-09-13T14:32:12.347 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.021802027000000002 2024-09-13T14:32:12.347 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.347 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.348 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:32:12.348 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.201157+0000", 2024-09-13T14:32:12.348 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.088838557999999998 2024-09-13T14:32:12.348 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.348 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.348 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:32:12.348 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.201216+0000", 2024-09-13T14:32:12.348 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.8804000000000001e-05 2024-09-13T14:32:12.348 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.348 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.348 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:32:12.348 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.202966+0000", 2024-09-13T14:32:12.348 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.001750195 2024-09-13T14:32:12.349 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:32:12.349 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:32:12.349 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:32:12.349 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.349 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.349 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:112879 5.b 5:d8d74457:::benchmark_data_smithi102_155989_object112878:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:32:12.349 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:27:54.093629+0000", 2024-09-13T14:32:12.349 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 258.22671023300001, 2024-09-13T14:32:12.349 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.110882463, 2024-09-13T14:32:12.349 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:32:12.349 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:32:12.349 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:32:12.349 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:32:12.350 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:32:12.350 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:32:12.350 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 112879 2024-09-13T14:32:12.350 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.350 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:32:12.350 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.350 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:32:12.350 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.093629+0000", 2024-09-13T14:32:12.350 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:32:12.350 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.350 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.350 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:32:12.351 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.093629+0000", 2024-09-13T14:32:12.351 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:32:12.351 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.351 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.351 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:32:12.351 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.093630+0000", 2024-09-13T14:32:12.351 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1489999999999999e-06 2024-09-13T14:32:12.351 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.351 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.351 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:32:12.351 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.093648+0000", 2024-09-13T14:32:12.351 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.8099999999999999e-05 2024-09-13T14:32:12.351 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.351 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.352 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:32:12.352 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.093658+0000", 2024-09-13T14:32:12.352 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.5719999999999992e-06 2024-09-13T14:32:12.352 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.352 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.352 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:32:12.352 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.093663+0000", 2024-09-13T14:32:12.352 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.882e-06 2024-09-13T14:32:12.352 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.352 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.352 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:32:12.352 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.093732+0000", 2024-09-13T14:32:12.352 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.8986000000000007e-05 2024-09-13T14:32:12.353 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.353 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.353 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:32:12.353 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:12.354 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.093840+0000", 2024-09-13T14:32:12.354 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000108448 2024-09-13T14:32:12.354 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.354 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.354 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 5,7", 2024-09-13T14:32:12.354 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.093960+0000", 2024-09-13T14:32:12.355 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000119764 2024-09-13T14:32:12.355 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.355 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.355 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:32:12.355 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.099419+0000", 2024-09-13T14:32:12.355 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0054587689999999996 2024-09-13T14:32:12.355 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.355 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.355 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:32:12.355 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.115925+0000", 2024-09-13T14:32:12.356 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.016506460000000001 2024-09-13T14:32:12.356 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.356 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.356 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:32:12.356 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.203105+0000", 2024-09-13T14:32:12.356 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.087180013000000001 2024-09-13T14:32:12.356 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.356 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.356 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:32:12.356 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.203177+0000", 2024-09-13T14:32:12.356 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.2250999999999997e-05 2024-09-13T14:32:12.357 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.357 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.357 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:32:12.357 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.204511+0000", 2024-09-13T14:32:12.357 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.001334069 2024-09-13T14:32:12.357 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:32:12.357 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:32:12.357 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:32:12.357 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.357 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.357 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:112880 5.b 5:d2a66a20:::benchmark_data_smithi102_155989_object112879:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:32:12.357 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:27:54.093777+0000", 2024-09-13T14:32:12.358 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 258.22656204399999, 2024-09-13T14:32:12.358 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.11214436699999999, 2024-09-13T14:32:12.358 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:32:12.358 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:32:12.358 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:32:12.358 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:32:12.358 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:32:12.358 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:32:12.358 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 112880 2024-09-13T14:32:12.358 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.358 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:32:12.358 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.358 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:32:12.358 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.093777+0000", 2024-09-13T14:32:12.359 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:32:12.359 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.359 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.359 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:32:12.359 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.093777+0000", 2024-09-13T14:32:12.359 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:32:12.359 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.359 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.359 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:32:12.359 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.093778+0000", 2024-09-13T14:32:12.359 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.5199999999999996e-07 2024-09-13T14:32:12.359 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.359 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.360 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:32:12.360 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.093797+0000", 2024-09-13T14:32:12.360 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.8828999999999999e-05 2024-09-13T14:32:12.360 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.360 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.360 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:32:12.360 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.093803+0000", 2024-09-13T14:32:12.360 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.2179999999999996e-06 2024-09-13T14:32:12.360 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.360 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.360 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:32:12.360 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.093807+0000", 2024-09-13T14:32:12.361 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.788e-06 2024-09-13T14:32:12.361 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.361 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.361 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:32:12.361 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.096849+0000", 2024-09-13T14:32:12.361 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0030418849999999998 2024-09-13T14:32:12.361 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.361 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.361 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:32:12.361 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.096972+0000", 2024-09-13T14:32:12.361 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00012337399999999999 2024-09-13T14:32:12.361 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.361 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.362 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 5,7", 2024-09-13T14:32:12.362 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.097086+0000", 2024-09-13T14:32:12.362 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000114218 2024-09-13T14:32:12.362 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.362 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.362 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:32:12.362 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.099560+0000", 2024-09-13T14:32:12.362 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0024740449999999998 2024-09-13T14:32:12.362 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.362 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.362 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:32:12.363 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.192059+0000", 2024-09-13T14:32:12.363 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.092498884000000003 2024-09-13T14:32:12.363 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.363 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.363 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:32:12.363 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.204583+0000", 2024-09-13T14:32:12.363 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.012524186 2024-09-13T14:32:12.363 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.363 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.363 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:32:12.363 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.204649+0000", 2024-09-13T14:32:12.363 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.5692999999999996e-05 2024-09-13T14:32:12.363 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.364 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.364 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:32:12.364 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:54.205921+0000", 2024-09-13T14:32:12.364 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.001272495 2024-09-13T14:32:12.364 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:32:12.364 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:32:12.364 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:32:12.364 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.364 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.364 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:116679 5.8 5:190cce24:::benchmark_data_smithi102_155989_object116678:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:32:12.364 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:27:59.371873+0000", 2024-09-13T14:32:12.364 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 252.94846608899999, 2024-09-13T14:32:12.365 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.121846316, 2024-09-13T14:32:12.365 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:32:12.365 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:32:12.365 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:32:12.365 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:32:12.365 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:32:12.365 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:32:12.365 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 116679 2024-09-13T14:32:12.365 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.365 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:32:12.365 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.365 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:32:12.366 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.371873+0000", 2024-09-13T14:32:12.366 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:32:12.366 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.366 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.366 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:32:12.366 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.371873+0000", 2024-09-13T14:32:12.366 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:32:12.366 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.366 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.366 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:32:12.366 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.371875+0000", 2024-09-13T14:32:12.366 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.567e-06 2024-09-13T14:32:12.366 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.366 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.367 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:32:12.367 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.371899+0000", 2024-09-13T14:32:12.367 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.4326000000000001e-05 2024-09-13T14:32:12.367 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.367 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.367 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:32:12.367 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.371914+0000", 2024-09-13T14:32:12.367 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4865000000000001e-05 2024-09-13T14:32:12.367 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.367 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.367 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:32:12.367 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.371922+0000", 2024-09-13T14:32:12.367 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.3850000000000002e-06 2024-09-13T14:32:12.368 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.368 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.368 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:32:12.368 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.371989+0000", 2024-09-13T14:32:12.368 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.7244999999999997e-05 2024-09-13T14:32:12.368 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.368 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.368 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:32:12.368 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.372167+0000", 2024-09-13T14:32:12.368 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000177861 2024-09-13T14:32:12.368 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.368 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.368 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,5", 2024-09-13T14:32:12.369 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.372358+0000", 2024-09-13T14:32:12.369 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00019059099999999999 2024-09-13T14:32:12.369 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.369 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.369 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:32:12.369 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.375821+0000", 2024-09-13T14:32:12.369 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0034634380000000001 2024-09-13T14:32:12.369 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.369 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.369 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:32:12.369 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.382391+0000", 2024-09-13T14:32:12.369 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0065692340000000002 2024-09-13T14:32:12.370 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.370 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.370 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:32:12.370 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.491965+0000", 2024-09-13T14:32:12.370 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.10957447200000001 2024-09-13T14:32:12.370 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.370 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.370 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:32:12.370 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.492009+0000", 2024-09-13T14:32:12.370 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.4187999999999998e-05 2024-09-13T14:32:12.370 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.370 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.371 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:32:12.371 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.493719+0000", 2024-09-13T14:32:12.371 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.001710144 2024-09-13T14:32:12.371 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:32:12.371 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:32:12.371 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:32:12.371 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.371 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.371 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:116682 5.8 5:11aa58c2:::benchmark_data_smithi102_155989_object116681:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:32:12.371 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:27:59.379192+0000", 2024-09-13T14:32:12.371 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 252.941146933, 2024-09-13T14:32:12.371 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.11629210199999999, 2024-09-13T14:32:12.371 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:32:12.372 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:32:12.372 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:32:12.372 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:32:12.372 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:32:12.372 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:32:12.372 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 116682 2024-09-13T14:32:12.372 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.372 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:32:12.372 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.372 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:32:12.372 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.379192+0000", 2024-09-13T14:32:12.372 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:32:12.372 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.373 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.373 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:32:12.373 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.379192+0000", 2024-09-13T14:32:12.373 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:32:12.373 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.373 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.373 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:32:12.373 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.379194+0000", 2024-09-13T14:32:12.373 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.5600000000000001e-06 2024-09-13T14:32:12.373 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.373 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.373 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:32:12.373 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.379220+0000", 2024-09-13T14:32:12.374 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.5782000000000002e-05 2024-09-13T14:32:12.374 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.374 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.374 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:32:12.374 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.379233+0000", 2024-09-13T14:32:12.374 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3799e-05 2024-09-13T14:32:12.374 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.374 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.374 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:32:12.374 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.379241+0000", 2024-09-13T14:32:12.374 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.3880000000000002e-06 2024-09-13T14:32:12.374 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.374 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.375 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:32:12.375 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.379290+0000", 2024-09-13T14:32:12.375 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.8825000000000001e-05 2024-09-13T14:32:12.375 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.375 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.375 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:32:12.375 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.379420+0000", 2024-09-13T14:32:12.375 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00013070900000000001 2024-09-13T14:32:12.375 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.375 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.375 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,5", 2024-09-13T14:32:12.375 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.379601+0000", 2024-09-13T14:32:12.375 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00018065999999999999 2024-09-13T14:32:12.375 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.376 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.376 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:32:12.376 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.382467+0000", 2024-09-13T14:32:12.376 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0028658009999999999 2024-09-13T14:32:12.376 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.376 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.376 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:32:12.376 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.385031+0000", 2024-09-13T14:32:12.376 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0025645640000000001 2024-09-13T14:32:12.376 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.376 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.376 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:32:12.376 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.493786+0000", 2024-09-13T14:32:12.377 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.108754976 2024-09-13T14:32:12.377 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.377 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.377 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:32:12.377 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.493858+0000", 2024-09-13T14:32:12.377 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.2057000000000005e-05 2024-09-13T14:32:12.377 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.377 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.377 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:32:12.377 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.495484+0000", 2024-09-13T14:32:12.377 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0016259810000000001 2024-09-13T14:32:12.377 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:32:12.377 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:32:12.378 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:32:12.378 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.378 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.378 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:116688 5.8 5:11d64788:::benchmark_data_smithi102_155989_object116687:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-13T14:32:12.378 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:27:59.385784+0000", 2024-09-13T14:32:12.378 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 252.934554934, 2024-09-13T14:32:12.378 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.111592976, 2024-09-13T14:32:12.378 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:32:12.378 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:32:12.378 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:32:12.378 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:32:12.378 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:32:12.378 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:32:12.378 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 116688 2024-09-13T14:32:12.379 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.379 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:32:12.379 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.379 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:32:12.379 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.385784+0000", 2024-09-13T14:32:12.379 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:32:12.379 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.379 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.379 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:32:12.379 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.385784+0000", 2024-09-13T14:32:12.379 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:32:12.379 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.379 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.380 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:32:12.380 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.385785+0000", 2024-09-13T14:32:12.380 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1829999999999999e-06 2024-09-13T14:32:12.380 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.380 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.380 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:32:12.380 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.385804+0000", 2024-09-13T14:32:12.380 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.8485999999999999e-05 2024-09-13T14:32:12.380 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.380 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.380 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:32:12.380 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.385813+0000", 2024-09-13T14:32:12.380 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.8689999999999993e-06 2024-09-13T14:32:12.381 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.381 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.381 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:32:12.381 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.385818+0000", 2024-09-13T14:32:12.381 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.8219999999999997e-06 2024-09-13T14:32:12.381 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.381 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.381 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:32:12.381 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.385875+0000", 2024-09-13T14:32:12.381 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.7768e-05 2024-09-13T14:32:12.381 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.381 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.381 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:32:12.381 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.386024+0000", 2024-09-13T14:32:12.382 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00014836999999999999 2024-09-13T14:32:12.382 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.382 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.382 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,5", 2024-09-13T14:32:12.382 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.386192+0000", 2024-09-13T14:32:12.382 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00016833500000000001 2024-09-13T14:32:12.382 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.382 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.383 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:32:12.384 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.389274+0000", 2024-09-13T14:32:12.384 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0030815410000000001 2024-09-13T14:32:12.384 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.384 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.384 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:32:12.384 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.488242+0000", 2024-09-13T14:32:12.384 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.098968889000000004 2024-09-13T14:32:12.384 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.384 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.384 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:32:12.384 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.495539+0000", 2024-09-13T14:32:12.384 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0072964249999999996 2024-09-13T14:32:12.385 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.385 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.385 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:32:12.385 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.495645+0000", 2024-09-13T14:32:12.385 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000106155 2024-09-13T14:32:12.385 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.385 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.385 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:32:12.385 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:27:59.497377+0000", 2024-09-13T14:32:12.385 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0017321330000000001 2024-09-13T14:32:12.385 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:32:12.385 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:32:12.385 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:32:12.386 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.386 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.386 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:214427 5.8 5:193eda92:::benchmark_data_smithi102_155989_object96509:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-09-13T14:32:12.386 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:30:16.264603+0000", 2024-09-13T14:32:12.386 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 116.055736279, 2024-09-13T14:32:12.386 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.109349495, 2024-09-13T14:32:12.386 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:32:12.386 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:32:12.386 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:32:12.386 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:32:12.386 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:32:12.386 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:32:12.386 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 214427 2024-09-13T14:32:12.387 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.387 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:32:12.387 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.387 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:32:12.387 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.264603+0000", 2024-09-13T14:32:12.387 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:32:12.387 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.387 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.387 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:32:12.387 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.264603+0000", 2024-09-13T14:32:12.388 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:32:12.388 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.388 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.388 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:32:12.388 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.264604+0000", 2024-09-13T14:32:12.388 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.0599999999999999e-07 2024-09-13T14:32:12.388 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.388 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.388 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:32:12.388 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.264608+0000", 2024-09-13T14:32:12.388 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.873e-06 2024-09-13T14:32:12.388 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.388 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.389 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:32:12.389 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.264618+0000", 2024-09-13T14:32:12.389 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0597999999999999e-05 2024-09-13T14:32:12.389 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.389 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.389 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:32:12.389 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.264625+0000", 2024-09-13T14:32:12.389 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.404e-06 2024-09-13T14:32:12.389 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.389 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.389 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:32:12.389 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.265543+0000", 2024-09-13T14:32:12.389 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000918559 2024-09-13T14:32:12.390 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.390 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.390 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:32:12.390 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.265623+0000", 2024-09-13T14:32:12.390 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.0292999999999998e-05 2024-09-13T14:32:12.390 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.390 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.390 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,5", 2024-09-13T14:32:12.390 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.265699+0000", 2024-09-13T14:32:12.390 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.5694999999999999e-05 2024-09-13T14:32:12.390 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.390 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.390 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:32:12.391 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.268272+0000", 2024-09-13T14:32:12.391 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.002573138 2024-09-13T14:32:12.391 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.391 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.391 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:32:12.391 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.277413+0000", 2024-09-13T14:32:12.391 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0091405940000000001 2024-09-13T14:32:12.391 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.391 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.391 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:32:12.391 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.372979+0000", 2024-09-13T14:32:12.391 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.095566377999999993 2024-09-13T14:32:12.391 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.391 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.392 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:32:12.392 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.373052+0000", 2024-09-13T14:32:12.392 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.2386999999999997e-05 2024-09-13T14:32:12.392 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.392 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.392 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:32:12.392 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.373952+0000", 2024-09-13T14:32:12.392 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00090076999999999996 2024-09-13T14:32:12.392 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:32:12.392 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:32:12.392 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:32:12.392 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.392 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.393 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:214593 5.8 5:14708bfc:::benchmark_data_smithi102_155989_object96675:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-09-13T14:32:12.393 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:30:16.496356+0000", 2024-09-13T14:32:12.393 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 115.82398296, 2024-09-13T14:32:12.393 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.13860318099999999, 2024-09-13T14:32:12.393 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:32:12.393 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:32:12.393 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:32:12.393 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:32:12.393 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:32:12.393 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:32:12.393 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 214593 2024-09-13T14:32:12.393 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.394 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:32:12.394 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.394 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:32:12.394 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.496356+0000", 2024-09-13T14:32:12.394 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:32:12.394 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.394 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.394 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:32:12.394 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.496356+0000", 2024-09-13T14:32:12.394 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:32:12.394 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.394 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.394 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:32:12.395 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.496357+0000", 2024-09-13T14:32:12.395 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.2200000000000004e-07 2024-09-13T14:32:12.395 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.395 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.395 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:32:12.395 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.496359+0000", 2024-09-13T14:32:12.395 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.6709999999999999e-06 2024-09-13T14:32:12.395 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.395 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.395 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:32:12.395 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.496366+0000", 2024-09-13T14:32:12.395 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.5169999999999996e-06 2024-09-13T14:32:12.395 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.396 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.396 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:32:12.396 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.496371+0000", 2024-09-13T14:32:12.396 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.0289999999999996e-06 2024-09-13T14:32:12.396 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.396 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.396 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:32:12.396 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.498993+0000", 2024-09-13T14:32:12.396 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.002622302 2024-09-13T14:32:12.396 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.396 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.396 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:32:12.397 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.499076+0000", 2024-09-13T14:32:12.397 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.2527000000000005e-05 2024-09-13T14:32:12.397 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.397 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.397 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,5", 2024-09-13T14:32:12.397 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.499153+0000", 2024-09-13T14:32:12.397 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.7348000000000006e-05 2024-09-13T14:32:12.397 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.397 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.397 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:32:12.397 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.500791+0000", 2024-09-13T14:32:12.397 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0016374269999999999 2024-09-13T14:32:12.397 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.398 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.398 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:32:12.398 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.511615+0000", 2024-09-13T14:32:12.398 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.010824375000000001 2024-09-13T14:32:12.398 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.398 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.398 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:32:12.398 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.527940+0000", 2024-09-13T14:32:12.398 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.016324939 2024-09-13T14:32:12.398 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.398 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.398 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:32:12.399 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.527971+0000", 2024-09-13T14:32:12.399 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.1355e-05 2024-09-13T14:32:12.399 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.399 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.399 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:32:12.399 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.634959+0000", 2024-09-13T14:32:12.399 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.10698816899999999 2024-09-13T14:32:12.399 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:32:12.399 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:32:12.399 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:32:12.399 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.399 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.399 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:214602 5.8 5:11b8c183:::benchmark_data_smithi102_155989_object96684:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-09-13T14:32:12.400 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:30:16.506765+0000", 2024-09-13T14:32:12.400 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 115.813574018, 2024-09-13T14:32:12.400 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.12949129400000001, 2024-09-13T14:32:12.400 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:32:12.400 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:32:12.401 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:32:12.401 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:32:12.402 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:32:12.402 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:32:12.402 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 214602 2024-09-13T14:32:12.402 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.402 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:32:12.402 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.402 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:32:12.402 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.506765+0000", 2024-09-13T14:32:12.402 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:32:12.402 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.403 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.403 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:32:12.403 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.506765+0000", 2024-09-13T14:32:12.403 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:32:12.403 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.403 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.403 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:32:12.403 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.506766+0000", 2024-09-13T14:32:12.403 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.037e-06 2024-09-13T14:32:12.403 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.403 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.403 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:32:12.403 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.506771+0000", 2024-09-13T14:32:12.404 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.9509999999999997e-06 2024-09-13T14:32:12.404 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.404 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.404 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:32:12.404 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.506783+0000", 2024-09-13T14:32:12.404 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2255999999999999e-05 2024-09-13T14:32:12.404 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.404 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.404 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:32:12.404 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.506791+0000", 2024-09-13T14:32:12.404 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.1070000000000005e-06 2024-09-13T14:32:12.404 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.404 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.404 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:32:12.405 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.506847+0000", 2024-09-13T14:32:12.405 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.5399000000000003e-05 2024-09-13T14:32:12.405 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.405 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.405 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:32:12.405 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.506981+0000", 2024-09-13T14:32:12.405 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00013389700000000001 2024-09-13T14:32:12.405 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.405 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.405 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,5", 2024-09-13T14:32:12.405 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.507113+0000", 2024-09-13T14:32:12.405 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00013256699999999999 2024-09-13T14:32:12.405 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.405 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.405 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:32:12.406 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.509621+0000", 2024-09-13T14:32:12.406 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0025073949999999999 2024-09-13T14:32:12.406 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.406 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.406 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:32:12.406 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.518307+0000", 2024-09-13T14:32:12.406 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0086859469999999994 2024-09-13T14:32:12.406 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.406 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.406 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:32:12.406 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.635115+0000", 2024-09-13T14:32:12.406 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.11680781799999999 2024-09-13T14:32:12.406 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.406 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.407 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:32:12.407 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.635166+0000", 2024-09-13T14:32:12.407 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.1688999999999999e-05 2024-09-13T14:32:12.407 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.407 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.407 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:32:12.407 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.636256+0000", 2024-09-13T14:32:12.407 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0010902310000000001 2024-09-13T14:32:12.407 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:32:12.407 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:32:12.407 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:32:12.407 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.407 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.408 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:214612 5.5 5:aeeffdc0:::benchmark_data_smithi102_155989_object96694:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-09-13T14:32:12.408 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:30:16.524423+0000", 2024-09-13T14:32:12.408 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 115.795916193, 2024-09-13T14:32:12.408 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.11197043399999999, 2024-09-13T14:32:12.408 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:32:12.408 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:32:12.408 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:32:12.408 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:32:12.408 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:32:12.408 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:32:12.408 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 214612 2024-09-13T14:32:12.408 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.408 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:32:12.408 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.409 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:32:12.409 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.524423+0000", 2024-09-13T14:32:12.409 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:32:12.409 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.409 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.409 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:32:12.409 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.524423+0000", 2024-09-13T14:32:12.409 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:32:12.409 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.409 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.409 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:32:12.409 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.524424+0000", 2024-09-13T14:32:12.409 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.9000000000000005e-07 2024-09-13T14:32:12.410 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.410 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.410 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:32:12.410 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.524429+0000", 2024-09-13T14:32:12.410 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.0100000000000003e-06 2024-09-13T14:32:12.410 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.410 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.410 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:32:12.410 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.524439+0000", 2024-09-13T14:32:12.410 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0389e-05 2024-09-13T14:32:12.410 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.410 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.411 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:32:12.411 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.524445+0000", 2024-09-13T14:32:12.411 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.1649999999999997e-06 2024-09-13T14:32:12.411 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.411 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.411 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:32:12.411 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.524488+0000", 2024-09-13T14:32:12.411 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.3918999999999997e-05 2024-09-13T14:32:12.411 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.411 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.411 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:32:12.411 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.524605+0000", 2024-09-13T14:32:12.411 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000116994 2024-09-13T14:32:12.411 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.412 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.412 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:32:12.412 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.524718+0000", 2024-09-13T14:32:12.412 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00011239999999999999 2024-09-13T14:32:12.412 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.412 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.412 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:32:12.412 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.634996+0000", 2024-09-13T14:32:12.412 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.11027838700000001 2024-09-13T14:32:12.412 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.412 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.412 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:32:12.412 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.635015+0000", 2024-09-13T14:32:12.413 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.8980000000000001e-05 2024-09-13T14:32:12.413 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.413 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.413 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:32:12.413 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.636316+0000", 2024-09-13T14:32:12.413 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0013011920000000001 2024-09-13T14:32:12.413 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.413 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.413 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:32:12.413 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.636361+0000", 2024-09-13T14:32:12.413 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.4953999999999997e-05 2024-09-13T14:32:12.413 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.413 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.413 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:32:12.414 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.636393+0000", 2024-09-13T14:32:12.414 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.2054000000000002e-05 2024-09-13T14:32:12.414 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:32:12.414 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:32:12.414 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:32:12.414 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.414 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.414 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.44546.0:214610 5.7 e150/116 5:ed8a2d11:::benchmark_data_smithi102_155989_object96692:head v 150'13566, mlcod=150'13563)", 2024-09-13T14:32:12.414 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:30:16.526051+0000", 2024-09-13T14:32:12.414 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 115.794288447, 2024-09-13T14:32:12.414 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.108924958, 2024-09-13T14:32:12.414 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:32:12.414 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:32:12.415 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:32:12.415 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:32:12.415 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.415 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:32:12.415 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.526051+0000", 2024-09-13T14:32:12.415 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:32:12.415 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.415 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.415 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:32:12.415 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.526051+0000", 2024-09-13T14:32:12.415 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:32:12.415 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.415 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.416 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:32:12.416 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.526051+0000", 2024-09-13T14:32:12.416 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.9999999999999998e-07 2024-09-13T14:32:12.416 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.416 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.416 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:32:12.416 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.526056+0000", 2024-09-13T14:32:12.416 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.6249999999999998e-06 2024-09-13T14:32:12.416 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.416 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.416 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:32:12.416 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.526063+0000", 2024-09-13T14:32:12.416 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.6800000000000004e-06 2024-09-13T14:32:12.417 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.417 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.417 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:32:12.417 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.526066+0000", 2024-09-13T14:32:12.417 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.9689999999999996e-06 2024-09-13T14:32:12.417 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.417 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.417 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:32:12.417 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.526106+0000", 2024-09-13T14:32:12.417 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.968e-05 2024-09-13T14:32:12.417 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.417 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.417 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:32:12.418 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.526119+0000", 2024-09-13T14:32:12.418 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3047000000000001e-05 2024-09-13T14:32:12.418 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.418 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.418 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:32:12.418 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.634927+0000", 2024-09-13T14:32:12.418 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.10880819899999999 2024-09-13T14:32:12.418 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.418 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.418 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:32:12.418 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.634976+0000", 2024-09-13T14:32:12.418 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.8257999999999998e-05 2024-09-13T14:32:12.418 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:32:12.419 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:32:12.419 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:32:12.419 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.419 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.419 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop_reply(client.44546.0:214602 5.8 e150/125 ondisk, result = 0)", 2024-09-13T14:32:12.419 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:30:16.526978+0000", 2024-09-13T14:32:12.419 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 115.793360853, 2024-09-13T14:32:12.419 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.109285152, 2024-09-13T14:32:12.419 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:32:12.419 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:32:12.419 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "started", 2024-09-13T14:32:12.419 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:32:12.420 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.420 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:32:12.420 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.526978+0000", 2024-09-13T14:32:12.420 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:32:12.420 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.420 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.420 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:32:12.420 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.526978+0000", 2024-09-13T14:32:12.420 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:32:12.420 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.420 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.420 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:32:12.420 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.526978+0000", 2024-09-13T14:32:12.420 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3e-07 2024-09-13T14:32:12.421 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.421 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.421 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:32:12.421 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.526980+0000", 2024-09-13T14:32:12.421 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2240000000000001e-06 2024-09-13T14:32:12.421 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.421 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.421 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:32:12.421 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.526983+0000", 2024-09-13T14:32:12.421 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.275e-06 2024-09-13T14:32:12.421 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.421 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.421 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:32:12.422 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.526985+0000", 2024-09-13T14:32:12.422 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.5339999999999998e-06 2024-09-13T14:32:12.422 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.422 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.422 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:32:12.422 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.635100+0000", 2024-09-13T14:32:12.422 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.10811415200000001 2024-09-13T14:32:12.422 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.422 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.422 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:32:12.422 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.635108+0000", 2024-09-13T14:32:12.422 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.3429999999999996e-06 2024-09-13T14:32:12.422 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.422 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.423 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:32:12.423 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.636263+0000", 2024-09-13T14:32:12.423 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.001155494 2024-09-13T14:32:12.423 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:32:12.423 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:32:12.423 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:32:12.423 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.423 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.423 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:214616 5.5 5:a7e320a8:::benchmark_data_smithi102_155989_object96698:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-09-13T14:32:12.423 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:30:16.527012+0000", 2024-09-13T14:32:12.423 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 115.79332682099999, 2024-09-13T14:32:12.423 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.11150837700000001, 2024-09-13T14:32:12.423 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:32:12.424 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:32:12.424 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:32:12.424 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:32:12.424 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:32:12.424 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:32:12.424 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 214616 2024-09-13T14:32:12.424 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.424 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:32:12.424 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.424 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:32:12.424 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.527012+0000", 2024-09-13T14:32:12.424 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:32:12.424 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.424 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.425 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:32:12.425 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.527012+0000", 2024-09-13T14:32:12.425 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:32:12.425 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.425 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.425 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:32:12.425 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.527013+0000", 2024-09-13T14:32:12.425 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.8899999999999998e-07 2024-09-13T14:32:12.425 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.425 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.425 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:32:12.425 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.527017+0000", 2024-09-13T14:32:12.425 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.844e-06 2024-09-13T14:32:12.425 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.426 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.426 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:32:12.426 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.527025+0000", 2024-09-13T14:32:12.426 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.503e-06 2024-09-13T14:32:12.426 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.426 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.426 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:32:12.426 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.527029+0000", 2024-09-13T14:32:12.426 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.941e-06 2024-09-13T14:32:12.426 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.426 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.426 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:32:12.426 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.527398+0000", 2024-09-13T14:32:12.426 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000369343 2024-09-13T14:32:12.427 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.427 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.427 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:32:12.427 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.527482+0000", 2024-09-13T14:32:12.427 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.4383999999999997e-05 2024-09-13T14:32:12.427 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.427 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.427 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:32:12.427 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.527563+0000", 2024-09-13T14:32:12.427 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.1247000000000003e-05 2024-09-13T14:32:12.427 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.427 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.427 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:32:12.427 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.635050+0000", 2024-09-13T14:32:12.428 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.10748634999999999 2024-09-13T14:32:12.428 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.428 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.428 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:32:12.428 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.636422+0000", 2024-09-13T14:32:12.428 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.001372177 2024-09-13T14:32:12.428 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.428 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.428 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:32:12.428 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.638408+0000", 2024-09-13T14:32:12.428 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0019862259999999998 2024-09-13T14:32:12.428 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.428 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.429 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:32:12.429 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.638479+0000", 2024-09-13T14:32:12.429 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.1080000000000004e-05 2024-09-13T14:32:12.429 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.429 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.429 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:32:12.429 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.638521+0000", 2024-09-13T14:32:12.429 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.1393000000000002e-05 2024-09-13T14:32:12.429 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:32:12.429 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:32:12.429 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:32:12.429 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.429 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.429 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.44546.0:214618 5.0 e150/116 5:09ce4356:::benchmark_data_smithi102_155989_object96700:head v 150'13609, mlcod=150'13607)", 2024-09-13T14:32:12.430 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:30:16.527857+0000", 2024-09-13T14:32:12.430 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 115.792482139, 2024-09-13T14:32:12.430 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.12100894399999999, 2024-09-13T14:32:12.430 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:32:12.430 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:32:12.430 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:32:12.430 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:32:12.430 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.430 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:32:12.430 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.527857+0000", 2024-09-13T14:32:12.430 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:32:12.430 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.430 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.430 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:32:12.431 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.527857+0000", 2024-09-13T14:32:12.431 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:32:12.431 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.431 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.431 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:32:12.431 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.527858+0000", 2024-09-13T14:32:12.431 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.2500000000000005e-07 2024-09-13T14:32:12.431 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.431 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.431 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:32:12.431 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.527863+0000", 2024-09-13T14:32:12.431 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.3329999999999996e-06 2024-09-13T14:32:12.431 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.432 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.432 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:32:12.432 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.527872+0000", 2024-09-13T14:32:12.432 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.6330000000000002e-06 2024-09-13T14:32:12.432 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.432 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.432 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:32:12.432 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.527876+0000", 2024-09-13T14:32:12.432 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.313e-06 2024-09-13T14:32:12.432 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.432 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.432 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:32:12.432 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.634996+0000", 2024-09-13T14:32:12.432 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.107120409 2024-09-13T14:32:12.433 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.433 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.433 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:32:12.433 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.635024+0000", 2024-09-13T14:32:12.433 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.7143000000000001e-05 2024-09-13T14:32:12.433 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.433 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.433 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:32:12.433 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.648820+0000", 2024-09-13T14:32:12.433 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.013796446 2024-09-13T14:32:12.433 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.433 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.433 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:32:12.434 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.648866+0000", 2024-09-13T14:32:12.434 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.5942000000000002e-05 2024-09-13T14:32:12.434 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:32:12.434 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:32:12.434 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:32:12.434 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.434 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.434 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.44546.0:214611 5.e e150/116 5:7939c0fa:::benchmark_data_smithi102_155989_object96693:head v 150'13437, mlcod=150'13433)", 2024-09-13T14:32:12.434 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:30:16.530619+0000", 2024-09-13T14:32:12.434 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 115.789720042, 2024-09-13T14:32:12.434 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.11163034300000001, 2024-09-13T14:32:12.434 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:32:12.434 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:32:12.435 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:32:12.435 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:32:12.435 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.435 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:32:12.435 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.530619+0000", 2024-09-13T14:32:12.435 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:32:12.435 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.435 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.435 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:32:12.435 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.530619+0000", 2024-09-13T14:32:12.435 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:32:12.435 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.435 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.435 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:32:12.435 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.530620+0000", 2024-09-13T14:32:12.436 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.0400000000000005e-07 2024-09-13T14:32:12.436 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.436 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.436 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:32:12.436 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.530629+0000", 2024-09-13T14:32:12.436 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.0289999999999994e-06 2024-09-13T14:32:12.436 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.436 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.436 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:32:12.436 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.634987+0000", 2024-09-13T14:32:12.436 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.104358387 2024-09-13T14:32:12.436 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.436 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.437 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:32:12.437 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.634993+0000", 2024-09-13T14:32:12.437 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.0700000000000003e-06 2024-09-13T14:32:12.437 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.437 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.437 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:32:12.437 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.635042+0000", 2024-09-13T14:32:12.437 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.8959999999999999e-05 2024-09-13T14:32:12.437 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.437 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.437 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:32:12.437 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.635060+0000", 2024-09-13T14:32:12.437 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.7455000000000001e-05 2024-09-13T14:32:12.437 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.438 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.438 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:32:12.438 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.642208+0000", 2024-09-13T14:32:12.438 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0071481560000000001 2024-09-13T14:32:12.438 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.438 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.438 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:32:12.438 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.642249+0000", 2024-09-13T14:32:12.438 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.1482e-05 2024-09-13T14:32:12.438 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:32:12.438 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:32:12.438 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:32:12.438 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.438 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.439 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.44546.0:214619 5.0 e150/116 5:04dd9390:::benchmark_data_smithi102_155989_object96701:head v 150'13610, mlcod=150'13607)", 2024-09-13T14:32:12.439 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:30:16.530878+0000", 2024-09-13T14:32:12.439 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 115.789461356, 2024-09-13T14:32:12.439 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.11801853, 2024-09-13T14:32:12.439 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:32:12.439 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:32:12.439 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:32:12.439 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:32:12.439 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.439 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:32:12.439 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.530878+0000", 2024-09-13T14:32:12.439 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:32:12.439 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.439 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.440 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:32:12.440 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.530878+0000", 2024-09-13T14:32:12.440 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:32:12.440 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.440 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.440 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:32:12.440 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.530878+0000", 2024-09-13T14:32:12.440 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.2600000000000002e-07 2024-09-13T14:32:12.440 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.440 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.440 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:32:12.440 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.530883+0000", 2024-09-13T14:32:12.440 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.3819999999999997e-06 2024-09-13T14:32:12.440 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.440 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.441 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:32:12.441 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.634909+0000", 2024-09-13T14:32:12.441 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.104026333 2024-09-13T14:32:12.441 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.441 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.441 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:32:12.441 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.634923+0000", 2024-09-13T14:32:12.441 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3699999999999999e-05 2024-09-13T14:32:12.441 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.441 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.441 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:32:12.441 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.638825+0000", 2024-09-13T14:32:12.441 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00390208 2024-09-13T14:32:12.441 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.441 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.442 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:32:12.442 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.638842+0000", 2024-09-13T14:32:12.442 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.7041000000000002e-05 2024-09-13T14:32:12.442 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.442 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.442 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:32:12.442 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.648869+0000", 2024-09-13T14:32:12.442 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.010027348 2024-09-13T14:32:12.442 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.442 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.442 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:32:12.442 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:16.648896+0000", 2024-09-13T14:32:12.442 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.7120000000000001e-05 2024-09-13T14:32:12.442 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:32:12.442 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:32:12.443 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:32:12.443 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.443 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.443 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.44546.0:233223 5.b 5:d1d53486:::benchmark_data_smithi102_155989_object115305:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-13T14:32:12.443 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:30:41.665412+0000", 2024-09-13T14:32:12.443 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 90.654927628999999, 2024-09-13T14:32:12.443 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.13849925799999999, 2024-09-13T14:32:12.443 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:32:12.443 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:32:12.443 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:32:12.443 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:32:12.443 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.44546", 2024-09-13T14:32:12.443 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:32:12.443 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 233223 2024-09-13T14:32:12.443 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.443 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:32:12.444 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.444 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:32:12.444 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:41.665412+0000", 2024-09-13T14:32:12.444 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:32:12.444 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.444 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.444 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:32:12.444 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:41.665412+0000", 2024-09-13T14:32:12.444 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:32:12.444 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.444 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.444 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:32:12.444 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:41.665413+0000", 2024-09-13T14:32:12.444 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.677e-06 2024-09-13T14:32:12.444 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.445 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.445 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:32:12.445 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:41.665419+0000", 2024-09-13T14:32:12.445 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.7189999999999998e-06 2024-09-13T14:32:12.445 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.445 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.445 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:32:12.445 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:41.665429+0000", 2024-09-13T14:32:12.445 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0071000000000001e-05 2024-09-13T14:32:12.445 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.445 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.445 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:32:12.445 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:41.665436+0000", 2024-09-13T14:32:12.445 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.2529999999999998e-06 2024-09-13T14:32:12.445 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.446 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.446 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:32:12.446 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:41.665485+0000", 2024-09-13T14:32:12.446 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.8977999999999998e-05 2024-09-13T14:32:12.446 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.446 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.446 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:32:12.446 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:41.665570+0000", 2024-09-13T14:32:12.446 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.4965999999999999e-05 2024-09-13T14:32:12.446 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.446 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.446 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 5,7", 2024-09-13T14:32:12.447 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:41.665658+0000", 2024-09-13T14:32:12.447 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.7795999999999998e-05 2024-09-13T14:32:12.447 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.447 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.447 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:32:12.447 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:41.668353+0000", 2024-09-13T14:32:12.447 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.002694952 2024-09-13T14:32:12.447 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.447 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.447 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:32:12.447 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:41.672201+0000", 2024-09-13T14:32:12.447 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0038476510000000001 2024-09-13T14:32:12.447 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.447 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.448 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:32:12.448 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:41.803829+0000", 2024-09-13T14:32:12.448 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.131628 2024-09-13T14:32:12.448 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.448 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.448 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:32:12.448 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:41.803872+0000", 2024-09-13T14:32:12.448 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.3025000000000002e-05 2024-09-13T14:32:12.448 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:32:12.448 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:32:12.448 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:32:12.448 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:30:41.803911+0000", 2024-09-13T14:32:12.448 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.9169999999999999e-05 2024-09-13T14:32:12.449 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:32:12.449 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:32:12.449 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:32:12.449 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:32:12.449 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:32:12.449 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:32:12.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:12.451 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:12.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:12.552 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:32:12.588 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:12.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:12.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:32:12.690 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:12.742 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:12.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:12.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:12.844 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:12.878 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:12.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:12.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:12.899 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 58 16 80714 80698 86.9512 5.7 0.0104312 0.0114959 2024-09-13T14:32:12.981 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:13.024 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:13.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:13.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:13.125 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:32:13.156 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:13.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:13.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:32:13.258 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:32:13.288 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:13.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:13.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:32:13.390 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:13.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:13 smithi102 ceph-mon[110183]: pgmap v530: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 4.8 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 233 KiB/s wr, 3 op/s; 4495/237195 objects misplaced (1.895%); 3.4 MiB/s, 54 objects/s recovering 2024-09-13T14:32:13.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:13 smithi102 ceph-mon[115269]: pgmap v530: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 4.8 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 233 KiB/s wr, 3 op/s; 4495/237195 objects misplaced (1.895%); 3.4 MiB/s, 54 objects/s recovering 2024-09-13T14:32:13.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:13.437 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:13.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:13.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:13 smithi195 ceph-mon[104262]: pgmap v530: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 4.8 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 233 KiB/s wr, 3 op/s; 4495/237195 objects misplaced (1.895%); 3.4 MiB/s, 54 objects/s recovering 2024-09-13T14:32:13.538 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:13.582 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:13.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:13.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:13.648 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:32:13.681 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T14:32:13.685 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:13.729 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:13.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:13.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:13.831 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:13.872 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:13.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:13.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:13.899 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 59 16 81795 81779 86.6224 67.5625 0.014321 0.0115391 2024-09-13T14:32:13.974 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:14.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:14.017 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:14.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:14.118 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:14.161 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:14.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:14.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:14.263 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:32:14.299 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:14.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:14.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:32:14.401 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:32:14.428 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:14.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:14.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:32:14.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:14 smithi195 ceph-mon[104262]: Health check failed: Degraded data redundancy: 4886/239640 objects degraded (2.039%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:32:14.529 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:32:14.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:14.560 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:14.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:32:14.661 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:14.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:14 smithi102 ceph-mon[110183]: Health check failed: Degraded data redundancy: 4886/239640 objects degraded (2.039%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:32:14.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:14 smithi102 ceph-mon[115269]: Health check failed: Degraded data redundancy: 4886/239640 objects degraded (2.039%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:32:14.696 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:14.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:14.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:14.797 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:32:14.831 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:14.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:14.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:32:14.899 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:2024-09-13T14:32:14.899870+0000 min lat: 0.00118498 max lat: 12.1717 avg lat: 0.0115818 2024-09-13T14:32:14.900 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:32:14.900 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 60 15 82873 82858 86.3024 67.4375 0.0121025 0.0115818 2024-09-13T14:32:14.933 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:32:14.970 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:14.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:14.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:32:15.072 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:15.106 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:15.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:15.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:15.208 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:15.245 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:15.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:15.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:15.347 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:15.382 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:15.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:15.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:15.471 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:15 smithi102 ceph-mon[110183]: pgmap v531: 89 pgs: 1 activating+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 86 active+clean; 4.9 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 5.3 MiB/s wr, 84 op/s; 4886/239640 objects degraded (2.039%); 8263/239640 objects misplaced (3.448%); 11 MiB/s, 176 objects/s recovering 2024-09-13T14:32:15.471 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:15 smithi102 ceph-mon[115269]: pgmap v531: 89 pgs: 1 activating+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 86 active+clean; 4.9 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 5.3 MiB/s wr, 84 op/s; 4886/239640 objects degraded (2.039%); 8263/239640 objects misplaced (3.448%); 11 MiB/s, 176 objects/s recovering 2024-09-13T14:32:15.485 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:32:15.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:15 smithi195 ceph-mon[104262]: pgmap v531: 89 pgs: 1 activating+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 86 active+clean; 4.9 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 5.3 MiB/s wr, 84 op/s; 4886/239640 objects degraded (2.039%); 8263/239640 objects misplaced (3.448%); 11 MiB/s, 176 objects/s recovering 2024-09-13T14:32:15.514 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:15.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:15.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:32:15.616 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:15.645 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:15.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:15.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:15.747 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:15.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:15.781 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:15.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:15.882 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:15.899 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 61 16 83955 83939 85.995 67.5625 0.0131893 0.0116231 2024-09-13T14:32:15.925 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:15.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:15.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:16.027 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:32:16.061 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:16.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:16.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:32:16.067 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:32:16.163 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:16.205 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:16.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:16.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:16.306 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:16.342 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:16.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:16.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:16.445 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:16.459 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:32:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:32:16] "GET /metrics HTTP/1.1" 200 36554 "" "Prometheus/2.51.0" 2024-09-13T14:32:16.489 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:16.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:16.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:16.592 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:16.628 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:16.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:16.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:16.730 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:32:16.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:16.754 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:16.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:32:16.855 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:32:16.884 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:16.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:16.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:32:16.900 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 62 15 84997 84982 85.6594 65.1875 0.00846837 0.0116695 2024-09-13T14:32:16.986 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:17.018 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:17.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:17.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:17.120 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:32:17.150 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:17.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:17.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:32:17.236 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is set 2024-09-13T14:32:17.252 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:17.293 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:17.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:17.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:17.382 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:32:17.397 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:17.466 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:17.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:17.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:17.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:17 smithi195 ceph-mon[104262]: pgmap v532: 89 pgs: 1 activating+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 86 active+clean; 4.9 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 4.7 MiB/s wr, 75 op/s; 4886/239640 objects degraded (2.039%); 8263/239640 objects misplaced (3.448%); 9.8 MiB/s, 157 objects/s recovering 2024-09-13T14:32:17.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:17 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/952046411' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:32:17.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:17 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:32:17.568 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:32:17.590 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:17.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:17.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:32:17.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:17 smithi102 ceph-mon[110183]: pgmap v532: 89 pgs: 1 activating+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 86 active+clean; 4.9 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 4.7 MiB/s wr, 75 op/s; 4886/239640 objects degraded (2.039%); 8263/239640 objects misplaced (3.448%); 9.8 MiB/s, 157 objects/s recovering 2024-09-13T14:32:17.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:17 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/952046411' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:32:17.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:17 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:32:17.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:17 smithi102 ceph-mon[115269]: pgmap v532: 89 pgs: 1 activating+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 86 active+clean; 4.9 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 4.7 MiB/s wr, 75 op/s; 4886/239640 objects degraded (2.039%); 8263/239640 objects misplaced (3.448%); 9.8 MiB/s, 157 objects/s recovering 2024-09-13T14:32:17.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:17 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/952046411' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:32:17.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:17 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:32:17.691 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:17.723 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:17.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:17.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:17.825 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:17.862 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:17.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:17.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:17.901 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 63 16 85909 85893 85.2012 56.9375 0.0141454 0.0117324 2024-09-13T14:32:17.964 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:32:17.992 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:17.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:17.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:32:18.095 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:32:18.125 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:18.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:18.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:32:18.227 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:18.260 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:18.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:18.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:18.362 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:18.397 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:18.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:18.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:18.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:18 smithi102 ceph-mon[110183]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:32:18.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:18 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:32:18.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:18 smithi102 ceph-mon[110183]: osdmap e175: 8 total, 8 up, 6 in 2024-09-13T14:32:18.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:18 smithi102 ceph-mon[115269]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:32:18.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:18 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:32:18.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:18 smithi102 ceph-mon[115269]: osdmap e175: 8 total, 8 up, 6 in 2024-09-13T14:32:18.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:18 smithi195 ceph-mon[104262]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:32:18.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:18 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:32:18.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:18 smithi195 ceph-mon[104262]: osdmap e175: 8 total, 8 up, 6 in 2024-09-13T14:32:18.499 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:18.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:18.537 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:18.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:18.638 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:18.669 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:18.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:18.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:18.771 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:18.799 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:18.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:18.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:18.901 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:18.902 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 64 16 86934 86918 84.8708 64.0625 0.0252101 0.0117775 2024-09-13T14:32:18.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:18.941 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:18.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:19.042 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:19.077 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:19.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:19.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:19.179 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:19.216 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:19.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:19.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:19.318 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:19.355 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:19.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:19.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:19.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:19 smithi102 ceph-mon[110183]: pgmap v534: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 86 active+clean; 5.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 33 MiB/s wr, 534 op/s; 8185/253221 objects degraded (3.232%); 4695/253221 objects misplaced (1.854%); 9.9 MiB/s, 158 objects/s recovering 2024-09-13T14:32:19.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:19 smithi102 ceph-mon[115269]: pgmap v534: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 86 active+clean; 5.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 33 MiB/s wr, 534 op/s; 8185/253221 objects degraded (3.232%); 4695/253221 objects misplaced (1.854%); 9.9 MiB/s, 158 objects/s recovering 2024-09-13T14:32:19.457 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:19.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:19.494 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:19.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:19.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:19 smithi195 ceph-mon[104262]: pgmap v534: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 86 active+clean; 5.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 33 MiB/s wr, 534 op/s; 8185/253221 objects degraded (3.232%); 4695/253221 objects misplaced (1.854%); 9.9 MiB/s, 158 objects/s recovering 2024-09-13T14:32:19.596 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:19.634 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:19.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:19.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:19.736 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:32:19.774 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:19.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:19.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:32:19.876 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:32:19.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:19.900 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:19.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:32:19.904 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 65 16 87874 87858 84.4689 58.75 0.00699937 0.011822 2024-09-13T14:32:20.001 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:20.041 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:20.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:20.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:20.143 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:20.174 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:20.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:20.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:20.276 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:20.307 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:20.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:20.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:20.409 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:32:20.437 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:20.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:20.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:32:20.539 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:32:20.569 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:20.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:20.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:32:20.599 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:20 smithi102 ceph-mon[110183]: pgmap v535: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 86 active+clean; 5.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 38 MiB/s wr, 604 op/s; 8185/255333 objects degraded (3.206%); 4695/255333 objects misplaced (1.839%); 9.9 MiB/s, 158 objects/s recovering 2024-09-13T14:32:20.599 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:20 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 8185/255333 objects degraded (3.206%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:20.599 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:20 smithi102 ceph-mon[115269]: pgmap v535: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 86 active+clean; 5.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 38 MiB/s wr, 604 op/s; 8185/255333 objects degraded (3.206%); 4695/255333 objects misplaced (1.839%); 9.9 MiB/s, 158 objects/s recovering 2024-09-13T14:32:20.599 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:20 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 8185/255333 objects degraded (3.206%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:20.611 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:32:20.671 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:20.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:20.706 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:20.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:20.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:20 smithi195 ceph-mon[104262]: pgmap v535: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 86 active+clean; 5.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 38 MiB/s wr, 604 op/s; 8185/255333 objects degraded (3.206%); 4695/255333 objects misplaced (1.839%); 9.9 MiB/s, 158 objects/s recovering 2024-09-13T14:32:20.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:20 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 8185/255333 objects degraded (3.206%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:20.807 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:20.833 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:20.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:20.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:20.902 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 66 16 88810 88794 84.0754 58.5 0.016389 0.0118892 2024-09-13T14:32:20.935 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:20.956 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:20.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:20.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:21.059 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:32:21.089 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:21.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:21.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:32:21.191 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:21.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:21.234 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:21.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:21.335 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:21.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:21.364 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:21.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:21.465 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:21.488 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:21.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:21.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:21.590 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:21.617 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:21.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:21.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:21.719 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:21.742 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:21.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:21.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:21.844 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:32:21.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:21.879 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:21.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:32:21.902 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 67 16 89657 89641 83.6106 52.9375 0.0363762 0.0119545 2024-09-13T14:32:21.981 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:22.017 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:22.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:22.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:22.119 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:32:22.149 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:22.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:22.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:32:22.251 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:22.289 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:22.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:22.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:22.391 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:32:22.421 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:22.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:22.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:32:22.523 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:22.553 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:22.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:22.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:22.654 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:22.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:22 smithi102 ceph-mon[110183]: pgmap v536: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 86 active+clean; 5.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 38 MiB/s wr, 604 op/s; 8185/255333 objects degraded (3.206%); 4695/255333 objects misplaced (1.839%); 9.9 MiB/s, 158 objects/s recovering 2024-09-13T14:32:22.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:22 smithi102 ceph-mon[115269]: pgmap v536: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 86 active+clean; 5.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 38 MiB/s wr, 604 op/s; 8185/255333 objects degraded (3.206%); 4695/255333 objects misplaced (1.839%); 9.9 MiB/s, 158 objects/s recovering 2024-09-13T14:32:22.694 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:22.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:22.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:22.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:22 smithi195 ceph-mon[104262]: pgmap v536: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 86 active+clean; 5.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 38 MiB/s wr, 604 op/s; 8185/255333 objects degraded (3.206%); 4695/255333 objects misplaced (1.839%); 9.9 MiB/s, 158 objects/s recovering 2024-09-13T14:32:22.796 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:22.835 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:22.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:22.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:22.902 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 68 16 90608 90592 83.255 59.4375 0.0369452 0.0120058 2024-09-13T14:32:22.936 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:22.968 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:22.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:22.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:23.070 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:23.109 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:23.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:23.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:23.211 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:23.234 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:23.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:23.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:23.336 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:23.367 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:23.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:23.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:23.468 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:23.510 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:23.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:23.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:23.611 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:23.643 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:23.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:23.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:23.744 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:32:23.773 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:23.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:23.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:32:23.875 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:23.902 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 69 16 91531 91515 82.8844 57.6875 0.0163958 0.0120597 2024-09-13T14:32:23.909 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:23.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:23.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:24.011 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:24.046 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:24.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:24.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:24.147 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:24.171 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:24.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:24.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:24.273 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:24.299 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:24.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:24.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:24.401 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:24.437 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:24.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:24.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:24.539 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:24.576 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:24.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:24.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:24.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:24 smithi102 ceph-mon[110183]: pgmap v537: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.5 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 62 MiB/s wr, 986 op/s; 8185/269241 objects degraded (3.040%); 4509/269241 objects misplaced (1.675%); 2.8 MiB/s, 45 objects/s recovering 2024-09-13T14:32:24.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:24 smithi102 ceph-mon[115269]: pgmap v537: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.5 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 62 MiB/s wr, 986 op/s; 8185/269241 objects degraded (3.040%); 4509/269241 objects misplaced (1.675%); 2.8 MiB/s, 45 objects/s recovering 2024-09-13T14:32:24.678 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:32:24.710 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:24.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:24.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:32:24.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:24 smithi195 ceph-mon[104262]: pgmap v537: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.5 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 62 MiB/s wr, 986 op/s; 8185/269241 objects degraded (3.040%); 4509/269241 objects misplaced (1.675%); 2.8 MiB/s, 45 objects/s recovering 2024-09-13T14:32:24.812 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:24.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:24.844 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:24.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:24.902 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 70 16 92501 92485 82.5664 60.625 0.0153204 0.0121064 2024-09-13T14:32:24.946 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:32:24.972 INFO:teuthology.orchestra.run.smithi102.stderr:no valid command found; 10 closest matches: 2024-09-13T14:32:24.973 INFO:teuthology.orchestra.run.smithi102.stderr:0 2024-09-13T14:32:24.973 INFO:teuthology.orchestra.run.smithi102.stderr:1 2024-09-13T14:32:24.973 INFO:teuthology.orchestra.run.smithi102.stderr:2 2024-09-13T14:32:24.973 INFO:teuthology.orchestra.run.smithi102.stderr:abort 2024-09-13T14:32:24.973 INFO:teuthology.orchestra.run.smithi102.stderr:assert 2024-09-13T14:32:24.973 INFO:teuthology.orchestra.run.smithi102.stderr:bench [] [] [] [] 2024-09-13T14:32:24.973 INFO:teuthology.orchestra.run.smithi102.stderr:bluefs debug_inject_read_zeros 2024-09-13T14:32:24.973 INFO:teuthology.orchestra.run.smithi102.stderr:bluefs files list 2024-09-13T14:32:24.973 INFO:teuthology.orchestra.run.smithi102.stderr:bluefs stats 2024-09-13T14:32:24.973 INFO:teuthology.orchestra.run.smithi102.stderr:bluestore allocator dump block 2024-09-13T14:32:24.974 INFO:teuthology.orchestra.run.smithi102.stderr:Error EINVAL: invalid command 2024-09-13T14:32:24.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:24.975 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:24.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:32:25.076 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:25.097 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:25.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:25.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:25.198 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:25.233 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:25.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:25.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:25.334 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:25.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:25.380 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:25.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:25.481 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:25.512 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:25.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:25.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:25.614 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:25.645 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:25.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:25.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:25.747 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:32:25.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:25.792 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:25.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:32:25.893 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:32:25.902 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 71 16 93232 93216 82.0469 45.6875 0.0355904 0.0121828 2024-09-13T14:32:25.923 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:25.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:25.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:32:26.025 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:26.057 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:26.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:26.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:26.159 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:32:26.196 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:26.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:26.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:32:26.298 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:26.335 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:26.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:26.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:26.437 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:32:26.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:26.476 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:26.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:32:26.577 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:32:26.604 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:26.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:26.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:32:26.661 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:32:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:32:26] "GET /metrics HTTP/1.1" 200 36547 "" "Prometheus/2.51.0" 2024-09-13T14:32:26.662 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:26 smithi102 ceph-mon[110183]: pgmap v538: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.5 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 62 MiB/s wr, 986 op/s; 8185/269241 objects degraded (3.040%); 4509/269241 objects misplaced (1.675%); 2.8 MiB/s, 45 objects/s recovering 2024-09-13T14:32:26.662 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:32:26.662 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:32:26.662 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:26 smithi102 ceph-mon[115269]: pgmap v538: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.5 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 62 MiB/s wr, 986 op/s; 8185/269241 objects degraded (3.040%); 4509/269241 objects misplaced (1.675%); 2.8 MiB/s, 45 objects/s recovering 2024-09-13T14:32:26.662 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:32:26.662 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:32:26.707 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:26.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:26 smithi195 ceph-mon[104262]: pgmap v538: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.5 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 62 MiB/s wr, 986 op/s; 8185/269241 objects degraded (3.040%); 4509/269241 objects misplaced (1.675%); 2.8 MiB/s, 45 objects/s recovering 2024-09-13T14:32:26.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:32:26.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:32:26.753 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:26.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:26.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:26.855 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:26.887 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:26.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:26.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:26.902 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 72 16 94105 94089 81.6651 54.5625 0.00979884 0.0122392 2024-09-13T14:32:26.989 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:32:27.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:27.023 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:27.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:32:27.125 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:27.162 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:27.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:27.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:27.264 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:27.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:27.298 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:27.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:27.399 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:27.435 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:27.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:27.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:27.537 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:27.572 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:27.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:27.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:27.674 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:32:27.706 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:27.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:27.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:32:27.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:27 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 8185/280218 objects degraded (2.921%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:27.809 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:32:27.841 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:27.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:27.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:32:27.902 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 73 16 95065 95049 81.3683 60 0.0205291 0.0122848 2024-09-13T14:32:27.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:27 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 8185/280218 objects degraded (2.921%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:27.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:27 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 8185/280218 objects degraded (2.921%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:27.942 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:27.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:27.978 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:27.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:28.079 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:28.118 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:28.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:28.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:28.220 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:28.257 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:28.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:28.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:28.346 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:32:28.346 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [], 2024-09-13T14:32:28.346 INFO:teuthology.orchestra.run.smithi102.stdout: "num_ops": 0 2024-09-13T14:32:28.346 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:32:28.359 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:32:28.388 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:28.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:28.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:32:28.490 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:28.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:28.538 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:28.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:28.639 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:28.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:28 smithi102 ceph-mon[110183]: pgmap v539: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.7 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 56 MiB/s wr, 890 op/s; 8185/280218 objects degraded (2.921%); 4326/280218 objects misplaced (1.544%); 2.3 MiB/s, 36 objects/s recovering 2024-09-13T14:32:28.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:28 smithi102 ceph-mon[115269]: pgmap v539: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.7 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 56 MiB/s wr, 890 op/s; 8185/280218 objects degraded (2.921%); 4326/280218 objects misplaced (1.544%); 2.3 MiB/s, 36 objects/s recovering 2024-09-13T14:32:28.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:28.685 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:28.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:28.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:28 smithi195 ceph-mon[104262]: pgmap v539: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.7 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 56 MiB/s wr, 890 op/s; 8185/280218 objects degraded (2.921%); 4326/280218 objects misplaced (1.544%); 2.3 MiB/s, 36 objects/s recovering 2024-09-13T14:32:28.750 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:32:28.787 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:32:28.817 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:28.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:28.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:32:28.902 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 74 16 96019 96003 81.0744 59.625 0.0109248 0.0123296 2024-09-13T14:32:28.920 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:32:28.953 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:28.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:28.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:32:29.055 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:29.092 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:29.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:29.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:29.194 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:29.229 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:29.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:29.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:29.331 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:32:29.347 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T14:32:29.374 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:29.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:29.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:32:29.476 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:32:29.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:29.523 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:29.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:32:29.624 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:32:29.656 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:29.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:29.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:32:29.761 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:29.791 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:29.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:29.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:29.893 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:29.902 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 75 16 97017 97001 80.825 62.375 0.0150412 0.012368 2024-09-13T14:32:29.927 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:29.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:29.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:29.978 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T14:32:29.978 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T14:32:29.979 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.4 'trim stale osdmaps' 2024-09-13T14:32:29.999 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:32:30.029 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:32:30.070 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:30.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:30.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:32:30.172 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:30.207 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:30.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:30.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:30.309 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:32:30.353 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:30.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:30.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:32:30.455 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:30.492 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:30.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:30.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:30.594 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:30.596 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:30 smithi102 ceph-mon[110183]: pgmap v540: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.7 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 50 MiB/s wr, 798 op/s; 8185/281967 objects degraded (2.903%); 4326/281967 objects misplaced (1.534%); 1.9 MiB/s, 30 objects/s recovering 2024-09-13T14:32:30.596 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:30 smithi102 ceph-mon[115269]: pgmap v540: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.7 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 50 MiB/s wr, 798 op/s; 8185/281967 objects degraded (2.903%); 4326/281967 objects misplaced (1.534%); 1.9 MiB/s, 30 objects/s recovering 2024-09-13T14:32:30.634 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:30.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:30.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:30.737 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:30.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:30 smithi195 ceph-mon[104262]: pgmap v540: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.7 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 50 MiB/s wr, 798 op/s; 8185/281967 objects degraded (2.903%); 4326/281967 objects misplaced (1.534%); 1.9 MiB/s, 30 objects/s recovering 2024-09-13T14:32:30.769 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:30.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:30.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:30.870 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:32:30.904 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 76 16 97992 97976 80.5633 60.9375 0.0206442 0.0124074 2024-09-13T14:32:30.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:30.910 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:30.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:32:31.011 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:32:31.044 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:31.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:31.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:32:31.146 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:31.176 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:31.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:31.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:31.278 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:31.314 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:31.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:31.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:31.416 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:31.446 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:31.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:31.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:31.548 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:31.583 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:31.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:31.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:31.685 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:31.722 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:31.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:31.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:31.824 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:31.860 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:31.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:31.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:31.903 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 77 16 98969 98953 80.3099 61.0625 0.00892449 0.0124455 2024-09-13T14:32:31.961 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:31.995 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:31.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:31.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:32.096 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:32:32.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:32.129 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:32.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:32:32.230 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:32.268 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:32.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:32.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:32.370 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:32.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:32.407 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:32.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:32.508 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:32.543 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:32.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:32.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:32.584 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:32 smithi102 ceph-mon[110183]: pgmap v541: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 54 MiB/s wr, 862 op/s; 8185/286410 objects degraded (2.858%); 4326/286410 objects misplaced (1.510%); 1.9 MiB/s, 30 objects/s recovering 2024-09-13T14:32:32.584 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:32 smithi102 ceph-mon[115269]: pgmap v541: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 54 MiB/s wr, 862 op/s; 8185/286410 objects degraded (2.858%); 4326/286410 objects misplaced (1.510%); 1.9 MiB/s, 30 objects/s recovering 2024-09-13T14:32:32.645 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:32.678 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:32.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:32.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:32.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:32 smithi195 ceph-mon[104262]: pgmap v541: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 54 MiB/s wr, 862 op/s; 8185/286410 objects degraded (2.858%); 4326/286410 objects misplaced (1.510%); 1.9 MiB/s, 30 objects/s recovering 2024-09-13T14:32:32.749 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:32:32.749 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:32:32.776 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:32:32.777 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:32:32.780 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:32:32.805 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:32:32.805 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:32:32.816 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:32.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:32.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:32:32.835 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:32:32.836 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:32:32.868 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:32:32.869 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:32:32.903 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 78 16 99943 99927 80.0607 60.875 0.0253941 0.0124856 2024-09-13T14:32:32.906 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:32:32.906 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:32:32.921 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:32.933 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:32:32.933 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:32:32.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:32.956 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:32.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:32.968 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:32:32.968 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:32:33.057 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:33.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:33.167 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:33.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:33.269 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:32:33.300 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:33.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:33.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:32:33.402 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:33.423 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:32:33.424 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:32:33.436 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:33.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:33.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:33.538 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:32:33.577 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:33.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:33.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:32:33.679 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:33.720 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:33.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:33.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:33.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:33 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3313148315' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:32:33.822 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:33.861 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:33.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:33.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:33.903 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 79 16 100871 100855 79.7814 58 0.0203918 0.0125293 2024-09-13T14:32:33.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:33 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3313148315' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:32:33.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:33 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3313148315' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:32:33.963 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:33.992 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:33.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:33.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:34.094 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:32:34.123 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:34.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:34.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:32:34.225 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:34.267 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:34.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:34.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:34.369 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:34.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:34.408 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:34.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:34.509 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:34.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:34.553 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:34.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:34.654 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:34.688 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:34.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:34.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:34.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:34 smithi195 ceph-mon[104262]: pgmap v542: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 6.0 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 72 MiB/s wr, 1.15k op/s; 8185/296613 objects degraded (2.759%); 4127/296613 objects misplaced (1.391%); 3.0 MiB/s, 47 objects/s recovering 2024-09-13T14:32:34.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:34 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 8185/296613 objects degraded (2.759%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:34.790 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:32:34.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:34.813 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:34.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:32:34.903 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:2024-09-13T14:32:34.903208+0000 min lat: 0.00118498 max lat: 12.1717 avg lat: 0.012579 2024-09-13T14:32:34.903 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:32:34.903 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 80 16 101742 101726 79.4646 54.4375 0.00853141 0.012579 2024-09-13T14:32:34.914 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:34.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:34 smithi102 ceph-mon[110183]: pgmap v542: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 6.0 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 72 MiB/s wr, 1.15k op/s; 8185/296613 objects degraded (2.759%); 4127/296613 objects misplaced (1.391%); 3.0 MiB/s, 47 objects/s recovering 2024-09-13T14:32:34.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:34 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 8185/296613 objects degraded (2.759%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:34.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:34 smithi102 ceph-mon[115269]: pgmap v542: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 6.0 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 72 MiB/s wr, 1.15k op/s; 8185/296613 objects degraded (2.759%); 4127/296613 objects misplaced (1.391%); 3.0 MiB/s, 47 objects/s recovering 2024-09-13T14:32:34.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:34 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 8185/296613 objects degraded (2.759%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:34.953 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:34.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:34.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:35.054 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:32:35.086 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:35.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:35.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:32:35.188 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:35.218 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:35.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:35.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:35.320 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:32:35.340 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:35.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:35.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:32:35.442 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:32:35.471 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:35.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:35.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:32:35.573 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:35.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:35.623 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:35.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:35.724 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:35.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:35.757 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:35.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:35.782 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:35 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:32:35.782 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:35 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:32:35.858 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:35.903 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 81 16 102576 102560 79.127 52.125 0.00967872 0.0126326 2024-09-13T14:32:35.907 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:35.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:35.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:35.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:35 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:32:36.009 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:36.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:36.066 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:36.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:36.167 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:36.222 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:36.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:36.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:36.325 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:36.360 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:36.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:36.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:36.462 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:36.491 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:36.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:36.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:36.593 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:36.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:36.654 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:36.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:36.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:36 smithi102 ceph-mon[110183]: pgmap v543: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 6.0 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 48 MiB/s wr, 760 op/s; 8185/296613 objects degraded (2.759%); 4127/296613 objects misplaced (1.391%); 2.0 MiB/s, 31 objects/s recovering 2024-09-13T14:32:36.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:36 smithi102 ceph-mon[115269]: pgmap v543: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 6.0 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 48 MiB/s wr, 760 op/s; 8185/296613 objects degraded (2.759%); 4127/296613 objects misplaced (1.391%); 2.0 MiB/s, 31 objects/s recovering 2024-09-13T14:32:36.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:32:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:32:36] "GET /metrics HTTP/1.1" 200 36547 "" "Prometheus/2.51.0" 2024-09-13T14:32:36.755 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:36.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:36.800 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:36.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:36.901 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:32:36.903 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 82 16 103458 103442 78.8342 55.125 0.0221271 0.0126801 2024-09-13T14:32:36.934 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:36.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:36.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:32:36.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:36 smithi195 ceph-mon[104262]: pgmap v543: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 6.0 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 48 MiB/s wr, 760 op/s; 8185/296613 objects degraded (2.759%); 4127/296613 objects misplaced (1.391%); 2.0 MiB/s, 31 objects/s recovering 2024-09-13T14:32:37.036 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:37.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:37.090 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:37.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:37.191 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:37.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:37.243 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:37.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:37.344 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:32:37.381 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:37.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:37.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:32:37.483 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:37.527 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:37.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:37.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:37.629 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:37.668 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:37.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:37.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:37.770 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:37.784 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:32:37.819 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:37.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:37.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:37.903 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 83 16 104322 104306 78.535 54 0.0133036 0.0127281 2024-09-13T14:32:37.920 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:32:37.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:37.954 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:37.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:32:38.055 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:38.092 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:38.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:38.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:38.194 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:38.233 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:38.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:38.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:38.336 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:32:38.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:38.377 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:38.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:32:38.478 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:38.523 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:38.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:38.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:38.625 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:38.661 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:38.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:38.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:38.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:38 smithi102 ceph-mon[110183]: pgmap v544: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 6.3 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 68 MiB/s wr, 1.09k op/s; 8185/308385 objects degraded (2.654%); 3888/308385 objects misplaced (1.261%); 3.2 MiB/s, 51 objects/s recovering 2024-09-13T14:32:38.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:38 smithi102 ceph-mon[115269]: pgmap v544: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 6.3 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 68 MiB/s wr, 1.09k op/s; 8185/308385 objects degraded (2.654%); 3888/308385 objects misplaced (1.261%); 3.2 MiB/s, 51 objects/s recovering 2024-09-13T14:32:38.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:38 smithi195 ceph-mon[104262]: pgmap v544: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 6.3 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 68 MiB/s wr, 1.09k op/s; 8185/308385 objects degraded (2.654%); 3888/308385 objects misplaced (1.261%); 3.2 MiB/s, 51 objects/s recovering 2024-09-13T14:32:38.763 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:38.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:38.814 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:38.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:38.903 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 84 16 105230 105214 78.2756 56.75 0.0137124 0.0127707 2024-09-13T14:32:38.915 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:38.963 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:38.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:38.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:38.981 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:32:39.065 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:39.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:39.109 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:39.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:39.210 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:32:39.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:39.246 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:39.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:32:39.347 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:39.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:39.379 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:39.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:39.480 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:39.513 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:39.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:39.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:39.543 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is set 2024-09-13T14:32:39.615 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:32:39.652 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:39.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:39.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:32:39.754 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:39.788 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:39.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:39.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:39.861 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:39 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1419724940' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:32:39.862 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:39 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:32:39.862 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:39 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 8185/308385 objects degraded (2.654%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:39.862 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:39 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1419724940' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:32:39.862 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:39 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:32:39.862 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:39 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 8185/308385 objects degraded (2.654%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:39.890 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:39.904 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 85 16 106112 106096 78.0029 55.125 0.0337166 0.0128144 2024-09-13T14:32:39.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:39.921 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:39.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:39.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:39 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1419724940' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:32:39.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:39 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:32:39.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:39 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 8185/308385 objects degraded (2.654%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:40.022 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:32:40.053 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:40.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:40.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:32:40.155 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:32:40.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:40.188 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:40.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:32:40.289 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:40.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:40.323 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:40.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:40.424 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:40.458 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:40.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:40.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:40.560 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:32:40.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:40.594 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:40.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:32:40.695 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:40.728 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:40.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:40.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:40.830 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:40.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:40.878 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:40.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:40.904 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 86 15 107025 107010 77.7601 57.125 0.0110399 0.0128547 2024-09-13T14:32:40.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:40 smithi102 ceph-mon[110183]: pgmap v545: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 6.3 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 52 MiB/s wr, 831 op/s; 8185/310161 objects degraded (2.639%); 3888/310161 objects misplaced (1.254%); 2.3 MiB/s, 36 objects/s recovering 2024-09-13T14:32:40.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:40 smithi102 ceph-mon[110183]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:32:40.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:40 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:32:40.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:40 smithi102 ceph-mon[110183]: osdmap e176: 8 total, 8 up, 6 in 2024-09-13T14:32:40.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:40 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:32:40.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:40 smithi102 ceph-mon[115269]: pgmap v545: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 6.3 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 52 MiB/s wr, 831 op/s; 8185/310161 objects degraded (2.639%); 3888/310161 objects misplaced (1.254%); 2.3 MiB/s, 36 objects/s recovering 2024-09-13T14:32:40.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:40 smithi102 ceph-mon[115269]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:32:40.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:40 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:32:40.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:40 smithi102 ceph-mon[115269]: osdmap e176: 8 total, 8 up, 6 in 2024-09-13T14:32:40.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:40 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:32:40.979 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:41.001 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:40 smithi195 ceph-mon[104262]: pgmap v545: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 6.3 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 52 MiB/s wr, 831 op/s; 8185/310161 objects degraded (2.639%); 3888/310161 objects misplaced (1.254%); 2.3 MiB/s, 36 objects/s recovering 2024-09-13T14:32:41.001 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:40 smithi195 ceph-mon[104262]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:32:41.001 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:40 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:32:41.001 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:40 smithi195 ceph-mon[104262]: osdmap e176: 8 total, 8 up, 6 in 2024-09-13T14:32:41.001 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:40 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:32:41.008 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:41.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:41.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:41.110 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:41.157 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:41.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:41.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:41.259 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:41.286 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:41.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:41.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:41.388 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:41.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:41.423 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:41.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:41.524 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:32:41.579 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:41.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:41.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:32:41.682 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:32:41.710 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:41.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:41.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:32:41.812 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:41.842 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:41.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:41.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:41.904 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 87 16 107935 107919 77.5193 56.8125 0.0223997 0.012895 2024-09-13T14:32:41.944 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:41.978 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:41.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:41.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:42.080 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:32:42.108 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:42.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:42.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:32:42.210 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:42.237 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:42.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:42.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:42.339 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:42.406 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:42.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:42.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:42.509 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:42.572 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:42.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:42.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:42.673 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:42.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:42.702 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:42.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:42.803 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:32:42.836 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:42.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:42.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:32:42.904 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 88 16 108749 108733 77.2165 50.875 0.0135907 0.0129457 2024-09-13T14:32:42.937 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:42.971 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:42.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:42.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:43.073 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:32:43.105 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:43.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:43.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:32:43.207 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:43.245 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:43.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:43.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:43.347 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:43.376 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:43.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:43.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:43.478 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:43.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:43 smithi195 ceph-mon[104262]: pgmap v547: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 6.4 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 58 MiB/s wr, 928 op/s; 8185/314274 objects degraded (2.604%); 3888/314274 objects misplaced (1.237%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:32:43.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:43 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:32:43.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:43 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:32:43.503 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:43.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:43.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:43.605 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:43.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:43.649 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:43.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:43.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:43 smithi102 ceph-mon[110183]: pgmap v547: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 6.4 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 58 MiB/s wr, 928 op/s; 8185/314274 objects degraded (2.604%); 3888/314274 objects misplaced (1.237%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:32:43.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:43 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:32:43.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:43 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:32:43.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:43 smithi102 ceph-mon[115269]: pgmap v547: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 6.4 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 58 MiB/s wr, 928 op/s; 8185/314274 objects degraded (2.604%); 3888/314274 objects misplaced (1.237%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:32:43.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:43 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:32:43.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:43 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:32:43.693 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:32:43.693 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:32:43.750 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:32:43.778 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:43.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:43.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:32:43.880 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:32:43.904 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 89 16 109675 109659 76.9991 57.875 0.0174959 0.0129825 2024-09-13T14:32:43.912 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:43.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:43.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:32:44.014 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:44.046 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:44.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:44.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:44.148 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:32:44.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:44.182 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:44.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:32:44.283 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:44.313 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:44.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:44.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:44.415 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:44.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:44 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 8185/314274 objects degraded (2.604%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:44.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:44 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 8185/314274 objects degraded (2.604%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:44.456 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:44.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:44.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:44.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:44 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 8185/314274 objects degraded (2.604%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:44.558 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:32:44.589 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:44.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:44.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:32:44.676 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T14:32:44.691 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:44.731 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:44.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:44.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:44.758 INFO:teuthology.orchestra.run.smithi102.stderr:no valid command found; 10 closest matches: 2024-09-13T14:32:44.758 INFO:teuthology.orchestra.run.smithi102.stderr:0 2024-09-13T14:32:44.758 INFO:teuthology.orchestra.run.smithi102.stderr:1 2024-09-13T14:32:44.758 INFO:teuthology.orchestra.run.smithi102.stderr:2 2024-09-13T14:32:44.758 INFO:teuthology.orchestra.run.smithi102.stderr:abort 2024-09-13T14:32:44.758 INFO:teuthology.orchestra.run.smithi102.stderr:assert 2024-09-13T14:32:44.759 INFO:teuthology.orchestra.run.smithi102.stderr:bench [] [] [] [] 2024-09-13T14:32:44.759 INFO:teuthology.orchestra.run.smithi102.stderr:bluefs debug_inject_read_zeros 2024-09-13T14:32:44.759 INFO:teuthology.orchestra.run.smithi102.stderr:bluefs files list 2024-09-13T14:32:44.759 INFO:teuthology.orchestra.run.smithi102.stderr:bluefs stats 2024-09-13T14:32:44.759 INFO:teuthology.orchestra.run.smithi102.stderr:bluestore allocator dump block 2024-09-13T14:32:44.759 INFO:teuthology.orchestra.run.smithi102.stderr:Error EINVAL: invalid command 2024-09-13T14:32:44.833 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:32:44.869 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:44.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:44.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:32:44.904 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Total time run: 90.007 2024-09-13T14:32:44.904 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Total writes made: 110621 2024-09-13T14:32:44.904 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Write size: 65536 2024-09-13T14:32:44.905 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Object size: 65536 2024-09-13T14:32:44.905 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Bandwidth (MB/sec): 76.8142 2024-09-13T14:32:44.905 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Stddev Bandwidth: 48.3894 2024-09-13T14:32:44.905 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Max bandwidth (MB/sec): 244.875 2024-09-13T14:32:44.905 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Min bandwidth (MB/sec): 0 2024-09-13T14:32:44.905 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Average IOPS: 1229 2024-09-13T14:32:44.905 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Stddev IOPS: 774.243 2024-09-13T14:32:44.905 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Max IOPS: 3918 2024-09-13T14:32:44.905 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Min IOPS: 0 2024-09-13T14:32:44.905 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Average Latency(s): 0.0130143 2024-09-13T14:32:44.905 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Stddev Latency(s): 0.136216 2024-09-13T14:32:44.906 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Max latency(s): 12.1717 2024-09-13T14:32:44.906 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Min latency(s): 0.00118498 2024-09-13T14:32:44.908 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Cleaning up (deleting benchmark objects) 2024-09-13T14:32:44.971 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:45.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:45.002 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:45.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:45.103 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:45.130 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:45.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:45.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:45.232 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:45.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:45.262 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:45.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:45.363 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:45.391 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:45.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:45.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:45.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:45 smithi102 ceph-mon[110183]: pgmap v548: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 6.6 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 56 MiB/s wr, 897 op/s; 8185/323535 objects degraded (2.530%); 3633/323535 objects misplaced (1.123%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T14:32:45.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:45 smithi102 ceph-mon[115269]: pgmap v548: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 6.6 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 56 MiB/s wr, 897 op/s; 8185/323535 objects degraded (2.530%); 3633/323535 objects misplaced (1.123%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T14:32:45.493 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:45.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:45 smithi195 ceph-mon[104262]: pgmap v548: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 6.6 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 56 MiB/s wr, 897 op/s; 8185/323535 objects degraded (2.530%); 3633/323535 objects misplaced (1.123%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T14:32:45.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:45.528 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:45.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:45.629 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:45.659 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:45.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:45.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:45.761 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:32:45.796 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:45.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:45.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:32:45.898 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:32:45.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:45.930 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:45.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:32:46.031 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:46.063 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:46.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:46.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:46.165 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:32:46.197 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:46.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:46.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:32:46.228 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T14:32:46.228 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T14:32:46.228 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.5 'trim stale osdmaps' 2024-09-13T14:32:46.300 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:46.338 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:46.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:46.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:46.440 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:46.455 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:32:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:32:46] "GET /metrics HTTP/1.1" 200 36550 "" "Prometheus/2.51.0" 2024-09-13T14:32:46.477 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:46.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:46.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:46.579 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:46.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:46.614 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:46.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:46.715 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:32:46.746 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:46.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:46.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:32:46.777 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:32:46.778 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:32:46.805 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:32:46.806 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:32:46.837 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:32:46.837 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:32:46.848 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:46.860 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:32:46.860 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:32:46.877 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:46.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:46.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:46.896 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:32:46.896 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:32:46.930 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:32:46.930 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:32:46.957 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:32:46.957 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:32:46.978 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:32:46.981 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:32:46.982 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:32:47.008 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:47.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:47.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:32:47.110 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:47.145 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:47.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:47.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:47.247 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:47.293 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:47.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:47.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:47.309 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:47 smithi102 ceph-mon[110183]: pgmap v549: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 6.6 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 56 MiB/s wr, 897 op/s; 8185/323535 objects degraded (2.530%); 3633/323535 objects misplaced (1.123%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T14:32:47.309 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:47 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2405502047' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:32:47.310 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:47 smithi102 ceph-mon[115269]: pgmap v549: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 6.6 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 56 MiB/s wr, 897 op/s; 8185/323535 objects degraded (2.530%); 3633/323535 objects misplaced (1.123%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T14:32:47.310 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:47 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2405502047' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:32:47.395 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:32:47.441 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:47.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:47.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:32:47.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:47 smithi195 ceph-mon[104262]: pgmap v549: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 6.6 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 56 MiB/s wr, 897 op/s; 8185/323535 objects degraded (2.530%); 3633/323535 objects misplaced (1.123%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T14:32:47.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:47 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2405502047' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:32:47.511 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:32:47.543 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:47.576 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:47.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:47.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:47.678 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:32:47.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:47.713 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:47.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:32:47.814 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:47.842 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:47.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:47.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:47.944 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:32:47.975 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:47.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:47.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:32:48.077 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:48.107 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:48.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:48.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:48.120 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:32:48.154 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:32:48.209 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:32:48.249 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:48.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:48.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:32:48.351 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:48.385 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:48.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:48.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:48.486 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:32:48.512 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:48.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:48.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:32:48.614 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:48.655 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:48.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:48.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:48.758 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:48.779 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:48.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:48.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:48.881 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:32:48.911 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:48.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:48.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:32:49.013 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:49.050 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:49.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:49.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:49.152 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:49.180 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:49.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:49.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:49.282 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:49.328 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:49.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:49.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:49.430 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:49.458 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:49.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:49.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:49.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:49 smithi195 ceph-mon[104262]: pgmap v550: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 6.6 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 102 B/s rd, 47 MiB/s wr, 914 op/s; 8185/326370 objects degraded (2.508%); 3392/326370 objects misplaced (1.039%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T14:32:49.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:49 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 8185/326370 objects degraded (2.508%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:49.551 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:49 smithi102 ceph-mon[110183]: pgmap v550: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 6.6 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 102 B/s rd, 47 MiB/s wr, 914 op/s; 8185/326370 objects degraded (2.508%); 3392/326370 objects misplaced (1.039%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T14:32:49.552 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:49 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 8185/326370 objects degraded (2.508%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:49.552 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:49 smithi102 ceph-mon[115269]: pgmap v550: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 6.6 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 102 B/s rd, 47 MiB/s wr, 914 op/s; 8185/326370 objects degraded (2.508%); 3392/326370 objects misplaced (1.039%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T14:32:49.552 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:49 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 8185/326370 objects degraded (2.508%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:49.560 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:49.594 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:49.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:49.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:49.696 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:49.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:49.725 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:49.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:49.826 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:49.860 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:49.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:49.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:49.962 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:49.990 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:49.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:49.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:50.092 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:32:50.136 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:50.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:50.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:32:50.157 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:32:50.238 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:50.282 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:50.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:50.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:50.384 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:32:50.430 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:50.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:50.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:32:50.532 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:32:50.568 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:50.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:50.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:32:50.669 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:32:50.699 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:50.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:50.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:32:50.805 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:50.830 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:50.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:50.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:50.932 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:50.960 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:50.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:50.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:51.062 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:51.092 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:51.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:51.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:51.194 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:32:51.231 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:51.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:51.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:32:51.333 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:51.371 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:51.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:51.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:51.472 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:32:51.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:51 smithi195 ceph-mon[104262]: pgmap v551: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 6.6 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 102 B/s rd, 45 MiB/s wr, 913 op/s; 8185/326322 objects degraded (2.508%); 3392/326322 objects misplaced (1.039%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T14:32:51.502 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:51.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:51.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:32:51.604 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:32:51.632 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:51.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:51.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:32:51.654 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:51 smithi102 ceph-mon[110183]: pgmap v551: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 6.6 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 102 B/s rd, 45 MiB/s wr, 913 op/s; 8185/326322 objects degraded (2.508%); 3392/326322 objects misplaced (1.039%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T14:32:51.654 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:51 smithi102 ceph-mon[115269]: pgmap v551: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 6.6 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 102 B/s rd, 45 MiB/s wr, 913 op/s; 8185/326322 objects degraded (2.508%); 3392/326322 objects misplaced (1.039%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T14:32:51.690 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:32:51.698 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:32:51.735 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:51.764 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:51.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:51.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:51.865 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:51.903 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:51.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:51.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:52.005 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:32:52.044 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:52.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:52.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:32:52.146 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:32:52.178 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:52.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:52.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:32:52.279 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:32:52.314 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:52.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:52.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:32:52.416 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:32:52.457 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:52.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:52.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:32:52.559 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:52.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:52.592 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:52.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:52.693 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:52.723 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:52.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:52.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:52.824 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:32:52.854 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:52.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:52.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:32:52.957 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:52.991 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:52.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:52.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:53.093 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:53.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:53.122 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:53.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:53.223 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:32:53.262 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:53.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:53.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:32:53.363 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:53.398 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:53.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:53.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:53.441 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:32:53.442 INFO:teuthology.orchestra.run.smithi102.stdout: "size": 20, 2024-09-13T14:32:53.442 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 600, 2024-09-13T14:32:53.442 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [] 2024-09-13T14:32:53.442 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:32:53.453 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:53 smithi102 ceph-mon[115269]: pgmap v552: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 6.5 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 86 B/s rd, 38 MiB/s wr, 901 op/s; 8185/321774 objects degraded (2.544%); 3392/321774 objects misplaced (1.054%); 2.6 MiB/s, 41 objects/s recovering 2024-09-13T14:32:53.457 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:53 smithi102 ceph-mon[110183]: pgmap v552: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 6.5 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 86 B/s rd, 38 MiB/s wr, 901 op/s; 8185/321774 objects degraded (2.544%); 3392/321774 objects misplaced (1.054%); 2.6 MiB/s, 41 objects/s recovering 2024-09-13T14:32:53.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:53 smithi195 ceph-mon[104262]: pgmap v552: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 6.5 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 86 B/s rd, 38 MiB/s wr, 901 op/s; 8185/321774 objects degraded (2.544%); 3392/321774 objects misplaced (1.054%); 2.6 MiB/s, 41 objects/s recovering 2024-09-13T14:32:53.502 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:32:53.545 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:53.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:53.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:32:53.647 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:53.677 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:53.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:53.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:53.779 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:53.815 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:53.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:53.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:53.917 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:53.957 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:53.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:53.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:54.059 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:54.101 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:54.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:54.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:54.203 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:32:54.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:54.255 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:54.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:32:54.356 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:54.391 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:54.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:54.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:54.400 INFO:teuthology.orchestra.run.smithi102.stderr:no valid command found; 10 closest matches: 2024-09-13T14:32:54.400 INFO:teuthology.orchestra.run.smithi102.stderr:0 2024-09-13T14:32:54.400 INFO:teuthology.orchestra.run.smithi102.stderr:1 2024-09-13T14:32:54.400 INFO:teuthology.orchestra.run.smithi102.stderr:2 2024-09-13T14:32:54.400 INFO:teuthology.orchestra.run.smithi102.stderr:abort 2024-09-13T14:32:54.400 INFO:teuthology.orchestra.run.smithi102.stderr:assert 2024-09-13T14:32:54.400 INFO:teuthology.orchestra.run.smithi102.stderr:bench [] [] [] [] 2024-09-13T14:32:54.400 INFO:teuthology.orchestra.run.smithi102.stderr:bluefs debug_inject_read_zeros 2024-09-13T14:32:54.400 INFO:teuthology.orchestra.run.smithi102.stderr:bluefs files list 2024-09-13T14:32:54.400 INFO:teuthology.orchestra.run.smithi102.stderr:bluefs stats 2024-09-13T14:32:54.401 INFO:teuthology.orchestra.run.smithi102.stderr:bluestore allocator dump block 2024-09-13T14:32:54.401 INFO:teuthology.orchestra.run.smithi102.stderr:Error EINVAL: invalid command 2024-09-13T14:32:54.493 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:54.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:54 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 8185/321774 objects degraded (2.544%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:54.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:54.535 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:54.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:54.636 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:32:54.664 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:54.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:54.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:32:54.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:54 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 8185/321774 objects degraded (2.544%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:54.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:54 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 8185/321774 objects degraded (2.544%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:54.767 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:54.796 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:54.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:54.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:54.897 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:32:54.928 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:54.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:54.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:32:55.030 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:55.057 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:55.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:55.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:55.159 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:32:55.184 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:55.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:55.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:32:55.286 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:55.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:55.315 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:55.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:55.416 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:32:55.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:55 smithi102 ceph-mon[115269]: pgmap v553: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 6.3 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 85 B/s rd, 31 MiB/s wr, 1.05k op/s; 8185/311922 objects degraded (2.624%); 3133/311922 objects misplaced (1.004%); 3.9 MiB/s, 62 objects/s recovering 2024-09-13T14:32:55.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:55 smithi102 ceph-mon[110183]: pgmap v553: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 6.3 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 85 B/s rd, 31 MiB/s wr, 1.05k op/s; 8185/311922 objects degraded (2.624%); 3133/311922 objects misplaced (1.004%); 3.9 MiB/s, 62 objects/s recovering 2024-09-13T14:32:55.421 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T14:32:55.422 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T14:32:55.422 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.6 'trim stale osdmaps' 2024-09-13T14:32:55.452 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:55.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:55.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:32:55.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:55 smithi195 ceph-mon[104262]: pgmap v553: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 6.3 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 85 B/s rd, 31 MiB/s wr, 1.05k op/s; 8185/311922 objects degraded (2.624%); 3133/311922 objects misplaced (1.004%); 3.9 MiB/s, 62 objects/s recovering 2024-09-13T14:32:55.554 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:55.589 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:55.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:55.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:55.691 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:55.728 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:55.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:55.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:55.830 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:55.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:55.866 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:55.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:55.967 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:56.005 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:56.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:56.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:56.008 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:32:56.008 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:32:56.008 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:32:56.008 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:32:56.008 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:32:56.008 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:32:56.013 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:32:56.014 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:32:56.043 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:32:56.044 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:32:56.078 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:32:56.078 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:32:56.106 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:32:56.118 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:32:56.118 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:32:56.135 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:56.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:56.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:32:56.151 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:32:56.151 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:32:56.238 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:32:56.272 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:56.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:56.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:32:56.351 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:32:56.352 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:56 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1869361963' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:32:56.352 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:32:56.352 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:56 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1869361963' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:32:56.374 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:32:56.410 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:56.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:56.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:32:56.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:32:56.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:56 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1869361963' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:32:56.512 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:56.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:56.546 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:56.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:56.647 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:56.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:32:56 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:32:56] "GET /metrics HTTP/1.1" 200 36557 "" "Prometheus/2.51.0" 2024-09-13T14:32:56.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:56.683 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:56.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:56.784 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:56.831 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:56.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:56.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:56.933 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:56.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:56.973 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:56.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:57.074 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:57.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:57.104 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:57.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:57.205 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:57.239 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:57.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:57.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:57.341 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:32:57.366 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:57.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:57.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:32:57.404 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T14:32:57.469 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:32:57.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:57 smithi195 ceph-mon[104262]: pgmap v554: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 6.3 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 85 B/s rd, 15 MiB/s wr, 789 op/s; 8185/311922 objects degraded (2.624%); 3133/311922 objects misplaced (1.004%); 2.6 MiB/s, 41 objects/s recovering 2024-09-13T14:32:57.506 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:57.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:57.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:32:57.608 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:32:57.642 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:57.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:57.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:32:57.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:57 smithi102 ceph-mon[110183]: pgmap v554: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 6.3 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 85 B/s rd, 15 MiB/s wr, 789 op/s; 8185/311922 objects degraded (2.624%); 3133/311922 objects misplaced (1.004%); 2.6 MiB/s, 41 objects/s recovering 2024-09-13T14:32:57.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:57 smithi102 ceph-mon[115269]: pgmap v554: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 6.3 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 85 B/s rd, 15 MiB/s wr, 789 op/s; 8185/311922 objects degraded (2.624%); 3133/311922 objects misplaced (1.004%); 2.6 MiB/s, 41 objects/s recovering 2024-09-13T14:32:57.744 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:57.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:57.778 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:57.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:57.878 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:57.920 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:57.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:57.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:58.022 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:58.058 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:58.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:58.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:58.160 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:58.189 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:58.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:58.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:58.276 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:32:58.291 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:32:58.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:58.340 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:58.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:32:58.441 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:32:58.474 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:58.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:58.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:32:58.576 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:58.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:58.615 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:58.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:58.716 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:32:58.759 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:58.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:58.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:32:58.835 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:32:58.861 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:32:58.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:58.898 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:58.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:32:58.999 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:32:59.025 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:59.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:59.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:32:59.127 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:32:59.163 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:59.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:59.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:32:59.265 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:59.298 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:59.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:59.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:59.400 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:32:59.444 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:59.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:59.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:32:59.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:59 smithi195 ceph-mon[104262]: pgmap v555: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 6.1 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 85 B/s rd, 15 MiB/s wr, 1.15k op/s; 8185/299031 objects degraded (2.737%); 2851/299031 objects misplaced (0.953%); 4.1 MiB/s, 65 objects/s recovering 2024-09-13T14:32:59.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:32:59 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 8185/299031 objects degraded (2.737%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:59.547 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:59.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:59.569 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:59.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:32:59.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:59 smithi102 ceph-mon[110183]: pgmap v555: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 6.1 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 85 B/s rd, 15 MiB/s wr, 1.15k op/s; 8185/299031 objects degraded (2.737%); 2851/299031 objects misplaced (0.953%); 4.1 MiB/s, 65 objects/s recovering 2024-09-13T14:32:59.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:32:59 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 8185/299031 objects degraded (2.737%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:59.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:59 smithi102 ceph-mon[115269]: pgmap v555: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 6.1 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 85 B/s rd, 15 MiB/s wr, 1.15k op/s; 8185/299031 objects degraded (2.737%); 2851/299031 objects misplaced (0.953%); 4.1 MiB/s, 65 objects/s recovering 2024-09-13T14:32:59.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:32:59 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 8185/299031 objects degraded (2.737%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:32:59.670 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:59.709 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:59.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:59.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:59.805 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:32:59.812 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:32:59.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:59.847 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:32:59.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:32:59.948 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:32:59.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:32:59.985 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:32:59.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:00.086 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:00.122 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:00.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:00.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:00.225 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:00.254 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:00.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:00.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:00.356 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:00.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:00.393 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:00.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:00.494 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:00.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:00.529 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:00.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:00.630 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:00.657 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:00.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:00.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:00.759 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:00.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:00.803 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:00.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:00.904 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:00.934 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:00.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:00.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:01.036 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:01.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:01.077 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:01.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:01.179 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:01.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:01.210 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:01.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:01.311 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:01.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:01.352 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:01.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:01.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:01 smithi102 ceph-mon[115269]: pgmap v556: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 6.1 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1.5 MiB/s wr, 855 op/s; 8185/297270 objects degraded (2.753%); 2851/297270 objects misplaced (0.959%); 2.8 MiB/s, 45 objects/s recovering 2024-09-13T14:33:01.405 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:01 smithi102 ceph-mon[110183]: pgmap v556: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 6.1 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1.5 MiB/s wr, 855 op/s; 8185/297270 objects degraded (2.753%); 2851/297270 objects misplaced (0.959%); 2.8 MiB/s, 45 objects/s recovering 2024-09-13T14:33:01.453 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:01.489 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:01.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:01.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:01.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:01 smithi195 ceph-mon[104262]: pgmap v556: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 6.1 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1.5 MiB/s wr, 855 op/s; 8185/297270 objects degraded (2.753%); 2851/297270 objects misplaced (0.959%); 2.8 MiB/s, 45 objects/s recovering 2024-09-13T14:33:01.591 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:01.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:01.627 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:01.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:01.728 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:01.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:01.772 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:01.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:01.873 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:33:01.915 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:01.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:01.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:33:02.017 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:02.044 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:02.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:02.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:02.146 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:02.179 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:02.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:02.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:02.248 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is unset 2024-09-13T14:33:02.281 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:02.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:02.313 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:02.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:02.414 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:33:02.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:02.454 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:02.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:33:02.555 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:02.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:02.589 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:02.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:02.594 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:02 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/777732152' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:33:02.595 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:02 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/777732152' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:33:02.690 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:02.727 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:02.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:02.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:02.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:02 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/777732152' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:33:02.830 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:02.870 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:02.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:02.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:02.972 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:33:03.012 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:03.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:03.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:33:03.114 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:33:03.154 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:03.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:03.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:33:03.256 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:03.294 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:03.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:03.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:03.397 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:33:03.430 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:03.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:03.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:33:03.532 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:03.561 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:03 smithi102 ceph-mon[110183]: pgmap v557: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 6.0 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 927 op/s; 8185/292914 objects degraded (2.794%); 2851/292914 objects misplaced (0.973%); 2.8 MiB/s, 45 objects/s recovering 2024-09-13T14:33:03.561 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:03 smithi102 ceph-mon[110183]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:33:03.562 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:03 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/777732152' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:33:03.562 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:03 smithi102 ceph-mon[110183]: osdmap e177: 8 total, 8 up, 6 in 2024-09-13T14:33:03.562 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:03 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:33:03.562 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:03 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:33:03.562 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:03 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:33:03.562 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:03 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:33:03.562 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:03 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:33:03.562 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:03 smithi102 ceph-mon[115269]: pgmap v557: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 6.0 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 927 op/s; 8185/292914 objects degraded (2.794%); 2851/292914 objects misplaced (0.973%); 2.8 MiB/s, 45 objects/s recovering 2024-09-13T14:33:03.562 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:03 smithi102 ceph-mon[115269]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:33:03.562 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:03 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/777732152' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:33:03.563 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:03 smithi102 ceph-mon[115269]: osdmap e177: 8 total, 8 up, 6 in 2024-09-13T14:33:03.563 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:03 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:33:03.563 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:03 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:33:03.563 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:03 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:33:03.563 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:03 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:33:03.563 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:03 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:33:03.566 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:03.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:03.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:03.668 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:33:03.711 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:03.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:03.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:33:03.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:03 smithi195 ceph-mon[104262]: pgmap v557: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 6.0 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 927 op/s; 8185/292914 objects degraded (2.794%); 2851/292914 objects misplaced (0.973%); 2.8 MiB/s, 45 objects/s recovering 2024-09-13T14:33:03.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:03 smithi195 ceph-mon[104262]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:33:03.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:03 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/777732152' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:33:03.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:03 smithi195 ceph-mon[104262]: osdmap e177: 8 total, 8 up, 6 in 2024-09-13T14:33:03.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:03 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:33:03.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:03 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:33:03.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:03 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:33:03.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:03 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:33:03.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:03 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:33:03.813 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:03.985 INFO:teuthology.orchestra.run.smithi102.stderr:no valid command found; 10 closest matches: 2024-09-13T14:33:03.985 INFO:teuthology.orchestra.run.smithi102.stderr:0 2024-09-13T14:33:03.986 INFO:teuthology.orchestra.run.smithi102.stderr:1 2024-09-13T14:33:03.986 INFO:teuthology.orchestra.run.smithi102.stderr:2 2024-09-13T14:33:03.986 INFO:teuthology.orchestra.run.smithi102.stderr:abort 2024-09-13T14:33:03.986 INFO:teuthology.orchestra.run.smithi102.stderr:assert 2024-09-13T14:33:03.986 INFO:teuthology.orchestra.run.smithi102.stderr:bench [] [] [] [] 2024-09-13T14:33:03.986 INFO:teuthology.orchestra.run.smithi102.stderr:bluefs debug_inject_read_zeros 2024-09-13T14:33:03.986 INFO:teuthology.orchestra.run.smithi102.stderr:bluefs files list 2024-09-13T14:33:03.986 INFO:teuthology.orchestra.run.smithi102.stderr:bluefs stats 2024-09-13T14:33:03.986 INFO:teuthology.orchestra.run.smithi102.stderr:bluestore allocator dump block 2024-09-13T14:33:03.986 INFO:teuthology.orchestra.run.smithi102.stderr:Error EINVAL: invalid command 2024-09-13T14:33:04.014 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:04.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:04.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:04.116 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:04.151 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:04.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:04.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:04.253 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:04.283 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:04.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:04.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:04.385 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:04.419 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:04.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:04.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:04.521 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:04.549 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:04.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:04.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:04.651 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:04.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:04 smithi102 ceph-mon[110183]: pgmap v559: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 956 op/s; 8185/283221 objects degraded (2.890%); 2548/283221 objects misplaced (0.900%); 3.7 MiB/s, 58 objects/s recovering 2024-09-13T14:33:04.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:04 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 8185/292914 objects degraded (2.794%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:33:04.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:04 smithi102 ceph-mon[115269]: pgmap v559: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 956 op/s; 8185/283221 objects degraded (2.890%); 2548/283221 objects misplaced (0.900%); 3.7 MiB/s, 58 objects/s recovering 2024-09-13T14:33:04.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:04 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 8185/292914 objects degraded (2.794%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:33:04.685 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:04.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:04.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:04.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:04 smithi195 ceph-mon[104262]: pgmap v559: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 956 op/s; 8185/283221 objects degraded (2.890%); 2548/283221 objects misplaced (0.900%); 3.7 MiB/s, 58 objects/s recovering 2024-09-13T14:33:04.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:04 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 8185/292914 objects degraded (2.794%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:33:04.787 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:04.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:04.829 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:04.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:04.931 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:04.964 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:04.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:04.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:05.066 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:05.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:05.106 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:05.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:05.208 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:05.234 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:05.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:05.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:05.320 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:33:05.336 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:05.372 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:05.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:05.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:05.474 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:05.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:05.506 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:05.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:05.607 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:05.637 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:05.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:05.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:05.739 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:33:05.779 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:05.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:05.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:33:05.881 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:05.907 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:05.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:05.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:06.009 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:06.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:06.043 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:06.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:06.144 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:33:06.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:06.185 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:06.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:33:06.286 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:06.316 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:06.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:06.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:06.339 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:33:06.369 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T14:33:06.370 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T14:33:06.370 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.7 'trim stale osdmaps' 2024-09-13T14:33:06.418 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:06.488 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:06.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:06.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:06.590 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:33:06.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:06.625 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:06.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:33:06.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:06 smithi102 ceph-mon[110183]: pgmap v560: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 956 op/s; 8185/283221 objects degraded (2.890%); 2548/283221 objects misplaced (0.900%); 3.7 MiB/s, 58 objects/s recovering 2024-09-13T14:33:06.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:06 smithi102 ceph-mon[115269]: pgmap v560: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 956 op/s; 8185/283221 objects degraded (2.890%); 2548/283221 objects misplaced (0.900%); 3.7 MiB/s, 58 objects/s recovering 2024-09-13T14:33:06.669 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:33:06 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:33:06] "GET /metrics HTTP/1.1" 200 36557 "" "Prometheus/2.51.0" 2024-09-13T14:33:06.726 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:06.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:06 smithi195 ceph-mon[104262]: pgmap v560: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 956 op/s; 8185/283221 objects degraded (2.890%); 2548/283221 objects misplaced (0.900%); 3.7 MiB/s, 58 objects/s recovering 2024-09-13T14:33:06.752 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:06.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:06.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:06.855 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:06.894 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:06.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:06.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:06.995 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:07.042 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:07.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:07.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:07.144 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:33:07.180 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:07.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:07.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:33:07.282 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:07.324 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:07.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:07.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:07.430 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:07.461 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:07.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:07.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:07.563 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:07.590 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:07.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:07.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:07.691 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:07.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:07.718 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:07.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:07.819 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:07.849 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:07.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:07.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:07.951 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:07.980 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:07.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:07.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:08.082 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:08.111 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:08.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:08.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:08.213 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:08.250 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:08.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:08.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:08.352 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:33:08.401 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:08.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:08.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:33:08.504 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:08.536 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:08.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:08.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:08.638 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:08.665 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:08.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:08.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:08.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:08 smithi102 ceph-mon[110183]: pgmap v561: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 943 op/s; 8185/270711 objects degraded (3.024%); 2229/270711 objects misplaced (0.823%); 3.9 MiB/s, 62 objects/s recovering 2024-09-13T14:33:08.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:08 smithi102 ceph-mon[115269]: pgmap v561: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 943 op/s; 8185/270711 objects degraded (3.024%); 2229/270711 objects misplaced (0.823%); 3.9 MiB/s, 62 objects/s recovering 2024-09-13T14:33:08.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:08 smithi195 ceph-mon[104262]: pgmap v561: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 943 op/s; 8185/270711 objects degraded (3.024%); 2229/270711 objects misplaced (0.823%); 3.9 MiB/s, 62 objects/s recovering 2024-09-13T14:33:08.767 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:08.796 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:08.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:08.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:08.898 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:33:08.938 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:08.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:08.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:33:09.039 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:09.075 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:09.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:09.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:09.177 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:09.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:09.210 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:09.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:09.311 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:09.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:09.342 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:09.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:09.443 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:09.479 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:09.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:09.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:09.581 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:09.618 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:09.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:09.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:09.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:09 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 8185/270711 objects degraded (3.024%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:09.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:09 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 8185/270711 objects degraded (3.024%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:09.720 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:33:09.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:09 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 8185/270711 objects degraded (3.024%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:09.755 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:09.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:09.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:33:09.857 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:09.888 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:09.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:09.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:09.989 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:33:10.034 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:10.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:10.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:33:10.136 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:10.166 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:10.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:10.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:10.268 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:10.291 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:10.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:10.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:10.393 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:10.433 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:10.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:10.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:10.498 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:33:10.498 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:33:10.535 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:33:10.567 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:10.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:10.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:33:10.668 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:10.699 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:10.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:10.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:10.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:10 smithi195 ceph-mon[104262]: pgmap v562: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 946 op/s; 8185/268875 objects degraded (3.044%); 2229/268875 objects misplaced (0.829%); 3.9 MiB/s, 62 objects/s recovering 2024-09-13T14:33:10.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:33:10.800 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:10.827 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:10.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:10.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:10.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:10 smithi102 ceph-mon[115269]: pgmap v562: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 946 op/s; 8185/268875 objects degraded (3.044%); 2229/268875 objects misplaced (0.829%); 3.9 MiB/s, 62 objects/s recovering 2024-09-13T14:33:10.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:33:10.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:10 smithi102 ceph-mon[110183]: pgmap v562: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 946 op/s; 8185/268875 objects degraded (3.044%); 2229/268875 objects misplaced (0.829%); 3.9 MiB/s, 62 objects/s recovering 2024-09-13T14:33:10.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:33:10.928 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:10.968 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:10.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:10.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:11.070 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:11.106 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:11.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:11.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:11.208 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:33:11.243 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:11.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:11.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:33:11.345 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:11.384 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:11.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:11.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:11.486 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:11.515 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:11.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:11.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:11.617 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:11.652 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:11.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:11.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:11.754 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:11.782 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:11.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:11.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:11.884 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:11.912 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:11.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:11.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:12.013 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:12.040 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:12.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:12.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:12.142 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:12.178 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:12.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:12.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:12.280 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:12.313 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:12.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:12.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:12.415 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:12.457 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:12.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:12.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:12.559 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:12.585 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:12.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:12.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:12.687 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:12.725 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:12.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:12.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:12.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:12 smithi195 ceph-mon[104262]: pgmap v563: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.4 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 938 op/s; 8185/264759 objects degraded (3.091%); 2229/264759 objects misplaced (0.842%); 3.9 MiB/s, 62 objects/s recovering 2024-09-13T14:33:12.828 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:33:12.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:12.868 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:12.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:33:12.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:12 smithi102 ceph-mon[110183]: pgmap v563: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.4 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 938 op/s; 8185/264759 objects degraded (3.091%); 2229/264759 objects misplaced (0.842%); 3.9 MiB/s, 62 objects/s recovering 2024-09-13T14:33:12.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:12 smithi102 ceph-mon[115269]: pgmap v563: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.4 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 938 op/s; 8185/264759 objects degraded (3.091%); 2229/264759 objects misplaced (0.842%); 3.9 MiB/s, 62 objects/s recovering 2024-09-13T14:33:12.969 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:12.996 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:12.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:12.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:13.028 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T14:33:13.099 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:13.152 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:13.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:13.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:13.256 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:13.292 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:13.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:13.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:13.394 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:13.422 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:13.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:13.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:13.524 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:13.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:13.552 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:13.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:13.653 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:33:13.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:13.684 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:13.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:33:13.785 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:33:13.801 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:33:13.820 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:13.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:13.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:33:13.922 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:13.954 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:13.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:13.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:14.056 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:14.092 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:14.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:14.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:14.194 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:14.220 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:14.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:14.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:14.322 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:14.355 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:14.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:14.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:14.457 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:14.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:14.489 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:14.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:14.530 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:14 smithi102 ceph-mon[110183]: pgmap v564: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.2 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 886 op/s; 8185/255510 objects degraded (3.203%); 1896/255510 objects misplaced (0.742%); 3.7 MiB/s, 58 objects/s recovering 2024-09-13T14:33:14.530 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:14 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 8185/255510 objects degraded (3.203%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:14.530 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:14 smithi102 ceph-mon[115269]: pgmap v564: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.2 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 886 op/s; 8185/255510 objects degraded (3.203%); 1896/255510 objects misplaced (0.742%); 3.7 MiB/s, 58 objects/s recovering 2024-09-13T14:33:14.530 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:14 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 8185/255510 objects degraded (3.203%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:14.591 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:14.618 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:14.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:14.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:14.720 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:14.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:14 smithi195 ceph-mon[104262]: pgmap v564: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.2 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 886 op/s; 8185/255510 objects degraded (3.203%); 1896/255510 objects misplaced (0.742%); 3.7 MiB/s, 58 objects/s recovering 2024-09-13T14:33:14.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:14 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 8185/255510 objects degraded (3.203%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:14.750 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:14.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:14.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:14.852 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:14.887 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:14.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:14.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:14.989 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:33:15.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:15.023 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:15.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:33:15.124 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:15.153 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:15.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:15.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:15.229 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:33:15.232 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:33:15.256 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:15.291 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:15.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:15.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:15.393 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:15.428 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:15.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:15.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:15.508 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:33:15.509 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:33:15.529 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:15.531 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:33:15.531 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:33:15.553 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:33:15.554 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:33:15.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:15.568 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:15.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:15.578 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:33:15.578 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:33:15.615 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:33:15.615 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:33:15.646 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:33:15.646 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:33:15.669 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:33:15.676 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:33:15.676 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:33:15.708 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:33:15.709 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:33:15.709 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:15.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:15.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:33:15.811 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:15.849 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:15.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:15.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:15.952 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:15.980 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:15.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:15.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:16.082 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:16.115 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:16.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:16.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:16.217 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:16.246 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:16.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:16.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:16.348 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:16.376 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:16.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:16.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:16.478 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:16.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:16.505 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:16.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:16.606 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:16.635 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:16.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:16.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:16.737 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:33:16.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:16 smithi195 ceph-mon[104262]: pgmap v565: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.2 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 769 op/s; 8185/255510 objects degraded (3.203%); 1896/255510 objects misplaced (0.742%); 3.4 MiB/s, 54 objects/s recovering 2024-09-13T14:33:16.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:16 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/905058768' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:33:16.768 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:16.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:16.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:33:16.870 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:33:16.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:16.906 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:16.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:33:16.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:16 smithi102 ceph-mon[110183]: pgmap v565: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.2 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 769 op/s; 8185/255510 objects degraded (3.203%); 1896/255510 objects misplaced (0.742%); 3.4 MiB/s, 54 objects/s recovering 2024-09-13T14:33:16.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:16 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/905058768' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:33:16.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:16 smithi102 ceph-mon[115269]: pgmap v565: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 5.2 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 769 op/s; 8185/255510 objects degraded (3.203%); 1896/255510 objects misplaced (0.742%); 3.4 MiB/s, 54 objects/s recovering 2024-09-13T14:33:16.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:16 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/905058768' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:33:16.919 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:33:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:33:16] "GET /metrics HTTP/1.1" 200 36552 "" "Prometheus/2.51.0" 2024-09-13T14:33:17.007 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:17.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:17.058 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:17.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:17.159 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:17.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:17.198 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:17.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:17.299 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:17.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:17.329 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:17.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:17.430 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:17.460 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:17.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:17.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:17.562 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:33:17.596 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:17.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:17.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:33:17.698 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:33:17.748 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:17.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:17.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:33:17.850 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:17.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:17.879 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:17.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:17.980 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:18.010 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:18.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:18.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:18.112 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:33:18.145 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:18.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:18.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:33:18.247 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:18.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:18.276 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:18.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:18.378 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:18.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:18.411 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:18.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:18.512 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:18.536 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:18.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:18.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:18.637 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:18.665 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:18.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:18.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:18.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:18 smithi195 ceph-mon[104262]: pgmap v566: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 1.11k op/s; 8185/243144 objects degraded (3.366%); 1480/243144 objects misplaced (0.609%); 5.6 MiB/s, 88 objects/s recovering 2024-09-13T14:33:18.767 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:18.771 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:18 smithi102 ceph-mon[110183]: pgmap v566: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 1.11k op/s; 8185/243144 objects degraded (3.366%); 1480/243144 objects misplaced (0.609%); 5.6 MiB/s, 88 objects/s recovering 2024-09-13T14:33:18.771 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:18 smithi102 ceph-mon[115269]: pgmap v566: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 1.11k op/s; 8185/243144 objects degraded (3.366%); 1480/243144 objects misplaced (0.609%); 5.6 MiB/s, 88 objects/s recovering 2024-09-13T14:33:18.795 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:18.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:18.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:18.898 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:18.935 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:18.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:18.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:19.036 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:19.065 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:19.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:19.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:19.167 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:19.193 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:19.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:19.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:19.295 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:19.333 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:19.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:19.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:19.435 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:19.464 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:19.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:19.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:19.566 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:19.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:19.609 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:19.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:19.630 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:19 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 8185/243144 objects degraded (3.366%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:19.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:19 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 8185/243144 objects degraded (3.366%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:19.710 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:19.740 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:19.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:19.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:19.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:19 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 8185/243144 objects degraded (3.366%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:19.841 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:19.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:19.872 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:19.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:19.973 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:20.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:20.018 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:20.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:20.118 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:33:20.161 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:20.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:20.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:33:20.263 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:20.293 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:20.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:20.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:20.394 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:20.420 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:20.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:20.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:20.522 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:20.552 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:20.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:20.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:20.594 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:20 smithi102 ceph-mon[110183]: pgmap v567: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 815 op/s; 8185/241329 objects degraded (3.392%); 1480/241329 objects misplaced (0.613%); 3.9 MiB/s, 62 objects/s recovering 2024-09-13T14:33:20.595 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:20 smithi102 ceph-mon[115269]: pgmap v567: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 815 op/s; 8185/241329 objects degraded (3.392%); 1480/241329 objects misplaced (0.613%); 3.9 MiB/s, 62 objects/s recovering 2024-09-13T14:33:20.654 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:20.684 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:20.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:20.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:20.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:20 smithi195 ceph-mon[104262]: pgmap v567: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 815 op/s; 8185/241329 objects degraded (3.392%); 1480/241329 objects misplaced (0.613%); 3.9 MiB/s, 62 objects/s recovering 2024-09-13T14:33:20.787 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:20.818 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:20.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:20.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:20.921 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:20.959 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:20.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:20.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:21.061 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:21.099 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:21.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:21.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:21.201 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:21.229 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:21.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:21.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:21.334 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:21.374 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:21.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:21.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:21.404 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:33:21.477 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:21.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:21.527 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:21.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:21.629 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:21.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:21.670 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:21.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:21.771 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:21.799 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:21.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:21.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:21.901 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:21.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:21.937 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:21.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:22.038 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:22.069 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:22.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:22.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:22.171 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:33:22.213 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:22.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:22.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:33:22.315 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:33:22.352 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:22.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:22.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:33:22.455 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:22.470 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is unset 2024-09-13T14:33:22.492 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:22.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:22.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:22.594 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:22.603 INFO:teuthology.orchestra.run.smithi102.stderr:no valid command found; 10 closest matches: 2024-09-13T14:33:22.603 INFO:teuthology.orchestra.run.smithi102.stderr:0 2024-09-13T14:33:22.603 INFO:teuthology.orchestra.run.smithi102.stderr:1 2024-09-13T14:33:22.604 INFO:teuthology.orchestra.run.smithi102.stderr:2 2024-09-13T14:33:22.604 INFO:teuthology.orchestra.run.smithi102.stderr:abort 2024-09-13T14:33:22.604 INFO:teuthology.orchestra.run.smithi102.stderr:assert 2024-09-13T14:33:22.604 INFO:teuthology.orchestra.run.smithi102.stderr:bench [] [] [] [] 2024-09-13T14:33:22.605 INFO:teuthology.orchestra.run.smithi102.stderr:bluefs debug_inject_read_zeros 2024-09-13T14:33:22.605 INFO:teuthology.orchestra.run.smithi102.stderr:bluefs files list 2024-09-13T14:33:22.605 INFO:teuthology.orchestra.run.smithi102.stderr:bluefs stats 2024-09-13T14:33:22.606 INFO:teuthology.orchestra.run.smithi102.stderr:bluestore allocator dump block 2024-09-13T14:33:22.606 INFO:teuthology.orchestra.run.smithi102.stderr:Error EINVAL: invalid command 2024-09-13T14:33:22.628 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:22.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:22.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:22.730 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:33:22.737 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:22 smithi102 ceph-mon[110183]: pgmap v568: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.9 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 791 op/s; 8185/240360 objects degraded (3.405%); 1480/240360 objects misplaced (0.616%); 3.9 MiB/s, 62 objects/s recovering 2024-09-13T14:33:22.737 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:22 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3361182004' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:33:22.737 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:22 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:33:22.737 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:22 smithi102 ceph-mon[115269]: pgmap v568: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.9 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 791 op/s; 8185/240360 objects degraded (3.405%); 1480/240360 objects misplaced (0.616%); 3.9 MiB/s, 62 objects/s recovering 2024-09-13T14:33:22.737 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:22 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3361182004' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:33:22.737 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:22 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:33:22.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:22 smithi195 ceph-mon[104262]: pgmap v568: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.9 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 791 op/s; 8185/240360 objects degraded (3.405%); 1480/240360 objects misplaced (0.616%); 3.9 MiB/s, 62 objects/s recovering 2024-09-13T14:33:22.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:22 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3361182004' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:33:22.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:22 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:33:22.769 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:22.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:22.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:33:22.870 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:22.899 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:22.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:22.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:23.001 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:23.027 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:23.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:23.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:23.130 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:23.158 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:23.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:23.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:23.260 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:33:23.310 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:23.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:23.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:33:23.412 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:33:23.448 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:23.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:23.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:33:23.549 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:23.593 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:23.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:23.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:23.695 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:23.730 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:23.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:23.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:23.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:23 smithi195 ceph-mon[104262]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:33:23.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:23 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:33:23.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:23 smithi195 ceph-mon[104262]: osdmap e178: 8 total, 8 up, 6 in 2024-09-13T14:33:23.833 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:23.860 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:23.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:23.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:23.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:23 smithi102 ceph-mon[110183]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:33:23.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:23 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:33:23.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:23 smithi102 ceph-mon[110183]: osdmap e178: 8 total, 8 up, 6 in 2024-09-13T14:33:23.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:23 smithi102 ceph-mon[115269]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:33:23.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:23 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:33:23.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:23 smithi102 ceph-mon[115269]: osdmap e178: 8 total, 8 up, 6 in 2024-09-13T14:33:23.962 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:23.989 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:23.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:23.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:24.092 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:24.120 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:24.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:24.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:24.222 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:24.253 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:24.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:24.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:24.355 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:24.402 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:24.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:24.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:24.468 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T14:33:24.469 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T14:33:24.507 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:24.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:24.546 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:24.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:24.647 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:24.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:24 smithi102 ceph-mon[115269]: pgmap v570: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 936 op/s; 8185/227406 objects degraded (3.599%); 981/227406 objects misplaced (0.431%); 5.7 MiB/s, 91 objects/s recovering 2024-09-13T14:33:24.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:24 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 8185/227406 objects degraded (3.599%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:24.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:24 smithi102 ceph-mon[110183]: pgmap v570: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 936 op/s; 8185/227406 objects degraded (3.599%); 981/227406 objects misplaced (0.431%); 5.7 MiB/s, 91 objects/s recovering 2024-09-13T14:33:24.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:24 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 8185/227406 objects degraded (3.599%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:24.686 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:24.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:24.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:24.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:24 smithi195 ceph-mon[104262]: pgmap v570: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 936 op/s; 8185/227406 objects degraded (3.599%); 981/227406 objects misplaced (0.431%); 5.7 MiB/s, 91 objects/s recovering 2024-09-13T14:33:24.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:24 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 8185/227406 objects degraded (3.599%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:24.788 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:24.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:24.825 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:24.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:24.926 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:24.963 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:24.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:24.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:25.065 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:33:25.108 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:25.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:25.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:33:25.213 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:25.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:25.235 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:25.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:25.336 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:25.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:25.381 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:25.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:25.482 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:25.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:25.523 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:25.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:25.537 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:33:25.624 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:25.661 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:25.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:25.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:25.763 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:25.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:25.795 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:25.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:25.894 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:33:25.894 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [], 2024-09-13T14:33:25.894 INFO:teuthology.orchestra.run.smithi102.stdout: "complaint_time": 30, 2024-09-13T14:33:25.894 INFO:teuthology.orchestra.run.smithi102.stdout: "num_blocked_ops": 0 2024-09-13T14:33:25.895 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:33:25.896 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:33:25.936 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:25.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:25.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:33:26.038 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:26.065 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:26.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:26.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:26.167 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:26.189 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:26.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:26.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:26.291 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:26.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:26.323 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:26.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:26.424 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:26.455 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:33:26.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:26.462 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:26.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:26.563 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:33:26.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:26.602 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:26.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:33:26.703 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:33:26.739 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:26.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:26.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:33:26.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:26 smithi195 ceph-mon[104262]: pgmap v571: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 936 op/s; 8185/227406 objects degraded (3.599%); 981/227406 objects misplaced (0.431%); 5.7 MiB/s, 91 objects/s recovering 2024-09-13T14:33:26.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:33:26.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:33:26.750 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:26 smithi102 ceph-mon[110183]: pgmap v571: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 936 op/s; 8185/227406 objects degraded (3.599%); 981/227406 objects misplaced (0.431%); 5.7 MiB/s, 91 objects/s recovering 2024-09-13T14:33:26.750 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:33:26.750 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:33:26.750 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:26 smithi102 ceph-mon[115269]: pgmap v571: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 936 op/s; 8185/227406 objects degraded (3.599%); 981/227406 objects misplaced (0.431%); 5.7 MiB/s, 91 objects/s recovering 2024-09-13T14:33:26.750 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:33:26.750 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:33:26.751 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:33:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:33:26] "GET /metrics HTTP/1.1" 200 36548 "" "Prometheus/2.51.0" 2024-09-13T14:33:26.841 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:33:26.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:26.890 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:26.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:33:26.991 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:27.020 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:27.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:27.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:27.122 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:27.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:27.171 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:27.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:27.272 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:27.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:27.304 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:27.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:27.315 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T14:33:27.405 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:27.446 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:27.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:27.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:27.547 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:27.577 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:27.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:27.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:27.678 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:27.702 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:27.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:27.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:27.777 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:33:27.804 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:27.839 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:27.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:27.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:27.941 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:27.961 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:27.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:27.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:28.063 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:33:28.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:28.108 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:28.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:33:28.209 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:33:28.247 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:28.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:28.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:33:28.349 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:28.393 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:28.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:28.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:28.494 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:28.533 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:28.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:28.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:28.586 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:28 smithi102 ceph-mon[110183]: pgmap v572: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.4 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 925 op/s; 8185/215376 objects degraded (3.800%); 388/215376 objects misplaced (0.180%); 6.8 MiB/s, 109 objects/s recovering 2024-09-13T14:33:28.586 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:28 smithi102 ceph-mon[115269]: pgmap v572: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.4 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 925 op/s; 8185/215376 objects degraded (3.800%); 388/215376 objects misplaced (0.180%); 6.8 MiB/s, 109 objects/s recovering 2024-09-13T14:33:28.635 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:33:28.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:28.680 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:28.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:33:28.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:28 smithi195 ceph-mon[104262]: pgmap v572: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.4 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 925 op/s; 8185/215376 objects degraded (3.800%); 388/215376 objects misplaced (0.180%); 6.8 MiB/s, 109 objects/s recovering 2024-09-13T14:33:28.781 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:28.807 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:28.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:28.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:28.909 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:28.943 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:28.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:28.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:29.046 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:29.087 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:29.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:29.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:29.189 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:29.216 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:29.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:29.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:29.220 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:33:29.318 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:33:29.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:29.354 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:29.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:33:29.455 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:29.469 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 7] out_osds: [1, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:33:29.469 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:33:29.470 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 3, 4, 5, 6, 7] 2024-09-13T14:33:29.470 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd out 5 2024-09-13T14:33:29.480 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:29.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:29.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:29.582 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:33:29.618 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:29.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:29.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:33:29.634 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:29 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 8185/215376 objects degraded (3.800%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:29.634 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:29 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 8185/215376 objects degraded (3.800%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:29.720 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:29.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:29 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 8185/215376 objects degraded (3.800%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:29.750 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:29.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:29.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:29.775 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:33:29.775 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:33:29.799 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:33:29.799 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:33:29.824 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:33:29.824 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:33:29.844 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:33:29.844 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:33:29.852 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:29.863 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:33:29.863 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:33:29.876 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:29.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:29.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:29.884 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:33:29.884 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:33:29.903 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:33:29.903 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:33:29.921 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:33:29.921 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:33:29.978 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:29.999 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:30.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:30.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:30.101 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:30.122 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:30.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:30.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:30.224 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:30.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:30.246 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:30.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:30.347 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:30.369 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:30.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:30.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:30.471 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:30.502 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:30.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:30.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:30.603 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:30.650 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:30.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:30.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:30.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:30 smithi195 ceph-mon[104262]: pgmap v573: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.4 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 920 op/s; 8185/213708 objects degraded (3.830%); 388/213708 objects misplaced (0.182%); 6.8 MiB/s, 109 objects/s recovering 2024-09-13T14:33:30.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:30 smithi195 ceph-mon[104262]: osdmap e179: 8 total, 8 up, 6 in 2024-09-13T14:33:30.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:30 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2146700797' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:33:30.752 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:33:30.785 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:30 smithi102 ceph-mon[110183]: pgmap v573: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.4 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 920 op/s; 8185/213708 objects degraded (3.830%); 388/213708 objects misplaced (0.182%); 6.8 MiB/s, 109 objects/s recovering 2024-09-13T14:33:30.785 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:30 smithi102 ceph-mon[110183]: osdmap e179: 8 total, 8 up, 6 in 2024-09-13T14:33:30.785 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:30 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2146700797' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:33:30.785 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:30 smithi102 ceph-mon[115269]: pgmap v573: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.4 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 920 op/s; 8185/213708 objects degraded (3.830%); 388/213708 objects misplaced (0.182%); 6.8 MiB/s, 109 objects/s recovering 2024-09-13T14:33:30.785 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:30 smithi102 ceph-mon[115269]: osdmap e179: 8 total, 8 up, 6 in 2024-09-13T14:33:30.785 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:30 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2146700797' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:33:30.797 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:30.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:30.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:33:30.899 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:30.924 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:30.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:30.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:31.026 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:33:31.058 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:31.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:31.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:33:31.160 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:31.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:31.190 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:31.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:31.291 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:31.318 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:31.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:31.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:31.419 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:31.459 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:31.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:31.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:31.561 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:31.601 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:31.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:31.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:31.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:31 smithi102 ceph-mon[110183]: osdmap e180: 8 total, 8 up, 6 in 2024-09-13T14:33:31.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:31 smithi102 ceph-mon[115269]: osdmap e180: 8 total, 8 up, 6 in 2024-09-13T14:33:31.703 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:31.729 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:31.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:31.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:31.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:31 smithi195 ceph-mon[104262]: osdmap e180: 8 total, 8 up, 6 in 2024-09-13T14:33:31.831 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:31.856 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:31.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:31.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:31.958 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:31.995 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:31.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:31.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:32.097 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:32.123 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:32.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:32.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:32.225 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:32.251 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:32.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:32.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:32.353 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:32.383 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:32.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:32.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:32.484 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:32.511 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:32.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:32.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:32.613 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:32.638 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:32.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:32.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:32.739 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:33:32.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:32 smithi195 ceph-mon[104262]: pgmap v576: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.3 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 598 op/s; 5860/213036 objects degraded (2.751%); 2592/213036 objects misplaced (1.217%); 5.6 MiB/s, 89 objects/s recovering 2024-09-13T14:33:32.780 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:32.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:32.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:33:32.884 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:32.909 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:32.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:32.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:32.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:32 smithi102 ceph-mon[110183]: pgmap v576: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.3 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 598 op/s; 5860/213036 objects degraded (2.751%); 2592/213036 objects misplaced (1.217%); 5.6 MiB/s, 89 objects/s recovering 2024-09-13T14:33:32.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:32 smithi102 ceph-mon[115269]: pgmap v576: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.3 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 598 op/s; 5860/213036 objects degraded (2.751%); 2592/213036 objects misplaced (1.217%); 5.6 MiB/s, 89 objects/s recovering 2024-09-13T14:33:33.012 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:33:33.052 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:33.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:33.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:33:33.154 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:33.179 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:33.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:33.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:33.260 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:33:33.281 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:33.336 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:33.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:33.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:33.438 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:33:33.474 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:33.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:33.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:33:33.576 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:33:33.618 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:33.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:33.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:33:33.720 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:33.748 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:33.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:33.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:33.849 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:33:33.881 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:33.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:33.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:33:33.982 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:34.015 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:34.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:34.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:34.116 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:34.156 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:34.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:34.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:34.258 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:34.283 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:34.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:34.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:34.384 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:33:34.419 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:34.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:34.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:33:34.521 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:34.549 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:34.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:34.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:34.565 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:34 smithi102 ceph-mon[115269]: pgmap v577: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 87 active+clean; 4.1 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 1.01k op/s; 3397/203190 objects degraded (1.672%); 4516/203190 objects misplaced (2.223%); 9.8 MiB/s, 156 objects/s recovering 2024-09-13T14:33:34.565 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:34 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 3397/203190 objects degraded (1.672%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:34.566 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:34 smithi102 ceph-mon[110183]: pgmap v577: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 87 active+clean; 4.1 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 1.01k op/s; 3397/203190 objects degraded (1.672%); 4516/203190 objects misplaced (2.223%); 9.8 MiB/s, 156 objects/s recovering 2024-09-13T14:33:34.566 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:34 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 3397/203190 objects degraded (1.672%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:34.651 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:34.695 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:34.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:34.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:34.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:34 smithi195 ceph-mon[104262]: pgmap v577: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 87 active+clean; 4.1 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 1.01k op/s; 3397/203190 objects degraded (1.672%); 4516/203190 objects misplaced (2.223%); 9.8 MiB/s, 156 objects/s recovering 2024-09-13T14:33:34.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:34 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 3397/203190 objects degraded (1.672%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:34.797 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:34.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:34.833 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:34.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:34.933 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:33:34.974 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:33:34.974 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:33:34.975 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:34.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:34.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:33:34.989 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:33:35.077 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:35.105 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:35.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:35.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:35.207 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:33:35.233 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:33:35.244 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:35.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:35.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:33:35.346 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:35.374 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:35.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:35.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:35.470 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is set 2024-09-13T14:33:35.475 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:35.515 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:35.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:35.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:35.617 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:33:35.662 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:35.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:35.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:33:35.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:35 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3146961358' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:33:35.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:35 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:33:35.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:35 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3146961358' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:33:35.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:35 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:33:35.702 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T14:33:35.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:35 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3146961358' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:33:35.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:35 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:33:35.764 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:35.792 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:35.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:35.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:35.893 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:35.928 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:35.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:35.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:36.031 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:36.060 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:36.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:36.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:36.162 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:36.194 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:36.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:36.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:36.296 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:36.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:36.338 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:36.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:36.439 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:36.466 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:36.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:36.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:36.568 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:36.616 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:36.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:36.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:36.621 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:33:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:33:36] "GET /metrics HTTP/1.1" 200 36548 "" "Prometheus/2.51.0" 2024-09-13T14:33:36.621 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:36 smithi102 ceph-mon[110183]: pgmap v578: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 87 active+clean; 4.1 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 507 op/s; 3397/203190 objects degraded (1.672%); 4516/203190 objects misplaced (2.223%); 5.2 MiB/s, 82 objects/s recovering 2024-09-13T14:33:36.621 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:36 smithi102 ceph-mon[110183]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:33:36.621 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:36 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:33:36.621 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:36 smithi102 ceph-mon[110183]: osdmap e181: 8 total, 8 up, 6 in 2024-09-13T14:33:36.621 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:36 smithi102 ceph-mon[115269]: pgmap v578: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 87 active+clean; 4.1 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 507 op/s; 3397/203190 objects degraded (1.672%); 4516/203190 objects misplaced (2.223%); 5.2 MiB/s, 82 objects/s recovering 2024-09-13T14:33:36.622 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:36 smithi102 ceph-mon[115269]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:33:36.622 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:36 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:33:36.622 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:36 smithi102 ceph-mon[115269]: osdmap e181: 8 total, 8 up, 6 in 2024-09-13T14:33:36.718 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:36.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:36 smithi195 ceph-mon[104262]: pgmap v578: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 87 active+clean; 4.1 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 507 op/s; 3397/203190 objects degraded (1.672%); 4516/203190 objects misplaced (2.223%); 5.2 MiB/s, 82 objects/s recovering 2024-09-13T14:33:36.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:36 smithi195 ceph-mon[104262]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:33:36.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:36 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:33:36.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:36 smithi195 ceph-mon[104262]: osdmap e181: 8 total, 8 up, 6 in 2024-09-13T14:33:36.766 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:36.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:36.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:36.867 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:36.906 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:36.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:36.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:37.008 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:37.045 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:37.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:37.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:37.147 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:37.174 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:37.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:37.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:37.276 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:37.302 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:37.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:37.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:37.404 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:37.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:37.433 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:37.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:37.534 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:33:37.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:37.590 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:37.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:33:37.691 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:37.716 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:37.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:37.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:37.818 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:37.841 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:37.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:37.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:37.943 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:37.982 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:37.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:37.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:38.084 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:38.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:38.126 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:38.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:38.227 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:38.251 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:38.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:38.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:38.353 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:38.384 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:38.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:38.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:38.486 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:38.512 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:38.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:38.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:38.614 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:33:38.658 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:38.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:38.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:33:38.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:38 smithi195 ceph-mon[104262]: pgmap v580: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 87 active+clean; 3.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 918 op/s; 3541/191880 objects degraded (1.845%); 4043/191880 objects misplaced (2.107%); 7.8 MiB/s, 124 objects/s recovering 2024-09-13T14:33:38.760 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:38.791 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:38.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:38.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:38.893 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:38.916 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:38.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:38.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:38.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:38 smithi102 ceph-mon[110183]: pgmap v580: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 87 active+clean; 3.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 918 op/s; 3541/191880 objects degraded (1.845%); 4043/191880 objects misplaced (2.107%); 7.8 MiB/s, 124 objects/s recovering 2024-09-13T14:33:38.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:38 smithi102 ceph-mon[115269]: pgmap v580: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 87 active+clean; 3.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 918 op/s; 3541/191880 objects degraded (1.845%); 4043/191880 objects misplaced (2.107%); 7.8 MiB/s, 124 objects/s recovering 2024-09-13T14:33:39.018 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:33:39.050 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:39.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:39.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:33:39.152 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:39.177 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:39.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:39.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:39.279 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:39.310 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:39.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:39.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:39.386 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:33:39.412 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:39.458 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:39.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:39.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:39.560 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:39.595 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:39.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:39.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:39.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:39 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 3541/191880 objects degraded (1.845%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:39.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:39 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 3541/191880 objects degraded (1.845%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:39.697 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:33:39.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:39.735 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:39.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:33:39.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:39 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 3541/191880 objects degraded (1.845%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:39.837 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:39.877 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:39.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:39.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:39.979 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:33:40.014 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:40.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:40.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:33:40.116 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:40.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:40.145 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:40.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:40.246 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:40.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:40.272 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:40.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:40.373 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:40.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:40.417 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:40.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:40.518 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:33:40.557 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:40.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:40.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:33:40.659 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:33:40.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:40.696 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:40.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:33:40.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:40 smithi195 ceph-mon[104262]: pgmap v581: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 87 active+clean; 3.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 876 op/s; 3541/190275 objects degraded (1.861%); 4043/190275 objects misplaced (2.125%); 6.9 MiB/s, 110 objects/s recovering 2024-09-13T14:33:40.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:40 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:33:40.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:40 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:33:40.797 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:40.825 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:40.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:40.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:40.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:40 smithi102 ceph-mon[110183]: pgmap v581: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 87 active+clean; 3.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 876 op/s; 3541/190275 objects degraded (1.861%); 4043/190275 objects misplaced (2.125%); 6.9 MiB/s, 110 objects/s recovering 2024-09-13T14:33:40.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:40 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:33:40.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:40 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:33:40.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:40 smithi102 ceph-mon[115269]: pgmap v581: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 87 active+clean; 3.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 876 op/s; 3541/190275 objects degraded (1.861%); 4043/190275 objects misplaced (2.125%); 6.9 MiB/s, 110 objects/s recovering 2024-09-13T14:33:40.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:40 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:33:40.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:40 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:33:40.927 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:40.955 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:40.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:40.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:41.057 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:41.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:41.079 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:41.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:41.180 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:41.217 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:41.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:41.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:41.319 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:41.345 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:41.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:41.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:41.447 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:33:41.485 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:41.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:41.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:33:41.587 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:41.618 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:41.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:41.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:41.720 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:41.754 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:41.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:41.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:41.856 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:41.884 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:41.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:41.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:41.986 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:33:42.026 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:42.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:42.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:33:42.128 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:42.155 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:42.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:42.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:42.257 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:42.287 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:42.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:42.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:42.317 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:33:42.389 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:33:42.427 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:42.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:42.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:33:42.529 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:42.560 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:42 smithi102 ceph-mon[110183]: pgmap v582: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 87 active+clean; 3.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 781 op/s; 3611/189564 objects degraded (1.905%); 3796/189564 objects misplaced (2.002%); 6.5 MiB/s, 104 objects/s recovering 2024-09-13T14:33:42.560 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:42 smithi102 ceph-mon[115269]: pgmap v582: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 87 active+clean; 3.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 781 op/s; 3611/189564 objects degraded (1.905%); 3796/189564 objects misplaced (2.002%); 6.5 MiB/s, 104 objects/s recovering 2024-09-13T14:33:42.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:42.563 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:42.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:42.622 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:33:42.623 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:33:42.649 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:33:42.649 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:33:42.664 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:42.674 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:33:42.674 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:33:42.693 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:42.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:42.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:42.701 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:33:42.701 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:33:42.736 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:33:42.736 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:33:42.762 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:33:42.762 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:33:42.792 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:33:42.792 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:33:42.795 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:42.815 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:42 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-a[110159]: 2024-09-13T14:33:42.602+0000 7fa3e7f42640 -1 mon.a@0(leader).osd e181 definitely_dead 0 2024-09-13T14:33:42.830 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:42.831 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:33:42.832 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:33:42.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:42.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:42.933 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:42.961 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:42.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:42.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:42.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:42 smithi195 ceph-mon[104262]: pgmap v582: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 87 active+clean; 3.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 781 op/s; 3611/189564 objects degraded (1.905%); 3796/189564 objects misplaced (2.002%); 6.5 MiB/s, 104 objects/s recovering 2024-09-13T14:33:43.063 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:43.089 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:43.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:43.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:43.190 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:43.217 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:43.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:43.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:43.319 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:43.360 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:43.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:43.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:43.462 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:33:43.506 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:43.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:43.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:33:43.510 INFO:teuthology.orchestra.run.smithi102.stderr:marked out osd.5. 2024-09-13T14:33:43.608 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:43.629 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:43.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:43.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:43.730 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:33:43.752 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:43.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:43.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:33:43.853 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:43.875 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:43.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:43.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:43.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:43 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/4086525272' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:33:43.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:43 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3077476653' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-13T14:33:43.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:43 smithi102 ceph-mon[110183]: Client client.admin marked osd.5 out, while it was still marked up 2024-09-13T14:33:43.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:43 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/4086525272' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:33:43.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:43 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3077476653' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-13T14:33:43.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:43 smithi102 ceph-mon[115269]: Client client.admin marked osd.5 out, while it was still marked up 2024-09-13T14:33:43.976 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:43.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:43 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/4086525272' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:33:43.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:43 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3077476653' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-13T14:33:43.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:43 smithi195 ceph-mon[104262]: Client client.admin marked osd.5 out, while it was still marked up 2024-09-13T14:33:43.998 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:43.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:43.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:44.100 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:44.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:44.122 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:44.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:44.176 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:33:44.222 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:44.243 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:44.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:44.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:44.345 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:44.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:44.368 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:44.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:44.469 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:33:44.490 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:44.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:44.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:33:44.592 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:44.613 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:44.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:44.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:44.715 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:44.736 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:44.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:44.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:44.838 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:44.864 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:44.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:44.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:44.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:44 smithi102 ceph-mon[115269]: pgmap v583: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 87 active+clean; 3.6 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 849 op/s; 3725/177690 objects degraded (2.096%); 3512/177690 objects misplaced (1.976%); 4.2 MiB/s, 67 objects/s recovering 2024-09-13T14:33:44.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:44 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3077476653' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-09-13T14:33:44.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:44 smithi102 ceph-mon[115269]: osdmap e182: 8 total, 8 up, 5 in 2024-09-13T14:33:44.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:44 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 3725/177690 objects degraded (2.096%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:44.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:44 smithi102 ceph-mon[115269]: osdmap e183: 8 total, 8 up, 5 in 2024-09-13T14:33:44.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:44 smithi102 ceph-mon[110183]: pgmap v583: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 87 active+clean; 3.6 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 849 op/s; 3725/177690 objects degraded (2.096%); 3512/177690 objects misplaced (1.976%); 4.2 MiB/s, 67 objects/s recovering 2024-09-13T14:33:44.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:44 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3077476653' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-09-13T14:33:44.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:44 smithi102 ceph-mon[110183]: osdmap e182: 8 total, 8 up, 5 in 2024-09-13T14:33:44.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:44 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 3725/177690 objects degraded (2.096%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:44.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:44 smithi102 ceph-mon[110183]: osdmap e183: 8 total, 8 up, 5 in 2024-09-13T14:33:44.966 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:44.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:44 smithi195 ceph-mon[104262]: pgmap v583: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 87 active+clean; 3.6 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 849 op/s; 3725/177690 objects degraded (2.096%); 3512/177690 objects misplaced (1.976%); 4.2 MiB/s, 67 objects/s recovering 2024-09-13T14:33:44.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:44 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3077476653' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-09-13T14:33:44.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:44 smithi195 ceph-mon[104262]: osdmap e182: 8 total, 8 up, 5 in 2024-09-13T14:33:44.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:44 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 3725/177690 objects degraded (2.096%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:33:44.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:44 smithi195 ceph-mon[104262]: osdmap e183: 8 total, 8 up, 5 in 2024-09-13T14:33:45.010 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:45.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:45.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:45.112 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:45.151 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:45.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:45.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:45.253 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:45.277 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:45.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:45.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:45.382 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:45.414 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:45.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:45.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:45.515 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:45.540 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:45.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:45.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:45.642 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:45.672 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:45.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:45.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:45.774 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:45.811 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:45.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:45.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:45.914 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:45.947 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:45.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:45.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:46.026 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:45 smithi102 ceph-mon[115269]: 6.d starting backfill to osd.4 from (0'0,0'0] MAX to 181'10342 2024-09-13T14:33:46.027 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:45 smithi102 ceph-mon[115269]: 6.0 starting backfill to osd.4 from (0'0,0'0] MAX to 181'10103 2024-09-13T14:33:46.027 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:45 smithi102 ceph-mon[115269]: osdmap e184: 8 total, 8 up, 5 in 2024-09-13T14:33:46.027 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:45 smithi102 ceph-mon[115269]: 6.1 starting backfill to osd.3 from (0'0,0'0] MAX to 181'10099 2024-09-13T14:33:46.027 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:45 smithi102 ceph-mon[115269]: 6.a starting backfill to osd.3 from (0'0,0'0] MAX to 181'10270 2024-09-13T14:33:46.027 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:45 smithi102 ceph-mon[115269]: 6.e starting backfill to osd.6 from (0'0,0'0] MAX to 181'10363 2024-09-13T14:33:46.027 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:45 smithi102 ceph-mon[115269]: 6.c starting backfill to osd.7 from (0'0,0'0] MAX to 181'10022 2024-09-13T14:33:46.027 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:45 smithi102 ceph-mon[115269]: 6.7 starting backfill to osd.7 from (0'0,0'0] MAX to 181'10370 2024-09-13T14:33:46.028 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:45 smithi102 ceph-mon[115269]: 6.4 starting backfill to osd.4 from (0'0,0'0] MAX to 181'10186 2024-09-13T14:33:46.028 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:45 smithi102 ceph-mon[110183]: 6.d starting backfill to osd.4 from (0'0,0'0] MAX to 181'10342 2024-09-13T14:33:46.028 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:45 smithi102 ceph-mon[110183]: 6.0 starting backfill to osd.4 from (0'0,0'0] MAX to 181'10103 2024-09-13T14:33:46.028 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:45 smithi102 ceph-mon[110183]: osdmap e184: 8 total, 8 up, 5 in 2024-09-13T14:33:46.028 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:45 smithi102 ceph-mon[110183]: 6.1 starting backfill to osd.3 from (0'0,0'0] MAX to 181'10099 2024-09-13T14:33:46.028 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:45 smithi102 ceph-mon[110183]: 6.a starting backfill to osd.3 from (0'0,0'0] MAX to 181'10270 2024-09-13T14:33:46.028 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:45 smithi102 ceph-mon[110183]: 6.e starting backfill to osd.6 from (0'0,0'0] MAX to 181'10363 2024-09-13T14:33:46.028 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:45 smithi102 ceph-mon[110183]: 6.c starting backfill to osd.7 from (0'0,0'0] MAX to 181'10022 2024-09-13T14:33:46.028 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:45 smithi102 ceph-mon[110183]: 6.7 starting backfill to osd.7 from (0'0,0'0] MAX to 181'10370 2024-09-13T14:33:46.029 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:45 smithi102 ceph-mon[110183]: 6.4 starting backfill to osd.4 from (0'0,0'0] MAX to 181'10186 2024-09-13T14:33:46.049 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:33:46.093 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:46.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:46.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:33:46.195 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:46.199 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:33:46.223 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:46.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:46.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:46.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:45 smithi195 ceph-mon[104262]: 6.d starting backfill to osd.4 from (0'0,0'0] MAX to 181'10342 2024-09-13T14:33:46.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:45 smithi195 ceph-mon[104262]: 6.0 starting backfill to osd.4 from (0'0,0'0] MAX to 181'10103 2024-09-13T14:33:46.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:45 smithi195 ceph-mon[104262]: osdmap e184: 8 total, 8 up, 5 in 2024-09-13T14:33:46.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:45 smithi195 ceph-mon[104262]: 6.1 starting backfill to osd.3 from (0'0,0'0] MAX to 181'10099 2024-09-13T14:33:46.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:45 smithi195 ceph-mon[104262]: 6.a starting backfill to osd.3 from (0'0,0'0] MAX to 181'10270 2024-09-13T14:33:46.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:45 smithi195 ceph-mon[104262]: 6.e starting backfill to osd.6 from (0'0,0'0] MAX to 181'10363 2024-09-13T14:33:46.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:45 smithi195 ceph-mon[104262]: 6.c starting backfill to osd.7 from (0'0,0'0] MAX to 181'10022 2024-09-13T14:33:46.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:45 smithi195 ceph-mon[104262]: 6.7 starting backfill to osd.7 from (0'0,0'0] MAX to 181'10370 2024-09-13T14:33:46.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:45 smithi195 ceph-mon[104262]: 6.4 starting backfill to osd.4 from (0'0,0'0] MAX to 181'10186 2024-09-13T14:33:46.325 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:46.355 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:46.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:46.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:46.457 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:46.483 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:46.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:46.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:46.585 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:33:46.625 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:46.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:46.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:33:46.727 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:46.754 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:46.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:46.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:46.829 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:33:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:33:46] "GET /metrics HTTP/1.1" 200 36557 "" "Prometheus/2.51.0" 2024-09-13T14:33:46.856 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:33:46.880 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:46.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:46.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:33:46.982 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:47.010 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:47.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:47.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:47.111 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:47.139 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:47.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:47.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:47.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:46 smithi102 ceph-mon[110183]: pgmap v587: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 87 active+clean; 3.6 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 698 op/s; 3725/177690 objects degraded (2.096%); 3512/177690 objects misplaced (1.976%); 3.6 MiB/s, 57 objects/s recovering 2024-09-13T14:33:47.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:46 smithi102 ceph-mon[110183]: osdmap e185: 8 total, 8 up, 5 in 2024-09-13T14:33:47.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:46 smithi102 ceph-mon[115269]: pgmap v587: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 87 active+clean; 3.6 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 698 op/s; 3725/177690 objects degraded (2.096%); 3512/177690 objects misplaced (1.976%); 3.6 MiB/s, 57 objects/s recovering 2024-09-13T14:33:47.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:46 smithi102 ceph-mon[115269]: osdmap e185: 8 total, 8 up, 5 in 2024-09-13T14:33:47.241 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:47.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:46 smithi195 ceph-mon[104262]: pgmap v587: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 87 active+clean; 3.6 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 698 op/s; 3725/177690 objects degraded (2.096%); 3512/177690 objects misplaced (1.976%); 3.6 MiB/s, 57 objects/s recovering 2024-09-13T14:33:47.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:46 smithi195 ceph-mon[104262]: osdmap e185: 8 total, 8 up, 5 in 2024-09-13T14:33:47.282 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:47.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:47.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:47.385 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:47.411 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:47.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:47.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:47.513 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:47.538 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:47.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:47.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:47.640 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:47.664 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:47.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:47.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:47.766 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:47.813 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:47.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:47.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:47.915 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:47.939 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:47.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:47.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:48.041 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:33:48.081 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:48.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:48.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:33:48.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:47 smithi102 ceph-mon[110183]: osdmap e186: 8 total, 8 up, 5 in 2024-09-13T14:33:48.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:47 smithi102 ceph-mon[115269]: osdmap e186: 8 total, 8 up, 5 in 2024-09-13T14:33:48.183 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:48.215 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:48.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:48.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:48.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:47 smithi195 ceph-mon[104262]: osdmap e186: 8 total, 8 up, 5 in 2024-09-13T14:33:48.317 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:33:48.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:48.339 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:48.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:33:48.440 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:48.469 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:48.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:48.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:48.571 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:48.590 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:33:48.590 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [ 2024-09-13T14:33:48.590 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.590 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35037.0:166544 6.b 6:dc20a272:::benchmark_data_smithi102_194590_object55920:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-13T14:33:48.590 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:33:48.577065+0000", 2024-09-13T14:33:48.590 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.011644270999999999, 2024-09-13T14:33:48.590 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.011670669, 2024-09-13T14:33:48.590 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:33:48.591 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:33:48.591 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:33:48.591 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:33:48.591 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35037", 2024-09-13T14:33:48.591 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:33:48.591 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 166544 2024-09-13T14:33:48.591 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.591 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:33:48.591 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.591 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:33:48.591 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.577065+0000", 2024-09-13T14:33:48.591 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:33:48.591 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.591 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.592 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:33:48.592 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.577065+0000", 2024-09-13T14:33:48.592 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:33:48.592 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.592 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.592 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:33:48.592 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.577066+0000", 2024-09-13T14:33:48.592 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.274e-06 2024-09-13T14:33:48.592 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.592 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.592 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:33:48.592 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.577071+0000", 2024-09-13T14:33:48.592 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.916e-06 2024-09-13T14:33:48.593 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.593 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.593 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:33:48.593 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.577083+0000", 2024-09-13T14:33:48.593 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2694000000000001e-05 2024-09-13T14:33:48.593 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.593 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.593 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:33:48.593 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.577090+0000", 2024-09-13T14:33:48.593 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.6939999999999998e-06 2024-09-13T14:33:48.593 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.593 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.593 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:33:48.594 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.581174+0000", 2024-09-13T14:33:48.594 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0040836509999999998 2024-09-13T14:33:48.594 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.594 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.594 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:33:48.594 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.581319+0000", 2024-09-13T14:33:48.594 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00014563600000000001 2024-09-13T14:33:48.594 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.594 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.594 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:33:48.594 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.581475+0000", 2024-09-13T14:33:48.594 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00015550199999999999 2024-09-13T14:33:48.594 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:33:48.595 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:33:48.595 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:33:48.595 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.595 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.595 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35037.0:166546 6.4 6:2b4dc619:::benchmark_data_smithi102_194590_object55922:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-13T14:33:48.595 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:33:48.578990+0000", 2024-09-13T14:33:48.595 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.0097192379999999998, 2024-09-13T14:33:48.595 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.009787334, 2024-09-13T14:33:48.595 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:33:48.595 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:33:48.595 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:33:48.595 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:33:48.595 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35037", 2024-09-13T14:33:48.595 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:33:48.596 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 166546 2024-09-13T14:33:48.596 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.596 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:33:48.596 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.596 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:33:48.596 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.578990+0000", 2024-09-13T14:33:48.596 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:33:48.596 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.596 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.596 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:33:48.596 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.578990+0000", 2024-09-13T14:33:48.596 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:33:48.597 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.597 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.597 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:33:48.597 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.578991+0000", 2024-09-13T14:33:48.597 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.198e-06 2024-09-13T14:33:48.597 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.597 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.597 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:33:48.597 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.578999+0000", 2024-09-13T14:33:48.597 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.3739999999999998e-06 2024-09-13T14:33:48.597 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.597 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.597 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:33:48.597 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.579013+0000", 2024-09-13T14:33:48.598 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4012999999999999e-05 2024-09-13T14:33:48.598 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.598 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.598 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:33:48.598 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.579021+0000", 2024-09-13T14:33:48.598 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.5800000000000003e-06 2024-09-13T14:33:48.598 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.598 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.598 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:33:48.598 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.579079+0000", 2024-09-13T14:33:48.598 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.8205000000000003e-05 2024-09-13T14:33:48.598 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.598 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.599 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:33:48.599 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.579204+0000", 2024-09-13T14:33:48.599 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00012461199999999999 2024-09-13T14:33:48.599 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.599 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.599 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,5,6", 2024-09-13T14:33:48.599 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.579326+0000", 2024-09-13T14:33:48.599 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000122853 2024-09-13T14:33:48.599 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.599 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.599 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:33:48.599 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.585134+0000", 2024-09-13T14:33:48.599 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0058078879999999998 2024-09-13T14:33:48.599 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.600 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.600 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:33:48.600 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.586227+0000", 2024-09-13T14:33:48.600 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0010923230000000001 2024-09-13T14:33:48.600 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.600 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.600 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:33:48.600 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.588590+0000", 2024-09-13T14:33:48.600 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0023634620000000002 2024-09-13T14:33:48.600 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:33:48.600 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:33:48.600 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:33:48.600 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.600 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.601 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35037.0:166547 6.4 6:29e735d8:::benchmark_data_smithi102_194590_object55923:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-13T14:33:48.601 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:33:48.579523+0000", 2024-09-13T14:33:48.601 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.0091853160000000007, 2024-09-13T14:33:48.601 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0092962400000000008, 2024-09-13T14:33:48.601 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:33:48.601 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:33:48.601 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:33:48.601 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:33:48.601 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35037", 2024-09-13T14:33:48.601 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:33:48.601 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 166547 2024-09-13T14:33:48.601 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.601 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:33:48.601 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.602 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:33:48.602 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.579523+0000", 2024-09-13T14:33:48.602 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:33:48.602 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.602 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.602 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:33:48.602 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.579523+0000", 2024-09-13T14:33:48.602 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:33:48.602 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.602 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.602 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:33:48.603 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.579524+0000", 2024-09-13T14:33:48.603 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.6700000000000002e-07 2024-09-13T14:33:48.603 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.603 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.603 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:33:48.603 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.579529+0000", 2024-09-13T14:33:48.603 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.2069999999999997e-06 2024-09-13T14:33:48.603 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.603 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.603 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:33:48.603 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.579539+0000", 2024-09-13T14:33:48.603 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0831e-05 2024-09-13T14:33:48.603 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.603 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.604 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:33:48.604 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.579547+0000", 2024-09-13T14:33:48.604 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.3000000000000004e-06 2024-09-13T14:33:48.604 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.604 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.604 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:33:48.604 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.582367+0000", 2024-09-13T14:33:48.604 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0028203799999999999 2024-09-13T14:33:48.604 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.604 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.605 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:33:48.605 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.582491+0000", 2024-09-13T14:33:48.605 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000124058 2024-09-13T14:33:48.605 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.605 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.605 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,5,6", 2024-09-13T14:33:48.605 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.582607+0000", 2024-09-13T14:33:48.605 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00011548900000000001 2024-09-13T14:33:48.605 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.605 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.605 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:33:48.606 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.585138+0000", 2024-09-13T14:33:48.606 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0025316290000000001 2024-09-13T14:33:48.606 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.606 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.606 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:33:48.606 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.586265+0000", 2024-09-13T14:33:48.606 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0011264999999999999 2024-09-13T14:33:48.606 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.606 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.606 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:33:48.606 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.588706+0000", 2024-09-13T14:33:48.606 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0024410109999999999 2024-09-13T14:33:48.606 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:33:48.606 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:33:48.607 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:33:48.607 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.607 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.607 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35037.0:166548 6.c 6:36e77202:::benchmark_data_smithi102_194590_object55924:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-13T14:33:48.607 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:33:48.579559+0000", 2024-09-13T14:33:48.607 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.0091494369999999998, 2024-09-13T14:33:48.607 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0093109179999999996, 2024-09-13T14:33:48.607 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:33:48.607 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:33:48.607 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:33:48.607 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:33:48.607 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35037", 2024-09-13T14:33:48.607 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:33:48.607 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 166548 2024-09-13T14:33:48.608 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.608 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:33:48.608 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.608 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:33:48.608 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.579559+0000", 2024-09-13T14:33:48.608 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:33:48.608 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.608 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.608 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:33:48.608 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.579559+0000", 2024-09-13T14:33:48.608 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:33:48.608 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.608 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.608 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:33:48.609 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.579560+0000", 2024-09-13T14:33:48.609 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.6899999999999998e-07 2024-09-13T14:33:48.609 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.609 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.609 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:33:48.609 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.579562+0000", 2024-09-13T14:33:48.609 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.0640000000000001e-06 2024-09-13T14:33:48.609 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.609 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.609 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:33:48.609 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.579569+0000", 2024-09-13T14:33:48.609 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.8029999999999999e-06 2024-09-13T14:33:48.610 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.610 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.610 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:33:48.610 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.579574+0000", 2024-09-13T14:33:48.610 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.0309999999999998e-06 2024-09-13T14:33:48.610 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.610 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.610 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:33:48.610 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.582298+0000", 2024-09-13T14:33:48.610 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0027239460000000001 2024-09-13T14:33:48.610 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.610 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.610 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:33:48.610 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.582960+0000", 2024-09-13T14:33:48.610 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00066216799999999998 2024-09-13T14:33:48.611 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.611 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.611 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 5,6,7", 2024-09-13T14:33:48.611 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.583085+0000", 2024-09-13T14:33:48.611 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000124731 2024-09-13T14:33:48.611 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.611 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.611 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:33:48.611 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.585258+0000", 2024-09-13T14:33:48.611 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0021731540000000001 2024-09-13T14:33:48.611 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.611 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.611 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:33:48.611 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.587570+0000", 2024-09-13T14:33:48.612 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.002312789 2024-09-13T14:33:48.612 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:33:48.612 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:33:48.612 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:33:48.612 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.612 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.612 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35037.0:166549 6.f 6:f392621a:::benchmark_data_smithi102_194590_object55925:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-13T14:33:48.612 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:33:48.582706+0000", 2024-09-13T14:33:48.612 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.0060024359999999999, 2024-09-13T14:33:48.612 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0062092320000000003, 2024-09-13T14:33:48.612 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:33:48.612 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:33:48.612 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:33:48.613 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:33:48.613 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35037", 2024-09-13T14:33:48.613 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:33:48.613 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 166549 2024-09-13T14:33:48.613 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.613 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:33:48.613 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.616 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:33:48.616 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.582706+0000", 2024-09-13T14:33:48.616 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:33:48.616 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.617 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.617 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:33:48.617 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.582706+0000", 2024-09-13T14:33:48.617 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:33:48.617 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.617 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.617 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:33:48.617 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.582708+0000", 2024-09-13T14:33:48.617 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.793e-06 2024-09-13T14:33:48.617 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.617 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.617 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:33:48.618 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.582713+0000", 2024-09-13T14:33:48.618 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.2610000000000001e-06 2024-09-13T14:33:48.618 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.618 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.618 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:33:48.618 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.582725+0000", 2024-09-13T14:33:48.618 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1283e-05 2024-09-13T14:33:48.618 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.618 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.618 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:33:48.618 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.582733+0000", 2024-09-13T14:33:48.618 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.551e-06 2024-09-13T14:33:48.618 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.619 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.619 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:33:48.619 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.582797+0000", 2024-09-13T14:33:48.619 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.3357999999999999e-05 2024-09-13T14:33:48.619 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.619 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.619 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:33:48.619 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.582915+0000", 2024-09-13T14:33:48.619 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000118336 2024-09-13T14:33:48.619 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.619 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.619 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:33:48.619 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.583015+0000", 2024-09-13T14:33:48.619 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0001 2024-09-13T14:33:48.620 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.620 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.620 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:33:48.620 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.585471+0000", 2024-09-13T14:33:48.620 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0024557060000000002 2024-09-13T14:33:48.620 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:33:48.620 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:33:48.620 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:33:48.620 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.620 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.620 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.35037.0:166539 6.5 e186/161 6:a5c037dd:::benchmark_data_smithi102_194590_object55915:head v 186'10334, mlcod=186'10330)", 2024-09-13T14:33:48.620 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:33:48.588218+0000", 2024-09-13T14:33:48.620 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.00049098499999999999, 2024-09-13T14:33:48.621 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000733211, 2024-09-13T14:33:48.621 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:33:48.621 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:33:48.621 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "started", 2024-09-13T14:33:48.621 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:33:48.621 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.621 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:33:48.621 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.588218+0000", 2024-09-13T14:33:48.621 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:33:48.621 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.621 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.621 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:33:48.621 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.588218+0000", 2024-09-13T14:33:48.622 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:33:48.622 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.622 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.622 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:33:48.622 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.588218+0000", 2024-09-13T14:33:48.622 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.9500000000000003e-07 2024-09-13T14:33:48.622 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.622 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.622 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:33:48.622 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.588225+0000", 2024-09-13T14:33:48.623 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.3380000000000001e-06 2024-09-13T14:33:48.623 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.623 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.623 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:33:48.623 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.588234+0000", 2024-09-13T14:33:48.623 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.217e-06 2024-09-13T14:33:48.624 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.624 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.624 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:33:48.624 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.588239+0000", 2024-09-13T14:33:48.624 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.4909999999999996e-06 2024-09-13T14:33:48.624 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.624 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.624 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:33:48.624 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.588296+0000", 2024-09-13T14:33:48.624 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.6507e-05 2024-09-13T14:33:48.624 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.624 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.625 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:33:48.625 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.588314+0000", 2024-09-13T14:33:48.625 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.8544000000000001e-05 2024-09-13T14:33:48.625 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:33:48.625 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:33:48.625 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:33:48.625 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.625 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.625 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop_reply(client.35037.0:166548 6.c e186/183)", 2024-09-13T14:33:48.625 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:33:48.588850+0000", 2024-09-13T14:33:48.625 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 4294967295.9998593, 2024-09-13T14:33:48.625 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00012839899999999999, 2024-09-13T14:33:48.625 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:33:48.626 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:33:48.626 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "no flag points reached", 2024-09-13T14:33:48.626 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:33:48.626 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.626 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:33:48.626 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.588850+0000", 2024-09-13T14:33:48.626 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:33:48.626 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.626 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.626 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:33:48.626 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.588850+0000", 2024-09-13T14:33:48.626 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:33:48.626 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.626 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.627 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:33:48.627 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.588858+0000", 2024-09-13T14:33:48.627 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.0819999999999999e-06 2024-09-13T14:33:48.627 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.627 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.627 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:33:48.627 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.588862+0000", 2024-09-13T14:33:48.627 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.5210000000000004e-06 2024-09-13T14:33:48.627 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:33:48.627 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:33:48.627 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:33:48.627 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:48.588872+0000", 2024-09-13T14:33:48.627 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.5920000000000007e-06 2024-09-13T14:33:48.627 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:33:48.627 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:33:48.628 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:33:48.628 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:33:48.628 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-09-13T14:33:48.628 INFO:teuthology.orchestra.run.smithi102.stdout: "num_ops": 7 2024-09-13T14:33:48.628 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:33:48.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:48.629 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:48.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:48.730 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:48.775 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:48.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:48.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:48.877 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:48.901 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:48.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:48.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:49.003 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:49.032 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:49.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:49.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:49.134 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:49.160 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:49.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:49.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:49.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:48 smithi102 ceph-mon[110183]: pgmap v590: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 peering, 1 active+recovering+undersized+degraded+remapped, 6 active+remapped+backfill_wait, 79 active+clean; 3.4 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 730 op/s; 2094/169245 objects degraded (1.237%); 24010/169245 objects misplaced (14.187%); 7.2 MiB/s, 112 objects/s recovering 2024-09-13T14:33:49.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:48 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 2094/169245 objects degraded (1.237%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:33:49.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:48 smithi102 ceph-mon[115269]: pgmap v590: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 peering, 1 active+recovering+undersized+degraded+remapped, 6 active+remapped+backfill_wait, 79 active+clean; 3.4 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 730 op/s; 2094/169245 objects degraded (1.237%); 24010/169245 objects misplaced (14.187%); 7.2 MiB/s, 112 objects/s recovering 2024-09-13T14:33:49.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:48 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 2094/169245 objects degraded (1.237%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:33:49.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:48 smithi195 ceph-mon[104262]: pgmap v590: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 peering, 1 active+recovering+undersized+degraded+remapped, 6 active+remapped+backfill_wait, 79 active+clean; 3.4 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 730 op/s; 2094/169245 objects degraded (1.237%); 24010/169245 objects misplaced (14.187%); 7.2 MiB/s, 112 objects/s recovering 2024-09-13T14:33:49.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:48 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 2094/169245 objects degraded (1.237%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:33:49.262 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:33:49.300 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:49.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:49.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:33:49.402 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:49.428 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:49.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:49.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:49.529 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:49.559 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:49.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:49.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:49.591 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T14:33:49.621 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 6, 7] out_osds: [1, 2, 5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:33:49.622 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:33:49.622 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-13T14:33:49.622 INFO:tasks.thrashosds.thrasher:Setting osd 6 primary_affinity to 0.000000 2024-09-13T14:33:49.622 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 6 0 2024-09-13T14:33:49.660 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:49.726 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:49.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:49.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:49.828 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:33:49.867 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:49.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:49.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:33:49.969 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:49.998 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:49.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:49.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:50.099 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:50.134 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:50.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:50.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:50.208 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:33:50.223 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:33:50.236 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:50.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:50.311 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:50.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:50.412 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:50.454 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:50.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:50.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:50.556 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:33:50.597 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:50.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:50.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:33:50.699 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:50.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:50.728 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:50.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:50.829 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:33:50.854 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is set 2024-09-13T14:33:50.872 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:50.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:50.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:33:50.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:50 smithi102 ceph-mon[110183]: pgmap v591: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 peering, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 78 active+clean; 3.4 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 578 op/s; 2094/168057 objects degraded (1.246%); 27588/168057 objects misplaced (16.416%); 5.0 MiB/s, 78 objects/s recovering 2024-09-13T14:33:50.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:50 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3844983525' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:33:50.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:50 smithi102 ceph-mon[115269]: pgmap v591: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 peering, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 78 active+clean; 3.4 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 578 op/s; 2094/168057 objects degraded (1.246%); 27588/168057 objects misplaced (16.416%); 5.0 MiB/s, 78 objects/s recovering 2024-09-13T14:33:50.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:50 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3844983525' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:33:50.974 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:51.017 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:51.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:51.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:51.119 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:51.162 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:51.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:51.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:51.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:50 smithi195 ceph-mon[104262]: pgmap v591: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 peering, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 78 active+clean; 3.4 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 578 op/s; 2094/168057 objects degraded (1.246%); 27588/168057 objects misplaced (16.416%); 5.0 MiB/s, 78 objects/s recovering 2024-09-13T14:33:51.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:50 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3844983525' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:33:51.265 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:51.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:51.294 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:51.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:51.395 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:33:51.430 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:51.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:51.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:33:51.532 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:51.560 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:51.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:51.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:51.662 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:51.687 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:51.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:51.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:51.789 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:51.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:51.816 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:51.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:51.917 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:51.953 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:51.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:51.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:52.054 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:52.073 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:51 smithi102 ceph-mon[115269]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:33:52.073 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:51 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3844983525' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:33:52.074 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:51 smithi102 ceph-mon[115269]: osdmap e187: 8 total, 8 up, 5 in 2024-09-13T14:33:52.074 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:51 smithi102 ceph-mon[110183]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:33:52.074 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:51 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3844983525' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:33:52.074 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:51 smithi102 ceph-mon[110183]: osdmap e187: 8 total, 8 up, 5 in 2024-09-13T14:33:52.094 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:52.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:52.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:52.196 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:52.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:52.238 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:52.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:52.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:51 smithi195 ceph-mon[104262]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:33:52.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:51 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3844983525' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:33:52.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:51 smithi195 ceph-mon[104262]: osdmap e187: 8 total, 8 up, 5 in 2024-09-13T14:33:52.339 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:52.367 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:52.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:52.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:52.469 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:52.506 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:52.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:52.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:52.608 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:52.635 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:52.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:52.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:52.737 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:33:52.776 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:52.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:52.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:33:52.878 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:52.904 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:52.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:52.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:53.006 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:53.039 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:53.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:53.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:53.141 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:53.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:52 smithi102 ceph-mon[110183]: pgmap v593: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 peering, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 78 active+clean; 3.4 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 534 op/s; 2094/168057 objects degraded (1.246%); 27588/168057 objects misplaced (16.416%); 4.6 MiB/s, 72 objects/s recovering 2024-09-13T14:33:53.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:52 smithi102 ceph-mon[115269]: pgmap v593: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 peering, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 78 active+clean; 3.4 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 534 op/s; 2094/168057 objects degraded (1.246%); 27588/168057 objects misplaced (16.416%); 4.6 MiB/s, 72 objects/s recovering 2024-09-13T14:33:53.176 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:53.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:53.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:53.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:52 smithi195 ceph-mon[104262]: pgmap v593: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 peering, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 78 active+clean; 3.4 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 534 op/s; 2094/168057 objects degraded (1.246%); 27588/168057 objects misplaced (16.416%); 4.6 MiB/s, 72 objects/s recovering 2024-09-13T14:33:53.278 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:53.323 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:53.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:53.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:53.425 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:33:53.462 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:53.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:53.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:33:53.564 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:53.591 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:53.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:53.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:53.693 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:33:53.729 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:53.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:53.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:33:53.831 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:33:53.867 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:53.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:53.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:33:53.969 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:53.995 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:53.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:53.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:54.097 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:33:54.137 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:54.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:54.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:33:54.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:53 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 2094/168057 objects degraded (1.246%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:33:54.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:53 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 2094/168057 objects degraded (1.246%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:33:54.239 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:33:54.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:53 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 2094/168057 objects degraded (1.246%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:33:54.275 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:54.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:54.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:33:54.377 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:54.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:54.405 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:54.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:54.506 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:54.531 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:54.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:54.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:54.633 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:33:54.666 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:54.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:54.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:33:54.768 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:33:54.808 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:54.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:54.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:33:54.910 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:54.944 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:54.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:54.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:55.046 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:55.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:55.075 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:55.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:55.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:54 smithi102 ceph-mon[110183]: pgmap v594: 89 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 3.2 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 1.01k op/s; 2094/154839 objects degraded (1.352%); 25186/154839 objects misplaced (16.266%); 7.3 MiB/s, 116 objects/s recovering 2024-09-13T14:33:55.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:54 smithi102 ceph-mon[115269]: pgmap v594: 89 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 3.2 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 1.01k op/s; 2094/154839 objects degraded (1.352%); 25186/154839 objects misplaced (16.266%); 7.3 MiB/s, 116 objects/s recovering 2024-09-13T14:33:55.176 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:55.218 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:55.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:55.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:55.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:54 smithi195 ceph-mon[104262]: pgmap v594: 89 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 3.2 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 1.01k op/s; 2094/154839 objects degraded (1.352%); 25186/154839 objects misplaced (16.266%); 7.3 MiB/s, 116 objects/s recovering 2024-09-13T14:33:55.320 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:33:55.346 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:33:55.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:55.366 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:55.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:33:55.380 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:33:55.381 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [], 2024-09-13T14:33:55.381 INFO:teuthology.orchestra.run.smithi102.stdout: "complaint_time": 30, 2024-09-13T14:33:55.381 INFO:teuthology.orchestra.run.smithi102.stdout: "num_blocked_ops": 0 2024-09-13T14:33:55.381 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:33:55.411 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:33:55.467 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:55.499 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:55.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:55.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:55.601 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:55.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:55.638 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:55.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:55.739 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:55.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:55.772 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:55.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:55.873 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:55.922 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:55.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:55.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:56.024 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:56.058 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:56.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:56.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:56.137 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:55 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:33:56.138 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:55 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:33:56.148 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:55 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:33:56.160 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:33:56.198 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:56.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:56.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:33:56.300 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:56.336 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:56.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:56.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:56.438 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:33:56.477 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:56.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:56.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:33:56.579 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:56.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:56.614 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:56.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:56.715 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:33:56.755 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:56.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:56.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:33:56.857 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:56.887 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:56.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:56.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:56.895 INFO:teuthology.orchestra.run.smithi102.stderr:set osd.6 primary-affinity to 0 (802) 2024-09-13T14:33:56.895 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:33:56 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:33:56] "GET /metrics HTTP/1.1" 200 36557 "" "Prometheus/2.51.0" 2024-09-13T14:33:56.989 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:57.013 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:57.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:57.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:57.096 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T14:33:57.114 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:33:57.136 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:57.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:57.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:33:57.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:56 smithi102 ceph-mon[115269]: pgmap v595: 89 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 3.2 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 774 op/s; 2094/154839 objects degraded (1.352%); 25186/154839 objects misplaced (16.266%); 5.4 MiB/s, 84 objects/s recovering 2024-09-13T14:33:57.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:56 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/149114219' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 0.0} : dispatch 2024-09-13T14:33:57.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:56 smithi102 ceph-mon[110183]: pgmap v595: 89 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 3.2 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 774 op/s; 2094/154839 objects degraded (1.352%); 25186/154839 objects misplaced (16.266%); 5.4 MiB/s, 84 objects/s recovering 2024-09-13T14:33:57.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:56 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/149114219' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 0.0} : dispatch 2024-09-13T14:33:57.238 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:57.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:56 smithi195 ceph-mon[104262]: pgmap v595: 89 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 3.2 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 774 op/s; 2094/154839 objects degraded (1.352%); 25186/154839 objects misplaced (16.266%); 5.4 MiB/s, 84 objects/s recovering 2024-09-13T14:33:57.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:56 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/149114219' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 0.0} : dispatch 2024-09-13T14:33:57.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:57.259 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:57.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:33:57.361 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:57.382 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:57.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:57.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:57.484 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:57.505 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:57.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:57.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:57.607 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:57.628 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:57.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:57.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:57.730 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:57.752 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:57.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:57.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:57.853 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:33:57.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:57.876 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:57.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:33:57.977 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:57.998 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:57.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:57.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:58.100 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:33:58.121 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:58.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:58.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:33:58.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:57 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/149114219' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.0}]': finished 2024-09-13T14:33:58.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:57 smithi102 ceph-mon[110183]: osdmap e188: 8 total, 8 up, 5 in 2024-09-13T14:33:58.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:57 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/149114219' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.0}]': finished 2024-09-13T14:33:58.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:57 smithi102 ceph-mon[115269]: osdmap e188: 8 total, 8 up, 5 in 2024-09-13T14:33:58.223 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:58.244 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:58.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:58.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:58.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:57 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/149114219' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.0}]': finished 2024-09-13T14:33:58.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:57 smithi195 ceph-mon[104262]: osdmap e188: 8 total, 8 up, 5 in 2024-09-13T14:33:58.346 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:33:58.367 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:58.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:58.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:33:58.469 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:58.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:58.493 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:58.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:58.594 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:33:58.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:58.617 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:58.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:33:58.718 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:58.730 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:33:58.744 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:58.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:58.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:58.846 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:33:58.867 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:58.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:58.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:33:58.969 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:58.998 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:58.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:58.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:59.014 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:58 smithi102 ceph-mon[115269]: pgmap v597: 89 pgs: 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 78 active+clean; 2.9 GiB data, 9.8 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 1.02k op/s; 2163/143598 objects degraded (1.506%); 23050/143598 objects misplaced (16.052%); 7.0 MiB/s, 110 objects/s recovering 2024-09-13T14:33:59.014 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:58 smithi102 ceph-mon[115269]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-13T14:33:59.015 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:58 smithi102 ceph-mon[115269]: osdmap e189: 8 total, 8 up, 5 in 2024-09-13T14:33:59.015 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:58 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 2163/143598 objects degraded (1.506%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:33:59.015 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:58 smithi102 ceph-mon[110183]: pgmap v597: 89 pgs: 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 78 active+clean; 2.9 GiB data, 9.8 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 1.02k op/s; 2163/143598 objects degraded (1.506%); 23050/143598 objects misplaced (16.052%); 7.0 MiB/s, 110 objects/s recovering 2024-09-13T14:33:59.015 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:58 smithi102 ceph-mon[110183]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-13T14:33:59.015 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:58 smithi102 ceph-mon[110183]: osdmap e189: 8 total, 8 up, 5 in 2024-09-13T14:33:59.015 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:58 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 2163/143598 objects degraded (1.506%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:33:59.100 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:59.120 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:59.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:59.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:59.222 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:33:59.245 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:59.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:59.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:33:59.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:58 smithi195 ceph-mon[104262]: pgmap v597: 89 pgs: 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 78 active+clean; 2.9 GiB data, 9.8 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 1.02k op/s; 2163/143598 objects degraded (1.506%); 23050/143598 objects misplaced (16.052%); 7.0 MiB/s, 110 objects/s recovering 2024-09-13T14:33:59.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:58 smithi195 ceph-mon[104262]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-13T14:33:59.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:58 smithi195 ceph-mon[104262]: osdmap e189: 8 total, 8 up, 5 in 2024-09-13T14:33:59.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:58 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 2163/143598 objects degraded (1.506%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:33:59.347 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:59.368 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:59.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:59.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:59.470 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:33:59.491 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:59.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:59.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:33:59.593 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:33:59.614 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:59.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:59.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:33:59.716 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:33:59.722 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:33:59.737 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:59.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:59.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:33:59.839 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:33:59.862 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:33:59.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:59.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:33:59.964 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:33:59.985 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:33:59.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:33:59.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:00.087 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:00.108 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:00.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:00.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:00.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:59 smithi102 ceph-mon[110183]: 6.a continuing backfill to osd.3 from (175'6032,187'10903] MIN to 187'10903 2024-09-13T14:34:00.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:59 smithi102 ceph-mon[110183]: osdmap e190: 8 total, 8 up, 5 in 2024-09-13T14:34:00.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:33:59 smithi102 ceph-mon[110183]: 6.1 starting backfill to osd.3 from (175'5861,187'10726] MIN to 187'10728 2024-09-13T14:34:00.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:59 smithi102 ceph-mon[115269]: 6.a continuing backfill to osd.3 from (175'6032,187'10903] MIN to 187'10903 2024-09-13T14:34:00.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:59 smithi102 ceph-mon[115269]: osdmap e190: 8 total, 8 up, 5 in 2024-09-13T14:34:00.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:33:59 smithi102 ceph-mon[115269]: 6.1 starting backfill to osd.3 from (175'5861,187'10726] MIN to 187'10728 2024-09-13T14:34:00.210 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:34:00.231 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:00.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:00.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:34:00.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:59 smithi195 ceph-mon[104262]: 6.a continuing backfill to osd.3 from (175'6032,187'10903] MIN to 187'10903 2024-09-13T14:34:00.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:59 smithi195 ceph-mon[104262]: osdmap e190: 8 total, 8 up, 5 in 2024-09-13T14:34:00.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:33:59 smithi195 ceph-mon[104262]: 6.1 starting backfill to osd.3 from (175'5861,187'10726] MIN to 187'10728 2024-09-13T14:34:00.333 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:00.355 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:00.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:00.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:00.457 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:00.477 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:00.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:00.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:00.579 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:34:00.601 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:00.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:00.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:34:00.699 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:34:00.699 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:34:00.702 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:34:00.726 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:00.727 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:34:00.727 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:34:00.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:00.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:34:00.748 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:34:00.749 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:34:00.779 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:34:00.779 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:34:00.798 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:34:00.798 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:34:00.816 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:34:00.816 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:34:00.828 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:00.834 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:34:00.835 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:34:00.851 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:00.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:00.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:00.853 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:34:00.853 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:34:00.953 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:00.975 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:00.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:00.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:01.077 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:01.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:01.100 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:01.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:01.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:00 smithi102 ceph-mon[110183]: pgmap v600: 89 pgs: 5 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 74 active+clean; 2.9 GiB data, 9.7 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 705 op/s; 2163/142131 objects degraded (1.522%); 22796/142131 objects misplaced (16.039%); 4.8 MiB/s, 72 objects/s recovering 2024-09-13T14:34:01.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:00 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1553141548' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:34:01.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:00 smithi102 ceph-mon[115269]: pgmap v600: 89 pgs: 5 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 74 active+clean; 2.9 GiB data, 9.7 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 705 op/s; 2163/142131 objects degraded (1.522%); 22796/142131 objects misplaced (16.039%); 4.8 MiB/s, 72 objects/s recovering 2024-09-13T14:34:01.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:00 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1553141548' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:34:01.201 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:01.222 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:01.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:01.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:01.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:00 smithi195 ceph-mon[104262]: pgmap v600: 89 pgs: 5 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 74 active+clean; 2.9 GiB data, 9.7 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 705 op/s; 2163/142131 objects degraded (1.522%); 22796/142131 objects misplaced (16.039%); 4.8 MiB/s, 72 objects/s recovering 2024-09-13T14:34:01.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:00 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1553141548' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:34:01.325 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:01.347 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:01.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:01.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:01.448 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:01.470 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:01.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:01.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:01.535 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:34:01.535 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:34:01.572 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:34:01.593 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:01.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:01.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:34:01.695 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:01.718 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:01.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:01.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:01.820 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:01.841 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:01.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:01.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:01.943 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:01.976 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:01.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:01.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:02.078 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:02.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:02.110 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:02.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:02.211 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:02.240 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:02.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:02.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:02.341 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:02.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:02.380 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:02.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:02.481 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:02.513 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:02.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:02.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:02.615 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:34:02.649 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:02.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:02.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:34:02.738 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 6, 7] out_osds: [1, 2, 5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:34:02.739 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:34:02.739 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_1 2024-09-13T14:34:02.739 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:34:02.750 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:02.818 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:02.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:02.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:02.920 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:34:02.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:02.969 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:02.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:34:03.070 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:03.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:03.110 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:03.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:03.211 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:03.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:02 smithi195 ceph-mon[104262]: pgmap v601: 89 pgs: 5 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 74 active+clean; 2.9 GiB data, 9.7 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 705 op/s; 2163/142131 objects degraded (1.522%); 22796/142131 objects misplaced (16.039%); 4.8 MiB/s, 72 objects/s recovering 2024-09-13T14:34:03.251 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:03.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:03.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:03.353 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:03.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:03.406 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:03.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:03.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:02 smithi102 ceph-mon[110183]: pgmap v601: 89 pgs: 5 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 74 active+clean; 2.9 GiB data, 9.7 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 705 op/s; 2163/142131 objects degraded (1.522%); 22796/142131 objects misplaced (16.039%); 4.8 MiB/s, 72 objects/s recovering 2024-09-13T14:34:03.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:02 smithi102 ceph-mon[115269]: pgmap v601: 89 pgs: 5 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 74 active+clean; 2.9 GiB data, 9.7 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 705 op/s; 2163/142131 objects degraded (1.522%); 22796/142131 objects misplaced (16.039%); 4.8 MiB/s, 72 objects/s recovering 2024-09-13T14:34:03.508 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:03.539 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:03.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:03.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:03.641 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:03.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:03.678 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:03.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:03.779 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:03.805 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:03.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:03.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:03.907 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:03.963 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:03.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:03.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:04.065 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:04.096 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:04.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:04.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:04.198 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:04.212 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:03 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:34:04.212 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:03 smithi102 ceph-mon[110183]: Health check update: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-09-13T14:34:04.212 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:03 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 2163/142131 objects degraded (1.522%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:34:04.212 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:03 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:34:04.212 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:03 smithi102 ceph-mon[115269]: Health check update: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-09-13T14:34:04.212 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:03 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 2163/142131 objects degraded (1.522%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:34:04.227 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:04.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:04.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:04.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:03 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:34:04.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:03 smithi195 ceph-mon[104262]: Health check update: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-09-13T14:34:04.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:03 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 2163/142131 objects degraded (1.522%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:34:04.329 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:04.358 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:04.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:04.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:04.460 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:04.488 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:04.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:04.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:04.590 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:34:04.635 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:04.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:04.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:34:04.737 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:04.789 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:04.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:04.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:04.891 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:04.921 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:04.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:04.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:05.023 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:05.049 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:05.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:05.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:05.151 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:05.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:04 smithi102 ceph-mon[110183]: pgmap v602: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 2.9 GiB data, 9.6 GiB used, 437 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 510 op/s; 4862/140898 objects degraded (3.451%); 20956/140898 objects misplaced (14.873%); 8.7 MiB/s, 143 objects/s recovering 2024-09-13T14:34:05.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:04 smithi102 ceph-mon[110183]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 5 pgs peering) 2024-09-13T14:34:05.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:04 smithi102 ceph-mon[115269]: pgmap v602: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 2.9 GiB data, 9.6 GiB used, 437 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 510 op/s; 4862/140898 objects degraded (3.451%); 20956/140898 objects misplaced (14.873%); 8.7 MiB/s, 143 objects/s recovering 2024-09-13T14:34:05.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:04 smithi102 ceph-mon[115269]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 5 pgs peering) 2024-09-13T14:34:05.212 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:05.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:05.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:05.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:04 smithi195 ceph-mon[104262]: pgmap v602: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 2.9 GiB data, 9.6 GiB used, 437 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 510 op/s; 4862/140898 objects degraded (3.451%); 20956/140898 objects misplaced (14.873%); 8.7 MiB/s, 143 objects/s recovering 2024-09-13T14:34:05.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:04 smithi195 ceph-mon[104262]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 5 pgs peering) 2024-09-13T14:34:05.319 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:05.349 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T14:34:05.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:05.370 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:05.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:05.372 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:34:05.470 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:05.496 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:05.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:05.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:05.598 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:05.625 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:05.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:05.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:05.726 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:05.758 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:05.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:05.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:05.938 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:05.959 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is unset 2024-09-13T14:34:05.974 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:05.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:05.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:05.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:05 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3368964214' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:34:05.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:05 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:34:06.076 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:06.101 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:34:06.127 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:05 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3368964214' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:34:06.127 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:05 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:34:06.127 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:05 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3368964214' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:34:06.128 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:05 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:34:06.140 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:06.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:06.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:06.171 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:34:06.242 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:06.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:06.293 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:06.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:06.394 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:06.423 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:06.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:06.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:06.524 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:06.551 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:06.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:06.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:06.653 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:06.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:06.697 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:06.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:06.768 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:34:06 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:34:06] "GET /metrics HTTP/1.1" 200 36557 "" "Prometheus/2.51.0" 2024-09-13T14:34:06.799 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:06.830 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:06.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:06.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:06.931 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:34:06.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:06.974 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:06.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:34:07.075 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:07.103 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:07.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:07.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:07.131 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:06 smithi102 ceph-mon[110183]: pgmap v603: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 2.9 GiB data, 9.6 GiB used, 437 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 113 op/s; 4862/140898 objects degraded (3.451%); 20956/140898 objects misplaced (14.873%); 5.6 MiB/s, 94 objects/s recovering 2024-09-13T14:34:07.131 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:06 smithi102 ceph-mon[110183]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:34:07.131 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:06 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:34:07.131 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:06 smithi102 ceph-mon[110183]: osdmap e191: 8 total, 8 up, 5 in 2024-09-13T14:34:07.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:06 smithi102 ceph-mon[115269]: pgmap v603: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 2.9 GiB data, 9.6 GiB used, 437 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 113 op/s; 4862/140898 objects degraded (3.451%); 20956/140898 objects misplaced (14.873%); 5.6 MiB/s, 94 objects/s recovering 2024-09-13T14:34:07.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:06 smithi102 ceph-mon[115269]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:34:07.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:06 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:34:07.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:06 smithi102 ceph-mon[115269]: osdmap e191: 8 total, 8 up, 5 in 2024-09-13T14:34:07.205 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:07.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:07.235 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:07.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:07.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:06 smithi195 ceph-mon[104262]: pgmap v603: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 2.9 GiB data, 9.6 GiB used, 437 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 113 op/s; 4862/140898 objects degraded (3.451%); 20956/140898 objects misplaced (14.873%); 5.6 MiB/s, 94 objects/s recovering 2024-09-13T14:34:07.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:06 smithi195 ceph-mon[104262]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:34:07.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:06 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:34:07.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:06 smithi195 ceph-mon[104262]: osdmap e191: 8 total, 8 up, 5 in 2024-09-13T14:34:07.336 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:07.362 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:07.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:07.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:07.464 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:07.491 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:07.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:07.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:07.593 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:34:07.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:07.615 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:07.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:34:07.717 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:07.747 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:07.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:07.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:07.849 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:07.876 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:07.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:07.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:07.978 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:08.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:08.019 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:08.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:08.120 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:08.157 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:08.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:08.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:08.258 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:08.287 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:08.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:08.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:08.389 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:08.413 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:08.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:08.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:08.515 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:08.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:08.543 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:08.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:08.644 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:08.674 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:08.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:08.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:08.776 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:08.806 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:08.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:08.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:08.817 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:34:08.909 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:34:08.957 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:08.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:08.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:34:09.004 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:34:09.006 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:34:09.058 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:09.100 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:09.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:09.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:09.202 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:09.240 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:09.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:09.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:09.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:08 smithi195 ceph-mon[104262]: pgmap v605: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 2.6 GiB data, 9.1 GiB used, 438 GiB / 447 GiB avail; 968 B/s rd, 0 B/s wr, 508 op/s; 4587/129249 objects degraded (3.549%); 19078/129249 objects misplaced (14.761%); 9.2 MiB/s, 153 objects/s recovering 2024-09-13T14:34:09.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:08 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 4587/129249 objects degraded (3.549%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:34:09.344 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:08 smithi102 ceph-mon[110183]: pgmap v605: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 2.6 GiB data, 9.1 GiB used, 438 GiB / 447 GiB avail; 968 B/s rd, 0 B/s wr, 508 op/s; 4587/129249 objects degraded (3.549%); 19078/129249 objects misplaced (14.761%); 9.2 MiB/s, 153 objects/s recovering 2024-09-13T14:34:09.344 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:08 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 4587/129249 objects degraded (3.549%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:34:09.344 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:08 smithi102 ceph-mon[115269]: pgmap v605: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 2.6 GiB data, 9.1 GiB used, 438 GiB / 447 GiB avail; 968 B/s rd, 0 B/s wr, 508 op/s; 4587/129249 objects degraded (3.549%); 19078/129249 objects misplaced (14.761%); 9.2 MiB/s, 153 objects/s recovering 2024-09-13T14:34:09.344 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:08 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 4587/129249 objects degraded (3.549%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:34:09.345 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:09.375 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:09.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:09.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:09.477 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:09.502 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:09.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:09.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:09.535 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:34:09.535 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:34:09.598 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:34:09.604 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:09.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:09.674 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:09.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:09.775 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:09.811 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:09.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:09.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:09.862 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":605,"stamp":"2024-09-13T14:34:07.372432+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2826377213,"num_objects":43083,"num_object_clones":0,"num_object_copies":129249,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4587,"num_objects_misplaced":19078,"num_objects_unfound":0,"num_objects_dirty":43083,"num_whiteouts":0,"num_read":6377,"num_read_kb":13834,"num_write":179312,"num_write_kb":7094786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10385,"num_bytes_recovered":677907962,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90477,"ondisk_log_size":90477,"up":267,"acting":265,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":270,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":9528292,"kb_used_data":7879224,"kb_used_omap":18,"kb_used_meta":1648941,"kb_avail":459197468,"statfs":{"total":479975178240,"available":470218207232,"internally_reserved":0,"allocated":8068325376,"data_stored":8063513376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18690,"internal_metadata":1688516350},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-281411584,"num_objects":-4294,"num_object_clones":0,"num_object_copies":-12882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2424,"num_objects_misplaced":-3718,"num_objects_unfound":0,"num_objects_dirty":-4294,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":4294,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1295,"num_bytes_recovered":81461229,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.461463"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":260,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-13T14:34:05.958760+0000","last_change":"2024-09-13T14:33:43.836678+0000","last_active":"2024-09-13T14:34:05.958760+0000","last_peered":"2024-09-13T14:34:05.958760+0000","last_clean":"2024-09-13T14:34:05.958760+0000","last_became_active":"2024-09-13T14:33:43.835921+0000","last_became_peered":"2024-09-13T14:33:43.835921+0000","last_unstale":"2024-09-13T14:34:05.958760+0000","last_undegraded":"2024-09-13T14:34:05.958760+0000","last_fullsized":"2024-09-13T14:34:05.958760+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:00:13.517399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":174,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-09-13T14:33:58.954001+0000","last_change":"2024-09-13T14:27:12.846760+0000","last_active":"2024-09-13T14:33:58.954001+0000","last_peered":"2024-09-13T14:33:58.954001+0000","last_clean":"2024-09-13T14:33:58.954001+0000","last_became_active":"2024-09-13T14:27:12.838041+0000","last_became_peered":"2024-09-13T14:27:12.838041+0000","last_unstale":"2024-09-13T14:33:58.954001+0000","last_undegraded":"2024-09-13T14:33:58.954001+0000","last_fullsized":"2024-09-13T14:33:58.954001+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:22.402859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":82,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-09-13T14:33:58.953691+0000","last_change":"2024-09-13T14:33:43.834858+0000","last_active":"2024-09-13T14:33:58.953691+0000","last_peered":"2024-09-13T14:33:58.953691+0000","last_clean":"2024-09-13T14:33:58.953691+0000","last_became_active":"2024-09-13T14:33:43.834249+0000","last_became_peered":"2024-09-13T14:33:43.834249+0000","last_unstale":"2024-09-13T14:33:58.953691+0000","last_undegraded":"2024-09-13T14:33:58.953691+0000","last_fullsized":"2024-09-13T14:33:58.953691+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:00:41.666063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":244,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-13T14:34:05.960915+0000","last_change":"2024-09-13T14:33:43.834086+0000","last_active":"2024-09-13T14:34:05.960915+0000","last_peered":"2024-09-13T14:34:05.960915+0000","last_clean":"2024-09-13T14:34:05.960915+0000","last_became_active":"2024-09-13T14:33:43.833787+0000","last_became_peered":"2024-09-13T14:33:43.833787+0000","last_unstale":"2024-09-13T14:34:05.960915+0000","last_undegraded":"2024-09-13T14:34:05.960915+0000","last_fullsized":"2024-09-13T14:34:05.960915+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:24:49.307711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":239,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-13T14:34:05.962763+0000","last_change":"2024-09-13T14:32:01.084730+0000","last_active":"2024-09-13T14:34:05.962763+0000","last_peered":"2024-09-13T14:34:05.962763+0000","last_clean":"2024-09-13T14:34:05.962763+0000","last_became_active":"2024-09-13T14:32:01.084049+0000","last_became_peered":"2024-09-13T14:32:01.084049+0000","last_unstale":"2024-09-13T14:34:05.962763+0000","last_undegraded":"2024-09-13T14:34:05.962763+0000","last_fullsized":"2024-09-13T14:34:05.962763+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:41:50.362080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":142,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-09-13T14:33:58.953336+0000","last_change":"2024-09-13T14:11:29.244452+0000","last_active":"2024-09-13T14:33:58.953336+0000","last_peered":"2024-09-13T14:33:58.953336+0000","last_clean":"2024-09-13T14:33:58.953336+0000","last_became_active":"2024-09-13T14:11:29.244102+0000","last_became_peered":"2024-09-13T14:11:29.244102+0000","last_unstale":"2024-09-13T14:33:58.953336+0000","last_undegraded":"2024-09-13T14:33:58.953336+0000","last_fullsized":"2024-09-13T14:33:58.953336+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:13:19.165534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":260,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-13T14:34:05.963766+0000","last_change":"2024-09-13T14:32:01.085443+0000","last_active":"2024-09-13T14:34:05.963766+0000","last_peered":"2024-09-13T14:34:05.963766+0000","last_clean":"2024-09-13T14:34:05.963766+0000","last_became_active":"2024-09-13T14:32:01.085067+0000","last_became_peered":"2024-09-13T14:32:01.085067+0000","last_unstale":"2024-09-13T14:34:05.963766+0000","last_undegraded":"2024-09-13T14:34:05.963766+0000","last_fullsized":"2024-09-13T14:34:05.963766+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:46.396351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":29,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-13T14:34:05.960132+0000","last_change":"2024-09-13T14:33:57.919141+0000","last_active":"2024-09-13T14:34:05.960132+0000","last_peered":"2024-09-13T14:34:05.960132+0000","last_clean":"2024-09-13T14:34:05.960132+0000","last_became_active":"2024-09-13T14:33:57.918690+0000","last_became_peered":"2024-09-13T14:33:57.918690+0000","last_unstale":"2024-09-13T14:34:05.960132+0000","last_undegraded":"2024-09-13T14:34:05.960132+0000","last_fullsized":"2024-09-13T14:34:05.960132+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:47.891878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"191'11170","reported_seq":11228,"reported_epoch":191,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:34:06.820006+0000","last_change":"2024-09-13T14:33:58.945696+0000","last_active":"2024-09-13T14:34:06.820006+0000","last_peered":"2024-09-13T14:34:06.820006+0000","last_clean":"2024-09-13T14:33:43.509452+0000","last_became_active":"2024-09-13T14:33:58.917173+0000","last_became_peered":"2024-09-13T14:33:58.917173+0000","last_unstale":"2024-09-13T14:34:06.820006+0000","last_undegraded":"2024-09-13T14:34:06.820006+0000","last_fullsized":"2024-09-13T14:34:06.820006+0000","mapping_epoch":189,"log_start":"175'6032","ondisk_log_start":"175'6032","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_clean_scrub_stamp":"2024-09-13T14:31:03.932624+0000","objects_scrubbed":0,"log_size":5138,"log_dups_size":0,"ondisk_log_size":5138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175505408,"num_objects":2678,"num_object_clones":0,"num_object_copies":8034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2678,"num_objects_unfound":0,"num_objects_dirty":2678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11170,"num_write_kb":443136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,5],"avail_no_missing":["0","5","6"],"object_location_counts":[{"shards":"0,5,6","objects":2678}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":60,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-13T14:33:57.915497+0000","last_change":"2024-09-13T14:33:43.846070+0000","last_active":"2024-09-13T14:33:57.915497+0000","last_peered":"2024-09-13T14:33:57.915497+0000","last_clean":"2024-09-13T14:33:57.915497+0000","last_became_active":"2024-09-13T14:33:43.831382+0000","last_became_peered":"2024-09-13T14:33:43.831382+0000","last_unstale":"2024-09-13T14:33:57.915497+0000","last_undegraded":"2024-09-13T14:33:57.915497+0000","last_fullsized":"2024-09-13T14:33:57.915497+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:12:14.302309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":136,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-13T14:33:57.920640+0000","last_change":"2024-09-13T14:33:57.920322+0000","last_active":"2024-09-13T14:33:57.920640+0000","last_peered":"2024-09-13T14:33:57.920640+0000","last_clean":"2024-09-13T14:33:57.920640+0000","last_became_active":"2024-09-13T14:33:57.920033+0000","last_became_peered":"2024-09-13T14:33:57.920033+0000","last_unstale":"2024-09-13T14:33:57.920640+0000","last_undegraded":"2024-09-13T14:33:57.920640+0000","last_fullsized":"2024-09-13T14:33:57.920640+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:43:29.491796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"191'11417","reported_seq":17345,"reported_epoch":191,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T14:34:07.190826+0000","last_change":"2024-09-13T14:33:43.863275+0000","last_active":"2024-09-13T14:34:07.190826+0000","last_peered":"2024-09-13T14:34:07.190826+0000","last_clean":"2024-09-13T14:32:00.048481+0000","last_became_active":"2024-09-13T14:32:02.347758+0000","last_became_peered":"2024-09-13T14:32:02.347758+0000","last_unstale":"2024-09-13T14:34:07.190826+0000","last_undegraded":"2024-09-13T14:32:12.050733+0000","last_fullsized":"2024-09-13T14:32:02.088903+0000","mapping_epoch":171,"log_start":"176'6858","ondisk_log_start":"176'6858","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_clean_scrub_stamp":"2024-09-13T14:31:03.932624+0000","objects_scrubbed":0,"log_size":4559,"log_dups_size":0,"ondisk_log_size":4559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177405952,"num_objects":2707,"num_object_clones":0,"num_object_copies":8121,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2154,"num_objects_misplaced":10,"num_objects_unfound":0,"num_objects_dirty":2707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11417,"num_write_kb":451968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2921,"num_bytes_recovered":191430656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6],"avail_no_missing":["3","6","7"],"object_location_counts":[{"shards":"3,6","objects":543},{"shards":"3,6,7","objects":10}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":60,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-13T14:33:57.915302+0000","last_change":"2024-09-13T14:33:43.853571+0000","last_active":"2024-09-13T14:33:57.915302+0000","last_peered":"2024-09-13T14:33:57.915302+0000","last_clean":"2024-09-13T14:33:57.915302+0000","last_became_active":"2024-09-13T14:33:43.831192+0000","last_became_peered":"2024-09-13T14:33:43.831192+0000","last_unstale":"2024-09-13T14:33:57.915302+0000","last_undegraded":"2024-09-13T14:33:57.915302+0000","last_fullsized":"2024-09-13T14:33:57.915302+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:48:09.947221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":179,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-13T14:33:57.912897+0000","last_change":"2024-09-13T14:26:15.824632+0000","last_active":"2024-09-13T14:33:57.912897+0000","last_peered":"2024-09-13T14:33:57.912897+0000","last_clean":"2024-09-13T14:33:57.912897+0000","last_became_active":"2024-09-13T14:26:15.824330+0000","last_became_peered":"2024-09-13T14:26:15.824330+0000","last_unstale":"2024-09-13T14:33:57.912897+0000","last_undegraded":"2024-09-13T14:33:57.912897+0000","last_fullsized":"2024-09-13T14:33:57.912897+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:25.288116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"191'11034","reported_seq":20873,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-13T14:34:07.167919+0000","last_change":"2024-09-13T14:33:30.478010+0000","last_active":"2024-09-13T14:34:07.167919+0000","last_peered":"2024-09-13T14:34:07.167919+0000","last_clean":"2024-09-13T14:34:07.167919+0000","last_became_active":"2024-09-13T14:33:30.455304+0000","last_became_peered":"2024-09-13T14:33:30.455304+0000","last_unstale":"2024-09-13T14:34:07.167919+0000","last_undegraded":"2024-09-13T14:34:07.167919+0000","last_fullsized":"2024-09-13T14:34:07.167919+0000","mapping_epoch":179,"log_start":"176'6537","ondisk_log_start":"176'6537","created":161,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_clean_scrub_stamp":"2024-09-13T14:31:03.932624+0000","objects_scrubbed":0,"log_size":4497,"log_dups_size":0,"ondisk_log_size":4497,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:42:20.589214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173408256,"num_objects":2646,"num_object_clones":0,"num_object_copies":7938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11034,"num_write_kb":437760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4883,"num_bytes_recovered":320012288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":157,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-09-13T14:34:02.198823+0000","last_change":"2024-09-13T14:33:43.843174+0000","last_active":"2024-09-13T14:34:02.198823+0000","last_peered":"2024-09-13T14:34:02.198823+0000","last_clean":"2024-09-13T14:34:02.198823+0000","last_became_active":"2024-09-13T14:33:43.833859+0000","last_became_peered":"2024-09-13T14:33:43.833859+0000","last_unstale":"2024-09-13T14:34:02.198823+0000","last_undegraded":"2024-09-13T14:34:02.198823+0000","last_fullsized":"2024-09-13T14:34:02.198823+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:37:56.292102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":184,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-13T14:34:05.960288+0000","last_change":"2024-09-13T14:33:57.919221+0000","last_active":"2024-09-13T14:34:05.960288+0000","last_peered":"2024-09-13T14:34:05.960288+0000","last_clean":"2024-09-13T14:34:05.960288+0000","last_became_active":"2024-09-13T14:33:57.918696+0000","last_became_peered":"2024-09-13T14:33:57.918696+0000","last_unstale":"2024-09-13T14:34:05.960288+0000","last_undegraded":"2024-09-13T14:34:05.960288+0000","last_fullsized":"2024-09-13T14:34:05.960288+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:13.384674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"191'10968","reported_seq":11040,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-13T14:34:06.883279+0000","last_change":"2024-09-13T14:31:04.953002+0000","last_active":"2024-09-13T14:34:06.883279+0000","last_peered":"2024-09-13T14:34:06.883279+0000","last_clean":"2024-09-13T14:34:06.883279+0000","last_became_active":"2024-09-13T14:31:04.952667+0000","last_became_peered":"2024-09-13T14:31:04.952667+0000","last_unstale":"2024-09-13T14:34:06.883279+0000","last_undegraded":"2024-09-13T14:34:06.883279+0000","last_fullsized":"2024-09-13T14:34:06.883279+0000","mapping_epoch":161,"log_start":"175'5364","ondisk_log_start":"175'5364","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_clean_scrub_stamp":"2024-09-13T14:31:03.932624+0000","objects_scrubbed":0,"log_size":5604,"log_dups_size":0,"ondisk_log_size":5604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:22:35.630717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170262528,"num_objects":2598,"num_object_clones":0,"num_object_copies":7794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10968,"num_write_kb":434112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":92,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-13T14:34:05.959301+0000","last_change":"2024-09-13T14:33:43.836901+0000","last_active":"2024-09-13T14:34:05.959301+0000","last_peered":"2024-09-13T14:34:05.959301+0000","last_clean":"2024-09-13T14:34:05.959301+0000","last_became_active":"2024-09-13T14:33:43.836361+0000","last_became_peered":"2024-09-13T14:33:43.836361+0000","last_unstale":"2024-09-13T14:34:05.959301+0000","last_undegraded":"2024-09-13T14:34:05.959301+0000","last_fullsized":"2024-09-13T14:34:05.959301+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:39.912565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":239,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-13T14:34:05.962207+0000","last_change":"2024-09-13T14:26:15.823316+0000","last_active":"2024-09-13T14:34:05.962207+0000","last_peered":"2024-09-13T14:34:05.962207+0000","last_clean":"2024-09-13T14:34:05.962207+0000","last_became_active":"2024-09-13T14:26:15.822980+0000","last_became_peered":"2024-09-13T14:26:15.822980+0000","last_unstale":"2024-09-13T14:34:05.962207+0000","last_undegraded":"2024-09-13T14:34:05.962207+0000","last_fullsized":"2024-09-13T14:34:05.962207+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:33:47.420548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"190'11101","reported_seq":11149,"reported_epoch":190,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:34:02.563233+0000","last_change":"2024-09-13T14:33:45.040440+0000","last_active":"2024-09-13T14:34:02.563233+0000","last_peered":"2024-09-13T14:34:02.563233+0000","last_clean":"2024-09-13T14:33:43.486093+0000","last_became_active":"2024-09-13T14:33:44.952268+0000","last_became_peered":"2024-09-13T14:33:44.952268+0000","last_unstale":"2024-09-13T14:34:02.563233+0000","last_undegraded":"2024-09-13T14:34:02.563233+0000","last_fullsized":"2024-09-13T14:34:02.563233+0000","mapping_epoch":183,"log_start":"175'5891","ondisk_log_start":"175'5891","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_clean_scrub_stamp":"2024-09-13T14:31:03.932624+0000","objects_scrubbed":0,"log_size":5210,"log_dups_size":0,"ondisk_log_size":5210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:26:38.807123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":195231744,"num_objects":2979,"num_object_clones":0,"num_object_copies":8937,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2979,"num_objects_unfound":0,"num_objects_dirty":2979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11101,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,5],"avail_no_missing":["4","5","7"],"object_location_counts":[{"shards":"4,5,7","objects":2979}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":238,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-13T14:34:05.959926+0000","last_change":"2024-09-13T14:24:55.284499+0000","last_active":"2024-09-13T14:34:05.959926+0000","last_peered":"2024-09-13T14:34:05.959926+0000","last_clean":"2024-09-13T14:34:05.959926+0000","last_became_active":"2024-09-13T14:24:55.284096+0000","last_became_peered":"2024-09-13T14:24:55.284096+0000","last_unstale":"2024-09-13T14:34:05.959926+0000","last_undegraded":"2024-09-13T14:34:05.959926+0000","last_fullsized":"2024-09-13T14:34:05.959926+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:02:06.836099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":155,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-13T14:33:57.913974+0000","last_change":"2024-09-13T14:24:55.291371+0000","last_active":"2024-09-13T14:33:57.913974+0000","last_peered":"2024-09-13T14:33:57.913974+0000","last_clean":"2024-09-13T14:33:57.913974+0000","last_became_active":"2024-09-13T14:24:55.291047+0000","last_became_peered":"2024-09-13T14:24:55.291047+0000","last_unstale":"2024-09-13T14:33:57.913974+0000","last_undegraded":"2024-09-13T14:33:57.913974+0000","last_fullsized":"2024-09-13T14:33:57.913974+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:43.912412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"191'11408","reported_seq":11480,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-13T14:34:07.183217+0000","last_change":"2024-09-13T14:31:04.956319+0000","last_active":"2024-09-13T14:34:07.183217+0000","last_peered":"2024-09-13T14:34:07.183217+0000","last_clean":"2024-09-13T14:34:07.183217+0000","last_became_active":"2024-09-13T14:31:04.956014+0000","last_became_peered":"2024-09-13T14:31:04.956014+0000","last_unstale":"2024-09-13T14:34:07.183217+0000","last_undegraded":"2024-09-13T14:34:07.183217+0000","last_fullsized":"2024-09-13T14:34:07.183217+0000","mapping_epoch":161,"log_start":"176'6926","ondisk_log_start":"176'6926","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_clean_scrub_stamp":"2024-09-13T14:31:03.932624+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:51:35.156658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177733632,"num_objects":2712,"num_object_clones":0,"num_object_copies":8136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11408,"num_write_kb":451840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":88,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-13T14:33:57.920645+0000","last_change":"2024-09-13T14:33:57.920222+0000","last_active":"2024-09-13T14:33:57.920645+0000","last_peered":"2024-09-13T14:33:57.920645+0000","last_clean":"2024-09-13T14:33:57.920645+0000","last_became_active":"2024-09-13T14:33:57.919863+0000","last_became_peered":"2024-09-13T14:33:57.919863+0000","last_unstale":"2024-09-13T14:33:57.920645+0000","last_undegraded":"2024-09-13T14:33:57.920645+0000","last_fullsized":"2024-09-13T14:33:57.920645+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:53:26.073480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":188,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-09-13T14:33:58.912993+0000","last_change":"2024-09-13T14:27:12.842604+0000","last_active":"2024-09-13T14:33:58.912993+0000","last_peered":"2024-09-13T14:33:58.912993+0000","last_clean":"2024-09-13T14:33:58.912993+0000","last_became_active":"2024-09-13T14:27:12.841643+0000","last_became_peered":"2024-09-13T14:27:12.841643+0000","last_unstale":"2024-09-13T14:33:58.912993+0000","last_undegraded":"2024-09-13T14:33:58.912993+0000","last_fullsized":"2024-09-13T14:33:58.912993+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:34:34.751790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"191'10985","reported_seq":11060,"reported_epoch":191,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:34:07.114284+0000","last_change":"2024-09-13T14:33:44.995636+0000","last_active":"2024-09-13T14:34:07.114284+0000","last_peered":"2024-09-13T14:34:07.114284+0000","last_clean":"2024-09-13T14:33:43.496064+0000","last_became_active":"2024-09-13T14:33:44.970102+0000","last_became_peered":"2024-09-13T14:33:44.970102+0000","last_unstale":"2024-09-13T14:34:07.114284+0000","last_undegraded":"2024-09-13T14:34:07.114284+0000","last_fullsized":"2024-09-13T14:34:07.114284+0000","mapping_epoch":183,"log_start":"175'6445","ondisk_log_start":"175'6445","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_clean_scrub_stamp":"2024-09-13T14:31:03.932624+0000","objects_scrubbed":0,"log_size":4540,"log_dups_size":0,"ondisk_log_size":4540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174522368,"num_objects":2663,"num_object_clones":0,"num_object_copies":7989,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2663,"num_objects_unfound":0,"num_objects_dirty":2663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10985,"num_write_kb":436736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,5],"avail_no_missing":["3","5","6"],"object_location_counts":[{"shards":"3,5,6","objects":2663}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":174,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-13T14:33:57.913182+0000","last_change":"2024-09-13T14:33:43.832502+0000","last_active":"2024-09-13T14:33:57.913182+0000","last_peered":"2024-09-13T14:33:57.913182+0000","last_clean":"2024-09-13T14:33:57.913182+0000","last_became_active":"2024-09-13T14:33:43.832200+0000","last_became_peered":"2024-09-13T14:33:43.832200+0000","last_unstale":"2024-09-13T14:33:57.913182+0000","last_undegraded":"2024-09-13T14:33:57.913182+0000","last_fullsized":"2024-09-13T14:33:57.913182+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:21:00.100158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":164,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-09-13T14:33:58.913013+0000","last_change":"2024-09-13T14:26:15.825304+0000","last_active":"2024-09-13T14:33:58.913013+0000","last_peered":"2024-09-13T14:33:58.913013+0000","last_clean":"2024-09-13T14:33:58.913013+0000","last_became_active":"2024-09-13T14:26:15.825018+0000","last_became_peered":"2024-09-13T14:26:15.825018+0000","last_unstale":"2024-09-13T14:33:58.913013+0000","last_undegraded":"2024-09-13T14:33:58.913013+0000","last_fullsized":"2024-09-13T14:33:58.913013+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:22:59.408928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"191'11328","reported_seq":11382,"reported_epoch":191,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:34:06.899131+0000","last_change":"2024-09-13T14:33:44.997490+0000","last_active":"2024-09-13T14:34:06.899131+0000","last_peered":"2024-09-13T14:34:06.899131+0000","last_clean":"2024-09-13T14:33:43.503130+0000","last_became_active":"2024-09-13T14:33:44.966494+0000","last_became_peered":"2024-09-13T14:33:44.966494+0000","last_unstale":"2024-09-13T14:34:06.899131+0000","last_undegraded":"2024-09-13T14:34:06.899131+0000","last_fullsized":"2024-09-13T14:34:06.899131+0000","mapping_epoch":183,"log_start":"175'5687","ondisk_log_start":"175'5687","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_clean_scrub_stamp":"2024-09-13T14:31:03.932624+0000","objects_scrubbed":0,"log_size":5641,"log_dups_size":0,"ondisk_log_size":5641,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176226328,"num_objects":2690,"num_object_clones":0,"num_object_copies":8070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2690,"num_objects_unfound":0,"num_objects_dirty":2690,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11328,"num_write_kb":448513,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,5],"avail_no_missing":["0","5","6"],"object_location_counts":[{"shards":"0,5,6","objects":2690}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":260,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-13T14:34:05.960058+0000","last_change":"2024-09-13T14:33:43.837111+0000","last_active":"2024-09-13T14:34:05.960058+0000","last_peered":"2024-09-13T14:34:05.960058+0000","last_clean":"2024-09-13T14:34:05.960058+0000","last_became_active":"2024-09-13T14:33:43.836673+0000","last_became_peered":"2024-09-13T14:33:43.836673+0000","last_unstale":"2024-09-13T14:34:05.960058+0000","last_undegraded":"2024-09-13T14:34:05.960058+0000","last_fullsized":"2024-09-13T14:34:05.960058+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:36:49.522297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":210,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-13T14:34:05.963905+0000","last_change":"2024-09-13T14:26:15.823198+0000","last_active":"2024-09-13T14:34:05.963905+0000","last_peered":"2024-09-13T14:34:05.963905+0000","last_clean":"2024-09-13T14:34:05.963905+0000","last_became_active":"2024-09-13T14:26:15.822717+0000","last_became_peered":"2024-09-13T14:26:15.822717+0000","last_unstale":"2024-09-13T14:34:05.963905+0000","last_undegraded":"2024-09-13T14:34:05.963905+0000","last_fullsized":"2024-09-13T14:34:05.963905+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:22.471622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"191'11348","reported_seq":11400,"reported_epoch":191,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:34:07.146425+0000","last_change":"2024-09-13T14:33:45.049970+0000","last_active":"2024-09-13T14:34:07.146425+0000","last_peered":"2024-09-13T14:34:07.146425+0000","last_clean":"2024-09-13T14:33:43.501971+0000","last_became_active":"2024-09-13T14:33:45.007619+0000","last_became_peered":"2024-09-13T14:33:45.007619+0000","last_unstale":"2024-09-13T14:34:07.146425+0000","last_undegraded":"2024-09-13T14:34:07.146425+0000","last_fullsized":"2024-09-13T14:34:07.146425+0000","mapping_epoch":183,"log_start":"176'6793","ondisk_log_start":"176'6793","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_clean_scrub_stamp":"2024-09-13T14:31:03.932624+0000","objects_scrubbed":0,"log_size":4555,"log_dups_size":0,"ondisk_log_size":4555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173539328,"num_objects":2648,"num_object_clones":0,"num_object_copies":7944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2648,"num_objects_unfound":0,"num_objects_dirty":2648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11348,"num_write_kb":447872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,5],"avail_no_missing":["3","4","5"],"object_location_counts":[{"shards":"3,4,5","objects":2648}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":425,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-13T14:34:05.960941+0000","last_change":"2024-09-13T14:32:04.102399+0000","last_active":"2024-09-13T14:34:05.960941+0000","last_peered":"2024-09-13T14:34:05.960941+0000","last_clean":"2024-09-13T14:34:05.960941+0000","last_became_active":"2024-09-13T14:32:04.102011+0000","last_became_peered":"2024-09-13T14:32:04.102011+0000","last_unstale":"2024-09-13T14:34:05.960941+0000","last_undegraded":"2024-09-13T14:34:05.960941+0000","last_fullsized":"2024-09-13T14:34:05.960941+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:07:41.927740+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:17.553502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":443,"reported_epoch":191,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:34:05.959535+0000","last_change":"2024-09-13T14:33:44.844136+0000","last_active":"2024-09-13T14:34:05.959535+0000","last_peered":"2024-09-13T14:34:05.959535+0000","last_clean":"2024-09-13T14:27:08.815217+0000","last_became_active":"2024-09-13T14:33:44.842500+0000","last_became_peered":"2024-09-13T14:33:44.842500+0000","last_unstale":"2024-09-13T14:34:05.959535+0000","last_undegraded":"2024-09-13T14:33:44.825636+0000","last_fullsized":"2024-09-13T14:33:44.823989+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6],"avail_no_missing":["3","5","6"],"object_location_counts":[{"shards":"3,5,6","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":196,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-13T14:34:05.958440+0000","last_change":"2024-09-13T14:33:43.840063+0000","last_active":"2024-09-13T14:34:05.958440+0000","last_peered":"2024-09-13T14:34:05.958440+0000","last_clean":"2024-09-13T14:34:05.958440+0000","last_became_active":"2024-09-13T14:33:43.839724+0000","last_became_peered":"2024-09-13T14:33:43.839724+0000","last_unstale":"2024-09-13T14:34:05.958440+0000","last_undegraded":"2024-09-13T14:34:05.958440+0000","last_fullsized":"2024-09-13T14:34:05.958440+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:26:08.978423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":15,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-13T14:33:57.916979+0000","last_change":"2024-09-13T14:33:57.916601+0000","last_active":"2024-09-13T14:33:57.916979+0000","last_peered":"2024-09-13T14:33:57.916979+0000","last_clean":"2024-09-13T14:33:57.916979+0000","last_became_active":"2024-09-13T14:33:57.916312+0000","last_became_peered":"2024-09-13T14:33:57.916312+0000","last_unstale":"2024-09-13T14:33:57.916979+0000","last_undegraded":"2024-09-13T14:33:57.916979+0000","last_fullsized":"2024-09-13T14:33:57.916979+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:28:36.646375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"191'11126","reported_seq":11180,"reported_epoch":191,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:34:07.193664+0000","last_change":"2024-09-13T14:33:45.045665+0000","last_active":"2024-09-13T14:34:07.193664+0000","last_peered":"2024-09-13T14:34:07.193664+0000","last_clean":"2024-09-13T14:33:43.419658+0000","last_became_active":"2024-09-13T14:33:45.007536+0000","last_became_peered":"2024-09-13T14:33:45.007536+0000","last_unstale":"2024-09-13T14:34:07.193664+0000","last_undegraded":"2024-09-13T14:34:07.193664+0000","last_fullsized":"2024-09-13T14:34:07.193664+0000","mapping_epoch":183,"log_start":"176'6609","ondisk_log_start":"176'6609","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_clean_scrub_stamp":"2024-09-13T14:31:03.932624+0000","objects_scrubbed":0,"log_size":4517,"log_dups_size":0,"ondisk_log_size":4517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172097536,"num_objects":2626,"num_object_clones":0,"num_object_copies":7878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2626,"num_objects_unfound":0,"num_objects_dirty":2626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11126,"num_write_kb":440064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,5],"avail_no_missing":["3","5","6"],"object_location_counts":[{"shards":"3,5,6","objects":2626}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":403,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-09-13T14:33:58.952980+0000","last_change":"2024-09-13T14:26:15.835178+0000","last_active":"2024-09-13T14:33:58.952980+0000","last_peered":"2024-09-13T14:33:58.952980+0000","last_clean":"2024-09-13T14:33:58.952980+0000","last_became_active":"2024-09-13T14:26:15.820283+0000","last_became_peered":"2024-09-13T14:26:15.820283+0000","last_unstale":"2024-09-13T14:33:58.952980+0000","last_undegraded":"2024-09-13T14:33:58.952980+0000","last_fullsized":"2024-09-13T14:33:58.952980+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:30.340038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":232,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-13T14:34:05.960731+0000","last_change":"2024-09-13T14:24:55.286576+0000","last_active":"2024-09-13T14:34:05.960731+0000","last_peered":"2024-09-13T14:34:05.960731+0000","last_clean":"2024-09-13T14:34:05.960731+0000","last_became_active":"2024-09-13T14:24:55.286284+0000","last_became_peered":"2024-09-13T14:24:55.286284+0000","last_unstale":"2024-09-13T14:34:05.960731+0000","last_undegraded":"2024-09-13T14:34:05.960731+0000","last_fullsized":"2024-09-13T14:34:05.960731+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:35.589736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":272,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-13T14:34:05.963031+0000","last_change":"2024-09-13T14:33:43.834021+0000","last_active":"2024-09-13T14:34:05.963031+0000","last_peered":"2024-09-13T14:34:05.963031+0000","last_clean":"2024-09-13T14:34:05.963031+0000","last_became_active":"2024-09-13T14:33:43.833725+0000","last_became_peered":"2024-09-13T14:33:43.833725+0000","last_unstale":"2024-09-13T14:34:05.963031+0000","last_undegraded":"2024-09-13T14:34:05.963031+0000","last_fullsized":"2024-09-13T14:34:05.963031+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:38:19.136728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"191'11069","reported_seq":11109,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-13T14:34:07.172682+0000","last_change":"2024-09-13T14:31:04.954224+0000","last_active":"2024-09-13T14:34:07.172682+0000","last_peered":"2024-09-13T14:34:07.172682+0000","last_clean":"2024-09-13T14:34:07.172682+0000","last_became_active":"2024-09-13T14:31:04.953723+0000","last_became_peered":"2024-09-13T14:31:04.953723+0000","last_unstale":"2024-09-13T14:34:07.172682+0000","last_undegraded":"2024-09-13T14:34:07.172682+0000","last_fullsized":"2024-09-13T14:34:07.172682+0000","mapping_epoch":161,"log_start":"164'1000","ondisk_log_start":"164'1000","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_clean_scrub_stamp":"2024-09-13T14:31:03.932624+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:41:04.638415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166133760,"num_objects":2535,"num_object_clones":0,"num_object_copies":7605,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11069,"num_write_kb":435328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":612,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-13T14:34:05.959431+0000","last_change":"2024-09-13T14:27:15.863966+0000","last_active":"2024-09-13T14:34:05.959431+0000","last_peered":"2024-09-13T14:34:05.959431+0000","last_clean":"2024-09-13T14:34:05.959431+0000","last_became_active":"2024-09-13T14:27:15.863124+0000","last_became_peered":"2024-09-13T14:27:15.863124+0000","last_unstale":"2024-09-13T14:34:05.959431+0000","last_undegraded":"2024-09-13T14:34:05.959431+0000","last_fullsized":"2024-09-13T14:34:05.959431+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:35:02.715013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":126,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-13T14:34:05.959684+0000","last_change":"2024-09-13T14:33:57.919106+0000","last_active":"2024-09-13T14:34:05.959684+0000","last_peered":"2024-09-13T14:34:05.959684+0000","last_clean":"2024-09-13T14:34:05.959684+0000","last_became_active":"2024-09-13T14:33:57.918730+0000","last_became_peered":"2024-09-13T14:33:57.918730+0000","last_unstale":"2024-09-13T14:34:05.959684+0000","last_undegraded":"2024-09-13T14:34:05.959684+0000","last_fullsized":"2024-09-13T14:34:05.959684+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:51:45.741430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":161,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-13T14:34:05.963513+0000","last_change":"2024-09-13T14:33:43.833058+0000","last_active":"2024-09-13T14:34:05.963513+0000","last_peered":"2024-09-13T14:34:05.963513+0000","last_clean":"2024-09-13T14:34:05.963513+0000","last_became_active":"2024-09-13T14:33:43.832640+0000","last_became_peered":"2024-09-13T14:33:43.832640+0000","last_unstale":"2024-09-13T14:34:05.963513+0000","last_undegraded":"2024-09-13T14:34:05.963513+0000","last_fullsized":"2024-09-13T14:34:05.963513+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:06:40.474738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"191'11149","reported_seq":11221,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-13T14:34:07.165589+0000","last_change":"2024-09-13T14:31:04.953392+0000","last_active":"2024-09-13T14:34:07.165589+0000","last_peered":"2024-09-13T14:34:07.165589+0000","last_clean":"2024-09-13T14:34:07.165589+0000","last_became_active":"2024-09-13T14:31:04.953114+0000","last_became_peered":"2024-09-13T14:31:04.953114+0000","last_unstale":"2024-09-13T14:34:07.165589+0000","last_undegraded":"2024-09-13T14:34:07.165589+0000","last_fullsized":"2024-09-13T14:34:07.165589+0000","mapping_epoch":161,"log_start":"176'6654","ondisk_log_start":"176'6654","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_clean_scrub_stamp":"2024-09-13T14:31:03.932624+0000","objects_scrubbed":0,"log_size":4495,"log_dups_size":0,"ondisk_log_size":4495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:41:13.856511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171900928,"num_objects":2623,"num_object_clones":0,"num_object_copies":7869,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11149,"num_write_kb":440704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":850,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-13T14:34:05.960373+0000","last_change":"2024-09-13T14:33:57.920053+0000","last_active":"2024-09-13T14:34:05.960373+0000","last_peered":"2024-09-13T14:34:05.960373+0000","last_clean":"2024-09-13T14:34:05.960373+0000","last_became_active":"2024-09-13T14:33:57.919717+0000","last_became_peered":"2024-09-13T14:33:57.919717+0000","last_unstale":"2024-09-13T14:34:05.960373+0000","last_undegraded":"2024-09-13T14:34:05.960373+0000","last_fullsized":"2024-09-13T14:34:05.960373+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:25:46.372325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":156,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-13T14:33:57.913685+0000","last_change":"2024-09-13T14:23:18.042891+0000","last_active":"2024-09-13T14:33:57.913685+0000","last_peered":"2024-09-13T14:33:57.913685+0000","last_clean":"2024-09-13T14:33:57.913685+0000","last_became_active":"2024-09-13T14:23:18.042562+0000","last_became_peered":"2024-09-13T14:23:18.042562+0000","last_unstale":"2024-09-13T14:33:57.913685+0000","last_undegraded":"2024-09-13T14:33:57.913685+0000","last_fullsized":"2024-09-13T14:33:57.913685+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:30:42.996661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":239,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-13T14:34:05.958505+0000","last_change":"2024-09-13T14:33:43.836569+0000","last_active":"2024-09-13T14:34:05.958505+0000","last_peered":"2024-09-13T14:34:05.958505+0000","last_clean":"2024-09-13T14:34:05.958505+0000","last_became_active":"2024-09-13T14:33:43.835446+0000","last_became_peered":"2024-09-13T14:33:43.835446+0000","last_unstale":"2024-09-13T14:34:05.958505+0000","last_undegraded":"2024-09-13T14:34:05.958505+0000","last_fullsized":"2024-09-13T14:34:05.958505+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:01:30.947962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"191'11001","reported_seq":11077,"reported_epoch":191,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:34:06.854105+0000","last_change":"2024-09-13T14:33:45.017502+0000","last_active":"2024-09-13T14:34:06.854105+0000","last_peered":"2024-09-13T14:34:06.854105+0000","last_clean":"2024-09-13T14:33:43.510448+0000","last_became_active":"2024-09-13T14:33:44.986411+0000","last_became_peered":"2024-09-13T14:33:44.986411+0000","last_unstale":"2024-09-13T14:34:06.854105+0000","last_undegraded":"2024-09-13T14:34:06.854105+0000","last_fullsized":"2024-09-13T14:34:06.854105+0000","mapping_epoch":183,"log_start":"175'5348","ondisk_log_start":"175'5348","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_clean_scrub_stamp":"2024-09-13T14:31:03.932624+0000","objects_scrubbed":0,"log_size":5653,"log_dups_size":0,"ondisk_log_size":5653,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174653440,"num_objects":2665,"num_object_clones":0,"num_object_copies":7995,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2665,"num_objects_unfound":0,"num_objects_dirty":2665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11001,"num_write_kb":437312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":2665}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":133,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-13T14:34:05.958594+0000","last_change":"2024-09-13T14:33:43.836763+0000","last_active":"2024-09-13T14:34:05.958594+0000","last_peered":"2024-09-13T14:34:05.958594+0000","last_clean":"2024-09-13T14:34:05.958594+0000","last_became_active":"2024-09-13T14:33:43.836126+0000","last_became_peered":"2024-09-13T14:33:43.836126+0000","last_unstale":"2024-09-13T14:34:05.958594+0000","last_undegraded":"2024-09-13T14:34:05.958594+0000","last_fullsized":"2024-09-13T14:34:05.958594+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:21:32.455751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":961,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-13T14:34:05.962171+0000","last_change":"2024-09-13T14:30:58.928302+0000","last_active":"2024-09-13T14:34:05.962171+0000","last_peered":"2024-09-13T14:34:05.962171+0000","last_clean":"2024-09-13T14:34:05.962171+0000","last_became_active":"2024-09-13T14:30:58.927718+0000","last_became_peered":"2024-09-13T14:30:58.927718+0000","last_unstale":"2024-09-13T14:34:05.962171+0000","last_undegraded":"2024-09-13T14:34:05.962171+0000","last_fullsized":"2024-09-13T14:34:05.962171+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:37.323036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":36,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-13T14:33:57.913098+0000","last_change":"2024-09-13T14:33:43.834853+0000","last_active":"2024-09-13T14:33:57.913098+0000","last_peered":"2024-09-13T14:33:57.913098+0000","last_clean":"2024-09-13T14:33:57.913098+0000","last_became_active":"2024-09-13T14:33:43.832792+0000","last_became_peered":"2024-09-13T14:33:43.832792+0000","last_unstale":"2024-09-13T14:33:57.913098+0000","last_undegraded":"2024-09-13T14:33:57.913098+0000","last_fullsized":"2024-09-13T14:33:57.913098+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:19:08.253840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"191'11003","reported_seq":16081,"reported_epoch":191,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-13T14:34:06.887046+0000","last_change":"2024-09-13T14:34:01.891290+0000","last_active":"2024-09-13T14:34:06.887046+0000","last_peered":"2024-09-13T14:34:06.887046+0000","last_clean":"2024-09-13T14:32:00.056530+0000","last_became_active":"2024-09-13T14:34:00.728204+0000","last_became_peered":"2024-09-13T14:34:00.728204+0000","last_unstale":"2024-09-13T14:34:06.887046+0000","last_undegraded":"2024-09-13T14:34:00.703418+0000","last_fullsized":"2024-09-13T14:34:06.887046+0000","mapping_epoch":189,"log_start":"175'5861","ondisk_log_start":"175'5861","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_clean_scrub_stamp":"2024-09-13T14:31:03.932624+0000","objects_scrubbed":0,"log_size":5142,"log_dups_size":0,"ondisk_log_size":5142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166658048,"num_objects":2543,"num_object_clones":0,"num_object_copies":7629,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2424,"num_objects_misplaced":119,"num_objects_unfound":0,"num_objects_dirty":2543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11003,"num_write_kb":433472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2507,"num_bytes_recovered":164102144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,5],"avail_no_missing":["0","6","7"],"object_location_counts":[{"shards":"0,5,6","objects":1843},{"shards":"0,6,7","objects":700}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":67,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-13T14:34:05.962082+0000","last_change":"2024-09-13T14:33:43.833965+0000","last_active":"2024-09-13T14:34:05.962082+0000","last_peered":"2024-09-13T14:34:05.962082+0000","last_clean":"2024-09-13T14:34:05.962082+0000","last_became_active":"2024-09-13T14:33:43.833585+0000","last_became_peered":"2024-09-13T14:33:43.833585+0000","last_unstale":"2024-09-13T14:34:05.962082+0000","last_undegraded":"2024-09-13T14:34:05.962082+0000","last_fullsized":"2024-09-13T14:34:05.962082+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:25:05.469973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":680,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-09-13T14:33:58.953892+0000","last_change":"2024-09-13T14:23:18.038847+0000","last_active":"2024-09-13T14:33:58.953892+0000","last_peered":"2024-09-13T14:33:58.953892+0000","last_clean":"2024-09-13T14:33:58.953892+0000","last_became_active":"2024-09-13T14:23:18.038188+0000","last_became_peered":"2024-09-13T14:23:18.038188+0000","last_unstale":"2024-09-13T14:33:58.953892+0000","last_undegraded":"2024-09-13T14:33:58.953892+0000","last_fullsized":"2024-09-13T14:33:58.953892+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:30:37.213496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":229,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-13T14:34:05.962170+0000","last_change":"2024-09-13T14:33:43.835321+0000","last_active":"2024-09-13T14:34:05.962170+0000","last_peered":"2024-09-13T14:34:05.962170+0000","last_clean":"2024-09-13T14:34:05.962170+0000","last_became_active":"2024-09-13T14:33:43.835013+0000","last_became_peered":"2024-09-13T14:33:43.835013+0000","last_unstale":"2024-09-13T14:34:05.962170+0000","last_undegraded":"2024-09-13T14:34:05.962170+0000","last_fullsized":"2024-09-13T14:34:05.962170+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:05.351205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"190'11072","reported_seq":11111,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-09-13T14:34:02.558664+0000","last_change":"2024-09-13T14:31:04.957425+0000","last_active":"2024-09-13T14:34:02.558664+0000","last_peered":"2024-09-13T14:34:02.558664+0000","last_clean":"2024-09-13T14:34:02.558664+0000","last_became_active":"2024-09-13T14:31:04.957101+0000","last_became_peered":"2024-09-13T14:31:04.957101+0000","last_unstale":"2024-09-13T14:34:02.558664+0000","last_undegraded":"2024-09-13T14:34:02.558664+0000","last_fullsized":"2024-09-13T14:34:02.558664+0000","mapping_epoch":161,"log_start":"175'5889","ondisk_log_start":"175'5889","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_clean_scrub_stamp":"2024-09-13T14:31:03.932624+0000","objects_scrubbed":0,"log_size":5183,"log_dups_size":0,"ondisk_log_size":5183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:14:30.110858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202768384,"num_objects":3094,"num_object_clones":0,"num_object_copies":9282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11072,"num_write_kb":453312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":260,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-13T14:34:05.963983+0000","last_change":"2024-09-13T14:33:43.852043+0000","last_active":"2024-09-13T14:34:05.963983+0000","last_peered":"2024-09-13T14:34:05.963983+0000","last_clean":"2024-09-13T14:34:05.963983+0000","last_became_active":"2024-09-13T14:33:43.838068+0000","last_became_peered":"2024-09-13T14:33:43.838068+0000","last_unstale":"2024-09-13T14:34:05.963983+0000","last_undegraded":"2024-09-13T14:34:05.963983+0000","last_fullsized":"2024-09-13T14:34:05.963983+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:07:49.882523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":471,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-13T14:33:57.920705+0000","last_change":"2024-09-13T14:33:57.920349+0000","last_active":"2024-09-13T14:33:57.920705+0000","last_peered":"2024-09-13T14:33:57.920705+0000","last_clean":"2024-09-13T14:33:57.920705+0000","last_became_active":"2024-09-13T14:33:57.919902+0000","last_became_peered":"2024-09-13T14:33:57.919902+0000","last_unstale":"2024-09-13T14:33:57.920705+0000","last_undegraded":"2024-09-13T14:33:57.920705+0000","last_fullsized":"2024-09-13T14:33:57.920705+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:15:17.273918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":248,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-13T14:34:05.959739+0000","last_change":"2024-09-13T14:33:43.836247+0000","last_active":"2024-09-13T14:34:05.959739+0000","last_peered":"2024-09-13T14:34:05.959739+0000","last_clean":"2024-09-13T14:34:05.959739+0000","last_became_active":"2024-09-13T14:33:43.835840+0000","last_became_peered":"2024-09-13T14:33:43.835840+0000","last_unstale":"2024-09-13T14:34:05.959739+0000","last_undegraded":"2024-09-13T14:34:05.959739+0000","last_fullsized":"2024-09-13T14:34:05.959739+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:37:02.209326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"191'11010","reported_seq":11050,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-13T14:34:07.146102+0000","last_change":"2024-09-13T14:31:04.956103+0000","last_active":"2024-09-13T14:34:07.146102+0000","last_peered":"2024-09-13T14:34:07.146102+0000","last_clean":"2024-09-13T14:34:07.146102+0000","last_became_active":"2024-09-13T14:31:04.955773+0000","last_became_peered":"2024-09-13T14:31:04.955773+0000","last_unstale":"2024-09-13T14:34:07.146102+0000","last_undegraded":"2024-09-13T14:34:07.146102+0000","last_fullsized":"2024-09-13T14:34:07.146102+0000","mapping_epoch":161,"log_start":"164'1000","ondisk_log_start":"164'1000","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:31:03.932624+0000","last_clean_scrub_stamp":"2024-09-13T14:31:03.932624+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:36:08.391308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173539328,"num_objects":2648,"num_object_clones":0,"num_object_copies":7944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11010,"num_write_kb":437056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":269,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-13T14:34:05.962608+0000","last_change":"2024-09-13T14:33:43.844935+0000","last_active":"2024-09-13T14:34:05.962608+0000","last_peered":"2024-09-13T14:34:05.962608+0000","last_clean":"2024-09-13T14:34:05.962608+0000","last_became_active":"2024-09-13T14:33:43.834372+0000","last_became_peered":"2024-09-13T14:33:43.834372+0000","last_unstale":"2024-09-13T14:34:05.962608+0000","last_undegraded":"2024-09-13T14:34:05.962608+0000","last_fullsized":"2024-09-13T14:34:05.962608+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:23:10.999813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":546,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-13T14:34:05.963673+0000","last_change":"2024-09-13T14:33:57.920925+0000","last_active":"2024-09-13T14:34:05.963673+0000","last_peered":"2024-09-13T14:34:05.963673+0000","last_clean":"2024-09-13T14:34:05.963673+0000","last_became_active":"2024-09-13T14:33:57.918989+0000","last_became_peered":"2024-09-13T14:33:57.918989+0000","last_unstale":"2024-09-13T14:34:05.963673+0000","last_undegraded":"2024-09-13T14:34:05.963673+0000","last_fullsized":"2024-09-13T14:34:05.963673+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:21:22.442058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":149,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-13T14:33:57.913045+0000","last_change":"2024-09-13T14:33:43.833979+0000","last_active":"2024-09-13T14:33:57.913045+0000","last_peered":"2024-09-13T14:33:57.913045+0000","last_clean":"2024-09-13T14:33:57.913045+0000","last_became_active":"2024-09-13T14:33:43.833688+0000","last_became_peered":"2024-09-13T14:33:43.833688+0000","last_unstale":"2024-09-13T14:33:57.913045+0000","last_undegraded":"2024-09-13T14:33:57.913045+0000","last_fullsized":"2024-09-13T14:33:57.913045+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:19:26.907941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":172,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-13T14:33:57.913443+0000","last_change":"2024-09-13T14:33:43.833545+0000","last_active":"2024-09-13T14:33:57.913443+0000","last_peered":"2024-09-13T14:33:57.913443+0000","last_clean":"2024-09-13T14:33:57.913443+0000","last_became_active":"2024-09-13T14:33:43.833220+0000","last_became_peered":"2024-09-13T14:33:43.833220+0000","last_unstale":"2024-09-13T14:33:57.913443+0000","last_undegraded":"2024-09-13T14:33:57.913443+0000","last_fullsized":"2024-09-13T14:33:57.913443+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:25:45.882019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":163,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-13T14:33:57.915571+0000","last_change":"2024-09-13T14:11:29.237013+0000","last_active":"2024-09-13T14:33:57.915571+0000","last_peered":"2024-09-13T14:33:57.915571+0000","last_clean":"2024-09-13T14:33:57.915571+0000","last_became_active":"2024-09-13T14:11:29.236643+0000","last_became_peered":"2024-09-13T14:11:29.236643+0000","last_unstale":"2024-09-13T14:33:57.915571+0000","last_undegraded":"2024-09-13T14:33:57.915571+0000","last_fullsized":"2024-09-13T14:33:57.915571+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:33:58.438693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":148,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-13T14:34:05.960453+0000","last_change":"2024-09-13T14:33:57.916903+0000","last_active":"2024-09-13T14:34:05.960453+0000","last_peered":"2024-09-13T14:34:05.960453+0000","last_clean":"2024-09-13T14:34:05.960453+0000","last_became_active":"2024-09-13T14:33:57.916557+0000","last_became_peered":"2024-09-13T14:33:57.916557+0000","last_unstale":"2024-09-13T14:34:05.960453+0000","last_undegraded":"2024-09-13T14:34:05.960453+0000","last_fullsized":"2024-09-13T14:34:05.960453+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:09:23.980869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":71,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-13T14:34:05.958491+0000","last_change":"2024-09-13T14:32:01.090445+0000","last_active":"2024-09-13T14:34:05.958491+0000","last_peered":"2024-09-13T14:34:05.958491+0000","last_clean":"2024-09-13T14:34:05.958491+0000","last_became_active":"2024-09-13T14:32:01.086157+0000","last_became_peered":"2024-09-13T14:32:01.086157+0000","last_unstale":"2024-09-13T14:34:05.958491+0000","last_undegraded":"2024-09-13T14:34:05.958491+0000","last_fullsized":"2024-09-13T14:34:05.958491+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:23:44.484442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":174,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-13T14:34:05.959373+0000","last_change":"2024-09-13T14:33:43.837424+0000","last_active":"2024-09-13T14:34:05.959373+0000","last_peered":"2024-09-13T14:34:05.959373+0000","last_clean":"2024-09-13T14:34:05.959373+0000","last_became_active":"2024-09-13T14:33:43.837064+0000","last_became_peered":"2024-09-13T14:33:43.837064+0000","last_unstale":"2024-09-13T14:34:05.959373+0000","last_undegraded":"2024-09-13T14:34:05.959373+0000","last_fullsized":"2024-09-13T14:34:05.959373+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:09.249443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":204,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-13T14:34:05.959098+0000","last_change":"2024-09-13T14:33:43.836075+0000","last_active":"2024-09-13T14:34:05.959098+0000","last_peered":"2024-09-13T14:34:05.959098+0000","last_clean":"2024-09-13T14:34:05.959098+0000","last_became_active":"2024-09-13T14:33:43.835705+0000","last_became_peered":"2024-09-13T14:33:43.835705+0000","last_unstale":"2024-09-13T14:34:05.959098+0000","last_undegraded":"2024-09-13T14:34:05.959098+0000","last_fullsized":"2024-09-13T14:34:05.959098+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:04.830722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":238,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-13T14:34:05.963325+0000","last_change":"2024-09-13T14:33:43.834053+0000","last_active":"2024-09-13T14:34:05.963325+0000","last_peered":"2024-09-13T14:34:05.963325+0000","last_clean":"2024-09-13T14:34:05.963325+0000","last_became_active":"2024-09-13T14:33:43.833735+0000","last_became_peered":"2024-09-13T14:33:43.833735+0000","last_unstale":"2024-09-13T14:34:05.963325+0000","last_undegraded":"2024-09-13T14:34:05.963325+0000","last_fullsized":"2024-09-13T14:34:05.963325+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:17:32.116269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":245,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-13T14:34:05.958796+0000","last_change":"2024-09-13T14:26:15.824540+0000","last_active":"2024-09-13T14:34:05.958796+0000","last_peered":"2024-09-13T14:34:05.958796+0000","last_clean":"2024-09-13T14:34:05.958796+0000","last_became_active":"2024-09-13T14:26:15.824118+0000","last_became_peered":"2024-09-13T14:26:15.824118+0000","last_unstale":"2024-09-13T14:34:05.958796+0000","last_undegraded":"2024-09-13T14:34:05.958796+0000","last_fullsized":"2024-09-13T14:34:05.958796+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:56.677401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":175,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-13T14:34:05.963155+0000","last_change":"2024-09-13T14:33:57.919095+0000","last_active":"2024-09-13T14:34:05.963155+0000","last_peered":"2024-09-13T14:34:05.963155+0000","last_clean":"2024-09-13T14:34:05.963155+0000","last_became_active":"2024-09-13T14:33:57.918725+0000","last_became_peered":"2024-09-13T14:33:57.918725+0000","last_unstale":"2024-09-13T14:34:05.963155+0000","last_undegraded":"2024-09-13T14:34:05.963155+0000","last_fullsized":"2024-09-13T14:34:05.963155+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:54:25.049712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":267,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-13T14:34:05.959961+0000","last_change":"2024-09-13T14:27:12.830642+0000","last_active":"2024-09-13T14:34:05.959961+0000","last_peered":"2024-09-13T14:34:05.959961+0000","last_clean":"2024-09-13T14:34:05.959961+0000","last_became_active":"2024-09-13T14:27:12.830026+0000","last_became_peered":"2024-09-13T14:27:12.830026+0000","last_unstale":"2024-09-13T14:34:05.959961+0000","last_undegraded":"2024-09-13T14:34:05.959961+0000","last_fullsized":"2024-09-13T14:34:05.959961+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:12:31.783552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":247,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-13T14:34:05.962922+0000","last_change":"2024-09-13T14:33:43.833127+0000","last_active":"2024-09-13T14:34:05.962922+0000","last_peered":"2024-09-13T14:34:05.962922+0000","last_clean":"2024-09-13T14:34:05.962922+0000","last_became_active":"2024-09-13T14:33:43.832601+0000","last_became_peered":"2024-09-13T14:33:43.832601+0000","last_unstale":"2024-09-13T14:34:05.962922+0000","last_undegraded":"2024-09-13T14:34:05.962922+0000","last_fullsized":"2024-09-13T14:34:05.962922+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:49:37.211195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":219,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-13T14:34:05.959333+0000","last_change":"2024-09-13T14:26:15.823524+0000","last_active":"2024-09-13T14:34:05.959333+0000","last_peered":"2024-09-13T14:34:05.959333+0000","last_clean":"2024-09-13T14:34:05.959333+0000","last_became_active":"2024-09-13T14:26:15.823162+0000","last_became_peered":"2024-09-13T14:26:15.823162+0000","last_unstale":"2024-09-13T14:34:05.959333+0000","last_undegraded":"2024-09-13T14:34:05.959333+0000","last_fullsized":"2024-09-13T14:34:05.959333+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:17:13.664812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":259,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-13T14:34:05.958286+0000","last_change":"2024-09-13T14:26:15.823703+0000","last_active":"2024-09-13T14:34:05.958286+0000","last_peered":"2024-09-13T14:34:05.958286+0000","last_clean":"2024-09-13T14:34:05.958286+0000","last_became_active":"2024-09-13T14:26:15.823249+0000","last_became_peered":"2024-09-13T14:26:15.823249+0000","last_unstale":"2024-09-13T14:34:05.958286+0000","last_undegraded":"2024-09-13T14:34:05.958286+0000","last_fullsized":"2024-09-13T14:34:05.958286+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:42:40.209150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":207,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-13T14:33:57.913307+0000","last_change":"2024-09-13T14:33:43.831885+0000","last_active":"2024-09-13T14:33:57.913307+0000","last_peered":"2024-09-13T14:33:57.913307+0000","last_clean":"2024-09-13T14:33:57.913307+0000","last_became_active":"2024-09-13T14:33:43.831544+0000","last_became_peered":"2024-09-13T14:33:43.831544+0000","last_unstale":"2024-09-13T14:33:57.913307+0000","last_undegraded":"2024-09-13T14:33:57.913307+0000","last_fullsized":"2024-09-13T14:33:57.913307+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:29:49.654536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":242,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-13T14:34:05.958791+0000","last_change":"2024-09-13T14:27:12.838453+0000","last_active":"2024-09-13T14:34:05.958791+0000","last_peered":"2024-09-13T14:34:05.958791+0000","last_clean":"2024-09-13T14:34:05.958791+0000","last_became_active":"2024-09-13T14:27:12.838124+0000","last_became_peered":"2024-09-13T14:27:12.838124+0000","last_unstale":"2024-09-13T14:34:05.958791+0000","last_undegraded":"2024-09-13T14:34:05.958791+0000","last_fullsized":"2024-09-13T14:34:05.958791+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:24.259191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":224,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-13T14:34:05.961944+0000","last_change":"2024-09-13T14:23:18.035101+0000","last_active":"2024-09-13T14:34:05.961944+0000","last_peered":"2024-09-13T14:34:05.961944+0000","last_clean":"2024-09-13T14:34:05.961944+0000","last_became_active":"2024-09-13T14:23:18.034724+0000","last_became_peered":"2024-09-13T14:23:18.034724+0000","last_unstale":"2024-09-13T14:34:05.961944+0000","last_undegraded":"2024-09-13T14:34:05.961944+0000","last_fullsized":"2024-09-13T14:34:05.961944+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:39.358144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":187,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-13T14:33:57.913488+0000","last_change":"2024-09-13T14:27:12.834931+0000","last_active":"2024-09-13T14:33:57.913488+0000","last_peered":"2024-09-13T14:33:57.913488+0000","last_clean":"2024-09-13T14:33:57.913488+0000","last_became_active":"2024-09-13T14:27:12.834572+0000","last_became_peered":"2024-09-13T14:27:12.834572+0000","last_unstale":"2024-09-13T14:33:57.913488+0000","last_undegraded":"2024-09-13T14:33:57.913488+0000","last_fullsized":"2024-09-13T14:33:57.913488+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:54:21.155689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":35,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-13T14:33:57.914145+0000","last_change":"2024-09-13T14:33:43.857044+0000","last_active":"2024-09-13T14:33:57.914145+0000","last_peered":"2024-09-13T14:33:57.914145+0000","last_clean":"2024-09-13T14:33:57.914145+0000","last_became_active":"2024-09-13T14:33:43.834729+0000","last_became_peered":"2024-09-13T14:33:43.834729+0000","last_unstale":"2024-09-13T14:33:57.914145+0000","last_undegraded":"2024-09-13T14:33:57.914145+0000","last_fullsized":"2024-09-13T14:33:57.914145+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:23:35.227896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":169,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-13T14:33:57.913067+0000","last_change":"2024-09-13T14:24:55.285384+0000","last_active":"2024-09-13T14:33:57.913067+0000","last_peered":"2024-09-13T14:33:57.913067+0000","last_clean":"2024-09-13T14:33:57.913067+0000","last_became_active":"2024-09-13T14:24:55.285081+0000","last_became_peered":"2024-09-13T14:24:55.285081+0000","last_unstale":"2024-09-13T14:33:57.913067+0000","last_undegraded":"2024-09-13T14:33:57.913067+0000","last_fullsized":"2024-09-13T14:33:57.913067+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:47:07.946120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":35,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-13T14:33:57.914123+0000","last_change":"2024-09-13T14:33:43.855300+0000","last_active":"2024-09-13T14:33:57.914123+0000","last_peered":"2024-09-13T14:33:57.914123+0000","last_clean":"2024-09-13T14:33:57.914123+0000","last_became_active":"2024-09-13T14:33:43.834723+0000","last_became_peered":"2024-09-13T14:33:43.834723+0000","last_unstale":"2024-09-13T14:33:57.914123+0000","last_undegraded":"2024-09-13T14:33:57.914123+0000","last_fullsized":"2024-09-13T14:33:57.914123+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:42.770152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":272,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-13T14:34:05.959840+0000","last_change":"2024-09-13T14:33:43.836412+0000","last_active":"2024-09-13T14:34:05.959840+0000","last_peered":"2024-09-13T14:34:05.959840+0000","last_clean":"2024-09-13T14:34:05.959840+0000","last_became_active":"2024-09-13T14:33:43.835943+0000","last_became_peered":"2024-09-13T14:33:43.835943+0000","last_unstale":"2024-09-13T14:34:05.959840+0000","last_undegraded":"2024-09-13T14:34:05.959840+0000","last_fullsized":"2024-09-13T14:34:05.959840+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:03.688293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":254,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-13T14:34:05.959677+0000","last_change":"2024-09-13T14:27:12.836567+0000","last_active":"2024-09-13T14:34:05.959677+0000","last_peered":"2024-09-13T14:34:05.959677+0000","last_clean":"2024-09-13T14:34:05.959677+0000","last_became_active":"2024-09-13T14:27:12.836271+0000","last_became_peered":"2024-09-13T14:27:12.836271+0000","last_unstale":"2024-09-13T14:34:05.959677+0000","last_undegraded":"2024-09-13T14:34:05.959677+0000","last_fullsized":"2024-09-13T14:34:05.959677+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:25:32.910438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":163,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-13T14:33:57.914263+0000","last_change":"2024-09-13T14:26:15.824075+0000","last_active":"2024-09-13T14:33:57.914263+0000","last_peered":"2024-09-13T14:33:57.914263+0000","last_clean":"2024-09-13T14:33:57.914263+0000","last_became_active":"2024-09-13T14:26:15.823762+0000","last_became_peered":"2024-09-13T14:26:15.823762+0000","last_unstale":"2024-09-13T14:33:57.914263+0000","last_undegraded":"2024-09-13T14:33:57.914263+0000","last_fullsized":"2024-09-13T14:33:57.914263+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:26:51.088388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":262,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-13T14:34:05.963234+0000","last_change":"2024-09-13T14:33:43.848824+0000","last_active":"2024-09-13T14:34:05.963234+0000","last_peered":"2024-09-13T14:34:05.963234+0000","last_clean":"2024-09-13T14:34:05.963234+0000","last_became_active":"2024-09-13T14:33:43.834398+0000","last_became_peered":"2024-09-13T14:33:43.834398+0000","last_unstale":"2024-09-13T14:34:05.963234+0000","last_undegraded":"2024-09-13T14:34:05.963234+0000","last_fullsized":"2024-09-13T14:34:05.963234+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:33:23.636845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":2821586968,"num_objects":43055,"num_object_clones":0,"num_object_copies":129165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4578,"num_objects_misplaced":19078,"num_objects_unfound":0,"num_objects_dirty":43055,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":178189,"num_write_kb":7079745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10311,"num_bytes_recovered":675545088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9338826752,"data_stored":9338814536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89295,"ondisk_log_size":89295,"up":48,"acting":47,"num_store_stats":6},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1402,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":17952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18272,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16793600,"data_stored":16777292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":23,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234283,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2010092,"kb_used_data":1722148,"kb_used_omap":1,"kb_used_meta":287870,"kb_avail":91735060,"statfs":{"total":95995035648,"available":93936701440,"internally_reserved":0,"allocated":1763479552,"data_stored":1762522389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":294779339},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495919,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2232940,"kb_used_data":1935204,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":91512212,"statfs":{"total":95995035648,"available":93708505088,"internally_reserved":0,"allocated":1981648896,"data_stored":1980684465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":304871879},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":0,"seq":163208757556,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019191,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1501928,"kb_used_data":1137760,"kb_used_omap":1,"kb_used_meta":364158,"kb_avail":92243224,"statfs":{"total":95995035648,"available":94457061376,"internally_reserved":0,"allocated":1165066240,"data_stored":1164105904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":372898246},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435375,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2156460,"kb_used_data":1885988,"kb_used_omap":5,"kb_used_meta":270458,"kb_avail":91588692,"statfs":{"total":95995035648,"available":93786820608,"internally_reserved":0,"allocated":1931251712,"data_stored":1930290233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":276949118},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":446676598903,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":425201762429,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726925956,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1626872,"kb_used_data":1198124,"kb_used_omap":7,"kb_used_meta":428728,"kb_avail":92118280,"statfs":{"total":95995035648,"available":94329118720,"internally_reserved":0,"allocated":1226878976,"data_stored":1225910385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":439017768},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1211961344,"data_stored":1211957272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1913716736,"data_stored":1913716736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1152122880,"data_stored":1152122880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1359941632,"data_stored":1359937560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1941442560,"data_stored":1941438488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1759641600,"data_stored":1759641600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:34:09.867 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_1 pgp_num 16 2024-09-13T14:34:09.913 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:09.944 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:09.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:09.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:09.965 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:34:10.046 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:10.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:10.088 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:10.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:10.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:09 smithi102 ceph-mon[110183]: from='client.35241 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:34:10.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:09 smithi102 ceph-mon[115269]: from='client.35241 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:34:10.190 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:10.193 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:09 smithi195 ceph-mon[104262]: from='client.35241 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:34:10.215 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:10.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:10.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:10.230 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T14:34:10.317 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:10.344 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:10.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:10.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:10.445 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:34:10.501 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:10.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:10.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:34:10.602 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:10.676 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:10.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:10.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:10.687 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:34:10.782 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:10.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:10.814 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:10.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:10.915 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:10.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:10.952 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:10.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:11.053 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:11.054 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:10 smithi102 ceph-mon[110183]: pgmap v606: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 2.6 GiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 818 B/s rd, 0 B/s wr, 490 op/s; 4587/127449 objects degraded (3.599%); 18771/127449 objects misplaced (14.728%); 7.8 MiB/s, 129 objects/s recovering 2024-09-13T14:34:11.054 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:34:11.055 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:10 smithi102 ceph-mon[115269]: pgmap v606: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 2.6 GiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 818 B/s rd, 0 B/s wr, 490 op/s; 4587/127449 objects degraded (3.599%); 18771/127449 objects misplaced (14.728%); 7.8 MiB/s, 129 objects/s recovering 2024-09-13T14:34:11.055 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:34:11.101 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:11.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:11.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:11.166 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:10 smithi195 ceph-mon[104262]: pgmap v606: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 2.6 GiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 818 B/s rd, 0 B/s wr, 490 op/s; 4587/127449 objects degraded (3.599%); 18771/127449 objects misplaced (14.728%); 7.8 MiB/s, 129 objects/s recovering 2024-09-13T14:34:11.166 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:34:11.203 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:11.235 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:11.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:11.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:11.337 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:11.361 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:11.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:11.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:11.463 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:11.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:11.491 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:11.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:11.592 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:11.622 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:11.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:11.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:11.721 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:34:11.723 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:11.758 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:11.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:11.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:11.861 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:11.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:11.904 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:11.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:12.007 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:12.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:12.037 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:12.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:12.138 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:34:12.175 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:12.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:12.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:34:12.277 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:12.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:12.319 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:12.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:12.421 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:12.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:12.472 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:12.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:12.573 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:12.603 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:12.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:12.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:12.705 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:12.747 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:12.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:12.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:12.849 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:12.890 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:12.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:12.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:12.991 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:13.022 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:13.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:13.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:13.124 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:34:13.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:13.157 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:13.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:34:13.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:12 smithi195 ceph-mon[104262]: pgmap v607: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 2.6 GiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 818 B/s rd, 0 B/s wr, 490 op/s; 4587/127449 objects degraded (3.599%); 18771/127449 objects misplaced (14.728%); 7.8 MiB/s, 129 objects/s recovering 2024-09-13T14:34:13.259 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:13.300 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:13.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:13.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:13.402 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:13.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:12 smithi102 ceph-mon[110183]: pgmap v607: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 2.6 GiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 818 B/s rd, 0 B/s wr, 490 op/s; 4587/127449 objects degraded (3.599%); 18771/127449 objects misplaced (14.728%); 7.8 MiB/s, 129 objects/s recovering 2024-09-13T14:34:13.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:12 smithi102 ceph-mon[115269]: pgmap v607: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 2.6 GiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 818 B/s rd, 0 B/s wr, 490 op/s; 4587/127449 objects degraded (3.599%); 18771/127449 objects misplaced (14.728%); 7.8 MiB/s, 129 objects/s recovering 2024-09-13T14:34:13.431 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:13.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:13.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:13.533 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:13.574 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:13.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:13.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:13.676 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:13.715 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:13.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:13.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:13.817 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:13.843 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:13.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:13.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:13.945 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:13.977 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:13.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:13.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:14.079 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:14.108 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:14.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:14.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:14.210 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:14.239 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:14.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:14.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:14.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:13 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 4587/127449 objects degraded (3.599%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:34:14.268 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:13 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 4587/127449 objects degraded (3.599%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:34:14.268 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:13 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 4587/127449 objects degraded (3.599%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:34:14.341 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:14.376 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:14.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:14.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:14.478 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:14.517 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:14.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:14.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:14.619 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:14.653 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:14.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:14.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:14.755 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:14.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:14.799 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:14.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:14.900 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:14.932 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:14.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:14.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:15.034 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:15.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:15.062 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:15.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:15.080 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:34:15.080 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:34:15.080 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:34:15.080 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:34:15.080 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:34:15.081 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:34:15.081 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:34:15.081 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:34:15.081 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:34:15.081 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:34:15.082 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:34:15.082 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:34:15.082 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:34:15.082 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:34:15.082 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:34:15.082 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:34:15.083 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:14 smithi102 ceph-mon[115269]: pgmap v608: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 2.3 GiB data, 8.4 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 888 op/s; 4075/114240 objects degraded (3.567%); 16797/114240 objects misplaced (14.703%); 4.8 MiB/s, 105 objects/s recovering 2024-09-13T14:34:15.083 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:14 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2921178723' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:34:15.083 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:14 smithi102 ceph-mon[110183]: pgmap v608: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 2.3 GiB data, 8.4 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 888 op/s; 4075/114240 objects degraded (3.567%); 16797/114240 objects misplaced (14.703%); 4.8 MiB/s, 105 objects/s recovering 2024-09-13T14:34:15.083 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:14 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2921178723' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:34:15.163 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:15.190 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:15.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:15.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:15.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:14 smithi195 ceph-mon[104262]: pgmap v608: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 2.3 GiB data, 8.4 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 888 op/s; 4075/114240 objects degraded (3.567%); 16797/114240 objects misplaced (14.703%); 4.8 MiB/s, 105 objects/s recovering 2024-09-13T14:34:15.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:14 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2921178723' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:34:15.292 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:15.320 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:15.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:15.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:15.422 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:15.456 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:15.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:15.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:15.558 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:15.587 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:15.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:15.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:15.689 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:15.737 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:15.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:15.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:15.840 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:15.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:15.910 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:15.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:15.992 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is unset 2024-09-13T14:34:16.001 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:34:16.001 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-13T14:34:16.001 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-09-13T14:34:16.001 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-09-13T14:34:16.001 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:34:16.011 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:16.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:16.056 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:16.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:16.157 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:16.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:15 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/156210067' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:34:16.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:15 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:34:16.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:15 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:34:16.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:15 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:34:16.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:15 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/156210067' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:34:16.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:15 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:34:16.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:15 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:34:16.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:15 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:34:16.177 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:16.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:16.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:16.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:15 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/156210067' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:34:16.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:15 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:34:16.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:15 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:34:16.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:15 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:34:16.280 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:16.305 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:16.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:16.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:16.407 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:16.440 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:16.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:16.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:16.542 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:16.570 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:16.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:16.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:16.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:34:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:34:16] "GET /metrics HTTP/1.1" 200 36547 "" "Prometheus/2.51.0" 2024-09-13T14:34:16.671 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:34:16.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:16.698 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:16.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:34:16.738 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T14:34:16.799 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:16.827 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:16.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:16.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:16.929 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:34:16.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:16.971 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:16.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:34:17.072 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:17.125 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:17.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:17.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:17.142 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:34:17.227 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:17.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:16 smithi195 ceph-mon[104262]: pgmap v609: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 2.3 GiB data, 8.4 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 888 op/s; 4075/114240 objects degraded (3.567%); 16797/114240 objects misplaced (14.703%); 4.8 MiB/s, 105 objects/s recovering 2024-09-13T14:34:17.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:16 smithi195 ceph-mon[104262]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:34:17.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:16 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:34:17.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:16 smithi195 ceph-mon[104262]: osdmap e192: 8 total, 8 up, 5 in 2024-09-13T14:34:17.254 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:17.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:17.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:17.356 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:34:17.404 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:17.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:17.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:34:17.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:16 smithi102 ceph-mon[110183]: pgmap v609: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 2.3 GiB data, 8.4 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 888 op/s; 4075/114240 objects degraded (3.567%); 16797/114240 objects misplaced (14.703%); 4.8 MiB/s, 105 objects/s recovering 2024-09-13T14:34:17.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:16 smithi102 ceph-mon[110183]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:34:17.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:16 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:34:17.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:16 smithi102 ceph-mon[110183]: osdmap e192: 8 total, 8 up, 5 in 2024-09-13T14:34:17.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:16 smithi102 ceph-mon[115269]: pgmap v609: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 2.3 GiB data, 8.4 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 888 op/s; 4075/114240 objects degraded (3.567%); 16797/114240 objects misplaced (14.703%); 4.8 MiB/s, 105 objects/s recovering 2024-09-13T14:34:17.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:16 smithi102 ceph-mon[115269]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:34:17.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:16 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:34:17.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:16 smithi102 ceph-mon[115269]: osdmap e192: 8 total, 8 up, 5 in 2024-09-13T14:34:17.507 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:17.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:17.537 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:17.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:17.638 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:17.679 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:17.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:17.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:17.780 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:17.809 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:17.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:17.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:17.843 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:34:17.911 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:17.948 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:17.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:17.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:18.050 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:18.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:18.073 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:18.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:18.174 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:18.204 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:18.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:18.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:18.305 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:34:18.327 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:18.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:18.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:34:18.429 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:18.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:18.471 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:18.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:18.572 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:18.601 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:18.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:18.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:18.703 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:18.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:18.740 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:18.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:18.842 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:18.885 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:18.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:18.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:18.987 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:19.024 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:19.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:19.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:19.126 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:19.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:19.153 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:19.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:19.213 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:34:19.213 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-09-13T14:34:19.213 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-09-13T14:34:19.214 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-13T14:34:19.214 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.214 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.44546.0:88026 5.6 e129/125 5:690e1d48:::benchmark_data_smithi102_155989_object88025:head v 129'5446, mlcod=0'0)", 2024-09-13T14:34:19.214 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:22.495761+0000", 2024-09-13T14:34:19.214 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 416.71314222000001, 2024-09-13T14:34:19.214 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.124147801, 2024-09-13T14:34:19.214 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:19.214 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:34:19.214 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:19.214 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.214 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:19.214 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.495761+0000", 2024-09-13T14:34:19.214 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:19.215 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.215 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.215 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:19.215 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.495761+0000", 2024-09-13T14:34:19.215 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:19.215 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.215 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.215 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:19.215 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.495760+0000", 2024-09-13T14:34:19.215 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:34:19.215 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.215 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.215 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:19.216 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.495782+0000", 2024-09-13T14:34:19.216 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1478999999999999e-05 2024-09-13T14:34:19.216 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.216 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.216 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:19.216 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.495792+0000", 2024-09-13T14:34:19.216 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0538e-05 2024-09-13T14:34:19.216 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.216 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.216 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:19.216 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.495798+0000", 2024-09-13T14:34:19.216 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0689999999999998e-06 2024-09-13T14:34:19.216 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.217 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.217 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:19.217 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.500510+0000", 2024-09-13T14:34:19.217 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0047120640000000002 2024-09-13T14:34:19.217 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.217 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.217 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:34:19.217 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.500532+0000", 2024-09-13T14:34:19.217 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1551000000000001e-05 2024-09-13T14:34:19.217 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.217 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.218 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:34:19.218 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.619795+0000", 2024-09-13T14:34:19.218 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.11926363600000001 2024-09-13T14:34:19.218 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.218 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.218 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:34:19.218 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:22.619909+0000", 2024-09-13T14:34:19.218 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011324600000000001 2024-09-13T14:34:19.218 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:19.218 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:19.218 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:19.218 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.218 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.218 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:91656 5.a 5:508f8413:::benchmark_data_smithi102_155989_object91655:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:34:19.219 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:27.316731+0000", 2024-09-13T14:34:19.219 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 411.89217244100001, 2024-09-13T14:34:19.219 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.12772304500000001, 2024-09-13T14:34:19.219 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:19.219 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:34:19.219 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:34:19.219 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:34:19.219 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:34:19.219 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 91656 2024-09-13T14:34:19.219 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.219 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:19.219 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.219 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:19.220 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.316731+0000", 2024-09-13T14:34:19.220 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:19.220 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.220 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.220 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:19.220 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.316731+0000", 2024-09-13T14:34:19.220 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:19.220 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.220 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.220 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:19.220 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.316730+0000", 2024-09-13T14:34:19.220 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:34:19.220 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.221 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.221 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:19.221 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.316764+0000", 2024-09-13T14:34:19.221 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.3442000000000003e-05 2024-09-13T14:34:19.221 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.221 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.221 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:19.221 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.316773+0000", 2024-09-13T14:34:19.221 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.2089999999999994e-06 2024-09-13T14:34:19.221 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.221 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.221 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:19.221 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.316778+0000", 2024-09-13T14:34:19.222 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0590000000000002e-06 2024-09-13T14:34:19.222 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.222 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.222 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:19.222 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.331802+0000", 2024-09-13T14:34:19.222 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.015023642 2024-09-13T14:34:19.222 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.222 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.222 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:34:19.222 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.331948+0000", 2024-09-13T14:34:19.222 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000146482 2024-09-13T14:34:19.222 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.222 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.222 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,3", 2024-09-13T14:34:19.223 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.332100+0000", 2024-09-13T14:34:19.223 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015182699999999999 2024-09-13T14:34:19.223 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.223 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.223 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:34:19.223 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.343857+0000", 2024-09-13T14:34:19.223 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.01175733 2024-09-13T14:34:19.223 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.223 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.223 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:34:19.224 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.343889+0000", 2024-09-13T14:34:19.224 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.1683000000000001e-05 2024-09-13T14:34:19.224 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.224 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.224 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:34:19.224 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.438087+0000", 2024-09-13T14:34:19.224 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.094197914999999993 2024-09-13T14:34:19.224 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.224 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.224 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:34:19.224 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.438139+0000", 2024-09-13T14:34:19.224 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2346000000000003e-05 2024-09-13T14:34:19.224 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.225 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.225 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:34:19.225 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.444454+0000", 2024-09-13T14:34:19.225 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0063145229999999998 2024-09-13T14:34:19.225 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:19.225 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:19.225 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:19.225 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.225 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.225 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:91657 5.a 5:51ff3ce2:::benchmark_data_smithi102_155989_object91656:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:34:19.225 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:27:27.316895+0000", 2024-09-13T14:34:19.225 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 411.89200833000001, 2024-09-13T14:34:19.225 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.13008430200000001, 2024-09-13T14:34:19.225 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:19.226 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:34:19.226 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:34:19.226 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:34:19.226 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:34:19.226 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 91657 2024-09-13T14:34:19.226 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.226 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:19.226 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.226 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:19.226 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.316895+0000", 2024-09-13T14:34:19.226 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:19.226 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.226 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.226 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:19.227 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.316895+0000", 2024-09-13T14:34:19.227 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:19.227 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.227 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.227 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:19.227 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.316894+0000", 2024-09-13T14:34:19.227 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:34:19.227 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.227 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.227 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:19.227 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.316964+0000", 2024-09-13T14:34:19.227 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.0160000000000006e-05 2024-09-13T14:34:19.227 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.228 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.228 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:19.228 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.316984+0000", 2024-09-13T14:34:19.228 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9992000000000001e-05 2024-09-13T14:34:19.228 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.228 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.228 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:19.228 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.316990+0000", 2024-09-13T14:34:19.228 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7649999999999999e-06 2024-09-13T14:34:19.228 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.228 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.228 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:19.228 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.335386+0000", 2024-09-13T14:34:19.229 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018395575000000001 2024-09-13T14:34:19.229 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.229 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.229 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:34:19.229 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.335511+0000", 2024-09-13T14:34:19.229 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012501499999999999 2024-09-13T14:34:19.229 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.229 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.229 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,3", 2024-09-13T14:34:19.229 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.335664+0000", 2024-09-13T14:34:19.229 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015311200000000001 2024-09-13T14:34:19.229 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.229 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.230 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:34:19.230 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.343907+0000", 2024-09-13T14:34:19.230 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0082433409999999995 2024-09-13T14:34:19.230 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.230 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.230 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:34:19.230 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.343936+0000", 2024-09-13T14:34:19.230 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9009e-05 2024-09-13T14:34:19.230 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.230 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.230 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:34:19.230 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.444517+0000", 2024-09-13T14:34:19.230 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.100580424 2024-09-13T14:34:19.230 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.230 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.231 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:34:19.231 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.444619+0000", 2024-09-13T14:34:19.231 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010208399999999999 2024-09-13T14:34:19.231 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.231 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.231 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:34:19.231 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:27:27.446979+0000", 2024-09-13T14:34:19.231 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0023604229999999999 2024-09-13T14:34:19.231 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:19.231 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:19.231 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:19.231 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.231 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.232 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:148479 5.a 5:51e5171a:::benchmark_data_smithi102_155989_object30561:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T14:34:19.232 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:28:43.161952+0000", 2024-09-13T14:34:19.232 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 336.04695095599999, 2024-09-13T14:34:19.232 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.12857813600000001, 2024-09-13T14:34:19.232 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:19.232 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:34:19.232 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:34:19.232 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:34:19.232 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:34:19.232 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 148479 2024-09-13T14:34:19.232 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.232 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:19.232 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.233 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:19.233 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.161952+0000", 2024-09-13T14:34:19.233 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:19.233 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.233 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.233 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:19.233 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.161952+0000", 2024-09-13T14:34:19.233 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:19.233 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.233 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.233 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:19.233 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.161950+0000", 2024-09-13T14:34:19.234 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:34:19.234 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.234 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.234 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:19.234 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.161958+0000", 2024-09-13T14:34:19.234 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3050000000000004e-06 2024-09-13T14:34:19.234 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.234 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.234 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:19.234 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.161971+0000", 2024-09-13T14:34:19.234 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3526999999999999e-05 2024-09-13T14:34:19.234 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.234 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.234 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:19.235 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.161978+0000", 2024-09-13T14:34:19.235 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1729999999999998e-06 2024-09-13T14:34:19.235 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.235 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.235 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:19.235 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.178715+0000", 2024-09-13T14:34:19.235 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016737031999999999 2024-09-13T14:34:19.235 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.235 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.235 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:34:19.235 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.178871+0000", 2024-09-13T14:34:19.235 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015592700000000001 2024-09-13T14:34:19.235 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.235 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.236 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,3", 2024-09-13T14:34:19.236 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.179040+0000", 2024-09-13T14:34:19.236 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016913299999999999 2024-09-13T14:34:19.236 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.236 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.236 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:34:19.236 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.183814+0000", 2024-09-13T14:34:19.236 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0047731980000000002 2024-09-13T14:34:19.236 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.236 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.236 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:34:19.236 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.191883+0000", 2024-09-13T14:34:19.236 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0080691019999999999 2024-09-13T14:34:19.237 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.237 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.237 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:34:19.237 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.280583+0000", 2024-09-13T14:34:19.237 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.088700627000000004 2024-09-13T14:34:19.237 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.237 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.237 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:34:19.237 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.280634+0000", 2024-09-13T14:34:19.237 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0494000000000001e-05 2024-09-13T14:34:19.237 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.237 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.237 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:34:19.237 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:28:43.290530+0000", 2024-09-13T14:34:19.238 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0098965500000000005 2024-09-13T14:34:19.238 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:19.238 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:19.238 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:19.238 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.238 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.238 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:214623 5.a 5:5bd7a124:::benchmark_data_smithi102_155989_object96705:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-09-13T14:34:19.238 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:16.537367+0000", 2024-09-13T14:34:19.238 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 242.67153615800001, 2024-09-13T14:34:19.238 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.124219291, 2024-09-13T14:34:19.238 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:19.238 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:34:19.238 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:34:19.239 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:34:19.239 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:34:19.239 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 214623 2024-09-13T14:34:19.239 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.239 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:19.239 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.239 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:19.239 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.537367+0000", 2024-09-13T14:34:19.239 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:19.239 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.239 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.239 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:19.239 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.537367+0000", 2024-09-13T14:34:19.239 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:19.240 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.240 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.240 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:19.240 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.537366+0000", 2024-09-13T14:34:19.240 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:34:19.240 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.240 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.240 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:19.240 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.537371+0000", 2024-09-13T14:34:19.240 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2530000000000004e-06 2024-09-13T14:34:19.240 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.240 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.240 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:19.240 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.537382+0000", 2024-09-13T14:34:19.241 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1054e-05 2024-09-13T14:34:19.241 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.241 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.241 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:19.241 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.537390+0000", 2024-09-13T14:34:19.241 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.204e-06 2024-09-13T14:34:19.241 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.241 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.241 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:19.241 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.539324+0000", 2024-09-13T14:34:19.241 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001934035 2024-09-13T14:34:19.241 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.241 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.241 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:34:19.242 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.539438+0000", 2024-09-13T14:34:19.242 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011466799999999999 2024-09-13T14:34:19.242 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.242 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.242 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,3", 2024-09-13T14:34:19.242 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.539558+0000", 2024-09-13T14:34:19.242 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012010600000000001 2024-09-13T14:34:19.243 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.243 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.243 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:34:19.243 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.542679+0000", 2024-09-13T14:34:19.243 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003120589 2024-09-13T14:34:19.243 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.243 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.243 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:34:19.243 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.542723+0000", 2024-09-13T14:34:19.243 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4011000000000003e-05 2024-09-13T14:34:19.243 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.243 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.244 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:34:19.244 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.660444+0000", 2024-09-13T14:34:19.244 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.117720959 2024-09-13T14:34:19.244 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.244 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.244 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:34:19.244 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.660492+0000", 2024-09-13T14:34:19.244 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7783e-05 2024-09-13T14:34:19.244 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.244 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.244 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:34:19.244 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:16.661586+0000", 2024-09-13T14:34:19.244 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0010945989999999999 2024-09-13T14:34:19.245 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:19.245 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:19.245 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:19.245 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.245 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.245 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:233222 5.a 5:5d43e0eb:::benchmark_data_smithi102_155989_object115304:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-13T14:34:19.245 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:41.665433+0000", 2024-09-13T14:34:19.245 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 217.543470257, 2024-09-13T14:34:19.245 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.17010931900000001, 2024-09-13T14:34:19.245 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:19.245 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:34:19.245 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:34:19.245 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:34:19.246 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:34:19.246 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 233222 2024-09-13T14:34:19.246 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.246 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:19.246 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.246 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:19.246 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.665433+0000", 2024-09-13T14:34:19.246 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:19.246 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.246 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.246 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:19.246 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.665433+0000", 2024-09-13T14:34:19.246 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:19.247 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.247 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.247 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:19.247 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.665432+0000", 2024-09-13T14:34:19.247 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:34:19.247 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.247 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.247 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:19.247 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.665437+0000", 2024-09-13T14:34:19.247 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9309999999999999e-06 2024-09-13T14:34:19.247 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.247 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.247 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:19.248 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.665447+0000", 2024-09-13T14:34:19.248 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0825000000000001e-05 2024-09-13T14:34:19.248 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.248 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.248 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:19.248 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.665454+0000", 2024-09-13T14:34:19.248 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6730000000000003e-06 2024-09-13T14:34:19.248 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.248 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.248 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:19.248 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.672469+0000", 2024-09-13T14:34:19.248 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0070153790000000004 2024-09-13T14:34:19.248 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.249 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.249 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:34:19.249 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.803548+0000", 2024-09-13T14:34:19.249 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.13107861000000001 2024-09-13T14:34:19.249 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.249 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.249 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,3", 2024-09-13T14:34:19.249 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.803692+0000", 2024-09-13T14:34:19.249 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014347599999999999 2024-09-13T14:34:19.249 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.249 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.249 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:34:19.249 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.817160+0000", 2024-09-13T14:34:19.249 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.01346893 2024-09-13T14:34:19.250 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.250 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.250 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:34:19.250 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.817190+0000", 2024-09-13T14:34:19.250 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9842999999999999e-05 2024-09-13T14:34:19.250 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.250 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.250 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:34:19.250 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.834483+0000", 2024-09-13T14:34:19.250 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017293059 2024-09-13T14:34:19.250 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.250 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.250 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:34:19.251 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.834549+0000", 2024-09-13T14:34:19.251 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5988000000000006e-05 2024-09-13T14:34:19.251 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.251 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.251 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:34:19.251 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.835542+0000", 2024-09-13T14:34:19.251 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00099279300000000006 2024-09-13T14:34:19.251 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:19.251 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:19.251 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:19.251 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.251 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.251 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:233224 5.e 5:73838589:::benchmark_data_smithi102_155989_object115306:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-13T14:34:19.252 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:41.667956+0000", 2024-09-13T14:34:19.252 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 217.54094707300001, 2024-09-13T14:34:19.252 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.15649626, 2024-09-13T14:34:19.252 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:19.252 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:34:19.252 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:34:19.252 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:34:19.252 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:34:19.252 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 233224 2024-09-13T14:34:19.252 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.252 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:19.252 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.252 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:19.253 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.667956+0000", 2024-09-13T14:34:19.253 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:19.253 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.253 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.253 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:19.253 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.667956+0000", 2024-09-13T14:34:19.253 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:19.253 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.253 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.253 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:19.253 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.667955+0000", 2024-09-13T14:34:19.253 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:34:19.253 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.253 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.254 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:19.254 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.667961+0000", 2024-09-13T14:34:19.254 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5010000000000001e-06 2024-09-13T14:34:19.254 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.254 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.254 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:19.254 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.667973+0000", 2024-09-13T14:34:19.254 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1996e-05 2024-09-13T14:34:19.254 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.254 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.254 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:19.254 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.667981+0000", 2024-09-13T14:34:19.254 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4080000000000001e-06 2024-09-13T14:34:19.254 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.255 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.255 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:19.255 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.668039+0000", 2024-09-13T14:34:19.255 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8029000000000003e-05 2024-09-13T14:34:19.255 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.255 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.255 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:34:19.255 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.668149+0000", 2024-09-13T14:34:19.255 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000110889 2024-09-13T14:34:19.255 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.255 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.255 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:34:19.255 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.668277+0000", 2024-09-13T14:34:19.255 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000127541 2024-09-13T14:34:19.256 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.256 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.256 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:34:19.256 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.672498+0000", 2024-09-13T14:34:19.256 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0042207640000000001 2024-09-13T14:34:19.256 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.256 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.256 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:34:19.256 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.672514+0000", 2024-09-13T14:34:19.256 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6135999999999999e-05 2024-09-13T14:34:19.256 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.256 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.256 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:34:19.257 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.824368+0000", 2024-09-13T14:34:19.257 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.15185453600000001 2024-09-13T14:34:19.257 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.257 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.257 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:34:19.257 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.824416+0000", 2024-09-13T14:34:19.257 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7236e-05 2024-09-13T14:34:19.257 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.257 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.257 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:34:19.257 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.824452+0000", 2024-09-13T14:34:19.257 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6653000000000003e-05 2024-09-13T14:34:19.257 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:19.258 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:19.258 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:19.258 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.258 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.258 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.44546.0:233211 5.f e154/116 5:fa93aecc:::benchmark_data_smithi102_155989_object115293:head v 154'14525, mlcod=154'14520)", 2024-09-13T14:34:19.258 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:41.668158+0000", 2024-09-13T14:34:19.258 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 217.54074462400001, 2024-09-13T14:34:19.258 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.13524302999999999, 2024-09-13T14:34:19.258 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:19.258 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:34:19.258 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:19.258 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.258 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:19.258 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.668158+0000", 2024-09-13T14:34:19.259 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:19.259 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.259 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.259 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:19.259 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.668158+0000", 2024-09-13T14:34:19.259 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:19.259 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.259 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.259 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:19.259 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.668158+0000", 2024-09-13T14:34:19.259 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:34:19.259 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.259 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.260 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:19.260 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.668162+0000", 2024-09-13T14:34:19.260 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1019999999999999e-06 2024-09-13T14:34:19.260 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.260 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.260 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:19.260 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.668168+0000", 2024-09-13T14:34:19.260 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.327e-06 2024-09-13T14:34:19.260 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.260 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.260 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:19.260 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.668171+0000", 2024-09-13T14:34:19.260 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8160000000000004e-06 2024-09-13T14:34:19.261 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.261 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.261 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:19.261 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.668259+0000", 2024-09-13T14:34:19.261 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7798000000000002e-05 2024-09-13T14:34:19.261 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.261 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.261 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:34:19.261 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.668277+0000", 2024-09-13T14:34:19.261 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7402999999999999e-05 2024-09-13T14:34:19.261 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.261 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.261 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:34:19.262 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.803363+0000", 2024-09-13T14:34:19.262 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.13508611200000001 2024-09-13T14:34:19.262 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.262 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.262 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:34:19.262 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.803402+0000", 2024-09-13T14:34:19.262 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8763999999999997e-05 2024-09-13T14:34:19.262 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:19.262 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:19.262 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:19.262 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.262 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.262 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.44546.0:233213 5.7 e154/116 5:ef469b63:::benchmark_data_smithi102_155989_object115295:head v 154'14788, mlcod=154'14786)", 2024-09-13T14:34:19.263 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:41.668182+0000", 2024-09-13T14:34:19.263 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 217.54072128600001, 2024-09-13T14:34:19.263 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.137443175, 2024-09-13T14:34:19.263 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:19.263 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:34:19.263 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:19.263 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.263 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:19.263 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.668182+0000", 2024-09-13T14:34:19.263 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:19.263 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.263 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.263 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:19.264 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.668182+0000", 2024-09-13T14:34:19.264 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:19.264 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.264 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.264 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:19.264 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.668182+0000", 2024-09-13T14:34:19.264 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967296 2024-09-13T14:34:19.264 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.264 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.264 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:19.264 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.668185+0000", 2024-09-13T14:34:19.264 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0680000000000001e-06 2024-09-13T14:34:19.264 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.264 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.265 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:19.265 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.668190+0000", 2024-09-13T14:34:19.265 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9620000000000001e-06 2024-09-13T14:34:19.265 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.265 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.265 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:19.265 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.668193+0000", 2024-09-13T14:34:19.265 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4659999999999999e-06 2024-09-13T14:34:19.265 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.265 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.265 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:19.265 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.669012+0000", 2024-09-13T14:34:19.265 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000818335 2024-09-13T14:34:19.266 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.266 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.266 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:34:19.266 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.669030+0000", 2024-09-13T14:34:19.266 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8102999999999999e-05 2024-09-13T14:34:19.266 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.266 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.266 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:34:19.266 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.803406+0000", 2024-09-13T14:34:19.266 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.13437612199999999 2024-09-13T14:34:19.266 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.266 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.266 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:34:19.267 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.805625+0000", 2024-09-13T14:34:19.267 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0022192700000000002 2024-09-13T14:34:19.267 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:19.267 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:19.267 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:19.267 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.267 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.267 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDPGRecoveryDeleteReply(5.a e154,125 5:51031a25:::benchmark_data_smithi102_155989_object5896:head,132'7840)", 2024-09-13T14:34:19.267 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:41.675340+0000", 2024-09-13T14:34:19.267 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 217.533562797, 2024-09-13T14:34:19.267 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.141782088, 2024-09-13T14:34:19.267 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:19.267 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-13T14:34:19.267 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:19.268 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.268 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:19.268 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.675340+0000", 2024-09-13T14:34:19.268 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:19.268 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.268 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.268 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:19.268 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.675340+0000", 2024-09-13T14:34:19.268 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:19.268 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.268 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.268 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:19.268 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.675338+0000", 2024-09-13T14:34:19.268 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:34:19.269 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.269 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.269 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:19.269 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.675350+0000", 2024-09-13T14:34:19.269 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1246999999999999e-05 2024-09-13T14:34:19.269 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.269 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.269 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:19.269 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.803436+0000", 2024-09-13T14:34:19.269 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.12808615000000001 2024-09-13T14:34:19.269 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.269 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.269 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:19.270 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.803449+0000", 2024-09-13T14:34:19.270 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2764e-05 2024-09-13T14:34:19.270 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.270 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.270 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:19.270 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.815507+0000", 2024-09-13T14:34:19.270 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012058615999999999 2024-09-13T14:34:19.270 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.270 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.270 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:34:19.270 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.817122+0000", 2024-09-13T14:34:19.270 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0016151550000000001 2024-09-13T14:34:19.270 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:19.270 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:19.271 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:19.271 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.271 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.271 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:85609 6.e 6:7b12e9c0:::benchmark_data_smithi102_194590_object85608:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-13T14:34:19.271 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:32:17.475387+0000", 2024-09-13T14:34:19.271 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 121.733515955, 2024-09-13T14:34:19.271 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.17257324399999999, 2024-09-13T14:34:19.271 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:19.271 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:34:19.272 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:18 smithi195 ceph-mon[104262]: pgmap v611: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 2.1 GiB data, 7.8 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 871 op/s; 3573/103086 objects degraded (3.466%); 15224/103086 objects misplaced (14.768%); 3.1 MiB/s, 102 objects/s recovering 2024-09-13T14:34:19.272 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:18 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 3573/103086 objects degraded (3.466%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:34:19.272 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:34:19.273 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:34:19.273 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:34:19.273 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 85609 2024-09-13T14:34:19.273 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.273 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:19.273 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.273 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:19.273 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.475387+0000", 2024-09-13T14:34:19.273 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:19.273 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.273 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.273 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:19.274 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.475387+0000", 2024-09-13T14:34:19.274 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:19.274 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.274 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.274 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:19.274 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.475386+0000", 2024-09-13T14:34:19.274 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:34:19.274 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.274 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.274 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:19.274 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.475437+0000", 2024-09-13T14:34:19.274 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1483999999999997e-05 2024-09-13T14:34:19.274 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.274 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.275 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:19.275 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.475451+0000", 2024-09-13T14:34:19.275 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3925e-05 2024-09-13T14:34:19.275 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.275 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.275 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:19.275 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.475458+0000", 2024-09-13T14:34:19.275 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8310000000000003e-06 2024-09-13T14:34:19.275 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.275 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.275 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:19.275 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.475498+0000", 2024-09-13T14:34:19.275 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0373999999999997e-05 2024-09-13T14:34:19.276 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.276 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.276 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:34:19.276 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.475602+0000", 2024-09-13T14:34:19.276 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000103912 2024-09-13T14:34:19.276 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.276 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.276 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,7", 2024-09-13T14:34:19.276 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.475710+0000", 2024-09-13T14:34:19.276 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000107833 2024-09-13T14:34:19.276 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.276 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.276 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:34:19.276 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.478173+0000", 2024-09-13T14:34:19.277 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0024634679999999999 2024-09-13T14:34:19.277 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.277 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.277 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:34:19.277 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.478402+0000", 2024-09-13T14:34:19.277 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00022860900000000001 2024-09-13T14:34:19.277 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.277 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.277 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:34:19.277 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.647799+0000", 2024-09-13T14:34:19.277 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.16939664700000001 2024-09-13T14:34:19.277 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.277 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.278 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:34:19.278 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.647910+0000", 2024-09-13T14:34:19.278 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000111189 2024-09-13T14:34:19.278 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.278 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.278 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:34:19.278 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.647960+0000", 2024-09-13T14:34:19.278 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0599000000000001e-05 2024-09-13T14:34:19.278 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:19.278 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:19.278 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:19.278 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.278 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.279 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:85610 6.e 6:722b26f6:::benchmark_data_smithi102_194590_object85609:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-13T14:34:19.279 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:32:17.478842+0000", 2024-09-13T14:34:19.279 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 121.73006140299999, 2024-09-13T14:34:19.279 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.17296241900000001, 2024-09-13T14:34:19.279 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:19.279 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:34:19.279 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:34:19.279 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:34:19.279 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:34:19.279 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 85610 2024-09-13T14:34:19.279 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.279 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:19.279 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.279 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:19.280 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.478842+0000", 2024-09-13T14:34:19.280 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:19.280 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.280 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.280 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:19.280 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.478842+0000", 2024-09-13T14:34:19.280 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:19.280 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.280 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.280 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:19.280 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.478841+0000", 2024-09-13T14:34:19.280 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:34:19.280 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.281 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.281 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:19.281 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.478869+0000", 2024-09-13T14:34:19.281 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7820000000000001e-05 2024-09-13T14:34:19.281 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.281 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.281 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:19.281 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.478878+0000", 2024-09-13T14:34:19.281 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.3999999999999998e-06 2024-09-13T14:34:19.281 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.281 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.281 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:19.281 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.478885+0000", 2024-09-13T14:34:19.282 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7549999999999997e-06 2024-09-13T14:34:19.282 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.282 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.282 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:19.282 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.478927+0000", 2024-09-13T14:34:19.282 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2305999999999998e-05 2024-09-13T14:34:19.282 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.282 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.282 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:34:19.282 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.479029+0000", 2024-09-13T14:34:19.282 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010176800000000001 2024-09-13T14:34:19.282 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.282 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.282 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,7", 2024-09-13T14:34:19.283 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.479131+0000", 2024-09-13T14:34:19.283 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010254700000000001 2024-09-13T14:34:19.283 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.283 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.283 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:34:19.283 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.481132+0000", 2024-09-13T14:34:19.283 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0020003070000000002 2024-09-13T14:34:19.283 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.283 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.283 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:34:19.283 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.481169+0000", 2024-09-13T14:34:19.283 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7768000000000002e-05 2024-09-13T14:34:19.283 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.284 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.284 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:34:19.284 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.651724+0000", 2024-09-13T14:34:19.284 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.170555026 2024-09-13T14:34:19.284 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.284 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.284 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:34:19.284 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.651765+0000", 2024-09-13T14:34:19.284 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0918999999999999e-05 2024-09-13T14:34:19.284 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.284 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.284 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:34:19.284 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.651804+0000", 2024-09-13T14:34:19.285 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8798999999999999e-05 2024-09-13T14:34:19.285 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:19.285 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:19.285 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:19.285 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.285 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.285 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:92672 6.2 6:45976def:::benchmark_data_smithi102_194590_object92671:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-13T14:34:19.285 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:32:25.069372+0000", 2024-09-13T14:34:19.285 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 114.139531456, 2024-09-13T14:34:19.285 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.13070775600000001, 2024-09-13T14:34:19.285 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:19.285 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:34:19.285 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:34:19.286 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:34:19.286 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:34:19.286 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 92672 2024-09-13T14:34:19.286 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.286 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:19.286 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.286 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:19.286 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:25.069372+0000", 2024-09-13T14:34:19.286 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:19.286 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.286 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.286 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:19.286 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:25.069372+0000", 2024-09-13T14:34:19.286 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:19.287 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.287 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.287 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:19.287 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:25.069370+0000", 2024-09-13T14:34:19.287 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:34:19.287 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.287 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.287 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:19.287 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:25.069407+0000", 2024-09-13T14:34:19.287 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7231999999999999e-05 2024-09-13T14:34:19.287 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.287 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.287 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:19.288 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:25.069426+0000", 2024-09-13T14:34:19.288 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8766999999999998e-05 2024-09-13T14:34:19.288 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.288 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.288 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:19.288 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:25.069434+0000", 2024-09-13T14:34:19.288 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5619999999999998e-06 2024-09-13T14:34:19.288 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.288 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.288 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:19.288 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:25.069478+0000", 2024-09-13T14:34:19.288 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4084e-05 2024-09-13T14:34:19.288 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.289 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.289 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:34:19.289 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:25.069605+0000", 2024-09-13T14:34:19.289 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012722100000000001 2024-09-13T14:34:19.289 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.289 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.289 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:34:19.289 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:25.069726+0000", 2024-09-13T14:34:19.289 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000121486 2024-09-13T14:34:19.289 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.289 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.289 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:34:19.289 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:25.073653+0000", 2024-09-13T14:34:19.290 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0039262619999999998 2024-09-13T14:34:19.290 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.290 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.290 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:34:19.290 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:25.073660+0000", 2024-09-13T14:34:19.290 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.849e-06 2024-09-13T14:34:19.290 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.290 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.290 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:34:19.290 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:25.199985+0000", 2024-09-13T14:34:19.290 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.12632539500000001 2024-09-13T14:34:19.290 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.290 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.291 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:34:19.291 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:25.200038+0000", 2024-09-13T14:34:19.291 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2895000000000001e-05 2024-09-13T14:34:19.291 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.291 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.291 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:34:19.291 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:25.200079+0000", 2024-09-13T14:34:19.291 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1564999999999999e-05 2024-09-13T14:34:19.291 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:19.291 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:19.291 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:19.291 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.291 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.292 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:150704 6.2 6:420df80d:::benchmark_data_smithi102_194590_object40080:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-13T14:34:19.292 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:33:27.750319+0000", 2024-09-13T14:34:19.292 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 51.458584066, 2024-09-13T14:34:19.292 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.29454005300000002, 2024-09-13T14:34:19.292 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:19.292 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:34:19.292 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:34:19.292 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:34:19.292 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:34:19.292 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 150704 2024-09-13T14:34:19.292 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.292 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:19.292 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.292 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:19.293 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.750319+0000", 2024-09-13T14:34:19.293 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:19.293 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.293 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.293 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:19.293 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.750319+0000", 2024-09-13T14:34:19.293 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:19.293 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.293 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.293 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:19.293 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.750317+0000", 2024-09-13T14:34:19.293 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:34:19.293 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.294 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.294 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:19.294 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.750326+0000", 2024-09-13T14:34:19.294 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1650000000000006e-06 2024-09-13T14:34:19.294 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.294 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.294 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:19.294 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.750341+0000", 2024-09-13T14:34:19.294 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5962999999999999e-05 2024-09-13T14:34:19.294 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.294 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.294 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:19.294 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.750350+0000", 2024-09-13T14:34:19.294 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5609999999999999e-06 2024-09-13T14:34:19.295 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.295 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.295 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:19.295 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.754163+0000", 2024-09-13T14:34:19.295 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0038129499999999998 2024-09-13T14:34:19.295 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.295 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.295 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:34:19.295 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.754287+0000", 2024-09-13T14:34:19.295 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012447200000000001 2024-09-13T14:34:19.295 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.295 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.295 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:34:19.295 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.754398+0000", 2024-09-13T14:34:19.296 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000110999 2024-09-13T14:34:19.296 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.296 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.296 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:34:19.296 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.044342+0000", 2024-09-13T14:34:19.296 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.28994310000000001 2024-09-13T14:34:19.296 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.296 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.296 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:34:19.296 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.044753+0000", 2024-09-13T14:34:19.296 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000411925 2024-09-13T14:34:19.296 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.296 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.297 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:34:19.297 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.044780+0000", 2024-09-13T14:34:19.297 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.6696999999999999e-05 2024-09-13T14:34:19.297 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.297 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.297 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:34:19.297 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.044820+0000", 2024-09-13T14:34:19.297 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9821999999999999e-05 2024-09-13T14:34:19.297 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.297 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.297 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:34:19.297 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.044859+0000", 2024-09-13T14:34:19.297 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9088999999999998e-05 2024-09-13T14:34:19.297 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:19.298 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:19.298 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:19.298 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.298 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.298 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:150708 6.2 6:4865044e:::benchmark_data_smithi102_194590_object40084:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-13T14:34:19.298 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:33:27.754180+0000", 2024-09-13T14:34:19.298 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 51.454722775999997, 2024-09-13T14:34:19.298 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.29088824600000002, 2024-09-13T14:34:19.298 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:19.298 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:34:19.298 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:34:19.298 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:34:19.298 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:34:19.298 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 150708 2024-09-13T14:34:19.299 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.299 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:19.299 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.299 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:19.299 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.754180+0000", 2024-09-13T14:34:19.299 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:19.299 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.299 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.299 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:19.299 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.754180+0000", 2024-09-13T14:34:19.299 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:19.299 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.299 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.299 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:19.300 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.754179+0000", 2024-09-13T14:34:19.300 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:34:19.300 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.300 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.300 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:19.300 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.754184+0000", 2024-09-13T14:34:19.300 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2429999999999996e-06 2024-09-13T14:34:19.300 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.300 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.300 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:19.300 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.754194+0000", 2024-09-13T14:34:19.300 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6490000000000003e-06 2024-09-13T14:34:19.300 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.300 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.301 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:19.301 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.754201+0000", 2024-09-13T14:34:19.301 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9190000000000002e-06 2024-09-13T14:34:19.301 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.301 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.301 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:19.301 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.757984+0000", 2024-09-13T14:34:19.301 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0037832400000000002 2024-09-13T14:34:19.301 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.301 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.301 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:34:19.301 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.758104+0000", 2024-09-13T14:34:19.301 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000120047 2024-09-13T14:34:19.302 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.302 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.302 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:34:19.302 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.758220+0000", 2024-09-13T14:34:19.302 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011538400000000001 2024-09-13T14:34:19.302 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.302 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.302 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:34:19.302 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.044352+0000", 2024-09-13T14:34:19.302 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.28613211500000002 2024-09-13T14:34:19.302 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.302 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.302 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:34:19.302 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.044970+0000", 2024-09-13T14:34:19.303 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00061775299999999997 2024-09-13T14:34:19.303 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.303 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.303 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:34:19.303 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.045002+0000", 2024-09-13T14:34:19.303 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2330999999999999e-05 2024-09-13T14:34:19.303 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.303 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.303 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:34:19.303 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.045037+0000", 2024-09-13T14:34:19.303 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5042000000000001e-05 2024-09-13T14:34:19.303 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.303 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.303 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:34:19.304 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.045069+0000", 2024-09-13T14:34:19.304 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.1695000000000001e-05 2024-09-13T14:34:19.304 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:19.304 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:19.304 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:19.304 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.304 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.304 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:150715 6.e 6:7aec09e5:::benchmark_data_smithi102_194590_object40091:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-13T14:34:19.304 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:33:27.761632+0000", 2024-09-13T14:34:19.304 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 51.447271344000001, 2024-09-13T14:34:19.304 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.29790872499999999, 2024-09-13T14:34:19.304 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:19.304 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:34:19.304 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:34:19.305 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:34:19.305 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:34:19.305 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 150715 2024-09-13T14:34:19.305 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.305 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:19.305 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.305 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:19.305 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.761632+0000", 2024-09-13T14:34:19.305 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:19.305 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.305 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.305 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:19.305 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.761632+0000", 2024-09-13T14:34:19.306 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:19.306 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.306 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.306 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:19.306 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.761630+0000", 2024-09-13T14:34:19.306 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:34:19.306 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.306 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.306 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:19.306 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.761638+0000", 2024-09-13T14:34:19.306 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1049999999999995e-06 2024-09-13T14:34:19.306 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.306 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.306 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:19.307 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.044265+0000", 2024-09-13T14:34:19.307 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.28262651100000002 2024-09-13T14:34:19.307 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.307 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.307 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:19.307 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.044280+0000", 2024-09-13T14:34:19.307 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5600999999999999e-05 2024-09-13T14:34:19.307 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.307 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.307 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:19.307 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.044408+0000", 2024-09-13T14:34:19.307 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012831499999999999 2024-09-13T14:34:19.307 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.307 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.308 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:34:19.308 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.044549+0000", 2024-09-13T14:34:19.308 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014089899999999999 2024-09-13T14:34:19.308 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.308 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.308 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,7", 2024-09-13T14:34:19.308 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.044679+0000", 2024-09-13T14:34:19.308 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012917399999999999 2024-09-13T14:34:19.308 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.308 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.308 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:34:19.308 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.059197+0000", 2024-09-13T14:34:19.308 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014518939999999999 2024-09-13T14:34:19.308 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.309 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.309 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:34:19.309 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.059225+0000", 2024-09-13T14:34:19.309 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7288999999999999e-05 2024-09-13T14:34:19.309 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.309 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.309 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:34:19.309 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.059358+0000", 2024-09-13T14:34:19.309 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000133122 2024-09-13T14:34:19.309 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.309 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.309 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:34:19.309 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.059449+0000", 2024-09-13T14:34:19.310 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1187000000000006e-05 2024-09-13T14:34:19.310 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.310 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.310 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:34:19.310 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.059540+0000", 2024-09-13T14:34:19.310 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1434000000000006e-05 2024-09-13T14:34:19.310 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:19.310 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:19.310 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:19.310 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.310 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.310 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35037.0:150736 6.7 e178/161 6:eea7a7c2:::benchmark_data_smithi102_194590_object40112:head v 178'9515, mlcod=178'9513)", 2024-09-13T14:34:19.310 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:33:27.781737+0000", 2024-09-13T14:34:19.310 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 51.427166577999998, 2024-09-13T14:34:19.311 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.283366118, 2024-09-13T14:34:19.311 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:19.311 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:34:19.311 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:19.311 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.311 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:19.311 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.781737+0000", 2024-09-13T14:34:19.311 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:19.311 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.311 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.311 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:19.311 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.781737+0000", 2024-09-13T14:34:19.311 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:19.312 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.312 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.312 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:19.312 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.781735+0000", 2024-09-13T14:34:19.312 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:34:19.312 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.312 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.312 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:19.312 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.781744+0000", 2024-09-13T14:34:19.312 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4760000000000004e-06 2024-09-13T14:34:19.312 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.312 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.312 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:19.312 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.044267+0000", 2024-09-13T14:34:19.313 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.26252262599999998 2024-09-13T14:34:19.313 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.313 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.313 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:19.313 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.044279+0000", 2024-09-13T14:34:19.313 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2354000000000001e-05 2024-09-13T14:34:19.313 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.313 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.313 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:19.313 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.044431+0000", 2024-09-13T14:34:19.313 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0001516 2024-09-13T14:34:19.313 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.313 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.314 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:34:19.314 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.044453+0000", 2024-09-13T14:34:19.314 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2124999999999999e-05 2024-09-13T14:34:19.314 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.314 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.314 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:34:19.314 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.065076+0000", 2024-09-13T14:34:19.314 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.020622628 2024-09-13T14:34:19.314 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.314 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.314 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:34:19.314 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.065103+0000", 2024-09-13T14:34:19.314 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7078999999999999e-05 2024-09-13T14:34:19.314 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:19.315 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:19.315 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:19.315 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.315 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.315 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:163490 6.e 6:7a5e8c84:::benchmark_data_smithi102_194590_object52866:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-13T14:34:19.315 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:33:43.515900+0000", 2024-09-13T14:34:19.315 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 35.693002767999999, 2024-09-13T14:34:19.315 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.555967597, 2024-09-13T14:34:19.315 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:19.315 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:34:19.315 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:34:19.315 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:34:19.315 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:34:19.316 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 163490 2024-09-13T14:34:19.316 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.316 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:19.316 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.316 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:19.316 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.515900+0000", 2024-09-13T14:34:19.316 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:19.316 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.316 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.316 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:19.316 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.515900+0000", 2024-09-13T14:34:19.316 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:19.316 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.316 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.317 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:19.317 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.515899+0000", 2024-09-13T14:34:19.317 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:34:19.317 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.317 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.317 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:19.317 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.515905+0000", 2024-09-13T14:34:19.317 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3219999999999998e-06 2024-09-13T14:34:19.317 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.317 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.317 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:19.317 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.515929+0000", 2024-09-13T14:34:19.317 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3445e-05 2024-09-13T14:34:19.318 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.318 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.318 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:19.318 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.515937+0000", 2024-09-13T14:34:19.318 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2530000000000004e-06 2024-09-13T14:34:19.318 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.318 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.318 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:19.318 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.533987+0000", 2024-09-13T14:34:19.318 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018049744 2024-09-13T14:34:19.318 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.318 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.318 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:34:19.319 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.533995+0000", 2024-09-13T14:34:19.319 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0250000000000002e-06 2024-09-13T14:34:19.319 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.319 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.319 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:19.319 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.820467+0000", 2024-09-13T14:34:19.319 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.28647251400000001 2024-09-13T14:34:19.319 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.319 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.319 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:34:19.319 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.820483+0000", 2024-09-13T14:34:19.319 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.596e-05 2024-09-13T14:34:19.319 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.319 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.320 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:19.320 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.025746+0000", 2024-09-13T14:34:19.320 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2052628699999999 2024-09-13T14:34:19.320 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.320 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.320 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:34:19.320 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.025880+0000", 2024-09-13T14:34:19.320 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000134039 2024-09-13T14:34:19.320 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.320 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.320 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6,7", 2024-09-13T14:34:19.320 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.026451+0000", 2024-09-13T14:34:19.320 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00057072799999999995 2024-09-13T14:34:19.321 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.321 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.321 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:34:19.321 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.061832+0000", 2024-09-13T14:34:19.321 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.035381126999999998 2024-09-13T14:34:19.321 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.321 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.321 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:34:19.321 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.063912+0000", 2024-09-13T14:34:19.321 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0020797739999999999 2024-09-13T14:34:19.321 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.321 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.321 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:34:19.322 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.066465+0000", 2024-09-13T14:34:19.322 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0025533280000000001 2024-09-13T14:34:19.322 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.322 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.322 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:34:19.322 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.071666+0000", 2024-09-13T14:34:19.322 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0052006839999999997 2024-09-13T14:34:19.322 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.322 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.322 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:34:19.322 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.071756+0000", 2024-09-13T14:34:19.322 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.0733e-05 2024-09-13T14:34:19.322 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.322 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.323 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:34:19.323 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.071868+0000", 2024-09-13T14:34:19.323 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000111537 2024-09-13T14:34:19.323 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:19.323 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:19.323 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:19.323 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.323 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.323 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:163524 6.e 6:7ba5d4da:::benchmark_data_smithi102_194590_object52900:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-09-13T14:34:19.323 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:33:43.559673+0000", 2024-09-13T14:34:19.323 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 35.649230492999997, 2024-09-13T14:34:19.323 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5124033290000001, 2024-09-13T14:34:19.324 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:19.324 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:34:19.324 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:34:19.324 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:34:19.324 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:34:19.324 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 163524 2024-09-13T14:34:19.324 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.324 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:19.324 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.324 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:19.324 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.559673+0000", 2024-09-13T14:34:19.324 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:19.325 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.325 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.325 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:19.325 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.559673+0000", 2024-09-13T14:34:19.325 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:19.325 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.325 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.325 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:19.325 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.559672+0000", 2024-09-13T14:34:19.325 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:34:19.325 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.325 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.325 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:19.325 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.559676+0000", 2024-09-13T14:34:19.326 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6859999999999997e-06 2024-09-13T14:34:19.326 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.326 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.326 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:19.326 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.559687+0000", 2024-09-13T14:34:19.326 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0514e-05 2024-09-13T14:34:19.326 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.326 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.326 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:19.326 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.559691+0000", 2024-09-13T14:34:19.326 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4989999999999996e-06 2024-09-13T14:34:19.326 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.326 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.326 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:19.327 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.559730+0000", 2024-09-13T14:34:19.327 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8816999999999998e-05 2024-09-13T14:34:19.327 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.327 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.327 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:34:19.327 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.559740+0000", 2024-09-13T14:34:19.327 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.5319999999999995e-06 2024-09-13T14:34:19.327 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.327 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.327 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:19.327 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.820610+0000", 2024-09-13T14:34:19.327 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.26086995699999999 2024-09-13T14:34:19.327 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.327 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.328 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:34:19.328 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.820618+0000", 2024-09-13T14:34:19.328 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.6580000000000008e-06 2024-09-13T14:34:19.328 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.328 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.328 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:19.328 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.040556+0000", 2024-09-13T14:34:19.328 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2199382860000001 2024-09-13T14:34:19.328 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.328 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.328 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:34:19.328 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.040691+0000", 2024-09-13T14:34:19.328 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000134497 2024-09-13T14:34:19.328 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.329 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.329 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6,7", 2024-09-13T14:34:19.329 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.040831+0000", 2024-09-13T14:34:19.329 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000140359 2024-09-13T14:34:19.329 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.329 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.329 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:34:19.329 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.061864+0000", 2024-09-13T14:34:19.329 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.021032660000000002 2024-09-13T14:34:19.329 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.329 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.329 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:34:19.329 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.066495+0000", 2024-09-13T14:34:19.330 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0046312369999999999 2024-09-13T14:34:19.330 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.330 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.330 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:34:19.330 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.068838+0000", 2024-09-13T14:34:19.330 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0023430949999999999 2024-09-13T14:34:19.330 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.330 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.330 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:34:19.330 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.071910+0000", 2024-09-13T14:34:19.330 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0030720309999999998 2024-09-13T14:34:19.330 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.330 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.330 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:34:19.331 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.071984+0000", 2024-09-13T14:34:19.331 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3590000000000005e-05 2024-09-13T14:34:19.331 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.331 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.331 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:34:19.331 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.072076+0000", 2024-09-13T14:34:19.331 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1986999999999999e-05 2024-09-13T14:34:19.331 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:19.331 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:19.331 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:19.331 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.331 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.331 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:163533 6.e 6:70a00fb8:::benchmark_data_smithi102_194590_object52909:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-09-13T14:34:19.331 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:33:43.580375+0000", 2024-09-13T14:34:19.332 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 35.628528490000001, 2024-09-13T14:34:19.332 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4918938339999999, 2024-09-13T14:34:19.332 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:19.332 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:34:19.332 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:34:19.332 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:34:19.332 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:34:19.332 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 163533 2024-09-13T14:34:19.332 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.332 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:19.332 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.332 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:19.333 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.580375+0000", 2024-09-13T14:34:19.333 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:19.333 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.333 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.333 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:19.333 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.580375+0000", 2024-09-13T14:34:19.333 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:19.333 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.333 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.333 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:19.333 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.580373+0000", 2024-09-13T14:34:19.333 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:34:19.333 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.333 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.334 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:19.334 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.580380+0000", 2024-09-13T14:34:19.334 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6980000000000001e-06 2024-09-13T14:34:19.334 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.334 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.334 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:19.334 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.580392+0000", 2024-09-13T14:34:19.334 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2675e-05 2024-09-13T14:34:19.334 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.334 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.334 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:19.334 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.580399+0000", 2024-09-13T14:34:19.334 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.674e-06 2024-09-13T14:34:19.335 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.335 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.335 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:19.335 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.580441+0000", 2024-09-13T14:34:19.335 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2077999999999999e-05 2024-09-13T14:34:19.335 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.335 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.335 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:34:19.335 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.580449+0000", 2024-09-13T14:34:19.335 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7300000000000005e-06 2024-09-13T14:34:19.335 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.335 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.335 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:19.335 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.820640+0000", 2024-09-13T14:34:19.336 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.240191126 2024-09-13T14:34:19.336 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.336 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.336 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:34:19.336 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.820648+0000", 2024-09-13T14:34:19.336 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0469999999999994e-06 2024-09-13T14:34:19.336 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.336 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.336 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:19.336 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.043836+0000", 2024-09-13T14:34:19.336 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2231876559999999 2024-09-13T14:34:19.336 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.336 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.336 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:34:19.337 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.043970+0000", 2024-09-13T14:34:19.337 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013394399999999999 2024-09-13T14:34:19.337 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.337 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.337 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6,7", 2024-09-13T14:34:19.337 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.044102+0000", 2024-09-13T14:34:19.337 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000132254 2024-09-13T14:34:19.337 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.337 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.337 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:34:19.337 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.061893+0000", 2024-09-13T14:34:19.337 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017791238000000001 2024-09-13T14:34:19.337 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.338 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.338 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:34:19.338 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.066525+0000", 2024-09-13T14:34:19.338 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0046314099999999999 2024-09-13T14:34:19.338 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.338 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.338 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:34:19.338 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.068872+0000", 2024-09-13T14:34:19.338 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002347605 2024-09-13T14:34:19.338 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.338 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.338 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:34:19.338 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.072115+0000", 2024-09-13T14:34:19.338 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0032423840000000001 2024-09-13T14:34:19.339 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.339 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.339 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:34:19.339 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.072177+0000", 2024-09-13T14:34:19.339 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2679000000000001e-05 2024-09-13T14:34:19.339 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:19.339 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:19.339 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:34:19.339 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.072268+0000", 2024-09-13T14:34:19.339 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1260000000000004e-05 2024-09-13T14:34:19.339 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:19.339 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:19.339 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:19.339 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:19.339 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:19.340 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:34:19.340 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:19.389 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:19.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:19.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:19.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:18 smithi102 ceph-mon[110183]: pgmap v611: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 2.1 GiB data, 7.8 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 871 op/s; 3573/103086 objects degraded (3.466%); 15224/103086 objects misplaced (14.768%); 3.1 MiB/s, 102 objects/s recovering 2024-09-13T14:34:19.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:18 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 3573/103086 objects degraded (3.466%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:34:19.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:18 smithi102 ceph-mon[115269]: pgmap v611: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 2.1 GiB data, 7.8 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 871 op/s; 3573/103086 objects degraded (3.466%); 15224/103086 objects misplaced (14.768%); 3.1 MiB/s, 102 objects/s recovering 2024-09-13T14:34:19.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:18 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 3573/103086 objects degraded (3.466%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:34:19.491 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:19.514 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:19.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:19.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:19.616 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:19.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:19.643 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:19.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:19.743 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:19.770 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:19.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:19.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:19.872 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:19.915 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:19.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:19.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:20.017 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:20.044 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:20.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:20.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:20.146 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:20.174 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:20.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:20.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:20.276 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:20.301 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:20.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:20.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:20.403 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:20.424 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:20.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:20.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:20.526 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:20.547 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T14:34:20.572 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:20.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:20.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:20.674 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:20.695 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:20.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:20.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:20.797 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:20.825 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:20.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:20.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:20.913 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:34:20.928 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:20.976 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:20.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:20.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:20.990 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:20 smithi195 ceph-mon[104262]: pgmap v612: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 2.1 GiB data, 7.7 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 865 op/s; 3573/101484 objects degraded (3.521%); 14944/101484 objects misplaced (14.725%); 3.1 MiB/s, 102 objects/s recovering 2024-09-13T14:34:21.077 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:21.120 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:21.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:21.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:21.223 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:21.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:21.245 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:21.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:21.346 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:21.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:21.372 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:21.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:21.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:20 smithi102 ceph-mon[110183]: pgmap v612: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 2.1 GiB data, 7.7 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 865 op/s; 3573/101484 objects degraded (3.521%); 14944/101484 objects misplaced (14.725%); 3.1 MiB/s, 102 objects/s recovering 2024-09-13T14:34:21.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:20 smithi102 ceph-mon[115269]: pgmap v612: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 2.1 GiB data, 7.7 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 865 op/s; 3573/101484 objects degraded (3.521%); 14944/101484 objects misplaced (14.725%); 3.1 MiB/s, 102 objects/s recovering 2024-09-13T14:34:21.474 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:21.518 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:21.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:21.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:21.620 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:21.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:21.647 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:21.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:21.748 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:21.783 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:21.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:21.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:21.886 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:21.918 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:21.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:21.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:22.020 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:22.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:22.063 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:22.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:22.164 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:34:22.199 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:22.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:22.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:34:22.301 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:34:22.323 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:22.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:22.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:34:22.425 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:22.445 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:22.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:22.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:22.547 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:22.569 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:22.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:22.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:22.670 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:22.698 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:22.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:22.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:22.800 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:22.848 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:22.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:22.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:22.950 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:22.992 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:22.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:22.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:23.094 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:34:23.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:23.142 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:23.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:34:23.243 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:23.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:22 smithi195 ceph-mon[104262]: pgmap v613: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 2.1 GiB data, 7.7 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 865 op/s; 3573/101484 objects degraded (3.521%); 14944/101484 objects misplaced (14.725%); 3.1 MiB/s, 102 objects/s recovering 2024-09-13T14:34:23.282 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:23.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:23.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:23.385 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:23.410 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:23.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:23.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:23.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:22 smithi102 ceph-mon[110183]: pgmap v613: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 2.1 GiB data, 7.7 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 865 op/s; 3573/101484 objects degraded (3.521%); 14944/101484 objects misplaced (14.725%); 3.1 MiB/s, 102 objects/s recovering 2024-09-13T14:34:23.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:22 smithi102 ceph-mon[115269]: pgmap v613: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 2.1 GiB data, 7.7 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 865 op/s; 3573/101484 objects degraded (3.521%); 14944/101484 objects misplaced (14.725%); 3.1 MiB/s, 102 objects/s recovering 2024-09-13T14:34:23.485 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:34:23.485 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-13T14:34:23.486 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:23.486 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35037.0:192367 6.a e192/189 6:5c33c571:::benchmark_data_smithi102_194590_object81743:head v 192'12053, mlcod=192'12051)", 2024-09-13T14:34:23.486 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:34:23.467874+0000", 2024-09-13T14:34:23.486 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.015457145, 2024-09-13T14:34:23.486 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.015484981, 2024-09-13T14:34:23.486 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:23.487 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T14:34:23.487 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:23.487 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:23.487 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:23.487 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:23.467874+0000", 2024-09-13T14:34:23.487 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:23.487 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:23.487 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:23.487 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:23.487 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:23.467874+0000", 2024-09-13T14:34:23.487 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:23.487 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:23.487 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:23.487 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:23.488 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:23.467873+0000", 2024-09-13T14:34:23.488 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:34:23.488 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:23.488 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:23.488 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:23.488 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:23.467883+0000", 2024-09-13T14:34:23.488 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4919999999999999e-06 2024-09-13T14:34:23.488 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:23.488 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:23.488 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:23.488 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:23.467899+0000", 2024-09-13T14:34:23.488 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6011e-05 2024-09-13T14:34:23.488 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:23.489 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:23.489 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:23.489 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:23.467905+0000", 2024-09-13T14:34:23.489 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0689999999999998e-06 2024-09-13T14:34:23.489 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:23.489 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:23.489 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:23.489 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:23.468355+0000", 2024-09-13T14:34:23.489 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00044965800000000002 2024-09-13T14:34:23.489 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:23.489 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:23.489 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:34:23.489 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:23.468402+0000", 2024-09-13T14:34:23.490 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6971999999999997e-05 2024-09-13T14:34:23.490 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:23.490 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:23.490 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:23.490 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:23.490 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:23.490 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35037.0:192373 6.a e192/189 6:5eba97c1:::benchmark_data_smithi102_194590_object81749:head v 192'12054, mlcod=192'12051)", 2024-09-13T14:34:23.490 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:34:23.477462+0000", 2024-09-13T14:34:23.490 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.0058695659999999997, 2024-09-13T14:34:23.490 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0059584, 2024-09-13T14:34:23.490 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:23.490 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T14:34:23.490 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:23.490 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:23.491 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:23.491 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:23.477462+0000", 2024-09-13T14:34:23.491 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:23.491 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:23.491 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:23.491 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:23.491 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:23.477462+0000", 2024-09-13T14:34:23.491 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:23.491 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:23.491 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:23.491 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:23.491 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:23.477461+0000", 2024-09-13T14:34:23.491 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:34:23.491 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:23.492 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:23.492 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:23.492 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:23.477467+0000", 2024-09-13T14:34:23.492 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8669999999999999e-06 2024-09-13T14:34:23.492 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:23.492 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:23.492 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:23.492 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:23.477479+0000", 2024-09-13T14:34:23.492 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2748999999999999e-05 2024-09-13T14:34:23.492 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:23.492 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:23.492 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:23.492 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:23.477486+0000", 2024-09-13T14:34:23.493 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3980000000000004e-06 2024-09-13T14:34:23.493 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:23.493 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:23.493 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:23.493 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:23.479966+0000", 2024-09-13T14:34:23.493 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0024799520000000001 2024-09-13T14:34:23.493 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:23.493 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:23.493 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:34:23.493 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:23.479987+0000", 2024-09-13T14:34:23.493 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1403000000000002e-05 2024-09-13T14:34:23.493 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:23.493 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:23.493 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:23.494 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:23.494 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:23.494 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35037.0:192375 6.1 e192/189 6:873cd6f1:::benchmark_data_smithi102_194590_object81751:head v 192'11833, mlcod=177'7856)", 2024-09-13T14:34:23.494 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:34:23.477499+0000", 2024-09-13T14:34:23.494 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.0058324969999999999, 2024-09-13T14:34:23.494 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0059751140000000001, 2024-09-13T14:34:23.494 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:23.494 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T14:34:23.494 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:23.494 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:23.494 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:23.494 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:23.477499+0000", 2024-09-13T14:34:23.494 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:23.495 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:23.495 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:23.495 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:23.495 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:23.477499+0000", 2024-09-13T14:34:23.495 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:23.495 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:23.495 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:23.495 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:23.495 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:23.477499+0000", 2024-09-13T14:34:23.495 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:34:23.495 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:23.495 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:23.495 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:23.496 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:23.477502+0000", 2024-09-13T14:34:23.496 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.822e-06 2024-09-13T14:34:23.496 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:23.496 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:23.496 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:23.496 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:23.477508+0000", 2024-09-13T14:34:23.496 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4990000000000002e-06 2024-09-13T14:34:23.496 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:23.496 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:23.496 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:23.496 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:23.477512+0000", 2024-09-13T14:34:23.496 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7139999999999999e-06 2024-09-13T14:34:23.496 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:23.496 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:23.496 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:23.497 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:23.477566+0000", 2024-09-13T14:34:23.497 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4387e-05 2024-09-13T14:34:23.497 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:23.497 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:23.497 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:34:23.497 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:23.477588+0000", 2024-09-13T14:34:23.497 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2370000000000001e-05 2024-09-13T14:34:23.497 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:23.497 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:23.497 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:23.497 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:23.497 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:23.497 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35037.0:192377 6.d e192/183 6:b4e5f2ac:::benchmark_data_smithi102_194590_object81753:head v 192'12201, mlcod=192'12199)", 2024-09-13T14:34:23.497 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:34:23.482355+0000", 2024-09-13T14:34:23.498 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.00097580300000000002, 2024-09-13T14:34:23.498 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001172738, 2024-09-13T14:34:23.498 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:23.498 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T14:34:23.498 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:23.498 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:23.498 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:23.498 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:23.482355+0000", 2024-09-13T14:34:23.498 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:23.498 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:23.498 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:23.498 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:23.498 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:23.482355+0000", 2024-09-13T14:34:23.499 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:23.499 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:23.499 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:23.499 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:23.499 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:23.482355+0000", 2024-09-13T14:34:23.499 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:34:23.499 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:23.499 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:23.499 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:23.499 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:23.482364+0000", 2024-09-13T14:34:23.499 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1050000000000001e-06 2024-09-13T14:34:23.499 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:23.499 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:23.499 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:23.499 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:23.482374+0000", 2024-09-13T14:34:23.500 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0479000000000001e-05 2024-09-13T14:34:23.500 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:23.500 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:23.500 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:23.500 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:23.482381+0000", 2024-09-13T14:34:23.500 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4620000000000001e-06 2024-09-13T14:34:23.500 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:23.500 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:23.500 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:23.500 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:23.482723+0000", 2024-09-13T14:34:23.500 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00034261399999999998 2024-09-13T14:34:23.500 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:23.500 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:23.501 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:34:23.501 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:23.482748+0000", 2024-09-13T14:34:23.501 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5072999999999999e-05 2024-09-13T14:34:23.501 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:23.501 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:23.501 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:23.501 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:23.501 INFO:teuthology.orchestra.run.smithi195.stdout: ], 2024-09-13T14:34:23.501 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 4 2024-09-13T14:34:23.501 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:34:23.513 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:23.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:23.535 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:23.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:23.636 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:23.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:23.664 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:23.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:23.765 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:23.793 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:23.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:23.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:23.895 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:34:23.932 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:23.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:23.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:34:24.034 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:24.073 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:24.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:24.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:24.175 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:24.204 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:24.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:24.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:24.231 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:24 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 3573/101484 objects degraded (3.521%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:34:24.267 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T14:34:24.305 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:24.333 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:24.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:24.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:24.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:24 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 3573/101484 objects degraded (3.521%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:34:24.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:24 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 3573/101484 objects degraded (3.521%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:34:24.435 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:24.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:24.485 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:24.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:24.560 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:34:24.586 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:24.614 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:24.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:24.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:24.715 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:24.743 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:24.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:24.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:24.844 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:24.872 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:24.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:24.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:24.975 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:25.011 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:25.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:25.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:25.112 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:25.141 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:25.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:25.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:25.243 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:25.249 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:25 smithi102 ceph-mon[110183]: pgmap v614: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 1.8 GiB data, 7.1 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 824 op/s; 3088/89490 objects degraded (3.451%); 13123/89490 objects misplaced (14.664%); 2.9 MiB/s, 98 objects/s recovering 2024-09-13T14:34:25.249 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:25 smithi102 ceph-mon[115269]: pgmap v614: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 1.8 GiB data, 7.1 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 824 op/s; 3088/89490 objects degraded (3.451%); 13123/89490 objects misplaced (14.664%); 2.9 MiB/s, 98 objects/s recovering 2024-09-13T14:34:25.285 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:25.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:25.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:25.386 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:25.418 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:25.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:25.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:25.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:25 smithi195 ceph-mon[104262]: pgmap v614: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 1.8 GiB data, 7.1 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 824 op/s; 3088/89490 objects degraded (3.451%); 13123/89490 objects misplaced (14.664%); 2.9 MiB/s, 98 objects/s recovering 2024-09-13T14:34:25.520 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:25.553 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:25.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:25.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:25.655 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:25.684 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:25.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:25.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:25.785 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:25.814 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:25.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:25.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:25.915 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:25.973 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:25.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:25.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:25.977 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:34:26.074 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:26.125 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:26.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:26.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:26.226 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:34:26.271 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:26.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:26.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:34:26.373 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:26.409 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:26.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:26.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:26.512 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:26.519 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:26 smithi195 ceph-mon[104262]: pgmap v615: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 1.8 GiB data, 7.1 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 824 op/s; 3088/89490 objects degraded (3.451%); 13123/89490 objects misplaced (14.664%); 2.9 MiB/s, 98 objects/s recovering 2024-09-13T14:34:26.520 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:34:26.520 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:34:26.527 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:26 smithi102 ceph-mon[110183]: pgmap v615: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 1.8 GiB data, 7.1 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 824 op/s; 3088/89490 objects degraded (3.451%); 13123/89490 objects misplaced (14.664%); 2.9 MiB/s, 98 objects/s recovering 2024-09-13T14:34:26.528 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:34:26.528 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:34:26.528 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:26 smithi102 ceph-mon[115269]: pgmap v615: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 1.8 GiB data, 7.1 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 824 op/s; 3088/89490 objects degraded (3.451%); 13123/89490 objects misplaced (14.664%); 2.9 MiB/s, 98 objects/s recovering 2024-09-13T14:34:26.529 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:34:26.529 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:34:26.529 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:34:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:34:26] "GET /metrics HTTP/1.1" 200 36549 "" "Prometheus/2.51.0" 2024-09-13T14:34:26.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:26.550 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:26.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:26.587 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:34:26.587 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-13T14:34:26.587 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-09-13T14:34:26.587 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-09-13T14:34:26.587 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:34:26.651 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:26.680 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:26.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:26.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:26.782 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:26.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:26.829 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:26.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:26.931 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:26.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:26.966 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:26.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:27.032 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:34:27.067 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:27.098 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:27.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:27.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:27.200 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:27.228 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:27.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:27.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:27.313 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T14:34:27.330 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:27.362 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:27.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:27.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:27.434 INFO:teuthology.orchestra.run.smithi102.stderr:set pool 6 pgp_num to 16 2024-09-13T14:34:27.464 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:34:27.528 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:27.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:27.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:34:27.630 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:27.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:27.683 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:27.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:27.691 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:34:27.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:27 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/535774568' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"} : dispatch 2024-09-13T14:34:27.785 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:27.836 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:27.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:27.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:27.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:27 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/535774568' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"} : dispatch 2024-09-13T14:34:27.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:27 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/535774568' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"} : dispatch 2024-09-13T14:34:27.938 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:34:27.983 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:27.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:27.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:34:28.085 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:28.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:28.130 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:28.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:28.232 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:28.267 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:28.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:28.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:28.369 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:28.414 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:28.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:28.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:28.516 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:28.546 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:28.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:28.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:28.648 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:28.658 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:28 smithi102 ceph-mon[115269]: pgmap v616: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 1.6 GiB data, 6.5 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 1.05k op/s; 2603/78198 objects degraded (3.329%); 11441/78198 objects misplaced (14.631%); 2.5 MiB/s, 113 objects/s recovering 2024-09-13T14:34:28.658 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:28 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/535774568' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]': finished 2024-09-13T14:34:28.658 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:28 smithi102 ceph-mon[115269]: osdmap e193: 8 total, 8 up, 5 in 2024-09-13T14:34:28.659 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:28 smithi102 ceph-mon[110183]: pgmap v616: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 1.6 GiB data, 6.5 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 1.05k op/s; 2603/78198 objects degraded (3.329%); 11441/78198 objects misplaced (14.631%); 2.5 MiB/s, 113 objects/s recovering 2024-09-13T14:34:28.659 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:28 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/535774568' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]': finished 2024-09-13T14:34:28.659 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:28 smithi102 ceph-mon[110183]: osdmap e193: 8 total, 8 up, 5 in 2024-09-13T14:34:28.677 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:28.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:28.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:28.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:28 smithi195 ceph-mon[104262]: pgmap v616: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 1.6 GiB data, 6.5 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 1.05k op/s; 2603/78198 objects degraded (3.329%); 11441/78198 objects misplaced (14.631%); 2.5 MiB/s, 113 objects/s recovering 2024-09-13T14:34:28.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:28 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/535774568' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]': finished 2024-09-13T14:34:28.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:28 smithi195 ceph-mon[104262]: osdmap e193: 8 total, 8 up, 5 in 2024-09-13T14:34:28.779 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:34:28.799 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:28.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:28.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:34:28.901 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:28.944 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:28.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:28.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:29.046 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:29.086 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:29.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:29.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:29.188 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:34:29.234 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:29.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:29.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:34:29.336 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:29.366 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:29.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:29.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:29.468 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:29.493 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:29.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:29.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:29.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:29 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 2603/78198 objects degraded (3.329%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:34:29.595 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:29.635 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:29.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:29.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:29.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:29 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 2603/78198 objects degraded (3.329%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:34:29.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:29 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 2603/78198 objects degraded (3.329%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:34:29.737 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:29.758 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:29.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:29.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:29.860 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:29.882 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:29.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:29.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:29.984 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:30.009 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:30.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:30.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:30.111 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:30.143 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:30.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:30.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:30.200 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:34:30.246 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:30.287 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:30.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:30.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:30.389 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:30.428 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:30.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:30.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:30.530 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:30.569 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:30.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:30.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:30.671 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:34:30.709 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:30.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:30.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:34:30.811 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:30.839 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:30.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:30.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:30.941 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:30.963 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:34:30.964 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:34:30.980 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:30.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:30.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:30.986 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:34:30.986 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:34:31.009 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:34:31.009 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:34:31.029 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:34:31.030 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:34:31.056 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:34:31.056 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:34:31.081 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:31.089 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:34:31.090 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:34:31.125 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:31.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:31.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:31.130 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:34:31.130 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:34:31.161 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:34:31.161 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:34:31.230 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:31.261 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:31.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:31.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:31.338 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:31 smithi195 ceph-mon[104262]: pgmap v618: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 1.6 GiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 835 op/s; 2603/76413 objects degraded (3.406%); 11155/76413 objects misplaced (14.598%); 1.3 MiB/s, 78 objects/s recovering 2024-09-13T14:34:31.338 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:31 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:34:31.338 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:31 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:34:31.338 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:31 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2239594239' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:34:31.362 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:31.411 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:31.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:31.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:31.513 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:31.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:31.557 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:31.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:31.659 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:31.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:31 smithi102 ceph-mon[110183]: pgmap v618: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 1.6 GiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 835 op/s; 2603/76413 objects degraded (3.406%); 11155/76413 objects misplaced (14.598%); 1.3 MiB/s, 78 objects/s recovering 2024-09-13T14:34:31.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:31 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:34:31.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:31 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:34:31.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:31 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2239594239' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:34:31.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:31 smithi102 ceph-mon[115269]: pgmap v618: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 1.6 GiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 835 op/s; 2603/76413 objects degraded (3.406%); 11155/76413 objects misplaced (14.598%); 1.3 MiB/s, 78 objects/s recovering 2024-09-13T14:34:31.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:31 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:34:31.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:31 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:34:31.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:31 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2239594239' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:34:31.684 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:31.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:31.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:31.786 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:31.812 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:31.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:31.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:31.914 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:34:31.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:31.955 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:31.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:34:32.056 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:34:32.084 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:34:32.084 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:34:32.101 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:32.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:32.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:34:32.203 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:32.246 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:32.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:32.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:32.348 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:32.374 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:32.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:32.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:32.476 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:32.502 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:32.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:32.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:32.604 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:32.630 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:32.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:32.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:32.732 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:32.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:32.769 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:32.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:32.860 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T14:34:32.870 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:32.898 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:32.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:32.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:33.000 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:33.031 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:33.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:33.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:33.133 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:33.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:33.187 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:33.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:33.265 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:34:33.287 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:33.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:33.340 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:33.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:33.366 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 6, 7] out_osds: [1, 2, 5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:34:33.366 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:34:33.367 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-13T14:34:33.367 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.0 'trim stale osdmaps' 2024-09-13T14:34:33.440 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:33.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:33.481 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:33.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:33.496 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:33 smithi102 ceph-mon[110183]: pgmap v619: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 1.6 GiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 835 op/s; 2603/76413 objects degraded (3.406%); 11155/76413 objects misplaced (14.598%); 1.3 MiB/s, 78 objects/s recovering 2024-09-13T14:34:33.496 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:33 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:34:33.496 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:33 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:34:33.496 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:33 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:34:33.497 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:33 smithi102 ceph-mon[115269]: pgmap v619: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 1.6 GiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 835 op/s; 2603/76413 objects degraded (3.406%); 11155/76413 objects misplaced (14.598%); 1.3 MiB/s, 78 objects/s recovering 2024-09-13T14:34:33.497 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:33 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:34:33.497 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:33 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:34:33.497 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:33 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:34:33.585 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:33.597 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:33 smithi195 ceph-mon[104262]: pgmap v619: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 1.6 GiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 835 op/s; 2603/76413 objects degraded (3.406%); 11155/76413 objects misplaced (14.598%); 1.3 MiB/s, 78 objects/s recovering 2024-09-13T14:34:33.597 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:33 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:34:33.598 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:33 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:34:33.598 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:33 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:34:33.632 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:33.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:33.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:33.681 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:34:33.734 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:33.770 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:33.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:33.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:33.872 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:33.897 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:33.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:33.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:33.999 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:34.039 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:34.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:34.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:34.141 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:34.185 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:34.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:34.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:34.191 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:34:34.279 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is set 2024-09-13T14:34:34.287 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:34.326 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:34.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:34.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:34.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:34 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 2603/76413 objects degraded (3.406%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:34:34.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:34 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1169814022' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:34:34.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:34 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:34:34.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:34 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 2603/76413 objects degraded (3.406%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:34:34.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:34 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1169814022' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:34:34.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:34 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:34:34.428 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:34.455 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:34.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:34.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:34.557 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:34.589 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:34.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:34.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:34.691 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:34.712 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:34.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:34.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:34.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:34 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 2603/76413 objects degraded (3.406%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:34:34.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:34 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1169814022' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:34:34.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:34 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:34:34.814 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:34:34.854 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:34.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:34.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:34:34.955 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:34.983 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:34.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:34.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:35.084 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:35.129 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:35.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:35.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:35.231 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:34:35.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:35.266 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:35.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:34:35.367 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:35.405 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:35.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:35.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:35.506 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:35.547 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:35.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:35.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:35.648 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:35.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:35 smithi102 ceph-mon[115269]: pgmap v620: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 1.3 GiB data, 5.8 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 862 op/s; 2011/63612 objects degraded (3.161%); 9253/63612 objects misplaced (14.546%); 0 B/s, 65 objects/s recovering 2024-09-13T14:34:35.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:35 smithi102 ceph-mon[115269]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:34:35.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:35 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:34:35.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:35 smithi102 ceph-mon[115269]: osdmap e194: 8 total, 8 up, 5 in 2024-09-13T14:34:35.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:35 smithi102 ceph-mon[110183]: pgmap v620: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 1.3 GiB data, 5.8 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 862 op/s; 2011/63612 objects degraded (3.161%); 9253/63612 objects misplaced (14.546%); 0 B/s, 65 objects/s recovering 2024-09-13T14:34:35.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:35 smithi102 ceph-mon[110183]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:34:35.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:35 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:34:35.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:35 smithi102 ceph-mon[110183]: osdmap e194: 8 total, 8 up, 5 in 2024-09-13T14:34:35.674 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:35.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:35.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:35.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:35 smithi195 ceph-mon[104262]: pgmap v620: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 1.3 GiB data, 5.8 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 862 op/s; 2011/63612 objects degraded (3.161%); 9253/63612 objects misplaced (14.546%); 0 B/s, 65 objects/s recovering 2024-09-13T14:34:35.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:35 smithi195 ceph-mon[104262]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:34:35.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:35 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:34:35.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:35 smithi195 ceph-mon[104262]: osdmap e194: 8 total, 8 up, 5 in 2024-09-13T14:34:35.776 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:35.810 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:35.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:35.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:35.838 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:34:35.912 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:35.963 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:35.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:35.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:36.065 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:36.107 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:36.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:36.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:36.208 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:36.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:36.239 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:36.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:36.340 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:36.366 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:36.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:36.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:36.468 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:36.499 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:36.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:36.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:36.568 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:34:36.569 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:34:36.601 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:36.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:36.647 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:36.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:36.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:34:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:34:36] "GET /metrics HTTP/1.1" 200 36549 "" "Prometheus/2.51.0" 2024-09-13T14:34:36.747 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:36.778 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:36.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:36.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:36.880 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:36.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:36.912 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:36.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:37.004 INFO:teuthology.orchestra.run.smithi102.stderr:no valid command found; 10 closest matches: 2024-09-13T14:34:37.023 INFO:teuthology.orchestra.run.smithi102.stderr:0 2024-09-13T14:34:37.024 INFO:teuthology.orchestra.run.smithi102.stderr:1 2024-09-13T14:34:37.024 INFO:teuthology.orchestra.run.smithi102.stderr:2 2024-09-13T14:34:37.024 INFO:teuthology.orchestra.run.smithi102.stderr:abort 2024-09-13T14:34:37.024 INFO:teuthology.orchestra.run.smithi102.stderr:assert 2024-09-13T14:34:37.024 INFO:teuthology.orchestra.run.smithi102.stderr:bench [] [] [] [] 2024-09-13T14:34:37.024 INFO:teuthology.orchestra.run.smithi102.stderr:bluefs debug_inject_read_zeros 2024-09-13T14:34:37.024 INFO:teuthology.orchestra.run.smithi102.stderr:bluefs files list 2024-09-13T14:34:37.024 INFO:teuthology.orchestra.run.smithi102.stderr:bluefs stats 2024-09-13T14:34:37.025 INFO:teuthology.orchestra.run.smithi102.stderr:bluestore allocator dump block 2024-09-13T14:34:37.025 INFO:teuthology.orchestra.run.smithi102.stderr:Error EINVAL: invalid command 2024-09-13T14:34:37.025 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:37.058 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:37.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:37.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:37.160 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:37.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:37.202 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:37.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:37.247 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:34:37.285 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T14:34:37.303 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:37.334 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:37.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:37.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:37.436 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:37.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:37 smithi102 ceph-mon[110183]: pgmap v622: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 1.3 GiB data, 5.8 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 607 op/s; 2011/63612 objects degraded (3.161%); 9253/63612 objects misplaced (14.546%); 0 B/s, 44 objects/s recovering 2024-09-13T14:34:37.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:37 smithi102 ceph-mon[115269]: pgmap v622: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 1.3 GiB data, 5.8 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 607 op/s; 2011/63612 objects degraded (3.161%); 9253/63612 objects misplaced (14.546%); 0 B/s, 44 objects/s recovering 2024-09-13T14:34:37.468 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:37.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:37.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:37.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:37 smithi195 ceph-mon[104262]: pgmap v622: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 1.3 GiB data, 5.8 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 607 op/s; 2011/63612 objects degraded (3.161%); 9253/63612 objects misplaced (14.546%); 0 B/s, 44 objects/s recovering 2024-09-13T14:34:37.569 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:37.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:37.627 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:37.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:37.665 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:34:37.729 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:37.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:37.772 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:37.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:37.873 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:37.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:37.905 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:37.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:38.006 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:38.033 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:38.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:38.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:38.135 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:38.168 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:38.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:38.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:38.253 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T14:34:38.253 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T14:34:38.253 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.3 'trim stale osdmaps' 2024-09-13T14:34:38.269 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:38.341 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:38.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:38.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:38.442 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:38.482 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:38.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:38.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:38.584 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:38.614 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:38.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:38.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:38.716 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:38.753 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:38.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:38.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:38.855 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:38.883 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:38.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:38.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:38.985 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:39.012 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:39.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:39.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:39.114 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:39.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:39.157 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:39.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:39.258 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:39.291 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:39.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:39.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:39.302 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:34:39.364 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:39 smithi102 ceph-mon[110183]: pgmap v623: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 1.1 GiB data, 5.2 GiB used, 442 GiB / 447 GiB avail; 0 B/s wr, 893 op/s; 1407/51552 objects degraded (2.729%); 7523/51552 objects misplaced (14.593%); 0 B/s, 69 objects/s recovering 2024-09-13T14:34:39.364 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:39 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 1407/51552 objects degraded (2.729%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:34:39.364 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:34:39.364 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:34:39.365 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:39 smithi102 ceph-mon[115269]: pgmap v623: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 1.1 GiB data, 5.2 GiB used, 442 GiB / 447 GiB avail; 0 B/s wr, 893 op/s; 1407/51552 objects degraded (2.729%); 7523/51552 objects misplaced (14.593%); 0 B/s, 69 objects/s recovering 2024-09-13T14:34:39.365 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:39 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 1407/51552 objects degraded (2.729%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:34:39.365 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:34:39.393 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:39.400 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:34:39.400 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:34:39.428 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:39.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:39.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:39.432 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:34:39.432 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:34:39.455 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:34:39.455 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:34:39.511 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:34:39.512 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:34:39.530 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:39.536 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:34:39.537 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:34:39.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:39.559 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:39.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:39.572 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:34:39.572 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:34:39.597 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:34:39.598 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:34:39.605 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:39 smithi195 ceph-mon[104262]: pgmap v623: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 1.1 GiB data, 5.2 GiB used, 442 GiB / 447 GiB avail; 0 B/s wr, 893 op/s; 1407/51552 objects degraded (2.729%); 7523/51552 objects misplaced (14.593%); 0 B/s, 69 objects/s recovering 2024-09-13T14:34:39.605 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:39 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 1407/51552 objects degraded (2.729%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:34:39.661 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:39.691 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:39.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:39.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:39.793 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:39.843 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:39.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:39.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:39.945 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:39.977 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:39.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:39.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:40.079 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:40.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:40.110 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:40.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:40.173 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:34:40.173 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-13T14:34:40.173 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-09-13T14:34:40.173 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-09-13T14:34:40.173 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:34:40.211 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:40.256 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:40.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:40.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:40.358 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:40.388 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:40.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:40.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:40.489 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:34:40.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:40 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1318488090' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:34:40.521 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:40.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:40.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:34:40.599 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:40 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1318488090' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:34:40.600 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:40 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1318488090' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:34:40.623 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:34:40.664 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:40.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:40.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:34:40.766 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:40.816 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:40.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:40.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:40.918 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:40.924 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T14:34:40.944 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:40.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:40.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:41.046 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:34:41.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:41.081 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:41.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:34:41.182 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:41.211 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:41.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:41.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:41.296 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:34:41.312 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:41.340 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:41.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:41.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:41.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:41 smithi102 ceph-mon[110183]: pgmap v624: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 1.0 GiB data, 5.1 GiB used, 442 GiB / 447 GiB avail; 0 B/s wr, 887 op/s; 1407/49773 objects degraded (2.827%); 7221/49773 objects misplaced (14.508%); 0 B/s, 68 objects/s recovering 2024-09-13T14:34:41.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:41 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:34:41.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:41 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:34:41.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:41 smithi102 ceph-mon[115269]: pgmap v624: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 1.0 GiB data, 5.1 GiB used, 442 GiB / 447 GiB avail; 0 B/s wr, 887 op/s; 1407/49773 objects degraded (2.827%); 7221/49773 objects misplaced (14.508%); 0 B/s, 68 objects/s recovering 2024-09-13T14:34:41.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:41 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:34:41.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:41 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:34:41.445 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:41.474 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:41.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:41.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:41.576 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:41.605 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:41 smithi195 ceph-mon[104262]: pgmap v624: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 1.0 GiB data, 5.1 GiB used, 442 GiB / 447 GiB avail; 0 B/s wr, 887 op/s; 1407/49773 objects degraded (2.827%); 7221/49773 objects misplaced (14.508%); 0 B/s, 68 objects/s recovering 2024-09-13T14:34:41.606 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:41 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:34:41.606 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:41 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:34:41.619 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:41.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:41.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:41.721 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:41.762 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:41.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:41.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:41.864 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:41.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:41.893 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:41.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:41.994 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:42.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:42.037 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:42.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:42.138 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:42.166 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:42.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:42.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:42.268 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:42.297 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:42.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:42.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:42.399 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:42.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:42.437 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:42.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:42.538 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:42.561 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:42.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:42.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:42.620 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:42 smithi195 ceph-mon[104262]: pgmap v625: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 1.0 GiB data, 5.1 GiB used, 442 GiB / 447 GiB avail; 0 B/s wr, 887 op/s; 1407/49773 objects degraded (2.827%); 7221/49773 objects misplaced (14.508%); 0 B/s, 68 objects/s recovering 2024-09-13T14:34:42.629 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:34:42.656 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:42 smithi102 ceph-mon[110183]: pgmap v625: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 1.0 GiB data, 5.1 GiB used, 442 GiB / 447 GiB avail; 0 B/s wr, 887 op/s; 1407/49773 objects degraded (2.827%); 7221/49773 objects misplaced (14.508%); 0 B/s, 68 objects/s recovering 2024-09-13T14:34:42.657 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:42 smithi102 ceph-mon[115269]: pgmap v625: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 1.0 GiB data, 5.1 GiB used, 442 GiB / 447 GiB avail; 0 B/s wr, 887 op/s; 1407/49773 objects degraded (2.827%); 7221/49773 objects misplaced (14.508%); 0 B/s, 68 objects/s recovering 2024-09-13T14:34:42.663 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:42.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:42.692 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:42.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:42.794 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:42.817 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:42.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:42.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:42.919 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:34:42.944 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:42.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:42.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:34:43.046 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:43.068 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:43.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:43.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:43.170 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:43.194 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:43.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:43.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:43.200 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:34:43.200 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-09-13T14:34:43.200 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-09-13T14:34:43.200 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-13T14:34:43.200 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.200 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:163497 6.1 6:88767735:::benchmark_data_smithi102_194590_object52873:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-13T14:34:43.200 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:33:43.521085+0000", 2024-09-13T14:34:43.200 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 59.676091778999997, 2024-09-13T14:34:43.200 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5959139229999999, 2024-09-13T14:34:43.200 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:43.201 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-13T14:34:43.201 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:34:43.201 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:34:43.201 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:34:43.201 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 163497 2024-09-13T14:34:43.201 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.201 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:43.201 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.201 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:43.202 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.521085+0000", 2024-09-13T14:34:43.202 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:43.202 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.202 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.202 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:43.202 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.521085+0000", 2024-09-13T14:34:43.202 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:43.202 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.202 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.202 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:43.202 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.521083+0000", 2024-09-13T14:34:43.202 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:34:43.203 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.203 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.203 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:43.203 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.521089+0000", 2024-09-13T14:34:43.203 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8769999999999998e-06 2024-09-13T14:34:43.203 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.203 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.203 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:43.203 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.521098+0000", 2024-09-13T14:34:43.203 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1810000000000007e-06 2024-09-13T14:34:43.203 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.203 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.203 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:43.204 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.521104+0000", 2024-09-13T14:34:43.204 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5600000000000001e-06 2024-09-13T14:34:43.204 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.204 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.204 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.204 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.521144+0000", 2024-09-13T14:34:43.204 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9910000000000002e-05 2024-09-13T14:34:43.204 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.204 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.204 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:34:43.204 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.521253+0000", 2024-09-13T14:34:43.204 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000108928 2024-09-13T14:34:43.205 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.205 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.205 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:34:43.205 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.521411+0000", 2024-09-13T14:34:43.205 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000158295 2024-09-13T14:34:43.205 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.205 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.205 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:34:43.205 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.553236+0000", 2024-09-13T14:34:43.206 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.031824516999999997 2024-09-13T14:34:43.206 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.206 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.206 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.206 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.560906+0000", 2024-09-13T14:34:43.206 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0076698729999999998 2024-09-13T14:34:43.206 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.206 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.206 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:34:43.206 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.560921+0000", 2024-09-13T14:34:43.206 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5485000000000002e-05 2024-09-13T14:34:43.206 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.206 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.207 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.207 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.826507+0000", 2024-09-13T14:34:43.207 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.26558596699999998 2024-09-13T14:34:43.207 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.207 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.207 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:34:43.207 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.826528+0000", 2024-09-13T14:34:43.207 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0537e-05 2024-09-13T14:34:43.207 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.207 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.207 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.207 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.035537+0000", 2024-09-13T14:34:43.207 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2090090600000001 2024-09-13T14:34:43.208 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.208 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.208 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:34:43.208 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.035668+0000", 2024-09-13T14:34:43.208 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013121000000000001 2024-09-13T14:34:43.208 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.208 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.208 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.208 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.116946+0000", 2024-09-13T14:34:43.208 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.081277985999999997 2024-09-13T14:34:43.209 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.209 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.209 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:34:43.209 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.116999+0000", 2024-09-13T14:34:43.209 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3399999999999997e-05 2024-09-13T14:34:43.209 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:43.209 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:43.209 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:43.209 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.209 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.209 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:163502 6.a 6:5c9d0108:::benchmark_data_smithi102_194590_object52878:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e182)", 2024-09-13T14:34:43.210 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:33:43.527092+0000", 2024-09-13T14:34:43.210 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 59.670084693, 2024-09-13T14:34:43.210 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4985434980000001, 2024-09-13T14:34:43.210 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:43.210 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:34:43.210 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:34:43.210 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:34:43.210 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:34:43.210 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 163502 2024-09-13T14:34:43.210 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.210 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:43.211 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.211 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:43.211 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.527092+0000", 2024-09-13T14:34:43.211 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:43.211 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.211 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.211 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:43.211 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.527092+0000", 2024-09-13T14:34:43.211 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:43.211 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.211 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.211 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:43.211 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.527091+0000", 2024-09-13T14:34:43.211 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:34:43.212 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.212 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.212 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:43.212 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.527096+0000", 2024-09-13T14:34:43.212 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.631e-06 2024-09-13T14:34:43.212 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.212 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.213 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:43.213 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.527107+0000", 2024-09-13T14:34:43.213 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0438999999999999e-05 2024-09-13T14:34:43.213 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.213 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.213 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:43.213 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.527113+0000", 2024-09-13T14:34:43.213 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6100000000000002e-06 2024-09-13T14:34:43.213 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.213 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.213 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.214 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.530662+0000", 2024-09-13T14:34:43.214 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0035484599999999998 2024-09-13T14:34:43.214 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.214 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.214 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-09-13T14:34:43.214 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.530669+0000", 2024-09-13T14:34:43.214 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7320000000000001e-06 2024-09-13T14:34:43.214 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.214 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.214 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.214 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.555942+0000", 2024-09-13T14:34:43.214 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.025273917999999999 2024-09-13T14:34:43.214 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.214 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.215 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:34:43.215 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.555959+0000", 2024-09-13T14:34:43.215 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6629e-05 2024-09-13T14:34:43.215 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.215 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.215 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.215 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.829180+0000", 2024-09-13T14:34:43.215 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.27322133500000001 2024-09-13T14:34:43.215 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.215 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.215 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:34:43.215 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.829191+0000", 2024-09-13T14:34:43.215 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0752e-05 2024-09-13T14:34:43.215 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.216 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.216 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.216 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:44.980656+0000", 2024-09-13T14:34:43.216 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1514644220000001 2024-09-13T14:34:43.216 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.216 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.216 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:34:43.217 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:44.984216+0000", 2024-09-13T14:34:43.217 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0035600039999999999 2024-09-13T14:34:43.217 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.217 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.217 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,3,5", 2024-09-13T14:34:43.217 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:44.984752+0000", 2024-09-13T14:34:43.218 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00053684100000000003 2024-09-13T14:34:43.218 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.218 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.218 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:34:43.218 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.000777+0000", 2024-09-13T14:34:43.218 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016024094999999999 2024-09-13T14:34:43.218 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.218 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.218 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:34:43.218 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.007705+0000", 2024-09-13T14:34:43.218 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0069286690000000001 2024-09-13T14:34:43.218 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.218 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.218 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:34:43.219 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.007946+0000", 2024-09-13T14:34:43.219 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000240666 2024-09-13T14:34:43.219 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.219 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.219 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:34:43.219 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.025465+0000", 2024-09-13T14:34:43.219 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017518907 2024-09-13T14:34:43.219 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.219 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.219 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:34:43.219 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.025521+0000", 2024-09-13T14:34:43.219 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.6359e-05 2024-09-13T14:34:43.219 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.219 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.220 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:34:43.220 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.025636+0000", 2024-09-13T14:34:43.220 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000114722 2024-09-13T14:34:43.220 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:43.220 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:43.220 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:43.220 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.220 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.221 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:163507 6.1 6:876d86fa:::benchmark_data_smithi102_194590_object52883:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-09-13T14:34:43.221 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:33:43.540150+0000", 2024-09-13T14:34:43.221 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 59.657027364999998, 2024-09-13T14:34:43.221 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.599987743, 2024-09-13T14:34:43.221 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:43.221 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:34:43.222 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:34:43.222 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:34:43.222 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:34:43.222 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 163507 2024-09-13T14:34:43.222 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.222 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:43.223 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.223 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:43.223 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.540150+0000", 2024-09-13T14:34:43.223 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:43.223 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.223 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.224 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:43.224 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.540150+0000", 2024-09-13T14:34:43.224 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:43.224 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.224 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.224 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:43.225 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.540149+0000", 2024-09-13T14:34:43.225 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:34:43.225 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.225 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.225 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:43.225 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.540153+0000", 2024-09-13T14:34:43.226 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8419999999999998e-06 2024-09-13T14:34:43.226 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.226 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.226 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:43.226 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.540161+0000", 2024-09-13T14:34:43.226 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5900000000000008e-06 2024-09-13T14:34:43.227 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.227 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.227 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:43.227 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.540167+0000", 2024-09-13T14:34:43.227 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8610000000000003e-06 2024-09-13T14:34:43.227 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.228 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.228 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.228 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.541571+0000", 2024-09-13T14:34:43.228 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0014040459999999999 2024-09-13T14:34:43.228 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.228 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.229 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-09-13T14:34:43.229 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.541579+0000", 2024-09-13T14:34:43.229 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6650000000000003e-06 2024-09-13T14:34:43.229 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.229 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.230 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.230 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.561786+0000", 2024-09-13T14:34:43.230 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.020207563000000001 2024-09-13T14:34:43.230 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.230 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.230 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:34:43.231 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.561795+0000", 2024-09-13T14:34:43.231 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2509999999999994e-06 2024-09-13T14:34:43.231 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.231 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.231 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.231 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.827015+0000", 2024-09-13T14:34:43.232 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.26521998800000002 2024-09-13T14:34:43.232 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.232 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.232 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:34:43.232 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.827023+0000", 2024-09-13T14:34:43.232 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8440000000000004e-06 2024-09-13T14:34:43.233 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.233 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.233 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.233 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.117059+0000", 2024-09-13T14:34:43.233 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2900357849999999 2024-09-13T14:34:43.233 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.234 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.234 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:34:43.234 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.117127+0000", 2024-09-13T14:34:43.234 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8052999999999994e-05 2024-09-13T14:34:43.234 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.234 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.235 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,3,5", 2024-09-13T14:34:43.235 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.117535+0000", 2024-09-13T14:34:43.235 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00040791800000000001 2024-09-13T14:34:43.235 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.235 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.236 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:34:43.236 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.129095+0000", 2024-09-13T14:34:43.236 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011559799000000001 2024-09-13T14:34:43.236 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.236 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.236 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:34:43.237 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.132781+0000", 2024-09-13T14:34:43.237 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0036863120000000002 2024-09-13T14:34:43.237 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.237 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.237 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:34:43.237 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.133782+0000", 2024-09-13T14:34:43.238 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001000969 2024-09-13T14:34:43.238 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.238 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.238 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:34:43.238 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.138468+0000", 2024-09-13T14:34:43.238 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0046855890000000004 2024-09-13T14:34:43.239 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.239 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.239 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:34:43.239 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.138519+0000", 2024-09-13T14:34:43.239 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1172999999999999e-05 2024-09-13T14:34:43.239 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.240 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.240 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:34:43.240 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.140137+0000", 2024-09-13T14:34:43.240 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0016184019999999999 2024-09-13T14:34:43.240 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:43.240 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:43.241 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:43.241 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.241 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.241 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:173911 6.1 6:8ae3e05e:::benchmark_data_smithi102_194590_object63287:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-13T14:34:43.241 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:33:56.868215+0000", 2024-09-13T14:34:43.242 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 46.328961681999999, 2024-09-13T14:34:43.242 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.043288818, 2024-09-13T14:34:43.242 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:43.242 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-13T14:34:43.243 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:34:43.243 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:34:43.243 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:34:43.243 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 173911 2024-09-13T14:34:43.243 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.244 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:43.244 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.244 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:43.244 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.868215+0000", 2024-09-13T14:34:43.244 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:43.244 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.245 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.245 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:43.245 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.868215+0000", 2024-09-13T14:34:43.245 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:43.245 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.245 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.246 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:43.246 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.868214+0000", 2024-09-13T14:34:43.246 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:34:43.246 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.246 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.247 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:43.247 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.868219+0000", 2024-09-13T14:34:43.247 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2839999999999997e-06 2024-09-13T14:34:43.247 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.247 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.247 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:43.248 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.868229+0000", 2024-09-13T14:34:43.248 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.5650000000000007e-06 2024-09-13T14:34:43.248 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.248 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.248 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:43.248 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.868235+0000", 2024-09-13T14:34:43.249 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4069999999999998e-06 2024-09-13T14:34:43.249 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.249 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.249 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.249 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.871197+0000", 2024-09-13T14:34:43.249 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0029621019999999999 2024-09-13T14:34:43.249 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.249 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.250 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for degraded object", 2024-09-13T14:34:43.250 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.872006+0000", 2024-09-13T14:34:43.250 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00080844599999999999 2024-09-13T14:34:43.250 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.250 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.250 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.250 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.913110+0000", 2024-09-13T14:34:43.250 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.041104373999999999 2024-09-13T14:34:43.251 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.251 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.251 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:34:43.251 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.913201+0000", 2024-09-13T14:34:43.251 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.0587999999999994e-05 2024-09-13T14:34:43.251 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.251 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.252 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,3,5", 2024-09-13T14:34:43.252 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.913323+0000", 2024-09-13T14:34:43.252 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012251699999999999 2024-09-13T14:34:43.252 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.252 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.252 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.252 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.925156+0000", 2024-09-13T14:34:43.252 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011832581 2024-09-13T14:34:43.253 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.253 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.253 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:34:43.253 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.925163+0000", 2024-09-13T14:34:43.253 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3459999999999996e-06 2024-09-13T14:34:43.253 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.253 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.254 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.254 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:57.911490+0000", 2024-09-13T14:34:43.254 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.98632671500000002 2024-09-13T14:34:43.254 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.254 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.254 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:34:43.254 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:57.911504+0000", 2024-09-13T14:34:43.254 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4319e-05 2024-09-13T14:34:43.254 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:43.254 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:43.254 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:43.254 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.255 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.255 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:173920 6.1 6:8052e2ea:::benchmark_data_smithi102_194590_object63296:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-13T14:34:43.255 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:33:56.876648+0000", 2024-09-13T14:34:43.255 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 46.320528613999997, 2024-09-13T14:34:43.255 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.034895049, 2024-09-13T14:34:43.255 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:43.255 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-13T14:34:43.255 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:34:43.255 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:34:43.255 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:34:43.255 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 173920 2024-09-13T14:34:43.255 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.256 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:43.256 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.256 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:43.256 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.876648+0000", 2024-09-13T14:34:43.256 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:43.256 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.256 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.256 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:43.256 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.876648+0000", 2024-09-13T14:34:43.256 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:43.256 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.256 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.256 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:43.257 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.876647+0000", 2024-09-13T14:34:43.257 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:34:43.257 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.257 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.257 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:43.257 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.876654+0000", 2024-09-13T14:34:43.257 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9419999999999998e-06 2024-09-13T14:34:43.257 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.257 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.257 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:43.257 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.876667+0000", 2024-09-13T14:34:43.257 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3429e-05 2024-09-13T14:34:43.258 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.258 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.258 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:43.258 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.876674+0000", 2024-09-13T14:34:43.258 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8889999999999996e-06 2024-09-13T14:34:43.258 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.259 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.259 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.259 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.887171+0000", 2024-09-13T14:34:43.259 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010497213 2024-09-13T14:34:43.259 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.259 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.260 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for degraded object", 2024-09-13T14:34:43.260 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.887229+0000", 2024-09-13T14:34:43.260 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7796e-05 2024-09-13T14:34:43.260 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.260 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.260 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.260 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.925194+0000", 2024-09-13T14:34:43.261 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.037964549 2024-09-13T14:34:43.261 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.261 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.261 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:34:43.261 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.925200+0000", 2024-09-13T14:34:43.261 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2759999999999997e-06 2024-09-13T14:34:43.261 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.262 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.262 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.262 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:57.911533+0000", 2024-09-13T14:34:43.262 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.98633309300000005 2024-09-13T14:34:43.262 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.262 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.262 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:34:43.262 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:57.911543+0000", 2024-09-13T14:34:43.262 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.047e-05 2024-09-13T14:34:43.263 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:43.263 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:43.263 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:43.263 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.263 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.263 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:173924 6.1 6:807d79c9:::benchmark_data_smithi102_194590_object63300:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-13T14:34:43.263 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:33:56.880707+0000", 2024-09-13T14:34:43.263 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 46.316469990000002, 2024-09-13T14:34:43.263 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0307343259999999, 2024-09-13T14:34:43.263 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:43.263 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-13T14:34:43.263 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:34:43.263 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:34:43.263 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:34:43.264 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 173924 2024-09-13T14:34:43.264 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.264 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:43.264 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.264 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:43.264 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.880707+0000", 2024-09-13T14:34:43.264 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:43.264 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.264 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.264 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:43.264 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.880707+0000", 2024-09-13T14:34:43.264 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:43.264 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.265 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.265 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:43.265 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.880705+0000", 2024-09-13T14:34:43.265 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:34:43.265 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.265 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.265 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:43.265 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.880712+0000", 2024-09-13T14:34:43.265 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8229999999999997e-06 2024-09-13T14:34:43.265 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.265 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.265 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:43.265 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.880745+0000", 2024-09-13T14:34:43.266 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2431000000000002e-05 2024-09-13T14:34:43.266 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.266 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.266 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:43.266 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.880755+0000", 2024-09-13T14:34:43.266 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0727000000000001e-05 2024-09-13T14:34:43.266 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.266 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.266 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.266 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.887252+0000", 2024-09-13T14:34:43.266 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0064968910000000003 2024-09-13T14:34:43.266 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.266 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.266 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for degraded object", 2024-09-13T14:34:43.267 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.887657+0000", 2024-09-13T14:34:43.267 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00040430700000000002 2024-09-13T14:34:43.267 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.267 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.267 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.267 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.905642+0000", 2024-09-13T14:34:43.267 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017985199 2024-09-13T14:34:43.267 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.267 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.267 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:34:43.267 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.907899+0000", 2024-09-13T14:34:43.267 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0022574259999999999 2024-09-13T14:34:43.267 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.268 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.268 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,3,5", 2024-09-13T14:34:43.268 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.908056+0000", 2024-09-13T14:34:43.268 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000156782 2024-09-13T14:34:43.268 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.268 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.268 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:34:43.268 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.916247+0000", 2024-09-13T14:34:43.268 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0081914540000000008 2024-09-13T14:34:43.268 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.268 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.268 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.268 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.924231+0000", 2024-09-13T14:34:43.268 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0079832029999999995 2024-09-13T14:34:43.269 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.269 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.269 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:34:43.269 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.924243+0000", 2024-09-13T14:34:43.269 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2404e-05 2024-09-13T14:34:43.269 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.269 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.269 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.269 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:57.911425+0000", 2024-09-13T14:34:43.269 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.98718179900000003 2024-09-13T14:34:43.269 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.269 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.269 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:34:43.270 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:57.911441+0000", 2024-09-13T14:34:43.270 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6492000000000001e-05 2024-09-13T14:34:43.270 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:43.270 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:43.270 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:43.270 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.270 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.270 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:173945 6.a 6.100ff67a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-09-13T14:34:43.270 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:33:56.913507+0000", 2024-09-13T14:34:43.270 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 46.283670203, 2024-09-13T14:34:43.270 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.99799647999999996, 2024-09-13T14:34:43.270 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:43.270 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-13T14:34:43.270 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:34:43.271 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:34:43.271 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:34:43.271 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 173945 2024-09-13T14:34:43.271 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.271 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:43.271 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.271 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:43.271 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.913507+0000", 2024-09-13T14:34:43.271 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:43.271 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.271 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.271 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:43.271 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.913507+0000", 2024-09-13T14:34:43.272 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:43.272 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.272 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.272 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:43.272 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.913505+0000", 2024-09-13T14:34:43.272 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:34:43.272 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.272 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.272 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:43.272 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.913512+0000", 2024-09-13T14:34:43.272 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.0190000000000001e-06 2024-09-13T14:34:43.272 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.272 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.272 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:43.273 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.913524+0000", 2024-09-13T14:34:43.273 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.217e-05 2024-09-13T14:34:43.273 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.273 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.273 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:43.273 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.913532+0000", 2024-09-13T14:34:43.273 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5009999999999998e-06 2024-09-13T14:34:43.273 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.273 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.273 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.273 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.922077+0000", 2024-09-13T14:34:43.273 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0085455900000000005 2024-09-13T14:34:43.273 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.274 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.274 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:34:43.274 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.922086+0000", 2024-09-13T14:34:43.274 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1770000000000004e-06 2024-09-13T14:34:43.274 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.274 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.274 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.274 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:57.911494+0000", 2024-09-13T14:34:43.274 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.98940765399999997 2024-09-13T14:34:43.274 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.274 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.274 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:34:43.274 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:57.911503+0000", 2024-09-13T14:34:43.274 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1749999999999994e-06 2024-09-13T14:34:43.275 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:43.275 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:43.275 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:43.275 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.275 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.275 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:173946 6.a 6.f2a790ba (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-13T14:34:43.275 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:33:56.915696+0000", 2024-09-13T14:34:43.275 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 46.281481118000002, 2024-09-13T14:34:43.275 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.99583630400000001, 2024-09-13T14:34:43.275 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:43.275 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-13T14:34:43.275 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:34:43.275 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:34:43.275 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:34:43.276 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 173946 2024-09-13T14:34:43.276 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.276 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:43.276 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.276 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:43.276 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.915696+0000", 2024-09-13T14:34:43.276 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:43.276 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.276 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.276 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:43.276 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.915696+0000", 2024-09-13T14:34:43.276 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:43.276 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.277 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.277 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:43.277 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.915695+0000", 2024-09-13T14:34:43.277 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:34:43.277 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.277 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.277 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:43.277 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.915700+0000", 2024-09-13T14:34:43.277 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0799999999999996e-06 2024-09-13T14:34:43.277 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.277 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.277 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:43.277 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.915710+0000", 2024-09-13T14:34:43.277 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0628000000000001e-05 2024-09-13T14:34:43.278 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.278 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.278 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:43.278 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.915717+0000", 2024-09-13T14:34:43.278 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.596e-06 2024-09-13T14:34:43.278 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.278 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.278 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.278 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.922106+0000", 2024-09-13T14:34:43.278 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0063887880000000003 2024-09-13T14:34:43.278 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.278 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.278 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:34:43.278 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.922112+0000", 2024-09-13T14:34:43.279 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7830000000000004e-06 2024-09-13T14:34:43.279 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.279 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.279 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.279 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:57.911523+0000", 2024-09-13T14:34:43.279 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.98941188899999999 2024-09-13T14:34:43.279 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.279 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.279 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:34:43.279 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:57.911532+0000", 2024-09-13T14:34:43.279 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7900000000000005e-06 2024-09-13T14:34:43.279 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:43.279 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:43.279 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:43.280 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.280 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.280 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:173949 6.a 6.2fdda32a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-13T14:34:43.280 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:33:56.919393+0000", 2024-09-13T14:34:43.280 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 46.277783626999998, 2024-09-13T14:34:43.280 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.99216720899999999, 2024-09-13T14:34:43.280 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:43.280 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-13T14:34:43.280 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:34:43.280 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:34:43.280 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:34:43.280 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 173949 2024-09-13T14:34:43.280 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.280 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:43.281 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.281 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:43.281 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.919393+0000", 2024-09-13T14:34:43.281 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:43.281 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.281 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.281 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:43.281 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.919393+0000", 2024-09-13T14:34:43.281 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:43.281 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.281 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.281 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:43.281 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.919392+0000", 2024-09-13T14:34:43.282 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:34:43.282 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.282 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.282 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:43.282 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.919398+0000", 2024-09-13T14:34:43.282 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2639999999999997e-06 2024-09-13T14:34:43.282 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.282 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.282 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:43.282 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.919409+0000", 2024-09-13T14:34:43.282 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0686000000000001e-05 2024-09-13T14:34:43.282 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.282 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.282 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:43.283 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.919416+0000", 2024-09-13T14:34:43.283 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.878e-06 2024-09-13T14:34:43.283 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.283 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.283 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.283 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.922129+0000", 2024-09-13T14:34:43.283 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002713147 2024-09-13T14:34:43.283 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.283 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.283 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:34:43.283 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.922134+0000", 2024-09-13T14:34:43.283 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.451e-06 2024-09-13T14:34:43.283 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.283 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.284 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.284 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:57.911552+0000", 2024-09-13T14:34:43.284 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.98941794900000002 2024-09-13T14:34:43.284 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.284 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.284 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:34:43.284 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:57.911561+0000", 2024-09-13T14:34:43.284 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2859999999999999e-06 2024-09-13T14:34:43.284 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:43.284 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:43.284 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:43.284 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.284 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.284 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:173968 6.1 6.d7be7821 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-13T14:34:43.285 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:33:56.937159+0000", 2024-09-13T14:34:43.285 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 46.260017912999999, 2024-09-13T14:34:43.285 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.97442019000000002, 2024-09-13T14:34:43.285 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:43.285 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-13T14:34:43.285 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:34:43.285 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:34:43.285 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:34:43.285 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 173968 2024-09-13T14:34:43.285 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.285 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:43.285 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.285 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:43.285 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.937159+0000", 2024-09-13T14:34:43.286 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:43.286 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.286 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.286 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:43.286 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.937159+0000", 2024-09-13T14:34:43.286 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:43.286 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.286 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.286 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:43.286 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.937157+0000", 2024-09-13T14:34:43.286 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:34:43.286 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.286 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.287 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:43.287 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.937163+0000", 2024-09-13T14:34:43.287 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.511e-06 2024-09-13T14:34:43.287 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.287 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.287 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:43.287 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.937176+0000", 2024-09-13T14:34:43.287 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2899e-05 2024-09-13T14:34:43.287 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.287 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.287 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:43.287 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.937183+0000", 2024-09-13T14:34:43.287 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7209999999999997e-06 2024-09-13T14:34:43.287 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.288 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.288 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.288 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.937260+0000", 2024-09-13T14:34:43.288 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.737e-05 2024-09-13T14:34:43.288 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.288 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.288 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:34:43.288 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.937270+0000", 2024-09-13T14:34:43.288 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.3489999999999998e-06 2024-09-13T14:34:43.288 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.288 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.288 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.288 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:57.911570+0000", 2024-09-13T14:34:43.289 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.97430051900000003 2024-09-13T14:34:43.289 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.289 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.289 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:34:43.289 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:57.911579+0000", 2024-09-13T14:34:43.289 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.9539999999999993e-06 2024-09-13T14:34:43.289 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:43.289 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:43.289 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:43.289 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.289 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.289 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:173979 6.a 6.105b7b1a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-13T14:34:43.289 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:33:56.948211+0000", 2024-09-13T14:34:43.289 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 46.248966447999997, 2024-09-13T14:34:43.290 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.96337822500000003, 2024-09-13T14:34:43.290 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:43.290 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-13T14:34:43.290 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:34:43.290 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:34:43.290 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:34:43.290 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 173979 2024-09-13T14:34:43.290 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.290 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:43.290 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.290 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:43.290 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.948211+0000", 2024-09-13T14:34:43.290 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:43.290 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.291 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.291 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:43.291 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.948211+0000", 2024-09-13T14:34:43.291 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:43.291 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.291 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.291 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:43.291 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.948209+0000", 2024-09-13T14:34:43.291 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:34:43.291 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.291 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.291 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:43.291 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.948218+0000", 2024-09-13T14:34:43.291 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.6160000000000002e-06 2024-09-13T14:34:43.292 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.292 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.292 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:43.292 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.948231+0000", 2024-09-13T14:34:43.292 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3667e-05 2024-09-13T14:34:43.292 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.292 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.292 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:43.292 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.948237+0000", 2024-09-13T14:34:43.292 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.48e-06 2024-09-13T14:34:43.292 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.292 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.292 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.293 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.948283+0000", 2024-09-13T14:34:43.293 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6369000000000003e-05 2024-09-13T14:34:43.293 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.293 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.293 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:34:43.293 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.948293+0000", 2024-09-13T14:34:43.293 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4599999999999992e-06 2024-09-13T14:34:43.293 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.293 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.293 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.293 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:57.911580+0000", 2024-09-13T14:34:43.293 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.96328718400000002 2024-09-13T14:34:43.293 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.293 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.293 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:34:43.294 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:57.911589+0000", 2024-09-13T14:34:43.294 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.054e-06 2024-09-13T14:34:43.294 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:43.294 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:43.294 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:43.294 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.294 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.294 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:173982 6.1 6.70adeab1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-13T14:34:43.294 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:33:56.950846+0000", 2024-09-13T14:34:43.294 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 46.246330598, 2024-09-13T14:34:43.294 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.96076499199999998, 2024-09-13T14:34:43.294 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:43.294 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-13T14:34:43.295 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:34:43.295 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:34:43.295 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:34:43.295 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 173982 2024-09-13T14:34:43.295 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.295 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:43.295 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.295 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:43.295 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.950846+0000", 2024-09-13T14:34:43.295 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:43.295 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.295 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.295 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:43.296 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:43.297 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.950846+0000", 2024-09-13T14:34:43.297 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:43.297 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.297 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.297 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:43.297 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.950845+0000", 2024-09-13T14:34:43.297 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:34:43.297 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.297 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.298 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:43.298 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.950852+0000", 2024-09-13T14:34:43.298 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3579999999999997e-06 2024-09-13T14:34:43.298 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.298 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.298 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:43.298 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.950865+0000", 2024-09-13T14:34:43.298 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2401999999999999e-05 2024-09-13T14:34:43.298 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.298 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.298 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:43.298 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.950872+0000", 2024-09-13T14:34:43.298 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5410000000000003e-06 2024-09-13T14:34:43.298 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.299 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.299 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.299 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.951196+0000", 2024-09-13T14:34:43.299 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00032409099999999998 2024-09-13T14:34:43.299 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.299 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.299 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:34:43.299 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.951206+0000", 2024-09-13T14:34:43.299 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0190999999999999e-05 2024-09-13T14:34:43.299 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.299 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.299 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.299 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:57.911604+0000", 2024-09-13T14:34:43.299 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.96039739099999999 2024-09-13T14:34:43.300 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.300 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.300 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:34:43.300 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:57.911611+0000", 2024-09-13T14:34:43.300 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5630000000000003e-06 2024-09-13T14:34:43.300 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:43.300 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:43.300 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:43.300 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.300 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.300 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:173989 6.1 6.d2c3881 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-13T14:34:43.300 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:33:56.960459+0000", 2024-09-13T14:34:43.300 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 46.236717714000001, 2024-09-13T14:34:43.301 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.95118071000000004, 2024-09-13T14:34:43.301 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:43.301 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-13T14:34:43.301 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:34:43.301 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:34:43.301 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:34:43.301 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 173989 2024-09-13T14:34:43.301 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.301 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:43.301 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.301 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:43.301 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.960459+0000", 2024-09-13T14:34:43.301 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:43.301 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.302 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.302 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:43.302 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.960459+0000", 2024-09-13T14:34:43.302 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:43.302 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.302 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.302 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:43.302 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.960458+0000", 2024-09-13T14:34:43.302 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:34:43.302 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.302 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.302 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:43.302 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.960465+0000", 2024-09-13T14:34:43.302 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5079999999999999e-06 2024-09-13T14:34:43.303 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.303 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.303 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:43.303 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.960479+0000", 2024-09-13T14:34:43.303 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3422e-05 2024-09-13T14:34:43.303 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.303 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.303 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:43.303 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.960486+0000", 2024-09-13T14:34:43.303 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5970000000000003e-06 2024-09-13T14:34:43.303 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.303 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.303 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.303 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.960550+0000", 2024-09-13T14:34:43.304 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3455999999999997e-05 2024-09-13T14:34:43.304 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.304 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.304 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:34:43.304 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.960559+0000", 2024-09-13T14:34:43.304 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.516e-06 2024-09-13T14:34:43.304 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.304 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.304 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.304 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:57.911632+0000", 2024-09-13T14:34:43.304 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.95107317999999996 2024-09-13T14:34:43.304 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.304 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.305 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:34:43.305 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:57.911640+0000", 2024-09-13T14:34:43.305 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6000000000000001e-06 2024-09-13T14:34:43.305 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:43.305 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:43.305 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:43.305 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.305 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.305 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:173993 6.a 6.26a446ea (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-13T14:34:43.305 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:33:56.964633+0000", 2024-09-13T14:34:43.305 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 46.232544476999998, 2024-09-13T14:34:43.305 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.94698310399999996, 2024-09-13T14:34:43.305 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:43.305 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-13T14:34:43.306 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:34:43.306 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:34:43.306 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:34:43.306 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 173993 2024-09-13T14:34:43.306 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.306 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:43.306 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.306 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:43.306 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.964633+0000", 2024-09-13T14:34:43.306 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:43.306 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.306 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.306 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:43.306 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.964633+0000", 2024-09-13T14:34:43.307 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:43.307 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.307 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.307 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:43.307 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.964631+0000", 2024-09-13T14:34:43.307 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:34:43.307 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.307 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.307 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:43.307 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.964636+0000", 2024-09-13T14:34:43.307 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6269999999999999e-06 2024-09-13T14:34:43.307 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.307 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.307 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:43.308 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.964645+0000", 2024-09-13T14:34:43.308 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.132e-06 2024-09-13T14:34:43.308 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.308 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.308 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:43.308 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.964651+0000", 2024-09-13T14:34:43.308 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5470000000000004e-06 2024-09-13T14:34:43.308 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.308 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.308 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.308 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.964711+0000", 2024-09-13T14:34:43.308 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.9960999999999997e-05 2024-09-13T14:34:43.308 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.309 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.309 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:34:43.309 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.964720+0000", 2024-09-13T14:34:43.309 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.2790000000000004e-06 2024-09-13T14:34:43.309 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.309 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.309 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.309 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:57.911607+0000", 2024-09-13T14:34:43.309 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.94688712100000005 2024-09-13T14:34:43.309 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.309 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.309 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:34:43.309 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:57.911616+0000", 2024-09-13T14:34:43.309 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.6610000000000006e-06 2024-09-13T14:34:43.310 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:43.310 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:43.310 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:43.310 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.310 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.310 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:173998 6.a 6.1868d37a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-13T14:34:43.310 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:33:56.969588+0000", 2024-09-13T14:34:43.310 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 46.227589017, 2024-09-13T14:34:43.310 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.94205431799999995, 2024-09-13T14:34:43.310 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:43.310 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-13T14:34:43.310 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:34:43.310 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:34:43.310 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:34:43.311 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 173998 2024-09-13T14:34:43.311 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.311 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:43.311 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.311 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:43.311 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.969588+0000", 2024-09-13T14:34:43.311 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:43.311 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.311 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.311 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:43.311 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.969588+0000", 2024-09-13T14:34:43.311 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:43.311 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.312 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.312 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:43.312 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.969586+0000", 2024-09-13T14:34:43.312 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:34:43.312 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.312 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.312 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:43.312 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.969593+0000", 2024-09-13T14:34:43.312 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.55e-06 2024-09-13T14:34:43.312 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.312 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.312 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:43.312 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.969605+0000", 2024-09-13T14:34:43.312 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1771e-05 2024-09-13T14:34:43.313 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.313 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.313 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:43.313 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.969611+0000", 2024-09-13T14:34:43.313 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7499999999999997e-06 2024-09-13T14:34:43.313 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.313 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.313 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.313 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.969661+0000", 2024-09-13T14:34:43.313 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9518000000000003e-05 2024-09-13T14:34:43.313 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.313 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.313 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:34:43.314 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.969671+0000", 2024-09-13T14:34:43.314 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0002e-05 2024-09-13T14:34:43.314 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.314 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.314 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.314 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:57.911634+0000", 2024-09-13T14:34:43.314 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.941963412 2024-09-13T14:34:43.314 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.314 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.314 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:34:43.314 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:57.911642+0000", 2024-09-13T14:34:43.314 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9179999999999994e-06 2024-09-13T14:34:43.314 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:43.314 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:43.315 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:43.315 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.315 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.315 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:174000 6.a 6.23f9d59a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-13T14:34:43.315 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:33:56.974756+0000", 2024-09-13T14:34:43.315 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 46.222420646000003, 2024-09-13T14:34:43.315 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.93710116200000004, 2024-09-13T14:34:43.315 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:43.315 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-13T14:34:43.315 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:34:43.315 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:34:43.315 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:34:43.315 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 174000 2024-09-13T14:34:43.315 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.315 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:43.316 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.316 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:43.316 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.974756+0000", 2024-09-13T14:34:43.316 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:43.316 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.316 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.316 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:43.316 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.974756+0000", 2024-09-13T14:34:43.316 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:43.316 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.316 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.316 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:43.316 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.974755+0000", 2024-09-13T14:34:43.317 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:34:43.317 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.317 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.317 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:43.317 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.974762+0000", 2024-09-13T14:34:43.317 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6290000000000004e-06 2024-09-13T14:34:43.317 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.317 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.317 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:43.317 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.974771+0000", 2024-09-13T14:34:43.317 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.8290000000000003e-06 2024-09-13T14:34:43.317 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.317 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.317 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:43.318 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.974777+0000", 2024-09-13T14:34:43.318 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.383e-06 2024-09-13T14:34:43.318 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.318 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.318 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.318 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.974812+0000", 2024-09-13T14:34:43.318 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5465e-05 2024-09-13T14:34:43.318 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.318 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.318 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:34:43.318 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.974823+0000", 2024-09-13T14:34:43.318 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0482999999999999e-05 2024-09-13T14:34:43.318 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.318 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.318 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.319 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:57.911661+0000", 2024-09-13T14:34:43.319 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.936838646 2024-09-13T14:34:43.319 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.319 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.319 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:34:43.319 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:57.911858+0000", 2024-09-13T14:34:43.319 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000196095 2024-09-13T14:34:43.319 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:43.319 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:43.319 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:43.319 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.319 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.319 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:174013 6.1 6.60fd2b91 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-13T14:34:43.319 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:33:56.995695+0000", 2024-09-13T14:34:43.320 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 46.201481655000002, 2024-09-13T14:34:43.320 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.91628301199999995, 2024-09-13T14:34:43.320 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:43.320 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-13T14:34:43.320 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:34:43.320 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:34:43.320 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:34:43.320 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 174013 2024-09-13T14:34:43.320 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.320 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:43.320 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.320 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:43.320 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.995695+0000", 2024-09-13T14:34:43.321 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:43.321 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.321 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.321 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:43.321 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.995695+0000", 2024-09-13T14:34:43.321 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:43.321 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.321 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.321 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:43.321 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.995694+0000", 2024-09-13T14:34:43.321 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:34:43.321 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.321 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.321 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:43.322 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.995700+0000", 2024-09-13T14:34:43.322 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7000000000000002e-06 2024-09-13T14:34:43.322 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.322 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.322 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:43.322 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.995712+0000", 2024-09-13T14:34:43.322 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.133e-05 2024-09-13T14:34:43.322 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.322 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.322 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:43.322 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.995717+0000", 2024-09-13T14:34:43.322 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3199999999999999e-06 2024-09-13T14:34:43.322 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.322 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.323 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.323 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.995772+0000", 2024-09-13T14:34:43.323 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4826e-05 2024-09-13T14:34:43.323 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.323 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.323 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:34:43.323 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:56.995782+0000", 2024-09-13T14:34:43.323 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0162e-05 2024-09-13T14:34:43.323 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.323 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.323 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.323 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:57.911664+0000", 2024-09-13T14:34:43.323 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.91588199400000003 2024-09-13T14:34:43.323 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.324 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.324 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:34:43.324 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:57.911978+0000", 2024-09-13T14:34:43.324 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00031440800000000002 2024-09-13T14:34:43.324 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:43.324 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:43.324 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:43.324 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.324 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.324 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:174019 6.a 6.f2c71c1a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-13T14:34:43.324 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:33:57.010589+0000", 2024-09-13T14:34:43.324 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 46.186587676999999, 2024-09-13T14:34:43.324 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.90130098800000003, 2024-09-13T14:34:43.324 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:43.325 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-13T14:34:43.325 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:34:43.325 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:34:43.325 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:34:43.325 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 174019 2024-09-13T14:34:43.325 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.325 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:43.325 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.325 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:43.325 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:57.010589+0000", 2024-09-13T14:34:43.325 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:43.325 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.325 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.325 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:43.326 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:57.010589+0000", 2024-09-13T14:34:43.326 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:43.326 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.326 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.326 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:43.326 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:57.010588+0000", 2024-09-13T14:34:43.326 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:34:43.326 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.326 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.326 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:43.326 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:57.010595+0000", 2024-09-13T14:34:43.326 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5880000000000003e-06 2024-09-13T14:34:43.326 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.326 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.327 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:43.327 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:57.010607+0000", 2024-09-13T14:34:43.327 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2013e-05 2024-09-13T14:34:43.327 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.327 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.327 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:43.327 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:57.010612+0000", 2024-09-13T14:34:43.327 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7799999999999997e-06 2024-09-13T14:34:43.327 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.327 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.327 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.327 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:57.010669+0000", 2024-09-13T14:34:43.327 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.6141999999999999e-05 2024-09-13T14:34:43.328 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.328 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.328 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:34:43.328 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:57.010679+0000", 2024-09-13T14:34:43.328 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0705e-05 2024-09-13T14:34:43.328 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.328 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.328 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.328 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:57.911882+0000", 2024-09-13T14:34:43.328 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.90120245499999996 2024-09-13T14:34:43.328 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.328 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.328 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:34:43.328 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:57.911890+0000", 2024-09-13T14:34:43.329 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.6039999999999993e-06 2024-09-13T14:34:43.329 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:43.329 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:43.329 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:43.329 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.329 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.329 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:174024 6.a 6.93a36e3a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-13T14:34:43.329 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:33:57.034790+0000", 2024-09-13T14:34:43.329 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 46.162387049000003, 2024-09-13T14:34:43.329 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.87713363600000005, 2024-09-13T14:34:43.329 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:43.329 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-13T14:34:43.329 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:34:43.329 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:34:43.330 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:34:43.330 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 174024 2024-09-13T14:34:43.330 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.330 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:43.330 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.330 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:43.330 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:57.034790+0000", 2024-09-13T14:34:43.330 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:43.330 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.330 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.330 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:43.330 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:57.034790+0000", 2024-09-13T14:34:43.330 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:43.330 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.331 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.331 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:43.331 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:57.034788+0000", 2024-09-13T14:34:43.331 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-09-13T14:34:43.331 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.331 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.331 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:43.331 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:57.034800+0000", 2024-09-13T14:34:43.331 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.255e-05 2024-09-13T14:34:43.331 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.331 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.331 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:43.331 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:57.034829+0000", 2024-09-13T14:34:43.331 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9359000000000001e-05 2024-09-13T14:34:43.332 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.332 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.332 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:43.332 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:57.034843+0000", 2024-09-13T14:34:43.332 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.366e-05 2024-09-13T14:34:43.332 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.332 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.332 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.332 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:57.034921+0000", 2024-09-13T14:34:43.332 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8387999999999996e-05 2024-09-13T14:34:43.332 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.332 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.332 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:34:43.333 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:57.034934+0000", 2024-09-13T14:34:43.333 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2709e-05 2024-09-13T14:34:43.333 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.333 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.333 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.333 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:57.911908+0000", 2024-09-13T14:34:43.333 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.87697388600000004 2024-09-13T14:34:43.333 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.333 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.333 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:34:43.333 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:57.911924+0000", 2024-09-13T14:34:43.333 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5520999999999999e-05 2024-09-13T14:34:43.333 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:43.333 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:43.334 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:43.334 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.334 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.334 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35037.0:198024 6.3 e193/161 6:c7640c5d:::benchmark_data_smithi102_194590_object87400:head v 193'12238, mlcod=193'12234)", 2024-09-13T14:34:43.334 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:34:29.686364+0000", 2024-09-13T14:34:43.334 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 13.510813356, 2024-09-13T14:34:43.334 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.22897640999999999, 2024-09-13T14:34:43.334 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:43.334 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:34:43.334 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:43.334 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.334 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:43.334 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.686364+0000", 2024-09-13T14:34:43.334 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:43.335 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.335 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.335 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:43.335 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.686364+0000", 2024-09-13T14:34:43.335 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:43.335 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.335 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.335 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:43.335 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.686363+0000", 2024-09-13T14:34:43.335 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967296 2024-09-13T14:34:43.335 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.335 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.335 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:43.335 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.686367+0000", 2024-09-13T14:34:43.335 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.1319999999999998e-06 2024-09-13T14:34:43.336 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.336 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.336 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:43.336 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.686370+0000", 2024-09-13T14:34:43.336 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2949999999999998e-06 2024-09-13T14:34:43.336 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.336 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.336 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:43.336 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.686372+0000", 2024-09-13T14:34:43.336 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3360000000000002e-06 2024-09-13T14:34:43.336 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.336 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.336 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:43.336 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.900035+0000", 2024-09-13T14:34:43.337 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.213663039 2024-09-13T14:34:43.337 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.337 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.337 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:34:43.337 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.900054+0000", 2024-09-13T14:34:43.337 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9233e-05 2024-09-13T14:34:43.337 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.337 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.337 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:34:43.337 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.915302+0000", 2024-09-13T14:34:43.337 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.015247686999999999 2024-09-13T14:34:43.337 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:43.337 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:43.337 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:34:43.338 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.915340+0000", 2024-09-13T14:34:43.338 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7898000000000002e-05 2024-09-13T14:34:43.338 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:43.338 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:43.338 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:43.338 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:43.338 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:43.338 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:34:43.364 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:43.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:43.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:43.466 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:43.490 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:43.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:43.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:43.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:43 smithi195 ceph-mon[104262]: osdmap e195: 8 total, 8 up, 5 in 2024-09-13T14:34:43.591 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:34:43.614 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:43.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:43.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:34:43.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:43 smithi102 ceph-mon[110183]: osdmap e195: 8 total, 8 up, 5 in 2024-09-13T14:34:43.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:43 smithi102 ceph-mon[115269]: osdmap e195: 8 total, 8 up, 5 in 2024-09-13T14:34:43.716 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:34:43.738 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:43.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:43.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:34:43.840 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:43.875 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:43.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:43.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:43.977 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:44.006 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:44.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:44.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:44.075 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T14:34:44.109 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:44.133 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:44.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:44.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:44.235 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:44.258 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:44.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:44.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:44.319 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:34:44.360 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:44.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:44.382 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:44.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:44.483 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:44.505 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:44.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:44.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:44.577 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:44 smithi195 ceph-mon[104262]: osdmap e196: 8 total, 8 up, 5 in 2024-09-13T14:34:44.577 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:44 smithi195 ceph-mon[104262]: pgmap v628: 89 pgs: 1 active+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 738 MiB data, 4.4 GiB used, 443 GiB / 447 GiB avail; 0 B/s wr, 1.18k op/s; 743/35304 objects degraded (2.105%); 5141/35304 objects misplaced (14.562%); 0 B/s, 87 objects/s recovering 2024-09-13T14:34:44.578 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:44 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 1407/49773 objects degraded (2.827%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:34:44.578 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:44 smithi195 ceph-mon[104262]: osdmap e197: 8 total, 8 up, 5 in 2024-09-13T14:34:44.607 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:44.628 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:44.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:44.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:44.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:44 smithi102 ceph-mon[110183]: osdmap e196: 8 total, 8 up, 5 in 2024-09-13T14:34:44.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:44 smithi102 ceph-mon[110183]: pgmap v628: 89 pgs: 1 active+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 738 MiB data, 4.4 GiB used, 443 GiB / 447 GiB avail; 0 B/s wr, 1.18k op/s; 743/35304 objects degraded (2.105%); 5141/35304 objects misplaced (14.562%); 0 B/s, 87 objects/s recovering 2024-09-13T14:34:44.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:44 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 1407/49773 objects degraded (2.827%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:34:44.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:44 smithi102 ceph-mon[110183]: osdmap e197: 8 total, 8 up, 5 in 2024-09-13T14:34:44.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:44 smithi102 ceph-mon[115269]: osdmap e196: 8 total, 8 up, 5 in 2024-09-13T14:34:44.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:44 smithi102 ceph-mon[115269]: pgmap v628: 89 pgs: 1 active+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 738 MiB data, 4.4 GiB used, 443 GiB / 447 GiB avail; 0 B/s wr, 1.18k op/s; 743/35304 objects degraded (2.105%); 5141/35304 objects misplaced (14.562%); 0 B/s, 87 objects/s recovering 2024-09-13T14:34:44.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:44 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 1407/49773 objects degraded (2.827%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:34:44.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:44 smithi102 ceph-mon[115269]: osdmap e197: 8 total, 8 up, 5 in 2024-09-13T14:34:44.730 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:44.751 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:44.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:44.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:44.853 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:44.874 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:44.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:44.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:44.976 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:45.005 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:45.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:45.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:45.107 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:45.138 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:45.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:45.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:45.240 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:45.265 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:45.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:45.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:45.367 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:45.388 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:45.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:45.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:45.490 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:45.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:45.520 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:45.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:45.621 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:45.652 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:45.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:45.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:45.754 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:45.785 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:45.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:45.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:45.887 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:45.916 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:45.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:45.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:45.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:45 smithi195 ceph-mon[104262]: osdmap e198: 8 total, 8 up, 5 in 2024-09-13T14:34:46.018 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:46.044 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:46.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:46.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:46.149 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:46.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:45 smithi102 ceph-mon[110183]: osdmap e198: 8 total, 8 up, 5 in 2024-09-13T14:34:46.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:45 smithi102 ceph-mon[115269]: osdmap e198: 8 total, 8 up, 5 in 2024-09-13T14:34:46.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:46.181 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:46.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:46.282 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:46.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:46.318 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:46.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:46.419 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:46.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:46.460 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:46.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:46.487 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:34:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:34:46] "GET /metrics HTTP/1.1" 200 36535 "" "Prometheus/2.51.0" 2024-09-13T14:34:46.562 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:46.595 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:46.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:46.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:46.697 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:46.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:46.733 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:46.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:46.834 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:46.871 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:46.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:46.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:46.973 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:47.023 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:47.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:47.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:47.059 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:46 smithi102 ceph-mon[110183]: pgmap v631: 89 pgs: 1 active+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 738 MiB data, 4.4 GiB used, 443 GiB / 447 GiB avail; 0 B/s wr, 1.21k op/s; 743/35304 objects degraded (2.105%); 5141/35304 objects misplaced (14.562%); 0 B/s, 91 objects/s recovering 2024-09-13T14:34:47.059 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:46 smithi102 ceph-mon[115269]: pgmap v631: 89 pgs: 1 active+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 738 MiB data, 4.4 GiB used, 443 GiB / 447 GiB avail; 0 B/s wr, 1.21k op/s; 743/35304 objects degraded (2.105%); 5141/35304 objects misplaced (14.562%); 0 B/s, 91 objects/s recovering 2024-09-13T14:34:47.099 INFO:teuthology.orchestra.run.smithi102.stderr:no valid command found; 10 closest matches: 2024-09-13T14:34:47.099 INFO:teuthology.orchestra.run.smithi102.stderr:0 2024-09-13T14:34:47.099 INFO:teuthology.orchestra.run.smithi102.stderr:1 2024-09-13T14:34:47.099 INFO:teuthology.orchestra.run.smithi102.stderr:2 2024-09-13T14:34:47.099 INFO:teuthology.orchestra.run.smithi102.stderr:abort 2024-09-13T14:34:47.100 INFO:teuthology.orchestra.run.smithi102.stderr:assert 2024-09-13T14:34:47.100 INFO:teuthology.orchestra.run.smithi102.stderr:bench [] [] [] [] 2024-09-13T14:34:47.100 INFO:teuthology.orchestra.run.smithi102.stderr:bluefs debug_inject_read_zeros 2024-09-13T14:34:47.100 INFO:teuthology.orchestra.run.smithi102.stderr:bluefs files list 2024-09-13T14:34:47.100 INFO:teuthology.orchestra.run.smithi102.stderr:bluefs stats 2024-09-13T14:34:47.100 INFO:teuthology.orchestra.run.smithi102.stderr:bluestore allocator dump block 2024-09-13T14:34:47.100 INFO:teuthology.orchestra.run.smithi102.stderr:Error EINVAL: invalid command 2024-09-13T14:34:47.102 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:46 smithi195 ceph-mon[104262]: pgmap v631: 89 pgs: 1 active+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 79 active+clean; 738 MiB data, 4.4 GiB used, 443 GiB / 447 GiB avail; 0 B/s wr, 1.21k op/s; 743/35304 objects degraded (2.105%); 5141/35304 objects misplaced (14.562%); 0 B/s, 91 objects/s recovering 2024-09-13T14:34:47.125 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:47.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:47.154 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:47.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:47.166 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:34:47.167 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-13T14:34:47.167 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.167 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35037.0:212329 6.c e198/183 6:3e52259a:::benchmark_data_smithi102_194590_object101705:head v 198'13085, mlcod=198'13081)", 2024-09-13T14:34:47.167 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:34:47.162596+0000", 2024-09-13T14:34:47.167 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.0020880759999999999, 2024-09-13T14:34:47.167 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0021165229999999999, 2024-09-13T14:34:47.167 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:47.167 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T14:34:47.167 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:47.167 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.167 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:47.168 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.162596+0000", 2024-09-13T14:34:47.168 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:47.168 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.168 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.168 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:47.168 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.162596+0000", 2024-09-13T14:34:47.168 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:47.168 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.168 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.168 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:47.168 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.162595+0000", 2024-09-13T14:34:47.168 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:34:47.168 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.168 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.169 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:47.169 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.162603+0000", 2024-09-13T14:34:47.169 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8930000000000005e-06 2024-09-13T14:34:47.169 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.169 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.169 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:47.169 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.162616+0000", 2024-09-13T14:34:47.169 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3305000000000001e-05 2024-09-13T14:34:47.169 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.169 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.169 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:47.169 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.162624+0000", 2024-09-13T14:34:47.169 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7039999999999994e-06 2024-09-13T14:34:47.170 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.170 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.170 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:47.170 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.162704+0000", 2024-09-13T14:34:47.170 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9523000000000004e-05 2024-09-13T14:34:47.170 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.170 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.170 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:34:47.170 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.162721+0000", 2024-09-13T14:34:47.170 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7592999999999999e-05 2024-09-13T14:34:47.170 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:47.170 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:47.170 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:47.170 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.171 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.171 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop_reply(client.35037.0:212325 6.3 e198/161)", 2024-09-13T14:34:47.171 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:34:47.163593+0000", 2024-09-13T14:34:47.171 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.0010908249999999999, 2024-09-13T14:34:47.171 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001175686, 2024-09-13T14:34:47.171 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:47.171 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "queued for pg", 2024-09-13T14:34:47.171 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:47.171 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.171 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:47.171 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.163593+0000", 2024-09-13T14:34:47.171 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:47.171 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.171 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.171 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:47.172 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.163593+0000", 2024-09-13T14:34:47.172 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:47.172 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.172 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.172 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:47.172 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.163593+0000", 2024-09-13T14:34:47.172 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:34:47.172 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.172 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.172 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:47.172 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.163602+0000", 2024-09-13T14:34:47.172 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1759999999999999e-06 2024-09-13T14:34:47.173 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.173 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.173 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:47.173 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.163614+0000", 2024-09-13T14:34:47.173 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1997000000000001e-05 2024-09-13T14:34:47.173 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.173 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.173 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:47.173 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.163621+0000", 2024-09-13T14:34:47.173 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.0990000000000001e-06 2024-09-13T14:34:47.173 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:47.173 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:47.174 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:47.174 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.174 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.174 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35037.0:212332 6.9 e198/161 6:972ad274:::benchmark_data_smithi102_194590_object101708:head v 198'13027, mlcod=198'13025)", 2024-09-13T14:34:47.174 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:34:47.164704+0000", 2024-09-13T14:34:47.174 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 4294967295.99998, 2024-09-13T14:34:47.174 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010273, 2024-09-13T14:34:47.174 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:47.174 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T14:34:47.174 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:47.174 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.174 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:47.174 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.164704+0000", 2024-09-13T14:34:47.175 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:47.175 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.175 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.175 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:47.175 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.164704+0000", 2024-09-13T14:34:47.175 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:47.175 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.175 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.175 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:47.175 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.164704+0000", 2024-09-13T14:34:47.175 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:34:47.175 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.176 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.176 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:47.176 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.164710+0000", 2024-09-13T14:34:47.176 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0730000000000001e-06 2024-09-13T14:34:47.176 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.176 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.176 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:47.176 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.164721+0000", 2024-09-13T14:34:47.176 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0759e-05 2024-09-13T14:34:47.176 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.176 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.176 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:47.176 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.164726+0000", 2024-09-13T14:34:47.177 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8710000000000002e-06 2024-09-13T14:34:47.177 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.177 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.177 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:47.177 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.164776+0000", 2024-09-13T14:34:47.177 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9246999999999998e-05 2024-09-13T14:34:47.177 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.177 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.177 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:34:47.177 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.164794+0000", 2024-09-13T14:34:47.177 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7907999999999999e-05 2024-09-13T14:34:47.177 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:47.177 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:47.177 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:47.178 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.178 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.178 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:212309 6.3 6:c3cc192e:::benchmark_data_smithi102_194590_object101685:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-13T14:34:47.178 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:34:47.134117+0000", 2024-09-13T14:34:47.178 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.030567637000000002, 2024-09-13T14:34:47.178 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.030739466999999999, 2024-09-13T14:34:47.178 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:47.178 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:34:47.178 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:34:47.178 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:34:47.178 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:34:47.178 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 212309 2024-09-13T14:34:47.178 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.178 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:47.179 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.179 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:47.179 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.134117+0000", 2024-09-13T14:34:47.179 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:47.179 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.179 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.179 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:47.179 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.134117+0000", 2024-09-13T14:34:47.179 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:47.179 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.179 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.179 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:47.179 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.134116+0000", 2024-09-13T14:34:47.179 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:34:47.180 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.180 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.180 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:47.180 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.134119+0000", 2024-09-13T14:34:47.180 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7e-06 2024-09-13T14:34:47.180 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.180 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.180 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:47.180 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.134125+0000", 2024-09-13T14:34:47.180 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1240000000000002e-06 2024-09-13T14:34:47.180 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.180 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.180 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:47.181 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.134129+0000", 2024-09-13T14:34:47.181 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4800000000000003e-06 2024-09-13T14:34:47.181 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.181 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.181 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:47.181 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.134968+0000", 2024-09-13T14:34:47.181 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000838097 2024-09-13T14:34:47.181 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.181 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.181 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:34:47.182 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.135083+0000", 2024-09-13T14:34:47.182 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000115054 2024-09-13T14:34:47.182 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.182 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.182 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,6", 2024-09-13T14:34:47.182 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.135186+0000", 2024-09-13T14:34:47.182 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000103368 2024-09-13T14:34:47.182 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.182 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.182 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:34:47.182 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.140637+0000", 2024-09-13T14:34:47.182 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0054507130000000003 2024-09-13T14:34:47.183 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.183 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.183 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:34:47.183 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.142060+0000", 2024-09-13T14:34:47.183 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0014229329999999999 2024-09-13T14:34:47.183 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:47.183 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:47.183 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:47.183 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.183 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.183 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:212314 6.5 6:ab289088:::benchmark_data_smithi102_194590_object101690:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-13T14:34:47.183 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:34:47.141469+0000", 2024-09-13T14:34:47.184 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.023215561999999999, 2024-09-13T14:34:47.184 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.023465667999999999, 2024-09-13T14:34:47.184 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:47.184 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:34:47.184 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:34:47.184 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:34:47.184 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:34:47.184 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 212314 2024-09-13T14:34:47.184 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.184 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:47.184 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.184 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:47.185 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.141469+0000", 2024-09-13T14:34:47.185 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:47.185 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.185 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.185 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:47.185 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.141469+0000", 2024-09-13T14:34:47.185 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:47.185 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.185 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.185 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:47.185 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.141467+0000", 2024-09-13T14:34:47.185 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:34:47.186 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.186 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.186 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:47.186 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.141473+0000", 2024-09-13T14:34:47.186 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4940000000000003e-06 2024-09-13T14:34:47.186 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.186 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.186 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:47.186 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.141485+0000", 2024-09-13T14:34:47.186 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2245e-05 2024-09-13T14:34:47.186 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.186 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.186 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:47.187 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.141491+0000", 2024-09-13T14:34:47.187 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.212e-06 2024-09-13T14:34:47.187 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.187 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.187 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:47.187 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.143325+0000", 2024-09-13T14:34:47.187 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0018339470000000001 2024-09-13T14:34:47.187 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.187 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.187 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:34:47.187 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.143441+0000", 2024-09-13T14:34:47.187 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011584200000000001 2024-09-13T14:34:47.187 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.187 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.188 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:34:47.188 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.143560+0000", 2024-09-13T14:34:47.188 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000118757 2024-09-13T14:34:47.188 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.188 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.188 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:34:47.188 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.155198+0000", 2024-09-13T14:34:47.188 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011638133 2024-09-13T14:34:47.188 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.188 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.188 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:34:47.188 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.155214+0000", 2024-09-13T14:34:47.188 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5692000000000002e-05 2024-09-13T14:34:47.189 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:47.189 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:47.189 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:47.189 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.189 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.189 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:212315 6.3 6:c622c195:::benchmark_data_smithi102_194590_object101691:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-13T14:34:47.189 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:34:47.144195+0000", 2024-09-13T14:34:47.189 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.020489211, 2024-09-13T14:34:47.189 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.020803407999999999, 2024-09-13T14:34:47.189 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:47.189 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:34:47.190 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:34:47.190 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:34:47.190 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:34:47.190 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 212315 2024-09-13T14:34:47.190 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.190 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:47.190 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.190 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:47.190 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.144195+0000", 2024-09-13T14:34:47.190 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:47.190 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.190 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.190 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:47.190 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.144195+0000", 2024-09-13T14:34:47.191 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:47.191 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.191 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.191 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:47.191 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.144194+0000", 2024-09-13T14:34:47.191 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:34:47.191 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.191 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.191 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:47.191 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.144200+0000", 2024-09-13T14:34:47.191 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8250000000000001e-06 2024-09-13T14:34:47.191 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.191 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.191 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:47.192 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.144215+0000", 2024-09-13T14:34:47.192 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.562e-05 2024-09-13T14:34:47.192 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.192 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.192 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:47.192 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.144222+0000", 2024-09-13T14:34:47.192 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6669999999999999e-06 2024-09-13T14:34:47.192 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.192 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.192 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:47.192 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.144285+0000", 2024-09-13T14:34:47.192 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3584999999999995e-05 2024-09-13T14:34:47.192 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.192 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.193 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:34:47.193 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.144506+0000", 2024-09-13T14:34:47.193 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00022075999999999999 2024-09-13T14:34:47.193 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.193 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.193 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,6", 2024-09-13T14:34:47.193 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.144615+0000", 2024-09-13T14:34:47.193 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000108436 2024-09-13T14:34:47.193 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.193 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.193 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:34:47.194 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.158380+0000", 2024-09-13T14:34:47.194 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013765539 2024-09-13T14:34:47.194 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.194 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.194 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:34:47.194 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.158387+0000", 2024-09-13T14:34:47.194 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8059999999999997e-06 2024-09-13T14:34:47.194 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:47.194 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:47.194 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:47.194 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.194 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.194 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:212320 6.3 6:c6723ad5:::benchmark_data_smithi102_194590_object101696:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-13T14:34:47.194 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:34:47.145438+0000", 2024-09-13T14:34:47.195 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.019245911000000001, 2024-09-13T14:34:47.195 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.019623062, 2024-09-13T14:34:47.195 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:47.195 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:34:47.195 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:34:47.195 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:34:47.195 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:34:47.195 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 212320 2024-09-13T14:34:47.195 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.195 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:47.195 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.195 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:47.195 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.145438+0000", 2024-09-13T14:34:47.195 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:47.195 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.196 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.196 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:47.196 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.145438+0000", 2024-09-13T14:34:47.196 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:47.196 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.196 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.196 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:47.196 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.145437+0000", 2024-09-13T14:34:47.196 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:34:47.196 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.196 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.196 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:47.196 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.145443+0000", 2024-09-13T14:34:47.196 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2909999999999998e-06 2024-09-13T14:34:47.197 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.197 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.197 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:47.197 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.145455+0000", 2024-09-13T14:34:47.197 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2098e-05 2024-09-13T14:34:47.197 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.197 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.197 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:47.197 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.145462+0000", 2024-09-13T14:34:47.197 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8929999999999999e-06 2024-09-13T14:34:47.197 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.197 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.197 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:47.197 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.151481+0000", 2024-09-13T14:34:47.198 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0060194200000000002 2024-09-13T14:34:47.198 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.198 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.198 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:34:47.198 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.151608+0000", 2024-09-13T14:34:47.198 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012742299999999999 2024-09-13T14:34:47.198 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.198 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.198 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,6", 2024-09-13T14:34:47.198 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.151718+0000", 2024-09-13T14:34:47.198 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000109406 2024-09-13T14:34:47.198 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.199 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.199 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:34:47.199 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.158419+0000", 2024-09-13T14:34:47.199 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0067012540000000002 2024-09-13T14:34:47.199 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.199 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.199 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:34:47.199 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.158426+0000", 2024-09-13T14:34:47.199 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.956e-06 2024-09-13T14:34:47.199 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:47.199 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:47.199 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:47.199 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.200 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.200 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:212325 6.3 6:c3317ddc:::benchmark_data_smithi102_194590_object101701:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-13T14:34:47.200 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:34:47.157316+0000", 2024-09-13T14:34:47.200 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.0073682169999999998, 2024-09-13T14:34:47.200 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0078176979999999997, 2024-09-13T14:34:47.200 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:47.200 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:34:47.200 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:34:47.200 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:34:47.200 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:34:47.200 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 212325 2024-09-13T14:34:47.200 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.201 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:47.201 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.201 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:47.201 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.157316+0000", 2024-09-13T14:34:47.201 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:47.201 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.201 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.201 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:47.201 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.157316+0000", 2024-09-13T14:34:47.201 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:47.201 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.201 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.202 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:47.202 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.157314+0000", 2024-09-13T14:34:47.202 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:34:47.202 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.202 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.202 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:47.202 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.157322+0000", 2024-09-13T14:34:47.202 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7619999999999995e-06 2024-09-13T14:34:47.202 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.202 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.202 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:47.202 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.157338+0000", 2024-09-13T14:34:47.202 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5886999999999999e-05 2024-09-13T14:34:47.203 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.203 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.203 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:47.203 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.157346+0000", 2024-09-13T14:34:47.203 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8550000000000002e-06 2024-09-13T14:34:47.203 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.203 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.203 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:47.203 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.158446+0000", 2024-09-13T14:34:47.203 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001100632 2024-09-13T14:34:47.203 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.203 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.203 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:34:47.203 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.158548+0000", 2024-09-13T14:34:47.204 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010128600000000001 2024-09-13T14:34:47.204 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.204 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.204 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,6", 2024-09-13T14:34:47.204 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.158652+0000", 2024-09-13T14:34:47.204 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000104681 2024-09-13T14:34:47.204 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:47.204 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:47.204 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:47.204 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.204 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.204 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:212327 6.5 6:a25f5b17:::benchmark_data_smithi102_194590_object101703:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-13T14:34:47.204 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:34:47.158078+0000", 2024-09-13T14:34:47.205 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.0066062279999999996, 2024-09-13T14:34:47.205 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0071248270000000002, 2024-09-13T14:34:47.205 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:34:47.205 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:34:47.205 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:34:47.205 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:34:47.205 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:34:47.205 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 212327 2024-09-13T14:34:47.205 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.205 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:34:47.205 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.205 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:34:47.205 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.158078+0000", 2024-09-13T14:34:47.205 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:47.206 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.206 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.206 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:34:47.206 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.158078+0000", 2024-09-13T14:34:47.206 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:34:47.206 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.206 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.206 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:34:47.206 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.158077+0000", 2024-09-13T14:34:47.206 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:34:47.206 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.206 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.206 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:34:47.206 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.158082+0000", 2024-09-13T14:34:47.207 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5580000000000002e-06 2024-09-13T14:34:47.207 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.207 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.207 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:34:47.207 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.158093+0000", 2024-09-13T14:34:47.207 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1126e-05 2024-09-13T14:34:47.207 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.207 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.207 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:34:47.207 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.158099+0000", 2024-09-13T14:34:47.207 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0360000000000003e-06 2024-09-13T14:34:47.207 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.207 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.207 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:34:47.207 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.158129+0000", 2024-09-13T14:34:47.208 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0103999999999998e-05 2024-09-13T14:34:47.208 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.208 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.208 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:34:47.208 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.158244+0000", 2024-09-13T14:34:47.208 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000115269 2024-09-13T14:34:47.208 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.208 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.208 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:34:47.208 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.158370+0000", 2024-09-13T14:34:47.208 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012605799999999999 2024-09-13T14:34:47.208 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.208 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.208 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:34:47.209 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.163469+0000", 2024-09-13T14:34:47.209 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0050983970000000002 2024-09-13T14:34:47.209 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:34:47.209 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:34:47.209 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:34:47.209 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:47.164170+0000", 2024-09-13T14:34:47.209 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00070151499999999999 2024-09-13T14:34:47.209 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:47.209 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:34:47.209 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:47.209 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:34:47.209 INFO:teuthology.orchestra.run.smithi195.stdout: ], 2024-09-13T14:34:47.209 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 9 2024-09-13T14:34:47.209 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:34:47.255 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:47.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:47.298 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:47.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:47.399 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:34:47.435 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:47.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:47.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:34:47.537 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:47.565 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:47.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:47.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:47.667 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:47.713 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:47.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:47.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:47.815 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:47.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:47.843 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:47.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:47.851 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is set 2024-09-13T14:34:47.902 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T14:34:47.902 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T14:34:47.903 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.4 'trim stale osdmaps' 2024-09-13T14:34:47.944 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:47.986 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:47.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:47.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:47.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:47 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1079405921' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:34:48.088 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:48.093 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T14:34:48.128 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:48.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:48.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:48.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:47 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1079405921' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:34:48.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:47 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1079405921' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:34:48.230 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:48.257 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:48.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:48.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:48.359 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:48.405 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:48.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:48.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:48.508 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:34:48.509 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:34:48.561 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:48.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:48.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:34:48.591 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:34:48.663 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:34:48.699 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:48.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:48.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:34:48.801 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:48.825 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:48.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:48.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:48.927 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:48.953 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:48.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:48.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:48.995 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:48 smithi195 ceph-mon[104262]: pgmap v632: 89 pgs: 1 active+recovery_wait, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait, 80 active+clean; 585 MiB data, 4.0 GiB used, 443 GiB / 447 GiB avail; 0 B/s wr, 601 op/s; 9/27927 objects degraded (0.032%); 3540/27927 objects misplaced (12.676%); 8.5 MiB/s, 301 objects/s recovering 2024-09-13T14:34:48.995 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:48 smithi195 ceph-mon[104262]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:34:48.995 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:48 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1079405921' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:34:48.995 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:48 smithi195 ceph-mon[104262]: osdmap e199: 8 total, 8 up, 5 in 2024-09-13T14:34:48.995 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:48 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 9/27927 objects degraded (0.032%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:34:49.024 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:48 smithi102 ceph-mon[110183]: pgmap v632: 89 pgs: 1 active+recovery_wait, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait, 80 active+clean; 585 MiB data, 4.0 GiB used, 443 GiB / 447 GiB avail; 0 B/s wr, 601 op/s; 9/27927 objects degraded (0.032%); 3540/27927 objects misplaced (12.676%); 8.5 MiB/s, 301 objects/s recovering 2024-09-13T14:34:49.025 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:48 smithi102 ceph-mon[110183]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:34:49.025 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:48 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1079405921' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:34:49.025 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:48 smithi102 ceph-mon[110183]: osdmap e199: 8 total, 8 up, 5 in 2024-09-13T14:34:49.025 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:48 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 9/27927 objects degraded (0.032%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:34:49.025 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:48 smithi102 ceph-mon[115269]: pgmap v632: 89 pgs: 1 active+recovery_wait, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait, 80 active+clean; 585 MiB data, 4.0 GiB used, 443 GiB / 447 GiB avail; 0 B/s wr, 601 op/s; 9/27927 objects degraded (0.032%); 3540/27927 objects misplaced (12.676%); 8.5 MiB/s, 301 objects/s recovering 2024-09-13T14:34:49.025 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:48 smithi102 ceph-mon[115269]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:34:49.025 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:48 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1079405921' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:34:49.025 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:48 smithi102 ceph-mon[115269]: osdmap e199: 8 total, 8 up, 5 in 2024-09-13T14:34:49.025 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:48 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 9/27927 objects degraded (0.032%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:34:49.055 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:49.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:49.097 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:49.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:49.199 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:49.227 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:49.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:49.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:49.328 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:49.353 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:34:49.360 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:49.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:49.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:49.462 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:49.491 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:49.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:49.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:49.593 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:49.620 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:49.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:49.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:49.722 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:49.749 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:49.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:49.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:49.851 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:49.890 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:49.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:49.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:49.992 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:50.020 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:50.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:50.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:50.122 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:50.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:50.166 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:50.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:50.268 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:50.297 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:50.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:50.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:50.399 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:50.444 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:50.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:50.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:50.521 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:34:50.546 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:34:50.546 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-13T14:34:50.546 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-09-13T14:34:50.546 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-09-13T14:34:50.546 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:34:50.546 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:50.592 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:50.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:50.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:50.694 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:50.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:50.733 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:50.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:50.834 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:50.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:50.867 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:50.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:50.968 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:51.011 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:51.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:51.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:51.112 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:51.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:51.144 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:51.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:51.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:50 smithi102 ceph-mon[110183]: pgmap v634: 89 pgs: 1 active+recovery_wait, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait, 80 active+clean; 559 MiB data, 4.0 GiB used, 443 GiB / 447 GiB avail; 0 B/s wr, 478 op/s; 9/26694 objects degraded (0.034%); 3355/26694 objects misplaced (12.568%); 7.2 MiB/s, 256 objects/s recovering 2024-09-13T14:34:51.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:50 smithi102 ceph-mon[115269]: pgmap v634: 89 pgs: 1 active+recovery_wait, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait, 80 active+clean; 559 MiB data, 4.0 GiB used, 443 GiB / 447 GiB avail; 0 B/s wr, 478 op/s; 9/26694 objects degraded (0.034%); 3355/26694 objects misplaced (12.568%); 7.2 MiB/s, 256 objects/s recovering 2024-09-13T14:34:51.204 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:50 smithi195 ceph-mon[104262]: pgmap v634: 89 pgs: 1 active+recovery_wait, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait, 80 active+clean; 559 MiB data, 4.0 GiB used, 443 GiB / 447 GiB avail; 0 B/s wr, 478 op/s; 9/26694 objects degraded (0.034%); 3355/26694 objects misplaced (12.568%); 7.2 MiB/s, 256 objects/s recovering 2024-09-13T14:34:51.237 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T14:34:51.246 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:34:51.342 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:51.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:51.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:34:51.444 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:51.471 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:51.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:51.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:51.573 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:51.580 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:34:51.594 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:51.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:51.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:51.696 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:51.730 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:51.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:51.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:51.833 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:51.874 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:51.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:51.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:51.976 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:51.981 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:51 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/254367814' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:34:51.981 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:51 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/254367814' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:34:51.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:51 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/254367814' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:34:52.020 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:52.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:52.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:52.123 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:52.165 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:52.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:52.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:52.267 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:52.309 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:52.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:52.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:52.411 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:52.451 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:52.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:52.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:52.553 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:52.562 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:34:52.562 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:34:52.562 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:34:52.562 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:34:52.562 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:34:52.563 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:34:52.563 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:34:52.563 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:34:52.563 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:34:52.563 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:34:52.563 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:34:52.563 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:34:52.564 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:34:52.564 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:34:52.564 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:34:52.564 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:34:52.583 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:52.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:52.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:52.685 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:52.724 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:52.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:52.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:52.825 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:52.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:52.848 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:52.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:52.949 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:52.977 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:52.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:52.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:53.079 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:34:53.115 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:53.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:53.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:34:53.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:52 smithi102 ceph-mon[110183]: pgmap v635: 89 pgs: 1 active+recovery_wait, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait, 80 active+clean; 559 MiB data, 4.0 GiB used, 443 GiB / 447 GiB avail; 0 B/s wr, 379 op/s; 9/26694 objects degraded (0.034%); 3355/26694 objects misplaced (12.568%); 5.7 MiB/s, 203 objects/s recovering 2024-09-13T14:34:53.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:52 smithi102 ceph-mon[115269]: pgmap v635: 89 pgs: 1 active+recovery_wait, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait, 80 active+clean; 559 MiB data, 4.0 GiB used, 443 GiB / 447 GiB avail; 0 B/s wr, 379 op/s; 9/26694 objects degraded (0.034%); 3355/26694 objects misplaced (12.568%); 5.7 MiB/s, 203 objects/s recovering 2024-09-13T14:34:53.215 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:52 smithi195 ceph-mon[104262]: pgmap v635: 89 pgs: 1 active+recovery_wait, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait, 80 active+clean; 559 MiB data, 4.0 GiB used, 443 GiB / 447 GiB avail; 0 B/s wr, 379 op/s; 9/26694 objects degraded (0.034%); 3355/26694 objects misplaced (12.568%); 5.7 MiB/s, 203 objects/s recovering 2024-09-13T14:34:53.217 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:53.257 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:53.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:53.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:53.358 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:53.386 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:53.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:53.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:53.488 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:53.510 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:53.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:53.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:53.611 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:53.667 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:53.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:53.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:53.769 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:53.808 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:53.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:53.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:53.909 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:53.952 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:53.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:53.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:54.054 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:54.059 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:53 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 9/26694 objects degraded (0.034%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:34:54.078 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:54.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:54.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:54.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:53 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 9/26694 objects degraded (0.034%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:34:54.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:53 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 9/26694 objects degraded (0.034%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:34:54.181 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:54.223 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:34:54.223 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:34:54.226 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:54.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:54.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:54.328 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:34:54.363 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:54.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:54.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:34:54.465 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:54.503 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:54.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:54.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:54.522 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:34:54.605 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:54.638 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:54.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:54.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:54.740 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:34:54.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:54.767 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:54.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:34:54.868 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:54.905 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:54.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:54.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:54.948 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T14:34:55.007 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:55.039 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:55.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:55.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:55.141 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:55.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:54 smithi102 ceph-mon[110183]: pgmap v636: 89 pgs: 1 active+recovery_wait, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait, 80 active+clean; 257 MiB data, 3.2 GiB used, 444 GiB / 447 GiB avail; 0 B/s wr, 900 op/s; 9/12216 objects degraded (0.074%); 1530/12216 objects misplaced (12.525%); 5.0 MiB/s, 220 objects/s recovering 2024-09-13T14:34:55.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:54 smithi102 ceph-mon[115269]: pgmap v636: 89 pgs: 1 active+recovery_wait, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait, 80 active+clean; 257 MiB data, 3.2 GiB used, 444 GiB / 447 GiB avail; 0 B/s wr, 900 op/s; 9/12216 objects degraded (0.074%); 1530/12216 objects misplaced (12.525%); 5.0 MiB/s, 220 objects/s recovering 2024-09-13T14:34:55.178 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:55.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:55.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:55.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:54 smithi195 ceph-mon[104262]: pgmap v636: 89 pgs: 1 active+recovery_wait, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait, 80 active+clean; 257 MiB data, 3.2 GiB used, 444 GiB / 447 GiB avail; 0 B/s wr, 900 op/s; 9/12216 objects degraded (0.074%); 1530/12216 objects misplaced (12.525%); 5.0 MiB/s, 220 objects/s recovering 2024-09-13T14:34:55.280 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:55.314 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:55.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:55.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:55.415 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:55.452 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:55.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:55.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:55.553 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:55.591 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:55.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:55.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:55.692 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:55.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:55.718 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:55.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:55.819 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:55.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:55.851 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:55.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:55.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:55 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:34:55.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:55 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:34:55.952 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:55.999 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:55.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:56.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:56.101 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:56.131 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:56.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:56.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:56.148 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:55 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:34:56.233 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:56.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:56.260 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:56.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:56.361 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:56.406 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:56.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:56.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:56.508 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:56.533 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:56.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:56.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:56.596 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Removed 110621 objects 2024-09-13T14:34:56.596 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Clean up completed and total clean up time :131.688 2024-09-13T14:34:56.634 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:56.656 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:56.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:56.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:56.758 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:56.780 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:56.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:56.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:56.878 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:34:56 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:34:56] "GET /metrics HTTP/1.1" 200 36542 "" "Prometheus/2.51.0" 2024-09-13T14:34:56.882 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:56.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:56.904 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:56.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:56.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:56 smithi195 ceph-mon[104262]: pgmap v637: 89 pgs: 1 active+recovery_wait, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait, 80 active+clean; 257 MiB data, 3.2 GiB used, 444 GiB / 447 GiB avail; 0 B/s wr, 769 op/s; 9/12216 objects degraded (0.074%); 1530/12216 objects misplaced (12.525%); 4.3 MiB/s, 188 objects/s recovering 2024-09-13T14:34:57.005 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:57.025 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:57.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:57.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:57.127 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:57.148 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:57.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:57.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:57.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:56 smithi102 ceph-mon[110183]: pgmap v637: 89 pgs: 1 active+recovery_wait, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait, 80 active+clean; 257 MiB data, 3.2 GiB used, 444 GiB / 447 GiB avail; 0 B/s wr, 769 op/s; 9/12216 objects degraded (0.074%); 1530/12216 objects misplaced (12.525%); 4.3 MiB/s, 188 objects/s recovering 2024-09-13T14:34:57.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:56 smithi102 ceph-mon[115269]: pgmap v637: 89 pgs: 1 active+recovery_wait, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait, 80 active+clean; 257 MiB data, 3.2 GiB used, 444 GiB / 447 GiB avail; 0 B/s wr, 769 op/s; 9/12216 objects degraded (0.074%); 1530/12216 objects misplaced (12.525%); 4.3 MiB/s, 188 objects/s recovering 2024-09-13T14:34:57.250 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:57.271 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:57.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:57.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:57.373 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:57.398 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:57.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:57.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:57.500 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:57.521 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:57.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:57.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:57.623 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:57.649 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:57.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:57.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:57.751 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:57.771 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:57.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:57.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:57.872 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:57.910 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:57.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:57.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:58.012 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:58.033 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:58.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:58.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:58.135 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:58.159 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:58.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:58.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:58.260 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:58.283 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:58.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:58.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:58.385 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:34:58.407 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:58.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:58.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:34:58.508 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:58.531 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:58.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:58.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:58.633 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:58.655 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:58.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:58.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:58.757 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:58.778 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:58.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:58.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:58.880 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:34:58.897 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:58 smithi102 ceph-mon[110183]: pgmap v638: 89 pgs: 3 active+clean+remapped, 1 active+recovering+remapped, 1 active+undersized+remapped, 3 active+remapped+backfill_wait, 81 active+clean; 33 MiB data, 2.6 GiB used, 444 GiB / 447 GiB avail; 0 B/s wr, 881 op/s; 225/1467 objects misplaced (15.337%); 0 B/s, 84 objects/s recovering 2024-09-13T14:34:58.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:58 smithi102 ceph-mon[110183]: osdmap e200: 8 total, 8 up, 5 in 2024-09-13T14:34:58.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:58 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-13T14:34:58.898 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:58 smithi102 ceph-mon[115269]: pgmap v638: 89 pgs: 3 active+clean+remapped, 1 active+recovering+remapped, 1 active+undersized+remapped, 3 active+remapped+backfill_wait, 81 active+clean; 33 MiB data, 2.6 GiB used, 444 GiB / 447 GiB avail; 0 B/s wr, 881 op/s; 225/1467 objects misplaced (15.337%); 0 B/s, 84 objects/s recovering 2024-09-13T14:34:58.898 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:58 smithi102 ceph-mon[115269]: osdmap e200: 8 total, 8 up, 5 in 2024-09-13T14:34:58.898 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:58 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-13T14:34:58.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:58.915 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:58.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:34:59.016 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:59.037 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:59.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:59.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:59.139 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:59.161 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:59.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:59.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:59.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:58 smithi195 ceph-mon[104262]: pgmap v638: 89 pgs: 3 active+clean+remapped, 1 active+recovering+remapped, 1 active+undersized+remapped, 3 active+remapped+backfill_wait, 81 active+clean; 33 MiB data, 2.6 GiB used, 444 GiB / 447 GiB avail; 0 B/s wr, 881 op/s; 225/1467 objects misplaced (15.337%); 0 B/s, 84 objects/s recovering 2024-09-13T14:34:59.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:58 smithi195 ceph-mon[104262]: osdmap e200: 8 total, 8 up, 5 in 2024-09-13T14:34:59.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:58 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-13T14:34:59.263 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:59.283 INFO:teuthology.orchestra.run.smithi102.stderr:no valid command found; 10 closest matches: 2024-09-13T14:34:59.283 INFO:teuthology.orchestra.run.smithi102.stderr:0 2024-09-13T14:34:59.283 INFO:teuthology.orchestra.run.smithi102.stderr:1 2024-09-13T14:34:59.284 INFO:teuthology.orchestra.run.smithi102.stderr:2 2024-09-13T14:34:59.284 INFO:teuthology.orchestra.run.smithi102.stderr:abort 2024-09-13T14:34:59.284 INFO:teuthology.orchestra.run.smithi102.stderr:assert 2024-09-13T14:34:59.284 INFO:teuthology.orchestra.run.smithi102.stderr:bench [] [] [] [] 2024-09-13T14:34:59.284 INFO:teuthology.orchestra.run.smithi102.stderr:bluefs debug_inject_read_zeros 2024-09-13T14:34:59.284 INFO:teuthology.orchestra.run.smithi102.stderr:bluefs files list 2024-09-13T14:34:59.284 INFO:teuthology.orchestra.run.smithi102.stderr:bluefs stats 2024-09-13T14:34:59.284 INFO:teuthology.orchestra.run.smithi102.stderr:bluestore allocator dump block 2024-09-13T14:34:59.284 INFO:teuthology.orchestra.run.smithi102.stderr:Error EINVAL: invalid command 2024-09-13T14:34:59.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:59.285 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:59.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:34:59.386 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:34:59.410 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:59.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:59.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:34:59.511 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:34:59.533 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:59.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:59.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:34:59.635 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:34:59.657 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:34:59.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:59.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:34:59.758 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:34:59.780 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:59.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:59.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:34:59.882 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:34:59.904 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:34:59.905 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:34:59.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:34:59.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:00.007 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:00.009 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T14:35:00.009 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T14:35:00.009 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.6 'trim stale osdmaps' 2024-09-13T14:35:00.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:00.030 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:00.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:00.131 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:00.153 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:00.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:00.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:00.160 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:34:59 smithi102 ceph-mon[115269]: osdmap e201: 8 total, 8 up, 5 in 2024-09-13T14:35:00.160 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:34:59 smithi102 ceph-mon[110183]: osdmap e201: 8 total, 8 up, 5 in 2024-09-13T14:35:00.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:34:59 smithi195 ceph-mon[104262]: osdmap e201: 8 total, 8 up, 5 in 2024-09-13T14:35:00.255 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:00.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:00.278 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:00.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:00.379 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:35:00.402 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:00.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:00.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:35:00.504 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:00.527 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:00.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:00.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:00.628 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:00.648 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-09-13T14:35:00.648 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-09-13T14:35:00.655 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:00.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:00.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:00.756 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:00.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:00.781 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:00.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:00.883 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:35:00.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:00.910 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:00.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:35:01.011 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:01.033 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:01.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:01.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:01.056 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:35:01.117 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:35:01.118 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:35:01.135 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:01.156 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:01.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:01.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:01.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:00 smithi102 ceph-mon[110183]: pgmap v641: 89 pgs: 4 active+clean+remapped, 1 active+recovering+remapped, 1 active+undersized+remapped, 2 active+remapped+backfill_wait, 81 active+clean; 4.6 MiB data, 2.5 GiB used, 445 GiB / 447 GiB avail; 0 B/s wr, 1.11k op/s; 0 B/s, 105 objects/s recovering 2024-09-13T14:35:01.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:00 smithi102 ceph-mon[115269]: pgmap v641: 89 pgs: 4 active+clean+remapped, 1 active+recovering+remapped, 1 active+undersized+remapped, 2 active+remapped+backfill_wait, 81 active+clean; 4.6 MiB data, 2.5 GiB used, 445 GiB / 447 GiB avail; 0 B/s wr, 1.11k op/s; 0 B/s, 105 objects/s recovering 2024-09-13T14:35:01.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:00 smithi195 ceph-mon[104262]: pgmap v641: 89 pgs: 4 active+clean+remapped, 1 active+recovering+remapped, 1 active+undersized+remapped, 2 active+remapped+backfill_wait, 81 active+clean; 4.6 MiB data, 2.5 GiB used, 445 GiB / 447 GiB avail; 0 B/s wr, 1.11k op/s; 0 B/s, 105 objects/s recovering 2024-09-13T14:35:01.259 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:01.284 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:01.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:01.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:01.352 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:35:01.386 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:01.411 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:01.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:01.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:01.513 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:01.535 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:01.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:01.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:01.637 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:35:01.663 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:01.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:01.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:35:01.765 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:01.790 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:01.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:01.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:01.892 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:01.911 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is unset 2024-09-13T14:35:01.915 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:01.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:01.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:02.017 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:02.038 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:02.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:02.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:02.073 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:01 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2289108920' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:35:02.074 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:01 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2289108920' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:35:02.140 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:02.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:02.163 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:02.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:02.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:01 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2289108920' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:35:02.264 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:02.290 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:02.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:02.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:02.391 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:02.417 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:02.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:02.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:02.519 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:02.544 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:02.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:02.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:02.646 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:02.667 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:02.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:02.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:02.769 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:02.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:02.793 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:02.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:02.894 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:02.916 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:02.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:02.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:02.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:02 smithi102 ceph-mon[115269]: pgmap v642: 89 pgs: 4 active+clean+remapped, 1 active+recovering+remapped, 1 active+undersized+remapped, 2 active+remapped+backfill_wait, 81 active+clean; 4.6 MiB data, 2.5 GiB used, 445 GiB / 447 GiB avail; 0 B/s wr, 505 op/s; 0 B/s, 61 objects/s recovering 2024-09-13T14:35:02.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:02 smithi102 ceph-mon[115269]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:35:02.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:02 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2289108920' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:35:02.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:02 smithi102 ceph-mon[115269]: osdmap e202: 8 total, 8 up, 5 in 2024-09-13T14:35:02.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:02 smithi102 ceph-mon[110183]: pgmap v642: 89 pgs: 4 active+clean+remapped, 1 active+recovering+remapped, 1 active+undersized+remapped, 2 active+remapped+backfill_wait, 81 active+clean; 4.6 MiB data, 2.5 GiB used, 445 GiB / 447 GiB avail; 0 B/s wr, 505 op/s; 0 B/s, 61 objects/s recovering 2024-09-13T14:35:02.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:02 smithi102 ceph-mon[110183]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:35:02.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:02 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2289108920' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:35:02.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:02 smithi102 ceph-mon[110183]: osdmap e202: 8 total, 8 up, 5 in 2024-09-13T14:35:03.017 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:03.039 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:03.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:03.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:03.140 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:03.163 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:03.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:03.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:03.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:02 smithi195 ceph-mon[104262]: pgmap v642: 89 pgs: 4 active+clean+remapped, 1 active+recovering+remapped, 1 active+undersized+remapped, 2 active+remapped+backfill_wait, 81 active+clean; 4.6 MiB data, 2.5 GiB used, 445 GiB / 447 GiB avail; 0 B/s wr, 505 op/s; 0 B/s, 61 objects/s recovering 2024-09-13T14:35:03.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:02 smithi195 ceph-mon[104262]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:35:03.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:02 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2289108920' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:35:03.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:02 smithi195 ceph-mon[104262]: osdmap e202: 8 total, 8 up, 5 in 2024-09-13T14:35:03.264 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:03.286 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:03.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:03.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:03.388 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:03.409 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:03.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:03.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:03.511 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:03.532 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:03.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:03.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:03.634 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:03.660 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:03.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:03.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:03.762 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:03.787 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:03.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:03.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:03.888 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:03.910 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:03.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:03.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:04.012 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:04.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:04.034 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:04.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:04.135 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:04.156 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:04.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:04.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:04.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:03 smithi102 ceph-mon[115269]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-13T14:35:04.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:03 smithi102 ceph-mon[110183]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-13T14:35:04.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:03 smithi195 ceph-mon[104262]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-13T14:35:04.259 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:04.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:04.285 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:04.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:04.386 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:04.408 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:04.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:04.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:04.510 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:04.531 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:04.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:04.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:04.633 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:04.656 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:04.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:04.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:04.759 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:04.785 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:04.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:04.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:04.887 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:35:04.913 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:04.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:04.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:35:05.015 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:35:05.040 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:05.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:05.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:35:05.142 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:35:05.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:04 smithi102 ceph-mon[110183]: pgmap v644: 89 pgs: 89 active+clean; 4.6 MiB data, 2.5 GiB used, 445 GiB / 447 GiB avail; 0 B/s wr, 76 op/s; 0 B/s, 7 objects/s recovering 2024-09-13T14:35:05.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:04 smithi102 ceph-mon[115269]: pgmap v644: 89 pgs: 89 active+clean; 4.6 MiB data, 2.5 GiB used, 445 GiB / 447 GiB avail; 0 B/s wr, 76 op/s; 0 B/s, 7 objects/s recovering 2024-09-13T14:35:05.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:05.170 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:05.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:35:05.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:04 smithi195 ceph-mon[104262]: pgmap v644: 89 pgs: 89 active+clean; 4.6 MiB data, 2.5 GiB used, 445 GiB / 447 GiB avail; 0 B/s wr, 76 op/s; 0 B/s, 7 objects/s recovering 2024-09-13T14:35:05.272 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:05.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:05.299 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:05.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:05.400 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:05.424 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:05.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:05.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:05.526 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:05.549 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:05.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:05.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:05.651 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:05.674 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:05.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:05.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:05.776 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:05.799 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:05.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:05.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:05.901 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:05.913 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:35:05.913 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:35:05.924 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:05.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:05.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:06.026 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:06.049 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:06.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:06.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:06.150 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:06.173 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:06.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:06.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:06.275 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:06.299 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:06.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:06.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:06.400 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:06.423 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:06.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:06.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:06.525 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:06.546 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:06.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:06.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:06.648 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:06.663 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:35:06.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:35:06 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:35:06] "GET /metrics HTTP/1.1" 200 36542 "" "Prometheus/2.51.0" 2024-09-13T14:35:06.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:06.671 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:06.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:06.771 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:06.793 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:06.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:06.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:06.895 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:06.917 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:06.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:06.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:07.018 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:35:07.045 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:07.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:07.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:35:07.146 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:07.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:07.168 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:07.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:07.237 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:06 smithi102 ceph-mon[110183]: pgmap v645: 89 pgs: 89 active+clean; 4.6 MiB data, 2.5 GiB used, 445 GiB / 447 GiB avail; 0 B/s, 6 objects/s recovering 2024-09-13T14:35:07.238 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:06 smithi102 ceph-mon[115269]: pgmap v645: 89 pgs: 89 active+clean; 4.6 MiB data, 2.5 GiB used, 445 GiB / 447 GiB avail; 0 B/s, 6 objects/s recovering 2024-09-13T14:35:07.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:06 smithi195 ceph-mon[104262]: pgmap v645: 89 pgs: 89 active+clean; 4.6 MiB data, 2.5 GiB used, 445 GiB / 447 GiB avail; 0 B/s, 6 objects/s recovering 2024-09-13T14:35:07.270 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:07.291 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:07.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:07.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:07.350 INFO:teuthology.orchestra.run.smithi102.stderr:no valid command found; 10 closest matches: 2024-09-13T14:35:07.350 INFO:teuthology.orchestra.run.smithi102.stderr:0 2024-09-13T14:35:07.351 INFO:teuthology.orchestra.run.smithi102.stderr:1 2024-09-13T14:35:07.351 INFO:teuthology.orchestra.run.smithi102.stderr:2 2024-09-13T14:35:07.351 INFO:teuthology.orchestra.run.smithi102.stderr:abort 2024-09-13T14:35:07.351 INFO:teuthology.orchestra.run.smithi102.stderr:assert 2024-09-13T14:35:07.351 INFO:teuthology.orchestra.run.smithi102.stderr:bench [] [] [] [] 2024-09-13T14:35:07.351 INFO:teuthology.orchestra.run.smithi102.stderr:bluefs debug_inject_read_zeros 2024-09-13T14:35:07.351 INFO:teuthology.orchestra.run.smithi102.stderr:bluefs files list 2024-09-13T14:35:07.351 INFO:teuthology.orchestra.run.smithi102.stderr:bluefs stats 2024-09-13T14:35:07.351 INFO:teuthology.orchestra.run.smithi102.stderr:bluestore allocator dump block 2024-09-13T14:35:07.351 INFO:teuthology.orchestra.run.smithi102.stderr:Error EINVAL: invalid command 2024-09-13T14:35:07.393 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:07.414 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:07.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:07.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:07.515 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:07.536 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:07.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:07.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:07.638 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:07.660 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:07.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:07.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:07.761 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:07.782 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:07.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:07.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:07.884 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:35:07.910 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:07.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:07.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:35:08.011 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:08.067 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:08.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:08.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:08.168 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:08.174 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:35:08.174 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:35:08.191 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:08.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:08.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:08.194 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:35:08.194 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:35:08.211 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:35:08.211 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:35:08.228 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:35:08.228 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:35:08.244 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:35:08.245 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:35:08.260 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:35:08.261 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:35:08.277 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:35:08.277 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:35:08.293 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:35:08.293 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:35:08.294 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:35:08.316 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:08.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:08.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:35:08.418 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:08.441 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:08.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:08.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:08.542 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:08.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:08.565 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:08.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:08.666 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:08.689 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:08.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:08.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:08.724 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T14:35:08.724 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T14:35:08.725 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.7 'trim stale osdmaps' 2024-09-13T14:35:08.791 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:08.829 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:08.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:08.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:08.931 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:08.955 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:08.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:08.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:09.057 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:09.083 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:09.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:09.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:09.185 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:09.207 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:09.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:09.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:09.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:08 smithi195 ceph-mon[104262]: pgmap v646: 89 pgs: 89 active+clean; 4.6 MiB data, 2.4 GiB used, 445 GiB / 447 GiB avail; 0 B/s, 5 objects/s recovering 2024-09-13T14:35:09.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:08 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3209978376' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:35:09.309 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:09.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:09.330 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:09.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:09.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:08 smithi102 ceph-mon[115269]: pgmap v646: 89 pgs: 89 active+clean; 4.6 MiB data, 2.4 GiB used, 445 GiB / 447 GiB avail; 0 B/s, 5 objects/s recovering 2024-09-13T14:35:09.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:08 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3209978376' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:35:09.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:08 smithi102 ceph-mon[110183]: pgmap v646: 89 pgs: 89 active+clean; 4.6 MiB data, 2.4 GiB used, 445 GiB / 447 GiB avail; 0 B/s, 5 objects/s recovering 2024-09-13T14:35:09.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:08 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3209978376' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:35:09.431 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:09.437 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T14:35:09.457 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:09.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:09.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:09.559 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:35:09.584 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:09.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:09.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:35:09.686 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:09.710 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:09.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:09.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:09.812 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:09.838 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:09.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:09.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:09.939 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:09.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:09.963 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:09.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:10.064 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:10.089 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:10.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:10.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:10.191 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:10.212 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:10.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:10.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:10.281 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:35:10.315 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:10.325 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:35:10.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:10.339 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:10.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:10.440 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:10.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:10.467 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:10.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:10.567 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:35:10.568 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:35:10.594 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:10.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:10.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:35:10.696 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:10.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:10.722 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:10.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:10.823 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:10.849 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:10.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:10.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:10.951 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:10.971 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'unique_pool_1' removed 2024-09-13T14:35:10.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:10.976 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:10.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:11.077 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:11.108 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:11.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:11.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:11.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:10 smithi102 ceph-mon[115269]: pgmap v647: 89 pgs: 89 active+clean; 4.6 MiB data, 2.4 GiB used, 445 GiB / 447 GiB avail; 0 B/s, 4 objects/s recovering 2024-09-13T14:35:11.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:10 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2960395436' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:35:11.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:10 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:35:11.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:35:11.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:35:11.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:10 smithi102 ceph-mon[110183]: pgmap v647: 89 pgs: 89 active+clean; 4.6 MiB data, 2.4 GiB used, 445 GiB / 447 GiB avail; 0 B/s, 4 objects/s recovering 2024-09-13T14:35:11.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:10 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2960395436' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:35:11.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:10 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:35:11.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:35:11.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:35:11.211 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:11.231 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:11.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:11.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:11.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:10 smithi195 ceph-mon[104262]: pgmap v647: 89 pgs: 89 active+clean; 4.6 MiB data, 2.4 GiB used, 445 GiB / 447 GiB avail; 0 B/s, 4 objects/s recovering 2024-09-13T14:35:11.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:10 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2960395436' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:35:11.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:10 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:35:11.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:35:11.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:35:11.333 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:11.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:11.356 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:11.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:11.457 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:11.478 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:11.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:11.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:11.580 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:11.601 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:11.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:11.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:11.703 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:35:11.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:11.725 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:11.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:35:11.825 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:11.848 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:11.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:11.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:11.950 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:11.972 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:11.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:11.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:12.074 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:12.095 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:12.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:12.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:12.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:11 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:35:12.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:11 smithi102 ceph-mon[115269]: osdmap e203: 8 total, 8 up, 5 in 2024-09-13T14:35:12.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:11 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:35:12.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:11 smithi102 ceph-mon[110183]: osdmap e203: 8 total, 8 up, 5 in 2024-09-13T14:35:12.200 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:12.221 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:12.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:12.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:12.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:11 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:35:12.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:11 smithi195 ceph-mon[104262]: osdmap e203: 8 total, 8 up, 5 in 2024-09-13T14:35:12.305 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:35:12.323 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:35:12.345 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:12.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:12.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:35:12.447 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:12.469 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:12.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:12.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:12.570 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:12.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:12.595 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:12.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:12.696 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:12.718 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:12.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:12.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:12.820 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:12.841 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:12.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:12.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:12.943 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:12.965 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:12.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:12.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:13.067 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:13.089 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:13.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:13.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:13.106 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-13T14:35:13.107 INFO:tasks.radosbench:Beginning radosbench... 2024-09-13T14:35:13.107 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-09-13T14:35:13.107 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_2 16 2024-09-13T14:35:13.191 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:13.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:13.214 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:13.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:13.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:12 smithi195 ceph-mon[104262]: pgmap v649: 73 pgs: 73 active+clean; 4.6 MiB data, 2.4 GiB used, 445 GiB / 447 GiB avail 2024-09-13T14:35:13.315 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:13.336 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:13.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:13.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:13.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:12 smithi102 ceph-mon[110183]: pgmap v649: 73 pgs: 73 active+clean; 4.6 MiB data, 2.4 GiB used, 445 GiB / 447 GiB avail 2024-09-13T14:35:13.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:12 smithi102 ceph-mon[115269]: pgmap v649: 73 pgs: 73 active+clean; 4.6 MiB data, 2.4 GiB used, 445 GiB / 447 GiB avail 2024-09-13T14:35:13.438 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:13.459 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:13.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:13.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:13.561 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:13.582 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:13.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:13.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:13.684 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:13.706 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:13.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:13.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:13.808 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:13.831 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:13.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:13.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:13.932 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:13.954 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:13.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:13.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:14.057 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:14.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:14.079 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:14.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:14.143 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:35:14.143 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:35:14.181 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:14.201 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:14.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:14.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:14.303 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:14.324 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:14.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:14.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:14.426 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:14.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:14.450 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:14.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:14.551 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:14.572 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:14.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:14.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:14.674 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:35:14.700 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:14.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:14.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:35:14.802 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:14.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:14.825 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:14.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:14.926 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:14.952 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:14.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:14.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:15.054 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:15.075 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:15.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:15.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:15.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:14 smithi102 ceph-mon[110183]: pgmap v650: 73 pgs: 73 active+clean; 4.6 MiB data, 2.4 GiB used, 445 GiB / 447 GiB avail 2024-09-13T14:35:15.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:14 smithi102 ceph-mon[115269]: pgmap v650: 73 pgs: 73 active+clean; 4.6 MiB data, 2.4 GiB used, 445 GiB / 447 GiB avail 2024-09-13T14:35:15.177 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:15.202 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:15.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:15.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:15.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:14 smithi195 ceph-mon[104262]: pgmap v650: 73 pgs: 73 active+clean; 4.6 MiB data, 2.4 GiB used, 445 GiB / 447 GiB avail 2024-09-13T14:35:15.305 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:15.327 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:15.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:15.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:15.429 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:15.454 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:15.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:15.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:15.556 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:15.581 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:15.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:15.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:15.683 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:15.704 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:15.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:15.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:15.806 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:15.832 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:15.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:15.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:15.934 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:15.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:15.957 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:15.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:16.059 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:16.066 INFO:teuthology.orchestra.run.smithi102.stderr:no valid command found; 10 closest matches: 2024-09-13T14:35:16.066 INFO:teuthology.orchestra.run.smithi102.stderr:0 2024-09-13T14:35:16.066 INFO:teuthology.orchestra.run.smithi102.stderr:1 2024-09-13T14:35:16.066 INFO:teuthology.orchestra.run.smithi102.stderr:2 2024-09-13T14:35:16.066 INFO:teuthology.orchestra.run.smithi102.stderr:abort 2024-09-13T14:35:16.066 INFO:teuthology.orchestra.run.smithi102.stderr:assert 2024-09-13T14:35:16.067 INFO:teuthology.orchestra.run.smithi102.stderr:bench [] [] [] [] 2024-09-13T14:35:16.067 INFO:teuthology.orchestra.run.smithi102.stderr:bluefs debug_inject_read_zeros 2024-09-13T14:35:16.067 INFO:teuthology.orchestra.run.smithi102.stderr:bluefs files list 2024-09-13T14:35:16.067 INFO:teuthology.orchestra.run.smithi102.stderr:bluefs stats 2024-09-13T14:35:16.067 INFO:teuthology.orchestra.run.smithi102.stderr:bluestore allocator dump block 2024-09-13T14:35:16.067 INFO:teuthology.orchestra.run.smithi102.stderr:Error EINVAL: invalid command 2024-09-13T14:35:16.084 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:16.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:16.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:16.186 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:35:16.212 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:16.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:16.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:35:16.314 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:16.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:16.337 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:16.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:16.438 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:16.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:16.459 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:16.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:16.561 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:35:16.586 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:16.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:16.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:35:16.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:35:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:35:16] "GET /metrics HTTP/1.1" 200 36519 "" "Prometheus/2.51.0" 2024-09-13T14:35:16.688 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:16.714 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:16.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:16.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:16.722 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T14:35:16.722 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T14:35:16.816 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:16.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:16.839 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:16.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:16.940 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:16.965 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:16.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:16.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:17.067 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:17.088 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:17.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:17.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:17.190 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:35:17.216 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:17.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:17.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:35:17.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:16 smithi195 ceph-mon[104262]: pgmap v651: 73 pgs: 73 active+clean; 4.6 MiB data, 2.4 GiB used, 445 GiB / 447 GiB avail 2024-09-13T14:35:17.257 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:16 smithi102 ceph-mon[110183]: pgmap v651: 73 pgs: 73 active+clean; 4.6 MiB data, 2.4 GiB used, 445 GiB / 447 GiB avail 2024-09-13T14:35:17.257 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:16 smithi102 ceph-mon[115269]: pgmap v651: 73 pgs: 73 active+clean; 4.6 MiB data, 2.4 GiB used, 445 GiB / 447 GiB avail 2024-09-13T14:35:17.319 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:17.340 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:17.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:17.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:17.442 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:17.463 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:17.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:17.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:17.566 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:35:17.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:17.594 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:17.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:35:17.695 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:17.717 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:17.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:17.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:17.818 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:17.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:17.840 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:17.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:17.941 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:35:17.967 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:17.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:17.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:35:18.069 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:18.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:18.091 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:18.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:18.192 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:18.214 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:18.214 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:35:18.214 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [], 2024-09-13T14:35:18.214 INFO:teuthology.orchestra.run.smithi102.stdout: "complaint_time": 30, 2024-09-13T14:35:18.214 INFO:teuthology.orchestra.run.smithi102.stdout: "num_blocked_ops": 0 2024-09-13T14:35:18.214 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:35:18.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:18.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:18.316 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:18.337 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:18.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:18.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:18.439 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:18.461 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:18.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:18.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:18.562 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:18.583 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:18.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:18.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:18.685 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:18.706 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:18.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:18.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:18.808 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:18.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:18.829 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:18.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:18.930 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:18.948 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T14:35:18.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:18.952 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:18.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:19.054 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:19.075 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:19.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:19.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:19.177 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:19.199 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:19.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:19.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:19.205 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:18 smithi102 ceph-mon[110183]: pgmap v652: 73 pgs: 73 active+clean; 4.6 MiB data, 2.4 GiB used, 445 GiB / 447 GiB avail 2024-09-13T14:35:19.205 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:18 smithi102 ceph-mon[115269]: pgmap v652: 73 pgs: 73 active+clean; 4.6 MiB data, 2.4 GiB used, 445 GiB / 447 GiB avail 2024-09-13T14:35:19.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:18 smithi195 ceph-mon[104262]: pgmap v652: 73 pgs: 73 active+clean; 4.6 MiB data, 2.4 GiB used, 445 GiB / 447 GiB avail 2024-09-13T14:35:19.301 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:19.322 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:19.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:19.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:19.424 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:19.445 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:19.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:19.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:19.547 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:19.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:19.573 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:19.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:19.674 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:35:19.700 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:19.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:19.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:35:19.802 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:19.827 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:19.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:19.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:19.928 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:19.950 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:19.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:19.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:19.988 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is unset 2024-09-13T14:35:20.052 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:35:20.078 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:20.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:20.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:35:20.180 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:20.205 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:20.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:20.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:20.235 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:19 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1299184813' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:35:20.235 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:19 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1299184813' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:35:20.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:19 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1299184813' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:35:20.254 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:35:20.254 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:35:20.254 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:35:20.254 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:35:20.269 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:35:20.269 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:35:20.285 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:35:20.285 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:35:20.302 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:35:20.302 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:35:20.306 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:20.318 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:35:20.318 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:35:20.331 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:20.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:20.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:20.336 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:35:20.336 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:35:20.352 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:35:20.353 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:35:20.432 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:20.453 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:20.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:20.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:20.555 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:35:20.581 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:20.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:20.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:35:20.682 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:20.707 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:20.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:20.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:20.809 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:20.831 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:20.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:20.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:20.933 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:20.959 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:20.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:20.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:21.060 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:35:21.085 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:21.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:21.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:35:21.130 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:20 smithi102 ceph-mon[110183]: pgmap v653: 73 pgs: 73 active+clean; 4.6 MiB data, 2.4 GiB used, 445 GiB / 447 GiB avail 2024-09-13T14:35:21.130 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:20 smithi102 ceph-mon[110183]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:35:21.130 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:20 smithi102 ceph-mon[110183]: Cluster is now healthy 2024-09-13T14:35:21.130 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:20 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1299184813' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:35:21.130 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:20 smithi102 ceph-mon[110183]: osdmap e204: 8 total, 8 up, 5 in 2024-09-13T14:35:21.130 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:20 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1701521342' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:35:21.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:20 smithi102 ceph-mon[115269]: pgmap v653: 73 pgs: 73 active+clean; 4.6 MiB data, 2.4 GiB used, 445 GiB / 447 GiB avail 2024-09-13T14:35:21.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:20 smithi102 ceph-mon[115269]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:35:21.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:20 smithi102 ceph-mon[115269]: Cluster is now healthy 2024-09-13T14:35:21.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:20 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1299184813' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:35:21.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:20 smithi102 ceph-mon[115269]: osdmap e204: 8 total, 8 up, 5 in 2024-09-13T14:35:21.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:20 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1701521342' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:35:21.187 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:21.208 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:21.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:21.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:21.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:20 smithi195 ceph-mon[104262]: pgmap v653: 73 pgs: 73 active+clean; 4.6 MiB data, 2.4 GiB used, 445 GiB / 447 GiB avail 2024-09-13T14:35:21.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:20 smithi195 ceph-mon[104262]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:35:21.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:20 smithi195 ceph-mon[104262]: Cluster is now healthy 2024-09-13T14:35:21.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:20 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1299184813' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:35:21.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:20 smithi195 ceph-mon[104262]: osdmap e204: 8 total, 8 up, 5 in 2024-09-13T14:35:21.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:20 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1701521342' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:35:21.310 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:35:21.336 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:21.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:21.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:35:21.438 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:21.462 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:21.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:21.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:21.564 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:21.586 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:21.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:21.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:21.688 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:21.709 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:21.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:21.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:21.723 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 6, 7] out_osds: [1, 2, 5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:35:21.724 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:35:21.724 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-09-13T14:35:21.724 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T14:35:21.724 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T14:35:21.724 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-09-13T14:35:21.811 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:21.831 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:21.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:21.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:21.933 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:21.955 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:21.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:21.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:21.970 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:35:22.057 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:22.078 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:22.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:22.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:22.180 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:22.201 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:22.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:22.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:22.303 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:22.314 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:35:22.324 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:22.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:22.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:22.426 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:22.448 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:22.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:22.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:22.549 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:35:22.570 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:22.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:22.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:35:22.672 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:22.694 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:22.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:22.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:22.796 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:22.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:22.818 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:22.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:22.919 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:22.941 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:22.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:22.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:22.991 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'unique_pool_2' created 2024-09-13T14:35:23.042 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:23.063 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:23.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:23.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:23.105 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:22 smithi195 ceph-mon[104262]: pgmap v655: 73 pgs: 73 active+clean; 4.6 MiB data, 2.4 GiB used, 445 GiB / 447 GiB avail 2024-09-13T14:35:23.105 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:22 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/753364998' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-13T14:35:23.165 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:23.188 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:23.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:23.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:23.285 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:22 smithi102 ceph-mon[115269]: pgmap v655: 73 pgs: 73 active+clean; 4.6 MiB data, 2.4 GiB used, 445 GiB / 447 GiB avail 2024-09-13T14:35:23.286 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:22 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/753364998' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-13T14:35:23.286 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:22 smithi102 ceph-mon[110183]: pgmap v655: 73 pgs: 73 active+clean; 4.6 MiB data, 2.4 GiB used, 445 GiB / 447 GiB avail 2024-09-13T14:35:23.286 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:22 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/753364998' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-13T14:35:23.289 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:23.311 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:23.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:23.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:23.413 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:23.433 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:23.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:23.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:23.535 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:35:23.558 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:23.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:23.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:35:23.652 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:35:23.653 INFO:teuthology.orchestra.run.smithi195.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-13T14:35:23.653 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:35:23.660 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:23.681 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:23.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:23.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:23.733 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-09-13T14:35:23.783 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:23.805 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:23.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:23.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:23.814 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:35:23.907 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:23.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:23.929 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:23.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:23.997 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:23 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/753364998' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-13T14:35:23.998 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:23 smithi102 ceph-mon[110183]: osdmap e205: 8 total, 8 up, 5 in 2024-09-13T14:35:23.998 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:23 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/753364998' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-13T14:35:23.998 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:23 smithi102 ceph-mon[115269]: osdmap e205: 8 total, 8 up, 5 in 2024-09-13T14:35:24.030 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:24.052 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:24.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:24.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:24.154 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:35:24.175 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:24.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:24.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:35:24.199 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:35:24.225 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:35:24.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:23 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/753364998' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-13T14:35:24.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:23 smithi195 ceph-mon[104262]: osdmap e205: 8 total, 8 up, 5 in 2024-09-13T14:35:24.277 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:24.298 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:24.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:24.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:24.400 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:24.421 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:24.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:24.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:24.522 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:24.545 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:24.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:24.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:24.647 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:24.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:24.669 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:24.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:24.770 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:24.792 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:24.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:24.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:24.893 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:24.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:24.915 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:24.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:24.922 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:35:24.922 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:35:25.017 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:25.038 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:25.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:25.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:25.140 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:25.161 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:25.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:25.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:25.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:24 smithi102 ceph-mon[115269]: pgmap v657: 89 pgs: 16 unknown, 73 active+clean; 4.6 MiB data, 2.3 GiB used, 445 GiB / 447 GiB avail 2024-09-13T14:35:25.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:24 smithi102 ceph-mon[115269]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:35:25.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:24 smithi102 ceph-mon[115269]: osdmap e206: 8 total, 8 up, 5 in 2024-09-13T14:35:25.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:24 smithi102 ceph-mon[110183]: pgmap v657: 89 pgs: 16 unknown, 73 active+clean; 4.6 MiB data, 2.3 GiB used, 445 GiB / 447 GiB avail 2024-09-13T14:35:25.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:24 smithi102 ceph-mon[110183]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:35:25.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:24 smithi102 ceph-mon[110183]: osdmap e206: 8 total, 8 up, 5 in 2024-09-13T14:35:25.263 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:35:25.285 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:25.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:25.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:35:25.387 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:25.408 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:25.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:25.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:25.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:24 smithi195 ceph-mon[104262]: pgmap v657: 89 pgs: 16 unknown, 73 active+clean; 4.6 MiB data, 2.3 GiB used, 445 GiB / 447 GiB avail 2024-09-13T14:35:25.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:24 smithi195 ceph-mon[104262]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:35:25.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:24 smithi195 ceph-mon[104262]: osdmap e206: 8 total, 8 up, 5 in 2024-09-13T14:35:25.510 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:25.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:25.531 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:25.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:25.632 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:25.648 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T14:35:25.653 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:25.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:25.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:25.755 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:25.778 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:25.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:25.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:25.880 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:25.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:25.902 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:25.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:26.003 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:26.025 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:26.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:26.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:26.053 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:35:26.096 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:35:26.128 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:26.149 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:26.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:26.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:26.251 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:26.272 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:26.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:26.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:26.300 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:26 smithi102 ceph-mon[110183]: osdmap e207: 8 total, 8 up, 5 in 2024-09-13T14:35:26.300 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:35:26.300 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:35:26.300 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:26 smithi102 ceph-mon[115269]: osdmap e207: 8 total, 8 up, 5 in 2024-09-13T14:35:26.300 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:35:26.300 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:35:26.374 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:26.395 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:26.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:26.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:26.399 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:26 smithi195 ceph-mon[104262]: osdmap e207: 8 total, 8 up, 5 in 2024-09-13T14:35:26.399 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:35:26.399 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:35:26.498 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:26.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:26.522 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:26.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:26.568 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:35:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:35:26] "GET /metrics HTTP/1.1" 200 36461 "" "Prometheus/2.51.0" 2024-09-13T14:35:26.623 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:26.644 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:26.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:26.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:26.746 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:26.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:26.770 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:26.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:26.871 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:35:26.892 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:26.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:26.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:35:26.994 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:27.015 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:27.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:27.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:27.117 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:27.139 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:27.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:27.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:27.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:27 smithi102 ceph-mon[110183]: pgmap v660: 89 pgs: 16 unknown, 73 active+clean; 4.6 MiB data, 2.3 GiB used, 445 GiB / 447 GiB avail 2024-09-13T14:35:27.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:27 smithi102 ceph-mon[115269]: pgmap v660: 89 pgs: 16 unknown, 73 active+clean; 4.6 MiB data, 2.3 GiB used, 445 GiB / 447 GiB avail 2024-09-13T14:35:27.241 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:35:27.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:27.264 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:27.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:35:27.365 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:27.386 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:27.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:27.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:27.488 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:27.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:27 smithi195 ceph-mon[104262]: pgmap v660: 89 pgs: 16 unknown, 73 active+clean; 4.6 MiB data, 2.3 GiB used, 445 GiB / 447 GiB avail 2024-09-13T14:35:27.509 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:27.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:27.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:27.605 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:35:27.611 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:27.634 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:27.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:27.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:27.736 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:27.757 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:27.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:27.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:27.859 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:35:27.881 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:27.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:27.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:35:27.982 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:28.004 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:28.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:28.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:28.105 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:28.127 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:28.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:28.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:28.229 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:28.249 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:28.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:28.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:28.352 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:28.373 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:28.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:28.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:28.475 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:35:28.499 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:28.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:28.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:35:28.600 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:28.624 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:28.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:28.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:28.726 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:28.750 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:28.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:28.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:28.851 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:28.875 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:28.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:28.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:28.977 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:28.999 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:29.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:29.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:29.101 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:29.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:29.127 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:29.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:29.228 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:35:29.253 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:29.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:29.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:35:29.284 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:29 smithi102 ceph-mon[110183]: pgmap v661: 89 pgs: 2 unknown, 87 active+clean; 4.6 MiB data, 2.3 GiB used, 445 GiB / 447 GiB avail 2024-09-13T14:35:29.285 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:29 smithi102 ceph-mon[115269]: pgmap v661: 89 pgs: 2 unknown, 87 active+clean; 4.6 MiB data, 2.3 GiB used, 445 GiB / 447 GiB avail 2024-09-13T14:35:29.309 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 6, 7] out_osds: [1, 2, 5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:35:29.309 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:35:29.310 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-09-13T14:35:29.310 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T14:35:29.310 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T14:35:29.310 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-09-13T14:35:29.337 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:29 smithi195 ceph-mon[104262]: pgmap v661: 89 pgs: 2 unknown, 87 active+clean; 4.6 MiB data, 2.3 GiB used, 445 GiB / 447 GiB avail 2024-09-13T14:35:29.355 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:29.379 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:29.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:29.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:29.481 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:29.504 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:29.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:29.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:29.526 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:35:29.606 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:29.627 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:29.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:29.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:29.728 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:29.749 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:29.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:29.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:29.851 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:29.873 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:29.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:29.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:29.975 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:29.996 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:29.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:29.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:30.098 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:30.121 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:30.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:30.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:30.223 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:30.244 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:30.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:30.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:30.346 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:30.367 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:30.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:30.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:30.469 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:30.491 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:30.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:30.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:30.593 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:30.614 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:30.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:30.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:30.717 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:30.737 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:30.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:30.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:30.839 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:30.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:30.862 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:30.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:30.963 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:30.984 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:30.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:30.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:31.058 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:31 smithi195 ceph-mon[104262]: pgmap v662: 89 pgs: 89 active+clean; 4.6 MiB data, 2.3 GiB used, 445 GiB / 447 GiB avail 2024-09-13T14:35:31.058 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:31 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/959070538' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:35:31.058 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:31 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:35:31.086 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:31.106 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:31.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:31.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:31.158 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:35:31.158 INFO:teuthology.orchestra.run.smithi195.stdout: "success": "" 2024-09-13T14:35:31.158 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:35:31.208 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:35:31.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:31.230 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:31.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:35:31.331 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:31.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:31.355 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:31.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:31.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:31 smithi102 ceph-mon[115269]: pgmap v662: 89 pgs: 89 active+clean; 4.6 MiB data, 2.3 GiB used, 445 GiB / 447 GiB avail 2024-09-13T14:35:31.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:31 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/959070538' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:35:31.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:31 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:35:31.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:31 smithi102 ceph-mon[110183]: pgmap v662: 89 pgs: 89 active+clean; 4.6 MiB data, 2.3 GiB used, 445 GiB / 447 GiB avail 2024-09-13T14:35:31.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:31 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/959070538' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:35:31.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:31 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:35:31.456 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:31.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:31.479 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:31.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:31.580 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:31.601 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:31.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:31.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:31.664 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:35:31.664 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:35:31.702 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:31.724 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:31.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:31.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:31.826 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:31.847 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:31.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:31.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:31.949 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:31.970 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:31.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:31.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:32.072 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:32.095 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:32.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:32.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:32.197 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:32.210 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:32 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:35:32.210 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:32 smithi102 ceph-mon[110183]: osdmap e208: 8 total, 8 up, 5 in 2024-09-13T14:35:32.210 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:32 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:35:32.210 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:32 smithi102 ceph-mon[115269]: osdmap e208: 8 total, 8 up, 5 in 2024-09-13T14:35:32.219 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:32.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:32.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:32.321 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:32.343 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:32.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:32.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:32.419 INFO:teuthology.orchestra.run.smithi102.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-09-13T14:35:32.445 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:32.468 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:32.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:32.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:32.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:32 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:35:32.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:32 smithi195 ceph-mon[104262]: osdmap e208: 8 total, 8 up, 5 in 2024-09-13T14:35:32.570 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:35:32.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:32.592 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:32.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:35:32.693 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:35:32.715 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:32.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:32.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:35:32.816 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:32.838 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:32.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:32.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:32.940 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:32.963 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:32.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:32.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:33.037 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is set 2024-09-13T14:35:33.064 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:33.086 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:33.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:33.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:33.188 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:33.210 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:33.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:33.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:33.311 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:33.332 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:33.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:33.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:33.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:33 smithi102 ceph-mon[110183]: pgmap v664: 89 pgs: 89 active+clean; 4.6 MiB data, 2.3 GiB used, 445 GiB / 447 GiB avail 2024-09-13T14:35:33.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:33 smithi102 ceph-mon[110183]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:35:33.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:33 smithi102 ceph-mon[110183]: Cluster is now healthy 2024-09-13T14:35:33.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:33 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1404910732' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:35:33.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:33 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:35:33.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:33 smithi102 ceph-mon[115269]: pgmap v664: 89 pgs: 89 active+clean; 4.6 MiB data, 2.3 GiB used, 445 GiB / 447 GiB avail 2024-09-13T14:35:33.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:33 smithi102 ceph-mon[115269]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:35:33.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:33 smithi102 ceph-mon[115269]: Cluster is now healthy 2024-09-13T14:35:33.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:33 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1404910732' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:35:33.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:33 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:35:33.433 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:33.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:33.456 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:33.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:33.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:33 smithi195 ceph-mon[104262]: pgmap v664: 89 pgs: 89 active+clean; 4.6 MiB data, 2.3 GiB used, 445 GiB / 447 GiB avail 2024-09-13T14:35:33.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:33 smithi195 ceph-mon[104262]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:35:33.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:33 smithi195 ceph-mon[104262]: Cluster is now healthy 2024-09-13T14:35:33.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:33 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1404910732' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:35:33.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:33 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:35:33.557 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:33.579 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:33.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:33.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:33.687 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:33.709 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:33.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:33.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:33.757 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T14:35:33.811 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:35:33.833 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:33.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:33.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:35:33.934 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:33.956 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:33.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:33.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:33.971 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:35:33.971 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:35:33.973 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:35:33.973 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:35:33.977 DEBUG:teuthology.orchestra.run.smithi102:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-09-13T14:35:33.980 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-13T14:35:33.980 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-13T14:35:33.994 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:35:33.994 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:35:34.013 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:35:34.013 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:35:34.030 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:35:34.030 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:35:34.033 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:hints = 1 2024-09-13T14:35:34.033 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-13T14:35:34.033 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Object prefix: benchmark_data_smithi102_227395 2024-09-13T14:35:34.033 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:35:34.033 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 0 0 0 0 0 0 - 0 2024-09-13T14:35:34.058 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:35:34.060 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:35:34.060 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:35:34.088 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:34.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:34.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:35:34.093 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:35:34.094 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:35:34.125 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:35:34.125 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:35:34.175 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:34 smithi195 ceph-mon[104262]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:35:34.175 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:34 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:35:34.175 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:34 smithi195 ceph-mon[104262]: osdmap e209: 8 total, 8 up, 5 in 2024-09-13T14:35:34.175 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:34 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:35:34.175 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:34 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/4103445730' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:35:34.190 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:34.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:34.229 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:34.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:34.330 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:35:34.362 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:34.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:34.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:35:34.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:34 smithi102 ceph-mon[110183]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:35:34.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:34 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:35:34.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:34 smithi102 ceph-mon[110183]: osdmap e209: 8 total, 8 up, 5 in 2024-09-13T14:35:34.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:34 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:35:34.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:34 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/4103445730' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:35:34.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:34 smithi102 ceph-mon[115269]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:35:34.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:34 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:35:34.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:34 smithi102 ceph-mon[115269]: osdmap e209: 8 total, 8 up, 5 in 2024-09-13T14:35:34.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:34 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:35:34.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:34 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/4103445730' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:35:34.464 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:34.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:34.506 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:34.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:34.607 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:34.642 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:34.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:34.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:34.744 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:34.777 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:34.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:34.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:34.878 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:34.913 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:34.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:34.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:35.015 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:35.033 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 1 16 4144 4128 257.986 258 0.00233668 0.00386633 2024-09-13T14:35:35.051 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:35.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:35.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:35.121 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:35 smithi195 ceph-mon[104262]: pgmap v666: 89 pgs: 89 active+clean; 4.6 MiB data, 2.3 GiB used, 445 GiB / 447 GiB avail 2024-09-13T14:35:35.153 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:35.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:35 smithi102 ceph-mon[110183]: pgmap v666: 89 pgs: 89 active+clean; 4.6 MiB data, 2.3 GiB used, 445 GiB / 447 GiB avail 2024-09-13T14:35:35.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:35 smithi102 ceph-mon[115269]: pgmap v666: 89 pgs: 89 active+clean; 4.6 MiB data, 2.3 GiB used, 445 GiB / 447 GiB avail 2024-09-13T14:35:35.187 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:35.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:35.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:35.263 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:35:35.290 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:35.318 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:35.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:35.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:35.420 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:35.459 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:35.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:35.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:35.561 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:35.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:35.600 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:35.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:35.701 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:35.738 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:35.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:35.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:35.840 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:35.875 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:35.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:35.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:35.977 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:36.011 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:36.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:36.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:36.034 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 2 16 7744 7728 241.465 225 0.00817884 0.00413125 2024-09-13T14:35:36.113 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:36.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:36.151 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:36.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:36.252 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:36.285 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:36.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:36.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:36.386 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:36.421 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:36.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:36.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:36.523 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:36.558 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:36.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:36.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:36.660 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:36.696 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:36.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:36.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:36.733 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 6, 7] out_osds: [1, 2, 5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:35:36.733 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:35:36.734 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-09-13T14:35:36.734 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T14:35:36.734 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T14:35:36.734 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-09-13T14:35:36.798 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:36.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:36.833 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:36.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:36.889 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:35:36.912 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:35:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:35:36] "GET /metrics HTTP/1.1" 200 36461 "" "Prometheus/2.51.0" 2024-09-13T14:35:36.934 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:36.969 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:36.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:36.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:37.034 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 3 16 11206 11190 233.095 216.375 0.00599513 0.00428203 2024-09-13T14:35:37.071 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:37.112 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:37.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:37.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:37.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:37 smithi102 ceph-mon[110183]: pgmap v667: 89 pgs: 89 active+clean; 4.6 MiB data, 2.3 GiB used, 445 GiB / 447 GiB avail 2024-09-13T14:35:37.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:37 smithi102 ceph-mon[115269]: pgmap v667: 89 pgs: 89 active+clean; 4.6 MiB data, 2.3 GiB used, 445 GiB / 447 GiB avail 2024-09-13T14:35:37.214 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:37.227 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:35:37.254 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:37.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:37.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:37.349 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:37 smithi195 ceph-mon[104262]: pgmap v667: 89 pgs: 89 active+clean; 4.6 MiB data, 2.3 GiB used, 445 GiB / 447 GiB avail 2024-09-13T14:35:37.356 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:37.396 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:37.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:37.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:37.498 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:35:37.537 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:37.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:37.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:35:37.565 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:35:37.639 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:37.675 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:37.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:37.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:37.777 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:37.807 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:37.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:37.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:37.909 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:37.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:37.960 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:37.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:38.034 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 4 16 14171 14155 221.128 185.312 0.00389111 0.00451457 2024-09-13T14:35:38.061 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:38.105 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:38.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:38.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:38.207 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:38.236 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:38.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:38.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:38.265 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:35:38.266 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [], 2024-09-13T14:35:38.266 INFO:teuthology.orchestra.run.smithi102.stdout: "complaint_time": 30, 2024-09-13T14:35:38.266 INFO:teuthology.orchestra.run.smithi102.stdout: "num_blocked_ops": 0 2024-09-13T14:35:38.266 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:35:38.338 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:38.374 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:38.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:38.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:38.476 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:38.507 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:38.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:38.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:38.609 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:38.641 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:38.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:38.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:38.743 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:38.776 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:38.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:38.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:38.878 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:38.916 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:38.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:38.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:39.018 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:39.030 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:35:39.034 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 5 16 17224 17208 215.062 190.812 0.00772032 0.00463984 2024-09-13T14:35:39.038 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:35:39.058 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T14:35:39.084 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:39.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:39.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:39.109 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:39 smithi195 ceph-mon[104262]: pgmap v668: 89 pgs: 89 active+clean; 629 MiB data, 3.6 GiB used, 443 GiB / 447 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-13T14:35:39.187 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:39.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:39.234 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:39.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:39.284 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:39 smithi102 ceph-mon[110183]: pgmap v668: 89 pgs: 89 active+clean; 629 MiB data, 3.6 GiB used, 443 GiB / 447 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-13T14:35:39.285 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:39 smithi102 ceph-mon[115269]: pgmap v668: 89 pgs: 89 active+clean; 629 MiB data, 3.6 GiB used, 443 GiB / 447 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-13T14:35:39.335 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:35:39.369 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:39.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:39.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:35:39.471 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:35:39.508 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:39.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:39.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:35:39.610 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:39.645 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:39.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:39.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:39.747 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:39.780 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:39.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:39.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:39.882 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:39.918 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:39.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:39.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:40.020 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:40.034 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 6 16 19979 19963 207.915 172.188 0.00429445 0.00480332 2024-09-13T14:35:40.051 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:40.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:40.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:40.153 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:40.187 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:40.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:40.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:40.289 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:40.321 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:40.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:40.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:40.423 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:40.448 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:40.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:40.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:40.550 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:40.583 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:40.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:40.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:40.685 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:35:40.705 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:40.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:40.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:35:40.807 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:40.843 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:40.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:40.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:40.945 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:40.980 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:40.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:40.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:41.034 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 7 16 22465 22449 200.407 155.375 0.00582045 0.00498463 2024-09-13T14:35:41.082 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:41.103 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:41.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:41.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:41.205 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:41.247 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:41.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:41 smithi195 ceph-mon[104262]: pgmap v669: 89 pgs: 89 active+clean; 732 MiB data, 4.0 GiB used, 443 GiB / 447 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-13T14:35:41.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:41 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:35:41.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:41 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:35:41.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:41.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:41.349 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:35:41.372 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:41.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:41.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:35:41.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:41 smithi102 ceph-mon[110183]: pgmap v669: 89 pgs: 89 active+clean; 732 MiB data, 4.0 GiB used, 443 GiB / 447 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-13T14:35:41.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:41 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:35:41.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:41 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:35:41.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:41 smithi102 ceph-mon[115269]: pgmap v669: 89 pgs: 89 active+clean; 732 MiB data, 4.0 GiB used, 443 GiB / 447 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-13T14:35:41.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:41 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:35:41.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:41 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:35:41.473 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:41.510 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:41.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:41.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:41.612 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:41.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:41.652 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:41.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:41.753 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:41.807 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:41.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:41.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:41.909 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:41.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:41.944 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:41.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:42.034 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 8 16 25004 24988 195.191 158.688 0.00608554 0.00511855 2024-09-13T14:35:42.046 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:42.080 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:42.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:42.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:42.182 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:42.218 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:42.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:42.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:42.320 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:35:42.352 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:42.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:42.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:35:42.454 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:42.468 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:35:42.480 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:42.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:42.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:42.520 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:35:42.520 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:35:42.554 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:35:42.555 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:35:42.582 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:42.586 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:35:42.586 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:35:42.613 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:42.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:42.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:42.618 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:35:42.618 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:35:42.651 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:35:42.651 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:35:42.682 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:35:42.682 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:35:42.715 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:42.722 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:35:42.722 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:35:42.753 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:35:42.753 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:35:42.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:42.757 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:42.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:42.859 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:42.895 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:42.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:42.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:42.997 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:43.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:43.032 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:43.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:43.034 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 9 16 27314 27298 189.544 144.375 0.00619486 0.0052707 2024-09-13T14:35:43.133 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:43.166 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:43.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:43.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:43.268 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:43.304 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:43.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:43.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:43.406 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:43.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:43 smithi102 ceph-mon[110183]: pgmap v670: 89 pgs: 89 active+clean; 732 MiB data, 4.0 GiB used, 443 GiB / 447 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-09-13T14:35:43.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:43 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1184551642' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:35:43.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:43 smithi102 ceph-mon[115269]: pgmap v670: 89 pgs: 89 active+clean; 732 MiB data, 4.0 GiB used, 443 GiB / 447 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-09-13T14:35:43.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:43 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1184551642' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:35:43.428 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:43.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:43.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:43.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:43 smithi195 ceph-mon[104262]: pgmap v670: 89 pgs: 89 active+clean; 732 MiB data, 4.0 GiB used, 443 GiB / 447 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-09-13T14:35:43.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:43 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1184551642' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:35:43.530 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:43.564 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:43.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:43.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:43.666 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:43.691 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:43.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:43.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:43.793 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:43.813 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:43.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:43.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:43.915 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:43.936 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:43.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:43.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:44.035 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 10 16 28640 28624 178.877 82.875 0.00593531 0.00558571 2024-09-13T14:35:44.038 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:44.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:44.075 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:44.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:44.176 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:44.197 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:44.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:44.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:44.299 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:44.330 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:44.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:44.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:44.361 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:35:44.361 INFO:teuthology.orchestra.run.smithi195.stdout: "success": "" 2024-09-13T14:35:44.361 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:35:44.361 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:44 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/90748644' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:35:44.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:44 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/90748644' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:35:44.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:44 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/90748644' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:35:44.433 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:44.467 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:44.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:44.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:44.569 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:44.604 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:44.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:44.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:44.707 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:44.747 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:44.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:44.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:44.849 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:44.884 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:44.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:44.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:44.986 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:35:45.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:45.024 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:45.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:35:45.035 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 11 16 30515 30499 173.268 117.188 0.00444316 0.00576658 2024-09-13T14:35:45.125 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:35:45.162 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:45.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:45.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:35:45.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:45 smithi195 ceph-mon[104262]: pgmap v671: 89 pgs: 89 active+clean; 1.6 GiB data, 6.7 GiB used, 440 GiB / 447 GiB avail; 153 MiB/s wr, 2.45k op/s 2024-09-13T14:35:45.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:45 smithi195 ceph-mon[104262]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:35:45.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:45 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/90748644' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:35:45.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:45 smithi195 ceph-mon[104262]: osdmap e210: 8 total, 8 up, 5 in 2024-09-13T14:35:45.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:45 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:35:45.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:45 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:35:45.264 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:45.288 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:45.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:45.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:45.390 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:45.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:45 smithi102 ceph-mon[110183]: pgmap v671: 89 pgs: 89 active+clean; 1.6 GiB data, 6.7 GiB used, 440 GiB / 447 GiB avail; 153 MiB/s wr, 2.45k op/s 2024-09-13T14:35:45.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:45 smithi102 ceph-mon[110183]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:35:45.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:45 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/90748644' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:35:45.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:45 smithi102 ceph-mon[110183]: osdmap e210: 8 total, 8 up, 5 in 2024-09-13T14:35:45.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:45 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:35:45.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:45 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:35:45.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:45 smithi102 ceph-mon[115269]: pgmap v671: 89 pgs: 89 active+clean; 1.6 GiB data, 6.7 GiB used, 440 GiB / 447 GiB avail; 153 MiB/s wr, 2.45k op/s 2024-09-13T14:35:45.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:45 smithi102 ceph-mon[115269]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:35:45.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:45 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/90748644' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:35:45.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:45 smithi102 ceph-mon[115269]: osdmap e210: 8 total, 8 up, 5 in 2024-09-13T14:35:45.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:45 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:35:45.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:45 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:35:45.422 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:45.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:45.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:45.524 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:45.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:45.559 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:45.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:45.660 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:45.695 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:45.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:45.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:45.797 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:45.832 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:45.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:45.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:45.936 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:45.966 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:45.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:45.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:46.035 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 12 16 32594 32578 169.656 129.938 0.00534051 0.00588933 2024-09-13T14:35:46.069 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:46.098 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:46.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:46.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:46.200 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:46.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:46.235 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:46.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:46.336 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:46.371 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:46.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:46.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:46.473 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:46.494 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:46.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:46.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:46.596 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:46.631 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:46.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:46.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:46.733 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:46.756 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:46.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:46.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:46.858 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:46.892 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:46.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:46.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:46.918 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:35:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:35:46] "GET /metrics HTTP/1.1" 200 36519 "" "Prometheus/2.51.0" 2024-09-13T14:35:46.995 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:47.016 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:47.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:47.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:47.035 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 13 16 34186 34170 164.259 99.5 0.00469878 0.0060484 2024-09-13T14:35:47.117 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:35:47.139 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:47.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:47.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:35:47.241 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:47.276 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:47.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:47.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:47.378 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:47.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:47.415 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:47.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:47.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:47 smithi102 ceph-mon[110183]: pgmap v673: 89 pgs: 89 active+clean; 1.6 GiB data, 6.7 GiB used, 440 GiB / 447 GiB avail; 159 MiB/s wr, 2.54k op/s 2024-09-13T14:35:47.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:47 smithi102 ceph-mon[115269]: pgmap v673: 89 pgs: 89 active+clean; 1.6 GiB data, 6.7 GiB used, 440 GiB / 447 GiB avail; 159 MiB/s wr, 2.54k op/s 2024-09-13T14:35:47.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:47 smithi195 ceph-mon[104262]: pgmap v673: 89 pgs: 89 active+clean; 1.6 GiB data, 6.7 GiB used, 440 GiB / 447 GiB avail; 159 MiB/s wr, 2.54k op/s 2024-09-13T14:35:47.516 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:47.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:47.551 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:47.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:47.652 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:47.689 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:47.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:47.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:47.791 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:47.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:47.817 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:47.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:47.918 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:47.956 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:47.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:47.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:48.035 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 14 16 36095 36079 161.048 119.312 0.0063471 0.00620491 2024-09-13T14:35:48.058 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:48.090 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:48.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:48.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:48.192 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:48.213 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:48.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:48.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:48.315 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:48.351 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:48.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:48.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:48.453 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:48.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:48.484 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:48.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:48.585 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:48.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:48.625 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:48.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:48.726 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:48.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:48.760 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:48.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:48.861 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:48.882 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:48.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:48.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:48.984 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:49.018 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:49.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:49.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:49.035 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 15 16 37807 37791 157.442 107 0.00691723 0.00634651 2024-09-13T14:35:49.120 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:35:49.144 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:49.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:49.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:35:49.246 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:49.280 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:49.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:49.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:49.382 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:49.418 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:49.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:49 smithi102 ceph-mon[110183]: pgmap v674: 89 pgs: 89 active+clean; 2.0 GiB data, 8.2 GiB used, 439 GiB / 447 GiB avail; 146 MiB/s wr, 2.33k op/s 2024-09-13T14:35:49.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:49 smithi102 ceph-mon[115269]: pgmap v674: 89 pgs: 89 active+clean; 2.0 GiB data, 8.2 GiB used, 439 GiB / 447 GiB avail; 146 MiB/s wr, 2.33k op/s 2024-09-13T14:35:49.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:49.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:49.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:49 smithi195 ceph-mon[104262]: pgmap v674: 89 pgs: 89 active+clean; 2.0 GiB data, 8.2 GiB used, 439 GiB / 447 GiB avail; 146 MiB/s wr, 2.33k op/s 2024-09-13T14:35:49.520 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:49.523 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is set 2024-09-13T14:35:49.556 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:49.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:49.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:49.658 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:49.692 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:49.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:49.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:49.794 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:49.828 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:49.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:49.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:49.930 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:49.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:49.962 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:49.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:50.028 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 6, 7] out_osds: [1, 2, 5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:35:50.028 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:35:50.028 INFO:tasks.thrashosds.thrasher:Killing osd 6, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:35:50.028 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-13T14:35:50.029 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ceph-08cee502-71d9-11ef-bceb-c7b262605968@osd.6 2024-09-13T14:35:50.035 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 16 16 39769 39753 155.265 122.625 0.00768967 0.00643599 2024-09-13T14:35:50.064 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:50.114 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:50.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:50.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:50.216 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:50.249 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:50.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:50.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:50.350 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:50.386 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:50.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:50.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:50.488 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:50.497 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 14:35:50 smithi195 systemd[1]: Stopping Ceph osd.6 for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T14:35:50.519 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:50.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:50.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:50.621 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:50.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:50.657 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:50.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:50.758 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:50.788 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:50.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:50.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:50.890 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:50.924 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:50.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:50.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:50.997 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 14:35:50 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[50503]: 2024-09-13T14:35:50.636+0000 7fb39ea2c640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T14:35:50.997 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 14:35:50 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[50503]: 2024-09-13T14:35:50.636+0000 7fb39ea2c640 -1 osd.6 210 *** Got signal Terminated *** 2024-09-13T14:35:50.997 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 14:35:50 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[50503]: 2024-09-13T14:35:50.636+0000 7fb39ea2c640 -1 osd.6 210 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T14:35:51.026 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:51.035 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 17 16 41881 41865 153.895 132 0.00926638 0.00649338 2024-09-13T14:35:51.061 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:51.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:51.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:51.163 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:51.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:51 smithi102 ceph-mon[110183]: pgmap v675: 89 pgs: 89 active+clean; 2.1 GiB data, 8.5 GiB used, 439 GiB / 447 GiB avail; 143 MiB/s wr, 2.28k op/s 2024-09-13T14:35:51.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:51 smithi102 ceph-mon[110183]: osd.6 marked itself down and dead 2024-09-13T14:35:51.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:51 smithi102 ceph-mon[115269]: pgmap v675: 89 pgs: 89 active+clean; 2.1 GiB data, 8.5 GiB used, 439 GiB / 447 GiB avail; 143 MiB/s wr, 2.28k op/s 2024-09-13T14:35:51.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:51 smithi102 ceph-mon[115269]: osd.6 marked itself down and dead 2024-09-13T14:35:51.184 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:51.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:51.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:51.286 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:51.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:51.307 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:51.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:51.408 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:51.430 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:51.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:51.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:51.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:51 smithi195 ceph-mon[104262]: pgmap v675: 89 pgs: 89 active+clean; 2.1 GiB data, 8.5 GiB used, 439 GiB / 447 GiB avail; 143 MiB/s wr, 2.28k op/s 2024-09-13T14:35:51.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:51 smithi195 ceph-mon[104262]: osd.6 marked itself down and dead 2024-09-13T14:35:51.532 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:51.553 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:51.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:51.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:51.655 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:51.676 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:51.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:51.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:51.778 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:51.800 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:51.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:51.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:51.902 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:51.923 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:51.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:51.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:51.997 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 14:35:51 smithi195 podman[160988]: 2024-09-13 14:35:51.71778693 +0000 UTC m=+1.291709813 container died 9cd070a6dd3d019fafaa81d4923267130bc6086284eec82df65444f313614369 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , RELEASE=reef-2c01458, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-13T14:35:52.025 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:52.035 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 18 16 42015 41999 145.811 8.375 0.00226316 0.00649827 2024-09-13T14:35:52.047 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:52.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:52.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:52.149 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:52.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:52 smithi102 ceph-mon[110183]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:35:52.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:52 smithi102 ceph-mon[110183]: osdmap e211: 8 total, 7 up, 5 in 2024-09-13T14:35:52.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:52 smithi102 ceph-mon[110183]: osdmap e212: 8 total, 7 up, 5 in 2024-09-13T14:35:52.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:52 smithi102 ceph-mon[115269]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:35:52.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:52 smithi102 ceph-mon[115269]: osdmap e211: 8 total, 7 up, 5 in 2024-09-13T14:35:52.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:52 smithi102 ceph-mon[115269]: osdmap e212: 8 total, 7 up, 5 in 2024-09-13T14:35:52.176 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:52.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:52.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:52.278 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:52.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:52.319 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:52.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:52.420 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:52.449 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:52.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:52.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:52.492 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:52 smithi195 ceph-mon[104262]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:35:52.492 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:52 smithi195 ceph-mon[104262]: osdmap e211: 8 total, 7 up, 5 in 2024-09-13T14:35:52.492 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:52 smithi195 ceph-mon[104262]: osdmap e212: 8 total, 7 up, 5 in 2024-09-13T14:35:52.492 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 14:35:52 smithi195 podman[160988]: 2024-09-13 14:35:52.180947755 +0000 UTC m=+1.754870647 container cleanup 9cd070a6dd3d019fafaa81d4923267130bc6086284eec82df65444f313614369 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, RELEASE=reef-2c01458, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-13T14:35:52.492 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 14:35:52 smithi195 bash[160988]: ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6 2024-09-13T14:35:52.551 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:52.581 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:52.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:52.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:52.683 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:52.726 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:52.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:52.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:52.747 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 14:35:52 smithi195 podman[161090]: 2024-09-13 14:35:52.491285501 +0000 UTC m=+0.744887968 container remove 9cd070a6dd3d019fafaa81d4923267130bc6086284eec82df65444f313614369 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=reef-2c01458, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-13T14:35:52.828 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:52.863 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:52.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:52.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:52.965 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:52.996 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:52.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:52.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:53.036 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 19 16 44039 44023 144.794 126.5 0.0083854 0.00690154 2024-09-13T14:35:53.098 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:53.134 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:53.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:53.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:53.197 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 14:35:52 smithi195 podman[161228]: 2024-09-13 14:35:52.814928647 +0000 UTC m=+0.027125870 image pull c129f5a2fd4eccce3e6eec43830eff1929af98bc48f3b850cf144ce5b326e8b5 quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11 2024-09-13T14:35:53.197 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 14:35:52 smithi195 podman[161228]: 2024-09-13 14:35:52.950808642 +0000 UTC m=+0.163005837 container create 8b936965f1fbabb43db636fddd8429db9bacf705aa016afbf27e82f5cb3e9975 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=reef-2c01458, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=) 2024-09-13T14:35:53.197 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:53 smithi195 ceph-mon[104262]: pgmap v677: 89 pgs: 89 active+clean; 2.1 GiB data, 8.5 GiB used, 439 GiB / 447 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-09-13T14:35:53.236 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:53.264 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:53.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:53.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:53.291 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:35:53.366 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:53.412 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:53.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:53.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:53.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:53 smithi102 ceph-mon[110183]: pgmap v677: 89 pgs: 89 active+clean; 2.1 GiB data, 8.5 GiB used, 439 GiB / 447 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-09-13T14:35:53.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:53 smithi102 ceph-mon[115269]: pgmap v677: 89 pgs: 89 active+clean; 2.1 GiB data, 8.5 GiB used, 439 GiB / 447 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-09-13T14:35:53.497 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 14:35:53 smithi195 podman[161228]: 2024-09-13 14:35:53.195687373 +0000 UTC m=+0.407884588 container init 8b936965f1fbabb43db636fddd8429db9bacf705aa016afbf27e82f5cb3e9975 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=reef-2c01458, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-13T14:35:53.498 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 14:35:53 smithi195 podman[161228]: 2024-09-13 14:35:53.204469036 +0000 UTC m=+0.416666235 container start 8b936965f1fbabb43db636fddd8429db9bacf705aa016afbf27e82f5cb3e9975 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-2c01458, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T14:35:53.498 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 14:35:53 smithi195 podman[161228]: 2024-09-13 14:35:53.320641204 +0000 UTC m=+0.532838406 container attach 8b936965f1fbabb43db636fddd8429db9bacf705aa016afbf27e82f5cb3e9975 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=reef-2c01458, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T14:35:53.514 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:53.554 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:53.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:53.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:53.656 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:53.704 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:53.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:53.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:53.749 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:35:53.749 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:35:53.806 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:53.817 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 14:35:53 smithi195 podman[161228]: 2024-09-13 14:35:53.498292436 +0000 UTC m=+0.710489646 container died 8b936965f1fbabb43db636fddd8429db9bacf705aa016afbf27e82f5cb3e9975 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=reef-2c01458, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2) 2024-09-13T14:35:53.849 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:53.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:53.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:53.951 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:53.988 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:53.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:53.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:54.036 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:2024-09-13T14:35:54.036109+0000 min lat: 0.00114883 max lat: 1.02478 avg lat: 0.00695922 2024-09-13T14:35:54.036 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:35:54.036 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 20 16 45976 45960 143.607 121.062 0.0088836 0.00695922 2024-09-13T14:35:54.089 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:54.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:54.127 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:54.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:54.228 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:54.256 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:54.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:54.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:54.279 DEBUG:teuthology.orchestra.run.smithi195:> sudo pkill -f 'journalctl -f -n 0 -u ceph-08cee502-71d9-11ef-bceb-c7b262605968@osd.6.service' 2024-09-13T14:35:54.357 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:54.390 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T14:35:54.390 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-13T14:35:54.393 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:54.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:54.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:54.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:54 smithi102 ceph-mon[115269]: Health check failed: Degraded data redundancy: 18552/127074 objects degraded (14.599%), 13 pgs degraded (PG_DEGRADED) 2024-09-13T14:35:54.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:54 smithi102 ceph-mon[110183]: Health check failed: Degraded data redundancy: 18552/127074 objects degraded (14.599%), 13 pgs degraded (PG_DEGRADED) 2024-09-13T14:35:54.495 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:54.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:54 smithi195 ceph-mon[104262]: Health check failed: Degraded data redundancy: 18552/127074 objects degraded (14.599%), 13 pgs degraded (PG_DEGRADED) 2024-09-13T14:35:54.523 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:54.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:54.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:54.625 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:54.652 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:54.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:54.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:54.754 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:54.789 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:54.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:54.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:54.819 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T14:35:54.891 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:54.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:54.941 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:54.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:55.036 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 21 16 48044 48028 142.923 129.25 0.00541254 0.00699217 2024-09-13T14:35:55.042 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:55.066 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:55.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:55.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:55.168 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:55.182 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:35:55.194 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:55.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:55.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:55.213 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:35:55.296 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:55.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:55.338 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:55.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:55.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:55 smithi102 ceph-mon[115269]: pgmap v679: 89 pgs: 30 active+undersized, 17 peering, 13 active+undersized+degraded, 29 active+clean; 2.6 GiB data, 9.7 GiB used, 437 GiB / 447 GiB avail; 132 MiB/s wr, 2.12k op/s; 18552/127074 objects degraded (14.599%) 2024-09-13T14:35:55.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:55 smithi102 ceph-mon[110183]: pgmap v679: 89 pgs: 30 active+undersized, 17 peering, 13 active+undersized+degraded, 29 active+clean; 2.6 GiB data, 9.7 GiB used, 437 GiB / 447 GiB avail; 132 MiB/s wr, 2.12k op/s; 18552/127074 objects degraded (14.599%) 2024-09-13T14:35:55.439 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:55.464 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:55.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:55.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:55.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:55 smithi195 ceph-mon[104262]: pgmap v679: 89 pgs: 30 active+undersized, 17 peering, 13 active+undersized+degraded, 29 active+clean; 2.6 GiB data, 9.7 GiB used, 437 GiB / 447 GiB avail; 132 MiB/s wr, 2.12k op/s; 18552/127074 objects degraded (14.599%) 2024-09-13T14:35:55.567 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:55.629 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:55.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:55.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:55.653 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:35:55.679 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:35:55.731 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:55.782 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:55.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:55.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:55.884 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:55.911 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:55.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:55.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:56.013 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:56.036 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 22 16 49587 49571 140.809 96.4375 0.00603029 0.00709622 2024-09-13T14:35:56.054 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:56.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:56.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:56.156 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:56.183 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:56.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:56.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:56.285 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:56.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:56.323 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:56.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:56.424 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:56.470 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:56.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:56.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:56.572 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:56.601 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:56 smithi102 ceph-mon[110183]: pgmap v680: 89 pgs: 30 active+undersized, 17 peering, 13 active+undersized+degraded, 29 active+clean; 2.6 GiB data, 9.7 GiB used, 437 GiB / 447 GiB avail; 70 MiB/s wr, 1.13k op/s; 18552/127074 objects degraded (14.599%) 2024-09-13T14:35:56.601 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:35:56.601 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:35:56.601 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:56 smithi102 ceph-mon[115269]: pgmap v680: 89 pgs: 30 active+undersized, 17 peering, 13 active+undersized+degraded, 29 active+clean; 2.6 GiB data, 9.7 GiB used, 437 GiB / 447 GiB avail; 70 MiB/s wr, 1.13k op/s; 18552/127074 objects degraded (14.599%) 2024-09-13T14:35:56.602 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:35:56.602 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:35:56.602 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:35:56 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:35:56] "GET /metrics HTTP/1.1" 200 36528 "" "Prometheus/2.51.0" 2024-09-13T14:35:56.611 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:56.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:56.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:56.713 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:56.747 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:56.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:56 smithi195 ceph-mon[104262]: pgmap v680: 89 pgs: 30 active+undersized, 17 peering, 13 active+undersized+degraded, 29 active+clean; 2.6 GiB data, 9.7 GiB used, 437 GiB / 447 GiB avail; 70 MiB/s wr, 1.13k op/s; 18552/127074 objects degraded (14.599%) 2024-09-13T14:35:56.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:35:56.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:35:56.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:56.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:56.849 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:56.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:56.893 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:56.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:56.993 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:57.020 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:57.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:57.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:57.037 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 23 16 51196 51180 139.054 100.562 0.0113467 0.00718655 2024-09-13T14:35:57.121 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:57.146 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:57.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:57.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:57.248 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:57.290 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:57.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:57.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:57.392 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:57.418 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:57.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:57.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:57.520 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:35:57.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:57.560 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:57.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:35:57.661 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:57.682 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:57.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:57.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:57.784 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:57.818 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:57.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:57.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:57.920 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:57.945 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:57.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:57.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:58.037 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 24 16 52617 52601 136.961 88.8125 0.01125 0.00729684 2024-09-13T14:35:58.046 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:58.077 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:58.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:58.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:58.179 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:58.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:58.220 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:58.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:58.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:57 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:35:58.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:57 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:35:58.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:57 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:35:58.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:57 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:35:58.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:57 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:35:58.321 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:58.350 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:58.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:58.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:58.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:57 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:35:58.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:57 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:35:58.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:57 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:35:58.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:57 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:35:58.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:57 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:35:58.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:57 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:35:58.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:57 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:35:58.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:57 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:35:58.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:57 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:35:58.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:57 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:35:58.451 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:58.481 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:58.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:58.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:58.583 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:58.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:58.618 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:58.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:58.719 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:58.747 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:58.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:58.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:58.849 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:58.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:58.875 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:58.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:58.976 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:35:59.010 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:59.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:59.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:35:59.037 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 25 16 54525 54509 136.252 119.25 0.0101445 0.00733521 2024-09-13T14:35:59.112 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:59.137 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:59.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:59.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:59.240 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:35:59.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:35:58 smithi195 ceph-mon[104262]: pgmap v681: 89 pgs: 39 active+undersized, 21 active+undersized+degraded, 29 active+clean; 3.1 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 124 MiB/s wr, 1.98k op/s; 31023/150939 objects degraded (20.553%) 2024-09-13T14:35:59.274 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:59.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:59.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:35:59.376 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:35:59.391 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 6, 7] out_osds: [1, 2, 5] dead_osds: [6] live_osds: [0, 1, 2, 3, 4, 5, 7] 2024-09-13T14:35:59.391 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:35:59.391 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-09-13T14:35:59.391 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd in 5 2024-09-13T14:35:59.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:59.397 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:35:59.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:35:59.417 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:35:58 smithi102 ceph-mon[110183]: pgmap v681: 89 pgs: 39 active+undersized, 21 active+undersized+degraded, 29 active+clean; 3.1 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 124 MiB/s wr, 1.98k op/s; 31023/150939 objects degraded (20.553%) 2024-09-13T14:35:59.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:35:58 smithi102 ceph-mon[115269]: pgmap v681: 89 pgs: 39 active+undersized, 21 active+undersized+degraded, 29 active+clean; 3.1 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 124 MiB/s wr, 1.98k op/s; 31023/150939 objects degraded (20.553%) 2024-09-13T14:35:59.498 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:35:59.524 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:59.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:59.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:35:59.626 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:59.653 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:59.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:59.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:35:59.755 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:35:59.782 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:59.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:59.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:35:59.884 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:35:59.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:35:59.911 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:35:59.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:00.012 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:00.037 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 26 16 56153 56137 134.925 101.75 0.0160086 0.00740714 2024-09-13T14:36:00.039 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:00.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:00.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:00.141 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:00.173 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:00.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:00.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:00.275 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:00.302 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:00.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:00.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:00.403 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:00.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:00 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 32147/154311 objects degraded (20.833%), 21 pgs degraded (PG_DEGRADED) 2024-09-13T14:36:00.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:00 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 32147/154311 objects degraded (20.833%), 21 pgs degraded (PG_DEGRADED) 2024-09-13T14:36:00.443 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:00.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:00.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:00.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:00 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 32147/154311 objects degraded (20.833%), 21 pgs degraded (PG_DEGRADED) 2024-09-13T14:36:00.546 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:00.571 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:00.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:00.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:00.673 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:00.697 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:00.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:00.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:00.799 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:00.835 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:00.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:00.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:00.937 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:00.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:00.985 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:00.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:01.037 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 27 16 57978 57962 134.151 114.062 0.00237504 0.00744525 2024-09-13T14:36:01.086 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:01.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:01.124 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:01.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:01.225 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:01.261 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:01.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:01.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:01.362 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:01.398 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:01.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:01.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:01.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:01 smithi102 ceph-mon[110183]: pgmap v682: 89 pgs: 39 active+undersized, 21 active+undersized+degraded, 29 active+clean; 3.1 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 128 MiB/s wr, 2.04k op/s; 32147/154311 objects degraded (20.833%) 2024-09-13T14:36:01.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:01 smithi102 ceph-mon[115269]: pgmap v682: 89 pgs: 39 active+undersized, 21 active+undersized+degraded, 29 active+clean; 3.1 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 128 MiB/s wr, 2.04k op/s; 32147/154311 objects degraded (20.833%) 2024-09-13T14:36:01.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:01 smithi195 ceph-mon[104262]: pgmap v682: 89 pgs: 39 active+undersized, 21 active+undersized+degraded, 29 active+clean; 3.1 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 128 MiB/s wr, 2.04k op/s; 32147/154311 objects degraded (20.833%) 2024-09-13T14:36:01.499 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:01.532 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:01.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:01.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:01.634 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:01.668 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:01.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:01.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:01.770 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:01.805 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:01.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:01.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:01.907 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:01.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:01.943 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:01.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:02.037 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 28 15 59591 59576 132.963 100.875 0.00559659 0.00751686 2024-09-13T14:36:02.044 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:02.080 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:02.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:02.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:02.182 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:02.217 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:02.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:02.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:02.319 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:02.346 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:02.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:02.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:02.448 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:02.473 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:02.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:02.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:02.575 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:02.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:02.613 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:02.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:02.714 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:02.743 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:02.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:02.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:02.845 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:02.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:02.867 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:02.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:02.968 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:03.005 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:03.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:03.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:03.037 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 29 16 61174 61158 131.787 98.875 0.00785808 0.00757626 2024-09-13T14:36:03.107 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:03.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:03.137 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:03.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:03.238 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:03.269 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:03.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:03.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:03.371 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:03.396 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:03.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:03.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:03.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:03 smithi102 ceph-mon[110183]: pgmap v683: 89 pgs: 39 active+undersized, 21 active+undersized+degraded, 29 active+clean; 3.1 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 106 MiB/s wr, 1.70k op/s; 32147/154311 objects degraded (20.833%) 2024-09-13T14:36:03.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:03 smithi102 ceph-mon[115269]: pgmap v683: 89 pgs: 39 active+undersized, 21 active+undersized+degraded, 29 active+clean; 3.1 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 106 MiB/s wr, 1.70k op/s; 32147/154311 objects degraded (20.833%) 2024-09-13T14:36:03.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:03 smithi195 ceph-mon[104262]: pgmap v683: 89 pgs: 39 active+undersized, 21 active+undersized+degraded, 29 active+clean; 3.1 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 106 MiB/s wr, 1.70k op/s; 32147/154311 objects degraded (20.833%) 2024-09-13T14:36:03.497 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:03.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:03.536 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:03.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:03.637 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:03.679 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:03.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:03.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:03.781 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:03.825 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:03.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:03.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:03.927 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:03.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:03.975 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:03.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:04.037 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 30 16 62844 62828 130.873 104.375 0.00771003 0.00763684 2024-09-13T14:36:04.076 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:04.103 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:04.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:04.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:04.205 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:04.230 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:04.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:04.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:04.332 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:04.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:04.373 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:04.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:04.474 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:04.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:04.521 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:04.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:04.623 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:04.644 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:04.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:04.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:04.746 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:04.786 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:04.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:04.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:04.888 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:04.911 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:04.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:04.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:05.013 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:05.037 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 31 16 64265 64249 129.516 88.8125 0.0096029 0.00771603 2024-09-13T14:36:05.045 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:05.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:05.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:05.049 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is unset 2024-09-13T14:36:05.147 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:05.168 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:05.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:05.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:05.270 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:05.297 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:05.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:05.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:05.328 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:05 smithi102 ceph-mon[110183]: pgmap v684: 89 pgs: 39 active+undersized, 21 active+undersized+degraded, 29 active+clean; 3.7 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 121 MiB/s wr, 1.94k op/s; 37432/179664 objects degraded (20.834%) 2024-09-13T14:36:05.329 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:05 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1845438068' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:36:05.329 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:05 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3255873620' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:36:05.329 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:05 smithi102 ceph-mon[115269]: pgmap v684: 89 pgs: 39 active+undersized, 21 active+undersized+degraded, 29 active+clean; 3.7 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 121 MiB/s wr, 1.94k op/s; 37432/179664 objects degraded (20.834%) 2024-09-13T14:36:05.329 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:05 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1845438068' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:36:05.329 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:05 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3255873620' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:36:05.398 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:05.419 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:05.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:05.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:05.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:05 smithi195 ceph-mon[104262]: pgmap v684: 89 pgs: 39 active+undersized, 21 active+undersized+degraded, 29 active+clean; 3.7 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 121 MiB/s wr, 1.94k op/s; 37432/179664 objects degraded (20.834%) 2024-09-13T14:36:05.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:05 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1845438068' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:36:05.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:05 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3255873620' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:36:05.521 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:05.553 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:05.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:05.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:05.655 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:05.690 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:05.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:05.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:05.791 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:05.826 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:05.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:05.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:05.927 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:05.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:05.965 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:05.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:06.039 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 32 16 65677 65661 128.223 88.25 0.00717178 0.00779346 2024-09-13T14:36:06.058 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:36:06.059 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:36:06.059 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:36:06.059 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:36:06.059 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:36:06.059 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:36:06.059 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:36:06.059 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:36:06.059 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:36:06.059 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:36:06.059 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:36:06.059 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:36:06.060 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:36:06.060 INFO:teuthology.orchestra.run.smithi102.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-13T14:36:06.060 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:36:06.066 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:06.093 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:06.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:06.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:06.195 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:06.229 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:06.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:06.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:06.331 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:06.356 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:06.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:06.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:06.381 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:06 smithi102 ceph-mon[110183]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:36:06.381 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:06 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3255873620' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:36:06.382 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:06 smithi102 ceph-mon[110183]: osdmap e213: 8 total, 7 up, 5 in 2024-09-13T14:36:06.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:06 smithi102 ceph-mon[115269]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:36:06.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:06 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3255873620' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:36:06.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:06 smithi102 ceph-mon[115269]: osdmap e213: 8 total, 7 up, 5 in 2024-09-13T14:36:06.457 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:06.484 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:06.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:06.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:06.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:06 smithi195 ceph-mon[104262]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:36:06.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:06 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3255873620' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:36:06.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:06 smithi195 ceph-mon[104262]: osdmap e213: 8 total, 7 up, 5 in 2024-09-13T14:36:06.586 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:06.624 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:06.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:06.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:06.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:36:06 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:36:06] "GET /metrics HTTP/1.1" 200 36528 "" "Prometheus/2.51.0" 2024-09-13T14:36:06.726 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:06.753 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:06.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:06.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:06.855 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:06.882 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:06.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:06.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:06.984 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:07.018 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:07.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:07.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:07.038 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 33 16 67361 67345 127.527 105.25 0.00955167 0.00783653 2024-09-13T14:36:07.119 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:07.131 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T14:36:07.132 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T14:36:07.156 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:07.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:07.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:07.258 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:07.296 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:07.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:07.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:07.311 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:36:07.346 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:07 smithi102 ceph-mon[115269]: pgmap v686: 89 pgs: 39 active+undersized, 21 active+undersized+degraded, 29 active+clean; 3.7 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 110 MiB/s wr, 1.75k op/s; 37432/179664 objects degraded (20.834%) 2024-09-13T14:36:07.346 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:07 smithi102 ceph-mon[110183]: pgmap v686: 89 pgs: 39 active+undersized, 21 active+undersized+degraded, 29 active+clean; 3.7 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 110 MiB/s wr, 1.75k op/s; 37432/179664 objects degraded (20.834%) 2024-09-13T14:36:07.398 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:07.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:07.436 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:07.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:07.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:07 smithi195 ceph-mon[104262]: pgmap v686: 89 pgs: 39 active+undersized, 21 active+undersized+degraded, 29 active+clean; 3.7 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 110 MiB/s wr, 1.75k op/s; 37432/179664 objects degraded (20.834%) 2024-09-13T14:36:07.537 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:07.578 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:07.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:07.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:07.654 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:36:07.654 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [], 2024-09-13T14:36:07.654 INFO:teuthology.orchestra.run.smithi102.stdout: "num_ops": 0 2024-09-13T14:36:07.654 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:36:07.680 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:07.706 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:07.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:07.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:07.808 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:07.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:07.845 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:07.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:07.946 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:07.971 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:07.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:07.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:08.039 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 34 16 68917 68901 126.636 97.25 0.00497001 0.00789209 2024-09-13T14:36:08.073 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:08.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:08.111 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:08.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:08.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:08 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 41356/200112 objects degraded (20.666%), 21 pgs degraded (PG_DEGRADED) 2024-09-13T14:36:08.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:08 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 41356/200112 objects degraded (20.666%), 21 pgs degraded (PG_DEGRADED) 2024-09-13T14:36:08.212 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:08.237 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:08.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:08.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:08.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:08 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 41356/200112 objects degraded (20.666%), 21 pgs degraded (PG_DEGRADED) 2024-09-13T14:36:08.339 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:08.368 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:08.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:08.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:08.470 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:08.505 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:08.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:08.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:08.607 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:08.639 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:08.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:08.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:08.702 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T14:36:08.741 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:08.787 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:08.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:08.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:08.889 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:08.942 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:08.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:08.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:09.039 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 35 16 70516 70500 125.873 99.9375 0.00480877 0.00794027 2024-09-13T14:36:09.043 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:09.068 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:09.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:09.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:09.132 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:36:09.170 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:09.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:09.223 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:09.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:09.324 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:09.380 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:09.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:09.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:09.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:09 smithi102 ceph-mon[110183]: pgmap v687: 89 pgs: 39 active+undersized, 21 active+undersized+degraded, 29 active+clean; 4.1 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 102 MiB/s wr, 1.64k op/s; 41356/200112 objects degraded (20.666%) 2024-09-13T14:36:09.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:09 smithi102 ceph-mon[115269]: pgmap v687: 89 pgs: 39 active+undersized, 21 active+undersized+degraded, 29 active+clean; 4.1 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 102 MiB/s wr, 1.64k op/s; 41356/200112 objects degraded (20.666%) 2024-09-13T14:36:09.481 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:09.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:09 smithi195 ceph-mon[104262]: pgmap v687: 89 pgs: 39 active+undersized, 21 active+undersized+degraded, 29 active+clean; 4.1 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 102 MiB/s wr, 1.64k op/s; 41356/200112 objects degraded (20.666%) 2024-09-13T14:36:09.518 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:09.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:09.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:09.620 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:09.644 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:09.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:09.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:09.746 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:09.773 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:09.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:09.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:09.875 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:09.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:09.914 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:09.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:10.015 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:10.039 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 36 16 72063 72047 125.062 96.6875 0.00596717 0.00799113 2024-09-13T14:36:10.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:10.047 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:10.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:10.148 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:10.157 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:36:10.168 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:36:10.200 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:10.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:10.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:10.302 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:10.328 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:10.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:10.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:10.429 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:10.471 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:10.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:10.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:10.574 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:10.607 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:10.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:10.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:10.709 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:10.748 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:10.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:10.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:10.850 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:10.885 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:10.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:10.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:10.986 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:11.021 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:11.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:11.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:11.039 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 37 16 73452 73436 124.028 86.8125 0.0218774 0.00805771 2024-09-13T14:36:11.123 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:11.148 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:11.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:11.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:11.250 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:11.278 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:11.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:11.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:11.379 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:11.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:11.415 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:11.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:11.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:11 smithi102 ceph-mon[110183]: pgmap v688: 89 pgs: 39 active+undersized, 21 active+undersized+degraded, 29 active+clean; 4.1 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 101 MiB/s wr, 1.62k op/s; 42298/202938 objects degraded (20.843%) 2024-09-13T14:36:11.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:11 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:36:11.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:11 smithi102 ceph-mon[115269]: pgmap v688: 89 pgs: 39 active+undersized, 21 active+undersized+degraded, 29 active+clean; 4.1 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 101 MiB/s wr, 1.62k op/s; 42298/202938 objects degraded (20.843%) 2024-09-13T14:36:11.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:11 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:36:11.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:11 smithi195 ceph-mon[104262]: pgmap v688: 89 pgs: 39 active+undersized, 21 active+undersized+degraded, 29 active+clean; 4.1 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 101 MiB/s wr, 1.62k op/s; 42298/202938 objects degraded (20.843%) 2024-09-13T14:36:11.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:11 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:36:11.517 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:11.543 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:11.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:11.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:11.609 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:36:11.645 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:11.704 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:11.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:11.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:11.806 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:11.829 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:11.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:11.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:11.931 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:11.954 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:11.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:11.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:12.039 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 38 16 74974 74958 123.267 95.125 0.00623373 0.00810763 2024-09-13T14:36:12.056 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:12.084 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:12.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:12.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:12.185 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:12.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:12.212 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:12.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:12.313 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:12.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:12.353 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:12.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:12.454 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:12.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:12.481 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:12.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:12.579 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:12 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-a[110159]: 2024-09-13T14:36:12.302+0000 7fa3e7f42640 -1 mon.a@0(leader).osd e213 definitely_dead 0 2024-09-13T14:36:12.582 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:12.607 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:12.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:12.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:12.708 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:12.747 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:12.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:12.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:12.849 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:12.876 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:12.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:12.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:12.978 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:13.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:13.013 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:13.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:13.039 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 39 15 76487 76472 122.533 94.625 0.00711389 0.00815639 2024-09-13T14:36:13.114 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:13.139 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:13.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:13.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:13.151 INFO:teuthology.orchestra.run.smithi102.stderr:marked in osd.5. 2024-09-13T14:36:13.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:13 smithi102 ceph-mon[115269]: pgmap v689: 89 pgs: 39 active+undersized, 21 active+undersized+degraded, 29 active+clean; 4.1 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 101 MiB/s wr, 1.62k op/s; 42298/202938 objects degraded (20.843%) 2024-09-13T14:36:13.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:13 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/990887796' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-13T14:36:13.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:13 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-13T14:36:13.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:13 smithi102 ceph-mon[110183]: pgmap v689: 89 pgs: 39 active+undersized, 21 active+undersized+degraded, 29 active+clean; 4.1 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 101 MiB/s wr, 1.62k op/s; 42298/202938 objects degraded (20.843%) 2024-09-13T14:36:13.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:13 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/990887796' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-13T14:36:13.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:13 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-13T14:36:13.241 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:13.262 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:13.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:13.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:13.364 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:13.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:13.386 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:13.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:13.486 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:13.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:13 smithi195 ceph-mon[104262]: pgmap v689: 89 pgs: 39 active+undersized, 21 active+undersized+degraded, 29 active+clean; 4.1 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 101 MiB/s wr, 1.62k op/s; 42298/202938 objects degraded (20.843%) 2024-09-13T14:36:13.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:13 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/990887796' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-13T14:36:13.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:13 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-13T14:36:13.508 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:13.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:13.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:13.610 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:13.631 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:13.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:13.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:13.733 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:13.754 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:13.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:13.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:13.856 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:13.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:13.884 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:13.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:13.985 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:14.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:14.007 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:14.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:14.039 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:2024-09-13T14:36:14.039559+0000 min lat: 0.00114883 max lat: 1.02478 avg lat: 0.00816553 2024-09-13T14:36:14.039 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:36:14.040 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 40 16 76702 76686 119.804 13.375 0.00280452 0.00816553 2024-09-13T14:36:14.108 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:14.129 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:14.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:14.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:14.231 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:14.253 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:14.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:14.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:14.355 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:14.377 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:14.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:14.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:14.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:14 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-09-13T14:36:14.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:14 smithi102 ceph-mon[110183]: osdmap e214: 8 total, 7 up, 6 in 2024-09-13T14:36:14.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:14 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 42298/202938 objects degraded (20.843%), 21 pgs degraded (PG_DEGRADED) 2024-09-13T14:36:14.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:14 smithi102 ceph-mon[110183]: osdmap e215: 8 total, 7 up, 6 in 2024-09-13T14:36:14.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:14 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-09-13T14:36:14.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:14 smithi102 ceph-mon[115269]: osdmap e214: 8 total, 7 up, 6 in 2024-09-13T14:36:14.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:14 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 42298/202938 objects degraded (20.843%), 21 pgs degraded (PG_DEGRADED) 2024-09-13T14:36:14.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:14 smithi102 ceph-mon[115269]: osdmap e215: 8 total, 7 up, 6 in 2024-09-13T14:36:14.435 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:36:14.479 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:14.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:14 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-09-13T14:36:14.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:14 smithi195 ceph-mon[104262]: osdmap e214: 8 total, 7 up, 6 in 2024-09-13T14:36:14.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:14 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 42298/202938 objects degraded (20.843%), 21 pgs degraded (PG_DEGRADED) 2024-09-13T14:36:14.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:14 smithi195 ceph-mon[104262]: osdmap e215: 8 total, 7 up, 6 in 2024-09-13T14:36:14.501 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:14.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:14.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:14.602 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:14.624 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:14.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:14.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:14.726 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:14.747 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:14.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:14.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:14.849 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:14.879 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:14.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:14.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:14.981 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:15.003 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:15.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:15.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:15.039 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 41 16 76740 76724 116.94 2.375 0.00333477 0.00832641 2024-09-13T14:36:15.105 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:15.115 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-09-13T14:36:15.137 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:15.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:15.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:15.239 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:15.267 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:15.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:15.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:15.336 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:15 smithi102 ceph-mon[110183]: pgmap v691: 89 pgs: 39 active+undersized, 21 active+undersized+degraded, 29 active+clean; 4.6 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 115 MiB/s wr, 1.84k op/s; 47068/225771 objects degraded (20.848%) 2024-09-13T14:36:15.337 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:15 smithi102 ceph-mon[110183]: osdmap e216: 8 total, 7 up, 6 in 2024-09-13T14:36:15.337 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:15 smithi102 ceph-mon[110183]: 7.a starting backfill to osd.5 from (0'0,0'0] MAX to 213'4846 2024-09-13T14:36:15.337 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:15 smithi102 ceph-mon[115269]: pgmap v691: 89 pgs: 39 active+undersized, 21 active+undersized+degraded, 29 active+clean; 4.6 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 115 MiB/s wr, 1.84k op/s; 47068/225771 objects degraded (20.848%) 2024-09-13T14:36:15.337 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:15 smithi102 ceph-mon[115269]: osdmap e216: 8 total, 7 up, 6 in 2024-09-13T14:36:15.337 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:15 smithi102 ceph-mon[115269]: 7.a starting backfill to osd.5 from (0'0,0'0] MAX to 213'4846 2024-09-13T14:36:15.368 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:15.394 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:15.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:15.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:15.496 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:15.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:15 smithi195 ceph-mon[104262]: pgmap v691: 89 pgs: 39 active+undersized, 21 active+undersized+degraded, 29 active+clean; 4.6 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 115 MiB/s wr, 1.84k op/s; 47068/225771 objects degraded (20.848%) 2024-09-13T14:36:15.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:15 smithi195 ceph-mon[104262]: osdmap e216: 8 total, 7 up, 6 in 2024-09-13T14:36:15.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:15 smithi195 ceph-mon[104262]: 7.a starting backfill to osd.5 from (0'0,0'0] MAX to 213'4846 2024-09-13T14:36:15.531 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:15.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:15.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:15.633 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:15.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:15.662 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:15.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:15.763 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:15.797 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:15.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:15.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:15.899 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:15.928 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:15.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:15.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:16.030 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:16.039 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 42 16 77874 77858 115.843 70.875 0.00860727 0.00862791 2024-09-13T14:36:16.069 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:16.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:16.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:16.171 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:16.205 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:16.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:16.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:16.306 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:16.334 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:16.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:16.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:16.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:16 smithi102 ceph-mon[110183]: 7.0 starting backfill to osd.5 from (0'0,0'0] MAX to 213'4780 2024-09-13T14:36:16.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:16 smithi102 ceph-mon[110183]: osdmap e217: 8 total, 7 up, 6 in 2024-09-13T14:36:16.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:16 smithi102 ceph-mon[115269]: 7.0 starting backfill to osd.5 from (0'0,0'0] MAX to 213'4780 2024-09-13T14:36:16.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:16 smithi102 ceph-mon[115269]: osdmap e217: 8 total, 7 up, 6 in 2024-09-13T14:36:16.436 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:16.469 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:16.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:16.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:16.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:16 smithi195 ceph-mon[104262]: 7.0 starting backfill to osd.5 from (0'0,0'0] MAX to 213'4780 2024-09-13T14:36:16.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:16 smithi195 ceph-mon[104262]: osdmap e217: 8 total, 7 up, 6 in 2024-09-13T14:36:16.571 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:16.614 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:16.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:16.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:16.716 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:16.737 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:36:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:36:16] "GET /metrics HTTP/1.1" 200 36531 "" "Prometheus/2.51.0" 2024-09-13T14:36:16.764 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:16.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:16.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:16.866 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:16.913 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:16.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:16.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:17.015 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:17.039 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 43 16 79073 79057 114.892 74.9375 0.0167769 0.0086984 2024-09-13T14:36:17.058 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:17.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:17.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:17.160 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:17.187 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:17.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:17.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:17.285 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:36:17.285 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:36:17.285 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:17 smithi102 ceph-mon[110183]: pgmap v694: 89 pgs: 39 active+undersized, 21 active+undersized+degraded, 29 active+clean; 4.6 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 79 MiB/s wr, 1.27k op/s; 47068/225771 objects degraded (20.848%) 2024-09-13T14:36:17.285 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:17 smithi102 ceph-mon[110183]: osdmap e218: 8 total, 7 up, 6 in 2024-09-13T14:36:17.286 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:17 smithi102 ceph-mon[115269]: pgmap v694: 89 pgs: 39 active+undersized, 21 active+undersized+degraded, 29 active+clean; 4.6 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 79 MiB/s wr, 1.27k op/s; 47068/225771 objects degraded (20.848%) 2024-09-13T14:36:17.286 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:17 smithi102 ceph-mon[115269]: osdmap e218: 8 total, 7 up, 6 in 2024-09-13T14:36:17.288 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:17.317 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:17.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:17.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:17.319 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:36:17.320 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:36:17.354 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:36:17.354 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:36:17.388 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:36:17.388 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:36:17.419 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:17.426 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:36:17.427 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:36:17.445 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:17.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:17.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:17.462 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:36:17.462 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:36:17.466 INFO:teuthology.orchestra.run.smithi102.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-13T14:36:17.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:17 smithi195 ceph-mon[104262]: pgmap v694: 89 pgs: 39 active+undersized, 21 active+undersized+degraded, 29 active+clean; 4.6 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 79 MiB/s wr, 1.27k op/s; 47068/225771 objects degraded (20.848%) 2024-09-13T14:36:17.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:17 smithi195 ceph-mon[104262]: osdmap e218: 8 total, 7 up, 6 in 2024-09-13T14:36:17.511 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:36:17.511 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:36:17.547 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:17.576 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:17.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:17.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:17.678 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:17.706 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:17.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:17.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:17.808 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:17.835 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:17.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:17.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:17.881 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:36:17.881 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [], 2024-09-13T14:36:17.881 INFO:teuthology.orchestra.run.smithi102.stdout: "complaint_time": 30, 2024-09-13T14:36:17.881 INFO:teuthology.orchestra.run.smithi102.stdout: "num_blocked_ops": 0 2024-09-13T14:36:17.881 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:36:17.937 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:17.972 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:17.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:17.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:18.039 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 44 16 80206 80190 113.89 70.8125 0.00560854 0.00877503 2024-09-13T14:36:18.074 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:18.107 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:18.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:18.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:18.191 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is unset 2024-09-13T14:36:18.209 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:18.234 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:18.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:18.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:18.336 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:18.365 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:18.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:18.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:18.466 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:18.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:18 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3513151689' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:36:18.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:18 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/941070013' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:36:18.501 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:18.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:18.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:18.603 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:18.638 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:18.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:18.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:18.662 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:18 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3513151689' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:36:18.662 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:18 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/941070013' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:36:18.662 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:18 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3513151689' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:36:18.663 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:18 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/941070013' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:36:18.700 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T14:36:18.740 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:18.784 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:18.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:18.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:18.886 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:18.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:18.923 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:18.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:19.024 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:19.040 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 45 16 81398 81382 113.014 74.5 0.0109346 0.00884355 2024-09-13T14:36:19.050 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:19.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:19.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:19.153 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:19.179 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:19.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:19.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:19.281 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:19.317 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:19.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:19.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:19.418 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:19.452 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:19.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:19.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:19.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:19 smithi195 ceph-mon[104262]: pgmap v697: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 16 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 20 active+undersized, 11 active+undersized+degraded, 35 active+clean; 4.8 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 58 MiB/s wr, 933 op/s; 58894/236973 objects degraded (24.853%); 14688/236973 objects misplaced (6.198%); 1.9 MiB/s, 19 objects/s recovering 2024-09-13T14:36:19.501 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:19 smithi195 ceph-mon[104262]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:36:19.501 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:19 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/941070013' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:36:19.501 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:19 smithi195 ceph-mon[104262]: osdmap e219: 8 total, 7 up, 6 in 2024-09-13T14:36:19.501 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:19 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 58894/236973 objects degraded (24.853%), 15 pgs degraded (PG_DEGRADED) 2024-09-13T14:36:19.553 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:19.567 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:19 smithi102 ceph-mon[110183]: pgmap v697: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 16 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 20 active+undersized, 11 active+undersized+degraded, 35 active+clean; 4.8 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 58 MiB/s wr, 933 op/s; 58894/236973 objects degraded (24.853%); 14688/236973 objects misplaced (6.198%); 1.9 MiB/s, 19 objects/s recovering 2024-09-13T14:36:19.567 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:19 smithi102 ceph-mon[110183]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:36:19.567 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:19 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/941070013' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:36:19.567 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:19 smithi102 ceph-mon[110183]: osdmap e219: 8 total, 7 up, 6 in 2024-09-13T14:36:19.567 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:19 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 58894/236973 objects degraded (24.853%), 15 pgs degraded (PG_DEGRADED) 2024-09-13T14:36:19.567 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:19 smithi102 ceph-mon[115269]: pgmap v697: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 16 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 20 active+undersized, 11 active+undersized+degraded, 35 active+clean; 4.8 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 58 MiB/s wr, 933 op/s; 58894/236973 objects degraded (24.853%); 14688/236973 objects misplaced (6.198%); 1.9 MiB/s, 19 objects/s recovering 2024-09-13T14:36:19.567 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:19 smithi102 ceph-mon[115269]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:36:19.567 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:19 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/941070013' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:36:19.568 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:19 smithi102 ceph-mon[115269]: osdmap e219: 8 total, 7 up, 6 in 2024-09-13T14:36:19.568 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:19 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 58894/236973 objects degraded (24.853%), 15 pgs degraded (PG_DEGRADED) 2024-09-13T14:36:19.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:19.593 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:19.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:19.610 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T14:36:19.611 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T14:36:19.694 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:19.723 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:19.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:19.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:19.824 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:19.856 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:19.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:19.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:19.958 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:19.991 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:19.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:19.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:20.040 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 46 16 82569 82553 112.149 73.1875 0.0038117 0.00891093 2024-09-13T14:36:20.093 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:20.116 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 6, 7, 5] out_osds: [1, 2] dead_osds: [6] live_osds: [0, 1, 2, 3, 4, 5, 7] 2024-09-13T14:36:20.116 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:36:20.117 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-09-13T14:36:20.117 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T14:36:20.117 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T14:36:20.117 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-09-13T14:36:20.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:20.120 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:20.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:20.222 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:20.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:20.256 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:20.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:20.357 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:20.358 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:36:20.389 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:20.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:20.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:20.412 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:36:20.491 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:20.521 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:20.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:20.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:20.623 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:20.669 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:20.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:20.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:20.771 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:20.801 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:20.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:20.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:20.903 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:20.936 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:20.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:20.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:21.037 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:21.040 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 47 16 83714 83698 111.285 71.5625 0.0144006 0.00898131 2024-09-13T14:36:21.072 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:21.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:21.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:21.174 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:21.216 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:21.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:21.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:21.318 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:21.354 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:21.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:21.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:21.456 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:21.494 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:21.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:21.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:21.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:21 smithi195 ceph-mon[104262]: pgmap v699: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 18 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 4.8 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 55 MiB/s wr, 874 op/s; 59142/237717 objects degraded (24.879%); 14688/237717 objects misplaced (6.179%); 1.7 MiB/s, 17 objects/s recovering 2024-09-13T14:36:21.596 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:21.611 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:36:21.627 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:21.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:21.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:21.647 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:21 smithi102 ceph-mon[110183]: pgmap v699: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 18 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 4.8 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 55 MiB/s wr, 874 op/s; 59142/237717 objects degraded (24.879%); 14688/237717 objects misplaced (6.179%); 1.7 MiB/s, 17 objects/s recovering 2024-09-13T14:36:21.648 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:21 smithi102 ceph-mon[115269]: pgmap v699: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 18 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 4.8 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 55 MiB/s wr, 874 op/s; 59142/237717 objects degraded (24.879%); 14688/237717 objects misplaced (6.179%); 1.7 MiB/s, 17 objects/s recovering 2024-09-13T14:36:21.729 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:21.769 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:21.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:21.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:21.871 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:21.923 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:21.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:21.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:22.025 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:22.040 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 48 16 84885 84869 110.491 73.1875 0.00857248 0.00904503 2024-09-13T14:36:22.057 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:22.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:22.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:22.160 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:22.190 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:22.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:22.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:22.292 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:22.297 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:36:22.326 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:22.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:22.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:22.428 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:22.456 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:22.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:22.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:22.475 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:36:22.531 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:36:22.531 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:36:22.558 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:22.606 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:22.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:22.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:22.708 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:22.735 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:22.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:22.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:22.837 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:22.869 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:22.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:22.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:22.971 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:23.005 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:23.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:23.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:23.040 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 49 16 85914 85898 109.548 64.3125 0.0193513 0.009123 2024-09-13T14:36:23.107 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:23.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:23.155 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:23.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:23.256 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:23.303 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:23.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:23.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:23.404 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:23.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:23.445 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:23.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:23.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:23 smithi195 ceph-mon[104262]: pgmap v700: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 18 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 4.8 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 41 MiB/s wr, 663 op/s; 59142/237717 objects degraded (24.879%); 14688/237717 objects misplaced (6.179%); 1.3 MiB/s, 13 objects/s recovering 2024-09-13T14:36:23.547 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:23.581 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:23.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:23.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:23.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:23 smithi102 ceph-mon[110183]: pgmap v700: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 18 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 4.8 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 41 MiB/s wr, 663 op/s; 59142/237717 objects degraded (24.879%); 14688/237717 objects misplaced (6.179%); 1.3 MiB/s, 13 objects/s recovering 2024-09-13T14:36:23.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:23 smithi102 ceph-mon[115269]: pgmap v700: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 18 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 4.8 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 41 MiB/s wr, 663 op/s; 59142/237717 objects degraded (24.879%); 14688/237717 objects misplaced (6.179%); 1.3 MiB/s, 13 objects/s recovering 2024-09-13T14:36:23.683 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:23.713 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:23.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:23.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:23.814 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:23.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:23.853 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:23.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:23.954 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:23.988 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:23.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:23.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:24.040 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 50 16 87057 87041 108.786 71.4375 0.00675379 0.00918653 2024-09-13T14:36:24.050 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:36:24.050 INFO:teuthology.orchestra.run.smithi195.stdout: "success": "" 2024-09-13T14:36:24.050 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:36:24.091 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:24.113 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:24.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:24.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:24.215 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:24.221 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T14:36:24.254 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:24.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:24.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:24.356 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:24.385 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:24.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:24.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:24.487 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:24.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:24 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 59142/237717 objects degraded (24.879%), 15 pgs degraded (PG_DEGRADED) 2024-09-13T14:36:24.523 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:24.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:24.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:24.624 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:24.644 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:24 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 59142/237717 objects degraded (24.879%), 15 pgs degraded (PG_DEGRADED) 2024-09-13T14:36:24.645 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:24 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 59142/237717 objects degraded (24.879%), 15 pgs degraded (PG_DEGRADED) 2024-09-13T14:36:24.651 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:24.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:24.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:24.752 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:36:24.752 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:24.807 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:24.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:24.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:24.858 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:36:24.909 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:24.952 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:24.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:24.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:25.040 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 51 16 88094 88078 107.924 64.8125 0.0148659 0.00925934 2024-09-13T14:36:25.054 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:25.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:25.089 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:25.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:25.190 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:25.217 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:25.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:25.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:25.319 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:25.347 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:25.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:25.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:25.449 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:25.477 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:25.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:25.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:25.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:25 smithi195 ceph-mon[104262]: pgmap v701: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 20 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 5.2 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 81 MiB/s wr, 1.29k op/s; 62793/255156 objects degraded (24.610%); 15287/255156 objects misplaced (5.991%); 2.0 MiB/s, 25 objects/s recovering 2024-09-13T14:36:25.579 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:25.602 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:25.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:25.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:25.608 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:25 smithi102 ceph-mon[110183]: pgmap v701: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 20 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 5.2 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 81 MiB/s wr, 1.29k op/s; 62793/255156 objects degraded (24.610%); 15287/255156 objects misplaced (5.991%); 2.0 MiB/s, 25 objects/s recovering 2024-09-13T14:36:25.608 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:25 smithi102 ceph-mon[115269]: pgmap v701: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 20 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 5.2 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 81 MiB/s wr, 1.29k op/s; 62793/255156 objects degraded (24.610%); 15287/255156 objects misplaced (5.991%); 2.0 MiB/s, 25 objects/s recovering 2024-09-13T14:36:25.705 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:25.736 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:25.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:25.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:25.838 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:25.876 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:25.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:25.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:25.978 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:26.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:26.007 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:26.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:26.040 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 52 16 89016 89000 106.956 57.625 0.0110216 0.00934496 2024-09-13T14:36:26.108 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:26.129 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:26.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:26.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:26.231 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:26.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:26.260 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:26.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:26.361 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:26.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:26.395 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:26.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:26.496 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:26.532 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:26.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:26.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:26.634 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:26.659 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:26.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:26.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:26.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:26 smithi102 ceph-mon[110183]: pgmap v702: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 20 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 5.2 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 72 MiB/s wr, 1.15k op/s; 62793/255156 objects degraded (24.610%); 15287/255156 objects misplaced (5.991%); 1.3 MiB/s, 21 objects/s recovering 2024-09-13T14:36:26.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:36:26.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:36:26.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:26 smithi102 ceph-mon[115269]: pgmap v702: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 20 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 5.2 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 72 MiB/s wr, 1.15k op/s; 62793/255156 objects degraded (24.610%); 15287/255156 objects misplaced (5.991%); 1.3 MiB/s, 21 objects/s recovering 2024-09-13T14:36:26.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:36:26.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:36:26.669 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:36:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:36:26] "GET /metrics HTTP/1.1" 200 36558 "" "Prometheus/2.51.0" 2024-09-13T14:36:26.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:26 smithi195 ceph-mon[104262]: pgmap v702: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 20 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 5.2 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 72 MiB/s wr, 1.15k op/s; 62793/255156 objects degraded (24.610%); 15287/255156 objects misplaced (5.991%); 1.3 MiB/s, 21 objects/s recovering 2024-09-13T14:36:26.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:36:26.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:36:26.761 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:26.789 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:26.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:26.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:26.892 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:26.917 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:26.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:26.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:27.019 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:27.041 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 53 16 89946 89930 106.035 58.125 0.00811807 0.00942505 2024-09-13T14:36:27.044 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:27.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:27.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:27.146 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:27.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:27.185 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:27.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:27.285 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:27.331 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:27.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:27.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:27.433 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:27.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:27.466 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:27.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:27.567 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:27.592 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:27.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:27.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:27.694 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:27.724 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:27.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:27.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:27.826 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:27.858 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:27.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:27.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:27.959 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:27.994 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:27.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:27.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:28.041 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 54 16 90952 90936 105.236 62.875 0.0324964 0.00949715 2024-09-13T14:36:28.096 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:28.126 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:28.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:28.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:28.229 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:28.251 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:28.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:28.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:28.353 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:28.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:28.401 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:28.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:28.502 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:28.540 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:28.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:28.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:28.628 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:28 smithi102 ceph-mon[110183]: pgmap v703: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 20 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 5.5 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 67 MiB/s wr, 1.08k op/s; 65628/269358 objects degraded (24.365%); 15784/269358 objects misplaced (5.860%); 1.5 MiB/s, 23 objects/s recovering 2024-09-13T14:36:28.628 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:28 smithi102 ceph-mon[115269]: pgmap v703: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 20 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 5.5 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 67 MiB/s wr, 1.08k op/s; 65628/269358 objects degraded (24.365%); 15784/269358 objects misplaced (5.860%); 1.5 MiB/s, 23 objects/s recovering 2024-09-13T14:36:28.642 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:28.662 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:28.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:28.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:28.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:28 smithi195 ceph-mon[104262]: pgmap v703: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 20 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 5.5 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 67 MiB/s wr, 1.08k op/s; 65628/269358 objects degraded (24.365%); 15784/269358 objects misplaced (5.860%); 1.5 MiB/s, 23 objects/s recovering 2024-09-13T14:36:28.764 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:28.795 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:28.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:28.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:28.897 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:28.938 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:28.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:28.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:29.040 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:29.041 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 55 16 91973 91957 104.482 63.8125 0.00765054 0.00956618 2024-09-13T14:36:29.066 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:29.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:29.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:29.168 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:29.198 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:29.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:29.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:29.299 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:29.339 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:29.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:29.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:29.385 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:36:29.385 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:36:29.385 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:36:29.385 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:36:29.385 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:36:29.385 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:36:29.386 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:36:29.386 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:36:29.386 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:36:29.386 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:36:29.386 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:36:29.386 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:36:29.386 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:36:29.386 INFO:teuthology.orchestra.run.smithi102.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-13T14:36:29.386 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:36:29.441 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:29.465 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:29.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:29.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:29.566 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:29.602 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:29.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:29.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:29.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:29 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1916501398' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:36:29.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:29 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 65628/269358 objects degraded (24.365%), 15 pgs degraded (PG_DEGRADED) 2024-09-13T14:36:29.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:29 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1916501398' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:36:29.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:29 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 65628/269358 objects degraded (24.365%), 15 pgs degraded (PG_DEGRADED) 2024-09-13T14:36:29.704 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:29.740 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:29.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:29.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:29.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:29 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1916501398' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:36:29.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:29 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 65628/269358 objects degraded (24.365%), 15 pgs degraded (PG_DEGRADED) 2024-09-13T14:36:29.841 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:29.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:29.870 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:29.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:29.971 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:30.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:30.012 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:30.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:30.041 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 56 16 93092 93076 103.865 69.9375 0.0112276 0.00962285 2024-09-13T14:36:30.113 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:30.141 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:30.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:30.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:30.243 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:30.272 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:30.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:30.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:30.374 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:30.403 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:30.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:30.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:30.505 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:30.541 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:30.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:30.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:30.558 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 6, 7, 5] out_osds: [1, 2] dead_osds: [6] live_osds: [0, 1, 2, 3, 4, 5, 7] 2024-09-13T14:36:30.558 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:36:30.558 INFO:tasks.thrashosds.thrasher:Reweighting osd 0 to 0.3110248273131276 2024-09-13T14:36:30.559 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd reweight 0 0.3110248273131276 2024-09-13T14:36:30.643 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:30.670 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:30.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:30.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:30.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:30 smithi195 ceph-mon[104262]: pgmap v704: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 20 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 5.5 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 61 MiB/s wr, 969 op/s; 65962/270360 objects degraded (24.398%); 15784/270360 objects misplaced (5.838%); 1.3 MiB/s, 21 objects/s recovering 2024-09-13T14:36:30.771 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:30.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:30.823 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:30.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:30.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:30 smithi102 ceph-mon[110183]: pgmap v704: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 20 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 5.5 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 61 MiB/s wr, 969 op/s; 65962/270360 objects degraded (24.398%); 15784/270360 objects misplaced (5.838%); 1.3 MiB/s, 21 objects/s recovering 2024-09-13T14:36:30.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:30 smithi102 ceph-mon[115269]: pgmap v704: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 20 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 5.5 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 61 MiB/s wr, 969 op/s; 65962/270360 objects degraded (24.398%); 15784/270360 objects misplaced (5.838%); 1.3 MiB/s, 21 objects/s recovering 2024-09-13T14:36:30.924 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:30.949 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:30.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:30.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:31.041 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 57 16 94078 94062 103.124 61.625 0.0138473 0.00969161 2024-09-13T14:36:31.050 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:31.075 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:31.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:31.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:31.176 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:31.218 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:31.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:31.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:31.319 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:31.365 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:31.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:31.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:31.468 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:31.494 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:31.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:31.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:31.596 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:31.631 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:31.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:31.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:31.733 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:31.761 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:31.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:31.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:31.863 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:31.889 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:31.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:31.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:31.907 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T14:36:31.907 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T14:36:31.991 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:32.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:32.034 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:32.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:32.041 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 58 16 95219 95203 102.576 71.3125 0.0121941 0.00974355 2024-09-13T14:36:32.136 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:32.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:32.166 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:32.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:32.267 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:32.293 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:32.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:32.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:32.395 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:32.430 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:32.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:32.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:32.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:32 smithi195 ceph-mon[104262]: pgmap v705: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 20 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 5.5 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 57 MiB/s wr, 906 op/s; 65962/270360 objects degraded (24.398%); 15784/270360 objects misplaced (5.838%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T14:36:32.497 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:36:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=cleanup t=2024-09-13T14:36:32.219903132Z level=info msg="Completed cleanup jobs" duration=22.983962ms 2024-09-13T14:36:32.497 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:36:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=grafana.update.checker t=2024-09-13T14:36:32.337868125Z level=info msg="Update check succeeded" duration=62.109374ms 2024-09-13T14:36:32.497 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:36:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=plugins.update.checker t=2024-09-13T14:36:32.344342082Z level=info msg="Update check succeeded" duration=66.204183ms 2024-09-13T14:36:32.532 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:32.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:32.568 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:32.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:32.669 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:32.701 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:32.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:32.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:32.802 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:32.811 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:32 smithi102 ceph-mon[110183]: pgmap v705: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 20 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 5.5 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 57 MiB/s wr, 906 op/s; 65962/270360 objects degraded (24.398%); 15784/270360 objects misplaced (5.838%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T14:36:32.812 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:32 smithi102 ceph-mon[115269]: pgmap v705: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 20 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 5.5 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 57 MiB/s wr, 906 op/s; 65962/270360 objects degraded (24.398%); 15784/270360 objects misplaced (5.838%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T14:36:32.830 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:32.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:32.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:32.932 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:32.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:32.974 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:32.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:33.041 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 59 16 96404 96388 102.092 74.0625 0.0133366 0.00978984 2024-09-13T14:36:33.076 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:33.107 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:33.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:33.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:33.208 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:33.235 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:33.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:33.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:33.336 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:33.366 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:33.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:33.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:33.468 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:33.502 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:33.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:33.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:33.603 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:33.629 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:33.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:33.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:33.731 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:33.757 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:33.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:33.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:33.858 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:33.889 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:33.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:33.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:33.909 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:36:33.991 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:34.017 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:34.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:34.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:34.041 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:2024-09-13T14:36:34.041522+0000 min lat: 0.00114883 max lat: 1.99103 avg lat: 0.00984896 2024-09-13T14:36:34.041 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:36:34.041 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 60 16 97451 97435 101.481 65.4375 0.005952 0.00984896 2024-09-13T14:36:34.119 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:34.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:34.168 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:34.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:34.269 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:34.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:34.307 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:34.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:34.408 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:34.441 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:34.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:34.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:34.543 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:34.566 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:34.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:34.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:34.667 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:34.714 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:34.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:34.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:34.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:34 smithi195 ceph-mon[104262]: pgmap v706: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 20 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 5.8 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 84 MiB/s wr, 1.35k op/s; 69276/286362 objects degraded (24.192%); 16333/286362 objects misplaced (5.704%); 1.8 MiB/s, 29 objects/s recovering 2024-09-13T14:36:34.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:34 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 69276/286362 objects degraded (24.192%), 15 pgs degraded (PG_DEGRADED) 2024-09-13T14:36:34.816 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:34.853 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:34.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:34.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:34.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:34 smithi102 ceph-mon[110183]: pgmap v706: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 20 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 5.8 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 84 MiB/s wr, 1.35k op/s; 69276/286362 objects degraded (24.192%); 16333/286362 objects misplaced (5.704%); 1.8 MiB/s, 29 objects/s recovering 2024-09-13T14:36:34.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:34 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 69276/286362 objects degraded (24.192%), 15 pgs degraded (PG_DEGRADED) 2024-09-13T14:36:34.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:34 smithi102 ceph-mon[115269]: pgmap v706: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 20 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 5.8 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 84 MiB/s wr, 1.35k op/s; 69276/286362 objects degraded (24.192%); 16333/286362 objects misplaced (5.704%); 1.8 MiB/s, 29 objects/s recovering 2024-09-13T14:36:34.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:34 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 69276/286362 objects degraded (24.192%), 15 pgs degraded (PG_DEGRADED) 2024-09-13T14:36:34.955 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:34.976 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:34.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:34.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:35.041 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 61 16 98453 98437 100.844 62.625 0.00464387 0.00990946 2024-09-13T14:36:35.079 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:35.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:35.112 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:35.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:35.213 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:35.247 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:35.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:35.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:35.349 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:35.388 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:35.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:35.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:35.490 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:35.517 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:35.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:35.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:35.619 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:35.649 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:35.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:35.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:35.750 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:35.787 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:35.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:35.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:35.889 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:35.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:35.929 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:35.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:36.030 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:36.041 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 62 16 99609 99593 100.383 72.25 0.0113105 0.00995646 2024-09-13T14:36:36.062 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:36.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:36.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:36.164 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:36.194 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:36.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:36.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:36.273 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:36:36.295 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:36.307 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:36:36.347 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:36.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:36.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:36.449 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:36.459 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is set 2024-09-13T14:36:36.489 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:36.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:36.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:36.591 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:36.621 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:36.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:36.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:36.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:36 smithi102 ceph-mon[110183]: pgmap v707: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 20 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 5.8 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 54 MiB/s wr, 866 op/s; 69276/286362 objects degraded (24.192%); 16333/286362 objects misplaced (5.704%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T14:36:36.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:36 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/57586958' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:36:36.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:36 smithi102 ceph-mon[115269]: pgmap v707: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 20 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 5.8 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 54 MiB/s wr, 866 op/s; 69276/286362 objects degraded (24.192%); 16333/286362 objects misplaced (5.704%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T14:36:36.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:36 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/57586958' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:36:36.669 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:36:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:36:36] "GET /metrics HTTP/1.1" 200 36558 "" "Prometheus/2.51.0" 2024-09-13T14:36:36.723 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:36.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:36 smithi195 ceph-mon[104262]: pgmap v707: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 20 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 5.8 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 54 MiB/s wr, 866 op/s; 69276/286362 objects degraded (24.192%); 16333/286362 objects misplaced (5.704%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T14:36:36.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:36 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/57586958' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:36:36.758 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:36.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:36.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:36.860 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:36.889 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:36.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:36.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:36.991 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:37.025 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:37.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:37.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:37.041 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 63 16 100531 100515 99.7042 57.625 0.0103678 0.0100244 2024-09-13T14:36:37.126 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:37.155 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:37.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:37.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:37.257 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:37.285 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:37.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:37.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:37.386 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:37.422 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:37.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:37.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:37.524 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:37.558 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:37.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:37.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:37.660 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:37.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:37.693 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:37.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:37.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:37 smithi195 ceph-mon[104262]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:36:37.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:37 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/57586958' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:36:37.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:37 smithi195 ceph-mon[104262]: osdmap e220: 8 total, 7 up, 6 in 2024-09-13T14:36:37.794 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:37.832 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:37.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:37.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:37.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:37 smithi102 ceph-mon[110183]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:36:37.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:37 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/57586958' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:36:37.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:37 smithi102 ceph-mon[110183]: osdmap e220: 8 total, 7 up, 6 in 2024-09-13T14:36:37.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:37 smithi102 ceph-mon[115269]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:36:37.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:37 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/57586958' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:36:37.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:37 smithi102 ceph-mon[115269]: osdmap e220: 8 total, 7 up, 6 in 2024-09-13T14:36:37.934 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:37.961 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:37.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:37.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:38.041 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 64 16 101480 101464 99.0731 59.3125 0.00769728 0.0100887 2024-09-13T14:36:38.063 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:38.093 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:38.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:38.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:38.195 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:38.235 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:38.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:38.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:38.337 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:38.364 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:38.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:38.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:38.466 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:38.508 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:38.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:38.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:38.610 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:38.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:38.658 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:38.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:38.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:38 smithi195 ceph-mon[104262]: pgmap v709: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 20 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 6.1 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 66 MiB/s wr, 1.06k op/s; 72257/301134 objects degraded (23.995%); 16918/301134 objects misplaced (5.618%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T14:36:38.759 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:38.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:38.805 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:38.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:38.818 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:38 smithi102 ceph-mon[110183]: pgmap v709: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 20 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 6.1 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 66 MiB/s wr, 1.06k op/s; 72257/301134 objects degraded (23.995%); 16918/301134 objects misplaced (5.618%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T14:36:38.818 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:38 smithi102 ceph-mon[115269]: pgmap v709: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 20 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 6.1 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 66 MiB/s wr, 1.06k op/s; 72257/301134 objects degraded (23.995%); 16918/301134 objects misplaced (5.618%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T14:36:38.906 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:38.951 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:38.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:38.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:39.042 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 65 16 102601 102585 98.6267 70.0625 0.00482339 0.0101345 2024-09-13T14:36:39.053 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:39.087 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:39.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:39.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:39.189 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:39.215 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:39.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:39.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:39.317 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:39.350 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:39.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:39.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:39.452 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:39.481 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:39.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:39.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:39.583 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:39.616 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:39.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:39.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:39.718 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:39.744 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:39.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:39.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:39.762 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:39 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 72257/301134 objects degraded (23.995%), 15 pgs degraded (PG_DEGRADED) 2024-09-13T14:36:39.763 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:39 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 72257/301134 objects degraded (23.995%), 15 pgs degraded (PG_DEGRADED) 2024-09-13T14:36:39.846 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:39.882 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:39.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:39.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:39.984 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:39.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:39 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 72257/301134 objects degraded (23.995%), 15 pgs degraded (PG_DEGRADED) 2024-09-13T14:36:40.031 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:40.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:40.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:40.042 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 66 16 103744 103728 98.2147 71.4375 0.022279 0.0101765 2024-09-13T14:36:40.132 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:40.165 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:40.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:40.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:40.267 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:40.299 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:40.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:40.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:40.401 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:40.427 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:40.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:40.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:40.529 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:40.556 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:40.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:40.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:40.657 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:40.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:40.693 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:40.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:40.794 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:40.804 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:40 smithi102 ceph-mon[110183]: pgmap v710: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 20 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 6.1 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 66 MiB/s wr, 1.06k op/s; 72581/302106 objects degraded (24.025%); 16918/302106 objects misplaced (5.600%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T14:36:40.804 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:40 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:36:40.805 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:40 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:36:40.805 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:40 smithi102 ceph-mon[115269]: pgmap v710: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 20 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 6.1 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 66 MiB/s wr, 1.06k op/s; 72581/302106 objects degraded (24.025%); 16918/302106 objects misplaced (5.600%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T14:36:40.805 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:40 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:36:40.805 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:40 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:36:40.820 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:40.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:40.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:40.861 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:36:40.861 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [ 2024-09-13T14:36:40.861 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:36:40.861 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.35379.0:104673 7.f e220/205 7:ff7db00a:::benchmark_data_smithi102_227395_object104672:head v 220'6559, mlcod=220'6555)", 2024-09-13T14:36:40.861 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:36:40.850702+0000", 2024-09-13T14:36:40.861 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.0082100070000000001, 2024-09-13T14:36:40.861 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0082375690000000001, 2024-09-13T14:36:40.861 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:36:40.862 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:36:40.862 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "started", 2024-09-13T14:36:40.862 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:36:40.862 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:36:40.862 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:36:40.862 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:40.850702+0000", 2024-09-13T14:36:40.862 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:36:40.862 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:36:40.862 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:36:40.862 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:36:40.862 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:40.850702+0000", 2024-09-13T14:36:40.863 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:36:40.863 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:36:40.863 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:36:40.863 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:36:40.863 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:40.850703+0000", 2024-09-13T14:36:40.863 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.5000000000000002e-07 2024-09-13T14:36:40.863 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:36:40.863 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:36:40.863 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:36:40.863 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:40.850744+0000", 2024-09-13T14:36:40.863 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.1607000000000003e-05 2024-09-13T14:36:40.863 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:36:40.863 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:36:40.864 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:36:40.864 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:40.850755+0000", 2024-09-13T14:36:40.864 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0505e-05 2024-09-13T14:36:40.864 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:36:40.864 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:36:40.864 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:36:40.864 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:40.850761+0000", 2024-09-13T14:36:40.864 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.9050000000000002e-06 2024-09-13T14:36:40.864 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:36:40.865 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:36:40.865 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:36:40.865 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:40.855315+0000", 2024-09-13T14:36:40.865 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0045542500000000001 2024-09-13T14:36:40.865 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:36:40.865 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:36:40.865 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:36:40.865 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:40.855337+0000", 2024-09-13T14:36:40.865 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.2226999999999999e-05 2024-09-13T14:36:40.865 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:36:40.866 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:36:40.866 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:36:40.866 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:36:40.866 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:36:40.866 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.35379.0:104670 7.6 e220/215 7:6e2e1102:::benchmark_data_smithi102_227395_object104669:head v 220'6559, mlcod=0'0)", 2024-09-13T14:36:40.866 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:36:40.854325+0000", 2024-09-13T14:36:40.866 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.0045871369999999998, 2024-09-13T14:36:40.866 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0046725839999999996, 2024-09-13T14:36:40.866 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:36:40.866 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:36:40.866 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:36:40.867 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:36:40.867 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:36:40.867 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:36:40.867 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:40.854325+0000", 2024-09-13T14:36:40.867 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:36:40.867 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:36:40.867 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:36:40.867 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:36:40.867 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:40.854325+0000", 2024-09-13T14:36:40.867 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:36:40.868 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:36:40.868 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:36:40.868 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:36:40.868 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:40.854326+0000", 2024-09-13T14:36:40.868 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.2399999999999998e-07 2024-09-13T14:36:40.868 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:36:40.868 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:36:40.868 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:36:40.868 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:40.854348+0000", 2024-09-13T14:36:40.868 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.2569e-05 2024-09-13T14:36:40.868 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:36:40.868 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:36:40.869 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:36:40.869 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:40.854362+0000", 2024-09-13T14:36:40.869 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3991999999999999e-05 2024-09-13T14:36:40.869 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:36:40.869 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:36:40.869 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:36:40.869 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:40.854369+0000", 2024-09-13T14:36:40.869 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.9399999999999996e-06 2024-09-13T14:36:40.869 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:36:40.869 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:36:40.869 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:36:40.869 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:40.854505+0000", 2024-09-13T14:36:40.869 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00013602699999999999 2024-09-13T14:36:40.870 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:36:40.870 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:36:40.870 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:36:40.870 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:40.854544+0000", 2024-09-13T14:36:40.870 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.8448999999999997e-05 2024-09-13T14:36:40.870 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:36:40.870 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:36:40.870 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:36:40.870 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:40.858006+0000", 2024-09-13T14:36:40.870 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0034623219999999999 2024-09-13T14:36:40.871 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:36:40.871 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:36:40.871 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:36:40.871 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:36:40.871 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:36:40.871 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.35379.0:104675 7.f e220/205)", 2024-09-13T14:36:40.871 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:36:40.856030+0000", 2024-09-13T14:36:40.871 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.0028816689999999999, 2024-09-13T14:36:40.871 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0030400039999999998, 2024-09-13T14:36:40.871 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:36:40.871 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:36:40.872 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "queued for pg", 2024-09-13T14:36:40.872 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:36:40.872 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:36:40.872 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:36:40.872 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:40.856030+0000", 2024-09-13T14:36:40.872 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:36:40.872 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:36:40.872 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:36:40.872 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:36:40.872 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:40.856030+0000", 2024-09-13T14:36:40.872 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:36:40.872 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:36:40.872 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:36:40.873 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:36:40.873 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:40.856031+0000", 2024-09-13T14:36:40.873 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.2399999999999997e-07 2024-09-13T14:36:40.873 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:36:40.873 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:36:40.873 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:36:40.873 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:40.856054+0000", 2024-09-13T14:36:40.873 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.2277999999999999e-05 2024-09-13T14:36:40.873 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:36:40.873 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:36:40.873 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:36:40.873 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:40.856069+0000", 2024-09-13T14:36:40.873 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.5641000000000002e-05 2024-09-13T14:36:40.874 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:36:40.874 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:36:40.874 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:36:40.874 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:40.856077+0000", 2024-09-13T14:36:40.874 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.559e-06 2024-09-13T14:36:40.874 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:36:40.874 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:36:40.874 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:36:40.874 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:36:40.874 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:36:40.874 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35379.0:104678 7.3 7:cad965c3:::benchmark_data_smithi102_227395_object104677:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e220)", 2024-09-13T14:36:40.875 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:36:40.857839+0000", 2024-09-13T14:36:40.875 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.001072882, 2024-09-13T14:36:40.875 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0012760569999999999, 2024-09-13T14:36:40.875 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:36:40.875 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:36:40.875 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:36:40.875 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:36:40.875 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35379", 2024-09-13T14:36:40.875 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:36:40.875 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 104678 2024-09-13T14:36:40.875 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:36:40.875 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:36:40.875 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:36:40.876 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:36:40.876 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:40.857839+0000", 2024-09-13T14:36:40.876 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:36:40.876 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:36:40.876 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:36:40.876 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:36:40.876 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:40.857839+0000", 2024-09-13T14:36:40.876 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:36:40.876 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:36:40.876 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:36:40.876 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:36:40.876 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:40.857841+0000", 2024-09-13T14:36:40.876 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.395e-06 2024-09-13T14:36:40.876 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:36:40.877 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:36:40.877 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:36:40.877 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:40.857878+0000", 2024-09-13T14:36:40.877 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.7530999999999997e-05 2024-09-13T14:36:40.877 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:36:40.877 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:36:40.877 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:36:40.877 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:40.857891+0000", 2024-09-13T14:36:40.877 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2537999999999999e-05 2024-09-13T14:36:40.877 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:36:40.877 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:36:40.877 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:36:40.878 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:40.857898+0000", 2024-09-13T14:36:40.878 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.6149999999999999e-06 2024-09-13T14:36:40.878 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:36:40.878 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:36:40.878 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:36:40.878 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:40.857959+0000", 2024-09-13T14:36:40.878 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.1094999999999997e-05 2024-09-13T14:36:40.878 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:36:40.878 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:36:40.878 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:36:40.878 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:40.858103+0000", 2024-09-13T14:36:40.878 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000143657 2024-09-13T14:36:40.878 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:36:40.879 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:36:40.879 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4", 2024-09-13T14:36:40.879 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:40.858274+0000", 2024-09-13T14:36:40.879 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00017091099999999999 2024-09-13T14:36:40.879 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:36:40.879 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:36:40.879 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:36:40.879 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:36:40.879 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:36:40.879 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.35379.0:104680 7.0 e220/215)", 2024-09-13T14:36:40.879 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:36:40.858644+0000", 2024-09-13T14:36:40.879 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.00026825500000000001, 2024-09-13T14:36:40.880 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00052794400000000003, 2024-09-13T14:36:40.880 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:36:40.880 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:36:40.880 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "queued for pg", 2024-09-13T14:36:40.880 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:36:40.880 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:36:40.881 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:36:40.881 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:40.858644+0000", 2024-09-13T14:36:40.881 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:36:40.881 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:36:40.881 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:36:40.881 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:36:40.881 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:40.858644+0000", 2024-09-13T14:36:40.881 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:36:40.881 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:36:40.881 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:36:40.881 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:36:40.881 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:40.858644+0000", 2024-09-13T14:36:40.881 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.0299999999999999e-07 2024-09-13T14:36:40.882 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:36:40.882 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:36:40.882 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:36:40.882 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:40.858668+0000", 2024-09-13T14:36:40.882 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.3349e-05 2024-09-13T14:36:40.882 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:36:40.882 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:36:40.882 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:36:40.882 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:40.858679+0000", 2024-09-13T14:36:40.882 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.117e-05 2024-09-13T14:36:40.882 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:36:40.882 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:36:40.882 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:36:40.883 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:40.858685+0000", 2024-09-13T14:36:40.883 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.5379999999999999e-06 2024-09-13T14:36:40.883 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:36:40.883 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:36:40.883 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:36:40.883 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:36:40.883 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-09-13T14:36:40.883 INFO:teuthology.orchestra.run.smithi102.stdout: "num_ops": 5 2024-09-13T14:36:40.883 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:36:40.921 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:40.952 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:40.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:40.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:40.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:40 smithi195 ceph-mon[104262]: pgmap v710: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 20 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 6.1 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 66 MiB/s wr, 1.06k op/s; 72581/302106 objects degraded (24.025%); 16918/302106 objects misplaced (5.600%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T14:36:40.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:40 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:36:40.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:40 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:36:41.042 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 67 16 104866 104850 97.7953 70.125 0.00398913 0.0102195 2024-09-13T14:36:41.055 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:41.093 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:41.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:41.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:41.195 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:41.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:41.232 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:41.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:41.333 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:41.368 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:41.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:41.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:41.470 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:41.502 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:41.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:41.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:41.604 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:41.630 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:41.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:41.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:41.732 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:41.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:41.770 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:41.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:41.872 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:41.908 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:41.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:41.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:42.010 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:42.042 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 68 16 106044 106028 97.4398 73.625 0.0145551 0.0102581 2024-09-13T14:36:42.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:42.051 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:42.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:42.111 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T14:36:42.152 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:42.201 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:42.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:42.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:42.303 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:42.344 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:42.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:42.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:42.446 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:42.474 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:42.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:42.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:42.576 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:42.610 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:42.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:42.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:42.617 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:36:42.712 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:42.740 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:42.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:42.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:42.842 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:42.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:42.895 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:42.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:42.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:42 smithi102 ceph-mon[110183]: pgmap v711: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 20 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 6.1 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 66 MiB/s wr, 1.06k op/s; 72581/302106 objects degraded (24.025%); 16918/302106 objects misplaced (5.600%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T14:36:42.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:42 smithi102 ceph-mon[115269]: pgmap v711: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 20 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 6.1 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 66 MiB/s wr, 1.06k op/s; 72581/302106 objects degraded (24.025%); 16918/302106 objects misplaced (5.600%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T14:36:42.996 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:42.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:42 smithi195 ceph-mon[104262]: pgmap v711: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 20 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 6.1 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 66 MiB/s wr, 1.06k op/s; 72581/302106 objects degraded (24.025%); 16918/302106 objects misplaced (5.600%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T14:36:43.028 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:43.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:43.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:43.042 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 69 16 107080 107064 96.966 64.75 0.0138663 0.0103077 2024-09-13T14:36:43.131 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:43.155 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:43.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:43.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:43.212 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:36:43.257 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:43.285 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:43.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:43.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:43.387 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:43.413 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:43.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:43.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:43.517 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:43.545 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:43.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:43.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:43.647 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:43.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:43.680 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:43.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:43.781 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:43.813 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:43.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:43.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:43.915 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:43.960 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:43.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:43.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:44.042 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 70 16 108088 108072 96.4806 63 0.0026779 0.0103531 2024-09-13T14:36:44.063 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:44.084 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:44.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:44.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:44.186 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:44.221 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:44.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:44.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:44.324 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:44.356 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:44.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:44.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:44.458 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:44.489 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:44.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:44.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:44.591 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:44.624 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:44.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:44.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:44.725 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:44.750 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:44.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:44.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:44.852 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:44.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:44.881 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:44.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:44.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:44 smithi102 ceph-mon[110183]: pgmap v712: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 20 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 6.5 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 68 MiB/s wr, 1.09k op/s; 76072/319020 objects degraded (23.846%); 17477/319020 objects misplaced (5.478%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:36:44.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:44 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 76072/319020 objects degraded (23.846%), 15 pgs degraded (PG_DEGRADED) 2024-09-13T14:36:44.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:44 smithi102 ceph-mon[115269]: pgmap v712: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 20 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 6.5 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 68 MiB/s wr, 1.09k op/s; 76072/319020 objects degraded (23.846%); 17477/319020 objects misplaced (5.478%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:36:44.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:44 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 76072/319020 objects degraded (23.846%), 15 pgs degraded (PG_DEGRADED) 2024-09-13T14:36:44.982 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:44.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:44 smithi195 ceph-mon[104262]: pgmap v712: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 20 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 6.5 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 68 MiB/s wr, 1.09k op/s; 76072/319020 objects degraded (23.846%); 17477/319020 objects misplaced (5.478%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:36:44.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:44 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 76072/319020 objects degraded (23.846%), 15 pgs degraded (PG_DEGRADED) 2024-09-13T14:36:45.014 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:45.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:45.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:45.042 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 71 16 109148 109132 96.0548 66.25 0.0146907 0.0104062 2024-09-13T14:36:45.117 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:45.152 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:45.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:45.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:45.254 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:45.282 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:45.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:45.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:45.384 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:45.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:45.424 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:45.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:45.525 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:45.552 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:45.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:45.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:45.653 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:45.692 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:45.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:45.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:45.794 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:45.817 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:45.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:45.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:45.919 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:45.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:45.955 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:45.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:46.042 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 72 16 110194 110178 95.6286 65.375 0.00635555 0.0104519 2024-09-13T14:36:46.056 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:46.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:46.098 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:46.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:46.199 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:46.232 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:46.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:46.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:46.334 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:46.363 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:46.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:46.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:46.465 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:46.490 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:46.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:46.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:46.592 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:46.628 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:46.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:46.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:46.721 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:46 smithi102 ceph-mon[110183]: pgmap v713: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 20 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 6.5 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 68 MiB/s wr, 1.09k op/s; 76072/319020 objects degraded (23.846%); 17477/319020 objects misplaced (5.478%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:36:46.721 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:36:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:36:46] "GET /metrics HTTP/1.1" 200 36555 "" "Prometheus/2.51.0" 2024-09-13T14:36:46.722 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:46 smithi102 ceph-mon[115269]: pgmap v713: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 20 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 6.5 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 68 MiB/s wr, 1.09k op/s; 76072/319020 objects degraded (23.846%); 17477/319020 objects misplaced (5.478%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:36:46.730 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:46.754 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:46.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:46.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:46.856 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:46.900 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:46.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:46.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:46.957 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:36:46.957 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:36:46.957 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:36:46.957 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:36:46.957 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:36:46.957 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:36:46.958 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:36:46.958 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:36:46.958 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:36:46.958 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:36:46.958 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:36:46.958 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:36:46.958 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:36:46.958 INFO:teuthology.orchestra.run.smithi102.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-13T14:36:46.958 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:36:46.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:46 smithi195 ceph-mon[104262]: pgmap v713: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 20 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 6.5 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 68 MiB/s wr, 1.09k op/s; 76072/319020 objects degraded (23.846%); 17477/319020 objects misplaced (5.478%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:36:47.002 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:47.034 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:47.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:47.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:47.042 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 73 16 111274 111258 95.2432 67.5 0.00748774 0.0104948 2024-09-13T14:36:47.136 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:47.176 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:47.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:47.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:47.213 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:36:47.278 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:47.302 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:47.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:47.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:47.404 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:47.431 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:47.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:47.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:47.533 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:47.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:47.556 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:47.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:47.657 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:47.682 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:47.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:47.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:47.784 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:47.794 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:47 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2330084995' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:36:47.794 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:47 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2330084995' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:36:47.815 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:47.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:47.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:47.916 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:47.943 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:47.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:47.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:47.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:47 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2330084995' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:36:48.042 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 74 16 112158 112142 94.7027 55.25 0.00477235 0.0105545 2024-09-13T14:36:48.045 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:48.074 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:48.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:48.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:48.176 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:48.202 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:48.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:48.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:48.304 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:48.332 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:48.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:48.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:48.434 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:48.463 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:48.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:48.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:48.485 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T14:36:48.485 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T14:36:48.546 INFO:teuthology.orchestra.run.smithi102.stderr:reweighted osd.0 to 0.311025 (4f9f) 2024-09-13T14:36:48.565 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:48.588 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:48.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:48.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:48.690 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:48.711 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:48.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:48.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:48.737 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:48 smithi102 ceph-mon[110183]: pgmap v714: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 20 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 6.8 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 90 MiB/s wr, 1.44k op/s; 79022/333783 objects degraded (23.675%); 17988/333783 objects misplaced (5.389%); 2.1 MiB/s, 33 objects/s recovering 2024-09-13T14:36:48.737 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:48 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3350889411' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 0.3110248273131276} : dispatch 2024-09-13T14:36:48.737 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:48 smithi102 ceph-mon[115269]: pgmap v714: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 20 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 6.8 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 90 MiB/s wr, 1.44k op/s; 79022/333783 objects degraded (23.675%); 17988/333783 objects misplaced (5.389%); 2.1 MiB/s, 33 objects/s recovering 2024-09-13T14:36:48.737 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:48 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3350889411' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 0.3110248273131276} : dispatch 2024-09-13T14:36:48.813 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:48.835 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:48.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:48.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:48.937 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:48.961 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:48.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:48.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:48.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:48 smithi195 ceph-mon[104262]: pgmap v714: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 20 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 6.8 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 90 MiB/s wr, 1.44k op/s; 79022/333783 objects degraded (23.675%); 17988/333783 objects misplaced (5.389%); 2.1 MiB/s, 33 objects/s recovering 2024-09-13T14:36:48.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:48 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3350889411' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 0.3110248273131276} : dispatch 2024-09-13T14:36:49.043 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 75 16 112734 112718 93.92 36 0.00439954 0.0105736 2024-09-13T14:36:49.063 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:49.086 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:49.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:49.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:49.187 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:49.209 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:49.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:49.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:49.310 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:49.331 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:49.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:49.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:49.433 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:49.455 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:49.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:49.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:49.556 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:49.579 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:49.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:49.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:49.680 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:49.702 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:49.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:49.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:49.804 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:49.825 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:49.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:49.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:49.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:49 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3350889411' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 0.3110248273131276}]': finished 2024-09-13T14:36:49.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:49 smithi102 ceph-mon[110183]: osdmap e221: 8 total, 7 up, 6 in 2024-09-13T14:36:49.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:49 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 79022/333783 objects degraded (23.675%), 15 pgs degraded (PG_DEGRADED) 2024-09-13T14:36:49.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:49 smithi102 ceph-mon[110183]: osdmap e222: 8 total, 7 up, 6 in 2024-09-13T14:36:49.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:49 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3350889411' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 0.3110248273131276}]': finished 2024-09-13T14:36:49.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:49 smithi102 ceph-mon[115269]: osdmap e221: 8 total, 7 up, 6 in 2024-09-13T14:36:49.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:49 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 79022/333783 objects degraded (23.675%), 15 pgs degraded (PG_DEGRADED) 2024-09-13T14:36:49.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:49 smithi102 ceph-mon[115269]: osdmap e222: 8 total, 7 up, 6 in 2024-09-13T14:36:49.927 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:49.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:49.965 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:49.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:49.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:49 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3350889411' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 0.3110248273131276}]': finished 2024-09-13T14:36:49.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:49 smithi195 ceph-mon[104262]: osdmap e221: 8 total, 7 up, 6 in 2024-09-13T14:36:49.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:49 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 79022/333783 objects degraded (23.675%), 15 pgs degraded (PG_DEGRADED) 2024-09-13T14:36:49.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:49 smithi195 ceph-mon[104262]: osdmap e222: 8 total, 7 up, 6 in 2024-09-13T14:36:50.043 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 76 16 112751 112735 92.6983 1.0625 0.00364408 0.010611 2024-09-13T14:36:50.066 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:50.096 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:50.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:50.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:50.198 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:50.227 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:50.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:50.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:50.330 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:50.355 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:50.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:50.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:50.457 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:50.479 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:50.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:50.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:50.486 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:36:50.580 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:50.603 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:50.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:50.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:50.705 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:50.728 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:50.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:50.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:50.829 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:50.852 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:50.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:50.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:50.954 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:50.976 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:50.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:50.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:51.043 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 77 16 112812 112796 91.5439 3.8125 0.00356543 0.0106664 2024-09-13T14:36:51.081 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:51.104 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:51.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:51.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:51.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:50 smithi102 ceph-mon[110183]: pgmap v717: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 20 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 6.8 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 85 MiB/s wr, 1.36k op/s; 79323/334686 objects degraded (23.701%); 17988/334686 objects misplaced (5.375%); 2.0 MiB/s, 31 objects/s recovering 2024-09-13T14:36:51.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:50 smithi102 ceph-mon[110183]: 7.4 starting backfill to osd.5 from (0'0,0'0] MAX to 220'7043 2024-09-13T14:36:51.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:50 smithi102 ceph-mon[110183]: osdmap e223: 8 total, 7 up, 6 in 2024-09-13T14:36:51.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:50 smithi102 ceph-mon[110183]: 7.f starting backfill to osd.4 from (0'0,0'0] MAX to 220'7040 2024-09-13T14:36:51.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:50 smithi102 ceph-mon[110183]: 7.7 starting backfill to osd.7 from (0'0,0'0] MAX to 220'6995 2024-09-13T14:36:51.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:50 smithi102 ceph-mon[115269]: pgmap v717: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 20 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 6.8 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 85 MiB/s wr, 1.36k op/s; 79323/334686 objects degraded (23.701%); 17988/334686 objects misplaced (5.375%); 2.0 MiB/s, 31 objects/s recovering 2024-09-13T14:36:51.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:50 smithi102 ceph-mon[115269]: 7.4 starting backfill to osd.5 from (0'0,0'0] MAX to 220'7043 2024-09-13T14:36:51.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:50 smithi102 ceph-mon[115269]: osdmap e223: 8 total, 7 up, 6 in 2024-09-13T14:36:51.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:50 smithi102 ceph-mon[115269]: 7.f starting backfill to osd.4 from (0'0,0'0] MAX to 220'7040 2024-09-13T14:36:51.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:50 smithi102 ceph-mon[115269]: 7.7 starting backfill to osd.7 from (0'0,0'0] MAX to 220'6995 2024-09-13T14:36:51.206 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:51.231 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:51.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:51.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:51.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:50 smithi195 ceph-mon[104262]: pgmap v717: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 20 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 6.8 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 85 MiB/s wr, 1.36k op/s; 79323/334686 objects degraded (23.701%); 17988/334686 objects misplaced (5.375%); 2.0 MiB/s, 31 objects/s recovering 2024-09-13T14:36:51.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:50 smithi195 ceph-mon[104262]: 7.4 starting backfill to osd.5 from (0'0,0'0] MAX to 220'7043 2024-09-13T14:36:51.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:50 smithi195 ceph-mon[104262]: osdmap e223: 8 total, 7 up, 6 in 2024-09-13T14:36:51.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:50 smithi195 ceph-mon[104262]: 7.f starting backfill to osd.4 from (0'0,0'0] MAX to 220'7040 2024-09-13T14:36:51.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:50 smithi195 ceph-mon[104262]: 7.7 starting backfill to osd.7 from (0'0,0'0] MAX to 220'6995 2024-09-13T14:36:51.333 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:51.355 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:51.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:51.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:51.457 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:51.479 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:51.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:51.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:51.581 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:51.603 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:51.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:51.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:51.705 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:51.727 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:51.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:51.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:51.829 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:51.851 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:51.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:51.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:51.954 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:51.976 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:51.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:51.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:52.043 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 78 16 112812 112796 90.3703 0 - 0.0106664 2024-09-13T14:36:52.078 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:52.101 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:52.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:52.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:52.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:51 smithi102 ceph-mon[110183]: osdmap e224: 8 total, 7 up, 6 in 2024-09-13T14:36:52.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:51 smithi102 ceph-mon[110183]: osdmap e225: 8 total, 7 up, 6 in 2024-09-13T14:36:52.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:51 smithi102 ceph-mon[115269]: osdmap e224: 8 total, 7 up, 6 in 2024-09-13T14:36:52.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:51 smithi102 ceph-mon[115269]: osdmap e225: 8 total, 7 up, 6 in 2024-09-13T14:36:52.203 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:52.228 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:52.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:52.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:52.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:51 smithi195 ceph-mon[104262]: osdmap e224: 8 total, 7 up, 6 in 2024-09-13T14:36:52.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:51 smithi195 ceph-mon[104262]: osdmap e225: 8 total, 7 up, 6 in 2024-09-13T14:36:52.330 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:52.354 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:52.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:52.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:52.456 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:52.478 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:52.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:52.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:52.580 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:52.602 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:52.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:52.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:52.704 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:52.726 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:52.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:52.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:52.829 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:52.852 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:52.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:52.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:52.954 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:52.977 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:52.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:52.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:53.043 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 79 16 112812 112796 89.2264 0 - 0.0106664 2024-09-13T14:36:53.078 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:53.102 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:53.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:53.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:53.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:52 smithi102 ceph-mon[115269]: pgmap v720: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 20 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 6.8 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 4.7 MiB/s wr, 75 op/s; 79323/334686 objects degraded (23.701%); 17988/334686 objects misplaced (5.375%) 2024-09-13T14:36:53.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:52 smithi102 ceph-mon[110183]: pgmap v720: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 20 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 6.8 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 4.7 MiB/s wr, 75 op/s; 79323/334686 objects degraded (23.701%); 17988/334686 objects misplaced (5.375%) 2024-09-13T14:36:53.204 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:53.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:53.230 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:53.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:53.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:52 smithi195 ceph-mon[104262]: pgmap v720: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 20 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 18 active+undersized, 11 active+undersized+degraded, 35 active+clean; 6.8 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 4.7 MiB/s wr, 75 op/s; 79323/334686 objects degraded (23.701%); 17988/334686 objects misplaced (5.375%) 2024-09-13T14:36:53.331 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:53.353 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:53.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:53.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:53.455 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:53.479 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:53.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:53.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:53.581 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:53.605 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:53.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:53.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:53.707 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:53.731 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:53.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:53.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:53.833 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:53.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:53.858 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:53.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:53.960 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:53.982 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:53.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:53.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:54.044 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:2024-09-13T14:36:54.043308+0000 min lat: 0.00114883 max lat: 1.99103 avg lat: 0.0106664 2024-09-13T14:36:54.045 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:36:54.045 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 80 16 112812 112796 88.1112 0 - 0.0106664 2024-09-13T14:36:54.076 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:53 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 79323/334686 objects degraded (23.701%), 15 pgs degraded (PG_DEGRADED) 2024-09-13T14:36:54.077 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:53 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 79323/334686 objects degraded (23.701%), 15 pgs degraded (PG_DEGRADED) 2024-09-13T14:36:54.084 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:54.108 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:54.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:54.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:54.210 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:54.234 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:54.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:54.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:54.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:53 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 79323/334686 objects degraded (23.701%), 15 pgs degraded (PG_DEGRADED) 2024-09-13T14:36:54.335 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:54.360 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:54.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:54.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:54.461 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:54.484 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:54.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:54.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:54.586 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:54.609 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:54.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:54.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:54.711 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:54.735 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:54.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:54.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:54.837 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:54.862 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:54.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:54.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:54.964 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:54.987 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:54.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:54.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:55.043 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 81 16 112812 112796 87.0234 0 - 0.0106664 2024-09-13T14:36:55.090 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:55.095 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:54 smithi102 ceph-mon[110183]: pgmap v722: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 activating+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 30 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 16 MiB/s wr, 259 op/s; 81115/338238 objects degraded (23.982%); 52806/338238 objects misplaced (15.612%); 25 MiB/s, 403 objects/s recovering 2024-09-13T14:36:55.095 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:54 smithi102 ceph-mon[110183]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-09-13T14:36:55.096 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:54 smithi102 ceph-mon[115269]: pgmap v722: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 activating+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 30 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 16 MiB/s wr, 259 op/s; 81115/338238 objects degraded (23.982%); 52806/338238 objects misplaced (15.612%); 25 MiB/s, 403 objects/s recovering 2024-09-13T14:36:55.096 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:54 smithi102 ceph-mon[115269]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-09-13T14:36:55.117 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:55.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:55.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:55.219 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:55.241 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:55.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:55.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:55.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:54 smithi195 ceph-mon[104262]: pgmap v722: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 activating+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 30 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 16 MiB/s wr, 259 op/s; 81115/338238 objects degraded (23.982%); 52806/338238 objects misplaced (15.612%); 25 MiB/s, 403 objects/s recovering 2024-09-13T14:36:55.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:54 smithi195 ceph-mon[104262]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-09-13T14:36:55.344 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:55.369 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:55.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:55.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:55.472 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:55.496 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:55.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:55.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:55.598 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:55.623 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:55.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:55.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:55.725 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:55.750 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:55.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:55.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:55.852 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:55.875 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:55.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:55.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:55.977 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:56.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:56.001 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:56.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:56.043 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 82 16 112812 112796 85.9622 0 - 0.0106664 2024-09-13T14:36:56.102 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:56.126 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:56.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:56.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:56.228 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:56.252 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:56.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:56.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:56.336 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:36:56.336 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:36:56.343 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:36:56.354 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:56.378 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:56.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:56.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:56.480 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:56.503 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:56.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:56.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:56.605 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:56.628 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:56.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:56.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:56.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:56 smithi102 ceph-mon[110183]: pgmap v723: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 activating+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 30 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 12 MiB/s wr, 197 op/s; 81115/338238 objects degraded (23.982%); 52806/338238 objects misplaced (15.612%); 19 MiB/s, 306 objects/s recovering 2024-09-13T14:36:56.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:36:56.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:36:56.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:56 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/475831406' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:36:56.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:56 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:36:56.669 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:36:56 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:36:56] "GET /metrics HTTP/1.1" 200 36574 "" "Prometheus/2.51.0" 2024-09-13T14:36:56.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:56 smithi102 ceph-mon[115269]: pgmap v723: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 activating+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 30 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 12 MiB/s wr, 197 op/s; 81115/338238 objects degraded (23.982%); 52806/338238 objects misplaced (15.612%); 19 MiB/s, 306 objects/s recovering 2024-09-13T14:36:56.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:36:56.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:36:56.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:56 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/475831406' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:36:56.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:56 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:36:56.730 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:56.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:56 smithi195 ceph-mon[104262]: pgmap v723: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 activating+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 30 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 12 MiB/s wr, 197 op/s; 81115/338238 objects degraded (23.982%); 52806/338238 objects misplaced (15.612%); 19 MiB/s, 306 objects/s recovering 2024-09-13T14:36:56.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:36:56.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:36:56.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:56 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/475831406' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:36:56.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:56 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:36:56.754 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:56.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:56.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:56.855 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:56.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:56.882 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:56.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:56.895 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is set 2024-09-13T14:36:56.984 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:57.005 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:57.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:57.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:57.043 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 83 16 112812 112796 84.9265 0 - 0.0106664 2024-09-13T14:36:57.107 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:57.131 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:57.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:57.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:57.233 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:57.257 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:57.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:57.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:57.358 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:57.381 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:57.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:57.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:57.482 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:57.506 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:57.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:57.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:57.608 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:57.630 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:57.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:57.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:57.732 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:57.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:57 smithi195 ceph-mon[104262]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:36:57.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:57 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:36:57.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:57 smithi195 ceph-mon[104262]: osdmap e226: 8 total, 7 up, 6 in 2024-09-13T14:36:57.758 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:57.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:57.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:57.860 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:57.882 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:57.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:57.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:57.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:57 smithi102 ceph-mon[110183]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:36:57.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:57 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:36:57.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:57 smithi102 ceph-mon[110183]: osdmap e226: 8 total, 7 up, 6 in 2024-09-13T14:36:57.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:57 smithi102 ceph-mon[115269]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:36:57.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:57 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:36:57.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:57 smithi102 ceph-mon[115269]: osdmap e226: 8 total, 7 up, 6 in 2024-09-13T14:36:57.984 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:58.007 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:58.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:58.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:58.043 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 84 16 112812 112796 83.9155 0 - 0.0106664 2024-09-13T14:36:58.109 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:58.132 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:58.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:58.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:58.233 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:36:58.256 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:58.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:58.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:36:58.357 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:58.380 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:58.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:58.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:58.482 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:58.504 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:58.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:58.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:58.606 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:58.628 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:58.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:58.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:58.730 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:58.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:58 smithi195 ceph-mon[104262]: pgmap v725: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 activating+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 30 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 11 MiB/s wr, 180 op/s; 78185/338238 objects degraded (23.115%); 51999/338238 objects misplaced (15.373%); 53 MiB/s, 850 objects/s recovering 2024-09-13T14:36:58.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:58 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:36:58.754 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:58.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:58.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:58.856 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:36:58.881 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:58.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:58.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:36:58.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:58 smithi102 ceph-mon[110183]: pgmap v725: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 activating+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 30 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 11 MiB/s wr, 180 op/s; 78185/338238 objects degraded (23.115%); 51999/338238 objects misplaced (15.373%); 53 MiB/s, 850 objects/s recovering 2024-09-13T14:36:58.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:58 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:36:58.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:58 smithi102 ceph-mon[115269]: pgmap v725: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 activating+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 30 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 11 MiB/s wr, 180 op/s; 78185/338238 objects degraded (23.115%); 51999/338238 objects misplaced (15.373%); 53 MiB/s, 850 objects/s recovering 2024-09-13T14:36:58.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:58 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:36:58.983 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:59.005 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:59.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:59.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:59.043 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 85 16 112812 112796 82.9283 0 - 0.0106664 2024-09-13T14:36:59.107 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:36:59.132 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:59.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:59.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:36:59.234 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:36:59.256 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:59.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:59.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:36:59.358 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:36:59.382 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:59.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:59.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:36:59.484 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:36:59.506 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:36:59.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:59.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:36:59.608 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:59.630 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:59.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:59.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:59.732 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:59.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:36:59 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 78185/338238 objects degraded (23.115%), 14 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-13T14:36:59.755 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:59.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:59.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:59.857 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:36:59.878 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:36:59.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:36:59.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:36:59.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:36:59 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 78185/338238 objects degraded (23.115%), 14 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-13T14:36:59.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:36:59 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 78185/338238 objects degraded (23.115%), 14 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-13T14:36:59.980 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:00.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:00.003 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:00.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:00.044 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 86 16 112812 112796 81.9641 0 - 0.0106664 2024-09-13T14:37:00.104 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:00.131 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:00.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:00.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:00.132 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 6, 7, 5] out_osds: [1, 2] dead_osds: [6] live_osds: [0, 1, 2, 3, 4, 5, 7] 2024-09-13T14:37:00.132 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:37:00.132 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-09-13T14:37:00.132 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T14:37:00.132 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T14:37:00.133 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-09-13T14:37:00.232 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:00.256 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:00.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:00.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:00.358 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:00.380 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:00.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:00.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:00.482 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:00.508 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:00.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:00.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:00.609 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:00.633 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:00.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:00.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:00.735 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:00.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:00 smithi195 ceph-mon[104262]: pgmap v726: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 activating+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 30 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 9.2 MiB/s wr, 147 op/s; 78185/338238 objects degraded (23.115%); 51999/338238 objects misplaced (15.373%); 44 MiB/s, 696 objects/s recovering 2024-09-13T14:37:00.762 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:00.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:00.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:00.863 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:00.889 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:00.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:00.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:00.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:00 smithi102 ceph-mon[110183]: pgmap v726: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 activating+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 30 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 9.2 MiB/s wr, 147 op/s; 78185/338238 objects degraded (23.115%); 51999/338238 objects misplaced (15.373%); 44 MiB/s, 696 objects/s recovering 2024-09-13T14:37:00.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:00 smithi102 ceph-mon[115269]: pgmap v726: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 activating+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 30 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 9.2 MiB/s wr, 147 op/s; 78185/338238 objects degraded (23.115%); 51999/338238 objects misplaced (15.373%); 44 MiB/s, 696 objects/s recovering 2024-09-13T14:37:00.991 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:01.013 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:01.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:01.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:01.043 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 87 16 112812 112796 81.022 0 - 0.0106664 2024-09-13T14:37:01.115 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:01.137 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:01.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:01.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:01.239 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:01.264 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:01.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:01.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:01.366 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:01.388 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:01.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:01.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:01.490 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:01.511 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:01.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:01.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:01.613 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:01.639 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:01.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:01.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:01.740 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:01.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:01 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:37:01.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:01 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:37:01.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:01.764 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:01.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:01.865 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:01.918 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:01.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:01 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:37:01.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:01 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:37:01.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:01 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:37:01.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:01 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:37:01.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:01.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:02.021 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:02.043 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 88 16 112812 112796 80.1014 0 - 0.0106664 2024-09-13T14:37:02.044 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:02.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:02.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:02.146 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:02.169 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:02.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:02.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:02.271 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:02.295 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:02.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:02.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:02.397 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:02.422 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:02.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:02.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:02.524 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:02.546 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:02.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:02.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:02.648 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:02.670 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:02.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:02.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:02.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:02 smithi195 ceph-mon[104262]: pgmap v727: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 activating+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 30 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 7.7 MiB/s wr, 123 op/s; 78185/338238 objects degraded (23.115%); 51999/338238 objects misplaced (15.373%); 36 MiB/s, 583 objects/s recovering 2024-09-13T14:37:02.773 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:02.796 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:02.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:02.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:02.897 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:02.912 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:02 smithi102 ceph-mon[110183]: pgmap v727: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 activating+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 30 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 7.7 MiB/s wr, 123 op/s; 78185/338238 objects degraded (23.115%); 51999/338238 objects misplaced (15.373%); 36 MiB/s, 583 objects/s recovering 2024-09-13T14:37:02.913 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:02 smithi102 ceph-mon[115269]: pgmap v727: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 activating+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 30 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 7.7 MiB/s wr, 123 op/s; 78185/338238 objects degraded (23.115%); 51999/338238 objects misplaced (15.373%); 36 MiB/s, 583 objects/s recovering 2024-09-13T14:37:02.920 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:02.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:02.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:03.022 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:03.044 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:03.044 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 89 16 112812 112796 79.2014 0 - 0.0106664 2024-09-13T14:37:03.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:03.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:03.146 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:03.171 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:03.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:03.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:03.273 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:03.296 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:03.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:03.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:03.398 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:03.421 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:03.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:03.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:03.524 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:03.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:03.549 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:03.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:03.652 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:03.675 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:03.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:03.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:03.777 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:03.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:03.801 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:03.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:03.902 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:03.926 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:03.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:03.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:04.029 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:04.044 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 90 16 112812 112796 78.3214 0 - 0.0106664 2024-09-13T14:37:04.053 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:04.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:04.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:04.155 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:04.178 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:04.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:04.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:04.280 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:04.302 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:04.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:04.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:04.404 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:04.426 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:04.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:04.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:04.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:04 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:37:04.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:04 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:37:04.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:04 smithi195 ceph-mon[104262]: osdmap e227: 8 total, 7 up, 6 in 2024-09-13T14:37:04.527 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:04.550 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:04.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:04.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:04.652 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:04.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:04 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:37:04.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:04 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:37:04.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:04 smithi102 ceph-mon[110183]: osdmap e227: 8 total, 7 up, 6 in 2024-09-13T14:37:04.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:04 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:37:04.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:04 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:37:04.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:04 smithi102 ceph-mon[115269]: osdmap e227: 8 total, 7 up, 6 in 2024-09-13T14:37:04.675 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:04.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:04.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:04.777 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:04.799 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:04.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:04.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:04.901 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:04.924 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:04.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:04.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:05.026 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:05.044 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 91 16 112812 112796 77.4607 0 - 0.0106664 2024-09-13T14:37:05.048 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:05.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:05.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:05.150 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:05.173 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:05.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:05.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:05.275 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:05.297 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:05.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:05.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:05.399 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:05.420 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:05.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:05.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:05.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:05 smithi195 ceph-mon[104262]: pgmap v728: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 activating+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 30 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 75611/338238 objects degraded (22.354%); 51125/338238 objects misplaced (15.115%); 45 MiB/s, 718 objects/s recovering 2024-09-13T14:37:05.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:05 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 75611/338238 objects degraded (22.354%), 14 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-13T14:37:05.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:05 smithi195 ceph-mon[104262]: osdmap e228: 8 total, 7 up, 6 in 2024-09-13T14:37:05.522 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:05.543 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:05.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:05.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:05.645 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:05.666 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:05.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:05.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:05.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:05 smithi102 ceph-mon[115269]: pgmap v728: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 activating+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 30 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 75611/338238 objects degraded (22.354%); 51125/338238 objects misplaced (15.115%); 45 MiB/s, 718 objects/s recovering 2024-09-13T14:37:05.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:05 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 75611/338238 objects degraded (22.354%), 14 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-13T14:37:05.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:05 smithi102 ceph-mon[115269]: osdmap e228: 8 total, 7 up, 6 in 2024-09-13T14:37:05.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:05 smithi102 ceph-mon[110183]: pgmap v728: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 activating+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 30 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 75611/338238 objects degraded (22.354%); 51125/338238 objects misplaced (15.115%); 45 MiB/s, 718 objects/s recovering 2024-09-13T14:37:05.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:05 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 75611/338238 objects degraded (22.354%), 14 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-13T14:37:05.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:05 smithi102 ceph-mon[110183]: osdmap e228: 8 total, 7 up, 6 in 2024-09-13T14:37:05.768 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:05.820 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:05.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:05.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:05.922 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:05.944 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:05.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:05.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:06.044 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 92 16 112812 112796 76.6188 0 - 0.0106664 2024-09-13T14:37:06.046 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:06.067 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:06.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:06.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:06.169 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:06.191 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:06.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:06.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:06.293 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:06.315 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:06.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:06.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:06.416 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:06.438 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:06.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:06.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:06.540 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:06.562 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:06.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:06.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:06.664 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:06.685 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:06.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:06.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:06.787 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:06.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:06.810 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:06.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:06.911 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:06.918 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:37:06 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:37:06] "GET /metrics HTTP/1.1" 200 36574 "" "Prometheus/2.51.0" 2024-09-13T14:37:06.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:06.935 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:06.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:06.964 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T14:37:07.037 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:07.044 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 93 16 112812 112796 75.795 0 - 0.0106664 2024-09-13T14:37:07.058 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:07.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:07.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:07.160 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:07.183 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:07.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:07.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:07.285 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:07.306 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:07.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:07.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:07.408 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:07.431 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:07.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:07.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:07.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:07 smithi195 ceph-mon[104262]: pgmap v731: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 activating+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 30 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 75611/338238 objects degraded (22.354%); 51125/338238 objects misplaced (15.115%); 27 MiB/s, 430 objects/s recovering 2024-09-13T14:37:07.533 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:07.555 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:07.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:07.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:07.657 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:07.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:07 smithi102 ceph-mon[110183]: pgmap v731: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 activating+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 30 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 75611/338238 objects degraded (22.354%); 51125/338238 objects misplaced (15.115%); 27 MiB/s, 430 objects/s recovering 2024-09-13T14:37:07.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:07 smithi102 ceph-mon[115269]: pgmap v731: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 activating+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 30 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 75611/338238 objects degraded (22.354%); 51125/338238 objects misplaced (15.115%); 27 MiB/s, 430 objects/s recovering 2024-09-13T14:37:07.678 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:07.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:07.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:07.779 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:07.801 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:07.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:07.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:07.903 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:07.929 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:07.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:07.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:08.031 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:08.044 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 94 16 112812 112796 74.9887 0 - 0.0106664 2024-09-13T14:37:08.053 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:08.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:08.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:08.155 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:08.177 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:08.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:08.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:08.279 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:08.302 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:08.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:08.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:08.404 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:08.426 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:08.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:08.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:08.528 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:08.549 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:08.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:08.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:08.652 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:08.674 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:08.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:08.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:08.776 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:08.799 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:08.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:08.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:08.901 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:08.923 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:08.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:08.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:09.025 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:09.044 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 95 16 112812 112796 74.1993 0 - 0.0106664 2024-09-13T14:37:09.046 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:09.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:09.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:09.148 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:09.169 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:09.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:09.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:09.271 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:09.294 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:09.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:09.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:09.308 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:37:09.334 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:37:09.396 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:09.419 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:09.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:09.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:09.520 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:09.545 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:09.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:09.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:09.588 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:09 smithi102 ceph-mon[110183]: pgmap v732: 89 pgs: 1 activating+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 31 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 75363/338238 objects degraded (22.281%); 49837/338238 objects misplaced (14.734%); 39 MiB/s, 622 objects/s recovering 2024-09-13T14:37:09.588 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:09 smithi102 ceph-mon[115269]: pgmap v732: 89 pgs: 1 activating+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 31 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 75363/338238 objects degraded (22.281%); 49837/338238 objects misplaced (14.734%); 39 MiB/s, 622 objects/s recovering 2024-09-13T14:37:09.646 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:09.668 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:09.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:09.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:09.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:09 smithi195 ceph-mon[104262]: pgmap v732: 89 pgs: 1 activating+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 31 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 75363/338238 objects degraded (22.281%); 49837/338238 objects misplaced (14.734%); 39 MiB/s, 622 objects/s recovering 2024-09-13T14:37:09.770 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:09.793 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:09.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:09.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:09.895 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:09.917 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:09.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:09.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:10.019 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:10.042 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:10.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:10.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:10.044 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 96 16 112812 112796 73.4265 0 - 0.0106664 2024-09-13T14:37:10.051 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:37:10.051 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:37:10.069 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:37:10.069 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:37:10.087 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:37:10.087 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:37:10.105 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:37:10.105 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:37:10.121 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:37:10.122 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:37:10.144 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:10.169 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:10.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:10.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:10.271 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:10.277 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:37:10.293 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:10.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:10.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:10.395 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:10.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:10 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2229343571' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:37:10.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:10 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2229343571' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:37:10.421 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:10.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:10.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:10.523 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:10.544 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:10.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:10.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:10.646 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:10.668 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:10.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:10.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:10.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:10 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2229343571' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:37:10.770 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:10.792 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:10.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:10.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:10.894 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:10.916 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:10.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:10.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:11.018 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:11.040 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:11.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:11.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:11.044 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 97 16 112812 112796 72.6695 0 - 0.0106664 2024-09-13T14:37:11.142 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:11.163 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:11.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:11.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:11.265 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:11.287 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:11.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:11.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:11.389 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:11.411 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:11.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:11.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:11.513 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:11.536 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:11.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:11.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:11.561 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:11 smithi102 ceph-mon[110183]: pgmap v733: 89 pgs: 1 activating+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 31 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 75363/338238 objects degraded (22.281%); 49837/338238 objects misplaced (14.734%); 39 MiB/s, 622 objects/s recovering 2024-09-13T14:37:11.561 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:11 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:37:11.561 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:11 smithi102 ceph-mon[115269]: pgmap v733: 89 pgs: 1 activating+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 31 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 75363/338238 objects degraded (22.281%); 49837/338238 objects misplaced (14.734%); 39 MiB/s, 622 objects/s recovering 2024-09-13T14:37:11.561 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:11 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:37:11.638 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:11.662 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:11.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:11.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:11.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:11 smithi195 ceph-mon[104262]: pgmap v733: 89 pgs: 1 activating+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 31 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 75363/338238 objects degraded (22.281%); 49837/338238 objects misplaced (14.734%); 39 MiB/s, 622 objects/s recovering 2024-09-13T14:37:11.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:11 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:37:11.764 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:11.786 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:11.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:11.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:11.888 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:11.910 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:11.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:11.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:12.012 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:12.035 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:12.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:12.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:12.045 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 98 16 112812 112796 71.9279 0 - 0.0106664 2024-09-13T14:37:12.137 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:12.159 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:12.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:12.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:12.261 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:12.275 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:37:12.275 INFO:teuthology.orchestra.run.smithi102.stdout: "size": 20, 2024-09-13T14:37:12.275 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 600, 2024-09-13T14:37:12.276 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [ 2024-09-13T14:37:12.276 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.276 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35037.0:163499 6.c 6:3f967e92:::benchmark_data_smithi102_194590_object52875:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-13T14:37:12.276 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:33:43.521329+0000", 2024-09-13T14:37:12.276 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 208.750051466, 2024-09-13T14:37:12.277 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4741819490000001, 2024-09-13T14:37:12.277 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:37:12.277 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:37:12.277 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "reached pg", 2024-09-13T14:37:12.277 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:37:12.277 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35037", 2024-09-13T14:37:12.278 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:37:12.278 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 163499 2024-09-13T14:37:12.278 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.278 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:37:12.278 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.279 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:37:12.279 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.521329+0000", 2024-09-13T14:37:12.279 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:37:12.279 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.279 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.280 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:37:12.280 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.521329+0000", 2024-09-13T14:37:12.280 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:37:12.280 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.280 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.280 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:37:12.281 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.521330+0000", 2024-09-13T14:37:12.281 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.9900000000000001e-07 2024-09-13T14:37:12.281 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.281 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.282 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:37:12.282 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.521332+0000", 2024-09-13T14:37:12.282 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.8369999999999999e-06 2024-09-13T14:37:12.282 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.282 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.282 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:37:12.283 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.521338+0000", 2024-09-13T14:37:12.283 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.6309999999999997e-06 2024-09-13T14:37:12.283 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.283 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.283 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:37:12.284 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.521342+0000", 2024-09-13T14:37:12.284 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.27e-06 2024-09-13T14:37:12.284 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.284 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.284 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.284 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.521398+0000", 2024-09-13T14:37:12.285 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.6119999999999998e-05 2024-09-13T14:37:12.285 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.285 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.285 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:37:12.285 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.521523+0000", 2024-09-13T14:37:12.285 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000125289 2024-09-13T14:37:12.285 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.285 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.285 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:37:12.285 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.521635+0000", 2024-09-13T14:37:12.285 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000112463 2024-09-13T14:37:12.285 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.285 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.286 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:37:12.286 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.525876+0000", 2024-09-13T14:37:12.286 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0042402200000000003 2024-09-13T14:37:12.286 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.286 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.286 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.286 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.537705+0000", 2024-09-13T14:37:12.286 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.011829223 2024-09-13T14:37:12.286 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.286 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.286 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:37:12.286 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.537716+0000", 2024-09-13T14:37:12.286 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1018e-05 2024-09-13T14:37:12.287 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.287 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.287 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.287 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.828453+0000", 2024-09-13T14:37:12.287 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.290736671 2024-09-13T14:37:12.287 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.287 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.287 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:37:12.287 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.828461+0000", 2024-09-13T14:37:12.287 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.6780000000000006e-06 2024-09-13T14:37:12.287 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.287 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.287 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.288 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:44.995434+0000", 2024-09-13T14:37:12.288 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1669728079999999 2024-09-13T14:37:12.288 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.288 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.288 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:37:12.288 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:44.995511+0000", 2024-09-13T14:37:12.288 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.7322000000000003e-05 2024-09-13T14:37:12.288 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:37:12.288 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:37:12.288 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:37:12.288 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.289 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.289 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35037.0:163517 6.4 6:202dcf30:::benchmark_data_smithi102_194590_object52893:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-09-13T14:37:12.289 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:33:43.555877+0000", 2024-09-13T14:37:12.289 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 208.71550382000001, 2024-09-13T14:37:12.289 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.54220369, 2024-09-13T14:37:12.289 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:37:12.289 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:37:12.289 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:12.289 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:37:12.289 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35037", 2024-09-13T14:37:12.289 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:37:12.289 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 163517 2024-09-13T14:37:12.289 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.290 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:37:12.290 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.290 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:37:12.290 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.555877+0000", 2024-09-13T14:37:12.290 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:37:12.290 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.290 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.290 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:37:12.290 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.555877+0000", 2024-09-13T14:37:12.290 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:37:12.290 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.290 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.290 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:37:12.291 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.555878+0000", 2024-09-13T14:37:12.291 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2720000000000001e-06 2024-09-13T14:37:12.291 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.291 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.291 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:37:12.291 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.555883+0000", 2024-09-13T14:37:12.291 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.3420000000000001e-06 2024-09-13T14:37:12.291 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.291 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.291 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:37:12.291 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.555893+0000", 2024-09-13T14:37:12.291 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0564000000000001e-05 2024-09-13T14:37:12.291 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.292 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.292 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:37:12.292 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.555901+0000", 2024-09-13T14:37:12.292 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.4939999999999997e-06 2024-09-13T14:37:12.292 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.292 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.292 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.292 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.555965+0000", 2024-09-13T14:37:12.292 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.3816e-05 2024-09-13T14:37:12.292 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.292 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.292 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:37:12.293 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.556003+0000", 2024-09-13T14:37:12.293 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.8167000000000002e-05 2024-09-13T14:37:12.293 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.293 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.293 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.293 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.828407+0000", 2024-09-13T14:37:12.293 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.27240448900000003 2024-09-13T14:37:12.293 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.293 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.293 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:37:12.293 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.828417+0000", 2024-09-13T14:37:12.293 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.7049999999999994e-06 2024-09-13T14:37:12.293 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.294 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.294 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.294 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.033750+0000", 2024-09-13T14:37:12.294 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2053327650000001 2024-09-13T14:37:12.294 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.294 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.294 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:37:12.294 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.036001+0000", 2024-09-13T14:37:12.294 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0022508929999999999 2024-09-13T14:37:12.294 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.294 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.294 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,5,6", 2024-09-13T14:37:12.294 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.036155+0000", 2024-09-13T14:37:12.294 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00015404000000000001 2024-09-13T14:37:12.295 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.295 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.295 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:37:12.295 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.050797+0000", 2024-09-13T14:37:12.295 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.014642471000000001 2024-09-13T14:37:12.295 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.295 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.295 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:12.295 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.069588+0000", 2024-09-13T14:37:12.295 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.018790478999999999 2024-09-13T14:37:12.295 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.295 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.295 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:12.295 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.074823+0000", 2024-09-13T14:37:12.296 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.005235033 2024-09-13T14:37:12.296 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.296 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.296 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:12.296 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.097906+0000", 2024-09-13T14:37:12.296 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.023083372000000001 2024-09-13T14:37:12.296 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.296 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.296 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:37:12.296 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.097968+0000", 2024-09-13T14:37:12.296 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.2368999999999999e-05 2024-09-13T14:37:12.296 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.296 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.296 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:37:12.297 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.098081+0000", 2024-09-13T14:37:12.297 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000112419 2024-09-13T14:37:12.297 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:37:12.297 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:37:12.297 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:37:12.297 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.297 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.297 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35037.0:163519 6.4 6:2757b5a3:::benchmark_data_smithi102_194590_object52895:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-09-13T14:37:12.297 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:33:43.557183+0000", 2024-09-13T14:37:12.297 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 208.71419836999999, 2024-09-13T14:37:12.297 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.541098818, 2024-09-13T14:37:12.297 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:37:12.297 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:37:12.297 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:12.298 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:37:12.298 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35037", 2024-09-13T14:37:12.298 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:37:12.298 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 163519 2024-09-13T14:37:12.298 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.298 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:37:12.298 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.298 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:37:12.298 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.557183+0000", 2024-09-13T14:37:12.298 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:37:12.298 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.298 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.298 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:37:12.298 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.557183+0000", 2024-09-13T14:37:12.299 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:37:12.299 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.299 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.299 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:37:12.299 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.557183+0000", 2024-09-13T14:37:12.299 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.9299999999999997e-07 2024-09-13T14:37:12.299 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.299 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.299 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:37:12.299 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.557187+0000", 2024-09-13T14:37:12.299 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.3230000000000002e-06 2024-09-13T14:37:12.299 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.299 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.299 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:37:12.300 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.557196+0000", 2024-09-13T14:37:12.300 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.7399999999999999e-06 2024-09-13T14:37:12.300 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.300 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.300 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:37:12.300 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.557201+0000", 2024-09-13T14:37:12.300 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.1630000000000004e-06 2024-09-13T14:37:12.300 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.300 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.300 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.300 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.557240+0000", 2024-09-13T14:37:12.300 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.8197e-05 2024-09-13T14:37:12.300 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.301 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.301 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:37:12.301 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.557245+0000", 2024-09-13T14:37:12.301 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.6590000000000004e-06 2024-09-13T14:37:12.301 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.301 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.301 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.301 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.828516+0000", 2024-09-13T14:37:12.301 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.27127105800000001 2024-09-13T14:37:12.301 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.301 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.301 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:37:12.301 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.828524+0000", 2024-09-13T14:37:12.302 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.3080000000000002e-06 2024-09-13T14:37:12.302 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.302 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.302 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.302 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.045895+0000", 2024-09-13T14:37:12.302 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.217371282 2024-09-13T14:37:12.302 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.302 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.302 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:37:12.302 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.046020+0000", 2024-09-13T14:37:12.302 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00012525899999999999 2024-09-13T14:37:12.302 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.302 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.303 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,5,6", 2024-09-13T14:37:12.303 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.046165+0000", 2024-09-13T14:37:12.303 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000144764 2024-09-13T14:37:12.303 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.303 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.303 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:37:12.303 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.050805+0000", 2024-09-13T14:37:12.303 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0046395940000000004 2024-09-13T14:37:12.303 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.303 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.303 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:12.303 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.069631+0000", 2024-09-13T14:37:12.303 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.018826664 2024-09-13T14:37:12.303 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.304 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.304 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:12.304 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.074871+0000", 2024-09-13T14:37:12.304 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.005240102 2024-09-13T14:37:12.304 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.304 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.304 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:12.304 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.098129+0000", 2024-09-13T14:37:12.304 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.023257654999999999 2024-09-13T14:37:12.304 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.304 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.304 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:37:12.305 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.098191+0000", 2024-09-13T14:37:12.305 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.2248000000000004e-05 2024-09-13T14:37:12.305 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.305 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.305 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:37:12.305 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.098281+0000", 2024-09-13T14:37:12.305 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.0109000000000001e-05 2024-09-13T14:37:12.305 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:37:12.305 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:37:12.305 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:37:12.305 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.306 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.306 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35037.0:163528 6.c 6:3ebf1a70:::benchmark_data_smithi102_194590_object52904:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-09-13T14:37:12.306 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:33:43.572042+0000", 2024-09-13T14:37:12.306 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 208.699338756, 2024-09-13T14:37:12.306 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4738058709999999, 2024-09-13T14:37:12.306 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:37:12.306 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:37:12.306 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:12.306 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:37:12.306 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35037", 2024-09-13T14:37:12.306 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:37:12.307 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 163528 2024-09-13T14:37:12.307 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.307 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:37:12.307 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.307 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:37:12.307 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.572042+0000", 2024-09-13T14:37:12.307 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:37:12.307 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.307 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.307 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:37:12.307 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.572042+0000", 2024-09-13T14:37:12.307 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:37:12.307 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.307 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.308 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:37:12.308 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.572043+0000", 2024-09-13T14:37:12.308 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1039999999999999e-06 2024-09-13T14:37:12.308 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.308 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.308 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:37:12.308 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.572047+0000", 2024-09-13T14:37:12.308 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.2119999999999997e-06 2024-09-13T14:37:12.308 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.308 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.308 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:37:12.308 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.572058+0000", 2024-09-13T14:37:12.308 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0477e-05 2024-09-13T14:37:12.309 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.309 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.309 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:37:12.309 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.572063+0000", 2024-09-13T14:37:12.309 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.0470000000000001e-06 2024-09-13T14:37:12.309 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.309 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.309 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.309 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.572112+0000", 2024-09-13T14:37:12.309 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.8888999999999999e-05 2024-09-13T14:37:12.309 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.309 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.310 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:37:12.310 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.572119+0000", 2024-09-13T14:37:12.310 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.4889999999999998e-06 2024-09-13T14:37:12.310 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.310 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.310 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.310 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.828485+0000", 2024-09-13T14:37:12.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:12.313 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:12.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:12.313 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.25636578100000001 2024-09-13T14:37:12.313 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.313 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.313 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:37:12.313 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.828492+0000", 2024-09-13T14:37:12.313 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.635e-06 2024-09-13T14:37:12.314 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.314 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.314 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.314 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:44.995745+0000", 2024-09-13T14:37:12.314 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1672535019999999 2024-09-13T14:37:12.314 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.314 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.314 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:37:12.314 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:44.995909+0000", 2024-09-13T14:37:12.314 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000163377 2024-09-13T14:37:12.314 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.314 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.314 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 5,6,7", 2024-09-13T14:37:12.314 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:44.996387+0000", 2024-09-13T14:37:12.315 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00047786500000000001 2024-09-13T14:37:12.315 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.315 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.315 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:37:12.315 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.006654+0000", 2024-09-13T14:37:12.315 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.010267837 2024-09-13T14:37:12.315 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.315 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.315 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:12.315 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.045545+0000", 2024-09-13T14:37:12.315 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.038890419000000002 2024-09-13T14:37:12.315 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.315 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.315 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:12.316 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.045595+0000", 2024-09-13T14:37:12.316 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.0451000000000002e-05 2024-09-13T14:37:12.316 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.316 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.316 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:12.316 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.045656+0000", 2024-09-13T14:37:12.316 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.0436000000000002e-05 2024-09-13T14:37:12.316 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.316 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.316 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:37:12.316 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.045725+0000", 2024-09-13T14:37:12.316 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.9461999999999993e-05 2024-09-13T14:37:12.317 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.317 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.317 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:37:12.317 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.045848+0000", 2024-09-13T14:37:12.317 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00012288800000000001 2024-09-13T14:37:12.317 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:37:12.317 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:37:12.317 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:37:12.317 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.317 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.317 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35037.0:163530 6.7 6:e0b0b793:::benchmark_data_smithi102_194590_object52906:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-09-13T14:37:12.317 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:33:43.577667+0000", 2024-09-13T14:37:12.317 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 208.693713949, 2024-09-13T14:37:12.317 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.492772931, 2024-09-13T14:37:12.318 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:37:12.318 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:37:12.318 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:12.318 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:37:12.318 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35037", 2024-09-13T14:37:12.318 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:37:12.318 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 163530 2024-09-13T14:37:12.318 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.318 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:37:12.318 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.318 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:37:12.318 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.577667+0000", 2024-09-13T14:37:12.318 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:37:12.318 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.319 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.319 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:37:12.319 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.577667+0000", 2024-09-13T14:37:12.319 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:37:12.319 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.319 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.319 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:37:12.319 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.577668+0000", 2024-09-13T14:37:12.319 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1000000000000001e-06 2024-09-13T14:37:12.319 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.319 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.319 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:37:12.319 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.577673+0000", 2024-09-13T14:37:12.319 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.6e-06 2024-09-13T14:37:12.320 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.320 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.320 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:37:12.320 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.577683+0000", 2024-09-13T14:37:12.320 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0562e-05 2024-09-13T14:37:12.320 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.320 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.320 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:37:12.320 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.577690+0000", 2024-09-13T14:37:12.320 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.1110000000000002e-06 2024-09-13T14:37:12.320 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.320 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.321 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.321 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.577748+0000", 2024-09-13T14:37:12.321 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.7583000000000001e-05 2024-09-13T14:37:12.321 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.321 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.321 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:37:12.321 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.577756+0000", 2024-09-13T14:37:12.321 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.0690000000000002e-06 2024-09-13T14:37:12.321 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.321 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.321 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.321 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.829762+0000", 2024-09-13T14:37:12.321 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.25200604700000001 2024-09-13T14:37:12.322 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.322 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.322 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:37:12.322 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.829771+0000", 2024-09-13T14:37:12.322 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.4369999999999996e-06 2024-09-13T14:37:12.322 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.322 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.322 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.322 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.032584+0000", 2024-09-13T14:37:12.322 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2028129830000001 2024-09-13T14:37:12.322 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.322 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.322 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:37:12.323 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.034765+0000", 2024-09-13T14:37:12.323 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.002180689 2024-09-13T14:37:12.323 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.323 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.323 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,5,7", 2024-09-13T14:37:12.323 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.035026+0000", 2024-09-13T14:37:12.323 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00026036199999999998 2024-09-13T14:37:12.323 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.323 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.323 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:37:12.323 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.051976+0000", 2024-09-13T14:37:12.323 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.016950914000000001 2024-09-13T14:37:12.323 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.324 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.324 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:12.324 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.066190+0000", 2024-09-13T14:37:12.324 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.014213676 2024-09-13T14:37:12.324 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.324 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.324 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:12.324 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.068991+0000", 2024-09-13T14:37:12.324 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.002801245 2024-09-13T14:37:12.324 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.324 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.324 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:12.325 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.070312+0000", 2024-09-13T14:37:12.325 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.001320983 2024-09-13T14:37:12.325 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.325 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.325 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:37:12.325 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.070347+0000", 2024-09-13T14:37:12.325 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.5150000000000001e-05 2024-09-13T14:37:12.325 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.325 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.325 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:37:12.325 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.070440+0000", 2024-09-13T14:37:12.325 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.2419999999999999e-05 2024-09-13T14:37:12.325 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:37:12.325 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:37:12.326 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:37:12.326 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.326 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.326 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35037.0:163536 6.7 6:e42f79f9:::benchmark_data_smithi102_194590_object52912:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-09-13T14:37:12.326 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:33:43.596961+0000", 2024-09-13T14:37:12.326 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 208.67442030000001, 2024-09-13T14:37:12.326 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4737024060000001, 2024-09-13T14:37:12.326 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:37:12.326 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:37:12.326 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:12.326 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:37:12.326 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35037", 2024-09-13T14:37:12.326 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:37:12.326 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 163536 2024-09-13T14:37:12.327 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.327 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:37:12.327 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.327 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:37:12.327 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.596961+0000", 2024-09-13T14:37:12.327 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:37:12.327 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.327 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.327 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:37:12.327 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.596961+0000", 2024-09-13T14:37:12.327 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:37:12.327 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.327 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.328 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:37:12.328 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.596962+0000", 2024-09-13T14:37:12.328 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4720000000000001e-06 2024-09-13T14:37:12.328 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.328 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.328 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:37:12.328 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.596967+0000", 2024-09-13T14:37:12.328 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.8570000000000003e-06 2024-09-13T14:37:12.328 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.328 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.328 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:37:12.328 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.596979+0000", 2024-09-13T14:37:12.328 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1676e-05 2024-09-13T14:37:12.329 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.329 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.329 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:37:12.329 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.596985+0000", 2024-09-13T14:37:12.329 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.8029999999999999e-06 2024-09-13T14:37:12.329 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.329 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.329 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.329 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.597047+0000", 2024-09-13T14:37:12.329 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.1739e-05 2024-09-13T14:37:12.329 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.329 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.329 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:37:12.329 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.597057+0000", 2024-09-13T14:37:12.330 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0251000000000001e-05 2024-09-13T14:37:12.330 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.330 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.330 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.330 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.829797+0000", 2024-09-13T14:37:12.330 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.23274002599999999 2024-09-13T14:37:12.330 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.330 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.330 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:37:12.330 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.829805+0000", 2024-09-13T14:37:12.330 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.4649999999999997e-06 2024-09-13T14:37:12.330 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.330 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.331 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.331 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.050110+0000", 2024-09-13T14:37:12.331 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.220305164 2024-09-13T14:37:12.331 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.331 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.331 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:37:12.331 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.050178+0000", 2024-09-13T14:37:12.331 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.8369999999999998e-05 2024-09-13T14:37:12.331 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.331 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.331 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,5,7", 2024-09-13T14:37:12.331 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.050251+0000", 2024-09-13T14:37:12.331 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.2557000000000003e-05 2024-09-13T14:37:12.331 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.332 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.332 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:37:12.332 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.052130+0000", 2024-09-13T14:37:12.332 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0018792450000000001 2024-09-13T14:37:12.332 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.332 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.332 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:12.332 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.066922+0000", 2024-09-13T14:37:12.332 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.014791759999999999 2024-09-13T14:37:12.332 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.332 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.332 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:12.332 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.069384+0000", 2024-09-13T14:37:12.333 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.002461773 2024-09-13T14:37:12.333 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.333 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.333 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:12.333 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.070483+0000", 2024-09-13T14:37:12.333 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0010996560000000001 2024-09-13T14:37:12.333 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.333 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.333 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:37:12.333 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.070540+0000", 2024-09-13T14:37:12.333 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.6307999999999997e-05 2024-09-13T14:37:12.333 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.333 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.333 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:37:12.334 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.070663+0000", 2024-09-13T14:37:12.334 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00012328400000000001 2024-09-13T14:37:12.334 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:37:12.334 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:37:12.334 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:37:12.334 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.334 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.334 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35037.0:209609 6.b 6:d3b1c0fa:::benchmark_data_smithi102_194590_object98985:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-13T14:37:12.334 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:34:42.286860+0000", 2024-09-13T14:37:12.334 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 149.98452110599999, 2024-09-13T14:37:12.334 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.5313817680000001, 2024-09-13T14:37:12.334 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:37:12.334 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:37:12.335 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "reached pg", 2024-09-13T14:37:12.335 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:37:12.335 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35037", 2024-09-13T14:37:12.335 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:37:12.335 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 209609 2024-09-13T14:37:12.335 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.335 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:37:12.335 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.335 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:37:12.335 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.286860+0000", 2024-09-13T14:37:12.335 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:37:12.335 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.335 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.336 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:37:12.336 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.286860+0000", 2024-09-13T14:37:12.336 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:37:12.336 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.336 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.336 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:37:12.336 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.286861+0000", 2024-09-13T14:37:12.336 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.78e-07 2024-09-13T14:37:12.336 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.336 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.336 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:37:12.336 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.286865+0000", 2024-09-13T14:37:12.336 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.1330000000000001e-06 2024-09-13T14:37:12.337 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.337 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.337 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:37:12.337 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.286884+0000", 2024-09-13T14:37:12.337 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.8777999999999999e-05 2024-09-13T14:37:12.337 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.337 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.337 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:37:12.337 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.286891+0000", 2024-09-13T14:37:12.337 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.25e-06 2024-09-13T14:37:12.337 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.337 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.337 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.338 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.286949+0000", 2024-09-13T14:37:12.338 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.8517000000000002e-05 2024-09-13T14:37:12.338 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.338 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.338 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:37:12.338 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.287058+0000", 2024-09-13T14:37:12.338 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00010861099999999999 2024-09-13T14:37:12.338 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.338 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.338 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:37:12.338 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.287171+0000", 2024-09-13T14:37:12.338 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000113503 2024-09-13T14:37:12.338 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.338 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.339 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:37:12.339 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.290197+0000", 2024-09-13T14:37:12.339 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0030256430000000002 2024-09-13T14:37:12.339 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.339 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.339 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.339 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.302327+0000", 2024-09-13T14:37:12.339 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.012129707999999999 2024-09-13T14:37:12.339 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.339 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.339 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:37:12.339 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.302335+0000", 2024-09-13T14:37:12.339 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.5939999999999994e-06 2024-09-13T14:37:12.340 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.340 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.340 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.340 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.818191+0000", 2024-09-13T14:37:12.340 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.5158554180000001 2024-09-13T14:37:12.340 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.340 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.340 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:37:12.340 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.818242+0000", 2024-09-13T14:37:12.340 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.0735000000000001e-05 2024-09-13T14:37:12.340 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:37:12.340 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:37:12.340 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:37:12.341 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.341 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.341 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35037.0:209615 6.b 6:ddcfd241:::benchmark_data_smithi102_194590_object98991:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-13T14:37:12.341 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:34:42.293446+0000", 2024-09-13T14:37:12.341 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 149.97793449, 2024-09-13T14:37:12.341 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.533916812, 2024-09-13T14:37:12.341 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:37:12.341 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:37:12.341 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "reached pg", 2024-09-13T14:37:12.341 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:37:12.341 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35037", 2024-09-13T14:37:12.341 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:37:12.341 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 209615 2024-09-13T14:37:12.342 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.342 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:37:12.342 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.342 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:37:12.342 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.293446+0000", 2024-09-13T14:37:12.342 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:37:12.342 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.342 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.342 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:37:12.342 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.293446+0000", 2024-09-13T14:37:12.342 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:37:12.342 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.342 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.342 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:37:12.343 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.293447+0000", 2024-09-13T14:37:12.343 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.7899999999999999e-07 2024-09-13T14:37:12.343 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.343 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.343 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:37:12.343 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.293449+0000", 2024-09-13T14:37:12.343 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.5799999999999999e-06 2024-09-13T14:37:12.343 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.343 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.343 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:37:12.343 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.293456+0000", 2024-09-13T14:37:12.343 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.6900000000000003e-06 2024-09-13T14:37:12.343 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.343 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.344 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:37:12.344 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.293461+0000", 2024-09-13T14:37:12.344 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.1490000000000002e-06 2024-09-13T14:37:12.344 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.344 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.344 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.344 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.294942+0000", 2024-09-13T14:37:12.344 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0014804849999999999 2024-09-13T14:37:12.344 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.344 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.344 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:37:12.344 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.295068+0000", 2024-09-13T14:37:12.344 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00012669800000000001 2024-09-13T14:37:12.345 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.345 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.345 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:37:12.345 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.295202+0000", 2024-09-13T14:37:12.345 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00013326200000000001 2024-09-13T14:37:12.345 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.345 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.345 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:37:12.345 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.298178+0000", 2024-09-13T14:37:12.345 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0029757910000000002 2024-09-13T14:37:12.345 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.345 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.345 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.346 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.302728+0000", 2024-09-13T14:37:12.346 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0045509219999999998 2024-09-13T14:37:12.346 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.346 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.346 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:37:12.346 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.302738+0000", 2024-09-13T14:37:12.346 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.6569999999999992e-06 2024-09-13T14:37:12.346 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.346 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.346 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.346 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.818432+0000", 2024-09-13T14:37:12.346 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.5156941260000001 2024-09-13T14:37:12.346 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.347 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.347 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for missing object", 2024-09-13T14:37:12.347 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.818508+0000", 2024-09-13T14:37:12.347 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.5945999999999994e-05 2024-09-13T14:37:12.347 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.347 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.347 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.347 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.827232+0000", 2024-09-13T14:37:12.347 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0087233269999999995 2024-09-13T14:37:12.347 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.347 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.347 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:37:12.347 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.827363+0000", 2024-09-13T14:37:12.347 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0001317 2024-09-13T14:37:12.348 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:37:12.348 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:37:12.348 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:37:12.348 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.348 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.348 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35037.0:209631 6.b 6:d1535ee8:::benchmark_data_smithi102_194590_object99007:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-09-13T14:37:12.348 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:34:42.315630+0000", 2024-09-13T14:37:12.348 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 149.955751209, 2024-09-13T14:37:12.348 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.5154516149999999, 2024-09-13T14:37:12.348 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:37:12.348 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:37:12.348 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:12.348 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:37:12.348 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35037", 2024-09-13T14:37:12.349 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:37:12.349 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 209631 2024-09-13T14:37:12.349 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.349 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:37:12.349 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.349 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:37:12.349 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.315630+0000", 2024-09-13T14:37:12.349 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:37:12.349 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.349 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.349 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:37:12.349 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.315630+0000", 2024-09-13T14:37:12.349 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:37:12.350 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.350 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.350 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:37:12.350 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.315630+0000", 2024-09-13T14:37:12.350 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.5300000000000004e-07 2024-09-13T14:37:12.350 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.350 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.350 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:37:12.350 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.315634+0000", 2024-09-13T14:37:12.350 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.298e-06 2024-09-13T14:37:12.350 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.350 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.350 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:37:12.351 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.315642+0000", 2024-09-13T14:37:12.351 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.6330000000000002e-06 2024-09-13T14:37:12.351 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.351 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.351 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:37:12.351 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.315647+0000", 2024-09-13T14:37:12.351 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.0300000000000001e-06 2024-09-13T14:37:12.351 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.351 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.351 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.351 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.315711+0000", 2024-09-13T14:37:12.351 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.3946000000000001e-05 2024-09-13T14:37:12.351 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.352 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.352 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:37:12.352 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.315720+0000", 2024-09-13T14:37:12.352 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.2979999999999997e-06 2024-09-13T14:37:12.352 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.352 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.352 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.352 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.818542+0000", 2024-09-13T14:37:12.352 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.5028214499999999 2024-09-13T14:37:12.352 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.352 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.352 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:37:12.353 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.818629+0000", 2024-09-13T14:37:12.353 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.7359000000000002e-05 2024-09-13T14:37:12.353 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.353 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.353 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:37:12.353 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.818703+0000", 2024-09-13T14:37:12.353 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.3827999999999999e-05 2024-09-13T14:37:12.353 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.353 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.353 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:37:12.353 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.825562+0000", 2024-09-13T14:37:12.353 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0068591590000000001 2024-09-13T14:37:12.354 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.354 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.354 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:12.354 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.827336+0000", 2024-09-13T14:37:12.354 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0017741569999999999 2024-09-13T14:37:12.354 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.354 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.354 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:12.354 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.830889+0000", 2024-09-13T14:37:12.354 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0035525439999999999 2024-09-13T14:37:12.354 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.354 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.354 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:37:12.355 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.830949+0000", 2024-09-13T14:37:12.355 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.0396999999999998e-05 2024-09-13T14:37:12.355 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.355 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.355 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:37:12.355 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.831081+0000", 2024-09-13T14:37:12.355 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000131963 2024-09-13T14:37:12.355 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:37:12.355 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:37:12.355 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:37:12.355 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.355 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.355 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35037.0:209642 6.b 6:d884ca38:::benchmark_data_smithi102_194590_object99018:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-09-13T14:37:12.356 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:34:42.327971+0000", 2024-09-13T14:37:12.356 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 149.943409597, 2024-09-13T14:37:12.356 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.5035762109999999, 2024-09-13T14:37:12.356 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:37:12.356 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:37:12.356 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:12.356 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:37:12.356 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35037", 2024-09-13T14:37:12.356 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:37:12.356 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 209642 2024-09-13T14:37:12.356 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.356 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:37:12.356 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.357 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:37:12.357 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.327971+0000", 2024-09-13T14:37:12.357 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:37:12.357 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.357 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.357 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:37:12.357 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.327971+0000", 2024-09-13T14:37:12.357 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:37:12.357 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.357 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.357 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:37:12.357 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.327972+0000", 2024-09-13T14:37:12.358 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.187e-06 2024-09-13T14:37:12.358 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.358 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.358 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:37:12.358 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.327977+0000", 2024-09-13T14:37:12.358 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.9019999999999998e-06 2024-09-13T14:37:12.358 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.358 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.358 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:37:12.358 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.327987+0000", 2024-09-13T14:37:12.358 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.3200000000000006e-06 2024-09-13T14:37:12.358 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.358 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.359 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:37:12.359 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.327993+0000", 2024-09-13T14:37:12.359 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.3899999999999998e-06 2024-09-13T14:37:12.359 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.359 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.359 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.359 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.342702+0000", 2024-09-13T14:37:12.359 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.014708718000000001 2024-09-13T14:37:12.359 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.359 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.359 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:37:12.359 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.342711+0000", 2024-09-13T14:37:12.359 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.38e-06 2024-09-13T14:37:12.360 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.360 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.360 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.360 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.820544+0000", 2024-09-13T14:37:12.360 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.477833231 2024-09-13T14:37:12.360 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.360 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.360 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:37:12.360 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.820606+0000", 2024-09-13T14:37:12.360 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.1655999999999994e-05 2024-09-13T14:37:12.360 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.360 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.360 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:37:12.360 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.820692+0000", 2024-09-13T14:37:12.360 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.5803999999999994e-05 2024-09-13T14:37:12.361 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.361 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.361 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:37:12.361 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.825577+0000", 2024-09-13T14:37:12.361 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0048847370000000001 2024-09-13T14:37:12.361 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.361 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.361 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:12.361 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.831148+0000", 2024-09-13T14:37:12.361 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0055713409999999996 2024-09-13T14:37:12.361 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.361 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.361 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:12.362 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.831393+0000", 2024-09-13T14:37:12.362 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000244952 2024-09-13T14:37:12.362 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.362 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.362 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:37:12.362 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.831449+0000", 2024-09-13T14:37:12.362 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.6057999999999998e-05 2024-09-13T14:37:12.362 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.362 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.362 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:37:12.362 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.831548+0000", 2024-09-13T14:37:12.362 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.8535000000000004e-05 2024-09-13T14:37:12.362 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:37:12.363 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:37:12.363 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:37:12.363 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.363 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.363 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35037.0:209648 6.b 6:d5f8e920:::benchmark_data_smithi102_194590_object99024:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-09-13T14:37:12.363 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:34:42.330317+0000", 2024-09-13T14:37:12.363 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 149.941063719, 2024-09-13T14:37:12.363 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.5014201520000001, 2024-09-13T14:37:12.363 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:37:12.363 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:37:12.363 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:12.363 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:37:12.363 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35037", 2024-09-13T14:37:12.363 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:37:12.364 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 209648 2024-09-13T14:37:12.364 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.364 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:37:12.364 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.364 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:37:12.364 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.330317+0000", 2024-09-13T14:37:12.364 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:37:12.364 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.364 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.364 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:37:12.364 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.330317+0000", 2024-09-13T14:37:12.364 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:37:12.364 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.364 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.365 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:37:12.365 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.330318+0000", 2024-09-13T14:37:12.365 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.5899999999999995e-07 2024-09-13T14:37:12.365 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.365 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.365 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:37:12.365 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.330322+0000", 2024-09-13T14:37:12.365 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.6399999999999999e-06 2024-09-13T14:37:12.365 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.365 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.365 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:37:12.365 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.330331+0000", 2024-09-13T14:37:12.365 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.0089999999999996e-06 2024-09-13T14:37:12.365 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.366 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.366 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:37:12.366 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.330336+0000", 2024-09-13T14:37:12.366 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.8810000000000001e-06 2024-09-13T14:37:12.366 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.366 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.366 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.366 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.342737+0000", 2024-09-13T14:37:12.366 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.012400665999999999 2024-09-13T14:37:12.366 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.366 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.366 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:37:12.366 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.342743+0000", 2024-09-13T14:37:12.367 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.0589999999999999e-06 2024-09-13T14:37:12.367 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.367 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.367 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.367 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.821812+0000", 2024-09-13T14:37:12.367 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4790690500000001 2024-09-13T14:37:12.367 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.367 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.367 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:37:12.367 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.821893+0000", 2024-09-13T14:37:12.367 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.0785999999999995e-05 2024-09-13T14:37:12.367 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.367 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.367 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:37:12.368 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.821976+0000", 2024-09-13T14:37:12.368 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.3030999999999998e-05 2024-09-13T14:37:12.368 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.368 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.368 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:37:12.368 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.826889+0000", 2024-09-13T14:37:12.368 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0049131729999999998 2024-09-13T14:37:12.368 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.368 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.368 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:12.368 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.831205+0000", 2024-09-13T14:37:12.368 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0043161989999999997 2024-09-13T14:37:12.368 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.368 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.369 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:12.369 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.831608+0000", 2024-09-13T14:37:12.369 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00040285000000000001 2024-09-13T14:37:12.369 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.369 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.369 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:37:12.369 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.831654+0000", 2024-09-13T14:37:12.369 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.5537000000000001e-05 2024-09-13T14:37:12.369 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.369 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.369 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:37:12.369 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.831737+0000", 2024-09-13T14:37:12.369 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.3511999999999996e-05 2024-09-13T14:37:12.369 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:37:12.369 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:37:12.370 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:37:12.370 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.370 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.370 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35037.0:209661 6.b 6:d207e695:::benchmark_data_smithi102_194590_object99037:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-09-13T14:37:12.370 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:34:42.345443+0000", 2024-09-13T14:37:12.370 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 149.92593771400001, 2024-09-13T14:37:12.370 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4932702879999999, 2024-09-13T14:37:12.370 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:37:12.370 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:37:12.370 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:12.370 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:37:12.370 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35037", 2024-09-13T14:37:12.370 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:37:12.370 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 209661 2024-09-13T14:37:12.371 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.371 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:37:12.371 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.371 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:37:12.371 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.345443+0000", 2024-09-13T14:37:12.371 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:37:12.371 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.371 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.371 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:37:12.371 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.345443+0000", 2024-09-13T14:37:12.371 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:37:12.371 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.371 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.371 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:37:12.371 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.345444+0000", 2024-09-13T14:37:12.372 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.238e-06 2024-09-13T14:37:12.372 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.372 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.372 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:37:12.372 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.345449+0000", 2024-09-13T14:37:12.372 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.8119999999999998e-06 2024-09-13T14:37:12.372 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.372 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.372 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:37:12.372 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.345462+0000", 2024-09-13T14:37:12.372 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2509e-05 2024-09-13T14:37:12.372 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.372 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.372 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:37:12.372 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.345469+0000", 2024-09-13T14:37:12.373 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.6229999999999997e-06 2024-09-13T14:37:12.373 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.373 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.373 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.373 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.345531+0000", 2024-09-13T14:37:12.373 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.1715000000000001e-05 2024-09-13T14:37:12.373 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.373 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.373 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:37:12.373 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.345541+0000", 2024-09-13T14:37:12.373 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0208e-05 2024-09-13T14:37:12.373 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.373 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.374 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.374 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.823205+0000", 2024-09-13T14:37:12.374 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4776633889999999 2024-09-13T14:37:12.374 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.374 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.374 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:37:12.374 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.823272+0000", 2024-09-13T14:37:12.374 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.7537000000000001e-05 2024-09-13T14:37:12.374 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.374 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.374 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:37:12.374 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.823353+0000", 2024-09-13T14:37:12.374 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.0846999999999994e-05 2024-09-13T14:37:12.374 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.374 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.375 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:37:12.375 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.826902+0000", 2024-09-13T14:37:12.375 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.003548668 2024-09-13T14:37:12.375 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.375 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.375 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:12.375 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.831276+0000", 2024-09-13T14:37:12.375 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0043738040000000002 2024-09-13T14:37:12.375 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.375 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.375 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:12.375 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.838530+0000", 2024-09-13T14:37:12.375 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.007254295 2024-09-13T14:37:12.375 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.376 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.376 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:37:12.376 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.838608+0000", 2024-09-13T14:37:12.376 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.7769999999999996e-05 2024-09-13T14:37:12.376 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.376 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.376 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:37:12.376 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.838713+0000", 2024-09-13T14:37:12.376 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00010587299999999999 2024-09-13T14:37:12.376 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:37:12.376 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:37:12.376 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:37:12.376 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.376 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.377 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35037.0:209666 6.b 6:d70c3924:::benchmark_data_smithi102_194590_object99042:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-09-13T14:37:12.377 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:34:42.353256+0000", 2024-09-13T14:37:12.377 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 149.91812443800001, 2024-09-13T14:37:12.377 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4857159209999999, 2024-09-13T14:37:12.377 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:37:12.377 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:37:12.377 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:12.377 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:37:12.377 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35037", 2024-09-13T14:37:12.377 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:37:12.377 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 209666 2024-09-13T14:37:12.377 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.377 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:37:12.378 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.378 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:37:12.378 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.353256+0000", 2024-09-13T14:37:12.378 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:37:12.378 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.378 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.378 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:37:12.378 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.353256+0000", 2024-09-13T14:37:12.378 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:37:12.378 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.378 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.378 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:37:12.378 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.353258+0000", 2024-09-13T14:37:12.379 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.31e-06 2024-09-13T14:37:12.379 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.379 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.379 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:37:12.379 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.353263+0000", 2024-09-13T14:37:12.379 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.7389999999999999e-06 2024-09-13T14:37:12.379 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.379 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.379 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:37:12.379 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.353274+0000", 2024-09-13T14:37:12.379 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1503e-05 2024-09-13T14:37:12.379 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.379 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.380 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:37:12.380 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.353281+0000", 2024-09-13T14:37:12.380 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.798e-06 2024-09-13T14:37:12.380 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.380 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.380 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.380 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.353341+0000", 2024-09-13T14:37:12.380 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.0559000000000001e-05 2024-09-13T14:37:12.380 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.380 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.380 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:37:12.380 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.353350+0000", 2024-09-13T14:37:12.380 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.7590000000000003e-06 2024-09-13T14:37:12.381 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.381 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.381 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.381 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.824302+0000", 2024-09-13T14:37:12.381 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.470951951 2024-09-13T14:37:12.381 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.381 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.381 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:37:12.381 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.824392+0000", 2024-09-13T14:37:12.381 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.9974000000000003e-05 2024-09-13T14:37:12.381 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.381 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.381 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:37:12.382 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.824502+0000", 2024-09-13T14:37:12.382 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000109684 2024-09-13T14:37:12.382 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.382 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.382 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:37:12.382 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.827282+0000", 2024-09-13T14:37:12.382 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.002780724 2024-09-13T14:37:12.382 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.382 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.382 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:12.382 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.831328+0000", 2024-09-13T14:37:12.382 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0040450929999999996 2024-09-13T14:37:12.382 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.383 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.383 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:12.383 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.838752+0000", 2024-09-13T14:37:12.383 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0074246160000000002 2024-09-13T14:37:12.383 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.383 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.383 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:37:12.383 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.838879+0000", 2024-09-13T14:37:12.383 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000126444 2024-09-13T14:37:12.383 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.383 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.383 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:37:12.383 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.838972+0000", 2024-09-13T14:37:12.384 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.3766999999999999e-05 2024-09-13T14:37:12.384 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:37:12.384 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:37:12.384 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:37:12.384 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.384 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.384 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35037.0:209686 6.b 6:da8cdc62:::benchmark_data_smithi102_194590_object99062:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-09-13T14:37:12.384 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:34:42.373005+0000", 2024-09-13T14:37:12.384 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 149.89837551900001, 2024-09-13T14:37:12.384 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.466189841, 2024-09-13T14:37:12.384 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:37:12.384 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:37:12.384 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:12.384 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:37:12.385 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35037", 2024-09-13T14:37:12.385 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:37:12.385 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 209686 2024-09-13T14:37:12.385 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.385 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:37:12.385 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.385 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:37:12.385 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.373005+0000", 2024-09-13T14:37:12.385 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:37:12.385 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.385 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.385 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:37:12.386 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.373005+0000", 2024-09-13T14:37:12.386 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:37:12.386 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.386 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.386 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:37:12.386 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.373006+0000", 2024-09-13T14:37:12.386 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.3399999999999998e-07 2024-09-13T14:37:12.386 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.386 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.386 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:37:12.386 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.373010+0000", 2024-09-13T14:37:12.386 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.9960000000000004e-06 2024-09-13T14:37:12.386 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.387 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.387 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:37:12.387 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.373020+0000", 2024-09-13T14:37:12.387 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.9620000000000005e-06 2024-09-13T14:37:12.387 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.387 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.387 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:37:12.387 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.373027+0000", 2024-09-13T14:37:12.387 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.9550000000000003e-06 2024-09-13T14:37:12.387 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.387 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.387 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.388 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.373107+0000", 2024-09-13T14:37:12.388 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.9721000000000005e-05 2024-09-13T14:37:12.388 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.388 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.388 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:37:12.388 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.373116+0000", 2024-09-13T14:37:12.388 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.2369999999999998e-06 2024-09-13T14:37:12.388 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.388 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.388 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.388 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.825620+0000", 2024-09-13T14:37:12.388 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4525041169999999 2024-09-13T14:37:12.388 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.389 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.389 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:37:12.389 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.825697+0000", 2024-09-13T14:37:12.389 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.6364000000000003e-05 2024-09-13T14:37:12.389 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.389 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.389 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:37:12.389 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.825762+0000", 2024-09-13T14:37:12.389 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.5034000000000001e-05 2024-09-13T14:37:12.389 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.389 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.389 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:37:12.389 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.827360+0000", 2024-09-13T14:37:12.390 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0015988230000000001 2024-09-13T14:37:12.390 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.390 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.390 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:12.390 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.831804+0000", 2024-09-13T14:37:12.390 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.004443504 2024-09-13T14:37:12.390 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.390 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.390 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:12.390 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.839014+0000", 2024-09-13T14:37:12.390 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.007209689 2024-09-13T14:37:12.390 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.390 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.391 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:37:12.391 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.839069+0000", 2024-09-13T14:37:12.391 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.5025999999999998e-05 2024-09-13T14:37:12.391 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.391 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.391 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:37:12.391 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.839195+0000", 2024-09-13T14:37:12.391 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00012657900000000001 2024-09-13T14:37:12.391 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:37:12.391 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:37:12.391 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:37:12.391 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.391 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.391 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35037.0:209707 6.b 6:d253cbca:::benchmark_data_smithi102_194590_object99083:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-09-13T14:37:12.391 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:34:42.406983+0000", 2024-09-13T14:37:12.392 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 149.86439788499999, 2024-09-13T14:37:12.392 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.432391996, 2024-09-13T14:37:12.392 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:37:12.392 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:37:12.392 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:12.392 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:37:12.392 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35037", 2024-09-13T14:37:12.392 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:37:12.392 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 209707 2024-09-13T14:37:12.392 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.392 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:37:12.392 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.393 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:37:12.393 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.406983+0000", 2024-09-13T14:37:12.393 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:37:12.393 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.393 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.393 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:37:12.393 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.406983+0000", 2024-09-13T14:37:12.393 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:37:12.393 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.393 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.393 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:37:12.393 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.406984+0000", 2024-09-13T14:37:12.393 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.2200000000000004e-07 2024-09-13T14:37:12.393 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.393 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.394 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:37:12.394 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.406986+0000", 2024-09-13T14:37:12.394 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.2680000000000001e-06 2024-09-13T14:37:12.394 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.394 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.394 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:37:12.394 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.406993+0000", 2024-09-13T14:37:12.394 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.7549999999999997e-06 2024-09-13T14:37:12.394 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.394 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.394 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:37:12.394 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.406998+0000", 2024-09-13T14:37:12.395 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.3839999999999996e-06 2024-09-13T14:37:12.395 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.395 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.395 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.395 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.407057+0000", 2024-09-13T14:37:12.395 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.8683e-05 2024-09-13T14:37:12.395 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.395 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.395 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:37:12.395 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.407066+0000", 2024-09-13T14:37:12.395 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.2459999999999992e-06 2024-09-13T14:37:12.395 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.395 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.395 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.395 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.827401+0000", 2024-09-13T14:37:12.396 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4203347909999999 2024-09-13T14:37:12.396 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.396 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.396 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:37:12.396 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.827504+0000", 2024-09-13T14:37:12.396 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00010348699999999999 2024-09-13T14:37:12.396 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.396 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.396 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:37:12.396 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.827646+0000", 2024-09-13T14:37:12.396 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000141805 2024-09-13T14:37:12.396 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.396 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.396 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:37:12.397 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.831219+0000", 2024-09-13T14:37:12.397 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0035728579999999999 2024-09-13T14:37:12.397 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.397 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.397 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:12.397 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.831877+0000", 2024-09-13T14:37:12.397 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000658258 2024-09-13T14:37:12.397 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.397 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.397 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:12.397 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.839235+0000", 2024-09-13T14:37:12.397 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0073578790000000003 2024-09-13T14:37:12.397 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.397 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.398 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:37:12.398 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.839284+0000", 2024-09-13T14:37:12.398 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.9466999999999999e-05 2024-09-13T14:37:12.398 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.398 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.398 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:37:12.398 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.839375+0000", 2024-09-13T14:37:12.398 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.0593000000000005e-05 2024-09-13T14:37:12.398 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:37:12.398 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:37:12.398 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:37:12.398 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.398 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.399 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35037.0:209713 6.b 6:d78fe956:::benchmark_data_smithi102_194590_object99089:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-09-13T14:37:12.399 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:34:42.413630+0000", 2024-09-13T14:37:12.399 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 149.85775074099999, 2024-09-13T14:37:12.399 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4259566239999999, 2024-09-13T14:37:12.399 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:37:12.399 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:37:12.399 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:12.399 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:37:12.399 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35037", 2024-09-13T14:37:12.399 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:37:12.399 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 209713 2024-09-13T14:37:12.399 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.399 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:37:12.400 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.400 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:37:12.400 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.413630+0000", 2024-09-13T14:37:12.400 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:37:12.400 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.400 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.400 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:37:12.400 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.413630+0000", 2024-09-13T14:37:12.400 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:37:12.400 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.400 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.400 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:37:12.400 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.413631+0000", 2024-09-13T14:37:12.401 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.8000000000000004e-07 2024-09-13T14:37:12.401 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.401 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.401 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:37:12.401 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.413636+0000", 2024-09-13T14:37:12.401 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.6009999999999997e-06 2024-09-13T14:37:12.401 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.401 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.401 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:37:12.401 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.413645+0000", 2024-09-13T14:37:12.401 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.8349999999999999e-06 2024-09-13T14:37:12.401 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.401 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.401 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:37:12.401 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.413652+0000", 2024-09-13T14:37:12.402 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.9949999999999999e-06 2024-09-13T14:37:12.402 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.402 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.402 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.402 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.413710+0000", 2024-09-13T14:37:12.402 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.7067000000000001e-05 2024-09-13T14:37:12.402 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.402 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.402 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:37:12.402 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.413719+0000", 2024-09-13T14:37:12.402 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.2720000000000003e-06 2024-09-13T14:37:12.402 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.402 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.402 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.402 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.829138+0000", 2024-09-13T14:37:12.402 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.415419623 2024-09-13T14:37:12.403 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.403 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.403 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:37:12.403 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.829241+0000", 2024-09-13T14:37:12.403 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00010207799999999999 2024-09-13T14:37:12.403 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.403 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.403 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:37:12.403 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.829310+0000", 2024-09-13T14:37:12.403 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.9657000000000001e-05 2024-09-13T14:37:12.403 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.403 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.403 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:37:12.404 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.831226+0000", 2024-09-13T14:37:12.404 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.001915992 2024-09-13T14:37:12.404 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.404 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.404 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:12.404 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.839409+0000", 2024-09-13T14:37:12.404 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0081829480000000007 2024-09-13T14:37:12.404 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.404 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.404 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:12.404 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.839442+0000", 2024-09-13T14:37:12.404 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.2616000000000001e-05 2024-09-13T14:37:12.404 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.404 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.405 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:37:12.405 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.839494+0000", 2024-09-13T14:37:12.405 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.2330999999999998e-05 2024-09-13T14:37:12.405 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.405 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.405 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:37:12.405 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.839587+0000", 2024-09-13T14:37:12.405 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.2728999999999999e-05 2024-09-13T14:37:12.405 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:37:12.405 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:37:12.406 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:37:12.406 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.406 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.406 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35037.0:209722 6.b 6:d818efa2:::benchmark_data_smithi102_194590_object99098:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-09-13T14:37:12.406 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:34:42.424323+0000", 2024-09-13T14:37:12.406 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 149.847058008, 2024-09-13T14:37:12.406 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.437535912, 2024-09-13T14:37:12.406 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:37:12.406 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:37:12.406 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:12.406 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:37:12.406 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35037", 2024-09-13T14:37:12.406 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:37:12.406 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 209722 2024-09-13T14:37:12.406 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.407 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:37:12.407 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.407 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:37:12.407 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.424323+0000", 2024-09-13T14:37:12.407 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:37:12.407 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.407 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.407 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:37:12.407 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.424323+0000", 2024-09-13T14:37:12.407 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:37:12.407 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.407 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.407 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:37:12.407 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.424324+0000", 2024-09-13T14:37:12.407 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.26e-06 2024-09-13T14:37:12.407 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.408 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.408 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:37:12.408 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.424329+0000", 2024-09-13T14:37:12.408 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.7729999999999999e-06 2024-09-13T14:37:12.408 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.408 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.408 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:37:12.408 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.424340+0000", 2024-09-13T14:37:12.408 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1308000000000001e-05 2024-09-13T14:37:12.408 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.408 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.408 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:37:12.409 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.424350+0000", 2024-09-13T14:37:12.409 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.7000000000000003e-06 2024-09-13T14:37:12.409 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.409 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.409 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.409 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.424413+0000", 2024-09-13T14:37:12.409 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.3103999999999998e-05 2024-09-13T14:37:12.409 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.409 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.409 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:37:12.409 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.424423+0000", 2024-09-13T14:37:12.409 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0342e-05 2024-09-13T14:37:12.409 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.409 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.409 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.409 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.831930+0000", 2024-09-13T14:37:12.410 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.407506575 2024-09-13T14:37:12.410 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.410 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.410 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:37:12.410 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.832054+0000", 2024-09-13T14:37:12.410 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00012403699999999999 2024-09-13T14:37:12.410 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.410 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.410 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:37:12.410 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.832188+0000", 2024-09-13T14:37:12.410 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00013378199999999999 2024-09-13T14:37:12.410 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.410 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.411 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:37:12.411 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.841913+0000", 2024-09-13T14:37:12.411 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0097248040000000001 2024-09-13T14:37:12.411 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.411 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.411 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:12.411 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.849617+0000", 2024-09-13T14:37:12.411 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0077045969999999997 2024-09-13T14:37:12.411 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.411 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.411 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:12.411 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.861692+0000", 2024-09-13T14:37:12.411 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.012074684 2024-09-13T14:37:12.411 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.412 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.412 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:37:12.412 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.861748+0000", 2024-09-13T14:37:12.412 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.6372000000000002e-05 2024-09-13T14:37:12.412 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.412 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.412 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:37:12.412 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.861859+0000", 2024-09-13T14:37:12.412 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000110574 2024-09-13T14:37:12.413 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:37:12.413 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:37:12.413 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:37:12.413 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.413 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.413 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35379.0:76660 7.a 7:55e7e51b:::benchmark_data_smithi102_227395_object76659:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-09-13T14:37:12.413 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:36:13.172282+0000", 2024-09-13T14:37:12.413 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 59.099099154999998, 2024-09-13T14:37:12.413 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.7816652209999999, 2024-09-13T14:37:12.413 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:37:12.414 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:37:12.414 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:12.414 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:37:12.414 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35379", 2024-09-13T14:37:12.414 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:37:12.414 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 76660 2024-09-13T14:37:12.414 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.414 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:37:12.414 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.414 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:37:12.414 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.172282+0000", 2024-09-13T14:37:12.414 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:37:12.414 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.414 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.415 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:37:12.415 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.172282+0000", 2024-09-13T14:37:12.415 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:37:12.415 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.415 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.415 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:37:12.415 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.172283+0000", 2024-09-13T14:37:12.415 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.6080000000000001e-06 2024-09-13T14:37:12.415 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.415 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.415 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:37:12.415 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.172380+0000", 2024-09-13T14:37:12.415 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.6696999999999999e-05 2024-09-13T14:37:12.415 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.416 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.416 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:37:12.416 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.172393+0000", 2024-09-13T14:37:12.416 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3363999999999999e-05 2024-09-13T14:37:12.416 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.416 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.416 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:37:12.416 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.172405+0000", 2024-09-13T14:37:12.416 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1265e-05 2024-09-13T14:37:12.416 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.416 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.416 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.416 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.172471+0000", 2024-09-13T14:37:12.417 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.6078e-05 2024-09-13T14:37:12.417 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.417 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.417 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:37:12.417 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.172482+0000", 2024-09-13T14:37:12.417 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1392e-05 2024-09-13T14:37:12.417 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.417 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.417 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.417 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.863789+0000", 2024-09-13T14:37:12.417 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.69130646100000004 2024-09-13T14:37:12.417 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.417 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.418 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:37:12.418 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.863802+0000", 2024-09-13T14:37:12.418 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3599e-05 2024-09-13T14:37:12.418 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.418 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.418 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.418 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.944957+0000", 2024-09-13T14:37:12.418 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.08115447 2024-09-13T14:37:12.418 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.418 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.418 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:37:12.418 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.945093+0000", 2024-09-13T14:37:12.418 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000136496 2024-09-13T14:37:12.419 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.419 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.419 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 0,4,5", 2024-09-13T14:37:12.419 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.945201+0000", 2024-09-13T14:37:12.419 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000107774 2024-09-13T14:37:12.419 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.419 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.419 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:12.419 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.948055+0000", 2024-09-13T14:37:12.419 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0028542900000000002 2024-09-13T14:37:12.419 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.419 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.419 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:37:12.419 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.953464+0000", 2024-09-13T14:37:12.420 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0054091709999999999 2024-09-13T14:37:12.420 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.420 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.420 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:12.420 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.953740+0000", 2024-09-13T14:37:12.420 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00027606199999999999 2024-09-13T14:37:12.420 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.420 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.420 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:12.420 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.953807+0000", 2024-09-13T14:37:12.420 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.6389000000000004e-05 2024-09-13T14:37:12.420 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.420 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.420 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:37:12.421 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.953863+0000", 2024-09-13T14:37:12.421 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.5942000000000001e-05 2024-09-13T14:37:12.421 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.421 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.421 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:37:12.421 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.953947+0000", 2024-09-13T14:37:12.421 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.4163e-05 2024-09-13T14:37:12.421 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:37:12.421 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:37:12.421 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:37:12.421 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.421 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.421 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35379.0:76665 7.a 7:55f8d188:::benchmark_data_smithi102_227395_object76664:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-09-13T14:37:12.421 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:36:13.177195+0000", 2024-09-13T14:37:12.422 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 59.094186235999999, 2024-09-13T14:37:12.422 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.777108328, 2024-09-13T14:37:12.422 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:37:12.422 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:37:12.422 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:12.422 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:37:12.422 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35379", 2024-09-13T14:37:12.422 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:37:12.422 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 76665 2024-09-13T14:37:12.422 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.422 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:37:12.422 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.423 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:37:12.423 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.177195+0000", 2024-09-13T14:37:12.423 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:37:12.423 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.423 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.423 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:37:12.423 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.177195+0000", 2024-09-13T14:37:12.423 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:37:12.423 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.423 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.423 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:37:12.423 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.177197+0000", 2024-09-13T14:37:12.423 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.187e-06 2024-09-13T14:37:12.423 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.423 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.424 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:37:12.424 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.177223+0000", 2024-09-13T14:37:12.424 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.5701e-05 2024-09-13T14:37:12.424 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.424 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.424 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:37:12.424 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.177240+0000", 2024-09-13T14:37:12.424 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.6946999999999999e-05 2024-09-13T14:37:12.424 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.424 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.424 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:37:12.424 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.177250+0000", 2024-09-13T14:37:12.424 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0169999999999999e-05 2024-09-13T14:37:12.425 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.425 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.425 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.425 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.179843+0000", 2024-09-13T14:37:12.425 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.002593167 2024-09-13T14:37:12.425 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.425 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.425 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:37:12.425 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.179861+0000", 2024-09-13T14:37:12.425 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.8014000000000001e-05 2024-09-13T14:37:12.425 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.425 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.425 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.426 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.863827+0000", 2024-09-13T14:37:12.426 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.68396646999999999 2024-09-13T14:37:12.426 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.426 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.426 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:37:12.426 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.863835+0000", 2024-09-13T14:37:12.426 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.0700000000000007e-06 2024-09-13T14:37:12.426 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.426 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.426 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.426 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.948168+0000", 2024-09-13T14:37:12.426 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.084332954 2024-09-13T14:37:12.426 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.427 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.427 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:37:12.427 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.948293+0000", 2024-09-13T14:37:12.427 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00012477300000000001 2024-09-13T14:37:12.427 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.427 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.427 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 0,4,5", 2024-09-13T14:37:12.427 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.948388+0000", 2024-09-13T14:37:12.427 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.4470999999999997e-05 2024-09-13T14:37:12.427 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.427 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.427 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:37:12.428 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.953757+0000", 2024-09-13T14:37:12.428 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.005369342 2024-09-13T14:37:12.428 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.428 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.428 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:12.428 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.954003+0000", 2024-09-13T14:37:12.428 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000246457 2024-09-13T14:37:12.428 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.428 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.428 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:12.428 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.954064+0000", 2024-09-13T14:37:12.428 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.0908e-05 2024-09-13T14:37:12.428 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.429 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.429 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:12.429 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.954114+0000", 2024-09-13T14:37:12.429 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.0107e-05 2024-09-13T14:37:12.429 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.429 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.429 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:37:12.429 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.954179+0000", 2024-09-13T14:37:12.429 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.4505999999999995e-05 2024-09-13T14:37:12.429 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.429 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.429 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:37:12.429 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.954303+0000", 2024-09-13T14:37:12.430 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000124084 2024-09-13T14:37:12.430 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:37:12.430 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:37:12.430 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:37:12.430 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.430 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.430 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35379.0:76673 7.a 7:5e41d42e:::benchmark_data_smithi102_227395_object76672:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-09-13T14:37:12.430 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:36:13.187226+0000", 2024-09-13T14:37:12.430 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 59.084155275000001, 2024-09-13T14:37:12.430 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.7674439820000001, 2024-09-13T14:37:12.430 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:37:12.430 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:37:12.430 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:12.431 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:37:12.431 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35379", 2024-09-13T14:37:12.431 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:37:12.431 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 76673 2024-09-13T14:37:12.431 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.431 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:37:12.431 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.431 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:37:12.431 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.187226+0000", 2024-09-13T14:37:12.431 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:37:12.431 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.431 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.431 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:37:12.431 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.187226+0000", 2024-09-13T14:37:12.432 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:37:12.432 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.432 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.432 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:37:12.432 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.187227+0000", 2024-09-13T14:37:12.432 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.403e-06 2024-09-13T14:37:12.432 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.432 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.432 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:37:12.432 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.187249+0000", 2024-09-13T14:37:12.432 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.2436999999999999e-05 2024-09-13T14:37:12.432 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.432 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.433 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:37:12.433 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.187264+0000", 2024-09-13T14:37:12.433 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4698999999999999e-05 2024-09-13T14:37:12.433 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.433 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.433 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:37:12.433 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.187271+0000", 2024-09-13T14:37:12.433 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.2350000000000002e-06 2024-09-13T14:37:12.433 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.433 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.433 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.433 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.187331+0000", 2024-09-13T14:37:12.433 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.9843999999999998e-05 2024-09-13T14:37:12.434 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.434 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.434 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:37:12.434 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.187340+0000", 2024-09-13T14:37:12.434 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.1779999999999992e-06 2024-09-13T14:37:12.434 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.434 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.434 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.434 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.863887+0000", 2024-09-13T14:37:12.434 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.676547016 2024-09-13T14:37:12.434 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.434 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.434 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:37:12.434 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.863896+0000", 2024-09-13T14:37:12.434 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.6400000000000003e-06 2024-09-13T14:37:12.435 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.435 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.435 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:37:12.435 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.950494+0000", 2024-09-13T14:37:12.435 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.086598349 2024-09-13T14:37:12.435 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.435 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.435 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:37:12.435 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.950658+0000", 2024-09-13T14:37:12.435 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00016312999999999999 2024-09-13T14:37:12.435 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.435 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.435 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 0,4,5", 2024-09-13T14:37:12.435 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.950840+0000", 2024-09-13T14:37:12.436 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000182638 2024-09-13T14:37:12.436 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.436 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.436 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:37:12.436 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.954015+0000", 2024-09-13T14:37:12.436 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0031750849999999998 2024-09-13T14:37:12.436 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.436 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.436 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:12.436 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.954379+0000", 2024-09-13T14:37:12.436 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00036403800000000001 2024-09-13T14:37:12.436 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.436 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.436 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:12.437 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.954448+0000", 2024-09-13T14:37:12.437 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.8753999999999999e-05 2024-09-13T14:37:12.437 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.437 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.437 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:12.437 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.954496+0000", 2024-09-13T14:37:12.437 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.7596000000000003e-05 2024-09-13T14:37:12.437 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.437 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.437 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:37:12.437 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.954553+0000", 2024-09-13T14:37:12.437 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.7571999999999997e-05 2024-09-13T14:37:12.437 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:37:12.437 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:37:12.438 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:37:12.438 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.954670+0000", 2024-09-13T14:37:12.438 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00011636799999999999 2024-09-13T14:37:12.438 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:37:12.438 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:37:12.438 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:37:12.438 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:37:12.438 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:37:12.438 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:37:12.438 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:12.459 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:12.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:12.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:12.561 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:12.583 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:12.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:12.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:12.685 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:12.708 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:12.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:12.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:12.810 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:12.832 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:12.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:12.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:12.934 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:12.955 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:12.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:12.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:13.045 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 99 16 112812 112796 71.2014 0 - 0.0106664 2024-09-13T14:37:13.057 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:13.081 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:13.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:13.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:13.182 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:13.204 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:13.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:13.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:13.305 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:13.327 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:13.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:13.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:13.429 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:13.450 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:13.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:13.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:13.552 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:13.572 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:13.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:13.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:13.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:13 smithi102 ceph-mon[110183]: pgmap v734: 89 pgs: 1 activating+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 31 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 75363/338238 objects degraded (22.281%); 49837/338238 objects misplaced (14.734%); 12 MiB/s, 191 objects/s recovering 2024-09-13T14:37:13.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:13 smithi102 ceph-mon[115269]: pgmap v734: 89 pgs: 1 activating+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 31 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 75363/338238 objects degraded (22.281%); 49837/338238 objects misplaced (14.734%); 12 MiB/s, 191 objects/s recovering 2024-09-13T14:37:13.675 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:13.696 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:13.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:13.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:13.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:13 smithi195 ceph-mon[104262]: pgmap v734: 89 pgs: 1 activating+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 31 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 75363/338238 objects degraded (22.281%); 49837/338238 objects misplaced (14.734%); 12 MiB/s, 191 objects/s recovering 2024-09-13T14:37:13.792 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:37:13.798 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:13.820 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:13.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:13.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:13.921 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:13.948 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:13.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:13.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:14.045 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:2024-09-13T14:37:14.045096+0000 min lat: 0.00114883 max lat: 1.99103 avg lat: 0.0106664 2024-09-13T14:37:14.045 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:37:14.045 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 100 16 112812 112796 70.4894 0 - 0.0106664 2024-09-13T14:37:14.050 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:14.071 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:14.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:14.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:14.173 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:14.193 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:14.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:14.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:14.295 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:14.321 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:14.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:14.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:14.422 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:14.444 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:14.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:14.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:14.546 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:14.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:14.568 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:14.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:14.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:14 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 75363/338238 objects degraded (22.281%), 13 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-13T14:37:14.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:14 smithi102 ceph-mon[110183]: osdmap e229: 8 total, 7 up, 6 in 2024-09-13T14:37:14.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:14 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 75363/338238 objects degraded (22.281%), 13 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-13T14:37:14.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:14 smithi102 ceph-mon[115269]: osdmap e229: 8 total, 7 up, 6 in 2024-09-13T14:37:14.669 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:14.695 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:14.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:14.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:14.715 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T14:37:14.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:14 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 75363/338238 objects degraded (22.281%), 13 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-13T14:37:14.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:14 smithi195 ceph-mon[104262]: osdmap e229: 8 total, 7 up, 6 in 2024-09-13T14:37:14.797 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:14.818 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:14.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:14.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:14.919 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:14.949 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:14.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:14.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:15.013 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:37:15.045 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 101 16 112812 112796 69.7915 0 - 0.0106664 2024-09-13T14:37:15.051 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:15.078 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:15.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:15.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:15.180 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:15.207 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:15.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:15.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:15.309 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:15.336 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:15.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:15.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:15.432 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:15 smithi102 ceph-mon[110183]: pgmap v735: 89 pgs: 1 activating+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 31 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 75363/338238 objects degraded (22.281%); 48445/338238 objects misplaced (14.323%); 19 MiB/s, 297 objects/s recovering 2024-09-13T14:37:15.432 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:15 smithi102 ceph-mon[110183]: osdmap e230: 8 total, 7 up, 6 in 2024-09-13T14:37:15.432 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:15 smithi102 ceph-mon[115269]: pgmap v735: 89 pgs: 1 activating+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 31 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 75363/338238 objects degraded (22.281%); 48445/338238 objects misplaced (14.323%); 19 MiB/s, 297 objects/s recovering 2024-09-13T14:37:15.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:15 smithi102 ceph-mon[115269]: osdmap e230: 8 total, 7 up, 6 in 2024-09-13T14:37:15.438 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:15.459 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:15.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:15.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:15.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:15 smithi195 ceph-mon[104262]: pgmap v735: 89 pgs: 1 activating+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 31 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 75363/338238 objects degraded (22.281%); 48445/338238 objects misplaced (14.323%); 19 MiB/s, 297 objects/s recovering 2024-09-13T14:37:15.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:15 smithi195 ceph-mon[104262]: osdmap e230: 8 total, 7 up, 6 in 2024-09-13T14:37:15.561 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:15.583 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:15.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:15.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:15.684 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:15.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:15.709 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:15.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:15.810 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:15.835 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:15.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:15.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:15.938 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:15.965 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:15.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:15.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:16.045 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 102 16 112812 112796 69.1072 0 - 0.0106664 2024-09-13T14:37:16.067 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:16.093 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:16.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:16.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:16.195 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:16.221 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:16.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:16.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:16.323 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:16.350 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:16.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:16.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:16.453 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:16.479 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:16.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:16.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:16.581 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:16.606 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:16.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:16.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:16.708 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:16.734 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:16.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:16.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:16.836 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:16.862 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:16.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:16.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:16.918 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:37:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:37:16] "GET /metrics HTTP/1.1" 200 36578 "" "Prometheus/2.51.0" 2024-09-13T14:37:16.964 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:16.981 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:37:16.981 INFO:teuthology.orchestra.run.smithi102.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-13T14:37:16.981 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:37:16.989 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:16.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:16.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:17.045 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 103 16 112812 112796 68.4362 0 - 0.0106664 2024-09-13T14:37:17.091 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:17.113 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:17.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:17.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:17.215 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:17.241 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:17.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:17.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:17.343 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:17.366 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:17.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:17.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:17.468 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:17.492 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:17.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:17.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:17.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:17 smithi195 ceph-mon[104262]: pgmap v738: 89 pgs: 1 activating+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 31 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 75363/338238 objects degraded (22.281%); 48445/338238 objects misplaced (14.323%); 11 MiB/s, 173 objects/s recovering 2024-09-13T14:37:17.578 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:17 smithi102 ceph-mon[115269]: pgmap v738: 89 pgs: 1 activating+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 31 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 75363/338238 objects degraded (22.281%); 48445/338238 objects misplaced (14.323%); 11 MiB/s, 173 objects/s recovering 2024-09-13T14:37:17.578 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:17 smithi102 ceph-mon[110183]: pgmap v738: 89 pgs: 1 activating+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 31 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 75363/338238 objects degraded (22.281%); 48445/338238 objects misplaced (14.323%); 11 MiB/s, 173 objects/s recovering 2024-09-13T14:37:17.594 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:17.617 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:17.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:17.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:17.719 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:17.742 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:17.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:17.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:17.844 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:17.867 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:17.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:17.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:17.969 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:17.992 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:17.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:17.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:18.045 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 104 16 112812 112796 67.7782 0 - 0.0106664 2024-09-13T14:37:18.094 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:18.121 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:18.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:18.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:18.223 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:18.245 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:18.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:18.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:18.347 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:18.373 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:18.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:18.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:18.475 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:18.497 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:18.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:18.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:18.599 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:18.620 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:18.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:18.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:18.722 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:18.743 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:18.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:18.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:18.845 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:18.869 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:18.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:18.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:18.971 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:18.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:18.994 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:18.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:19.045 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 105 16 112812 112796 67.1327 0 - 0.0106664 2024-09-13T14:37:19.094 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:19.116 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:19.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:19.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:19.218 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:19.243 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:19.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:19.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:19.277 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is unset 2024-09-13T14:37:19.345 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:19.371 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:19.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:19.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:19.473 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:19.495 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:19.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:19.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:19.590 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:19 smithi195 ceph-mon[104262]: pgmap v739: 89 pgs: 1 activating+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 31 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 75363/338238 objects degraded (22.281%); 48174/338238 objects misplaced (14.243%); 13 MiB/s, 207 objects/s recovering 2024-09-13T14:37:19.590 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:19 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 75363/338238 objects degraded (22.281%), 13 pgs degraded, 24 pgs undersized (PG_DEGRADED) 2024-09-13T14:37:19.590 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:19 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1032406535' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:37:19.590 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:19 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:37:19.596 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:19.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:19.619 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:19.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:19.667 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:19 smithi102 ceph-mon[110183]: pgmap v739: 89 pgs: 1 activating+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 31 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 75363/338238 objects degraded (22.281%); 48174/338238 objects misplaced (14.243%); 13 MiB/s, 207 objects/s recovering 2024-09-13T14:37:19.667 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:19 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 75363/338238 objects degraded (22.281%), 13 pgs degraded, 24 pgs undersized (PG_DEGRADED) 2024-09-13T14:37:19.667 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:19 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1032406535' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:37:19.667 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:19 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:37:19.667 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:19 smithi102 ceph-mon[115269]: pgmap v739: 89 pgs: 1 activating+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 31 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 75363/338238 objects degraded (22.281%); 48174/338238 objects misplaced (14.243%); 13 MiB/s, 207 objects/s recovering 2024-09-13T14:37:19.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:19 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 75363/338238 objects degraded (22.281%), 13 pgs degraded, 24 pgs undersized (PG_DEGRADED) 2024-09-13T14:37:19.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:19 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1032406535' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:37:19.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:19 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:37:19.719 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:19.740 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:19.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:19.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:19.842 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:19.863 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:19.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:19.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:19.965 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:19.992 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:19.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:19.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:20.045 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 106 16 112812 112796 66.4994 0 - 0.0106664 2024-09-13T14:37:20.094 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:20.116 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:20.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:20.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:20.217 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:20.245 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:20.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:20.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:20.347 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:20.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:20.369 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:20.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:20.470 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:20.491 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:20.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:20.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:20.593 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:20.614 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:20.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:20.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:20.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:20 smithi102 ceph-mon[110183]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:37:20.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:20 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:37:20.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:20 smithi102 ceph-mon[110183]: osdmap e231: 8 total, 7 up, 6 in 2024-09-13T14:37:20.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:20 smithi102 ceph-mon[115269]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:37:20.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:20 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:37:20.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:20 smithi102 ceph-mon[115269]: osdmap e231: 8 total, 7 up, 6 in 2024-09-13T14:37:20.716 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:20.737 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:20.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:20.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:20.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:20 smithi195 ceph-mon[104262]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:37:20.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:20 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:37:20.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:20 smithi195 ceph-mon[104262]: osdmap e231: 8 total, 7 up, 6 in 2024-09-13T14:37:20.839 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:20.867 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:20.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:20.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:20.969 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:20.992 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:20.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:20.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:21.046 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 107 16 112812 112796 65.8779 0 - 0.0106664 2024-09-13T14:37:21.096 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:21.118 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:21.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:21.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:21.220 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:21.245 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:21.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:21.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:21.347 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:21.373 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:21.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:21.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:21.475 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:21.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:21.497 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:21.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:21.561 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:21 smithi102 ceph-mon[115269]: pgmap v741: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 activating+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 31 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 72551/338238 objects degraded (21.450%); 48174/338238 objects misplaced (14.243%); 32 MiB/s, 513 objects/s recovering 2024-09-13T14:37:21.561 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:21 smithi102 ceph-mon[110183]: pgmap v741: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 activating+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 31 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 72551/338238 objects degraded (21.450%); 48174/338238 objects misplaced (14.243%); 32 MiB/s, 513 objects/s recovering 2024-09-13T14:37:21.597 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:21.619 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:21.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:21.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:21.721 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:21.742 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:21.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:21.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:21.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:21 smithi195 ceph-mon[104262]: pgmap v741: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 activating+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 31 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 72551/338238 objects degraded (21.450%); 48174/338238 objects misplaced (14.243%); 32 MiB/s, 513 objects/s recovering 2024-09-13T14:37:21.844 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:21.866 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:21.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:21.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:21.969 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:21.980 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:37:21.995 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:21.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:21.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:22.046 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 108 16 112812 112796 65.2679 0 - 0.0106664 2024-09-13T14:37:22.097 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:22.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:22.119 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:22.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:22.202 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:37:22.221 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:22.246 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:22.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:22.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:22.347 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:22.368 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:22.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:22.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:22.470 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:22.491 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:22.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:22.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:22.593 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:22.619 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:22.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:22.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:22.721 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:22.742 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:22.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:22.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:22.843 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:22.865 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:22.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:22.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:22.967 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:22.988 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:22.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:22.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:23.046 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 109 16 112812 112796 64.6691 0 - 0.0106664 2024-09-13T14:37:23.090 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:23.112 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:23.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:23.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:23.213 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:23.235 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:23.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:23.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:23.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:23 smithi195 ceph-mon[104262]: pgmap v742: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 activating+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 31 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 72551/338238 objects degraded (21.450%); 48174/338238 objects misplaced (14.243%); 26 MiB/s, 408 objects/s recovering 2024-09-13T14:37:23.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:23 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:37:23.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:23 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:37:23.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:23 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:37:23.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:23 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:37:23.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:23 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:37:23.336 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:23.357 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:23.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:23.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:23.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:23 smithi102 ceph-mon[110183]: pgmap v742: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 activating+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 31 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 72551/338238 objects degraded (21.450%); 48174/338238 objects misplaced (14.243%); 26 MiB/s, 408 objects/s recovering 2024-09-13T14:37:23.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:23 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:37:23.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:23 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:37:23.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:23 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:37:23.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:23 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:37:23.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:23 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:37:23.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:23 smithi102 ceph-mon[115269]: pgmap v742: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 activating+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 31 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 72551/338238 objects degraded (21.450%); 48174/338238 objects misplaced (14.243%); 26 MiB/s, 408 objects/s recovering 2024-09-13T14:37:23.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:23 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:37:23.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:23 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:37:23.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:23 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:37:23.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:23 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:37:23.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:23 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:37:23.459 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:23.480 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:23.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:23.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:23.582 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:23.608 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:23.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:23.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:23.710 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:23.723 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 6, 7, 5] out_osds: [1, 2] dead_osds: [6] live_osds: [0, 1, 2, 3, 4, 5, 7] 2024-09-13T14:37:23.723 INFO:tasks.thrashosds.thrasher:Reviving osd 6 2024-09-13T14:37:23.723 INFO:tasks.cephadm.osd.6:Restarting osd.6 (starting--it wasn't running)... 2024-09-13T14:37:23.724 DEBUG:teuthology.orchestra.run.smithi195:osd.6> sudo journalctl -f -n 0 -u ceph-08cee502-71d9-11ef-bceb-c7b262605968@osd.6.service 2024-09-13T14:37:23.736 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:23.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:23.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:23.738 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl start ceph-08cee502-71d9-11ef-bceb-c7b262605968@osd.6 2024-09-13T14:37:23.838 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:23.860 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:23.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:23.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:23.962 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:23.990 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:23.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:23.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:24.034 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 14:37:23 smithi195 systemd[1]: Starting Ceph osd.6 for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T14:37:24.046 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 110 16 112812 112796 64.0813 0 - 0.0106664 2024-09-13T14:37:24.092 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:24.118 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:24.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:24.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:24.220 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:24.241 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:24.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:24.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:24.343 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:24.364 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:24.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:24.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:24.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:24 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 72551/338238 objects degraded (21.450%), 13 pgs degraded, 24 pgs undersized (PG_DEGRADED) 2024-09-13T14:37:24.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:24 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 72551/338238 objects degraded (21.450%), 13 pgs degraded, 24 pgs undersized (PG_DEGRADED) 2024-09-13T14:37:24.466 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:24.488 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:24.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:24.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:24.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:24 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 72551/338238 objects degraded (21.450%), 13 pgs degraded, 24 pgs undersized (PG_DEGRADED) 2024-09-13T14:37:24.589 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:24.614 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:24.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:24.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:24.716 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:24.738 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:24.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:24.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:24.839 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:24.862 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:24.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:24.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:24.963 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:24.987 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:24.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:24.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:25.040 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is unset 2024-09-13T14:37:25.046 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 111 16 112812 112796 63.504 0 - 0.0106664 2024-09-13T14:37:25.089 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:25.113 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:25.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:25.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:25.215 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:25.236 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:25.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:25.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:25.319 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:25 smithi102 ceph-mon[110183]: pgmap v743: 89 pgs: 1 activating+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 31 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 69347/338238 objects degraded (20.502%); 47920/338238 objects misplaced (14.168%); 48 MiB/s, 764 objects/s recovering 2024-09-13T14:37:25.319 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:25 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3342475070' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:37:25.319 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:25 smithi102 ceph-mon[115269]: pgmap v743: 89 pgs: 1 activating+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 31 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 69347/338238 objects degraded (20.502%); 47920/338238 objects misplaced (14.168%); 48 MiB/s, 764 objects/s recovering 2024-09-13T14:37:25.320 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:25 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3342475070' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:37:25.338 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:25.362 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:25.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:25.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:25.464 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:25.486 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:25.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:25.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:25.497 INFO:journalctl@ceph.osd.4.smithi195.stdout:Sep 13 14:37:25 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-4[38505]: 2024-09-13T14:37:25.129+0000 7fb6840a8640 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-09-13T14:36:59.808621+0000 front 2024-09-13T14:36:59.808658+0000 (oldest deadline 2024-09-13T14:37:25.108617+0000) 2024-09-13T14:37:25.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:25 smithi195 ceph-mon[104262]: pgmap v743: 89 pgs: 1 activating+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 31 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 69347/338238 objects degraded (20.502%); 47920/338238 objects misplaced (14.168%); 48 MiB/s, 764 objects/s recovering 2024-09-13T14:37:25.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:25 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3342475070' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:37:25.549 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:37:25.550 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:37:25.550 INFO:teuthology.orchestra.run.smithi102.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-13T14:37:25.569 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:37:25.569 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:37:25.587 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:25.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:25.611 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:25.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:25.712 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:25.734 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:25.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:25.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:25.836 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:25.860 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:25.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:25.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:25.873 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 14:37:25 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1[132573]: 2024-09-13T14:37:25.697+0000 7f1d2e09b640 -1 osd.1 231 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-09-13T14:37:01.744780+0000 front 2024-09-13T14:37:01.744827+0000 (oldest deadline 2024-09-13T14:37:25.244655+0000) 2024-09-13T14:37:25.961 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:25.982 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:25.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:25.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:26.046 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 112 6 112812 112806 62.9426 0.0178571 37.4588 0.0139453 2024-09-13T14:37:26.084 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:26.105 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:26.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:26.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:26.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:26 smithi102 ceph-mon[115269]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:37:26.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:26 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3342475070' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:37:26.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:26 smithi102 ceph-mon[115269]: osdmap e232: 8 total, 7 up, 6 in 2024-09-13T14:37:26.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:26 smithi102 ceph-mon[115269]: osd.5 reported failed by osd.4 2024-09-13T14:37:26.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:37:26.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:37:26.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:26 smithi102 ceph-mon[115269]: osd.5 failure report canceled by osd.4 2024-09-13T14:37:26.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:26 smithi102 ceph-mon[110183]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:37:26.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:26 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3342475070' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:37:26.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:26 smithi102 ceph-mon[110183]: osdmap e232: 8 total, 7 up, 6 in 2024-09-13T14:37:26.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:26 smithi102 ceph-mon[110183]: osd.5 reported failed by osd.4 2024-09-13T14:37:26.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:37:26.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:37:26.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:26 smithi102 ceph-mon[110183]: osd.5 failure report canceled by osd.4 2024-09-13T14:37:26.208 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:26.232 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:26.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:26.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:26.334 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:26.355 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:26.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:26.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:26.399 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:26 smithi195 ceph-mon[104262]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:37:26.399 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:26 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3342475070' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:37:26.399 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:26 smithi195 ceph-mon[104262]: osdmap e232: 8 total, 7 up, 6 in 2024-09-13T14:37:26.399 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:26 smithi195 ceph-mon[104262]: osd.5 reported failed by osd.4 2024-09-13T14:37:26.399 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:37:26.399 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:37:26.399 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:26 smithi195 ceph-mon[104262]: osd.5 failure report canceled by osd.4 2024-09-13T14:37:26.457 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:26.482 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:26.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:26.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:26.583 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:26.586 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:37:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0[127501]: 2024-09-13T14:37:26.425+0000 7fe9a0458640 -1 osd.0 233 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-09-13T14:37:00.309202+0000 front 2024-09-13T14:37:00.309254+0000 (oldest deadline 2024-09-13T14:37:26.209185+0000) 2024-09-13T14:37:26.587 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:37:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:37:26] "GET /metrics HTTP/1.1" 200 36574 "" "Prometheus/2.51.0" 2024-09-13T14:37:26.605 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:26.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:26.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:26.608 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T14:37:26.608 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T14:37:26.707 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:26.731 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:26.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:26.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:26.833 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:26.856 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:26.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:26.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:26.958 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:26.982 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:26.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:26.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:27.046 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 113 1 112812 112811 62.3883 0.3125 36.6305 0.0155603 2024-09-13T14:37:27.084 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:27.108 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:27.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:27.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:27.209 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:27.232 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:27.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:27.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:27.334 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:27.355 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:27.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:27.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:27.377 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:27 smithi195 ceph-mon[104262]: pgmap v745: 89 pgs: 1 activating+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 31 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 69347/338238 objects degraded (20.502%); 47920/338238 objects misplaced (14.168%); 49 MiB/s, 783 objects/s recovering 2024-09-13T14:37:27.377 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:27 smithi195 ceph-mon[104262]: osdmap e233: 8 total, 7 up, 6 in 2024-09-13T14:37:27.388 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:27 smithi102 ceph-mon[110183]: pgmap v745: 89 pgs: 1 activating+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 31 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 69347/338238 objects degraded (20.502%); 47920/338238 objects misplaced (14.168%); 49 MiB/s, 783 objects/s recovering 2024-09-13T14:37:27.388 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:27 smithi102 ceph-mon[110183]: osdmap e233: 8 total, 7 up, 6 in 2024-09-13T14:37:27.389 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:27 smithi102 ceph-mon[115269]: pgmap v745: 89 pgs: 1 activating+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 2 activating+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 31 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 27 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 69347/338238 objects degraded (20.502%); 47920/338238 objects misplaced (14.168%); 49 MiB/s, 783 objects/s recovering 2024-09-13T14:37:27.389 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:27 smithi102 ceph-mon[115269]: osdmap e233: 8 total, 7 up, 6 in 2024-09-13T14:37:27.457 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:27.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:27.480 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:27.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:27.581 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:27.607 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:27.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:27.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:27.668 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:37:27 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0[127501]: 2024-09-13T14:37:27.387+0000 7fe9a0458640 -1 osd.0 233 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-09-13T14:37:00.309202+0000 front 2024-09-13T14:37:00.309254+0000 (oldest deadline 2024-09-13T14:37:26.209185+0000) 2024-09-13T14:37:27.709 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:27.731 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:27.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:27.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:27.833 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:27.862 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:27.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:27.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:27.900 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:37:27.964 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:27.987 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:27.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:27.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:27.995 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:37:27.996 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-13T14:37:27.996 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 0 2024-09-13T14:37:27.996 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:37:28.046 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Total time run: 113.253 2024-09-13T14:37:28.046 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Total writes made: 112812 2024-09-13T14:37:28.047 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Write size: 65536 2024-09-13T14:37:28.047 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Object size: 65536 2024-09-13T14:37:28.047 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Bandwidth (MB/sec): 62.2566 2024-09-13T14:37:28.047 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Stddev Bandwidth: 57.9026 2024-09-13T14:37:28.047 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Max bandwidth (MB/sec): 258 2024-09-13T14:37:28.047 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Min bandwidth (MB/sec): 0 2024-09-13T14:37:28.047 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Average IOPS: 996 2024-09-13T14:37:28.047 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Stddev IOPS: 926.444 2024-09-13T14:37:28.047 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Max IOPS: 4128 2024-09-13T14:37:28.047 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Min IOPS: 0 2024-09-13T14:37:28.047 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Average Latency(s): 0.0159035 2024-09-13T14:37:28.047 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Stddev Latency(s): 0.440813 2024-09-13T14:37:28.048 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Max latency(s): 38.738 2024-09-13T14:37:28.048 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Min latency(s): 0.00114883 2024-09-13T14:37:28.052 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Cleaning up (deleting benchmark objects) 2024-09-13T14:37:28.089 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:28.126 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:28.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:28.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:28.151 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:37:28.228 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:28.249 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:28.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:28.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:28.351 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:28.378 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:28.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:28.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:28.480 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:28.506 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:28.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:28.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:28.557 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:37:28 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0[127501]: 2024-09-13T14:37:28.374+0000 7fe9a0458640 -1 osd.0 233 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-09-13T14:37:00.309202+0000 front 2024-09-13T14:37:00.309254+0000 (oldest deadline 2024-09-13T14:37:26.209185+0000) 2024-09-13T14:37:28.609 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:28.609 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:37:28.642 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:28.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:28.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:28.661 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T14:37:28.743 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:28.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:28.769 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:28.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:28.870 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:28.892 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:28.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:28.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:28.943 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 14:37:28 smithi195 podman[167991]: 2024-09-13 14:37:28.675880341 +0000 UTC m=+0.454288297 image pull c129f5a2fd4eccce3e6eec43830eff1929af98bc48f3b850cf144ce5b326e8b5 quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11 2024-09-13T14:37:28.943 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 14:37:28 smithi195 podman[167991]: 2024-09-13 14:37:28.820406658 +0000 UTC m=+0.598814591 container create e033601dd6f34520aab9872367191ba0494587c95c0dc5bae333fdfb9f51cea9 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=reef-2c01458, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-13T14:37:28.993 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:29.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:29.029 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:29.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:29.130 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:29.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:29.153 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:29.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:29.208 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 14:37:29 smithi195 podman[167991]: 2024-09-13 14:37:29.107320196 +0000 UTC m=+0.885728137 container init e033601dd6f34520aab9872367191ba0494587c95c0dc5bae333fdfb9f51cea9 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6-activate, io.buildah.version=1.37.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-2c01458, org.label-schema.build-date=20240731) 2024-09-13T14:37:29.208 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 14:37:29 smithi195 podman[167991]: 2024-09-13 14:37:29.114107913 +0000 UTC m=+0.892515844 container start e033601dd6f34520aab9872367191ba0494587c95c0dc5bae333fdfb9f51cea9 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-2c01458) 2024-09-13T14:37:29.208 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:29 smithi195 ceph-mon[104262]: pgmap v747: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfilling, 2 activating+remapped, 34 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 14 active+undersized, 7 active+undersized+degraded, 25 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 712 KiB/s wr, 11 op/s; 62914/338505 objects degraded (18.586%); 54198/338505 objects misplaced (16.011%); 46 MiB/s, 727 objects/s recovering 2024-09-13T14:37:29.208 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:29 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 62914/338505 objects degraded (18.586%), 12 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-09-13T14:37:29.255 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:29.258 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:37:29.277 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:29.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:29.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:29.378 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:29.382 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:29 smithi102 ceph-mon[110183]: pgmap v747: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfilling, 2 activating+remapped, 34 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 14 active+undersized, 7 active+undersized+degraded, 25 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 712 KiB/s wr, 11 op/s; 62914/338505 objects degraded (18.586%); 54198/338505 objects misplaced (16.011%); 46 MiB/s, 727 objects/s recovering 2024-09-13T14:37:29.383 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:29 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 62914/338505 objects degraded (18.586%), 12 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-09-13T14:37:29.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:29 smithi102 ceph-mon[115269]: pgmap v747: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 2 active+undersized+degraded+remapped+backfilling, 2 activating+remapped, 34 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 14 active+undersized, 7 active+undersized+degraded, 25 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 712 KiB/s wr, 11 op/s; 62914/338505 objects degraded (18.586%); 54198/338505 objects misplaced (16.011%); 46 MiB/s, 727 objects/s recovering 2024-09-13T14:37:29.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:29 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 62914/338505 objects degraded (18.586%), 12 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-09-13T14:37:29.407 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:29.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:29.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:29.497 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 14:37:29 smithi195 podman[167991]: 2024-09-13 14:37:29.207196231 +0000 UTC m=+0.985604163 container attach e033601dd6f34520aab9872367191ba0494587c95c0dc5bae333fdfb9f51cea9 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6-activate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=reef-2c01458, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-09-13T14:37:29.509 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:29.537 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:29.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:29.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:29.639 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:29.639 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:37:29 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0[127501]: 2024-09-13T14:37:29.381+0000 7fe9a0458640 -1 osd.0 233 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-09-13T14:37:00.309202+0000 front 2024-09-13T14:37:00.309254+0000 (oldest deadline 2024-09-13T14:37:26.209185+0000) 2024-09-13T14:37:29.661 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:29.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:29.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:29.763 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:29.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:29.787 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:29.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:29.864 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 14:37:29 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6-activate[168075]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-13T14:37:29.864 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 14:37:29 smithi195 bash[167991]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-13T14:37:29.864 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 14:37:29 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6-activate[168075]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-13T14:37:29.864 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 14:37:29 smithi195 bash[167991]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-13T14:37:29.864 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 14:37:29 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6-activate[168075]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-13T14:37:29.865 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 14:37:29 smithi195 bash[167991]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-13T14:37:29.865 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 14:37:29 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6-activate[168075]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-13T14:37:29.865 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 14:37:29 smithi195 bash[167991]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-13T14:37:29.865 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 14:37:29 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6-activate[168075]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-09-13T14:37:29.865 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 14:37:29 smithi195 bash[167991]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-09-13T14:37:29.865 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 14:37:29 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6-activate[168075]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-13T14:37:29.865 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 14:37:29 smithi195 bash[167991]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-13T14:37:29.865 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 14:37:29 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6-activate[168075]: --> ceph-volume raw activate successful for osd ID: 6 2024-09-13T14:37:29.865 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 14:37:29 smithi195 bash[167991]: --> ceph-volume raw activate successful for osd ID: 6 2024-09-13T14:37:29.865 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 14:37:29 smithi195 podman[167991]: 2024-09-13 14:37:29.576797134 +0000 UTC m=+1.355205070 container died e033601dd6f34520aab9872367191ba0494587c95c0dc5bae333fdfb9f51cea9 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=reef-2c01458, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-09-13T14:37:29.889 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:29.918 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:29.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:29.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:30.020 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:30.034 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:37:30.046 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:30.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:30.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:30.147 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:30.169 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:30.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:30.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:30.271 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:30.296 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:30.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:30.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:30.398 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:30.421 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:30.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:30.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:30.524 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:30.552 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:30.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:30.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:30.654 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:30.668 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 14:37:30 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0[127501]: 2024-09-13T14:37:30.425+0000 7fe9a0458640 -1 osd.0 233 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-09-13T14:37:00.309202+0000 front 2024-09-13T14:37:00.309254+0000 (oldest deadline 2024-09-13T14:37:26.209185+0000) 2024-09-13T14:37:30.677 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:30.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:30.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:30.719 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 14:37:30 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T14:37:30.300+0000 7f6a052c3640 -1 osd.7 233 heartbeat_check: no reply from 172.21.15.195:6812 osd.5 since back 2024-09-13T14:37:04.195994+0000 front 2024-09-13T14:37:04.196104+0000 (oldest deadline 2024-09-13T14:37:30.096018+0000) 2024-09-13T14:37:30.779 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:30.802 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:30.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:30.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:30.904 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:30.927 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:30.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:30.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:31.029 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:31.051 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:31.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:31.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:31.058 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is set 2024-09-13T14:37:31.153 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:31.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:31 smithi102 ceph-mon[110183]: pgmap v748: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 activating+remapped, 36 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 14 active+undersized, 7 active+undersized+degraded, 25 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 712 KiB/s wr, 11 op/s; 61877/338505 objects degraded (18.279%); 54198/338505 objects misplaced (16.011%); 54 MiB/s, 856 objects/s recovering 2024-09-13T14:37:31.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:31 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2371392640' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:37:31.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:31 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:37:31.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:31 smithi102 ceph-mon[110183]: osd.5 reported failed by osd.0 2024-09-13T14:37:31.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:31 smithi102 ceph-mon[110183]: osd.5 failure report canceled by osd.0 2024-09-13T14:37:31.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:31 smithi102 ceph-mon[115269]: pgmap v748: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 activating+remapped, 36 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 14 active+undersized, 7 active+undersized+degraded, 25 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 712 KiB/s wr, 11 op/s; 61877/338505 objects degraded (18.279%); 54198/338505 objects misplaced (16.011%); 54 MiB/s, 856 objects/s recovering 2024-09-13T14:37:31.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:31 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2371392640' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:37:31.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:31 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:37:31.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:31 smithi102 ceph-mon[115269]: osd.5 reported failed by osd.0 2024-09-13T14:37:31.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:31 smithi102 ceph-mon[115269]: osd.5 failure report canceled by osd.0 2024-09-13T14:37:31.179 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:31.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:31.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:31.280 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:31.302 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:31.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:31.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:31.404 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:31.430 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:31.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:31.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:31.457 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:31 smithi195 ceph-mon[104262]: pgmap v748: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 activating+remapped, 36 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 14 active+undersized, 7 active+undersized+degraded, 25 active+clean; 6.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 712 KiB/s wr, 11 op/s; 61877/338505 objects degraded (18.279%); 54198/338505 objects misplaced (16.011%); 54 MiB/s, 856 objects/s recovering 2024-09-13T14:37:31.457 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:31 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2371392640' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:37:31.457 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:31 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:37:31.457 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:31 smithi195 ceph-mon[104262]: osd.5 reported failed by osd.0 2024-09-13T14:37:31.457 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:31 smithi195 ceph-mon[104262]: osd.5 failure report canceled by osd.0 2024-09-13T14:37:31.533 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:31.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:31.562 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:31.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:31.663 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:31.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:31.689 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:31.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:31.747 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 14:37:31 smithi195 podman[168319]: 2024-09-13 14:37:31.455740923 +0000 UTC m=+1.866608882 container remove e033601dd6f34520aab9872367191ba0494587c95c0dc5bae333fdfb9f51cea9 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6-activate, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, RELEASE=reef-2c01458, ceph=True, GIT_BRANCH=HEAD) 2024-09-13T14:37:31.789 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:31.813 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:31.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:31.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:31.915 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:31.937 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:31.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:31.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:32.040 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:32.065 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:32.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:32.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:32.167 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:32.191 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:32.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:32.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:32.247 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 14:37:31 smithi195 podman[168529]: 2024-09-13 14:37:31.826424764 +0000 UTC m=+0.251650756 image pull c129f5a2fd4eccce3e6eec43830eff1929af98bc48f3b850cf144ce5b326e8b5 quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11 2024-09-13T14:37:32.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:32 smithi195 ceph-mon[104262]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:37:32.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:32 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:37:32.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:32 smithi195 ceph-mon[104262]: osdmap e234: 8 total, 7 up, 6 in 2024-09-13T14:37:32.293 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:32.300 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:32 smithi102 ceph-mon[115269]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:37:32.301 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:32 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:37:32.301 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:32 smithi102 ceph-mon[115269]: osdmap e234: 8 total, 7 up, 6 in 2024-09-13T14:37:32.301 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:32 smithi102 ceph-mon[110183]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:37:32.301 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:32 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:37:32.301 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:32 smithi102 ceph-mon[110183]: osdmap e234: 8 total, 7 up, 6 in 2024-09-13T14:37:32.318 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:32.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:32.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:32.419 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:32.444 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:32.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:32.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:32.545 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:32.552 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:37:32.552 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:37:32.569 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:32.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:32.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:32.572 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:37:32.572 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:37:32.589 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:37:32.590 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:37:32.608 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:37:32.608 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:37:32.625 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:37:32.625 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:37:32.644 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:37:32.644 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:37:32.644 INFO:teuthology.orchestra.run.smithi102.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-13T14:37:32.663 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:37:32.663 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:37:32.671 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:32.694 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:32.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:32.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:32.796 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:32.818 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:32.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:32.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:32.919 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:32.942 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:32.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:32.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:33.044 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:33.069 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:33.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:33.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:33.171 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:33.193 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:33.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:33.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:33.295 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:33.318 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:33.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:33.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:33.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:33 smithi102 ceph-mon[115269]: pgmap v750: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 37 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 14 active+undersized, 7 active+undersized+degraded, 25 active+clean; 6.9 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 942 KiB/s wr, 19 op/s; 54874/338427 objects degraded (16.214%); 61151/338427 objects misplaced (18.069%); 33 MiB/s, 533 objects/s recovering 2024-09-13T14:37:33.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:33 smithi102 ceph-mon[115269]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-09-13T14:37:33.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:33 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3082205734' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:37:33.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:33 smithi102 ceph-mon[110183]: pgmap v750: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 37 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 14 active+undersized, 7 active+undersized+degraded, 25 active+clean; 6.9 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 942 KiB/s wr, 19 op/s; 54874/338427 objects degraded (16.214%); 61151/338427 objects misplaced (18.069%); 33 MiB/s, 533 objects/s recovering 2024-09-13T14:37:33.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:33 smithi102 ceph-mon[110183]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-09-13T14:37:33.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:33 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3082205734' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:37:33.420 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:33.442 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:33.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:33.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:33.467 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T14:37:33.467 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T14:37:33.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:33 smithi195 ceph-mon[104262]: pgmap v750: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 37 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 14 active+undersized, 7 active+undersized+degraded, 25 active+clean; 6.9 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 942 KiB/s wr, 19 op/s; 54874/338427 objects degraded (16.214%); 61151/338427 objects misplaced (18.069%); 33 MiB/s, 533 objects/s recovering 2024-09-13T14:37:33.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:33 smithi195 ceph-mon[104262]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-09-13T14:37:33.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:33 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3082205734' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:37:33.544 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:33.571 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:33.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:33.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:33.673 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:33.695 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:33.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:33.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:33.796 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:33.818 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:33.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:33.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:33.920 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:33.941 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:33.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:33.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:34.043 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:34.066 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:34.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:34.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:34.070 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:37:34.168 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:34.191 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:34.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:34.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:34.293 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:34.318 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:34.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:34.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:34.331 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:37:34.359 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:37:34.360 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:37:34.360 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:34 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 54874/338427 objects degraded (16.214%), 11 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-09-13T14:37:34.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:34 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 54874/338427 objects degraded (16.214%), 11 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-09-13T14:37:34.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:34 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 54874/338427 objects degraded (16.214%), 11 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-09-13T14:37:34.421 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:34.442 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:34.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:34.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:34.544 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:34.565 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:34.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:34.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:34.667 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:34.689 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:34.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:34.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:34.791 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:34.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:34.814 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:34.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:34.915 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:34.937 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:34.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:34.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:35.039 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:35.062 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:35.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:35.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:35.164 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:35.184 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:35.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:35.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:35.286 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:35.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:35.312 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:35.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:35.413 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:35.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:35 smithi102 ceph-mon[110183]: pgmap v751: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 37 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 14 active+undersized, 7 active+undersized+degraded, 25 active+clean; 6.9 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 127 B/s rd, 752 KiB/s wr, 64 op/s; 51757/337263 objects degraded (15.346%); 60660/337263 objects misplaced (17.986%); 52 MiB/s, 832 objects/s recovering 2024-09-13T14:37:35.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:35 smithi102 ceph-mon[115269]: pgmap v751: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 37 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 14 active+undersized, 7 active+undersized+degraded, 25 active+clean; 6.9 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 127 B/s rd, 752 KiB/s wr, 64 op/s; 51757/337263 objects degraded (15.346%); 60660/337263 objects misplaced (17.986%); 52 MiB/s, 832 objects/s recovering 2024-09-13T14:37:35.436 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:35.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:35.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:35.468 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:37:35.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:35 smithi195 ceph-mon[104262]: pgmap v751: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 37 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 14 active+undersized, 7 active+undersized+degraded, 25 active+clean; 6.9 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 127 B/s rd, 752 KiB/s wr, 64 op/s; 51757/337263 objects degraded (15.346%); 60660/337263 objects misplaced (17.986%); 52 MiB/s, 832 objects/s recovering 2024-09-13T14:37:35.538 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:35.563 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:35.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:35.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:35.665 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:35.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:35.689 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:35.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:35.792 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:35.815 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:35.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:35.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:35.916 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:35.942 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:35.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:35.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:36.044 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:36.069 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:36.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:36.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:36.171 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:36.180 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:37:36.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:36.201 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:36.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:36.302 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:36.323 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:36.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:36.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:36.425 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:36.449 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:36.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:36.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:36.617 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:36.626 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:37:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:37:36] "GET /metrics HTTP/1.1" 200 36574 "" "Prometheus/2.51.0" 2024-09-13T14:37:36.642 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:36.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:36.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:36.744 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:36.765 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:36.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:36.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:36.867 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:36.890 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:36.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:36.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:36.938 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T14:37:36.991 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:37.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:37.017 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:37.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:37.065 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is set 2024-09-13T14:37:37.120 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:37.146 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:37.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:37.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:37.195 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:37 smithi195 ceph-mon[104262]: pgmap v752: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 37 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 14 active+undersized, 7 active+undersized+degraded, 25 active+clean; 6.9 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 109 B/s rd, 55 KiB/s wr, 45 op/s; 51757/337263 objects degraded (15.346%); 60660/337263 objects misplaced (17.986%); 42 MiB/s, 671 objects/s recovering 2024-09-13T14:37:37.195 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:37 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/837158419' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:37:37.196 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 14:37:37 smithi195 podman[168529]: 2024-09-13 14:37:37.027864009 +0000 UTC m=+5.453089994 container create 6927a7bcecccb5b49ac842264f0b3b2312145c9eec7a599f20dac4e3aced5246 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=reef-2c01458, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-09-13T14:37:37.248 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:37.270 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:37.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:37.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:37.372 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:37.397 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:37.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:37.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:37.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:37 smithi102 ceph-mon[115269]: pgmap v752: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 37 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 14 active+undersized, 7 active+undersized+degraded, 25 active+clean; 6.9 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 109 B/s rd, 55 KiB/s wr, 45 op/s; 51757/337263 objects degraded (15.346%); 60660/337263 objects misplaced (17.986%); 42 MiB/s, 671 objects/s recovering 2024-09-13T14:37:37.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:37 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/837158419' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:37:37.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:37 smithi102 ceph-mon[110183]: pgmap v752: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 37 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 14 active+undersized, 7 active+undersized+degraded, 25 active+clean; 6.9 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 109 B/s rd, 55 KiB/s wr, 45 op/s; 51757/337263 objects degraded (15.346%); 60660/337263 objects misplaced (17.986%); 42 MiB/s, 671 objects/s recovering 2024-09-13T14:37:37.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:37 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/837158419' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:37:37.430 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T14:37:37.471 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:37:37.499 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:37.521 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:37.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:37.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:37.597 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 14:37:37 smithi195 podman[168529]: 2024-09-13 14:37:37.323199404 +0000 UTC m=+5.748425381 container init 6927a7bcecccb5b49ac842264f0b3b2312145c9eec7a599f20dac4e3aced5246 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=reef-2c01458) 2024-09-13T14:37:37.597 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 14:37:37 smithi195 podman[168529]: 2024-09-13 14:37:37.328752215 +0000 UTC m=+5.753978192 container start 6927a7bcecccb5b49ac842264f0b3b2312145c9eec7a599f20dac4e3aced5246 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=reef-2c01458, ceph=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T14:37:37.597 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 14:37:37 smithi195 ceph-osd[168912]: -- 172.21.15.195:0/1910205410 <== mon.1 v2:172.21.15.195:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557f69e9b1e0 con 0x557f6ac1a000 2024-09-13T14:37:37.597 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 14:37:37 smithi195 bash[168529]: 6927a7bcecccb5b49ac842264f0b3b2312145c9eec7a599f20dac4e3aced5246 2024-09-13T14:37:37.597 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 14:37:37 smithi195 systemd[1]: Started Ceph osd.6 for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T14:37:37.623 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:37.645 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:37.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:37.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:37.747 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:37.775 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:37.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:37.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:37.877 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:37.904 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:37.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:37.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:38.006 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:38.029 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:38.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:38.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:38.131 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:38.155 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:38.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:38.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:38.248 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 14:37:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T14:37:37.875+0000 7f8111415640 -1 Falling back to public interface 2024-09-13T14:37:38.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:38 smithi195 ceph-mon[104262]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:37:38.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:38 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/837158419' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:37:38.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:38 smithi195 ceph-mon[104262]: osdmap e235: 8 total, 7 up, 6 in 2024-09-13T14:37:38.256 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:38.279 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:38.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:38.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:38.297 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:37:38.381 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:38.403 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:38.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:38.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:38.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:38 smithi102 ceph-mon[110183]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:37:38.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:38 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/837158419' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:37:38.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:38 smithi102 ceph-mon[110183]: osdmap e235: 8 total, 7 up, 6 in 2024-09-13T14:37:38.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:38 smithi102 ceph-mon[115269]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:37:38.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:38 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/837158419' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:37:38.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:38 smithi102 ceph-mon[115269]: osdmap e235: 8 total, 7 up, 6 in 2024-09-13T14:37:38.505 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:38.527 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:38.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:38.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:38.629 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:38.653 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:38.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:38.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:38.757 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:38.783 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:38.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:38.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:38.885 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:38.909 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:38.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:38.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:39.011 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:39.037 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:39.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:39.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:39.139 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:39.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:39.163 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:39.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:39.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:39 smithi195 ceph-mon[104262]: pgmap v754: 89 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 37 active+clean+remapped, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 25 active+clean; 6.9 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 127 B/s rd, 40 KiB/s wr, 80 op/s; 51653/336618 objects degraded (15.345%); 59663/336618 objects misplaced (17.724%); 33 MiB/s, 526 objects/s recovering 2024-09-13T14:37:39.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:39 smithi195 ceph-mon[104262]: osdmap e236: 8 total, 7 up, 6 in 2024-09-13T14:37:39.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:39 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 51653/336618 objects degraded (15.345%), 11 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-09-13T14:37:39.268 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:39.291 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:39.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:39.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:39.325 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:39 smithi102 ceph-mon[110183]: pgmap v754: 89 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 37 active+clean+remapped, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 25 active+clean; 6.9 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 127 B/s rd, 40 KiB/s wr, 80 op/s; 51653/336618 objects degraded (15.345%); 59663/336618 objects misplaced (17.724%); 33 MiB/s, 526 objects/s recovering 2024-09-13T14:37:39.325 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:39 smithi102 ceph-mon[110183]: osdmap e236: 8 total, 7 up, 6 in 2024-09-13T14:37:39.325 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:39 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 51653/336618 objects degraded (15.345%), 11 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-09-13T14:37:39.325 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:39 smithi102 ceph-mon[115269]: pgmap v754: 89 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 37 active+clean+remapped, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 25 active+clean; 6.9 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 127 B/s rd, 40 KiB/s wr, 80 op/s; 51653/336618 objects degraded (15.345%); 59663/336618 objects misplaced (17.724%); 33 MiB/s, 526 objects/s recovering 2024-09-13T14:37:39.325 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:39 smithi102 ceph-mon[115269]: osdmap e236: 8 total, 7 up, 6 in 2024-09-13T14:37:39.325 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:39 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 51653/336618 objects degraded (15.345%), 11 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-09-13T14:37:39.334 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:37:39.334 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:37:39.342 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:37:39.343 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:37:39.359 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:37:39.360 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:37:39.379 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:37:39.379 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:37:39.393 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:39.398 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:37:39.398 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:37:39.415 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:39.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:39.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:39.418 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:37:39.419 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:37:39.419 INFO:teuthology.orchestra.run.smithi102.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-13T14:37:39.438 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:37:39.438 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:37:39.517 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:39.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:39.539 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:39.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:39.640 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:39.662 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:39.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:39.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:39.764 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:39.790 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:39.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:39.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:39.892 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:39.917 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:39.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:39.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:40.019 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:40.042 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:40.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:40.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:40.144 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:40.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:40.169 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:40.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:40.187 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T14:37:40.187 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T14:37:40.270 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:40.288 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:37:40.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:40.293 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:40.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:40.308 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:40 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3842317557' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:37:40.394 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:40.416 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:40.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:40.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:40.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:40 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3842317557' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:37:40.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:40 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3842317557' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:37:40.500 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:37:40.518 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:40.541 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:40.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:40.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:40.643 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:40.664 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:40.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:40.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:40.766 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:40.794 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:40.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:40.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:40.896 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:40.918 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:40.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:40.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:41.020 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:41.043 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:41.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:41.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:41.145 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:41.171 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:41.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:41.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:41.273 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:41.300 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:41.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:41.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:41.401 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:41.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:41 smithi102 ceph-mon[110183]: pgmap v756: 89 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 37 active+clean+remapped, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 25 active+clean; 6.8 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 127 B/s rd, 127 B/s wr, 81 op/s; 51633/336474 objects degraded (15.345%); 59635/336474 objects misplaced (17.724%); 33 MiB/s, 525 objects/s recovering 2024-09-13T14:37:41.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:41 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:37:41.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:41 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:37:41.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:41 smithi102 ceph-mon[115269]: pgmap v756: 89 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 37 active+clean+remapped, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 25 active+clean; 6.8 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 127 B/s rd, 127 B/s wr, 81 op/s; 51633/336474 objects degraded (15.345%); 59635/336474 objects misplaced (17.724%); 33 MiB/s, 525 objects/s recovering 2024-09-13T14:37:41.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:41 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:37:41.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:41 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:37:41.427 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:41.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:41.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:41.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:41 smithi195 ceph-mon[104262]: pgmap v756: 89 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 37 active+clean+remapped, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 25 active+clean; 6.8 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 127 B/s rd, 127 B/s wr, 81 op/s; 51633/336474 objects degraded (15.345%); 59635/336474 objects misplaced (17.724%); 33 MiB/s, 525 objects/s recovering 2024-09-13T14:37:41.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:41 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:37:41.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:41 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:37:41.529 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:41.550 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:41.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:41.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:41.652 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:41.678 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:41.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:41.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:41.780 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:41.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:41.804 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:41.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:41.905 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:41.931 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:41.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:41.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:42.033 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:42.056 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:42.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:42.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:42.158 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:42.179 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:42.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:42.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:42.189 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:37:42.282 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:42.305 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:42.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:42.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:42.407 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:42.431 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:42.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:42.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:42.533 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:42.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:42.556 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:42.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:42.657 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:42.686 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:42.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:42.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:42.788 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:42.810 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:42.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:42.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:42.912 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:42.941 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:42.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:42.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:43.042 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:43.066 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:43.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:43.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:43.167 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:43.189 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:43.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:43.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:43.291 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:43.316 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:43.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:43.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:43.418 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:43.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:43 smithi102 ceph-mon[110183]: pgmap v757: 89 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 38 active+clean+remapped, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 25 active+clean; 6.8 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 40 op/s; 49249/336294 objects degraded (14.645%); 59573/336294 objects misplaced (17.715%); 26 MiB/s, 419 objects/s recovering 2024-09-13T14:37:43.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:43 smithi102 ceph-mon[115269]: pgmap v757: 89 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 38 active+clean+remapped, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 25 active+clean; 6.8 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 40 op/s; 49249/336294 objects degraded (14.645%); 59573/336294 objects misplaced (17.715%); 26 MiB/s, 419 objects/s recovering 2024-09-13T14:37:43.442 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:43.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:43.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:43.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:43 smithi195 ceph-mon[104262]: pgmap v757: 89 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 38 active+clean+remapped, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 25 active+clean; 6.8 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 40 op/s; 49249/336294 objects degraded (14.645%); 59573/336294 objects misplaced (17.715%); 26 MiB/s, 419 objects/s recovering 2024-09-13T14:37:43.544 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:43.570 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:43.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:43.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:43.672 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:43.700 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:43.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:43.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:43.802 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:43.831 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:43.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:43.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:43.927 INFO:teuthology.orchestra.run.smithi195.stderr:no valid command found; 10 closest matches: 2024-09-13T14:37:43.927 INFO:teuthology.orchestra.run.smithi195.stderr:0 2024-09-13T14:37:43.928 INFO:teuthology.orchestra.run.smithi195.stderr:1 2024-09-13T14:37:43.928 INFO:teuthology.orchestra.run.smithi195.stderr:2 2024-09-13T14:37:43.928 INFO:teuthology.orchestra.run.smithi195.stderr:abort 2024-09-13T14:37:43.928 INFO:teuthology.orchestra.run.smithi195.stderr:assert 2024-09-13T14:37:43.928 INFO:teuthology.orchestra.run.smithi195.stderr:bluefs debug_inject_read_zeros 2024-09-13T14:37:43.928 INFO:teuthology.orchestra.run.smithi195.stderr:bluefs files list 2024-09-13T14:37:43.928 INFO:teuthology.orchestra.run.smithi195.stderr:bluefs stats 2024-09-13T14:37:43.928 INFO:teuthology.orchestra.run.smithi195.stderr:bluestore allocator dump block 2024-09-13T14:37:43.928 INFO:teuthology.orchestra.run.smithi195.stderr:bluestore allocator fragmentation block 2024-09-13T14:37:43.928 INFO:teuthology.orchestra.run.smithi195.stderr:admin_socket: invalid command 2024-09-13T14:37:43.934 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:43.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:43.957 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:43.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:44.058 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:44.080 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:44.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:44.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:44.181 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is unset 2024-09-13T14:37:44.184 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:44.205 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:44.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:44.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:44.307 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:44.333 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:44.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:44.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:44.372 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:44 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3167115547' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:37:44.372 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:44 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:37:44.372 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:44 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 49249/336294 objects degraded (14.645%), 10 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-13T14:37:44.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:44 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3167115547' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:37:44.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:44 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:37:44.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:44 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 49249/336294 objects degraded (14.645%), 10 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-13T14:37:44.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:44 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3167115547' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:37:44.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:44 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:37:44.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:44 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 49249/336294 objects degraded (14.645%), 10 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-13T14:37:44.420 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:37:44.420 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-09-13T14:37:44.421 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-09-13T14:37:44.421 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-13T14:37:44.421 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.421 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:233222 5.a 5:5d43e0eb:::benchmark_data_smithi102_155989_object115304:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-13T14:37:44.421 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:41.665433+0000", 2024-09-13T14:37:44.421 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 422.75184619700002, 2024-09-13T14:37:44.421 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.17010931900000001, 2024-09-13T14:37:44.421 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:44.421 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:44.421 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:37:44.421 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:37:44.421 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:37:44.422 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 233222 2024-09-13T14:37:44.422 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.422 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:44.422 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.422 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:44.422 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.665433+0000", 2024-09-13T14:37:44.422 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:44.422 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.422 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.422 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:44.422 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.665433+0000", 2024-09-13T14:37:44.422 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:44.422 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.423 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.423 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:44.423 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.665432+0000", 2024-09-13T14:37:44.423 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:37:44.423 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.423 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.423 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:44.423 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.665437+0000", 2024-09-13T14:37:44.423 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9309999999999999e-06 2024-09-13T14:37:44.423 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.423 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.423 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:44.423 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.665447+0000", 2024-09-13T14:37:44.423 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0825000000000001e-05 2024-09-13T14:37:44.424 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.424 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.424 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:44.424 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.665454+0000", 2024-09-13T14:37:44.424 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6730000000000003e-06 2024-09-13T14:37:44.424 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.424 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.424 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:44.424 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.672469+0000", 2024-09-13T14:37:44.424 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0070153790000000004 2024-09-13T14:37:44.424 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.424 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.425 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:37:44.425 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.803548+0000", 2024-09-13T14:37:44.425 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.13107861000000001 2024-09-13T14:37:44.425 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.425 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.425 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,3", 2024-09-13T14:37:44.425 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.803692+0000", 2024-09-13T14:37:44.425 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014347599999999999 2024-09-13T14:37:44.425 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.425 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.425 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:44.425 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.817160+0000", 2024-09-13T14:37:44.426 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.01346893 2024-09-13T14:37:44.426 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.426 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.426 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:44.426 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.817190+0000", 2024-09-13T14:37:44.426 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9842999999999999e-05 2024-09-13T14:37:44.426 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.426 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.426 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:37:44.426 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.834483+0000", 2024-09-13T14:37:44.426 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017293059 2024-09-13T14:37:44.426 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.426 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.427 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:37:44.427 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.834549+0000", 2024-09-13T14:37:44.427 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5988000000000006e-05 2024-09-13T14:37:44.427 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.427 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.427 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:37:44.427 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.835542+0000", 2024-09-13T14:37:44.427 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00099279300000000006 2024-09-13T14:37:44.427 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:44.427 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:44.427 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:44.427 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.427 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.428 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:233224 5.e 5:73838589:::benchmark_data_smithi102_155989_object115306:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-13T14:37:44.428 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:41.667956+0000", 2024-09-13T14:37:44.428 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 422.74932301299998, 2024-09-13T14:37:44.428 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.15649626, 2024-09-13T14:37:44.428 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:44.428 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:44.428 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:37:44.428 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:37:44.428 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:37:44.428 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 233224 2024-09-13T14:37:44.428 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.428 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:44.428 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.429 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:44.429 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.667956+0000", 2024-09-13T14:37:44.429 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:44.429 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.429 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.429 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:44.429 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.667956+0000", 2024-09-13T14:37:44.429 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:44.429 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.429 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.429 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:44.429 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.667955+0000", 2024-09-13T14:37:44.429 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:37:44.429 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.430 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.430 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:44.430 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.667961+0000", 2024-09-13T14:37:44.430 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5010000000000001e-06 2024-09-13T14:37:44.430 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.430 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.430 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:44.430 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.667973+0000", 2024-09-13T14:37:44.430 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1996e-05 2024-09-13T14:37:44.430 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.430 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.430 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:44.430 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.667981+0000", 2024-09-13T14:37:44.431 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4080000000000001e-06 2024-09-13T14:37:44.431 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.431 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.431 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:44.431 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.668039+0000", 2024-09-13T14:37:44.431 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8029000000000003e-05 2024-09-13T14:37:44.431 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.431 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.431 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:37:44.431 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.668149+0000", 2024-09-13T14:37:44.431 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000110889 2024-09-13T14:37:44.431 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.431 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.432 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:37:44.432 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.668277+0000", 2024-09-13T14:37:44.432 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000127541 2024-09-13T14:37:44.432 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.432 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.432 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:44.432 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.672498+0000", 2024-09-13T14:37:44.432 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0042207640000000001 2024-09-13T14:37:44.432 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.432 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.432 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:37:44.433 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.672514+0000", 2024-09-13T14:37:44.433 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6135999999999999e-05 2024-09-13T14:37:44.433 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.433 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.433 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:44.433 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.824368+0000", 2024-09-13T14:37:44.433 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.15185453600000001 2024-09-13T14:37:44.433 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.433 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.433 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:37:44.433 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.824416+0000", 2024-09-13T14:37:44.433 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7236e-05 2024-09-13T14:37:44.433 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.433 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.434 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:37:44.434 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:41.824452+0000", 2024-09-13T14:37:44.434 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6653000000000003e-05 2024-09-13T14:37:44.434 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:44.434 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:44.434 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:44.434 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.434 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.434 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:85609 6.e 6:7b12e9c0:::benchmark_data_smithi102_194590_object85608:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-13T14:37:44.434 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:32:17.475387+0000", 2024-09-13T14:37:44.434 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 326.94189189500003, 2024-09-13T14:37:44.434 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.17257324399999999, 2024-09-13T14:37:44.434 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:44.434 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:44.435 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:37:44.435 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:37:44.435 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:37:44.435 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 85609 2024-09-13T14:37:44.435 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.435 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:44.435 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:44.436 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.436 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:44.436 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.475387+0000", 2024-09-13T14:37:44.437 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:44.437 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.437 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.437 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:44.437 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.475387+0000", 2024-09-13T14:37:44.437 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:44.437 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.437 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.437 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:44.437 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.475386+0000", 2024-09-13T14:37:44.437 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:37:44.437 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.437 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.438 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:44.438 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.475437+0000", 2024-09-13T14:37:44.438 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1483999999999997e-05 2024-09-13T14:37:44.438 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.438 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.438 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:44.438 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.475451+0000", 2024-09-13T14:37:44.438 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3925e-05 2024-09-13T14:37:44.438 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.438 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.438 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:44.438 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.475458+0000", 2024-09-13T14:37:44.438 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8310000000000003e-06 2024-09-13T14:37:44.438 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.439 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.439 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:44.439 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.475498+0000", 2024-09-13T14:37:44.439 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0373999999999997e-05 2024-09-13T14:37:44.439 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.439 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.439 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:37:44.439 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.475602+0000", 2024-09-13T14:37:44.439 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000103912 2024-09-13T14:37:44.439 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.439 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.439 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,7", 2024-09-13T14:37:44.439 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.475710+0000", 2024-09-13T14:37:44.440 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000107833 2024-09-13T14:37:44.440 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.440 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.440 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:37:44.440 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.478173+0000", 2024-09-13T14:37:44.440 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0024634679999999999 2024-09-13T14:37:44.440 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.440 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.440 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:44.440 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.478402+0000", 2024-09-13T14:37:44.440 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00022860900000000001 2024-09-13T14:37:44.440 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.440 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.441 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:44.441 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.647799+0000", 2024-09-13T14:37:44.441 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.16939664700000001 2024-09-13T14:37:44.441 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.441 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.441 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:37:44.441 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.647910+0000", 2024-09-13T14:37:44.441 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000111189 2024-09-13T14:37:44.441 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.441 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.441 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:37:44.441 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.647960+0000", 2024-09-13T14:37:44.441 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0599000000000001e-05 2024-09-13T14:37:44.442 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:44.442 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:44.442 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:44.442 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.442 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.442 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:85610 6.e 6:722b26f6:::benchmark_data_smithi102_194590_object85609:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-13T14:37:44.442 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:32:17.478842+0000", 2024-09-13T14:37:44.442 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 326.93843734299998, 2024-09-13T14:37:44.442 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.17296241900000001, 2024-09-13T14:37:44.442 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:44.442 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:44.442 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:37:44.443 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:37:44.443 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:37:44.443 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 85610 2024-09-13T14:37:44.443 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.443 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:44.443 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.443 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:44.443 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.478842+0000", 2024-09-13T14:37:44.443 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:44.443 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.443 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.443 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:44.443 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.478842+0000", 2024-09-13T14:37:44.444 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:44.444 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.444 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.444 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:44.444 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.478841+0000", 2024-09-13T14:37:44.444 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:37:44.444 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.444 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.444 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:44.444 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.478869+0000", 2024-09-13T14:37:44.444 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7820000000000001e-05 2024-09-13T14:37:44.444 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.445 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.445 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:44.445 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.478878+0000", 2024-09-13T14:37:44.445 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.3999999999999998e-06 2024-09-13T14:37:44.445 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.445 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.445 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:44.445 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.478885+0000", 2024-09-13T14:37:44.445 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7549999999999997e-06 2024-09-13T14:37:44.445 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.445 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.445 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:44.445 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.478927+0000", 2024-09-13T14:37:44.445 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2305999999999998e-05 2024-09-13T14:37:44.446 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.446 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.446 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:37:44.446 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.479029+0000", 2024-09-13T14:37:44.446 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010176800000000001 2024-09-13T14:37:44.446 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.446 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.446 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,7", 2024-09-13T14:37:44.446 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.479131+0000", 2024-09-13T14:37:44.446 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010254700000000001 2024-09-13T14:37:44.446 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.446 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.446 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:37:44.446 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.481132+0000", 2024-09-13T14:37:44.447 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0020003070000000002 2024-09-13T14:37:44.447 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.447 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.447 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:44.447 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.481169+0000", 2024-09-13T14:37:44.447 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7768000000000002e-05 2024-09-13T14:37:44.447 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.447 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.447 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:44.447 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.651724+0000", 2024-09-13T14:37:44.447 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.170555026 2024-09-13T14:37:44.447 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.447 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.448 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:37:44.448 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.651765+0000", 2024-09-13T14:37:44.448 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0918999999999999e-05 2024-09-13T14:37:44.448 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.448 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.448 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:37:44.448 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.651804+0000", 2024-09-13T14:37:44.448 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8798999999999999e-05 2024-09-13T14:37:44.448 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:44.448 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:44.448 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:44.448 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.448 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.448 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:150704 6.2 6:420df80d:::benchmark_data_smithi102_194590_object40080:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-13T14:37:44.449 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:33:27.750319+0000", 2024-09-13T14:37:44.449 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 256.66696000600001, 2024-09-13T14:37:44.449 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.29454005300000002, 2024-09-13T14:37:44.449 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:44.449 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:44.449 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:37:44.449 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:37:44.449 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:37:44.449 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 150704 2024-09-13T14:37:44.449 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.449 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:44.449 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.449 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:44.449 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.750319+0000", 2024-09-13T14:37:44.450 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:44.450 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.450 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.450 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:44.450 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.750319+0000", 2024-09-13T14:37:44.450 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:44.450 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.450 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.450 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:44.450 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.750317+0000", 2024-09-13T14:37:44.450 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:37:44.450 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.450 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.451 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:44.451 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.750326+0000", 2024-09-13T14:37:44.451 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1650000000000006e-06 2024-09-13T14:37:44.451 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.451 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.451 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:44.451 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.750341+0000", 2024-09-13T14:37:44.451 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5962999999999999e-05 2024-09-13T14:37:44.451 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.452 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.452 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:44.452 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.750350+0000", 2024-09-13T14:37:44.452 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5609999999999999e-06 2024-09-13T14:37:44.452 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.452 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.452 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:44.452 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.754163+0000", 2024-09-13T14:37:44.452 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0038129499999999998 2024-09-13T14:37:44.452 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.452 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.452 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:37:44.452 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.754287+0000", 2024-09-13T14:37:44.453 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012447200000000001 2024-09-13T14:37:44.453 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.453 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.453 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:37:44.453 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.754398+0000", 2024-09-13T14:37:44.453 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000110999 2024-09-13T14:37:44.453 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.453 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.453 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:37:44.453 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.044342+0000", 2024-09-13T14:37:44.453 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.28994310000000001 2024-09-13T14:37:44.453 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.453 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.454 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:44.454 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.044753+0000", 2024-09-13T14:37:44.454 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000411925 2024-09-13T14:37:44.454 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.454 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.454 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:44.454 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.044780+0000", 2024-09-13T14:37:44.454 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.6696999999999999e-05 2024-09-13T14:37:44.454 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.454 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.454 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:37:44.454 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.044820+0000", 2024-09-13T14:37:44.454 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9821999999999999e-05 2024-09-13T14:37:44.454 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.455 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.455 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:37:44.455 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.044859+0000", 2024-09-13T14:37:44.455 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9088999999999998e-05 2024-09-13T14:37:44.455 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:44.455 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:44.455 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:44.455 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.455 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.455 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:150708 6.2 6:4865044e:::benchmark_data_smithi102_194590_object40084:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-13T14:37:44.455 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:33:27.754180+0000", 2024-09-13T14:37:44.455 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 256.66309871599998, 2024-09-13T14:37:44.455 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.29088824600000002, 2024-09-13T14:37:44.456 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:44.456 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:44.456 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:37:44.456 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:37:44.456 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:37:44.456 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 150708 2024-09-13T14:37:44.456 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.456 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:44.456 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.456 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:44.456 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.754180+0000", 2024-09-13T14:37:44.456 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:44.456 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.457 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.457 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:44.457 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.754180+0000", 2024-09-13T14:37:44.457 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:44.457 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.457 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.457 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:44.457 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.754179+0000", 2024-09-13T14:37:44.457 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:37:44.457 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.458 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.458 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:44.458 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.754184+0000", 2024-09-13T14:37:44.458 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2429999999999996e-06 2024-09-13T14:37:44.458 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.458 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.458 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:44.458 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.754194+0000", 2024-09-13T14:37:44.458 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6490000000000003e-06 2024-09-13T14:37:44.458 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.458 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.458 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:44.458 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.754201+0000", 2024-09-13T14:37:44.459 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9190000000000002e-06 2024-09-13T14:37:44.459 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.459 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.459 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:44.459 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.757984+0000", 2024-09-13T14:37:44.459 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0037832400000000002 2024-09-13T14:37:44.459 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.459 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.459 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:37:44.459 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.758104+0000", 2024-09-13T14:37:44.459 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000120047 2024-09-13T14:37:44.459 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.459 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.459 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:37:44.460 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.758220+0000", 2024-09-13T14:37:44.460 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011538400000000001 2024-09-13T14:37:44.460 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.460 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.460 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:37:44.460 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.044352+0000", 2024-09-13T14:37:44.460 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.28613211500000002 2024-09-13T14:37:44.460 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.460 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.460 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:44.460 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.044970+0000", 2024-09-13T14:37:44.460 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00061775299999999997 2024-09-13T14:37:44.460 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.461 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.461 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:44.461 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.045002+0000", 2024-09-13T14:37:44.461 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2330999999999999e-05 2024-09-13T14:37:44.461 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.461 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.461 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:37:44.461 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.045037+0000", 2024-09-13T14:37:44.461 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5042000000000001e-05 2024-09-13T14:37:44.461 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.461 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.461 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:37:44.461 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.045069+0000", 2024-09-13T14:37:44.461 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.1695000000000001e-05 2024-09-13T14:37:44.462 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:44.462 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:44.462 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:44.462 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.462 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.462 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:150715 6.e 6:7aec09e5:::benchmark_data_smithi102_194590_object40091:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-13T14:37:44.462 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:33:27.761632+0000", 2024-09-13T14:37:44.462 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 256.655647284, 2024-09-13T14:37:44.462 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.29790872499999999, 2024-09-13T14:37:44.462 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:44.462 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:44.462 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:37:44.462 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:37:44.463 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:37:44.463 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 150715 2024-09-13T14:37:44.463 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.463 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:44.463 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.463 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:44.463 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.761632+0000", 2024-09-13T14:37:44.463 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:44.463 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.463 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.463 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:44.463 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.761632+0000", 2024-09-13T14:37:44.463 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:44.463 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.464 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.464 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:44.464 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.761630+0000", 2024-09-13T14:37:44.464 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:37:44.464 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.464 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.464 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:44.464 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.761638+0000", 2024-09-13T14:37:44.464 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1049999999999995e-06 2024-09-13T14:37:44.464 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.464 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.464 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:44.464 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.044265+0000", 2024-09-13T14:37:44.465 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.28262651100000002 2024-09-13T14:37:44.465 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.465 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.465 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:44.465 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.044280+0000", 2024-09-13T14:37:44.465 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5600999999999999e-05 2024-09-13T14:37:44.465 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.465 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.465 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:44.465 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.044408+0000", 2024-09-13T14:37:44.465 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012831499999999999 2024-09-13T14:37:44.465 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.465 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.465 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:37:44.466 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.044549+0000", 2024-09-13T14:37:44.466 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014089899999999999 2024-09-13T14:37:44.466 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.466 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.466 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,7", 2024-09-13T14:37:44.466 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.044679+0000", 2024-09-13T14:37:44.466 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012917399999999999 2024-09-13T14:37:44.466 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.466 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.466 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:44.466 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.059197+0000", 2024-09-13T14:37:44.466 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014518939999999999 2024-09-13T14:37:44.466 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.467 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.467 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:44.467 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.059225+0000", 2024-09-13T14:37:44.467 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7288999999999999e-05 2024-09-13T14:37:44.467 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.467 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.467 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:37:44.467 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.059358+0000", 2024-09-13T14:37:44.467 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000133122 2024-09-13T14:37:44.467 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.467 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.467 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:37:44.467 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.059449+0000", 2024-09-13T14:37:44.468 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1187000000000006e-05 2024-09-13T14:37:44.468 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.468 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.468 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:37:44.468 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.059540+0000", 2024-09-13T14:37:44.468 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1434000000000006e-05 2024-09-13T14:37:44.468 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:44.468 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:44.468 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:44.468 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.468 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.468 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35037.0:150736 6.7 e178/161 6:eea7a7c2:::benchmark_data_smithi102_194590_object40112:head v 178'9515, mlcod=178'9513)", 2024-09-13T14:37:44.468 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:33:27.781737+0000", 2024-09-13T14:37:44.469 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 256.63554251800002, 2024-09-13T14:37:44.469 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.283366118, 2024-09-13T14:37:44.469 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:44.469 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:44.469 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:44.469 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.469 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:44.469 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.781737+0000", 2024-09-13T14:37:44.469 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:44.469 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.469 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.469 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:44.469 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.781737+0000", 2024-09-13T14:37:44.469 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:44.470 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.470 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.470 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:44.470 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.781735+0000", 2024-09-13T14:37:44.470 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:37:44.470 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.470 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.470 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:44.470 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.781744+0000", 2024-09-13T14:37:44.470 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4760000000000004e-06 2024-09-13T14:37:44.470 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.470 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.470 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:44.471 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.044267+0000", 2024-09-13T14:37:44.471 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.26252262599999998 2024-09-13T14:37:44.471 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.471 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.471 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:44.471 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.044279+0000", 2024-09-13T14:37:44.471 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2354000000000001e-05 2024-09-13T14:37:44.471 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.471 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.471 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:44.471 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.044431+0000", 2024-09-13T14:37:44.471 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0001516 2024-09-13T14:37:44.471 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.471 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.472 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:37:44.472 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.044453+0000", 2024-09-13T14:37:44.472 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2124999999999999e-05 2024-09-13T14:37:44.472 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.472 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.472 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:37:44.472 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.065076+0000", 2024-09-13T14:37:44.472 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.020622628 2024-09-13T14:37:44.472 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.472 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.472 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:37:44.472 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.065103+0000", 2024-09-13T14:37:44.472 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7078999999999999e-05 2024-09-13T14:37:44.473 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:44.473 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:44.473 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:44.473 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.473 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.473 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:163490 6.e 6:7a5e8c84:::benchmark_data_smithi102_194590_object52866:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-13T14:37:44.473 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:33:43.515900+0000", 2024-09-13T14:37:44.473 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 240.90137870800001, 2024-09-13T14:37:44.473 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.555967597, 2024-09-13T14:37:44.473 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:44.473 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:44.473 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:37:44.473 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:37:44.474 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:37:44.474 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 163490 2024-09-13T14:37:44.474 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.474 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:44.474 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.474 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:44.474 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.515900+0000", 2024-09-13T14:37:44.474 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:44.474 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.474 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.474 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:44.474 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.515900+0000", 2024-09-13T14:37:44.474 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:44.474 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.475 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.475 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:44.475 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.515899+0000", 2024-09-13T14:37:44.475 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:37:44.475 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.475 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.475 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:44.475 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.515905+0000", 2024-09-13T14:37:44.475 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3219999999999998e-06 2024-09-13T14:37:44.475 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.475 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.475 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:44.475 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.515929+0000", 2024-09-13T14:37:44.476 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3445e-05 2024-09-13T14:37:44.476 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.476 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.476 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:44.476 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.515937+0000", 2024-09-13T14:37:44.476 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2530000000000004e-06 2024-09-13T14:37:44.476 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.476 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.476 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:44.476 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.533987+0000", 2024-09-13T14:37:44.476 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018049744 2024-09-13T14:37:44.476 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.477 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.477 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:37:44.477 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.533995+0000", 2024-09-13T14:37:44.477 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0250000000000002e-06 2024-09-13T14:37:44.477 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.477 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.477 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:44.477 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.820467+0000", 2024-09-13T14:37:44.477 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.28647251400000001 2024-09-13T14:37:44.477 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.477 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.477 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:37:44.477 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.820483+0000", 2024-09-13T14:37:44.477 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.596e-05 2024-09-13T14:37:44.478 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.478 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.478 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:44.478 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.025746+0000", 2024-09-13T14:37:44.478 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2052628699999999 2024-09-13T14:37:44.478 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.478 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.478 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:37:44.478 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.025880+0000", 2024-09-13T14:37:44.478 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000134039 2024-09-13T14:37:44.478 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.478 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.478 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6,7", 2024-09-13T14:37:44.478 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.026451+0000", 2024-09-13T14:37:44.479 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00057072799999999995 2024-09-13T14:37:44.479 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.479 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.479 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:44.479 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.061832+0000", 2024-09-13T14:37:44.479 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.035381126999999998 2024-09-13T14:37:44.479 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.479 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.479 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:37:44.479 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.063912+0000", 2024-09-13T14:37:44.479 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0020797739999999999 2024-09-13T14:37:44.479 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.479 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.479 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:44.480 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.066465+0000", 2024-09-13T14:37:44.481 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0025533280000000001 2024-09-13T14:37:44.481 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.481 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.481 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:44.481 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.071666+0000", 2024-09-13T14:37:44.481 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0052006839999999997 2024-09-13T14:37:44.481 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.481 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.481 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:37:44.482 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.071756+0000", 2024-09-13T14:37:44.482 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.0733e-05 2024-09-13T14:37:44.482 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.482 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.482 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:37:44.482 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.071868+0000", 2024-09-13T14:37:44.482 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000111537 2024-09-13T14:37:44.482 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:44.482 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:44.482 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:44.482 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.482 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.482 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:163524 6.e 6:7ba5d4da:::benchmark_data_smithi102_194590_object52900:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-09-13T14:37:44.483 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:33:43.559673+0000", 2024-09-13T14:37:44.483 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 240.857606433, 2024-09-13T14:37:44.483 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5124033290000001, 2024-09-13T14:37:44.483 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:44.483 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:44.483 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:37:44.483 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:37:44.483 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:37:44.483 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 163524 2024-09-13T14:37:44.483 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.483 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:44.483 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.483 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:44.483 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.559673+0000", 2024-09-13T14:37:44.484 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:44.484 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.484 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.484 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:44.484 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.559673+0000", 2024-09-13T14:37:44.484 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:44.484 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.484 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.484 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:44.484 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.559672+0000", 2024-09-13T14:37:44.484 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:37:44.484 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.484 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.484 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:44.485 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.559676+0000", 2024-09-13T14:37:44.485 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6859999999999997e-06 2024-09-13T14:37:44.485 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.485 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.485 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:44.485 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.559687+0000", 2024-09-13T14:37:44.485 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0514e-05 2024-09-13T14:37:44.485 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.485 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.485 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:44.485 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.559691+0000", 2024-09-13T14:37:44.485 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4989999999999996e-06 2024-09-13T14:37:44.485 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.485 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.486 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:44.486 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.559730+0000", 2024-09-13T14:37:44.486 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8816999999999998e-05 2024-09-13T14:37:44.486 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.486 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.486 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:37:44.486 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.559740+0000", 2024-09-13T14:37:44.486 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.5319999999999995e-06 2024-09-13T14:37:44.486 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.486 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.486 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:44.486 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.820610+0000", 2024-09-13T14:37:44.486 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.26086995699999999 2024-09-13T14:37:44.487 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.487 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.487 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:37:44.487 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.820618+0000", 2024-09-13T14:37:44.487 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.6580000000000008e-06 2024-09-13T14:37:44.487 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.487 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.487 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:44.487 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.040556+0000", 2024-09-13T14:37:44.487 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2199382860000001 2024-09-13T14:37:44.487 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.487 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.487 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:37:44.487 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.040691+0000", 2024-09-13T14:37:44.488 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000134497 2024-09-13T14:37:44.488 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.488 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.488 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6,7", 2024-09-13T14:37:44.488 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.040831+0000", 2024-09-13T14:37:44.488 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000140359 2024-09-13T14:37:44.488 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.488 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.488 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:44.488 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.061864+0000", 2024-09-13T14:37:44.488 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.021032660000000002 2024-09-13T14:37:44.488 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.488 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.489 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:44.489 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.066495+0000", 2024-09-13T14:37:44.489 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0046312369999999999 2024-09-13T14:37:44.489 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.489 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.489 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:37:44.489 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.068838+0000", 2024-09-13T14:37:44.489 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0023430949999999999 2024-09-13T14:37:44.489 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.489 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.489 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:44.489 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.071910+0000", 2024-09-13T14:37:44.489 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0030720309999999998 2024-09-13T14:37:44.489 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.490 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.490 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:37:44.490 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.071984+0000", 2024-09-13T14:37:44.490 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3590000000000005e-05 2024-09-13T14:37:44.490 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.490 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.490 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:37:44.490 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.072076+0000", 2024-09-13T14:37:44.490 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1986999999999999e-05 2024-09-13T14:37:44.490 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:44.490 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:44.490 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:44.490 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.490 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.491 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:163533 6.e 6:70a00fb8:::benchmark_data_smithi102_194590_object52909:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-09-13T14:37:44.491 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:33:43.580375+0000", 2024-09-13T14:37:44.491 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 240.83690443, 2024-09-13T14:37:44.491 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4918938339999999, 2024-09-13T14:37:44.491 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:44.491 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:44.491 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:37:44.491 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:37:44.491 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:37:44.491 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 163533 2024-09-13T14:37:44.491 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.491 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:44.491 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.491 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:44.492 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.580375+0000", 2024-09-13T14:37:44.492 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:44.492 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.492 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.492 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:44.492 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.580375+0000", 2024-09-13T14:37:44.492 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:44.492 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.492 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.492 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:44.492 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.580373+0000", 2024-09-13T14:37:44.492 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:37:44.492 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.493 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.493 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:44.493 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.580380+0000", 2024-09-13T14:37:44.493 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6980000000000001e-06 2024-09-13T14:37:44.493 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.493 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.493 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:44.493 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.580392+0000", 2024-09-13T14:37:44.493 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2675e-05 2024-09-13T14:37:44.493 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.493 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.493 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:44.493 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.580399+0000", 2024-09-13T14:37:44.493 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.674e-06 2024-09-13T14:37:44.494 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.494 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.494 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:44.494 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.580441+0000", 2024-09-13T14:37:44.494 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2077999999999999e-05 2024-09-13T14:37:44.494 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.494 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.494 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:37:44.494 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.580449+0000", 2024-09-13T14:37:44.494 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7300000000000005e-06 2024-09-13T14:37:44.494 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.494 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.494 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:44.494 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.820640+0000", 2024-09-13T14:37:44.495 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.240191126 2024-09-13T14:37:44.495 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.495 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.495 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:37:44.495 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.820648+0000", 2024-09-13T14:37:44.495 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0469999999999994e-06 2024-09-13T14:37:44.495 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.495 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.495 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:44.495 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.043836+0000", 2024-09-13T14:37:44.495 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2231876559999999 2024-09-13T14:37:44.495 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.495 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.495 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:37:44.496 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.043970+0000", 2024-09-13T14:37:44.496 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013394399999999999 2024-09-13T14:37:44.496 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.496 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.496 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6,7", 2024-09-13T14:37:44.496 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.044102+0000", 2024-09-13T14:37:44.496 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000132254 2024-09-13T14:37:44.496 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.496 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.496 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:44.496 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.061893+0000", 2024-09-13T14:37:44.496 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017791238000000001 2024-09-13T14:37:44.496 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.497 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.497 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:44.497 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.066525+0000", 2024-09-13T14:37:44.497 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0046314099999999999 2024-09-13T14:37:44.497 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.497 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.497 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:37:44.497 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.068872+0000", 2024-09-13T14:37:44.497 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002347605 2024-09-13T14:37:44.497 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.497 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.497 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:44.497 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.072115+0000", 2024-09-13T14:37:44.497 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0032423840000000001 2024-09-13T14:37:44.498 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.498 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.498 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:37:44.498 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.072177+0000", 2024-09-13T14:37:44.498 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2679000000000001e-05 2024-09-13T14:37:44.498 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.498 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.498 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:37:44.498 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.072268+0000", 2024-09-13T14:37:44.498 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1260000000000004e-05 2024-09-13T14:37:44.498 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:44.498 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:44.498 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:44.498 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.499 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.499 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:191635 6.e 6:71f9b24e:::benchmark_data_smithi102_194590_object81011:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-09-13T14:37:44.499 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:34:22.367198+0000", 2024-09-13T14:37:44.499 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 202.05008097199999, 2024-09-13T14:37:44.499 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.26435987, 2024-09-13T14:37:44.499 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:44.499 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:44.499 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:37:44.499 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:37:44.499 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:37:44.499 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 191635 2024-09-13T14:37:44.499 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.499 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:44.499 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.500 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:44.500 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.367198+0000", 2024-09-13T14:37:44.500 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:44.500 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.500 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.500 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:44.500 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.367198+0000", 2024-09-13T14:37:44.500 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:44.500 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.500 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.500 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:44.500 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.367197+0000", 2024-09-13T14:37:44.500 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:37:44.501 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.501 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.501 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:44.501 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.367203+0000", 2024-09-13T14:37:44.501 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.455e-06 2024-09-13T14:37:44.501 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.501 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.501 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:44.501 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.367222+0000", 2024-09-13T14:37:44.501 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8655e-05 2024-09-13T14:37:44.501 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.501 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.501 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:44.502 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.367230+0000", 2024-09-13T14:37:44.502 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7379999999999994e-06 2024-09-13T14:37:44.502 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.502 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.502 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:44.502 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.367927+0000", 2024-09-13T14:37:44.502 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00069707899999999997 2024-09-13T14:37:44.502 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.502 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.502 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:37:44.502 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.368049+0000", 2024-09-13T14:37:44.502 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000122211 2024-09-13T14:37:44.502 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.502 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.503 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6,7", 2024-09-13T14:37:44.503 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.368168+0000", 2024-09-13T14:37:44.503 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000119357 2024-09-13T14:37:44.503 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.503 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.503 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:37:44.503 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.371968+0000", 2024-09-13T14:37:44.503 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003800159 2024-09-13T14:37:44.503 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.503 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.503 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:44.503 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.380532+0000", 2024-09-13T14:37:44.503 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0085635420000000004 2024-09-13T14:37:44.504 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.504 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.504 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:44.504 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.380575+0000", 2024-09-13T14:37:44.504 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3392000000000001e-05 2024-09-13T14:37:44.504 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.504 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.504 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:44.504 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.631340+0000", 2024-09-13T14:37:44.504 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.250764879 2024-09-13T14:37:44.504 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.504 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.504 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:37:44.505 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.631451+0000", 2024-09-13T14:37:44.505 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000111078 2024-09-13T14:37:44.505 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.505 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.505 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:37:44.505 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.631558+0000", 2024-09-13T14:37:44.505 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010673799999999999 2024-09-13T14:37:44.505 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:44.505 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:44.505 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:44.505 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.505 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.505 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:191643 6.e 6:7b30bcbb:::benchmark_data_smithi102_194590_object81019:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-09-13T14:37:44.506 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:34:22.372207+0000", 2024-09-13T14:37:44.506 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 202.04507224400001, 2024-09-13T14:37:44.506 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.26177498799999999, 2024-09-13T14:37:44.506 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:44.506 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:44.506 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:37:44.506 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:37:44.506 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:37:44.506 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 191643 2024-09-13T14:37:44.506 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.506 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:44.506 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.506 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:44.506 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.372207+0000", 2024-09-13T14:37:44.506 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:44.507 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.507 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.507 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:44.507 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.372207+0000", 2024-09-13T14:37:44.507 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:44.507 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.507 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.507 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:44.507 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.372205+0000", 2024-09-13T14:37:44.507 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:37:44.507 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.507 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.507 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:44.507 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.372213+0000", 2024-09-13T14:37:44.508 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1400000000000002e-06 2024-09-13T14:37:44.508 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.508 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.508 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:44.508 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.372225+0000", 2024-09-13T14:37:44.508 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2577e-05 2024-09-13T14:37:44.508 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.508 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.508 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:44.508 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.372232+0000", 2024-09-13T14:37:44.508 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7139999999999996e-06 2024-09-13T14:37:44.508 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.508 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.509 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:44.509 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.372269+0000", 2024-09-13T14:37:44.509 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7327999999999999e-05 2024-09-13T14:37:44.509 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.509 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.509 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:37:44.509 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.372400+0000", 2024-09-13T14:37:44.509 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000131085 2024-09-13T14:37:44.509 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.509 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.509 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6,7", 2024-09-13T14:37:44.509 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.372529+0000", 2024-09-13T14:37:44.509 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012839700000000001 2024-09-13T14:37:44.509 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.510 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.510 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:37:44.510 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.380540+0000", 2024-09-13T14:37:44.510 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0080113100000000007 2024-09-13T14:37:44.510 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.510 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.510 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:44.510 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.380608+0000", 2024-09-13T14:37:44.510 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7725000000000006e-05 2024-09-13T14:37:44.510 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.510 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.510 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:44.510 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.380655+0000", 2024-09-13T14:37:44.510 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7085000000000001e-05 2024-09-13T14:37:44.511 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.511 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.511 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:44.511 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.631591+0000", 2024-09-13T14:37:44.511 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.25093650699999998 2024-09-13T14:37:44.511 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.511 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.511 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:37:44.511 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.633854+0000", 2024-09-13T14:37:44.511 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0022624239999999999 2024-09-13T14:37:44.511 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.511 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.511 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:37:44.511 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.633982+0000", 2024-09-13T14:37:44.512 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012812899999999999 2024-09-13T14:37:44.512 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:44.512 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:44.512 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:44.512 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.512 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.512 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:191647 6.e 6:76d424e6:::benchmark_data_smithi102_194590_object81023:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-09-13T14:37:44.512 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:34:22.373106+0000", 2024-09-13T14:37:44.512 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 202.04417277600001, 2024-09-13T14:37:44.512 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.26104791700000002, 2024-09-13T14:37:44.512 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:44.512 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:44.513 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:37:44.513 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:37:44.513 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:37:44.513 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 191647 2024-09-13T14:37:44.513 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.513 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:44.513 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.513 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:44.513 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.373106+0000", 2024-09-13T14:37:44.513 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:44.513 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.513 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.513 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:44.513 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.373106+0000", 2024-09-13T14:37:44.514 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:44.514 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.514 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.514 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:44.514 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.373105+0000", 2024-09-13T14:37:44.514 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:37:44.514 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.514 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.514 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:44.514 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.373110+0000", 2024-09-13T14:37:44.514 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5220000000000001e-06 2024-09-13T14:37:44.514 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.514 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.515 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:44.515 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.373134+0000", 2024-09-13T14:37:44.515 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4409e-05 2024-09-13T14:37:44.515 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.515 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.515 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:44.515 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.373141+0000", 2024-09-13T14:37:44.515 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4860000000000002e-06 2024-09-13T14:37:44.515 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.515 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.515 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:44.515 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.377126+0000", 2024-09-13T14:37:44.515 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003985321 2024-09-13T14:37:44.515 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.516 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.516 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:37:44.516 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.377248+0000", 2024-09-13T14:37:44.516 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012149199999999999 2024-09-13T14:37:44.516 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.516 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.516 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6,7", 2024-09-13T14:37:44.516 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.377456+0000", 2024-09-13T14:37:44.516 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00020867600000000001 2024-09-13T14:37:44.517 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.517 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.517 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:37:44.517 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.381454+0000", 2024-09-13T14:37:44.517 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0039976029999999997 2024-09-13T14:37:44.517 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.517 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.517 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:44.517 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.382225+0000", 2024-09-13T14:37:44.517 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00077125399999999997 2024-09-13T14:37:44.517 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.517 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.517 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:44.518 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.383736+0000", 2024-09-13T14:37:44.518 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0015105279999999999 2024-09-13T14:37:44.518 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.518 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.518 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:44.518 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.634027+0000", 2024-09-13T14:37:44.518 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.25029173100000002 2024-09-13T14:37:44.518 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.518 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.518 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:37:44.518 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.634065+0000", 2024-09-13T14:37:44.518 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7135000000000003e-05 2024-09-13T14:37:44.518 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.518 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.519 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:37:44.519 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.634154+0000", 2024-09-13T14:37:44.519 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.9624999999999997e-05 2024-09-13T14:37:44.519 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:44.519 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:44.519 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:44.519 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.519 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.519 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:191656 6.e 6:73cf8b7b:::benchmark_data_smithi102_194590_object81032:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-09-13T14:37:44.519 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:34:22.389704+0000", 2024-09-13T14:37:44.519 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 202.027575321, 2024-09-13T14:37:44.519 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.24469366300000001, 2024-09-13T14:37:44.519 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:44.519 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:44.520 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:37:44.520 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:37:44.520 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:37:44.520 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 191656 2024-09-13T14:37:44.520 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.520 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:44.520 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.520 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:44.520 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.389704+0000", 2024-09-13T14:37:44.520 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:44.520 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.520 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.520 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:44.520 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.389704+0000", 2024-09-13T14:37:44.521 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:44.521 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.521 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.521 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:44.521 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.389702+0000", 2024-09-13T14:37:44.521 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:37:44.521 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.521 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.521 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:44.521 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.389708+0000", 2024-09-13T14:37:44.521 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2670000000000003e-06 2024-09-13T14:37:44.521 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.521 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.521 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:44.522 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.389723+0000", 2024-09-13T14:37:44.522 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4175000000000001e-05 2024-09-13T14:37:44.522 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.522 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.522 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:44.522 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.389730+0000", 2024-09-13T14:37:44.522 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5480000000000004e-06 2024-09-13T14:37:44.522 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.522 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.522 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:44.522 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.389767+0000", 2024-09-13T14:37:44.522 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7123999999999999e-05 2024-09-13T14:37:44.522 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.522 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.523 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:37:44.523 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.389868+0000", 2024-09-13T14:37:44.523 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010078599999999999 2024-09-13T14:37:44.523 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.523 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.523 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6,7", 2024-09-13T14:37:44.523 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.390004+0000", 2024-09-13T14:37:44.523 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000135578 2024-09-13T14:37:44.523 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.523 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.523 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:44.523 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.393200+0000", 2024-09-13T14:37:44.523 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0031962649999999998 2024-09-13T14:37:44.524 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.524 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.524 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:37:44.524 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.396346+0000", 2024-09-13T14:37:44.524 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003146009 2024-09-13T14:37:44.524 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.524 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.524 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:44.524 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.396394+0000", 2024-09-13T14:37:44.524 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8588000000000003e-05 2024-09-13T14:37:44.524 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.524 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.524 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:44.524 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.634261+0000", 2024-09-13T14:37:44.525 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.237866258 2024-09-13T14:37:44.525 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.525 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.525 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:37:44.525 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.634301+0000", 2024-09-13T14:37:44.525 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.019e-05 2024-09-13T14:37:44.525 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.525 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.525 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:37:44.525 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.634397+0000", 2024-09-13T14:37:44.525 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6477999999999993e-05 2024-09-13T14:37:44.525 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:44.525 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:44.525 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:44.526 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.526 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.526 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:191660 6.e 6:738104a9:::benchmark_data_smithi102_194590_object81036:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-09-13T14:37:44.526 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:34:22.392930+0000", 2024-09-13T14:37:44.526 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 202.024348766, 2024-09-13T14:37:44.526 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.24195140200000001, 2024-09-13T14:37:44.526 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:44.526 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:44.526 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:37:44.526 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:37:44.526 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:37:44.526 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 191660 2024-09-13T14:37:44.526 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.526 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:44.527 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.527 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:44.527 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.392930+0000", 2024-09-13T14:37:44.527 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:44.527 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.527 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.527 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:44.527 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.392930+0000", 2024-09-13T14:37:44.527 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:44.527 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.527 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.527 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:44.527 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.392929+0000", 2024-09-13T14:37:44.528 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:37:44.528 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.528 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.528 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:44.528 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.392935+0000", 2024-09-13T14:37:44.528 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0859999999999998e-06 2024-09-13T14:37:44.528 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.528 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.528 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:44.528 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.392948+0000", 2024-09-13T14:37:44.528 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2529e-05 2024-09-13T14:37:44.528 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.528 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.528 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:44.529 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.392954+0000", 2024-09-13T14:37:44.529 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9920000000000001e-06 2024-09-13T14:37:44.529 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.529 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.529 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:44.529 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.393326+0000", 2024-09-13T14:37:44.529 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00037141000000000002 2024-09-13T14:37:44.529 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.529 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.529 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:37:44.529 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.393461+0000", 2024-09-13T14:37:44.529 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013538499999999999 2024-09-13T14:37:44.529 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.530 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.530 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6,7", 2024-09-13T14:37:44.530 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.393618+0000", 2024-09-13T14:37:44.530 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000156541 2024-09-13T14:37:44.530 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.530 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.530 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:44.530 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.396432+0000", 2024-09-13T14:37:44.530 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00281445 2024-09-13T14:37:44.530 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.530 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.530 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:37:44.530 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.396443+0000", 2024-09-13T14:37:44.530 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0852999999999999e-05 2024-09-13T14:37:44.530 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.531 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.531 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:44.531 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.396763+0000", 2024-09-13T14:37:44.531 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00031982700000000001 2024-09-13T14:37:44.531 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.531 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.531 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:44.531 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.634751+0000", 2024-09-13T14:37:44.531 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.23798829299999999 2024-09-13T14:37:44.531 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.531 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.531 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:37:44.531 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.634794+0000", 2024-09-13T14:37:44.531 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2425999999999997e-05 2024-09-13T14:37:44.532 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.532 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.532 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:37:44.532 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.634882+0000", 2024-09-13T14:37:44.532 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8011999999999997e-05 2024-09-13T14:37:44.532 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:44.532 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:44.532 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:44.532 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.532 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.532 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:198028 6.e 6:71106726:::benchmark_data_smithi102_194590_object87404:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-09-13T14:37:44.532 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:34:29.683014+0000", 2024-09-13T14:37:44.532 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 194.73426470499999, 2024-09-13T14:37:44.533 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.22173479500000001, 2024-09-13T14:37:44.533 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:44.533 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:44.533 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:37:44.533 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:37:44.533 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:37:44.533 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 198028 2024-09-13T14:37:44.533 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.533 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:44.533 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.533 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:44.533 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.683014+0000", 2024-09-13T14:37:44.533 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:44.533 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.533 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.534 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:44.534 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.683014+0000", 2024-09-13T14:37:44.534 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:44.534 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.534 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.534 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:44.534 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.683013+0000", 2024-09-13T14:37:44.534 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:37:44.534 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.534 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.534 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:44.534 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.683019+0000", 2024-09-13T14:37:44.534 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1550000000000004e-06 2024-09-13T14:37:44.535 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.535 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.535 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:44.535 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.683034+0000", 2024-09-13T14:37:44.535 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4796e-05 2024-09-13T14:37:44.535 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.535 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.535 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:44.535 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.683041+0000", 2024-09-13T14:37:44.535 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4699999999999996e-06 2024-09-13T14:37:44.535 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.535 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.535 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:44.535 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.685659+0000", 2024-09-13T14:37:44.536 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0026179699999999998 2024-09-13T14:37:44.536 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.536 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.536 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:37:44.536 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.685968+0000", 2024-09-13T14:37:44.536 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00030874299999999999 2024-09-13T14:37:44.536 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.536 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.536 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6,7", 2024-09-13T14:37:44.536 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.686387+0000", 2024-09-13T14:37:44.536 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00041937000000000002 2024-09-13T14:37:44.536 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.536 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.537 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:44.537 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.689774+0000", 2024-09-13T14:37:44.537 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0033865100000000001 2024-09-13T14:37:44.537 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.537 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.537 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:44.537 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.691584+0000", 2024-09-13T14:37:44.537 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0018108029999999999 2024-09-13T14:37:44.537 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.537 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.537 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:37:44.537 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.692138+0000", 2024-09-13T14:37:44.537 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00055365500000000001 2024-09-13T14:37:44.537 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.538 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.538 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:44.538 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.904593+0000", 2024-09-13T14:37:44.538 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.21245457600000001 2024-09-13T14:37:44.538 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.538 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.538 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:37:44.538 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.904642+0000", 2024-09-13T14:37:44.538 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9833999999999998e-05 2024-09-13T14:37:44.538 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.538 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.538 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:37:44.538 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.904749+0000", 2024-09-13T14:37:44.538 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000106693 2024-09-13T14:37:44.539 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:44.539 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:44.539 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:44.539 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.539 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.539 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:198030 6.2 6:416e1cf0:::benchmark_data_smithi102_194590_object87406:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-09-13T14:37:44.539 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:34:29.683047+0000", 2024-09-13T14:37:44.539 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 194.73423198899999, 2024-09-13T14:37:44.539 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.221895961, 2024-09-13T14:37:44.539 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:44.539 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:44.539 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:37:44.539 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:37:44.539 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:37:44.540 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 198030 2024-09-13T14:37:44.540 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.540 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:44.540 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.540 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:44.540 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.683047+0000", 2024-09-13T14:37:44.540 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:44.540 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.540 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.540 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:44.540 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.683047+0000", 2024-09-13T14:37:44.540 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:44.541 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.541 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.541 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:44.541 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.683047+0000", 2024-09-13T14:37:44.541 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:37:44.541 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.541 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.541 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:44.541 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.683049+0000", 2024-09-13T14:37:44.541 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.424e-06 2024-09-13T14:37:44.541 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.541 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.541 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:44.541 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.683058+0000", 2024-09-13T14:37:44.542 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7450000000000001e-06 2024-09-13T14:37:44.542 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.542 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.542 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:44.542 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.683063+0000", 2024-09-13T14:37:44.542 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0089999999999998e-06 2024-09-13T14:37:44.542 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.542 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.542 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:44.542 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.683110+0000", 2024-09-13T14:37:44.542 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7601e-05 2024-09-13T14:37:44.542 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.542 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.543 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:37:44.543 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.683250+0000", 2024-09-13T14:37:44.543 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013993600000000001 2024-09-13T14:37:44.543 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.543 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.543 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:37:44.543 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.683385+0000", 2024-09-13T14:37:44.543 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013476499999999999 2024-09-13T14:37:44.543 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.543 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.543 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:37:44.543 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.686273+0000", 2024-09-13T14:37:44.543 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0028874199999999999 2024-09-13T14:37:44.543 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.544 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.544 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:44.544 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.686610+0000", 2024-09-13T14:37:44.544 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00033694600000000002 2024-09-13T14:37:44.544 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.544 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.544 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:44.544 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.904866+0000", 2024-09-13T14:37:44.544 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.21825642100000001 2024-09-13T14:37:44.544 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.544 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.544 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:37:44.544 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.904903+0000", 2024-09-13T14:37:44.545 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7057000000000001e-05 2024-09-13T14:37:44.545 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.545 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.545 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:37:44.545 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.904943+0000", 2024-09-13T14:37:44.545 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0006999999999998e-05 2024-09-13T14:37:44.545 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:44.545 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:44.545 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:44.545 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.545 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.545 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDPGPushReply(7.4 231/222 [PushReplyOp(7:2bfb3853:::benchmark_data_smithi102_227395_object42539:head)])", 2024-09-13T14:37:44.545 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:21.387032+0000", 2024-09-13T14:37:44.545 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 23.0302474, 2024-09-13T14:37:44.546 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.16156214199999999, 2024-09-13T14:37:44.546 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:44.546 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-13T14:37:44.546 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:44.546 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.546 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:44.546 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:21.387032+0000", 2024-09-13T14:37:44.546 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:44.546 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.546 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.546 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:44.546 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:21.387032+0000", 2024-09-13T14:37:44.546 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:44.546 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.547 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.547 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:44.547 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:21.387031+0000", 2024-09-13T14:37:44.547 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:37:44.547 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.547 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.547 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:44.547 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:21.387034+0000", 2024-09-13T14:37:44.547 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8930000000000001e-06 2024-09-13T14:37:44.547 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.547 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.547 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:44.547 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:21.387040+0000", 2024-09-13T14:37:44.547 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1750000000000002e-06 2024-09-13T14:37:44.547 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.548 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.548 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:44.548 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:21.387045+0000", 2024-09-13T14:37:44.548 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0209999999999999e-06 2024-09-13T14:37:44.548 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.548 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.548 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:44.548 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:21.548310+0000", 2024-09-13T14:37:44.548 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.16126428000000001 2024-09-13T14:37:44.548 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.548 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.548 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:37:44.548 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:21.548594+0000", 2024-09-13T14:37:44.549 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000284102 2024-09-13T14:37:44.549 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:44.549 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:44.549 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:44.549 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.549 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.549 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDPGPush(7.5 231/222 [PushOp(7:a35ea534:::benchmark_data_smithi102_227395_object28148:head, version: 209'1749, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:a35ea534:::benchmark_data_smithi102_227395_object28148:head@209'1749, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T14:37:44.549 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:21.394245+0000", 2024-09-13T14:37:44.549 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 23.023034484, 2024-09-13T14:37:44.549 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.15708040200000001, 2024-09-13T14:37:44.549 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:44.549 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T14:37:44.549 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:44.549 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.550 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:44.550 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:21.394245+0000", 2024-09-13T14:37:44.550 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:44.550 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.550 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.550 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:44.550 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:21.394245+0000", 2024-09-13T14:37:44.550 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:44.550 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.550 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.550 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:44.550 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:21.394243+0000", 2024-09-13T14:37:44.550 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:37:44.550 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.551 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.551 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:44.551 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:21.394277+0000", 2024-09-13T14:37:44.551 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4430000000000001e-05 2024-09-13T14:37:44.551 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.551 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.551 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:44.551 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:21.548466+0000", 2024-09-13T14:37:44.551 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.15418857699999999 2024-09-13T14:37:44.551 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.551 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.551 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:44.551 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:21.548504+0000", 2024-09-13T14:37:44.551 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8098e-05 2024-09-13T14:37:44.552 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.552 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.552 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:44.552 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:21.548671+0000", 2024-09-13T14:37:44.552 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000167188 2024-09-13T14:37:44.552 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.552 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.552 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:37:44.552 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:21.548683+0000", 2024-09-13T14:37:44.552 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1892000000000001e-05 2024-09-13T14:37:44.552 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:44.552 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:44.552 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:37:44.552 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:21.551325+0000", 2024-09-13T14:37:44.553 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0026417849999999998 2024-09-13T14:37:44.553 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:44.553 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:44.553 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:44.553 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:44.553 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:44.553 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:37:44.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:44.555 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:44.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:44.655 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:44.677 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:44.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:44.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:44.778 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:44.806 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:44.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:44.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:44.908 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:44.929 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:44.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:44.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:45.032 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:45.053 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:45.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:45.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:45.155 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:45.178 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:45.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:45.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:45.247 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 14:37:45 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T14:37:45.172+0000 7f8111415640 -1 osd.6 210 log_to_monitors true 2024-09-13T14:37:45.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:45 smithi195 ceph-mon[104262]: pgmap v758: 89 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 38 active+clean+remapped, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 25 active+clean; 6.8 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 64 op/s; 49148/335709 objects degraded (14.640%); 56247/335709 objects misplaced (16.755%); 52 MiB/s, 827 objects/s recovering 2024-09-13T14:37:45.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:45 smithi195 ceph-mon[104262]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:37:45.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:45 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:37:45.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:45 smithi195 ceph-mon[104262]: osdmap e237: 8 total, 7 up, 6 in 2024-09-13T14:37:45.279 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:45.303 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:45.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:45.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:45.405 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:45.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:45 smithi102 ceph-mon[110183]: pgmap v758: 89 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 38 active+clean+remapped, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 25 active+clean; 6.8 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 64 op/s; 49148/335709 objects degraded (14.640%); 56247/335709 objects misplaced (16.755%); 52 MiB/s, 827 objects/s recovering 2024-09-13T14:37:45.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:45 smithi102 ceph-mon[110183]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:37:45.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:45 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:37:45.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:45 smithi102 ceph-mon[110183]: osdmap e237: 8 total, 7 up, 6 in 2024-09-13T14:37:45.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:45 smithi102 ceph-mon[115269]: pgmap v758: 89 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 38 active+clean+remapped, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 25 active+clean; 6.8 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 64 op/s; 49148/335709 objects degraded (14.640%); 56247/335709 objects misplaced (16.755%); 52 MiB/s, 827 objects/s recovering 2024-09-13T14:37:45.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:45 smithi102 ceph-mon[115269]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:37:45.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:45 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:37:45.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:45 smithi102 ceph-mon[115269]: osdmap e237: 8 total, 7 up, 6 in 2024-09-13T14:37:45.429 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:45.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:45.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:45.454 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T14:37:45.454 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-6, ['dump_ops_in_flight'] 2024-09-13T14:37:45.531 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:45.553 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:45.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:45.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:45.655 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:45.681 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:45.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:45.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:45.783 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:45.807 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:45.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:45.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:45.909 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:45.931 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:45.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:45.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:46.033 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:46.054 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:46.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:46.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:46.156 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:46.184 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:46.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:46.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:46.186 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T14:37:46.286 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:46.308 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:46.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:46.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:46.410 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:46.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:46 smithi102 ceph-mon[110183]: from='osd.6 [v2:172.21.15.195:6816/3802194762,v1:172.21.15.195:6817/3802194762]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-13T14:37:46.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:46 smithi102 ceph-mon[110183]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-13T14:37:46.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:46 smithi102 ceph-mon[115269]: from='osd.6 [v2:172.21.15.195:6816/3802194762,v1:172.21.15.195:6817/3802194762]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-13T14:37:46.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:46 smithi102 ceph-mon[115269]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-13T14:37:46.435 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:46.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:46.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:46.482 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:37:46.497 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 14:37:46 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T14:37:46.169+0000 7f81051b7640 -1 osd.6 238 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T14:37:46.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:46 smithi195 ceph-mon[104262]: from='osd.6 [v2:172.21.15.195:6816/3802194762,v1:172.21.15.195:6817/3802194762]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-13T14:37:46.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:46 smithi195 ceph-mon[104262]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-13T14:37:46.537 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:46.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:46.559 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:46.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:46.660 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:46.685 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:46.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:46.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:46.787 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:46.811 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:46.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:46.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:46.913 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:46.918 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:37:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:37:46] "GET /metrics HTTP/1.1" 200 36578 "" "Prometheus/2.51.0" 2024-09-13T14:37:46.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:46.935 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:46.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:47.036 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:47.063 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:47.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:47.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:47.165 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:47.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:47.190 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:47.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:47.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:47 smithi195 ceph-mon[104262]: pgmap v760: 89 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 38 active+clean+remapped, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 25 active+clean; 6.8 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 37 op/s; 49148/335709 objects degraded (14.640%); 56247/335709 objects misplaced (16.755%); 44 MiB/s, 708 objects/s recovering 2024-09-13T14:37:47.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:47 smithi195 ceph-mon[104262]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-13T14:37:47.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:47 smithi195 ceph-mon[104262]: osdmap e238: 8 total, 7 up, 6 in 2024-09-13T14:37:47.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:47 smithi195 ceph-mon[104262]: from='osd.6 [v2:172.21.15.195:6816/3802194762,v1:172.21.15.195:6817/3802194762]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi195", "root=default"]} : dispatch 2024-09-13T14:37:47.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:47 smithi195 ceph-mon[104262]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi195", "root=default"]} : dispatch 2024-09-13T14:37:47.291 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:47.312 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:47.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:47.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:47.413 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:47 smithi102 ceph-mon[110183]: pgmap v760: 89 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 38 active+clean+remapped, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 25 active+clean; 6.8 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 37 op/s; 49148/335709 objects degraded (14.640%); 56247/335709 objects misplaced (16.755%); 44 MiB/s, 708 objects/s recovering 2024-09-13T14:37:47.413 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:47 smithi102 ceph-mon[110183]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-13T14:37:47.413 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:47 smithi102 ceph-mon[110183]: osdmap e238: 8 total, 7 up, 6 in 2024-09-13T14:37:47.414 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:47 smithi102 ceph-mon[110183]: from='osd.6 [v2:172.21.15.195:6816/3802194762,v1:172.21.15.195:6817/3802194762]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi195", "root=default"]} : dispatch 2024-09-13T14:37:47.414 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:47 smithi102 ceph-mon[110183]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi195", "root=default"]} : dispatch 2024-09-13T14:37:47.414 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:47 smithi102 ceph-mon[115269]: pgmap v760: 89 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 38 active+clean+remapped, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+undersized, 7 active+undersized+degraded, 25 active+clean; 6.8 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 37 op/s; 49148/335709 objects degraded (14.640%); 56247/335709 objects misplaced (16.755%); 44 MiB/s, 708 objects/s recovering 2024-09-13T14:37:47.414 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:47 smithi102 ceph-mon[115269]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-13T14:37:47.414 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:47 smithi102 ceph-mon[115269]: osdmap e238: 8 total, 7 up, 6 in 2024-09-13T14:37:47.414 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:47 smithi102 ceph-mon[115269]: from='osd.6 [v2:172.21.15.195:6816/3802194762,v1:172.21.15.195:6817/3802194762]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi195", "root=default"]} : dispatch 2024-09-13T14:37:47.414 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:47 smithi102 ceph-mon[115269]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi195", "root=default"]} : dispatch 2024-09-13T14:37:47.415 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:47.435 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:47.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:47.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:47.537 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:47.559 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:47.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:47.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:47.661 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:47.682 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:47.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:47.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:47.784 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:47.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:47.807 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:47.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:47.908 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:47.931 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:47.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:47.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:48.033 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:48.058 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:48.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:48.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:48.160 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:48.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:48.183 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:48.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:48.285 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:48.300 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:48 smithi195 ceph-mon[104262]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:37:48.300 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:48 smithi195 ceph-mon[104262]: osd.6 [v2:172.21.15.195:6816/3802194762,v1:172.21.15.195:6817/3802194762] boot 2024-09-13T14:37:48.300 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:48 smithi195 ceph-mon[104262]: osdmap e239: 8 total, 8 up, 6 in 2024-09-13T14:37:48.300 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:48 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-13T14:37:48.300 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:48 smithi195 ceph-mon[104262]: osdmap e240: 8 total, 8 up, 6 in 2024-09-13T14:37:48.307 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:48.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:48.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:48.314 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:37:48.315 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-13T14:37:48.315 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.315 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113976 7.0 7.23360e40 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-13T14:37:48.315 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:47.285996+0000", 2024-09-13T14:37:48.315 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 1.026567708, 2024-09-13T14:37:48.315 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0265896670000001, 2024-09-13T14:37:48.315 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:48.315 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "waiting for peered", 2024-09-13T14:37:48.315 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:37:48.315 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:37:48.315 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:37:48.315 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113976 2024-09-13T14:37:48.316 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.316 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:48.316 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.316 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:48.316 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.285996+0000", 2024-09-13T14:37:48.316 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:48.316 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.316 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.316 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:48.316 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.285996+0000", 2024-09-13T14:37:48.316 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:48.316 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.317 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.317 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:48.317 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.285994+0000", 2024-09-13T14:37:48.317 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:37:48.317 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.317 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.317 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:48.317 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.286006+0000", 2024-09-13T14:37:48.317 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1901e-05 2024-09-13T14:37:48.317 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.317 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.318 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:48.318 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.286019+0000", 2024-09-13T14:37:48.318 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3175e-05 2024-09-13T14:37:48.318 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.318 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.318 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:48.318 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.286026+0000", 2024-09-13T14:37:48.318 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8240000000000002e-06 2024-09-13T14:37:48.318 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.318 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.318 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:48.318 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.286083+0000", 2024-09-13T14:37:48.318 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7114999999999998e-05 2024-09-13T14:37:48.319 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.319 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.319 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:37:48.319 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.286093+0000", 2024-09-13T14:37:48.319 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4099999999999997e-06 2024-09-13T14:37:48.319 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.319 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.319 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:48.319 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.142541+0000", 2024-09-13T14:37:48.319 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.85644865699999995 2024-09-13T14:37:48.319 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.319 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.319 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:37:48.320 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.142554+0000", 2024-09-13T14:37:48.320 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2408999999999999e-05 2024-09-13T14:37:48.320 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:48.320 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:48.320 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:48.320 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.320 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.320 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDPGPush(7.6 239/222 [PushOp(7:6b509e1e:::benchmark_data_smithi102_227395_object1182:head, version: 209'73, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:6b509e1e:::benchmark_data_smithi102_227395_object1182:head@209'73, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T14:37:48.320 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:47.290709+0000", 2024-09-13T14:37:48.320 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 1.0218547929999999, 2024-09-13T14:37:48.320 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0219409239999999, 2024-09-13T14:37:48.321 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:48.321 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "queued for pg", 2024-09-13T14:37:48.321 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:48.321 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.321 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:48.321 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.290709+0000", 2024-09-13T14:37:48.321 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:48.321 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.321 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.321 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:48.321 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.290709+0000", 2024-09-13T14:37:48.321 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:48.321 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.322 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.322 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:48.322 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.290707+0000", 2024-09-13T14:37:48.322 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:37:48.322 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.322 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.322 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:48.322 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.290762+0000", 2024-09-13T14:37:48.322 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4667999999999999e-05 2024-09-13T14:37:48.322 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.322 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.322 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:48.323 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.290783+0000", 2024-09-13T14:37:48.323 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0361e-05 2024-09-13T14:37:48.323 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.323 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.323 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:48.323 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.290799+0000", 2024-09-13T14:37:48.323 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6869000000000001e-05 2024-09-13T14:37:48.323 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:48.323 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:48.323 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:48.323 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.323 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.323 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113947 7.e 7:72d65fe5:::benchmark_data_smithi102_227395_object1132:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-13T14:37:48.324 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:45.881740+0000", 2024-09-13T14:37:48.324 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 2.4308231679999999, 2024-09-13T14:37:48.324 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.430937702, 2024-09-13T14:37:48.324 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:48.324 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "waiting for rw locks", 2024-09-13T14:37:48.324 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:37:48.324 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:37:48.324 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:37:48.324 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113947 2024-09-13T14:37:48.324 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.324 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:48.324 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.324 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:48.324 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.881740+0000", 2024-09-13T14:37:48.325 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:48.325 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.325 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.325 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:48.325 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.881740+0000", 2024-09-13T14:37:48.325 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:48.325 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.325 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.325 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:48.325 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.881739+0000", 2024-09-13T14:37:48.325 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:37:48.325 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.325 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.326 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:48.326 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.881746+0000", 2024-09-13T14:37:48.326 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6309999999999997e-06 2024-09-13T14:37:48.326 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.326 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.326 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:48.326 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.881755+0000", 2024-09-13T14:37:48.326 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.0189999999999995e-06 2024-09-13T14:37:48.326 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.326 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.326 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:48.326 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.881760+0000", 2024-09-13T14:37:48.326 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8489999999999997e-06 2024-09-13T14:37:48.326 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.326 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.327 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:48.327 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.899950+0000", 2024-09-13T14:37:48.327 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018189839999999999 2024-09-13T14:37:48.327 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.327 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.327 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:37:48.327 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.917306+0000", 2024-09-13T14:37:48.327 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017356068999999998 2024-09-13T14:37:48.327 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.327 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.327 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:48.327 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.076013+0000", 2024-09-13T14:37:48.327 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.158707658 2024-09-13T14:37:48.327 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.328 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.328 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:37:48.328 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.092645+0000", 2024-09-13T14:37:48.328 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016631877999999999 2024-09-13T14:37:48.328 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:48.328 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:48.328 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:48.328 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.328 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.328 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113986 7.e 7:7dc5417c:::benchmark_data_smithi102_227395_object1171:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-13T14:37:48.328 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:47.635035+0000", 2024-09-13T14:37:48.329 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.67752841600000002, 2024-09-13T14:37:48.329 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.67768630500000004, 2024-09-13T14:37:48.329 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:48.329 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "waiting for rw locks", 2024-09-13T14:37:48.329 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:37:48.329 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:37:48.329 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:37:48.330 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113986 2024-09-13T14:37:48.330 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.330 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:48.330 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.330 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:48.330 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.635035+0000", 2024-09-13T14:37:48.330 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:48.330 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.330 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.330 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:48.330 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.635035+0000", 2024-09-13T14:37:48.330 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:48.330 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.331 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.331 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:48.331 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.635034+0000", 2024-09-13T14:37:48.331 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:37:48.331 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.331 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.331 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:48.331 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.635041+0000", 2024-09-13T14:37:48.331 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2749999999999998e-06 2024-09-13T14:37:48.331 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.331 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.331 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:48.331 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.635053+0000", 2024-09-13T14:37:48.332 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2386e-05 2024-09-13T14:37:48.332 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.332 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.332 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:48.332 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.635063+0000", 2024-09-13T14:37:48.332 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4050000000000006e-06 2024-09-13T14:37:48.332 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.332 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.332 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:48.332 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.685520+0000", 2024-09-13T14:37:48.332 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.050456986000000002 2024-09-13T14:37:48.332 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.332 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.332 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:37:48.333 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.698223+0000", 2024-09-13T14:37:48.333 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012702854 2024-09-13T14:37:48.333 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.333 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.333 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:48.333 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.813155+0000", 2024-09-13T14:37:48.333 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.114932643 2024-09-13T14:37:48.333 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.333 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.333 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:37:48.333 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.822588+0000", 2024-09-13T14:37:48.333 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0094329040000000006 2024-09-13T14:37:48.335 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:48.335 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:48.335 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:48.335 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.335 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.335 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113950 7.e 7:7bc9cd2e:::benchmark_data_smithi102_227395_object1135:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-13T14:37:48.335 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:46.127745+0000", 2024-09-13T14:37:48.335 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 2.184818173, 2024-09-13T14:37:48.335 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1850157810000002, 2024-09-13T14:37:48.335 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:48.335 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "waiting for rw locks", 2024-09-13T14:37:48.336 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:37:48.336 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:37:48.336 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:37:48.336 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113950 2024-09-13T14:37:48.336 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.336 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:48.336 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.336 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:48.336 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.127745+0000", 2024-09-13T14:37:48.336 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:48.336 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.336 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.336 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:48.336 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.127745+0000", 2024-09-13T14:37:48.337 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:48.337 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.337 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.337 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:48.337 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.127744+0000", 2024-09-13T14:37:48.337 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:37:48.337 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.337 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.337 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:48.337 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.127751+0000", 2024-09-13T14:37:48.337 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8630000000000002e-06 2024-09-13T14:37:48.338 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.338 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.338 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:48.338 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.127763+0000", 2024-09-13T14:37:48.338 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1819e-05 2024-09-13T14:37:48.338 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.338 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.338 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:48.338 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.127768+0000", 2024-09-13T14:37:48.338 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5119999999999999e-06 2024-09-13T14:37:48.338 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.338 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.338 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:48.339 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.467726+0000", 2024-09-13T14:37:48.339 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.339957642 2024-09-13T14:37:48.339 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.339 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.339 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:37:48.339 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.487879+0000", 2024-09-13T14:37:48.339 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.020152837 2024-09-13T14:37:48.339 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.339 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.339 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:48.339 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.651549+0000", 2024-09-13T14:37:48.339 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.16367005200000001 2024-09-13T14:37:48.339 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.339 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.339 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:37:48.340 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.670285+0000", 2024-09-13T14:37:48.340 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018735781999999999 2024-09-13T14:37:48.340 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:48.340 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:48.340 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:48.340 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.340 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.340 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113987 7.e 7:7d70cc81:::benchmark_data_smithi102_227395_object1172:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-13T14:37:48.340 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:47.663783+0000", 2024-09-13T14:37:48.340 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.64878095099999999, 2024-09-13T14:37:48.340 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.64901811300000001, 2024-09-13T14:37:48.340 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:48.341 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "waiting for rw locks", 2024-09-13T14:37:48.341 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:37:48.341 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:37:48.341 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:37:48.341 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113987 2024-09-13T14:37:48.341 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.341 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:48.341 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.341 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:48.341 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.663783+0000", 2024-09-13T14:37:48.341 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:48.341 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.341 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.342 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:48.342 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.663783+0000", 2024-09-13T14:37:48.342 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:48.342 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.342 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.342 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:48.342 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.663781+0000", 2024-09-13T14:37:48.342 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:37:48.342 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.342 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.342 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:48.342 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.663790+0000", 2024-09-13T14:37:48.342 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.4849999999999992e-06 2024-09-13T14:37:48.343 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.343 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.343 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:48.343 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.663802+0000", 2024-09-13T14:37:48.343 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2849e-05 2024-09-13T14:37:48.343 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.343 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.343 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:48.343 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.663811+0000", 2024-09-13T14:37:48.343 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5099999999999998e-06 2024-09-13T14:37:48.343 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.343 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.343 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:48.343 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.699497+0000", 2024-09-13T14:37:48.344 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.035686155999999997 2024-09-13T14:37:48.344 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.344 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.344 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:37:48.344 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.709342+0000", 2024-09-13T14:37:48.344 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0098446760000000001 2024-09-13T14:37:48.344 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.344 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.344 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:48.344 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.858875+0000", 2024-09-13T14:37:48.344 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.149532793 2024-09-13T14:37:48.344 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.344 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.345 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:37:48.345 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.868437+0000", 2024-09-13T14:37:48.345 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0095622699999999995 2024-09-13T14:37:48.345 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:48.345 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:48.345 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:48.345 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.345 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.345 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113723 7.e 7:765a5df7:::benchmark_data_smithi102_227395_object908:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-13T14:37:48.345 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:41.735065+0000", 2024-09-13T14:37:48.345 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 6.5774986479999997, 2024-09-13T14:37:48.345 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5777784439999998, 2024-09-13T14:37:48.346 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:48.346 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:37:48.346 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:37:48.346 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:37:48.346 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:37:48.346 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113723 2024-09-13T14:37:48.346 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.346 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:48.346 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.346 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:48.346 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.735065+0000", 2024-09-13T14:37:48.346 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:48.347 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.347 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.347 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:48.347 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.735065+0000", 2024-09-13T14:37:48.347 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:48.347 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.347 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.347 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:48.347 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.735064+0000", 2024-09-13T14:37:48.347 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:37:48.347 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.347 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.347 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:48.347 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.735071+0000", 2024-09-13T14:37:48.347 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2289999999999997e-06 2024-09-13T14:37:48.348 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.348 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.348 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:48.348 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.735085+0000", 2024-09-13T14:37:48.348 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4436e-05 2024-09-13T14:37:48.348 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.348 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.348 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:48.348 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.735092+0000", 2024-09-13T14:37:48.348 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.216e-06 2024-09-13T14:37:48.348 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.348 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.349 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:48.349 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.016651+0000", 2024-09-13T14:37:48.349 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.28155898499999998 2024-09-13T14:37:48.349 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.349 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.349 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:37:48.349 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.027127+0000", 2024-09-13T14:37:48.349 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010475241999999999 2024-09-13T14:37:48.349 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.349 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.349 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:48.349 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.819782+0000", 2024-09-13T14:37:48.349 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7926552459999998 2024-09-13T14:37:48.350 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.350 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.350 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:37:48.350 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.838097+0000", 2024-09-13T14:37:48.350 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018315084999999998 2024-09-13T14:37:48.350 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.350 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.350 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:48.350 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.103166+0000", 2024-09-13T14:37:48.350 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2650688400000001 2024-09-13T14:37:48.350 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.350 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.351 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:37:48.351 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.113423+0000", 2024-09-13T14:37:48.351 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010256919 2024-09-13T14:37:48.351 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.351 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.351 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:37:48.351 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.159715+0000", 2024-09-13T14:37:48.351 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.046292424999999998 2024-09-13T14:37:48.351 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.351 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.351 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:37:48.351 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.242694+0000", 2024-09-13T14:37:48.351 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.082978417999999998 2024-09-13T14:37:48.352 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.352 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.352 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:48.352 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.310555+0000", 2024-09-13T14:37:48.352 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.067861006000000001 2024-09-13T14:37:48.352 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:48.352 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:48.352 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:48.352 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.352 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.352 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113957 7.e 7:7d24aebc:::benchmark_data_smithi102_227395_object1142:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-13T14:37:48.352 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:46.417450+0000", 2024-09-13T14:37:48.352 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 1.8951133439999999, 2024-09-13T14:37:48.353 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.89544655, 2024-09-13T14:37:48.353 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:48.353 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "waiting for rw locks", 2024-09-13T14:37:48.353 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:37:48.353 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:37:48.353 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:37:48.353 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113957 2024-09-13T14:37:48.353 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.353 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:48.353 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.353 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:48.353 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.417450+0000", 2024-09-13T14:37:48.354 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:48.354 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.354 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.354 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:48.354 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.417450+0000", 2024-09-13T14:37:48.354 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:48.354 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.354 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.354 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:48.354 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.417449+0000", 2024-09-13T14:37:48.354 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:37:48.354 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.354 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.354 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:48.355 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.417456+0000", 2024-09-13T14:37:48.355 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1380000000000001e-06 2024-09-13T14:37:48.355 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.355 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.355 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:48.355 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.417469+0000", 2024-09-13T14:37:48.355 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3412e-05 2024-09-13T14:37:48.355 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.355 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.355 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:48.355 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.417477+0000", 2024-09-13T14:37:48.355 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3490000000000003e-06 2024-09-13T14:37:48.355 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.355 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.356 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:48.356 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.490197+0000", 2024-09-13T14:37:48.356 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.072720160000000006 2024-09-13T14:37:48.356 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.356 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.356 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:37:48.356 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.511607+0000", 2024-09-13T14:37:48.356 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.021409804000000001 2024-09-13T14:37:48.356 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.356 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.356 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:48.356 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.729832+0000", 2024-09-13T14:37:48.357 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.21822541300000001 2024-09-13T14:37:48.358 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.358 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.358 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:37:48.358 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.745044+0000", 2024-09-13T14:37:48.358 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.01521162 2024-09-13T14:37:48.358 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:48.358 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:48.358 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:48.358 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.358 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.358 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDPGPush(7.6 240/222 [PushOp(7:6ae589aa:::benchmark_data_smithi102_227395_object1210:head, version: 209'74, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:6ae589aa:::benchmark_data_smithi102_227395_object1210:head@209'74, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T14:37:48.358 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:48.146505+0000", 2024-09-13T14:37:48.359 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.166058282, 2024-09-13T14:37:48.359 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.16645562699999999, 2024-09-13T14:37:48.359 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:48.359 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "queued for pg", 2024-09-13T14:37:48.359 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:48.359 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.359 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:48.359 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.146505+0000", 2024-09-13T14:37:48.359 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:48.359 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.359 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.359 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:48.359 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.146505+0000", 2024-09-13T14:37:48.360 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:48.360 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.360 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.360 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:48.360 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.146505+0000", 2024-09-13T14:37:48.360 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:37:48.360 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.360 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.360 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:48.360 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.146543+0000", 2024-09-13T14:37:48.360 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8121000000000003e-05 2024-09-13T14:37:48.360 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.360 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.361 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:48.361 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.146559+0000", 2024-09-13T14:37:48.361 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6118999999999999e-05 2024-09-13T14:37:48.361 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.361 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.361 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:48.361 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.146580+0000", 2024-09-13T14:37:48.361 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0786000000000001e-05 2024-09-13T14:37:48.361 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:48.361 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:48.361 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:48.361 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.362 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.362 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop_reply(client.35379.0:113723 7.e e240/222)", 2024-09-13T14:37:48.362 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:48.181764+0000", 2024-09-13T14:37:48.362 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.13079992100000001, 2024-09-13T14:37:48.362 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.13122113199999999, 2024-09-13T14:37:48.362 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:48.362 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-13T14:37:48.362 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:48.362 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.362 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:48.362 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.181764+0000", 2024-09-13T14:37:48.362 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:48.362 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.362 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.363 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:48.363 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.181764+0000", 2024-09-13T14:37:48.363 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:48.363 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.363 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.363 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:48.363 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.181762+0000", 2024-09-13T14:37:48.363 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:37:48.363 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.363 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.363 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:48.363 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.181772+0000", 2024-09-13T14:37:48.364 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.3579999999999992e-06 2024-09-13T14:37:48.364 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.364 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.364 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:48.364 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.181789+0000", 2024-09-13T14:37:48.364 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7598e-05 2024-09-13T14:37:48.364 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.364 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.364 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:48.364 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.181799+0000", 2024-09-13T14:37:48.364 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0010000000000001e-05 2024-09-13T14:37:48.365 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.365 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.365 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:48.365 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.312468+0000", 2024-09-13T14:37:48.365 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.13066855599999999 2024-09-13T14:37:48.365 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:48.365 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:48.365 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:48.365 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.365 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.365 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113992 7.0 7.a8fefc80 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-13T14:37:48.365 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:48.220617+0000", 2024-09-13T14:37:48.365 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.091946331000000006, 2024-09-13T14:37:48.366 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.092394034999999999, 2024-09-13T14:37:48.366 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:48.366 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "waiting for peered", 2024-09-13T14:37:48.366 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:37:48.366 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:37:48.366 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:37:48.366 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113992 2024-09-13T14:37:48.366 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.366 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:48.366 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.366 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:48.366 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.220617+0000", 2024-09-13T14:37:48.366 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:48.367 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.367 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.367 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:48.367 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.220617+0000", 2024-09-13T14:37:48.367 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:48.367 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.367 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.367 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:48.367 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.220615+0000", 2024-09-13T14:37:48.367 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:37:48.367 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.367 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.367 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:48.368 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.220625+0000", 2024-09-13T14:37:48.368 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.5729999999999997e-06 2024-09-13T14:37:48.368 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.368 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.368 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:48.368 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.220643+0000", 2024-09-13T14:37:48.368 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8295999999999999e-05 2024-09-13T14:37:48.368 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.368 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.368 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:48.368 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.220654+0000", 2024-09-13T14:37:48.368 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0789e-05 2024-09-13T14:37:48.369 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.369 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.369 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:48.369 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.220721+0000", 2024-09-13T14:37:48.369 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6859999999999993e-05 2024-09-13T14:37:48.369 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.369 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.369 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:37:48.369 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.220733+0000", 2024-09-13T14:37:48.369 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2447e-05 2024-09-13T14:37:48.369 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:48.369 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:48.369 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:48.370 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.370 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.370 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113948 7.e 7:70c743c6:::benchmark_data_smithi102_227395_object1133:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-13T14:37:48.370 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:45.960644+0000", 2024-09-13T14:37:48.370 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 2.351919927, 2024-09-13T14:37:48.370 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.352406738, 2024-09-13T14:37:48.370 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:48.370 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "waiting for rw locks", 2024-09-13T14:37:48.370 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:37:48.370 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:37:48.370 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:37:48.370 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113948 2024-09-13T14:37:48.370 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.371 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:48.371 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.371 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:48.371 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.960644+0000", 2024-09-13T14:37:48.371 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:48.371 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.371 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.371 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:48.371 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.960644+0000", 2024-09-13T14:37:48.371 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:48.371 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.371 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.371 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:48.372 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.960642+0000", 2024-09-13T14:37:48.372 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:37:48.372 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.372 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.372 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:48.372 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.960649+0000", 2024-09-13T14:37:48.372 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3790000000000003e-06 2024-09-13T14:37:48.372 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.372 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.372 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:48.372 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.960658+0000", 2024-09-13T14:37:48.372 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.0899999999999994e-06 2024-09-13T14:37:48.372 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.373 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.373 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:48.373 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.960665+0000", 2024-09-13T14:37:48.373 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6000000000000003e-06 2024-09-13T14:37:48.373 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.373 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.373 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:48.373 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.439612+0000", 2024-09-13T14:37:48.373 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.47894750699999999 2024-09-13T14:37:48.373 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.373 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.373 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:37:48.373 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.465491+0000", 2024-09-13T14:37:48.374 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.025878531999999999 2024-09-13T14:37:48.374 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.374 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.374 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:48.374 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.572697+0000", 2024-09-13T14:37:48.374 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.10720668999999999 2024-09-13T14:37:48.374 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:48.374 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:48.374 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:37:48.374 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.589981+0000", 2024-09-13T14:37:48.374 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017284119000000001 2024-09-13T14:37:48.374 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:48.374 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:48.374 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:48.375 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:48.375 INFO:teuthology.orchestra.run.smithi195.stdout: ], 2024-09-13T14:37:48.375 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 12 2024-09-13T14:37:48.375 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:37:48.408 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:48.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:48 smithi102 ceph-mon[110183]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:37:48.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:48 smithi102 ceph-mon[110183]: osd.6 [v2:172.21.15.195:6816/3802194762,v1:172.21.15.195:6817/3802194762] boot 2024-09-13T14:37:48.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:48 smithi102 ceph-mon[110183]: osdmap e239: 8 total, 8 up, 6 in 2024-09-13T14:37:48.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:48 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-13T14:37:48.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:48 smithi102 ceph-mon[110183]: osdmap e240: 8 total, 8 up, 6 in 2024-09-13T14:37:48.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:48 smithi102 ceph-mon[115269]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:37:48.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:48 smithi102 ceph-mon[115269]: osd.6 [v2:172.21.15.195:6816/3802194762,v1:172.21.15.195:6817/3802194762] boot 2024-09-13T14:37:48.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:48 smithi102 ceph-mon[115269]: osdmap e239: 8 total, 8 up, 6 in 2024-09-13T14:37:48.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:48 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-13T14:37:48.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:48 smithi102 ceph-mon[115269]: osdmap e240: 8 total, 8 up, 6 in 2024-09-13T14:37:48.432 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:48.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:48.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:48.534 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:48.561 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:48.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:48.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:48.664 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:48.689 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:48.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:48.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:48.791 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:48.816 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:48.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:48.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:48.880 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:37:48.918 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:48.944 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:48.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:48.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:49.046 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:49.067 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:49.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:49.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:49.145 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T14:37:49.169 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:49.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:49.195 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:49.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:49.296 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:49.318 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:49.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:49.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:49.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:49 smithi102 ceph-mon[110183]: pgmap v763: 89 pgs: 1 active+recovering+undersized+remapped, 13 remapped+peering, 12 peering, 26 active+clean+remapped, 2 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 7 active+undersized, 2 active+undersized+degraded, 25 active+clean; 6.8 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 61 op/s; 7103/335193 objects degraded (2.119%); 49875/335193 objects misplaced (14.879%); 63 MiB/s, 1.01k objects/s recovering 2024-09-13T14:37:49.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:49 smithi102 ceph-mon[110183]: 7.9 starting backfill to osd.6 from (0'0,0'0] MAX to 238'7149 2024-09-13T14:37:49.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:49 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 7103/335193 objects degraded (2.119%), 3 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-13T14:37:49.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:49 smithi102 ceph-mon[110183]: osdmap e241: 8 total, 8 up, 6 in 2024-09-13T14:37:49.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:49 smithi102 ceph-mon[115269]: pgmap v763: 89 pgs: 1 active+recovering+undersized+remapped, 13 remapped+peering, 12 peering, 26 active+clean+remapped, 2 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 7 active+undersized, 2 active+undersized+degraded, 25 active+clean; 6.8 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 61 op/s; 7103/335193 objects degraded (2.119%); 49875/335193 objects misplaced (14.879%); 63 MiB/s, 1.01k objects/s recovering 2024-09-13T14:37:49.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:49 smithi102 ceph-mon[115269]: 7.9 starting backfill to osd.6 from (0'0,0'0] MAX to 238'7149 2024-09-13T14:37:49.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:49 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 7103/335193 objects degraded (2.119%), 3 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-13T14:37:49.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:49 smithi102 ceph-mon[115269]: osdmap e241: 8 total, 8 up, 6 in 2024-09-13T14:37:49.420 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:49.442 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:49.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:49.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:49.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:49 smithi195 ceph-mon[104262]: pgmap v763: 89 pgs: 1 active+recovering+undersized+remapped, 13 remapped+peering, 12 peering, 26 active+clean+remapped, 2 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 7 active+undersized, 2 active+undersized+degraded, 25 active+clean; 6.8 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 61 op/s; 7103/335193 objects degraded (2.119%); 49875/335193 objects misplaced (14.879%); 63 MiB/s, 1.01k objects/s recovering 2024-09-13T14:37:49.501 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:49 smithi195 ceph-mon[104262]: 7.9 starting backfill to osd.6 from (0'0,0'0] MAX to 238'7149 2024-09-13T14:37:49.501 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:49 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 7103/335193 objects degraded (2.119%), 3 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-13T14:37:49.501 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:49 smithi195 ceph-mon[104262]: osdmap e241: 8 total, 8 up, 6 in 2024-09-13T14:37:49.532 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:37:49.544 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:49.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:49.567 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:49.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:49.668 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:49.693 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:49.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:49.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:49.795 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:49.816 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:49.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:49.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:49.918 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:49.941 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:49.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:49.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:50.043 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:50.065 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:50.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:50.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:50.167 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:50.192 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:50.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:50.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:50.294 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:50.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:50.317 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:50.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:50.395 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:50 smithi102 ceph-mon[110183]: 7.3 starting backfill to osd.6 from (0'0,210'2607] MAX to 238'7279 2024-09-13T14:37:50.396 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:50 smithi102 ceph-mon[115269]: 7.3 starting backfill to osd.6 from (0'0,210'2607] MAX to 238'7279 2024-09-13T14:37:50.418 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:50.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:50.442 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:50.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:50.455 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T14:37:50.477 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:50 smithi195 ceph-mon[104262]: 7.3 starting backfill to osd.6 from (0'0,210'2607] MAX to 238'7279 2024-09-13T14:37:50.544 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:50.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:50.567 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:50.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:50.668 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:50.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:50.690 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:50.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:50.791 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:50.797 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:37:50.798 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:37:50.812 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:50.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:50.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:50.814 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:37:50.814 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:37:50.827 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:37:50.831 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:37:50.831 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:37:50.854 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:37:50.854 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:37:50.871 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:37:50.872 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:37:50.888 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:37:50.889 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:37:50.906 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:37:50.906 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_max_backfills = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-09-13T14:37:50.914 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:50.925 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:37:50.929 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:37:50.941 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:50.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:50.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:51.044 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:51.065 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:51.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:51.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:51.167 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:51.191 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:51.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:51.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:51.292 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:51.314 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:51.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:51.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:51.416 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:51.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:51 smithi102 ceph-mon[115269]: pgmap v766: 89 pgs: 1 active+recovering+undersized+remapped, 22 remapped+peering, 21 peering, 17 active+clean+remapped, 2 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 25 active+clean; 6.8 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 50 op/s; 1/335100 objects degraded (0.000%); 35862/335100 objects misplaced (10.702%); 44 MiB/s, 704 objects/s recovering 2024-09-13T14:37:51.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:51 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2783030702' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:37:51.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:51 smithi102 ceph-mon[110183]: pgmap v766: 89 pgs: 1 active+recovering+undersized+remapped, 22 remapped+peering, 21 peering, 17 active+clean+remapped, 2 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 25 active+clean; 6.8 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 50 op/s; 1/335100 objects degraded (0.000%); 35862/335100 objects misplaced (10.702%); 44 MiB/s, 704 objects/s recovering 2024-09-13T14:37:51.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:51 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2783030702' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:37:51.438 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:51.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:51.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:51.491 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:51 smithi195 ceph-mon[104262]: pgmap v766: 89 pgs: 1 active+recovering+undersized+remapped, 22 remapped+peering, 21 peering, 17 active+clean+remapped, 2 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 25 active+clean; 6.8 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 50 op/s; 1/335100 objects degraded (0.000%); 35862/335100 objects misplaced (10.702%); 44 MiB/s, 704 objects/s recovering 2024-09-13T14:37:51.492 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:51 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2783030702' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:37:51.540 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:51.565 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:51.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:51.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:51.667 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:51.693 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:51.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:51.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:51.755 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:37:51.794 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:51.817 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:51.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:51.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:51.919 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:51.940 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:51.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:51.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:52.041 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:52.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:52.064 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:52.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:52.165 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:52.187 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:52.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:52.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:52.288 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:52.310 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:52.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:52.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:52.412 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:52.433 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:52.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:52.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:52.535 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:52.557 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:52.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:52.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:52.663 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:52.686 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:52.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:52.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:52.788 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:52.810 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:52.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:52.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:52.913 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:52.935 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:52.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:52.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:53.037 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:53.039 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:37:53.059 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:53.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:53.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:53.160 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:53.183 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:53.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:53.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:53.284 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:53.306 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:53.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:53.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:53.318 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:53 smithi195 ceph-mon[104262]: pgmap v767: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 22 remapped+peering, 21 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 40 active+clean; 6.8 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 39 op/s; 8923/334929 objects degraded (2.664%); 22018/334929 objects misplaced (6.574%); 33 MiB/s, 530 objects/s recovering 2024-09-13T14:37:53.370 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:53 smithi102 ceph-mon[110183]: pgmap v767: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 22 remapped+peering, 21 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 40 active+clean; 6.8 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 39 op/s; 8923/334929 objects degraded (2.664%); 22018/334929 objects misplaced (6.574%); 33 MiB/s, 530 objects/s recovering 2024-09-13T14:37:53.370 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:53 smithi102 ceph-mon[115269]: pgmap v767: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 22 remapped+peering, 21 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 40 active+clean; 6.8 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 39 op/s; 8923/334929 objects degraded (2.664%); 22018/334929 objects misplaced (6.574%); 33 MiB/s, 530 objects/s recovering 2024-09-13T14:37:53.408 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:53.431 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:53.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:53.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:53.533 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:53.560 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:53.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:53.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:53.662 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:53.685 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:53.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:53.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:53.746 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:37:53.790 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:53.811 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:53.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:53.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:53.913 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:53.935 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:53.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:53.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:53.982 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:37:54.037 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:54.058 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:54.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:54.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:54.150 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is unset 2024-09-13T14:37:54.159 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:54.187 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:54.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:54.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:54.219 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:37:54.219 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:37:54.289 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:54.311 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:54.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:54.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:54.413 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:54.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:54 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 8923/334929 objects degraded (2.664%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:37:54.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:54 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2811506502' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:37:54.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:54 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 8923/334929 objects degraded (2.664%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:37:54.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:54 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2811506502' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:37:54.435 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:54.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:54.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:54.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:54 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 8923/334929 objects degraded (2.664%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:37:54.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:54 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2811506502' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:37:54.537 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:54.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:54.559 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:54.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:54.660 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:54.691 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:54.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:54.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:54.793 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:54.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:54.818 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:54.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:54.891 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T14:37:54.919 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:54.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:54.941 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:54.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:55.043 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:55.066 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:55.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:55.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:55.150 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:55 smithi195 ceph-mon[104262]: pgmap v768: 89 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 73 active+clean; 6.8 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 96 op/s; 51078/333738 objects degraded (15.305%); 28148/333738 objects misplaced (8.434%); 36 MiB/s, 578 objects/s recovering 2024-09-13T14:37:55.151 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:55 smithi195 ceph-mon[104262]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:37:55.169 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:55.191 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:55.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:55.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:55.293 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:55.314 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:55.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:55.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:55.416 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:55.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:55 smithi102 ceph-mon[110183]: pgmap v768: 89 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 73 active+clean; 6.8 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 96 op/s; 51078/333738 objects degraded (15.305%); 28148/333738 objects misplaced (8.434%); 36 MiB/s, 578 objects/s recovering 2024-09-13T14:37:55.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:55 smithi102 ceph-mon[110183]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:37:55.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:55 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2811506502' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:37:55.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:55 smithi102 ceph-mon[110183]: osdmap e242: 8 total, 8 up, 6 in 2024-09-13T14:37:55.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:55 smithi102 ceph-mon[115269]: pgmap v768: 89 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 73 active+clean; 6.8 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 96 op/s; 51078/333738 objects degraded (15.305%); 28148/333738 objects misplaced (8.434%); 36 MiB/s, 578 objects/s recovering 2024-09-13T14:37:55.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:55 smithi102 ceph-mon[115269]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:37:55.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:55 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2811506502' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:37:55.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:55 smithi102 ceph-mon[115269]: osdmap e242: 8 total, 8 up, 6 in 2024-09-13T14:37:55.444 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:55.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:55.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:55.484 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:55 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2811506502' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:37:55.485 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:55 smithi195 ceph-mon[104262]: osdmap e242: 8 total, 8 up, 6 in 2024-09-13T14:37:55.546 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:55.572 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:55.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:55.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:55.673 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:55.695 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:55.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:55.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:55.797 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:55.826 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:55.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:55.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:55.928 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:55.950 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:55.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:55.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:56.052 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:56.060 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:37:56.061 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:37:56.075 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:56.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:56.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:56.177 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:56.201 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:56.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:56.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:56.303 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:56.325 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:56.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:56.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:56.427 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:56.434 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:37:56.434 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:37:56.451 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:37:56.451 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:37:56.451 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:56.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:56.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:56.467 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:37:56.467 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:37:56.467 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:37:56.467 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:37:56.467 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:56 smithi102 ceph-mon[110183]: pgmap v770: 89 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 73 active+clean; 6.8 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 61 op/s; 51078/333738 objects degraded (15.305%); 28148/333738 objects misplaced (8.434%); 7.3 MiB/s, 116 objects/s recovering 2024-09-13T14:37:56.468 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:56 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/246316716' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:37:56.468 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:37:56.468 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:37:56.468 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:56 smithi102 ceph-mon[115269]: pgmap v770: 89 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 73 active+clean; 6.8 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 61 op/s; 51078/333738 objects degraded (15.305%); 28148/333738 objects misplaced (8.434%); 7.3 MiB/s, 116 objects/s recovering 2024-09-13T14:37:56.468 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:56 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/246316716' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:37:56.468 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:37:56 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:37:56] "GET /metrics HTTP/1.1" 200 36574 "" "Prometheus/2.51.0" 2024-09-13T14:37:56.483 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:37:56.483 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:37:56.500 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:37:56.500 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:37:56.515 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:37:56.516 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:37:56.533 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:37:56.533 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:37:56.548 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:37:56.549 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:37:56.553 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:56.577 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:56.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:56.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:56.679 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:56.700 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:56.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:56.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:56.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:37:56.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:37:56.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:56 smithi195 ceph-mon[104262]: pgmap v770: 89 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 73 active+clean; 6.8 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 61 op/s; 51078/333738 objects degraded (15.305%); 28148/333738 objects misplaced (8.434%); 7.3 MiB/s, 116 objects/s recovering 2024-09-13T14:37:56.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:56 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/246316716' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:37:56.802 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:56.824 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:56.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:56.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:56.891 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:37:56.915 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-13T14:37:56.926 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:56.949 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:56.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:56.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:57.039 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:37:57.051 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:57.105 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:57.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:57.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:57.207 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:57.228 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:57.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:57.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:57.330 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:57.352 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:57.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:57.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:57.359 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:37:57.454 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:57.475 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:57.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:57.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:57.576 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:57.597 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:57.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:57.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:57.699 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:57.722 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:57.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:57.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:57.823 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:57.846 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:57.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:57.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:57.948 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:57.969 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:57.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:57.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:58.071 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:58.091 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:58.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:58.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:58.193 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:58.216 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:58.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:58.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:58.318 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:58.340 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:58.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:58.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:58.428 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:37:58.442 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:58.471 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:58.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:58.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:58.573 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:58.595 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:58.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:58.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:58.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:58 smithi102 ceph-mon[110183]: pgmap v771: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 73 active+clean; 6.8 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 109 op/s; 51045/332364 objects degraded (15.358%); 28041/332364 objects misplaced (8.437%); 7.0 MiB/s, 111 objects/s recovering 2024-09-13T14:37:58.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:58 smithi102 ceph-mon[115269]: pgmap v771: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 73 active+clean; 6.8 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 109 op/s; 51045/332364 objects degraded (15.358%); 28041/332364 objects misplaced (8.437%); 7.0 MiB/s, 111 objects/s recovering 2024-09-13T14:37:58.697 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:58.726 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:58.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:58.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:58.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:58 smithi195 ceph-mon[104262]: pgmap v771: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 73 active+clean; 6.8 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 109 op/s; 51045/332364 objects degraded (15.358%); 28041/332364 objects misplaced (8.437%); 7.0 MiB/s, 111 objects/s recovering 2024-09-13T14:37:58.828 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:58.847 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:37:58.847 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-09-13T14:37:58.847 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-09-13T14:37:58.847 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-13T14:37:58.847 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.847 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:112933 7.e 7:71539828:::benchmark_data_smithi102_227395_object118:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-09-13T14:37:58.847 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:28.169308+0000", 2024-09-13T14:37:58.847 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 30.673650518999999, 2024-09-13T14:37:58.848 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.67251101, 2024-09-13T14:37:58.848 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:58.848 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:58.848 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:37:58.848 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:37:58.848 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:37:58.848 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 112933 2024-09-13T14:37:58.848 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.848 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:58.848 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.848 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:58.848 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.169308+0000", 2024-09-13T14:37:58.848 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:58.849 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.849 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.849 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:58.849 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.169308+0000", 2024-09-13T14:37:58.849 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:58.849 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.849 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.849 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:58.849 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.169306+0000", 2024-09-13T14:37:58.849 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:37:58.849 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.849 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.849 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:58.850 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.169314+0000", 2024-09-13T14:37:58.850 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5399999999999998e-06 2024-09-13T14:37:58.850 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.850 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.850 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:58.850 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.169327+0000", 2024-09-13T14:37:58.850 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3589e-05 2024-09-13T14:37:58.850 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.850 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.850 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:58.850 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.169336+0000", 2024-09-13T14:37:58.850 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1599999999999998e-06 2024-09-13T14:37:58.850 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.851 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.851 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.851 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.036088+0000", 2024-09-13T14:37:58.851 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.86675195199999999 2024-09-13T14:37:58.851 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.851 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.851 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:37:58.851 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.056928+0000", 2024-09-13T14:37:58.851 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.020840121 2024-09-13T14:37:58.851 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.851 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.851 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.851 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.228620+0000", 2024-09-13T14:37:58.852 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.17169189700000001 2024-09-13T14:37:58.852 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.852 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.852 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:37:58.852 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.256794+0000", 2024-09-13T14:37:58.852 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.028174637999999998 2024-09-13T14:37:58.852 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.852 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.852 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.852 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.612105+0000", 2024-09-13T14:37:58.852 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3553110849999999 2024-09-13T14:37:58.852 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.852 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.853 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:37:58.853 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.625533+0000", 2024-09-13T14:37:58.853 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013427308000000001 2024-09-13T14:37:58.853 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.853 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.853 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:37:58.853 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.650011+0000", 2024-09-13T14:37:58.853 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024478033999999999 2024-09-13T14:37:58.853 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.853 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.853 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:58.853 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.779644+0000", 2024-09-13T14:37:58.853 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.12963343899999999 2024-09-13T14:37:58.854 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.854 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.854 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:37:58.854 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.781386+0000", 2024-09-13T14:37:58.854 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0017419709999999999 2024-09-13T14:37:58.854 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.854 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.854 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:58.854 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.786230+0000", 2024-09-13T14:37:58.854 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0048441839999999996 2024-09-13T14:37:58.854 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.854 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.854 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:37:58.855 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.792473+0000", 2024-09-13T14:37:58.855 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0062430649999999999 2024-09-13T14:37:58.855 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.855 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.855 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:37:58.855 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.841819+0000", 2024-09-13T14:37:58.855 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.049345554999999999 2024-09-13T14:37:58.855 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:58.855 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:58.855 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:58.855 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.855 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.855 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:112953 7.e 7:72a6485e:::benchmark_data_smithi102_227395_object138:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-09-13T14:37:58.856 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:28.219817+0000", 2024-09-13T14:37:58.856 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 30.623141503999999, 2024-09-13T14:37:58.856 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.688959026, 2024-09-13T14:37:58.856 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:58.856 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:58.856 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:37:58.856 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:37:58.856 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:37:58.856 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 112953 2024-09-13T14:37:58.856 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.856 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:58.856 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.856 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:58.857 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.219817+0000", 2024-09-13T14:37:58.857 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:58.857 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.857 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.857 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:58.857 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.219817+0000", 2024-09-13T14:37:58.857 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:58.857 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.857 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.857 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:58.857 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.219815+0000", 2024-09-13T14:37:58.857 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:37:58.857 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.858 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.858 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:58.858 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.219824+0000", 2024-09-13T14:37:58.858 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.3449999999999995e-06 2024-09-13T14:37:58.858 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.858 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.858 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:58.858 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.219841+0000", 2024-09-13T14:37:58.858 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6330000000000001e-05 2024-09-13T14:37:58.858 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.858 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.858 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:58.858 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.219850+0000", 2024-09-13T14:37:58.859 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.2280000000000004e-06 2024-09-13T14:37:58.859 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.859 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.859 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.859 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.266772+0000", 2024-09-13T14:37:58.859 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.046922012 2024-09-13T14:37:58.859 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.859 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.859 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:37:58.859 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.295174+0000", 2024-09-13T14:37:58.859 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.028402107999999999 2024-09-13T14:37:58.859 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.859 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.859 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.860 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.443017+0000", 2024-09-13T14:37:58.860 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.14784280499999999 2024-09-13T14:37:58.860 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.860 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.860 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:37:58.860 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.465867+0000", 2024-09-13T14:37:58.860 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.022849813 2024-09-13T14:37:58.860 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.860 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.860 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.860 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.721401+0000", 2024-09-13T14:37:58.860 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.255534258 2024-09-13T14:37:58.860 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.861 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.861 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:37:58.861 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.733248+0000", 2024-09-13T14:37:58.861 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.01184754 2024-09-13T14:37:58.861 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.861 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.861 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:37:58.861 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.757733+0000", 2024-09-13T14:37:58.861 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024485033 2024-09-13T14:37:58.861 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.861 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.861 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:58.862 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.847315+0000", 2024-09-13T14:37:58.862 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.089581220000000003 2024-09-13T14:37:58.862 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.862 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.862 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:37:58.862 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.849721+0000", 2024-09-13T14:37:58.862 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0024068840000000002 2024-09-13T14:37:58.863 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.863 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.863 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:58.863 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.856740+0000", 2024-09-13T14:37:58.863 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0070188029999999997 2024-09-13T14:37:58.863 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.863 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.863 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:37:58.863 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.867725+0000", 2024-09-13T14:37:58.863 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010984910000000001 2024-09-13T14:37:58.863 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.863 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.863 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:37:58.864 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.908776+0000", 2024-09-13T14:37:58.864 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.041050775999999997 2024-09-13T14:37:58.864 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:58.864 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:58.864 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:58.864 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.864 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.864 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:112957 7.e 7:73c81a1b:::benchmark_data_smithi102_227395_object142:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-09-13T14:37:58.864 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:28.238422+0000", 2024-09-13T14:37:58.864 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 30.604536059000001, 2024-09-13T14:37:58.864 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9574894330000001, 2024-09-13T14:37:58.864 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:58.864 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:58.864 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:37:58.865 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:37:58.865 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:37:58.865 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 112957 2024-09-13T14:37:58.865 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.865 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:58.865 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.865 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:58.865 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.238422+0000", 2024-09-13T14:37:58.865 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:58.865 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.865 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.865 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:58.865 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.238422+0000", 2024-09-13T14:37:58.866 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:58.866 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.866 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.866 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:58.866 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.238421+0000", 2024-09-13T14:37:58.866 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:37:58.866 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.866 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.866 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:58.866 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.238429+0000", 2024-09-13T14:37:58.866 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9349999999999994e-06 2024-09-13T14:37:58.866 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.866 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.866 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:58.867 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.238442+0000", 2024-09-13T14:37:58.867 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3501e-05 2024-09-13T14:37:58.867 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.867 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.867 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:58.867 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.238448+0000", 2024-09-13T14:37:58.867 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3759999999999996e-06 2024-09-13T14:37:58.867 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.867 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.867 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.867 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.297744+0000", 2024-09-13T14:37:58.867 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.059295036 2024-09-13T14:37:58.867 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.868 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.868 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:37:58.868 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.319171+0000", 2024-09-13T14:37:58.868 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.021427883000000002 2024-09-13T14:37:58.868 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.868 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.868 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.868 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.554554+0000", 2024-09-13T14:37:58.868 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.23538229799999999 2024-09-13T14:37:58.868 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.868 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.868 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:37:58.868 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.579299+0000", 2024-09-13T14:37:58.869 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024745277 2024-09-13T14:37:58.869 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.869 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.869 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.869 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.992160+0000", 2024-09-13T14:37:58.869 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4128609559999998 2024-09-13T14:37:58.869 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.869 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.869 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:37:58.869 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.011466+0000", 2024-09-13T14:37:58.869 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.019306104000000001 2024-09-13T14:37:58.869 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.869 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.870 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:37:58.870 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.039599+0000", 2024-09-13T14:37:58.870 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.028132766999999999 2024-09-13T14:37:58.870 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.870 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.870 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:58.870 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.138317+0000", 2024-09-13T14:37:58.870 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.098718599000000004 2024-09-13T14:37:58.870 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.870 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.870 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:37:58.870 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.139902+0000", 2024-09-13T14:37:58.870 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001584184 2024-09-13T14:37:58.871 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.871 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.871 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:58.871 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.148487+0000", 2024-09-13T14:37:58.871 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0085852440000000006 2024-09-13T14:37:58.871 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.871 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.871 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:37:58.871 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.156727+0000", 2024-09-13T14:37:58.871 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0082399269999999993 2024-09-13T14:37:58.871 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.871 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.871 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:37:58.871 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.195912+0000", 2024-09-13T14:37:58.872 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.039185042000000003 2024-09-13T14:37:58.872 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:58.872 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:58.872 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:58.872 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.872 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.872 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:112968 7.e 7:7cd0abb8:::benchmark_data_smithi102_227395_object153:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-09-13T14:37:58.872 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:28.367834+0000", 2024-09-13T14:37:58.872 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 30.475124228999999, 2024-09-13T14:37:58.872 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1335673269999997, 2024-09-13T14:37:58.872 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:58.872 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:58.872 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:37:58.873 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:37:58.873 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:37:58.873 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 112968 2024-09-13T14:37:58.873 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.873 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:58.873 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.873 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:58.873 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.367834+0000", 2024-09-13T14:37:58.873 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:58.873 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.873 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.873 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:58.873 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.367834+0000", 2024-09-13T14:37:58.874 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:58.874 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.874 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.874 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:58.874 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.367832+0000", 2024-09-13T14:37:58.874 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:37:58.874 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.874 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.874 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:58.874 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.367841+0000", 2024-09-13T14:37:58.874 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7539999999999995e-06 2024-09-13T14:37:58.874 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.874 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.874 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:58.875 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.367856+0000", 2024-09-13T14:37:58.875 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4499e-05 2024-09-13T14:37:58.875 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.875 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.875 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:58.875 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.367863+0000", 2024-09-13T14:37:58.875 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6690000000000006e-06 2024-09-13T14:37:58.875 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.875 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.875 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.875 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.321756+0000", 2024-09-13T14:37:58.875 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.95389310000000005 2024-09-13T14:37:58.875 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.876 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.876 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:37:58.876 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.346608+0000", 2024-09-13T14:37:58.876 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024851504 2024-09-13T14:37:58.876 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.876 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.876 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.876 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.658373+0000", 2024-09-13T14:37:58.876 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.31176493100000002 2024-09-13T14:37:58.876 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.876 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.876 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:37:58.876 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.677721+0000", 2024-09-13T14:37:58.877 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.019348260999999999 2024-09-13T14:37:58.877 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.877 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.877 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.877 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.244806+0000", 2024-09-13T14:37:58.877 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.567085134 2024-09-13T14:37:58.877 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.877 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.877 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:37:58.877 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.260918+0000", 2024-09-13T14:37:58.877 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016111721999999998 2024-09-13T14:37:58.877 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.877 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.877 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:37:58.878 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.285995+0000", 2024-09-13T14:37:58.878 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.025077351000000001 2024-09-13T14:37:58.878 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.878 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.878 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:58.878 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.451788+0000", 2024-09-13T14:37:58.878 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.16579242799999999 2024-09-13T14:37:58.878 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.878 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.878 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:37:58.878 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.453689+0000", 2024-09-13T14:37:58.878 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0019015049999999999 2024-09-13T14:37:58.878 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.879 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.879 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:58.879 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.459427+0000", 2024-09-13T14:37:58.879 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0057378819999999997 2024-09-13T14:37:58.879 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.879 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.879 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:37:58.879 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.465958+0000", 2024-09-13T14:37:58.879 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0065305709999999998 2024-09-13T14:37:58.879 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.879 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.879 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:37:58.879 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.501402+0000", 2024-09-13T14:37:58.879 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.035443745999999998 2024-09-13T14:37:58.880 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:58.880 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:58.880 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:58.880 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.880 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.880 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113005 7.e 7:78e9e6f9:::benchmark_data_smithi102_227395_object190:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-09-13T14:37:58.880 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:28.544298+0000", 2024-09-13T14:37:58.880 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 30.298660693999999, 2024-09-13T14:37:58.880 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0162806550000001, 2024-09-13T14:37:58.880 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:58.880 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:58.880 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:37:58.880 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:37:58.881 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:37:58.881 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113005 2024-09-13T14:37:58.881 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.881 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:58.881 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.881 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:58.881 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.544298+0000", 2024-09-13T14:37:58.881 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:58.881 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.881 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.881 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:58.881 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.544298+0000", 2024-09-13T14:37:58.881 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:58.882 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.882 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.882 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:58.882 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.544296+0000", 2024-09-13T14:37:58.882 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:37:58.882 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.882 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.882 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:58.882 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.544305+0000", 2024-09-13T14:37:58.882 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.0310000000000004e-06 2024-09-13T14:37:58.882 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.882 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.882 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:58.882 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.544318+0000", 2024-09-13T14:37:58.883 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3301e-05 2024-09-13T14:37:58.883 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.883 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.883 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:58.883 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.544325+0000", 2024-09-13T14:37:58.883 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9380000000000003e-06 2024-09-13T14:37:58.883 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.883 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.883 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.883 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.679520+0000", 2024-09-13T14:37:58.883 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1351946159999999 2024-09-13T14:37:58.883 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.883 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.883 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:37:58.884 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.697460+0000", 2024-09-13T14:37:58.884 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017939749000000001 2024-09-13T14:37:58.884 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.884 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.884 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.884 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.780770+0000", 2024-09-13T14:37:58.884 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.083310181999999997 2024-09-13T14:37:58.884 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.884 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.884 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:37:58.884 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.798851+0000", 2024-09-13T14:37:58.884 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018080978000000001 2024-09-13T14:37:58.884 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.885 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.885 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.885 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.382351+0000", 2024-09-13T14:37:58.885 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5834998100000002 2024-09-13T14:37:58.885 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.885 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.885 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:37:58.885 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.398282+0000", 2024-09-13T14:37:58.885 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.015931325999999999 2024-09-13T14:37:58.885 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.886 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.886 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:37:58.886 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.426371+0000", 2024-09-13T14:37:58.886 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.028089050000000001 2024-09-13T14:37:58.886 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.886 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.886 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:37:58.886 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.503108+0000", 2024-09-13T14:37:58.886 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.076737284000000003 2024-09-13T14:37:58.886 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.886 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.886 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:58.886 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.507418+0000", 2024-09-13T14:37:58.887 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0043095690000000001 2024-09-13T14:37:58.887 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.887 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.887 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:58.887 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.514013+0000", 2024-09-13T14:37:58.887 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0065951639999999997 2024-09-13T14:37:58.887 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.887 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.887 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:37:58.887 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.521065+0000", 2024-09-13T14:37:58.887 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.007051753 2024-09-13T14:37:58.887 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.887 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.887 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:37:58.888 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.560578+0000", 2024-09-13T14:37:58.888 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.039513581999999998 2024-09-13T14:37:58.888 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:58.888 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:58.888 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:58.888 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.888 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.888 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113646 7.e 7:7fda021e:::benchmark_data_smithi102_227395_object831:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-13T14:37:58.888 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:39.650939+0000", 2024-09-13T14:37:58.888 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 19.192018955000002, 2024-09-13T14:37:58.888 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4310964820000001, 2024-09-13T14:37:58.888 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:58.888 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:58.889 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:37:58.889 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:37:58.889 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:37:58.889 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113646 2024-09-13T14:37:58.889 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.889 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:58.889 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.889 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:58.889 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:39.650939+0000", 2024-09-13T14:37:58.889 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:58.889 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.889 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.889 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:58.889 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:39.650939+0000", 2024-09-13T14:37:58.890 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:58.890 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.890 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.890 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:58.890 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:39.650938+0000", 2024-09-13T14:37:58.890 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:37:58.890 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.890 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.890 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:58.890 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:39.650945+0000", 2024-09-13T14:37:58.890 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4200000000000001e-06 2024-09-13T14:37:58.890 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.890 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.890 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:58.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:58.892 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:58.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:58.893 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:39.650960+0000", 2024-09-13T14:37:58.893 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5028000000000001e-05 2024-09-13T14:37:58.894 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.894 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.894 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:58.894 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:39.650969+0000", 2024-09-13T14:37:58.894 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.4319999999999999e-06 2024-09-13T14:37:58.894 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.894 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.894 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.894 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:39.791532+0000", 2024-09-13T14:37:58.894 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.140563199 2024-09-13T14:37:58.894 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.894 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.894 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:37:58.894 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:39.808048+0000", 2024-09-13T14:37:58.895 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016515914999999999 2024-09-13T14:37:58.895 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.895 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.895 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.895 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:39.937277+0000", 2024-09-13T14:37:58.895 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.129229236 2024-09-13T14:37:58.895 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.895 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.895 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:37:58.895 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:39.953949+0000", 2024-09-13T14:37:58.895 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016672356999999999 2024-09-13T14:37:58.895 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.895 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.896 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.896 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.812544+0000", 2024-09-13T14:37:58.896 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8585950149999997 2024-09-13T14:37:58.896 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.896 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.896 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:37:58.896 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.829480+0000", 2024-09-13T14:37:58.896 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016936072999999999 2024-09-13T14:37:58.896 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.896 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.896 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:37:58.896 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.857149+0000", 2024-09-13T14:37:58.896 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.027668132000000002 2024-09-13T14:37:58.897 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.897 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.897 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:58.897 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.013996+0000", 2024-09-13T14:37:58.897 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.15684716000000001 2024-09-13T14:37:58.897 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.897 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.897 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:58.897 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.021795+0000", 2024-09-13T14:37:58.897 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0077991079999999999 2024-09-13T14:37:58.897 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.897 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.897 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:37:58.897 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.023954+0000", 2024-09-13T14:37:58.898 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0021587970000000001 2024-09-13T14:37:58.898 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.898 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.898 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:37:58.898 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.034566+0000", 2024-09-13T14:37:58.898 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010612244 2024-09-13T14:37:58.898 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.898 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.898 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:37:58.898 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.082036+0000", 2024-09-13T14:37:58.898 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.047470036 2024-09-13T14:37:58.898 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:58.898 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:58.899 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:58.899 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.899 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.899 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113660 7.e 7:72e391fe:::benchmark_data_smithi102_227395_object845:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-13T14:37:58.899 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:39.881206+0000", 2024-09-13T14:37:58.899 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 18.961752291, 2024-09-13T14:37:58.899 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2702958029999998, 2024-09-13T14:37:58.899 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:58.899 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:58.899 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:37:58.899 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:37:58.899 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:37:58.899 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113660 2024-09-13T14:37:58.900 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.900 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:58.900 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.900 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:58.900 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:39.881206+0000", 2024-09-13T14:37:58.900 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:58.900 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.900 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.900 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:58.900 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:39.881206+0000", 2024-09-13T14:37:58.900 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:58.900 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.900 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.900 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:58.901 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:39.881205+0000", 2024-09-13T14:37:58.901 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:37:58.901 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.901 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.901 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:58.901 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:39.881212+0000", 2024-09-13T14:37:58.901 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.17e-06 2024-09-13T14:37:58.901 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.901 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.901 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:58.901 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:39.881223+0000", 2024-09-13T14:37:58.901 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1214e-05 2024-09-13T14:37:58.901 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.902 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.902 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:58.902 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:39.881229+0000", 2024-09-13T14:37:58.902 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5559999999999998e-06 2024-09-13T14:37:58.902 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.902 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.902 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.902 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.072810+0000", 2024-09-13T14:37:58.902 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.19158088300000001 2024-09-13T14:37:58.902 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.902 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.902 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:37:58.902 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.098533+0000", 2024-09-13T14:37:58.903 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.025723314000000001 2024-09-13T14:37:58.903 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.903 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.903 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.903 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.225733+0000", 2024-09-13T14:37:58.903 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.12720013499999999 2024-09-13T14:37:58.903 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.903 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.903 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:37:58.903 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.243239+0000", 2024-09-13T14:37:58.903 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017505427 2024-09-13T14:37:58.903 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.903 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.903 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.904 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.929831+0000", 2024-09-13T14:37:58.904 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6865921210000003 2024-09-13T14:37:58.904 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.904 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.904 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:37:58.904 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.951285+0000", 2024-09-13T14:37:58.904 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.021454606000000001 2024-09-13T14:37:58.904 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.904 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.904 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:37:58.904 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.980812+0000", 2024-09-13T14:37:58.904 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.029526611000000001 2024-09-13T14:37:58.904 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.905 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.905 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:37:58.905 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.084340+0000", 2024-09-13T14:37:58.905 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.10352797699999999 2024-09-13T14:37:58.905 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.905 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.905 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:58.905 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.090949+0000", 2024-09-13T14:37:58.905 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0066088229999999998 2024-09-13T14:37:58.905 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.905 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.905 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:58.905 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.095550+0000", 2024-09-13T14:37:58.905 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0046012689999999998 2024-09-13T14:37:58.906 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.906 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.906 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:37:58.906 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.104748+0000", 2024-09-13T14:37:58.906 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0091979999999999996 2024-09-13T14:37:58.906 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.906 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.906 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:37:58.906 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.151502+0000", 2024-09-13T14:37:58.906 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.046753905999999998 2024-09-13T14:37:58.906 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:58.906 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:58.906 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:58.906 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.907 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.907 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113667 7.e 7:7011123e:::benchmark_data_smithi102_227395_object852:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-13T14:37:58.907 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:40.145165+0000", 2024-09-13T14:37:58.907 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 18.697793318999999, 2024-09-13T14:37:58.907 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2681343839999997, 2024-09-13T14:37:58.907 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:58.907 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:58.907 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:37:58.907 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:37:58.907 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:37:58.907 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113667 2024-09-13T14:37:58.907 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.907 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:58.908 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.908 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:58.908 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.145165+0000", 2024-09-13T14:37:58.908 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:58.908 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.908 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.908 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:58.908 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.145165+0000", 2024-09-13T14:37:58.908 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:58.909 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.909 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.909 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:58.909 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.145164+0000", 2024-09-13T14:37:58.909 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:37:58.909 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.909 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.909 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:58.909 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.145170+0000", 2024-09-13T14:37:58.909 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8379999999999998e-06 2024-09-13T14:37:58.909 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.909 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.909 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:58.909 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.145181+0000", 2024-09-13T14:37:58.910 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1521e-05 2024-09-13T14:37:58.910 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.910 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.910 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:58.910 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.145186+0000", 2024-09-13T14:37:58.910 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0200000000000002e-06 2024-09-13T14:37:58.910 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.910 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.910 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.910 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.244931+0000", 2024-09-13T14:37:58.910 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.099744219999999995 2024-09-13T14:37:58.910 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.910 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.911 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:37:58.911 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.263033+0000", 2024-09-13T14:37:58.911 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018102917 2024-09-13T14:37:58.911 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.911 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.911 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.911 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.467278+0000", 2024-09-13T14:37:58.911 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.20424453100000001 2024-09-13T14:37:58.911 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.911 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.911 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:37:58.911 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.487283+0000", 2024-09-13T14:37:58.911 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.020004724000000002 2024-09-13T14:37:58.912 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.912 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.912 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.912 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.223527+0000", 2024-09-13T14:37:58.912 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7362443320000001 2024-09-13T14:37:58.912 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.912 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.912 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:37:58.912 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.237432+0000", 2024-09-13T14:37:58.912 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013905244000000001 2024-09-13T14:37:58.912 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.912 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.912 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:37:58.913 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.264564+0000", 2024-09-13T14:37:58.913 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.027132177 2024-09-13T14:37:58.913 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.913 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.913 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:58.913 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.369627+0000", 2024-09-13T14:37:58.913 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.10506285 2024-09-13T14:37:58.913 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.913 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.913 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:37:58.913 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.370616+0000", 2024-09-13T14:37:58.913 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00098851600000000005 2024-09-13T14:37:58.913 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.913 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.914 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:58.914 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.376243+0000", 2024-09-13T14:37:58.914 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0056276249999999998 2024-09-13T14:37:58.914 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.914 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.914 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:37:58.914 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.381212+0000", 2024-09-13T14:37:58.914 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0049688850000000001 2024-09-13T14:37:58.914 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.914 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.914 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:37:58.914 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.413299+0000", 2024-09-13T14:37:58.914 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.032087169999999998 2024-09-13T14:37:58.915 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:58.915 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:58.915 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:58.915 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.915 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.915 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113673 7.e 7:707c08ea:::benchmark_data_smithi102_227395_object858:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-13T14:37:58.915 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:40.852904+0000", 2024-09-13T14:37:58.915 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 17.990054843999999, 2024-09-13T14:37:58.915 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0436350230000002, 2024-09-13T14:37:58.915 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:58.915 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:58.915 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:37:58.915 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:37:58.915 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:37:58.916 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113673 2024-09-13T14:37:58.916 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.916 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:58.916 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.916 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:58.916 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.852904+0000", 2024-09-13T14:37:58.916 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:58.916 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.916 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.916 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:58.916 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.852904+0000", 2024-09-13T14:37:58.916 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:58.917 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.917 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.917 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:58.917 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.852902+0000", 2024-09-13T14:37:58.917 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:37:58.917 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.917 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.917 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:58.917 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.852909+0000", 2024-09-13T14:37:58.917 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8900000000000001e-06 2024-09-13T14:37:58.917 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.917 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.917 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:58.917 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.852927+0000", 2024-09-13T14:37:58.918 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8281999999999999e-05 2024-09-13T14:37:58.918 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.918 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.918 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:58.918 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.852933+0000", 2024-09-13T14:37:58.918 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0220000000000001e-06 2024-09-13T14:37:58.918 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.918 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.918 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.918 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.932659+0000", 2024-09-13T14:37:58.918 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.079725349000000001 2024-09-13T14:37:58.918 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.918 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.918 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:37:58.919 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.961885+0000", 2024-09-13T14:37:58.919 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.029226683999999999 2024-09-13T14:37:58.919 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.919 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.919 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.919 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.190540+0000", 2024-09-13T14:37:58.919 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.22865431899999999 2024-09-13T14:37:58.919 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.919 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.919 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:37:58.919 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.213124+0000", 2024-09-13T14:37:58.919 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.022584453000000001 2024-09-13T14:37:58.919 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.920 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.920 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.920 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.694534+0000", 2024-09-13T14:37:58.920 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4814095549999999 2024-09-13T14:37:58.920 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.920 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.920 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:37:58.920 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.708113+0000", 2024-09-13T14:37:58.920 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013578961000000001 2024-09-13T14:37:58.920 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.920 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.920 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:37:58.920 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.734818+0000", 2024-09-13T14:37:58.920 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.026704998000000001 2024-09-13T14:37:58.921 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.921 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.921 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:58.921 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.843296+0000", 2024-09-13T14:37:58.921 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.10847884000000001 2024-09-13T14:37:58.921 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.921 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.921 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:37:58.921 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.844461+0000", 2024-09-13T14:37:58.921 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001164111 2024-09-13T14:37:58.921 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.921 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.921 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:58.922 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.853549+0000", 2024-09-13T14:37:58.922 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0090885069999999991 2024-09-13T14:37:58.922 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.922 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.922 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:37:58.922 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.859381+0000", 2024-09-13T14:37:58.922 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.005832266 2024-09-13T14:37:58.922 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.922 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.922 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:37:58.922 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.896539+0000", 2024-09-13T14:37:58.922 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.037157218999999998 2024-09-13T14:37:58.922 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:58.922 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:58.923 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:58.923 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.923 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.923 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDPGPush(7.e 236/222 [PushOp(7:765a5df7:::benchmark_data_smithi102_227395_object908:head, version: 209'56, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:765a5df7:::benchmark_data_smithi102_227395_object908:head@209'56, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T14:37:58.923 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:41.327036+0000", 2024-09-13T14:37:58.923 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 17.515922107000002, 2024-09-13T14:37:58.923 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4911342149999998, 2024-09-13T14:37:58.923 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:58.923 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T14:37:58.923 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:58.923 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.923 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:58.923 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.327036+0000", 2024-09-13T14:37:58.924 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:58.924 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.924 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.924 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:58.924 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.327036+0000", 2024-09-13T14:37:58.924 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:58.924 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.924 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.924 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:58.924 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.327035+0000", 2024-09-13T14:37:58.924 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:37:58.924 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.924 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.925 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:58.925 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.327094+0000", 2024-09-13T14:37:58.925 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8471999999999999e-05 2024-09-13T14:37:58.925 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.925 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.925 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:58.925 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.327110+0000", 2024-09-13T14:37:58.925 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6333000000000001e-05 2024-09-13T14:37:58.925 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.925 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.925 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:58.925 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.327125+0000", 2024-09-13T14:37:58.925 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4766999999999999e-05 2024-09-13T14:37:58.925 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.926 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.926 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.926 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.756102+0000", 2024-09-13T14:37:58.926 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4289765430000001 2024-09-13T14:37:58.926 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.926 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.926 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:37:58.926 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.757954+0000", 2024-09-13T14:37:58.926 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0018523560000000001 2024-09-13T14:37:58.926 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.926 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.926 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:37:58.926 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.818171+0000", 2024-09-13T14:37:58.927 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.060216609999999997 2024-09-13T14:37:58.927 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:58.927 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:58.927 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:58.927 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.927 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.927 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113723 7.e 7:765a5df7:::benchmark_data_smithi102_227395_object908:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-13T14:37:58.927 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:41.735065+0000", 2024-09-13T14:37:58.927 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 17.107893397000002, 2024-09-13T14:37:58.927 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6293769879999997, 2024-09-13T14:37:58.927 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:58.927 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:58.927 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:37:58.928 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:37:58.928 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:37:58.928 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113723 2024-09-13T14:37:58.928 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.928 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:58.928 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.928 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:58.928 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.735065+0000", 2024-09-13T14:37:58.928 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:58.928 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.928 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.928 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:58.928 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.735065+0000", 2024-09-13T14:37:58.929 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:58.929 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.929 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.929 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:58.929 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.735064+0000", 2024-09-13T14:37:58.929 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:37:58.929 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.929 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.929 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:58.929 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.735071+0000", 2024-09-13T14:37:58.929 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2289999999999997e-06 2024-09-13T14:37:58.929 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.929 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.929 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:58.930 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.735085+0000", 2024-09-13T14:37:58.930 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4436e-05 2024-09-13T14:37:58.930 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.930 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.930 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:58.930 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.735092+0000", 2024-09-13T14:37:58.930 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.216e-06 2024-09-13T14:37:58.930 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.930 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.930 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.930 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.016651+0000", 2024-09-13T14:37:58.930 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.28155898499999998 2024-09-13T14:37:58.930 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.931 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.931 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:37:58.931 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.027127+0000", 2024-09-13T14:37:58.931 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010475241999999999 2024-09-13T14:37:58.931 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.931 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.931 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.931 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.819782+0000", 2024-09-13T14:37:58.931 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7926552459999998 2024-09-13T14:37:58.931 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.931 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.931 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:37:58.931 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.838097+0000", 2024-09-13T14:37:58.932 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018315084999999998 2024-09-13T14:37:58.932 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.932 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.932 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.932 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.103166+0000", 2024-09-13T14:37:58.932 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2650688400000001 2024-09-13T14:37:58.932 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.932 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.932 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:37:58.932 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.113423+0000", 2024-09-13T14:37:58.932 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010256919 2024-09-13T14:37:58.932 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.932 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.933 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:37:58.933 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.159715+0000", 2024-09-13T14:37:58.933 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.046292424999999998 2024-09-13T14:37:58.933 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.933 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.933 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:37:58.933 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.242694+0000", 2024-09-13T14:37:58.933 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.082978417999999998 2024-09-13T14:37:58.933 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.933 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.933 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:58.933 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.310555+0000", 2024-09-13T14:37:58.933 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.067861006000000001 2024-09-13T14:37:58.933 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.934 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.934 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:58.934 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.316591+0000", 2024-09-13T14:37:58.934 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0060365549999999999 2024-09-13T14:37:58.934 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.934 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.934 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:37:58.934 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.328028+0000", 2024-09-13T14:37:58.934 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.01143724 2024-09-13T14:37:58.934 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.934 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.934 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:37:58.934 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.364442+0000", 2024-09-13T14:37:58.934 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.036413661 2024-09-13T14:37:58.935 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:58.935 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:58.935 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:58.935 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.935 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.935 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113733 7.e 7:71f1f434:::benchmark_data_smithi102_227395_object918:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-13T14:37:58.935 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:41.768799+0000", 2024-09-13T14:37:58.935 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 17.074159858000002, 2024-09-13T14:37:58.935 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4105988570000001, 2024-09-13T14:37:58.935 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:58.935 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:58.935 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:37:58.935 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:37:58.936 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:37:58.936 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113733 2024-09-13T14:37:58.936 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.936 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:58.936 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.936 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:58.936 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.768799+0000", 2024-09-13T14:37:58.936 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:58.936 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.936 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.936 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:58.936 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.768799+0000", 2024-09-13T14:37:58.936 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:58.937 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.937 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.937 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:58.937 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.768797+0000", 2024-09-13T14:37:58.937 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:37:58.937 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.937 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.937 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:58.937 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.768805+0000", 2024-09-13T14:37:58.937 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1880000000000002e-06 2024-09-13T14:37:58.937 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.937 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.937 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:58.938 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.768818+0000", 2024-09-13T14:37:58.938 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3322999999999999e-05 2024-09-13T14:37:58.938 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.938 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.938 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:58.938 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.768826+0000", 2024-09-13T14:37:58.938 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5070000000000003e-06 2024-09-13T14:37:58.938 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.938 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.938 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.938 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.029999+0000", 2024-09-13T14:37:58.938 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.261173402 2024-09-13T14:37:58.938 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.938 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.939 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:37:58.939 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.071555+0000", 2024-09-13T14:37:58.939 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.041556211000000003 2024-09-13T14:37:58.939 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.939 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.939 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.939 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.202418+0000", 2024-09-13T14:37:58.939 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.13086251099999999 2024-09-13T14:37:58.939 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.939 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.939 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:37:58.939 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.224452+0000", 2024-09-13T14:37:58.939 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.022033740999999999 2024-09-13T14:37:58.940 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.940 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.940 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.940 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.962029+0000", 2024-09-13T14:37:58.940 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7375775129999997 2024-09-13T14:37:58.940 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.940 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.940 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:37:58.940 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.976076+0000", 2024-09-13T14:37:58.940 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014046856999999999 2024-09-13T14:37:58.940 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.940 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.940 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:37:58.941 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.000274+0000", 2024-09-13T14:37:58.941 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024197749000000001 2024-09-13T14:37:58.941 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.941 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.941 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:58.941 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.099261+0000", 2024-09-13T14:37:58.941 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.098987027000000005 2024-09-13T14:37:58.941 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.941 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.941 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:37:58.941 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.101138+0000", 2024-09-13T14:37:58.941 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0018768680000000001 2024-09-13T14:37:58.941 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.942 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.942 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:58.942 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.111923+0000", 2024-09-13T14:37:58.942 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010785444999999999 2024-09-13T14:37:58.942 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.942 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.942 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:37:58.942 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.124357+0000", 2024-09-13T14:37:58.942 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012433962999999999 2024-09-13T14:37:58.942 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.942 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.942 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:37:58.942 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.179397+0000", 2024-09-13T14:37:58.942 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.055040402000000002 2024-09-13T14:37:58.943 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:58.943 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:58.943 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:58.943 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.943 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.943 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113763 7.e 7:7a44c088:::benchmark_data_smithi102_227395_object948:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-13T14:37:58.943 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:41.987161+0000", 2024-09-13T14:37:58.943 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 16.855797640999999, 2024-09-13T14:37:58.943 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.450241471, 2024-09-13T14:37:58.943 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:58.943 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:58.943 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:37:58.943 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:37:58.944 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:37:58.944 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113763 2024-09-13T14:37:58.944 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.944 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:58.944 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.944 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:58.944 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.987161+0000", 2024-09-13T14:37:58.944 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:58.944 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.944 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.944 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:58.944 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.987161+0000", 2024-09-13T14:37:58.944 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:58.945 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.945 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.945 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:58.945 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.987159+0000", 2024-09-13T14:37:58.945 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:37:58.945 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.945 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.945 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:58.945 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.987166+0000", 2024-09-13T14:37:58.945 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5080000000000002e-06 2024-09-13T14:37:58.945 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.945 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.945 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:58.945 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.987178+0000", 2024-09-13T14:37:58.946 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2441e-05 2024-09-13T14:37:58.946 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.946 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.946 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:58.946 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.987184+0000", 2024-09-13T14:37:58.946 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5879999999999997e-06 2024-09-13T14:37:58.946 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.946 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.946 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.946 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.073662+0000", 2024-09-13T14:37:58.946 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.086478476999999998 2024-09-13T14:37:58.946 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.946 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.946 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:37:58.947 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.099105+0000", 2024-09-13T14:37:58.947 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.025442959000000001 2024-09-13T14:37:58.947 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.947 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.947 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.947 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.302626+0000", 2024-09-13T14:37:58.947 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.203520283 2024-09-13T14:37:58.947 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.947 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.947 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:37:58.947 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.324932+0000", 2024-09-13T14:37:58.947 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.022305898000000001 2024-09-13T14:37:58.947 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.948 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.948 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.948 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.224175+0000", 2024-09-13T14:37:58.948 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8992434930000002 2024-09-13T14:37:58.948 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.948 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.948 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:37:58.948 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.241799+0000", 2024-09-13T14:37:58.948 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017623608999999998 2024-09-13T14:37:58.948 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.948 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.948 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:37:58.948 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.274010+0000", 2024-09-13T14:37:58.948 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.032211245999999999 2024-09-13T14:37:58.949 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.949 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.949 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:58.949 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.364856+0000", 2024-09-13T14:37:58.949 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.090845578999999996 2024-09-13T14:37:58.949 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.949 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.949 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:58.949 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.372367+0000", 2024-09-13T14:37:58.949 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0075111149999999996 2024-09-13T14:37:58.949 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.949 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.949 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:37:58.950 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.374485+0000", 2024-09-13T14:37:58.950 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0021184810000000002 2024-09-13T14:37:58.950 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.950 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.950 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:37:58.950 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.386174+0000", 2024-09-13T14:37:58.950 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011689181999999999 2024-09-13T14:37:58.950 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.950 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.950 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:37:58.950 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.437402+0000", 2024-09-13T14:37:58.950 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.051227946000000003 2024-09-13T14:37:58.950 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:58.950 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:58.951 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:58.951 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.951 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.951 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113767 7.e 7:78506392:::benchmark_data_smithi102_227395_object952:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-13T14:37:58.951 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:42.006143+0000", 2024-09-13T14:37:58.951 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 16.8368158, 2024-09-13T14:37:58.951 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0048697080000002, 2024-09-13T14:37:58.951 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:58.951 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:58.951 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:37:58.951 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:37:58.951 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:37:58.951 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113767 2024-09-13T14:37:58.952 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.952 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:58.952 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.952 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:58.952 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.006143+0000", 2024-09-13T14:37:58.952 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:58.952 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.952 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.952 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:58.952 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.006143+0000", 2024-09-13T14:37:58.952 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:58.952 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.952 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.953 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:58.953 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.006141+0000", 2024-09-13T14:37:58.953 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:37:58.953 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.953 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.953 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:58.953 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.006149+0000", 2024-09-13T14:37:58.953 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8699999999999992e-06 2024-09-13T14:37:58.953 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.953 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.953 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:58.953 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.006161+0000", 2024-09-13T14:37:58.953 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1939999999999999e-05 2024-09-13T14:37:58.953 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.954 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.954 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:58.954 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.006168+0000", 2024-09-13T14:37:58.954 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4390000000000003e-06 2024-09-13T14:37:58.954 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.954 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.954 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.954 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.101455+0000", 2024-09-13T14:37:58.954 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.095286518000000001 2024-09-13T14:37:58.954 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.954 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.954 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:37:58.954 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.125347+0000", 2024-09-13T14:37:58.955 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.023891539999999999 2024-09-13T14:37:58.955 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.955 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.955 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.955 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.397577+0000", 2024-09-13T14:37:58.955 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.27223060100000002 2024-09-13T14:37:58.955 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.955 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.955 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:37:58.955 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.417123+0000", 2024-09-13T14:37:58.955 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.019546392999999999 2024-09-13T14:37:58.955 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.955 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.955 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.956 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.791931+0000", 2024-09-13T14:37:58.956 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3748072650000003 2024-09-13T14:37:58.956 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.956 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.956 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:37:58.956 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.809411+0000", 2024-09-13T14:37:58.956 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017480623000000001 2024-09-13T14:37:58.956 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.956 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.956 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:37:58.956 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.831143+0000", 2024-09-13T14:37:58.956 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.021731145 2024-09-13T14:37:58.956 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.957 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.957 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:58.957 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.959387+0000", 2024-09-13T14:37:58.957 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.12824418500000001 2024-09-13T14:37:58.957 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.957 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.957 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:58.957 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.964317+0000", 2024-09-13T14:37:58.957 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0049304350000000004 2024-09-13T14:37:58.957 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.957 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.957 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:37:58.957 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.966146+0000", 2024-09-13T14:37:58.957 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001829041 2024-09-13T14:37:58.958 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.958 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.958 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:37:58.958 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.972871+0000", 2024-09-13T14:37:58.958 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0067250080000000002 2024-09-13T14:37:58.958 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.958 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.958 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:37:58.958 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.011012+0000", 2024-09-13T14:37:58.958 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.038141126999999997 2024-09-13T14:37:58.958 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:58.958 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:58.958 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:58.959 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.959 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.959 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113770 7.e 7:7f1c5c68:::benchmark_data_smithi102_227395_object955:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-13T14:37:58.959 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:42.134010+0000", 2024-09-13T14:37:58.959 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 16.708948793000001, 2024-09-13T14:37:58.959 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9464087430000001, 2024-09-13T14:37:58.959 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:58.959 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:58.959 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:37:58.959 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:37:58.959 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:37:58.959 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113770 2024-09-13T14:37:58.959 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.959 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:58.960 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.960 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:58.960 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.134010+0000", 2024-09-13T14:37:58.960 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:58.960 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.960 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.960 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:58.960 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.134010+0000", 2024-09-13T14:37:58.960 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:58.960 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.960 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.960 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:58.960 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.134008+0000", 2024-09-13T14:37:58.960 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:37:58.961 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.961 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.961 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:58.961 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.134019+0000", 2024-09-13T14:37:58.961 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0458e-05 2024-09-13T14:37:58.961 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.961 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.961 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:58.961 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.134030+0000", 2024-09-13T14:37:58.961 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1311999999999999e-05 2024-09-13T14:37:58.961 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.961 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.961 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:58.962 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.134036+0000", 2024-09-13T14:37:58.962 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0229999999999998e-06 2024-09-13T14:37:58.962 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.962 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.962 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.962 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.419198+0000", 2024-09-13T14:37:58.962 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.28516167399999998 2024-09-13T14:37:58.962 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.962 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.962 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:37:58.962 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.440868+0000", 2024-09-13T14:37:58.962 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.021670779000000001 2024-09-13T14:37:58.962 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.962 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.963 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.963 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.554028+0000", 2024-09-13T14:37:58.963 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.113159653 2024-09-13T14:37:58.963 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.963 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.963 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:37:58.963 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.576302+0000", 2024-09-13T14:37:58.963 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.022274200000000001 2024-09-13T14:37:58.963 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.963 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.963 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.963 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.890592+0000", 2024-09-13T14:37:58.963 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3142900390000003 2024-09-13T14:37:58.964 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.964 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.964 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:37:58.964 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.904275+0000", 2024-09-13T14:37:58.964 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013682952 2024-09-13T14:37:58.964 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.964 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.964 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:37:58.964 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.931383+0000", 2024-09-13T14:37:58.964 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.027107368 2024-09-13T14:37:58.964 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.964 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.964 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:37:58.965 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.012801+0000", 2024-09-13T14:37:58.965 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.081418328999999998 2024-09-13T14:37:58.965 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.965 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.965 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:58.965 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.018242+0000", 2024-09-13T14:37:58.965 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0054409819999999996 2024-09-13T14:37:58.965 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.965 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.965 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:58.965 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.025406+0000", 2024-09-13T14:37:58.965 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0071640150000000001 2024-09-13T14:37:58.965 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.965 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.966 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:37:58.966 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.036309+0000", 2024-09-13T14:37:58.966 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010902949 2024-09-13T14:37:58.966 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.966 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.966 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:37:58.966 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.080418+0000", 2024-09-13T14:37:58.966 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.044109451000000001 2024-09-13T14:37:58.966 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:58.966 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:58.966 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:58.966 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.966 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.967 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113800 7.e 7:745cc90f:::benchmark_data_smithi102_227395_object985:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-13T14:37:58.967 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:42.931912+0000", 2024-09-13T14:37:58.967 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 15.911046646999999, 2024-09-13T14:37:58.967 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3762813339999997, 2024-09-13T14:37:58.967 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:58.967 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:58.967 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:37:58.967 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:37:58.967 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:37:58.967 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113800 2024-09-13T14:37:58.967 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.967 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:58.968 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.968 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:58.968 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.931912+0000", 2024-09-13T14:37:58.968 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:58.968 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.968 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.968 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:58.968 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.931912+0000", 2024-09-13T14:37:58.968 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:58.968 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.968 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.968 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:58.968 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.931910+0000", 2024-09-13T14:37:58.969 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:37:58.969 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.969 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.969 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:58.969 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.931924+0000", 2024-09-13T14:37:58.969 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3915e-05 2024-09-13T14:37:58.969 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.969 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.969 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:58.969 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.931937+0000", 2024-09-13T14:37:58.969 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2835e-05 2024-09-13T14:37:58.969 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.969 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.969 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:58.970 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.931942+0000", 2024-09-13T14:37:58.970 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.181e-06 2024-09-13T14:37:58.970 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.970 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.970 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.970 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:43.100264+0000", 2024-09-13T14:37:58.970 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.16832238599999999 2024-09-13T14:37:58.970 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.970 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.970 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:37:58.970 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:43.124629+0000", 2024-09-13T14:37:58.970 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024364495 2024-09-13T14:37:58.970 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.971 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.971 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.971 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:43.260475+0000", 2024-09-13T14:37:58.971 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.13584602100000001 2024-09-13T14:37:58.971 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.971 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.971 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:37:58.971 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:43.281800+0000", 2024-09-13T14:37:58.971 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.021324864999999998 2024-09-13T14:37:58.971 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.971 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.971 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.971 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.133937+0000", 2024-09-13T14:37:58.972 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8521372940000003 2024-09-13T14:37:58.972 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.972 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.972 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:37:58.972 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.152762+0000", 2024-09-13T14:37:58.972 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018825346 2024-09-13T14:37:58.972 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.972 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.972 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:37:58.972 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.167553+0000", 2024-09-13T14:37:58.972 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014790905 2024-09-13T14:37:58.972 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.972 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.973 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:37:58.973 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.272976+0000", 2024-09-13T14:37:58.973 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.105422497 2024-09-13T14:37:58.973 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.974 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.974 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:58.974 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.275989+0000", 2024-09-13T14:37:58.974 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0030131899999999998 2024-09-13T14:37:58.974 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.974 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.974 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:58.974 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.279778+0000", 2024-09-13T14:37:58.974 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0037891349999999999 2024-09-13T14:37:58.974 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.974 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.974 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:37:58.974 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.285756+0000", 2024-09-13T14:37:58.975 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0059774989999999998 2024-09-13T14:37:58.975 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.975 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.975 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:37:58.975 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.308193+0000", 2024-09-13T14:37:58.975 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.022437436000000002 2024-09-13T14:37:58.975 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:58.975 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:58.975 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:58.975 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.975 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.975 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113838 7.e 7:725b9897:::benchmark_data_smithi102_227395_object1023:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-13T14:37:58.975 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:43.215448+0000", 2024-09-13T14:37:58.976 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 15.62750995, 2024-09-13T14:37:58.976 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1232826060000001, 2024-09-13T14:37:58.976 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:58.976 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:58.976 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:37:58.976 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:37:58.976 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:37:58.976 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113838 2024-09-13T14:37:58.976 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.976 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:58.976 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.976 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:58.976 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:43.215448+0000", 2024-09-13T14:37:58.977 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:58.977 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.977 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.978 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:58.978 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:43.215448+0000", 2024-09-13T14:37:58.978 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:58.978 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.978 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.978 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:58.978 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:43.215447+0000", 2024-09-13T14:37:58.978 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:37:58.978 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.978 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.978 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:58.978 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:43.215455+0000", 2024-09-13T14:37:58.978 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8089999999999993e-06 2024-09-13T14:37:58.979 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.979 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.979 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:58.979 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:43.215469+0000", 2024-09-13T14:37:58.979 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4056000000000001e-05 2024-09-13T14:37:58.979 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.979 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.979 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:58.979 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:43.215476+0000", 2024-09-13T14:37:58.979 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3749999999999997e-06 2024-09-13T14:37:58.979 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.979 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.979 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.980 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:43.413654+0000", 2024-09-13T14:37:58.980 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.19817833800000001 2024-09-13T14:37:58.980 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.980 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.980 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:37:58.980 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:43.438154+0000", 2024-09-13T14:37:58.980 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024499441 2024-09-13T14:37:58.980 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.980 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.980 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.980 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:43.627551+0000", 2024-09-13T14:37:58.980 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.189397542 2024-09-13T14:37:58.980 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.981 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.981 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:37:58.981 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:43.651356+0000", 2024-09-13T14:37:58.981 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.023804698999999999 2024-09-13T14:37:58.981 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.981 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.981 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.981 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.210574+0000", 2024-09-13T14:37:58.981 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5592174459999999 2024-09-13T14:37:58.981 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.981 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.981 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:37:58.981 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.219313+0000", 2024-09-13T14:37:58.982 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0087393239999999997 2024-09-13T14:37:58.982 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.982 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.982 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:37:58.982 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.233331+0000", 2024-09-13T14:37:58.982 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014018256999999999 2024-09-13T14:37:58.982 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.982 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.982 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:37:58.982 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.309091+0000", 2024-09-13T14:37:58.982 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.075759601999999995 2024-09-13T14:37:58.982 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.982 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.983 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:58.983 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.311847+0000", 2024-09-13T14:37:58.983 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0027565770000000001 2024-09-13T14:37:58.983 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.983 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.983 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:58.983 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.315219+0000", 2024-09-13T14:37:58.983 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0033721850000000002 2024-09-13T14:37:58.983 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.984 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.984 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:37:58.984 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.319648+0000", 2024-09-13T14:37:58.984 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0044280630000000003 2024-09-13T14:37:58.984 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.984 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.984 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:37:58.984 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.338731+0000", 2024-09-13T14:37:58.984 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.019083547999999999 2024-09-13T14:37:58.984 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:58.984 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:58.984 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:58.985 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.985 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.985 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113868 7.e 7:729791b6:::benchmark_data_smithi102_227395_object1053:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-13T14:37:58.985 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:44.019108+0000", 2024-09-13T14:37:58.985 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 14.823850572, 2024-09-13T14:37:58.985 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6053882540000002, 2024-09-13T14:37:58.985 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:58.985 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:58.985 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:37:58.985 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:37:58.985 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:37:58.985 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113868 2024-09-13T14:37:58.986 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.986 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:58.986 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.986 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:58.986 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.019108+0000", 2024-09-13T14:37:58.986 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:58.986 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.986 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.986 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:58.986 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.019108+0000", 2024-09-13T14:37:58.986 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:58.986 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.986 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.987 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:58.987 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.019107+0000", 2024-09-13T14:37:58.987 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:37:58.987 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.987 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.987 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:58.987 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.019112+0000", 2024-09-13T14:37:58.987 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7239999999999997e-06 2024-09-13T14:37:58.987 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.987 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.987 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:58.987 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.019122+0000", 2024-09-13T14:37:58.988 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6539999999999994e-06 2024-09-13T14:37:58.988 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.988 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.988 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:58.988 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.019128+0000", 2024-09-13T14:37:58.988 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8749999999999997e-06 2024-09-13T14:37:58.988 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.988 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.988 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.988 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.114948+0000", 2024-09-13T14:37:58.988 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.095819748999999996 2024-09-13T14:37:58.988 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.989 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.989 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:37:58.989 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.144054+0000", 2024-09-13T14:37:58.989 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.029106476999999999 2024-09-13T14:37:58.989 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.989 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.989 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.989 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.443107+0000", 2024-09-13T14:37:58.989 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.29905287200000003 2024-09-13T14:37:58.989 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.989 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.989 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:37:58.990 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.470900+0000", 2024-09-13T14:37:58.990 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.027792595999999999 2024-09-13T14:37:58.990 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.990 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.990 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.990 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.424154+0000", 2024-09-13T14:37:58.990 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9532546819999999 2024-09-13T14:37:58.990 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.990 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.990 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:37:58.990 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.431994+0000", 2024-09-13T14:37:58.990 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0078397650000000003 2024-09-13T14:37:58.990 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.991 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.991 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:37:58.991 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.445713+0000", 2024-09-13T14:37:58.991 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013719080999999999 2024-09-13T14:37:58.991 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.991 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.991 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:58.991 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.463051+0000", 2024-09-13T14:37:58.991 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017337680000000001 2024-09-13T14:37:58.991 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.991 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.991 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:58.992 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.465649+0000", 2024-09-13T14:37:58.992 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002598684 2024-09-13T14:37:58.992 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.992 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.992 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:37:58.992 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.600753+0000", 2024-09-13T14:37:58.992 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.13510388200000001 2024-09-13T14:37:58.992 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.992 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.992 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:37:58.992 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.605058+0000", 2024-09-13T14:37:58.992 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0043048879999999998 2024-09-13T14:37:58.992 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.993 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.993 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:37:58.993 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.624496+0000", 2024-09-13T14:37:58.993 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.019437904999999998 2024-09-13T14:37:58.993 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:58.993 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:58.993 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:58.993 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.993 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.993 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113882 7.e 7:73c217af:::benchmark_data_smithi102_227395_object1067:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-13T14:37:58.993 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:44.061776+0000", 2024-09-13T14:37:58.993 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 14.781182244, 2024-09-13T14:37:58.993 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.59134807, 2024-09-13T14:37:58.993 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:58.994 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:58.994 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:37:58.994 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:37:58.994 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:37:58.994 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113882 2024-09-13T14:37:58.994 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.994 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:58.994 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.994 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:58.994 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.061776+0000", 2024-09-13T14:37:58.994 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:58.994 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.995 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.995 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:58.995 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.061776+0000", 2024-09-13T14:37:58.995 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:58.995 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.995 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.995 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:37:58.996 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:58.997 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.061775+0000", 2024-09-13T14:37:58.997 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:37:58.997 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.997 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.997 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:58.997 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.061781+0000", 2024-09-13T14:37:58.997 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0839999999999997e-06 2024-09-13T14:37:58.997 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.997 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.997 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:58.997 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.061793+0000", 2024-09-13T14:37:58.997 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1939e-05 2024-09-13T14:37:58.997 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.998 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.998 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:58.998 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.061801+0000", 2024-09-13T14:37:58.998 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2490000000000001e-06 2024-09-13T14:37:58.998 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.998 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.998 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.998 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.146275+0000", 2024-09-13T14:37:58.998 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.084473626999999996 2024-09-13T14:37:58.998 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.998 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.998 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:37:58.999 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.170652+0000", 2024-09-13T14:37:58.999 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024377374 2024-09-13T14:37:58.999 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.999 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.999 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:58.999 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.548899+0000", 2024-09-13T14:37:58.999 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.37824670900000001 2024-09-13T14:37:58.999 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:58.999 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:58.999 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:37:59.000 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.610957+0000", 2024-09-13T14:37:59.000 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.062058073999999998 2024-09-13T14:37:59.000 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:59.000 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:59.000 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:59.000 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.488800+0000", 2024-09-13T14:37:59.000 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8778428460000001 2024-09-13T14:37:59.000 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:59.000 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:59.000 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:37:59.000 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.496393+0000", 2024-09-13T14:37:59.000 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0075932910000000003 2024-09-13T14:37:59.000 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:59.001 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:59.001 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:37:59.001 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.509330+0000", 2024-09-13T14:37:59.001 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012937051999999999 2024-09-13T14:37:59.001 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:59.001 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:59.001 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:37:59.001 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.625334+0000", 2024-09-13T14:37:59.001 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.116003785 2024-09-13T14:37:59.001 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:59.001 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:59.001 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:59.002 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.627836+0000", 2024-09-13T14:37:59.002 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0025019109999999999 2024-09-13T14:37:59.002 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:59.002 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:59.002 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:59.002 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.630456+0000", 2024-09-13T14:37:59.002 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0026199159999999999 2024-09-13T14:37:59.002 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:59.002 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:59.002 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:37:59.002 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.634782+0000", 2024-09-13T14:37:59.002 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0043260019999999998 2024-09-13T14:37:59.002 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:59.003 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:59.003 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:37:59.003 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.653124+0000", 2024-09-13T14:37:59.003 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018342607 2024-09-13T14:37:59.003 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:59.003 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:59.003 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:59.003 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:59.003 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:59.003 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113893 7.e 7:70855f2b:::benchmark_data_smithi102_227395_object1078:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-13T14:37:59.003 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:44.143011+0000", 2024-09-13T14:37:59.003 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 14.699947097999999, 2024-09-13T14:37:59.003 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.54156615, 2024-09-13T14:37:59.004 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:37:59.004 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:37:59.004 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:37:59.004 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:37:59.004 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:37:59.004 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113893 2024-09-13T14:37:59.004 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:59.004 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:37:59.004 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:59.004 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:37:59.004 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.143011+0000", 2024-09-13T14:37:59.004 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:59.005 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:59.005 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:59.005 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:37:59.005 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.143011+0000", 2024-09-13T14:37:59.005 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:37:59.005 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:59.005 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:59.005 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:37:59.005 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.143010+0000", 2024-09-13T14:37:59.005 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:37:59.005 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:59.005 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:59.006 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:37:59.006 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.143017+0000", 2024-09-13T14:37:59.006 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.0389999999999999e-06 2024-09-13T14:37:59.006 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:59.006 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:59.006 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:37:59.006 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.143028+0000", 2024-09-13T14:37:59.006 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0519999999999999e-05 2024-09-13T14:37:59.006 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:59.006 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:59.006 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:37:59.006 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.143033+0000", 2024-09-13T14:37:59.007 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5110000000000003e-06 2024-09-13T14:37:59.007 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:59.007 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:59.007 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:59.007 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.613507+0000", 2024-09-13T14:37:59.007 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.47047437800000003 2024-09-13T14:37:59.007 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:59.007 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:59.007 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:37:59.007 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.642684+0000", 2024-09-13T14:37:59.007 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.029176705000000001 2024-09-13T14:37:59.007 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:59.008 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:59.008 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:59.008 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.784620+0000", 2024-09-13T14:37:59.008 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.14193537000000001 2024-09-13T14:37:59.008 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:59.008 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:59.008 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:37:59.008 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.810347+0000", 2024-09-13T14:37:59.008 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.025727015999999998 2024-09-13T14:37:59.008 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:59.008 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:59.008 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:37:59.009 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.567851+0000", 2024-09-13T14:37:59.009 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7575047000000001 2024-09-13T14:37:59.009 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:59.009 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:59.009 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:37:59.009 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.575637+0000", 2024-09-13T14:37:59.009 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0077859319999999997 2024-09-13T14:37:59.009 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:59.009 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:59.009 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:37:59.009 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.590198+0000", 2024-09-13T14:37:59.009 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.01456061 2024-09-13T14:37:59.010 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:59.010 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:59.010 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:37:59.010 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.653927+0000", 2024-09-13T14:37:59.010 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.063728952000000005 2024-09-13T14:37:59.010 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:59.010 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:59.010 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:59.010 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.656466+0000", 2024-09-13T14:37:59.010 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0025390679999999998 2024-09-13T14:37:59.010 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:59.010 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:59.011 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:37:59.011 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.659062+0000", 2024-09-13T14:37:59.011 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0025957469999999998 2024-09-13T14:37:59.011 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:59.011 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:59.011 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:37:59.011 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.663437+0000", 2024-09-13T14:37:59.011 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0043751850000000002 2024-09-13T14:37:59.011 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:37:59.011 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:37:59.011 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:37:59.012 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.684577+0000", 2024-09-13T14:37:59.012 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.021140732999999998 2024-09-13T14:37:59.012 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:59.012 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:59.012 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:59.012 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:37:59.012 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:37:59.012 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:37:59.017 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:59.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:59.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:37:59.119 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:37:59.141 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:59.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:59.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:37:59.243 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:37:59.266 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:59.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:59.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:37:59.310 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:37:59.368 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:37:59.389 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:59.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:59.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:37:59.491 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:37:59.514 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:59.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:59.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:37:59.616 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:37:59.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:59.639 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:59.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:37:59.709 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T14:37:59.740 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:37:59.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:37:59 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 51045/332364 objects degraded (15.358%), 12 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:37:59.761 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:37:59.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:59.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:37:59.815 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:37:59 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 51045/332364 objects degraded (15.358%), 12 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:37:59.816 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:37:59 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 51045/332364 objects degraded (15.358%), 12 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:37:59.863 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:37:59.886 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:37:59.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:37:59.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:37:59.955 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:37:59.989 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:00.010 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:00.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:00.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:00.112 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:00.135 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:00.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:00.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:00.237 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:00.260 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:00.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:00.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:00.362 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:00.385 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:00.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:00.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:00.487 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:00.509 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:00.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:00.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:00.611 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:00.633 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:00.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:00.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:00.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:00 smithi102 ceph-mon[110183]: pgmap v772: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 73 active+clean; 6.8 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 100 op/s; 51045/332091 objects degraded (15.371%); 28016/332091 objects misplaced (8.436%); 5.8 MiB/s, 92 objects/s recovering 2024-09-13T14:38:00.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:00 smithi102 ceph-mon[115269]: pgmap v772: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 73 active+clean; 6.8 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 100 op/s; 51045/332091 objects degraded (15.371%); 28016/332091 objects misplaced (8.436%); 5.8 MiB/s, 92 objects/s recovering 2024-09-13T14:38:00.735 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:00.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:00 smithi195 ceph-mon[104262]: pgmap v772: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 73 active+clean; 6.8 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 100 op/s; 51045/332091 objects degraded (15.371%); 28016/332091 objects misplaced (8.436%); 5.8 MiB/s, 92 objects/s recovering 2024-09-13T14:38:00.759 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:00.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:00.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:00.861 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:00.883 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:00.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:00.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:00.984 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:01.009 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:01.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:01.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:01.111 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:01.133 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:01.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:01.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:01.235 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:01.256 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:01.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:01.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:01.358 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:01.380 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:01.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:01.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:01.482 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:01.504 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:01.504 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is set 2024-09-13T14:38:01.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:01.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:01.606 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:01.631 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:01.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:01.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:01.733 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:01.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:01 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2597633086' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:38:01.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:01 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:38:01.755 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:01.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:01.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:01.757 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:38:01.757 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-13T14:38:01.757 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 0 2024-09-13T14:38:01.757 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:38:01.856 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:01.865 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:01 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2597633086' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:38:01.865 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:01 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:38:01.865 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:01 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2597633086' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:38:01.866 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:01 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:38:01.878 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:01.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:01.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:01.980 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:02.002 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:02.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:02.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:02.104 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:02.130 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:02.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:02.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:02.232 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:02.253 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:02.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:02.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:02.309 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:38:02.355 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:02.377 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:02.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:02.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:02.479 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:02.501 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:02.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:02.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:02.512 INFO:teuthology.orchestra.run.smithi102.stderr:reweighted osd.0 to 1 (10000) 2024-09-13T14:38:02.514 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T14:38:02.603 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:02.629 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:02.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:02.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:02.730 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:02.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:02 smithi195 ceph-mon[104262]: pgmap v773: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 73 active+clean; 6.8 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 103 op/s; 51045/331836 objects degraded (15.383%); 27987/331836 objects misplaced (8.434%); 5.8 MiB/s, 92 objects/s recovering 2024-09-13T14:38:02.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:02 smithi195 ceph-mon[104262]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:38:02.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:02 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:38:02.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:02 smithi195 ceph-mon[104262]: osdmap e243: 8 total, 8 up, 6 in 2024-09-13T14:38:02.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:02 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2818046232' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:38:02.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:02 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:38:02.753 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:02.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:02.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:02.766 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:38:02.854 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:02.876 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:02.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:02.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:02.909 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:02 smithi102 ceph-mon[110183]: pgmap v773: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 73 active+clean; 6.8 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 103 op/s; 51045/331836 objects degraded (15.383%); 27987/331836 objects misplaced (8.434%); 5.8 MiB/s, 92 objects/s recovering 2024-09-13T14:38:02.909 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:02 smithi102 ceph-mon[110183]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:38:02.909 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:02 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:38:02.910 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:02 smithi102 ceph-mon[110183]: osdmap e243: 8 total, 8 up, 6 in 2024-09-13T14:38:02.910 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:02 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2818046232' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:38:02.910 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:02 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:38:02.910 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:02 smithi102 ceph-mon[115269]: pgmap v773: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 73 active+clean; 6.8 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 103 op/s; 51045/331836 objects degraded (15.383%); 27987/331836 objects misplaced (8.434%); 5.8 MiB/s, 92 objects/s recovering 2024-09-13T14:38:02.910 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:02 smithi102 ceph-mon[115269]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:38:02.910 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:02 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:38:02.910 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:02 smithi102 ceph-mon[115269]: osdmap e243: 8 total, 8 up, 6 in 2024-09-13T14:38:02.910 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:02 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2818046232' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:38:02.910 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:02 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:38:02.978 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:03.001 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:03.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:03.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:03.102 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:03.124 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:03.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:03.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:03.226 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:03.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:03.249 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:03.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:03.349 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:03.351 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-13T14:38:03.372 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:03.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:03.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:03.474 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:03.495 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:03.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:03.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:03.597 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:03.620 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:03.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:03.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:03.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:03 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T14:38:03.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:03 smithi102 ceph-mon[110183]: osdmap e244: 8 total, 8 up, 6 in 2024-09-13T14:38:03.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:03 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T14:38:03.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:03 smithi102 ceph-mon[115269]: osdmap e244: 8 total, 8 up, 6 in 2024-09-13T14:38:03.722 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:03.744 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:03.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:03.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:03.846 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:03.867 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:03.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:03.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:03.946 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:03 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T14:38:03.946 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:03 smithi195 ceph-mon[104262]: osdmap e244: 8 total, 8 up, 6 in 2024-09-13T14:38:03.969 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:03.991 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:03.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:03.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:04.093 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:04.115 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:04.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:04.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:04.217 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:04.239 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:04.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:04.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:04.263 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:38:04.341 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:04.363 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:04.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:04.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:04.465 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:04.476 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:38:04.476 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-13T14:38:04.476 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-09-13T14:38:04.476 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-09-13T14:38:04.477 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:38:04.486 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:04.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:04.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:04.588 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:04.617 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:04.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:04.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:04.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:04 smithi102 ceph-mon[110183]: pgmap v776: 89 pgs: 6 peering, 1 active+recovering+undersized+remapped, 2 remapped+peering, 1 active+undersized+degraded+remapped+backfill_wait, 9 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 67 active+clean; 6.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 130 op/s; 46518/330609 objects degraded (14.070%); 21766/330609 objects misplaced (6.584%); 1024 KiB/s, 15 objects/s recovering 2024-09-13T14:38:04.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:04 smithi102 ceph-mon[110183]: osdmap e245: 8 total, 8 up, 6 in 2024-09-13T14:38:04.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:04 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 46518/330609 objects degraded (14.070%), 11 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:38:04.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:04 smithi102 ceph-mon[115269]: pgmap v776: 89 pgs: 6 peering, 1 active+recovering+undersized+remapped, 2 remapped+peering, 1 active+undersized+degraded+remapped+backfill_wait, 9 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 67 active+clean; 6.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 130 op/s; 46518/330609 objects degraded (14.070%); 21766/330609 objects misplaced (6.584%); 1024 KiB/s, 15 objects/s recovering 2024-09-13T14:38:04.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:04 smithi102 ceph-mon[115269]: osdmap e245: 8 total, 8 up, 6 in 2024-09-13T14:38:04.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:04 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 46518/330609 objects degraded (14.070%), 11 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:38:04.719 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:04.741 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:04.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:04.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:04.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:04 smithi195 ceph-mon[104262]: pgmap v776: 89 pgs: 6 peering, 1 active+recovering+undersized+remapped, 2 remapped+peering, 1 active+undersized+degraded+remapped+backfill_wait, 9 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 67 active+clean; 6.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 130 op/s; 46518/330609 objects degraded (14.070%); 21766/330609 objects misplaced (6.584%); 1024 KiB/s, 15 objects/s recovering 2024-09-13T14:38:04.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:04 smithi195 ceph-mon[104262]: osdmap e245: 8 total, 8 up, 6 in 2024-09-13T14:38:04.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:04 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 46518/330609 objects degraded (14.070%), 11 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:38:04.843 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:04.865 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:04.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:04.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:04.967 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:04.989 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:04.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:04.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:05.091 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:05.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:05.115 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:05.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:05.170 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T14:38:05.216 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:05.242 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:05.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:05.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:05.344 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:05.367 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:05.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:05.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:05.384 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:38:05.469 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:05.491 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:05.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:05.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:05.593 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:05.615 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:05.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:05.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:05.718 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:05.740 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:05.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:05.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:05.828 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:05 smithi195 ceph-mon[104262]: osdmap e246: 8 total, 8 up, 6 in 2024-09-13T14:38:05.842 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:05.866 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:05.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:05.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:05.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:05 smithi102 ceph-mon[110183]: osdmap e246: 8 total, 8 up, 6 in 2024-09-13T14:38:05.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:05 smithi102 ceph-mon[115269]: osdmap e246: 8 total, 8 up, 6 in 2024-09-13T14:38:05.968 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:05.993 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:05.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:05.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:06.095 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:06.119 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:06.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:06.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:06.221 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:06.244 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:06.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:06.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:06.346 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:06.371 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:06.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:06.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:06.474 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:06.497 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:06.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:06.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:06.529 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:38:06 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:38:06] "GET /metrics HTTP/1.1" 200 36574 "" "Prometheus/2.51.0" 2024-09-13T14:38:06.599 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:06.614 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:06 smithi195 ceph-mon[104262]: pgmap v779: 89 pgs: 6 peering, 1 active+recovering+undersized+remapped, 2 remapped+peering, 1 active+undersized+degraded+remapped+backfill_wait, 9 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 67 active+clean; 6.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 102 op/s; 46518/330609 objects degraded (14.070%); 21766/330609 objects misplaced (6.584%); 496 KiB/s, 7 objects/s recovering 2024-09-13T14:38:06.615 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:06 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/601112229' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:38:06.622 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:06.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:06.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:06.725 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:06.749 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:06.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:06.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:06.851 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:06.874 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:06.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:06.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:06.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:06 smithi102 ceph-mon[110183]: pgmap v779: 89 pgs: 6 peering, 1 active+recovering+undersized+remapped, 2 remapped+peering, 1 active+undersized+degraded+remapped+backfill_wait, 9 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 67 active+clean; 6.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 102 op/s; 46518/330609 objects degraded (14.070%); 21766/330609 objects misplaced (6.584%); 496 KiB/s, 7 objects/s recovering 2024-09-13T14:38:06.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:06 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/601112229' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:38:06.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:06 smithi102 ceph-mon[115269]: pgmap v779: 89 pgs: 6 peering, 1 active+recovering+undersized+remapped, 2 remapped+peering, 1 active+undersized+degraded+remapped+backfill_wait, 9 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 67 active+clean; 6.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 102 op/s; 46518/330609 objects degraded (14.070%); 21766/330609 objects misplaced (6.584%); 496 KiB/s, 7 objects/s recovering 2024-09-13T14:38:06.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:06 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/601112229' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:38:06.976 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:07.000 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:07.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:07.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:07.102 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:07.126 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:07.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:07.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:07.149 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:38:07.149 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-09-13T14:38:07.149 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-09-13T14:38:07.149 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-13T14:38:07.150 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.150 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:227896 5.d 5:b81995d9:::benchmark_data_smithi102_155989_object109978:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-13T14:38:07.150 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:33.751073+0000", 2024-09-13T14:38:07.150 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 453.39562663800001, 2024-09-13T14:38:07.150 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0751938379999999, 2024-09-13T14:38:07.150 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:38:07.150 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-13T14:38:07.150 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:38:07.151 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:38:07.151 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:38:07.151 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 227896 2024-09-13T14:38:07.151 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.151 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:38:07.151 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.151 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:38:07.151 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.751073+0000", 2024-09-13T14:38:07.151 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:38:07.151 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.151 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.151 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:38:07.151 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.751073+0000", 2024-09-13T14:38:07.152 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:38:07.152 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.152 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.152 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:38:07.152 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.751071+0000", 2024-09-13T14:38:07.152 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:38:07.152 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.152 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.152 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:38:07.152 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.751077+0000", 2024-09-13T14:38:07.152 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.818e-06 2024-09-13T14:38:07.152 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.152 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.152 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:38:07.153 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.751089+0000", 2024-09-13T14:38:07.153 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1321000000000001e-05 2024-09-13T14:38:07.153 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.153 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.153 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:38:07.153 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.751095+0000", 2024-09-13T14:38:07.153 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0599999999999996e-06 2024-09-13T14:38:07.153 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.153 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.153 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:38:07.153 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.758089+0000", 2024-09-13T14:38:07.153 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0069944409999999997 2024-09-13T14:38:07.153 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.154 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.154 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:38:07.154 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.758217+0000", 2024-09-13T14:38:07.154 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012764699999999999 2024-09-13T14:38:07.154 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.154 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.154 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:38:07.154 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.758352+0000", 2024-09-13T14:38:07.154 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013484 2024-09-13T14:38:07.154 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.154 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.154 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:38:07.154 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.762219+0000", 2024-09-13T14:38:07.154 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0038677770000000002 2024-09-13T14:38:07.155 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.155 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.155 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:38:07.155 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.762712+0000", 2024-09-13T14:38:07.155 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00049232300000000002 2024-09-13T14:38:07.155 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.155 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.155 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:38:07.155 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.791924+0000", 2024-09-13T14:38:07.155 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.029212175999999999 2024-09-13T14:38:07.155 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.155 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.155 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:38:07.155 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.791937+0000", 2024-09-13T14:38:07.156 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3121e-05 2024-09-13T14:38:07.156 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.156 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.156 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:38:07.156 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.826186+0000", 2024-09-13T14:38:07.156 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0342485370000001 2024-09-13T14:38:07.156 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.156 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.156 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:38:07.156 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.826267+0000", 2024-09-13T14:38:07.156 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1280999999999996e-05 2024-09-13T14:38:07.156 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:38:07.156 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:38:07.157 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:38:07.157 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.157 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.157 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:227905 5.d 5:b4feef4a:::benchmark_data_smithi102_155989_object109987:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-13T14:38:07.157 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:33.762563+0000", 2024-09-13T14:38:07.157 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 453.384136453, 2024-09-13T14:38:07.157 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.064023068, 2024-09-13T14:38:07.157 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:38:07.157 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-13T14:38:07.157 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:38:07.158 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:38:07.158 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:38:07.158 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 227905 2024-09-13T14:38:07.158 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.159 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:38:07.159 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.159 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:38:07.159 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.762563+0000", 2024-09-13T14:38:07.159 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:38:07.159 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.160 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.160 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:38:07.160 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.762563+0000", 2024-09-13T14:38:07.160 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:38:07.160 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.160 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.160 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:38:07.161 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.762562+0000", 2024-09-13T14:38:07.161 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:38:07.161 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.161 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.161 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:38:07.161 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.762569+0000", 2024-09-13T14:38:07.161 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5870000000000004e-06 2024-09-13T14:38:07.162 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.162 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.162 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:38:07.162 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.762582+0000", 2024-09-13T14:38:07.162 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2828e-05 2024-09-13T14:38:07.162 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.162 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.162 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:38:07.163 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.762590+0000", 2024-09-13T14:38:07.163 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7989999999999994e-06 2024-09-13T14:38:07.163 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.163 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.163 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:38:07.163 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.762762+0000", 2024-09-13T14:38:07.164 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00017276699999999999 2024-09-13T14:38:07.164 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.164 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.164 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:38:07.164 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.762887+0000", 2024-09-13T14:38:07.164 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012444299999999999 2024-09-13T14:38:07.267 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.267 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.267 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:38:07.267 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.763018+0000", 2024-09-13T14:38:07.267 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013129999999999999 2024-09-13T14:38:07.267 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.267 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.268 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:38:07.268 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.766939+0000", 2024-09-13T14:38:07.268 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0039210190000000004 2024-09-13T14:38:07.268 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.268 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.268 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:38:07.268 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.775873+0000", 2024-09-13T14:38:07.268 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0089337670000000004 2024-09-13T14:38:07.268 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.269 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.269 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:38:07.269 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.791965+0000", 2024-09-13T14:38:07.269 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016091520000000002 2024-09-13T14:38:07.269 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.269 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.269 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:38:07.269 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.791973+0000", 2024-09-13T14:38:07.269 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0539999999999995e-06 2024-09-13T14:38:07.269 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.269 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.269 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:38:07.269 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.826539+0000", 2024-09-13T14:38:07.270 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0345661960000001 2024-09-13T14:38:07.270 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.270 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.270 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:38:07.270 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.826586+0000", 2024-09-13T14:38:07.270 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7432999999999998e-05 2024-09-13T14:38:07.270 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:38:07.270 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:38:07.270 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:38:07.270 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.270 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.270 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:227906 5.d 5:b54f7af9:::benchmark_data_smithi102_155989_object109988:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-13T14:38:07.270 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:33.764422+0000", 2024-09-13T14:38:07.270 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 453.382277821, 2024-09-13T14:38:07.271 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:07.272 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:38:07.272 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:38:07.272 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:38:07.272 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:38:07.272 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:38:07.273 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:38:07.273 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:38:07.273 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:38:07.273 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:38:07.273 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:38:07.274 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:38:07.275 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0622330689999999, 2024-09-13T14:38:07.275 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:38:07.275 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-13T14:38:07.275 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:38:07.275 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:38:07.275 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:38:07.275 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 227906 2024-09-13T14:38:07.275 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.276 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:38:07.276 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.276 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:38:07.276 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.764422+0000", 2024-09-13T14:38:07.276 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:38:07.276 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.276 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.276 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:38:07.276 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.764422+0000", 2024-09-13T14:38:07.276 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:38:07.276 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.276 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.277 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:38:07.277 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.764420+0000", 2024-09-13T14:38:07.277 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:38:07.277 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.277 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.277 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:38:07.277 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.764429+0000", 2024-09-13T14:38:07.277 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7690000000000002e-06 2024-09-13T14:38:07.277 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.277 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.277 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:38:07.277 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.764446+0000", 2024-09-13T14:38:07.278 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7343e-05 2024-09-13T14:38:07.278 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.278 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.278 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:38:07.278 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.764454+0000", 2024-09-13T14:38:07.278 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3930000000000002e-06 2024-09-13T14:38:07.278 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.278 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.278 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:38:07.278 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.766963+0000", 2024-09-13T14:38:07.278 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0025088789999999999 2024-09-13T14:38:07.278 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.278 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.278 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:38:07.279 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.767080+0000", 2024-09-13T14:38:07.279 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011749 2024-09-13T14:38:07.279 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.279 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.279 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:38:07.279 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.767200+0000", 2024-09-13T14:38:07.279 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000120247 2024-09-13T14:38:07.279 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.279 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.279 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:38:07.279 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.771536+0000", 2024-09-13T14:38:07.279 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0043359749999999997 2024-09-13T14:38:07.279 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.279 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.280 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:38:07.280 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.775881+0000", 2024-09-13T14:38:07.280 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0043450520000000003 2024-09-13T14:38:07.280 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.280 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.280 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:38:07.280 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.792001+0000", 2024-09-13T14:38:07.280 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016119228999999999 2024-09-13T14:38:07.280 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.280 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.280 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:38:07.280 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.792009+0000", 2024-09-13T14:38:07.280 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2320000000000001e-06 2024-09-13T14:38:07.281 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.281 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.281 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:38:07.281 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.826613+0000", 2024-09-13T14:38:07.281 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.034603977 2024-09-13T14:38:07.281 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.281 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.281 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:38:07.281 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.826655+0000", 2024-09-13T14:38:07.281 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2023000000000001e-05 2024-09-13T14:38:07.281 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:38:07.281 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:38:07.281 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:38:07.282 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.282 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.282 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:227907 5.d 5:be3572e2:::benchmark_data_smithi102_155989_object109989:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-13T14:38:07.282 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:33.768928+0000", 2024-09-13T14:38:07.282 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 453.377771634, 2024-09-13T14:38:07.282 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.05785475, 2024-09-13T14:38:07.282 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:38:07.282 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-13T14:38:07.282 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:38:07.282 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:38:07.282 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:38:07.282 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 227907 2024-09-13T14:38:07.282 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.282 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:38:07.283 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.283 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:38:07.283 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.768928+0000", 2024-09-13T14:38:07.283 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:38:07.283 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.283 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.283 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:38:07.283 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.768928+0000", 2024-09-13T14:38:07.283 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:38:07.283 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.283 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.283 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:38:07.283 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.768926+0000", 2024-09-13T14:38:07.283 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:38:07.284 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.284 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.284 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:38:07.284 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.768934+0000", 2024-09-13T14:38:07.284 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9389999999999997e-06 2024-09-13T14:38:07.284 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.284 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.284 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:38:07.284 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.768946+0000", 2024-09-13T14:38:07.285 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2251e-05 2024-09-13T14:38:07.285 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.285 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.285 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:38:07.285 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.768954+0000", 2024-09-13T14:38:07.285 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8019999999999992e-06 2024-09-13T14:38:07.285 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.285 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.285 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:38:07.285 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.771557+0000", 2024-09-13T14:38:07.285 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002602942 2024-09-13T14:38:07.285 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.285 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.285 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:38:07.286 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.771676+0000", 2024-09-13T14:38:07.286 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011906200000000001 2024-09-13T14:38:07.286 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.286 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.286 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:38:07.286 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.771793+0000", 2024-09-13T14:38:07.286 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011686800000000001 2024-09-13T14:38:07.286 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.286 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.286 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:38:07.286 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.782402+0000", 2024-09-13T14:38:07.286 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010609034999999999 2024-09-13T14:38:07.286 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.286 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.287 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:38:07.287 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.782881+0000", 2024-09-13T14:38:07.287 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00047899999999999999 2024-09-13T14:38:07.287 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.287 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.287 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:38:07.287 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.792131+0000", 2024-09-13T14:38:07.287 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0092503280000000004 2024-09-13T14:38:07.287 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.287 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.287 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:38:07.287 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.792142+0000", 2024-09-13T14:38:07.287 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0261999999999999e-05 2024-09-13T14:38:07.287 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.288 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.288 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:38:07.288 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.826703+0000", 2024-09-13T14:38:07.288 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0345612500000001 2024-09-13T14:38:07.288 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.288 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.288 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:38:07.288 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.826783+0000", 2024-09-13T14:38:07.288 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9729999999999997e-05 2024-09-13T14:38:07.288 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:38:07.288 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:38:07.288 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:38:07.288 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.288 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.289 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:227932 5.d 5:b1de39af:::benchmark_data_smithi102_155989_object110014:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-09-13T14:38:07.289 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:33.813605+0000", 2024-09-13T14:38:07.289 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 453.333094211, 2024-09-13T14:38:07.289 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.028329944, 2024-09-13T14:38:07.289 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:38:07.289 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:38:07.289 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:38:07.289 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:38:07.289 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:38:07.289 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 227932 2024-09-13T14:38:07.290 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.290 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:38:07.290 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.290 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:38:07.290 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.813605+0000", 2024-09-13T14:38:07.290 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:38:07.290 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.290 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.290 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:38:07.290 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.813605+0000", 2024-09-13T14:38:07.290 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:38:07.290 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.291 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.291 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:38:07.291 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.813604+0000", 2024-09-13T14:38:07.291 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:38:07.291 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.291 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.291 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:38:07.291 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.813610+0000", 2024-09-13T14:38:07.291 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1709999999999999e-06 2024-09-13T14:38:07.291 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.291 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.291 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:38:07.291 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.813621+0000", 2024-09-13T14:38:07.291 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0560000000000001e-05 2024-09-13T14:38:07.292 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.292 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.292 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:38:07.292 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.813629+0000", 2024-09-13T14:38:07.292 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8490000000000006e-06 2024-09-13T14:38:07.292 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.292 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.292 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:38:07.292 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.813961+0000", 2024-09-13T14:38:07.292 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00033266799999999999 2024-09-13T14:38:07.292 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.292 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.292 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:38:07.293 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.813971+0000", 2024-09-13T14:38:07.293 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4099999999999997e-06 2024-09-13T14:38:07.293 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.293 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.293 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:38:07.293 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.826810+0000", 2024-09-13T14:38:07.293 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0128397709999999 2024-09-13T14:38:07.293 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.293 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.293 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:38:07.293 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.826931+0000", 2024-09-13T14:38:07.293 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000120252 2024-09-13T14:38:07.293 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.293 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.294 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:38:07.294 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.827056+0000", 2024-09-13T14:38:07.294 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012539199999999999 2024-09-13T14:38:07.294 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.294 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.294 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:38:07.294 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.841785+0000", 2024-09-13T14:38:07.294 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014728890999999999 2024-09-13T14:38:07.294 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.294 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.295 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:38:07.295 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.841794+0000", 2024-09-13T14:38:07.295 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8740000000000001e-06 2024-09-13T14:38:07.295 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.295 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.295 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:38:07.295 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.841838+0000", 2024-09-13T14:38:07.295 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4066999999999997e-05 2024-09-13T14:38:07.295 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.295 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.295 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:38:07.295 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.841887+0000", 2024-09-13T14:38:07.295 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8823000000000003e-05 2024-09-13T14:38:07.296 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.296 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.296 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:38:07.296 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.841935+0000", 2024-09-13T14:38:07.296 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8547000000000001e-05 2024-09-13T14:38:07.296 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:38:07.296 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:38:07.296 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:38:07.296 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.296 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.296 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:227960 5.d 5:b3b8d790:::benchmark_data_smithi102_155989_object110042:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-09-13T14:38:07.296 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:33.860733+0000", 2024-09-13T14:38:07.296 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 453.28596694700002, 2024-09-13T14:38:07.297 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.98902412699999998, 2024-09-13T14:38:07.297 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:38:07.297 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:38:07.297 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:38:07.297 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:38:07.297 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:38:07.297 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 227960 2024-09-13T14:38:07.297 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.297 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:38:07.297 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.297 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:38:07.297 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.860733+0000", 2024-09-13T14:38:07.297 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:38:07.297 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.298 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.298 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:38:07.298 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.860733+0000", 2024-09-13T14:38:07.298 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:38:07.298 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.298 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.298 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:38:07.298 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.860731+0000", 2024-09-13T14:38:07.298 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:38:07.298 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.298 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.298 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:38:07.298 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.860740+0000", 2024-09-13T14:38:07.298 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.2259999999999994e-06 2024-09-13T14:38:07.299 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.299 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.299 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:38:07.299 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.860751+0000", 2024-09-13T14:38:07.299 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0334e-05 2024-09-13T14:38:07.299 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.299 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.299 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:38:07.299 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.860757+0000", 2024-09-13T14:38:07.299 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0990000000000004e-06 2024-09-13T14:38:07.300 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.300 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.300 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:38:07.300 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.860935+0000", 2024-09-13T14:38:07.300 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000178701 2024-09-13T14:38:07.300 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.300 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.300 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:38:07.300 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.860960+0000", 2024-09-13T14:38:07.300 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4587999999999999e-05 2024-09-13T14:38:07.300 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.300 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.300 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:38:07.300 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.830632+0000", 2024-09-13T14:38:07.301 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.96967207399999999 2024-09-13T14:38:07.301 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.301 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.301 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:38:07.301 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.830754+0000", 2024-09-13T14:38:07.301 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012228099999999999 2024-09-13T14:38:07.301 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.301 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.301 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:38:07.301 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.830894+0000", 2024-09-13T14:38:07.301 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013926300000000001 2024-09-13T14:38:07.301 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.301 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.302 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:38:07.302 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.841985+0000", 2024-09-13T14:38:07.302 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011091247 2024-09-13T14:38:07.302 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.302 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.302 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:38:07.302 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.841996+0000", 2024-09-13T14:38:07.302 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0798999999999999e-05 2024-09-13T14:38:07.302 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.302 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.302 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:38:07.302 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.849677+0000", 2024-09-13T14:38:07.302 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0076816180000000003 2024-09-13T14:38:07.302 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.303 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.303 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:38:07.303 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.849722+0000", 2024-09-13T14:38:07.303 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4997000000000003e-05 2024-09-13T14:38:07.303 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.303 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.303 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:38:07.303 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.849757+0000", 2024-09-13T14:38:07.303 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4422999999999999e-05 2024-09-13T14:38:07.303 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:38:07.303 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:38:07.304 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:38:07.304 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.304 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.304 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:227961 5.d 5:ba86b973:::benchmark_data_smithi102_155989_object110043:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-09-13T14:38:07.304 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:33.860768+0000", 2024-09-13T14:38:07.304 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 453.28593117100002, 2024-09-13T14:38:07.304 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.98916236300000004, 2024-09-13T14:38:07.304 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:38:07.304 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:38:07.304 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:38:07.304 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:38:07.304 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:38:07.305 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 227961 2024-09-13T14:38:07.305 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.305 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:38:07.305 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.305 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:38:07.305 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.860768+0000", 2024-09-13T14:38:07.305 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:38:07.305 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.305 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.305 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:38:07.305 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.860768+0000", 2024-09-13T14:38:07.305 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:38:07.305 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.305 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.306 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:38:07.306 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.860768+0000", 2024-09-13T14:38:07.306 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:38:07.306 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.306 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.306 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:38:07.306 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.860770+0000", 2024-09-13T14:38:07.307 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0810000000000001e-06 2024-09-13T14:38:07.308 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.308 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.308 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:38:07.308 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.860775+0000", 2024-09-13T14:38:07.308 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3299999999999998e-06 2024-09-13T14:38:07.308 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.308 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.308 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:38:07.308 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.860780+0000", 2024-09-13T14:38:07.308 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1370000000000004e-06 2024-09-13T14:38:07.308 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.309 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.309 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:38:07.309 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.861060+0000", 2024-09-13T14:38:07.309 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00028065399999999997 2024-09-13T14:38:07.309 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.309 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.309 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:38:07.309 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.861082+0000", 2024-09-13T14:38:07.309 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1608e-05 2024-09-13T14:38:07.309 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.309 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.309 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:38:07.309 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.834509+0000", 2024-09-13T14:38:07.310 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.97342739199999995 2024-09-13T14:38:07.310 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.310 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.310 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:38:07.310 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.834663+0000", 2024-09-13T14:38:07.310 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015392699999999999 2024-09-13T14:38:07.310 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.310 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.310 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:38:07.310 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.834789+0000", 2024-09-13T14:38:07.310 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012604700000000001 2024-09-13T14:38:07.310 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.310 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.310 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:38:07.310 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.842000+0000", 2024-09-13T14:38:07.311 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0072104860000000003 2024-09-13T14:38:07.311 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.311 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.311 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:38:07.311 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.849800+0000", 2024-09-13T14:38:07.311 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0078000250000000004 2024-09-13T14:38:07.311 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.311 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.311 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:38:07.311 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.849842+0000", 2024-09-13T14:38:07.311 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1857000000000003e-05 2024-09-13T14:38:07.311 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.311 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.311 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:38:07.312 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.849891+0000", 2024-09-13T14:38:07.312 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8897000000000003e-05 2024-09-13T14:38:07.312 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.312 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.312 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:38:07.312 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.849931+0000", 2024-09-13T14:38:07.312 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0268999999999997e-05 2024-09-13T14:38:07.312 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:38:07.312 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:38:07.312 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:38:07.312 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.312 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.312 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:227970 5.d 5:be0539d0:::benchmark_data_smithi102_155989_object110052:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-09-13T14:38:07.312 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:33.872593+0000", 2024-09-13T14:38:07.313 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 453.27410652399999, 2024-09-13T14:38:07.313 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.97752432700000003, 2024-09-13T14:38:07.313 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:38:07.313 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:38:07.313 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:38:07.313 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:38:07.313 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:38:07.313 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 227970 2024-09-13T14:38:07.313 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.313 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:38:07.313 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.313 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:38:07.313 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.872593+0000", 2024-09-13T14:38:07.313 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:38:07.314 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.314 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.314 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:38:07.314 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.872593+0000", 2024-09-13T14:38:07.314 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:38:07.314 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.314 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.314 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:38:07.314 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.872592+0000", 2024-09-13T14:38:07.314 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:38:07.314 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.314 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.314 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:38:07.314 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.872597+0000", 2024-09-13T14:38:07.315 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4700000000000004e-06 2024-09-13T14:38:07.315 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.315 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.315 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:38:07.315 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.872605+0000", 2024-09-13T14:38:07.315 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8970000000000008e-06 2024-09-13T14:38:07.315 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.315 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.315 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:38:07.315 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.872611+0000", 2024-09-13T14:38:07.316 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5899999999999996e-06 2024-09-13T14:38:07.316 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.316 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.316 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:38:07.316 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.872668+0000", 2024-09-13T14:38:07.316 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.6808999999999998e-05 2024-09-13T14:38:07.316 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.316 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.316 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:38:07.316 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.872688+0000", 2024-09-13T14:38:07.316 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9898000000000002e-05 2024-09-13T14:38:07.316 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.316 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.317 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:38:07.317 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.838196+0000", 2024-09-13T14:38:07.317 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.96550749599999997 2024-09-13T14:38:07.317 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.317 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.317 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:38:07.317 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.838308+0000", 2024-09-13T14:38:07.317 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000112339 2024-09-13T14:38:07.317 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.317 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.317 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:38:07.318 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.838416+0000", 2024-09-13T14:38:07.318 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000108495 2024-09-13T14:38:07.318 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.318 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.318 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:38:07.318 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.849806+0000", 2024-09-13T14:38:07.318 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011389644000000001 2024-09-13T14:38:07.318 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.318 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.318 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:38:07.318 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.849965+0000", 2024-09-13T14:38:07.318 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000159025 2024-09-13T14:38:07.318 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.318 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.319 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:38:07.319 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.850005+0000", 2024-09-13T14:38:07.319 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9731000000000003e-05 2024-09-13T14:38:07.319 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.319 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.319 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:38:07.319 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.850073+0000", 2024-09-13T14:38:07.319 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8078999999999997e-05 2024-09-13T14:38:07.319 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.319 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.319 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:38:07.319 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.850117+0000", 2024-09-13T14:38:07.319 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.456e-05 2024-09-13T14:38:07.319 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:38:07.319 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:38:07.320 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:38:07.320 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.320 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.320 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:227976 5.d 5:baf0dd50:::benchmark_data_smithi102_155989_object110058:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-09-13T14:38:07.320 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:33.882421+0000", 2024-09-13T14:38:07.320 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 453.26427894300002, 2024-09-13T14:38:07.320 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.96790013699999999, 2024-09-13T14:38:07.320 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:38:07.320 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:38:07.320 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:38:07.320 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:38:07.320 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:38:07.320 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 227976 2024-09-13T14:38:07.320 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.321 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:38:07.321 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.321 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:38:07.321 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.882421+0000", 2024-09-13T14:38:07.321 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:38:07.321 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.321 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.321 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:38:07.321 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.882421+0000", 2024-09-13T14:38:07.321 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:38:07.321 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.321 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.321 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:38:07.321 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.882419+0000", 2024-09-13T14:38:07.322 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:38:07.322 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.322 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.322 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:38:07.322 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.882426+0000", 2024-09-13T14:38:07.322 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8889999999999996e-06 2024-09-13T14:38:07.322 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.322 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.322 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:38:07.322 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.882440+0000", 2024-09-13T14:38:07.322 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3852000000000001e-05 2024-09-13T14:38:07.322 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.322 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.323 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:38:07.323 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.882447+0000", 2024-09-13T14:38:07.323 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4540000000000001e-06 2024-09-13T14:38:07.323 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.323 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.323 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:38:07.323 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.882588+0000", 2024-09-13T14:38:07.323 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000140419 2024-09-13T14:38:07.323 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.323 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.323 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:38:07.324 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.882614+0000", 2024-09-13T14:38:07.324 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.6061999999999999e-05 2024-09-13T14:38:07.324 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.324 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.324 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:38:07.324 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.842047+0000", 2024-09-13T14:38:07.324 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.95943292300000005 2024-09-13T14:38:07.324 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.324 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.324 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:38:07.324 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.842169+0000", 2024-09-13T14:38:07.324 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012222000000000001 2024-09-13T14:38:07.324 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.324 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.325 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:38:07.325 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.842289+0000", 2024-09-13T14:38:07.325 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012015000000000001 2024-09-13T14:38:07.325 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.325 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.325 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:38:07.325 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.849810+0000", 2024-09-13T14:38:07.325 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0075213420000000003 2024-09-13T14:38:07.325 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.325 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.325 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:38:07.325 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.850170+0000", 2024-09-13T14:38:07.325 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00035967200000000001 2024-09-13T14:38:07.325 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.326 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.326 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:38:07.326 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.850229+0000", 2024-09-13T14:38:07.326 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8646000000000001e-05 2024-09-13T14:38:07.326 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.326 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.326 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:38:07.326 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.850291+0000", 2024-09-13T14:38:07.326 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1736000000000007e-05 2024-09-13T14:38:07.326 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.326 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.326 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:38:07.326 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.850321+0000", 2024-09-13T14:38:07.326 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0270999999999999e-05 2024-09-13T14:38:07.327 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:38:07.327 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:38:07.327 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:38:07.327 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.327 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.327 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:227980 5.d 5:bdcdc726:::benchmark_data_smithi102_155989_object110062:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-09-13T14:38:07.327 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:33.888564+0000", 2024-09-13T14:38:07.327 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 453.25813543999999, 2024-09-13T14:38:07.327 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.98116541899999998, 2024-09-13T14:38:07.327 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:38:07.327 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:38:07.327 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:38:07.327 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:38:07.327 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:38:07.327 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 227980 2024-09-13T14:38:07.328 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.328 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:38:07.328 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.328 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:38:07.328 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.888564+0000", 2024-09-13T14:38:07.328 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:38:07.328 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.328 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.328 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:38:07.328 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.888564+0000", 2024-09-13T14:38:07.328 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:38:07.329 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.329 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.329 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:38:07.329 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.888562+0000", 2024-09-13T14:38:07.329 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:38:07.329 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.329 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.329 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:38:07.329 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.888573+0000", 2024-09-13T14:38:07.329 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0964999999999999e-05 2024-09-13T14:38:07.329 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.329 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.329 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:38:07.330 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.888593+0000", 2024-09-13T14:38:07.330 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9604999999999999e-05 2024-09-13T14:38:07.330 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.330 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.330 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:38:07.330 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.888601+0000", 2024-09-13T14:38:07.330 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2239999999999995e-06 2024-09-13T14:38:07.330 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.330 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.330 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:38:07.330 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.888661+0000", 2024-09-13T14:38:07.330 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0241000000000001e-05 2024-09-13T14:38:07.330 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.330 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.330 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:38:07.331 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.888674+0000", 2024-09-13T14:38:07.331 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2904e-05 2024-09-13T14:38:07.331 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.331 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.331 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:38:07.331 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.845769+0000", 2024-09-13T14:38:07.331 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.95709459900000005 2024-09-13T14:38:07.331 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.331 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.331 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:38:07.331 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.845868+0000", 2024-09-13T14:38:07.331 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.8649999999999999e-05 2024-09-13T14:38:07.331 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.331 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.332 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:38:07.332 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.845979+0000", 2024-09-13T14:38:07.332 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011089799999999999 2024-09-13T14:38:07.332 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.332 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.332 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:38:07.332 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.849815+0000", 2024-09-13T14:38:07.332 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0038365529999999999 2024-09-13T14:38:07.332 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.332 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.332 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:38:07.332 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.858462+0000", 2024-09-13T14:38:07.332 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.008646473 2024-09-13T14:38:07.332 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.333 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.333 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:38:07.333 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.869652+0000", 2024-09-13T14:38:07.333 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011190054 2024-09-13T14:38:07.333 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.333 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.333 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:38:07.333 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.869694+0000", 2024-09-13T14:38:07.333 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2077999999999999e-05 2024-09-13T14:38:07.333 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.333 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.333 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:38:07.333 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.869730+0000", 2024-09-13T14:38:07.334 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5876e-05 2024-09-13T14:38:07.334 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:38:07.334 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:38:07.334 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:38:07.334 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.334 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.334 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:227991 5.d 5:bf4fea11:::benchmark_data_smithi102_155989_object110073:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-09-13T14:38:07.334 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:33.906105+0000", 2024-09-13T14:38:07.334 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 453.24059503699999, 2024-09-13T14:38:07.334 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.97247658400000003, 2024-09-13T14:38:07.334 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:38:07.334 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:38:07.334 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:38:07.334 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:38:07.335 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:38:07.335 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 227991 2024-09-13T14:38:07.335 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.335 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:38:07.335 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.335 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:38:07.335 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.906105+0000", 2024-09-13T14:38:07.335 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:38:07.335 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.335 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.335 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:38:07.335 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.906105+0000", 2024-09-13T14:38:07.335 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:38:07.336 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.336 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.336 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:38:07.336 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.906102+0000", 2024-09-13T14:38:07.336 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-09-13T14:38:07.336 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.336 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.336 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:38:07.336 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.906112+0000", 2024-09-13T14:38:07.336 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.3899999999999999e-06 2024-09-13T14:38:07.336 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.336 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.337 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:38:07.337 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.906131+0000", 2024-09-13T14:38:07.337 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9721e-05 2024-09-13T14:38:07.337 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.337 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.337 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:38:07.337 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.906140+0000", 2024-09-13T14:38:07.337 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.9190000000000005e-06 2024-09-13T14:38:07.337 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.337 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.337 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:38:07.338 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.906199+0000", 2024-09-13T14:38:07.338 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8479e-05 2024-09-13T14:38:07.338 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.338 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.338 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:38:07.338 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.906208+0000", 2024-09-13T14:38:07.338 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1670000000000005e-06 2024-09-13T14:38:07.338 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.338 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.338 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:38:07.338 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.850361+0000", 2024-09-13T14:38:07.338 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.94415305500000002 2024-09-13T14:38:07.338 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.338 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.339 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:38:07.339 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.850472+0000", 2024-09-13T14:38:07.339 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011134899999999999 2024-09-13T14:38:07.339 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.339 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.339 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:38:07.339 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.850602+0000", 2024-09-13T14:38:07.339 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012967799999999999 2024-09-13T14:38:07.339 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.339 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.339 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:38:07.339 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.858500+0000", 2024-09-13T14:38:07.339 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0078974240000000001 2024-09-13T14:38:07.339 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.340 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.340 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:38:07.340 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.858507+0000", 2024-09-13T14:38:07.340 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4189999999999996e-06 2024-09-13T14:38:07.340 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.340 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.340 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:38:07.340 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.878498+0000", 2024-09-13T14:38:07.340 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.019990785 2024-09-13T14:38:07.340 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.340 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.340 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:38:07.340 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.878545+0000", 2024-09-13T14:38:07.341 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7302000000000002e-05 2024-09-13T14:38:07.341 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.341 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.341 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:38:07.341 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.878581+0000", 2024-09-13T14:38:07.341 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6074000000000001e-05 2024-09-13T14:38:07.341 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:38:07.341 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:38:07.341 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:38:07.341 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.341 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.341 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:228001 5.d 5:b17ce506:::benchmark_data_smithi102_155989_object110083:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-09-13T14:38:07.341 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:33.921096+0000", 2024-09-13T14:38:07.341 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 453.22560367099999, 2024-09-13T14:38:07.342 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.96293915699999999, 2024-09-13T14:38:07.342 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:38:07.342 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:38:07.342 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:38:07.342 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:38:07.342 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:38:07.342 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 228001 2024-09-13T14:38:07.342 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.342 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:38:07.342 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.342 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:38:07.342 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.921096+0000", 2024-09-13T14:38:07.343 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:38:07.343 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.343 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.343 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:38:07.343 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.921096+0000", 2024-09-13T14:38:07.343 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:38:07.343 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.343 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.343 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:38:07.343 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.921095+0000", 2024-09-13T14:38:07.343 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:38:07.343 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.344 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.344 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:38:07.344 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.921101+0000", 2024-09-13T14:38:07.344 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4389999999999997e-06 2024-09-13T14:38:07.344 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.344 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.344 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:38:07.344 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.921114+0000", 2024-09-13T14:38:07.344 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3173000000000001e-05 2024-09-13T14:38:07.344 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.344 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.344 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:38:07.344 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.921121+0000", 2024-09-13T14:38:07.345 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3430000000000001e-06 2024-09-13T14:38:07.345 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.345 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.345 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:38:07.345 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.921250+0000", 2024-09-13T14:38:07.345 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012937199999999999 2024-09-13T14:38:07.345 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.345 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.345 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:38:07.345 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.921281+0000", 2024-09-13T14:38:07.345 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0942000000000003e-05 2024-09-13T14:38:07.345 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.345 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.345 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:38:07.346 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.854267+0000", 2024-09-13T14:38:07.346 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.93298647899999998 2024-09-13T14:38:07.346 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.346 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.346 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:38:07.346 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.854367+0000", 2024-09-13T14:38:07.346 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.9473000000000001e-05 2024-09-13T14:38:07.346 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.346 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.346 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:38:07.346 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.854478+0000", 2024-09-13T14:38:07.346 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000110863 2024-09-13T14:38:07.346 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.347 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.347 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:38:07.347 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.858513+0000", 2024-09-13T14:38:07.347 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0040354140000000002 2024-09-13T14:38:07.347 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.347 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.347 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:38:07.347 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.858617+0000", 2024-09-13T14:38:07.347 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000103667 2024-09-13T14:38:07.347 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.348 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.348 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:38:07.348 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.883942+0000", 2024-09-13T14:38:07.348 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.025325754999999998 2024-09-13T14:38:07.348 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.348 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.348 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:38:07.348 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.883991+0000", 2024-09-13T14:38:07.348 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8579999999999999e-05 2024-09-13T14:38:07.348 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.348 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.348 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:38:07.348 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.884035+0000", 2024-09-13T14:38:07.348 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4027999999999999e-05 2024-09-13T14:38:07.349 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:38:07.349 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:38:07.349 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:38:07.349 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.349 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.349 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:228031 5.d 5:b8c11733:::benchmark_data_smithi102_155989_object110113:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-09-13T14:38:07.349 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:33.981537+0000", 2024-09-13T14:38:07.349 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 453.16516255599998, 2024-09-13T14:38:07.349 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.91229017300000004, 2024-09-13T14:38:07.349 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:38:07.349 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:38:07.349 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:38:07.349 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:38:07.349 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:38:07.350 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 228031 2024-09-13T14:38:07.350 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.350 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:38:07.350 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.350 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:38:07.350 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.981537+0000", 2024-09-13T14:38:07.350 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:38:07.350 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.350 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.350 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:38:07.350 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.981537+0000", 2024-09-13T14:38:07.350 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:38:07.350 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.350 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.350 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:38:07.351 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.981535+0000", 2024-09-13T14:38:07.351 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-09-13T14:38:07.351 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.351 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.351 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:38:07.351 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.981545+0000", 2024-09-13T14:38:07.351 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0327e-05 2024-09-13T14:38:07.351 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.351 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.351 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:38:07.351 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.981570+0000", 2024-09-13T14:38:07.351 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4828e-05 2024-09-13T14:38:07.351 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.351 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.352 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:38:07.352 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.981580+0000", 2024-09-13T14:38:07.352 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0069e-05 2024-09-13T14:38:07.352 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.352 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.352 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:38:07.352 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.981646+0000", 2024-09-13T14:38:07.352 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5489999999999998e-05 2024-09-13T14:38:07.352 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.352 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.352 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:38:07.353 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:33.981660+0000", 2024-09-13T14:38:07.353 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4430000000000001e-05 2024-09-13T14:38:07.353 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.353 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.353 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:38:07.353 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.858663+0000", 2024-09-13T14:38:07.353 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.87700307899999996 2024-09-13T14:38:07.353 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.353 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.353 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:38:07.353 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.858778+0000", 2024-09-13T14:38:07.353 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000115408 2024-09-13T14:38:07.354 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.354 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.354 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:38:07.354 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.858894+0000", 2024-09-13T14:38:07.354 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000115122 2024-09-13T14:38:07.354 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.354 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.354 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:38:07.354 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.869617+0000", 2024-09-13T14:38:07.354 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010723425999999999 2024-09-13T14:38:07.354 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.354 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.354 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:38:07.355 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.869768+0000", 2024-09-13T14:38:07.355 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000150598 2024-09-13T14:38:07.355 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.355 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.355 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:38:07.355 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.893709+0000", 2024-09-13T14:38:07.355 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.023941535999999999 2024-09-13T14:38:07.355 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.355 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.355 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:38:07.355 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.893783+0000", 2024-09-13T14:38:07.355 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3767e-05 2024-09-13T14:38:07.355 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.355 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.355 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:38:07.356 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.893827+0000", 2024-09-13T14:38:07.356 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4342000000000003e-05 2024-09-13T14:38:07.356 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:38:07.356 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:38:07.356 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:38:07.356 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.356 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.356 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:228049 5.d 5:b5b2849e:::benchmark_data_smithi102_155989_object110131:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-09-13T14:38:07.356 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:34.031254+0000", 2024-09-13T14:38:07.356 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 453.11544577500001, 2024-09-13T14:38:07.356 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.871372916, 2024-09-13T14:38:07.356 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:38:07.356 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:38:07.357 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:38:07.357 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:38:07.357 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:38:07.357 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 228049 2024-09-13T14:38:07.357 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.357 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:38:07.357 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.357 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:38:07.357 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.031254+0000", 2024-09-13T14:38:07.357 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:38:07.357 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.357 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.357 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:38:07.357 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.031254+0000", 2024-09-13T14:38:07.357 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:38:07.358 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.358 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.358 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:38:07.358 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.031253+0000", 2024-09-13T14:38:07.358 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:38:07.358 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.358 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.358 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:38:07.358 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.031258+0000", 2024-09-13T14:38:07.358 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.6559999999999997e-06 2024-09-13T14:38:07.358 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.358 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.358 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:38:07.358 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.031269+0000", 2024-09-13T14:38:07.359 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0147e-05 2024-09-13T14:38:07.359 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.359 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.359 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:38:07.359 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.031275+0000", 2024-09-13T14:38:07.359 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1800000000000001e-06 2024-09-13T14:38:07.359 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.359 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.359 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:38:07.359 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.031346+0000", 2024-09-13T14:38:07.359 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1205999999999996e-05 2024-09-13T14:38:07.359 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.360 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.360 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:38:07.360 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.031359+0000", 2024-09-13T14:38:07.360 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2853999999999999e-05 2024-09-13T14:38:07.360 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.360 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.360 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:38:07.360 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.862342+0000", 2024-09-13T14:38:07.360 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.83098279200000003 2024-09-13T14:38:07.360 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.360 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.361 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:38:07.361 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.862593+0000", 2024-09-13T14:38:07.361 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00025126 2024-09-13T14:38:07.361 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.361 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.361 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:38:07.361 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.862696+0000", 2024-09-13T14:38:07.361 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010274500000000001 2024-09-13T14:38:07.361 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.361 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.361 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:38:07.361 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.869738+0000", 2024-09-13T14:38:07.361 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0070420980000000001 2024-09-13T14:38:07.361 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.362 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.362 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:38:07.362 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.869796+0000", 2024-09-13T14:38:07.362 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8091000000000003e-05 2024-09-13T14:38:07.362 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.362 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.362 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:38:07.362 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.902543+0000", 2024-09-13T14:38:07.362 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.03274726 2024-09-13T14:38:07.362 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.362 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.362 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:38:07.362 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.902591+0000", 2024-09-13T14:38:07.362 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7809999999999998e-05 2024-09-13T14:38:07.363 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.363 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.363 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:38:07.363 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.902627+0000", 2024-09-13T14:38:07.363 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5815999999999997e-05 2024-09-13T14:38:07.363 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:38:07.363 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:38:07.363 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:38:07.363 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.363 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.363 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:228051 5.d 5:be5f511d:::benchmark_data_smithi102_155989_object110133:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-09-13T14:38:07.363 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:34.037246+0000", 2024-09-13T14:38:07.363 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 453.10945358399999, 2024-09-13T14:38:07.363 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.86547465199999996, 2024-09-13T14:38:07.364 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:38:07.364 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:38:07.364 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:38:07.364 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:38:07.364 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:38:07.364 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 228051 2024-09-13T14:38:07.364 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.364 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:38:07.364 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.364 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:38:07.364 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.037246+0000", 2024-09-13T14:38:07.364 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:38:07.364 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.364 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.365 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:38:07.365 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.037246+0000", 2024-09-13T14:38:07.365 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:38:07.365 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.365 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.365 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:38:07.365 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.037245+0000", 2024-09-13T14:38:07.365 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:38:07.365 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.365 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.365 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:38:07.365 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.037251+0000", 2024-09-13T14:38:07.365 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1600000000000003e-06 2024-09-13T14:38:07.365 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.366 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.366 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:38:07.366 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.037263+0000", 2024-09-13T14:38:07.366 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1881e-05 2024-09-13T14:38:07.366 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.366 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.366 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:38:07.366 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.037271+0000", 2024-09-13T14:38:07.366 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3680000000000002e-06 2024-09-13T14:38:07.366 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.366 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.367 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:38:07.367 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.037322+0000", 2024-09-13T14:38:07.367 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1171000000000002e-05 2024-09-13T14:38:07.367 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.367 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.367 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:38:07.367 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.037333+0000", 2024-09-13T14:38:07.367 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0463e-05 2024-09-13T14:38:07.367 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.367 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.367 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:38:07.367 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.866056+0000", 2024-09-13T14:38:07.367 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.82872323199999998 2024-09-13T14:38:07.367 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.368 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.368 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:38:07.368 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.866158+0000", 2024-09-13T14:38:07.368 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010219500000000001 2024-09-13T14:38:07.368 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.368 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.368 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:38:07.368 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.866273+0000", 2024-09-13T14:38:07.368 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011437700000000001 2024-09-13T14:38:07.368 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.368 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.368 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:38:07.368 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.873381+0000", 2024-09-13T14:38:07.369 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0071080099999999997 2024-09-13T14:38:07.369 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.369 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.369 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:38:07.369 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.876997+0000", 2024-09-13T14:38:07.369 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0036160010000000002 2024-09-13T14:38:07.369 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.369 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.369 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:38:07.369 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.902661+0000", 2024-09-13T14:38:07.369 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.025664585 2024-09-13T14:38:07.369 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.369 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.369 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:38:07.370 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.902692+0000", 2024-09-13T14:38:07.370 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.1047000000000003e-05 2024-09-13T14:38:07.370 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.370 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.370 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:38:07.370 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.902721+0000", 2024-09-13T14:38:07.370 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8388999999999999e-05 2024-09-13T14:38:07.370 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:38:07.370 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:38:07.370 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:38:07.370 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.370 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.371 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.44546.0:228072 5.d 5:bb5a3f8e:::benchmark_data_smithi102_155989_object110154:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-09-13T14:38:07.371 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:30:34.139716+0000", 2024-09-13T14:38:07.371 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 453.00698341999998, 2024-09-13T14:38:07.371 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.76309005699999999, 2024-09-13T14:38:07.371 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:38:07.371 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:38:07.371 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:38:07.371 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.44546", 2024-09-13T14:38:07.371 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3171604237", 2024-09-13T14:38:07.371 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 228072 2024-09-13T14:38:07.371 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.371 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:38:07.371 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.371 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:38:07.372 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.139716+0000", 2024-09-13T14:38:07.372 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:38:07.372 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.372 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.372 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:38:07.372 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.139716+0000", 2024-09-13T14:38:07.372 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:38:07.372 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.372 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.372 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:38:07.372 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.139714+0000", 2024-09-13T14:38:07.372 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-09-13T14:38:07.372 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.372 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.373 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:38:07.373 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.139725+0000", 2024-09-13T14:38:07.373 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0658e-05 2024-09-13T14:38:07.373 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.373 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.373 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:38:07.373 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.139745+0000", 2024-09-13T14:38:07.373 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0801999999999998e-05 2024-09-13T14:38:07.373 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.373 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.373 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:38:07.373 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.139755+0000", 2024-09-13T14:38:07.373 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4690000000000003e-06 2024-09-13T14:38:07.373 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.374 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.374 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:38:07.374 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.139821+0000", 2024-09-13T14:38:07.374 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5639e-05 2024-09-13T14:38:07.374 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.374 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.374 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:38:07.374 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.139834+0000", 2024-09-13T14:38:07.374 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3393999999999999e-05 2024-09-13T14:38:07.374 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.374 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.374 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:38:07.374 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.869818+0000", 2024-09-13T14:38:07.374 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.72998423099999998 2024-09-13T14:38:07.375 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.375 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.375 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:38:07.375 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.869919+0000", 2024-09-13T14:38:07.375 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000101116 2024-09-13T14:38:07.375 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.375 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.375 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:38:07.375 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.870020+0000", 2024-09-13T14:38:07.375 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000100689 2024-09-13T14:38:07.376 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.376 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.376 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:38:07.376 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.877004+0000", 2024-09-13T14:38:07.376 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0069837550000000003 2024-09-13T14:38:07.376 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.376 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.376 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:38:07.376 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.884289+0000", 2024-09-13T14:38:07.376 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0072851160000000003 2024-09-13T14:38:07.376 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.376 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.377 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:38:07.377 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.902746+0000", 2024-09-13T14:38:07.377 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.01845757 2024-09-13T14:38:07.377 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.377 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.377 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:38:07.377 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.902776+0000", 2024-09-13T14:38:07.377 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9428e-05 2024-09-13T14:38:07.377 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.377 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.377 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:38:07.377 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:30:34.902806+0000", 2024-09-13T14:38:07.377 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0366e-05 2024-09-13T14:38:07.377 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:38:07.378 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:38:07.378 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:38:07.378 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.378 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.378 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:198014 6.3 6:c6a15bb2:::benchmark_data_smithi102_194590_object87390:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-09-13T14:38:07.378 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:34:29.666548+0000", 2024-09-13T14:38:07.378 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 217.48015165000001, 2024-09-13T14:38:07.378 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.24791018200000001, 2024-09-13T14:38:07.378 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:38:07.378 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:38:07.378 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:38:07.378 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:38:07.378 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:38:07.378 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 198014 2024-09-13T14:38:07.379 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.379 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:38:07.379 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.379 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:38:07.379 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.666548+0000", 2024-09-13T14:38:07.379 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:38:07.379 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.379 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.379 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:38:07.379 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.666548+0000", 2024-09-13T14:38:07.379 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:38:07.379 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.379 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.380 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:38:07.380 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.666547+0000", 2024-09-13T14:38:07.380 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:38:07.380 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.380 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.380 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:38:07.380 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.666551+0000", 2024-09-13T14:38:07.380 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.1499999999999999e-06 2024-09-13T14:38:07.381 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.381 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.381 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:38:07.381 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.666560+0000", 2024-09-13T14:38:07.381 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.8630000000000003e-06 2024-09-13T14:38:07.381 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.381 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.381 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:38:07.381 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.666566+0000", 2024-09-13T14:38:07.381 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7409999999999998e-06 2024-09-13T14:38:07.381 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.381 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.381 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:38:07.381 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.677707+0000", 2024-09-13T14:38:07.382 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011140399 2024-09-13T14:38:07.382 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.382 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.382 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:38:07.382 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.677829+0000", 2024-09-13T14:38:07.382 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000122621 2024-09-13T14:38:07.382 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.382 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.382 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,6", 2024-09-13T14:38:07.382 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.677950+0000", 2024-09-13T14:38:07.382 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000120926 2024-09-13T14:38:07.382 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.382 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.382 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:38:07.383 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.694719+0000", 2024-09-13T14:38:07.383 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016769071 2024-09-13T14:38:07.383 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.383 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.383 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:38:07.383 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.694726+0000", 2024-09-13T14:38:07.383 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7870000000000004e-06 2024-09-13T14:38:07.383 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.383 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.383 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:38:07.383 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.914382+0000", 2024-09-13T14:38:07.383 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.219656197 2024-09-13T14:38:07.383 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.383 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.384 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:38:07.384 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.914419+0000", 2024-09-13T14:38:07.384 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6971000000000002e-05 2024-09-13T14:38:07.384 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.384 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.384 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:38:07.384 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.914458+0000", 2024-09-13T14:38:07.384 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9035000000000001e-05 2024-09-13T14:38:07.384 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:38:07.384 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:38:07.385 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:38:07.385 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.385 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.385 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:198032 6.3 6:cf2708b1:::benchmark_data_smithi102_194590_object87408:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-09-13T14:38:07.385 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:34:29.686356+0000", 2024-09-13T14:38:07.385 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 217.460343292, 2024-09-13T14:38:07.385 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.234390241, 2024-09-13T14:38:07.385 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:38:07.385 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:38:07.385 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:38:07.385 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:38:07.385 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:38:07.386 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 198032 2024-09-13T14:38:07.386 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.386 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:38:07.386 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.386 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:38:07.386 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.686356+0000", 2024-09-13T14:38:07.386 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:38:07.386 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.386 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.386 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:38:07.386 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.686356+0000", 2024-09-13T14:38:07.386 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:38:07.386 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.386 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.387 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:38:07.387 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.686355+0000", 2024-09-13T14:38:07.387 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:38:07.387 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.387 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.387 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:38:07.387 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.686360+0000", 2024-09-13T14:38:07.387 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1070000000000004e-06 2024-09-13T14:38:07.387 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.387 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.388 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:38:07.388 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.686374+0000", 2024-09-13T14:38:07.388 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3878e-05 2024-09-13T14:38:07.388 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.388 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.388 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:38:07.388 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.686381+0000", 2024-09-13T14:38:07.388 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8510000000000001e-06 2024-09-13T14:38:07.388 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.388 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.388 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:38:07.388 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.688802+0000", 2024-09-13T14:38:07.388 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0024206000000000002 2024-09-13T14:38:07.388 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.389 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.389 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:38:07.389 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.688928+0000", 2024-09-13T14:38:07.389 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012623499999999999 2024-09-13T14:38:07.389 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.389 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.389 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,6", 2024-09-13T14:38:07.389 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.690008+0000", 2024-09-13T14:38:07.389 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001080714 2024-09-13T14:38:07.389 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.389 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.389 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:38:07.389 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.694768+0000", 2024-09-13T14:38:07.389 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0047593569999999997 2024-09-13T14:38:07.390 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.390 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.390 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:38:07.390 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.695055+0000", 2024-09-13T14:38:07.390 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00028694099999999998 2024-09-13T14:38:07.390 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.390 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.390 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:38:07.390 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.920646+0000", 2024-09-13T14:38:07.390 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.225591033 2024-09-13T14:38:07.390 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.390 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.390 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:38:07.390 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.920712+0000", 2024-09-13T14:38:07.391 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5730999999999998e-05 2024-09-13T14:38:07.391 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.391 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.391 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:38:07.391 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.920747+0000", 2024-09-13T14:38:07.391 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5027000000000002e-05 2024-09-13T14:38:07.391 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:38:07.391 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:38:07.391 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:38:07.391 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.391 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.391 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113978 7.5 7:a1bc8a89:::benchmark_data_smithi102_227395_object1163:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-13T14:38:07.391 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:47.323151+0000", 2024-09-13T14:38:07.391 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 19.823548827, 2024-09-13T14:38:07.392 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.366077738, 2024-09-13T14:38:07.392 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:38:07.392 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:38:07.392 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:38:07.392 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:38:07.392 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:38:07.392 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113978 2024-09-13T14:38:07.392 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.392 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:38:07.392 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.392 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:38:07.392 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.323151+0000", 2024-09-13T14:38:07.393 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:38:07.393 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.393 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.393 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:38:07.393 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.323151+0000", 2024-09-13T14:38:07.393 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:38:07.393 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.393 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.393 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:38:07.393 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.323149+0000", 2024-09-13T14:38:07.393 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:38:07.393 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.394 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.394 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:38:07.394 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.323159+0000", 2024-09-13T14:38:07.394 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.7049999999999994e-06 2024-09-13T14:38:07.394 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.394 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.394 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:38:07.394 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.323176+0000", 2024-09-13T14:38:07.394 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7649999999999999e-05 2024-09-13T14:38:07.394 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.394 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.394 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:38:07.394 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.323184+0000", 2024-09-13T14:38:07.394 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7579999999999992e-06 2024-09-13T14:38:07.395 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.395 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.395 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:38:07.395 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.323232+0000", 2024-09-13T14:38:07.395 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8402999999999997e-05 2024-09-13T14:38:07.395 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.395 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.395 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:38:07.395 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.323241+0000", 2024-09-13T14:38:07.395 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1990000000000006e-06 2024-09-13T14:38:07.395 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.395 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.395 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:38:07.395 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.120037+0000", 2024-09-13T14:38:07.396 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.79679608999999996 2024-09-13T14:38:07.396 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.396 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.396 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:38:07.396 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.120050+0000", 2024-09-13T14:38:07.396 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2903e-05 2024-09-13T14:38:07.396 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.396 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.396 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:38:07.396 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.628936+0000", 2024-09-13T14:38:07.396 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.508886086 2024-09-13T14:38:07.397 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.397 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.397 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:38:07.397 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.629091+0000", 2024-09-13T14:38:07.397 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015541700000000001 2024-09-13T14:38:07.397 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.397 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.397 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:38:07.397 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.629244+0000", 2024-09-13T14:38:07.397 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015319199999999999 2024-09-13T14:38:07.397 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.397 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.397 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:38:07.398 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.635107+0000", 2024-09-13T14:38:07.398 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0058630619999999996 2024-09-13T14:38:07.398 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.398 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.398 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:38:07.398 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.643339+0000", 2024-09-13T14:38:07.398 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0082320020000000004 2024-09-13T14:38:07.398 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.398 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.398 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:38:07.398 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.685466+0000", 2024-09-13T14:38:07.398 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.042126602999999999 2024-09-13T14:38:07.398 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.398 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.399 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:38:07.399 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.685526+0000", 2024-09-13T14:38:07.399 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0146999999999999e-05 2024-09-13T14:38:07.399 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.399 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.399 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:38:07.399 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.689229+0000", 2024-09-13T14:38:07.399 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0037024200000000001 2024-09-13T14:38:07.399 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:38:07.399 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:38:07.399 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:38:07.399 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.399 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.399 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113996 7.5 7:a0c0436d:::benchmark_data_smithi102_227395_object1181:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-13T14:38:07.400 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:48.567765+0000", 2024-09-13T14:38:07.400 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 18.578934774, 2024-09-13T14:38:07.400 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1252143800000001, 2024-09-13T14:38:07.400 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:38:07.400 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:38:07.400 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:38:07.400 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:38:07.400 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:38:07.400 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113996 2024-09-13T14:38:07.400 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.400 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:38:07.400 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.400 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:38:07.400 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.567765+0000", 2024-09-13T14:38:07.401 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:38:07.401 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.401 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.401 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:38:07.401 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.567765+0000", 2024-09-13T14:38:07.401 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:38:07.401 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.401 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.401 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:38:07.401 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.567763+0000", 2024-09-13T14:38:07.401 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-09-13T14:38:07.401 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.401 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.401 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:38:07.402 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.567773+0000", 2024-09-13T14:38:07.402 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0569e-05 2024-09-13T14:38:07.402 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.402 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.402 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:38:07.402 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.567795+0000", 2024-09-13T14:38:07.402 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1739999999999999e-05 2024-09-13T14:38:07.402 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.402 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.402 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:38:07.402 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.567805+0000", 2024-09-13T14:38:07.402 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.8109999999999997e-06 2024-09-13T14:38:07.402 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.402 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.403 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:38:07.403 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.567869+0000", 2024-09-13T14:38:07.403 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4679999999999997e-05 2024-09-13T14:38:07.403 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.403 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.403 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:38:07.403 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.567883+0000", 2024-09-13T14:38:07.403 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3978999999999999e-05 2024-09-13T14:38:07.403 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.403 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.403 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:38:07.403 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.635139+0000", 2024-09-13T14:38:07.403 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0672552609999999 2024-09-13T14:38:07.403 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.404 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.404 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:38:07.404 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.635297+0000", 2024-09-13T14:38:07.404 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000158488 2024-09-13T14:38:07.404 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.404 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.404 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:38:07.404 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.635456+0000", 2024-09-13T14:38:07.404 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015900899999999999 2024-09-13T14:38:07.404 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.404 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.404 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:38:07.404 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.639619+0000", 2024-09-13T14:38:07.404 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0041628070000000001 2024-09-13T14:38:07.405 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.405 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.405 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:38:07.405 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.643415+0000", 2024-09-13T14:38:07.405 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003795625 2024-09-13T14:38:07.405 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.405 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.405 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:38:07.405 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.689289+0000", 2024-09-13T14:38:07.405 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.045873979000000002 2024-09-13T14:38:07.405 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.405 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.405 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:38:07.405 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.689334+0000", 2024-09-13T14:38:07.406 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5918000000000003e-05 2024-09-13T14:38:07.406 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:38:07.406 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:38:07.406 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:38:07.406 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.692979+0000", 2024-09-13T14:38:07.406 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0036447760000000002 2024-09-13T14:38:07.406 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:38:07.406 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:38:07.406 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:38:07.407 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:38:07.407 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:38:07.407 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:38:07.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:07.408 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:07.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:07.409 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:38:07.512 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:07.533 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:07.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:07.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:07.635 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:07.657 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:07.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:07.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:07.759 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:07.782 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:07.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:07.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:07.884 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:07.909 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:07.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:07.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:08.011 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:08.032 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:08.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:08.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:08.134 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:08.162 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:08.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:08.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:08.235 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T14:38:08.264 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:08.299 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:08.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:08.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:08.402 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:08.425 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:08.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:08.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:08.451 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:38:08.526 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:08.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:08.549 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:08.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:08.650 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:08.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:08 smithi102 ceph-mon[115269]: pgmap v780: 89 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 6 peering, 1 active+undersized+degraded+remapped+backfill_wait, 2 remapped+peering, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 67 active+clean; 6.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 68 op/s; 47404/330609 objects degraded (14.338%); 14842/330609 objects misplaced (4.489%); 4.0 MiB/s, 65 objects/s recovering 2024-09-13T14:38:08.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:08 smithi102 ceph-mon[110183]: pgmap v780: 89 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 6 peering, 1 active+undersized+degraded+remapped+backfill_wait, 2 remapped+peering, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 67 active+clean; 6.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 68 op/s; 47404/330609 objects degraded (14.338%); 14842/330609 objects misplaced (4.489%); 4.0 MiB/s, 65 objects/s recovering 2024-09-13T14:38:08.676 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:08.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:08.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:08.778 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:08.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:08.805 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:08.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:08.906 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:08.914 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:08 smithi195 ceph-mon[104262]: pgmap v780: 89 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 6 peering, 1 active+undersized+degraded+remapped+backfill_wait, 2 remapped+peering, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 67 active+clean; 6.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 68 op/s; 47404/330609 objects degraded (14.338%); 14842/330609 objects misplaced (4.489%); 4.0 MiB/s, 65 objects/s recovering 2024-09-13T14:38:08.930 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:08.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:08.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:09.032 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:09.056 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:09.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:09.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:09.157 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:09.182 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:09.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:09.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:09.284 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:09.310 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:09.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:09.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:09.411 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:09.436 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:09.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:09.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:09.538 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:09.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:09.563 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:09.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:09.664 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:09.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:09.686 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:09.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:09.703 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:09 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 47404/330609 objects degraded (14.338%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:38:09.716 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:09 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 47404/330609 objects degraded (14.338%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:38:09.716 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:09 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 47404/330609 objects degraded (14.338%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:38:09.788 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:09.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:09.811 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:09.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:09.912 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:09.936 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:09.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:09.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:10.038 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:10.062 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:10.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:10.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:10.163 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:10.188 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:10.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:10.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:10.257 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:38:10.257 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-13T14:38:10.257 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 0 2024-09-13T14:38:10.257 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:38:10.289 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:10.313 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:10.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:10.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:10.414 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:10.436 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:10.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:10.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:10.443 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:38:10.443 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:38:10.478 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:38:10.478 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:38:10.499 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:38:10.499 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:38:10.538 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:10.559 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:10.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:10.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:10.661 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:10.686 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:10.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:10.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:10.776 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:10 smithi102 ceph-mon[110183]: pgmap v781: 89 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 remapped+peering, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 73 active+clean; 6.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 12 op/s; 47403/330609 objects degraded (14.338%); 14842/330609 objects misplaced (4.489%); 3.2 MiB/s, 52 objects/s recovering 2024-09-13T14:38:10.776 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:38:10.776 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:38:10.776 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:10 smithi102 ceph-mon[115269]: pgmap v781: 89 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 remapped+peering, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 73 active+clean; 6.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 12 op/s; 47403/330609 objects degraded (14.338%); 14842/330609 objects misplaced (4.489%); 3.2 MiB/s, 52 objects/s recovering 2024-09-13T14:38:10.776 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:38:10.777 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:38:10.789 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:10.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:10.814 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:10.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:10.915 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:10.941 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:10.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:10.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:10.954 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:10 smithi195 ceph-mon[104262]: pgmap v781: 89 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 remapped+peering, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 73 active+clean; 6.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 12 op/s; 47403/330609 objects degraded (14.338%); 14842/330609 objects misplaced (4.489%); 3.2 MiB/s, 52 objects/s recovering 2024-09-13T14:38:10.955 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:38:10.955 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:38:10.980 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T14:38:11.043 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:11.066 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:11.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:11.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:11.167 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:11.192 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:11.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:11.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:11.198 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:38:11.294 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:11.315 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:11.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:11.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:11.417 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:11.438 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:11.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:11.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:11.540 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:11.548 INFO:teuthology.orchestra.run.smithi102.stderr:reweighted osd.3 to 1 (10000) 2024-09-13T14:38:11.574 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:11.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:11.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:11.650 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:11 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2474351083' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:38:11.650 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:11 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:38:11.675 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:11.699 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:11.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:11.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:11.800 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:11.822 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:11.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:11.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:11.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:11 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2474351083' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:38:11.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:11 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:38:11.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:11 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2474351083' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:38:11.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:11 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:38:11.924 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:11.949 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:11.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:11.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:12.051 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:12.073 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:12.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:12.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:12.175 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:12.199 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:12.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:12.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:12.262 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-13T14:38:12.300 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:12.328 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:12.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:12.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:12.430 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:12.453 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:12.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:12.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:12.554 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:12.586 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:12.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:12.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:12.688 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:12.709 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:12.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:12.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:12.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:12 smithi195 ceph-mon[104262]: pgmap v782: 89 pgs: 2 activating+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 remapped+peering, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 73 active+clean; 6.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 1 op/s; 52189/330561 objects degraded (15.788%); 7983/330561 objects misplaced (2.415%); 2.8 MiB/s, 46 objects/s recovering 2024-09-13T14:38:12.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:12 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T14:38:12.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:12 smithi195 ceph-mon[104262]: osdmap e247: 8 total, 8 up, 6 in 2024-09-13T14:38:12.812 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:12.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:12.837 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:12.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:12.853 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:38:12.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:12 smithi102 ceph-mon[110183]: pgmap v782: 89 pgs: 2 activating+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 remapped+peering, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 73 active+clean; 6.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 1 op/s; 52189/330561 objects degraded (15.788%); 7983/330561 objects misplaced (2.415%); 2.8 MiB/s, 46 objects/s recovering 2024-09-13T14:38:12.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:12 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T14:38:12.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:12 smithi102 ceph-mon[110183]: osdmap e247: 8 total, 8 up, 6 in 2024-09-13T14:38:12.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:12 smithi102 ceph-mon[115269]: pgmap v782: 89 pgs: 2 activating+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 remapped+peering, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 73 active+clean; 6.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 1 op/s; 52189/330561 objects degraded (15.788%); 7983/330561 objects misplaced (2.415%); 2.8 MiB/s, 46 objects/s recovering 2024-09-13T14:38:12.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:12 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T14:38:12.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:12 smithi102 ceph-mon[115269]: osdmap e247: 8 total, 8 up, 6 in 2024-09-13T14:38:12.939 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:12.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:12.963 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:12.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:13.034 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:38:13.034 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:38:13.064 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:13.090 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:13.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:13.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:13.192 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:13.219 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:13.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:13.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:13.321 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:13.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:13.347 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:13.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:13.448 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:13.470 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:13.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:13.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:13.488 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:38:13.564 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is set 2024-09-13T14:38:13.571 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:13.592 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:13.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:13.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:13.694 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:13.715 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:13.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:13.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:13.731 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:13 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2168923047' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:38:13.731 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:13 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:38:13.754 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T14:38:13.817 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:13.861 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:13.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:13.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:13.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:13 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2168923047' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:38:13.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:13 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:38:13.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:13 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2168923047' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:38:13.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:13 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:38:13.963 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:13.986 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:13.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:13.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:14.011 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:38:14.088 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:14.111 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:14.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:14.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:14.212 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:14.235 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:14.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:14.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:14.337 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:14.360 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:14.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:14.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:14.462 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:14.486 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:14.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:14.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:14.588 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:14.612 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:14.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:14.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:14.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:14 smithi102 ceph-mon[110183]: pgmap v784: 89 pgs: 1 active+recovery_wait+remapped, 1 activating+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 remapped+peering, 7 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 73 active+clean; 6.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 2 op/s; 51744/330540 objects degraded (15.654%); 7983/330540 objects misplaced (2.415%); 5.6 MiB/s, 91 objects/s recovering 2024-09-13T14:38:14.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:14 smithi102 ceph-mon[110183]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:38:14.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:14 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:38:14.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:14 smithi102 ceph-mon[110183]: osdmap e248: 8 total, 8 up, 6 in 2024-09-13T14:38:14.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:14 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 51744/330540 objects degraded (15.654%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:38:14.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:14 smithi102 ceph-mon[115269]: pgmap v784: 89 pgs: 1 active+recovery_wait+remapped, 1 activating+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 remapped+peering, 7 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 73 active+clean; 6.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 2 op/s; 51744/330540 objects degraded (15.654%); 7983/330540 objects misplaced (2.415%); 5.6 MiB/s, 91 objects/s recovering 2024-09-13T14:38:14.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:14 smithi102 ceph-mon[115269]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:38:14.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:14 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:38:14.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:14 smithi102 ceph-mon[115269]: osdmap e248: 8 total, 8 up, 6 in 2024-09-13T14:38:14.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:14 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 51744/330540 objects degraded (15.654%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:38:14.714 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:14.737 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:14.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:14.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:14.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:14 smithi195 ceph-mon[104262]: pgmap v784: 89 pgs: 1 active+recovery_wait+remapped, 1 activating+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 remapped+peering, 7 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 73 active+clean; 6.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 2 op/s; 51744/330540 objects degraded (15.654%); 7983/330540 objects misplaced (2.415%); 5.6 MiB/s, 91 objects/s recovering 2024-09-13T14:38:14.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:14 smithi195 ceph-mon[104262]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:38:14.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:14 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:38:14.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:14 smithi195 ceph-mon[104262]: osdmap e248: 8 total, 8 up, 6 in 2024-09-13T14:38:14.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:14 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 51744/330540 objects degraded (15.654%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:38:14.839 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:14.864 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:14.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:14.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:14.965 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:14.988 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:14.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:14.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:15.090 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:15.112 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:15.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:15.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:15.213 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:15.236 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:15.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:15.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:15.338 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:15.363 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:15.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:15.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:15.465 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:15.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:15.487 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:15.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:15.588 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:15.618 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:15.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:15.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:15.720 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:15.745 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:15.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:15.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:15.797 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:38:15.798 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:38:15.847 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:15.861 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:38:15.870 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:15.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:15.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:15.972 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:15.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:15.999 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:15.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:16.100 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:16.128 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:16.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:16.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:16.230 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:16.255 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:16.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:16.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:16.357 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:16.379 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:16.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:16.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:16.465 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T14:38:16.480 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:16.542 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:16.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:16.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:16.569 INFO:teuthology.orchestra.run.smithi102.stderr:reweighted osd.4 to 1 (10000) 2024-09-13T14:38:16.644 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:16.665 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:16.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:16.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:16.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:16 smithi195 ceph-mon[104262]: pgmap v786: 89 pgs: 1 active+recovery_wait+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 remapped+peering, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 73 active+clean; 6.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 4 op/s; 51743/330504 objects degraded (15.656%); 7983/330504 objects misplaced (2.415%); 3.5 MiB/s, 56 objects/s recovering 2024-09-13T14:38:16.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:16 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3159327627' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:38:16.767 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:16.788 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:16.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:16.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:16.790 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:38:16.817 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:16 smithi102 ceph-mon[110183]: pgmap v786: 89 pgs: 1 active+recovery_wait+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 remapped+peering, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 73 active+clean; 6.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 4 op/s; 51743/330504 objects degraded (15.656%); 7983/330504 objects misplaced (2.415%); 3.5 MiB/s, 56 objects/s recovering 2024-09-13T14:38:16.817 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:16 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3159327627' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:38:16.818 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:16 smithi102 ceph-mon[115269]: pgmap v786: 89 pgs: 1 active+recovery_wait+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 remapped+peering, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 73 active+clean; 6.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 4 op/s; 51743/330504 objects degraded (15.656%); 7983/330504 objects misplaced (2.415%); 3.5 MiB/s, 56 objects/s recovering 2024-09-13T14:38:16.818 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:16 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3159327627' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:38:16.819 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:38:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:38:16] "GET /metrics HTTP/1.1" 200 36552 "" "Prometheus/2.51.0" 2024-09-13T14:38:16.889 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:16.910 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:16.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:16.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:17.012 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:17.034 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:17.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:17.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:17.136 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:17.160 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:17.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:17.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:17.262 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:17.289 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:17.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:17.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:17.392 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:17.412 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:17.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:17.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:17.683 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:17.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:17.711 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:17.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:17.772 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-13T14:38:17.813 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:17.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:17.836 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:17.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:17.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:17 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3159327627' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T14:38:17.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:17 smithi102 ceph-mon[110183]: osdmap e249: 8 total, 8 up, 6 in 2024-09-13T14:38:17.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:17 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3159327627' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T14:38:17.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:17 smithi102 ceph-mon[115269]: osdmap e249: 8 total, 8 up, 6 in 2024-09-13T14:38:17.937 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:17.968 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:17.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:17.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:17.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:17 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3159327627' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T14:38:17.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:17 smithi195 ceph-mon[104262]: osdmap e249: 8 total, 8 up, 6 in 2024-09-13T14:38:18.071 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:18.097 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:18.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:18.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:18.198 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:18.224 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:18.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:18.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:18.326 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:18.348 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:18.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:18.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:18.364 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:38:18.384 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:38:18.450 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:18.473 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:18.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:18.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:18.575 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:18.597 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:18.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:18.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:18.699 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:18.727 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:18.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:18.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:18.829 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:18.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:18.854 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:18.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:18.866 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:18 smithi102 ceph-mon[110183]: pgmap v788: 89 pgs: 1 active+recovery_wait+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 remapped+peering, 7 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 73 active+clean; 6.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 10 op/s; 46450/330372 objects degraded (14.060%); 12634/330372 objects misplaced (3.824%); 9.5 MiB/s, 173 objects/s recovering 2024-09-13T14:38:18.866 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:18 smithi102 ceph-mon[115269]: pgmap v788: 89 pgs: 1 active+recovery_wait+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 remapped+peering, 7 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 73 active+clean; 6.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 10 op/s; 46450/330372 objects degraded (14.060%); 12634/330372 objects misplaced (3.824%); 9.5 MiB/s, 173 objects/s recovering 2024-09-13T14:38:18.956 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:18.979 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:18.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:18.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:18.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:18 smithi195 ceph-mon[104262]: pgmap v788: 89 pgs: 1 active+recovery_wait+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 remapped+peering, 7 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 73 active+clean; 6.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 10 op/s; 46450/330372 objects degraded (14.060%); 12634/330372 objects misplaced (3.824%); 9.5 MiB/s, 173 objects/s recovering 2024-09-13T14:38:19.082 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:19.103 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:19.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:19.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:19.205 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:19.230 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:19.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:19.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:19.280 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:38:19.281 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:38:19.305 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:38:19.305 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:38:19.327 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:38:19.327 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:38:19.332 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:19.350 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:38:19.350 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:38:19.357 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:19.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:19.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:19.372 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:38:19.373 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:38:19.392 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:38:19.392 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:38:19.412 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:38:19.413 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:38:19.438 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:38:19.439 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:38:19.459 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:19.485 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:19.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:19.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:19.587 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:19.612 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:19.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:19.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:19.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:19 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 46450/330372 objects degraded (14.060%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:38:19.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:19 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2320607551' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:38:19.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:19 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 46450/330372 objects degraded (14.060%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:38:19.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:19 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2320607551' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:38:19.714 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:19.736 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:19.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:19.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:19.838 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:19.863 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:19.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:19.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:19.965 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:19.989 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:19.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:19.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:19.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:19 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 46450/330372 objects degraded (14.060%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:38:19.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:19 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2320607551' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:38:20.091 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:20.113 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:20.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:20.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:20.216 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:20.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:20.239 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:20.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:20.340 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:20.366 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:20.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:20.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:20.468 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:20.492 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:20.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:20.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:20.594 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:20.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:20.622 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:20.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:20.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:20 smithi102 ceph-mon[110183]: pgmap v789: 89 pgs: 1 active+degraded, 1 active+recovery_wait+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 6 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 75 active+clean; 6.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 12 op/s; 51036/330270 objects degraded (15.453%); 5811/330270 objects misplaced (1.759%); 8.0 MiB/s, 144 objects/s recovering 2024-09-13T14:38:20.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:20 smithi102 ceph-mon[115269]: pgmap v789: 89 pgs: 1 active+degraded, 1 active+recovery_wait+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 6 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 75 active+clean; 6.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 12 op/s; 51036/330270 objects degraded (15.453%); 5811/330270 objects misplaced (1.759%); 8.0 MiB/s, 144 objects/s recovering 2024-09-13T14:38:20.724 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:20.748 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:20.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:20.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:20.850 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:20.872 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:20.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:20.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:20.974 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:20.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:20 smithi195 ceph-mon[104262]: pgmap v789: 89 pgs: 1 active+degraded, 1 active+recovery_wait+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 6 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 75 active+clean; 6.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 12 op/s; 51036/330270 objects degraded (15.453%); 5811/330270 objects misplaced (1.759%); 8.0 MiB/s, 144 objects/s recovering 2024-09-13T14:38:20.998 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:20.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:20.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:21.100 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:21.123 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:21.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:21.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:21.225 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:21.250 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:21.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:21.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:21.352 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:21.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:21.380 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:21.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:21.481 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:21.504 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:21.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:21.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:21.606 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:21.630 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:21.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:21.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:21.654 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:38:21.732 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:21.759 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:21.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:21.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:21.861 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:21.886 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:21.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:21.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:21.987 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:22.012 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:22.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:22.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:22.114 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:22.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:22.141 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:22.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:22.241 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:22.267 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:22.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:22.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:22.369 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:22.394 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:22.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:22.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:22.496 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:22.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:22.521 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:22.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:22.622 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:22.645 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:22.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:22.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:22.747 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:22.773 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:22.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:22.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:22.786 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:38:22.875 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:22.900 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:22.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:22.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:22.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:22 smithi102 ceph-mon[110183]: pgmap v790: 89 pgs: 1 active+degraded, 1 active+recovery_wait+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 6 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 75 active+clean; 6.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 22 op/s; 51002/330003 objects degraded (15.455%); 5811/330003 objects misplaced (1.761%); 4.6 MiB/s, 90 objects/s recovering 2024-09-13T14:38:22.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:22 smithi102 ceph-mon[115269]: pgmap v790: 89 pgs: 1 active+degraded, 1 active+recovery_wait+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 6 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 75 active+clean; 6.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 22 op/s; 51002/330003 objects degraded (15.455%); 5811/330003 objects misplaced (1.761%); 4.6 MiB/s, 90 objects/s recovering 2024-09-13T14:38:22.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:22 smithi195 ceph-mon[104262]: pgmap v790: 89 pgs: 1 active+degraded, 1 active+recovery_wait+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 6 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 75 active+clean; 6.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 22 op/s; 51002/330003 objects degraded (15.455%); 5811/330003 objects misplaced (1.761%); 4.6 MiB/s, 90 objects/s recovering 2024-09-13T14:38:23.002 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:23.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:23.028 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:23.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:23.129 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:23.152 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:23.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:23.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:23.254 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:23.279 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:23.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:23.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:23.381 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:23.403 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:23.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:23.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:23.505 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:23.528 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:23.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:23.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:23.630 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:23.652 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:23.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:23.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:23.755 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:23.779 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:23.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:23.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:23.881 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:23.886 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:23 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:38:23.886 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:23 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:38:23.886 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:23 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:38:23.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:23.908 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:23.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:23.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:23 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:38:23.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:23 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:38:23.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:23 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:38:23.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:23 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:38:23.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:23 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:38:23.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:23 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:38:24.009 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:24.033 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:24.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:24.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:24.135 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:24.159 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:24.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:24.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:24.262 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:24.286 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:24.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:24.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:24.390 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:24.413 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:24.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:24.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:24.515 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:24.538 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:24.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:24.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:24.640 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:24.666 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:24.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:24.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:24.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:24 smithi102 ceph-mon[115269]: pgmap v791: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 75 active+clean; 6.7 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 86 op/s; 50460/327963 objects degraded (15.386%); 5750/327963 objects misplaced (1.753%); 7.2 MiB/s, 132 objects/s recovering 2024-09-13T14:38:24.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:24 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 51002/330003 objects degraded (15.455%), 12 pgs degraded (PG_DEGRADED) 2024-09-13T14:38:24.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:24 smithi102 ceph-mon[110183]: pgmap v791: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 75 active+clean; 6.7 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 86 op/s; 50460/327963 objects degraded (15.386%); 5750/327963 objects misplaced (1.753%); 7.2 MiB/s, 132 objects/s recovering 2024-09-13T14:38:24.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:24 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 51002/330003 objects degraded (15.455%), 12 pgs degraded (PG_DEGRADED) 2024-09-13T14:38:24.769 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:24.792 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:24.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:24.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:24.855 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:24 smithi195 ceph-mon[104262]: pgmap v791: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 75 active+clean; 6.7 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 86 op/s; 50460/327963 objects degraded (15.386%); 5750/327963 objects misplaced (1.753%); 7.2 MiB/s, 132 objects/s recovering 2024-09-13T14:38:24.855 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:24 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 51002/330003 objects degraded (15.455%), 12 pgs degraded (PG_DEGRADED) 2024-09-13T14:38:24.894 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:24.921 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:24.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:24.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:25.022 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:25.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:25.048 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:25.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:25.149 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:25.173 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:25.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:25.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:25.275 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:25.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:25.301 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:25.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:25.402 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:25.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:25.425 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:25.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:25.526 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:25.549 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:25.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:25.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:25.651 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:25.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:25.676 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:25.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:25.777 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:25.800 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:25.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:25.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:25.851 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:25 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:38:25.902 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:25.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:25 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:38:25.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:25 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:38:25.926 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:25.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:25.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:26.029 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:26.055 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:26.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:26.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:26.156 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:26.181 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:26.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:26.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:26.282 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:26.306 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:26.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:26.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:26.409 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:26.431 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:26.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:26.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:26.533 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:26.554 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:26.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:26.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:26.656 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:26.681 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:26.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:26.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:26.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:26 smithi195 ceph-mon[104262]: pgmap v792: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 75 active+clean; 6.7 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 84 op/s; 50460/327963 objects degraded (15.386%); 5750/327963 objects misplaced (1.753%); 7.1 MiB/s, 130 objects/s recovering 2024-09-13T14:38:26.783 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:26.808 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:26.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:26.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:26.910 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:26.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:26 smithi102 ceph-mon[110183]: pgmap v792: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 75 active+clean; 6.7 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 84 op/s; 50460/327963 objects degraded (15.386%); 5750/327963 objects misplaced (1.753%); 7.1 MiB/s, 130 objects/s recovering 2024-09-13T14:38:26.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:26 smithi102 ceph-mon[115269]: pgmap v792: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 75 active+clean; 6.7 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 84 op/s; 50460/327963 objects degraded (15.386%); 5750/327963 objects misplaced (1.753%); 7.1 MiB/s, 130 objects/s recovering 2024-09-13T14:38:26.918 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:38:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:38:26] "GET /metrics HTTP/1.1" 200 36555 "" "Prometheus/2.51.0" 2024-09-13T14:38:26.934 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:26.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:26.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:27.036 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:27.064 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:27.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:27.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:27.166 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:27.196 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:27.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:27.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:27.298 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:27.321 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:27.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:27.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:27.422 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:27.446 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:27.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:27.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:27.548 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:27.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:27.573 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:27.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:27.674 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:27.697 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:27.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:27.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:27.799 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:27.822 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:27.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:27.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:27.924 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:27.947 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:27.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:27.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:28.049 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:28.073 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:28.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:28.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:28.175 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:28.199 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:28.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:28.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:28.301 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:28.324 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:28.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:28.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:28.426 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:28.450 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:28.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:28.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:28.552 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:28.574 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:28.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:28.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:28.676 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:28.701 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:28.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:28.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:28.803 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:28.825 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:28.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:28.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:28.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:28 smithi102 ceph-mon[115269]: pgmap v793: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 75 active+clean; 6.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 117 op/s; 49793/326676 objects degraded (15.242%); 5698/326676 objects misplaced (1.744%); 10 MiB/s, 181 objects/s recovering 2024-09-13T14:38:28.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:28 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:38:28.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:28 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:38:28.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:28 smithi102 ceph-mon[110183]: pgmap v793: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 75 active+clean; 6.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 117 op/s; 49793/326676 objects degraded (15.242%); 5698/326676 objects misplaced (1.744%); 10 MiB/s, 181 objects/s recovering 2024-09-13T14:38:28.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:28 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:38:28.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:28 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:38:28.927 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:28.952 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:28.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:28.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:28.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:28 smithi195 ceph-mon[104262]: pgmap v793: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 75 active+clean; 6.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 117 op/s; 49793/326676 objects degraded (15.242%); 5698/326676 objects misplaced (1.744%); 10 MiB/s, 181 objects/s recovering 2024-09-13T14:38:28.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:28 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:38:28.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:28 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:38:29.054 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:29.078 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:29.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:29.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:29.180 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:29.203 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:29.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:29.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:29.305 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:29.327 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:29.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:29.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:29.429 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:29.451 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:29.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:29.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:29.553 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:29.580 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:29.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:29.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:29.681 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:29.704 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:29.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:29.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:29.808 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:29.833 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:29.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:29.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:29.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:29 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 49793/326676 objects degraded (15.242%), 12 pgs degraded (PG_DEGRADED) 2024-09-13T14:38:29.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:29 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 49793/326676 objects degraded (15.242%), 12 pgs degraded (PG_DEGRADED) 2024-09-13T14:38:29.936 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:29.959 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:29.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:29.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:29.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:29 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 49793/326676 objects degraded (15.242%), 12 pgs degraded (PG_DEGRADED) 2024-09-13T14:38:30.061 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:30.083 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:30.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:30.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:30.186 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:30.209 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:30.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:30.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:30.311 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:30.333 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:30.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:30.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:30.435 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:30.460 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:30.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:30.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:30.562 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:30.584 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:30.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:30.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:30.686 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:30.708 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:30.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:30.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:30.810 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:30.832 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:30.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:30.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:30.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:30 smithi102 ceph-mon[110183]: pgmap v794: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 75 active+clean; 6.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 126 op/s; 49793/325818 objects degraded (15.282%); 5698/325818 objects misplaced (1.749%); 7.0 MiB/s, 114 objects/s recovering 2024-09-13T14:38:30.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:30 smithi102 ceph-mon[115269]: pgmap v794: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 75 active+clean; 6.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 126 op/s; 49793/325818 objects degraded (15.282%); 5698/325818 objects misplaced (1.749%); 7.0 MiB/s, 114 objects/s recovering 2024-09-13T14:38:30.934 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:30.957 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:30.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:30.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:30.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:30 smithi195 ceph-mon[104262]: pgmap v794: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 75 active+clean; 6.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 126 op/s; 49793/325818 objects degraded (15.282%); 5698/325818 objects misplaced (1.749%); 7.0 MiB/s, 114 objects/s recovering 2024-09-13T14:38:31.059 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:31.081 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:31.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:31.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:31.183 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:31.204 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:31.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:31.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:31.306 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:31.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:31.329 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:31.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:31.430 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:31.458 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:31.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:31.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:31.560 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:31.590 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:31.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:31.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:31.692 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:31.716 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:31.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:31.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:31.817 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:31.841 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:31.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:31.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:31.944 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:31.968 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:31.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:31.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:32.070 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:32.093 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:32.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:32.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:32.195 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:32.217 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:32.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:32.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:32.319 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:32.341 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:32.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:32.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:32.443 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:32.465 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:32.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:32.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:32.567 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:32.589 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:32.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:32.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:32.691 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:32.713 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:32.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:32.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:32.815 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:32.836 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:32.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:32.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:32.853 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:32 smithi102 ceph-mon[110183]: pgmap v795: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 75 active+clean; 6.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 135 op/s; 49754/325374 objects degraded (15.291%); 5698/325374 objects misplaced (1.751%); 6.7 MiB/s, 109 objects/s recovering 2024-09-13T14:38:32.853 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:32 smithi102 ceph-mon[115269]: pgmap v795: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 75 active+clean; 6.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 135 op/s; 49754/325374 objects degraded (15.291%); 5698/325374 objects misplaced (1.751%); 6.7 MiB/s, 109 objects/s recovering 2024-09-13T14:38:32.938 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:32.964 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:32.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:32.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:32.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:32 smithi195 ceph-mon[104262]: pgmap v795: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 75 active+clean; 6.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 135 op/s; 49754/325374 objects degraded (15.291%); 5698/325374 objects misplaced (1.751%); 6.7 MiB/s, 109 objects/s recovering 2024-09-13T14:38:33.067 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:33.089 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:33.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:33.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:33.191 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:33.216 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:33.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:33.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:33.317 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:33.340 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:33.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:33.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:33.442 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:33.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:33.468 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:33.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:33.555 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:38:33.570 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:33.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:33.598 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:33.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:33.616 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is unset 2024-09-13T14:38:33.699 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:33.721 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:33.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:33.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:33.823 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:33.846 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:33.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:33.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:33.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:33 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1482714114' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:38:33.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:33 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:38:33.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:33 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1482714114' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:38:33.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:33 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:38:33.948 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:33.971 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:33.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:33.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:33.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:33 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1482714114' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:38:33.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:33 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:38:34.073 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:34.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:34.099 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:34.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:34.200 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:34.222 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:34.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:34.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:34.323 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:34.347 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:34.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:34.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:34.448 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:34.472 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:34.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:34.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:34.574 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:34.599 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:34.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:34.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:34.701 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:34.725 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:34.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:34.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:34.826 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:34.851 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:34.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:34.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:34.906 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:34 smithi102 ceph-mon[115269]: pgmap v796: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 75 active+clean; 6.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 180 op/s; 48947/323502 objects degraded (15.130%); 5650/323502 objects misplaced (1.747%); 11 MiB/s, 175 objects/s recovering 2024-09-13T14:38:34.906 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:34 smithi102 ceph-mon[115269]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:38:34.906 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:34 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:38:34.906 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:34 smithi102 ceph-mon[115269]: osdmap e250: 8 total, 8 up, 6 in 2024-09-13T14:38:34.906 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:34 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 48947/323502 objects degraded (15.130%), 12 pgs degraded (PG_DEGRADED) 2024-09-13T14:38:34.906 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:34 smithi102 ceph-mon[110183]: pgmap v796: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 75 active+clean; 6.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 180 op/s; 48947/323502 objects degraded (15.130%); 5650/323502 objects misplaced (1.747%); 11 MiB/s, 175 objects/s recovering 2024-09-13T14:38:34.906 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:34 smithi102 ceph-mon[110183]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:38:34.906 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:34 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:38:34.907 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:34 smithi102 ceph-mon[110183]: osdmap e250: 8 total, 8 up, 6 in 2024-09-13T14:38:34.907 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:34 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 48947/323502 objects degraded (15.130%), 12 pgs degraded (PG_DEGRADED) 2024-09-13T14:38:34.953 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:34.977 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:34.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:34.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:34.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:34 smithi195 ceph-mon[104262]: pgmap v796: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 75 active+clean; 6.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 180 op/s; 48947/323502 objects degraded (15.130%); 5650/323502 objects misplaced (1.747%); 11 MiB/s, 175 objects/s recovering 2024-09-13T14:38:34.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:34 smithi195 ceph-mon[104262]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:38:34.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:34 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:38:34.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:34 smithi195 ceph-mon[104262]: osdmap e250: 8 total, 8 up, 6 in 2024-09-13T14:38:34.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:34 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 48947/323502 objects degraded (15.130%), 12 pgs degraded (PG_DEGRADED) 2024-09-13T14:38:35.031 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:38:35.031 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [ 2024-09-13T14:38:35.031 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:38:35.031 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35379.0:118327 7.4 7:245f859d:::benchmark_data_smithi102_227395_object5512:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-09-13T14:38:35.032 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:38:34.973480+0000", 2024-09-13T14:38:35.032 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.048448514999999998, 2024-09-13T14:38:35.032 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.048469390000000001, 2024-09-13T14:38:35.032 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:38:35.032 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:38:35.032 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:38:35.032 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:38:35.032 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35379", 2024-09-13T14:38:35.032 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:38:35.032 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 118327 2024-09-13T14:38:35.032 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:38:35.032 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:38:35.033 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:38:35.033 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:38:35.033 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:34.973480+0000", 2024-09-13T14:38:35.033 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:38:35.033 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:38:35.033 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:38:35.033 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:38:35.033 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:34.973480+0000", 2024-09-13T14:38:35.033 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:38:35.033 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:38:35.033 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:38:35.033 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:38:35.033 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:34.973482+0000", 2024-09-13T14:38:35.034 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2950000000000001e-06 2024-09-13T14:38:35.034 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:38:35.034 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:38:35.034 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:38:35.034 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:34.973487+0000", 2024-09-13T14:38:35.034 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.7960000000000001e-06 2024-09-13T14:38:35.034 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:38:35.034 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:38:35.034 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:38:35.034 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:34.973499+0000", 2024-09-13T14:38:35.034 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1929e-05 2024-09-13T14:38:35.034 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:38:35.035 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:38:35.035 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:38:35.035 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:34.973506+0000", 2024-09-13T14:38:35.035 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.6900000000000003e-06 2024-09-13T14:38:35.035 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:38:35.035 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:38:35.035 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:38:35.035 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:35.003826+0000", 2024-09-13T14:38:35.035 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.030320514999999999 2024-09-13T14:38:35.035 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:38:35.035 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:38:35.035 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:38:35.035 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:35.011313+0000", 2024-09-13T14:38:35.036 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0074868620000000004 2024-09-13T14:38:35.036 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:38:35.036 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:38:35.036 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:38:35.036 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:35.020211+0000", 2024-09-13T14:38:35.036 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0088977080000000007 2024-09-13T14:38:35.036 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:38:35.036 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:38:35.036 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:38:35.036 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:38:35.036 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:38:35.037 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop_reply(client.35379.0:118324 7.4 e250/245)", 2024-09-13T14:38:35.037 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:38:34.997755+0000", 2024-09-13T14:38:35.037 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.024173461, 2024-09-13T14:38:35.037 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.024247711000000002, 2024-09-13T14:38:35.037 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:38:35.037 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:38:35.037 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "queued for pg", 2024-09-13T14:38:35.037 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:38:35.037 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:38:35.037 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:38:35.037 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:34.997755+0000", 2024-09-13T14:38:35.037 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:38:35.037 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:38:35.038 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:38:35.038 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:38:35.038 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:34.997755+0000", 2024-09-13T14:38:35.038 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:38:35.038 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:38:35.038 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:38:35.038 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:38:35.038 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:34.997760+0000", 2024-09-13T14:38:35.038 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.0089999999999998e-06 2024-09-13T14:38:35.038 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:38:35.038 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:38:35.038 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:38:35.038 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:34.997769+0000", 2024-09-13T14:38:35.038 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.5590000000000006e-06 2024-09-13T14:38:35.039 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:38:35.039 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:38:35.039 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:38:35.039 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:34.997785+0000", 2024-09-13T14:38:35.039 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.6526999999999999e-05 2024-09-13T14:38:35.039 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:38:35.039 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:38:35.039 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:38:35.039 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:34.997798+0000", 2024-09-13T14:38:35.039 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2670000000000001e-05 2024-09-13T14:38:35.039 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:38:35.039 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:38:35.039 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:38:35.040 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:38:35.040 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:38:35.040 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop_reply(client.35379.0:118324 7.4 e250/245)", 2024-09-13T14:38:35.040 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:38:34.999370+0000", 2024-09-13T14:38:35.040 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.022558783999999998, 2024-09-13T14:38:35.040 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.022663396999999998, 2024-09-13T14:38:35.040 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:38:35.040 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:38:35.041 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "queued for pg", 2024-09-13T14:38:35.041 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:38:35.041 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:38:35.041 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:38:35.041 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:34.999370+0000", 2024-09-13T14:38:35.041 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:38:35.041 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:38:35.041 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:38:35.041 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:38:35.041 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:34.999370+0000", 2024-09-13T14:38:35.041 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:38:35.041 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:38:35.041 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:38:35.042 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:38:35.042 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:34.999371+0000", 2024-09-13T14:38:35.042 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.3799999999999996e-07 2024-09-13T14:38:35.042 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:38:35.042 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:38:35.042 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:38:35.042 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:34.999378+0000", 2024-09-13T14:38:35.042 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.8889999999999996e-06 2024-09-13T14:38:35.042 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:38:35.042 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:38:35.042 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:38:35.042 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:34.999390+0000", 2024-09-13T14:38:35.042 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2680999999999999e-05 2024-09-13T14:38:35.042 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:38:35.043 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:38:35.043 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:38:35.043 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:34.999398+0000", 2024-09-13T14:38:35.043 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.9230000000000002e-06 2024-09-13T14:38:35.043 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:38:35.043 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:38:35.043 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:38:35.043 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:38:35.043 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:38:35.043 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35379.0:118324 7.4 7:2215b66d:::benchmark_data_smithi102_227395_object5509:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-09-13T14:38:35.043 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:38:34.962091+0000", 2024-09-13T14:38:35.043 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.059837656000000003, 2024-09-13T14:38:35.043 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.059966865000000001, 2024-09-13T14:38:35.044 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:38:35.044 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:38:35.044 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:38:35.044 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:38:35.044 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35379", 2024-09-13T14:38:35.044 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:38:35.044 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 118324 2024-09-13T14:38:35.044 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:38:35.044 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:38:35.044 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:38:35.044 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:38:35.044 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:34.962091+0000", 2024-09-13T14:38:35.044 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:38:35.045 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:38:35.045 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:38:35.045 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:38:35.045 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:34.962091+0000", 2024-09-13T14:38:35.045 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:38:35.045 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:38:35.045 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:38:35.045 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:38:35.045 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:34.962093+0000", 2024-09-13T14:38:35.045 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.5260000000000001e-06 2024-09-13T14:38:35.045 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:38:35.045 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:38:35.045 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:38:35.046 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:34.962099+0000", 2024-09-13T14:38:35.046 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.798e-06 2024-09-13T14:38:35.046 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:38:35.046 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:38:35.046 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:38:35.046 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:34.962112+0000", 2024-09-13T14:38:35.046 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2597e-05 2024-09-13T14:38:35.046 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:38:35.046 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:38:35.046 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:38:35.046 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:34.962122+0000", 2024-09-13T14:38:35.046 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.7650000000000005e-06 2024-09-13T14:38:35.046 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:38:35.046 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:38:35.047 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:38:35.047 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:34.976221+0000", 2024-09-13T14:38:35.047 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.014099008 2024-09-13T14:38:35.047 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:38:35.047 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:38:35.047 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:38:35.047 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:34.983236+0000", 2024-09-13T14:38:35.047 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0070153799999999999 2024-09-13T14:38:35.047 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:38:35.047 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:38:35.047 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:38:35.047 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:34.994900+0000", 2024-09-13T14:38:35.047 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.011663659 2024-09-13T14:38:35.048 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:38:35.048 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:38:35.048 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:38:35.048 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:38:35.048 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-09-13T14:38:35.048 INFO:teuthology.orchestra.run.smithi102.stdout: "num_ops": 4 2024-09-13T14:38:35.048 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:38:35.078 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:35.104 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:35.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:35.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:35.206 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:35.229 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:35.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:35.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:35.331 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:35.355 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:35.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:35.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:35.458 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:35.481 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:35.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:35.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:35.583 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:35.607 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:35.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:35.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:35.709 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:35.733 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:35.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:35.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:35.835 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:35.859 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:35.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:35.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:35.961 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:35.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:35.988 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:35.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:36.089 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:36.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:36.114 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:36.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:36.215 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:36.239 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:36.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:36.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:36.314 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T14:38:36.341 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:36.365 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:36.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:36.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:36.467 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:36.497 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:36.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:36.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:36.599 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:36.609 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:38:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:38:36] "GET /metrics HTTP/1.1" 200 36555 "" "Prometheus/2.51.0" 2024-09-13T14:38:36.626 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:36.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:36.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:36.728 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:36.750 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:36.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:36.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:36.852 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:36.876 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:36.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:36.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:36.911 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:36 smithi102 ceph-mon[110183]: pgmap v798: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 75 active+clean; 6.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 148 op/s; 48947/323502 objects degraded (15.130%); 5650/323502 objects misplaced (1.747%); 9.5 MiB/s, 151 objects/s recovering 2024-09-13T14:38:36.912 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:36 smithi102 ceph-mon[115269]: pgmap v798: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 75 active+clean; 6.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 148 op/s; 48947/323502 objects degraded (15.130%); 5650/323502 objects misplaced (1.747%); 9.5 MiB/s, 151 objects/s recovering 2024-09-13T14:38:36.977 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:36.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:36 smithi195 ceph-mon[104262]: pgmap v798: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 75 active+clean; 6.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 148 op/s; 48947/323502 objects degraded (15.130%); 5650/323502 objects misplaced (1.747%); 9.5 MiB/s, 151 objects/s recovering 2024-09-13T14:38:37.003 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:37.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:37.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:37.105 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:37.128 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:37.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:37.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:37.230 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:37.256 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:37.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:37.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:37.358 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:37.382 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:37.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:37.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:37.484 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:37.505 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:37.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:37.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:37.607 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:37.631 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:37.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:37.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:37.733 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:37.756 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:37.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:37.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:37.858 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:37.883 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:37.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:37.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:37.989 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:38.015 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:38.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:38.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:38.116 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:38.140 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:38.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:38.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:38.242 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:38.265 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:38.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:38.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:38.367 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:38.389 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:38.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:38.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:38.491 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:38.518 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:38.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:38.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:38.620 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:38.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:38.654 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:38.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:38.657 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:38 smithi102 ceph-mon[110183]: pgmap v799: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 75 active+clean; 6.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 158 op/s; 47709/321909 objects degraded (14.821%); 5589/321909 objects misplaced (1.736%); 13 MiB/s, 207 objects/s recovering 2024-09-13T14:38:38.658 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:38 smithi102 ceph-mon[115269]: pgmap v799: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 75 active+clean; 6.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 158 op/s; 47709/321909 objects degraded (14.821%); 5589/321909 objects misplaced (1.736%); 13 MiB/s, 207 objects/s recovering 2024-09-13T14:38:38.756 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:38.779 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:38.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:38.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:38.881 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:38.903 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:38.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:38.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:38.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:38 smithi195 ceph-mon[104262]: pgmap v799: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 75 active+clean; 6.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 158 op/s; 47709/321909 objects degraded (14.821%); 5589/321909 objects misplaced (1.736%); 13 MiB/s, 207 objects/s recovering 2024-09-13T14:38:39.005 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:39.027 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:39.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:39.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:39.129 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:39.151 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:39.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:39.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:39.253 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:39.275 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:39.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:39.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:39.376 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:39.399 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:39.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:39.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:39.501 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:39.524 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:39.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:39.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:39.626 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:39.661 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:39.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:39.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:39.718 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:38:39.763 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:39.787 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:39.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:39.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:39.889 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:39.911 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:39.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:39.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:39.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:39 smithi102 ceph-mon[110183]: osdmap e251: 8 total, 8 up, 6 in 2024-09-13T14:38:39.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:39 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 47709/321909 objects degraded (14.821%), 12 pgs degraded (PG_DEGRADED) 2024-09-13T14:38:39.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:39 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/769794495' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:38:39.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:39 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2919313330' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:38:39.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:39 smithi102 ceph-mon[115269]: osdmap e251: 8 total, 8 up, 6 in 2024-09-13T14:38:39.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:39 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 47709/321909 objects degraded (14.821%), 12 pgs degraded (PG_DEGRADED) 2024-09-13T14:38:39.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:39 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/769794495' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:38:39.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:39 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2919313330' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:38:39.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:39 smithi195 ceph-mon[104262]: osdmap e251: 8 total, 8 up, 6 in 2024-09-13T14:38:39.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:39 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 47709/321909 objects degraded (14.821%), 12 pgs degraded (PG_DEGRADED) 2024-09-13T14:38:39.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:39 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/769794495' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:38:39.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:39 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2919313330' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:38:40.013 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:40.035 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:40.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:40.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:40.137 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:40.158 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:40.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:40.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:40.163 INFO:teuthology.orchestra.run.smithi102.stderr:reweighted osd.6 to 1 (10000) 2024-09-13T14:38:40.260 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:40.281 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:40.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:40.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:40.383 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:40.408 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:40.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:40.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:40.510 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:40.532 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:40.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:40.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:40.634 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:40.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:40.660 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:40.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:40.761 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:40.786 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:40.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:40.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:40.888 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:40.910 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:40.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:40.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:40.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:40 smithi102 ceph-mon[110183]: pgmap v801: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 75 active+clean; 6.5 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 181 op/s; 47709/321018 objects degraded (14.862%); 5589/321018 objects misplaced (1.741%); 16 MiB/s, 254 objects/s recovering 2024-09-13T14:38:40.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:40 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2919313330' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T14:38:40.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:40 smithi102 ceph-mon[110183]: osdmap e252: 8 total, 8 up, 6 in 2024-09-13T14:38:40.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:40 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:38:40.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:40 smithi102 ceph-mon[115269]: pgmap v801: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 75 active+clean; 6.5 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 181 op/s; 47709/321018 objects degraded (14.862%); 5589/321018 objects misplaced (1.741%); 16 MiB/s, 254 objects/s recovering 2024-09-13T14:38:40.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:40 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2919313330' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T14:38:40.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:40 smithi102 ceph-mon[115269]: osdmap e252: 8 total, 8 up, 6 in 2024-09-13T14:38:40.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:40 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:38:40.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:40 smithi195 ceph-mon[104262]: pgmap v801: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 75 active+clean; 6.5 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 181 op/s; 47709/321018 objects degraded (14.862%); 5589/321018 objects misplaced (1.741%); 16 MiB/s, 254 objects/s recovering 2024-09-13T14:38:40.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:40 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2919313330' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T14:38:40.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:40 smithi195 ceph-mon[104262]: osdmap e252: 8 total, 8 up, 6 in 2024-09-13T14:38:40.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:40 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:38:41.013 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:41.034 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:41.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:41.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:41.136 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:41.159 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:41.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:41.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:41.261 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:41.283 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:41.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:41.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:41.385 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:41.410 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:41.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:41.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:41.511 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:41.535 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:41.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:41.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:41.636 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:41.661 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:41.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:41.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:41.763 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:41.789 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:41.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:41.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:41.890 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:41.912 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:41.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:41.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:42.013 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:42.036 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:42.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:42.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:42.138 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:42.159 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:42.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:42.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:42.261 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:42.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:42.286 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:42.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:42.387 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:42.409 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:42.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:42.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:42.494 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:38:42.494 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:38:42.494 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:38:42.494 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:38:42.495 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:38:42.495 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:38:42.495 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:38:42.495 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:38:42.495 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:38:42.495 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:38:42.496 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:38:42.496 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:38:42.496 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:38:42.496 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:38:42.496 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:38:42.496 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:38:42.511 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:42.533 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:42.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:42.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:42.635 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:42.658 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:42.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:42.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:42.760 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:42.781 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:42.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:42.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:42.883 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:42.901 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:42 smithi102 ceph-mon[115269]: pgmap v803: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 75 active+clean; 6.5 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 123 op/s; 47662/320601 objects degraded (14.866%); 5589/320601 objects misplaced (1.743%); 10 MiB/s, 163 objects/s recovering 2024-09-13T14:38:42.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:42 smithi102 ceph-mon[110183]: pgmap v803: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 75 active+clean; 6.5 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 123 op/s; 47662/320601 objects degraded (14.866%); 5589/320601 objects misplaced (1.743%); 10 MiB/s, 163 objects/s recovering 2024-09-13T14:38:42.905 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:42.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:42.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:42.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:42 smithi195 ceph-mon[104262]: pgmap v803: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 75 active+clean; 6.5 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 123 op/s; 47662/320601 objects degraded (14.866%); 5589/320601 objects misplaced (1.743%); 10 MiB/s, 163 objects/s recovering 2024-09-13T14:38:43.008 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:43.028 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:43.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:43.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:43.130 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:43.153 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:43.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:43.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:43.256 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:43.280 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:43.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:43.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:43.382 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:43.406 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:43.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:43.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:43.508 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:43.530 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:43.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:43.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:43.632 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:43.662 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:43.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:43.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:43.726 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:38:43.764 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:43.787 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:43.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:43.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:43.888 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:43.911 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:43.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:43.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:44.013 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:44.040 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:44.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:44.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:44.141 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:44.164 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:44.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:44.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:44.266 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:44.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:44.291 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:44.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:44.392 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:44.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:44.414 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:44.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:44.516 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:44.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:44.541 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:44.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:44.642 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:44.664 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:44.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:44.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:44.766 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:44.791 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:44.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:44.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:44.893 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:44.915 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:44.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:44.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:44.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:44 smithi102 ceph-mon[115269]: pgmap v804: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 76 active+clean; 6.5 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 180 op/s; 47095/319164 objects degraded (14.756%); 5549/319164 objects misplaced (1.739%); 14 MiB/s, 231 objects/s recovering 2024-09-13T14:38:44.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:44 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 47095/319164 objects degraded (14.756%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:38:44.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:44 smithi102 ceph-mon[110183]: pgmap v804: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 76 active+clean; 6.5 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 180 op/s; 47095/319164 objects degraded (14.756%); 5549/319164 objects misplaced (1.739%); 14 MiB/s, 231 objects/s recovering 2024-09-13T14:38:44.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:44 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 47095/319164 objects degraded (14.756%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:38:44.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:44 smithi195 ceph-mon[104262]: pgmap v804: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 76 active+clean; 6.5 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 180 op/s; 47095/319164 objects degraded (14.756%); 5549/319164 objects misplaced (1.739%); 14 MiB/s, 231 objects/s recovering 2024-09-13T14:38:44.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:44 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 47095/319164 objects degraded (14.756%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:38:45.017 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:45.040 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:45.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:45.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:45.142 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:45.166 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:45.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:45.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:45.267 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:45.289 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:45.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:45.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:45.391 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:45.413 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:45.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:45.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:45.515 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:45.537 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:45.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:45.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:45.639 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:45.661 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:45.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:45.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:45.671 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:38:45.763 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:45.793 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:45.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:45.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:45.895 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:45.921 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:45.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:45.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:46.023 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:46.045 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:46.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:46.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:46.147 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:46.170 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:46.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:46.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:46.272 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:46.300 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:46.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:46.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:46.402 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:46.425 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:46.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:46.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:46.527 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:46.554 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:46.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:46.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:46.632 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:38:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:38:46] "GET /metrics HTTP/1.1" 200 36559 "" "Prometheus/2.51.0" 2024-09-13T14:38:46.656 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:46.683 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:46.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:46.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:46.784 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:46.806 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:46.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:46.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:46.908 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:46.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:46 smithi102 ceph-mon[115269]: pgmap v805: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 76 active+clean; 6.5 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 114 op/s; 47095/319164 objects degraded (14.756%); 5549/319164 objects misplaced (1.739%); 4.3 MiB/s, 77 objects/s recovering 2024-09-13T14:38:46.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:46 smithi102 ceph-mon[110183]: pgmap v805: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 76 active+clean; 6.5 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 114 op/s; 47095/319164 objects degraded (14.756%); 5549/319164 objects misplaced (1.739%); 4.3 MiB/s, 77 objects/s recovering 2024-09-13T14:38:46.932 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:46.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:46.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:46.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:46 smithi195 ceph-mon[104262]: pgmap v805: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 76 active+clean; 6.5 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 114 op/s; 47095/319164 objects degraded (14.756%); 5549/319164 objects misplaced (1.739%); 4.3 MiB/s, 77 objects/s recovering 2024-09-13T14:38:47.035 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:47.059 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:47.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:47.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:47.162 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:47.191 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:47.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:47.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:47.293 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:47.315 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:47.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:47.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:47.417 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:47.448 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:47.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:47.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:47.550 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:47.577 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:47.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:47.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:47.679 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:47.703 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:47.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:47.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:47.805 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:47.827 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:47.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:47.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:47.882 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-13T14:38:47.929 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:47.954 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:47.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:47.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:48.056 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:48.077 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:48.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:48.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:48.179 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:48.201 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:48.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:48.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:48.303 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:48.329 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:48.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:48.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:48.417 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:38:48.418 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:38:48.418 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:38:48.431 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:48.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:48.473 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:48.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:48.574 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:48.598 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:48.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:48.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:48.652 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:38:48.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:48 smithi102 ceph-mon[110183]: pgmap v806: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 76 active+clean; 6.5 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 119 op/s; 46638/317847 objects degraded (14.673%); 5473/317847 objects misplaced (1.722%); 7.3 MiB/s, 124 objects/s recovering 2024-09-13T14:38:48.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:48 smithi102 ceph-mon[115269]: pgmap v806: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 76 active+clean; 6.5 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 119 op/s; 46638/317847 objects degraded (14.673%); 5473/317847 objects misplaced (1.722%); 7.3 MiB/s, 124 objects/s recovering 2024-09-13T14:38:48.700 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:48.724 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:48.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:48.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:48.826 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:48.852 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:48.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:48.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:48.953 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:48.977 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:48.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:48.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:48.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:48 smithi195 ceph-mon[104262]: pgmap v806: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 76 active+clean; 6.5 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 119 op/s; 46638/317847 objects degraded (14.673%); 5473/317847 objects misplaced (1.722%); 7.3 MiB/s, 124 objects/s recovering 2024-09-13T14:38:49.079 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:49.103 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:49.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:49.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:49.205 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:49.230 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:49.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:49.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:49.332 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:49.359 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:49.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:49.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:49.460 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:49.486 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:49.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:49.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:49.588 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:49.613 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:49.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:49.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:49.714 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:49.740 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:49.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:49.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:49.842 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:49.866 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:49.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:49.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:49.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:49 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 46638/317847 objects degraded (14.673%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:38:49.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:49 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 46638/317847 objects degraded (14.673%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:38:49.969 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:49.994 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:49.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:49.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:49.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:49 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 46638/317847 objects degraded (14.673%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:38:50.095 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:50.121 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:50.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:50.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:50.223 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:50.244 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:50.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:50.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:50.346 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:50.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:50.372 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:50.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:50.473 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:50.494 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:50.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:50.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:50.596 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:50.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:50.621 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:50.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:50.722 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:50.746 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:50.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:50.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:50.848 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:50.870 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:50.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:50.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:50.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:50 smithi102 ceph-mon[110183]: pgmap v807: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 76 active+clean; 6.4 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 137 op/s; 46638/316881 objects degraded (14.718%); 5473/316881 objects misplaced (1.727%); 6.4 MiB/s, 110 objects/s recovering 2024-09-13T14:38:50.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:50 smithi102 ceph-mon[115269]: pgmap v807: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 76 active+clean; 6.4 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 137 op/s; 46638/316881 objects degraded (14.718%); 5473/316881 objects misplaced (1.727%); 6.4 MiB/s, 110 objects/s recovering 2024-09-13T14:38:50.972 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:50.995 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:50.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:50.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:50.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:50 smithi195 ceph-mon[104262]: pgmap v807: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 76 active+clean; 6.4 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 137 op/s; 46638/316881 objects degraded (14.718%); 5473/316881 objects misplaced (1.727%); 6.4 MiB/s, 110 objects/s recovering 2024-09-13T14:38:51.098 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:51.119 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:51.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:51.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:51.221 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:51.243 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:51.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:51.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:51.345 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:51.369 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:51.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:51.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:51.471 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:51.494 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:51.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:51.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:51.596 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:51.619 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:51.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:51.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:51.721 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:51.744 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:51.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:51.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:51.761 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:38:51.790 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T14:38:51.846 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:51.873 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:51.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:51.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:51.975 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:51.999 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:51.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:52.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:52.102 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:52.127 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:52.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:52.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:52.229 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:52.250 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:52.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:52.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:52.352 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:52.375 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:52.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:52.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:52.477 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:52.502 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:52.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:52.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:52.605 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:52.628 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:52.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:52.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:52.730 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:52.752 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:52.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:52.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:52.854 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:52.877 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:52.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:52.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:52.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:52 smithi102 ceph-mon[110183]: pgmap v808: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 76 active+clean; 6.4 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 130 op/s; 46590/316380 objects degraded (14.726%); 5473/316380 objects misplaced (1.730%); 5.7 MiB/s, 97 objects/s recovering 2024-09-13T14:38:52.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:52 smithi102 ceph-mon[115269]: pgmap v808: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 76 active+clean; 6.4 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 130 op/s; 46590/316380 objects degraded (14.726%); 5473/316380 objects misplaced (1.730%); 5.7 MiB/s, 97 objects/s recovering 2024-09-13T14:38:52.981 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:52.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:52 smithi195 ceph-mon[104262]: pgmap v808: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 76 active+clean; 6.4 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 130 op/s; 46590/316380 objects degraded (14.726%); 5473/316380 objects misplaced (1.730%); 5.7 MiB/s, 97 objects/s recovering 2024-09-13T14:38:53.003 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:53.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:53.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:53.105 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:53.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:53.129 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:53.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:53.230 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:53.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:53.257 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:53.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:53.358 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:53.380 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:53.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:53.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:53.482 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:53.506 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:53.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:53.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:53.608 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:53.630 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:53.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:53.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:53.650 INFO:teuthology.orchestra.run.smithi102.stderr:reweighted osd.7 to 1 (10000) 2024-09-13T14:38:53.732 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:53.754 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:53.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:53.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:53.855 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:53.880 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:53.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:53.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:53.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:53 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3874360587' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T14:38:53.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:53 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3874360587' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T14:38:53.982 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:53.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:53 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3874360587' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T14:38:54.008 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:54.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:54.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:54.110 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:54.132 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:54.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:54.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:54.234 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:54.258 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:54.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:54.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:54.360 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:54.385 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:54.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:54.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:54.412 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:38:54.412 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:38:54.433 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:38:54.433 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:38:54.454 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:38:54.454 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:38:54.474 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:38:54.474 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:38:54.487 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:54.493 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:38:54.493 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:38:54.510 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:38:54.510 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:38:54.511 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:54.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:54.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:54.528 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:38:54.528 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:38:54.547 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:38:54.547 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:38:54.613 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:54.637 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:54.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:54.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:54.703 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:54 smithi102 ceph-mon[115269]: pgmap v809: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 76 active+clean; 6.4 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 172 op/s; 46103/314400 objects degraded (14.664%); 5403/314400 objects misplaced (1.719%); 7.9 MiB/s, 132 objects/s recovering 2024-09-13T14:38:54.703 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:54 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3874360587' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T14:38:54.703 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:54 smithi102 ceph-mon[115269]: osdmap e253: 8 total, 8 up, 6 in 2024-09-13T14:38:54.703 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:54 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 46103/314400 objects degraded (14.664%), 11 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:38:54.703 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:54 smithi102 ceph-mon[115269]: osdmap e254: 8 total, 8 up, 6 in 2024-09-13T14:38:54.703 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:54 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3037026471' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:38:54.704 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:54 smithi102 ceph-mon[110183]: pgmap v809: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 76 active+clean; 6.4 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 172 op/s; 46103/314400 objects degraded (14.664%); 5403/314400 objects misplaced (1.719%); 7.9 MiB/s, 132 objects/s recovering 2024-09-13T14:38:54.704 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:54 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3874360587' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T14:38:54.704 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:54 smithi102 ceph-mon[110183]: osdmap e253: 8 total, 8 up, 6 in 2024-09-13T14:38:54.704 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:54 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 46103/314400 objects degraded (14.664%), 11 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:38:54.704 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:54 smithi102 ceph-mon[110183]: osdmap e254: 8 total, 8 up, 6 in 2024-09-13T14:38:54.704 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:54 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3037026471' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:38:54.739 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:54.764 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:54.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:54.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:38:54.866 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:54.893 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:54.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:54.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:54.995 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:54.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:54 smithi195 ceph-mon[104262]: pgmap v809: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 76 active+clean; 6.4 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 172 op/s; 46103/314400 objects degraded (14.664%); 5403/314400 objects misplaced (1.719%); 7.9 MiB/s, 132 objects/s recovering 2024-09-13T14:38:54.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:54 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3874360587' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T14:38:54.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:54 smithi195 ceph-mon[104262]: osdmap e253: 8 total, 8 up, 6 in 2024-09-13T14:38:54.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:54 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 46103/314400 objects degraded (14.664%), 11 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:38:54.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:54 smithi195 ceph-mon[104262]: osdmap e254: 8 total, 8 up, 6 in 2024-09-13T14:38:54.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:54 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3037026471' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:38:55.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:55.019 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:55.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:55.121 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:55.149 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:55.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:55.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:55.251 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:55.278 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:55.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:55.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:55.326 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-13T14:38:55.380 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:55.405 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:55.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:55.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:55.507 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:55.535 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:55.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:55.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:55.637 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:55.665 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:55.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:55.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:55.767 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:55.794 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:55.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:55.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:55.896 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:55.918 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:55.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:55.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:56.020 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:56.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:56.043 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:56.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:56.056 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:55 smithi102 ceph-mon[110183]: osdmap e255: 8 total, 8 up, 6 in 2024-09-13T14:38:56.056 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:55 smithi102 ceph-mon[110183]: 7.6 starting backfill to osd.7 from (0'0,0'0] MAX to 252'7633 2024-09-13T14:38:56.057 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:55 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:38:56.057 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:55 smithi102 ceph-mon[115269]: osdmap e255: 8 total, 8 up, 6 in 2024-09-13T14:38:56.057 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:55 smithi102 ceph-mon[115269]: 7.6 starting backfill to osd.7 from (0'0,0'0] MAX to 252'7633 2024-09-13T14:38:56.057 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:55 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:38:56.144 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:56.171 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:56.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:56.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:56.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:55 smithi195 ceph-mon[104262]: osdmap e255: 8 total, 8 up, 6 in 2024-09-13T14:38:56.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:55 smithi195 ceph-mon[104262]: 7.6 starting backfill to osd.7 from (0'0,0'0] MAX to 252'7633 2024-09-13T14:38:56.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:55 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:38:56.273 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:56.298 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:56.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:56.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:56.371 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:38:56.399 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:56.424 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:56.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:56.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:56.526 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:56.547 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:56.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:56.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:56.649 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:56.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:38:56 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:38:56] "GET /metrics HTTP/1.1" 200 36535 "" "Prometheus/2.51.0" 2024-09-13T14:38:56.672 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:56.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:56.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:56.774 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:56.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:56.798 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:56.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:56.895 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is unset 2024-09-13T14:38:56.899 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:56.926 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:56.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:56.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:56.957 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:56 smithi102 ceph-mon[110183]: pgmap v813: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 76 active+clean; 6.4 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 137 op/s; 46103/314400 objects degraded (14.664%); 5403/314400 objects misplaced (1.719%); 5.6 MiB/s, 89 objects/s recovering 2024-09-13T14:38:56.957 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:56 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2463665858' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:38:56.957 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:56 smithi102 ceph-mon[115269]: pgmap v813: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 76 active+clean; 6.4 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 137 op/s; 46103/314400 objects degraded (14.664%); 5403/314400 objects misplaced (1.719%); 5.6 MiB/s, 89 objects/s recovering 2024-09-13T14:38:56.957 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:56 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2463665858' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:38:57.028 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:57.052 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:57.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:57.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:57.154 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:57.168 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:56 smithi195 ceph-mon[104262]: pgmap v813: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 76 active+clean; 6.4 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 137 op/s; 46103/314400 objects degraded (14.664%); 5403/314400 objects misplaced (1.719%); 5.6 MiB/s, 89 objects/s recovering 2024-09-13T14:38:57.168 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:56 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2463665858' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:38:57.182 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:57.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:57.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:57.284 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:57.308 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:57.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:57.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:57.410 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:57.432 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:57.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:57.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:57.534 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:57.556 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:57.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:57.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:57.658 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:57.683 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:57.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:57.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:57.785 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:57.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:57.811 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:57.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:57.912 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:57.941 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:57.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:57.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:58.043 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:58.069 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:58.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:58.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:58.081 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:38:58.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:57 smithi102 ceph-mon[110183]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:38:58.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:57 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2463665858' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:38:58.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:57 smithi102 ceph-mon[110183]: osdmap e256: 8 total, 8 up, 6 in 2024-09-13T14:38:58.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:57 smithi102 ceph-mon[115269]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:38:58.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:57 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2463665858' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:38:58.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:57 smithi102 ceph-mon[115269]: osdmap e256: 8 total, 8 up, 6 in 2024-09-13T14:38:58.171 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:58.196 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:58.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:58.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:58.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:57 smithi195 ceph-mon[104262]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:38:58.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:57 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2463665858' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:38:58.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:57 smithi195 ceph-mon[104262]: osdmap e256: 8 total, 8 up, 6 in 2024-09-13T14:38:58.298 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:58.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:58.325 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:58.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:58.426 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:58.459 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:58.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:58.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:58.835 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:38:58.858 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:58.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:58.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:38:58.960 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:38:58.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:58.984 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:58.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:38:58.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:58 smithi195 ceph-mon[104262]: pgmap v815: 89 pgs: 1 peering, 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 74 active+clean; 6.4 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 74 op/s; 44882/313503 objects degraded (14.316%); 5049/313503 objects misplaced (1.611%); 23 MiB/s, 383 objects/s recovering 2024-09-13T14:38:58.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:58 smithi195 ceph-mon[104262]: osdmap e257: 8 total, 8 up, 6 in 2024-09-13T14:38:58.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:38:58 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 44882/313503 objects degraded (14.316%), 14 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:38:59.085 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:59.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:59.109 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:59.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:59.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:58 smithi102 ceph-mon[110183]: pgmap v815: 89 pgs: 1 peering, 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 74 active+clean; 6.4 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 74 op/s; 44882/313503 objects degraded (14.316%); 5049/313503 objects misplaced (1.611%); 23 MiB/s, 383 objects/s recovering 2024-09-13T14:38:59.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:58 smithi102 ceph-mon[110183]: osdmap e257: 8 total, 8 up, 6 in 2024-09-13T14:38:59.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:38:58 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 44882/313503 objects degraded (14.316%), 14 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:38:59.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:58 smithi102 ceph-mon[115269]: pgmap v815: 89 pgs: 1 peering, 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 74 active+clean; 6.4 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 74 op/s; 44882/313503 objects degraded (14.316%); 5049/313503 objects misplaced (1.611%); 23 MiB/s, 383 objects/s recovering 2024-09-13T14:38:59.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:58 smithi102 ceph-mon[115269]: osdmap e257: 8 total, 8 up, 6 in 2024-09-13T14:38:59.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:38:58 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 44882/313503 objects degraded (14.316%), 14 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:38:59.210 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:59.234 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:59.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:59.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:59.336 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:38:59.360 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:59.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:59.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:38:59.464 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:38:59.491 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:59.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:59.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:38:59.593 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:38:59.614 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:59.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:59.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:38:59.716 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:38:59.737 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:59.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:59.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:38:59.839 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:38:59.863 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:38:59.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:59.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:38:59.964 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:38:59.986 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:38:59.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:38:59.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:00.088 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:00.112 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:00.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:00.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:00.215 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:00.238 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:00.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:00.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:00.341 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:00.364 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:00.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:00.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:00.465 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:00.490 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:00.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:00.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:00.592 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:00.615 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:00.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:00.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:00.717 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:00.739 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:00.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:00.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:00.840 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:00.862 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:00.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:00.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:00.964 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:00.986 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:00.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:00.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:01.088 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:01.113 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:01.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:01.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:01.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:00 smithi102 ceph-mon[110183]: pgmap v817: 89 pgs: 1 peering, 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 74 active+clean; 6.4 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 96 op/s; 49161/312780 objects degraded (15.717%); 2730/312780 objects misplaced (0.873%); 17 MiB/s, 280 objects/s recovering 2024-09-13T14:39:01.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:00 smithi102 ceph-mon[115269]: pgmap v817: 89 pgs: 1 peering, 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 74 active+clean; 6.4 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 96 op/s; 49161/312780 objects degraded (15.717%); 2730/312780 objects misplaced (0.873%); 17 MiB/s, 280 objects/s recovering 2024-09-13T14:39:01.215 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:01.241 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:01.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:01.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:01.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:00 smithi195 ceph-mon[104262]: pgmap v817: 89 pgs: 1 peering, 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 74 active+clean; 6.4 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 96 op/s; 49161/312780 objects degraded (15.717%); 2730/312780 objects misplaced (0.873%); 17 MiB/s, 280 objects/s recovering 2024-09-13T14:39:01.343 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:01.368 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:01.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:01.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:01.470 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:01.492 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:01.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:01.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:01.594 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:01.620 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:01.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:01.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:01.722 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:01.748 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:01.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:01.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:01.850 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:01.872 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:01.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:01.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:01.974 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:01.996 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:01.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:01.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:02.098 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:02.121 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:02.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:02.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:02.223 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:02.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:02.249 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:02.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:02.263 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:39:02.263 INFO:teuthology.orchestra.run.smithi102.stdout: "size": 20, 2024-09-13T14:39:02.263 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 600, 2024-09-13T14:39:02.264 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [ 2024-09-13T14:39:02.264 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.264 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35037.0:163505 6.d 6:b0890c21:::benchmark_data_smithi102_194590_object52881:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-09-13T14:39:02.264 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:33:43.535981+0000", 2024-09-13T14:39:02.264 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 318.72461980399999, 2024-09-13T14:39:02.264 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4839642450000001, 2024-09-13T14:39:02.264 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:39:02.264 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:39:02.264 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:39:02.264 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:39:02.264 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35037", 2024-09-13T14:39:02.264 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:39:02.264 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 163505 2024-09-13T14:39:02.265 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.265 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:39:02.265 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.265 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:39:02.265 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.535981+0000", 2024-09-13T14:39:02.265 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:39:02.265 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.265 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.265 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:39:02.265 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.535981+0000", 2024-09-13T14:39:02.265 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:39:02.265 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.265 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.266 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:39:02.266 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.535983+0000", 2024-09-13T14:39:02.266 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.57e-06 2024-09-13T14:39:02.266 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.266 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.266 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:39:02.266 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.535986+0000", 2024-09-13T14:39:02.266 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.1939999999999998e-06 2024-09-13T14:39:02.266 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.266 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.266 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:39:02.266 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.535999+0000", 2024-09-13T14:39:02.266 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2639999999999999e-05 2024-09-13T14:39:02.266 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.267 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.267 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:39:02.267 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.536007+0000", 2024-09-13T14:39:02.267 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.3860000000000007e-06 2024-09-13T14:39:02.267 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.267 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.267 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:39:02.267 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.536146+0000", 2024-09-13T14:39:02.267 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00013886599999999999 2024-09-13T14:39:02.267 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.267 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.267 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:39:02.267 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.536156+0000", 2024-09-13T14:39:02.267 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0271e-05 2024-09-13T14:39:02.268 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.268 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.268 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:39:02.268 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.827509+0000", 2024-09-13T14:39:02.268 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.29135231 2024-09-13T14:39:02.268 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.268 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.268 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:39:02.268 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.827522+0000", 2024-09-13T14:39:02.268 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3206e-05 2024-09-13T14:39:02.268 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.268 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.269 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:39:02.269 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:44.986034+0000", 2024-09-13T14:39:02.269 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1585123930000001 2024-09-13T14:39:02.269 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.269 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.269 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:39:02.269 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:44.988651+0000", 2024-09-13T14:39:02.269 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.002616592 2024-09-13T14:39:02.269 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.269 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.269 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,5,6", 2024-09-13T14:39:02.270 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:44.988919+0000", 2024-09-13T14:39:02.270 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00026805299999999998 2024-09-13T14:39:02.270 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.270 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.270 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:39:02.270 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.004082+0000", 2024-09-13T14:39:02.270 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.015162881 2024-09-13T14:39:02.270 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.270 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.270 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:02.270 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.015837+0000", 2024-09-13T14:39:02.270 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.011754865 2024-09-13T14:39:02.270 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.270 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.271 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:02.271 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.016344+0000", 2024-09-13T14:39:02.271 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00050761999999999997 2024-09-13T14:39:02.271 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.271 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.271 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:02.271 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.019780+0000", 2024-09-13T14:39:02.271 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0034358370000000002 2024-09-13T14:39:02.271 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.271 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.271 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:39:02.271 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.019831+0000", 2024-09-13T14:39:02.271 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.1065e-05 2024-09-13T14:39:02.271 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.272 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.272 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:39:02.272 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.019946+0000", 2024-09-13T14:39:02.272 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000114496 2024-09-13T14:39:02.272 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:39:02.272 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:39:02.272 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:39:02.272 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.272 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.272 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35037.0:163510 6.d 6:baea5a4b:::benchmark_data_smithi102_194590_object52886:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-09-13T14:39:02.272 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:33:43.541828+0000", 2024-09-13T14:39:02.272 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 318.718772683, 2024-09-13T14:39:02.272 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.479087115, 2024-09-13T14:39:02.273 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:39:02.273 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:39:02.273 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:39:02.273 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:39:02.273 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35037", 2024-09-13T14:39:02.273 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:39:02.273 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 163510 2024-09-13T14:39:02.273 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.273 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:39:02.273 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.273 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:39:02.273 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.541828+0000", 2024-09-13T14:39:02.273 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:39:02.273 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.274 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.274 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:39:02.274 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.541828+0000", 2024-09-13T14:39:02.274 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:39:02.274 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.274 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.274 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:39:02.274 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.541830+0000", 2024-09-13T14:39:02.274 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.6789999999999999e-06 2024-09-13T14:39:02.274 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.274 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.274 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:39:02.274 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.541836+0000", 2024-09-13T14:39:02.274 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.84e-06 2024-09-13T14:39:02.275 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.275 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.275 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:39:02.275 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.541856+0000", 2024-09-13T14:39:02.275 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.0508000000000001e-05 2024-09-13T14:39:02.275 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.275 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.275 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:39:02.275 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.541866+0000", 2024-09-13T14:39:02.275 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.064e-06 2024-09-13T14:39:02.275 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.275 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.275 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:39:02.275 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.541933+0000", 2024-09-13T14:39:02.276 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.7755000000000004e-05 2024-09-13T14:39:02.276 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.276 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.276 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:39:02.276 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.541944+0000", 2024-09-13T14:39:02.276 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0468999999999999e-05 2024-09-13T14:39:02.276 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.276 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.276 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:39:02.276 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.828944+0000", 2024-09-13T14:39:02.276 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.28699979599999997 2024-09-13T14:39:02.276 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.276 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.277 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:39:02.277 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.828952+0000", 2024-09-13T14:39:02.277 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.67e-06 2024-09-13T14:39:02.277 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.277 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.277 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:39:02.277 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:44.997620+0000", 2024-09-13T14:39:02.277 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1686680250000001 2024-09-13T14:39:02.277 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.277 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.277 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:39:02.277 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:44.997737+0000", 2024-09-13T14:39:02.277 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000116585 2024-09-13T14:39:02.277 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.277 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.278 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,5,6", 2024-09-13T14:39:02.279 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:44.997915+0000", 2024-09-13T14:39:02.279 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00017786200000000001 2024-09-13T14:39:02.279 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.279 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.279 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:39:02.279 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.004087+0000", 2024-09-13T14:39:02.280 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0061725729999999998 2024-09-13T14:39:02.280 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.280 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.280 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:02.280 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.019989+0000", 2024-09-13T14:39:02.280 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.015901950000000002 2024-09-13T14:39:02.280 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.280 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.280 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:02.280 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.020036+0000", 2024-09-13T14:39:02.280 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.6854999999999998e-05 2024-09-13T14:39:02.280 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.280 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.281 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:02.281 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.020761+0000", 2024-09-13T14:39:02.281 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000724693 2024-09-13T14:39:02.281 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.281 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.281 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:39:02.281 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.020795+0000", 2024-09-13T14:39:02.281 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.4592999999999999e-05 2024-09-13T14:39:02.281 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.281 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.281 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:39:02.281 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.020916+0000", 2024-09-13T14:39:02.281 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000120198 2024-09-13T14:39:02.281 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:39:02.282 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:39:02.282 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:39:02.282 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.282 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.282 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35037.0:163513 6.d 6:b81ff245:::benchmark_data_smithi102_194590_object52889:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-09-13T14:39:02.282 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:33:43.547716+0000", 2024-09-13T14:39:02.282 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 318.71288559300001, 2024-09-13T14:39:02.282 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4780098859999999, 2024-09-13T14:39:02.282 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:39:02.282 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:39:02.282 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:39:02.282 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:39:02.282 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35037", 2024-09-13T14:39:02.282 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:39:02.283 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 163513 2024-09-13T14:39:02.283 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.283 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:39:02.283 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.283 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:39:02.283 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.547716+0000", 2024-09-13T14:39:02.283 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:39:02.283 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.283 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.283 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:39:02.283 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.547716+0000", 2024-09-13T14:39:02.283 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:39:02.283 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.283 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.284 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:39:02.284 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.547717+0000", 2024-09-13T14:39:02.284 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4410000000000001e-06 2024-09-13T14:39:02.284 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.284 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.284 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:39:02.284 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.547721+0000", 2024-09-13T14:39:02.284 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.0249999999999996e-06 2024-09-13T14:39:02.284 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.284 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.284 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:39:02.284 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.547731+0000", 2024-09-13T14:39:02.284 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0001e-05 2024-09-13T14:39:02.285 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.285 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.285 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:39:02.285 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.547738+0000", 2024-09-13T14:39:02.285 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.934e-06 2024-09-13T14:39:02.285 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.285 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.285 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:39:02.285 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.547796+0000", 2024-09-13T14:39:02.285 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.8443999999999998e-05 2024-09-13T14:39:02.285 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.285 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.285 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:39:02.285 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.547808+0000", 2024-09-13T14:39:02.286 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1603999999999999e-05 2024-09-13T14:39:02.286 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.286 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.286 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:39:02.286 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.828990+0000", 2024-09-13T14:39:02.286 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.28118225800000002 2024-09-13T14:39:02.286 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.286 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.286 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:39:02.286 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.828998+0000", 2024-09-13T14:39:02.286 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.0010000000000001e-06 2024-09-13T14:39:02.286 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.286 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.286 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:39:02.287 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.000933+0000", 2024-09-13T14:39:02.287 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1719350230000001 2024-09-13T14:39:02.287 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.287 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.287 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:39:02.287 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.001062+0000", 2024-09-13T14:39:02.287 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00012916199999999999 2024-09-13T14:39:02.287 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.287 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.287 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,5,6", 2024-09-13T14:39:02.287 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.001214+0000", 2024-09-13T14:39:02.287 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00015121500000000001 2024-09-13T14:39:02.287 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.287 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.288 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:39:02.288 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.004240+0000", 2024-09-13T14:39:02.288 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0030266469999999999 2024-09-13T14:39:02.288 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.288 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.288 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:02.288 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.020084+0000", 2024-09-13T14:39:02.288 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.015844035999999999 2024-09-13T14:39:02.288 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.288 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.288 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:02.288 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.020958+0000", 2024-09-13T14:39:02.288 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00087404400000000004 2024-09-13T14:39:02.288 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.289 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.289 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:02.289 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.025526+0000", 2024-09-13T14:39:02.289 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.004567733 2024-09-13T14:39:02.289 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.289 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.289 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:39:02.289 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.025596+0000", 2024-09-13T14:39:02.289 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.0400000000000004e-05 2024-09-13T14:39:02.289 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.289 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.289 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:39:02.289 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.025725+0000", 2024-09-13T14:39:02.289 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000128918 2024-09-13T14:39:02.290 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:39:02.290 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:39:02.290 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:39:02.290 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.290 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.290 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35037.0:163534 6.0 6:0531286c:::benchmark_data_smithi102_194590_object52910:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-09-13T14:39:02.290 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:33:43.582825+0000", 2024-09-13T14:39:02.290 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 318.67777644500001, 2024-09-13T14:39:02.290 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.450685134, 2024-09-13T14:39:02.290 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:39:02.290 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:39:02.290 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:39:02.290 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:39:02.290 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35037", 2024-09-13T14:39:02.291 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:39:02.291 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 163534 2024-09-13T14:39:02.291 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.291 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:39:02.291 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.291 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:39:02.291 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.582825+0000", 2024-09-13T14:39:02.291 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:39:02.291 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.291 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.291 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:39:02.291 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.582825+0000", 2024-09-13T14:39:02.291 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:39:02.291 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.291 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.292 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:39:02.292 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.582826+0000", 2024-09-13T14:39:02.292 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1829999999999999e-06 2024-09-13T14:39:02.292 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.292 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.292 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:39:02.292 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.582830+0000", 2024-09-13T14:39:02.292 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.6450000000000004e-06 2024-09-13T14:39:02.292 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.292 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.292 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:39:02.292 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.582844+0000", 2024-09-13T14:39:02.292 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3321e-05 2024-09-13T14:39:02.293 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.293 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.293 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:39:02.293 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.582855+0000", 2024-09-13T14:39:02.293 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1590000000000001e-05 2024-09-13T14:39:02.293 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.293 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.293 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:39:02.293 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.582917+0000", 2024-09-13T14:39:02.293 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.1248999999999996e-05 2024-09-13T14:39:02.293 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.293 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.293 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:39:02.294 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.582928+0000", 2024-09-13T14:39:02.294 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1242e-05 2024-09-13T14:39:02.294 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.294 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.294 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:39:02.294 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.829377+0000", 2024-09-13T14:39:02.294 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.24644956200000001 2024-09-13T14:39:02.294 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.294 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.294 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:39:02.294 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.829387+0000", 2024-09-13T14:39:02.294 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.5270000000000004e-06 2024-09-13T14:39:02.294 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.294 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.294 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:39:02.295 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.009379+0000", 2024-09-13T14:39:02.295 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1799916719999999 2024-09-13T14:39:02.295 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.295 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.295 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:39:02.295 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.009507+0000", 2024-09-13T14:39:02.295 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000128097 2024-09-13T14:39:02.295 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.295 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.295 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 3,4,5", 2024-09-13T14:39:02.295 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.009986+0000", 2024-09-13T14:39:02.295 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00047883499999999999 2024-09-13T14:39:02.295 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.295 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.296 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:39:02.296 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.017895+0000", 2024-09-13T14:39:02.296 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.007909737 2024-09-13T14:39:02.296 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.296 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.296 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:02.296 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.019053+0000", 2024-09-13T14:39:02.296 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0011575929999999999 2024-09-13T14:39:02.296 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.296 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.296 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:02.296 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.025397+0000", 2024-09-13T14:39:02.296 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0063443930000000003 2024-09-13T14:39:02.297 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.297 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.297 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:02.297 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.033378+0000", 2024-09-13T14:39:02.297 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0079806919999999993 2024-09-13T14:39:02.297 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.297 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.297 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:39:02.297 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.033419+0000", 2024-09-13T14:39:02.297 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.0524e-05 2024-09-13T14:39:02.297 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.297 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.297 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:39:02.297 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.033510+0000", 2024-09-13T14:39:02.298 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.1272000000000003e-05 2024-09-13T14:39:02.298 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:39:02.298 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:39:02.298 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:39:02.298 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.298 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.298 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35037.0:174000 6.a 6:59ab9fc4:::benchmark_data_smithi102_194590_object63376:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e189)", 2024-09-13T14:39:02.298 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:33:57.912783+0000", 2024-09-13T14:39:02.298 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 304.34781780499998, 2024-09-13T14:39:02.298 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0524105340000001, 2024-09-13T14:39:02.298 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:39:02.298 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:39:02.298 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:39:02.299 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:39:02.299 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35037", 2024-09-13T14:39:02.299 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:39:02.299 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 174000 2024-09-13T14:39:02.299 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.299 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:39:02.299 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.299 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:39:02.299 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:57.912783+0000", 2024-09-13T14:39:02.299 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:39:02.300 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.300 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.300 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:39:02.300 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:57.912783+0000", 2024-09-13T14:39:02.300 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:39:02.300 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.300 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.300 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:39:02.300 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:57.912784+0000", 2024-09-13T14:39:02.300 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.37e-07 2024-09-13T14:39:02.300 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.300 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.300 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:39:02.300 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:57.912786+0000", 2024-09-13T14:39:02.301 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.328e-06 2024-09-13T14:39:02.301 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.301 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.301 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:39:02.301 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:57.912793+0000", 2024-09-13T14:39:02.301 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.596e-06 2024-09-13T14:39:02.301 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.301 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.301 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:39:02.301 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:57.912799+0000", 2024-09-13T14:39:02.301 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.9329999999999998e-06 2024-09-13T14:39:02.301 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.301 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.301 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:39:02.302 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:57.914035+0000", 2024-09-13T14:39:02.302 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0012363000000000001 2024-09-13T14:39:02.302 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.302 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.302 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:39:02.302 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:57.914044+0000", 2024-09-13T14:39:02.302 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.7509999999999997e-06 2024-09-13T14:39:02.302 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.303 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.303 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:39:02.303 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:58.961292+0000", 2024-09-13T14:39:02.303 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.047248559 2024-09-13T14:39:02.303 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.303 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.303 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:39:02.303 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:58.961437+0000", 2024-09-13T14:39:02.303 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00014468099999999999 2024-09-13T14:39:02.303 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.303 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.303 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 3,5,6", 2024-09-13T14:39:02.303 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:58.961595+0000", 2024-09-13T14:39:02.303 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00015761599999999999 2024-09-13T14:39:02.304 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.304 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.304 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:39:02.304 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:58.964535+0000", 2024-09-13T14:39:02.304 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0029401269999999998 2024-09-13T14:39:02.304 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.304 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.304 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:02.304 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:58.964758+0000", 2024-09-13T14:39:02.304 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00022280299999999999 2024-09-13T14:39:02.304 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.304 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.304 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:02.304 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:58.964799+0000", 2024-09-13T14:39:02.305 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.1492999999999997e-05 2024-09-13T14:39:02.305 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.305 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.305 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:02.305 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:58.964848+0000", 2024-09-13T14:39:02.305 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.9085000000000002e-05 2024-09-13T14:39:02.305 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.305 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.305 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:39:02.305 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:58.964905+0000", 2024-09-13T14:39:02.305 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.6478999999999999e-05 2024-09-13T14:39:02.305 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.305 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.305 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:39:02.306 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:58.965194+0000", 2024-09-13T14:39:02.306 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000289246 2024-09-13T14:39:02.306 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:39:02.306 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:39:02.306 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:39:02.306 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.306 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.306 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35037.0:174019 6.a 6:5838e34f:::benchmark_data_smithi102_194590_object63395:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e189)", 2024-09-13T14:39:02.306 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:33:57.912826+0000", 2024-09-13T14:39:02.306 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 304.34777523999998, 2024-09-13T14:39:02.307 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0638388990000001, 2024-09-13T14:39:02.307 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:39:02.307 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:39:02.307 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:39:02.307 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:39:02.307 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35037", 2024-09-13T14:39:02.307 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:39:02.307 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 174019 2024-09-13T14:39:02.307 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.307 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:39:02.307 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.307 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:39:02.307 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:57.912826+0000", 2024-09-13T14:39:02.307 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:39:02.308 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.308 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.308 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:39:02.308 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:57.912826+0000", 2024-09-13T14:39:02.308 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:39:02.308 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.308 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.308 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:39:02.308 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:57.912826+0000", 2024-09-13T14:39:02.308 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.5999999999999999e-07 2024-09-13T14:39:02.308 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.308 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.308 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:39:02.309 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:57.912828+0000", 2024-09-13T14:39:02.309 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.531e-06 2024-09-13T14:39:02.309 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.309 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.309 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:39:02.309 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:57.912833+0000", 2024-09-13T14:39:02.309 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.7300000000000002e-06 2024-09-13T14:39:02.309 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.309 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.309 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:39:02.309 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:57.912838+0000", 2024-09-13T14:39:02.309 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.0010000000000001e-06 2024-09-13T14:39:02.309 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.309 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.310 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:39:02.310 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:57.914106+0000", 2024-09-13T14:39:02.310 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0012673350000000001 2024-09-13T14:39:02.310 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.310 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.310 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:39:02.310 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:57.914112+0000", 2024-09-13T14:39:02.310 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.6790000000000002e-06 2024-09-13T14:39:02.310 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.310 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.310 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:39:02.310 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:58.965232+0000", 2024-09-13T14:39:02.310 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.051120222 2024-09-13T14:39:02.310 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.311 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.311 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:39:02.311 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:58.965339+0000", 2024-09-13T14:39:02.311 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000107025 2024-09-13T14:39:02.311 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.311 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.311 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 3,5,6", 2024-09-13T14:39:02.311 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:58.965476+0000", 2024-09-13T14:39:02.311 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00013721 2024-09-13T14:39:02.311 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.311 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.311 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:39:02.311 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:58.971184+0000", 2024-09-13T14:39:02.311 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0057081279999999998 2024-09-13T14:39:02.312 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.312 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.312 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:02.312 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:58.973517+0000", 2024-09-13T14:39:02.312 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0023324550000000002 2024-09-13T14:39:02.312 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.312 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.312 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:02.312 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:58.974072+0000", 2024-09-13T14:39:02.312 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000555438 2024-09-13T14:39:02.312 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.312 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.312 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:02.313 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:58.976467+0000", 2024-09-13T14:39:02.313 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.002395267 2024-09-13T14:39:02.313 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.313 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.313 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:39:02.313 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:58.976538+0000", 2024-09-13T14:39:02.313 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.0292999999999999e-05 2024-09-13T14:39:02.313 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.313 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.313 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:39:02.313 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:58.976665+0000", 2024-09-13T14:39:02.313 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00012722499999999999 2024-09-13T14:39:02.313 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:39:02.314 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:39:02.314 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:39:02.314 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.314 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.314 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35037.0:174024 6.a 6:5c76c5c9:::benchmark_data_smithi102_194590_object63400:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e189)", 2024-09-13T14:39:02.314 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:33:57.912844+0000", 2024-09-13T14:39:02.314 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 304.34775681899998, 2024-09-13T14:39:02.314 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0641217169999999, 2024-09-13T14:39:02.314 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:39:02.314 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:39:02.314 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:39:02.314 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:39:02.314 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35037", 2024-09-13T14:39:02.315 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:39:02.315 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 174024 2024-09-13T14:39:02.315 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.315 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:39:02.315 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.315 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:39:02.315 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:57.912844+0000", 2024-09-13T14:39:02.315 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:39:02.315 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.315 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.315 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:39:02.315 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:57.912844+0000", 2024-09-13T14:39:02.315 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:39:02.315 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.316 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.316 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:39:02.316 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:57.912845+0000", 2024-09-13T14:39:02.316 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.9900000000000001e-07 2024-09-13T14:39:02.316 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.316 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.316 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:39:02.316 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:57.912846+0000", 2024-09-13T14:39:02.316 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.793e-06 2024-09-13T14:39:02.316 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.316 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.316 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:39:02.316 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:57.912871+0000", 2024-09-13T14:39:02.317 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.4148999999999999e-05 2024-09-13T14:39:02.317 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.317 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.317 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:39:02.317 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:57.912876+0000", 2024-09-13T14:39:02.317 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.4299999999999997e-06 2024-09-13T14:39:02.317 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.317 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.317 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:39:02.317 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:57.914129+0000", 2024-09-13T14:39:02.317 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.001252925 2024-09-13T14:39:02.317 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.318 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.318 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:39:02.318 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:57.914134+0000", 2024-09-13T14:39:02.318 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.3040000000000004e-06 2024-09-13T14:39:02.318 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.318 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.318 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:39:02.318 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:58.968200+0000", 2024-09-13T14:39:02.318 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0540658949999999 2024-09-13T14:39:02.318 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.318 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.318 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:39:02.318 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:58.968324+0000", 2024-09-13T14:39:02.318 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00012334800000000001 2024-09-13T14:39:02.319 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.319 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.319 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 3,5,6", 2024-09-13T14:39:02.319 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:58.968446+0000", 2024-09-13T14:39:02.319 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000122018 2024-09-13T14:39:02.319 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.319 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.319 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:39:02.319 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:58.971430+0000", 2024-09-13T14:39:02.319 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0029840930000000002 2024-09-13T14:39:02.319 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.319 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.319 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:02.319 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:58.973558+0000", 2024-09-13T14:39:02.320 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0021287210000000001 2024-09-13T14:39:02.320 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.320 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.320 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:02.320 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:58.974121+0000", 2024-09-13T14:39:02.320 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00056267600000000002 2024-09-13T14:39:02.320 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.320 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.320 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:02.320 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:58.976790+0000", 2024-09-13T14:39:02.320 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.002669449 2024-09-13T14:39:02.320 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.320 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.320 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:39:02.321 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:58.976848+0000", 2024-09-13T14:39:02.321 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.7127999999999999e-05 2024-09-13T14:39:02.321 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.321 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.321 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:39:02.321 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:58.976966+0000", 2024-09-13T14:39:02.321 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00011838899999999999 2024-09-13T14:39:02.321 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:39:02.321 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:39:02.321 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:39:02.321 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.321 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.321 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35379.0:76658 7.6 7:696a83fa:::benchmark_data_smithi102_227395_object76657:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-09-13T14:39:02.321 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:36:13.173006+0000", 2024-09-13T14:39:02.322 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 169.087594975, 2024-09-13T14:39:02.322 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.8755242050000001, 2024-09-13T14:39:02.322 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:39:02.322 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:39:02.322 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:39:02.322 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:39:02.322 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35379", 2024-09-13T14:39:02.322 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:39:02.322 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 76658 2024-09-13T14:39:02.322 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.322 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:39:02.322 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.322 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:39:02.323 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.173006+0000", 2024-09-13T14:39:02.323 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:39:02.323 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.323 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.323 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:39:02.323 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.173006+0000", 2024-09-13T14:39:02.323 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:39:02.323 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.323 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.323 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:39:02.323 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.173008+0000", 2024-09-13T14:39:02.323 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.437e-06 2024-09-13T14:39:02.323 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.323 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.324 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:39:02.324 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.173035+0000", 2024-09-13T14:39:02.324 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.7438999999999999e-05 2024-09-13T14:39:02.324 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.324 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.324 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:39:02.324 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.173054+0000", 2024-09-13T14:39:02.324 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.9281000000000001e-05 2024-09-13T14:39:02.324 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.324 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.324 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:39:02.325 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.173063+0000", 2024-09-13T14:39:02.325 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.5290000000000008e-06 2024-09-13T14:39:02.325 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.325 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.325 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:39:02.325 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.181990+0000", 2024-09-13T14:39:02.325 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0089272929999999993 2024-09-13T14:39:02.325 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.325 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.325 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:39:02.325 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.181998+0000", 2024-09-13T14:39:02.325 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.0320000000000003e-06 2024-09-13T14:39:02.325 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.326 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.326 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:39:02.326 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.860183+0000", 2024-09-13T14:39:02.326 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.67818476299999997 2024-09-13T14:39:02.326 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.326 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.326 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:39:02.326 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.860192+0000", 2024-09-13T14:39:02.326 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.8089999999999999e-06 2024-09-13T14:39:02.326 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.326 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.326 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:39:02.326 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.029027+0000", 2024-09-13T14:39:02.326 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.168835517 2024-09-13T14:39:02.327 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.327 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.327 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:39:02.327 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.029178+0000", 2024-09-13T14:39:02.327 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000150679 2024-09-13T14:39:02.327 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.327 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.327 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:39:02.327 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.029269+0000", 2024-09-13T14:39:02.327 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.1199999999999994e-05 2024-09-13T14:39:02.327 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.327 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.327 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:02.328 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.037391+0000", 2024-09-13T14:39:02.328 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0081219800000000009 2024-09-13T14:39:02.328 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.328 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.328 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:39:02.328 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.037401+0000", 2024-09-13T14:39:02.328 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.6590000000000002e-06 2024-09-13T14:39:02.329 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.329 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.329 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:02.329 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.045353+0000", 2024-09-13T14:39:02.329 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0079517930000000004 2024-09-13T14:39:02.329 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.329 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.329 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:39:02.329 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.045411+0000", 2024-09-13T14:39:02.329 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.8805999999999999e-05 2024-09-13T14:39:02.329 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.329 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.329 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:39:02.329 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.048530+0000", 2024-09-13T14:39:02.330 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0031189880000000001 2024-09-13T14:39:02.330 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:39:02.330 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:39:02.330 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:39:02.330 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.330 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.330 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35379.0:76677 7.6 7:68bf8003:::benchmark_data_smithi102_227395_object76676:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-09-13T14:39:02.330 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:36:13.189798+0000", 2024-09-13T14:39:02.330 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 169.07080336000001, 2024-09-13T14:39:02.330 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.862190115, 2024-09-13T14:39:02.330 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:39:02.330 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:39:02.331 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:39:02.331 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:39:02.331 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35379", 2024-09-13T14:39:02.331 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:39:02.331 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 76677 2024-09-13T14:39:02.331 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.331 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:39:02.331 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.331 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:39:02.331 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.189798+0000", 2024-09-13T14:39:02.331 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:39:02.331 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.332 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.332 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:39:02.332 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.189798+0000", 2024-09-13T14:39:02.332 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:39:02.332 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.332 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.332 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:39:02.332 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.189799+0000", 2024-09-13T14:39:02.332 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.22e-06 2024-09-13T14:39:02.332 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.332 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.332 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:39:02.332 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.189819+0000", 2024-09-13T14:39:02.333 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.0472999999999999e-05 2024-09-13T14:39:02.333 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.333 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.333 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:39:02.333 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.189831+0000", 2024-09-13T14:39:02.333 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1224e-05 2024-09-13T14:39:02.333 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.333 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.333 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:39:02.333 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.189837+0000", 2024-09-13T14:39:02.333 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.5139999999999998e-06 2024-09-13T14:39:02.333 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.333 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.333 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:39:02.334 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.189964+0000", 2024-09-13T14:39:02.334 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00012710600000000001 2024-09-13T14:39:02.334 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.334 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.334 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:39:02.334 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.189977+0000", 2024-09-13T14:39:02.334 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2719e-05 2024-09-13T14:39:02.334 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.334 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.334 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:39:02.334 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.860215+0000", 2024-09-13T14:39:02.336 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.67023824799999998 2024-09-13T14:39:02.336 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.336 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.336 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:39:02.336 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.860223+0000", 2024-09-13T14:39:02.336 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.3150000000000003e-06 2024-09-13T14:39:02.336 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.336 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.336 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:39:02.336 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.034495+0000", 2024-09-13T14:39:02.336 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1742722569999999 2024-09-13T14:39:02.336 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.336 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.337 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:39:02.337 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.034606+0000", 2024-09-13T14:39:02.337 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00011078699999999999 2024-09-13T14:39:02.337 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.337 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.337 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:39:02.337 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.034714+0000", 2024-09-13T14:39:02.337 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00010840099999999999 2024-09-13T14:39:02.337 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.337 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.337 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:02.337 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.037437+0000", 2024-09-13T14:39:02.337 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0027231600000000001 2024-09-13T14:39:02.338 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.338 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.339 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:39:02.339 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.045217+0000", 2024-09-13T14:39:02.339 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0077802840000000002 2024-09-13T14:39:02.339 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.339 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.339 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:02.339 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.048739+0000", 2024-09-13T14:39:02.340 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.003521621 2024-09-13T14:39:02.340 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.340 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.340 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:39:02.340 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.048781+0000", 2024-09-13T14:39:02.340 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.2409e-05 2024-09-13T14:39:02.340 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.340 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.340 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:39:02.340 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.051988+0000", 2024-09-13T14:39:02.340 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0032063769999999998 2024-09-13T14:39:02.340 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:39:02.340 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:39:02.341 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:39:02.341 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.341 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.341 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35379.0:76682 7.6 7:6caaba2d:::benchmark_data_smithi102_227395_object76681:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-09-13T14:39:02.341 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:36:13.192732+0000", 2024-09-13T14:39:02.341 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 169.067869215, 2024-09-13T14:39:02.341 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.869393316, 2024-09-13T14:39:02.341 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:39:02.341 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:39:02.341 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:39:02.341 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:39:02.341 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35379", 2024-09-13T14:39:02.342 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:39:02.342 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 76682 2024-09-13T14:39:02.342 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.342 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:39:02.342 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.342 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:39:02.342 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.192732+0000", 2024-09-13T14:39:02.342 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:39:02.342 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.342 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.342 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:39:02.342 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.192732+0000", 2024-09-13T14:39:02.342 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:39:02.343 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.343 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.343 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:39:02.343 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.192733+0000", 2024-09-13T14:39:02.343 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0920000000000001e-06 2024-09-13T14:39:02.343 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.343 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.343 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:39:02.343 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.192752+0000", 2024-09-13T14:39:02.343 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.9259999999999999e-05 2024-09-13T14:39:02.343 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.343 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.343 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:39:02.344 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.192784+0000", 2024-09-13T14:39:02.344 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.1625999999999998e-05 2024-09-13T14:39:02.344 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.344 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.344 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:39:02.344 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.192795+0000", 2024-09-13T14:39:02.344 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1371e-05 2024-09-13T14:39:02.344 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.344 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.344 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:39:02.344 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.192896+0000", 2024-09-13T14:39:02.344 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00010060699999999999 2024-09-13T14:39:02.345 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.345 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.345 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:39:02.345 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.192906+0000", 2024-09-13T14:39:02.345 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0219e-05 2024-09-13T14:39:02.345 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.345 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.345 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:39:02.345 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.860244+0000", 2024-09-13T14:39:02.345 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.66733834800000003 2024-09-13T14:39:02.345 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.345 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.346 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:39:02.346 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.860252+0000", 2024-09-13T14:39:02.346 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.2089999999999999e-06 2024-09-13T14:39:02.346 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.346 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.346 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:39:02.346 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.037460+0000", 2024-09-13T14:39:02.346 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.177207903 2024-09-13T14:39:02.346 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.346 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.346 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:39:02.346 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.037566+0000", 2024-09-13T14:39:02.346 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00010692999999999999 2024-09-13T14:39:02.346 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.347 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.347 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:39:02.347 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.037673+0000", 2024-09-13T14:39:02.347 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000106956 2024-09-13T14:39:02.347 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.347 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.347 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:02.347 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.045290+0000", 2024-09-13T14:39:02.347 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0076169189999999998 2024-09-13T14:39:02.347 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.347 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.347 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:39:02.348 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.048551+0000", 2024-09-13T14:39:02.348 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0032610320000000001 2024-09-13T14:39:02.348 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.348 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.348 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:02.348 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.058686+0000", 2024-09-13T14:39:02.348 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.010134811000000001 2024-09-13T14:39:02.348 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.348 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.348 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:39:02.348 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.058741+0000", 2024-09-13T14:39:02.348 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.4817e-05 2024-09-13T14:39:02.348 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.349 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.349 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:39:02.349 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.062125+0000", 2024-09-13T14:39:02.349 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0033842159999999998 2024-09-13T14:39:02.349 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:39:02.349 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:39:02.349 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:39:02.349 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.349 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.349 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35379.0:76646 7.e 7:7fa694df:::benchmark_data_smithi102_227395_object76645:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e215)", 2024-09-13T14:39:02.349 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:36:13.856866+0000", 2024-09-13T14:39:02.349 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 168.40373521999999, 2024-09-13T14:39:02.349 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2743484139999999, 2024-09-13T14:39:02.350 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:39:02.350 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:39:02.350 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "reached pg", 2024-09-13T14:39:02.350 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:39:02.350 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35379", 2024-09-13T14:39:02.350 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:39:02.350 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 76646 2024-09-13T14:39:02.350 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.350 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:39:02.350 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.350 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:39:02.350 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.856866+0000", 2024-09-13T14:39:02.350 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:39:02.351 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.351 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.351 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:39:02.351 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.856866+0000", 2024-09-13T14:39:02.351 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:39:02.351 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.351 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.351 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:39:02.351 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.856868+0000", 2024-09-13T14:39:02.351 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.7209999999999999e-06 2024-09-13T14:39:02.351 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.351 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.351 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:39:02.352 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.856896+0000", 2024-09-13T14:39:02.352 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.7908000000000001e-05 2024-09-13T14:39:02.352 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.352 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.352 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:39:02.352 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.856907+0000", 2024-09-13T14:39:02.352 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1449e-05 2024-09-13T14:39:02.352 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.352 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.352 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:39:02.352 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.856917+0000", 2024-09-13T14:39:02.352 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0268999999999999e-05 2024-09-13T14:39:02.352 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.353 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.353 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:39:02.353 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.860273+0000", 2024-09-13T14:39:02.353 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.003355631 2024-09-13T14:39:02.353 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.353 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.353 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:39:02.353 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.860279+0000", 2024-09-13T14:39:02.353 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.2179999999999996e-06 2024-09-13T14:39:02.353 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.353 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.353 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:39:02.353 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.131122+0000", 2024-09-13T14:39:02.353 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.270842515 2024-09-13T14:39:02.354 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.354 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.354 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:39:02.354 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.131214+0000", 2024-09-13T14:39:02.354 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.2702999999999997e-05 2024-09-13T14:39:02.354 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:39:02.354 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:39:02.354 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:39:02.354 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.354 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.354 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35379.0:76657 7.e 7:7bbd5dc3:::benchmark_data_smithi102_227395_object76656:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e215)", 2024-09-13T14:39:02.355 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:36:13.856930+0000", 2024-09-13T14:39:02.355 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 168.40367102100001, 2024-09-13T14:39:02.355 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.293766059, 2024-09-13T14:39:02.355 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:39:02.355 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:39:02.355 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:39:02.355 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:39:02.355 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35379", 2024-09-13T14:39:02.355 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:39:02.355 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 76657 2024-09-13T14:39:02.355 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.355 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:39:02.355 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.356 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:39:02.356 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.856930+0000", 2024-09-13T14:39:02.356 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:39:02.356 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.356 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.356 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:39:02.356 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.856930+0000", 2024-09-13T14:39:02.356 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:39:02.356 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.356 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.356 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:39:02.356 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.856931+0000", 2024-09-13T14:39:02.356 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.2799999999999996e-07 2024-09-13T14:39:02.357 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.357 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.357 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:39:02.357 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.856960+0000", 2024-09-13T14:39:02.357 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.9102e-05 2024-09-13T14:39:02.357 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.357 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.357 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:39:02.357 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.856968+0000", 2024-09-13T14:39:02.357 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.9060000000000002e-06 2024-09-13T14:39:02.357 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.357 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.357 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:39:02.358 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.856973+0000", 2024-09-13T14:39:02.358 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.5840000000000002e-06 2024-09-13T14:39:02.358 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.358 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.358 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:39:02.358 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.860301+0000", 2024-09-13T14:39:02.358 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.003328077 2024-09-13T14:39:02.358 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.358 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.358 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:39:02.358 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.860308+0000", 2024-09-13T14:39:02.358 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.5420000000000002e-06 2024-09-13T14:39:02.358 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.359 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.359 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:39:02.359 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.133654+0000", 2024-09-13T14:39:02.359 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.273345838 2024-09-13T14:39:02.359 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.359 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.359 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:39:02.359 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.133805+0000", 2024-09-13T14:39:02.359 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00015133799999999999 2024-09-13T14:39:02.359 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.359 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.359 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:39:02.359 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.133981+0000", 2024-09-13T14:39:02.360 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00017645599999999999 2024-09-13T14:39:02.360 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.360 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.360 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:02.360 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.143351+0000", 2024-09-13T14:39:02.360 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0093690749999999993 2024-09-13T14:39:02.360 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.360 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.360 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:02.360 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.143392+0000", 2024-09-13T14:39:02.360 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.1454e-05 2024-09-13T14:39:02.360 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.361 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.361 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:39:02.361 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.148655+0000", 2024-09-13T14:39:02.361 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0052630669999999997 2024-09-13T14:39:02.361 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.361 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.361 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:39:02.361 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.148708+0000", 2024-09-13T14:39:02.361 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.2775999999999998e-05 2024-09-13T14:39:02.361 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.361 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.361 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:39:02.361 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.150696+0000", 2024-09-13T14:39:02.361 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.001988316 2024-09-13T14:39:02.362 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:39:02.362 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:39:02.362 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:39:02.362 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.362 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.362 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35379.0:76661 7.e 7:7993c651:::benchmark_data_smithi102_227395_object76660:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e215)", 2024-09-13T14:39:02.362 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:36:13.856981+0000", 2024-09-13T14:39:02.362 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 168.403620002, 2024-09-13T14:39:02.362 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.304796106, 2024-09-13T14:39:02.362 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:39:02.362 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:39:02.362 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:39:02.362 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:39:02.363 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35379", 2024-09-13T14:39:02.363 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:39:02.363 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 76661 2024-09-13T14:39:02.363 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.363 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:39:02.363 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.363 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:39:02.363 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.856981+0000", 2024-09-13T14:39:02.363 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:39:02.363 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.363 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.363 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:39:02.363 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.856981+0000", 2024-09-13T14:39:02.363 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:39:02.364 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.364 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.364 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:39:02.364 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.856982+0000", 2024-09-13T14:39:02.364 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.8699999999999995e-07 2024-09-13T14:39:02.364 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.364 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.364 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:39:02.364 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.857003+0000", 2024-09-13T14:39:02.364 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.1554999999999999e-05 2024-09-13T14:39:02.364 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.364 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.364 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:39:02.365 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.857010+0000", 2024-09-13T14:39:02.365 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.0670000000000003e-06 2024-09-13T14:39:02.365 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.365 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.365 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:39:02.365 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.857016+0000", 2024-09-13T14:39:02.365 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.4870000000000002e-06 2024-09-13T14:39:02.365 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.365 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.365 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:39:02.365 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.860330+0000", 2024-09-13T14:39:02.365 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.003314202 2024-09-13T14:39:02.365 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.365 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.366 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:39:02.366 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.860336+0000", 2024-09-13T14:39:02.366 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.4219999999999997e-06 2024-09-13T14:39:02.366 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.366 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.366 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:39:02.366 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.137164+0000", 2024-09-13T14:39:02.366 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2768280430000001 2024-09-13T14:39:02.366 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.366 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.366 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:39:02.366 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.137332+0000", 2024-09-13T14:39:02.366 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00016787900000000001 2024-09-13T14:39:02.366 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.367 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.367 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:39:02.367 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.137500+0000", 2024-09-13T14:39:02.367 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00016808900000000001 2024-09-13T14:39:02.367 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.367 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.367 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:02.367 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.143436+0000", 2024-09-13T14:39:02.367 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0059355880000000003 2024-09-13T14:39:02.367 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.367 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.367 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:39:02.367 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.153527+0000", 2024-09-13T14:39:02.368 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.010090745 2024-09-13T14:39:02.368 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.368 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.368 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:02.368 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.159507+0000", 2024-09-13T14:39:02.368 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0059797940000000001 2024-09-13T14:39:02.368 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.368 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.368 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:39:02.368 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.159583+0000", 2024-09-13T14:39:02.368 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.6637999999999994e-05 2024-09-13T14:39:02.368 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.369 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.369 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:39:02.369 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.161777+0000", 2024-09-13T14:39:02.369 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0021940100000000001 2024-09-13T14:39:02.369 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:39:02.369 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:39:02.369 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:39:02.369 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.369 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.369 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35379.0:76679 7.e 7:708fcb06:::benchmark_data_smithi102_227395_object76678:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e215)", 2024-09-13T14:39:02.369 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:36:13.857023+0000", 2024-09-13T14:39:02.369 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 168.40357828699999, 2024-09-13T14:39:02.369 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3069522490000001, 2024-09-13T14:39:02.370 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:39:02.370 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:39:02.370 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:39:02.370 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:39:02.370 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35379", 2024-09-13T14:39:02.370 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:39:02.370 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 76679 2024-09-13T14:39:02.370 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.370 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:39:02.370 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.370 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:39:02.370 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.857023+0000", 2024-09-13T14:39:02.370 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:39:02.371 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.371 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.371 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:39:02.371 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.857023+0000", 2024-09-13T14:39:02.371 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:39:02.371 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.371 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.371 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:39:02.371 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.857023+0000", 2024-09-13T14:39:02.371 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.3099999999999998e-07 2024-09-13T14:39:02.371 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.371 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.371 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:39:02.372 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.857055+0000", 2024-09-13T14:39:02.372 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.2091999999999997e-05 2024-09-13T14:39:02.372 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.372 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.372 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:39:02.372 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.857062+0000", 2024-09-13T14:39:02.372 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.1269999999999997e-06 2024-09-13T14:39:02.372 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.372 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.372 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:39:02.372 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.857068+0000", 2024-09-13T14:39:02.372 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.4229999999999996e-06 2024-09-13T14:39:02.372 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.373 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.373 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:39:02.373 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.860358+0000", 2024-09-13T14:39:02.373 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0032896610000000001 2024-09-13T14:39:02.373 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.373 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.373 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:39:02.373 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.860364+0000", 2024-09-13T14:39:02.373 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.1999999999999999e-06 2024-09-13T14:39:02.373 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.373 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.373 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:39:02.373 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.140517+0000", 2024-09-13T14:39:02.374 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2801531640000001 2024-09-13T14:39:02.374 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.374 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.374 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:39:02.374 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.140686+0000", 2024-09-13T14:39:02.374 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00016929600000000001 2024-09-13T14:39:02.374 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.374 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.374 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:39:02.374 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.140862+0000", 2024-09-13T14:39:02.374 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00017592399999999999 2024-09-13T14:39:02.374 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.374 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.374 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:02.375 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.143482+0000", 2024-09-13T14:39:02.375 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.002619485 2024-09-13T14:39:02.375 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.375 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.375 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:39:02.375 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.153564+0000", 2024-09-13T14:39:02.375 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.010082593000000001 2024-09-13T14:39:02.375 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.375 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.375 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:02.375 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.161846+0000", 2024-09-13T14:39:02.375 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0082813079999999994 2024-09-13T14:39:02.375 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.375 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.376 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:39:02.376 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.161915+0000", 2024-09-13T14:39:02.376 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.9947e-05 2024-09-13T14:39:02.376 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.376 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.376 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:39:02.376 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.163975+0000", 2024-09-13T14:39:02.376 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0020595980000000002 2024-09-13T14:39:02.376 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:39:02.376 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:39:02.376 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:39:02.376 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.376 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.377 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35379.0:76683 7.e 7:73aa7300:::benchmark_data_smithi102_227395_object76682:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e215)", 2024-09-13T14:39:02.377 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:36:13.857074+0000", 2024-09-13T14:39:02.377 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 168.403526833, 2024-09-13T14:39:02.377 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3149687059999999, 2024-09-13T14:39:02.377 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:39:02.377 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:39:02.377 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:39:02.377 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:39:02.377 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35379", 2024-09-13T14:39:02.377 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:39:02.377 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 76683 2024-09-13T14:39:02.377 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.377 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:39:02.378 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.378 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:39:02.378 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.857074+0000", 2024-09-13T14:39:02.378 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:39:02.378 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.378 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.378 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:39:02.378 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.857074+0000", 2024-09-13T14:39:02.378 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:39:02.378 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.378 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.378 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:39:02.378 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.857075+0000", 2024-09-13T14:39:02.378 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.3599999999999999e-07 2024-09-13T14:39:02.378 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.379 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.379 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:39:02.379 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.857178+0000", 2024-09-13T14:39:02.379 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000103577 2024-09-13T14:39:02.379 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.379 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.379 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:39:02.379 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.857187+0000", 2024-09-13T14:39:02.379 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.3639999999999999e-06 2024-09-13T14:39:02.379 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.379 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.379 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:39:02.379 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.857192+0000", 2024-09-13T14:39:02.380 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.6489999999999996e-06 2024-09-13T14:39:02.380 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.380 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.380 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:39:02.380 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.860385+0000", 2024-09-13T14:39:02.380 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.003192669 2024-09-13T14:39:02.380 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.380 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.380 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:39:02.380 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.860391+0000", 2024-09-13T14:39:02.380 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.1410000000000002e-06 2024-09-13T14:39:02.380 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.381 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.381 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:39:02.381 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.143516+0000", 2024-09-13T14:39:02.381 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.283124685 2024-09-13T14:39:02.381 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.381 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.381 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:39:02.381 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.143668+0000", 2024-09-13T14:39:02.381 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00015190599999999999 2024-09-13T14:39:02.381 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.381 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.381 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:39:02.381 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.143829+0000", 2024-09-13T14:39:02.381 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000161084 2024-09-13T14:39:02.381 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.382 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.382 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:02.382 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.153510+0000", 2024-09-13T14:39:02.382 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0096813999999999997 2024-09-13T14:39:02.382 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.382 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.382 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:39:02.382 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.153577+0000", 2024-09-13T14:39:02.382 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.6601999999999997e-05 2024-09-13T14:39:02.382 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.382 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.382 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:02.382 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.169714+0000", 2024-09-13T14:39:02.383 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.016137308999999999 2024-09-13T14:39:02.383 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.383 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.383 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:39:02.383 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.169795+0000", 2024-09-13T14:39:02.383 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.0839000000000003e-05 2024-09-13T14:39:02.383 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.383 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.383 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:39:02.383 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.172043+0000", 2024-09-13T14:39:02.383 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0022480450000000002 2024-09-13T14:39:02.383 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:39:02.383 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:39:02.383 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:39:02.384 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.384 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.384 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35379.0:76688 7.e 7:7be77187:::benchmark_data_smithi102_227395_object76687:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e215)", 2024-09-13T14:39:02.384 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:36:13.857202+0000", 2024-09-13T14:39:02.384 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 168.40339957699999, 2024-09-13T14:39:02.384 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3174651050000001, 2024-09-13T14:39:02.384 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:39:02.384 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:39:02.384 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:39:02.384 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:39:02.384 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35379", 2024-09-13T14:39:02.384 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:39:02.384 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 76688 2024-09-13T14:39:02.385 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.385 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:39:02.385 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.385 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:39:02.385 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.857202+0000", 2024-09-13T14:39:02.385 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:39:02.385 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.385 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.385 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:39:02.385 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.857202+0000", 2024-09-13T14:39:02.385 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:39:02.385 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.385 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.386 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:39:02.386 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.857202+0000", 2024-09-13T14:39:02.386 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.8299999999999997e-07 2024-09-13T14:39:02.386 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.386 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.386 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:39:02.386 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.857258+0000", 2024-09-13T14:39:02.386 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.6124e-05 2024-09-13T14:39:02.386 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.386 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.386 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:39:02.386 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.857266+0000", 2024-09-13T14:39:02.386 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.644e-06 2024-09-13T14:39:02.386 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.387 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.387 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:39:02.387 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.857272+0000", 2024-09-13T14:39:02.387 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.6500000000000001e-06 2024-09-13T14:39:02.387 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.387 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.387 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:39:02.387 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.860414+0000", 2024-09-13T14:39:02.387 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0031422170000000001 2024-09-13T14:39:02.387 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.387 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.387 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:39:02.387 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.860420+0000", 2024-09-13T14:39:02.387 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.3720000000000001e-06 2024-09-13T14:39:02.388 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.388 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.388 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:39:02.388 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.146349+0000", 2024-09-13T14:39:02.388 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.285929068 2024-09-13T14:39:02.388 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.388 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.388 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:39:02.388 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.146501+0000", 2024-09-13T14:39:02.388 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000152175 2024-09-13T14:39:02.388 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.388 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.388 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:39:02.389 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.146614+0000", 2024-09-13T14:39:02.389 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00011299299999999999 2024-09-13T14:39:02.389 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.389 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.389 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:39:02.389 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.153583+0000", 2024-09-13T14:39:02.389 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0069685449999999996 2024-09-13T14:39:02.389 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.389 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.389 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:02.389 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.153628+0000", 2024-09-13T14:39:02.389 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.5163000000000001e-05 2024-09-13T14:39:02.389 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.389 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.390 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:02.390 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.172397+0000", 2024-09-13T14:39:02.390 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.018768671000000001 2024-09-13T14:39:02.390 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.390 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.390 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:39:02.390 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.172465+0000", 2024-09-13T14:39:02.390 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.8726000000000006e-05 2024-09-13T14:39:02.390 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.390 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.390 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:39:02.390 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.174667+0000", 2024-09-13T14:39:02.390 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0022011740000000002 2024-09-13T14:39:02.391 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:39:02.391 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:39:02.391 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:39:02.391 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.391 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.391 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35379.0:76689 7.0 7:0a9a6ae2:::benchmark_data_smithi102_227395_object76688:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e215)", 2024-09-13T14:39:02.391 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:36:13.857279+0000", 2024-09-13T14:39:02.391 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 168.403321858, 2024-09-13T14:39:02.391 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0959833189999999, 2024-09-13T14:39:02.391 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:39:02.391 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:39:02.391 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:39:02.391 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:39:02.392 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35379", 2024-09-13T14:39:02.392 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:39:02.392 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 76689 2024-09-13T14:39:02.392 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.392 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:39:02.392 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.392 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:39:02.392 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.857279+0000", 2024-09-13T14:39:02.392 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:39:02.392 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.392 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.392 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:39:02.392 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.857279+0000", 2024-09-13T14:39:02.392 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:39:02.393 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.393 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.393 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:39:02.393 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.857280+0000", 2024-09-13T14:39:02.393 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.13e-07 2024-09-13T14:39:02.393 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.393 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.393 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:39:02.393 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.857321+0000", 2024-09-13T14:39:02.393 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.1176e-05 2024-09-13T14:39:02.393 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.393 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.393 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:39:02.394 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.857328+0000", 2024-09-13T14:39:02.394 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.968e-06 2024-09-13T14:39:02.394 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.394 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.394 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:39:02.394 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.857333+0000", 2024-09-13T14:39:02.394 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.2299999999999999e-06 2024-09-13T14:39:02.394 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.394 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.394 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:39:02.394 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.858731+0000", 2024-09-13T14:39:02.394 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.001398332 2024-09-13T14:39:02.394 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.395 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.395 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:39:02.395 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.858739+0000", 2024-09-13T14:39:02.395 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.0059999999999992e-06 2024-09-13T14:39:02.395 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.395 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.395 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:39:02.395 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.943771+0000", 2024-09-13T14:39:02.395 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0850315859999999 2024-09-13T14:39:02.395 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.395 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.395 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:39:02.395 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.945227+0000", 2024-09-13T14:39:02.396 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.001455912 2024-09-13T14:39:02.396 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.396 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.396 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:39:02.396 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.945374+0000", 2024-09-13T14:39:02.396 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00014741299999999999 2024-09-13T14:39:02.396 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.396 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.396 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:02.396 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.947977+0000", 2024-09-13T14:39:02.396 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0026024529999999998 2024-09-13T14:39:02.396 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.396 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.397 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:02.397 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.953072+0000", 2024-09-13T14:39:02.397 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0050956550000000001 2024-09-13T14:39:02.397 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.397 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.397 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:39:02.397 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.953099+0000", 2024-09-13T14:39:02.397 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.6809999999999999e-05 2024-09-13T14:39:02.397 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.397 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.397 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:39:02.397 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.953150+0000", 2024-09-13T14:39:02.397 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.0321999999999997e-05 2024-09-13T14:39:02.398 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.398 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.398 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:39:02.398 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.953263+0000", 2024-09-13T14:39:02.398 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00011294300000000001 2024-09-13T14:39:02.398 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:39:02.398 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:39:02.398 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:39:02.398 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.398 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.398 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35379.0:76690 7.0 7:078e550a:::benchmark_data_smithi102_227395_object76689:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e215)", 2024-09-13T14:39:02.398 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:36:13.857341+0000", 2024-09-13T14:39:02.398 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 168.40326045, 2024-09-13T14:39:02.398 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.096312524, 2024-09-13T14:39:02.398 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:39:02.399 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:39:02.399 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:39:02.399 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:39:02.399 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35379", 2024-09-13T14:39:02.399 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:39:02.399 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 76690 2024-09-13T14:39:02.399 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.399 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:39:02.399 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.399 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:39:02.399 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.857341+0000", 2024-09-13T14:39:02.399 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:39:02.399 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.400 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.400 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:39:02.400 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.857341+0000", 2024-09-13T14:39:02.400 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:39:02.400 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.400 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.400 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:39:02.400 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.857341+0000", 2024-09-13T14:39:02.400 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.3700000000000001e-07 2024-09-13T14:39:02.400 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.400 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.400 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:39:02.400 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.857381+0000", 2024-09-13T14:39:02.401 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.9873000000000003e-05 2024-09-13T14:39:02.401 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.401 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.401 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:39:02.401 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.857388+0000", 2024-09-13T14:39:02.401 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.5409999999999997e-06 2024-09-13T14:39:02.401 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.401 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.401 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:39:02.401 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.857393+0000", 2024-09-13T14:39:02.401 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.2850000000000002e-06 2024-09-13T14:39:02.401 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.401 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.401 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:39:02.402 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.858771+0000", 2024-09-13T14:39:02.402 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00137802 2024-09-13T14:39:02.402 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.402 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.402 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:39:02.402 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.858778+0000", 2024-09-13T14:39:02.402 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.6900000000000003e-06 2024-09-13T14:39:02.402 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.402 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.402 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:39:02.402 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.948005+0000", 2024-09-13T14:39:02.402 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0892270829999999 2024-09-13T14:39:02.403 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.403 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.403 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:39:02.403 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.948198+0000", 2024-09-13T14:39:02.403 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00019328499999999999 2024-09-13T14:39:02.403 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.403 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.403 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:39:02.403 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.948317+0000", 2024-09-13T14:39:02.403 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000119558 2024-09-13T14:39:02.403 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.403 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.403 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:02.403 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.953353+0000", 2024-09-13T14:39:02.404 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0050355280000000001 2024-09-13T14:39:02.404 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.404 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.404 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:02.404 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.953410+0000", 2024-09-13T14:39:02.404 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.7290000000000002e-05 2024-09-13T14:39:02.404 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.404 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.404 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:39:02.404 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.953425+0000", 2024-09-13T14:39:02.404 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4688000000000001e-05 2024-09-13T14:39:02.404 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.404 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.405 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:39:02.405 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.953486+0000", 2024-09-13T14:39:02.405 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.0980999999999998e-05 2024-09-13T14:39:02.405 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.405 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.405 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:39:02.405 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.953653+0000", 2024-09-13T14:39:02.405 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00016726499999999999 2024-09-13T14:39:02.405 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:39:02.405 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:39:02.405 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:39:02.405 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.405 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.406 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35379.0:76700 7.b 7:d66e47c8:::benchmark_data_smithi102_227395_object76699:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e215)", 2024-09-13T14:39:02.406 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:36:13.857401+0000", 2024-09-13T14:39:02.406 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 168.40320043200001, 2024-09-13T14:39:02.406 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.164722195, 2024-09-13T14:39:02.406 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:39:02.406 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:39:02.406 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:39:02.406 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:39:02.406 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35379", 2024-09-13T14:39:02.406 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:39:02.406 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 76700 2024-09-13T14:39:02.406 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.406 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:39:02.406 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.407 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:39:02.407 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.857401+0000", 2024-09-13T14:39:02.407 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:39:02.407 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.407 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.407 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:39:02.407 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.857401+0000", 2024-09-13T14:39:02.407 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:39:02.407 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.407 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.407 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:39:02.407 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.857401+0000", 2024-09-13T14:39:02.407 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.7300000000000001e-07 2024-09-13T14:39:02.408 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.408 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.408 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:39:02.408 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.857446+0000", 2024-09-13T14:39:02.408 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.4421e-05 2024-09-13T14:39:02.408 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.408 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.408 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:39:02.408 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.857453+0000", 2024-09-13T14:39:02.408 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.4080000000000001e-06 2024-09-13T14:39:02.408 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.408 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.408 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:39:02.409 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.857459+0000", 2024-09-13T14:39:02.409 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.6200000000000004e-06 2024-09-13T14:39:02.409 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.409 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.409 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:39:02.409 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.857687+0000", 2024-09-13T14:39:02.409 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00022821000000000001 2024-09-13T14:39:02.409 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.409 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.409 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:39:02.409 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.857697+0000", 2024-09-13T14:39:02.409 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0000000000000001e-05 2024-09-13T14:39:02.409 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.410 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.410 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:39:02.410 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.011240+0000", 2024-09-13T14:39:02.410 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1535428809999999 2024-09-13T14:39:02.410 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.410 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.410 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:39:02.410 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.012463+0000", 2024-09-13T14:39:02.410 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0012236529999999999 2024-09-13T14:39:02.410 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.410 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.410 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,5", 2024-09-13T14:39:02.410 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.012645+0000", 2024-09-13T14:39:02.410 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000181578 2024-09-13T14:39:02.411 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.411 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.411 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:02.411 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.019384+0000", 2024-09-13T14:39:02.411 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0067395210000000001 2024-09-13T14:39:02.411 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.411 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.411 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:39:02.411 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.019394+0000", 2024-09-13T14:39:02.411 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.4050000000000006e-06 2024-09-13T14:39:02.411 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.411 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.411 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:02.412 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.019523+0000", 2024-09-13T14:39:02.412 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000129388 2024-09-13T14:39:02.412 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.412 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.412 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:39:02.412 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.019563+0000", 2024-09-13T14:39:02.412 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.9350000000000001e-05 2024-09-13T14:39:02.412 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.412 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.412 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:39:02.412 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:15.022123+0000", 2024-09-13T14:39:02.412 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0025602870000000001 2024-09-13T14:39:02.412 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:39:02.413 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:39:02.413 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:39:02.413 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.413 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.413 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35379.0:76702 7.0 7:008088d7:::benchmark_data_smithi102_227395_object76701:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e215)", 2024-09-13T14:39:02.413 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:36:13.857472+0000", 2024-09-13T14:39:02.413 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 168.403128944, 2024-09-13T14:39:02.413 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0965724830000001, 2024-09-13T14:39:02.413 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:39:02.413 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:39:02.413 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:39:02.413 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:39:02.413 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35379", 2024-09-13T14:39:02.414 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:39:02.414 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 76702 2024-09-13T14:39:02.414 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.414 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:39:02.414 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.414 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:39:02.414 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.857472+0000", 2024-09-13T14:39:02.414 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:39:02.414 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.414 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.414 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:39:02.414 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.857472+0000", 2024-09-13T14:39:02.414 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:39:02.414 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.415 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.415 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:39:02.415 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.857473+0000", 2024-09-13T14:39:02.415 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.9299999999999998e-07 2024-09-13T14:39:02.415 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.415 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.415 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:39:02.415 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.857516+0000", 2024-09-13T14:39:02.415 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.3135e-05 2024-09-13T14:39:02.415 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.415 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.415 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:39:02.415 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.857523+0000", 2024-09-13T14:39:02.416 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.0729999999999999e-06 2024-09-13T14:39:02.416 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.416 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.416 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:39:02.416 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.857528+0000", 2024-09-13T14:39:02.416 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.0860000000000001e-06 2024-09-13T14:39:02.416 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.416 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.416 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:39:02.416 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.859363+0000", 2024-09-13T14:39:02.416 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0018348290000000001 2024-09-13T14:39:02.416 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.416 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.416 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:39:02.417 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.859371+0000", 2024-09-13T14:39:02.417 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.7619999999999995e-06 2024-09-13T14:39:02.417 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.417 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.417 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:39:02.417 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.950777+0000", 2024-09-13T14:39:02.417 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.091406906 2024-09-13T14:39:02.417 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.417 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.417 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:39:02.417 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.950923+0000", 2024-09-13T14:39:02.418 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000145356 2024-09-13T14:39:02.418 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.418 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.418 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:39:02.418 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.951078+0000", 2024-09-13T14:39:02.418 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00015483399999999999 2024-09-13T14:39:02.418 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.418 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.418 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:39:02.418 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.953664+0000", 2024-09-13T14:39:02.418 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.002586292 2024-09-13T14:39:02.418 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.418 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.418 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:02.419 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.953813+0000", 2024-09-13T14:39:02.419 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00014946 2024-09-13T14:39:02.419 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.419 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.419 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:39:02.419 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.953864+0000", 2024-09-13T14:39:02.419 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.1047000000000001e-05 2024-09-13T14:39:02.419 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.419 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.419 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:39:02.419 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.953926+0000", 2024-09-13T14:39:02.419 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.1593999999999993e-05 2024-09-13T14:39:02.419 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:39:02.419 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:39:02.420 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:39:02.420 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.954045+0000", 2024-09-13T14:39:02.420 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000118616 2024-09-13T14:39:02.420 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:39:02.420 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:39:02.420 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:39:02.420 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:39:02.420 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:39:02.420 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:39:02.420 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:02.448 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:02.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:02.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:02.550 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:02.573 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:02.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:02.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:02.675 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:02.699 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:02.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:02.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:02.800 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:02.826 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:02.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:02.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:02.928 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:02.954 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:02.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:02.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:03.055 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:03.079 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:03.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:03.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:03.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:02 smithi102 ceph-mon[110183]: pgmap v818: 89 pgs: 1 peering, 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 74 active+clean; 6.4 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 100 op/s; 49119/312414 objects degraded (15.722%); 2730/312414 objects misplaced (0.874%); 14 MiB/s, 244 objects/s recovering 2024-09-13T14:39:03.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:02 smithi102 ceph-mon[115269]: pgmap v818: 89 pgs: 1 peering, 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 74 active+clean; 6.4 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 100 op/s; 49119/312414 objects degraded (15.722%); 2730/312414 objects misplaced (0.874%); 14 MiB/s, 244 objects/s recovering 2024-09-13T14:39:03.181 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:03.204 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:03.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:03.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:03.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:02 smithi195 ceph-mon[104262]: pgmap v818: 89 pgs: 1 peering, 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 74 active+clean; 6.4 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 100 op/s; 49119/312414 objects degraded (15.722%); 2730/312414 objects misplaced (0.874%); 14 MiB/s, 244 objects/s recovering 2024-09-13T14:39:03.306 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:03.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:03.330 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:03.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:03.431 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:03.455 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:03.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:03.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:03.557 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:03.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:03.590 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:03.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:03.691 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:03.721 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:03.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:03.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:03.823 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:03.846 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:03.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:03.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:03.948 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:03.973 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:03.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:03.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:04.074 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:04.099 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:04.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:04.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:04.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:03 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 49119/312414 objects degraded (15.722%), 14 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:04.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:03 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 49119/312414 objects degraded (15.722%), 14 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:04.201 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:04.223 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:04.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:04.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:04.228 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T14:39:04.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:03 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 49119/312414 objects degraded (15.722%), 14 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:04.299 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:39:04.300 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:39:04.325 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:04.347 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:04.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:04.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:04.449 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:04.474 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:04.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:04.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:04.576 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:04.603 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:04.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:04.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:04.705 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:04.727 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:04.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:04.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:04.829 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:04.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:04.851 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:04.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:04.952 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:04.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:04.976 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:04.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:05.077 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:05.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:05.100 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:05.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:05.108 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:04 smithi102 ceph-mon[110183]: pgmap v819: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.3 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 161 op/s; 49028/310533 objects degraded (15.788%); 2375/310533 objects misplaced (0.765%); 15 MiB/s, 250 objects/s recovering 2024-09-13T14:39:05.109 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:04 smithi102 ceph-mon[115269]: pgmap v819: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.3 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 161 op/s; 49028/310533 objects degraded (15.788%); 2375/310533 objects misplaced (0.765%); 15 MiB/s, 250 objects/s recovering 2024-09-13T14:39:05.200 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:05.224 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:05.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:05.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:05.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:04 smithi195 ceph-mon[104262]: pgmap v819: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.3 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 161 op/s; 49028/310533 objects degraded (15.788%); 2375/310533 objects misplaced (0.765%); 15 MiB/s, 250 objects/s recovering 2024-09-13T14:39:05.326 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:05.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:05.348 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:05.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:05.449 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:05.471 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:05.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:05.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:05.573 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:05.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:05.595 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:05.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:05.696 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:05.704 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:39:05.722 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:05.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:05.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:05.824 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:05.848 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:05.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:05.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:05.950 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:05.976 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:05.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:05.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:06.078 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:06.103 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:06.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:06.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:06.205 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:06.230 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:06.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:06.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:06.331 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:06.357 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:06.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:06.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:06.459 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:06.482 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:06.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:06.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:06.583 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:06.606 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:06.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:06.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:06.707 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:06.733 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:06.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:06.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:06.738 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:39:06 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:39:06] "GET /metrics HTTP/1.1" 200 36535 "" "Prometheus/2.51.0" 2024-09-13T14:39:06.836 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:06.861 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:06.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:06.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:06.963 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:06.986 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:06.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:06.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:07.015 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:06 smithi102 ceph-mon[110183]: pgmap v820: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.3 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 138 op/s; 49028/310533 objects degraded (15.788%); 2375/310533 objects misplaced (0.765%); 14 MiB/s, 229 objects/s recovering 2024-09-13T14:39:07.015 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:06 smithi102 ceph-mon[115269]: pgmap v820: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.3 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 138 op/s; 49028/310533 objects degraded (15.788%); 2375/310533 objects misplaced (0.765%); 14 MiB/s, 229 objects/s recovering 2024-09-13T14:39:07.088 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:07.113 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:07.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:07.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:07.215 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:07.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:07.246 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:07.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:07.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:06 smithi195 ceph-mon[104262]: pgmap v820: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.3 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 138 op/s; 49028/310533 objects degraded (15.788%); 2375/310533 objects misplaced (0.765%); 14 MiB/s, 229 objects/s recovering 2024-09-13T14:39:07.347 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:07.374 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:07.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:07.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:07.476 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:07.504 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:07.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:07.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:07.605 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:07.628 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:07.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:07.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:07.730 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:07.755 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:07.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:07.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:07.857 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:07.882 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:07.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:07.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:07.915 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:39:07.922 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:39:07.944 INFO:teuthology.orchestra.run.smithi102.stderr:reweighted osd.5 to 1 (10000) 2024-09-13T14:39:07.984 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:08.007 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:08.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:08.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:08.109 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:08.132 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:08.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:08.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:08.202 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:07 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3331997272' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:39:08.202 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:07 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:39:08.202 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:07 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3331997272' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:39:08.203 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:07 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:39:08.234 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:08.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:07 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3331997272' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:39:08.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:07 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:39:08.259 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:08.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:08.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:08.360 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:08.385 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:08.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:08.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:08.486 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:08.511 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:08.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:08.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:08.613 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:08.638 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:08.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:08.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:08.654 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:39:08.654 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:39:08.654 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:39:08.654 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:39:08.654 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:39:08.654 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:39:08.654 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:39:08.655 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:39:08.655 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:39:08.655 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:39:08.658 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:39:08.658 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:39:08.679 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:39:08.679 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:39:08.696 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:39:08.696 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:39:08.740 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:08.763 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:08.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:08.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:08.865 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:08.889 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:08.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:08.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:08.990 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:09.013 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:09.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:09.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:09.115 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:09.141 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:09.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:09.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:09.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:08 smithi102 ceph-mon[110183]: pgmap v821: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.3 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 140 op/s; 48910/309288 objects degraded (15.814%); 1997/309288 objects misplaced (0.646%); 5.7 MiB/s, 91 objects/s recovering 2024-09-13T14:39:09.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:08 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T14:39:09.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:08 smithi102 ceph-mon[110183]: osdmap e258: 8 total, 8 up, 6 in 2024-09-13T14:39:09.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:08 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/540883063' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:39:09.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:08 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 48910/309288 objects degraded (15.814%), 14 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:09.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:08 smithi102 ceph-mon[115269]: pgmap v821: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.3 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 140 op/s; 48910/309288 objects degraded (15.814%); 1997/309288 objects misplaced (0.646%); 5.7 MiB/s, 91 objects/s recovering 2024-09-13T14:39:09.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:08 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T14:39:09.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:08 smithi102 ceph-mon[115269]: osdmap e258: 8 total, 8 up, 6 in 2024-09-13T14:39:09.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:08 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/540883063' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:39:09.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:08 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 48910/309288 objects degraded (15.814%), 14 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:09.242 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:09.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:08 smithi195 ceph-mon[104262]: pgmap v821: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.3 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 140 op/s; 48910/309288 objects degraded (15.814%); 1997/309288 objects misplaced (0.646%); 5.7 MiB/s, 91 objects/s recovering 2024-09-13T14:39:09.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:08 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T14:39:09.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:08 smithi195 ceph-mon[104262]: osdmap e258: 8 total, 8 up, 6 in 2024-09-13T14:39:09.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:08 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/540883063' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:39:09.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:08 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 48910/309288 objects degraded (15.814%), 14 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:09.267 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:09.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:09.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:09.368 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:09.396 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:09.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:09.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:09.498 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:09.521 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:09.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:09.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:09.623 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:09.646 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:09.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:09.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:09.748 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:09.774 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:09.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:09.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:09.876 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:09.906 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:09.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:09.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:10.007 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:10.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:10.039 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:10.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:10.140 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:10.164 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:10.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:10.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:10.266 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:10.288 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:10.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:10.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:10.390 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:10.416 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:10.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:10.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:10.518 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:10.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:10.541 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:10.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:10.642 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:10.664 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:10.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:10.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:10.766 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:10.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:10.789 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:10.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:10.890 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:10.913 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:10.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:10.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:11.015 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:11.037 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:11.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:11.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:11.139 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:11.162 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:11.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:11.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:11.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:10 smithi102 ceph-mon[110183]: pgmap v823: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.3 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 150 op/s; 48853/308277 objects degraded (15.847%); 1997/308277 objects misplaced (0.648%); 5.9 MiB/s, 94 objects/s recovering 2024-09-13T14:39:11.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:39:11.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:39:11.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:10 smithi102 ceph-mon[115269]: pgmap v823: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.3 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 150 op/s; 48853/308277 objects degraded (15.847%); 1997/308277 objects misplaced (0.648%); 5.9 MiB/s, 94 objects/s recovering 2024-09-13T14:39:11.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:39:11.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:39:11.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:10 smithi195 ceph-mon[104262]: pgmap v823: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.3 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 150 op/s; 48853/308277 objects degraded (15.847%); 1997/308277 objects misplaced (0.648%); 5.9 MiB/s, 94 objects/s recovering 2024-09-13T14:39:11.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:39:11.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:39:11.264 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:11.288 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:11.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:11.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:11.379 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-13T14:39:11.379 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:39:11.390 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:11.413 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:11.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:11.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:11.515 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:11.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:11.539 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:11.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:11.639 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:11.663 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:11.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:11.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:11.765 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:11.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:11.788 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:11.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:11.864 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:39:11.864 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:39:11.888 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:11.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:11.911 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:11.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:12.013 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:12.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:12.035 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:12.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:12.137 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:12.161 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:12.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:12.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:12.263 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:12.284 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:12.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:12.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:12.386 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:12.408 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:12.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:12.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:12.510 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:12.533 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:12.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:12.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:12.535 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T14:39:12.635 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:12.661 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:12.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:12.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:12.762 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:12.785 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:12.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:12.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:12.887 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:12.909 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:12.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:12.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:13.011 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:13.036 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:13.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:13.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:13.138 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:13.161 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:13.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:13.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:13.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:12 smithi102 ceph-mon[110183]: pgmap v824: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.3 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 153 op/s; 48812/307803 objects degraded (15.858%); 1997/307803 objects misplaced (0.649%); 5.9 MiB/s, 94 objects/s recovering 2024-09-13T14:39:13.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:12 smithi102 ceph-mon[115269]: pgmap v824: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.3 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 153 op/s; 48812/307803 objects degraded (15.858%); 1997/307803 objects misplaced (0.649%); 5.9 MiB/s, 94 objects/s recovering 2024-09-13T14:39:13.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:12 smithi195 ceph-mon[104262]: pgmap v824: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.3 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 153 op/s; 48812/307803 objects degraded (15.858%); 1997/307803 objects misplaced (0.649%); 5.9 MiB/s, 94 objects/s recovering 2024-09-13T14:39:13.263 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:13.287 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:13.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:13.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:13.389 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:13.414 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:13.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:13.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:13.516 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:13.538 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:13.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:13.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:13.639 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:13.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:13.661 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:13.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:13.762 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:13.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:13.786 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:13.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:13.888 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:13.912 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:13.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:13.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:14.014 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:14.040 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:14.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:14.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:14.142 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:14.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:13 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 48812/307803 objects degraded (15.858%), 14 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:14.154 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:13 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 48812/307803 objects degraded (15.858%), 14 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:14.167 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:14.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:14.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:14.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:13 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 48812/307803 objects degraded (15.858%), 14 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:14.269 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:14.296 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:14.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:14.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:14.398 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:14.425 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:14.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:14.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:14.492 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:39:14.527 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:14.555 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:14.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:14.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:14.634 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:39:14.658 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:14.683 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:14.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:14.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:14.785 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:14.810 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:14.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:14.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:14.912 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:14.938 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:14.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:14.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:15.040 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:15.065 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:15.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:15.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:15.067 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:14 smithi102 ceph-mon[110183]: pgmap v825: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.2 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 158 op/s; 48276/305781 objects degraded (15.788%); 1997/305781 objects misplaced (0.653%); 6.4 MiB/s, 102 objects/s recovering 2024-09-13T14:39:15.067 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:14 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:39:15.067 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:14 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:39:15.068 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:14 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/4177080976' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:39:15.068 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:14 smithi102 ceph-mon[115269]: pgmap v825: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.2 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 158 op/s; 48276/305781 objects degraded (15.788%); 1997/305781 objects misplaced (0.653%); 6.4 MiB/s, 102 objects/s recovering 2024-09-13T14:39:15.068 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:14 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:39:15.068 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:14 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:39:15.069 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:14 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/4177080976' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:39:15.167 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:15.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:15.194 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:15.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:15.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:14 smithi195 ceph-mon[104262]: pgmap v825: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.2 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 158 op/s; 48276/305781 objects degraded (15.788%); 1997/305781 objects misplaced (0.653%); 6.4 MiB/s, 102 objects/s recovering 2024-09-13T14:39:15.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:14 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:39:15.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:14 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:39:15.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:14 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/4177080976' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:39:15.295 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:15.319 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:15.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:15.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:15.403 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is set 2024-09-13T14:39:15.420 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:15.444 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:15.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:15.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:15.546 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:15.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:15.569 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:15.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:15.639 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:39:15.670 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:15.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:15.694 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:15.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:15.795 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:15.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:15.821 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:15.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:15.922 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:15.947 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:15.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:15.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:16.049 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:16.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:16.075 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:16.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:16.176 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:16.200 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:16.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:16.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:16.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:15 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:39:16.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:15 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:39:16.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:15 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:39:16.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:15 smithi195 ceph-mon[104262]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:39:16.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:15 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/4177080976' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:39:16.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:15 smithi195 ceph-mon[104262]: osdmap e259: 8 total, 8 up, 6 in 2024-09-13T14:39:16.277 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:15 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:39:16.278 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:15 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:39:16.278 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:15 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:39:16.278 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:15 smithi102 ceph-mon[115269]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:39:16.278 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:15 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/4177080976' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:39:16.278 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:15 smithi102 ceph-mon[115269]: osdmap e259: 8 total, 8 up, 6 in 2024-09-13T14:39:16.278 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:15 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:39:16.278 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:15 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:39:16.278 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:15 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:39:16.278 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:15 smithi102 ceph-mon[110183]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:39:16.278 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:15 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/4177080976' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:39:16.278 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:15 smithi102 ceph-mon[110183]: osdmap e259: 8 total, 8 up, 6 in 2024-09-13T14:39:16.302 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:16.324 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:16.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:16.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:16.426 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:16.450 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:16.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:16.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:16.552 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:16.579 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:16.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:16.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:16.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:39:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:39:16] "GET /metrics HTTP/1.1" 200 36541 "" "Prometheus/2.51.0" 2024-09-13T14:39:16.682 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:16.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:16.711 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:16.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:16.812 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:16.842 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:16.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:16.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:16.944 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:16.969 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:16.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:16.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:17.070 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:17.095 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:17.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:17.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:17.157 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:16 smithi102 ceph-mon[110183]: pgmap v827: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.2 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 146 op/s; 48276/305781 objects degraded (15.788%); 1997/305781 objects misplaced (0.653%); 4.5 MiB/s, 72 objects/s recovering 2024-09-13T14:39:17.157 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:16 smithi102 ceph-mon[115269]: pgmap v827: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.2 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 146 op/s; 48276/305781 objects degraded (15.788%); 1997/305781 objects misplaced (0.653%); 4.5 MiB/s, 72 objects/s recovering 2024-09-13T14:39:17.196 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:17.217 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:17.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:17.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:17.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:16 smithi195 ceph-mon[104262]: pgmap v827: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.2 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 146 op/s; 48276/305781 objects degraded (15.788%); 1997/305781 objects misplaced (0.653%); 4.5 MiB/s, 72 objects/s recovering 2024-09-13T14:39:17.319 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:17.342 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:17.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:17.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:17.444 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:17.468 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:17.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:17.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:17.570 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:17.593 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:17.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:17.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:17.695 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:17.720 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:17.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:17.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:17.822 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:17.848 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:17.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:17.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:17.950 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:17.975 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:17.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:17.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:18.077 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:18.098 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:18.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:18.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:18.201 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:18.224 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:18.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:18.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:18.326 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:18.350 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:18.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:18.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:18.406 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:39:18.421 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:39:18.421 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:39:18.453 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:18.475 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:18.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:18.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:18.578 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:18.603 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:18.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:18.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:18.705 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:18.726 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:18.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:18.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:18.828 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:18.851 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:18.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:18.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:18.953 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:18.978 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:18.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:18.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:19.056 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:18 smithi102 ceph-mon[110183]: pgmap v828: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.2 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 168 op/s; 47716/304347 objects degraded (15.678%); 1997/304347 objects misplaced (0.656%); 7.1 MiB/s, 113 objects/s recovering 2024-09-13T14:39:19.057 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:18 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 47716/304347 objects degraded (15.678%), 14 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:19.057 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:18 smithi102 ceph-mon[115269]: pgmap v828: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.2 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 168 op/s; 47716/304347 objects degraded (15.678%); 1997/304347 objects misplaced (0.656%); 7.1 MiB/s, 113 objects/s recovering 2024-09-13T14:39:19.057 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:18 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 47716/304347 objects degraded (15.678%), 14 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:19.079 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:19.104 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:19.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:19.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:19.182 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:39:19.207 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:19.231 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:19.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:19.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:19.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:18 smithi195 ceph-mon[104262]: pgmap v828: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.2 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 168 op/s; 47716/304347 objects degraded (15.678%); 1997/304347 objects misplaced (0.656%); 7.1 MiB/s, 113 objects/s recovering 2024-09-13T14:39:19.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:18 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 47716/304347 objects degraded (15.678%), 14 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:19.334 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:19.357 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:19.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:19.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:19.371 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:39:19.372 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:39:19.460 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:19.483 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:19.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:19.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:19.585 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:19.610 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:19.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:19.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:19.711 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:19.734 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:19.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:19.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:19.836 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:19.860 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T14:39:19.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:19.861 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:19.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:19.962 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:19.987 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:19.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:19.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:20.088 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:20.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:20.115 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:20.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:20.216 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:20.241 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:20.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:20.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:20.343 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:20.368 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:20.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:20.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:20.470 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:20.498 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:20.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:20.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:20.508 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":828,"stamp":"2024-09-13T14:39:17.457234+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6651451389,"num_objects":101449,"num_object_clones":0,"num_object_copies":304347,"num_objects_missing_on_primary":4311,"num_objects_missing":4311,"num_objects_degraded":47716,"num_objects_misplaced":1997,"num_objects_unfound":0,"num_objects_dirty":101449,"num_whiteouts":0,"num_read":6377,"num_read_kb":13834,"num_write":125328,"num_write_kb":7235010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46507,"num_bytes_recovered":3024817269,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88110,"ondisk_log_size":88110,"up":267,"acting":259,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":309,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":22622884,"kb_used_data":19734316,"kb_used_omap":26,"kb_used_meta":2888485,"kb_avail":539848028,"statfs":{"total":575970213888,"available":552804380672,"internally_reserved":0,"allocated":20207939584,"data_stored":20199556986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27144,"internal_metadata":2957809144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-104857600,"num_objects":-1600,"num_object_clones":0,"num_object_copies":-4800,"num_objects_missing_on_primary":-84,"num_objects_missing":-84,"num_objects_degraded":-1194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1600,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1083,"num_bytes_recovered":70975488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.520169"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":410,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.421501+0000","last_change":"2024-09-13T14:36:13.859927+0000","last_active":"2024-09-13T14:39:15.421501+0000","last_peered":"2024-09-13T14:39:15.421501+0000","last_clean":"2024-09-13T14:39:15.421501+0000","last_became_active":"2024-09-13T14:36:13.859505+0000","last_became_peered":"2024-09-13T14:36:13.859505+0000","last_unstale":"2024-09-13T14:39:15.421501+0000","last_undegraded":"2024-09-13T14:39:15.421501+0000","last_fullsized":"2024-09-13T14:39:15.421501+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:12:20.989627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":259,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:39:07.944104+0000","last_change":"2024-09-13T14:37:48.130968+0000","last_active":"2024-09-13T14:39:07.944104+0000","last_peered":"2024-09-13T14:39:07.944104+0000","last_clean":"2024-09-13T14:39:07.944104+0000","last_became_active":"2024-09-13T14:37:48.130603+0000","last_became_peered":"2024-09-13T14:37:48.130603+0000","last_unstale":"2024-09-13T14:39:07.944104+0000","last_undegraded":"2024-09-13T14:39:07.944104+0000","last_fullsized":"2024-09-13T14:39:07.944104+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:42.626386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":56,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:39:07.964129+0000","last_change":"2024-09-13T14:36:13.862181+0000","last_active":"2024-09-13T14:39:07.964129+0000","last_peered":"2024-09-13T14:39:07.964129+0000","last_clean":"2024-09-13T14:39:07.964129+0000","last_became_active":"2024-09-13T14:36:13.861873+0000","last_became_peered":"2024-09-13T14:36:13.861873+0000","last_unstale":"2024-09-13T14:39:07.964129+0000","last_undegraded":"2024-09-13T14:39:07.964129+0000","last_fullsized":"2024-09-13T14:39:07.964129+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:14.085700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":45,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:39:07.949985+0000","last_change":"2024-09-13T14:38:53.896672+0000","last_active":"2024-09-13T14:39:07.949985+0000","last_peered":"2024-09-13T14:39:07.949985+0000","last_clean":"2024-09-13T14:39:07.949985+0000","last_became_active":"2024-09-13T14:38:53.896346+0000","last_became_peered":"2024-09-13T14:38:53.896346+0000","last_unstale":"2024-09-13T14:39:07.949985+0000","last_undegraded":"2024-09-13T14:39:07.949985+0000","last_fullsized":"2024-09-13T14:39:07.949985+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:57.789686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":45,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:39:07.950039+0000","last_change":"2024-09-13T14:38:53.893021+0000","last_active":"2024-09-13T14:39:07.950039+0000","last_peered":"2024-09-13T14:39:07.950039+0000","last_clean":"2024-09-13T14:39:07.950039+0000","last_became_active":"2024-09-13T14:38:53.892668+0000","last_became_peered":"2024-09-13T14:38:53.892668+0000","last_unstale":"2024-09-13T14:39:07.950039+0000","last_undegraded":"2024-09-13T14:39:07.950039+0000","last_fullsized":"2024-09-13T14:39:07.950039+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:57.180738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":227,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:39:07.943912+0000","last_change":"2024-09-13T14:37:48.134642+0000","last_active":"2024-09-13T14:39:07.943912+0000","last_peered":"2024-09-13T14:39:07.943912+0000","last_clean":"2024-09-13T14:39:07.943912+0000","last_became_active":"2024-09-13T14:37:48.134402+0000","last_became_peered":"2024-09-13T14:37:48.134402+0000","last_unstale":"2024-09-13T14:39:07.943912+0000","last_undegraded":"2024-09-13T14:39:07.943912+0000","last_fullsized":"2024-09-13T14:39:07.943912+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:51.313663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":428,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.421527+0000","last_change":"2024-09-13T14:38:53.903022+0000","last_active":"2024-09-13T14:39:15.421527+0000","last_peered":"2024-09-13T14:39:15.421527+0000","last_clean":"2024-09-13T14:39:15.421527+0000","last_became_active":"2024-09-13T14:38:53.902448+0000","last_became_peered":"2024-09-13T14:38:53.902448+0000","last_unstale":"2024-09-13T14:39:15.421527+0000","last_undegraded":"2024-09-13T14:39:15.421527+0000","last_fullsized":"2024-09-13T14:39:15.421527+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:24.690401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":48,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:39:07.949030+0000","last_change":"2024-09-13T14:38:03.554500+0000","last_active":"2024-09-13T14:39:07.949030+0000","last_peered":"2024-09-13T14:39:07.949030+0000","last_clean":"2024-09-13T14:39:07.949030+0000","last_became_active":"2024-09-13T14:38:03.532910+0000","last_became_peered":"2024-09-13T14:38:03.532910+0000","last_unstale":"2024-09-13T14:39:07.949030+0000","last_undegraded":"2024-09-13T14:39:07.949030+0000","last_fullsized":"2024-09-13T14:39:07.949030+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:27:50.843498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"259'7667","reported_seq":9750,"reported_epoch":259,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:39:15.986956+0000","last_change":"2024-09-13T14:37:49.715669+0000","last_active":"2024-09-13T14:39:15.986956+0000","last_peered":"2024-09-13T14:39:15.986956+0000","last_clean":"2024-09-13T14:37:46.417003+0000","last_became_active":"2024-09-13T14:37:49.690961+0000","last_became_peered":"2024-09-13T14:37:49.690961+0000","last_unstale":"2024-09-13T14:39:15.986956+0000","last_undegraded":"2024-09-13T14:37:49.182535+0000","last_fullsized":"2024-09-13T14:37:49.126746+0000","mapping_epoch":240,"log_start":"210'2000","ondisk_log_start":"210'2000","created":205,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5667,"log_dups_size":0,"ondisk_log_size":5667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:48:52.526429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":407175168,"num_objects":6213,"num_object_clones":0,"num_object_copies":18639,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4387,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7667,"num_write_kb":444160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4765,"num_bytes_recovered":312279040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,4],"avail_no_missing":["5","0","4"],"object_location_counts":[{"shards":"0,4,5","objects":4308},{"shards":"4,5","objects":1905}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":20,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:39:07.944484+0000","last_change":"2024-09-13T14:38:53.940238+0000","last_active":"2024-09-13T14:39:07.944484+0000","last_peered":"2024-09-13T14:39:07.944484+0000","last_clean":"2024-09-13T14:39:07.944484+0000","last_became_active":"2024-09-13T14:38:53.892825+0000","last_became_peered":"2024-09-13T14:38:53.892825+0000","last_unstale":"2024-09-13T14:39:07.944484+0000","last_undegraded":"2024-09-13T14:39:07.944484+0000","last_fullsized":"2024-09-13T14:39:07.944484+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:46.666581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":76,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:39:07.962822+0000","last_change":"2024-09-13T14:38:53.898907+0000","last_active":"2024-09-13T14:39:07.962822+0000","last_peered":"2024-09-13T14:39:07.962822+0000","last_clean":"2024-09-13T14:39:07.962822+0000","last_became_active":"2024-09-13T14:38:53.898572+0000","last_became_peered":"2024-09-13T14:38:53.898572+0000","last_unstale":"2024-09-13T14:39:07.962822+0000","last_undegraded":"2024-09-13T14:39:07.962822+0000","last_fullsized":"2024-09-13T14:39:07.962822+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:29.222150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"259'7902","reported_seq":22028,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:17.253118+0000","last_change":"2024-09-13T14:38:57.930032+0000","last_active":"2024-09-13T14:39:17.253118+0000","last_peered":"2024-09-13T14:39:17.253118+0000","last_clean":"2024-09-13T14:39:17.253118+0000","last_became_active":"2024-09-13T14:38:57.921458+0000","last_became_peered":"2024-09-13T14:38:57.921458+0000","last_unstale":"2024-09-13T14:39:17.253118+0000","last_undegraded":"2024-09-13T14:39:17.253118+0000","last_fullsized":"2024-09-13T14:39:17.253118+0000","mapping_epoch":256,"log_start":"212'3139","ondisk_log_start":"212'3139","created":205,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":4763,"log_dups_size":0,"ondisk_log_size":4763,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:27:12.294038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":415498240,"num_objects":6340,"num_object_clones":0,"num_object_copies":19020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7902,"num_write_kb":455744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6990,"num_bytes_recovered":458096640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":20,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:39:07.944450+0000","last_change":"2024-09-13T14:38:53.904885+0000","last_active":"2024-09-13T14:39:07.944450+0000","last_peered":"2024-09-13T14:39:07.944450+0000","last_clean":"2024-09-13T14:39:07.944450+0000","last_became_active":"2024-09-13T14:38:53.891726+0000","last_became_peered":"2024-09-13T14:38:53.891726+0000","last_unstale":"2024-09-13T14:39:07.944450+0000","last_undegraded":"2024-09-13T14:39:07.944450+0000","last_fullsized":"2024-09-13T14:39:07.944450+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:57.079205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":264,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:38:57.912059+0000","last_change":"2024-09-13T14:37:48.160504+0000","last_active":"2024-09-13T14:38:57.912059+0000","last_peered":"2024-09-13T14:38:57.912059+0000","last_clean":"2024-09-13T14:38:57.912059+0000","last_became_active":"2024-09-13T14:37:48.160181+0000","last_became_peered":"2024-09-13T14:37:48.160181+0000","last_unstale":"2024-09-13T14:38:57.912059+0000","last_undegraded":"2024-09-13T14:38:57.912059+0000","last_fullsized":"2024-09-13T14:38:57.912059+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:37.087008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"258'7779","reported_seq":9525,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:39:12.645994+0000","last_change":"2024-09-13T14:38:04.531914+0000","last_active":"2024-09-13T14:39:12.645994+0000","last_peered":"2024-09-13T14:39:12.645994+0000","last_clean":"2024-09-13T14:39:12.645994+0000","last_became_active":"2024-09-13T14:38:04.523448+0000","last_became_peered":"2024-09-13T14:38:04.523448+0000","last_unstale":"2024-09-13T14:39:12.645994+0000","last_undegraded":"2024-09-13T14:39:12.645994+0000","last_fullsized":"2024-09-13T14:39:12.645994+0000","mapping_epoch":245,"log_start":"210'2000","ondisk_log_start":"210'2000","created":205,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5779,"log_dups_size":0,"ondisk_log_size":5779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:06.536896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":419102720,"num_objects":6395,"num_object_clones":0,"num_object_copies":19185,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7779,"num_write_kb":453568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":811,"num_bytes_recovered":53149696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":257,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:39:07.948680+0000","last_change":"2024-09-13T14:37:49.611774+0000","last_active":"2024-09-13T14:39:07.948680+0000","last_peered":"2024-09-13T14:39:07.948680+0000","last_clean":"2024-09-13T14:39:07.948680+0000","last_became_active":"2024-09-13T14:37:49.611088+0000","last_became_peered":"2024-09-13T14:37:49.611088+0000","last_unstale":"2024-09-13T14:39:07.948680+0000","last_undegraded":"2024-09-13T14:39:07.948680+0000","last_fullsized":"2024-09-13T14:39:07.948680+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:08.933213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":16,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:39:07.943833+0000","last_change":"2024-09-13T14:38:53.899211+0000","last_active":"2024-09-13T14:39:07.943833+0000","last_peered":"2024-09-13T14:39:07.943833+0000","last_clean":"2024-09-13T14:39:07.943833+0000","last_became_active":"2024-09-13T14:38:53.898927+0000","last_became_peered":"2024-09-13T14:38:53.898927+0000","last_unstale":"2024-09-13T14:39:07.943833+0000","last_undegraded":"2024-09-13T14:39:07.943833+0000","last_fullsized":"2024-09-13T14:39:07.943833+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:00:28.857611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"259'7732","reported_seq":7796,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:17.271978+0000","last_change":"2024-09-13T14:35:24.004651+0000","last_active":"2024-09-13T14:39:17.271978+0000","last_peered":"2024-09-13T14:39:17.271978+0000","last_clean":"2024-09-13T14:39:17.271978+0000","last_became_active":"2024-09-13T14:35:24.004358+0000","last_became_peered":"2024-09-13T14:35:24.004358+0000","last_unstale":"2024-09-13T14:39:17.271978+0000","last_undegraded":"2024-09-13T14:39:17.271978+0000","last_fullsized":"2024-09-13T14:39:17.271978+0000","mapping_epoch":205,"log_start":"209'1384","ondisk_log_start":"209'1384","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6348,"log_dups_size":0,"ondisk_log_size":6348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:57:38.023669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":408944640,"num_objects":6240,"num_object_clones":0,"num_object_copies":18720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7732,"num_write_kb":447104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":67,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:39:07.963659+0000","last_change":"2024-09-13T14:37:49.168003+0000","last_active":"2024-09-13T14:39:07.963659+0000","last_peered":"2024-09-13T14:39:07.963659+0000","last_clean":"2024-09-13T14:39:07.963659+0000","last_became_active":"2024-09-13T14:37:49.167638+0000","last_became_peered":"2024-09-13T14:37:49.167638+0000","last_unstale":"2024-09-13T14:39:07.963659+0000","last_undegraded":"2024-09-13T14:39:07.963659+0000","last_fullsized":"2024-09-13T14:39:07.963659+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:29.051981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":38,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:39:07.950587+0000","last_change":"2024-09-13T14:38:03.528381+0000","last_active":"2024-09-13T14:39:07.950587+0000","last_peered":"2024-09-13T14:39:07.950587+0000","last_clean":"2024-09-13T14:39:07.950587+0000","last_became_active":"2024-09-13T14:38:03.527282+0000","last_became_peered":"2024-09-13T14:38:03.527282+0000","last_unstale":"2024-09-13T14:39:07.950587+0000","last_undegraded":"2024-09-13T14:39:07.950587+0000","last_fullsized":"2024-09-13T14:39:07.950587+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:24.435067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.f","version":"258'7705","reported_seq":7946,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:39:11.837041+0000","last_change":"2024-09-13T14:38:16.371079+0000","last_active":"2024-09-13T14:39:11.837041+0000","last_peered":"2024-09-13T14:39:11.837041+0000","last_clean":"2024-09-13T14:39:11.837041+0000","last_became_active":"2024-09-13T14:38:12.859883+0000","last_became_peered":"2024-09-13T14:38:12.859883+0000","last_unstale":"2024-09-13T14:39:11.837041+0000","last_undegraded":"2024-09-13T14:39:11.837041+0000","last_fullsized":"2024-09-13T14:39:11.837041+0000","mapping_epoch":245,"log_start":"212'2663","ondisk_log_start":"212'2663","created":205,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5042,"log_dups_size":0,"ondisk_log_size":5042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:08.436532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":418185216,"num_objects":6381,"num_object_clones":0,"num_object_copies":19143,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7705,"num_write_kb":450752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":3670016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":398,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.421707+0000","last_change":"2024-09-13T14:38:03.549595+0000","last_active":"2024-09-13T14:39:15.421707+0000","last_peered":"2024-09-13T14:39:15.421707+0000","last_clean":"2024-09-13T14:39:15.421707+0000","last_became_active":"2024-09-13T14:38:03.533331+0000","last_became_peered":"2024-09-13T14:38:03.533331+0000","last_unstale":"2024-09-13T14:39:15.421707+0000","last_undegraded":"2024-09-13T14:39:15.421707+0000","last_fullsized":"2024-09-13T14:39:15.421707+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:59:02.470343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":240,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:38:57.912949+0000","last_change":"2024-09-13T14:37:48.127600+0000","last_active":"2024-09-13T14:38:57.912949+0000","last_peered":"2024-09-13T14:38:57.912949+0000","last_clean":"2024-09-13T14:38:57.912949+0000","last_became_active":"2024-09-13T14:37:48.127349+0000","last_became_peered":"2024-09-13T14:37:48.127349+0000","last_unstale":"2024-09-13T14:38:57.912949+0000","last_undegraded":"2024-09-13T14:38:57.912949+0000","last_fullsized":"2024-09-13T14:38:57.912949+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:57.268374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"259'7665","reported_seq":9082,"reported_epoch":259,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:39:15.987349+0000","last_change":"2024-09-13T14:38:14.979229+0000","last_active":"2024-09-13T14:39:15.987349+0000","last_peered":"2024-09-13T14:39:15.987349+0000","last_clean":"2024-09-13T14:36:13.131873+0000","last_became_active":"2024-09-13T14:38:10.443954+0000","last_became_peered":"2024-09-13T14:38:10.443954+0000","last_unstale":"2024-09-13T14:39:15.987349+0000","last_undegraded":"2024-09-13T14:38:10.384643+0000","last_fullsized":"2024-09-13T14:38:04.517046+0000","mapping_epoch":245,"log_start":"210'2000","ondisk_log_start":"210'2000","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5665,"log_dups_size":0,"ondisk_log_size":5665,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:26:53.493170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":415825920,"num_objects":6345,"num_object_clones":0,"num_object_copies":19035,"num_objects_missing_on_primary":4121,"num_objects_missing":4121,"num_objects_degraded":4262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7665,"num_write_kb":448320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":655,"num_bytes_recovered":42860544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":1},{"shards":"0,3,5,7","objects":9},{"shards":"0,3,7","objects":4120},{"shards":"0,5","objects":2210},{"shards":"3,5,7","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":173,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:38:57.912213+0000","last_change":"2024-09-13T14:37:48.127604+0000","last_active":"2024-09-13T14:38:57.912213+0000","last_peered":"2024-09-13T14:38:57.912213+0000","last_clean":"2024-09-13T14:38:57.912213+0000","last_became_active":"2024-09-13T14:37:48.127300+0000","last_became_peered":"2024-09-13T14:37:48.127300+0000","last_unstale":"2024-09-13T14:38:57.912213+0000","last_undegraded":"2024-09-13T14:38:57.912213+0000","last_fullsized":"2024-09-13T14:38:57.912213+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:51.999506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":293,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:38:57.912415+0000","last_change":"2024-09-13T14:38:03.529967+0000","last_active":"2024-09-13T14:38:57.912415+0000","last_peered":"2024-09-13T14:38:57.912415+0000","last_clean":"2024-09-13T14:38:57.912415+0000","last_became_active":"2024-09-13T14:38:03.529646+0000","last_became_peered":"2024-09-13T14:38:03.529646+0000","last_unstale":"2024-09-13T14:38:57.912415+0000","last_undegraded":"2024-09-13T14:38:57.912415+0000","last_fullsized":"2024-09-13T14:38:57.912415+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:52:05.193279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"258'7764","reported_seq":8085,"reported_epoch":258,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:39:12.700008+0000","last_change":"2024-09-13T14:38:03.544515+0000","last_active":"2024-09-13T14:39:12.700008+0000","last_peered":"2024-09-13T14:39:12.700008+0000","last_clean":"2024-09-13T14:35:51.090657+0000","last_became_active":"2024-09-13T14:37:49.619314+0000","last_became_peered":"2024-09-13T14:37:49.619314+0000","last_unstale":"2024-09-13T14:39:12.700008+0000","last_undegraded":"2024-09-13T14:37:49.147721+0000","last_fullsized":"2024-09-13T14:37:49.120361+0000","mapping_epoch":240,"log_start":"209'1800","ondisk_log_start":"209'1800","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5964,"log_dups_size":0,"ondisk_log_size":5964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":417792024,"num_objects":6376,"num_object_clones":0,"num_object_copies":19128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4393,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6376,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7764,"num_write_kb":452417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":6291456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":2048}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":253,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:38:57.912454+0000","last_change":"2024-09-13T14:36:13.860367+0000","last_active":"2024-09-13T14:38:57.912454+0000","last_peered":"2024-09-13T14:38:57.912454+0000","last_clean":"2024-09-13T14:38:57.912454+0000","last_became_active":"2024-09-13T14:36:13.859982+0000","last_became_peered":"2024-09-13T14:36:13.859982+0000","last_unstale":"2024-09-13T14:38:57.912454+0000","last_undegraded":"2024-09-13T14:38:57.912454+0000","last_fullsized":"2024-09-13T14:38:57.912454+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:08:23.311350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":249,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:38:57.912935+0000","last_change":"2024-09-13T14:37:48.147360+0000","last_active":"2024-09-13T14:38:57.912935+0000","last_peered":"2024-09-13T14:38:57.912935+0000","last_clean":"2024-09-13T14:38:57.912935+0000","last_became_active":"2024-09-13T14:37:48.146885+0000","last_became_peered":"2024-09-13T14:37:48.146885+0000","last_unstale":"2024-09-13T14:38:57.912935+0000","last_undegraded":"2024-09-13T14:38:57.912935+0000","last_fullsized":"2024-09-13T14:38:57.912935+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:27.841810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"259'7738","reported_seq":16963,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:17.275618+0000","last_change":"2024-09-13T14:38:39.650332+0000","last_active":"2024-09-13T14:39:17.275618+0000","last_peered":"2024-09-13T14:39:17.275618+0000","last_clean":"2024-09-13T14:39:17.275618+0000","last_became_active":"2024-09-13T14:38:39.640835+0000","last_became_peered":"2024-09-13T14:38:39.640835+0000","last_unstale":"2024-09-13T14:39:17.275618+0000","last_undegraded":"2024-09-13T14:39:17.275618+0000","last_fullsized":"2024-09-13T14:39:17.275618+0000","mapping_epoch":251,"log_start":"212'3051","ondisk_log_start":"212'3051","created":205,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":4687,"log_dups_size":0,"ondisk_log_size":4687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:48.176606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":415236096,"num_objects":6336,"num_object_clones":0,"num_object_copies":19008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7738,"num_write_kb":450368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4544,"num_bytes_recovered":292945920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":416,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.420600+0000","last_change":"2024-09-13T14:38:53.900041+0000","last_active":"2024-09-13T14:39:15.420600+0000","last_peered":"2024-09-13T14:39:15.420600+0000","last_clean":"2024-09-13T14:39:15.420600+0000","last_became_active":"2024-09-13T14:38:53.899203+0000","last_became_peered":"2024-09-13T14:38:53.899203+0000","last_unstale":"2024-09-13T14:39:15.420600+0000","last_undegraded":"2024-09-13T14:39:15.420600+0000","last_fullsized":"2024-09-13T14:39:15.420600+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:28:07.883094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":348,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.421619+0000","last_change":"2024-09-13T14:26:15.823198+0000","last_active":"2024-09-13T14:39:15.421619+0000","last_peered":"2024-09-13T14:39:15.421619+0000","last_clean":"2024-09-13T14:39:15.421619+0000","last_became_active":"2024-09-13T14:26:15.822717+0000","last_became_peered":"2024-09-13T14:26:15.822717+0000","last_unstale":"2024-09-13T14:39:15.421619+0000","last_undegraded":"2024-09-13T14:39:15.421619+0000","last_fullsized":"2024-09-13T14:39:15.421619+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:22.471622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.6","version":"258'7823","reported_seq":9191,"reported_epoch":258,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-13T14:39:12.714504+0000","last_change":"2024-09-13T14:38:55.132683+0000","last_active":"2024-09-13T14:39:12.714504+0000","last_peered":"2024-09-13T14:39:12.714504+0000","last_clean":"2024-09-13T14:36:13.124368+0000","last_became_active":"2024-09-13T14:38:55.120883+0000","last_became_peered":"2024-09-13T14:38:55.120883+0000","last_unstale":"2024-09-13T14:39:12.714504+0000","last_undegraded":"2024-09-13T14:38:55.026610+0000","last_fullsized":"2024-09-13T14:39:12.714504+0000","mapping_epoch":254,"log_start":"212'3119","ondisk_log_start":"212'3119","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":4704,"log_dups_size":0,"ondisk_log_size":4704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":417792000,"num_objects":6375,"num_object_clones":0,"num_object_copies":19125,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4388,"num_objects_misplaced":1987,"num_objects_unfound":0,"num_objects_dirty":6375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7823,"num_write_kb":454336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":617,"num_bytes_recovered":29229056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,3,5],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3,4","objects":4109},{"shards":"0,3,5","objects":2266}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":412,"reported_epoch":259,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-13T14:39:15.420235+0000","last_change":"2024-09-13T14:38:54.899624+0000","last_active":"2024-09-13T14:39:15.420235+0000","last_peered":"2024-09-13T14:39:15.420235+0000","last_clean":"2024-09-13T14:17:27.401592+0000","last_became_active":"2024-09-13T14:38:54.899282+0000","last_became_peered":"2024-09-13T14:38:54.899282+0000","last_unstale":"2024-09-13T14:39:15.420235+0000","last_undegraded":"2024-09-13T14:38:54.891176+0000","last_fullsized":"2024-09-13T14:39:15.420235+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:07:41.927740+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[3,0,6],"avail_no_missing":["3","4","6"],"object_location_counts":[{"shards":"3,4,6","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":496,"reported_epoch":258,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:39:07.963177+0000","last_change":"2024-09-13T14:38:54.904329+0000","last_active":"2024-09-13T14:39:07.963177+0000","last_peered":"2024-09-13T14:39:07.963177+0000","last_clean":"2024-09-13T14:38:39.648017+0000","last_became_active":"2024-09-13T14:38:54.903798+0000","last_became_peered":"2024-09-13T14:38:54.903798+0000","last_unstale":"2024-09-13T14:39:07.963177+0000","last_undegraded":"2024-09-13T14:38:54.890505+0000","last_fullsized":"2024-09-13T14:38:54.889701+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:18.120931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,3],"avail_no_missing":["5","3","6"],"object_location_counts":[{"shards":"3,5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":350,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.421299+0000","last_change":"2024-09-13T14:37:49.160719+0000","last_active":"2024-09-13T14:39:15.421299+0000","last_peered":"2024-09-13T14:39:15.421299+0000","last_clean":"2024-09-13T14:39:15.421299+0000","last_became_active":"2024-09-13T14:37:49.160401+0000","last_became_peered":"2024-09-13T14:37:49.160401+0000","last_unstale":"2024-09-13T14:39:15.421299+0000","last_undegraded":"2024-09-13T14:39:15.421299+0000","last_fullsized":"2024-09-13T14:39:15.421299+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:21.737945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":100,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:38:57.912332+0000","last_change":"2024-09-13T14:37:48.172940+0000","last_active":"2024-09-13T14:38:57.912332+0000","last_peered":"2024-09-13T14:38:57.912332+0000","last_clean":"2024-09-13T14:38:57.912332+0000","last_became_active":"2024-09-13T14:37:48.172060+0000","last_became_peered":"2024-09-13T14:37:48.172060+0000","last_unstale":"2024-09-13T14:38:57.912332+0000","last_undegraded":"2024-09-13T14:38:57.912332+0000","last_fullsized":"2024-09-13T14:38:57.912332+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:07.480645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.5","version":"259'7791","reported_seq":18570,"reported_epoch":259,"state":"active+recovering+degraded","last_fresh":"2024-09-13T14:39:17.376216+0000","last_change":"2024-09-13T14:38:53.941671+0000","last_active":"2024-09-13T14:39:17.376216+0000","last_peered":"2024-09-13T14:39:17.376216+0000","last_clean":"2024-09-13T14:35:51.091736+0000","last_became_active":"2024-09-13T14:38:04.600616+0000","last_became_peered":"2024-09-13T14:38:04.600616+0000","last_unstale":"2024-09-13T14:39:17.376216+0000","last_undegraded":"2024-09-13T14:38:04.591411+0000","last_fullsized":"2024-09-13T14:39:17.376216+0000","mapping_epoch":245,"log_start":"209'1516","ondisk_log_start":"209'1516","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6275,"log_dups_size":0,"ondisk_log_size":6275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:00:44.321678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":416219136,"num_objects":6351,"num_object_clones":0,"num_object_copies":19053,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2692,"num_objects_misplaced":8,"num_objects_unfound":0,"num_objects_dirty":6351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7791,"num_write_kb":452544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5341,"num_bytes_recovered":344129536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,4,7","objects":8},{"shards":"0,6,7","objects":3651},{"shards":"0,7","objects":2692}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":474,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:39:07.943879+0000","last_change":"2024-09-13T14:26:15.835178+0000","last_active":"2024-09-13T14:39:07.943879+0000","last_peered":"2024-09-13T14:39:07.943879+0000","last_clean":"2024-09-13T14:39:07.943879+0000","last_became_active":"2024-09-13T14:26:15.820283+0000","last_became_peered":"2024-09-13T14:26:15.820283+0000","last_unstale":"2024-09-13T14:39:07.943879+0000","last_undegraded":"2024-09-13T14:39:07.943879+0000","last_fullsized":"2024-09-13T14:39:07.943879+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:30.340038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":38,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:39:07.949030+0000","last_change":"2024-09-13T14:38:03.525682+0000","last_active":"2024-09-13T14:39:07.949030+0000","last_peered":"2024-09-13T14:39:07.949030+0000","last_clean":"2024-09-13T14:39:07.949030+0000","last_became_active":"2024-09-13T14:38:03.525406+0000","last_became_peered":"2024-09-13T14:38:03.525406+0000","last_unstale":"2024-09-13T14:39:07.949030+0000","last_undegraded":"2024-09-13T14:39:07.949030+0000","last_fullsized":"2024-09-13T14:39:07.949030+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:41:28.970300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":45,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:39:07.950115+0000","last_change":"2024-09-13T14:38:53.991738+0000","last_active":"2024-09-13T14:39:07.950115+0000","last_peered":"2024-09-13T14:39:07.950115+0000","last_clean":"2024-09-13T14:39:07.950115+0000","last_became_active":"2024-09-13T14:38:53.991417+0000","last_became_peered":"2024-09-13T14:38:53.991417+0000","last_unstale":"2024-09-13T14:39:07.950115+0000","last_undegraded":"2024-09-13T14:39:07.950115+0000","last_fullsized":"2024-09-13T14:39:07.950115+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:50:38.559838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"258'7747","reported_seq":21885,"reported_epoch":258,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-13T14:39:11.952872+0000","last_change":"2024-09-13T14:38:17.472071+0000","last_active":"2024-09-13T14:39:11.952872+0000","last_peered":"2024-09-13T14:39:11.952872+0000","last_clean":"2024-09-13T14:37:46.290754+0000","last_became_active":"2024-09-13T14:38:17.381753+0000","last_became_peered":"2024-09-13T14:38:17.381753+0000","last_unstale":"2024-09-13T14:39:11.952872+0000","last_undegraded":"2024-09-13T14:38:13.504140+0000","last_fullsized":"2024-09-13T14:39:11.952872+0000","mapping_epoch":245,"log_start":"210'1959","ondisk_log_start":"210'1959","created":205,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5788,"log_dups_size":0,"ondisk_log_size":5788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":416743424,"num_objects":6359,"num_object_clones":0,"num_object_copies":19077,"num_objects_missing_on_primary":83,"num_objects_missing":83,"num_objects_degraded":4582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7747,"num_write_kb":451392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7053,"num_bytes_recovered":462225408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"0,4,5","objects":4413},{"shards":"0,4,6","objects":1946}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":761,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.421057+0000","last_change":"2024-09-13T14:37:48.131799+0000","last_active":"2024-09-13T14:39:15.421057+0000","last_peered":"2024-09-13T14:39:15.421057+0000","last_clean":"2024-09-13T14:39:15.421057+0000","last_became_active":"2024-09-13T14:37:48.130837+0000","last_became_peered":"2024-09-13T14:37:48.130837+0000","last_unstale":"2024-09-13T14:39:15.421057+0000","last_undegraded":"2024-09-13T14:39:15.421057+0000","last_fullsized":"2024-09-13T14:39:15.421057+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:06:32.663770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":278,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:39:07.949313+0000","last_change":"2024-09-13T14:37:49.301531+0000","last_active":"2024-09-13T14:39:07.949313+0000","last_peered":"2024-09-13T14:39:07.949313+0000","last_clean":"2024-09-13T14:39:07.949313+0000","last_became_active":"2024-09-13T14:37:49.301249+0000","last_became_peered":"2024-09-13T14:37:49.301249+0000","last_unstale":"2024-09-13T14:39:07.949313+0000","last_undegraded":"2024-09-13T14:39:07.949313+0000","last_fullsized":"2024-09-13T14:39:07.949313+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:10.977710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":83,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:39:07.963309+0000","last_change":"2024-09-13T14:38:53.901427+0000","last_active":"2024-09-13T14:39:07.963309+0000","last_peered":"2024-09-13T14:39:07.963309+0000","last_clean":"2024-09-13T14:39:07.963309+0000","last_became_active":"2024-09-13T14:38:53.901157+0000","last_became_peered":"2024-09-13T14:38:53.901157+0000","last_unstale":"2024-09-13T14:39:07.963309+0000","last_undegraded":"2024-09-13T14:39:07.963309+0000","last_fullsized":"2024-09-13T14:39:07.963309+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:28:21.713025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"258'7689","reported_seq":21717,"reported_epoch":258,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-13T14:39:12.694738+0000","last_change":"2024-09-13T14:38:16.552896+0000","last_active":"2024-09-13T14:39:12.694738+0000","last_peered":"2024-09-13T14:39:12.694738+0000","last_clean":"2024-09-13T14:37:46.405415+0000","last_became_active":"2024-09-13T14:38:16.427510+0000","last_became_peered":"2024-09-13T14:38:16.427510+0000","last_unstale":"2024-09-13T14:39:12.694738+0000","last_undegraded":"2024-09-13T14:38:12.738464+0000","last_fullsized":"2024-09-13T14:39:12.694738+0000","mapping_epoch":245,"log_start":"210'2518","ondisk_log_start":"210'2518","created":205,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5171,"log_dups_size":0,"ondisk_log_size":5171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":413466624,"num_objects":6309,"num_object_clones":0,"num_object_copies":18927,"num_objects_missing_on_primary":100,"num_objects_missing":100,"num_objects_degraded":4572,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7689,"num_write_kb":447936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6999,"num_bytes_recovered":458686464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"0,3,6","objects":1904},{"shards":"0,3,7","objects":4405}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":965,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:39:07.965123+0000","last_change":"2024-09-13T14:37:49.147498+0000","last_active":"2024-09-13T14:39:07.965123+0000","last_peered":"2024-09-13T14:39:07.965123+0000","last_clean":"2024-09-13T14:39:07.965123+0000","last_became_active":"2024-09-13T14:37:49.147028+0000","last_became_peered":"2024-09-13T14:37:49.147028+0000","last_unstale":"2024-09-13T14:39:07.965123+0000","last_undegraded":"2024-09-13T14:39:07.965123+0000","last_fullsized":"2024-09-13T14:39:07.965123+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:11:29.050178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":262,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:38:57.912752+0000","last_change":"2024-09-13T14:38:03.528133+0000","last_active":"2024-09-13T14:38:57.912752+0000","last_peered":"2024-09-13T14:38:57.912752+0000","last_clean":"2024-09-13T14:38:57.912752+0000","last_became_active":"2024-09-13T14:38:03.527829+0000","last_became_peered":"2024-09-13T14:38:03.527829+0000","last_unstale":"2024-09-13T14:38:57.912752+0000","last_undegraded":"2024-09-13T14:38:57.912752+0000","last_fullsized":"2024-09-13T14:38:57.912752+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:48.254169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":400,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.421417+0000","last_change":"2024-09-13T14:38:53.899755+0000","last_active":"2024-09-13T14:39:15.421417+0000","last_peered":"2024-09-13T14:39:15.421417+0000","last_clean":"2024-09-13T14:39:15.421417+0000","last_became_active":"2024-09-13T14:38:53.899448+0000","last_became_peered":"2024-09-13T14:38:53.899448+0000","last_unstale":"2024-09-13T14:39:15.421417+0000","last_undegraded":"2024-09-13T14:39:15.421417+0000","last_fullsized":"2024-09-13T14:39:15.421417+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:55.897035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"258'7852","reported_seq":7933,"reported_epoch":258,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:39:12.418752+0000","last_change":"2024-09-13T14:38:03.567892+0000","last_active":"2024-09-13T14:39:12.418752+0000","last_peered":"2024-09-13T14:39:12.418752+0000","last_clean":"2024-09-13T14:35:51.090791+0000","last_became_active":"2024-09-13T14:37:49.665025+0000","last_became_peered":"2024-09-13T14:37:49.665025+0000","last_unstale":"2024-09-13T14:39:12.418752+0000","last_undegraded":"2024-09-13T14:37:49.169101+0000","last_fullsized":"2024-09-13T14:37:49.121596+0000","mapping_epoch":240,"log_start":"210'2063","ondisk_log_start":"210'2063","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5789,"log_dups_size":0,"ondisk_log_size":5789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:02.321504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":422707200,"num_objects":6450,"num_object_clones":0,"num_object_copies":19350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4501,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7852,"num_write_kb":457664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":2015}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":67,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:39:07.963980+0000","last_change":"2024-09-13T14:37:49.149770+0000","last_active":"2024-09-13T14:39:07.963980+0000","last_peered":"2024-09-13T14:39:07.963980+0000","last_clean":"2024-09-13T14:39:07.963980+0000","last_became_active":"2024-09-13T14:37:49.148991+0000","last_became_peered":"2024-09-13T14:37:49.148991+0000","last_unstale":"2024-09-13T14:39:07.963980+0000","last_undegraded":"2024-09-13T14:39:07.963980+0000","last_fullsized":"2024-09-13T14:39:07.963980+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:04:04.193366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":1097,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:39:07.994149+0000","last_change":"2024-09-13T14:30:58.928302+0000","last_active":"2024-09-13T14:39:07.994149+0000","last_peered":"2024-09-13T14:39:07.994149+0000","last_clean":"2024-09-13T14:39:07.994149+0000","last_became_active":"2024-09-13T14:30:58.927718+0000","last_became_peered":"2024-09-13T14:30:58.927718+0000","last_unstale":"2024-09-13T14:39:07.994149+0000","last_undegraded":"2024-09-13T14:39:07.994149+0000","last_fullsized":"2024-09-13T14:39:07.994149+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:37.323036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":74,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:39:07.965504+0000","last_change":"2024-09-13T14:38:03.534628+0000","last_active":"2024-09-13T14:39:07.965504+0000","last_peered":"2024-09-13T14:39:07.965504+0000","last_clean":"2024-09-13T14:39:07.965504+0000","last_became_active":"2024-09-13T14:38:03.534262+0000","last_became_peered":"2024-09-13T14:38:03.534262+0000","last_unstale":"2024-09-13T14:39:07.965504+0000","last_undegraded":"2024-09-13T14:39:07.965504+0000","last_fullsized":"2024-09-13T14:39:07.965504+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:31.168783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"259'7730","reported_seq":21874,"reported_epoch":259,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:39:15.996863+0000","last_change":"2024-09-13T14:38:53.944111+0000","last_active":"2024-09-13T14:39:15.996863+0000","last_peered":"2024-09-13T14:39:15.996863+0000","last_clean":"2024-09-13T14:37:47.047672+0000","last_became_active":"2024-09-13T14:37:49.664294+0000","last_became_peered":"2024-09-13T14:37:49.664294+0000","last_unstale":"2024-09-13T14:39:15.996863+0000","last_undegraded":"2024-09-13T14:38:53.940822+0000","last_fullsized":"2024-09-13T14:37:49.126627+0000","mapping_epoch":240,"log_start":"210'2000","ondisk_log_start":"210'2000","created":205,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5730,"log_dups_size":0,"ondisk_log_size":5730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:11.766692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":414318592,"num_objects":6322,"num_object_clones":0,"num_object_copies":18966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4535,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7730,"num_write_kb":449664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7009,"num_bytes_recovered":459341824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3],"avail_no_missing":["5","0","3"],"object_location_counts":[{"shards":"0,3,5","objects":4460},{"shards":"3,5","objects":1862}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":86,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:39:07.963758+0000","last_change":"2024-09-13T14:38:03.534303+0000","last_active":"2024-09-13T14:39:07.963758+0000","last_peered":"2024-09-13T14:39:07.963758+0000","last_clean":"2024-09-13T14:39:07.963758+0000","last_became_active":"2024-09-13T14:38:03.533171+0000","last_became_peered":"2024-09-13T14:38:03.533171+0000","last_unstale":"2024-09-13T14:39:07.963758+0000","last_undegraded":"2024-09-13T14:39:07.963758+0000","last_fullsized":"2024-09-13T14:39:07.963758+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:00:16.553526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":778,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:39:07.948760+0000","last_change":"2024-09-13T14:38:04.596192+0000","last_active":"2024-09-13T14:39:07.948760+0000","last_peered":"2024-09-13T14:39:07.948760+0000","last_clean":"2024-09-13T14:39:07.948760+0000","last_became_active":"2024-09-13T14:38:04.593299+0000","last_became_peered":"2024-09-13T14:38:04.593299+0000","last_unstale":"2024-09-13T14:39:07.948760+0000","last_undegraded":"2024-09-13T14:39:07.948760+0000","last_fullsized":"2024-09-13T14:39:07.948760+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:34.241410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":38,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:39:07.950664+0000","last_change":"2024-09-13T14:38:03.533120+0000","last_active":"2024-09-13T14:39:07.950664+0000","last_peered":"2024-09-13T14:39:07.950664+0000","last_clean":"2024-09-13T14:39:07.950664+0000","last_became_active":"2024-09-13T14:38:03.532829+0000","last_became_peered":"2024-09-13T14:38:03.532829+0000","last_unstale":"2024-09-13T14:39:07.950664+0000","last_undegraded":"2024-09-13T14:39:07.950664+0000","last_fullsized":"2024-09-13T14:39:07.950664+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:24.514661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"259'7944","reported_seq":8071,"reported_epoch":259,"state":"active+undersized+degraded+remapped+backfill_wait","last_fresh":"2024-09-13T14:39:17.268425+0000","last_change":"2024-09-13T14:37:49.295861+0000","last_active":"2024-09-13T14:39:17.268425+0000","last_peered":"2024-09-13T14:39:17.268425+0000","last_clean":"2024-09-13T14:35:51.078708+0000","last_became_active":"2024-09-13T14:37:49.290403+0000","last_became_peered":"2024-09-13T14:37:49.290403+0000","last_unstale":"2024-09-13T14:39:17.268425+0000","last_undegraded":"2024-09-13T14:37:49.147402+0000","last_fullsized":"2024-09-13T14:37:49.117202+0000","mapping_epoch":240,"log_start":"212'3218","ondisk_log_start":"212'3218","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":4726,"log_dups_size":0,"ondisk_log_size":4726,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":422969344,"num_objects":6454,"num_object_clones":0,"num_object_copies":19362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6454,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7944,"num_write_kb":460736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":6454}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":442,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.421646+0000","last_change":"2024-09-13T14:38:53.938227+0000","last_active":"2024-09-13T14:39:15.421646+0000","last_peered":"2024-09-13T14:39:15.421646+0000","last_clean":"2024-09-13T14:39:15.421646+0000","last_became_active":"2024-09-13T14:38:53.902867+0000","last_became_peered":"2024-09-13T14:38:53.902867+0000","last_unstale":"2024-09-13T14:39:15.421646+0000","last_undegraded":"2024-09-13T14:39:15.421646+0000","last_fullsized":"2024-09-13T14:39:15.421646+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:16.831756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":556,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:38:57.913087+0000","last_change":"2024-09-13T14:37:48.147677+0000","last_active":"2024-09-13T14:38:57.913087+0000","last_peered":"2024-09-13T14:38:57.913087+0000","last_clean":"2024-09-13T14:38:57.913087+0000","last_became_active":"2024-09-13T14:37:48.147001+0000","last_became_peered":"2024-09-13T14:37:48.147001+0000","last_unstale":"2024-09-13T14:38:57.913087+0000","last_undegraded":"2024-09-13T14:38:57.913087+0000","last_fullsized":"2024-09-13T14:38:57.913087+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:35:00.653576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":424,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.420152+0000","last_change":"2024-09-13T14:38:53.900107+0000","last_active":"2024-09-13T14:39:15.420152+0000","last_peered":"2024-09-13T14:39:15.420152+0000","last_clean":"2024-09-13T14:39:15.420152+0000","last_became_active":"2024-09-13T14:38:53.899556+0000","last_became_peered":"2024-09-13T14:38:53.899556+0000","last_unstale":"2024-09-13T14:39:15.420152+0000","last_undegraded":"2024-09-13T14:39:15.420152+0000","last_fullsized":"2024-09-13T14:39:15.420152+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:15:47.918163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"259'7677","reported_seq":10734,"reported_epoch":259,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:39:17.213100+0000","last_change":"2024-09-13T14:38:53.903357+0000","last_active":"2024-09-13T14:39:17.213100+0000","last_peered":"2024-09-13T14:39:17.213100+0000","last_clean":"2024-09-13T14:35:51.088129+0000","last_became_active":"2024-09-13T14:37:49.691874+0000","last_became_peered":"2024-09-13T14:37:49.691874+0000","last_unstale":"2024-09-13T14:39:17.213100+0000","last_undegraded":"2024-09-13T14:37:49.158458+0000","last_fullsized":"2024-09-13T14:37:49.116993+0000","mapping_epoch":240,"log_start":"212'2847","ondisk_log_start":"212'2847","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":4830,"log_dups_size":0,"ondisk_log_size":4830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404684800,"num_objects":6175,"num_object_clones":0,"num_object_copies":18525,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2932,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7677,"num_write_kb":443264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1463,"num_bytes_recovered":95879168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":3308}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":53,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:39:07.949953+0000","last_change":"2024-09-13T14:38:53.901806+0000","last_active":"2024-09-13T14:39:07.949953+0000","last_peered":"2024-09-13T14:39:07.949953+0000","last_clean":"2024-09-13T14:39:07.949953+0000","last_became_active":"2024-09-13T14:38:53.891055+0000","last_became_peered":"2024-09-13T14:38:53.891055+0000","last_unstale":"2024-09-13T14:39:07.949953+0000","last_undegraded":"2024-09-13T14:39:07.949953+0000","last_fullsized":"2024-09-13T14:39:07.949953+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:03:00.416198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":14,"reported_epoch":258,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-13T14:39:07.943662+0000","last_change":"2024-09-13T14:38:53.900902+0000","last_active":"2024-09-13T14:39:07.943662+0000","last_peered":"2024-09-13T14:39:07.943662+0000","last_clean":"2024-09-13T14:38:39.635779+0000","last_became_active":"2024-09-13T14:38:53.900626+0000","last_became_peered":"2024-09-13T14:38:53.900626+0000","last_unstale":"2024-09-13T14:39:07.943662+0000","last_undegraded":"2024-09-13T14:38:53.885778+0000","last_fullsized":"2024-09-13T14:39:07.943662+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:37:21.715778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":7,"num_objects_missing":7,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":["3","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":246,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:38:57.912333+0000","last_change":"2024-09-13T14:37:49.172218+0000","last_active":"2024-09-13T14:38:57.912333+0000","last_peered":"2024-09-13T14:38:57.912333+0000","last_clean":"2024-09-13T14:38:57.912333+0000","last_became_active":"2024-09-13T14:37:49.171718+0000","last_became_peered":"2024-09-13T14:37:49.171718+0000","last_unstale":"2024-09-13T14:38:57.912333+0000","last_undegraded":"2024-09-13T14:38:57.912333+0000","last_fullsized":"2024-09-13T14:38:57.912333+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:46:54.626521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":268,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:38:57.912613+0000","last_change":"2024-09-13T14:37:49.230094+0000","last_active":"2024-09-13T14:38:57.912613+0000","last_peered":"2024-09-13T14:38:57.912613+0000","last_clean":"2024-09-13T14:38:57.912613+0000","last_became_active":"2024-09-13T14:37:49.229393+0000","last_became_peered":"2024-09-13T14:37:49.229393+0000","last_unstale":"2024-09-13T14:38:57.912613+0000","last_undegraded":"2024-09-13T14:38:57.912613+0000","last_fullsized":"2024-09-13T14:38:57.912613+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:02:28.690318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":248,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:38:57.913079+0000","last_change":"2024-09-13T14:37:48.128447+0000","last_active":"2024-09-13T14:38:57.913079+0000","last_peered":"2024-09-13T14:38:57.913079+0000","last_clean":"2024-09-13T14:38:57.913079+0000","last_became_active":"2024-09-13T14:37:48.128143+0000","last_became_peered":"2024-09-13T14:37:48.128143+0000","last_unstale":"2024-09-13T14:38:57.913079+0000","last_undegraded":"2024-09-13T14:38:57.913079+0000","last_fullsized":"2024-09-13T14:38:57.913079+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:22.204039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":38,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:39:07.949902+0000","last_change":"2024-09-13T14:38:03.531683+0000","last_active":"2024-09-13T14:39:07.949902+0000","last_peered":"2024-09-13T14:39:07.949902+0000","last_clean":"2024-09-13T14:39:07.949902+0000","last_became_active":"2024-09-13T14:38:03.531476+0000","last_became_peered":"2024-09-13T14:38:03.531476+0000","last_unstale":"2024-09-13T14:39:07.949902+0000","last_undegraded":"2024-09-13T14:39:07.949902+0000","last_fullsized":"2024-09-13T14:39:07.949902+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:41.557569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":22,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:39:07.944204+0000","last_change":"2024-09-13T14:38:53.954281+0000","last_active":"2024-09-13T14:39:07.944204+0000","last_peered":"2024-09-13T14:39:07.944204+0000","last_clean":"2024-09-13T14:39:07.944204+0000","last_became_active":"2024-09-13T14:38:53.900912+0000","last_became_peered":"2024-09-13T14:38:53.900912+0000","last_unstale":"2024-09-13T14:39:07.944204+0000","last_undegraded":"2024-09-13T14:39:07.944204+0000","last_fullsized":"2024-09-13T14:39:07.944204+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:47.876275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":74,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:39:08.005649+0000","last_change":"2024-09-13T14:38:54.000782+0000","last_active":"2024-09-13T14:39:08.005649+0000","last_peered":"2024-09-13T14:39:08.005649+0000","last_clean":"2024-09-13T14:39:08.005649+0000","last_became_active":"2024-09-13T14:38:54.000322+0000","last_became_peered":"2024-09-13T14:38:54.000322+0000","last_unstale":"2024-09-13T14:39:08.005649+0000","last_undegraded":"2024-09-13T14:39:08.005649+0000","last_fullsized":"2024-09-13T14:39:08.005649+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:09.057597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":366,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.420793+0000","last_change":"2024-09-13T14:38:53.899535+0000","last_active":"2024-09-13T14:39:15.420793+0000","last_peered":"2024-09-13T14:39:15.420793+0000","last_clean":"2024-09-13T14:39:15.420793+0000","last_became_active":"2024-09-13T14:38:53.899183+0000","last_became_peered":"2024-09-13T14:38:53.899183+0000","last_unstale":"2024-09-13T14:39:15.420793+0000","last_undegraded":"2024-09-13T14:39:15.420793+0000","last_fullsized":"2024-09-13T14:39:15.420793+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:33.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":38,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:39:07.954287+0000","last_change":"2024-09-13T14:38:03.543426+0000","last_active":"2024-09-13T14:39:07.954287+0000","last_peered":"2024-09-13T14:39:07.954287+0000","last_clean":"2024-09-13T14:39:07.954287+0000","last_became_active":"2024-09-13T14:38:03.542661+0000","last_became_peered":"2024-09-13T14:38:03.542661+0000","last_unstale":"2024-09-13T14:39:07.954287+0000","last_undegraded":"2024-09-13T14:39:07.954287+0000","last_fullsized":"2024-09-13T14:39:07.954287+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:54.050127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":395,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.420075+0000","last_change":"2024-09-13T14:37:48.131321+0000","last_active":"2024-09-13T14:39:15.420075+0000","last_peered":"2024-09-13T14:39:15.420075+0000","last_clean":"2024-09-13T14:39:15.420075+0000","last_became_active":"2024-09-13T14:37:48.130998+0000","last_became_peered":"2024-09-13T14:37:48.130998+0000","last_unstale":"2024-09-13T14:39:15.420075+0000","last_undegraded":"2024-09-13T14:39:15.420075+0000","last_fullsized":"2024-09-13T14:39:15.420075+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:33:26.597825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":86,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:39:07.963236+0000","last_change":"2024-09-13T14:38:03.535454+0000","last_active":"2024-09-13T14:39:07.963236+0000","last_peered":"2024-09-13T14:39:07.963236+0000","last_clean":"2024-09-13T14:39:07.963236+0000","last_became_active":"2024-09-13T14:38:03.535131+0000","last_became_peered":"2024-09-13T14:38:03.535131+0000","last_unstale":"2024-09-13T14:39:07.963236+0000","last_undegraded":"2024-09-13T14:39:07.963236+0000","last_fullsized":"2024-09-13T14:39:07.963236+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:24.477601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":419,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.420514+0000","last_change":"2024-09-13T14:38:03.524939+0000","last_active":"2024-09-13T14:39:15.420514+0000","last_peered":"2024-09-13T14:39:15.420514+0000","last_clean":"2024-09-13T14:39:15.420514+0000","last_became_active":"2024-09-13T14:38:03.524406+0000","last_became_peered":"2024-09-13T14:38:03.524406+0000","last_unstale":"2024-09-13T14:39:15.420514+0000","last_undegraded":"2024-09-13T14:39:15.420514+0000","last_fullsized":"2024-09-13T14:39:15.420514+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:19:45.257424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":45,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:39:07.949871+0000","last_change":"2024-09-13T14:38:53.892948+0000","last_active":"2024-09-13T14:39:07.949871+0000","last_peered":"2024-09-13T14:39:07.949871+0000","last_clean":"2024-09-13T14:39:07.949871+0000","last_became_active":"2024-09-13T14:38:53.892665+0000","last_became_peered":"2024-09-13T14:38:53.892665+0000","last_unstale":"2024-09-13T14:39:07.949871+0000","last_undegraded":"2024-09-13T14:39:07.949871+0000","last_fullsized":"2024-09-13T14:39:07.949871+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:08.292359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":368,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.420970+0000","last_change":"2024-09-13T14:37:48.139348+0000","last_active":"2024-09-13T14:39:15.420970+0000","last_peered":"2024-09-13T14:39:15.420970+0000","last_clean":"2024-09-13T14:39:15.420970+0000","last_became_active":"2024-09-13T14:37:48.139115+0000","last_became_peered":"2024-09-13T14:37:48.139115+0000","last_unstale":"2024-09-13T14:39:15.420970+0000","last_undegraded":"2024-09-13T14:39:15.420970+0000","last_fullsized":"2024-09-13T14:39:15.420970+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:32:20.413346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":408,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.421675+0000","last_change":"2024-09-13T14:37:48.172088+0000","last_active":"2024-09-13T14:39:15.421675+0000","last_peered":"2024-09-13T14:39:15.421675+0000","last_clean":"2024-09-13T14:39:15.421675+0000","last_became_active":"2024-09-13T14:37:48.171779+0000","last_became_peered":"2024-09-13T14:37:48.171779+0000","last_unstale":"2024-09-13T14:39:15.421675+0000","last_undegraded":"2024-09-13T14:39:15.421675+0000","last_fullsized":"2024-09-13T14:39:15.421675+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:02.852712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":313,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:38:57.912284+0000","last_change":"2024-09-13T14:38:53.890236+0000","last_active":"2024-09-13T14:38:57.912284+0000","last_peered":"2024-09-13T14:38:57.912284+0000","last_clean":"2024-09-13T14:38:57.912284+0000","last_became_active":"2024-09-13T14:38:53.889582+0000","last_became_peered":"2024-09-13T14:38:53.889582+0000","last_unstale":"2024-09-13T14:38:57.912284+0000","last_undegraded":"2024-09-13T14:38:57.912284+0000","last_fullsized":"2024-09-13T14:38:57.912284+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:14:29.334774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":394,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.421776+0000","last_change":"2024-09-13T14:38:03.534097+0000","last_active":"2024-09-13T14:39:15.421776+0000","last_peered":"2024-09-13T14:39:15.421776+0000","last_clean":"2024-09-13T14:39:15.421776+0000","last_became_active":"2024-09-13T14:38:03.533794+0000","last_became_peered":"2024-09-13T14:38:03.533794+0000","last_unstale":"2024-09-13T14:39:15.421776+0000","last_undegraded":"2024-09-13T14:39:15.421776+0000","last_fullsized":"2024-09-13T14:39:15.421776+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:23.272098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":38,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:39:07.950493+0000","last_change":"2024-09-13T14:38:03.527021+0000","last_active":"2024-09-13T14:39:07.950493+0000","last_peered":"2024-09-13T14:39:07.950493+0000","last_clean":"2024-09-13T14:39:07.950493+0000","last_became_active":"2024-09-13T14:38:03.526698+0000","last_became_peered":"2024-09-13T14:38:03.526698+0000","last_unstale":"2024-09-13T14:39:07.950493+0000","last_undegraded":"2024-09-13T14:39:07.950493+0000","last_fullsized":"2024-09-13T14:39:07.950493+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:01:35.734086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":257,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:38:57.912560+0000","last_change":"2024-09-13T14:27:12.834931+0000","last_active":"2024-09-13T14:38:57.912560+0000","last_peered":"2024-09-13T14:38:57.912560+0000","last_clean":"2024-09-13T14:38:57.912560+0000","last_became_active":"2024-09-13T14:27:12.834572+0000","last_became_peered":"2024-09-13T14:27:12.834572+0000","last_unstale":"2024-09-13T14:38:57.912560+0000","last_undegraded":"2024-09-13T14:38:57.912560+0000","last_fullsized":"2024-09-13T14:38:57.912560+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:54:21.155689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":71,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:39:07.963748+0000","last_change":"2024-09-13T14:37:49.176796+0000","last_active":"2024-09-13T14:39:07.963748+0000","last_peered":"2024-09-13T14:39:07.963748+0000","last_clean":"2024-09-13T14:39:07.963748+0000","last_became_active":"2024-09-13T14:37:49.176246+0000","last_became_peered":"2024-09-13T14:37:49.176246+0000","last_unstale":"2024-09-13T14:39:07.963748+0000","last_undegraded":"2024-09-13T14:39:07.963748+0000","last_fullsized":"2024-09-13T14:39:07.963748+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:29:24.936718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":268,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:38:57.912198+0000","last_change":"2024-09-13T14:38:03.526333+0000","last_active":"2024-09-13T14:38:57.912198+0000","last_peered":"2024-09-13T14:38:57.912198+0000","last_clean":"2024-09-13T14:38:57.912198+0000","last_became_active":"2024-09-13T14:38:03.526001+0000","last_became_peered":"2024-09-13T14:38:03.526001+0000","last_unstale":"2024-09-13T14:38:57.912198+0000","last_undegraded":"2024-09-13T14:38:57.912198+0000","last_fullsized":"2024-09-13T14:38:57.912198+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:25:25.181271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":71,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:39:07.964058+0000","last_change":"2024-09-13T14:37:49.149582+0000","last_active":"2024-09-13T14:39:07.964058+0000","last_peered":"2024-09-13T14:39:07.964058+0000","last_clean":"2024-09-13T14:39:07.964058+0000","last_became_active":"2024-09-13T14:37:49.148525+0000","last_became_peered":"2024-09-13T14:37:49.148525+0000","last_unstale":"2024-09-13T14:39:07.964058+0000","last_undegraded":"2024-09-13T14:39:07.964058+0000","last_fullsized":"2024-09-13T14:39:07.964058+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:48.655665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":440,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.421133+0000","last_change":"2024-09-13T14:38:03.535255+0000","last_active":"2024-09-13T14:39:15.421133+0000","last_peered":"2024-09-13T14:39:15.421133+0000","last_clean":"2024-09-13T14:39:15.421133+0000","last_became_active":"2024-09-13T14:38:03.533917+0000","last_became_peered":"2024-09-13T14:38:03.533917+0000","last_unstale":"2024-09-13T14:39:15.421133+0000","last_undegraded":"2024-09-13T14:39:15.421133+0000","last_fullsized":"2024-09-13T14:39:15.421133+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:33.559788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":404,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.422446+0000","last_change":"2024-09-13T14:37:48.146784+0000","last_active":"2024-09-13T14:39:15.422446+0000","last_peered":"2024-09-13T14:39:15.422446+0000","last_clean":"2024-09-13T14:39:15.422446+0000","last_became_active":"2024-09-13T14:37:48.146491+0000","last_became_peered":"2024-09-13T14:37:48.146491+0000","last_unstale":"2024-09-13T14:39:15.422446+0000","last_undegraded":"2024-09-13T14:39:15.422446+0000","last_fullsized":"2024-09-13T14:39:15.422446+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:30:54.772513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":248,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:38:57.912074+0000","last_change":"2024-09-13T14:37:48.127607+0000","last_active":"2024-09-13T14:38:57.912074+0000","last_peered":"2024-09-13T14:38:57.912074+0000","last_clean":"2024-09-13T14:38:57.912074+0000","last_became_active":"2024-09-13T14:37:48.127316+0000","last_became_peered":"2024-09-13T14:37:48.127316+0000","last_unstale":"2024-09-13T14:38:57.912074+0000","last_undegraded":"2024-09-13T14:38:57.912074+0000","last_fullsized":"2024-09-13T14:38:57.912074+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:53:37.678861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":44,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-13T14:39:07.954196+0000","last_change":"2024-09-13T14:38:03.563200+0000","last_active":"2024-09-13T14:39:07.954196+0000","last_peered":"2024-09-13T14:39:07.954196+0000","last_clean":"2024-09-13T14:39:07.954196+0000","last_became_active":"2024-09-13T14:38:03.534258+0000","last_became_peered":"2024-09-13T14:38:03.534258+0000","last_unstale":"2024-09-13T14:39:07.954196+0000","last_undegraded":"2024-09-13T14:39:07.954196+0000","last_fullsized":"2024-09-13T14:39:07.954196+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:49:07.814419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":6646661144,"num_objects":101421,"num_object_clones":0,"num_object_copies":304263,"num_objects_missing_on_primary":4304,"num_objects_missing":4304,"num_objects_degraded":47698,"num_objects_misplaced":1995,"num_objects_unfound":0,"num_objects_dirty":101421,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124205,"num_write_kb":7219969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46400,"num_bytes_recovered":3018850304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20159209472,"data_stored":20159201328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86928,"ondisk_log_size":86928,"up":48,"acting":41,"num_store_stats":6},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1538,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":17412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25132,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":7,"num_objects_missing":7,"num_objects_degraded":16,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16793600,"data_stored":16777292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":23,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234345,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3611824,"kb_used_data":3157416,"kb_used_omap":1,"kb_used_meta":454398,"kb_avail":90133328,"statfs":{"total":95995035648,"available":92296527872,"internally_reserved":0,"allocated":3233193984,"data_stored":3231791264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":465304011},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":239,"seq":1026497183763,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1894712,"kb_used_data":1796268,"kb_used_omap":8,"kb_used_meta":98423,"kb_avail":91850440,"statfs":{"total":95995035648,"available":94054850560,"internally_reserved":0,"allocated":1839378432,"data_stored":1837972707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8453,"internal_metadata":100785915},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757610,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2532436,"kb_used_data":1945932,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":91212716,"statfs":{"total":95995035648,"available":93401821184,"internally_reserved":0,"allocated":1992634368,"data_stored":1991247570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":600570310},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019253,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3956384,"kb_used_data":3375640,"kb_used_omap":1,"kb_used_meta":580734,"kb_avail":89788768,"statfs":{"total":95995035648,"available":91943698432,"internally_reserved":0,"allocated":3456655360,"data_stored":3455261085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":594672070},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435437,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5410860,"kb_used_data":4865504,"kb_used_omap":5,"kb_used_meta":545338,"kb_avail":88334292,"statfs":{"total":95995035648,"available":90454315008,"internally_reserved":0,"allocated":4982276096,"data_stored":4980877619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":558426238},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":446676598965,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":425201762491,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926015,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5216668,"kb_used_data":4593556,"kb_used_omap":7,"kb_used_meta":623096,"kb_avail":88528484,"statfs":{"total":95995035648,"available":90653167616,"internally_reserved":0,"allocated":4703801344,"data_stored":4702406741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":638050600},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6860,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4694937600,"data_stored":4694933528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4968812544,"data_stored":4968808472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3451846656,"data_stored":3451846656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1983709184,"data_stored":1983709184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1830944768,"data_stored":1830944768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3228958720,"data_stored":3228958720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:39:20.577 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:39:20.600 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:20.624 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:20.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:20.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:20.726 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:20.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:20.749 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:20.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:20.850 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:20.871 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:20.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:20.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:20.974 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:20.999 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:21.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:21.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:21.101 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:21.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:21.127 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:21.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:21.127 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:20 smithi102 ceph-mon[115269]: from='client.35616 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:39:21.128 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:20 smithi102 ceph-mon[115269]: pgmap v829: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.2 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 169 op/s; 47662/303204 objects degraded (15.719%); 1997/303204 objects misplaced (0.659%); 6.7 MiB/s, 107 objects/s recovering 2024-09-13T14:39:21.128 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:20 smithi102 ceph-mon[110183]: from='client.35616 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:39:21.128 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:20 smithi102 ceph-mon[110183]: pgmap v829: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.2 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 169 op/s; 47662/303204 objects degraded (15.719%); 1997/303204 objects misplaced (0.659%); 6.7 MiB/s, 107 objects/s recovering 2024-09-13T14:39:21.227 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:21.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:20 smithi195 ceph-mon[104262]: from='client.35616 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:39:21.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:20 smithi195 ceph-mon[104262]: pgmap v829: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.2 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 169 op/s; 47662/303204 objects degraded (15.719%); 1997/303204 objects misplaced (0.659%); 6.7 MiB/s, 107 objects/s recovering 2024-09-13T14:39:21.252 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:21.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:21.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:21.354 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:21.376 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:21.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:21.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:21.477 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:39:21.478 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:21.481 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:39:21.501 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:21.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:21.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:21.603 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:21.627 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:21.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:21.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:21.728 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:21.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:21.751 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:21.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:21.852 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:21.878 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:21.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:21.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:21.980 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:22.004 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:22.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:22.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:22.106 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:22.131 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:22.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:22.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:22.233 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:22.256 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:22.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:22.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:22.358 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:22.384 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:22.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:22.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:22.486 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:22.511 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:22.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:22.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:22.613 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:22.635 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:22.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:22.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:22.737 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:22.758 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:22.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:22.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:22.860 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:22.882 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:22.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:22.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:22.986 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:23.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:23.012 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:23.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:23.049 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:22 smithi102 ceph-mon[110183]: pgmap v830: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.2 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 166 op/s; 47614/302805 objects degraded (15.724%); 1997/302805 objects misplaced (0.660%); 6.8 MiB/s, 108 objects/s recovering 2024-09-13T14:39:23.049 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:22 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3455475010' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:39:23.050 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:22 smithi102 ceph-mon[115269]: pgmap v830: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.2 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 166 op/s; 47614/302805 objects degraded (15.724%); 1997/302805 objects misplaced (0.660%); 6.8 MiB/s, 108 objects/s recovering 2024-09-13T14:39:23.050 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:22 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3455475010' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:39:23.112 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:23.134 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:23.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:23.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:23.141 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:39:23.141 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:39:23.141 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:39:23.141 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:39:23.142 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:39:23.142 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:39:23.142 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:39:23.142 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:39:23.142 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:39:23.142 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:39:23.142 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:39:23.142 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:39:23.143 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:39:23.143 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:39:23.143 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:39:23.143 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:39:23.235 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:23.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:22 smithi195 ceph-mon[104262]: pgmap v830: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.2 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 166 op/s; 47614/302805 objects degraded (15.724%); 1997/302805 objects misplaced (0.660%); 6.8 MiB/s, 108 objects/s recovering 2024-09-13T14:39:23.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:22 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3455475010' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:39:23.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:23.259 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:23.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:23.360 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:23.384 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:23.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:23.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:23.486 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:23.509 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:23.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:23.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:23.611 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:23.633 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:23.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:23.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:23.735 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:23.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:23.757 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:23.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:23.858 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:23.881 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:23.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:23.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:23.983 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:24.005 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:24.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:24.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:24.107 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:24.134 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:24.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:24.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:24.199 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:23 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 47614/302805 objects degraded (15.724%), 14 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:24.199 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:23 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 47614/302805 objects degraded (15.724%), 14 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:24.236 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:24.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:23 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 47614/302805 objects degraded (15.724%), 14 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:24.258 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:24.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:24.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:24.360 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:24.383 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:24.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:24.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:24.485 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:24.509 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:24.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:24.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:24.611 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:24.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:24.633 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:24.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:24.734 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:24.756 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:24.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:24.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:24.858 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:24.882 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:24.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:24.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:24.976 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is set 2024-09-13T14:39:24.984 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:25.008 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:25.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:25.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:25.110 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:25.133 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:25.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:25.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:25.235 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:25.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:24 smithi195 ceph-mon[104262]: pgmap v831: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.1 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 166 op/s; 46977/300777 objects degraded (15.619%); 1997/300777 objects misplaced (0.664%); 7.5 MiB/s, 119 objects/s recovering 2024-09-13T14:39:25.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:24 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3699490844' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:39:25.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:24 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:39:25.260 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:25.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:25.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:25.351 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:24 smithi102 ceph-mon[110183]: pgmap v831: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.1 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 166 op/s; 46977/300777 objects degraded (15.619%); 1997/300777 objects misplaced (0.664%); 7.5 MiB/s, 119 objects/s recovering 2024-09-13T14:39:25.351 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:24 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3699490844' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:39:25.351 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:24 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:39:25.352 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:24 smithi102 ceph-mon[115269]: pgmap v831: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.1 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 166 op/s; 46977/300777 objects degraded (15.619%); 1997/300777 objects misplaced (0.664%); 7.5 MiB/s, 119 objects/s recovering 2024-09-13T14:39:25.352 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:24 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3699490844' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:39:25.352 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:24 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:39:25.362 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:25.385 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:25.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:25.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:25.487 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:25.508 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:25.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:25.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:25.611 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:25.634 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:25.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:25.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:25.737 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:25.763 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:25.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:25.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:25.865 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:25.890 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:25.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:25.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:25.992 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:26.016 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:26.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:26.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:26.022 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:25 smithi102 ceph-mon[110183]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:39:26.023 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:25 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:39:26.023 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:25 smithi102 ceph-mon[110183]: osdmap e260: 8 total, 8 up, 6 in 2024-09-13T14:39:26.023 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:25 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:39:26.023 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:25 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:39:26.024 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:25 smithi102 ceph-mon[115269]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:39:26.024 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:25 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:39:26.024 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:25 smithi102 ceph-mon[115269]: osdmap e260: 8 total, 8 up, 6 in 2024-09-13T14:39:26.024 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:25 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:39:26.024 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:25 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:39:26.118 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:26.139 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:26.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:26.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:26.241 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:26.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:25 smithi195 ceph-mon[104262]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:39:26.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:25 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:39:26.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:25 smithi195 ceph-mon[104262]: osdmap e260: 8 total, 8 up, 6 in 2024-09-13T14:39:26.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:25 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:39:26.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:25 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:39:26.266 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:26.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:26.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:26.368 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:26.391 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:26.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:26.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:26.425 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:39:26.493 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:26.515 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:26.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:26.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:26.617 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:26.643 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:26.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:26.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:26.745 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:26.766 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:26.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:26.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:26.847 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:39:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:39:26] "GET /metrics HTTP/1.1" 200 36544 "" "Prometheus/2.51.0" 2024-09-13T14:39:26.878 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:26.953 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:26.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:26.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:27.056 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:27.092 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:27.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:27.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:27.134 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:39:27.135 INFO:teuthology.orchestra.run.smithi102.stdout: "size": 20, 2024-09-13T14:39:27.135 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 600, 2024-09-13T14:39:27.135 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [] 2024-09-13T14:39:27.135 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:39:27.135 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:26 smithi102 ceph-mon[110183]: pgmap v833: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.1 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 166 op/s; 46977/300777 objects degraded (15.619%); 1997/300777 objects misplaced (0.664%); 7.5 MiB/s, 119 objects/s recovering 2024-09-13T14:39:27.135 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:26 smithi102 ceph-mon[115269]: pgmap v833: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.1 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 166 op/s; 46977/300777 objects degraded (15.619%); 1997/300777 objects misplaced (0.664%); 7.5 MiB/s, 119 objects/s recovering 2024-09-13T14:39:27.194 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:27.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:27.224 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:27.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:27.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:26 smithi195 ceph-mon[104262]: pgmap v833: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.1 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 166 op/s; 46977/300777 objects degraded (15.619%); 1997/300777 objects misplaced (0.664%); 7.5 MiB/s, 119 objects/s recovering 2024-09-13T14:39:27.325 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:27.350 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:27.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:27.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:27.452 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:27.479 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:27.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:27.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:27.581 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:27.604 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:27.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:27.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:27.706 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:27.730 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:27.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:27.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:27.833 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:27.857 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:27.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:27.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:27.959 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:27.983 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:27.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:27.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:28.085 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:28.107 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:28.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:28.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:28.209 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:28.220 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:39:28.231 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:28.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:28.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:28.333 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:28.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:28.357 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:28.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:28.458 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:28.481 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:28.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:28.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:28.574 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T14:39:28.583 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:28.605 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:28.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:28.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:28.625 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:39:28.707 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:28.732 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:28.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:28.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:28.834 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:28.857 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:28.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:28.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:28.881 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:39:28.959 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:28.986 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:28.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:28.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:29.088 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:29.109 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:28 smithi102 ceph-mon[110183]: pgmap v834: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.1 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 154 op/s; 46203/299712 objects degraded (15.416%); 1997/299712 objects misplaced (0.666%); 9.0 MiB/s, 143 objects/s recovering 2024-09-13T14:39:29.109 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:28 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 46203/299712 objects degraded (15.416%), 14 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:29.109 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:28 smithi102 ceph-mon[115269]: pgmap v834: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.1 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 154 op/s; 46203/299712 objects degraded (15.416%); 1997/299712 objects misplaced (0.666%); 9.0 MiB/s, 143 objects/s recovering 2024-09-13T14:39:29.109 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:28 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 46203/299712 objects degraded (15.416%), 14 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:29.114 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:29.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:29.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:29.216 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:29.242 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:29.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:29.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:29.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:28 smithi195 ceph-mon[104262]: pgmap v834: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.1 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 154 op/s; 46203/299712 objects degraded (15.416%); 1997/299712 objects misplaced (0.666%); 9.0 MiB/s, 143 objects/s recovering 2024-09-13T14:39:29.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:28 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 46203/299712 objects degraded (15.416%), 14 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:29.343 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:29.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:29.372 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:29.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:29.473 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:29.497 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:29.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:29.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:29.598 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:29.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:29.627 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:29.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:29.728 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:29.752 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:29.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:29.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:29.854 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:29.878 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:29.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:29.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:29.980 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:30.008 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:30.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:30.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:30.110 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:30.136 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:30.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:30.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:30.238 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:30.264 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:30.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:30.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:30.303 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:39:30.304 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:39:30.365 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:30.392 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:30.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:30.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:30.494 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:30.519 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:30.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:30.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:30.621 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:30.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:30.645 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:30.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:30.748 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:30.772 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:30.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:30.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:30.875 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:30.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:30.901 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:30.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:31.002 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:31.025 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:31.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:31.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:31.128 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:31.153 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:31.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:31.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:31.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:30 smithi195 ceph-mon[104262]: pgmap v835: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.1 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 145 op/s; 46165/298824 objects degraded (15.449%); 1997/298824 objects misplaced (0.668%); 8.9 MiB/s, 141 objects/s recovering 2024-09-13T14:39:31.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:30 smithi195 ceph-mon[104262]: from='client.35634 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:39:31.258 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:31.262 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:39:31.287 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:31.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:31.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:31.366 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:30 smithi102 ceph-mon[115269]: pgmap v835: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.1 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 145 op/s; 46165/298824 objects degraded (15.449%); 1997/298824 objects misplaced (0.668%); 8.9 MiB/s, 141 objects/s recovering 2024-09-13T14:39:31.367 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:30 smithi102 ceph-mon[115269]: from='client.35634 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:39:31.367 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:30 smithi102 ceph-mon[110183]: pgmap v835: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.1 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 145 op/s; 46165/298824 objects degraded (15.449%); 1997/298824 objects misplaced (0.668%); 8.9 MiB/s, 141 objects/s recovering 2024-09-13T14:39:31.367 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:30 smithi102 ceph-mon[110183]: from='client.35634 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:39:31.388 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:31.414 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:31.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:31.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:31.516 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:31.541 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:31.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:31.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:31.643 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:31.666 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:31.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:31.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:31.768 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:31.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:31.791 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:31.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:31.892 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:31.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:31.915 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:31.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:32.016 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:32.039 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:32.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:32.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:32.085 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:39:32.141 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:32.166 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:32.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:32.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:32.268 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:32.294 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:32.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:32.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:32.395 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:32.418 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:32.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:32.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:32.520 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:32.546 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:32.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:32.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:32.648 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:32.672 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:32.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:32.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:32.774 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:32.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:32.798 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:32.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:32.899 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:32.921 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:32.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:32.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:33.023 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:33.048 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:33.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:33.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:33.150 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:33.175 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:33.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:33.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:33.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:32 smithi195 ceph-mon[104262]: pgmap v836: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.1 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 147 op/s; 46120/298392 objects degraded (15.456%); 1997/298392 objects misplaced (0.669%); 8.9 MiB/s, 142 objects/s recovering 2024-09-13T14:39:33.276 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:32 smithi102 ceph-mon[110183]: pgmap v836: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.1 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 147 op/s; 46120/298392 objects degraded (15.456%); 1997/298392 objects misplaced (0.669%); 8.9 MiB/s, 142 objects/s recovering 2024-09-13T14:39:33.276 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:32 smithi102 ceph-mon[115269]: pgmap v836: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.1 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 147 op/s; 46120/298392 objects degraded (15.456%); 1997/298392 objects misplaced (0.669%); 8.9 MiB/s, 142 objects/s recovering 2024-09-13T14:39:33.277 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:33.300 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:33.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:33.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:33.403 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:33.430 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:33.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:33.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:33.532 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:33.558 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:33.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:33.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:33.660 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:33.662 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":835,"stamp":"2024-09-13T14:39:29.460934+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6530799613,"num_objects":99608,"num_object_clones":0,"num_object_copies":298824,"num_objects_missing_on_primary":4216,"num_objects_missing":4216,"num_objects_degraded":46165,"num_objects_misplaced":1997,"num_objects_unfound":0,"num_objects_dirty":99608,"num_whiteouts":0,"num_read":6377,"num_read_kb":13834,"num_write":127169,"num_write_kb":7235010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47978,"num_bytes_recovered":3121220725,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89451,"ondisk_log_size":89451,"up":267,"acting":259,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":309,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":22417844,"kb_used_data":19493244,"kb_used_omap":26,"kb_used_meta":2924517,"kb_avail":540053068,"statfs":{"total":575970213888,"available":553014341632,"internally_reserved":0,"allocated":19961081856,"data_stored":19952628041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27144,"internal_metadata":2994705912},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1,1,1,0,2,1,6,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-95682560,"num_objects":-1460,"num_object_clones":0,"num_object_copies":-4380,"num_objects_missing_on_primary":-95,"num_objects_missing":-95,"num_objects_degraded":-1497,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1460,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1417,"num_bytes_recovered":92864512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003040"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":412,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.974503+0000","last_change":"2024-09-13T14:36:13.859927+0000","last_active":"2024-09-13T14:39:24.974503+0000","last_peered":"2024-09-13T14:39:24.974503+0000","last_clean":"2024-09-13T14:39:24.974503+0000","last_became_active":"2024-09-13T14:36:13.859505+0000","last_became_peered":"2024-09-13T14:36:13.859505+0000","last_unstale":"2024-09-13T14:39:24.974503+0000","last_undegraded":"2024-09-13T14:39:24.974503+0000","last_fullsized":"2024-09-13T14:39:24.974503+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:12:20.989627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":260,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.407286+0000","last_change":"2024-09-13T14:37:48.130968+0000","last_active":"2024-09-13T14:39:15.407286+0000","last_peered":"2024-09-13T14:39:15.407286+0000","last_clean":"2024-09-13T14:39:15.407286+0000","last_became_active":"2024-09-13T14:37:48.130603+0000","last_became_peered":"2024-09-13T14:37:48.130603+0000","last_unstale":"2024-09-13T14:39:15.407286+0000","last_undegraded":"2024-09-13T14:39:15.407286+0000","last_fullsized":"2024-09-13T14:39:15.407286+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:42.626386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":57,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.408732+0000","last_change":"2024-09-13T14:36:13.862181+0000","last_active":"2024-09-13T14:39:15.408732+0000","last_peered":"2024-09-13T14:39:15.408732+0000","last_clean":"2024-09-13T14:39:15.408732+0000","last_became_active":"2024-09-13T14:36:13.861873+0000","last_became_peered":"2024-09-13T14:36:13.861873+0000","last_unstale":"2024-09-13T14:39:15.408732+0000","last_undegraded":"2024-09-13T14:39:15.408732+0000","last_fullsized":"2024-09-13T14:39:15.408732+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:14.085700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":49,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.977601+0000","last_change":"2024-09-13T14:38:53.896672+0000","last_active":"2024-09-13T14:39:24.977601+0000","last_peered":"2024-09-13T14:39:24.977601+0000","last_clean":"2024-09-13T14:39:24.977601+0000","last_became_active":"2024-09-13T14:38:53.896346+0000","last_became_peered":"2024-09-13T14:38:53.896346+0000","last_unstale":"2024-09-13T14:39:24.977601+0000","last_undegraded":"2024-09-13T14:39:24.977601+0000","last_fullsized":"2024-09-13T14:39:24.977601+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:57.789686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":49,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.977933+0000","last_change":"2024-09-13T14:38:53.893021+0000","last_active":"2024-09-13T14:39:24.977933+0000","last_peered":"2024-09-13T14:39:24.977933+0000","last_clean":"2024-09-13T14:39:24.977933+0000","last_became_active":"2024-09-13T14:38:53.892668+0000","last_became_peered":"2024-09-13T14:38:53.892668+0000","last_unstale":"2024-09-13T14:39:24.977933+0000","last_undegraded":"2024-09-13T14:39:24.977933+0000","last_fullsized":"2024-09-13T14:39:24.977933+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:57.180738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":229,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.975628+0000","last_change":"2024-09-13T14:37:48.134642+0000","last_active":"2024-09-13T14:39:24.975628+0000","last_peered":"2024-09-13T14:39:24.975628+0000","last_clean":"2024-09-13T14:39:24.975628+0000","last_became_active":"2024-09-13T14:37:48.134402+0000","last_became_peered":"2024-09-13T14:37:48.134402+0000","last_unstale":"2024-09-13T14:39:24.975628+0000","last_undegraded":"2024-09-13T14:39:24.975628+0000","last_fullsized":"2024-09-13T14:39:24.975628+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:51.313663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":431,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.974523+0000","last_change":"2024-09-13T14:38:53.903022+0000","last_active":"2024-09-13T14:39:24.974523+0000","last_peered":"2024-09-13T14:39:24.974523+0000","last_clean":"2024-09-13T14:39:24.974523+0000","last_became_active":"2024-09-13T14:38:53.902448+0000","last_became_peered":"2024-09-13T14:38:53.902448+0000","last_unstale":"2024-09-13T14:39:24.974523+0000","last_undegraded":"2024-09-13T14:39:24.974523+0000","last_fullsized":"2024-09-13T14:39:24.974523+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:24.690401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":52,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.977064+0000","last_change":"2024-09-13T14:38:03.554500+0000","last_active":"2024-09-13T14:39:24.977064+0000","last_peered":"2024-09-13T14:39:24.977064+0000","last_clean":"2024-09-13T14:39:24.977064+0000","last_became_active":"2024-09-13T14:38:03.532910+0000","last_became_peered":"2024-09-13T14:38:03.532910+0000","last_unstale":"2024-09-13T14:39:24.977064+0000","last_undegraded":"2024-09-13T14:39:24.977064+0000","last_fullsized":"2024-09-13T14:39:24.977064+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:27:50.843498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"260'7751","reported_seq":9835,"reported_epoch":260,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:39:25.938482+0000","last_change":"2024-09-13T14:37:49.715669+0000","last_active":"2024-09-13T14:39:25.938482+0000","last_peered":"2024-09-13T14:39:25.938482+0000","last_clean":"2024-09-13T14:37:46.417003+0000","last_became_active":"2024-09-13T14:37:49.690961+0000","last_became_peered":"2024-09-13T14:37:49.690961+0000","last_unstale":"2024-09-13T14:39:25.938482+0000","last_undegraded":"2024-09-13T14:37:49.182535+0000","last_fullsized":"2024-09-13T14:37:49.126746+0000","mapping_epoch":240,"log_start":"210'2000","ondisk_log_start":"210'2000","created":205,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5751,"log_dups_size":0,"ondisk_log_size":5751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:48:52.526429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":401670144,"num_objects":6129,"num_object_clones":0,"num_object_copies":18387,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4387,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7751,"num_write_kb":444160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4765,"num_bytes_recovered":312279040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,4],"avail_no_missing":["5","0","4"],"object_location_counts":[{"shards":"0,4,5","objects":4308},{"shards":"4,5","objects":1821}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":21,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.408576+0000","last_change":"2024-09-13T14:38:53.940238+0000","last_active":"2024-09-13T14:39:15.408576+0000","last_peered":"2024-09-13T14:39:15.408576+0000","last_clean":"2024-09-13T14:39:15.408576+0000","last_became_active":"2024-09-13T14:38:53.892825+0000","last_became_peered":"2024-09-13T14:38:53.892825+0000","last_unstale":"2024-09-13T14:39:15.408576+0000","last_undegraded":"2024-09-13T14:39:15.408576+0000","last_fullsized":"2024-09-13T14:39:15.408576+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:46.666581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":77,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.407458+0000","last_change":"2024-09-13T14:38:53.898907+0000","last_active":"2024-09-13T14:39:15.407458+0000","last_peered":"2024-09-13T14:39:15.407458+0000","last_clean":"2024-09-13T14:39:15.407458+0000","last_became_active":"2024-09-13T14:38:53.898572+0000","last_became_peered":"2024-09-13T14:38:53.898572+0000","last_unstale":"2024-09-13T14:39:15.407458+0000","last_undegraded":"2024-09-13T14:39:15.407458+0000","last_fullsized":"2024-09-13T14:39:15.407458+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:29.222150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"260'7985","reported_seq":22113,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:27.072824+0000","last_change":"2024-09-13T14:38:57.930032+0000","last_active":"2024-09-13T14:39:27.072824+0000","last_peered":"2024-09-13T14:39:27.072824+0000","last_clean":"2024-09-13T14:39:27.072824+0000","last_became_active":"2024-09-13T14:38:57.921458+0000","last_became_peered":"2024-09-13T14:38:57.921458+0000","last_unstale":"2024-09-13T14:39:27.072824+0000","last_undegraded":"2024-09-13T14:39:27.072824+0000","last_fullsized":"2024-09-13T14:39:27.072824+0000","mapping_epoch":256,"log_start":"212'3139","ondisk_log_start":"212'3139","created":205,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":4846,"log_dups_size":0,"ondisk_log_size":4846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:27:12.294038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410058752,"num_objects":6257,"num_object_clones":0,"num_object_copies":18771,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7985,"num_write_kb":455744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6990,"num_bytes_recovered":458096640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":21,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.407895+0000","last_change":"2024-09-13T14:38:53.904885+0000","last_active":"2024-09-13T14:39:15.407895+0000","last_peered":"2024-09-13T14:39:15.407895+0000","last_clean":"2024-09-13T14:39:15.407895+0000","last_became_active":"2024-09-13T14:38:53.891726+0000","last_became_peered":"2024-09-13T14:38:53.891726+0000","last_unstale":"2024-09-13T14:39:15.407895+0000","last_undegraded":"2024-09-13T14:39:15.407895+0000","last_fullsized":"2024-09-13T14:39:15.407895+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:57.079205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":266,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.404181+0000","last_change":"2024-09-13T14:37:48.160504+0000","last_active":"2024-09-13T14:39:15.404181+0000","last_peered":"2024-09-13T14:39:15.404181+0000","last_clean":"2024-09-13T14:39:15.404181+0000","last_became_active":"2024-09-13T14:37:48.160181+0000","last_became_peered":"2024-09-13T14:37:48.160181+0000","last_unstale":"2024-09-13T14:39:15.404181+0000","last_undegraded":"2024-09-13T14:39:15.404181+0000","last_fullsized":"2024-09-13T14:39:15.404181+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:37.087008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"260'7922","reported_seq":9672,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:27.846686+0000","last_change":"2024-09-13T14:38:04.531914+0000","last_active":"2024-09-13T14:39:27.846686+0000","last_peered":"2024-09-13T14:39:27.846686+0000","last_clean":"2024-09-13T14:39:27.846686+0000","last_became_active":"2024-09-13T14:38:04.523448+0000","last_became_peered":"2024-09-13T14:38:04.523448+0000","last_unstale":"2024-09-13T14:39:27.846686+0000","last_undegraded":"2024-09-13T14:39:27.846686+0000","last_fullsized":"2024-09-13T14:39:27.846686+0000","mapping_epoch":245,"log_start":"210'2000","ondisk_log_start":"210'2000","created":205,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5922,"log_dups_size":0,"ondisk_log_size":5922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:06.536896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":409731072,"num_objects":6252,"num_object_clones":0,"num_object_copies":18756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7922,"num_write_kb":453568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":811,"num_bytes_recovered":53149696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":258,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.407466+0000","last_change":"2024-09-13T14:37:49.611774+0000","last_active":"2024-09-13T14:39:15.407466+0000","last_peered":"2024-09-13T14:39:15.407466+0000","last_clean":"2024-09-13T14:39:15.407466+0000","last_became_active":"2024-09-13T14:37:49.611088+0000","last_became_peered":"2024-09-13T14:37:49.611088+0000","last_unstale":"2024-09-13T14:39:15.407466+0000","last_undegraded":"2024-09-13T14:39:15.407466+0000","last_fullsized":"2024-09-13T14:39:15.407466+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:08.933213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.408165+0000","last_change":"2024-09-13T14:38:53.899211+0000","last_active":"2024-09-13T14:39:15.408165+0000","last_peered":"2024-09-13T14:39:15.408165+0000","last_clean":"2024-09-13T14:39:15.408165+0000","last_became_active":"2024-09-13T14:38:53.898927+0000","last_became_peered":"2024-09-13T14:38:53.898927+0000","last_unstale":"2024-09-13T14:39:15.408165+0000","last_undegraded":"2024-09-13T14:39:15.408165+0000","last_fullsized":"2024-09-13T14:39:15.408165+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:00:28.857611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"260'7811","reported_seq":7876,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:27.387252+0000","last_change":"2024-09-13T14:35:24.004651+0000","last_active":"2024-09-13T14:39:27.387252+0000","last_peered":"2024-09-13T14:39:27.387252+0000","last_clean":"2024-09-13T14:39:27.387252+0000","last_became_active":"2024-09-13T14:35:24.004358+0000","last_became_peered":"2024-09-13T14:35:24.004358+0000","last_unstale":"2024-09-13T14:39:27.387252+0000","last_undegraded":"2024-09-13T14:39:27.387252+0000","last_fullsized":"2024-09-13T14:39:27.387252+0000","mapping_epoch":205,"log_start":"209'1484","ondisk_log_start":"209'1484","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6327,"log_dups_size":0,"ondisk_log_size":6327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:57:38.023669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403767296,"num_objects":6161,"num_object_clones":0,"num_object_copies":18483,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7811,"num_write_kb":447104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":68,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.408511+0000","last_change":"2024-09-13T14:37:49.168003+0000","last_active":"2024-09-13T14:39:15.408511+0000","last_peered":"2024-09-13T14:39:15.408511+0000","last_clean":"2024-09-13T14:39:15.408511+0000","last_became_active":"2024-09-13T14:37:49.167638+0000","last_became_peered":"2024-09-13T14:37:49.167638+0000","last_unstale":"2024-09-13T14:39:15.408511+0000","last_undegraded":"2024-09-13T14:39:15.408511+0000","last_fullsized":"2024-09-13T14:39:15.408511+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:29.051981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":42,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.977664+0000","last_change":"2024-09-13T14:38:03.528381+0000","last_active":"2024-09-13T14:39:24.977664+0000","last_peered":"2024-09-13T14:39:24.977664+0000","last_clean":"2024-09-13T14:39:24.977664+0000","last_became_active":"2024-09-13T14:38:03.527282+0000","last_became_peered":"2024-09-13T14:38:03.527282+0000","last_unstale":"2024-09-13T14:39:24.977664+0000","last_undegraded":"2024-09-13T14:39:24.977664+0000","last_fullsized":"2024-09-13T14:39:24.977664+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:24.435067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.f","version":"260'7861","reported_seq":8106,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:27.875947+0000","last_change":"2024-09-13T14:38:16.371079+0000","last_active":"2024-09-13T14:39:27.875947+0000","last_peered":"2024-09-13T14:39:27.875947+0000","last_clean":"2024-09-13T14:39:27.875947+0000","last_became_active":"2024-09-13T14:38:12.859883+0000","last_became_peered":"2024-09-13T14:38:12.859883+0000","last_unstale":"2024-09-13T14:39:27.875947+0000","last_undegraded":"2024-09-13T14:39:27.875947+0000","last_fullsized":"2024-09-13T14:39:27.875947+0000","mapping_epoch":245,"log_start":"212'2663","ondisk_log_start":"212'2663","created":205,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5198,"log_dups_size":0,"ondisk_log_size":5198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:08.436532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":407961600,"num_objects":6225,"num_object_clones":0,"num_object_copies":18675,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7861,"num_write_kb":450752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":3670016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":400,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.973720+0000","last_change":"2024-09-13T14:38:03.549595+0000","last_active":"2024-09-13T14:39:24.973720+0000","last_peered":"2024-09-13T14:39:24.973720+0000","last_clean":"2024-09-13T14:39:24.973720+0000","last_became_active":"2024-09-13T14:38:03.533331+0000","last_became_peered":"2024-09-13T14:38:03.533331+0000","last_unstale":"2024-09-13T14:39:24.973720+0000","last_undegraded":"2024-09-13T14:39:24.973720+0000","last_fullsized":"2024-09-13T14:39:24.973720+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:59:02.470343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":242,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.404385+0000","last_change":"2024-09-13T14:37:48.127600+0000","last_active":"2024-09-13T14:39:15.404385+0000","last_peered":"2024-09-13T14:39:15.404385+0000","last_clean":"2024-09-13T14:39:15.404385+0000","last_became_active":"2024-09-13T14:37:48.127349+0000","last_became_peered":"2024-09-13T14:37:48.127349+0000","last_unstale":"2024-09-13T14:39:15.404385+0000","last_undegraded":"2024-09-13T14:39:15.404385+0000","last_fullsized":"2024-09-13T14:39:15.404385+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:57.268374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"260'7757","reported_seq":9361,"reported_epoch":260,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:39:26.098934+0000","last_change":"2024-09-13T14:38:14.979229+0000","last_active":"2024-09-13T14:39:26.098934+0000","last_peered":"2024-09-13T14:39:26.098934+0000","last_clean":"2024-09-13T14:36:13.131873+0000","last_became_active":"2024-09-13T14:38:10.443954+0000","last_became_peered":"2024-09-13T14:38:10.443954+0000","last_unstale":"2024-09-13T14:39:26.098934+0000","last_undegraded":"2024-09-13T14:38:10.384643+0000","last_fullsized":"2024-09-13T14:38:04.517046+0000","mapping_epoch":245,"log_start":"210'2000","ondisk_log_start":"210'2000","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5757,"log_dups_size":0,"ondisk_log_size":5757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:26:53.493170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":409796608,"num_objects":6253,"num_object_clones":0,"num_object_copies":18759,"num_objects_missing_on_primary":4026,"num_objects_missing":4026,"num_objects_degraded":4167,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7757,"num_write_kb":448320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":747,"num_bytes_recovered":48889856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":1},{"shards":"0,3,5,7","objects":12},{"shards":"0,3,7","objects":4025},{"shards":"0,5","objects":2210},{"shards":"3,5,7","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":175,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.404008+0000","last_change":"2024-09-13T14:37:48.127604+0000","last_active":"2024-09-13T14:39:15.404008+0000","last_peered":"2024-09-13T14:39:15.404008+0000","last_clean":"2024-09-13T14:39:15.404008+0000","last_became_active":"2024-09-13T14:37:48.127300+0000","last_became_peered":"2024-09-13T14:37:48.127300+0000","last_unstale":"2024-09-13T14:39:15.404008+0000","last_undegraded":"2024-09-13T14:39:15.404008+0000","last_fullsized":"2024-09-13T14:39:15.404008+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:51.999506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":295,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.404533+0000","last_change":"2024-09-13T14:38:03.529967+0000","last_active":"2024-09-13T14:39:15.404533+0000","last_peered":"2024-09-13T14:39:15.404533+0000","last_clean":"2024-09-13T14:39:15.404533+0000","last_became_active":"2024-09-13T14:38:03.529646+0000","last_became_peered":"2024-09-13T14:38:03.529646+0000","last_unstale":"2024-09-13T14:39:15.404533+0000","last_undegraded":"2024-09-13T14:39:15.404533+0000","last_fullsized":"2024-09-13T14:39:15.404533+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:52:05.193279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"260'7909","reported_seq":8234,"reported_epoch":260,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:39:27.821685+0000","last_change":"2024-09-13T14:38:03.544515+0000","last_active":"2024-09-13T14:39:27.821685+0000","last_peered":"2024-09-13T14:39:27.821685+0000","last_clean":"2024-09-13T14:35:51.090657+0000","last_became_active":"2024-09-13T14:37:49.619314+0000","last_became_peered":"2024-09-13T14:37:49.619314+0000","last_unstale":"2024-09-13T14:39:27.821685+0000","last_undegraded":"2024-09-13T14:37:49.147721+0000","last_fullsized":"2024-09-13T14:37:49.120361+0000","mapping_epoch":240,"log_start":"210'2000","ondisk_log_start":"210'2000","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5909,"log_dups_size":0,"ondisk_log_size":5909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":408289304,"num_objects":6231,"num_object_clones":0,"num_object_copies":18693,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4393,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6231,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7909,"num_write_kb":452417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":6291456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":1903}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":255,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.404525+0000","last_change":"2024-09-13T14:36:13.860367+0000","last_active":"2024-09-13T14:39:15.404525+0000","last_peered":"2024-09-13T14:39:15.404525+0000","last_clean":"2024-09-13T14:39:15.404525+0000","last_became_active":"2024-09-13T14:36:13.859982+0000","last_became_peered":"2024-09-13T14:36:13.859982+0000","last_unstale":"2024-09-13T14:39:15.404525+0000","last_undegraded":"2024-09-13T14:39:15.404525+0000","last_fullsized":"2024-09-13T14:39:15.404525+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:08:23.311350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":251,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.404625+0000","last_change":"2024-09-13T14:37:48.147360+0000","last_active":"2024-09-13T14:39:15.404625+0000","last_peered":"2024-09-13T14:39:15.404625+0000","last_clean":"2024-09-13T14:39:15.404625+0000","last_became_active":"2024-09-13T14:37:48.146885+0000","last_became_peered":"2024-09-13T14:37:48.146885+0000","last_unstale":"2024-09-13T14:39:15.404625+0000","last_undegraded":"2024-09-13T14:39:15.404625+0000","last_fullsized":"2024-09-13T14:39:15.404625+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:27.841810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"260'7833","reported_seq":17060,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:27.212946+0000","last_change":"2024-09-13T14:38:39.650332+0000","last_active":"2024-09-13T14:39:27.212946+0000","last_peered":"2024-09-13T14:39:27.212946+0000","last_clean":"2024-09-13T14:39:27.212946+0000","last_became_active":"2024-09-13T14:38:39.640835+0000","last_became_peered":"2024-09-13T14:38:39.640835+0000","last_unstale":"2024-09-13T14:39:27.212946+0000","last_undegraded":"2024-09-13T14:39:27.212946+0000","last_fullsized":"2024-09-13T14:39:27.212946+0000","mapping_epoch":251,"log_start":"212'3051","ondisk_log_start":"212'3051","created":205,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":4782,"log_dups_size":0,"ondisk_log_size":4782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:48.176606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":409010176,"num_objects":6241,"num_object_clones":0,"num_object_copies":18723,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7833,"num_write_kb":450368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4544,"num_bytes_recovered":292945920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":418,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.973940+0000","last_change":"2024-09-13T14:38:53.900041+0000","last_active":"2024-09-13T14:39:24.973940+0000","last_peered":"2024-09-13T14:39:24.973940+0000","last_clean":"2024-09-13T14:39:24.973940+0000","last_became_active":"2024-09-13T14:38:53.899203+0000","last_became_peered":"2024-09-13T14:38:53.899203+0000","last_unstale":"2024-09-13T14:39:24.973940+0000","last_undegraded":"2024-09-13T14:39:24.973940+0000","last_fullsized":"2024-09-13T14:39:24.973940+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:28:07.883094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":350,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.974619+0000","last_change":"2024-09-13T14:26:15.823198+0000","last_active":"2024-09-13T14:39:24.974619+0000","last_peered":"2024-09-13T14:39:24.974619+0000","last_clean":"2024-09-13T14:39:24.974619+0000","last_became_active":"2024-09-13T14:26:15.822717+0000","last_became_peered":"2024-09-13T14:26:15.822717+0000","last_unstale":"2024-09-13T14:39:24.974619+0000","last_undegraded":"2024-09-13T14:39:24.974619+0000","last_fullsized":"2024-09-13T14:39:24.974619+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:22.471622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.6","version":"260'7969","reported_seq":9633,"reported_epoch":260,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-13T14:39:27.933537+0000","last_change":"2024-09-13T14:38:55.132683+0000","last_active":"2024-09-13T14:39:27.933537+0000","last_peered":"2024-09-13T14:39:27.933537+0000","last_clean":"2024-09-13T14:36:13.124368+0000","last_became_active":"2024-09-13T14:38:55.120883+0000","last_became_peered":"2024-09-13T14:38:55.120883+0000","last_unstale":"2024-09-13T14:39:27.933537+0000","last_undegraded":"2024-09-13T14:38:55.026610+0000","last_fullsized":"2024-09-13T14:39:27.933537+0000","mapping_epoch":254,"log_start":"212'3119","ondisk_log_start":"212'3119","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":4850,"log_dups_size":0,"ondisk_log_size":4850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":408223744,"num_objects":6229,"num_object_clones":0,"num_object_copies":18687,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4242,"num_objects_misplaced":1987,"num_objects_unfound":0,"num_objects_dirty":6229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7969,"num_write_kb":454336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":763,"num_bytes_recovered":38797312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,3,5],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3,4","objects":3963},{"shards":"0,3,5","objects":2266}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":414,"reported_epoch":260,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-13T14:39:24.973761+0000","last_change":"2024-09-13T14:38:54.899624+0000","last_active":"2024-09-13T14:39:24.973761+0000","last_peered":"2024-09-13T14:39:24.973761+0000","last_clean":"2024-09-13T14:17:27.401592+0000","last_became_active":"2024-09-13T14:38:54.899282+0000","last_became_peered":"2024-09-13T14:38:54.899282+0000","last_unstale":"2024-09-13T14:39:24.973761+0000","last_undegraded":"2024-09-13T14:38:54.891176+0000","last_fullsized":"2024-09-13T14:39:24.973761+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:07:41.927740+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[3,0,6],"avail_no_missing":["3","4","6"],"object_location_counts":[{"shards":"3,4,6","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":497,"reported_epoch":259,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:39:15.408494+0000","last_change":"2024-09-13T14:38:54.904329+0000","last_active":"2024-09-13T14:39:15.408494+0000","last_peered":"2024-09-13T14:39:15.408494+0000","last_clean":"2024-09-13T14:38:39.648017+0000","last_became_active":"2024-09-13T14:38:54.903798+0000","last_became_peered":"2024-09-13T14:38:54.903798+0000","last_unstale":"2024-09-13T14:39:15.408494+0000","last_undegraded":"2024-09-13T14:38:54.890505+0000","last_fullsized":"2024-09-13T14:38:54.889701+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:18.120931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,3],"avail_no_missing":["5","3","6"],"object_location_counts":[{"shards":"3,5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":353,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.975540+0000","last_change":"2024-09-13T14:37:49.160719+0000","last_active":"2024-09-13T14:39:24.975540+0000","last_peered":"2024-09-13T14:39:24.975540+0000","last_clean":"2024-09-13T14:39:24.975540+0000","last_became_active":"2024-09-13T14:37:49.160401+0000","last_became_peered":"2024-09-13T14:37:49.160401+0000","last_unstale":"2024-09-13T14:39:24.975540+0000","last_undegraded":"2024-09-13T14:39:24.975540+0000","last_fullsized":"2024-09-13T14:39:24.975540+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:21.737945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":102,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.404461+0000","last_change":"2024-09-13T14:37:48.172940+0000","last_active":"2024-09-13T14:39:15.404461+0000","last_peered":"2024-09-13T14:39:15.404461+0000","last_clean":"2024-09-13T14:39:15.404461+0000","last_became_active":"2024-09-13T14:37:48.172060+0000","last_became_peered":"2024-09-13T14:37:48.172060+0000","last_unstale":"2024-09-13T14:39:15.404461+0000","last_undegraded":"2024-09-13T14:39:15.404461+0000","last_fullsized":"2024-09-13T14:39:15.404461+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:07.480645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.5","version":"260'7873","reported_seq":21119,"reported_epoch":260,"state":"active+recovering+degraded","last_fresh":"2024-09-13T14:39:27.387414+0000","last_change":"2024-09-13T14:38:53.941671+0000","last_active":"2024-09-13T14:39:27.387414+0000","last_peered":"2024-09-13T14:39:27.387414+0000","last_clean":"2024-09-13T14:35:51.091736+0000","last_became_active":"2024-09-13T14:38:04.600616+0000","last_became_peered":"2024-09-13T14:38:04.600616+0000","last_unstale":"2024-09-13T14:39:27.387414+0000","last_undegraded":"2024-09-13T14:38:04.591411+0000","last_fullsized":"2024-09-13T14:39:27.387414+0000","mapping_epoch":245,"log_start":"209'1616","ondisk_log_start":"209'1616","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6257,"log_dups_size":0,"ondisk_log_size":6257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:00:44.321678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410845184,"num_objects":6269,"num_object_clones":0,"num_object_copies":18807,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1459,"num_objects_misplaced":8,"num_objects_unfound":0,"num_objects_dirty":6269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7873,"num_write_kb":452544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6574,"num_bytes_recovered":424935424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,4,7","objects":8},{"shards":"0,6,7","objects":4802},{"shards":"0,7","objects":1459}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":475,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.406763+0000","last_change":"2024-09-13T14:26:15.835178+0000","last_active":"2024-09-13T14:39:15.406763+0000","last_peered":"2024-09-13T14:39:15.406763+0000","last_clean":"2024-09-13T14:39:15.406763+0000","last_became_active":"2024-09-13T14:26:15.820283+0000","last_became_peered":"2024-09-13T14:26:15.820283+0000","last_unstale":"2024-09-13T14:39:15.406763+0000","last_undegraded":"2024-09-13T14:39:15.406763+0000","last_fullsized":"2024-09-13T14:39:15.406763+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:30.340038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":42,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.977110+0000","last_change":"2024-09-13T14:38:03.525682+0000","last_active":"2024-09-13T14:39:24.977110+0000","last_peered":"2024-09-13T14:39:24.977110+0000","last_clean":"2024-09-13T14:39:24.977110+0000","last_became_active":"2024-09-13T14:38:03.525406+0000","last_became_peered":"2024-09-13T14:38:03.525406+0000","last_unstale":"2024-09-13T14:39:24.977110+0000","last_undegraded":"2024-09-13T14:39:24.977110+0000","last_fullsized":"2024-09-13T14:39:24.977110+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:41:28.970300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":49,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.977969+0000","last_change":"2024-09-13T14:38:53.991738+0000","last_active":"2024-09-13T14:39:24.977969+0000","last_peered":"2024-09-13T14:39:24.977969+0000","last_clean":"2024-09-13T14:39:24.977969+0000","last_became_active":"2024-09-13T14:38:53.991417+0000","last_became_peered":"2024-09-13T14:38:53.991417+0000","last_unstale":"2024-09-13T14:39:24.977969+0000","last_undegraded":"2024-09-13T14:39:24.977969+0000","last_fullsized":"2024-09-13T14:39:24.977969+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:50:38.559838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"260'7903","reported_seq":22045,"reported_epoch":260,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-13T14:39:27.872134+0000","last_change":"2024-09-13T14:38:17.472071+0000","last_active":"2024-09-13T14:39:27.872134+0000","last_peered":"2024-09-13T14:39:27.872134+0000","last_clean":"2024-09-13T14:37:46.290754+0000","last_became_active":"2024-09-13T14:38:17.381753+0000","last_became_peered":"2024-09-13T14:38:17.381753+0000","last_unstale":"2024-09-13T14:39:27.872134+0000","last_undegraded":"2024-09-13T14:38:13.504140+0000","last_fullsized":"2024-09-13T14:39:27.872134+0000","mapping_epoch":245,"log_start":"210'1959","ondisk_log_start":"210'1959","created":205,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5944,"log_dups_size":0,"ondisk_log_size":5944,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406519808,"num_objects":6203,"num_object_clones":0,"num_object_copies":18609,"num_objects_missing_on_primary":83,"num_objects_missing":83,"num_objects_degraded":4582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7903,"num_write_kb":451392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7053,"num_bytes_recovered":462225408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"0,4,5","objects":4413},{"shards":"0,4,6","objects":1790}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":763,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.973876+0000","last_change":"2024-09-13T14:37:48.131799+0000","last_active":"2024-09-13T14:39:24.973876+0000","last_peered":"2024-09-13T14:39:24.973876+0000","last_clean":"2024-09-13T14:39:24.973876+0000","last_became_active":"2024-09-13T14:37:48.130837+0000","last_became_peered":"2024-09-13T14:37:48.130837+0000","last_unstale":"2024-09-13T14:39:24.973876+0000","last_undegraded":"2024-09-13T14:39:24.973876+0000","last_fullsized":"2024-09-13T14:39:24.973876+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:06:32.663770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":282,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.977282+0000","last_change":"2024-09-13T14:37:49.301531+0000","last_active":"2024-09-13T14:39:24.977282+0000","last_peered":"2024-09-13T14:39:24.977282+0000","last_clean":"2024-09-13T14:39:24.977282+0000","last_became_active":"2024-09-13T14:37:49.301249+0000","last_became_peered":"2024-09-13T14:37:49.301249+0000","last_unstale":"2024-09-13T14:39:24.977282+0000","last_undegraded":"2024-09-13T14:39:24.977282+0000","last_fullsized":"2024-09-13T14:39:24.977282+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:10.977710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":84,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.409175+0000","last_change":"2024-09-13T14:38:53.901427+0000","last_active":"2024-09-13T14:39:15.409175+0000","last_peered":"2024-09-13T14:39:15.409175+0000","last_clean":"2024-09-13T14:39:15.409175+0000","last_became_active":"2024-09-13T14:38:53.901157+0000","last_became_peered":"2024-09-13T14:38:53.901157+0000","last_unstale":"2024-09-13T14:39:15.409175+0000","last_undegraded":"2024-09-13T14:39:15.409175+0000","last_fullsized":"2024-09-13T14:39:15.409175+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:28:21.713025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"260'7842","reported_seq":21874,"reported_epoch":260,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-13T14:39:27.779599+0000","last_change":"2024-09-13T14:38:16.552896+0000","last_active":"2024-09-13T14:39:27.779599+0000","last_peered":"2024-09-13T14:39:27.779599+0000","last_clean":"2024-09-13T14:37:46.405415+0000","last_became_active":"2024-09-13T14:38:16.427510+0000","last_became_peered":"2024-09-13T14:38:16.427510+0000","last_unstale":"2024-09-13T14:39:27.779599+0000","last_undegraded":"2024-09-13T14:38:12.738464+0000","last_fullsized":"2024-09-13T14:39:27.779599+0000","mapping_epoch":245,"log_start":"210'2518","ondisk_log_start":"210'2518","created":205,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403439616,"num_objects":6156,"num_object_clones":0,"num_object_copies":18468,"num_objects_missing_on_primary":100,"num_objects_missing":100,"num_objects_degraded":4572,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7842,"num_write_kb":447936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6999,"num_bytes_recovered":458686464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"0,3,6","objects":1751},{"shards":"0,3,7","objects":4405}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":966,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.407974+0000","last_change":"2024-09-13T14:37:49.147498+0000","last_active":"2024-09-13T14:39:15.407974+0000","last_peered":"2024-09-13T14:39:15.407974+0000","last_clean":"2024-09-13T14:39:15.407974+0000","last_became_active":"2024-09-13T14:37:49.147028+0000","last_became_peered":"2024-09-13T14:37:49.147028+0000","last_unstale":"2024-09-13T14:39:15.407974+0000","last_undegraded":"2024-09-13T14:39:15.407974+0000","last_fullsized":"2024-09-13T14:39:15.407974+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:11:29.050178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":264,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.404399+0000","last_change":"2024-09-13T14:38:03.528133+0000","last_active":"2024-09-13T14:39:15.404399+0000","last_peered":"2024-09-13T14:39:15.404399+0000","last_clean":"2024-09-13T14:39:15.404399+0000","last_became_active":"2024-09-13T14:38:03.527829+0000","last_became_peered":"2024-09-13T14:38:03.527829+0000","last_unstale":"2024-09-13T14:39:15.404399+0000","last_undegraded":"2024-09-13T14:39:15.404399+0000","last_fullsized":"2024-09-13T14:39:15.404399+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:48.254169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":402,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.973514+0000","last_change":"2024-09-13T14:38:53.899755+0000","last_active":"2024-09-13T14:39:24.973514+0000","last_peered":"2024-09-13T14:39:24.973514+0000","last_clean":"2024-09-13T14:39:24.973514+0000","last_became_active":"2024-09-13T14:38:53.899448+0000","last_became_peered":"2024-09-13T14:38:53.899448+0000","last_unstale":"2024-09-13T14:39:24.973514+0000","last_undegraded":"2024-09-13T14:39:24.973514+0000","last_fullsized":"2024-09-13T14:39:24.973514+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:55.897035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"260'8007","reported_seq":8090,"reported_epoch":260,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:39:27.713453+0000","last_change":"2024-09-13T14:38:03.567892+0000","last_active":"2024-09-13T14:39:27.713453+0000","last_peered":"2024-09-13T14:39:27.713453+0000","last_clean":"2024-09-13T14:35:51.090791+0000","last_became_active":"2024-09-13T14:37:49.665025+0000","last_became_peered":"2024-09-13T14:37:49.665025+0000","last_unstale":"2024-09-13T14:39:27.713453+0000","last_undegraded":"2024-09-13T14:37:49.169101+0000","last_fullsized":"2024-09-13T14:37:49.121596+0000","mapping_epoch":240,"log_start":"210'2063","ondisk_log_start":"210'2063","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5944,"log_dups_size":0,"ondisk_log_size":5944,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:02.321504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":412549120,"num_objects":6295,"num_object_clones":0,"num_object_copies":18885,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4501,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8007,"num_write_kb":457664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":1860}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":68,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.408581+0000","last_change":"2024-09-13T14:37:49.149770+0000","last_active":"2024-09-13T14:39:15.408581+0000","last_peered":"2024-09-13T14:39:15.408581+0000","last_clean":"2024-09-13T14:39:15.408581+0000","last_became_active":"2024-09-13T14:37:49.148991+0000","last_became_peered":"2024-09-13T14:37:49.148991+0000","last_unstale":"2024-09-13T14:39:15.408581+0000","last_undegraded":"2024-09-13T14:39:15.408581+0000","last_fullsized":"2024-09-13T14:39:15.408581+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:04:04.193366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":1101,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.977985+0000","last_change":"2024-09-13T14:30:58.928302+0000","last_active":"2024-09-13T14:39:24.977985+0000","last_peered":"2024-09-13T14:39:24.977985+0000","last_clean":"2024-09-13T14:39:24.977985+0000","last_became_active":"2024-09-13T14:30:58.927718+0000","last_became_peered":"2024-09-13T14:30:58.927718+0000","last_unstale":"2024-09-13T14:39:24.977985+0000","last_undegraded":"2024-09-13T14:39:24.977985+0000","last_fullsized":"2024-09-13T14:39:24.977985+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:37.323036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":75,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.408126+0000","last_change":"2024-09-13T14:38:03.534628+0000","last_active":"2024-09-13T14:39:15.408126+0000","last_peered":"2024-09-13T14:39:15.408126+0000","last_clean":"2024-09-13T14:39:15.408126+0000","last_became_active":"2024-09-13T14:38:03.534262+0000","last_became_peered":"2024-09-13T14:38:03.534262+0000","last_unstale":"2024-09-13T14:39:15.408126+0000","last_undegraded":"2024-09-13T14:39:15.408126+0000","last_fullsized":"2024-09-13T14:39:15.408126+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:31.168783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"260'7833","reported_seq":21978,"reported_epoch":260,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:39:25.929242+0000","last_change":"2024-09-13T14:38:53.944111+0000","last_active":"2024-09-13T14:39:25.929242+0000","last_peered":"2024-09-13T14:39:25.929242+0000","last_clean":"2024-09-13T14:37:47.047672+0000","last_became_active":"2024-09-13T14:37:49.664294+0000","last_became_peered":"2024-09-13T14:37:49.664294+0000","last_unstale":"2024-09-13T14:39:25.929242+0000","last_undegraded":"2024-09-13T14:38:53.940822+0000","last_fullsized":"2024-09-13T14:37:49.126627+0000","mapping_epoch":240,"log_start":"210'2100","ondisk_log_start":"210'2100","created":205,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5733,"log_dups_size":0,"ondisk_log_size":5733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:11.766692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":407568384,"num_objects":6219,"num_object_clones":0,"num_object_copies":18657,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4535,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7833,"num_write_kb":449664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7009,"num_bytes_recovered":459341824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3],"avail_no_missing":["5","0","3"],"object_location_counts":[{"shards":"0,3,5","objects":4460},{"shards":"3,5","objects":1759}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":87,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.408587+0000","last_change":"2024-09-13T14:38:03.534303+0000","last_active":"2024-09-13T14:39:15.408587+0000","last_peered":"2024-09-13T14:39:15.408587+0000","last_clean":"2024-09-13T14:39:15.408587+0000","last_became_active":"2024-09-13T14:38:03.533171+0000","last_became_peered":"2024-09-13T14:38:03.533171+0000","last_unstale":"2024-09-13T14:39:15.408587+0000","last_undegraded":"2024-09-13T14:39:15.408587+0000","last_fullsized":"2024-09-13T14:39:15.408587+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:00:16.553526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":779,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.407539+0000","last_change":"2024-09-13T14:38:04.596192+0000","last_active":"2024-09-13T14:39:15.407539+0000","last_peered":"2024-09-13T14:39:15.407539+0000","last_clean":"2024-09-13T14:39:15.407539+0000","last_became_active":"2024-09-13T14:38:04.593299+0000","last_became_peered":"2024-09-13T14:38:04.593299+0000","last_unstale":"2024-09-13T14:39:15.407539+0000","last_undegraded":"2024-09-13T14:39:15.407539+0000","last_fullsized":"2024-09-13T14:39:15.407539+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:34.241410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":42,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.977663+0000","last_change":"2024-09-13T14:38:03.533120+0000","last_active":"2024-09-13T14:39:24.977663+0000","last_peered":"2024-09-13T14:39:24.977663+0000","last_clean":"2024-09-13T14:39:24.977663+0000","last_became_active":"2024-09-13T14:38:03.532829+0000","last_became_peered":"2024-09-13T14:38:03.532829+0000","last_unstale":"2024-09-13T14:39:24.977663+0000","last_undegraded":"2024-09-13T14:39:24.977663+0000","last_fullsized":"2024-09-13T14:39:24.977663+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:24.514661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"260'8021","reported_seq":8150,"reported_epoch":260,"state":"active+undersized+degraded+remapped+backfill_wait","last_fresh":"2024-09-13T14:39:27.373126+0000","last_change":"2024-09-13T14:37:49.295861+0000","last_active":"2024-09-13T14:39:27.373126+0000","last_peered":"2024-09-13T14:39:27.373126+0000","last_clean":"2024-09-13T14:35:51.078708+0000","last_became_active":"2024-09-13T14:37:49.290403+0000","last_became_peered":"2024-09-13T14:37:49.290403+0000","last_unstale":"2024-09-13T14:39:27.373126+0000","last_undegraded":"2024-09-13T14:37:49.147402+0000","last_fullsized":"2024-09-13T14:37:49.117202+0000","mapping_epoch":240,"log_start":"212'3218","ondisk_log_start":"212'3218","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":4803,"log_dups_size":0,"ondisk_log_size":4803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":417923072,"num_objects":6377,"num_object_clones":0,"num_object_copies":19131,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6377,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8021,"num_write_kb":460736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":6377}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":444,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.974699+0000","last_change":"2024-09-13T14:38:53.938227+0000","last_active":"2024-09-13T14:39:24.974699+0000","last_peered":"2024-09-13T14:39:24.974699+0000","last_clean":"2024-09-13T14:39:24.974699+0000","last_became_active":"2024-09-13T14:38:53.902867+0000","last_became_peered":"2024-09-13T14:38:53.902867+0000","last_unstale":"2024-09-13T14:39:24.974699+0000","last_undegraded":"2024-09-13T14:39:24.974699+0000","last_fullsized":"2024-09-13T14:39:24.974699+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:16.831756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":558,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.404701+0000","last_change":"2024-09-13T14:37:48.147677+0000","last_active":"2024-09-13T14:39:15.404701+0000","last_peered":"2024-09-13T14:39:15.404701+0000","last_clean":"2024-09-13T14:39:15.404701+0000","last_became_active":"2024-09-13T14:37:48.147001+0000","last_became_peered":"2024-09-13T14:37:48.147001+0000","last_unstale":"2024-09-13T14:39:15.404701+0000","last_undegraded":"2024-09-13T14:39:15.404701+0000","last_fullsized":"2024-09-13T14:39:15.404701+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:35:00.653576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":426,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.973670+0000","last_change":"2024-09-13T14:38:53.900107+0000","last_active":"2024-09-13T14:39:24.973670+0000","last_peered":"2024-09-13T14:39:24.973670+0000","last_clean":"2024-09-13T14:39:24.973670+0000","last_became_active":"2024-09-13T14:38:53.899556+0000","last_became_peered":"2024-09-13T14:38:53.899556+0000","last_unstale":"2024-09-13T14:39:24.973670+0000","last_undegraded":"2024-09-13T14:39:24.973670+0000","last_fullsized":"2024-09-13T14:39:24.973670+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:15:47.918163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"260'7769","reported_seq":10828,"reported_epoch":260,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:39:27.261964+0000","last_change":"2024-09-13T14:38:53.903357+0000","last_active":"2024-09-13T14:39:27.261964+0000","last_peered":"2024-09-13T14:39:27.261964+0000","last_clean":"2024-09-13T14:35:51.088129+0000","last_became_active":"2024-09-13T14:37:49.691874+0000","last_became_peered":"2024-09-13T14:37:49.691874+0000","last_unstale":"2024-09-13T14:39:27.261964+0000","last_undegraded":"2024-09-13T14:37:49.158458+0000","last_fullsized":"2024-09-13T14:37:49.116993+0000","mapping_epoch":240,"log_start":"212'2847","ondisk_log_start":"212'2847","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":4922,"log_dups_size":0,"ondisk_log_size":4922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398655488,"num_objects":6083,"num_object_clones":0,"num_object_copies":18249,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2932,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7769,"num_write_kb":443264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1463,"num_bytes_recovered":95879168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":3216}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":57,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.977695+0000","last_change":"2024-09-13T14:38:53.901806+0000","last_active":"2024-09-13T14:39:24.977695+0000","last_peered":"2024-09-13T14:39:24.977695+0000","last_clean":"2024-09-13T14:39:24.977695+0000","last_became_active":"2024-09-13T14:38:53.891055+0000","last_became_peered":"2024-09-13T14:38:53.891055+0000","last_unstale":"2024-09-13T14:39:24.977695+0000","last_undegraded":"2024-09-13T14:39:24.977695+0000","last_fullsized":"2024-09-13T14:39:24.977695+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:03:00.416198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":15,"reported_epoch":259,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-13T14:39:15.406431+0000","last_change":"2024-09-13T14:38:53.900902+0000","last_active":"2024-09-13T14:39:15.406431+0000","last_peered":"2024-09-13T14:39:15.406431+0000","last_clean":"2024-09-13T14:38:39.635779+0000","last_became_active":"2024-09-13T14:38:53.900626+0000","last_became_peered":"2024-09-13T14:38:53.900626+0000","last_unstale":"2024-09-13T14:39:15.406431+0000","last_undegraded":"2024-09-13T14:38:53.885778+0000","last_fullsized":"2024-09-13T14:39:15.406431+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:37:21.715778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":7,"num_objects_missing":7,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":["3","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":248,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.404302+0000","last_change":"2024-09-13T14:37:49.172218+0000","last_active":"2024-09-13T14:39:15.404302+0000","last_peered":"2024-09-13T14:39:15.404302+0000","last_clean":"2024-09-13T14:39:15.404302+0000","last_became_active":"2024-09-13T14:37:49.171718+0000","last_became_peered":"2024-09-13T14:37:49.171718+0000","last_unstale":"2024-09-13T14:39:15.404302+0000","last_undegraded":"2024-09-13T14:39:15.404302+0000","last_fullsized":"2024-09-13T14:39:15.404302+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:46:54.626521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":270,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.404225+0000","last_change":"2024-09-13T14:37:49.230094+0000","last_active":"2024-09-13T14:39:15.404225+0000","last_peered":"2024-09-13T14:39:15.404225+0000","last_clean":"2024-09-13T14:39:15.404225+0000","last_became_active":"2024-09-13T14:37:49.229393+0000","last_became_peered":"2024-09-13T14:37:49.229393+0000","last_unstale":"2024-09-13T14:39:15.404225+0000","last_undegraded":"2024-09-13T14:39:15.404225+0000","last_fullsized":"2024-09-13T14:39:15.404225+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:02:28.690318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":250,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.404474+0000","last_change":"2024-09-13T14:37:48.128447+0000","last_active":"2024-09-13T14:39:15.404474+0000","last_peered":"2024-09-13T14:39:15.404474+0000","last_clean":"2024-09-13T14:39:15.404474+0000","last_became_active":"2024-09-13T14:37:48.128143+0000","last_became_peered":"2024-09-13T14:37:48.128143+0000","last_unstale":"2024-09-13T14:39:15.404474+0000","last_undegraded":"2024-09-13T14:39:15.404474+0000","last_fullsized":"2024-09-13T14:39:15.404474+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:22.204039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":42,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.977340+0000","last_change":"2024-09-13T14:38:03.531683+0000","last_active":"2024-09-13T14:39:24.977340+0000","last_peered":"2024-09-13T14:39:24.977340+0000","last_clean":"2024-09-13T14:39:24.977340+0000","last_became_active":"2024-09-13T14:38:03.531476+0000","last_became_peered":"2024-09-13T14:38:03.531476+0000","last_unstale":"2024-09-13T14:39:24.977340+0000","last_undegraded":"2024-09-13T14:39:24.977340+0000","last_fullsized":"2024-09-13T14:39:24.977340+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:41.557569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":23,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.406982+0000","last_change":"2024-09-13T14:38:53.954281+0000","last_active":"2024-09-13T14:39:15.406982+0000","last_peered":"2024-09-13T14:39:15.406982+0000","last_clean":"2024-09-13T14:39:15.406982+0000","last_became_active":"2024-09-13T14:38:53.900912+0000","last_became_peered":"2024-09-13T14:38:53.900912+0000","last_unstale":"2024-09-13T14:39:15.406982+0000","last_undegraded":"2024-09-13T14:39:15.406982+0000","last_fullsized":"2024-09-13T14:39:15.406982+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:47.876275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":75,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.417828+0000","last_change":"2024-09-13T14:38:54.000782+0000","last_active":"2024-09-13T14:39:15.417828+0000","last_peered":"2024-09-13T14:39:15.417828+0000","last_clean":"2024-09-13T14:39:15.417828+0000","last_became_active":"2024-09-13T14:38:54.000322+0000","last_became_peered":"2024-09-13T14:38:54.000322+0000","last_unstale":"2024-09-13T14:39:15.417828+0000","last_undegraded":"2024-09-13T14:39:15.417828+0000","last_fullsized":"2024-09-13T14:39:15.417828+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:09.057597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":368,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.973516+0000","last_change":"2024-09-13T14:38:53.899535+0000","last_active":"2024-09-13T14:39:24.973516+0000","last_peered":"2024-09-13T14:39:24.973516+0000","last_clean":"2024-09-13T14:39:24.973516+0000","last_became_active":"2024-09-13T14:38:53.899183+0000","last_became_peered":"2024-09-13T14:38:53.899183+0000","last_unstale":"2024-09-13T14:39:24.973516+0000","last_undegraded":"2024-09-13T14:39:24.973516+0000","last_fullsized":"2024-09-13T14:39:24.973516+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:33.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":42,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.977804+0000","last_change":"2024-09-13T14:38:03.543426+0000","last_active":"2024-09-13T14:39:24.977804+0000","last_peered":"2024-09-13T14:39:24.977804+0000","last_clean":"2024-09-13T14:39:24.977804+0000","last_became_active":"2024-09-13T14:38:03.542661+0000","last_became_peered":"2024-09-13T14:38:03.542661+0000","last_unstale":"2024-09-13T14:39:24.977804+0000","last_undegraded":"2024-09-13T14:39:24.977804+0000","last_fullsized":"2024-09-13T14:39:24.977804+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:54.050127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":397,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.973587+0000","last_change":"2024-09-13T14:37:48.131321+0000","last_active":"2024-09-13T14:39:24.973587+0000","last_peered":"2024-09-13T14:39:24.973587+0000","last_clean":"2024-09-13T14:39:24.973587+0000","last_became_active":"2024-09-13T14:37:48.130998+0000","last_became_peered":"2024-09-13T14:37:48.130998+0000","last_unstale":"2024-09-13T14:39:24.973587+0000","last_undegraded":"2024-09-13T14:39:24.973587+0000","last_fullsized":"2024-09-13T14:39:24.973587+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:33:26.597825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":87,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.409107+0000","last_change":"2024-09-13T14:38:03.535454+0000","last_active":"2024-09-13T14:39:15.409107+0000","last_peered":"2024-09-13T14:39:15.409107+0000","last_clean":"2024-09-13T14:39:15.409107+0000","last_became_active":"2024-09-13T14:38:03.535131+0000","last_became_peered":"2024-09-13T14:38:03.535131+0000","last_unstale":"2024-09-13T14:39:15.409107+0000","last_undegraded":"2024-09-13T14:39:15.409107+0000","last_fullsized":"2024-09-13T14:39:15.409107+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:24.477601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":422,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.973884+0000","last_change":"2024-09-13T14:38:03.524939+0000","last_active":"2024-09-13T14:39:24.973884+0000","last_peered":"2024-09-13T14:39:24.973884+0000","last_clean":"2024-09-13T14:39:24.973884+0000","last_became_active":"2024-09-13T14:38:03.524406+0000","last_became_peered":"2024-09-13T14:38:03.524406+0000","last_unstale":"2024-09-13T14:39:24.973884+0000","last_undegraded":"2024-09-13T14:39:24.973884+0000","last_fullsized":"2024-09-13T14:39:24.973884+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:19:45.257424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":49,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.977611+0000","last_change":"2024-09-13T14:38:53.892948+0000","last_active":"2024-09-13T14:39:24.977611+0000","last_peered":"2024-09-13T14:39:24.977611+0000","last_clean":"2024-09-13T14:39:24.977611+0000","last_became_active":"2024-09-13T14:38:53.892665+0000","last_became_peered":"2024-09-13T14:38:53.892665+0000","last_unstale":"2024-09-13T14:39:24.977611+0000","last_undegraded":"2024-09-13T14:39:24.977611+0000","last_fullsized":"2024-09-13T14:39:24.977611+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:08.292359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":370,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.973780+0000","last_change":"2024-09-13T14:37:48.139348+0000","last_active":"2024-09-13T14:39:24.973780+0000","last_peered":"2024-09-13T14:39:24.973780+0000","last_clean":"2024-09-13T14:39:24.973780+0000","last_became_active":"2024-09-13T14:37:48.139115+0000","last_became_peered":"2024-09-13T14:37:48.139115+0000","last_unstale":"2024-09-13T14:39:24.973780+0000","last_undegraded":"2024-09-13T14:39:24.973780+0000","last_fullsized":"2024-09-13T14:39:24.973780+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:32:20.413346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":410,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.974266+0000","last_change":"2024-09-13T14:37:48.172088+0000","last_active":"2024-09-13T14:39:24.974266+0000","last_peered":"2024-09-13T14:39:24.974266+0000","last_clean":"2024-09-13T14:39:24.974266+0000","last_became_active":"2024-09-13T14:37:48.171779+0000","last_became_peered":"2024-09-13T14:37:48.171779+0000","last_unstale":"2024-09-13T14:39:24.974266+0000","last_undegraded":"2024-09-13T14:39:24.974266+0000","last_fullsized":"2024-09-13T14:39:24.974266+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:02.852712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":315,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.404066+0000","last_change":"2024-09-13T14:38:53.890236+0000","last_active":"2024-09-13T14:39:15.404066+0000","last_peered":"2024-09-13T14:39:15.404066+0000","last_clean":"2024-09-13T14:39:15.404066+0000","last_became_active":"2024-09-13T14:38:53.889582+0000","last_became_peered":"2024-09-13T14:38:53.889582+0000","last_unstale":"2024-09-13T14:39:15.404066+0000","last_undegraded":"2024-09-13T14:39:15.404066+0000","last_fullsized":"2024-09-13T14:39:15.404066+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:14:29.334774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":396,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.974669+0000","last_change":"2024-09-13T14:38:03.534097+0000","last_active":"2024-09-13T14:39:24.974669+0000","last_peered":"2024-09-13T14:39:24.974669+0000","last_clean":"2024-09-13T14:39:24.974669+0000","last_became_active":"2024-09-13T14:38:03.533794+0000","last_became_peered":"2024-09-13T14:38:03.533794+0000","last_unstale":"2024-09-13T14:39:24.974669+0000","last_undegraded":"2024-09-13T14:39:24.974669+0000","last_fullsized":"2024-09-13T14:39:24.974669+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:23.272098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":42,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.977494+0000","last_change":"2024-09-13T14:38:03.527021+0000","last_active":"2024-09-13T14:39:24.977494+0000","last_peered":"2024-09-13T14:39:24.977494+0000","last_clean":"2024-09-13T14:39:24.977494+0000","last_became_active":"2024-09-13T14:38:03.526698+0000","last_became_peered":"2024-09-13T14:38:03.526698+0000","last_unstale":"2024-09-13T14:39:24.977494+0000","last_undegraded":"2024-09-13T14:39:24.977494+0000","last_fullsized":"2024-09-13T14:39:24.977494+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:01:35.734086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":259,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.404154+0000","last_change":"2024-09-13T14:27:12.834931+0000","last_active":"2024-09-13T14:39:15.404154+0000","last_peered":"2024-09-13T14:39:15.404154+0000","last_clean":"2024-09-13T14:39:15.404154+0000","last_became_active":"2024-09-13T14:27:12.834572+0000","last_became_peered":"2024-09-13T14:27:12.834572+0000","last_unstale":"2024-09-13T14:39:15.404154+0000","last_undegraded":"2024-09-13T14:39:15.404154+0000","last_fullsized":"2024-09-13T14:39:15.404154+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:54:21.155689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":72,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.408528+0000","last_change":"2024-09-13T14:37:49.176796+0000","last_active":"2024-09-13T14:39:15.408528+0000","last_peered":"2024-09-13T14:39:15.408528+0000","last_clean":"2024-09-13T14:39:15.408528+0000","last_became_active":"2024-09-13T14:37:49.176246+0000","last_became_peered":"2024-09-13T14:37:49.176246+0000","last_unstale":"2024-09-13T14:39:15.408528+0000","last_undegraded":"2024-09-13T14:39:15.408528+0000","last_fullsized":"2024-09-13T14:39:15.408528+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:29:24.936718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":270,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.403933+0000","last_change":"2024-09-13T14:38:03.526333+0000","last_active":"2024-09-13T14:39:15.403933+0000","last_peered":"2024-09-13T14:39:15.403933+0000","last_clean":"2024-09-13T14:39:15.403933+0000","last_became_active":"2024-09-13T14:38:03.526001+0000","last_became_peered":"2024-09-13T14:38:03.526001+0000","last_unstale":"2024-09-13T14:39:15.403933+0000","last_undegraded":"2024-09-13T14:39:15.403933+0000","last_fullsized":"2024-09-13T14:39:15.403933+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:25:25.181271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":72,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.408662+0000","last_change":"2024-09-13T14:37:49.149582+0000","last_active":"2024-09-13T14:39:15.408662+0000","last_peered":"2024-09-13T14:39:15.408662+0000","last_clean":"2024-09-13T14:39:15.408662+0000","last_became_active":"2024-09-13T14:37:49.148525+0000","last_became_peered":"2024-09-13T14:37:49.148525+0000","last_unstale":"2024-09-13T14:39:15.408662+0000","last_undegraded":"2024-09-13T14:39:15.408662+0000","last_fullsized":"2024-09-13T14:39:15.408662+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:48.655665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":443,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.973998+0000","last_change":"2024-09-13T14:38:03.535255+0000","last_active":"2024-09-13T14:39:24.973998+0000","last_peered":"2024-09-13T14:39:24.973998+0000","last_clean":"2024-09-13T14:39:24.973998+0000","last_became_active":"2024-09-13T14:38:03.533917+0000","last_became_peered":"2024-09-13T14:38:03.533917+0000","last_unstale":"2024-09-13T14:39:24.973998+0000","last_undegraded":"2024-09-13T14:39:24.973998+0000","last_fullsized":"2024-09-13T14:39:24.973998+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:33.559788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":406,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.975599+0000","last_change":"2024-09-13T14:37:48.146784+0000","last_active":"2024-09-13T14:39:24.975599+0000","last_peered":"2024-09-13T14:39:24.975599+0000","last_clean":"2024-09-13T14:39:24.975599+0000","last_became_active":"2024-09-13T14:37:48.146491+0000","last_became_peered":"2024-09-13T14:37:48.146491+0000","last_unstale":"2024-09-13T14:39:24.975599+0000","last_undegraded":"2024-09-13T14:39:24.975599+0000","last_fullsized":"2024-09-13T14:39:24.975599+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:30:54.772513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":250,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-13T14:39:15.403904+0000","last_change":"2024-09-13T14:37:48.127607+0000","last_active":"2024-09-13T14:39:15.403904+0000","last_peered":"2024-09-13T14:39:15.403904+0000","last_clean":"2024-09-13T14:39:15.403904+0000","last_became_active":"2024-09-13T14:37:48.127316+0000","last_became_peered":"2024-09-13T14:37:48.127316+0000","last_unstale":"2024-09-13T14:39:15.403904+0000","last_undegraded":"2024-09-13T14:39:15.403904+0000","last_fullsized":"2024-09-13T14:39:15.403904+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:53:37.678861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":48,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.977649+0000","last_change":"2024-09-13T14:38:03.563200+0000","last_active":"2024-09-13T14:39:24.977649+0000","last_peered":"2024-09-13T14:39:24.977649+0000","last_clean":"2024-09-13T14:39:24.977649+0000","last_became_active":"2024-09-13T14:38:03.534258+0000","last_became_peered":"2024-09-13T14:38:03.534258+0000","last_unstale":"2024-09-13T14:39:24.977649+0000","last_undegraded":"2024-09-13T14:39:24.977649+0000","last_fullsized":"2024-09-13T14:39:24.977649+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:49:07.814419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":6526009368,"num_objects":99580,"num_object_clones":0,"num_object_copies":298740,"num_objects_missing_on_primary":4209,"num_objects_missing":4209,"num_objects_degraded":46147,"num_objects_misplaced":1995,"num_objects_unfound":0,"num_objects_dirty":99580,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":126046,"num_write_kb":7219969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47871,"num_bytes_recovered":3115253760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19912794112,"data_stored":19912785968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88269,"ondisk_log_size":88269,"up":48,"acting":41,"num_store_stats":6},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1538,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":17412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25132,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":7,"num_objects_missing":7,"num_objects_degraded":16,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16793600,"data_stored":16777292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":23,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234347,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3582272,"kb_used_data":3123768,"kb_used_omap":1,"kb_used_meta":458494,"kb_avail":90162880,"statfs":{"total":95995035648,"available":92326789120,"internally_reserved":0,"allocated":3198738432,"data_stored":3197327799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":469498315},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":239,"seq":1026497183765,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1957384,"kb_used_data":1814652,"kb_used_omap":8,"kb_used_meta":142711,"kb_avail":91787768,"statfs":{"total":95995035648,"available":93990674432,"internally_reserved":0,"allocated":1858203648,"data_stored":1856790010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8453,"internal_metadata":146136827},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757612,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2519096,"kb_used_data":1928492,"kb_used_omap":1,"kb_used_meta":590590,"kb_avail":91226056,"statfs":{"total":95995035648,"available":93415481344,"internally_reserved":0,"allocated":1974775808,"data_stored":1973373184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":604764614},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1,0,2,1,6,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019256,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3883072,"kb_used_data":3314616,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":89862080,"statfs":{"total":95995035648,"available":92018769920,"internally_reserved":0,"allocated":3394166784,"data_stored":3392756683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":582089158},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435439,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5329592,"kb_used_data":4792560,"kb_used_omap":5,"kb_used_meta":537018,"kb_avail":88415560,"statfs":{"total":95995035648,"available":90537533440,"internally_reserved":0,"allocated":4907581440,"data_stored":4906175050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":549906558},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":446676598967,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":425201762494,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926018,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5146428,"kb_used_data":4519156,"kb_used_omap":7,"kb_used_meta":627256,"kb_avail":88598724,"statfs":{"total":95995035648,"available":90725093376,"internally_reserved":0,"allocated":4627615744,"data_stored":4626205315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":642310440},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6860,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4617670656,"data_stored":4617666584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4897640448,"data_stored":4897636376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3387359232,"data_stored":3387359232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1965621248,"data_stored":1965621248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1850474496,"data_stored":1850474496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3194028032,"data_stored":3194028032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:39:33.666 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph status --format=json 2024-09-13T14:39:33.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:33.694 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:33.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:33.795 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:33.819 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:33.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:33.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:33.921 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:33.947 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:33.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:33.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:34.053 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:34.077 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:34.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:34.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:34.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:33 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 46120/298392 objects degraded (15.456%), 14 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:34.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:33 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 46120/298392 objects degraded (15.456%), 14 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:34.178 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:34.201 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:34.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:34.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:34.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:33 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 46120/298392 objects degraded (15.456%), 14 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:34.304 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:34.326 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:34.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:34.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:34.404 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:39:34.404 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:39:34.423 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:39:34.423 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:39:34.429 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:34.441 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:39:34.445 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:39:34.452 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:34.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:34.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:34.459 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:39:34.459 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:39:34.478 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:39:34.478 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:39:34.499 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:39:34.499 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:39:34.521 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:39:34.521 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:39:34.541 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:39:34.541 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:39:34.554 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:34.575 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:34.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:34.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:34.677 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:34.702 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:34.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:34.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:34.804 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:34.828 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:34.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:34.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:34.930 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:34.952 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:34.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:34.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:35.057 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:35.084 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:35.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:35.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:35.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:34 smithi102 ceph-mon[110183]: pgmap v837: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.0 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 149 op/s; 45115/296301 objects degraded (15.226%); 1997/296301 objects misplaced (0.674%); 11 MiB/s, 178 objects/s recovering 2024-09-13T14:39:35.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:34 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2211675195' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:39:35.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:34 smithi102 ceph-mon[115269]: pgmap v837: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.0 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 149 op/s; 45115/296301 objects degraded (15.226%); 1997/296301 objects misplaced (0.674%); 11 MiB/s, 178 objects/s recovering 2024-09-13T14:39:35.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:34 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2211675195' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:39:35.186 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:35.211 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:35.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:35.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:35.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:34 smithi195 ceph-mon[104262]: pgmap v837: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.0 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 149 op/s; 45115/296301 objects degraded (15.226%); 1997/296301 objects misplaced (0.674%); 11 MiB/s, 178 objects/s recovering 2024-09-13T14:39:35.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:34 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2211675195' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:39:35.313 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:35.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:35.337 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:35.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:35.438 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:35.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:35.466 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:35.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:35.567 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:35.591 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:35.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:35.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:35.693 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:35.718 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:35.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:35.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:35.820 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:35.845 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:35.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:35.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:35.947 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:35.969 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:35.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:35.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:36.072 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:36.099 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:36.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:36.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:36.202 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:36.224 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:36.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:36.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:36.326 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:36.349 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:36.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:36.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:36.450 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:36.474 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:36.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:36.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:36.576 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:36.602 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:36.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:36.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:36.705 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:36.720 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:39:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:39:36] "GET /metrics HTTP/1.1" 200 36544 "" "Prometheus/2.51.0" 2024-09-13T14:39:36.730 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:36.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:36.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:36.832 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:36.856 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:36.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:36.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:36.958 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:36.983 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:36.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:36.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:37.085 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:37.118 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:37.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:37.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:37.220 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:37.247 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:37.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:36 smithi195 ceph-mon[104262]: pgmap v838: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.0 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 142 op/s; 45115/296301 objects degraded (15.226%); 1997/296301 objects misplaced (0.674%); 11 MiB/s, 169 objects/s recovering 2024-09-13T14:39:37.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:37.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:37.293 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:36 smithi102 ceph-mon[110183]: pgmap v838: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.0 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 142 op/s; 45115/296301 objects degraded (15.226%); 1997/296301 objects misplaced (0.674%); 11 MiB/s, 169 objects/s recovering 2024-09-13T14:39:37.294 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:36 smithi102 ceph-mon[115269]: pgmap v838: 89 pgs: 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 75 active+clean; 6.0 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 142 op/s; 45115/296301 objects degraded (15.226%); 1997/296301 objects misplaced (0.674%); 11 MiB/s, 169 objects/s recovering 2024-09-13T14:39:37.349 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:37.373 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:37.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:37.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:37.409 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:39:37.432 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:39:37.432 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:39:37.475 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:37.502 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:37.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:37.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:37.604 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:37.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:37.639 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:37.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:37.740 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:37.775 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:37.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:37.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:37.877 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:37.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:37.900 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:37.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:37.985 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is unset 2024-09-13T14:39:38.002 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:38.028 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:38.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:38.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:38.109 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:39:38.131 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:38.157 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:38.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:38.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:38.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:37 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/185964693' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:39:38.260 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:38.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:38.288 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:38.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:38.390 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:38.418 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:38.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:37 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/185964693' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:39:38.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:37 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/185964693' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:39:38.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:38.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:38.520 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:38.544 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:38.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:38.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:38.646 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:38.669 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:38.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:38.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:38.771 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:38.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:38.795 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:38.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:38.896 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:38.920 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:38.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:38.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:39.022 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:39.047 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:39.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:39.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:39.149 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:39.177 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:39.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:39.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:39.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:38 smithi195 ceph-mon[104262]: pgmap v839: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 76 active+clean; 6.0 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 160 op/s; 44099/294987 objects degraded (14.949%); 1989/294987 objects misplaced (0.674%); 14 MiB/s, 230 objects/s recovering 2024-09-13T14:39:39.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:38 smithi195 ceph-mon[104262]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:39:39.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:38 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/185964693' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:39:39.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:38 smithi195 ceph-mon[104262]: osdmap e261: 8 total, 8 up, 6 in 2024-09-13T14:39:39.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:38 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 44099/294987 objects degraded (14.949%), 13 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:39.279 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:39.305 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:39.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:39.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:39.407 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:39.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:38 smithi102 ceph-mon[110183]: pgmap v839: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 76 active+clean; 6.0 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 160 op/s; 44099/294987 objects degraded (14.949%); 1989/294987 objects misplaced (0.674%); 14 MiB/s, 230 objects/s recovering 2024-09-13T14:39:39.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:38 smithi102 ceph-mon[110183]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:39:39.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:38 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/185964693' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:39:39.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:38 smithi102 ceph-mon[110183]: osdmap e261: 8 total, 8 up, 6 in 2024-09-13T14:39:39.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:38 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 44099/294987 objects degraded (14.949%), 13 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:39.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:38 smithi102 ceph-mon[115269]: pgmap v839: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 76 active+clean; 6.0 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 160 op/s; 44099/294987 objects degraded (14.949%); 1989/294987 objects misplaced (0.674%); 14 MiB/s, 230 objects/s recovering 2024-09-13T14:39:39.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:38 smithi102 ceph-mon[115269]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:39:39.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:38 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/185964693' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:39:39.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:38 smithi102 ceph-mon[115269]: osdmap e261: 8 total, 8 up, 6 in 2024-09-13T14:39:39.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:38 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 44099/294987 objects degraded (14.949%), 13 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:39.431 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:39.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:39.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:39.534 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:39.561 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:39.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:39.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:39.663 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:39.688 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:39.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:39.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:39.790 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:39.812 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:39.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:39.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:39.914 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:39.938 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:39.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:39.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:40.040 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:40.043 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T14:39:40.111 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:40.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:40.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:40.213 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:40.238 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:40.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:40.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:40.340 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:40.362 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:40.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:40.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:40.464 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:40.486 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:40.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:40.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:40.588 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:40.612 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:40.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:40.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:40.714 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:40.738 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:40.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:40.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:40.834 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:39:40.840 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:40.846 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:39:40.867 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:40.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:40.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:40.969 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:40.991 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:40.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:40.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:41.070 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:40 smithi102 ceph-mon[110183]: pgmap v841: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 76 active+clean; 6.0 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 171 op/s; 44039/293682 objects degraded (14.995%); 1990/293682 objects misplaced (0.678%); 12 MiB/s, 204 objects/s recovering 2024-09-13T14:39:41.070 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:40 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:39:41.070 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:40 smithi102 ceph-mon[115269]: pgmap v841: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 76 active+clean; 6.0 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 171 op/s; 44039/293682 objects degraded (14.995%); 1990/293682 objects misplaced (0.678%); 12 MiB/s, 204 objects/s recovering 2024-09-13T14:39:41.070 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:40 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:39:41.093 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:41.116 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:41.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:41.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:41.218 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:41.243 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:41.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:41.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:41.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:40 smithi195 ceph-mon[104262]: pgmap v841: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 76 active+clean; 6.0 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 171 op/s; 44039/293682 objects degraded (14.995%); 1990/293682 objects misplaced (0.678%); 12 MiB/s, 204 objects/s recovering 2024-09-13T14:39:41.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:40 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:39:41.345 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:41.370 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:41.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:41.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:41.471 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:41.498 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:41.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:41.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:41.600 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:41.626 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:41.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:41.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:41.728 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:41.750 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:41.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:41.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:41.851 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:41.875 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:41.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:41.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:41.977 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:42.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:42.004 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:42.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:42.105 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:42.129 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:42.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:42.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:42.231 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:42.256 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:42.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:42.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:42.358 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:42.385 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:42.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:42.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:42.487 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:42.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:42.511 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:42.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:42.612 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:42.638 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:42.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:42.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:42.740 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:42.764 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:42.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:42.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:42.803 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:39:42.866 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:42.895 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:42.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:42.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:42.997 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:43.025 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:43.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:43.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:43.091 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:42 smithi102 ceph-mon[110183]: pgmap v842: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 76 active+clean; 6.0 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 175 op/s; 43985/293136 objects degraded (15.005%); 1990/293136 objects misplaced (0.679%); 12 MiB/s, 204 objects/s recovering 2024-09-13T14:39:43.091 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:42 smithi102 ceph-mon[115269]: pgmap v842: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 76 active+clean; 6.0 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 175 op/s; 43985/293136 objects degraded (15.005%); 1990/293136 objects misplaced (0.679%); 12 MiB/s, 204 objects/s recovering 2024-09-13T14:39:43.127 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:43.152 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:43.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:43.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:43.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:42 smithi195 ceph-mon[104262]: pgmap v842: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 76 active+clean; 6.0 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 175 op/s; 43985/293136 objects degraded (15.005%); 1990/293136 objects misplaced (0.679%); 12 MiB/s, 204 objects/s recovering 2024-09-13T14:39:43.254 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:43.277 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:43.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:43.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:43.378 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:43.400 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:43.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:43.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:43.502 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:43.525 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:43.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:43.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:43.627 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:43.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:43.649 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:43.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:43.750 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:43.775 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:43.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:43.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:43.877 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:43.899 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:43.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:43.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:44.001 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:44.025 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:44.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:44.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:44.127 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:44.152 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:44.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:44.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:44.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:43 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 43985/293136 objects degraded (15.005%), 13 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:44.254 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:44.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:44.278 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:44.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:44.349 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:43 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 43985/293136 objects degraded (15.005%), 13 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:44.349 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:43 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 43985/293136 objects degraded (15.005%), 13 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:44.380 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:44.402 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:44.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:44.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:44.466 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:39:44.466 INFO:teuthology.orchestra.run.smithi102.stdout:{"fsid":"08cee502-71d9-11ef-bceb-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 43295/291045 objects degraded (14.876%), 13 pgs degraded, 7 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1207,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":261,"num_osds":8,"num_up_osds":8,"osd_up_since":1726238267,"num_in_osds":6,"osd_in_since":1726238173,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":76},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+recovery_wait+degraded","count":3},{"state_name":"active+recovery_wait+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_wait","count":1}],"num_pgs":89,"num_pools":5,"num_objects":97015,"data_bytes":6360864765,"bytes_used":22685298688,"bytes_avail":553284915200,"bytes_total":575970213888,"degraded_objects":43295,"degraded_total":291045,"degraded_ratio":0.1487570650586679,"misplaced_objects":1989,"misplaced_total":291045,"misplaced_ratio":0.0068339947430809669,"recovering_objects_per_sec":173,"recovering_bytes_per_sec":10823720,"recovering_keys_per_sec":0,"num_objects_recovered":1731,"num_bytes_recovered":108265472,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":175},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi102.njetxr","status":"up:active","gid":24521}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.102:8443/","prometheus":"http://172.21.15.102:9283/"}},"servicemap":{"epoch":27,"modified":"2024-09-13T14:38:47.449556+0000","services":{}},"progress_events":{}} 2024-09-13T14:39:44.505 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:44.529 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:44.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:44.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:44.631 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:44.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:44.657 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:44.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:44.758 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:44.788 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:44.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:44.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:44.891 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:44.918 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:44.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:44.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:45.020 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:45.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:45.044 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:45.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:45.138 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:44 smithi102 ceph-mon[110183]: pgmap v843: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 76 active+clean; 5.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 175 op/s; 43295/291045 objects degraded (14.876%); 1989/291045 objects misplaced (0.683%); 10 MiB/s, 173 objects/s recovering 2024-09-13T14:39:45.139 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:44 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3714692604' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:39:45.139 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:44 smithi102 ceph-mon[115269]: pgmap v843: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 76 active+clean; 5.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 175 op/s; 43295/291045 objects degraded (14.876%); 1989/291045 objects misplaced (0.683%); 10 MiB/s, 173 objects/s recovering 2024-09-13T14:39:45.139 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:44 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3714692604' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:39:45.145 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:45.169 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:45.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:45.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:45.178 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-13T14:39:45.178 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:39:45.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:44 smithi195 ceph-mon[104262]: pgmap v843: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 76 active+clean; 5.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 175 op/s; 43295/291045 objects degraded (14.876%); 1989/291045 objects misplaced (0.683%); 10 MiB/s, 173 objects/s recovering 2024-09-13T14:39:45.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:44 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3714692604' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:39:45.271 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:45.293 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:45.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:45.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:45.395 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:45.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:45.418 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:45.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:45.520 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:45.543 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:45.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:45.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:45.624 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:39:45.646 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:45.668 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:45.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:45.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:45.770 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:45.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:45.793 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:45.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:45.894 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:45.919 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:45.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:45.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:46.021 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:46.043 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:46.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:46.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:46.145 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:46.168 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:46.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:46.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:46.269 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:46.293 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:46.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:46.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:46.344 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:39:46.344 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:39:46.396 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:46.418 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:46.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:46.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:46.520 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:46.543 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:46.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:46.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:46.645 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:46.668 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:46.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:46.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:46.769 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:39:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:39:46] "GET /metrics HTTP/1.1" 200 36543 "" "Prometheus/2.51.0" 2024-09-13T14:39:46.770 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:46.775 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:39:46.791 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:46.792 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:39:46.793 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:39:46.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:46.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:46.814 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:39:46.815 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:39:46.835 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:39:46.835 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:39:46.854 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:39:46.855 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:39:46.870 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:39:46.870 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:39:46.890 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:39:46.890 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:39:46.895 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:46.917 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:39:46.917 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:39:46.919 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:46.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:46.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:46.939 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:39:46.939 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:39:47.021 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:47.031 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:46 smithi102 ceph-mon[110183]: pgmap v844: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 76 active+clean; 5.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 175 op/s; 43295/291045 objects degraded (14.876%); 1989/291045 objects misplaced (0.683%); 10 MiB/s, 173 objects/s recovering 2024-09-13T14:39:47.031 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:46 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/769032802' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:39:47.031 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:46 smithi102 ceph-mon[115269]: pgmap v844: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 76 active+clean; 5.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 175 op/s; 43295/291045 objects degraded (14.876%); 1989/291045 objects misplaced (0.683%); 10 MiB/s, 173 objects/s recovering 2024-09-13T14:39:47.032 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:46 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/769032802' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:39:47.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:47.044 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:47.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:47.145 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:47.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:47.168 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:47.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:47.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:46 smithi195 ceph-mon[104262]: pgmap v844: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 76 active+clean; 5.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 175 op/s; 43295/291045 objects degraded (14.876%); 1989/291045 objects misplaced (0.683%); 10 MiB/s, 173 objects/s recovering 2024-09-13T14:39:47.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:46 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/769032802' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:39:47.270 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:47.296 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:47.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:47.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:47.397 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:47.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:47.426 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:47.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:47.527 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:47.552 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:47.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:47.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:47.654 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:47.678 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:47.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:47.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:47.780 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:47.803 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:47.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:47.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:47.905 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:47.930 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:47.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:47.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:48.032 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:48.053 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:48.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:48.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:48.155 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:48.179 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:48.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:48.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:48.281 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:48.305 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:48.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:48.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:48.407 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:48.429 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:48.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:48.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:48.531 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:48.555 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:48.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:48.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:48.656 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:48.680 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:48.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:48.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:48.782 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:48.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:48.808 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:48.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:48.909 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:48.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:48.933 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:48.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:49.034 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:49.059 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:49.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:49.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:49.161 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:49.185 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:49.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:49.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:49.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:48 smithi195 ceph-mon[104262]: pgmap v845: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 76 active+clean; 5.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 158 op/s; 43242/290235 objects degraded (14.899%); 1989/290235 objects misplaced (0.685%); 5.0 MiB/s, 80 objects/s recovering 2024-09-13T14:39:49.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:48 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 43242/290235 objects degraded (14.899%), 13 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:49.288 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:49.311 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:49.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:49.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:49.413 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:49.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:48 smithi102 ceph-mon[115269]: pgmap v845: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 76 active+clean; 5.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 158 op/s; 43242/290235 objects degraded (14.899%); 1989/290235 objects misplaced (0.685%); 5.0 MiB/s, 80 objects/s recovering 2024-09-13T14:39:49.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:48 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 43242/290235 objects degraded (14.899%), 13 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:49.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:48 smithi102 ceph-mon[110183]: pgmap v845: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 76 active+clean; 5.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 158 op/s; 43242/290235 objects degraded (14.899%); 1989/290235 objects misplaced (0.685%); 5.0 MiB/s, 80 objects/s recovering 2024-09-13T14:39:49.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:48 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 43242/290235 objects degraded (14.899%), 13 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:49.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:49.437 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:49.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:49.538 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:49.562 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:49.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:49.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:49.666 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:49.690 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:49.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:49.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:49.793 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:49.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:49.815 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:49.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:49.916 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:49.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:49.939 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:49.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:50.040 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:50.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:50.063 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:50.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:50.164 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:50.186 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:50.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:50.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:50.213 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:39:50.288 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:50.312 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:50.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:50.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:50.414 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:50.438 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:50.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:50.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:50.527 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T14:39:50.540 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:50.598 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:50.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:50.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:50.700 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:50.725 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:50.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:50.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:50.827 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:50.850 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:50.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:50.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:50.951 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:50.975 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:50.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:50.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:51.057 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:50 smithi102 ceph-mon[115269]: pgmap v846: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 76 active+clean; 5.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 181 op/s; 42412/288534 objects degraded (14.699%); 1989/288534 objects misplaced (0.689%); 8.6 MiB/s, 138 objects/s recovering 2024-09-13T14:39:51.057 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:50 smithi102 ceph-mon[110183]: pgmap v846: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 76 active+clean; 5.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 181 op/s; 42412/288534 objects degraded (14.699%); 1989/288534 objects misplaced (0.689%); 8.6 MiB/s, 138 objects/s recovering 2024-09-13T14:39:51.077 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:51.099 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:51.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:51.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:51.201 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:51.226 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:51.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:51.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:51.329 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:51.352 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:51.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:51.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:51.454 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:51.484 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:51.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:51.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:51.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:50 smithi195 ceph-mon[104262]: pgmap v846: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 76 active+clean; 5.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 181 op/s; 42412/288534 objects degraded (14.699%); 1989/288534 objects misplaced (0.689%); 8.6 MiB/s, 138 objects/s recovering 2024-09-13T14:39:51.562 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:39:51.567 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:39:51.568 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:39:51.586 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:51.622 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:51.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:51.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:51.724 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:51.751 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:51.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:51.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:51.796 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:39:51.852 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:51.877 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:51.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:51.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:51.978 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:52.002 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:52.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:52.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:52.007 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is unset 2024-09-13T14:39:52.104 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:52.131 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:52.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:52.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:52.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:51 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2284278646' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:39:52.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:51 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:39:52.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:51 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2284278646' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:39:52.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:51 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:39:52.233 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:52.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:52.259 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:52.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:52.360 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:52.387 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:52.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:52.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:52.490 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:52.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:51 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2284278646' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:39:52.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:51 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:39:52.516 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:52.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:52.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:52.618 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:52.645 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:52.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:52.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:52.747 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:52.769 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:52.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:52.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:52.871 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:52.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:52.894 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:52.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:52.995 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:53.018 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:53.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:53.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:53.119 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:53.143 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:53.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:53.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:53.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:53 smithi102 ceph-mon[110183]: pgmap v847: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 76 active+clean; 5.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 155 op/s; 42359/288093 objects degraded (14.703%); 1989/288093 objects misplaced (0.690%); 8.2 MiB/s, 131 objects/s recovering 2024-09-13T14:39:53.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:53 smithi102 ceph-mon[110183]: from='client.45581 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:39:53.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:53 smithi102 ceph-mon[110183]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:39:53.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:53 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:39:53.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:53 smithi102 ceph-mon[110183]: osdmap e262: 8 total, 8 up, 6 in 2024-09-13T14:39:53.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:53 smithi102 ceph-mon[115269]: pgmap v847: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 76 active+clean; 5.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 155 op/s; 42359/288093 objects degraded (14.703%); 1989/288093 objects misplaced (0.690%); 8.2 MiB/s, 131 objects/s recovering 2024-09-13T14:39:53.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:53 smithi102 ceph-mon[115269]: from='client.45581 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:39:53.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:53 smithi102 ceph-mon[115269]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:39:53.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:53 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:39:53.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:53 smithi102 ceph-mon[115269]: osdmap e262: 8 total, 8 up, 6 in 2024-09-13T14:39:53.246 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:53.265 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":847,"stamp":"2024-09-13T14:39:51.467561+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6296377341,"num_objects":96031,"num_object_clones":0,"num_object_copies":288093,"num_objects_missing_on_primary":3959,"num_objects_missing":3959,"num_objects_degraded":42359,"num_objects_misplaced":1989,"num_objects_unfound":0,"num_objects_dirty":96031,"num_whiteouts":0,"num_read":6377,"num_read_kb":13834,"num_write":130746,"num_write_kb":7235010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51605,"num_bytes_recovered":3353742453,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91028,"ondisk_log_size":91028,"up":267,"acting":259,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":309,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":22088588,"kb_used_data":19093212,"kb_used_omap":26,"kb_used_meta":2995301,"kb_avail":540382324,"statfs":{"total":575970213888,"available":553351499776,"internally_reserved":0,"allocated":19551449088,"data_stored":19542947795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27144,"internal_metadata":3067188728},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,1,2,2,2,2,6,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-122093568,"num_objects":-1863,"num_object_clones":0,"num_object_copies":-5589,"num_objects_missing_on_primary":-160,"num_objects_missing":-160,"num_objects_degraded":-1680,"num_objects_misplaced":-1,"num_objects_unfound":0,"num_objects_dirty":-1863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1863,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1582,"num_bytes_recovered":103677952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003992"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":414,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:39:38.057042+0000","last_change":"2024-09-13T14:36:13.859927+0000","last_active":"2024-09-13T14:39:38.057042+0000","last_peered":"2024-09-13T14:39:38.057042+0000","last_clean":"2024-09-13T14:39:38.057042+0000","last_became_active":"2024-09-13T14:36:13.859505+0000","last_became_peered":"2024-09-13T14:36:13.859505+0000","last_unstale":"2024-09-13T14:39:38.057042+0000","last_undegraded":"2024-09-13T14:39:38.057042+0000","last_fullsized":"2024-09-13T14:39:38.057042+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:12:20.989627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":262,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:32.392112+0000","last_change":"2024-09-13T14:37:48.130968+0000","last_active":"2024-09-13T14:39:32.392112+0000","last_peered":"2024-09-13T14:39:32.392112+0000","last_clean":"2024-09-13T14:39:32.392112+0000","last_became_active":"2024-09-13T14:37:48.130603+0000","last_became_peered":"2024-09-13T14:37:48.130603+0000","last_unstale":"2024-09-13T14:39:32.392112+0000","last_undegraded":"2024-09-13T14:39:32.392112+0000","last_fullsized":"2024-09-13T14:39:32.392112+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:42.626386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":58,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.978799+0000","last_change":"2024-09-13T14:36:13.862181+0000","last_active":"2024-09-13T14:39:24.978799+0000","last_peered":"2024-09-13T14:39:24.978799+0000","last_clean":"2024-09-13T14:39:24.978799+0000","last_became_active":"2024-09-13T14:36:13.861873+0000","last_became_peered":"2024-09-13T14:36:13.861873+0000","last_unstale":"2024-09-13T14:39:24.978799+0000","last_undegraded":"2024-09-13T14:39:24.978799+0000","last_fullsized":"2024-09-13T14:39:24.978799+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:14.085700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":51,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:39:38.016810+0000","last_change":"2024-09-13T14:38:53.896672+0000","last_active":"2024-09-13T14:39:38.016810+0000","last_peered":"2024-09-13T14:39:38.016810+0000","last_clean":"2024-09-13T14:39:38.016810+0000","last_became_active":"2024-09-13T14:38:53.896346+0000","last_became_peered":"2024-09-13T14:38:53.896346+0000","last_unstale":"2024-09-13T14:39:38.016810+0000","last_undegraded":"2024-09-13T14:39:38.016810+0000","last_fullsized":"2024-09-13T14:39:38.016810+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:57.789686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":51,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:39:38.017797+0000","last_change":"2024-09-13T14:38:53.893021+0000","last_active":"2024-09-13T14:39:38.017797+0000","last_peered":"2024-09-13T14:39:38.017797+0000","last_clean":"2024-09-13T14:39:38.017797+0000","last_became_active":"2024-09-13T14:38:53.892668+0000","last_became_peered":"2024-09-13T14:38:53.892668+0000","last_unstale":"2024-09-13T14:39:38.017797+0000","last_undegraded":"2024-09-13T14:39:38.017797+0000","last_fullsized":"2024-09-13T14:39:38.017797+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:57.180738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":230,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:27.391671+0000","last_change":"2024-09-13T14:37:48.134642+0000","last_active":"2024-09-13T14:39:27.391671+0000","last_peered":"2024-09-13T14:39:27.391671+0000","last_clean":"2024-09-13T14:39:27.391671+0000","last_became_active":"2024-09-13T14:37:48.134402+0000","last_became_peered":"2024-09-13T14:37:48.134402+0000","last_unstale":"2024-09-13T14:39:27.391671+0000","last_undegraded":"2024-09-13T14:39:27.391671+0000","last_fullsized":"2024-09-13T14:39:27.391671+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:51.313663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":433,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:39:38.058182+0000","last_change":"2024-09-13T14:38:53.903022+0000","last_active":"2024-09-13T14:39:38.058182+0000","last_peered":"2024-09-13T14:39:38.058182+0000","last_clean":"2024-09-13T14:39:38.058182+0000","last_became_active":"2024-09-13T14:38:53.902448+0000","last_became_peered":"2024-09-13T14:38:53.902448+0000","last_unstale":"2024-09-13T14:39:38.058182+0000","last_undegraded":"2024-09-13T14:39:38.058182+0000","last_fullsized":"2024-09-13T14:39:38.058182+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:24.690401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":54,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:39:38.016367+0000","last_change":"2024-09-13T14:38:03.554500+0000","last_active":"2024-09-13T14:39:38.016367+0000","last_peered":"2024-09-13T14:39:38.016367+0000","last_clean":"2024-09-13T14:39:38.016367+0000","last_became_active":"2024-09-13T14:38:03.532910+0000","last_became_peered":"2024-09-13T14:38:03.532910+0000","last_unstale":"2024-09-13T14:39:38.016367+0000","last_undegraded":"2024-09-13T14:39:38.016367+0000","last_fullsized":"2024-09-13T14:39:38.016367+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:27:50.843498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"261'8005","reported_seq":10091,"reported_epoch":261,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:39:51.253170+0000","last_change":"2024-09-13T14:37:49.715669+0000","last_active":"2024-09-13T14:39:51.253170+0000","last_peered":"2024-09-13T14:39:51.253170+0000","last_clean":"2024-09-13T14:37:46.417003+0000","last_became_active":"2024-09-13T14:37:49.690961+0000","last_became_peered":"2024-09-13T14:37:49.690961+0000","last_unstale":"2024-09-13T14:39:51.253170+0000","last_undegraded":"2024-09-13T14:37:49.182535+0000","last_fullsized":"2024-09-13T14:37:49.126746+0000","mapping_epoch":240,"log_start":"210'2300","ondisk_log_start":"210'2300","created":205,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5705,"log_dups_size":0,"ondisk_log_size":5705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:48:52.526429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385024000,"num_objects":5875,"num_object_clones":0,"num_object_copies":17625,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4387,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8005,"num_write_kb":444160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4765,"num_bytes_recovered":312279040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,4],"avail_no_missing":["5","0","4"],"object_location_counts":[{"shards":"0,4,5","objects":4308},{"shards":"4,5","objects":1567}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":22,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.976099+0000","last_change":"2024-09-13T14:38:53.940238+0000","last_active":"2024-09-13T14:39:24.976099+0000","last_peered":"2024-09-13T14:39:24.976099+0000","last_clean":"2024-09-13T14:39:24.976099+0000","last_became_active":"2024-09-13T14:38:53.892825+0000","last_became_peered":"2024-09-13T14:38:53.892825+0000","last_unstale":"2024-09-13T14:39:24.976099+0000","last_undegraded":"2024-09-13T14:39:24.976099+0000","last_fullsized":"2024-09-13T14:39:24.976099+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:46.666581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":78,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.977849+0000","last_change":"2024-09-13T14:38:53.898907+0000","last_active":"2024-09-13T14:39:24.977849+0000","last_peered":"2024-09-13T14:39:24.977849+0000","last_clean":"2024-09-13T14:39:24.977849+0000","last_became_active":"2024-09-13T14:38:53.898572+0000","last_became_peered":"2024-09-13T14:38:53.898572+0000","last_unstale":"2024-09-13T14:39:24.977849+0000","last_undegraded":"2024-09-13T14:39:24.977849+0000","last_fullsized":"2024-09-13T14:39:24.977849+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:29.222150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"261'8207","reported_seq":22338,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:39:47.406078+0000","last_change":"2024-09-13T14:38:57.930032+0000","last_active":"2024-09-13T14:39:47.406078+0000","last_peered":"2024-09-13T14:39:47.406078+0000","last_clean":"2024-09-13T14:39:47.406078+0000","last_became_active":"2024-09-13T14:38:57.921458+0000","last_became_peered":"2024-09-13T14:38:57.921458+0000","last_unstale":"2024-09-13T14:39:47.406078+0000","last_undegraded":"2024-09-13T14:39:47.406078+0000","last_fullsized":"2024-09-13T14:39:47.406078+0000","mapping_epoch":256,"log_start":"212'3139","ondisk_log_start":"212'3139","created":205,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5068,"log_dups_size":0,"ondisk_log_size":5068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:27:12.294038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395509760,"num_objects":6035,"num_object_clones":0,"num_object_copies":18105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8207,"num_write_kb":455744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6990,"num_bytes_recovered":458096640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":22,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.976087+0000","last_change":"2024-09-13T14:38:53.904885+0000","last_active":"2024-09-13T14:39:24.976087+0000","last_peered":"2024-09-13T14:39:24.976087+0000","last_clean":"2024-09-13T14:39:24.976087+0000","last_became_active":"2024-09-13T14:38:53.891726+0000","last_became_peered":"2024-09-13T14:38:53.891726+0000","last_unstale":"2024-09-13T14:39:24.976087+0000","last_undegraded":"2024-09-13T14:39:24.976087+0000","last_fullsized":"2024-09-13T14:39:24.976087+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:57.079205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":268,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:39:37.988450+0000","last_change":"2024-09-13T14:37:48.160504+0000","last_active":"2024-09-13T14:39:37.988450+0000","last_peered":"2024-09-13T14:39:37.988450+0000","last_clean":"2024-09-13T14:39:37.988450+0000","last_became_active":"2024-09-13T14:37:48.160181+0000","last_became_peered":"2024-09-13T14:37:48.160181+0000","last_unstale":"2024-09-13T14:39:37.988450+0000","last_undegraded":"2024-09-13T14:39:37.988450+0000","last_fullsized":"2024-09-13T14:39:37.988450+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:37.087008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"261'8143","reported_seq":9895,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:39:47.825791+0000","last_change":"2024-09-13T14:38:04.531914+0000","last_active":"2024-09-13T14:39:47.825791+0000","last_peered":"2024-09-13T14:39:47.825791+0000","last_clean":"2024-09-13T14:39:47.825791+0000","last_became_active":"2024-09-13T14:38:04.523448+0000","last_became_peered":"2024-09-13T14:38:04.523448+0000","last_unstale":"2024-09-13T14:39:47.825791+0000","last_undegraded":"2024-09-13T14:39:47.825791+0000","last_fullsized":"2024-09-13T14:39:47.825791+0000","mapping_epoch":245,"log_start":"210'2200","ondisk_log_start":"210'2200","created":205,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5943,"log_dups_size":0,"ondisk_log_size":5943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:06.536896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395247616,"num_objects":6031,"num_object_clones":0,"num_object_copies":18093,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8143,"num_write_kb":453568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":811,"num_bytes_recovered":53149696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":259,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.976040+0000","last_change":"2024-09-13T14:37:49.611774+0000","last_active":"2024-09-13T14:39:24.976040+0000","last_peered":"2024-09-13T14:39:24.976040+0000","last_clean":"2024-09-13T14:39:24.976040+0000","last_became_active":"2024-09-13T14:37:49.611088+0000","last_became_peered":"2024-09-13T14:37:49.611088+0000","last_unstale":"2024-09-13T14:39:24.976040+0000","last_undegraded":"2024-09-13T14:39:24.976040+0000","last_fullsized":"2024-09-13T14:39:24.976040+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:08.933213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.975518+0000","last_change":"2024-09-13T14:38:53.899211+0000","last_active":"2024-09-13T14:39:24.975518+0000","last_peered":"2024-09-13T14:39:24.975518+0000","last_clean":"2024-09-13T14:39:24.975518+0000","last_became_active":"2024-09-13T14:38:53.898927+0000","last_became_peered":"2024-09-13T14:38:53.898927+0000","last_unstale":"2024-09-13T14:39:24.975518+0000","last_undegraded":"2024-09-13T14:39:24.975518+0000","last_fullsized":"2024-09-13T14:39:24.975518+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:00:28.857611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"261'8015","reported_seq":8082,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:39:47.386181+0000","last_change":"2024-09-13T14:35:24.004651+0000","last_active":"2024-09-13T14:39:47.386181+0000","last_peered":"2024-09-13T14:39:47.386181+0000","last_clean":"2024-09-13T14:39:47.386181+0000","last_became_active":"2024-09-13T14:35:24.004358+0000","last_became_peered":"2024-09-13T14:35:24.004358+0000","last_unstale":"2024-09-13T14:39:47.386181+0000","last_undegraded":"2024-09-13T14:39:47.386181+0000","last_fullsized":"2024-09-13T14:39:47.386181+0000","mapping_epoch":205,"log_start":"209'1684","ondisk_log_start":"209'1684","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6331,"log_dups_size":0,"ondisk_log_size":6331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:57:38.023669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390397952,"num_objects":5957,"num_object_clones":0,"num_object_copies":17871,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8015,"num_write_kb":447104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":69,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.978666+0000","last_change":"2024-09-13T14:37:49.168003+0000","last_active":"2024-09-13T14:39:24.978666+0000","last_peered":"2024-09-13T14:39:24.978666+0000","last_clean":"2024-09-13T14:39:24.978666+0000","last_became_active":"2024-09-13T14:37:49.167638+0000","last_became_peered":"2024-09-13T14:37:49.167638+0000","last_unstale":"2024-09-13T14:39:24.978666+0000","last_undegraded":"2024-09-13T14:39:24.978666+0000","last_fullsized":"2024-09-13T14:39:24.978666+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:29.051981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":44,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:39:38.016752+0000","last_change":"2024-09-13T14:38:03.528381+0000","last_active":"2024-09-13T14:39:38.016752+0000","last_peered":"2024-09-13T14:39:38.016752+0000","last_clean":"2024-09-13T14:39:38.016752+0000","last_became_active":"2024-09-13T14:38:03.527282+0000","last_became_peered":"2024-09-13T14:38:03.527282+0000","last_unstale":"2024-09-13T14:39:38.016752+0000","last_undegraded":"2024-09-13T14:39:38.016752+0000","last_fullsized":"2024-09-13T14:39:38.016752+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:24.435067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.f","version":"261'8082","reported_seq":8329,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:39:47.932614+0000","last_change":"2024-09-13T14:38:16.371079+0000","last_active":"2024-09-13T14:39:47.932614+0000","last_peered":"2024-09-13T14:39:47.932614+0000","last_clean":"2024-09-13T14:39:47.932614+0000","last_became_active":"2024-09-13T14:38:12.859883+0000","last_became_peered":"2024-09-13T14:38:12.859883+0000","last_unstale":"2024-09-13T14:39:47.932614+0000","last_undegraded":"2024-09-13T14:39:47.932614+0000","last_fullsized":"2024-09-13T14:39:47.932614+0000","mapping_epoch":245,"log_start":"212'2663","ondisk_log_start":"212'2663","created":205,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5419,"log_dups_size":0,"ondisk_log_size":5419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:08.436532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393478144,"num_objects":6004,"num_object_clones":0,"num_object_copies":18012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8082,"num_write_kb":450752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":3670016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":402,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:39:38.057169+0000","last_change":"2024-09-13T14:38:03.549595+0000","last_active":"2024-09-13T14:39:38.057169+0000","last_peered":"2024-09-13T14:39:38.057169+0000","last_clean":"2024-09-13T14:39:38.057169+0000","last_became_active":"2024-09-13T14:38:03.533331+0000","last_became_peered":"2024-09-13T14:38:03.533331+0000","last_unstale":"2024-09-13T14:39:38.057169+0000","last_undegraded":"2024-09-13T14:39:38.057169+0000","last_fullsized":"2024-09-13T14:39:38.057169+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:59:02.470343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":243,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.972983+0000","last_change":"2024-09-13T14:37:48.127600+0000","last_active":"2024-09-13T14:39:24.972983+0000","last_peered":"2024-09-13T14:39:24.972983+0000","last_clean":"2024-09-13T14:39:24.972983+0000","last_became_active":"2024-09-13T14:37:48.127349+0000","last_became_peered":"2024-09-13T14:37:48.127349+0000","last_unstale":"2024-09-13T14:39:24.972983+0000","last_undegraded":"2024-09-13T14:39:24.972983+0000","last_fullsized":"2024-09-13T14:39:24.972983+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:57.268374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"261'8020","reported_seq":10147,"reported_epoch":261,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:39:51.413027+0000","last_change":"2024-09-13T14:38:14.979229+0000","last_active":"2024-09-13T14:39:51.413027+0000","last_peered":"2024-09-13T14:39:51.413027+0000","last_clean":"2024-09-13T14:36:13.131873+0000","last_became_active":"2024-09-13T14:38:10.443954+0000","last_became_peered":"2024-09-13T14:38:10.443954+0000","last_unstale":"2024-09-13T14:39:51.413027+0000","last_undegraded":"2024-09-13T14:38:10.384643+0000","last_fullsized":"2024-09-13T14:38:04.517046+0000","mapping_epoch":245,"log_start":"210'2300","ondisk_log_start":"210'2300","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5720,"log_dups_size":0,"ondisk_log_size":5720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:26:53.493170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":392560640,"num_objects":5990,"num_object_clones":0,"num_object_copies":17970,"num_objects_missing_on_primary":3769,"num_objects_missing":3769,"num_objects_degraded":3910,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8020,"num_write_kb":448320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1010,"num_bytes_recovered":66125824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":1},{"shards":"0,3,5,7","objects":6},{"shards":"0,3,7","objects":3768},{"shards":"0,5","objects":2210},{"shards":"3,5,7","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":176,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.972393+0000","last_change":"2024-09-13T14:37:48.127604+0000","last_active":"2024-09-13T14:39:24.972393+0000","last_peered":"2024-09-13T14:39:24.972393+0000","last_clean":"2024-09-13T14:39:24.972393+0000","last_became_active":"2024-09-13T14:37:48.127300+0000","last_became_peered":"2024-09-13T14:37:48.127300+0000","last_unstale":"2024-09-13T14:39:24.972393+0000","last_undegraded":"2024-09-13T14:39:24.972393+0000","last_fullsized":"2024-09-13T14:39:24.972393+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:51.999506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":296,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.972659+0000","last_change":"2024-09-13T14:38:03.529967+0000","last_active":"2024-09-13T14:39:24.972659+0000","last_peered":"2024-09-13T14:39:24.972659+0000","last_clean":"2024-09-13T14:39:24.972659+0000","last_became_active":"2024-09-13T14:38:03.529646+0000","last_became_peered":"2024-09-13T14:38:03.529646+0000","last_unstale":"2024-09-13T14:39:24.972659+0000","last_undegraded":"2024-09-13T14:39:24.972659+0000","last_fullsized":"2024-09-13T14:39:24.972659+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:52:05.193279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"261'8125","reported_seq":8452,"reported_epoch":261,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:39:47.732191+0000","last_change":"2024-09-13T14:38:03.544515+0000","last_active":"2024-09-13T14:39:47.732191+0000","last_peered":"2024-09-13T14:39:47.732191+0000","last_clean":"2024-09-13T14:35:51.090657+0000","last_became_active":"2024-09-13T14:37:49.619314+0000","last_became_peered":"2024-09-13T14:37:49.619314+0000","last_unstale":"2024-09-13T14:39:47.732191+0000","last_undegraded":"2024-09-13T14:37:49.147721+0000","last_fullsized":"2024-09-13T14:37:49.120361+0000","mapping_epoch":240,"log_start":"210'2200","ondisk_log_start":"210'2200","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5925,"log_dups_size":0,"ondisk_log_size":5925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394133528,"num_objects":6015,"num_object_clones":0,"num_object_copies":18045,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4393,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6015,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8125,"num_write_kb":452417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":6291456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":1687}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":257,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:39:37.988700+0000","last_change":"2024-09-13T14:36:13.860367+0000","last_active":"2024-09-13T14:39:37.988700+0000","last_peered":"2024-09-13T14:39:37.988700+0000","last_clean":"2024-09-13T14:39:37.988700+0000","last_became_active":"2024-09-13T14:36:13.859982+0000","last_became_peered":"2024-09-13T14:36:13.859982+0000","last_unstale":"2024-09-13T14:39:37.988700+0000","last_undegraded":"2024-09-13T14:39:37.988700+0000","last_fullsized":"2024-09-13T14:39:37.988700+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:08:23.311350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":252,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.973407+0000","last_change":"2024-09-13T14:37:48.147360+0000","last_active":"2024-09-13T14:39:24.973407+0000","last_peered":"2024-09-13T14:39:24.973407+0000","last_clean":"2024-09-13T14:39:24.973407+0000","last_became_active":"2024-09-13T14:37:48.146885+0000","last_became_peered":"2024-09-13T14:37:48.146885+0000","last_unstale":"2024-09-13T14:39:24.973407+0000","last_undegraded":"2024-09-13T14:39:24.973407+0000","last_fullsized":"2024-09-13T14:39:24.973407+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:27.841810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"261'8055","reported_seq":17284,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:39:47.278259+0000","last_change":"2024-09-13T14:38:39.650332+0000","last_active":"2024-09-13T14:39:47.278259+0000","last_peered":"2024-09-13T14:39:47.278259+0000","last_clean":"2024-09-13T14:39:47.278259+0000","last_became_active":"2024-09-13T14:38:39.640835+0000","last_became_peered":"2024-09-13T14:38:39.640835+0000","last_unstale":"2024-09-13T14:39:47.278259+0000","last_undegraded":"2024-09-13T14:39:47.278259+0000","last_fullsized":"2024-09-13T14:39:47.278259+0000","mapping_epoch":251,"log_start":"212'3051","ondisk_log_start":"212'3051","created":205,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5004,"log_dups_size":0,"ondisk_log_size":5004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:48.176606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394461184,"num_objects":6019,"num_object_clones":0,"num_object_copies":18057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8055,"num_write_kb":450368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4544,"num_bytes_recovered":292945920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":420,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:39:38.057532+0000","last_change":"2024-09-13T14:38:53.900041+0000","last_active":"2024-09-13T14:39:38.057532+0000","last_peered":"2024-09-13T14:39:38.057532+0000","last_clean":"2024-09-13T14:39:38.057532+0000","last_became_active":"2024-09-13T14:38:53.899203+0000","last_became_peered":"2024-09-13T14:38:53.899203+0000","last_unstale":"2024-09-13T14:39:38.057532+0000","last_undegraded":"2024-09-13T14:39:38.057532+0000","last_fullsized":"2024-09-13T14:39:38.057532+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:28:07.883094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":352,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:39:38.058257+0000","last_change":"2024-09-13T14:26:15.823198+0000","last_active":"2024-09-13T14:39:38.058257+0000","last_peered":"2024-09-13T14:39:38.058257+0000","last_clean":"2024-09-13T14:39:38.058257+0000","last_became_active":"2024-09-13T14:26:15.822717+0000","last_became_peered":"2024-09-13T14:26:15.822717+0000","last_unstale":"2024-09-13T14:39:38.058257+0000","last_undegraded":"2024-09-13T14:39:38.058257+0000","last_fullsized":"2024-09-13T14:39:38.058257+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:22.471622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.6","version":"261'8194","reported_seq":10310,"reported_epoch":261,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-13T14:39:48.071563+0000","last_change":"2024-09-13T14:38:55.132683+0000","last_active":"2024-09-13T14:39:48.071563+0000","last_peered":"2024-09-13T14:39:48.071563+0000","last_clean":"2024-09-13T14:36:13.124368+0000","last_became_active":"2024-09-13T14:38:55.120883+0000","last_became_peered":"2024-09-13T14:38:55.120883+0000","last_unstale":"2024-09-13T14:39:48.071563+0000","last_undegraded":"2024-09-13T14:38:55.026610+0000","last_fullsized":"2024-09-13T14:39:48.071563+0000","mapping_epoch":254,"log_start":"212'3119","ondisk_log_start":"212'3119","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5075,"log_dups_size":0,"ondisk_log_size":5075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393478144,"num_objects":6004,"num_object_clones":0,"num_object_copies":18012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4017,"num_objects_misplaced":1987,"num_objects_unfound":0,"num_objects_dirty":6004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8194,"num_write_kb":454336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":988,"num_bytes_recovered":53542912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,3,5],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3,4","objects":3738},{"shards":"0,3,5","objects":2266}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":416,"reported_epoch":261,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-13T14:39:38.056846+0000","last_change":"2024-09-13T14:38:54.899624+0000","last_active":"2024-09-13T14:39:38.056846+0000","last_peered":"2024-09-13T14:39:38.056846+0000","last_clean":"2024-09-13T14:17:27.401592+0000","last_became_active":"2024-09-13T14:38:54.899282+0000","last_became_peered":"2024-09-13T14:38:54.899282+0000","last_unstale":"2024-09-13T14:39:38.056846+0000","last_undegraded":"2024-09-13T14:38:54.891176+0000","last_fullsized":"2024-09-13T14:39:38.056846+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:07:41.927740+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[3,0,6],"avail_no_missing":["3","4","6"],"object_location_counts":[{"shards":"3,4,6","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":498,"reported_epoch":260,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:39:24.978710+0000","last_change":"2024-09-13T14:38:54.904329+0000","last_active":"2024-09-13T14:39:24.978710+0000","last_peered":"2024-09-13T14:39:24.978710+0000","last_clean":"2024-09-13T14:38:39.648017+0000","last_became_active":"2024-09-13T14:38:54.903798+0000","last_became_peered":"2024-09-13T14:38:54.903798+0000","last_unstale":"2024-09-13T14:39:24.978710+0000","last_undegraded":"2024-09-13T14:38:54.890505+0000","last_fullsized":"2024-09-13T14:38:54.889701+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:18.120931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,3],"avail_no_missing":["5","3","6"],"object_location_counts":[{"shards":"3,5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":355,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:39:38.057584+0000","last_change":"2024-09-13T14:37:49.160719+0000","last_active":"2024-09-13T14:39:38.057584+0000","last_peered":"2024-09-13T14:39:38.057584+0000","last_clean":"2024-09-13T14:39:38.057584+0000","last_became_active":"2024-09-13T14:37:49.160401+0000","last_became_peered":"2024-09-13T14:37:49.160401+0000","last_unstale":"2024-09-13T14:39:38.057584+0000","last_undegraded":"2024-09-13T14:39:38.057584+0000","last_fullsized":"2024-09-13T14:39:38.057584+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:21.737945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":103,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.972589+0000","last_change":"2024-09-13T14:37:48.172940+0000","last_active":"2024-09-13T14:39:24.972589+0000","last_peered":"2024-09-13T14:39:24.972589+0000","last_clean":"2024-09-13T14:39:24.972589+0000","last_became_active":"2024-09-13T14:37:48.172060+0000","last_became_peered":"2024-09-13T14:37:48.172060+0000","last_unstale":"2024-09-13T14:39:24.972589+0000","last_undegraded":"2024-09-13T14:39:24.972589+0000","last_fullsized":"2024-09-13T14:39:24.972589+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:07.480645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.5","version":"261'8072","reported_seq":24415,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:39:47.231298+0000","last_change":"2024-09-13T14:39:34.446699+0000","last_active":"2024-09-13T14:39:47.231298+0000","last_peered":"2024-09-13T14:39:47.231298+0000","last_clean":"2024-09-13T14:39:47.231298+0000","last_became_active":"2024-09-13T14:38:04.600616+0000","last_became_peered":"2024-09-13T14:38:04.600616+0000","last_unstale":"2024-09-13T14:39:47.231298+0000","last_undegraded":"2024-09-13T14:39:47.231298+0000","last_fullsized":"2024-09-13T14:39:47.231298+0000","mapping_epoch":245,"log_start":"210'1816","ondisk_log_start":"210'1816","created":205,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6256,"log_dups_size":0,"ondisk_log_size":6256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:00:44.321678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":397803520,"num_objects":6070,"num_object_clones":0,"num_object_copies":18210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8072,"num_write_kb":452544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8120,"num_bytes_recovered":521076736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":476,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.974815+0000","last_change":"2024-09-13T14:26:15.835178+0000","last_active":"2024-09-13T14:39:24.974815+0000","last_peered":"2024-09-13T14:39:24.974815+0000","last_clean":"2024-09-13T14:39:24.974815+0000","last_became_active":"2024-09-13T14:26:15.820283+0000","last_became_peered":"2024-09-13T14:26:15.820283+0000","last_unstale":"2024-09-13T14:39:24.974815+0000","last_undegraded":"2024-09-13T14:39:24.974815+0000","last_fullsized":"2024-09-13T14:39:24.974815+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:30.340038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":44,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:39:38.017097+0000","last_change":"2024-09-13T14:38:03.525682+0000","last_active":"2024-09-13T14:39:38.017097+0000","last_peered":"2024-09-13T14:39:38.017097+0000","last_clean":"2024-09-13T14:39:38.017097+0000","last_became_active":"2024-09-13T14:38:03.525406+0000","last_became_peered":"2024-09-13T14:38:03.525406+0000","last_unstale":"2024-09-13T14:39:38.017097+0000","last_undegraded":"2024-09-13T14:39:38.017097+0000","last_fullsized":"2024-09-13T14:39:38.017097+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:41:28.970300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":51,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:39:38.017884+0000","last_change":"2024-09-13T14:38:53.991738+0000","last_active":"2024-09-13T14:39:38.017884+0000","last_peered":"2024-09-13T14:39:38.017884+0000","last_clean":"2024-09-13T14:39:38.017884+0000","last_became_active":"2024-09-13T14:38:53.991417+0000","last_became_peered":"2024-09-13T14:38:53.991417+0000","last_unstale":"2024-09-13T14:39:38.017884+0000","last_undegraded":"2024-09-13T14:39:38.017884+0000","last_fullsized":"2024-09-13T14:39:38.017884+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:50:38.559838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"261'8126","reported_seq":22270,"reported_epoch":261,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-13T14:39:47.966173+0000","last_change":"2024-09-13T14:38:17.472071+0000","last_active":"2024-09-13T14:39:47.966173+0000","last_peered":"2024-09-13T14:39:47.966173+0000","last_clean":"2024-09-13T14:37:46.290754+0000","last_became_active":"2024-09-13T14:38:17.381753+0000","last_became_peered":"2024-09-13T14:38:17.381753+0000","last_unstale":"2024-09-13T14:39:47.966173+0000","last_undegraded":"2024-09-13T14:38:13.504140+0000","last_fullsized":"2024-09-13T14:39:47.966173+0000","mapping_epoch":245,"log_start":"210'2159","ondisk_log_start":"210'2159","created":205,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5967,"log_dups_size":0,"ondisk_log_size":5967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":391905280,"num_objects":5980,"num_object_clones":0,"num_object_copies":17940,"num_objects_missing_on_primary":83,"num_objects_missing":83,"num_objects_degraded":4582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8126,"num_write_kb":451392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7053,"num_bytes_recovered":462225408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"0,4,5","objects":4413},{"shards":"0,4,6","objects":1567}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":766,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:39:38.056889+0000","last_change":"2024-09-13T14:37:48.131799+0000","last_active":"2024-09-13T14:39:38.056889+0000","last_peered":"2024-09-13T14:39:38.056889+0000","last_clean":"2024-09-13T14:39:38.056889+0000","last_became_active":"2024-09-13T14:37:48.130837+0000","last_became_peered":"2024-09-13T14:37:48.130837+0000","last_unstale":"2024-09-13T14:39:38.056889+0000","last_undegraded":"2024-09-13T14:39:38.056889+0000","last_fullsized":"2024-09-13T14:39:38.056889+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:06:32.663770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":284,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:39:38.016595+0000","last_change":"2024-09-13T14:37:49.301531+0000","last_active":"2024-09-13T14:39:38.016595+0000","last_peered":"2024-09-13T14:39:38.016595+0000","last_clean":"2024-09-13T14:39:38.016595+0000","last_became_active":"2024-09-13T14:37:49.301249+0000","last_became_peered":"2024-09-13T14:37:49.301249+0000","last_unstale":"2024-09-13T14:39:38.016595+0000","last_undegraded":"2024-09-13T14:39:38.016595+0000","last_fullsized":"2024-09-13T14:39:38.016595+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:10.977710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":85,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.979179+0000","last_change":"2024-09-13T14:38:53.901427+0000","last_active":"2024-09-13T14:39:24.979179+0000","last_peered":"2024-09-13T14:39:24.979179+0000","last_clean":"2024-09-13T14:39:24.979179+0000","last_became_active":"2024-09-13T14:38:53.901157+0000","last_became_peered":"2024-09-13T14:38:53.901157+0000","last_unstale":"2024-09-13T14:39:24.979179+0000","last_undegraded":"2024-09-13T14:39:24.979179+0000","last_fullsized":"2024-09-13T14:39:24.979179+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:28:21.713025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"261'8062","reported_seq":22096,"reported_epoch":261,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-13T14:39:47.977900+0000","last_change":"2024-09-13T14:38:16.552896+0000","last_active":"2024-09-13T14:39:47.977900+0000","last_peered":"2024-09-13T14:39:47.977900+0000","last_clean":"2024-09-13T14:37:46.405415+0000","last_became_active":"2024-09-13T14:38:16.427510+0000","last_became_peered":"2024-09-13T14:38:16.427510+0000","last_unstale":"2024-09-13T14:39:47.977900+0000","last_undegraded":"2024-09-13T14:38:12.738464+0000","last_fullsized":"2024-09-13T14:39:47.977900+0000","mapping_epoch":245,"log_start":"210'2518","ondisk_log_start":"210'2518","created":205,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5544,"log_dups_size":0,"ondisk_log_size":5544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":389021696,"num_objects":5936,"num_object_clones":0,"num_object_copies":17808,"num_objects_missing_on_primary":100,"num_objects_missing":100,"num_objects_degraded":4572,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8062,"num_write_kb":447936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6999,"num_bytes_recovered":458686464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"0,3,6","objects":1531},{"shards":"0,3,7","objects":4405}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":967,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.978419+0000","last_change":"2024-09-13T14:37:49.147498+0000","last_active":"2024-09-13T14:39:24.978419+0000","last_peered":"2024-09-13T14:39:24.978419+0000","last_clean":"2024-09-13T14:39:24.978419+0000","last_became_active":"2024-09-13T14:37:49.147028+0000","last_became_peered":"2024-09-13T14:37:49.147028+0000","last_unstale":"2024-09-13T14:39:24.978419+0000","last_undegraded":"2024-09-13T14:39:24.978419+0000","last_fullsized":"2024-09-13T14:39:24.978419+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:11:29.050178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":265,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.972906+0000","last_change":"2024-09-13T14:38:03.528133+0000","last_active":"2024-09-13T14:39:24.972906+0000","last_peered":"2024-09-13T14:39:24.972906+0000","last_clean":"2024-09-13T14:39:24.972906+0000","last_became_active":"2024-09-13T14:38:03.527829+0000","last_became_peered":"2024-09-13T14:38:03.527829+0000","last_unstale":"2024-09-13T14:39:24.972906+0000","last_undegraded":"2024-09-13T14:39:24.972906+0000","last_fullsized":"2024-09-13T14:39:24.972906+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:48.254169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":405,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:39:38.057059+0000","last_change":"2024-09-13T14:38:53.899755+0000","last_active":"2024-09-13T14:39:38.057059+0000","last_peered":"2024-09-13T14:39:38.057059+0000","last_clean":"2024-09-13T14:39:38.057059+0000","last_became_active":"2024-09-13T14:38:53.899448+0000","last_became_peered":"2024-09-13T14:38:53.899448+0000","last_unstale":"2024-09-13T14:39:38.057059+0000","last_undegraded":"2024-09-13T14:39:38.057059+0000","last_fullsized":"2024-09-13T14:39:38.057059+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:55.897035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"261'8250","reported_seq":8335,"reported_epoch":261,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:39:47.707115+0000","last_change":"2024-09-13T14:38:03.567892+0000","last_active":"2024-09-13T14:39:47.707115+0000","last_peered":"2024-09-13T14:39:47.707115+0000","last_clean":"2024-09-13T14:35:51.090791+0000","last_became_active":"2024-09-13T14:37:49.665025+0000","last_became_peered":"2024-09-13T14:37:49.665025+0000","last_unstale":"2024-09-13T14:39:47.707115+0000","last_undegraded":"2024-09-13T14:37:49.169101+0000","last_fullsized":"2024-09-13T14:37:49.121596+0000","mapping_epoch":240,"log_start":"210'2163","ondisk_log_start":"210'2163","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6087,"log_dups_size":0,"ondisk_log_size":6087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:02.321504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":396623872,"num_objects":6052,"num_object_clones":0,"num_object_copies":18156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4501,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8250,"num_write_kb":457664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":1617}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":69,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.978561+0000","last_change":"2024-09-13T14:37:49.149770+0000","last_active":"2024-09-13T14:39:24.978561+0000","last_peered":"2024-09-13T14:39:24.978561+0000","last_clean":"2024-09-13T14:39:24.978561+0000","last_became_active":"2024-09-13T14:37:49.148991+0000","last_became_peered":"2024-09-13T14:37:49.148991+0000","last_unstale":"2024-09-13T14:39:24.978561+0000","last_undegraded":"2024-09-13T14:39:24.978561+0000","last_fullsized":"2024-09-13T14:39:24.978561+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:04:04.193366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":1103,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:39:38.063743+0000","last_change":"2024-09-13T14:30:58.928302+0000","last_active":"2024-09-13T14:39:38.063743+0000","last_peered":"2024-09-13T14:39:38.063743+0000","last_clean":"2024-09-13T14:39:38.063743+0000","last_became_active":"2024-09-13T14:30:58.927718+0000","last_became_peered":"2024-09-13T14:30:58.927718+0000","last_unstale":"2024-09-13T14:39:38.063743+0000","last_undegraded":"2024-09-13T14:39:38.063743+0000","last_fullsized":"2024-09-13T14:39:38.063743+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:37.323036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":76,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.978596+0000","last_change":"2024-09-13T14:38:03.534628+0000","last_active":"2024-09-13T14:39:24.978596+0000","last_peered":"2024-09-13T14:39:24.978596+0000","last_clean":"2024-09-13T14:39:24.978596+0000","last_became_active":"2024-09-13T14:38:03.534262+0000","last_became_peered":"2024-09-13T14:38:03.534262+0000","last_unstale":"2024-09-13T14:39:24.978596+0000","last_undegraded":"2024-09-13T14:39:24.978596+0000","last_fullsized":"2024-09-13T14:39:24.978596+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:31.168783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"261'8093","reported_seq":22240,"reported_epoch":261,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:39:51.264615+0000","last_change":"2024-09-13T14:38:53.944111+0000","last_active":"2024-09-13T14:39:51.264615+0000","last_peered":"2024-09-13T14:39:51.264615+0000","last_clean":"2024-09-13T14:37:47.047672+0000","last_became_active":"2024-09-13T14:37:49.664294+0000","last_became_peered":"2024-09-13T14:37:49.664294+0000","last_unstale":"2024-09-13T14:39:51.264615+0000","last_undegraded":"2024-09-13T14:38:53.940822+0000","last_fullsized":"2024-09-13T14:37:49.126627+0000","mapping_epoch":240,"log_start":"210'2400","ondisk_log_start":"210'2400","created":205,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5693,"log_dups_size":0,"ondisk_log_size":5693,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:11.766692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390529024,"num_objects":5959,"num_object_clones":0,"num_object_copies":17877,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4535,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8093,"num_write_kb":449664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7009,"num_bytes_recovered":459341824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3],"avail_no_missing":["5","0","3"],"object_location_counts":[{"shards":"0,3,5","objects":4460},{"shards":"3,5","objects":1499}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":88,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.978795+0000","last_change":"2024-09-13T14:38:03.534303+0000","last_active":"2024-09-13T14:39:24.978795+0000","last_peered":"2024-09-13T14:39:24.978795+0000","last_clean":"2024-09-13T14:39:24.978795+0000","last_became_active":"2024-09-13T14:38:03.533171+0000","last_became_peered":"2024-09-13T14:38:03.533171+0000","last_unstale":"2024-09-13T14:39:24.978795+0000","last_undegraded":"2024-09-13T14:39:24.978795+0000","last_fullsized":"2024-09-13T14:39:24.978795+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:00:16.553526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":781,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:32.392145+0000","last_change":"2024-09-13T14:38:04.596192+0000","last_active":"2024-09-13T14:39:32.392145+0000","last_peered":"2024-09-13T14:39:32.392145+0000","last_clean":"2024-09-13T14:39:32.392145+0000","last_became_active":"2024-09-13T14:38:04.593299+0000","last_became_peered":"2024-09-13T14:38:04.593299+0000","last_unstale":"2024-09-13T14:39:32.392145+0000","last_undegraded":"2024-09-13T14:39:32.392145+0000","last_fullsized":"2024-09-13T14:39:32.392145+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:34.241410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":44,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:39:38.016826+0000","last_change":"2024-09-13T14:38:03.533120+0000","last_active":"2024-09-13T14:39:38.016826+0000","last_peered":"2024-09-13T14:39:38.016826+0000","last_clean":"2024-09-13T14:39:38.016826+0000","last_became_active":"2024-09-13T14:38:03.532829+0000","last_became_peered":"2024-09-13T14:38:03.532829+0000","last_unstale":"2024-09-13T14:39:38.016826+0000","last_undegraded":"2024-09-13T14:39:38.016826+0000","last_fullsized":"2024-09-13T14:39:38.016826+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:24.514661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"261'8228","reported_seq":8360,"reported_epoch":261,"state":"active+undersized+degraded+remapped+backfill_wait","last_fresh":"2024-09-13T14:39:47.253558+0000","last_change":"2024-09-13T14:37:49.295861+0000","last_active":"2024-09-13T14:39:47.253558+0000","last_peered":"2024-09-13T14:39:47.253558+0000","last_clean":"2024-09-13T14:35:51.078708+0000","last_became_active":"2024-09-13T14:37:49.290403+0000","last_became_peered":"2024-09-13T14:37:49.290403+0000","last_unstale":"2024-09-13T14:39:47.253558+0000","last_undegraded":"2024-09-13T14:37:49.147402+0000","last_fullsized":"2024-09-13T14:37:49.117202+0000","mapping_epoch":240,"log_start":"212'3218","ondisk_log_start":"212'3218","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5010,"log_dups_size":0,"ondisk_log_size":5010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404357120,"num_objects":6170,"num_object_clones":0,"num_object_copies":18510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8228,"num_write_kb":460736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":6170}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":446,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:39:38.058263+0000","last_change":"2024-09-13T14:38:53.938227+0000","last_active":"2024-09-13T14:39:38.058263+0000","last_peered":"2024-09-13T14:39:38.058263+0000","last_clean":"2024-09-13T14:39:38.058263+0000","last_became_active":"2024-09-13T14:38:53.902867+0000","last_became_peered":"2024-09-13T14:38:53.902867+0000","last_unstale":"2024-09-13T14:39:38.058263+0000","last_undegraded":"2024-09-13T14:39:38.058263+0000","last_fullsized":"2024-09-13T14:39:38.058263+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:16.831756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":559,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.973532+0000","last_change":"2024-09-13T14:37:48.147677+0000","last_active":"2024-09-13T14:39:24.973532+0000","last_peered":"2024-09-13T14:39:24.973532+0000","last_clean":"2024-09-13T14:39:24.973532+0000","last_became_active":"2024-09-13T14:37:48.147001+0000","last_became_peered":"2024-09-13T14:37:48.147001+0000","last_unstale":"2024-09-13T14:39:24.973532+0000","last_undegraded":"2024-09-13T14:39:24.973532+0000","last_fullsized":"2024-09-13T14:39:24.973532+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:35:00.653576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":428,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:39:38.056756+0000","last_change":"2024-09-13T14:38:53.900107+0000","last_active":"2024-09-13T14:39:38.056756+0000","last_peered":"2024-09-13T14:39:38.056756+0000","last_clean":"2024-09-13T14:39:38.056756+0000","last_became_active":"2024-09-13T14:38:53.899556+0000","last_became_peered":"2024-09-13T14:38:53.899556+0000","last_unstale":"2024-09-13T14:39:38.056756+0000","last_undegraded":"2024-09-13T14:39:38.056756+0000","last_fullsized":"2024-09-13T14:39:38.056756+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:15:47.918163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"261'7946","reported_seq":14195,"reported_epoch":261,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T14:39:47.463100+0000","last_change":"2024-09-13T14:39:34.447300+0000","last_active":"2024-09-13T14:39:47.463100+0000","last_peered":"2024-09-13T14:39:47.463100+0000","last_clean":"2024-09-13T14:35:51.088129+0000","last_became_active":"2024-09-13T14:37:49.691874+0000","last_became_peered":"2024-09-13T14:37:49.691874+0000","last_unstale":"2024-09-13T14:39:47.463100+0000","last_undegraded":"2024-09-13T14:37:49.158458+0000","last_fullsized":"2024-09-13T14:37:49.116993+0000","mapping_epoch":240,"log_start":"212'2847","ondisk_log_start":"212'2847","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5099,"log_dups_size":0,"ondisk_log_size":5099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":387055616,"num_objects":5906,"num_object_clones":0,"num_object_copies":17718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7946,"num_write_kb":443264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3056,"num_bytes_recovered":200278016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":4632}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":59,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:39:38.017713+0000","last_change":"2024-09-13T14:38:53.901806+0000","last_active":"2024-09-13T14:39:38.017713+0000","last_peered":"2024-09-13T14:39:38.017713+0000","last_clean":"2024-09-13T14:39:38.017713+0000","last_became_active":"2024-09-13T14:38:53.891055+0000","last_became_peered":"2024-09-13T14:38:53.891055+0000","last_unstale":"2024-09-13T14:39:38.017713+0000","last_undegraded":"2024-09-13T14:39:38.017713+0000","last_fullsized":"2024-09-13T14:39:38.017713+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:03:00.416198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":16,"reported_epoch":260,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-13T14:39:24.975689+0000","last_change":"2024-09-13T14:38:53.900902+0000","last_active":"2024-09-13T14:39:24.975689+0000","last_peered":"2024-09-13T14:39:24.975689+0000","last_clean":"2024-09-13T14:38:39.635779+0000","last_became_active":"2024-09-13T14:38:53.900626+0000","last_became_peered":"2024-09-13T14:38:53.900626+0000","last_unstale":"2024-09-13T14:39:24.975689+0000","last_undegraded":"2024-09-13T14:38:53.885778+0000","last_fullsized":"2024-09-13T14:39:24.975689+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:37:21.715778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":7,"num_objects_missing":7,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":["3","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":249,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.972451+0000","last_change":"2024-09-13T14:37:49.172218+0000","last_active":"2024-09-13T14:39:24.972451+0000","last_peered":"2024-09-13T14:39:24.972451+0000","last_clean":"2024-09-13T14:39:24.972451+0000","last_became_active":"2024-09-13T14:37:49.171718+0000","last_became_peered":"2024-09-13T14:37:49.171718+0000","last_unstale":"2024-09-13T14:39:24.972451+0000","last_undegraded":"2024-09-13T14:39:24.972451+0000","last_fullsized":"2024-09-13T14:39:24.972451+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:46:54.626521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":272,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:32.758653+0000","last_change":"2024-09-13T14:37:49.230094+0000","last_active":"2024-09-13T14:39:32.758653+0000","last_peered":"2024-09-13T14:39:32.758653+0000","last_clean":"2024-09-13T14:39:32.758653+0000","last_became_active":"2024-09-13T14:37:49.229393+0000","last_became_peered":"2024-09-13T14:37:49.229393+0000","last_unstale":"2024-09-13T14:39:32.758653+0000","last_undegraded":"2024-09-13T14:39:32.758653+0000","last_fullsized":"2024-09-13T14:39:32.758653+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:02:28.690318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":252,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:37.761015+0000","last_change":"2024-09-13T14:37:48.128447+0000","last_active":"2024-09-13T14:39:37.761015+0000","last_peered":"2024-09-13T14:39:37.761015+0000","last_clean":"2024-09-13T14:39:37.761015+0000","last_became_active":"2024-09-13T14:37:48.128143+0000","last_became_peered":"2024-09-13T14:37:48.128143+0000","last_unstale":"2024-09-13T14:39:37.761015+0000","last_undegraded":"2024-09-13T14:39:37.761015+0000","last_fullsized":"2024-09-13T14:39:37.761015+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:22.204039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":44,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:39:38.016724+0000","last_change":"2024-09-13T14:38:03.531683+0000","last_active":"2024-09-13T14:39:38.016724+0000","last_peered":"2024-09-13T14:39:38.016724+0000","last_clean":"2024-09-13T14:39:38.016724+0000","last_became_active":"2024-09-13T14:38:03.531476+0000","last_became_peered":"2024-09-13T14:38:03.531476+0000","last_unstale":"2024-09-13T14:39:38.016724+0000","last_undegraded":"2024-09-13T14:39:38.016724+0000","last_fullsized":"2024-09-13T14:39:38.016724+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:41.557569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":24,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.974965+0000","last_change":"2024-09-13T14:38:53.954281+0000","last_active":"2024-09-13T14:39:24.974965+0000","last_peered":"2024-09-13T14:39:24.974965+0000","last_clean":"2024-09-13T14:39:24.974965+0000","last_became_active":"2024-09-13T14:38:53.900912+0000","last_became_peered":"2024-09-13T14:38:53.900912+0000","last_unstale":"2024-09-13T14:39:24.974965+0000","last_undegraded":"2024-09-13T14:39:24.974965+0000","last_fullsized":"2024-09-13T14:39:24.974965+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:47.876275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":76,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:25.003967+0000","last_change":"2024-09-13T14:38:54.000782+0000","last_active":"2024-09-13T14:39:25.003967+0000","last_peered":"2024-09-13T14:39:25.003967+0000","last_clean":"2024-09-13T14:39:25.003967+0000","last_became_active":"2024-09-13T14:38:54.000322+0000","last_became_peered":"2024-09-13T14:38:54.000322+0000","last_unstale":"2024-09-13T14:39:25.003967+0000","last_undegraded":"2024-09-13T14:39:25.003967+0000","last_fullsized":"2024-09-13T14:39:25.003967+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:09.057597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":370,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:39:38.056635+0000","last_change":"2024-09-13T14:38:53.899535+0000","last_active":"2024-09-13T14:39:38.056635+0000","last_peered":"2024-09-13T14:39:38.056635+0000","last_clean":"2024-09-13T14:39:38.056635+0000","last_became_active":"2024-09-13T14:38:53.899183+0000","last_became_peered":"2024-09-13T14:38:53.899183+0000","last_unstale":"2024-09-13T14:39:38.056635+0000","last_undegraded":"2024-09-13T14:39:38.056635+0000","last_fullsized":"2024-09-13T14:39:38.056635+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:33.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":44,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:39:38.018072+0000","last_change":"2024-09-13T14:38:03.543426+0000","last_active":"2024-09-13T14:39:38.018072+0000","last_peered":"2024-09-13T14:39:38.018072+0000","last_clean":"2024-09-13T14:39:38.018072+0000","last_became_active":"2024-09-13T14:38:03.542661+0000","last_became_peered":"2024-09-13T14:38:03.542661+0000","last_unstale":"2024-09-13T14:39:38.018072+0000","last_undegraded":"2024-09-13T14:39:38.018072+0000","last_fullsized":"2024-09-13T14:39:38.018072+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:54.050127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":399,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:39:38.056671+0000","last_change":"2024-09-13T14:37:48.131321+0000","last_active":"2024-09-13T14:39:38.056671+0000","last_peered":"2024-09-13T14:39:38.056671+0000","last_clean":"2024-09-13T14:39:38.056671+0000","last_became_active":"2024-09-13T14:37:48.130998+0000","last_became_peered":"2024-09-13T14:37:48.130998+0000","last_unstale":"2024-09-13T14:39:38.056671+0000","last_undegraded":"2024-09-13T14:39:38.056671+0000","last_fullsized":"2024-09-13T14:39:38.056671+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:33:26.597825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":88,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.979149+0000","last_change":"2024-09-13T14:38:03.535454+0000","last_active":"2024-09-13T14:39:24.979149+0000","last_peered":"2024-09-13T14:39:24.979149+0000","last_clean":"2024-09-13T14:39:24.979149+0000","last_became_active":"2024-09-13T14:38:03.535131+0000","last_became_peered":"2024-09-13T14:38:03.535131+0000","last_unstale":"2024-09-13T14:39:24.979149+0000","last_undegraded":"2024-09-13T14:39:24.979149+0000","last_fullsized":"2024-09-13T14:39:24.979149+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:24.477601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":424,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:39:38.057446+0000","last_change":"2024-09-13T14:38:03.524939+0000","last_active":"2024-09-13T14:39:38.057446+0000","last_peered":"2024-09-13T14:39:38.057446+0000","last_clean":"2024-09-13T14:39:38.057446+0000","last_became_active":"2024-09-13T14:38:03.524406+0000","last_became_peered":"2024-09-13T14:38:03.524406+0000","last_unstale":"2024-09-13T14:39:38.057446+0000","last_undegraded":"2024-09-13T14:39:38.057446+0000","last_fullsized":"2024-09-13T14:39:38.057446+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:19:45.257424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":51,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:39:38.017631+0000","last_change":"2024-09-13T14:38:53.892948+0000","last_active":"2024-09-13T14:39:38.017631+0000","last_peered":"2024-09-13T14:39:38.017631+0000","last_clean":"2024-09-13T14:39:38.017631+0000","last_became_active":"2024-09-13T14:38:53.892665+0000","last_became_peered":"2024-09-13T14:38:53.892665+0000","last_unstale":"2024-09-13T14:39:38.017631+0000","last_undegraded":"2024-09-13T14:39:38.017631+0000","last_fullsized":"2024-09-13T14:39:38.017631+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:08.292359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":372,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:39:38.056806+0000","last_change":"2024-09-13T14:37:48.139348+0000","last_active":"2024-09-13T14:39:38.056806+0000","last_peered":"2024-09-13T14:39:38.056806+0000","last_clean":"2024-09-13T14:39:38.056806+0000","last_became_active":"2024-09-13T14:37:48.139115+0000","last_became_peered":"2024-09-13T14:37:48.139115+0000","last_unstale":"2024-09-13T14:39:38.056806+0000","last_undegraded":"2024-09-13T14:39:38.056806+0000","last_fullsized":"2024-09-13T14:39:38.056806+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:32:20.413346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":412,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:39:38.057815+0000","last_change":"2024-09-13T14:37:48.172088+0000","last_active":"2024-09-13T14:39:38.057815+0000","last_peered":"2024-09-13T14:39:38.057815+0000","last_clean":"2024-09-13T14:39:38.057815+0000","last_became_active":"2024-09-13T14:37:48.171779+0000","last_became_peered":"2024-09-13T14:37:48.171779+0000","last_unstale":"2024-09-13T14:39:38.057815+0000","last_undegraded":"2024-09-13T14:39:38.057815+0000","last_fullsized":"2024-09-13T14:39:38.057815+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:02.852712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":316,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.972485+0000","last_change":"2024-09-13T14:38:53.890236+0000","last_active":"2024-09-13T14:39:24.972485+0000","last_peered":"2024-09-13T14:39:24.972485+0000","last_clean":"2024-09-13T14:39:24.972485+0000","last_became_active":"2024-09-13T14:38:53.889582+0000","last_became_peered":"2024-09-13T14:38:53.889582+0000","last_unstale":"2024-09-13T14:39:24.972485+0000","last_undegraded":"2024-09-13T14:39:24.972485+0000","last_fullsized":"2024-09-13T14:39:24.972485+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:14:29.334774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":399,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:39:38.057273+0000","last_change":"2024-09-13T14:38:03.534097+0000","last_active":"2024-09-13T14:39:38.057273+0000","last_peered":"2024-09-13T14:39:38.057273+0000","last_clean":"2024-09-13T14:39:38.057273+0000","last_became_active":"2024-09-13T14:38:03.533794+0000","last_became_peered":"2024-09-13T14:38:03.533794+0000","last_unstale":"2024-09-13T14:39:38.057273+0000","last_undegraded":"2024-09-13T14:39:38.057273+0000","last_fullsized":"2024-09-13T14:39:38.057273+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:23.272098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":44,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:39:38.016660+0000","last_change":"2024-09-13T14:38:03.527021+0000","last_active":"2024-09-13T14:39:38.016660+0000","last_peered":"2024-09-13T14:39:38.016660+0000","last_clean":"2024-09-13T14:39:38.016660+0000","last_became_active":"2024-09-13T14:38:03.526698+0000","last_became_peered":"2024-09-13T14:38:03.526698+0000","last_unstale":"2024-09-13T14:39:38.016660+0000","last_undegraded":"2024-09-13T14:39:38.016660+0000","last_fullsized":"2024-09-13T14:39:38.016660+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:01:35.734086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":260,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.972768+0000","last_change":"2024-09-13T14:27:12.834931+0000","last_active":"2024-09-13T14:39:24.972768+0000","last_peered":"2024-09-13T14:39:24.972768+0000","last_clean":"2024-09-13T14:39:24.972768+0000","last_became_active":"2024-09-13T14:27:12.834572+0000","last_became_peered":"2024-09-13T14:27:12.834572+0000","last_unstale":"2024-09-13T14:39:24.972768+0000","last_undegraded":"2024-09-13T14:39:24.972768+0000","last_fullsized":"2024-09-13T14:39:24.972768+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:54:21.155689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":73,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.978871+0000","last_change":"2024-09-13T14:37:49.176796+0000","last_active":"2024-09-13T14:39:24.978871+0000","last_peered":"2024-09-13T14:39:24.978871+0000","last_clean":"2024-09-13T14:39:24.978871+0000","last_became_active":"2024-09-13T14:37:49.176246+0000","last_became_peered":"2024-09-13T14:37:49.176246+0000","last_unstale":"2024-09-13T14:39:24.978871+0000","last_undegraded":"2024-09-13T14:39:24.978871+0000","last_fullsized":"2024-09-13T14:39:24.978871+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:29:24.936718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":271,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.972378+0000","last_change":"2024-09-13T14:38:03.526333+0000","last_active":"2024-09-13T14:39:24.972378+0000","last_peered":"2024-09-13T14:39:24.972378+0000","last_clean":"2024-09-13T14:39:24.972378+0000","last_became_active":"2024-09-13T14:38:03.526001+0000","last_became_peered":"2024-09-13T14:38:03.526001+0000","last_unstale":"2024-09-13T14:39:24.972378+0000","last_undegraded":"2024-09-13T14:39:24.972378+0000","last_fullsized":"2024-09-13T14:39:24.972378+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:25:25.181271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":73,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.978727+0000","last_change":"2024-09-13T14:37:49.149582+0000","last_active":"2024-09-13T14:39:24.978727+0000","last_peered":"2024-09-13T14:39:24.978727+0000","last_clean":"2024-09-13T14:39:24.978727+0000","last_became_active":"2024-09-13T14:37:49.148525+0000","last_became_peered":"2024-09-13T14:37:49.148525+0000","last_unstale":"2024-09-13T14:39:24.978727+0000","last_undegraded":"2024-09-13T14:39:24.978727+0000","last_fullsized":"2024-09-13T14:39:24.978727+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:48.655665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":445,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:39:38.056902+0000","last_change":"2024-09-13T14:38:03.535255+0000","last_active":"2024-09-13T14:39:38.056902+0000","last_peered":"2024-09-13T14:39:38.056902+0000","last_clean":"2024-09-13T14:39:38.056902+0000","last_became_active":"2024-09-13T14:38:03.533917+0000","last_became_peered":"2024-09-13T14:38:03.533917+0000","last_unstale":"2024-09-13T14:39:38.056902+0000","last_undegraded":"2024-09-13T14:39:38.056902+0000","last_fullsized":"2024-09-13T14:39:38.056902+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:33.559788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":408,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:39:38.058706+0000","last_change":"2024-09-13T14:37:48.146784+0000","last_active":"2024-09-13T14:39:38.058706+0000","last_peered":"2024-09-13T14:39:38.058706+0000","last_clean":"2024-09-13T14:39:38.058706+0000","last_became_active":"2024-09-13T14:37:48.146491+0000","last_became_peered":"2024-09-13T14:37:48.146491+0000","last_unstale":"2024-09-13T14:39:38.058706+0000","last_undegraded":"2024-09-13T14:39:38.058706+0000","last_fullsized":"2024-09-13T14:39:38.058706+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:30:54.772513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":251,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-13T14:39:24.972928+0000","last_change":"2024-09-13T14:37:48.127607+0000","last_active":"2024-09-13T14:39:24.972928+0000","last_peered":"2024-09-13T14:39:24.972928+0000","last_clean":"2024-09-13T14:39:24.972928+0000","last_became_active":"2024-09-13T14:37:48.127316+0000","last_became_peered":"2024-09-13T14:37:48.127316+0000","last_unstale":"2024-09-13T14:39:24.972928+0000","last_undegraded":"2024-09-13T14:39:24.972928+0000","last_fullsized":"2024-09-13T14:39:24.972928+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:53:37.678861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":50,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:39:38.018016+0000","last_change":"2024-09-13T14:38:03.563200+0000","last_active":"2024-09-13T14:39:38.018016+0000","last_peered":"2024-09-13T14:39:38.018016+0000","last_clean":"2024-09-13T14:39:38.018016+0000","last_became_active":"2024-09-13T14:38:03.534258+0000","last_became_peered":"2024-09-13T14:38:03.534258+0000","last_unstale":"2024-09-13T14:39:38.018016+0000","last_undegraded":"2024-09-13T14:39:38.018016+0000","last_fullsized":"2024-09-13T14:39:38.018016+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:49:07.814419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":6291587096,"num_objects":96003,"num_object_clones":0,"num_object_copies":288009,"num_objects_missing_on_primary":3952,"num_objects_missing":3952,"num_objects_degraded":42341,"num_objects_misplaced":1987,"num_objects_unfound":0,"num_objects_dirty":96003,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129623,"num_write_kb":7219969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51498,"num_bytes_recovered":3347775488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19503980544,"data_stored":19503972400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89846,"ondisk_log_size":89846,"up":48,"acting":41,"num_store_stats":6},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1538,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":17412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25132,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":7,"num_objects_missing":7,"num_objects_degraded":16,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16793600,"data_stored":16777292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":23,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234351,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3489296,"kb_used_data":3043080,"kb_used_omap":1,"kb_used_meta":446206,"kb_avail":90255856,"statfs":{"total":95995035648,"available":92421996544,"internally_reserved":0,"allocated":3116113920,"data_stored":3114695374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":456915403},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":239,"seq":1026497183770,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2056980,"kb_used_data":1918348,"kb_used_omap":8,"kb_used_meta":138615,"kb_avail":91688172,"statfs":{"total":95995035648,"available":93888688128,"internally_reserved":0,"allocated":1964388352,"data_stored":1962966801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8453,"internal_metadata":141942523},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757617,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2494532,"kb_used_data":1877692,"kb_used_omap":1,"kb_used_meta":616830,"kb_avail":91250620,"statfs":{"total":95995035648,"available":93440634880,"internally_reserved":0,"allocated":1922756608,"data_stored":1921346071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":631634374},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1,2,2,2,6,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019260,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3792912,"kb_used_data":3216264,"kb_used_omap":1,"kb_used_meta":576638,"kb_avail":89952240,"statfs":{"total":95995035648,"available":92111093760,"internally_reserved":0,"allocated":3293454336,"data_stored":3292036322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":590477766},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435443,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5174024,"kb_used_data":4628672,"kb_used_omap":5,"kb_used_meta":545338,"kb_avail":88571128,"statfs":{"total":95995035648,"available":90696835072,"internally_reserved":0,"allocated":4739760128,"data_stored":4738345825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":558426238},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":446676598972,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":425201762498,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926022,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5080844,"kb_used_data":4409156,"kb_used_omap":7,"kb_used_meta":671672,"kb_avail":88664308,"statfs":{"total":95995035648,"available":90792251392,"internally_reserved":0,"allocated":4514975744,"data_stored":4513557402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":687792424},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6860,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4504293376,"data_stored":4504289304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4728950784,"data_stored":4728946712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3286958080,"data_stored":3286958080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1914044416,"data_stored":1914044416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1957494784,"data_stored":1957494784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3112239104,"data_stored":3112239104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:39:53.271 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:53.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:53.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:53.373 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:53.398 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:53.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:53.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:53.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:53 smithi195 ceph-mon[104262]: pgmap v847: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 76 active+clean; 5.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 155 op/s; 42359/288093 objects degraded (14.703%); 1989/288093 objects misplaced (0.690%); 8.2 MiB/s, 131 objects/s recovering 2024-09-13T14:39:53.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:53 smithi195 ceph-mon[104262]: from='client.45581 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:39:53.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:53 smithi195 ceph-mon[104262]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:39:53.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:53 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:39:53.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:53 smithi195 ceph-mon[104262]: osdmap e262: 8 total, 8 up, 6 in 2024-09-13T14:39:53.500 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:53.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:53.527 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:53.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:53.628 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:53.656 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:53.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:53.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:53.757 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:53.781 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:53.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:53.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:53.882 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:53.911 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:53.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:53.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:54.013 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:54.035 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:54.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:54.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:54.137 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:54.204 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:54.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:54.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:54.306 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:54.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:54.329 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:54.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:54.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:54 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 42359/288093 objects degraded (14.703%), 13 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:54.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:54 smithi102 ceph-mon[110183]: osdmap e263: 8 total, 8 up, 6 in 2024-09-13T14:39:54.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:54 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 42359/288093 objects degraded (14.703%), 13 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:54.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:54 smithi102 ceph-mon[115269]: osdmap e263: 8 total, 8 up, 6 in 2024-09-13T14:39:54.431 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:54.453 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:54.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:54.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:54.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:54 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 42359/288093 objects degraded (14.703%), 13 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:54.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:54 smithi195 ceph-mon[104262]: osdmap e263: 8 total, 8 up, 6 in 2024-09-13T14:39:54.555 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:54.579 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:54.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:54.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:54.683 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:54.712 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:54.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:54.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:54.815 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:54.837 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:54.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:54.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:54.939 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:54.968 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:54.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:54.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:55.069 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:55.092 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:55.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:55.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:55.107 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:55 smithi102 ceph-mon[110183]: pgmap v849: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 76 active+clean; 5.8 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 161 op/s; 41110/286212 objects degraded (14.363%); 1989/286212 objects misplaced (0.695%); 13 MiB/s, 202 objects/s recovering 2024-09-13T14:39:55.107 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:55 smithi102 ceph-mon[110183]: osdmap e264: 8 total, 8 up, 6 in 2024-09-13T14:39:55.107 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:55 smithi102 ceph-mon[115269]: pgmap v849: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 76 active+clean; 5.8 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 161 op/s; 41110/286212 objects degraded (14.363%); 1989/286212 objects misplaced (0.695%); 13 MiB/s, 202 objects/s recovering 2024-09-13T14:39:55.107 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:55 smithi102 ceph-mon[115269]: osdmap e264: 8 total, 8 up, 6 in 2024-09-13T14:39:55.194 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:55.217 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:55.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:55.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:55.319 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:55.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:55.345 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:55.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:55.446 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:55.468 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:55.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:55.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:55.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:55 smithi195 ceph-mon[104262]: pgmap v849: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 76 active+clean; 5.8 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 161 op/s; 41110/286212 objects degraded (14.363%); 1989/286212 objects misplaced (0.695%); 13 MiB/s, 202 objects/s recovering 2024-09-13T14:39:55.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:55 smithi195 ceph-mon[104262]: osdmap e264: 8 total, 8 up, 6 in 2024-09-13T14:39:55.519 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:39:55.519 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:39:55.547 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:39:55.547 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:39:55.570 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:55.574 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:39:55.575 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:39:55.597 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:55.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:55.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:55.602 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:39:55.602 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:39:55.620 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:39:55.621 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:39:55.647 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:39:55.647 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:39:55.666 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:39:55.666 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:39:55.691 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:39:55.692 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:39:55.700 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:55.724 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:55.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:55.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:55.826 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:55.848 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:55.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:55.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:55.950 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:39:55.977 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:55.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:55.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:39:56.079 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:56.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:56.109 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:56.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:56.139 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:39:56.210 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:56.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:56.234 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:56.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:56.271 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:39:56.335 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:56.357 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:56.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:56.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:56.459 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:56.486 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:56.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:56.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:56.587 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:39:56.609 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:56.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:56.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:39:56.630 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:39:56.630 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:39:56.630 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:56 smithi102 ceph-mon[110183]: pgmap v852: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 76 active+clean; 5.8 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 128 op/s; 41110/286212 objects degraded (14.363%); 1989/286212 objects misplaced (0.695%); 12 MiB/s, 198 objects/s recovering 2024-09-13T14:39:56.630 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:56 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/669988667' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:39:56.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:39:56.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:39:56.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:56 smithi102 ceph-mon[115269]: pgmap v852: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 76 active+clean; 5.8 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 128 op/s; 41110/286212 objects degraded (14.363%); 1989/286212 objects misplaced (0.695%); 12 MiB/s, 198 objects/s recovering 2024-09-13T14:39:56.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:56 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/669988667' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:39:56.631 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:39:56 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:39:56] "GET /metrics HTTP/1.1" 200 36541 "" "Prometheus/2.51.0" 2024-09-13T14:39:56.711 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:56.739 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:56.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:56.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:56.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:39:56.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:39:56.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:56 smithi195 ceph-mon[104262]: pgmap v852: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 76 active+clean; 5.8 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 128 op/s; 41110/286212 objects degraded (14.363%); 1989/286212 objects misplaced (0.695%); 12 MiB/s, 198 objects/s recovering 2024-09-13T14:39:56.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:56 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/669988667' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:39:56.841 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:56.862 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:56.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:56.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:56.964 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:56.991 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:56.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:56.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:57.093 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:57.119 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:57.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:57.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:57.220 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:57.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:57.246 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:57.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:57.347 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:57.385 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:57.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:57.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:57.486 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:57.509 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:57.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:57.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:57.611 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:57.623 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:39:57.634 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:57.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:57.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:57.736 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:57.758 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:57.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:57.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:57.860 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:57.883 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:57.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:57.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:57.985 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:58.010 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:58.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:58.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:58.112 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:58.125 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:39:58.125 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:39:58.139 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:58.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:58.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:58.241 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:58.265 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:58.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:58.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:58.334 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:39:58.367 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:39:58.390 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:58.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:58.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:39:58.492 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:58.515 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:58.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:58.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:58.617 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:58.638 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:58.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:58.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:58.740 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:58.762 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:58.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:58.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:58.864 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:58.874 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:58 smithi102 ceph-mon[110183]: pgmap v853: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 76 active+clean; 5.8 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 147 op/s; 41056/285438 objects degraded (14.384%); 1989/285438 objects misplaced (0.697%); 12 MiB/s, 198 objects/s recovering 2024-09-13T14:39:58.874 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:58 smithi102 ceph-mon[115269]: pgmap v853: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 76 active+clean; 5.8 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 147 op/s; 41056/285438 objects degraded (14.384%); 1989/285438 objects misplaced (0.697%); 12 MiB/s, 198 objects/s recovering 2024-09-13T14:39:58.893 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:58.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:58.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:58.907 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T14:39:58.995 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:39:58.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:58 smithi195 ceph-mon[104262]: pgmap v853: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 76 active+clean; 5.8 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 147 op/s; 41056/285438 objects degraded (14.384%); 1989/285438 objects misplaced (0.697%); 12 MiB/s, 198 objects/s recovering 2024-09-13T14:39:59.017 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:59.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:59.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:39:59.119 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:59.143 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:59.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:59.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:59.244 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:39:59.269 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:59.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:59.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:39:59.313 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:39:59.371 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:59.393 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:59.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:59.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:59.495 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:39:59.518 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:59.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:59.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:39:59.620 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:59.644 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:59.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:59.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:59.746 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:39:59.772 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:39:59.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:59.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:39:59.874 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:39:59.896 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:39:59.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:39:59.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:39:59.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:39:59 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 41056/285438 objects degraded (14.384%), 13 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:59.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:39:59 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 41056/285438 objects degraded (14.384%), 13 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:59.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:39:59 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 41056/285438 objects degraded (14.384%), 13 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-09-13T14:39:59.998 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:00.023 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:00.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:00.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:00.125 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:00.161 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:00.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:00.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:00.263 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:00.284 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:00.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:00.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:00.386 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:00.408 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:00.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:00.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:00.510 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:00.535 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:00.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:00.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:00.606 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:00.611 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:00.637 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:00.662 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:00.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:00.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:00.764 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:00.788 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:00.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:00.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:00.890 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:00.914 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:00.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:00.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:01.016 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:01.041 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:01.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:01.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:01.057 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:00 smithi102 ceph-mon[110183]: pgmap v854: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.8 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 204 op/s; 40418/283509 objects degraded (14.256%); 1990/283509 objects misplaced (0.702%); 15 MiB/s, 246 objects/s recovering 2024-09-13T14:40:01.057 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:00 smithi102 ceph-mon[110183]: Health detail: HEALTH_WARN Degraded data redundancy: 40418/283509 objects degraded (14.256%), 12 pgs degraded, 7 pgs undersized 2024-09-13T14:40:01.057 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:00 smithi102 ceph-mon[110183]: [WRN] PG_DEGRADED: Degraded data redundancy: 40418/283509 objects degraded (14.256%), 12 pgs degraded, 7 pgs undersized 2024-09-13T14:40:01.057 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:00 smithi102 ceph-mon[110183]: pg 1.0 is active+recovery_wait+degraded+remapped, acting [3,0,6] 2024-09-13T14:40:01.057 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:00 smithi102 ceph-mon[110183]: pg 2.3 is stuck undersized for 64s, current state active+recovery_wait+undersized+degraded+remapped, last acting [5,3] 2024-09-13T14:40:01.057 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:00 smithi102 ceph-mon[110183]: pg 2.7 is active+recovery_wait+degraded, acting [7,6,3] 2024-09-13T14:40:01.058 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:00 smithi102 ceph-mon[110183]: pg 7.0 is stuck undersized for 2m, current state active+recovery_wait+undersized+degraded+remapped, last acting [5,3] 2024-09-13T14:40:01.058 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:00 smithi102 ceph-mon[110183]: pg 7.1 is stuck undersized for 2m, current state active+recovery_wait+undersized+degraded+remapped, last acting [4,3] 2024-09-13T14:40:01.058 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:00 smithi102 ceph-mon[110183]: pg 7.3 is stuck undersized for 2m, current state active+undersized+degraded+remapped+backfill_wait, last acting [3,4] 2024-09-13T14:40:01.058 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:00 smithi102 ceph-mon[110183]: pg 7.4 is active+recovery_wait+degraded, acting [0,4,6] 2024-09-13T14:40:01.058 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:00 smithi102 ceph-mon[110183]: pg 7.6 is active+recovery_wait+degraded+remapped, acting [0,3,5] 2024-09-13T14:40:01.058 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:00 smithi102 ceph-mon[110183]: pg 7.7 is active+recovery_wait+degraded, acting [0,6,3] 2024-09-13T14:40:01.058 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:00 smithi102 ceph-mon[110183]: pg 7.b is stuck undersized for 2m, current state active+recovery_wait+undersized+degraded+remapped, last acting [5,4] 2024-09-13T14:40:01.058 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:00 smithi102 ceph-mon[110183]: pg 7.d is stuck undersized for 2m, current state active+recovering+undersized+degraded+remapped, last acting [0,3] 2024-09-13T14:40:01.058 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:00 smithi102 ceph-mon[110183]: pg 7.e is stuck undersized for 114s, current state active+recovery_wait+undersized+degraded+remapped, last acting [5,0] 2024-09-13T14:40:01.059 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:00 smithi102 ceph-mon[115269]: pgmap v854: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.8 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 204 op/s; 40418/283509 objects degraded (14.256%); 1990/283509 objects misplaced (0.702%); 15 MiB/s, 246 objects/s recovering 2024-09-13T14:40:01.059 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:00 smithi102 ceph-mon[115269]: Health detail: HEALTH_WARN Degraded data redundancy: 40418/283509 objects degraded (14.256%), 12 pgs degraded, 7 pgs undersized 2024-09-13T14:40:01.060 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:00 smithi102 ceph-mon[115269]: [WRN] PG_DEGRADED: Degraded data redundancy: 40418/283509 objects degraded (14.256%), 12 pgs degraded, 7 pgs undersized 2024-09-13T14:40:01.060 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:00 smithi102 ceph-mon[115269]: pg 1.0 is active+recovery_wait+degraded+remapped, acting [3,0,6] 2024-09-13T14:40:01.060 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:00 smithi102 ceph-mon[115269]: pg 2.3 is stuck undersized for 64s, current state active+recovery_wait+undersized+degraded+remapped, last acting [5,3] 2024-09-13T14:40:01.060 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:00 smithi102 ceph-mon[115269]: pg 2.7 is active+recovery_wait+degraded, acting [7,6,3] 2024-09-13T14:40:01.060 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:00 smithi102 ceph-mon[115269]: pg 7.0 is stuck undersized for 2m, current state active+recovery_wait+undersized+degraded+remapped, last acting [5,3] 2024-09-13T14:40:01.060 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:00 smithi102 ceph-mon[115269]: pg 7.1 is stuck undersized for 2m, current state active+recovery_wait+undersized+degraded+remapped, last acting [4,3] 2024-09-13T14:40:01.060 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:00 smithi102 ceph-mon[115269]: pg 7.3 is stuck undersized for 2m, current state active+undersized+degraded+remapped+backfill_wait, last acting [3,4] 2024-09-13T14:40:01.060 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:00 smithi102 ceph-mon[115269]: pg 7.4 is active+recovery_wait+degraded, acting [0,4,6] 2024-09-13T14:40:01.060 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:00 smithi102 ceph-mon[115269]: pg 7.6 is active+recovery_wait+degraded+remapped, acting [0,3,5] 2024-09-13T14:40:01.060 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:00 smithi102 ceph-mon[115269]: pg 7.7 is active+recovery_wait+degraded, acting [0,6,3] 2024-09-13T14:40:01.060 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:00 smithi102 ceph-mon[115269]: pg 7.b is stuck undersized for 2m, current state active+recovery_wait+undersized+degraded+remapped, last acting [5,4] 2024-09-13T14:40:01.060 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:00 smithi102 ceph-mon[115269]: pg 7.d is stuck undersized for 2m, current state active+recovering+undersized+degraded+remapped, last acting [0,3] 2024-09-13T14:40:01.061 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:00 smithi102 ceph-mon[115269]: pg 7.e is stuck undersized for 114s, current state active+recovery_wait+undersized+degraded+remapped, last acting [5,0] 2024-09-13T14:40:01.143 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:01.165 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:01.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:01.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:01.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:00 smithi195 ceph-mon[104262]: pgmap v854: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.8 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 204 op/s; 40418/283509 objects degraded (14.256%); 1990/283509 objects misplaced (0.702%); 15 MiB/s, 246 objects/s recovering 2024-09-13T14:40:01.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:00 smithi195 ceph-mon[104262]: Health detail: HEALTH_WARN Degraded data redundancy: 40418/283509 objects degraded (14.256%), 12 pgs degraded, 7 pgs undersized 2024-09-13T14:40:01.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:00 smithi195 ceph-mon[104262]: [WRN] PG_DEGRADED: Degraded data redundancy: 40418/283509 objects degraded (14.256%), 12 pgs degraded, 7 pgs undersized 2024-09-13T14:40:01.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:00 smithi195 ceph-mon[104262]: pg 1.0 is active+recovery_wait+degraded+remapped, acting [3,0,6] 2024-09-13T14:40:01.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:00 smithi195 ceph-mon[104262]: pg 2.3 is stuck undersized for 64s, current state active+recovery_wait+undersized+degraded+remapped, last acting [5,3] 2024-09-13T14:40:01.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:00 smithi195 ceph-mon[104262]: pg 2.7 is active+recovery_wait+degraded, acting [7,6,3] 2024-09-13T14:40:01.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:00 smithi195 ceph-mon[104262]: pg 7.0 is stuck undersized for 2m, current state active+recovery_wait+undersized+degraded+remapped, last acting [5,3] 2024-09-13T14:40:01.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:00 smithi195 ceph-mon[104262]: pg 7.1 is stuck undersized for 2m, current state active+recovery_wait+undersized+degraded+remapped, last acting [4,3] 2024-09-13T14:40:01.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:00 smithi195 ceph-mon[104262]: pg 7.3 is stuck undersized for 2m, current state active+undersized+degraded+remapped+backfill_wait, last acting [3,4] 2024-09-13T14:40:01.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:00 smithi195 ceph-mon[104262]: pg 7.4 is active+recovery_wait+degraded, acting [0,4,6] 2024-09-13T14:40:01.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:00 smithi195 ceph-mon[104262]: pg 7.6 is active+recovery_wait+degraded+remapped, acting [0,3,5] 2024-09-13T14:40:01.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:00 smithi195 ceph-mon[104262]: pg 7.7 is active+recovery_wait+degraded, acting [0,6,3] 2024-09-13T14:40:01.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:00 smithi195 ceph-mon[104262]: pg 7.b is stuck undersized for 2m, current state active+recovery_wait+undersized+degraded+remapped, last acting [5,4] 2024-09-13T14:40:01.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:00 smithi195 ceph-mon[104262]: pg 7.d is stuck undersized for 2m, current state active+recovering+undersized+degraded+remapped, last acting [0,3] 2024-09-13T14:40:01.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:00 smithi195 ceph-mon[104262]: pg 7.e is stuck undersized for 114s, current state active+recovery_wait+undersized+degraded+remapped, last acting [5,0] 2024-09-13T14:40:01.267 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:01.295 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:01.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:01.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:01.397 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:01.419 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:01.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:01.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:01.520 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:01.546 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:01.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:01.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:01.648 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:01.675 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:01.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:01.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:01.777 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:01.805 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:01.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:01.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:01.907 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:01.950 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:01.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:01.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:02.053 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:02.082 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:02.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:02.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:02.184 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:02.206 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:02.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:02.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:02.308 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:02.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:02.333 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:02.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:02.434 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:02.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:02.463 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:02.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:02.564 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:02.587 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:02.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:02.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:02.689 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:02.712 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:02.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:02.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:02.814 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:02.837 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:02.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:02.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:02.939 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:02.960 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:02.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:02.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:03.062 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:03.086 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:03.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:03.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:03.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:02 smithi102 ceph-mon[110183]: pgmap v855: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.8 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 112 op/s; 40418/283509 objects degraded (14.256%); 1990/283509 objects misplaced (0.702%); 5.0 MiB/s, 88 objects/s recovering 2024-09-13T14:40:03.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:02 smithi102 ceph-mon[115269]: pgmap v855: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.8 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 112 op/s; 40418/283509 objects degraded (14.256%); 1990/283509 objects misplaced (0.702%); 5.0 MiB/s, 88 objects/s recovering 2024-09-13T14:40:03.189 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:03.214 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:03.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:03.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:03.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:02 smithi195 ceph-mon[104262]: pgmap v855: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.8 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 112 op/s; 40418/283509 objects degraded (14.256%); 1990/283509 objects misplaced (0.702%); 5.0 MiB/s, 88 objects/s recovering 2024-09-13T14:40:03.316 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:03.341 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:03.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:03.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:03.443 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:03.465 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:03.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:03.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:03.567 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:03.591 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:03.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:03.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:03.692 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:03.716 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:03.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:03.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:03.818 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:03.841 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:03.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:03.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:03.943 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:03.966 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:03.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:03.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:04.068 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:04.092 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:04.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:04.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:04.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:03 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 40418/283509 objects degraded (14.256%), 12 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:04.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:03 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 40418/283509 objects degraded (14.256%), 12 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:04.194 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:04.219 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:04.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:04.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:04.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:03 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 40418/283509 objects degraded (14.256%), 12 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:04.321 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:04.345 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:04.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:04.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:04.447 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:04.474 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:04.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:04.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:04.576 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:04.599 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:04.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:04.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:04.700 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:04.723 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:04.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:04.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:04.825 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:04.847 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:04.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:04.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:04.949 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:04.972 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:04.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:04.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:05.074 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:05.101 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:05.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:05.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:05.203 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:05.230 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:05.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:05.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:05.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:04 smithi195 ceph-mon[104262]: pgmap v856: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.7 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 202 op/s; 39872/280386 objects degraded (14.220%); 1989/280386 objects misplaced (0.709%); 7.3 MiB/s, 123 objects/s recovering 2024-09-13T14:40:05.332 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:05.357 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:05.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:05.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:05.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:04 smithi102 ceph-mon[110183]: pgmap v856: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.7 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 202 op/s; 39872/280386 objects degraded (14.220%); 1989/280386 objects misplaced (0.709%); 7.3 MiB/s, 123 objects/s recovering 2024-09-13T14:40:05.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:04 smithi102 ceph-mon[115269]: pgmap v856: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.7 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 202 op/s; 39872/280386 objects degraded (14.220%); 1989/280386 objects misplaced (0.709%); 7.3 MiB/s, 123 objects/s recovering 2024-09-13T14:40:05.459 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:05.480 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:05.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:05.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:05.582 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:05.604 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:05.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:05.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:05.708 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:05.732 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:05.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:05.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:05.834 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:05.855 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:05.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:05.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:05.957 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:05.983 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:05.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:05.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:06.085 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:06.109 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:06.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:06.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:06.211 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:06.232 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:06.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:06.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:06.334 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:06.360 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:06.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:06.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:06.462 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:06.488 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:06.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:06.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:06.590 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:06.612 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:06.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:06.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:06.702 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:40:06 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:40:06] "GET /metrics HTTP/1.1" 200 36541 "" "Prometheus/2.51.0" 2024-09-13T14:40:06.715 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:06.738 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:06.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:06.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:06.840 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:06.869 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:06.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:06.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:06.971 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:06.999 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:06.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:07.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:07.101 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:07.123 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:07.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:07.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:07.225 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:07.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:06 smithi195 ceph-mon[104262]: pgmap v857: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.7 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 183 op/s; 39872/280386 objects degraded (14.220%); 1989/280386 objects misplaced (0.709%); 6.6 MiB/s, 111 objects/s recovering 2024-09-13T14:40:07.250 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:07.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:07.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:07.260 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:06 smithi102 ceph-mon[110183]: pgmap v857: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.7 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 183 op/s; 39872/280386 objects degraded (14.220%); 1989/280386 objects misplaced (0.709%); 6.6 MiB/s, 111 objects/s recovering 2024-09-13T14:40:07.260 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:06 smithi102 ceph-mon[115269]: pgmap v857: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.7 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 183 op/s; 39872/280386 objects degraded (14.220%); 1989/280386 objects misplaced (0.709%); 6.6 MiB/s, 111 objects/s recovering 2024-09-13T14:40:07.353 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:07.375 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:07.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:07.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:07.476 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:07.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:07.501 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:07.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:07.602 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:07.624 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:07.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:07.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:07.726 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:07.750 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:07.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:07.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:07.854 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:07.879 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:07.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:07.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:07.981 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:08.005 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:08.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:08.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:08.107 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:08.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:08.130 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:08.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:08.231 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:08.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:07 smithi195 ceph-mon[104262]: from='client.55258 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:40:08.256 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:08.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:08.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:08.358 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:08.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:08.382 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:08.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:08.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:07 smithi102 ceph-mon[110183]: from='client.55258 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:40:08.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:07 smithi102 ceph-mon[115269]: from='client.55258 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:40:08.483 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:08.505 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:08.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:08.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:08.607 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:08.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:08.631 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:08.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:08.732 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:08.755 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:08.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:08.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:08.857 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:08.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:08.886 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:08.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:08.987 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:09.016 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:09.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:09.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:09.118 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:09.143 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:09.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:09.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:09.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:08 smithi102 ceph-mon[110183]: pgmap v858: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.7 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 208 op/s; 39762/278697 objects degraded (14.267%); 1989/278697 objects misplaced (0.714%); 6.1 MiB/s, 103 objects/s recovering 2024-09-13T14:40:09.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:08 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 39762/278697 objects degraded (14.267%), 12 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:09.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:08 smithi102 ceph-mon[115269]: pgmap v858: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.7 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 208 op/s; 39762/278697 objects degraded (14.267%); 1989/278697 objects misplaced (0.714%); 6.1 MiB/s, 103 objects/s recovering 2024-09-13T14:40:09.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:08 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 39762/278697 objects degraded (14.267%), 12 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:09.245 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:09.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:08 smithi195 ceph-mon[104262]: pgmap v858: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.7 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 208 op/s; 39762/278697 objects degraded (14.267%); 1989/278697 objects misplaced (0.714%); 6.1 MiB/s, 103 objects/s recovering 2024-09-13T14:40:09.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:08 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 39762/278697 objects degraded (14.267%), 12 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:09.269 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:09.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:09.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:09.370 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:09.393 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:09.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:09.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:09.496 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:09.518 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:09.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:09.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:09.620 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:09.644 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:09.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:09.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:09.746 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:09.772 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:09.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:09.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:09.874 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:09.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:09.900 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:09.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:10.001 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:10.024 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:10.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:10.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:10.126 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:10.149 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:10.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:10.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:10.251 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:10.275 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:10.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:10.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:10.377 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:10.401 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:10.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:10.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:10.503 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:10.528 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:10.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:10.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:10.630 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:10.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:10.654 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:10.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:10.755 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:10.776 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:10.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:10.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:10.878 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:10.903 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:10.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:10.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:11.004 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:11.031 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:11.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:11.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:11.133 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:11.158 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:11.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:11.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:11.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:10 smithi195 ceph-mon[104262]: pgmap v859: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.6 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 221 op/s; 39298/277467 objects degraded (14.163%); 1989/277467 objects misplaced (0.717%); 8.2 MiB/s, 137 objects/s recovering 2024-09-13T14:40:11.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:40:11.260 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:11.283 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:11.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:11.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:11.384 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:11.411 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:11.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:11.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:11.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:10 smithi102 ceph-mon[110183]: pgmap v859: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.6 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 221 op/s; 39298/277467 objects degraded (14.163%); 1989/277467 objects misplaced (0.717%); 8.2 MiB/s, 137 objects/s recovering 2024-09-13T14:40:11.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:40:11.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:10 smithi102 ceph-mon[115269]: pgmap v859: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.6 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 221 op/s; 39298/277467 objects degraded (14.163%); 1989/277467 objects misplaced (0.717%); 8.2 MiB/s, 137 objects/s recovering 2024-09-13T14:40:11.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:40:11.513 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:11.535 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:11.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:11.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:11.637 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:11.662 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:11.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:11.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:11.764 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:11.786 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:11.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:11.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:11.888 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:11.913 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:11.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:11.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:12.015 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:12.039 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:12.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:12.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:12.140 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:12.163 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:12.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:12.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:12.265 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:12.287 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:12.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:12.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:12.389 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:12.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:12.412 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:12.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:12.513 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:12.539 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:12.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:12.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:12.641 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:12.664 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:12.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:12.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:12.766 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:12.790 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:12.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:12.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:12.892 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:12.914 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:12.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:12.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:13.016 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:13.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:13.039 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:13.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:13.140 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:13.163 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:13.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:13.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:13.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:12 smithi195 ceph-mon[104262]: pgmap v860: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.6 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 167 op/s; 39298/277467 objects degraded (14.163%); 1989/277467 objects misplaced (0.717%); 5.2 MiB/s, 83 objects/s recovering 2024-09-13T14:40:13.265 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:13.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:13.288 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:13.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:13.389 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:13.410 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:13.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:13.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:13.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:12 smithi102 ceph-mon[110183]: pgmap v860: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.6 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 167 op/s; 39298/277467 objects degraded (14.163%); 1989/277467 objects misplaced (0.717%); 5.2 MiB/s, 83 objects/s recovering 2024-09-13T14:40:13.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:12 smithi102 ceph-mon[115269]: pgmap v860: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.6 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 167 op/s; 39298/277467 objects degraded (14.163%); 1989/277467 objects misplaced (0.717%); 5.2 MiB/s, 83 objects/s recovering 2024-09-13T14:40:13.513 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:13.536 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:13.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:13.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:13.638 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:13.665 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:13.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:13.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:13.767 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:13.790 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:13.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:13.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:13.892 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:13.917 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:13.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:13.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:14.020 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:14.043 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:14.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:14.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:14.145 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:14.170 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:14.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:14.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:14.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:13 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 39298/277467 objects degraded (14.163%), 12 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:14.271 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:14.294 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:14.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:14.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:14.396 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:14.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:13 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 39298/277467 objects degraded (14.163%), 12 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:14.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:13 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 39298/277467 objects degraded (14.163%), 12 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:14.420 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:14.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:14.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:14.522 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:14.544 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:14.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:14.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:14.646 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:14.672 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:14.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:14.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:14.774 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:14.799 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:14.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:14.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:14.901 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:14.925 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:14.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:14.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:15.027 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:15.052 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:15.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:15.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:15.082 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:14 smithi102 ceph-mon[110183]: pgmap v861: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.6 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 235 op/s; 38637/275028 objects degraded (14.048%); 1990/275028 objects misplaced (0.724%); 8.3 MiB/s, 132 objects/s recovering 2024-09-13T14:40:15.082 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:14 smithi102 ceph-mon[115269]: pgmap v861: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.6 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 235 op/s; 38637/275028 objects degraded (14.048%); 1990/275028 objects misplaced (0.724%); 8.3 MiB/s, 132 objects/s recovering 2024-09-13T14:40:15.154 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:15.181 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:15.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:15.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:15.244 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:40:15.247 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:40:15.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:14 smithi195 ceph-mon[104262]: pgmap v861: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.6 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 235 op/s; 38637/275028 objects degraded (14.048%); 1990/275028 objects misplaced (0.724%); 8.3 MiB/s, 132 objects/s recovering 2024-09-13T14:40:15.283 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:15.312 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:15.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:15.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:15.414 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:15.438 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:15.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:15.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:15.540 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:15.570 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:15.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:15.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:15.672 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:15.700 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:15.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:15.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:15.802 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:15.828 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:15.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:15.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:15.930 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:15.953 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:15.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:15.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:15.998 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is set 2024-09-13T14:40:16.055 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:16.080 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:16.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:16.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:16.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:15 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/440727510' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:40:16.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:15 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:40:16.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:15 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:40:16.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:15 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/440727510' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:40:16.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:15 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:40:16.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:15 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:40:16.182 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:16.204 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:16.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:16.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:16.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:15 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/440727510' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:40:16.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:15 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:40:16.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:15 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:40:16.307 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:16.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:16.337 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:16.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:16.438 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:16.462 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:16.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:16.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:16.564 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:16.587 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:16.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:16.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:16.590 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:40:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:40:16] "GET /metrics HTTP/1.1" 200 36540 "" "Prometheus/2.51.0" 2024-09-13T14:40:16.690 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:16.715 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:16.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:16.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:16.817 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:16.843 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:16.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:16.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:16.944 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:16.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:16.970 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:16.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:17.071 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:17.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:17.101 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:17.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:17.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:16 smithi102 ceph-mon[110183]: pgmap v862: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.6 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 148 op/s; 38637/275028 objects degraded (14.048%); 1990/275028 objects misplaced (0.724%); 5.8 MiB/s, 93 objects/s recovering 2024-09-13T14:40:17.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:16 smithi102 ceph-mon[110183]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:40:17.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:16 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:40:17.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:16 smithi102 ceph-mon[110183]: osdmap e265: 8 total, 8 up, 6 in 2024-09-13T14:40:17.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:16 smithi102 ceph-mon[115269]: pgmap v862: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.6 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 148 op/s; 38637/275028 objects degraded (14.048%); 1990/275028 objects misplaced (0.724%); 5.8 MiB/s, 93 objects/s recovering 2024-09-13T14:40:17.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:16 smithi102 ceph-mon[115269]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:40:17.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:16 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:40:17.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:16 smithi102 ceph-mon[115269]: osdmap e265: 8 total, 8 up, 6 in 2024-09-13T14:40:17.201 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:17.226 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:17.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:17.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:17.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:16 smithi195 ceph-mon[104262]: pgmap v862: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.6 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 148 op/s; 38637/275028 objects degraded (14.048%); 1990/275028 objects misplaced (0.724%); 5.8 MiB/s, 93 objects/s recovering 2024-09-13T14:40:17.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:16 smithi195 ceph-mon[104262]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:40:17.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:16 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:40:17.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:16 smithi195 ceph-mon[104262]: osdmap e265: 8 total, 8 up, 6 in 2024-09-13T14:40:17.328 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:17.353 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:17.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:17.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:17.416 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":857,"stamp":"2024-09-13T14:40:05.471795+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6128015357,"num_objects":93462,"num_object_clones":0,"num_object_copies":280386,"num_objects_missing_on_primary":3845,"num_objects_missing":3845,"num_objects_degraded":39872,"num_objects_misplaced":1989,"num_objects_unfound":0,"num_objects_dirty":93462,"num_whiteouts":0,"num_read":6377,"num_read_kb":13834,"num_write":133315,"num_write_kb":7235010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53921,"num_bytes_recovered":3501263989,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92197,"ondisk_log_size":92197,"up":267,"acting":260,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":309,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":21679468,"kb_used_data":18700796,"kb_used_omap":26,"kb_used_meta":2978597,"kb_avail":540791444,"statfs":{"total":575970213888,"available":553770438656,"internally_reserved":0,"allocated":19149615104,"data_stored":19140971251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27144,"internal_metadata":3050083832},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-127270912,"num_objects":-1942,"num_object_clones":0,"num_object_copies":-5826,"num_objects_missing_on_primary":-114,"num_objects_missing":-114,"num_objects_degraded":-1238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1942,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1182,"num_bytes_recovered":73203712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.573931"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":420,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:39:54.904456+0000","last_change":"2024-09-13T14:36:13.859927+0000","last_active":"2024-09-13T14:39:54.904456+0000","last_peered":"2024-09-13T14:39:54.904456+0000","last_clean":"2024-09-13T14:39:54.904456+0000","last_became_active":"2024-09-13T14:36:13.859505+0000","last_became_peered":"2024-09-13T14:36:13.859505+0000","last_unstale":"2024-09-13T14:39:54.904456+0000","last_undegraded":"2024-09-13T14:39:54.904456+0000","last_fullsized":"2024-09-13T14:39:54.904456+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:12:20.989627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":265,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:39:53.898121+0000","last_change":"2024-09-13T14:37:48.130968+0000","last_active":"2024-09-13T14:39:53.898121+0000","last_peered":"2024-09-13T14:39:53.898121+0000","last_clean":"2024-09-13T14:39:53.898121+0000","last_became_active":"2024-09-13T14:37:48.130603+0000","last_became_peered":"2024-09-13T14:37:48.130603+0000","last_unstale":"2024-09-13T14:39:53.898121+0000","last_undegraded":"2024-09-13T14:39:53.898121+0000","last_fullsized":"2024-09-13T14:39:53.898121+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:42.626386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":61,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:39:53.919224+0000","last_change":"2024-09-13T14:36:13.862181+0000","last_active":"2024-09-13T14:39:53.919224+0000","last_peered":"2024-09-13T14:39:53.919224+0000","last_clean":"2024-09-13T14:39:53.919224+0000","last_became_active":"2024-09-13T14:36:13.861873+0000","last_became_peered":"2024-09-13T14:36:13.861873+0000","last_unstale":"2024-09-13T14:39:53.919224+0000","last_undegraded":"2024-09-13T14:39:53.919224+0000","last_fullsized":"2024-09-13T14:39:53.919224+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:14.085700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":57,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:39:54.899610+0000","last_change":"2024-09-13T14:38:53.896672+0000","last_active":"2024-09-13T14:39:54.899610+0000","last_peered":"2024-09-13T14:39:54.899610+0000","last_clean":"2024-09-13T14:39:54.899610+0000","last_became_active":"2024-09-13T14:38:53.896346+0000","last_became_peered":"2024-09-13T14:38:53.896346+0000","last_unstale":"2024-09-13T14:39:54.899610+0000","last_undegraded":"2024-09-13T14:39:54.899610+0000","last_fullsized":"2024-09-13T14:39:54.899610+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:57.789686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":57,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:39:54.900433+0000","last_change":"2024-09-13T14:38:53.893021+0000","last_active":"2024-09-13T14:39:54.900433+0000","last_peered":"2024-09-13T14:39:54.900433+0000","last_clean":"2024-09-13T14:39:54.900433+0000","last_became_active":"2024-09-13T14:38:53.892668+0000","last_became_peered":"2024-09-13T14:38:53.892668+0000","last_unstale":"2024-09-13T14:39:54.900433+0000","last_undegraded":"2024-09-13T14:39:54.900433+0000","last_fullsized":"2024-09-13T14:39:54.900433+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:57.180738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":233,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:39:53.897896+0000","last_change":"2024-09-13T14:37:48.134642+0000","last_active":"2024-09-13T14:39:53.897896+0000","last_peered":"2024-09-13T14:39:53.897896+0000","last_clean":"2024-09-13T14:39:53.897896+0000","last_became_active":"2024-09-13T14:37:48.134402+0000","last_became_peered":"2024-09-13T14:37:48.134402+0000","last_unstale":"2024-09-13T14:39:53.897896+0000","last_undegraded":"2024-09-13T14:39:53.897896+0000","last_fullsized":"2024-09-13T14:39:53.897896+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:51.313663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":439,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:39:54.904032+0000","last_change":"2024-09-13T14:38:53.903022+0000","last_active":"2024-09-13T14:39:54.904032+0000","last_peered":"2024-09-13T14:39:54.904032+0000","last_clean":"2024-09-13T14:39:54.904032+0000","last_became_active":"2024-09-13T14:38:53.902448+0000","last_became_peered":"2024-09-13T14:38:53.902448+0000","last_unstale":"2024-09-13T14:39:54.904032+0000","last_undegraded":"2024-09-13T14:39:54.904032+0000","last_fullsized":"2024-09-13T14:39:54.904032+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:24.690401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":60,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:39:54.898844+0000","last_change":"2024-09-13T14:38:03.554500+0000","last_active":"2024-09-13T14:39:54.898844+0000","last_peered":"2024-09-13T14:39:54.898844+0000","last_clean":"2024-09-13T14:39:54.898844+0000","last_became_active":"2024-09-13T14:38:03.532910+0000","last_became_peered":"2024-09-13T14:38:03.532910+0000","last_unstale":"2024-09-13T14:39:54.898844+0000","last_undegraded":"2024-09-13T14:39:54.898844+0000","last_fullsized":"2024-09-13T14:39:54.898844+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:27:50.843498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"264'8133","reported_seq":10222,"reported_epoch":264,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:40:01.382784+0000","last_change":"2024-09-13T14:37:49.715669+0000","last_active":"2024-09-13T14:40:01.382784+0000","last_peered":"2024-09-13T14:40:01.382784+0000","last_clean":"2024-09-13T14:37:46.417003+0000","last_became_active":"2024-09-13T14:37:49.690961+0000","last_became_peered":"2024-09-13T14:37:49.690961+0000","last_unstale":"2024-09-13T14:40:01.382784+0000","last_undegraded":"2024-09-13T14:37:49.182535+0000","last_fullsized":"2024-09-13T14:37:49.126746+0000","mapping_epoch":240,"log_start":"210'2400","ondisk_log_start":"210'2400","created":205,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5733,"log_dups_size":0,"ondisk_log_size":5733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:48:52.526429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376635392,"num_objects":5747,"num_object_clones":0,"num_object_copies":17241,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4387,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8133,"num_write_kb":444160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4765,"num_bytes_recovered":312279040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,4],"avail_no_missing":["5","0","4"],"object_location_counts":[{"shards":"0,4,5","objects":4308},{"shards":"4,5","objects":1439}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":25,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:39:53.898887+0000","last_change":"2024-09-13T14:38:53.940238+0000","last_active":"2024-09-13T14:39:53.898887+0000","last_peered":"2024-09-13T14:39:53.898887+0000","last_clean":"2024-09-13T14:39:53.898887+0000","last_became_active":"2024-09-13T14:38:53.892825+0000","last_became_peered":"2024-09-13T14:38:53.892825+0000","last_unstale":"2024-09-13T14:39:53.898887+0000","last_undegraded":"2024-09-13T14:39:53.898887+0000","last_fullsized":"2024-09-13T14:39:53.898887+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:46.666581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":81,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:39:53.918697+0000","last_change":"2024-09-13T14:38:53.898907+0000","last_active":"2024-09-13T14:39:53.918697+0000","last_peered":"2024-09-13T14:39:53.918697+0000","last_clean":"2024-09-13T14:39:53.918697+0000","last_became_active":"2024-09-13T14:38:53.898572+0000","last_became_peered":"2024-09-13T14:38:53.898572+0000","last_unstale":"2024-09-13T14:39:53.918697+0000","last_undegraded":"2024-09-13T14:39:53.918697+0000","last_fullsized":"2024-09-13T14:39:53.918697+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:29.222150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"264'8367","reported_seq":22504,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:40:02.233309+0000","last_change":"2024-09-13T14:38:57.930032+0000","last_active":"2024-09-13T14:40:02.233309+0000","last_peered":"2024-09-13T14:40:02.233309+0000","last_clean":"2024-09-13T14:40:02.233309+0000","last_became_active":"2024-09-13T14:38:57.921458+0000","last_became_peered":"2024-09-13T14:38:57.921458+0000","last_unstale":"2024-09-13T14:40:02.233309+0000","last_undegraded":"2024-09-13T14:40:02.233309+0000","last_fullsized":"2024-09-13T14:40:02.233309+0000","mapping_epoch":256,"log_start":"212'3139","ondisk_log_start":"212'3139","created":205,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5228,"log_dups_size":0,"ondisk_log_size":5228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:27:12.294038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385024000,"num_objects":5875,"num_object_clones":0,"num_object_copies":17625,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8367,"num_write_kb":455744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6990,"num_bytes_recovered":458096640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":25,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:39:53.898812+0000","last_change":"2024-09-13T14:38:53.904885+0000","last_active":"2024-09-13T14:39:53.898812+0000","last_peered":"2024-09-13T14:39:53.898812+0000","last_clean":"2024-09-13T14:39:53.898812+0000","last_became_active":"2024-09-13T14:38:53.891726+0000","last_became_peered":"2024-09-13T14:38:53.891726+0000","last_unstale":"2024-09-13T14:39:53.898812+0000","last_undegraded":"2024-09-13T14:39:53.898812+0000","last_fullsized":"2024-09-13T14:39:53.898812+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:57.079205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":271,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:39:53.895732+0000","last_change":"2024-09-13T14:37:48.160504+0000","last_active":"2024-09-13T14:39:53.895732+0000","last_peered":"2024-09-13T14:39:53.895732+0000","last_clean":"2024-09-13T14:39:53.895732+0000","last_became_active":"2024-09-13T14:37:48.160181+0000","last_became_peered":"2024-09-13T14:37:48.160181+0000","last_unstale":"2024-09-13T14:39:53.895732+0000","last_undegraded":"2024-09-13T14:39:53.895732+0000","last_fullsized":"2024-09-13T14:39:53.895732+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:37.087008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"264'8302","reported_seq":10060,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:40:03.164106+0000","last_change":"2024-09-13T14:38:04.531914+0000","last_active":"2024-09-13T14:40:03.164106+0000","last_peered":"2024-09-13T14:40:03.164106+0000","last_clean":"2024-09-13T14:40:03.164106+0000","last_became_active":"2024-09-13T14:38:04.523448+0000","last_became_peered":"2024-09-13T14:38:04.523448+0000","last_unstale":"2024-09-13T14:40:03.164106+0000","last_undegraded":"2024-09-13T14:40:03.164106+0000","last_fullsized":"2024-09-13T14:40:03.164106+0000","mapping_epoch":245,"log_start":"210'2400","ondisk_log_start":"210'2400","created":205,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5902,"log_dups_size":0,"ondisk_log_size":5902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:06.536896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384827392,"num_objects":5872,"num_object_clones":0,"num_object_copies":17616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8302,"num_write_kb":453568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":811,"num_bytes_recovered":53149696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":262,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:39:53.898645+0000","last_change":"2024-09-13T14:37:49.611774+0000","last_active":"2024-09-13T14:39:53.898645+0000","last_peered":"2024-09-13T14:39:53.898645+0000","last_clean":"2024-09-13T14:39:53.898645+0000","last_became_active":"2024-09-13T14:37:49.611088+0000","last_became_peered":"2024-09-13T14:37:49.611088+0000","last_unstale":"2024-09-13T14:39:53.898645+0000","last_undegraded":"2024-09-13T14:39:53.898645+0000","last_fullsized":"2024-09-13T14:39:53.898645+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:08.933213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":21,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:39:53.897974+0000","last_change":"2024-09-13T14:38:53.899211+0000","last_active":"2024-09-13T14:39:53.897974+0000","last_peered":"2024-09-13T14:39:53.897974+0000","last_clean":"2024-09-13T14:39:53.897974+0000","last_became_active":"2024-09-13T14:38:53.898927+0000","last_became_peered":"2024-09-13T14:38:53.898927+0000","last_unstale":"2024-09-13T14:39:53.897974+0000","last_undegraded":"2024-09-13T14:39:53.897974+0000","last_fullsized":"2024-09-13T14:39:53.897974+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:00:28.857611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"264'8172","reported_seq":8242,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:40:02.243790+0000","last_change":"2024-09-13T14:35:24.004651+0000","last_active":"2024-09-13T14:40:02.243790+0000","last_peered":"2024-09-13T14:40:02.243790+0000","last_clean":"2024-09-13T14:40:02.243790+0000","last_became_active":"2024-09-13T14:35:24.004358+0000","last_became_peered":"2024-09-13T14:35:24.004358+0000","last_unstale":"2024-09-13T14:40:02.243790+0000","last_undegraded":"2024-09-13T14:40:02.243790+0000","last_fullsized":"2024-09-13T14:40:02.243790+0000","mapping_epoch":205,"log_start":"210'1884","ondisk_log_start":"210'1884","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6288,"log_dups_size":0,"ondisk_log_size":6288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:57:38.023669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":380108800,"num_objects":5800,"num_object_clones":0,"num_object_copies":17400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8172,"num_write_kb":447104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":72,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:39:53.918965+0000","last_change":"2024-09-13T14:37:49.168003+0000","last_active":"2024-09-13T14:39:53.918965+0000","last_peered":"2024-09-13T14:39:53.918965+0000","last_clean":"2024-09-13T14:39:53.918965+0000","last_became_active":"2024-09-13T14:37:49.167638+0000","last_became_peered":"2024-09-13T14:37:49.167638+0000","last_unstale":"2024-09-13T14:39:53.918965+0000","last_undegraded":"2024-09-13T14:39:53.918965+0000","last_fullsized":"2024-09-13T14:39:53.918965+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:29.051981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":50,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:39:54.900091+0000","last_change":"2024-09-13T14:38:03.528381+0000","last_active":"2024-09-13T14:39:54.900091+0000","last_peered":"2024-09-13T14:39:54.900091+0000","last_clean":"2024-09-13T14:39:54.900091+0000","last_became_active":"2024-09-13T14:38:03.527282+0000","last_became_peered":"2024-09-13T14:38:03.527282+0000","last_unstale":"2024-09-13T14:39:54.900091+0000","last_undegraded":"2024-09-13T14:39:54.900091+0000","last_fullsized":"2024-09-13T14:39:54.900091+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:24.435067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.f","version":"264'8243","reported_seq":8496,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:40:03.286139+0000","last_change":"2024-09-13T14:38:16.371079+0000","last_active":"2024-09-13T14:40:03.286139+0000","last_peered":"2024-09-13T14:40:03.286139+0000","last_clean":"2024-09-13T14:40:03.286139+0000","last_became_active":"2024-09-13T14:38:12.859883+0000","last_became_peered":"2024-09-13T14:38:12.859883+0000","last_unstale":"2024-09-13T14:40:03.286139+0000","last_undegraded":"2024-09-13T14:40:03.286139+0000","last_fullsized":"2024-09-13T14:40:03.286139+0000","mapping_epoch":245,"log_start":"212'2663","ondisk_log_start":"212'2663","created":205,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5580,"log_dups_size":0,"ondisk_log_size":5580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:08.436532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382926848,"num_objects":5843,"num_object_clones":0,"num_object_copies":17529,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8243,"num_write_kb":450752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":3670016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":408,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:39:54.906039+0000","last_change":"2024-09-13T14:38:03.549595+0000","last_active":"2024-09-13T14:39:54.906039+0000","last_peered":"2024-09-13T14:39:54.906039+0000","last_clean":"2024-09-13T14:39:54.906039+0000","last_became_active":"2024-09-13T14:38:03.533331+0000","last_became_peered":"2024-09-13T14:38:03.533331+0000","last_unstale":"2024-09-13T14:39:54.906039+0000","last_undegraded":"2024-09-13T14:39:54.906039+0000","last_fullsized":"2024-09-13T14:39:54.906039+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:59:02.470343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":246,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:39:53.895993+0000","last_change":"2024-09-13T14:37:48.127600+0000","last_active":"2024-09-13T14:39:53.895993+0000","last_peered":"2024-09-13T14:39:53.895993+0000","last_clean":"2024-09-13T14:39:53.895993+0000","last_became_active":"2024-09-13T14:37:48.127349+0000","last_became_peered":"2024-09-13T14:37:48.127349+0000","last_unstale":"2024-09-13T14:39:53.895993+0000","last_undegraded":"2024-09-13T14:39:53.895993+0000","last_fullsized":"2024-09-13T14:39:53.895993+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:57.268374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"264'8132","reported_seq":10488,"reported_epoch":264,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:40:01.472987+0000","last_change":"2024-09-13T14:38:14.979229+0000","last_active":"2024-09-13T14:40:01.472987+0000","last_peered":"2024-09-13T14:40:01.472987+0000","last_clean":"2024-09-13T14:36:13.131873+0000","last_became_active":"2024-09-13T14:38:10.443954+0000","last_became_peered":"2024-09-13T14:38:10.443954+0000","last_unstale":"2024-09-13T14:40:01.472987+0000","last_undegraded":"2024-09-13T14:38:10.384643+0000","last_fullsized":"2024-09-13T14:38:04.517046+0000","mapping_epoch":245,"log_start":"210'2400","ondisk_log_start":"210'2400","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5732,"log_dups_size":0,"ondisk_log_size":5732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:26:53.493170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385220608,"num_objects":5878,"num_object_clones":0,"num_object_copies":17634,"num_objects_missing_on_primary":3655,"num_objects_missing":3655,"num_objects_degraded":3796,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8132,"num_write_kb":448320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1122,"num_bytes_recovered":73465856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":1},{"shards":"0,3,5,7","objects":8},{"shards":"0,3,7","objects":3654},{"shards":"0,5","objects":2210},{"shards":"3,5,7","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":179,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:39:53.895642+0000","last_change":"2024-09-13T14:37:48.127604+0000","last_active":"2024-09-13T14:39:53.895642+0000","last_peered":"2024-09-13T14:39:53.895642+0000","last_clean":"2024-09-13T14:39:53.895642+0000","last_became_active":"2024-09-13T14:37:48.127300+0000","last_became_peered":"2024-09-13T14:37:48.127300+0000","last_unstale":"2024-09-13T14:39:53.895642+0000","last_undegraded":"2024-09-13T14:39:53.895642+0000","last_fullsized":"2024-09-13T14:39:53.895642+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:51.999506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":299,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:39:53.896119+0000","last_change":"2024-09-13T14:38:03.529967+0000","last_active":"2024-09-13T14:39:53.896119+0000","last_peered":"2024-09-13T14:39:53.896119+0000","last_clean":"2024-09-13T14:39:53.896119+0000","last_became_active":"2024-09-13T14:38:03.529646+0000","last_became_peered":"2024-09-13T14:38:03.529646+0000","last_unstale":"2024-09-13T14:39:53.896119+0000","last_undegraded":"2024-09-13T14:39:53.896119+0000","last_fullsized":"2024-09-13T14:39:53.896119+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:52:05.193279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"264'8314","reported_seq":10041,"reported_epoch":264,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T14:40:03.350603+0000","last_change":"2024-09-13T14:39:53.111523+0000","last_active":"2024-09-13T14:40:03.350603+0000","last_peered":"2024-09-13T14:40:03.350603+0000","last_clean":"2024-09-13T14:35:51.090657+0000","last_became_active":"2024-09-13T14:37:49.619314+0000","last_became_peered":"2024-09-13T14:37:49.619314+0000","last_unstale":"2024-09-13T14:40:03.350603+0000","last_undegraded":"2024-09-13T14:37:49.147721+0000","last_fullsized":"2024-09-13T14:37:49.120361+0000","mapping_epoch":240,"log_start":"210'2400","ondisk_log_start":"210'2400","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5914,"log_dups_size":0,"ondisk_log_size":5914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381747224,"num_objects":5826,"num_object_clones":0,"num_object_copies":17478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3632,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5826,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8314,"num_write_kb":452417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":792,"num_bytes_recovered":51904512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":2194}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":260,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:39:53.895977+0000","last_change":"2024-09-13T14:36:13.860367+0000","last_active":"2024-09-13T14:39:53.895977+0000","last_peered":"2024-09-13T14:39:53.895977+0000","last_clean":"2024-09-13T14:39:53.895977+0000","last_became_active":"2024-09-13T14:36:13.859982+0000","last_became_peered":"2024-09-13T14:36:13.859982+0000","last_unstale":"2024-09-13T14:39:53.895977+0000","last_undegraded":"2024-09-13T14:39:53.895977+0000","last_fullsized":"2024-09-13T14:39:53.895977+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:08:23.311350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":255,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:39:53.896251+0000","last_change":"2024-09-13T14:37:48.147360+0000","last_active":"2024-09-13T14:39:53.896251+0000","last_peered":"2024-09-13T14:39:53.896251+0000","last_clean":"2024-09-13T14:39:53.896251+0000","last_became_active":"2024-09-13T14:37:48.146885+0000","last_became_peered":"2024-09-13T14:37:48.146885+0000","last_unstale":"2024-09-13T14:39:53.896251+0000","last_undegraded":"2024-09-13T14:39:53.896251+0000","last_fullsized":"2024-09-13T14:39:53.896251+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:27.841810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"264'8249","reported_seq":17485,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:40:02.113045+0000","last_change":"2024-09-13T14:38:39.650332+0000","last_active":"2024-09-13T14:40:02.113045+0000","last_peered":"2024-09-13T14:40:02.113045+0000","last_clean":"2024-09-13T14:40:02.113045+0000","last_became_active":"2024-09-13T14:38:39.640835+0000","last_became_peered":"2024-09-13T14:38:39.640835+0000","last_unstale":"2024-09-13T14:40:02.113045+0000","last_undegraded":"2024-09-13T14:40:02.113045+0000","last_fullsized":"2024-09-13T14:40:02.113045+0000","mapping_epoch":251,"log_start":"212'3051","ondisk_log_start":"212'3051","created":205,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5198,"log_dups_size":0,"ondisk_log_size":5198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:48.176606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381747200,"num_objects":5825,"num_object_clones":0,"num_object_copies":17475,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8249,"num_write_kb":450368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4544,"num_bytes_recovered":292945920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":427,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:39:54.903522+0000","last_change":"2024-09-13T14:38:53.900041+0000","last_active":"2024-09-13T14:39:54.903522+0000","last_peered":"2024-09-13T14:39:54.903522+0000","last_clean":"2024-09-13T14:39:54.903522+0000","last_became_active":"2024-09-13T14:38:53.899203+0000","last_became_peered":"2024-09-13T14:38:53.899203+0000","last_unstale":"2024-09-13T14:39:54.903522+0000","last_undegraded":"2024-09-13T14:39:54.903522+0000","last_fullsized":"2024-09-13T14:39:54.903522+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:28:07.883094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":359,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:39:54.904127+0000","last_change":"2024-09-13T14:26:15.823198+0000","last_active":"2024-09-13T14:39:54.904127+0000","last_peered":"2024-09-13T14:39:54.904127+0000","last_clean":"2024-09-13T14:39:54.904127+0000","last_became_active":"2024-09-13T14:26:15.822717+0000","last_became_peered":"2024-09-13T14:26:15.822717+0000","last_unstale":"2024-09-13T14:39:54.904127+0000","last_undegraded":"2024-09-13T14:39:54.904127+0000","last_fullsized":"2024-09-13T14:39:54.904127+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:22.471622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.6","version":"264'8363","reported_seq":10823,"reported_epoch":264,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-13T14:40:03.298164+0000","last_change":"2024-09-13T14:38:55.132683+0000","last_active":"2024-09-13T14:40:03.298164+0000","last_peered":"2024-09-13T14:40:03.298164+0000","last_clean":"2024-09-13T14:36:13.124368+0000","last_became_active":"2024-09-13T14:38:55.120883+0000","last_became_peered":"2024-09-13T14:38:55.120883+0000","last_unstale":"2024-09-13T14:40:03.298164+0000","last_undegraded":"2024-09-13T14:38:55.026610+0000","last_fullsized":"2024-09-13T14:40:03.298164+0000","mapping_epoch":254,"log_start":"212'3119","ondisk_log_start":"212'3119","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5244,"log_dups_size":0,"ondisk_log_size":5244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382402560,"num_objects":5835,"num_object_clones":0,"num_object_copies":17505,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3848,"num_objects_misplaced":1987,"num_objects_unfound":0,"num_objects_dirty":5835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8363,"num_write_kb":454336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1157,"num_bytes_recovered":64618496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,3,5],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3,4","objects":3569},{"shards":"0,3,5","objects":2266}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":422,"reported_epoch":264,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-13T14:39:54.903635+0000","last_change":"2024-09-13T14:38:54.899624+0000","last_active":"2024-09-13T14:39:54.903635+0000","last_peered":"2024-09-13T14:39:54.903635+0000","last_clean":"2024-09-13T14:17:27.401592+0000","last_became_active":"2024-09-13T14:38:54.899282+0000","last_became_peered":"2024-09-13T14:38:54.899282+0000","last_unstale":"2024-09-13T14:39:54.903635+0000","last_undegraded":"2024-09-13T14:38:54.891176+0000","last_fullsized":"2024-09-13T14:39:54.903635+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:07:41.927740+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[3,0,6],"avail_no_missing":["3","4","6"],"object_location_counts":[{"shards":"3,4,6","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":501,"reported_epoch":263,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:39:53.918751+0000","last_change":"2024-09-13T14:38:54.904329+0000","last_active":"2024-09-13T14:39:53.918751+0000","last_peered":"2024-09-13T14:39:53.918751+0000","last_clean":"2024-09-13T14:38:39.648017+0000","last_became_active":"2024-09-13T14:38:54.903798+0000","last_became_peered":"2024-09-13T14:38:54.903798+0000","last_unstale":"2024-09-13T14:39:53.918751+0000","last_undegraded":"2024-09-13T14:38:54.890505+0000","last_fullsized":"2024-09-13T14:38:54.889701+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:18.120931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,3],"avail_no_missing":["5","3","6"],"object_location_counts":[{"shards":"3,5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":361,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:39:54.905872+0000","last_change":"2024-09-13T14:37:49.160719+0000","last_active":"2024-09-13T14:39:54.905872+0000","last_peered":"2024-09-13T14:39:54.905872+0000","last_clean":"2024-09-13T14:39:54.905872+0000","last_became_active":"2024-09-13T14:37:49.160401+0000","last_became_peered":"2024-09-13T14:37:49.160401+0000","last_unstale":"2024-09-13T14:39:54.905872+0000","last_undegraded":"2024-09-13T14:39:54.905872+0000","last_fullsized":"2024-09-13T14:39:54.905872+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:21.737945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":106,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:39:53.896044+0000","last_change":"2024-09-13T14:37:48.172940+0000","last_active":"2024-09-13T14:39:53.896044+0000","last_peered":"2024-09-13T14:39:53.896044+0000","last_clean":"2024-09-13T14:39:53.896044+0000","last_became_active":"2024-09-13T14:37:48.172060+0000","last_became_peered":"2024-09-13T14:37:48.172060+0000","last_unstale":"2024-09-13T14:39:53.896044+0000","last_undegraded":"2024-09-13T14:39:53.896044+0000","last_fullsized":"2024-09-13T14:39:53.896044+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:07.480645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.5","version":"264'8256","reported_seq":24602,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:40:02.123331+0000","last_change":"2024-09-13T14:39:34.446699+0000","last_active":"2024-09-13T14:40:02.123331+0000","last_peered":"2024-09-13T14:40:02.123331+0000","last_clean":"2024-09-13T14:40:02.123331+0000","last_became_active":"2024-09-13T14:38:04.600616+0000","last_became_peered":"2024-09-13T14:38:04.600616+0000","last_unstale":"2024-09-13T14:40:02.123331+0000","last_undegraded":"2024-09-13T14:40:02.123331+0000","last_fullsized":"2024-09-13T14:40:02.123331+0000","mapping_epoch":245,"log_start":"210'1916","ondisk_log_start":"210'1916","created":205,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6340,"log_dups_size":0,"ondisk_log_size":6340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:00:44.321678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385744896,"num_objects":5886,"num_object_clones":0,"num_object_copies":17658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8256,"num_write_kb":452544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8120,"num_bytes_recovered":521076736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":479,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:39:53.897809+0000","last_change":"2024-09-13T14:26:15.835178+0000","last_active":"2024-09-13T14:39:53.897809+0000","last_peered":"2024-09-13T14:39:53.897809+0000","last_clean":"2024-09-13T14:39:53.897809+0000","last_became_active":"2024-09-13T14:26:15.820283+0000","last_became_peered":"2024-09-13T14:26:15.820283+0000","last_unstale":"2024-09-13T14:39:53.897809+0000","last_undegraded":"2024-09-13T14:39:53.897809+0000","last_fullsized":"2024-09-13T14:39:53.897809+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:30.340038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":50,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:39:54.899051+0000","last_change":"2024-09-13T14:38:03.525682+0000","last_active":"2024-09-13T14:39:54.899051+0000","last_peered":"2024-09-13T14:39:54.899051+0000","last_clean":"2024-09-13T14:39:54.899051+0000","last_became_active":"2024-09-13T14:38:03.525406+0000","last_became_peered":"2024-09-13T14:38:03.525406+0000","last_unstale":"2024-09-13T14:39:54.899051+0000","last_undegraded":"2024-09-13T14:39:54.899051+0000","last_fullsized":"2024-09-13T14:39:54.899051+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:41:28.970300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":57,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:39:54.900437+0000","last_change":"2024-09-13T14:38:53.991738+0000","last_active":"2024-09-13T14:39:54.900437+0000","last_peered":"2024-09-13T14:39:54.900437+0000","last_clean":"2024-09-13T14:39:54.900437+0000","last_became_active":"2024-09-13T14:38:53.991417+0000","last_became_peered":"2024-09-13T14:38:53.991417+0000","last_unstale":"2024-09-13T14:39:54.900437+0000","last_undegraded":"2024-09-13T14:39:54.900437+0000","last_fullsized":"2024-09-13T14:39:54.900437+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:50:38.559838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"264'8297","reported_seq":22447,"reported_epoch":264,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-13T14:40:03.312809+0000","last_change":"2024-09-13T14:38:17.472071+0000","last_active":"2024-09-13T14:40:03.312809+0000","last_peered":"2024-09-13T14:40:03.312809+0000","last_clean":"2024-09-13T14:37:46.290754+0000","last_became_active":"2024-09-13T14:38:17.381753+0000","last_became_peered":"2024-09-13T14:38:17.381753+0000","last_unstale":"2024-09-13T14:40:03.312809+0000","last_undegraded":"2024-09-13T14:38:13.504140+0000","last_fullsized":"2024-09-13T14:40:03.312809+0000","mapping_epoch":245,"log_start":"210'2359","ondisk_log_start":"210'2359","created":205,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5938,"log_dups_size":0,"ondisk_log_size":5938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":380698624,"num_objects":5809,"num_object_clones":0,"num_object_copies":17427,"num_objects_missing_on_primary":83,"num_objects_missing":83,"num_objects_degraded":4582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8297,"num_write_kb":451392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7053,"num_bytes_recovered":462225408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"0,4,5","objects":4413},{"shards":"0,4,6","objects":1396}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":772,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:39:54.903513+0000","last_change":"2024-09-13T14:37:48.131799+0000","last_active":"2024-09-13T14:39:54.903513+0000","last_peered":"2024-09-13T14:39:54.903513+0000","last_clean":"2024-09-13T14:39:54.903513+0000","last_became_active":"2024-09-13T14:37:48.130837+0000","last_became_peered":"2024-09-13T14:37:48.130837+0000","last_unstale":"2024-09-13T14:39:54.903513+0000","last_undegraded":"2024-09-13T14:39:54.903513+0000","last_fullsized":"2024-09-13T14:39:54.903513+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:06:32.663770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":291,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:39:54.899124+0000","last_change":"2024-09-13T14:37:49.301531+0000","last_active":"2024-09-13T14:39:54.899124+0000","last_peered":"2024-09-13T14:39:54.899124+0000","last_clean":"2024-09-13T14:39:54.899124+0000","last_became_active":"2024-09-13T14:37:49.301249+0000","last_became_peered":"2024-09-13T14:37:49.301249+0000","last_unstale":"2024-09-13T14:39:54.899124+0000","last_undegraded":"2024-09-13T14:39:54.899124+0000","last_fullsized":"2024-09-13T14:39:54.899124+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:10.977710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":88,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:39:53.919406+0000","last_change":"2024-09-13T14:38:53.901427+0000","last_active":"2024-09-13T14:39:53.919406+0000","last_peered":"2024-09-13T14:39:53.919406+0000","last_clean":"2024-09-13T14:39:53.919406+0000","last_became_active":"2024-09-13T14:38:53.901157+0000","last_became_peered":"2024-09-13T14:38:53.901157+0000","last_unstale":"2024-09-13T14:39:53.919406+0000","last_undegraded":"2024-09-13T14:39:53.919406+0000","last_fullsized":"2024-09-13T14:39:53.919406+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:28:21.713025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"264'8241","reported_seq":22281,"reported_epoch":264,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-13T14:40:03.296431+0000","last_change":"2024-09-13T14:38:16.552896+0000","last_active":"2024-09-13T14:40:03.296431+0000","last_peered":"2024-09-13T14:40:03.296431+0000","last_clean":"2024-09-13T14:37:46.405415+0000","last_became_active":"2024-09-13T14:38:16.427510+0000","last_became_peered":"2024-09-13T14:38:16.427510+0000","last_unstale":"2024-09-13T14:40:03.296431+0000","last_undegraded":"2024-09-13T14:38:12.738464+0000","last_fullsized":"2024-09-13T14:40:03.296431+0000","mapping_epoch":245,"log_start":"210'2518","ondisk_log_start":"210'2518","created":205,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5723,"log_dups_size":0,"ondisk_log_size":5723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":377290752,"num_objects":5757,"num_object_clones":0,"num_object_copies":17271,"num_objects_missing_on_primary":100,"num_objects_missing":100,"num_objects_degraded":4572,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8241,"num_write_kb":447936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6999,"num_bytes_recovered":458686464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"0,3,6","objects":1352},{"shards":"0,3,7","objects":4405}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":970,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:39:53.918756+0000","last_change":"2024-09-13T14:37:49.147498+0000","last_active":"2024-09-13T14:39:53.918756+0000","last_peered":"2024-09-13T14:39:53.918756+0000","last_clean":"2024-09-13T14:39:53.918756+0000","last_became_active":"2024-09-13T14:37:49.147028+0000","last_became_peered":"2024-09-13T14:37:49.147028+0000","last_unstale":"2024-09-13T14:39:53.918756+0000","last_undegraded":"2024-09-13T14:39:53.918756+0000","last_fullsized":"2024-09-13T14:39:53.918756+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:11:29.050178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":268,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:39:53.896737+0000","last_change":"2024-09-13T14:38:03.528133+0000","last_active":"2024-09-13T14:39:53.896737+0000","last_peered":"2024-09-13T14:39:53.896737+0000","last_clean":"2024-09-13T14:39:53.896737+0000","last_became_active":"2024-09-13T14:38:03.527829+0000","last_became_peered":"2024-09-13T14:38:03.527829+0000","last_unstale":"2024-09-13T14:39:53.896737+0000","last_undegraded":"2024-09-13T14:39:53.896737+0000","last_fullsized":"2024-09-13T14:39:53.896737+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:48.254169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":411,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:39:54.904364+0000","last_change":"2024-09-13T14:38:53.899755+0000","last_active":"2024-09-13T14:39:54.904364+0000","last_peered":"2024-09-13T14:39:54.904364+0000","last_clean":"2024-09-13T14:39:54.904364+0000","last_became_active":"2024-09-13T14:38:53.899448+0000","last_became_peered":"2024-09-13T14:38:53.899448+0000","last_unstale":"2024-09-13T14:39:54.904364+0000","last_undegraded":"2024-09-13T14:39:54.904364+0000","last_fullsized":"2024-09-13T14:39:54.904364+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:55.897035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"264'8415","reported_seq":8503,"reported_epoch":264,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:40:02.429081+0000","last_change":"2024-09-13T14:38:03.567892+0000","last_active":"2024-09-13T14:40:02.429081+0000","last_peered":"2024-09-13T14:40:02.429081+0000","last_clean":"2024-09-13T14:35:51.090791+0000","last_became_active":"2024-09-13T14:37:49.665025+0000","last_became_peered":"2024-09-13T14:37:49.665025+0000","last_unstale":"2024-09-13T14:40:02.429081+0000","last_undegraded":"2024-09-13T14:37:49.169101+0000","last_fullsized":"2024-09-13T14:37:49.121596+0000","mapping_epoch":240,"log_start":"210'2363","ondisk_log_start":"210'2363","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6052,"log_dups_size":0,"ondisk_log_size":6052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:02.321504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385810432,"num_objects":5887,"num_object_clones":0,"num_object_copies":17661,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4501,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8415,"num_write_kb":457664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":1452}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":72,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:39:53.919149+0000","last_change":"2024-09-13T14:37:49.149770+0000","last_active":"2024-09-13T14:39:53.919149+0000","last_peered":"2024-09-13T14:39:53.919149+0000","last_clean":"2024-09-13T14:39:53.919149+0000","last_became_active":"2024-09-13T14:37:49.148991+0000","last_became_peered":"2024-09-13T14:37:49.148991+0000","last_unstale":"2024-09-13T14:39:53.919149+0000","last_undegraded":"2024-09-13T14:39:53.919149+0000","last_fullsized":"2024-09-13T14:39:53.919149+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:04:04.193366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":1109,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:39:54.900333+0000","last_change":"2024-09-13T14:30:58.928302+0000","last_active":"2024-09-13T14:39:54.900333+0000","last_peered":"2024-09-13T14:39:54.900333+0000","last_clean":"2024-09-13T14:39:54.900333+0000","last_became_active":"2024-09-13T14:30:58.927718+0000","last_became_peered":"2024-09-13T14:30:58.927718+0000","last_unstale":"2024-09-13T14:39:54.900333+0000","last_undegraded":"2024-09-13T14:39:54.900333+0000","last_fullsized":"2024-09-13T14:39:54.900333+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:37.323036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":79,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:39:53.918862+0000","last_change":"2024-09-13T14:38:03.534628+0000","last_active":"2024-09-13T14:39:53.918862+0000","last_peered":"2024-09-13T14:39:53.918862+0000","last_clean":"2024-09-13T14:39:53.918862+0000","last_became_active":"2024-09-13T14:38:03.534262+0000","last_became_peered":"2024-09-13T14:38:03.534262+0000","last_unstale":"2024-09-13T14:39:53.918862+0000","last_undegraded":"2024-09-13T14:39:53.918862+0000","last_fullsized":"2024-09-13T14:39:53.918862+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:31.168783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"264'8200","reported_seq":22350,"reported_epoch":264,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:40:01.323644+0000","last_change":"2024-09-13T14:38:53.944111+0000","last_active":"2024-09-13T14:40:01.323644+0000","last_peered":"2024-09-13T14:40:01.323644+0000","last_clean":"2024-09-13T14:37:47.047672+0000","last_became_active":"2024-09-13T14:37:49.664294+0000","last_became_peered":"2024-09-13T14:37:49.664294+0000","last_unstale":"2024-09-13T14:40:01.323644+0000","last_undegraded":"2024-09-13T14:38:53.940822+0000","last_fullsized":"2024-09-13T14:37:49.126627+0000","mapping_epoch":240,"log_start":"210'2500","ondisk_log_start":"210'2500","created":205,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5700,"log_dups_size":0,"ondisk_log_size":5700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:11.766692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383516672,"num_objects":5852,"num_object_clones":0,"num_object_copies":17556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4535,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8200,"num_write_kb":449664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7009,"num_bytes_recovered":459341824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3],"avail_no_missing":["5","0","3"],"object_location_counts":[{"shards":"0,3,5","objects":4460},{"shards":"3,5","objects":1392}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":91,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:39:53.919045+0000","last_change":"2024-09-13T14:38:03.534303+0000","last_active":"2024-09-13T14:39:53.919045+0000","last_peered":"2024-09-13T14:39:53.919045+0000","last_clean":"2024-09-13T14:39:53.919045+0000","last_became_active":"2024-09-13T14:38:03.533171+0000","last_became_peered":"2024-09-13T14:38:03.533171+0000","last_unstale":"2024-09-13T14:39:53.919045+0000","last_undegraded":"2024-09-13T14:39:53.919045+0000","last_fullsized":"2024-09-13T14:39:53.919045+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:00:16.553526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":784,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:39:53.898631+0000","last_change":"2024-09-13T14:38:04.596192+0000","last_active":"2024-09-13T14:39:53.898631+0000","last_peered":"2024-09-13T14:39:53.898631+0000","last_clean":"2024-09-13T14:39:53.898631+0000","last_became_active":"2024-09-13T14:38:04.593299+0000","last_became_peered":"2024-09-13T14:38:04.593299+0000","last_unstale":"2024-09-13T14:39:53.898631+0000","last_undegraded":"2024-09-13T14:39:53.898631+0000","last_fullsized":"2024-09-13T14:39:53.898631+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:34.241410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":50,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:39:54.900167+0000","last_change":"2024-09-13T14:38:03.533120+0000","last_active":"2024-09-13T14:39:54.900167+0000","last_peered":"2024-09-13T14:39:54.900167+0000","last_clean":"2024-09-13T14:39:54.900167+0000","last_became_active":"2024-09-13T14:38:03.532829+0000","last_became_peered":"2024-09-13T14:38:03.532829+0000","last_unstale":"2024-09-13T14:39:54.900167+0000","last_undegraded":"2024-09-13T14:39:54.900167+0000","last_fullsized":"2024-09-13T14:39:54.900167+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:24.514661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"264'8397","reported_seq":8535,"reported_epoch":264,"state":"active+undersized+degraded+remapped+backfill_wait","last_fresh":"2024-09-13T14:40:02.210104+0000","last_change":"2024-09-13T14:37:49.295861+0000","last_active":"2024-09-13T14:40:02.210104+0000","last_peered":"2024-09-13T14:40:02.210104+0000","last_clean":"2024-09-13T14:35:51.078708+0000","last_became_active":"2024-09-13T14:37:49.290403+0000","last_became_peered":"2024-09-13T14:37:49.290403+0000","last_unstale":"2024-09-13T14:40:02.210104+0000","last_undegraded":"2024-09-13T14:37:49.147402+0000","last_fullsized":"2024-09-13T14:37:49.117202+0000","mapping_epoch":240,"log_start":"212'3218","ondisk_log_start":"212'3218","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5179,"log_dups_size":0,"ondisk_log_size":5179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393281536,"num_objects":6001,"num_object_clones":0,"num_object_copies":18003,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6001,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8397,"num_write_kb":460736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":6001}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":453,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:39:54.904204+0000","last_change":"2024-09-13T14:38:53.938227+0000","last_active":"2024-09-13T14:39:54.904204+0000","last_peered":"2024-09-13T14:39:54.904204+0000","last_clean":"2024-09-13T14:39:54.904204+0000","last_became_active":"2024-09-13T14:38:53.902867+0000","last_became_peered":"2024-09-13T14:38:53.902867+0000","last_unstale":"2024-09-13T14:39:54.904204+0000","last_undegraded":"2024-09-13T14:39:54.904204+0000","last_fullsized":"2024-09-13T14:39:54.904204+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:16.831756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":562,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:39:53.896383+0000","last_change":"2024-09-13T14:37:48.147677+0000","last_active":"2024-09-13T14:39:53.896383+0000","last_peered":"2024-09-13T14:39:53.896383+0000","last_clean":"2024-09-13T14:39:53.896383+0000","last_became_active":"2024-09-13T14:37:48.147001+0000","last_became_peered":"2024-09-13T14:37:48.147001+0000","last_unstale":"2024-09-13T14:39:53.896383+0000","last_undegraded":"2024-09-13T14:39:53.896383+0000","last_fullsized":"2024-09-13T14:39:53.896383+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:35:00.653576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":434,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:39:54.903291+0000","last_change":"2024-09-13T14:38:53.900107+0000","last_active":"2024-09-13T14:39:54.903291+0000","last_peered":"2024-09-13T14:39:54.903291+0000","last_clean":"2024-09-13T14:39:54.903291+0000","last_became_active":"2024-09-13T14:38:53.899556+0000","last_became_peered":"2024-09-13T14:38:53.899556+0000","last_unstale":"2024-09-13T14:39:54.903291+0000","last_undegraded":"2024-09-13T14:39:54.903291+0000","last_fullsized":"2024-09-13T14:39:54.903291+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:15:47.918163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"264'8111","reported_seq":17052,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:40:02.118986+0000","last_change":"2024-09-13T14:39:54.921239+0000","last_active":"2024-09-13T14:40:02.118986+0000","last_peered":"2024-09-13T14:40:02.118986+0000","last_clean":"2024-09-13T14:40:02.118986+0000","last_became_active":"2024-09-13T14:39:54.907894+0000","last_became_peered":"2024-09-13T14:39:54.907894+0000","last_unstale":"2024-09-13T14:40:02.118986+0000","last_undegraded":"2024-09-13T14:40:02.118986+0000","last_fullsized":"2024-09-13T14:40:02.118986+0000","mapping_epoch":263,"log_start":"212'2847","ondisk_log_start":"212'2847","created":205,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5264,"log_dups_size":0,"ondisk_log_size":5264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:52:04.619238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376242176,"num_objects":5741,"num_object_clones":0,"num_object_copies":17223,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8111,"num_write_kb":443264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4395,"num_bytes_recovered":283770880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":65,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:39:54.900337+0000","last_change":"2024-09-13T14:38:53.901806+0000","last_active":"2024-09-13T14:39:54.900337+0000","last_peered":"2024-09-13T14:39:54.900337+0000","last_clean":"2024-09-13T14:39:54.900337+0000","last_became_active":"2024-09-13T14:38:53.891055+0000","last_became_peered":"2024-09-13T14:38:53.891055+0000","last_unstale":"2024-09-13T14:39:54.900337+0000","last_undegraded":"2024-09-13T14:39:54.900337+0000","last_fullsized":"2024-09-13T14:39:54.900337+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:03:00.416198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":19,"reported_epoch":263,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-13T14:39:53.897701+0000","last_change":"2024-09-13T14:38:53.900902+0000","last_active":"2024-09-13T14:39:53.897701+0000","last_peered":"2024-09-13T14:39:53.897701+0000","last_clean":"2024-09-13T14:38:39.635779+0000","last_became_active":"2024-09-13T14:38:53.900626+0000","last_became_peered":"2024-09-13T14:38:53.900626+0000","last_unstale":"2024-09-13T14:39:53.897701+0000","last_undegraded":"2024-09-13T14:38:53.885778+0000","last_fullsized":"2024-09-13T14:39:53.897701+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:37:21.715778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":7,"num_objects_missing":7,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":["3","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":252,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:39:53.895850+0000","last_change":"2024-09-13T14:37:49.172218+0000","last_active":"2024-09-13T14:39:53.895850+0000","last_peered":"2024-09-13T14:39:53.895850+0000","last_clean":"2024-09-13T14:39:53.895850+0000","last_became_active":"2024-09-13T14:37:49.171718+0000","last_became_peered":"2024-09-13T14:37:49.171718+0000","last_unstale":"2024-09-13T14:39:53.895850+0000","last_undegraded":"2024-09-13T14:39:53.895850+0000","last_fullsized":"2024-09-13T14:39:53.895850+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:46:54.626521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":275,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:39:53.896585+0000","last_change":"2024-09-13T14:37:49.230094+0000","last_active":"2024-09-13T14:39:53.896585+0000","last_peered":"2024-09-13T14:39:53.896585+0000","last_clean":"2024-09-13T14:39:53.896585+0000","last_became_active":"2024-09-13T14:37:49.229393+0000","last_became_peered":"2024-09-13T14:37:49.229393+0000","last_unstale":"2024-09-13T14:39:53.896585+0000","last_undegraded":"2024-09-13T14:39:53.896585+0000","last_fullsized":"2024-09-13T14:39:53.896585+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:02:28.690318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":255,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:39:53.896136+0000","last_change":"2024-09-13T14:37:48.128447+0000","last_active":"2024-09-13T14:39:53.896136+0000","last_peered":"2024-09-13T14:39:53.896136+0000","last_clean":"2024-09-13T14:39:53.896136+0000","last_became_active":"2024-09-13T14:37:48.128143+0000","last_became_peered":"2024-09-13T14:37:48.128143+0000","last_unstale":"2024-09-13T14:39:53.896136+0000","last_undegraded":"2024-09-13T14:39:53.896136+0000","last_fullsized":"2024-09-13T14:39:53.896136+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:22.204039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":50,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:39:54.899531+0000","last_change":"2024-09-13T14:38:03.531683+0000","last_active":"2024-09-13T14:39:54.899531+0000","last_peered":"2024-09-13T14:39:54.899531+0000","last_clean":"2024-09-13T14:39:54.899531+0000","last_became_active":"2024-09-13T14:38:03.531476+0000","last_became_peered":"2024-09-13T14:38:03.531476+0000","last_unstale":"2024-09-13T14:39:54.899531+0000","last_undegraded":"2024-09-13T14:39:54.899531+0000","last_fullsized":"2024-09-13T14:39:54.899531+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:41.557569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":27,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:39:53.897922+0000","last_change":"2024-09-13T14:38:53.954281+0000","last_active":"2024-09-13T14:39:53.897922+0000","last_peered":"2024-09-13T14:39:53.897922+0000","last_clean":"2024-09-13T14:39:53.897922+0000","last_became_active":"2024-09-13T14:38:53.900912+0000","last_became_peered":"2024-09-13T14:38:53.900912+0000","last_unstale":"2024-09-13T14:39:53.897922+0000","last_undegraded":"2024-09-13T14:39:53.897922+0000","last_fullsized":"2024-09-13T14:39:53.897922+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:47.876275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":79,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:39:53.958023+0000","last_change":"2024-09-13T14:38:54.000782+0000","last_active":"2024-09-13T14:39:53.958023+0000","last_peered":"2024-09-13T14:39:53.958023+0000","last_clean":"2024-09-13T14:39:53.958023+0000","last_became_active":"2024-09-13T14:38:54.000322+0000","last_became_peered":"2024-09-13T14:38:54.000322+0000","last_unstale":"2024-09-13T14:39:53.958023+0000","last_undegraded":"2024-09-13T14:39:53.958023+0000","last_fullsized":"2024-09-13T14:39:53.958023+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:09.057597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":376,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:39:54.903184+0000","last_change":"2024-09-13T14:38:53.899535+0000","last_active":"2024-09-13T14:39:54.903184+0000","last_peered":"2024-09-13T14:39:54.903184+0000","last_clean":"2024-09-13T14:39:54.903184+0000","last_became_active":"2024-09-13T14:38:53.899183+0000","last_became_peered":"2024-09-13T14:38:53.899183+0000","last_unstale":"2024-09-13T14:39:54.903184+0000","last_undegraded":"2024-09-13T14:39:54.903184+0000","last_fullsized":"2024-09-13T14:39:54.903184+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:33.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":50,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:39:54.900794+0000","last_change":"2024-09-13T14:38:03.543426+0000","last_active":"2024-09-13T14:39:54.900794+0000","last_peered":"2024-09-13T14:39:54.900794+0000","last_clean":"2024-09-13T14:39:54.900794+0000","last_became_active":"2024-09-13T14:38:03.542661+0000","last_became_peered":"2024-09-13T14:38:03.542661+0000","last_unstale":"2024-09-13T14:39:54.900794+0000","last_undegraded":"2024-09-13T14:39:54.900794+0000","last_fullsized":"2024-09-13T14:39:54.900794+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:54.050127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":405,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:39:54.903249+0000","last_change":"2024-09-13T14:37:48.131321+0000","last_active":"2024-09-13T14:39:54.903249+0000","last_peered":"2024-09-13T14:39:54.903249+0000","last_clean":"2024-09-13T14:39:54.903249+0000","last_became_active":"2024-09-13T14:37:48.130998+0000","last_became_peered":"2024-09-13T14:37:48.130998+0000","last_unstale":"2024-09-13T14:39:54.903249+0000","last_undegraded":"2024-09-13T14:39:54.903249+0000","last_fullsized":"2024-09-13T14:39:54.903249+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:33:26.597825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":91,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:39:53.919382+0000","last_change":"2024-09-13T14:38:03.535454+0000","last_active":"2024-09-13T14:39:53.919382+0000","last_peered":"2024-09-13T14:39:53.919382+0000","last_clean":"2024-09-13T14:39:53.919382+0000","last_became_active":"2024-09-13T14:38:03.535131+0000","last_became_peered":"2024-09-13T14:38:03.535131+0000","last_unstale":"2024-09-13T14:39:53.919382+0000","last_undegraded":"2024-09-13T14:39:53.919382+0000","last_fullsized":"2024-09-13T14:39:53.919382+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:24.477601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":430,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:39:54.903394+0000","last_change":"2024-09-13T14:38:03.524939+0000","last_active":"2024-09-13T14:39:54.903394+0000","last_peered":"2024-09-13T14:39:54.903394+0000","last_clean":"2024-09-13T14:39:54.903394+0000","last_became_active":"2024-09-13T14:38:03.524406+0000","last_became_peered":"2024-09-13T14:38:03.524406+0000","last_unstale":"2024-09-13T14:39:54.903394+0000","last_undegraded":"2024-09-13T14:39:54.903394+0000","last_fullsized":"2024-09-13T14:39:54.903394+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:19:45.257424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":57,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:39:54.899176+0000","last_change":"2024-09-13T14:38:53.892948+0000","last_active":"2024-09-13T14:39:54.899176+0000","last_peered":"2024-09-13T14:39:54.899176+0000","last_clean":"2024-09-13T14:39:54.899176+0000","last_became_active":"2024-09-13T14:38:53.892665+0000","last_became_peered":"2024-09-13T14:38:53.892665+0000","last_unstale":"2024-09-13T14:39:54.899176+0000","last_undegraded":"2024-09-13T14:39:54.899176+0000","last_fullsized":"2024-09-13T14:39:54.899176+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:08.292359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":378,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:39:54.903422+0000","last_change":"2024-09-13T14:37:48.139348+0000","last_active":"2024-09-13T14:39:54.903422+0000","last_peered":"2024-09-13T14:39:54.903422+0000","last_clean":"2024-09-13T14:39:54.903422+0000","last_became_active":"2024-09-13T14:37:48.139115+0000","last_became_peered":"2024-09-13T14:37:48.139115+0000","last_unstale":"2024-09-13T14:39:54.903422+0000","last_undegraded":"2024-09-13T14:39:54.903422+0000","last_fullsized":"2024-09-13T14:39:54.903422+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:32:20.413346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":419,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:39:54.903551+0000","last_change":"2024-09-13T14:37:48.172088+0000","last_active":"2024-09-13T14:39:54.903551+0000","last_peered":"2024-09-13T14:39:54.903551+0000","last_clean":"2024-09-13T14:39:54.903551+0000","last_became_active":"2024-09-13T14:37:48.171779+0000","last_became_peered":"2024-09-13T14:37:48.171779+0000","last_unstale":"2024-09-13T14:39:54.903551+0000","last_undegraded":"2024-09-13T14:39:54.903551+0000","last_fullsized":"2024-09-13T14:39:54.903551+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:02.852712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":319,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:39:53.895691+0000","last_change":"2024-09-13T14:38:53.890236+0000","last_active":"2024-09-13T14:39:53.895691+0000","last_peered":"2024-09-13T14:39:53.895691+0000","last_clean":"2024-09-13T14:39:53.895691+0000","last_became_active":"2024-09-13T14:38:53.889582+0000","last_became_peered":"2024-09-13T14:38:53.889582+0000","last_unstale":"2024-09-13T14:39:53.895691+0000","last_undegraded":"2024-09-13T14:39:53.895691+0000","last_fullsized":"2024-09-13T14:39:53.895691+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:14:29.334774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":405,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:39:54.904627+0000","last_change":"2024-09-13T14:38:03.534097+0000","last_active":"2024-09-13T14:39:54.904627+0000","last_peered":"2024-09-13T14:39:54.904627+0000","last_clean":"2024-09-13T14:39:54.904627+0000","last_became_active":"2024-09-13T14:38:03.533794+0000","last_became_peered":"2024-09-13T14:38:03.533794+0000","last_unstale":"2024-09-13T14:39:54.904627+0000","last_undegraded":"2024-09-13T14:39:54.904627+0000","last_fullsized":"2024-09-13T14:39:54.904627+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:23.272098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":50,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:39:54.899998+0000","last_change":"2024-09-13T14:38:03.527021+0000","last_active":"2024-09-13T14:39:54.899998+0000","last_peered":"2024-09-13T14:39:54.899998+0000","last_clean":"2024-09-13T14:39:54.899998+0000","last_became_active":"2024-09-13T14:38:03.526698+0000","last_became_peered":"2024-09-13T14:38:03.526698+0000","last_unstale":"2024-09-13T14:39:54.899998+0000","last_undegraded":"2024-09-13T14:39:54.899998+0000","last_fullsized":"2024-09-13T14:39:54.899998+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:01:35.734086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":263,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:39:53.896515+0000","last_change":"2024-09-13T14:27:12.834931+0000","last_active":"2024-09-13T14:39:53.896515+0000","last_peered":"2024-09-13T14:39:53.896515+0000","last_clean":"2024-09-13T14:39:53.896515+0000","last_became_active":"2024-09-13T14:27:12.834572+0000","last_became_peered":"2024-09-13T14:27:12.834572+0000","last_unstale":"2024-09-13T14:39:53.896515+0000","last_undegraded":"2024-09-13T14:39:53.896515+0000","last_fullsized":"2024-09-13T14:39:53.896515+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:54:21.155689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":76,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:39:53.919005+0000","last_change":"2024-09-13T14:37:49.176796+0000","last_active":"2024-09-13T14:39:53.919005+0000","last_peered":"2024-09-13T14:39:53.919005+0000","last_clean":"2024-09-13T14:39:53.919005+0000","last_became_active":"2024-09-13T14:37:49.176246+0000","last_became_peered":"2024-09-13T14:37:49.176246+0000","last_unstale":"2024-09-13T14:39:53.919005+0000","last_undegraded":"2024-09-13T14:39:53.919005+0000","last_fullsized":"2024-09-13T14:39:53.919005+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:29:24.936718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":274,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:39:53.895602+0000","last_change":"2024-09-13T14:38:03.526333+0000","last_active":"2024-09-13T14:39:53.895602+0000","last_peered":"2024-09-13T14:39:53.895602+0000","last_clean":"2024-09-13T14:39:53.895602+0000","last_became_active":"2024-09-13T14:38:03.526001+0000","last_became_peered":"2024-09-13T14:38:03.526001+0000","last_unstale":"2024-09-13T14:39:53.895602+0000","last_undegraded":"2024-09-13T14:39:53.895602+0000","last_fullsized":"2024-09-13T14:39:53.895602+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:25:25.181271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":76,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:39:53.919175+0000","last_change":"2024-09-13T14:37:49.149582+0000","last_active":"2024-09-13T14:39:53.919175+0000","last_peered":"2024-09-13T14:39:53.919175+0000","last_clean":"2024-09-13T14:39:53.919175+0000","last_became_active":"2024-09-13T14:37:49.148525+0000","last_became_peered":"2024-09-13T14:37:49.148525+0000","last_unstale":"2024-09-13T14:39:53.919175+0000","last_undegraded":"2024-09-13T14:39:53.919175+0000","last_fullsized":"2024-09-13T14:39:53.919175+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:48.655665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":451,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:39:54.903604+0000","last_change":"2024-09-13T14:38:03.535255+0000","last_active":"2024-09-13T14:39:54.903604+0000","last_peered":"2024-09-13T14:39:54.903604+0000","last_clean":"2024-09-13T14:39:54.903604+0000","last_became_active":"2024-09-13T14:38:03.533917+0000","last_became_peered":"2024-09-13T14:38:03.533917+0000","last_unstale":"2024-09-13T14:39:54.903604+0000","last_undegraded":"2024-09-13T14:39:54.903604+0000","last_fullsized":"2024-09-13T14:39:54.903604+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:33.559788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":414,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:39:54.904542+0000","last_change":"2024-09-13T14:37:48.146784+0000","last_active":"2024-09-13T14:39:54.904542+0000","last_peered":"2024-09-13T14:39:54.904542+0000","last_clean":"2024-09-13T14:39:54.904542+0000","last_became_active":"2024-09-13T14:37:48.146491+0000","last_became_peered":"2024-09-13T14:37:48.146491+0000","last_unstale":"2024-09-13T14:39:54.904542+0000","last_undegraded":"2024-09-13T14:39:54.904542+0000","last_fullsized":"2024-09-13T14:39:54.904542+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:30:54.772513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":254,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:39:53.895732+0000","last_change":"2024-09-13T14:37:48.127607+0000","last_active":"2024-09-13T14:39:53.895732+0000","last_peered":"2024-09-13T14:39:53.895732+0000","last_clean":"2024-09-13T14:39:53.895732+0000","last_became_active":"2024-09-13T14:37:48.127316+0000","last_became_peered":"2024-09-13T14:37:48.127316+0000","last_unstale":"2024-09-13T14:39:53.895732+0000","last_undegraded":"2024-09-13T14:39:53.895732+0000","last_fullsized":"2024-09-13T14:39:53.895732+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:53:37.678861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":56,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:39:54.900705+0000","last_change":"2024-09-13T14:38:03.563200+0000","last_active":"2024-09-13T14:39:54.900705+0000","last_peered":"2024-09-13T14:39:54.900705+0000","last_clean":"2024-09-13T14:39:54.900705+0000","last_became_active":"2024-09-13T14:38:03.534258+0000","last_became_peered":"2024-09-13T14:38:03.534258+0000","last_unstale":"2024-09-13T14:39:54.900705+0000","last_undegraded":"2024-09-13T14:39:54.900705+0000","last_fullsized":"2024-09-13T14:39:54.900705+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:49:07.814419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":6123225112,"num_objects":93434,"num_object_clones":0,"num_object_copies":280302,"num_objects_missing_on_primary":3838,"num_objects_missing":3838,"num_objects_degraded":39854,"num_objects_misplaced":1987,"num_objects_unfound":0,"num_objects_dirty":93434,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132192,"num_write_kb":7219969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53814,"num_bytes_recovered":3495297024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19107160064,"data_stored":19107151920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91015,"ondisk_log_size":91015,"up":48,"acting":42,"num_store_stats":6},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1538,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":17412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25132,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":7,"num_objects_missing":7,"num_objects_degraded":16,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16793600,"data_stored":16777292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":23,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234354,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3427840,"kb_used_data":2977528,"kb_used_omap":1,"kb_used_meta":450302,"kb_avail":90317312,"statfs":{"total":95995035648,"available":92484927488,"internally_reserved":0,"allocated":3048988672,"data_stored":3047546366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":461109707},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":239,"seq":1026497183772,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2059332,"kb_used_data":1912508,"kb_used_omap":8,"kb_used_meta":146807,"kb_avail":91685820,"statfs":{"total":95995035648,"available":93886279680,"internally_reserved":0,"allocated":1958408192,"data_stored":1956962881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8453,"internal_metadata":150331131},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757619,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2460148,"kb_used_data":1855596,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":91285004,"statfs":{"total":95995035648,"available":93475844096,"internally_reserved":0,"allocated":1900130304,"data_stored":1898696007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":619051462},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019263,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3723200,"kb_used_data":3142456,"kb_used_omap":1,"kb_used_meta":580734,"kb_avail":90021952,"statfs":{"total":95995035648,"available":92182478848,"internally_reserved":0,"allocated":3217874944,"data_stored":3216433170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":594672070},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435446,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5029304,"kb_used_data":4492272,"kb_used_omap":5,"kb_used_meta":537018,"kb_avail":88715848,"statfs":{"total":95995035648,"available":90845028352,"internally_reserved":0,"allocated":4600086528,"data_stored":4598648465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":549906558},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":446676598974,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":425201762501,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926025,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4979644,"kb_used_data":4320436,"kb_used_omap":7,"kb_used_meta":659192,"kb_avail":88765508,"statfs":{"total":95995035648,"available":90895880192,"internally_reserved":0,"allocated":4424126464,"data_stored":4422684362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":675012904},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6860,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4413001728,"data_stored":4412997656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4595912704,"data_stored":4595908632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3212181504,"data_stored":3212181504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1891565568,"data_stored":1891565568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1948909568,"data_stored":1948909568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3045588992,"data_stored":3045588992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:40:17.422 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph status --format=json 2024-09-13T14:40:17.456 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:17.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:17.492 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:17.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:17.593 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:17.620 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:17.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:17.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:17.722 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:17.748 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:17.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:17.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:17.850 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:17.876 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:17.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:17.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:17.977 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:18.003 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:18.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:18.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:18.105 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:18.134 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:18.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:18.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:18.205 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:40:18.205 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:40:18.227 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:40:18.227 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:40:18.235 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:18.246 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:40:18.246 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:40:18.263 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:18.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:18.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:18.272 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:40:18.272 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:40:18.301 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:40:18.301 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:40:18.319 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:40:18.319 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:40:18.338 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:40:18.339 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:40:18.359 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:40:18.359 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:40:18.365 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:18.392 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:18.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:18.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:18.499 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:18.525 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:18.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:18.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:18.627 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:18.654 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:18.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:18.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:18.755 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:18.781 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:18.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:18.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:18.883 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:18.910 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:18.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:18.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:19.011 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:19.038 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:19.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:19.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:19.140 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:19.167 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:19.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:19.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:19.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:18 smithi195 ceph-mon[104262]: pgmap v864: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.6 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 169 op/s; 38536/273603 objects degraded (14.085%); 1990/273603 objects misplaced (0.727%); 7.0 MiB/s, 112 objects/s recovering 2024-09-13T14:40:19.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:18 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3833237844' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:40:19.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:18 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 38536/273603 objects degraded (14.085%), 12 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:19.269 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:19.294 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:19.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:19.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:19.334 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:18 smithi102 ceph-mon[110183]: pgmap v864: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.6 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 169 op/s; 38536/273603 objects degraded (14.085%); 1990/273603 objects misplaced (0.727%); 7.0 MiB/s, 112 objects/s recovering 2024-09-13T14:40:19.334 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:18 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3833237844' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:40:19.334 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:18 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 38536/273603 objects degraded (14.085%), 12 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:19.334 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:18 smithi102 ceph-mon[115269]: pgmap v864: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.6 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 169 op/s; 38536/273603 objects degraded (14.085%); 1990/273603 objects misplaced (0.727%); 7.0 MiB/s, 112 objects/s recovering 2024-09-13T14:40:19.335 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:18 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3833237844' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:40:19.335 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:18 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 38536/273603 objects degraded (14.085%), 12 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:19.396 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:19.421 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:19.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:19.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:19.523 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:19.552 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:19.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:19.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:19.653 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:19.658 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:40:19.658 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [], 2024-09-13T14:40:19.658 INFO:teuthology.orchestra.run.smithi102.stdout: "num_ops": 0 2024-09-13T14:40:19.658 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:40:19.660 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:19.687 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:19.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:19.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:19.789 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:19.813 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:19.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:19.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:19.915 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:19.940 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:19.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:19.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:20.042 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:20.069 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:20.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:20.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:20.171 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:20.200 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:20.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:20.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:20.301 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:20.323 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:20.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:20.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:20.426 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:20.448 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:20.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:20.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:20.549 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:20.577 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:20.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:20.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:20.679 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:20.701 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:20.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:20.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:20.722 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:40:20.803 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:20.831 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:20.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:20.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:20.933 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:20.958 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:20.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:20.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:21.060 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:21.088 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:21.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:21.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:21.190 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:21.221 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:21.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:21.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:21.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:20 smithi195 ceph-mon[104262]: pgmap v865: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.5 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 172 op/s; 38001/272280 objects degraded (13.957%); 1989/272280 objects misplaced (0.730%); 7.4 MiB/s, 119 objects/s recovering 2024-09-13T14:40:21.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:20 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:40:21.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:20 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:40:21.323 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:21.327 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:20 smithi102 ceph-mon[115269]: pgmap v865: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.5 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 172 op/s; 38001/272280 objects degraded (13.957%); 1989/272280 objects misplaced (0.730%); 7.4 MiB/s, 119 objects/s recovering 2024-09-13T14:40:21.327 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:20 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:40:21.328 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:20 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:40:21.328 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:20 smithi102 ceph-mon[110183]: pgmap v865: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.5 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 172 op/s; 38001/272280 objects degraded (13.957%); 1989/272280 objects misplaced (0.730%); 7.4 MiB/s, 119 objects/s recovering 2024-09-13T14:40:21.328 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:20 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:40:21.328 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:20 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:40:21.358 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:21.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:21.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:21.362 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:40:21.460 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:21.488 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:21.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:21.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:21.590 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:21.613 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:21.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:21.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:21.716 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:21.739 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:21.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:21.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:21.841 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:21.864 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:21.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:21.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:21.966 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:21.991 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:21.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:21.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:22.093 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:22.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:22.119 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:22.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:22.220 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:22.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:22.245 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:22.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:22.346 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:22.370 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:22.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:22.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:22.471 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:22.494 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:22.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:22.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:22.596 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:22.621 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:22.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:22.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:22.724 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:22.752 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:22.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:22.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:22.854 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:22.880 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:22.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:22.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:22.982 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:23.008 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:23.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:23.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:23.075 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T14:40:23.110 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:23.134 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:23.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:23.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:23.235 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:23.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:22 smithi195 ceph-mon[104262]: pgmap v866: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.5 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 172 op/s; 38001/272280 objects degraded (13.957%); 1989/272280 objects misplaced (0.730%); 7.4 MiB/s, 119 objects/s recovering 2024-09-13T14:40:23.259 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:23.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:23.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:23.361 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:23.384 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:23.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:23.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:23.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:22 smithi102 ceph-mon[110183]: pgmap v866: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.5 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 172 op/s; 38001/272280 objects degraded (13.957%); 1989/272280 objects misplaced (0.730%); 7.4 MiB/s, 119 objects/s recovering 2024-09-13T14:40:23.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:22 smithi102 ceph-mon[115269]: pgmap v866: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.5 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 172 op/s; 38001/272280 objects degraded (13.957%); 1989/272280 objects misplaced (0.730%); 7.4 MiB/s, 119 objects/s recovering 2024-09-13T14:40:23.485 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:23.509 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:23.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:23.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:23.611 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:23.636 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:23.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:23.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:23.738 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:23.763 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:23.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:23.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:23.865 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:23.886 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:23.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:23.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:23.988 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:24.014 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:24.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:24.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:24.116 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:24.138 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:24.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:24.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:24.155 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:24 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 38001/272280 objects degraded (13.957%), 12 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:24.155 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:24 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 38001/272280 objects degraded (13.957%), 12 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:24.240 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:24.265 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:24.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:24.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:24.366 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:24.396 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:24.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:24.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:24.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:24 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 38001/272280 objects degraded (13.957%), 12 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:24.497 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:24.524 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:24.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:24.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:24.626 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:24.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:24.649 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:24.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:24.747 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:40:24.747 INFO:teuthology.orchestra.run.smithi102.stdout:{"fsid":"08cee502-71d9-11ef-bceb-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 37875/270306 objects degraded (14.012%), 12 pgs degraded, 7 pgs undersized","count":19},"muted":false}},"mutes":[]},"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1248,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":265,"num_osds":8,"num_up_osds":8,"osd_up_since":1726238267,"num_in_osds":6,"osd_in_since":1726238173,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":77},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovery_wait+degraded","count":3},{"state_name":"active+recovery_wait+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_wait","count":1}],"num_pgs":89,"num_pools":5,"num_objects":90102,"data_bytes":5907814397,"bytes_used":21689610240,"bytes_avail":554280603648,"bytes_total":575970213888,"degraded_objects":37875,"degraded_total":270306,"degraded_ratio":0.14011897627133693,"misplaced_objects":1989,"misplaced_total":270306,"misplaced_ratio":0.0073583272291403079,"recovering_objects_per_sec":65,"recovering_bytes_per_sec":4264947,"recovering_keys_per_sec":0,"num_objects_recovered":651,"num_bytes_recovered":42663936,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":157},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi102.njetxr","status":"up:active","gid":24521}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.102:8443/","prometheus":"http://172.21.15.102:9283/"}},"servicemap":{"epoch":28,"modified":"2024-09-13T14:40:21.476600+0000","services":{}},"progress_events":{}} 2024-09-13T14:40:24.751 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:24.781 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:24.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:24.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:24.883 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:24.905 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:24.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:24.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:25.007 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:25.035 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:25.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:25.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:25.137 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:25.160 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:25.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:25.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:25.262 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:25.288 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:25.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:25.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:25.390 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:25.412 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:25.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:25.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:25.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:25 smithi102 ceph-mon[110183]: pgmap v867: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.5 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 157 op/s; 37875/270306 objects degraded (14.012%); 1989/270306 objects misplaced (0.736%); 4.1 MiB/s, 65 objects/s recovering 2024-09-13T14:40:25.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:25 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/922548040' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:40:25.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:25 smithi102 ceph-mon[115269]: pgmap v867: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.5 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 157 op/s; 37875/270306 objects degraded (14.012%); 1989/270306 objects misplaced (0.736%); 4.1 MiB/s, 65 objects/s recovering 2024-09-13T14:40:25.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:25 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/922548040' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:40:25.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:25 smithi195 ceph-mon[104262]: pgmap v867: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.5 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 157 op/s; 37875/270306 objects degraded (14.012%); 1989/270306 objects misplaced (0.736%); 4.1 MiB/s, 65 objects/s recovering 2024-09-13T14:40:25.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:25 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/922548040' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:40:25.513 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:25.513 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:25.521 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-13T14:40:25.521 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:40:25.554 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:25.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:25.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:25.579 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:25.582 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:25.656 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:25.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:25.685 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:25.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:25.787 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:25.811 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:25.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:25.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:25.913 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:25.937 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:25.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:25.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:26.039 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:26.061 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:26.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:26.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:26.165 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:26.188 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:26.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:26.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:26.290 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:26.313 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:26.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:26.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:26.415 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:26.439 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:26.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:26.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:26.542 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:26.566 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:26.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:26.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:26.611 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:40:26.611 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:40:26.611 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:26 smithi102 ceph-mon[110183]: pgmap v868: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.5 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 200 op/s; 37243/269010 objects degraded (13.844%); 1989/269010 objects misplaced (0.739%); 8.0 MiB/s, 128 objects/s recovering 2024-09-13T14:40:26.612 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:40:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:40:26] "GET /metrics HTTP/1.1" 200 36538 "" "Prometheus/2.51.0" 2024-09-13T14:40:26.612 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:40:26.612 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:40:26.612 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:26 smithi102 ceph-mon[115269]: pgmap v868: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.5 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 200 op/s; 37243/269010 objects degraded (13.844%); 1989/269010 objects misplaced (0.739%); 8.0 MiB/s, 128 objects/s recovering 2024-09-13T14:40:26.668 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:26.691 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:26.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:26.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:26.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:40:26.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:40:26.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:26 smithi195 ceph-mon[104262]: pgmap v868: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.5 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 200 op/s; 37243/269010 objects degraded (13.844%); 1989/269010 objects misplaced (0.739%); 8.0 MiB/s, 128 objects/s recovering 2024-09-13T14:40:26.793 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:26.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:26.820 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:26.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:26.921 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:26.944 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:26.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:26.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:27.046 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:27.072 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:27.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:27.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:27.174 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:27.197 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:27.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:27.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:27.299 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:27.323 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:27.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:27.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:27.425 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:27.453 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:27.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:27.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:27.555 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:27.579 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:27.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:27.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:27.681 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:27.703 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:27.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:27.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:27.805 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:27.827 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:27.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:27.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:27.929 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:27.953 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:27.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:27.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:28.055 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:28.079 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:28.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:28.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:28.181 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:28.202 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:28.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:28.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:28.304 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:28.328 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:28.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:28.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:28.430 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:28.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:28.453 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:28.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:28.554 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:28.578 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:28.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:28.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:28.680 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:28.702 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:28.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:28.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:28.804 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:28.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:28.826 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:28.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:28.840 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:28 smithi195 ceph-mon[104262]: pgmap v869: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.4 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 220 op/s; 37106/267432 objects degraded (13.875%); 1989/267432 objects misplaced (0.744%); 7.3 MiB/s, 117 objects/s recovering 2024-09-13T14:40:28.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:28 smithi102 ceph-mon[110183]: pgmap v869: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.4 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 220 op/s; 37106/267432 objects degraded (13.875%); 1989/267432 objects misplaced (0.744%); 7.3 MiB/s, 117 objects/s recovering 2024-09-13T14:40:28.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:28 smithi102 ceph-mon[115269]: pgmap v869: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.4 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 220 op/s; 37106/267432 objects degraded (13.875%); 1989/267432 objects misplaced (0.744%); 7.3 MiB/s, 117 objects/s recovering 2024-09-13T14:40:28.927 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:28.949 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:28.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:28.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:29.051 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:29.074 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:29.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:29.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:29.176 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:29.198 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:29.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:29.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:29.300 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:29.324 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:29.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:29.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:29.426 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:29.452 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:29.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:29.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:29.554 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:29.576 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:29.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:29.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:29.679 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:29.701 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:29.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:29.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:29.803 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:29.827 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:29.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:29.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:29.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:29 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 37106/267432 objects degraded (13.875%), 12 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:29.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:29 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 37106/267432 objects degraded (13.875%), 12 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:29.929 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:29.954 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:29.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:29.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:29.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:29 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 37106/267432 objects degraded (13.875%), 12 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:30.056 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:30.084 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:30.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:30.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:30.186 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:30.209 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:30.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:30.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:30.311 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:30.333 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:30.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:30.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:30.435 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:30.459 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:30.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:30.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:30.561 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:30.583 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:30.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:30.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:30.685 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:30.709 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:30.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:30.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:30.793 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:30 smithi102 ceph-mon[110183]: pgmap v870: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.4 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 203 op/s; 36276/266283 objects degraded (13.623%); 1989/266283 objects misplaced (0.747%); 11 MiB/s, 175 objects/s recovering 2024-09-13T14:40:30.793 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:30 smithi102 ceph-mon[115269]: pgmap v870: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.4 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 203 op/s; 36276/266283 objects degraded (13.623%); 1989/266283 objects misplaced (0.747%); 11 MiB/s, 175 objects/s recovering 2024-09-13T14:40:30.811 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:30.841 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:30.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:30.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:30.942 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:30.967 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:30.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:30.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:30.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:30 smithi195 ceph-mon[104262]: pgmap v870: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.4 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 203 op/s; 36276/266283 objects degraded (13.623%); 1989/266283 objects misplaced (0.747%); 11 MiB/s, 175 objects/s recovering 2024-09-13T14:40:31.069 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:31.092 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:31.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:31.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:31.194 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:31.222 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:31.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:31.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:31.323 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:31.347 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:31.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:31.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:31.449 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:31.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:31.480 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:31.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:31.581 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:31.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:31.610 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:31.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:31.711 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:31.743 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:31.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:31.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:31.845 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:31.868 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:31.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:31.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:31.970 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:31.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:31.997 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:31.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:32.098 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:32.123 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:32.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:32.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:32.225 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:32.252 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:32.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:32.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:32.355 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:32.379 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:32.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:32.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:32.481 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:32.503 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:32.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:32.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:32.605 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:32.627 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:32.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:32.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:32.729 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:32.753 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:32.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:32.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:32.803 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:32 smithi102 ceph-mon[110183]: pgmap v871: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.4 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 166 op/s; 36276/266283 objects degraded (13.623%); 1989/266283 objects misplaced (0.747%); 8.2 MiB/s, 131 objects/s recovering 2024-09-13T14:40:32.804 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:32 smithi102 ceph-mon[115269]: pgmap v871: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.4 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 166 op/s; 36276/266283 objects degraded (13.623%); 1989/266283 objects misplaced (0.747%); 8.2 MiB/s, 131 objects/s recovering 2024-09-13T14:40:32.855 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:32.889 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:32.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:32.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:32.990 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:32.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:32 smithi195 ceph-mon[104262]: pgmap v871: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.4 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 166 op/s; 36276/266283 objects degraded (13.623%); 1989/266283 objects misplaced (0.747%); 8.2 MiB/s, 131 objects/s recovering 2024-09-13T14:40:33.014 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:33.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:33.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:33.117 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:33.176 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:33.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:33.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:33.278 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:33.299 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:33.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:33.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:33.401 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:33.426 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:33.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:33.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:33.528 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:33.555 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:33.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:33.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:33.657 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:33.682 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:33.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:33.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:33.783 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:33.812 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:33.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:33.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:33.846 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:40:33.847 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:40:33.847 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:40:33.847 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:40:33.847 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:40:33.847 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:40:33.847 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:40:33.847 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:40:33.848 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:40:33.848 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:40:33.848 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:40:33.848 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:40:33.851 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:40:33.852 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:40:33.871 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:40:33.871 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:40:33.914 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:33.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:33.939 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:33.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:34.040 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:34.063 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:34.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:34.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:34.165 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:34.189 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:34.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:34.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:34.291 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:34.315 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:34.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:34.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:34.417 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:34.438 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:34.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:34.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:34.541 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:34.561 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is set 2024-09-13T14:40:34.567 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:34.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:34.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:34.669 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:34.694 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:34.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:34.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:34.796 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:34.819 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:34.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:34.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:34.889 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:34 smithi102 ceph-mon[110183]: pgmap v872: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.4 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 209 op/s; 36162/264729 objects degraded (13.660%); 1989/264729 objects misplaced (0.751%); 8.5 MiB/s, 136 objects/s recovering 2024-09-13T14:40:34.889 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:34 smithi102 ceph-mon[110183]: osdmap e266: 8 total, 8 up, 6 in 2024-09-13T14:40:34.890 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:34 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1419262957' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:40:34.890 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:34 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 36162/264729 objects degraded (13.660%), 12 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:34.890 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:34 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3362630103' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:40:34.890 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:34 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:40:34.894 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:34 smithi102 ceph-mon[115269]: pgmap v872: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.4 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 209 op/s; 36162/264729 objects degraded (13.660%); 1989/264729 objects misplaced (0.751%); 8.5 MiB/s, 136 objects/s recovering 2024-09-13T14:40:34.894 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:34 smithi102 ceph-mon[115269]: osdmap e266: 8 total, 8 up, 6 in 2024-09-13T14:40:34.894 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:34 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1419262957' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:40:34.894 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:34 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 36162/264729 objects degraded (13.660%), 12 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:34.894 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:34 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3362630103' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:40:34.894 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:34 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:40:34.920 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:34.943 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:34.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:34.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:34.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:34 smithi195 ceph-mon[104262]: pgmap v872: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.4 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 209 op/s; 36162/264729 objects degraded (13.660%); 1989/264729 objects misplaced (0.751%); 8.5 MiB/s, 136 objects/s recovering 2024-09-13T14:40:34.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:34 smithi195 ceph-mon[104262]: osdmap e266: 8 total, 8 up, 6 in 2024-09-13T14:40:34.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:34 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1419262957' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:40:34.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:34 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 36162/264729 objects degraded (13.660%), 12 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:34.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:34 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3362630103' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:40:34.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:34 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:40:35.046 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:35.071 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:35.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:35.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:35.173 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:35.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:35.196 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:35.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:35.297 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:35.318 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:35.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:35.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:35.377 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:40:35.377 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:40:35.420 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:35.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:35.442 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:35.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:35.543 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:35.565 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:35.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:35.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:35.669 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:35.694 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:35.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:35.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:35.797 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:35.820 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:35.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:35.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:35.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:35 smithi102 ceph-mon[110183]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:40:35.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:35 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:40:35.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:35 smithi102 ceph-mon[110183]: osdmap e267: 8 total, 8 up, 6 in 2024-09-13T14:40:35.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:35 smithi102 ceph-mon[115269]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:40:35.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:35 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:40:35.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:35 smithi102 ceph-mon[115269]: osdmap e267: 8 total, 8 up, 6 in 2024-09-13T14:40:35.922 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:35.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:35.947 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:35.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:35.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:35 smithi195 ceph-mon[104262]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:40:35.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:35 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:40:35.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:35 smithi195 ceph-mon[104262]: osdmap e267: 8 total, 8 up, 6 in 2024-09-13T14:40:36.048 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:36.072 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:36.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:36.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:36.174 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:36.200 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:36.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:36.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:36.302 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:36.324 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:36.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:36.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:36.361 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:36.387 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T14:40:36.426 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:36.454 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:36.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:36.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:36.556 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:36.580 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:36.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:36.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:36.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:36 smithi102 ceph-mon[110183]: pgmap v875: 89 pgs: 1 peering, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.4 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 152 op/s; 35172/263775 objects degraded (13.334%); 1989/263775 objects misplaced (0.754%); 15 MiB/s, 243 objects/s recovering 2024-09-13T14:40:36.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:40:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:40:36] "GET /metrics HTTP/1.1" 200 36538 "" "Prometheus/2.51.0" 2024-09-13T14:40:36.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:36 smithi102 ceph-mon[115269]: pgmap v875: 89 pgs: 1 peering, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.4 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 152 op/s; 35172/263775 objects degraded (13.334%); 1989/263775 objects misplaced (0.754%); 15 MiB/s, 243 objects/s recovering 2024-09-13T14:40:36.682 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:36.705 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:36.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:36.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:36.807 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:36.832 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:36.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:36.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:36.924 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:40:36.934 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:36.995 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:36.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:36.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:36.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:36 smithi195 ceph-mon[104262]: pgmap v875: 89 pgs: 1 peering, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.4 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 152 op/s; 35172/263775 objects degraded (13.334%); 1989/263775 objects misplaced (0.754%); 15 MiB/s, 243 objects/s recovering 2024-09-13T14:40:37.096 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:37.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:37.120 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:37.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:37.221 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:37.242 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:37.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:37.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:37.344 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:37.368 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:37.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:37.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:37.470 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:37.493 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:37.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:37.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:37.595 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:37.616 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:37.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:37.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:37.718 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:37.742 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:37.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:37.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:37.844 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:37.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:37.866 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:37.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:37.967 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:37.990 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:37.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:37.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:38.091 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:38.114 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:38.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:38.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:38.216 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:38.238 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:38.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:38.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:38.339 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:38.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:38.362 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:38.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:38.463 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:38.484 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:38.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:38.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:38.586 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:38.611 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:38.612 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:38.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:38.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:38.714 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:38.737 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:38.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:38.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:38.840 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:38.869 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:38.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:38.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:38.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:38 smithi102 ceph-mon[110183]: pgmap v876: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 1 peering, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.3 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 169 op/s; 30682/262203 objects degraded (11.702%); 6067/262203 objects misplaced (2.314%); 10 MiB/s, 175 objects/s recovering 2024-09-13T14:40:38.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:38 smithi102 ceph-mon[115269]: pgmap v876: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 1 peering, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.3 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 169 op/s; 30682/262203 objects degraded (11.702%); 6067/262203 objects misplaced (2.314%); 10 MiB/s, 175 objects/s recovering 2024-09-13T14:40:38.971 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:38.994 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:38.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:38.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:38.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:38 smithi195 ceph-mon[104262]: pgmap v876: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 1 peering, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 77 active+clean; 5.3 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 169 op/s; 30682/262203 objects degraded (11.702%); 6067/262203 objects misplaced (2.314%); 10 MiB/s, 175 objects/s recovering 2024-09-13T14:40:39.097 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:39.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:39.121 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:39.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:39.223 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:39.245 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:39.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:39.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:39.347 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:39.369 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:39.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:39.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:39.471 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:39.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:39.498 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:39.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:39.588 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:39.599 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:39.622 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:39.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:39.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:39.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:39 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 30682/262203 objects degraded (11.702%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:39.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:39 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 30682/262203 objects degraded (11.702%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:39.724 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:39.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:39 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 30682/262203 objects degraded (11.702%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:39.747 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:39.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:39.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:39.849 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:39.876 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:39.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:39.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:39.978 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:40.003 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:40.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:40.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:40.104 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:40.131 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:40.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:40.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:40.149 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:40:40.233 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:40.255 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:40.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:40.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:40.357 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:40.383 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:40.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:40.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:40.485 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:40.516 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:40.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:40.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:40.618 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:40.630 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:40 smithi102 ceph-mon[110183]: pgmap v877: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 5.3 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 218 op/s; 30632/261042 objects degraded (11.735%); 6067/261042 objects misplaced (2.324%); 11 MiB/s, 181 objects/s recovering 2024-09-13T14:40:40.630 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:40 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:40:40.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:40 smithi102 ceph-mon[115269]: pgmap v877: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 5.3 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 218 op/s; 30632/261042 objects degraded (11.735%); 6067/261042 objects misplaced (2.324%); 11 MiB/s, 181 objects/s recovering 2024-09-13T14:40:40.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:40 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:40:40.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:40.641 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:40.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:40.743 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:40.771 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:40.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:40.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:40.873 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:40.897 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:40.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:40.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:40.955 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:40:40.956 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:40:40.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:40 smithi195 ceph-mon[104262]: pgmap v877: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 5.3 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 218 op/s; 30632/261042 objects degraded (11.735%); 6067/261042 objects misplaced (2.324%); 11 MiB/s, 181 objects/s recovering 2024-09-13T14:40:40.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:40 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:40:40.998 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:41.020 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:41.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:41.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:41.121 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:41.144 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:41.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:41.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:41.247 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:41.269 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:41.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:41.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:41.371 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:41.396 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:41.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:41.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:41.498 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:41.520 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:41.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:41.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:41.622 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:41.644 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:41.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:41.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:41.746 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:41.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:41.768 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:41.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:41.847 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:41 smithi102 ceph-mon[110183]: from='client.45635 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:40:41.847 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:41 smithi102 ceph-mon[115269]: from='client.45635 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:40:41.869 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:41.892 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:41.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:41.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:41.994 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:41.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:41 smithi195 ceph-mon[104262]: from='client.45635 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:40:42.027 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:42.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:42.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:42.130 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:42.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:42.153 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:42.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:42.254 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:42.276 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:42.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:42.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:42.378 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:42.401 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:42.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:42.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:42.503 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:42.526 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:42.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:42.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:42.629 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:42.653 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:42.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:42.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:42.755 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:42.783 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:42.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:42.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:42.885 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:42.909 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:42.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:42.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:42.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:42 smithi102 ceph-mon[110183]: pgmap v878: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 5.3 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 153 op/s; 30632/261042 objects degraded (11.735%); 6067/261042 objects misplaced (2.324%); 10 MiB/s, 174 objects/s recovering 2024-09-13T14:40:42.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:42 smithi102 ceph-mon[115269]: pgmap v878: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 5.3 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 153 op/s; 30632/261042 objects degraded (11.735%); 6067/261042 objects misplaced (2.324%); 10 MiB/s, 174 objects/s recovering 2024-09-13T14:40:42.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:42 smithi195 ceph-mon[104262]: pgmap v878: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 5.3 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 153 op/s; 30632/261042 objects degraded (11.735%); 6067/261042 objects misplaced (2.324%); 10 MiB/s, 174 objects/s recovering 2024-09-13T14:40:43.012 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:43.039 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:43.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:43.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:43.142 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:43.166 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:43.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:43.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:43.268 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:43.292 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:43.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:43.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:43.394 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:43.420 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:43.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:43.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:43.522 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:43.548 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:43.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:43.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:43.650 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:43.675 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:43.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:43.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:43.777 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:43.798 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:43.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:43.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:43.900 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:43.925 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:43.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:43.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:43.966 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":877,"stamp":"2024-09-13T14:40:39.482161+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5705439229,"num_objects":87014,"num_object_clones":0,"num_object_copies":261042,"num_objects_missing_on_primary":3428,"num_objects_missing":3428,"num_objects_degraded":30632,"num_objects_misplaced":6067,"num_objects_unfound":0,"num_objects_dirty":87014,"num_whiteouts":0,"num_read":6377,"num_read_kb":13834,"num_write":139763,"num_write_kb":7235010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58668,"num_bytes_recovered":3808038029,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93445,"ondisk_log_size":93445,"up":267,"acting":261,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":309,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":20845032,"kb_used_data":17791408,"kb_used_omap":26,"kb_used_meta":3053541,"kb_avail":541625880,"statfs":{"total":575970213888,"available":554624901120,"internally_reserved":0,"allocated":18218401792,"data_stored":18209659937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27144,"internal_metadata":3126826488},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2,1,1,0,5,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-114491392,"num_objects":-1747,"num_object_clones":0,"num_object_copies":-5241,"num_objects_missing_on_primary":-123,"num_objects_missing":-123,"num_objects_degraded":-5644,"num_objects_misplaced":4078,"num_objects_unfound":0,"num_objects_dirty":-1747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1747,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1455,"num_bytes_recovered":91029528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002032"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":426,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.559102+0000","last_change":"2024-09-13T14:36:13.859927+0000","last_active":"2024-09-13T14:40:34.559102+0000","last_peered":"2024-09-13T14:40:34.559102+0000","last_clean":"2024-09-13T14:40:34.559102+0000","last_became_active":"2024-09-13T14:36:13.859505+0000","last_became_peered":"2024-09-13T14:36:13.859505+0000","last_unstale":"2024-09-13T14:40:34.559102+0000","last_undegraded":"2024-09-13T14:40:34.559102+0000","last_fullsized":"2024-09-13T14:40:34.559102+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:12:20.989627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":268,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.553366+0000","last_change":"2024-09-13T14:37:48.130968+0000","last_active":"2024-09-13T14:40:33.553366+0000","last_peered":"2024-09-13T14:40:33.553366+0000","last_clean":"2024-09-13T14:40:33.553366+0000","last_became_active":"2024-09-13T14:37:48.130603+0000","last_became_peered":"2024-09-13T14:37:48.130603+0000","last_unstale":"2024-09-13T14:40:33.553366+0000","last_undegraded":"2024-09-13T14:40:33.553366+0000","last_fullsized":"2024-09-13T14:40:33.553366+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:42.626386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":64,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.556939+0000","last_change":"2024-09-13T14:36:13.862181+0000","last_active":"2024-09-13T14:40:33.556939+0000","last_peered":"2024-09-13T14:40:33.556939+0000","last_clean":"2024-09-13T14:40:33.556939+0000","last_became_active":"2024-09-13T14:36:13.861873+0000","last_became_peered":"2024-09-13T14:36:13.861873+0000","last_unstale":"2024-09-13T14:40:33.556939+0000","last_undegraded":"2024-09-13T14:40:33.556939+0000","last_fullsized":"2024-09-13T14:40:33.556939+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:14.085700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":64,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.561499+0000","last_change":"2024-09-13T14:38:53.896672+0000","last_active":"2024-09-13T14:40:34.561499+0000","last_peered":"2024-09-13T14:40:34.561499+0000","last_clean":"2024-09-13T14:40:34.561499+0000","last_became_active":"2024-09-13T14:38:53.896346+0000","last_became_peered":"2024-09-13T14:38:53.896346+0000","last_unstale":"2024-09-13T14:40:34.561499+0000","last_undegraded":"2024-09-13T14:40:34.561499+0000","last_fullsized":"2024-09-13T14:40:34.561499+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:57.789686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":64,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.561760+0000","last_change":"2024-09-13T14:38:53.893021+0000","last_active":"2024-09-13T14:40:34.561760+0000","last_peered":"2024-09-13T14:40:34.561760+0000","last_clean":"2024-09-13T14:40:34.561760+0000","last_became_active":"2024-09-13T14:38:53.892668+0000","last_became_peered":"2024-09-13T14:38:53.892668+0000","last_unstale":"2024-09-13T14:40:34.561760+0000","last_undegraded":"2024-09-13T14:40:34.561760+0000","last_fullsized":"2024-09-13T14:40:34.561760+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:57.180738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":236,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.553185+0000","last_change":"2024-09-13T14:37:48.134642+0000","last_active":"2024-09-13T14:40:33.553185+0000","last_peered":"2024-09-13T14:40:33.553185+0000","last_clean":"2024-09-13T14:40:33.553185+0000","last_became_active":"2024-09-13T14:37:48.134402+0000","last_became_peered":"2024-09-13T14:37:48.134402+0000","last_unstale":"2024-09-13T14:40:33.553185+0000","last_undegraded":"2024-09-13T14:40:33.553185+0000","last_fullsized":"2024-09-13T14:40:33.553185+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:51.313663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":445,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.559667+0000","last_change":"2024-09-13T14:38:53.903022+0000","last_active":"2024-09-13T14:40:34.559667+0000","last_peered":"2024-09-13T14:40:34.559667+0000","last_clean":"2024-09-13T14:40:34.559667+0000","last_became_active":"2024-09-13T14:38:53.902448+0000","last_became_peered":"2024-09-13T14:38:53.902448+0000","last_unstale":"2024-09-13T14:40:34.559667+0000","last_undegraded":"2024-09-13T14:40:34.559667+0000","last_fullsized":"2024-09-13T14:40:34.559667+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:24.690401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":67,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.561052+0000","last_change":"2024-09-13T14:38:03.554500+0000","last_active":"2024-09-13T14:40:34.561052+0000","last_peered":"2024-09-13T14:40:34.561052+0000","last_clean":"2024-09-13T14:40:34.561052+0000","last_became_active":"2024-09-13T14:38:03.532910+0000","last_became_peered":"2024-09-13T14:38:03.532910+0000","last_unstale":"2024-09-13T14:40:34.561052+0000","last_undegraded":"2024-09-13T14:40:34.561052+0000","last_fullsized":"2024-09-13T14:40:34.561052+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:27:50.843498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"267'8548","reported_seq":11101,"reported_epoch":267,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-13T14:40:36.633151+0000","last_change":"2024-09-13T14:40:32.799848+0000","last_active":"2024-09-13T14:40:36.633151+0000","last_peered":"2024-09-13T14:40:36.633151+0000","last_clean":"2024-09-13T14:37:46.417003+0000","last_became_active":"2024-09-13T14:37:49.690961+0000","last_became_peered":"2024-09-13T14:37:49.690961+0000","last_unstale":"2024-09-13T14:40:36.633151+0000","last_undegraded":"2024-09-13T14:40:36.633151+0000","last_fullsized":"2024-09-13T14:37:49.126746+0000","mapping_epoch":240,"log_start":"212'2800","ondisk_log_start":"212'2800","created":205,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5748,"log_dups_size":0,"ondisk_log_size":5748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:48:52.526429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":349437952,"num_objects":5332,"num_object_clones":0,"num_object_copies":15996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4078,"num_objects_unfound":0,"num_objects_dirty":5332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8548,"num_write_kb":444160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4995,"num_bytes_recovered":327352320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,4],"avail_no_missing":["5","0","4"],"object_location_counts":[{"shards":"0,4,5","objects":4078},{"shards":"4,5","objects":1254}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":28,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.553961+0000","last_change":"2024-09-13T14:38:53.940238+0000","last_active":"2024-09-13T14:40:33.553961+0000","last_peered":"2024-09-13T14:40:33.553961+0000","last_clean":"2024-09-13T14:40:33.553961+0000","last_became_active":"2024-09-13T14:38:53.892825+0000","last_became_peered":"2024-09-13T14:38:53.892825+0000","last_unstale":"2024-09-13T14:40:33.553961+0000","last_undegraded":"2024-09-13T14:40:33.553961+0000","last_fullsized":"2024-09-13T14:40:33.553961+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:46.666581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":85,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.573478+0000","last_change":"2024-09-13T14:38:53.898907+0000","last_active":"2024-09-13T14:40:34.573478+0000","last_peered":"2024-09-13T14:40:34.573478+0000","last_clean":"2024-09-13T14:40:34.573478+0000","last_became_active":"2024-09-13T14:38:53.898572+0000","last_became_peered":"2024-09-13T14:38:53.898572+0000","last_unstale":"2024-09-13T14:40:34.573478+0000","last_undegraded":"2024-09-13T14:40:34.573478+0000","last_fullsized":"2024-09-13T14:40:34.573478+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:29.222150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"267'8728","reported_seq":22871,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:36.805005+0000","last_change":"2024-09-13T14:38:57.930032+0000","last_active":"2024-09-13T14:40:36.805005+0000","last_peered":"2024-09-13T14:40:36.805005+0000","last_clean":"2024-09-13T14:40:36.805005+0000","last_became_active":"2024-09-13T14:38:57.921458+0000","last_became_peered":"2024-09-13T14:38:57.921458+0000","last_unstale":"2024-09-13T14:40:36.805005+0000","last_undegraded":"2024-09-13T14:40:36.805005+0000","last_fullsized":"2024-09-13T14:40:36.805005+0000","mapping_epoch":256,"log_start":"212'3439","ondisk_log_start":"212'3439","created":205,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5289,"log_dups_size":0,"ondisk_log_size":5289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:27:12.294038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":361365504,"num_objects":5514,"num_object_clones":0,"num_object_copies":16542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8728,"num_write_kb":455744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6990,"num_bytes_recovered":458096640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":28,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.553782+0000","last_change":"2024-09-13T14:38:53.904885+0000","last_active":"2024-09-13T14:40:33.553782+0000","last_peered":"2024-09-13T14:40:33.553782+0000","last_clean":"2024-09-13T14:40:33.553782+0000","last_became_active":"2024-09-13T14:38:53.891726+0000","last_became_peered":"2024-09-13T14:38:53.891726+0000","last_unstale":"2024-09-13T14:40:33.553782+0000","last_undegraded":"2024-09-13T14:40:33.553782+0000","last_fullsized":"2024-09-13T14:40:33.553782+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:57.079205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":274,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.551024+0000","last_change":"2024-09-13T14:37:48.160504+0000","last_active":"2024-09-13T14:40:33.551024+0000","last_peered":"2024-09-13T14:40:33.551024+0000","last_clean":"2024-09-13T14:40:33.551024+0000","last_became_active":"2024-09-13T14:37:48.160181+0000","last_became_peered":"2024-09-13T14:37:48.160181+0000","last_unstale":"2024-09-13T14:40:33.551024+0000","last_undegraded":"2024-09-13T14:40:33.551024+0000","last_fullsized":"2024-09-13T14:40:33.551024+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:37.087008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"267'8719","reported_seq":10484,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:38.301288+0000","last_change":"2024-09-13T14:38:04.531914+0000","last_active":"2024-09-13T14:40:38.301288+0000","last_peered":"2024-09-13T14:40:38.301288+0000","last_clean":"2024-09-13T14:40:38.301288+0000","last_became_active":"2024-09-13T14:38:04.523448+0000","last_became_peered":"2024-09-13T14:38:04.523448+0000","last_unstale":"2024-09-13T14:40:38.301288+0000","last_undegraded":"2024-09-13T14:40:38.301288+0000","last_fullsized":"2024-09-13T14:40:38.301288+0000","mapping_epoch":245,"log_start":"212'2800","ondisk_log_start":"212'2800","created":205,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5919,"log_dups_size":0,"ondisk_log_size":5919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:06.536896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":357498880,"num_objects":5455,"num_object_clones":0,"num_object_copies":16365,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8719,"num_write_kb":453568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":811,"num_bytes_recovered":53149696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":265,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.553681+0000","last_change":"2024-09-13T14:37:49.611774+0000","last_active":"2024-09-13T14:40:33.553681+0000","last_peered":"2024-09-13T14:40:33.553681+0000","last_clean":"2024-09-13T14:40:33.553681+0000","last_became_active":"2024-09-13T14:37:49.611088+0000","last_became_peered":"2024-09-13T14:37:49.611088+0000","last_unstale":"2024-09-13T14:40:33.553681+0000","last_undegraded":"2024-09-13T14:40:33.553681+0000","last_fullsized":"2024-09-13T14:40:33.553681+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:08.933213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":24,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.553252+0000","last_change":"2024-09-13T14:38:53.899211+0000","last_active":"2024-09-13T14:40:33.553252+0000","last_peered":"2024-09-13T14:40:33.553252+0000","last_clean":"2024-09-13T14:40:33.553252+0000","last_became_active":"2024-09-13T14:38:53.898927+0000","last_became_peered":"2024-09-13T14:38:53.898927+0000","last_unstale":"2024-09-13T14:40:33.553252+0000","last_undegraded":"2024-09-13T14:40:33.553252+0000","last_fullsized":"2024-09-13T14:40:33.553252+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:00:28.857611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"267'8580","reported_seq":8653,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:37.125884+0000","last_change":"2024-09-13T14:35:24.004651+0000","last_active":"2024-09-13T14:40:37.125884+0000","last_peered":"2024-09-13T14:40:37.125884+0000","last_clean":"2024-09-13T14:40:37.125884+0000","last_became_active":"2024-09-13T14:35:24.004358+0000","last_became_peered":"2024-09-13T14:35:24.004358+0000","last_unstale":"2024-09-13T14:40:37.125884+0000","last_undegraded":"2024-09-13T14:40:37.125884+0000","last_fullsized":"2024-09-13T14:40:37.125884+0000","mapping_epoch":205,"log_start":"210'2284","ondisk_log_start":"210'2284","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6296,"log_dups_size":0,"ondisk_log_size":6296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:57:38.023669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353370112,"num_objects":5392,"num_object_clones":0,"num_object_copies":16176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8580,"num_write_kb":447104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":75,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.556613+0000","last_change":"2024-09-13T14:37:49.168003+0000","last_active":"2024-09-13T14:40:33.556613+0000","last_peered":"2024-09-13T14:40:33.556613+0000","last_clean":"2024-09-13T14:40:33.556613+0000","last_became_active":"2024-09-13T14:37:49.167638+0000","last_became_peered":"2024-09-13T14:37:49.167638+0000","last_unstale":"2024-09-13T14:40:33.556613+0000","last_undegraded":"2024-09-13T14:40:33.556613+0000","last_fullsized":"2024-09-13T14:40:33.556613+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:29.051981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":57,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.561426+0000","last_change":"2024-09-13T14:38:03.528381+0000","last_active":"2024-09-13T14:40:34.561426+0000","last_peered":"2024-09-13T14:40:34.561426+0000","last_clean":"2024-09-13T14:40:34.561426+0000","last_became_active":"2024-09-13T14:38:03.527282+0000","last_became_peered":"2024-09-13T14:38:03.527282+0000","last_unstale":"2024-09-13T14:40:34.561426+0000","last_undegraded":"2024-09-13T14:40:34.561426+0000","last_fullsized":"2024-09-13T14:40:34.561426+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:24.435067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.f","version":"267'8657","reported_seq":8917,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:38.497260+0000","last_change":"2024-09-13T14:38:16.371079+0000","last_active":"2024-09-13T14:40:38.497260+0000","last_peered":"2024-09-13T14:40:38.497260+0000","last_clean":"2024-09-13T14:40:38.497260+0000","last_became_active":"2024-09-13T14:38:12.859883+0000","last_became_peered":"2024-09-13T14:38:12.859883+0000","last_unstale":"2024-09-13T14:40:38.497260+0000","last_undegraded":"2024-09-13T14:40:38.497260+0000","last_fullsized":"2024-09-13T14:40:38.497260+0000","mapping_epoch":245,"log_start":"212'2763","ondisk_log_start":"212'2763","created":205,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5894,"log_dups_size":0,"ondisk_log_size":5894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:08.436532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":355794944,"num_objects":5429,"num_object_clones":0,"num_object_copies":16287,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8657,"num_write_kb":450752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":3670016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":415,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.558977+0000","last_change":"2024-09-13T14:38:03.549595+0000","last_active":"2024-09-13T14:40:34.558977+0000","last_peered":"2024-09-13T14:40:34.558977+0000","last_clean":"2024-09-13T14:40:34.558977+0000","last_became_active":"2024-09-13T14:38:03.533331+0000","last_became_peered":"2024-09-13T14:38:03.533331+0000","last_unstale":"2024-09-13T14:40:34.558977+0000","last_undegraded":"2024-09-13T14:40:34.558977+0000","last_fullsized":"2024-09-13T14:40:34.558977+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:59:02.470343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":249,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.551263+0000","last_change":"2024-09-13T14:37:48.127600+0000","last_active":"2024-09-13T14:40:33.551263+0000","last_peered":"2024-09-13T14:40:33.551263+0000","last_clean":"2024-09-13T14:40:33.551263+0000","last_became_active":"2024-09-13T14:37:48.127349+0000","last_became_peered":"2024-09-13T14:37:48.127349+0000","last_unstale":"2024-09-13T14:40:33.551263+0000","last_undegraded":"2024-09-13T14:40:33.551263+0000","last_fullsized":"2024-09-13T14:40:33.551263+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:57.268374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"267'8545","reported_seq":11734,"reported_epoch":267,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:40:36.727065+0000","last_change":"2024-09-13T14:38:14.979229+0000","last_active":"2024-09-13T14:40:36.727065+0000","last_peered":"2024-09-13T14:40:36.727065+0000","last_clean":"2024-09-13T14:36:13.131873+0000","last_became_active":"2024-09-13T14:38:10.443954+0000","last_became_peered":"2024-09-13T14:38:10.443954+0000","last_unstale":"2024-09-13T14:40:36.727065+0000","last_undegraded":"2024-09-13T14:38:10.384643+0000","last_fullsized":"2024-09-13T14:38:04.517046+0000","mapping_epoch":245,"log_start":"212'2800","ondisk_log_start":"212'2800","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5745,"log_dups_size":0,"ondisk_log_size":5745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:26:53.493170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358154240,"num_objects":5465,"num_object_clones":0,"num_object_copies":16395,"num_objects_missing_on_primary":3238,"num_objects_missing":3238,"num_objects_degraded":3379,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8545,"num_write_kb":448320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1535,"num_bytes_recovered":100532224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":1},{"shards":"0,3,5,7","objects":12},{"shards":"0,3,7","objects":3237},{"shards":"0,5","objects":2210},{"shards":"3,5,7","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":183,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.551100+0000","last_change":"2024-09-13T14:37:48.127604+0000","last_active":"2024-09-13T14:40:33.551100+0000","last_peered":"2024-09-13T14:40:33.551100+0000","last_clean":"2024-09-13T14:40:33.551100+0000","last_became_active":"2024-09-13T14:37:48.127300+0000","last_became_peered":"2024-09-13T14:37:48.127300+0000","last_unstale":"2024-09-13T14:40:33.551100+0000","last_undegraded":"2024-09-13T14:40:33.551100+0000","last_fullsized":"2024-09-13T14:40:33.551100+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:51.999506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":302,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.551708+0000","last_change":"2024-09-13T14:38:03.529967+0000","last_active":"2024-09-13T14:40:33.551708+0000","last_peered":"2024-09-13T14:40:33.551708+0000","last_clean":"2024-09-13T14:40:33.551708+0000","last_became_active":"2024-09-13T14:38:03.529646+0000","last_became_peered":"2024-09-13T14:38:03.529646+0000","last_unstale":"2024-09-13T14:40:33.551708+0000","last_undegraded":"2024-09-13T14:40:33.551708+0000","last_fullsized":"2024-09-13T14:40:33.551708+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:52:05.193279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"267'8706","reported_seq":17841,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:38.249872+0000","last_change":"2024-09-13T14:40:34.583237+0000","last_active":"2024-09-13T14:40:38.249872+0000","last_peered":"2024-09-13T14:40:38.249872+0000","last_clean":"2024-09-13T14:40:38.249872+0000","last_became_active":"2024-09-13T14:40:34.563989+0000","last_became_peered":"2024-09-13T14:40:34.563989+0000","last_unstale":"2024-09-13T14:40:38.249872+0000","last_undegraded":"2024-09-13T14:40:38.249872+0000","last_fullsized":"2024-09-13T14:40:38.249872+0000","mapping_epoch":266,"log_start":"212'2800","ondisk_log_start":"212'2800","created":205,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5906,"log_dups_size":0,"ondisk_log_size":5906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:34:43.006051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":356057112,"num_objects":5434,"num_object_clones":0,"num_object_copies":16302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5434,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8706,"num_write_kb":452417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4489,"num_bytes_recovered":289865752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":263,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.551273+0000","last_change":"2024-09-13T14:36:13.860367+0000","last_active":"2024-09-13T14:40:33.551273+0000","last_peered":"2024-09-13T14:40:33.551273+0000","last_clean":"2024-09-13T14:40:33.551273+0000","last_became_active":"2024-09-13T14:36:13.859982+0000","last_became_peered":"2024-09-13T14:36:13.859982+0000","last_unstale":"2024-09-13T14:40:33.551273+0000","last_undegraded":"2024-09-13T14:40:33.551273+0000","last_fullsized":"2024-09-13T14:40:33.551273+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:08:23.311350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":258,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.551655+0000","last_change":"2024-09-13T14:37:48.147360+0000","last_active":"2024-09-13T14:40:33.551655+0000","last_peered":"2024-09-13T14:40:33.551655+0000","last_clean":"2024-09-13T14:40:33.551655+0000","last_became_active":"2024-09-13T14:37:48.146885+0000","last_became_peered":"2024-09-13T14:37:48.146885+0000","last_unstale":"2024-09-13T14:40:33.551655+0000","last_undegraded":"2024-09-13T14:40:33.551655+0000","last_fullsized":"2024-09-13T14:40:33.551655+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:27.841810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"267'8641","reported_seq":17883,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:37.330996+0000","last_change":"2024-09-13T14:38:39.650332+0000","last_active":"2024-09-13T14:40:37.330996+0000","last_peered":"2024-09-13T14:40:37.330996+0000","last_clean":"2024-09-13T14:40:37.330996+0000","last_became_active":"2024-09-13T14:38:39.640835+0000","last_became_peered":"2024-09-13T14:38:39.640835+0000","last_unstale":"2024-09-13T14:40:37.330996+0000","last_undegraded":"2024-09-13T14:40:37.330996+0000","last_fullsized":"2024-09-13T14:40:37.330996+0000","mapping_epoch":251,"log_start":"212'3351","ondisk_log_start":"212'3351","created":205,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5290,"log_dups_size":0,"ondisk_log_size":5290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:48.176606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":356057088,"num_objects":5433,"num_object_clones":0,"num_object_copies":16299,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8641,"num_write_kb":450368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4544,"num_bytes_recovered":292945920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":433,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.558682+0000","last_change":"2024-09-13T14:38:53.900041+0000","last_active":"2024-09-13T14:40:34.558682+0000","last_peered":"2024-09-13T14:40:34.558682+0000","last_clean":"2024-09-13T14:40:34.558682+0000","last_became_active":"2024-09-13T14:38:53.899203+0000","last_became_peered":"2024-09-13T14:38:53.899203+0000","last_unstale":"2024-09-13T14:40:34.558682+0000","last_undegraded":"2024-09-13T14:40:34.558682+0000","last_fullsized":"2024-09-13T14:40:34.558682+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:28:07.883094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":365,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.559811+0000","last_change":"2024-09-13T14:26:15.823198+0000","last_active":"2024-09-13T14:40:34.559811+0000","last_peered":"2024-09-13T14:40:34.559811+0000","last_clean":"2024-09-13T14:40:34.559811+0000","last_became_active":"2024-09-13T14:26:15.822717+0000","last_became_peered":"2024-09-13T14:26:15.822717+0000","last_unstale":"2024-09-13T14:40:34.559811+0000","last_undegraded":"2024-09-13T14:40:34.559811+0000","last_fullsized":"2024-09-13T14:40:34.559811+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:22.471622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.6","version":"267'8770","reported_seq":12050,"reported_epoch":267,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-13T14:40:38.332705+0000","last_change":"2024-09-13T14:38:55.132683+0000","last_active":"2024-09-13T14:40:38.332705+0000","last_peered":"2024-09-13T14:40:38.332705+0000","last_clean":"2024-09-13T14:36:13.124368+0000","last_became_active":"2024-09-13T14:38:55.120883+0000","last_became_peered":"2024-09-13T14:38:55.120883+0000","last_unstale":"2024-09-13T14:40:38.332705+0000","last_undegraded":"2024-09-13T14:38:55.026610+0000","last_fullsized":"2024-09-13T14:40:38.332705+0000","mapping_epoch":254,"log_start":"212'3119","ondisk_log_start":"212'3119","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5651,"log_dups_size":0,"ondisk_log_size":5651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":355729408,"num_objects":5428,"num_object_clones":0,"num_object_copies":16284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3441,"num_objects_misplaced":1987,"num_objects_unfound":0,"num_objects_dirty":5428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8770,"num_write_kb":454336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1564,"num_bytes_recovered":91291648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,3,5],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3,4","objects":3162},{"shards":"0,3,5","objects":2266}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":429,"reported_epoch":267,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-13T14:40:34.558590+0000","last_change":"2024-09-13T14:38:54.899624+0000","last_active":"2024-09-13T14:40:34.558590+0000","last_peered":"2024-09-13T14:40:34.558590+0000","last_clean":"2024-09-13T14:17:27.401592+0000","last_became_active":"2024-09-13T14:38:54.899282+0000","last_became_peered":"2024-09-13T14:38:54.899282+0000","last_unstale":"2024-09-13T14:40:34.558590+0000","last_undegraded":"2024-09-13T14:38:54.891176+0000","last_fullsized":"2024-09-13T14:40:34.558590+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:07:41.927740+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[3,0,6],"avail_no_missing":["3","4","6"],"object_location_counts":[{"shards":"3,4,6","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":505,"reported_epoch":266,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:40:33.556725+0000","last_change":"2024-09-13T14:38:54.904329+0000","last_active":"2024-09-13T14:40:33.556725+0000","last_peered":"2024-09-13T14:40:33.556725+0000","last_clean":"2024-09-13T14:38:39.648017+0000","last_became_active":"2024-09-13T14:38:54.903798+0000","last_became_peered":"2024-09-13T14:38:54.903798+0000","last_unstale":"2024-09-13T14:40:33.556725+0000","last_undegraded":"2024-09-13T14:38:54.890505+0000","last_fullsized":"2024-09-13T14:38:54.889701+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:18.120931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,3],"avail_no_missing":["5","3","6"],"object_location_counts":[{"shards":"3,5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":367,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.559320+0000","last_change":"2024-09-13T14:37:49.160719+0000","last_active":"2024-09-13T14:40:34.559320+0000","last_peered":"2024-09-13T14:40:34.559320+0000","last_clean":"2024-09-13T14:40:34.559320+0000","last_became_active":"2024-09-13T14:37:49.160401+0000","last_became_peered":"2024-09-13T14:37:49.160401+0000","last_unstale":"2024-09-13T14:40:34.559320+0000","last_undegraded":"2024-09-13T14:40:34.559320+0000","last_fullsized":"2024-09-13T14:40:34.559320+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:21.737945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":110,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.551634+0000","last_change":"2024-09-13T14:37:48.172940+0000","last_active":"2024-09-13T14:40:33.551634+0000","last_peered":"2024-09-13T14:40:33.551634+0000","last_clean":"2024-09-13T14:40:33.551634+0000","last_became_active":"2024-09-13T14:37:48.172060+0000","last_became_peered":"2024-09-13T14:37:48.172060+0000","last_unstale":"2024-09-13T14:40:33.551634+0000","last_undegraded":"2024-09-13T14:40:33.551634+0000","last_fullsized":"2024-09-13T14:40:33.551634+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:07.480645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.5","version":"267'8663","reported_seq":25012,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:37.072019+0000","last_change":"2024-09-13T14:39:34.446699+0000","last_active":"2024-09-13T14:40:37.072019+0000","last_peered":"2024-09-13T14:40:37.072019+0000","last_clean":"2024-09-13T14:40:37.072019+0000","last_became_active":"2024-09-13T14:38:04.600616+0000","last_became_peered":"2024-09-13T14:38:04.600616+0000","last_unstale":"2024-09-13T14:40:37.072019+0000","last_undegraded":"2024-09-13T14:40:37.072019+0000","last_fullsized":"2024-09-13T14:40:37.072019+0000","mapping_epoch":245,"log_start":"210'2316","ondisk_log_start":"210'2316","created":205,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6347,"log_dups_size":0,"ondisk_log_size":6347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:00:44.321678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":359071744,"num_objects":5479,"num_object_clones":0,"num_object_copies":16437,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8663,"num_write_kb":452544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8120,"num_bytes_recovered":521076736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":483,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.553001+0000","last_change":"2024-09-13T14:26:15.835178+0000","last_active":"2024-09-13T14:40:33.553001+0000","last_peered":"2024-09-13T14:40:33.553001+0000","last_clean":"2024-09-13T14:40:33.553001+0000","last_became_active":"2024-09-13T14:26:15.820283+0000","last_became_peered":"2024-09-13T14:26:15.820283+0000","last_unstale":"2024-09-13T14:40:33.553001+0000","last_undegraded":"2024-09-13T14:40:33.553001+0000","last_fullsized":"2024-09-13T14:40:33.553001+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:30.340038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":57,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.561056+0000","last_change":"2024-09-13T14:38:03.525682+0000","last_active":"2024-09-13T14:40:34.561056+0000","last_peered":"2024-09-13T14:40:34.561056+0000","last_clean":"2024-09-13T14:40:34.561056+0000","last_became_active":"2024-09-13T14:38:03.525406+0000","last_became_peered":"2024-09-13T14:38:03.525406+0000","last_unstale":"2024-09-13T14:40:34.561056+0000","last_undegraded":"2024-09-13T14:40:34.561056+0000","last_fullsized":"2024-09-13T14:40:34.561056+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:41:28.970300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":64,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.561826+0000","last_change":"2024-09-13T14:38:53.991738+0000","last_active":"2024-09-13T14:40:34.561826+0000","last_peered":"2024-09-13T14:40:34.561826+0000","last_clean":"2024-09-13T14:40:34.561826+0000","last_became_active":"2024-09-13T14:38:53.991417+0000","last_became_peered":"2024-09-13T14:38:53.991417+0000","last_unstale":"2024-09-13T14:40:34.561826+0000","last_undegraded":"2024-09-13T14:40:34.561826+0000","last_fullsized":"2024-09-13T14:40:34.561826+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:50:38.559838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"267'8713","reported_seq":22870,"reported_epoch":267,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-13T14:40:38.288043+0000","last_change":"2024-09-13T14:38:17.472071+0000","last_active":"2024-09-13T14:40:38.288043+0000","last_peered":"2024-09-13T14:40:38.288043+0000","last_clean":"2024-09-13T14:37:46.290754+0000","last_became_active":"2024-09-13T14:38:17.381753+0000","last_became_peered":"2024-09-13T14:38:17.381753+0000","last_unstale":"2024-09-13T14:40:38.288043+0000","last_undegraded":"2024-09-13T14:38:13.504140+0000","last_fullsized":"2024-09-13T14:40:38.288043+0000","mapping_epoch":245,"log_start":"212'2759","ondisk_log_start":"212'2759","created":205,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5954,"log_dups_size":0,"ondisk_log_size":5954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353435648,"num_objects":5393,"num_object_clones":0,"num_object_copies":16179,"num_objects_missing_on_primary":83,"num_objects_missing":83,"num_objects_degraded":4582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8713,"num_write_kb":451392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7053,"num_bytes_recovered":462225408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"0,4,5","objects":4413},{"shards":"0,4,6","objects":980}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":778,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.558741+0000","last_change":"2024-09-13T14:37:48.131799+0000","last_active":"2024-09-13T14:40:34.558741+0000","last_peered":"2024-09-13T14:40:34.558741+0000","last_clean":"2024-09-13T14:40:34.558741+0000","last_became_active":"2024-09-13T14:37:48.130837+0000","last_became_peered":"2024-09-13T14:37:48.130837+0000","last_unstale":"2024-09-13T14:40:34.558741+0000","last_undegraded":"2024-09-13T14:40:34.558741+0000","last_fullsized":"2024-09-13T14:40:34.558741+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:06:32.663770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":297,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.561243+0000","last_change":"2024-09-13T14:37:49.301531+0000","last_active":"2024-09-13T14:40:34.561243+0000","last_peered":"2024-09-13T14:40:34.561243+0000","last_clean":"2024-09-13T14:40:34.561243+0000","last_became_active":"2024-09-13T14:37:49.301249+0000","last_became_peered":"2024-09-13T14:37:49.301249+0000","last_unstale":"2024-09-13T14:40:34.561243+0000","last_undegraded":"2024-09-13T14:40:34.561243+0000","last_fullsized":"2024-09-13T14:40:34.561243+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:10.977710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":92,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.573575+0000","last_change":"2024-09-13T14:38:53.901427+0000","last_active":"2024-09-13T14:40:34.573575+0000","last_peered":"2024-09-13T14:40:34.573575+0000","last_clean":"2024-09-13T14:40:34.573575+0000","last_became_active":"2024-09-13T14:38:53.901157+0000","last_became_peered":"2024-09-13T14:38:53.901157+0000","last_unstale":"2024-09-13T14:40:34.573575+0000","last_undegraded":"2024-09-13T14:40:34.573575+0000","last_fullsized":"2024-09-13T14:40:34.573575+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:28:21.713025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"267'8627","reported_seq":22674,"reported_epoch":267,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-13T14:40:38.250555+0000","last_change":"2024-09-13T14:38:16.552896+0000","last_active":"2024-09-13T14:40:38.250555+0000","last_peered":"2024-09-13T14:40:38.250555+0000","last_clean":"2024-09-13T14:37:46.405415+0000","last_became_active":"2024-09-13T14:38:16.427510+0000","last_became_peered":"2024-09-13T14:38:16.427510+0000","last_unstale":"2024-09-13T14:40:38.250555+0000","last_undegraded":"2024-09-13T14:38:12.738464+0000","last_fullsized":"2024-09-13T14:40:38.250555+0000","mapping_epoch":245,"log_start":"212'2718","ondisk_log_start":"212'2718","created":205,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5909,"log_dups_size":0,"ondisk_log_size":5909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":351993856,"num_objects":5371,"num_object_clones":0,"num_object_copies":16113,"num_objects_missing_on_primary":100,"num_objects_missing":100,"num_objects_degraded":4572,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8627,"num_write_kb":447936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6999,"num_bytes_recovered":458686464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"0,3,6","objects":966},{"shards":"0,3,7","objects":4405}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":974,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.556444+0000","last_change":"2024-09-13T14:37:49.147498+0000","last_active":"2024-09-13T14:40:33.556444+0000","last_peered":"2024-09-13T14:40:33.556444+0000","last_clean":"2024-09-13T14:40:33.556444+0000","last_became_active":"2024-09-13T14:37:49.147028+0000","last_became_peered":"2024-09-13T14:37:49.147028+0000","last_unstale":"2024-09-13T14:40:33.556444+0000","last_undegraded":"2024-09-13T14:40:33.556444+0000","last_fullsized":"2024-09-13T14:40:33.556444+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:11:29.050178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":271,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.552110+0000","last_change":"2024-09-13T14:38:03.528133+0000","last_active":"2024-09-13T14:40:33.552110+0000","last_peered":"2024-09-13T14:40:33.552110+0000","last_clean":"2024-09-13T14:40:33.552110+0000","last_became_active":"2024-09-13T14:38:03.527829+0000","last_became_peered":"2024-09-13T14:38:03.527829+0000","last_unstale":"2024-09-13T14:40:33.552110+0000","last_undegraded":"2024-09-13T14:40:33.552110+0000","last_fullsized":"2024-09-13T14:40:33.552110+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:48.254169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":417,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.558794+0000","last_change":"2024-09-13T14:38:53.899755+0000","last_active":"2024-09-13T14:40:34.558794+0000","last_peered":"2024-09-13T14:40:34.558794+0000","last_clean":"2024-09-13T14:40:34.558794+0000","last_became_active":"2024-09-13T14:38:53.899448+0000","last_became_peered":"2024-09-13T14:38:53.899448+0000","last_unstale":"2024-09-13T14:40:34.558794+0000","last_undegraded":"2024-09-13T14:40:34.558794+0000","last_fullsized":"2024-09-13T14:40:34.558794+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:55.897035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"267'8831","reported_seq":8922,"reported_epoch":267,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:40:37.442693+0000","last_change":"2024-09-13T14:38:03.567892+0000","last_active":"2024-09-13T14:40:37.442693+0000","last_peered":"2024-09-13T14:40:37.442693+0000","last_clean":"2024-09-13T14:35:51.090791+0000","last_became_active":"2024-09-13T14:37:49.665025+0000","last_became_peered":"2024-09-13T14:37:49.665025+0000","last_unstale":"2024-09-13T14:40:37.442693+0000","last_undegraded":"2024-09-13T14:37:49.169101+0000","last_fullsized":"2024-09-13T14:37:49.121596+0000","mapping_epoch":240,"log_start":"212'2763","ondisk_log_start":"212'2763","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6068,"log_dups_size":0,"ondisk_log_size":6068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:02.321504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358547456,"num_objects":5471,"num_object_clones":0,"num_object_copies":16413,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4501,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8831,"num_write_kb":457664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":1036}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":76,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.556806+0000","last_change":"2024-09-13T14:37:49.149770+0000","last_active":"2024-09-13T14:40:33.556806+0000","last_peered":"2024-09-13T14:40:33.556806+0000","last_clean":"2024-09-13T14:40:33.556806+0000","last_became_active":"2024-09-13T14:37:49.148991+0000","last_became_peered":"2024-09-13T14:37:49.148991+0000","last_unstale":"2024-09-13T14:40:33.556806+0000","last_undegraded":"2024-09-13T14:40:33.556806+0000","last_fullsized":"2024-09-13T14:40:33.556806+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:04:04.193366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":1115,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.561745+0000","last_change":"2024-09-13T14:30:58.928302+0000","last_active":"2024-09-13T14:40:34.561745+0000","last_peered":"2024-09-13T14:40:34.561745+0000","last_clean":"2024-09-13T14:40:34.561745+0000","last_became_active":"2024-09-13T14:30:58.927718+0000","last_became_peered":"2024-09-13T14:30:58.927718+0000","last_unstale":"2024-09-13T14:40:34.561745+0000","last_undegraded":"2024-09-13T14:40:34.561745+0000","last_fullsized":"2024-09-13T14:40:34.561745+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:37.323036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":83,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.556567+0000","last_change":"2024-09-13T14:38:03.534628+0000","last_active":"2024-09-13T14:40:33.556567+0000","last_peered":"2024-09-13T14:40:33.556567+0000","last_clean":"2024-09-13T14:40:33.556567+0000","last_became_active":"2024-09-13T14:38:03.534262+0000","last_became_peered":"2024-09-13T14:38:03.534262+0000","last_unstale":"2024-09-13T14:40:33.556567+0000","last_undegraded":"2024-09-13T14:40:33.556567+0000","last_fullsized":"2024-09-13T14:40:33.556567+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:31.168783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"267'8592","reported_seq":22745,"reported_epoch":267,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:40:36.564704+0000","last_change":"2024-09-13T14:38:53.944111+0000","last_active":"2024-09-13T14:40:36.564704+0000","last_peered":"2024-09-13T14:40:36.564704+0000","last_clean":"2024-09-13T14:37:47.047672+0000","last_became_active":"2024-09-13T14:37:49.664294+0000","last_became_peered":"2024-09-13T14:37:49.664294+0000","last_unstale":"2024-09-13T14:40:36.564704+0000","last_undegraded":"2024-09-13T14:38:53.940822+0000","last_fullsized":"2024-09-13T14:37:49.126627+0000","mapping_epoch":240,"log_start":"212'2900","ondisk_log_start":"212'2900","created":205,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5692,"log_dups_size":0,"ondisk_log_size":5692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:11.766692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":357826560,"num_objects":5460,"num_object_clones":0,"num_object_copies":16380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4535,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8592,"num_write_kb":449664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7009,"num_bytes_recovered":459341824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3],"avail_no_missing":["5","0","3"],"object_location_counts":[{"shards":"0,3,5","objects":4460},{"shards":"3,5","objects":1000}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":95,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.556692+0000","last_change":"2024-09-13T14:38:03.534303+0000","last_active":"2024-09-13T14:40:33.556692+0000","last_peered":"2024-09-13T14:40:33.556692+0000","last_clean":"2024-09-13T14:40:33.556692+0000","last_became_active":"2024-09-13T14:38:03.533171+0000","last_became_peered":"2024-09-13T14:38:03.533171+0000","last_unstale":"2024-09-13T14:40:33.556692+0000","last_undegraded":"2024-09-13T14:40:33.556692+0000","last_fullsized":"2024-09-13T14:40:33.556692+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:00:16.553526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":787,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.553699+0000","last_change":"2024-09-13T14:38:04.596192+0000","last_active":"2024-09-13T14:40:33.553699+0000","last_peered":"2024-09-13T14:40:33.553699+0000","last_clean":"2024-09-13T14:40:33.553699+0000","last_became_active":"2024-09-13T14:38:04.593299+0000","last_became_peered":"2024-09-13T14:38:04.593299+0000","last_unstale":"2024-09-13T14:40:33.553699+0000","last_undegraded":"2024-09-13T14:40:33.553699+0000","last_fullsized":"2024-09-13T14:40:33.553699+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:34.241410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":57,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.561575+0000","last_change":"2024-09-13T14:38:03.533120+0000","last_active":"2024-09-13T14:40:34.561575+0000","last_peered":"2024-09-13T14:40:34.561575+0000","last_clean":"2024-09-13T14:40:34.561575+0000","last_became_active":"2024-09-13T14:38:03.532829+0000","last_became_peered":"2024-09-13T14:38:03.532829+0000","last_unstale":"2024-09-13T14:40:34.561575+0000","last_undegraded":"2024-09-13T14:40:34.561575+0000","last_fullsized":"2024-09-13T14:40:34.561575+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:24.514661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"267'8794","reported_seq":8938,"reported_epoch":267,"state":"active+undersized+degraded+remapped+backfill_wait","last_fresh":"2024-09-13T14:40:36.827738+0000","last_change":"2024-09-13T14:37:49.295861+0000","last_active":"2024-09-13T14:40:36.827738+0000","last_peered":"2024-09-13T14:40:36.827738+0000","last_clean":"2024-09-13T14:35:51.078708+0000","last_became_active":"2024-09-13T14:37:49.290403+0000","last_became_peered":"2024-09-13T14:37:49.290403+0000","last_unstale":"2024-09-13T14:40:36.827738+0000","last_undegraded":"2024-09-13T14:37:49.147402+0000","last_fullsized":"2024-09-13T14:37:49.117202+0000","mapping_epoch":240,"log_start":"212'3518","ondisk_log_start":"212'3518","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5276,"log_dups_size":0,"ondisk_log_size":5276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":367263744,"num_objects":5604,"num_object_clones":0,"num_object_copies":16812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5604,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8794,"num_write_kb":460736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":5604}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":459,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.559917+0000","last_change":"2024-09-13T14:38:53.938227+0000","last_active":"2024-09-13T14:40:34.559917+0000","last_peered":"2024-09-13T14:40:34.559917+0000","last_clean":"2024-09-13T14:40:34.559917+0000","last_became_active":"2024-09-13T14:38:53.902867+0000","last_became_peered":"2024-09-13T14:38:53.902867+0000","last_unstale":"2024-09-13T14:40:34.559917+0000","last_undegraded":"2024-09-13T14:40:34.559917+0000","last_fullsized":"2024-09-13T14:40:34.559917+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:16.831756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":566,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.551786+0000","last_change":"2024-09-13T14:37:48.147677+0000","last_active":"2024-09-13T14:40:33.551786+0000","last_peered":"2024-09-13T14:40:33.551786+0000","last_clean":"2024-09-13T14:40:33.551786+0000","last_became_active":"2024-09-13T14:37:48.147001+0000","last_became_peered":"2024-09-13T14:37:48.147001+0000","last_unstale":"2024-09-13T14:40:33.551786+0000","last_undegraded":"2024-09-13T14:40:33.551786+0000","last_fullsized":"2024-09-13T14:40:33.551786+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:35:00.653576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":440,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.558499+0000","last_change":"2024-09-13T14:38:53.900107+0000","last_active":"2024-09-13T14:40:34.558499+0000","last_peered":"2024-09-13T14:40:34.558499+0000","last_clean":"2024-09-13T14:40:34.558499+0000","last_became_active":"2024-09-13T14:38:53.899556+0000","last_became_peered":"2024-09-13T14:38:53.899556+0000","last_unstale":"2024-09-13T14:40:34.558499+0000","last_undegraded":"2024-09-13T14:40:34.558499+0000","last_fullsized":"2024-09-13T14:40:34.558499+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:15:47.918163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"267'8526","reported_seq":17473,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:37.105142+0000","last_change":"2024-09-13T14:39:54.921239+0000","last_active":"2024-09-13T14:40:37.105142+0000","last_peered":"2024-09-13T14:40:37.105142+0000","last_clean":"2024-09-13T14:40:37.105142+0000","last_became_active":"2024-09-13T14:39:54.907894+0000","last_became_peered":"2024-09-13T14:39:54.907894+0000","last_unstale":"2024-09-13T14:40:37.105142+0000","last_undegraded":"2024-09-13T14:40:37.105142+0000","last_fullsized":"2024-09-13T14:40:37.105142+0000","mapping_epoch":263,"log_start":"212'3247","ondisk_log_start":"212'3247","created":205,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5279,"log_dups_size":0,"ondisk_log_size":5279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:52:04.619238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":349044736,"num_objects":5326,"num_object_clones":0,"num_object_copies":15978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8526,"num_write_kb":443264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4395,"num_bytes_recovered":283770880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":72,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.562953+0000","last_change":"2024-09-13T14:38:53.901806+0000","last_active":"2024-09-13T14:40:34.562953+0000","last_peered":"2024-09-13T14:40:34.562953+0000","last_clean":"2024-09-13T14:40:34.562953+0000","last_became_active":"2024-09-13T14:38:53.891055+0000","last_became_peered":"2024-09-13T14:38:53.891055+0000","last_unstale":"2024-09-13T14:40:34.562953+0000","last_undegraded":"2024-09-13T14:40:34.562953+0000","last_fullsized":"2024-09-13T14:40:34.562953+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:03:00.416198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":22,"reported_epoch":266,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-13T14:40:33.552853+0000","last_change":"2024-09-13T14:38:53.900902+0000","last_active":"2024-09-13T14:40:33.552853+0000","last_peered":"2024-09-13T14:40:33.552853+0000","last_clean":"2024-09-13T14:38:39.635779+0000","last_became_active":"2024-09-13T14:38:53.900626+0000","last_became_peered":"2024-09-13T14:38:53.900626+0000","last_unstale":"2024-09-13T14:40:33.552853+0000","last_undegraded":"2024-09-13T14:38:53.885778+0000","last_fullsized":"2024-09-13T14:40:33.552853+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:37:21.715778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":7,"num_objects_missing":7,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":["3","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":256,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.563796+0000","last_change":"2024-09-13T14:37:49.172218+0000","last_active":"2024-09-13T14:40:34.563796+0000","last_peered":"2024-09-13T14:40:34.563796+0000","last_clean":"2024-09-13T14:40:34.563796+0000","last_became_active":"2024-09-13T14:37:49.171718+0000","last_became_peered":"2024-09-13T14:37:49.171718+0000","last_unstale":"2024-09-13T14:40:34.563796+0000","last_undegraded":"2024-09-13T14:40:34.563796+0000","last_fullsized":"2024-09-13T14:40:34.563796+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:46:54.626521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":278,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.551944+0000","last_change":"2024-09-13T14:37:49.230094+0000","last_active":"2024-09-13T14:40:33.551944+0000","last_peered":"2024-09-13T14:40:33.551944+0000","last_clean":"2024-09-13T14:40:33.551944+0000","last_became_active":"2024-09-13T14:37:49.229393+0000","last_became_peered":"2024-09-13T14:37:49.229393+0000","last_unstale":"2024-09-13T14:40:33.551944+0000","last_undegraded":"2024-09-13T14:40:33.551944+0000","last_fullsized":"2024-09-13T14:40:33.551944+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:02:28.690318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":258,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.551423+0000","last_change":"2024-09-13T14:37:48.128447+0000","last_active":"2024-09-13T14:40:33.551423+0000","last_peered":"2024-09-13T14:40:33.551423+0000","last_clean":"2024-09-13T14:40:33.551423+0000","last_became_active":"2024-09-13T14:37:48.128143+0000","last_became_peered":"2024-09-13T14:37:48.128143+0000","last_unstale":"2024-09-13T14:40:33.551423+0000","last_undegraded":"2024-09-13T14:40:33.551423+0000","last_fullsized":"2024-09-13T14:40:33.551423+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:22.204039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":57,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.561401+0000","last_change":"2024-09-13T14:38:03.531683+0000","last_active":"2024-09-13T14:40:34.561401+0000","last_peered":"2024-09-13T14:40:34.561401+0000","last_clean":"2024-09-13T14:40:34.561401+0000","last_became_active":"2024-09-13T14:38:03.531476+0000","last_became_peered":"2024-09-13T14:38:03.531476+0000","last_unstale":"2024-09-13T14:40:34.561401+0000","last_undegraded":"2024-09-13T14:40:34.561401+0000","last_fullsized":"2024-09-13T14:40:34.561401+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:41.557569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":30,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.553557+0000","last_change":"2024-09-13T14:38:53.954281+0000","last_active":"2024-09-13T14:40:33.553557+0000","last_peered":"2024-09-13T14:40:33.553557+0000","last_clean":"2024-09-13T14:40:33.553557+0000","last_became_active":"2024-09-13T14:38:53.900912+0000","last_became_peered":"2024-09-13T14:38:53.900912+0000","last_unstale":"2024-09-13T14:40:33.553557+0000","last_undegraded":"2024-09-13T14:40:33.553557+0000","last_fullsized":"2024-09-13T14:40:33.553557+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:47.876275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":82,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.588179+0000","last_change":"2024-09-13T14:38:54.000782+0000","last_active":"2024-09-13T14:40:33.588179+0000","last_peered":"2024-09-13T14:40:33.588179+0000","last_clean":"2024-09-13T14:40:33.588179+0000","last_became_active":"2024-09-13T14:38:54.000322+0000","last_became_peered":"2024-09-13T14:38:54.000322+0000","last_unstale":"2024-09-13T14:40:33.588179+0000","last_undegraded":"2024-09-13T14:40:33.588179+0000","last_fullsized":"2024-09-13T14:40:33.588179+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:09.057597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":382,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.558438+0000","last_change":"2024-09-13T14:38:53.899535+0000","last_active":"2024-09-13T14:40:34.558438+0000","last_peered":"2024-09-13T14:40:34.558438+0000","last_clean":"2024-09-13T14:40:34.558438+0000","last_became_active":"2024-09-13T14:38:53.899183+0000","last_became_peered":"2024-09-13T14:38:53.899183+0000","last_unstale":"2024-09-13T14:40:34.558438+0000","last_undegraded":"2024-09-13T14:40:34.558438+0000","last_fullsized":"2024-09-13T14:40:34.558438+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:33.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":56,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.561876+0000","last_change":"2024-09-13T14:38:03.543426+0000","last_active":"2024-09-13T14:40:34.561876+0000","last_peered":"2024-09-13T14:40:34.561876+0000","last_clean":"2024-09-13T14:40:34.561876+0000","last_became_active":"2024-09-13T14:38:03.542661+0000","last_became_peered":"2024-09-13T14:38:03.542661+0000","last_unstale":"2024-09-13T14:40:34.561876+0000","last_undegraded":"2024-09-13T14:40:34.561876+0000","last_fullsized":"2024-09-13T14:40:34.561876+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:54.050127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":411,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.558473+0000","last_change":"2024-09-13T14:37:48.131321+0000","last_active":"2024-09-13T14:40:34.558473+0000","last_peered":"2024-09-13T14:40:34.558473+0000","last_clean":"2024-09-13T14:40:34.558473+0000","last_became_active":"2024-09-13T14:37:48.130998+0000","last_became_peered":"2024-09-13T14:37:48.130998+0000","last_unstale":"2024-09-13T14:40:34.558473+0000","last_undegraded":"2024-09-13T14:40:34.558473+0000","last_fullsized":"2024-09-13T14:40:34.558473+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:33:26.597825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":95,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.557033+0000","last_change":"2024-09-13T14:38:03.535454+0000","last_active":"2024-09-13T14:40:33.557033+0000","last_peered":"2024-09-13T14:40:33.557033+0000","last_clean":"2024-09-13T14:40:33.557033+0000","last_became_active":"2024-09-13T14:38:03.535131+0000","last_became_peered":"2024-09-13T14:38:03.535131+0000","last_unstale":"2024-09-13T14:40:33.557033+0000","last_undegraded":"2024-09-13T14:40:33.557033+0000","last_fullsized":"2024-09-13T14:40:33.557033+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:24.477601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":436,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.558593+0000","last_change":"2024-09-13T14:38:03.524939+0000","last_active":"2024-09-13T14:40:34.558593+0000","last_peered":"2024-09-13T14:40:34.558593+0000","last_clean":"2024-09-13T14:40:34.558593+0000","last_became_active":"2024-09-13T14:38:03.524406+0000","last_became_peered":"2024-09-13T14:38:03.524406+0000","last_unstale":"2024-09-13T14:40:34.558593+0000","last_undegraded":"2024-09-13T14:40:34.558593+0000","last_fullsized":"2024-09-13T14:40:34.558593+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:19:45.257424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":64,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.561516+0000","last_change":"2024-09-13T14:38:53.892948+0000","last_active":"2024-09-13T14:40:34.561516+0000","last_peered":"2024-09-13T14:40:34.561516+0000","last_clean":"2024-09-13T14:40:34.561516+0000","last_became_active":"2024-09-13T14:38:53.892665+0000","last_became_peered":"2024-09-13T14:38:53.892665+0000","last_unstale":"2024-09-13T14:40:34.561516+0000","last_undegraded":"2024-09-13T14:40:34.561516+0000","last_fullsized":"2024-09-13T14:40:34.561516+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:08.292359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":384,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.558647+0000","last_change":"2024-09-13T14:37:48.139348+0000","last_active":"2024-09-13T14:40:34.558647+0000","last_peered":"2024-09-13T14:40:34.558647+0000","last_clean":"2024-09-13T14:40:34.558647+0000","last_became_active":"2024-09-13T14:37:48.139115+0000","last_became_peered":"2024-09-13T14:37:48.139115+0000","last_unstale":"2024-09-13T14:40:34.558647+0000","last_undegraded":"2024-09-13T14:40:34.558647+0000","last_fullsized":"2024-09-13T14:40:34.558647+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:32:20.413346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":425,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.559146+0000","last_change":"2024-09-13T14:37:48.172088+0000","last_active":"2024-09-13T14:40:34.559146+0000","last_peered":"2024-09-13T14:40:34.559146+0000","last_clean":"2024-09-13T14:40:34.559146+0000","last_became_active":"2024-09-13T14:37:48.171779+0000","last_became_peered":"2024-09-13T14:37:48.171779+0000","last_unstale":"2024-09-13T14:40:34.559146+0000","last_undegraded":"2024-09-13T14:40:34.559146+0000","last_fullsized":"2024-09-13T14:40:34.559146+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:02.852712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":322,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.551186+0000","last_change":"2024-09-13T14:38:53.890236+0000","last_active":"2024-09-13T14:40:33.551186+0000","last_peered":"2024-09-13T14:40:33.551186+0000","last_clean":"2024-09-13T14:40:33.551186+0000","last_became_active":"2024-09-13T14:38:53.889582+0000","last_became_peered":"2024-09-13T14:38:53.889582+0000","last_unstale":"2024-09-13T14:40:33.551186+0000","last_undegraded":"2024-09-13T14:40:33.551186+0000","last_fullsized":"2024-09-13T14:40:33.551186+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:14:29.334774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":411,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.559293+0000","last_change":"2024-09-13T14:38:03.534097+0000","last_active":"2024-09-13T14:40:34.559293+0000","last_peered":"2024-09-13T14:40:34.559293+0000","last_clean":"2024-09-13T14:40:34.559293+0000","last_became_active":"2024-09-13T14:38:03.533794+0000","last_became_peered":"2024-09-13T14:38:03.533794+0000","last_unstale":"2024-09-13T14:40:34.559293+0000","last_undegraded":"2024-09-13T14:40:34.559293+0000","last_fullsized":"2024-09-13T14:40:34.559293+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:23.272098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":56,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.561392+0000","last_change":"2024-09-13T14:38:03.527021+0000","last_active":"2024-09-13T14:40:34.561392+0000","last_peered":"2024-09-13T14:40:34.561392+0000","last_clean":"2024-09-13T14:40:34.561392+0000","last_became_active":"2024-09-13T14:38:03.526698+0000","last_became_peered":"2024-09-13T14:38:03.526698+0000","last_unstale":"2024-09-13T14:40:34.561392+0000","last_undegraded":"2024-09-13T14:40:34.561392+0000","last_fullsized":"2024-09-13T14:40:34.561392+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:01:35.734086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":266,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.551869+0000","last_change":"2024-09-13T14:27:12.834931+0000","last_active":"2024-09-13T14:40:33.551869+0000","last_peered":"2024-09-13T14:40:33.551869+0000","last_clean":"2024-09-13T14:40:33.551869+0000","last_became_active":"2024-09-13T14:27:12.834572+0000","last_became_peered":"2024-09-13T14:27:12.834572+0000","last_unstale":"2024-09-13T14:40:33.551869+0000","last_undegraded":"2024-09-13T14:40:33.551869+0000","last_fullsized":"2024-09-13T14:40:33.551869+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:54:21.155689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":79,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.556662+0000","last_change":"2024-09-13T14:37:49.176796+0000","last_active":"2024-09-13T14:40:33.556662+0000","last_peered":"2024-09-13T14:40:33.556662+0000","last_clean":"2024-09-13T14:40:33.556662+0000","last_became_active":"2024-09-13T14:37:49.176246+0000","last_became_peered":"2024-09-13T14:37:49.176246+0000","last_unstale":"2024-09-13T14:40:33.556662+0000","last_undegraded":"2024-09-13T14:40:33.556662+0000","last_fullsized":"2024-09-13T14:40:33.556662+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:29:24.936718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":277,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.551022+0000","last_change":"2024-09-13T14:38:03.526333+0000","last_active":"2024-09-13T14:40:33.551022+0000","last_peered":"2024-09-13T14:40:33.551022+0000","last_clean":"2024-09-13T14:40:33.551022+0000","last_became_active":"2024-09-13T14:38:03.526001+0000","last_became_peered":"2024-09-13T14:38:03.526001+0000","last_unstale":"2024-09-13T14:40:33.551022+0000","last_undegraded":"2024-09-13T14:40:33.551022+0000","last_fullsized":"2024-09-13T14:40:33.551022+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:25:25.181271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":79,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.556905+0000","last_change":"2024-09-13T14:37:49.149582+0000","last_active":"2024-09-13T14:40:33.556905+0000","last_peered":"2024-09-13T14:40:33.556905+0000","last_clean":"2024-09-13T14:40:33.556905+0000","last_became_active":"2024-09-13T14:37:49.148525+0000","last_became_peered":"2024-09-13T14:37:49.148525+0000","last_unstale":"2024-09-13T14:40:33.556905+0000","last_undegraded":"2024-09-13T14:40:33.556905+0000","last_fullsized":"2024-09-13T14:40:33.556905+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:48.655665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":457,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.558846+0000","last_change":"2024-09-13T14:38:03.535255+0000","last_active":"2024-09-13T14:40:34.558846+0000","last_peered":"2024-09-13T14:40:34.558846+0000","last_clean":"2024-09-13T14:40:34.558846+0000","last_became_active":"2024-09-13T14:38:03.533917+0000","last_became_peered":"2024-09-13T14:38:03.533917+0000","last_unstale":"2024-09-13T14:40:34.558846+0000","last_undegraded":"2024-09-13T14:40:34.558846+0000","last_fullsized":"2024-09-13T14:40:34.558846+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:33.559788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":420,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.559823+0000","last_change":"2024-09-13T14:37:48.146784+0000","last_active":"2024-09-13T14:40:34.559823+0000","last_peered":"2024-09-13T14:40:34.559823+0000","last_clean":"2024-09-13T14:40:34.559823+0000","last_became_active":"2024-09-13T14:37:48.146491+0000","last_became_peered":"2024-09-13T14:37:48.146491+0000","last_unstale":"2024-09-13T14:40:34.559823+0000","last_undegraded":"2024-09-13T14:40:34.559823+0000","last_fullsized":"2024-09-13T14:40:34.559823+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:30:54.772513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":258,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.550870+0000","last_change":"2024-09-13T14:37:48.127607+0000","last_active":"2024-09-13T14:40:33.550870+0000","last_peered":"2024-09-13T14:40:33.550870+0000","last_clean":"2024-09-13T14:40:33.550870+0000","last_became_active":"2024-09-13T14:37:48.127316+0000","last_became_peered":"2024-09-13T14:37:48.127316+0000","last_unstale":"2024-09-13T14:40:33.550870+0000","last_undegraded":"2024-09-13T14:40:33.550870+0000","last_fullsized":"2024-09-13T14:40:33.550870+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:53:37.678861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":63,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.561874+0000","last_change":"2024-09-13T14:38:03.563200+0000","last_active":"2024-09-13T14:40:34.561874+0000","last_peered":"2024-09-13T14:40:34.561874+0000","last_clean":"2024-09-13T14:40:34.561874+0000","last_became_active":"2024-09-13T14:38:03.534258+0000","last_became_peered":"2024-09-13T14:38:03.534258+0000","last_unstale":"2024-09-13T14:40:34.561874+0000","last_undegraded":"2024-09-13T14:40:34.561874+0000","last_fullsized":"2024-09-13T14:40:34.561874+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:49:07.814419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":5700648984,"num_objects":86986,"num_object_clones":0,"num_object_copies":260958,"num_objects_missing_on_primary":3421,"num_objects_missing":3421,"num_objects_degraded":30614,"num_objects_misplaced":6065,"num_objects_unfound":0,"num_objects_dirty":86986,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":138640,"num_write_kb":7219969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58561,"num_bytes_recovered":3802071064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18174521344,"data_stored":18174509128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92263,"ondisk_log_size":92263,"up":48,"acting":43,"num_store_stats":6},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1538,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":17412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25132,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":7,"num_objects_missing":7,"num_objects_degraded":16,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16793600,"data_stored":16777292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":23,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234361,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3268200,"kb_used_data":2821984,"kb_used_omap":1,"kb_used_meta":446206,"kb_avail":90476952,"statfs":{"total":95995035648,"available":92648398848,"internally_reserved":0,"allocated":2889711616,"data_stored":2888253655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":456915403},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":239,"seq":1026497183779,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2111352,"kb_used_data":1928424,"kb_used_omap":8,"kb_used_meta":182903,"kb_avail":91633800,"statfs":{"total":95995035648,"available":93833011200,"internally_reserved":0,"allocated":1974706176,"data_stored":1973241138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8453,"internal_metadata":187293435},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757626,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2412316,"kb_used_data":1781524,"kb_used_omap":1,"kb_used_meta":630782,"kb_avail":91332836,"statfs":{"total":95995035648,"available":93524824064,"internally_reserved":0,"allocated":1824280576,"data_stored":1822830624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":645921222},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2,1,1,0,5,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019270,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3538152,"kb_used_data":2961504,"kb_used_omap":1,"kb_used_meta":576638,"kb_avail":90207000,"statfs":{"total":95995035648,"available":92371968000,"internally_reserved":0,"allocated":3032580096,"data_stored":3031122667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":590477766},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435453,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4727712,"kb_used_data":4186520,"kb_used_omap":5,"kb_used_meta":541178,"kb_avail":89017440,"statfs":{"total":95995035648,"available":91153858560,"internally_reserved":0,"allocated":4286996480,"data_stored":4285542762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":554166398},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":446676598981,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":425201762508,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926032,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4787300,"kb_used_data":4111452,"kb_used_omap":7,"kb_used_meta":675832,"kb_avail":88957852,"statfs":{"total":95995035648,"available":91092840448,"internally_reserved":0,"allocated":4210126848,"data_stored":4208669091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":692052264},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6860,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4199878656,"data_stored":4199874584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4282650624,"data_stored":4282646552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3026911232,"data_stored":3026911232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1814429696,"data_stored":1814429696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1964838912,"data_stored":1964834840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2885812224,"data_stored":2885812224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:40:44.026 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:44.047 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:44.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:44.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:44.054 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:44 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:40:44.054 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:44 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:40:44.054 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:44 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:40:44.054 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:44 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:40:44.054 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:44 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:40:44.054 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:44 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 30632/261042 objects degraded (11.735%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:44.055 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:44 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:40:44.055 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:44 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:40:44.055 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:44 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:40:44.055 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:44 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:40:44.055 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:44 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:40:44.055 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:44 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 30632/261042 objects degraded (11.735%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:44.152 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:44.174 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:44.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:44.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:44.277 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:44.298 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:44.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:44.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:44.400 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:44.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:44.422 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:44.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:44.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:44 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:40:44.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:44 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:40:44.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:44 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:40:44.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:44 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:40:44.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:44 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:40:44.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:44 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 30632/261042 objects degraded (11.735%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:44.509 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:44.516 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:40:44.516 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:40:44.523 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:44.546 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:44.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:44.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:44.647 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:44.672 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:44.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:44.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:44.773 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:44.799 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:44.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:44.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:44.901 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:44.924 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:44.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:44.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:45.026 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:45.048 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:45.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:45.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:45.150 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:45.176 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:45.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:45.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:45.264 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:45 smithi102 ceph-mon[110183]: pgmap v879: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 5.3 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 178 op/s; 30519/259404 objects degraded (11.765%); 6008/259404 objects misplaced (2.316%); 9.1 MiB/s, 152 objects/s recovering 2024-09-13T14:40:45.265 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:45 smithi102 ceph-mon[115269]: pgmap v879: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 5.3 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 178 op/s; 30519/259404 objects degraded (11.765%); 6008/259404 objects misplaced (2.316%); 9.1 MiB/s, 152 objects/s recovering 2024-09-13T14:40:45.278 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:45.303 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:45.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:45.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:45.405 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:45.430 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:45.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:45.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:45.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:45 smithi195 ceph-mon[104262]: pgmap v879: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 5.3 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 178 op/s; 30519/259404 objects degraded (11.765%); 6008/259404 objects misplaced (2.316%); 9.1 MiB/s, 152 objects/s recovering 2024-09-13T14:40:45.531 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:45.556 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:45.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:45.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:45.658 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:45.685 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:45.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:45.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:45.788 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:45.810 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:45.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:45.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:45.912 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:45.935 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:45.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:45.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:46.037 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:46.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:46.065 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:46.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:46.166 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:46.170 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T14:40:46.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:46.191 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:46.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:46.292 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:46.314 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:46.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:46.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:46.415 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:46.418 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:40:46.439 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:46.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:46.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:46.541 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:46.563 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:46.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:46.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:46.665 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:46.688 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:46.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:46.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:46.760 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:40:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:40:46] "GET /metrics HTTP/1.1" 200 36541 "" "Prometheus/2.51.0" 2024-09-13T14:40:46.790 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:46.813 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:46.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:46.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:46.915 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:46.937 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:46.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:46.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:46.971 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:40:47.012 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:47 smithi102 ceph-mon[110183]: pgmap v880: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 5.3 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 168 op/s; 30456/258267 objects degraded (11.792%); 6008/258267 objects misplaced (2.326%); 3.0 MiB/s, 48 objects/s recovering 2024-09-13T14:40:47.040 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:47.062 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:47.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:47.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:47.163 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:47.189 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:47.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:47.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:47.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:47 smithi195 ceph-mon[104262]: pgmap v880: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 5.3 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 168 op/s; 30456/258267 objects degraded (11.792%); 6008/258267 objects misplaced (2.326%); 3.0 MiB/s, 48 objects/s recovering 2024-09-13T14:40:47.280 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:47 smithi102 ceph-mon[115269]: pgmap v880: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 5.3 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 168 op/s; 30456/258267 objects degraded (11.792%); 6008/258267 objects misplaced (2.326%); 3.0 MiB/s, 48 objects/s recovering 2024-09-13T14:40:47.290 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:47.299 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:40:47.299 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:40:47.318 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:47.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:47.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:47.320 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:40:47.320 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:40:47.349 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:40:47.349 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:40:47.375 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:40:47.376 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:40:47.404 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:40:47.404 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:40:47.420 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:47.426 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:40:47.426 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:40:47.446 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:47.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:47.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:47.448 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:40:47.448 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:40:47.470 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:40:47.470 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:40:47.548 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:47.574 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:47.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:47.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:47.676 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:47.700 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:47.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:47.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:47.801 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:47.826 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:47.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:47.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:47.928 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:47.950 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:47.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:47.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:48.049 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:48.052 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:48.076 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:48.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:48.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:48.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:48 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1340056883' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:40:48.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:48 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1340056883' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:40:48.177 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:48.200 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:48.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:48.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:48.302 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:48.324 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:48.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:48.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:48.349 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:48 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1340056883' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:40:48.426 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:48.450 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:48.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:48.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:48.468 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:40:48.468 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-13T14:40:48.468 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 0 2024-09-13T14:40:48.469 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:40:48.552 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:48.576 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:48.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:48.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:48.678 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:48.705 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:48.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:48.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:48.807 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:48.830 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:48.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:48.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:48.933 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:48.954 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:48.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:48.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:49.018 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is unset 2024-09-13T14:40:49.056 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:49.079 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:49.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:49.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:49.113 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:49 smithi195 ceph-mon[104262]: pgmap v881: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 5.2 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 200 op/s; 30331/256554 objects degraded (11.822%); 5839/256554 objects misplaced (2.276%); 4.0 MiB/s, 63 objects/s recovering 2024-09-13T14:40:49.113 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:49 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 30331/256554 objects degraded (11.822%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:49.113 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:49 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1746945636' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:40:49.137 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T14:40:49.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:49 smithi102 ceph-mon[110183]: pgmap v881: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 5.2 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 200 op/s; 30331/256554 objects degraded (11.822%); 5839/256554 objects misplaced (2.276%); 4.0 MiB/s, 63 objects/s recovering 2024-09-13T14:40:49.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:49 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 30331/256554 objects degraded (11.822%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:49.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:49 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1746945636' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:40:49.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:49 smithi102 ceph-mon[115269]: pgmap v881: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 5.2 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 200 op/s; 30331/256554 objects degraded (11.822%); 5839/256554 objects misplaced (2.276%); 4.0 MiB/s, 63 objects/s recovering 2024-09-13T14:40:49.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:49 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 30331/256554 objects degraded (11.822%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:49.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:49 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1746945636' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:40:49.181 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:49.205 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:49.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:49.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:49.307 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:49.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:49.340 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:49.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:49.413 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:40:49.441 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:49.465 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:49.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:49.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:49.569 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:49.594 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:49.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:49.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:49.696 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:49.718 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:49.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:49.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:49.820 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:49.843 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:49.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:49.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:49.945 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:49.967 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:49.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:49.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:50.069 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:50.094 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:50.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:50.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:50.196 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:50.220 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:50.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:50.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:50.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:50 smithi195 ceph-mon[104262]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:40:50.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:50 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1746945636' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:40:50.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:50 smithi195 ceph-mon[104262]: osdmap e268: 8 total, 8 up, 6 in 2024-09-13T14:40:50.322 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:50.345 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:50.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:50.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:50.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:50 smithi102 ceph-mon[115269]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:40:50.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:50 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1746945636' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:40:50.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:50 smithi102 ceph-mon[115269]: osdmap e268: 8 total, 8 up, 6 in 2024-09-13T14:40:50.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:50 smithi102 ceph-mon[110183]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:40:50.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:50 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1746945636' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:40:50.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:50 smithi102 ceph-mon[110183]: osdmap e268: 8 total, 8 up, 6 in 2024-09-13T14:40:50.447 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:50.469 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:50.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:50.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:50.571 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:50.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:50.592 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:50.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:50.694 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:50.718 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:50.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:50.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:50.820 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:50.847 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:50.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:50.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:50.949 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:50.972 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:50.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:50.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:51.075 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:51.098 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:51.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:51.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:51.200 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:51.228 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:51.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:51.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:51.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:51 smithi195 ceph-mon[104262]: pgmap v883: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 5.2 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 194 op/s; 30270/255219 objects degraded (11.860%); 5840/255219 objects misplaced (2.288%); 3.0 MiB/s, 48 objects/s recovering 2024-09-13T14:40:51.305 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:51 smithi102 ceph-mon[110183]: pgmap v883: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 5.2 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 194 op/s; 30270/255219 objects degraded (11.860%); 5840/255219 objects misplaced (2.288%); 3.0 MiB/s, 48 objects/s recovering 2024-09-13T14:40:51.305 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:51 smithi102 ceph-mon[115269]: pgmap v883: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 5.2 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 194 op/s; 30270/255219 objects degraded (11.860%); 5840/255219 objects misplaced (2.288%); 3.0 MiB/s, 48 objects/s recovering 2024-09-13T14:40:51.331 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:51.353 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:51.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:51.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:51.455 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:51.480 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:51.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:51.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:51.581 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:51.612 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:51.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:51.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:51.704 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:40:51.704 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-13T14:40:51.704 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-09-13T14:40:51.704 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-09-13T14:40:51.705 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:40:51.714 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:51.739 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:51.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:51.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:51.841 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:51.868 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:51.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:51.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:51.937 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:40:51.938 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:40:51.970 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:51.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:51.998 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:51.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:52.071 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:40:52.098 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:52.125 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:52.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:52.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:52.226 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:52.252 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:52.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:52.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:52.354 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:52.379 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:52.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:52.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:52.481 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:52.507 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:52.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:52.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:52.560 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T14:40:52.600 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:40:52.609 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:52.667 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:52.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:52.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:52.769 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:52.782 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":884,"stamp":"2024-09-13T14:40:51.485602+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5578233853,"num_objects":85073,"num_object_clones":0,"num_object_copies":255219,"num_objects_missing_on_primary":3296,"num_objects_missing":3296,"num_objects_degraded":30270,"num_objects_misplaced":5840,"num_objects_unfound":0,"num_objects_dirty":85073,"num_whiteouts":0,"num_read":6377,"num_read_kb":13834,"num_write":141704,"num_write_kb":7235010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59152,"num_bytes_recovered":3839757453,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93386,"ondisk_log_size":93386,"up":267,"acting":261,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":309,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":20485104,"kb_used_data":17435708,"kb_used_omap":26,"kb_used_meta":3049317,"kb_avail":541985808,"statfs":{"total":575970213888,"available":554993467392,"internally_reserved":0,"allocated":17854164992,"data_stored":17845419262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27144,"internal_metadata":3122501112},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-127205376,"num_objects":-1941,"num_object_clones":0,"num_object_copies":-5823,"num_objects_missing_on_primary":-132,"num_objects_missing":-132,"num_objects_degraded":-362,"num_objects_misplaced":-227,"num_objects_unfound":0,"num_objects_dirty":-1941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1941,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":484,"num_bytes_recovered":31719424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002752"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":426,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.559102+0000","last_change":"2024-09-13T14:36:13.859927+0000","last_active":"2024-09-13T14:40:34.559102+0000","last_peered":"2024-09-13T14:40:34.559102+0000","last_clean":"2024-09-13T14:40:34.559102+0000","last_became_active":"2024-09-13T14:36:13.859505+0000","last_became_peered":"2024-09-13T14:36:13.859505+0000","last_unstale":"2024-09-13T14:40:34.559102+0000","last_undegraded":"2024-09-13T14:40:34.559102+0000","last_fullsized":"2024-09-13T14:40:34.559102+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:12:20.989627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":268,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.553366+0000","last_change":"2024-09-13T14:37:48.130968+0000","last_active":"2024-09-13T14:40:33.553366+0000","last_peered":"2024-09-13T14:40:33.553366+0000","last_clean":"2024-09-13T14:40:33.553366+0000","last_became_active":"2024-09-13T14:37:48.130603+0000","last_became_peered":"2024-09-13T14:37:48.130603+0000","last_unstale":"2024-09-13T14:40:33.553366+0000","last_undegraded":"2024-09-13T14:40:33.553366+0000","last_fullsized":"2024-09-13T14:40:33.553366+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:42.626386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":64,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.556939+0000","last_change":"2024-09-13T14:36:13.862181+0000","last_active":"2024-09-13T14:40:33.556939+0000","last_peered":"2024-09-13T14:40:33.556939+0000","last_clean":"2024-09-13T14:40:33.556939+0000","last_became_active":"2024-09-13T14:36:13.861873+0000","last_became_peered":"2024-09-13T14:36:13.861873+0000","last_unstale":"2024-09-13T14:40:33.556939+0000","last_undegraded":"2024-09-13T14:40:33.556939+0000","last_fullsized":"2024-09-13T14:40:33.556939+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:14.085700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":64,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.561499+0000","last_change":"2024-09-13T14:38:53.896672+0000","last_active":"2024-09-13T14:40:34.561499+0000","last_peered":"2024-09-13T14:40:34.561499+0000","last_clean":"2024-09-13T14:40:34.561499+0000","last_became_active":"2024-09-13T14:38:53.896346+0000","last_became_peered":"2024-09-13T14:38:53.896346+0000","last_unstale":"2024-09-13T14:40:34.561499+0000","last_undegraded":"2024-09-13T14:40:34.561499+0000","last_fullsized":"2024-09-13T14:40:34.561499+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:57.789686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":64,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.561760+0000","last_change":"2024-09-13T14:38:53.893021+0000","last_active":"2024-09-13T14:40:34.561760+0000","last_peered":"2024-09-13T14:40:34.561760+0000","last_clean":"2024-09-13T14:40:34.561760+0000","last_became_active":"2024-09-13T14:38:53.892668+0000","last_became_peered":"2024-09-13T14:38:53.892668+0000","last_unstale":"2024-09-13T14:40:34.561760+0000","last_undegraded":"2024-09-13T14:40:34.561760+0000","last_fullsized":"2024-09-13T14:40:34.561760+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:57.180738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":236,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.553185+0000","last_change":"2024-09-13T14:37:48.134642+0000","last_active":"2024-09-13T14:40:33.553185+0000","last_peered":"2024-09-13T14:40:33.553185+0000","last_clean":"2024-09-13T14:40:33.553185+0000","last_became_active":"2024-09-13T14:37:48.134402+0000","last_became_peered":"2024-09-13T14:37:48.134402+0000","last_unstale":"2024-09-13T14:40:33.553185+0000","last_undegraded":"2024-09-13T14:40:33.553185+0000","last_fullsized":"2024-09-13T14:40:33.553185+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:51.313663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":445,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.559667+0000","last_change":"2024-09-13T14:38:53.903022+0000","last_active":"2024-09-13T14:40:34.559667+0000","last_peered":"2024-09-13T14:40:34.559667+0000","last_clean":"2024-09-13T14:40:34.559667+0000","last_became_active":"2024-09-13T14:38:53.902448+0000","last_became_peered":"2024-09-13T14:38:53.902448+0000","last_unstale":"2024-09-13T14:40:34.559667+0000","last_undegraded":"2024-09-13T14:40:34.559667+0000","last_fullsized":"2024-09-13T14:40:34.559667+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:24.690401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":67,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.561052+0000","last_change":"2024-09-13T14:38:03.554500+0000","last_active":"2024-09-13T14:40:34.561052+0000","last_peered":"2024-09-13T14:40:34.561052+0000","last_clean":"2024-09-13T14:40:34.561052+0000","last_became_active":"2024-09-13T14:38:03.532910+0000","last_became_peered":"2024-09-13T14:38:03.532910+0000","last_unstale":"2024-09-13T14:40:34.561052+0000","last_undegraded":"2024-09-13T14:40:34.561052+0000","last_fullsized":"2024-09-13T14:40:34.561052+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:27:50.843498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"267'8672","reported_seq":11681,"reported_epoch":267,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-13T14:40:46.370193+0000","last_change":"2024-09-13T14:40:32.799848+0000","last_active":"2024-09-13T14:40:46.370193+0000","last_peered":"2024-09-13T14:40:46.370193+0000","last_clean":"2024-09-13T14:37:46.417003+0000","last_became_active":"2024-09-13T14:37:49.690961+0000","last_became_peered":"2024-09-13T14:37:49.690961+0000","last_unstale":"2024-09-13T14:40:46.370193+0000","last_undegraded":"2024-09-13T14:40:46.370193+0000","last_fullsized":"2024-09-13T14:37:49.126746+0000","mapping_epoch":240,"log_start":"212'3000","ondisk_log_start":"212'3000","created":205,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5672,"log_dups_size":0,"ondisk_log_size":5672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:48:52.526429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341311488,"num_objects":5208,"num_object_clones":0,"num_object_copies":15624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3850,"num_objects_unfound":0,"num_objects_dirty":5208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8672,"num_write_kb":444160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5223,"num_bytes_recovered":342294528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,4],"avail_no_missing":["5","0","4"],"object_location_counts":[{"shards":"0,4,5","objects":3850},{"shards":"4,5","objects":1358}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":28,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.553961+0000","last_change":"2024-09-13T14:38:53.940238+0000","last_active":"2024-09-13T14:40:33.553961+0000","last_peered":"2024-09-13T14:40:33.553961+0000","last_clean":"2024-09-13T14:40:33.553961+0000","last_became_active":"2024-09-13T14:38:53.892825+0000","last_became_peered":"2024-09-13T14:38:53.892825+0000","last_unstale":"2024-09-13T14:40:33.553961+0000","last_undegraded":"2024-09-13T14:40:33.553961+0000","last_fullsized":"2024-09-13T14:40:33.553961+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:46.666581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":85,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.573478+0000","last_change":"2024-09-13T14:38:53.898907+0000","last_active":"2024-09-13T14:40:34.573478+0000","last_peered":"2024-09-13T14:40:34.573478+0000","last_clean":"2024-09-13T14:40:34.573478+0000","last_became_active":"2024-09-13T14:38:53.898572+0000","last_became_peered":"2024-09-13T14:38:53.898572+0000","last_unstale":"2024-09-13T14:40:34.573478+0000","last_undegraded":"2024-09-13T14:40:34.573478+0000","last_fullsized":"2024-09-13T14:40:34.573478+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:29.222150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"267'8846","reported_seq":22989,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:47.259977+0000","last_change":"2024-09-13T14:38:57.930032+0000","last_active":"2024-09-13T14:40:47.259977+0000","last_peered":"2024-09-13T14:40:47.259977+0000","last_clean":"2024-09-13T14:40:47.259977+0000","last_became_active":"2024-09-13T14:38:57.921458+0000","last_became_peered":"2024-09-13T14:38:57.921458+0000","last_unstale":"2024-09-13T14:40:47.259977+0000","last_undegraded":"2024-09-13T14:40:47.259977+0000","last_fullsized":"2024-09-13T14:40:47.259977+0000","mapping_epoch":256,"log_start":"212'3539","ondisk_log_start":"212'3539","created":205,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5307,"log_dups_size":0,"ondisk_log_size":5307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:27:12.294038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353632256,"num_objects":5396,"num_object_clones":0,"num_object_copies":16188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8846,"num_write_kb":455744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6990,"num_bytes_recovered":458096640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":28,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.553782+0000","last_change":"2024-09-13T14:38:53.904885+0000","last_active":"2024-09-13T14:40:33.553782+0000","last_peered":"2024-09-13T14:40:33.553782+0000","last_clean":"2024-09-13T14:40:33.553782+0000","last_became_active":"2024-09-13T14:38:53.891726+0000","last_became_peered":"2024-09-13T14:38:53.891726+0000","last_unstale":"2024-09-13T14:40:33.553782+0000","last_undegraded":"2024-09-13T14:40:33.553782+0000","last_fullsized":"2024-09-13T14:40:33.553782+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:57.079205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":274,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.551024+0000","last_change":"2024-09-13T14:37:48.160504+0000","last_active":"2024-09-13T14:40:33.551024+0000","last_peered":"2024-09-13T14:40:33.551024+0000","last_clean":"2024-09-13T14:40:33.551024+0000","last_became_active":"2024-09-13T14:37:48.160181+0000","last_became_peered":"2024-09-13T14:37:48.160181+0000","last_unstale":"2024-09-13T14:40:33.551024+0000","last_undegraded":"2024-09-13T14:40:33.551024+0000","last_fullsized":"2024-09-13T14:40:33.551024+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:37.087008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"267'8844","reported_seq":10609,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:48.746855+0000","last_change":"2024-09-13T14:38:04.531914+0000","last_active":"2024-09-13T14:40:48.746855+0000","last_peered":"2024-09-13T14:40:48.746855+0000","last_clean":"2024-09-13T14:40:48.746855+0000","last_became_active":"2024-09-13T14:38:04.523448+0000","last_became_peered":"2024-09-13T14:38:04.523448+0000","last_unstale":"2024-09-13T14:40:48.746855+0000","last_undegraded":"2024-09-13T14:40:48.746855+0000","last_fullsized":"2024-09-13T14:40:48.746855+0000","mapping_epoch":245,"log_start":"212'2900","ondisk_log_start":"212'2900","created":205,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5944,"log_dups_size":0,"ondisk_log_size":5944,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:06.536896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":349306880,"num_objects":5330,"num_object_clones":0,"num_object_copies":15990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8844,"num_write_kb":453568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":811,"num_bytes_recovered":53149696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":266,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.562108+0000","last_change":"2024-09-13T14:37:49.611774+0000","last_active":"2024-09-13T14:40:34.562108+0000","last_peered":"2024-09-13T14:40:34.562108+0000","last_clean":"2024-09-13T14:40:34.562108+0000","last_became_active":"2024-09-13T14:37:49.611088+0000","last_became_peered":"2024-09-13T14:37:49.611088+0000","last_unstale":"2024-09-13T14:40:34.562108+0000","last_undegraded":"2024-09-13T14:40:34.562108+0000","last_fullsized":"2024-09-13T14:40:34.562108+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:08.933213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":24,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.553252+0000","last_change":"2024-09-13T14:38:53.899211+0000","last_active":"2024-09-13T14:40:33.553252+0000","last_peered":"2024-09-13T14:40:33.553252+0000","last_clean":"2024-09-13T14:40:33.553252+0000","last_became_active":"2024-09-13T14:38:53.898927+0000","last_became_peered":"2024-09-13T14:38:53.898927+0000","last_unstale":"2024-09-13T14:40:33.553252+0000","last_undegraded":"2024-09-13T14:40:33.553252+0000","last_fullsized":"2024-09-13T14:40:33.553252+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:00:28.857611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"267'8686","reported_seq":8759,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:47.336395+0000","last_change":"2024-09-13T14:35:24.004651+0000","last_active":"2024-09-13T14:40:47.336395+0000","last_peered":"2024-09-13T14:40:47.336395+0000","last_clean":"2024-09-13T14:40:47.336395+0000","last_became_active":"2024-09-13T14:35:24.004358+0000","last_became_peered":"2024-09-13T14:35:24.004358+0000","last_unstale":"2024-09-13T14:40:47.336395+0000","last_undegraded":"2024-09-13T14:40:47.336395+0000","last_fullsized":"2024-09-13T14:40:47.336395+0000","mapping_epoch":205,"log_start":"210'2384","ondisk_log_start":"210'2384","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6302,"log_dups_size":0,"ondisk_log_size":6302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:57:38.023669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":346423296,"num_objects":5286,"num_object_clones":0,"num_object_copies":15858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8686,"num_write_kb":447104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":76,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.572816+0000","last_change":"2024-09-13T14:37:49.168003+0000","last_active":"2024-09-13T14:40:34.572816+0000","last_peered":"2024-09-13T14:40:34.572816+0000","last_clean":"2024-09-13T14:40:34.572816+0000","last_became_active":"2024-09-13T14:37:49.167638+0000","last_became_peered":"2024-09-13T14:37:49.167638+0000","last_unstale":"2024-09-13T14:40:34.572816+0000","last_undegraded":"2024-09-13T14:40:34.572816+0000","last_fullsized":"2024-09-13T14:40:34.572816+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:29.051981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":57,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.561426+0000","last_change":"2024-09-13T14:38:03.528381+0000","last_active":"2024-09-13T14:40:34.561426+0000","last_peered":"2024-09-13T14:40:34.561426+0000","last_clean":"2024-09-13T14:40:34.561426+0000","last_became_active":"2024-09-13T14:38:03.527282+0000","last_became_peered":"2024-09-13T14:38:03.527282+0000","last_unstale":"2024-09-13T14:40:34.561426+0000","last_undegraded":"2024-09-13T14:40:34.561426+0000","last_fullsized":"2024-09-13T14:40:34.561426+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:24.435067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.f","version":"267'8773","reported_seq":9033,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:48.736092+0000","last_change":"2024-09-13T14:38:16.371079+0000","last_active":"2024-09-13T14:40:48.736092+0000","last_peered":"2024-09-13T14:40:48.736092+0000","last_clean":"2024-09-13T14:40:48.736092+0000","last_became_active":"2024-09-13T14:38:12.859883+0000","last_became_peered":"2024-09-13T14:38:12.859883+0000","last_unstale":"2024-09-13T14:40:48.736092+0000","last_undegraded":"2024-09-13T14:40:48.736092+0000","last_fullsized":"2024-09-13T14:40:48.736092+0000","mapping_epoch":245,"log_start":"212'2863","ondisk_log_start":"212'2863","created":205,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5910,"log_dups_size":0,"ondisk_log_size":5910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:08.436532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":348192768,"num_objects":5313,"num_object_clones":0,"num_object_copies":15939,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8773,"num_write_kb":450752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":3670016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":415,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.558977+0000","last_change":"2024-09-13T14:38:03.549595+0000","last_active":"2024-09-13T14:40:34.558977+0000","last_peered":"2024-09-13T14:40:34.558977+0000","last_clean":"2024-09-13T14:40:34.558977+0000","last_became_active":"2024-09-13T14:38:03.533331+0000","last_became_peered":"2024-09-13T14:38:03.533331+0000","last_unstale":"2024-09-13T14:40:34.558977+0000","last_undegraded":"2024-09-13T14:40:34.558977+0000","last_fullsized":"2024-09-13T14:40:34.558977+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:59:02.470343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":250,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.558600+0000","last_change":"2024-09-13T14:37:48.127600+0000","last_active":"2024-09-13T14:40:34.558600+0000","last_peered":"2024-09-13T14:40:34.558600+0000","last_clean":"2024-09-13T14:40:34.558600+0000","last_became_active":"2024-09-13T14:37:48.127349+0000","last_became_peered":"2024-09-13T14:37:48.127349+0000","last_unstale":"2024-09-13T14:40:34.558600+0000","last_undegraded":"2024-09-13T14:40:34.558600+0000","last_fullsized":"2024-09-13T14:40:34.558600+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:57.268374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"267'8676","reported_seq":12129,"reported_epoch":267,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:40:46.846079+0000","last_change":"2024-09-13T14:38:14.979229+0000","last_active":"2024-09-13T14:40:46.846079+0000","last_peered":"2024-09-13T14:40:46.846079+0000","last_clean":"2024-09-13T14:36:13.131873+0000","last_became_active":"2024-09-13T14:38:10.443954+0000","last_became_peered":"2024-09-13T14:38:10.443954+0000","last_unstale":"2024-09-13T14:40:46.846079+0000","last_undegraded":"2024-09-13T14:38:10.384643+0000","last_fullsized":"2024-09-13T14:38:04.517046+0000","mapping_epoch":245,"log_start":"212'3000","ondisk_log_start":"212'3000","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5676,"log_dups_size":0,"ondisk_log_size":5676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:26:53.493170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":349569024,"num_objects":5334,"num_object_clones":0,"num_object_copies":16002,"num_objects_missing_on_primary":3106,"num_objects_missing":3106,"num_objects_degraded":3247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8676,"num_write_kb":448320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1667,"num_bytes_recovered":109182976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":1},{"shards":"0,3,5,7","objects":12},{"shards":"0,3,7","objects":3105},{"shards":"0,5","objects":2211},{"shards":"3,5,7","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":183,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.551100+0000","last_change":"2024-09-13T14:37:48.127604+0000","last_active":"2024-09-13T14:40:33.551100+0000","last_peered":"2024-09-13T14:40:33.551100+0000","last_clean":"2024-09-13T14:40:33.551100+0000","last_became_active":"2024-09-13T14:37:48.127300+0000","last_became_peered":"2024-09-13T14:37:48.127300+0000","last_unstale":"2024-09-13T14:40:33.551100+0000","last_undegraded":"2024-09-13T14:40:33.551100+0000","last_fullsized":"2024-09-13T14:40:33.551100+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:51.999506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":303,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.559644+0000","last_change":"2024-09-13T14:38:03.529967+0000","last_active":"2024-09-13T14:40:34.559644+0000","last_peered":"2024-09-13T14:40:34.559644+0000","last_clean":"2024-09-13T14:40:34.559644+0000","last_became_active":"2024-09-13T14:38:03.529646+0000","last_became_peered":"2024-09-13T14:38:03.529646+0000","last_unstale":"2024-09-13T14:40:34.559644+0000","last_undegraded":"2024-09-13T14:40:34.559644+0000","last_fullsized":"2024-09-13T14:40:34.559644+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:52:05.193279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"267'8838","reported_seq":17973,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:48.623688+0000","last_change":"2024-09-13T14:40:34.583237+0000","last_active":"2024-09-13T14:40:48.623688+0000","last_peered":"2024-09-13T14:40:48.623688+0000","last_clean":"2024-09-13T14:40:48.623688+0000","last_became_active":"2024-09-13T14:40:34.563989+0000","last_became_peered":"2024-09-13T14:40:34.563989+0000","last_unstale":"2024-09-13T14:40:48.623688+0000","last_undegraded":"2024-09-13T14:40:48.623688+0000","last_fullsized":"2024-09-13T14:40:48.623688+0000","mapping_epoch":266,"log_start":"212'2900","ondisk_log_start":"212'2900","created":205,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5938,"log_dups_size":0,"ondisk_log_size":5938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:34:43.006051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":347406360,"num_objects":5302,"num_object_clones":0,"num_object_copies":15906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5302,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8838,"num_write_kb":452417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4489,"num_bytes_recovered":289865752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":263,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.551273+0000","last_change":"2024-09-13T14:36:13.860367+0000","last_active":"2024-09-13T14:40:33.551273+0000","last_peered":"2024-09-13T14:40:33.551273+0000","last_clean":"2024-09-13T14:40:33.551273+0000","last_became_active":"2024-09-13T14:36:13.859982+0000","last_became_peered":"2024-09-13T14:36:13.859982+0000","last_unstale":"2024-09-13T14:40:33.551273+0000","last_undegraded":"2024-09-13T14:40:33.551273+0000","last_fullsized":"2024-09-13T14:40:33.551273+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:08:23.311350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":259,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.558923+0000","last_change":"2024-09-13T14:37:48.147360+0000","last_active":"2024-09-13T14:40:34.558923+0000","last_peered":"2024-09-13T14:40:34.558923+0000","last_clean":"2024-09-13T14:40:34.558923+0000","last_became_active":"2024-09-13T14:37:48.146885+0000","last_became_peered":"2024-09-13T14:37:48.146885+0000","last_unstale":"2024-09-13T14:40:34.558923+0000","last_undegraded":"2024-09-13T14:40:34.558923+0000","last_fullsized":"2024-09-13T14:40:34.558923+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:27.841810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"267'8757","reported_seq":17999,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:47.406244+0000","last_change":"2024-09-13T14:38:39.650332+0000","last_active":"2024-09-13T14:40:47.406244+0000","last_peered":"2024-09-13T14:40:47.406244+0000","last_clean":"2024-09-13T14:40:47.406244+0000","last_became_active":"2024-09-13T14:38:39.640835+0000","last_became_peered":"2024-09-13T14:38:39.640835+0000","last_unstale":"2024-09-13T14:40:47.406244+0000","last_undegraded":"2024-09-13T14:40:47.406244+0000","last_fullsized":"2024-09-13T14:40:47.406244+0000","mapping_epoch":251,"log_start":"212'3451","ondisk_log_start":"212'3451","created":205,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5306,"log_dups_size":0,"ondisk_log_size":5306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:48.176606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":348454912,"num_objects":5317,"num_object_clones":0,"num_object_copies":15951,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8757,"num_write_kb":450368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4544,"num_bytes_recovered":292945920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":433,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.558682+0000","last_change":"2024-09-13T14:38:53.900041+0000","last_active":"2024-09-13T14:40:34.558682+0000","last_peered":"2024-09-13T14:40:34.558682+0000","last_clean":"2024-09-13T14:40:34.558682+0000","last_became_active":"2024-09-13T14:38:53.899203+0000","last_became_peered":"2024-09-13T14:38:53.899203+0000","last_unstale":"2024-09-13T14:40:34.558682+0000","last_undegraded":"2024-09-13T14:40:34.558682+0000","last_fullsized":"2024-09-13T14:40:34.558682+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:28:07.883094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":365,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.559811+0000","last_change":"2024-09-13T14:26:15.823198+0000","last_active":"2024-09-13T14:40:34.559811+0000","last_peered":"2024-09-13T14:40:34.559811+0000","last_clean":"2024-09-13T14:40:34.559811+0000","last_became_active":"2024-09-13T14:26:15.822717+0000","last_became_peered":"2024-09-13T14:26:15.822717+0000","last_unstale":"2024-09-13T14:40:34.559811+0000","last_undegraded":"2024-09-13T14:40:34.559811+0000","last_fullsized":"2024-09-13T14:40:34.559811+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:22.471622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.6","version":"267'8893","reported_seq":12421,"reported_epoch":267,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-13T14:40:48.769066+0000","last_change":"2024-09-13T14:38:55.132683+0000","last_active":"2024-09-13T14:40:48.769066+0000","last_peered":"2024-09-13T14:40:48.769066+0000","last_clean":"2024-09-13T14:36:13.124368+0000","last_became_active":"2024-09-13T14:38:55.120883+0000","last_became_peered":"2024-09-13T14:38:55.120883+0000","last_unstale":"2024-09-13T14:40:48.769066+0000","last_undegraded":"2024-09-13T14:38:55.026610+0000","last_fullsized":"2024-09-13T14:40:48.769066+0000","mapping_epoch":254,"log_start":"212'3119","ondisk_log_start":"212'3119","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5774,"log_dups_size":0,"ondisk_log_size":5774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":347668480,"num_objects":5305,"num_object_clones":0,"num_object_copies":15915,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3317,"num_objects_misplaced":1988,"num_objects_unfound":0,"num_objects_dirty":5305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8893,"num_write_kb":454336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1688,"num_bytes_recovered":99418112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,3,5],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3,4","objects":3038},{"shards":"0,3,5","objects":2267}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":429,"reported_epoch":267,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-13T14:40:34.558590+0000","last_change":"2024-09-13T14:38:54.899624+0000","last_active":"2024-09-13T14:40:34.558590+0000","last_peered":"2024-09-13T14:40:34.558590+0000","last_clean":"2024-09-13T14:17:27.401592+0000","last_became_active":"2024-09-13T14:38:54.899282+0000","last_became_peered":"2024-09-13T14:38:54.899282+0000","last_unstale":"2024-09-13T14:40:34.558590+0000","last_undegraded":"2024-09-13T14:38:54.891176+0000","last_fullsized":"2024-09-13T14:40:34.558590+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:07:41.927740+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[3,0,6],"avail_no_missing":["3","4","6"],"object_location_counts":[{"shards":"3,4,6","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":505,"reported_epoch":266,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:40:33.556725+0000","last_change":"2024-09-13T14:38:54.904329+0000","last_active":"2024-09-13T14:40:33.556725+0000","last_peered":"2024-09-13T14:40:33.556725+0000","last_clean":"2024-09-13T14:38:39.648017+0000","last_became_active":"2024-09-13T14:38:54.903798+0000","last_became_peered":"2024-09-13T14:38:54.903798+0000","last_unstale":"2024-09-13T14:40:33.556725+0000","last_undegraded":"2024-09-13T14:38:54.890505+0000","last_fullsized":"2024-09-13T14:38:54.889701+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:18.120931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,3],"avail_no_missing":["5","3","6"],"object_location_counts":[{"shards":"3,5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":367,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.559320+0000","last_change":"2024-09-13T14:37:49.160719+0000","last_active":"2024-09-13T14:40:34.559320+0000","last_peered":"2024-09-13T14:40:34.559320+0000","last_clean":"2024-09-13T14:40:34.559320+0000","last_became_active":"2024-09-13T14:37:49.160401+0000","last_became_peered":"2024-09-13T14:37:49.160401+0000","last_unstale":"2024-09-13T14:40:34.559320+0000","last_undegraded":"2024-09-13T14:40:34.559320+0000","last_fullsized":"2024-09-13T14:40:34.559320+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:21.737945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":110,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.551634+0000","last_change":"2024-09-13T14:37:48.172940+0000","last_active":"2024-09-13T14:40:33.551634+0000","last_peered":"2024-09-13T14:40:33.551634+0000","last_clean":"2024-09-13T14:40:33.551634+0000","last_became_active":"2024-09-13T14:37:48.172060+0000","last_became_peered":"2024-09-13T14:37:48.172060+0000","last_unstale":"2024-09-13T14:40:33.551634+0000","last_undegraded":"2024-09-13T14:40:33.551634+0000","last_fullsized":"2024-09-13T14:40:33.551634+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:07.480645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.5","version":"267'8790","reported_seq":25139,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:47.331268+0000","last_change":"2024-09-13T14:39:34.446699+0000","last_active":"2024-09-13T14:40:47.331268+0000","last_peered":"2024-09-13T14:40:47.331268+0000","last_clean":"2024-09-13T14:40:47.331268+0000","last_became_active":"2024-09-13T14:38:04.600616+0000","last_became_peered":"2024-09-13T14:38:04.600616+0000","last_unstale":"2024-09-13T14:40:47.331268+0000","last_undegraded":"2024-09-13T14:40:47.331268+0000","last_fullsized":"2024-09-13T14:40:47.331268+0000","mapping_epoch":245,"log_start":"210'2516","ondisk_log_start":"210'2516","created":205,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6274,"log_dups_size":0,"ondisk_log_size":6274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:00:44.321678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350748672,"num_objects":5352,"num_object_clones":0,"num_object_copies":16056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8790,"num_write_kb":452544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8120,"num_bytes_recovered":521076736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":483,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.553001+0000","last_change":"2024-09-13T14:26:15.835178+0000","last_active":"2024-09-13T14:40:33.553001+0000","last_peered":"2024-09-13T14:40:33.553001+0000","last_clean":"2024-09-13T14:40:33.553001+0000","last_became_active":"2024-09-13T14:26:15.820283+0000","last_became_peered":"2024-09-13T14:26:15.820283+0000","last_unstale":"2024-09-13T14:40:33.553001+0000","last_undegraded":"2024-09-13T14:40:33.553001+0000","last_fullsized":"2024-09-13T14:40:33.553001+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:30.340038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":57,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.561056+0000","last_change":"2024-09-13T14:38:03.525682+0000","last_active":"2024-09-13T14:40:34.561056+0000","last_peered":"2024-09-13T14:40:34.561056+0000","last_clean":"2024-09-13T14:40:34.561056+0000","last_became_active":"2024-09-13T14:38:03.525406+0000","last_became_peered":"2024-09-13T14:38:03.525406+0000","last_unstale":"2024-09-13T14:40:34.561056+0000","last_undegraded":"2024-09-13T14:40:34.561056+0000","last_fullsized":"2024-09-13T14:40:34.561056+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:41:28.970300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":64,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.561826+0000","last_change":"2024-09-13T14:38:53.991738+0000","last_active":"2024-09-13T14:40:34.561826+0000","last_peered":"2024-09-13T14:40:34.561826+0000","last_clean":"2024-09-13T14:40:34.561826+0000","last_became_active":"2024-09-13T14:38:53.991417+0000","last_became_peered":"2024-09-13T14:38:53.991417+0000","last_unstale":"2024-09-13T14:40:34.561826+0000","last_undegraded":"2024-09-13T14:40:34.561826+0000","last_fullsized":"2024-09-13T14:40:34.561826+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:50:38.559838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"267'8847","reported_seq":23004,"reported_epoch":267,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-13T14:40:48.598121+0000","last_change":"2024-09-13T14:38:17.472071+0000","last_active":"2024-09-13T14:40:48.598121+0000","last_peered":"2024-09-13T14:40:48.598121+0000","last_clean":"2024-09-13T14:37:46.290754+0000","last_became_active":"2024-09-13T14:38:17.381753+0000","last_became_peered":"2024-09-13T14:38:17.381753+0000","last_unstale":"2024-09-13T14:40:48.598121+0000","last_undegraded":"2024-09-13T14:38:13.504140+0000","last_fullsized":"2024-09-13T14:40:48.598121+0000","mapping_epoch":245,"log_start":"212'2959","ondisk_log_start":"212'2959","created":205,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5888,"log_dups_size":0,"ondisk_log_size":5888,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":344653824,"num_objects":5259,"num_object_clones":0,"num_object_copies":15777,"num_objects_missing_on_primary":83,"num_objects_missing":83,"num_objects_degraded":4582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8847,"num_write_kb":451392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7053,"num_bytes_recovered":462225408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"0,4,5","objects":4413},{"shards":"0,4,6","objects":846}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":778,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.558741+0000","last_change":"2024-09-13T14:37:48.131799+0000","last_active":"2024-09-13T14:40:34.558741+0000","last_peered":"2024-09-13T14:40:34.558741+0000","last_clean":"2024-09-13T14:40:34.558741+0000","last_became_active":"2024-09-13T14:37:48.130837+0000","last_became_peered":"2024-09-13T14:37:48.130837+0000","last_unstale":"2024-09-13T14:40:34.558741+0000","last_undegraded":"2024-09-13T14:40:34.558741+0000","last_fullsized":"2024-09-13T14:40:34.558741+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:06:32.663770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":297,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.561243+0000","last_change":"2024-09-13T14:37:49.301531+0000","last_active":"2024-09-13T14:40:34.561243+0000","last_peered":"2024-09-13T14:40:34.561243+0000","last_clean":"2024-09-13T14:40:34.561243+0000","last_became_active":"2024-09-13T14:37:49.301249+0000","last_became_peered":"2024-09-13T14:37:49.301249+0000","last_unstale":"2024-09-13T14:40:34.561243+0000","last_undegraded":"2024-09-13T14:40:34.561243+0000","last_fullsized":"2024-09-13T14:40:34.561243+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:10.977710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":92,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.573575+0000","last_change":"2024-09-13T14:38:53.901427+0000","last_active":"2024-09-13T14:40:34.573575+0000","last_peered":"2024-09-13T14:40:34.573575+0000","last_clean":"2024-09-13T14:40:34.573575+0000","last_became_active":"2024-09-13T14:38:53.901157+0000","last_became_peered":"2024-09-13T14:38:53.901157+0000","last_unstale":"2024-09-13T14:40:34.573575+0000","last_undegraded":"2024-09-13T14:40:34.573575+0000","last_fullsized":"2024-09-13T14:40:34.573575+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:28:21.713025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"267'8753","reported_seq":22800,"reported_epoch":267,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-13T14:40:48.685045+0000","last_change":"2024-09-13T14:38:16.552896+0000","last_active":"2024-09-13T14:40:48.685045+0000","last_peered":"2024-09-13T14:40:48.685045+0000","last_clean":"2024-09-13T14:37:46.405415+0000","last_became_active":"2024-09-13T14:38:16.427510+0000","last_became_peered":"2024-09-13T14:38:16.427510+0000","last_unstale":"2024-09-13T14:40:48.685045+0000","last_undegraded":"2024-09-13T14:38:12.738464+0000","last_fullsized":"2024-09-13T14:40:48.685045+0000","mapping_epoch":245,"log_start":"212'2818","ondisk_log_start":"212'2818","created":205,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5935,"log_dups_size":0,"ondisk_log_size":5935,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343736320,"num_objects":5245,"num_object_clones":0,"num_object_copies":15735,"num_objects_missing_on_primary":100,"num_objects_missing":100,"num_objects_degraded":4572,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8753,"num_write_kb":447936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6999,"num_bytes_recovered":458686464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"0,3,6","objects":840},{"shards":"0,3,7","objects":4405}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":974,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.556444+0000","last_change":"2024-09-13T14:37:49.147498+0000","last_active":"2024-09-13T14:40:33.556444+0000","last_peered":"2024-09-13T14:40:33.556444+0000","last_clean":"2024-09-13T14:40:33.556444+0000","last_became_active":"2024-09-13T14:37:49.147028+0000","last_became_peered":"2024-09-13T14:37:49.147028+0000","last_unstale":"2024-09-13T14:40:33.556444+0000","last_undegraded":"2024-09-13T14:40:33.556444+0000","last_fullsized":"2024-09-13T14:40:33.556444+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:11:29.050178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":272,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.560061+0000","last_change":"2024-09-13T14:38:03.528133+0000","last_active":"2024-09-13T14:40:34.560061+0000","last_peered":"2024-09-13T14:40:34.560061+0000","last_clean":"2024-09-13T14:40:34.560061+0000","last_became_active":"2024-09-13T14:38:03.527829+0000","last_became_peered":"2024-09-13T14:38:03.527829+0000","last_unstale":"2024-09-13T14:40:34.560061+0000","last_undegraded":"2024-09-13T14:40:34.560061+0000","last_fullsized":"2024-09-13T14:40:34.560061+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:48.254169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":417,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.558794+0000","last_change":"2024-09-13T14:38:53.899755+0000","last_active":"2024-09-13T14:40:34.558794+0000","last_peered":"2024-09-13T14:40:34.558794+0000","last_clean":"2024-09-13T14:40:34.558794+0000","last_became_active":"2024-09-13T14:38:53.899448+0000","last_became_peered":"2024-09-13T14:38:53.899448+0000","last_unstale":"2024-09-13T14:40:34.558794+0000","last_undegraded":"2024-09-13T14:40:34.558794+0000","last_fullsized":"2024-09-13T14:40:34.558794+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:55.897035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"267'8964","reported_seq":9055,"reported_epoch":267,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:40:47.711504+0000","last_change":"2024-09-13T14:38:03.567892+0000","last_active":"2024-09-13T14:40:47.711504+0000","last_peered":"2024-09-13T14:40:47.711504+0000","last_clean":"2024-09-13T14:35:51.090791+0000","last_became_active":"2024-09-13T14:37:49.665025+0000","last_became_peered":"2024-09-13T14:37:49.665025+0000","last_unstale":"2024-09-13T14:40:47.711504+0000","last_undegraded":"2024-09-13T14:37:49.169101+0000","last_fullsized":"2024-09-13T14:37:49.121596+0000","mapping_epoch":240,"log_start":"212'2963","ondisk_log_start":"212'2963","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6001,"log_dups_size":0,"ondisk_log_size":6001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:02.321504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":349831168,"num_objects":5338,"num_object_clones":0,"num_object_copies":16014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4501,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8964,"num_write_kb":457664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":903}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":76,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.556806+0000","last_change":"2024-09-13T14:37:49.149770+0000","last_active":"2024-09-13T14:40:33.556806+0000","last_peered":"2024-09-13T14:40:33.556806+0000","last_clean":"2024-09-13T14:40:33.556806+0000","last_became_active":"2024-09-13T14:37:49.148991+0000","last_became_peered":"2024-09-13T14:37:49.148991+0000","last_unstale":"2024-09-13T14:40:33.556806+0000","last_undegraded":"2024-09-13T14:40:33.556806+0000","last_fullsized":"2024-09-13T14:40:33.556806+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:04:04.193366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":1115,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.561745+0000","last_change":"2024-09-13T14:30:58.928302+0000","last_active":"2024-09-13T14:40:34.561745+0000","last_peered":"2024-09-13T14:40:34.561745+0000","last_clean":"2024-09-13T14:40:34.561745+0000","last_became_active":"2024-09-13T14:30:58.927718+0000","last_became_peered":"2024-09-13T14:30:58.927718+0000","last_unstale":"2024-09-13T14:40:34.561745+0000","last_undegraded":"2024-09-13T14:40:34.561745+0000","last_fullsized":"2024-09-13T14:40:34.561745+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:37.323036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":83,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.556567+0000","last_change":"2024-09-13T14:38:03.534628+0000","last_active":"2024-09-13T14:40:33.556567+0000","last_peered":"2024-09-13T14:40:33.556567+0000","last_clean":"2024-09-13T14:40:33.556567+0000","last_became_active":"2024-09-13T14:38:03.534262+0000","last_became_peered":"2024-09-13T14:38:03.534262+0000","last_unstale":"2024-09-13T14:40:33.556567+0000","last_undegraded":"2024-09-13T14:40:33.556567+0000","last_fullsized":"2024-09-13T14:40:33.556567+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:31.168783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"267'8712","reported_seq":22865,"reported_epoch":267,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:40:46.743141+0000","last_change":"2024-09-13T14:38:53.944111+0000","last_active":"2024-09-13T14:40:46.743141+0000","last_peered":"2024-09-13T14:40:46.743141+0000","last_clean":"2024-09-13T14:37:47.047672+0000","last_became_active":"2024-09-13T14:37:49.664294+0000","last_became_peered":"2024-09-13T14:37:49.664294+0000","last_unstale":"2024-09-13T14:40:46.743141+0000","last_undegraded":"2024-09-13T14:38:53.940822+0000","last_fullsized":"2024-09-13T14:37:49.126627+0000","mapping_epoch":240,"log_start":"212'3000","ondisk_log_start":"212'3000","created":205,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5712,"log_dups_size":0,"ondisk_log_size":5712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:11.766692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":349962240,"num_objects":5340,"num_object_clones":0,"num_object_copies":16020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4535,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8712,"num_write_kb":449664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7009,"num_bytes_recovered":459341824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3],"avail_no_missing":["5","0","3"],"object_location_counts":[{"shards":"0,3,5","objects":4460},{"shards":"3,5","objects":880}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":95,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.556692+0000","last_change":"2024-09-13T14:38:03.534303+0000","last_active":"2024-09-13T14:40:33.556692+0000","last_peered":"2024-09-13T14:40:33.556692+0000","last_clean":"2024-09-13T14:40:33.556692+0000","last_became_active":"2024-09-13T14:38:03.533171+0000","last_became_peered":"2024-09-13T14:38:03.533171+0000","last_unstale":"2024-09-13T14:40:33.556692+0000","last_undegraded":"2024-09-13T14:40:33.556692+0000","last_fullsized":"2024-09-13T14:40:33.556692+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:00:16.553526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":787,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.553699+0000","last_change":"2024-09-13T14:38:04.596192+0000","last_active":"2024-09-13T14:40:33.553699+0000","last_peered":"2024-09-13T14:40:33.553699+0000","last_clean":"2024-09-13T14:40:33.553699+0000","last_became_active":"2024-09-13T14:38:04.593299+0000","last_became_peered":"2024-09-13T14:38:04.593299+0000","last_unstale":"2024-09-13T14:40:33.553699+0000","last_undegraded":"2024-09-13T14:40:33.553699+0000","last_fullsized":"2024-09-13T14:40:33.553699+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:34.241410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":57,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.561575+0000","last_change":"2024-09-13T14:38:03.533120+0000","last_active":"2024-09-13T14:40:34.561575+0000","last_peered":"2024-09-13T14:40:34.561575+0000","last_clean":"2024-09-13T14:40:34.561575+0000","last_became_active":"2024-09-13T14:38:03.532829+0000","last_became_peered":"2024-09-13T14:38:03.532829+0000","last_unstale":"2024-09-13T14:40:34.561575+0000","last_undegraded":"2024-09-13T14:40:34.561575+0000","last_fullsized":"2024-09-13T14:40:34.561575+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:24.514661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"267'8900","reported_seq":9044,"reported_epoch":267,"state":"active+undersized+degraded+remapped+backfill_wait","last_fresh":"2024-09-13T14:40:47.134749+0000","last_change":"2024-09-13T14:37:49.295861+0000","last_active":"2024-09-13T14:40:47.134749+0000","last_peered":"2024-09-13T14:40:47.134749+0000","last_clean":"2024-09-13T14:35:51.078708+0000","last_became_active":"2024-09-13T14:37:49.290403+0000","last_became_peered":"2024-09-13T14:37:49.290403+0000","last_unstale":"2024-09-13T14:40:47.134749+0000","last_undegraded":"2024-09-13T14:37:49.147402+0000","last_fullsized":"2024-09-13T14:37:49.117202+0000","mapping_epoch":240,"log_start":"212'3618","ondisk_log_start":"212'3618","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5282,"log_dups_size":0,"ondisk_log_size":5282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360316928,"num_objects":5498,"num_object_clones":0,"num_object_copies":16494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5498,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8900,"num_write_kb":460736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":5498}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":459,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.559917+0000","last_change":"2024-09-13T14:38:53.938227+0000","last_active":"2024-09-13T14:40:34.559917+0000","last_peered":"2024-09-13T14:40:34.559917+0000","last_clean":"2024-09-13T14:40:34.559917+0000","last_became_active":"2024-09-13T14:38:53.902867+0000","last_became_peered":"2024-09-13T14:38:53.902867+0000","last_unstale":"2024-09-13T14:40:34.559917+0000","last_undegraded":"2024-09-13T14:40:34.559917+0000","last_fullsized":"2024-09-13T14:40:34.559917+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:16.831756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":566,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.551786+0000","last_change":"2024-09-13T14:37:48.147677+0000","last_active":"2024-09-13T14:40:33.551786+0000","last_peered":"2024-09-13T14:40:33.551786+0000","last_clean":"2024-09-13T14:40:33.551786+0000","last_became_active":"2024-09-13T14:37:48.147001+0000","last_became_peered":"2024-09-13T14:37:48.147001+0000","last_unstale":"2024-09-13T14:40:33.551786+0000","last_undegraded":"2024-09-13T14:40:33.551786+0000","last_fullsized":"2024-09-13T14:40:33.551786+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:35:00.653576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":440,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.558499+0000","last_change":"2024-09-13T14:38:53.900107+0000","last_active":"2024-09-13T14:40:34.558499+0000","last_peered":"2024-09-13T14:40:34.558499+0000","last_clean":"2024-09-13T14:40:34.558499+0000","last_became_active":"2024-09-13T14:38:53.899556+0000","last_became_peered":"2024-09-13T14:38:53.899556+0000","last_unstale":"2024-09-13T14:40:34.558499+0000","last_undegraded":"2024-09-13T14:40:34.558499+0000","last_fullsized":"2024-09-13T14:40:34.558499+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:15:47.918163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"267'8630","reported_seq":17577,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:47.416197+0000","last_change":"2024-09-13T14:39:54.921239+0000","last_active":"2024-09-13T14:40:47.416197+0000","last_peered":"2024-09-13T14:40:47.416197+0000","last_clean":"2024-09-13T14:40:47.416197+0000","last_became_active":"2024-09-13T14:39:54.907894+0000","last_became_peered":"2024-09-13T14:39:54.907894+0000","last_unstale":"2024-09-13T14:40:47.416197+0000","last_undegraded":"2024-09-13T14:40:47.416197+0000","last_fullsized":"2024-09-13T14:40:47.416197+0000","mapping_epoch":263,"log_start":"212'3347","ondisk_log_start":"212'3347","created":205,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5283,"log_dups_size":0,"ondisk_log_size":5283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:52:04.619238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":342228992,"num_objects":5222,"num_object_clones":0,"num_object_copies":15666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8630,"num_write_kb":443264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4395,"num_bytes_recovered":283770880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":72,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.562953+0000","last_change":"2024-09-13T14:38:53.901806+0000","last_active":"2024-09-13T14:40:34.562953+0000","last_peered":"2024-09-13T14:40:34.562953+0000","last_clean":"2024-09-13T14:40:34.562953+0000","last_became_active":"2024-09-13T14:38:53.891055+0000","last_became_peered":"2024-09-13T14:38:53.891055+0000","last_unstale":"2024-09-13T14:40:34.562953+0000","last_undegraded":"2024-09-13T14:40:34.562953+0000","last_fullsized":"2024-09-13T14:40:34.562953+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:03:00.416198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":22,"reported_epoch":266,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-13T14:40:33.552853+0000","last_change":"2024-09-13T14:38:53.900902+0000","last_active":"2024-09-13T14:40:33.552853+0000","last_peered":"2024-09-13T14:40:33.552853+0000","last_clean":"2024-09-13T14:38:39.635779+0000","last_became_active":"2024-09-13T14:38:53.900626+0000","last_became_peered":"2024-09-13T14:38:53.900626+0000","last_unstale":"2024-09-13T14:40:33.552853+0000","last_undegraded":"2024-09-13T14:38:53.885778+0000","last_fullsized":"2024-09-13T14:40:33.552853+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:37:21.715778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":7,"num_objects_missing":7,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":["3","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":256,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.563796+0000","last_change":"2024-09-13T14:37:49.172218+0000","last_active":"2024-09-13T14:40:34.563796+0000","last_peered":"2024-09-13T14:40:34.563796+0000","last_clean":"2024-09-13T14:40:34.563796+0000","last_became_active":"2024-09-13T14:37:49.171718+0000","last_became_peered":"2024-09-13T14:37:49.171718+0000","last_unstale":"2024-09-13T14:40:34.563796+0000","last_undegraded":"2024-09-13T14:40:34.563796+0000","last_fullsized":"2024-09-13T14:40:34.563796+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:46:54.626521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":278,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.551944+0000","last_change":"2024-09-13T14:37:49.230094+0000","last_active":"2024-09-13T14:40:33.551944+0000","last_peered":"2024-09-13T14:40:33.551944+0000","last_clean":"2024-09-13T14:40:33.551944+0000","last_became_active":"2024-09-13T14:37:49.229393+0000","last_became_peered":"2024-09-13T14:37:49.229393+0000","last_unstale":"2024-09-13T14:40:33.551944+0000","last_undegraded":"2024-09-13T14:40:33.551944+0000","last_fullsized":"2024-09-13T14:40:33.551944+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:02:28.690318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":258,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.551423+0000","last_change":"2024-09-13T14:37:48.128447+0000","last_active":"2024-09-13T14:40:33.551423+0000","last_peered":"2024-09-13T14:40:33.551423+0000","last_clean":"2024-09-13T14:40:33.551423+0000","last_became_active":"2024-09-13T14:37:48.128143+0000","last_became_peered":"2024-09-13T14:37:48.128143+0000","last_unstale":"2024-09-13T14:40:33.551423+0000","last_undegraded":"2024-09-13T14:40:33.551423+0000","last_fullsized":"2024-09-13T14:40:33.551423+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:22.204039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":57,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.561401+0000","last_change":"2024-09-13T14:38:03.531683+0000","last_active":"2024-09-13T14:40:34.561401+0000","last_peered":"2024-09-13T14:40:34.561401+0000","last_clean":"2024-09-13T14:40:34.561401+0000","last_became_active":"2024-09-13T14:38:03.531476+0000","last_became_peered":"2024-09-13T14:38:03.531476+0000","last_unstale":"2024-09-13T14:40:34.561401+0000","last_undegraded":"2024-09-13T14:40:34.561401+0000","last_fullsized":"2024-09-13T14:40:34.561401+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:41.557569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":30,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.553557+0000","last_change":"2024-09-13T14:38:53.954281+0000","last_active":"2024-09-13T14:40:33.553557+0000","last_peered":"2024-09-13T14:40:33.553557+0000","last_clean":"2024-09-13T14:40:33.553557+0000","last_became_active":"2024-09-13T14:38:53.900912+0000","last_became_peered":"2024-09-13T14:38:53.900912+0000","last_unstale":"2024-09-13T14:40:33.553557+0000","last_undegraded":"2024-09-13T14:40:33.553557+0000","last_fullsized":"2024-09-13T14:40:33.553557+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:47.876275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":82,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.588179+0000","last_change":"2024-09-13T14:38:54.000782+0000","last_active":"2024-09-13T14:40:33.588179+0000","last_peered":"2024-09-13T14:40:33.588179+0000","last_clean":"2024-09-13T14:40:33.588179+0000","last_became_active":"2024-09-13T14:38:54.000322+0000","last_became_peered":"2024-09-13T14:38:54.000322+0000","last_unstale":"2024-09-13T14:40:33.588179+0000","last_undegraded":"2024-09-13T14:40:33.588179+0000","last_fullsized":"2024-09-13T14:40:33.588179+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:09.057597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":382,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.558438+0000","last_change":"2024-09-13T14:38:53.899535+0000","last_active":"2024-09-13T14:40:34.558438+0000","last_peered":"2024-09-13T14:40:34.558438+0000","last_clean":"2024-09-13T14:40:34.558438+0000","last_became_active":"2024-09-13T14:38:53.899183+0000","last_became_peered":"2024-09-13T14:38:53.899183+0000","last_unstale":"2024-09-13T14:40:34.558438+0000","last_undegraded":"2024-09-13T14:40:34.558438+0000","last_fullsized":"2024-09-13T14:40:34.558438+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:33.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":57,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.561910+0000","last_change":"2024-09-13T14:38:03.543426+0000","last_active":"2024-09-13T14:40:34.561910+0000","last_peered":"2024-09-13T14:40:34.561910+0000","last_clean":"2024-09-13T14:40:34.561910+0000","last_became_active":"2024-09-13T14:38:03.542661+0000","last_became_peered":"2024-09-13T14:38:03.542661+0000","last_unstale":"2024-09-13T14:40:34.561910+0000","last_undegraded":"2024-09-13T14:40:34.561910+0000","last_fullsized":"2024-09-13T14:40:34.561910+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:54.050127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":411,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.558473+0000","last_change":"2024-09-13T14:37:48.131321+0000","last_active":"2024-09-13T14:40:34.558473+0000","last_peered":"2024-09-13T14:40:34.558473+0000","last_clean":"2024-09-13T14:40:34.558473+0000","last_became_active":"2024-09-13T14:37:48.130998+0000","last_became_peered":"2024-09-13T14:37:48.130998+0000","last_unstale":"2024-09-13T14:40:34.558473+0000","last_undegraded":"2024-09-13T14:40:34.558473+0000","last_fullsized":"2024-09-13T14:40:34.558473+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:33:26.597825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":95,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.557033+0000","last_change":"2024-09-13T14:38:03.535454+0000","last_active":"2024-09-13T14:40:33.557033+0000","last_peered":"2024-09-13T14:40:33.557033+0000","last_clean":"2024-09-13T14:40:33.557033+0000","last_became_active":"2024-09-13T14:38:03.535131+0000","last_became_peered":"2024-09-13T14:38:03.535131+0000","last_unstale":"2024-09-13T14:40:33.557033+0000","last_undegraded":"2024-09-13T14:40:33.557033+0000","last_fullsized":"2024-09-13T14:40:33.557033+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:24.477601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":436,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.558593+0000","last_change":"2024-09-13T14:38:03.524939+0000","last_active":"2024-09-13T14:40:34.558593+0000","last_peered":"2024-09-13T14:40:34.558593+0000","last_clean":"2024-09-13T14:40:34.558593+0000","last_became_active":"2024-09-13T14:38:03.524406+0000","last_became_peered":"2024-09-13T14:38:03.524406+0000","last_unstale":"2024-09-13T14:40:34.558593+0000","last_undegraded":"2024-09-13T14:40:34.558593+0000","last_fullsized":"2024-09-13T14:40:34.558593+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:19:45.257424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":64,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.561516+0000","last_change":"2024-09-13T14:38:53.892948+0000","last_active":"2024-09-13T14:40:34.561516+0000","last_peered":"2024-09-13T14:40:34.561516+0000","last_clean":"2024-09-13T14:40:34.561516+0000","last_became_active":"2024-09-13T14:38:53.892665+0000","last_became_peered":"2024-09-13T14:38:53.892665+0000","last_unstale":"2024-09-13T14:40:34.561516+0000","last_undegraded":"2024-09-13T14:40:34.561516+0000","last_fullsized":"2024-09-13T14:40:34.561516+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:08.292359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":384,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.558647+0000","last_change":"2024-09-13T14:37:48.139348+0000","last_active":"2024-09-13T14:40:34.558647+0000","last_peered":"2024-09-13T14:40:34.558647+0000","last_clean":"2024-09-13T14:40:34.558647+0000","last_became_active":"2024-09-13T14:37:48.139115+0000","last_became_peered":"2024-09-13T14:37:48.139115+0000","last_unstale":"2024-09-13T14:40:34.558647+0000","last_undegraded":"2024-09-13T14:40:34.558647+0000","last_fullsized":"2024-09-13T14:40:34.558647+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:32:20.413346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":425,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.559146+0000","last_change":"2024-09-13T14:37:48.172088+0000","last_active":"2024-09-13T14:40:34.559146+0000","last_peered":"2024-09-13T14:40:34.559146+0000","last_clean":"2024-09-13T14:40:34.559146+0000","last_became_active":"2024-09-13T14:37:48.171779+0000","last_became_peered":"2024-09-13T14:37:48.171779+0000","last_unstale":"2024-09-13T14:40:34.559146+0000","last_undegraded":"2024-09-13T14:40:34.559146+0000","last_fullsized":"2024-09-13T14:40:34.559146+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:02.852712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":322,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.551186+0000","last_change":"2024-09-13T14:38:53.890236+0000","last_active":"2024-09-13T14:40:33.551186+0000","last_peered":"2024-09-13T14:40:33.551186+0000","last_clean":"2024-09-13T14:40:33.551186+0000","last_became_active":"2024-09-13T14:38:53.889582+0000","last_became_peered":"2024-09-13T14:38:53.889582+0000","last_unstale":"2024-09-13T14:40:33.551186+0000","last_undegraded":"2024-09-13T14:40:33.551186+0000","last_fullsized":"2024-09-13T14:40:33.551186+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:14:29.334774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":411,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.559293+0000","last_change":"2024-09-13T14:38:03.534097+0000","last_active":"2024-09-13T14:40:34.559293+0000","last_peered":"2024-09-13T14:40:34.559293+0000","last_clean":"2024-09-13T14:40:34.559293+0000","last_became_active":"2024-09-13T14:38:03.533794+0000","last_became_peered":"2024-09-13T14:38:03.533794+0000","last_unstale":"2024-09-13T14:40:34.559293+0000","last_undegraded":"2024-09-13T14:40:34.559293+0000","last_fullsized":"2024-09-13T14:40:34.559293+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:23.272098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":57,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.561444+0000","last_change":"2024-09-13T14:38:03.527021+0000","last_active":"2024-09-13T14:40:34.561444+0000","last_peered":"2024-09-13T14:40:34.561444+0000","last_clean":"2024-09-13T14:40:34.561444+0000","last_became_active":"2024-09-13T14:38:03.526698+0000","last_became_peered":"2024-09-13T14:38:03.526698+0000","last_unstale":"2024-09-13T14:40:34.561444+0000","last_undegraded":"2024-09-13T14:40:34.561444+0000","last_fullsized":"2024-09-13T14:40:34.561444+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:01:35.734086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":266,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.551869+0000","last_change":"2024-09-13T14:27:12.834931+0000","last_active":"2024-09-13T14:40:33.551869+0000","last_peered":"2024-09-13T14:40:33.551869+0000","last_clean":"2024-09-13T14:40:33.551869+0000","last_became_active":"2024-09-13T14:27:12.834572+0000","last_became_peered":"2024-09-13T14:27:12.834572+0000","last_unstale":"2024-09-13T14:40:33.551869+0000","last_undegraded":"2024-09-13T14:40:33.551869+0000","last_fullsized":"2024-09-13T14:40:33.551869+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:54:21.155689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":79,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.556662+0000","last_change":"2024-09-13T14:37:49.176796+0000","last_active":"2024-09-13T14:40:33.556662+0000","last_peered":"2024-09-13T14:40:33.556662+0000","last_clean":"2024-09-13T14:40:33.556662+0000","last_became_active":"2024-09-13T14:37:49.176246+0000","last_became_peered":"2024-09-13T14:37:49.176246+0000","last_unstale":"2024-09-13T14:40:33.556662+0000","last_undegraded":"2024-09-13T14:40:33.556662+0000","last_fullsized":"2024-09-13T14:40:33.556662+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:29:24.936718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":277,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.551022+0000","last_change":"2024-09-13T14:38:03.526333+0000","last_active":"2024-09-13T14:40:33.551022+0000","last_peered":"2024-09-13T14:40:33.551022+0000","last_clean":"2024-09-13T14:40:33.551022+0000","last_became_active":"2024-09-13T14:38:03.526001+0000","last_became_peered":"2024-09-13T14:38:03.526001+0000","last_unstale":"2024-09-13T14:40:33.551022+0000","last_undegraded":"2024-09-13T14:40:33.551022+0000","last_fullsized":"2024-09-13T14:40:33.551022+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:25:25.181271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":79,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.556905+0000","last_change":"2024-09-13T14:37:49.149582+0000","last_active":"2024-09-13T14:40:33.556905+0000","last_peered":"2024-09-13T14:40:33.556905+0000","last_clean":"2024-09-13T14:40:33.556905+0000","last_became_active":"2024-09-13T14:37:49.148525+0000","last_became_peered":"2024-09-13T14:37:49.148525+0000","last_unstale":"2024-09-13T14:40:33.556905+0000","last_undegraded":"2024-09-13T14:40:33.556905+0000","last_fullsized":"2024-09-13T14:40:33.556905+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:48.655665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":457,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.558846+0000","last_change":"2024-09-13T14:38:03.535255+0000","last_active":"2024-09-13T14:40:34.558846+0000","last_peered":"2024-09-13T14:40:34.558846+0000","last_clean":"2024-09-13T14:40:34.558846+0000","last_became_active":"2024-09-13T14:38:03.533917+0000","last_became_peered":"2024-09-13T14:38:03.533917+0000","last_unstale":"2024-09-13T14:40:34.558846+0000","last_undegraded":"2024-09-13T14:40:34.558846+0000","last_fullsized":"2024-09-13T14:40:34.558846+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:33.559788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":420,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.559823+0000","last_change":"2024-09-13T14:37:48.146784+0000","last_active":"2024-09-13T14:40:34.559823+0000","last_peered":"2024-09-13T14:40:34.559823+0000","last_clean":"2024-09-13T14:40:34.559823+0000","last_became_active":"2024-09-13T14:37:48.146491+0000","last_became_peered":"2024-09-13T14:37:48.146491+0000","last_unstale":"2024-09-13T14:40:34.559823+0000","last_undegraded":"2024-09-13T14:40:34.559823+0000","last_fullsized":"2024-09-13T14:40:34.559823+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:30:54.772513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":258,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-13T14:40:33.550870+0000","last_change":"2024-09-13T14:37:48.127607+0000","last_active":"2024-09-13T14:40:33.550870+0000","last_peered":"2024-09-13T14:40:33.550870+0000","last_clean":"2024-09-13T14:40:33.550870+0000","last_became_active":"2024-09-13T14:37:48.127316+0000","last_became_peered":"2024-09-13T14:37:48.127316+0000","last_unstale":"2024-09-13T14:40:33.550870+0000","last_undegraded":"2024-09-13T14:40:33.550870+0000","last_fullsized":"2024-09-13T14:40:33.550870+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:53:37.678861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":63,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-13T14:40:34.561874+0000","last_change":"2024-09-13T14:38:03.563200+0000","last_active":"2024-09-13T14:40:34.561874+0000","last_peered":"2024-09-13T14:40:34.561874+0000","last_clean":"2024-09-13T14:40:34.561874+0000","last_became_active":"2024-09-13T14:38:03.534258+0000","last_became_peered":"2024-09-13T14:38:03.534258+0000","last_unstale":"2024-09-13T14:40:34.561874+0000","last_undegraded":"2024-09-13T14:40:34.561874+0000","last_fullsized":"2024-09-13T14:40:34.561874+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:49:07.814419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":5573443608,"num_objects":85045,"num_object_clones":0,"num_object_copies":255135,"num_objects_missing_on_primary":3289,"num_objects_missing":3289,"num_objects_degraded":30252,"num_objects_misplaced":5838,"num_objects_unfound":0,"num_objects_dirty":85045,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140581,"num_write_kb":7219969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59045,"num_bytes_recovered":3833790488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17804898304,"data_stored":17804886088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92204,"ondisk_log_size":92204,"up":48,"acting":43,"num_store_stats":6},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1538,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":17412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25132,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":7,"num_objects_missing":7,"num_objects_degraded":16,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16793600,"data_stored":16777292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":23,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234363,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3230248,"kb_used_data":2779936,"kb_used_omap":1,"kb_used_meta":450302,"kb_avail":90514904,"statfs":{"total":95995035648,"available":92687261696,"internally_reserved":0,"allocated":2846654464,"data_stored":2845196503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":461109707},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":239,"seq":1026497183782,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2018880,"kb_used_data":1844148,"kb_used_omap":8,"kb_used_meta":174711,"kb_avail":91726272,"statfs":{"total":95995035648,"available":93927702528,"internally_reserved":0,"allocated":1888407552,"data_stored":1886938639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8453,"internal_metadata":178904827},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757628,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2391132,"kb_used_data":1756244,"kb_used_omap":1,"kb_used_meta":634878,"kb_avail":91354020,"statfs":{"total":95995035648,"available":93546516480,"internally_reserved":0,"allocated":1798393856,"data_stored":1796943904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":650115526},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019272,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3490280,"kb_used_data":2909536,"kb_used_omap":1,"kb_used_meta":580734,"kb_avail":90254872,"statfs":{"total":95995035648,"available":92420988928,"internally_reserved":0,"allocated":2979364864,"data_stored":2977907435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":594672070},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435455,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4644256,"kb_used_data":4098904,"kb_used_omap":5,"kb_used_meta":545338,"kb_avail":89100896,"statfs":{"total":95995035648,"available":91239317504,"internally_reserved":0,"allocated":4197277696,"data_stored":4195823978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":558426238},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":446676598984,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":425201762510,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926034,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4710308,"kb_used_data":4046940,"kb_used_omap":7,"kb_used_meta":663352,"kb_avail":89034844,"statfs":{"total":95995035648,"available":91171680256,"internally_reserved":0,"allocated":4144066560,"data_stored":4142608803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":679272744},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6860,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4132638720,"data_stored":4132634648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4190769152,"data_stored":4190765080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2971074560,"data_stored":2971074560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1790705664,"data_stored":1790705664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1879052288,"data_stored":1879048216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2840657920,"data_stored":2840657920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:40:52.786 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph status --format=json 2024-09-13T14:40:52.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:52.802 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:52.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:52.810 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:40:52.830 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:52.903 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:52.926 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:52.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:52.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:53.028 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:53.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:53.051 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:53.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:53.153 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:53.177 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:53.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:53.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:53.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:53 smithi195 ceph-mon[104262]: pgmap v884: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 5.2 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 194 op/s; 30270/255219 objects degraded (11.860%); 5840/255219 objects misplaced (2.288%); 3.0 MiB/s, 48 objects/s recovering 2024-09-13T14:40:53.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:53 smithi195 ceph-mon[104262]: from='client.35736 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:40:53.252 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:53 smithi102 ceph-mon[110183]: pgmap v884: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 5.2 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 194 op/s; 30270/255219 objects degraded (11.860%); 5840/255219 objects misplaced (2.288%); 3.0 MiB/s, 48 objects/s recovering 2024-09-13T14:40:53.252 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:53 smithi102 ceph-mon[110183]: from='client.35736 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:40:53.253 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:53 smithi102 ceph-mon[115269]: pgmap v884: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 5.2 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 194 op/s; 30270/255219 objects degraded (11.860%); 5840/255219 objects misplaced (2.288%); 3.0 MiB/s, 48 objects/s recovering 2024-09-13T14:40:53.253 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:53 smithi102 ceph-mon[115269]: from='client.35736 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:40:53.279 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:53.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:53.305 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:53.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:53.406 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:53.430 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:53.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:53.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:53.532 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:53.554 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:53.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:53.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:53.603 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:53.658 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:53.683 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:53.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:53.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:53.786 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:53.808 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:53.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:53.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:53.910 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:53.934 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:53.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:53.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:54.036 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:54.061 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:54.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:54.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:54.163 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:54.186 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:54.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:54.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:54.288 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:54.312 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:54.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:54.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:54.353 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:54 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 30270/255219 objects degraded (11.860%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:54.354 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:54 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 30270/255219 objects degraded (11.860%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:54.413 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:54.429 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:54 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 30270/255219 objects degraded (11.860%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:54.438 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:54.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:54.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:54.506 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:54.540 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:54.568 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:54.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:54.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:54.670 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:54.695 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:54.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:54.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:54.797 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:54.816 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:40:54.816 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:40:54.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:54.831 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:54.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:54.836 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:40:54.837 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:40:54.854 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:40:54.854 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:40:54.873 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:40:54.873 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:40:54.883 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:40:54.883 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-09-13T14:40:54.883 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-09-13T14:40:54.883 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-13T14:40:54.883 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.884 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:85610 6.e 6:722b26f6:::benchmark_data_smithi102_194590_object85609:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-13T14:40:54.884 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:32:17.478842+0000", 2024-09-13T14:40:54.884 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 517.40147421100005, 2024-09-13T14:40:54.884 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.17296241900000001, 2024-09-13T14:40:54.884 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:40:54.884 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:40:54.884 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:40:54.884 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:40:54.884 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:40:54.884 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 85610 2024-09-13T14:40:54.884 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.884 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:40:54.884 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.885 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:40:54.885 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.478842+0000", 2024-09-13T14:40:54.885 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:40:54.885 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.885 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.885 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:40:54.885 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.478842+0000", 2024-09-13T14:40:54.885 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:40:54.885 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.885 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.885 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:40:54.886 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.478841+0000", 2024-09-13T14:40:54.886 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:40:54.886 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.886 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.886 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:40:54.886 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.478869+0000", 2024-09-13T14:40:54.886 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7820000000000001e-05 2024-09-13T14:40:54.886 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.886 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.886 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:40:54.886 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.478878+0000", 2024-09-13T14:40:54.886 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.3999999999999998e-06 2024-09-13T14:40:54.886 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.887 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.887 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:40:54.887 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.478885+0000", 2024-09-13T14:40:54.887 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7549999999999997e-06 2024-09-13T14:40:54.887 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.887 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.887 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:40:54.887 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.478927+0000", 2024-09-13T14:40:54.887 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2305999999999998e-05 2024-09-13T14:40:54.887 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.887 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.887 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:40:54.887 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.479029+0000", 2024-09-13T14:40:54.887 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010176800000000001 2024-09-13T14:40:54.888 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.888 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.888 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,7", 2024-09-13T14:40:54.888 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.479131+0000", 2024-09-13T14:40:54.888 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010254700000000001 2024-09-13T14:40:54.888 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.888 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.888 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:40:54.888 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.481132+0000", 2024-09-13T14:40:54.888 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0020003070000000002 2024-09-13T14:40:54.888 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.888 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.888 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:40:54.889 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.481169+0000", 2024-09-13T14:40:54.889 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7768000000000002e-05 2024-09-13T14:40:54.889 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.889 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.889 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:40:54.889 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.651724+0000", 2024-09-13T14:40:54.889 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.170555026 2024-09-13T14:40:54.889 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.889 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.889 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:40:54.889 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.651765+0000", 2024-09-13T14:40:54.890 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0918999999999999e-05 2024-09-13T14:40:54.890 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.890 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.890 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:40:54.890 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:32:17.651804+0000", 2024-09-13T14:40:54.890 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8798999999999999e-05 2024-09-13T14:40:54.890 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:40:54.890 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:40:54.890 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:40:54.890 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.890 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.890 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:150704 6.2 6:420df80d:::benchmark_data_smithi102_194590_object40080:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-13T14:40:54.891 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:33:27.750319+0000", 2024-09-13T14:40:54.891 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 447.12999687400003, 2024-09-13T14:40:54.891 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.29454005300000002, 2024-09-13T14:40:54.891 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:40:54.891 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:40:54.891 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:40:54.891 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:40:54.891 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:40:54.891 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 150704 2024-09-13T14:40:54.891 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.891 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:40:54.891 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.891 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:40:54.891 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.750319+0000", 2024-09-13T14:40:54.892 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:40:54.892 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.892 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.892 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:40:54.892 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.750319+0000", 2024-09-13T14:40:54.892 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:40:54.892 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.892 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.892 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:40:54.892 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.750317+0000", 2024-09-13T14:40:54.892 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:40:54.892 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.892 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.893 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:40:54.893 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.750326+0000", 2024-09-13T14:40:54.893 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1650000000000006e-06 2024-09-13T14:40:54.893 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.893 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.893 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:40:54.893 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.750341+0000", 2024-09-13T14:40:54.893 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5962999999999999e-05 2024-09-13T14:40:54.893 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.893 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.893 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:40:54.893 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.750350+0000", 2024-09-13T14:40:54.893 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5609999999999999e-06 2024-09-13T14:40:54.893 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.893 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.894 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:40:54.894 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.754163+0000", 2024-09-13T14:40:54.894 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0038129499999999998 2024-09-13T14:40:54.894 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.894 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.894 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:40:54.894 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.754287+0000", 2024-09-13T14:40:54.894 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012447200000000001 2024-09-13T14:40:54.894 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.894 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.894 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:40:54.894 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.754398+0000", 2024-09-13T14:40:54.894 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000110999 2024-09-13T14:40:54.894 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.895 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.895 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:40:54.895 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.044342+0000", 2024-09-13T14:40:54.895 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.28994310000000001 2024-09-13T14:40:54.895 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.895 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.895 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:40:54.895 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.044753+0000", 2024-09-13T14:40:54.895 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000411925 2024-09-13T14:40:54.895 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.895 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.895 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:40:54.895 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.044780+0000", 2024-09-13T14:40:54.895 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.6696999999999999e-05 2024-09-13T14:40:54.896 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.896 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.896 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:40:54.896 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.044820+0000", 2024-09-13T14:40:54.896 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9821999999999999e-05 2024-09-13T14:40:54.896 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.896 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.896 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:40:54.896 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.044859+0000", 2024-09-13T14:40:54.896 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9088999999999998e-05 2024-09-13T14:40:54.896 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:40:54.896 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:40:54.896 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:40:54.896 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.897 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.897 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:150708 6.2 6:4865044e:::benchmark_data_smithi102_194590_object40084:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-13T14:40:54.897 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:33:27.754180+0000", 2024-09-13T14:40:54.897 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 447.126135584, 2024-09-13T14:40:54.897 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.29088824600000002, 2024-09-13T14:40:54.897 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:40:54.897 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:40:54.897 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:40:54.897 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:40:54.897 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:40:54.897 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 150708 2024-09-13T14:40:54.898 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:40:54.898 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:40:54.899 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.899 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:40:54.899 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.899 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:40:54.899 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.754180+0000", 2024-09-13T14:40:54.899 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:40:54.899 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.899 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.899 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:40:54.899 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.754180+0000", 2024-09-13T14:40:54.899 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:40:54.899 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.900 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.900 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:40:54.900 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.754179+0000", 2024-09-13T14:40:54.900 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:40:54.900 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.900 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.900 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:40:54.900 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.754184+0000", 2024-09-13T14:40:54.900 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2429999999999996e-06 2024-09-13T14:40:54.900 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.900 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.900 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:40:54.901 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.754194+0000", 2024-09-13T14:40:54.901 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6490000000000003e-06 2024-09-13T14:40:54.901 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.901 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.901 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:40:54.901 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.754201+0000", 2024-09-13T14:40:54.901 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9190000000000002e-06 2024-09-13T14:40:54.901 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.901 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.901 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:40:54.901 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.757984+0000", 2024-09-13T14:40:54.901 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0037832400000000002 2024-09-13T14:40:54.902 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.902 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.902 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:40:54.902 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.758104+0000", 2024-09-13T14:40:54.902 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000120047 2024-09-13T14:40:54.902 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.902 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.902 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:40:54.902 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.758220+0000", 2024-09-13T14:40:54.902 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011538400000000001 2024-09-13T14:40:54.902 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.902 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.902 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:40:54.902 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.044352+0000", 2024-09-13T14:40:54.903 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.28613211500000002 2024-09-13T14:40:54.903 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.903 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.903 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:40:54.903 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.044970+0000", 2024-09-13T14:40:54.903 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00061775299999999997 2024-09-13T14:40:54.903 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.903 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.903 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:40:54.903 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.045002+0000", 2024-09-13T14:40:54.903 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2330999999999999e-05 2024-09-13T14:40:54.903 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.903 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.904 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:40:54.904 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.045037+0000", 2024-09-13T14:40:54.904 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5042000000000001e-05 2024-09-13T14:40:54.904 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.904 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.904 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:40:54.904 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.045069+0000", 2024-09-13T14:40:54.904 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.1695000000000001e-05 2024-09-13T14:40:54.904 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:40:54.904 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:40:54.904 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:40:54.904 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.904 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.904 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:150715 6.e 6:7aec09e5:::benchmark_data_smithi102_194590_object40091:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-13T14:40:54.905 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:33:27.761632+0000", 2024-09-13T14:40:54.905 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 447.11868415200001, 2024-09-13T14:40:54.905 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.29790872499999999, 2024-09-13T14:40:54.905 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:40:54.905 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:40:54.905 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:40:54.905 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:40:54.905 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:40:54.905 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 150715 2024-09-13T14:40:54.905 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.905 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:40:54.905 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.906 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:40:54.906 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.761632+0000", 2024-09-13T14:40:54.906 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:40:54.906 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.906 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.906 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:40:54.906 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.761632+0000", 2024-09-13T14:40:54.906 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:40:54.906 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.906 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.906 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:40:54.906 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.761630+0000", 2024-09-13T14:40:54.906 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:40:54.907 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.907 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.907 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:40:54.907 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.761638+0000", 2024-09-13T14:40:54.907 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1049999999999995e-06 2024-09-13T14:40:54.907 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.907 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.907 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:40:54.907 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.044265+0000", 2024-09-13T14:40:54.907 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.28262651100000002 2024-09-13T14:40:54.907 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.907 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.907 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:40:54.907 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.044280+0000", 2024-09-13T14:40:54.908 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5600999999999999e-05 2024-09-13T14:40:54.908 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.908 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.908 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:40:54.908 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.044408+0000", 2024-09-13T14:40:54.908 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012831499999999999 2024-09-13T14:40:54.908 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.908 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.908 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:40:54.908 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.044549+0000", 2024-09-13T14:40:54.908 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014089899999999999 2024-09-13T14:40:54.908 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.908 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.908 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,7", 2024-09-13T14:40:54.909 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.044679+0000", 2024-09-13T14:40:54.909 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012917399999999999 2024-09-13T14:40:54.909 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.909 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.909 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:40:54.909 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.059197+0000", 2024-09-13T14:40:54.909 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014518939999999999 2024-09-13T14:40:54.909 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.909 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.909 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:40:54.909 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.059225+0000", 2024-09-13T14:40:54.909 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7288999999999999e-05 2024-09-13T14:40:54.909 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.910 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.910 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:40:54.910 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.059358+0000", 2024-09-13T14:40:54.910 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000133122 2024-09-13T14:40:54.910 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.910 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.910 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:40:54.910 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.059449+0000", 2024-09-13T14:40:54.910 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1187000000000006e-05 2024-09-13T14:40:54.910 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.910 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.910 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:40:54.910 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.059540+0000", 2024-09-13T14:40:54.910 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1434000000000006e-05 2024-09-13T14:40:54.911 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:40:54.911 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:40:54.911 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:40:54.911 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.911 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.911 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35037.0:150736 6.7 e178/161 6:eea7a7c2:::benchmark_data_smithi102_194590_object40112:head v 178'9515, mlcod=178'9513)", 2024-09-13T14:40:54.911 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:33:27.781737+0000", 2024-09-13T14:40:54.911 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 447.09857938599998, 2024-09-13T14:40:54.911 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.283366118, 2024-09-13T14:40:54.911 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:40:54.911 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:40:54.911 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:40:54.911 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.911 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:40:54.912 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.781737+0000", 2024-09-13T14:40:54.912 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:40:54.912 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.912 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.912 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:40:54.912 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.781737+0000", 2024-09-13T14:40:54.912 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:40:54.912 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.912 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.912 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:40:54.912 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.781735+0000", 2024-09-13T14:40:54.912 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:40:54.912 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.913 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.913 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:40:54.913 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:27.781744+0000", 2024-09-13T14:40:54.913 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4760000000000004e-06 2024-09-13T14:40:54.913 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.913 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.913 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:40:54.913 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.044267+0000", 2024-09-13T14:40:54.913 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.26252262599999998 2024-09-13T14:40:54.913 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.913 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.913 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:40:54.913 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.044279+0000", 2024-09-13T14:40:54.913 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2354000000000001e-05 2024-09-13T14:40:54.914 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.914 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.914 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:40:54.914 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.044431+0000", 2024-09-13T14:40:54.914 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0001516 2024-09-13T14:40:54.914 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.914 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.914 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:40:54.914 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.044453+0000", 2024-09-13T14:40:54.914 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2124999999999999e-05 2024-09-13T14:40:54.914 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.914 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.914 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:40:54.915 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.065076+0000", 2024-09-13T14:40:54.915 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.020622628 2024-09-13T14:40:54.915 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.915 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.915 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:40:54.915 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:28.065103+0000", 2024-09-13T14:40:54.915 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7078999999999999e-05 2024-09-13T14:40:54.915 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:40:54.915 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:40:54.915 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:40:54.915 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.915 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.915 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:163490 6.e 6:7a5e8c84:::benchmark_data_smithi102_194590_object52866:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-13T14:40:54.915 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:33:43.515900+0000", 2024-09-13T14:40:54.915 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 431.364415576, 2024-09-13T14:40:54.916 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.555967597, 2024-09-13T14:40:54.916 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:40:54.916 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:40:54.916 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:40:54.916 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:40:54.916 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:40:54.916 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 163490 2024-09-13T14:40:54.916 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.916 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:40:54.916 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.916 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:40:54.916 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.515900+0000", 2024-09-13T14:40:54.916 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:40:54.917 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.917 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.917 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:40:54.917 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.515900+0000", 2024-09-13T14:40:54.917 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:40:54.917 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.917 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.917 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:40:54.917 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.515899+0000", 2024-09-13T14:40:54.917 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:40:54.917 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.917 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.917 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:40:54.917 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.515905+0000", 2024-09-13T14:40:54.918 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3219999999999998e-06 2024-09-13T14:40:54.918 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.918 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.918 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:40:54.918 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.515929+0000", 2024-09-13T14:40:54.918 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3445e-05 2024-09-13T14:40:54.918 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.918 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.918 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:40:54.918 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.515937+0000", 2024-09-13T14:40:54.918 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2530000000000004e-06 2024-09-13T14:40:54.918 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.918 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.919 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:40:54.919 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.533987+0000", 2024-09-13T14:40:54.919 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018049744 2024-09-13T14:40:54.919 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.919 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.919 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:40:54.919 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.533995+0000", 2024-09-13T14:40:54.919 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0250000000000002e-06 2024-09-13T14:40:54.919 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.919 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.919 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:40:54.919 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.820467+0000", 2024-09-13T14:40:54.919 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.28647251400000001 2024-09-13T14:40:54.919 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.920 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.920 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:40:54.920 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.820483+0000", 2024-09-13T14:40:54.920 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.596e-05 2024-09-13T14:40:54.920 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.920 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.920 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:40:54.920 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.025746+0000", 2024-09-13T14:40:54.920 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2052628699999999 2024-09-13T14:40:54.920 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.920 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.920 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:40:54.920 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.025880+0000", 2024-09-13T14:40:54.920 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000134039 2024-09-13T14:40:54.920 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.921 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.921 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6,7", 2024-09-13T14:40:54.921 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.026451+0000", 2024-09-13T14:40:54.921 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00057072799999999995 2024-09-13T14:40:54.921 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.921 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.921 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:40:54.921 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.061832+0000", 2024-09-13T14:40:54.921 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.035381126999999998 2024-09-13T14:40:54.921 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.921 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.921 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:40:54.921 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.063912+0000", 2024-09-13T14:40:54.922 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0020797739999999999 2024-09-13T14:40:54.922 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.922 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.922 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:40:54.922 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.066465+0000", 2024-09-13T14:40:54.922 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0025533280000000001 2024-09-13T14:40:54.922 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.922 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.922 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:40:54.922 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.071666+0000", 2024-09-13T14:40:54.922 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0052006839999999997 2024-09-13T14:40:54.922 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.922 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.922 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:40:54.922 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.071756+0000", 2024-09-13T14:40:54.923 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.0733e-05 2024-09-13T14:40:54.923 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.923 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.923 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:40:54.923 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.071868+0000", 2024-09-13T14:40:54.923 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000111537 2024-09-13T14:40:54.923 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:40:54.923 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:40:54.923 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:40:54.923 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.923 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.923 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:163524 6.e 6:7ba5d4da:::benchmark_data_smithi102_194590_object52900:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-09-13T14:40:54.923 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:33:43.559673+0000", 2024-09-13T14:40:54.923 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 431.32064330100002, 2024-09-13T14:40:54.924 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5124033290000001, 2024-09-13T14:40:54.924 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:40:54.924 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:40:54.924 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:40:54.924 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:40:54.924 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:40:54.924 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 163524 2024-09-13T14:40:54.924 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.924 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:40:54.924 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.924 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:40:54.924 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.559673+0000", 2024-09-13T14:40:54.924 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:40:54.924 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.925 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.925 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:40:54.925 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.559673+0000", 2024-09-13T14:40:54.925 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:40:54.925 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.925 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.925 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:40:54.925 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.559672+0000", 2024-09-13T14:40:54.925 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:40:54.925 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.925 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.925 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:40:54.925 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.559676+0000", 2024-09-13T14:40:54.925 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6859999999999997e-06 2024-09-13T14:40:54.925 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.926 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.926 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:40:54.926 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.559687+0000", 2024-09-13T14:40:54.926 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0514e-05 2024-09-13T14:40:54.926 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.926 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.926 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:40:54.926 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.559691+0000", 2024-09-13T14:40:54.926 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4989999999999996e-06 2024-09-13T14:40:54.926 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.926 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.926 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:40:54.926 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.559730+0000", 2024-09-13T14:40:54.927 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8816999999999998e-05 2024-09-13T14:40:54.927 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.927 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.927 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:40:54.927 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.559740+0000", 2024-09-13T14:40:54.927 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.5319999999999995e-06 2024-09-13T14:40:54.927 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.927 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.927 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:40:54.927 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.820610+0000", 2024-09-13T14:40:54.927 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.26086995699999999 2024-09-13T14:40:54.927 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.927 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.927 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:40:54.928 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.820618+0000", 2024-09-13T14:40:54.928 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.6580000000000008e-06 2024-09-13T14:40:54.928 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.928 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.928 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:40:54.928 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.040556+0000", 2024-09-13T14:40:54.928 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2199382860000001 2024-09-13T14:40:54.928 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.928 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.928 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:40:54.928 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.040691+0000", 2024-09-13T14:40:54.928 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000134497 2024-09-13T14:40:54.928 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.929 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.929 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6,7", 2024-09-13T14:40:54.929 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.040831+0000", 2024-09-13T14:40:54.929 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000140359 2024-09-13T14:40:54.929 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.929 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.929 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:40:54.929 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.061864+0000", 2024-09-13T14:40:54.929 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.021032660000000002 2024-09-13T14:40:54.929 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.929 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.929 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:40:54.929 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.066495+0000", 2024-09-13T14:40:54.930 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0046312369999999999 2024-09-13T14:40:54.930 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.930 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.930 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:40:54.930 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.068838+0000", 2024-09-13T14:40:54.930 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0023430949999999999 2024-09-13T14:40:54.930 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.930 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.930 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:40:54.930 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.071910+0000", 2024-09-13T14:40:54.930 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0030720309999999998 2024-09-13T14:40:54.930 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.930 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.930 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:40:54.931 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.071984+0000", 2024-09-13T14:40:54.931 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3590000000000005e-05 2024-09-13T14:40:54.931 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.931 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.931 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:40:54.931 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.072076+0000", 2024-09-13T14:40:54.931 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1986999999999999e-05 2024-09-13T14:40:54.931 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:40:54.931 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:40:54.931 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:40:54.931 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.931 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.931 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:163533 6.e 6:70a00fb8:::benchmark_data_smithi102_194590_object52909:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-09-13T14:40:54.931 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:33:43.580375+0000", 2024-09-13T14:40:54.932 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 431.29994129800002, 2024-09-13T14:40:54.932 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4918938339999999, 2024-09-13T14:40:54.932 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:40:54.932 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:40:54.932 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:40:54.932 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:40:54.932 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:40:54.932 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 163533 2024-09-13T14:40:54.932 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.932 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:40:54.932 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.932 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:40:54.932 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.580375+0000", 2024-09-13T14:40:54.933 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:40:54.933 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.933 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.933 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:40:54.933 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.580375+0000", 2024-09-13T14:40:54.933 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:40:54.933 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.933 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.933 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:40:54.933 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.580373+0000", 2024-09-13T14:40:54.933 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:40:54.933 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.933 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.933 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:40:54.934 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.580380+0000", 2024-09-13T14:40:54.934 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6980000000000001e-06 2024-09-13T14:40:54.934 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.934 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.934 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:40:54.934 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.580392+0000", 2024-09-13T14:40:54.934 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2675e-05 2024-09-13T14:40:54.934 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.934 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.934 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:40:54.934 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.580399+0000", 2024-09-13T14:40:54.934 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.674e-06 2024-09-13T14:40:54.934 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.935 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.935 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:40:54.935 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.580441+0000", 2024-09-13T14:40:54.935 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2077999999999999e-05 2024-09-13T14:40:54.935 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.935 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.935 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:40:54.935 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.580449+0000", 2024-09-13T14:40:54.935 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7300000000000005e-06 2024-09-13T14:40:54.935 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.935 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.935 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:40:54.936 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.820640+0000", 2024-09-13T14:40:54.936 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.240191126 2024-09-13T14:40:54.936 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.936 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.936 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:40:54.936 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:43.820648+0000", 2024-09-13T14:40:54.936 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0469999999999994e-06 2024-09-13T14:40:54.936 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.936 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.936 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:40:54.936 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.043836+0000", 2024-09-13T14:40:54.936 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2231876559999999 2024-09-13T14:40:54.936 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.937 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.937 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:40:54.937 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.043970+0000", 2024-09-13T14:40:54.937 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013394399999999999 2024-09-13T14:40:54.937 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.937 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.937 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6,7", 2024-09-13T14:40:54.937 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.044102+0000", 2024-09-13T14:40:54.937 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000132254 2024-09-13T14:40:54.937 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.937 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.937 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:40:54.937 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.061893+0000", 2024-09-13T14:40:54.937 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017791238000000001 2024-09-13T14:40:54.938 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.938 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.938 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:40:54.938 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.066525+0000", 2024-09-13T14:40:54.938 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0046314099999999999 2024-09-13T14:40:54.938 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.938 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.938 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:40:54.938 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.068872+0000", 2024-09-13T14:40:54.938 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002347605 2024-09-13T14:40:54.938 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.938 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.938 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:40:54.938 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.072115+0000", 2024-09-13T14:40:54.939 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0032423840000000001 2024-09-13T14:40:54.939 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.939 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.939 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:40:54.939 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.072177+0000", 2024-09-13T14:40:54.939 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2679000000000001e-05 2024-09-13T14:40:54.939 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.939 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.939 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:40:54.939 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:33:45.072268+0000", 2024-09-13T14:40:54.939 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1260000000000004e-05 2024-09-13T14:40:54.939 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:40:54.939 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:40:54.940 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:40:54.940 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.940 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.940 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:191635 6.e 6:71f9b24e:::benchmark_data_smithi102_194590_object81011:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-09-13T14:40:54.940 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:34:22.367198+0000", 2024-09-13T14:40:54.940 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 392.51311784000001, 2024-09-13T14:40:54.940 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.26435987, 2024-09-13T14:40:54.940 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:40:54.940 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:40:54.940 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:40:54.940 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:40:54.940 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:40:54.940 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 191635 2024-09-13T14:40:54.940 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.941 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:40:54.941 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.941 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:40:54.941 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.367198+0000", 2024-09-13T14:40:54.941 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:54.942 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:40:54.942 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:40:54.943 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:40:54.944 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:40:54.944 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:40:54.944 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.944 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.944 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:40:54.945 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.367198+0000", 2024-09-13T14:40:54.945 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:40:54.945 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.945 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.945 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:40:54.945 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.367197+0000", 2024-09-13T14:40:54.946 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:40:54.946 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.946 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.946 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:40:54.946 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.367203+0000", 2024-09-13T14:40:54.946 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.455e-06 2024-09-13T14:40:54.946 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.946 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.946 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:40:54.946 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.367222+0000", 2024-09-13T14:40:54.946 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8655e-05 2024-09-13T14:40:54.946 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.947 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.947 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:40:54.947 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.367230+0000", 2024-09-13T14:40:54.947 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7379999999999994e-06 2024-09-13T14:40:54.947 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.947 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.947 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:40:54.947 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.367927+0000", 2024-09-13T14:40:54.947 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00069707899999999997 2024-09-13T14:40:54.947 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.948 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.948 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:40:54.948 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.368049+0000", 2024-09-13T14:40:54.948 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000122211 2024-09-13T14:40:54.948 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.948 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.948 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6,7", 2024-09-13T14:40:54.948 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.368168+0000", 2024-09-13T14:40:54.948 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000119357 2024-09-13T14:40:54.948 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.948 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.948 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:40:54.948 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.371968+0000", 2024-09-13T14:40:54.949 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003800159 2024-09-13T14:40:54.949 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.949 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.949 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:40:54.949 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.380532+0000", 2024-09-13T14:40:54.949 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0085635420000000004 2024-09-13T14:40:54.949 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.949 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.949 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:40:54.949 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.380575+0000", 2024-09-13T14:40:54.949 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3392000000000001e-05 2024-09-13T14:40:54.949 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.949 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.949 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:40:54.950 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.631340+0000", 2024-09-13T14:40:54.950 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.250764879 2024-09-13T14:40:54.950 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.950 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.950 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:40:54.950 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.631451+0000", 2024-09-13T14:40:54.950 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000111078 2024-09-13T14:40:54.950 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.950 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.950 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:40:54.950 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.631558+0000", 2024-09-13T14:40:54.950 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010673799999999999 2024-09-13T14:40:54.950 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:40:54.950 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:40:54.951 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:40:54.951 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.951 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.951 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:191643 6.e 6:7b30bcbb:::benchmark_data_smithi102_194590_object81019:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-09-13T14:40:54.951 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:34:22.372207+0000", 2024-09-13T14:40:54.951 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 392.508109112, 2024-09-13T14:40:54.951 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.26177498799999999, 2024-09-13T14:40:54.951 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:40:54.951 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:40:54.951 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:40:54.951 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:40:54.951 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:40:54.951 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 191643 2024-09-13T14:40:54.952 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.952 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:40:54.952 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.952 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:40:54.952 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.372207+0000", 2024-09-13T14:40:54.952 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:40:54.952 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.952 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.952 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:40:54.952 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.372207+0000", 2024-09-13T14:40:54.952 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:40:54.952 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.952 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.952 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:40:54.953 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.372205+0000", 2024-09-13T14:40:54.953 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:40:54.953 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.953 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.953 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:40:54.953 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.372213+0000", 2024-09-13T14:40:54.953 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1400000000000002e-06 2024-09-13T14:40:54.953 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.953 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.953 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:40:54.953 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.372225+0000", 2024-09-13T14:40:54.953 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2577e-05 2024-09-13T14:40:54.953 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.954 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.954 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:40:54.954 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.372232+0000", 2024-09-13T14:40:54.954 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7139999999999996e-06 2024-09-13T14:40:54.954 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.954 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.954 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:40:54.954 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.372269+0000", 2024-09-13T14:40:54.954 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7327999999999999e-05 2024-09-13T14:40:54.954 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.954 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.954 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:40:54.954 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.372400+0000", 2024-09-13T14:40:54.954 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000131085 2024-09-13T14:40:54.955 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.955 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.955 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6,7", 2024-09-13T14:40:54.955 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.372529+0000", 2024-09-13T14:40:54.955 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012839700000000001 2024-09-13T14:40:54.955 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.955 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.955 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:40:54.955 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.380540+0000", 2024-09-13T14:40:54.955 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0080113100000000007 2024-09-13T14:40:54.955 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.955 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.955 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:40:54.955 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.380608+0000", 2024-09-13T14:40:54.956 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7725000000000006e-05 2024-09-13T14:40:54.956 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.956 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.956 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:40:54.956 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.380655+0000", 2024-09-13T14:40:54.956 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7085000000000001e-05 2024-09-13T14:40:54.956 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.956 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.956 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:40:54.956 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.631591+0000", 2024-09-13T14:40:54.956 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.25093650699999998 2024-09-13T14:40:54.956 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.956 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.956 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:40:54.957 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.633854+0000", 2024-09-13T14:40:54.957 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0022624239999999999 2024-09-13T14:40:54.957 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.957 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.957 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:40:54.957 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.633982+0000", 2024-09-13T14:40:54.957 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012812899999999999 2024-09-13T14:40:54.957 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:40:54.957 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:40:54.957 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:40:54.957 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.957 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.957 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:191647 6.e 6:76d424e6:::benchmark_data_smithi102_194590_object81023:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-09-13T14:40:54.957 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:34:22.373106+0000", 2024-09-13T14:40:54.958 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 392.507209644, 2024-09-13T14:40:54.958 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.26104791700000002, 2024-09-13T14:40:54.958 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:40:54.958 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:40:54.958 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:40:54.958 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:40:54.958 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:40:54.958 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 191647 2024-09-13T14:40:54.958 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.958 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:40:54.958 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.958 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:40:54.958 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.373106+0000", 2024-09-13T14:40:54.958 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:40:54.959 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.959 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.959 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:40:54.959 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.373106+0000", 2024-09-13T14:40:54.959 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:40:54.959 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.959 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.959 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:40:54.959 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.373105+0000", 2024-09-13T14:40:54.959 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:40:54.959 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.959 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.959 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:40:54.959 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.373110+0000", 2024-09-13T14:40:54.960 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5220000000000001e-06 2024-09-13T14:40:54.960 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.960 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.960 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:40:54.960 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.373134+0000", 2024-09-13T14:40:54.960 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4409e-05 2024-09-13T14:40:54.960 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.960 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.960 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:40:54.960 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.373141+0000", 2024-09-13T14:40:54.960 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4860000000000002e-06 2024-09-13T14:40:54.960 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.960 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.961 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:40:54.961 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.377126+0000", 2024-09-13T14:40:54.961 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003985321 2024-09-13T14:40:54.961 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.961 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.961 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:40:54.961 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.377248+0000", 2024-09-13T14:40:54.961 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012149199999999999 2024-09-13T14:40:54.961 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.961 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.961 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6,7", 2024-09-13T14:40:54.961 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.377456+0000", 2024-09-13T14:40:54.961 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00020867600000000001 2024-09-13T14:40:54.961 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.962 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.962 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:40:54.962 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.381454+0000", 2024-09-13T14:40:54.962 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0039976029999999997 2024-09-13T14:40:54.962 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.962 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.962 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:40:54.962 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.382225+0000", 2024-09-13T14:40:54.962 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00077125399999999997 2024-09-13T14:40:54.962 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.962 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.962 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:40:54.962 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.383736+0000", 2024-09-13T14:40:54.962 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0015105279999999999 2024-09-13T14:40:54.963 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.963 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.963 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:40:54.963 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.634027+0000", 2024-09-13T14:40:54.963 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.25029173100000002 2024-09-13T14:40:54.963 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.963 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.963 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:40:54.963 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.634065+0000", 2024-09-13T14:40:54.963 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7135000000000003e-05 2024-09-13T14:40:54.963 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.963 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.963 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:40:54.963 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.634154+0000", 2024-09-13T14:40:54.964 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.9624999999999997e-05 2024-09-13T14:40:54.964 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:40:54.964 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:40:54.964 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:40:54.964 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.964 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.964 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:191656 6.e 6:73cf8b7b:::benchmark_data_smithi102_194590_object81032:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-09-13T14:40:54.965 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:34:22.389704+0000", 2024-09-13T14:40:54.965 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 392.49061218899999, 2024-09-13T14:40:54.965 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.24469366300000001, 2024-09-13T14:40:54.965 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:40:54.965 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:40:54.965 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:40:54.965 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:40:54.966 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:40:54.966 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 191656 2024-09-13T14:40:54.966 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.966 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:40:54.966 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.966 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:40:54.966 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.389704+0000", 2024-09-13T14:40:54.966 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:40:54.966 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.967 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.967 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:40:54.967 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.389704+0000", 2024-09-13T14:40:54.967 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:40:54.967 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.967 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.967 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:40:54.967 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.389702+0000", 2024-09-13T14:40:54.967 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:40:54.967 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.967 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.967 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:40:54.967 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.389708+0000", 2024-09-13T14:40:54.967 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2670000000000003e-06 2024-09-13T14:40:54.968 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.968 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.968 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:40:54.968 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.389723+0000", 2024-09-13T14:40:54.968 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4175000000000001e-05 2024-09-13T14:40:54.968 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.968 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.968 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:40:54.968 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.389730+0000", 2024-09-13T14:40:54.968 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5480000000000004e-06 2024-09-13T14:40:54.968 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.968 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.968 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:40:54.969 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.389767+0000", 2024-09-13T14:40:54.969 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7123999999999999e-05 2024-09-13T14:40:54.969 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.969 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.969 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:40:54.969 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.389868+0000", 2024-09-13T14:40:54.969 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010078599999999999 2024-09-13T14:40:54.969 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.969 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.969 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6,7", 2024-09-13T14:40:54.969 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.390004+0000", 2024-09-13T14:40:54.969 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000135578 2024-09-13T14:40:54.969 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.969 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.970 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:40:54.970 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.393200+0000", 2024-09-13T14:40:54.970 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0031962649999999998 2024-09-13T14:40:54.970 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.970 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.970 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:40:54.970 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.396346+0000", 2024-09-13T14:40:54.970 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003146009 2024-09-13T14:40:54.970 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.970 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.970 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:40:54.970 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.396394+0000", 2024-09-13T14:40:54.970 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8588000000000003e-05 2024-09-13T14:40:54.970 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.970 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.971 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:40:54.971 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.634261+0000", 2024-09-13T14:40:54.971 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.237866258 2024-09-13T14:40:54.971 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.971 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.971 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:40:54.971 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.634301+0000", 2024-09-13T14:40:54.971 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.019e-05 2024-09-13T14:40:54.971 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.971 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.971 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:40:54.971 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.634397+0000", 2024-09-13T14:40:54.971 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6477999999999993e-05 2024-09-13T14:40:54.972 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:40:54.972 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:40:54.972 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:40:54.972 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.972 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.972 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:191660 6.e 6:738104a9:::benchmark_data_smithi102_194590_object81036:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-09-13T14:40:54.972 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:34:22.392930+0000", 2024-09-13T14:40:54.972 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 392.48738563400002, 2024-09-13T14:40:54.972 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.24195140200000001, 2024-09-13T14:40:54.972 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:40:54.972 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:40:54.972 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:40:54.972 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:40:54.972 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:40:54.972 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 191660 2024-09-13T14:40:54.973 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.973 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:40:54.973 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.973 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:40:54.973 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.392930+0000", 2024-09-13T14:40:54.973 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:40:54.973 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.973 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.973 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:40:54.973 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.392930+0000", 2024-09-13T14:40:54.973 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:40:54.973 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.973 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.973 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:40:54.974 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.392929+0000", 2024-09-13T14:40:54.974 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:40:54.974 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.974 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.974 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:40:54.974 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.392935+0000", 2024-09-13T14:40:54.974 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0859999999999998e-06 2024-09-13T14:40:54.974 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.974 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.974 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:40:54.974 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.392948+0000", 2024-09-13T14:40:54.974 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2529e-05 2024-09-13T14:40:54.974 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.975 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.975 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:40:54.975 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.392954+0000", 2024-09-13T14:40:54.975 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9920000000000001e-06 2024-09-13T14:40:54.975 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.975 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.975 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:40:54.975 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.393326+0000", 2024-09-13T14:40:54.975 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00037141000000000002 2024-09-13T14:40:54.975 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.975 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.975 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:40:54.976 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.393461+0000", 2024-09-13T14:40:54.976 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013538499999999999 2024-09-13T14:40:54.976 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.976 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.976 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6,7", 2024-09-13T14:40:54.976 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.393618+0000", 2024-09-13T14:40:54.976 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000156541 2024-09-13T14:40:54.976 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.976 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.976 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:40:54.976 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.396432+0000", 2024-09-13T14:40:54.976 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00281445 2024-09-13T14:40:54.976 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.976 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.977 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:40:54.977 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.396443+0000", 2024-09-13T14:40:54.977 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0852999999999999e-05 2024-09-13T14:40:54.977 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.977 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.977 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:40:54.977 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.396763+0000", 2024-09-13T14:40:54.977 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00031982700000000001 2024-09-13T14:40:54.977 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.977 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.977 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:40:54.977 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.634751+0000", 2024-09-13T14:40:54.977 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.23798829299999999 2024-09-13T14:40:54.978 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.978 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.978 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:40:54.978 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.634794+0000", 2024-09-13T14:40:54.978 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2425999999999997e-05 2024-09-13T14:40:54.978 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.978 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.978 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:40:54.978 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.634882+0000", 2024-09-13T14:40:54.978 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8011999999999997e-05 2024-09-13T14:40:54.979 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:40:54.979 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:40:54.979 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:40:54.979 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.979 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.979 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:198028 6.e 6:71106726:::benchmark_data_smithi102_194590_object87404:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-09-13T14:40:54.979 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:34:29.683014+0000", 2024-09-13T14:40:54.979 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 385.197301573, 2024-09-13T14:40:54.979 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.22173479500000001, 2024-09-13T14:40:54.979 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:40:54.979 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:40:54.979 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:40:54.979 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:40:54.980 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:40:54.980 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 198028 2024-09-13T14:40:54.980 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.980 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:40:54.980 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.980 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:40:54.980 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.683014+0000", 2024-09-13T14:40:54.980 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:40:54.980 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.980 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.980 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:40:54.980 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.683014+0000", 2024-09-13T14:40:54.980 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:40:54.981 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.981 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.981 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:40:54.981 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.683013+0000", 2024-09-13T14:40:54.981 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:40:54.981 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.981 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.981 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:40:54.981 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.683019+0000", 2024-09-13T14:40:54.982 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1550000000000004e-06 2024-09-13T14:40:54.982 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.982 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.982 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:40:54.982 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.683034+0000", 2024-09-13T14:40:54.982 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4796e-05 2024-09-13T14:40:54.982 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.982 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.982 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:40:54.982 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.683041+0000", 2024-09-13T14:40:54.983 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4699999999999996e-06 2024-09-13T14:40:54.983 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.983 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.983 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:40:54.983 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.685659+0000", 2024-09-13T14:40:54.983 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0026179699999999998 2024-09-13T14:40:54.983 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.983 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.983 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:40:54.983 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.685968+0000", 2024-09-13T14:40:54.983 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00030874299999999999 2024-09-13T14:40:54.983 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.983 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.983 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6,7", 2024-09-13T14:40:54.984 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.686387+0000", 2024-09-13T14:40:54.984 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00041937000000000002 2024-09-13T14:40:54.984 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.984 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.984 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:40:54.984 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.689774+0000", 2024-09-13T14:40:54.984 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0033865100000000001 2024-09-13T14:40:54.984 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.984 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.984 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:40:54.984 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.691584+0000", 2024-09-13T14:40:54.984 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0018108029999999999 2024-09-13T14:40:54.984 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.985 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.985 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:40:54.985 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.692138+0000", 2024-09-13T14:40:54.985 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00055365500000000001 2024-09-13T14:40:54.985 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.985 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.985 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:40:54.985 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.904593+0000", 2024-09-13T14:40:54.985 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.21245457600000001 2024-09-13T14:40:54.985 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.985 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.985 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:40:54.985 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.904642+0000", 2024-09-13T14:40:54.986 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9833999999999998e-05 2024-09-13T14:40:54.986 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.986 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.986 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:40:54.986 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.904749+0000", 2024-09-13T14:40:54.986 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000106693 2024-09-13T14:40:54.986 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:40:54.986 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:40:54.986 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:40:54.986 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.986 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.986 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:198030 6.2 6:416e1cf0:::benchmark_data_smithi102_194590_object87406:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-09-13T14:40:54.986 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:34:29.683047+0000", 2024-09-13T14:40:54.987 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 385.19726885699998, 2024-09-13T14:40:54.987 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.221895961, 2024-09-13T14:40:54.987 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:40:54.987 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:40:54.987 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:40:54.987 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:40:54.987 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:40:54.987 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 198030 2024-09-13T14:40:54.987 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.987 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:40:54.987 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.987 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:40:54.987 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.683047+0000", 2024-09-13T14:40:54.987 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:40:54.988 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.988 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.988 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:40:54.988 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.683047+0000", 2024-09-13T14:40:54.988 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:40:54.988 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.988 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.988 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:40:54.988 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.683047+0000", 2024-09-13T14:40:54.988 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:40:54.988 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.988 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.988 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:40:54.988 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.683049+0000", 2024-09-13T14:40:54.989 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.424e-06 2024-09-13T14:40:54.989 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.989 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.989 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:40:54.989 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.683058+0000", 2024-09-13T14:40:54.989 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7450000000000001e-06 2024-09-13T14:40:54.989 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.989 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.989 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:40:54.989 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.683063+0000", 2024-09-13T14:40:54.989 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0089999999999998e-06 2024-09-13T14:40:54.989 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.990 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.990 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:40:54.990 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.683110+0000", 2024-09-13T14:40:54.990 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7601e-05 2024-09-13T14:40:54.990 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.990 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.990 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:40:54.990 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.683250+0000", 2024-09-13T14:40:54.990 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013993600000000001 2024-09-13T14:40:54.990 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.990 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.991 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:40:54.991 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.683385+0000", 2024-09-13T14:40:54.991 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013476499999999999 2024-09-13T14:40:54.991 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.991 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.991 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:40:54.991 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.686273+0000", 2024-09-13T14:40:54.991 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0028874199999999999 2024-09-13T14:40:54.991 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.991 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.991 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:40:54.991 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.686610+0000", 2024-09-13T14:40:54.991 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00033694600000000002 2024-09-13T14:40:54.991 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.992 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.992 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:40:54.992 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.904866+0000", 2024-09-13T14:40:54.992 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.21825642100000001 2024-09-13T14:40:54.992 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.992 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.992 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:40:54.992 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.904903+0000", 2024-09-13T14:40:54.992 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7057000000000001e-05 2024-09-13T14:40:54.992 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.992 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.992 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:40:54.992 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.904943+0000", 2024-09-13T14:40:54.992 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0006999999999998e-05 2024-09-13T14:40:54.993 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:40:54.993 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:40:54.993 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:40:54.993 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.993 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.993 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113979 7.4 7:2d0634a9:::benchmark_data_smithi102_227395_object1164:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-13T14:40:54.993 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:47.339159+0000", 2024-09-13T14:40:54.993 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 187.54115666300001, 2024-09-13T14:40:54.993 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.356370809, 2024-09-13T14:40:54.993 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:40:54.993 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:40:54.993 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:40:54.993 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:40:54.993 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:40:54.994 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113979 2024-09-13T14:40:54.994 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.994 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:40:54.994 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.994 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:40:54.994 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.339159+0000", 2024-09-13T14:40:54.994 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:40:54.994 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.994 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.994 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:40:54.994 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.339159+0000", 2024-09-13T14:40:54.994 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:40:54.994 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.994 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.995 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:40:54.995 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.339158+0000", 2024-09-13T14:40:54.995 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:40:54.995 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.995 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.995 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:40:54.995 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.339165+0000", 2024-09-13T14:40:54.995 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4719999999999998e-06 2024-09-13T14:40:54.995 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.995 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.995 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:40:54.995 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.339178+0000", 2024-09-13T14:40:54.995 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.306e-05 2024-09-13T14:40:54.995 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.996 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.996 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:40:54.996 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.339185+0000", 2024-09-13T14:40:54.996 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1509999999999998e-06 2024-09-13T14:40:54.996 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.996 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.996 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:40:54.996 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.339232+0000", 2024-09-13T14:40:54.996 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6953000000000003e-05 2024-09-13T14:40:54.996 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.996 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.996 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:40:54.996 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.339241+0000", 2024-09-13T14:40:54.997 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0900000000000005e-06 2024-09-13T14:40:54.997 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.997 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.997 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:40:54.997 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.112527+0000", 2024-09-13T14:40:54.997 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.77328639700000001 2024-09-13T14:40:54.997 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.997 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.997 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:40:54.997 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.112538+0000", 2024-09-13T14:40:54.997 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1090999999999999e-05 2024-09-13T14:40:54.997 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.997 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.997 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:40:54.997 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.664075+0000", 2024-09-13T14:40:54.998 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5515367659999999 2024-09-13T14:40:54.998 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.998 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.998 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:40:54.998 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.664233+0000", 2024-09-13T14:40:54.998 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015868300000000001 2024-09-13T14:40:54.998 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.998 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.998 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:40:54.998 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.664384+0000", 2024-09-13T14:40:54.998 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000150371 2024-09-13T14:40:54.998 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.998 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.998 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:40:54.999 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.673923+0000", 2024-09-13T14:40:54.999 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0095388030000000002 2024-09-13T14:40:54.999 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.999 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.999 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:40:54.999 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.677432+0000", 2024-09-13T14:40:54.999 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0035092560000000001 2024-09-13T14:40:54.999 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.999 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:54.999 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:40:54.999 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.692151+0000", 2024-09-13T14:40:54.999 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014718716999999999 2024-09-13T14:40:54.999 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:54.999 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.000 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:40:55.000 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.692201+0000", 2024-09-13T14:40:55.000 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0145000000000002e-05 2024-09-13T14:40:55.000 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.000 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.000 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:40:55.000 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.695530+0000", 2024-09-13T14:40:55.000 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0033293260000000001 2024-09-13T14:40:55.000 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:40:55.000 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:40:55.000 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:40:55.000 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.000 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.001 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113982 7.4 7:256588d9:::benchmark_data_smithi102_227395_object1167:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-13T14:40:55.001 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:47.404500+0000", 2024-09-13T14:40:55.001 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 187.47581587900001, 2024-09-13T14:40:55.001 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.296665693, 2024-09-13T14:40:55.001 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:40:55.001 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:40:55.001 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:40:55.001 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:40:55.001 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:40:55.001 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113982 2024-09-13T14:40:55.001 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.002 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:40:55.002 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.002 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:40:55.002 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.404500+0000", 2024-09-13T14:40:55.002 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:40:55.002 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.002 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.002 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:40:55.002 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.404500+0000", 2024-09-13T14:40:55.002 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:40:55.002 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.002 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.002 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:40:55.002 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.404499+0000", 2024-09-13T14:40:55.003 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:40:55.003 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.003 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.003 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:40:55.003 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.404505+0000", 2024-09-13T14:40:55.003 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.438e-06 2024-09-13T14:40:55.003 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.003 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.003 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:40:55.003 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.404517+0000", 2024-09-13T14:40:55.003 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2130000000000001e-05 2024-09-13T14:40:55.003 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.003 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.003 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:40:55.004 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.404522+0000", 2024-09-13T14:40:55.004 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2449999999999998e-06 2024-09-13T14:40:55.004 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.004 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.004 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:40:55.004 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.404570+0000", 2024-09-13T14:40:55.004 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7508e-05 2024-09-13T14:40:55.004 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.004 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.004 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:40:55.004 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.404578+0000", 2024-09-13T14:40:55.004 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2519999999999999e-06 2024-09-13T14:40:55.004 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.005 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.005 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:40:55.005 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.112562+0000", 2024-09-13T14:40:55.005 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.707983323 2024-09-13T14:40:55.005 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.005 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.005 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:40:55.005 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.112569+0000", 2024-09-13T14:40:55.005 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9379999999999992e-06 2024-09-13T14:40:55.005 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.005 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.005 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:40:55.005 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.670326+0000", 2024-09-13T14:40:55.006 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.557756707 2024-09-13T14:40:55.006 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.006 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.006 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:40:55.006 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.670465+0000", 2024-09-13T14:40:55.006 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013909099999999999 2024-09-13T14:40:55.006 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.006 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.006 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:40:55.006 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.670607+0000", 2024-09-13T14:40:55.006 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014218999999999999 2024-09-13T14:40:55.006 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.006 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.006 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:40:55.007 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.677438+0000", 2024-09-13T14:40:55.007 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.006830839 2024-09-13T14:40:55.007 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.007 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.007 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:40:55.007 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.678258+0000", 2024-09-13T14:40:55.007 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00081936699999999999 2024-09-13T14:40:55.007 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.007 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.007 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:40:55.008 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.695584+0000", 2024-09-13T14:40:55.008 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017326105000000001 2024-09-13T14:40:55.008 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.008 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.008 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:40:55.008 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.695638+0000", 2024-09-13T14:40:55.008 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3949000000000001e-05 2024-09-13T14:40:55.008 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.008 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.008 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:40:55.009 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.701166+0000", 2024-09-13T14:40:55.009 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0055280310000000001 2024-09-13T14:40:55.009 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:40:55.009 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:40:55.009 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:40:55.009 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.009 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.009 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113988 7.9 7:9e6eb1fd:::benchmark_data_smithi102_227395_object1173:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-13T14:40:55.009 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:47.878570+0000", 2024-09-13T14:40:55.009 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 187.00174549400001, 2024-09-13T14:40:55.009 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.34139049300000002, 2024-09-13T14:40:55.009 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:40:55.010 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:40:55.010 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:40:55.010 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:40:55.010 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:40:55.010 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113988 2024-09-13T14:40:55.010 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.010 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:40:55.010 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.010 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:40:55.010 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.878570+0000", 2024-09-13T14:40:55.010 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:40:55.010 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.010 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.010 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:40:55.011 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.878570+0000", 2024-09-13T14:40:55.011 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:40:55.011 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.011 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.011 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:40:55.011 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.878569+0000", 2024-09-13T14:40:55.011 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:40:55.011 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.011 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.011 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:40:55.011 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.878578+0000", 2024-09-13T14:40:55.011 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.8339999999999994e-06 2024-09-13T14:40:55.011 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.011 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.011 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:40:55.012 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.878595+0000", 2024-09-13T14:40:55.012 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7007000000000002e-05 2024-09-13T14:40:55.012 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.012 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.012 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:40:55.012 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.878623+0000", 2024-09-13T14:40:55.012 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7189999999999999e-05 2024-09-13T14:40:55.012 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.012 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.012 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:40:55.012 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.878685+0000", 2024-09-13T14:40:55.012 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2661000000000002e-05 2024-09-13T14:40:55.012 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.012 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.013 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:40:55.013 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.878698+0000", 2024-09-13T14:40:55.013 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3139000000000001e-05 2024-09-13T14:40:55.013 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.013 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.013 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:40:55.013 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.213485+0000", 2024-09-13T14:40:55.013 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.33478628199999999 2024-09-13T14:40:55.013 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.013 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.013 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:40:55.013 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.213573+0000", 2024-09-13T14:40:55.013 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8212000000000002e-05 2024-09-13T14:40:55.014 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.014 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.014 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,6,7", 2024-09-13T14:40:55.014 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.213671+0000", 2024-09-13T14:40:55.014 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.8344999999999994e-05 2024-09-13T14:40:55.014 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.014 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.014 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:40:55.014 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.216736+0000", 2024-09-13T14:40:55.014 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0030646760000000001 2024-09-13T14:40:55.014 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.014 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.014 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:40:55.014 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.216745+0000", 2024-09-13T14:40:55.015 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4220000000000006e-06 2024-09-13T14:40:55.015 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.015 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.015 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:40:55.015 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.216773+0000", 2024-09-13T14:40:55.015 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7970000000000002e-05 2024-09-13T14:40:55.015 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.015 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.015 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:40:55.015 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.219756+0000", 2024-09-13T14:40:55.015 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0029822120000000001 2024-09-13T14:40:55.015 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.015 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.015 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:40:55.016 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.219817+0000", 2024-09-13T14:40:55.016 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1235000000000006e-05 2024-09-13T14:40:55.016 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.016 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.016 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:40:55.016 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.219961+0000", 2024-09-13T14:40:55.016 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014409899999999999 2024-09-13T14:40:55.016 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:40:55.016 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:40:55.016 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:40:55.016 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.016 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.016 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113993 7.4 7:2f29f4ff:::benchmark_data_smithi102_227395_object1178:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-13T14:40:55.017 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:48.328273+0000", 2024-09-13T14:40:55.017 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 186.55204280999999, 2024-09-13T14:40:55.017 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3770136559999999, 2024-09-13T14:40:55.017 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:40:55.017 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:40:55.017 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:40:55.017 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:40:55.017 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:40:55.017 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113993 2024-09-13T14:40:55.017 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.017 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:40:55.017 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.017 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:40:55.017 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.328273+0000", 2024-09-13T14:40:55.017 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:40:55.018 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.018 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.018 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:40:55.018 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.328273+0000", 2024-09-13T14:40:55.018 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:40:55.018 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.018 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.018 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:40:55.018 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.328271+0000", 2024-09-13T14:40:55.018 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:40:55.018 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.018 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.018 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:40:55.019 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.328279+0000", 2024-09-13T14:40:55.019 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0609999999999996e-06 2024-09-13T14:40:55.019 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.019 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.019 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:40:55.019 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.328294+0000", 2024-09-13T14:40:55.019 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4405e-05 2024-09-13T14:40:55.019 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.019 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.019 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:40:55.019 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.328301+0000", 2024-09-13T14:40:55.019 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6750000000000002e-06 2024-09-13T14:40:55.019 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.019 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.020 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:40:55.020 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.328376+0000", 2024-09-13T14:40:55.020 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4040000000000003e-05 2024-09-13T14:40:55.020 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.020 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.020 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:40:55.020 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.328389+0000", 2024-09-13T14:40:55.020 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3478e-05 2024-09-13T14:40:55.020 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.020 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.020 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:40:55.020 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.673990+0000", 2024-09-13T14:40:55.020 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.345601324 2024-09-13T14:40:55.021 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.021 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.021 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:40:55.021 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.674152+0000", 2024-09-13T14:40:55.021 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016154299999999999 2024-09-13T14:40:55.021 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.021 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.021 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:40:55.021 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.674339+0000", 2024-09-13T14:40:55.021 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00018710999999999999 2024-09-13T14:40:55.021 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.021 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.021 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:40:55.022 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.678268+0000", 2024-09-13T14:40:55.022 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0039291450000000002 2024-09-13T14:40:55.022 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.022 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.022 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:40:55.022 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.678582+0000", 2024-09-13T14:40:55.022 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00031387 2024-09-13T14:40:55.022 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.022 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.022 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:40:55.022 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.701236+0000", 2024-09-13T14:40:55.022 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.022653889 2024-09-13T14:40:55.022 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.022 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.022 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:40:55.023 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.701392+0000", 2024-09-13T14:40:55.023 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015579999999999999 2024-09-13T14:40:55.023 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.023 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.023 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:40:55.023 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.705287+0000", 2024-09-13T14:40:55.023 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003895098 2024-09-13T14:40:55.023 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:40:55.023 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:40:55.023 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:40:55.023 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.023 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.023 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113998 7.1 7:8cc51271:::benchmark_data_smithi102_227395_object1183:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-13T14:40:55.024 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:48.829978+0000", 2024-09-13T14:40:55.024 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 186.050337781, 2024-09-13T14:40:55.024 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.86212481600000002, 2024-09-13T14:40:55.024 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:40:55.024 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:40:55.024 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:40:55.024 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:40:55.024 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:40:55.024 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113998 2024-09-13T14:40:55.024 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.024 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:40:55.024 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.024 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:40:55.024 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.829978+0000", 2024-09-13T14:40:55.025 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:40:55.025 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.025 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.025 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:40:55.025 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.829978+0000", 2024-09-13T14:40:55.025 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:40:55.025 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.025 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.025 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:40:55.025 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.829977+0000", 2024-09-13T14:40:55.025 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:40:55.025 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.025 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.025 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:40:55.025 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.829984+0000", 2024-09-13T14:40:55.026 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7479999999999996e-06 2024-09-13T14:40:55.026 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.026 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.026 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:40:55.026 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.829999+0000", 2024-09-13T14:40:55.026 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5233e-05 2024-09-13T14:40:55.026 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.026 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.026 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:40:55.026 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.830004+0000", 2024-09-13T14:40:55.026 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4519999999999996e-06 2024-09-13T14:40:55.026 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.026 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.026 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:40:55.027 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.830059+0000", 2024-09-13T14:40:55.027 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4829e-05 2024-09-13T14:40:55.027 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.027 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.027 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:40:55.027 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.830072+0000", 2024-09-13T14:40:55.027 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3410999999999999e-05 2024-09-13T14:40:55.027 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.027 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.027 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:40:55.027 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.675677+0000", 2024-09-13T14:40:55.027 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.84560498200000001 2024-09-13T14:40:55.027 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.027 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.028 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:40:55.028 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.675810+0000", 2024-09-13T14:40:55.028 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000132786 2024-09-13T14:40:55.028 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.028 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.028 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:40:55.028 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.675943+0000", 2024-09-13T14:40:55.028 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000132898 2024-09-13T14:40:55.028 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.028 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.028 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:40:55.028 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.681639+0000", 2024-09-13T14:40:55.028 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0056959269999999999 2024-09-13T14:40:55.028 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.029 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.029 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:40:55.029 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.681746+0000", 2024-09-13T14:40:55.029 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010723299999999999 2024-09-13T14:40:55.029 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.029 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.029 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:40:55.029 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.689715+0000", 2024-09-13T14:40:55.029 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0079683510000000003 2024-09-13T14:40:55.029 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.029 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.029 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:40:55.029 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.689761+0000", 2024-09-13T14:40:55.029 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5936999999999997e-05 2024-09-13T14:40:55.029 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:40:55.030 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:40:55.030 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:40:55.030 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.692103+0000", 2024-09-13T14:40:55.030 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002342392 2024-09-13T14:40:55.030 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:40:55.030 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:40:55.030 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:40:55.030 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:40:55.030 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:40:55.030 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:40:55.031 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:40:55.031 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:40:55.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:55.032 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:55.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:55.133 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:55.155 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:55.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:55.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:55.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:55 smithi102 ceph-mon[110183]: pgmap v885: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 5.2 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 206 op/s; 30149/253221 objects degraded (11.906%); 5724/253221 objects misplaced (2.260%); 3.4 MiB/s, 53 objects/s recovering 2024-09-13T14:40:55.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:55 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/557132945' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:40:55.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:55 smithi102 ceph-mon[115269]: pgmap v885: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 5.2 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 206 op/s; 30149/253221 objects degraded (11.906%); 5724/253221 objects misplaced (2.260%); 3.4 MiB/s, 53 objects/s recovering 2024-09-13T14:40:55.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:55 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/557132945' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:40:55.257 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:55.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:55.280 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:55.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:55.382 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:55.405 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:55.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:55.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:55.482 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:55 smithi195 ceph-mon[104262]: pgmap v885: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 5.2 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 206 op/s; 30149/253221 objects degraded (11.906%); 5724/253221 objects misplaced (2.260%); 3.4 MiB/s, 53 objects/s recovering 2024-09-13T14:40:55.482 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:55 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/557132945' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:40:55.507 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:55.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:55.530 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:55.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:55.631 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:55.652 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:55.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:55.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:55.754 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:55.775 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:55.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:55.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:55.877 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:55.901 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:55.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:55.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:55.924 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T14:40:56.003 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:56.029 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:56.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:56.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:56.130 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:56.153 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:56.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:56.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:56.154 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:40:56.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:40:56.255 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:56.278 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:56.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:56.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:56.379 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:56.398 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:40:56.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:40:56.406 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:56.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:56.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:56.508 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:56.542 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:56.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:56.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:56.644 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:40:56.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:40:56 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:40:56] "GET /metrics HTTP/1.1" 200 36536 "" "Prometheus/2.51.0" 2024-09-13T14:40:56.669 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:56.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:56.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:40:56.772 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:56.799 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:56.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:56.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:56.901 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:56.924 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:56.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:56.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:57.026 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:57.051 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:57.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:57.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:57.089 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:57 smithi102 ceph-mon[110183]: pgmap v886: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 5.1 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 210 op/s; 30063/251943 objects degraded (11.932%); 5723/251943 objects misplaced (2.272%); 3.5 MiB/s, 56 objects/s recovering 2024-09-13T14:40:57.089 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:57 smithi102 ceph-mon[115269]: pgmap v886: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 5.1 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 210 op/s; 30063/251943 objects degraded (11.932%); 5723/251943 objects misplaced (2.272%); 3.5 MiB/s, 56 objects/s recovering 2024-09-13T14:40:57.154 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:57.176 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:57.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:57.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:57.278 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:57.303 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:57.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:57.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:57.405 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:57.426 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:57.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:57.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:57.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:57 smithi195 ceph-mon[104262]: pgmap v886: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 5.1 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 210 op/s; 30063/251943 objects degraded (11.932%); 5723/251943 objects misplaced (2.272%); 3.5 MiB/s, 56 objects/s recovering 2024-09-13T14:40:57.527 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:57.551 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:57.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:57.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:57.653 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:57.677 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:57.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:57.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:57.745 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:40:57.779 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:57.818 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:57.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:57.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:57.920 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:40:57.945 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:57.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:57.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:40:58.046 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:58.074 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:58.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:58.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:58.177 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:58.201 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:58.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:58.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:58.302 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:40:58.326 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:58.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:58.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:40:58.428 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:40:58.450 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:58.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:58.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:40:58.552 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:58.586 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:58.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:58.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:58.687 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:58.717 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:58.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:58.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:58.819 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:58.844 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:58.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:58.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:58.946 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:58.973 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:40:58.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:58.981 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:58.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:59.028 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:40:59.028 INFO:teuthology.orchestra.run.smithi102.stdout:{"fsid":"08cee502-71d9-11ef-bceb-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29934/250377 objects degraded (11.956%), 10 pgs degraded, 6 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1282,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":268,"num_osds":8,"num_up_osds":8,"osd_up_since":1726238267,"num_in_osds":6,"osd_in_since":1726238173,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":78},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovery_wait+degraded","count":3},{"state_name":"active+recovery_wait+degraded+remapped","count":2},{"state_name":"active+undersized+degraded+remapped+backfill_wait","count":1},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":89,"num_pools":5,"num_objects":83459,"data_bytes":5472458749,"bytes_used":20720820224,"bytes_avail":555249393664,"bytes_total":575970213888,"degraded_objects":29934,"degraded_total":250377,"degraded_ratio":0.11955570998933608,"misplaced_objects":5502,"misplaced_total":250377,"misplaced_ratio":0.021974861908242372,"recovering_objects_per_sec":61,"recovering_bytes_per_sec":4029410,"recovering_keys_per_sec":0,"num_objects_recovered":615,"num_bytes_recovered":40304640,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":205},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi102.njetxr","status":"up:active","gid":24521}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.102:8443/","prometheus":"http://172.21.15.102:9283/"}},"servicemap":{"epoch":28,"modified":"2024-09-13T14:40:21.476600+0000","services":{}},"progress_events":{}} 2024-09-13T14:40:59.082 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:59.102 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:59 smithi102 ceph-mon[115269]: pgmap v887: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 5.1 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 205 op/s; 29934/250377 objects degraded (11.956%); 5502/250377 objects misplaced (2.197%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:40:59.102 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:59 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 29934/250377 objects degraded (11.956%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:59.102 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:40:59 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3718293830' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:40:59.102 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:59 smithi102 ceph-mon[110183]: pgmap v887: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 5.1 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 205 op/s; 29934/250377 objects degraded (11.956%); 5502/250377 objects misplaced (2.197%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:40:59.102 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:59 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 29934/250377 objects degraded (11.956%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:59.102 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:40:59 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3718293830' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:40:59.110 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:59.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:59.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:59.212 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:59.237 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:59.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:59.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:59.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:59 smithi195 ceph-mon[104262]: pgmap v887: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 5.1 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 205 op/s; 29934/250377 objects degraded (11.956%); 5502/250377 objects misplaced (2.197%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:40:59.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:59 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 29934/250377 objects degraded (11.956%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:40:59.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:40:59 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3718293830' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:40:59.338 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:59.362 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:59.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:59.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:59.464 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:40:59.488 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:59.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:59.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:40:59.590 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:40:59.615 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:59.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:59.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:40:59.716 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:40:59.738 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:40:59.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:59.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:40:59.840 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:59.865 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:59.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:59.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:40:59.967 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:40:59.991 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:40:59.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:40:59.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:00.058 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is unset 2024-09-13T14:41:00.093 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:00.117 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:00.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:00.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:00.219 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:00.244 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:00.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:00.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:00.303 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:00 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3777009341' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:41:00.303 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:00 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:41:00.303 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:00 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3777009341' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:41:00.303 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:00 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:41:00.347 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:00.361 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:00 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3777009341' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:41:00.361 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:00 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:41:00.372 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:00.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:00.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:00.474 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:00.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:00.501 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:00.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:00.602 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:00.627 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:00.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:00.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:00.663 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-13T14:41:00.663 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:41:00.729 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:00.755 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:00.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:00.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:00.857 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:00.882 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:00.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:00.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:00.984 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:01.010 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:01.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:01.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:01.112 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:01.138 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:01.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:01.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:01.239 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:01.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:01 smithi195 ceph-mon[104262]: pgmap v888: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 5.1 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 195 op/s; 29877/249063 objects degraded (11.996%); 5502/249063 objects misplaced (2.209%); 3.6 MiB/s, 58 objects/s recovering 2024-09-13T14:41:01.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:01 smithi195 ceph-mon[104262]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:41:01.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:01 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:41:01.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:01 smithi195 ceph-mon[104262]: osdmap e269: 8 total, 8 up, 6 in 2024-09-13T14:41:01.263 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:01.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:01.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:01.365 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:01.387 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:01.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:01.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:01.414 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:01 smithi102 ceph-mon[110183]: pgmap v888: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 5.1 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 195 op/s; 29877/249063 objects degraded (11.996%); 5502/249063 objects misplaced (2.209%); 3.6 MiB/s, 58 objects/s recovering 2024-09-13T14:41:01.414 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:01 smithi102 ceph-mon[110183]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:41:01.414 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:01 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:41:01.414 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:01 smithi102 ceph-mon[110183]: osdmap e269: 8 total, 8 up, 6 in 2024-09-13T14:41:01.415 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:01 smithi102 ceph-mon[115269]: pgmap v888: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 5.1 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 195 op/s; 29877/249063 objects degraded (11.996%); 5502/249063 objects misplaced (2.209%); 3.6 MiB/s, 58 objects/s recovering 2024-09-13T14:41:01.415 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:01 smithi102 ceph-mon[115269]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:41:01.415 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:01 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:41:01.415 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:01 smithi102 ceph-mon[115269]: osdmap e269: 8 total, 8 up, 6 in 2024-09-13T14:41:01.489 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:01.514 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:01.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:01.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:01.616 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:01.647 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:01.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:01.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:01.735 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:41:01.735 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:41:01.750 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:01.752 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:41:01.752 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:41:01.774 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:41:01.774 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:41:01.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:01.775 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:01.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:01.794 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:41:01.794 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:41:01.812 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:41:01.813 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:41:01.830 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:41:01.830 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:41:01.848 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:41:01.849 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:41:01.874 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:41:01.875 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:41:01.876 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:01.903 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:01.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:01.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:01.938 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:41:01.938 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-13T14:41:01.938 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.938 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:143491 7.e 7:724aa6b6:::benchmark_data_smithi102_227395_object30676:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-09-13T14:41:01.938 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:41:01.858190+0000", 2024-09-13T14:41:01.938 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.077521196000000001, 2024-09-13T14:41:01.938 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.077544169999999996, 2024-09-13T14:41:01.938 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:41:01.939 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "waiting for missing object", 2024-09-13T14:41:01.939 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:41:01.939 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:41:01.939 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:41:01.939 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 143491 2024-09-13T14:41:01.939 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.939 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:41:01.939 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.939 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:41:01.939 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.858190+0000", 2024-09-13T14:41:01.939 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:01.939 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.939 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.940 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:41:01.940 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.858190+0000", 2024-09-13T14:41:01.940 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:01.940 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.940 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.940 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:41:01.940 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.858189+0000", 2024-09-13T14:41:01.940 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:41:01.940 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.940 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.940 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:41:01.940 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.858196+0000", 2024-09-13T14:41:01.941 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8499999999999994e-06 2024-09-13T14:41:01.941 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.941 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.941 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:41:01.941 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.858211+0000", 2024-09-13T14:41:01.941 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4524e-05 2024-09-13T14:41:01.941 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.941 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.941 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:41:01.941 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.858219+0000", 2024-09-13T14:41:01.941 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7910000000000005e-06 2024-09-13T14:41:01.941 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.941 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.941 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:01.942 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.921522+0000", 2024-09-13T14:41:01.942 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.063302997999999999 2024-09-13T14:41:01.942 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.942 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.942 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:41:01.942 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.926052+0000", 2024-09-13T14:41:01.942 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0045306629999999999 2024-09-13T14:41:01.942 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:01.942 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:41:01.942 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:01.942 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.942 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.942 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35379.0:143493 7.6 e269/254 7:64067919:::benchmark_data_smithi102_227395_object30678:head v 269'9071, mlcod=0'0)", 2024-09-13T14:41:01.943 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:41:01.924456+0000", 2024-09-13T14:41:01.943 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.01125571, 2024-09-13T14:41:01.943 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011323448999999999, 2024-09-13T14:41:01.943 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:41:01.943 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T14:41:01.943 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:41:01.943 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.943 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:41:01.943 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.924456+0000", 2024-09-13T14:41:01.943 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:01.943 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.943 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.943 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:41:01.944 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.924456+0000", 2024-09-13T14:41:01.944 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:01.944 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.944 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.944 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:41:01.944 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.924455+0000", 2024-09-13T14:41:01.944 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:41:01.944 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.944 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.944 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:41:01.944 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.924463+0000", 2024-09-13T14:41:01.944 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6029999999999999e-06 2024-09-13T14:41:01.944 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.945 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.945 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:41:01.945 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.924474+0000", 2024-09-13T14:41:01.945 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1017e-05 2024-09-13T14:41:01.945 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.945 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.945 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:41:01.945 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.924480+0000", 2024-09-13T14:41:01.945 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3409999999999999e-06 2024-09-13T14:41:01.945 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.945 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.945 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:01.945 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.926087+0000", 2024-09-13T14:41:01.946 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0016065840000000001 2024-09-13T14:41:01.946 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.946 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.946 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:41:01.946 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.926115+0000", 2024-09-13T14:41:01.946 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8296000000000001e-05 2024-09-13T14:41:01.946 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:01.946 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:41:01.946 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:01.946 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.946 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.946 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:143502 7.0 7:04ec3e53:::benchmark_data_smithi102_227395_object30687:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-09-13T14:41:01.946 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:41:01.926693+0000", 2024-09-13T14:41:01.946 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.0090190359999999994, 2024-09-13T14:41:01.947 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0091198990000000008, 2024-09-13T14:41:01.947 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:41:01.947 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:01.947 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:41:01.947 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:41:01.947 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:41:01.947 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 143502 2024-09-13T14:41:01.947 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.947 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:41:01.947 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.947 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:41:01.947 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.926693+0000", 2024-09-13T14:41:01.947 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:01.947 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.947 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.948 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:41:01.948 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.926693+0000", 2024-09-13T14:41:01.948 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:01.948 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.948 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.948 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:41:01.948 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.926690+0000", 2024-09-13T14:41:01.948 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:41:01.948 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.948 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.948 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:41:01.948 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.926700+0000", 2024-09-13T14:41:01.948 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.3670000000000003e-06 2024-09-13T14:41:01.949 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.949 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.949 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:41:01.949 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.926718+0000", 2024-09-13T14:41:01.949 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7683e-05 2024-09-13T14:41:01.949 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.949 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.949 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:41:01.949 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.926729+0000", 2024-09-13T14:41:01.949 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1094000000000001e-05 2024-09-13T14:41:01.949 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.949 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.949 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:01.949 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.926787+0000", 2024-09-13T14:41:01.950 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8208999999999998e-05 2024-09-13T14:41:01.950 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.950 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.950 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:41:01.950 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.926961+0000", 2024-09-13T14:41:01.950 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000173983 2024-09-13T14:41:01.950 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.950 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.950 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:41:01.950 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.927124+0000", 2024-09-13T14:41:01.950 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000162898 2024-09-13T14:41:01.950 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.950 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.951 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:41:01.951 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.929541+0000", 2024-09-13T14:41:01.951 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0024175199999999998 2024-09-13T14:41:01.951 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.951 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.951 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:01.951 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.929842+0000", 2024-09-13T14:41:01.951 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00030119499999999998 2024-09-13T14:41:01.951 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.951 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.951 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:01.951 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.934004+0000", 2024-09-13T14:41:01.951 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0041613800000000001 2024-09-13T14:41:01.951 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.952 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.952 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:41:01.952 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.934042+0000", 2024-09-13T14:41:01.952 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8018000000000001e-05 2024-09-13T14:41:01.952 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:01.952 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:41:01.952 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:01.952 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.952 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.952 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDPGPush(7.e 269/245 [PushOp(7:724aa6b6:::benchmark_data_smithi102_227395_object30676:head, version: 210'1879, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:724aa6b6:::benchmark_data_smithi102_227395_object30676:head@210'1879, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T14:41:01.952 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:41:01.927239+0000", 2024-09-13T14:41:01.952 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.0084720750000000008, 2024-09-13T14:41:01.952 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0086356550000000008, 2024-09-13T14:41:01.953 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:41:01.953 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T14:41:01.953 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:41:01.953 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.953 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:41:01.953 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.927239+0000", 2024-09-13T14:41:01.953 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:01.953 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.953 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.953 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:41:01.953 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.927239+0000", 2024-09-13T14:41:01.953 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:01.953 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.954 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.954 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:41:01.954 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.927239+0000", 2024-09-13T14:41:01.954 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:41:01.954 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.954 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.954 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:41:01.954 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.927314+0000", 2024-09-13T14:41:01.954 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5257000000000001e-05 2024-09-13T14:41:01.954 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.954 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.954 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:41:01.954 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.927340+0000", 2024-09-13T14:41:01.955 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5931999999999998e-05 2024-09-13T14:41:01.955 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.955 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.955 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:41:01.955 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.927402+0000", 2024-09-13T14:41:01.955 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2107999999999996e-05 2024-09-13T14:41:01.955 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.955 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.955 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:01.955 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.928159+0000", 2024-09-13T14:41:01.955 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00075739100000000005 2024-09-13T14:41:01.955 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.955 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.955 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:41:01.956 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.928663+0000", 2024-09-13T14:41:01.956 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00050368199999999998 2024-09-13T14:41:01.956 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:01.956 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:41:01.956 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:01.956 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.956 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.956 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop_reply(client.35379.0:143502 7.0 e269/240 ondisk, result = 0)", 2024-09-13T14:41:01.956 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:41:01.933896+0000", 2024-09-13T14:41:01.956 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.0018158849999999999, 2024-09-13T14:41:01.956 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0020083869999999999, 2024-09-13T14:41:01.956 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:41:01.956 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T14:41:01.957 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:41:01.957 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.957 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:41:01.957 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.933896+0000", 2024-09-13T14:41:01.957 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:01.957 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.957 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.957 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:41:01.957 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.933896+0000", 2024-09-13T14:41:01.957 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:01.957 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.957 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.957 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:41:01.958 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.933894+0000", 2024-09-13T14:41:01.958 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:41:01.958 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.958 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.958 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:41:01.958 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.933904+0000", 2024-09-13T14:41:01.958 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0122000000000001e-05 2024-09-13T14:41:01.958 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.958 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.958 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:41:01.958 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.933932+0000", 2024-09-13T14:41:01.958 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7460000000000001e-05 2024-09-13T14:41:01.958 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.959 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.959 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:41:01.959 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.933942+0000", 2024-09-13T14:41:01.959 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0030000000000001e-05 2024-09-13T14:41:01.959 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.959 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.959 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:01.959 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.933991+0000", 2024-09-13T14:41:01.959 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8974000000000002e-05 2024-09-13T14:41:01.959 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.959 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.959 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:41:01.959 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.933999+0000", 2024-09-13T14:41:01.960 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0790000000000001e-06 2024-09-13T14:41:01.960 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:01.960 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:41:01.960 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:01.960 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.960 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.960 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDPGPush(7.6 269/254 [PushOp(7:6b522d8b:::benchmark_data_smithi102_227395_object30688:head, version: 210'1974, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:6b522d8b:::benchmark_data_smithi102_227395_object30688:head@210'1974, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T14:41:01.960 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:41:01.935004+0000", 2024-09-13T14:41:01.960 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.000707444, 2024-09-13T14:41:01.960 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00095775999999999999, 2024-09-13T14:41:01.960 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:41:01.960 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "queued for pg", 2024-09-13T14:41:01.960 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:41:01.961 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.961 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:41:01.961 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.935004+0000", 2024-09-13T14:41:01.961 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:01.961 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.961 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.961 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:41:01.961 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.935004+0000", 2024-09-13T14:41:01.961 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:01.961 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.961 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.961 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:41:01.962 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.935003+0000", 2024-09-13T14:41:01.962 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:41:01.962 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.962 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.962 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:41:01.962 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.935051+0000", 2024-09-13T14:41:01.962 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.808e-05 2024-09-13T14:41:01.962 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.962 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.962 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:41:01.962 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.935567+0000", 2024-09-13T14:41:01.962 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00051524199999999996 2024-09-13T14:41:01.962 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:01.963 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:01.963 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:41:01.963 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:01.935584+0000", 2024-09-13T14:41:01.963 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7034e-05 2024-09-13T14:41:01.963 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:01.963 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:41:01.963 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:01.963 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:01.963 INFO:teuthology.orchestra.run.smithi195.stdout: ], 2024-09-13T14:41:01.963 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 6 2024-09-13T14:41:01.963 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:41:02.005 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:02.028 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:02.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:02.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:02.130 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:02.156 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:02.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:02.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:02.258 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:02.285 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:02.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:02.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:02.387 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:02.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:02.412 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:02.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:02.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:02 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3695232646' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:41:02.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:02 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3695232646' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:41:02.433 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:02 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3695232646' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:41:02.513 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:02.535 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:02.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:02.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:02.637 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:02.658 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:02.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:02.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:02.730 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T14:41:02.760 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:02.782 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:02.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:02.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:02.884 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:02.906 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:02.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:02.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:02.972 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:41:03.008 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:03.033 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:03.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:03.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:03.134 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:03.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:03.161 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:03.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:03.261 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:03.287 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:03.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:03.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:03.389 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:03.412 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:03.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:03.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:03.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:03 smithi102 ceph-mon[110183]: pgmap v890: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 5.1 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 205 op/s; 29877/249063 objects degraded (11.996%); 5502/249063 objects misplaced (2.209%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:41:03.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:03 smithi102 ceph-mon[115269]: pgmap v890: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 5.1 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 205 op/s; 29877/249063 objects degraded (11.996%); 5502/249063 objects misplaced (2.209%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:41:03.448 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:03 smithi195 ceph-mon[104262]: pgmap v890: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 5.1 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 205 op/s; 29877/249063 objects degraded (11.996%); 5502/249063 objects misplaced (2.209%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:41:03.515 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:03.539 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:03.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:03.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:03.641 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:03.665 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:03.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:03.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:03.767 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:03.791 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:03.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:03.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:03.893 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:03.920 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:03.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:03.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:04.022 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:04.044 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:04.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:04.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:04.146 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:04.170 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:04.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:04.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:04.217 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:04 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 29877/249063 objects degraded (11.996%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:41:04.257 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:04 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 29877/249063 objects degraded (11.996%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:41:04.257 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:04 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 29877/249063 objects degraded (11.996%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:41:04.272 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:04.297 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:04.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:04.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:04.308 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:04.399 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:04.422 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:04.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:04.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:04.524 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:04.549 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:04.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:04.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:04.651 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:04.674 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:04.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:04.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:04.776 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:04.801 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:04.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:04.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:04.852 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:41:04.852 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-13T14:41:04.852 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-09-13T14:41:04.852 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-09-13T14:41:04.852 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:41:04.903 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:04.924 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:04.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:04.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:05.026 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:05.047 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:05.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:05.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:05.149 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:05.170 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:05.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:05.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:05.172 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:41:05.227 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:05 smithi195 ceph-mon[104262]: pgmap v891: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 5.0 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 203 op/s; 29752/247122 objects degraded (12.039%); 5281/247122 objects misplaced (2.137%); 4.5 MiB/s, 71 objects/s recovering 2024-09-13T14:41:05.272 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:05.295 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:05.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:05.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:05.397 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:05.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:05 smithi102 ceph-mon[115269]: pgmap v891: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 5.0 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 203 op/s; 29752/247122 objects degraded (12.039%); 5281/247122 objects misplaced (2.137%); 4.5 MiB/s, 71 objects/s recovering 2024-09-13T14:41:05.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:05 smithi102 ceph-mon[110183]: pgmap v891: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 5.0 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 203 op/s; 29752/247122 objects degraded (12.039%); 5281/247122 objects misplaced (2.137%); 4.5 MiB/s, 71 objects/s recovering 2024-09-13T14:41:05.424 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:05.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:05.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:05.526 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:05.550 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:05.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:05.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:05.652 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:05.660 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T14:41:05.682 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:05.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:05.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:05.784 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:05.813 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:05.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:05.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:05.915 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:05.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:05.940 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:05.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:05.950 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:41:06.041 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:06.065 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:06.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:06.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:06.128 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:06.166 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:06.188 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:06.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:06.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:06.282 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:41:06.289 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:06.312 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:06.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:06.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:06.393 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:41:06.394 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:41:06.413 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:06.435 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:06.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:06.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:06.537 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:06.563 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:06.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:06.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:06.666 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:06.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:41:06 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:41:06] "GET /metrics HTTP/1.1" 200 36536 "" "Prometheus/2.51.0" 2024-09-13T14:41:06.693 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:06.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:06.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:06.796 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:06.818 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:06.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:06.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:06.920 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:06.941 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:06.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:06.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:07.043 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:07.072 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:07.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:07.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:07.174 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:07.198 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:07.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:07.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:07.226 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":892,"stamp":"2024-09-13T14:41:05.489515+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5374154749,"num_objects":81959,"num_object_clones":0,"num_object_copies":245877,"num_objects_missing_on_primary":3101,"num_objects_missing":3101,"num_objects_degraded":29697,"num_objects_misplaced":5281,"num_objects_unfound":0,"num_objects_dirty":81959,"num_whiteouts":0,"num_read":6377,"num_read_kb":13834,"num_write":144818,"num_write_kb":7235010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60102,"num_bytes_recovered":3902016653,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94000,"ondisk_log_size":94000,"up":267,"acting":261,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":309,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":19999804,"kb_used_data":16944704,"kb_used_omap":26,"kb_used_meta":3055013,"kb_avail":542471108,"statfs":{"total":575970213888,"available":555490414592,"internally_reserved":0,"allocated":17351376896,"data_stored":17342588541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27144,"internal_metadata":3128333816},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-132513792,"num_objects":-2022,"num_object_clones":0,"num_object_copies":-6066,"num_objects_missing_on_primary":-135,"num_objects_missing":-135,"num_objects_degraded":-366,"num_objects_misplaced":-442,"num_objects_unfound":0,"num_objects_dirty":-2022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2022,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":682,"num_bytes_recovered":44695552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002602"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":430,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.056575+0000","last_change":"2024-09-13T14:36:13.859927+0000","last_active":"2024-09-13T14:41:00.056575+0000","last_peered":"2024-09-13T14:41:00.056575+0000","last_clean":"2024-09-13T14:41:00.056575+0000","last_became_active":"2024-09-13T14:36:13.859505+0000","last_became_peered":"2024-09-13T14:36:13.859505+0000","last_unstale":"2024-09-13T14:41:00.056575+0000","last_undegraded":"2024-09-13T14:41:00.056575+0000","last_fullsized":"2024-09-13T14:41:00.056575+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:12:20.989627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":270,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:40:49.019596+0000","last_change":"2024-09-13T14:37:48.130968+0000","last_active":"2024-09-13T14:40:49.019596+0000","last_peered":"2024-09-13T14:40:49.019596+0000","last_clean":"2024-09-13T14:40:49.019596+0000","last_became_active":"2024-09-13T14:37:48.130603+0000","last_became_peered":"2024-09-13T14:37:48.130603+0000","last_unstale":"2024-09-13T14:40:49.019596+0000","last_undegraded":"2024-09-13T14:40:49.019596+0000","last_fullsized":"2024-09-13T14:40:49.019596+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:42.626386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":67,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.072994+0000","last_change":"2024-09-13T14:36:13.862181+0000","last_active":"2024-09-13T14:41:00.072994+0000","last_peered":"2024-09-13T14:41:00.072994+0000","last_clean":"2024-09-13T14:41:00.072994+0000","last_became_active":"2024-09-13T14:36:13.861873+0000","last_became_peered":"2024-09-13T14:36:13.861873+0000","last_unstale":"2024-09-13T14:41:00.072994+0000","last_undegraded":"2024-09-13T14:41:00.072994+0000","last_fullsized":"2024-09-13T14:41:00.072994+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:14.085700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":68,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.067078+0000","last_change":"2024-09-13T14:38:53.896672+0000","last_active":"2024-09-13T14:41:00.067078+0000","last_peered":"2024-09-13T14:41:00.067078+0000","last_clean":"2024-09-13T14:41:00.067078+0000","last_became_active":"2024-09-13T14:38:53.896346+0000","last_became_peered":"2024-09-13T14:38:53.896346+0000","last_unstale":"2024-09-13T14:41:00.067078+0000","last_undegraded":"2024-09-13T14:41:00.067078+0000","last_fullsized":"2024-09-13T14:41:00.067078+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:57.789686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":68,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.067484+0000","last_change":"2024-09-13T14:38:53.893021+0000","last_active":"2024-09-13T14:41:00.067484+0000","last_peered":"2024-09-13T14:41:00.067484+0000","last_clean":"2024-09-13T14:41:00.067484+0000","last_became_active":"2024-09-13T14:38:53.892668+0000","last_became_peered":"2024-09-13T14:38:53.892668+0000","last_unstale":"2024-09-13T14:41:00.067484+0000","last_undegraded":"2024-09-13T14:41:00.067484+0000","last_fullsized":"2024-09-13T14:41:00.067484+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:57.180738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":238,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:40:49.019372+0000","last_change":"2024-09-13T14:37:48.134642+0000","last_active":"2024-09-13T14:40:49.019372+0000","last_peered":"2024-09-13T14:40:49.019372+0000","last_clean":"2024-09-13T14:40:49.019372+0000","last_became_active":"2024-09-13T14:37:48.134402+0000","last_became_peered":"2024-09-13T14:37:48.134402+0000","last_unstale":"2024-09-13T14:40:49.019372+0000","last_undegraded":"2024-09-13T14:40:49.019372+0000","last_fullsized":"2024-09-13T14:40:49.019372+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:51.313663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":449,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.057158+0000","last_change":"2024-09-13T14:38:53.903022+0000","last_active":"2024-09-13T14:41:00.057158+0000","last_peered":"2024-09-13T14:41:00.057158+0000","last_clean":"2024-09-13T14:41:00.057158+0000","last_became_active":"2024-09-13T14:38:53.902448+0000","last_became_peered":"2024-09-13T14:38:53.902448+0000","last_unstale":"2024-09-13T14:41:00.057158+0000","last_undegraded":"2024-09-13T14:41:00.057158+0000","last_fullsized":"2024-09-13T14:41:00.057158+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:24.690401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":71,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.066620+0000","last_change":"2024-09-13T14:38:03.554500+0000","last_active":"2024-09-13T14:41:00.066620+0000","last_peered":"2024-09-13T14:41:00.066620+0000","last_clean":"2024-09-13T14:41:00.066620+0000","last_became_active":"2024-09-13T14:38:03.532910+0000","last_became_peered":"2024-09-13T14:38:03.532910+0000","last_unstale":"2024-09-13T14:41:00.066620+0000","last_undegraded":"2024-09-13T14:41:00.066620+0000","last_fullsized":"2024-09-13T14:41:00.066620+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:27:50.843498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"269'8875","reported_seq":13002,"reported_epoch":269,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-13T14:41:01.828818+0000","last_change":"2024-09-13T14:40:32.799848+0000","last_active":"2024-09-13T14:41:01.828818+0000","last_peered":"2024-09-13T14:41:01.828818+0000","last_clean":"2024-09-13T14:37:46.417003+0000","last_became_active":"2024-09-13T14:37:49.690961+0000","last_became_peered":"2024-09-13T14:37:49.690961+0000","last_unstale":"2024-09-13T14:41:01.828818+0000","last_undegraded":"2024-09-13T14:41:01.828818+0000","last_fullsized":"2024-09-13T14:37:49.126746+0000","mapping_epoch":240,"log_start":"212'3200","ondisk_log_start":"212'3200","created":205,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5675,"log_dups_size":0,"ondisk_log_size":5675,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:48:52.526429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328007680,"num_objects":5005,"num_object_clones":0,"num_object_copies":15015,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3292,"num_objects_unfound":0,"num_objects_dirty":5005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8875,"num_write_kb":444160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5781,"num_bytes_recovered":378863616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,4],"avail_no_missing":["5","0","4"],"object_location_counts":[{"shards":"0,4,5","objects":3292},{"shards":"4,5","objects":1713}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":30,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:40:49.020199+0000","last_change":"2024-09-13T14:38:53.940238+0000","last_active":"2024-09-13T14:40:49.020199+0000","last_peered":"2024-09-13T14:40:49.020199+0000","last_clean":"2024-09-13T14:40:49.020199+0000","last_became_active":"2024-09-13T14:38:53.892825+0000","last_became_peered":"2024-09-13T14:38:53.892825+0000","last_unstale":"2024-09-13T14:40:49.020199+0000","last_undegraded":"2024-09-13T14:40:49.020199+0000","last_fullsized":"2024-09-13T14:40:49.020199+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:46.666581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":87,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:40:49.038366+0000","last_change":"2024-09-13T14:38:53.898907+0000","last_active":"2024-09-13T14:40:49.038366+0000","last_peered":"2024-09-13T14:40:49.038366+0000","last_clean":"2024-09-13T14:40:49.038366+0000","last_became_active":"2024-09-13T14:38:53.898572+0000","last_became_peered":"2024-09-13T14:38:53.898572+0000","last_unstale":"2024-09-13T14:40:49.038366+0000","last_undegraded":"2024-09-13T14:40:49.038366+0000","last_fullsized":"2024-09-13T14:40:49.038366+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:29.222150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"269'9033","reported_seq":23180,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:02.236577+0000","last_change":"2024-09-13T14:38:57.930032+0000","last_active":"2024-09-13T14:41:02.236577+0000","last_peered":"2024-09-13T14:41:02.236577+0000","last_clean":"2024-09-13T14:41:02.236577+0000","last_became_active":"2024-09-13T14:38:57.921458+0000","last_became_peered":"2024-09-13T14:38:57.921458+0000","last_unstale":"2024-09-13T14:41:02.236577+0000","last_undegraded":"2024-09-13T14:41:02.236577+0000","last_fullsized":"2024-09-13T14:41:02.236577+0000","mapping_epoch":256,"log_start":"212'3739","ondisk_log_start":"212'3739","created":205,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5294,"log_dups_size":0,"ondisk_log_size":5294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:27:12.294038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341377024,"num_objects":5209,"num_object_clones":0,"num_object_copies":15627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9033,"num_write_kb":455744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6990,"num_bytes_recovered":458096640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":30,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:40:49.020004+0000","last_change":"2024-09-13T14:38:53.904885+0000","last_active":"2024-09-13T14:40:49.020004+0000","last_peered":"2024-09-13T14:40:49.020004+0000","last_clean":"2024-09-13T14:40:49.020004+0000","last_became_active":"2024-09-13T14:38:53.891726+0000","last_became_peered":"2024-09-13T14:38:53.891726+0000","last_unstale":"2024-09-13T14:40:49.020004+0000","last_undegraded":"2024-09-13T14:40:49.020004+0000","last_fullsized":"2024-09-13T14:40:49.020004+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:57.079205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":276,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:40:49.028531+0000","last_change":"2024-09-13T14:37:48.160504+0000","last_active":"2024-09-13T14:40:49.028531+0000","last_peered":"2024-09-13T14:40:49.028531+0000","last_clean":"2024-09-13T14:40:49.028531+0000","last_became_active":"2024-09-13T14:37:48.160181+0000","last_became_peered":"2024-09-13T14:37:48.160181+0000","last_unstale":"2024-09-13T14:40:49.028531+0000","last_undegraded":"2024-09-13T14:40:49.028531+0000","last_fullsized":"2024-09-13T14:40:49.028531+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:37.087008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"269'9056","reported_seq":10825,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:03.869012+0000","last_change":"2024-09-13T14:38:04.531914+0000","last_active":"2024-09-13T14:41:03.869012+0000","last_peered":"2024-09-13T14:41:03.869012+0000","last_clean":"2024-09-13T14:41:03.869012+0000","last_became_active":"2024-09-13T14:38:04.523448+0000","last_became_peered":"2024-09-13T14:38:04.523448+0000","last_unstale":"2024-09-13T14:41:03.869012+0000","last_undegraded":"2024-09-13T14:41:03.869012+0000","last_fullsized":"2024-09-13T14:41:03.869012+0000","mapping_epoch":245,"log_start":"212'3100","ondisk_log_start":"212'3100","created":205,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5956,"log_dups_size":0,"ondisk_log_size":5956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:06.536896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335413248,"num_objects":5118,"num_object_clones":0,"num_object_copies":15354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9056,"num_write_kb":453568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":811,"num_bytes_recovered":53149696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":268,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:40:49.019957+0000","last_change":"2024-09-13T14:37:49.611774+0000","last_active":"2024-09-13T14:40:49.019957+0000","last_peered":"2024-09-13T14:40:49.019957+0000","last_clean":"2024-09-13T14:40:49.019957+0000","last_became_active":"2024-09-13T14:37:49.611088+0000","last_became_peered":"2024-09-13T14:37:49.611088+0000","last_unstale":"2024-09-13T14:40:49.019957+0000","last_undegraded":"2024-09-13T14:40:49.019957+0000","last_fullsized":"2024-09-13T14:40:49.019957+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:08.933213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":26,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:40:49.019377+0000","last_change":"2024-09-13T14:38:53.899211+0000","last_active":"2024-09-13T14:40:49.019377+0000","last_peered":"2024-09-13T14:40:49.019377+0000","last_clean":"2024-09-13T14:40:49.019377+0000","last_became_active":"2024-09-13T14:38:53.898927+0000","last_became_peered":"2024-09-13T14:38:53.898927+0000","last_unstale":"2024-09-13T14:40:49.019377+0000","last_undegraded":"2024-09-13T14:40:49.019377+0000","last_fullsized":"2024-09-13T14:40:49.019377+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:00:28.857611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"269'8880","reported_seq":8955,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:02.344658+0000","last_change":"2024-09-13T14:35:24.004651+0000","last_active":"2024-09-13T14:41:02.344658+0000","last_peered":"2024-09-13T14:41:02.344658+0000","last_clean":"2024-09-13T14:41:02.344658+0000","last_became_active":"2024-09-13T14:35:24.004358+0000","last_became_peered":"2024-09-13T14:35:24.004358+0000","last_unstale":"2024-09-13T14:41:02.344658+0000","last_undegraded":"2024-09-13T14:41:02.344658+0000","last_fullsized":"2024-09-13T14:41:02.344658+0000","mapping_epoch":205,"log_start":"210'2584","ondisk_log_start":"210'2584","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6296,"log_dups_size":0,"ondisk_log_size":6296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:57:38.023669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333709312,"num_objects":5092,"num_object_clones":0,"num_object_copies":15276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8880,"num_write_kb":447104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":78,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:40:49.039237+0000","last_change":"2024-09-13T14:37:49.168003+0000","last_active":"2024-09-13T14:40:49.039237+0000","last_peered":"2024-09-13T14:40:49.039237+0000","last_clean":"2024-09-13T14:40:49.039237+0000","last_became_active":"2024-09-13T14:37:49.167638+0000","last_became_peered":"2024-09-13T14:37:49.167638+0000","last_unstale":"2024-09-13T14:40:49.039237+0000","last_undegraded":"2024-09-13T14:40:49.039237+0000","last_fullsized":"2024-09-13T14:40:49.039237+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:29.051981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":61,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.067020+0000","last_change":"2024-09-13T14:38:03.528381+0000","last_active":"2024-09-13T14:41:00.067020+0000","last_peered":"2024-09-13T14:41:00.067020+0000","last_clean":"2024-09-13T14:41:00.067020+0000","last_became_active":"2024-09-13T14:38:03.527282+0000","last_became_peered":"2024-09-13T14:38:03.527282+0000","last_unstale":"2024-09-13T14:41:00.067020+0000","last_undegraded":"2024-09-13T14:41:00.067020+0000","last_fullsized":"2024-09-13T14:41:00.067020+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:24.435067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.f","version":"269'8994","reported_seq":9258,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:03.638643+0000","last_change":"2024-09-13T14:38:16.371079+0000","last_active":"2024-09-13T14:41:03.638643+0000","last_peered":"2024-09-13T14:41:03.638643+0000","last_clean":"2024-09-13T14:41:03.638643+0000","last_became_active":"2024-09-13T14:38:12.859883+0000","last_became_peered":"2024-09-13T14:38:12.859883+0000","last_unstale":"2024-09-13T14:41:03.638643+0000","last_undegraded":"2024-09-13T14:41:03.638643+0000","last_fullsized":"2024-09-13T14:41:03.638643+0000","mapping_epoch":245,"log_start":"212'3063","ondisk_log_start":"212'3063","created":205,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5931,"log_dups_size":0,"ondisk_log_size":5931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:08.436532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333709312,"num_objects":5092,"num_object_clones":0,"num_object_copies":15276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8994,"num_write_kb":450752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":3670016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":419,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.056675+0000","last_change":"2024-09-13T14:38:03.549595+0000","last_active":"2024-09-13T14:41:00.056675+0000","last_peered":"2024-09-13T14:41:00.056675+0000","last_clean":"2024-09-13T14:41:00.056675+0000","last_became_active":"2024-09-13T14:38:03.533331+0000","last_became_peered":"2024-09-13T14:38:03.533331+0000","last_unstale":"2024-09-13T14:41:00.056675+0000","last_undegraded":"2024-09-13T14:41:00.056675+0000","last_fullsized":"2024-09-13T14:41:00.056675+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:59:02.470343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":252,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:40:49.028745+0000","last_change":"2024-09-13T14:37:48.127600+0000","last_active":"2024-09-13T14:40:49.028745+0000","last_peered":"2024-09-13T14:40:49.028745+0000","last_clean":"2024-09-13T14:40:49.028745+0000","last_became_active":"2024-09-13T14:37:48.127349+0000","last_became_peered":"2024-09-13T14:37:48.127349+0000","last_unstale":"2024-09-13T14:40:49.028745+0000","last_undegraded":"2024-09-13T14:40:49.028745+0000","last_fullsized":"2024-09-13T14:40:49.028745+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:57.268374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"269'8871","reported_seq":12715,"reported_epoch":269,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:41:01.901827+0000","last_change":"2024-09-13T14:38:14.979229+0000","last_active":"2024-09-13T14:41:01.901827+0000","last_peered":"2024-09-13T14:41:01.901827+0000","last_clean":"2024-09-13T14:36:13.131873+0000","last_became_active":"2024-09-13T14:38:10.443954+0000","last_became_peered":"2024-09-13T14:38:10.443954+0000","last_unstale":"2024-09-13T14:41:01.901827+0000","last_undegraded":"2024-09-13T14:38:10.384643+0000","last_fullsized":"2024-09-13T14:38:04.517046+0000","mapping_epoch":245,"log_start":"212'3200","ondisk_log_start":"212'3200","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5671,"log_dups_size":0,"ondisk_log_size":5671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:26:53.493170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336789504,"num_objects":5139,"num_object_clones":0,"num_object_copies":15417,"num_objects_missing_on_primary":2911,"num_objects_missing":2911,"num_objects_degraded":3052,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8871,"num_write_kb":448320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1861,"num_bytes_recovered":121896960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":1},{"shards":"0,3,5,7","objects":13},{"shards":"0,3,7","objects":2910},{"shards":"0,5","objects":2210},{"shards":"3,5,7","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":185,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:40:49.028622+0000","last_change":"2024-09-13T14:37:48.127604+0000","last_active":"2024-09-13T14:40:49.028622+0000","last_peered":"2024-09-13T14:40:49.028622+0000","last_clean":"2024-09-13T14:40:49.028622+0000","last_became_active":"2024-09-13T14:37:48.127300+0000","last_became_peered":"2024-09-13T14:37:48.127300+0000","last_unstale":"2024-09-13T14:40:49.028622+0000","last_undegraded":"2024-09-13T14:40:49.028622+0000","last_fullsized":"2024-09-13T14:40:49.028622+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:51.999506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":305,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:40:49.029129+0000","last_change":"2024-09-13T14:38:03.529967+0000","last_active":"2024-09-13T14:40:49.029129+0000","last_peered":"2024-09-13T14:40:49.029129+0000","last_clean":"2024-09-13T14:40:49.029129+0000","last_became_active":"2024-09-13T14:38:03.529646+0000","last_became_peered":"2024-09-13T14:38:03.529646+0000","last_unstale":"2024-09-13T14:40:49.029129+0000","last_undegraded":"2024-09-13T14:40:49.029129+0000","last_fullsized":"2024-09-13T14:40:49.029129+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:52:05.193279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"269'9028","reported_seq":18167,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:03.883162+0000","last_change":"2024-09-13T14:40:34.583237+0000","last_active":"2024-09-13T14:41:03.883162+0000","last_peered":"2024-09-13T14:41:03.883162+0000","last_clean":"2024-09-13T14:41:03.883162+0000","last_became_active":"2024-09-13T14:40:34.563989+0000","last_became_peered":"2024-09-13T14:40:34.563989+0000","last_unstale":"2024-09-13T14:41:03.883162+0000","last_undegraded":"2024-09-13T14:41:03.883162+0000","last_fullsized":"2024-09-13T14:41:03.883162+0000","mapping_epoch":266,"log_start":"212'3100","ondisk_log_start":"212'3100","created":205,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5928,"log_dups_size":0,"ondisk_log_size":5928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:34:43.006051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334954520,"num_objects":5112,"num_object_clones":0,"num_object_copies":15336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5112,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9028,"num_write_kb":452417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4489,"num_bytes_recovered":289865752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":265,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:40:49.028806+0000","last_change":"2024-09-13T14:36:13.860367+0000","last_active":"2024-09-13T14:40:49.028806+0000","last_peered":"2024-09-13T14:40:49.028806+0000","last_clean":"2024-09-13T14:40:49.028806+0000","last_became_active":"2024-09-13T14:36:13.859982+0000","last_became_peered":"2024-09-13T14:36:13.859982+0000","last_unstale":"2024-09-13T14:40:49.028806+0000","last_undegraded":"2024-09-13T14:40:49.028806+0000","last_fullsized":"2024-09-13T14:40:49.028806+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:08:23.311350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":261,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:40:49.029015+0000","last_change":"2024-09-13T14:37:48.147360+0000","last_active":"2024-09-13T14:40:49.029015+0000","last_peered":"2024-09-13T14:40:49.029015+0000","last_clean":"2024-09-13T14:40:49.029015+0000","last_became_active":"2024-09-13T14:37:48.146885+0000","last_became_peered":"2024-09-13T14:37:48.146885+0000","last_unstale":"2024-09-13T14:40:49.029015+0000","last_undegraded":"2024-09-13T14:40:49.029015+0000","last_fullsized":"2024-09-13T14:40:49.029015+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:27.841810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"269'8968","reported_seq":18214,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:02.362677+0000","last_change":"2024-09-13T14:38:39.650332+0000","last_active":"2024-09-13T14:41:02.362677+0000","last_peered":"2024-09-13T14:41:02.362677+0000","last_clean":"2024-09-13T14:41:02.362677+0000","last_became_active":"2024-09-13T14:38:39.640835+0000","last_became_peered":"2024-09-13T14:38:39.640835+0000","last_unstale":"2024-09-13T14:41:02.362677+0000","last_undegraded":"2024-09-13T14:41:02.362677+0000","last_fullsized":"2024-09-13T14:41:02.362677+0000","mapping_epoch":251,"log_start":"212'3651","ondisk_log_start":"212'3651","created":205,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5317,"log_dups_size":0,"ondisk_log_size":5317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:48.176606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334626816,"num_objects":5106,"num_object_clones":0,"num_object_copies":15318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8968,"num_write_kb":450368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4544,"num_bytes_recovered":292945920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":437,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.056444+0000","last_change":"2024-09-13T14:38:53.900041+0000","last_active":"2024-09-13T14:41:00.056444+0000","last_peered":"2024-09-13T14:41:00.056444+0000","last_clean":"2024-09-13T14:41:00.056444+0000","last_became_active":"2024-09-13T14:38:53.899203+0000","last_became_peered":"2024-09-13T14:38:53.899203+0000","last_unstale":"2024-09-13T14:41:00.056444+0000","last_undegraded":"2024-09-13T14:41:00.056444+0000","last_fullsized":"2024-09-13T14:41:00.056444+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:28:07.883094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":369,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.057249+0000","last_change":"2024-09-13T14:26:15.823198+0000","last_active":"2024-09-13T14:41:00.057249+0000","last_peered":"2024-09-13T14:41:00.057249+0000","last_clean":"2024-09-13T14:41:00.057249+0000","last_became_active":"2024-09-13T14:26:15.822717+0000","last_became_peered":"2024-09-13T14:26:15.822717+0000","last_unstale":"2024-09-13T14:41:00.057249+0000","last_undegraded":"2024-09-13T14:41:00.057249+0000","last_fullsized":"2024-09-13T14:41:00.057249+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:22.471622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.6","version":"269'9092","reported_seq":13021,"reported_epoch":269,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-13T14:41:03.942176+0000","last_change":"2024-09-13T14:38:55.132683+0000","last_active":"2024-09-13T14:41:03.942176+0000","last_peered":"2024-09-13T14:41:03.942176+0000","last_clean":"2024-09-13T14:36:13.124368+0000","last_became_active":"2024-09-13T14:38:55.120883+0000","last_became_peered":"2024-09-13T14:38:55.120883+0000","last_unstale":"2024-09-13T14:41:03.942176+0000","last_undegraded":"2024-09-13T14:38:55.026610+0000","last_fullsized":"2024-09-13T14:41:03.942176+0000","mapping_epoch":254,"log_start":"212'3119","ondisk_log_start":"212'3119","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5973,"log_dups_size":0,"ondisk_log_size":5973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334626816,"num_objects":5106,"num_object_clones":0,"num_object_copies":15318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3119,"num_objects_misplaced":1987,"num_objects_unfound":0,"num_objects_dirty":5106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9092,"num_write_kb":454336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1886,"num_bytes_recovered":112394240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,3,5],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3,4","objects":2840},{"shards":"0,3,5","objects":2266}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":433,"reported_epoch":269,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-13T14:41:00.056078+0000","last_change":"2024-09-13T14:38:54.899624+0000","last_active":"2024-09-13T14:41:00.056078+0000","last_peered":"2024-09-13T14:41:00.056078+0000","last_clean":"2024-09-13T14:17:27.401592+0000","last_became_active":"2024-09-13T14:38:54.899282+0000","last_became_peered":"2024-09-13T14:38:54.899282+0000","last_unstale":"2024-09-13T14:41:00.056078+0000","last_undegraded":"2024-09-13T14:38:54.891176+0000","last_fullsized":"2024-09-13T14:41:00.056078+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:07:41.927740+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[3,0,6],"avail_no_missing":["3","4","6"],"object_location_counts":[{"shards":"3,4,6","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":507,"reported_epoch":268,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:40:49.039001+0000","last_change":"2024-09-13T14:38:54.904329+0000","last_active":"2024-09-13T14:40:49.039001+0000","last_peered":"2024-09-13T14:40:49.039001+0000","last_clean":"2024-09-13T14:38:39.648017+0000","last_became_active":"2024-09-13T14:38:54.903798+0000","last_became_peered":"2024-09-13T14:38:54.903798+0000","last_unstale":"2024-09-13T14:40:49.039001+0000","last_undegraded":"2024-09-13T14:38:54.890505+0000","last_fullsized":"2024-09-13T14:38:54.889701+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:18.120931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,3],"avail_no_missing":["5","3","6"],"object_location_counts":[{"shards":"3,5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":371,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.056321+0000","last_change":"2024-09-13T14:37:49.160719+0000","last_active":"2024-09-13T14:41:00.056321+0000","last_peered":"2024-09-13T14:41:00.056321+0000","last_clean":"2024-09-13T14:41:00.056321+0000","last_became_active":"2024-09-13T14:37:49.160401+0000","last_became_peered":"2024-09-13T14:37:49.160401+0000","last_unstale":"2024-09-13T14:41:00.056321+0000","last_undegraded":"2024-09-13T14:41:00.056321+0000","last_fullsized":"2024-09-13T14:41:00.056321+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:21.737945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":112,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:40:49.029064+0000","last_change":"2024-09-13T14:37:48.172940+0000","last_active":"2024-09-13T14:40:49.029064+0000","last_peered":"2024-09-13T14:40:49.029064+0000","last_clean":"2024-09-13T14:40:49.029064+0000","last_became_active":"2024-09-13T14:37:48.172060+0000","last_became_peered":"2024-09-13T14:37:48.172060+0000","last_unstale":"2024-09-13T14:40:49.029064+0000","last_undegraded":"2024-09-13T14:40:49.029064+0000","last_fullsized":"2024-09-13T14:40:49.029064+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:07.480645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.5","version":"269'8966","reported_seq":25317,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:02.259135+0000","last_change":"2024-09-13T14:39:34.446699+0000","last_active":"2024-09-13T14:41:02.259135+0000","last_peered":"2024-09-13T14:41:02.259135+0000","last_clean":"2024-09-13T14:41:02.259135+0000","last_became_active":"2024-09-13T14:38:04.600616+0000","last_became_peered":"2024-09-13T14:38:04.600616+0000","last_unstale":"2024-09-13T14:41:02.259135+0000","last_undegraded":"2024-09-13T14:41:02.259135+0000","last_fullsized":"2024-09-13T14:41:02.259135+0000","mapping_epoch":245,"log_start":"212'2616","ondisk_log_start":"212'2616","created":205,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6350,"log_dups_size":0,"ondisk_log_size":6350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:00:44.321678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339214336,"num_objects":5176,"num_object_clones":0,"num_object_copies":15528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8966,"num_write_kb":452544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8120,"num_bytes_recovered":521076736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":485,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:40:49.019371+0000","last_change":"2024-09-13T14:26:15.835178+0000","last_active":"2024-09-13T14:40:49.019371+0000","last_peered":"2024-09-13T14:40:49.019371+0000","last_clean":"2024-09-13T14:40:49.019371+0000","last_became_active":"2024-09-13T14:26:15.820283+0000","last_became_peered":"2024-09-13T14:26:15.820283+0000","last_unstale":"2024-09-13T14:40:49.019371+0000","last_undegraded":"2024-09-13T14:40:49.019371+0000","last_fullsized":"2024-09-13T14:40:49.019371+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:30.340038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":61,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.066667+0000","last_change":"2024-09-13T14:38:03.525682+0000","last_active":"2024-09-13T14:41:00.066667+0000","last_peered":"2024-09-13T14:41:00.066667+0000","last_clean":"2024-09-13T14:41:00.066667+0000","last_became_active":"2024-09-13T14:38:03.525406+0000","last_became_peered":"2024-09-13T14:38:03.525406+0000","last_unstale":"2024-09-13T14:41:00.066667+0000","last_undegraded":"2024-09-13T14:41:00.066667+0000","last_fullsized":"2024-09-13T14:41:00.066667+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:41:28.970300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":68,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.067555+0000","last_change":"2024-09-13T14:38:53.991738+0000","last_active":"2024-09-13T14:41:00.067555+0000","last_peered":"2024-09-13T14:41:00.067555+0000","last_clean":"2024-09-13T14:41:00.067555+0000","last_became_active":"2024-09-13T14:38:53.991417+0000","last_became_peered":"2024-09-13T14:38:53.991417+0000","last_unstale":"2024-09-13T14:41:00.067555+0000","last_undegraded":"2024-09-13T14:41:00.067555+0000","last_fullsized":"2024-09-13T14:41:00.067555+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:50:38.559838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"269'9039","reported_seq":23200,"reported_epoch":269,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-13T14:41:03.829413+0000","last_change":"2024-09-13T14:38:17.472071+0000","last_active":"2024-09-13T14:41:03.829413+0000","last_peered":"2024-09-13T14:41:03.829413+0000","last_clean":"2024-09-13T14:37:46.290754+0000","last_became_active":"2024-09-13T14:38:17.381753+0000","last_became_peered":"2024-09-13T14:38:17.381753+0000","last_unstale":"2024-09-13T14:41:03.829413+0000","last_undegraded":"2024-09-13T14:38:13.504140+0000","last_fullsized":"2024-09-13T14:41:03.829413+0000","mapping_epoch":245,"log_start":"212'3059","ondisk_log_start":"212'3059","created":205,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5980,"log_dups_size":0,"ondisk_log_size":5980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332070912,"num_objects":5067,"num_object_clones":0,"num_object_copies":15201,"num_objects_missing_on_primary":83,"num_objects_missing":83,"num_objects_degraded":4582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9039,"num_write_kb":451392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7053,"num_bytes_recovered":462225408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"0,4,5","objects":4413},{"shards":"0,4,6","objects":654}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":782,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.056263+0000","last_change":"2024-09-13T14:37:48.131799+0000","last_active":"2024-09-13T14:41:00.056263+0000","last_peered":"2024-09-13T14:41:00.056263+0000","last_clean":"2024-09-13T14:41:00.056263+0000","last_became_active":"2024-09-13T14:37:48.130837+0000","last_became_peered":"2024-09-13T14:37:48.130837+0000","last_unstale":"2024-09-13T14:41:00.056263+0000","last_undegraded":"2024-09-13T14:41:00.056263+0000","last_fullsized":"2024-09-13T14:41:00.056263+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:06:32.663770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":301,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.066887+0000","last_change":"2024-09-13T14:37:49.301531+0000","last_active":"2024-09-13T14:41:00.066887+0000","last_peered":"2024-09-13T14:41:00.066887+0000","last_clean":"2024-09-13T14:41:00.066887+0000","last_became_active":"2024-09-13T14:37:49.301249+0000","last_became_peered":"2024-09-13T14:37:49.301249+0000","last_unstale":"2024-09-13T14:41:00.066887+0000","last_undegraded":"2024-09-13T14:41:00.066887+0000","last_fullsized":"2024-09-13T14:41:00.066887+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:10.977710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":94,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:40:49.039721+0000","last_change":"2024-09-13T14:38:53.901427+0000","last_active":"2024-09-13T14:40:49.039721+0000","last_peered":"2024-09-13T14:40:49.039721+0000","last_clean":"2024-09-13T14:40:49.039721+0000","last_became_active":"2024-09-13T14:38:53.901157+0000","last_became_peered":"2024-09-13T14:38:53.901157+0000","last_unstale":"2024-09-13T14:40:49.039721+0000","last_undegraded":"2024-09-13T14:40:49.039721+0000","last_fullsized":"2024-09-13T14:40:49.039721+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:28:21.713025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"269'8948","reported_seq":22999,"reported_epoch":269,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-13T14:41:03.878965+0000","last_change":"2024-09-13T14:38:16.552896+0000","last_active":"2024-09-13T14:41:03.878965+0000","last_peered":"2024-09-13T14:41:03.878965+0000","last_clean":"2024-09-13T14:37:46.405415+0000","last_became_active":"2024-09-13T14:38:16.427510+0000","last_became_peered":"2024-09-13T14:38:16.427510+0000","last_unstale":"2024-09-13T14:41:03.878965+0000","last_undegraded":"2024-09-13T14:38:12.738464+0000","last_fullsized":"2024-09-13T14:41:03.878965+0000","mapping_epoch":245,"log_start":"212'3018","ondisk_log_start":"212'3018","created":205,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5930,"log_dups_size":0,"ondisk_log_size":5930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330956800,"num_objects":5050,"num_object_clones":0,"num_object_copies":15150,"num_objects_missing_on_primary":100,"num_objects_missing":100,"num_objects_degraded":4572,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8948,"num_write_kb":447936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6999,"num_bytes_recovered":458686464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"0,3,6","objects":645},{"shards":"0,3,7","objects":4405}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":976,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:40:49.038558+0000","last_change":"2024-09-13T14:37:49.147498+0000","last_active":"2024-09-13T14:40:49.038558+0000","last_peered":"2024-09-13T14:40:49.038558+0000","last_clean":"2024-09-13T14:40:49.038558+0000","last_became_active":"2024-09-13T14:37:49.147028+0000","last_became_peered":"2024-09-13T14:37:49.147028+0000","last_unstale":"2024-09-13T14:40:49.038558+0000","last_undegraded":"2024-09-13T14:40:49.038558+0000","last_fullsized":"2024-09-13T14:40:49.038558+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:11:29.050178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":274,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:40:49.029049+0000","last_change":"2024-09-13T14:38:03.528133+0000","last_active":"2024-09-13T14:40:49.029049+0000","last_peered":"2024-09-13T14:40:49.029049+0000","last_clean":"2024-09-13T14:40:49.029049+0000","last_became_active":"2024-09-13T14:38:03.527829+0000","last_became_peered":"2024-09-13T14:38:03.527829+0000","last_unstale":"2024-09-13T14:40:49.029049+0000","last_undegraded":"2024-09-13T14:40:49.029049+0000","last_fullsized":"2024-09-13T14:40:49.029049+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:48.254169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":421,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.056561+0000","last_change":"2024-09-13T14:38:53.899755+0000","last_active":"2024-09-13T14:41:00.056561+0000","last_peered":"2024-09-13T14:41:00.056561+0000","last_clean":"2024-09-13T14:41:00.056561+0000","last_became_active":"2024-09-13T14:38:53.899448+0000","last_became_peered":"2024-09-13T14:38:53.899448+0000","last_unstale":"2024-09-13T14:41:00.056561+0000","last_undegraded":"2024-09-13T14:41:00.056561+0000","last_fullsized":"2024-09-13T14:41:00.056561+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:55.897035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"269'9152","reported_seq":9245,"reported_epoch":269,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:41:02.255686+0000","last_change":"2024-09-13T14:38:03.567892+0000","last_active":"2024-09-13T14:41:02.255686+0000","last_peered":"2024-09-13T14:41:02.255686+0000","last_clean":"2024-09-13T14:35:51.090791+0000","last_became_active":"2024-09-13T14:37:49.665025+0000","last_became_peered":"2024-09-13T14:37:49.665025+0000","last_unstale":"2024-09-13T14:41:02.255686+0000","last_undegraded":"2024-09-13T14:37:49.169101+0000","last_fullsized":"2024-09-13T14:37:49.121596+0000","mapping_epoch":240,"log_start":"212'3063","ondisk_log_start":"212'3063","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6089,"log_dups_size":0,"ondisk_log_size":6089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:02.321504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":337510400,"num_objects":5150,"num_object_clones":0,"num_object_copies":15450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4501,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9152,"num_write_kb":457664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":715}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":78,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:40:49.039306+0000","last_change":"2024-09-13T14:37:49.149770+0000","last_active":"2024-09-13T14:40:49.039306+0000","last_peered":"2024-09-13T14:40:49.039306+0000","last_clean":"2024-09-13T14:40:49.039306+0000","last_became_active":"2024-09-13T14:37:49.148991+0000","last_became_peered":"2024-09-13T14:37:49.148991+0000","last_unstale":"2024-09-13T14:40:49.039306+0000","last_undegraded":"2024-09-13T14:40:49.039306+0000","last_fullsized":"2024-09-13T14:40:49.039306+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:04:04.193366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":1120,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.076693+0000","last_change":"2024-09-13T14:30:58.928302+0000","last_active":"2024-09-13T14:41:00.076693+0000","last_peered":"2024-09-13T14:41:00.076693+0000","last_clean":"2024-09-13T14:41:00.076693+0000","last_became_active":"2024-09-13T14:30:58.927718+0000","last_became_peered":"2024-09-13T14:30:58.927718+0000","last_unstale":"2024-09-13T14:41:00.076693+0000","last_undegraded":"2024-09-13T14:41:00.076693+0000","last_fullsized":"2024-09-13T14:41:00.076693+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:37.323036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":85,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:40:49.038735+0000","last_change":"2024-09-13T14:38:03.534628+0000","last_active":"2024-09-13T14:40:49.038735+0000","last_peered":"2024-09-13T14:40:49.038735+0000","last_clean":"2024-09-13T14:40:49.038735+0000","last_became_active":"2024-09-13T14:38:03.534262+0000","last_became_peered":"2024-09-13T14:38:03.534262+0000","last_unstale":"2024-09-13T14:40:49.038735+0000","last_undegraded":"2024-09-13T14:40:49.038735+0000","last_fullsized":"2024-09-13T14:40:49.038735+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:31.168783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"269'8909","reported_seq":23064,"reported_epoch":269,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:41:01.900253+0000","last_change":"2024-09-13T14:38:53.944111+0000","last_active":"2024-09-13T14:41:01.900253+0000","last_peered":"2024-09-13T14:41:01.900253+0000","last_clean":"2024-09-13T14:37:47.047672+0000","last_became_active":"2024-09-13T14:37:49.664294+0000","last_became_peered":"2024-09-13T14:37:49.664294+0000","last_unstale":"2024-09-13T14:41:01.900253+0000","last_undegraded":"2024-09-13T14:38:53.940822+0000","last_fullsized":"2024-09-13T14:37:49.126627+0000","mapping_epoch":240,"log_start":"212'3200","ondisk_log_start":"212'3200","created":205,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5709,"log_dups_size":0,"ondisk_log_size":5709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:11.766692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":337051648,"num_objects":5143,"num_object_clones":0,"num_object_copies":15429,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4535,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8909,"num_write_kb":449664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7009,"num_bytes_recovered":459341824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3],"avail_no_missing":["5","0","3"],"object_location_counts":[{"shards":"0,3,5","objects":4460},{"shards":"3,5","objects":683}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":97,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:40:49.039405+0000","last_change":"2024-09-13T14:38:03.534303+0000","last_active":"2024-09-13T14:40:49.039405+0000","last_peered":"2024-09-13T14:40:49.039405+0000","last_clean":"2024-09-13T14:40:49.039405+0000","last_became_active":"2024-09-13T14:38:03.533171+0000","last_became_peered":"2024-09-13T14:38:03.533171+0000","last_unstale":"2024-09-13T14:40:49.039405+0000","last_undegraded":"2024-09-13T14:40:49.039405+0000","last_fullsized":"2024-09-13T14:40:49.039405+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:00:16.553526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":789,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:40:49.020033+0000","last_change":"2024-09-13T14:38:04.596192+0000","last_active":"2024-09-13T14:40:49.020033+0000","last_peered":"2024-09-13T14:40:49.020033+0000","last_clean":"2024-09-13T14:40:49.020033+0000","last_became_active":"2024-09-13T14:38:04.593299+0000","last_became_peered":"2024-09-13T14:38:04.593299+0000","last_unstale":"2024-09-13T14:40:49.020033+0000","last_undegraded":"2024-09-13T14:40:49.020033+0000","last_fullsized":"2024-09-13T14:40:49.020033+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:34.241410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":61,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.067077+0000","last_change":"2024-09-13T14:38:03.533120+0000","last_active":"2024-09-13T14:41:00.067077+0000","last_peered":"2024-09-13T14:41:00.067077+0000","last_clean":"2024-09-13T14:41:00.067077+0000","last_became_active":"2024-09-13T14:38:03.532829+0000","last_became_peered":"2024-09-13T14:38:03.532829+0000","last_unstale":"2024-09-13T14:41:00.067077+0000","last_undegraded":"2024-09-13T14:41:00.067077+0000","last_fullsized":"2024-09-13T14:41:00.067077+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:24.514661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"269'9080","reported_seq":9228,"reported_epoch":269,"state":"active+undersized+degraded+remapped+backfill_wait","last_fresh":"2024-09-13T14:41:02.466842+0000","last_change":"2024-09-13T14:37:49.295861+0000","last_active":"2024-09-13T14:41:02.466842+0000","last_peered":"2024-09-13T14:41:02.466842+0000","last_clean":"2024-09-13T14:35:51.078708+0000","last_became_active":"2024-09-13T14:37:49.290403+0000","last_became_peered":"2024-09-13T14:37:49.290403+0000","last_unstale":"2024-09-13T14:41:02.466842+0000","last_undegraded":"2024-09-13T14:37:49.147402+0000","last_fullsized":"2024-09-13T14:37:49.117202+0000","mapping_epoch":240,"log_start":"212'3718","ondisk_log_start":"212'3718","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5362,"log_dups_size":0,"ondisk_log_size":5362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":348520448,"num_objects":5318,"num_object_clones":0,"num_object_copies":15954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5318,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9080,"num_write_kb":460736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":5318}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":463,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.057260+0000","last_change":"2024-09-13T14:38:53.938227+0000","last_active":"2024-09-13T14:41:00.057260+0000","last_peered":"2024-09-13T14:41:00.057260+0000","last_clean":"2024-09-13T14:41:00.057260+0000","last_became_active":"2024-09-13T14:38:53.902867+0000","last_became_peered":"2024-09-13T14:38:53.902867+0000","last_unstale":"2024-09-13T14:41:00.057260+0000","last_undegraded":"2024-09-13T14:41:00.057260+0000","last_fullsized":"2024-09-13T14:41:00.057260+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:16.831756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":568,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:40:49.029133+0000","last_change":"2024-09-13T14:37:48.147677+0000","last_active":"2024-09-13T14:40:49.029133+0000","last_peered":"2024-09-13T14:40:49.029133+0000","last_clean":"2024-09-13T14:40:49.029133+0000","last_became_active":"2024-09-13T14:37:48.147001+0000","last_became_peered":"2024-09-13T14:37:48.147001+0000","last_unstale":"2024-09-13T14:40:49.029133+0000","last_undegraded":"2024-09-13T14:40:49.029133+0000","last_fullsized":"2024-09-13T14:40:49.029133+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:35:00.653576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":445,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.056006+0000","last_change":"2024-09-13T14:38:53.900107+0000","last_active":"2024-09-13T14:41:00.056006+0000","last_peered":"2024-09-13T14:41:00.056006+0000","last_clean":"2024-09-13T14:41:00.056006+0000","last_became_active":"2024-09-13T14:38:53.899556+0000","last_became_peered":"2024-09-13T14:38:53.899556+0000","last_unstale":"2024-09-13T14:41:00.056006+0000","last_undegraded":"2024-09-13T14:41:00.056006+0000","last_fullsized":"2024-09-13T14:41:00.056006+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:15:47.918163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"269'8804","reported_seq":17755,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:02.205757+0000","last_change":"2024-09-13T14:39:54.921239+0000","last_active":"2024-09-13T14:41:02.205757+0000","last_peered":"2024-09-13T14:41:02.205757+0000","last_clean":"2024-09-13T14:41:02.205757+0000","last_became_active":"2024-09-13T14:39:54.907894+0000","last_became_peered":"2024-09-13T14:39:54.907894+0000","last_unstale":"2024-09-13T14:41:02.205757+0000","last_undegraded":"2024-09-13T14:41:02.205757+0000","last_fullsized":"2024-09-13T14:41:02.205757+0000","mapping_epoch":263,"log_start":"212'3447","ondisk_log_start":"212'3447","created":205,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5357,"log_dups_size":0,"ondisk_log_size":5357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:52:04.619238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330825728,"num_objects":5048,"num_object_clones":0,"num_object_copies":15144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8804,"num_write_kb":443264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4395,"num_bytes_recovered":283770880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":76,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.067403+0000","last_change":"2024-09-13T14:38:53.901806+0000","last_active":"2024-09-13T14:41:00.067403+0000","last_peered":"2024-09-13T14:41:00.067403+0000","last_clean":"2024-09-13T14:41:00.067403+0000","last_became_active":"2024-09-13T14:38:53.891055+0000","last_became_peered":"2024-09-13T14:38:53.891055+0000","last_unstale":"2024-09-13T14:41:00.067403+0000","last_undegraded":"2024-09-13T14:41:00.067403+0000","last_fullsized":"2024-09-13T14:41:00.067403+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:03:00.416198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":24,"reported_epoch":268,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-13T14:40:49.020240+0000","last_change":"2024-09-13T14:38:53.900902+0000","last_active":"2024-09-13T14:40:49.020240+0000","last_peered":"2024-09-13T14:40:49.020240+0000","last_clean":"2024-09-13T14:38:39.635779+0000","last_became_active":"2024-09-13T14:38:53.900626+0000","last_became_peered":"2024-09-13T14:38:53.900626+0000","last_unstale":"2024-09-13T14:40:49.020240+0000","last_undegraded":"2024-09-13T14:38:53.885778+0000","last_fullsized":"2024-09-13T14:40:49.020240+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:37:21.715778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":7,"num_objects_missing":7,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":["3","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":258,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:40:49.028667+0000","last_change":"2024-09-13T14:37:49.172218+0000","last_active":"2024-09-13T14:40:49.028667+0000","last_peered":"2024-09-13T14:40:49.028667+0000","last_clean":"2024-09-13T14:40:49.028667+0000","last_became_active":"2024-09-13T14:37:49.171718+0000","last_became_peered":"2024-09-13T14:37:49.171718+0000","last_unstale":"2024-09-13T14:40:49.028667+0000","last_undegraded":"2024-09-13T14:40:49.028667+0000","last_fullsized":"2024-09-13T14:40:49.028667+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:46:54.626521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":280,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:40:49.028849+0000","last_change":"2024-09-13T14:37:49.230094+0000","last_active":"2024-09-13T14:40:49.028849+0000","last_peered":"2024-09-13T14:40:49.028849+0000","last_clean":"2024-09-13T14:40:49.028849+0000","last_became_active":"2024-09-13T14:37:49.229393+0000","last_became_peered":"2024-09-13T14:37:49.229393+0000","last_unstale":"2024-09-13T14:40:49.028849+0000","last_undegraded":"2024-09-13T14:40:49.028849+0000","last_fullsized":"2024-09-13T14:40:49.028849+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:02:28.690318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":260,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:40:49.028879+0000","last_change":"2024-09-13T14:37:48.128447+0000","last_active":"2024-09-13T14:40:49.028879+0000","last_peered":"2024-09-13T14:40:49.028879+0000","last_clean":"2024-09-13T14:40:49.028879+0000","last_became_active":"2024-09-13T14:37:48.128143+0000","last_became_peered":"2024-09-13T14:37:48.128143+0000","last_unstale":"2024-09-13T14:40:49.028879+0000","last_undegraded":"2024-09-13T14:40:49.028879+0000","last_fullsized":"2024-09-13T14:40:49.028879+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:22.204039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":61,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.066996+0000","last_change":"2024-09-13T14:38:03.531683+0000","last_active":"2024-09-13T14:41:00.066996+0000","last_peered":"2024-09-13T14:41:00.066996+0000","last_clean":"2024-09-13T14:41:00.066996+0000","last_became_active":"2024-09-13T14:38:03.531476+0000","last_became_peered":"2024-09-13T14:38:03.531476+0000","last_unstale":"2024-09-13T14:41:00.066996+0000","last_undegraded":"2024-09-13T14:41:00.066996+0000","last_fullsized":"2024-09-13T14:41:00.066996+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:41.557569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":32,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:40:49.019686+0000","last_change":"2024-09-13T14:38:53.954281+0000","last_active":"2024-09-13T14:40:49.019686+0000","last_peered":"2024-09-13T14:40:49.019686+0000","last_clean":"2024-09-13T14:40:49.019686+0000","last_became_active":"2024-09-13T14:38:53.900912+0000","last_became_peered":"2024-09-13T14:38:53.900912+0000","last_unstale":"2024-09-13T14:40:49.019686+0000","last_undegraded":"2024-09-13T14:40:49.019686+0000","last_fullsized":"2024-09-13T14:40:49.019686+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:47.876275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":84,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:40:49.049645+0000","last_change":"2024-09-13T14:38:54.000782+0000","last_active":"2024-09-13T14:40:49.049645+0000","last_peered":"2024-09-13T14:40:49.049645+0000","last_clean":"2024-09-13T14:40:49.049645+0000","last_became_active":"2024-09-13T14:38:54.000322+0000","last_became_peered":"2024-09-13T14:38:54.000322+0000","last_unstale":"2024-09-13T14:40:49.049645+0000","last_undegraded":"2024-09-13T14:40:49.049645+0000","last_fullsized":"2024-09-13T14:40:49.049645+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:09.057597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":386,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.056031+0000","last_change":"2024-09-13T14:38:53.899535+0000","last_active":"2024-09-13T14:41:00.056031+0000","last_peered":"2024-09-13T14:41:00.056031+0000","last_clean":"2024-09-13T14:41:00.056031+0000","last_became_active":"2024-09-13T14:38:53.899183+0000","last_became_peered":"2024-09-13T14:38:53.899183+0000","last_unstale":"2024-09-13T14:41:00.056031+0000","last_undegraded":"2024-09-13T14:41:00.056031+0000","last_fullsized":"2024-09-13T14:41:00.056031+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:33.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":61,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.068697+0000","last_change":"2024-09-13T14:38:03.543426+0000","last_active":"2024-09-13T14:41:00.068697+0000","last_peered":"2024-09-13T14:41:00.068697+0000","last_clean":"2024-09-13T14:41:00.068697+0000","last_became_active":"2024-09-13T14:38:03.542661+0000","last_became_peered":"2024-09-13T14:38:03.542661+0000","last_unstale":"2024-09-13T14:41:00.068697+0000","last_undegraded":"2024-09-13T14:41:00.068697+0000","last_fullsized":"2024-09-13T14:41:00.068697+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:54.050127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":415,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.055968+0000","last_change":"2024-09-13T14:37:48.131321+0000","last_active":"2024-09-13T14:41:00.055968+0000","last_peered":"2024-09-13T14:41:00.055968+0000","last_clean":"2024-09-13T14:41:00.055968+0000","last_became_active":"2024-09-13T14:37:48.130998+0000","last_became_peered":"2024-09-13T14:37:48.130998+0000","last_unstale":"2024-09-13T14:41:00.055968+0000","last_undegraded":"2024-09-13T14:41:00.055968+0000","last_fullsized":"2024-09-13T14:41:00.055968+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:33:26.597825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":97,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:40:49.039696+0000","last_change":"2024-09-13T14:38:03.535454+0000","last_active":"2024-09-13T14:40:49.039696+0000","last_peered":"2024-09-13T14:40:49.039696+0000","last_clean":"2024-09-13T14:40:49.039696+0000","last_became_active":"2024-09-13T14:38:03.535131+0000","last_became_peered":"2024-09-13T14:38:03.535131+0000","last_unstale":"2024-09-13T14:40:49.039696+0000","last_undegraded":"2024-09-13T14:40:49.039696+0000","last_fullsized":"2024-09-13T14:40:49.039696+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:24.477601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":440,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.056410+0000","last_change":"2024-09-13T14:38:03.524939+0000","last_active":"2024-09-13T14:41:00.056410+0000","last_peered":"2024-09-13T14:41:00.056410+0000","last_clean":"2024-09-13T14:41:00.056410+0000","last_became_active":"2024-09-13T14:38:03.524406+0000","last_became_peered":"2024-09-13T14:38:03.524406+0000","last_unstale":"2024-09-13T14:41:00.056410+0000","last_undegraded":"2024-09-13T14:41:00.056410+0000","last_fullsized":"2024-09-13T14:41:00.056410+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:19:45.257424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":68,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.067328+0000","last_change":"2024-09-13T14:38:53.892948+0000","last_active":"2024-09-13T14:41:00.067328+0000","last_peered":"2024-09-13T14:41:00.067328+0000","last_clean":"2024-09-13T14:41:00.067328+0000","last_became_active":"2024-09-13T14:38:53.892665+0000","last_became_peered":"2024-09-13T14:38:53.892665+0000","last_unstale":"2024-09-13T14:41:00.067328+0000","last_undegraded":"2024-09-13T14:41:00.067328+0000","last_fullsized":"2024-09-13T14:41:00.067328+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:08.292359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":389,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.056256+0000","last_change":"2024-09-13T14:37:48.139348+0000","last_active":"2024-09-13T14:41:00.056256+0000","last_peered":"2024-09-13T14:41:00.056256+0000","last_clean":"2024-09-13T14:41:00.056256+0000","last_became_active":"2024-09-13T14:37:48.139115+0000","last_became_peered":"2024-09-13T14:37:48.139115+0000","last_unstale":"2024-09-13T14:41:00.056256+0000","last_undegraded":"2024-09-13T14:41:00.056256+0000","last_fullsized":"2024-09-13T14:41:00.056256+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:32:20.413346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":429,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.056634+0000","last_change":"2024-09-13T14:37:48.172088+0000","last_active":"2024-09-13T14:41:00.056634+0000","last_peered":"2024-09-13T14:41:00.056634+0000","last_clean":"2024-09-13T14:41:00.056634+0000","last_became_active":"2024-09-13T14:37:48.171779+0000","last_became_peered":"2024-09-13T14:37:48.171779+0000","last_unstale":"2024-09-13T14:41:00.056634+0000","last_undegraded":"2024-09-13T14:41:00.056634+0000","last_fullsized":"2024-09-13T14:41:00.056634+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:02.852712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":324,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:40:49.028723+0000","last_change":"2024-09-13T14:38:53.890236+0000","last_active":"2024-09-13T14:40:49.028723+0000","last_peered":"2024-09-13T14:40:49.028723+0000","last_clean":"2024-09-13T14:40:49.028723+0000","last_became_active":"2024-09-13T14:38:53.889582+0000","last_became_peered":"2024-09-13T14:38:53.889582+0000","last_unstale":"2024-09-13T14:40:49.028723+0000","last_undegraded":"2024-09-13T14:40:49.028723+0000","last_fullsized":"2024-09-13T14:40:49.028723+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:14:29.334774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":415,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.056674+0000","last_change":"2024-09-13T14:38:03.534097+0000","last_active":"2024-09-13T14:41:00.056674+0000","last_peered":"2024-09-13T14:41:00.056674+0000","last_clean":"2024-09-13T14:41:00.056674+0000","last_became_active":"2024-09-13T14:38:03.533794+0000","last_became_peered":"2024-09-13T14:38:03.533794+0000","last_unstale":"2024-09-13T14:41:00.056674+0000","last_undegraded":"2024-09-13T14:41:00.056674+0000","last_fullsized":"2024-09-13T14:41:00.056674+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:23.272098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":61,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.066980+0000","last_change":"2024-09-13T14:38:03.527021+0000","last_active":"2024-09-13T14:41:00.066980+0000","last_peered":"2024-09-13T14:41:00.066980+0000","last_clean":"2024-09-13T14:41:00.066980+0000","last_became_active":"2024-09-13T14:38:03.526698+0000","last_became_peered":"2024-09-13T14:38:03.526698+0000","last_unstale":"2024-09-13T14:41:00.066980+0000","last_undegraded":"2024-09-13T14:41:00.066980+0000","last_fullsized":"2024-09-13T14:41:00.066980+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:01:35.734086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":268,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:40:49.028779+0000","last_change":"2024-09-13T14:27:12.834931+0000","last_active":"2024-09-13T14:40:49.028779+0000","last_peered":"2024-09-13T14:40:49.028779+0000","last_clean":"2024-09-13T14:40:49.028779+0000","last_became_active":"2024-09-13T14:27:12.834572+0000","last_became_peered":"2024-09-13T14:27:12.834572+0000","last_unstale":"2024-09-13T14:40:49.028779+0000","last_undegraded":"2024-09-13T14:40:49.028779+0000","last_fullsized":"2024-09-13T14:40:49.028779+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:54:21.155689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":81,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:40:49.039318+0000","last_change":"2024-09-13T14:37:49.176796+0000","last_active":"2024-09-13T14:40:49.039318+0000","last_peered":"2024-09-13T14:40:49.039318+0000","last_clean":"2024-09-13T14:40:49.039318+0000","last_became_active":"2024-09-13T14:37:49.176246+0000","last_became_peered":"2024-09-13T14:37:49.176246+0000","last_unstale":"2024-09-13T14:40:49.039318+0000","last_undegraded":"2024-09-13T14:40:49.039318+0000","last_fullsized":"2024-09-13T14:40:49.039318+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:29:24.936718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":279,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:40:49.028555+0000","last_change":"2024-09-13T14:38:03.526333+0000","last_active":"2024-09-13T14:40:49.028555+0000","last_peered":"2024-09-13T14:40:49.028555+0000","last_clean":"2024-09-13T14:40:49.028555+0000","last_became_active":"2024-09-13T14:38:03.526001+0000","last_became_peered":"2024-09-13T14:38:03.526001+0000","last_unstale":"2024-09-13T14:40:49.028555+0000","last_undegraded":"2024-09-13T14:40:49.028555+0000","last_fullsized":"2024-09-13T14:40:49.028555+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:25:25.181271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":81,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:40:49.039399+0000","last_change":"2024-09-13T14:37:49.149582+0000","last_active":"2024-09-13T14:40:49.039399+0000","last_peered":"2024-09-13T14:40:49.039399+0000","last_clean":"2024-09-13T14:40:49.039399+0000","last_became_active":"2024-09-13T14:37:49.148525+0000","last_became_peered":"2024-09-13T14:37:49.148525+0000","last_unstale":"2024-09-13T14:40:49.039399+0000","last_undegraded":"2024-09-13T14:40:49.039399+0000","last_fullsized":"2024-09-13T14:40:49.039399+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:48.655665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":461,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.056364+0000","last_change":"2024-09-13T14:38:03.535255+0000","last_active":"2024-09-13T14:41:00.056364+0000","last_peered":"2024-09-13T14:41:00.056364+0000","last_clean":"2024-09-13T14:41:00.056364+0000","last_became_active":"2024-09-13T14:38:03.533917+0000","last_became_peered":"2024-09-13T14:38:03.533917+0000","last_unstale":"2024-09-13T14:41:00.056364+0000","last_undegraded":"2024-09-13T14:41:00.056364+0000","last_fullsized":"2024-09-13T14:41:00.056364+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:33.559788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":425,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.057120+0000","last_change":"2024-09-13T14:37:48.146784+0000","last_active":"2024-09-13T14:41:00.057120+0000","last_peered":"2024-09-13T14:41:00.057120+0000","last_clean":"2024-09-13T14:41:00.057120+0000","last_became_active":"2024-09-13T14:37:48.146491+0000","last_became_peered":"2024-09-13T14:37:48.146491+0000","last_unstale":"2024-09-13T14:41:00.057120+0000","last_undegraded":"2024-09-13T14:41:00.057120+0000","last_fullsized":"2024-09-13T14:41:00.057120+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:30:54.772513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":260,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-13T14:40:49.028488+0000","last_change":"2024-09-13T14:37:48.127607+0000","last_active":"2024-09-13T14:40:49.028488+0000","last_peered":"2024-09-13T14:40:49.028488+0000","last_clean":"2024-09-13T14:40:49.028488+0000","last_became_active":"2024-09-13T14:37:48.127316+0000","last_became_peered":"2024-09-13T14:37:48.127316+0000","last_unstale":"2024-09-13T14:40:49.028488+0000","last_undegraded":"2024-09-13T14:40:49.028488+0000","last_fullsized":"2024-09-13T14:40:49.028488+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:53:37.678861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":67,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.068596+0000","last_change":"2024-09-13T14:38:03.563200+0000","last_active":"2024-09-13T14:41:00.068596+0000","last_peered":"2024-09-13T14:41:00.068596+0000","last_clean":"2024-09-13T14:41:00.068596+0000","last_became_active":"2024-09-13T14:38:03.534258+0000","last_became_peered":"2024-09-13T14:38:03.534258+0000","last_unstale":"2024-09-13T14:41:00.068596+0000","last_undegraded":"2024-09-13T14:41:00.068596+0000","last_fullsized":"2024-09-13T14:41:00.068596+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:49:07.814419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":5369364504,"num_objects":81931,"num_object_clones":0,"num_object_copies":245793,"num_objects_missing_on_primary":3094,"num_objects_missing":3094,"num_objects_degraded":29679,"num_objects_misplaced":5279,"num_objects_unfound":0,"num_objects_dirty":81931,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143695,"num_write_kb":7219969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59995,"num_bytes_recovered":3896049688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17302171648,"data_stored":17302159432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92818,"ondisk_log_size":92818,"up":48,"acting":43,"num_store_stats":6},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1538,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":17412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25132,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":7,"num_objects_missing":7,"num_objects_degraded":16,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16793600,"data_stored":16777292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":23,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234366,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3156736,"kb_used_data":2702328,"kb_used_omap":1,"kb_used_meta":454398,"kb_avail":90588416,"statfs":{"total":95995035648,"available":92762537984,"internally_reserved":0,"allocated":2767183872,"data_stored":2765718161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":465304011},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":239,"seq":1026497183784,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1984076,"kb_used_data":1801152,"kb_used_omap":8,"kb_used_meta":182903,"kb_avail":91761076,"statfs":{"total":95995035648,"available":93963341824,"internally_reserved":0,"allocated":1844379648,"data_stored":1842906860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8453,"internal_metadata":187293435},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757631,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2348216,"kb_used_data":1721516,"kb_used_omap":1,"kb_used_meta":626686,"kb_avail":91396936,"statfs":{"total":95995035648,"available":93590462464,"internally_reserved":0,"allocated":1762832384,"data_stored":1761374682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":641726918},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019275,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3390464,"kb_used_data":2822008,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":90354688,"statfs":{"total":95995035648,"available":92523200512,"internally_reserved":0,"allocated":2889736192,"data_stored":2888271013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":582089158},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435458,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4504956,"kb_used_data":3949872,"kb_used_omap":5,"kb_used_meta":555066,"kb_avail":89240196,"statfs":{"total":95995035648,"available":91381960704,"internally_reserved":0,"allocated":4044668928,"data_stored":4043207460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":568387710},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":446676598986,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":425201762513,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926037,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4615356,"kb_used_data":3947828,"kb_used_omap":7,"kb_used_meta":667512,"kb_avail":89129796,"statfs":{"total":95995035648,"available":91268911104,"internally_reserved":0,"allocated":4042575872,"data_stored":4041110365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":683532584},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6860,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4029091840,"data_stored":4029087768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4037677056,"data_stored":4037672984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2883846144,"data_stored":2883846144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1752629248,"data_stored":1752629248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1835274240,"data_stored":1835270168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2763653120,"data_stored":2763653120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:41:07.299 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:07.325 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:07.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:07.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:07.371 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:07 smithi195 ceph-mon[104262]: pgmap v892: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 5.0 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 202 op/s; 29697/245877 objects degraded (12.078%); 5281/245877 objects misplaced (2.148%); 4.3 MiB/s, 68 objects/s recovering 2024-09-13T14:41:07.380 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:07 smithi102 ceph-mon[110183]: pgmap v892: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 5.0 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 202 op/s; 29697/245877 objects degraded (12.078%); 5281/245877 objects misplaced (2.148%); 4.3 MiB/s, 68 objects/s recovering 2024-09-13T14:41:07.381 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:07 smithi102 ceph-mon[115269]: pgmap v892: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 5.0 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 202 op/s; 29697/245877 objects degraded (12.078%); 5281/245877 objects misplaced (2.148%); 4.3 MiB/s, 68 objects/s recovering 2024-09-13T14:41:07.426 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:07.452 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:07.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:07.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:07.554 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:07.576 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:07.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:07.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:07.677 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:07.704 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:07.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:07.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:07.805 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:07.827 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:07.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:07.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:07.929 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:07.950 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:07.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:07.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:07.968 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:41:07.968 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-09-13T14:41:07.968 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-09-13T14:41:07.968 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-13T14:41:07.968 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.968 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:112933 7.e 7:71539828:::benchmark_data_smithi102_227395_object118:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-09-13T14:41:07.968 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:28.169308+0000", 2024-09-13T14:41:07.968 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 219.78521151499999, 2024-09-13T14:41:07.968 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.67251101, 2024-09-13T14:41:07.968 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:41:07.969 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:07.969 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:41:07.969 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:41:07.969 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:41:07.969 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 112933 2024-09-13T14:41:07.969 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.969 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:41:07.969 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.969 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:41:07.969 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.169308+0000", 2024-09-13T14:41:07.969 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:07.969 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.969 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.969 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:41:07.970 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.169308+0000", 2024-09-13T14:41:07.970 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:07.970 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.970 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.970 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:41:07.970 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.169306+0000", 2024-09-13T14:41:07.970 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:41:07.970 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.970 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.970 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:41:07.970 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.169314+0000", 2024-09-13T14:41:07.970 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5399999999999998e-06 2024-09-13T14:41:07.970 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.971 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.971 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:41:07.971 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.169327+0000", 2024-09-13T14:41:07.971 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3589e-05 2024-09-13T14:41:07.971 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.971 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.971 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:41:07.971 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.169336+0000", 2024-09-13T14:41:07.971 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1599999999999998e-06 2024-09-13T14:41:07.971 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.971 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.971 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:07.971 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.036088+0000", 2024-09-13T14:41:07.972 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.86675195199999999 2024-09-13T14:41:07.972 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.972 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.972 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:41:07.972 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.056928+0000", 2024-09-13T14:41:07.972 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.020840121 2024-09-13T14:41:07.972 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.972 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.972 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:07.972 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.228620+0000", 2024-09-13T14:41:07.972 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.17169189700000001 2024-09-13T14:41:07.972 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.972 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.973 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:41:07.973 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.256794+0000", 2024-09-13T14:41:07.973 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.028174637999999998 2024-09-13T14:41:07.973 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.973 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.973 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:07.973 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.612105+0000", 2024-09-13T14:41:07.973 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3553110849999999 2024-09-13T14:41:07.973 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.974 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.974 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:41:07.974 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.625533+0000", 2024-09-13T14:41:07.974 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013427308000000001 2024-09-13T14:41:07.974 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.974 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.974 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:41:07.974 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.650011+0000", 2024-09-13T14:41:07.974 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024478033999999999 2024-09-13T14:41:07.974 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.974 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.974 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:07.974 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.779644+0000", 2024-09-13T14:41:07.974 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.12963343899999999 2024-09-13T14:41:07.975 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.975 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.975 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:41:07.975 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.781386+0000", 2024-09-13T14:41:07.975 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0017419709999999999 2024-09-13T14:41:07.975 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.975 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.975 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:07.975 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.786230+0000", 2024-09-13T14:41:07.975 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0048441839999999996 2024-09-13T14:41:07.975 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.975 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.975 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:41:07.975 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.792473+0000", 2024-09-13T14:41:07.976 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0062430649999999999 2024-09-13T14:41:07.976 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.976 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.976 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:41:07.976 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.841819+0000", 2024-09-13T14:41:07.976 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.049345554999999999 2024-09-13T14:41:07.976 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:07.976 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:41:07.976 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:07.976 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.976 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.976 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:112953 7.e 7:72a6485e:::benchmark_data_smithi102_227395_object138:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-09-13T14:41:07.976 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:28.219817+0000", 2024-09-13T14:41:07.977 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 219.7347025, 2024-09-13T14:41:07.977 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.688959026, 2024-09-13T14:41:07.977 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:41:07.977 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:07.977 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:41:07.977 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:41:07.977 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:41:07.977 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 112953 2024-09-13T14:41:07.977 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.977 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:41:07.977 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.977 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:41:07.977 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.219817+0000", 2024-09-13T14:41:07.978 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:07.978 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.978 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.978 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:41:07.978 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.219817+0000", 2024-09-13T14:41:07.978 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:07.978 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.978 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.978 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:41:07.978 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.219815+0000", 2024-09-13T14:41:07.978 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:41:07.978 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.978 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.978 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:41:07.979 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.219824+0000", 2024-09-13T14:41:07.979 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.3449999999999995e-06 2024-09-13T14:41:07.979 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.979 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.979 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:41:07.979 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.219841+0000", 2024-09-13T14:41:07.979 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6330000000000001e-05 2024-09-13T14:41:07.979 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.979 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.979 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:41:07.979 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.219850+0000", 2024-09-13T14:41:07.979 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.2280000000000004e-06 2024-09-13T14:41:07.979 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.979 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.980 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:07.980 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.266772+0000", 2024-09-13T14:41:07.980 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.046922012 2024-09-13T14:41:07.980 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.980 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.980 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:41:07.980 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.295174+0000", 2024-09-13T14:41:07.980 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.028402107999999999 2024-09-13T14:41:07.980 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.980 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.980 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:07.980 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.443017+0000", 2024-09-13T14:41:07.980 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.14784280499999999 2024-09-13T14:41:07.980 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.981 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.981 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:41:07.981 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.465867+0000", 2024-09-13T14:41:07.981 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.022849813 2024-09-13T14:41:07.981 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.981 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.981 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:07.981 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.721401+0000", 2024-09-13T14:41:07.981 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.255534258 2024-09-13T14:41:07.981 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.981 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.981 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:41:07.981 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.733248+0000", 2024-09-13T14:41:07.981 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.01184754 2024-09-13T14:41:07.982 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.982 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.982 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:41:07.982 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.757733+0000", 2024-09-13T14:41:07.982 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024485033 2024-09-13T14:41:07.982 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.982 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.982 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:07.982 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.847315+0000", 2024-09-13T14:41:07.982 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.089581220000000003 2024-09-13T14:41:07.982 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.982 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.982 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:41:07.983 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.849721+0000", 2024-09-13T14:41:07.983 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0024068840000000002 2024-09-13T14:41:07.983 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.983 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.983 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:07.983 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.856740+0000", 2024-09-13T14:41:07.983 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0070188029999999997 2024-09-13T14:41:07.983 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.983 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.983 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:41:07.983 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.867725+0000", 2024-09-13T14:41:07.983 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010984910000000001 2024-09-13T14:41:07.983 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.984 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.984 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:41:07.984 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.908776+0000", 2024-09-13T14:41:07.984 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.041050775999999997 2024-09-13T14:41:07.984 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:07.984 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:41:07.984 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:07.984 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.984 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.984 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:112957 7.e 7:73c81a1b:::benchmark_data_smithi102_227395_object142:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-09-13T14:41:07.984 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:28.238422+0000", 2024-09-13T14:41:07.984 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 219.71609705500001, 2024-09-13T14:41:07.984 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9574894330000001, 2024-09-13T14:41:07.985 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:41:07.985 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:07.985 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:41:07.985 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:41:07.985 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:41:07.985 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 112957 2024-09-13T14:41:07.985 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.985 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:41:07.985 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.985 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:41:07.985 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.238422+0000", 2024-09-13T14:41:07.985 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:07.985 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.986 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.986 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:41:07.986 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.238422+0000", 2024-09-13T14:41:07.986 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:07.986 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.986 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.986 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:41:07.986 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.238421+0000", 2024-09-13T14:41:07.986 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:41:07.986 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.986 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.986 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:41:07.986 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.238429+0000", 2024-09-13T14:41:07.986 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9349999999999994e-06 2024-09-13T14:41:07.987 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.987 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.987 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:41:07.987 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.238442+0000", 2024-09-13T14:41:07.987 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3501e-05 2024-09-13T14:41:07.987 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.987 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.987 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:41:07.987 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.238448+0000", 2024-09-13T14:41:07.987 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3759999999999996e-06 2024-09-13T14:41:07.987 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.987 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.987 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:07.988 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.297744+0000", 2024-09-13T14:41:07.988 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.059295036 2024-09-13T14:41:07.988 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.988 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.988 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:41:07.988 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.319171+0000", 2024-09-13T14:41:07.988 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.021427883000000002 2024-09-13T14:41:07.988 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.988 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.988 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:07.988 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.554554+0000", 2024-09-13T14:41:07.988 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.23538229799999999 2024-09-13T14:41:07.989 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.989 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.989 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:41:07.989 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.579299+0000", 2024-09-13T14:41:07.989 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024745277 2024-09-13T14:41:07.989 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.989 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.989 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:07.989 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.992160+0000", 2024-09-13T14:41:07.989 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4128609559999998 2024-09-13T14:41:07.989 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.989 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.989 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:41:07.989 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.011466+0000", 2024-09-13T14:41:07.990 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.019306104000000001 2024-09-13T14:41:07.990 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.990 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.990 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:41:07.990 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.039599+0000", 2024-09-13T14:41:07.990 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.028132766999999999 2024-09-13T14:41:07.990 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.990 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.990 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:07.991 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.138317+0000", 2024-09-13T14:41:07.991 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.098718599000000004 2024-09-13T14:41:07.991 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.991 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.991 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:41:07.991 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.139902+0000", 2024-09-13T14:41:07.991 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001584184 2024-09-13T14:41:07.991 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.991 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.991 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:07.991 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.148487+0000", 2024-09-13T14:41:07.991 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0085852440000000006 2024-09-13T14:41:07.991 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.992 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.992 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:41:07.992 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.156727+0000", 2024-09-13T14:41:07.992 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0082399269999999993 2024-09-13T14:41:07.992 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.992 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.992 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:41:07.992 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.195912+0000", 2024-09-13T14:41:07.992 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.039185042000000003 2024-09-13T14:41:07.992 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:07.992 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:41:07.992 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:07.992 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.993 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.993 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:112968 7.e 7:7cd0abb8:::benchmark_data_smithi102_227395_object153:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-09-13T14:41:07.993 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:28.367834+0000", 2024-09-13T14:41:07.993 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 219.586685225, 2024-09-13T14:41:07.993 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1335673269999997, 2024-09-13T14:41:07.993 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:41:07.993 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:07.993 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:41:07.993 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:41:07.993 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:41:07.993 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 112968 2024-09-13T14:41:07.993 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.993 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:41:07.994 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.994 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:41:07.994 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.367834+0000", 2024-09-13T14:41:07.994 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:07.994 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.994 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.994 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:41:07.994 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.367834+0000", 2024-09-13T14:41:07.994 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:07.994 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.994 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.994 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:41:07.994 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.367832+0000", 2024-09-13T14:41:07.994 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:41:07.995 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.995 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.995 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:41:07.995 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.367841+0000", 2024-09-13T14:41:07.995 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7539999999999995e-06 2024-09-13T14:41:07.995 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.995 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.995 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:41:07.995 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.367856+0000", 2024-09-13T14:41:07.995 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4499e-05 2024-09-13T14:41:07.995 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.995 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.995 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:41:07.996 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.367863+0000", 2024-09-13T14:41:07.996 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6690000000000006e-06 2024-09-13T14:41:07.996 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.996 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.996 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:07.996 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.321756+0000", 2024-09-13T14:41:07.996 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.95389310000000005 2024-09-13T14:41:07.996 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.996 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.996 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:41:07.996 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.346608+0000", 2024-09-13T14:41:07.996 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024851504 2024-09-13T14:41:07.996 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.997 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.997 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:07.997 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.658373+0000", 2024-09-13T14:41:07.997 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.31176493100000002 2024-09-13T14:41:07.997 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.997 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.997 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:41:07.997 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.677721+0000", 2024-09-13T14:41:07.997 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.019348260999999999 2024-09-13T14:41:07.997 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.997 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.997 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:07.997 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.244806+0000", 2024-09-13T14:41:07.998 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.567085134 2024-09-13T14:41:07.998 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.998 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.998 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:41:07.998 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.260918+0000", 2024-09-13T14:41:07.998 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016111721999999998 2024-09-13T14:41:07.998 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.998 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.998 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:41:07.998 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.285995+0000", 2024-09-13T14:41:07.998 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.025077351000000001 2024-09-13T14:41:07.998 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.998 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.998 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:07.999 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.451788+0000", 2024-09-13T14:41:07.999 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.16579242799999999 2024-09-13T14:41:07.999 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.999 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.999 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:41:07.999 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.453689+0000", 2024-09-13T14:41:07.999 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0019015049999999999 2024-09-13T14:41:07.999 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.999 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:07.999 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:07.999 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.459427+0000", 2024-09-13T14:41:07.999 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0057378819999999997 2024-09-13T14:41:07.999 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:07.999 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.000 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:41:08.000 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.465958+0000", 2024-09-13T14:41:08.000 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0065305709999999998 2024-09-13T14:41:08.000 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.000 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.000 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:41:08.000 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.501402+0000", 2024-09-13T14:41:08.000 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.035443745999999998 2024-09-13T14:41:08.000 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:08.000 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:41:08.000 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:08.000 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.000 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.001 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113005 7.e 7:78e9e6f9:::benchmark_data_smithi102_227395_object190:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-09-13T14:41:08.001 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:28.544298+0000", 2024-09-13T14:41:08.001 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 219.41022168999999, 2024-09-13T14:41:08.001 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0162806550000001, 2024-09-13T14:41:08.001 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:41:08.001 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:08.001 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:41:08.001 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:41:08.001 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:41:08.001 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113005 2024-09-13T14:41:08.001 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.001 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:41:08.001 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.001 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:41:08.002 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.544298+0000", 2024-09-13T14:41:08.002 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:08.002 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.002 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.002 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:41:08.002 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.544298+0000", 2024-09-13T14:41:08.002 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:08.002 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.002 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.002 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:41:08.002 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.544296+0000", 2024-09-13T14:41:08.002 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:41:08.002 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.003 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.003 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:41:08.003 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.544305+0000", 2024-09-13T14:41:08.003 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.0310000000000004e-06 2024-09-13T14:41:08.003 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.003 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.003 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:41:08.003 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.544318+0000", 2024-09-13T14:41:08.003 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3301e-05 2024-09-13T14:41:08.003 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.003 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.003 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:41:08.003 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.544325+0000", 2024-09-13T14:41:08.003 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9380000000000003e-06 2024-09-13T14:41:08.004 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.004 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.004 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:08.004 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.679520+0000", 2024-09-13T14:41:08.004 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1351946159999999 2024-09-13T14:41:08.004 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.004 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.004 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:41:08.004 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.697460+0000", 2024-09-13T14:41:08.004 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017939749000000001 2024-09-13T14:41:08.004 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.004 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.004 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:08.005 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.780770+0000", 2024-09-13T14:41:08.005 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.083310181999999997 2024-09-13T14:41:08.005 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.005 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.005 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:41:08.005 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.798851+0000", 2024-09-13T14:41:08.005 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018080978000000001 2024-09-13T14:41:08.005 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.005 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.005 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:08.005 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.382351+0000", 2024-09-13T14:41:08.005 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5834998100000002 2024-09-13T14:41:08.005 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.005 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.006 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:41:08.006 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.398282+0000", 2024-09-13T14:41:08.006 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.015931325999999999 2024-09-13T14:41:08.006 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.006 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.006 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:41:08.006 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.426371+0000", 2024-09-13T14:41:08.006 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.028089050000000001 2024-09-13T14:41:08.006 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.006 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.006 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:41:08.006 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.503108+0000", 2024-09-13T14:41:08.006 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.076737284000000003 2024-09-13T14:41:08.006 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.007 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.007 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:08.007 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.507418+0000", 2024-09-13T14:41:08.007 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0043095690000000001 2024-09-13T14:41:08.007 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.007 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.007 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:08.007 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.514013+0000", 2024-09-13T14:41:08.007 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0065951639999999997 2024-09-13T14:41:08.007 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.007 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.007 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:41:08.007 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.521065+0000", 2024-09-13T14:41:08.007 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.007051753 2024-09-13T14:41:08.008 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.008 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.008 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:41:08.008 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.560578+0000", 2024-09-13T14:41:08.008 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.039513581999999998 2024-09-13T14:41:08.008 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:08.008 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:41:08.008 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:08.008 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.008 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.008 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113646 7.e 7:7fda021e:::benchmark_data_smithi102_227395_object831:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-13T14:41:08.008 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:39.650939+0000", 2024-09-13T14:41:08.008 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 208.30357995099999, 2024-09-13T14:41:08.008 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4310964820000001, 2024-09-13T14:41:08.009 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:41:08.009 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:08.009 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:41:08.009 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:41:08.009 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:41:08.009 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113646 2024-09-13T14:41:08.009 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.009 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:41:08.009 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.009 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:41:08.009 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:39.650939+0000", 2024-09-13T14:41:08.009 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:08.009 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.010 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.010 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:41:08.010 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:39.650939+0000", 2024-09-13T14:41:08.010 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:08.010 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.010 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.010 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:41:08.010 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:39.650938+0000", 2024-09-13T14:41:08.010 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:41:08.010 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.010 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.010 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:41:08.010 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:39.650945+0000", 2024-09-13T14:41:08.010 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4200000000000001e-06 2024-09-13T14:41:08.011 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.011 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.011 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:41:08.011 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:39.650960+0000", 2024-09-13T14:41:08.011 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5028000000000001e-05 2024-09-13T14:41:08.011 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.011 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.011 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:41:08.011 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:39.650969+0000", 2024-09-13T14:41:08.011 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.4319999999999999e-06 2024-09-13T14:41:08.011 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.011 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.011 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:08.012 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:39.791532+0000", 2024-09-13T14:41:08.012 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.140563199 2024-09-13T14:41:08.012 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.012 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.012 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:41:08.012 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:39.808048+0000", 2024-09-13T14:41:08.012 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016515914999999999 2024-09-13T14:41:08.012 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.012 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.012 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:08.012 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:39.937277+0000", 2024-09-13T14:41:08.012 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.129229236 2024-09-13T14:41:08.012 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.012 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.013 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:41:08.013 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:39.953949+0000", 2024-09-13T14:41:08.013 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016672356999999999 2024-09-13T14:41:08.013 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.013 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.013 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:08.013 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.812544+0000", 2024-09-13T14:41:08.013 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8585950149999997 2024-09-13T14:41:08.013 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.013 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.013 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:41:08.013 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.829480+0000", 2024-09-13T14:41:08.013 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016936072999999999 2024-09-13T14:41:08.013 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.014 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.014 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:41:08.014 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.857149+0000", 2024-09-13T14:41:08.014 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.027668132000000002 2024-09-13T14:41:08.014 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.014 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.014 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:08.014 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.013996+0000", 2024-09-13T14:41:08.014 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.15684716000000001 2024-09-13T14:41:08.014 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.014 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.014 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:08.014 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.021795+0000", 2024-09-13T14:41:08.014 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0077991079999999999 2024-09-13T14:41:08.015 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.015 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.015 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:41:08.015 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.023954+0000", 2024-09-13T14:41:08.015 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0021587970000000001 2024-09-13T14:41:08.015 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.015 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.015 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:41:08.015 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.034566+0000", 2024-09-13T14:41:08.015 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010612244 2024-09-13T14:41:08.015 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.015 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.015 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:41:08.016 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.082036+0000", 2024-09-13T14:41:08.016 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.047470036 2024-09-13T14:41:08.016 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:08.016 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:41:08.016 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:08.016 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.016 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.016 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113660 7.e 7:72e391fe:::benchmark_data_smithi102_227395_object845:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-13T14:41:08.016 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:39.881206+0000", 2024-09-13T14:41:08.016 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 208.07331328699999, 2024-09-13T14:41:08.016 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2702958029999998, 2024-09-13T14:41:08.016 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:41:08.016 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:08.016 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:41:08.017 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:41:08.017 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:41:08.017 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113660 2024-09-13T14:41:08.017 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.017 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:41:08.017 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.017 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:41:08.017 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:39.881206+0000", 2024-09-13T14:41:08.017 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:08.017 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.017 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.017 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:41:08.017 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:39.881206+0000", 2024-09-13T14:41:08.017 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:08.018 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.018 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.018 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:41:08.018 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:39.881205+0000", 2024-09-13T14:41:08.018 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:41:08.018 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.018 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.018 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:41:08.018 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:39.881212+0000", 2024-09-13T14:41:08.018 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.17e-06 2024-09-13T14:41:08.018 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.018 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.018 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:41:08.018 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:39.881223+0000", 2024-09-13T14:41:08.019 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1214e-05 2024-09-13T14:41:08.019 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.019 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.019 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:41:08.019 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:39.881229+0000", 2024-09-13T14:41:08.019 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5559999999999998e-06 2024-09-13T14:41:08.019 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.019 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.019 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:08.019 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.072810+0000", 2024-09-13T14:41:08.019 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.19158088300000001 2024-09-13T14:41:08.019 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.019 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.020 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:41:08.020 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.098533+0000", 2024-09-13T14:41:08.020 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.025723314000000001 2024-09-13T14:41:08.020 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.020 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.020 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:08.020 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.225733+0000", 2024-09-13T14:41:08.020 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.12720013499999999 2024-09-13T14:41:08.020 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.020 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.020 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:41:08.020 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.243239+0000", 2024-09-13T14:41:08.020 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017505427 2024-09-13T14:41:08.020 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.021 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.021 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:08.021 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.929831+0000", 2024-09-13T14:41:08.021 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6865921210000003 2024-09-13T14:41:08.021 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.021 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.021 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:41:08.021 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.951285+0000", 2024-09-13T14:41:08.021 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.021454606000000001 2024-09-13T14:41:08.021 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.021 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.021 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:41:08.021 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.980812+0000", 2024-09-13T14:41:08.022 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.029526611000000001 2024-09-13T14:41:08.022 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.022 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.022 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:41:08.022 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.084340+0000", 2024-09-13T14:41:08.022 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.10352797699999999 2024-09-13T14:41:08.022 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.022 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.022 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:08.022 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.090949+0000", 2024-09-13T14:41:08.022 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0066088229999999998 2024-09-13T14:41:08.022 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.022 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.022 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:08.023 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.095550+0000", 2024-09-13T14:41:08.023 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0046012689999999998 2024-09-13T14:41:08.023 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.023 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.023 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:41:08.023 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.104748+0000", 2024-09-13T14:41:08.023 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0091979999999999996 2024-09-13T14:41:08.023 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.023 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.023 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:41:08.023 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.151502+0000", 2024-09-13T14:41:08.023 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.046753905999999998 2024-09-13T14:41:08.023 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:08.023 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:41:08.024 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:08.024 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.024 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.024 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113667 7.e 7:7011123e:::benchmark_data_smithi102_227395_object852:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-13T14:41:08.024 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:40.145165+0000", 2024-09-13T14:41:08.024 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 207.80935431500001, 2024-09-13T14:41:08.024 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2681343839999997, 2024-09-13T14:41:08.024 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:41:08.024 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:08.024 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:41:08.024 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:41:08.024 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:41:08.024 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113667 2024-09-13T14:41:08.024 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.025 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:41:08.026 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.026 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:41:08.026 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.145165+0000", 2024-09-13T14:41:08.026 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:08.026 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.026 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.026 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:41:08.027 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.145165+0000", 2024-09-13T14:41:08.027 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:08.027 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.027 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.027 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:41:08.027 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.145164+0000", 2024-09-13T14:41:08.027 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:41:08.027 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.027 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.027 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:41:08.027 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.145170+0000", 2024-09-13T14:41:08.027 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8379999999999998e-06 2024-09-13T14:41:08.027 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.028 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.028 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:41:08.028 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.145181+0000", 2024-09-13T14:41:08.028 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1521e-05 2024-09-13T14:41:08.028 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.028 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.028 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:41:08.028 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.145186+0000", 2024-09-13T14:41:08.028 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0200000000000002e-06 2024-09-13T14:41:08.028 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.028 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.028 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:08.028 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.244931+0000", 2024-09-13T14:41:08.029 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.099744219999999995 2024-09-13T14:41:08.029 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.029 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.029 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:41:08.029 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.263033+0000", 2024-09-13T14:41:08.029 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018102917 2024-09-13T14:41:08.029 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.029 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.029 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:08.029 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.467278+0000", 2024-09-13T14:41:08.029 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.20424453100000001 2024-09-13T14:41:08.029 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.029 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.029 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:41:08.030 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.487283+0000", 2024-09-13T14:41:08.030 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.020004724000000002 2024-09-13T14:41:08.030 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.030 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.030 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:08.030 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.223527+0000", 2024-09-13T14:41:08.030 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7362443320000001 2024-09-13T14:41:08.030 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.030 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.030 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:41:08.030 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.237432+0000", 2024-09-13T14:41:08.030 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013905244000000001 2024-09-13T14:41:08.030 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.031 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.031 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:41:08.031 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.264564+0000", 2024-09-13T14:41:08.031 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.027132177 2024-09-13T14:41:08.031 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.031 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.031 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:08.031 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.369627+0000", 2024-09-13T14:41:08.031 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.10506285 2024-09-13T14:41:08.031 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.031 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.031 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:41:08.031 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.370616+0000", 2024-09-13T14:41:08.032 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00098851600000000005 2024-09-13T14:41:08.032 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.032 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.032 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:08.032 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.376243+0000", 2024-09-13T14:41:08.032 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0056276249999999998 2024-09-13T14:41:08.032 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.032 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.032 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:41:08.032 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.381212+0000", 2024-09-13T14:41:08.032 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0049688850000000001 2024-09-13T14:41:08.032 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.032 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.033 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:41:08.033 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.413299+0000", 2024-09-13T14:41:08.033 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.032087169999999998 2024-09-13T14:41:08.033 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:08.033 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:41:08.033 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:08.033 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.033 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.033 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113673 7.e 7:707c08ea:::benchmark_data_smithi102_227395_object858:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-13T14:41:08.033 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:40.852904+0000", 2024-09-13T14:41:08.033 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 207.10161583999999, 2024-09-13T14:41:08.034 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0436350230000002, 2024-09-13T14:41:08.034 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:41:08.034 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:08.034 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:41:08.034 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:41:08.034 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:41:08.034 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113673 2024-09-13T14:41:08.034 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.034 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:41:08.035 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.035 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:41:08.035 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.852904+0000", 2024-09-13T14:41:08.035 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:08.035 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.035 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.035 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:41:08.035 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.852904+0000", 2024-09-13T14:41:08.035 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:08.035 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.035 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.035 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:41:08.035 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.852902+0000", 2024-09-13T14:41:08.035 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:41:08.036 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.036 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.036 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:41:08.036 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.852909+0000", 2024-09-13T14:41:08.036 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8900000000000001e-06 2024-09-13T14:41:08.036 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.036 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.036 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:41:08.036 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.852927+0000", 2024-09-13T14:41:08.036 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8281999999999999e-05 2024-09-13T14:41:08.036 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.036 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.037 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:41:08.037 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.852933+0000", 2024-09-13T14:41:08.037 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0220000000000001e-06 2024-09-13T14:41:08.037 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.037 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.037 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:08.037 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.932659+0000", 2024-09-13T14:41:08.037 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.079725349000000001 2024-09-13T14:41:08.037 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.037 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.037 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:41:08.037 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.961885+0000", 2024-09-13T14:41:08.037 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.029226683999999999 2024-09-13T14:41:08.037 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.038 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.038 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:08.038 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.190540+0000", 2024-09-13T14:41:08.038 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.22865431899999999 2024-09-13T14:41:08.038 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.038 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.038 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:41:08.038 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.213124+0000", 2024-09-13T14:41:08.038 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.022584453000000001 2024-09-13T14:41:08.038 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.038 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.038 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:08.038 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.694534+0000", 2024-09-13T14:41:08.038 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4814095549999999 2024-09-13T14:41:08.039 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.039 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.039 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:41:08.039 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.708113+0000", 2024-09-13T14:41:08.039 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013578961000000001 2024-09-13T14:41:08.039 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.039 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.039 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:41:08.039 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.734818+0000", 2024-09-13T14:41:08.039 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.026704998000000001 2024-09-13T14:41:08.039 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.039 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.039 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:08.040 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.843296+0000", 2024-09-13T14:41:08.040 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.10847884000000001 2024-09-13T14:41:08.040 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.040 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.040 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:41:08.040 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.844461+0000", 2024-09-13T14:41:08.040 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001164111 2024-09-13T14:41:08.040 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.040 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.040 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:08.040 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.853549+0000", 2024-09-13T14:41:08.040 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0090885069999999991 2024-09-13T14:41:08.040 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.040 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.041 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:41:08.041 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.859381+0000", 2024-09-13T14:41:08.041 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.005832266 2024-09-13T14:41:08.041 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.041 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.041 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:41:08.041 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.896539+0000", 2024-09-13T14:41:08.041 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.037157218999999998 2024-09-13T14:41:08.041 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:08.041 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:41:08.041 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:08.041 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.041 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.042 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDPGPush(7.e 236/222 [PushOp(7:765a5df7:::benchmark_data_smithi102_227395_object908:head, version: 209'56, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:765a5df7:::benchmark_data_smithi102_227395_object908:head@209'56, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T14:41:08.042 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:41.327036+0000", 2024-09-13T14:41:08.042 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 206.627483103, 2024-09-13T14:41:08.042 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4911342149999998, 2024-09-13T14:41:08.042 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:41:08.042 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T14:41:08.042 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:41:08.042 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.042 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:41:08.042 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.327036+0000", 2024-09-13T14:41:08.042 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:08.042 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.042 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.043 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:41:08.043 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.327036+0000", 2024-09-13T14:41:08.043 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:08.043 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.043 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.043 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:41:08.043 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.327035+0000", 2024-09-13T14:41:08.043 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:41:08.043 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.043 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.043 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:41:08.043 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.327094+0000", 2024-09-13T14:41:08.043 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8471999999999999e-05 2024-09-13T14:41:08.043 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.044 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.044 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:41:08.044 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.327110+0000", 2024-09-13T14:41:08.044 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6333000000000001e-05 2024-09-13T14:41:08.044 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.045 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.045 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:41:08.045 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.327125+0000", 2024-09-13T14:41:08.045 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4766999999999999e-05 2024-09-13T14:41:08.045 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.045 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.045 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:08.045 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.756102+0000", 2024-09-13T14:41:08.045 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4289765430000001 2024-09-13T14:41:08.045 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.046 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.046 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:41:08.046 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.757954+0000", 2024-09-13T14:41:08.046 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0018523560000000001 2024-09-13T14:41:08.046 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.046 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.046 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:41:08.046 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.818171+0000", 2024-09-13T14:41:08.046 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.060216609999999997 2024-09-13T14:41:08.046 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:08.046 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:41:08.047 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:08.047 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.047 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.047 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113723 7.e 7:765a5df7:::benchmark_data_smithi102_227395_object908:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-13T14:41:08.047 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:41.735065+0000", 2024-09-13T14:41:08.047 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 206.21945439300001, 2024-09-13T14:41:08.047 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6293769879999997, 2024-09-13T14:41:08.047 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:41:08.047 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:08.047 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:41:08.048 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:41:08.048 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:41:08.048 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113723 2024-09-13T14:41:08.048 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.048 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:41:08.048 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.048 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:41:08.048 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.735065+0000", 2024-09-13T14:41:08.048 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:08.048 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.048 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.048 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:41:08.048 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.735065+0000", 2024-09-13T14:41:08.049 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:08.049 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.049 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.049 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:41:08.049 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.735064+0000", 2024-09-13T14:41:08.049 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:41:08.049 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.049 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.049 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:41:08.049 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.735071+0000", 2024-09-13T14:41:08.049 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2289999999999997e-06 2024-09-13T14:41:08.049 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.049 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.050 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:41:08.050 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.735085+0000", 2024-09-13T14:41:08.050 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4436e-05 2024-09-13T14:41:08.050 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.050 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.050 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:41:08.050 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.735092+0000", 2024-09-13T14:41:08.050 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.216e-06 2024-09-13T14:41:08.051 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.051 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.051 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:08.051 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.016651+0000", 2024-09-13T14:41:08.051 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.28155898499999998 2024-09-13T14:41:08.051 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.051 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.051 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:41:08.051 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.027127+0000", 2024-09-13T14:41:08.051 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010475241999999999 2024-09-13T14:41:08.051 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.051 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.052 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:08.052 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.819782+0000", 2024-09-13T14:41:08.052 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7926552459999998 2024-09-13T14:41:08.052 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.052 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.052 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:41:08.052 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.838097+0000", 2024-09-13T14:41:08.052 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018315084999999998 2024-09-13T14:41:08.052 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.052 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.052 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:08.053 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.103166+0000", 2024-09-13T14:41:08.053 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2650688400000001 2024-09-13T14:41:08.053 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.053 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.053 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:41:08.053 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.113423+0000", 2024-09-13T14:41:08.053 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010256919 2024-09-13T14:41:08.053 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.053 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.053 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:41:08.053 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.159715+0000", 2024-09-13T14:41:08.053 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.046292424999999998 2024-09-13T14:41:08.053 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.054 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.054 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:41:08.054 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.242694+0000", 2024-09-13T14:41:08.054 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.082978417999999998 2024-09-13T14:41:08.054 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.054 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.054 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:08.054 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.310555+0000", 2024-09-13T14:41:08.054 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.067861006000000001 2024-09-13T14:41:08.054 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.054 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.054 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:08.054 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.316591+0000", 2024-09-13T14:41:08.054 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0060365549999999999 2024-09-13T14:41:08.055 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.055 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.055 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:41:08.055 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.328028+0000", 2024-09-13T14:41:08.055 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.01143724 2024-09-13T14:41:08.055 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.055 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.055 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:41:08.055 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.364442+0000", 2024-09-13T14:41:08.055 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.036413661 2024-09-13T14:41:08.055 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:08.055 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:41:08.055 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:08.055 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.056 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.056 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113733 7.e 7:71f1f434:::benchmark_data_smithi102_227395_object918:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-13T14:41:08.056 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:41.768799+0000", 2024-09-13T14:41:08.056 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 206.18572085400001, 2024-09-13T14:41:08.056 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4105988570000001, 2024-09-13T14:41:08.056 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:41:08.056 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:08.056 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:41:08.056 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:41:08.056 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:41:08.056 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113733 2024-09-13T14:41:08.056 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.057 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:41:08.057 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.057 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:41:08.057 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.768799+0000", 2024-09-13T14:41:08.057 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:08.057 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.057 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.057 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:41:08.057 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.768799+0000", 2024-09-13T14:41:08.057 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:08.057 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.057 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.057 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:41:08.057 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.768797+0000", 2024-09-13T14:41:08.058 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:41:08.058 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.058 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.058 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:41:08.058 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.768805+0000", 2024-09-13T14:41:08.058 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1880000000000002e-06 2024-09-13T14:41:08.058 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.058 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.058 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:41:08.058 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.768818+0000", 2024-09-13T14:41:08.058 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3322999999999999e-05 2024-09-13T14:41:08.058 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.058 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.058 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:41:08.059 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.768826+0000", 2024-09-13T14:41:08.059 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5070000000000003e-06 2024-09-13T14:41:08.059 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.059 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.059 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:08.059 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.029999+0000", 2024-09-13T14:41:08.059 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.261173402 2024-09-13T14:41:08.059 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.059 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.059 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:41:08.059 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.071555+0000", 2024-09-13T14:41:08.059 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.041556211000000003 2024-09-13T14:41:08.059 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.059 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.060 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:08.060 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.202418+0000", 2024-09-13T14:41:08.060 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.13086251099999999 2024-09-13T14:41:08.060 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.060 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.060 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:41:08.060 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.224452+0000", 2024-09-13T14:41:08.060 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.022033740999999999 2024-09-13T14:41:08.060 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.060 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.060 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:08.060 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.962029+0000", 2024-09-13T14:41:08.060 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7375775129999997 2024-09-13T14:41:08.061 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.061 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.061 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:41:08.061 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.976076+0000", 2024-09-13T14:41:08.061 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014046856999999999 2024-09-13T14:41:08.061 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.061 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.061 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:41:08.061 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.000274+0000", 2024-09-13T14:41:08.061 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024197749000000001 2024-09-13T14:41:08.061 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.061 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.061 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:08.061 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.099261+0000", 2024-09-13T14:41:08.062 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.098987027000000005 2024-09-13T14:41:08.062 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.062 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.062 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:41:08.062 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.101138+0000", 2024-09-13T14:41:08.062 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0018768680000000001 2024-09-13T14:41:08.062 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.062 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.062 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:08.062 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.111923+0000", 2024-09-13T14:41:08.062 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010785444999999999 2024-09-13T14:41:08.062 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.062 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.062 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:41:08.062 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.124357+0000", 2024-09-13T14:41:08.063 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012433962999999999 2024-09-13T14:41:08.063 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.063 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.063 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:41:08.063 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.179397+0000", 2024-09-13T14:41:08.063 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.055040402000000002 2024-09-13T14:41:08.063 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:08.063 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:41:08.063 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:08.063 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.063 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.063 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113763 7.e 7:7a44c088:::benchmark_data_smithi102_227395_object948:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-13T14:41:08.063 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:41.987161+0000", 2024-09-13T14:41:08.064 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 205.96735863699999, 2024-09-13T14:41:08.064 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.450241471, 2024-09-13T14:41:08.064 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:41:08.064 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:08.064 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:41:08.064 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:41:08.064 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:41:08.064 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113763 2024-09-13T14:41:08.064 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.064 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:41:08.064 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.064 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:41:08.064 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.987161+0000", 2024-09-13T14:41:08.065 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:08.065 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.065 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.065 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:41:08.065 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.987161+0000", 2024-09-13T14:41:08.065 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:08.065 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.065 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.065 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:41:08.065 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.987159+0000", 2024-09-13T14:41:08.065 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:41:08.065 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.065 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.065 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:41:08.066 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.987166+0000", 2024-09-13T14:41:08.066 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5080000000000002e-06 2024-09-13T14:41:08.066 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.066 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.066 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:41:08.066 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.987178+0000", 2024-09-13T14:41:08.066 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2441e-05 2024-09-13T14:41:08.066 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.066 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.066 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:41:08.066 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.987184+0000", 2024-09-13T14:41:08.066 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5879999999999997e-06 2024-09-13T14:41:08.066 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.066 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.067 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:08.067 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.073662+0000", 2024-09-13T14:41:08.067 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.086478476999999998 2024-09-13T14:41:08.067 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.067 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.067 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:41:08.067 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.099105+0000", 2024-09-13T14:41:08.067 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.025442959000000001 2024-09-13T14:41:08.067 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.067 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.067 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:08.067 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.302626+0000", 2024-09-13T14:41:08.067 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.203520283 2024-09-13T14:41:08.067 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.067 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.068 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:41:08.068 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.324932+0000", 2024-09-13T14:41:08.068 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.022305898000000001 2024-09-13T14:41:08.068 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.068 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.068 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:08.068 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.224175+0000", 2024-09-13T14:41:08.068 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8992434930000002 2024-09-13T14:41:08.068 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.068 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.068 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:41:08.068 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.241799+0000", 2024-09-13T14:41:08.068 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017623608999999998 2024-09-13T14:41:08.069 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.069 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.069 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:41:08.069 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.274010+0000", 2024-09-13T14:41:08.069 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.032211245999999999 2024-09-13T14:41:08.069 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.069 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.069 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:08.069 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.364856+0000", 2024-09-13T14:41:08.069 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.090845578999999996 2024-09-13T14:41:08.069 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.069 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.070 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:08.070 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.372367+0000", 2024-09-13T14:41:08.070 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0075111149999999996 2024-09-13T14:41:08.070 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.070 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.070 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:41:08.070 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.374485+0000", 2024-09-13T14:41:08.070 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0021184810000000002 2024-09-13T14:41:08.070 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.070 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.070 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:41:08.070 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.386174+0000", 2024-09-13T14:41:08.071 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011689181999999999 2024-09-13T14:41:08.071 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.071 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.071 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:41:08.071 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.437402+0000", 2024-09-13T14:41:08.071 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.051227946000000003 2024-09-13T14:41:08.071 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:08.071 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:41:08.071 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:08.071 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.071 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.071 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113767 7.e 7:78506392:::benchmark_data_smithi102_227395_object952:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-13T14:41:08.071 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:42.006143+0000", 2024-09-13T14:41:08.071 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 205.94837679599999, 2024-09-13T14:41:08.072 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0048697080000002, 2024-09-13T14:41:08.072 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:41:08.072 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:08.072 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:41:08.072 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:41:08.072 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:41:08.072 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113767 2024-09-13T14:41:08.072 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.072 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:41:08.072 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.072 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:41:08.072 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.006143+0000", 2024-09-13T14:41:08.072 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:08.072 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.073 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.073 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:41:08.073 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.006143+0000", 2024-09-13T14:41:08.073 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:08.073 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.073 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.073 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:41:08.073 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.006141+0000", 2024-09-13T14:41:08.073 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:41:08.073 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.073 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.073 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:41:08.073 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.006149+0000", 2024-09-13T14:41:08.074 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8699999999999992e-06 2024-09-13T14:41:08.074 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.074 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.074 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:41:08.074 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.006161+0000", 2024-09-13T14:41:08.074 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1939999999999999e-05 2024-09-13T14:41:08.074 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.074 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.074 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:41:08.074 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.006168+0000", 2024-09-13T14:41:08.074 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4390000000000003e-06 2024-09-13T14:41:08.074 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.074 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.075 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:08.075 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.101455+0000", 2024-09-13T14:41:08.075 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.095286518000000001 2024-09-13T14:41:08.075 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.075 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.075 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:41:08.075 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.125347+0000", 2024-09-13T14:41:08.075 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.023891539999999999 2024-09-13T14:41:08.075 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.075 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.075 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:08.075 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.397577+0000", 2024-09-13T14:41:08.075 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.27223060100000002 2024-09-13T14:41:08.075 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.076 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.076 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:41:08.076 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.417123+0000", 2024-09-13T14:41:08.076 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.019546392999999999 2024-09-13T14:41:08.076 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.076 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.076 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:08.076 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.791931+0000", 2024-09-13T14:41:08.076 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3748072650000003 2024-09-13T14:41:08.076 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.076 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.076 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:41:08.076 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.809411+0000", 2024-09-13T14:41:08.077 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017480623000000001 2024-09-13T14:41:08.077 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.077 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.077 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:41:08.077 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.831143+0000", 2024-09-13T14:41:08.077 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.021731145 2024-09-13T14:41:08.077 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.077 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.077 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:08.077 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.959387+0000", 2024-09-13T14:41:08.077 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.12824418500000001 2024-09-13T14:41:08.078 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.078 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.078 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:08.078 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.964317+0000", 2024-09-13T14:41:08.078 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0049304350000000004 2024-09-13T14:41:08.078 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.078 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.078 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:41:08.078 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.966146+0000", 2024-09-13T14:41:08.078 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001829041 2024-09-13T14:41:08.078 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.078 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.078 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:41:08.079 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.972871+0000", 2024-09-13T14:41:08.079 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0067250080000000002 2024-09-13T14:41:08.079 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.079 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.079 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:41:08.079 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.011012+0000", 2024-09-13T14:41:08.079 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.038141126999999997 2024-09-13T14:41:08.079 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:08.079 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:41:08.079 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:08.079 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.079 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.079 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113770 7.e 7:7f1c5c68:::benchmark_data_smithi102_227395_object955:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-13T14:41:08.080 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:42.134010+0000", 2024-09-13T14:41:08.080 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 205.820509789, 2024-09-13T14:41:08.080 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9464087430000001, 2024-09-13T14:41:08.080 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:41:08.080 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:08.080 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:41:08.080 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:41:08.080 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:41:08.080 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113770 2024-09-13T14:41:08.080 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.080 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:41:08.080 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.080 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:41:08.080 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.134010+0000", 2024-09-13T14:41:08.081 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:08.081 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.081 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.081 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:41:08.081 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.134010+0000", 2024-09-13T14:41:08.081 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:08.081 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.081 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.081 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:41:08.081 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.134008+0000", 2024-09-13T14:41:08.081 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:41:08.081 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.081 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.081 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:41:08.082 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.134019+0000", 2024-09-13T14:41:08.082 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0458e-05 2024-09-13T14:41:08.082 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.082 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.082 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:41:08.082 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.134030+0000", 2024-09-13T14:41:08.082 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1311999999999999e-05 2024-09-13T14:41:08.082 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.082 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.082 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:41:08.082 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.134036+0000", 2024-09-13T14:41:08.082 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0229999999999998e-06 2024-09-13T14:41:08.082 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.083 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.083 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:08.083 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.419198+0000", 2024-09-13T14:41:08.083 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.28516167399999998 2024-09-13T14:41:08.083 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.083 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.083 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:41:08.083 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.440868+0000", 2024-09-13T14:41:08.083 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.021670779000000001 2024-09-13T14:41:08.083 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.083 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.083 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:08.083 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.554028+0000", 2024-09-13T14:41:08.084 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.113159653 2024-09-13T14:41:08.084 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.084 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.084 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:41:08.084 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.576302+0000", 2024-09-13T14:41:08.084 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.022274200000000001 2024-09-13T14:41:08.084 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.084 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.084 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:08.084 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.890592+0000", 2024-09-13T14:41:08.084 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3142900390000003 2024-09-13T14:41:08.085 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.085 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.085 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:41:08.085 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.904275+0000", 2024-09-13T14:41:08.085 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013682952 2024-09-13T14:41:08.085 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.085 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.085 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:41:08.085 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.931383+0000", 2024-09-13T14:41:08.085 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.027107368 2024-09-13T14:41:08.085 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.085 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.086 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:41:08.086 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.012801+0000", 2024-09-13T14:41:08.086 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.081418328999999998 2024-09-13T14:41:08.086 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.086 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.086 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:08.086 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.018242+0000", 2024-09-13T14:41:08.086 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0054409819999999996 2024-09-13T14:41:08.086 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.086 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.087 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:08.087 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.025406+0000", 2024-09-13T14:41:08.087 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0071640150000000001 2024-09-13T14:41:08.087 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.087 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.087 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:41:08.087 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.036309+0000", 2024-09-13T14:41:08.087 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010902949 2024-09-13T14:41:08.087 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.087 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.087 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:41:08.087 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.080418+0000", 2024-09-13T14:41:08.087 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.044109451000000001 2024-09-13T14:41:08.087 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:08.088 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:41:08.088 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:08.088 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.088 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.088 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113800 7.e 7:745cc90f:::benchmark_data_smithi102_227395_object985:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-13T14:41:08.088 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:42.931912+0000", 2024-09-13T14:41:08.088 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 205.02260764299999, 2024-09-13T14:41:08.088 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3762813339999997, 2024-09-13T14:41:08.088 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:41:08.088 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:08.088 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:41:08.088 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:41:08.088 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:41:08.088 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113800 2024-09-13T14:41:08.089 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.089 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:41:08.089 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.089 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:41:08.089 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.931912+0000", 2024-09-13T14:41:08.089 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:08.089 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.089 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.089 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:41:08.089 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.931912+0000", 2024-09-13T14:41:08.089 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:08.089 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.089 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.090 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:41:08.090 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.931910+0000", 2024-09-13T14:41:08.090 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:41:08.090 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.090 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.090 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:41:08.090 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.931924+0000", 2024-09-13T14:41:08.090 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3915e-05 2024-09-13T14:41:08.090 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.090 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.090 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:41:08.090 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.931937+0000", 2024-09-13T14:41:08.091 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2835e-05 2024-09-13T14:41:08.091 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.091 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.091 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:41:08.091 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.931942+0000", 2024-09-13T14:41:08.091 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.181e-06 2024-09-13T14:41:08.091 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.091 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.091 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:08.091 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:43.100264+0000", 2024-09-13T14:41:08.091 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.16832238599999999 2024-09-13T14:41:08.091 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.091 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.091 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:41:08.091 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:43.124629+0000", 2024-09-13T14:41:08.092 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024364495 2024-09-13T14:41:08.092 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.092 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.092 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:08.092 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:43.260475+0000", 2024-09-13T14:41:08.092 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.13584602100000001 2024-09-13T14:41:08.092 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.092 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.092 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:41:08.092 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:43.281800+0000", 2024-09-13T14:41:08.092 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.021324864999999998 2024-09-13T14:41:08.092 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.092 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.093 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:08.093 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.133937+0000", 2024-09-13T14:41:08.093 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8521372940000003 2024-09-13T14:41:08.093 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.093 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.093 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:41:08.093 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.152762+0000", 2024-09-13T14:41:08.093 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018825346 2024-09-13T14:41:08.093 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.093 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.093 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:41:08.093 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.167553+0000", 2024-09-13T14:41:08.093 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014790905 2024-09-13T14:41:08.094 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.094 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.094 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:41:08.094 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.272976+0000", 2024-09-13T14:41:08.094 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.105422497 2024-09-13T14:41:08.094 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.094 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.094 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:08.094 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.275989+0000", 2024-09-13T14:41:08.094 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0030131899999999998 2024-09-13T14:41:08.094 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.094 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.094 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:08.095 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.279778+0000", 2024-09-13T14:41:08.095 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0037891349999999999 2024-09-13T14:41:08.095 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.095 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.095 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:41:08.095 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.285756+0000", 2024-09-13T14:41:08.095 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0059774989999999998 2024-09-13T14:41:08.095 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.095 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.095 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:41:08.095 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.308193+0000", 2024-09-13T14:41:08.095 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.022437436000000002 2024-09-13T14:41:08.095 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:08.096 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:41:08.096 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:08.096 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.096 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.096 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113838 7.e 7:725b9897:::benchmark_data_smithi102_227395_object1023:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-13T14:41:08.096 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:43.215448+0000", 2024-09-13T14:41:08.096 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 204.739070946, 2024-09-13T14:41:08.096 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1232826060000001, 2024-09-13T14:41:08.096 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:41:08.096 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:08.096 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:41:08.096 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:41:08.097 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:41:08.097 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113838 2024-09-13T14:41:08.097 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.097 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:41:08.097 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.097 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:41:08.097 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:43.215448+0000", 2024-09-13T14:41:08.097 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:08.097 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.097 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.097 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:41:08.097 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:43.215448+0000", 2024-09-13T14:41:08.097 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:08.098 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.098 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.098 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:41:08.098 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:43.215447+0000", 2024-09-13T14:41:08.098 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:41:08.098 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.098 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.098 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:41:08.098 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:43.215455+0000", 2024-09-13T14:41:08.098 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8089999999999993e-06 2024-09-13T14:41:08.098 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.098 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.098 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:41:08.099 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:43.215469+0000", 2024-09-13T14:41:08.099 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4056000000000001e-05 2024-09-13T14:41:08.099 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.099 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.099 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:41:08.099 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:43.215476+0000", 2024-09-13T14:41:08.099 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3749999999999997e-06 2024-09-13T14:41:08.099 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.099 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.099 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:08.099 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:43.413654+0000", 2024-09-13T14:41:08.099 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.19817833800000001 2024-09-13T14:41:08.099 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.100 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.100 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:41:08.100 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:43.438154+0000", 2024-09-13T14:41:08.100 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024499441 2024-09-13T14:41:08.100 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.100 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.101 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:08.101 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:43.627551+0000", 2024-09-13T14:41:08.101 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.189397542 2024-09-13T14:41:08.101 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.101 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.101 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:41:08.101 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:43.651356+0000", 2024-09-13T14:41:08.101 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.023804698999999999 2024-09-13T14:41:08.101 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.101 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.101 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:08.101 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.210574+0000", 2024-09-13T14:41:08.102 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5592174459999999 2024-09-13T14:41:08.102 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.102 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.102 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:41:08.102 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.219313+0000", 2024-09-13T14:41:08.102 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0087393239999999997 2024-09-13T14:41:08.102 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.102 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.102 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:41:08.102 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.233331+0000", 2024-09-13T14:41:08.102 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014018256999999999 2024-09-13T14:41:08.102 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.103 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.103 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:41:08.103 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.309091+0000", 2024-09-13T14:41:08.103 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.075759601999999995 2024-09-13T14:41:08.103 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.103 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.103 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:08.103 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.311847+0000", 2024-09-13T14:41:08.103 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0027565770000000001 2024-09-13T14:41:08.103 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.103 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.103 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:08.103 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.315219+0000", 2024-09-13T14:41:08.104 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0033721850000000002 2024-09-13T14:41:08.104 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.104 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.104 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:41:08.104 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.319648+0000", 2024-09-13T14:41:08.104 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0044280630000000003 2024-09-13T14:41:08.104 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.104 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.104 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:41:08.104 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.338731+0000", 2024-09-13T14:41:08.104 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.019083547999999999 2024-09-13T14:41:08.104 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:08.104 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:41:08.105 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:08.105 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.105 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.105 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113868 7.e 7:729791b6:::benchmark_data_smithi102_227395_object1053:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-13T14:41:08.105 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:44.019108+0000", 2024-09-13T14:41:08.105 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 203.93541156800001, 2024-09-13T14:41:08.105 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6053882540000002, 2024-09-13T14:41:08.105 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:41:08.105 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:08.105 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:41:08.105 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:41:08.105 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:41:08.105 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113868 2024-09-13T14:41:08.106 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.106 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:41:08.106 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.106 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:41:08.106 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.019108+0000", 2024-09-13T14:41:08.106 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:08.106 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.106 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.106 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:41:08.106 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.019108+0000", 2024-09-13T14:41:08.106 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:08.106 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.106 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.107 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:41:08.107 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.019107+0000", 2024-09-13T14:41:08.107 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:41:08.107 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.107 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.107 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:41:08.107 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.019112+0000", 2024-09-13T14:41:08.107 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7239999999999997e-06 2024-09-13T14:41:08.107 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.107 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.107 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:41:08.107 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.019122+0000", 2024-09-13T14:41:08.107 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6539999999999994e-06 2024-09-13T14:41:08.107 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.108 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.108 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:41:08.108 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.019128+0000", 2024-09-13T14:41:08.108 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8749999999999997e-06 2024-09-13T14:41:08.108 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.108 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.108 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:08.108 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.114948+0000", 2024-09-13T14:41:08.108 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.095819748999999996 2024-09-13T14:41:08.108 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.108 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.108 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:41:08.109 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.144054+0000", 2024-09-13T14:41:08.109 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.029106476999999999 2024-09-13T14:41:08.109 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.109 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.109 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:08.109 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.443107+0000", 2024-09-13T14:41:08.109 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.29905287200000003 2024-09-13T14:41:08.109 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.109 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.109 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:41:08.109 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.470900+0000", 2024-09-13T14:41:08.109 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.027792595999999999 2024-09-13T14:41:08.109 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.109 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.110 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:08.110 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.424154+0000", 2024-09-13T14:41:08.110 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9532546819999999 2024-09-13T14:41:08.110 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.110 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.110 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:41:08.110 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.431994+0000", 2024-09-13T14:41:08.110 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0078397650000000003 2024-09-13T14:41:08.110 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.110 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.110 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:41:08.110 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.445713+0000", 2024-09-13T14:41:08.110 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013719080999999999 2024-09-13T14:41:08.110 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.111 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.111 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:08.111 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.463051+0000", 2024-09-13T14:41:08.111 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017337680000000001 2024-09-13T14:41:08.111 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.111 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.111 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:08.111 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.465649+0000", 2024-09-13T14:41:08.111 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002598684 2024-09-13T14:41:08.111 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.111 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.111 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:41:08.111 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.600753+0000", 2024-09-13T14:41:08.111 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.13510388200000001 2024-09-13T14:41:08.112 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.112 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.112 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:41:08.112 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.605058+0000", 2024-09-13T14:41:08.112 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0043048879999999998 2024-09-13T14:41:08.112 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.112 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.112 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:41:08.112 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.624496+0000", 2024-09-13T14:41:08.112 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.019437904999999998 2024-09-13T14:41:08.112 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:08.112 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:41:08.112 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:08.113 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.113 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.113 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113882 7.e 7:73c217af:::benchmark_data_smithi102_227395_object1067:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-13T14:41:08.113 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:44.061776+0000", 2024-09-13T14:41:08.113 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 203.89274323999999, 2024-09-13T14:41:08.113 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.59134807, 2024-09-13T14:41:08.113 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:41:08.113 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:08.113 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:41:08.113 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:41:08.113 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:41:08.113 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113882 2024-09-13T14:41:08.113 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.114 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:41:08.114 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.114 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:41:08.114 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.061776+0000", 2024-09-13T14:41:08.114 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:08.114 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.114 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.114 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:41:08.114 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.061776+0000", 2024-09-13T14:41:08.114 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:08.114 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.114 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.114 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:41:08.115 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.061775+0000", 2024-09-13T14:41:08.115 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:41:08.115 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.115 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.115 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:41:08.115 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.061781+0000", 2024-09-13T14:41:08.115 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0839999999999997e-06 2024-09-13T14:41:08.115 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.115 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.115 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:41:08.115 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.061793+0000", 2024-09-13T14:41:08.115 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1939e-05 2024-09-13T14:41:08.115 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.116 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.116 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:41:08.116 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.061801+0000", 2024-09-13T14:41:08.116 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2490000000000001e-06 2024-09-13T14:41:08.116 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.116 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.116 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:08.116 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.146275+0000", 2024-09-13T14:41:08.116 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.084473626999999996 2024-09-13T14:41:08.116 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.116 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.116 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:41:08.116 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.170652+0000", 2024-09-13T14:41:08.117 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024377374 2024-09-13T14:41:08.117 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.117 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.117 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:08.117 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.548899+0000", 2024-09-13T14:41:08.117 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.37824670900000001 2024-09-13T14:41:08.117 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.117 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.117 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:41:08.117 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.610957+0000", 2024-09-13T14:41:08.117 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.062058073999999998 2024-09-13T14:41:08.117 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.117 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.117 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:08.118 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.488800+0000", 2024-09-13T14:41:08.118 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8778428460000001 2024-09-13T14:41:08.118 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.118 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.118 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:41:08.118 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.496393+0000", 2024-09-13T14:41:08.118 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0075932910000000003 2024-09-13T14:41:08.118 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.118 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.118 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:41:08.118 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.509330+0000", 2024-09-13T14:41:08.118 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012937051999999999 2024-09-13T14:41:08.118 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.119 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.119 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:41:08.119 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.625334+0000", 2024-09-13T14:41:08.119 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.116003785 2024-09-13T14:41:08.119 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.119 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.119 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:08.119 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.627836+0000", 2024-09-13T14:41:08.119 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0025019109999999999 2024-09-13T14:41:08.119 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.119 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.119 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:08.120 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.630456+0000", 2024-09-13T14:41:08.120 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0026199159999999999 2024-09-13T14:41:08.120 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.120 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.120 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:41:08.120 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.634782+0000", 2024-09-13T14:41:08.120 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0043260019999999998 2024-09-13T14:41:08.120 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.120 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.120 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:41:08.120 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.653124+0000", 2024-09-13T14:41:08.120 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018342607 2024-09-13T14:41:08.120 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:08.121 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:41:08.121 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:08.121 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.121 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.121 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113893 7.e 7:70855f2b:::benchmark_data_smithi102_227395_object1078:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-13T14:41:08.121 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:44.143011+0000", 2024-09-13T14:41:08.121 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 203.811508094, 2024-09-13T14:41:08.121 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.54156615, 2024-09-13T14:41:08.121 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:41:08.121 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:08.121 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:41:08.121 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:41:08.121 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:41:08.122 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113893 2024-09-13T14:41:08.122 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.122 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:41:08.122 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.122 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:41:08.122 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.143011+0000", 2024-09-13T14:41:08.122 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:08.122 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.122 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.122 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:41:08.122 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.143011+0000", 2024-09-13T14:41:08.122 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:08.122 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.123 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.123 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:41:08.123 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.143010+0000", 2024-09-13T14:41:08.123 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:41:08.123 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.123 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.123 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:41:08.123 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.143017+0000", 2024-09-13T14:41:08.123 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.0389999999999999e-06 2024-09-13T14:41:08.123 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.123 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.123 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:41:08.123 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.143028+0000", 2024-09-13T14:41:08.124 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0519999999999999e-05 2024-09-13T14:41:08.124 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.124 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.124 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:41:08.124 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.143033+0000", 2024-09-13T14:41:08.124 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5110000000000003e-06 2024-09-13T14:41:08.124 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.124 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.124 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:08.124 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.613507+0000", 2024-09-13T14:41:08.124 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.47047437800000003 2024-09-13T14:41:08.124 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.125 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.125 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:41:08.125 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.642684+0000", 2024-09-13T14:41:08.125 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.029176705000000001 2024-09-13T14:41:08.125 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.125 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.125 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:08.125 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.784620+0000", 2024-09-13T14:41:08.125 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.14193537000000001 2024-09-13T14:41:08.125 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.125 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.125 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:41:08.125 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.810347+0000", 2024-09-13T14:41:08.126 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.025727015999999998 2024-09-13T14:41:08.126 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.126 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.126 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:08.126 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.567851+0000", 2024-09-13T14:41:08.126 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7575047000000001 2024-09-13T14:41:08.126 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.126 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.126 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:41:08.126 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.575637+0000", 2024-09-13T14:41:08.126 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0077859319999999997 2024-09-13T14:41:08.126 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.127 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.127 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:41:08.127 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.590198+0000", 2024-09-13T14:41:08.127 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.01456061 2024-09-13T14:41:08.127 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.127 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.127 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:41:08.127 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.653927+0000", 2024-09-13T14:41:08.127 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.063728952000000005 2024-09-13T14:41:08.127 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.127 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.127 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:08.127 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.656466+0000", 2024-09-13T14:41:08.128 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0025390679999999998 2024-09-13T14:41:08.128 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.128 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.128 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:41:08.128 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.659062+0000", 2024-09-13T14:41:08.128 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0025957469999999998 2024-09-13T14:41:08.128 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.128 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.128 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:41:08.128 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.663437+0000", 2024-09-13T14:41:08.128 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0043751850000000002 2024-09-13T14:41:08.128 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:08.128 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:08.128 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:41:08.129 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.684577+0000", 2024-09-13T14:41:08.129 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.021140732999999998 2024-09-13T14:41:08.129 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:08.129 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:41:08.129 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:08.129 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:08.129 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:41:08.129 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:41:08.129 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:08.131 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:08 smithi102 ceph-mon[110183]: from='client.35766 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:41:08.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:08 smithi102 ceph-mon[115269]: from='client.35766 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:41:08.156 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:08.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:08.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:08.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:08 smithi195 ceph-mon[104262]: from='client.35766 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:41:08.258 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:08.282 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:08.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:08.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:08.383 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:08.407 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:08.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:08.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:08.509 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:08.531 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:08.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:08.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:08.632 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:08.655 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:08.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:08.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:08.757 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:08.780 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:08.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:08.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:08.881 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:08.886 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T14:41:08.906 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:08.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:08.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:09.008 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:09.032 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:09.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:09.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:09.123 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:41:09.133 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:09.156 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:09.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:09.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:09.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:09 smithi195 ceph-mon[104262]: pgmap v893: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 5.0 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 208 op/s; 29552/244107 objects degraded (12.106%); 5208/244107 objects misplaced (2.133%); 3.4 MiB/s, 53 objects/s recovering 2024-09-13T14:41:09.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:09 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 29552/244107 objects degraded (12.106%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:41:09.258 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:09.259 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:09.284 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:09.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:09.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:09.386 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:09 smithi102 ceph-mon[110183]: pgmap v893: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 5.0 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 208 op/s; 29552/244107 objects degraded (12.106%); 5208/244107 objects misplaced (2.133%); 3.4 MiB/s, 53 objects/s recovering 2024-09-13T14:41:09.386 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:09 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 29552/244107 objects degraded (12.106%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:41:09.386 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:09 smithi102 ceph-mon[115269]: pgmap v893: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 5.0 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 208 op/s; 29552/244107 objects degraded (12.106%); 5208/244107 objects misplaced (2.133%); 3.4 MiB/s, 53 objects/s recovering 2024-09-13T14:41:09.387 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:09 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 29552/244107 objects degraded (12.106%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:41:09.387 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:09.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:09.413 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:09.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:09.515 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:09.538 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:09.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:09.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:09.641 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:09.663 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:09.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:09.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:09.764 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:09.791 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:09.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:09.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:09.893 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:09.923 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:09.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:09.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:10.024 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:10.046 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:10.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:10.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:10.080 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is set 2024-09-13T14:41:10.148 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:10.177 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:10.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:10.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:10.230 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:41:10.278 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:10.304 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:10.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:10.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:10.378 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:10 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/4037253753' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:41:10.406 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:10.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:10 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/4037253753' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:41:10.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:10 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/4037253753' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:41:10.429 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:10.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:10.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:10.531 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:10.555 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:10.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:10.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:10.657 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:10.681 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:10.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:10.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:10.783 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:10.805 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:10.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:10.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:10.907 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:10.931 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:10.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:10.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:10.972 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:41:10.972 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-13T14:41:10.972 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 0 2024-09-13T14:41:10.972 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:41:11.033 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:11.056 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:11.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:11.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:11.157 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:11.180 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:11.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:11.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:11.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:11 smithi195 ceph-mon[104262]: pgmap v894: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.9 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 203 op/s; 29504/242964 objects degraded (12.143%); 5208/242964 objects misplaced (2.144%); 3.3 MiB/s, 53 objects/s recovering 2024-09-13T14:41:11.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:11 smithi195 ceph-mon[104262]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:41:11.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:11 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/4037253753' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:41:11.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:11 smithi195 ceph-mon[104262]: osdmap e270: 8 total, 8 up, 6 in 2024-09-13T14:41:11.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:11 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:41:11.282 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:11.308 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:11.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:11.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:11.366 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:11 smithi102 ceph-mon[110183]: pgmap v894: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.9 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 203 op/s; 29504/242964 objects degraded (12.143%); 5208/242964 objects misplaced (2.144%); 3.3 MiB/s, 53 objects/s recovering 2024-09-13T14:41:11.367 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:11 smithi102 ceph-mon[110183]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:41:11.367 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:11 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/4037253753' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:41:11.367 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:11 smithi102 ceph-mon[110183]: osdmap e270: 8 total, 8 up, 6 in 2024-09-13T14:41:11.367 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:11 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:41:11.367 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:11 smithi102 ceph-mon[115269]: pgmap v894: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.9 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 203 op/s; 29504/242964 objects degraded (12.143%); 5208/242964 objects misplaced (2.144%); 3.3 MiB/s, 53 objects/s recovering 2024-09-13T14:41:11.367 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:11 smithi102 ceph-mon[115269]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:41:11.367 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:11 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/4037253753' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:41:11.368 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:11 smithi102 ceph-mon[115269]: osdmap e270: 8 total, 8 up, 6 in 2024-09-13T14:41:11.368 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:11 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:41:11.410 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:11.437 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:11.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:11.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:11.440 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:11.541 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:11.564 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:11.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:11.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:11.666 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:11.687 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T14:41:11.691 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:11.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:11.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:11.793 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:11.817 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:11.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:11.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:11.920 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:11.957 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:41:11.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:11.958 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:11.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:12.059 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:12.082 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:12.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:12.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:12.165 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:41:12.165 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:41:12.184 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:12.185 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:41:12.185 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:41:12.206 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:41:12.207 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:41:12.208 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:12.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:12.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:12.224 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:41:12.224 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:41:12.243 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:41:12.243 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:41:12.265 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:41:12.265 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:41:12.283 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:41:12.284 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:41:12.309 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:12.312 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:41:12.312 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:41:12.335 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:12.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:12.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:12.437 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:12.462 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:12.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:12.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:12.564 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:12.590 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:12.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:12.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:12.692 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:12.714 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:12.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:12.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:12.816 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:12.837 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:12.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:12.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:12.939 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:12.961 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:12.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:12.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:13.064 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:13.088 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:13.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:13.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:13.190 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:13.214 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:13.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:13.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:13.316 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:13.338 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:13.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:13.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:13.348 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:13 smithi195 ceph-mon[104262]: pgmap v896: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.9 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 203 op/s; 29504/242964 objects degraded (12.143%); 5208/242964 objects misplaced (2.144%); 3.3 MiB/s, 53 objects/s recovering 2024-09-13T14:41:13.349 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:13 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1462525402' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:41:13.401 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:41:13.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:13 smithi102 ceph-mon[110183]: pgmap v896: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.9 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 203 op/s; 29504/242964 objects degraded (12.143%); 5208/242964 objects misplaced (2.144%); 3.3 MiB/s, 53 objects/s recovering 2024-09-13T14:41:13.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:13 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1462525402' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:41:13.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:13 smithi102 ceph-mon[115269]: pgmap v896: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.9 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 203 op/s; 29504/242964 objects degraded (12.143%); 5208/242964 objects misplaced (2.144%); 3.3 MiB/s, 53 objects/s recovering 2024-09-13T14:41:13.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:13 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1462525402' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:41:13.440 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:13.466 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:13.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:13.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:13.569 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:13.597 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:13.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:13.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:13.698 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:13.726 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:13.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:13.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:13.827 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:13.859 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:13.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:13.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:13.950 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:41:13.951 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:41:13.961 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:13.982 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:13.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:13.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:14.084 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:14.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:14.109 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:14.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:14.210 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:14.235 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:14.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:14.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:14.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:14 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 29504/242964 objects degraded (12.143%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:41:14.338 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:14.363 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:14.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:14.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:14.366 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:14 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 29504/242964 objects degraded (12.143%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:41:14.366 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:14 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 29504/242964 objects degraded (12.143%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:41:14.465 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:14.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:14.490 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:14.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:14.591 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:14.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:14.623 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:14.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:14.626 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T14:41:14.696 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:14.724 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:14.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:14.750 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:14.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:14.831 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:41:14.838 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:41:14.851 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:14.882 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:14.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:14.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:14.917 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:41:14.984 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:15.016 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:15.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:15.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:15.118 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:15.142 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:15.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:15.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:15.245 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:15.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:15.269 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:15.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:15.366 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:15 smithi195 ceph-mon[104262]: pgmap v897: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.9 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 194 op/s; 29375/241284 objects degraded (12.174%); 5128/241284 objects misplaced (2.125%); 2.5 MiB/s, 39 objects/s recovering 2024-09-13T14:41:15.370 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:15.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:15.393 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:15.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:15.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:15 smithi102 ceph-mon[110183]: pgmap v897: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.9 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 194 op/s; 29375/241284 objects degraded (12.174%); 5128/241284 objects misplaced (2.125%); 2.5 MiB/s, 39 objects/s recovering 2024-09-13T14:41:15.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:15 smithi102 ceph-mon[115269]: pgmap v897: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.9 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 194 op/s; 29375/241284 objects degraded (12.174%); 5128/241284 objects misplaced (2.125%); 2.5 MiB/s, 39 objects/s recovering 2024-09-13T14:41:15.495 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:15.516 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:15.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:15.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:15.618 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:15.642 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:15.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:15.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:15.744 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:15.753 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":897,"stamp":"2024-09-13T14:41:13.491879+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5273819133,"num_objects":80428,"num_object_clones":0,"num_object_copies":241284,"num_objects_missing_on_primary":2961,"num_objects_missing":2961,"num_objects_degraded":29375,"num_objects_misplaced":5128,"num_objects_unfound":0,"num_objects_dirty":80428,"num_whiteouts":0,"num_read":6377,"num_read_kb":13834,"num_write":146349,"num_write_kb":7235010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60447,"num_bytes_recovered":3924626573,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93631,"ondisk_log_size":93631,"up":267,"acting":261,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":309,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":19710004,"kb_used_data":16658812,"kb_used_omap":26,"kb_used_meta":3051109,"kb_avail":542760908,"statfs":{"total":575970213888,"available":555787169792,"internally_reserved":0,"allocated":17058623488,"data_stored":17049815758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27144,"internal_metadata":3124336120},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,2,3,4,1,1,4,4],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-127533056,"num_objects":-1946,"num_object_clones":0,"num_object_copies":-5838,"num_objects_missing_on_primary":-140,"num_objects_missing":-140,"num_objects_degraded":-377,"num_objects_misplaced":-153,"num_objects_unfound":0,"num_objects_dirty":-1946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1946,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":400,"num_bytes_recovered":26214400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002897"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":432,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:41:10.106514+0000","last_change":"2024-09-13T14:36:13.859927+0000","last_active":"2024-09-13T14:41:10.106514+0000","last_peered":"2024-09-13T14:41:10.106514+0000","last_clean":"2024-09-13T14:41:10.106514+0000","last_became_active":"2024-09-13T14:36:13.859505+0000","last_became_peered":"2024-09-13T14:36:13.859505+0000","last_unstale":"2024-09-13T14:41:10.106514+0000","last_undegraded":"2024-09-13T14:41:10.106514+0000","last_fullsized":"2024-09-13T14:41:10.106514+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:12:20.989627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":271,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.060096+0000","last_change":"2024-09-13T14:37:48.130968+0000","last_active":"2024-09-13T14:41:00.060096+0000","last_peered":"2024-09-13T14:41:00.060096+0000","last_clean":"2024-09-13T14:41:00.060096+0000","last_became_active":"2024-09-13T14:37:48.130603+0000","last_became_peered":"2024-09-13T14:37:48.130603+0000","last_unstale":"2024-09-13T14:41:00.060096+0000","last_undegraded":"2024-09-13T14:41:00.060096+0000","last_fullsized":"2024-09-13T14:41:00.060096+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:42.626386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":68,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:01.945860+0000","last_change":"2024-09-13T14:36:13.862181+0000","last_active":"2024-09-13T14:41:01.945860+0000","last_peered":"2024-09-13T14:41:01.945860+0000","last_clean":"2024-09-13T14:41:01.945860+0000","last_became_active":"2024-09-13T14:36:13.861873+0000","last_became_peered":"2024-09-13T14:36:13.861873+0000","last_unstale":"2024-09-13T14:41:01.945860+0000","last_undegraded":"2024-09-13T14:41:01.945860+0000","last_fullsized":"2024-09-13T14:41:01.945860+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:14.085700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":68,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.067078+0000","last_change":"2024-09-13T14:38:53.896672+0000","last_active":"2024-09-13T14:41:00.067078+0000","last_peered":"2024-09-13T14:41:00.067078+0000","last_clean":"2024-09-13T14:41:00.067078+0000","last_became_active":"2024-09-13T14:38:53.896346+0000","last_became_peered":"2024-09-13T14:38:53.896346+0000","last_unstale":"2024-09-13T14:41:00.067078+0000","last_undegraded":"2024-09-13T14:41:00.067078+0000","last_fullsized":"2024-09-13T14:41:00.067078+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:57.789686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":68,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.067484+0000","last_change":"2024-09-13T14:38:53.893021+0000","last_active":"2024-09-13T14:41:00.067484+0000","last_peered":"2024-09-13T14:41:00.067484+0000","last_clean":"2024-09-13T14:41:00.067484+0000","last_became_active":"2024-09-13T14:38:53.892668+0000","last_became_peered":"2024-09-13T14:38:53.892668+0000","last_unstale":"2024-09-13T14:41:00.067484+0000","last_undegraded":"2024-09-13T14:41:00.067484+0000","last_fullsized":"2024-09-13T14:41:00.067484+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:57.180738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":239,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.059871+0000","last_change":"2024-09-13T14:37:48.134642+0000","last_active":"2024-09-13T14:41:00.059871+0000","last_peered":"2024-09-13T14:41:00.059871+0000","last_clean":"2024-09-13T14:41:00.059871+0000","last_became_active":"2024-09-13T14:37:48.134402+0000","last_became_peered":"2024-09-13T14:37:48.134402+0000","last_unstale":"2024-09-13T14:41:00.059871+0000","last_undegraded":"2024-09-13T14:41:00.059871+0000","last_fullsized":"2024-09-13T14:41:00.059871+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:51.313663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":451,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:41:10.107500+0000","last_change":"2024-09-13T14:38:53.903022+0000","last_active":"2024-09-13T14:41:10.107500+0000","last_peered":"2024-09-13T14:41:10.107500+0000","last_clean":"2024-09-13T14:41:10.107500+0000","last_became_active":"2024-09-13T14:38:53.902448+0000","last_became_peered":"2024-09-13T14:38:53.902448+0000","last_unstale":"2024-09-13T14:41:10.107500+0000","last_undegraded":"2024-09-13T14:41:10.107500+0000","last_fullsized":"2024-09-13T14:41:10.107500+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:24.690401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":71,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.066620+0000","last_change":"2024-09-13T14:38:03.554500+0000","last_active":"2024-09-13T14:41:00.066620+0000","last_peered":"2024-09-13T14:41:00.066620+0000","last_clean":"2024-09-13T14:41:00.066620+0000","last_became_active":"2024-09-13T14:38:03.532910+0000","last_became_peered":"2024-09-13T14:38:03.532910+0000","last_unstale":"2024-09-13T14:41:00.066620+0000","last_undegraded":"2024-09-13T14:41:00.066620+0000","last_fullsized":"2024-09-13T14:41:00.066620+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:27:50.843498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"270'8998","reported_seq":13432,"reported_epoch":270,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-13T14:41:11.977485+0000","last_change":"2024-09-13T14:40:32.799848+0000","last_active":"2024-09-13T14:41:11.977485+0000","last_peered":"2024-09-13T14:41:11.977485+0000","last_clean":"2024-09-13T14:37:46.417003+0000","last_became_active":"2024-09-13T14:37:49.690961+0000","last_became_peered":"2024-09-13T14:37:49.690961+0000","last_unstale":"2024-09-13T14:41:11.977485+0000","last_undegraded":"2024-09-13T14:41:11.977485+0000","last_fullsized":"2024-09-13T14:37:49.126746+0000","mapping_epoch":240,"log_start":"212'3300","ondisk_log_start":"212'3300","created":205,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5698,"log_dups_size":0,"ondisk_log_size":5698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:48:52.526429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319946752,"num_objects":4882,"num_object_clones":0,"num_object_copies":14646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3139,"num_objects_unfound":0,"num_objects_dirty":4882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8998,"num_write_kb":444160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5934,"num_bytes_recovered":388890624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,4],"avail_no_missing":["5","0","4"],"object_location_counts":[{"shards":"0,4,5","objects":3139},{"shards":"4,5","objects":1743}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":32,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:41:10.085903+0000","last_change":"2024-09-13T14:38:53.940238+0000","last_active":"2024-09-13T14:41:10.085903+0000","last_peered":"2024-09-13T14:41:10.085903+0000","last_clean":"2024-09-13T14:41:10.085903+0000","last_became_active":"2024-09-13T14:38:53.892825+0000","last_became_peered":"2024-09-13T14:38:53.892825+0000","last_unstale":"2024-09-13T14:41:10.085903+0000","last_undegraded":"2024-09-13T14:41:10.085903+0000","last_fullsized":"2024-09-13T14:41:10.085903+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:46.666581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":88,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.072235+0000","last_change":"2024-09-13T14:38:53.898907+0000","last_active":"2024-09-13T14:41:00.072235+0000","last_peered":"2024-09-13T14:41:00.072235+0000","last_clean":"2024-09-13T14:41:00.072235+0000","last_became_active":"2024-09-13T14:38:53.898572+0000","last_became_peered":"2024-09-13T14:38:53.898572+0000","last_unstale":"2024-09-13T14:41:00.072235+0000","last_undegraded":"2024-09-13T14:41:00.072235+0000","last_fullsized":"2024-09-13T14:41:00.072235+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:29.222150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"270'9170","reported_seq":23319,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:41:12.447270+0000","last_change":"2024-09-13T14:38:57.930032+0000","last_active":"2024-09-13T14:41:12.447270+0000","last_peered":"2024-09-13T14:41:12.447270+0000","last_clean":"2024-09-13T14:41:12.447270+0000","last_became_active":"2024-09-13T14:38:57.921458+0000","last_became_peered":"2024-09-13T14:38:57.921458+0000","last_unstale":"2024-09-13T14:41:12.447270+0000","last_undegraded":"2024-09-13T14:41:12.447270+0000","last_fullsized":"2024-09-13T14:41:12.447270+0000","mapping_epoch":256,"log_start":"212'3839","ondisk_log_start":"212'3839","created":205,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5331,"log_dups_size":0,"ondisk_log_size":5331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:27:12.294038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332398592,"num_objects":5072,"num_object_clones":0,"num_object_copies":15216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9170,"num_write_kb":455744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6990,"num_bytes_recovered":458096640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":32,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:41:10.085743+0000","last_change":"2024-09-13T14:38:53.904885+0000","last_active":"2024-09-13T14:41:10.085743+0000","last_peered":"2024-09-13T14:41:10.085743+0000","last_clean":"2024-09-13T14:41:10.085743+0000","last_became_active":"2024-09-13T14:38:53.891726+0000","last_became_peered":"2024-09-13T14:38:53.891726+0000","last_unstale":"2024-09-13T14:41:10.085743+0000","last_undegraded":"2024-09-13T14:41:10.085743+0000","last_fullsized":"2024-09-13T14:41:10.085743+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:57.079205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":277,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.065672+0000","last_change":"2024-09-13T14:37:48.160504+0000","last_active":"2024-09-13T14:41:00.065672+0000","last_peered":"2024-09-13T14:41:00.065672+0000","last_clean":"2024-09-13T14:41:00.065672+0000","last_became_active":"2024-09-13T14:37:48.160181+0000","last_became_peered":"2024-09-13T14:37:48.160181+0000","last_unstale":"2024-09-13T14:41:00.065672+0000","last_undegraded":"2024-09-13T14:41:00.065672+0000","last_fullsized":"2024-09-13T14:41:00.065672+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:37.087008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"269'9105","reported_seq":10874,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:08.770413+0000","last_change":"2024-09-13T14:38:04.531914+0000","last_active":"2024-09-13T14:41:08.770413+0000","last_peered":"2024-09-13T14:41:08.770413+0000","last_clean":"2024-09-13T14:41:08.770413+0000","last_became_active":"2024-09-13T14:38:04.523448+0000","last_became_peered":"2024-09-13T14:38:04.523448+0000","last_unstale":"2024-09-13T14:41:08.770413+0000","last_undegraded":"2024-09-13T14:41:08.770413+0000","last_fullsized":"2024-09-13T14:41:08.770413+0000","mapping_epoch":245,"log_start":"212'3200","ondisk_log_start":"212'3200","created":205,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5905,"log_dups_size":0,"ondisk_log_size":5905,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:06.536896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332201984,"num_objects":5069,"num_object_clones":0,"num_object_copies":15207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9105,"num_write_kb":453568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":811,"num_bytes_recovered":53149696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":269,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.061232+0000","last_change":"2024-09-13T14:37:49.611774+0000","last_active":"2024-09-13T14:41:00.061232+0000","last_peered":"2024-09-13T14:41:00.061232+0000","last_clean":"2024-09-13T14:41:00.061232+0000","last_became_active":"2024-09-13T14:37:49.611088+0000","last_became_peered":"2024-09-13T14:37:49.611088+0000","last_unstale":"2024-09-13T14:41:00.061232+0000","last_undegraded":"2024-09-13T14:41:00.061232+0000","last_fullsized":"2024-09-13T14:41:00.061232+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:08.933213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":28,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:07.412358+0000","last_change":"2024-09-13T14:38:53.899211+0000","last_active":"2024-09-13T14:41:07.412358+0000","last_peered":"2024-09-13T14:41:07.412358+0000","last_clean":"2024-09-13T14:41:07.412358+0000","last_became_active":"2024-09-13T14:38:53.898927+0000","last_became_peered":"2024-09-13T14:38:53.898927+0000","last_unstale":"2024-09-13T14:41:07.412358+0000","last_undegraded":"2024-09-13T14:41:07.412358+0000","last_fullsized":"2024-09-13T14:41:07.412358+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:00:28.857611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"270'9004","reported_seq":9080,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:41:12.278384+0000","last_change":"2024-09-13T14:35:24.004651+0000","last_active":"2024-09-13T14:41:12.278384+0000","last_peered":"2024-09-13T14:41:12.278384+0000","last_clean":"2024-09-13T14:41:12.278384+0000","last_became_active":"2024-09-13T14:35:24.004358+0000","last_became_peered":"2024-09-13T14:35:24.004358+0000","last_unstale":"2024-09-13T14:41:12.278384+0000","last_undegraded":"2024-09-13T14:41:12.278384+0000","last_fullsized":"2024-09-13T14:41:12.278384+0000","mapping_epoch":205,"log_start":"212'2684","ondisk_log_start":"212'2684","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6320,"log_dups_size":0,"ondisk_log_size":6320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:57:38.023669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":325582848,"num_objects":4968,"num_object_clones":0,"num_object_copies":14904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9004,"num_write_kb":447104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":79,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.073288+0000","last_change":"2024-09-13T14:37:49.168003+0000","last_active":"2024-09-13T14:41:00.073288+0000","last_peered":"2024-09-13T14:41:00.073288+0000","last_clean":"2024-09-13T14:41:00.073288+0000","last_became_active":"2024-09-13T14:37:49.167638+0000","last_became_peered":"2024-09-13T14:37:49.167638+0000","last_unstale":"2024-09-13T14:41:00.073288+0000","last_undegraded":"2024-09-13T14:41:00.073288+0000","last_fullsized":"2024-09-13T14:41:00.073288+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:29.051981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":61,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.067020+0000","last_change":"2024-09-13T14:38:03.528381+0000","last_active":"2024-09-13T14:41:00.067020+0000","last_peered":"2024-09-13T14:41:00.067020+0000","last_clean":"2024-09-13T14:41:00.067020+0000","last_became_active":"2024-09-13T14:38:03.527282+0000","last_became_peered":"2024-09-13T14:38:03.527282+0000","last_unstale":"2024-09-13T14:41:00.067020+0000","last_undegraded":"2024-09-13T14:41:00.067020+0000","last_fullsized":"2024-09-13T14:41:00.067020+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:24.435067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.f","version":"269'9052","reported_seq":9316,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:08.848746+0000","last_change":"2024-09-13T14:38:16.371079+0000","last_active":"2024-09-13T14:41:08.848746+0000","last_peered":"2024-09-13T14:41:08.848746+0000","last_clean":"2024-09-13T14:41:08.848746+0000","last_became_active":"2024-09-13T14:38:12.859883+0000","last_became_peered":"2024-09-13T14:38:12.859883+0000","last_unstale":"2024-09-13T14:41:08.848746+0000","last_undegraded":"2024-09-13T14:41:08.848746+0000","last_fullsized":"2024-09-13T14:41:08.848746+0000","mapping_epoch":245,"log_start":"212'3163","ondisk_log_start":"212'3163","created":205,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5889,"log_dups_size":0,"ondisk_log_size":5889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:08.436532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":329908224,"num_objects":5034,"num_object_clones":0,"num_object_copies":15102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9052,"num_write_kb":450752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":3670016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":421,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:41:10.108243+0000","last_change":"2024-09-13T14:38:03.549595+0000","last_active":"2024-09-13T14:41:10.108243+0000","last_peered":"2024-09-13T14:41:10.108243+0000","last_clean":"2024-09-13T14:41:10.108243+0000","last_became_active":"2024-09-13T14:38:03.533331+0000","last_became_peered":"2024-09-13T14:38:03.533331+0000","last_unstale":"2024-09-13T14:41:10.108243+0000","last_undegraded":"2024-09-13T14:41:10.108243+0000","last_fullsized":"2024-09-13T14:41:10.108243+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:59:02.470343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":253,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.056672+0000","last_change":"2024-09-13T14:37:48.127600+0000","last_active":"2024-09-13T14:41:00.056672+0000","last_peered":"2024-09-13T14:41:00.056672+0000","last_clean":"2024-09-13T14:41:00.056672+0000","last_became_active":"2024-09-13T14:37:48.127349+0000","last_became_peered":"2024-09-13T14:37:48.127349+0000","last_unstale":"2024-09-13T14:41:00.056672+0000","last_undegraded":"2024-09-13T14:41:00.056672+0000","last_fullsized":"2024-09-13T14:41:00.056672+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:57.268374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"270'9015","reported_seq":13144,"reported_epoch":270,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:41:11.984969+0000","last_change":"2024-09-13T14:38:14.979229+0000","last_active":"2024-09-13T14:41:11.984969+0000","last_peered":"2024-09-13T14:41:11.984969+0000","last_clean":"2024-09-13T14:36:13.131873+0000","last_became_active":"2024-09-13T14:38:10.443954+0000","last_became_peered":"2024-09-13T14:38:10.443954+0000","last_unstale":"2024-09-13T14:41:11.984969+0000","last_undegraded":"2024-09-13T14:38:10.384643+0000","last_fullsized":"2024-09-13T14:38:04.517046+0000","mapping_epoch":245,"log_start":"212'3300","ondisk_log_start":"212'3300","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5715,"log_dups_size":0,"ondisk_log_size":5715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:26:53.493170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327352320,"num_objects":4995,"num_object_clones":0,"num_object_copies":14985,"num_objects_missing_on_primary":2771,"num_objects_missing":2771,"num_objects_degraded":2912,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9015,"num_write_kb":448320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2005,"num_bytes_recovered":131334144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":1},{"shards":"0,3,5,7","objects":9},{"shards":"0,3,7","objects":2770},{"shards":"0,5","objects":2210},{"shards":"3,5,7","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":186,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.056822+0000","last_change":"2024-09-13T14:37:48.127604+0000","last_active":"2024-09-13T14:41:00.056822+0000","last_peered":"2024-09-13T14:41:00.056822+0000","last_clean":"2024-09-13T14:41:00.056822+0000","last_became_active":"2024-09-13T14:37:48.127300+0000","last_became_peered":"2024-09-13T14:37:48.127300+0000","last_unstale":"2024-09-13T14:41:00.056822+0000","last_undegraded":"2024-09-13T14:41:00.056822+0000","last_fullsized":"2024-09-13T14:41:00.056822+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:51.999506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":306,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.057213+0000","last_change":"2024-09-13T14:38:03.529967+0000","last_active":"2024-09-13T14:41:00.057213+0000","last_peered":"2024-09-13T14:41:00.057213+0000","last_clean":"2024-09-13T14:41:00.057213+0000","last_became_active":"2024-09-13T14:38:03.529646+0000","last_became_peered":"2024-09-13T14:38:03.529646+0000","last_unstale":"2024-09-13T14:41:00.057213+0000","last_undegraded":"2024-09-13T14:41:00.057213+0000","last_fullsized":"2024-09-13T14:41:00.057213+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:52:05.193279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"269'9087","reported_seq":18226,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:08.727660+0000","last_change":"2024-09-13T14:40:34.583237+0000","last_active":"2024-09-13T14:41:08.727660+0000","last_peered":"2024-09-13T14:41:08.727660+0000","last_clean":"2024-09-13T14:41:08.727660+0000","last_became_active":"2024-09-13T14:40:34.563989+0000","last_became_peered":"2024-09-13T14:40:34.563989+0000","last_unstale":"2024-09-13T14:41:08.727660+0000","last_undegraded":"2024-09-13T14:41:08.727660+0000","last_fullsized":"2024-09-13T14:41:08.727660+0000","mapping_epoch":266,"log_start":"212'3200","ondisk_log_start":"212'3200","created":205,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5887,"log_dups_size":0,"ondisk_log_size":5887,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:34:43.006051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331087896,"num_objects":5053,"num_object_clones":0,"num_object_copies":15159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5053,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9087,"num_write_kb":452417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4489,"num_bytes_recovered":289865752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":266,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.066060+0000","last_change":"2024-09-13T14:36:13.860367+0000","last_active":"2024-09-13T14:41:00.066060+0000","last_peered":"2024-09-13T14:41:00.066060+0000","last_clean":"2024-09-13T14:41:00.066060+0000","last_became_active":"2024-09-13T14:36:13.859982+0000","last_became_peered":"2024-09-13T14:36:13.859982+0000","last_unstale":"2024-09-13T14:41:00.066060+0000","last_undegraded":"2024-09-13T14:41:00.066060+0000","last_fullsized":"2024-09-13T14:41:00.066060+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:08:23.311350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":262,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.057609+0000","last_change":"2024-09-13T14:37:48.147360+0000","last_active":"2024-09-13T14:41:00.057609+0000","last_peered":"2024-09-13T14:41:00.057609+0000","last_clean":"2024-09-13T14:41:00.057609+0000","last_became_active":"2024-09-13T14:37:48.146885+0000","last_became_peered":"2024-09-13T14:37:48.146885+0000","last_unstale":"2024-09-13T14:41:00.057609+0000","last_undegraded":"2024-09-13T14:41:00.057609+0000","last_fullsized":"2024-09-13T14:41:00.057609+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:27.841810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"270'9055","reported_seq":18303,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:41:12.444656+0000","last_change":"2024-09-13T14:38:39.650332+0000","last_active":"2024-09-13T14:41:12.444656+0000","last_peered":"2024-09-13T14:41:12.444656+0000","last_clean":"2024-09-13T14:41:12.444656+0000","last_became_active":"2024-09-13T14:38:39.640835+0000","last_became_peered":"2024-09-13T14:38:39.640835+0000","last_unstale":"2024-09-13T14:41:12.444656+0000","last_undegraded":"2024-09-13T14:41:12.444656+0000","last_fullsized":"2024-09-13T14:41:12.444656+0000","mapping_epoch":251,"log_start":"212'3751","ondisk_log_start":"212'3751","created":205,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5304,"log_dups_size":0,"ondisk_log_size":5304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:48.176606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328925184,"num_objects":5019,"num_object_clones":0,"num_object_copies":15057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9055,"num_write_kb":450368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4544,"num_bytes_recovered":292945920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":439,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:41:10.109759+0000","last_change":"2024-09-13T14:38:53.900041+0000","last_active":"2024-09-13T14:41:10.109759+0000","last_peered":"2024-09-13T14:41:10.109759+0000","last_clean":"2024-09-13T14:41:10.109759+0000","last_became_active":"2024-09-13T14:38:53.899203+0000","last_became_peered":"2024-09-13T14:38:53.899203+0000","last_unstale":"2024-09-13T14:41:10.109759+0000","last_undegraded":"2024-09-13T14:41:10.109759+0000","last_fullsized":"2024-09-13T14:41:10.109759+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:28:07.883094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":371,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:41:10.107635+0000","last_change":"2024-09-13T14:26:15.823198+0000","last_active":"2024-09-13T14:41:10.107635+0000","last_peered":"2024-09-13T14:41:10.107635+0000","last_clean":"2024-09-13T14:41:10.107635+0000","last_became_active":"2024-09-13T14:26:15.822717+0000","last_became_peered":"2024-09-13T14:26:15.822717+0000","last_unstale":"2024-09-13T14:41:10.107635+0000","last_undegraded":"2024-09-13T14:41:10.107635+0000","last_fullsized":"2024-09-13T14:41:10.107635+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:22.471622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.6","version":"269'9140","reported_seq":13165,"reported_epoch":269,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-13T14:41:08.514127+0000","last_change":"2024-09-13T14:38:55.132683+0000","last_active":"2024-09-13T14:41:08.514127+0000","last_peered":"2024-09-13T14:41:08.514127+0000","last_clean":"2024-09-13T14:36:13.124368+0000","last_became_active":"2024-09-13T14:38:55.120883+0000","last_became_peered":"2024-09-13T14:38:55.120883+0000","last_unstale":"2024-09-13T14:41:08.514127+0000","last_undegraded":"2024-09-13T14:38:55.026610+0000","last_fullsized":"2024-09-13T14:41:08.514127+0000","mapping_epoch":254,"log_start":"212'3219","ondisk_log_start":"212'3219","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5921,"log_dups_size":0,"ondisk_log_size":5921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331481088,"num_objects":5058,"num_object_clones":0,"num_object_copies":15174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3071,"num_objects_misplaced":1987,"num_objects_unfound":0,"num_objects_dirty":5058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9140,"num_write_kb":454336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1934,"num_bytes_recovered":115539968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,3,5],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3,4","objects":2792},{"shards":"0,3,5","objects":2266}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":435,"reported_epoch":270,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-13T14:41:10.109568+0000","last_change":"2024-09-13T14:38:54.899624+0000","last_active":"2024-09-13T14:41:10.109568+0000","last_peered":"2024-09-13T14:41:10.109568+0000","last_clean":"2024-09-13T14:17:27.401592+0000","last_became_active":"2024-09-13T14:38:54.899282+0000","last_became_peered":"2024-09-13T14:38:54.899282+0000","last_unstale":"2024-09-13T14:41:10.109568+0000","last_undegraded":"2024-09-13T14:38:54.891176+0000","last_fullsized":"2024-09-13T14:41:10.109568+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:07:41.927740+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[3,0,6],"avail_no_missing":["3","4","6"],"object_location_counts":[{"shards":"3,4,6","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":508,"reported_epoch":269,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:41:00.072705+0000","last_change":"2024-09-13T14:38:54.904329+0000","last_active":"2024-09-13T14:41:00.072705+0000","last_peered":"2024-09-13T14:41:00.072705+0000","last_clean":"2024-09-13T14:38:39.648017+0000","last_became_active":"2024-09-13T14:38:54.903798+0000","last_became_peered":"2024-09-13T14:38:54.903798+0000","last_unstale":"2024-09-13T14:41:00.072705+0000","last_undegraded":"2024-09-13T14:38:54.890505+0000","last_fullsized":"2024-09-13T14:38:54.889701+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:18.120931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,3],"avail_no_missing":["5","3","6"],"object_location_counts":[{"shards":"3,5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":373,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:41:10.106882+0000","last_change":"2024-09-13T14:37:49.160719+0000","last_active":"2024-09-13T14:41:10.106882+0000","last_peered":"2024-09-13T14:41:10.106882+0000","last_clean":"2024-09-13T14:41:10.106882+0000","last_became_active":"2024-09-13T14:37:49.160401+0000","last_became_peered":"2024-09-13T14:37:49.160401+0000","last_unstale":"2024-09-13T14:41:10.106882+0000","last_undegraded":"2024-09-13T14:41:10.106882+0000","last_fullsized":"2024-09-13T14:41:10.106882+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:21.737945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":113,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.057143+0000","last_change":"2024-09-13T14:37:48.172940+0000","last_active":"2024-09-13T14:41:00.057143+0000","last_peered":"2024-09-13T14:41:00.057143+0000","last_clean":"2024-09-13T14:41:00.057143+0000","last_became_active":"2024-09-13T14:37:48.172060+0000","last_became_peered":"2024-09-13T14:37:48.172060+0000","last_unstale":"2024-09-13T14:41:00.057143+0000","last_undegraded":"2024-09-13T14:41:00.057143+0000","last_fullsized":"2024-09-13T14:41:00.057143+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:07.480645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.5","version":"270'9060","reported_seq":25412,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:41:12.402861+0000","last_change":"2024-09-13T14:39:34.446699+0000","last_active":"2024-09-13T14:41:12.402861+0000","last_peered":"2024-09-13T14:41:12.402861+0000","last_clean":"2024-09-13T14:41:12.402861+0000","last_became_active":"2024-09-13T14:38:04.600616+0000","last_became_peered":"2024-09-13T14:38:04.600616+0000","last_unstale":"2024-09-13T14:41:12.402861+0000","last_undegraded":"2024-09-13T14:41:12.402861+0000","last_fullsized":"2024-09-13T14:41:12.402861+0000","mapping_epoch":245,"log_start":"212'2716","ondisk_log_start":"212'2716","created":205,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6344,"log_dups_size":0,"ondisk_log_size":6344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:00:44.321678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333053952,"num_objects":5082,"num_object_clones":0,"num_object_copies":15246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9060,"num_write_kb":452544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8120,"num_bytes_recovered":521076736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":486,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.060440+0000","last_change":"2024-09-13T14:26:15.835178+0000","last_active":"2024-09-13T14:41:00.060440+0000","last_peered":"2024-09-13T14:41:00.060440+0000","last_clean":"2024-09-13T14:41:00.060440+0000","last_became_active":"2024-09-13T14:26:15.820283+0000","last_became_peered":"2024-09-13T14:26:15.820283+0000","last_unstale":"2024-09-13T14:41:00.060440+0000","last_undegraded":"2024-09-13T14:41:00.060440+0000","last_fullsized":"2024-09-13T14:41:00.060440+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:30.340038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":61,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.066667+0000","last_change":"2024-09-13T14:38:03.525682+0000","last_active":"2024-09-13T14:41:00.066667+0000","last_peered":"2024-09-13T14:41:00.066667+0000","last_clean":"2024-09-13T14:41:00.066667+0000","last_became_active":"2024-09-13T14:38:03.525406+0000","last_became_peered":"2024-09-13T14:38:03.525406+0000","last_unstale":"2024-09-13T14:41:00.066667+0000","last_undegraded":"2024-09-13T14:41:00.066667+0000","last_fullsized":"2024-09-13T14:41:00.066667+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:41:28.970300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":68,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.067555+0000","last_change":"2024-09-13T14:38:53.991738+0000","last_active":"2024-09-13T14:41:00.067555+0000","last_peered":"2024-09-13T14:41:00.067555+0000","last_clean":"2024-09-13T14:41:00.067555+0000","last_became_active":"2024-09-13T14:38:53.991417+0000","last_became_peered":"2024-09-13T14:38:53.991417+0000","last_unstale":"2024-09-13T14:41:00.067555+0000","last_undegraded":"2024-09-13T14:41:00.067555+0000","last_fullsized":"2024-09-13T14:41:00.067555+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:50:38.559838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"269'9090","reported_seq":23251,"reported_epoch":269,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-13T14:41:08.730180+0000","last_change":"2024-09-13T14:38:17.472071+0000","last_active":"2024-09-13T14:41:08.730180+0000","last_peered":"2024-09-13T14:41:08.730180+0000","last_clean":"2024-09-13T14:37:46.290754+0000","last_became_active":"2024-09-13T14:38:17.381753+0000","last_became_peered":"2024-09-13T14:38:17.381753+0000","last_unstale":"2024-09-13T14:41:08.730180+0000","last_undegraded":"2024-09-13T14:38:13.504140+0000","last_fullsized":"2024-09-13T14:41:08.730180+0000","mapping_epoch":245,"log_start":"212'3159","ondisk_log_start":"212'3159","created":205,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5931,"log_dups_size":0,"ondisk_log_size":5931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328728576,"num_objects":5016,"num_object_clones":0,"num_object_copies":15048,"num_objects_missing_on_primary":83,"num_objects_missing":83,"num_objects_degraded":4582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9090,"num_write_kb":451392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7053,"num_bytes_recovered":462225408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"0,4,5","objects":4413},{"shards":"0,4,6","objects":603}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":784,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:41:10.106415+0000","last_change":"2024-09-13T14:37:48.131799+0000","last_active":"2024-09-13T14:41:10.106415+0000","last_peered":"2024-09-13T14:41:10.106415+0000","last_clean":"2024-09-13T14:41:10.106415+0000","last_became_active":"2024-09-13T14:37:48.130837+0000","last_became_peered":"2024-09-13T14:37:48.130837+0000","last_unstale":"2024-09-13T14:41:10.106415+0000","last_undegraded":"2024-09-13T14:41:10.106415+0000","last_fullsized":"2024-09-13T14:41:10.106415+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:06:32.663770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":301,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.066887+0000","last_change":"2024-09-13T14:37:49.301531+0000","last_active":"2024-09-13T14:41:00.066887+0000","last_peered":"2024-09-13T14:41:00.066887+0000","last_clean":"2024-09-13T14:41:00.066887+0000","last_became_active":"2024-09-13T14:37:49.301249+0000","last_became_peered":"2024-09-13T14:37:49.301249+0000","last_unstale":"2024-09-13T14:41:00.066887+0000","last_undegraded":"2024-09-13T14:41:00.066887+0000","last_fullsized":"2024-09-13T14:41:00.066887+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:10.977710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":95,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.073060+0000","last_change":"2024-09-13T14:38:53.901427+0000","last_active":"2024-09-13T14:41:00.073060+0000","last_peered":"2024-09-13T14:41:00.073060+0000","last_clean":"2024-09-13T14:41:00.073060+0000","last_became_active":"2024-09-13T14:38:53.901157+0000","last_became_peered":"2024-09-13T14:38:53.901157+0000","last_unstale":"2024-09-13T14:41:00.073060+0000","last_undegraded":"2024-09-13T14:41:00.073060+0000","last_fullsized":"2024-09-13T14:41:00.073060+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:28:21.713025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"269'9008","reported_seq":23059,"reported_epoch":269,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-13T14:41:08.829470+0000","last_change":"2024-09-13T14:38:16.552896+0000","last_active":"2024-09-13T14:41:08.829470+0000","last_peered":"2024-09-13T14:41:08.829470+0000","last_clean":"2024-09-13T14:37:46.405415+0000","last_became_active":"2024-09-13T14:38:16.427510+0000","last_became_peered":"2024-09-13T14:38:16.427510+0000","last_unstale":"2024-09-13T14:41:08.829470+0000","last_undegraded":"2024-09-13T14:38:12.738464+0000","last_fullsized":"2024-09-13T14:41:08.829470+0000","mapping_epoch":245,"log_start":"212'3118","ondisk_log_start":"212'3118","created":205,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5890,"log_dups_size":0,"ondisk_log_size":5890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327024640,"num_objects":4990,"num_object_clones":0,"num_object_copies":14970,"num_objects_missing_on_primary":100,"num_objects_missing":100,"num_objects_degraded":4572,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9008,"num_write_kb":447936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6999,"num_bytes_recovered":458686464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"0,3,6","objects":585},{"shards":"0,3,7","objects":4405}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":977,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.072910+0000","last_change":"2024-09-13T14:37:49.147498+0000","last_active":"2024-09-13T14:41:00.072910+0000","last_peered":"2024-09-13T14:41:00.072910+0000","last_clean":"2024-09-13T14:41:00.072910+0000","last_became_active":"2024-09-13T14:37:49.147028+0000","last_became_peered":"2024-09-13T14:37:49.147028+0000","last_unstale":"2024-09-13T14:41:00.072910+0000","last_undegraded":"2024-09-13T14:41:00.072910+0000","last_fullsized":"2024-09-13T14:41:00.072910+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:11:29.050178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":275,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.058741+0000","last_change":"2024-09-13T14:38:03.528133+0000","last_active":"2024-09-13T14:41:00.058741+0000","last_peered":"2024-09-13T14:41:00.058741+0000","last_clean":"2024-09-13T14:41:00.058741+0000","last_became_active":"2024-09-13T14:38:03.527829+0000","last_became_peered":"2024-09-13T14:38:03.527829+0000","last_unstale":"2024-09-13T14:41:00.058741+0000","last_undegraded":"2024-09-13T14:41:00.058741+0000","last_fullsized":"2024-09-13T14:41:00.058741+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:48.254169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":423,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:41:10.106416+0000","last_change":"2024-09-13T14:38:53.899755+0000","last_active":"2024-09-13T14:41:10.106416+0000","last_peered":"2024-09-13T14:41:10.106416+0000","last_clean":"2024-09-13T14:41:10.106416+0000","last_became_active":"2024-09-13T14:38:53.899448+0000","last_became_peered":"2024-09-13T14:38:53.899448+0000","last_unstale":"2024-09-13T14:41:10.106416+0000","last_undegraded":"2024-09-13T14:41:10.106416+0000","last_fullsized":"2024-09-13T14:41:10.106416+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:55.897035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"270'9274","reported_seq":9368,"reported_epoch":270,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:41:12.605819+0000","last_change":"2024-09-13T14:38:03.567892+0000","last_active":"2024-09-13T14:41:12.605819+0000","last_peered":"2024-09-13T14:41:12.605819+0000","last_clean":"2024-09-13T14:35:51.090791+0000","last_became_active":"2024-09-13T14:37:49.665025+0000","last_became_peered":"2024-09-13T14:37:49.665025+0000","last_unstale":"2024-09-13T14:41:12.605819+0000","last_undegraded":"2024-09-13T14:37:49.169101+0000","last_fullsized":"2024-09-13T14:37:49.121596+0000","mapping_epoch":240,"log_start":"212'3263","ondisk_log_start":"212'3263","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6011,"log_dups_size":0,"ondisk_log_size":6011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:02.321504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":329515008,"num_objects":5028,"num_object_clones":0,"num_object_copies":15084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4501,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9274,"num_write_kb":457664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":593}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":79,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.072831+0000","last_change":"2024-09-13T14:37:49.149770+0000","last_active":"2024-09-13T14:41:00.072831+0000","last_peered":"2024-09-13T14:41:00.072831+0000","last_clean":"2024-09-13T14:41:00.072831+0000","last_became_active":"2024-09-13T14:37:49.148991+0000","last_became_peered":"2024-09-13T14:37:49.148991+0000","last_unstale":"2024-09-13T14:41:00.072831+0000","last_undegraded":"2024-09-13T14:41:00.072831+0000","last_fullsized":"2024-09-13T14:41:00.072831+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:04:04.193366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":1120,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.076693+0000","last_change":"2024-09-13T14:30:58.928302+0000","last_active":"2024-09-13T14:41:00.076693+0000","last_peered":"2024-09-13T14:41:00.076693+0000","last_clean":"2024-09-13T14:41:00.076693+0000","last_became_active":"2024-09-13T14:30:58.927718+0000","last_became_peered":"2024-09-13T14:30:58.927718+0000","last_unstale":"2024-09-13T14:41:00.076693+0000","last_undegraded":"2024-09-13T14:41:00.076693+0000","last_fullsized":"2024-09-13T14:41:00.076693+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:37.323036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":86,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.073105+0000","last_change":"2024-09-13T14:38:03.534628+0000","last_active":"2024-09-13T14:41:00.073105+0000","last_peered":"2024-09-13T14:41:00.073105+0000","last_clean":"2024-09-13T14:41:00.073105+0000","last_became_active":"2024-09-13T14:38:03.534262+0000","last_became_peered":"2024-09-13T14:38:03.534262+0000","last_unstale":"2024-09-13T14:41:00.073105+0000","last_undegraded":"2024-09-13T14:41:00.073105+0000","last_fullsized":"2024-09-13T14:41:00.073105+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:31.168783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"270'9018","reported_seq":23174,"reported_epoch":270,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:41:11.856817+0000","last_change":"2024-09-13T14:38:53.944111+0000","last_active":"2024-09-13T14:41:11.856817+0000","last_peered":"2024-09-13T14:41:11.856817+0000","last_clean":"2024-09-13T14:37:47.047672+0000","last_became_active":"2024-09-13T14:37:49.664294+0000","last_became_peered":"2024-09-13T14:37:49.664294+0000","last_unstale":"2024-09-13T14:41:11.856817+0000","last_undegraded":"2024-09-13T14:38:53.940822+0000","last_fullsized":"2024-09-13T14:37:49.126627+0000","mapping_epoch":240,"log_start":"212'3300","ondisk_log_start":"212'3300","created":205,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5718,"log_dups_size":0,"ondisk_log_size":5718,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:11.766692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":329908224,"num_objects":5034,"num_object_clones":0,"num_object_copies":15102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4535,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9018,"num_write_kb":449664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7009,"num_bytes_recovered":459341824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3],"avail_no_missing":["5","0","3"],"object_location_counts":[{"shards":"0,3,5","objects":4460},{"shards":"3,5","objects":574}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":98,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.073366+0000","last_change":"2024-09-13T14:38:03.534303+0000","last_active":"2024-09-13T14:41:00.073366+0000","last_peered":"2024-09-13T14:41:00.073366+0000","last_clean":"2024-09-13T14:41:00.073366+0000","last_became_active":"2024-09-13T14:38:03.533171+0000","last_became_peered":"2024-09-13T14:38:03.533171+0000","last_unstale":"2024-09-13T14:41:00.073366+0000","last_undegraded":"2024-09-13T14:41:00.073366+0000","last_fullsized":"2024-09-13T14:41:00.073366+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:00:16.553526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":790,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.061302+0000","last_change":"2024-09-13T14:38:04.596192+0000","last_active":"2024-09-13T14:41:00.061302+0000","last_peered":"2024-09-13T14:41:00.061302+0000","last_clean":"2024-09-13T14:41:00.061302+0000","last_became_active":"2024-09-13T14:38:04.593299+0000","last_became_peered":"2024-09-13T14:38:04.593299+0000","last_unstale":"2024-09-13T14:41:00.061302+0000","last_undegraded":"2024-09-13T14:41:00.061302+0000","last_fullsized":"2024-09-13T14:41:00.061302+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:34.241410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":61,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.067077+0000","last_change":"2024-09-13T14:38:03.533120+0000","last_active":"2024-09-13T14:41:00.067077+0000","last_peered":"2024-09-13T14:41:00.067077+0000","last_clean":"2024-09-13T14:41:00.067077+0000","last_became_active":"2024-09-13T14:38:03.532829+0000","last_became_peered":"2024-09-13T14:38:03.532829+0000","last_unstale":"2024-09-13T14:41:00.067077+0000","last_undegraded":"2024-09-13T14:41:00.067077+0000","last_fullsized":"2024-09-13T14:41:00.067077+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:24.514661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"270'9214","reported_seq":9364,"reported_epoch":270,"state":"active+undersized+degraded+remapped+backfill_wait","last_fresh":"2024-09-13T14:41:12.333379+0000","last_change":"2024-09-13T14:37:49.295861+0000","last_active":"2024-09-13T14:41:12.333379+0000","last_peered":"2024-09-13T14:41:12.333379+0000","last_clean":"2024-09-13T14:35:51.078708+0000","last_became_active":"2024-09-13T14:37:49.290403+0000","last_became_peered":"2024-09-13T14:37:49.290403+0000","last_unstale":"2024-09-13T14:41:12.333379+0000","last_undegraded":"2024-09-13T14:37:49.147402+0000","last_fullsized":"2024-09-13T14:37:49.117202+0000","mapping_epoch":240,"log_start":"212'3918","ondisk_log_start":"212'3918","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5296,"log_dups_size":0,"ondisk_log_size":5296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339738624,"num_objects":5184,"num_object_clones":0,"num_object_copies":15552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5184,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9214,"num_write_kb":460736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":5184}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":465,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:41:10.107714+0000","last_change":"2024-09-13T14:38:53.938227+0000","last_active":"2024-09-13T14:41:10.107714+0000","last_peered":"2024-09-13T14:41:10.107714+0000","last_clean":"2024-09-13T14:41:10.107714+0000","last_became_active":"2024-09-13T14:38:53.902867+0000","last_became_peered":"2024-09-13T14:38:53.902867+0000","last_unstale":"2024-09-13T14:41:10.107714+0000","last_undegraded":"2024-09-13T14:41:10.107714+0000","last_fullsized":"2024-09-13T14:41:10.107714+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:16.831756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":569,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.057746+0000","last_change":"2024-09-13T14:37:48.147677+0000","last_active":"2024-09-13T14:41:00.057746+0000","last_peered":"2024-09-13T14:41:00.057746+0000","last_clean":"2024-09-13T14:41:00.057746+0000","last_became_active":"2024-09-13T14:37:48.147001+0000","last_became_peered":"2024-09-13T14:37:48.147001+0000","last_unstale":"2024-09-13T14:41:00.057746+0000","last_undegraded":"2024-09-13T14:41:00.057746+0000","last_fullsized":"2024-09-13T14:41:00.057746+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:35:00.653576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":447,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:41:10.109512+0000","last_change":"2024-09-13T14:38:53.900107+0000","last_active":"2024-09-13T14:41:10.109512+0000","last_peered":"2024-09-13T14:41:10.109512+0000","last_clean":"2024-09-13T14:41:10.109512+0000","last_became_active":"2024-09-13T14:38:53.899556+0000","last_became_peered":"2024-09-13T14:38:53.899556+0000","last_unstale":"2024-09-13T14:41:10.109512+0000","last_undegraded":"2024-09-13T14:41:10.109512+0000","last_fullsized":"2024-09-13T14:41:10.109512+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:15:47.918163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"270'8936","reported_seq":17889,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:41:12.409581+0000","last_change":"2024-09-13T14:39:54.921239+0000","last_active":"2024-09-13T14:41:12.409581+0000","last_peered":"2024-09-13T14:41:12.409581+0000","last_clean":"2024-09-13T14:41:12.409581+0000","last_became_active":"2024-09-13T14:39:54.907894+0000","last_became_peered":"2024-09-13T14:39:54.907894+0000","last_unstale":"2024-09-13T14:41:12.409581+0000","last_undegraded":"2024-09-13T14:41:12.409581+0000","last_fullsized":"2024-09-13T14:41:12.409581+0000","mapping_epoch":263,"log_start":"212'3647","ondisk_log_start":"212'3647","created":205,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5289,"log_dups_size":0,"ondisk_log_size":5289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:52:04.619238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":322174976,"num_objects":4916,"num_object_clones":0,"num_object_copies":14748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8936,"num_write_kb":443264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4395,"num_bytes_recovered":283770880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":76,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.067403+0000","last_change":"2024-09-13T14:38:53.901806+0000","last_active":"2024-09-13T14:41:00.067403+0000","last_peered":"2024-09-13T14:41:00.067403+0000","last_clean":"2024-09-13T14:41:00.067403+0000","last_became_active":"2024-09-13T14:38:53.891055+0000","last_became_peered":"2024-09-13T14:38:53.891055+0000","last_unstale":"2024-09-13T14:41:00.067403+0000","last_undegraded":"2024-09-13T14:41:00.067403+0000","last_fullsized":"2024-09-13T14:41:00.067403+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:03:00.416198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":26,"reported_epoch":269,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-13T14:41:07.412449+0000","last_change":"2024-09-13T14:38:53.900902+0000","last_active":"2024-09-13T14:41:07.412449+0000","last_peered":"2024-09-13T14:41:07.412449+0000","last_clean":"2024-09-13T14:38:39.635779+0000","last_became_active":"2024-09-13T14:38:53.900626+0000","last_became_peered":"2024-09-13T14:38:53.900626+0000","last_unstale":"2024-09-13T14:41:07.412449+0000","last_undegraded":"2024-09-13T14:38:53.885778+0000","last_fullsized":"2024-09-13T14:41:07.412449+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:37:21.715778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":7,"num_objects_missing":7,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":["3","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":259,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.065749+0000","last_change":"2024-09-13T14:37:49.172218+0000","last_active":"2024-09-13T14:41:00.065749+0000","last_peered":"2024-09-13T14:41:00.065749+0000","last_clean":"2024-09-13T14:41:00.065749+0000","last_became_active":"2024-09-13T14:37:49.171718+0000","last_became_peered":"2024-09-13T14:37:49.171718+0000","last_unstale":"2024-09-13T14:41:00.065749+0000","last_undegraded":"2024-09-13T14:41:00.065749+0000","last_fullsized":"2024-09-13T14:41:00.065749+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:46:54.626521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":281,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.058540+0000","last_change":"2024-09-13T14:37:49.230094+0000","last_active":"2024-09-13T14:41:00.058540+0000","last_peered":"2024-09-13T14:41:00.058540+0000","last_clean":"2024-09-13T14:41:00.058540+0000","last_became_active":"2024-09-13T14:37:49.229393+0000","last_became_peered":"2024-09-13T14:37:49.229393+0000","last_unstale":"2024-09-13T14:41:00.058540+0000","last_undegraded":"2024-09-13T14:41:00.058540+0000","last_fullsized":"2024-09-13T14:41:00.058540+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:02:28.690318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":261,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.056885+0000","last_change":"2024-09-13T14:37:48.128447+0000","last_active":"2024-09-13T14:41:00.056885+0000","last_peered":"2024-09-13T14:41:00.056885+0000","last_clean":"2024-09-13T14:41:00.056885+0000","last_became_active":"2024-09-13T14:37:48.128143+0000","last_became_peered":"2024-09-13T14:37:48.128143+0000","last_unstale":"2024-09-13T14:41:00.056885+0000","last_undegraded":"2024-09-13T14:41:00.056885+0000","last_fullsized":"2024-09-13T14:41:00.056885+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:22.204039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":61,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.066996+0000","last_change":"2024-09-13T14:38:03.531683+0000","last_active":"2024-09-13T14:41:00.066996+0000","last_peered":"2024-09-13T14:41:00.066996+0000","last_clean":"2024-09-13T14:41:00.066996+0000","last_became_active":"2024-09-13T14:38:03.531476+0000","last_became_peered":"2024-09-13T14:38:03.531476+0000","last_unstale":"2024-09-13T14:41:00.066996+0000","last_undegraded":"2024-09-13T14:41:00.066996+0000","last_fullsized":"2024-09-13T14:41:00.066996+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:41.557569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":33,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.060986+0000","last_change":"2024-09-13T14:38:53.954281+0000","last_active":"2024-09-13T14:41:00.060986+0000","last_peered":"2024-09-13T14:41:00.060986+0000","last_clean":"2024-09-13T14:41:00.060986+0000","last_became_active":"2024-09-13T14:38:53.900912+0000","last_became_peered":"2024-09-13T14:38:53.900912+0000","last_unstale":"2024-09-13T14:41:00.060986+0000","last_undegraded":"2024-09-13T14:41:00.060986+0000","last_fullsized":"2024-09-13T14:41:00.060986+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:47.876275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":86,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:06.962019+0000","last_change":"2024-09-13T14:38:54.000782+0000","last_active":"2024-09-13T14:41:06.962019+0000","last_peered":"2024-09-13T14:41:06.962019+0000","last_clean":"2024-09-13T14:41:06.962019+0000","last_became_active":"2024-09-13T14:38:54.000322+0000","last_became_peered":"2024-09-13T14:38:54.000322+0000","last_unstale":"2024-09-13T14:41:06.962019+0000","last_undegraded":"2024-09-13T14:41:06.962019+0000","last_fullsized":"2024-09-13T14:41:06.962019+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:09.057597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":389,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:41:10.106101+0000","last_change":"2024-09-13T14:38:53.899535+0000","last_active":"2024-09-13T14:41:10.106101+0000","last_peered":"2024-09-13T14:41:10.106101+0000","last_clean":"2024-09-13T14:41:10.106101+0000","last_became_active":"2024-09-13T14:38:53.899183+0000","last_became_peered":"2024-09-13T14:38:53.899183+0000","last_unstale":"2024-09-13T14:41:10.106101+0000","last_undegraded":"2024-09-13T14:41:10.106101+0000","last_fullsized":"2024-09-13T14:41:10.106101+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:33.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":61,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.068697+0000","last_change":"2024-09-13T14:38:03.543426+0000","last_active":"2024-09-13T14:41:00.068697+0000","last_peered":"2024-09-13T14:41:00.068697+0000","last_clean":"2024-09-13T14:41:00.068697+0000","last_became_active":"2024-09-13T14:38:03.542661+0000","last_became_peered":"2024-09-13T14:38:03.542661+0000","last_unstale":"2024-09-13T14:41:00.068697+0000","last_undegraded":"2024-09-13T14:41:00.068697+0000","last_fullsized":"2024-09-13T14:41:00.068697+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:54.050127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":418,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:41:10.109511+0000","last_change":"2024-09-13T14:37:48.131321+0000","last_active":"2024-09-13T14:41:10.109511+0000","last_peered":"2024-09-13T14:41:10.109511+0000","last_clean":"2024-09-13T14:41:10.109511+0000","last_became_active":"2024-09-13T14:37:48.130998+0000","last_became_peered":"2024-09-13T14:37:48.130998+0000","last_unstale":"2024-09-13T14:41:10.109511+0000","last_undegraded":"2024-09-13T14:41:10.109511+0000","last_fullsized":"2024-09-13T14:41:10.109511+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:33:26.597825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":98,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.072984+0000","last_change":"2024-09-13T14:38:03.535454+0000","last_active":"2024-09-13T14:41:00.072984+0000","last_peered":"2024-09-13T14:41:00.072984+0000","last_clean":"2024-09-13T14:41:00.072984+0000","last_became_active":"2024-09-13T14:38:03.535131+0000","last_became_peered":"2024-09-13T14:38:03.535131+0000","last_unstale":"2024-09-13T14:41:00.072984+0000","last_undegraded":"2024-09-13T14:41:00.072984+0000","last_fullsized":"2024-09-13T14:41:00.072984+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:24.477601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":442,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:41:10.109651+0000","last_change":"2024-09-13T14:38:03.524939+0000","last_active":"2024-09-13T14:41:10.109651+0000","last_peered":"2024-09-13T14:41:10.109651+0000","last_clean":"2024-09-13T14:41:10.109651+0000","last_became_active":"2024-09-13T14:38:03.524406+0000","last_became_peered":"2024-09-13T14:38:03.524406+0000","last_unstale":"2024-09-13T14:41:10.109651+0000","last_undegraded":"2024-09-13T14:41:10.109651+0000","last_fullsized":"2024-09-13T14:41:10.109651+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:19:45.257424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":68,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.067328+0000","last_change":"2024-09-13T14:38:53.892948+0000","last_active":"2024-09-13T14:41:00.067328+0000","last_peered":"2024-09-13T14:41:00.067328+0000","last_clean":"2024-09-13T14:41:00.067328+0000","last_became_active":"2024-09-13T14:38:53.892665+0000","last_became_peered":"2024-09-13T14:38:53.892665+0000","last_unstale":"2024-09-13T14:41:00.067328+0000","last_undegraded":"2024-09-13T14:41:00.067328+0000","last_fullsized":"2024-09-13T14:41:00.067328+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:08.292359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":391,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:41:10.106262+0000","last_change":"2024-09-13T14:37:48.139348+0000","last_active":"2024-09-13T14:41:10.106262+0000","last_peered":"2024-09-13T14:41:10.106262+0000","last_clean":"2024-09-13T14:41:10.106262+0000","last_became_active":"2024-09-13T14:37:48.139115+0000","last_became_peered":"2024-09-13T14:37:48.139115+0000","last_unstale":"2024-09-13T14:41:10.106262+0000","last_undegraded":"2024-09-13T14:41:10.106262+0000","last_fullsized":"2024-09-13T14:41:10.106262+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:32:20.413346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":431,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:41:10.106999+0000","last_change":"2024-09-13T14:37:48.172088+0000","last_active":"2024-09-13T14:41:10.106999+0000","last_peered":"2024-09-13T14:41:10.106999+0000","last_clean":"2024-09-13T14:41:10.106999+0000","last_became_active":"2024-09-13T14:37:48.171779+0000","last_became_peered":"2024-09-13T14:37:48.171779+0000","last_unstale":"2024-09-13T14:41:10.106999+0000","last_undegraded":"2024-09-13T14:41:10.106999+0000","last_fullsized":"2024-09-13T14:41:10.106999+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:02.852712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":325,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.056908+0000","last_change":"2024-09-13T14:38:53.890236+0000","last_active":"2024-09-13T14:41:00.056908+0000","last_peered":"2024-09-13T14:41:00.056908+0000","last_clean":"2024-09-13T14:41:00.056908+0000","last_became_active":"2024-09-13T14:38:53.889582+0000","last_became_peered":"2024-09-13T14:38:53.889582+0000","last_unstale":"2024-09-13T14:41:00.056908+0000","last_undegraded":"2024-09-13T14:41:00.056908+0000","last_fullsized":"2024-09-13T14:41:00.056908+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:14:29.334774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":417,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:41:10.106704+0000","last_change":"2024-09-13T14:38:03.534097+0000","last_active":"2024-09-13T14:41:10.106704+0000","last_peered":"2024-09-13T14:41:10.106704+0000","last_clean":"2024-09-13T14:41:10.106704+0000","last_became_active":"2024-09-13T14:38:03.533794+0000","last_became_peered":"2024-09-13T14:38:03.533794+0000","last_unstale":"2024-09-13T14:41:10.106704+0000","last_undegraded":"2024-09-13T14:41:10.106704+0000","last_fullsized":"2024-09-13T14:41:10.106704+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:23.272098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":61,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.066980+0000","last_change":"2024-09-13T14:38:03.527021+0000","last_active":"2024-09-13T14:41:00.066980+0000","last_peered":"2024-09-13T14:41:00.066980+0000","last_clean":"2024-09-13T14:41:00.066980+0000","last_became_active":"2024-09-13T14:38:03.526698+0000","last_became_peered":"2024-09-13T14:38:03.526698+0000","last_unstale":"2024-09-13T14:41:00.066980+0000","last_undegraded":"2024-09-13T14:41:00.066980+0000","last_fullsized":"2024-09-13T14:41:00.066980+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:01:35.734086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":269,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.058467+0000","last_change":"2024-09-13T14:27:12.834931+0000","last_active":"2024-09-13T14:41:00.058467+0000","last_peered":"2024-09-13T14:41:00.058467+0000","last_clean":"2024-09-13T14:41:00.058467+0000","last_became_active":"2024-09-13T14:27:12.834572+0000","last_became_peered":"2024-09-13T14:27:12.834572+0000","last_unstale":"2024-09-13T14:41:00.058467+0000","last_undegraded":"2024-09-13T14:41:00.058467+0000","last_fullsized":"2024-09-13T14:41:00.058467+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:54:21.155689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":82,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.073337+0000","last_change":"2024-09-13T14:37:49.176796+0000","last_active":"2024-09-13T14:41:00.073337+0000","last_peered":"2024-09-13T14:41:00.073337+0000","last_clean":"2024-09-13T14:41:00.073337+0000","last_became_active":"2024-09-13T14:37:49.176246+0000","last_became_peered":"2024-09-13T14:37:49.176246+0000","last_unstale":"2024-09-13T14:41:00.073337+0000","last_undegraded":"2024-09-13T14:41:00.073337+0000","last_fullsized":"2024-09-13T14:41:00.073337+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:29:24.936718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":280,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.056749+0000","last_change":"2024-09-13T14:38:03.526333+0000","last_active":"2024-09-13T14:41:00.056749+0000","last_peered":"2024-09-13T14:41:00.056749+0000","last_clean":"2024-09-13T14:41:00.056749+0000","last_became_active":"2024-09-13T14:38:03.526001+0000","last_became_peered":"2024-09-13T14:38:03.526001+0000","last_unstale":"2024-09-13T14:41:00.056749+0000","last_undegraded":"2024-09-13T14:41:00.056749+0000","last_fullsized":"2024-09-13T14:41:00.056749+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:25:25.181271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":82,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.072952+0000","last_change":"2024-09-13T14:37:49.149582+0000","last_active":"2024-09-13T14:41:00.072952+0000","last_peered":"2024-09-13T14:41:00.072952+0000","last_clean":"2024-09-13T14:41:00.072952+0000","last_became_active":"2024-09-13T14:37:49.148525+0000","last_became_peered":"2024-09-13T14:37:49.148525+0000","last_unstale":"2024-09-13T14:41:00.072952+0000","last_undegraded":"2024-09-13T14:41:00.072952+0000","last_fullsized":"2024-09-13T14:41:00.072952+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:48.655665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":463,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:41:10.106503+0000","last_change":"2024-09-13T14:38:03.535255+0000","last_active":"2024-09-13T14:41:10.106503+0000","last_peered":"2024-09-13T14:41:10.106503+0000","last_clean":"2024-09-13T14:41:10.106503+0000","last_became_active":"2024-09-13T14:38:03.533917+0000","last_became_peered":"2024-09-13T14:38:03.533917+0000","last_unstale":"2024-09-13T14:41:10.106503+0000","last_undegraded":"2024-09-13T14:41:10.106503+0000","last_fullsized":"2024-09-13T14:41:10.106503+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:33.559788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":427,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:41:10.108695+0000","last_change":"2024-09-13T14:37:48.146784+0000","last_active":"2024-09-13T14:41:10.108695+0000","last_peered":"2024-09-13T14:41:10.108695+0000","last_clean":"2024-09-13T14:41:10.108695+0000","last_became_active":"2024-09-13T14:37:48.146491+0000","last_became_peered":"2024-09-13T14:37:48.146491+0000","last_unstale":"2024-09-13T14:41:10.108695+0000","last_undegraded":"2024-09-13T14:41:10.108695+0000","last_fullsized":"2024-09-13T14:41:10.108695+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:30:54.772513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":261,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.056219+0000","last_change":"2024-09-13T14:37:48.127607+0000","last_active":"2024-09-13T14:41:00.056219+0000","last_peered":"2024-09-13T14:41:00.056219+0000","last_clean":"2024-09-13T14:41:00.056219+0000","last_became_active":"2024-09-13T14:37:48.127316+0000","last_became_peered":"2024-09-13T14:37:48.127316+0000","last_unstale":"2024-09-13T14:41:00.056219+0000","last_undegraded":"2024-09-13T14:41:00.056219+0000","last_fullsized":"2024-09-13T14:41:00.056219+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:53:37.678861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":67,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-13T14:41:00.068596+0000","last_change":"2024-09-13T14:38:03.563200+0000","last_active":"2024-09-13T14:41:00.068596+0000","last_peered":"2024-09-13T14:41:00.068596+0000","last_clean":"2024-09-13T14:41:00.068596+0000","last_became_active":"2024-09-13T14:38:03.534258+0000","last_became_peered":"2024-09-13T14:38:03.534258+0000","last_unstale":"2024-09-13T14:41:00.068596+0000","last_undegraded":"2024-09-13T14:41:00.068596+0000","last_fullsized":"2024-09-13T14:41:00.068596+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:49:07.814419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":5269028888,"num_objects":80400,"num_object_clones":0,"num_object_copies":241200,"num_objects_missing_on_primary":2954,"num_objects_missing":2954,"num_objects_degraded":29357,"num_objects_misplaced":5126,"num_objects_unfound":0,"num_objects_dirty":80400,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145226,"num_write_kb":7219969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60340,"num_bytes_recovered":3918659608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16997298176,"data_stored":16997285960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92449,"ondisk_log_size":92449,"up":48,"acting":43,"num_store_stats":6},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1538,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":17412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25132,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":7,"num_objects_missing":7,"num_objects_degraded":16,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16793600,"data_stored":16777292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":23,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234368,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3106444,"kb_used_data":2664324,"kb_used_omap":1,"kb_used_meta":442110,"kb_avail":90638708,"statfs":{"total":95995035648,"available":92814036992,"internally_reserved":0,"allocated":2728267776,"data_stored":2726798190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":452721099},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":239,"seq":1026497183786,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1920344,"kb_used_data":1749708,"kb_used_omap":8,"kb_used_meta":170615,"kb_avail":91824808,"statfs":{"total":95995035648,"available":94028603392,"internally_reserved":0,"allocated":1791700992,"data_stored":1790224329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8453,"internal_metadata":174710523},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757633,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2328580,"kb_used_data":1697784,"kb_used_omap":1,"kb_used_meta":630782,"kb_avail":91416572,"statfs":{"total":95995035648,"available":93610569728,"internally_reserved":0,"allocated":1738530816,"data_stored":1737069239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":645921222},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,1,2,1,1,4,4],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019277,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3341516,"kb_used_data":2768964,"kb_used_omap":1,"kb_used_meta":572542,"kb_avail":90403636,"statfs":{"total":95995035648,"available":92573323264,"internally_reserved":0,"allocated":2835419136,"data_stored":2833950082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":586283462},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435460,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4422852,"kb_used_data":3859452,"kb_used_omap":5,"kb_used_meta":563386,"kb_avail":89322300,"statfs":{"total":95995035648,"available":91466035200,"internally_reserved":0,"allocated":3952078848,"data_stored":3950613505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":576907390},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":446676598988,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":425201762514,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926038,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4590268,"kb_used_data":3918580,"kb_used_omap":7,"kb_used_meta":671672,"kb_avail":89154884,"statfs":{"total":95995035648,"available":91294601216,"internally_reserved":0,"allocated":4012625920,"data_stored":4011160413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":687792424},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6860,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4000256000,"data_stored":4000251928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3942977536,"data_stored":3942973464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2827026432,"data_stored":2827026432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1728380928,"data_stored":1728380928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1778126848,"data_stored":1778122776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2720530432,"data_stored":2720530432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:41:15.758 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph status --format=json 2024-09-13T14:41:15.766 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:15.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:15.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:15.868 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:15.890 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:15.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:15.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:15.992 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:16.013 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:16.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:16.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:16.115 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:16.137 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:16.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:16.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:16.138 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:16 smithi195 ceph-mon[104262]: from='client.35784 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:41:16.150 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:41:16.239 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:16.259 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:16.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:16.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:16.361 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:16.386 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:16.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:16.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:16.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:16 smithi102 ceph-mon[115269]: from='client.35784 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:41:16.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:16 smithi102 ceph-mon[110183]: from='client.35784 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:41:16.488 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:16.510 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:16.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:16.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:16.612 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:16.635 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:16.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:16.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:16.737 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:16.742 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:41:16.742 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:41:16.763 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:16.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:16.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:16.814 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:41:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:41:16] "GET /metrics HTTP/1.1" 200 36539 "" "Prometheus/2.51.0" 2024-09-13T14:41:16.865 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:16.891 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:16.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:16.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:16.993 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:17.023 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:17.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:17.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:17.125 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:17.147 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:17.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:17.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:17.249 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:17.273 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:17.275 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:17.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:17.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:17.377 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:17.379 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:17 smithi102 ceph-mon[110183]: pgmap v898: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.9 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 187 op/s; 29313/240252 objects degraded (12.201%); 5128/240252 objects misplaced (2.134%); 2.5 MiB/s, 40 objects/s recovering 2024-09-13T14:41:17.379 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:17 smithi102 ceph-mon[115269]: pgmap v898: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.9 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 187 op/s; 29313/240252 objects degraded (12.201%); 5128/240252 objects misplaced (2.134%); 2.5 MiB/s, 40 objects/s recovering 2024-09-13T14:41:17.403 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:17.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:17.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:17.441 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:17 smithi195 ceph-mon[104262]: pgmap v898: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.9 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 187 op/s; 29313/240252 objects degraded (12.201%); 5128/240252 objects misplaced (2.134%); 2.5 MiB/s, 40 objects/s recovering 2024-09-13T14:41:17.478 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T14:41:17.506 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:17.509 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:17.532 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:17.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:17.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:17.634 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:17.658 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:17.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:17.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:17.745 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:41:17.759 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:17.784 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:17.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:17.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:17.886 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:17.913 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:17.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:17.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:18.015 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:18.039 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:18.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:18.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:18.094 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is set 2024-09-13T14:41:18.137 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:18 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2061918619' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:41:18.141 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:18.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:18.167 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:18.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:18.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:18 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2061918619' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:41:18.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:18 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2061918619' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:41:18.269 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:18.298 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:18.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:18.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:18.400 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:18.424 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:18.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:18.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:18.526 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:18.551 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:18.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:18.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:18.653 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:18.681 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:18.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:18.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:18.783 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:18.808 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:18.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:18.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:18.911 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:18.936 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:18.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:18.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:19.039 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:19.064 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:19.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:19.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:19.166 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:19.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:19 smithi102 ceph-mon[110183]: pgmap v899: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.9 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 185 op/s; 29179/238536 objects degraded (12.233%); 5079/238536 objects misplaced (2.129%); 2.4 MiB/s, 38 objects/s recovering 2024-09-13T14:41:19.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:19 smithi102 ceph-mon[110183]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:41:19.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:19 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2061918619' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:41:19.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:19 smithi102 ceph-mon[110183]: osdmap e271: 8 total, 8 up, 6 in 2024-09-13T14:41:19.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:19 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 29179/238536 objects degraded (12.233%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:41:19.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:19 smithi102 ceph-mon[115269]: pgmap v899: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.9 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 185 op/s; 29179/238536 objects degraded (12.233%); 5079/238536 objects misplaced (2.129%); 2.4 MiB/s, 38 objects/s recovering 2024-09-13T14:41:19.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:19 smithi102 ceph-mon[115269]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:41:19.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:19 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2061918619' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:41:19.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:19 smithi102 ceph-mon[115269]: osdmap e271: 8 total, 8 up, 6 in 2024-09-13T14:41:19.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:19 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 29179/238536 objects degraded (12.233%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:41:19.195 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:19.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:19.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:19.297 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:19.322 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:19.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:19.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:19.424 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:19.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:19.454 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:19.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:19.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:19 smithi195 ceph-mon[104262]: pgmap v899: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.9 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 185 op/s; 29179/238536 objects degraded (12.233%); 5079/238536 objects misplaced (2.129%); 2.4 MiB/s, 38 objects/s recovering 2024-09-13T14:41:19.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:19 smithi195 ceph-mon[104262]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:41:19.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:19 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2061918619' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:41:19.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:19 smithi195 ceph-mon[104262]: osdmap e271: 8 total, 8 up, 6 in 2024-09-13T14:41:19.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:19 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 29179/238536 objects degraded (12.233%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:41:19.555 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:19.577 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:19.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:19.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:19.680 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:19.706 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:19.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:19.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:19.808 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:19.837 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:19.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:19.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:19.939 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:19.962 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:19.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:19.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:20.064 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:20.089 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:20.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:20.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:20.191 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:20.219 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:20.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:20.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:20.321 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:20.347 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:20.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:20.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:20.399 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:41:20.400 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:41:20.450 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:20.474 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:20.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:20.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:20.576 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:20.600 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:20.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:20.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:20.702 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:20.724 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:20.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:20.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:20.826 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:20.850 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:20.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:20.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:20.952 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:20.974 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:20.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:20.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:21.076 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:21.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:21.101 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:21.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:21.128 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:21 smithi102 ceph-mon[110183]: pgmap v901: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.8 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 204 op/s; 29107/237195 objects degraded (12.271%); 5080/237195 objects misplaced (2.142%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:41:21.128 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:21 smithi102 ceph-mon[115269]: pgmap v901: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.8 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 204 op/s; 29107/237195 objects degraded (12.271%); 5080/237195 objects misplaced (2.142%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:41:21.202 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:21.208 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T14:41:21.224 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:21.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:21.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:21.326 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:21.351 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:21.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:21.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:21.386 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:41:21.386 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:41:21.406 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:41:21.406 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:41:21.424 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:41:21.424 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:41:21.442 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:41:21.443 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:41:21.453 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:21.462 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:41:21.462 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:41:21.464 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:41:21.477 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:21.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:21.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:21.483 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:41:21.483 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:41:21.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:21 smithi195 ceph-mon[104262]: pgmap v901: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.8 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 204 op/s; 29107/237195 objects degraded (12.271%); 5080/237195 objects misplaced (2.142%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:41:21.501 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:41:21.501 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:41:21.519 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:41:21.519 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:41:21.579 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:21.605 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:21.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:21.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:21.707 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:21.731 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:21.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:21.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:21.833 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:21.848 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:41:21.860 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:21.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:21.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:21.925 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:41:21.925 INFO:teuthology.orchestra.run.smithi102.stdout:{"fsid":"08cee502-71d9-11ef-bceb-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29107/237195 objects degraded (12.271%), 10 pgs degraded, 6 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1305,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":271,"num_osds":8,"num_up_osds":8,"osd_up_since":1726238267,"num_in_osds":6,"osd_in_since":1726238173,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":78},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovery_wait+degraded","count":3},{"state_name":"active+recovery_wait+degraded+remapped","count":2},{"state_name":"active+undersized+degraded+remapped+backfill_wait","count":1},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":89,"num_pools":5,"num_objects":79065,"data_bytes":5184493565,"bytes_used":20001726464,"bytes_avail":555968487424,"bytes_total":575970213888,"degraded_objects":29107,"degraded_total":237195,"degraded_ratio":0.12271337928708446,"misplaced_objects":5080,"misplaced_total":237195,"misplaced_ratio":0.021416977592276395,"recovering_objects_per_sec":43,"recovering_bytes_per_sec":2855085,"recovering_keys_per_sec":0,"num_objects_recovered":410,"num_bytes_recovered":26869760,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":204},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi102.njetxr","status":"up:active","gid":24521}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.102:8443/","prometheus":"http://172.21.15.102:9283/"}},"servicemap":{"epoch":28,"modified":"2024-09-13T14:40:21.476600+0000","services":{}},"progress_events":{}} 2024-09-13T14:41:21.962 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:21.989 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:21.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:21.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:22.091 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:22.123 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:22.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:22.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:22.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:22 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/366660157' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:41:22.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:22 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1860421966' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:41:22.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:22 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/366660157' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:41:22.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:22 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1860421966' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:41:22.225 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:22.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:22 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/366660157' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:41:22.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:22 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1860421966' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:41:22.250 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:22.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:22.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:22.352 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:22.373 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:22.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:22.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:22.475 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:22.500 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:22.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:22.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:22.602 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:22.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:22.624 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:22.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:22.725 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:22.750 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:22.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:22.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:22.852 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:22.876 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:22.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:22.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:22.978 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:23.001 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:23.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:23.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:23.103 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:23.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:23.129 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:23.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:23.131 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-13T14:41:23.132 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:41:23.197 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:23.230 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:23.256 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:23.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:23.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:23.353 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:23 smithi102 ceph-mon[110183]: pgmap v902: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.8 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 192 op/s; 29107/237195 objects degraded (12.271%); 5080/237195 objects misplaced (2.142%); 2.6 MiB/s, 40 objects/s recovering 2024-09-13T14:41:23.353 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:23 smithi102 ceph-mon[115269]: pgmap v902: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.8 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 192 op/s; 29107/237195 objects degraded (12.271%); 5080/237195 objects misplaced (2.142%); 2.6 MiB/s, 40 objects/s recovering 2024-09-13T14:41:23.357 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:23.380 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:23.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:23.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:23.482 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:23.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:23 smithi195 ceph-mon[104262]: pgmap v902: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.8 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 192 op/s; 29107/237195 objects degraded (12.271%); 5080/237195 objects misplaced (2.142%); 2.6 MiB/s, 40 objects/s recovering 2024-09-13T14:41:23.508 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:23.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:23.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:23.613 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:23.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:23.639 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:23.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:23.740 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:23.765 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:23.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:23.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:23.867 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:23.892 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:23.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:23.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:23.976 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:23.993 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:24.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:24.022 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:24.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:24.123 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:24.149 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:24.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:24.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:24.251 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:24.274 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:24.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:24.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:24.375 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:24.405 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:24.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:24.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:24.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:24 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 29107/237195 objects degraded (12.271%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:41:24.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:24 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 29107/237195 objects degraded (12.271%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:41:24.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:24 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 29107/237195 objects degraded (12.271%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:41:24.507 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:24.532 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:24.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:24.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:24.634 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:24.659 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:24.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:24.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:24.666 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:41:24.761 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:24.784 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:24.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:24.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:24.886 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:24.908 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:24.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:24.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:25.010 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:25.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:25.040 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:25.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:25.141 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:25.164 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:25.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:25.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:25.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:25 smithi102 ceph-mon[115269]: pgmap v903: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.8 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 210 op/s; 28964/234963 objects degraded (12.327%); 4831/234963 objects misplaced (2.056%); 3.6 MiB/s, 57 objects/s recovering 2024-09-13T14:41:25.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:25 smithi102 ceph-mon[110183]: pgmap v903: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.8 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 210 op/s; 28964/234963 objects degraded (12.327%); 4831/234963 objects misplaced (2.056%); 3.6 MiB/s, 57 objects/s recovering 2024-09-13T14:41:25.266 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:25.294 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:25.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:25.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:25.395 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:25.419 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:25.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:25.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:25.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:25 smithi195 ceph-mon[104262]: pgmap v903: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.8 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 210 op/s; 28964/234963 objects degraded (12.327%); 4831/234963 objects misplaced (2.056%); 3.6 MiB/s, 57 objects/s recovering 2024-09-13T14:41:25.521 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:25.546 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:25.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:25.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:25.648 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:25.681 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:25.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:25.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:25.783 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:25.806 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:25.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:25.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:25.908 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:25.931 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:25.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:25.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:26.033 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:26.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:26.062 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:26.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:26.124 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is unset 2024-09-13T14:41:26.162 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:26.191 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:26.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:26.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:26.293 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:26.320 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:26.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:26.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:26.371 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:41:26.371 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:26 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/397674316' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:41:26.371 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:26 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:41:26.372 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:41:26.372 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:26 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/397674316' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:41:26.372 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:26 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:41:26.421 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:26.442 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:41:26.443 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:26 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/397674316' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:41:26.443 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:26 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:41:26.449 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:26.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:26.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:26.551 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:26.575 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:26.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:26.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:26.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:41:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:41:26] "GET /metrics HTTP/1.1" 200 36539 "" "Prometheus/2.51.0" 2024-09-13T14:41:26.678 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:26.705 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:26.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:26.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:26.807 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:26.823 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:41:26.823 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-13T14:41:26.824 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-09-13T14:41:26.824 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-09-13T14:41:26.824 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:41:26.835 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:26.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:26.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:26.937 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:26.961 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:26.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:26.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:27.063 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:27.091 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:27.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:27.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:27.193 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:27.194 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:27.215 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:27.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:27.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:27.317 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:27.341 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:27.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:27.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:27.383 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:27 smithi102 ceph-mon[110183]: pgmap v904: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.8 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 222 op/s; 28867/233574 objects degraded (12.359%); 4830/233574 objects misplaced (2.068%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:41:27.383 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:27 smithi102 ceph-mon[110183]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:41:27.383 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:27 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:41:27.384 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:27 smithi102 ceph-mon[110183]: osdmap e272: 8 total, 8 up, 6 in 2024-09-13T14:41:27.384 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:27 smithi102 ceph-mon[115269]: pgmap v904: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.8 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 222 op/s; 28867/233574 objects degraded (12.359%); 4830/233574 objects misplaced (2.068%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:41:27.384 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:27 smithi102 ceph-mon[115269]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:41:27.384 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:27 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:41:27.384 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:27 smithi102 ceph-mon[115269]: osdmap e272: 8 total, 8 up, 6 in 2024-09-13T14:41:27.443 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:27.466 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:27.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:27.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:27.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:27 smithi195 ceph-mon[104262]: pgmap v904: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.8 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 222 op/s; 28867/233574 objects degraded (12.359%); 4830/233574 objects misplaced (2.068%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:41:27.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:27 smithi195 ceph-mon[104262]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:41:27.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:27 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:41:27.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:27 smithi195 ceph-mon[104262]: osdmap e272: 8 total, 8 up, 6 in 2024-09-13T14:41:27.568 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:27.586 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T14:41:27.592 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:27.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:27.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:27.694 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:27.722 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:27.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:27.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:27.824 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:27.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:27.853 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:27.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:27.883 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:41:27.954 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:27.978 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:27.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:27.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:28.080 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:28.105 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:28.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:28.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:28.207 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:28.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:28.232 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:28.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:28.333 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:28.363 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:28.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:28.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:28.447 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:41:28.448 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:41:28.465 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:28.490 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:28.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:28.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:28.592 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:28.618 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:28.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:28.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:28.720 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:28.743 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:28.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:28.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:28.845 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:28.873 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:28.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:28.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:28.974 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:28.998 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:28.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:28.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:29.100 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:29.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:29.123 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:29.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:29.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:29 smithi102 ceph-mon[115269]: pgmap v906: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.7 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 236 op/s; 28726/231660 objects degraded (12.400%); 4409/231660 objects misplaced (1.903%); 6.5 MiB/s, 104 objects/s recovering 2024-09-13T14:41:29.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:29 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 28726/231660 objects degraded (12.400%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:41:29.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:29 smithi102 ceph-mon[110183]: pgmap v906: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.7 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 236 op/s; 28726/231660 objects degraded (12.400%); 4409/231660 objects misplaced (1.903%); 6.5 MiB/s, 104 objects/s recovering 2024-09-13T14:41:29.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:29 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 28726/231660 objects degraded (12.400%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:41:29.224 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:29.246 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:29.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:29.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:29.348 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:29.360 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":906,"stamp":"2024-09-13T14:41:27.496012+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5063579645,"num_objects":77220,"num_object_clones":0,"num_object_copies":231660,"num_objects_missing_on_primary":2743,"num_objects_missing":2743,"num_objects_degraded":28726,"num_objects_misplaced":4409,"num_objects_unfound":0,"num_objects_dirty":77220,"num_whiteouts":0,"num_read":6377,"num_read_kb":13834,"num_write":149557,"num_write_kb":7235010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61616,"num_bytes_recovered":4001238157,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93839,"ondisk_log_size":93839,"up":267,"acting":261,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":309,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":19209304,"kb_used_data":16136232,"kb_used_omap":26,"kb_used_meta":3072933,"kb_avail":543261608,"statfs":{"total":575970213888,"available":556299886592,"internally_reserved":0,"allocated":16523501568,"data_stored":16514658963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27144,"internal_metadata":3146683896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,0,1,1,0,1,3,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-145883136,"num_objects":-2226,"num_object_clones":0,"num_object_copies":-6678,"num_objects_missing_on_primary":-143,"num_objects_missing":-143,"num_objects_degraded":-453,"num_objects_misplaced":-670,"num_objects_unfound":0,"num_objects_dirty":-2226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2226,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":984,"num_bytes_recovered":64487424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.401925"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":437,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:41:26.135273+0000","last_change":"2024-09-13T14:36:13.859927+0000","last_active":"2024-09-13T14:41:26.135273+0000","last_peered":"2024-09-13T14:41:26.135273+0000","last_clean":"2024-09-13T14:41:26.135273+0000","last_became_active":"2024-09-13T14:36:13.859505+0000","last_became_peered":"2024-09-13T14:36:13.859505+0000","last_unstale":"2024-09-13T14:41:26.135273+0000","last_undegraded":"2024-09-13T14:41:26.135273+0000","last_fullsized":"2024-09-13T14:41:26.135273+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:12:20.989627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":273,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:18.121069+0000","last_change":"2024-09-13T14:37:48.130968+0000","last_active":"2024-09-13T14:41:18.121069+0000","last_peered":"2024-09-13T14:41:18.121069+0000","last_clean":"2024-09-13T14:41:18.121069+0000","last_became_active":"2024-09-13T14:37:48.130603+0000","last_became_peered":"2024-09-13T14:37:48.130603+0000","last_unstale":"2024-09-13T14:41:18.121069+0000","last_undegraded":"2024-09-13T14:41:18.121069+0000","last_fullsized":"2024-09-13T14:41:18.121069+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:42.626386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":70,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:18.109202+0000","last_change":"2024-09-13T14:36:13.862181+0000","last_active":"2024-09-13T14:41:18.109202+0000","last_peered":"2024-09-13T14:41:18.109202+0000","last_clean":"2024-09-13T14:41:18.109202+0000","last_became_active":"2024-09-13T14:36:13.861873+0000","last_became_peered":"2024-09-13T14:36:13.861873+0000","last_unstale":"2024-09-13T14:41:18.109202+0000","last_undegraded":"2024-09-13T14:41:18.109202+0000","last_fullsized":"2024-09-13T14:41:18.109202+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:14.085700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":72,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:18.095972+0000","last_change":"2024-09-13T14:38:53.896672+0000","last_active":"2024-09-13T14:41:18.095972+0000","last_peered":"2024-09-13T14:41:18.095972+0000","last_clean":"2024-09-13T14:41:18.095972+0000","last_became_active":"2024-09-13T14:38:53.896346+0000","last_became_peered":"2024-09-13T14:38:53.896346+0000","last_unstale":"2024-09-13T14:41:18.095972+0000","last_undegraded":"2024-09-13T14:41:18.095972+0000","last_fullsized":"2024-09-13T14:41:18.095972+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:57.789686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":72,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:18.096687+0000","last_change":"2024-09-13T14:38:53.893021+0000","last_active":"2024-09-13T14:41:18.096687+0000","last_peered":"2024-09-13T14:41:18.096687+0000","last_clean":"2024-09-13T14:41:18.096687+0000","last_became_active":"2024-09-13T14:38:53.892668+0000","last_became_peered":"2024-09-13T14:38:53.892668+0000","last_unstale":"2024-09-13T14:41:18.096687+0000","last_undegraded":"2024-09-13T14:41:18.096687+0000","last_fullsized":"2024-09-13T14:41:18.096687+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:57.180738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":241,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:18.121309+0000","last_change":"2024-09-13T14:37:48.134642+0000","last_active":"2024-09-13T14:41:18.121309+0000","last_peered":"2024-09-13T14:41:18.121309+0000","last_clean":"2024-09-13T14:41:18.121309+0000","last_became_active":"2024-09-13T14:37:48.134402+0000","last_became_peered":"2024-09-13T14:37:48.134402+0000","last_unstale":"2024-09-13T14:41:18.121309+0000","last_undegraded":"2024-09-13T14:41:18.121309+0000","last_fullsized":"2024-09-13T14:41:18.121309+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:51.313663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":455,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:41:26.136643+0000","last_change":"2024-09-13T14:38:53.903022+0000","last_active":"2024-09-13T14:41:26.136643+0000","last_peered":"2024-09-13T14:41:26.136643+0000","last_clean":"2024-09-13T14:41:26.136643+0000","last_became_active":"2024-09-13T14:38:53.902448+0000","last_became_peered":"2024-09-13T14:38:53.902448+0000","last_unstale":"2024-09-13T14:41:26.136643+0000","last_undegraded":"2024-09-13T14:41:26.136643+0000","last_fullsized":"2024-09-13T14:41:26.136643+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:24.690401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":75,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:18.095406+0000","last_change":"2024-09-13T14:38:03.554500+0000","last_active":"2024-09-13T14:41:18.095406+0000","last_peered":"2024-09-13T14:41:18.095406+0000","last_clean":"2024-09-13T14:41:18.095406+0000","last_became_active":"2024-09-13T14:38:03.532910+0000","last_became_peered":"2024-09-13T14:38:03.532910+0000","last_unstale":"2024-09-13T14:41:18.095406+0000","last_undegraded":"2024-09-13T14:41:18.095406+0000","last_fullsized":"2024-09-13T14:41:18.095406+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:27:50.843498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"272'9206","reported_seq":15080,"reported_epoch":272,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-13T14:41:27.066375+0000","last_change":"2024-09-13T14:40:32.799848+0000","last_active":"2024-09-13T14:41:27.066375+0000","last_peered":"2024-09-13T14:41:27.066375+0000","last_clean":"2024-09-13T14:37:46.417003+0000","last_became_active":"2024-09-13T14:37:49.690961+0000","last_became_peered":"2024-09-13T14:37:49.690961+0000","last_unstale":"2024-09-13T14:41:27.066375+0000","last_undegraded":"2024-09-13T14:41:27.066375+0000","last_fullsized":"2024-09-13T14:37:49.126746+0000","mapping_epoch":240,"log_start":"212'3500","ondisk_log_start":"212'3500","created":205,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5706,"log_dups_size":0,"ondisk_log_size":5706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:48:52.526429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306315264,"num_objects":4674,"num_object_clones":0,"num_object_copies":14022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2420,"num_objects_unfound":0,"num_objects_dirty":4674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9206,"num_write_kb":444160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6653,"num_bytes_recovered":436011008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,4],"avail_no_missing":["5","0","4"],"object_location_counts":[{"shards":"0,4,5","objects":2420},{"shards":"4,5","objects":2254}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":34,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:18.120812+0000","last_change":"2024-09-13T14:38:53.940238+0000","last_active":"2024-09-13T14:41:18.120812+0000","last_peered":"2024-09-13T14:41:18.120812+0000","last_clean":"2024-09-13T14:41:18.120812+0000","last_became_active":"2024-09-13T14:38:53.892825+0000","last_became_peered":"2024-09-13T14:38:53.892825+0000","last_unstale":"2024-09-13T14:41:18.120812+0000","last_undegraded":"2024-09-13T14:41:18.120812+0000","last_fullsized":"2024-09-13T14:41:18.120812+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:46.666581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":90,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:18.108795+0000","last_change":"2024-09-13T14:38:53.898907+0000","last_active":"2024-09-13T14:41:18.108795+0000","last_peered":"2024-09-13T14:41:18.108795+0000","last_clean":"2024-09-13T14:41:18.108795+0000","last_became_active":"2024-09-13T14:38:53.898572+0000","last_became_peered":"2024-09-13T14:38:53.898572+0000","last_unstale":"2024-09-13T14:41:18.108795+0000","last_undegraded":"2024-09-13T14:41:18.108795+0000","last_fullsized":"2024-09-13T14:41:18.108795+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:29.222150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"272'9375","reported_seq":23528,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:41:27.352040+0000","last_change":"2024-09-13T14:38:57.930032+0000","last_active":"2024-09-13T14:41:27.352040+0000","last_peered":"2024-09-13T14:41:27.352040+0000","last_clean":"2024-09-13T14:41:27.352040+0000","last_became_active":"2024-09-13T14:38:57.921458+0000","last_became_peered":"2024-09-13T14:38:57.921458+0000","last_unstale":"2024-09-13T14:41:27.352040+0000","last_undegraded":"2024-09-13T14:41:27.352040+0000","last_fullsized":"2024-09-13T14:41:27.352040+0000","mapping_epoch":256,"log_start":"212'4039","ondisk_log_start":"212'4039","created":205,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5336,"log_dups_size":0,"ondisk_log_size":5336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:27:12.294038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318963712,"num_objects":4867,"num_object_clones":0,"num_object_copies":14601,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9375,"num_write_kb":455744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6990,"num_bytes_recovered":458096640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":34,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:18.121290+0000","last_change":"2024-09-13T14:38:53.904885+0000","last_active":"2024-09-13T14:41:18.121290+0000","last_peered":"2024-09-13T14:41:18.121290+0000","last_clean":"2024-09-13T14:41:18.121290+0000","last_became_active":"2024-09-13T14:38:53.891726+0000","last_became_peered":"2024-09-13T14:38:53.891726+0000","last_unstale":"2024-09-13T14:41:18.121290+0000","last_undegraded":"2024-09-13T14:41:18.121290+0000","last_fullsized":"2024-09-13T14:41:18.121290+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:57.079205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":278,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:41:10.101108+0000","last_change":"2024-09-13T14:37:48.160504+0000","last_active":"2024-09-13T14:41:10.101108+0000","last_peered":"2024-09-13T14:41:10.101108+0000","last_clean":"2024-09-13T14:41:10.101108+0000","last_became_active":"2024-09-13T14:37:48.160181+0000","last_became_peered":"2024-09-13T14:37:48.160181+0000","last_unstale":"2024-09-13T14:41:10.101108+0000","last_undegraded":"2024-09-13T14:41:10.101108+0000","last_fullsized":"2024-09-13T14:41:10.101108+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:37.087008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"271'9304","reported_seq":11077,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:24.046059+0000","last_change":"2024-09-13T14:38:04.531914+0000","last_active":"2024-09-13T14:41:24.046059+0000","last_peered":"2024-09-13T14:41:24.046059+0000","last_clean":"2024-09-13T14:41:24.046059+0000","last_became_active":"2024-09-13T14:38:04.523448+0000","last_became_peered":"2024-09-13T14:38:04.523448+0000","last_unstale":"2024-09-13T14:41:24.046059+0000","last_undegraded":"2024-09-13T14:41:24.046059+0000","last_fullsized":"2024-09-13T14:41:24.046059+0000","mapping_epoch":245,"log_start":"212'3400","ondisk_log_start":"212'3400","created":205,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5904,"log_dups_size":0,"ondisk_log_size":5904,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:06.536896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319160320,"num_objects":4870,"num_object_clones":0,"num_object_copies":14610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9304,"num_write_kb":453568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":811,"num_bytes_recovered":53149696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":271,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:18.121705+0000","last_change":"2024-09-13T14:37:49.611774+0000","last_active":"2024-09-13T14:41:18.121705+0000","last_peered":"2024-09-13T14:41:18.121705+0000","last_clean":"2024-09-13T14:41:18.121705+0000","last_became_active":"2024-09-13T14:37:49.611088+0000","last_became_peered":"2024-09-13T14:37:49.611088+0000","last_unstale":"2024-09-13T14:41:18.121705+0000","last_undegraded":"2024-09-13T14:41:18.121705+0000","last_fullsized":"2024-09-13T14:41:18.121705+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:08.933213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":30,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:18.122063+0000","last_change":"2024-09-13T14:38:53.899211+0000","last_active":"2024-09-13T14:41:18.122063+0000","last_peered":"2024-09-13T14:41:18.122063+0000","last_clean":"2024-09-13T14:41:18.122063+0000","last_became_active":"2024-09-13T14:38:53.898927+0000","last_became_peered":"2024-09-13T14:38:53.898927+0000","last_unstale":"2024-09-13T14:41:18.122063+0000","last_undegraded":"2024-09-13T14:41:18.122063+0000","last_fullsized":"2024-09-13T14:41:18.122063+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:00:28.857611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"272'9213","reported_seq":9291,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:41:27.160586+0000","last_change":"2024-09-13T14:35:24.004651+0000","last_active":"2024-09-13T14:41:27.160586+0000","last_peered":"2024-09-13T14:41:27.160586+0000","last_clean":"2024-09-13T14:41:27.160586+0000","last_became_active":"2024-09-13T14:35:24.004358+0000","last_became_peered":"2024-09-13T14:35:24.004358+0000","last_unstale":"2024-09-13T14:41:27.160586+0000","last_undegraded":"2024-09-13T14:41:27.160586+0000","last_fullsized":"2024-09-13T14:41:27.160586+0000","mapping_epoch":205,"log_start":"212'2884","ondisk_log_start":"212'2884","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6329,"log_dups_size":0,"ondisk_log_size":6329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:57:38.023669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":311885824,"num_objects":4759,"num_object_clones":0,"num_object_copies":14277,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9213,"num_write_kb":447104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":81,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:18.109225+0000","last_change":"2024-09-13T14:37:49.168003+0000","last_active":"2024-09-13T14:41:18.109225+0000","last_peered":"2024-09-13T14:41:18.109225+0000","last_clean":"2024-09-13T14:41:18.109225+0000","last_became_active":"2024-09-13T14:37:49.167638+0000","last_became_peered":"2024-09-13T14:37:49.167638+0000","last_unstale":"2024-09-13T14:41:18.109225+0000","last_undegraded":"2024-09-13T14:41:18.109225+0000","last_fullsized":"2024-09-13T14:41:18.109225+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:29.051981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":65,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:18.096027+0000","last_change":"2024-09-13T14:38:03.528381+0000","last_active":"2024-09-13T14:41:18.096027+0000","last_peered":"2024-09-13T14:41:18.096027+0000","last_clean":"2024-09-13T14:41:18.096027+0000","last_became_active":"2024-09-13T14:38:03.527282+0000","last_became_peered":"2024-09-13T14:38:03.527282+0000","last_unstale":"2024-09-13T14:41:18.096027+0000","last_undegraded":"2024-09-13T14:41:18.096027+0000","last_fullsized":"2024-09-13T14:41:18.096027+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:24.435067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.f","version":"271'9223","reported_seq":9491,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:23.770104+0000","last_change":"2024-09-13T14:38:16.371079+0000","last_active":"2024-09-13T14:41:23.770104+0000","last_peered":"2024-09-13T14:41:23.770104+0000","last_clean":"2024-09-13T14:41:23.770104+0000","last_became_active":"2024-09-13T14:38:12.859883+0000","last_became_peered":"2024-09-13T14:38:12.859883+0000","last_unstale":"2024-09-13T14:41:23.770104+0000","last_undegraded":"2024-09-13T14:41:23.770104+0000","last_fullsized":"2024-09-13T14:41:23.770104+0000","mapping_epoch":245,"log_start":"212'3263","ondisk_log_start":"212'3263","created":205,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5960,"log_dups_size":0,"ondisk_log_size":5960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:08.436532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318701568,"num_objects":4863,"num_object_clones":0,"num_object_copies":14589,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9223,"num_write_kb":450752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":3670016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":425,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:41:26.136342+0000","last_change":"2024-09-13T14:38:03.549595+0000","last_active":"2024-09-13T14:41:26.136342+0000","last_peered":"2024-09-13T14:41:26.136342+0000","last_clean":"2024-09-13T14:41:26.136342+0000","last_became_active":"2024-09-13T14:38:03.533331+0000","last_became_peered":"2024-09-13T14:38:03.533331+0000","last_unstale":"2024-09-13T14:41:26.136342+0000","last_undegraded":"2024-09-13T14:41:26.136342+0000","last_fullsized":"2024-09-13T14:41:26.136342+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:59:02.470343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":254,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:41:10.101377+0000","last_change":"2024-09-13T14:37:48.127600+0000","last_active":"2024-09-13T14:41:10.101377+0000","last_peered":"2024-09-13T14:41:10.101377+0000","last_clean":"2024-09-13T14:41:10.101377+0000","last_became_active":"2024-09-13T14:37:48.127349+0000","last_became_peered":"2024-09-13T14:37:48.127349+0000","last_unstale":"2024-09-13T14:41:10.101377+0000","last_undegraded":"2024-09-13T14:41:10.101377+0000","last_fullsized":"2024-09-13T14:41:10.101377+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:57.268374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"272'9234","reported_seq":13801,"reported_epoch":272,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:41:27.060975+0000","last_change":"2024-09-13T14:38:14.979229+0000","last_active":"2024-09-13T14:41:27.060975+0000","last_peered":"2024-09-13T14:41:27.060975+0000","last_clean":"2024-09-13T14:36:13.131873+0000","last_became_active":"2024-09-13T14:38:10.443954+0000","last_became_peered":"2024-09-13T14:38:10.443954+0000","last_unstale":"2024-09-13T14:41:27.060975+0000","last_undegraded":"2024-09-13T14:38:10.384643+0000","last_fullsized":"2024-09-13T14:38:04.517046+0000","mapping_epoch":245,"log_start":"212'3500","ondisk_log_start":"212'3500","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5734,"log_dups_size":0,"ondisk_log_size":5734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:26:53.493170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":312999936,"num_objects":4776,"num_object_clones":0,"num_object_copies":14328,"num_objects_missing_on_primary":2553,"num_objects_missing":2553,"num_objects_degraded":2694,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9234,"num_write_kb":448320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2224,"num_bytes_recovered":145686528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":1},{"shards":"0,3,5,7","objects":8},{"shards":"0,3,7","objects":2552},{"shards":"0,5","objects":2210},{"shards":"3,5,7","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":187,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:41:10.101294+0000","last_change":"2024-09-13T14:37:48.127604+0000","last_active":"2024-09-13T14:41:10.101294+0000","last_peered":"2024-09-13T14:41:10.101294+0000","last_clean":"2024-09-13T14:41:10.101294+0000","last_became_active":"2024-09-13T14:37:48.127300+0000","last_became_peered":"2024-09-13T14:37:48.127300+0000","last_unstale":"2024-09-13T14:41:10.101294+0000","last_undegraded":"2024-09-13T14:41:10.101294+0000","last_fullsized":"2024-09-13T14:41:10.101294+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:51.999506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":307,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:41:10.102025+0000","last_change":"2024-09-13T14:38:03.529967+0000","last_active":"2024-09-13T14:41:10.102025+0000","last_peered":"2024-09-13T14:41:10.102025+0000","last_clean":"2024-09-13T14:41:10.102025+0000","last_became_active":"2024-09-13T14:38:03.529646+0000","last_became_peered":"2024-09-13T14:38:03.529646+0000","last_unstale":"2024-09-13T14:41:10.102025+0000","last_undegraded":"2024-09-13T14:41:10.102025+0000","last_fullsized":"2024-09-13T14:41:10.102025+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:52:05.193279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"271'9294","reported_seq":18437,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:24.018055+0000","last_change":"2024-09-13T14:40:34.583237+0000","last_active":"2024-09-13T14:41:24.018055+0000","last_peered":"2024-09-13T14:41:24.018055+0000","last_clean":"2024-09-13T14:41:24.018055+0000","last_became_active":"2024-09-13T14:40:34.563989+0000","last_became_peered":"2024-09-13T14:40:34.563989+0000","last_unstale":"2024-09-13T14:41:24.018055+0000","last_undegraded":"2024-09-13T14:41:24.018055+0000","last_fullsized":"2024-09-13T14:41:24.018055+0000","mapping_epoch":266,"log_start":"212'3400","ondisk_log_start":"212'3400","created":205,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5894,"log_dups_size":0,"ondisk_log_size":5894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:34:43.006051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":317521944,"num_objects":4846,"num_object_clones":0,"num_object_copies":14538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4846,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9294,"num_write_kb":452417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4489,"num_bytes_recovered":289865752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":267,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:41:10.101370+0000","last_change":"2024-09-13T14:36:13.860367+0000","last_active":"2024-09-13T14:41:10.101370+0000","last_peered":"2024-09-13T14:41:10.101370+0000","last_clean":"2024-09-13T14:41:10.101370+0000","last_became_active":"2024-09-13T14:36:13.859982+0000","last_became_peered":"2024-09-13T14:36:13.859982+0000","last_unstale":"2024-09-13T14:41:10.101370+0000","last_undegraded":"2024-09-13T14:41:10.101370+0000","last_fullsized":"2024-09-13T14:41:10.101370+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:08:23.311350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":263,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:41:10.101674+0000","last_change":"2024-09-13T14:37:48.147360+0000","last_active":"2024-09-13T14:41:10.101674+0000","last_peered":"2024-09-13T14:41:10.101674+0000","last_clean":"2024-09-13T14:41:10.101674+0000","last_became_active":"2024-09-13T14:37:48.146885+0000","last_became_peered":"2024-09-13T14:37:48.146885+0000","last_unstale":"2024-09-13T14:41:10.101674+0000","last_undegraded":"2024-09-13T14:41:10.101674+0000","last_fullsized":"2024-09-13T14:41:10.101674+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:27.841810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"272'9262","reported_seq":18514,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:41:27.398434+0000","last_change":"2024-09-13T14:38:39.650332+0000","last_active":"2024-09-13T14:41:27.398434+0000","last_peered":"2024-09-13T14:41:27.398434+0000","last_clean":"2024-09-13T14:41:27.398434+0000","last_became_active":"2024-09-13T14:38:39.640835+0000","last_became_peered":"2024-09-13T14:38:39.640835+0000","last_unstale":"2024-09-13T14:41:27.398434+0000","last_undegraded":"2024-09-13T14:41:27.398434+0000","last_fullsized":"2024-09-13T14:41:27.398434+0000","mapping_epoch":251,"log_start":"212'3951","ondisk_log_start":"212'3951","created":205,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5311,"log_dups_size":0,"ondisk_log_size":5311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:48.176606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":315359232,"num_objects":4812,"num_object_clones":0,"num_object_copies":14436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9262,"num_write_kb":450368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4544,"num_bytes_recovered":292945920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":443,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:41:26.135124+0000","last_change":"2024-09-13T14:38:53.900041+0000","last_active":"2024-09-13T14:41:26.135124+0000","last_peered":"2024-09-13T14:41:26.135124+0000","last_clean":"2024-09-13T14:41:26.135124+0000","last_became_active":"2024-09-13T14:38:53.899203+0000","last_became_peered":"2024-09-13T14:38:53.899203+0000","last_unstale":"2024-09-13T14:41:26.135124+0000","last_undegraded":"2024-09-13T14:41:26.135124+0000","last_fullsized":"2024-09-13T14:41:26.135124+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:28:07.883094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":375,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:41:26.136777+0000","last_change":"2024-09-13T14:26:15.823198+0000","last_active":"2024-09-13T14:41:26.136777+0000","last_peered":"2024-09-13T14:41:26.136777+0000","last_clean":"2024-09-13T14:41:26.136777+0000","last_became_active":"2024-09-13T14:26:15.822717+0000","last_became_peered":"2024-09-13T14:26:15.822717+0000","last_unstale":"2024-09-13T14:41:26.136777+0000","last_undegraded":"2024-09-13T14:41:26.136777+0000","last_fullsized":"2024-09-13T14:41:26.136777+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:22.471622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.6","version":"271'9371","reported_seq":13862,"reported_epoch":271,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-13T14:41:24.148233+0000","last_change":"2024-09-13T14:38:55.132683+0000","last_active":"2024-09-13T14:41:24.148233+0000","last_peered":"2024-09-13T14:41:24.148233+0000","last_clean":"2024-09-13T14:36:13.124368+0000","last_became_active":"2024-09-13T14:38:55.120883+0000","last_became_peered":"2024-09-13T14:38:55.120883+0000","last_unstale":"2024-09-13T14:41:24.148233+0000","last_undegraded":"2024-09-13T14:38:55.026610+0000","last_fullsized":"2024-09-13T14:41:24.148233+0000","mapping_epoch":254,"log_start":"212'3419","ondisk_log_start":"212'3419","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5952,"log_dups_size":0,"ondisk_log_size":5952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316342272,"num_objects":4827,"num_object_clones":0,"num_object_copies":14481,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2840,"num_objects_misplaced":1987,"num_objects_unfound":0,"num_objects_dirty":4827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9371,"num_write_kb":454336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2165,"num_bytes_recovered":130678784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,3,5],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3,4","objects":2561},{"shards":"0,3,5","objects":2266}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":439,"reported_epoch":272,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-13T14:41:26.136952+0000","last_change":"2024-09-13T14:38:54.899624+0000","last_active":"2024-09-13T14:41:26.136952+0000","last_peered":"2024-09-13T14:41:26.136952+0000","last_clean":"2024-09-13T14:17:27.401592+0000","last_became_active":"2024-09-13T14:38:54.899282+0000","last_became_peered":"2024-09-13T14:38:54.899282+0000","last_unstale":"2024-09-13T14:41:26.136952+0000","last_undegraded":"2024-09-13T14:38:54.891176+0000","last_fullsized":"2024-09-13T14:41:26.136952+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:07:41.927740+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[3,0,6],"avail_no_missing":["3","4","6"],"object_location_counts":[{"shards":"3,4,6","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":510,"reported_epoch":271,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:41:18.109180+0000","last_change":"2024-09-13T14:38:54.904329+0000","last_active":"2024-09-13T14:41:18.109180+0000","last_peered":"2024-09-13T14:41:18.109180+0000","last_clean":"2024-09-13T14:38:39.648017+0000","last_became_active":"2024-09-13T14:38:54.903798+0000","last_became_peered":"2024-09-13T14:38:54.903798+0000","last_unstale":"2024-09-13T14:41:18.109180+0000","last_undegraded":"2024-09-13T14:38:54.890505+0000","last_fullsized":"2024-09-13T14:38:54.889701+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:18.120931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,3],"avail_no_missing":["5","3","6"],"object_location_counts":[{"shards":"3,5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":377,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:41:26.137677+0000","last_change":"2024-09-13T14:37:49.160719+0000","last_active":"2024-09-13T14:41:26.137677+0000","last_peered":"2024-09-13T14:41:26.137677+0000","last_clean":"2024-09-13T14:41:26.137677+0000","last_became_active":"2024-09-13T14:37:49.160401+0000","last_became_peered":"2024-09-13T14:37:49.160401+0000","last_unstale":"2024-09-13T14:41:26.137677+0000","last_undegraded":"2024-09-13T14:41:26.137677+0000","last_fullsized":"2024-09-13T14:41:26.137677+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:21.737945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":114,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:41:10.101977+0000","last_change":"2024-09-13T14:37:48.172940+0000","last_active":"2024-09-13T14:41:10.101977+0000","last_peered":"2024-09-13T14:41:10.101977+0000","last_clean":"2024-09-13T14:41:10.101977+0000","last_became_active":"2024-09-13T14:37:48.172060+0000","last_became_peered":"2024-09-13T14:37:48.172060+0000","last_unstale":"2024-09-13T14:41:10.101977+0000","last_undegraded":"2024-09-13T14:41:10.101977+0000","last_fullsized":"2024-09-13T14:41:10.101977+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:07.480645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.5","version":"272'9280","reported_seq":25634,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:41:27.197781+0000","last_change":"2024-09-13T14:39:34.446699+0000","last_active":"2024-09-13T14:41:27.197781+0000","last_peered":"2024-09-13T14:41:27.197781+0000","last_clean":"2024-09-13T14:41:27.197781+0000","last_became_active":"2024-09-13T14:38:04.600616+0000","last_became_peered":"2024-09-13T14:38:04.600616+0000","last_unstale":"2024-09-13T14:41:27.197781+0000","last_undegraded":"2024-09-13T14:41:27.197781+0000","last_fullsized":"2024-09-13T14:41:27.197781+0000","mapping_epoch":245,"log_start":"212'3016","ondisk_log_start":"212'3016","created":205,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6264,"log_dups_size":0,"ondisk_log_size":6264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:00:44.321678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318636032,"num_objects":4862,"num_object_clones":0,"num_object_copies":14586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9280,"num_write_kb":452544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8120,"num_bytes_recovered":521076736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":488,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:18.121352+0000","last_change":"2024-09-13T14:26:15.835178+0000","last_active":"2024-09-13T14:41:18.121352+0000","last_peered":"2024-09-13T14:41:18.121352+0000","last_clean":"2024-09-13T14:41:18.121352+0000","last_became_active":"2024-09-13T14:26:15.820283+0000","last_became_peered":"2024-09-13T14:26:15.820283+0000","last_unstale":"2024-09-13T14:41:18.121352+0000","last_undegraded":"2024-09-13T14:41:18.121352+0000","last_fullsized":"2024-09-13T14:41:18.121352+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:30.340038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":65,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:18.095347+0000","last_change":"2024-09-13T14:38:03.525682+0000","last_active":"2024-09-13T14:41:18.095347+0000","last_peered":"2024-09-13T14:41:18.095347+0000","last_clean":"2024-09-13T14:41:18.095347+0000","last_became_active":"2024-09-13T14:38:03.525406+0000","last_became_peered":"2024-09-13T14:38:03.525406+0000","last_unstale":"2024-09-13T14:41:18.095347+0000","last_undegraded":"2024-09-13T14:41:18.095347+0000","last_fullsized":"2024-09-13T14:41:18.095347+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:41:28.970300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":72,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:18.096766+0000","last_change":"2024-09-13T14:38:53.991738+0000","last_active":"2024-09-13T14:41:18.096766+0000","last_peered":"2024-09-13T14:41:18.096766+0000","last_clean":"2024-09-13T14:41:18.096766+0000","last_became_active":"2024-09-13T14:38:53.991417+0000","last_became_peered":"2024-09-13T14:38:53.991417+0000","last_unstale":"2024-09-13T14:41:18.096766+0000","last_undegraded":"2024-09-13T14:41:18.096766+0000","last_fullsized":"2024-09-13T14:41:18.096766+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:50:38.559838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"271'9280","reported_seq":23445,"reported_epoch":271,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-13T14:41:24.074658+0000","last_change":"2024-09-13T14:38:17.472071+0000","last_active":"2024-09-13T14:41:24.074658+0000","last_peered":"2024-09-13T14:41:24.074658+0000","last_clean":"2024-09-13T14:37:46.290754+0000","last_became_active":"2024-09-13T14:38:17.381753+0000","last_became_peered":"2024-09-13T14:38:17.381753+0000","last_unstale":"2024-09-13T14:41:24.074658+0000","last_undegraded":"2024-09-13T14:38:13.504140+0000","last_fullsized":"2024-09-13T14:41:24.074658+0000","mapping_epoch":245,"log_start":"212'3359","ondisk_log_start":"212'3359","created":205,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5921,"log_dups_size":0,"ondisk_log_size":5921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316276736,"num_objects":4826,"num_object_clones":0,"num_object_copies":14478,"num_objects_missing_on_primary":83,"num_objects_missing":83,"num_objects_degraded":4582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9280,"num_write_kb":451392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7053,"num_bytes_recovered":462225408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"0,4,5","objects":4413},{"shards":"0,4,6","objects":413}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":788,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:41:26.134629+0000","last_change":"2024-09-13T14:37:48.131799+0000","last_active":"2024-09-13T14:41:26.134629+0000","last_peered":"2024-09-13T14:41:26.134629+0000","last_clean":"2024-09-13T14:41:26.134629+0000","last_became_active":"2024-09-13T14:37:48.130837+0000","last_became_peered":"2024-09-13T14:37:48.130837+0000","last_unstale":"2024-09-13T14:41:26.134629+0000","last_undegraded":"2024-09-13T14:41:26.134629+0000","last_fullsized":"2024-09-13T14:41:26.134629+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:06:32.663770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":305,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:18.096341+0000","last_change":"2024-09-13T14:37:49.301531+0000","last_active":"2024-09-13T14:41:18.096341+0000","last_peered":"2024-09-13T14:41:18.096341+0000","last_clean":"2024-09-13T14:41:18.096341+0000","last_became_active":"2024-09-13T14:37:49.301249+0000","last_became_peered":"2024-09-13T14:37:49.301249+0000","last_unstale":"2024-09-13T14:41:18.096341+0000","last_undegraded":"2024-09-13T14:41:18.096341+0000","last_fullsized":"2024-09-13T14:41:18.096341+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:10.977710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":97,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:18.109610+0000","last_change":"2024-09-13T14:38:53.901427+0000","last_active":"2024-09-13T14:41:18.109610+0000","last_peered":"2024-09-13T14:41:18.109610+0000","last_clean":"2024-09-13T14:41:18.109610+0000","last_became_active":"2024-09-13T14:38:53.901157+0000","last_became_peered":"2024-09-13T14:38:53.901157+0000","last_unstale":"2024-09-13T14:41:18.109610+0000","last_undegraded":"2024-09-13T14:41:18.109610+0000","last_fullsized":"2024-09-13T14:41:18.109610+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:28:21.713025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"271'9198","reported_seq":23253,"reported_epoch":271,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-13T14:41:24.092167+0000","last_change":"2024-09-13T14:38:16.552896+0000","last_active":"2024-09-13T14:41:24.092167+0000","last_peered":"2024-09-13T14:41:24.092167+0000","last_clean":"2024-09-13T14:37:46.405415+0000","last_became_active":"2024-09-13T14:38:16.427510+0000","last_became_peered":"2024-09-13T14:38:16.427510+0000","last_unstale":"2024-09-13T14:41:24.092167+0000","last_undegraded":"2024-09-13T14:38:12.738464+0000","last_fullsized":"2024-09-13T14:41:24.092167+0000","mapping_epoch":245,"log_start":"212'3218","ondisk_log_start":"212'3218","created":205,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5980,"log_dups_size":0,"ondisk_log_size":5980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314572800,"num_objects":4800,"num_object_clones":0,"num_object_copies":14400,"num_objects_missing_on_primary":100,"num_objects_missing":100,"num_objects_degraded":4572,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9198,"num_write_kb":447936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6999,"num_bytes_recovered":458686464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"0,3,6","objects":395},{"shards":"0,3,7","objects":4405}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":979,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:18.108867+0000","last_change":"2024-09-13T14:37:49.147498+0000","last_active":"2024-09-13T14:41:18.108867+0000","last_peered":"2024-09-13T14:41:18.108867+0000","last_clean":"2024-09-13T14:41:18.108867+0000","last_became_active":"2024-09-13T14:37:49.147028+0000","last_became_peered":"2024-09-13T14:37:49.147028+0000","last_unstale":"2024-09-13T14:41:18.108867+0000","last_undegraded":"2024-09-13T14:41:18.108867+0000","last_fullsized":"2024-09-13T14:41:18.108867+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:11:29.050178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":276,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:41:10.103220+0000","last_change":"2024-09-13T14:38:03.528133+0000","last_active":"2024-09-13T14:41:10.103220+0000","last_peered":"2024-09-13T14:41:10.103220+0000","last_clean":"2024-09-13T14:41:10.103220+0000","last_became_active":"2024-09-13T14:38:03.527829+0000","last_became_peered":"2024-09-13T14:38:03.527829+0000","last_unstale":"2024-09-13T14:41:10.103220+0000","last_undegraded":"2024-09-13T14:41:10.103220+0000","last_fullsized":"2024-09-13T14:41:10.103220+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:48.254169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":427,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:41:26.135038+0000","last_change":"2024-09-13T14:38:53.899755+0000","last_active":"2024-09-13T14:41:26.135038+0000","last_peered":"2024-09-13T14:41:26.135038+0000","last_clean":"2024-09-13T14:41:26.135038+0000","last_became_active":"2024-09-13T14:38:53.899448+0000","last_became_peered":"2024-09-13T14:38:53.899448+0000","last_unstale":"2024-09-13T14:41:26.135038+0000","last_undegraded":"2024-09-13T14:41:26.135038+0000","last_fullsized":"2024-09-13T14:41:26.135038+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:55.897035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"271'9415","reported_seq":9510,"reported_epoch":271,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:41:22.690294+0000","last_change":"2024-09-13T14:38:03.567892+0000","last_active":"2024-09-13T14:41:22.690294+0000","last_peered":"2024-09-13T14:41:22.690294+0000","last_clean":"2024-09-13T14:35:51.090791+0000","last_became_active":"2024-09-13T14:37:49.665025+0000","last_became_peered":"2024-09-13T14:37:49.665025+0000","last_unstale":"2024-09-13T14:41:22.690294+0000","last_undegraded":"2024-09-13T14:37:49.169101+0000","last_fullsized":"2024-09-13T14:37:49.121596+0000","mapping_epoch":240,"log_start":"212'3363","ondisk_log_start":"212'3363","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6052,"log_dups_size":0,"ondisk_log_size":6052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:02.321504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320274432,"num_objects":4887,"num_object_clones":0,"num_object_copies":14661,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4501,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9415,"num_write_kb":457664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":452}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":81,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:18.109094+0000","last_change":"2024-09-13T14:37:49.149770+0000","last_active":"2024-09-13T14:41:18.109094+0000","last_peered":"2024-09-13T14:41:18.109094+0000","last_clean":"2024-09-13T14:41:18.109094+0000","last_became_active":"2024-09-13T14:37:49.148991+0000","last_became_peered":"2024-09-13T14:37:49.148991+0000","last_unstale":"2024-09-13T14:41:18.109094+0000","last_undegraded":"2024-09-13T14:41:18.109094+0000","last_fullsized":"2024-09-13T14:41:18.109094+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:04:04.193366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":1124,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:18.129003+0000","last_change":"2024-09-13T14:30:58.928302+0000","last_active":"2024-09-13T14:41:18.129003+0000","last_peered":"2024-09-13T14:41:18.129003+0000","last_clean":"2024-09-13T14:41:18.129003+0000","last_became_active":"2024-09-13T14:30:58.927718+0000","last_became_peered":"2024-09-13T14:30:58.927718+0000","last_unstale":"2024-09-13T14:41:18.129003+0000","last_undegraded":"2024-09-13T14:41:18.129003+0000","last_fullsized":"2024-09-13T14:41:18.129003+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:37.323036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":88,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:18.108946+0000","last_change":"2024-09-13T14:38:03.534628+0000","last_active":"2024-09-13T14:41:18.108946+0000","last_peered":"2024-09-13T14:41:18.108946+0000","last_clean":"2024-09-13T14:41:18.108946+0000","last_became_active":"2024-09-13T14:38:03.534262+0000","last_became_peered":"2024-09-13T14:38:03.534262+0000","last_unstale":"2024-09-13T14:41:18.108946+0000","last_undegraded":"2024-09-13T14:41:18.108946+0000","last_fullsized":"2024-09-13T14:41:18.108946+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:31.168783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"272'9247","reported_seq":23405,"reported_epoch":272,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:41:26.911143+0000","last_change":"2024-09-13T14:38:53.944111+0000","last_active":"2024-09-13T14:41:26.911143+0000","last_peered":"2024-09-13T14:41:26.911143+0000","last_clean":"2024-09-13T14:37:47.047672+0000","last_became_active":"2024-09-13T14:37:49.664294+0000","last_became_peered":"2024-09-13T14:37:49.664294+0000","last_unstale":"2024-09-13T14:41:26.911143+0000","last_undegraded":"2024-09-13T14:38:53.940822+0000","last_fullsized":"2024-09-13T14:37:49.126627+0000","mapping_epoch":240,"log_start":"212'3500","ondisk_log_start":"212'3500","created":205,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5747,"log_dups_size":0,"ondisk_log_size":5747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:11.766692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314900480,"num_objects":4805,"num_object_clones":0,"num_object_copies":14415,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4535,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9247,"num_write_kb":449664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7009,"num_bytes_recovered":459341824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3],"avail_no_missing":["5","0","3"],"object_location_counts":[{"shards":"0,3,5","objects":4460},{"shards":"3,5","objects":345}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":100,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:18.109384+0000","last_change":"2024-09-13T14:38:03.534303+0000","last_active":"2024-09-13T14:41:18.109384+0000","last_peered":"2024-09-13T14:41:18.109384+0000","last_clean":"2024-09-13T14:41:18.109384+0000","last_became_active":"2024-09-13T14:38:03.533171+0000","last_became_peered":"2024-09-13T14:38:03.533171+0000","last_unstale":"2024-09-13T14:41:18.109384+0000","last_undegraded":"2024-09-13T14:41:18.109384+0000","last_fullsized":"2024-09-13T14:41:18.109384+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:00:16.553526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":792,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:18.121774+0000","last_change":"2024-09-13T14:38:04.596192+0000","last_active":"2024-09-13T14:41:18.121774+0000","last_peered":"2024-09-13T14:41:18.121774+0000","last_clean":"2024-09-13T14:41:18.121774+0000","last_became_active":"2024-09-13T14:38:04.593299+0000","last_became_peered":"2024-09-13T14:38:04.593299+0000","last_unstale":"2024-09-13T14:41:18.121774+0000","last_undegraded":"2024-09-13T14:41:18.121774+0000","last_fullsized":"2024-09-13T14:41:18.121774+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:34.241410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":65,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:18.096104+0000","last_change":"2024-09-13T14:38:03.533120+0000","last_active":"2024-09-13T14:41:18.096104+0000","last_peered":"2024-09-13T14:41:18.096104+0000","last_clean":"2024-09-13T14:41:18.096104+0000","last_became_active":"2024-09-13T14:38:03.532829+0000","last_became_peered":"2024-09-13T14:38:03.532829+0000","last_unstale":"2024-09-13T14:41:18.096104+0000","last_undegraded":"2024-09-13T14:41:18.096104+0000","last_fullsized":"2024-09-13T14:41:18.096104+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:24.514661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"272'9414","reported_seq":9568,"reported_epoch":272,"state":"active+undersized+degraded+remapped+backfill_wait","last_fresh":"2024-09-13T14:41:27.358929+0000","last_change":"2024-09-13T14:37:49.295861+0000","last_active":"2024-09-13T14:41:27.358929+0000","last_peered":"2024-09-13T14:41:27.358929+0000","last_clean":"2024-09-13T14:35:51.078708+0000","last_became_active":"2024-09-13T14:37:49.290403+0000","last_became_peered":"2024-09-13T14:37:49.290403+0000","last_unstale":"2024-09-13T14:41:27.358929+0000","last_undegraded":"2024-09-13T14:37:49.147402+0000","last_fullsized":"2024-09-13T14:37:49.117202+0000","mapping_epoch":240,"log_start":"213'4118","ondisk_log_start":"213'4118","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5296,"log_dups_size":0,"ondisk_log_size":5296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326631424,"num_objects":4984,"num_object_clones":0,"num_object_copies":14952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4984,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9414,"num_write_kb":460736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":4984}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":469,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:41:26.136866+0000","last_change":"2024-09-13T14:38:53.938227+0000","last_active":"2024-09-13T14:41:26.136866+0000","last_peered":"2024-09-13T14:41:26.136866+0000","last_clean":"2024-09-13T14:41:26.136866+0000","last_became_active":"2024-09-13T14:38:53.902867+0000","last_became_peered":"2024-09-13T14:38:53.902867+0000","last_unstale":"2024-09-13T14:41:26.136866+0000","last_undegraded":"2024-09-13T14:41:26.136866+0000","last_fullsized":"2024-09-13T14:41:26.136866+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:16.831756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":570,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:41:10.101819+0000","last_change":"2024-09-13T14:37:48.147677+0000","last_active":"2024-09-13T14:41:10.101819+0000","last_peered":"2024-09-13T14:41:10.101819+0000","last_clean":"2024-09-13T14:41:10.101819+0000","last_became_active":"2024-09-13T14:37:48.147001+0000","last_became_peered":"2024-09-13T14:37:48.147001+0000","last_unstale":"2024-09-13T14:41:10.101819+0000","last_undegraded":"2024-09-13T14:41:10.101819+0000","last_fullsized":"2024-09-13T14:41:10.101819+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:35:00.653576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":451,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:41:26.134942+0000","last_change":"2024-09-13T14:38:53.900107+0000","last_active":"2024-09-13T14:41:26.134942+0000","last_peered":"2024-09-13T14:41:26.134942+0000","last_clean":"2024-09-13T14:41:26.134942+0000","last_became_active":"2024-09-13T14:38:53.899556+0000","last_became_peered":"2024-09-13T14:38:53.899556+0000","last_unstale":"2024-09-13T14:41:26.134942+0000","last_undegraded":"2024-09-13T14:41:26.134942+0000","last_fullsized":"2024-09-13T14:41:26.134942+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:15:47.918163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"272'9118","reported_seq":18075,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:41:27.304535+0000","last_change":"2024-09-13T14:39:54.921239+0000","last_active":"2024-09-13T14:41:27.304535+0000","last_peered":"2024-09-13T14:41:27.304535+0000","last_clean":"2024-09-13T14:41:27.304535+0000","last_became_active":"2024-09-13T14:39:54.907894+0000","last_became_peered":"2024-09-13T14:39:54.907894+0000","last_unstale":"2024-09-13T14:41:27.304535+0000","last_undegraded":"2024-09-13T14:41:27.304535+0000","last_fullsized":"2024-09-13T14:41:27.304535+0000","mapping_epoch":263,"log_start":"212'3847","ondisk_log_start":"212'3847","created":205,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5271,"log_dups_size":0,"ondisk_log_size":5271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:52:04.619238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310247424,"num_objects":4734,"num_object_clones":0,"num_object_copies":14202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9118,"num_write_kb":443264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4395,"num_bytes_recovered":283770880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":80,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:18.096598+0000","last_change":"2024-09-13T14:38:53.901806+0000","last_active":"2024-09-13T14:41:18.096598+0000","last_peered":"2024-09-13T14:41:18.096598+0000","last_clean":"2024-09-13T14:41:18.096598+0000","last_became_active":"2024-09-13T14:38:53.891055+0000","last_became_peered":"2024-09-13T14:38:53.891055+0000","last_unstale":"2024-09-13T14:41:18.096598+0000","last_undegraded":"2024-09-13T14:41:18.096598+0000","last_fullsized":"2024-09-13T14:41:18.096598+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:03:00.416198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":28,"reported_epoch":271,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-13T14:41:18.120651+0000","last_change":"2024-09-13T14:38:53.900902+0000","last_active":"2024-09-13T14:41:18.120651+0000","last_peered":"2024-09-13T14:41:18.120651+0000","last_clean":"2024-09-13T14:38:39.635779+0000","last_became_active":"2024-09-13T14:38:53.900626+0000","last_became_peered":"2024-09-13T14:38:53.900626+0000","last_unstale":"2024-09-13T14:41:18.120651+0000","last_undegraded":"2024-09-13T14:38:53.885778+0000","last_fullsized":"2024-09-13T14:41:18.120651+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:37:21.715778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":7,"num_objects_missing":7,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":["3","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":260,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:41:10.101242+0000","last_change":"2024-09-13T14:37:49.172218+0000","last_active":"2024-09-13T14:41:10.101242+0000","last_peered":"2024-09-13T14:41:10.101242+0000","last_clean":"2024-09-13T14:41:10.101242+0000","last_became_active":"2024-09-13T14:37:49.171718+0000","last_became_peered":"2024-09-13T14:37:49.171718+0000","last_unstale":"2024-09-13T14:41:10.101242+0000","last_undegraded":"2024-09-13T14:41:10.101242+0000","last_fullsized":"2024-09-13T14:41:10.101242+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:46:54.626521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":282,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:41:10.103151+0000","last_change":"2024-09-13T14:37:49.230094+0000","last_active":"2024-09-13T14:41:10.103151+0000","last_peered":"2024-09-13T14:41:10.103151+0000","last_clean":"2024-09-13T14:41:10.103151+0000","last_became_active":"2024-09-13T14:37:49.229393+0000","last_became_peered":"2024-09-13T14:37:49.229393+0000","last_unstale":"2024-09-13T14:41:10.103151+0000","last_undegraded":"2024-09-13T14:41:10.103151+0000","last_fullsized":"2024-09-13T14:41:10.103151+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:02:28.690318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":262,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:41:10.101491+0000","last_change":"2024-09-13T14:37:48.128447+0000","last_active":"2024-09-13T14:41:10.101491+0000","last_peered":"2024-09-13T14:41:10.101491+0000","last_clean":"2024-09-13T14:41:10.101491+0000","last_became_active":"2024-09-13T14:37:48.128143+0000","last_became_peered":"2024-09-13T14:37:48.128143+0000","last_unstale":"2024-09-13T14:41:10.101491+0000","last_undegraded":"2024-09-13T14:41:10.101491+0000","last_fullsized":"2024-09-13T14:41:10.101491+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:22.204039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":65,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:18.095881+0000","last_change":"2024-09-13T14:38:03.531683+0000","last_active":"2024-09-13T14:41:18.095881+0000","last_peered":"2024-09-13T14:41:18.095881+0000","last_clean":"2024-09-13T14:41:18.095881+0000","last_became_active":"2024-09-13T14:38:03.531476+0000","last_became_peered":"2024-09-13T14:38:03.531476+0000","last_unstale":"2024-09-13T14:41:18.095881+0000","last_undegraded":"2024-09-13T14:41:18.095881+0000","last_fullsized":"2024-09-13T14:41:18.095881+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:41.557569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":36,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:22.414512+0000","last_change":"2024-09-13T14:38:53.954281+0000","last_active":"2024-09-13T14:41:22.414512+0000","last_peered":"2024-09-13T14:41:22.414512+0000","last_clean":"2024-09-13T14:41:22.414512+0000","last_became_active":"2024-09-13T14:38:53.900912+0000","last_became_peered":"2024-09-13T14:38:53.900912+0000","last_unstale":"2024-09-13T14:41:22.414512+0000","last_undegraded":"2024-09-13T14:41:22.414512+0000","last_fullsized":"2024-09-13T14:41:22.414512+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:47.876275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":88,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:18.116101+0000","last_change":"2024-09-13T14:38:54.000782+0000","last_active":"2024-09-13T14:41:18.116101+0000","last_peered":"2024-09-13T14:41:18.116101+0000","last_clean":"2024-09-13T14:41:18.116101+0000","last_became_active":"2024-09-13T14:38:54.000322+0000","last_became_peered":"2024-09-13T14:38:54.000322+0000","last_unstale":"2024-09-13T14:41:18.116101+0000","last_undegraded":"2024-09-13T14:41:18.116101+0000","last_fullsized":"2024-09-13T14:41:18.116101+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:09.057597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":393,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:41:26.134481+0000","last_change":"2024-09-13T14:38:53.899535+0000","last_active":"2024-09-13T14:41:26.134481+0000","last_peered":"2024-09-13T14:41:26.134481+0000","last_clean":"2024-09-13T14:41:26.134481+0000","last_became_active":"2024-09-13T14:38:53.899183+0000","last_became_peered":"2024-09-13T14:38:53.899183+0000","last_unstale":"2024-09-13T14:41:26.134481+0000","last_undegraded":"2024-09-13T14:41:26.134481+0000","last_fullsized":"2024-09-13T14:41:26.134481+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:33.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":65,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:18.100328+0000","last_change":"2024-09-13T14:38:03.543426+0000","last_active":"2024-09-13T14:41:18.100328+0000","last_peered":"2024-09-13T14:41:18.100328+0000","last_clean":"2024-09-13T14:41:18.100328+0000","last_became_active":"2024-09-13T14:38:03.542661+0000","last_became_peered":"2024-09-13T14:38:03.542661+0000","last_unstale":"2024-09-13T14:41:18.100328+0000","last_undegraded":"2024-09-13T14:41:18.100328+0000","last_fullsized":"2024-09-13T14:41:18.100328+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:54.050127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":422,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:41:26.134824+0000","last_change":"2024-09-13T14:37:48.131321+0000","last_active":"2024-09-13T14:41:26.134824+0000","last_peered":"2024-09-13T14:41:26.134824+0000","last_clean":"2024-09-13T14:41:26.134824+0000","last_became_active":"2024-09-13T14:37:48.130998+0000","last_became_peered":"2024-09-13T14:37:48.130998+0000","last_unstale":"2024-09-13T14:41:26.134824+0000","last_undegraded":"2024-09-13T14:41:26.134824+0000","last_fullsized":"2024-09-13T14:41:26.134824+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:33:26.597825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":100,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:18.109547+0000","last_change":"2024-09-13T14:38:03.535454+0000","last_active":"2024-09-13T14:41:18.109547+0000","last_peered":"2024-09-13T14:41:18.109547+0000","last_clean":"2024-09-13T14:41:18.109547+0000","last_became_active":"2024-09-13T14:38:03.535131+0000","last_became_peered":"2024-09-13T14:38:03.535131+0000","last_unstale":"2024-09-13T14:41:18.109547+0000","last_undegraded":"2024-09-13T14:41:18.109547+0000","last_fullsized":"2024-09-13T14:41:18.109547+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:24.477601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":446,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:41:26.135034+0000","last_change":"2024-09-13T14:38:03.524939+0000","last_active":"2024-09-13T14:41:26.135034+0000","last_peered":"2024-09-13T14:41:26.135034+0000","last_clean":"2024-09-13T14:41:26.135034+0000","last_became_active":"2024-09-13T14:38:03.524406+0000","last_became_peered":"2024-09-13T14:38:03.524406+0000","last_unstale":"2024-09-13T14:41:26.135034+0000","last_undegraded":"2024-09-13T14:41:26.135034+0000","last_fullsized":"2024-09-13T14:41:26.135034+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:19:45.257424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":72,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:18.096516+0000","last_change":"2024-09-13T14:38:53.892948+0000","last_active":"2024-09-13T14:41:18.096516+0000","last_peered":"2024-09-13T14:41:18.096516+0000","last_clean":"2024-09-13T14:41:18.096516+0000","last_became_active":"2024-09-13T14:38:53.892665+0000","last_became_peered":"2024-09-13T14:38:53.892665+0000","last_unstale":"2024-09-13T14:41:18.096516+0000","last_undegraded":"2024-09-13T14:41:18.096516+0000","last_fullsized":"2024-09-13T14:41:18.096516+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:08.292359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":395,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:41:26.134598+0000","last_change":"2024-09-13T14:37:48.139348+0000","last_active":"2024-09-13T14:41:26.134598+0000","last_peered":"2024-09-13T14:41:26.134598+0000","last_clean":"2024-09-13T14:41:26.134598+0000","last_became_active":"2024-09-13T14:37:48.139115+0000","last_became_peered":"2024-09-13T14:37:48.139115+0000","last_unstale":"2024-09-13T14:41:26.134598+0000","last_undegraded":"2024-09-13T14:41:26.134598+0000","last_fullsized":"2024-09-13T14:41:26.134598+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:32:20.413346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":435,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:41:26.136174+0000","last_change":"2024-09-13T14:37:48.172088+0000","last_active":"2024-09-13T14:41:26.136174+0000","last_peered":"2024-09-13T14:41:26.136174+0000","last_clean":"2024-09-13T14:41:26.136174+0000","last_became_active":"2024-09-13T14:37:48.171779+0000","last_became_peered":"2024-09-13T14:37:48.171779+0000","last_unstale":"2024-09-13T14:41:26.136174+0000","last_undegraded":"2024-09-13T14:41:26.136174+0000","last_fullsized":"2024-09-13T14:41:26.136174+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:02.852712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":326,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:41:10.101374+0000","last_change":"2024-09-13T14:38:53.890236+0000","last_active":"2024-09-13T14:41:10.101374+0000","last_peered":"2024-09-13T14:41:10.101374+0000","last_clean":"2024-09-13T14:41:10.101374+0000","last_became_active":"2024-09-13T14:38:53.889582+0000","last_became_peered":"2024-09-13T14:38:53.889582+0000","last_unstale":"2024-09-13T14:41:10.101374+0000","last_undegraded":"2024-09-13T14:41:10.101374+0000","last_fullsized":"2024-09-13T14:41:10.101374+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:14:29.334774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":421,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:41:26.135431+0000","last_change":"2024-09-13T14:38:03.534097+0000","last_active":"2024-09-13T14:41:26.135431+0000","last_peered":"2024-09-13T14:41:26.135431+0000","last_clean":"2024-09-13T14:41:26.135431+0000","last_became_active":"2024-09-13T14:38:03.533794+0000","last_became_peered":"2024-09-13T14:38:03.533794+0000","last_unstale":"2024-09-13T14:41:26.135431+0000","last_undegraded":"2024-09-13T14:41:26.135431+0000","last_fullsized":"2024-09-13T14:41:26.135431+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:23.272098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":65,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:18.095916+0000","last_change":"2024-09-13T14:38:03.527021+0000","last_active":"2024-09-13T14:41:18.095916+0000","last_peered":"2024-09-13T14:41:18.095916+0000","last_clean":"2024-09-13T14:41:18.095916+0000","last_became_active":"2024-09-13T14:38:03.526698+0000","last_became_peered":"2024-09-13T14:38:03.526698+0000","last_unstale":"2024-09-13T14:41:18.095916+0000","last_undegraded":"2024-09-13T14:41:18.095916+0000","last_fullsized":"2024-09-13T14:41:18.095916+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:01:35.734086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":270,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:41:10.103092+0000","last_change":"2024-09-13T14:27:12.834931+0000","last_active":"2024-09-13T14:41:10.103092+0000","last_peered":"2024-09-13T14:41:10.103092+0000","last_clean":"2024-09-13T14:41:10.103092+0000","last_became_active":"2024-09-13T14:27:12.834572+0000","last_became_peered":"2024-09-13T14:27:12.834572+0000","last_unstale":"2024-09-13T14:41:10.103092+0000","last_undegraded":"2024-09-13T14:41:10.103092+0000","last_fullsized":"2024-09-13T14:41:10.103092+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:54:21.155689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":85,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:18.109307+0000","last_change":"2024-09-13T14:37:49.176796+0000","last_active":"2024-09-13T14:41:18.109307+0000","last_peered":"2024-09-13T14:41:18.109307+0000","last_clean":"2024-09-13T14:41:18.109307+0000","last_became_active":"2024-09-13T14:37:49.176246+0000","last_became_peered":"2024-09-13T14:37:49.176246+0000","last_unstale":"2024-09-13T14:41:18.109307+0000","last_undegraded":"2024-09-13T14:41:18.109307+0000","last_fullsized":"2024-09-13T14:41:18.109307+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:29:24.936718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":281,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:41:10.101220+0000","last_change":"2024-09-13T14:38:03.526333+0000","last_active":"2024-09-13T14:41:10.101220+0000","last_peered":"2024-09-13T14:41:10.101220+0000","last_clean":"2024-09-13T14:41:10.101220+0000","last_became_active":"2024-09-13T14:38:03.526001+0000","last_became_peered":"2024-09-13T14:38:03.526001+0000","last_unstale":"2024-09-13T14:41:10.101220+0000","last_undegraded":"2024-09-13T14:41:10.101220+0000","last_fullsized":"2024-09-13T14:41:10.101220+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:25:25.181271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":85,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:18.109134+0000","last_change":"2024-09-13T14:37:49.149582+0000","last_active":"2024-09-13T14:41:18.109134+0000","last_peered":"2024-09-13T14:41:18.109134+0000","last_clean":"2024-09-13T14:41:18.109134+0000","last_became_active":"2024-09-13T14:37:49.148525+0000","last_became_peered":"2024-09-13T14:37:49.148525+0000","last_unstale":"2024-09-13T14:41:18.109134+0000","last_undegraded":"2024-09-13T14:41:18.109134+0000","last_fullsized":"2024-09-13T14:41:18.109134+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:48.655665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":467,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:41:26.134684+0000","last_change":"2024-09-13T14:38:03.535255+0000","last_active":"2024-09-13T14:41:26.134684+0000","last_peered":"2024-09-13T14:41:26.134684+0000","last_clean":"2024-09-13T14:41:26.134684+0000","last_became_active":"2024-09-13T14:38:03.533917+0000","last_became_peered":"2024-09-13T14:38:03.533917+0000","last_unstale":"2024-09-13T14:41:26.134684+0000","last_undegraded":"2024-09-13T14:41:26.134684+0000","last_fullsized":"2024-09-13T14:41:26.134684+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:33.559788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":431,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:41:26.136009+0000","last_change":"2024-09-13T14:37:48.146784+0000","last_active":"2024-09-13T14:41:26.136009+0000","last_peered":"2024-09-13T14:41:26.136009+0000","last_clean":"2024-09-13T14:41:26.136009+0000","last_became_active":"2024-09-13T14:37:48.146491+0000","last_became_peered":"2024-09-13T14:37:48.146491+0000","last_unstale":"2024-09-13T14:41:26.136009+0000","last_undegraded":"2024-09-13T14:41:26.136009+0000","last_fullsized":"2024-09-13T14:41:26.136009+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:30:54.772513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":262,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-13T14:41:10.101080+0000","last_change":"2024-09-13T14:37:48.127607+0000","last_active":"2024-09-13T14:41:10.101080+0000","last_peered":"2024-09-13T14:41:10.101080+0000","last_clean":"2024-09-13T14:41:10.101080+0000","last_became_active":"2024-09-13T14:37:48.127316+0000","last_became_peered":"2024-09-13T14:37:48.127316+0000","last_unstale":"2024-09-13T14:41:10.101080+0000","last_undegraded":"2024-09-13T14:41:10.101080+0000","last_fullsized":"2024-09-13T14:41:10.101080+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:53:37.678861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":71,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:18.100209+0000","last_change":"2024-09-13T14:38:03.563200+0000","last_active":"2024-09-13T14:41:18.100209+0000","last_peered":"2024-09-13T14:41:18.100209+0000","last_clean":"2024-09-13T14:41:18.100209+0000","last_became_active":"2024-09-13T14:38:03.534258+0000","last_became_peered":"2024-09-13T14:38:03.534258+0000","last_unstale":"2024-09-13T14:41:18.100209+0000","last_undegraded":"2024-09-13T14:41:18.100209+0000","last_fullsized":"2024-09-13T14:41:18.100209+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:49:07.814419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":5058789400,"num_objects":77192,"num_object_clones":0,"num_object_copies":231576,"num_objects_missing_on_primary":2736,"num_objects_missing":2736,"num_objects_degraded":28708,"num_objects_misplaced":4407,"num_objects_unfound":0,"num_objects_dirty":77192,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148434,"num_write_kb":7219969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61509,"num_bytes_recovered":3995271192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16466849792,"data_stored":16466837576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92657,"ondisk_log_size":92657,"up":48,"acting":43,"num_store_stats":6},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1538,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":17412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25132,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":7,"num_objects_missing":7,"num_objects_degraded":16,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16793600,"data_stored":16777292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":23,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234371,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3032932,"kb_used_data":2586716,"kb_used_omap":1,"kb_used_meta":446206,"kb_avail":90712220,"statfs":{"total":95995035648,"available":92889313280,"internally_reserved":0,"allocated":2648797184,"data_stored":2647319848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":456915403},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":239,"seq":1026497183789,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1856288,"kb_used_data":1673304,"kb_used_omap":8,"kb_used_meta":182903,"kb_avail":91888864,"statfs":{"total":95995035648,"available":94094196736,"internally_reserved":0,"allocated":1713463296,"data_stored":1711982758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8453,"internal_metadata":187293435},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757636,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2291788,"kb_used_data":1656900,"kb_used_omap":1,"kb_used_meta":634878,"kb_avail":91453364,"statfs":{"total":95995035648,"available":93648244736,"internally_reserved":0,"allocated":1696665600,"data_stored":1695200148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":650115526},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,1,0,1,3,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019279,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3301144,"kb_used_data":2706448,"kb_used_omap":1,"kb_used_meta":594686,"kb_avail":90444008,"statfs":{"total":95995035648,"available":92614664192,"internally_reserved":0,"allocated":2771402752,"data_stored":2769929823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":608958918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435463,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4255708,"kb_used_data":3700628,"kb_used_omap":5,"kb_used_meta":555066,"kb_avail":89489444,"statfs":{"total":95995035648,"available":91637190656,"internally_reserved":0,"allocated":3789443072,"data_stored":3787969979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":568387710},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":446676598991,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":425201762517,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926041,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4471444,"kb_used_data":3812236,"kb_used_omap":7,"kb_used_meta":659192,"kb_avail":89273708,"statfs":{"total":95995035648,"available":91416276992,"internally_reserved":0,"allocated":3903729664,"data_stored":3902256407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":675012904},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6860,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3895201792,"data_stored":3895197720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3779203072,"data_stored":3779199000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2764242944,"data_stored":2764242944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1686241280,"data_stored":1686241280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1701449728,"data_stored":1701445656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2640510976,"data_stored":2640510976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:41:29.372 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:29.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:29.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:29.474 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:29.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:29 smithi195 ceph-mon[104262]: pgmap v906: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.7 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 236 op/s; 28726/231660 objects degraded (12.400%); 4409/231660 objects misplaced (1.903%); 6.5 MiB/s, 104 objects/s recovering 2024-09-13T14:41:29.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:29 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 28726/231660 objects degraded (12.400%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:41:29.502 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:29.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:29.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:29.604 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:29.629 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:29.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:29.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:29.734 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:29.757 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:29.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:29.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:29.859 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:29.882 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:29.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:29.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:29.984 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:29.988 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:41:30.011 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:30.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:30.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:30.113 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:30.143 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:30.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:30.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:30.245 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:30.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:30 smithi195 ceph-mon[104262]: from='client.45731 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:41:30.249 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:30 smithi102 ceph-mon[110183]: from='client.45731 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:41:30.249 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:30 smithi102 ceph-mon[115269]: from='client.45731 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:41:30.271 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:30.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:30.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:30.358 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:30.374 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:30.399 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:30.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:30.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:30.500 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:30.526 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:30.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:30.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:30.628 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:30.652 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:30.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:30.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:30.754 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:30.777 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:30.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:30.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:30.879 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:30.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:30.904 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:30.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:31.006 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:31.028 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:31.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:31.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:31.130 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:31.154 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:31.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:31.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:31.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:31 smithi102 ceph-mon[110183]: pgmap v907: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.7 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 236 op/s; 28642/230100 objects degraded (12.448%); 4409/230100 objects misplaced (1.916%); 6.2 MiB/s, 99 objects/s recovering 2024-09-13T14:41:31.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:31 smithi102 ceph-mon[115269]: pgmap v907: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.7 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 236 op/s; 28642/230100 objects degraded (12.448%); 4409/230100 objects misplaced (1.916%); 6.2 MiB/s, 99 objects/s recovering 2024-09-13T14:41:31.231 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:41:31.231 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-09-13T14:41:31.231 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-09-13T14:41:31.231 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-13T14:41:31.231 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.231 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35379.0:113999 7.3 e241/240 7:ce075efd:::benchmark_data_smithi102_227395_object1184:head v 241'7281, mlcod=0'0)", 2024-09-13T14:41:31.231 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:49.296141+0000", 2024-09-13T14:41:31.231 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 221.93193585, 2024-09-13T14:41:31.231 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.028084437, 2024-09-13T14:41:31.231 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:41:31.232 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:31.232 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:41:31.232 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.232 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:41:31.232 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.296141+0000", 2024-09-13T14:41:31.232 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:31.232 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.232 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.232 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:41:31.232 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.296141+0000", 2024-09-13T14:41:31.232 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:31.232 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.233 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.233 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:41:31.233 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.296140+0000", 2024-09-13T14:41:31.233 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:41:31.233 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.233 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.233 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:41:31.233 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.296149+0000", 2024-09-13T14:41:31.233 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2169999999999994e-06 2024-09-13T14:41:31.233 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.233 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.233 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:41:31.233 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.296159+0000", 2024-09-13T14:41:31.233 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0032e-05 2024-09-13T14:41:31.234 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.234 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.234 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:41:31.234 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.296168+0000", 2024-09-13T14:41:31.234 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.268e-06 2024-09-13T14:41:31.234 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.234 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.234 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:31.234 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.296419+0000", 2024-09-13T14:41:31.234 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00025090000000000003 2024-09-13T14:41:31.234 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.234 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.234 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:41:31.235 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.296441+0000", 2024-09-13T14:41:31.235 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2118999999999999e-05 2024-09-13T14:41:31.235 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.235 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.235 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:41:31.235 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.305258+0000", 2024-09-13T14:41:31.235 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0088167409999999995 2024-09-13T14:41:31.235 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.235 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.235 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:41:31.235 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.324226+0000", 2024-09-13T14:41:31.235 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018968160000000001 2024-09-13T14:41:31.235 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:31.236 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:41:31.236 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:31.236 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.236 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.236 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35379.0:114001 7.d e241/240 7:b833ca5f:::benchmark_data_smithi102_227395_object1186:head v 241'7136, mlcod=0'0)", 2024-09-13T14:41:31.236 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:49.625200+0000", 2024-09-13T14:41:31.236 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 221.60287704199999, 2024-09-13T14:41:31.236 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.060053484999999997, 2024-09-13T14:41:31.236 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:41:31.236 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:31.236 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:41:31.236 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.236 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:41:31.236 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.625200+0000", 2024-09-13T14:41:31.237 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:31.237 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.237 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.237 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:41:31.237 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.625200+0000", 2024-09-13T14:41:31.237 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:31.237 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.237 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.237 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:41:31.237 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.625200+0000", 2024-09-13T14:41:31.237 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967296 2024-09-13T14:41:31.237 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.237 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.238 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:41:31.238 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.625211+0000", 2024-09-13T14:41:31.238 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0567000000000001e-05 2024-09-13T14:41:31.238 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.238 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.238 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:41:31.238 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.625232+0000", 2024-09-13T14:41:31.238 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1081999999999999e-05 2024-09-13T14:41:31.238 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.238 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.238 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:41:31.238 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.625245+0000", 2024-09-13T14:41:31.239 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3210999999999999e-05 2024-09-13T14:41:31.239 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.239 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.239 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:31.239 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.636891+0000", 2024-09-13T14:41:31.239 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011645900000000001 2024-09-13T14:41:31.239 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.239 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.239 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:41:31.239 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.636938+0000", 2024-09-13T14:41:31.239 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7448999999999999e-05 2024-09-13T14:41:31.239 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.239 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.240 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:41:31.240 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.685219+0000", 2024-09-13T14:41:31.240 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.048280494 2024-09-13T14:41:31.240 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.240 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.240 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:41:31.240 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.685254+0000", 2024-09-13T14:41:31.240 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4968999999999997e-05 2024-09-13T14:41:31.240 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:31.240 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:41:31.240 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:31.240 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.240 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.240 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35379.0:114005 7.d e241/240 7:b04d2f86:::benchmark_data_smithi102_227395_object1190:head v 241'7137, mlcod=0'0)", 2024-09-13T14:41:31.241 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:49.629094+0000", 2024-09-13T14:41:31.241 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 221.59898359900001, 2024-09-13T14:41:31.241 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.056369791000000002, 2024-09-13T14:41:31.241 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:41:31.241 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:31.241 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:41:31.241 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.241 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:41:31.241 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.629094+0000", 2024-09-13T14:41:31.241 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:31.241 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.241 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.241 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:41:31.242 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.629094+0000", 2024-09-13T14:41:31.242 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:31.242 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.242 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.242 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:41:31.242 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.629091+0000", 2024-09-13T14:41:31.242 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999971 2024-09-13T14:41:31.242 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.242 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.242 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:41:31.242 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.629109+0000", 2024-09-13T14:41:31.242 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8346999999999999e-05 2024-09-13T14:41:31.242 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.242 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.243 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:41:31.243 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.629136+0000", 2024-09-13T14:41:31.243 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7036999999999999e-05 2024-09-13T14:41:31.243 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.243 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.243 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:41:31.243 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.629159+0000", 2024-09-13T14:41:31.243 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2529999999999999e-05 2024-09-13T14:41:31.243 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.243 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.243 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:31.243 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.644193+0000", 2024-09-13T14:41:31.243 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.015034084 2024-09-13T14:41:31.243 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.243 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.244 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:41:31.244 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.644222+0000", 2024-09-13T14:41:31.244 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9450000000000001e-05 2024-09-13T14:41:31.244 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.244 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.244 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:41:31.244 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.685434+0000", 2024-09-13T14:41:31.244 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.041211283000000001 2024-09-13T14:41:31.244 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.244 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.244 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:41:31.244 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.685463+0000", 2024-09-13T14:41:31.244 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9750000000000001e-05 2024-09-13T14:41:31.245 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:31.245 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:41:31.245 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:31.245 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.245 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.245 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35379.0:113978 7.5 e241/240 7:a1bc8a89:::benchmark_data_smithi102_227395_object1163:head v 241'7151, mlcod=0'0)", 2024-09-13T14:41:31.245 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:49.629364+0000", 2024-09-13T14:41:31.245 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 221.59871318, 2024-09-13T14:41:31.245 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.056066579999999998, 2024-09-13T14:41:31.245 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:41:31.245 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:31.245 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:41:31.245 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.245 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:41:31.246 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.629364+0000", 2024-09-13T14:41:31.246 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:31.246 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.246 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.246 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:41:31.246 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.629364+0000", 2024-09-13T14:41:31.246 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:31.246 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.246 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.246 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:41:31.246 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.629363+0000", 2024-09-13T14:41:31.246 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:41:31.246 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.246 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.247 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:41:31.247 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.629375+0000", 2024-09-13T14:41:31.247 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1226000000000001e-05 2024-09-13T14:41:31.247 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.247 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.247 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:41:31.247 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.629398+0000", 2024-09-13T14:41:31.247 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3434999999999998e-05 2024-09-13T14:41:31.247 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.247 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.247 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:41:31.247 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.629409+0000", 2024-09-13T14:41:31.247 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0713000000000001e-05 2024-09-13T14:41:31.247 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.248 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.248 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:31.248 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.643993+0000", 2024-09-13T14:41:31.248 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014584674000000001 2024-09-13T14:41:31.248 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.248 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.248 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:41:31.248 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.644054+0000", 2024-09-13T14:41:31.248 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0618999999999997e-05 2024-09-13T14:41:31.248 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.248 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.248 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:41:31.248 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.685257+0000", 2024-09-13T14:41:31.248 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.041203374000000001 2024-09-13T14:41:31.249 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.249 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.249 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:41:31.249 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.685431+0000", 2024-09-13T14:41:31.249 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000173182 2024-09-13T14:41:31.249 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:31.249 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:41:31.249 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:31.249 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.250 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.250 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35379.0:113996 7.5 e241/240 7:a0c0436d:::benchmark_data_smithi102_227395_object1181:head v 241'7152, mlcod=0'0)", 2024-09-13T14:41:31.250 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:49.635640+0000", 2024-09-13T14:41:31.250 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 221.59243717199999, 2024-09-13T14:41:31.250 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.049945839999999998, 2024-09-13T14:41:31.250 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:41:31.250 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:31.250 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:41:31.250 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.250 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:41:31.250 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.635640+0000", 2024-09-13T14:41:31.250 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:31.251 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.251 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.251 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:41:31.251 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.635640+0000", 2024-09-13T14:41:31.251 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:31.251 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.251 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.251 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:41:31.251 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.635637+0000", 2024-09-13T14:41:31.251 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999971 2024-09-13T14:41:31.251 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.251 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.251 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:41:31.251 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.635655+0000", 2024-09-13T14:41:31.252 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7524e-05 2024-09-13T14:41:31.252 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.252 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.252 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:41:31.252 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.635686+0000", 2024-09-13T14:41:31.252 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0959e-05 2024-09-13T14:41:31.252 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.252 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.252 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:41:31.252 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.635698+0000", 2024-09-13T14:41:31.252 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2639e-05 2024-09-13T14:41:31.252 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.252 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.252 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:31.253 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.651499+0000", 2024-09-13T14:41:31.253 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.015801030000000001 2024-09-13T14:41:31.253 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.253 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.253 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:41:31.253 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.651532+0000", 2024-09-13T14:41:31.253 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2076000000000003e-05 2024-09-13T14:41:31.253 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.253 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.253 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:41:31.253 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.685466+0000", 2024-09-13T14:41:31.253 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.033934480000000003 2024-09-13T14:41:31.253 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.253 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.254 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:41:31.254 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.685586+0000", 2024-09-13T14:41:31.254 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000119936 2024-09-13T14:41:31.254 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:31.254 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:41:31.254 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:31.254 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.254 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.254 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35379.0:113979 7.4 e241/240 7:2d0634a9:::benchmark_data_smithi102_227395_object1164:head v 241'7140, mlcod=0'0)", 2024-09-13T14:41:31.254 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:49.664954+0000", 2024-09-13T14:41:31.254 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 221.56312298700001, 2024-09-13T14:41:31.254 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.026864203999999999, 2024-09-13T14:41:31.254 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:41:31.254 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:31.255 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:41:31.255 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.255 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:41:31.255 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.664954+0000", 2024-09-13T14:41:31.255 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:31.255 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.255 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.255 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:41:31.255 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.664954+0000", 2024-09-13T14:41:31.255 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:31.255 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.255 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.255 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:41:31.255 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.664954+0000", 2024-09-13T14:41:31.256 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:41:31.256 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.256 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.256 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:41:31.256 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.664962+0000", 2024-09-13T14:41:31.256 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9990000000000008e-06 2024-09-13T14:41:31.256 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.256 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.256 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:41:31.256 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.664975+0000", 2024-09-13T14:41:31.256 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3298999999999999e-05 2024-09-13T14:41:31.256 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.256 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.257 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:41:31.257 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.664983+0000", 2024-09-13T14:41:31.257 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8699999999999992e-06 2024-09-13T14:41:31.257 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.257 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.257 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:31.257 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.665018+0000", 2024-09-13T14:41:31.257 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5663000000000001e-05 2024-09-13T14:41:31.257 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.257 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.257 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:41:31.257 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.665040+0000", 2024-09-13T14:41:31.257 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1225000000000001e-05 2024-09-13T14:41:31.257 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.258 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.258 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:41:31.258 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.691728+0000", 2024-09-13T14:41:31.258 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.026688046999999999 2024-09-13T14:41:31.258 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.258 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.258 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:41:31.258 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.691818+0000", 2024-09-13T14:41:31.258 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.0839000000000002e-05 2024-09-13T14:41:31.258 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:31.258 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:41:31.258 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:31.258 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.259 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.259 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35379.0:113989 7.2 e241/240 7:4d58294e:::benchmark_data_smithi102_227395_object1174:head v 241'6992, mlcod=0'0)", 2024-09-13T14:41:31.259 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:49.700538+0000", 2024-09-13T14:41:31.259 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 221.52753919899999, 2024-09-13T14:41:31.259 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.041952102999999998, 2024-09-13T14:41:31.259 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:41:31.259 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:31.259 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:41:31.259 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.259 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:41:31.259 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.700538+0000", 2024-09-13T14:41:31.259 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:31.259 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.259 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.260 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:41:31.260 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.700538+0000", 2024-09-13T14:41:31.260 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:31.260 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.260 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.260 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:31.261 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:41:31.261 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.700535+0000", 2024-09-13T14:41:31.262 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999971 2024-09-13T14:41:31.262 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.262 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.262 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:41:31.262 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.700543+0000", 2024-09-13T14:41:31.262 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.374e-06 2024-09-13T14:41:31.262 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.262 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.262 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:41:31.262 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.700553+0000", 2024-09-13T14:41:31.262 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0402e-05 2024-09-13T14:41:31.262 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.262 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.262 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:41:31.263 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.700560+0000", 2024-09-13T14:41:31.263 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.793e-06 2024-09-13T14:41:31.263 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.263 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.263 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:31.263 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.700623+0000", 2024-09-13T14:41:31.263 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3621999999999995e-05 2024-09-13T14:41:31.263 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.263 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.263 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:41:31.263 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.700646+0000", 2024-09-13T14:41:31.263 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2289e-05 2024-09-13T14:41:31.263 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.263 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.264 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:41:31.264 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.742453+0000", 2024-09-13T14:41:31.264 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.041807122000000002 2024-09-13T14:41:31.264 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.264 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.264 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:41:31.264 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.742490+0000", 2024-09-13T14:41:31.264 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7277000000000002e-05 2024-09-13T14:41:31.264 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:31.264 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:41:31.264 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:31.264 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.264 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.265 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35379.0:113995 7.2 e241/240 7:4425f875:::benchmark_data_smithi102_227395_object1180:head v 241'6993, mlcod=0'0)", 2024-09-13T14:41:31.265 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:49.712220+0000", 2024-09-13T14:41:31.265 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 221.515857735, 2024-09-13T14:41:31.265 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.030292321000000001, 2024-09-13T14:41:31.265 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:41:31.265 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:31.265 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:41:31.265 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.265 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:41:31.265 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.712220+0000", 2024-09-13T14:41:31.265 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:31.265 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.265 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.266 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:41:31.266 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.712220+0000", 2024-09-13T14:41:31.266 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:31.266 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.266 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.266 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:41:31.266 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.712219+0000", 2024-09-13T14:41:31.266 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:41:31.266 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.266 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.266 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:41:31.266 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.712227+0000", 2024-09-13T14:41:31.266 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2989999999999997e-06 2024-09-13T14:41:31.267 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.267 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.267 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:41:31.267 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.712237+0000", 2024-09-13T14:41:31.267 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.9229999999999997e-06 2024-09-13T14:41:31.267 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.267 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.267 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:41:31.267 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.712242+0000", 2024-09-13T14:41:31.267 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0730000000000004e-06 2024-09-13T14:41:31.267 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.267 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.267 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:31.267 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.715716+0000", 2024-09-13T14:41:31.267 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0034740069999999999 2024-09-13T14:41:31.268 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.268 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.268 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:41:31.268 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.715772+0000", 2024-09-13T14:41:31.268 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5578999999999998e-05 2024-09-13T14:41:31.268 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.268 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.268 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:41:31.268 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.742494+0000", 2024-09-13T14:41:31.268 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.026721967999999999 2024-09-13T14:41:31.268 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.268 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.268 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:41:31.269 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.742512+0000", 2024-09-13T14:41:31.269 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8102999999999999e-05 2024-09-13T14:41:31.269 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:31.269 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:41:31.269 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:31.269 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.269 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.269 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35379.0:114019 7.2 e241/240 7:4353a217:::benchmark_data_smithi102_227395_object1204:head v 241'6994, mlcod=0'0)", 2024-09-13T14:41:31.269 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:49.712271+0000", 2024-09-13T14:41:31.269 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 221.515806416, 2024-09-13T14:41:31.269 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.030264374, 2024-09-13T14:41:31.269 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:41:31.269 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:31.269 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:41:31.270 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.270 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:41:31.270 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.712271+0000", 2024-09-13T14:41:31.270 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:31.270 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.270 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.270 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:41:31.270 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.712271+0000", 2024-09-13T14:41:31.270 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:31.270 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.270 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.270 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:41:31.270 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.712271+0000", 2024-09-13T14:41:31.271 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967296 2024-09-13T14:41:31.271 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.271 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.271 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:41:31.271 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.712278+0000", 2024-09-13T14:41:31.271 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.934e-06 2024-09-13T14:41:31.271 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.271 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.271 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:41:31.271 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.712283+0000", 2024-09-13T14:41:31.271 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.967e-06 2024-09-13T14:41:31.271 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.271 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.271 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:41:31.271 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.712286+0000", 2024-09-13T14:41:31.272 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.3830000000000001e-06 2024-09-13T14:41:31.272 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.272 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.272 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:31.272 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.729335+0000", 2024-09-13T14:41:31.272 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.01704872 2024-09-13T14:41:31.272 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.272 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.272 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:41:31.272 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.729764+0000", 2024-09-13T14:41:31.272 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000429694 2024-09-13T14:41:31.272 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.272 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.272 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:41:31.273 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.742514+0000", 2024-09-13T14:41:31.273 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012750005 2024-09-13T14:41:31.273 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.273 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.273 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:41:31.273 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.742535+0000", 2024-09-13T14:41:31.273 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0835e-05 2024-09-13T14:41:31.273 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:31.273 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:41:31.273 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:31.273 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.273 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.273 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35379.0:114020 7.2 e241/240 7:4af54330:::benchmark_data_smithi102_227395_object1205:head v 241'6995, mlcod=0'0)", 2024-09-13T14:41:31.273 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:49.715242+0000", 2024-09-13T14:41:31.274 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 221.51283504700001, 2024-09-13T14:41:31.274 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.027314150999999998, 2024-09-13T14:41:31.274 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:41:31.274 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:31.274 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:41:31.274 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.274 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:41:31.274 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.715242+0000", 2024-09-13T14:41:31.274 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:31.274 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.274 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.274 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:41:31.274 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.715242+0000", 2024-09-13T14:41:31.274 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:31.275 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.275 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.275 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:41:31.275 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.715241+0000", 2024-09-13T14:41:31.275 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:41:31.275 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.275 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.275 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:41:31.275 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.715250+0000", 2024-09-13T14:41:31.275 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8890000000000008e-06 2024-09-13T14:41:31.275 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.275 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.275 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:41:31.275 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.715271+0000", 2024-09-13T14:41:31.275 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1503999999999999e-05 2024-09-13T14:41:31.276 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.276 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.276 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:41:31.276 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.715279+0000", 2024-09-13T14:41:31.276 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3170000000000004e-06 2024-09-13T14:41:31.276 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.276 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.276 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:31.276 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.736274+0000", 2024-09-13T14:41:31.276 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.020995538000000001 2024-09-13T14:41:31.276 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.276 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.277 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:41:31.277 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.736320+0000", 2024-09-13T14:41:31.277 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6121000000000001e-05 2024-09-13T14:41:31.277 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.277 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.277 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:41:31.277 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.742537+0000", 2024-09-13T14:41:31.277 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0062166599999999997 2024-09-13T14:41:31.277 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.277 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.277 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:41:31.277 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.742556+0000", 2024-09-13T14:41:31.277 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9315000000000001e-05 2024-09-13T14:41:31.277 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:31.278 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:41:31.278 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:31.278 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.278 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.278 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35379.0:114032 7.d e241/240 7:bcc99e0b:::benchmark_data_smithi102_227395_object1217:head v 241'7140, mlcod=210'2646)", 2024-09-13T14:41:31.278 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:49.730937+0000", 2024-09-13T14:41:31.278 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 221.49714044999999, 2024-09-13T14:41:31.278 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.028271039000000001, 2024-09-13T14:41:31.278 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:41:31.278 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:31.278 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:41:31.278 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.278 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:41:31.279 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.730937+0000", 2024-09-13T14:41:31.279 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:31.279 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.279 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.279 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:41:31.279 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.730937+0000", 2024-09-13T14:41:31.279 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:31.279 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.279 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.279 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:41:31.279 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.730936+0000", 2024-09-13T14:41:31.279 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:41:31.279 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.280 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.280 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:41:31.280 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.730946+0000", 2024-09-13T14:41:31.280 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0370000000000001e-05 2024-09-13T14:41:31.280 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.280 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.280 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:41:31.280 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.730957+0000", 2024-09-13T14:41:31.280 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0771000000000001e-05 2024-09-13T14:41:31.280 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.280 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.280 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:41:31.280 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.730964+0000", 2024-09-13T14:41:31.280 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7659999999999998e-06 2024-09-13T14:41:31.281 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.281 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.281 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:31.281 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.731094+0000", 2024-09-13T14:41:31.281 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012904999999999999 2024-09-13T14:41:31.281 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.281 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.281 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:41:31.281 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.731116+0000", 2024-09-13T14:41:31.281 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2249e-05 2024-09-13T14:41:31.281 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.281 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.281 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:41:31.281 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.759114+0000", 2024-09-13T14:41:31.282 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.027998282999999999 2024-09-13T14:41:31.282 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.282 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.282 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:41:31.282 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.759208+0000", 2024-09-13T14:41:31.282 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.3758999999999995e-05 2024-09-13T14:41:31.282 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:31.282 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:41:31.282 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:31.282 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.282 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.282 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35379.0:114041 7.2 e241/240 7:4ea322ce:::benchmark_data_smithi102_227395_object1226:head v 241'6996, mlcod=210'2596)", 2024-09-13T14:41:31.282 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:49.748559+0000", 2024-09-13T14:41:31.283 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 221.479518243, 2024-09-13T14:41:31.283 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.038689679999999997, 2024-09-13T14:41:31.283 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:41:31.283 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:31.283 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:41:31.283 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.283 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:41:31.283 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.748559+0000", 2024-09-13T14:41:31.283 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:31.283 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.283 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.283 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:41:31.283 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.748559+0000", 2024-09-13T14:41:31.283 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:31.284 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.284 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.284 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:41:31.284 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.748558+0000", 2024-09-13T14:41:31.284 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:41:31.284 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.284 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.284 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:41:31.284 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.748566+0000", 2024-09-13T14:41:31.284 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0430000000000008e-06 2024-09-13T14:41:31.284 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.284 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.284 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:41:31.284 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.748579+0000", 2024-09-13T14:41:31.285 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2661e-05 2024-09-13T14:41:31.285 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.285 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.285 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:41:31.285 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.748586+0000", 2024-09-13T14:41:31.285 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6869999999999994e-06 2024-09-13T14:41:31.285 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.285 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.285 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:31.285 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.753724+0000", 2024-09-13T14:41:31.285 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0051372129999999998 2024-09-13T14:41:31.285 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.285 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.285 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:41:31.286 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.753748+0000", 2024-09-13T14:41:31.286 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4862999999999999e-05 2024-09-13T14:41:31.286 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.286 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.286 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:41:31.286 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.787223+0000", 2024-09-13T14:41:31.286 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.033474059 2024-09-13T14:41:31.286 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.286 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.286 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:41:31.286 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.787249+0000", 2024-09-13T14:41:31.286 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.6149999999999999e-05 2024-09-13T14:41:31.286 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:31.287 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:41:31.287 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:31.287 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.287 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.287 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35379.0:114045 7.2 e241/240 7:44c27eea:::benchmark_data_smithi102_227395_object1230:head v 241'6997, mlcod=210'2596)", 2024-09-13T14:41:31.287 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:49.753636+0000", 2024-09-13T14:41:31.287 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 221.474441329, 2024-09-13T14:41:31.287 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.033627046000000001, 2024-09-13T14:41:31.287 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:41:31.287 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:31.287 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:41:31.287 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.287 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:41:31.287 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.753636+0000", 2024-09-13T14:41:31.288 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:31.288 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.288 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.288 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:41:31.288 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.753636+0000", 2024-09-13T14:41:31.288 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:31.288 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.288 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.288 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:41:31.288 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.753635+0000", 2024-09-13T14:41:31.288 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:41:31.288 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.288 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.288 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:41:31.289 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.753644+0000", 2024-09-13T14:41:31.289 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1120000000000002e-06 2024-09-13T14:41:31.289 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.289 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.289 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:41:31.289 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.753656+0000", 2024-09-13T14:41:31.289 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.168e-05 2024-09-13T14:41:31.289 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.289 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.289 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:41:31.289 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.753662+0000", 2024-09-13T14:41:31.289 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4289999999999998e-06 2024-09-13T14:41:31.289 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.289 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.290 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:31.290 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.760505+0000", 2024-09-13T14:41:31.290 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0068426099999999998 2024-09-13T14:41:31.290 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.290 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.290 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:41:31.290 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.760526+0000", 2024-09-13T14:41:31.290 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1497000000000001e-05 2024-09-13T14:41:31.290 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.290 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.290 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:41:31.290 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.787251+0000", 2024-09-13T14:41:31.290 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.026724372 2024-09-13T14:41:31.291 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.291 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.291 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:41:31.291 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.787263+0000", 2024-09-13T14:41:31.291 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2157000000000001e-05 2024-09-13T14:41:31.291 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:31.291 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:41:31.291 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:31.291 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.291 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.291 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35379.0:114067 7.1 e241/240 7:896def22:::benchmark_data_smithi102_227395_object1252:head v 241'7223, mlcod=210'2715)", 2024-09-13T14:41:31.291 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:49.816693+0000", 2024-09-13T14:41:31.291 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 221.41138463600001, 2024-09-13T14:41:31.292 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.034224615, 2024-09-13T14:41:31.292 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:41:31.292 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:31.292 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:41:31.292 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.292 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:41:31.292 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.816693+0000", 2024-09-13T14:41:31.292 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:31.292 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.292 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.292 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:41:31.292 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.816693+0000", 2024-09-13T14:41:31.292 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:31.292 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.293 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.293 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:41:31.293 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.816692+0000", 2024-09-13T14:41:31.293 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:41:31.293 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.293 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.293 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:41:31.293 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.816701+0000", 2024-09-13T14:41:31.293 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.0550000000000005e-06 2024-09-13T14:41:31.293 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.293 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.293 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:41:31.293 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.816717+0000", 2024-09-13T14:41:31.293 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6021000000000001e-05 2024-09-13T14:41:31.294 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.294 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.294 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:41:31.294 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.833127+0000", 2024-09-13T14:41:31.294 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016410062999999999 2024-09-13T14:41:31.294 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.294 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.294 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:31.294 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.833383+0000", 2024-09-13T14:41:31.294 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00025658899999999999 2024-09-13T14:41:31.294 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.294 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.294 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:41:31.294 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.833411+0000", 2024-09-13T14:41:31.295 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7287000000000001e-05 2024-09-13T14:41:31.295 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.295 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.295 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:41:31.295 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.850872+0000", 2024-09-13T14:41:31.295 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017461438999999999 2024-09-13T14:41:31.295 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.295 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.295 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:41:31.295 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.850917+0000", 2024-09-13T14:41:31.295 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5158999999999999e-05 2024-09-13T14:41:31.295 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:31.295 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:41:31.295 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:31.296 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.296 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.296 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35379.0:114084 7.d e241/240 7:b5346ec3:::benchmark_data_smithi102_227395_object1269:head v 241'7145, mlcod=210'2646)", 2024-09-13T14:41:31.296 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:49.819351+0000", 2024-09-13T14:41:31.296 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 221.408726497, 2024-09-13T14:41:31.296 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.026640918999999999, 2024-09-13T14:41:31.296 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:41:31.296 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:31.296 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:41:31.296 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.296 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:41:31.296 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.819351+0000", 2024-09-13T14:41:31.296 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:31.297 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.297 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.297 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:41:31.297 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.819351+0000", 2024-09-13T14:41:31.297 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:31.297 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.297 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.297 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:41:31.297 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.819350+0000", 2024-09-13T14:41:31.297 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:41:31.297 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.297 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.297 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:41:31.297 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.819359+0000", 2024-09-13T14:41:31.298 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4029999999999996e-06 2024-09-13T14:41:31.298 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.298 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.298 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:41:31.298 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.834151+0000", 2024-09-13T14:41:31.298 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014791561999999999 2024-09-13T14:41:31.298 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.298 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.298 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:41:31.298 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.834163+0000", 2024-09-13T14:41:31.298 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2537999999999999e-05 2024-09-13T14:41:31.298 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.298 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.299 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:31.299 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.835934+0000", 2024-09-13T14:41:31.299 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0017711420000000001 2024-09-13T14:41:31.299 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.299 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.299 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:41:31.299 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.835985+0000", 2024-09-13T14:41:31.299 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0968999999999999e-05 2024-09-13T14:41:31.299 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.299 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.299 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:41:31.299 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.844563+0000", 2024-09-13T14:41:31.299 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0085775039999999997 2024-09-13T14:41:31.299 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.299 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.300 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:41:31.300 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.845992+0000", 2024-09-13T14:41:31.300 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0014289859999999999 2024-09-13T14:41:31.300 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:31.300 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:41:31.300 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:31.300 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.300 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.301 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35379.0:114083 7.4 e241/240 7:2f9c8d6d:::benchmark_data_smithi102_227395_object1268:head v 241'7153, mlcod=210'2640)", 2024-09-13T14:41:31.301 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:49.833244+0000", 2024-09-13T14:41:31.301 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 221.394833651, 2024-09-13T14:41:31.301 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.036063532000000002, 2024-09-13T14:41:31.301 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:41:31.301 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:31.301 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:41:31.301 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.301 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:41:31.301 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.833244+0000", 2024-09-13T14:41:31.301 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:31.301 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.301 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.301 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:41:31.302 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.833244+0000", 2024-09-13T14:41:31.302 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:31.302 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.302 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.302 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:41:31.302 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.833243+0000", 2024-09-13T14:41:31.302 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967296 2024-09-13T14:41:31.302 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.302 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.302 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:41:31.302 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.833253+0000", 2024-09-13T14:41:31.302 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.7869999999999996e-06 2024-09-13T14:41:31.302 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.303 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.303 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:41:31.303 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.833258+0000", 2024-09-13T14:41:31.303 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.215e-06 2024-09-13T14:41:31.303 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.303 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.303 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:41:31.303 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.833262+0000", 2024-09-13T14:41:31.303 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9990000000000002e-06 2024-09-13T14:41:31.303 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.303 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.303 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:31.303 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.849288+0000", 2024-09-13T14:41:31.303 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016025239 2024-09-13T14:41:31.303 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.304 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.304 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:41:31.304 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.849315+0000", 2024-09-13T14:41:31.304 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7251e-05 2024-09-13T14:41:31.304 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.304 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.304 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:41:31.304 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.869224+0000", 2024-09-13T14:41:31.304 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.019908766000000001 2024-09-13T14:41:31.304 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.304 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.304 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:41:31.304 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.869307+0000", 2024-09-13T14:41:31.305 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3387000000000006e-05 2024-09-13T14:41:31.305 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:31.305 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:41:31.305 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:31.305 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.305 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.305 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35379.0:114101 7.1 e241/240 7:86bf0281:::benchmark_data_smithi102_227395_object1286:head v 241'7227, mlcod=210'2715)", 2024-09-13T14:41:31.305 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:49.867656+0000", 2024-09-13T14:41:31.305 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 221.360421503, 2024-09-13T14:41:31.305 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.027272266, 2024-09-13T14:41:31.305 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:41:31.305 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:31.305 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:41:31.306 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.306 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:41:31.306 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.867656+0000", 2024-09-13T14:41:31.306 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:31.306 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.306 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.306 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:41:31.306 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.867656+0000", 2024-09-13T14:41:31.306 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:31.306 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.306 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.306 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:41:31.306 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.867655+0000", 2024-09-13T14:41:31.306 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:41:31.307 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.307 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.307 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:41:31.307 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.867664+0000", 2024-09-13T14:41:31.307 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.8870000000000004e-06 2024-09-13T14:41:31.307 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.307 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.307 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:41:31.307 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.867679+0000", 2024-09-13T14:41:31.307 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4822e-05 2024-09-13T14:41:31.307 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.307 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.307 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:41:31.308 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.867686+0000", 2024-09-13T14:41:31.308 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2099999999999996e-06 2024-09-13T14:41:31.308 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.308 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.308 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:31.308 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.867761+0000", 2024-09-13T14:41:31.308 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4995999999999998e-05 2024-09-13T14:41:31.308 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.308 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.308 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:41:31.308 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.867791+0000", 2024-09-13T14:41:31.308 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9499e-05 2024-09-13T14:41:31.308 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.309 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.309 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:41:31.309 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.894808+0000", 2024-09-13T14:41:31.309 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.027017023000000001 2024-09-13T14:41:31.309 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.309 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.309 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:41:31.309 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.894928+0000", 2024-09-13T14:41:31.309 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000120004 2024-09-13T14:41:31.309 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:31.309 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:41:31.309 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:31.309 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.310 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.310 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35379.0:148608 7.2 e272/263 7:44ad0776:::benchmark_data_smithi102_227395_object35793:head v 272'9111, mlcod=272'9109)", 2024-09-13T14:41:31.310 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:41:26.480081+0000", 2024-09-13T14:41:31.310 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 4.7479958480000004, 2024-09-13T14:41:31.310 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.070670319999999995, 2024-09-13T14:41:31.310 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:41:31.310 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:31.310 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:41:31.310 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.310 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:41:31.310 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.480081+0000", 2024-09-13T14:41:31.310 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:31.310 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.311 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.311 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:41:31.311 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.480081+0000", 2024-09-13T14:41:31.311 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:31.311 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.311 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.311 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:41:31.311 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.480080+0000", 2024-09-13T14:41:31.311 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:41:31.311 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.311 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.311 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:41:31.311 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.480090+0000", 2024-09-13T14:41:31.312 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.7799999999999995e-06 2024-09-13T14:41:31.312 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.312 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.312 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:41:31.312 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.480106+0000", 2024-09-13T14:41:31.312 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5481999999999998e-05 2024-09-13T14:41:31.312 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.312 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.312 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:41:31.312 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.480113+0000", 2024-09-13T14:41:31.312 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9820000000000008e-06 2024-09-13T14:41:31.312 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.312 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.312 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:31.313 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.480156+0000", 2024-09-13T14:41:31.313 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.295e-05 2024-09-13T14:41:31.313 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.313 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.313 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:41:31.313 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.480177+0000", 2024-09-13T14:41:31.313 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0839000000000002e-05 2024-09-13T14:41:31.313 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.313 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.313 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:41:31.313 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.550706+0000", 2024-09-13T14:41:31.313 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.070529013000000002 2024-09-13T14:41:31.313 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.314 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.314 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:41:31.314 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.550752+0000", 2024-09-13T14:41:31.314 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5435999999999997e-05 2024-09-13T14:41:31.314 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:31.314 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:41:31.314 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:31.314 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.314 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.314 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDPGPush(7.b 272/240 [PushOp(7:d58d7887:::benchmark_data_smithi102_227395_object72243:head, version: 213'4482, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:d58d7887:::benchmark_data_smithi102_227395_object72243:head@213'4482, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T14:41:31.314 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:41:26.483641+0000", 2024-09-13T14:41:31.314 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 4.7444360909999999, 2024-09-13T14:41:31.314 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.066958958999999998, 2024-09-13T14:41:31.315 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:41:31.315 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T14:41:31.315 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:41:31.315 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.315 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:41:31.315 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.483641+0000", 2024-09-13T14:41:31.315 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:31.315 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.315 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.315 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:41:31.315 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.483641+0000", 2024-09-13T14:41:31.315 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:31.315 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.315 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.316 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:41:31.316 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.483640+0000", 2024-09-13T14:41:31.316 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:41:31.316 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.316 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.316 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:41:31.316 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.483684+0000", 2024-09-13T14:41:31.316 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3866999999999999e-05 2024-09-13T14:41:31.316 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.316 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.316 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:41:31.316 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.548048+0000", 2024-09-13T14:41:31.316 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.064363715000000002 2024-09-13T14:41:31.316 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.317 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.317 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:41:31.317 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.548073+0000", 2024-09-13T14:41:31.317 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5522999999999999e-05 2024-09-13T14:41:31.317 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.317 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.317 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:31.317 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.548173+0000", 2024-09-13T14:41:31.317 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.9853999999999996e-05 2024-09-13T14:41:31.317 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.317 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.317 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:41:31.317 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.548187+0000", 2024-09-13T14:41:31.317 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4132e-05 2024-09-13T14:41:31.318 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.318 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.318 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:41:31.318 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.550600+0000", 2024-09-13T14:41:31.318 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0024130250000000001 2024-09-13T14:41:31.318 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:31.318 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:41:31.318 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:31.318 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.318 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.318 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35379.0:148609 7.c e272/251 7:313103c6:::benchmark_data_smithi102_227395_object35794:head v 272'9246, mlcod=272'9244)", 2024-09-13T14:41:31.318 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:41:26.488089+0000", 2024-09-13T14:41:31.318 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 4.7399886159999998, 2024-09-13T14:41:31.318 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.066262520000000005, 2024-09-13T14:41:31.319 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:41:31.319 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:41:31.319 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:41:31.319 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.319 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:41:31.319 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.488089+0000", 2024-09-13T14:41:31.319 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:31.319 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.319 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.319 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:41:31.319 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.488089+0000", 2024-09-13T14:41:31.319 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:41:31.319 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.320 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.320 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:41:31.320 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.488088+0000", 2024-09-13T14:41:31.320 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:41:31.320 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.320 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.320 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:41:31.320 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.488096+0000", 2024-09-13T14:41:31.320 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3380000000000005e-06 2024-09-13T14:41:31.320 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.320 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.320 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:41:31.320 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.548070+0000", 2024-09-13T14:41:31.320 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.059973789999999999 2024-09-13T14:41:31.321 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.321 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.321 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:41:31.321 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.548100+0000", 2024-09-13T14:41:31.321 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0759000000000001e-05 2024-09-13T14:41:31.321 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.321 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.321 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:41:31.321 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.548263+0000", 2024-09-13T14:41:31.321 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000162399 2024-09-13T14:41:31.321 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.321 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.321 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:41:31.321 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.548286+0000", 2024-09-13T14:41:31.322 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3048e-05 2024-09-13T14:41:31.322 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.322 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.322 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:41:31.322 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.554312+0000", 2024-09-13T14:41:31.322 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0060261400000000001 2024-09-13T14:41:31.322 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:41:31.322 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:41:31.322 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:41:31.322 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.554351+0000", 2024-09-13T14:41:31.322 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9165000000000002e-05 2024-09-13T14:41:31.322 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:31.322 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:41:31.322 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:31.323 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:41:31.323 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:41:31.323 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:41:31.350 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:31.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:31.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:31.452 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:31.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:31.475 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:31.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:31.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:31 smithi195 ceph-mon[104262]: pgmap v907: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.7 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 236 op/s; 28642/230100 objects degraded (12.448%); 4409/230100 objects misplaced (1.916%); 6.2 MiB/s, 99 objects/s recovering 2024-09-13T14:41:31.576 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:31.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:31.605 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:31.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:31.706 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:31.737 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:31.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:31.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:31.839 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:31.862 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:31.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:31.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:31.964 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:31.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:31.990 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:31.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:32.091 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:32.116 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:32.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:32.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:32.218 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:32.247 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:32.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:32.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:32.336 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T14:41:32.349 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:32.366 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:41:32.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:32.408 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:32.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:32.510 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:32.536 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:32.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:32.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:32.638 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:32.662 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:32.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:32.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:32.764 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:32.792 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:32.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:32.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:32.817 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:41:32.817 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:41:32.841 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:41:32.842 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:41:32.860 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:32.869 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:41:32.869 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:41:32.886 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:41:32.886 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:41:32.893 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:32.905 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:41:32.905 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:41:32.918 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:32.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:32.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:32.925 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:41:32.926 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:41:32.947 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:41:32.948 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:41:32.973 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:41:32.973 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:41:33.020 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:33.046 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:33.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:33.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:33.147 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:33.172 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:33.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:33.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:33.274 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:33.300 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:33.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:33.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:33.402 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:33.406 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:33 smithi102 ceph-mon[110183]: pgmap v908: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.7 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 236 op/s; 28642/230100 objects degraded (12.448%); 4409/230100 objects misplaced (1.916%); 6.2 MiB/s, 99 objects/s recovering 2024-09-13T14:41:33.406 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:33 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1153830131' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:41:33.406 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:33 smithi102 ceph-mon[115269]: pgmap v908: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.7 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 236 op/s; 28642/230100 objects degraded (12.448%); 4409/230100 objects misplaced (1.916%); 6.2 MiB/s, 99 objects/s recovering 2024-09-13T14:41:33.407 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:33 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1153830131' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:41:33.424 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:33.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:33.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:33.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:33 smithi195 ceph-mon[104262]: pgmap v908: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.7 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 236 op/s; 28642/230100 objects degraded (12.448%); 4409/230100 objects misplaced (1.916%); 6.2 MiB/s, 99 objects/s recovering 2024-09-13T14:41:33.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:33 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1153830131' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:41:33.526 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:33.555 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:33.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:33.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:33.657 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:33.680 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:33.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:33.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:33.782 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:33.807 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:33.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:33.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:33.909 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:33.931 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:33.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:33.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:34.033 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:34.055 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:34.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:34.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:34.158 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:34.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:34.180 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:34.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:34.281 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:34.303 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:34.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:34.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:34.405 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:34.412 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:34 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 28642/230100 objects degraded (12.448%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:41:34.412 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:34 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 28642/230100 objects degraded (12.448%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:41:34.428 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:34.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:34.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:34.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:34 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 28642/230100 objects degraded (12.448%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:41:34.529 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:34.550 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:34.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:34.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:34.652 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:34.674 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:34.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:34.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:34.776 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:34.799 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:34.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:34.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:34.901 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:34.923 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:34.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:34.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:35.025 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:35.050 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:35.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:35.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:35.153 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:35.181 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:35.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:35.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:35.283 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:35.308 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:35.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:35.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:35.410 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:35.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:35 smithi102 ceph-mon[110183]: pgmap v909: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.6 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 232 op/s; 28499/227973 objects degraded (12.501%); 4091/227973 objects misplaced (1.795%); 6.7 MiB/s, 106 objects/s recovering 2024-09-13T14:41:35.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:35 smithi102 ceph-mon[115269]: pgmap v909: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.6 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 232 op/s; 28499/227973 objects degraded (12.501%); 4091/227973 objects misplaced (1.795%); 6.7 MiB/s, 106 objects/s recovering 2024-09-13T14:41:35.434 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:35.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:35.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:35.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:35 smithi195 ceph-mon[104262]: pgmap v909: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.6 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 232 op/s; 28499/227973 objects degraded (12.501%); 4091/227973 objects misplaced (1.795%); 6.7 MiB/s, 106 objects/s recovering 2024-09-13T14:41:35.537 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:35.565 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:35.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:35.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:35.764 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:35.788 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:35.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:35.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:35.891 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:35.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:35.918 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:35.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:36.019 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:36.044 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:36.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:36.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:36.122 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:41:36.123 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:41:36.146 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:36.169 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:36.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:36.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:36.271 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:36.295 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:36.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:36.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:36.298 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:41:36.397 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:36.421 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:36.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:36.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:36.523 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:36.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:36.547 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:36.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:36.570 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:36.649 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:36.673 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:36.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:36.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:36.773 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:41:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:41:36] "GET /metrics HTTP/1.1" 200 36539 "" "Prometheus/2.51.0" 2024-09-13T14:41:36.775 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:36.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:36.800 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:36.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:36.901 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:36.923 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:36.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:36.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:37.025 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:37.048 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:37.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:37.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:37.051 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T14:41:37.113 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:37.146 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is unset 2024-09-13T14:41:37.150 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:37.181 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:37.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:37.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:37.283 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:37.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:37.308 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:37.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:37.366 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:37 smithi102 ceph-mon[110183]: pgmap v910: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.6 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 225 op/s; 28439/226803 objects degraded (12.539%); 4091/226803 objects misplaced (1.804%); 6.5 MiB/s, 103 objects/s recovering 2024-09-13T14:41:37.366 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:37 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2164673587' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:41:37.366 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:37 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:41:37.366 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:37 smithi102 ceph-mon[115269]: pgmap v910: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.6 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 225 op/s; 28439/226803 objects degraded (12.539%); 4091/226803 objects misplaced (1.804%); 6.5 MiB/s, 103 objects/s recovering 2024-09-13T14:41:37.367 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:37 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2164673587' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:41:37.367 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:37 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:41:37.409 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:37.437 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:37.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:37.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:37.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:37 smithi195 ceph-mon[104262]: pgmap v910: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.6 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 225 op/s; 28439/226803 objects degraded (12.539%); 4091/226803 objects misplaced (1.804%); 6.5 MiB/s, 103 objects/s recovering 2024-09-13T14:41:37.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:37 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2164673587' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:41:37.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:37 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:41:37.539 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:37.565 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:37.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:37.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:37.667 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:37.692 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:37.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:37.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:37.793 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:37.818 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:37.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:37.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:37.919 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:37.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:37.941 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:37.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:38.042 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:38.067 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:38.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:38.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:38.169 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:38.193 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:38.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:38.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:38.295 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:38.320 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:38.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:38.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:38.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:38 smithi102 ceph-mon[110183]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:41:38.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:38 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:41:38.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:38 smithi102 ceph-mon[110183]: osdmap e273: 8 total, 8 up, 6 in 2024-09-13T14:41:38.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:38 smithi102 ceph-mon[115269]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:41:38.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:38 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:41:38.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:38 smithi102 ceph-mon[115269]: osdmap e273: 8 total, 8 up, 6 in 2024-09-13T14:41:38.421 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:38.450 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:38.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:38.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:38.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:38 smithi195 ceph-mon[104262]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:41:38.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:38 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:41:38.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:38 smithi195 ceph-mon[104262]: osdmap e273: 8 total, 8 up, 6 in 2024-09-13T14:41:38.552 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:38.578 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:38.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:38.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:38.680 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:38.709 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:38.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:38.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:38.810 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:38.833 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:38.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:38.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:38.935 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:38.957 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:38.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:38.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:39.058 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:39.081 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:39.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:39.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:39.183 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:39.207 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:39.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:39.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:39.309 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:39.333 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:39.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:39.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:39.337 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:39 smithi102 ceph-mon[110183]: pgmap v912: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.6 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 221 op/s; 28292/225009 objects degraded (12.574%); 3891/225009 objects misplaced (1.729%); 5.1 MiB/s, 82 objects/s recovering 2024-09-13T14:41:39.337 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:39 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 28292/225009 objects degraded (12.574%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:41:39.337 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:39 smithi102 ceph-mon[115269]: pgmap v912: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.6 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 221 op/s; 28292/225009 objects degraded (12.574%); 3891/225009 objects misplaced (1.729%); 5.1 MiB/s, 82 objects/s recovering 2024-09-13T14:41:39.337 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:39 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 28292/225009 objects degraded (12.574%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:41:39.436 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:39.461 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:39.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:39.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:39.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:39 smithi195 ceph-mon[104262]: pgmap v912: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.6 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 221 op/s; 28292/225009 objects degraded (12.574%); 3891/225009 objects misplaced (1.729%); 5.1 MiB/s, 82 objects/s recovering 2024-09-13T14:41:39.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:39 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 28292/225009 objects degraded (12.574%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:41:39.564 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:39.592 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:39.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:39.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:39.694 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:39.718 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:39.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:39.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:39.820 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:39.845 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:39.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:39.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:39.947 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:39.970 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:39.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:39.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:40.072 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:40.101 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:40.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:40.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:40.204 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:40.229 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:40.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:40.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:40.251 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:40.331 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:40.358 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:40.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:40.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:40.462 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:40.486 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:40.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:40.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:40.588 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:40.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:40.614 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:40.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:40.716 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:40.744 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:40.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:40.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:40.846 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:40.870 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:40.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:40.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:40.972 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:40.998 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:40.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:40.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:41.100 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:41.129 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:41.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:41.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:41.231 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:41.256 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:41.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:41.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:41.358 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:41.364 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:41:41.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:41.385 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:41.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:41.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:41 smithi102 ceph-mon[110183]: pgmap v913: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.6 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 218 op/s; 28227/223551 objects degraded (12.627%); 3892/223551 objects misplaced (1.741%); 5.0 MiB/s, 80 objects/s recovering 2024-09-13T14:41:41.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:41 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:41:41.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:41 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:41:41.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:41 smithi102 ceph-mon[115269]: pgmap v913: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.6 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 218 op/s; 28227/223551 objects degraded (12.627%); 3892/223551 objects misplaced (1.741%); 5.0 MiB/s, 80 objects/s recovering 2024-09-13T14:41:41.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:41 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:41:41.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:41 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:41:41.485 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:41.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:41 smithi195 ceph-mon[104262]: pgmap v913: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.6 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 218 op/s; 28227/223551 objects degraded (12.627%); 3892/223551 objects misplaced (1.741%); 5.0 MiB/s, 80 objects/s recovering 2024-09-13T14:41:41.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:41 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:41:41.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:41 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:41:41.515 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:41.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:41.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:41.617 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:41.643 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:41.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:41.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:41.745 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:41.770 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:41.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:41.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:41.872 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:41.895 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:41.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:41.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:41.915 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:41:41.916 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:41:41.997 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:42.022 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:42.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:42.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:42.124 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:42.153 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:42.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:42.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:42.254 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:42.278 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:42.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:42.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:42.381 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:42.407 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:42.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:42.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:42.461 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:41:42.461 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:41:42.492 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:41:42.493 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:41:42.509 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:42.513 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:41:42.513 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:41:42.534 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:42.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:42.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:42.536 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:41:42.536 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:41:42.557 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:41:42.557 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:41:42.576 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:41:42.576 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:41:42.595 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:41:42.595 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:41:42.612 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:41:42.613 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:41:42.636 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:42.659 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:42.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:42.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:42.761 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:42.787 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:42.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:42.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:42.889 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:42.912 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:42.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:42.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:43.014 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:43.044 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:43.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:43.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:43.145 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:43.174 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:43.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:43.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:43.276 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:43.301 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:43.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:43.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:43.303 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:43 smithi102 ceph-mon[110183]: pgmap v914: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.6 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 218 op/s; 28227/223551 objects degraded (12.627%); 3892/223551 objects misplaced (1.741%); 5.0 MiB/s, 80 objects/s recovering 2024-09-13T14:41:43.303 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:43 smithi102 ceph-mon[110183]: from='client.45749 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:41:43.303 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:43 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2707491223' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:41:43.303 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:43 smithi102 ceph-mon[115269]: pgmap v914: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.6 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 218 op/s; 28227/223551 objects degraded (12.627%); 3892/223551 objects misplaced (1.741%); 5.0 MiB/s, 80 objects/s recovering 2024-09-13T14:41:43.303 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:43 smithi102 ceph-mon[115269]: from='client.45749 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:41:43.303 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:43 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2707491223' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:41:43.403 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:43.429 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:43.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:43.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:43.436 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:41:43.436 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [], 2024-09-13T14:41:43.436 INFO:teuthology.orchestra.run.smithi102.stdout: "complaint_time": 30, 2024-09-13T14:41:43.436 INFO:teuthology.orchestra.run.smithi102.stdout: "num_blocked_ops": 0 2024-09-13T14:41:43.436 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:41:43.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:43 smithi195 ceph-mon[104262]: pgmap v914: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.6 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 218 op/s; 28227/223551 objects degraded (12.627%); 3892/223551 objects misplaced (1.741%); 5.0 MiB/s, 80 objects/s recovering 2024-09-13T14:41:43.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:43 smithi195 ceph-mon[104262]: from='client.45749 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:41:43.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:43 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2707491223' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:41:43.533 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:43.557 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:43.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:43.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:43.659 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:43.683 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:43.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:43.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:43.785 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:43.821 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:43.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:43.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:43.922 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:43.950 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:43.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:43.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:44.052 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:44.075 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:44.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:44.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:44.177 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:44.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:44.201 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:44.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:44.302 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:44.324 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:44.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:44.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:44.426 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:44.448 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:44.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:44.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:44.495 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:44 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:41:44.495 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:44 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 28227/223551 objects degraded (12.627%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:41:44.496 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:44 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:41:44.496 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:44 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 28227/223551 objects degraded (12.627%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:41:44.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:44 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:41:44.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:44 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 28227/223551 objects degraded (12.627%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:41:44.529 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":914,"stamp":"2024-09-13T14:41:41.500336+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4886435837,"num_objects":74517,"num_object_clones":0,"num_object_copies":223551,"num_objects_missing_on_primary":2582,"num_objects_missing":2582,"num_objects_degraded":28227,"num_objects_misplaced":3892,"num_objects_unfound":0,"num_objects_dirty":74517,"num_whiteouts":0,"num_read":6377,"num_read_kb":13834,"num_write":152260,"num_write_kb":7235010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62503,"num_bytes_recovered":4059368589,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93742,"ondisk_log_size":93742,"up":267,"acting":261,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":309,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":18801584,"kb_used_data":15684028,"kb_used_omap":26,"kb_used_meta":3117477,"kb_avail":543669328,"statfs":{"total":575970213888,"available":556717391872,"internally_reserved":0,"allocated":16060444672,"data_stored":16051574942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27144,"internal_metadata":3192296952},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-143065088,"num_objects":-2183,"num_object_clones":0,"num_object_copies":-6549,"num_objects_missing_on_primary":-161,"num_objects_missing":-161,"num_objects_degraded":-415,"num_objects_misplaced":-517,"num_objects_unfound":0,"num_objects_dirty":-2183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2183,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":803,"num_bytes_recovered":52625408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002743"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":439,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-13T14:41:37.149871+0000","last_change":"2024-09-13T14:36:13.859927+0000","last_active":"2024-09-13T14:41:37.149871+0000","last_peered":"2024-09-13T14:41:37.149871+0000","last_clean":"2024-09-13T14:41:37.149871+0000","last_became_active":"2024-09-13T14:36:13.859505+0000","last_became_peered":"2024-09-13T14:36:13.859505+0000","last_unstale":"2024-09-13T14:41:37.149871+0000","last_undegraded":"2024-09-13T14:41:37.149871+0000","last_fullsized":"2024-09-13T14:41:37.149871+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:12:20.989627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":274,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:41:26.123062+0000","last_change":"2024-09-13T14:37:48.130968+0000","last_active":"2024-09-13T14:41:26.123062+0000","last_peered":"2024-09-13T14:41:26.123062+0000","last_clean":"2024-09-13T14:41:26.123062+0000","last_became_active":"2024-09-13T14:37:48.130603+0000","last_became_peered":"2024-09-13T14:37:48.130603+0000","last_unstale":"2024-09-13T14:41:26.123062+0000","last_undegraded":"2024-09-13T14:41:26.123062+0000","last_fullsized":"2024-09-13T14:41:26.123062+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:42.626386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":70,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:18.109202+0000","last_change":"2024-09-13T14:36:13.862181+0000","last_active":"2024-09-13T14:41:18.109202+0000","last_peered":"2024-09-13T14:41:18.109202+0000","last_clean":"2024-09-13T14:41:18.109202+0000","last_became_active":"2024-09-13T14:36:13.861873+0000","last_became_peered":"2024-09-13T14:36:13.861873+0000","last_unstale":"2024-09-13T14:41:18.109202+0000","last_undegraded":"2024-09-13T14:41:18.109202+0000","last_fullsized":"2024-09-13T14:41:18.109202+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:14.085700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":76,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-13T14:41:37.161840+0000","last_change":"2024-09-13T14:38:53.896672+0000","last_active":"2024-09-13T14:41:37.161840+0000","last_peered":"2024-09-13T14:41:37.161840+0000","last_clean":"2024-09-13T14:41:37.161840+0000","last_became_active":"2024-09-13T14:38:53.896346+0000","last_became_peered":"2024-09-13T14:38:53.896346+0000","last_unstale":"2024-09-13T14:41:37.161840+0000","last_undegraded":"2024-09-13T14:41:37.161840+0000","last_fullsized":"2024-09-13T14:41:37.161840+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:57.789686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":76,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-13T14:41:37.161458+0000","last_change":"2024-09-13T14:38:53.893021+0000","last_active":"2024-09-13T14:41:37.161458+0000","last_peered":"2024-09-13T14:41:37.161458+0000","last_clean":"2024-09-13T14:41:37.161458+0000","last_became_active":"2024-09-13T14:38:53.892668+0000","last_became_peered":"2024-09-13T14:38:53.892668+0000","last_unstale":"2024-09-13T14:41:37.161458+0000","last_undegraded":"2024-09-13T14:41:37.161458+0000","last_fullsized":"2024-09-13T14:41:37.161458+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:57.180738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":242,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:41:26.123154+0000","last_change":"2024-09-13T14:37:48.134642+0000","last_active":"2024-09-13T14:41:26.123154+0000","last_peered":"2024-09-13T14:41:26.123154+0000","last_clean":"2024-09-13T14:41:26.123154+0000","last_became_active":"2024-09-13T14:37:48.134402+0000","last_became_peered":"2024-09-13T14:37:48.134402+0000","last_unstale":"2024-09-13T14:41:26.123154+0000","last_undegraded":"2024-09-13T14:41:26.123154+0000","last_fullsized":"2024-09-13T14:41:26.123154+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:51.313663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":457,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-13T14:41:37.149792+0000","last_change":"2024-09-13T14:38:53.903022+0000","last_active":"2024-09-13T14:41:37.149792+0000","last_peered":"2024-09-13T14:41:37.149792+0000","last_clean":"2024-09-13T14:41:37.149792+0000","last_became_active":"2024-09-13T14:38:53.902448+0000","last_became_peered":"2024-09-13T14:38:53.902448+0000","last_unstale":"2024-09-13T14:41:37.149792+0000","last_undegraded":"2024-09-13T14:41:37.149792+0000","last_fullsized":"2024-09-13T14:41:37.149792+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:24.690401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":79,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-13T14:41:37.160798+0000","last_change":"2024-09-13T14:38:03.554500+0000","last_active":"2024-09-13T14:41:37.160798+0000","last_peered":"2024-09-13T14:41:37.160798+0000","last_clean":"2024-09-13T14:41:37.160798+0000","last_became_active":"2024-09-13T14:38:03.532910+0000","last_became_peered":"2024-09-13T14:38:03.532910+0000","last_unstale":"2024-09-13T14:41:37.160798+0000","last_undegraded":"2024-09-13T14:41:37.160798+0000","last_fullsized":"2024-09-13T14:41:37.160798+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:27:50.843498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"272'9341","reported_seq":16251,"reported_epoch":272,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-13T14:41:37.084508+0000","last_change":"2024-09-13T14:40:32.799848+0000","last_active":"2024-09-13T14:41:37.084508+0000","last_peered":"2024-09-13T14:41:37.084508+0000","last_clean":"2024-09-13T14:37:46.417003+0000","last_became_active":"2024-09-13T14:37:49.690961+0000","last_became_peered":"2024-09-13T14:37:49.690961+0000","last_unstale":"2024-09-13T14:41:37.084508+0000","last_undegraded":"2024-09-13T14:41:37.084508+0000","last_fullsized":"2024-09-13T14:37:49.126746+0000","mapping_epoch":240,"log_start":"212'3600","ondisk_log_start":"212'3600","created":205,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5741,"log_dups_size":0,"ondisk_log_size":5741,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:48:52.526429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297467904,"num_objects":4539,"num_object_clones":0,"num_object_copies":13617,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1902,"num_objects_unfound":0,"num_objects_dirty":4539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9341,"num_write_kb":444160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7171,"num_bytes_recovered":469958656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,4],"avail_no_missing":["5","0","4"],"object_location_counts":[{"shards":"0,4,5","objects":1902},{"shards":"4,5","objects":2637}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":35,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:41:26.124327+0000","last_change":"2024-09-13T14:38:53.940238+0000","last_active":"2024-09-13T14:41:26.124327+0000","last_peered":"2024-09-13T14:41:26.124327+0000","last_clean":"2024-09-13T14:41:26.124327+0000","last_became_active":"2024-09-13T14:38:53.892825+0000","last_became_peered":"2024-09-13T14:38:53.892825+0000","last_unstale":"2024-09-13T14:41:26.124327+0000","last_undegraded":"2024-09-13T14:41:26.124327+0000","last_fullsized":"2024-09-13T14:41:26.124327+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:46.666581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":90,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:18.108795+0000","last_change":"2024-09-13T14:38:53.898907+0000","last_active":"2024-09-13T14:41:18.108795+0000","last_peered":"2024-09-13T14:41:18.108795+0000","last_clean":"2024-09-13T14:41:18.108795+0000","last_became_active":"2024-09-13T14:38:53.898572+0000","last_became_peered":"2024-09-13T14:38:53.898572+0000","last_unstale":"2024-09-13T14:41:18.108795+0000","last_undegraded":"2024-09-13T14:41:18.108795+0000","last_fullsized":"2024-09-13T14:41:18.108795+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:29.222150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"273'9519","reported_seq":23674,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-13T14:41:37.484783+0000","last_change":"2024-09-13T14:38:57.930032+0000","last_active":"2024-09-13T14:41:37.484783+0000","last_peered":"2024-09-13T14:41:37.484783+0000","last_clean":"2024-09-13T14:41:37.484783+0000","last_became_active":"2024-09-13T14:38:57.921458+0000","last_became_peered":"2024-09-13T14:38:57.921458+0000","last_unstale":"2024-09-13T14:41:37.484783+0000","last_undegraded":"2024-09-13T14:41:37.484783+0000","last_fullsized":"2024-09-13T14:41:37.484783+0000","mapping_epoch":256,"log_start":"213'4239","ondisk_log_start":"213'4239","created":205,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5280,"log_dups_size":0,"ondisk_log_size":5280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:27:12.294038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":309526528,"num_objects":4723,"num_object_clones":0,"num_object_copies":14169,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9519,"num_write_kb":455744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6990,"num_bytes_recovered":458096640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":35,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:41:26.123966+0000","last_change":"2024-09-13T14:38:53.904885+0000","last_active":"2024-09-13T14:41:26.123966+0000","last_peered":"2024-09-13T14:41:26.123966+0000","last_clean":"2024-09-13T14:41:26.123966+0000","last_became_active":"2024-09-13T14:38:53.891726+0000","last_became_peered":"2024-09-13T14:38:53.891726+0000","last_unstale":"2024-09-13T14:41:26.123966+0000","last_undegraded":"2024-09-13T14:41:26.123966+0000","last_fullsized":"2024-09-13T14:41:26.123966+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:57.079205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":280,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:41:26.136127+0000","last_change":"2024-09-13T14:37:48.160504+0000","last_active":"2024-09-13T14:41:26.136127+0000","last_peered":"2024-09-13T14:41:26.136127+0000","last_clean":"2024-09-13T14:41:26.136127+0000","last_became_active":"2024-09-13T14:37:48.160181+0000","last_became_peered":"2024-09-13T14:37:48.160181+0000","last_unstale":"2024-09-13T14:41:26.136127+0000","last_undegraded":"2024-09-13T14:41:26.136127+0000","last_fullsized":"2024-09-13T14:41:26.136127+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:37.087008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"273'9527","reported_seq":11304,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-13T14:41:39.233768+0000","last_change":"2024-09-13T14:38:04.531914+0000","last_active":"2024-09-13T14:41:39.233768+0000","last_peered":"2024-09-13T14:41:39.233768+0000","last_clean":"2024-09-13T14:41:39.233768+0000","last_became_active":"2024-09-13T14:38:04.523448+0000","last_became_peered":"2024-09-13T14:38:04.523448+0000","last_unstale":"2024-09-13T14:41:39.233768+0000","last_undegraded":"2024-09-13T14:41:39.233768+0000","last_fullsized":"2024-09-13T14:41:39.233768+0000","mapping_epoch":245,"log_start":"212'3600","ondisk_log_start":"212'3600","created":205,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5927,"log_dups_size":0,"ondisk_log_size":5927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:06.536896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304545792,"num_objects":4647,"num_object_clones":0,"num_object_copies":13941,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9527,"num_write_kb":453568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":811,"num_bytes_recovered":53149696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":272,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:41:26.123615+0000","last_change":"2024-09-13T14:37:49.611774+0000","last_active":"2024-09-13T14:41:26.123615+0000","last_peered":"2024-09-13T14:41:26.123615+0000","last_clean":"2024-09-13T14:41:26.123615+0000","last_became_active":"2024-09-13T14:37:49.611088+0000","last_became_peered":"2024-09-13T14:37:49.611088+0000","last_unstale":"2024-09-13T14:41:26.123615+0000","last_undegraded":"2024-09-13T14:41:26.123615+0000","last_fullsized":"2024-09-13T14:41:26.123615+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:08.933213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":31,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:41:26.123250+0000","last_change":"2024-09-13T14:38:53.899211+0000","last_active":"2024-09-13T14:41:26.123250+0000","last_peered":"2024-09-13T14:41:26.123250+0000","last_clean":"2024-09-13T14:41:26.123250+0000","last_became_active":"2024-09-13T14:38:53.898927+0000","last_became_peered":"2024-09-13T14:38:53.898927+0000","last_unstale":"2024-09-13T14:41:26.123250+0000","last_undegraded":"2024-09-13T14:41:26.123250+0000","last_fullsized":"2024-09-13T14:41:26.123250+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:00:28.857611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"273'9369","reported_seq":9448,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-13T14:41:37.147228+0000","last_change":"2024-09-13T14:35:24.004651+0000","last_active":"2024-09-13T14:41:37.147228+0000","last_peered":"2024-09-13T14:41:37.147228+0000","last_clean":"2024-09-13T14:41:37.147228+0000","last_became_active":"2024-09-13T14:35:24.004358+0000","last_became_peered":"2024-09-13T14:35:24.004358+0000","last_unstale":"2024-09-13T14:41:37.147228+0000","last_undegraded":"2024-09-13T14:41:37.147228+0000","last_fullsized":"2024-09-13T14:41:37.147228+0000","mapping_epoch":205,"log_start":"212'3084","ondisk_log_start":"212'3084","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6285,"log_dups_size":0,"ondisk_log_size":6285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:57:38.023669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301662208,"num_objects":4603,"num_object_clones":0,"num_object_copies":13809,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9369,"num_write_kb":447104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":81,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:18.109225+0000","last_change":"2024-09-13T14:37:49.168003+0000","last_active":"2024-09-13T14:41:18.109225+0000","last_peered":"2024-09-13T14:41:18.109225+0000","last_clean":"2024-09-13T14:41:18.109225+0000","last_became_active":"2024-09-13T14:37:49.167638+0000","last_became_peered":"2024-09-13T14:37:49.167638+0000","last_unstale":"2024-09-13T14:41:18.109225+0000","last_undegraded":"2024-09-13T14:41:18.109225+0000","last_fullsized":"2024-09-13T14:41:18.109225+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:29.051981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":69,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-13T14:41:37.161445+0000","last_change":"2024-09-13T14:38:03.528381+0000","last_active":"2024-09-13T14:41:37.161445+0000","last_peered":"2024-09-13T14:41:37.161445+0000","last_clean":"2024-09-13T14:41:37.161445+0000","last_became_active":"2024-09-13T14:38:03.527282+0000","last_became_peered":"2024-09-13T14:38:03.527282+0000","last_unstale":"2024-09-13T14:41:37.161445+0000","last_undegraded":"2024-09-13T14:41:37.161445+0000","last_fullsized":"2024-09-13T14:41:37.161445+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:24.435067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.f","version":"273'9429","reported_seq":9701,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-13T14:41:39.267544+0000","last_change":"2024-09-13T14:38:16.371079+0000","last_active":"2024-09-13T14:41:39.267544+0000","last_peered":"2024-09-13T14:41:39.267544+0000","last_clean":"2024-09-13T14:41:39.267544+0000","last_became_active":"2024-09-13T14:38:12.859883+0000","last_became_peered":"2024-09-13T14:38:12.859883+0000","last_unstale":"2024-09-13T14:41:39.267544+0000","last_undegraded":"2024-09-13T14:41:39.267544+0000","last_fullsized":"2024-09-13T14:41:39.267544+0000","mapping_epoch":245,"log_start":"212'3463","ondisk_log_start":"212'3463","created":205,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5966,"log_dups_size":0,"ondisk_log_size":5966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:08.436532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305201152,"num_objects":4657,"num_object_clones":0,"num_object_copies":13971,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9429,"num_write_kb":450752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":3670016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":427,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-13T14:41:37.149665+0000","last_change":"2024-09-13T14:38:03.549595+0000","last_active":"2024-09-13T14:41:37.149665+0000","last_peered":"2024-09-13T14:41:37.149665+0000","last_clean":"2024-09-13T14:41:37.149665+0000","last_became_active":"2024-09-13T14:38:03.533331+0000","last_became_peered":"2024-09-13T14:38:03.533331+0000","last_unstale":"2024-09-13T14:41:37.149665+0000","last_undegraded":"2024-09-13T14:41:37.149665+0000","last_fullsized":"2024-09-13T14:41:37.149665+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:59:02.470343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":256,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:41:26.136377+0000","last_change":"2024-09-13T14:37:48.127600+0000","last_active":"2024-09-13T14:41:26.136377+0000","last_peered":"2024-09-13T14:41:26.136377+0000","last_clean":"2024-09-13T14:41:26.136377+0000","last_became_active":"2024-09-13T14:37:48.127349+0000","last_became_peered":"2024-09-13T14:37:48.127349+0000","last_unstale":"2024-09-13T14:41:26.136377+0000","last_undegraded":"2024-09-13T14:41:26.136377+0000","last_fullsized":"2024-09-13T14:41:26.136377+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:57.268374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"272'9393","reported_seq":14282,"reported_epoch":272,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:41:37.127119+0000","last_change":"2024-09-13T14:38:14.979229+0000","last_active":"2024-09-13T14:41:37.127119+0000","last_peered":"2024-09-13T14:41:37.127119+0000","last_clean":"2024-09-13T14:36:13.131873+0000","last_became_active":"2024-09-13T14:38:10.443954+0000","last_became_peered":"2024-09-13T14:38:10.443954+0000","last_unstale":"2024-09-13T14:41:37.127119+0000","last_undegraded":"2024-09-13T14:38:10.384643+0000","last_fullsized":"2024-09-13T14:38:04.517046+0000","mapping_epoch":245,"log_start":"212'3700","ondisk_log_start":"212'3700","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5693,"log_dups_size":0,"ondisk_log_size":5693,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:26:53.493170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302579712,"num_objects":4617,"num_object_clones":0,"num_object_copies":13851,"num_objects_missing_on_primary":2392,"num_objects_missing":2392,"num_objects_degraded":2533,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9393,"num_write_kb":448320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2384,"num_bytes_recovered":156172288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":1},{"shards":"0,3,5,7","objects":9},{"shards":"0,3,7","objects":2391},{"shards":"0,5","objects":2211},{"shards":"3,5,7","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":189,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:41:26.138208+0000","last_change":"2024-09-13T14:37:48.127604+0000","last_active":"2024-09-13T14:41:26.138208+0000","last_peered":"2024-09-13T14:41:26.138208+0000","last_clean":"2024-09-13T14:41:26.138208+0000","last_became_active":"2024-09-13T14:37:48.127300+0000","last_became_peered":"2024-09-13T14:37:48.127300+0000","last_unstale":"2024-09-13T14:41:26.138208+0000","last_undegraded":"2024-09-13T14:41:26.138208+0000","last_fullsized":"2024-09-13T14:41:26.138208+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:51.999506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":309,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:41:26.135984+0000","last_change":"2024-09-13T14:38:03.529967+0000","last_active":"2024-09-13T14:41:26.135984+0000","last_peered":"2024-09-13T14:41:26.135984+0000","last_clean":"2024-09-13T14:41:26.135984+0000","last_became_active":"2024-09-13T14:38:03.529646+0000","last_became_peered":"2024-09-13T14:38:03.529646+0000","last_unstale":"2024-09-13T14:41:26.135984+0000","last_undegraded":"2024-09-13T14:41:26.135984+0000","last_fullsized":"2024-09-13T14:41:26.135984+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:52:05.193279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"273'9511","reported_seq":18658,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-13T14:41:39.187814+0000","last_change":"2024-09-13T14:40:34.583237+0000","last_active":"2024-09-13T14:41:39.187814+0000","last_peered":"2024-09-13T14:41:39.187814+0000","last_clean":"2024-09-13T14:41:39.187814+0000","last_became_active":"2024-09-13T14:40:34.563989+0000","last_became_peered":"2024-09-13T14:40:34.563989+0000","last_unstale":"2024-09-13T14:41:39.187814+0000","last_undegraded":"2024-09-13T14:41:39.187814+0000","last_fullsized":"2024-09-13T14:41:39.187814+0000","mapping_epoch":266,"log_start":"212'3600","ondisk_log_start":"212'3600","created":205,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5911,"log_dups_size":0,"ondisk_log_size":5911,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:34:43.006051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303300632,"num_objects":4629,"num_object_clones":0,"num_object_copies":13887,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4629,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9511,"num_write_kb":452417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4489,"num_bytes_recovered":289865752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":269,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:41:26.136614+0000","last_change":"2024-09-13T14:36:13.860367+0000","last_active":"2024-09-13T14:41:26.136614+0000","last_peered":"2024-09-13T14:41:26.136614+0000","last_clean":"2024-09-13T14:41:26.136614+0000","last_became_active":"2024-09-13T14:36:13.859982+0000","last_became_peered":"2024-09-13T14:36:13.859982+0000","last_unstale":"2024-09-13T14:41:26.136614+0000","last_undegraded":"2024-09-13T14:41:26.136614+0000","last_fullsized":"2024-09-13T14:41:26.136614+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:08:23.311350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":265,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:41:26.136127+0000","last_change":"2024-09-13T14:37:48.147360+0000","last_active":"2024-09-13T14:41:26.136127+0000","last_peered":"2024-09-13T14:41:26.136127+0000","last_clean":"2024-09-13T14:41:26.136127+0000","last_became_active":"2024-09-13T14:37:48.146885+0000","last_became_peered":"2024-09-13T14:37:48.146885+0000","last_unstale":"2024-09-13T14:41:26.136127+0000","last_undegraded":"2024-09-13T14:41:26.136127+0000","last_fullsized":"2024-09-13T14:41:26.136127+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:27.841810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"273'9377","reported_seq":18631,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-13T14:41:37.196355+0000","last_change":"2024-09-13T14:38:39.650332+0000","last_active":"2024-09-13T14:41:37.196355+0000","last_peered":"2024-09-13T14:41:37.196355+0000","last_clean":"2024-09-13T14:41:37.196355+0000","last_became_active":"2024-09-13T14:38:39.640835+0000","last_became_peered":"2024-09-13T14:38:39.640835+0000","last_unstale":"2024-09-13T14:41:37.196355+0000","last_undegraded":"2024-09-13T14:41:37.196355+0000","last_fullsized":"2024-09-13T14:41:37.196355+0000","mapping_epoch":251,"log_start":"213'4051","ondisk_log_start":"213'4051","created":205,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5326,"log_dups_size":0,"ondisk_log_size":5326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:48.176606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":307822592,"num_objects":4697,"num_object_clones":0,"num_object_copies":14091,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9377,"num_write_kb":450368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4544,"num_bytes_recovered":292945920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":445,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-13T14:41:37.149253+0000","last_change":"2024-09-13T14:38:53.900041+0000","last_active":"2024-09-13T14:41:37.149253+0000","last_peered":"2024-09-13T14:41:37.149253+0000","last_clean":"2024-09-13T14:41:37.149253+0000","last_became_active":"2024-09-13T14:38:53.899203+0000","last_became_peered":"2024-09-13T14:38:53.899203+0000","last_unstale":"2024-09-13T14:41:37.149253+0000","last_undegraded":"2024-09-13T14:41:37.149253+0000","last_fullsized":"2024-09-13T14:41:37.149253+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:28:07.883094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":377,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-13T14:41:37.149937+0000","last_change":"2024-09-13T14:26:15.823198+0000","last_active":"2024-09-13T14:41:37.149937+0000","last_peered":"2024-09-13T14:41:37.149937+0000","last_clean":"2024-09-13T14:41:37.149937+0000","last_became_active":"2024-09-13T14:26:15.822717+0000","last_became_peered":"2024-09-13T14:26:15.822717+0000","last_unstale":"2024-09-13T14:41:37.149937+0000","last_undegraded":"2024-09-13T14:41:37.149937+0000","last_fullsized":"2024-09-13T14:41:37.149937+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:22.471622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.6","version":"273'9579","reported_seq":14491,"reported_epoch":273,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-13T14:41:39.322070+0000","last_change":"2024-09-13T14:38:55.132683+0000","last_active":"2024-09-13T14:41:39.322070+0000","last_peered":"2024-09-13T14:41:39.322070+0000","last_clean":"2024-09-13T14:36:13.124368+0000","last_became_active":"2024-09-13T14:38:55.120883+0000","last_became_peered":"2024-09-13T14:38:55.120883+0000","last_unstale":"2024-09-13T14:41:39.322070+0000","last_undegraded":"2024-09-13T14:38:55.026610+0000","last_fullsized":"2024-09-13T14:41:39.322070+0000","mapping_epoch":254,"log_start":"212'3619","ondisk_log_start":"212'3619","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5960,"log_dups_size":0,"ondisk_log_size":5960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302710784,"num_objects":4619,"num_object_clones":0,"num_object_copies":13857,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2631,"num_objects_misplaced":1988,"num_objects_unfound":0,"num_objects_dirty":4619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9579,"num_write_kb":454336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2374,"num_bytes_recovered":144375808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,3,5],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3,4","objects":2352},{"shards":"0,3,5","objects":2267}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":441,"reported_epoch":273,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-13T14:41:37.148869+0000","last_change":"2024-09-13T14:38:54.899624+0000","last_active":"2024-09-13T14:41:37.148869+0000","last_peered":"2024-09-13T14:41:37.148869+0000","last_clean":"2024-09-13T14:17:27.401592+0000","last_became_active":"2024-09-13T14:38:54.899282+0000","last_became_peered":"2024-09-13T14:38:54.899282+0000","last_unstale":"2024-09-13T14:41:37.148869+0000","last_undegraded":"2024-09-13T14:38:54.891176+0000","last_fullsized":"2024-09-13T14:41:37.148869+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:07:41.927740+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[3,0,6],"avail_no_missing":["3","4","6"],"object_location_counts":[{"shards":"3,4,6","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":510,"reported_epoch":271,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:41:18.109180+0000","last_change":"2024-09-13T14:38:54.904329+0000","last_active":"2024-09-13T14:41:18.109180+0000","last_peered":"2024-09-13T14:41:18.109180+0000","last_clean":"2024-09-13T14:38:39.648017+0000","last_became_active":"2024-09-13T14:38:54.903798+0000","last_became_peered":"2024-09-13T14:38:54.903798+0000","last_unstale":"2024-09-13T14:41:18.109180+0000","last_undegraded":"2024-09-13T14:38:54.890505+0000","last_fullsized":"2024-09-13T14:38:54.889701+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:18.120931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,3],"avail_no_missing":["5","3","6"],"object_location_counts":[{"shards":"3,5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":380,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-13T14:41:37.148990+0000","last_change":"2024-09-13T14:37:49.160719+0000","last_active":"2024-09-13T14:41:37.148990+0000","last_peered":"2024-09-13T14:41:37.148990+0000","last_clean":"2024-09-13T14:41:37.148990+0000","last_became_active":"2024-09-13T14:37:49.160401+0000","last_became_peered":"2024-09-13T14:37:49.160401+0000","last_unstale":"2024-09-13T14:41:37.148990+0000","last_undegraded":"2024-09-13T14:41:37.148990+0000","last_fullsized":"2024-09-13T14:41:37.148990+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:21.737945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":116,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:41:26.135898+0000","last_change":"2024-09-13T14:37:48.172940+0000","last_active":"2024-09-13T14:41:26.135898+0000","last_peered":"2024-09-13T14:41:26.135898+0000","last_clean":"2024-09-13T14:41:26.135898+0000","last_became_active":"2024-09-13T14:37:48.172060+0000","last_became_peered":"2024-09-13T14:37:48.172060+0000","last_unstale":"2024-09-13T14:41:26.135898+0000","last_undegraded":"2024-09-13T14:41:26.135898+0000","last_fullsized":"2024-09-13T14:41:26.135898+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:07.480645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.5","version":"273'9411","reported_seq":25766,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-13T14:41:37.348456+0000","last_change":"2024-09-13T14:39:34.446699+0000","last_active":"2024-09-13T14:41:37.348456+0000","last_peered":"2024-09-13T14:41:37.348456+0000","last_clean":"2024-09-13T14:41:37.348456+0000","last_became_active":"2024-09-13T14:38:04.600616+0000","last_became_peered":"2024-09-13T14:38:04.600616+0000","last_unstale":"2024-09-13T14:41:37.348456+0000","last_undegraded":"2024-09-13T14:41:37.348456+0000","last_fullsized":"2024-09-13T14:41:37.348456+0000","mapping_epoch":245,"log_start":"212'3116","ondisk_log_start":"212'3116","created":205,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6295,"log_dups_size":0,"ondisk_log_size":6295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:00:44.321678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310050816,"num_objects":4731,"num_object_clones":0,"num_object_copies":14193,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9411,"num_write_kb":452544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8120,"num_bytes_recovered":521076736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":489,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:41:26.122949+0000","last_change":"2024-09-13T14:26:15.835178+0000","last_active":"2024-09-13T14:41:26.122949+0000","last_peered":"2024-09-13T14:41:26.122949+0000","last_clean":"2024-09-13T14:41:26.122949+0000","last_became_active":"2024-09-13T14:26:15.820283+0000","last_became_peered":"2024-09-13T14:26:15.820283+0000","last_unstale":"2024-09-13T14:41:26.122949+0000","last_undegraded":"2024-09-13T14:41:26.122949+0000","last_fullsized":"2024-09-13T14:41:26.122949+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:30.340038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":69,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-13T14:41:37.160865+0000","last_change":"2024-09-13T14:38:03.525682+0000","last_active":"2024-09-13T14:41:37.160865+0000","last_peered":"2024-09-13T14:41:37.160865+0000","last_clean":"2024-09-13T14:41:37.160865+0000","last_became_active":"2024-09-13T14:38:03.525406+0000","last_became_peered":"2024-09-13T14:38:03.525406+0000","last_unstale":"2024-09-13T14:41:37.160865+0000","last_undegraded":"2024-09-13T14:41:37.160865+0000","last_fullsized":"2024-09-13T14:41:37.160865+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:41:28.970300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":76,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-13T14:41:37.161572+0000","last_change":"2024-09-13T14:38:53.991738+0000","last_active":"2024-09-13T14:41:37.161572+0000","last_peered":"2024-09-13T14:41:37.161572+0000","last_clean":"2024-09-13T14:41:37.161572+0000","last_became_active":"2024-09-13T14:38:53.991417+0000","last_became_peered":"2024-09-13T14:38:53.991417+0000","last_unstale":"2024-09-13T14:41:37.161572+0000","last_undegraded":"2024-09-13T14:41:37.161572+0000","last_fullsized":"2024-09-13T14:41:37.161572+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:50:38.559838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"273'9483","reported_seq":23652,"reported_epoch":273,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-13T14:41:39.260284+0000","last_change":"2024-09-13T14:38:17.472071+0000","last_active":"2024-09-13T14:41:39.260284+0000","last_peered":"2024-09-13T14:41:39.260284+0000","last_clean":"2024-09-13T14:37:46.290754+0000","last_became_active":"2024-09-13T14:38:17.381753+0000","last_became_peered":"2024-09-13T14:38:17.381753+0000","last_unstale":"2024-09-13T14:41:39.260284+0000","last_undegraded":"2024-09-13T14:38:13.504140+0000","last_fullsized":"2024-09-13T14:41:39.260284+0000","mapping_epoch":245,"log_start":"212'3559","ondisk_log_start":"212'3559","created":205,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5924,"log_dups_size":0,"ondisk_log_size":5924,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302972928,"num_objects":4623,"num_object_clones":0,"num_object_copies":13869,"num_objects_missing_on_primary":83,"num_objects_missing":83,"num_objects_degraded":4582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9483,"num_write_kb":451392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7053,"num_bytes_recovered":462225408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"0,4,5","objects":4413},{"shards":"0,4,6","objects":210}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":790,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-13T14:41:37.150053+0000","last_change":"2024-09-13T14:37:48.131799+0000","last_active":"2024-09-13T14:41:37.150053+0000","last_peered":"2024-09-13T14:41:37.150053+0000","last_clean":"2024-09-13T14:41:37.150053+0000","last_became_active":"2024-09-13T14:37:48.130837+0000","last_became_peered":"2024-09-13T14:37:48.130837+0000","last_unstale":"2024-09-13T14:41:37.150053+0000","last_undegraded":"2024-09-13T14:41:37.150053+0000","last_fullsized":"2024-09-13T14:41:37.150053+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:06:32.663770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":309,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-13T14:41:37.161048+0000","last_change":"2024-09-13T14:37:49.301531+0000","last_active":"2024-09-13T14:41:37.161048+0000","last_peered":"2024-09-13T14:41:37.161048+0000","last_clean":"2024-09-13T14:41:37.161048+0000","last_became_active":"2024-09-13T14:37:49.301249+0000","last_became_peered":"2024-09-13T14:37:49.301249+0000","last_unstale":"2024-09-13T14:41:37.161048+0000","last_undegraded":"2024-09-13T14:41:37.161048+0000","last_fullsized":"2024-09-13T14:41:37.161048+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:10.977710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":97,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:18.109610+0000","last_change":"2024-09-13T14:38:53.901427+0000","last_active":"2024-09-13T14:41:18.109610+0000","last_peered":"2024-09-13T14:41:18.109610+0000","last_clean":"2024-09-13T14:41:18.109610+0000","last_became_active":"2024-09-13T14:38:53.901157+0000","last_became_peered":"2024-09-13T14:38:53.901157+0000","last_unstale":"2024-09-13T14:41:18.109610+0000","last_undegraded":"2024-09-13T14:41:18.109610+0000","last_fullsized":"2024-09-13T14:41:18.109610+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:28:21.713025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"273'9414","reported_seq":23472,"reported_epoch":273,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-13T14:41:39.253086+0000","last_change":"2024-09-13T14:38:16.552896+0000","last_active":"2024-09-13T14:41:39.253086+0000","last_peered":"2024-09-13T14:41:39.253086+0000","last_clean":"2024-09-13T14:37:46.405415+0000","last_became_active":"2024-09-13T14:38:16.427510+0000","last_became_peered":"2024-09-13T14:38:16.427510+0000","last_unstale":"2024-09-13T14:41:39.253086+0000","last_undegraded":"2024-09-13T14:38:12.738464+0000","last_fullsized":"2024-09-13T14:41:39.253086+0000","mapping_epoch":245,"log_start":"212'3518","ondisk_log_start":"212'3518","created":205,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5896,"log_dups_size":0,"ondisk_log_size":5896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300417024,"num_objects":4584,"num_object_clones":0,"num_object_copies":13752,"num_objects_missing_on_primary":100,"num_objects_missing":100,"num_objects_degraded":4572,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9414,"num_write_kb":447936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6999,"num_bytes_recovered":458686464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"0,3,6","objects":179},{"shards":"0,3,7","objects":4405}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":979,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:18.108867+0000","last_change":"2024-09-13T14:37:49.147498+0000","last_active":"2024-09-13T14:41:18.108867+0000","last_peered":"2024-09-13T14:41:18.108867+0000","last_clean":"2024-09-13T14:41:18.108867+0000","last_became_active":"2024-09-13T14:37:49.147028+0000","last_became_peered":"2024-09-13T14:37:49.147028+0000","last_unstale":"2024-09-13T14:41:18.108867+0000","last_undegraded":"2024-09-13T14:41:18.108867+0000","last_fullsized":"2024-09-13T14:41:18.108867+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:11:29.050178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":278,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:41:26.136944+0000","last_change":"2024-09-13T14:38:03.528133+0000","last_active":"2024-09-13T14:41:26.136944+0000","last_peered":"2024-09-13T14:41:26.136944+0000","last_clean":"2024-09-13T14:41:26.136944+0000","last_became_active":"2024-09-13T14:38:03.527829+0000","last_became_peered":"2024-09-13T14:38:03.527829+0000","last_unstale":"2024-09-13T14:41:26.136944+0000","last_undegraded":"2024-09-13T14:41:26.136944+0000","last_fullsized":"2024-09-13T14:41:26.136944+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:48.254169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":429,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-13T14:41:37.149779+0000","last_change":"2024-09-13T14:38:53.899755+0000","last_active":"2024-09-13T14:41:37.149779+0000","last_peered":"2024-09-13T14:41:37.149779+0000","last_clean":"2024-09-13T14:41:37.149779+0000","last_became_active":"2024-09-13T14:38:53.899448+0000","last_became_peered":"2024-09-13T14:38:53.899448+0000","last_unstale":"2024-09-13T14:41:37.149779+0000","last_undegraded":"2024-09-13T14:41:37.149779+0000","last_fullsized":"2024-09-13T14:41:37.149779+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:55.897035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"273'9614","reported_seq":9711,"reported_epoch":273,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:41:37.635153+0000","last_change":"2024-09-13T14:38:03.567892+0000","last_active":"2024-09-13T14:41:37.635153+0000","last_peered":"2024-09-13T14:41:37.635153+0000","last_clean":"2024-09-13T14:35:51.090791+0000","last_became_active":"2024-09-13T14:37:49.665025+0000","last_became_peered":"2024-09-13T14:37:49.665025+0000","last_unstale":"2024-09-13T14:41:37.635153+0000","last_undegraded":"2024-09-13T14:37:49.169101+0000","last_fullsized":"2024-09-13T14:37:49.121596+0000","mapping_epoch":240,"log_start":"212'3563","ondisk_log_start":"212'3563","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6051,"log_dups_size":0,"ondisk_log_size":6051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:02.321504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":307232768,"num_objects":4688,"num_object_clones":0,"num_object_copies":14064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4501,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9614,"num_write_kb":457664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":253}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":81,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:18.109094+0000","last_change":"2024-09-13T14:37:49.149770+0000","last_active":"2024-09-13T14:41:18.109094+0000","last_peered":"2024-09-13T14:41:18.109094+0000","last_clean":"2024-09-13T14:41:18.109094+0000","last_became_active":"2024-09-13T14:37:49.148991+0000","last_became_peered":"2024-09-13T14:37:49.148991+0000","last_unstale":"2024-09-13T14:41:18.109094+0000","last_undegraded":"2024-09-13T14:41:18.109094+0000","last_fullsized":"2024-09-13T14:41:18.109094+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:04:04.193366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":1128,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-13T14:41:37.197504+0000","last_change":"2024-09-13T14:30:58.928302+0000","last_active":"2024-09-13T14:41:37.197504+0000","last_peered":"2024-09-13T14:41:37.197504+0000","last_clean":"2024-09-13T14:41:37.197504+0000","last_became_active":"2024-09-13T14:30:58.927718+0000","last_became_peered":"2024-09-13T14:30:58.927718+0000","last_unstale":"2024-09-13T14:41:37.197504+0000","last_undegraded":"2024-09-13T14:41:37.197504+0000","last_fullsized":"2024-09-13T14:41:37.197504+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:37.323036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":88,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:18.108946+0000","last_change":"2024-09-13T14:38:03.534628+0000","last_active":"2024-09-13T14:41:18.108946+0000","last_peered":"2024-09-13T14:41:18.108946+0000","last_clean":"2024-09-13T14:41:18.108946+0000","last_became_active":"2024-09-13T14:38:03.534262+0000","last_became_peered":"2024-09-13T14:38:03.534262+0000","last_unstale":"2024-09-13T14:41:18.108946+0000","last_undegraded":"2024-09-13T14:41:18.108946+0000","last_fullsized":"2024-09-13T14:41:18.108946+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:31.168783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"272'9369","reported_seq":23527,"reported_epoch":272,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:41:37.090709+0000","last_change":"2024-09-13T14:38:53.944111+0000","last_active":"2024-09-13T14:41:37.090709+0000","last_peered":"2024-09-13T14:41:37.090709+0000","last_clean":"2024-09-13T14:37:47.047672+0000","last_became_active":"2024-09-13T14:37:49.664294+0000","last_became_peered":"2024-09-13T14:37:49.664294+0000","last_unstale":"2024-09-13T14:41:37.090709+0000","last_undegraded":"2024-09-13T14:38:53.940822+0000","last_fullsized":"2024-09-13T14:37:49.126627+0000","mapping_epoch":240,"log_start":"212'3700","ondisk_log_start":"212'3700","created":205,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5669,"log_dups_size":0,"ondisk_log_size":5669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:11.766692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306905088,"num_objects":4683,"num_object_clones":0,"num_object_copies":14049,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4535,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9369,"num_write_kb":449664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7009,"num_bytes_recovered":459341824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3],"avail_no_missing":["5","0","3"],"object_location_counts":[{"shards":"0,3,5","objects":4460},{"shards":"3,5","objects":223}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":100,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:18.109384+0000","last_change":"2024-09-13T14:38:03.534303+0000","last_active":"2024-09-13T14:41:18.109384+0000","last_peered":"2024-09-13T14:41:18.109384+0000","last_clean":"2024-09-13T14:41:18.109384+0000","last_became_active":"2024-09-13T14:38:03.533171+0000","last_became_peered":"2024-09-13T14:38:03.533171+0000","last_unstale":"2024-09-13T14:41:18.109384+0000","last_undegraded":"2024-09-13T14:41:18.109384+0000","last_fullsized":"2024-09-13T14:41:18.109384+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:00:16.553526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":794,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-13T14:41:37.144815+0000","last_change":"2024-09-13T14:38:04.596192+0000","last_active":"2024-09-13T14:41:37.144815+0000","last_peered":"2024-09-13T14:41:37.144815+0000","last_clean":"2024-09-13T14:41:37.144815+0000","last_became_active":"2024-09-13T14:38:04.593299+0000","last_became_peered":"2024-09-13T14:38:04.593299+0000","last_unstale":"2024-09-13T14:41:37.144815+0000","last_undegraded":"2024-09-13T14:41:37.144815+0000","last_fullsized":"2024-09-13T14:41:37.144815+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:34.241410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":69,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-13T14:41:37.161525+0000","last_change":"2024-09-13T14:38:03.533120+0000","last_active":"2024-09-13T14:41:37.161525+0000","last_peered":"2024-09-13T14:41:37.161525+0000","last_clean":"2024-09-13T14:41:37.161525+0000","last_became_active":"2024-09-13T14:38:03.532829+0000","last_became_peered":"2024-09-13T14:38:03.532829+0000","last_unstale":"2024-09-13T14:41:37.161525+0000","last_undegraded":"2024-09-13T14:41:37.161525+0000","last_fullsized":"2024-09-13T14:41:37.161525+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:24.514661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"273'9543","reported_seq":9699,"reported_epoch":273,"state":"active+undersized+degraded+remapped+backfill_wait","last_fresh":"2024-09-13T14:41:37.333401+0000","last_change":"2024-09-13T14:37:49.295861+0000","last_active":"2024-09-13T14:41:37.333401+0000","last_peered":"2024-09-13T14:41:37.333401+0000","last_clean":"2024-09-13T14:35:51.078708+0000","last_became_active":"2024-09-13T14:37:49.290403+0000","last_became_peered":"2024-09-13T14:37:49.290403+0000","last_unstale":"2024-09-13T14:41:37.333401+0000","last_undegraded":"2024-09-13T14:37:49.147402+0000","last_fullsized":"2024-09-13T14:37:49.117202+0000","mapping_epoch":240,"log_start":"213'4218","ondisk_log_start":"213'4218","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5325,"log_dups_size":0,"ondisk_log_size":5325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318177280,"num_objects":4855,"num_object_clones":0,"num_object_copies":14565,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9543,"num_write_kb":460736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":4855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":471,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-13T14:41:37.150023+0000","last_change":"2024-09-13T14:38:53.938227+0000","last_active":"2024-09-13T14:41:37.150023+0000","last_peered":"2024-09-13T14:41:37.150023+0000","last_clean":"2024-09-13T14:41:37.150023+0000","last_became_active":"2024-09-13T14:38:53.902867+0000","last_became_peered":"2024-09-13T14:38:53.902867+0000","last_unstale":"2024-09-13T14:41:37.150023+0000","last_undegraded":"2024-09-13T14:41:37.150023+0000","last_fullsized":"2024-09-13T14:41:37.150023+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:16.831756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":572,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:41:26.136257+0000","last_change":"2024-09-13T14:37:48.147677+0000","last_active":"2024-09-13T14:41:26.136257+0000","last_peered":"2024-09-13T14:41:26.136257+0000","last_clean":"2024-09-13T14:41:26.136257+0000","last_became_active":"2024-09-13T14:37:48.147001+0000","last_became_peered":"2024-09-13T14:37:48.147001+0000","last_unstale":"2024-09-13T14:41:26.136257+0000","last_undegraded":"2024-09-13T14:41:26.136257+0000","last_fullsized":"2024-09-13T14:41:26.136257+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:35:00.653576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":453,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-13T14:41:37.148733+0000","last_change":"2024-09-13T14:38:53.900107+0000","last_active":"2024-09-13T14:41:37.148733+0000","last_peered":"2024-09-13T14:41:37.148733+0000","last_clean":"2024-09-13T14:41:37.148733+0000","last_became_active":"2024-09-13T14:38:53.899556+0000","last_became_peered":"2024-09-13T14:38:53.899556+0000","last_unstale":"2024-09-13T14:41:37.148733+0000","last_undegraded":"2024-09-13T14:41:37.148733+0000","last_fullsized":"2024-09-13T14:41:37.148733+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:15:47.918163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"273'9258","reported_seq":18217,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-13T14:41:37.423348+0000","last_change":"2024-09-13T14:39:54.921239+0000","last_active":"2024-09-13T14:41:37.423348+0000","last_peered":"2024-09-13T14:41:37.423348+0000","last_clean":"2024-09-13T14:41:37.423348+0000","last_became_active":"2024-09-13T14:39:54.907894+0000","last_became_peered":"2024-09-13T14:39:54.907894+0000","last_unstale":"2024-09-13T14:41:37.423348+0000","last_undegraded":"2024-09-13T14:41:37.423348+0000","last_fullsized":"2024-09-13T14:41:37.423348+0000","mapping_epoch":263,"log_start":"213'3947","ondisk_log_start":"213'3947","created":205,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5311,"log_dups_size":0,"ondisk_log_size":5311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:52:04.619238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301072384,"num_objects":4594,"num_object_clones":0,"num_object_copies":13782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9258,"num_write_kb":443264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4395,"num_bytes_recovered":283770880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":84,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-13T14:41:37.161371+0000","last_change":"2024-09-13T14:38:53.901806+0000","last_active":"2024-09-13T14:41:37.161371+0000","last_peered":"2024-09-13T14:41:37.161371+0000","last_clean":"2024-09-13T14:41:37.161371+0000","last_became_active":"2024-09-13T14:38:53.891055+0000","last_became_peered":"2024-09-13T14:38:53.891055+0000","last_unstale":"2024-09-13T14:41:37.161371+0000","last_undegraded":"2024-09-13T14:41:37.161371+0000","last_fullsized":"2024-09-13T14:41:37.161371+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:03:00.416198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":29,"reported_epoch":272,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-13T14:41:26.124412+0000","last_change":"2024-09-13T14:38:53.900902+0000","last_active":"2024-09-13T14:41:26.124412+0000","last_peered":"2024-09-13T14:41:26.124412+0000","last_clean":"2024-09-13T14:38:39.635779+0000","last_became_active":"2024-09-13T14:38:53.900626+0000","last_became_peered":"2024-09-13T14:38:53.900626+0000","last_unstale":"2024-09-13T14:41:26.124412+0000","last_undegraded":"2024-09-13T14:38:53.885778+0000","last_fullsized":"2024-09-13T14:41:26.124412+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:37:21.715778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":7,"num_objects_missing":7,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":["3","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":262,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:41:26.136250+0000","last_change":"2024-09-13T14:37:49.172218+0000","last_active":"2024-09-13T14:41:26.136250+0000","last_peered":"2024-09-13T14:41:26.136250+0000","last_clean":"2024-09-13T14:41:26.136250+0000","last_became_active":"2024-09-13T14:37:49.171718+0000","last_became_peered":"2024-09-13T14:37:49.171718+0000","last_unstale":"2024-09-13T14:41:26.136250+0000","last_undegraded":"2024-09-13T14:41:26.136250+0000","last_fullsized":"2024-09-13T14:41:26.136250+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:46:54.626521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":284,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:41:26.137047+0000","last_change":"2024-09-13T14:37:49.230094+0000","last_active":"2024-09-13T14:41:26.137047+0000","last_peered":"2024-09-13T14:41:26.137047+0000","last_clean":"2024-09-13T14:41:26.137047+0000","last_became_active":"2024-09-13T14:37:49.229393+0000","last_became_peered":"2024-09-13T14:37:49.229393+0000","last_unstale":"2024-09-13T14:41:26.137047+0000","last_undegraded":"2024-09-13T14:41:26.137047+0000","last_fullsized":"2024-09-13T14:41:26.137047+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:02:28.690318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":264,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:41:26.136516+0000","last_change":"2024-09-13T14:37:48.128447+0000","last_active":"2024-09-13T14:41:26.136516+0000","last_peered":"2024-09-13T14:41:26.136516+0000","last_clean":"2024-09-13T14:41:26.136516+0000","last_became_active":"2024-09-13T14:37:48.128143+0000","last_became_peered":"2024-09-13T14:37:48.128143+0000","last_unstale":"2024-09-13T14:41:26.136516+0000","last_undegraded":"2024-09-13T14:41:26.136516+0000","last_fullsized":"2024-09-13T14:41:26.136516+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:22.204039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":69,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-13T14:41:37.161759+0000","last_change":"2024-09-13T14:38:03.531683+0000","last_active":"2024-09-13T14:41:37.161759+0000","last_peered":"2024-09-13T14:41:37.161759+0000","last_clean":"2024-09-13T14:41:37.161759+0000","last_became_active":"2024-09-13T14:38:03.531476+0000","last_became_peered":"2024-09-13T14:38:03.531476+0000","last_unstale":"2024-09-13T14:41:37.161759+0000","last_undegraded":"2024-09-13T14:41:37.161759+0000","last_fullsized":"2024-09-13T14:41:37.161759+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:41.557569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":37,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:41:26.122826+0000","last_change":"2024-09-13T14:38:53.954281+0000","last_active":"2024-09-13T14:41:26.122826+0000","last_peered":"2024-09-13T14:41:26.122826+0000","last_clean":"2024-09-13T14:41:26.122826+0000","last_became_active":"2024-09-13T14:38:53.900912+0000","last_became_peered":"2024-09-13T14:38:53.900912+0000","last_unstale":"2024-09-13T14:41:26.122826+0000","last_undegraded":"2024-09-13T14:41:26.122826+0000","last_fullsized":"2024-09-13T14:41:26.122826+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:47.876275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":88,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:18.116101+0000","last_change":"2024-09-13T14:38:54.000782+0000","last_active":"2024-09-13T14:41:18.116101+0000","last_peered":"2024-09-13T14:41:18.116101+0000","last_clean":"2024-09-13T14:41:18.116101+0000","last_became_active":"2024-09-13T14:38:54.000322+0000","last_became_peered":"2024-09-13T14:38:54.000322+0000","last_unstale":"2024-09-13T14:41:18.116101+0000","last_undegraded":"2024-09-13T14:41:18.116101+0000","last_fullsized":"2024-09-13T14:41:18.116101+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:09.057597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":395,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-13T14:41:37.148701+0000","last_change":"2024-09-13T14:38:53.899535+0000","last_active":"2024-09-13T14:41:37.148701+0000","last_peered":"2024-09-13T14:41:37.148701+0000","last_clean":"2024-09-13T14:41:37.148701+0000","last_became_active":"2024-09-13T14:38:53.899183+0000","last_became_peered":"2024-09-13T14:38:53.899183+0000","last_unstale":"2024-09-13T14:41:37.148701+0000","last_undegraded":"2024-09-13T14:41:37.148701+0000","last_fullsized":"2024-09-13T14:41:37.148701+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:33.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":69,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-13T14:41:37.164915+0000","last_change":"2024-09-13T14:38:03.543426+0000","last_active":"2024-09-13T14:41:37.164915+0000","last_peered":"2024-09-13T14:41:37.164915+0000","last_clean":"2024-09-13T14:41:37.164915+0000","last_became_active":"2024-09-13T14:38:03.542661+0000","last_became_peered":"2024-09-13T14:38:03.542661+0000","last_unstale":"2024-09-13T14:41:37.164915+0000","last_undegraded":"2024-09-13T14:41:37.164915+0000","last_fullsized":"2024-09-13T14:41:37.164915+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:54.050127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":424,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-13T14:41:37.148727+0000","last_change":"2024-09-13T14:37:48.131321+0000","last_active":"2024-09-13T14:41:37.148727+0000","last_peered":"2024-09-13T14:41:37.148727+0000","last_clean":"2024-09-13T14:41:37.148727+0000","last_became_active":"2024-09-13T14:37:48.130998+0000","last_became_peered":"2024-09-13T14:37:48.130998+0000","last_unstale":"2024-09-13T14:41:37.148727+0000","last_undegraded":"2024-09-13T14:41:37.148727+0000","last_fullsized":"2024-09-13T14:41:37.148727+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:33:26.597825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":100,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:18.109547+0000","last_change":"2024-09-13T14:38:03.535454+0000","last_active":"2024-09-13T14:41:18.109547+0000","last_peered":"2024-09-13T14:41:18.109547+0000","last_clean":"2024-09-13T14:41:18.109547+0000","last_became_active":"2024-09-13T14:38:03.535131+0000","last_became_peered":"2024-09-13T14:38:03.535131+0000","last_unstale":"2024-09-13T14:41:18.109547+0000","last_undegraded":"2024-09-13T14:41:18.109547+0000","last_fullsized":"2024-09-13T14:41:18.109547+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:24.477601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":448,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-13T14:41:37.149155+0000","last_change":"2024-09-13T14:38:03.524939+0000","last_active":"2024-09-13T14:41:37.149155+0000","last_peered":"2024-09-13T14:41:37.149155+0000","last_clean":"2024-09-13T14:41:37.149155+0000","last_became_active":"2024-09-13T14:38:03.524406+0000","last_became_peered":"2024-09-13T14:38:03.524406+0000","last_unstale":"2024-09-13T14:41:37.149155+0000","last_undegraded":"2024-09-13T14:41:37.149155+0000","last_fullsized":"2024-09-13T14:41:37.149155+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:19:45.257424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":76,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-13T14:41:37.161291+0000","last_change":"2024-09-13T14:38:53.892948+0000","last_active":"2024-09-13T14:41:37.161291+0000","last_peered":"2024-09-13T14:41:37.161291+0000","last_clean":"2024-09-13T14:41:37.161291+0000","last_became_active":"2024-09-13T14:38:53.892665+0000","last_became_peered":"2024-09-13T14:38:53.892665+0000","last_unstale":"2024-09-13T14:41:37.161291+0000","last_undegraded":"2024-09-13T14:41:37.161291+0000","last_fullsized":"2024-09-13T14:41:37.161291+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:08.292359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":397,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-13T14:41:37.149949+0000","last_change":"2024-09-13T14:37:48.139348+0000","last_active":"2024-09-13T14:41:37.149949+0000","last_peered":"2024-09-13T14:41:37.149949+0000","last_clean":"2024-09-13T14:41:37.149949+0000","last_became_active":"2024-09-13T14:37:48.139115+0000","last_became_peered":"2024-09-13T14:37:48.139115+0000","last_unstale":"2024-09-13T14:41:37.149949+0000","last_undegraded":"2024-09-13T14:41:37.149949+0000","last_fullsized":"2024-09-13T14:41:37.149949+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:32:20.413346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":437,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-13T14:41:37.149276+0000","last_change":"2024-09-13T14:37:48.172088+0000","last_active":"2024-09-13T14:41:37.149276+0000","last_peered":"2024-09-13T14:41:37.149276+0000","last_clean":"2024-09-13T14:41:37.149276+0000","last_became_active":"2024-09-13T14:37:48.171779+0000","last_became_peered":"2024-09-13T14:37:48.171779+0000","last_unstale":"2024-09-13T14:41:37.149276+0000","last_undegraded":"2024-09-13T14:41:37.149276+0000","last_fullsized":"2024-09-13T14:41:37.149276+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:02.852712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":329,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:41:27.779422+0000","last_change":"2024-09-13T14:38:53.890236+0000","last_active":"2024-09-13T14:41:27.779422+0000","last_peered":"2024-09-13T14:41:27.779422+0000","last_clean":"2024-09-13T14:41:27.779422+0000","last_became_active":"2024-09-13T14:38:53.889582+0000","last_became_peered":"2024-09-13T14:38:53.889582+0000","last_unstale":"2024-09-13T14:41:27.779422+0000","last_undegraded":"2024-09-13T14:41:27.779422+0000","last_fullsized":"2024-09-13T14:41:27.779422+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:14:29.334774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":423,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-13T14:41:37.150055+0000","last_change":"2024-09-13T14:38:03.534097+0000","last_active":"2024-09-13T14:41:37.150055+0000","last_peered":"2024-09-13T14:41:37.150055+0000","last_clean":"2024-09-13T14:41:37.150055+0000","last_became_active":"2024-09-13T14:38:03.533794+0000","last_became_peered":"2024-09-13T14:38:03.533794+0000","last_unstale":"2024-09-13T14:41:37.150055+0000","last_undegraded":"2024-09-13T14:41:37.150055+0000","last_fullsized":"2024-09-13T14:41:37.150055+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:23.272098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":69,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-13T14:41:37.161346+0000","last_change":"2024-09-13T14:38:03.527021+0000","last_active":"2024-09-13T14:41:37.161346+0000","last_peered":"2024-09-13T14:41:37.161346+0000","last_clean":"2024-09-13T14:41:37.161346+0000","last_became_active":"2024-09-13T14:38:03.526698+0000","last_became_peered":"2024-09-13T14:38:03.526698+0000","last_unstale":"2024-09-13T14:41:37.161346+0000","last_undegraded":"2024-09-13T14:41:37.161346+0000","last_fullsized":"2024-09-13T14:41:37.161346+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:01:35.734086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":273,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:41:27.779441+0000","last_change":"2024-09-13T14:27:12.834931+0000","last_active":"2024-09-13T14:41:27.779441+0000","last_peered":"2024-09-13T14:41:27.779441+0000","last_clean":"2024-09-13T14:41:27.779441+0000","last_became_active":"2024-09-13T14:27:12.834572+0000","last_became_peered":"2024-09-13T14:27:12.834572+0000","last_unstale":"2024-09-13T14:41:27.779441+0000","last_undegraded":"2024-09-13T14:41:27.779441+0000","last_fullsized":"2024-09-13T14:41:27.779441+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:54:21.155689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":85,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:18.109307+0000","last_change":"2024-09-13T14:37:49.176796+0000","last_active":"2024-09-13T14:41:18.109307+0000","last_peered":"2024-09-13T14:41:18.109307+0000","last_clean":"2024-09-13T14:41:18.109307+0000","last_became_active":"2024-09-13T14:37:49.176246+0000","last_became_peered":"2024-09-13T14:37:49.176246+0000","last_unstale":"2024-09-13T14:41:18.109307+0000","last_undegraded":"2024-09-13T14:41:18.109307+0000","last_fullsized":"2024-09-13T14:41:18.109307+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:29:24.936718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":284,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:41:27.779392+0000","last_change":"2024-09-13T14:38:03.526333+0000","last_active":"2024-09-13T14:41:27.779392+0000","last_peered":"2024-09-13T14:41:27.779392+0000","last_clean":"2024-09-13T14:41:27.779392+0000","last_became_active":"2024-09-13T14:38:03.526001+0000","last_became_peered":"2024-09-13T14:38:03.526001+0000","last_unstale":"2024-09-13T14:41:27.779392+0000","last_undegraded":"2024-09-13T14:41:27.779392+0000","last_fullsized":"2024-09-13T14:41:27.779392+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:25:25.181271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":85,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-13T14:41:18.109134+0000","last_change":"2024-09-13T14:37:49.149582+0000","last_active":"2024-09-13T14:41:18.109134+0000","last_peered":"2024-09-13T14:41:18.109134+0000","last_clean":"2024-09-13T14:41:18.109134+0000","last_became_active":"2024-09-13T14:37:49.148525+0000","last_became_peered":"2024-09-13T14:37:49.148525+0000","last_unstale":"2024-09-13T14:41:18.109134+0000","last_undegraded":"2024-09-13T14:41:18.109134+0000","last_fullsized":"2024-09-13T14:41:18.109134+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:48.655665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":469,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-13T14:41:37.150096+0000","last_change":"2024-09-13T14:38:03.535255+0000","last_active":"2024-09-13T14:41:37.150096+0000","last_peered":"2024-09-13T14:41:37.150096+0000","last_clean":"2024-09-13T14:41:37.150096+0000","last_became_active":"2024-09-13T14:38:03.533917+0000","last_became_peered":"2024-09-13T14:38:03.533917+0000","last_unstale":"2024-09-13T14:41:37.150096+0000","last_undegraded":"2024-09-13T14:41:37.150096+0000","last_fullsized":"2024-09-13T14:41:37.150096+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:33.559788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":433,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-13T14:41:37.149996+0000","last_change":"2024-09-13T14:37:48.146784+0000","last_active":"2024-09-13T14:41:37.149996+0000","last_peered":"2024-09-13T14:41:37.149996+0000","last_clean":"2024-09-13T14:41:37.149996+0000","last_became_active":"2024-09-13T14:37:48.146491+0000","last_became_peered":"2024-09-13T14:37:48.146491+0000","last_unstale":"2024-09-13T14:41:37.149996+0000","last_undegraded":"2024-09-13T14:41:37.149996+0000","last_fullsized":"2024-09-13T14:41:37.149996+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:30:54.772513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":264,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-13T14:41:26.135992+0000","last_change":"2024-09-13T14:37:48.127607+0000","last_active":"2024-09-13T14:41:26.135992+0000","last_peered":"2024-09-13T14:41:26.135992+0000","last_clean":"2024-09-13T14:41:26.135992+0000","last_became_active":"2024-09-13T14:37:48.127316+0000","last_became_peered":"2024-09-13T14:37:48.127316+0000","last_unstale":"2024-09-13T14:41:26.135992+0000","last_undegraded":"2024-09-13T14:41:26.135992+0000","last_fullsized":"2024-09-13T14:41:26.135992+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:53:37.678861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":75,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-13T14:41:37.164799+0000","last_change":"2024-09-13T14:38:03.563200+0000","last_active":"2024-09-13T14:41:37.164799+0000","last_peered":"2024-09-13T14:41:37.164799+0000","last_clean":"2024-09-13T14:41:37.164799+0000","last_became_active":"2024-09-13T14:38:03.534258+0000","last_became_peered":"2024-09-13T14:38:03.534258+0000","last_unstale":"2024-09-13T14:41:37.164799+0000","last_undegraded":"2024-09-13T14:41:37.164799+0000","last_fullsized":"2024-09-13T14:41:37.164799+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:49:07.814419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":4881645592,"num_objects":74489,"num_object_clones":0,"num_object_copies":223467,"num_objects_missing_on_primary":2575,"num_objects_missing":2575,"num_objects_degraded":28209,"num_objects_misplaced":3890,"num_objects_unfound":0,"num_objects_dirty":74489,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151137,"num_write_kb":7219969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62396,"num_bytes_recovered":4053401624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16007573504,"data_stored":16007561288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92560,"ondisk_log_size":92560,"up":48,"acting":43,"num_store_stats":6},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1538,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":17412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25132,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":7,"num_objects_missing":7,"num_objects_degraded":16,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16793600,"data_stored":16777292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":23,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234373,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2986020,"kb_used_data":2535708,"kb_used_omap":1,"kb_used_meta":450302,"kb_avail":90759132,"statfs":{"total":95995035648,"available":92937351168,"internally_reserved":0,"allocated":2596564992,"data_stored":2595087656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":461109707},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":239,"seq":1026497183792,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1820920,"kb_used_data":1606000,"kb_used_omap":8,"kb_used_meta":214903,"kb_avail":91924232,"statfs":{"total":95995035648,"available":94130413568,"internally_reserved":0,"allocated":1644544000,"data_stored":1643055712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8453,"internal_metadata":220061435},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757638,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2251800,"kb_used_data":1629200,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":91493352,"statfs":{"total":95995035648,"available":93689192448,"internally_reserved":0,"allocated":1668300800,"data_stored":1666831473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":637532614},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019282,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3212916,"kb_used_data":2614120,"kb_used_omap":1,"kb_used_meta":598782,"kb_avail":90532236,"statfs":{"total":95995035648,"available":92705009664,"internally_reserved":0,"allocated":2676858880,"data_stored":2675378201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":613153222},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435465,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4159324,"kb_used_data":3595924,"kb_used_omap":5,"kb_used_meta":563386,"kb_avail":89585828,"statfs":{"total":95995035648,"available":91735887872,"internally_reserved":0,"allocated":3682226176,"data_stored":3680753083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":576907390},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":446676598994,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":425201762520,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926044,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4370604,"kb_used_data":3703076,"kb_used_omap":7,"kb_used_meta":667512,"kb_avail":89374548,"statfs":{"total":95995035648,"available":91519537152,"internally_reserved":0,"allocated":3791949824,"data_stored":3790468817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":683532584},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6860,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3781955584,"data_stored":3781951512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3672510464,"data_stored":3672506392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2669281280,"data_stored":2669281280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1659830272,"data_stored":1659830272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1636175872,"data_stored":1636171800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2587820032,"data_stored":2587820032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:41:44.534 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph status --format=json 2024-09-13T14:41:44.550 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:44.599 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:44.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:44.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:44.702 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:44.727 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:44.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:44.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:44.829 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:44.852 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:44.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:44.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:44.954 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:44.984 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:44.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:44.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:44.993 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:45.086 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:45.112 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:45.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:45.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:45.214 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:45.242 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:45.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:45.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:45.344 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:45.367 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:45.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:45.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:45.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:45 smithi102 ceph-mon[110183]: pgmap v915: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.5 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 229 op/s; 28048/221091 objects degraded (12.686%); 3504/221091 objects misplaced (1.585%); 5.5 MiB/s, 88 objects/s recovering 2024-09-13T14:41:45.428 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:45 smithi102 ceph-mon[115269]: pgmap v915: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.5 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 229 op/s; 28048/221091 objects degraded (12.686%); 3504/221091 objects misplaced (1.585%); 5.5 MiB/s, 88 objects/s recovering 2024-09-13T14:41:45.459 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T14:41:45.462 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:45.469 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:45.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:45 smithi195 ceph-mon[104262]: pgmap v915: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.5 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 229 op/s; 28048/221091 objects degraded (12.686%); 3504/221091 objects misplaced (1.585%); 5.5 MiB/s, 88 objects/s recovering 2024-09-13T14:41:45.542 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:45.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:45.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:45.643 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:45.673 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:45.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:45.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:45.775 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:45.802 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:45.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:45.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:45.904 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:45.931 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:45.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:45.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:46.033 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:46.056 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:46.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:46.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:46.158 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:46.187 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:46.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:46.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:46.288 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:46.324 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:46.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:46.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:46.426 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:46.455 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:46.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:46.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:46.557 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:46.582 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:46.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:46.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:46.598 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:41:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:41:46] "GET /metrics HTTP/1.1" 200 36539 "" "Prometheus/2.51.0" 2024-09-13T14:41:46.683 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:46.707 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:46.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:46.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:46.808 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:46.837 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:46.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:46.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:46.925 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:41:46.939 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:46.968 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:46.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:46.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:47.070 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:47.096 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:47.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:47.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:47.198 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:47.224 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:47.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:47.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:47.225 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:47 smithi102 ceph-mon[115269]: pgmap v916: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.5 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 236 op/s; 27966/219708 objects degraded (12.729%); 3504/219708 objects misplaced (1.595%); 5.6 MiB/s, 90 objects/s recovering 2024-09-13T14:41:47.226 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:47 smithi102 ceph-mon[110183]: pgmap v916: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.5 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 236 op/s; 27966/219708 objects degraded (12.729%); 3504/219708 objects misplaced (1.595%); 5.6 MiB/s, 90 objects/s recovering 2024-09-13T14:41:47.325 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:47.351 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:47.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:47.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:47.453 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:47.480 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:47.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:47.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:47.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:47 smithi195 ceph-mon[104262]: pgmap v916: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.5 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 236 op/s; 27966/219708 objects degraded (12.729%); 3504/219708 objects misplaced (1.595%); 5.6 MiB/s, 90 objects/s recovering 2024-09-13T14:41:47.582 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:47.606 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:47.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:47.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:47.708 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:47.733 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:47.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:47.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:47.827 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:47.834 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:47.865 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:47.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:47.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:47.968 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:47.992 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:47.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:47.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:48.094 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:48.123 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:48.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:48.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:48.133 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:48.225 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:48.256 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:48.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:48.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:48.358 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:48.391 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:48.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:48.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:48.493 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:48.520 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:48.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:48.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:48.622 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:48.646 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:48.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:48.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:48.748 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:48.775 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:48.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:48.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:48.877 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:48.900 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:48.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:48.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:49.002 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:49.026 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:49.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:49.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:49.129 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:49.153 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:49.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:49.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:49.182 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is set 2024-09-13T14:41:49.255 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:49.287 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:49.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:49.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:49.389 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:49.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:49.412 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:49.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:49.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:49 smithi102 ceph-mon[110183]: pgmap v917: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.4 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 302 op/s; 27803/217389 objects degraded (12.790%); 2789/217389 objects misplaced (1.283%); 10 MiB/s, 164 objects/s recovering 2024-09-13T14:41:49.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:49 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3022770839' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:41:49.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:49 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 27803/217389 objects degraded (12.790%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:41:49.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:49 smithi102 ceph-mon[115269]: pgmap v917: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.4 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 302 op/s; 27803/217389 objects degraded (12.790%); 2789/217389 objects misplaced (1.283%); 10 MiB/s, 164 objects/s recovering 2024-09-13T14:41:49.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:49 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3022770839' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:41:49.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:49 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 27803/217389 objects degraded (12.790%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:41:49.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:49 smithi195 ceph-mon[104262]: pgmap v917: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.4 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 302 op/s; 27803/217389 objects degraded (12.790%); 2789/217389 objects misplaced (1.283%); 10 MiB/s, 164 objects/s recovering 2024-09-13T14:41:49.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:49 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3022770839' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:41:49.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:49 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 27803/217389 objects degraded (12.790%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:41:49.514 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:49.536 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:49.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:49.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:49.638 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:49.665 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:49.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:49.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:49.767 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:49.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:49.790 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:49.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:49.891 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:49.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:49.917 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:49.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:50.018 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:50.041 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:50.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:50.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:50.143 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:50.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:50.168 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:50.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:50.269 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:50.291 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:50.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:50.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:50.381 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:50 smithi102 ceph-mon[110183]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:41:50.381 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:50 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3022770839' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:41:50.381 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:50 smithi102 ceph-mon[110183]: osdmap e274: 8 total, 8 up, 6 in 2024-09-13T14:41:50.381 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:50 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:41:50.381 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:50 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:41:50.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:50 smithi102 ceph-mon[115269]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:41:50.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:50 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3022770839' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:41:50.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:50 smithi102 ceph-mon[115269]: osdmap e274: 8 total, 8 up, 6 in 2024-09-13T14:41:50.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:50 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:41:50.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:50 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:41:50.392 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:50.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:50.415 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:50.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:50.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:50 smithi195 ceph-mon[104262]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:41:50.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:50 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3022770839' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:41:50.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:50 smithi195 ceph-mon[104262]: osdmap e274: 8 total, 8 up, 6 in 2024-09-13T14:41:50.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:50 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:41:50.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:50 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:41:50.517 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:50.540 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:50.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:50.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:50.642 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:50.674 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:50.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:50.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:50.776 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:50.802 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:50.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:50.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:50.904 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:50.927 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:50.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:50.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:51.029 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:51.054 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:51.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:51.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:51.156 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:51.182 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:51.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:51.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:51.283 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:51.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:51.313 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:51.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:51.414 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:51.445 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:51.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:51.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:51.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:51 smithi195 ceph-mon[104262]: pgmap v919: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.4 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 214 op/s; 27803/217122 objects degraded (12.805%); 2789/217122 objects misplaced (1.285%); 8.4 MiB/s, 135 objects/s recovering 2024-09-13T14:41:51.546 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:51.572 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:51.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:51.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:51.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:51 smithi102 ceph-mon[110183]: pgmap v919: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.4 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 214 op/s; 27803/217122 objects degraded (12.805%); 2789/217122 objects misplaced (1.285%); 8.4 MiB/s, 135 objects/s recovering 2024-09-13T14:41:51.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:51 smithi102 ceph-mon[115269]: pgmap v919: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.4 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 214 op/s; 27803/217122 objects degraded (12.805%); 2789/217122 objects misplaced (1.285%); 8.4 MiB/s, 135 objects/s recovering 2024-09-13T14:41:51.674 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:51.699 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:51.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:51.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:51.801 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:51.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:51.829 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:51.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:51.931 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:51.958 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:51.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:51.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:52.060 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:52.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:52.093 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:52.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:52.195 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:52.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:52.222 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:52.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:52.323 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:52.347 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:52.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:52.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:52.450 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:52.475 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:52.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:52.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:52.577 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:52.601 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:52.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:52.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:52.703 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:52.726 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:52.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:52.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:52.828 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:52.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:52.855 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:52.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:52.957 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:52.978 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:52.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:52.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:53.080 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:53.103 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:53.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:53.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:53.205 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:53.233 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:53.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:53.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:53.339 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:53.366 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:53.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:53.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:53.408 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:53 smithi102 ceph-mon[110183]: pgmap v920: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.4 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 263 op/s; 27722/215640 objects degraded (12.856%); 2789/215640 objects misplaced (1.293%); 8.9 MiB/s, 143 objects/s recovering 2024-09-13T14:41:53.408 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:53 smithi102 ceph-mon[110183]: osdmap e275: 8 total, 8 up, 6 in 2024-09-13T14:41:53.408 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:53 smithi102 ceph-mon[115269]: pgmap v920: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.4 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 263 op/s; 27722/215640 objects degraded (12.856%); 2789/215640 objects misplaced (1.293%); 8.9 MiB/s, 143 objects/s recovering 2024-09-13T14:41:53.408 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:53 smithi102 ceph-mon[115269]: osdmap e275: 8 total, 8 up, 6 in 2024-09-13T14:41:53.468 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:53.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:53.495 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:53.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:53.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:53 smithi195 ceph-mon[104262]: pgmap v920: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.4 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 263 op/s; 27722/215640 objects degraded (12.856%); 2789/215640 objects misplaced (1.293%); 8.9 MiB/s, 143 objects/s recovering 2024-09-13T14:41:53.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:53 smithi195 ceph-mon[104262]: osdmap e275: 8 total, 8 up, 6 in 2024-09-13T14:41:53.597 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:53.625 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:53.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:53.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:53.727 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:53.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:53.755 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:53.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:53.856 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:53.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:53.884 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:53.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:53.887 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:41:53.888 INFO:teuthology.orchestra.run.smithi102.stdout:{"fsid":"08cee502-71d9-11ef-bceb-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27722/215640 objects degraded (12.856%), 10 pgs degraded, 6 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1337,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":276,"num_osds":8,"num_up_osds":8,"osd_up_since":1726238267,"num_in_osds":6,"osd_in_since":1726238173,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":78},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovery_wait+degraded","count":3},{"state_name":"active+recovery_wait+degraded+remapped","count":2},{"state_name":"active+undersized+degraded+remapped+backfill_wait","count":1},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":89,"num_pools":5,"num_objects":71880,"data_bytes":4713617405,"bytes_used":18846416896,"bytes_avail":557123796992,"bytes_total":575970213888,"degraded_objects":27722,"degraded_total":215640,"degraded_ratio":0.12855685401595252,"misplaced_objects":2789,"misplaced_total":215640,"misplaced_ratio":0.012933593025412724,"recovering_objects_per_sec":143,"recovering_bytes_per_sec":9382070,"recovering_keys_per_sec":0,"num_objects_recovered":1432,"num_bytes_recovered":93847552,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":263},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi102.njetxr","status":"up:active","gid":24521}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.102:8443/","prometheus":"http://172.21.15.102:9283/"}},"servicemap":{"epoch":28,"modified":"2024-09-13T14:40:21.476600+0000","services":{}},"progress_events":{}} 2024-09-13T14:41:53.986 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:54.008 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:54.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:54.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:54.110 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:54.133 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:54.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:54.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:54.222 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:41:54.222 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:41:54.222 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:41:54.223 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:41:54.223 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:41:54.223 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:41:54.223 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:41:54.223 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:41:54.223 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:41:54.223 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:41:54.223 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:41:54.223 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:41:54.223 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:41:54.224 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:41:54.224 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:41:54.224 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:41:54.235 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:54.261 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:54.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:54.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:54.363 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:54.387 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:54.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:54.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:54.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:54 smithi102 ceph-mon[115269]: osdmap e276: 8 total, 8 up, 6 in 2024-09-13T14:41:54.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:54 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2947435727' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:41:54.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:54 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/433192149' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:41:54.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:54 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 27722/215640 objects degraded (12.856%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:41:54.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:54 smithi102 ceph-mon[110183]: osdmap e276: 8 total, 8 up, 6 in 2024-09-13T14:41:54.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:54 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2947435727' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:41:54.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:54 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/433192149' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:41:54.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:54 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 27722/215640 objects degraded (12.856%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:41:54.488 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:54.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:54 smithi195 ceph-mon[104262]: osdmap e276: 8 total, 8 up, 6 in 2024-09-13T14:41:54.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:54 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2947435727' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:41:54.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:54 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/433192149' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:41:54.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:54 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 27722/215640 objects degraded (12.856%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:41:54.512 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:54.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:54.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:54.614 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:54.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:54.640 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:54.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:54.742 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:54.767 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:54.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:54.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:54.869 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:54.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:54.894 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:54.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:54.995 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:55.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:55.026 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:55.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:55.127 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:55.154 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:55.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:55.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:55.226 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:41:55.226 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:41:55.257 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:55.278 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:55.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:55.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:55.380 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:55.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:55.404 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:55.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:55.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:55 smithi102 ceph-mon[110183]: pgmap v923: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 peering, 1 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.3 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 234 op/s; 27444/213162 objects degraded (12.875%); 1988/213162 objects misplaced (0.933%); 11 MiB/s, 191 objects/s recovering 2024-09-13T14:41:55.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:55 smithi102 ceph-mon[110183]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-13T14:41:55.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:55 smithi102 ceph-mon[115269]: pgmap v923: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 peering, 1 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.3 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 234 op/s; 27444/213162 objects degraded (12.875%); 1988/213162 objects misplaced (0.933%); 11 MiB/s, 191 objects/s recovering 2024-09-13T14:41:55.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:55 smithi102 ceph-mon[115269]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-13T14:41:55.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:55 smithi195 ceph-mon[104262]: pgmap v923: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 peering, 1 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.3 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 234 op/s; 27444/213162 objects degraded (12.875%); 1988/213162 objects misplaced (0.933%); 11 MiB/s, 191 objects/s recovering 2024-09-13T14:41:55.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:55 smithi195 ceph-mon[104262]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-13T14:41:55.506 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:55.532 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:55.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:55.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:55.634 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:55.660 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:55.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:55.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:55.762 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:55.785 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:55.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:55.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:55.887 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:55.917 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:55.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:55.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:56.019 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:56.042 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:56.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:56.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:56.144 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:56.145 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:41:56.171 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:56.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:56.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:56.273 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:56.300 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:56.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:56.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:56.402 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:56.428 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:56.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:56.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:56.530 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:56.557 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:56.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:56.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:56.637 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-13T14:41:56.637 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:41:56.659 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:56.719 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:56.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:56.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:56.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:41:56.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:41:56.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:56 smithi195 ceph-mon[104262]: pgmap v924: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 2 peering, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.3 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 265 op/s; 27355/212085 objects degraded (12.898%); 1987/212085 objects misplaced (0.937%); 11 MiB/s, 195 objects/s recovering 2024-09-13T14:41:56.821 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:41:56.843 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:56.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:56.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:41:56.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:41:56.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:41:56.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:56 smithi102 ceph-mon[110183]: pgmap v924: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 2 peering, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.3 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 265 op/s; 27355/212085 objects degraded (12.898%); 1987/212085 objects misplaced (0.937%); 11 MiB/s, 195 objects/s recovering 2024-09-13T14:41:56.921 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:41:56 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:41:56] "GET /metrics HTTP/1.1" 200 36541 "" "Prometheus/2.51.0" 2024-09-13T14:41:56.921 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:41:56.921 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:41:56.921 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:56 smithi102 ceph-mon[115269]: pgmap v924: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 2 peering, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 78 active+clean; 4.3 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 265 op/s; 27355/212085 objects degraded (12.898%); 1987/212085 objects misplaced (0.937%); 11 MiB/s, 195 objects/s recovering 2024-09-13T14:41:56.944 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:56.973 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:56.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:56.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:57.030 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:41:57.076 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:57.100 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:57.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:57.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:57.202 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:57.225 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:57.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:57.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:57.328 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:57.356 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:57.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:57.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:57.458 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:57.482 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:57.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:57.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:57.584 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:57.609 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:57.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:57.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:57.711 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:57.739 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:57.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:57.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:57.841 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:41:57.864 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:57.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:57.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:41:57.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:57 smithi102 ceph-mon[110183]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-13T14:41:57.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:57 smithi102 ceph-mon[115269]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-13T14:41:57.966 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:57.989 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:57.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:57.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:57.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:57 smithi195 ceph-mon[104262]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-13T14:41:58.092 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:58.092 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:58.114 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:58.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:58.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:58.216 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:58.242 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:58.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:58.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:58.344 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:58.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:58.368 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:58.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:58.470 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:58.498 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:58.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:58.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:58.600 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:58.627 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:58.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:58.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:58.729 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:58.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:58.757 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:58.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:58.858 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:58.882 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:58.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:58.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:58.898 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:58 smithi195 ceph-mon[104262]: pgmap v925: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 80 active+clean; 4.3 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 260 op/s; 22973/210867 objects degraded (10.895%); 3982/210867 objects misplaced (1.888%); 26 MiB/s, 431 objects/s recovering 2024-09-13T14:41:58.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:58 smithi102 ceph-mon[110183]: pgmap v925: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 80 active+clean; 4.3 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 260 op/s; 22973/210867 objects degraded (10.895%); 3982/210867 objects misplaced (1.888%); 26 MiB/s, 431 objects/s recovering 2024-09-13T14:41:58.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:58 smithi102 ceph-mon[115269]: pgmap v925: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 80 active+clean; 4.3 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 260 op/s; 22973/210867 objects degraded (10.895%); 3982/210867 objects misplaced (1.888%); 26 MiB/s, 431 objects/s recovering 2024-09-13T14:41:58.985 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:41:59.007 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:59.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:59.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:41:59.109 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:59.139 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:59.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:59.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:59.241 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:41:59.264 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:59.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:59.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:41:59.366 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:59.395 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:59.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:59.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:59.497 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:41:59.509 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T14:41:59.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:59.529 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:59.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:41:59.548 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:59.620 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:59.630 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:41:59.655 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:41:59.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:59.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:41:59.738 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:41:59 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 22973/210867 objects degraded (10.895%), 9 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:41:59.738 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:41:59 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 22973/210867 objects degraded (10.895%), 9 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:41:59.757 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:41:59.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:59.790 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:59.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:41:59.891 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:41:59.920 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:41:59.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:41:59.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:41:59.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:41:59 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 22973/210867 objects degraded (10.895%), 9 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:42:00.022 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:00.048 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:00.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:00.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:00.149 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:00.173 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:00.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:00.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:00.275 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:00.303 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:00.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:00.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:00.405 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:00.428 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:00.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:00.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:00.530 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:00.555 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:00.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:00.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:00.657 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:00.682 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:00.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:00.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:00.784 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:00.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:00.808 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:00.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:00.909 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:00.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:00 smithi102 ceph-mon[115269]: pgmap v926: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 80 active+clean; 4.3 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 204 op/s; 22973/210738 objects degraded (10.901%); 3982/210738 objects misplaced (1.890%); 26 MiB/s, 421 objects/s recovering 2024-09-13T14:42:00.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:00 smithi102 ceph-mon[110183]: pgmap v926: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 80 active+clean; 4.3 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 204 op/s; 22973/210738 objects degraded (10.901%); 3982/210738 objects misplaced (1.890%); 26 MiB/s, 421 objects/s recovering 2024-09-13T14:42:00.936 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:00.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:00.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:00.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:00 smithi195 ceph-mon[104262]: pgmap v926: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 80 active+clean; 4.3 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 204 op/s; 22973/210738 objects degraded (10.901%); 3982/210738 objects misplaced (1.890%); 26 MiB/s, 421 objects/s recovering 2024-09-13T14:42:01.037 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:01.062 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:01.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:01.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:01.164 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:01.189 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:01.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:01.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:01.290 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:01.317 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:01.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:01.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:01.419 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:01.443 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:01.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:01.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:01.545 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:01.567 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:01.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:01.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:01.669 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:01.696 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:01.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:01.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:01.799 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:01.822 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:01.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:01.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:01.924 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:01.946 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:01.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:01.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:02.048 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:02.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:02.072 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:02.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:02.173 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:02.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:02.203 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:02.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:02.304 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:02.325 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:02.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:02.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:02.427 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:02.452 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:02.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:02.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:02.554 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:02.581 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:02.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:02.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:02.683 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:02.705 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:02.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:02.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:02.807 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:02.834 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:02.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:02.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:02.878 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:02 smithi102 ceph-mon[110183]: pgmap v927: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 80 active+clean; 4.3 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 205 op/s; 22830/209910 objects degraded (10.876%); 3982/209910 objects misplaced (1.897%); 23 MiB/s, 377 objects/s recovering 2024-09-13T14:42:02.878 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:02 smithi102 ceph-mon[115269]: pgmap v927: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 80 active+clean; 4.3 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 205 op/s; 22830/209910 objects degraded (10.876%); 3982/209910 objects misplaced (1.897%); 23 MiB/s, 377 objects/s recovering 2024-09-13T14:42:02.936 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:02.962 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:02.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:02.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:02.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:02 smithi195 ceph-mon[104262]: pgmap v927: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 80 active+clean; 4.3 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 205 op/s; 22830/209910 objects degraded (10.876%); 3982/209910 objects misplaced (1.897%); 23 MiB/s, 377 objects/s recovering 2024-09-13T14:42:03.064 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:03.091 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:03.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:03.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:03.192 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:03.214 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:03.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:03.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:03.316 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:03.342 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:03.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:03.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:03.443 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:03.475 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:03.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:03.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:03.577 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:03.599 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:03.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:03.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:03.701 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:03.725 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:03.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:03.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:03.827 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:03.849 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:03.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:03.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:03.950 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:03.971 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:03.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:03.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:04.071 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:04.073 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:04.102 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:04.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:04.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:04.204 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:04.218 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:42:04.219 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:42:04.225 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:04.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:04.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:04.326 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:04.347 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:04.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:04.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:04.449 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:04.471 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:04.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:04.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:04.572 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:04.595 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:04.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:04.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:04.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:04 smithi102 ceph-mon[110183]: pgmap v928: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 80 active+clean; 4.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 147 op/s; 20749/208602 objects degraded (9.947%); 3809/208602 objects misplaced (1.826%); 28 MiB/s, 454 objects/s recovering 2024-09-13T14:42:04.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:04 smithi102 ceph-mon[110183]: osdmap e277: 8 total, 8 up, 6 in 2024-09-13T14:42:04.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:04 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 20749/208602 objects degraded (9.947%), 9 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:42:04.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:04 smithi102 ceph-mon[110183]: from='client.35856 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:42:04.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:04 smithi102 ceph-mon[115269]: pgmap v928: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 80 active+clean; 4.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 147 op/s; 20749/208602 objects degraded (9.947%); 3809/208602 objects misplaced (1.826%); 28 MiB/s, 454 objects/s recovering 2024-09-13T14:42:04.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:04 smithi102 ceph-mon[115269]: osdmap e277: 8 total, 8 up, 6 in 2024-09-13T14:42:04.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:04 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 20749/208602 objects degraded (9.947%), 9 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:42:04.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:04 smithi102 ceph-mon[115269]: from='client.35856 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:42:04.696 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:04.723 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:04.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:04.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:04.824 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:04.846 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:04.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:04.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:04.948 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:04.974 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:04.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:04.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:04.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:04 smithi195 ceph-mon[104262]: pgmap v928: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 80 active+clean; 4.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 147 op/s; 20749/208602 objects degraded (9.947%); 3809/208602 objects misplaced (1.826%); 28 MiB/s, 454 objects/s recovering 2024-09-13T14:42:04.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:04 smithi195 ceph-mon[104262]: osdmap e277: 8 total, 8 up, 6 in 2024-09-13T14:42:04.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:04 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 20749/208602 objects degraded (9.947%), 9 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:42:04.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:04 smithi195 ceph-mon[104262]: from='client.35856 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:42:05.076 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:05.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:05.101 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:05.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:05.202 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:05.209 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":929,"stamp":"2024-09-13T14:42:03.639592+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4559869949,"num_objects":69534,"num_object_clones":0,"num_object_copies":208602,"num_objects_missing_on_primary":2008,"num_objects_missing":2008,"num_objects_degraded":20749,"num_objects_misplaced":3809,"num_objects_unfound":0,"num_objects_dirty":69534,"num_whiteouts":0,"num_read":6377,"num_read_kb":13834,"num_write":157243,"num_write_kb":7235010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69690,"num_bytes_recovered":4525198477,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94225,"ondisk_log_size":94225,"up":267,"acting":262,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":302,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":18111976,"kb_used_data":14998648,"kb_used_omap":26,"kb_used_meta":3113253,"kb_avail":544358936,"statfs":{"total":575970213888,"available":557423550464,"internally_reserved":0,"allocated":15358615552,"data_stored":15349664669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27144,"internal_metadata":3187971576},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-99614720,"num_objects":-1520,"num_object_clones":0,"num_object_copies":-4560,"num_objects_missing_on_primary":-329,"num_objects_missing":-329,"num_objects_degraded":-6695,"num_objects_misplaced":1821,"num_objects_unfound":0,"num_objects_dirty":-1520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1520,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4686,"num_bytes_recovered":307101696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.135816"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":445,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T14:41:53.200871+0000","last_change":"2024-09-13T14:36:13.859927+0000","last_active":"2024-09-13T14:41:53.200871+0000","last_peered":"2024-09-13T14:41:53.200871+0000","last_clean":"2024-09-13T14:41:53.200871+0000","last_became_active":"2024-09-13T14:36:13.859505+0000","last_became_peered":"2024-09-13T14:36:13.859505+0000","last_unstale":"2024-09-13T14:41:53.200871+0000","last_undegraded":"2024-09-13T14:41:53.200871+0000","last_fullsized":"2024-09-13T14:41:53.200871+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:12:20.989627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":278,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.417902+0000","last_change":"2024-09-13T14:37:48.130968+0000","last_active":"2024-09-13T14:41:52.417902+0000","last_peered":"2024-09-13T14:41:52.417902+0000","last_clean":"2024-09-13T14:41:52.417902+0000","last_became_active":"2024-09-13T14:37:48.130603+0000","last_became_peered":"2024-09-13T14:37:48.130603+0000","last_unstale":"2024-09-13T14:41:52.417902+0000","last_undegraded":"2024-09-13T14:41:52.417902+0000","last_fullsized":"2024-09-13T14:41:52.417902+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:42.626386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":74,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.211096+0000","last_change":"2024-09-13T14:36:13.862181+0000","last_active":"2024-09-13T14:41:52.211096+0000","last_peered":"2024-09-13T14:41:52.211096+0000","last_clean":"2024-09-13T14:41:52.211096+0000","last_became_active":"2024-09-13T14:36:13.861873+0000","last_became_peered":"2024-09-13T14:36:13.861873+0000","last_unstale":"2024-09-13T14:41:52.211096+0000","last_undegraded":"2024-09-13T14:41:52.211096+0000","last_fullsized":"2024-09-13T14:41:52.211096+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:14.085700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":82,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T14:41:53.195349+0000","last_change":"2024-09-13T14:38:53.896672+0000","last_active":"2024-09-13T14:41:53.195349+0000","last_peered":"2024-09-13T14:41:53.195349+0000","last_clean":"2024-09-13T14:41:53.195349+0000","last_became_active":"2024-09-13T14:38:53.896346+0000","last_became_peered":"2024-09-13T14:38:53.896346+0000","last_unstale":"2024-09-13T14:41:53.195349+0000","last_undegraded":"2024-09-13T14:41:53.195349+0000","last_fullsized":"2024-09-13T14:41:53.195349+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:57.789686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":82,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T14:41:53.195743+0000","last_change":"2024-09-13T14:38:53.893021+0000","last_active":"2024-09-13T14:41:53.195743+0000","last_peered":"2024-09-13T14:41:53.195743+0000","last_clean":"2024-09-13T14:41:53.195743+0000","last_became_active":"2024-09-13T14:38:53.892668+0000","last_became_peered":"2024-09-13T14:38:53.892668+0000","last_unstale":"2024-09-13T14:41:53.195743+0000","last_undegraded":"2024-09-13T14:41:53.195743+0000","last_fullsized":"2024-09-13T14:41:53.195743+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:57.180738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":246,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.192465+0000","last_change":"2024-09-13T14:37:48.134642+0000","last_active":"2024-09-13T14:41:52.192465+0000","last_peered":"2024-09-13T14:41:52.192465+0000","last_clean":"2024-09-13T14:41:52.192465+0000","last_became_active":"2024-09-13T14:37:48.134402+0000","last_became_peered":"2024-09-13T14:37:48.134402+0000","last_unstale":"2024-09-13T14:41:52.192465+0000","last_undegraded":"2024-09-13T14:41:52.192465+0000","last_fullsized":"2024-09-13T14:41:52.192465+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:51.313663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":464,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T14:41:53.201683+0000","last_change":"2024-09-13T14:38:53.903022+0000","last_active":"2024-09-13T14:41:53.201683+0000","last_peered":"2024-09-13T14:41:53.201683+0000","last_clean":"2024-09-13T14:41:53.201683+0000","last_became_active":"2024-09-13T14:38:53.902448+0000","last_became_peered":"2024-09-13T14:38:53.902448+0000","last_unstale":"2024-09-13T14:41:53.201683+0000","last_undegraded":"2024-09-13T14:41:53.201683+0000","last_fullsized":"2024-09-13T14:41:53.201683+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:24.690401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":85,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T14:41:53.194651+0000","last_change":"2024-09-13T14:38:03.554500+0000","last_active":"2024-09-13T14:41:53.194651+0000","last_peered":"2024-09-13T14:41:53.194651+0000","last_clean":"2024-09-13T14:41:53.194651+0000","last_became_active":"2024-09-13T14:38:03.532910+0000","last_became_peered":"2024-09-13T14:38:03.532910+0000","last_unstale":"2024-09-13T14:41:53.194651+0000","last_undegraded":"2024-09-13T14:41:53.194651+0000","last_fullsized":"2024-09-13T14:41:53.194651+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:27:50.843498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"276'9667","reported_seq":20554,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T14:42:01.756291+0000","last_change":"2024-09-13T14:41:53.220083+0000","last_active":"2024-09-13T14:42:01.756291+0000","last_peered":"2024-09-13T14:42:01.756291+0000","last_clean":"2024-09-13T14:42:01.756291+0000","last_became_active":"2024-09-13T14:41:53.200084+0000","last_became_peered":"2024-09-13T14:41:53.200084+0000","last_unstale":"2024-09-13T14:42:01.756291+0000","last_undegraded":"2024-09-13T14:42:01.756291+0000","last_fullsized":"2024-09-13T14:42:01.756291+0000","mapping_epoch":275,"log_start":"212'4000","ondisk_log_start":"212'4000","created":205,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5667,"log_dups_size":0,"ondisk_log_size":5667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:08.897524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276103168,"num_objects":4213,"num_object_clones":0,"num_object_copies":12639,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9667,"num_write_kb":444160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9152,"num_bytes_recovered":594608128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":38,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.193031+0000","last_change":"2024-09-13T14:38:53.940238+0000","last_active":"2024-09-13T14:41:52.193031+0000","last_peered":"2024-09-13T14:41:52.193031+0000","last_clean":"2024-09-13T14:41:52.193031+0000","last_became_active":"2024-09-13T14:38:53.892825+0000","last_became_peered":"2024-09-13T14:38:53.892825+0000","last_unstale":"2024-09-13T14:41:52.193031+0000","last_undegraded":"2024-09-13T14:41:52.193031+0000","last_fullsized":"2024-09-13T14:41:52.193031+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:46.666581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":94,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.194785+0000","last_change":"2024-09-13T14:38:53.898907+0000","last_active":"2024-09-13T14:41:52.194785+0000","last_peered":"2024-09-13T14:41:52.194785+0000","last_clean":"2024-09-13T14:41:52.194785+0000","last_became_active":"2024-09-13T14:38:53.898572+0000","last_became_peered":"2024-09-13T14:38:53.898572+0000","last_unstale":"2024-09-13T14:41:52.194785+0000","last_undegraded":"2024-09-13T14:41:52.194785+0000","last_fullsized":"2024-09-13T14:41:52.194785+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:29.222150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"276'9887","reported_seq":24049,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T14:42:02.013630+0000","last_change":"2024-09-13T14:38:57.930032+0000","last_active":"2024-09-13T14:42:02.013630+0000","last_peered":"2024-09-13T14:42:02.013630+0000","last_clean":"2024-09-13T14:42:02.013630+0000","last_became_active":"2024-09-13T14:38:57.921458+0000","last_became_peered":"2024-09-13T14:38:57.921458+0000","last_unstale":"2024-09-13T14:42:02.013630+0000","last_undegraded":"2024-09-13T14:42:02.013630+0000","last_fullsized":"2024-09-13T14:42:02.013630+0000","mapping_epoch":256,"log_start":"213'4439","ondisk_log_start":"213'4439","created":205,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5448,"log_dups_size":0,"ondisk_log_size":5448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:27:12.294038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":285409280,"num_objects":4355,"num_object_clones":0,"num_object_copies":13065,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9887,"num_write_kb":455744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6990,"num_bytes_recovered":458096640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":38,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.192628+0000","last_change":"2024-09-13T14:38:53.904885+0000","last_active":"2024-09-13T14:41:52.192628+0000","last_peered":"2024-09-13T14:41:52.192628+0000","last_clean":"2024-09-13T14:41:52.192628+0000","last_became_active":"2024-09-13T14:38:53.891726+0000","last_became_peered":"2024-09-13T14:38:53.891726+0000","last_unstale":"2024-09-13T14:41:52.192628+0000","last_undegraded":"2024-09-13T14:41:52.192628+0000","last_fullsized":"2024-09-13T14:41:52.192628+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:57.079205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":284,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.787082+0000","last_change":"2024-09-13T14:37:48.160504+0000","last_active":"2024-09-13T14:41:52.787082+0000","last_peered":"2024-09-13T14:41:52.787082+0000","last_clean":"2024-09-13T14:41:52.787082+0000","last_became_active":"2024-09-13T14:37:48.160181+0000","last_became_peered":"2024-09-13T14:37:48.160181+0000","last_unstale":"2024-09-13T14:41:52.787082+0000","last_undegraded":"2024-09-13T14:41:52.787082+0000","last_fullsized":"2024-09-13T14:41:52.787082+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:37.087008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"276'9816","reported_seq":11599,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T14:41:59.678805+0000","last_change":"2024-09-13T14:38:04.531914+0000","last_active":"2024-09-13T14:41:59.678805+0000","last_peered":"2024-09-13T14:41:59.678805+0000","last_clean":"2024-09-13T14:41:59.678805+0000","last_became_active":"2024-09-13T14:38:04.523448+0000","last_became_peered":"2024-09-13T14:38:04.523448+0000","last_unstale":"2024-09-13T14:41:59.678805+0000","last_undegraded":"2024-09-13T14:41:59.678805+0000","last_fullsized":"2024-09-13T14:41:59.678805+0000","mapping_epoch":245,"log_start":"212'3900","ondisk_log_start":"212'3900","created":205,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5916,"log_dups_size":0,"ondisk_log_size":5916,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:06.536896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":285605888,"num_objects":4358,"num_object_clones":0,"num_object_copies":13074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9816,"num_write_kb":453568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":811,"num_bytes_recovered":53149696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":275,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.192751+0000","last_change":"2024-09-13T14:37:49.611774+0000","last_active":"2024-09-13T14:41:52.192751+0000","last_peered":"2024-09-13T14:41:52.192751+0000","last_clean":"2024-09-13T14:41:52.192751+0000","last_became_active":"2024-09-13T14:37:49.611088+0000","last_became_peered":"2024-09-13T14:37:49.611088+0000","last_unstale":"2024-09-13T14:41:52.192751+0000","last_undegraded":"2024-09-13T14:41:52.192751+0000","last_fullsized":"2024-09-13T14:41:52.192751+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:08.933213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":34,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.192387+0000","last_change":"2024-09-13T14:38:53.899211+0000","last_active":"2024-09-13T14:41:52.192387+0000","last_peered":"2024-09-13T14:41:52.192387+0000","last_clean":"2024-09-13T14:41:52.192387+0000","last_became_active":"2024-09-13T14:38:53.898927+0000","last_became_peered":"2024-09-13T14:38:53.898927+0000","last_unstale":"2024-09-13T14:41:52.192387+0000","last_undegraded":"2024-09-13T14:41:52.192387+0000","last_fullsized":"2024-09-13T14:41:52.192387+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:00:28.857611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"276'9679","reported_seq":9762,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T14:42:01.743211+0000","last_change":"2024-09-13T14:35:24.004651+0000","last_active":"2024-09-13T14:42:01.743211+0000","last_peered":"2024-09-13T14:42:01.743211+0000","last_clean":"2024-09-13T14:42:01.743211+0000","last_became_active":"2024-09-13T14:35:24.004358+0000","last_became_peered":"2024-09-13T14:35:24.004358+0000","last_unstale":"2024-09-13T14:42:01.743211+0000","last_undegraded":"2024-09-13T14:42:01.743211+0000","last_fullsized":"2024-09-13T14:42:01.743211+0000","mapping_epoch":205,"log_start":"212'3384","ondisk_log_start":"212'3384","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6295,"log_dups_size":0,"ondisk_log_size":6295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:57:38.023669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281346048,"num_objects":4293,"num_object_clones":0,"num_object_copies":12879,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9679,"num_write_kb":447104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":85,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.208124+0000","last_change":"2024-09-13T14:37:49.168003+0000","last_active":"2024-09-13T14:41:52.208124+0000","last_peered":"2024-09-13T14:41:52.208124+0000","last_clean":"2024-09-13T14:41:52.208124+0000","last_became_active":"2024-09-13T14:37:49.167638+0000","last_became_peered":"2024-09-13T14:37:49.167638+0000","last_unstale":"2024-09-13T14:41:52.208124+0000","last_undegraded":"2024-09-13T14:41:52.208124+0000","last_fullsized":"2024-09-13T14:41:52.208124+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:29.051981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":75,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T14:41:53.195471+0000","last_change":"2024-09-13T14:38:03.528381+0000","last_active":"2024-09-13T14:41:53.195471+0000","last_peered":"2024-09-13T14:41:53.195471+0000","last_clean":"2024-09-13T14:41:53.195471+0000","last_became_active":"2024-09-13T14:38:03.527282+0000","last_became_peered":"2024-09-13T14:38:03.527282+0000","last_unstale":"2024-09-13T14:41:53.195471+0000","last_undegraded":"2024-09-13T14:41:53.195471+0000","last_fullsized":"2024-09-13T14:41:53.195471+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:24.435067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.f","version":"276'9689","reported_seq":9967,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T14:41:59.749976+0000","last_change":"2024-09-13T14:38:16.371079+0000","last_active":"2024-09-13T14:41:59.749976+0000","last_peered":"2024-09-13T14:41:59.749976+0000","last_clean":"2024-09-13T14:41:59.749976+0000","last_became_active":"2024-09-13T14:38:12.859883+0000","last_became_peered":"2024-09-13T14:38:12.859883+0000","last_unstale":"2024-09-13T14:41:59.749976+0000","last_undegraded":"2024-09-13T14:41:59.749976+0000","last_fullsized":"2024-09-13T14:41:59.749976+0000","mapping_epoch":245,"log_start":"212'3763","ondisk_log_start":"212'3763","created":205,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5926,"log_dups_size":0,"ondisk_log_size":5926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:08.436532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288161792,"num_objects":4397,"num_object_clones":0,"num_object_copies":13191,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9689,"num_write_kb":450752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":3670016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":433,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T14:41:53.200831+0000","last_change":"2024-09-13T14:38:03.549595+0000","last_active":"2024-09-13T14:41:53.200831+0000","last_peered":"2024-09-13T14:41:53.200831+0000","last_clean":"2024-09-13T14:41:53.200831+0000","last_became_active":"2024-09-13T14:38:03.533331+0000","last_became_peered":"2024-09-13T14:38:03.533331+0000","last_unstale":"2024-09-13T14:41:53.200831+0000","last_undegraded":"2024-09-13T14:41:53.200831+0000","last_fullsized":"2024-09-13T14:41:53.200831+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:59:02.470343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":259,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.191491+0000","last_change":"2024-09-13T14:37:48.127600+0000","last_active":"2024-09-13T14:41:52.191491+0000","last_peered":"2024-09-13T14:41:52.191491+0000","last_clean":"2024-09-13T14:41:52.191491+0000","last_became_active":"2024-09-13T14:37:48.127349+0000","last_became_peered":"2024-09-13T14:37:48.127349+0000","last_unstale":"2024-09-13T14:41:52.191491+0000","last_undegraded":"2024-09-13T14:41:52.191491+0000","last_fullsized":"2024-09-13T14:41:52.191491+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:57.268374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"276'9744","reported_seq":15796,"reported_epoch":276,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T14:42:02.236897+0000","last_change":"2024-09-13T14:41:52.247680+0000","last_active":"2024-09-13T14:42:02.236897+0000","last_peered":"2024-09-13T14:42:02.236897+0000","last_clean":"2024-09-13T14:36:13.131873+0000","last_became_active":"2024-09-13T14:38:10.443954+0000","last_became_peered":"2024-09-13T14:38:10.443954+0000","last_unstale":"2024-09-13T14:42:02.236897+0000","last_undegraded":"2024-09-13T14:38:10.384643+0000","last_fullsized":"2024-09-13T14:38:04.517046+0000","mapping_epoch":245,"log_start":"212'4000","ondisk_log_start":"212'4000","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5744,"log_dups_size":0,"ondisk_log_size":5744,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:26:53.493170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279576576,"num_objects":4266,"num_object_clones":0,"num_object_copies":12798,"num_objects_missing_on_primary":1818,"num_objects_missing":1818,"num_objects_degraded":1,"num_objects_misplaced":1822,"num_objects_unfound":0,"num_objects_dirty":4266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9744,"num_write_kb":448320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2967,"num_bytes_recovered":194379776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":1},{"shards":"0,3,7","objects":1817},{"shards":"0,5","objects":2443},{"shards":"3,5,7","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":192,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.191051+0000","last_change":"2024-09-13T14:37:48.127604+0000","last_active":"2024-09-13T14:41:52.191051+0000","last_peered":"2024-09-13T14:41:52.191051+0000","last_clean":"2024-09-13T14:41:52.191051+0000","last_became_active":"2024-09-13T14:37:48.127300+0000","last_became_peered":"2024-09-13T14:37:48.127300+0000","last_unstale":"2024-09-13T14:41:52.191051+0000","last_undegraded":"2024-09-13T14:41:52.191051+0000","last_fullsized":"2024-09-13T14:41:52.191051+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:51.999506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":312,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.191725+0000","last_change":"2024-09-13T14:38:03.529967+0000","last_active":"2024-09-13T14:41:52.191725+0000","last_peered":"2024-09-13T14:41:52.191725+0000","last_clean":"2024-09-13T14:41:52.191725+0000","last_became_active":"2024-09-13T14:38:03.529646+0000","last_became_peered":"2024-09-13T14:38:03.529646+0000","last_unstale":"2024-09-13T14:41:52.191725+0000","last_undegraded":"2024-09-13T14:41:52.191725+0000","last_fullsized":"2024-09-13T14:41:52.191725+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:52:05.193279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"276'9769","reported_seq":18922,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T14:41:59.524105+0000","last_change":"2024-09-13T14:40:34.583237+0000","last_active":"2024-09-13T14:41:59.524105+0000","last_peered":"2024-09-13T14:41:59.524105+0000","last_clean":"2024-09-13T14:41:59.524105+0000","last_became_active":"2024-09-13T14:40:34.563989+0000","last_became_peered":"2024-09-13T14:40:34.563989+0000","last_unstale":"2024-09-13T14:41:59.524105+0000","last_undegraded":"2024-09-13T14:41:59.524105+0000","last_fullsized":"2024-09-13T14:41:59.524105+0000","mapping_epoch":266,"log_start":"212'3800","ondisk_log_start":"212'3800","created":205,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5969,"log_dups_size":0,"ondisk_log_size":5969,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:34:43.006051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":286392344,"num_objects":4371,"num_object_clones":0,"num_object_copies":13113,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4371,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9769,"num_write_kb":452417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4489,"num_bytes_recovered":289865752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":273,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.787102+0000","last_change":"2024-09-13T14:36:13.860367+0000","last_active":"2024-09-13T14:41:52.787102+0000","last_peered":"2024-09-13T14:41:52.787102+0000","last_clean":"2024-09-13T14:41:52.787102+0000","last_became_active":"2024-09-13T14:36:13.859982+0000","last_became_peered":"2024-09-13T14:36:13.859982+0000","last_unstale":"2024-09-13T14:41:52.787102+0000","last_undegraded":"2024-09-13T14:41:52.787102+0000","last_fullsized":"2024-09-13T14:41:52.787102+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:08:23.311350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":268,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.192052+0000","last_change":"2024-09-13T14:37:48.147360+0000","last_active":"2024-09-13T14:41:52.192052+0000","last_peered":"2024-09-13T14:41:52.192052+0000","last_clean":"2024-09-13T14:41:52.192052+0000","last_became_active":"2024-09-13T14:37:48.146885+0000","last_became_peered":"2024-09-13T14:37:48.146885+0000","last_unstale":"2024-09-13T14:41:52.192052+0000","last_undegraded":"2024-09-13T14:41:52.192052+0000","last_fullsized":"2024-09-13T14:41:52.192052+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:27.841810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"276'9692","reported_seq":18952,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T14:42:02.443170+0000","last_change":"2024-09-13T14:38:39.650332+0000","last_active":"2024-09-13T14:42:02.443170+0000","last_peered":"2024-09-13T14:42:02.443170+0000","last_clean":"2024-09-13T14:42:02.443170+0000","last_became_active":"2024-09-13T14:38:39.640835+0000","last_became_peered":"2024-09-13T14:38:39.640835+0000","last_unstale":"2024-09-13T14:42:02.443170+0000","last_undegraded":"2024-09-13T14:42:02.443170+0000","last_fullsized":"2024-09-13T14:42:02.443170+0000","mapping_epoch":251,"log_start":"213'4251","ondisk_log_start":"213'4251","created":205,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5441,"log_dups_size":0,"ondisk_log_size":5441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:48.176606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287178752,"num_objects":4382,"num_object_clones":0,"num_object_copies":13146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9692,"num_write_kb":450368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4544,"num_bytes_recovered":292945920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":451,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T14:41:53.200463+0000","last_change":"2024-09-13T14:38:53.900041+0000","last_active":"2024-09-13T14:41:53.200463+0000","last_peered":"2024-09-13T14:41:53.200463+0000","last_clean":"2024-09-13T14:41:53.200463+0000","last_became_active":"2024-09-13T14:38:53.899203+0000","last_became_peered":"2024-09-13T14:38:53.899203+0000","last_unstale":"2024-09-13T14:41:53.200463+0000","last_undegraded":"2024-09-13T14:41:53.200463+0000","last_fullsized":"2024-09-13T14:41:53.200463+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:28:07.883094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":383,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T14:41:53.201811+0000","last_change":"2024-09-13T14:26:15.823198+0000","last_active":"2024-09-13T14:41:53.201811+0000","last_peered":"2024-09-13T14:41:53.201811+0000","last_clean":"2024-09-13T14:41:53.201811+0000","last_became_active":"2024-09-13T14:26:15.822717+0000","last_became_peered":"2024-09-13T14:26:15.822717+0000","last_unstale":"2024-09-13T14:41:53.201811+0000","last_undegraded":"2024-09-13T14:41:53.201811+0000","last_fullsized":"2024-09-13T14:41:53.201811+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:22.471622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.6","version":"276'9851","reported_seq":15312,"reported_epoch":276,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-13T14:41:59.725488+0000","last_change":"2024-09-13T14:38:55.132683+0000","last_active":"2024-09-13T14:41:59.725488+0000","last_peered":"2024-09-13T14:41:59.725488+0000","last_clean":"2024-09-13T14:36:13.124368+0000","last_became_active":"2024-09-13T14:38:55.120883+0000","last_became_peered":"2024-09-13T14:38:55.120883+0000","last_unstale":"2024-09-13T14:41:59.725488+0000","last_undegraded":"2024-09-13T14:38:55.026610+0000","last_fullsized":"2024-09-13T14:41:59.725488+0000","mapping_epoch":254,"log_start":"212'3919","ondisk_log_start":"212'3919","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5932,"log_dups_size":0,"ondisk_log_size":5932,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284884992,"num_objects":4347,"num_object_clones":0,"num_object_copies":13041,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2360,"num_objects_misplaced":1987,"num_objects_unfound":0,"num_objects_dirty":4347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9851,"num_write_kb":454336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2645,"num_bytes_recovered":162136064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,3,5],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3,4","objects":2081},{"shards":"0,3,5","objects":2266}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":409,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T14:41:53.196835+0000","last_change":"2024-09-13T14:41:53.196835+0000","last_active":"2024-09-13T14:41:53.196835+0000","last_peered":"2024-09-13T14:41:53.196835+0000","last_clean":"2024-09-13T14:41:53.196835+0000","last_became_active":"2024-09-13T14:41:53.196387+0000","last_became_peered":"2024-09-13T14:41:53.196387+0000","last_unstale":"2024-09-13T14:41:53.196835+0000","last_undegraded":"2024-09-13T14:41:53.196835+0000","last_fullsized":"2024-09-13T14:41:53.196835+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:07:41.927740+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:49.972303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":514,"reported_epoch":275,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:41:52.208181+0000","last_change":"2024-09-13T14:38:54.904329+0000","last_active":"2024-09-13T14:41:52.208181+0000","last_peered":"2024-09-13T14:41:52.208181+0000","last_clean":"2024-09-13T14:38:39.648017+0000","last_became_active":"2024-09-13T14:38:54.903798+0000","last_became_peered":"2024-09-13T14:38:54.903798+0000","last_unstale":"2024-09-13T14:41:52.208181+0000","last_undegraded":"2024-09-13T14:38:54.890505+0000","last_fullsized":"2024-09-13T14:38:54.889701+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:18.120931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,3],"avail_no_missing":["5","3","6"],"object_location_counts":[{"shards":"3,5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":386,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T14:41:53.200546+0000","last_change":"2024-09-13T14:37:49.160719+0000","last_active":"2024-09-13T14:41:53.200546+0000","last_peered":"2024-09-13T14:41:53.200546+0000","last_clean":"2024-09-13T14:41:53.200546+0000","last_became_active":"2024-09-13T14:37:49.160401+0000","last_became_peered":"2024-09-13T14:37:49.160401+0000","last_unstale":"2024-09-13T14:41:53.200546+0000","last_undegraded":"2024-09-13T14:41:53.200546+0000","last_fullsized":"2024-09-13T14:41:53.200546+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:21.737945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":119,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.191677+0000","last_change":"2024-09-13T14:37:48.172940+0000","last_active":"2024-09-13T14:41:52.191677+0000","last_peered":"2024-09-13T14:41:52.191677+0000","last_clean":"2024-09-13T14:41:52.191677+0000","last_became_active":"2024-09-13T14:37:48.172060+0000","last_became_peered":"2024-09-13T14:37:48.172060+0000","last_unstale":"2024-09-13T14:41:52.191677+0000","last_undegraded":"2024-09-13T14:41:52.191677+0000","last_fullsized":"2024-09-13T14:41:52.191677+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:07.480645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.5","version":"276'9741","reported_seq":26100,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T14:42:01.764881+0000","last_change":"2024-09-13T14:39:34.446699+0000","last_active":"2024-09-13T14:42:01.764881+0000","last_peered":"2024-09-13T14:42:01.764881+0000","last_clean":"2024-09-13T14:42:01.764881+0000","last_became_active":"2024-09-13T14:38:04.600616+0000","last_became_peered":"2024-09-13T14:38:04.600616+0000","last_unstale":"2024-09-13T14:42:01.764881+0000","last_undegraded":"2024-09-13T14:42:01.764881+0000","last_fullsized":"2024-09-13T14:42:01.764881+0000","mapping_epoch":245,"log_start":"212'3416","ondisk_log_start":"212'3416","created":205,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6325,"log_dups_size":0,"ondisk_log_size":6325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:00:44.321678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288423936,"num_objects":4401,"num_object_clones":0,"num_object_copies":13203,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9741,"num_write_kb":452544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8120,"num_bytes_recovered":521076736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":492,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.193770+0000","last_change":"2024-09-13T14:26:15.835178+0000","last_active":"2024-09-13T14:41:52.193770+0000","last_peered":"2024-09-13T14:41:52.193770+0000","last_clean":"2024-09-13T14:41:52.193770+0000","last_became_active":"2024-09-13T14:26:15.820283+0000","last_became_peered":"2024-09-13T14:26:15.820283+0000","last_unstale":"2024-09-13T14:41:52.193770+0000","last_undegraded":"2024-09-13T14:41:52.193770+0000","last_fullsized":"2024-09-13T14:41:52.193770+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:30.340038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":75,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T14:41:53.194303+0000","last_change":"2024-09-13T14:38:03.525682+0000","last_active":"2024-09-13T14:41:53.194303+0000","last_peered":"2024-09-13T14:41:53.194303+0000","last_clean":"2024-09-13T14:41:53.194303+0000","last_became_active":"2024-09-13T14:38:03.525406+0000","last_became_peered":"2024-09-13T14:38:03.525406+0000","last_unstale":"2024-09-13T14:41:53.194303+0000","last_undegraded":"2024-09-13T14:41:53.194303+0000","last_fullsized":"2024-09-13T14:41:53.194303+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:41:28.970300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":82,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T14:41:53.195833+0000","last_change":"2024-09-13T14:38:53.991738+0000","last_active":"2024-09-13T14:41:53.195833+0000","last_peered":"2024-09-13T14:41:53.195833+0000","last_clean":"2024-09-13T14:41:53.195833+0000","last_became_active":"2024-09-13T14:38:53.991417+0000","last_became_peered":"2024-09-13T14:38:53.991417+0000","last_unstale":"2024-09-13T14:41:53.195833+0000","last_undegraded":"2024-09-13T14:41:53.195833+0000","last_fullsized":"2024-09-13T14:41:53.195833+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:50:38.559838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"276'9751","reported_seq":24042,"reported_epoch":276,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-13T14:41:59.751928+0000","last_change":"2024-09-13T14:38:17.472071+0000","last_active":"2024-09-13T14:41:59.751928+0000","last_peered":"2024-09-13T14:41:59.751928+0000","last_clean":"2024-09-13T14:37:46.290754+0000","last_became_active":"2024-09-13T14:38:17.381753+0000","last_became_peered":"2024-09-13T14:38:17.381753+0000","last_unstale":"2024-09-13T14:41:59.751928+0000","last_undegraded":"2024-09-13T14:38:13.504140+0000","last_fullsized":"2024-09-13T14:41:59.751928+0000","mapping_epoch":245,"log_start":"212'3859","ondisk_log_start":"212'3859","created":205,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5892,"log_dups_size":0,"ondisk_log_size":5892,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":285409280,"num_objects":4355,"num_object_clones":0,"num_object_copies":13065,"num_objects_missing_on_primary":83,"num_objects_missing":83,"num_objects_degraded":4523,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9751,"num_write_kb":451392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7112,"num_bytes_recovered":466092032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"0,4,5","objects":4354},{"shards":"0,4,6","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":797,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T14:41:53.200233+0000","last_change":"2024-09-13T14:37:48.131799+0000","last_active":"2024-09-13T14:41:53.200233+0000","last_peered":"2024-09-13T14:41:53.200233+0000","last_clean":"2024-09-13T14:41:53.200233+0000","last_became_active":"2024-09-13T14:37:48.130837+0000","last_became_peered":"2024-09-13T14:37:48.130837+0000","last_unstale":"2024-09-13T14:41:53.200233+0000","last_undegraded":"2024-09-13T14:41:53.200233+0000","last_fullsized":"2024-09-13T14:41:53.200233+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:06:32.663770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":315,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T14:41:53.194865+0000","last_change":"2024-09-13T14:37:49.301531+0000","last_active":"2024-09-13T14:41:53.194865+0000","last_peered":"2024-09-13T14:41:53.194865+0000","last_clean":"2024-09-13T14:41:53.194865+0000","last_became_active":"2024-09-13T14:37:49.301249+0000","last_became_peered":"2024-09-13T14:37:49.301249+0000","last_unstale":"2024-09-13T14:41:53.194865+0000","last_undegraded":"2024-09-13T14:41:53.194865+0000","last_fullsized":"2024-09-13T14:41:53.194865+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:10.977710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":101,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.208836+0000","last_change":"2024-09-13T14:38:53.901427+0000","last_active":"2024-09-13T14:41:52.208836+0000","last_peered":"2024-09-13T14:41:52.208836+0000","last_clean":"2024-09-13T14:41:52.208836+0000","last_became_active":"2024-09-13T14:38:53.901157+0000","last_became_peered":"2024-09-13T14:38:53.901157+0000","last_unstale":"2024-09-13T14:41:52.208836+0000","last_undegraded":"2024-09-13T14:41:52.208836+0000","last_fullsized":"2024-09-13T14:41:52.208836+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:28:21.713025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"276'9657","reported_seq":23852,"reported_epoch":276,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-13T14:41:59.752208+0000","last_change":"2024-09-13T14:38:16.552896+0000","last_active":"2024-09-13T14:41:59.752208+0000","last_peered":"2024-09-13T14:41:59.752208+0000","last_clean":"2024-09-13T14:37:46.405415+0000","last_became_active":"2024-09-13T14:38:16.427510+0000","last_became_peered":"2024-09-13T14:38:16.427510+0000","last_unstale":"2024-09-13T14:41:59.752208+0000","last_undegraded":"2024-09-13T14:38:12.738464+0000","last_fullsized":"2024-09-13T14:41:59.752208+0000","mapping_epoch":245,"log_start":"212'3718","ondisk_log_start":"212'3718","created":205,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5939,"log_dups_size":0,"ondisk_log_size":5939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284491776,"num_objects":4341,"num_object_clones":0,"num_object_copies":13023,"num_objects_missing_on_primary":100,"num_objects_missing":100,"num_objects_degraded":4507,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9657,"num_write_kb":447936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7064,"num_bytes_recovered":462946304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"0,3,6","objects":1},{"shards":"0,3,7","objects":4340}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":983,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.209021+0000","last_change":"2024-09-13T14:37:49.147498+0000","last_active":"2024-09-13T14:41:52.209021+0000","last_peered":"2024-09-13T14:41:52.209021+0000","last_clean":"2024-09-13T14:41:52.209021+0000","last_became_active":"2024-09-13T14:37:49.147028+0000","last_became_peered":"2024-09-13T14:37:49.147028+0000","last_unstale":"2024-09-13T14:41:52.209021+0000","last_undegraded":"2024-09-13T14:41:52.209021+0000","last_fullsized":"2024-09-13T14:41:52.209021+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:11:29.050178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":281,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.191275+0000","last_change":"2024-09-13T14:38:03.528133+0000","last_active":"2024-09-13T14:41:52.191275+0000","last_peered":"2024-09-13T14:41:52.191275+0000","last_clean":"2024-09-13T14:41:52.191275+0000","last_became_active":"2024-09-13T14:38:03.527829+0000","last_became_peered":"2024-09-13T14:38:03.527829+0000","last_unstale":"2024-09-13T14:41:52.191275+0000","last_undegraded":"2024-09-13T14:41:52.191275+0000","last_fullsized":"2024-09-13T14:41:52.191275+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:48.254169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":436,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T14:41:53.200783+0000","last_change":"2024-09-13T14:38:53.899755+0000","last_active":"2024-09-13T14:41:53.200783+0000","last_peered":"2024-09-13T14:41:53.200783+0000","last_clean":"2024-09-13T14:41:53.200783+0000","last_became_active":"2024-09-13T14:38:53.899448+0000","last_became_peered":"2024-09-13T14:38:53.899448+0000","last_unstale":"2024-09-13T14:41:53.200783+0000","last_undegraded":"2024-09-13T14:41:53.200783+0000","last_fullsized":"2024-09-13T14:41:53.200783+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:55.897035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"276'9961","reported_seq":10075,"reported_epoch":276,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:42:02.415295+0000","last_change":"2024-09-13T14:41:52.203221+0000","last_active":"2024-09-13T14:42:02.415295+0000","last_peered":"2024-09-13T14:42:02.415295+0000","last_clean":"2024-09-13T14:35:51.090791+0000","last_became_active":"2024-09-13T14:37:49.665025+0000","last_became_peered":"2024-09-13T14:37:49.665025+0000","last_unstale":"2024-09-13T14:42:02.415295+0000","last_undegraded":"2024-09-13T14:37:49.169101+0000","last_fullsized":"2024-09-13T14:37:49.121596+0000","mapping_epoch":240,"log_start":"212'3863","ondisk_log_start":"212'3863","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6098,"log_dups_size":0,"ondisk_log_size":6098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:02.321504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284491776,"num_objects":4341,"num_object_clones":0,"num_object_copies":13023,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4496,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9961,"num_write_kb":457664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":393216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":-89}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":85,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.208477+0000","last_change":"2024-09-13T14:37:49.149770+0000","last_active":"2024-09-13T14:41:52.208477+0000","last_peered":"2024-09-13T14:41:52.208477+0000","last_clean":"2024-09-13T14:41:52.208477+0000","last_became_active":"2024-09-13T14:37:49.148991+0000","last_became_peered":"2024-09-13T14:37:49.148991+0000","last_unstale":"2024-09-13T14:41:52.208477+0000","last_undegraded":"2024-09-13T14:41:52.208477+0000","last_fullsized":"2024-09-13T14:41:52.208477+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:04:04.193366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":1134,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T14:41:53.206894+0000","last_change":"2024-09-13T14:30:58.928302+0000","last_active":"2024-09-13T14:41:53.206894+0000","last_peered":"2024-09-13T14:41:53.206894+0000","last_clean":"2024-09-13T14:41:53.206894+0000","last_became_active":"2024-09-13T14:30:58.927718+0000","last_became_peered":"2024-09-13T14:30:58.927718+0000","last_unstale":"2024-09-13T14:41:53.206894+0000","last_undegraded":"2024-09-13T14:41:53.206894+0000","last_fullsized":"2024-09-13T14:41:53.206894+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:37.323036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":92,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.208602+0000","last_change":"2024-09-13T14:38:03.534628+0000","last_active":"2024-09-13T14:41:52.208602+0000","last_peered":"2024-09-13T14:41:52.208602+0000","last_clean":"2024-09-13T14:41:52.208602+0000","last_became_active":"2024-09-13T14:38:03.534262+0000","last_became_peered":"2024-09-13T14:38:03.534262+0000","last_unstale":"2024-09-13T14:41:52.208602+0000","last_undegraded":"2024-09-13T14:41:52.208602+0000","last_fullsized":"2024-09-13T14:41:52.208602+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:31.168783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"276'9693","reported_seq":23856,"reported_epoch":276,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:42:01.865562+0000","last_change":"2024-09-13T14:38:53.944111+0000","last_active":"2024-09-13T14:42:01.865562+0000","last_peered":"2024-09-13T14:42:01.865562+0000","last_clean":"2024-09-13T14:37:47.047672+0000","last_became_active":"2024-09-13T14:37:49.664294+0000","last_became_peered":"2024-09-13T14:37:49.664294+0000","last_unstale":"2024-09-13T14:42:01.865562+0000","last_undegraded":"2024-09-13T14:38:53.940822+0000","last_fullsized":"2024-09-13T14:37:49.126627+0000","mapping_epoch":240,"log_start":"213'4000","ondisk_log_start":"213'4000","created":205,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5693,"log_dups_size":0,"ondisk_log_size":5693,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:11.766692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":285671424,"num_objects":4359,"num_object_clones":0,"num_object_copies":13077,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4535,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9693,"num_write_kb":449664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7009,"num_bytes_recovered":459341824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3],"avail_no_missing":["5","0","3"],"object_location_counts":[{"shards":"0,3,5","objects":4359},{"shards":"3,5","objects":-101}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":104,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.209083+0000","last_change":"2024-09-13T14:38:03.534303+0000","last_active":"2024-09-13T14:41:52.209083+0000","last_peered":"2024-09-13T14:41:52.209083+0000","last_clean":"2024-09-13T14:41:52.209083+0000","last_became_active":"2024-09-13T14:38:03.533171+0000","last_became_peered":"2024-09-13T14:38:03.533171+0000","last_unstale":"2024-09-13T14:41:52.209083+0000","last_undegraded":"2024-09-13T14:41:52.209083+0000","last_fullsized":"2024-09-13T14:41:52.209083+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:00:16.553526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":797,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.192838+0000","last_change":"2024-09-13T14:38:04.596192+0000","last_active":"2024-09-13T14:41:52.192838+0000","last_peered":"2024-09-13T14:41:52.192838+0000","last_clean":"2024-09-13T14:41:52.192838+0000","last_became_active":"2024-09-13T14:38:04.593299+0000","last_became_peered":"2024-09-13T14:38:04.593299+0000","last_unstale":"2024-09-13T14:41:52.192838+0000","last_undegraded":"2024-09-13T14:41:52.192838+0000","last_fullsized":"2024-09-13T14:41:52.192838+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:34.241410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":75,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T14:41:53.195789+0000","last_change":"2024-09-13T14:38:03.533120+0000","last_active":"2024-09-13T14:41:53.195789+0000","last_peered":"2024-09-13T14:41:53.195789+0000","last_clean":"2024-09-13T14:41:53.195789+0000","last_became_active":"2024-09-13T14:38:03.532829+0000","last_became_peered":"2024-09-13T14:38:03.532829+0000","last_unstale":"2024-09-13T14:41:53.195789+0000","last_undegraded":"2024-09-13T14:41:53.195789+0000","last_fullsized":"2024-09-13T14:41:53.195789+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:24.514661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"276'9906","reported_seq":18516,"reported_epoch":276,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-13T14:42:02.501414+0000","last_change":"2024-09-13T14:41:52.205922+0000","last_active":"2024-09-13T14:42:02.501414+0000","last_peered":"2024-09-13T14:42:02.501414+0000","last_clean":"2024-09-13T14:35:51.078708+0000","last_became_active":"2024-09-13T14:37:49.290403+0000","last_became_peered":"2024-09-13T14:37:49.290403+0000","last_unstale":"2024-09-13T14:42:02.501414+0000","last_undegraded":"2024-09-13T14:37:49.147402+0000","last_fullsized":"2024-09-13T14:37:49.117202+0000","mapping_epoch":240,"log_start":"213'4518","ondisk_log_start":"213'4518","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5388,"log_dups_size":0,"ondisk_log_size":5388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294387712,"num_objects":4492,"num_object_clones":0,"num_object_copies":13476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9906,"num_write_kb":460736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4223,"num_bytes_recovered":276758528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":4492}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":477,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T14:41:53.201932+0000","last_change":"2024-09-13T14:38:53.938227+0000","last_active":"2024-09-13T14:41:53.201932+0000","last_peered":"2024-09-13T14:41:53.201932+0000","last_clean":"2024-09-13T14:41:53.201932+0000","last_became_active":"2024-09-13T14:38:53.902867+0000","last_became_peered":"2024-09-13T14:38:53.902867+0000","last_unstale":"2024-09-13T14:41:53.201932+0000","last_undegraded":"2024-09-13T14:41:53.201932+0000","last_fullsized":"2024-09-13T14:41:53.201932+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:16.831756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":575,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.192135+0000","last_change":"2024-09-13T14:37:48.147677+0000","last_active":"2024-09-13T14:41:52.192135+0000","last_peered":"2024-09-13T14:41:52.192135+0000","last_clean":"2024-09-13T14:41:52.192135+0000","last_became_active":"2024-09-13T14:37:48.147001+0000","last_became_peered":"2024-09-13T14:37:48.147001+0000","last_unstale":"2024-09-13T14:41:52.192135+0000","last_undegraded":"2024-09-13T14:41:52.192135+0000","last_fullsized":"2024-09-13T14:41:52.192135+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:35:00.653576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":459,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T14:41:53.200277+0000","last_change":"2024-09-13T14:38:53.900107+0000","last_active":"2024-09-13T14:41:53.200277+0000","last_peered":"2024-09-13T14:41:53.200277+0000","last_clean":"2024-09-13T14:41:53.200277+0000","last_became_active":"2024-09-13T14:38:53.899556+0000","last_became_peered":"2024-09-13T14:38:53.899556+0000","last_unstale":"2024-09-13T14:41:53.200277+0000","last_undegraded":"2024-09-13T14:41:53.200277+0000","last_fullsized":"2024-09-13T14:41:53.200277+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:15:47.918163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"276'9617","reported_seq":18583,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T14:42:02.285894+0000","last_change":"2024-09-13T14:39:54.921239+0000","last_active":"2024-09-13T14:42:02.285894+0000","last_peered":"2024-09-13T14:42:02.285894+0000","last_clean":"2024-09-13T14:42:02.285894+0000","last_became_active":"2024-09-13T14:39:54.907894+0000","last_became_peered":"2024-09-13T14:39:54.907894+0000","last_unstale":"2024-09-13T14:42:02.285894+0000","last_undegraded":"2024-09-13T14:42:02.285894+0000","last_fullsized":"2024-09-13T14:42:02.285894+0000","mapping_epoch":263,"log_start":"213'4247","ondisk_log_start":"213'4247","created":205,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5370,"log_dups_size":0,"ondisk_log_size":5370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:52:04.619238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":277544960,"num_objects":4235,"num_object_clones":0,"num_object_copies":12705,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9617,"num_write_kb":443264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4395,"num_bytes_recovered":283770880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":90,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T14:41:53.195619+0000","last_change":"2024-09-13T14:38:53.901806+0000","last_active":"2024-09-13T14:41:53.195619+0000","last_peered":"2024-09-13T14:41:53.195619+0000","last_clean":"2024-09-13T14:41:53.195619+0000","last_became_active":"2024-09-13T14:38:53.891055+0000","last_became_peered":"2024-09-13T14:38:53.891055+0000","last_unstale":"2024-09-13T14:41:53.195619+0000","last_undegraded":"2024-09-13T14:41:53.195619+0000","last_fullsized":"2024-09-13T14:41:53.195619+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:03:00.416198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":32,"reported_epoch":275,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-13T14:41:52.193121+0000","last_change":"2024-09-13T14:38:53.900902+0000","last_active":"2024-09-13T14:41:52.193121+0000","last_peered":"2024-09-13T14:41:52.193121+0000","last_clean":"2024-09-13T14:38:39.635779+0000","last_became_active":"2024-09-13T14:38:53.900626+0000","last_became_peered":"2024-09-13T14:38:53.900626+0000","last_unstale":"2024-09-13T14:41:52.193121+0000","last_undegraded":"2024-09-13T14:38:53.885778+0000","last_fullsized":"2024-09-13T14:41:52.193121+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:37:21.715778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":7,"num_objects_missing":7,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":["3","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":265,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.190728+0000","last_change":"2024-09-13T14:37:49.172218+0000","last_active":"2024-09-13T14:41:52.190728+0000","last_peered":"2024-09-13T14:41:52.190728+0000","last_clean":"2024-09-13T14:41:52.190728+0000","last_became_active":"2024-09-13T14:37:49.171718+0000","last_became_peered":"2024-09-13T14:37:49.171718+0000","last_unstale":"2024-09-13T14:41:52.190728+0000","last_undegraded":"2024-09-13T14:41:52.190728+0000","last_fullsized":"2024-09-13T14:41:52.190728+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:46:54.626521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":288,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.787134+0000","last_change":"2024-09-13T14:37:49.230094+0000","last_active":"2024-09-13T14:41:52.787134+0000","last_peered":"2024-09-13T14:41:52.787134+0000","last_clean":"2024-09-13T14:41:52.787134+0000","last_became_active":"2024-09-13T14:37:49.229393+0000","last_became_peered":"2024-09-13T14:37:49.229393+0000","last_unstale":"2024-09-13T14:41:52.787134+0000","last_undegraded":"2024-09-13T14:41:52.787134+0000","last_fullsized":"2024-09-13T14:41:52.787134+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:02:28.690318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":268,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T14:41:53.199735+0000","last_change":"2024-09-13T14:37:48.128447+0000","last_active":"2024-09-13T14:41:53.199735+0000","last_peered":"2024-09-13T14:41:53.199735+0000","last_clean":"2024-09-13T14:41:53.199735+0000","last_became_active":"2024-09-13T14:37:48.128143+0000","last_became_peered":"2024-09-13T14:37:48.128143+0000","last_unstale":"2024-09-13T14:41:53.199735+0000","last_undegraded":"2024-09-13T14:41:53.199735+0000","last_fullsized":"2024-09-13T14:41:53.199735+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:22.204039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":75,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T14:41:53.195146+0000","last_change":"2024-09-13T14:38:03.531683+0000","last_active":"2024-09-13T14:41:53.195146+0000","last_peered":"2024-09-13T14:41:53.195146+0000","last_clean":"2024-09-13T14:41:53.195146+0000","last_became_active":"2024-09-13T14:38:03.531476+0000","last_became_peered":"2024-09-13T14:38:03.531476+0000","last_unstale":"2024-09-13T14:41:53.195146+0000","last_undegraded":"2024-09-13T14:41:53.195146+0000","last_fullsized":"2024-09-13T14:41:53.195146+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:41.557569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":40,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.192457+0000","last_change":"2024-09-13T14:38:53.954281+0000","last_active":"2024-09-13T14:41:52.192457+0000","last_peered":"2024-09-13T14:41:52.192457+0000","last_clean":"2024-09-13T14:41:52.192457+0000","last_became_active":"2024-09-13T14:38:53.900912+0000","last_became_peered":"2024-09-13T14:38:53.900912+0000","last_unstale":"2024-09-13T14:41:52.192457+0000","last_undegraded":"2024-09-13T14:41:52.192457+0000","last_fullsized":"2024-09-13T14:41:52.192457+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:47.876275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":92,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.237620+0000","last_change":"2024-09-13T14:38:54.000782+0000","last_active":"2024-09-13T14:41:52.237620+0000","last_peered":"2024-09-13T14:41:52.237620+0000","last_clean":"2024-09-13T14:41:52.237620+0000","last_became_active":"2024-09-13T14:38:54.000322+0000","last_became_peered":"2024-09-13T14:38:54.000322+0000","last_unstale":"2024-09-13T14:41:52.237620+0000","last_undegraded":"2024-09-13T14:41:52.237620+0000","last_fullsized":"2024-09-13T14:41:52.237620+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:09.057597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":401,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T14:41:53.199812+0000","last_change":"2024-09-13T14:38:53.899535+0000","last_active":"2024-09-13T14:41:53.199812+0000","last_peered":"2024-09-13T14:41:53.199812+0000","last_clean":"2024-09-13T14:41:53.199812+0000","last_became_active":"2024-09-13T14:38:53.899183+0000","last_became_peered":"2024-09-13T14:38:53.899183+0000","last_unstale":"2024-09-13T14:41:53.199812+0000","last_undegraded":"2024-09-13T14:41:53.199812+0000","last_fullsized":"2024-09-13T14:41:53.199812+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:33.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":75,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T14:41:53.219087+0000","last_change":"2024-09-13T14:38:03.543426+0000","last_active":"2024-09-13T14:41:53.219087+0000","last_peered":"2024-09-13T14:41:53.219087+0000","last_clean":"2024-09-13T14:41:53.219087+0000","last_became_active":"2024-09-13T14:38:03.542661+0000","last_became_peered":"2024-09-13T14:38:03.542661+0000","last_unstale":"2024-09-13T14:41:53.219087+0000","last_undegraded":"2024-09-13T14:41:53.219087+0000","last_fullsized":"2024-09-13T14:41:53.219087+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:54.050127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":430,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T14:41:53.200188+0000","last_change":"2024-09-13T14:37:48.131321+0000","last_active":"2024-09-13T14:41:53.200188+0000","last_peered":"2024-09-13T14:41:53.200188+0000","last_clean":"2024-09-13T14:41:53.200188+0000","last_became_active":"2024-09-13T14:37:48.130998+0000","last_became_peered":"2024-09-13T14:37:48.130998+0000","last_unstale":"2024-09-13T14:41:53.200188+0000","last_undegraded":"2024-09-13T14:41:53.200188+0000","last_fullsized":"2024-09-13T14:41:53.200188+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:33:26.597825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":104,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.208742+0000","last_change":"2024-09-13T14:38:03.535454+0000","last_active":"2024-09-13T14:41:52.208742+0000","last_peered":"2024-09-13T14:41:52.208742+0000","last_clean":"2024-09-13T14:41:52.208742+0000","last_became_active":"2024-09-13T14:38:03.535131+0000","last_became_peered":"2024-09-13T14:38:03.535131+0000","last_unstale":"2024-09-13T14:41:52.208742+0000","last_undegraded":"2024-09-13T14:41:52.208742+0000","last_fullsized":"2024-09-13T14:41:52.208742+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:24.477601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":455,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T14:41:53.200375+0000","last_change":"2024-09-13T14:38:03.524939+0000","last_active":"2024-09-13T14:41:53.200375+0000","last_peered":"2024-09-13T14:41:53.200375+0000","last_clean":"2024-09-13T14:41:53.200375+0000","last_became_active":"2024-09-13T14:38:03.524406+0000","last_became_peered":"2024-09-13T14:38:03.524406+0000","last_unstale":"2024-09-13T14:41:53.200375+0000","last_undegraded":"2024-09-13T14:41:53.200375+0000","last_fullsized":"2024-09-13T14:41:53.200375+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:19:45.257424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":82,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T14:41:53.195504+0000","last_change":"2024-09-13T14:38:53.892948+0000","last_active":"2024-09-13T14:41:53.195504+0000","last_peered":"2024-09-13T14:41:53.195504+0000","last_clean":"2024-09-13T14:41:53.195504+0000","last_became_active":"2024-09-13T14:38:53.892665+0000","last_became_peered":"2024-09-13T14:38:53.892665+0000","last_unstale":"2024-09-13T14:41:53.195504+0000","last_undegraded":"2024-09-13T14:41:53.195504+0000","last_fullsized":"2024-09-13T14:41:53.195504+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:08.292359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":403,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T14:41:53.200143+0000","last_change":"2024-09-13T14:37:48.139348+0000","last_active":"2024-09-13T14:41:53.200143+0000","last_peered":"2024-09-13T14:41:53.200143+0000","last_clean":"2024-09-13T14:41:53.200143+0000","last_became_active":"2024-09-13T14:37:48.139115+0000","last_became_peered":"2024-09-13T14:37:48.139115+0000","last_unstale":"2024-09-13T14:41:53.200143+0000","last_undegraded":"2024-09-13T14:41:53.200143+0000","last_fullsized":"2024-09-13T14:41:53.200143+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:32:20.413346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":443,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T14:41:53.201278+0000","last_change":"2024-09-13T14:37:48.172088+0000","last_active":"2024-09-13T14:41:53.201278+0000","last_peered":"2024-09-13T14:41:53.201278+0000","last_clean":"2024-09-13T14:41:53.201278+0000","last_became_active":"2024-09-13T14:37:48.171779+0000","last_became_peered":"2024-09-13T14:37:48.171779+0000","last_unstale":"2024-09-13T14:41:53.201278+0000","last_undegraded":"2024-09-13T14:41:53.201278+0000","last_fullsized":"2024-09-13T14:41:53.201278+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:02.852712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":332,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.191132+0000","last_change":"2024-09-13T14:38:53.890236+0000","last_active":"2024-09-13T14:41:52.191132+0000","last_peered":"2024-09-13T14:41:52.191132+0000","last_clean":"2024-09-13T14:41:52.191132+0000","last_became_active":"2024-09-13T14:38:53.889582+0000","last_became_peered":"2024-09-13T14:38:53.889582+0000","last_unstale":"2024-09-13T14:41:52.191132+0000","last_undegraded":"2024-09-13T14:41:52.191132+0000","last_fullsized":"2024-09-13T14:41:52.191132+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:14:29.334774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":429,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T14:41:53.201036+0000","last_change":"2024-09-13T14:38:03.534097+0000","last_active":"2024-09-13T14:41:53.201036+0000","last_peered":"2024-09-13T14:41:53.201036+0000","last_clean":"2024-09-13T14:41:53.201036+0000","last_became_active":"2024-09-13T14:38:03.533794+0000","last_became_peered":"2024-09-13T14:38:03.533794+0000","last_unstale":"2024-09-13T14:41:53.201036+0000","last_undegraded":"2024-09-13T14:41:53.201036+0000","last_fullsized":"2024-09-13T14:41:53.201036+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:23.272098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":75,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T14:41:53.195370+0000","last_change":"2024-09-13T14:38:03.527021+0000","last_active":"2024-09-13T14:41:53.195370+0000","last_peered":"2024-09-13T14:41:53.195370+0000","last_clean":"2024-09-13T14:41:53.195370+0000","last_became_active":"2024-09-13T14:38:03.526698+0000","last_became_peered":"2024-09-13T14:38:03.526698+0000","last_unstale":"2024-09-13T14:41:53.195370+0000","last_undegraded":"2024-09-13T14:41:53.195370+0000","last_fullsized":"2024-09-13T14:41:53.195370+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:01:35.734086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":276,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.191134+0000","last_change":"2024-09-13T14:27:12.834931+0000","last_active":"2024-09-13T14:41:52.191134+0000","last_peered":"2024-09-13T14:41:52.191134+0000","last_clean":"2024-09-13T14:41:52.191134+0000","last_became_active":"2024-09-13T14:27:12.834572+0000","last_became_peered":"2024-09-13T14:27:12.834572+0000","last_unstale":"2024-09-13T14:41:52.191134+0000","last_undegraded":"2024-09-13T14:41:52.191134+0000","last_fullsized":"2024-09-13T14:41:52.191134+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:54:21.155689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":89,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.208265+0000","last_change":"2024-09-13T14:37:49.176796+0000","last_active":"2024-09-13T14:41:52.208265+0000","last_peered":"2024-09-13T14:41:52.208265+0000","last_clean":"2024-09-13T14:41:52.208265+0000","last_became_active":"2024-09-13T14:37:49.176246+0000","last_became_peered":"2024-09-13T14:37:49.176246+0000","last_unstale":"2024-09-13T14:41:52.208265+0000","last_undegraded":"2024-09-13T14:41:52.208265+0000","last_fullsized":"2024-09-13T14:41:52.208265+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:29:24.936718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":287,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.190972+0000","last_change":"2024-09-13T14:38:03.526333+0000","last_active":"2024-09-13T14:41:52.190972+0000","last_peered":"2024-09-13T14:41:52.190972+0000","last_clean":"2024-09-13T14:41:52.190972+0000","last_became_active":"2024-09-13T14:38:03.526001+0000","last_became_peered":"2024-09-13T14:38:03.526001+0000","last_unstale":"2024-09-13T14:41:52.190972+0000","last_undegraded":"2024-09-13T14:41:52.190972+0000","last_fullsized":"2024-09-13T14:41:52.190972+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:25:25.181271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":89,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.208550+0000","last_change":"2024-09-13T14:37:49.149582+0000","last_active":"2024-09-13T14:41:52.208550+0000","last_peered":"2024-09-13T14:41:52.208550+0000","last_clean":"2024-09-13T14:41:52.208550+0000","last_became_active":"2024-09-13T14:37:49.148525+0000","last_became_peered":"2024-09-13T14:37:49.148525+0000","last_unstale":"2024-09-13T14:41:52.208550+0000","last_undegraded":"2024-09-13T14:41:52.208550+0000","last_fullsized":"2024-09-13T14:41:52.208550+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:48.655665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":476,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T14:41:53.200331+0000","last_change":"2024-09-13T14:38:03.535255+0000","last_active":"2024-09-13T14:41:53.200331+0000","last_peered":"2024-09-13T14:41:53.200331+0000","last_clean":"2024-09-13T14:41:53.200331+0000","last_became_active":"2024-09-13T14:38:03.533917+0000","last_became_peered":"2024-09-13T14:38:03.533917+0000","last_unstale":"2024-09-13T14:41:53.200331+0000","last_undegraded":"2024-09-13T14:41:53.200331+0000","last_fullsized":"2024-09-13T14:41:53.200331+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:33.559788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":439,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T14:41:53.202316+0000","last_change":"2024-09-13T14:37:48.146784+0000","last_active":"2024-09-13T14:41:53.202316+0000","last_peered":"2024-09-13T14:41:53.202316+0000","last_clean":"2024-09-13T14:41:53.202316+0000","last_became_active":"2024-09-13T14:37:48.146491+0000","last_became_peered":"2024-09-13T14:37:48.146491+0000","last_unstale":"2024-09-13T14:41:53.202316+0000","last_undegraded":"2024-09-13T14:41:53.202316+0000","last_fullsized":"2024-09-13T14:41:53.202316+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:30:54.772513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":267,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-13T14:41:52.190588+0000","last_change":"2024-09-13T14:37:48.127607+0000","last_active":"2024-09-13T14:41:52.190588+0000","last_peered":"2024-09-13T14:41:52.190588+0000","last_clean":"2024-09-13T14:41:52.190588+0000","last_became_active":"2024-09-13T14:37:48.127316+0000","last_became_peered":"2024-09-13T14:37:48.127316+0000","last_unstale":"2024-09-13T14:41:52.190588+0000","last_undegraded":"2024-09-13T14:41:52.190588+0000","last_fullsized":"2024-09-13T14:41:52.190588+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:53:37.678861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":81,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-13T14:41:53.219026+0000","last_change":"2024-09-13T14:38:03.563200+0000","last_active":"2024-09-13T14:41:53.219026+0000","last_peered":"2024-09-13T14:41:53.219026+0000","last_clean":"2024-09-13T14:41:53.219026+0000","last_became_active":"2024-09-13T14:38:03.534258+0000","last_became_peered":"2024-09-13T14:38:03.534258+0000","last_unstale":"2024-09-13T14:41:53.219026+0000","last_undegraded":"2024-09-13T14:41:53.219026+0000","last_fullsized":"2024-09-13T14:41:53.219026+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:49:07.814419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":4555079704,"num_objects":69506,"num_object_clones":0,"num_object_copies":208518,"num_objects_missing_on_primary":2001,"num_objects_missing":2001,"num_objects_degraded":20733,"num_objects_misplaced":3809,"num_objects_unfound":0,"num_objects_dirty":69506,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":156120,"num_write_kb":7219969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69583,"num_bytes_recovered":4519231512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15316955136,"data_stored":15316942920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93043,"ondisk_log_size":93043,"up":48,"acting":44,"num_store_stats":6},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1538,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":17390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25132,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":7,"num_objects_missing":7,"num_objects_degraded":16,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16793600,"data_stored":16777292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":23,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2969600,"data_stored":2951840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234378,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2852312,"kb_used_data":2406096,"kb_used_omap":1,"kb_used_meta":446206,"kb_avail":90892840,"statfs":{"total":95995035648,"available":93074268160,"internally_reserved":0,"allocated":2463842304,"data_stored":2462345866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":456915403},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":239,"seq":1026497183796,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1890844,"kb_used_data":1675924,"kb_used_omap":8,"kb_used_meta":214903,"kb_avail":91854308,"statfs":{"total":95995035648,"available":94058811392,"internally_reserved":0,"allocated":1716146176,"data_stored":1714646213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8453,"internal_metadata":220061435},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757643,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2211528,"kb_used_data":1576640,"kb_used_omap":1,"kb_used_meta":634878,"kb_avail":91533624,"statfs":{"total":95995035648,"available":93730430976,"internally_reserved":0,"allocated":1614479360,"data_stored":1612994483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":650115526},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019287,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3030420,"kb_used_data":2435724,"kb_used_omap":1,"kb_used_meta":594686,"kb_avail":90714732,"statfs":{"total":95995035648,"available":92891885568,"internally_reserved":0,"allocated":2494181376,"data_stored":2492689022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":608958918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435470,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3902084,"kb_used_data":3338684,"kb_used_omap":5,"kb_used_meta":563386,"kb_avail":89843068,"statfs":{"total":95995035648,"available":91999301632,"internally_reserved":0,"allocated":3418812416,"data_stored":3417331399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":576907390},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":446676598998,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":425201762524,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926048,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4224788,"kb_used_data":3565580,"kb_used_omap":7,"kb_used_meta":659192,"kb_avail":89520364,"statfs":{"total":95995035648,"available":91668852736,"internally_reserved":0,"allocated":3651153920,"data_stored":3649657686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":675012904},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6860,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3640201216,"data_stored":3640197144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3414036480,"data_stored":3414032408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2488926208,"data_stored":2488926208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1607335936,"data_stored":1607335936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1708265472,"data_stored":1708261400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2458189824,"data_stored":2458189824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:42:05.226 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:05.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:05.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:05.328 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:05.351 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:05.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:05.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:05.453 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:05.479 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:05.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:05.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:05.581 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:05.602 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:05.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:05.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:05.704 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:05.725 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:05.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:05.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:05.827 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:05.850 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:05.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:05.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:05.907 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:05 smithi102 ceph-mon[110183]: osdmap e278: 8 total, 8 up, 6 in 2024-09-13T14:42:05.907 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:05 smithi102 ceph-mon[115269]: osdmap e278: 8 total, 8 up, 6 in 2024-09-13T14:42:05.951 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:05.973 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:05.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:05.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:05.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:05 smithi195 ceph-mon[104262]: osdmap e278: 8 total, 8 up, 6 in 2024-09-13T14:42:06.075 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:06.100 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:06.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:06.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:06.202 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:06.229 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:06.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:06.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:06.331 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:06.354 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:06.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:06.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:06.457 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:06.479 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:06.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:06.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:06.581 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:07.009 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:06 smithi195 ceph-mon[104262]: pgmap v931: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 80 active+clean; 4.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 126 op/s; 20628/207840 objects degraded (9.925%); 3809/207840 objects misplaced (1.833%); 20 MiB/s, 313 objects/s recovering 2024-09-13T14:42:07.010 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:42:06 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:42:06] "GET /metrics HTTP/1.1" 200 36541 "" "Prometheus/2.51.0" 2024-09-13T14:42:07.010 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:06 smithi102 ceph-mon[110183]: pgmap v931: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 80 active+clean; 4.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 126 op/s; 20628/207840 objects degraded (9.925%); 3809/207840 objects misplaced (1.833%); 20 MiB/s, 313 objects/s recovering 2024-09-13T14:42:07.010 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:06 smithi102 ceph-mon[115269]: pgmap v931: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 80 active+clean; 4.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 126 op/s; 20628/207840 objects degraded (9.925%); 3809/207840 objects misplaced (1.833%); 20 MiB/s, 313 objects/s recovering 2024-09-13T14:42:07.032 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:07.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:07.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:07.134 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:07.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:07.157 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:07.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:07.258 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:07.283 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:07.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:07.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:07.385 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:07.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:07.410 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:07.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:07.511 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:07.534 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:07.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:07.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:07.636 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:07.658 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:07.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:07.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:07.760 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:07.782 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:07.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:07.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:07.884 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:07.907 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:07.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:07.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:08.009 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:08.031 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:08.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:08.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:08.133 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:08.153 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:08.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:08.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:08.214 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:42:08.255 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:08.286 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:08.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:08.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:08.387 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:08.410 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:08.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:08.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:08.464 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:42:08.465 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:42:08.483 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:42:08.483 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:42:08.500 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:42:08.500 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:42:08.513 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:08.518 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:42:08.518 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:42:08.537 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:08.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:08.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:08.543 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:42:08.544 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:42:08.565 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:42:08.565 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:42:08.592 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:42:08.592 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:42:08.616 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:42:08.617 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:42:08.638 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:08.659 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:08.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:08.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:08.761 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:08.784 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:08.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:08.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:08.885 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:08.906 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:08.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:08.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:08.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:08 smithi102 ceph-mon[110183]: pgmap v932: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 81 active+clean; 4.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 181 op/s; 20317/206370 objects degraded (9.845%); 3622/206370 objects misplaced (1.755%); 23 MiB/s, 375 objects/s recovering 2024-09-13T14:42:08.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:08 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/430211741' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:42:08.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:08 smithi102 ceph-mon[115269]: pgmap v932: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 81 active+clean; 4.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 181 op/s; 20317/206370 objects degraded (9.845%); 3622/206370 objects misplaced (1.755%); 23 MiB/s, 375 objects/s recovering 2024-09-13T14:42:08.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:08 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/430211741' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:42:08.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:08 smithi195 ceph-mon[104262]: pgmap v932: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 81 active+clean; 4.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 181 op/s; 20317/206370 objects degraded (9.845%); 3622/206370 objects misplaced (1.755%); 23 MiB/s, 375 objects/s recovering 2024-09-13T14:42:08.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:08 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/430211741' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:42:09.009 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:09.031 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:09.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:09.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:09.132 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:09.153 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:09.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:09.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:09.255 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:09.281 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:09.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:09.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:09.382 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:09.406 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:09.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:09.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:09.465 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:09.508 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:09.531 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:09.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:09.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:09.633 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:09.655 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:09.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:09.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:09.659 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is set 2024-09-13T14:42:09.721 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:09 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1809414718' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:42:09.721 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:09 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 20317/206370 objects degraded (9.845%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:42:09.721 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:09 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1809414718' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:42:09.721 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:09 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 20317/206370 objects degraded (9.845%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:42:09.756 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:09.781 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:09.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:09.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:09.883 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:09.907 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:09.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:09.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:09.918 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:09 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1809414718' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:42:09.918 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:09 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 20317/206370 objects degraded (9.845%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:42:10.009 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:10.031 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:10.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:10.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:10.133 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:10.158 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:10.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:10.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:10.260 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:10.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:10.281 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:10.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:10.382 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:10.404 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:10.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:10.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:10.505 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:10.529 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:10.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:10.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:10.631 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:10.654 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:10.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:10.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:10.702 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:10 smithi102 ceph-mon[110183]: pgmap v933: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 81 active+clean; 4.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 153 op/s; 20317/206214 objects degraded (9.852%); 3622/206214 objects misplaced (1.756%); 22 MiB/s, 357 objects/s recovering 2024-09-13T14:42:10.703 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:10 smithi102 ceph-mon[110183]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:42:10.703 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:10 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1809414718' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:42:10.703 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:10 smithi102 ceph-mon[110183]: osdmap e279: 8 total, 8 up, 6 in 2024-09-13T14:42:10.703 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:42:10.703 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:42:10.703 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:10 smithi102 ceph-mon[115269]: pgmap v933: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 81 active+clean; 4.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 153 op/s; 20317/206214 objects degraded (9.852%); 3622/206214 objects misplaced (1.756%); 22 MiB/s, 357 objects/s recovering 2024-09-13T14:42:10.703 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:10 smithi102 ceph-mon[115269]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:42:10.703 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:10 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1809414718' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:42:10.703 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:10 smithi102 ceph-mon[115269]: osdmap e279: 8 total, 8 up, 6 in 2024-09-13T14:42:10.703 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:42:10.703 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:42:10.756 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:10.783 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:10.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:10.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:10.885 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:10.909 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:10.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:10.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:10.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:10 smithi195 ceph-mon[104262]: pgmap v933: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 81 active+clean; 4.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 153 op/s; 20317/206214 objects degraded (9.852%); 3622/206214 objects misplaced (1.756%); 22 MiB/s, 357 objects/s recovering 2024-09-13T14:42:10.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:10 smithi195 ceph-mon[104262]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:42:10.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:10 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1809414718' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:42:10.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:10 smithi195 ceph-mon[104262]: osdmap e279: 8 total, 8 up, 6 in 2024-09-13T14:42:10.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:42:10.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:42:11.011 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:11.037 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:11.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:11.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:11.139 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:11.161 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:11.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:11.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:11.263 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:11.291 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:11.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:11.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:11.393 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:11.406 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:42:11.407 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [], 2024-09-13T14:42:11.407 INFO:teuthology.orchestra.run.smithi102.stdout: "num_ops": 0 2024-09-13T14:42:11.407 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:42:11.417 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:11.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:11.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:11.474 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:42:11.520 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:11.545 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:11.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:11.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:11.647 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:11.671 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:11.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:11.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:11.773 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:11.796 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:11.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:11.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:11.898 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:11.926 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:11.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:11.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:12.027 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:12.048 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:12.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:12.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:12.150 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:12.173 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:12.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:12.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:12.274 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:12.299 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:12.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:12.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:12.401 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:12.425 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:12.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:12.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:12.483 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:42:12.527 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:12.553 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:12.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:12.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:12.587 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T14:42:12.655 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:12.694 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:12.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:12.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:12.795 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:12.817 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:12.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:12.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:12.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:12 smithi102 ceph-mon[110183]: pgmap v935: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 81 active+clean; 4.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 139 op/s; 20156/205302 objects degraded (9.818%); 3622/205302 objects misplaced (1.764%); 6.2 MiB/s, 98 objects/s recovering 2024-09-13T14:42:12.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:12 smithi102 ceph-mon[115269]: pgmap v935: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 81 active+clean; 4.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 139 op/s; 20156/205302 objects degraded (9.818%); 3622/205302 objects misplaced (1.764%); 6.2 MiB/s, 98 objects/s recovering 2024-09-13T14:42:12.918 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:12.940 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:12.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:12.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:12.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:12 smithi195 ceph-mon[104262]: pgmap v935: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 81 active+clean; 4.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 139 op/s; 20156/205302 objects degraded (9.818%); 3622/205302 objects misplaced (1.764%); 6.2 MiB/s, 98 objects/s recovering 2024-09-13T14:42:13.042 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:13.066 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:13.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:13.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:13.167 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:13.189 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:13.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:13.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:13.290 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:13.316 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:13.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:13.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:13.417 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:13.438 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:13.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:13.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:13.540 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:13.563 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:13.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:13.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:13.664 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:13.685 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:13.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:13.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:13.714 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:13.788 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:13.816 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:13.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:13.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:13.886 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:13.918 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:13.939 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:13.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:13.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:14.041 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:14.065 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:14.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:14.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:14.167 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:14.188 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:14.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:14.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:14.287 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:14 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:42:14.287 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:14 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:42:14.287 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:14 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:42:14.288 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:14 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:42:14.288 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:14 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:42:14.288 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:14 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 20156/205302 objects degraded (9.818%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:42:14.288 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:14 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:42:14.288 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:14 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:42:14.288 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:14 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:42:14.288 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:14 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:42:14.288 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:14 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:42:14.288 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:14 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 20156/205302 objects degraded (9.818%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:42:14.290 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:14.311 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:14.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:14.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:14.413 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:14.444 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:14.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:14.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:14.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:14 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:42:14.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:14 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:42:14.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:14 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:42:14.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:14 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:42:14.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:14 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:42:14.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:14 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 20156/205302 objects degraded (9.818%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:42:14.546 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:14.570 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:14.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:14.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:14.672 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:14.695 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:14.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:14.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:14.797 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:14.820 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:14.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:14.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:14.922 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:14.945 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:14.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:14.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:15.047 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:15.069 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:15.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:15.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:15.171 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:15.193 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:15.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:15.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:15.294 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:15.315 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:15.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:15.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:15.417 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:15.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:15 smithi102 ceph-mon[110183]: pgmap v936: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 81 active+clean; 4.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 172 op/s; 20156/204009 objects degraded (9.880%); 3388/204009 objects misplaced (1.661%); 7.1 MiB/s, 114 objects/s recovering 2024-09-13T14:42:15.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:15 smithi102 ceph-mon[115269]: pgmap v936: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 81 active+clean; 4.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 172 op/s; 20156/204009 objects degraded (9.880%); 3388/204009 objects misplaced (1.661%); 7.1 MiB/s, 114 objects/s recovering 2024-09-13T14:42:15.440 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:15.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:15.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:15.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:15 smithi195 ceph-mon[104262]: pgmap v936: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 81 active+clean; 4.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 172 op/s; 20156/204009 objects degraded (9.880%); 3388/204009 objects misplaced (1.661%); 7.1 MiB/s, 114 objects/s recovering 2024-09-13T14:42:15.542 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:15.563 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:15.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:15.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:15.665 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:15.693 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:15.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:15.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:15.794 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:15.818 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:15.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:15.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:15.920 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:15.944 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:15.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:15.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:16.046 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:16.071 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:16.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:16.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:16.173 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:16.196 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:16.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:16.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:16.298 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:16.321 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:16.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:16.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:16.372 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:42:16.373 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:42:16.423 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:16.445 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:16.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:16.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:16.547 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:16.572 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:16.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:16.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:16.629 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:42:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:42:16] "GET /metrics HTTP/1.1" 200 36551 "" "Prometheus/2.51.0" 2024-09-13T14:42:16.674 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:16.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:16.697 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:16.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:16.799 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:16.820 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:16.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:16.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:16.922 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:16.945 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:16.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:16.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:17.047 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:17.073 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:17.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:17.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:17.175 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:17.196 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:17.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:17.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:17.297 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:17.322 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:17.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:17.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:17.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:17 smithi102 ceph-mon[110183]: pgmap v937: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 81 active+clean; 4.1 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 157 op/s; 19994/203112 objects degraded (9.844%); 3388/203112 objects misplaced (1.668%); 6.6 MiB/s, 105 objects/s recovering 2024-09-13T14:42:17.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:17 smithi102 ceph-mon[110183]: from='client.55447 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:42:17.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:17 smithi102 ceph-mon[115269]: pgmap v937: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 81 active+clean; 4.1 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 157 op/s; 19994/203112 objects degraded (9.844%); 3388/203112 objects misplaced (1.668%); 6.6 MiB/s, 105 objects/s recovering 2024-09-13T14:42:17.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:17 smithi102 ceph-mon[115269]: from='client.55447 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:42:17.424 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:17.446 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:17.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:17.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:17.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:17 smithi195 ceph-mon[104262]: pgmap v937: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 81 active+clean; 4.1 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 157 op/s; 19994/203112 objects degraded (9.844%); 3388/203112 objects misplaced (1.668%); 6.6 MiB/s, 105 objects/s recovering 2024-09-13T14:42:17.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:17 smithi195 ceph-mon[104262]: from='client.55447 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:42:17.549 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:17.571 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:17.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:17.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:17.673 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:17.696 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:17.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:17.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:17.798 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:17.821 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:17.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:17.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:17.923 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:17.950 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:17.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:17.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:18.052 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:18.075 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:18.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:18.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:18.177 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:18.200 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:18.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:18.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:18.302 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:18.323 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:18.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:18.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:18.425 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:18.449 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:18.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:18.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:18.550 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:18.578 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:18.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:18.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:18.680 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:18.705 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:18.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:18.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:18.807 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:18.828 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:18.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:18.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:18.930 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:18.955 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:18.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:18.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:19.057 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:19.083 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:19.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:19.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:19.113 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:42:19.113 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:42:19.133 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:42:19.133 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:42:19.153 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:42:19.153 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:42:19.181 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:42:19.182 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:42:19.185 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:19.202 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:42:19.202 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:42:19.209 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:19.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:19.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:19.225 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:42:19.225 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:42:19.225 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:19 smithi102 ceph-mon[110183]: pgmap v938: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 81 active+clean; 4.1 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 149 op/s; 19994/201870 objects degraded (9.904%); 3119/201870 objects misplaced (1.545%); 5.2 MiB/s, 82 objects/s recovering 2024-09-13T14:42:19.226 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:19 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 19994/201870 objects degraded (9.904%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:42:19.226 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:19 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/384834836' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:42:19.226 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:19 smithi102 ceph-mon[115269]: pgmap v938: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 81 active+clean; 4.1 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 149 op/s; 19994/201870 objects degraded (9.904%); 3119/201870 objects misplaced (1.545%); 5.2 MiB/s, 82 objects/s recovering 2024-09-13T14:42:19.226 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:19 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 19994/201870 objects degraded (9.904%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:42:19.226 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:19 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/384834836' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:42:19.244 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:42:19.245 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:42:19.263 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:42:19.263 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:42:19.311 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:19.334 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:19.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:19.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:19.436 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:19.457 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:19.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:19.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:19.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:19 smithi195 ceph-mon[104262]: pgmap v938: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 81 active+clean; 4.1 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 149 op/s; 19994/201870 objects degraded (9.904%); 3119/201870 objects misplaced (1.545%); 5.2 MiB/s, 82 objects/s recovering 2024-09-13T14:42:19.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:19 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 19994/201870 objects degraded (9.904%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:42:19.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:19 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/384834836' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:42:19.560 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:19.585 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:19.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:19.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:19.687 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:19.712 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:19.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:19.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:19.814 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:19.841 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:19.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:19.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:19.943 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:19.971 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:19.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:19.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:20.073 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:20.096 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:20.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:20.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:20.198 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:20.224 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:20.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:20.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:20.326 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:20.351 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:20.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:20.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:20.453 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:20.477 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:20.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:20.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:20.579 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:20.602 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:20.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:20.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:20.705 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:20.727 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:20.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:20.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:20.829 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:20.851 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:20.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:20.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:20.896 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":937,"stamp":"2024-09-13T14:42:15.509904+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4439939069,"num_objects":67704,"num_object_clones":0,"num_object_copies":203112,"num_objects_missing_on_primary":1587,"num_objects_missing":1587,"num_objects_degraded":19994,"num_objects_misplaced":3388,"num_objects_unfound":0,"num_objects_dirty":67704,"num_whiteouts":0,"num_read":6377,"num_read_kb":13834,"num_write":159073,"num_write_kb":7235010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70865,"num_bytes_recovered":4602203277,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94855,"ondisk_log_size":94855,"up":267,"acting":263,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":302,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":17888708,"kb_used_data":14747600,"kb_used_omap":26,"kb_used_meta":3141029,"kb_avail":544582204,"statfs":{"total":575970213888,"available":557652176896,"internally_reserved":0,"allocated":15101542400,"data_stored":15092520351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27144,"internal_metadata":3216414200},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,0,3,2,3,4,4,7],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-103284736,"num_objects":-1576,"num_object_clones":0,"num_object_copies":-4728,"num_objects_missing_on_primary":-421,"num_objects_missing":-421,"num_objects_degraded":-634,"num_objects_misplaced":-421,"num_objects_unfound":0,"num_objects_dirty":-1576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1576,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1054,"num_bytes_recovered":69074944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002906"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":451,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:42:09.674783+0000","last_change":"2024-09-13T14:36:13.859927+0000","last_active":"2024-09-13T14:42:09.674783+0000","last_peered":"2024-09-13T14:42:09.674783+0000","last_clean":"2024-09-13T14:42:09.674783+0000","last_became_active":"2024-09-13T14:36:13.859505+0000","last_became_peered":"2024-09-13T14:36:13.859505+0000","last_unstale":"2024-09-13T14:42:09.674783+0000","last_undegraded":"2024-09-13T14:42:09.674783+0000","last_fullsized":"2024-09-13T14:42:09.674783+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:12:20.989627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":281,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:42:04.639839+0000","last_change":"2024-09-13T14:37:48.130968+0000","last_active":"2024-09-13T14:42:04.639839+0000","last_peered":"2024-09-13T14:42:04.639839+0000","last_clean":"2024-09-13T14:42:04.639839+0000","last_became_active":"2024-09-13T14:37:48.130603+0000","last_became_peered":"2024-09-13T14:37:48.130603+0000","last_unstale":"2024-09-13T14:42:04.639839+0000","last_undegraded":"2024-09-13T14:42:04.639839+0000","last_fullsized":"2024-09-13T14:42:04.639839+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:42.626386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":77,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:42:04.652593+0000","last_change":"2024-09-13T14:36:13.862181+0000","last_active":"2024-09-13T14:42:04.652593+0000","last_peered":"2024-09-13T14:42:04.652593+0000","last_clean":"2024-09-13T14:42:04.652593+0000","last_became_active":"2024-09-13T14:36:13.861873+0000","last_became_peered":"2024-09-13T14:36:13.861873+0000","last_unstale":"2024-09-13T14:42:04.652593+0000","last_undegraded":"2024-09-13T14:42:04.652593+0000","last_fullsized":"2024-09-13T14:42:04.652593+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:14.085700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":88,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:42:09.710639+0000","last_change":"2024-09-13T14:38:53.896672+0000","last_active":"2024-09-13T14:42:09.710639+0000","last_peered":"2024-09-13T14:42:09.710639+0000","last_clean":"2024-09-13T14:42:09.710639+0000","last_became_active":"2024-09-13T14:38:53.896346+0000","last_became_peered":"2024-09-13T14:38:53.896346+0000","last_unstale":"2024-09-13T14:42:09.710639+0000","last_undegraded":"2024-09-13T14:42:09.710639+0000","last_fullsized":"2024-09-13T14:42:09.710639+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:57.789686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":88,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:42:09.711431+0000","last_change":"2024-09-13T14:38:53.893021+0000","last_active":"2024-09-13T14:42:09.711431+0000","last_peered":"2024-09-13T14:42:09.711431+0000","last_clean":"2024-09-13T14:42:09.711431+0000","last_became_active":"2024-09-13T14:38:53.892668+0000","last_became_peered":"2024-09-13T14:38:53.892668+0000","last_unstale":"2024-09-13T14:42:09.711431+0000","last_undegraded":"2024-09-13T14:42:09.711431+0000","last_fullsized":"2024-09-13T14:42:09.711431+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:57.180738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":249,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:42:04.639488+0000","last_change":"2024-09-13T14:37:48.134642+0000","last_active":"2024-09-13T14:42:04.639488+0000","last_peered":"2024-09-13T14:42:04.639488+0000","last_clean":"2024-09-13T14:42:04.639488+0000","last_became_active":"2024-09-13T14:37:48.134402+0000","last_became_peered":"2024-09-13T14:37:48.134402+0000","last_unstale":"2024-09-13T14:42:04.639488+0000","last_undegraded":"2024-09-13T14:42:04.639488+0000","last_fullsized":"2024-09-13T14:42:04.639488+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:51.313663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":470,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:42:09.675020+0000","last_change":"2024-09-13T14:38:53.903022+0000","last_active":"2024-09-13T14:42:09.675020+0000","last_peered":"2024-09-13T14:42:09.675020+0000","last_clean":"2024-09-13T14:42:09.675020+0000","last_became_active":"2024-09-13T14:38:53.902448+0000","last_became_peered":"2024-09-13T14:38:53.902448+0000","last_unstale":"2024-09-13T14:42:09.675020+0000","last_undegraded":"2024-09-13T14:42:09.675020+0000","last_fullsized":"2024-09-13T14:42:09.675020+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:24.690401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":91,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:42:09.710137+0000","last_change":"2024-09-13T14:38:03.554500+0000","last_active":"2024-09-13T14:42:09.710137+0000","last_peered":"2024-09-13T14:42:09.710137+0000","last_clean":"2024-09-13T14:42:09.710137+0000","last_became_active":"2024-09-13T14:38:03.532910+0000","last_became_peered":"2024-09-13T14:38:03.532910+0000","last_unstale":"2024-09-13T14:42:09.710137+0000","last_undegraded":"2024-09-13T14:42:09.710137+0000","last_fullsized":"2024-09-13T14:42:09.710137+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:27:50.843498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"279'9755","reported_seq":20645,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:42:11.896038+0000","last_change":"2024-09-13T14:41:53.220083+0000","last_active":"2024-09-13T14:42:11.896038+0000","last_peered":"2024-09-13T14:42:11.896038+0000","last_clean":"2024-09-13T14:42:11.896038+0000","last_became_active":"2024-09-13T14:41:53.200084+0000","last_became_peered":"2024-09-13T14:41:53.200084+0000","last_unstale":"2024-09-13T14:42:11.896038+0000","last_undegraded":"2024-09-13T14:42:11.896038+0000","last_fullsized":"2024-09-13T14:42:11.896038+0000","mapping_epoch":275,"log_start":"212'4000","ondisk_log_start":"212'4000","created":205,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5755,"log_dups_size":0,"ondisk_log_size":5755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:08.897524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":270336000,"num_objects":4125,"num_object_clones":0,"num_object_copies":12375,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9755,"num_write_kb":444160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9152,"num_bytes_recovered":594608128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":41,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:42:04.640325+0000","last_change":"2024-09-13T14:38:53.940238+0000","last_active":"2024-09-13T14:42:04.640325+0000","last_peered":"2024-09-13T14:42:04.640325+0000","last_clean":"2024-09-13T14:42:04.640325+0000","last_became_active":"2024-09-13T14:38:53.892825+0000","last_became_peered":"2024-09-13T14:38:53.892825+0000","last_unstale":"2024-09-13T14:42:04.640325+0000","last_undegraded":"2024-09-13T14:42:04.640325+0000","last_fullsized":"2024-09-13T14:42:04.640325+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:46.666581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":97,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:42:04.652160+0000","last_change":"2024-09-13T14:38:53.898907+0000","last_active":"2024-09-13T14:42:04.652160+0000","last_peered":"2024-09-13T14:42:04.652160+0000","last_clean":"2024-09-13T14:42:04.652160+0000","last_became_active":"2024-09-13T14:38:53.898572+0000","last_became_peered":"2024-09-13T14:38:53.898572+0000","last_unstale":"2024-09-13T14:42:04.652160+0000","last_undegraded":"2024-09-13T14:42:04.652160+0000","last_fullsized":"2024-09-13T14:42:04.652160+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:29.222150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"279'9998","reported_seq":24166,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:42:12.211837+0000","last_change":"2024-09-13T14:38:57.930032+0000","last_active":"2024-09-13T14:42:12.211837+0000","last_peered":"2024-09-13T14:42:12.211837+0000","last_clean":"2024-09-13T14:42:12.211837+0000","last_became_active":"2024-09-13T14:38:57.921458+0000","last_became_peered":"2024-09-13T14:38:57.921458+0000","last_unstale":"2024-09-13T14:42:12.211837+0000","last_undegraded":"2024-09-13T14:42:12.211837+0000","last_fullsized":"2024-09-13T14:42:12.211837+0000","mapping_epoch":256,"log_start":"213'4439","ondisk_log_start":"213'4439","created":205,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5559,"log_dups_size":0,"ondisk_log_size":5559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:27:12.294038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278134784,"num_objects":4244,"num_object_clones":0,"num_object_copies":12732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9998,"num_write_kb":455744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6990,"num_bytes_recovered":458096640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":41,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:42:04.640244+0000","last_change":"2024-09-13T14:38:53.904885+0000","last_active":"2024-09-13T14:42:04.640244+0000","last_peered":"2024-09-13T14:42:04.640244+0000","last_clean":"2024-09-13T14:42:04.640244+0000","last_became_active":"2024-09-13T14:38:53.891726+0000","last_became_peered":"2024-09-13T14:38:53.891726+0000","last_unstale":"2024-09-13T14:42:04.640244+0000","last_undegraded":"2024-09-13T14:42:04.640244+0000","last_fullsized":"2024-09-13T14:42:04.640244+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:57.079205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":287,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:42:04.637417+0000","last_change":"2024-09-13T14:37:48.160504+0000","last_active":"2024-09-13T14:42:04.637417+0000","last_peered":"2024-09-13T14:42:04.637417+0000","last_clean":"2024-09-13T14:42:04.637417+0000","last_became_active":"2024-09-13T14:37:48.160181+0000","last_became_peered":"2024-09-13T14:37:48.160181+0000","last_unstale":"2024-09-13T14:42:04.637417+0000","last_undegraded":"2024-09-13T14:42:04.637417+0000","last_fullsized":"2024-09-13T14:42:04.637417+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:37.087008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"279'9960","reported_seq":11749,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.836961+0000","last_change":"2024-09-13T14:38:04.531914+0000","last_active":"2024-09-13T14:42:14.836961+0000","last_peered":"2024-09-13T14:42:14.836961+0000","last_clean":"2024-09-13T14:42:14.836961+0000","last_became_active":"2024-09-13T14:38:04.523448+0000","last_became_peered":"2024-09-13T14:38:04.523448+0000","last_unstale":"2024-09-13T14:42:14.836961+0000","last_undegraded":"2024-09-13T14:42:14.836961+0000","last_fullsized":"2024-09-13T14:42:14.836961+0000","mapping_epoch":245,"log_start":"213'4000","ondisk_log_start":"213'4000","created":205,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5960,"log_dups_size":0,"ondisk_log_size":5960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:06.536896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276168704,"num_objects":4214,"num_object_clones":0,"num_object_copies":12642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9960,"num_write_kb":453568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":811,"num_bytes_recovered":53149696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":278,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:42:04.640107+0000","last_change":"2024-09-13T14:37:49.611774+0000","last_active":"2024-09-13T14:42:04.640107+0000","last_peered":"2024-09-13T14:42:04.640107+0000","last_clean":"2024-09-13T14:42:04.640107+0000","last_became_active":"2024-09-13T14:37:49.611088+0000","last_became_peered":"2024-09-13T14:37:49.611088+0000","last_unstale":"2024-09-13T14:42:04.640107+0000","last_undegraded":"2024-09-13T14:42:04.640107+0000","last_fullsized":"2024-09-13T14:42:04.640107+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:08.933213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":37,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:42:04.639473+0000","last_change":"2024-09-13T14:38:53.899211+0000","last_active":"2024-09-13T14:42:04.639473+0000","last_peered":"2024-09-13T14:42:04.639473+0000","last_clean":"2024-09-13T14:42:04.639473+0000","last_became_active":"2024-09-13T14:38:53.898927+0000","last_became_peered":"2024-09-13T14:38:53.898927+0000","last_unstale":"2024-09-13T14:42:04.639473+0000","last_undegraded":"2024-09-13T14:42:04.639473+0000","last_fullsized":"2024-09-13T14:42:04.639473+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:00:28.857611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"279'9776","reported_seq":9862,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:42:12.255169+0000","last_change":"2024-09-13T14:35:24.004651+0000","last_active":"2024-09-13T14:42:12.255169+0000","last_peered":"2024-09-13T14:42:12.255169+0000","last_clean":"2024-09-13T14:42:12.255169+0000","last_became_active":"2024-09-13T14:35:24.004358+0000","last_became_peered":"2024-09-13T14:35:24.004358+0000","last_unstale":"2024-09-13T14:42:12.255169+0000","last_undegraded":"2024-09-13T14:42:12.255169+0000","last_fullsized":"2024-09-13T14:42:12.255169+0000","mapping_epoch":205,"log_start":"212'3484","ondisk_log_start":"212'3484","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6292,"log_dups_size":0,"ondisk_log_size":6292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:57:38.023669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":274989056,"num_objects":4196,"num_object_clones":0,"num_object_copies":12588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9776,"num_write_kb":447104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":88,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:42:04.652726+0000","last_change":"2024-09-13T14:37:49.168003+0000","last_active":"2024-09-13T14:42:04.652726+0000","last_peered":"2024-09-13T14:42:04.652726+0000","last_clean":"2024-09-13T14:42:04.652726+0000","last_became_active":"2024-09-13T14:37:49.167638+0000","last_became_peered":"2024-09-13T14:37:49.167638+0000","last_unstale":"2024-09-13T14:42:04.652726+0000","last_undegraded":"2024-09-13T14:42:04.652726+0000","last_fullsized":"2024-09-13T14:42:04.652726+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:29.051981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":81,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:42:09.710825+0000","last_change":"2024-09-13T14:38:03.528381+0000","last_active":"2024-09-13T14:42:09.710825+0000","last_peered":"2024-09-13T14:42:09.710825+0000","last_clean":"2024-09-13T14:42:09.710825+0000","last_became_active":"2024-09-13T14:38:03.527282+0000","last_became_peered":"2024-09-13T14:38:03.527282+0000","last_unstale":"2024-09-13T14:42:09.710825+0000","last_undegraded":"2024-09-13T14:42:09.710825+0000","last_fullsized":"2024-09-13T14:42:09.710825+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:24.435067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.f","version":"279'9810","reported_seq":10093,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.855593+0000","last_change":"2024-09-13T14:38:16.371079+0000","last_active":"2024-09-13T14:42:14.855593+0000","last_peered":"2024-09-13T14:42:14.855593+0000","last_clean":"2024-09-13T14:42:14.855593+0000","last_became_active":"2024-09-13T14:38:12.859883+0000","last_became_peered":"2024-09-13T14:38:12.859883+0000","last_unstale":"2024-09-13T14:42:14.855593+0000","last_undegraded":"2024-09-13T14:42:14.855593+0000","last_fullsized":"2024-09-13T14:42:14.855593+0000","mapping_epoch":245,"log_start":"212'3863","ondisk_log_start":"212'3863","created":205,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5947,"log_dups_size":0,"ondisk_log_size":5947,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:08.436532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280231936,"num_objects":4276,"num_object_clones":0,"num_object_copies":12828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9810,"num_write_kb":450752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":3670016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":439,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:42:09.676477+0000","last_change":"2024-09-13T14:38:03.549595+0000","last_active":"2024-09-13T14:42:09.676477+0000","last_peered":"2024-09-13T14:42:09.676477+0000","last_clean":"2024-09-13T14:42:09.676477+0000","last_became_active":"2024-09-13T14:38:03.533331+0000","last_became_peered":"2024-09-13T14:38:03.533331+0000","last_unstale":"2024-09-13T14:42:09.676477+0000","last_undegraded":"2024-09-13T14:42:09.676477+0000","last_fullsized":"2024-09-13T14:42:09.676477+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:59:02.470343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":262,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:42:04.637613+0000","last_change":"2024-09-13T14:37:48.127600+0000","last_active":"2024-09-13T14:42:04.637613+0000","last_peered":"2024-09-13T14:42:04.637613+0000","last_clean":"2024-09-13T14:42:04.637613+0000","last_became_active":"2024-09-13T14:37:48.127349+0000","last_became_peered":"2024-09-13T14:37:48.127349+0000","last_unstale":"2024-09-13T14:42:04.637613+0000","last_undegraded":"2024-09-13T14:42:04.637613+0000","last_fullsized":"2024-09-13T14:42:04.637613+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:57.268374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"279'9837","reported_seq":16733,"reported_epoch":279,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T14:42:12.245707+0000","last_change":"2024-09-13T14:41:52.247680+0000","last_active":"2024-09-13T14:42:12.245707+0000","last_peered":"2024-09-13T14:42:12.245707+0000","last_clean":"2024-09-13T14:36:13.131873+0000","last_became_active":"2024-09-13T14:38:10.443954+0000","last_became_peered":"2024-09-13T14:38:10.443954+0000","last_unstale":"2024-09-13T14:42:12.245707+0000","last_undegraded":"2024-09-13T14:38:10.384643+0000","last_fullsized":"2024-09-13T14:38:04.517046+0000","mapping_epoch":245,"log_start":"213'4100","ondisk_log_start":"213'4100","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5737,"log_dups_size":0,"ondisk_log_size":5737,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:26:53.493170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":273481728,"num_objects":4173,"num_object_clones":0,"num_object_copies":12519,"num_objects_missing_on_primary":1397,"num_objects_missing":1397,"num_objects_degraded":1,"num_objects_misplaced":1401,"num_objects_unfound":0,"num_objects_dirty":4173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9837,"num_write_kb":448320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3388,"num_bytes_recovered":221970432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0],"avail_no_missing":["3"],"object_location_counts":[{"shards":"0,3","objects":1},{"shards":"0,3,7","objects":1396},{"shards":"0,5","objects":2771},{"shards":"3,5,7","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":195,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:42:04.637750+0000","last_change":"2024-09-13T14:37:48.127604+0000","last_active":"2024-09-13T14:42:04.637750+0000","last_peered":"2024-09-13T14:42:04.637750+0000","last_clean":"2024-09-13T14:42:04.637750+0000","last_became_active":"2024-09-13T14:37:48.127300+0000","last_became_peered":"2024-09-13T14:37:48.127300+0000","last_unstale":"2024-09-13T14:42:04.637750+0000","last_undegraded":"2024-09-13T14:42:04.637750+0000","last_fullsized":"2024-09-13T14:42:04.637750+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:51.999506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":315,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:42:04.637836+0000","last_change":"2024-09-13T14:38:03.529967+0000","last_active":"2024-09-13T14:42:04.637836+0000","last_peered":"2024-09-13T14:42:04.637836+0000","last_clean":"2024-09-13T14:42:04.637836+0000","last_became_active":"2024-09-13T14:38:03.529646+0000","last_became_peered":"2024-09-13T14:38:03.529646+0000","last_unstale":"2024-09-13T14:42:04.637836+0000","last_undegraded":"2024-09-13T14:42:04.637836+0000","last_fullsized":"2024-09-13T14:42:04.637836+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:52:05.193279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"279'9917","reported_seq":19077,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:42:14.938296+0000","last_change":"2024-09-13T14:40:34.583237+0000","last_active":"2024-09-13T14:42:14.938296+0000","last_peered":"2024-09-13T14:42:14.938296+0000","last_clean":"2024-09-13T14:42:14.938296+0000","last_became_active":"2024-09-13T14:40:34.563989+0000","last_became_peered":"2024-09-13T14:40:34.563989+0000","last_unstale":"2024-09-13T14:42:14.938296+0000","last_undegraded":"2024-09-13T14:42:14.938296+0000","last_fullsized":"2024-09-13T14:42:14.938296+0000","mapping_epoch":266,"log_start":"212'4000","ondisk_log_start":"212'4000","created":205,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5917,"log_dups_size":0,"ondisk_log_size":5917,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:34:43.006051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276693016,"num_objects":4223,"num_object_clones":0,"num_object_copies":12669,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4223,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9917,"num_write_kb":452417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4489,"num_bytes_recovered":289865752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":276,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:42:04.637669+0000","last_change":"2024-09-13T14:36:13.860367+0000","last_active":"2024-09-13T14:42:04.637669+0000","last_peered":"2024-09-13T14:42:04.637669+0000","last_clean":"2024-09-13T14:42:04.637669+0000","last_became_active":"2024-09-13T14:36:13.859982+0000","last_became_peered":"2024-09-13T14:36:13.859982+0000","last_unstale":"2024-09-13T14:42:04.637669+0000","last_undegraded":"2024-09-13T14:42:04.637669+0000","last_fullsized":"2024-09-13T14:42:04.637669+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:08:23.311350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":271,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:42:04.638028+0000","last_change":"2024-09-13T14:37:48.147360+0000","last_active":"2024-09-13T14:42:04.638028+0000","last_peered":"2024-09-13T14:42:04.638028+0000","last_clean":"2024-09-13T14:42:04.638028+0000","last_became_active":"2024-09-13T14:37:48.146885+0000","last_became_peered":"2024-09-13T14:37:48.146885+0000","last_unstale":"2024-09-13T14:42:04.638028+0000","last_undegraded":"2024-09-13T14:42:04.638028+0000","last_fullsized":"2024-09-13T14:42:04.638028+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:27.841810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"279'9793","reported_seq":19060,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:42:12.262346+0000","last_change":"2024-09-13T14:38:39.650332+0000","last_active":"2024-09-13T14:42:12.262346+0000","last_peered":"2024-09-13T14:42:12.262346+0000","last_clean":"2024-09-13T14:42:12.262346+0000","last_became_active":"2024-09-13T14:38:39.640835+0000","last_became_peered":"2024-09-13T14:38:39.640835+0000","last_unstale":"2024-09-13T14:42:12.262346+0000","last_undegraded":"2024-09-13T14:42:12.262346+0000","last_fullsized":"2024-09-13T14:42:12.262346+0000","mapping_epoch":251,"log_start":"213'4251","ondisk_log_start":"213'4251","created":205,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5542,"log_dups_size":0,"ondisk_log_size":5542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:48.176606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280559616,"num_objects":4281,"num_object_clones":0,"num_object_copies":12843,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9793,"num_write_kb":450368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4544,"num_bytes_recovered":292945920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":458,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:42:09.674642+0000","last_change":"2024-09-13T14:38:53.900041+0000","last_active":"2024-09-13T14:42:09.674642+0000","last_peered":"2024-09-13T14:42:09.674642+0000","last_clean":"2024-09-13T14:42:09.674642+0000","last_became_active":"2024-09-13T14:38:53.899203+0000","last_became_peered":"2024-09-13T14:38:53.899203+0000","last_unstale":"2024-09-13T14:42:09.674642+0000","last_undegraded":"2024-09-13T14:42:09.674642+0000","last_fullsized":"2024-09-13T14:42:09.674642+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:28:07.883094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":390,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:42:09.675167+0000","last_change":"2024-09-13T14:26:15.823198+0000","last_active":"2024-09-13T14:42:09.675167+0000","last_peered":"2024-09-13T14:42:09.675167+0000","last_clean":"2024-09-13T14:42:09.675167+0000","last_became_active":"2024-09-13T14:26:15.822717+0000","last_became_peered":"2024-09-13T14:26:15.822717+0000","last_unstale":"2024-09-13T14:42:09.675167+0000","last_undegraded":"2024-09-13T14:42:09.675167+0000","last_fullsized":"2024-09-13T14:42:09.675167+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:22.471622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.6","version":"279'9994","reported_seq":15747,"reported_epoch":279,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-13T14:42:14.927980+0000","last_change":"2024-09-13T14:38:55.132683+0000","last_active":"2024-09-13T14:42:14.927980+0000","last_peered":"2024-09-13T14:42:14.927980+0000","last_clean":"2024-09-13T14:36:13.124368+0000","last_became_active":"2024-09-13T14:38:55.120883+0000","last_became_peered":"2024-09-13T14:38:55.120883+0000","last_unstale":"2024-09-13T14:42:14.927980+0000","last_undegraded":"2024-09-13T14:38:55.026610+0000","last_fullsized":"2024-09-13T14:42:14.927980+0000","mapping_epoch":254,"log_start":"212'4019","ondisk_log_start":"212'4019","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5975,"log_dups_size":0,"ondisk_log_size":5975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":275513344,"num_objects":4204,"num_object_clones":0,"num_object_copies":12612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2217,"num_objects_misplaced":1987,"num_objects_unfound":0,"num_objects_dirty":4204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9994,"num_write_kb":454336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2788,"num_bytes_recovered":171507712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,3,5],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3,4","objects":1938},{"shards":"0,3,5","objects":2266}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":412,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:42:04.639465+0000","last_change":"2024-09-13T14:41:53.196835+0000","last_active":"2024-09-13T14:42:04.639465+0000","last_peered":"2024-09-13T14:42:04.639465+0000","last_clean":"2024-09-13T14:42:04.639465+0000","last_became_active":"2024-09-13T14:41:53.196387+0000","last_became_peered":"2024-09-13T14:41:53.196387+0000","last_unstale":"2024-09-13T14:42:04.639465+0000","last_undegraded":"2024-09-13T14:42:04.639465+0000","last_fullsized":"2024-09-13T14:42:04.639465+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:07:41.927740+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:49.972303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":517,"reported_epoch":278,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:42:04.652511+0000","last_change":"2024-09-13T14:38:54.904329+0000","last_active":"2024-09-13T14:42:04.652511+0000","last_peered":"2024-09-13T14:42:04.652511+0000","last_clean":"2024-09-13T14:38:39.648017+0000","last_became_active":"2024-09-13T14:38:54.903798+0000","last_became_peered":"2024-09-13T14:38:54.903798+0000","last_unstale":"2024-09-13T14:42:04.652511+0000","last_undegraded":"2024-09-13T14:38:54.890505+0000","last_fullsized":"2024-09-13T14:38:54.889701+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:18.120931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,3],"avail_no_missing":["5","3","6"],"object_location_counts":[{"shards":"3,5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":392,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:42:09.675121+0000","last_change":"2024-09-13T14:37:49.160719+0000","last_active":"2024-09-13T14:42:09.675121+0000","last_peered":"2024-09-13T14:42:09.675121+0000","last_clean":"2024-09-13T14:42:09.675121+0000","last_became_active":"2024-09-13T14:37:49.160401+0000","last_became_peered":"2024-09-13T14:37:49.160401+0000","last_unstale":"2024-09-13T14:42:09.675121+0000","last_undegraded":"2024-09-13T14:42:09.675121+0000","last_fullsized":"2024-09-13T14:42:09.675121+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:21.737945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":122,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:42:04.637805+0000","last_change":"2024-09-13T14:37:48.172940+0000","last_active":"2024-09-13T14:42:04.637805+0000","last_peered":"2024-09-13T14:42:04.637805+0000","last_clean":"2024-09-13T14:42:04.637805+0000","last_became_active":"2024-09-13T14:37:48.172060+0000","last_became_peered":"2024-09-13T14:37:48.172060+0000","last_unstale":"2024-09-13T14:42:04.637805+0000","last_undegraded":"2024-09-13T14:42:04.637805+0000","last_fullsized":"2024-09-13T14:42:04.637805+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:07.480645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.5","version":"279'9848","reported_seq":26210,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:42:12.197051+0000","last_change":"2024-09-13T14:39:34.446699+0000","last_active":"2024-09-13T14:42:12.197051+0000","last_peered":"2024-09-13T14:42:12.197051+0000","last_clean":"2024-09-13T14:42:12.197051+0000","last_became_active":"2024-09-13T14:38:04.600616+0000","last_became_peered":"2024-09-13T14:38:04.600616+0000","last_unstale":"2024-09-13T14:42:12.197051+0000","last_undegraded":"2024-09-13T14:42:12.197051+0000","last_fullsized":"2024-09-13T14:42:12.197051+0000","mapping_epoch":245,"log_start":"212'3516","ondisk_log_start":"212'3516","created":205,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6332,"log_dups_size":0,"ondisk_log_size":6332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:00:44.321678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281411584,"num_objects":4294,"num_object_clones":0,"num_object_copies":12882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9848,"num_write_kb":452544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8120,"num_bytes_recovered":521076736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":495,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:42:04.639315+0000","last_change":"2024-09-13T14:26:15.835178+0000","last_active":"2024-09-13T14:42:04.639315+0000","last_peered":"2024-09-13T14:42:04.639315+0000","last_clean":"2024-09-13T14:42:04.639315+0000","last_became_active":"2024-09-13T14:26:15.820283+0000","last_became_peered":"2024-09-13T14:26:15.820283+0000","last_unstale":"2024-09-13T14:42:04.639315+0000","last_undegraded":"2024-09-13T14:42:04.639315+0000","last_fullsized":"2024-09-13T14:42:04.639315+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:30.340038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":81,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:42:09.710224+0000","last_change":"2024-09-13T14:38:03.525682+0000","last_active":"2024-09-13T14:42:09.710224+0000","last_peered":"2024-09-13T14:42:09.710224+0000","last_clean":"2024-09-13T14:42:09.710224+0000","last_became_active":"2024-09-13T14:38:03.525406+0000","last_became_peered":"2024-09-13T14:38:03.525406+0000","last_unstale":"2024-09-13T14:42:09.710224+0000","last_undegraded":"2024-09-13T14:42:09.710224+0000","last_fullsized":"2024-09-13T14:42:09.710224+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:41:28.970300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":88,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:42:09.711487+0000","last_change":"2024-09-13T14:38:53.991738+0000","last_active":"2024-09-13T14:42:09.711487+0000","last_peered":"2024-09-13T14:42:09.711487+0000","last_clean":"2024-09-13T14:42:09.711487+0000","last_became_active":"2024-09-13T14:38:53.991417+0000","last_became_peered":"2024-09-13T14:38:53.991417+0000","last_unstale":"2024-09-13T14:42:09.711487+0000","last_undegraded":"2024-09-13T14:42:09.711487+0000","last_fullsized":"2024-09-13T14:42:09.711487+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:50:38.559838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"279'9899","reported_seq":24493,"reported_epoch":279,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-13T14:42:15.102442+0000","last_change":"2024-09-13T14:38:17.472071+0000","last_active":"2024-09-13T14:42:15.102442+0000","last_peered":"2024-09-13T14:42:15.102442+0000","last_clean":"2024-09-13T14:37:46.290754+0000","last_became_active":"2024-09-13T14:38:17.381753+0000","last_became_peered":"2024-09-13T14:38:17.381753+0000","last_unstale":"2024-09-13T14:42:15.102442+0000","last_undegraded":"2024-09-13T14:38:13.504140+0000","last_fullsized":"2024-09-13T14:42:15.102442+0000","mapping_epoch":245,"log_start":"212'3959","ondisk_log_start":"212'3959","created":205,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5940,"log_dups_size":0,"ondisk_log_size":5940,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":275709952,"num_objects":4207,"num_object_clones":0,"num_object_copies":12621,"num_objects_missing_on_primary":83,"num_objects_missing":83,"num_objects_degraded":4375,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9899,"num_write_kb":451392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7260,"num_bytes_recovered":475791360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"0,4,5","objects":4206},{"shards":"0,4,6","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":803,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:42:09.674402+0000","last_change":"2024-09-13T14:37:48.131799+0000","last_active":"2024-09-13T14:42:09.674402+0000","last_peered":"2024-09-13T14:42:09.674402+0000","last_clean":"2024-09-13T14:42:09.674402+0000","last_became_active":"2024-09-13T14:37:48.130837+0000","last_became_peered":"2024-09-13T14:37:48.130837+0000","last_unstale":"2024-09-13T14:42:09.674402+0000","last_undegraded":"2024-09-13T14:42:09.674402+0000","last_fullsized":"2024-09-13T14:42:09.674402+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:06:32.663770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":322,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:42:09.710312+0000","last_change":"2024-09-13T14:37:49.301531+0000","last_active":"2024-09-13T14:42:09.710312+0000","last_peered":"2024-09-13T14:42:09.710312+0000","last_clean":"2024-09-13T14:42:09.710312+0000","last_became_active":"2024-09-13T14:37:49.301249+0000","last_became_peered":"2024-09-13T14:37:49.301249+0000","last_unstale":"2024-09-13T14:42:09.710312+0000","last_undegraded":"2024-09-13T14:42:09.710312+0000","last_fullsized":"2024-09-13T14:42:09.710312+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:10.977710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":104,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:42:04.652819+0000","last_change":"2024-09-13T14:38:53.901427+0000","last_active":"2024-09-13T14:42:04.652819+0000","last_peered":"2024-09-13T14:42:04.652819+0000","last_clean":"2024-09-13T14:42:04.652819+0000","last_became_active":"2024-09-13T14:38:53.901157+0000","last_became_peered":"2024-09-13T14:38:53.901157+0000","last_unstale":"2024-09-13T14:42:04.652819+0000","last_undegraded":"2024-09-13T14:42:04.652819+0000","last_fullsized":"2024-09-13T14:42:04.652819+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:28:21.713025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"279'9810","reported_seq":24317,"reported_epoch":279,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-13T14:42:15.142913+0000","last_change":"2024-09-13T14:38:16.552896+0000","last_active":"2024-09-13T14:42:15.142913+0000","last_peered":"2024-09-13T14:42:15.142913+0000","last_clean":"2024-09-13T14:37:46.405415+0000","last_became_active":"2024-09-13T14:38:16.427510+0000","last_became_peered":"2024-09-13T14:38:16.427510+0000","last_unstale":"2024-09-13T14:42:15.142913+0000","last_undegraded":"2024-09-13T14:38:12.738464+0000","last_fullsized":"2024-09-13T14:42:15.142913+0000","mapping_epoch":245,"log_start":"212'3918","ondisk_log_start":"212'3918","created":205,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5892,"log_dups_size":0,"ondisk_log_size":5892,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":274464768,"num_objects":4188,"num_object_clones":0,"num_object_copies":12564,"num_objects_missing_on_primary":100,"num_objects_missing":100,"num_objects_degraded":4354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9810,"num_write_kb":447936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7217,"num_bytes_recovered":472973312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"0,3,6","objects":1},{"shards":"0,3,7","objects":4187}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":986,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:42:04.652316+0000","last_change":"2024-09-13T14:37:49.147498+0000","last_active":"2024-09-13T14:42:04.652316+0000","last_peered":"2024-09-13T14:42:04.652316+0000","last_clean":"2024-09-13T14:42:04.652316+0000","last_became_active":"2024-09-13T14:37:49.147028+0000","last_became_peered":"2024-09-13T14:37:49.147028+0000","last_unstale":"2024-09-13T14:42:04.652316+0000","last_undegraded":"2024-09-13T14:42:04.652316+0000","last_fullsized":"2024-09-13T14:42:04.652316+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:11:29.050178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":284,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:42:04.638134+0000","last_change":"2024-09-13T14:38:03.528133+0000","last_active":"2024-09-13T14:42:04.638134+0000","last_peered":"2024-09-13T14:42:04.638134+0000","last_clean":"2024-09-13T14:42:04.638134+0000","last_became_active":"2024-09-13T14:38:03.527829+0000","last_became_peered":"2024-09-13T14:38:03.527829+0000","last_unstale":"2024-09-13T14:42:04.638134+0000","last_undegraded":"2024-09-13T14:42:04.638134+0000","last_fullsized":"2024-09-13T14:42:04.638134+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:48.254169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":442,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:42:09.674693+0000","last_change":"2024-09-13T14:38:53.899755+0000","last_active":"2024-09-13T14:42:09.674693+0000","last_peered":"2024-09-13T14:42:09.674693+0000","last_clean":"2024-09-13T14:42:09.674693+0000","last_became_active":"2024-09-13T14:38:53.899448+0000","last_became_peered":"2024-09-13T14:38:53.899448+0000","last_unstale":"2024-09-13T14:42:09.674693+0000","last_undegraded":"2024-09-13T14:42:09.674693+0000","last_fullsized":"2024-09-13T14:42:09.674693+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:55.897035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"279'10052","reported_seq":10169,"reported_epoch":279,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:42:12.612125+0000","last_change":"2024-09-13T14:41:52.203221+0000","last_active":"2024-09-13T14:42:12.612125+0000","last_peered":"2024-09-13T14:42:12.612125+0000","last_clean":"2024-09-13T14:35:51.090791+0000","last_became_active":"2024-09-13T14:37:49.665025+0000","last_became_peered":"2024-09-13T14:37:49.665025+0000","last_unstale":"2024-09-13T14:42:12.612125+0000","last_undegraded":"2024-09-13T14:37:49.169101+0000","last_fullsized":"2024-09-13T14:37:49.121596+0000","mapping_epoch":240,"log_start":"212'3863","ondisk_log_start":"212'3863","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6189,"log_dups_size":0,"ondisk_log_size":6189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:02.321504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278528000,"num_objects":4250,"num_object_clones":0,"num_object_copies":12750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4496,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10052,"num_write_kb":457664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":393216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":-180}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":88,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:42:04.652658+0000","last_change":"2024-09-13T14:37:49.149770+0000","last_active":"2024-09-13T14:42:04.652658+0000","last_peered":"2024-09-13T14:42:04.652658+0000","last_clean":"2024-09-13T14:42:04.652658+0000","last_became_active":"2024-09-13T14:37:49.148991+0000","last_became_peered":"2024-09-13T14:37:49.148991+0000","last_unstale":"2024-09-13T14:42:04.652658+0000","last_undegraded":"2024-09-13T14:42:04.652658+0000","last_fullsized":"2024-09-13T14:42:04.652658+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:04:04.193366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":1140,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:42:09.737094+0000","last_change":"2024-09-13T14:30:58.928302+0000","last_active":"2024-09-13T14:42:09.737094+0000","last_peered":"2024-09-13T14:42:09.737094+0000","last_clean":"2024-09-13T14:42:09.737094+0000","last_became_active":"2024-09-13T14:30:58.927718+0000","last_became_peered":"2024-09-13T14:30:58.927718+0000","last_unstale":"2024-09-13T14:42:09.737094+0000","last_undegraded":"2024-09-13T14:42:09.737094+0000","last_fullsized":"2024-09-13T14:42:09.737094+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:37.323036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":95,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:42:04.652388+0000","last_change":"2024-09-13T14:38:03.534628+0000","last_active":"2024-09-13T14:42:04.652388+0000","last_peered":"2024-09-13T14:42:04.652388+0000","last_clean":"2024-09-13T14:42:04.652388+0000","last_became_active":"2024-09-13T14:38:03.534262+0000","last_became_peered":"2024-09-13T14:38:03.534262+0000","last_unstale":"2024-09-13T14:42:04.652388+0000","last_undegraded":"2024-09-13T14:42:04.652388+0000","last_fullsized":"2024-09-13T14:42:04.652388+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:31.168783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"279'9780","reported_seq":23946,"reported_epoch":279,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:42:11.938197+0000","last_change":"2024-09-13T14:38:53.944111+0000","last_active":"2024-09-13T14:42:11.938197+0000","last_peered":"2024-09-13T14:42:11.938197+0000","last_clean":"2024-09-13T14:37:47.047672+0000","last_became_active":"2024-09-13T14:37:49.664294+0000","last_became_peered":"2024-09-13T14:37:49.664294+0000","last_unstale":"2024-09-13T14:42:11.938197+0000","last_undegraded":"2024-09-13T14:38:53.940822+0000","last_fullsized":"2024-09-13T14:37:49.126627+0000","mapping_epoch":240,"log_start":"213'4100","ondisk_log_start":"213'4100","created":205,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5680,"log_dups_size":0,"ondisk_log_size":5680,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:11.766692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279969792,"num_objects":4272,"num_object_clones":0,"num_object_copies":12816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4535,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9780,"num_write_kb":449664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7009,"num_bytes_recovered":459341824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3],"avail_no_missing":["5","0","3"],"object_location_counts":[{"shards":"0,3,5","objects":4272},{"shards":"3,5","objects":-188}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":107,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:42:04.652620+0000","last_change":"2024-09-13T14:38:03.534303+0000","last_active":"2024-09-13T14:42:04.652620+0000","last_peered":"2024-09-13T14:42:04.652620+0000","last_clean":"2024-09-13T14:42:04.652620+0000","last_became_active":"2024-09-13T14:38:03.533171+0000","last_became_peered":"2024-09-13T14:38:03.533171+0000","last_unstale":"2024-09-13T14:42:04.652620+0000","last_undegraded":"2024-09-13T14:42:04.652620+0000","last_fullsized":"2024-09-13T14:42:04.652620+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:00:16.553526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":800,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:42:04.640154+0000","last_change":"2024-09-13T14:38:04.596192+0000","last_active":"2024-09-13T14:42:04.640154+0000","last_peered":"2024-09-13T14:42:04.640154+0000","last_clean":"2024-09-13T14:42:04.640154+0000","last_became_active":"2024-09-13T14:38:04.593299+0000","last_became_peered":"2024-09-13T14:38:04.593299+0000","last_unstale":"2024-09-13T14:42:04.640154+0000","last_undegraded":"2024-09-13T14:42:04.640154+0000","last_fullsized":"2024-09-13T14:42:04.640154+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:34.241410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":81,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:42:09.710910+0000","last_change":"2024-09-13T14:38:03.533120+0000","last_active":"2024-09-13T14:42:09.710910+0000","last_peered":"2024-09-13T14:42:09.710910+0000","last_clean":"2024-09-13T14:42:09.710910+0000","last_became_active":"2024-09-13T14:38:03.532829+0000","last_became_peered":"2024-09-13T14:38:03.532829+0000","last_unstale":"2024-09-13T14:42:09.710910+0000","last_undegraded":"2024-09-13T14:42:09.710910+0000","last_fullsized":"2024-09-13T14:42:09.710910+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:24.514661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"279'10013","reported_seq":19259,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:42:12.169627+0000","last_change":"2024-09-13T14:42:04.660877+0000","last_active":"2024-09-13T14:42:12.169627+0000","last_peered":"2024-09-13T14:42:12.169627+0000","last_clean":"2024-09-13T14:42:12.169627+0000","last_became_active":"2024-09-13T14:42:04.641680+0000","last_became_peered":"2024-09-13T14:42:04.641680+0000","last_unstale":"2024-09-13T14:42:12.169627+0000","last_undegraded":"2024-09-13T14:42:12.169627+0000","last_fullsized":"2024-09-13T14:42:12.169627+0000","mapping_epoch":277,"log_start":"213'4518","ondisk_log_start":"213'4518","created":205,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5495,"log_dups_size":0,"ondisk_log_size":5495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:22:18.955428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287375360,"num_objects":4385,"num_object_clones":0,"num_object_copies":13155,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10013,"num_write_kb":460736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4533,"num_bytes_recovered":297074688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":484,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:42:09.675273+0000","last_change":"2024-09-13T14:38:53.938227+0000","last_active":"2024-09-13T14:42:09.675273+0000","last_peered":"2024-09-13T14:42:09.675273+0000","last_clean":"2024-09-13T14:42:09.675273+0000","last_became_active":"2024-09-13T14:38:53.902867+0000","last_became_peered":"2024-09-13T14:38:53.902867+0000","last_unstale":"2024-09-13T14:42:09.675273+0000","last_undegraded":"2024-09-13T14:42:09.675273+0000","last_fullsized":"2024-09-13T14:42:09.675273+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:16.831756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":578,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:42:04.638215+0000","last_change":"2024-09-13T14:37:48.147677+0000","last_active":"2024-09-13T14:42:04.638215+0000","last_peered":"2024-09-13T14:42:04.638215+0000","last_clean":"2024-09-13T14:42:04.638215+0000","last_became_active":"2024-09-13T14:37:48.147001+0000","last_became_peered":"2024-09-13T14:37:48.147001+0000","last_unstale":"2024-09-13T14:42:04.638215+0000","last_undegraded":"2024-09-13T14:42:04.638215+0000","last_fullsized":"2024-09-13T14:42:04.638215+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:35:00.653576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":465,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:42:09.674425+0000","last_change":"2024-09-13T14:38:53.900107+0000","last_active":"2024-09-13T14:42:09.674425+0000","last_peered":"2024-09-13T14:42:09.674425+0000","last_clean":"2024-09-13T14:42:09.674425+0000","last_became_active":"2024-09-13T14:38:53.899556+0000","last_became_peered":"2024-09-13T14:38:53.899556+0000","last_unstale":"2024-09-13T14:42:09.674425+0000","last_undegraded":"2024-09-13T14:42:09.674425+0000","last_fullsized":"2024-09-13T14:42:09.674425+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:15:47.918163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"279'9708","reported_seq":18680,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:42:12.280286+0000","last_change":"2024-09-13T14:39:54.921239+0000","last_active":"2024-09-13T14:42:12.280286+0000","last_peered":"2024-09-13T14:42:12.280286+0000","last_clean":"2024-09-13T14:42:12.280286+0000","last_became_active":"2024-09-13T14:39:54.907894+0000","last_became_peered":"2024-09-13T14:39:54.907894+0000","last_unstale":"2024-09-13T14:42:12.280286+0000","last_undegraded":"2024-09-13T14:42:12.280286+0000","last_fullsized":"2024-09-13T14:42:12.280286+0000","mapping_epoch":263,"log_start":"213'4247","ondisk_log_start":"213'4247","created":205,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5461,"log_dups_size":0,"ondisk_log_size":5461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:52:04.619238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271581184,"num_objects":4144,"num_object_clones":0,"num_object_copies":12432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9708,"num_write_kb":443264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4395,"num_bytes_recovered":283770880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":96,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:42:09.711392+0000","last_change":"2024-09-13T14:38:53.901806+0000","last_active":"2024-09-13T14:42:09.711392+0000","last_peered":"2024-09-13T14:42:09.711392+0000","last_clean":"2024-09-13T14:42:09.711392+0000","last_became_active":"2024-09-13T14:38:53.891055+0000","last_became_peered":"2024-09-13T14:38:53.891055+0000","last_unstale":"2024-09-13T14:42:09.711392+0000","last_undegraded":"2024-09-13T14:42:09.711392+0000","last_fullsized":"2024-09-13T14:42:09.711392+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:03:00.416198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":35,"reported_epoch":278,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-13T14:42:04.639191+0000","last_change":"2024-09-13T14:38:53.900902+0000","last_active":"2024-09-13T14:42:04.639191+0000","last_peered":"2024-09-13T14:42:04.639191+0000","last_clean":"2024-09-13T14:38:39.635779+0000","last_became_active":"2024-09-13T14:38:53.900626+0000","last_became_peered":"2024-09-13T14:38:53.900626+0000","last_unstale":"2024-09-13T14:42:04.639191+0000","last_undegraded":"2024-09-13T14:38:53.885778+0000","last_fullsized":"2024-09-13T14:42:04.639191+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:37:21.715778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":7,"num_objects_missing":7,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":["3","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":268,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:42:04.637547+0000","last_change":"2024-09-13T14:37:49.172218+0000","last_active":"2024-09-13T14:42:04.637547+0000","last_peered":"2024-09-13T14:42:04.637547+0000","last_clean":"2024-09-13T14:42:04.637547+0000","last_became_active":"2024-09-13T14:37:49.171718+0000","last_became_peered":"2024-09-13T14:37:49.171718+0000","last_unstale":"2024-09-13T14:42:04.637547+0000","last_undegraded":"2024-09-13T14:42:04.637547+0000","last_fullsized":"2024-09-13T14:42:04.637547+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:46:54.626521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":291,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:42:04.637962+0000","last_change":"2024-09-13T14:37:49.230094+0000","last_active":"2024-09-13T14:42:04.637962+0000","last_peered":"2024-09-13T14:42:04.637962+0000","last_clean":"2024-09-13T14:42:04.637962+0000","last_became_active":"2024-09-13T14:37:49.229393+0000","last_became_peered":"2024-09-13T14:37:49.229393+0000","last_unstale":"2024-09-13T14:42:04.637962+0000","last_undegraded":"2024-09-13T14:42:04.637962+0000","last_fullsized":"2024-09-13T14:42:04.637962+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:02:28.690318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":271,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:42:04.637994+0000","last_change":"2024-09-13T14:37:48.128447+0000","last_active":"2024-09-13T14:42:04.637994+0000","last_peered":"2024-09-13T14:42:04.637994+0000","last_clean":"2024-09-13T14:42:04.637994+0000","last_became_active":"2024-09-13T14:37:48.128143+0000","last_became_peered":"2024-09-13T14:37:48.128143+0000","last_unstale":"2024-09-13T14:42:04.637994+0000","last_undegraded":"2024-09-13T14:42:04.637994+0000","last_fullsized":"2024-09-13T14:42:04.637994+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:22.204039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":81,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:42:09.710637+0000","last_change":"2024-09-13T14:38:03.531683+0000","last_active":"2024-09-13T14:42:09.710637+0000","last_peered":"2024-09-13T14:42:09.710637+0000","last_clean":"2024-09-13T14:42:09.710637+0000","last_became_active":"2024-09-13T14:38:03.531476+0000","last_became_peered":"2024-09-13T14:38:03.531476+0000","last_unstale":"2024-09-13T14:42:09.710637+0000","last_undegraded":"2024-09-13T14:42:09.710637+0000","last_fullsized":"2024-09-13T14:42:09.710637+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:41.557569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":43,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:42:04.639944+0000","last_change":"2024-09-13T14:38:53.954281+0000","last_active":"2024-09-13T14:42:04.639944+0000","last_peered":"2024-09-13T14:42:04.639944+0000","last_clean":"2024-09-13T14:42:04.639944+0000","last_became_active":"2024-09-13T14:38:53.900912+0000","last_became_peered":"2024-09-13T14:38:53.900912+0000","last_unstale":"2024-09-13T14:42:04.639944+0000","last_undegraded":"2024-09-13T14:42:04.639944+0000","last_fullsized":"2024-09-13T14:42:04.639944+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:47.876275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":95,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:42:04.656016+0000","last_change":"2024-09-13T14:38:54.000782+0000","last_active":"2024-09-13T14:42:04.656016+0000","last_peered":"2024-09-13T14:42:04.656016+0000","last_clean":"2024-09-13T14:42:04.656016+0000","last_became_active":"2024-09-13T14:38:54.000322+0000","last_became_peered":"2024-09-13T14:38:54.000322+0000","last_unstale":"2024-09-13T14:42:04.656016+0000","last_undegraded":"2024-09-13T14:42:04.656016+0000","last_fullsized":"2024-09-13T14:42:04.656016+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:09.057597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":407,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:42:09.674119+0000","last_change":"2024-09-13T14:38:53.899535+0000","last_active":"2024-09-13T14:42:09.674119+0000","last_peered":"2024-09-13T14:42:09.674119+0000","last_clean":"2024-09-13T14:42:09.674119+0000","last_became_active":"2024-09-13T14:38:53.899183+0000","last_became_peered":"2024-09-13T14:38:53.899183+0000","last_unstale":"2024-09-13T14:42:09.674119+0000","last_undegraded":"2024-09-13T14:42:09.674119+0000","last_fullsized":"2024-09-13T14:42:09.674119+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:33.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":81,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:42:09.730471+0000","last_change":"2024-09-13T14:38:03.543426+0000","last_active":"2024-09-13T14:42:09.730471+0000","last_peered":"2024-09-13T14:42:09.730471+0000","last_clean":"2024-09-13T14:42:09.730471+0000","last_became_active":"2024-09-13T14:38:03.542661+0000","last_became_peered":"2024-09-13T14:38:03.542661+0000","last_unstale":"2024-09-13T14:42:09.730471+0000","last_undegraded":"2024-09-13T14:42:09.730471+0000","last_fullsized":"2024-09-13T14:42:09.730471+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:54.050127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":436,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:42:09.674345+0000","last_change":"2024-09-13T14:37:48.131321+0000","last_active":"2024-09-13T14:42:09.674345+0000","last_peered":"2024-09-13T14:42:09.674345+0000","last_clean":"2024-09-13T14:42:09.674345+0000","last_became_active":"2024-09-13T14:37:48.130998+0000","last_became_peered":"2024-09-13T14:37:48.130998+0000","last_unstale":"2024-09-13T14:42:09.674345+0000","last_undegraded":"2024-09-13T14:42:09.674345+0000","last_fullsized":"2024-09-13T14:42:09.674345+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:33:26.597825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":107,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:42:04.652785+0000","last_change":"2024-09-13T14:38:03.535454+0000","last_active":"2024-09-13T14:42:04.652785+0000","last_peered":"2024-09-13T14:42:04.652785+0000","last_clean":"2024-09-13T14:42:04.652785+0000","last_became_active":"2024-09-13T14:38:03.535131+0000","last_became_peered":"2024-09-13T14:38:03.535131+0000","last_unstale":"2024-09-13T14:42:04.652785+0000","last_undegraded":"2024-09-13T14:42:04.652785+0000","last_fullsized":"2024-09-13T14:42:04.652785+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:24.477601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":461,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:42:09.674508+0000","last_change":"2024-09-13T14:38:03.524939+0000","last_active":"2024-09-13T14:42:09.674508+0000","last_peered":"2024-09-13T14:42:09.674508+0000","last_clean":"2024-09-13T14:42:09.674508+0000","last_became_active":"2024-09-13T14:38:03.524406+0000","last_became_peered":"2024-09-13T14:38:03.524406+0000","last_unstale":"2024-09-13T14:42:09.674508+0000","last_undegraded":"2024-09-13T14:42:09.674508+0000","last_fullsized":"2024-09-13T14:42:09.674508+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:19:45.257424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":88,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:42:09.711337+0000","last_change":"2024-09-13T14:38:53.892948+0000","last_active":"2024-09-13T14:42:09.711337+0000","last_peered":"2024-09-13T14:42:09.711337+0000","last_clean":"2024-09-13T14:42:09.711337+0000","last_became_active":"2024-09-13T14:38:53.892665+0000","last_became_peered":"2024-09-13T14:38:53.892665+0000","last_unstale":"2024-09-13T14:42:09.711337+0000","last_undegraded":"2024-09-13T14:42:09.711337+0000","last_fullsized":"2024-09-13T14:42:09.711337+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:08.292359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":409,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:42:09.674316+0000","last_change":"2024-09-13T14:37:48.139348+0000","last_active":"2024-09-13T14:42:09.674316+0000","last_peered":"2024-09-13T14:42:09.674316+0000","last_clean":"2024-09-13T14:42:09.674316+0000","last_became_active":"2024-09-13T14:37:48.139115+0000","last_became_peered":"2024-09-13T14:37:48.139115+0000","last_unstale":"2024-09-13T14:42:09.674316+0000","last_undegraded":"2024-09-13T14:42:09.674316+0000","last_fullsized":"2024-09-13T14:42:09.674316+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:32:20.413346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":449,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:42:09.674782+0000","last_change":"2024-09-13T14:37:48.172088+0000","last_active":"2024-09-13T14:42:09.674782+0000","last_peered":"2024-09-13T14:42:09.674782+0000","last_clean":"2024-09-13T14:42:09.674782+0000","last_became_active":"2024-09-13T14:37:48.171779+0000","last_became_peered":"2024-09-13T14:37:48.171779+0000","last_unstale":"2024-09-13T14:42:09.674782+0000","last_undegraded":"2024-09-13T14:42:09.674782+0000","last_fullsized":"2024-09-13T14:42:09.674782+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:02.852712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":335,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:42:04.637618+0000","last_change":"2024-09-13T14:38:53.890236+0000","last_active":"2024-09-13T14:42:04.637618+0000","last_peered":"2024-09-13T14:42:04.637618+0000","last_clean":"2024-09-13T14:42:04.637618+0000","last_became_active":"2024-09-13T14:38:53.889582+0000","last_became_peered":"2024-09-13T14:38:53.889582+0000","last_unstale":"2024-09-13T14:42:04.637618+0000","last_undegraded":"2024-09-13T14:42:04.637618+0000","last_fullsized":"2024-09-13T14:42:04.637618+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:14:29.334774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":436,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:42:09.675047+0000","last_change":"2024-09-13T14:38:03.534097+0000","last_active":"2024-09-13T14:42:09.675047+0000","last_peered":"2024-09-13T14:42:09.675047+0000","last_clean":"2024-09-13T14:42:09.675047+0000","last_became_active":"2024-09-13T14:38:03.533794+0000","last_became_peered":"2024-09-13T14:38:03.533794+0000","last_unstale":"2024-09-13T14:42:09.675047+0000","last_undegraded":"2024-09-13T14:42:09.675047+0000","last_fullsized":"2024-09-13T14:42:09.675047+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:23.272098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":81,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:42:09.710734+0000","last_change":"2024-09-13T14:38:03.527021+0000","last_active":"2024-09-13T14:42:09.710734+0000","last_peered":"2024-09-13T14:42:09.710734+0000","last_clean":"2024-09-13T14:42:09.710734+0000","last_became_active":"2024-09-13T14:38:03.526698+0000","last_became_peered":"2024-09-13T14:38:03.526698+0000","last_unstale":"2024-09-13T14:42:09.710734+0000","last_undegraded":"2024-09-13T14:42:09.710734+0000","last_fullsized":"2024-09-13T14:42:09.710734+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:01:35.734086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":279,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:42:04.637879+0000","last_change":"2024-09-13T14:27:12.834931+0000","last_active":"2024-09-13T14:42:04.637879+0000","last_peered":"2024-09-13T14:42:04.637879+0000","last_clean":"2024-09-13T14:42:04.637879+0000","last_became_active":"2024-09-13T14:27:12.834572+0000","last_became_peered":"2024-09-13T14:27:12.834572+0000","last_unstale":"2024-09-13T14:42:04.637879+0000","last_undegraded":"2024-09-13T14:42:04.637879+0000","last_fullsized":"2024-09-13T14:42:04.637879+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:54:21.155689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":92,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:42:04.652555+0000","last_change":"2024-09-13T14:37:49.176796+0000","last_active":"2024-09-13T14:42:04.652555+0000","last_peered":"2024-09-13T14:42:04.652555+0000","last_clean":"2024-09-13T14:42:04.652555+0000","last_became_active":"2024-09-13T14:37:49.176246+0000","last_became_peered":"2024-09-13T14:37:49.176246+0000","last_unstale":"2024-09-13T14:42:04.652555+0000","last_undegraded":"2024-09-13T14:42:04.652555+0000","last_fullsized":"2024-09-13T14:42:04.652555+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:29:24.936718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":290,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:42:04.637525+0000","last_change":"2024-09-13T14:38:03.526333+0000","last_active":"2024-09-13T14:42:04.637525+0000","last_peered":"2024-09-13T14:42:04.637525+0000","last_clean":"2024-09-13T14:42:04.637525+0000","last_became_active":"2024-09-13T14:38:03.526001+0000","last_became_peered":"2024-09-13T14:38:03.526001+0000","last_unstale":"2024-09-13T14:42:04.637525+0000","last_undegraded":"2024-09-13T14:42:04.637525+0000","last_fullsized":"2024-09-13T14:42:04.637525+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:25:25.181271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":92,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:42:04.652515+0000","last_change":"2024-09-13T14:37:49.149582+0000","last_active":"2024-09-13T14:42:04.652515+0000","last_peered":"2024-09-13T14:42:04.652515+0000","last_clean":"2024-09-13T14:42:04.652515+0000","last_became_active":"2024-09-13T14:37:49.148525+0000","last_became_peered":"2024-09-13T14:37:49.148525+0000","last_unstale":"2024-09-13T14:42:04.652515+0000","last_undegraded":"2024-09-13T14:42:04.652515+0000","last_fullsized":"2024-09-13T14:42:04.652515+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:48.655665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":482,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:42:09.674489+0000","last_change":"2024-09-13T14:38:03.535255+0000","last_active":"2024-09-13T14:42:09.674489+0000","last_peered":"2024-09-13T14:42:09.674489+0000","last_clean":"2024-09-13T14:42:09.674489+0000","last_became_active":"2024-09-13T14:38:03.533917+0000","last_became_peered":"2024-09-13T14:38:03.533917+0000","last_unstale":"2024-09-13T14:42:09.674489+0000","last_undegraded":"2024-09-13T14:42:09.674489+0000","last_fullsized":"2024-09-13T14:42:09.674489+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:33.559788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":445,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:42:09.675160+0000","last_change":"2024-09-13T14:37:48.146784+0000","last_active":"2024-09-13T14:42:09.675160+0000","last_peered":"2024-09-13T14:42:09.675160+0000","last_clean":"2024-09-13T14:42:09.675160+0000","last_became_active":"2024-09-13T14:37:48.146491+0000","last_became_peered":"2024-09-13T14:37:48.146491+0000","last_unstale":"2024-09-13T14:42:09.675160+0000","last_undegraded":"2024-09-13T14:42:09.675160+0000","last_fullsized":"2024-09-13T14:42:09.675160+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:30:54.772513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":270,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:42:04.637952+0000","last_change":"2024-09-13T14:37:48.127607+0000","last_active":"2024-09-13T14:42:04.637952+0000","last_peered":"2024-09-13T14:42:04.637952+0000","last_clean":"2024-09-13T14:42:04.637952+0000","last_became_active":"2024-09-13T14:37:48.127316+0000","last_became_peered":"2024-09-13T14:37:48.127316+0000","last_unstale":"2024-09-13T14:42:04.637952+0000","last_undegraded":"2024-09-13T14:42:04.637952+0000","last_fullsized":"2024-09-13T14:42:04.637952+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:53:37.678861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":87,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:42:09.730345+0000","last_change":"2024-09-13T14:38:03.563200+0000","last_active":"2024-09-13T14:42:09.730345+0000","last_peered":"2024-09-13T14:42:09.730345+0000","last_clean":"2024-09-13T14:42:09.730345+0000","last_became_active":"2024-09-13T14:38:03.534258+0000","last_became_peered":"2024-09-13T14:38:03.534258+0000","last_unstale":"2024-09-13T14:42:09.730345+0000","last_undegraded":"2024-09-13T14:42:09.730345+0000","last_fullsized":"2024-09-13T14:42:09.730345+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:49:07.814419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":4435148824,"num_objects":67676,"num_object_clones":0,"num_object_copies":203028,"num_objects_missing_on_primary":1580,"num_objects_missing":1580,"num_objects_degraded":19978,"num_objects_misplaced":3388,"num_objects_unfound":0,"num_objects_dirty":67676,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157950,"num_write_kb":7219969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70758,"num_bytes_recovered":4596236312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15050289152,"data_stored":15050276936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93673,"ondisk_log_size":93673,"up":48,"acting":45,"num_store_stats":6},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1538,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":17390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25132,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":7,"num_objects_missing":7,"num_objects_degraded":16,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16793600,"data_stored":16777292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":23,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2969600,"data_stored":2951840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234380,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2818300,"kb_used_data":2372084,"kb_used_omap":1,"kb_used_meta":446206,"kb_avail":90926852,"statfs":{"total":95995035648,"available":93109096448,"internally_reserved":0,"allocated":2429014016,"data_stored":2427505717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":456915403},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":239,"seq":1026497183798,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1909380,"kb_used_data":1662456,"kb_used_omap":8,"kb_used_meta":246903,"kb_avail":91835772,"statfs":{"total":95995035648,"available":94039830528,"internally_reserved":0,"allocated":1702354944,"data_stored":1700843120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8453,"internal_metadata":252829435},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757645,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2218732,"kb_used_data":1579748,"kb_used_omap":1,"kb_used_meta":638974,"kb_avail":91526420,"statfs":{"total":95995035648,"available":93723054080,"internally_reserved":0,"allocated":1617661952,"data_stored":1616165214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":654309830},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019289,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2967416,"kb_used_data":2372720,"kb_used_omap":1,"kb_used_meta":594686,"kb_avail":90777736,"statfs":{"total":95995035648,"available":92956401664,"internally_reserved":0,"allocated":2429665280,"data_stored":2428161065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":608958918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435472,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3816808,"kb_used_data":3265888,"kb_used_omap":5,"kb_used_meta":550906,"kb_avail":89928344,"statfs":{"total":95995035648,"available":92086624256,"internally_reserved":0,"allocated":3344269312,"data_stored":3342776434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":564127870},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":446676599000,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":425201762527,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926051,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4158072,"kb_used_data":3494704,"kb_used_omap":7,"kb_used_meta":663352,"kb_avail":89587080,"statfs":{"total":95995035648,"available":91737169920,"internally_reserved":0,"allocated":3578576896,"data_stored":3577068801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":679272744},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,3,2,3,4,4,7],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6860,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3564572672,"data_stored":3564568600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3337162752,"data_stored":3337158680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2424832000,"data_stored":2424832000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1610088448,"data_stored":1610088448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1692733440,"data_stored":1692729368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2420899840,"data_stored":2420899840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:42:20.901 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph status --format=json 2024-09-13T14:42:20.953 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:20.974 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:20.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:20.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:21.076 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:21.099 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:21.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:21.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:21.156 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:21.200 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:21.226 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:21.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:21.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:21.327 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:21.350 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:21.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:21.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:21.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:21 smithi102 ceph-mon[110183]: pgmap v939: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 81 active+clean; 4.1 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 149 op/s; 19994/201723 objects degraded (9.912%); 3119/201723 objects misplaced (1.546%); 5.2 MiB/s, 82 objects/s recovering 2024-09-13T14:42:21.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:21 smithi102 ceph-mon[115269]: pgmap v939: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 81 active+clean; 4.1 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 149 op/s; 19994/201723 objects degraded (9.912%); 3119/201723 objects misplaced (1.546%); 5.2 MiB/s, 82 objects/s recovering 2024-09-13T14:42:21.452 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:21.473 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:21.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:21.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:21.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:21 smithi195 ceph-mon[104262]: pgmap v939: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 81 active+clean; 4.1 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 149 op/s; 19994/201723 objects degraded (9.912%); 3119/201723 objects misplaced (1.546%); 5.2 MiB/s, 82 objects/s recovering 2024-09-13T14:42:21.575 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:21.599 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:21.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:21.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:21.701 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:21.726 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:21.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:21.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:21.827 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:21.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:21.851 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:21.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:21.952 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:21.975 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:21.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:21.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:22.077 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:22.098 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:22.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:22.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:22.172 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is unset 2024-09-13T14:42:22.200 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:22.234 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:22.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:22.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:22.337 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:22.368 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:22.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:22.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:22.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:22 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/88839756' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:42:22.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:22 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:42:22.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:22 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/88839756' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:42:22.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:22 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:42:22.469 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:22.476 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:42:22.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:22 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/88839756' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:42:22.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:22 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:42:22.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:22.500 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:22.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:22.602 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:22.631 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:22.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:22.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:22.733 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:22.758 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:22.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:22.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:22.861 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:22.886 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:22.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:22.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:22.988 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:23.010 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:23.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:23.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:23.112 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:23.133 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:23.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:23.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:23.235 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:23.259 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:23.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:23.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:23.361 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:23.387 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:23.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:23.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:23.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:23 smithi102 ceph-mon[110183]: pgmap v940: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 81 active+clean; 4.1 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 154 op/s; 19813/200718 objects degraded (9.871%); 3120/200718 objects misplaced (1.554%); 5.3 MiB/s, 84 objects/s recovering 2024-09-13T14:42:23.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:23 smithi102 ceph-mon[110183]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:42:23.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:23 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:42:23.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:23 smithi102 ceph-mon[110183]: osdmap e280: 8 total, 8 up, 6 in 2024-09-13T14:42:23.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:23 smithi102 ceph-mon[115269]: pgmap v940: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 81 active+clean; 4.1 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 154 op/s; 19813/200718 objects degraded (9.871%); 3120/200718 objects misplaced (1.554%); 5.3 MiB/s, 84 objects/s recovering 2024-09-13T14:42:23.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:23 smithi102 ceph-mon[115269]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:42:23.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:23 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:42:23.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:23 smithi102 ceph-mon[115269]: osdmap e280: 8 total, 8 up, 6 in 2024-09-13T14:42:23.488 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:23.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:23 smithi195 ceph-mon[104262]: pgmap v940: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 81 active+clean; 4.1 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 154 op/s; 19813/200718 objects degraded (9.871%); 3120/200718 objects misplaced (1.554%); 5.3 MiB/s, 84 objects/s recovering 2024-09-13T14:42:23.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:23 smithi195 ceph-mon[104262]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:42:23.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:23 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:42:23.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:23 smithi195 ceph-mon[104262]: osdmap e280: 8 total, 8 up, 6 in 2024-09-13T14:42:23.513 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:23.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:23.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:23.615 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:23.636 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:23.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:23.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:23.738 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:23.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:23.761 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:23.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:23.862 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:23.885 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:23.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:23.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:23.987 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:24.011 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:24.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:24.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:24.113 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:24.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:24.137 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:24.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:24.159 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:24.159 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:24.177 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:42:24.177 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:42:24.238 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:24.261 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:24.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:24.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:24.362 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:24.384 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:24.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:24.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:24.489 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:24.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:24 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 19813/200718 objects degraded (9.871%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:42:24.511 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:24.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:24.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:24.588 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:24 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 19813/200718 objects degraded (9.871%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:42:24.588 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:24 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 19813/200718 objects degraded (9.871%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:42:24.613 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:24.637 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:24.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:24.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:24.739 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:24.762 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:24.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:24.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:24.864 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:24.886 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:24.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:24.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:24.987 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:25.009 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:25.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:25.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:25.089 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:42:25.110 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:25.157 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:25.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:25.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:25.258 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:25.286 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:25.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:25.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:25.387 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:25.410 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:25.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:25.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:25.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:25 smithi195 ceph-mon[104262]: pgmap v942: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 81 active+clean; 4.1 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 163 op/s; 19813/199110 objects degraded (9.951%); 2793/199110 objects misplaced (1.403%); 5.9 MiB/s, 93 objects/s recovering 2024-09-13T14:42:25.506 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:25 smithi102 ceph-mon[110183]: pgmap v942: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 81 active+clean; 4.1 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 163 op/s; 19813/199110 objects degraded (9.951%); 2793/199110 objects misplaced (1.403%); 5.9 MiB/s, 93 objects/s recovering 2024-09-13T14:42:25.506 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:25 smithi102 ceph-mon[115269]: pgmap v942: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 81 active+clean; 4.1 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 163 op/s; 19813/199110 objects degraded (9.951%); 2793/199110 objects misplaced (1.403%); 5.9 MiB/s, 93 objects/s recovering 2024-09-13T14:42:25.512 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:25.534 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:25.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:25.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:25.636 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:25.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:25.661 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:25.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:25.762 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:25.785 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:25.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:25.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:25.887 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:25.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:25.918 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:25.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:26.020 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:26.053 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:26.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:26.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:26.155 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:26.176 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:26.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:26.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:26.278 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:26.307 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:26.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:26.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:26.409 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:26.436 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:26.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:26.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:26.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:42:26.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:42:26.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:42:26.540 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:26.574 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:26.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:26.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:26.676 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:26.700 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:26.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:26.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:26.724 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:42:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:42:26] "GET /metrics HTTP/1.1" 200 36552 "" "Prometheus/2.51.0" 2024-09-13T14:42:26.804 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:26.826 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:26.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:26.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:26.928 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:26.950 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:26.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:26.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:27.052 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:27.054 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T14:42:27.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:27.074 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:27.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:27.176 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:27.199 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:27.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:27.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:27.301 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:27.323 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:27.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:27.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:27.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:27 smithi102 ceph-mon[110183]: pgmap v943: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 81 active+clean; 4.1 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 133 op/s; 19813/199110 objects degraded (9.951%); 2793/199110 objects misplaced (1.403%); 4.9 MiB/s, 77 objects/s recovering 2024-09-13T14:42:27.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:27 smithi102 ceph-mon[115269]: pgmap v943: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 81 active+clean; 4.1 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 133 op/s; 19813/199110 objects degraded (9.951%); 2793/199110 objects misplaced (1.403%); 4.9 MiB/s, 77 objects/s recovering 2024-09-13T14:42:27.425 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:27.453 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:27.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:27.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:27.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:27 smithi195 ceph-mon[104262]: pgmap v943: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 81 active+clean; 4.1 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 133 op/s; 19813/199110 objects degraded (9.951%); 2793/199110 objects misplaced (1.403%); 4.9 MiB/s, 77 objects/s recovering 2024-09-13T14:42:27.557 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:27.580 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:27.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:27.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:27.682 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:27.705 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:27.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:27.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:27.807 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:27.833 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:27.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:27.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:27.935 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:27.959 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:27.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:27.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:28.060 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:28.082 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:28.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:28.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:28.184 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:28.208 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:28.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:28.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:28.305 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:42:28.305 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:42:28.310 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:28.323 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:42:28.323 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:42:28.330 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:28.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:28.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:28.340 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:42:28.340 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:42:28.358 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:42:28.358 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:42:28.386 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:42:28.387 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:42:28.415 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:42:28.415 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:42:28.432 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:28.443 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:42:28.443 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:42:28.455 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:28.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:28.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:28.469 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:42:28.469 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:42:28.556 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:28.582 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:28.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:28.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:28.683 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:28.706 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:28.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:28.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:28.808 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:28.833 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:28.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:28.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:28.856 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:42:28.856 INFO:teuthology.orchestra.run.smithi102.stdout:{"fsid":"08cee502-71d9-11ef-bceb-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19645/196716 objects degraded (9.986%), 8 pgs degraded, 4 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1372,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":280,"num_osds":8,"num_up_osds":8,"osd_up_since":1726238267,"num_in_osds":6,"osd_in_since":1726238173,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":81},{"state_name":"active+recovery_wait+degraded","count":3},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+recovery_wait+degraded+remapped","count":1}],"num_pgs":89,"num_pools":5,"num_objects":65572,"data_bytes":4300216317,"bytes_used":18006945792,"bytes_avail":557963268096,"bytes_total":575970213888,"degraded_objects":19645,"degraded_total":196716,"degraded_ratio":0.099864779682384774,"misplaced_objects":2273,"misplaced_total":196716,"misplaced_ratio":0.011554728644340064,"recovering_objects_per_sec":119,"recovering_bytes_per_sec":7836126,"recovering_keys_per_sec":0,"num_objects_recovered":1196,"num_bytes_recovered":78381056,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":171},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi102.njetxr","status":"up:active","gid":24521}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.102:8443/","prometheus":"http://172.21.15.102:9283/"}},"servicemap":{"epoch":28,"modified":"2024-09-13T14:40:21.476600+0000","services":{}},"progress_events":{}} 2024-09-13T14:42:28.935 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:28.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:28.958 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:28.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:29.059 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:29.083 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:29.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:29.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:29.185 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:29.212 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:29.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:29.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:29.314 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:29.335 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:29.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:29.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:29.437 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:29.458 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:29.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:29.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:29.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:29 smithi195 ceph-mon[104262]: pgmap v944: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 81 active+clean; 4.0 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 171 op/s; 19645/196716 objects degraded (9.986%); 2273/196716 objects misplaced (1.155%); 7.5 MiB/s, 119 objects/s recovering 2024-09-13T14:42:29.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:29 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2169905927' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:42:29.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:29 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1942523363' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:42:29.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:29 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 19645/196716 objects degraded (9.986%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:42:29.560 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:29.588 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:29.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:29.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:29.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:29 smithi102 ceph-mon[110183]: pgmap v944: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 81 active+clean; 4.0 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 171 op/s; 19645/196716 objects degraded (9.986%); 2273/196716 objects misplaced (1.155%); 7.5 MiB/s, 119 objects/s recovering 2024-09-13T14:42:29.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:29 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2169905927' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:42:29.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:29 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1942523363' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:42:29.650 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:29 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 19645/196716 objects degraded (9.986%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:42:29.650 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:29 smithi102 ceph-mon[115269]: pgmap v944: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 81 active+clean; 4.0 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 171 op/s; 19645/196716 objects degraded (9.986%); 2273/196716 objects misplaced (1.155%); 7.5 MiB/s, 119 objects/s recovering 2024-09-13T14:42:29.650 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:29 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2169905927' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:42:29.650 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:29 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1942523363' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:42:29.650 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:29 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 19645/196716 objects degraded (9.986%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:42:29.689 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:29.711 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:29.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:29.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:29.813 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:29.836 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:29.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:29.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:29.938 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:29.959 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:29.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:29.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:29.976 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-13T14:42:29.976 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:42:30.021 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:30.061 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:30.063 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:30.084 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:30.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:30.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:30.186 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:30.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:30.213 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:30.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:30.314 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:30.343 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:30.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:30.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:30.444 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:30.470 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:30.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:30.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:30.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:30 smithi195 ceph-mon[104262]: osdmap e281: 8 total, 8 up, 6 in 2024-09-13T14:42:30.539 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:30 smithi102 ceph-mon[115269]: osdmap e281: 8 total, 8 up, 6 in 2024-09-13T14:42:30.539 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:30 smithi102 ceph-mon[110183]: osdmap e281: 8 total, 8 up, 6 in 2024-09-13T14:42:30.573 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:30.595 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:30.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:30.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:30.697 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:30.722 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:30.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:30.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:30.824 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:30.847 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:30.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:30.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:30.948 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:30.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:30.971 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:30.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:31.072 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:31.096 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:31.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:31.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:31.198 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:31.221 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:31.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:31.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:31.296 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:42:31.323 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:31.366 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:31.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:31.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:31.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:31 smithi102 ceph-mon[110183]: pgmap v946: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 81 active+clean; 4.0 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 173 op/s; 19645/196557 objects degraded (9.995%); 2273/196557 objects misplaced (1.156%); 7.9 MiB/s, 126 objects/s recovering 2024-09-13T14:42:31.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:31 smithi102 ceph-mon[110183]: osdmap e282: 8 total, 8 up, 6 in 2024-09-13T14:42:31.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:31 smithi102 ceph-mon[115269]: pgmap v946: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 81 active+clean; 4.0 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 173 op/s; 19645/196557 objects degraded (9.995%); 2273/196557 objects misplaced (1.156%); 7.9 MiB/s, 126 objects/s recovering 2024-09-13T14:42:31.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:31 smithi102 ceph-mon[115269]: osdmap e282: 8 total, 8 up, 6 in 2024-09-13T14:42:31.468 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:31.491 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:31.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:31.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:31.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:31 smithi195 ceph-mon[104262]: pgmap v946: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 81 active+clean; 4.0 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 173 op/s; 19645/196557 objects degraded (9.995%); 2273/196557 objects misplaced (1.156%); 7.9 MiB/s, 126 objects/s recovering 2024-09-13T14:42:31.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:31 smithi195 ceph-mon[104262]: osdmap e282: 8 total, 8 up, 6 in 2024-09-13T14:42:31.593 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:31.615 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:31.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:31.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:31.717 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:31.742 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:31.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:31.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:31.843 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:31.866 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:31.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:31.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:31.968 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:31.993 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:31.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:31.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:32.095 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:32.123 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:32.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:32.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:32.225 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:32.248 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:32.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:32.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:32.265 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:32.351 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:32.374 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:32.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:32.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:32.476 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:32.501 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:32.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:32.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:32.602 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:32.631 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:32.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:32.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:32.732 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:32.755 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:32.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:32.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:32.857 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:32.880 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:32.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:32.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:32.982 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:33.004 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:33.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:33.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:33.105 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:33.127 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:33.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:33.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:33.229 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:33.256 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:33.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:33.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:33.358 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:33.379 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:33.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:33.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:33.462 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:33 smithi102 ceph-mon[110183]: pgmap v948: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 81 active+clean; 4.0 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 142 op/s; 15314/195687 objects degraded (7.826%); 6306/195687 objects misplaced (3.222%); 6.4 MiB/s, 104 objects/s recovering 2024-09-13T14:42:33.462 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:33 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1722342107' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:42:33.462 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:33 smithi102 ceph-mon[115269]: pgmap v948: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 81 active+clean; 4.0 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 142 op/s; 15314/195687 objects degraded (7.826%); 6306/195687 objects misplaced (3.222%); 6.4 MiB/s, 104 objects/s recovering 2024-09-13T14:42:33.463 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:33 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1722342107' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:42:33.481 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:33.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:33 smithi195 ceph-mon[104262]: pgmap v948: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 81 active+clean; 4.0 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 142 op/s; 15314/195687 objects degraded (7.826%); 6306/195687 objects misplaced (3.222%); 6.4 MiB/s, 104 objects/s recovering 2024-09-13T14:42:33.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:33 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1722342107' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:42:33.504 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:33.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:33.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:33.605 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:33.627 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:33.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:33.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:33.729 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:33.752 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:33.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:33.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:33.854 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:33.878 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:33.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:33.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:33.980 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:34.001 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:34.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:34.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:34.103 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:34.126 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:34.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:34.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:34.209 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is unset 2024-09-13T14:42:34.228 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:34.252 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:34.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:34.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:34.354 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:34.380 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:34.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:34.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:34.426 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:34.482 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:34.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:34 smithi195 ceph-mon[104262]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:42:34.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:34 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1722342107' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:42:34.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:34 smithi195 ceph-mon[104262]: osdmap e283: 8 total, 8 up, 6 in 2024-09-13T14:42:34.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:34 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 15314/195687 objects degraded (7.826%), 7 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:42:34.503 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:34.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:34.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:34.531 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:34 smithi102 ceph-mon[115269]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:42:34.531 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:34 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1722342107' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:42:34.531 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:34 smithi102 ceph-mon[115269]: osdmap e283: 8 total, 8 up, 6 in 2024-09-13T14:42:34.531 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:34 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 15314/195687 objects degraded (7.826%), 7 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:42:34.531 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:34 smithi102 ceph-mon[110183]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:42:34.531 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:34 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1722342107' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:42:34.531 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:34 smithi102 ceph-mon[110183]: osdmap e283: 8 total, 8 up, 6 in 2024-09-13T14:42:34.531 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:34 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 15314/195687 objects degraded (7.826%), 7 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:42:34.605 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:34.627 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:34.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:34.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:34.729 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:34.750 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:34.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:34.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:34.852 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:34.876 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:34.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:34.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:34.978 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:35.001 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:35.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:35.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:35.103 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:35.129 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:35.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:35.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:35.230 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:35.254 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:35.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:35.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:35.356 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:35.379 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:35.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:35.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:35.481 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:35.484 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:35 smithi102 ceph-mon[110183]: pgmap v950: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 4.0 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 140 op/s; 15313/194181 objects degraded (7.886%); 6020/194181 objects misplaced (3.100%); 4.4 MiB/s, 95 objects/s recovering 2024-09-13T14:42:35.484 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:35 smithi102 ceph-mon[115269]: pgmap v950: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 4.0 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 140 op/s; 15313/194181 objects degraded (7.886%); 6020/194181 objects misplaced (3.100%); 4.4 MiB/s, 95 objects/s recovering 2024-09-13T14:42:35.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:35 smithi195 ceph-mon[104262]: pgmap v950: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 4.0 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 140 op/s; 15313/194181 objects degraded (7.886%); 6020/194181 objects misplaced (3.100%); 4.4 MiB/s, 95 objects/s recovering 2024-09-13T14:42:35.499 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:42:35.499 INFO:teuthology.orchestra.run.smithi102.stdout: "size": 20, 2024-09-13T14:42:35.499 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 600, 2024-09-13T14:42:35.500 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [] 2024-09-13T14:42:35.500 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:42:35.511 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:35.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:35.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:35.612 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:35.636 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:35.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:35.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:35.737 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:35.762 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:35.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:35.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:35.864 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:35.889 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:35.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:35.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:35.991 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:36.012 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:36.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:36.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:36.114 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:36.135 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:36.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:36.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:36.193 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T14:42:36.237 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:36.259 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:36.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:36.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:36.360 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:36.382 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:36.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:36.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:36.484 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:36.505 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:36.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:36.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:36.607 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:36.634 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:36.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:36.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:36.736 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:36.739 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:42:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:42:36] "GET /metrics HTTP/1.1" 200 36552 "" "Prometheus/2.51.0" 2024-09-13T14:42:36.758 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:36.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:36.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:36.860 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:36.884 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:36.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:36.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:36.966 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:42:36.986 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:37.032 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:37.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:37.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:37.134 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:37.156 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:37.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:37.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:37.258 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:37.282 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:37.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:37.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:37.293 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:37 smithi102 ceph-mon[115269]: pgmap v951: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 4.0 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 125 op/s; 15313/194181 objects degraded (7.886%); 6020/194181 objects misplaced (3.100%); 4.1 MiB/s, 90 objects/s recovering 2024-09-13T14:42:37.293 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:37 smithi102 ceph-mon[110183]: pgmap v951: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 4.0 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 125 op/s; 15313/194181 objects degraded (7.886%); 6020/194181 objects misplaced (3.100%); 4.1 MiB/s, 90 objects/s recovering 2024-09-13T14:42:37.384 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:37.407 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:37.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:37.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:37.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:37 smithi195 ceph-mon[104262]: pgmap v951: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 4.0 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 125 op/s; 15313/194181 objects degraded (7.886%); 6020/194181 objects misplaced (3.100%); 4.1 MiB/s, 90 objects/s recovering 2024-09-13T14:42:37.509 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:37.530 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:37.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:37.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:37.633 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:37.660 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:37.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:37.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:37.762 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:37.981 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:38.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:38.004 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:38.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:38.105 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:38.127 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:38.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:38.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:38.229 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:38.252 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:38.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:38.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:38.354 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:38.381 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:38.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:38.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:38.483 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:38.505 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:38.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:38.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:38.606 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:38.629 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:38.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:38.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:38.731 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:38.755 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:38.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:38.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:38.857 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:38.883 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:38.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:38.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:38.985 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:39.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:39.012 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:39.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:39.113 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:39.135 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:39.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:39.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:39.175 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:42:39.177 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:42:39.236 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:39.259 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:39.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:39.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:39.361 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:39.387 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:39.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:39.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:39.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:39 smithi102 ceph-mon[115269]: pgmap v952: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.9 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 202 op/s; 15210/191697 objects degraded (7.934%); 5976/191697 objects misplaced (3.117%); 4.4 MiB/s, 94 objects/s recovering 2024-09-13T14:42:39.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:39 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 15210/191697 objects degraded (7.934%), 6 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:42:39.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:39 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2964970018' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:42:39.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:39 smithi102 ceph-mon[110183]: pgmap v952: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.9 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 202 op/s; 15210/191697 objects degraded (7.934%); 5976/191697 objects misplaced (3.117%); 4.4 MiB/s, 94 objects/s recovering 2024-09-13T14:42:39.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:39 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 15210/191697 objects degraded (7.934%), 6 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:42:39.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:39 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2964970018' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:42:39.488 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:39.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:39 smithi195 ceph-mon[104262]: pgmap v952: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.9 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 202 op/s; 15210/191697 objects degraded (7.934%); 5976/191697 objects misplaced (3.117%); 4.4 MiB/s, 94 objects/s recovering 2024-09-13T14:42:39.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:39 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 15210/191697 objects degraded (7.934%), 6 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:42:39.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:39 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2964970018' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:42:39.511 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:39.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:39.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:39.613 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:39.634 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:39.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:39.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:39.736 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:39.762 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:39.762 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:39.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:39.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:39.864 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:39.887 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:39.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:39.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:39.989 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:40.011 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:40.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:40.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:40.114 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:40.135 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:40.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:40.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:40.237 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:40.263 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:40.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:40.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:40.365 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:40.390 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:40.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:40.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:40.492 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:40.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:40 smithi195 ceph-mon[104262]: from='client.45830 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:42:40.518 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:40.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:40.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:40.620 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:40.644 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:40.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:40.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:40.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:40 smithi102 ceph-mon[110183]: from='client.45830 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:42:40.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:40 smithi102 ceph-mon[115269]: from='client.45830 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:42:40.746 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:40.767 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:40.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:40.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:40.869 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:40.893 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:40.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:40.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:40.995 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:41.016 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:41.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:41.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:41.118 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:41.141 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:41.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:41.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:41.233 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:41 smithi102 ceph-mon[110183]: pgmap v953: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.9 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 179 op/s; 15210/191538 objects degraded (7.941%); 5976/191538 objects misplaced (3.120%); 3.8 MiB/s, 81 objects/s recovering 2024-09-13T14:42:41.243 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:41.270 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:41.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:41.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:41.372 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:41.396 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:41.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:41.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:41.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:41 smithi195 ceph-mon[104262]: pgmap v953: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.9 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 179 op/s; 15210/191538 objects degraded (7.941%); 5976/191538 objects misplaced (3.120%); 3.8 MiB/s, 81 objects/s recovering 2024-09-13T14:42:41.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:41 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:42:41.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:41 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:42:41.498 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:41.524 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:41.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:41.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:41.626 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:41.649 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:41.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:41.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:41.654 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:41 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:42:41.654 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:41 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:42:41.655 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:41 smithi102 ceph-mon[115269]: pgmap v953: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.9 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 179 op/s; 15210/191538 objects degraded (7.941%); 5976/191538 objects misplaced (3.120%); 3.8 MiB/s, 81 objects/s recovering 2024-09-13T14:42:41.655 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:41 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:42:41.655 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:41 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:42:41.751 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:41.777 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:41.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:41.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:41.878 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:41.903 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:41.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:41.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:42.005 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:42.029 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:42.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:42.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:42.131 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:42.152 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:42.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:42.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:42.248 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is set 2024-09-13T14:42:42.255 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:42.281 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:42.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:42.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:42.383 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:42.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:42.405 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:42.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:42.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:42 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2094011391' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:42:42.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:42 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:42:42.507 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:42.531 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:42.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:42.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:42.632 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:42:42.632 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:42:42.632 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:42:42.632 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:42:42.632 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:42:42.633 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:42:42.633 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:42:42.633 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:42:42.633 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:42:42.633 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:42:42.633 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:42:42.634 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:42:42.634 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:42:42.634 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:42:42.634 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:42:42.634 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:42:42.634 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:42 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2094011391' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:42:42.634 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:42 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:42:42.635 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:42 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2094011391' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:42:42.635 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:42 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:42:42.635 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:42.661 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:42.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:42.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:42.763 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:42.787 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:42.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:42.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:42.889 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:42.917 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:42.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:42.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:43.019 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:43.044 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:43.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:43.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:43.146 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:43.174 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:43.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:43.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:43.276 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:43.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:43.299 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:43.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:43.399 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:43.425 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:43.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:43.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:43.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:43 smithi195 ceph-mon[104262]: pgmap v954: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.9 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 166 op/s; 15107/190686 objects degraded (7.922%); 5932/190686 objects misplaced (3.111%); 3.6 MiB/s, 79 objects/s recovering 2024-09-13T14:42:43.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:43 smithi195 ceph-mon[104262]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:42:43.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:43 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:42:43.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:43 smithi195 ceph-mon[104262]: osdmap e284: 8 total, 8 up, 6 in 2024-09-13T14:42:43.508 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:43 smithi102 ceph-mon[110183]: pgmap v954: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.9 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 166 op/s; 15107/190686 objects degraded (7.922%); 5932/190686 objects misplaced (3.111%); 3.6 MiB/s, 79 objects/s recovering 2024-09-13T14:42:43.508 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:43 smithi102 ceph-mon[110183]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:42:43.509 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:43 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:42:43.509 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:43 smithi102 ceph-mon[110183]: osdmap e284: 8 total, 8 up, 6 in 2024-09-13T14:42:43.509 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:43 smithi102 ceph-mon[115269]: pgmap v954: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.9 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 166 op/s; 15107/190686 objects degraded (7.922%); 5932/190686 objects misplaced (3.111%); 3.6 MiB/s, 79 objects/s recovering 2024-09-13T14:42:43.509 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:43 smithi102 ceph-mon[115269]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:42:43.509 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:43 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:42:43.509 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:43 smithi102 ceph-mon[115269]: osdmap e284: 8 total, 8 up, 6 in 2024-09-13T14:42:43.528 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:43.550 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:43.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:43.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:43.652 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:43.676 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:43.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:43.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:43.778 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:43.805 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:43.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:43.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:43.856 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":952,"stamp":"2024-09-13T14:42:37.515181+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4190574589,"num_objects":63899,"num_object_clones":0,"num_object_copies":191697,"num_objects_missing_on_primary":134,"num_objects_missing":134,"num_objects_degraded":15210,"num_objects_misplaced":5976,"num_objects_unfound":0,"num_objects_dirty":63899,"num_whiteouts":0,"num_read":6377,"num_read_kb":13834,"num_write":162878,"num_write_kb":7235010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73085,"num_bytes_recovered":4735175821,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95860,"ondisk_log_size":95860,"up":267,"acting":264,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":302,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":17352856,"kb_used_data":14128612,"kb_used_omap":26,"kb_used_meta":3224165,"kb_avail":545118056,"statfs":{"total":575970213888,"available":558200889344,"internally_reserved":0,"allocated":14467698688,"data_stored":14458585150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27144,"internal_metadata":3301545464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2,0,5,3,2,7,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-106168320,"num_objects":-1620,"num_object_clones":0,"num_object_copies":-4860,"num_objects_missing_on_primary":-338,"num_objects_missing":-338,"num_objects_degraded":-4435,"num_objects_misplaced":3703,"num_objects_unfound":0,"num_objects_dirty":-1620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1620,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":756,"num_bytes_recovered":37027840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001705"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":459,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.213325+0000","last_change":"2024-09-13T14:36:13.859927+0000","last_active":"2024-09-13T14:42:33.213325+0000","last_peered":"2024-09-13T14:42:33.213325+0000","last_clean":"2024-09-13T14:42:33.213325+0000","last_became_active":"2024-09-13T14:36:13.859505+0000","last_became_peered":"2024-09-13T14:36:13.859505+0000","last_unstale":"2024-09-13T14:42:33.213325+0000","last_undegraded":"2024-09-13T14:42:33.213325+0000","last_fullsized":"2024-09-13T14:42:33.213325+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:12:20.989627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":285,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-13T14:42:30.204589+0000","last_change":"2024-09-13T14:37:48.130968+0000","last_active":"2024-09-13T14:42:30.204589+0000","last_peered":"2024-09-13T14:42:30.204589+0000","last_clean":"2024-09-13T14:42:30.204589+0000","last_became_active":"2024-09-13T14:37:48.130603+0000","last_became_peered":"2024-09-13T14:37:48.130603+0000","last_unstale":"2024-09-13T14:42:30.204589+0000","last_undegraded":"2024-09-13T14:42:30.204589+0000","last_fullsized":"2024-09-13T14:42:30.204589+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:42.626386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":81,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-13T14:42:30.204479+0000","last_change":"2024-09-13T14:36:13.862181+0000","last_active":"2024-09-13T14:42:30.204479+0000","last_peered":"2024-09-13T14:42:30.204479+0000","last_clean":"2024-09-13T14:42:30.204479+0000","last_became_active":"2024-09-13T14:36:13.861873+0000","last_became_peered":"2024-09-13T14:36:13.861873+0000","last_unstale":"2024-09-13T14:42:30.204479+0000","last_undegraded":"2024-09-13T14:42:30.204479+0000","last_fullsized":"2024-09-13T14:42:30.204479+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:14.085700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":96,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.215966+0000","last_change":"2024-09-13T14:38:53.896672+0000","last_active":"2024-09-13T14:42:33.215966+0000","last_peered":"2024-09-13T14:42:33.215966+0000","last_clean":"2024-09-13T14:42:33.215966+0000","last_became_active":"2024-09-13T14:38:53.896346+0000","last_became_peered":"2024-09-13T14:38:53.896346+0000","last_unstale":"2024-09-13T14:42:33.215966+0000","last_undegraded":"2024-09-13T14:42:33.215966+0000","last_fullsized":"2024-09-13T14:42:33.215966+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:57.789686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":96,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.216446+0000","last_change":"2024-09-13T14:38:53.893021+0000","last_active":"2024-09-13T14:42:33.216446+0000","last_peered":"2024-09-13T14:42:33.216446+0000","last_clean":"2024-09-13T14:42:33.216446+0000","last_became_active":"2024-09-13T14:38:53.892668+0000","last_became_peered":"2024-09-13T14:38:53.892668+0000","last_unstale":"2024-09-13T14:42:33.216446+0000","last_undegraded":"2024-09-13T14:42:33.216446+0000","last_fullsized":"2024-09-13T14:42:33.216446+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:57.180738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":253,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-13T14:42:30.203536+0000","last_change":"2024-09-13T14:37:48.134642+0000","last_active":"2024-09-13T14:42:30.203536+0000","last_peered":"2024-09-13T14:42:30.203536+0000","last_clean":"2024-09-13T14:42:30.203536+0000","last_became_active":"2024-09-13T14:37:48.134402+0000","last_became_peered":"2024-09-13T14:37:48.134402+0000","last_unstale":"2024-09-13T14:42:30.203536+0000","last_undegraded":"2024-09-13T14:42:30.203536+0000","last_fullsized":"2024-09-13T14:42:30.203536+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:51.313663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":478,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.212734+0000","last_change":"2024-09-13T14:38:53.903022+0000","last_active":"2024-09-13T14:42:33.212734+0000","last_peered":"2024-09-13T14:42:33.212734+0000","last_clean":"2024-09-13T14:42:33.212734+0000","last_became_active":"2024-09-13T14:38:53.902448+0000","last_became_peered":"2024-09-13T14:38:53.902448+0000","last_unstale":"2024-09-13T14:42:33.212734+0000","last_undegraded":"2024-09-13T14:42:33.212734+0000","last_fullsized":"2024-09-13T14:42:33.212734+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:24.690401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":99,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.215440+0000","last_change":"2024-09-13T14:38:03.554500+0000","last_active":"2024-09-13T14:42:33.215440+0000","last_peered":"2024-09-13T14:42:33.215440+0000","last_clean":"2024-09-13T14:42:33.215440+0000","last_became_active":"2024-09-13T14:38:03.532910+0000","last_became_peered":"2024-09-13T14:38:03.532910+0000","last_unstale":"2024-09-13T14:42:33.215440+0000","last_undegraded":"2024-09-13T14:42:33.215440+0000","last_fullsized":"2024-09-13T14:42:33.215440+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:27:50.843498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"283'10043","reported_seq":20937,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:37.255073+0000","last_change":"2024-09-13T14:41:53.220083+0000","last_active":"2024-09-13T14:42:37.255073+0000","last_peered":"2024-09-13T14:42:37.255073+0000","last_clean":"2024-09-13T14:42:37.255073+0000","last_became_active":"2024-09-13T14:41:53.200084+0000","last_became_peered":"2024-09-13T14:41:53.200084+0000","last_unstale":"2024-09-13T14:42:37.255073+0000","last_undegraded":"2024-09-13T14:42:37.255073+0000","last_fullsized":"2024-09-13T14:42:37.255073+0000","mapping_epoch":275,"log_start":"213'4300","ondisk_log_start":"213'4300","created":205,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5743,"log_dups_size":0,"ondisk_log_size":5743,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:08.897524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251461632,"num_objects":3837,"num_object_clones":0,"num_object_copies":11511,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10043,"num_write_kb":444160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9152,"num_bytes_recovered":594608128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":45,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-13T14:42:30.205119+0000","last_change":"2024-09-13T14:38:53.940238+0000","last_active":"2024-09-13T14:42:30.205119+0000","last_peered":"2024-09-13T14:42:30.205119+0000","last_clean":"2024-09-13T14:42:30.205119+0000","last_became_active":"2024-09-13T14:38:53.892825+0000","last_became_peered":"2024-09-13T14:38:53.892825+0000","last_unstale":"2024-09-13T14:42:30.205119+0000","last_undegraded":"2024-09-13T14:42:30.205119+0000","last_fullsized":"2024-09-13T14:42:30.205119+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:46.666581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":101,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-13T14:42:30.203231+0000","last_change":"2024-09-13T14:38:53.898907+0000","last_active":"2024-09-13T14:42:30.203231+0000","last_peered":"2024-09-13T14:42:30.203231+0000","last_clean":"2024-09-13T14:42:30.203231+0000","last_became_active":"2024-09-13T14:38:53.898572+0000","last_became_peered":"2024-09-13T14:38:53.898572+0000","last_unstale":"2024-09-13T14:42:30.203231+0000","last_undegraded":"2024-09-13T14:42:30.203231+0000","last_fullsized":"2024-09-13T14:42:30.203231+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:29.222150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"283'10277","reported_seq":24453,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:37.215446+0000","last_change":"2024-09-13T14:38:57.930032+0000","last_active":"2024-09-13T14:42:37.215446+0000","last_peered":"2024-09-13T14:42:37.215446+0000","last_clean":"2024-09-13T14:42:37.215446+0000","last_became_active":"2024-09-13T14:38:57.921458+0000","last_became_peered":"2024-09-13T14:38:57.921458+0000","last_unstale":"2024-09-13T14:42:37.215446+0000","last_undegraded":"2024-09-13T14:42:37.215446+0000","last_fullsized":"2024-09-13T14:42:37.215446+0000","mapping_epoch":256,"log_start":"213'4539","ondisk_log_start":"213'4539","created":205,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5738,"log_dups_size":0,"ondisk_log_size":5738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:27:12.294038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":259850240,"num_objects":3965,"num_object_clones":0,"num_object_copies":11895,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10277,"num_write_kb":455744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6990,"num_bytes_recovered":458096640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":45,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-13T14:42:30.205117+0000","last_change":"2024-09-13T14:38:53.904885+0000","last_active":"2024-09-13T14:42:30.205117+0000","last_peered":"2024-09-13T14:42:30.205117+0000","last_clean":"2024-09-13T14:42:30.205117+0000","last_became_active":"2024-09-13T14:38:53.891726+0000","last_became_peered":"2024-09-13T14:38:53.891726+0000","last_unstale":"2024-09-13T14:42:30.205117+0000","last_undegraded":"2024-09-13T14:42:30.205117+0000","last_fullsized":"2024-09-13T14:42:30.205117+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:57.079205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":290,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-13T14:42:29.201804+0000","last_change":"2024-09-13T14:37:48.160504+0000","last_active":"2024-09-13T14:42:29.201804+0000","last_peered":"2024-09-13T14:42:29.201804+0000","last_clean":"2024-09-13T14:42:29.201804+0000","last_became_active":"2024-09-13T14:37:48.160181+0000","last_became_peered":"2024-09-13T14:37:48.160181+0000","last_unstale":"2024-09-13T14:42:29.201804+0000","last_undegraded":"2024-09-13T14:42:29.201804+0000","last_fullsized":"2024-09-13T14:42:29.201804+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:37.087008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"283'10168","reported_seq":11965,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:35.638296+0000","last_change":"2024-09-13T14:38:04.531914+0000","last_active":"2024-09-13T14:42:35.638296+0000","last_peered":"2024-09-13T14:42:35.638296+0000","last_clean":"2024-09-13T14:42:35.638296+0000","last_became_active":"2024-09-13T14:38:04.523448+0000","last_became_peered":"2024-09-13T14:38:04.523448+0000","last_unstale":"2024-09-13T14:42:35.638296+0000","last_undegraded":"2024-09-13T14:42:35.638296+0000","last_fullsized":"2024-09-13T14:42:35.638296+0000","mapping_epoch":245,"log_start":"213'4200","ondisk_log_start":"213'4200","created":205,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5968,"log_dups_size":0,"ondisk_log_size":5968,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:06.536896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262537216,"num_objects":4006,"num_object_clones":0,"num_object_copies":12018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10168,"num_write_kb":453568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":811,"num_bytes_recovered":53149696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":282,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-13T14:42:30.203564+0000","last_change":"2024-09-13T14:37:49.611774+0000","last_active":"2024-09-13T14:42:30.203564+0000","last_peered":"2024-09-13T14:42:30.203564+0000","last_clean":"2024-09-13T14:42:30.203564+0000","last_became_active":"2024-09-13T14:37:49.611088+0000","last_became_peered":"2024-09-13T14:37:49.611088+0000","last_unstale":"2024-09-13T14:42:30.203564+0000","last_undegraded":"2024-09-13T14:42:30.203564+0000","last_fullsized":"2024-09-13T14:42:30.203564+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:08.933213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":41,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-13T14:42:30.204165+0000","last_change":"2024-09-13T14:38:53.899211+0000","last_active":"2024-09-13T14:42:30.204165+0000","last_peered":"2024-09-13T14:42:30.204165+0000","last_clean":"2024-09-13T14:42:30.204165+0000","last_became_active":"2024-09-13T14:38:53.898927+0000","last_became_peered":"2024-09-13T14:38:53.898927+0000","last_unstale":"2024-09-13T14:42:30.204165+0000","last_undegraded":"2024-09-13T14:42:30.204165+0000","last_fullsized":"2024-09-13T14:42:30.204165+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:00:28.857611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"283'10021","reported_seq":10111,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:36.937786+0000","last_change":"2024-09-13T14:35:24.004651+0000","last_active":"2024-09-13T14:42:36.937786+0000","last_peered":"2024-09-13T14:42:36.937786+0000","last_clean":"2024-09-13T14:42:36.937786+0000","last_became_active":"2024-09-13T14:35:24.004358+0000","last_became_peered":"2024-09-13T14:35:24.004358+0000","last_unstale":"2024-09-13T14:42:36.937786+0000","last_undegraded":"2024-09-13T14:42:36.937786+0000","last_fullsized":"2024-09-13T14:42:36.937786+0000","mapping_epoch":205,"log_start":"212'3684","ondisk_log_start":"212'3684","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6337,"log_dups_size":0,"ondisk_log_size":6337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:57:38.023669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":258932736,"num_objects":3951,"num_object_clones":0,"num_object_copies":11853,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10021,"num_write_kb":447104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":92,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-13T14:42:30.203787+0000","last_change":"2024-09-13T14:37:49.168003+0000","last_active":"2024-09-13T14:42:30.203787+0000","last_peered":"2024-09-13T14:42:30.203787+0000","last_clean":"2024-09-13T14:42:30.203787+0000","last_became_active":"2024-09-13T14:37:49.167638+0000","last_became_peered":"2024-09-13T14:37:49.167638+0000","last_unstale":"2024-09-13T14:42:30.203787+0000","last_undegraded":"2024-09-13T14:42:30.203787+0000","last_fullsized":"2024-09-13T14:42:30.203787+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:29.051981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":89,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.216103+0000","last_change":"2024-09-13T14:38:03.528381+0000","last_active":"2024-09-13T14:42:33.216103+0000","last_peered":"2024-09-13T14:42:33.216103+0000","last_clean":"2024-09-13T14:42:33.216103+0000","last_became_active":"2024-09-13T14:38:03.527282+0000","last_became_peered":"2024-09-13T14:38:03.527282+0000","last_unstale":"2024-09-13T14:42:33.216103+0000","last_undegraded":"2024-09-13T14:42:33.216103+0000","last_fullsized":"2024-09-13T14:42:33.216103+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:24.435067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.f","version":"283'10022","reported_seq":10314,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:35.689979+0000","last_change":"2024-09-13T14:38:16.371079+0000","last_active":"2024-09-13T14:42:35.689979+0000","last_peered":"2024-09-13T14:42:35.689979+0000","last_clean":"2024-09-13T14:42:35.689979+0000","last_became_active":"2024-09-13T14:38:12.859883+0000","last_became_peered":"2024-09-13T14:38:12.859883+0000","last_unstale":"2024-09-13T14:42:35.689979+0000","last_undegraded":"2024-09-13T14:42:35.689979+0000","last_fullsized":"2024-09-13T14:42:35.689979+0000","mapping_epoch":245,"log_start":"213'4063","ondisk_log_start":"213'4063","created":205,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5959,"log_dups_size":0,"ondisk_log_size":5959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:08.436532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266338304,"num_objects":4064,"num_object_clones":0,"num_object_copies":12192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10022,"num_write_kb":450752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":3670016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":447,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.213045+0000","last_change":"2024-09-13T14:38:03.549595+0000","last_active":"2024-09-13T14:42:33.213045+0000","last_peered":"2024-09-13T14:42:33.213045+0000","last_clean":"2024-09-13T14:42:33.213045+0000","last_became_active":"2024-09-13T14:38:03.533331+0000","last_became_peered":"2024-09-13T14:38:03.533331+0000","last_unstale":"2024-09-13T14:42:33.213045+0000","last_undegraded":"2024-09-13T14:42:33.213045+0000","last_fullsized":"2024-09-13T14:42:33.213045+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:59:02.470343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":265,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-13T14:42:29.201941+0000","last_change":"2024-09-13T14:37:48.127600+0000","last_active":"2024-09-13T14:42:29.201941+0000","last_peered":"2024-09-13T14:42:29.201941+0000","last_clean":"2024-09-13T14:42:29.201941+0000","last_became_active":"2024-09-13T14:37:48.127349+0000","last_became_peered":"2024-09-13T14:37:48.127349+0000","last_unstale":"2024-09-13T14:42:29.201941+0000","last_undegraded":"2024-09-13T14:42:29.201941+0000","last_fullsized":"2024-09-13T14:42:29.201941+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:57.268374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"283'10099","reported_seq":20086,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:37.075517+0000","last_change":"2024-09-13T14:42:30.227259+0000","last_active":"2024-09-13T14:42:37.075517+0000","last_peered":"2024-09-13T14:42:37.075517+0000","last_clean":"2024-09-13T14:42:37.075517+0000","last_became_active":"2024-09-13T14:42:30.209102+0000","last_became_peered":"2024-09-13T14:42:30.209102+0000","last_unstale":"2024-09-13T14:42:37.075517+0000","last_undegraded":"2024-09-13T14:42:37.075517+0000","last_fullsized":"2024-09-13T14:42:37.075517+0000","mapping_epoch":281,"log_start":"213'4400","ondisk_log_start":"213'4400","created":205,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5699,"log_dups_size":0,"ondisk_log_size":5699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:56:16.193287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256311296,"num_objects":3911,"num_object_clones":0,"num_object_copies":11733,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10099,"num_write_kb":448320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4926,"num_bytes_recovered":313851904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":198,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-13T14:42:29.201946+0000","last_change":"2024-09-13T14:37:48.127604+0000","last_active":"2024-09-13T14:42:29.201946+0000","last_peered":"2024-09-13T14:42:29.201946+0000","last_clean":"2024-09-13T14:42:29.201946+0000","last_became_active":"2024-09-13T14:37:48.127300+0000","last_became_peered":"2024-09-13T14:37:48.127300+0000","last_unstale":"2024-09-13T14:42:29.201946+0000","last_undegraded":"2024-09-13T14:42:29.201946+0000","last_fullsized":"2024-09-13T14:42:29.201946+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:51.999506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":318,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-13T14:42:29.202478+0000","last_change":"2024-09-13T14:38:03.529967+0000","last_active":"2024-09-13T14:42:29.202478+0000","last_peered":"2024-09-13T14:42:29.202478+0000","last_clean":"2024-09-13T14:42:29.202478+0000","last_became_active":"2024-09-13T14:38:03.529646+0000","last_became_peered":"2024-09-13T14:38:03.529646+0000","last_unstale":"2024-09-13T14:42:29.202478+0000","last_undegraded":"2024-09-13T14:42:29.202478+0000","last_fullsized":"2024-09-13T14:42:29.202478+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:52:05.193279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"283'10134","reported_seq":19302,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:35.704467+0000","last_change":"2024-09-13T14:40:34.583237+0000","last_active":"2024-09-13T14:42:35.704467+0000","last_peered":"2024-09-13T14:42:35.704467+0000","last_clean":"2024-09-13T14:42:35.704467+0000","last_became_active":"2024-09-13T14:40:34.563989+0000","last_became_peered":"2024-09-13T14:40:34.563989+0000","last_unstale":"2024-09-13T14:42:35.704467+0000","last_undegraded":"2024-09-13T14:42:35.704467+0000","last_fullsized":"2024-09-13T14:42:35.704467+0000","mapping_epoch":266,"log_start":"213'4200","ondisk_log_start":"213'4200","created":205,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5934,"log_dups_size":0,"ondisk_log_size":5934,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:34:43.006051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262471704,"num_objects":4006,"num_object_clones":0,"num_object_copies":12018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4006,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10134,"num_write_kb":452417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4489,"num_bytes_recovered":289865752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":279,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-13T14:42:29.202143+0000","last_change":"2024-09-13T14:36:13.860367+0000","last_active":"2024-09-13T14:42:29.202143+0000","last_peered":"2024-09-13T14:42:29.202143+0000","last_clean":"2024-09-13T14:42:29.202143+0000","last_became_active":"2024-09-13T14:36:13.859982+0000","last_became_peered":"2024-09-13T14:36:13.859982+0000","last_unstale":"2024-09-13T14:42:29.202143+0000","last_undegraded":"2024-09-13T14:42:29.202143+0000","last_fullsized":"2024-09-13T14:42:29.202143+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:08:23.311350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":274,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-13T14:42:29.202183+0000","last_change":"2024-09-13T14:37:48.147360+0000","last_active":"2024-09-13T14:42:29.202183+0000","last_peered":"2024-09-13T14:42:29.202183+0000","last_clean":"2024-09-13T14:42:29.202183+0000","last_became_active":"2024-09-13T14:37:48.146885+0000","last_became_peered":"2024-09-13T14:37:48.146885+0000","last_unstale":"2024-09-13T14:42:29.202183+0000","last_undegraded":"2024-09-13T14:42:29.202183+0000","last_fullsized":"2024-09-13T14:42:29.202183+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:27.841810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"283'10055","reported_seq":19330,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:37.360986+0000","last_change":"2024-09-13T14:38:39.650332+0000","last_active":"2024-09-13T14:42:37.360986+0000","last_peered":"2024-09-13T14:42:37.360986+0000","last_clean":"2024-09-13T14:42:37.360986+0000","last_became_active":"2024-09-13T14:38:39.640835+0000","last_became_peered":"2024-09-13T14:38:39.640835+0000","last_unstale":"2024-09-13T14:42:37.360986+0000","last_undegraded":"2024-09-13T14:42:37.360986+0000","last_fullsized":"2024-09-13T14:42:37.360986+0000","mapping_epoch":251,"log_start":"213'4351","ondisk_log_start":"213'4351","created":205,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5704,"log_dups_size":0,"ondisk_log_size":5704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:48.176606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263389184,"num_objects":4019,"num_object_clones":0,"num_object_copies":12057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10055,"num_write_kb":450368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4544,"num_bytes_recovered":292945920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":466,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.213094+0000","last_change":"2024-09-13T14:38:53.900041+0000","last_active":"2024-09-13T14:42:33.213094+0000","last_peered":"2024-09-13T14:42:33.213094+0000","last_clean":"2024-09-13T14:42:33.213094+0000","last_became_active":"2024-09-13T14:38:53.899203+0000","last_became_peered":"2024-09-13T14:38:53.899203+0000","last_unstale":"2024-09-13T14:42:33.213094+0000","last_undegraded":"2024-09-13T14:42:33.213094+0000","last_fullsized":"2024-09-13T14:42:33.213094+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:28:07.883094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":398,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.212879+0000","last_change":"2024-09-13T14:26:15.823198+0000","last_active":"2024-09-13T14:42:33.212879+0000","last_peered":"2024-09-13T14:42:33.212879+0000","last_clean":"2024-09-13T14:42:33.212879+0000","last_became_active":"2024-09-13T14:26:15.822717+0000","last_became_peered":"2024-09-13T14:26:15.822717+0000","last_unstale":"2024-09-13T14:42:33.212879+0000","last_undegraded":"2024-09-13T14:42:33.212879+0000","last_fullsized":"2024-09-13T14:42:33.212879+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:22.471622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.6","version":"283'10209","reported_seq":16400,"reported_epoch":283,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-13T14:42:35.739956+0000","last_change":"2024-09-13T14:38:55.132683+0000","last_active":"2024-09-13T14:42:35.739956+0000","last_peered":"2024-09-13T14:42:35.739956+0000","last_clean":"2024-09-13T14:36:13.124368+0000","last_became_active":"2024-09-13T14:38:55.120883+0000","last_became_peered":"2024-09-13T14:38:55.120883+0000","last_unstale":"2024-09-13T14:42:35.739956+0000","last_undegraded":"2024-09-13T14:38:55.026610+0000","last_fullsized":"2024-09-13T14:42:35.739956+0000","mapping_epoch":254,"log_start":"213'4319","ondisk_log_start":"213'4319","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5890,"log_dups_size":0,"ondisk_log_size":5890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261423104,"num_objects":3989,"num_object_clones":0,"num_object_copies":11967,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":1987,"num_objects_unfound":0,"num_objects_dirty":3989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10209,"num_write_kb":454336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3003,"num_bytes_recovered":185597952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,3,5],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3,4","objects":1723},{"shards":"0,3,5","objects":2266}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":416,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-13T14:42:30.203256+0000","last_change":"2024-09-13T14:41:53.196835+0000","last_active":"2024-09-13T14:42:30.203256+0000","last_peered":"2024-09-13T14:42:30.203256+0000","last_clean":"2024-09-13T14:42:30.203256+0000","last_became_active":"2024-09-13T14:41:53.196387+0000","last_became_peered":"2024-09-13T14:41:53.196387+0000","last_unstale":"2024-09-13T14:42:30.203256+0000","last_undegraded":"2024-09-13T14:42:30.203256+0000","last_fullsized":"2024-09-13T14:42:30.203256+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:07:41.927740+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:49.972303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":522,"reported_epoch":282,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:42:32.278398+0000","last_change":"2024-09-13T14:38:54.904329+0000","last_active":"2024-09-13T14:42:32.278398+0000","last_peered":"2024-09-13T14:42:32.278398+0000","last_clean":"2024-09-13T14:38:39.648017+0000","last_became_active":"2024-09-13T14:38:54.903798+0000","last_became_peered":"2024-09-13T14:38:54.903798+0000","last_unstale":"2024-09-13T14:42:32.278398+0000","last_undegraded":"2024-09-13T14:38:54.890505+0000","last_fullsized":"2024-09-13T14:38:54.889701+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:18.120931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,3],"avail_no_missing":["5","3","6"],"object_location_counts":[{"shards":"3,5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":400,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.214085+0000","last_change":"2024-09-13T14:37:49.160719+0000","last_active":"2024-09-13T14:42:33.214085+0000","last_peered":"2024-09-13T14:42:33.214085+0000","last_clean":"2024-09-13T14:42:33.214085+0000","last_became_active":"2024-09-13T14:37:49.160401+0000","last_became_peered":"2024-09-13T14:37:49.160401+0000","last_unstale":"2024-09-13T14:42:33.214085+0000","last_undegraded":"2024-09-13T14:42:33.214085+0000","last_fullsized":"2024-09-13T14:42:33.214085+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:21.737945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":126,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-13T14:42:29.202450+0000","last_change":"2024-09-13T14:37:48.172940+0000","last_active":"2024-09-13T14:42:29.202450+0000","last_peered":"2024-09-13T14:42:29.202450+0000","last_clean":"2024-09-13T14:42:29.202450+0000","last_became_active":"2024-09-13T14:37:48.172060+0000","last_became_peered":"2024-09-13T14:37:48.172060+0000","last_unstale":"2024-09-13T14:42:29.202450+0000","last_undegraded":"2024-09-13T14:42:29.202450+0000","last_fullsized":"2024-09-13T14:42:29.202450+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:07.480645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.5","version":"283'10110","reported_seq":26476,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:37.258890+0000","last_change":"2024-09-13T14:39:34.446699+0000","last_active":"2024-09-13T14:42:37.258890+0000","last_peered":"2024-09-13T14:42:37.258890+0000","last_clean":"2024-09-13T14:42:37.258890+0000","last_became_active":"2024-09-13T14:38:04.600616+0000","last_became_peered":"2024-09-13T14:38:04.600616+0000","last_unstale":"2024-09-13T14:42:37.258890+0000","last_undegraded":"2024-09-13T14:42:37.258890+0000","last_fullsized":"2024-09-13T14:42:37.258890+0000","mapping_epoch":245,"log_start":"212'3816","ondisk_log_start":"212'3816","created":205,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6294,"log_dups_size":0,"ondisk_log_size":6294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:00:44.321678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264241152,"num_objects":4032,"num_object_clones":0,"num_object_copies":12096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10110,"num_write_kb":452544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8120,"num_bytes_recovered":521076736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":500,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-13T14:42:30.203168+0000","last_change":"2024-09-13T14:26:15.835178+0000","last_active":"2024-09-13T14:42:30.203168+0000","last_peered":"2024-09-13T14:42:30.203168+0000","last_clean":"2024-09-13T14:42:30.203168+0000","last_became_active":"2024-09-13T14:26:15.820283+0000","last_became_peered":"2024-09-13T14:26:15.820283+0000","last_unstale":"2024-09-13T14:42:30.203168+0000","last_undegraded":"2024-09-13T14:42:30.203168+0000","last_fullsized":"2024-09-13T14:42:30.203168+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:30.340038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":90,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.215388+0000","last_change":"2024-09-13T14:38:03.525682+0000","last_active":"2024-09-13T14:42:33.215388+0000","last_peered":"2024-09-13T14:42:33.215388+0000","last_clean":"2024-09-13T14:42:33.215388+0000","last_became_active":"2024-09-13T14:38:03.525406+0000","last_became_peered":"2024-09-13T14:38:03.525406+0000","last_unstale":"2024-09-13T14:42:33.215388+0000","last_undegraded":"2024-09-13T14:42:33.215388+0000","last_fullsized":"2024-09-13T14:42:33.215388+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:41:28.970300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":97,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.216476+0000","last_change":"2024-09-13T14:38:53.991738+0000","last_active":"2024-09-13T14:42:33.216476+0000","last_peered":"2024-09-13T14:42:33.216476+0000","last_clean":"2024-09-13T14:42:33.216476+0000","last_became_active":"2024-09-13T14:38:53.991417+0000","last_became_peered":"2024-09-13T14:38:53.991417+0000","last_unstale":"2024-09-13T14:42:33.216476+0000","last_undegraded":"2024-09-13T14:42:33.216476+0000","last_fullsized":"2024-09-13T14:42:33.216476+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:50:38.559838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"283'10113","reported_seq":25145,"reported_epoch":283,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-13T14:42:35.724133+0000","last_change":"2024-09-13T14:38:17.472071+0000","last_active":"2024-09-13T14:42:35.724133+0000","last_peered":"2024-09-13T14:42:35.724133+0000","last_clean":"2024-09-13T14:37:46.290754+0000","last_became_active":"2024-09-13T14:38:17.381753+0000","last_became_peered":"2024-09-13T14:38:17.381753+0000","last_unstale":"2024-09-13T14:42:35.724133+0000","last_undegraded":"2024-09-13T14:38:13.504140+0000","last_fullsized":"2024-09-13T14:42:35.724133+0000","mapping_epoch":245,"log_start":"213'4159","ondisk_log_start":"213'4159","created":205,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5954,"log_dups_size":0,"ondisk_log_size":5954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261685248,"num_objects":3993,"num_object_clones":0,"num_object_copies":11979,"num_objects_missing_on_primary":83,"num_objects_missing":83,"num_objects_degraded":4161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10113,"num_write_kb":451392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7474,"num_bytes_recovered":489816064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"0,4,5","objects":3992},{"shards":"0,4,6","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":811,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.214892+0000","last_change":"2024-09-13T14:37:48.131799+0000","last_active":"2024-09-13T14:42:33.214892+0000","last_peered":"2024-09-13T14:42:33.214892+0000","last_clean":"2024-09-13T14:42:33.214892+0000","last_became_active":"2024-09-13T14:37:48.130837+0000","last_became_peered":"2024-09-13T14:37:48.130837+0000","last_unstale":"2024-09-13T14:42:33.214892+0000","last_undegraded":"2024-09-13T14:42:33.214892+0000","last_fullsized":"2024-09-13T14:42:33.214892+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:06:32.663770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":330,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.215711+0000","last_change":"2024-09-13T14:37:49.301531+0000","last_active":"2024-09-13T14:42:33.215711+0000","last_peered":"2024-09-13T14:42:33.215711+0000","last_clean":"2024-09-13T14:42:33.215711+0000","last_became_active":"2024-09-13T14:37:49.301249+0000","last_became_peered":"2024-09-13T14:37:49.301249+0000","last_unstale":"2024-09-13T14:42:33.215711+0000","last_undegraded":"2024-09-13T14:42:33.215711+0000","last_fullsized":"2024-09-13T14:42:33.215711+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:10.977710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":108,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-13T14:42:30.204932+0000","last_change":"2024-09-13T14:38:53.901427+0000","last_active":"2024-09-13T14:42:30.204932+0000","last_peered":"2024-09-13T14:42:30.204932+0000","last_clean":"2024-09-13T14:42:30.204932+0000","last_became_active":"2024-09-13T14:38:53.901157+0000","last_became_peered":"2024-09-13T14:38:53.901157+0000","last_unstale":"2024-09-13T14:42:30.204932+0000","last_undegraded":"2024-09-13T14:42:30.204932+0000","last_fullsized":"2024-09-13T14:42:30.204932+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:28:21.713025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"283'10008","reported_seq":25032,"reported_epoch":283,"state":"active+recovering","last_fresh":"2024-09-13T14:42:35.768515+0000","last_change":"2024-09-13T14:42:29.105072+0000","last_active":"2024-09-13T14:42:35.768515+0000","last_peered":"2024-09-13T14:42:35.768515+0000","last_clean":"2024-09-13T14:37:46.405415+0000","last_became_active":"2024-09-13T14:38:16.427510+0000","last_became_peered":"2024-09-13T14:38:16.427510+0000","last_unstale":"2024-09-13T14:42:35.768515+0000","last_undegraded":"2024-09-13T14:42:35.768515+0000","last_fullsized":"2024-09-13T14:42:35.768515+0000","mapping_epoch":245,"log_start":"213'4118","ondisk_log_start":"213'4118","created":205,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5890,"log_dups_size":0,"ondisk_log_size":5890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261488640,"num_objects":3990,"num_object_clones":0,"num_object_copies":11970,"num_objects_missing_on_primary":44,"num_objects_missing":44,"num_objects_degraded":0,"num_objects_misplaced":3989,"num_objects_unfound":0,"num_objects_dirty":3990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10008,"num_write_kb":447936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7470,"num_bytes_recovered":485949440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"0,3,6","objects":1},{"shards":"0,3,7","objects":3989}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":991,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-13T14:42:32.278368+0000","last_change":"2024-09-13T14:37:49.147498+0000","last_active":"2024-09-13T14:42:32.278368+0000","last_peered":"2024-09-13T14:42:32.278368+0000","last_clean":"2024-09-13T14:42:32.278368+0000","last_became_active":"2024-09-13T14:37:49.147028+0000","last_became_peered":"2024-09-13T14:37:49.147028+0000","last_unstale":"2024-09-13T14:42:32.278368+0000","last_undegraded":"2024-09-13T14:42:32.278368+0000","last_fullsized":"2024-09-13T14:42:32.278368+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:11:29.050178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":287,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-13T14:42:29.204490+0000","last_change":"2024-09-13T14:38:03.528133+0000","last_active":"2024-09-13T14:42:29.204490+0000","last_peered":"2024-09-13T14:42:29.204490+0000","last_clean":"2024-09-13T14:42:29.204490+0000","last_became_active":"2024-09-13T14:38:03.527829+0000","last_became_peered":"2024-09-13T14:38:03.527829+0000","last_unstale":"2024-09-13T14:42:29.204490+0000","last_undegraded":"2024-09-13T14:42:29.204490+0000","last_fullsized":"2024-09-13T14:42:29.204490+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:48.254169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":450,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.213251+0000","last_change":"2024-09-13T14:38:53.899755+0000","last_active":"2024-09-13T14:42:33.213251+0000","last_peered":"2024-09-13T14:42:33.213251+0000","last_clean":"2024-09-13T14:42:33.213251+0000","last_became_active":"2024-09-13T14:38:53.899448+0000","last_became_peered":"2024-09-13T14:38:53.899448+0000","last_unstale":"2024-09-13T14:42:33.213251+0000","last_undegraded":"2024-09-13T14:42:33.213251+0000","last_fullsized":"2024-09-13T14:42:33.213251+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:55.897035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"282'10274","reported_seq":10394,"reported_epoch":282,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:42:32.672073+0000","last_change":"2024-09-13T14:41:52.203221+0000","last_active":"2024-09-13T14:42:32.672073+0000","last_peered":"2024-09-13T14:42:32.672073+0000","last_clean":"2024-09-13T14:35:51.090791+0000","last_became_active":"2024-09-13T14:37:49.665025+0000","last_became_peered":"2024-09-13T14:37:49.665025+0000","last_unstale":"2024-09-13T14:42:32.672073+0000","last_undegraded":"2024-09-13T14:37:49.169101+0000","last_fullsized":"2024-09-13T14:37:49.121596+0000","mapping_epoch":240,"log_start":"212'3863","ondisk_log_start":"212'3863","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6411,"log_dups_size":0,"ondisk_log_size":6411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:02.321504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263979008,"num_objects":4028,"num_object_clones":0,"num_object_copies":12084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4496,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10274,"num_write_kb":457664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":393216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":-402}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":93,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.222936+0000","last_change":"2024-09-13T14:37:49.149770+0000","last_active":"2024-09-13T14:42:33.222936+0000","last_peered":"2024-09-13T14:42:33.222936+0000","last_clean":"2024-09-13T14:42:33.222936+0000","last_became_active":"2024-09-13T14:37:49.148991+0000","last_became_peered":"2024-09-13T14:37:49.148991+0000","last_unstale":"2024-09-13T14:42:33.222936+0000","last_undegraded":"2024-09-13T14:42:33.222936+0000","last_fullsized":"2024-09-13T14:42:33.222936+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:04:04.193366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":1148,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.223134+0000","last_change":"2024-09-13T14:30:58.928302+0000","last_active":"2024-09-13T14:42:33.223134+0000","last_peered":"2024-09-13T14:42:33.223134+0000","last_clean":"2024-09-13T14:42:33.223134+0000","last_became_active":"2024-09-13T14:30:58.927718+0000","last_became_peered":"2024-09-13T14:30:58.927718+0000","last_unstale":"2024-09-13T14:42:33.223134+0000","last_undegraded":"2024-09-13T14:42:33.223134+0000","last_fullsized":"2024-09-13T14:42:33.223134+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:37.323036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":100,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-13T14:42:32.278385+0000","last_change":"2024-09-13T14:38:03.534628+0000","last_active":"2024-09-13T14:42:32.278385+0000","last_peered":"2024-09-13T14:42:32.278385+0000","last_clean":"2024-09-13T14:42:32.278385+0000","last_became_active":"2024-09-13T14:38:03.534262+0000","last_became_peered":"2024-09-13T14:38:03.534262+0000","last_unstale":"2024-09-13T14:42:32.278385+0000","last_undegraded":"2024-09-13T14:42:32.278385+0000","last_fullsized":"2024-09-13T14:42:32.278385+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:31.168783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"283'10028","reported_seq":24198,"reported_epoch":283,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:42:37.056592+0000","last_change":"2024-09-13T14:38:53.944111+0000","last_active":"2024-09-13T14:42:37.056592+0000","last_peered":"2024-09-13T14:42:37.056592+0000","last_clean":"2024-09-13T14:37:47.047672+0000","last_became_active":"2024-09-13T14:37:49.664294+0000","last_became_peered":"2024-09-13T14:37:49.664294+0000","last_unstale":"2024-09-13T14:42:37.056592+0000","last_undegraded":"2024-09-13T14:38:53.940822+0000","last_fullsized":"2024-09-13T14:37:49.126627+0000","mapping_epoch":240,"log_start":"213'4300","ondisk_log_start":"213'4300","created":205,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5728,"log_dups_size":0,"ondisk_log_size":5728,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:11.766692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263716864,"num_objects":4024,"num_object_clones":0,"num_object_copies":12072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4535,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10028,"num_write_kb":449664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7009,"num_bytes_recovered":459341824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3],"avail_no_missing":["5","0","3"],"object_location_counts":[{"shards":"0,3,5","objects":4024},{"shards":"3,5","objects":-436}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":112,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.222886+0000","last_change":"2024-09-13T14:38:03.534303+0000","last_active":"2024-09-13T14:42:33.222886+0000","last_peered":"2024-09-13T14:42:33.222886+0000","last_clean":"2024-09-13T14:42:33.222886+0000","last_became_active":"2024-09-13T14:38:03.533171+0000","last_became_peered":"2024-09-13T14:38:03.533171+0000","last_unstale":"2024-09-13T14:42:33.222886+0000","last_undegraded":"2024-09-13T14:42:33.222886+0000","last_fullsized":"2024-09-13T14:42:33.222886+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:00:16.553526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":804,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-13T14:42:30.203685+0000","last_change":"2024-09-13T14:38:04.596192+0000","last_active":"2024-09-13T14:42:30.203685+0000","last_peered":"2024-09-13T14:42:30.203685+0000","last_clean":"2024-09-13T14:42:30.203685+0000","last_became_active":"2024-09-13T14:38:04.593299+0000","last_became_peered":"2024-09-13T14:38:04.593299+0000","last_unstale":"2024-09-13T14:42:30.203685+0000","last_undegraded":"2024-09-13T14:42:30.203685+0000","last_fullsized":"2024-09-13T14:42:30.203685+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:34.241410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":90,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.216194+0000","last_change":"2024-09-13T14:38:03.533120+0000","last_active":"2024-09-13T14:42:33.216194+0000","last_peered":"2024-09-13T14:42:33.216194+0000","last_clean":"2024-09-13T14:42:33.216194+0000","last_became_active":"2024-09-13T14:38:03.532829+0000","last_became_peered":"2024-09-13T14:38:03.532829+0000","last_unstale":"2024-09-13T14:42:33.216194+0000","last_undegraded":"2024-09-13T14:42:33.216194+0000","last_fullsized":"2024-09-13T14:42:33.216194+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:24.514661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"283'10255","reported_seq":19509,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:37.176671+0000","last_change":"2024-09-13T14:42:04.660877+0000","last_active":"2024-09-13T14:42:37.176671+0000","last_peered":"2024-09-13T14:42:37.176671+0000","last_clean":"2024-09-13T14:42:37.176671+0000","last_became_active":"2024-09-13T14:42:04.641680+0000","last_became_peered":"2024-09-13T14:42:04.641680+0000","last_unstale":"2024-09-13T14:42:37.176671+0000","last_undegraded":"2024-09-13T14:42:37.176671+0000","last_fullsized":"2024-09-13T14:42:37.176671+0000","mapping_epoch":277,"log_start":"213'4518","ondisk_log_start":"213'4518","created":205,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5737,"log_dups_size":0,"ondisk_log_size":5737,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:22:18.955428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271515648,"num_objects":4143,"num_object_clones":0,"num_object_copies":12429,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10255,"num_write_kb":460736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4533,"num_bytes_recovered":297074688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":492,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.212955+0000","last_change":"2024-09-13T14:38:53.938227+0000","last_active":"2024-09-13T14:42:33.212955+0000","last_peered":"2024-09-13T14:42:33.212955+0000","last_clean":"2024-09-13T14:42:33.212955+0000","last_became_active":"2024-09-13T14:38:53.902867+0000","last_became_peered":"2024-09-13T14:38:53.902867+0000","last_unstale":"2024-09-13T14:42:33.212955+0000","last_undegraded":"2024-09-13T14:42:33.212955+0000","last_fullsized":"2024-09-13T14:42:33.212955+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:16.831756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":581,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-13T14:42:29.202340+0000","last_change":"2024-09-13T14:37:48.147677+0000","last_active":"2024-09-13T14:42:29.202340+0000","last_peered":"2024-09-13T14:42:29.202340+0000","last_clean":"2024-09-13T14:42:29.202340+0000","last_became_active":"2024-09-13T14:37:48.147001+0000","last_became_peered":"2024-09-13T14:37:48.147001+0000","last_unstale":"2024-09-13T14:42:29.202340+0000","last_undegraded":"2024-09-13T14:42:29.202340+0000","last_fullsized":"2024-09-13T14:42:29.202340+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:35:00.653576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":473,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.212468+0000","last_change":"2024-09-13T14:38:53.900107+0000","last_active":"2024-09-13T14:42:33.212468+0000","last_peered":"2024-09-13T14:42:33.212468+0000","last_clean":"2024-09-13T14:42:33.212468+0000","last_became_active":"2024-09-13T14:38:53.899556+0000","last_became_peered":"2024-09-13T14:38:53.899556+0000","last_unstale":"2024-09-13T14:42:33.212468+0000","last_undegraded":"2024-09-13T14:42:33.212468+0000","last_fullsized":"2024-09-13T14:42:33.212468+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:15:47.918163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"283'9939","reported_seq":18919,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:37.462541+0000","last_change":"2024-09-13T14:39:54.921239+0000","last_active":"2024-09-13T14:42:37.462541+0000","last_peered":"2024-09-13T14:42:37.462541+0000","last_clean":"2024-09-13T14:42:37.462541+0000","last_became_active":"2024-09-13T14:39:54.907894+0000","last_became_peered":"2024-09-13T14:39:54.907894+0000","last_unstale":"2024-09-13T14:42:37.462541+0000","last_undegraded":"2024-09-13T14:42:37.462541+0000","last_fullsized":"2024-09-13T14:42:37.462541+0000","mapping_epoch":263,"log_start":"213'4247","ondisk_log_start":"213'4247","created":205,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5692,"log_dups_size":0,"ondisk_log_size":5692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:52:04.619238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256442368,"num_objects":3913,"num_object_clones":0,"num_object_copies":11739,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9939,"num_write_kb":443264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4395,"num_bytes_recovered":283770880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":105,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.216378+0000","last_change":"2024-09-13T14:38:53.901806+0000","last_active":"2024-09-13T14:42:33.216378+0000","last_peered":"2024-09-13T14:42:33.216378+0000","last_clean":"2024-09-13T14:42:33.216378+0000","last_became_active":"2024-09-13T14:38:53.891055+0000","last_became_peered":"2024-09-13T14:38:53.891055+0000","last_unstale":"2024-09-13T14:42:33.216378+0000","last_undegraded":"2024-09-13T14:42:33.216378+0000","last_fullsized":"2024-09-13T14:42:33.216378+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:03:00.416198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":39,"reported_epoch":282,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-13T14:42:30.205218+0000","last_change":"2024-09-13T14:38:53.900902+0000","last_active":"2024-09-13T14:42:30.205218+0000","last_peered":"2024-09-13T14:42:30.205218+0000","last_clean":"2024-09-13T14:38:39.635779+0000","last_became_active":"2024-09-13T14:38:53.900626+0000","last_became_peered":"2024-09-13T14:38:53.900626+0000","last_unstale":"2024-09-13T14:42:30.205218+0000","last_undegraded":"2024-09-13T14:38:53.885778+0000","last_fullsized":"2024-09-13T14:42:30.205218+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:37:21.715778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":7,"num_objects_missing":7,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":["3","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":271,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-13T14:42:29.201994+0000","last_change":"2024-09-13T14:37:49.172218+0000","last_active":"2024-09-13T14:42:29.201994+0000","last_peered":"2024-09-13T14:42:29.201994+0000","last_clean":"2024-09-13T14:42:29.201994+0000","last_became_active":"2024-09-13T14:37:49.171718+0000","last_became_peered":"2024-09-13T14:37:49.171718+0000","last_unstale":"2024-09-13T14:42:29.201994+0000","last_undegraded":"2024-09-13T14:42:29.201994+0000","last_fullsized":"2024-09-13T14:42:29.201994+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:46:54.626521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":294,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-13T14:42:29.204421+0000","last_change":"2024-09-13T14:37:49.230094+0000","last_active":"2024-09-13T14:42:29.204421+0000","last_peered":"2024-09-13T14:42:29.204421+0000","last_clean":"2024-09-13T14:42:29.204421+0000","last_became_active":"2024-09-13T14:37:49.229393+0000","last_became_peered":"2024-09-13T14:37:49.229393+0000","last_unstale":"2024-09-13T14:42:29.204421+0000","last_undegraded":"2024-09-13T14:42:29.204421+0000","last_fullsized":"2024-09-13T14:42:29.204421+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:02:28.690318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":274,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-13T14:42:29.202111+0000","last_change":"2024-09-13T14:37:48.128447+0000","last_active":"2024-09-13T14:42:29.202111+0000","last_peered":"2024-09-13T14:42:29.202111+0000","last_clean":"2024-09-13T14:42:29.202111+0000","last_became_active":"2024-09-13T14:37:48.128143+0000","last_became_peered":"2024-09-13T14:37:48.128143+0000","last_unstale":"2024-09-13T14:42:29.202111+0000","last_undegraded":"2024-09-13T14:42:29.202111+0000","last_fullsized":"2024-09-13T14:42:29.202111+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:22.204039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":89,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.215901+0000","last_change":"2024-09-13T14:38:03.531683+0000","last_active":"2024-09-13T14:42:33.215901+0000","last_peered":"2024-09-13T14:42:33.215901+0000","last_clean":"2024-09-13T14:42:33.215901+0000","last_became_active":"2024-09-13T14:38:03.531476+0000","last_became_peered":"2024-09-13T14:38:03.531476+0000","last_unstale":"2024-09-13T14:42:33.215901+0000","last_undegraded":"2024-09-13T14:42:33.215901+0000","last_fullsized":"2024-09-13T14:42:33.215901+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:41.557569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":47,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-13T14:42:30.203338+0000","last_change":"2024-09-13T14:38:53.954281+0000","last_active":"2024-09-13T14:42:30.203338+0000","last_peered":"2024-09-13T14:42:30.203338+0000","last_clean":"2024-09-13T14:42:30.203338+0000","last_became_active":"2024-09-13T14:38:53.900912+0000","last_became_peered":"2024-09-13T14:38:53.900912+0000","last_unstale":"2024-09-13T14:42:30.203338+0000","last_undegraded":"2024-09-13T14:42:30.203338+0000","last_fullsized":"2024-09-13T14:42:30.203338+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:47.876275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":99,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-13T14:42:30.204005+0000","last_change":"2024-09-13T14:38:54.000782+0000","last_active":"2024-09-13T14:42:30.204005+0000","last_peered":"2024-09-13T14:42:30.204005+0000","last_clean":"2024-09-13T14:42:30.204005+0000","last_became_active":"2024-09-13T14:38:54.000322+0000","last_became_peered":"2024-09-13T14:38:54.000322+0000","last_unstale":"2024-09-13T14:42:30.204005+0000","last_undegraded":"2024-09-13T14:42:30.204005+0000","last_fullsized":"2024-09-13T14:42:30.204005+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:09.057597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":415,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.214694+0000","last_change":"2024-09-13T14:38:53.899535+0000","last_active":"2024-09-13T14:42:33.214694+0000","last_peered":"2024-09-13T14:42:33.214694+0000","last_clean":"2024-09-13T14:42:33.214694+0000","last_became_active":"2024-09-13T14:38:53.899183+0000","last_became_peered":"2024-09-13T14:38:53.899183+0000","last_unstale":"2024-09-13T14:42:33.214694+0000","last_undegraded":"2024-09-13T14:42:33.214694+0000","last_fullsized":"2024-09-13T14:42:33.214694+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:33.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":89,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.268610+0000","last_change":"2024-09-13T14:38:03.543426+0000","last_active":"2024-09-13T14:42:33.268610+0000","last_peered":"2024-09-13T14:42:33.268610+0000","last_clean":"2024-09-13T14:42:33.268610+0000","last_became_active":"2024-09-13T14:38:03.542661+0000","last_became_peered":"2024-09-13T14:38:03.542661+0000","last_unstale":"2024-09-13T14:42:33.268610+0000","last_undegraded":"2024-09-13T14:42:33.268610+0000","last_fullsized":"2024-09-13T14:42:33.268610+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:54.050127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":444,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.212385+0000","last_change":"2024-09-13T14:37:48.131321+0000","last_active":"2024-09-13T14:42:33.212385+0000","last_peered":"2024-09-13T14:42:33.212385+0000","last_clean":"2024-09-13T14:42:33.212385+0000","last_became_active":"2024-09-13T14:37:48.130998+0000","last_became_peered":"2024-09-13T14:37:48.130998+0000","last_unstale":"2024-09-13T14:42:33.212385+0000","last_undegraded":"2024-09-13T14:42:33.212385+0000","last_fullsized":"2024-09-13T14:42:33.212385+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:33:26.597825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":112,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-13T14:42:30.204850+0000","last_change":"2024-09-13T14:38:03.535454+0000","last_active":"2024-09-13T14:42:30.204850+0000","last_peered":"2024-09-13T14:42:30.204850+0000","last_clean":"2024-09-13T14:42:30.204850+0000","last_became_active":"2024-09-13T14:38:03.535131+0000","last_became_peered":"2024-09-13T14:38:03.535131+0000","last_unstale":"2024-09-13T14:42:30.204850+0000","last_undegraded":"2024-09-13T14:42:30.204850+0000","last_fullsized":"2024-09-13T14:42:30.204850+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:24.477601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":469,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.212543+0000","last_change":"2024-09-13T14:38:03.524939+0000","last_active":"2024-09-13T14:42:33.212543+0000","last_peered":"2024-09-13T14:42:33.212543+0000","last_clean":"2024-09-13T14:42:33.212543+0000","last_became_active":"2024-09-13T14:38:03.524406+0000","last_became_peered":"2024-09-13T14:38:03.524406+0000","last_unstale":"2024-09-13T14:42:33.212543+0000","last_undegraded":"2024-09-13T14:42:33.212543+0000","last_fullsized":"2024-09-13T14:42:33.212543+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:19:45.257424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":97,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.216374+0000","last_change":"2024-09-13T14:38:53.892948+0000","last_active":"2024-09-13T14:42:33.216374+0000","last_peered":"2024-09-13T14:42:33.216374+0000","last_clean":"2024-09-13T14:42:33.216374+0000","last_became_active":"2024-09-13T14:38:53.892665+0000","last_became_peered":"2024-09-13T14:38:53.892665+0000","last_unstale":"2024-09-13T14:42:33.216374+0000","last_undegraded":"2024-09-13T14:42:33.216374+0000","last_fullsized":"2024-09-13T14:42:33.216374+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:08.292359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":417,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.215179+0000","last_change":"2024-09-13T14:37:48.139348+0000","last_active":"2024-09-13T14:42:33.215179+0000","last_peered":"2024-09-13T14:42:33.215179+0000","last_clean":"2024-09-13T14:42:33.215179+0000","last_became_active":"2024-09-13T14:37:48.139115+0000","last_became_peered":"2024-09-13T14:37:48.139115+0000","last_unstale":"2024-09-13T14:42:33.215179+0000","last_undegraded":"2024-09-13T14:42:33.215179+0000","last_fullsized":"2024-09-13T14:42:33.215179+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:32:20.413346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":458,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.212540+0000","last_change":"2024-09-13T14:37:48.172088+0000","last_active":"2024-09-13T14:42:33.212540+0000","last_peered":"2024-09-13T14:42:33.212540+0000","last_clean":"2024-09-13T14:42:33.212540+0000","last_became_active":"2024-09-13T14:37:48.171779+0000","last_became_peered":"2024-09-13T14:37:48.171779+0000","last_unstale":"2024-09-13T14:42:33.212540+0000","last_undegraded":"2024-09-13T14:42:33.212540+0000","last_fullsized":"2024-09-13T14:42:33.212540+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:02.852712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":338,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-13T14:42:29.202071+0000","last_change":"2024-09-13T14:38:53.890236+0000","last_active":"2024-09-13T14:42:29.202071+0000","last_peered":"2024-09-13T14:42:29.202071+0000","last_clean":"2024-09-13T14:42:29.202071+0000","last_became_active":"2024-09-13T14:38:53.889582+0000","last_became_peered":"2024-09-13T14:38:53.889582+0000","last_unstale":"2024-09-13T14:42:29.202071+0000","last_undegraded":"2024-09-13T14:42:29.202071+0000","last_fullsized":"2024-09-13T14:42:29.202071+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:14:29.334774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":444,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.213481+0000","last_change":"2024-09-13T14:38:03.534097+0000","last_active":"2024-09-13T14:42:33.213481+0000","last_peered":"2024-09-13T14:42:33.213481+0000","last_clean":"2024-09-13T14:42:33.213481+0000","last_became_active":"2024-09-13T14:38:03.533794+0000","last_became_peered":"2024-09-13T14:38:03.533794+0000","last_unstale":"2024-09-13T14:42:33.213481+0000","last_undegraded":"2024-09-13T14:42:33.213481+0000","last_fullsized":"2024-09-13T14:42:33.213481+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:23.272098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":89,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.216097+0000","last_change":"2024-09-13T14:38:03.527021+0000","last_active":"2024-09-13T14:42:33.216097+0000","last_peered":"2024-09-13T14:42:33.216097+0000","last_clean":"2024-09-13T14:42:33.216097+0000","last_became_active":"2024-09-13T14:38:03.526698+0000","last_became_peered":"2024-09-13T14:38:03.526698+0000","last_unstale":"2024-09-13T14:42:33.216097+0000","last_undegraded":"2024-09-13T14:42:33.216097+0000","last_fullsized":"2024-09-13T14:42:33.216097+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:01:35.734086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":282,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-13T14:42:29.204404+0000","last_change":"2024-09-13T14:27:12.834931+0000","last_active":"2024-09-13T14:42:29.204404+0000","last_peered":"2024-09-13T14:42:29.204404+0000","last_clean":"2024-09-13T14:42:29.204404+0000","last_became_active":"2024-09-13T14:27:12.834572+0000","last_became_peered":"2024-09-13T14:27:12.834572+0000","last_unstale":"2024-09-13T14:42:29.204404+0000","last_undegraded":"2024-09-13T14:42:29.204404+0000","last_fullsized":"2024-09-13T14:42:29.204404+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:54:21.155689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":96,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-13T14:42:30.203904+0000","last_change":"2024-09-13T14:37:49.176796+0000","last_active":"2024-09-13T14:42:30.203904+0000","last_peered":"2024-09-13T14:42:30.203904+0000","last_clean":"2024-09-13T14:42:30.203904+0000","last_became_active":"2024-09-13T14:37:49.176246+0000","last_became_peered":"2024-09-13T14:37:49.176246+0000","last_unstale":"2024-09-13T14:42:30.203904+0000","last_undegraded":"2024-09-13T14:42:30.203904+0000","last_fullsized":"2024-09-13T14:42:30.203904+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:29:24.936718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":293,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-13T14:42:29.201860+0000","last_change":"2024-09-13T14:38:03.526333+0000","last_active":"2024-09-13T14:42:29.201860+0000","last_peered":"2024-09-13T14:42:29.201860+0000","last_clean":"2024-09-13T14:42:29.201860+0000","last_became_active":"2024-09-13T14:38:03.526001+0000","last_became_peered":"2024-09-13T14:38:03.526001+0000","last_unstale":"2024-09-13T14:42:29.201860+0000","last_undegraded":"2024-09-13T14:42:29.201860+0000","last_fullsized":"2024-09-13T14:42:29.201860+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:25:25.181271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":96,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-13T14:42:30.204371+0000","last_change":"2024-09-13T14:37:49.149582+0000","last_active":"2024-09-13T14:42:30.204371+0000","last_peered":"2024-09-13T14:42:30.204371+0000","last_clean":"2024-09-13T14:42:30.204371+0000","last_became_active":"2024-09-13T14:37:49.148525+0000","last_became_peered":"2024-09-13T14:37:49.148525+0000","last_unstale":"2024-09-13T14:42:30.204371+0000","last_undegraded":"2024-09-13T14:42:30.204371+0000","last_fullsized":"2024-09-13T14:42:30.204371+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:48.655665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":490,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.215044+0000","last_change":"2024-09-13T14:38:03.535255+0000","last_active":"2024-09-13T14:42:33.215044+0000","last_peered":"2024-09-13T14:42:33.215044+0000","last_clean":"2024-09-13T14:42:33.215044+0000","last_became_active":"2024-09-13T14:38:03.533917+0000","last_became_peered":"2024-09-13T14:38:03.533917+0000","last_unstale":"2024-09-13T14:42:33.215044+0000","last_undegraded":"2024-09-13T14:42:33.215044+0000","last_fullsized":"2024-09-13T14:42:33.215044+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:33.559788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":453,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.214195+0000","last_change":"2024-09-13T14:37:48.146784+0000","last_active":"2024-09-13T14:42:33.214195+0000","last_peered":"2024-09-13T14:42:33.214195+0000","last_clean":"2024-09-13T14:42:33.214195+0000","last_became_active":"2024-09-13T14:37:48.146491+0000","last_became_peered":"2024-09-13T14:37:48.146491+0000","last_unstale":"2024-09-13T14:42:33.214195+0000","last_undegraded":"2024-09-13T14:42:33.214195+0000","last_fullsized":"2024-09-13T14:42:33.214195+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:30:54.772513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":273,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-13T14:42:29.201748+0000","last_change":"2024-09-13T14:37:48.127607+0000","last_active":"2024-09-13T14:42:29.201748+0000","last_peered":"2024-09-13T14:42:29.201748+0000","last_clean":"2024-09-13T14:42:29.201748+0000","last_became_active":"2024-09-13T14:37:48.127316+0000","last_became_peered":"2024-09-13T14:37:48.127316+0000","last_unstale":"2024-09-13T14:42:29.201748+0000","last_undegraded":"2024-09-13T14:42:29.201748+0000","last_fullsized":"2024-09-13T14:42:29.201748+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:53:37.678861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":96,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.268477+0000","last_change":"2024-09-13T14:38:03.563200+0000","last_active":"2024-09-13T14:42:33.268477+0000","last_peered":"2024-09-13T14:42:33.268477+0000","last_clean":"2024-09-13T14:42:33.268477+0000","last_became_active":"2024-09-13T14:38:03.534258+0000","last_became_peered":"2024-09-13T14:38:03.534258+0000","last_unstale":"2024-09-13T14:42:33.268477+0000","last_undegraded":"2024-09-13T14:42:33.268477+0000","last_fullsized":"2024-09-13T14:42:33.268477+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:49:07.814419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":4185784344,"num_objects":63871,"num_object_clones":0,"num_object_copies":191613,"num_objects_missing_on_primary":127,"num_objects_missing":127,"num_objects_degraded":15194,"num_objects_misplaced":5976,"num_objects_unfound":0,"num_objects_dirty":63871,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161755,"num_write_kb":7219969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72978,"num_bytes_recovered":4729208856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14416818176,"data_stored":14416805960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94678,"ondisk_log_size":94678,"up":48,"acting":46,"num_store_stats":6},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1538,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":17390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25132,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":7,"num_objects_missing":7,"num_objects_degraded":16,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16793600,"data_stored":16777292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":23,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2969600,"data_stored":2951840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234385,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2730348,"kb_used_data":2275940,"kb_used_omap":1,"kb_used_meta":454398,"kb_avail":91014804,"statfs":{"total":95995035648,"available":93199159296,"internally_reserved":0,"allocated":2330562560,"data_stored":2329038347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":465304011},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":239,"seq":1026497183803,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1808564,"kb_used_data":1565736,"kb_used_omap":8,"kb_used_meta":242807,"kb_avail":91936588,"statfs":{"total":95995035648,"available":94143066112,"internally_reserved":0,"allocated":1603313664,"data_stored":1601785926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8453,"internal_metadata":248635131},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757650,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2272540,"kb_used_data":1601556,"kb_used_omap":1,"kb_used_meta":670974,"kb_avail":91472612,"statfs":{"total":95995035648,"available":93667954688,"internally_reserved":0,"allocated":1639993344,"data_stored":1638480692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":687077830},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019293,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2846108,"kb_used_data":2243220,"kb_used_omap":1,"kb_used_meta":602878,"kb_avail":90899044,"statfs":{"total":95995035648,"available":93080621056,"internally_reserved":0,"allocated":2297057280,"data_stored":2295541146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":617347526},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435477,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3633176,"kb_used_data":3069776,"kb_used_omap":5,"kb_used_meta":563386,"kb_avail":90111976,"statfs":{"total":95995035648,"available":92274663424,"internally_reserved":0,"allocated":3143450624,"data_stored":3141941832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":576907390},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":446676599005,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":425201762531,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926055,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4062120,"kb_used_data":3372384,"kb_used_omap":7,"kb_used_meta":689720,"kb_avail":89683032,"statfs":{"total":95995035648,"available":91835424768,"internally_reserved":0,"allocated":3453321216,"data_stored":3451797207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":706273576},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2,0,5,3,2,7,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6860,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3441233920,"data_stored":3441229848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3137409024,"data_stored":3137404952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2290417664,"data_stored":2290417664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1631059968,"data_stored":1631059968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1593446400,"data_stored":1593442328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2323251200,"data_stored":2323251200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:42:43.907 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:43.931 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:43.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:43.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:44.033 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:44.057 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:44.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:44.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:44.158 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:44.179 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:44.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:44.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:44.281 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:44.305 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:44.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:44.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:44.407 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:44.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:44 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 15107/190686 objects degraded (7.922%), 6 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:42:44.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:44 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 15107/190686 objects degraded (7.922%), 6 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:42:44.434 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:44.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:44.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:44.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:44 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 15107/190686 objects degraded (7.922%), 6 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:42:44.537 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:44.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:44.558 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:44.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:44.659 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:44.687 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:44.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:44.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:44.762 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:42:44.762 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:42:44.790 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:44.811 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:44.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:44.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:44.913 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:44.938 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:44.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:44.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:45.040 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:45.065 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:45.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:45.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:45.167 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:45.190 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:45.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:45.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:45.292 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:45.309 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:42:45.315 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:45.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:45.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:45.416 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:45.439 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:45.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:45.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:45.541 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:45.564 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:45.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:45.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:45.665 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:45.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:45 smithi102 ceph-mon[110183]: pgmap v956: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.9 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 159 op/s; 15107/189381 objects degraded (7.977%); 5932/189381 objects misplaced (3.132%); 1.8 MiB/s, 36 objects/s recovering 2024-09-13T14:42:45.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:45 smithi102 ceph-mon[115269]: pgmap v956: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.9 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 159 op/s; 15107/189381 objects degraded (7.977%); 5932/189381 objects misplaced (3.132%); 1.8 MiB/s, 36 objects/s recovering 2024-09-13T14:42:45.687 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:45.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:45.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:45.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:45 smithi195 ceph-mon[104262]: pgmap v956: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.9 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 159 op/s; 15107/189381 objects degraded (7.977%); 5932/189381 objects misplaced (3.132%); 1.8 MiB/s, 36 objects/s recovering 2024-09-13T14:42:45.790 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:45.811 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:45.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:45.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:45.913 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:45.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:45.936 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:45.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:46.037 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:46.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:46.065 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:46.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:46.166 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:46.169 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:42:46.189 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:46.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:46.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:46.291 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:46.315 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:46.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:46.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:46.417 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:46.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:46.441 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:46.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:46.542 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:46.564 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:46.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:46.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:46.665 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:46.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:42:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:42:46] "GET /metrics HTTP/1.1" 200 36548 "" "Prometheus/2.51.0" 2024-09-13T14:42:46.687 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:46.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:46.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:46.789 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:46.816 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:46.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:46.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:46.862 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:42:46.918 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:46.941 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:46.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:46.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:47.017 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T14:42:47.043 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:47.099 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:47.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:47.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:47.114 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:47.201 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:47.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:47.230 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:47.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:47.256 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:47.257 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:47.331 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:47.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:47.358 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:47.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:47.459 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:47.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:47.483 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:47.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:47.532 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:47 smithi102 ceph-mon[110183]: pgmap v957: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.9 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 159 op/s; 15107/189381 objects degraded (7.977%); 5932/189381 objects misplaced (3.132%); 1.8 MiB/s, 36 objects/s recovering 2024-09-13T14:42:47.532 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:47 smithi102 ceph-mon[115269]: pgmap v957: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.9 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 159 op/s; 15107/189381 objects degraded (7.977%); 5932/189381 objects misplaced (3.132%); 1.8 MiB/s, 36 objects/s recovering 2024-09-13T14:42:47.584 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:47.608 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:47.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:47.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:47.710 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:47.734 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:47.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:47.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:47.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:47 smithi195 ceph-mon[104262]: pgmap v957: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.9 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 159 op/s; 15107/189381 objects degraded (7.977%); 5932/189381 objects misplaced (3.132%); 1.8 MiB/s, 36 objects/s recovering 2024-09-13T14:42:47.837 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:47.858 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:47.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:47.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:47.960 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:47.984 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:47.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:47.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:48.085 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:48.106 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:48.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:48.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:48.208 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:48.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:48.230 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:48.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:48.331 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:48.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:48.355 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:48.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:48.456 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:48.478 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:48.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:48.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:48.581 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:48.605 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:48.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:48.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:48.707 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:48.733 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:48.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:48.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:48.835 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:48.857 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:48.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:48.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:48.959 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:48.984 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:48.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:48.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:49.086 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:49.110 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:49.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:49.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:49.212 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:49.238 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:49.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:49.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:49.339 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:49.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:49.363 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:49.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:49.464 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:49.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:49.491 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:49.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:49.592 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:49.601 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:49 smithi102 ceph-mon[110183]: pgmap v958: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.8 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 164 op/s; 14996/186765 objects degraded (8.029%); 5716/186765 objects misplaced (3.061%); 3.0 MiB/s, 51 objects/s recovering 2024-09-13T14:42:49.601 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:49 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 14996/186765 objects degraded (8.029%), 6 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:42:49.601 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:49 smithi102 ceph-mon[115269]: pgmap v958: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.8 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 164 op/s; 14996/186765 objects degraded (8.029%); 5716/186765 objects misplaced (3.061%); 3.0 MiB/s, 51 objects/s recovering 2024-09-13T14:42:49.601 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:49 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 14996/186765 objects degraded (8.029%), 6 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:42:49.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:49.614 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:49.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:49.715 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:49.742 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:49.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:49.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:49.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:49 smithi195 ceph-mon[104262]: pgmap v958: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.8 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 164 op/s; 14996/186765 objects degraded (8.029%); 5716/186765 objects misplaced (3.061%); 3.0 MiB/s, 51 objects/s recovering 2024-09-13T14:42:49.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:49 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 14996/186765 objects degraded (8.029%), 6 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:42:49.843 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:49.868 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:49.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:49.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:49.969 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:49.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:49.993 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:49.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:50.094 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:50.117 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:50.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:50.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:50.219 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:50.241 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:50.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:50.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:50.343 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:50.368 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:50.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:50.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:50.470 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:50.492 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:50.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:50.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:50.594 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:50.615 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:50.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:50.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:50.717 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:50.739 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:50.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:50.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:50.840 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:50.862 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:50.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:50.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:50.964 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:50.987 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:50.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:50.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:51.089 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:51.116 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:51.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:51.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:51.218 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:51.242 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:51.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:51.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:51.344 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:51.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:51.370 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:51.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:51.471 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:51.493 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:51.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:51.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:51.594 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:51.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:51.617 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:51.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:51.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:51 smithi102 ceph-mon[110183]: pgmap v959: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.8 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 164 op/s; 14996/186606 objects degraded (8.036%); 5716/186606 objects misplaced (3.063%); 3.0 MiB/s, 51 objects/s recovering 2024-09-13T14:42:51.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:51 smithi102 ceph-mon[115269]: pgmap v959: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.8 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 164 op/s; 14996/186606 objects degraded (8.036%); 5716/186606 objects misplaced (3.063%); 3.0 MiB/s, 51 objects/s recovering 2024-09-13T14:42:51.718 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:51.744 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:51.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:51.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:51.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:51 smithi195 ceph-mon[104262]: pgmap v959: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.8 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 164 op/s; 14996/186606 objects degraded (8.036%); 5716/186606 objects misplaced (3.063%); 3.0 MiB/s, 51 objects/s recovering 2024-09-13T14:42:51.848 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:51.870 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:51.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:51.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:51.971 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:51.996 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:51.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:51.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:52.098 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:52.121 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:52.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:52.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:52.222 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:52.251 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:52.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:52.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:52.354 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:52.375 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:52.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:52.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:52.477 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:52.498 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:52.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:52.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:52.512 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:52.600 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:52.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:52.623 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:52.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:52.724 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:52.748 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:52.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:52.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:52.850 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:52.875 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:52.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:52.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:52.976 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:52.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:52.999 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:53.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:53.101 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:53.129 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:53.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:53.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:53.149 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:42:53.149 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:42:53.171 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:42:53.171 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:42:53.199 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:42:53.200 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:42:53.223 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:42:53.223 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:42:53.231 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:53.244 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:42:53.245 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:42:53.255 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:53.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:53.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:53.266 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:42:53.267 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:42:53.285 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:42:53.285 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:42:53.306 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:42:53.307 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:42:53.356 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:53.380 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:53.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:53.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:53.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:53 smithi102 ceph-mon[110183]: pgmap v960: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.8 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 171 op/s; 14861/185544 objects degraded (8.009%); 5295/185544 objects misplaced (2.854%); 5.5 MiB/s, 89 objects/s recovering 2024-09-13T14:42:53.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:53 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1660446154' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:42:53.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:53 smithi102 ceph-mon[115269]: pgmap v960: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.8 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 171 op/s; 14861/185544 objects degraded (8.009%); 5295/185544 objects misplaced (2.854%); 5.5 MiB/s, 89 objects/s recovering 2024-09-13T14:42:53.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:53 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1660446154' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:42:53.481 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:53.504 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:53.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:53.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:53.606 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:53.632 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:53.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:53.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:53.735 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:53.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:53 smithi195 ceph-mon[104262]: pgmap v960: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.8 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 171 op/s; 14861/185544 objects degraded (8.009%); 5295/185544 objects misplaced (2.854%); 5.5 MiB/s, 89 objects/s recovering 2024-09-13T14:42:53.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:53 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1660446154' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:42:53.757 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:53.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:53.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:53.859 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:53.883 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:53.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:53.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:53.985 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:54.011 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:54.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:54.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:54.113 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:54.135 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:54.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:54.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:54.237 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:54.259 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:54.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:54.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:54.360 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:54.383 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:54.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:54.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:54.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:54 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 14861/185544 objects degraded (8.009%), 6 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:42:54.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:54 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 14861/185544 objects degraded (8.009%), 6 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:42:54.486 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:54.513 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:54.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:54.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:54.614 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:54.636 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:54.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:54.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:54.738 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:54.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:54 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 14861/185544 objects degraded (8.009%), 6 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:42:54.759 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:54.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:54.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:54.861 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:54.886 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:54.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:54.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:54.985 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:42:54.986 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:42:54.988 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:55.018 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:55.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:55.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:55.121 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:55.145 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:55.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:55.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:55.247 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:55.271 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:55.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:55.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:55.373 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:55.394 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:55.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:55.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:55.495 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:55.519 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:55.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:55.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:55.621 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:55.641 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:55 smithi102 ceph-mon[110183]: pgmap v961: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.7 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 208 op/s; 14861/183636 objects degraded (8.093%); 5295/183636 objects misplaced (2.883%); 4.9 MiB/s, 78 objects/s recovering 2024-09-13T14:42:55.642 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:55 smithi102 ceph-mon[115269]: pgmap v961: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.7 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 208 op/s; 14861/183636 objects degraded (8.093%); 5295/183636 objects misplaced (2.883%); 4.9 MiB/s, 78 objects/s recovering 2024-09-13T14:42:55.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:55.643 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:55.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:55.744 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:55.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:55 smithi195 ceph-mon[104262]: pgmap v961: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.7 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 208 op/s; 14861/183636 objects degraded (8.093%); 5295/183636 objects misplaced (2.883%); 4.9 MiB/s, 78 objects/s recovering 2024-09-13T14:42:55.765 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:55.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:55.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:55.867 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:55.890 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:55.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:55.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:55.991 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:56.015 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:56.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:56.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:56.117 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:56.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:56.143 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:56.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:56.244 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:56.266 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:56.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:56.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:56.368 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:56.390 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:56.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:56.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:56.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:56 smithi102 ceph-mon[115269]: from='client.45854 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:42:56.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:42:56.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:42:56.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:56 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1677160535' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:42:56.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:56 smithi102 ceph-mon[110183]: from='client.45854 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:42:56.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:42:56.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:42:56.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:56 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1677160535' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:42:56.436 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is set 2024-09-13T14:42:56.492 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:56.517 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:56.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:56.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:56.619 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:56.642 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:56.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:56.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:56.744 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:56.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:56 smithi195 ceph-mon[104262]: from='client.45854 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:42:56.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:42:56.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:42:56.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:56 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1677160535' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:42:56.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:56.770 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:56.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:56.871 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:56.894 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:56.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:56.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:56.918 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:42:56 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:42:56] "GET /metrics HTTP/1.1" 200 36551 "" "Prometheus/2.51.0" 2024-09-13T14:42:56.997 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:57.019 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:57.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:57.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:57.120 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:57.144 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:57.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:57.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:57.245 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:57.276 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:57.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:57.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:57.378 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:57.402 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:57.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:57.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:57.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:57 smithi102 ceph-mon[110183]: pgmap v962: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.7 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 159 op/s; 14861/183636 objects degraded (8.093%); 5295/183636 objects misplaced (2.883%); 4.6 MiB/s, 74 objects/s recovering 2024-09-13T14:42:57.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:57 smithi102 ceph-mon[110183]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:42:57.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:57 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1677160535' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:42:57.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:57 smithi102 ceph-mon[110183]: osdmap e285: 8 total, 8 up, 6 in 2024-09-13T14:42:57.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:57 smithi102 ceph-mon[115269]: pgmap v962: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.7 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 159 op/s; 14861/183636 objects degraded (8.093%); 5295/183636 objects misplaced (2.883%); 4.6 MiB/s, 74 objects/s recovering 2024-09-13T14:42:57.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:57 smithi102 ceph-mon[115269]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:42:57.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:57 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1677160535' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:42:57.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:57 smithi102 ceph-mon[115269]: osdmap e285: 8 total, 8 up, 6 in 2024-09-13T14:42:57.504 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:57.527 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:57.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:57.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:57.630 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:57.654 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:57.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:57.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:57.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:57 smithi195 ceph-mon[104262]: pgmap v962: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.7 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 159 op/s; 14861/183636 objects degraded (8.093%); 5295/183636 objects misplaced (2.883%); 4.6 MiB/s, 74 objects/s recovering 2024-09-13T14:42:57.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:57 smithi195 ceph-mon[104262]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:42:57.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:57 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1677160535' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:42:57.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:57 smithi195 ceph-mon[104262]: osdmap e285: 8 total, 8 up, 6 in 2024-09-13T14:42:57.756 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:57.780 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:57.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:57.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:57.882 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:57.907 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:57.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:57.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:58.008 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:58.032 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:58.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:58.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:58.134 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:42:58.161 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:58.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:58.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:42:58.221 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":961,"stamp":"2024-09-13T14:42:53.519575+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4014479357,"num_objects":61212,"num_object_clones":0,"num_object_copies":183636,"num_objects_missing_on_primary":90,"num_objects_missing":90,"num_objects_degraded":14861,"num_objects_misplaced":5295,"num_objects_unfound":0,"num_objects_dirty":61212,"num_whiteouts":0,"num_read":6377,"num_read_kb":13834,"num_write":165565,"num_write_kb":7235010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74161,"num_bytes_recovered":4802743437,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95647,"ondisk_log_size":95647,"up":267,"acting":264,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":302,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":16860456,"kb_used_data":13643000,"kb_used_omap":26,"kb_used_meta":3217381,"kb_avail":545610456,"statfs":{"total":575970213888,"available":558705106944,"internally_reserved":0,"allocated":13970432000,"data_stored":13961290497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27144,"internal_metadata":3294598648},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-154009600,"num_objects":-2350,"num_object_clones":0,"num_object_copies":-7050,"num_objects_missing_on_primary":-7,"num_objects_missing":-7,"num_objects_degraded":-246,"num_objects_misplaced":-637,"num_objects_unfound":0,"num_objects_dirty":-2350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2350,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":891,"num_bytes_recovered":57933824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.279321"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":461,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-13T14:42:42.245869+0000","last_change":"2024-09-13T14:36:13.859927+0000","last_active":"2024-09-13T14:42:42.245869+0000","last_peered":"2024-09-13T14:42:42.245869+0000","last_clean":"2024-09-13T14:42:42.245869+0000","last_became_active":"2024-09-13T14:36:13.859505+0000","last_became_peered":"2024-09-13T14:36:13.859505+0000","last_unstale":"2024-09-13T14:42:42.245869+0000","last_undegraded":"2024-09-13T14:42:42.245869+0000","last_fullsized":"2024-09-13T14:42:42.245869+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:12:20.989627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":286,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.208451+0000","last_change":"2024-09-13T14:37:48.130968+0000","last_active":"2024-09-13T14:42:33.208451+0000","last_peered":"2024-09-13T14:42:33.208451+0000","last_clean":"2024-09-13T14:42:33.208451+0000","last_became_active":"2024-09-13T14:37:48.130603+0000","last_became_peered":"2024-09-13T14:37:48.130603+0000","last_unstale":"2024-09-13T14:42:33.208451+0000","last_undegraded":"2024-09-13T14:42:33.208451+0000","last_fullsized":"2024-09-13T14:42:33.208451+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:42.626386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":82,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.223032+0000","last_change":"2024-09-13T14:36:13.862181+0000","last_active":"2024-09-13T14:42:33.223032+0000","last_peered":"2024-09-13T14:42:33.223032+0000","last_clean":"2024-09-13T14:42:33.223032+0000","last_became_active":"2024-09-13T14:36:13.861873+0000","last_became_peered":"2024-09-13T14:36:13.861873+0000","last_unstale":"2024-09-13T14:42:33.223032+0000","last_undegraded":"2024-09-13T14:42:33.223032+0000","last_fullsized":"2024-09-13T14:42:33.223032+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:14.085700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":98,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-13T14:42:42.249680+0000","last_change":"2024-09-13T14:38:53.896672+0000","last_active":"2024-09-13T14:42:42.249680+0000","last_peered":"2024-09-13T14:42:42.249680+0000","last_clean":"2024-09-13T14:42:42.249680+0000","last_became_active":"2024-09-13T14:38:53.896346+0000","last_became_peered":"2024-09-13T14:38:53.896346+0000","last_unstale":"2024-09-13T14:42:42.249680+0000","last_undegraded":"2024-09-13T14:42:42.249680+0000","last_fullsized":"2024-09-13T14:42:42.249680+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:57.789686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":98,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-13T14:42:42.250472+0000","last_change":"2024-09-13T14:38:53.893021+0000","last_active":"2024-09-13T14:42:42.250472+0000","last_peered":"2024-09-13T14:42:42.250472+0000","last_clean":"2024-09-13T14:42:42.250472+0000","last_became_active":"2024-09-13T14:38:53.892668+0000","last_became_peered":"2024-09-13T14:38:53.892668+0000","last_unstale":"2024-09-13T14:42:42.250472+0000","last_undegraded":"2024-09-13T14:42:42.250472+0000","last_fullsized":"2024-09-13T14:42:42.250472+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:57.180738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":254,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.208323+0000","last_change":"2024-09-13T14:37:48.134642+0000","last_active":"2024-09-13T14:42:33.208323+0000","last_peered":"2024-09-13T14:42:33.208323+0000","last_clean":"2024-09-13T14:42:33.208323+0000","last_became_active":"2024-09-13T14:37:48.134402+0000","last_became_peered":"2024-09-13T14:37:48.134402+0000","last_unstale":"2024-09-13T14:42:33.208323+0000","last_undegraded":"2024-09-13T14:42:33.208323+0000","last_fullsized":"2024-09-13T14:42:33.208323+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:51.313663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":480,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-13T14:42:42.245415+0000","last_change":"2024-09-13T14:38:53.903022+0000","last_active":"2024-09-13T14:42:42.245415+0000","last_peered":"2024-09-13T14:42:42.245415+0000","last_clean":"2024-09-13T14:42:42.245415+0000","last_became_active":"2024-09-13T14:38:53.902448+0000","last_became_peered":"2024-09-13T14:38:53.902448+0000","last_unstale":"2024-09-13T14:42:42.245415+0000","last_undegraded":"2024-09-13T14:42:42.245415+0000","last_fullsized":"2024-09-13T14:42:42.245415+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:24.690401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":102,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-13T14:42:42.249281+0000","last_change":"2024-09-13T14:38:03.554500+0000","last_active":"2024-09-13T14:42:42.249281+0000","last_peered":"2024-09-13T14:42:42.249281+0000","last_clean":"2024-09-13T14:42:42.249281+0000","last_became_active":"2024-09-13T14:38:03.532910+0000","last_became_peered":"2024-09-13T14:38:03.532910+0000","last_unstale":"2024-09-13T14:42:42.249281+0000","last_undegraded":"2024-09-13T14:42:42.249281+0000","last_fullsized":"2024-09-13T14:42:42.249281+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:27:50.843498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"284'10194","reported_seq":21089,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-13T14:42:52.087958+0000","last_change":"2024-09-13T14:41:53.220083+0000","last_active":"2024-09-13T14:42:52.087958+0000","last_peered":"2024-09-13T14:42:52.087958+0000","last_clean":"2024-09-13T14:42:52.087958+0000","last_became_active":"2024-09-13T14:41:53.200084+0000","last_became_peered":"2024-09-13T14:41:53.200084+0000","last_unstale":"2024-09-13T14:42:52.087958+0000","last_undegraded":"2024-09-13T14:42:52.087958+0000","last_fullsized":"2024-09-13T14:42:52.087958+0000","mapping_epoch":275,"log_start":"213'4500","ondisk_log_start":"213'4500","created":205,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5694,"log_dups_size":0,"ondisk_log_size":5694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:08.897524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241565696,"num_objects":3686,"num_object_clones":0,"num_object_copies":11058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10194,"num_write_kb":444160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9152,"num_bytes_recovered":594608128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":46,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.208961+0000","last_change":"2024-09-13T14:38:53.940238+0000","last_active":"2024-09-13T14:42:33.208961+0000","last_peered":"2024-09-13T14:42:33.208961+0000","last_clean":"2024-09-13T14:42:33.208961+0000","last_became_active":"2024-09-13T14:38:53.892825+0000","last_became_peered":"2024-09-13T14:38:53.892825+0000","last_unstale":"2024-09-13T14:42:33.208961+0000","last_undegraded":"2024-09-13T14:42:33.208961+0000","last_fullsized":"2024-09-13T14:42:33.208961+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:46.666581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":103,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-13T14:42:42.289551+0000","last_change":"2024-09-13T14:38:53.898907+0000","last_active":"2024-09-13T14:42:42.289551+0000","last_peered":"2024-09-13T14:42:42.289551+0000","last_clean":"2024-09-13T14:42:42.289551+0000","last_became_active":"2024-09-13T14:38:53.898572+0000","last_became_peered":"2024-09-13T14:38:53.898572+0000","last_unstale":"2024-09-13T14:42:42.289551+0000","last_undegraded":"2024-09-13T14:42:42.289551+0000","last_fullsized":"2024-09-13T14:42:42.289551+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:29.222150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"284'10444","reported_seq":24622,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-13T14:42:52.490493+0000","last_change":"2024-09-13T14:38:57.930032+0000","last_active":"2024-09-13T14:42:52.490493+0000","last_peered":"2024-09-13T14:42:52.490493+0000","last_clean":"2024-09-13T14:42:52.490493+0000","last_became_active":"2024-09-13T14:38:57.921458+0000","last_became_peered":"2024-09-13T14:38:57.921458+0000","last_unstale":"2024-09-13T14:42:52.490493+0000","last_undegraded":"2024-09-13T14:42:52.490493+0000","last_fullsized":"2024-09-13T14:42:52.490493+0000","mapping_epoch":256,"log_start":"213'4739","ondisk_log_start":"213'4739","created":205,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5705,"log_dups_size":0,"ondisk_log_size":5705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:27:12.294038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248905728,"num_objects":3798,"num_object_clones":0,"num_object_copies":11394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10444,"num_write_kb":455744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6990,"num_bytes_recovered":458096640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":46,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.208815+0000","last_change":"2024-09-13T14:38:53.904885+0000","last_active":"2024-09-13T14:42:33.208815+0000","last_peered":"2024-09-13T14:42:33.208815+0000","last_clean":"2024-09-13T14:42:33.208815+0000","last_became_active":"2024-09-13T14:38:53.891726+0000","last_became_peered":"2024-09-13T14:38:53.891726+0000","last_unstale":"2024-09-13T14:42:33.208815+0000","last_undegraded":"2024-09-13T14:42:33.208815+0000","last_fullsized":"2024-09-13T14:42:33.208815+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:57.079205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":292,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.206603+0000","last_change":"2024-09-13T14:37:48.160504+0000","last_active":"2024-09-13T14:42:33.206603+0000","last_peered":"2024-09-13T14:42:33.206603+0000","last_clean":"2024-09-13T14:42:33.206603+0000","last_became_active":"2024-09-13T14:37:48.160181+0000","last_became_peered":"2024-09-13T14:37:48.160181+0000","last_unstale":"2024-09-13T14:42:33.206603+0000","last_undegraded":"2024-09-13T14:42:33.206603+0000","last_fullsized":"2024-09-13T14:42:33.206603+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:37.087008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"284'10326","reported_seq":12126,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-13T14:42:50.407304+0000","last_change":"2024-09-13T14:38:04.531914+0000","last_active":"2024-09-13T14:42:50.407304+0000","last_peered":"2024-09-13T14:42:50.407304+0000","last_clean":"2024-09-13T14:42:50.407304+0000","last_became_active":"2024-09-13T14:38:04.523448+0000","last_became_peered":"2024-09-13T14:38:04.523448+0000","last_unstale":"2024-09-13T14:42:50.407304+0000","last_undegraded":"2024-09-13T14:42:50.407304+0000","last_fullsized":"2024-09-13T14:42:50.407304+0000","mapping_epoch":245,"log_start":"213'4400","ondisk_log_start":"213'4400","created":205,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5926,"log_dups_size":0,"ondisk_log_size":5926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:06.536896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252182528,"num_objects":3848,"num_object_clones":0,"num_object_copies":11544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10326,"num_write_kb":453568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":811,"num_bytes_recovered":53149696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":283,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.208813+0000","last_change":"2024-09-13T14:37:49.611774+0000","last_active":"2024-09-13T14:42:33.208813+0000","last_peered":"2024-09-13T14:42:33.208813+0000","last_clean":"2024-09-13T14:42:33.208813+0000","last_became_active":"2024-09-13T14:37:49.611088+0000","last_became_peered":"2024-09-13T14:37:49.611088+0000","last_unstale":"2024-09-13T14:42:33.208813+0000","last_undegraded":"2024-09-13T14:42:33.208813+0000","last_fullsized":"2024-09-13T14:42:33.208813+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:08.933213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":42,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.208268+0000","last_change":"2024-09-13T14:38:53.899211+0000","last_active":"2024-09-13T14:42:33.208268+0000","last_peered":"2024-09-13T14:42:33.208268+0000","last_clean":"2024-09-13T14:42:33.208268+0000","last_became_active":"2024-09-13T14:38:53.898927+0000","last_became_peered":"2024-09-13T14:38:53.898927+0000","last_unstale":"2024-09-13T14:42:33.208268+0000","last_undegraded":"2024-09-13T14:42:33.208268+0000","last_fullsized":"2024-09-13T14:42:33.208268+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:00:28.857611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"284'10181","reported_seq":10272,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-13T14:42:52.180615+0000","last_change":"2024-09-13T14:35:24.004651+0000","last_active":"2024-09-13T14:42:52.180615+0000","last_peered":"2024-09-13T14:42:52.180615+0000","last_clean":"2024-09-13T14:42:52.180615+0000","last_became_active":"2024-09-13T14:35:24.004358+0000","last_became_peered":"2024-09-13T14:35:24.004358+0000","last_unstale":"2024-09-13T14:42:52.180615+0000","last_undegraded":"2024-09-13T14:42:52.180615+0000","last_fullsized":"2024-09-13T14:42:52.180615+0000","mapping_epoch":205,"log_start":"212'3884","ondisk_log_start":"212'3884","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6297,"log_dups_size":0,"ondisk_log_size":6297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:57:38.023669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248446976,"num_objects":3791,"num_object_clones":0,"num_object_copies":11373,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10181,"num_write_kb":447104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":93,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.222801+0000","last_change":"2024-09-13T14:37:49.168003+0000","last_active":"2024-09-13T14:42:33.222801+0000","last_peered":"2024-09-13T14:42:33.222801+0000","last_clean":"2024-09-13T14:42:33.222801+0000","last_became_active":"2024-09-13T14:37:49.167638+0000","last_became_peered":"2024-09-13T14:37:49.167638+0000","last_unstale":"2024-09-13T14:42:33.222801+0000","last_undegraded":"2024-09-13T14:42:33.222801+0000","last_fullsized":"2024-09-13T14:42:33.222801+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:29.051981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":92,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-13T14:42:42.250073+0000","last_change":"2024-09-13T14:38:03.528381+0000","last_active":"2024-09-13T14:42:42.250073+0000","last_peered":"2024-09-13T14:42:42.250073+0000","last_clean":"2024-09-13T14:42:42.250073+0000","last_became_active":"2024-09-13T14:38:03.527282+0000","last_became_peered":"2024-09-13T14:38:03.527282+0000","last_unstale":"2024-09-13T14:42:42.250073+0000","last_undegraded":"2024-09-13T14:42:42.250073+0000","last_fullsized":"2024-09-13T14:42:42.250073+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:24.435067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.f","version":"284'10165","reported_seq":10460,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-13T14:42:50.783847+0000","last_change":"2024-09-13T14:38:16.371079+0000","last_active":"2024-09-13T14:42:50.783847+0000","last_peered":"2024-09-13T14:42:50.783847+0000","last_clean":"2024-09-13T14:42:50.783847+0000","last_became_active":"2024-09-13T14:38:12.859883+0000","last_became_peered":"2024-09-13T14:38:12.859883+0000","last_unstale":"2024-09-13T14:42:50.783847+0000","last_undegraded":"2024-09-13T14:42:50.783847+0000","last_fullsized":"2024-09-13T14:42:50.783847+0000","mapping_epoch":245,"log_start":"213'4263","ondisk_log_start":"213'4263","created":205,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5902,"log_dups_size":0,"ondisk_log_size":5902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:08.436532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256966656,"num_objects":3921,"num_object_clones":0,"num_object_copies":11763,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10165,"num_write_kb":450752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":3670016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":450,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-13T14:42:42.245905+0000","last_change":"2024-09-13T14:38:03.549595+0000","last_active":"2024-09-13T14:42:42.245905+0000","last_peered":"2024-09-13T14:42:42.245905+0000","last_clean":"2024-09-13T14:42:42.245905+0000","last_became_active":"2024-09-13T14:38:03.533331+0000","last_became_peered":"2024-09-13T14:38:03.533331+0000","last_unstale":"2024-09-13T14:42:42.245905+0000","last_undegraded":"2024-09-13T14:42:42.245905+0000","last_fullsized":"2024-09-13T14:42:42.245905+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:59:02.470343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":267,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.206798+0000","last_change":"2024-09-13T14:37:48.127600+0000","last_active":"2024-09-13T14:42:33.206798+0000","last_peered":"2024-09-13T14:42:33.206798+0000","last_clean":"2024-09-13T14:42:33.206798+0000","last_became_active":"2024-09-13T14:37:48.127349+0000","last_became_peered":"2024-09-13T14:37:48.127349+0000","last_unstale":"2024-09-13T14:42:33.206798+0000","last_undegraded":"2024-09-13T14:42:33.206798+0000","last_fullsized":"2024-09-13T14:42:33.206798+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:57.268374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"284'10274","reported_seq":20262,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-13T14:42:52.248865+0000","last_change":"2024-09-13T14:42:30.227259+0000","last_active":"2024-09-13T14:42:52.248865+0000","last_peered":"2024-09-13T14:42:52.248865+0000","last_clean":"2024-09-13T14:42:52.248865+0000","last_became_active":"2024-09-13T14:42:30.209102+0000","last_became_peered":"2024-09-13T14:42:30.209102+0000","last_unstale":"2024-09-13T14:42:52.248865+0000","last_undegraded":"2024-09-13T14:42:52.248865+0000","last_fullsized":"2024-09-13T14:42:52.248865+0000","mapping_epoch":281,"log_start":"213'4600","ondisk_log_start":"213'4600","created":205,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5674,"log_dups_size":0,"ondisk_log_size":5674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:56:16.193287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244842496,"num_objects":3736,"num_object_clones":0,"num_object_copies":11208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10274,"num_write_kb":448320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4926,"num_bytes_recovered":313851904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":201,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:37.796782+0000","last_change":"2024-09-13T14:37:48.127604+0000","last_active":"2024-09-13T14:42:37.796782+0000","last_peered":"2024-09-13T14:42:37.796782+0000","last_clean":"2024-09-13T14:42:37.796782+0000","last_became_active":"2024-09-13T14:37:48.127300+0000","last_became_peered":"2024-09-13T14:37:48.127300+0000","last_unstale":"2024-09-13T14:42:37.796782+0000","last_undegraded":"2024-09-13T14:42:37.796782+0000","last_fullsized":"2024-09-13T14:42:37.796782+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:51.999506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":320,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.206922+0000","last_change":"2024-09-13T14:38:03.529967+0000","last_active":"2024-09-13T14:42:33.206922+0000","last_peered":"2024-09-13T14:42:33.206922+0000","last_clean":"2024-09-13T14:42:33.206922+0000","last_became_active":"2024-09-13T14:38:03.529646+0000","last_became_peered":"2024-09-13T14:38:03.529646+0000","last_unstale":"2024-09-13T14:42:33.206922+0000","last_undegraded":"2024-09-13T14:42:33.206922+0000","last_fullsized":"2024-09-13T14:42:33.206922+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:52:05.193279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"284'10299","reported_seq":19469,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-13T14:42:50.547320+0000","last_change":"2024-09-13T14:40:34.583237+0000","last_active":"2024-09-13T14:42:50.547320+0000","last_peered":"2024-09-13T14:42:50.547320+0000","last_clean":"2024-09-13T14:42:50.547320+0000","last_became_active":"2024-09-13T14:40:34.563989+0000","last_became_peered":"2024-09-13T14:40:34.563989+0000","last_unstale":"2024-09-13T14:42:50.547320+0000","last_undegraded":"2024-09-13T14:42:50.547320+0000","last_fullsized":"2024-09-13T14:42:50.547320+0000","mapping_epoch":266,"log_start":"213'4400","ondisk_log_start":"213'4400","created":205,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5899,"log_dups_size":0,"ondisk_log_size":5899,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:34:43.006051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251658264,"num_objects":3841,"num_object_clones":0,"num_object_copies":11523,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3841,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10299,"num_write_kb":452417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4489,"num_bytes_recovered":289865752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":281,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.206955+0000","last_change":"2024-09-13T14:36:13.860367+0000","last_active":"2024-09-13T14:42:33.206955+0000","last_peered":"2024-09-13T14:42:33.206955+0000","last_clean":"2024-09-13T14:42:33.206955+0000","last_became_active":"2024-09-13T14:36:13.859982+0000","last_became_peered":"2024-09-13T14:36:13.859982+0000","last_unstale":"2024-09-13T14:42:33.206955+0000","last_undegraded":"2024-09-13T14:42:33.206955+0000","last_fullsized":"2024-09-13T14:42:33.206955+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:08:23.311350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":276,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.207017+0000","last_change":"2024-09-13T14:37:48.147360+0000","last_active":"2024-09-13T14:42:33.207017+0000","last_peered":"2024-09-13T14:42:33.207017+0000","last_clean":"2024-09-13T14:42:33.207017+0000","last_became_active":"2024-09-13T14:37:48.146885+0000","last_became_peered":"2024-09-13T14:37:48.146885+0000","last_unstale":"2024-09-13T14:42:33.207017+0000","last_undegraded":"2024-09-13T14:42:33.207017+0000","last_fullsized":"2024-09-13T14:42:33.207017+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:27.841810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"284'10241","reported_seq":19518,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-13T14:42:52.240859+0000","last_change":"2024-09-13T14:38:39.650332+0000","last_active":"2024-09-13T14:42:52.240859+0000","last_peered":"2024-09-13T14:42:52.240859+0000","last_clean":"2024-09-13T14:42:52.240859+0000","last_became_active":"2024-09-13T14:38:39.640835+0000","last_became_peered":"2024-09-13T14:38:39.640835+0000","last_unstale":"2024-09-13T14:42:52.240859+0000","last_undegraded":"2024-09-13T14:42:52.240859+0000","last_fullsized":"2024-09-13T14:42:52.240859+0000","mapping_epoch":251,"log_start":"213'4551","ondisk_log_start":"213'4551","created":205,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5690,"log_dups_size":0,"ondisk_log_size":5690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:48.176606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251199488,"num_objects":3833,"num_object_clones":0,"num_object_copies":11499,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10241,"num_write_kb":450368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4544,"num_bytes_recovered":292945920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":468,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-13T14:42:42.245654+0000","last_change":"2024-09-13T14:38:53.900041+0000","last_active":"2024-09-13T14:42:42.245654+0000","last_peered":"2024-09-13T14:42:42.245654+0000","last_clean":"2024-09-13T14:42:42.245654+0000","last_became_active":"2024-09-13T14:38:53.899203+0000","last_became_peered":"2024-09-13T14:38:53.899203+0000","last_unstale":"2024-09-13T14:42:42.245654+0000","last_undegraded":"2024-09-13T14:42:42.245654+0000","last_fullsized":"2024-09-13T14:42:42.245654+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:28:07.883094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":400,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-13T14:42:42.245554+0000","last_change":"2024-09-13T14:26:15.823198+0000","last_active":"2024-09-13T14:42:42.245554+0000","last_peered":"2024-09-13T14:42:42.245554+0000","last_clean":"2024-09-13T14:42:42.245554+0000","last_became_active":"2024-09-13T14:26:15.822717+0000","last_became_peered":"2024-09-13T14:26:15.822717+0000","last_unstale":"2024-09-13T14:42:42.245554+0000","last_undegraded":"2024-09-13T14:42:42.245554+0000","last_fullsized":"2024-09-13T14:42:42.245554+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:22.471622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.6","version":"284'10378","reported_seq":16911,"reported_epoch":284,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-13T14:42:51.014462+0000","last_change":"2024-09-13T14:38:55.132683+0000","last_active":"2024-09-13T14:42:51.014462+0000","last_peered":"2024-09-13T14:42:51.014462+0000","last_clean":"2024-09-13T14:36:13.124368+0000","last_became_active":"2024-09-13T14:38:55.120883+0000","last_became_peered":"2024-09-13T14:38:55.120883+0000","last_unstale":"2024-09-13T14:42:51.014462+0000","last_undegraded":"2024-09-13T14:38:55.026610+0000","last_fullsized":"2024-09-13T14:42:51.014462+0000","mapping_epoch":254,"log_start":"213'4419","ondisk_log_start":"213'4419","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5959,"log_dups_size":0,"ondisk_log_size":5959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250347520,"num_objects":3820,"num_object_clones":0,"num_object_copies":11460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1832,"num_objects_misplaced":1988,"num_objects_unfound":0,"num_objects_dirty":3820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10378,"num_write_kb":454336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3173,"num_bytes_recovered":196739072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,3,5],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3,4","objects":1553},{"shards":"0,3,5","objects":2267}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":417,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.208368+0000","last_change":"2024-09-13T14:41:53.196835+0000","last_active":"2024-09-13T14:42:33.208368+0000","last_peered":"2024-09-13T14:42:33.208368+0000","last_clean":"2024-09-13T14:42:33.208368+0000","last_became_active":"2024-09-13T14:41:53.196387+0000","last_became_peered":"2024-09-13T14:41:53.196387+0000","last_unstale":"2024-09-13T14:42:33.208368+0000","last_undegraded":"2024-09-13T14:42:33.208368+0000","last_fullsized":"2024-09-13T14:42:33.208368+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:07:41.927740+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:49.972303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":523,"reported_epoch":283,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:42:33.222746+0000","last_change":"2024-09-13T14:38:54.904329+0000","last_active":"2024-09-13T14:42:33.222746+0000","last_peered":"2024-09-13T14:42:33.222746+0000","last_clean":"2024-09-13T14:38:39.648017+0000","last_became_active":"2024-09-13T14:38:54.903798+0000","last_became_peered":"2024-09-13T14:38:54.903798+0000","last_unstale":"2024-09-13T14:42:33.222746+0000","last_undegraded":"2024-09-13T14:38:54.890505+0000","last_fullsized":"2024-09-13T14:38:54.889701+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:18.120931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,3],"avail_no_missing":["5","3","6"],"object_location_counts":[{"shards":"3,5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":402,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-13T14:42:42.246194+0000","last_change":"2024-09-13T14:37:49.160719+0000","last_active":"2024-09-13T14:42:42.246194+0000","last_peered":"2024-09-13T14:42:42.246194+0000","last_clean":"2024-09-13T14:42:42.246194+0000","last_became_active":"2024-09-13T14:37:49.160401+0000","last_became_peered":"2024-09-13T14:37:49.160401+0000","last_unstale":"2024-09-13T14:42:42.246194+0000","last_undegraded":"2024-09-13T14:42:42.246194+0000","last_fullsized":"2024-09-13T14:42:42.246194+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:21.737945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":128,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.206871+0000","last_change":"2024-09-13T14:37:48.172940+0000","last_active":"2024-09-13T14:42:33.206871+0000","last_peered":"2024-09-13T14:42:33.206871+0000","last_clean":"2024-09-13T14:42:33.206871+0000","last_became_active":"2024-09-13T14:37:48.172060+0000","last_became_peered":"2024-09-13T14:37:48.172060+0000","last_unstale":"2024-09-13T14:42:33.206871+0000","last_undegraded":"2024-09-13T14:42:33.206871+0000","last_fullsized":"2024-09-13T14:42:33.206871+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:07.480645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.5","version":"284'10273","reported_seq":26640,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-13T14:42:52.125101+0000","last_change":"2024-09-13T14:39:34.446699+0000","last_active":"2024-09-13T14:42:52.125101+0000","last_peered":"2024-09-13T14:42:52.125101+0000","last_clean":"2024-09-13T14:42:52.125101+0000","last_became_active":"2024-09-13T14:38:04.600616+0000","last_became_peered":"2024-09-13T14:38:04.600616+0000","last_unstale":"2024-09-13T14:42:52.125101+0000","last_undegraded":"2024-09-13T14:42:52.125101+0000","last_fullsized":"2024-09-13T14:42:52.125101+0000","mapping_epoch":245,"log_start":"212'4016","ondisk_log_start":"212'4016","created":205,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6257,"log_dups_size":0,"ondisk_log_size":6257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:00:44.321678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":253558784,"num_objects":3869,"num_object_clones":0,"num_object_copies":11607,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10273,"num_write_kb":452544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8120,"num_bytes_recovered":521076736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":501,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.208334+0000","last_change":"2024-09-13T14:26:15.835178+0000","last_active":"2024-09-13T14:42:33.208334+0000","last_peered":"2024-09-13T14:42:33.208334+0000","last_clean":"2024-09-13T14:42:33.208334+0000","last_became_active":"2024-09-13T14:26:15.820283+0000","last_became_peered":"2024-09-13T14:26:15.820283+0000","last_unstale":"2024-09-13T14:42:33.208334+0000","last_undegraded":"2024-09-13T14:42:33.208334+0000","last_fullsized":"2024-09-13T14:42:33.208334+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:30.340038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":92,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-13T14:42:42.249262+0000","last_change":"2024-09-13T14:38:03.525682+0000","last_active":"2024-09-13T14:42:42.249262+0000","last_peered":"2024-09-13T14:42:42.249262+0000","last_clean":"2024-09-13T14:42:42.249262+0000","last_became_active":"2024-09-13T14:38:03.525406+0000","last_became_peered":"2024-09-13T14:38:03.525406+0000","last_unstale":"2024-09-13T14:42:42.249262+0000","last_undegraded":"2024-09-13T14:42:42.249262+0000","last_fullsized":"2024-09-13T14:42:42.249262+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:41:28.970300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":99,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-13T14:42:42.250562+0000","last_change":"2024-09-13T14:38:53.991738+0000","last_active":"2024-09-13T14:42:42.250562+0000","last_peered":"2024-09-13T14:42:42.250562+0000","last_clean":"2024-09-13T14:42:42.250562+0000","last_became_active":"2024-09-13T14:38:53.991417+0000","last_became_peered":"2024-09-13T14:38:53.991417+0000","last_unstale":"2024-09-13T14:42:42.250562+0000","last_undegraded":"2024-09-13T14:42:42.250562+0000","last_fullsized":"2024-09-13T14:42:42.250562+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:50:38.559838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"284'10293","reported_seq":25685,"reported_epoch":284,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-13T14:42:50.963889+0000","last_change":"2024-09-13T14:38:17.472071+0000","last_active":"2024-09-13T14:42:50.963889+0000","last_peered":"2024-09-13T14:42:50.963889+0000","last_clean":"2024-09-13T14:37:46.290754+0000","last_became_active":"2024-09-13T14:38:17.381753+0000","last_became_peered":"2024-09-13T14:38:17.381753+0000","last_unstale":"2024-09-13T14:42:50.963889+0000","last_undegraded":"2024-09-13T14:38:13.504140+0000","last_fullsized":"2024-09-13T14:42:50.963889+0000","mapping_epoch":245,"log_start":"213'4359","ondisk_log_start":"213'4359","created":205,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5934,"log_dups_size":0,"ondisk_log_size":5934,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249888768,"num_objects":3813,"num_object_clones":0,"num_object_copies":11439,"num_objects_missing_on_primary":83,"num_objects_missing":83,"num_objects_degraded":3982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10293,"num_write_kb":451392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7653,"num_bytes_recovered":501547008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"0,4,5","objects":3813}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":813,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-13T14:42:42.245427+0000","last_change":"2024-09-13T14:37:48.131799+0000","last_active":"2024-09-13T14:42:42.245427+0000","last_peered":"2024-09-13T14:42:42.245427+0000","last_clean":"2024-09-13T14:42:42.245427+0000","last_became_active":"2024-09-13T14:37:48.130837+0000","last_became_peered":"2024-09-13T14:37:48.130837+0000","last_unstale":"2024-09-13T14:42:42.245427+0000","last_undegraded":"2024-09-13T14:42:42.245427+0000","last_fullsized":"2024-09-13T14:42:42.245427+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:06:32.663770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":332,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-13T14:42:42.249773+0000","last_change":"2024-09-13T14:37:49.301531+0000","last_active":"2024-09-13T14:42:42.249773+0000","last_peered":"2024-09-13T14:42:42.249773+0000","last_clean":"2024-09-13T14:42:42.249773+0000","last_became_active":"2024-09-13T14:37:49.301249+0000","last_became_peered":"2024-09-13T14:37:49.301249+0000","last_unstale":"2024-09-13T14:42:42.249773+0000","last_undegraded":"2024-09-13T14:42:42.249773+0000","last_fullsized":"2024-09-13T14:42:42.249773+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:10.977710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":110,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-13T14:42:42.288237+0000","last_change":"2024-09-13T14:38:53.901427+0000","last_active":"2024-09-13T14:42:42.288237+0000","last_peered":"2024-09-13T14:42:42.288237+0000","last_clean":"2024-09-13T14:42:42.288237+0000","last_became_active":"2024-09-13T14:38:53.901157+0000","last_became_peered":"2024-09-13T14:38:53.901157+0000","last_unstale":"2024-09-13T14:42:42.288237+0000","last_undegraded":"2024-09-13T14:42:42.288237+0000","last_fullsized":"2024-09-13T14:42:42.288237+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:28:21.713025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"284'10176","reported_seq":26655,"reported_epoch":284,"state":"active+recovering","last_fresh":"2024-09-13T14:42:51.014479+0000","last_change":"2024-09-13T14:42:29.105072+0000","last_active":"2024-09-13T14:42:51.014479+0000","last_peered":"2024-09-13T14:42:51.014479+0000","last_clean":"2024-09-13T14:37:46.405415+0000","last_became_active":"2024-09-13T14:38:16.427510+0000","last_became_peered":"2024-09-13T14:38:16.427510+0000","last_unstale":"2024-09-13T14:42:51.014479+0000","last_undegraded":"2024-09-13T14:42:51.014479+0000","last_fullsized":"2024-09-13T14:42:51.014479+0000","mapping_epoch":245,"log_start":"213'4218","ondisk_log_start":"213'4218","created":205,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5958,"log_dups_size":0,"ondisk_log_size":5958,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250478592,"num_objects":3822,"num_object_clones":0,"num_object_copies":11466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3307,"num_objects_unfound":0,"num_objects_dirty":3822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10176,"num_write_kb":447936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8197,"num_bytes_recovered":530644992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":["0","3","7"],"object_location_counts":[{"shards":"0,3,6","objects":515},{"shards":"0,3,7","objects":3307}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":992,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.222610+0000","last_change":"2024-09-13T14:37:49.147498+0000","last_active":"2024-09-13T14:42:33.222610+0000","last_peered":"2024-09-13T14:42:33.222610+0000","last_clean":"2024-09-13T14:42:33.222610+0000","last_became_active":"2024-09-13T14:37:49.147028+0000","last_became_peered":"2024-09-13T14:37:49.147028+0000","last_unstale":"2024-09-13T14:42:33.222610+0000","last_undegraded":"2024-09-13T14:42:33.222610+0000","last_fullsized":"2024-09-13T14:42:33.222610+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:11:29.050178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":290,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-13T14:42:42.241889+0000","last_change":"2024-09-13T14:38:03.528133+0000","last_active":"2024-09-13T14:42:42.241889+0000","last_peered":"2024-09-13T14:42:42.241889+0000","last_clean":"2024-09-13T14:42:42.241889+0000","last_became_active":"2024-09-13T14:38:03.527829+0000","last_became_peered":"2024-09-13T14:38:03.527829+0000","last_unstale":"2024-09-13T14:42:42.241889+0000","last_undegraded":"2024-09-13T14:42:42.241889+0000","last_fullsized":"2024-09-13T14:42:42.241889+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:48.254169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":452,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-13T14:42:42.245788+0000","last_change":"2024-09-13T14:38:53.899755+0000","last_active":"2024-09-13T14:42:42.245788+0000","last_peered":"2024-09-13T14:42:42.245788+0000","last_clean":"2024-09-13T14:42:42.245788+0000","last_became_active":"2024-09-13T14:38:53.899448+0000","last_became_peered":"2024-09-13T14:38:53.899448+0000","last_unstale":"2024-09-13T14:42:42.245788+0000","last_undegraded":"2024-09-13T14:42:42.245788+0000","last_fullsized":"2024-09-13T14:42:42.245788+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:55.897035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"284'10494","reported_seq":10616,"reported_epoch":284,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:42:52.698988+0000","last_change":"2024-09-13T14:41:52.203221+0000","last_active":"2024-09-13T14:42:52.698988+0000","last_peered":"2024-09-13T14:42:52.698988+0000","last_clean":"2024-09-13T14:35:51.090791+0000","last_became_active":"2024-09-13T14:37:49.665025+0000","last_became_peered":"2024-09-13T14:37:49.665025+0000","last_unstale":"2024-09-13T14:42:52.698988+0000","last_undegraded":"2024-09-13T14:37:49.169101+0000","last_fullsized":"2024-09-13T14:37:49.121596+0000","mapping_epoch":240,"log_start":"212'4063","ondisk_log_start":"212'4063","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6431,"log_dups_size":0,"ondisk_log_size":6431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:02.321504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249561088,"num_objects":3808,"num_object_clones":0,"num_object_copies":11424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4496,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10494,"num_write_kb":457664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":393216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":-622}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":94,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:37.278927+0000","last_change":"2024-09-13T14:37:49.149770+0000","last_active":"2024-09-13T14:42:37.278927+0000","last_peered":"2024-09-13T14:42:37.278927+0000","last_clean":"2024-09-13T14:42:37.278927+0000","last_became_active":"2024-09-13T14:37:49.148991+0000","last_became_peered":"2024-09-13T14:37:49.148991+0000","last_unstale":"2024-09-13T14:42:37.278927+0000","last_undegraded":"2024-09-13T14:42:37.278927+0000","last_fullsized":"2024-09-13T14:42:37.278927+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:04:04.193366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":1150,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-13T14:42:42.296001+0000","last_change":"2024-09-13T14:30:58.928302+0000","last_active":"2024-09-13T14:42:42.296001+0000","last_peered":"2024-09-13T14:42:42.296001+0000","last_clean":"2024-09-13T14:42:42.296001+0000","last_became_active":"2024-09-13T14:30:58.927718+0000","last_became_peered":"2024-09-13T14:30:58.927718+0000","last_unstale":"2024-09-13T14:42:42.296001+0000","last_undegraded":"2024-09-13T14:42:42.296001+0000","last_fullsized":"2024-09-13T14:42:42.296001+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:37.323036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":101,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.222712+0000","last_change":"2024-09-13T14:38:03.534628+0000","last_active":"2024-09-13T14:42:33.222712+0000","last_peered":"2024-09-13T14:42:33.222712+0000","last_clean":"2024-09-13T14:42:33.222712+0000","last_became_active":"2024-09-13T14:38:03.534262+0000","last_became_peered":"2024-09-13T14:38:03.534262+0000","last_unstale":"2024-09-13T14:42:33.222712+0000","last_undegraded":"2024-09-13T14:42:33.222712+0000","last_fullsized":"2024-09-13T14:42:33.222712+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:31.168783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"284'10179","reported_seq":24350,"reported_epoch":284,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:42:52.245335+0000","last_change":"2024-09-13T14:38:53.944111+0000","last_active":"2024-09-13T14:42:52.245335+0000","last_peered":"2024-09-13T14:42:52.245335+0000","last_clean":"2024-09-13T14:37:47.047672+0000","last_became_active":"2024-09-13T14:37:49.664294+0000","last_became_peered":"2024-09-13T14:37:49.664294+0000","last_unstale":"2024-09-13T14:42:52.245335+0000","last_undegraded":"2024-09-13T14:38:53.940822+0000","last_fullsized":"2024-09-13T14:37:49.126627+0000","mapping_epoch":240,"log_start":"213'4500","ondisk_log_start":"213'4500","created":205,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5679,"log_dups_size":0,"ondisk_log_size":5679,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:11.766692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":253820928,"num_objects":3873,"num_object_clones":0,"num_object_copies":11619,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4535,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10179,"num_write_kb":449664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7009,"num_bytes_recovered":459341824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3],"avail_no_missing":["5","0","3"],"object_location_counts":[{"shards":"0,3,5","objects":3873},{"shards":"3,5","objects":-587}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":113,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:37.278909+0000","last_change":"2024-09-13T14:38:03.534303+0000","last_active":"2024-09-13T14:42:37.278909+0000","last_peered":"2024-09-13T14:42:37.278909+0000","last_clean":"2024-09-13T14:42:37.278909+0000","last_became_active":"2024-09-13T14:38:03.533171+0000","last_became_peered":"2024-09-13T14:38:03.533171+0000","last_unstale":"2024-09-13T14:42:37.278909+0000","last_undegraded":"2024-09-13T14:42:37.278909+0000","last_fullsized":"2024-09-13T14:42:37.278909+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:00:16.553526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":805,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.208890+0000","last_change":"2024-09-13T14:38:04.596192+0000","last_active":"2024-09-13T14:42:33.208890+0000","last_peered":"2024-09-13T14:42:33.208890+0000","last_clean":"2024-09-13T14:42:33.208890+0000","last_became_active":"2024-09-13T14:38:04.593299+0000","last_became_peered":"2024-09-13T14:38:04.593299+0000","last_unstale":"2024-09-13T14:42:33.208890+0000","last_undegraded":"2024-09-13T14:42:33.208890+0000","last_fullsized":"2024-09-13T14:42:33.208890+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:34.241410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":92,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-13T14:42:42.250190+0000","last_change":"2024-09-13T14:38:03.533120+0000","last_active":"2024-09-13T14:42:42.250190+0000","last_peered":"2024-09-13T14:42:42.250190+0000","last_clean":"2024-09-13T14:42:42.250190+0000","last_became_active":"2024-09-13T14:38:03.532829+0000","last_became_peered":"2024-09-13T14:38:03.532829+0000","last_unstale":"2024-09-13T14:42:42.250190+0000","last_undegraded":"2024-09-13T14:42:42.250190+0000","last_fullsized":"2024-09-13T14:42:42.250190+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:24.514661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"284'10422","reported_seq":19678,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-13T14:42:52.274129+0000","last_change":"2024-09-13T14:42:04.660877+0000","last_active":"2024-09-13T14:42:52.274129+0000","last_peered":"2024-09-13T14:42:52.274129+0000","last_clean":"2024-09-13T14:42:52.274129+0000","last_became_active":"2024-09-13T14:42:04.641680+0000","last_became_peered":"2024-09-13T14:42:04.641680+0000","last_unstale":"2024-09-13T14:42:52.274129+0000","last_undegraded":"2024-09-13T14:42:52.274129+0000","last_fullsized":"2024-09-13T14:42:52.274129+0000","mapping_epoch":277,"log_start":"213'4718","ondisk_log_start":"213'4718","created":205,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5704,"log_dups_size":0,"ondisk_log_size":5704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:22:18.955428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260571136,"num_objects":3976,"num_object_clones":0,"num_object_copies":11928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10422,"num_write_kb":460736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4533,"num_bytes_recovered":297074688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":494,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-13T14:42:42.245633+0000","last_change":"2024-09-13T14:38:53.938227+0000","last_active":"2024-09-13T14:42:42.245633+0000","last_peered":"2024-09-13T14:42:42.245633+0000","last_clean":"2024-09-13T14:42:42.245633+0000","last_became_active":"2024-09-13T14:38:53.902867+0000","last_became_peered":"2024-09-13T14:38:53.902867+0000","last_unstale":"2024-09-13T14:42:42.245633+0000","last_undegraded":"2024-09-13T14:42:42.245633+0000","last_fullsized":"2024-09-13T14:42:42.245633+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:16.831756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":584,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:37.796879+0000","last_change":"2024-09-13T14:37:48.147677+0000","last_active":"2024-09-13T14:42:37.796879+0000","last_peered":"2024-09-13T14:42:37.796879+0000","last_clean":"2024-09-13T14:42:37.796879+0000","last_became_active":"2024-09-13T14:37:48.147001+0000","last_became_peered":"2024-09-13T14:37:48.147001+0000","last_unstale":"2024-09-13T14:42:37.796879+0000","last_undegraded":"2024-09-13T14:42:37.796879+0000","last_fullsized":"2024-09-13T14:42:37.796879+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:35:00.653576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":475,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-13T14:42:42.245484+0000","last_change":"2024-09-13T14:38:53.900107+0000","last_active":"2024-09-13T14:42:42.245484+0000","last_peered":"2024-09-13T14:42:42.245484+0000","last_clean":"2024-09-13T14:42:42.245484+0000","last_became_active":"2024-09-13T14:38:53.899556+0000","last_became_peered":"2024-09-13T14:38:53.899556+0000","last_unstale":"2024-09-13T14:42:42.245484+0000","last_undegraded":"2024-09-13T14:42:42.245484+0000","last_fullsized":"2024-09-13T14:42:42.245484+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:15:47.918163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"284'10103","reported_seq":19085,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-13T14:42:52.469796+0000","last_change":"2024-09-13T14:39:54.921239+0000","last_active":"2024-09-13T14:42:52.469796+0000","last_peered":"2024-09-13T14:42:52.469796+0000","last_clean":"2024-09-13T14:42:52.469796+0000","last_became_active":"2024-09-13T14:39:54.907894+0000","last_became_peered":"2024-09-13T14:39:54.907894+0000","last_unstale":"2024-09-13T14:42:52.469796+0000","last_undegraded":"2024-09-13T14:42:52.469796+0000","last_fullsized":"2024-09-13T14:42:52.469796+0000","mapping_epoch":263,"log_start":"213'4347","ondisk_log_start":"213'4347","created":205,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5756,"log_dups_size":0,"ondisk_log_size":5756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:52:04.619238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":245694464,"num_objects":3749,"num_object_clones":0,"num_object_copies":11247,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10103,"num_write_kb":443264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4395,"num_bytes_recovered":283770880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":107,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-13T14:42:42.250448+0000","last_change":"2024-09-13T14:38:53.901806+0000","last_active":"2024-09-13T14:42:42.250448+0000","last_peered":"2024-09-13T14:42:42.250448+0000","last_clean":"2024-09-13T14:42:42.250448+0000","last_became_active":"2024-09-13T14:38:53.891055+0000","last_became_peered":"2024-09-13T14:38:53.891055+0000","last_unstale":"2024-09-13T14:42:42.250448+0000","last_undegraded":"2024-09-13T14:42:42.250448+0000","last_fullsized":"2024-09-13T14:42:42.250448+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:03:00.416198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":40,"reported_epoch":283,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-13T14:42:33.208095+0000","last_change":"2024-09-13T14:38:53.900902+0000","last_active":"2024-09-13T14:42:33.208095+0000","last_peered":"2024-09-13T14:42:33.208095+0000","last_clean":"2024-09-13T14:38:39.635779+0000","last_became_active":"2024-09-13T14:38:53.900626+0000","last_became_peered":"2024-09-13T14:38:53.900626+0000","last_unstale":"2024-09-13T14:42:33.208095+0000","last_undegraded":"2024-09-13T14:38:53.885778+0000","last_fullsized":"2024-09-13T14:42:33.208095+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:37:21.715778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":7,"num_objects_missing":7,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":["3","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":274,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-13T14:42:42.241394+0000","last_change":"2024-09-13T14:37:49.172218+0000","last_active":"2024-09-13T14:42:42.241394+0000","last_peered":"2024-09-13T14:42:42.241394+0000","last_clean":"2024-09-13T14:42:42.241394+0000","last_became_active":"2024-09-13T14:37:49.171718+0000","last_became_peered":"2024-09-13T14:37:49.171718+0000","last_unstale":"2024-09-13T14:42:42.241394+0000","last_undegraded":"2024-09-13T14:42:42.241394+0000","last_fullsized":"2024-09-13T14:42:42.241394+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:46:54.626521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":296,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.206763+0000","last_change":"2024-09-13T14:37:49.230094+0000","last_active":"2024-09-13T14:42:33.206763+0000","last_peered":"2024-09-13T14:42:33.206763+0000","last_clean":"2024-09-13T14:42:33.206763+0000","last_became_active":"2024-09-13T14:37:49.229393+0000","last_became_peered":"2024-09-13T14:37:49.229393+0000","last_unstale":"2024-09-13T14:42:33.206763+0000","last_undegraded":"2024-09-13T14:42:33.206763+0000","last_fullsized":"2024-09-13T14:42:33.206763+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:02:28.690318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":276,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.206884+0000","last_change":"2024-09-13T14:37:48.128447+0000","last_active":"2024-09-13T14:42:33.206884+0000","last_peered":"2024-09-13T14:42:33.206884+0000","last_clean":"2024-09-13T14:42:33.206884+0000","last_became_active":"2024-09-13T14:37:48.128143+0000","last_became_peered":"2024-09-13T14:37:48.128143+0000","last_unstale":"2024-09-13T14:42:33.206884+0000","last_undegraded":"2024-09-13T14:42:33.206884+0000","last_fullsized":"2024-09-13T14:42:33.206884+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:22.204039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":91,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-13T14:42:42.249638+0000","last_change":"2024-09-13T14:38:03.531683+0000","last_active":"2024-09-13T14:42:42.249638+0000","last_peered":"2024-09-13T14:42:42.249638+0000","last_clean":"2024-09-13T14:42:42.249638+0000","last_became_active":"2024-09-13T14:38:03.531476+0000","last_became_peered":"2024-09-13T14:38:03.531476+0000","last_unstale":"2024-09-13T14:42:42.249638+0000","last_undegraded":"2024-09-13T14:42:42.249638+0000","last_fullsized":"2024-09-13T14:42:42.249638+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:41.557569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":48,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.208940+0000","last_change":"2024-09-13T14:38:53.954281+0000","last_active":"2024-09-13T14:42:33.208940+0000","last_peered":"2024-09-13T14:42:33.208940+0000","last_clean":"2024-09-13T14:42:33.208940+0000","last_became_active":"2024-09-13T14:38:53.900912+0000","last_became_peered":"2024-09-13T14:38:53.900912+0000","last_unstale":"2024-09-13T14:42:33.208940+0000","last_undegraded":"2024-09-13T14:42:33.208940+0000","last_fullsized":"2024-09-13T14:42:33.208940+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:47.876275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":100,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.223145+0000","last_change":"2024-09-13T14:38:54.000782+0000","last_active":"2024-09-13T14:42:33.223145+0000","last_peered":"2024-09-13T14:42:33.223145+0000","last_clean":"2024-09-13T14:42:33.223145+0000","last_became_active":"2024-09-13T14:38:54.000322+0000","last_became_peered":"2024-09-13T14:38:54.000322+0000","last_unstale":"2024-09-13T14:42:33.223145+0000","last_undegraded":"2024-09-13T14:42:33.223145+0000","last_fullsized":"2024-09-13T14:42:33.223145+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:09.057597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":417,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-13T14:42:42.245143+0000","last_change":"2024-09-13T14:38:53.899535+0000","last_active":"2024-09-13T14:42:42.245143+0000","last_peered":"2024-09-13T14:42:42.245143+0000","last_clean":"2024-09-13T14:42:42.245143+0000","last_became_active":"2024-09-13T14:38:53.899183+0000","last_became_peered":"2024-09-13T14:38:53.899183+0000","last_unstale":"2024-09-13T14:42:42.245143+0000","last_undegraded":"2024-09-13T14:42:42.245143+0000","last_fullsized":"2024-09-13T14:42:42.245143+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:33.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":91,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-13T14:42:42.257342+0000","last_change":"2024-09-13T14:38:03.543426+0000","last_active":"2024-09-13T14:42:42.257342+0000","last_peered":"2024-09-13T14:42:42.257342+0000","last_clean":"2024-09-13T14:42:42.257342+0000","last_became_active":"2024-09-13T14:38:03.542661+0000","last_became_peered":"2024-09-13T14:38:03.542661+0000","last_unstale":"2024-09-13T14:42:42.257342+0000","last_undegraded":"2024-09-13T14:42:42.257342+0000","last_fullsized":"2024-09-13T14:42:42.257342+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:54.050127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":446,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-13T14:42:42.245172+0000","last_change":"2024-09-13T14:37:48.131321+0000","last_active":"2024-09-13T14:42:42.245172+0000","last_peered":"2024-09-13T14:42:42.245172+0000","last_clean":"2024-09-13T14:42:42.245172+0000","last_became_active":"2024-09-13T14:37:48.130998+0000","last_became_peered":"2024-09-13T14:37:48.130998+0000","last_unstale":"2024-09-13T14:42:42.245172+0000","last_undegraded":"2024-09-13T14:42:42.245172+0000","last_fullsized":"2024-09-13T14:42:42.245172+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:33:26.597825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":113,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.223168+0000","last_change":"2024-09-13T14:38:03.535454+0000","last_active":"2024-09-13T14:42:33.223168+0000","last_peered":"2024-09-13T14:42:33.223168+0000","last_clean":"2024-09-13T14:42:33.223168+0000","last_became_active":"2024-09-13T14:38:03.535131+0000","last_became_peered":"2024-09-13T14:38:03.535131+0000","last_unstale":"2024-09-13T14:42:33.223168+0000","last_undegraded":"2024-09-13T14:42:33.223168+0000","last_fullsized":"2024-09-13T14:42:33.223168+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:24.477601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":471,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-13T14:42:42.245569+0000","last_change":"2024-09-13T14:38:03.524939+0000","last_active":"2024-09-13T14:42:42.245569+0000","last_peered":"2024-09-13T14:42:42.245569+0000","last_clean":"2024-09-13T14:42:42.245569+0000","last_became_active":"2024-09-13T14:38:03.524406+0000","last_became_peered":"2024-09-13T14:38:03.524406+0000","last_unstale":"2024-09-13T14:42:42.245569+0000","last_undegraded":"2024-09-13T14:42:42.245569+0000","last_fullsized":"2024-09-13T14:42:42.245569+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:19:45.257424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":99,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-13T14:42:42.250395+0000","last_change":"2024-09-13T14:38:53.892948+0000","last_active":"2024-09-13T14:42:42.250395+0000","last_peered":"2024-09-13T14:42:42.250395+0000","last_clean":"2024-09-13T14:42:42.250395+0000","last_became_active":"2024-09-13T14:38:53.892665+0000","last_became_peered":"2024-09-13T14:38:53.892665+0000","last_unstale":"2024-09-13T14:42:42.250395+0000","last_undegraded":"2024-09-13T14:42:42.250395+0000","last_fullsized":"2024-09-13T14:42:42.250395+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:08.292359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":419,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-13T14:42:42.245340+0000","last_change":"2024-09-13T14:37:48.139348+0000","last_active":"2024-09-13T14:42:42.245340+0000","last_peered":"2024-09-13T14:42:42.245340+0000","last_clean":"2024-09-13T14:42:42.245340+0000","last_became_active":"2024-09-13T14:37:48.139115+0000","last_became_peered":"2024-09-13T14:37:48.139115+0000","last_unstale":"2024-09-13T14:42:42.245340+0000","last_undegraded":"2024-09-13T14:42:42.245340+0000","last_fullsized":"2024-09-13T14:42:42.245340+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:32:20.413346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":460,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-13T14:42:42.246018+0000","last_change":"2024-09-13T14:37:48.172088+0000","last_active":"2024-09-13T14:42:42.246018+0000","last_peered":"2024-09-13T14:42:42.246018+0000","last_clean":"2024-09-13T14:42:42.246018+0000","last_became_active":"2024-09-13T14:37:48.171779+0000","last_became_peered":"2024-09-13T14:37:48.171779+0000","last_unstale":"2024-09-13T14:42:42.246018+0000","last_undegraded":"2024-09-13T14:42:42.246018+0000","last_fullsized":"2024-09-13T14:42:42.246018+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:02.852712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":340,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.206688+0000","last_change":"2024-09-13T14:38:53.890236+0000","last_active":"2024-09-13T14:42:33.206688+0000","last_peered":"2024-09-13T14:42:33.206688+0000","last_clean":"2024-09-13T14:42:33.206688+0000","last_became_active":"2024-09-13T14:38:53.889582+0000","last_became_peered":"2024-09-13T14:38:53.889582+0000","last_unstale":"2024-09-13T14:42:33.206688+0000","last_undegraded":"2024-09-13T14:42:33.206688+0000","last_fullsized":"2024-09-13T14:42:33.206688+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:14:29.334774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":446,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-13T14:42:42.246959+0000","last_change":"2024-09-13T14:38:03.534097+0000","last_active":"2024-09-13T14:42:42.246959+0000","last_peered":"2024-09-13T14:42:42.246959+0000","last_clean":"2024-09-13T14:42:42.246959+0000","last_became_active":"2024-09-13T14:38:03.533794+0000","last_became_peered":"2024-09-13T14:38:03.533794+0000","last_unstale":"2024-09-13T14:42:42.246959+0000","last_undegraded":"2024-09-13T14:42:42.246959+0000","last_fullsized":"2024-09-13T14:42:42.246959+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:23.272098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":91,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-13T14:42:42.249965+0000","last_change":"2024-09-13T14:38:03.527021+0000","last_active":"2024-09-13T14:42:42.249965+0000","last_peered":"2024-09-13T14:42:42.249965+0000","last_clean":"2024-09-13T14:42:42.249965+0000","last_became_active":"2024-09-13T14:38:03.526698+0000","last_became_peered":"2024-09-13T14:38:03.526698+0000","last_unstale":"2024-09-13T14:42:42.249965+0000","last_undegraded":"2024-09-13T14:42:42.249965+0000","last_fullsized":"2024-09-13T14:42:42.249965+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:01:35.734086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":284,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.206716+0000","last_change":"2024-09-13T14:27:12.834931+0000","last_active":"2024-09-13T14:42:33.206716+0000","last_peered":"2024-09-13T14:42:33.206716+0000","last_clean":"2024-09-13T14:42:33.206716+0000","last_became_active":"2024-09-13T14:27:12.834572+0000","last_became_peered":"2024-09-13T14:27:12.834572+0000","last_unstale":"2024-09-13T14:42:33.206716+0000","last_undegraded":"2024-09-13T14:42:33.206716+0000","last_fullsized":"2024-09-13T14:42:33.206716+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:54:21.155689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":97,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.222806+0000","last_change":"2024-09-13T14:37:49.176796+0000","last_active":"2024-09-13T14:42:33.222806+0000","last_peered":"2024-09-13T14:42:33.222806+0000","last_clean":"2024-09-13T14:42:33.222806+0000","last_became_active":"2024-09-13T14:37:49.176246+0000","last_became_peered":"2024-09-13T14:37:49.176246+0000","last_unstale":"2024-09-13T14:42:33.222806+0000","last_undegraded":"2024-09-13T14:42:33.222806+0000","last_fullsized":"2024-09-13T14:42:33.222806+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:29:24.936718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":295,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.206606+0000","last_change":"2024-09-13T14:38:03.526333+0000","last_active":"2024-09-13T14:42:33.206606+0000","last_peered":"2024-09-13T14:42:33.206606+0000","last_clean":"2024-09-13T14:42:33.206606+0000","last_became_active":"2024-09-13T14:38:03.526001+0000","last_became_peered":"2024-09-13T14:38:03.526001+0000","last_unstale":"2024-09-13T14:42:33.206606+0000","last_undegraded":"2024-09-13T14:42:33.206606+0000","last_fullsized":"2024-09-13T14:42:33.206606+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:25:25.181271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":97,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.222940+0000","last_change":"2024-09-13T14:37:49.149582+0000","last_active":"2024-09-13T14:42:33.222940+0000","last_peered":"2024-09-13T14:42:33.222940+0000","last_clean":"2024-09-13T14:42:33.222940+0000","last_became_active":"2024-09-13T14:37:49.148525+0000","last_became_peered":"2024-09-13T14:37:49.148525+0000","last_unstale":"2024-09-13T14:42:33.222940+0000","last_undegraded":"2024-09-13T14:42:33.222940+0000","last_fullsized":"2024-09-13T14:42:33.222940+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:48.655665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":492,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-13T14:42:42.245515+0000","last_change":"2024-09-13T14:38:03.535255+0000","last_active":"2024-09-13T14:42:42.245515+0000","last_peered":"2024-09-13T14:42:42.245515+0000","last_clean":"2024-09-13T14:42:42.245515+0000","last_became_active":"2024-09-13T14:38:03.533917+0000","last_became_peered":"2024-09-13T14:38:03.533917+0000","last_unstale":"2024-09-13T14:42:42.245515+0000","last_undegraded":"2024-09-13T14:42:42.245515+0000","last_fullsized":"2024-09-13T14:42:42.245515+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:33.559788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":455,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-13T14:42:42.246022+0000","last_change":"2024-09-13T14:37:48.146784+0000","last_active":"2024-09-13T14:42:42.246022+0000","last_peered":"2024-09-13T14:42:42.246022+0000","last_clean":"2024-09-13T14:42:42.246022+0000","last_became_active":"2024-09-13T14:37:48.146491+0000","last_became_peered":"2024-09-13T14:37:48.146491+0000","last_unstale":"2024-09-13T14:42:42.246022+0000","last_undegraded":"2024-09-13T14:42:42.246022+0000","last_fullsized":"2024-09-13T14:42:42.246022+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:30:54.772513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":275,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-13T14:42:33.206494+0000","last_change":"2024-09-13T14:37:48.127607+0000","last_active":"2024-09-13T14:42:33.206494+0000","last_peered":"2024-09-13T14:42:33.206494+0000","last_clean":"2024-09-13T14:42:33.206494+0000","last_became_active":"2024-09-13T14:37:48.127316+0000","last_became_peered":"2024-09-13T14:37:48.127316+0000","last_unstale":"2024-09-13T14:42:33.206494+0000","last_undegraded":"2024-09-13T14:42:33.206494+0000","last_fullsized":"2024-09-13T14:42:33.206494+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:53:37.678861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":98,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-13T14:42:42.257236+0000","last_change":"2024-09-13T14:38:03.563200+0000","last_active":"2024-09-13T14:42:42.257236+0000","last_peered":"2024-09-13T14:42:42.257236+0000","last_clean":"2024-09-13T14:42:42.257236+0000","last_became_active":"2024-09-13T14:38:03.534258+0000","last_became_peered":"2024-09-13T14:38:03.534258+0000","last_unstale":"2024-09-13T14:42:42.257236+0000","last_undegraded":"2024-09-13T14:42:42.257236+0000","last_fullsized":"2024-09-13T14:42:42.257236+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:49:07.814419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":4009689112,"num_objects":61184,"num_object_clones":0,"num_object_copies":183552,"num_objects_missing_on_primary":83,"num_objects_missing":83,"num_objects_degraded":14845,"num_objects_misplaced":5295,"num_objects_unfound":0,"num_objects_dirty":61184,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164442,"num_write_kb":7219969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74054,"num_bytes_recovered":4796776472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13917958144,"data_stored":13917945928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94465,"ondisk_log_size":94465,"up":48,"acting":46,"num_store_stats":6},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1538,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":17390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25132,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":7,"num_objects_missing":7,"num_objects_degraded":16,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16793600,"data_stored":16777292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":23,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2969600,"data_stored":2951840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234388,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2657016,"kb_used_data":2214896,"kb_used_omap":1,"kb_used_meta":442110,"kb_avail":91088136,"statfs":{"total":95995035648,"available":93274251264,"internally_reserved":0,"allocated":2268053504,"data_stored":2266525296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":452721099},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":239,"seq":1026497183806,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1788924,"kb_used_data":1533812,"kb_used_omap":8,"kb_used_meta":255095,"kb_avail":91956228,"statfs":{"total":95995035648,"available":94163177472,"internally_reserved":0,"allocated":1570623488,"data_stored":1569091755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8453,"internal_metadata":261218043},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757653,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2235240,"kb_used_data":1560160,"kb_used_omap":1,"kb_used_meta":675070,"kb_avail":91509912,"statfs":{"total":95995035648,"available":93706149888,"internally_reserved":0,"allocated":1597603840,"data_stored":1596087193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":691272134},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019297,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2696308,"kb_used_data":2105708,"kb_used_omap":1,"kb_used_meta":590590,"kb_avail":91048844,"statfs":{"total":95995035648,"available":93234016256,"internally_reserved":0,"allocated":2156244992,"data_stored":2154720868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":604764614},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435480,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3517092,"kb_used_data":2943964,"kb_used_omap":5,"kb_used_meta":573114,"kb_avail":90228060,"statfs":{"total":95995035648,"available":92393533440,"internally_reserved":0,"allocated":3014619136,"data_stored":3013106349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":586868862},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":446676599008,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":425201762534,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926058,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3965876,"kb_used_data":3284460,"kb_used_omap":7,"kb_used_meta":681400,"kb_avail":89779276,"statfs":{"total":95995035648,"available":91933978624,"internally_reserved":0,"allocated":3363287040,"data_stored":3361759036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":697753896},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6860,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3351318528,"data_stored":3351314456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3006402560,"data_stored":3006398488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2149253120,"data_stored":2149253120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1588920320,"data_stored":1588920320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1563168768,"data_stored":1563164696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2258894848,"data_stored":2258894848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:42:58.225 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph status --format=json 2024-09-13T14:42:58.263 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:58.298 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:58.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:58.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:58.400 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:58.421 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:58.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:58.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:58.523 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:58.545 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:58.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:58.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:58.647 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:58.670 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:58.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:58.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:58.749 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:42:58.749 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [], 2024-09-13T14:42:58.749 INFO:teuthology.orchestra.run.smithi102.stdout: "complaint_time": 30, 2024-09-13T14:42:58.749 INFO:teuthology.orchestra.run.smithi102.stdout: "num_blocked_ops": 0 2024-09-13T14:42:58.749 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:42:58.772 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:42:58.794 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:58.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:58.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:42:58.895 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:58.919 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:58.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:58.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:59.021 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:59.043 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:59.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:59.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:59.142 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:42:59.146 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:59.168 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:59.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:59.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:59.270 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:59.292 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:59.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:59.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:59.394 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:42:59.416 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:59.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:59.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:42:59.499 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:59 smithi102 ceph-mon[110183]: pgmap v964: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.7 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 202 op/s; 14732/180678 objects degraded (8.154%); 4855/180678 objects misplaced (2.687%); 7.0 MiB/s, 112 objects/s recovering 2024-09-13T14:42:59.499 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:42:59 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 14732/180678 objects degraded (8.154%), 6 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:42:59.500 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:59 smithi102 ceph-mon[115269]: pgmap v964: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.7 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 202 op/s; 14732/180678 objects degraded (8.154%); 4855/180678 objects misplaced (2.687%); 7.0 MiB/s, 112 objects/s recovering 2024-09-13T14:42:59.500 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:42:59 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 14732/180678 objects degraded (8.154%), 6 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:42:59.518 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:42:59.540 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:59.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:59.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:42:59.548 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:59 smithi195 ceph-mon[104262]: pgmap v964: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.7 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 202 op/s; 14732/180678 objects degraded (8.154%); 4855/180678 objects misplaced (2.687%); 7.0 MiB/s, 112 objects/s recovering 2024-09-13T14:42:59.548 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:42:59 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 14732/180678 objects degraded (8.154%), 6 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:42:59.601 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:59.624 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:42:59.641 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:59.666 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:42:59.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:59.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:59.768 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:59.794 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:59.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:59.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:59.896 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:59.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:59.921 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:42:59.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:00.022 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:00.044 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:00.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:00.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:00.146 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:00.170 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:00.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:00.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:00.272 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:00.296 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:00.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:00.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:00.398 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:00.421 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:00.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:00.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:00.523 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:00.547 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:00.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:00.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:00.649 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:00.671 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:00.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:00.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:00.773 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:00.796 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:00.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:00.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:00.898 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:00.921 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:00.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:00.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:01.023 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:01.046 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:01.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:01.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:01.147 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:01.170 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:01.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:01.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:01.271 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:01.294 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:01.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:01.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:01.397 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:01.424 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:01.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:01.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:01.526 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:01.548 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:01.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:01.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:01.579 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:01 smithi102 ceph-mon[110183]: pgmap v965: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.7 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 204 op/s; 14732/180477 objects degraded (8.163%); 4855/180477 objects misplaced (2.690%); 7.0 MiB/s, 112 objects/s recovering 2024-09-13T14:43:01.579 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:01 smithi102 ceph-mon[115269]: pgmap v965: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.7 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 204 op/s; 14732/180477 objects degraded (8.163%); 4855/180477 objects misplaced (2.690%); 7.0 MiB/s, 112 objects/s recovering 2024-09-13T14:43:01.650 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:01.674 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:01.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:01.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:01.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:01 smithi195 ceph-mon[104262]: pgmap v965: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.7 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 204 op/s; 14732/180477 objects degraded (8.163%); 4855/180477 objects misplaced (2.690%); 7.0 MiB/s, 112 objects/s recovering 2024-09-13T14:43:01.776 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:01.797 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:01.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:01.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:01.899 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:01.921 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:01.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:01.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:02.023 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:02.049 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:02.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:02.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:02.150 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:02.174 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:02.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:02.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:02.276 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:02.301 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:02.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:02.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:02.384 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T14:43:02.400 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:02.401 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:02.403 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:02.425 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:02.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:02.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:02.526 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:02.548 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:02.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:02.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:02.650 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:02.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:02.672 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:02.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:02.773 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:02.796 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:02.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:02.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:02.898 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:02.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:02.921 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:02.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:03.022 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:03.031 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:03.051 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:03.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:03.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:03.153 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:03.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:03.176 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:03.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:03.277 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:03.283 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:43:03.283 INFO:teuthology.orchestra.run.smithi102.stdout:{"fsid":"08cee502-71d9-11ef-bceb-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 14609/179238 objects degraded (8.151%), 6 pgs degraded, 3 pgs undersized","count":9},"muted":false}},"mutes":[]},"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1406,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":285,"num_osds":8,"num_up_osds":8,"osd_up_since":1726238267,"num_in_osds":6,"osd_in_since":1726238173,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":82},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovery_wait+degraded","count":2},{"state_name":"active+recovering","count":1},{"state_name":"active+recovery_wait+degraded+remapped","count":1}],"num_pgs":89,"num_pools":5,"num_objects":59746,"data_bytes":3918403581,"bytes_used":17073364992,"bytes_avail":558896848896,"bytes_total":575970213888,"degraded_objects":14609,"degraded_total":179238,"degraded_ratio":0.081506153828987155,"misplaced_objects":4380,"misplaced_total":179238,"misplaced_ratio":0.02443678237873665,"recovering_objects_per_sec":116,"recovering_bytes_per_sec":7639479,"recovering_keys_per_sec":0,"num_objects_recovered":1166,"num_bytes_recovered":76414976,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":210},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi102.njetxr","status":"up:active","gid":24521}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.102:8443/","prometheus":"http://172.21.15.102:9283/"}},"servicemap":{"epoch":28,"modified":"2024-09-13T14:40:21.476600+0000","services":{}},"progress_events":{}} 2024-09-13T14:43:03.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:03.305 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:03.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:03.406 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:03.428 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:03.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:03.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:03.530 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:03.555 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:03.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:03.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:03.590 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:03 smithi102 ceph-mon[115269]: pgmap v966: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.6 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 210 op/s; 14609/179238 objects degraded (8.151%); 4380/179238 objects misplaced (2.444%); 7.3 MiB/s, 116 objects/s recovering 2024-09-13T14:43:03.590 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:03 smithi102 ceph-mon[110183]: pgmap v966: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.6 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 210 op/s; 14609/179238 objects degraded (8.151%); 4380/179238 objects misplaced (2.444%); 7.3 MiB/s, 116 objects/s recovering 2024-09-13T14:43:03.656 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:03.680 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:03.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:03.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:03.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:03 smithi195 ceph-mon[104262]: pgmap v966: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.6 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 210 op/s; 14609/179238 objects degraded (8.151%); 4380/179238 objects misplaced (2.444%); 7.3 MiB/s, 116 objects/s recovering 2024-09-13T14:43:03.782 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:03.810 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:03.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:03.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:03.911 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:03.935 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:03.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:03.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:04.036 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:04.061 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:04.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:04.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:04.163 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:04.189 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:04.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:04.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:04.291 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:04.314 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:04.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:04.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:04.415 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:04.443 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:04.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:04.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:04.545 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:04.564 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:04 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/411375201' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:43:04.564 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:04 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 14609/179238 objects degraded (8.151%), 6 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:43:04.564 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:04 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/411375201' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:43:04.565 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:04 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 14609/179238 objects degraded (8.151%), 6 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:43:04.568 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:04.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:04.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:04.669 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:04.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:04.696 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:04.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:04.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:04 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/411375201' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:43:04.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:04 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 14609/179238 objects degraded (8.151%), 6 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:43:04.797 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:04.820 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:04.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:04.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:04.922 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:04.944 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:04.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:04.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:05.049 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:05.075 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:05.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:05.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:05.177 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:05.198 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:05.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:05.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:05.212 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-13T14:43:05.212 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:43:05.300 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:05.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:05.324 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:05.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:05.424 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:05.446 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:05.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:05.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:05.548 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:05.570 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:05.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:05.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:05.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:05 smithi102 ceph-mon[110183]: pgmap v967: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.6 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 215 op/s; 14609/177156 objects degraded (8.246%); 4380/177156 objects misplaced (2.472%); 7.3 MiB/s, 116 objects/s recovering 2024-09-13T14:43:05.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:05 smithi102 ceph-mon[115269]: pgmap v967: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.6 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 215 op/s; 14609/177156 objects degraded (8.246%); 4380/177156 objects misplaced (2.472%); 7.3 MiB/s, 116 objects/s recovering 2024-09-13T14:43:05.672 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:05.696 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:05.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:05.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:05.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:05 smithi195 ceph-mon[104262]: pgmap v967: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.6 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 215 op/s; 14609/177156 objects degraded (8.246%); 4380/177156 objects misplaced (2.472%); 7.3 MiB/s, 116 objects/s recovering 2024-09-13T14:43:05.799 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:05.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:05.822 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:05.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:05.923 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:05.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:05.953 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:05.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:06.054 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:06.076 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:06.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:06.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:06.178 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:06.201 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:06.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:06.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:06.302 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:06.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:06.325 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:06.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:06.427 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:06.451 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:06.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:06.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:06.553 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:06.577 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:06.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:06.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:06.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:43:06 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:43:06] "GET /metrics HTTP/1.1" 200 36551 "" "Prometheus/2.51.0" 2024-09-13T14:43:06.678 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:06.701 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:06.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:06.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:06.803 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:06.830 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:06.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:06.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:06.933 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:06.955 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:06.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:06.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:07.057 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:07.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:07.085 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:07.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:07.189 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:07.216 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:07.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:07.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:07.318 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:07.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:07.343 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:07.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:07.444 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:07.463 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:07 smithi102 ceph-mon[110183]: pgmap v968: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.6 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 215 op/s; 14609/177156 objects degraded (8.246%); 4380/177156 objects misplaced (2.472%); 7.3 MiB/s, 116 objects/s recovering 2024-09-13T14:43:07.463 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:07 smithi102 ceph-mon[115269]: pgmap v968: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.6 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 215 op/s; 14609/177156 objects degraded (8.246%); 4380/177156 objects misplaced (2.472%); 7.3 MiB/s, 116 objects/s recovering 2024-09-13T14:43:07.468 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:07.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:07.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:07.569 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:07.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:07.600 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:07.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:07.701 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:07.724 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:07.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:07.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:07.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:07 smithi195 ceph-mon[104262]: pgmap v968: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.6 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 215 op/s; 14609/177156 objects degraded (8.246%); 4380/177156 objects misplaced (2.472%); 7.3 MiB/s, 116 objects/s recovering 2024-09-13T14:43:07.826 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:07.848 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:07.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:07.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:07.950 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:07.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:07.975 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:07.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:08.076 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:08.098 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:08.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:08.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:08.200 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:08.223 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:08.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:08.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:08.325 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:08.347 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:08.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:08.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:08.438 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:43:08.438 INFO:teuthology.orchestra.run.smithi102.stdout: "size": 20, 2024-09-13T14:43:08.438 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 600, 2024-09-13T14:43:08.438 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [] 2024-09-13T14:43:08.439 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:43:08.450 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:08.473 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:08.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:08.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:08.575 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:08.597 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:08.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:08.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:08.699 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:08.724 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:08.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:08.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:08.826 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:08.852 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:08.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:08.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:08.954 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:08.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:08.977 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:08.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:09.078 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:09.100 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:09.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:09.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:09.202 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:09.225 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:09.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:09.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:09.327 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:09.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:09.349 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:09.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:09.451 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:09.475 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:09.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:09.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:09.578 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:09.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:09.601 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:09.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:09.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:09 smithi102 ceph-mon[110183]: pgmap v969: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.5 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 251 op/s; 14467/173955 objects degraded (8.317%); 3813/173955 objects misplaced (2.192%); 7.4 MiB/s, 117 objects/s recovering 2024-09-13T14:43:09.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:09 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 14467/173955 objects degraded (8.317%), 6 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:43:09.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:09 smithi102 ceph-mon[115269]: pgmap v969: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.5 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 251 op/s; 14467/173955 objects degraded (8.317%); 3813/173955 objects misplaced (2.192%); 7.4 MiB/s, 117 objects/s recovering 2024-09-13T14:43:09.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:09 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 14467/173955 objects degraded (8.317%), 6 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:43:09.702 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:09.724 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:09.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:09.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:09.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:09 smithi195 ceph-mon[104262]: pgmap v969: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.5 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 251 op/s; 14467/173955 objects degraded (8.317%); 3813/173955 objects misplaced (2.192%); 7.4 MiB/s, 117 objects/s recovering 2024-09-13T14:43:09.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:09 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 14467/173955 objects degraded (8.317%), 6 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:43:09.826 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:09.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:09.849 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:09.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:09.950 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:09.975 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:09.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:09.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:10.077 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:10.099 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:10.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:10.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:10.201 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:10.228 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:10.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:10.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:10.330 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:10.354 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:10.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:10.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:10.456 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:10.479 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:10.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:10.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:10.581 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:10.604 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:10.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:10.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:10.706 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:10.727 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:10.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:10.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:10.828 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:10.856 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:10.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:10.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:10.957 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:10.979 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:10.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:10.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:11.081 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:11.103 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:11.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:11.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:11.205 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:11.231 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:11.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:11.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:11.333 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:11.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:11.355 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:11.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:11.456 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:11.480 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:11.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:11.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:11.582 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:11.605 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:11.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:11.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:11.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:11 smithi102 ceph-mon[110183]: pgmap v970: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.5 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 192 op/s; 14467/173745 objects degraded (8.327%); 3813/173745 objects misplaced (2.195%); 6.8 MiB/s, 108 objects/s recovering 2024-09-13T14:43:11.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:11 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:43:11.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:11 smithi102 ceph-mon[115269]: pgmap v970: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.5 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 192 op/s; 14467/173745 objects degraded (8.327%); 3813/173745 objects misplaced (2.195%); 6.8 MiB/s, 108 objects/s recovering 2024-09-13T14:43:11.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:11 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:43:11.707 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:11.731 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:11.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:11.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:11.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:11 smithi195 ceph-mon[104262]: pgmap v970: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.5 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 192 op/s; 14467/173745 objects degraded (8.327%); 3813/173745 objects misplaced (2.195%); 6.8 MiB/s, 108 objects/s recovering 2024-09-13T14:43:11.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:11 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:43:11.833 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:11.857 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:11.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:11.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:11.959 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:11.986 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:11.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:11.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:12.088 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:12.113 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:12.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:12.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:12.215 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:12.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:12.237 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:12.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:12.338 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:12.361 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:12.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:12.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:12.463 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:12.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:12.486 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:12.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:12.588 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:12.611 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:12.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:12.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:12.713 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:12.737 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:12.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:12.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:12.839 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:12.864 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:12.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:12.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:12.966 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:12.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:12.993 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:12.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:13.094 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:13.116 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:13.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:13.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:13.218 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:13.241 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:13.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:13.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:13.343 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:13.366 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:13.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:13.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:13.468 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:13.493 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:13.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:13.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:13.595 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:13.618 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:13.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:13.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:13.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:13 smithi102 ceph-mon[110183]: pgmap v971: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.5 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 219 op/s; 14344/172560 objects degraded (8.312%); 3092/172560 objects misplaced (1.792%); 11 MiB/s, 179 objects/s recovering 2024-09-13T14:43:13.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:13 smithi102 ceph-mon[115269]: pgmap v971: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.5 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 219 op/s; 14344/172560 objects degraded (8.312%); 3092/172560 objects misplaced (1.792%); 11 MiB/s, 179 objects/s recovering 2024-09-13T14:43:13.720 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:13.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:13 smithi195 ceph-mon[104262]: pgmap v971: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.5 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 219 op/s; 14344/172560 objects degraded (8.312%); 3092/172560 objects misplaced (1.792%); 11 MiB/s, 179 objects/s recovering 2024-09-13T14:43:13.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:13.748 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:13.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:13.849 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:13.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:13.876 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:13.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:13.978 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:13.999 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:14.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:14.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:14.101 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:14.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:14.126 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:14.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:14.227 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:14.252 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:14.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:14.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:14.354 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:14.385 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:14.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:14.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:14.487 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:14.512 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:14.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:14.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:14.605 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:14 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 14344/172560 objects degraded (8.312%), 6 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:43:14.605 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:14 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:43:14.606 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:14 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 14344/172560 objects degraded (8.312%), 6 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:43:14.606 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:14 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:43:14.614 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:14.636 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:14.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:14.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:14.738 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:14.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:14 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 14344/172560 objects degraded (8.312%), 6 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:43:14.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:14 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:43:14.761 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:14.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:14.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:14.863 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:14.885 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:14.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:14.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:14.987 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:15.010 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:15.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:15.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:15.048 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T14:43:15.112 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:15.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:15.159 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:15.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:15.260 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:15.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:15.286 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:15.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:15.387 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:15.417 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:15.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:15.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:15.519 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:15.544 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:15.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:15.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:15.562 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:15 smithi102 ceph-mon[110183]: pgmap v972: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.5 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 245 op/s; 14344/170382 objects degraded (8.419%); 3092/170382 objects misplaced (1.815%); 8.1 MiB/s, 129 objects/s recovering 2024-09-13T14:43:15.562 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:15 smithi102 ceph-mon[115269]: pgmap v972: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.5 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 245 op/s; 14344/170382 objects degraded (8.419%); 3092/170382 objects misplaced (1.815%); 8.1 MiB/s, 129 objects/s recovering 2024-09-13T14:43:15.646 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:15.673 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:15.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:15.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:15.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:15 smithi195 ceph-mon[104262]: pgmap v972: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.5 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 245 op/s; 14344/170382 objects degraded (8.419%); 3092/170382 objects misplaced (1.815%); 8.1 MiB/s, 129 objects/s recovering 2024-09-13T14:43:15.775 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:15.798 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:15.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:15.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:15.900 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:15.923 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:15.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:15.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:15.993 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:43:15.993 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:43:16.013 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:43:16.013 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:43:16.024 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:16.041 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:43:16.042 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:43:16.047 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:16.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:16.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:16.062 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:43:16.062 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:43:16.083 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:16.084 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:16.085 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:43:16.085 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:43:16.108 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:43:16.109 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:43:16.133 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:43:16.133 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:43:16.149 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:16.153 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:43:16.153 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:43:16.177 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:16.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:16.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:16.279 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:16.301 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is unset 2024-09-13T14:43:16.305 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:16.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:16.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:16.395 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:16 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/78642064' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:43:16.395 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:16 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:43:16.395 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:16 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3593448434' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:43:16.396 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:16 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/78642064' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:43:16.396 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:16 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:43:16.396 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:16 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3593448434' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:43:16.406 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:16.430 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:16.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:16.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:16.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:16 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/78642064' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:43:16.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:16 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:43:16.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:16 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3593448434' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:43:16.532 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:16.566 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:16.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:16.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:16.668 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:16.669 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:43:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:43:16] "GET /metrics HTTP/1.1" 200 36549 "" "Prometheus/2.51.0" 2024-09-13T14:43:16.691 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:16.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:16.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:16.793 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:16.816 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:16.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:16.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:16.918 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:16.941 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:16.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:16.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:17.043 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:17.065 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:17.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:17.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:17.167 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:17.189 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:17.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:17.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:17.291 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:17.319 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:17.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:17.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:17.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:17 smithi102 ceph-mon[115269]: pgmap v973: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.5 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 188 op/s; 14344/170382 objects degraded (8.419%); 3092/170382 objects misplaced (1.815%); 8.1 MiB/s, 129 objects/s recovering 2024-09-13T14:43:17.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:17 smithi102 ceph-mon[115269]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:43:17.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:17 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:43:17.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:17 smithi102 ceph-mon[115269]: osdmap e286: 8 total, 8 up, 6 in 2024-09-13T14:43:17.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:17 smithi102 ceph-mon[110183]: pgmap v973: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.5 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 188 op/s; 14344/170382 objects degraded (8.419%); 3092/170382 objects misplaced (1.815%); 8.1 MiB/s, 129 objects/s recovering 2024-09-13T14:43:17.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:17 smithi102 ceph-mon[110183]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:43:17.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:17 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:43:17.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:17 smithi102 ceph-mon[110183]: osdmap e286: 8 total, 8 up, 6 in 2024-09-13T14:43:17.420 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:17.442 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:17.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:17.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:17.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:17 smithi195 ceph-mon[104262]: pgmap v973: 89 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering, 82 active+clean; 3.5 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 188 op/s; 14344/170382 objects degraded (8.419%); 3092/170382 objects misplaced (1.815%); 8.1 MiB/s, 129 objects/s recovering 2024-09-13T14:43:17.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:17 smithi195 ceph-mon[104262]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:43:17.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:17 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:43:17.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:17 smithi195 ceph-mon[104262]: osdmap e286: 8 total, 8 up, 6 in 2024-09-13T14:43:17.543 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:17.571 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:17.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:17.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:17.673 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:17.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:17.701 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:17.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:17.802 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:17.825 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:17.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:17.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:17.927 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:17.949 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:17.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:17.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:18.052 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:18.078 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:18.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:18.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:18.180 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:18.203 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:18.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:18.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:18.305 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:18.327 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:18.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:18.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:18.429 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:18.454 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:18.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:18.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:18.556 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:18.582 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:18.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:18.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:18.684 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:18.710 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:18.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:18.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:18.812 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:18.835 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:18.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:18.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:18.937 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:18.962 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:18.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:18.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:19.064 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:19.091 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:19.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:19.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:19.193 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:19.219 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:19.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:19.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:19.321 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:19.349 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:19.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:19.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:19.366 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:19 smithi195 ceph-mon[104262]: pgmap v975: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 83 active+clean; 3.4 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 228 op/s; 14187/167085 objects degraded (8.491%); 2091/167085 objects misplaced (1.251%); 12 MiB/s, 200 objects/s recovering 2024-09-13T14:43:19.366 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:19 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 14187/167085 objects degraded (8.491%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:43:19.450 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:19.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:19.478 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:19.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:19.580 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:19.608 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:19.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:19.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:19.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:19 smithi102 ceph-mon[110183]: pgmap v975: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 83 active+clean; 3.4 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 228 op/s; 14187/167085 objects degraded (8.491%); 2091/167085 objects misplaced (1.251%); 12 MiB/s, 200 objects/s recovering 2024-09-13T14:43:19.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:19 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 14187/167085 objects degraded (8.491%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:43:19.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:19 smithi102 ceph-mon[115269]: pgmap v975: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 83 active+clean; 3.4 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 228 op/s; 14187/167085 objects degraded (8.491%); 2091/167085 objects misplaced (1.251%); 12 MiB/s, 200 objects/s recovering 2024-09-13T14:43:19.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:19 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 14187/167085 objects degraded (8.491%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:43:19.710 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:19.733 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:19.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:19.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:19.835 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:19.862 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:19.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:19.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:19.964 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:19.987 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:19.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:19.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:20.089 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:20.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:20.112 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:20.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:20.214 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:20.241 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:20.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:20.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:20.344 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:20.367 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:20.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:20.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:20.469 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:20.490 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:20.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:20.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:20.592 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:20.617 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:20.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:20.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:20.718 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:20.741 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:20.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:20.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:20.843 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:20.870 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:20.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:20.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:20.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:20 smithi102 ceph-mon[110183]: pgmap v976: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 83 active+clean; 3.4 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 228 op/s; 14040/166896 objects degraded (8.412%); 2091/166896 objects misplaced (1.253%); 13 MiB/s, 208 objects/s recovering 2024-09-13T14:43:20.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:20 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:43:20.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:20 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:43:20.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:20 smithi102 ceph-mon[115269]: pgmap v976: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 83 active+clean; 3.4 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 228 op/s; 14040/166896 objects degraded (8.412%); 2091/166896 objects misplaced (1.253%); 13 MiB/s, 208 objects/s recovering 2024-09-13T14:43:20.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:20 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:43:20.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:20 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:43:20.972 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:20.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:20 smithi195 ceph-mon[104262]: pgmap v976: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 83 active+clean; 3.4 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 228 op/s; 14040/166896 objects degraded (8.412%); 2091/166896 objects misplaced (1.253%); 13 MiB/s, 208 objects/s recovering 2024-09-13T14:43:20.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:20 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:43:20.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:20 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:43:21.000 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:21.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:21.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:21.102 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:21.125 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:21.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:21.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:21.227 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:21.253 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:21.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:21.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:21.355 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:21.363 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:43:21.363 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:43:21.378 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:21.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:21.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:21.481 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:21.504 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:21.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:21.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:21.606 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:21.628 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:21.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:21.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:21.730 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:21.751 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:21.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:21.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:21.857 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:21.879 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:21.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:21.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:21.981 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:21.990 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:43:22.003 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:22.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:22.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:22.104 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:22.132 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:22.133 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T14:43:22.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:22.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:22.235 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:22.258 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:22.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:22.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:22.360 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:22.385 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:22.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:22.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:22.487 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:22.512 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:22.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:22.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:22.614 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:22.636 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:22.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:22.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:22.738 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:22.759 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:22.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:22.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:22.861 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:22.888 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:22.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:22.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:22.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:22 smithi102 ceph-mon[110183]: pgmap v977: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.4 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 242 op/s; 13870/165279 objects degraded (8.392%); 1987/165279 objects misplaced (1.202%); 9.4 MiB/s, 157 objects/s recovering 2024-09-13T14:43:22.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:22 smithi102 ceph-mon[115269]: pgmap v977: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.4 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 242 op/s; 13870/165279 objects degraded (8.392%); 1987/165279 objects misplaced (1.202%); 9.4 MiB/s, 157 objects/s recovering 2024-09-13T14:43:22.990 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:22.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:22 smithi195 ceph-mon[104262]: pgmap v977: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.4 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 242 op/s; 13870/165279 objects degraded (8.392%); 1987/165279 objects misplaced (1.202%); 9.4 MiB/s, 157 objects/s recovering 2024-09-13T14:43:23.004 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:43:23.015 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:23.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:23.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:23.107 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:23.108 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:23.117 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:23.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:23.172 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:23.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:23.273 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:23.294 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:23.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:23.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:23.383 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:43:23.384 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:43:23.397 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:23.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:23.422 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:23.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:23.523 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:23.552 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:23.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:23.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:23.654 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:23.678 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:23.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:23.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:23.780 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:23.804 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:23.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:23.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:23.906 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:23.931 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:23.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:23.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:24.032 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:24.054 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:24.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:24.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:24.156 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:24.180 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:24.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:24.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:24.280 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":977,"stamp":"2024-09-13T14:43:21.526609+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3613464573,"num_objects":55093,"num_object_clones":0,"num_object_copies":165279,"num_objects_missing_on_primary":83,"num_objects_missing":83,"num_objects_degraded":13870,"num_objects_misplaced":1987,"num_objects_unfound":0,"num_objects_dirty":55093,"num_whiteouts":0,"num_read":6377,"num_read_kb":13834,"num_write":171684,"num_write_kb":7235010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78460,"num_bytes_recovered":5079633037,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95966,"ondisk_log_size":95966,"up":267,"acting":264,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":302,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":16031880,"kb_used_data":12752136,"kb_used_omap":26,"kb_used_meta":3279653,"kb_avail":546439032,"statfs":{"total":575970213888,"available":559553568768,"internally_reserved":0,"allocated":13058187264,"data_stored":13048997821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27144,"internal_metadata":3358365176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-159055872,"num_objects":-2427,"num_object_clones":0,"num_object_copies":-7281,"num_objects_missing_on_primary":-7,"num_objects_missing":-7,"num_objects_degraded":-474,"num_objects_misplaced":-1105,"num_objects_unfound":0,"num_objects_dirty":-2427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2427,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1580,"num_bytes_recovered":98697216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002628"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":465,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.493577+0000","last_change":"2024-09-13T14:36:13.859927+0000","last_active":"2024-09-13T14:43:16.493577+0000","last_peered":"2024-09-13T14:43:16.493577+0000","last_clean":"2024-09-13T14:43:16.493577+0000","last_became_active":"2024-09-13T14:36:13.859505+0000","last_became_peered":"2024-09-13T14:36:13.859505+0000","last_unstale":"2024-09-13T14:43:16.493577+0000","last_undegraded":"2024-09-13T14:43:16.493577+0000","last_fullsized":"2024-09-13T14:43:16.493577+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:12:20.989627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":288,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.439721+0000","last_change":"2024-09-13T14:37:48.130968+0000","last_active":"2024-09-13T14:42:56.439721+0000","last_peered":"2024-09-13T14:42:56.439721+0000","last_clean":"2024-09-13T14:42:56.439721+0000","last_became_active":"2024-09-13T14:37:48.130603+0000","last_became_peered":"2024-09-13T14:37:48.130603+0000","last_unstale":"2024-09-13T14:42:56.439721+0000","last_undegraded":"2024-09-13T14:42:56.439721+0000","last_fullsized":"2024-09-13T14:42:56.439721+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:42.626386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":84,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.454141+0000","last_change":"2024-09-13T14:36:13.862181+0000","last_active":"2024-09-13T14:42:56.454141+0000","last_peered":"2024-09-13T14:42:56.454141+0000","last_clean":"2024-09-13T14:42:56.454141+0000","last_became_active":"2024-09-13T14:36:13.861873+0000","last_became_peered":"2024-09-13T14:36:13.861873+0000","last_unstale":"2024-09-13T14:42:56.454141+0000","last_undegraded":"2024-09-13T14:42:56.454141+0000","last_fullsized":"2024-09-13T14:42:56.454141+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:14.085700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":103,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.353148+0000","last_change":"2024-09-13T14:38:53.896672+0000","last_active":"2024-09-13T14:43:16.353148+0000","last_peered":"2024-09-13T14:43:16.353148+0000","last_clean":"2024-09-13T14:43:16.353148+0000","last_became_active":"2024-09-13T14:38:53.896346+0000","last_became_peered":"2024-09-13T14:38:53.896346+0000","last_unstale":"2024-09-13T14:43:16.353148+0000","last_undegraded":"2024-09-13T14:43:16.353148+0000","last_fullsized":"2024-09-13T14:43:16.353148+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:57.789686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":103,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.353656+0000","last_change":"2024-09-13T14:38:53.893021+0000","last_active":"2024-09-13T14:43:16.353656+0000","last_peered":"2024-09-13T14:43:16.353656+0000","last_clean":"2024-09-13T14:43:16.353656+0000","last_became_active":"2024-09-13T14:38:53.892668+0000","last_became_peered":"2024-09-13T14:38:53.892668+0000","last_unstale":"2024-09-13T14:43:16.353656+0000","last_undegraded":"2024-09-13T14:43:16.353656+0000","last_fullsized":"2024-09-13T14:43:16.353656+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:57.180738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":256,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.439552+0000","last_change":"2024-09-13T14:37:48.134642+0000","last_active":"2024-09-13T14:42:56.439552+0000","last_peered":"2024-09-13T14:42:56.439552+0000","last_clean":"2024-09-13T14:42:56.439552+0000","last_became_active":"2024-09-13T14:37:48.134402+0000","last_became_peered":"2024-09-13T14:37:48.134402+0000","last_unstale":"2024-09-13T14:42:56.439552+0000","last_undegraded":"2024-09-13T14:42:56.439552+0000","last_fullsized":"2024-09-13T14:42:56.439552+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:51.313663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":484,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.494009+0000","last_change":"2024-09-13T14:38:53.903022+0000","last_active":"2024-09-13T14:43:16.494009+0000","last_peered":"2024-09-13T14:43:16.494009+0000","last_clean":"2024-09-13T14:43:16.494009+0000","last_became_active":"2024-09-13T14:38:53.902448+0000","last_became_peered":"2024-09-13T14:38:53.902448+0000","last_unstale":"2024-09-13T14:43:16.494009+0000","last_undegraded":"2024-09-13T14:43:16.494009+0000","last_fullsized":"2024-09-13T14:43:16.494009+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:24.690401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":106,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.352448+0000","last_change":"2024-09-13T14:38:03.554500+0000","last_active":"2024-09-13T14:43:16.352448+0000","last_peered":"2024-09-13T14:43:16.352448+0000","last_clean":"2024-09-13T14:43:16.352448+0000","last_became_active":"2024-09-13T14:38:03.532910+0000","last_became_peered":"2024-09-13T14:38:03.532910+0000","last_unstale":"2024-09-13T14:43:16.352448+0000","last_undegraded":"2024-09-13T14:43:16.352448+0000","last_fullsized":"2024-09-13T14:43:16.352448+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:27:50.843498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"286'10522","reported_seq":21419,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.764431+0000","last_change":"2024-09-13T14:41:53.220083+0000","last_active":"2024-09-13T14:43:16.764431+0000","last_peered":"2024-09-13T14:43:16.764431+0000","last_clean":"2024-09-13T14:43:16.764431+0000","last_became_active":"2024-09-13T14:41:53.200084+0000","last_became_peered":"2024-09-13T14:41:53.200084+0000","last_unstale":"2024-09-13T14:43:16.764431+0000","last_undegraded":"2024-09-13T14:43:16.764431+0000","last_fullsized":"2024-09-13T14:43:16.764431+0000","mapping_epoch":275,"log_start":"216'4800","ondisk_log_start":"216'4800","created":205,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5722,"log_dups_size":0,"ondisk_log_size":5722,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:08.897524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220069888,"num_objects":3358,"num_object_clones":0,"num_object_copies":10074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10522,"num_write_kb":444160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9152,"num_bytes_recovered":594608128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":48,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.440224+0000","last_change":"2024-09-13T14:38:53.940238+0000","last_active":"2024-09-13T14:42:56.440224+0000","last_peered":"2024-09-13T14:42:56.440224+0000","last_clean":"2024-09-13T14:42:56.440224+0000","last_became_active":"2024-09-13T14:38:53.892825+0000","last_became_peered":"2024-09-13T14:38:53.892825+0000","last_unstale":"2024-09-13T14:42:56.440224+0000","last_undegraded":"2024-09-13T14:42:56.440224+0000","last_fullsized":"2024-09-13T14:42:56.440224+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:46.666581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":105,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.453169+0000","last_change":"2024-09-13T14:38:53.898907+0000","last_active":"2024-09-13T14:42:56.453169+0000","last_peered":"2024-09-13T14:42:56.453169+0000","last_clean":"2024-09-13T14:42:56.453169+0000","last_became_active":"2024-09-13T14:38:53.898572+0000","last_became_peered":"2024-09-13T14:38:53.898572+0000","last_unstale":"2024-09-13T14:42:56.453169+0000","last_undegraded":"2024-09-13T14:42:56.453169+0000","last_fullsized":"2024-09-13T14:42:56.453169+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:29.222150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"286'10773","reported_seq":24955,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:17.510942+0000","last_change":"2024-09-13T14:38:57.930032+0000","last_active":"2024-09-13T14:43:17.510942+0000","last_peered":"2024-09-13T14:43:17.510942+0000","last_clean":"2024-09-13T14:43:17.510942+0000","last_became_active":"2024-09-13T14:38:57.921458+0000","last_became_peered":"2024-09-13T14:38:57.921458+0000","last_unstale":"2024-09-13T14:43:17.510942+0000","last_undegraded":"2024-09-13T14:43:17.510942+0000","last_fullsized":"2024-09-13T14:43:17.510942+0000","mapping_epoch":256,"log_start":"218'5039","ondisk_log_start":"218'5039","created":205,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5734,"log_dups_size":0,"ondisk_log_size":5734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:27:12.294038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227344384,"num_objects":3469,"num_object_clones":0,"num_object_copies":10407,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10773,"num_write_kb":455744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6990,"num_bytes_recovered":458096640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":48,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.440156+0000","last_change":"2024-09-13T14:38:53.904885+0000","last_active":"2024-09-13T14:42:56.440156+0000","last_peered":"2024-09-13T14:42:56.440156+0000","last_clean":"2024-09-13T14:42:56.440156+0000","last_became_active":"2024-09-13T14:38:53.891726+0000","last_became_peered":"2024-09-13T14:38:53.891726+0000","last_unstale":"2024-09-13T14:42:56.440156+0000","last_undegraded":"2024-09-13T14:42:56.440156+0000","last_fullsized":"2024-09-13T14:42:56.440156+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:57.079205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":294,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.437588+0000","last_change":"2024-09-13T14:37:48.160504+0000","last_active":"2024-09-13T14:42:56.437588+0000","last_peered":"2024-09-13T14:42:56.437588+0000","last_clean":"2024-09-13T14:42:56.437588+0000","last_became_active":"2024-09-13T14:37:48.160181+0000","last_became_peered":"2024-09-13T14:37:48.160181+0000","last_unstale":"2024-09-13T14:42:56.437588+0000","last_undegraded":"2024-09-13T14:42:56.437588+0000","last_fullsized":"2024-09-13T14:42:56.437588+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:37.087008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"286'10750","reported_seq":12554,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:21.209656+0000","last_change":"2024-09-13T14:38:04.531914+0000","last_active":"2024-09-13T14:43:21.209656+0000","last_peered":"2024-09-13T14:43:21.209656+0000","last_clean":"2024-09-13T14:43:21.209656+0000","last_became_active":"2024-09-13T14:38:04.523448+0000","last_became_peered":"2024-09-13T14:38:04.523448+0000","last_unstale":"2024-09-13T14:43:21.209656+0000","last_undegraded":"2024-09-13T14:43:21.209656+0000","last_fullsized":"2024-09-13T14:43:21.209656+0000","mapping_epoch":245,"log_start":"216'4800","ondisk_log_start":"216'4800","created":205,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5950,"log_dups_size":0,"ondisk_log_size":5950,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:06.536896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224395264,"num_objects":3424,"num_object_clones":0,"num_object_copies":10272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10750,"num_write_kb":453568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":811,"num_bytes_recovered":53149696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":286,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:43:02.427494+0000","last_change":"2024-09-13T14:37:49.611774+0000","last_active":"2024-09-13T14:43:02.427494+0000","last_peered":"2024-09-13T14:43:02.427494+0000","last_clean":"2024-09-13T14:43:02.427494+0000","last_became_active":"2024-09-13T14:37:49.611088+0000","last_became_peered":"2024-09-13T14:37:49.611088+0000","last_unstale":"2024-09-13T14:43:02.427494+0000","last_undegraded":"2024-09-13T14:43:02.427494+0000","last_fullsized":"2024-09-13T14:43:02.427494+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:08.933213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":45,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.302107+0000","last_change":"2024-09-13T14:38:53.899211+0000","last_active":"2024-09-13T14:43:16.302107+0000","last_peered":"2024-09-13T14:43:16.302107+0000","last_clean":"2024-09-13T14:43:16.302107+0000","last_became_active":"2024-09-13T14:38:53.898927+0000","last_became_peered":"2024-09-13T14:38:53.898927+0000","last_unstale":"2024-09-13T14:43:16.302107+0000","last_undegraded":"2024-09-13T14:43:16.302107+0000","last_fullsized":"2024-09-13T14:43:16.302107+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:00:28.857611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"286'10527","reported_seq":10620,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:17.307777+0000","last_change":"2024-09-13T14:35:24.004651+0000","last_active":"2024-09-13T14:43:17.307777+0000","last_peered":"2024-09-13T14:43:17.307777+0000","last_clean":"2024-09-13T14:43:17.307777+0000","last_became_active":"2024-09-13T14:35:24.004358+0000","last_became_peered":"2024-09-13T14:35:24.004358+0000","last_unstale":"2024-09-13T14:43:17.307777+0000","last_undegraded":"2024-09-13T14:43:17.307777+0000","last_fullsized":"2024-09-13T14:43:17.307777+0000","mapping_epoch":205,"log_start":"213'4184","ondisk_log_start":"213'4184","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6343,"log_dups_size":0,"ondisk_log_size":6343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:57:38.023669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":225771520,"num_objects":3445,"num_object_clones":0,"num_object_copies":10335,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10527,"num_write_kb":447104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":96,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:43:02.285014+0000","last_change":"2024-09-13T14:37:49.168003+0000","last_active":"2024-09-13T14:43:02.285014+0000","last_peered":"2024-09-13T14:43:02.285014+0000","last_clean":"2024-09-13T14:43:02.285014+0000","last_became_active":"2024-09-13T14:37:49.167638+0000","last_became_peered":"2024-09-13T14:37:49.167638+0000","last_unstale":"2024-09-13T14:43:02.285014+0000","last_undegraded":"2024-09-13T14:43:02.285014+0000","last_fullsized":"2024-09-13T14:43:02.285014+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:29.051981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":96,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.353190+0000","last_change":"2024-09-13T14:38:03.528381+0000","last_active":"2024-09-13T14:43:16.353190+0000","last_peered":"2024-09-13T14:43:16.353190+0000","last_clean":"2024-09-13T14:43:16.353190+0000","last_became_active":"2024-09-13T14:38:03.527282+0000","last_became_peered":"2024-09-13T14:38:03.527282+0000","last_unstale":"2024-09-13T14:43:16.353190+0000","last_undegraded":"2024-09-13T14:43:16.353190+0000","last_fullsized":"2024-09-13T14:43:16.353190+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:24.435067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.f","version":"286'10662","reported_seq":10961,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:21.317202+0000","last_change":"2024-09-13T14:38:16.371079+0000","last_active":"2024-09-13T14:43:21.317202+0000","last_peered":"2024-09-13T14:43:21.317202+0000","last_clean":"2024-09-13T14:43:21.317202+0000","last_became_active":"2024-09-13T14:38:12.859883+0000","last_became_peered":"2024-09-13T14:38:12.859883+0000","last_unstale":"2024-09-13T14:43:21.317202+0000","last_undegraded":"2024-09-13T14:43:21.317202+0000","last_fullsized":"2024-09-13T14:43:21.317202+0000","mapping_epoch":245,"log_start":"213'4763","ondisk_log_start":"213'4763","created":205,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5899,"log_dups_size":0,"ondisk_log_size":5899,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:08.436532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224395264,"num_objects":3424,"num_object_clones":0,"num_object_copies":10272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10662,"num_write_kb":450752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":3670016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":454,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.493550+0000","last_change":"2024-09-13T14:38:03.549595+0000","last_active":"2024-09-13T14:43:16.493550+0000","last_peered":"2024-09-13T14:43:16.493550+0000","last_clean":"2024-09-13T14:43:16.493550+0000","last_became_active":"2024-09-13T14:38:03.533331+0000","last_became_peered":"2024-09-13T14:38:03.533331+0000","last_unstale":"2024-09-13T14:43:16.493550+0000","last_undegraded":"2024-09-13T14:43:16.493550+0000","last_fullsized":"2024-09-13T14:43:16.493550+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:59:02.470343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":270,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:57.799891+0000","last_change":"2024-09-13T14:37:48.127600+0000","last_active":"2024-09-13T14:42:57.799891+0000","last_peered":"2024-09-13T14:42:57.799891+0000","last_clean":"2024-09-13T14:42:57.799891+0000","last_became_active":"2024-09-13T14:37:48.127349+0000","last_became_peered":"2024-09-13T14:37:48.127349+0000","last_unstale":"2024-09-13T14:42:57.799891+0000","last_undegraded":"2024-09-13T14:42:57.799891+0000","last_fullsized":"2024-09-13T14:42:57.799891+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:57.268374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"286'10623","reported_seq":20613,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:17.112945+0000","last_change":"2024-09-13T14:42:30.227259+0000","last_active":"2024-09-13T14:43:17.112945+0000","last_peered":"2024-09-13T14:43:17.112945+0000","last_clean":"2024-09-13T14:43:17.112945+0000","last_became_active":"2024-09-13T14:42:30.209102+0000","last_became_peered":"2024-09-13T14:42:30.209102+0000","last_unstale":"2024-09-13T14:43:17.112945+0000","last_undegraded":"2024-09-13T14:43:17.112945+0000","last_fullsized":"2024-09-13T14:43:17.112945+0000","mapping_epoch":281,"log_start":"217'4900","ondisk_log_start":"217'4900","created":205,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5723,"log_dups_size":0,"ondisk_log_size":5723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:56:16.193287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":221970432,"num_objects":3387,"num_object_clones":0,"num_object_copies":10161,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10623,"num_write_kb":448320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4926,"num_bytes_recovered":313851904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":203,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.438029+0000","last_change":"2024-09-13T14:37:48.127604+0000","last_active":"2024-09-13T14:42:56.438029+0000","last_peered":"2024-09-13T14:42:56.438029+0000","last_clean":"2024-09-13T14:42:56.438029+0000","last_became_active":"2024-09-13T14:37:48.127300+0000","last_became_peered":"2024-09-13T14:37:48.127300+0000","last_unstale":"2024-09-13T14:42:56.438029+0000","last_undegraded":"2024-09-13T14:42:56.438029+0000","last_fullsized":"2024-09-13T14:42:56.438029+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:51.999506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":323,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:43:02.800497+0000","last_change":"2024-09-13T14:38:03.529967+0000","last_active":"2024-09-13T14:43:02.800497+0000","last_peered":"2024-09-13T14:43:02.800497+0000","last_clean":"2024-09-13T14:43:02.800497+0000","last_became_active":"2024-09-13T14:38:03.529646+0000","last_became_peered":"2024-09-13T14:38:03.529646+0000","last_unstale":"2024-09-13T14:43:02.800497+0000","last_undegraded":"2024-09-13T14:43:02.800497+0000","last_fullsized":"2024-09-13T14:43:02.800497+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:52:05.193279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"286'10745","reported_seq":19919,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:21.337272+0000","last_change":"2024-09-13T14:40:34.583237+0000","last_active":"2024-09-13T14:43:21.337272+0000","last_peered":"2024-09-13T14:43:21.337272+0000","last_clean":"2024-09-13T14:43:21.337272+0000","last_became_active":"2024-09-13T14:40:34.563989+0000","last_became_peered":"2024-09-13T14:40:34.563989+0000","last_unstale":"2024-09-13T14:43:21.337272+0000","last_undegraded":"2024-09-13T14:43:21.337272+0000","last_fullsized":"2024-09-13T14:43:21.337272+0000","mapping_epoch":266,"log_start":"216'4800","ondisk_log_start":"216'4800","created":205,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5945,"log_dups_size":0,"ondisk_log_size":5945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:34:43.006051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222429208,"num_objects":3395,"num_object_clones":0,"num_object_copies":10185,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3395,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10745,"num_write_kb":452417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4489,"num_bytes_recovered":289865752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":283,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.437833+0000","last_change":"2024-09-13T14:36:13.860367+0000","last_active":"2024-09-13T14:42:56.437833+0000","last_peered":"2024-09-13T14:42:56.437833+0000","last_clean":"2024-09-13T14:42:56.437833+0000","last_became_active":"2024-09-13T14:36:13.859982+0000","last_became_peered":"2024-09-13T14:36:13.859982+0000","last_unstale":"2024-09-13T14:42:56.437833+0000","last_undegraded":"2024-09-13T14:42:56.437833+0000","last_fullsized":"2024-09-13T14:42:56.437833+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:08:23.311350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":279,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:43:02.800520+0000","last_change":"2024-09-13T14:37:48.147360+0000","last_active":"2024-09-13T14:43:02.800520+0000","last_peered":"2024-09-13T14:43:02.800520+0000","last_clean":"2024-09-13T14:43:02.800520+0000","last_became_active":"2024-09-13T14:37:48.146885+0000","last_became_peered":"2024-09-13T14:37:48.146885+0000","last_unstale":"2024-09-13T14:43:02.800520+0000","last_undegraded":"2024-09-13T14:43:02.800520+0000","last_fullsized":"2024-09-13T14:43:02.800520+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:27.841810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"286'10575","reported_seq":19856,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:17.504690+0000","last_change":"2024-09-13T14:38:39.650332+0000","last_active":"2024-09-13T14:43:17.504690+0000","last_peered":"2024-09-13T14:43:17.504690+0000","last_clean":"2024-09-13T14:43:17.504690+0000","last_became_active":"2024-09-13T14:38:39.640835+0000","last_became_peered":"2024-09-13T14:38:39.640835+0000","last_unstale":"2024-09-13T14:43:17.504690+0000","last_undegraded":"2024-09-13T14:43:17.504690+0000","last_fullsized":"2024-09-13T14:43:17.504690+0000","mapping_epoch":251,"log_start":"216'4851","ondisk_log_start":"216'4851","created":205,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5724,"log_dups_size":0,"ondisk_log_size":5724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:48.176606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229310464,"num_objects":3499,"num_object_clones":0,"num_object_copies":10497,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10575,"num_write_kb":450368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4544,"num_bytes_recovered":292945920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":472,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.493522+0000","last_change":"2024-09-13T14:38:53.900041+0000","last_active":"2024-09-13T14:43:16.493522+0000","last_peered":"2024-09-13T14:43:16.493522+0000","last_clean":"2024-09-13T14:43:16.493522+0000","last_became_active":"2024-09-13T14:38:53.899203+0000","last_became_peered":"2024-09-13T14:38:53.899203+0000","last_unstale":"2024-09-13T14:43:16.493522+0000","last_undegraded":"2024-09-13T14:43:16.493522+0000","last_fullsized":"2024-09-13T14:43:16.493522+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:28:07.883094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":404,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.494092+0000","last_change":"2024-09-13T14:26:15.823198+0000","last_active":"2024-09-13T14:43:16.494092+0000","last_peered":"2024-09-13T14:43:16.494092+0000","last_clean":"2024-09-13T14:43:16.494092+0000","last_became_active":"2024-09-13T14:26:15.822717+0000","last_became_peered":"2024-09-13T14:26:15.822717+0000","last_unstale":"2024-09-13T14:43:16.494092+0000","last_undegraded":"2024-09-13T14:43:16.494092+0000","last_fullsized":"2024-09-13T14:43:16.494092+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:22.471622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.6","version":"286'10808","reported_seq":18204,"reported_epoch":286,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-13T14:43:21.283757+0000","last_change":"2024-09-13T14:38:55.132683+0000","last_active":"2024-09-13T14:43:21.283757+0000","last_peered":"2024-09-13T14:43:21.283757+0000","last_clean":"2024-09-13T14:36:13.124368+0000","last_became_active":"2024-09-13T14:38:55.120883+0000","last_became_peered":"2024-09-13T14:38:55.120883+0000","last_unstale":"2024-09-13T14:43:21.283757+0000","last_undegraded":"2024-09-13T14:38:55.026610+0000","last_fullsized":"2024-09-13T14:43:21.283757+0000","mapping_epoch":254,"log_start":"217'4919","ondisk_log_start":"217'4919","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5889,"log_dups_size":0,"ondisk_log_size":5889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222167040,"num_objects":3390,"num_object_clones":0,"num_object_copies":10170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1403,"num_objects_misplaced":1987,"num_objects_unfound":0,"num_objects_dirty":3390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10808,"num_write_kb":454336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3602,"num_bytes_recovered":224854016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,3,5],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3,4","objects":1124},{"shards":"0,3,5","objects":2266}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":419,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.439461+0000","last_change":"2024-09-13T14:41:53.196835+0000","last_active":"2024-09-13T14:42:56.439461+0000","last_peered":"2024-09-13T14:42:56.439461+0000","last_clean":"2024-09-13T14:42:56.439461+0000","last_became_active":"2024-09-13T14:41:53.196387+0000","last_became_peered":"2024-09-13T14:41:53.196387+0000","last_unstale":"2024-09-13T14:42:56.439461+0000","last_undegraded":"2024-09-13T14:42:56.439461+0000","last_fullsized":"2024-09-13T14:42:56.439461+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:07:41.927740+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:49.972303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":525,"reported_epoch":285,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:42:56.453618+0000","last_change":"2024-09-13T14:38:54.904329+0000","last_active":"2024-09-13T14:42:56.453618+0000","last_peered":"2024-09-13T14:42:56.453618+0000","last_clean":"2024-09-13T14:38:39.648017+0000","last_became_active":"2024-09-13T14:38:54.903798+0000","last_became_peered":"2024-09-13T14:38:54.903798+0000","last_unstale":"2024-09-13T14:42:56.453618+0000","last_undegraded":"2024-09-13T14:38:54.890505+0000","last_fullsized":"2024-09-13T14:38:54.889701+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:18.120931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,3],"avail_no_missing":["5","3","6"],"object_location_counts":[{"shards":"3,5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":406,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.493570+0000","last_change":"2024-09-13T14:37:49.160719+0000","last_active":"2024-09-13T14:43:16.493570+0000","last_peered":"2024-09-13T14:43:16.493570+0000","last_clean":"2024-09-13T14:43:16.493570+0000","last_became_active":"2024-09-13T14:37:49.160401+0000","last_became_peered":"2024-09-13T14:37:49.160401+0000","last_unstale":"2024-09-13T14:43:16.493570+0000","last_undegraded":"2024-09-13T14:43:16.493570+0000","last_fullsized":"2024-09-13T14:43:16.493570+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:21.737945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":130,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.437752+0000","last_change":"2024-09-13T14:37:48.172940+0000","last_active":"2024-09-13T14:42:56.437752+0000","last_peered":"2024-09-13T14:42:56.437752+0000","last_clean":"2024-09-13T14:42:56.437752+0000","last_became_active":"2024-09-13T14:37:48.172060+0000","last_became_peered":"2024-09-13T14:37:48.172060+0000","last_unstale":"2024-09-13T14:42:56.437752+0000","last_undegraded":"2024-09-13T14:42:56.437752+0000","last_fullsized":"2024-09-13T14:42:56.437752+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:07.480645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.5","version":"286'10663","reported_seq":27032,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:17.322090+0000","last_change":"2024-09-13T14:39:34.446699+0000","last_active":"2024-09-13T14:43:17.322090+0000","last_peered":"2024-09-13T14:43:17.322090+0000","last_clean":"2024-09-13T14:43:17.322090+0000","last_became_active":"2024-09-13T14:38:04.600616+0000","last_became_peered":"2024-09-13T14:38:04.600616+0000","last_unstale":"2024-09-13T14:43:17.322090+0000","last_undegraded":"2024-09-13T14:43:17.322090+0000","last_fullsized":"2024-09-13T14:43:17.322090+0000","mapping_epoch":245,"log_start":"213'4316","ondisk_log_start":"213'4316","created":205,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6347,"log_dups_size":0,"ondisk_log_size":6347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:00:44.321678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227999744,"num_objects":3479,"num_object_clones":0,"num_object_copies":10437,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10663,"num_write_kb":452544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8120,"num_bytes_recovered":521076736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":503,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.439431+0000","last_change":"2024-09-13T14:26:15.835178+0000","last_active":"2024-09-13T14:42:56.439431+0000","last_peered":"2024-09-13T14:42:56.439431+0000","last_clean":"2024-09-13T14:42:56.439431+0000","last_became_active":"2024-09-13T14:26:15.820283+0000","last_became_peered":"2024-09-13T14:26:15.820283+0000","last_unstale":"2024-09-13T14:42:56.439431+0000","last_undegraded":"2024-09-13T14:42:56.439431+0000","last_fullsized":"2024-09-13T14:42:56.439431+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:30.340038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":96,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.352500+0000","last_change":"2024-09-13T14:38:03.525682+0000","last_active":"2024-09-13T14:43:16.352500+0000","last_peered":"2024-09-13T14:43:16.352500+0000","last_clean":"2024-09-13T14:43:16.352500+0000","last_became_active":"2024-09-13T14:38:03.525406+0000","last_became_peered":"2024-09-13T14:38:03.525406+0000","last_unstale":"2024-09-13T14:43:16.352500+0000","last_undegraded":"2024-09-13T14:43:16.352500+0000","last_fullsized":"2024-09-13T14:43:16.352500+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:41:28.970300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":103,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.353659+0000","last_change":"2024-09-13T14:38:53.991738+0000","last_active":"2024-09-13T14:43:16.353659+0000","last_peered":"2024-09-13T14:43:16.353659+0000","last_clean":"2024-09-13T14:43:16.353659+0000","last_became_active":"2024-09-13T14:38:53.991417+0000","last_became_peered":"2024-09-13T14:38:53.991417+0000","last_unstale":"2024-09-13T14:43:16.353659+0000","last_undegraded":"2024-09-13T14:43:16.353659+0000","last_fullsized":"2024-09-13T14:43:16.353659+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:50:38.559838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"286'10700","reported_seq":26910,"reported_epoch":286,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-13T14:43:21.330906+0000","last_change":"2024-09-13T14:38:17.472071+0000","last_active":"2024-09-13T14:43:21.330906+0000","last_peered":"2024-09-13T14:43:21.330906+0000","last_clean":"2024-09-13T14:37:46.290754+0000","last_became_active":"2024-09-13T14:38:17.381753+0000","last_became_peered":"2024-09-13T14:38:17.381753+0000","last_unstale":"2024-09-13T14:43:21.330906+0000","last_undegraded":"2024-09-13T14:38:13.504140+0000","last_fullsized":"2024-09-13T14:43:21.330906+0000","mapping_epoch":245,"log_start":"216'4759","ondisk_log_start":"216'4759","created":205,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5941,"log_dups_size":0,"ondisk_log_size":5941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223215616,"num_objects":3406,"num_object_clones":0,"num_object_copies":10218,"num_objects_missing_on_primary":83,"num_objects_missing":83,"num_objects_degraded":3574,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10700,"num_write_kb":451392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8061,"num_bytes_recovered":528285696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"0,4,5","objects":3405},{"shards":"0,4,6","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":817,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.493315+0000","last_change":"2024-09-13T14:37:48.131799+0000","last_active":"2024-09-13T14:43:16.493315+0000","last_peered":"2024-09-13T14:43:16.493315+0000","last_clean":"2024-09-13T14:43:16.493315+0000","last_became_active":"2024-09-13T14:37:48.130837+0000","last_became_peered":"2024-09-13T14:37:48.130837+0000","last_unstale":"2024-09-13T14:43:16.493315+0000","last_undegraded":"2024-09-13T14:43:16.493315+0000","last_fullsized":"2024-09-13T14:43:16.493315+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:06:32.663770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":336,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.352626+0000","last_change":"2024-09-13T14:37:49.301531+0000","last_active":"2024-09-13T14:43:16.352626+0000","last_peered":"2024-09-13T14:43:16.352626+0000","last_clean":"2024-09-13T14:43:16.352626+0000","last_became_active":"2024-09-13T14:37:49.301249+0000","last_became_peered":"2024-09-13T14:37:49.301249+0000","last_unstale":"2024-09-13T14:43:16.352626+0000","last_undegraded":"2024-09-13T14:43:16.352626+0000","last_fullsized":"2024-09-13T14:43:16.352626+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:10.977710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":112,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.454314+0000","last_change":"2024-09-13T14:38:53.901427+0000","last_active":"2024-09-13T14:42:56.454314+0000","last_peered":"2024-09-13T14:42:56.454314+0000","last_clean":"2024-09-13T14:42:56.454314+0000","last_became_active":"2024-09-13T14:38:53.901157+0000","last_became_peered":"2024-09-13T14:38:53.901157+0000","last_unstale":"2024-09-13T14:42:56.454314+0000","last_undegraded":"2024-09-13T14:42:56.454314+0000","last_fullsized":"2024-09-13T14:42:56.454314+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:28:21.713025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"286'10645","reported_seq":33880,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:21.349726+0000","last_change":"2024-09-13T14:43:16.851222+0000","last_active":"2024-09-13T14:43:21.349726+0000","last_peered":"2024-09-13T14:43:21.349726+0000","last_clean":"2024-09-13T14:43:21.349726+0000","last_became_active":"2024-09-13T14:38:16.427510+0000","last_became_peered":"2024-09-13T14:38:16.427510+0000","last_unstale":"2024-09-13T14:43:21.349726+0000","last_undegraded":"2024-09-13T14:43:21.349726+0000","last_fullsized":"2024-09-13T14:43:21.349726+0000","mapping_epoch":245,"log_start":"213'4718","ondisk_log_start":"213'4718","created":205,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5927,"log_dups_size":0,"ondisk_log_size":5927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:27:26.281938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219742208,"num_objects":3353,"num_object_clones":0,"num_object_copies":10059,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10645,"num_write_kb":447936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11571,"num_bytes_recovered":747372544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":994,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.453656+0000","last_change":"2024-09-13T14:37:49.147498+0000","last_active":"2024-09-13T14:42:56.453656+0000","last_peered":"2024-09-13T14:42:56.453656+0000","last_clean":"2024-09-13T14:42:56.453656+0000","last_became_active":"2024-09-13T14:37:49.147028+0000","last_became_peered":"2024-09-13T14:37:49.147028+0000","last_unstale":"2024-09-13T14:42:56.453656+0000","last_undegraded":"2024-09-13T14:42:56.453656+0000","last_fullsized":"2024-09-13T14:42:56.453656+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:11:29.050178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":292,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.438039+0000","last_change":"2024-09-13T14:38:03.528133+0000","last_active":"2024-09-13T14:42:56.438039+0000","last_peered":"2024-09-13T14:42:56.438039+0000","last_clean":"2024-09-13T14:42:56.438039+0000","last_became_active":"2024-09-13T14:38:03.527829+0000","last_became_peered":"2024-09-13T14:38:03.527829+0000","last_unstale":"2024-09-13T14:42:56.438039+0000","last_undegraded":"2024-09-13T14:42:56.438039+0000","last_fullsized":"2024-09-13T14:42:56.438039+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:48.254169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":456,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.493487+0000","last_change":"2024-09-13T14:38:53.899755+0000","last_active":"2024-09-13T14:43:16.493487+0000","last_peered":"2024-09-13T14:43:16.493487+0000","last_clean":"2024-09-13T14:43:16.493487+0000","last_became_active":"2024-09-13T14:38:53.899448+0000","last_became_peered":"2024-09-13T14:38:53.899448+0000","last_unstale":"2024-09-13T14:43:16.493487+0000","last_undegraded":"2024-09-13T14:43:16.493487+0000","last_fullsized":"2024-09-13T14:43:16.493487+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:55.897035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"286'10827","reported_seq":11114,"reported_epoch":286,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T14:43:17.797658+0000","last_change":"2024-09-13T14:43:16.854832+0000","last_active":"2024-09-13T14:43:17.797658+0000","last_peered":"2024-09-13T14:43:17.797658+0000","last_clean":"2024-09-13T14:35:51.090791+0000","last_became_active":"2024-09-13T14:37:49.665025+0000","last_became_peered":"2024-09-13T14:37:49.665025+0000","last_unstale":"2024-09-13T14:43:17.797658+0000","last_undegraded":"2024-09-13T14:37:49.169101+0000","last_fullsized":"2024-09-13T14:37:49.121596+0000","mapping_epoch":240,"log_start":"213'4363","ondisk_log_start":"213'4363","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6464,"log_dups_size":0,"ondisk_log_size":6464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:02.321504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227737600,"num_objects":3475,"num_object_clones":0,"num_object_copies":10425,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4349,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10827,"num_write_kb":457664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":5701632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":-874}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":96,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.454048+0000","last_change":"2024-09-13T14:37:49.149770+0000","last_active":"2024-09-13T14:42:56.454048+0000","last_peered":"2024-09-13T14:42:56.454048+0000","last_clean":"2024-09-13T14:42:56.454048+0000","last_became_active":"2024-09-13T14:37:49.148991+0000","last_became_peered":"2024-09-13T14:37:49.148991+0000","last_unstale":"2024-09-13T14:42:56.454048+0000","last_undegraded":"2024-09-13T14:42:56.454048+0000","last_fullsized":"2024-09-13T14:42:56.454048+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:04:04.193366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":1155,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.375419+0000","last_change":"2024-09-13T14:30:58.928302+0000","last_active":"2024-09-13T14:43:16.375419+0000","last_peered":"2024-09-13T14:43:16.375419+0000","last_clean":"2024-09-13T14:43:16.375419+0000","last_became_active":"2024-09-13T14:30:58.927718+0000","last_became_peered":"2024-09-13T14:30:58.927718+0000","last_unstale":"2024-09-13T14:43:16.375419+0000","last_undegraded":"2024-09-13T14:43:16.375419+0000","last_fullsized":"2024-09-13T14:43:16.375419+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:37.323036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":103,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.453812+0000","last_change":"2024-09-13T14:38:03.534628+0000","last_active":"2024-09-13T14:42:56.453812+0000","last_peered":"2024-09-13T14:42:56.453812+0000","last_clean":"2024-09-13T14:42:56.453812+0000","last_became_active":"2024-09-13T14:38:03.534262+0000","last_became_peered":"2024-09-13T14:38:03.534262+0000","last_unstale":"2024-09-13T14:42:56.453812+0000","last_undegraded":"2024-09-13T14:42:56.453812+0000","last_fullsized":"2024-09-13T14:42:56.453812+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:31.168783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"286'10536","reported_seq":24709,"reported_epoch":286,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:43:17.128113+0000","last_change":"2024-09-13T14:38:53.944111+0000","last_active":"2024-09-13T14:43:17.128113+0000","last_peered":"2024-09-13T14:43:17.128113+0000","last_clean":"2024-09-13T14:37:47.047672+0000","last_became_active":"2024-09-13T14:37:49.664294+0000","last_became_peered":"2024-09-13T14:37:49.664294+0000","last_unstale":"2024-09-13T14:43:17.128113+0000","last_undegraded":"2024-09-13T14:38:53.940822+0000","last_fullsized":"2024-09-13T14:37:49.126627+0000","mapping_epoch":240,"log_start":"216'4800","ondisk_log_start":"216'4800","created":205,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5736,"log_dups_size":0,"ondisk_log_size":5736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:11.766692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230424576,"num_objects":3516,"num_object_clones":0,"num_object_copies":10548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4535,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10536,"num_write_kb":449664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7009,"num_bytes_recovered":459341824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3],"avail_no_missing":["5","0","3"],"object_location_counts":[{"shards":"0,3,5","objects":3516},{"shards":"3,5","objects":-944}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":115,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.454140+0000","last_change":"2024-09-13T14:38:03.534303+0000","last_active":"2024-09-13T14:42:56.454140+0000","last_peered":"2024-09-13T14:42:56.454140+0000","last_clean":"2024-09-13T14:42:56.454140+0000","last_became_active":"2024-09-13T14:38:03.533171+0000","last_became_peered":"2024-09-13T14:38:03.533171+0000","last_unstale":"2024-09-13T14:42:56.454140+0000","last_undegraded":"2024-09-13T14:42:56.454140+0000","last_fullsized":"2024-09-13T14:42:56.454140+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:00:16.553526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":807,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.440019+0000","last_change":"2024-09-13T14:38:04.596192+0000","last_active":"2024-09-13T14:42:56.440019+0000","last_peered":"2024-09-13T14:42:56.440019+0000","last_clean":"2024-09-13T14:42:56.440019+0000","last_became_active":"2024-09-13T14:38:04.593299+0000","last_became_peered":"2024-09-13T14:38:04.593299+0000","last_unstale":"2024-09-13T14:42:56.440019+0000","last_undegraded":"2024-09-13T14:42:56.440019+0000","last_fullsized":"2024-09-13T14:42:56.440019+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:34.241410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":96,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.353314+0000","last_change":"2024-09-13T14:38:03.533120+0000","last_active":"2024-09-13T14:43:16.353314+0000","last_peered":"2024-09-13T14:43:16.353314+0000","last_clean":"2024-09-13T14:43:16.353314+0000","last_became_active":"2024-09-13T14:38:03.532829+0000","last_became_peered":"2024-09-13T14:38:03.532829+0000","last_unstale":"2024-09-13T14:43:16.353314+0000","last_undegraded":"2024-09-13T14:43:16.353314+0000","last_fullsized":"2024-09-13T14:43:16.353314+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:24.514661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"286'10763","reported_seq":20023,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:17.495763+0000","last_change":"2024-09-13T14:42:04.660877+0000","last_active":"2024-09-13T14:43:17.495763+0000","last_peered":"2024-09-13T14:43:17.495763+0000","last_clean":"2024-09-13T14:43:17.495763+0000","last_became_active":"2024-09-13T14:42:04.641680+0000","last_became_peered":"2024-09-13T14:42:04.641680+0000","last_unstale":"2024-09-13T14:43:17.495763+0000","last_undegraded":"2024-09-13T14:43:17.495763+0000","last_fullsized":"2024-09-13T14:43:17.495763+0000","mapping_epoch":277,"log_start":"218'5018","ondisk_log_start":"218'5018","created":205,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5745,"log_dups_size":0,"ondisk_log_size":5745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:22:18.955428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238223360,"num_objects":3635,"num_object_clones":0,"num_object_copies":10905,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10763,"num_write_kb":460736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4533,"num_bytes_recovered":297074688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":498,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.494152+0000","last_change":"2024-09-13T14:38:53.938227+0000","last_active":"2024-09-13T14:43:16.494152+0000","last_peered":"2024-09-13T14:43:16.494152+0000","last_clean":"2024-09-13T14:43:16.494152+0000","last_became_active":"2024-09-13T14:38:53.902867+0000","last_became_peered":"2024-09-13T14:38:53.902867+0000","last_unstale":"2024-09-13T14:43:16.494152+0000","last_undegraded":"2024-09-13T14:43:16.494152+0000","last_fullsized":"2024-09-13T14:43:16.494152+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:16.831756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":586,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.438207+0000","last_change":"2024-09-13T14:37:48.147677+0000","last_active":"2024-09-13T14:42:56.438207+0000","last_peered":"2024-09-13T14:42:56.438207+0000","last_clean":"2024-09-13T14:42:56.438207+0000","last_became_active":"2024-09-13T14:37:48.147001+0000","last_became_peered":"2024-09-13T14:37:48.147001+0000","last_unstale":"2024-09-13T14:42:56.438207+0000","last_undegraded":"2024-09-13T14:42:56.438207+0000","last_fullsized":"2024-09-13T14:42:56.438207+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:35:00.653576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":480,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.493312+0000","last_change":"2024-09-13T14:38:53.900107+0000","last_active":"2024-09-13T14:43:16.493312+0000","last_peered":"2024-09-13T14:43:16.493312+0000","last_clean":"2024-09-13T14:43:16.493312+0000","last_became_active":"2024-09-13T14:38:53.899556+0000","last_became_peered":"2024-09-13T14:38:53.899556+0000","last_unstale":"2024-09-13T14:43:16.493312+0000","last_undegraded":"2024-09-13T14:43:16.493312+0000","last_fullsized":"2024-09-13T14:43:16.493312+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:15:47.918163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"286'10442","reported_seq":19428,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:17.331371+0000","last_change":"2024-09-13T14:39:54.921239+0000","last_active":"2024-09-13T14:43:17.331371+0000","last_peered":"2024-09-13T14:43:17.331371+0000","last_clean":"2024-09-13T14:43:17.331371+0000","last_became_active":"2024-09-13T14:39:54.907894+0000","last_became_peered":"2024-09-13T14:39:54.907894+0000","last_unstale":"2024-09-13T14:43:17.331371+0000","last_undegraded":"2024-09-13T14:43:17.331371+0000","last_fullsized":"2024-09-13T14:43:17.331371+0000","mapping_epoch":263,"log_start":"216'4747","ondisk_log_start":"216'4747","created":205,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5695,"log_dups_size":0,"ondisk_log_size":5695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:52:04.619238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223477760,"num_objects":3410,"num_object_clones":0,"num_object_copies":10230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10442,"num_write_kb":443264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4395,"num_bytes_recovered":283770880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":111,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.353565+0000","last_change":"2024-09-13T14:38:53.901806+0000","last_active":"2024-09-13T14:43:16.353565+0000","last_peered":"2024-09-13T14:43:16.353565+0000","last_clean":"2024-09-13T14:43:16.353565+0000","last_became_active":"2024-09-13T14:38:53.891055+0000","last_became_peered":"2024-09-13T14:38:53.891055+0000","last_unstale":"2024-09-13T14:43:16.353565+0000","last_undegraded":"2024-09-13T14:43:16.353565+0000","last_fullsized":"2024-09-13T14:43:16.353565+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:03:00.416198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":61,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.854553+0000","last_change":"2024-09-13T14:43:16.854299+0000","last_active":"2024-09-13T14:43:16.854553+0000","last_peered":"2024-09-13T14:43:16.854553+0000","last_clean":"2024-09-13T14:43:16.854553+0000","last_became_active":"2024-09-13T14:38:53.900626+0000","last_became_peered":"2024-09-13T14:38:53.900626+0000","last_unstale":"2024-09-13T14:43:16.854553+0000","last_undegraded":"2024-09-13T14:43:16.854553+0000","last_fullsized":"2024-09-13T14:43:16.854553+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:37:21.715778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":276,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.437712+0000","last_change":"2024-09-13T14:37:49.172218+0000","last_active":"2024-09-13T14:42:56.437712+0000","last_peered":"2024-09-13T14:42:56.437712+0000","last_clean":"2024-09-13T14:42:56.437712+0000","last_became_active":"2024-09-13T14:37:49.171718+0000","last_became_peered":"2024-09-13T14:37:49.171718+0000","last_unstale":"2024-09-13T14:42:56.437712+0000","last_undegraded":"2024-09-13T14:42:56.437712+0000","last_fullsized":"2024-09-13T14:42:56.437712+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:46:54.626521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":298,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.437958+0000","last_change":"2024-09-13T14:37:49.230094+0000","last_active":"2024-09-13T14:42:56.437958+0000","last_peered":"2024-09-13T14:42:56.437958+0000","last_clean":"2024-09-13T14:42:56.437958+0000","last_became_active":"2024-09-13T14:37:49.229393+0000","last_became_peered":"2024-09-13T14:37:49.229393+0000","last_unstale":"2024-09-13T14:42:56.437958+0000","last_undegraded":"2024-09-13T14:42:56.437958+0000","last_fullsized":"2024-09-13T14:42:56.437958+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:02:28.690318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":278,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.438136+0000","last_change":"2024-09-13T14:37:48.128447+0000","last_active":"2024-09-13T14:42:56.438136+0000","last_peered":"2024-09-13T14:42:56.438136+0000","last_clean":"2024-09-13T14:42:56.438136+0000","last_became_active":"2024-09-13T14:37:48.128143+0000","last_became_peered":"2024-09-13T14:37:48.128143+0000","last_unstale":"2024-09-13T14:42:56.438136+0000","last_undegraded":"2024-09-13T14:42:56.438136+0000","last_fullsized":"2024-09-13T14:42:56.438136+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:22.204039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":96,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.353100+0000","last_change":"2024-09-13T14:38:03.531683+0000","last_active":"2024-09-13T14:43:16.353100+0000","last_peered":"2024-09-13T14:43:16.353100+0000","last_clean":"2024-09-13T14:43:16.353100+0000","last_became_active":"2024-09-13T14:38:03.531476+0000","last_became_peered":"2024-09-13T14:38:03.531476+0000","last_unstale":"2024-09-13T14:43:16.353100+0000","last_undegraded":"2024-09-13T14:43:16.353100+0000","last_fullsized":"2024-09-13T14:43:16.353100+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:41.557569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":50,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.439755+0000","last_change":"2024-09-13T14:38:53.954281+0000","last_active":"2024-09-13T14:42:56.439755+0000","last_peered":"2024-09-13T14:42:56.439755+0000","last_clean":"2024-09-13T14:42:56.439755+0000","last_became_active":"2024-09-13T14:38:53.900912+0000","last_became_peered":"2024-09-13T14:38:53.900912+0000","last_unstale":"2024-09-13T14:42:56.439755+0000","last_undegraded":"2024-09-13T14:42:56.439755+0000","last_fullsized":"2024-09-13T14:42:56.439755+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:47.876275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":102,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.455014+0000","last_change":"2024-09-13T14:38:54.000782+0000","last_active":"2024-09-13T14:42:56.455014+0000","last_peered":"2024-09-13T14:42:56.455014+0000","last_clean":"2024-09-13T14:42:56.455014+0000","last_became_active":"2024-09-13T14:38:54.000322+0000","last_became_peered":"2024-09-13T14:38:54.000322+0000","last_unstale":"2024-09-13T14:42:56.455014+0000","last_undegraded":"2024-09-13T14:42:56.455014+0000","last_fullsized":"2024-09-13T14:42:56.455014+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:09.057597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":421,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.493143+0000","last_change":"2024-09-13T14:38:53.899535+0000","last_active":"2024-09-13T14:43:16.493143+0000","last_peered":"2024-09-13T14:43:16.493143+0000","last_clean":"2024-09-13T14:43:16.493143+0000","last_became_active":"2024-09-13T14:38:53.899183+0000","last_became_peered":"2024-09-13T14:38:53.899183+0000","last_unstale":"2024-09-13T14:43:16.493143+0000","last_undegraded":"2024-09-13T14:43:16.493143+0000","last_fullsized":"2024-09-13T14:43:16.493143+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:33.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":96,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.353862+0000","last_change":"2024-09-13T14:38:03.543426+0000","last_active":"2024-09-13T14:43:16.353862+0000","last_peered":"2024-09-13T14:43:16.353862+0000","last_clean":"2024-09-13T14:43:16.353862+0000","last_became_active":"2024-09-13T14:38:03.542661+0000","last_became_peered":"2024-09-13T14:38:03.542661+0000","last_unstale":"2024-09-13T14:43:16.353862+0000","last_undegraded":"2024-09-13T14:43:16.353862+0000","last_fullsized":"2024-09-13T14:43:16.353862+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:54.050127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":450,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.493222+0000","last_change":"2024-09-13T14:37:48.131321+0000","last_active":"2024-09-13T14:43:16.493222+0000","last_peered":"2024-09-13T14:43:16.493222+0000","last_clean":"2024-09-13T14:43:16.493222+0000","last_became_active":"2024-09-13T14:37:48.130998+0000","last_became_peered":"2024-09-13T14:37:48.130998+0000","last_unstale":"2024-09-13T14:43:16.493222+0000","last_undegraded":"2024-09-13T14:43:16.493222+0000","last_fullsized":"2024-09-13T14:43:16.493222+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:33:26.597825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":115,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.454306+0000","last_change":"2024-09-13T14:38:03.535454+0000","last_active":"2024-09-13T14:42:56.454306+0000","last_peered":"2024-09-13T14:42:56.454306+0000","last_clean":"2024-09-13T14:42:56.454306+0000","last_became_active":"2024-09-13T14:38:03.535131+0000","last_became_peered":"2024-09-13T14:38:03.535131+0000","last_unstale":"2024-09-13T14:42:56.454306+0000","last_undegraded":"2024-09-13T14:42:56.454306+0000","last_fullsized":"2024-09-13T14:42:56.454306+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:24.477601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":475,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.493400+0000","last_change":"2024-09-13T14:38:03.524939+0000","last_active":"2024-09-13T14:43:16.493400+0000","last_peered":"2024-09-13T14:43:16.493400+0000","last_clean":"2024-09-13T14:43:16.493400+0000","last_became_active":"2024-09-13T14:38:03.524406+0000","last_became_peered":"2024-09-13T14:38:03.524406+0000","last_unstale":"2024-09-13T14:43:16.493400+0000","last_undegraded":"2024-09-13T14:43:16.493400+0000","last_fullsized":"2024-09-13T14:43:16.493400+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:19:45.257424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":103,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.353478+0000","last_change":"2024-09-13T14:38:53.892948+0000","last_active":"2024-09-13T14:43:16.353478+0000","last_peered":"2024-09-13T14:43:16.353478+0000","last_clean":"2024-09-13T14:43:16.353478+0000","last_became_active":"2024-09-13T14:38:53.892665+0000","last_became_peered":"2024-09-13T14:38:53.892665+0000","last_unstale":"2024-09-13T14:43:16.353478+0000","last_undegraded":"2024-09-13T14:43:16.353478+0000","last_fullsized":"2024-09-13T14:43:16.353478+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:08.292359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":424,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.493636+0000","last_change":"2024-09-13T14:37:48.139348+0000","last_active":"2024-09-13T14:43:16.493636+0000","last_peered":"2024-09-13T14:43:16.493636+0000","last_clean":"2024-09-13T14:43:16.493636+0000","last_became_active":"2024-09-13T14:37:48.139115+0000","last_became_peered":"2024-09-13T14:37:48.139115+0000","last_unstale":"2024-09-13T14:43:16.493636+0000","last_undegraded":"2024-09-13T14:43:16.493636+0000","last_fullsized":"2024-09-13T14:43:16.493636+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:32:20.413346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":464,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.493760+0000","last_change":"2024-09-13T14:37:48.172088+0000","last_active":"2024-09-13T14:43:16.493760+0000","last_peered":"2024-09-13T14:43:16.493760+0000","last_clean":"2024-09-13T14:43:16.493760+0000","last_became_active":"2024-09-13T14:37:48.171779+0000","last_became_peered":"2024-09-13T14:37:48.171779+0000","last_unstale":"2024-09-13T14:43:16.493760+0000","last_undegraded":"2024-09-13T14:43:16.493760+0000","last_fullsized":"2024-09-13T14:43:16.493760+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:02.852712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":342,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.437691+0000","last_change":"2024-09-13T14:38:53.890236+0000","last_active":"2024-09-13T14:42:56.437691+0000","last_peered":"2024-09-13T14:42:56.437691+0000","last_clean":"2024-09-13T14:42:56.437691+0000","last_became_active":"2024-09-13T14:38:53.889582+0000","last_became_peered":"2024-09-13T14:38:53.889582+0000","last_unstale":"2024-09-13T14:42:56.437691+0000","last_undegraded":"2024-09-13T14:42:56.437691+0000","last_fullsized":"2024-09-13T14:42:56.437691+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:14:29.334774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":450,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.493729+0000","last_change":"2024-09-13T14:38:03.534097+0000","last_active":"2024-09-13T14:43:16.493729+0000","last_peered":"2024-09-13T14:43:16.493729+0000","last_clean":"2024-09-13T14:43:16.493729+0000","last_became_active":"2024-09-13T14:38:03.533794+0000","last_became_peered":"2024-09-13T14:38:03.533794+0000","last_unstale":"2024-09-13T14:43:16.493729+0000","last_undegraded":"2024-09-13T14:43:16.493729+0000","last_fullsized":"2024-09-13T14:43:16.493729+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:23.272098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":96,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.353376+0000","last_change":"2024-09-13T14:38:03.527021+0000","last_active":"2024-09-13T14:43:16.353376+0000","last_peered":"2024-09-13T14:43:16.353376+0000","last_clean":"2024-09-13T14:43:16.353376+0000","last_became_active":"2024-09-13T14:38:03.526698+0000","last_became_peered":"2024-09-13T14:38:03.526698+0000","last_unstale":"2024-09-13T14:43:16.353376+0000","last_undegraded":"2024-09-13T14:43:16.353376+0000","last_fullsized":"2024-09-13T14:43:16.353376+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:01:35.734086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":286,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.437900+0000","last_change":"2024-09-13T14:27:12.834931+0000","last_active":"2024-09-13T14:42:56.437900+0000","last_peered":"2024-09-13T14:42:56.437900+0000","last_clean":"2024-09-13T14:42:56.437900+0000","last_became_active":"2024-09-13T14:27:12.834572+0000","last_became_peered":"2024-09-13T14:27:12.834572+0000","last_unstale":"2024-09-13T14:42:56.437900+0000","last_undegraded":"2024-09-13T14:42:56.437900+0000","last_fullsized":"2024-09-13T14:42:56.437900+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:54:21.155689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":99,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.454062+0000","last_change":"2024-09-13T14:37:49.176796+0000","last_active":"2024-09-13T14:42:56.454062+0000","last_peered":"2024-09-13T14:42:56.454062+0000","last_clean":"2024-09-13T14:42:56.454062+0000","last_became_active":"2024-09-13T14:37:49.176246+0000","last_became_peered":"2024-09-13T14:37:49.176246+0000","last_unstale":"2024-09-13T14:42:56.454062+0000","last_undegraded":"2024-09-13T14:42:56.454062+0000","last_fullsized":"2024-09-13T14:42:56.454062+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:29:24.936718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":297,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.437594+0000","last_change":"2024-09-13T14:38:03.526333+0000","last_active":"2024-09-13T14:42:56.437594+0000","last_peered":"2024-09-13T14:42:56.437594+0000","last_clean":"2024-09-13T14:42:56.437594+0000","last_became_active":"2024-09-13T14:38:03.526001+0000","last_became_peered":"2024-09-13T14:38:03.526001+0000","last_unstale":"2024-09-13T14:42:56.437594+0000","last_undegraded":"2024-09-13T14:42:56.437594+0000","last_fullsized":"2024-09-13T14:42:56.437594+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:25:25.181271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":99,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.454140+0000","last_change":"2024-09-13T14:37:49.149582+0000","last_active":"2024-09-13T14:42:56.454140+0000","last_peered":"2024-09-13T14:42:56.454140+0000","last_clean":"2024-09-13T14:42:56.454140+0000","last_became_active":"2024-09-13T14:37:49.148525+0000","last_became_peered":"2024-09-13T14:37:49.148525+0000","last_unstale":"2024-09-13T14:42:56.454140+0000","last_undegraded":"2024-09-13T14:42:56.454140+0000","last_fullsized":"2024-09-13T14:42:56.454140+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:48.655665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":496,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.493493+0000","last_change":"2024-09-13T14:38:03.535255+0000","last_active":"2024-09-13T14:43:16.493493+0000","last_peered":"2024-09-13T14:43:16.493493+0000","last_clean":"2024-09-13T14:43:16.493493+0000","last_became_active":"2024-09-13T14:38:03.533917+0000","last_became_peered":"2024-09-13T14:38:03.533917+0000","last_unstale":"2024-09-13T14:43:16.493493+0000","last_undegraded":"2024-09-13T14:43:16.493493+0000","last_fullsized":"2024-09-13T14:43:16.493493+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:33.559788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":460,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.495256+0000","last_change":"2024-09-13T14:37:48.146784+0000","last_active":"2024-09-13T14:43:16.495256+0000","last_peered":"2024-09-13T14:43:16.495256+0000","last_clean":"2024-09-13T14:43:16.495256+0000","last_became_active":"2024-09-13T14:37:48.146491+0000","last_became_peered":"2024-09-13T14:37:48.146491+0000","last_unstale":"2024-09-13T14:43:16.495256+0000","last_undegraded":"2024-09-13T14:43:16.495256+0000","last_fullsized":"2024-09-13T14:43:16.495256+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:30:54.772513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":278,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:57.799812+0000","last_change":"2024-09-13T14:37:48.127607+0000","last_active":"2024-09-13T14:42:57.799812+0000","last_peered":"2024-09-13T14:42:57.799812+0000","last_clean":"2024-09-13T14:42:57.799812+0000","last_became_active":"2024-09-13T14:37:48.127316+0000","last_became_peered":"2024-09-13T14:37:48.127316+0000","last_unstale":"2024-09-13T14:42:57.799812+0000","last_undegraded":"2024-09-13T14:42:57.799812+0000","last_fullsized":"2024-09-13T14:42:57.799812+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:53:37.678861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":102,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.353755+0000","last_change":"2024-09-13T14:38:03.563200+0000","last_active":"2024-09-13T14:43:16.353755+0000","last_peered":"2024-09-13T14:43:16.353755+0000","last_clean":"2024-09-13T14:43:16.353755+0000","last_became_active":"2024-09-13T14:38:03.534258+0000","last_became_peered":"2024-09-13T14:38:03.534258+0000","last_unstale":"2024-09-13T14:43:16.353755+0000","last_undegraded":"2024-09-13T14:43:16.353755+0000","last_fullsized":"2024-09-13T14:43:16.353755+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:49:07.814419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":3608674328,"num_objects":55065,"num_object_clones":0,"num_object_copies":165195,"num_objects_missing_on_primary":83,"num_objects_missing":83,"num_objects_degraded":13861,"num_objects_misplaced":1987,"num_objects_unfound":0,"num_objects_dirty":55065,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170561,"num_write_kb":7219969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78346,"num_bytes_recovered":5073666072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12999536640,"data_stored":12999524424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94784,"ondisk_log_size":94784,"up":48,"acting":46,"num_store_stats":6},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1538,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":17390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25132,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16793600,"data_stored":16777292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":23,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2969600,"data_stored":2951840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234393,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2547348,"kb_used_data":2078984,"kb_used_omap":1,"kb_used_meta":468350,"kb_avail":91197804,"statfs":{"total":95995035648,"available":93386551296,"internally_reserved":0,"allocated":2128879616,"data_stored":2127343418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":479590859},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":239,"seq":1026497183812,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1854804,"kb_used_data":1563596,"kb_used_omap":8,"kb_used_meta":291191,"kb_avail":91890348,"statfs":{"total":95995035648,"available":94095716352,"internally_reserved":0,"allocated":1601122304,"data_stored":1599582581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8453,"internal_metadata":298180347},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757658,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2139588,"kb_used_data":1472696,"kb_used_omap":1,"kb_used_meta":666878,"kb_avail":91605564,"statfs":{"total":95995035648,"available":93804097536,"internally_reserved":0,"allocated":1508040704,"data_stored":1506516067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":682883526},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019302,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2511632,"kb_used_data":1908740,"kb_used_omap":1,"kb_used_meta":602878,"kb_avail":91233520,"statfs":{"total":95995035648,"available":93423124480,"internally_reserved":0,"allocated":1954549760,"data_stored":1953017646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":617347526},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435485,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3249600,"kb_used_data":2676468,"kb_used_omap":5,"kb_used_meta":573114,"kb_avail":90495552,"statfs":{"total":95995035648,"available":92667445248,"internally_reserved":0,"allocated":2740703232,"data_stored":2739182455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":586868862},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":446676599014,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":425201762540,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926064,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3728908,"kb_used_data":3051652,"kb_used_omap":7,"kb_used_meta":677240,"kb_avail":90016244,"statfs":{"total":95995035648,"available":92176633856,"internally_reserved":0,"allocated":3124891648,"data_stored":3123355654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":693494056},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6860,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3114143744,"data_stored":3114139672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2733772800,"data_stored":2733768728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1946157056,"data_stored":1946157056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1499660288,"data_stored":1499660288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1589841920,"data_stored":1589837848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2115960832,"data_stored":2115960832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:43:24.284 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:24.312 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:24.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:24.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:24.414 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:24.437 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:24.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:24.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:24.539 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:24.560 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:24.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:24.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:24.662 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:24.686 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:24.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:24.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:24.788 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:24.811 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:24.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:24.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:24.912 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:24.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:24 smithi102 ceph-mon[110183]: from='client.35958 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:43:24.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:24 smithi102 ceph-mon[110183]: pgmap v978: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.3 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 265 op/s; 13455/162423 objects degraded (8.284%); 1987/162423 objects misplaced (1.223%); 12 MiB/s, 199 objects/s recovering 2024-09-13T14:43:24.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:24 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 13455/162423 objects degraded (8.284%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:43:24.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:24 smithi102 ceph-mon[115269]: from='client.35958 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:43:24.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:24 smithi102 ceph-mon[115269]: pgmap v978: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.3 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 265 op/s; 13455/162423 objects degraded (8.284%); 1987/162423 objects misplaced (1.223%); 12 MiB/s, 199 objects/s recovering 2024-09-13T14:43:24.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:24 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 13455/162423 objects degraded (8.284%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:43:24.936 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:24.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:24.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:24.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:24 smithi195 ceph-mon[104262]: from='client.35958 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:43:24.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:24 smithi195 ceph-mon[104262]: pgmap v978: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.3 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 265 op/s; 13455/162423 objects degraded (8.284%); 1987/162423 objects misplaced (1.223%); 12 MiB/s, 199 objects/s recovering 2024-09-13T14:43:24.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:24 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 13455/162423 objects degraded (8.284%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:43:25.038 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:25.061 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:25.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:25.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:25.163 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:25.187 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:25.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:25.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:25.289 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:25.314 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:25.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:25.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:25.416 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:25.438 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:25.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:25.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:25.540 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:25.563 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:25.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:25.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:25.664 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:25.692 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:25.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:25.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:25.794 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:25.817 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:25.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:25.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:25.862 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:25 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:43:25.863 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:25 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:43:25.919 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:25.941 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:25.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:25.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:25.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:25 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:43:26.042 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:26.067 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:26.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:26.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:26.169 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:26.192 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:26.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:26.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:26.294 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:26.321 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:26.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:26.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:26.423 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:26.450 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:26.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:26.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:26.552 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:26.575 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:26.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:26.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:26.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:43:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:43:26] "GET /metrics HTTP/1.1" 200 36550 "" "Prometheus/2.51.0" 2024-09-13T14:43:26.676 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:26.698 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:26.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:26.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:26.800 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:26.823 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:26.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:26.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:26.925 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:26.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:26 smithi102 ceph-mon[110183]: pgmap v979: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.3 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 265 op/s; 13455/162423 objects degraded (8.284%); 1987/162423 objects misplaced (1.223%); 12 MiB/s, 199 objects/s recovering 2024-09-13T14:43:26.931 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:26 smithi102 ceph-mon[115269]: pgmap v979: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.3 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 265 op/s; 13455/162423 objects degraded (8.284%); 1987/162423 objects misplaced (1.223%); 12 MiB/s, 199 objects/s recovering 2024-09-13T14:43:26.948 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:26.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:26.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:26.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:26 smithi195 ceph-mon[104262]: pgmap v979: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.3 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 265 op/s; 13455/162423 objects degraded (8.284%); 1987/162423 objects misplaced (1.223%); 12 MiB/s, 199 objects/s recovering 2024-09-13T14:43:27.050 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:27.062 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:43:27.063 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:43:27.074 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:27.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:27.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:27.084 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:43:27.084 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:43:27.102 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:43:27.103 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:43:27.134 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:43:27.135 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:43:27.166 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:43:27.166 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:43:27.175 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:27.188 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:43:27.188 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:43:27.200 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:27.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:27.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:27.207 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:43:27.207 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:43:27.225 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:43:27.226 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:43:27.285 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:43:27.302 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:27.361 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:27.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:27.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:27.463 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:27.486 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:27.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:27.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:27.588 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:27.615 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:27.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:27.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:27.717 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:27.739 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:27.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:27.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:27.810 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:43:27.810 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [], 2024-09-13T14:43:27.810 INFO:teuthology.orchestra.run.smithi102.stdout: "complaint_time": 30, 2024-09-13T14:43:27.810 INFO:teuthology.orchestra.run.smithi102.stdout: "num_blocked_ops": 0 2024-09-13T14:43:27.810 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:43:27.841 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:27.863 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:27.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:27.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:27.965 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:27.990 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:27.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:27.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:27.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:27 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2706829489' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:43:28.091 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:28.114 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:28.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:28.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:28.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:27 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2706829489' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:43:28.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:27 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2706829489' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:43:28.216 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:28.238 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:28.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:28.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:28.339 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:28.364 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:28.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:28.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:28.466 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:28.488 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:28.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:28.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:28.578 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T14:43:28.590 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:28.612 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:28.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:28.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:28.714 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:28.740 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:28.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:28.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:28.841 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:28.862 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:28.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:28.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:28.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:28 smithi102 ceph-mon[110183]: pgmap v980: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 338 op/s; 13297/158976 objects degraded (8.364%); 1987/158976 objects misplaced (1.250%); 12 MiB/s, 191 objects/s recovering 2024-09-13T14:43:28.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:28 smithi102 ceph-mon[115269]: pgmap v980: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 338 op/s; 13297/158976 objects degraded (8.364%); 1987/158976 objects misplaced (1.250%); 12 MiB/s, 191 objects/s recovering 2024-09-13T14:43:28.963 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:28.991 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:28.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:28.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:28.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:28 smithi195 ceph-mon[104262]: pgmap v980: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 338 op/s; 13297/158976 objects degraded (8.364%); 1987/158976 objects misplaced (1.250%); 12 MiB/s, 191 objects/s recovering 2024-09-13T14:43:29.092 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:29.093 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:29.110 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:29.115 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:29.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:29.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:29.217 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:29.239 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:29.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:29.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:29.340 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:29.366 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:29.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:29.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:29.468 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:29.489 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:29.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:29.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:29.591 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:29.619 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:29.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:29.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:29.721 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:29.734 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:29.748 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:29.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:29.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:29.860 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:29.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:29.883 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:29.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:29.945 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:29 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 13297/158976 objects degraded (8.364%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:43:29.946 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:29 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 13297/158976 objects degraded (8.364%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:43:29.984 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:29.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:29 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 13297/158976 objects degraded (8.364%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:43:30.006 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:30.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:30.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:30.108 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:30.134 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:30.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:30.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:30.236 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:30.258 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:30.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:30.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:30.360 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:30.382 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:30.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:30.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:30.484 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:30.506 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:30.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:30.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:30.608 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:30.632 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:30.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:30.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:30.733 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:30.755 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:30.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:30.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:30.857 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:30.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:30.881 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:30.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:30.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:30 smithi102 ceph-mon[110183]: pgmap v981: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.2 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 231 op/s; 12809/158739 objects degraded (8.069%); 1987/158739 objects misplaced (1.252%); 7.4 MiB/s, 123 objects/s recovering 2024-09-13T14:43:30.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:30 smithi102 ceph-mon[115269]: pgmap v981: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.2 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 231 op/s; 12809/158739 objects degraded (8.069%); 1987/158739 objects misplaced (1.252%); 7.4 MiB/s, 123 objects/s recovering 2024-09-13T14:43:30.982 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:30.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:30 smithi195 ceph-mon[104262]: pgmap v981: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.2 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 231 op/s; 12809/158739 objects degraded (8.069%); 1987/158739 objects misplaced (1.252%); 7.4 MiB/s, 123 objects/s recovering 2024-09-13T14:43:31.006 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:31.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:31.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:31.061 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:43:31.108 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:31.129 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:31.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:31.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:31.231 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:31.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:31.254 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:31.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:31.355 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:31.378 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:31.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:31.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:31.480 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:31.501 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:31.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:31.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:31.603 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:31.624 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:31.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:31.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:31.726 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:31.754 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:31.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:31.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:31.855 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:31.879 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:31.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:31.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:31.981 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:32.003 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:32.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:32.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:32.105 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:32.129 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:32.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:32.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:32.232 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:32.253 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:32.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:32.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:32.355 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:32.381 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:32.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:32.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:32.483 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:32.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:32.513 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:32.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:32.614 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:32.640 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:32.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:32.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:32.742 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:32.765 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:32.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:32.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:32.867 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:32.893 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:32.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:32.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:32.907 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:32 smithi102 ceph-mon[110183]: pgmap v982: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.2 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 256 op/s; 12677/157662 objects degraded (8.041%); 1987/157662 objects misplaced (1.260%); 7.6 MiB/s, 127 objects/s recovering 2024-09-13T14:43:32.908 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:32 smithi102 ceph-mon[115269]: pgmap v982: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.2 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 256 op/s; 12677/157662 objects degraded (8.041%); 1987/157662 objects misplaced (1.260%); 7.6 MiB/s, 127 objects/s recovering 2024-09-13T14:43:32.994 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:32.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:32 smithi195 ceph-mon[104262]: pgmap v982: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.2 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 256 op/s; 12677/157662 objects degraded (8.041%); 1987/157662 objects misplaced (1.260%); 7.6 MiB/s, 127 objects/s recovering 2024-09-13T14:43:33.021 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:33.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:33.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:33.123 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:33.153 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:33.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:33.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:33.255 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:33.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:33.283 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:33.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:33.384 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:33.407 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:33.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:33.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:33.509 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:33.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:33.537 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:33.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:33.638 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:33.662 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:33.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:33.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:33.764 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:33.786 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:33.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:33.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:33.888 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:33.913 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:33.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:33.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:34.015 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:34.040 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:34.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:34.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:34.141 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:34.165 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:34.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:34.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:34.266 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:34.289 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:34.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:34.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:34.391 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:34.413 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:34.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:34.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:34.515 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:34.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:34.540 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:34.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:34.613 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:43:34.614 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:43:34.641 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:34.666 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:34.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:34.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:34.707 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:34.768 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:34.790 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:34.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:34.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:34.892 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:34.906 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:34 smithi102 ceph-mon[115269]: pgmap v983: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.2 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 263 op/s; 12122/155796 objects degraded (7.781%); 1987/155796 objects misplaced (1.275%); 9.1 MiB/s, 145 objects/s recovering 2024-09-13T14:43:34.906 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:34 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 12122/155796 objects degraded (7.781%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:43:34.906 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:34 smithi102 ceph-mon[110183]: pgmap v983: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.2 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 263 op/s; 12122/155796 objects degraded (7.781%); 1987/155796 objects misplaced (1.275%); 9.1 MiB/s, 145 objects/s recovering 2024-09-13T14:43:34.907 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:34 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 12122/155796 objects degraded (7.781%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:43:34.920 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:34.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:34.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:34.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:34 smithi195 ceph-mon[104262]: pgmap v983: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.2 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 263 op/s; 12122/155796 objects degraded (7.781%); 1987/155796 objects misplaced (1.275%); 9.1 MiB/s, 145 objects/s recovering 2024-09-13T14:43:34.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:34 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 12122/155796 objects degraded (7.781%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:43:35.022 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:35.044 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:35.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:35.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:35.146 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:35.169 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:35.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:35.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:35.271 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:35.294 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:35.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:35.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:35.396 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:35.421 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:35.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:35.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:35.523 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:35.548 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:35.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:35.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:35.650 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:35.671 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:35.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:35.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:35.773 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:35.796 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:35.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:35.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:35.898 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:35.926 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:35.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:35.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:35.953 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:35 smithi102 ceph-mon[110183]: from='client.45896 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:43:35.953 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:35 smithi102 ceph-mon[115269]: from='client.45896 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:43:35.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:35 smithi195 ceph-mon[104262]: from='client.45896 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:43:36.027 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:36.051 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:36.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:36.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:36.153 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:36.178 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:36.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:36.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:36.281 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:36.308 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:36.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:36.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:36.410 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:36.433 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:36.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:36.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:36.535 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:36.559 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:36.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:36.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:36.661 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:36.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:43:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:43:36] "GET /metrics HTTP/1.1" 200 36550 "" "Prometheus/2.51.0" 2024-09-13T14:43:36.689 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:36.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:36.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:36.709 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is unset 2024-09-13T14:43:36.790 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:36.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:36.813 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:36.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:36.914 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:36.936 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:36.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:36.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:36.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:36 smithi195 ceph-mon[104262]: pgmap v984: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.2 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 184 op/s; 12122/155796 objects degraded (7.781%); 1987/155796 objects misplaced (1.275%); 6.9 MiB/s, 111 objects/s recovering 2024-09-13T14:43:36.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:36 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/4215048457' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:43:37.038 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:37.062 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:37.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:37.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:37.154 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:36 smithi102 ceph-mon[110183]: pgmap v984: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.2 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 184 op/s; 12122/155796 objects degraded (7.781%); 1987/155796 objects misplaced (1.275%); 6.9 MiB/s, 111 objects/s recovering 2024-09-13T14:43:37.154 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:36 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/4215048457' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:43:37.155 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:36 smithi102 ceph-mon[115269]: pgmap v984: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.2 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 184 op/s; 12122/155796 objects degraded (7.781%); 1987/155796 objects misplaced (1.275%); 6.9 MiB/s, 111 objects/s recovering 2024-09-13T14:43:37.155 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:36 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/4215048457' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:43:37.156 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":983,"stamp":"2024-09-13T14:43:33.529521+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3406305277,"num_objects":51932,"num_object_clones":0,"num_object_copies":155796,"num_objects_missing_on_primary":83,"num_objects_missing":83,"num_objects_degraded":12122,"num_objects_misplaced":1987,"num_objects_unfound":0,"num_objects_dirty":51932,"num_whiteouts":0,"num_read":6377,"num_read_kb":13834,"num_write":174845,"num_write_kb":7235010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80208,"num_bytes_recovered":5194189965,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95727,"ondisk_log_size":95727,"up":267,"acting":264,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":302,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":15420856,"kb_used_data":12144968,"kb_used_omap":26,"kb_used_meta":3275813,"kb_avail":547050056,"statfs":{"total":575970213888,"available":560179257344,"internally_reserved":0,"allocated":12436447232,"data_stored":12427257789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27144,"internal_metadata":3354433016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2,3,1,0,6,7],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-207159296,"num_objects":-3161,"num_object_clones":0,"num_object_copies":-9483,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1748,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3161,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1748,"num_bytes_recovered":114556928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002912"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":465,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.493577+0000","last_change":"2024-09-13T14:36:13.859927+0000","last_active":"2024-09-13T14:43:16.493577+0000","last_peered":"2024-09-13T14:43:16.493577+0000","last_clean":"2024-09-13T14:43:16.493577+0000","last_became_active":"2024-09-13T14:36:13.859505+0000","last_became_peered":"2024-09-13T14:36:13.859505+0000","last_unstale":"2024-09-13T14:43:16.493577+0000","last_undegraded":"2024-09-13T14:43:16.493577+0000","last_fullsized":"2024-09-13T14:43:16.493577+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:12:20.989627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":288,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.439721+0000","last_change":"2024-09-13T14:37:48.130968+0000","last_active":"2024-09-13T14:42:56.439721+0000","last_peered":"2024-09-13T14:42:56.439721+0000","last_clean":"2024-09-13T14:42:56.439721+0000","last_became_active":"2024-09-13T14:37:48.130603+0000","last_became_peered":"2024-09-13T14:37:48.130603+0000","last_unstale":"2024-09-13T14:42:56.439721+0000","last_undegraded":"2024-09-13T14:42:56.439721+0000","last_fullsized":"2024-09-13T14:42:56.439721+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:42.626386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":85,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.300864+0000","last_change":"2024-09-13T14:36:13.862181+0000","last_active":"2024-09-13T14:43:16.300864+0000","last_peered":"2024-09-13T14:43:16.300864+0000","last_clean":"2024-09-13T14:43:16.300864+0000","last_became_active":"2024-09-13T14:36:13.861873+0000","last_became_peered":"2024-09-13T14:36:13.861873+0000","last_unstale":"2024-09-13T14:43:16.300864+0000","last_undegraded":"2024-09-13T14:43:16.300864+0000","last_fullsized":"2024-09-13T14:43:16.300864+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:14.085700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":103,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.353148+0000","last_change":"2024-09-13T14:38:53.896672+0000","last_active":"2024-09-13T14:43:16.353148+0000","last_peered":"2024-09-13T14:43:16.353148+0000","last_clean":"2024-09-13T14:43:16.353148+0000","last_became_active":"2024-09-13T14:38:53.896346+0000","last_became_peered":"2024-09-13T14:38:53.896346+0000","last_unstale":"2024-09-13T14:43:16.353148+0000","last_undegraded":"2024-09-13T14:43:16.353148+0000","last_fullsized":"2024-09-13T14:43:16.353148+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:57.789686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":103,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.353656+0000","last_change":"2024-09-13T14:38:53.893021+0000","last_active":"2024-09-13T14:43:16.353656+0000","last_peered":"2024-09-13T14:43:16.353656+0000","last_clean":"2024-09-13T14:43:16.353656+0000","last_became_active":"2024-09-13T14:38:53.892668+0000","last_became_peered":"2024-09-13T14:38:53.892668+0000","last_unstale":"2024-09-13T14:43:16.353656+0000","last_undegraded":"2024-09-13T14:43:16.353656+0000","last_fullsized":"2024-09-13T14:43:16.353656+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:57.180738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":256,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.439552+0000","last_change":"2024-09-13T14:37:48.134642+0000","last_active":"2024-09-13T14:42:56.439552+0000","last_peered":"2024-09-13T14:42:56.439552+0000","last_clean":"2024-09-13T14:42:56.439552+0000","last_became_active":"2024-09-13T14:37:48.134402+0000","last_became_peered":"2024-09-13T14:37:48.134402+0000","last_unstale":"2024-09-13T14:42:56.439552+0000","last_undegraded":"2024-09-13T14:42:56.439552+0000","last_fullsized":"2024-09-13T14:42:56.439552+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:51.313663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":484,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.494009+0000","last_change":"2024-09-13T14:38:53.903022+0000","last_active":"2024-09-13T14:43:16.494009+0000","last_peered":"2024-09-13T14:43:16.494009+0000","last_clean":"2024-09-13T14:43:16.494009+0000","last_became_active":"2024-09-13T14:38:53.902448+0000","last_became_peered":"2024-09-13T14:38:53.902448+0000","last_unstale":"2024-09-13T14:43:16.494009+0000","last_undegraded":"2024-09-13T14:43:16.494009+0000","last_fullsized":"2024-09-13T14:43:16.494009+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:24.690401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":106,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.352448+0000","last_change":"2024-09-13T14:38:03.554500+0000","last_active":"2024-09-13T14:43:16.352448+0000","last_peered":"2024-09-13T14:43:16.352448+0000","last_clean":"2024-09-13T14:43:16.352448+0000","last_became_active":"2024-09-13T14:38:03.532910+0000","last_became_peered":"2024-09-13T14:38:03.532910+0000","last_unstale":"2024-09-13T14:43:16.352448+0000","last_undegraded":"2024-09-13T14:43:16.352448+0000","last_fullsized":"2024-09-13T14:43:16.352448+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:27:50.843498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"286'10755","reported_seq":21652,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:31.947101+0000","last_change":"2024-09-13T14:41:53.220083+0000","last_active":"2024-09-13T14:43:31.947101+0000","last_peered":"2024-09-13T14:43:31.947101+0000","last_clean":"2024-09-13T14:43:31.947101+0000","last_became_active":"2024-09-13T14:41:53.200084+0000","last_became_peered":"2024-09-13T14:41:53.200084+0000","last_unstale":"2024-09-13T14:43:31.947101+0000","last_undegraded":"2024-09-13T14:43:31.947101+0000","last_fullsized":"2024-09-13T14:43:31.947101+0000","mapping_epoch":275,"log_start":"219'5000","ondisk_log_start":"219'5000","created":205,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5755,"log_dups_size":0,"ondisk_log_size":5755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:08.897524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204800000,"num_objects":3125,"num_object_clones":0,"num_object_copies":9375,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10755,"num_write_kb":444160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9152,"num_bytes_recovered":594608128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":49,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.303726+0000","last_change":"2024-09-13T14:38:53.940238+0000","last_active":"2024-09-13T14:43:16.303726+0000","last_peered":"2024-09-13T14:43:16.303726+0000","last_clean":"2024-09-13T14:43:16.303726+0000","last_became_active":"2024-09-13T14:38:53.892825+0000","last_became_peered":"2024-09-13T14:38:53.892825+0000","last_unstale":"2024-09-13T14:43:16.303726+0000","last_undegraded":"2024-09-13T14:43:16.303726+0000","last_fullsized":"2024-09-13T14:43:16.303726+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:46.666581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":105,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.453169+0000","last_change":"2024-09-13T14:38:53.898907+0000","last_active":"2024-09-13T14:42:56.453169+0000","last_peered":"2024-09-13T14:42:56.453169+0000","last_clean":"2024-09-13T14:42:56.453169+0000","last_became_active":"2024-09-13T14:38:53.898572+0000","last_became_peered":"2024-09-13T14:38:53.898572+0000","last_unstale":"2024-09-13T14:42:56.453169+0000","last_undegraded":"2024-09-13T14:42:56.453169+0000","last_fullsized":"2024-09-13T14:42:56.453169+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:29.222150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"286'11011","reported_seq":25193,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:32.521344+0000","last_change":"2024-09-13T14:38:57.930032+0000","last_active":"2024-09-13T14:43:32.521344+0000","last_peered":"2024-09-13T14:43:32.521344+0000","last_clean":"2024-09-13T14:43:32.521344+0000","last_became_active":"2024-09-13T14:38:57.921458+0000","last_became_peered":"2024-09-13T14:38:57.921458+0000","last_unstale":"2024-09-13T14:43:32.521344+0000","last_undegraded":"2024-09-13T14:43:32.521344+0000","last_fullsized":"2024-09-13T14:43:32.521344+0000","mapping_epoch":256,"log_start":"219'5339","ondisk_log_start":"219'5339","created":205,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5672,"log_dups_size":0,"ondisk_log_size":5672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:27:12.294038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211746816,"num_objects":3231,"num_object_clones":0,"num_object_copies":9693,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11011,"num_write_kb":455744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6990,"num_bytes_recovered":458096640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":49,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.303625+0000","last_change":"2024-09-13T14:38:53.904885+0000","last_active":"2024-09-13T14:43:16.303625+0000","last_peered":"2024-09-13T14:43:16.303625+0000","last_clean":"2024-09-13T14:43:16.303625+0000","last_became_active":"2024-09-13T14:38:53.891726+0000","last_became_peered":"2024-09-13T14:38:53.891726+0000","last_unstale":"2024-09-13T14:43:16.303625+0000","last_undegraded":"2024-09-13T14:43:16.303625+0000","last_fullsized":"2024-09-13T14:43:16.303625+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:57.079205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":294,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.437588+0000","last_change":"2024-09-13T14:37:48.160504+0000","last_active":"2024-09-13T14:42:56.437588+0000","last_peered":"2024-09-13T14:42:56.437588+0000","last_clean":"2024-09-13T14:42:56.437588+0000","last_became_active":"2024-09-13T14:37:48.160181+0000","last_became_peered":"2024-09-13T14:37:48.160181+0000","last_unstale":"2024-09-13T14:42:56.437588+0000","last_undegraded":"2024-09-13T14:42:56.437588+0000","last_fullsized":"2024-09-13T14:42:56.437588+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:37.087008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"286'10881","reported_seq":12685,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:30.891860+0000","last_change":"2024-09-13T14:38:04.531914+0000","last_active":"2024-09-13T14:43:30.891860+0000","last_peered":"2024-09-13T14:43:30.891860+0000","last_clean":"2024-09-13T14:43:30.891860+0000","last_became_active":"2024-09-13T14:38:04.523448+0000","last_became_peered":"2024-09-13T14:38:04.523448+0000","last_unstale":"2024-09-13T14:43:30.891860+0000","last_undegraded":"2024-09-13T14:43:30.891860+0000","last_fullsized":"2024-09-13T14:43:30.891860+0000","mapping_epoch":245,"log_start":"217'4900","ondisk_log_start":"217'4900","created":205,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5981,"log_dups_size":0,"ondisk_log_size":5981,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:06.536896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215810048,"num_objects":3293,"num_object_clones":0,"num_object_copies":9879,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10881,"num_write_kb":453568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":811,"num_bytes_recovered":53149696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":286,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:43:02.427494+0000","last_change":"2024-09-13T14:37:49.611774+0000","last_active":"2024-09-13T14:43:02.427494+0000","last_peered":"2024-09-13T14:43:02.427494+0000","last_clean":"2024-09-13T14:43:02.427494+0000","last_became_active":"2024-09-13T14:37:49.611088+0000","last_became_peered":"2024-09-13T14:37:49.611088+0000","last_unstale":"2024-09-13T14:43:02.427494+0000","last_undegraded":"2024-09-13T14:43:02.427494+0000","last_fullsized":"2024-09-13T14:43:02.427494+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:08.933213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":45,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.302107+0000","last_change":"2024-09-13T14:38:53.899211+0000","last_active":"2024-09-13T14:43:16.302107+0000","last_peered":"2024-09-13T14:43:16.302107+0000","last_clean":"2024-09-13T14:43:16.302107+0000","last_became_active":"2024-09-13T14:38:53.898927+0000","last_became_peered":"2024-09-13T14:38:53.898927+0000","last_unstale":"2024-09-13T14:43:16.302107+0000","last_undegraded":"2024-09-13T14:43:16.302107+0000","last_fullsized":"2024-09-13T14:43:16.302107+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:00:28.857611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"286'10742","reported_seq":10835,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:32.411575+0000","last_change":"2024-09-13T14:35:24.004651+0000","last_active":"2024-09-13T14:43:32.411575+0000","last_peered":"2024-09-13T14:43:32.411575+0000","last_clean":"2024-09-13T14:43:32.411575+0000","last_became_active":"2024-09-13T14:35:24.004358+0000","last_became_peered":"2024-09-13T14:35:24.004358+0000","last_unstale":"2024-09-13T14:43:32.411575+0000","last_undegraded":"2024-09-13T14:43:32.411575+0000","last_fullsized":"2024-09-13T14:43:32.411575+0000","mapping_epoch":205,"log_start":"213'4484","ondisk_log_start":"213'4484","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6258,"log_dups_size":0,"ondisk_log_size":6258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:57:38.023669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211681280,"num_objects":3230,"num_object_clones":0,"num_object_copies":9690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10742,"num_write_kb":447104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":96,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:43:02.285014+0000","last_change":"2024-09-13T14:37:49.168003+0000","last_active":"2024-09-13T14:43:02.285014+0000","last_peered":"2024-09-13T14:43:02.285014+0000","last_clean":"2024-09-13T14:43:02.285014+0000","last_became_active":"2024-09-13T14:37:49.167638+0000","last_became_peered":"2024-09-13T14:37:49.167638+0000","last_unstale":"2024-09-13T14:43:02.285014+0000","last_undegraded":"2024-09-13T14:43:02.285014+0000","last_fullsized":"2024-09-13T14:43:02.285014+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:29.051981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":96,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.353190+0000","last_change":"2024-09-13T14:38:03.528381+0000","last_active":"2024-09-13T14:43:16.353190+0000","last_peered":"2024-09-13T14:43:16.353190+0000","last_clean":"2024-09-13T14:43:16.353190+0000","last_became_active":"2024-09-13T14:38:03.527282+0000","last_became_peered":"2024-09-13T14:38:03.527282+0000","last_unstale":"2024-09-13T14:43:16.353190+0000","last_undegraded":"2024-09-13T14:43:16.353190+0000","last_fullsized":"2024-09-13T14:43:16.353190+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:24.435067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.f","version":"286'10800","reported_seq":11099,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:31.001894+0000","last_change":"2024-09-13T14:38:16.371079+0000","last_active":"2024-09-13T14:43:31.001894+0000","last_peered":"2024-09-13T14:43:31.001894+0000","last_clean":"2024-09-13T14:43:31.001894+0000","last_became_active":"2024-09-13T14:38:12.859883+0000","last_became_peered":"2024-09-13T14:38:12.859883+0000","last_unstale":"2024-09-13T14:43:31.001894+0000","last_undegraded":"2024-09-13T14:43:31.001894+0000","last_fullsized":"2024-09-13T14:43:31.001894+0000","mapping_epoch":245,"log_start":"217'4863","ondisk_log_start":"217'4863","created":205,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5937,"log_dups_size":0,"ondisk_log_size":5937,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:08.436532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215351296,"num_objects":3286,"num_object_clones":0,"num_object_copies":9858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10800,"num_write_kb":450752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":3670016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":454,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.493550+0000","last_change":"2024-09-13T14:38:03.549595+0000","last_active":"2024-09-13T14:43:16.493550+0000","last_peered":"2024-09-13T14:43:16.493550+0000","last_clean":"2024-09-13T14:43:16.493550+0000","last_became_active":"2024-09-13T14:38:03.533331+0000","last_became_peered":"2024-09-13T14:38:03.533331+0000","last_unstale":"2024-09-13T14:43:16.493550+0000","last_undegraded":"2024-09-13T14:43:16.493550+0000","last_fullsized":"2024-09-13T14:43:16.493550+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:59:02.470343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":270,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:57.799891+0000","last_change":"2024-09-13T14:37:48.127600+0000","last_active":"2024-09-13T14:42:57.799891+0000","last_peered":"2024-09-13T14:42:57.799891+0000","last_clean":"2024-09-13T14:42:57.799891+0000","last_became_active":"2024-09-13T14:37:48.127349+0000","last_became_peered":"2024-09-13T14:37:48.127349+0000","last_unstale":"2024-09-13T14:42:57.799891+0000","last_undegraded":"2024-09-13T14:42:57.799891+0000","last_fullsized":"2024-09-13T14:42:57.799891+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:57.268374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"286'10859","reported_seq":20849,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:32.172212+0000","last_change":"2024-09-13T14:42:30.227259+0000","last_active":"2024-09-13T14:43:32.172212+0000","last_peered":"2024-09-13T14:43:32.172212+0000","last_clean":"2024-09-13T14:43:32.172212+0000","last_became_active":"2024-09-13T14:42:30.209102+0000","last_became_peered":"2024-09-13T14:42:30.209102+0000","last_unstale":"2024-09-13T14:43:32.172212+0000","last_undegraded":"2024-09-13T14:43:32.172212+0000","last_fullsized":"2024-09-13T14:43:32.172212+0000","mapping_epoch":281,"log_start":"219'5100","ondisk_log_start":"219'5100","created":205,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5759,"log_dups_size":0,"ondisk_log_size":5759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:56:16.193287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206503936,"num_objects":3151,"num_object_clones":0,"num_object_copies":9453,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10859,"num_write_kb":448320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4926,"num_bytes_recovered":313851904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":203,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.438029+0000","last_change":"2024-09-13T14:37:48.127604+0000","last_active":"2024-09-13T14:42:56.438029+0000","last_peered":"2024-09-13T14:42:56.438029+0000","last_clean":"2024-09-13T14:42:56.438029+0000","last_became_active":"2024-09-13T14:37:48.127300+0000","last_became_peered":"2024-09-13T14:37:48.127300+0000","last_unstale":"2024-09-13T14:42:56.438029+0000","last_undegraded":"2024-09-13T14:42:56.438029+0000","last_fullsized":"2024-09-13T14:42:56.438029+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:51.999506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":323,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:43:02.800497+0000","last_change":"2024-09-13T14:38:03.529967+0000","last_active":"2024-09-13T14:43:02.800497+0000","last_peered":"2024-09-13T14:43:02.800497+0000","last_clean":"2024-09-13T14:43:02.800497+0000","last_became_active":"2024-09-13T14:38:03.529646+0000","last_became_peered":"2024-09-13T14:38:03.529646+0000","last_unstale":"2024-09-13T14:43:02.800497+0000","last_undegraded":"2024-09-13T14:43:02.800497+0000","last_fullsized":"2024-09-13T14:43:02.800497+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:52:05.193279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"286'10897","reported_seq":20071,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:31.338730+0000","last_change":"2024-09-13T14:40:34.583237+0000","last_active":"2024-09-13T14:43:31.338730+0000","last_peered":"2024-09-13T14:43:31.338730+0000","last_clean":"2024-09-13T14:43:31.338730+0000","last_became_active":"2024-09-13T14:40:34.563989+0000","last_became_peered":"2024-09-13T14:40:34.563989+0000","last_unstale":"2024-09-13T14:43:31.338730+0000","last_undegraded":"2024-09-13T14:43:31.338730+0000","last_fullsized":"2024-09-13T14:43:31.338730+0000","mapping_epoch":266,"log_start":"218'5000","ondisk_log_start":"218'5000","created":205,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5897,"log_dups_size":0,"ondisk_log_size":5897,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:34:43.006051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212467736,"num_objects":3243,"num_object_clones":0,"num_object_copies":9729,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3243,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10897,"num_write_kb":452417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4489,"num_bytes_recovered":289865752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":283,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.437833+0000","last_change":"2024-09-13T14:36:13.860367+0000","last_active":"2024-09-13T14:42:56.437833+0000","last_peered":"2024-09-13T14:42:56.437833+0000","last_clean":"2024-09-13T14:42:56.437833+0000","last_became_active":"2024-09-13T14:36:13.859982+0000","last_became_peered":"2024-09-13T14:36:13.859982+0000","last_unstale":"2024-09-13T14:42:56.437833+0000","last_undegraded":"2024-09-13T14:42:56.437833+0000","last_fullsized":"2024-09-13T14:42:56.437833+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:08:23.311350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":279,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:43:02.800520+0000","last_change":"2024-09-13T14:37:48.147360+0000","last_active":"2024-09-13T14:43:02.800520+0000","last_peered":"2024-09-13T14:43:02.800520+0000","last_clean":"2024-09-13T14:43:02.800520+0000","last_became_active":"2024-09-13T14:37:48.146885+0000","last_became_peered":"2024-09-13T14:37:48.146885+0000","last_unstale":"2024-09-13T14:43:02.800520+0000","last_undegraded":"2024-09-13T14:43:02.800520+0000","last_fullsized":"2024-09-13T14:43:02.800520+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:27.841810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"286'10814","reported_seq":20095,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:32.487165+0000","last_change":"2024-09-13T14:38:39.650332+0000","last_active":"2024-09-13T14:43:32.487165+0000","last_peered":"2024-09-13T14:43:32.487165+0000","last_clean":"2024-09-13T14:43:32.487165+0000","last_became_active":"2024-09-13T14:38:39.640835+0000","last_became_peered":"2024-09-13T14:38:39.640835+0000","last_unstale":"2024-09-13T14:43:32.487165+0000","last_undegraded":"2024-09-13T14:43:32.487165+0000","last_fullsized":"2024-09-13T14:43:32.487165+0000","mapping_epoch":251,"log_start":"219'5151","ondisk_log_start":"219'5151","created":205,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5663,"log_dups_size":0,"ondisk_log_size":5663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:48.176606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213647360,"num_objects":3260,"num_object_clones":0,"num_object_copies":9780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10814,"num_write_kb":450368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4544,"num_bytes_recovered":292945920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":472,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.493522+0000","last_change":"2024-09-13T14:38:53.900041+0000","last_active":"2024-09-13T14:43:16.493522+0000","last_peered":"2024-09-13T14:43:16.493522+0000","last_clean":"2024-09-13T14:43:16.493522+0000","last_became_active":"2024-09-13T14:38:53.899203+0000","last_became_peered":"2024-09-13T14:38:53.899203+0000","last_unstale":"2024-09-13T14:43:16.493522+0000","last_undegraded":"2024-09-13T14:43:16.493522+0000","last_fullsized":"2024-09-13T14:43:16.493522+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:28:07.883094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":404,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.494092+0000","last_change":"2024-09-13T14:26:15.823198+0000","last_active":"2024-09-13T14:43:16.494092+0000","last_peered":"2024-09-13T14:43:16.494092+0000","last_clean":"2024-09-13T14:43:16.494092+0000","last_became_active":"2024-09-13T14:26:15.822717+0000","last_became_peered":"2024-09-13T14:26:15.822717+0000","last_unstale":"2024-09-13T14:43:16.494092+0000","last_undegraded":"2024-09-13T14:43:16.494092+0000","last_fullsized":"2024-09-13T14:43:16.494092+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:22.471622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.6","version":"286'10943","reported_seq":18609,"reported_epoch":286,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-13T14:43:31.365976+0000","last_change":"2024-09-13T14:38:55.132683+0000","last_active":"2024-09-13T14:43:31.365976+0000","last_peered":"2024-09-13T14:43:31.365976+0000","last_clean":"2024-09-13T14:36:13.124368+0000","last_became_active":"2024-09-13T14:38:55.120883+0000","last_became_peered":"2024-09-13T14:38:55.120883+0000","last_unstale":"2024-09-13T14:43:31.365976+0000","last_undegraded":"2024-09-13T14:38:55.026610+0000","last_fullsized":"2024-09-13T14:43:31.365976+0000","mapping_epoch":254,"log_start":"218'5019","ondisk_log_start":"218'5019","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5924,"log_dups_size":0,"ondisk_log_size":5924,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213319680,"num_objects":3255,"num_object_clones":0,"num_object_copies":9765,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1268,"num_objects_misplaced":1987,"num_objects_unfound":0,"num_objects_dirty":3255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10943,"num_write_kb":454336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3737,"num_bytes_recovered":233701376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,3,5],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3,4","objects":989},{"shards":"0,3,5","objects":2266}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":419,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.439461+0000","last_change":"2024-09-13T14:41:53.196835+0000","last_active":"2024-09-13T14:42:56.439461+0000","last_peered":"2024-09-13T14:42:56.439461+0000","last_clean":"2024-09-13T14:42:56.439461+0000","last_became_active":"2024-09-13T14:41:53.196387+0000","last_became_peered":"2024-09-13T14:41:53.196387+0000","last_unstale":"2024-09-13T14:42:56.439461+0000","last_undegraded":"2024-09-13T14:42:56.439461+0000","last_fullsized":"2024-09-13T14:42:56.439461+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:07:41.927740+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:49.972303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":525,"reported_epoch":285,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:42:56.453618+0000","last_change":"2024-09-13T14:38:54.904329+0000","last_active":"2024-09-13T14:42:56.453618+0000","last_peered":"2024-09-13T14:42:56.453618+0000","last_clean":"2024-09-13T14:38:39.648017+0000","last_became_active":"2024-09-13T14:38:54.903798+0000","last_became_peered":"2024-09-13T14:38:54.903798+0000","last_unstale":"2024-09-13T14:42:56.453618+0000","last_undegraded":"2024-09-13T14:38:54.890505+0000","last_fullsized":"2024-09-13T14:38:54.889701+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:18.120931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,3],"avail_no_missing":["5","3","6"],"object_location_counts":[{"shards":"3,5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":406,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.493570+0000","last_change":"2024-09-13T14:37:49.160719+0000","last_active":"2024-09-13T14:43:16.493570+0000","last_peered":"2024-09-13T14:43:16.493570+0000","last_clean":"2024-09-13T14:43:16.493570+0000","last_became_active":"2024-09-13T14:37:49.160401+0000","last_became_peered":"2024-09-13T14:37:49.160401+0000","last_unstale":"2024-09-13T14:43:16.493570+0000","last_undegraded":"2024-09-13T14:43:16.493570+0000","last_fullsized":"2024-09-13T14:43:16.493570+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:21.737945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":130,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.437752+0000","last_change":"2024-09-13T14:37:48.172940+0000","last_active":"2024-09-13T14:42:56.437752+0000","last_peered":"2024-09-13T14:42:56.437752+0000","last_clean":"2024-09-13T14:42:56.437752+0000","last_became_active":"2024-09-13T14:37:48.172060+0000","last_became_peered":"2024-09-13T14:37:48.172060+0000","last_unstale":"2024-09-13T14:42:56.437752+0000","last_undegraded":"2024-09-13T14:42:56.437752+0000","last_fullsized":"2024-09-13T14:42:56.437752+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:07.480645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.5","version":"286'10921","reported_seq":27290,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:32.393782+0000","last_change":"2024-09-13T14:39:34.446699+0000","last_active":"2024-09-13T14:43:32.393782+0000","last_peered":"2024-09-13T14:43:32.393782+0000","last_clean":"2024-09-13T14:43:32.393782+0000","last_became_active":"2024-09-13T14:38:04.600616+0000","last_became_peered":"2024-09-13T14:38:04.600616+0000","last_unstale":"2024-09-13T14:43:32.393782+0000","last_undegraded":"2024-09-13T14:43:32.393782+0000","last_fullsized":"2024-09-13T14:43:32.393782+0000","mapping_epoch":245,"log_start":"213'4616","ondisk_log_start":"213'4616","created":205,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6305,"log_dups_size":0,"ondisk_log_size":6305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:00:44.321678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211091456,"num_objects":3221,"num_object_clones":0,"num_object_copies":9663,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10921,"num_write_kb":452544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8120,"num_bytes_recovered":521076736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":503,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.439431+0000","last_change":"2024-09-13T14:26:15.835178+0000","last_active":"2024-09-13T14:42:56.439431+0000","last_peered":"2024-09-13T14:42:56.439431+0000","last_clean":"2024-09-13T14:42:56.439431+0000","last_became_active":"2024-09-13T14:26:15.820283+0000","last_became_peered":"2024-09-13T14:26:15.820283+0000","last_unstale":"2024-09-13T14:42:56.439431+0000","last_undegraded":"2024-09-13T14:42:56.439431+0000","last_fullsized":"2024-09-13T14:42:56.439431+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:30.340038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":96,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.352500+0000","last_change":"2024-09-13T14:38:03.525682+0000","last_active":"2024-09-13T14:43:16.352500+0000","last_peered":"2024-09-13T14:43:16.352500+0000","last_clean":"2024-09-13T14:43:16.352500+0000","last_became_active":"2024-09-13T14:38:03.525406+0000","last_became_peered":"2024-09-13T14:38:03.525406+0000","last_unstale":"2024-09-13T14:43:16.352500+0000","last_undegraded":"2024-09-13T14:43:16.352500+0000","last_fullsized":"2024-09-13T14:43:16.352500+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:41:28.970300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":103,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.353659+0000","last_change":"2024-09-13T14:38:53.991738+0000","last_active":"2024-09-13T14:43:16.353659+0000","last_peered":"2024-09-13T14:43:16.353659+0000","last_clean":"2024-09-13T14:43:16.353659+0000","last_became_active":"2024-09-13T14:38:53.991417+0000","last_became_peered":"2024-09-13T14:38:53.991417+0000","last_unstale":"2024-09-13T14:43:16.353659+0000","last_undegraded":"2024-09-13T14:43:16.353659+0000","last_fullsized":"2024-09-13T14:43:16.353659+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:50:38.559838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"286'10855","reported_seq":27375,"reported_epoch":286,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-13T14:43:31.472963+0000","last_change":"2024-09-13T14:38:17.472071+0000","last_active":"2024-09-13T14:43:31.472963+0000","last_peered":"2024-09-13T14:43:31.472963+0000","last_clean":"2024-09-13T14:37:46.290754+0000","last_became_active":"2024-09-13T14:38:17.381753+0000","last_became_peered":"2024-09-13T14:38:17.381753+0000","last_unstale":"2024-09-13T14:43:31.472963+0000","last_undegraded":"2024-09-13T14:38:13.504140+0000","last_fullsized":"2024-09-13T14:43:31.472963+0000","mapping_epoch":245,"log_start":"218'4959","ondisk_log_start":"218'4959","created":205,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5896,"log_dups_size":0,"ondisk_log_size":5896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213057536,"num_objects":3251,"num_object_clones":0,"num_object_copies":9753,"num_objects_missing_on_primary":83,"num_objects_missing":83,"num_objects_degraded":3419,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10855,"num_write_kb":451392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8216,"num_bytes_recovered":538443776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"0,4,5","objects":3250},{"shards":"0,4,6","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":817,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.493315+0000","last_change":"2024-09-13T14:37:48.131799+0000","last_active":"2024-09-13T14:43:16.493315+0000","last_peered":"2024-09-13T14:43:16.493315+0000","last_clean":"2024-09-13T14:43:16.493315+0000","last_became_active":"2024-09-13T14:37:48.130837+0000","last_became_peered":"2024-09-13T14:37:48.130837+0000","last_unstale":"2024-09-13T14:43:16.493315+0000","last_undegraded":"2024-09-13T14:43:16.493315+0000","last_fullsized":"2024-09-13T14:43:16.493315+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:06:32.663770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":336,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.352626+0000","last_change":"2024-09-13T14:37:49.301531+0000","last_active":"2024-09-13T14:43:16.352626+0000","last_peered":"2024-09-13T14:43:16.352626+0000","last_clean":"2024-09-13T14:43:16.352626+0000","last_became_active":"2024-09-13T14:37:49.301249+0000","last_became_peered":"2024-09-13T14:37:49.301249+0000","last_unstale":"2024-09-13T14:43:16.352626+0000","last_undegraded":"2024-09-13T14:43:16.352626+0000","last_fullsized":"2024-09-13T14:43:16.352626+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:10.977710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":112,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.454314+0000","last_change":"2024-09-13T14:38:53.901427+0000","last_active":"2024-09-13T14:42:56.454314+0000","last_peered":"2024-09-13T14:42:56.454314+0000","last_clean":"2024-09-13T14:42:56.454314+0000","last_became_active":"2024-09-13T14:38:53.901157+0000","last_became_peered":"2024-09-13T14:38:53.901157+0000","last_unstale":"2024-09-13T14:42:56.454314+0000","last_undegraded":"2024-09-13T14:42:56.454314+0000","last_fullsized":"2024-09-13T14:42:56.454314+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:28:21.713025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"286'10768","reported_seq":34003,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:31.179689+0000","last_change":"2024-09-13T14:43:16.851222+0000","last_active":"2024-09-13T14:43:31.179689+0000","last_peered":"2024-09-13T14:43:31.179689+0000","last_clean":"2024-09-13T14:43:31.179689+0000","last_became_active":"2024-09-13T14:38:16.427510+0000","last_became_peered":"2024-09-13T14:38:16.427510+0000","last_unstale":"2024-09-13T14:43:31.179689+0000","last_undegraded":"2024-09-13T14:43:31.179689+0000","last_fullsized":"2024-09-13T14:43:31.179689+0000","mapping_epoch":245,"log_start":"217'4818","ondisk_log_start":"217'4818","created":205,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5950,"log_dups_size":0,"ondisk_log_size":5950,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:27:26.281938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211681280,"num_objects":3230,"num_object_clones":0,"num_object_copies":9690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10768,"num_write_kb":447936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11571,"num_bytes_recovered":747372544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":994,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.453656+0000","last_change":"2024-09-13T14:37:49.147498+0000","last_active":"2024-09-13T14:42:56.453656+0000","last_peered":"2024-09-13T14:42:56.453656+0000","last_clean":"2024-09-13T14:42:56.453656+0000","last_became_active":"2024-09-13T14:37:49.147028+0000","last_became_peered":"2024-09-13T14:37:49.147028+0000","last_unstale":"2024-09-13T14:42:56.453656+0000","last_undegraded":"2024-09-13T14:42:56.453656+0000","last_fullsized":"2024-09-13T14:42:56.453656+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:11:29.050178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":292,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.438039+0000","last_change":"2024-09-13T14:38:03.528133+0000","last_active":"2024-09-13T14:42:56.438039+0000","last_peered":"2024-09-13T14:42:56.438039+0000","last_clean":"2024-09-13T14:42:56.438039+0000","last_became_active":"2024-09-13T14:38:03.527829+0000","last_became_peered":"2024-09-13T14:38:03.527829+0000","last_unstale":"2024-09-13T14:42:56.438039+0000","last_undegraded":"2024-09-13T14:42:56.438039+0000","last_fullsized":"2024-09-13T14:42:56.438039+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:48.254169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":456,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.493487+0000","last_change":"2024-09-13T14:38:53.899755+0000","last_active":"2024-09-13T14:43:16.493487+0000","last_peered":"2024-09-13T14:43:16.493487+0000","last_clean":"2024-09-13T14:43:16.493487+0000","last_became_active":"2024-09-13T14:38:53.899448+0000","last_became_peered":"2024-09-13T14:38:53.899448+0000","last_unstale":"2024-09-13T14:43:16.493487+0000","last_undegraded":"2024-09-13T14:43:16.493487+0000","last_fullsized":"2024-09-13T14:43:16.493487+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:55.897035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"286'11073","reported_seq":14276,"reported_epoch":286,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T14:43:32.810352+0000","last_change":"2024-09-13T14:43:16.854832+0000","last_active":"2024-09-13T14:43:32.810352+0000","last_peered":"2024-09-13T14:43:32.810352+0000","last_clean":"2024-09-13T14:35:51.090791+0000","last_became_active":"2024-09-13T14:37:49.665025+0000","last_became_peered":"2024-09-13T14:37:49.665025+0000","last_unstale":"2024-09-13T14:43:32.810352+0000","last_undegraded":"2024-09-13T14:37:49.169101+0000","last_fullsized":"2024-09-13T14:37:49.121596+0000","mapping_epoch":240,"log_start":"213'4663","ondisk_log_start":"213'4663","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6410,"log_dups_size":0,"ondisk_log_size":6410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:02.321504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211615744,"num_objects":3229,"num_object_clones":0,"num_object_copies":9687,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2891,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11073,"num_write_kb":457664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1545,"num_bytes_recovered":101253120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":338}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":96,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.454048+0000","last_change":"2024-09-13T14:37:49.149770+0000","last_active":"2024-09-13T14:42:56.454048+0000","last_peered":"2024-09-13T14:42:56.454048+0000","last_clean":"2024-09-13T14:42:56.454048+0000","last_became_active":"2024-09-13T14:37:49.148991+0000","last_became_peered":"2024-09-13T14:37:49.148991+0000","last_unstale":"2024-09-13T14:42:56.454048+0000","last_undegraded":"2024-09-13T14:42:56.454048+0000","last_fullsized":"2024-09-13T14:42:56.454048+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:04:04.193366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":1155,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.375419+0000","last_change":"2024-09-13T14:30:58.928302+0000","last_active":"2024-09-13T14:43:16.375419+0000","last_peered":"2024-09-13T14:43:16.375419+0000","last_clean":"2024-09-13T14:43:16.375419+0000","last_became_active":"2024-09-13T14:30:58.927718+0000","last_became_peered":"2024-09-13T14:30:58.927718+0000","last_unstale":"2024-09-13T14:43:16.375419+0000","last_undegraded":"2024-09-13T14:43:16.375419+0000","last_fullsized":"2024-09-13T14:43:16.375419+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:37.323036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":103,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.453812+0000","last_change":"2024-09-13T14:38:03.534628+0000","last_active":"2024-09-13T14:42:56.453812+0000","last_peered":"2024-09-13T14:42:56.453812+0000","last_clean":"2024-09-13T14:42:56.453812+0000","last_became_active":"2024-09-13T14:38:03.534262+0000","last_became_peered":"2024-09-13T14:38:03.534262+0000","last_unstale":"2024-09-13T14:42:56.453812+0000","last_undegraded":"2024-09-13T14:42:56.453812+0000","last_fullsized":"2024-09-13T14:42:56.453812+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:31.168783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"286'10757","reported_seq":24930,"reported_epoch":286,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:43:32.160531+0000","last_change":"2024-09-13T14:38:53.944111+0000","last_active":"2024-09-13T14:43:32.160531+0000","last_peered":"2024-09-13T14:43:32.160531+0000","last_clean":"2024-09-13T14:37:47.047672+0000","last_became_active":"2024-09-13T14:37:49.664294+0000","last_became_peered":"2024-09-13T14:37:49.664294+0000","last_unstale":"2024-09-13T14:43:32.160531+0000","last_undegraded":"2024-09-13T14:38:53.940822+0000","last_fullsized":"2024-09-13T14:37:49.126627+0000","mapping_epoch":240,"log_start":"218'5000","ondisk_log_start":"218'5000","created":205,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5757,"log_dups_size":0,"ondisk_log_size":5757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:11.766692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215941120,"num_objects":3295,"num_object_clones":0,"num_object_copies":9885,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4535,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10757,"num_write_kb":449664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7009,"num_bytes_recovered":459341824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3],"avail_no_missing":["5","0","3"],"object_location_counts":[{"shards":"0,3,5","objects":3295},{"shards":"3,5","objects":-1165}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":115,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.454140+0000","last_change":"2024-09-13T14:38:03.534303+0000","last_active":"2024-09-13T14:42:56.454140+0000","last_peered":"2024-09-13T14:42:56.454140+0000","last_clean":"2024-09-13T14:42:56.454140+0000","last_became_active":"2024-09-13T14:38:03.533171+0000","last_became_peered":"2024-09-13T14:38:03.533171+0000","last_unstale":"2024-09-13T14:42:56.454140+0000","last_undegraded":"2024-09-13T14:42:56.454140+0000","last_fullsized":"2024-09-13T14:42:56.454140+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:00:16.553526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":807,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.440019+0000","last_change":"2024-09-13T14:38:04.596192+0000","last_active":"2024-09-13T14:42:56.440019+0000","last_peered":"2024-09-13T14:42:56.440019+0000","last_clean":"2024-09-13T14:42:56.440019+0000","last_became_active":"2024-09-13T14:38:04.593299+0000","last_became_peered":"2024-09-13T14:38:04.593299+0000","last_unstale":"2024-09-13T14:42:56.440019+0000","last_undegraded":"2024-09-13T14:42:56.440019+0000","last_fullsized":"2024-09-13T14:42:56.440019+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:34.241410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":96,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.353314+0000","last_change":"2024-09-13T14:38:03.533120+0000","last_active":"2024-09-13T14:43:16.353314+0000","last_peered":"2024-09-13T14:43:16.353314+0000","last_clean":"2024-09-13T14:43:16.353314+0000","last_became_active":"2024-09-13T14:38:03.532829+0000","last_became_peered":"2024-09-13T14:38:03.532829+0000","last_unstale":"2024-09-13T14:43:16.353314+0000","last_undegraded":"2024-09-13T14:43:16.353314+0000","last_fullsized":"2024-09-13T14:43:16.353314+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:24.514661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"286'11003","reported_seq":20263,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:32.466432+0000","last_change":"2024-09-13T14:42:04.660877+0000","last_active":"2024-09-13T14:43:32.466432+0000","last_peered":"2024-09-13T14:43:32.466432+0000","last_clean":"2024-09-13T14:43:32.466432+0000","last_became_active":"2024-09-13T14:42:04.641680+0000","last_became_peered":"2024-09-13T14:42:04.641680+0000","last_unstale":"2024-09-13T14:43:32.466432+0000","last_undegraded":"2024-09-13T14:43:32.466432+0000","last_fullsized":"2024-09-13T14:43:32.466432+0000","mapping_epoch":277,"log_start":"219'5318","ondisk_log_start":"219'5318","created":205,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5685,"log_dups_size":0,"ondisk_log_size":5685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:22:18.955428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222494720,"num_objects":3395,"num_object_clones":0,"num_object_copies":10185,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11003,"num_write_kb":460736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4533,"num_bytes_recovered":297074688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":498,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.494152+0000","last_change":"2024-09-13T14:38:53.938227+0000","last_active":"2024-09-13T14:43:16.494152+0000","last_peered":"2024-09-13T14:43:16.494152+0000","last_clean":"2024-09-13T14:43:16.494152+0000","last_became_active":"2024-09-13T14:38:53.902867+0000","last_became_peered":"2024-09-13T14:38:53.902867+0000","last_unstale":"2024-09-13T14:43:16.494152+0000","last_undegraded":"2024-09-13T14:43:16.494152+0000","last_fullsized":"2024-09-13T14:43:16.494152+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:16.831756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":586,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.438207+0000","last_change":"2024-09-13T14:37:48.147677+0000","last_active":"2024-09-13T14:42:56.438207+0000","last_peered":"2024-09-13T14:42:56.438207+0000","last_clean":"2024-09-13T14:42:56.438207+0000","last_became_active":"2024-09-13T14:37:48.147001+0000","last_became_peered":"2024-09-13T14:37:48.147001+0000","last_unstale":"2024-09-13T14:42:56.438207+0000","last_undegraded":"2024-09-13T14:42:56.438207+0000","last_fullsized":"2024-09-13T14:42:56.438207+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:35:00.653576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":480,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.493312+0000","last_change":"2024-09-13T14:38:53.900107+0000","last_active":"2024-09-13T14:43:16.493312+0000","last_peered":"2024-09-13T14:43:16.493312+0000","last_clean":"2024-09-13T14:43:16.493312+0000","last_became_active":"2024-09-13T14:38:53.899556+0000","last_became_peered":"2024-09-13T14:38:53.899556+0000","last_unstale":"2024-09-13T14:43:16.493312+0000","last_undegraded":"2024-09-13T14:43:16.493312+0000","last_fullsized":"2024-09-13T14:43:16.493312+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:15:47.918163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"286'10643","reported_seq":19629,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:32.463426+0000","last_change":"2024-09-13T14:39:54.921239+0000","last_active":"2024-09-13T14:43:32.463426+0000","last_peered":"2024-09-13T14:43:32.463426+0000","last_clean":"2024-09-13T14:43:32.463426+0000","last_became_active":"2024-09-13T14:39:54.907894+0000","last_became_peered":"2024-09-13T14:39:54.907894+0000","last_unstale":"2024-09-13T14:43:32.463426+0000","last_undegraded":"2024-09-13T14:43:32.463426+0000","last_fullsized":"2024-09-13T14:43:32.463426+0000","mapping_epoch":263,"log_start":"219'4947","ondisk_log_start":"219'4947","created":205,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5696,"log_dups_size":0,"ondisk_log_size":5696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:52:04.619238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210305024,"num_objects":3209,"num_object_clones":0,"num_object_copies":9627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10643,"num_write_kb":443264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4395,"num_bytes_recovered":283770880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":111,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.353565+0000","last_change":"2024-09-13T14:38:53.901806+0000","last_active":"2024-09-13T14:43:16.353565+0000","last_peered":"2024-09-13T14:43:16.353565+0000","last_clean":"2024-09-13T14:43:16.353565+0000","last_became_active":"2024-09-13T14:38:53.891055+0000","last_became_peered":"2024-09-13T14:38:53.891055+0000","last_unstale":"2024-09-13T14:43:16.353565+0000","last_undegraded":"2024-09-13T14:43:16.353565+0000","last_fullsized":"2024-09-13T14:43:16.353565+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:03:00.416198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":61,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.854553+0000","last_change":"2024-09-13T14:43:16.854299+0000","last_active":"2024-09-13T14:43:16.854553+0000","last_peered":"2024-09-13T14:43:16.854553+0000","last_clean":"2024-09-13T14:43:16.854553+0000","last_became_active":"2024-09-13T14:38:53.900626+0000","last_became_peered":"2024-09-13T14:38:53.900626+0000","last_unstale":"2024-09-13T14:43:16.854553+0000","last_undegraded":"2024-09-13T14:43:16.854553+0000","last_fullsized":"2024-09-13T14:43:16.854553+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:37:21.715778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":276,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.437712+0000","last_change":"2024-09-13T14:37:49.172218+0000","last_active":"2024-09-13T14:42:56.437712+0000","last_peered":"2024-09-13T14:42:56.437712+0000","last_clean":"2024-09-13T14:42:56.437712+0000","last_became_active":"2024-09-13T14:37:49.171718+0000","last_became_peered":"2024-09-13T14:37:49.171718+0000","last_unstale":"2024-09-13T14:42:56.437712+0000","last_undegraded":"2024-09-13T14:42:56.437712+0000","last_fullsized":"2024-09-13T14:42:56.437712+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:46:54.626521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":298,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.437958+0000","last_change":"2024-09-13T14:37:49.230094+0000","last_active":"2024-09-13T14:42:56.437958+0000","last_peered":"2024-09-13T14:42:56.437958+0000","last_clean":"2024-09-13T14:42:56.437958+0000","last_became_active":"2024-09-13T14:37:49.229393+0000","last_became_peered":"2024-09-13T14:37:49.229393+0000","last_unstale":"2024-09-13T14:42:56.437958+0000","last_undegraded":"2024-09-13T14:42:56.437958+0000","last_fullsized":"2024-09-13T14:42:56.437958+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:02:28.690318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":278,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.438136+0000","last_change":"2024-09-13T14:37:48.128447+0000","last_active":"2024-09-13T14:42:56.438136+0000","last_peered":"2024-09-13T14:42:56.438136+0000","last_clean":"2024-09-13T14:42:56.438136+0000","last_became_active":"2024-09-13T14:37:48.128143+0000","last_became_peered":"2024-09-13T14:37:48.128143+0000","last_unstale":"2024-09-13T14:42:56.438136+0000","last_undegraded":"2024-09-13T14:42:56.438136+0000","last_fullsized":"2024-09-13T14:42:56.438136+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:22.204039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":96,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.353100+0000","last_change":"2024-09-13T14:38:03.531683+0000","last_active":"2024-09-13T14:43:16.353100+0000","last_peered":"2024-09-13T14:43:16.353100+0000","last_clean":"2024-09-13T14:43:16.353100+0000","last_became_active":"2024-09-13T14:38:03.531476+0000","last_became_peered":"2024-09-13T14:38:03.531476+0000","last_unstale":"2024-09-13T14:43:16.353100+0000","last_undegraded":"2024-09-13T14:43:16.353100+0000","last_fullsized":"2024-09-13T14:43:16.353100+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:41.557569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":50,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.439755+0000","last_change":"2024-09-13T14:38:53.954281+0000","last_active":"2024-09-13T14:42:56.439755+0000","last_peered":"2024-09-13T14:42:56.439755+0000","last_clean":"2024-09-13T14:42:56.439755+0000","last_became_active":"2024-09-13T14:38:53.900912+0000","last_became_peered":"2024-09-13T14:38:53.900912+0000","last_unstale":"2024-09-13T14:42:56.439755+0000","last_undegraded":"2024-09-13T14:42:56.439755+0000","last_fullsized":"2024-09-13T14:42:56.439755+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:47.876275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":103,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.301721+0000","last_change":"2024-09-13T14:38:54.000782+0000","last_active":"2024-09-13T14:43:16.301721+0000","last_peered":"2024-09-13T14:43:16.301721+0000","last_clean":"2024-09-13T14:43:16.301721+0000","last_became_active":"2024-09-13T14:38:54.000322+0000","last_became_peered":"2024-09-13T14:38:54.000322+0000","last_unstale":"2024-09-13T14:43:16.301721+0000","last_undegraded":"2024-09-13T14:43:16.301721+0000","last_fullsized":"2024-09-13T14:43:16.301721+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:09.057597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":422,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.493191+0000","last_change":"2024-09-13T14:38:53.899535+0000","last_active":"2024-09-13T14:43:16.493191+0000","last_peered":"2024-09-13T14:43:16.493191+0000","last_clean":"2024-09-13T14:43:16.493191+0000","last_became_active":"2024-09-13T14:38:53.899183+0000","last_became_peered":"2024-09-13T14:38:53.899183+0000","last_unstale":"2024-09-13T14:43:16.493191+0000","last_undegraded":"2024-09-13T14:43:16.493191+0000","last_fullsized":"2024-09-13T14:43:16.493191+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:33.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":96,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.353862+0000","last_change":"2024-09-13T14:38:03.543426+0000","last_active":"2024-09-13T14:43:16.353862+0000","last_peered":"2024-09-13T14:43:16.353862+0000","last_clean":"2024-09-13T14:43:16.353862+0000","last_became_active":"2024-09-13T14:38:03.542661+0000","last_became_peered":"2024-09-13T14:38:03.542661+0000","last_unstale":"2024-09-13T14:43:16.353862+0000","last_undegraded":"2024-09-13T14:43:16.353862+0000","last_fullsized":"2024-09-13T14:43:16.353862+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:54.050127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":451,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.493264+0000","last_change":"2024-09-13T14:37:48.131321+0000","last_active":"2024-09-13T14:43:16.493264+0000","last_peered":"2024-09-13T14:43:16.493264+0000","last_clean":"2024-09-13T14:43:16.493264+0000","last_became_active":"2024-09-13T14:37:48.130998+0000","last_became_peered":"2024-09-13T14:37:48.130998+0000","last_unstale":"2024-09-13T14:43:16.493264+0000","last_undegraded":"2024-09-13T14:43:16.493264+0000","last_fullsized":"2024-09-13T14:43:16.493264+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:33:26.597825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":115,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.454306+0000","last_change":"2024-09-13T14:38:03.535454+0000","last_active":"2024-09-13T14:42:56.454306+0000","last_peered":"2024-09-13T14:42:56.454306+0000","last_clean":"2024-09-13T14:42:56.454306+0000","last_became_active":"2024-09-13T14:38:03.535131+0000","last_became_peered":"2024-09-13T14:38:03.535131+0000","last_unstale":"2024-09-13T14:42:56.454306+0000","last_undegraded":"2024-09-13T14:42:56.454306+0000","last_fullsized":"2024-09-13T14:42:56.454306+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:24.477601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":475,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.493400+0000","last_change":"2024-09-13T14:38:03.524939+0000","last_active":"2024-09-13T14:43:16.493400+0000","last_peered":"2024-09-13T14:43:16.493400+0000","last_clean":"2024-09-13T14:43:16.493400+0000","last_became_active":"2024-09-13T14:38:03.524406+0000","last_became_peered":"2024-09-13T14:38:03.524406+0000","last_unstale":"2024-09-13T14:43:16.493400+0000","last_undegraded":"2024-09-13T14:43:16.493400+0000","last_fullsized":"2024-09-13T14:43:16.493400+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:19:45.257424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":103,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.353478+0000","last_change":"2024-09-13T14:38:53.892948+0000","last_active":"2024-09-13T14:43:16.353478+0000","last_peered":"2024-09-13T14:43:16.353478+0000","last_clean":"2024-09-13T14:43:16.353478+0000","last_became_active":"2024-09-13T14:38:53.892665+0000","last_became_peered":"2024-09-13T14:38:53.892665+0000","last_unstale":"2024-09-13T14:43:16.353478+0000","last_undegraded":"2024-09-13T14:43:16.353478+0000","last_fullsized":"2024-09-13T14:43:16.353478+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:08.292359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":424,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.493636+0000","last_change":"2024-09-13T14:37:48.139348+0000","last_active":"2024-09-13T14:43:16.493636+0000","last_peered":"2024-09-13T14:43:16.493636+0000","last_clean":"2024-09-13T14:43:16.493636+0000","last_became_active":"2024-09-13T14:37:48.139115+0000","last_became_peered":"2024-09-13T14:37:48.139115+0000","last_unstale":"2024-09-13T14:43:16.493636+0000","last_undegraded":"2024-09-13T14:43:16.493636+0000","last_fullsized":"2024-09-13T14:43:16.493636+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:32:20.413346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":464,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.493760+0000","last_change":"2024-09-13T14:37:48.172088+0000","last_active":"2024-09-13T14:43:16.493760+0000","last_peered":"2024-09-13T14:43:16.493760+0000","last_clean":"2024-09-13T14:43:16.493760+0000","last_became_active":"2024-09-13T14:37:48.171779+0000","last_became_peered":"2024-09-13T14:37:48.171779+0000","last_unstale":"2024-09-13T14:43:16.493760+0000","last_undegraded":"2024-09-13T14:43:16.493760+0000","last_fullsized":"2024-09-13T14:43:16.493760+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:02.852712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":342,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.437691+0000","last_change":"2024-09-13T14:38:53.890236+0000","last_active":"2024-09-13T14:42:56.437691+0000","last_peered":"2024-09-13T14:42:56.437691+0000","last_clean":"2024-09-13T14:42:56.437691+0000","last_became_active":"2024-09-13T14:38:53.889582+0000","last_became_peered":"2024-09-13T14:38:53.889582+0000","last_unstale":"2024-09-13T14:42:56.437691+0000","last_undegraded":"2024-09-13T14:42:56.437691+0000","last_fullsized":"2024-09-13T14:42:56.437691+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:14:29.334774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":450,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.493729+0000","last_change":"2024-09-13T14:38:03.534097+0000","last_active":"2024-09-13T14:43:16.493729+0000","last_peered":"2024-09-13T14:43:16.493729+0000","last_clean":"2024-09-13T14:43:16.493729+0000","last_became_active":"2024-09-13T14:38:03.533794+0000","last_became_peered":"2024-09-13T14:38:03.533794+0000","last_unstale":"2024-09-13T14:43:16.493729+0000","last_undegraded":"2024-09-13T14:43:16.493729+0000","last_fullsized":"2024-09-13T14:43:16.493729+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:23.272098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":96,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.353376+0000","last_change":"2024-09-13T14:38:03.527021+0000","last_active":"2024-09-13T14:43:16.353376+0000","last_peered":"2024-09-13T14:43:16.353376+0000","last_clean":"2024-09-13T14:43:16.353376+0000","last_became_active":"2024-09-13T14:38:03.526698+0000","last_became_peered":"2024-09-13T14:38:03.526698+0000","last_unstale":"2024-09-13T14:43:16.353376+0000","last_undegraded":"2024-09-13T14:43:16.353376+0000","last_fullsized":"2024-09-13T14:43:16.353376+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:01:35.734086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":286,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.437900+0000","last_change":"2024-09-13T14:27:12.834931+0000","last_active":"2024-09-13T14:42:56.437900+0000","last_peered":"2024-09-13T14:42:56.437900+0000","last_clean":"2024-09-13T14:42:56.437900+0000","last_became_active":"2024-09-13T14:27:12.834572+0000","last_became_peered":"2024-09-13T14:27:12.834572+0000","last_unstale":"2024-09-13T14:42:56.437900+0000","last_undegraded":"2024-09-13T14:42:56.437900+0000","last_fullsized":"2024-09-13T14:42:56.437900+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:54:21.155689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":99,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.454062+0000","last_change":"2024-09-13T14:37:49.176796+0000","last_active":"2024-09-13T14:42:56.454062+0000","last_peered":"2024-09-13T14:42:56.454062+0000","last_clean":"2024-09-13T14:42:56.454062+0000","last_became_active":"2024-09-13T14:37:49.176246+0000","last_became_peered":"2024-09-13T14:37:49.176246+0000","last_unstale":"2024-09-13T14:42:56.454062+0000","last_undegraded":"2024-09-13T14:42:56.454062+0000","last_fullsized":"2024-09-13T14:42:56.454062+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:29:24.936718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":297,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.437594+0000","last_change":"2024-09-13T14:38:03.526333+0000","last_active":"2024-09-13T14:42:56.437594+0000","last_peered":"2024-09-13T14:42:56.437594+0000","last_clean":"2024-09-13T14:42:56.437594+0000","last_became_active":"2024-09-13T14:38:03.526001+0000","last_became_peered":"2024-09-13T14:38:03.526001+0000","last_unstale":"2024-09-13T14:42:56.437594+0000","last_undegraded":"2024-09-13T14:42:56.437594+0000","last_fullsized":"2024-09-13T14:42:56.437594+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:25:25.181271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":99,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:56.454140+0000","last_change":"2024-09-13T14:37:49.149582+0000","last_active":"2024-09-13T14:42:56.454140+0000","last_peered":"2024-09-13T14:42:56.454140+0000","last_clean":"2024-09-13T14:42:56.454140+0000","last_became_active":"2024-09-13T14:37:49.148525+0000","last_became_peered":"2024-09-13T14:37:49.148525+0000","last_unstale":"2024-09-13T14:42:56.454140+0000","last_undegraded":"2024-09-13T14:42:56.454140+0000","last_fullsized":"2024-09-13T14:42:56.454140+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:48.655665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":496,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.493493+0000","last_change":"2024-09-13T14:38:03.535255+0000","last_active":"2024-09-13T14:43:16.493493+0000","last_peered":"2024-09-13T14:43:16.493493+0000","last_clean":"2024-09-13T14:43:16.493493+0000","last_became_active":"2024-09-13T14:38:03.533917+0000","last_became_peered":"2024-09-13T14:38:03.533917+0000","last_unstale":"2024-09-13T14:43:16.493493+0000","last_undegraded":"2024-09-13T14:43:16.493493+0000","last_fullsized":"2024-09-13T14:43:16.493493+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:33.559788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":460,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.495256+0000","last_change":"2024-09-13T14:37:48.146784+0000","last_active":"2024-09-13T14:43:16.495256+0000","last_peered":"2024-09-13T14:43:16.495256+0000","last_clean":"2024-09-13T14:43:16.495256+0000","last_became_active":"2024-09-13T14:37:48.146491+0000","last_became_peered":"2024-09-13T14:37:48.146491+0000","last_unstale":"2024-09-13T14:43:16.495256+0000","last_undegraded":"2024-09-13T14:43:16.495256+0000","last_fullsized":"2024-09-13T14:43:16.495256+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:30:54.772513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":278,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-13T14:42:57.799812+0000","last_change":"2024-09-13T14:37:48.127607+0000","last_active":"2024-09-13T14:42:57.799812+0000","last_peered":"2024-09-13T14:42:57.799812+0000","last_clean":"2024-09-13T14:42:57.799812+0000","last_became_active":"2024-09-13T14:37:48.127316+0000","last_became_peered":"2024-09-13T14:37:48.127316+0000","last_unstale":"2024-09-13T14:42:57.799812+0000","last_undegraded":"2024-09-13T14:42:57.799812+0000","last_fullsized":"2024-09-13T14:42:57.799812+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:53:37.678861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":102,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.353755+0000","last_change":"2024-09-13T14:38:03.563200+0000","last_active":"2024-09-13T14:43:16.353755+0000","last_peered":"2024-09-13T14:43:16.353755+0000","last_clean":"2024-09-13T14:43:16.353755+0000","last_became_active":"2024-09-13T14:38:03.534258+0000","last_became_peered":"2024-09-13T14:38:03.534258+0000","last_unstale":"2024-09-13T14:43:16.353755+0000","last_undegraded":"2024-09-13T14:43:16.353755+0000","last_fullsized":"2024-09-13T14:43:16.353755+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:49:07.814419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":3401515032,"num_objects":51904,"num_object_clones":0,"num_object_copies":155712,"num_objects_missing_on_primary":83,"num_objects_missing":83,"num_objects_degraded":12113,"num_objects_misplaced":1987,"num_objects_unfound":0,"num_objects_dirty":51904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173722,"num_write_kb":7219969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80094,"num_bytes_recovered":5188223000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12388216832,"data_stored":12388204616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94545,"ondisk_log_size":94545,"up":48,"acting":46,"num_store_stats":6},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1538,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":17390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25132,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16793600,"data_stored":16777292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":23,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2969600,"data_stored":2951840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234396,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2384720,"kb_used_data":1924552,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":91360432,"statfs":{"total":95995035648,"available":93553082368,"internally_reserved":0,"allocated":1970741248,"data_stored":1969205050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":471202251},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":239,"seq":1026497183814,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1838932,"kb_used_data":1551820,"kb_used_omap":8,"kb_used_meta":287095,"kb_avail":91906220,"statfs":{"total":95995035648,"available":94111969280,"internally_reserved":0,"allocated":1589063680,"data_stored":1587523957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8453,"internal_metadata":293986043},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757661,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2084544,"kb_used_data":1413560,"kb_used_omap":1,"kb_used_meta":670974,"kb_avail":91660608,"statfs":{"total":95995035648,"available":93860462592,"internally_reserved":0,"allocated":1447485440,"data_stored":1445960803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":687077830},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019305,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2363660,"kb_used_data":1773060,"kb_used_omap":1,"kb_used_meta":590590,"kb_avail":91381492,"statfs":{"total":95995035648,"available":93574647808,"internally_reserved":0,"allocated":1815613440,"data_stored":1814081326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":604764614},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435488,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3089084,"kb_used_data":2503476,"kb_used_omap":5,"kb_used_meta":585594,"kb_avail":90656068,"statfs":{"total":95995035648,"available":92831813632,"internally_reserved":0,"allocated":2563559424,"data_stored":2562038647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":599648382},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":446676599016,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":425201762542,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926066,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3659916,"kb_used_data":2978500,"kb_used_omap":7,"kb_used_meta":681400,"kb_avail":90085236,"statfs":{"total":95995035648,"available":92247281664,"internally_reserved":0,"allocated":3049984000,"data_stored":3048448006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":697753896},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2,3,1,0,6,7],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6860,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3040481280,"data_stored":3040477208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2550796288,"data_stored":2550792216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1809055744,"data_stored":1809055744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1439432704,"data_stored":1439432704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1582043136,"data_stored":1582039064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1966407680,"data_stored":1966407680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:43:37.163 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph status --format=json 2024-09-13T14:43:37.164 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:37.188 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:37.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:37.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:37.290 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:37.314 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:37.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:37.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:37.416 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:37.444 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:37.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:37.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:37.546 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:37.570 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:37.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:37.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:37.672 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:37.695 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:37.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:37.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:37.797 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:37.823 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:37.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:37.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:37.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:37 smithi102 ceph-mon[115269]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:43:37.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:37 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/4215048457' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:43:37.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:37 smithi102 ceph-mon[115269]: osdmap e287: 8 total, 8 up, 6 in 2024-09-13T14:43:37.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:37 smithi102 ceph-mon[110183]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:43:37.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:37 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/4215048457' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:43:37.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:37 smithi102 ceph-mon[110183]: osdmap e287: 8 total, 8 up, 6 in 2024-09-13T14:43:37.924 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:37.949 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:37.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:37.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:37.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:37 smithi195 ceph-mon[104262]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:43:37.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:37 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/4215048457' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:43:37.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:37 smithi195 ceph-mon[104262]: osdmap e287: 8 total, 8 up, 6 in 2024-09-13T14:43:37.998 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:43:37.998 INFO:teuthology.orchestra.run.smithi102.stdout: "size": 20, 2024-09-13T14:43:37.998 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 600, 2024-09-13T14:43:37.998 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [ 2024-09-13T14:43:37.998 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:37.998 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35037.0:163517 6.4 6:202dcf30:::benchmark_data_smithi102_194590_object52893:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-09-13T14:43:37.998 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:33:43.555877+0000", 2024-09-13T14:43:37.999 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 594.43916319000004, 2024-09-13T14:43:37.999 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.54220369, 2024-09-13T14:43:37.999 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:43:37.999 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:43:37.999 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:37.999 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:43:37.999 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35037", 2024-09-13T14:43:37.999 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:43:37.999 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 163517 2024-09-13T14:43:37.999 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:37.999 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:43:37.999 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:37.999 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:43:38.000 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.555877+0000", 2024-09-13T14:43:38.000 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:43:38.000 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.000 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.000 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:43:38.000 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.555877+0000", 2024-09-13T14:43:38.000 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:43:38.000 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.000 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.000 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:43:38.000 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.555878+0000", 2024-09-13T14:43:38.000 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2720000000000001e-06 2024-09-13T14:43:38.001 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.001 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.001 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:43:38.001 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.555883+0000", 2024-09-13T14:43:38.001 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.3420000000000001e-06 2024-09-13T14:43:38.001 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.001 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.001 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:43:38.001 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.555893+0000", 2024-09-13T14:43:38.001 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0564000000000001e-05 2024-09-13T14:43:38.001 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.001 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.001 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:43:38.002 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.555901+0000", 2024-09-13T14:43:38.002 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.4939999999999997e-06 2024-09-13T14:43:38.002 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.002 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.002 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.002 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.555965+0000", 2024-09-13T14:43:38.002 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.3816e-05 2024-09-13T14:43:38.002 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.002 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.002 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:43:38.002 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.556003+0000", 2024-09-13T14:43:38.002 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.8167000000000002e-05 2024-09-13T14:43:38.002 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.002 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.003 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.003 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.828407+0000", 2024-09-13T14:43:38.003 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.27240448900000003 2024-09-13T14:43:38.003 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.003 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.003 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:43:38.003 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.828417+0000", 2024-09-13T14:43:38.003 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.7049999999999994e-06 2024-09-13T14:43:38.003 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.003 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.003 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.003 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.033750+0000", 2024-09-13T14:43:38.003 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2053327650000001 2024-09-13T14:43:38.004 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.004 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.004 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:43:38.004 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.036001+0000", 2024-09-13T14:43:38.004 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0022508929999999999 2024-09-13T14:43:38.004 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.004 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.004 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,5,6", 2024-09-13T14:43:38.004 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.036155+0000", 2024-09-13T14:43:38.004 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00015404000000000001 2024-09-13T14:43:38.004 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.004 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.004 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:43:38.004 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.050797+0000", 2024-09-13T14:43:38.005 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.014642471000000001 2024-09-13T14:43:38.005 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.005 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.005 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:38.005 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.069588+0000", 2024-09-13T14:43:38.005 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.018790478999999999 2024-09-13T14:43:38.005 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.005 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.005 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:38.006 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.074823+0000", 2024-09-13T14:43:38.006 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.005235033 2024-09-13T14:43:38.006 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.006 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.006 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:38.006 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.097906+0000", 2024-09-13T14:43:38.006 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.023083372000000001 2024-09-13T14:43:38.006 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.006 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.006 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:43:38.006 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.097968+0000", 2024-09-13T14:43:38.006 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.2368999999999999e-05 2024-09-13T14:43:38.006 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.006 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.007 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:43:38.007 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.098081+0000", 2024-09-13T14:43:38.007 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000112419 2024-09-13T14:43:38.007 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:43:38.007 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:43:38.007 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:43:38.007 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.007 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.007 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35037.0:163519 6.4 6:2757b5a3:::benchmark_data_smithi102_194590_object52895:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-09-13T14:43:38.007 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:33:43.557183+0000", 2024-09-13T14:43:38.007 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 594.43785774000003, 2024-09-13T14:43:38.007 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.541098818, 2024-09-13T14:43:38.007 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:43:38.008 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:43:38.008 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:38.008 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:43:38.008 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35037", 2024-09-13T14:43:38.008 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:43:38.008 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 163519 2024-09-13T14:43:38.008 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.008 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:43:38.008 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.008 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:43:38.008 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.557183+0000", 2024-09-13T14:43:38.008 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:43:38.008 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.009 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.009 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:43:38.009 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.557183+0000", 2024-09-13T14:43:38.009 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:43:38.009 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.009 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.009 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:43:38.009 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.557183+0000", 2024-09-13T14:43:38.009 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.9299999999999997e-07 2024-09-13T14:43:38.009 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.009 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.009 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:43:38.009 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.557187+0000", 2024-09-13T14:43:38.009 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.3230000000000002e-06 2024-09-13T14:43:38.010 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.010 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.010 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:43:38.010 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.557196+0000", 2024-09-13T14:43:38.010 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.7399999999999999e-06 2024-09-13T14:43:38.010 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.010 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.010 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:43:38.010 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.557201+0000", 2024-09-13T14:43:38.010 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.1630000000000004e-06 2024-09-13T14:43:38.010 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.010 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.010 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.011 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.557240+0000", 2024-09-13T14:43:38.011 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.8197e-05 2024-09-13T14:43:38.011 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.011 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.011 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:43:38.011 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.557245+0000", 2024-09-13T14:43:38.011 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.6590000000000004e-06 2024-09-13T14:43:38.011 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.011 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.011 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.011 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.828516+0000", 2024-09-13T14:43:38.011 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.27127105800000001 2024-09-13T14:43:38.011 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.011 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.012 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:43:38.012 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:43.828524+0000", 2024-09-13T14:43:38.012 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.3080000000000002e-06 2024-09-13T14:43:38.012 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.012 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.012 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.012 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.045895+0000", 2024-09-13T14:43:38.012 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.217371282 2024-09-13T14:43:38.012 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.012 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.012 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:43:38.012 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.046020+0000", 2024-09-13T14:43:38.012 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00012525899999999999 2024-09-13T14:43:38.013 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.013 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.013 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,5,6", 2024-09-13T14:43:38.013 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.046165+0000", 2024-09-13T14:43:38.013 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000144764 2024-09-13T14:43:38.013 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.013 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.013 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:43:38.013 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.050805+0000", 2024-09-13T14:43:38.013 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0046395940000000004 2024-09-13T14:43:38.013 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.013 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.013 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:38.014 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.069631+0000", 2024-09-13T14:43:38.014 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.018826664 2024-09-13T14:43:38.014 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.014 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.014 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:38.014 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.074871+0000", 2024-09-13T14:43:38.014 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.005240102 2024-09-13T14:43:38.014 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.014 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.014 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:38.014 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.098129+0000", 2024-09-13T14:43:38.014 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.023257654999999999 2024-09-13T14:43:38.014 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.014 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.015 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:43:38.015 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.098191+0000", 2024-09-13T14:43:38.015 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.2248000000000004e-05 2024-09-13T14:43:38.015 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.015 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.015 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:43:38.015 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:33:45.098281+0000", 2024-09-13T14:43:38.015 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.0109000000000001e-05 2024-09-13T14:43:38.015 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:43:38.015 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:43:38.015 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:43:38.015 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.015 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.015 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35037.0:209609 6.b 6:d3b1c0fa:::benchmark_data_smithi102_194590_object98985:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-13T14:43:38.016 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:34:42.286860+0000", 2024-09-13T14:43:38.016 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 535.70818047600005, 2024-09-13T14:43:38.016 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.5313817680000001, 2024-09-13T14:43:38.016 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:43:38.016 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:43:38.016 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "reached pg", 2024-09-13T14:43:38.016 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:43:38.016 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35037", 2024-09-13T14:43:38.016 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:43:38.016 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 209609 2024-09-13T14:43:38.016 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.016 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:43:38.016 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.016 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:43:38.017 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.286860+0000", 2024-09-13T14:43:38.017 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:43:38.017 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.017 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.017 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:43:38.017 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.286860+0000", 2024-09-13T14:43:38.017 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:43:38.017 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.017 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.017 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:43:38.017 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.286861+0000", 2024-09-13T14:43:38.017 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.78e-07 2024-09-13T14:43:38.017 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.018 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.018 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:43:38.018 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.286865+0000", 2024-09-13T14:43:38.018 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.1330000000000001e-06 2024-09-13T14:43:38.018 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.018 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.018 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:43:38.018 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.286884+0000", 2024-09-13T14:43:38.018 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.8777999999999999e-05 2024-09-13T14:43:38.018 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.018 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.018 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:43:38.018 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.286891+0000", 2024-09-13T14:43:38.018 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.25e-06 2024-09-13T14:43:38.019 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.019 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.019 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.019 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.286949+0000", 2024-09-13T14:43:38.019 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.8517000000000002e-05 2024-09-13T14:43:38.019 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.019 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.019 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:43:38.019 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.287058+0000", 2024-09-13T14:43:38.019 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00010861099999999999 2024-09-13T14:43:38.019 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.019 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.019 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:43:38.019 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.287171+0000", 2024-09-13T14:43:38.020 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000113503 2024-09-13T14:43:38.020 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.020 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.020 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:43:38.020 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.290197+0000", 2024-09-13T14:43:38.020 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0030256430000000002 2024-09-13T14:43:38.020 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.020 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.020 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.020 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.302327+0000", 2024-09-13T14:43:38.020 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.012129707999999999 2024-09-13T14:43:38.020 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.020 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.020 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:43:38.021 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.302335+0000", 2024-09-13T14:43:38.021 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.5939999999999994e-06 2024-09-13T14:43:38.021 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.021 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.021 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.021 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.818191+0000", 2024-09-13T14:43:38.021 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.5158554180000001 2024-09-13T14:43:38.021 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.021 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.021 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:43:38.021 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.818242+0000", 2024-09-13T14:43:38.021 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.0735000000000001e-05 2024-09-13T14:43:38.021 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:43:38.021 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:43:38.022 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:43:38.022 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.022 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.022 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35037.0:209615 6.b 6:ddcfd241:::benchmark_data_smithi102_194590_object98991:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-13T14:43:38.022 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:34:42.293446+0000", 2024-09-13T14:43:38.022 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 535.70159386, 2024-09-13T14:43:38.022 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.533916812, 2024-09-13T14:43:38.022 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:43:38.022 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:43:38.022 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "reached pg", 2024-09-13T14:43:38.022 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:43:38.022 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35037", 2024-09-13T14:43:38.022 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:43:38.022 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 209615 2024-09-13T14:43:38.023 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.023 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:43:38.023 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.023 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:43:38.023 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.293446+0000", 2024-09-13T14:43:38.023 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:43:38.023 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.023 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.023 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:43:38.023 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.293446+0000", 2024-09-13T14:43:38.023 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:43:38.023 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.023 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.023 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:43:38.023 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.293447+0000", 2024-09-13T14:43:38.024 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.7899999999999999e-07 2024-09-13T14:43:38.024 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.024 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.024 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:43:38.024 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.293449+0000", 2024-09-13T14:43:38.024 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.5799999999999999e-06 2024-09-13T14:43:38.024 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.024 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.024 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:43:38.024 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.293456+0000", 2024-09-13T14:43:38.024 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.6900000000000003e-06 2024-09-13T14:43:38.024 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.024 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.024 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:43:38.025 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.293461+0000", 2024-09-13T14:43:38.025 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.1490000000000002e-06 2024-09-13T14:43:38.025 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.025 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.025 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.025 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.294942+0000", 2024-09-13T14:43:38.025 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0014804849999999999 2024-09-13T14:43:38.025 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.025 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.025 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:43:38.025 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.295068+0000", 2024-09-13T14:43:38.025 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00012669800000000001 2024-09-13T14:43:38.025 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.025 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.026 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:43:38.026 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.295202+0000", 2024-09-13T14:43:38.026 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00013326200000000001 2024-09-13T14:43:38.026 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.026 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.026 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:43:38.026 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.298178+0000", 2024-09-13T14:43:38.026 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0029757910000000002 2024-09-13T14:43:38.026 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.026 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.026 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.026 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.302728+0000", 2024-09-13T14:43:38.027 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0045509219999999998 2024-09-13T14:43:38.027 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.027 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.028 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:43:38.028 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.302738+0000", 2024-09-13T14:43:38.028 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.6569999999999992e-06 2024-09-13T14:43:38.028 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.028 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.028 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.028 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.818432+0000", 2024-09-13T14:43:38.028 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.5156941260000001 2024-09-13T14:43:38.028 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.028 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.028 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for missing object", 2024-09-13T14:43:38.028 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.818508+0000", 2024-09-13T14:43:38.028 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.5945999999999994e-05 2024-09-13T14:43:38.028 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.029 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.029 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.029 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.827232+0000", 2024-09-13T14:43:38.029 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0087233269999999995 2024-09-13T14:43:38.029 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.029 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.029 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:43:38.029 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.827363+0000", 2024-09-13T14:43:38.029 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0001317 2024-09-13T14:43:38.029 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:43:38.029 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:43:38.029 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:43:38.029 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.029 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.030 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35037.0:209631 6.b 6:d1535ee8:::benchmark_data_smithi102_194590_object99007:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-09-13T14:43:38.030 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:34:42.315630+0000", 2024-09-13T14:43:38.030 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 535.67941057899998, 2024-09-13T14:43:38.030 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.5154516149999999, 2024-09-13T14:43:38.030 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:43:38.030 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:43:38.030 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:38.030 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:43:38.030 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35037", 2024-09-13T14:43:38.030 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:43:38.030 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 209631 2024-09-13T14:43:38.030 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.030 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:43:38.030 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.031 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:43:38.031 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.315630+0000", 2024-09-13T14:43:38.031 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:43:38.031 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.031 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.031 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:43:38.031 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.315630+0000", 2024-09-13T14:43:38.031 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:43:38.031 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.031 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.031 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:43:38.031 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.315630+0000", 2024-09-13T14:43:38.031 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.5300000000000004e-07 2024-09-13T14:43:38.031 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.032 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.032 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:43:38.032 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.315634+0000", 2024-09-13T14:43:38.032 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.298e-06 2024-09-13T14:43:38.032 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.032 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.032 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:43:38.032 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.315642+0000", 2024-09-13T14:43:38.032 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.6330000000000002e-06 2024-09-13T14:43:38.032 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.032 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.032 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:43:38.032 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.315647+0000", 2024-09-13T14:43:38.033 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.0300000000000001e-06 2024-09-13T14:43:38.033 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.033 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.033 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.033 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.315711+0000", 2024-09-13T14:43:38.033 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.3946000000000001e-05 2024-09-13T14:43:38.033 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.033 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.033 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:43:38.033 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.315720+0000", 2024-09-13T14:43:38.033 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.2979999999999997e-06 2024-09-13T14:43:38.033 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.033 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.033 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.034 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.818542+0000", 2024-09-13T14:43:38.034 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.5028214499999999 2024-09-13T14:43:38.034 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.034 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.034 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:43:38.034 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.818629+0000", 2024-09-13T14:43:38.034 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.7359000000000002e-05 2024-09-13T14:43:38.034 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.034 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.034 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:43:38.034 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.818703+0000", 2024-09-13T14:43:38.034 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.3827999999999999e-05 2024-09-13T14:43:38.034 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.035 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.035 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:43:38.035 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.825562+0000", 2024-09-13T14:43:38.035 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0068591590000000001 2024-09-13T14:43:38.035 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.035 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.035 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:38.035 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.827336+0000", 2024-09-13T14:43:38.035 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0017741569999999999 2024-09-13T14:43:38.035 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.035 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.035 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:38.035 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.830889+0000", 2024-09-13T14:43:38.035 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0035525439999999999 2024-09-13T14:43:38.036 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.036 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.036 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:43:38.036 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.830949+0000", 2024-09-13T14:43:38.036 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.0396999999999998e-05 2024-09-13T14:43:38.036 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.036 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.036 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:43:38.036 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.831081+0000", 2024-09-13T14:43:38.036 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000131963 2024-09-13T14:43:38.036 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:43:38.036 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:43:38.036 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:43:38.036 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.037 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.037 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35037.0:209642 6.b 6:d884ca38:::benchmark_data_smithi102_194590_object99018:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-09-13T14:43:38.037 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:34:42.327971+0000", 2024-09-13T14:43:38.037 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 535.66706896699998, 2024-09-13T14:43:38.037 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.5035762109999999, 2024-09-13T14:43:38.037 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:43:38.037 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:43:38.037 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:38.037 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:43:38.037 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35037", 2024-09-13T14:43:38.037 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:43:38.037 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 209642 2024-09-13T14:43:38.037 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.037 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:43:38.038 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.038 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:43:38.038 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.327971+0000", 2024-09-13T14:43:38.038 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:43:38.038 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.038 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.038 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:43:38.038 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.327971+0000", 2024-09-13T14:43:38.038 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:43:38.038 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.038 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.038 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:43:38.038 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.327972+0000", 2024-09-13T14:43:38.038 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.187e-06 2024-09-13T14:43:38.039 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.039 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.039 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:43:38.039 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.327977+0000", 2024-09-13T14:43:38.039 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.9019999999999998e-06 2024-09-13T14:43:38.039 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.039 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.039 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:43:38.039 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.327987+0000", 2024-09-13T14:43:38.039 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.3200000000000006e-06 2024-09-13T14:43:38.039 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.039 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.039 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:43:38.039 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.327993+0000", 2024-09-13T14:43:38.039 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.3899999999999998e-06 2024-09-13T14:43:38.040 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.040 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.040 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.040 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.342702+0000", 2024-09-13T14:43:38.040 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.014708718000000001 2024-09-13T14:43:38.040 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.040 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.040 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:43:38.040 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:42.342711+0000", 2024-09-13T14:43:38.040 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.38e-06 2024-09-13T14:43:38.040 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.040 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.040 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.040 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.820544+0000", 2024-09-13T14:43:38.041 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.477833231 2024-09-13T14:43:38.041 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.041 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.041 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:43:38.041 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.820606+0000", 2024-09-13T14:43:38.041 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.1655999999999994e-05 2024-09-13T14:43:38.041 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.041 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.041 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:43:38.041 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.820692+0000", 2024-09-13T14:43:38.041 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.5803999999999994e-05 2024-09-13T14:43:38.041 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.041 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.042 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:43:38.042 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.825577+0000", 2024-09-13T14:43:38.042 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0048847370000000001 2024-09-13T14:43:38.042 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.042 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.042 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:38.042 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.831148+0000", 2024-09-13T14:43:38.042 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0055713409999999996 2024-09-13T14:43:38.042 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.042 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.042 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:38.042 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.831393+0000", 2024-09-13T14:43:38.042 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000244952 2024-09-13T14:43:38.042 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.043 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.043 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:43:38.043 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.831449+0000", 2024-09-13T14:43:38.043 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.6057999999999998e-05 2024-09-13T14:43:38.043 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.043 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.043 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:43:38.043 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:34:43.831548+0000", 2024-09-13T14:43:38.043 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.8535000000000004e-05 2024-09-13T14:43:38.043 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:43:38.043 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:43:38.043 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:43:38.043 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.043 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.044 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35379.0:76660 7.a 7:55e7e51b:::benchmark_data_smithi102_227395_object76659:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-09-13T14:43:38.044 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:36:13.172282+0000", 2024-09-13T14:43:38.044 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 444.82275852499998, 2024-09-13T14:43:38.044 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.7816652209999999, 2024-09-13T14:43:38.044 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:43:38.044 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:43:38.044 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:38.044 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:43:38.044 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35379", 2024-09-13T14:43:38.044 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:43:38.044 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 76660 2024-09-13T14:43:38.044 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.044 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:43:38.044 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.045 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:43:38.045 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.172282+0000", 2024-09-13T14:43:38.045 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:43:38.045 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.045 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.045 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:43:38.045 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.172282+0000", 2024-09-13T14:43:38.045 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:43:38.045 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.045 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.045 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:43:38.045 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.172283+0000", 2024-09-13T14:43:38.045 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.6080000000000001e-06 2024-09-13T14:43:38.045 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.046 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.046 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:43:38.046 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.172380+0000", 2024-09-13T14:43:38.046 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.6696999999999999e-05 2024-09-13T14:43:38.046 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.046 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.046 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:43:38.046 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.172393+0000", 2024-09-13T14:43:38.046 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3363999999999999e-05 2024-09-13T14:43:38.046 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.046 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.046 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:43:38.046 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.172405+0000", 2024-09-13T14:43:38.046 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1265e-05 2024-09-13T14:43:38.047 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.047 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.047 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.047 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.172471+0000", 2024-09-13T14:43:38.047 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.6078e-05 2024-09-13T14:43:38.047 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.047 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.047 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:43:38.047 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.172482+0000", 2024-09-13T14:43:38.047 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1392e-05 2024-09-13T14:43:38.047 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.047 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.047 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.047 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.863789+0000", 2024-09-13T14:43:38.048 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.69130646100000004 2024-09-13T14:43:38.048 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.048 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.048 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:43:38.048 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.863802+0000", 2024-09-13T14:43:38.048 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3599e-05 2024-09-13T14:43:38.048 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.048 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.048 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.048 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.944957+0000", 2024-09-13T14:43:38.048 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.08115447 2024-09-13T14:43:38.048 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.049 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.049 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:43:38.049 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.945093+0000", 2024-09-13T14:43:38.049 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000136496 2024-09-13T14:43:38.049 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.049 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.049 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 0,4,5", 2024-09-13T14:43:38.049 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.945201+0000", 2024-09-13T14:43:38.049 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000107774 2024-09-13T14:43:38.049 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.049 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.049 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:38.049 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.948055+0000", 2024-09-13T14:43:38.049 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0028542900000000002 2024-09-13T14:43:38.049 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.050 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.050 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:43:38.050 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.953464+0000", 2024-09-13T14:43:38.050 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0054091709999999999 2024-09-13T14:43:38.050 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.050 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.050 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:38.050 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.953740+0000", 2024-09-13T14:43:38.050 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00027606199999999999 2024-09-13T14:43:38.050 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.050 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.050 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:38.050 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.953807+0000", 2024-09-13T14:43:38.050 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.6389000000000004e-05 2024-09-13T14:43:38.051 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.051 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.051 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:43:38.051 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.953863+0000", 2024-09-13T14:43:38.051 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.5942000000000001e-05 2024-09-13T14:43:38.051 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.051 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.051 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:43:38.051 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.953947+0000", 2024-09-13T14:43:38.051 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.4163e-05 2024-09-13T14:43:38.051 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:43:38.051 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:43:38.051 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:43:38.051 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.052 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.052 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35379.0:76665 7.a 7:55f8d188:::benchmark_data_smithi102_227395_object76664:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-09-13T14:43:38.052 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:36:13.177195+0000", 2024-09-13T14:43:38.052 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 444.81784560599999, 2024-09-13T14:43:38.052 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.777108328, 2024-09-13T14:43:38.052 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:43:38.052 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:43:38.052 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:38.052 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:43:38.052 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35379", 2024-09-13T14:43:38.052 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:43:38.052 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 76665 2024-09-13T14:43:38.052 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.053 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:43:38.053 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.053 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:43:38.053 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.177195+0000", 2024-09-13T14:43:38.053 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:43:38.053 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.053 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.053 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:43:38.053 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.177195+0000", 2024-09-13T14:43:38.053 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:43:38.053 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.053 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.053 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:43:38.053 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.177197+0000", 2024-09-13T14:43:38.053 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.187e-06 2024-09-13T14:43:38.054 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.054 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.054 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:43:38.054 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.177223+0000", 2024-09-13T14:43:38.054 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.5701e-05 2024-09-13T14:43:38.054 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.054 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.054 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:43:38.054 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.177240+0000", 2024-09-13T14:43:38.054 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.6946999999999999e-05 2024-09-13T14:43:38.054 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.054 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.054 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:43:38.054 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.177250+0000", 2024-09-13T14:43:38.055 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0169999999999999e-05 2024-09-13T14:43:38.055 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.055 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.055 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.055 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.179843+0000", 2024-09-13T14:43:38.055 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.002593167 2024-09-13T14:43:38.055 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.055 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.055 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:43:38.055 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.179861+0000", 2024-09-13T14:43:38.055 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.8014000000000001e-05 2024-09-13T14:43:38.055 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.055 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.056 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.056 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.863827+0000", 2024-09-13T14:43:38.056 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.68396646999999999 2024-09-13T14:43:38.056 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.056 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.056 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:43:38.056 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.863835+0000", 2024-09-13T14:43:38.056 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.0700000000000007e-06 2024-09-13T14:43:38.056 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.056 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.056 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.056 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.948168+0000", 2024-09-13T14:43:38.056 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.084332954 2024-09-13T14:43:38.057 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.057 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.057 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:43:38.057 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.948293+0000", 2024-09-13T14:43:38.057 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00012477300000000001 2024-09-13T14:43:38.057 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.057 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.057 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 0,4,5", 2024-09-13T14:43:38.057 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.948388+0000", 2024-09-13T14:43:38.057 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.4470999999999997e-05 2024-09-13T14:43:38.057 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.057 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.057 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:43:38.057 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.953757+0000", 2024-09-13T14:43:38.058 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.005369342 2024-09-13T14:43:38.058 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.058 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.058 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:38.058 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.954003+0000", 2024-09-13T14:43:38.058 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000246457 2024-09-13T14:43:38.058 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.058 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.058 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:38.058 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.954064+0000", 2024-09-13T14:43:38.058 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.0908e-05 2024-09-13T14:43:38.058 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.058 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.058 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:38.059 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.954114+0000", 2024-09-13T14:43:38.059 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.0107e-05 2024-09-13T14:43:38.059 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.059 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.059 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:43:38.059 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.954179+0000", 2024-09-13T14:43:38.059 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.4505999999999995e-05 2024-09-13T14:43:38.059 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.059 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.059 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:43:38.059 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.954303+0000", 2024-09-13T14:43:38.059 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000124084 2024-09-13T14:43:38.059 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:43:38.059 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:43:38.060 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:43:38.060 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.060 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.060 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35379.0:76673 7.a 7:5e41d42e:::benchmark_data_smithi102_227395_object76672:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-09-13T14:43:38.060 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:36:13.187226+0000", 2024-09-13T14:43:38.060 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 444.80781464500001, 2024-09-13T14:43:38.060 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.7674439820000001, 2024-09-13T14:43:38.060 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:43:38.060 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:43:38.060 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:38.060 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:43:38.060 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35379", 2024-09-13T14:43:38.060 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:43:38.061 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 76673 2024-09-13T14:43:38.061 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.061 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:43:38.061 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.061 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:43:38.061 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.187226+0000", 2024-09-13T14:43:38.061 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:43:38.061 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.061 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.061 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:43:38.061 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.187226+0000", 2024-09-13T14:43:38.061 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:43:38.061 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.061 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.062 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:43:38.062 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.187227+0000", 2024-09-13T14:43:38.062 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.403e-06 2024-09-13T14:43:38.062 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.062 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.062 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:43:38.062 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.187249+0000", 2024-09-13T14:43:38.062 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.2436999999999999e-05 2024-09-13T14:43:38.062 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.062 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.062 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:43:38.062 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.187264+0000", 2024-09-13T14:43:38.062 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4698999999999999e-05 2024-09-13T14:43:38.062 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.063 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.063 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:43:38.063 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.187271+0000", 2024-09-13T14:43:38.063 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.2350000000000002e-06 2024-09-13T14:43:38.063 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.063 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.063 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.063 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.187331+0000", 2024-09-13T14:43:38.063 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.9843999999999998e-05 2024-09-13T14:43:38.063 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.063 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.063 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:43:38.063 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.187340+0000", 2024-09-13T14:43:38.063 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.1779999999999992e-06 2024-09-13T14:43:38.064 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.064 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.064 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.064 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.863887+0000", 2024-09-13T14:43:38.064 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.676547016 2024-09-13T14:43:38.064 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.064 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.064 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:43:38.064 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:13.863896+0000", 2024-09-13T14:43:38.064 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.6400000000000003e-06 2024-09-13T14:43:38.064 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.064 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.064 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.064 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.950494+0000", 2024-09-13T14:43:38.065 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.086598349 2024-09-13T14:43:38.065 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.065 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.065 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:43:38.065 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.950658+0000", 2024-09-13T14:43:38.065 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00016312999999999999 2024-09-13T14:43:38.065 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.065 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.065 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 0,4,5", 2024-09-13T14:43:38.065 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.950840+0000", 2024-09-13T14:43:38.065 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000182638 2024-09-13T14:43:38.065 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.065 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.065 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:43:38.066 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.954015+0000", 2024-09-13T14:43:38.066 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0031750849999999998 2024-09-13T14:43:38.066 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.066 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.066 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:38.066 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.954379+0000", 2024-09-13T14:43:38.066 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00036403800000000001 2024-09-13T14:43:38.066 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.066 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.066 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:38.066 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.954448+0000", 2024-09-13T14:43:38.066 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.8753999999999999e-05 2024-09-13T14:43:38.066 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.066 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.067 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:38.067 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.954496+0000", 2024-09-13T14:43:38.067 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.7596000000000003e-05 2024-09-13T14:43:38.067 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.067 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.067 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:43:38.067 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.954553+0000", 2024-09-13T14:43:38.067 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.7571999999999997e-05 2024-09-13T14:43:38.067 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.067 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.067 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:43:38.067 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:36:14.954670+0000", 2024-09-13T14:43:38.067 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00011636799999999999 2024-09-13T14:43:38.067 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:43:38.068 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:43:38.068 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:43:38.068 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.068 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.068 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35379.0:113983 7.3 7:c9a5e2c3:::benchmark_data_smithi102_227395_object1168:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-13T14:43:38.068 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:37:47.523718+0000", 2024-09-13T14:43:38.068 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 350.47132194900001, 2024-09-13T14:43:38.068 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.777319997, 2024-09-13T14:43:38.068 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:43:38.068 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:43:38.068 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:38.068 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:43:38.068 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35379", 2024-09-13T14:43:38.069 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:43:38.069 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 113983 2024-09-13T14:43:38.069 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.069 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:43:38.069 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.069 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:43:38.069 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:47.523718+0000", 2024-09-13T14:43:38.069 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:43:38.069 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.069 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.069 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:43:38.069 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:47.523718+0000", 2024-09-13T14:43:38.069 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:43:38.069 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.070 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.070 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:43:38.070 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:47.523720+0000", 2024-09-13T14:43:38.070 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.387e-06 2024-09-13T14:43:38.070 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.070 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.070 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:43:38.070 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:47.523725+0000", 2024-09-13T14:43:38.070 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.6699999999999999e-06 2024-09-13T14:43:38.070 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.070 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.070 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:43:38.070 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:47.523739+0000", 2024-09-13T14:43:38.070 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3465999999999999e-05 2024-09-13T14:43:38.071 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.071 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.071 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:43:38.071 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:47.523745+0000", 2024-09-13T14:43:38.071 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.6159999999999998e-06 2024-09-13T14:43:38.071 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.071 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.071 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.071 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:47.523812+0000", 2024-09-13T14:43:38.071 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.6700999999999997e-05 2024-09-13T14:43:38.071 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.071 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.071 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:43:38.072 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:47.523824+0000", 2024-09-13T14:43:38.072 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1956999999999999e-05 2024-09-13T14:43:38.072 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.072 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.072 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.072 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:48.118386+0000", 2024-09-13T14:43:38.072 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.594561487 2024-09-13T14:43:38.072 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.072 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.072 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:43:38.072 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:48.118399+0000", 2024-09-13T14:43:38.072 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2998e-05 2024-09-13T14:43:38.072 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.073 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.073 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.073 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:49.293750+0000", 2024-09-13T14:43:38.073 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1753509959999999 2024-09-13T14:43:38.073 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.073 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.073 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:43:38.073 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:49.293873+0000", 2024-09-13T14:43:38.073 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00012321700000000001 2024-09-13T14:43:38.073 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.073 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.073 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:43:38.073 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:49.293956+0000", 2024-09-13T14:43:38.073 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.2769999999999995e-05 2024-09-13T14:43:38.074 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.074 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.074 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:43:38.074 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:49.297291+0000", 2024-09-13T14:43:38.074 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0033351909999999999 2024-09-13T14:43:38.074 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.074 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.074 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:38.074 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:49.299277+0000", 2024-09-13T14:43:38.074 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0019862429999999999 2024-09-13T14:43:38.074 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.074 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.074 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:38.074 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:49.300864+0000", 2024-09-13T14:43:38.075 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0015870599999999999 2024-09-13T14:43:38.075 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.075 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.075 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:43:38.075 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:49.300924+0000", 2024-09-13T14:43:38.075 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.9969999999999997e-05 2024-09-13T14:43:38.075 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.075 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.075 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:43:38.075 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:49.301038+0000", 2024-09-13T14:43:38.075 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000114268 2024-09-13T14:43:38.075 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:43:38.075 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:43:38.075 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:43:38.076 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.076 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.076 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35379.0:113985 7.7 7:ec57fab1:::benchmark_data_smithi102_227395_object1170:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-13T14:43:38.076 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:37:47.609442+0000", 2024-09-13T14:43:38.076 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 350.38559817800001, 2024-09-13T14:43:38.076 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.1115280009999999, 2024-09-13T14:43:38.076 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:43:38.076 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:43:38.076 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:38.076 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:43:38.076 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35379", 2024-09-13T14:43:38.076 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:43:38.076 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 113985 2024-09-13T14:43:38.076 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.077 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:43:38.077 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.077 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:43:38.077 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:47.609442+0000", 2024-09-13T14:43:38.077 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:43:38.077 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.077 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.077 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:43:38.077 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:47.609442+0000", 2024-09-13T14:43:38.077 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:43:38.077 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.077 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.077 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:43:38.078 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:47.609443+0000", 2024-09-13T14:43:38.078 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.393e-06 2024-09-13T14:43:38.078 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.078 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.078 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:43:38.078 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:47.609447+0000", 2024-09-13T14:43:38.078 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.7689999999999998e-06 2024-09-13T14:43:38.078 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.078 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.078 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:43:38.078 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:47.609460+0000", 2024-09-13T14:43:38.078 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2581e-05 2024-09-13T14:43:38.078 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.078 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.079 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:43:38.079 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:47.609465+0000", 2024-09-13T14:43:38.079 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.412e-06 2024-09-13T14:43:38.079 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.079 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.079 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.079 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:47.609522+0000", 2024-09-13T14:43:38.079 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.6518999999999999e-05 2024-09-13T14:43:38.079 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.079 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.079 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:43:38.079 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:47.609530+0000", 2024-09-13T14:43:38.079 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.5259999999999993e-06 2024-09-13T14:43:38.079 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.080 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.080 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.080 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:48.121514+0000", 2024-09-13T14:43:38.080 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.51198324299999998 2024-09-13T14:43:38.080 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.080 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.080 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:43:38.080 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:48.121525+0000", 2024-09-13T14:43:38.080 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1493e-05 2024-09-13T14:43:38.080 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.080 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.080 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.080 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:49.698236+0000", 2024-09-13T14:43:38.080 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.5767113940000002 2024-09-13T14:43:38.081 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.081 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.081 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:43:38.081 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:49.698338+0000", 2024-09-13T14:43:38.081 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000101347 2024-09-13T14:43:38.081 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.081 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.081 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:43:38.081 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:49.698439+0000", 2024-09-13T14:43:38.081 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000101585 2024-09-13T14:43:38.081 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.081 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.081 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:43:38.081 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:49.703931+0000", 2024-09-13T14:43:38.082 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0054914780000000002 2024-09-13T14:43:38.082 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.082 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.082 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:38.082 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:49.712262+0000", 2024-09-13T14:43:38.082 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0083308419999999998 2024-09-13T14:43:38.082 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.082 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.082 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:38.082 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:38.084 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:49.718910+0000", 2024-09-13T14:43:38.084 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0066487100000000004 2024-09-13T14:43:38.084 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.084 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.084 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:43:38.084 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:49.718974+0000", 2024-09-13T14:43:38.084 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.3274000000000004e-05 2024-09-13T14:43:38.084 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.084 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.084 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:43:38.085 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:49.720970+0000", 2024-09-13T14:43:38.085 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.001996435 2024-09-13T14:43:38.085 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:43:38.085 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:43:38.085 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:43:38.085 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.085 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.085 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35379.0:113989 7.2 7:4d58294e:::benchmark_data_smithi102_227395_object1174:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-13T14:43:38.085 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:37:48.033615+0000", 2024-09-13T14:43:38.085 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 349.96142483800003, 2024-09-13T14:43:38.085 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.7111446429999999, 2024-09-13T14:43:38.085 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:43:38.085 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:43:38.086 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:38.086 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:43:38.086 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35379", 2024-09-13T14:43:38.086 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:43:38.086 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 113989 2024-09-13T14:43:38.086 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.086 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:43:38.086 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.086 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:43:38.086 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:48.033615+0000", 2024-09-13T14:43:38.086 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:43:38.086 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.086 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.086 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:43:38.087 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:48.033615+0000", 2024-09-13T14:43:38.087 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:43:38.087 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.087 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.087 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:43:38.087 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:48.033617+0000", 2024-09-13T14:43:38.087 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.533e-06 2024-09-13T14:43:38.087 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.087 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.087 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:43:38.087 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:48.033623+0000", 2024-09-13T14:43:38.087 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.5779999999999998e-06 2024-09-13T14:43:38.087 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.087 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.088 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:43:38.088 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:48.033637+0000", 2024-09-13T14:43:38.088 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4588e-05 2024-09-13T14:43:38.088 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.088 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.088 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:43:38.088 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:48.033643+0000", 2024-09-13T14:43:38.088 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.2349999999999996e-06 2024-09-13T14:43:38.088 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.088 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.088 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.088 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:48.033703+0000", 2024-09-13T14:43:38.088 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.9756000000000002e-05 2024-09-13T14:43:38.089 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.089 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.089 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:43:38.089 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:48.033711+0000", 2024-09-13T14:43:38.089 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.554e-06 2024-09-13T14:43:38.089 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.089 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.089 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.089 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:48.125812+0000", 2024-09-13T14:43:38.089 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.092101059999999998 2024-09-13T14:43:38.089 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.089 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.089 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:43:38.089 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:48.125823+0000", 2024-09-13T14:43:38.090 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1141000000000001e-05 2024-09-13T14:43:38.090 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.090 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.090 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.090 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:49.700174+0000", 2024-09-13T14:43:38.090 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.574350828 2024-09-13T14:43:38.090 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.090 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.090 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:43:38.090 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:49.700291+0000", 2024-09-13T14:43:38.090 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00011712 2024-09-13T14:43:38.090 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.090 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.091 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:43:38.091 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:49.700396+0000", 2024-09-13T14:43:38.091 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000105586 2024-09-13T14:43:38.091 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.091 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.091 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:43:38.091 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:49.708636+0000", 2024-09-13T14:43:38.091 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0082397430000000008 2024-09-13T14:43:38.091 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.091 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.091 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:38.091 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:49.712242+0000", 2024-09-13T14:43:38.091 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0036055179999999998 2024-09-13T14:43:38.091 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.092 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.092 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:38.092 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:49.742705+0000", 2024-09-13T14:43:38.092 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.030463707 2024-09-13T14:43:38.092 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.092 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.092 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:43:38.092 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:49.742771+0000", 2024-09-13T14:43:38.092 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.5347999999999998e-05 2024-09-13T14:43:38.092 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.092 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.092 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:43:38.092 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:49.744760+0000", 2024-09-13T14:43:38.092 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0019893480000000002 2024-09-13T14:43:38.093 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:43:38.093 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:43:38.093 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:43:38.093 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.093 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.093 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35379.0:114910 7.6 7:6e389841:::benchmark_data_smithi102_227395_object2095:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e242)", 2024-09-13T14:43:38.093 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:37:56.749897+0000", 2024-09-13T14:43:38.093 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 341.24514345, 2024-09-13T14:43:38.093 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.6553098880000001, 2024-09-13T14:43:38.093 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:43:38.093 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:43:38.093 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:38.093 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:43:38.093 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35379", 2024-09-13T14:43:38.094 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:43:38.094 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 114910 2024-09-13T14:43:38.094 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.094 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:43:38.094 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.094 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:43:38.094 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:56.749897+0000", 2024-09-13T14:43:38.094 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:43:38.094 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.094 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.094 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:43:38.094 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:56.749897+0000", 2024-09-13T14:43:38.094 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:43:38.094 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.094 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.095 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:43:38.095 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:56.749898+0000", 2024-09-13T14:43:38.095 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.539e-06 2024-09-13T14:43:38.095 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.095 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.095 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:43:38.095 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:56.749903+0000", 2024-09-13T14:43:38.095 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.9189999999999998e-06 2024-09-13T14:43:38.095 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.095 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.095 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:43:38.095 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:56.749915+0000", 2024-09-13T14:43:38.095 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2045000000000001e-05 2024-09-13T14:43:38.095 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.096 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.096 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:43:38.096 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:56.749922+0000", 2024-09-13T14:43:38.096 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.4019999999999999e-06 2024-09-13T14:43:38.096 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.096 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.096 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.096 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:56.764462+0000", 2024-09-13T14:43:38.096 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.014540712000000001 2024-09-13T14:43:38.096 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.096 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.096 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for degraded object", 2024-09-13T14:43:38.096 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:56.764528+0000", 2024-09-13T14:43:38.096 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.5779999999999997e-05 2024-09-13T14:43:38.097 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.097 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.097 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.097 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:58.251470+0000", 2024-09-13T14:43:38.097 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4869421039999999 2024-09-13T14:43:38.097 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.097 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.097 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:43:38.097 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:58.251532+0000", 2024-09-13T14:43:38.097 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.1839000000000002e-05 2024-09-13T14:43:38.097 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.097 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.097 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,5", 2024-09-13T14:43:38.097 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:58.251620+0000", 2024-09-13T14:43:38.098 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.7398999999999994e-05 2024-09-13T14:43:38.098 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.098 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.098 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:38.098 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:58.259370+0000", 2024-09-13T14:43:38.098 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0077499329999999996 2024-09-13T14:43:38.098 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.098 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.098 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:43:38.098 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:58.263582+0000", 2024-09-13T14:43:38.098 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0042128959999999998 2024-09-13T14:43:38.098 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.098 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.099 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:38.099 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:58.400821+0000", 2024-09-13T14:43:38.099 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.13723897400000001 2024-09-13T14:43:38.099 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.099 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.099 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:43:38.099 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:58.400864+0000", 2024-09-13T14:43:38.099 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.3028000000000002e-05 2024-09-13T14:43:38.099 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.099 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.099 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:43:38.099 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:58.405207+0000", 2024-09-13T14:43:38.099 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0043423180000000004 2024-09-13T14:43:38.099 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:43:38.100 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:43:38.100 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:43:38.100 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.100 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.100 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35379.0:114934 7.6 7:6d27adcb:::benchmark_data_smithi102_227395_object2119:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e242)", 2024-09-13T14:43:38.100 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:37:56.805710+0000", 2024-09-13T14:43:38.100 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 341.18933047500002, 2024-09-13T14:43:38.100 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.6124777610000001, 2024-09-13T14:43:38.100 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:43:38.100 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:43:38.100 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:38.100 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:43:38.100 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35379", 2024-09-13T14:43:38.100 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:43:38.101 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 114934 2024-09-13T14:43:38.101 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.101 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:43:38.101 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.101 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:43:38.101 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:56.805710+0000", 2024-09-13T14:43:38.101 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:43:38.101 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.101 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.101 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:43:38.101 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:56.805710+0000", 2024-09-13T14:43:38.101 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:43:38.101 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.101 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.102 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:43:38.102 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:56.805710+0000", 2024-09-13T14:43:38.102 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.2e-07 2024-09-13T14:43:38.102 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.102 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.102 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:43:38.102 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:56.805713+0000", 2024-09-13T14:43:38.102 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.6690000000000002e-06 2024-09-13T14:43:38.102 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.102 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.102 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:43:38.102 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:56.805721+0000", 2024-09-13T14:43:38.102 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.8099999999999998e-06 2024-09-13T14:43:38.103 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.103 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.103 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:43:38.103 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:56.805726+0000", 2024-09-13T14:43:38.103 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.3360000000000002e-06 2024-09-13T14:43:38.103 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.103 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.103 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.103 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:56.805777+0000", 2024-09-13T14:43:38.103 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.0581000000000002e-05 2024-09-13T14:43:38.103 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.103 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.103 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for degraded object", 2024-09-13T14:43:38.103 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:56.805846+0000", 2024-09-13T14:43:38.104 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.9467000000000004e-05 2024-09-13T14:43:38.104 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.104 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.104 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.104 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:58.257491+0000", 2024-09-13T14:43:38.104 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4516446700000001 2024-09-13T14:43:38.104 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.104 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.104 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:43:38.104 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:58.257550+0000", 2024-09-13T14:43:38.104 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.9361000000000003e-05 2024-09-13T14:43:38.104 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.104 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.105 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,5", 2024-09-13T14:43:38.105 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:58.257633+0000", 2024-09-13T14:43:38.105 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.2905000000000007e-05 2024-09-13T14:43:38.105 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.105 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.105 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:38.105 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:58.265538+0000", 2024-09-13T14:43:38.105 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0079044790000000007 2024-09-13T14:43:38.105 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.105 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.105 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:43:38.105 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:58.265549+0000", 2024-09-13T14:43:38.105 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1685000000000001e-05 2024-09-13T14:43:38.105 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.106 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.106 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:38.106 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:58.413171+0000", 2024-09-13T14:43:38.106 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.14762181399999999 2024-09-13T14:43:38.106 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.106 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.106 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:43:38.106 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:58.413233+0000", 2024-09-13T14:43:38.106 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.2069999999999994e-05 2024-09-13T14:43:38.106 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.106 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.106 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:43:38.106 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:58.418188+0000", 2024-09-13T14:43:38.106 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.004954394 2024-09-13T14:43:38.107 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:43:38.107 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:43:38.107 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:43:38.107 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.107 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.107 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35379.0:114936 7.6 7:6fc4c4cd:::benchmark_data_smithi102_227395_object2121:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e242)", 2024-09-13T14:43:38.107 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:37:56.813162+0000", 2024-09-13T14:43:38.107 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 341.18187816199998, 2024-09-13T14:43:38.107 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.6096162039999999, 2024-09-13T14:43:38.107 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:43:38.107 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:43:38.107 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:38.107 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:43:38.107 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35379", 2024-09-13T14:43:38.108 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:43:38.108 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 114936 2024-09-13T14:43:38.108 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.108 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:43:38.108 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.108 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:43:38.108 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:56.813162+0000", 2024-09-13T14:43:38.108 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:43:38.108 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.108 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.108 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:43:38.108 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:56.813162+0000", 2024-09-13T14:43:38.108 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:43:38.109 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.109 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.109 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:43:38.109 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:56.813163+0000", 2024-09-13T14:43:38.109 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.8500000000000001e-07 2024-09-13T14:43:38.109 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.109 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.109 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:43:38.109 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:56.813165+0000", 2024-09-13T14:43:38.109 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.5600000000000001e-06 2024-09-13T14:43:38.109 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.109 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.109 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:43:38.109 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:56.813173+0000", 2024-09-13T14:43:38.110 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.4429999999999997e-06 2024-09-13T14:43:38.110 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.110 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.110 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:43:38.110 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:56.813177+0000", 2024-09-13T14:43:38.110 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.2080000000000002e-06 2024-09-13T14:43:38.110 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.110 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.110 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.110 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:56.813218+0000", 2024-09-13T14:43:38.110 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.1498000000000001e-05 2024-09-13T14:43:38.110 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.110 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.110 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for degraded object", 2024-09-13T14:43:38.110 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:56.813267+0000", 2024-09-13T14:43:38.111 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.8492000000000003e-05 2024-09-13T14:43:38.111 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.111 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.111 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.111 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:58.263613+0000", 2024-09-13T14:43:38.111 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.450346073 2024-09-13T14:43:38.111 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.111 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.111 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:43:38.111 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:58.263679+0000", 2024-09-13T14:43:38.111 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.6354999999999997e-05 2024-09-13T14:43:38.111 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.111 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.112 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,5", 2024-09-13T14:43:38.112 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:58.263765+0000", 2024-09-13T14:43:38.112 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.5976000000000005e-05 2024-09-13T14:43:38.112 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.112 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.112 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:38.112 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:58.271875+0000", 2024-09-13T14:43:38.112 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0081095620000000007 2024-09-13T14:43:38.112 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.112 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.112 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:43:38.112 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:58.276115+0000", 2024-09-13T14:43:38.112 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0042398619999999996 2024-09-13T14:43:38.113 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.113 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.113 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:38.113 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:58.418257+0000", 2024-09-13T14:43:38.113 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.142142557 2024-09-13T14:43:38.113 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.113 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.113 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:43:38.113 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:58.418327+0000", 2024-09-13T14:43:38.113 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.9704999999999998e-05 2024-09-13T14:43:38.113 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.113 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.113 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:43:38.113 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:58.422778+0000", 2024-09-13T14:43:38.114 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0044513280000000001 2024-09-13T14:43:38.114 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:43:38.114 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:43:38.114 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:43:38.114 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.114 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.114 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35379.0:114951 7.6 7:6de7ffc4:::benchmark_data_smithi102_227395_object2136:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e242)", 2024-09-13T14:43:38.114 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:37:56.914361+0000", 2024-09-13T14:43:38.114 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 341.08067905399997, 2024-09-13T14:43:38.114 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.575557248, 2024-09-13T14:43:38.114 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:43:38.114 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:43:38.114 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:38.115 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:43:38.115 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35379", 2024-09-13T14:43:38.115 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:43:38.115 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 114951 2024-09-13T14:43:38.115 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.115 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:43:38.115 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.115 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:43:38.115 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:56.914361+0000", 2024-09-13T14:43:38.115 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:43:38.115 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.115 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.115 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:43:38.115 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:56.914361+0000", 2024-09-13T14:43:38.115 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:43:38.116 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.116 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.116 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:43:38.116 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:56.914363+0000", 2024-09-13T14:43:38.116 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3999999999999999e-06 2024-09-13T14:43:38.116 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.116 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.116 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:43:38.116 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:56.914368+0000", 2024-09-13T14:43:38.116 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.8370000000000002e-06 2024-09-13T14:43:38.116 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.117 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.117 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:43:38.117 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:56.914380+0000", 2024-09-13T14:43:38.117 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1749000000000001e-05 2024-09-13T14:43:38.117 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.117 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.117 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:43:38.117 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:56.914386+0000", 2024-09-13T14:43:38.117 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.0839999999999997e-06 2024-09-13T14:43:38.117 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.117 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.117 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.117 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:56.920171+0000", 2024-09-13T14:43:38.117 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0057844280000000003 2024-09-13T14:43:38.118 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.118 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.118 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for degraded object", 2024-09-13T14:43:38.118 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:56.920230+0000", 2024-09-13T14:43:38.118 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.9209000000000002e-05 2024-09-13T14:43:38.118 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.118 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.118 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.118 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:58.269910+0000", 2024-09-13T14:43:38.118 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.349679973 2024-09-13T14:43:38.118 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.118 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.118 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:43:38.118 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:58.269974+0000", 2024-09-13T14:43:38.119 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.3688000000000004e-05 2024-09-13T14:43:38.119 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.119 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.119 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,5", 2024-09-13T14:43:38.119 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:58.270061+0000", 2024-09-13T14:43:38.119 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.7534000000000006e-05 2024-09-13T14:43:38.119 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.119 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.119 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:38.119 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:58.278118+0000", 2024-09-13T14:43:38.119 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0080570920000000001 2024-09-13T14:43:38.119 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.119 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.120 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:43:38.120 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:58.278130+0000", 2024-09-13T14:43:38.120 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2201000000000001e-05 2024-09-13T14:43:38.120 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.120 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.120 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:38.120 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:58.486305+0000", 2024-09-13T14:43:38.120 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.20817459799999999 2024-09-13T14:43:38.120 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.120 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.120 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:43:38.120 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:58.486341+0000", 2024-09-13T14:43:38.120 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.5506000000000002e-05 2024-09-13T14:43:38.120 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.121 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.121 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:43:38.121 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:58.489918+0000", 2024-09-13T14:43:38.121 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.003577949 2024-09-13T14:43:38.121 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:43:38.121 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:43:38.121 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:43:38.121 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.121 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.121 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35379.0:115018 7.6 7:65c2675d:::benchmark_data_smithi102_227395_object2203:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e242)", 2024-09-13T14:43:38.121 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:37:58.188828+0000", 2024-09-13T14:43:38.121 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 339.80621199299998, 2024-09-13T14:43:38.121 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.5236193579999999, 2024-09-13T14:43:38.121 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:43:38.121 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:43:38.122 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:38.122 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:43:38.122 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35379", 2024-09-13T14:43:38.122 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:43:38.122 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 115018 2024-09-13T14:43:38.122 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.122 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:43:38.122 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.122 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:43:38.122 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:58.188828+0000", 2024-09-13T14:43:38.122 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:43:38.122 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.122 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.122 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:43:38.123 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:58.188828+0000", 2024-09-13T14:43:38.123 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:43:38.123 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.123 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.123 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:43:38.123 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:58.188830+0000", 2024-09-13T14:43:38.123 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.6339999999999999e-06 2024-09-13T14:43:38.123 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.123 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.123 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:43:38.123 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:58.188835+0000", 2024-09-13T14:43:38.123 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.0350000000000001e-06 2024-09-13T14:43:38.123 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.123 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.124 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:43:38.124 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:58.188847+0000", 2024-09-13T14:43:38.124 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1798e-05 2024-09-13T14:43:38.124 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.124 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.124 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:43:38.124 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:58.188865+0000", 2024-09-13T14:43:38.124 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.7915999999999999e-05 2024-09-13T14:43:38.124 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.124 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.124 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.124 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:58.198710+0000", 2024-09-13T14:43:38.124 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0098455409999999993 2024-09-13T14:43:38.124 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.125 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.125 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for degraded object", 2024-09-13T14:43:38.125 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:58.198751+0000", 2024-09-13T14:43:38.125 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.0748999999999999e-05 2024-09-13T14:43:38.125 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.125 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.125 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.125 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:59.649104+0000", 2024-09-13T14:43:38.125 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4503528160000001 2024-09-13T14:43:38.125 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.125 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.125 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:43:38.125 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:59.649173+0000", 2024-09-13T14:43:38.125 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.9197999999999997e-05 2024-09-13T14:43:38.126 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.126 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.126 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,5", 2024-09-13T14:43:38.126 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:59.649264+0000", 2024-09-13T14:43:38.126 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.1175999999999996e-05 2024-09-13T14:43:38.126 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.126 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.126 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:43:38.126 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:59.658739+0000", 2024-09-13T14:43:38.126 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0094745669999999997 2024-09-13T14:43:38.126 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.126 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.126 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:38.127 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:59.658776+0000", 2024-09-13T14:43:38.127 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.7518000000000002e-05 2024-09-13T14:43:38.127 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.127 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.127 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:38.127 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:59.709970+0000", 2024-09-13T14:43:38.127 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.051193434000000003 2024-09-13T14:43:38.127 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.127 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.127 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:43:38.127 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:59.710007+0000", 2024-09-13T14:43:38.127 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.7812999999999999e-05 2024-09-13T14:43:38.127 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.127 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.128 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:43:38.128 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:37:59.712448+0000", 2024-09-13T14:43:38.128 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.002440163 2024-09-13T14:43:38.128 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:43:38.128 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:43:38.128 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:43:38.128 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.128 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.128 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35379.0:115252 7.6 7:6500c2c6:::benchmark_data_smithi102_227395_object2437:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e242)", 2024-09-13T14:43:38.128 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:38:00.015728+0000", 2024-09-13T14:43:38.128 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 337.97931207300002, 2024-09-13T14:43:38.128 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.5060196480000001, 2024-09-13T14:43:38.128 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:43:38.129 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:43:38.129 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:38.129 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:43:38.129 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35379", 2024-09-13T14:43:38.129 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:43:38.129 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 115252 2024-09-13T14:43:38.129 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.129 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:43:38.129 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.129 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:43:38.129 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:00.015728+0000", 2024-09-13T14:43:38.129 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:43:38.129 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.129 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.130 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:43:38.130 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:00.015728+0000", 2024-09-13T14:43:38.130 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:43:38.130 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.130 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.130 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:43:38.130 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:00.015731+0000", 2024-09-13T14:43:38.130 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.8260000000000001e-06 2024-09-13T14:43:38.130 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.130 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.130 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:43:38.130 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:00.015742+0000", 2024-09-13T14:43:38.130 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0550000000000001e-05 2024-09-13T14:43:38.130 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.131 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.131 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:43:38.131 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:00.015771+0000", 2024-09-13T14:43:38.131 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.9513e-05 2024-09-13T14:43:38.131 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.131 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.131 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:43:38.131 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:00.015786+0000", 2024-09-13T14:43:38.131 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4783000000000001e-05 2024-09-13T14:43:38.131 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.131 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.131 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.131 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:00.015903+0000", 2024-09-13T14:43:38.131 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00011705399999999999 2024-09-13T14:43:38.132 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.132 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.132 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for degraded object", 2024-09-13T14:43:38.132 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:00.015958+0000", 2024-09-13T14:43:38.132 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.5194e-05 2024-09-13T14:43:38.132 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.132 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.132 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.132 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:01.432417+0000", 2024-09-13T14:43:38.132 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4164584840000001 2024-09-13T14:43:38.132 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.132 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.132 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:43:38.132 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:01.432519+0000", 2024-09-13T14:43:38.133 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000102599 2024-09-13T14:43:38.133 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.133 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.133 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,5", 2024-09-13T14:43:38.133 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:01.432671+0000", 2024-09-13T14:43:38.133 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000152015 2024-09-13T14:43:38.133 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.133 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.133 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:43:38.133 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:01.443345+0000", 2024-09-13T14:43:38.133 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.010674131999999999 2024-09-13T14:43:38.133 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.133 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.133 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:38.134 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:01.443407+0000", 2024-09-13T14:43:38.134 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.1716000000000003e-05 2024-09-13T14:43:38.134 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.134 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.134 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:38.134 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:01.516765+0000", 2024-09-13T14:43:38.134 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.073358163000000004 2024-09-13T14:43:38.134 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.134 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.134 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:43:38.134 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:01.516819+0000", 2024-09-13T14:43:38.134 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.4030000000000003e-05 2024-09-13T14:43:38.134 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.134 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.135 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:43:38.135 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:01.521748+0000", 2024-09-13T14:43:38.135 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0049285889999999997 2024-09-13T14:43:38.135 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:43:38.135 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:43:38.135 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:43:38.135 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.135 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.135 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35379.0:115254 7.6 7:6b0627e8:::benchmark_data_smithi102_227395_object2439:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e242)", 2024-09-13T14:43:38.135 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:38:00.188024+0000", 2024-09-13T14:43:38.135 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 337.80701624, 2024-09-13T14:43:38.135 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.739526294, 2024-09-13T14:43:38.135 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:43:38.135 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:43:38.135 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:38.136 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:43:38.136 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35379", 2024-09-13T14:43:38.136 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:43:38.136 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 115254 2024-09-13T14:43:38.136 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.136 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:43:38.136 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.136 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:43:38.136 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:00.188024+0000", 2024-09-13T14:43:38.136 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:43:38.136 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.136 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.136 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:43:38.137 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:00.188024+0000", 2024-09-13T14:43:38.137 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:43:38.137 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.137 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.137 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:43:38.137 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:00.188025+0000", 2024-09-13T14:43:38.137 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.141e-06 2024-09-13T14:43:38.137 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.137 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.137 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:43:38.137 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:00.188030+0000", 2024-09-13T14:43:38.137 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.4769999999999997e-06 2024-09-13T14:43:38.137 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.137 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.138 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:43:38.138 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:00.188041+0000", 2024-09-13T14:43:38.138 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0896999999999999e-05 2024-09-13T14:43:38.138 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.138 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.138 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:43:38.138 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:00.188046+0000", 2024-09-13T14:43:38.138 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.288e-06 2024-09-13T14:43:38.138 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.138 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.138 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.138 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:00.193867+0000", 2024-09-13T14:43:38.138 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0058214670000000003 2024-09-13T14:43:38.138 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.138 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.139 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for degraded object", 2024-09-13T14:43:38.139 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:00.193936+0000", 2024-09-13T14:43:38.139 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.8564000000000003e-05 2024-09-13T14:43:38.139 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.139 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.139 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.139 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:01.708466+0000", 2024-09-13T14:43:38.139 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.5145303239999999 2024-09-13T14:43:38.139 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.139 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.139 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:43:38.139 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:01.708530+0000", 2024-09-13T14:43:38.139 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.3752999999999998e-05 2024-09-13T14:43:38.140 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.140 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.140 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,5", 2024-09-13T14:43:38.140 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:01.708618+0000", 2024-09-13T14:43:38.140 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.8344000000000007e-05 2024-09-13T14:43:38.140 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.140 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.140 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:43:38.140 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:01.718693+0000", 2024-09-13T14:43:38.140 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.010074408999999999 2024-09-13T14:43:38.140 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.140 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.140 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:38.140 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:01.718726+0000", 2024-09-13T14:43:38.141 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.3649000000000003e-05 2024-09-13T14:43:38.141 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.141 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.141 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:38.141 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:01.923691+0000", 2024-09-13T14:43:38.141 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.20496487299999999 2024-09-13T14:43:38.141 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.141 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.141 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:43:38.141 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:01.923731+0000", 2024-09-13T14:43:38.141 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.9657000000000003e-05 2024-09-13T14:43:38.141 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.141 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.141 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:43:38.141 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:01.927550+0000", 2024-09-13T14:43:38.142 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0038194510000000002 2024-09-13T14:43:38.142 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:43:38.142 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:43:38.142 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:43:38.142 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.142 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.142 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35379.0:115283 7.6 7:650e2755:::benchmark_data_smithi102_227395_object2468:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e242)", 2024-09-13T14:43:38.142 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:38:00.440578+0000", 2024-09-13T14:43:38.142 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 337.55446255499999, 2024-09-13T14:43:38.142 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.6250139940000001, 2024-09-13T14:43:38.142 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:43:38.142 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:43:38.142 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:38.142 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:43:38.143 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35379", 2024-09-13T14:43:38.143 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:43:38.143 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 115283 2024-09-13T14:43:38.143 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.143 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:43:38.143 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.143 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:43:38.143 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:00.440578+0000", 2024-09-13T14:43:38.143 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:43:38.143 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.143 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.143 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:43:38.143 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:00.440578+0000", 2024-09-13T14:43:38.143 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:43:38.143 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.144 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.144 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:43:38.144 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:00.440578+0000", 2024-09-13T14:43:38.144 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.4899999999999995e-07 2024-09-13T14:43:38.144 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.144 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.144 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:43:38.144 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:00.440582+0000", 2024-09-13T14:43:38.144 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.1429999999999998e-06 2024-09-13T14:43:38.144 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.144 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.144 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:43:38.144 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:00.440590+0000", 2024-09-13T14:43:38.144 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.4270000000000008e-06 2024-09-13T14:43:38.145 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.145 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.145 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:43:38.145 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:00.440595+0000", 2024-09-13T14:43:38.145 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.6650000000000002e-06 2024-09-13T14:43:38.145 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.145 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.145 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.145 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:00.440648+0000", 2024-09-13T14:43:38.145 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.2917999999999997e-05 2024-09-13T14:43:38.145 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.145 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.145 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for degraded object", 2024-09-13T14:43:38.145 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:00.440696+0000", 2024-09-13T14:43:38.146 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.8390999999999998e-05 2024-09-13T14:43:38.146 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.146 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.146 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:43:38.146 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:01.932765+0000", 2024-09-13T14:43:38.146 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4920688769999999 2024-09-13T14:43:38.146 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.146 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.146 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:43:38.146 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:01.932831+0000", 2024-09-13T14:43:38.146 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.6441999999999998e-05 2024-09-13T14:43:38.146 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.146 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.146 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,5", 2024-09-13T14:43:38.147 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:01.932941+0000", 2024-09-13T14:43:38.147 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000110022 2024-09-13T14:43:38.147 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.147 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.147 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:38.147 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:01.947299+0000", 2024-09-13T14:43:38.147 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.014357653 2024-09-13T14:43:38.147 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.147 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.147 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:43:38.147 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:01.947318+0000", 2024-09-13T14:43:38.147 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.8975999999999999e-05 2024-09-13T14:43:38.147 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.147 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.147 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:38.148 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:02.063350+0000", 2024-09-13T14:43:38.148 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.11603247 2024-09-13T14:43:38.148 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.148 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.148 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:43:38.148 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:02.063409+0000", 2024-09-13T14:43:38.148 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.8573000000000003e-05 2024-09-13T14:43:38.148 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:43:38.148 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:43:38.148 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:43:38.148 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:38:02.065592+0000", 2024-09-13T14:43:38.148 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0021827880000000002 2024-09-13T14:43:38.148 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:43:38.148 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:43:38.149 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:43:38.149 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:43:38.149 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:43:38.149 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:43:38.170 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:38.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:38.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:38.272 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:38.295 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:38.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:38.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:38.397 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:38.420 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:38.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:38.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:38.522 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:38.549 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:38.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:38.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:38.651 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:38.675 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:38.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:38.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:38.776 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:38.799 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:38.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:38.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:38.900 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:38.923 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:38.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:38.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:38.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:38 smithi195 ceph-mon[104262]: pgmap v986: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.1 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 256 op/s; 11959/151284 objects degraded (7.905%); 1987/151284 objects misplaced (1.313%); 8.4 MiB/s, 133 objects/s recovering 2024-09-13T14:43:39.024 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:39.047 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:39.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:39.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:39.101 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:38 smithi102 ceph-mon[115269]: pgmap v986: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.1 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 256 op/s; 11959/151284 objects degraded (7.905%); 1987/151284 objects misplaced (1.313%); 8.4 MiB/s, 133 objects/s recovering 2024-09-13T14:43:39.101 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:38 smithi102 ceph-mon[110183]: pgmap v986: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.1 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 256 op/s; 11959/151284 objects degraded (7.905%); 1987/151284 objects misplaced (1.313%); 8.4 MiB/s, 133 objects/s recovering 2024-09-13T14:43:39.149 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:39.175 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:39.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:39.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:39.277 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:39.304 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:39.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:39.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:39.406 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:39.428 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:39.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:39.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:39.529 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:39.554 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:39.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:39.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:39.656 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:39.681 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:39.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:39.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:39.782 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:39.807 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:39.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:39.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:39.909 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:39.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:39 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 11959/151284 objects degraded (7.905%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:43:39.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:39 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 11959/151284 objects degraded (7.905%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:43:39.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:39.932 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:39.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:39.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:39 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 11959/151284 objects degraded (7.905%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:43:40.040 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:40.065 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:40.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:40.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:40.069 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T14:43:40.167 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:40.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:40.198 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:40.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:40.299 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:40.330 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:40.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:40.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:40.338 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:43:40.432 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:40.456 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:40.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:40.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:40.558 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:40.587 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:40.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:40.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:40.689 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:40.723 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:40.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:40.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:40.825 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:40.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:40.854 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:40.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:40.954 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:40.976 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:40.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:40.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:40.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:40 smithi195 ceph-mon[104262]: pgmap v987: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.1 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 259 op/s; 11351/150942 objects degraded (7.520%); 1987/150942 objects misplaced (1.316%); 9.1 MiB/s, 145 objects/s recovering 2024-09-13T14:43:40.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:40 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:43:40.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:40 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:43:41.049 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:40 smithi102 ceph-mon[110183]: pgmap v987: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.1 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 259 op/s; 11351/150942 objects degraded (7.520%); 1987/150942 objects misplaced (1.316%); 9.1 MiB/s, 145 objects/s recovering 2024-09-13T14:43:41.049 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:40 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:43:41.049 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:40 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:43:41.050 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:40 smithi102 ceph-mon[115269]: pgmap v987: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.1 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 259 op/s; 11351/150942 objects degraded (7.520%); 1987/150942 objects misplaced (1.316%); 9.1 MiB/s, 145 objects/s recovering 2024-09-13T14:43:41.050 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:40 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:43:41.050 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:40 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:43:41.078 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:41.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:41.104 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:41.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:41.205 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:41.229 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:41.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:41.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:41.331 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:41.354 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:41.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:41.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:41.456 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:41.482 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:41.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:41.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:41.584 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:41.606 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:41.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:41.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:41.708 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:41.732 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:41.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:41.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:41.833 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:41.857 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:41.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:41.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:41.885 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:43:41.885 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:43:41.895 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:41.910 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:43:41.910 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:43:41.939 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:43:41.939 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:43:41.959 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:41.968 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:43:41.968 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:43:41.981 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:41.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:41.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:41.997 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:43:41.997 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:43:42.024 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:43:42.024 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:43:42.055 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:43:42.055 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:43:42.082 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:42.086 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:43:42.086 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:43:42.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:42.106 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:42.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:42.207 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:42.236 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:42.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:42.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:42.338 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:42.360 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:42.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:42.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:42.416 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:43:42.416 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:43:42.463 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:42.486 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:42.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:42.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:42.588 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:42.613 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:42.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:42.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:42.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:42 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:43:42.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:42 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:43:42.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:42 smithi102 ceph-mon[110183]: pgmap v988: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.1 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 223 op/s; 11351/150942 objects degraded (7.520%); 1987/150942 objects misplaced (1.316%); 8.3 MiB/s, 132 objects/s recovering 2024-09-13T14:43:42.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:42 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3696031107' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:43:42.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:42 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:43:42.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:42 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:43:42.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:42 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:43:42.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:42 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:43:42.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:42 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:43:42.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:42 smithi102 ceph-mon[115269]: pgmap v988: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.1 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 223 op/s; 11351/150942 objects degraded (7.520%); 1987/150942 objects misplaced (1.316%); 8.3 MiB/s, 132 objects/s recovering 2024-09-13T14:43:42.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:42 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3696031107' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:43:42.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:42 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:43:42.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:42 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:43:42.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:42 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:43:42.716 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:42.739 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:42.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:42.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:42.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:42 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:43:42.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:42 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:43:42.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:42 smithi195 ceph-mon[104262]: pgmap v988: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.1 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 223 op/s; 11351/150942 objects degraded (7.520%); 1987/150942 objects misplaced (1.316%); 8.3 MiB/s, 132 objects/s recovering 2024-09-13T14:43:42.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:42 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3696031107' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:43:42.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:42 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:43:42.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:42 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:43:42.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:42 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:43:42.840 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:42.862 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:42.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:42.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:42.940 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:43:42.964 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:43.029 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:43.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:43.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:43.131 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:43.160 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:43.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:43.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:43.262 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:43.288 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:43.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:43.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:43.390 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:43.415 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:43.416 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:43.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:43.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:43.518 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:43.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:43.543 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:43.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:43.645 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:43.668 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:43.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:43.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:43.770 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:43.792 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:43.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:43.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:43.894 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:43.917 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T14:43:43.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:43.918 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:43.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:44.018 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:44.041 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:44.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:44.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:44.143 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:44.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:44.169 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:44.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:44.177 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:43:44.270 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:44.294 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:44.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:44.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:44.396 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:44.421 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:44.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:44.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:44.522 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:44.548 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:44.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:44.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:44.650 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:44.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:44.673 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:44.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:44.777 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:44.801 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:44.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:44.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:44.904 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:44.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:44 smithi102 ceph-mon[110183]: pgmap v989: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.0 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 303 op/s; 10477/146694 objects degraded (7.142%); 1987/146694 objects misplaced (1.355%); 10 MiB/s, 164 objects/s recovering 2024-09-13T14:43:44.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:44 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 10477/146694 objects degraded (7.142%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:43:44.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:44 smithi102 ceph-mon[115269]: pgmap v989: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.0 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 303 op/s; 10477/146694 objects degraded (7.142%); 1987/146694 objects misplaced (1.355%); 10 MiB/s, 164 objects/s recovering 2024-09-13T14:43:44.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:44 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 10477/146694 objects degraded (7.142%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:43:44.929 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:44.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:44.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:44.969 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:43:44.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:44 smithi195 ceph-mon[104262]: pgmap v989: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.0 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 303 op/s; 10477/146694 objects degraded (7.142%); 1987/146694 objects misplaced (1.355%); 10 MiB/s, 164 objects/s recovering 2024-09-13T14:43:44.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:44 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 10477/146694 objects degraded (7.142%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:43:45.032 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:45.054 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:45.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:45.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:45.156 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:45.178 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:45.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:45.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:45.279 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:45.303 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:45.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:45.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:45.405 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:45.437 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:45.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:45.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:45.539 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:45.567 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:45.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:45.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:45.669 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:45.697 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:45.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:45.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:45.798 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:45.824 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:45.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:45.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:45.926 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:45.951 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:45.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:45.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:46.002 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:43:46.002 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:43:46.048 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:43:46.049 INFO:teuthology.orchestra.run.smithi102.stdout:{"fsid":"08cee502-71d9-11ef-bceb-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 10477/146694 objects degraded (7.142%), 5 pgs degraded, 3 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1449,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":287,"num_osds":8,"num_up_osds":8,"osd_up_since":1726238267,"num_in_osds":6,"osd_in_since":1726238173,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":84},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+recovery_wait+degraded","count":1},{"state_name":"active+recovery_wait+degraded+remapped","count":1}],"num_pgs":89,"num_pools":5,"num_objects":48898,"data_bytes":3207469053,"bytes_used":15258222592,"bytes_avail":560711991296,"bytes_total":575970213888,"degraded_objects":10477,"degraded_total":146694,"degraded_ratio":0.071420780672692821,"misplaced_objects":1987,"misplaced_total":146694,"misplaced_ratio":0.013545202939452192,"recovering_objects_per_sec":164,"recovering_bytes_per_sec":10777943,"recovering_keys_per_sec":0,"num_objects_recovered":1645,"num_bytes_recovered":107806720,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":303},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi102.njetxr","status":"up:active","gid":24521}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.102:8443/","prometheus":"http://172.21.15.102:9283/"}},"servicemap":{"epoch":28,"modified":"2024-09-13T14:40:21.476600+0000","services":{}},"progress_events":{}} 2024-09-13T14:43:46.053 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:46.077 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:46.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:46.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:46.179 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:46.204 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:46.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:46.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:46.306 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:46.331 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:46.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:46.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:46.433 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:46.459 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:46.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:46.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:46.561 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:46.587 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:46.588 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is set 2024-09-13T14:43:46.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:46.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:46.685 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T14:43:46.690 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:46.750 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:46.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:46.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:46.852 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:46.879 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:46.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:46.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:46.916 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:43:46.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:46 smithi102 ceph-mon[110183]: pgmap v990: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.0 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 303 op/s; 10477/146694 objects degraded (7.142%); 1987/146694 objects misplaced (1.355%); 10 MiB/s, 164 objects/s recovering 2024-09-13T14:43:46.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:46 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/940132559' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:43:46.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:46 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3327234' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:43:46.918 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:43:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:43:46] "GET /metrics HTTP/1.1" 200 36552 "" "Prometheus/2.51.0" 2024-09-13T14:43:46.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:46 smithi102 ceph-mon[115269]: pgmap v990: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.0 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 303 op/s; 10477/146694 objects degraded (7.142%); 1987/146694 objects misplaced (1.355%); 10 MiB/s, 164 objects/s recovering 2024-09-13T14:43:46.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:46 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/940132559' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:43:46.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:46 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3327234' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:43:46.981 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:46.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:46 smithi195 ceph-mon[104262]: pgmap v990: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.0 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 303 op/s; 10477/146694 objects degraded (7.142%); 1987/146694 objects misplaced (1.355%); 10 MiB/s, 164 objects/s recovering 2024-09-13T14:43:46.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:46 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/940132559' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:43:46.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:46 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3327234' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:43:47.003 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:47.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:47.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:47.068 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:47.105 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:47.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:47.129 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:47.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:47.230 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:47.253 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:47.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:47.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:47.355 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:47.381 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:47.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:47.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:47.483 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:47.505 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:47.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:47.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:47.607 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:47.631 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:47.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:47.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:47.733 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:47.758 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:47.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:47.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:47.855 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:47 smithi195 ceph-mon[104262]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:43:47.855 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:47 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3327234' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:43:47.855 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:47 smithi195 ceph-mon[104262]: osdmap e288: 8 total, 8 up, 6 in 2024-09-13T14:43:47.859 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:47.881 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:47.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:47.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:47.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:47 smithi102 ceph-mon[110183]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:43:47.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:47 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3327234' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:43:47.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:47 smithi102 ceph-mon[110183]: osdmap e288: 8 total, 8 up, 6 in 2024-09-13T14:43:47.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:47 smithi102 ceph-mon[115269]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:43:47.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:47 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3327234' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:43:47.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:47 smithi102 ceph-mon[115269]: osdmap e288: 8 total, 8 up, 6 in 2024-09-13T14:43:47.983 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:48.006 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:48.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:48.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:48.108 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:48.131 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:48.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:48.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:48.233 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:48.257 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:48.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:48.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:48.359 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:48.383 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:48.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:48.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:48.485 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:48.508 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:48.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:48.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:48.610 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:48.637 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:48.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:48.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:48.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:48 smithi102 ceph-mon[110183]: pgmap v992: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 2.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 300 op/s; 10299/142260 objects degraded (7.240%); 1987/142260 objects misplaced (1.397%); 10 MiB/s, 165 objects/s recovering 2024-09-13T14:43:48.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:48 smithi102 ceph-mon[115269]: pgmap v992: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 2.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 300 op/s; 10299/142260 objects degraded (7.240%); 1987/142260 objects misplaced (1.397%); 10 MiB/s, 165 objects/s recovering 2024-09-13T14:43:48.739 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:48.761 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:48.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:48.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:48.863 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:48.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:48.885 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:48.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:48.948 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:48 smithi195 ceph-mon[104262]: pgmap v992: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 2.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 300 op/s; 10299/142260 objects degraded (7.240%); 1987/142260 objects misplaced (1.397%); 10 MiB/s, 165 objects/s recovering 2024-09-13T14:43:48.986 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:49.007 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:49.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:49.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:49.109 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:49.132 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:49.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:49.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:49.233 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:49.255 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:49.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:49.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:49.357 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:49.381 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:49.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:49.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:49.484 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:49.507 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:49.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:49.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:49.551 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-13T14:43:49.551 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:43:49.609 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:49.632 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:49.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:49.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:49.734 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:49.758 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:49.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:49.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:49.773 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:49 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 10299/142260 objects degraded (7.240%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:43:49.773 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:49 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 10299/142260 objects degraded (7.240%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:43:49.860 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:49.884 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:49.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:49.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:49.905 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:49 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 10299/142260 objects degraded (7.240%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:43:49.986 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:50.008 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:50.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:50.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:50.015 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:43:50.015 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:43:50.033 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:43:50.033 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:43:50.049 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:50.051 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:43:50.052 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:43:50.081 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:43:50.081 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:43:50.098 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:43:50.099 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:43:50.110 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:50.114 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:43:50.115 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:43:50.131 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:43:50.131 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:43:50.132 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:50.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:50.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:50.147 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:43:50.147 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:43:50.234 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:50.255 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:50.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:50.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:50.357 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:50.385 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:50.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:50.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:50.487 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:50.509 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:50.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:50.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:50.612 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:50.635 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:50.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:50.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:50.737 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:50.759 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:50.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:50.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:50.860 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:50.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:50.884 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:50.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:50.888 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:50 smithi195 ceph-mon[104262]: pgmap v993: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 2.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 299 op/s; 9421/141954 objects degraded (6.637%); 1987/141954 objects misplaced (1.400%); 11 MiB/s, 199 objects/s recovering 2024-09-13T14:43:50.888 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:50 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1200711991' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:43:50.917 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:43:50.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:50 smithi102 ceph-mon[115269]: pgmap v993: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 2.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 299 op/s; 9421/141954 objects degraded (6.637%); 1987/141954 objects misplaced (1.400%); 11 MiB/s, 199 objects/s recovering 2024-09-13T14:43:50.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:50 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1200711991' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:43:50.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:50 smithi102 ceph-mon[110183]: pgmap v993: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 2.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 299 op/s; 9421/141954 objects degraded (6.637%); 1987/141954 objects misplaced (1.400%); 11 MiB/s, 199 objects/s recovering 2024-09-13T14:43:50.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:50 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1200711991' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:43:50.985 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:51.008 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:51.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:51.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:51.110 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:51.132 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:51.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:51.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:51.235 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:51.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:51.257 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:51.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:51.358 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:51.381 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:51.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:51.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:51.483 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:51.506 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:51.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:51.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:51.608 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:51.631 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:51.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:51.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:51.732 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:43:51.732 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-09-13T14:43:51.732 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-09-13T14:43:51.733 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-13T14:43:51.733 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.733 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:191635 6.e 6:71f9b24e:::benchmark_data_smithi102_194590_object81011:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-09-13T14:43:51.733 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:34:22.367198+0000", 2024-09-13T14:43:51.733 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 568.519682999, 2024-09-13T14:43:51.733 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.26435987, 2024-09-13T14:43:51.733 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:43:51.733 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:51.733 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:43:51.733 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:43:51.733 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:43:51.733 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 191635 2024-09-13T14:43:51.734 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.734 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:43:51.734 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.734 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:43:51.734 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.367198+0000", 2024-09-13T14:43:51.734 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:43:51.734 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.734 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.734 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:43:51.734 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.367198+0000", 2024-09-13T14:43:51.734 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:43:51.734 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.734 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.735 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:43:51.735 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.367197+0000", 2024-09-13T14:43:51.735 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:43:51.735 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.735 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.735 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:43:51.735 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.367203+0000", 2024-09-13T14:43:51.735 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.455e-06 2024-09-13T14:43:51.736 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.736 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.736 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:43:51.736 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.367222+0000", 2024-09-13T14:43:51.736 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8655e-05 2024-09-13T14:43:51.736 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.736 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.736 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:43:51.736 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.367230+0000", 2024-09-13T14:43:51.736 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7379999999999994e-06 2024-09-13T14:43:51.736 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.736 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.737 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:43:51.737 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.367927+0000", 2024-09-13T14:43:51.737 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00069707899999999997 2024-09-13T14:43:51.737 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.737 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.737 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:43:51.737 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.368049+0000", 2024-09-13T14:43:51.737 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000122211 2024-09-13T14:43:51.737 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.737 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.737 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6,7", 2024-09-13T14:43:51.737 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.368168+0000", 2024-09-13T14:43:51.737 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000119357 2024-09-13T14:43:51.738 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.738 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.738 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:43:51.738 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.371968+0000", 2024-09-13T14:43:51.738 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003800159 2024-09-13T14:43:51.738 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.738 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.738 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:51.738 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.380532+0000", 2024-09-13T14:43:51.738 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0085635420000000004 2024-09-13T14:43:51.738 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.738 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.738 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:51.739 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.380575+0000", 2024-09-13T14:43:51.739 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3392000000000001e-05 2024-09-13T14:43:51.739 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.739 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.739 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:51.739 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.631340+0000", 2024-09-13T14:43:51.739 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.250764879 2024-09-13T14:43:51.739 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.739 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.739 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:43:51.739 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.631451+0000", 2024-09-13T14:43:51.739 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000111078 2024-09-13T14:43:51.739 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.740 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.740 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:43:51.740 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.631558+0000", 2024-09-13T14:43:51.740 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010673799999999999 2024-09-13T14:43:51.740 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:43:51.740 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:43:51.740 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:43:51.740 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.740 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.740 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:191643 6.e 6:7b30bcbb:::benchmark_data_smithi102_194590_object81019:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-09-13T14:43:51.740 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:34:22.372207+0000", 2024-09-13T14:43:51.740 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 568.51467427099999, 2024-09-13T14:43:51.741 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.26177498799999999, 2024-09-13T14:43:51.741 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:43:51.741 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:51.741 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:43:51.741 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:43:51.741 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:43:51.741 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 191643 2024-09-13T14:43:51.741 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.741 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:43:51.742 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.742 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:43:51.742 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.372207+0000", 2024-09-13T14:43:51.742 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:43:51.742 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.742 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.742 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:43:51.742 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.372207+0000", 2024-09-13T14:43:51.742 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:43:51.743 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.743 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.743 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:43:51.743 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.372205+0000", 2024-09-13T14:43:51.743 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:43:51.743 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.743 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.743 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:43:51.743 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.372213+0000", 2024-09-13T14:43:51.743 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1400000000000002e-06 2024-09-13T14:43:51.743 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.743 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.743 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:43:51.743 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.372225+0000", 2024-09-13T14:43:51.744 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2577e-05 2024-09-13T14:43:51.744 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.744 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.744 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:43:51.744 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.372232+0000", 2024-09-13T14:43:51.744 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7139999999999996e-06 2024-09-13T14:43:51.744 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.744 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.744 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:43:51.744 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.372269+0000", 2024-09-13T14:43:51.744 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7327999999999999e-05 2024-09-13T14:43:51.744 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.744 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.745 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:43:51.745 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.372400+0000", 2024-09-13T14:43:51.745 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000131085 2024-09-13T14:43:51.745 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.745 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.745 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6,7", 2024-09-13T14:43:51.745 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.372529+0000", 2024-09-13T14:43:51.745 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012839700000000001 2024-09-13T14:43:51.745 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.745 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.745 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:43:51.745 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.380540+0000", 2024-09-13T14:43:51.745 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0080113100000000007 2024-09-13T14:43:51.746 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.746 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.746 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:51.746 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.380608+0000", 2024-09-13T14:43:51.746 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7725000000000006e-05 2024-09-13T14:43:51.746 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.746 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.746 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:51.746 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.380655+0000", 2024-09-13T14:43:51.746 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7085000000000001e-05 2024-09-13T14:43:51.747 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.747 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.747 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:51.747 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.631591+0000", 2024-09-13T14:43:51.747 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.25093650699999998 2024-09-13T14:43:51.747 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.747 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.747 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:43:51.747 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.633854+0000", 2024-09-13T14:43:51.748 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0022624239999999999 2024-09-13T14:43:51.748 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.748 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.748 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:43:51.748 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.633982+0000", 2024-09-13T14:43:51.748 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012812899999999999 2024-09-13T14:43:51.748 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:43:51.748 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:43:51.748 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:43:51.749 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.749 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.749 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:191647 6.e 6:76d424e6:::benchmark_data_smithi102_194590_object81023:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-09-13T14:43:51.749 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:34:22.373106+0000", 2024-09-13T14:43:51.749 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 568.51377480300005, 2024-09-13T14:43:51.749 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.26104791700000002, 2024-09-13T14:43:51.749 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:43:51.749 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:51.750 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:43:51.750 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:43:51.750 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:43:51.750 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 191647 2024-09-13T14:43:51.750 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.750 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:43:51.751 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.751 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:43:51.751 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.373106+0000", 2024-09-13T14:43:51.751 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:43:51.751 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.751 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.751 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:43:51.751 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.373106+0000", 2024-09-13T14:43:51.751 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:43:51.751 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.751 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.751 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:43:51.751 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.373105+0000", 2024-09-13T14:43:51.752 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:43:51.752 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.752 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.752 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:43:51.752 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.373110+0000", 2024-09-13T14:43:51.752 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5220000000000001e-06 2024-09-13T14:43:51.752 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.752 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.752 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:43:51.752 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.373134+0000", 2024-09-13T14:43:51.752 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4409e-05 2024-09-13T14:43:51.752 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.752 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.753 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:43:51.753 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.373141+0000", 2024-09-13T14:43:51.753 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4860000000000002e-06 2024-09-13T14:43:51.753 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.753 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.753 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:43:51.753 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.377126+0000", 2024-09-13T14:43:51.753 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003985321 2024-09-13T14:43:51.753 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.753 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.753 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:43:51.753 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.377248+0000", 2024-09-13T14:43:51.753 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012149199999999999 2024-09-13T14:43:51.753 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.754 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.754 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6,7", 2024-09-13T14:43:51.754 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.377456+0000", 2024-09-13T14:43:51.754 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00020867600000000001 2024-09-13T14:43:51.754 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.754 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.754 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:43:51.754 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.381454+0000", 2024-09-13T14:43:51.754 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0039976029999999997 2024-09-13T14:43:51.754 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.754 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.754 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:51.754 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.382225+0000", 2024-09-13T14:43:51.755 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00077125399999999997 2024-09-13T14:43:51.755 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.755 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.755 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:51.755 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.383736+0000", 2024-09-13T14:43:51.755 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0015105279999999999 2024-09-13T14:43:51.755 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.755 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.755 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:51.755 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.634027+0000", 2024-09-13T14:43:51.755 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.25029173100000002 2024-09-13T14:43:51.755 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.755 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.756 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:43:51.756 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.634065+0000", 2024-09-13T14:43:51.756 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7135000000000003e-05 2024-09-13T14:43:51.756 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.756 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.756 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:43:51.756 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.634154+0000", 2024-09-13T14:43:51.756 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.9624999999999997e-05 2024-09-13T14:43:51.756 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:43:51.757 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:43:51.757 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:43:51.757 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.757 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.757 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:191656 6.e 6:73cf8b7b:::benchmark_data_smithi102_194590_object81032:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-09-13T14:43:51.757 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:34:22.389704+0000", 2024-09-13T14:43:51.757 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 568.49717734800004, 2024-09-13T14:43:51.757 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.24469366300000001, 2024-09-13T14:43:51.757 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:43:51.757 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:51.757 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:43:51.758 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:43:51.758 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:43:51.758 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 191656 2024-09-13T14:43:51.758 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.758 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:43:51.758 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.758 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:43:51.758 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.389704+0000", 2024-09-13T14:43:51.758 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:43:51.758 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.758 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.758 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:43:51.758 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.389704+0000", 2024-09-13T14:43:51.758 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:43:51.759 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.759 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.759 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:43:51.759 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.389702+0000", 2024-09-13T14:43:51.759 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:43:51.759 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.759 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.759 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:43:51.759 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.389708+0000", 2024-09-13T14:43:51.759 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2670000000000003e-06 2024-09-13T14:43:51.759 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.759 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.759 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:43:51.760 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.389723+0000", 2024-09-13T14:43:51.760 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4175000000000001e-05 2024-09-13T14:43:51.760 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.760 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.760 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:43:51.760 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.389730+0000", 2024-09-13T14:43:51.760 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5480000000000004e-06 2024-09-13T14:43:51.760 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.760 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.760 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:43:51.760 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.389767+0000", 2024-09-13T14:43:51.760 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7123999999999999e-05 2024-09-13T14:43:51.760 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.760 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.761 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:43:51.761 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.389868+0000", 2024-09-13T14:43:51.761 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010078599999999999 2024-09-13T14:43:51.761 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.761 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.761 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6,7", 2024-09-13T14:43:51.761 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.390004+0000", 2024-09-13T14:43:51.761 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000135578 2024-09-13T14:43:51.761 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.761 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.761 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:51.761 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.393200+0000", 2024-09-13T14:43:51.761 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0031962649999999998 2024-09-13T14:43:51.762 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.762 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.762 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:43:51.762 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.396346+0000", 2024-09-13T14:43:51.762 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003146009 2024-09-13T14:43:51.762 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.762 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.762 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:51.762 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.396394+0000", 2024-09-13T14:43:51.762 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8588000000000003e-05 2024-09-13T14:43:51.762 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.762 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.762 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:51.762 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.634261+0000", 2024-09-13T14:43:51.762 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.237866258 2024-09-13T14:43:51.763 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.763 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.763 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:43:51.763 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.634301+0000", 2024-09-13T14:43:51.763 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.019e-05 2024-09-13T14:43:51.763 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.763 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.763 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:43:51.763 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.634397+0000", 2024-09-13T14:43:51.763 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6477999999999993e-05 2024-09-13T14:43:51.763 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:43:51.763 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:43:51.763 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:43:51.763 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.764 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.764 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:191660 6.e 6:738104a9:::benchmark_data_smithi102_194590_object81036:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-09-13T14:43:51.764 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:34:22.392930+0000", 2024-09-13T14:43:51.764 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 568.49395079299995, 2024-09-13T14:43:51.764 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.24195140200000001, 2024-09-13T14:43:51.764 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:43:51.764 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:51.764 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:43:51.764 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:43:51.764 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:43:51.764 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 191660 2024-09-13T14:43:51.764 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.765 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:43:51.765 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.765 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:43:51.765 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.392930+0000", 2024-09-13T14:43:51.765 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:43:51.765 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.765 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.765 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:43:51.765 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.392930+0000", 2024-09-13T14:43:51.765 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:43:51.765 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.765 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.765 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:43:51.765 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.392929+0000", 2024-09-13T14:43:51.766 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:43:51.766 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.766 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.766 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:43:51.766 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.392935+0000", 2024-09-13T14:43:51.766 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0859999999999998e-06 2024-09-13T14:43:51.766 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.766 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.766 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:43:51.766 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.392948+0000", 2024-09-13T14:43:51.766 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2529e-05 2024-09-13T14:43:51.766 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.767 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.767 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:43:51.767 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.392954+0000", 2024-09-13T14:43:51.767 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9920000000000001e-06 2024-09-13T14:43:51.767 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.767 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.767 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:43:51.767 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.393326+0000", 2024-09-13T14:43:51.768 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00037141000000000002 2024-09-13T14:43:51.768 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.768 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.768 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:43:51.768 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.393461+0000", 2024-09-13T14:43:51.768 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013538499999999999 2024-09-13T14:43:51.768 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.768 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.768 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6,7", 2024-09-13T14:43:51.768 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.393618+0000", 2024-09-13T14:43:51.768 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000156541 2024-09-13T14:43:51.768 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.768 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.769 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:51.769 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.396432+0000", 2024-09-13T14:43:51.769 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00281445 2024-09-13T14:43:51.769 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.769 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.769 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:43:51.769 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.396443+0000", 2024-09-13T14:43:51.769 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0852999999999999e-05 2024-09-13T14:43:51.769 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.769 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.769 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:51.769 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.396763+0000", 2024-09-13T14:43:51.769 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00031982700000000001 2024-09-13T14:43:51.770 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.770 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.770 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:51.770 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.634751+0000", 2024-09-13T14:43:51.770 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.23798829299999999 2024-09-13T14:43:51.770 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.770 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.770 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:43:51.770 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.634794+0000", 2024-09-13T14:43:51.770 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2425999999999997e-05 2024-09-13T14:43:51.770 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.770 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.770 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:43:51.770 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:22.634882+0000", 2024-09-13T14:43:51.771 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8011999999999997e-05 2024-09-13T14:43:51.771 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:43:51.771 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:43:51.771 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:43:51.771 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.771 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.771 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:198028 6.e 6:71106726:::benchmark_data_smithi102_194590_object87404:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-09-13T14:43:51.771 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:34:29.683014+0000", 2024-09-13T14:43:51.771 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 561.20386673200005, 2024-09-13T14:43:51.771 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.22173479500000001, 2024-09-13T14:43:51.771 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:43:51.771 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:51.771 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:43:51.772 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:43:51.772 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:43:51.772 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 198028 2024-09-13T14:43:51.772 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.772 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:43:51.772 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.772 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:43:51.772 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.683014+0000", 2024-09-13T14:43:51.772 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:43:51.772 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.772 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.772 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:43:51.772 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.683014+0000", 2024-09-13T14:43:51.772 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:43:51.773 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.773 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.773 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:43:51.773 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.683013+0000", 2024-09-13T14:43:51.773 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:43:51.773 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.773 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.773 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:43:51.773 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.683019+0000", 2024-09-13T14:43:51.773 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1550000000000004e-06 2024-09-13T14:43:51.774 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.774 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.774 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:43:51.774 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.683034+0000", 2024-09-13T14:43:51.774 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4796e-05 2024-09-13T14:43:51.774 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.774 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.774 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:43:51.774 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.683041+0000", 2024-09-13T14:43:51.774 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4699999999999996e-06 2024-09-13T14:43:51.775 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.775 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.775 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:43:51.775 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.685659+0000", 2024-09-13T14:43:51.775 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0026179699999999998 2024-09-13T14:43:51.775 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.775 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.775 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:43:51.775 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.685968+0000", 2024-09-13T14:43:51.775 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00030874299999999999 2024-09-13T14:43:51.775 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.775 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.776 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6,7", 2024-09-13T14:43:51.776 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.686387+0000", 2024-09-13T14:43:51.776 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00041937000000000002 2024-09-13T14:43:51.776 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.776 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.776 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:51.776 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.689774+0000", 2024-09-13T14:43:51.776 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0033865100000000001 2024-09-13T14:43:51.776 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.776 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.776 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:51.776 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.691584+0000", 2024-09-13T14:43:51.776 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0018108029999999999 2024-09-13T14:43:51.777 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.777 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.777 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:43:51.777 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.692138+0000", 2024-09-13T14:43:51.777 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00055365500000000001 2024-09-13T14:43:51.777 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.777 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.777 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:51.777 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.904593+0000", 2024-09-13T14:43:51.777 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.21245457600000001 2024-09-13T14:43:51.777 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.777 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.777 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:43:51.778 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.904642+0000", 2024-09-13T14:43:51.778 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9833999999999998e-05 2024-09-13T14:43:51.778 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.778 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.778 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:43:51.778 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.904749+0000", 2024-09-13T14:43:51.778 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000106693 2024-09-13T14:43:51.778 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:43:51.778 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:43:51.778 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:43:51.778 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.778 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.778 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:198030 6.2 6:416e1cf0:::benchmark_data_smithi102_194590_object87406:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-09-13T14:43:51.778 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:34:29.683047+0000", 2024-09-13T14:43:51.779 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 561.20383401599997, 2024-09-13T14:43:51.779 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.221895961, 2024-09-13T14:43:51.779 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:43:51.779 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:51.779 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:43:51.779 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:43:51.779 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:43:51.779 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 198030 2024-09-13T14:43:51.779 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.779 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:43:51.779 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.779 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:43:51.779 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.683047+0000", 2024-09-13T14:43:51.779 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:43:51.780 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.780 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.780 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:43:51.780 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.683047+0000", 2024-09-13T14:43:51.780 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:43:51.780 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.780 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.780 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:43:51.780 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.683047+0000", 2024-09-13T14:43:51.780 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:43:51.780 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.780 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.780 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:43:51.781 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.683049+0000", 2024-09-13T14:43:51.781 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.424e-06 2024-09-13T14:43:51.781 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.781 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.781 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:43:51.781 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.683058+0000", 2024-09-13T14:43:51.781 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7450000000000001e-06 2024-09-13T14:43:51.781 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.781 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.781 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:43:51.781 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.683063+0000", 2024-09-13T14:43:51.781 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0089999999999998e-06 2024-09-13T14:43:51.781 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.782 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.782 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:43:51.782 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.683110+0000", 2024-09-13T14:43:51.782 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7601e-05 2024-09-13T14:43:51.782 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.782 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.782 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:43:51.782 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.683250+0000", 2024-09-13T14:43:51.782 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013993600000000001 2024-09-13T14:43:51.782 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.782 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.782 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:43:51.782 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.683385+0000", 2024-09-13T14:43:51.783 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013476499999999999 2024-09-13T14:43:51.783 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.783 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.783 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:43:51.783 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.686273+0000", 2024-09-13T14:43:51.783 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0028874199999999999 2024-09-13T14:43:51.783 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.783 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.783 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:51.783 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.686610+0000", 2024-09-13T14:43:51.783 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00033694600000000002 2024-09-13T14:43:51.783 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.783 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.783 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:51.784 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.904866+0000", 2024-09-13T14:43:51.784 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.21825642100000001 2024-09-13T14:43:51.784 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.784 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.784 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:43:51.784 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.904903+0000", 2024-09-13T14:43:51.784 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7057000000000001e-05 2024-09-13T14:43:51.784 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.784 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.784 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:43:51.784 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.904943+0000", 2024-09-13T14:43:51.784 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0006999999999998e-05 2024-09-13T14:43:51.784 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:43:51.785 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:43:51.785 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:43:51.785 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.785 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.785 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113979 7.4 7:2d0634a9:::benchmark_data_smithi102_227395_object1164:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-13T14:43:51.785 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:47.339159+0000", 2024-09-13T14:43:51.785 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 363.54772182200003, 2024-09-13T14:43:51.785 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.356370809, 2024-09-13T14:43:51.785 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:43:51.785 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:51.785 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:43:51.785 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:43:51.786 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:43:51.786 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113979 2024-09-13T14:43:51.786 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.786 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:43:51.786 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.786 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:43:51.786 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.339159+0000", 2024-09-13T14:43:51.786 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:43:51.786 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.786 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.786 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:43:51.786 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.339159+0000", 2024-09-13T14:43:51.787 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:43:51.787 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.787 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.787 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:43:51.787 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.339158+0000", 2024-09-13T14:43:51.787 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:43:51.787 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.787 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.787 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:43:51.787 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.339165+0000", 2024-09-13T14:43:51.787 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4719999999999998e-06 2024-09-13T14:43:51.787 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.788 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.788 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:43:51.788 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.339178+0000", 2024-09-13T14:43:51.788 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.306e-05 2024-09-13T14:43:51.788 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.788 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.789 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:43:51.789 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.339185+0000", 2024-09-13T14:43:51.789 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1509999999999998e-06 2024-09-13T14:43:51.789 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.789 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.789 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:43:51.789 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.339232+0000", 2024-09-13T14:43:51.790 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6953000000000003e-05 2024-09-13T14:43:51.790 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.790 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.790 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:43:51.790 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.339241+0000", 2024-09-13T14:43:51.790 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0900000000000005e-06 2024-09-13T14:43:51.790 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.790 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.790 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:43:51.790 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.112527+0000", 2024-09-13T14:43:51.790 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.77328639700000001 2024-09-13T14:43:51.790 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.790 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.791 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:43:51.791 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.112538+0000", 2024-09-13T14:43:51.791 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1090999999999999e-05 2024-09-13T14:43:51.791 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.791 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.791 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:43:51.791 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.664075+0000", 2024-09-13T14:43:51.791 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5515367659999999 2024-09-13T14:43:51.791 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.791 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.791 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:43:51.791 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.664233+0000", 2024-09-13T14:43:51.791 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015868300000000001 2024-09-13T14:43:51.791 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.791 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.792 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:43:51.792 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.664384+0000", 2024-09-13T14:43:51.792 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000150371 2024-09-13T14:43:51.792 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.792 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.792 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:51.792 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.673923+0000", 2024-09-13T14:43:51.792 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0095388030000000002 2024-09-13T14:43:51.792 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.792 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.792 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:43:51.792 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.677432+0000", 2024-09-13T14:43:51.792 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0035092560000000001 2024-09-13T14:43:51.793 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.793 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.793 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:51.793 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.692151+0000", 2024-09-13T14:43:51.793 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014718716999999999 2024-09-13T14:43:51.793 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.793 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.793 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:43:51.793 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.692201+0000", 2024-09-13T14:43:51.793 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0145000000000002e-05 2024-09-13T14:43:51.793 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.793 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.793 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:43:51.793 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.695530+0000", 2024-09-13T14:43:51.794 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0033293260000000001 2024-09-13T14:43:51.794 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:43:51.794 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:43:51.794 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:43:51.794 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.794 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.794 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113982 7.4 7:256588d9:::benchmark_data_smithi102_227395_object1167:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-13T14:43:51.794 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:47.404500+0000", 2024-09-13T14:43:51.794 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 363.48238103800003, 2024-09-13T14:43:51.794 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.296665693, 2024-09-13T14:43:51.794 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:43:51.794 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:51.794 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:43:51.794 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:43:51.794 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:43:51.795 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113982 2024-09-13T14:43:51.795 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.795 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:43:51.795 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.795 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:43:51.795 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.404500+0000", 2024-09-13T14:43:51.795 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:43:51.795 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.795 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.795 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:43:51.795 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.404500+0000", 2024-09-13T14:43:51.795 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:43:51.795 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.796 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.796 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:43:51.796 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.404499+0000", 2024-09-13T14:43:51.796 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:43:51.796 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.796 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.796 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:43:51.796 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.404505+0000", 2024-09-13T14:43:51.796 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.438e-06 2024-09-13T14:43:51.796 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.796 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.797 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:43:51.797 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.404517+0000", 2024-09-13T14:43:51.797 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2130000000000001e-05 2024-09-13T14:43:51.797 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.797 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.797 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:43:51.797 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.404522+0000", 2024-09-13T14:43:51.797 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2449999999999998e-06 2024-09-13T14:43:51.797 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.797 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.797 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:43:51.797 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.404570+0000", 2024-09-13T14:43:51.797 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7508e-05 2024-09-13T14:43:51.797 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.798 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.798 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:43:51.798 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.404578+0000", 2024-09-13T14:43:51.798 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2519999999999999e-06 2024-09-13T14:43:51.798 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.798 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.798 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:43:51.798 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.112562+0000", 2024-09-13T14:43:51.798 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.707983323 2024-09-13T14:43:51.798 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.798 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.798 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:43:51.798 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.112569+0000", 2024-09-13T14:43:51.798 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9379999999999992e-06 2024-09-13T14:43:51.799 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.799 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.799 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:43:51.799 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.670326+0000", 2024-09-13T14:43:51.799 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.557756707 2024-09-13T14:43:51.799 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.799 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.799 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:43:51.799 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.670465+0000", 2024-09-13T14:43:51.799 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013909099999999999 2024-09-13T14:43:51.799 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.799 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.799 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:43:51.799 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.670607+0000", 2024-09-13T14:43:51.800 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014218999999999999 2024-09-13T14:43:51.800 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.800 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.800 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:43:51.800 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.677438+0000", 2024-09-13T14:43:51.800 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.006830839 2024-09-13T14:43:51.800 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.800 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.800 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:51.800 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.678258+0000", 2024-09-13T14:43:51.800 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00081936699999999999 2024-09-13T14:43:51.800 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.801 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.801 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:51.801 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.695584+0000", 2024-09-13T14:43:51.801 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017326105000000001 2024-09-13T14:43:51.801 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.801 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.801 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:43:51.801 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.695638+0000", 2024-09-13T14:43:51.801 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3949000000000001e-05 2024-09-13T14:43:51.801 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.801 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.801 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:43:51.801 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.701166+0000", 2024-09-13T14:43:51.801 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0055280310000000001 2024-09-13T14:43:51.802 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:43:51.802 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:43:51.802 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:43:51.802 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.802 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.802 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113988 7.9 7:9e6eb1fd:::benchmark_data_smithi102_227395_object1173:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-13T14:43:51.802 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:47.878570+0000", 2024-09-13T14:43:51.802 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 363.00831065300002, 2024-09-13T14:43:51.802 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.34139049300000002, 2024-09-13T14:43:51.802 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:43:51.802 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:51.802 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:43:51.802 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:43:51.802 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:43:51.803 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113988 2024-09-13T14:43:51.803 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.803 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:43:51.803 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.803 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:43:51.803 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.878570+0000", 2024-09-13T14:43:51.803 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:43:51.803 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.803 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.803 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:43:51.803 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.878570+0000", 2024-09-13T14:43:51.803 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:43:51.803 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.803 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.804 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:43:51.804 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.878569+0000", 2024-09-13T14:43:51.804 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:43:51.804 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.804 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.804 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:43:51.804 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.878578+0000", 2024-09-13T14:43:51.804 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.8339999999999994e-06 2024-09-13T14:43:51.804 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.804 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.804 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:43:51.804 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.878595+0000", 2024-09-13T14:43:51.804 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7007000000000002e-05 2024-09-13T14:43:51.805 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.805 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.805 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:43:51.805 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.878623+0000", 2024-09-13T14:43:51.805 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7189999999999999e-05 2024-09-13T14:43:51.805 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.805 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.805 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:43:51.805 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.878685+0000", 2024-09-13T14:43:51.805 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2661000000000002e-05 2024-09-13T14:43:51.805 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.805 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.805 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:43:51.805 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.878698+0000", 2024-09-13T14:43:51.806 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3139000000000001e-05 2024-09-13T14:43:51.806 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.806 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.806 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:43:51.806 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.213485+0000", 2024-09-13T14:43:51.806 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.33478628199999999 2024-09-13T14:43:51.806 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.806 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.806 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:43:51.806 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.213573+0000", 2024-09-13T14:43:51.806 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8212000000000002e-05 2024-09-13T14:43:51.806 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.806 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.806 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,6,7", 2024-09-13T14:43:51.807 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.213671+0000", 2024-09-13T14:43:51.807 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.8344999999999994e-05 2024-09-13T14:43:51.807 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.807 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.807 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:51.807 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.216736+0000", 2024-09-13T14:43:51.807 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0030646760000000001 2024-09-13T14:43:51.807 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.807 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.807 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:43:51.807 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.216745+0000", 2024-09-13T14:43:51.807 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4220000000000006e-06 2024-09-13T14:43:51.807 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.807 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.807 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:51.808 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.216773+0000", 2024-09-13T14:43:51.808 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7970000000000002e-05 2024-09-13T14:43:51.808 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.808 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.808 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:51.808 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.219756+0000", 2024-09-13T14:43:51.808 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0029822120000000001 2024-09-13T14:43:51.808 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.808 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.808 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:43:51.808 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.219817+0000", 2024-09-13T14:43:51.808 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1235000000000006e-05 2024-09-13T14:43:51.808 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.809 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.809 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:43:51.809 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.219961+0000", 2024-09-13T14:43:51.809 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014409899999999999 2024-09-13T14:43:51.809 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:43:51.809 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:43:51.809 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:43:51.809 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.809 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.809 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113993 7.4 7:2f29f4ff:::benchmark_data_smithi102_227395_object1178:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-13T14:43:51.809 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:48.328273+0000", 2024-09-13T14:43:51.809 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 362.55860796899998, 2024-09-13T14:43:51.809 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3770136559999999, 2024-09-13T14:43:51.809 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:43:51.810 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:51.810 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:43:51.810 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:43:51.810 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:43:51.810 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113993 2024-09-13T14:43:51.810 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.810 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:43:51.810 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.810 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:43:51.810 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.328273+0000", 2024-09-13T14:43:51.810 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:43:51.810 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.810 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.811 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:43:51.811 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.328273+0000", 2024-09-13T14:43:51.811 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:43:51.811 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.811 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.811 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:43:51.811 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.328271+0000", 2024-09-13T14:43:51.811 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:43:51.811 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.811 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.811 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:43:51.811 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.328279+0000", 2024-09-13T14:43:51.811 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0609999999999996e-06 2024-09-13T14:43:51.811 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.812 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.812 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:43:51.812 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.328294+0000", 2024-09-13T14:43:51.812 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4405e-05 2024-09-13T14:43:51.812 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.812 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.812 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:43:51.812 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.328301+0000", 2024-09-13T14:43:51.812 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6750000000000002e-06 2024-09-13T14:43:51.812 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.812 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.812 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:43:51.812 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.328376+0000", 2024-09-13T14:43:51.813 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4040000000000003e-05 2024-09-13T14:43:51.813 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.813 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.813 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:43:51.813 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.328389+0000", 2024-09-13T14:43:51.813 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3478e-05 2024-09-13T14:43:51.813 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.813 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.813 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:43:51.813 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.673990+0000", 2024-09-13T14:43:51.813 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.345601324 2024-09-13T14:43:51.813 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.813 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.813 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:43:51.814 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.674152+0000", 2024-09-13T14:43:51.814 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016154299999999999 2024-09-13T14:43:51.814 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.814 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.814 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:43:51.814 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.674339+0000", 2024-09-13T14:43:51.814 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00018710999999999999 2024-09-13T14:43:51.814 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.814 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.814 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:43:51.814 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.678268+0000", 2024-09-13T14:43:51.814 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0039291450000000002 2024-09-13T14:43:51.814 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.814 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.815 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:51.815 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.678582+0000", 2024-09-13T14:43:51.815 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00031387 2024-09-13T14:43:51.815 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.815 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.815 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:51.815 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.701236+0000", 2024-09-13T14:43:51.815 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.022653889 2024-09-13T14:43:51.815 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.815 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.815 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:43:51.815 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.701392+0000", 2024-09-13T14:43:51.815 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015579999999999999 2024-09-13T14:43:51.815 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.816 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.816 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:43:51.816 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.705287+0000", 2024-09-13T14:43:51.816 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003895098 2024-09-13T14:43:51.816 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:43:51.816 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:43:51.816 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:43:51.816 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.816 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.816 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113998 7.1 7:8cc51271:::benchmark_data_smithi102_227395_object1183:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-13T14:43:51.816 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:48.829978+0000", 2024-09-13T14:43:51.816 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 362.05690293999999, 2024-09-13T14:43:51.816 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.86212481600000002, 2024-09-13T14:43:51.817 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:43:51.817 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:51.817 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:43:51.817 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:43:51.817 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:43:51.817 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113998 2024-09-13T14:43:51.817 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.817 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:43:51.817 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.817 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:43:51.817 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.829978+0000", 2024-09-13T14:43:51.817 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:43:51.817 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.817 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.818 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:43:51.818 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.829978+0000", 2024-09-13T14:43:51.818 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:43:51.818 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.818 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.818 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:43:51.818 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.829977+0000", 2024-09-13T14:43:51.818 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:43:51.818 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.818 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.818 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:43:51.818 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.829984+0000", 2024-09-13T14:43:51.818 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7479999999999996e-06 2024-09-13T14:43:51.818 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.819 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.819 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:43:51.819 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.829999+0000", 2024-09-13T14:43:51.819 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5233e-05 2024-09-13T14:43:51.819 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.819 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.819 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:43:51.819 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.830004+0000", 2024-09-13T14:43:51.819 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4519999999999996e-06 2024-09-13T14:43:51.819 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.819 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.819 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:43:51.819 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.830059+0000", 2024-09-13T14:43:51.819 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4829e-05 2024-09-13T14:43:51.820 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.820 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.820 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:43:51.820 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.830072+0000", 2024-09-13T14:43:51.820 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3410999999999999e-05 2024-09-13T14:43:51.820 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.820 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.820 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:43:51.820 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.675677+0000", 2024-09-13T14:43:51.820 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.84560498200000001 2024-09-13T14:43:51.820 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.820 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.820 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:43:51.820 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.675810+0000", 2024-09-13T14:43:51.821 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000132786 2024-09-13T14:43:51.821 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.821 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.821 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:43:51.821 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.675943+0000", 2024-09-13T14:43:51.821 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000132898 2024-09-13T14:43:51.821 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.821 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.821 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:51.821 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.681639+0000", 2024-09-13T14:43:51.821 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0056959269999999999 2024-09-13T14:43:51.821 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.821 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.821 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:43:51.822 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.681746+0000", 2024-09-13T14:43:51.822 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010723299999999999 2024-09-13T14:43:51.822 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.822 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.822 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:51.822 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.689715+0000", 2024-09-13T14:43:51.822 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0079683510000000003 2024-09-13T14:43:51.822 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.822 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.822 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:43:51.822 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.689761+0000", 2024-09-13T14:43:51.822 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5936999999999997e-05 2024-09-13T14:43:51.822 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.822 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.823 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:43:51.823 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.692103+0000", 2024-09-13T14:43:51.823 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002342392 2024-09-13T14:43:51.823 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:43:51.823 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:43:51.823 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:43:51.823 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.823 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.823 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:173483 7.1 7:8955485b:::benchmark_data_smithi102_227395_object60668:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-09-13T14:43:51.823 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:43:30.826623+0000", 2024-09-13T14:43:51.823 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 20.060258256000001, 2024-09-13T14:43:51.823 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.019487589999999999, 2024-09-13T14:43:51.823 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:43:51.823 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:51.824 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:43:51.824 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:43:51.824 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:43:51.824 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 173483 2024-09-13T14:43:51.824 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.824 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:43:51.824 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.824 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:43:51.824 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:30.826623+0000", 2024-09-13T14:43:51.824 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:43:51.824 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.824 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.824 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:43:51.825 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:30.826623+0000", 2024-09-13T14:43:51.825 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:43:51.825 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.825 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.825 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:43:51.825 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:30.826620+0000", 2024-09-13T14:43:51.825 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-09-13T14:43:51.825 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.825 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.825 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:43:51.825 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:30.826628+0000", 2024-09-13T14:43:51.825 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1300000000000001e-06 2024-09-13T14:43:51.825 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.825 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.826 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:43:51.826 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:30.826642+0000", 2024-09-13T14:43:51.826 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3062e-05 2024-09-13T14:43:51.826 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.826 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.826 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:43:51.826 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:30.826648+0000", 2024-09-13T14:43:51.826 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8170000000000001e-06 2024-09-13T14:43:51.826 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.826 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.826 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:43:51.826 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:30.831472+0000", 2024-09-13T14:43:51.826 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.004823417 2024-09-13T14:43:51.826 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.827 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.827 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:43:51.827 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:30.831603+0000", 2024-09-13T14:43:51.827 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013108 2024-09-13T14:43:51.827 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.827 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.827 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:43:51.827 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:30.831731+0000", 2024-09-13T14:43:51.827 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000128405 2024-09-13T14:43:51.827 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.827 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.827 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:51.827 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:30.838097+0000", 2024-09-13T14:43:51.827 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0063660959999999999 2024-09-13T14:43:51.828 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.828 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.828 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:43:51.828 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:30.838111+0000", 2024-09-13T14:43:51.828 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3755e-05 2024-09-13T14:43:51.828 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.828 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.828 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:51.828 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:30.844909+0000", 2024-09-13T14:43:51.828 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0067977000000000003 2024-09-13T14:43:51.828 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.828 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.828 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:43:51.828 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:30.844974+0000", 2024-09-13T14:43:51.829 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5625999999999998e-05 2024-09-13T14:43:51.829 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.829 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.829 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:43:51.829 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:30.846110+0000", 2024-09-13T14:43:51.829 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001135968 2024-09-13T14:43:51.829 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:43:51.829 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:43:51.829 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:43:51.829 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.829 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.829 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:173857 7.1 7:8c395768:::benchmark_data_smithi102_227395_object61042:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-09-13T14:43:51.829 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:43:32.488117+0000", 2024-09-13T14:43:51.830 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 18.39876422, 2024-09-13T14:43:51.830 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.021468803000000002, 2024-09-13T14:43:51.830 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:43:51.830 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:51.830 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:43:51.830 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:43:51.830 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:43:51.830 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 173857 2024-09-13T14:43:51.830 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.830 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:43:51.830 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.830 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:43:51.830 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:32.488117+0000", 2024-09-13T14:43:51.830 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:43:51.831 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.831 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.831 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:43:51.831 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:32.488117+0000", 2024-09-13T14:43:51.831 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:43:51.831 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.831 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.831 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:43:51.831 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:32.488115+0000", 2024-09-13T14:43:51.831 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:43:51.831 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.831 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.831 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:43:51.831 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:32.488122+0000", 2024-09-13T14:43:51.832 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6490000000000002e-06 2024-09-13T14:43:51.832 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.832 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.832 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:43:51.832 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:32.488134+0000", 2024-09-13T14:43:51.832 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2316e-05 2024-09-13T14:43:51.832 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.832 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.832 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:43:51.832 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:32.488141+0000", 2024-09-13T14:43:51.832 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8249999999999999e-06 2024-09-13T14:43:51.832 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.832 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.832 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:43:51.833 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:32.489047+0000", 2024-09-13T14:43:51.833 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00090605799999999995 2024-09-13T14:43:51.833 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.833 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.833 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:43:51.833 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:32.489177+0000", 2024-09-13T14:43:51.833 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012959399999999999 2024-09-13T14:43:51.833 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.833 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.833 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:43:51.833 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:32.489326+0000", 2024-09-13T14:43:51.833 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000149092 2024-09-13T14:43:51.833 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.833 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.834 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:51.834 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:32.497774+0000", 2024-09-13T14:43:51.834 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0084484009999999995 2024-09-13T14:43:51.834 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.834 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.834 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:51.834 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:32.497840+0000", 2024-09-13T14:43:51.834 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6018000000000004e-05 2024-09-13T14:43:51.834 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.834 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.834 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:43:51.834 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:32.508092+0000", 2024-09-13T14:43:51.834 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010251837999999999 2024-09-13T14:43:51.834 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.835 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.835 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:43:51.835 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:32.508156+0000", 2024-09-13T14:43:51.835 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3478000000000004e-05 2024-09-13T14:43:51.835 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.835 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.835 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:43:51.835 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:32.509586+0000", 2024-09-13T14:43:51.835 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0014300929999999999 2024-09-13T14:43:51.835 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:43:51.835 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:43:51.835 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:43:51.835 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.835 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.836 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:173867 7.1 7:8c13f615:::benchmark_data_smithi102_227395_object61052:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-09-13T14:43:51.836 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:43:32.509243+0000", 2024-09-13T14:43:51.836 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 18.377637747000001, 2024-09-13T14:43:51.836 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.020158024, 2024-09-13T14:43:51.836 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:43:51.836 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:51.836 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:43:51.836 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:43:51.836 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:43:51.836 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 173867 2024-09-13T14:43:51.836 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.836 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:43:51.836 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.836 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:43:51.837 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:32.509243+0000", 2024-09-13T14:43:51.837 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:43:51.837 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.837 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.837 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:43:51.837 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:32.509243+0000", 2024-09-13T14:43:51.837 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:43:51.837 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.837 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.837 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:43:51.837 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:32.509242+0000", 2024-09-13T14:43:51.837 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:43:51.837 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.837 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.838 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:43:51.838 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:32.509248+0000", 2024-09-13T14:43:51.838 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7159999999999997e-06 2024-09-13T14:43:51.838 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.838 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.838 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:43:51.838 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:32.509260+0000", 2024-09-13T14:43:51.838 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1564e-05 2024-09-13T14:43:51.838 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.838 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.838 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:43:51.838 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:32.509269+0000", 2024-09-13T14:43:51.838 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8599999999999999e-06 2024-09-13T14:43:51.839 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.839 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.839 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:43:51.839 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:32.509612+0000", 2024-09-13T14:43:51.839 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00034328599999999999 2024-09-13T14:43:51.839 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.839 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.839 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:43:51.839 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:32.509742+0000", 2024-09-13T14:43:51.839 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000129505 2024-09-13T14:43:51.839 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.839 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.839 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:43:51.839 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:32.509840+0000", 2024-09-13T14:43:51.840 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.8518999999999996e-05 2024-09-13T14:43:51.840 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.840 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.840 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:51.840 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:32.516806+0000", 2024-09-13T14:43:51.840 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0069658630000000001 2024-09-13T14:43:51.840 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.840 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.840 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:43:51.840 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:32.522785+0000", 2024-09-13T14:43:51.840 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0059793629999999997 2024-09-13T14:43:51.840 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.840 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.840 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:51.841 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:32.527954+0000", 2024-09-13T14:43:51.841 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0051691630000000001 2024-09-13T14:43:51.841 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.841 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.841 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:43:51.841 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:32.528013+0000", 2024-09-13T14:43:51.841 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8445e-05 2024-09-13T14:43:51.841 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.841 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.841 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:43:51.841 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:32.529401+0000", 2024-09-13T14:43:51.841 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0013884450000000001 2024-09-13T14:43:51.841 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:43:51.841 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:43:51.842 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:43:51.842 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.842 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.842 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:174081 7.1 7:80d99951:::benchmark_data_smithi102_227395_object61266:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-09-13T14:43:51.842 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:43:33.080953+0000", 2024-09-13T14:43:51.842 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 17.805928149, 2024-09-13T14:43:51.842 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.021799749, 2024-09-13T14:43:51.842 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:43:51.842 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:51.842 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:43:51.842 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:43:51.842 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:43:51.842 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 174081 2024-09-13T14:43:51.842 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.843 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:43:51.843 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.843 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:43:51.843 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:33.080953+0000", 2024-09-13T14:43:51.843 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:43:51.843 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.843 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.843 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:43:51.843 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:33.080953+0000", 2024-09-13T14:43:51.843 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:43:51.843 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.843 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.843 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:43:51.843 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:33.080951+0000", 2024-09-13T14:43:51.844 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:43:51.844 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.844 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.844 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:43:51.844 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:33.080960+0000", 2024-09-13T14:43:51.844 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5790000000000004e-06 2024-09-13T14:43:51.844 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.844 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.844 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:43:51.844 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:33.080975+0000", 2024-09-13T14:43:51.844 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5367e-05 2024-09-13T14:43:51.844 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.844 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.845 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:43:51.845 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:33.080983+0000", 2024-09-13T14:43:51.845 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1920000000000005e-06 2024-09-13T14:43:51.845 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.845 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.845 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:43:51.845 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:33.082327+0000", 2024-09-13T14:43:51.845 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0013437189999999999 2024-09-13T14:43:51.845 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.845 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.845 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:43:51.845 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:33.082477+0000", 2024-09-13T14:43:51.845 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015001700000000001 2024-09-13T14:43:51.845 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.846 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.846 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:43:51.846 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:33.082638+0000", 2024-09-13T14:43:51.846 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016096200000000001 2024-09-13T14:43:51.846 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.846 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.846 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:43:51.846 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:33.086319+0000", 2024-09-13T14:43:51.846 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003681035 2024-09-13T14:43:51.846 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.846 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.846 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:51.846 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:33.086551+0000", 2024-09-13T14:43:51.846 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00023172899999999999 2024-09-13T14:43:51.847 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.847 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.847 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:51.847 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:33.101499+0000", 2024-09-13T14:43:51.847 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014948189000000001 2024-09-13T14:43:51.847 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.847 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.847 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:43:51.847 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:33.101550+0000", 2024-09-13T14:43:51.847 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1193999999999998e-05 2024-09-13T14:43:51.847 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.847 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.847 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:43:51.847 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:33.102753+0000", 2024-09-13T14:43:51.848 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0012025359999999999 2024-09-13T14:43:51.848 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:43:51.848 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:43:51.848 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:43:51.848 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.848 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.848 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:174092 7.1 7:858d1779:::benchmark_data_smithi102_227395_object61277:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-09-13T14:43:51.848 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:43:33.096797+0000", 2024-09-13T14:43:51.848 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 17.790084556, 2024-09-13T14:43:51.848 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.020503454000000001, 2024-09-13T14:43:51.848 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:43:51.848 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:51.848 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:43:51.848 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:43:51.849 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:43:51.849 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 174092 2024-09-13T14:43:51.849 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.849 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:43:51.849 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.849 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:43:51.849 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:33.096797+0000", 2024-09-13T14:43:51.849 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:43:51.849 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.849 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.849 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:43:51.849 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:33.096797+0000", 2024-09-13T14:43:51.849 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:43:51.849 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.850 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.850 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:43:51.850 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:33.096795+0000", 2024-09-13T14:43:51.850 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:43:51.850 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.850 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.850 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:43:51.850 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:33.096805+0000", 2024-09-13T14:43:51.850 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0088000000000001e-05 2024-09-13T14:43:51.850 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.850 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.850 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:43:51.850 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:33.096818+0000", 2024-09-13T14:43:51.850 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2652e-05 2024-09-13T14:43:51.851 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.851 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.851 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:43:51.851 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:33.096826+0000", 2024-09-13T14:43:51.851 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1300000000000001e-06 2024-09-13T14:43:51.851 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.851 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.851 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:43:51.851 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:33.106675+0000", 2024-09-13T14:43:51.851 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0098492010000000001 2024-09-13T14:43:51.851 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.851 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.851 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:43:51.851 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:33.106819+0000", 2024-09-13T14:43:51.852 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014368900000000001 2024-09-13T14:43:51.852 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.852 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.852 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:43:51.852 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:33.106985+0000", 2024-09-13T14:43:51.852 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000166696 2024-09-13T14:43:51.852 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.852 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.852 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:51.852 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:33.112067+0000", 2024-09-13T14:43:51.852 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0050816239999999999 2024-09-13T14:43:51.852 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.852 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.853 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:51.853 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:33.115898+0000", 2024-09-13T14:43:51.853 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0038307670000000001 2024-09-13T14:43:51.853 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.853 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.853 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:43:51.853 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:33.115928+0000", 2024-09-13T14:43:51.853 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0233e-05 2024-09-13T14:43:51.853 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.853 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.853 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:43:51.853 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:33.115984+0000", 2024-09-13T14:43:51.853 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5748999999999998e-05 2024-09-13T14:43:51.853 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.854 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.854 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:43:51.854 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:33.117300+0000", 2024-09-13T14:43:51.854 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001316179 2024-09-13T14:43:51.854 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:43:51.854 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:43:51.854 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:43:51.854 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.854 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.854 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:179030 7.1 7:88439225:::benchmark_data_smithi102_227395_object66215:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-09-13T14:43:51.854 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:43:50.347682+0000", 2024-09-13T14:43:51.854 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.53919903800000002, 2024-09-13T14:43:51.854 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0036871690000000001, 2024-09-13T14:43:51.855 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:43:51.855 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:51.855 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:43:51.855 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:43:51.855 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:43:51.855 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 179030 2024-09-13T14:43:51.855 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.855 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:43:51.855 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.855 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:43:51.855 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:50.347682+0000", 2024-09-13T14:43:51.855 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:43:51.855 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.855 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.856 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:43:51.856 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:50.347682+0000", 2024-09-13T14:43:51.856 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:43:51.856 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.856 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.856 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:43:51.856 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:50.347681+0000", 2024-09-13T14:43:51.856 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:43:51.856 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.856 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.856 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:43:51.856 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:50.347687+0000", 2024-09-13T14:43:51.856 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2790000000000004e-06 2024-09-13T14:43:51.856 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.857 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.857 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:43:51.857 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:50.347698+0000", 2024-09-13T14:43:51.857 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0779e-05 2024-09-13T14:43:51.857 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.857 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.857 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:43:51.857 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:50.347703+0000", 2024-09-13T14:43:51.857 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5140000000000001e-06 2024-09-13T14:43:51.857 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.857 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.857 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:43:51.857 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:50.347737+0000", 2024-09-13T14:43:51.857 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.3356999999999999e-05 2024-09-13T14:43:51.858 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.858 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.858 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:43:51.858 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:50.347828+0000", 2024-09-13T14:43:51.858 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1671999999999999e-05 2024-09-13T14:43:51.858 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.858 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.858 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:43:51.858 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:50.347908+0000", 2024-09-13T14:43:51.858 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.928e-05 2024-09-13T14:43:51.858 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.858 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.858 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:51.859 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:50.350591+0000", 2024-09-13T14:43:51.859 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0026831590000000001 2024-09-13T14:43:51.859 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.859 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.859 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:43:51.859 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:50.350682+0000", 2024-09-13T14:43:51.859 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1719999999999996e-05 2024-09-13T14:43:51.859 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.859 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.859 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:51.859 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:50.351267+0000", 2024-09-13T14:43:51.859 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00058466200000000001 2024-09-13T14:43:51.859 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.859 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.860 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:43:51.860 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:50.351301+0000", 2024-09-13T14:43:51.860 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.3420000000000002e-05 2024-09-13T14:43:51.860 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.860 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.860 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:43:51.860 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:50.351369+0000", 2024-09-13T14:43:51.860 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8696999999999996e-05 2024-09-13T14:43:51.860 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:43:51.860 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:43:51.860 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:43:51.860 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.860 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.861 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:179047 7.1 7:8eedc5b8:::benchmark_data_smithi102_227395_object66232:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-09-13T14:43:51.861 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:43:50.380360+0000", 2024-09-13T14:43:51.861 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.50652062099999995, 2024-09-13T14:43:51.861 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0044444130000000004, 2024-09-13T14:43:51.861 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:43:51.861 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:51.861 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:43:51.861 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:43:51.861 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:43:51.861 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 179047 2024-09-13T14:43:51.861 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.861 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:43:51.861 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.861 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:43:51.862 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:50.380360+0000", 2024-09-13T14:43:51.862 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:43:51.862 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.862 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.862 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:43:51.862 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:50.380360+0000", 2024-09-13T14:43:51.862 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:43:51.862 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.862 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.862 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:43:51.862 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:50.380359+0000", 2024-09-13T14:43:51.862 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:43:51.862 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.862 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.863 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:43:51.863 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:50.380366+0000", 2024-09-13T14:43:51.863 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.34e-06 2024-09-13T14:43:51.863 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.863 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.863 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:43:51.863 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:50.380382+0000", 2024-09-13T14:43:51.863 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6049000000000002e-05 2024-09-13T14:43:51.863 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.863 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.863 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:43:51.863 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:50.380389+0000", 2024-09-13T14:43:51.863 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8179999999999998e-06 2024-09-13T14:43:51.864 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.864 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.864 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:43:51.864 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:50.380437+0000", 2024-09-13T14:43:51.864 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7341e-05 2024-09-13T14:43:51.864 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.864 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.864 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:43:51.864 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:50.380536+0000", 2024-09-13T14:43:51.864 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.8991000000000001e-05 2024-09-13T14:43:51.864 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.864 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.864 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:43:51.864 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:50.380619+0000", 2024-09-13T14:43:51.865 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3733999999999995e-05 2024-09-13T14:43:51.865 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.865 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.865 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:51.865 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:50.383556+0000", 2024-09-13T14:43:51.865 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0029369909999999999 2024-09-13T14:43:51.865 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.865 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.865 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:43:51.865 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:50.383634+0000", 2024-09-13T14:43:51.865 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7971000000000003e-05 2024-09-13T14:43:51.865 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.865 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.865 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:51.866 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:50.384718+0000", 2024-09-13T14:43:51.866 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001083709 2024-09-13T14:43:51.866 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.866 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.866 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:43:51.866 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:50.384749+0000", 2024-09-13T14:43:51.866 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.1001999999999999e-05 2024-09-13T14:43:51.866 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.866 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.866 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:43:51.866 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:50.384805+0000", 2024-09-13T14:43:51.866 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5943999999999998e-05 2024-09-13T14:43:51.866 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:43:51.867 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:43:51.867 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:43:51.867 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.867 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.867 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:179074 7.1 7:886614cf:::benchmark_data_smithi102_227395_object66259:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-09-13T14:43:51.867 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:43:50.518661+0000", 2024-09-13T14:43:51.867 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.368219939, 2024-09-13T14:43:51.867 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003737387, 2024-09-13T14:43:51.867 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:43:51.867 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:51.867 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:43:51.867 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:43:51.867 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:43:51.867 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 179074 2024-09-13T14:43:51.868 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.868 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:43:51.868 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.868 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:43:51.868 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:50.518661+0000", 2024-09-13T14:43:51.868 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:43:51.868 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.868 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.868 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:43:51.868 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:50.518661+0000", 2024-09-13T14:43:51.868 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:43:51.868 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.868 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.868 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:43:51.869 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:50.518661+0000", 2024-09-13T14:43:51.869 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:43:51.869 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.869 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.869 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:43:51.869 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:50.518664+0000", 2024-09-13T14:43:51.869 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7009999999999998e-06 2024-09-13T14:43:51.869 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.869 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.869 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:43:51.869 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:50.518673+0000", 2024-09-13T14:43:51.869 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.4130000000000006e-06 2024-09-13T14:43:51.869 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.869 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.870 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:43:51.870 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:50.518677+0000", 2024-09-13T14:43:51.870 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6450000000000004e-06 2024-09-13T14:43:51.870 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.870 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.870 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:43:51.870 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:50.518713+0000", 2024-09-13T14:43:51.870 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5438000000000002e-05 2024-09-13T14:43:51.870 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.870 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.870 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:43:51.870 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:50.518796+0000", 2024-09-13T14:43:51.870 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3435000000000003e-05 2024-09-13T14:43:51.870 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.871 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.871 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:43:51.871 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:50.518865+0000", 2024-09-13T14:43:51.871 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8398000000000005e-05 2024-09-13T14:43:51.871 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.871 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.871 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:43:51.871 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:50.521176+0000", 2024-09-13T14:43:51.871 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0023116769999999998 2024-09-13T14:43:51.871 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.871 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.871 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:51.871 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:50.521595+0000", 2024-09-13T14:43:51.871 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00041890800000000001 2024-09-13T14:43:51.872 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.872 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.872 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:51.872 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:50.522235+0000", 2024-09-13T14:43:51.872 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00063991299999999998 2024-09-13T14:43:51.872 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.872 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.872 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:43:51.872 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:50.522290+0000", 2024-09-13T14:43:51.872 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5429000000000001e-05 2024-09-13T14:43:51.872 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:43:51.872 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:43:51.872 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:43:51.872 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:50.522399+0000", 2024-09-13T14:43:51.873 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000108065 2024-09-13T14:43:51.873 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:43:51.873 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:43:51.873 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:43:51.873 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:43:51.873 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:43:51.873 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:43:51.873 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:51.895 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:51.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:51.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:51.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:51 smithi102 ceph-mon[110183]: osdmap e289: 8 total, 8 up, 6 in 2024-09-13T14:43:51.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:51 smithi102 ceph-mon[115269]: osdmap e289: 8 total, 8 up, 6 in 2024-09-13T14:43:51.997 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:51.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:51 smithi195 ceph-mon[104262]: osdmap e289: 8 total, 8 up, 6 in 2024-09-13T14:43:52.019 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:52.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:52.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:52.121 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:52.143 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:52.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:52.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:52.154 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:52.245 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:52.270 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:52.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:52.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:52.372 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:52.394 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:52.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:52.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:52.496 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:52.517 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:52.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:52.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:52.619 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:52.644 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:52.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:52.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:52.746 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:52.772 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:52.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:52.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:52.874 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:52.877 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:52 smithi102 ceph-mon[110183]: pgmap v995: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 2.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 197 op/s; 9421/141954 objects degraded (6.637%); 1987/141954 objects misplaced (1.400%); 6.4 MiB/s, 140 objects/s recovering 2024-09-13T14:43:52.877 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:52 smithi102 ceph-mon[110183]: osdmap e290: 8 total, 8 up, 6 in 2024-09-13T14:43:52.878 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:52 smithi102 ceph-mon[115269]: pgmap v995: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 2.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 197 op/s; 9421/141954 objects degraded (6.637%); 1987/141954 objects misplaced (1.400%); 6.4 MiB/s, 140 objects/s recovering 2024-09-13T14:43:52.878 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:52 smithi102 ceph-mon[115269]: osdmap e290: 8 total, 8 up, 6 in 2024-09-13T14:43:52.895 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:52.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:52.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:52.997 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:52.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:52 smithi195 ceph-mon[104262]: pgmap v995: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 2.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 197 op/s; 9421/141954 objects degraded (6.637%); 1987/141954 objects misplaced (1.400%); 6.4 MiB/s, 140 objects/s recovering 2024-09-13T14:43:52.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:52 smithi195 ceph-mon[104262]: osdmap e290: 8 total, 8 up, 6 in 2024-09-13T14:43:53.021 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:53.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:53.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:53.123 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:53.143 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:53.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:53.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:53.245 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:53.268 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:53.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:53.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:53.369 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:53.391 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:53.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:53.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:53.493 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:53.514 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:53.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:53.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:53.556 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:43:53.616 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:53.646 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:53.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:53.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:53.716 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:43:53.717 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:43:53.748 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:53.770 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:53.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:53.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:53.871 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:53.894 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:53.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:53.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:53.995 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:54.022 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:54.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:54.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:54.124 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:54.147 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:54.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:54.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:54.249 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:54.272 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:54.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:54.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:54.374 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:54.396 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:54.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:54.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:54.498 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:54.521 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:54.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:54.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:54.623 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:54.648 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:54.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:54.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:54.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:54 smithi102 ceph-mon[110183]: pgmap v997: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.8 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 374 op/s; 5472/138885 objects degraded (3.940%); 4903/138885 objects misplaced (3.530%); 8.8 MiB/s, 285 objects/s recovering 2024-09-13T14:43:54.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:54 smithi102 ceph-mon[110183]: from='client.36006 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:43:54.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:54 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 5472/138885 objects degraded (3.940%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:43:54.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:54 smithi102 ceph-mon[115269]: pgmap v997: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.8 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 374 op/s; 5472/138885 objects degraded (3.940%); 4903/138885 objects misplaced (3.530%); 8.8 MiB/s, 285 objects/s recovering 2024-09-13T14:43:54.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:54 smithi102 ceph-mon[115269]: from='client.36006 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:43:54.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:54 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 5472/138885 objects degraded (3.940%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:43:54.749 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:54.770 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:54.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:54.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:54.872 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:54.895 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:54.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:54.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:54.996 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:54.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:54 smithi195 ceph-mon[104262]: pgmap v997: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.8 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 374 op/s; 5472/138885 objects degraded (3.940%); 4903/138885 objects misplaced (3.530%); 8.8 MiB/s, 285 objects/s recovering 2024-09-13T14:43:54.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:54 smithi195 ceph-mon[104262]: from='client.36006 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:43:54.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:54 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 5472/138885 objects degraded (3.940%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:43:55.019 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:55.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:55.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:55.103 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":997,"stamp":"2024-09-13T14:43:53.534490+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3036878845,"num_objects":46295,"num_object_clones":0,"num_object_copies":138885,"num_objects_missing_on_primary":83,"num_objects_missing":83,"num_objects_degraded":5472,"num_objects_misplaced":4903,"num_objects_unfound":0,"num_objects_dirty":46295,"num_whiteouts":0,"num_read":6377,"num_read_kb":13834,"num_write":180482,"num_write_kb":7235010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83839,"num_bytes_recovered":5365959821,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95964,"ondisk_log_size":95964,"up":267,"acting":265,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":299,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":14446940,"kb_used_data":11175436,"kb_used_omap":26,"kb_used_meta":3271397,"kb_avail":548023972,"statfs":{"total":575970213888,"available":561176547328,"internally_reserved":0,"allocated":11443646464,"data_stored":11434372599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27144,"internal_metadata":3349911032},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-170590208,"num_objects":-2603,"num_object_clones":0,"num_object_copies":-7809,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-5005,"num_objects_misplaced":2916,"num_objects_unfound":0,"num_objects_dirty":-2603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2603,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1986,"num_bytes_recovered":63963136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.945146"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":474,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.626279+0000","last_change":"2024-09-13T14:36:13.859927+0000","last_active":"2024-09-13T14:43:51.626279+0000","last_peered":"2024-09-13T14:43:51.626279+0000","last_clean":"2024-09-13T14:43:51.626279+0000","last_became_active":"2024-09-13T14:36:13.859505+0000","last_became_peered":"2024-09-13T14:36:13.859505+0000","last_unstale":"2024-09-13T14:43:51.626279+0000","last_undegraded":"2024-09-13T14:43:51.626279+0000","last_fullsized":"2024-09-13T14:43:51.626279+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:12:20.989627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":292,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.690420+0000","last_change":"2024-09-13T14:37:48.130968+0000","last_active":"2024-09-13T14:43:50.690420+0000","last_peered":"2024-09-13T14:43:50.690420+0000","last_clean":"2024-09-13T14:43:50.690420+0000","last_became_active":"2024-09-13T14:37:48.130603+0000","last_became_peered":"2024-09-13T14:37:48.130603+0000","last_unstale":"2024-09-13T14:43:50.690420+0000","last_undegraded":"2024-09-13T14:43:50.690420+0000","last_fullsized":"2024-09-13T14:43:50.690420+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:42.626386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":89,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.620441+0000","last_change":"2024-09-13T14:36:13.862181+0000","last_active":"2024-09-13T14:43:50.620441+0000","last_peered":"2024-09-13T14:43:50.620441+0000","last_clean":"2024-09-13T14:43:50.620441+0000","last_became_active":"2024-09-13T14:36:13.861873+0000","last_became_peered":"2024-09-13T14:36:13.861873+0000","last_unstale":"2024-09-13T14:43:50.620441+0000","last_undegraded":"2024-09-13T14:43:50.620441+0000","last_fullsized":"2024-09-13T14:43:50.620441+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:14.085700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":111,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.650001+0000","last_change":"2024-09-13T14:38:53.896672+0000","last_active":"2024-09-13T14:43:51.650001+0000","last_peered":"2024-09-13T14:43:51.650001+0000","last_clean":"2024-09-13T14:43:51.650001+0000","last_became_active":"2024-09-13T14:38:53.896346+0000","last_became_peered":"2024-09-13T14:38:53.896346+0000","last_unstale":"2024-09-13T14:43:51.650001+0000","last_undegraded":"2024-09-13T14:43:51.650001+0000","last_fullsized":"2024-09-13T14:43:51.650001+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:57.789686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":111,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.650492+0000","last_change":"2024-09-13T14:38:53.893021+0000","last_active":"2024-09-13T14:43:51.650492+0000","last_peered":"2024-09-13T14:43:51.650492+0000","last_clean":"2024-09-13T14:43:51.650492+0000","last_became_active":"2024-09-13T14:38:53.892668+0000","last_became_peered":"2024-09-13T14:38:53.892668+0000","last_unstale":"2024-09-13T14:43:51.650492+0000","last_undegraded":"2024-09-13T14:43:51.650492+0000","last_fullsized":"2024-09-13T14:43:51.650492+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:57.180738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":260,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.690490+0000","last_change":"2024-09-13T14:37:48.134642+0000","last_active":"2024-09-13T14:43:50.690490+0000","last_peered":"2024-09-13T14:43:50.690490+0000","last_clean":"2024-09-13T14:43:50.690490+0000","last_became_active":"2024-09-13T14:37:48.134402+0000","last_became_peered":"2024-09-13T14:37:48.134402+0000","last_unstale":"2024-09-13T14:43:50.690490+0000","last_undegraded":"2024-09-13T14:43:50.690490+0000","last_fullsized":"2024-09-13T14:43:50.690490+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:51.313663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":492,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.625758+0000","last_change":"2024-09-13T14:38:53.903022+0000","last_active":"2024-09-13T14:43:51.625758+0000","last_peered":"2024-09-13T14:43:51.625758+0000","last_clean":"2024-09-13T14:43:51.625758+0000","last_became_active":"2024-09-13T14:38:53.902448+0000","last_became_peered":"2024-09-13T14:38:53.902448+0000","last_unstale":"2024-09-13T14:43:51.625758+0000","last_undegraded":"2024-09-13T14:43:51.625758+0000","last_fullsized":"2024-09-13T14:43:51.625758+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:24.690401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":114,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.649644+0000","last_change":"2024-09-13T14:38:03.554500+0000","last_active":"2024-09-13T14:43:51.649644+0000","last_peered":"2024-09-13T14:43:51.649644+0000","last_clean":"2024-09-13T14:43:51.649644+0000","last_became_active":"2024-09-13T14:38:03.532910+0000","last_became_peered":"2024-09-13T14:38:03.532910+0000","last_unstale":"2024-09-13T14:43:51.649644+0000","last_undegraded":"2024-09-13T14:43:51.649644+0000","last_fullsized":"2024-09-13T14:43:51.649644+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:27:50.843498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"290'11082","reported_seq":21983,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:52.281211+0000","last_change":"2024-09-13T14:41:53.220083+0000","last_active":"2024-09-13T14:43:52.281211+0000","last_peered":"2024-09-13T14:43:52.281211+0000","last_clean":"2024-09-13T14:43:52.281211+0000","last_became_active":"2024-09-13T14:41:53.200084+0000","last_became_peered":"2024-09-13T14:41:53.200084+0000","last_unstale":"2024-09-13T14:43:52.281211+0000","last_undegraded":"2024-09-13T14:43:52.281211+0000","last_fullsized":"2024-09-13T14:43:52.281211+0000","mapping_epoch":275,"log_start":"219'5400","ondisk_log_start":"219'5400","created":205,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5682,"log_dups_size":0,"ondisk_log_size":5682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:08.897524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183369728,"num_objects":2798,"num_object_clones":0,"num_object_copies":8394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11082,"num_write_kb":444160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9152,"num_bytes_recovered":594608128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":53,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.690686+0000","last_change":"2024-09-13T14:38:53.940238+0000","last_active":"2024-09-13T14:43:50.690686+0000","last_peered":"2024-09-13T14:43:50.690686+0000","last_clean":"2024-09-13T14:43:50.690686+0000","last_became_active":"2024-09-13T14:38:53.892825+0000","last_became_peered":"2024-09-13T14:38:53.892825+0000","last_unstale":"2024-09-13T14:43:50.690686+0000","last_undegraded":"2024-09-13T14:43:50.690686+0000","last_fullsized":"2024-09-13T14:43:50.690686+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:46.666581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":109,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.619817+0000","last_change":"2024-09-13T14:38:53.898907+0000","last_active":"2024-09-13T14:43:50.619817+0000","last_peered":"2024-09-13T14:43:50.619817+0000","last_clean":"2024-09-13T14:43:50.619817+0000","last_became_active":"2024-09-13T14:38:53.898572+0000","last_became_peered":"2024-09-13T14:38:53.898572+0000","last_unstale":"2024-09-13T14:43:50.619817+0000","last_undegraded":"2024-09-13T14:43:50.619817+0000","last_fullsized":"2024-09-13T14:43:50.619817+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:29.222150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"290'11373","reported_seq":25563,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:52.422987+0000","last_change":"2024-09-13T14:38:57.930032+0000","last_active":"2024-09-13T14:43:52.422987+0000","last_peered":"2024-09-13T14:43:52.422987+0000","last_clean":"2024-09-13T14:43:52.422987+0000","last_became_active":"2024-09-13T14:38:57.921458+0000","last_became_peered":"2024-09-13T14:38:57.921458+0000","last_unstale":"2024-09-13T14:43:52.422987+0000","last_undegraded":"2024-09-13T14:43:52.422987+0000","last_fullsized":"2024-09-13T14:43:52.422987+0000","mapping_epoch":256,"log_start":"219'5639","ondisk_log_start":"219'5639","created":205,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5734,"log_dups_size":0,"ondisk_log_size":5734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:27:12.294038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188022784,"num_objects":2869,"num_object_clones":0,"num_object_copies":8607,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11373,"num_write_kb":455744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6990,"num_bytes_recovered":458096640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":53,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.690681+0000","last_change":"2024-09-13T14:38:53.904885+0000","last_active":"2024-09-13T14:43:50.690681+0000","last_peered":"2024-09-13T14:43:50.690681+0000","last_clean":"2024-09-13T14:43:50.690681+0000","last_became_active":"2024-09-13T14:38:53.891726+0000","last_became_peered":"2024-09-13T14:38:53.891726+0000","last_unstale":"2024-09-13T14:43:50.690681+0000","last_undegraded":"2024-09-13T14:43:50.690681+0000","last_fullsized":"2024-09-13T14:43:50.690681+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:57.079205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":298,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.622236+0000","last_change":"2024-09-13T14:37:48.160504+0000","last_active":"2024-09-13T14:43:50.622236+0000","last_peered":"2024-09-13T14:43:50.622236+0000","last_clean":"2024-09-13T14:43:50.622236+0000","last_became_active":"2024-09-13T14:37:48.160181+0000","last_became_peered":"2024-09-13T14:37:48.160181+0000","last_unstale":"2024-09-13T14:43:50.622236+0000","last_undegraded":"2024-09-13T14:43:50.622236+0000","last_fullsized":"2024-09-13T14:43:50.622236+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:37.087008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"290'11202","reported_seq":13014,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.659871+0000","last_change":"2024-09-13T14:38:04.531914+0000","last_active":"2024-09-13T14:43:51.659871+0000","last_peered":"2024-09-13T14:43:51.659871+0000","last_clean":"2024-09-13T14:43:51.659871+0000","last_became_active":"2024-09-13T14:38:04.523448+0000","last_became_peered":"2024-09-13T14:38:04.523448+0000","last_unstale":"2024-09-13T14:43:51.659871+0000","last_undegraded":"2024-09-13T14:43:51.659871+0000","last_fullsized":"2024-09-13T14:43:51.659871+0000","mapping_epoch":245,"log_start":"219'5300","ondisk_log_start":"219'5300","created":205,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5902,"log_dups_size":0,"ondisk_log_size":5902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:06.536896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194772992,"num_objects":2972,"num_object_clones":0,"num_object_copies":8916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11202,"num_write_kb":453568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":811,"num_bytes_recovered":53149696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":290,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.690418+0000","last_change":"2024-09-13T14:37:49.611774+0000","last_active":"2024-09-13T14:43:50.690418+0000","last_peered":"2024-09-13T14:43:50.690418+0000","last_clean":"2024-09-13T14:43:50.690418+0000","last_became_active":"2024-09-13T14:37:49.611088+0000","last_became_peered":"2024-09-13T14:37:49.611088+0000","last_unstale":"2024-09-13T14:43:50.690418+0000","last_undegraded":"2024-09-13T14:43:50.690418+0000","last_fullsized":"2024-09-13T14:43:50.690418+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:08.933213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":49,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.690168+0000","last_change":"2024-09-13T14:38:53.899211+0000","last_active":"2024-09-13T14:43:50.690168+0000","last_peered":"2024-09-13T14:43:50.690168+0000","last_clean":"2024-09-13T14:43:50.690168+0000","last_became_active":"2024-09-13T14:38:53.898927+0000","last_became_peered":"2024-09-13T14:38:53.898927+0000","last_unstale":"2024-09-13T14:43:50.690168+0000","last_undegraded":"2024-09-13T14:43:50.690168+0000","last_fullsized":"2024-09-13T14:43:50.690168+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:00:28.857611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"290'11110","reported_seq":11207,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:52.336890+0000","last_change":"2024-09-13T14:35:24.004651+0000","last_active":"2024-09-13T14:43:52.336890+0000","last_peered":"2024-09-13T14:43:52.336890+0000","last_clean":"2024-09-13T14:43:52.336890+0000","last_became_active":"2024-09-13T14:35:24.004358+0000","last_became_peered":"2024-09-13T14:35:24.004358+0000","last_unstale":"2024-09-13T14:43:52.336890+0000","last_undegraded":"2024-09-13T14:43:52.336890+0000","last_fullsized":"2024-09-13T14:43:52.336890+0000","mapping_epoch":205,"log_start":"216'4784","ondisk_log_start":"216'4784","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6326,"log_dups_size":0,"ondisk_log_size":6326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:57:38.023669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187564032,"num_objects":2862,"num_object_clones":0,"num_object_copies":8586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11110,"num_write_kb":447104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":100,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.620360+0000","last_change":"2024-09-13T14:37:49.168003+0000","last_active":"2024-09-13T14:43:50.620360+0000","last_peered":"2024-09-13T14:43:50.620360+0000","last_clean":"2024-09-13T14:43:50.620360+0000","last_became_active":"2024-09-13T14:37:49.167638+0000","last_became_peered":"2024-09-13T14:37:49.167638+0000","last_unstale":"2024-09-13T14:43:50.620360+0000","last_undegraded":"2024-09-13T14:43:50.620360+0000","last_fullsized":"2024-09-13T14:43:50.620360+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:29.051981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":104,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.650052+0000","last_change":"2024-09-13T14:38:03.528381+0000","last_active":"2024-09-13T14:43:51.650052+0000","last_peered":"2024-09-13T14:43:51.650052+0000","last_clean":"2024-09-13T14:43:51.650052+0000","last_became_active":"2024-09-13T14:38:03.527282+0000","last_became_peered":"2024-09-13T14:38:03.527282+0000","last_unstale":"2024-09-13T14:43:51.650052+0000","last_undegraded":"2024-09-13T14:43:51.650052+0000","last_fullsized":"2024-09-13T14:43:51.650052+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:24.435067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.f","version":"290'11163","reported_seq":11470,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.676129+0000","last_change":"2024-09-13T14:38:16.371079+0000","last_active":"2024-09-13T14:43:51.676129+0000","last_peered":"2024-09-13T14:43:51.676129+0000","last_clean":"2024-09-13T14:43:51.676129+0000","last_became_active":"2024-09-13T14:38:12.859883+0000","last_became_peered":"2024-09-13T14:38:12.859883+0000","last_unstale":"2024-09-13T14:43:51.676129+0000","last_undegraded":"2024-09-13T14:43:51.676129+0000","last_fullsized":"2024-09-13T14:43:51.676129+0000","mapping_epoch":245,"log_start":"219'5263","ondisk_log_start":"219'5263","created":205,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5900,"log_dups_size":0,"ondisk_log_size":5900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:08.436532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191561728,"num_objects":2923,"num_object_clones":0,"num_object_copies":8769,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11163,"num_write_kb":450752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":3670016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":462,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.625571+0000","last_change":"2024-09-13T14:38:03.549595+0000","last_active":"2024-09-13T14:43:51.625571+0000","last_peered":"2024-09-13T14:43:51.625571+0000","last_clean":"2024-09-13T14:43:51.625571+0000","last_became_active":"2024-09-13T14:38:03.533331+0000","last_became_peered":"2024-09-13T14:38:03.533331+0000","last_unstale":"2024-09-13T14:43:51.625571+0000","last_undegraded":"2024-09-13T14:43:51.625571+0000","last_fullsized":"2024-09-13T14:43:51.625571+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:59:02.470343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":274,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.622456+0000","last_change":"2024-09-13T14:37:48.127600+0000","last_active":"2024-09-13T14:43:50.622456+0000","last_peered":"2024-09-13T14:43:50.622456+0000","last_clean":"2024-09-13T14:43:50.622456+0000","last_became_active":"2024-09-13T14:37:48.127349+0000","last_became_peered":"2024-09-13T14:37:48.127349+0000","last_unstale":"2024-09-13T14:43:50.622456+0000","last_undegraded":"2024-09-13T14:43:50.622456+0000","last_fullsized":"2024-09-13T14:43:50.622456+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:57.268374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"290'11207","reported_seq":21201,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:52.269886+0000","last_change":"2024-09-13T14:42:30.227259+0000","last_active":"2024-09-13T14:43:52.269886+0000","last_peered":"2024-09-13T14:43:52.269886+0000","last_clean":"2024-09-13T14:43:52.269886+0000","last_became_active":"2024-09-13T14:42:30.209102+0000","last_became_peered":"2024-09-13T14:42:30.209102+0000","last_unstale":"2024-09-13T14:43:52.269886+0000","last_undegraded":"2024-09-13T14:43:52.269886+0000","last_fullsized":"2024-09-13T14:43:52.269886+0000","mapping_epoch":281,"log_start":"219'5500","ondisk_log_start":"219'5500","created":205,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5707,"log_dups_size":0,"ondisk_log_size":5707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:56:16.193287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183697408,"num_objects":2803,"num_object_clones":0,"num_object_copies":8409,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11207,"num_write_kb":448320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4926,"num_bytes_recovered":313851904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":207,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.622331+0000","last_change":"2024-09-13T14:37:48.127604+0000","last_active":"2024-09-13T14:43:50.622331+0000","last_peered":"2024-09-13T14:43:50.622331+0000","last_clean":"2024-09-13T14:43:50.622331+0000","last_became_active":"2024-09-13T14:37:48.127300+0000","last_became_peered":"2024-09-13T14:37:48.127300+0000","last_unstale":"2024-09-13T14:43:50.622331+0000","last_undegraded":"2024-09-13T14:43:50.622331+0000","last_fullsized":"2024-09-13T14:43:50.622331+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:51.999506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":327,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.622553+0000","last_change":"2024-09-13T14:38:03.529967+0000","last_active":"2024-09-13T14:43:50.622553+0000","last_peered":"2024-09-13T14:43:50.622553+0000","last_clean":"2024-09-13T14:43:50.622553+0000","last_became_active":"2024-09-13T14:38:03.529646+0000","last_became_peered":"2024-09-13T14:38:03.529646+0000","last_unstale":"2024-09-13T14:43:50.622553+0000","last_undegraded":"2024-09-13T14:43:50.622553+0000","last_fullsized":"2024-09-13T14:43:50.622553+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:52:05.193279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"290'11256","reported_seq":20438,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.650263+0000","last_change":"2024-09-13T14:40:34.583237+0000","last_active":"2024-09-13T14:43:51.650263+0000","last_peered":"2024-09-13T14:43:51.650263+0000","last_clean":"2024-09-13T14:43:51.650263+0000","last_became_active":"2024-09-13T14:40:34.563989+0000","last_became_peered":"2024-09-13T14:40:34.563989+0000","last_unstale":"2024-09-13T14:43:51.650263+0000","last_undegraded":"2024-09-13T14:43:51.650263+0000","last_fullsized":"2024-09-13T14:43:51.650263+0000","mapping_epoch":266,"log_start":"219'5300","ondisk_log_start":"219'5300","created":205,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5956,"log_dups_size":0,"ondisk_log_size":5956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:34:43.006051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188940312,"num_objects":2884,"num_object_clones":0,"num_object_copies":8652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2884,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11256,"num_write_kb":452417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4489,"num_bytes_recovered":289865752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":287,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.622874+0000","last_change":"2024-09-13T14:36:13.860367+0000","last_active":"2024-09-13T14:43:50.622874+0000","last_peered":"2024-09-13T14:43:50.622874+0000","last_clean":"2024-09-13T14:43:50.622874+0000","last_became_active":"2024-09-13T14:36:13.859982+0000","last_became_peered":"2024-09-13T14:36:13.859982+0000","last_unstale":"2024-09-13T14:43:50.622874+0000","last_undegraded":"2024-09-13T14:43:50.622874+0000","last_fullsized":"2024-09-13T14:43:50.622874+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:08:23.311350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":283,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.622725+0000","last_change":"2024-09-13T14:37:48.147360+0000","last_active":"2024-09-13T14:43:50.622725+0000","last_peered":"2024-09-13T14:43:50.622725+0000","last_clean":"2024-09-13T14:43:50.622725+0000","last_became_active":"2024-09-13T14:37:48.146885+0000","last_became_peered":"2024-09-13T14:37:48.146885+0000","last_unstale":"2024-09-13T14:43:50.622725+0000","last_undegraded":"2024-09-13T14:43:50.622725+0000","last_fullsized":"2024-09-13T14:43:50.622725+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:27.841810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"290'11158","reported_seq":20447,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:52.433958+0000","last_change":"2024-09-13T14:38:39.650332+0000","last_active":"2024-09-13T14:43:52.433958+0000","last_peered":"2024-09-13T14:43:52.433958+0000","last_clean":"2024-09-13T14:43:52.433958+0000","last_became_active":"2024-09-13T14:38:39.640835+0000","last_became_peered":"2024-09-13T14:38:39.640835+0000","last_unstale":"2024-09-13T14:43:52.433958+0000","last_undegraded":"2024-09-13T14:43:52.433958+0000","last_fullsized":"2024-09-13T14:43:52.433958+0000","mapping_epoch":251,"log_start":"219'5451","ondisk_log_start":"219'5451","created":205,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5707,"log_dups_size":0,"ondisk_log_size":5707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:48.176606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191102976,"num_objects":2916,"num_object_clones":0,"num_object_copies":8748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11158,"num_write_kb":450368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4544,"num_bytes_recovered":292945920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":480,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.626610+0000","last_change":"2024-09-13T14:38:53.900041+0000","last_active":"2024-09-13T14:43:51.626610+0000","last_peered":"2024-09-13T14:43:51.626610+0000","last_clean":"2024-09-13T14:43:51.626610+0000","last_became_active":"2024-09-13T14:38:53.899203+0000","last_became_peered":"2024-09-13T14:38:53.899203+0000","last_unstale":"2024-09-13T14:43:51.626610+0000","last_undegraded":"2024-09-13T14:43:51.626610+0000","last_fullsized":"2024-09-13T14:43:51.626610+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:28:07.883094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":412,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.625960+0000","last_change":"2024-09-13T14:26:15.823198+0000","last_active":"2024-09-13T14:43:51.625960+0000","last_peered":"2024-09-13T14:43:51.625960+0000","last_clean":"2024-09-13T14:43:51.625960+0000","last_became_active":"2024-09-13T14:26:15.822717+0000","last_became_peered":"2024-09-13T14:26:15.822717+0000","last_unstale":"2024-09-13T14:43:51.625960+0000","last_undegraded":"2024-09-13T14:43:51.625960+0000","last_fullsized":"2024-09-13T14:43:51.625960+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:22.471622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.6","version":"289'11283","reported_seq":19637,"reported_epoch":290,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-13T14:43:51.650624+0000","last_change":"2024-09-13T14:38:55.132683+0000","last_active":"2024-09-13T14:43:51.650624+0000","last_peered":"2024-09-13T14:43:51.650624+0000","last_clean":"2024-09-13T14:36:13.124368+0000","last_became_active":"2024-09-13T14:38:55.120883+0000","last_became_peered":"2024-09-13T14:38:55.120883+0000","last_unstale":"2024-09-13T14:43:51.650624+0000","last_undegraded":"2024-09-13T14:38:55.026610+0000","last_fullsized":"2024-09-13T14:43:51.650624+0000","mapping_epoch":254,"log_start":"219'5319","ondisk_log_start":"219'5319","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5964,"log_dups_size":0,"ondisk_log_size":5964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191037440,"num_objects":2915,"num_object_clones":0,"num_object_copies":8745,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":928,"num_objects_misplaced":1987,"num_objects_unfound":0,"num_objects_dirty":2915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11283,"num_write_kb":454336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4077,"num_bytes_recovered":255983616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,3,5],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3,4","objects":649},{"shards":"0,3,5","objects":2266}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":424,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.657167+0000","last_change":"2024-09-13T14:41:53.196835+0000","last_active":"2024-09-13T14:43:51.657167+0000","last_peered":"2024-09-13T14:43:51.657167+0000","last_clean":"2024-09-13T14:43:51.657167+0000","last_became_active":"2024-09-13T14:41:53.196387+0000","last_became_peered":"2024-09-13T14:41:53.196387+0000","last_unstale":"2024-09-13T14:43:51.657167+0000","last_undegraded":"2024-09-13T14:43:51.657167+0000","last_fullsized":"2024-09-13T14:43:51.657167+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:07:41.927740+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:49.972303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":529,"reported_epoch":289,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:43:50.620126+0000","last_change":"2024-09-13T14:38:54.904329+0000","last_active":"2024-09-13T14:43:50.620126+0000","last_peered":"2024-09-13T14:43:50.620126+0000","last_clean":"2024-09-13T14:38:39.648017+0000","last_became_active":"2024-09-13T14:38:54.903798+0000","last_became_peered":"2024-09-13T14:38:54.903798+0000","last_unstale":"2024-09-13T14:43:50.620126+0000","last_undegraded":"2024-09-13T14:38:54.890505+0000","last_fullsized":"2024-09-13T14:38:54.889701+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:18.120931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,3],"avail_no_missing":["5","3","6"],"object_location_counts":[{"shards":"3,5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":415,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.626125+0000","last_change":"2024-09-13T14:37:49.160719+0000","last_active":"2024-09-13T14:43:51.626125+0000","last_peered":"2024-09-13T14:43:51.626125+0000","last_clean":"2024-09-13T14:43:51.626125+0000","last_became_active":"2024-09-13T14:37:49.160401+0000","last_became_peered":"2024-09-13T14:37:49.160401+0000","last_unstale":"2024-09-13T14:43:51.626125+0000","last_undegraded":"2024-09-13T14:43:51.626125+0000","last_fullsized":"2024-09-13T14:43:51.626125+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:21.737945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":134,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.622540+0000","last_change":"2024-09-13T14:37:48.172940+0000","last_active":"2024-09-13T14:43:50.622540+0000","last_peered":"2024-09-13T14:43:50.622540+0000","last_clean":"2024-09-13T14:43:50.622540+0000","last_became_active":"2024-09-13T14:37:48.172060+0000","last_became_peered":"2024-09-13T14:37:48.172060+0000","last_unstale":"2024-09-13T14:43:50.622540+0000","last_undegraded":"2024-09-13T14:43:50.622540+0000","last_fullsized":"2024-09-13T14:43:50.622540+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:07.480645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.5","version":"290'11239","reported_seq":27613,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:52.304996+0000","last_change":"2024-09-13T14:39:34.446699+0000","last_active":"2024-09-13T14:43:52.304996+0000","last_peered":"2024-09-13T14:43:52.304996+0000","last_clean":"2024-09-13T14:43:52.304996+0000","last_became_active":"2024-09-13T14:38:04.600616+0000","last_became_peered":"2024-09-13T14:38:04.600616+0000","last_unstale":"2024-09-13T14:43:52.304996+0000","last_undegraded":"2024-09-13T14:43:52.304996+0000","last_fullsized":"2024-09-13T14:43:52.304996+0000","mapping_epoch":245,"log_start":"217'4916","ondisk_log_start":"217'4916","created":205,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6323,"log_dups_size":0,"ondisk_log_size":6323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:00:44.321678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190251008,"num_objects":2903,"num_object_clones":0,"num_object_copies":8709,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11239,"num_write_kb":452544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8120,"num_bytes_recovered":521076736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":507,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.690066+0000","last_change":"2024-09-13T14:26:15.835178+0000","last_active":"2024-09-13T14:43:50.690066+0000","last_peered":"2024-09-13T14:43:50.690066+0000","last_clean":"2024-09-13T14:43:50.690066+0000","last_became_active":"2024-09-13T14:26:15.820283+0000","last_became_peered":"2024-09-13T14:26:15.820283+0000","last_unstale":"2024-09-13T14:43:50.690066+0000","last_undegraded":"2024-09-13T14:43:50.690066+0000","last_fullsized":"2024-09-13T14:43:50.690066+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:30.340038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":104,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.649704+0000","last_change":"2024-09-13T14:38:03.525682+0000","last_active":"2024-09-13T14:43:51.649704+0000","last_peered":"2024-09-13T14:43:51.649704+0000","last_clean":"2024-09-13T14:43:51.649704+0000","last_became_active":"2024-09-13T14:38:03.525406+0000","last_became_peered":"2024-09-13T14:38:03.525406+0000","last_unstale":"2024-09-13T14:43:51.649704+0000","last_undegraded":"2024-09-13T14:43:51.649704+0000","last_fullsized":"2024-09-13T14:43:51.649704+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:41:28.970300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":111,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.650537+0000","last_change":"2024-09-13T14:38:53.991738+0000","last_active":"2024-09-13T14:43:51.650537+0000","last_peered":"2024-09-13T14:43:51.650537+0000","last_clean":"2024-09-13T14:43:51.650537+0000","last_became_active":"2024-09-13T14:38:53.991417+0000","last_became_peered":"2024-09-13T14:38:53.991417+0000","last_unstale":"2024-09-13T14:43:51.650537+0000","last_undegraded":"2024-09-13T14:43:51.650537+0000","last_fullsized":"2024-09-13T14:43:51.650537+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:50:38.559838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"289'11190","reported_seq":28387,"reported_epoch":290,"state":"active+recovering","last_fresh":"2024-09-13T14:43:51.668872+0000","last_change":"2024-09-13T14:43:50.028785+0000","last_active":"2024-09-13T14:43:51.668872+0000","last_peered":"2024-09-13T14:43:51.668872+0000","last_clean":"2024-09-13T14:37:46.290754+0000","last_became_active":"2024-09-13T14:38:17.381753+0000","last_became_peered":"2024-09-13T14:38:17.381753+0000","last_unstale":"2024-09-13T14:43:51.668872+0000","last_undegraded":"2024-09-13T14:43:51.668872+0000","last_fullsized":"2024-09-13T14:43:51.668872+0000","mapping_epoch":245,"log_start":"219'5259","ondisk_log_start":"219'5259","created":205,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5931,"log_dups_size":0,"ondisk_log_size":5931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191102976,"num_objects":2916,"num_object_clones":0,"num_object_copies":8748,"num_objects_missing_on_primary":83,"num_objects_missing":83,"num_objects_degraded":0,"num_objects_misplaced":2916,"num_objects_unfound":0,"num_objects_dirty":2916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11190,"num_write_kb":451392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8550,"num_bytes_recovered":560332800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"0,4,5","objects":2916}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":826,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.625800+0000","last_change":"2024-09-13T14:37:48.131799+0000","last_active":"2024-09-13T14:43:51.625800+0000","last_peered":"2024-09-13T14:43:51.625800+0000","last_clean":"2024-09-13T14:43:51.625800+0000","last_became_active":"2024-09-13T14:37:48.130837+0000","last_became_peered":"2024-09-13T14:37:48.130837+0000","last_unstale":"2024-09-13T14:43:51.625800+0000","last_undegraded":"2024-09-13T14:43:51.625800+0000","last_fullsized":"2024-09-13T14:43:51.625800+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:06:32.663770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":344,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.649946+0000","last_change":"2024-09-13T14:37:49.301531+0000","last_active":"2024-09-13T14:43:51.649946+0000","last_peered":"2024-09-13T14:43:51.649946+0000","last_clean":"2024-09-13T14:43:51.649946+0000","last_became_active":"2024-09-13T14:37:49.301249+0000","last_became_peered":"2024-09-13T14:37:49.301249+0000","last_unstale":"2024-09-13T14:43:51.649946+0000","last_undegraded":"2024-09-13T14:43:51.649946+0000","last_fullsized":"2024-09-13T14:43:51.649946+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:10.977710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":116,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.620681+0000","last_change":"2024-09-13T14:38:53.901427+0000","last_active":"2024-09-13T14:43:50.620681+0000","last_peered":"2024-09-13T14:43:50.620681+0000","last_clean":"2024-09-13T14:43:50.620681+0000","last_became_active":"2024-09-13T14:38:53.901157+0000","last_became_peered":"2024-09-13T14:38:53.901157+0000","last_unstale":"2024-09-13T14:43:50.620681+0000","last_undegraded":"2024-09-13T14:43:50.620681+0000","last_fullsized":"2024-09-13T14:43:50.620681+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:28:21.713025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"289'11124","reported_seq":34367,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.650688+0000","last_change":"2024-09-13T14:43:16.851222+0000","last_active":"2024-09-13T14:43:51.650688+0000","last_peered":"2024-09-13T14:43:51.650688+0000","last_clean":"2024-09-13T14:43:51.650688+0000","last_became_active":"2024-09-13T14:38:16.427510+0000","last_became_peered":"2024-09-13T14:38:16.427510+0000","last_unstale":"2024-09-13T14:43:51.650688+0000","last_undegraded":"2024-09-13T14:43:51.650688+0000","last_fullsized":"2024-09-13T14:43:51.650688+0000","mapping_epoch":245,"log_start":"219'5218","ondisk_log_start":"219'5218","created":205,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5906,"log_dups_size":0,"ondisk_log_size":5906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:27:26.281938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188350464,"num_objects":2874,"num_object_clones":0,"num_object_copies":8622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11124,"num_write_kb":447936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11571,"num_bytes_recovered":747372544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":998,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.619897+0000","last_change":"2024-09-13T14:37:49.147498+0000","last_active":"2024-09-13T14:43:50.619897+0000","last_peered":"2024-09-13T14:43:50.619897+0000","last_clean":"2024-09-13T14:43:50.619897+0000","last_became_active":"2024-09-13T14:37:49.147028+0000","last_became_peered":"2024-09-13T14:37:49.147028+0000","last_unstale":"2024-09-13T14:43:50.619897+0000","last_undegraded":"2024-09-13T14:43:50.619897+0000","last_fullsized":"2024-09-13T14:43:50.619897+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:11:29.050178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":296,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.622736+0000","last_change":"2024-09-13T14:38:03.528133+0000","last_active":"2024-09-13T14:43:50.622736+0000","last_peered":"2024-09-13T14:43:50.622736+0000","last_clean":"2024-09-13T14:43:50.622736+0000","last_became_active":"2024-09-13T14:38:03.527829+0000","last_became_peered":"2024-09-13T14:38:03.527829+0000","last_unstale":"2024-09-13T14:43:50.622736+0000","last_undegraded":"2024-09-13T14:43:50.622736+0000","last_fullsized":"2024-09-13T14:43:50.622736+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:48.254169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":464,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.626203+0000","last_change":"2024-09-13T14:38:53.899755+0000","last_active":"2024-09-13T14:43:51.626203+0000","last_peered":"2024-09-13T14:43:51.626203+0000","last_clean":"2024-09-13T14:43:51.626203+0000","last_became_active":"2024-09-13T14:38:53.899448+0000","last_became_peered":"2024-09-13T14:38:53.899448+0000","last_unstale":"2024-09-13T14:43:51.626203+0000","last_undegraded":"2024-09-13T14:43:51.626203+0000","last_fullsized":"2024-09-13T14:43:51.626203+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:55.897035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"290'11452","reported_seq":20584,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:52.712171+0000","last_change":"2024-09-13T14:43:51.649951+0000","last_active":"2024-09-13T14:43:52.712171+0000","last_peered":"2024-09-13T14:43:52.712171+0000","last_clean":"2024-09-13T14:43:52.712171+0000","last_became_active":"2024-09-13T14:43:51.626628+0000","last_became_peered":"2024-09-13T14:43:51.626628+0000","last_unstale":"2024-09-13T14:43:52.712171+0000","last_undegraded":"2024-09-13T14:43:52.712171+0000","last_fullsized":"2024-09-13T14:43:52.712171+0000","mapping_epoch":289,"log_start":"213'4863","ondisk_log_start":"213'4863","created":205,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6589,"log_dups_size":0,"ondisk_log_size":6589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:37:06.795173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186777600,"num_objects":2850,"num_object_clones":0,"num_object_copies":8550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11452,"num_write_kb":457664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4502,"num_bytes_recovered":228851712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":100,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.620326+0000","last_change":"2024-09-13T14:37:49.149770+0000","last_active":"2024-09-13T14:43:50.620326+0000","last_peered":"2024-09-13T14:43:50.620326+0000","last_clean":"2024-09-13T14:43:50.620326+0000","last_became_active":"2024-09-13T14:37:49.148991+0000","last_became_peered":"2024-09-13T14:37:49.148991+0000","last_unstale":"2024-09-13T14:43:50.620326+0000","last_undegraded":"2024-09-13T14:43:50.620326+0000","last_fullsized":"2024-09-13T14:43:50.620326+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:04:04.193366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":1163,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.667716+0000","last_change":"2024-09-13T14:30:58.928302+0000","last_active":"2024-09-13T14:43:51.667716+0000","last_peered":"2024-09-13T14:43:51.667716+0000","last_clean":"2024-09-13T14:43:51.667716+0000","last_became_active":"2024-09-13T14:30:58.927718+0000","last_became_peered":"2024-09-13T14:30:58.927718+0000","last_unstale":"2024-09-13T14:43:51.667716+0000","last_undegraded":"2024-09-13T14:43:51.667716+0000","last_fullsized":"2024-09-13T14:43:51.667716+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:37.323036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":107,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.620008+0000","last_change":"2024-09-13T14:38:03.534628+0000","last_active":"2024-09-13T14:43:50.620008+0000","last_peered":"2024-09-13T14:43:50.620008+0000","last_clean":"2024-09-13T14:43:50.620008+0000","last_became_active":"2024-09-13T14:38:03.534262+0000","last_became_peered":"2024-09-13T14:38:03.534262+0000","last_unstale":"2024-09-13T14:43:50.620008+0000","last_undegraded":"2024-09-13T14:43:50.620008+0000","last_fullsized":"2024-09-13T14:43:50.620008+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:31.168783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"290'11170","reported_seq":25348,"reported_epoch":290,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:43:52.287052+0000","last_change":"2024-09-13T14:38:53.944111+0000","last_active":"2024-09-13T14:43:52.287052+0000","last_peered":"2024-09-13T14:43:52.287052+0000","last_clean":"2024-09-13T14:37:47.047672+0000","last_became_active":"2024-09-13T14:37:49.664294+0000","last_became_peered":"2024-09-13T14:37:49.664294+0000","last_unstale":"2024-09-13T14:43:52.287052+0000","last_undegraded":"2024-09-13T14:38:53.940822+0000","last_fullsized":"2024-09-13T14:37:49.126627+0000","mapping_epoch":240,"log_start":"219'5500","ondisk_log_start":"219'5500","created":205,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5670,"log_dups_size":0,"ondisk_log_size":5670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:11.766692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188874752,"num_objects":2882,"num_object_clones":0,"num_object_copies":8646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4535,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11170,"num_write_kb":449664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7009,"num_bytes_recovered":459341824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3],"avail_no_missing":["5","0","3"],"object_location_counts":[{"shards":"0,3,5","objects":2882},{"shards":"3,5","objects":-1578}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":119,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.620290+0000","last_change":"2024-09-13T14:38:03.534303+0000","last_active":"2024-09-13T14:43:50.620290+0000","last_peered":"2024-09-13T14:43:50.620290+0000","last_clean":"2024-09-13T14:43:50.620290+0000","last_became_active":"2024-09-13T14:38:03.533171+0000","last_became_peered":"2024-09-13T14:38:03.533171+0000","last_unstale":"2024-09-13T14:43:50.620290+0000","last_undegraded":"2024-09-13T14:43:50.620290+0000","last_fullsized":"2024-09-13T14:43:50.620290+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:00:16.553526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":812,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.690438+0000","last_change":"2024-09-13T14:38:04.596192+0000","last_active":"2024-09-13T14:43:50.690438+0000","last_peered":"2024-09-13T14:43:50.690438+0000","last_clean":"2024-09-13T14:43:50.690438+0000","last_became_active":"2024-09-13T14:38:04.593299+0000","last_became_peered":"2024-09-13T14:38:04.593299+0000","last_unstale":"2024-09-13T14:43:50.690438+0000","last_undegraded":"2024-09-13T14:43:50.690438+0000","last_fullsized":"2024-09-13T14:43:50.690438+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:34.241410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":104,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.650109+0000","last_change":"2024-09-13T14:38:03.533120+0000","last_active":"2024-09-13T14:43:51.650109+0000","last_peered":"2024-09-13T14:43:51.650109+0000","last_clean":"2024-09-13T14:43:51.650109+0000","last_became_active":"2024-09-13T14:38:03.532829+0000","last_became_peered":"2024-09-13T14:38:03.532829+0000","last_unstale":"2024-09-13T14:43:51.650109+0000","last_undegraded":"2024-09-13T14:43:51.650109+0000","last_fullsized":"2024-09-13T14:43:51.650109+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:24.514661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"290'11377","reported_seq":20646,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:52.429116+0000","last_change":"2024-09-13T14:42:04.660877+0000","last_active":"2024-09-13T14:43:52.429116+0000","last_peered":"2024-09-13T14:43:52.429116+0000","last_clean":"2024-09-13T14:43:52.429116+0000","last_became_active":"2024-09-13T14:42:04.641680+0000","last_became_peered":"2024-09-13T14:42:04.641680+0000","last_unstale":"2024-09-13T14:43:52.429116+0000","last_undegraded":"2024-09-13T14:43:52.429116+0000","last_fullsized":"2024-09-13T14:43:52.429116+0000","mapping_epoch":277,"log_start":"219'5618","ondisk_log_start":"219'5618","created":205,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5759,"log_dups_size":0,"ondisk_log_size":5759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:22:18.955428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197984256,"num_objects":3021,"num_object_clones":0,"num_object_copies":9063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11377,"num_write_kb":460736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4533,"num_bytes_recovered":297074688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":506,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.626040+0000","last_change":"2024-09-13T14:38:53.938227+0000","last_active":"2024-09-13T14:43:51.626040+0000","last_peered":"2024-09-13T14:43:51.626040+0000","last_clean":"2024-09-13T14:43:51.626040+0000","last_became_active":"2024-09-13T14:38:53.902867+0000","last_became_peered":"2024-09-13T14:38:53.902867+0000","last_unstale":"2024-09-13T14:43:51.626040+0000","last_undegraded":"2024-09-13T14:43:51.626040+0000","last_fullsized":"2024-09-13T14:43:51.626040+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:16.831756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":590,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.622816+0000","last_change":"2024-09-13T14:37:48.147677+0000","last_active":"2024-09-13T14:43:50.622816+0000","last_peered":"2024-09-13T14:43:50.622816+0000","last_clean":"2024-09-13T14:43:50.622816+0000","last_became_active":"2024-09-13T14:37:48.147001+0000","last_became_peered":"2024-09-13T14:37:48.147001+0000","last_unstale":"2024-09-13T14:43:50.622816+0000","last_undegraded":"2024-09-13T14:43:50.622816+0000","last_fullsized":"2024-09-13T14:43:50.622816+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:35:00.653576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":488,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.626531+0000","last_change":"2024-09-13T14:38:53.900107+0000","last_active":"2024-09-13T14:43:51.626531+0000","last_peered":"2024-09-13T14:43:51.626531+0000","last_clean":"2024-09-13T14:43:51.626531+0000","last_became_active":"2024-09-13T14:38:53.899556+0000","last_became_peered":"2024-09-13T14:38:53.899556+0000","last_unstale":"2024-09-13T14:43:51.626531+0000","last_undegraded":"2024-09-13T14:43:51.626531+0000","last_fullsized":"2024-09-13T14:43:51.626531+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:15:47.918163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"290'10973","reported_seq":19968,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:52.447143+0000","last_change":"2024-09-13T14:39:54.921239+0000","last_active":"2024-09-13T14:43:52.447143+0000","last_peered":"2024-09-13T14:43:52.447143+0000","last_clean":"2024-09-13T14:43:52.447143+0000","last_became_active":"2024-09-13T14:39:54.907894+0000","last_became_peered":"2024-09-13T14:39:54.907894+0000","last_unstale":"2024-09-13T14:43:52.447143+0000","last_undegraded":"2024-09-13T14:43:52.447143+0000","last_fullsized":"2024-09-13T14:43:52.447143+0000","mapping_epoch":263,"log_start":"219'5247","ondisk_log_start":"219'5247","created":205,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5726,"log_dups_size":0,"ondisk_log_size":5726,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:52:04.619238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188678144,"num_objects":2879,"num_object_clones":0,"num_object_copies":8637,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10973,"num_write_kb":443264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4395,"num_bytes_recovered":283770880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":119,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.650419+0000","last_change":"2024-09-13T14:38:53.901806+0000","last_active":"2024-09-13T14:43:51.650419+0000","last_peered":"2024-09-13T14:43:51.650419+0000","last_clean":"2024-09-13T14:43:51.650419+0000","last_became_active":"2024-09-13T14:38:53.891055+0000","last_became_peered":"2024-09-13T14:38:53.891055+0000","last_unstale":"2024-09-13T14:43:51.650419+0000","last_undegraded":"2024-09-13T14:43:51.650419+0000","last_fullsized":"2024-09-13T14:43:51.650419+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:03:00.416198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":65,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.690702+0000","last_change":"2024-09-13T14:43:16.854299+0000","last_active":"2024-09-13T14:43:50.690702+0000","last_peered":"2024-09-13T14:43:50.690702+0000","last_clean":"2024-09-13T14:43:50.690702+0000","last_became_active":"2024-09-13T14:38:53.900626+0000","last_became_peered":"2024-09-13T14:38:53.900626+0000","last_unstale":"2024-09-13T14:43:50.690702+0000","last_undegraded":"2024-09-13T14:43:50.690702+0000","last_fullsized":"2024-09-13T14:43:50.690702+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:37:21.715778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":280,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.622385+0000","last_change":"2024-09-13T14:37:49.172218+0000","last_active":"2024-09-13T14:43:50.622385+0000","last_peered":"2024-09-13T14:43:50.622385+0000","last_clean":"2024-09-13T14:43:50.622385+0000","last_became_active":"2024-09-13T14:37:49.171718+0000","last_became_peered":"2024-09-13T14:37:49.171718+0000","last_unstale":"2024-09-13T14:43:50.622385+0000","last_undegraded":"2024-09-13T14:43:50.622385+0000","last_fullsized":"2024-09-13T14:43:50.622385+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:46:54.626521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":302,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.622454+0000","last_change":"2024-09-13T14:37:49.230094+0000","last_active":"2024-09-13T14:43:50.622454+0000","last_peered":"2024-09-13T14:43:50.622454+0000","last_clean":"2024-09-13T14:43:50.622454+0000","last_became_active":"2024-09-13T14:37:49.229393+0000","last_became_peered":"2024-09-13T14:37:49.229393+0000","last_unstale":"2024-09-13T14:43:50.622454+0000","last_undegraded":"2024-09-13T14:43:50.622454+0000","last_fullsized":"2024-09-13T14:43:50.622454+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:02:28.690318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":282,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.622830+0000","last_change":"2024-09-13T14:37:48.128447+0000","last_active":"2024-09-13T14:43:50.622830+0000","last_peered":"2024-09-13T14:43:50.622830+0000","last_clean":"2024-09-13T14:43:50.622830+0000","last_became_active":"2024-09-13T14:37:48.128143+0000","last_became_peered":"2024-09-13T14:37:48.128143+0000","last_unstale":"2024-09-13T14:43:50.622830+0000","last_undegraded":"2024-09-13T14:43:50.622830+0000","last_fullsized":"2024-09-13T14:43:50.622830+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:22.204039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":104,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.649966+0000","last_change":"2024-09-13T14:38:03.531683+0000","last_active":"2024-09-13T14:43:51.649966+0000","last_peered":"2024-09-13T14:43:51.649966+0000","last_clean":"2024-09-13T14:43:51.649966+0000","last_became_active":"2024-09-13T14:38:03.531476+0000","last_became_peered":"2024-09-13T14:38:03.531476+0000","last_unstale":"2024-09-13T14:43:51.649966+0000","last_undegraded":"2024-09-13T14:43:51.649966+0000","last_fullsized":"2024-09-13T14:43:51.649966+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:41.557569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":55,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.690220+0000","last_change":"2024-09-13T14:38:53.954281+0000","last_active":"2024-09-13T14:43:50.690220+0000","last_peered":"2024-09-13T14:43:50.690220+0000","last_clean":"2024-09-13T14:43:50.690220+0000","last_became_active":"2024-09-13T14:38:53.900912+0000","last_became_peered":"2024-09-13T14:38:53.900912+0000","last_unstale":"2024-09-13T14:43:50.690220+0000","last_undegraded":"2024-09-13T14:43:50.690220+0000","last_fullsized":"2024-09-13T14:43:50.690220+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:47.876275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":107,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.620645+0000","last_change":"2024-09-13T14:38:54.000782+0000","last_active":"2024-09-13T14:43:50.620645+0000","last_peered":"2024-09-13T14:43:50.620645+0000","last_clean":"2024-09-13T14:43:50.620645+0000","last_became_active":"2024-09-13T14:38:54.000322+0000","last_became_peered":"2024-09-13T14:38:54.000322+0000","last_unstale":"2024-09-13T14:43:50.620645+0000","last_undegraded":"2024-09-13T14:43:50.620645+0000","last_fullsized":"2024-09-13T14:43:50.620645+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:09.057597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":430,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.625481+0000","last_change":"2024-09-13T14:38:53.899535+0000","last_active":"2024-09-13T14:43:51.625481+0000","last_peered":"2024-09-13T14:43:51.625481+0000","last_clean":"2024-09-13T14:43:51.625481+0000","last_became_active":"2024-09-13T14:38:53.899183+0000","last_became_peered":"2024-09-13T14:38:53.899183+0000","last_unstale":"2024-09-13T14:43:51.625481+0000","last_undegraded":"2024-09-13T14:43:51.625481+0000","last_fullsized":"2024-09-13T14:43:51.625481+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:33.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":104,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.650600+0000","last_change":"2024-09-13T14:38:03.543426+0000","last_active":"2024-09-13T14:43:51.650600+0000","last_peered":"2024-09-13T14:43:51.650600+0000","last_clean":"2024-09-13T14:43:51.650600+0000","last_became_active":"2024-09-13T14:38:03.542661+0000","last_became_peered":"2024-09-13T14:38:03.542661+0000","last_unstale":"2024-09-13T14:43:51.650600+0000","last_undegraded":"2024-09-13T14:43:51.650600+0000","last_fullsized":"2024-09-13T14:43:51.650600+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:54.050127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":459,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.626504+0000","last_change":"2024-09-13T14:37:48.131321+0000","last_active":"2024-09-13T14:43:51.626504+0000","last_peered":"2024-09-13T14:43:51.626504+0000","last_clean":"2024-09-13T14:43:51.626504+0000","last_became_active":"2024-09-13T14:37:48.130998+0000","last_became_peered":"2024-09-13T14:37:48.130998+0000","last_unstale":"2024-09-13T14:43:51.626504+0000","last_undegraded":"2024-09-13T14:43:51.626504+0000","last_fullsized":"2024-09-13T14:43:51.626504+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:33:26.597825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":119,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.620641+0000","last_change":"2024-09-13T14:38:03.535454+0000","last_active":"2024-09-13T14:43:50.620641+0000","last_peered":"2024-09-13T14:43:50.620641+0000","last_clean":"2024-09-13T14:43:50.620641+0000","last_became_active":"2024-09-13T14:38:03.535131+0000","last_became_peered":"2024-09-13T14:38:03.535131+0000","last_unstale":"2024-09-13T14:43:50.620641+0000","last_undegraded":"2024-09-13T14:43:50.620641+0000","last_fullsized":"2024-09-13T14:43:50.620641+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:24.477601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":483,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.626587+0000","last_change":"2024-09-13T14:38:03.524939+0000","last_active":"2024-09-13T14:43:51.626587+0000","last_peered":"2024-09-13T14:43:51.626587+0000","last_clean":"2024-09-13T14:43:51.626587+0000","last_became_active":"2024-09-13T14:38:03.524406+0000","last_became_peered":"2024-09-13T14:38:03.524406+0000","last_unstale":"2024-09-13T14:43:51.626587+0000","last_undegraded":"2024-09-13T14:43:51.626587+0000","last_fullsized":"2024-09-13T14:43:51.626587+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:19:45.257424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":111,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.650398+0000","last_change":"2024-09-13T14:38:53.892948+0000","last_active":"2024-09-13T14:43:51.650398+0000","last_peered":"2024-09-13T14:43:51.650398+0000","last_clean":"2024-09-13T14:43:51.650398+0000","last_became_active":"2024-09-13T14:38:53.892665+0000","last_became_peered":"2024-09-13T14:38:53.892665+0000","last_unstale":"2024-09-13T14:43:51.650398+0000","last_undegraded":"2024-09-13T14:43:51.650398+0000","last_fullsized":"2024-09-13T14:43:51.650398+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:08.292359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":432,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.625674+0000","last_change":"2024-09-13T14:37:48.139348+0000","last_active":"2024-09-13T14:43:51.625674+0000","last_peered":"2024-09-13T14:43:51.625674+0000","last_clean":"2024-09-13T14:43:51.625674+0000","last_became_active":"2024-09-13T14:37:48.139115+0000","last_became_peered":"2024-09-13T14:37:48.139115+0000","last_unstale":"2024-09-13T14:43:51.625674+0000","last_undegraded":"2024-09-13T14:43:51.625674+0000","last_fullsized":"2024-09-13T14:43:51.625674+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:32:20.413346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":472,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.626096+0000","last_change":"2024-09-13T14:37:48.172088+0000","last_active":"2024-09-13T14:43:51.626096+0000","last_peered":"2024-09-13T14:43:51.626096+0000","last_clean":"2024-09-13T14:43:51.626096+0000","last_became_active":"2024-09-13T14:37:48.171779+0000","last_became_peered":"2024-09-13T14:37:48.171779+0000","last_unstale":"2024-09-13T14:43:51.626096+0000","last_undegraded":"2024-09-13T14:43:51.626096+0000","last_fullsized":"2024-09-13T14:43:51.626096+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:02.852712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":346,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.622498+0000","last_change":"2024-09-13T14:38:53.890236+0000","last_active":"2024-09-13T14:43:50.622498+0000","last_peered":"2024-09-13T14:43:50.622498+0000","last_clean":"2024-09-13T14:43:50.622498+0000","last_became_active":"2024-09-13T14:38:53.889582+0000","last_became_peered":"2024-09-13T14:38:53.889582+0000","last_unstale":"2024-09-13T14:43:50.622498+0000","last_undegraded":"2024-09-13T14:43:50.622498+0000","last_fullsized":"2024-09-13T14:43:50.622498+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:14:29.334774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":458,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.626361+0000","last_change":"2024-09-13T14:38:03.534097+0000","last_active":"2024-09-13T14:43:51.626361+0000","last_peered":"2024-09-13T14:43:51.626361+0000","last_clean":"2024-09-13T14:43:51.626361+0000","last_became_active":"2024-09-13T14:38:03.533794+0000","last_became_peered":"2024-09-13T14:38:03.533794+0000","last_unstale":"2024-09-13T14:43:51.626361+0000","last_undegraded":"2024-09-13T14:43:51.626361+0000","last_fullsized":"2024-09-13T14:43:51.626361+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:23.272098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":104,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.649969+0000","last_change":"2024-09-13T14:38:03.527021+0000","last_active":"2024-09-13T14:43:51.649969+0000","last_peered":"2024-09-13T14:43:51.649969+0000","last_clean":"2024-09-13T14:43:51.649969+0000","last_became_active":"2024-09-13T14:38:03.526698+0000","last_became_peered":"2024-09-13T14:38:03.526698+0000","last_unstale":"2024-09-13T14:43:51.649969+0000","last_undegraded":"2024-09-13T14:43:51.649969+0000","last_fullsized":"2024-09-13T14:43:51.649969+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:01:35.734086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":290,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.622538+0000","last_change":"2024-09-13T14:27:12.834931+0000","last_active":"2024-09-13T14:43:50.622538+0000","last_peered":"2024-09-13T14:43:50.622538+0000","last_clean":"2024-09-13T14:43:50.622538+0000","last_became_active":"2024-09-13T14:27:12.834572+0000","last_became_peered":"2024-09-13T14:27:12.834572+0000","last_unstale":"2024-09-13T14:43:50.622538+0000","last_undegraded":"2024-09-13T14:43:50.622538+0000","last_fullsized":"2024-09-13T14:43:50.622538+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:54:21.155689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":104,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.620211+0000","last_change":"2024-09-13T14:37:49.176796+0000","last_active":"2024-09-13T14:43:50.620211+0000","last_peered":"2024-09-13T14:43:50.620211+0000","last_clean":"2024-09-13T14:43:50.620211+0000","last_became_active":"2024-09-13T14:37:49.176246+0000","last_became_peered":"2024-09-13T14:37:49.176246+0000","last_unstale":"2024-09-13T14:43:50.620211+0000","last_undegraded":"2024-09-13T14:43:50.620211+0000","last_fullsized":"2024-09-13T14:43:50.620211+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:29:24.936718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":301,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.622250+0000","last_change":"2024-09-13T14:38:03.526333+0000","last_active":"2024-09-13T14:43:50.622250+0000","last_peered":"2024-09-13T14:43:50.622250+0000","last_clean":"2024-09-13T14:43:50.622250+0000","last_became_active":"2024-09-13T14:38:03.526001+0000","last_became_peered":"2024-09-13T14:38:03.526001+0000","last_unstale":"2024-09-13T14:43:50.622250+0000","last_undegraded":"2024-09-13T14:43:50.622250+0000","last_fullsized":"2024-09-13T14:43:50.622250+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:25:25.181271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":104,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.620403+0000","last_change":"2024-09-13T14:37:49.149582+0000","last_active":"2024-09-13T14:43:50.620403+0000","last_peered":"2024-09-13T14:43:50.620403+0000","last_clean":"2024-09-13T14:43:50.620403+0000","last_became_active":"2024-09-13T14:37:49.148525+0000","last_became_peered":"2024-09-13T14:37:49.148525+0000","last_unstale":"2024-09-13T14:43:50.620403+0000","last_undegraded":"2024-09-13T14:43:50.620403+0000","last_fullsized":"2024-09-13T14:43:50.620403+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:48.655665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":504,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.625858+0000","last_change":"2024-09-13T14:38:03.535255+0000","last_active":"2024-09-13T14:43:51.625858+0000","last_peered":"2024-09-13T14:43:51.625858+0000","last_clean":"2024-09-13T14:43:51.625858+0000","last_became_active":"2024-09-13T14:38:03.533917+0000","last_became_peered":"2024-09-13T14:38:03.533917+0000","last_unstale":"2024-09-13T14:43:51.625858+0000","last_undegraded":"2024-09-13T14:43:51.625858+0000","last_fullsized":"2024-09-13T14:43:51.625858+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:33.559788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":468,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.626797+0000","last_change":"2024-09-13T14:37:48.146784+0000","last_active":"2024-09-13T14:43:51.626797+0000","last_peered":"2024-09-13T14:43:51.626797+0000","last_clean":"2024-09-13T14:43:51.626797+0000","last_became_active":"2024-09-13T14:37:48.146491+0000","last_became_peered":"2024-09-13T14:37:48.146491+0000","last_unstale":"2024-09-13T14:43:51.626797+0000","last_undegraded":"2024-09-13T14:43:51.626797+0000","last_fullsized":"2024-09-13T14:43:51.626797+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:30:54.772513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":282,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.622204+0000","last_change":"2024-09-13T14:37:48.127607+0000","last_active":"2024-09-13T14:43:50.622204+0000","last_peered":"2024-09-13T14:43:50.622204+0000","last_clean":"2024-09-13T14:43:50.622204+0000","last_became_active":"2024-09-13T14:37:48.127316+0000","last_became_peered":"2024-09-13T14:37:48.127316+0000","last_unstale":"2024-09-13T14:43:50.622204+0000","last_undegraded":"2024-09-13T14:43:50.622204+0000","last_fullsized":"2024-09-13T14:43:50.622204+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:53:37.678861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":110,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.650547+0000","last_change":"2024-09-13T14:38:03.563200+0000","last_active":"2024-09-13T14:43:51.650547+0000","last_peered":"2024-09-13T14:43:51.650547+0000","last_clean":"2024-09-13T14:43:51.650547+0000","last_became_active":"2024-09-13T14:38:03.534258+0000","last_became_peered":"2024-09-13T14:38:03.534258+0000","last_unstale":"2024-09-13T14:43:51.650547+0000","last_undegraded":"2024-09-13T14:43:51.650547+0000","last_fullsized":"2024-09-13T14:43:51.650547+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:49:07.814419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":3032088600,"num_objects":46267,"num_object_clones":0,"num_object_copies":138801,"num_objects_missing_on_primary":83,"num_objects_missing":83,"num_objects_degraded":5463,"num_objects_misplaced":4903,"num_objects_unfound":0,"num_objects_dirty":46267,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":179359,"num_write_kb":7219969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83725,"num_bytes_recovered":5359992856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11391021056,"data_stored":11391008840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94782,"ondisk_log_size":94782,"up":48,"acting":47,"num_store_stats":6},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1538,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":17390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25132,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16793600,"data_stored":16777292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":23,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2969600,"data_stored":2951840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234400,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2190728,"kb_used_data":1718264,"kb_used_omap":1,"kb_used_meta":472446,"kb_avail":91554424,"statfs":{"total":95995035648,"available":93751730176,"internally_reserved":0,"allocated":1759502336,"data_stored":1757950137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":483785163},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":239,"seq":1026497183818,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1794876,"kb_used_data":1507760,"kb_used_omap":8,"kb_used_meta":287095,"kb_avail":91950276,"statfs":{"total":95995035648,"available":94157082624,"internally_reserved":0,"allocated":1543946240,"data_stored":1542394532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8453,"internal_metadata":293986043},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757665,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1983988,"kb_used_data":1321192,"kb_used_omap":1,"kb_used_meta":662782,"kb_avail":91761164,"statfs":{"total":95995035648,"available":93963431936,"internally_reserved":0,"allocated":1352900608,"data_stored":1351359970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":678689222},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019309,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2201984,"kb_used_data":1599088,"kb_used_omap":1,"kb_used_meta":602878,"kb_avail":91543168,"statfs":{"total":95995035648,"available":93740204032,"internally_reserved":0,"allocated":1637466112,"data_stored":1635921549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":617347526},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435492,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2794928,"kb_used_data":2225956,"kb_used_omap":5,"kb_used_meta":568954,"kb_avail":90950224,"statfs":{"total":95995035648,"available":93133029376,"internally_reserved":0,"allocated":2279378944,"data_stored":2277842166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":582609022},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":446676599020,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":425201762546,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926070,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3480436,"kb_used_data":2803176,"kb_used_omap":7,"kb_used_meta":677240,"kb_avail":90264716,"statfs":{"total":95995035648,"available":92431069184,"internally_reserved":0,"allocated":2870452224,"data_stored":2868904245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":693494056},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6860,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2860257280,"data_stored":2860253208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2268991488,"data_stored":2268987416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1629159424,"data_stored":1629159424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1343815680,"data_stored":1343815680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1534267392,"data_stored":1534263320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1754529792,"data_stored":1754529792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:43:55.120 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:55.146 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:55.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:55.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:55.248 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:55.271 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:55.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:55.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:55.373 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:55.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:55.400 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:55.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:55.501 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:55.532 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:55.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:55.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:55.634 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:55.647 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T14:43:55.662 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:55.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:55.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:55.764 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:55.787 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:55.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:55.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:55.886 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:43:55.889 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:55.914 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:55.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:55.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:56.016 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:56.043 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:56.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:56.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:56.145 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:56.171 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:56.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:56.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:56.273 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:56.304 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:56.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:56.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:56.361 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:56.407 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:56.430 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:56.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:56.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:56.532 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:56.557 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:56.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:56.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:56.641 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:43:56 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:43:56] "GET /metrics HTTP/1.1" 200 36549 "" "Prometheus/2.51.0" 2024-09-13T14:43:56.642 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:43:56.642 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:43:56.642 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:56 smithi102 ceph-mon[110183]: pgmap v998: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.8 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 140 op/s; 5472/138885 objects degraded (3.940%); 4903/138885 objects misplaced (3.530%); 6.2 MiB/s, 225 objects/s recovering 2024-09-13T14:43:56.642 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:43:56.642 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:43:56.642 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:56 smithi102 ceph-mon[115269]: pgmap v998: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.8 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 140 op/s; 5472/138885 objects degraded (3.940%); 4903/138885 objects misplaced (3.530%); 6.2 MiB/s, 225 objects/s recovering 2024-09-13T14:43:56.659 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:56.680 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:56.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:56.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:56.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:43:56.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:43:56.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:56 smithi195 ceph-mon[104262]: pgmap v998: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.8 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 140 op/s; 5472/138885 objects degraded (3.940%); 4903/138885 objects misplaced (3.530%); 6.2 MiB/s, 225 objects/s recovering 2024-09-13T14:43:56.783 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:56.810 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:56.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:56.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:56.911 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:56.935 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:56.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:56.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:57.040 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:57.065 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:57.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:57.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:57.167 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:57.189 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:57.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:57.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:57.291 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:57.314 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:57.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:57.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:57.416 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:57.440 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:57.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:57.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:57.441 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is set 2024-09-13T14:43:57.542 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:57.564 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:57.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:57.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:57.666 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:57.667 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:43:57.667 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-13T14:43:57.667 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 0 2024-09-13T14:43:57.667 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:43:57.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:57 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3198554199' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:43:57.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:57 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:43:57.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:57 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3198554199' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:43:57.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:57 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:43:57.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:57.691 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:57.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:57.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:57 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3198554199' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:43:57.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:57 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:43:57.792 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:57.816 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:57.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:57.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:57.918 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:57.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:57.945 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:57.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:58.046 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:58.071 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:58.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:58.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:58.108 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:43:58.173 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:58.198 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:58.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:58.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:58.300 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:58.321 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:58.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:58.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:58.342 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T14:43:58.423 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:58.445 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:58.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:58.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:58.547 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:58.563 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:43:58.569 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:58.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:58.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:58.671 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:58.687 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:58 smithi195 ceph-mon[104262]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:43:58.687 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:58 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:43:58.687 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:58 smithi195 ceph-mon[104262]: osdmap e291: 8 total, 8 up, 6 in 2024-09-13T14:43:58.687 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:58 smithi195 ceph-mon[104262]: pgmap v1000: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.7 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 363 op/s; 5379/134418 objects degraded (4.002%); 4816/134418 objects misplaced (3.583%); 3.0 MiB/s, 151 objects/s recovering 2024-09-13T14:43:58.693 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:58.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:58.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:58.744 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:58 smithi102 ceph-mon[110183]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:43:58.744 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:58 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:43:58.744 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:58 smithi102 ceph-mon[110183]: osdmap e291: 8 total, 8 up, 6 in 2024-09-13T14:43:58.745 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:58 smithi102 ceph-mon[110183]: pgmap v1000: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.7 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 363 op/s; 5379/134418 objects degraded (4.002%); 4816/134418 objects misplaced (3.583%); 3.0 MiB/s, 151 objects/s recovering 2024-09-13T14:43:58.745 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:58 smithi102 ceph-mon[115269]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:43:58.745 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:58 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:43:58.745 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:58 smithi102 ceph-mon[115269]: osdmap e291: 8 total, 8 up, 6 in 2024-09-13T14:43:58.745 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:58 smithi102 ceph-mon[115269]: pgmap v1000: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.7 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 363 op/s; 5379/134418 objects degraded (4.002%); 4816/134418 objects misplaced (3.583%); 3.0 MiB/s, 151 objects/s recovering 2024-09-13T14:43:58.795 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:58.818 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:58.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:58.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:58.919 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:58.943 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:58.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:58.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:59.044 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:59.068 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:59.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:59.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:59.169 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:59.193 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:59.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:59.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:59.297 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:59.318 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:59.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:59.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:59.381 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:43:59.381 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:43:59.381 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:43:59.381 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:43:59.381 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:43:59.381 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:43:59.382 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:43:59.382 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:43:59.382 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:43:59.382 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:43:59.382 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:43:59.382 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:43:59.382 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:43:59.382 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:43:59.383 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:43:59.383 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:43:59.420 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:59.441 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:43:59.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:59.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:59.543 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:59.568 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:59.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:59.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:59.629 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:59.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:59 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/59437455' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:43:59.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:43:59 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 5379/134418 objects degraded (4.002%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:43:59.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:59 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/59437455' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:43:59.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:43:59 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 5379/134418 objects degraded (4.002%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:43:59.670 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:59.691 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:59.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:59.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:59.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:59 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/59437455' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:43:59.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:43:59 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 5379/134418 objects degraded (4.002%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:43:59.793 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:59.815 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:59.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:59.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:59.917 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:59.945 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:43:59.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:59.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:00.047 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:00.072 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:00.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:00.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:00.174 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:00.197 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:00.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:00.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:00.299 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:00.321 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:00.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:00.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:00.423 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:00.447 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:00.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:00.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:00.482 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:00 smithi195 ceph-mon[104262]: pgmap v1001: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.7 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 325 op/s; 5379/134136 objects degraded (4.010%); 4816/134136 objects misplaced (3.590%); 2.6 MiB/s, 130 objects/s recovering 2024-09-13T14:44:00.482 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:44:00.482 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:44:00.548 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:00.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:00.570 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:00.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:00.671 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:00.697 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:00.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:00.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:00.799 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:00.820 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:00.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:00.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:00.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:00 smithi102 ceph-mon[110183]: pgmap v1001: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.7 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 325 op/s; 5379/134136 objects degraded (4.010%); 4816/134136 objects misplaced (3.590%); 2.6 MiB/s, 130 objects/s recovering 2024-09-13T14:44:00.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:00 smithi102 ceph-mon[115269]: pgmap v1001: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.7 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 325 op/s; 5379/134136 objects degraded (4.010%); 4816/134136 objects misplaced (3.590%); 2.6 MiB/s, 130 objects/s recovering 2024-09-13T14:44:00.923 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:00.945 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:00.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:00.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:01.047 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:01.072 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:01.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:01.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:01.162 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T14:44:01.173 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:01.200 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:01.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:01.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:01.291 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:44:01.302 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:01.362 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:01.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:01.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:01.393 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:44:01.464 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:01.486 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:01.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:01.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:01.588 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:01.614 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:01.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:01.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:01.716 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:01.739 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:01.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:01.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:01.841 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:01.870 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:01.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:01.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:01.972 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:01.993 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:01.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:01.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:02.095 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:02.116 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:02.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:02.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:02.218 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:02.241 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:02.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:02.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:02.307 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:02.342 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:02.372 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:02.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:02.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:02.472 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:44:02.473 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:44:02.474 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:02.507 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:02.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:02.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:02.609 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:02.633 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:02.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:02.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:02.735 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:02.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:02 smithi195 ceph-mon[104262]: pgmap v1002: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.7 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 262 op/s; 5379/134136 objects degraded (4.010%); 4816/134136 objects misplaced (3.590%); 2.1 MiB/s, 105 objects/s recovering 2024-09-13T14:44:02.764 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:02.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:02.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:02.865 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:02.892 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:02.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:02.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:02.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:02 smithi102 ceph-mon[110183]: pgmap v1002: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.7 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 262 op/s; 5379/134136 objects degraded (4.010%); 4816/134136 objects misplaced (3.590%); 2.1 MiB/s, 105 objects/s recovering 2024-09-13T14:44:02.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:02 smithi102 ceph-mon[115269]: pgmap v1002: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.7 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 262 op/s; 5379/134136 objects degraded (4.010%); 4816/134136 objects misplaced (3.590%); 2.1 MiB/s, 105 objects/s recovering 2024-09-13T14:44:02.994 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:03.019 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:03.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:03.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:03.121 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:03.144 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:03.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:03.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:03.246 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:03.268 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:03.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:03.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:03.371 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:03.392 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:03.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:03.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:03.494 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:03.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:03.517 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:03.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:03.619 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:03.642 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:03.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:03.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:03.744 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:03.754 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:44:03.767 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:03.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:03.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:03.817 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:44:03.817 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:44:03.818 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:03 smithi195 ceph-mon[104262]: from='client.36024 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:44:03.868 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:03.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:03.893 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:03.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:03.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:03 smithi102 ceph-mon[115269]: from='client.36024 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:44:03.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:03 smithi102 ceph-mon[110183]: from='client.36024 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:44:03.995 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:04.020 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:04.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:04.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:04.121 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:04.144 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:04.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:04.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:04.245 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:04.270 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:04.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:04.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:04.372 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:04.393 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:04.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:04.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:04.478 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T14:44:04.495 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:04.516 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:04.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:04.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:04.618 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:04.644 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:04.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:04.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:04.727 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:44:04.746 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:04.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:04 smithi195 ceph-mon[104262]: pgmap v1003: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.6 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 314 op/s; 5286/129435 objects degraded (4.084%); 4724/129435 objects misplaced (3.650%); 2.3 MiB/s, 36 objects/s recovering 2024-09-13T14:44:04.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:04 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 5286/129435 objects degraded (4.084%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:04.773 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:04.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:04.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:04.875 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:04.897 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:04.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:04.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:04.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:04 smithi102 ceph-mon[115269]: pgmap v1003: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.6 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 314 op/s; 5286/129435 objects degraded (4.084%); 4724/129435 objects misplaced (3.650%); 2.3 MiB/s, 36 objects/s recovering 2024-09-13T14:44:04.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:04 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 5286/129435 objects degraded (4.084%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:04.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:04 smithi102 ceph-mon[110183]: pgmap v1003: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.6 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 314 op/s; 5286/129435 objects degraded (4.084%); 4724/129435 objects misplaced (3.650%); 2.3 MiB/s, 36 objects/s recovering 2024-09-13T14:44:04.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:04 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 5286/129435 objects degraded (4.084%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:04.999 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:05.025 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:05.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:05.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:05.052 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":1002,"stamp":"2024-09-13T14:44:01.536485+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2933135357,"num_objects":44712,"num_object_clones":0,"num_object_copies":134136,"num_objects_missing_on_primary":82,"num_objects_missing":82,"num_objects_degraded":5379,"num_objects_misplaced":4816,"num_objects_unfound":0,"num_objects_dirty":44712,"num_whiteouts":0,"num_read":6377,"num_read_kb":13834,"num_write":182065,"num_write_kb":7235010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84020,"num_bytes_recovered":5377756301,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95947,"ondisk_log_size":95947,"up":267,"acting":265,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":299,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":14107312,"kb_used_data":10835708,"kb_used_omap":26,"kb_used_meta":3271525,"kb_avail":548363600,"statfs":{"total":575970213888,"available":561524326400,"internally_reserved":0,"allocated":11095764992,"data_stored":11086475047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27144,"internal_metadata":3350042104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,0,0,2,1,3,5,4],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-170786816,"num_objects":-2606,"num_object_clones":0,"num_object_copies":-7818,"num_objects_missing_on_primary":-1,"num_objects_missing":-1,"num_objects_degraded":-4042,"num_objects_misplaced":2829,"num_objects_unfound":0,"num_objects_dirty":-2606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2606,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1045,"num_bytes_recovered":21692416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.913434"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":474,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.626279+0000","last_change":"2024-09-13T14:36:13.859927+0000","last_active":"2024-09-13T14:43:51.626279+0000","last_peered":"2024-09-13T14:43:51.626279+0000","last_clean":"2024-09-13T14:43:51.626279+0000","last_became_active":"2024-09-13T14:36:13.859505+0000","last_became_peered":"2024-09-13T14:36:13.859505+0000","last_unstale":"2024-09-13T14:43:51.626279+0000","last_undegraded":"2024-09-13T14:43:51.626279+0000","last_fullsized":"2024-09-13T14:43:51.626279+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:12:20.989627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":292,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.690420+0000","last_change":"2024-09-13T14:37:48.130968+0000","last_active":"2024-09-13T14:43:50.690420+0000","last_peered":"2024-09-13T14:43:50.690420+0000","last_clean":"2024-09-13T14:43:50.690420+0000","last_became_active":"2024-09-13T14:37:48.130603+0000","last_became_peered":"2024-09-13T14:37:48.130603+0000","last_unstale":"2024-09-13T14:43:50.690420+0000","last_undegraded":"2024-09-13T14:43:50.690420+0000","last_fullsized":"2024-09-13T14:43:50.690420+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:42.626386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":89,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.620441+0000","last_change":"2024-09-13T14:36:13.862181+0000","last_active":"2024-09-13T14:43:50.620441+0000","last_peered":"2024-09-13T14:43:50.620441+0000","last_clean":"2024-09-13T14:43:50.620441+0000","last_became_active":"2024-09-13T14:36:13.861873+0000","last_became_peered":"2024-09-13T14:36:13.861873+0000","last_unstale":"2024-09-13T14:43:50.620441+0000","last_undegraded":"2024-09-13T14:43:50.620441+0000","last_fullsized":"2024-09-13T14:43:50.620441+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:14.085700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":111,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.650001+0000","last_change":"2024-09-13T14:38:53.896672+0000","last_active":"2024-09-13T14:43:51.650001+0000","last_peered":"2024-09-13T14:43:51.650001+0000","last_clean":"2024-09-13T14:43:51.650001+0000","last_became_active":"2024-09-13T14:38:53.896346+0000","last_became_peered":"2024-09-13T14:38:53.896346+0000","last_unstale":"2024-09-13T14:43:51.650001+0000","last_undegraded":"2024-09-13T14:43:51.650001+0000","last_fullsized":"2024-09-13T14:43:51.650001+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:57.789686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":111,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.650492+0000","last_change":"2024-09-13T14:38:53.893021+0000","last_active":"2024-09-13T14:43:51.650492+0000","last_peered":"2024-09-13T14:43:51.650492+0000","last_clean":"2024-09-13T14:43:51.650492+0000","last_became_active":"2024-09-13T14:38:53.892668+0000","last_became_peered":"2024-09-13T14:38:53.892668+0000","last_unstale":"2024-09-13T14:43:51.650492+0000","last_undegraded":"2024-09-13T14:43:51.650492+0000","last_fullsized":"2024-09-13T14:43:51.650492+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:57.180738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":260,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.690490+0000","last_change":"2024-09-13T14:37:48.134642+0000","last_active":"2024-09-13T14:43:50.690490+0000","last_peered":"2024-09-13T14:43:50.690490+0000","last_clean":"2024-09-13T14:43:50.690490+0000","last_became_active":"2024-09-13T14:37:48.134402+0000","last_became_peered":"2024-09-13T14:37:48.134402+0000","last_unstale":"2024-09-13T14:43:50.690490+0000","last_undegraded":"2024-09-13T14:43:50.690490+0000","last_fullsized":"2024-09-13T14:43:50.690490+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:51.313663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":492,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.625758+0000","last_change":"2024-09-13T14:38:53.903022+0000","last_active":"2024-09-13T14:43:51.625758+0000","last_peered":"2024-09-13T14:43:51.625758+0000","last_clean":"2024-09-13T14:43:51.625758+0000","last_became_active":"2024-09-13T14:38:53.902448+0000","last_became_peered":"2024-09-13T14:38:53.902448+0000","last_unstale":"2024-09-13T14:43:51.625758+0000","last_undegraded":"2024-09-13T14:43:51.625758+0000","last_fullsized":"2024-09-13T14:43:51.625758+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:24.690401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":114,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.649644+0000","last_change":"2024-09-13T14:38:03.554500+0000","last_active":"2024-09-13T14:43:51.649644+0000","last_peered":"2024-09-13T14:43:51.649644+0000","last_clean":"2024-09-13T14:43:51.649644+0000","last_became_active":"2024-09-13T14:38:03.532910+0000","last_became_peered":"2024-09-13T14:38:03.532910+0000","last_unstale":"2024-09-13T14:43:51.649644+0000","last_undegraded":"2024-09-13T14:43:51.649644+0000","last_fullsized":"2024-09-13T14:43:51.649644+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:27:50.843498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"290'11171","reported_seq":22072,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:56.940922+0000","last_change":"2024-09-13T14:41:53.220083+0000","last_active":"2024-09-13T14:43:56.940922+0000","last_peered":"2024-09-13T14:43:56.940922+0000","last_clean":"2024-09-13T14:43:56.940922+0000","last_became_active":"2024-09-13T14:41:53.200084+0000","last_became_peered":"2024-09-13T14:41:53.200084+0000","last_unstale":"2024-09-13T14:43:56.940922+0000","last_undegraded":"2024-09-13T14:43:56.940922+0000","last_fullsized":"2024-09-13T14:43:56.940922+0000","mapping_epoch":275,"log_start":"219'5500","ondisk_log_start":"219'5500","created":205,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5671,"log_dups_size":0,"ondisk_log_size":5671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:08.897524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177537024,"num_objects":2709,"num_object_clones":0,"num_object_copies":8127,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11171,"num_write_kb":444160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9152,"num_bytes_recovered":594608128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":53,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.690686+0000","last_change":"2024-09-13T14:38:53.940238+0000","last_active":"2024-09-13T14:43:50.690686+0000","last_peered":"2024-09-13T14:43:50.690686+0000","last_clean":"2024-09-13T14:43:50.690686+0000","last_became_active":"2024-09-13T14:38:53.892825+0000","last_became_peered":"2024-09-13T14:38:53.892825+0000","last_unstale":"2024-09-13T14:43:50.690686+0000","last_undegraded":"2024-09-13T14:43:50.690686+0000","last_fullsized":"2024-09-13T14:43:50.690686+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:46.666581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":109,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.619817+0000","last_change":"2024-09-13T14:38:53.898907+0000","last_active":"2024-09-13T14:43:50.619817+0000","last_peered":"2024-09-13T14:43:50.619817+0000","last_clean":"2024-09-13T14:43:50.619817+0000","last_became_active":"2024-09-13T14:38:53.898572+0000","last_became_peered":"2024-09-13T14:38:53.898572+0000","last_unstale":"2024-09-13T14:43:50.619817+0000","last_undegraded":"2024-09-13T14:43:50.619817+0000","last_fullsized":"2024-09-13T14:43:50.619817+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:29.222150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"290'11486","reported_seq":25676,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:57.392148+0000","last_change":"2024-09-13T14:38:57.930032+0000","last_active":"2024-09-13T14:43:57.392148+0000","last_peered":"2024-09-13T14:43:57.392148+0000","last_clean":"2024-09-13T14:43:57.392148+0000","last_became_active":"2024-09-13T14:38:57.921458+0000","last_became_peered":"2024-09-13T14:38:57.921458+0000","last_unstale":"2024-09-13T14:43:57.392148+0000","last_undegraded":"2024-09-13T14:43:57.392148+0000","last_fullsized":"2024-09-13T14:43:57.392148+0000","mapping_epoch":256,"log_start":"219'5739","ondisk_log_start":"219'5739","created":205,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5747,"log_dups_size":0,"ondisk_log_size":5747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:27:12.294038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180617216,"num_objects":2756,"num_object_clones":0,"num_object_copies":8268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11486,"num_write_kb":455744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6990,"num_bytes_recovered":458096640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":53,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.690681+0000","last_change":"2024-09-13T14:38:53.904885+0000","last_active":"2024-09-13T14:43:50.690681+0000","last_peered":"2024-09-13T14:43:50.690681+0000","last_clean":"2024-09-13T14:43:50.690681+0000","last_became_active":"2024-09-13T14:38:53.891726+0000","last_became_peered":"2024-09-13T14:38:53.891726+0000","last_unstale":"2024-09-13T14:43:50.690681+0000","last_undegraded":"2024-09-13T14:43:50.690681+0000","last_fullsized":"2024-09-13T14:43:50.690681+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:57.079205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":299,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.624028+0000","last_change":"2024-09-13T14:37:48.160504+0000","last_active":"2024-09-13T14:43:51.624028+0000","last_peered":"2024-09-13T14:43:51.624028+0000","last_clean":"2024-09-13T14:43:51.624028+0000","last_became_active":"2024-09-13T14:37:48.160181+0000","last_became_peered":"2024-09-13T14:37:48.160181+0000","last_unstale":"2024-09-13T14:43:51.624028+0000","last_undegraded":"2024-09-13T14:43:51.624028+0000","last_fullsized":"2024-09-13T14:43:51.624028+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:37.087008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"290'11314","reported_seq":13126,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:56.558679+0000","last_change":"2024-09-13T14:38:04.531914+0000","last_active":"2024-09-13T14:43:56.558679+0000","last_peered":"2024-09-13T14:43:56.558679+0000","last_clean":"2024-09-13T14:43:56.558679+0000","last_became_active":"2024-09-13T14:38:04.523448+0000","last_became_peered":"2024-09-13T14:38:04.523448+0000","last_unstale":"2024-09-13T14:43:56.558679+0000","last_undegraded":"2024-09-13T14:43:56.558679+0000","last_fullsized":"2024-09-13T14:43:56.558679+0000","mapping_epoch":245,"log_start":"219'5400","ondisk_log_start":"219'5400","created":205,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5914,"log_dups_size":0,"ondisk_log_size":5914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:06.536896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187432960,"num_objects":2860,"num_object_clones":0,"num_object_copies":8580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11314,"num_write_kb":453568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":811,"num_bytes_recovered":53149696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":290,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.690418+0000","last_change":"2024-09-13T14:37:49.611774+0000","last_active":"2024-09-13T14:43:50.690418+0000","last_peered":"2024-09-13T14:43:50.690418+0000","last_clean":"2024-09-13T14:43:50.690418+0000","last_became_active":"2024-09-13T14:37:49.611088+0000","last_became_peered":"2024-09-13T14:37:49.611088+0000","last_unstale":"2024-09-13T14:43:50.690418+0000","last_undegraded":"2024-09-13T14:43:50.690418+0000","last_fullsized":"2024-09-13T14:43:50.690418+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:08.933213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":49,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.690168+0000","last_change":"2024-09-13T14:38:53.899211+0000","last_active":"2024-09-13T14:43:50.690168+0000","last_peered":"2024-09-13T14:43:50.690168+0000","last_clean":"2024-09-13T14:43:50.690168+0000","last_became_active":"2024-09-13T14:38:53.898927+0000","last_became_peered":"2024-09-13T14:38:53.898927+0000","last_unstale":"2024-09-13T14:43:50.690168+0000","last_undegraded":"2024-09-13T14:43:50.690168+0000","last_fullsized":"2024-09-13T14:43:50.690168+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:00:28.857611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"290'11210","reported_seq":11307,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:57.250694+0000","last_change":"2024-09-13T14:35:24.004651+0000","last_active":"2024-09-13T14:43:57.250694+0000","last_peered":"2024-09-13T14:43:57.250694+0000","last_clean":"2024-09-13T14:43:57.250694+0000","last_became_active":"2024-09-13T14:35:24.004358+0000","last_became_peered":"2024-09-13T14:35:24.004358+0000","last_unstale":"2024-09-13T14:43:57.250694+0000","last_undegraded":"2024-09-13T14:43:57.250694+0000","last_fullsized":"2024-09-13T14:43:57.250694+0000","mapping_epoch":205,"log_start":"217'4884","ondisk_log_start":"217'4884","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6326,"log_dups_size":0,"ondisk_log_size":6326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:57:38.023669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181010432,"num_objects":2762,"num_object_clones":0,"num_object_copies":8286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11210,"num_write_kb":447104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":100,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.620360+0000","last_change":"2024-09-13T14:37:49.168003+0000","last_active":"2024-09-13T14:43:50.620360+0000","last_peered":"2024-09-13T14:43:50.620360+0000","last_clean":"2024-09-13T14:43:50.620360+0000","last_became_active":"2024-09-13T14:37:49.167638+0000","last_became_peered":"2024-09-13T14:37:49.167638+0000","last_unstale":"2024-09-13T14:43:50.620360+0000","last_undegraded":"2024-09-13T14:43:50.620360+0000","last_fullsized":"2024-09-13T14:43:50.620360+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:29.051981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":104,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.650052+0000","last_change":"2024-09-13T14:38:03.528381+0000","last_active":"2024-09-13T14:43:51.650052+0000","last_peered":"2024-09-13T14:43:51.650052+0000","last_clean":"2024-09-13T14:43:51.650052+0000","last_became_active":"2024-09-13T14:38:03.527282+0000","last_became_peered":"2024-09-13T14:38:03.527282+0000","last_unstale":"2024-09-13T14:43:51.650052+0000","last_undegraded":"2024-09-13T14:43:51.650052+0000","last_fullsized":"2024-09-13T14:43:51.650052+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:24.435067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.f","version":"290'11255","reported_seq":11562,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:56.564806+0000","last_change":"2024-09-13T14:38:16.371079+0000","last_active":"2024-09-13T14:43:56.564806+0000","last_peered":"2024-09-13T14:43:56.564806+0000","last_clean":"2024-09-13T14:43:56.564806+0000","last_became_active":"2024-09-13T14:38:12.859883+0000","last_became_peered":"2024-09-13T14:38:12.859883+0000","last_unstale":"2024-09-13T14:43:56.564806+0000","last_undegraded":"2024-09-13T14:43:56.564806+0000","last_fullsized":"2024-09-13T14:43:56.564806+0000","mapping_epoch":245,"log_start":"219'5363","ondisk_log_start":"219'5363","created":205,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5892,"log_dups_size":0,"ondisk_log_size":5892,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:08.436532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185532416,"num_objects":2831,"num_object_clones":0,"num_object_copies":8493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11255,"num_write_kb":450752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":3670016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":462,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.625571+0000","last_change":"2024-09-13T14:38:03.549595+0000","last_active":"2024-09-13T14:43:51.625571+0000","last_peered":"2024-09-13T14:43:51.625571+0000","last_clean":"2024-09-13T14:43:51.625571+0000","last_became_active":"2024-09-13T14:38:03.533331+0000","last_became_peered":"2024-09-13T14:38:03.533331+0000","last_unstale":"2024-09-13T14:43:51.625571+0000","last_undegraded":"2024-09-13T14:43:51.625571+0000","last_fullsized":"2024-09-13T14:43:51.625571+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:59:02.470343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":275,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.624410+0000","last_change":"2024-09-13T14:37:48.127600+0000","last_active":"2024-09-13T14:43:51.624410+0000","last_peered":"2024-09-13T14:43:51.624410+0000","last_clean":"2024-09-13T14:43:51.624410+0000","last_became_active":"2024-09-13T14:37:48.127349+0000","last_became_peered":"2024-09-13T14:37:48.127349+0000","last_unstale":"2024-09-13T14:43:51.624410+0000","last_undegraded":"2024-09-13T14:43:51.624410+0000","last_fullsized":"2024-09-13T14:43:51.624410+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:57.268374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"290'11303","reported_seq":21297,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:57.231671+0000","last_change":"2024-09-13T14:42:30.227259+0000","last_active":"2024-09-13T14:43:57.231671+0000","last_peered":"2024-09-13T14:43:57.231671+0000","last_clean":"2024-09-13T14:43:57.231671+0000","last_became_active":"2024-09-13T14:42:30.209102+0000","last_became_peered":"2024-09-13T14:42:30.209102+0000","last_unstale":"2024-09-13T14:43:57.231671+0000","last_undegraded":"2024-09-13T14:43:57.231671+0000","last_fullsized":"2024-09-13T14:43:57.231671+0000","mapping_epoch":281,"log_start":"219'5600","ondisk_log_start":"219'5600","created":205,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5703,"log_dups_size":0,"ondisk_log_size":5703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:56:16.193287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177405952,"num_objects":2707,"num_object_clones":0,"num_object_copies":8121,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11303,"num_write_kb":448320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4926,"num_bytes_recovered":313851904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":208,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.624099+0000","last_change":"2024-09-13T14:37:48.127604+0000","last_active":"2024-09-13T14:43:51.624099+0000","last_peered":"2024-09-13T14:43:51.624099+0000","last_clean":"2024-09-13T14:43:51.624099+0000","last_became_active":"2024-09-13T14:37:48.127300+0000","last_became_peered":"2024-09-13T14:37:48.127300+0000","last_unstale":"2024-09-13T14:43:51.624099+0000","last_undegraded":"2024-09-13T14:43:51.624099+0000","last_fullsized":"2024-09-13T14:43:51.624099+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:51.999506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":328,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.624513+0000","last_change":"2024-09-13T14:38:03.529967+0000","last_active":"2024-09-13T14:43:51.624513+0000","last_peered":"2024-09-13T14:43:51.624513+0000","last_clean":"2024-09-13T14:43:51.624513+0000","last_became_active":"2024-09-13T14:38:03.529646+0000","last_became_peered":"2024-09-13T14:38:03.529646+0000","last_unstale":"2024-09-13T14:43:51.624513+0000","last_undegraded":"2024-09-13T14:43:51.624513+0000","last_fullsized":"2024-09-13T14:43:51.624513+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:52:05.193279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"290'11366","reported_seq":20548,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:56.631489+0000","last_change":"2024-09-13T14:40:34.583237+0000","last_active":"2024-09-13T14:43:56.631489+0000","last_peered":"2024-09-13T14:43:56.631489+0000","last_clean":"2024-09-13T14:43:56.631489+0000","last_became_active":"2024-09-13T14:40:34.563989+0000","last_became_peered":"2024-09-13T14:40:34.563989+0000","last_unstale":"2024-09-13T14:43:56.631489+0000","last_undegraded":"2024-09-13T14:43:56.631489+0000","last_fullsized":"2024-09-13T14:43:56.631489+0000","mapping_epoch":266,"log_start":"219'5400","ondisk_log_start":"219'5400","created":205,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5966,"log_dups_size":0,"ondisk_log_size":5966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:34:43.006051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181731352,"num_objects":2774,"num_object_clones":0,"num_object_copies":8322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2774,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11366,"num_write_kb":452417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4489,"num_bytes_recovered":289865752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":288,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.624320+0000","last_change":"2024-09-13T14:36:13.860367+0000","last_active":"2024-09-13T14:43:51.624320+0000","last_peered":"2024-09-13T14:43:51.624320+0000","last_clean":"2024-09-13T14:43:51.624320+0000","last_became_active":"2024-09-13T14:36:13.859982+0000","last_became_peered":"2024-09-13T14:36:13.859982+0000","last_unstale":"2024-09-13T14:43:51.624320+0000","last_undegraded":"2024-09-13T14:43:51.624320+0000","last_fullsized":"2024-09-13T14:43:51.624320+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:08:23.311350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":284,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.624834+0000","last_change":"2024-09-13T14:37:48.147360+0000","last_active":"2024-09-13T14:43:51.624834+0000","last_peered":"2024-09-13T14:43:51.624834+0000","last_clean":"2024-09-13T14:43:51.624834+0000","last_became_active":"2024-09-13T14:37:48.146885+0000","last_became_peered":"2024-09-13T14:37:48.146885+0000","last_unstale":"2024-09-13T14:43:51.624834+0000","last_undegraded":"2024-09-13T14:43:51.624834+0000","last_fullsized":"2024-09-13T14:43:51.624834+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:27.841810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"290'11260","reported_seq":20549,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:57.142876+0000","last_change":"2024-09-13T14:38:39.650332+0000","last_active":"2024-09-13T14:43:57.142876+0000","last_peered":"2024-09-13T14:43:57.142876+0000","last_clean":"2024-09-13T14:43:57.142876+0000","last_became_active":"2024-09-13T14:38:39.640835+0000","last_became_peered":"2024-09-13T14:38:39.640835+0000","last_unstale":"2024-09-13T14:43:57.142876+0000","last_undegraded":"2024-09-13T14:43:57.142876+0000","last_fullsized":"2024-09-13T14:43:57.142876+0000","mapping_epoch":251,"log_start":"219'5551","ondisk_log_start":"219'5551","created":205,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5709,"log_dups_size":0,"ondisk_log_size":5709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:48.176606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184418304,"num_objects":2814,"num_object_clones":0,"num_object_copies":8442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11260,"num_write_kb":450368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4544,"num_bytes_recovered":292945920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":480,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.626610+0000","last_change":"2024-09-13T14:38:53.900041+0000","last_active":"2024-09-13T14:43:51.626610+0000","last_peered":"2024-09-13T14:43:51.626610+0000","last_clean":"2024-09-13T14:43:51.626610+0000","last_became_active":"2024-09-13T14:38:53.899203+0000","last_became_peered":"2024-09-13T14:38:53.899203+0000","last_unstale":"2024-09-13T14:43:51.626610+0000","last_undegraded":"2024-09-13T14:43:51.626610+0000","last_fullsized":"2024-09-13T14:43:51.626610+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:28:07.883094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":412,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.625960+0000","last_change":"2024-09-13T14:26:15.823198+0000","last_active":"2024-09-13T14:43:51.625960+0000","last_peered":"2024-09-13T14:43:51.625960+0000","last_clean":"2024-09-13T14:43:51.625960+0000","last_became_active":"2024-09-13T14:26:15.822717+0000","last_became_peered":"2024-09-13T14:26:15.822717+0000","last_unstale":"2024-09-13T14:43:51.625960+0000","last_undegraded":"2024-09-13T14:43:51.625960+0000","last_fullsized":"2024-09-13T14:43:51.625960+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:22.471622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.6","version":"290'11376","reported_seq":19916,"reported_epoch":290,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-13T14:43:56.703863+0000","last_change":"2024-09-13T14:38:55.132683+0000","last_active":"2024-09-13T14:43:56.703863+0000","last_peered":"2024-09-13T14:43:56.703863+0000","last_clean":"2024-09-13T14:36:13.124368+0000","last_became_active":"2024-09-13T14:38:55.120883+0000","last_became_peered":"2024-09-13T14:38:55.120883+0000","last_unstale":"2024-09-13T14:43:56.703863+0000","last_undegraded":"2024-09-13T14:38:55.026610+0000","last_fullsized":"2024-09-13T14:43:56.703863+0000","mapping_epoch":254,"log_start":"219'5419","ondisk_log_start":"219'5419","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5957,"log_dups_size":0,"ondisk_log_size":5957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184942592,"num_objects":2822,"num_object_clones":0,"num_object_copies":8466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":835,"num_objects_misplaced":1987,"num_objects_unfound":0,"num_objects_dirty":2822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11376,"num_write_kb":454336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4170,"num_bytes_recovered":262078464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,3,5],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3,4","objects":556},{"shards":"0,3,5","objects":2266}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":424,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.657167+0000","last_change":"2024-09-13T14:41:53.196835+0000","last_active":"2024-09-13T14:43:51.657167+0000","last_peered":"2024-09-13T14:43:51.657167+0000","last_clean":"2024-09-13T14:43:51.657167+0000","last_became_active":"2024-09-13T14:41:53.196387+0000","last_became_peered":"2024-09-13T14:41:53.196387+0000","last_unstale":"2024-09-13T14:43:51.657167+0000","last_undegraded":"2024-09-13T14:43:51.657167+0000","last_fullsized":"2024-09-13T14:43:51.657167+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:07:41.927740+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:49.972303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":529,"reported_epoch":289,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:43:50.620126+0000","last_change":"2024-09-13T14:38:54.904329+0000","last_active":"2024-09-13T14:43:50.620126+0000","last_peered":"2024-09-13T14:43:50.620126+0000","last_clean":"2024-09-13T14:38:39.648017+0000","last_became_active":"2024-09-13T14:38:54.903798+0000","last_became_peered":"2024-09-13T14:38:54.903798+0000","last_unstale":"2024-09-13T14:43:50.620126+0000","last_undegraded":"2024-09-13T14:38:54.890505+0000","last_fullsized":"2024-09-13T14:38:54.889701+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:18.120931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,3],"avail_no_missing":["5","3","6"],"object_location_counts":[{"shards":"3,5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":415,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.626125+0000","last_change":"2024-09-13T14:37:49.160719+0000","last_active":"2024-09-13T14:43:51.626125+0000","last_peered":"2024-09-13T14:43:51.626125+0000","last_clean":"2024-09-13T14:43:51.626125+0000","last_became_active":"2024-09-13T14:37:49.160401+0000","last_became_peered":"2024-09-13T14:37:49.160401+0000","last_unstale":"2024-09-13T14:43:51.626125+0000","last_undegraded":"2024-09-13T14:43:51.626125+0000","last_fullsized":"2024-09-13T14:43:51.626125+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:21.737945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":135,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.624439+0000","last_change":"2024-09-13T14:37:48.172940+0000","last_active":"2024-09-13T14:43:51.624439+0000","last_peered":"2024-09-13T14:43:51.624439+0000","last_clean":"2024-09-13T14:43:51.624439+0000","last_became_active":"2024-09-13T14:37:48.172060+0000","last_became_peered":"2024-09-13T14:37:48.172060+0000","last_unstale":"2024-09-13T14:43:51.624439+0000","last_undegraded":"2024-09-13T14:43:51.624439+0000","last_fullsized":"2024-09-13T14:43:51.624439+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:07.480645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.5","version":"290'11340","reported_seq":27714,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:57.316828+0000","last_change":"2024-09-13T14:39:34.446699+0000","last_active":"2024-09-13T14:43:57.316828+0000","last_peered":"2024-09-13T14:43:57.316828+0000","last_clean":"2024-09-13T14:43:57.316828+0000","last_became_active":"2024-09-13T14:38:04.600616+0000","last_became_peered":"2024-09-13T14:38:04.600616+0000","last_unstale":"2024-09-13T14:43:57.316828+0000","last_undegraded":"2024-09-13T14:43:57.316828+0000","last_fullsized":"2024-09-13T14:43:57.316828+0000","mapping_epoch":245,"log_start":"218'5016","ondisk_log_start":"218'5016","created":205,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6324,"log_dups_size":0,"ondisk_log_size":6324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:00:44.321678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183631872,"num_objects":2802,"num_object_clones":0,"num_object_copies":8406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11340,"num_write_kb":452544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8120,"num_bytes_recovered":521076736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":507,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.690066+0000","last_change":"2024-09-13T14:26:15.835178+0000","last_active":"2024-09-13T14:43:50.690066+0000","last_peered":"2024-09-13T14:43:50.690066+0000","last_clean":"2024-09-13T14:43:50.690066+0000","last_became_active":"2024-09-13T14:26:15.820283+0000","last_became_peered":"2024-09-13T14:26:15.820283+0000","last_unstale":"2024-09-13T14:43:50.690066+0000","last_undegraded":"2024-09-13T14:43:50.690066+0000","last_fullsized":"2024-09-13T14:43:50.690066+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:30.340038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":104,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.649704+0000","last_change":"2024-09-13T14:38:03.525682+0000","last_active":"2024-09-13T14:43:51.649704+0000","last_peered":"2024-09-13T14:43:51.649704+0000","last_clean":"2024-09-13T14:43:51.649704+0000","last_became_active":"2024-09-13T14:38:03.525406+0000","last_became_peered":"2024-09-13T14:38:03.525406+0000","last_unstale":"2024-09-13T14:43:51.649704+0000","last_undegraded":"2024-09-13T14:43:51.649704+0000","last_fullsized":"2024-09-13T14:43:51.649704+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:41:28.970300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":111,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.650537+0000","last_change":"2024-09-13T14:38:53.991738+0000","last_active":"2024-09-13T14:43:51.650537+0000","last_peered":"2024-09-13T14:43:51.650537+0000","last_clean":"2024-09-13T14:43:51.650537+0000","last_became_active":"2024-09-13T14:38:53.991417+0000","last_became_peered":"2024-09-13T14:38:53.991417+0000","last_unstale":"2024-09-13T14:43:51.650537+0000","last_undegraded":"2024-09-13T14:43:51.650537+0000","last_fullsized":"2024-09-13T14:43:51.650537+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:50:38.559838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"290'11277","reported_seq":28652,"reported_epoch":290,"state":"active+recovering","last_fresh":"2024-09-13T14:43:56.698591+0000","last_change":"2024-09-13T14:43:50.028785+0000","last_active":"2024-09-13T14:43:56.698591+0000","last_peered":"2024-09-13T14:43:56.698591+0000","last_clean":"2024-09-13T14:37:46.290754+0000","last_became_active":"2024-09-13T14:38:17.381753+0000","last_became_peered":"2024-09-13T14:38:17.381753+0000","last_unstale":"2024-09-13T14:43:56.698591+0000","last_undegraded":"2024-09-13T14:43:56.698591+0000","last_fullsized":"2024-09-13T14:43:56.698591+0000","mapping_epoch":245,"log_start":"219'5359","ondisk_log_start":"219'5359","created":205,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5918,"log_dups_size":0,"ondisk_log_size":5918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185401344,"num_objects":2829,"num_object_clones":0,"num_object_copies":8487,"num_objects_missing_on_primary":82,"num_objects_missing":82,"num_objects_degraded":0,"num_objects_misplaced":2829,"num_objects_unfound":0,"num_objects_dirty":2829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11277,"num_write_kb":451392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8638,"num_bytes_recovered":566034432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"0,4,5","objects":2829}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":826,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.625800+0000","last_change":"2024-09-13T14:37:48.131799+0000","last_active":"2024-09-13T14:43:51.625800+0000","last_peered":"2024-09-13T14:43:51.625800+0000","last_clean":"2024-09-13T14:43:51.625800+0000","last_became_active":"2024-09-13T14:37:48.130837+0000","last_became_peered":"2024-09-13T14:37:48.130837+0000","last_unstale":"2024-09-13T14:43:51.625800+0000","last_undegraded":"2024-09-13T14:43:51.625800+0000","last_fullsized":"2024-09-13T14:43:51.625800+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:06:32.663770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":344,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.649946+0000","last_change":"2024-09-13T14:37:49.301531+0000","last_active":"2024-09-13T14:43:51.649946+0000","last_peered":"2024-09-13T14:43:51.649946+0000","last_clean":"2024-09-13T14:43:51.649946+0000","last_became_active":"2024-09-13T14:37:49.301249+0000","last_became_peered":"2024-09-13T14:37:49.301249+0000","last_unstale":"2024-09-13T14:43:51.649946+0000","last_undegraded":"2024-09-13T14:43:51.649946+0000","last_fullsized":"2024-09-13T14:43:51.649946+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:10.977710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":116,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.620681+0000","last_change":"2024-09-13T14:38:53.901427+0000","last_active":"2024-09-13T14:43:50.620681+0000","last_peered":"2024-09-13T14:43:50.620681+0000","last_clean":"2024-09-13T14:43:50.620681+0000","last_became_active":"2024-09-13T14:38:53.901157+0000","last_became_peered":"2024-09-13T14:38:53.901157+0000","last_unstale":"2024-09-13T14:43:50.620681+0000","last_undegraded":"2024-09-13T14:43:50.620681+0000","last_fullsized":"2024-09-13T14:43:50.620681+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:28:21.713025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"290'11218","reported_seq":34461,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:56.689737+0000","last_change":"2024-09-13T14:43:16.851222+0000","last_active":"2024-09-13T14:43:56.689737+0000","last_peered":"2024-09-13T14:43:56.689737+0000","last_clean":"2024-09-13T14:43:56.689737+0000","last_became_active":"2024-09-13T14:38:16.427510+0000","last_became_peered":"2024-09-13T14:38:16.427510+0000","last_unstale":"2024-09-13T14:43:56.689737+0000","last_undegraded":"2024-09-13T14:43:56.689737+0000","last_fullsized":"2024-09-13T14:43:56.689737+0000","mapping_epoch":245,"log_start":"219'5318","ondisk_log_start":"219'5318","created":205,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5900,"log_dups_size":0,"ondisk_log_size":5900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:27:26.281938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182190080,"num_objects":2780,"num_object_clones":0,"num_object_copies":8340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11218,"num_write_kb":447936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11571,"num_bytes_recovered":747372544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":998,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.619897+0000","last_change":"2024-09-13T14:37:49.147498+0000","last_active":"2024-09-13T14:43:50.619897+0000","last_peered":"2024-09-13T14:43:50.619897+0000","last_clean":"2024-09-13T14:43:50.619897+0000","last_became_active":"2024-09-13T14:37:49.147028+0000","last_became_peered":"2024-09-13T14:37:49.147028+0000","last_unstale":"2024-09-13T14:43:50.619897+0000","last_undegraded":"2024-09-13T14:43:50.619897+0000","last_fullsized":"2024-09-13T14:43:50.619897+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:11:29.050178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":297,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.624462+0000","last_change":"2024-09-13T14:38:03.528133+0000","last_active":"2024-09-13T14:43:51.624462+0000","last_peered":"2024-09-13T14:43:51.624462+0000","last_clean":"2024-09-13T14:43:51.624462+0000","last_became_active":"2024-09-13T14:38:03.527829+0000","last_became_peered":"2024-09-13T14:38:03.527829+0000","last_unstale":"2024-09-13T14:43:51.624462+0000","last_undegraded":"2024-09-13T14:43:51.624462+0000","last_fullsized":"2024-09-13T14:43:51.624462+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:48.254169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":465,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.626238+0000","last_change":"2024-09-13T14:38:53.899755+0000","last_active":"2024-09-13T14:43:51.626238+0000","last_peered":"2024-09-13T14:43:51.626238+0000","last_clean":"2024-09-13T14:43:51.626238+0000","last_became_active":"2024-09-13T14:38:53.899448+0000","last_became_peered":"2024-09-13T14:38:53.899448+0000","last_unstale":"2024-09-13T14:43:51.626238+0000","last_undegraded":"2024-09-13T14:43:51.626238+0000","last_fullsized":"2024-09-13T14:43:51.626238+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:55.897035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"291'11546","reported_seq":20679,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-13T14:43:57.792895+0000","last_change":"2024-09-13T14:43:51.649951+0000","last_active":"2024-09-13T14:43:57.792895+0000","last_peered":"2024-09-13T14:43:57.792895+0000","last_clean":"2024-09-13T14:43:57.792895+0000","last_became_active":"2024-09-13T14:43:51.626628+0000","last_became_peered":"2024-09-13T14:43:51.626628+0000","last_unstale":"2024-09-13T14:43:57.792895+0000","last_undegraded":"2024-09-13T14:43:57.792895+0000","last_fullsized":"2024-09-13T14:43:57.792895+0000","mapping_epoch":289,"log_start":"213'4863","ondisk_log_start":"213'4863","created":205,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6683,"log_dups_size":0,"ondisk_log_size":6683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:37:06.795173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180617216,"num_objects":2756,"num_object_clones":0,"num_object_copies":8268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11546,"num_write_kb":457664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4502,"num_bytes_recovered":228851712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":100,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.620326+0000","last_change":"2024-09-13T14:37:49.149770+0000","last_active":"2024-09-13T14:43:50.620326+0000","last_peered":"2024-09-13T14:43:50.620326+0000","last_clean":"2024-09-13T14:43:50.620326+0000","last_became_active":"2024-09-13T14:37:49.148991+0000","last_became_peered":"2024-09-13T14:37:49.148991+0000","last_unstale":"2024-09-13T14:43:50.620326+0000","last_undegraded":"2024-09-13T14:43:50.620326+0000","last_fullsized":"2024-09-13T14:43:50.620326+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:04:04.193366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":1163,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.667716+0000","last_change":"2024-09-13T14:30:58.928302+0000","last_active":"2024-09-13T14:43:51.667716+0000","last_peered":"2024-09-13T14:43:51.667716+0000","last_clean":"2024-09-13T14:43:51.667716+0000","last_became_active":"2024-09-13T14:30:58.927718+0000","last_became_peered":"2024-09-13T14:30:58.927718+0000","last_unstale":"2024-09-13T14:43:51.667716+0000","last_undegraded":"2024-09-13T14:43:51.667716+0000","last_fullsized":"2024-09-13T14:43:51.667716+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:37.323036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":107,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.620008+0000","last_change":"2024-09-13T14:38:03.534628+0000","last_active":"2024-09-13T14:43:50.620008+0000","last_peered":"2024-09-13T14:43:50.620008+0000","last_clean":"2024-09-13T14:43:50.620008+0000","last_became_active":"2024-09-13T14:38:03.534262+0000","last_became_peered":"2024-09-13T14:38:03.534262+0000","last_unstale":"2024-09-13T14:43:50.620008+0000","last_undegraded":"2024-09-13T14:43:50.620008+0000","last_fullsized":"2024-09-13T14:43:50.620008+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:31.168783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"290'11262","reported_seq":25440,"reported_epoch":290,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:43:57.174429+0000","last_change":"2024-09-13T14:38:53.944111+0000","last_active":"2024-09-13T14:43:57.174429+0000","last_peered":"2024-09-13T14:43:57.174429+0000","last_clean":"2024-09-13T14:37:47.047672+0000","last_became_active":"2024-09-13T14:37:49.664294+0000","last_became_peered":"2024-09-13T14:37:49.664294+0000","last_unstale":"2024-09-13T14:43:57.174429+0000","last_undegraded":"2024-09-13T14:38:53.940822+0000","last_fullsized":"2024-09-13T14:37:49.126627+0000","mapping_epoch":240,"log_start":"219'5600","ondisk_log_start":"219'5600","created":205,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5662,"log_dups_size":0,"ondisk_log_size":5662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:11.766692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182845440,"num_objects":2790,"num_object_clones":0,"num_object_copies":8370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4535,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11262,"num_write_kb":449664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7009,"num_bytes_recovered":459341824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3],"avail_no_missing":["5","0","3"],"object_location_counts":[{"shards":"0,3,5","objects":2790},{"shards":"3,5","objects":-1670}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":119,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.620290+0000","last_change":"2024-09-13T14:38:03.534303+0000","last_active":"2024-09-13T14:43:50.620290+0000","last_peered":"2024-09-13T14:43:50.620290+0000","last_clean":"2024-09-13T14:43:50.620290+0000","last_became_active":"2024-09-13T14:38:03.533171+0000","last_became_peered":"2024-09-13T14:38:03.533171+0000","last_unstale":"2024-09-13T14:43:50.620290+0000","last_undegraded":"2024-09-13T14:43:50.620290+0000","last_fullsized":"2024-09-13T14:43:50.620290+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:00:16.553526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":812,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.690438+0000","last_change":"2024-09-13T14:38:04.596192+0000","last_active":"2024-09-13T14:43:50.690438+0000","last_peered":"2024-09-13T14:43:50.690438+0000","last_clean":"2024-09-13T14:43:50.690438+0000","last_became_active":"2024-09-13T14:38:04.593299+0000","last_became_peered":"2024-09-13T14:38:04.593299+0000","last_unstale":"2024-09-13T14:43:50.690438+0000","last_undegraded":"2024-09-13T14:43:50.690438+0000","last_fullsized":"2024-09-13T14:43:50.690438+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:34.241410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":104,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.650109+0000","last_change":"2024-09-13T14:38:03.533120+0000","last_active":"2024-09-13T14:43:51.650109+0000","last_peered":"2024-09-13T14:43:51.650109+0000","last_clean":"2024-09-13T14:43:51.650109+0000","last_became_active":"2024-09-13T14:38:03.532829+0000","last_became_peered":"2024-09-13T14:38:03.532829+0000","last_unstale":"2024-09-13T14:43:51.650109+0000","last_undegraded":"2024-09-13T14:43:51.650109+0000","last_fullsized":"2024-09-13T14:43:51.650109+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:24.514661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"290'11483","reported_seq":20752,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:57.275987+0000","last_change":"2024-09-13T14:42:04.660877+0000","last_active":"2024-09-13T14:43:57.275987+0000","last_peered":"2024-09-13T14:43:57.275987+0000","last_clean":"2024-09-13T14:43:57.275987+0000","last_became_active":"2024-09-13T14:42:04.641680+0000","last_became_peered":"2024-09-13T14:42:04.641680+0000","last_unstale":"2024-09-13T14:43:57.275987+0000","last_undegraded":"2024-09-13T14:43:57.275987+0000","last_fullsized":"2024-09-13T14:43:57.275987+0000","mapping_epoch":277,"log_start":"219'5818","ondisk_log_start":"219'5818","created":205,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5665,"log_dups_size":0,"ondisk_log_size":5665,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:22:18.955428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191037440,"num_objects":2915,"num_object_clones":0,"num_object_copies":8745,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11483,"num_write_kb":460736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4533,"num_bytes_recovered":297074688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":506,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.626040+0000","last_change":"2024-09-13T14:38:53.938227+0000","last_active":"2024-09-13T14:43:51.626040+0000","last_peered":"2024-09-13T14:43:51.626040+0000","last_clean":"2024-09-13T14:43:51.626040+0000","last_became_active":"2024-09-13T14:38:53.902867+0000","last_became_peered":"2024-09-13T14:38:53.902867+0000","last_unstale":"2024-09-13T14:43:51.626040+0000","last_undegraded":"2024-09-13T14:43:51.626040+0000","last_fullsized":"2024-09-13T14:43:51.626040+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:16.831756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":591,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.624879+0000","last_change":"2024-09-13T14:37:48.147677+0000","last_active":"2024-09-13T14:43:51.624879+0000","last_peered":"2024-09-13T14:43:51.624879+0000","last_clean":"2024-09-13T14:43:51.624879+0000","last_became_active":"2024-09-13T14:37:48.147001+0000","last_became_peered":"2024-09-13T14:37:48.147001+0000","last_unstale":"2024-09-13T14:43:51.624879+0000","last_undegraded":"2024-09-13T14:43:51.624879+0000","last_fullsized":"2024-09-13T14:43:51.624879+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:35:00.653576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":488,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.626531+0000","last_change":"2024-09-13T14:38:53.900107+0000","last_active":"2024-09-13T14:43:51.626531+0000","last_peered":"2024-09-13T14:43:51.626531+0000","last_clean":"2024-09-13T14:43:51.626531+0000","last_became_active":"2024-09-13T14:38:53.899556+0000","last_became_peered":"2024-09-13T14:38:53.899556+0000","last_unstale":"2024-09-13T14:43:51.626531+0000","last_undegraded":"2024-09-13T14:43:51.626531+0000","last_fullsized":"2024-09-13T14:43:51.626531+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:15:47.918163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"290'11075","reported_seq":20070,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:57.153075+0000","last_change":"2024-09-13T14:39:54.921239+0000","last_active":"2024-09-13T14:43:57.153075+0000","last_peered":"2024-09-13T14:43:57.153075+0000","last_clean":"2024-09-13T14:43:57.153075+0000","last_became_active":"2024-09-13T14:39:54.907894+0000","last_became_peered":"2024-09-13T14:39:54.907894+0000","last_unstale":"2024-09-13T14:43:57.153075+0000","last_undegraded":"2024-09-13T14:43:57.153075+0000","last_fullsized":"2024-09-13T14:43:57.153075+0000","mapping_epoch":263,"log_start":"219'5347","ondisk_log_start":"219'5347","created":205,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5728,"log_dups_size":0,"ondisk_log_size":5728,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:52:04.619238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181993472,"num_objects":2777,"num_object_clones":0,"num_object_copies":8331,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11075,"num_write_kb":443264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4395,"num_bytes_recovered":283770880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":119,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.650419+0000","last_change":"2024-09-13T14:38:53.901806+0000","last_active":"2024-09-13T14:43:51.650419+0000","last_peered":"2024-09-13T14:43:51.650419+0000","last_clean":"2024-09-13T14:43:51.650419+0000","last_became_active":"2024-09-13T14:38:53.891055+0000","last_became_peered":"2024-09-13T14:38:53.891055+0000","last_unstale":"2024-09-13T14:43:51.650419+0000","last_undegraded":"2024-09-13T14:43:51.650419+0000","last_fullsized":"2024-09-13T14:43:51.650419+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:03:00.416198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":65,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.690702+0000","last_change":"2024-09-13T14:43:16.854299+0000","last_active":"2024-09-13T14:43:50.690702+0000","last_peered":"2024-09-13T14:43:50.690702+0000","last_clean":"2024-09-13T14:43:50.690702+0000","last_became_active":"2024-09-13T14:38:53.900626+0000","last_became_peered":"2024-09-13T14:38:53.900626+0000","last_unstale":"2024-09-13T14:43:50.690702+0000","last_undegraded":"2024-09-13T14:43:50.690702+0000","last_fullsized":"2024-09-13T14:43:50.690702+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:37:21.715778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":281,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.624165+0000","last_change":"2024-09-13T14:37:49.172218+0000","last_active":"2024-09-13T14:43:51.624165+0000","last_peered":"2024-09-13T14:43:51.624165+0000","last_clean":"2024-09-13T14:43:51.624165+0000","last_became_active":"2024-09-13T14:37:49.171718+0000","last_became_peered":"2024-09-13T14:37:49.171718+0000","last_unstale":"2024-09-13T14:43:51.624165+0000","last_undegraded":"2024-09-13T14:43:51.624165+0000","last_fullsized":"2024-09-13T14:43:51.624165+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:46:54.626521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":303,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.624327+0000","last_change":"2024-09-13T14:37:49.230094+0000","last_active":"2024-09-13T14:43:51.624327+0000","last_peered":"2024-09-13T14:43:51.624327+0000","last_clean":"2024-09-13T14:43:51.624327+0000","last_became_active":"2024-09-13T14:37:49.229393+0000","last_became_peered":"2024-09-13T14:37:49.229393+0000","last_unstale":"2024-09-13T14:43:51.624327+0000","last_undegraded":"2024-09-13T14:43:51.624327+0000","last_fullsized":"2024-09-13T14:43:51.624327+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:02:28.690318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":283,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.624494+0000","last_change":"2024-09-13T14:37:48.128447+0000","last_active":"2024-09-13T14:43:51.624494+0000","last_peered":"2024-09-13T14:43:51.624494+0000","last_clean":"2024-09-13T14:43:51.624494+0000","last_became_active":"2024-09-13T14:37:48.128143+0000","last_became_peered":"2024-09-13T14:37:48.128143+0000","last_unstale":"2024-09-13T14:43:51.624494+0000","last_undegraded":"2024-09-13T14:43:51.624494+0000","last_fullsized":"2024-09-13T14:43:51.624494+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:22.204039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":104,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.649966+0000","last_change":"2024-09-13T14:38:03.531683+0000","last_active":"2024-09-13T14:43:51.649966+0000","last_peered":"2024-09-13T14:43:51.649966+0000","last_clean":"2024-09-13T14:43:51.649966+0000","last_became_active":"2024-09-13T14:38:03.531476+0000","last_became_peered":"2024-09-13T14:38:03.531476+0000","last_unstale":"2024-09-13T14:43:51.649966+0000","last_undegraded":"2024-09-13T14:43:51.649966+0000","last_fullsized":"2024-09-13T14:43:51.649966+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:41.557569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":55,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.690220+0000","last_change":"2024-09-13T14:38:53.954281+0000","last_active":"2024-09-13T14:43:50.690220+0000","last_peered":"2024-09-13T14:43:50.690220+0000","last_clean":"2024-09-13T14:43:50.690220+0000","last_became_active":"2024-09-13T14:38:53.900912+0000","last_became_peered":"2024-09-13T14:38:53.900912+0000","last_unstale":"2024-09-13T14:43:50.690220+0000","last_undegraded":"2024-09-13T14:43:50.690220+0000","last_fullsized":"2024-09-13T14:43:50.690220+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:47.876275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":107,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.620645+0000","last_change":"2024-09-13T14:38:54.000782+0000","last_active":"2024-09-13T14:43:50.620645+0000","last_peered":"2024-09-13T14:43:50.620645+0000","last_clean":"2024-09-13T14:43:50.620645+0000","last_became_active":"2024-09-13T14:38:54.000322+0000","last_became_peered":"2024-09-13T14:38:54.000322+0000","last_unstale":"2024-09-13T14:43:50.620645+0000","last_undegraded":"2024-09-13T14:43:50.620645+0000","last_fullsized":"2024-09-13T14:43:50.620645+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:09.057597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":430,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.625481+0000","last_change":"2024-09-13T14:38:53.899535+0000","last_active":"2024-09-13T14:43:51.625481+0000","last_peered":"2024-09-13T14:43:51.625481+0000","last_clean":"2024-09-13T14:43:51.625481+0000","last_became_active":"2024-09-13T14:38:53.899183+0000","last_became_peered":"2024-09-13T14:38:53.899183+0000","last_unstale":"2024-09-13T14:43:51.625481+0000","last_undegraded":"2024-09-13T14:43:51.625481+0000","last_fullsized":"2024-09-13T14:43:51.625481+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:33.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":104,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.650600+0000","last_change":"2024-09-13T14:38:03.543426+0000","last_active":"2024-09-13T14:43:51.650600+0000","last_peered":"2024-09-13T14:43:51.650600+0000","last_clean":"2024-09-13T14:43:51.650600+0000","last_became_active":"2024-09-13T14:38:03.542661+0000","last_became_peered":"2024-09-13T14:38:03.542661+0000","last_unstale":"2024-09-13T14:43:51.650600+0000","last_undegraded":"2024-09-13T14:43:51.650600+0000","last_fullsized":"2024-09-13T14:43:51.650600+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:54.050127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":459,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.626504+0000","last_change":"2024-09-13T14:37:48.131321+0000","last_active":"2024-09-13T14:43:51.626504+0000","last_peered":"2024-09-13T14:43:51.626504+0000","last_clean":"2024-09-13T14:43:51.626504+0000","last_became_active":"2024-09-13T14:37:48.130998+0000","last_became_peered":"2024-09-13T14:37:48.130998+0000","last_unstale":"2024-09-13T14:43:51.626504+0000","last_undegraded":"2024-09-13T14:43:51.626504+0000","last_fullsized":"2024-09-13T14:43:51.626504+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:33:26.597825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":119,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.620641+0000","last_change":"2024-09-13T14:38:03.535454+0000","last_active":"2024-09-13T14:43:50.620641+0000","last_peered":"2024-09-13T14:43:50.620641+0000","last_clean":"2024-09-13T14:43:50.620641+0000","last_became_active":"2024-09-13T14:38:03.535131+0000","last_became_peered":"2024-09-13T14:38:03.535131+0000","last_unstale":"2024-09-13T14:43:50.620641+0000","last_undegraded":"2024-09-13T14:43:50.620641+0000","last_fullsized":"2024-09-13T14:43:50.620641+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:24.477601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":483,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.626587+0000","last_change":"2024-09-13T14:38:03.524939+0000","last_active":"2024-09-13T14:43:51.626587+0000","last_peered":"2024-09-13T14:43:51.626587+0000","last_clean":"2024-09-13T14:43:51.626587+0000","last_became_active":"2024-09-13T14:38:03.524406+0000","last_became_peered":"2024-09-13T14:38:03.524406+0000","last_unstale":"2024-09-13T14:43:51.626587+0000","last_undegraded":"2024-09-13T14:43:51.626587+0000","last_fullsized":"2024-09-13T14:43:51.626587+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:19:45.257424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":111,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.650398+0000","last_change":"2024-09-13T14:38:53.892948+0000","last_active":"2024-09-13T14:43:51.650398+0000","last_peered":"2024-09-13T14:43:51.650398+0000","last_clean":"2024-09-13T14:43:51.650398+0000","last_became_active":"2024-09-13T14:38:53.892665+0000","last_became_peered":"2024-09-13T14:38:53.892665+0000","last_unstale":"2024-09-13T14:43:51.650398+0000","last_undegraded":"2024-09-13T14:43:51.650398+0000","last_fullsized":"2024-09-13T14:43:51.650398+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:08.292359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":432,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.625674+0000","last_change":"2024-09-13T14:37:48.139348+0000","last_active":"2024-09-13T14:43:51.625674+0000","last_peered":"2024-09-13T14:43:51.625674+0000","last_clean":"2024-09-13T14:43:51.625674+0000","last_became_active":"2024-09-13T14:37:48.139115+0000","last_became_peered":"2024-09-13T14:37:48.139115+0000","last_unstale":"2024-09-13T14:43:51.625674+0000","last_undegraded":"2024-09-13T14:43:51.625674+0000","last_fullsized":"2024-09-13T14:43:51.625674+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:32:20.413346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":472,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.626096+0000","last_change":"2024-09-13T14:37:48.172088+0000","last_active":"2024-09-13T14:43:51.626096+0000","last_peered":"2024-09-13T14:43:51.626096+0000","last_clean":"2024-09-13T14:43:51.626096+0000","last_became_active":"2024-09-13T14:37:48.171779+0000","last_became_peered":"2024-09-13T14:37:48.171779+0000","last_unstale":"2024-09-13T14:43:51.626096+0000","last_undegraded":"2024-09-13T14:43:51.626096+0000","last_fullsized":"2024-09-13T14:43:51.626096+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:02.852712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":348,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-13T14:43:57.479682+0000","last_change":"2024-09-13T14:38:53.890236+0000","last_active":"2024-09-13T14:43:57.479682+0000","last_peered":"2024-09-13T14:43:57.479682+0000","last_clean":"2024-09-13T14:43:57.479682+0000","last_became_active":"2024-09-13T14:38:53.889582+0000","last_became_peered":"2024-09-13T14:38:53.889582+0000","last_unstale":"2024-09-13T14:43:57.479682+0000","last_undegraded":"2024-09-13T14:43:57.479682+0000","last_fullsized":"2024-09-13T14:43:57.479682+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:14:29.334774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":458,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.626361+0000","last_change":"2024-09-13T14:38:03.534097+0000","last_active":"2024-09-13T14:43:51.626361+0000","last_peered":"2024-09-13T14:43:51.626361+0000","last_clean":"2024-09-13T14:43:51.626361+0000","last_became_active":"2024-09-13T14:38:03.533794+0000","last_became_peered":"2024-09-13T14:38:03.533794+0000","last_unstale":"2024-09-13T14:43:51.626361+0000","last_undegraded":"2024-09-13T14:43:51.626361+0000","last_fullsized":"2024-09-13T14:43:51.626361+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:23.272098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":104,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.649969+0000","last_change":"2024-09-13T14:38:03.527021+0000","last_active":"2024-09-13T14:43:51.649969+0000","last_peered":"2024-09-13T14:43:51.649969+0000","last_clean":"2024-09-13T14:43:51.649969+0000","last_became_active":"2024-09-13T14:38:03.526698+0000","last_became_peered":"2024-09-13T14:38:03.526698+0000","last_unstale":"2024-09-13T14:43:51.649969+0000","last_undegraded":"2024-09-13T14:43:51.649969+0000","last_fullsized":"2024-09-13T14:43:51.649969+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:01:35.734086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":292,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-13T14:43:57.479684+0000","last_change":"2024-09-13T14:27:12.834931+0000","last_active":"2024-09-13T14:43:57.479684+0000","last_peered":"2024-09-13T14:43:57.479684+0000","last_clean":"2024-09-13T14:43:57.479684+0000","last_became_active":"2024-09-13T14:27:12.834572+0000","last_became_peered":"2024-09-13T14:27:12.834572+0000","last_unstale":"2024-09-13T14:43:57.479684+0000","last_undegraded":"2024-09-13T14:43:57.479684+0000","last_fullsized":"2024-09-13T14:43:57.479684+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:54:21.155689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":104,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.620211+0000","last_change":"2024-09-13T14:37:49.176796+0000","last_active":"2024-09-13T14:43:50.620211+0000","last_peered":"2024-09-13T14:43:50.620211+0000","last_clean":"2024-09-13T14:43:50.620211+0000","last_became_active":"2024-09-13T14:37:49.176246+0000","last_became_peered":"2024-09-13T14:37:49.176246+0000","last_unstale":"2024-09-13T14:43:50.620211+0000","last_undegraded":"2024-09-13T14:43:50.620211+0000","last_fullsized":"2024-09-13T14:43:50.620211+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:29:24.936718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":303,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-13T14:43:57.479601+0000","last_change":"2024-09-13T14:38:03.526333+0000","last_active":"2024-09-13T14:43:57.479601+0000","last_peered":"2024-09-13T14:43:57.479601+0000","last_clean":"2024-09-13T14:43:57.479601+0000","last_became_active":"2024-09-13T14:38:03.526001+0000","last_became_peered":"2024-09-13T14:38:03.526001+0000","last_unstale":"2024-09-13T14:43:57.479601+0000","last_undegraded":"2024-09-13T14:43:57.479601+0000","last_fullsized":"2024-09-13T14:43:57.479601+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:25:25.181271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":104,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-09-13T14:43:50.620403+0000","last_change":"2024-09-13T14:37:49.149582+0000","last_active":"2024-09-13T14:43:50.620403+0000","last_peered":"2024-09-13T14:43:50.620403+0000","last_clean":"2024-09-13T14:43:50.620403+0000","last_became_active":"2024-09-13T14:37:49.148525+0000","last_became_peered":"2024-09-13T14:37:49.148525+0000","last_unstale":"2024-09-13T14:43:50.620403+0000","last_undegraded":"2024-09-13T14:43:50.620403+0000","last_fullsized":"2024-09-13T14:43:50.620403+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:48.655665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":504,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.625858+0000","last_change":"2024-09-13T14:38:03.535255+0000","last_active":"2024-09-13T14:43:51.625858+0000","last_peered":"2024-09-13T14:43:51.625858+0000","last_clean":"2024-09-13T14:43:51.625858+0000","last_became_active":"2024-09-13T14:38:03.533917+0000","last_became_peered":"2024-09-13T14:38:03.533917+0000","last_unstale":"2024-09-13T14:43:51.625858+0000","last_undegraded":"2024-09-13T14:43:51.625858+0000","last_fullsized":"2024-09-13T14:43:51.625858+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:33.559788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":468,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.626797+0000","last_change":"2024-09-13T14:37:48.146784+0000","last_active":"2024-09-13T14:43:51.626797+0000","last_peered":"2024-09-13T14:43:51.626797+0000","last_clean":"2024-09-13T14:43:51.626797+0000","last_became_active":"2024-09-13T14:37:48.146491+0000","last_became_peered":"2024-09-13T14:37:48.146491+0000","last_unstale":"2024-09-13T14:43:51.626797+0000","last_undegraded":"2024-09-13T14:43:51.626797+0000","last_fullsized":"2024-09-13T14:43:51.626797+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:30:54.772513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":283,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.623975+0000","last_change":"2024-09-13T14:37:48.127607+0000","last_active":"2024-09-13T14:43:51.623975+0000","last_peered":"2024-09-13T14:43:51.623975+0000","last_clean":"2024-09-13T14:43:51.623975+0000","last_became_active":"2024-09-13T14:37:48.127316+0000","last_became_peered":"2024-09-13T14:37:48.127316+0000","last_unstale":"2024-09-13T14:43:51.623975+0000","last_undegraded":"2024-09-13T14:43:51.623975+0000","last_fullsized":"2024-09-13T14:43:51.623975+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:53:37.678861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":110,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-13T14:43:51.650547+0000","last_change":"2024-09-13T14:38:03.563200+0000","last_active":"2024-09-13T14:43:51.650547+0000","last_peered":"2024-09-13T14:43:51.650547+0000","last_clean":"2024-09-13T14:43:51.650547+0000","last_became_active":"2024-09-13T14:38:03.534258+0000","last_became_peered":"2024-09-13T14:38:03.534258+0000","last_unstale":"2024-09-13T14:43:51.650547+0000","last_undegraded":"2024-09-13T14:43:51.650547+0000","last_fullsized":"2024-09-13T14:43:51.650547+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:49:07.814419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":2928345112,"num_objects":44684,"num_object_clones":0,"num_object_copies":134052,"num_objects_missing_on_primary":82,"num_objects_missing":82,"num_objects_degraded":5370,"num_objects_misplaced":4816,"num_objects_unfound":0,"num_objects_dirty":44684,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180942,"num_write_kb":7219969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83906,"num_bytes_recovered":5371789336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11033391104,"data_stored":11033378888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94765,"ondisk_log_size":94765,"up":48,"acting":47,"num_store_stats":6},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1538,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":17390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25132,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16793600,"data_stored":16777292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":23,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2969600,"data_stored":2951840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234401,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2141632,"kb_used_data":1669176,"kb_used_omap":1,"kb_used_meta":472446,"kb_avail":91603520,"statfs":{"total":95995035648,"available":93802004480,"internally_reserved":0,"allocated":1709236224,"data_stored":1707684025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":483785163},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":239,"seq":1026497183820,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1706000,"kb_used_data":1414792,"kb_used_omap":8,"kb_used_meta":291191,"kb_avail":92039152,"statfs":{"total":95995035648,"available":94248091648,"internally_reserved":0,"allocated":1448747008,"data_stored":1447187260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8453,"internal_metadata":298180347},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757666,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1962608,"kb_used_data":1295720,"kb_used_omap":1,"kb_used_meta":666878,"kb_avail":91782544,"statfs":{"total":95995035648,"available":93985325056,"internally_reserved":0,"allocated":1326817280,"data_stored":1325276642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":682883526},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019310,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2140808,"kb_used_data":1554300,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":91604344,"statfs":{"total":95995035648,"available":93802848256,"internally_reserved":0,"allocated":1591603200,"data_stored":1590054613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":600570310},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435493,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2718956,"kb_used_data":2145828,"kb_used_omap":5,"kb_used_meta":573114,"kb_avail":91026196,"statfs":{"total":95995035648,"available":93210824704,"internally_reserved":0,"allocated":2197327872,"data_stored":2195791094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":586868862},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":446676599022,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":425201762548,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926071,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3437308,"kb_used_data":2755892,"kb_used_omap":7,"kb_used_meta":681400,"kb_avail":90307844,"statfs":{"total":95995035648,"available":92475232256,"internally_reserved":0,"allocated":2822033408,"data_stored":2820481413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":697753896},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2,1,3,5,4],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6860,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2809401344,"data_stored":2809397272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2191134720,"data_stored":2191130648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1585643520,"data_stored":1585643520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1318191104,"data_stored":1318191104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1435045888,"data_stored":1435041816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1693974528,"data_stored":1693974528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:44:05.057 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph status --format=json 2024-09-13T14:44:05.127 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:05.156 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:05.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:05.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:05.258 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:05.288 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:05.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:05.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:05.390 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:05.414 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:05.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:05.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:05.516 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:05.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:05.540 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:05.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:05.564 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:05.611 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:05.641 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:05.664 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:05.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:05.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:05.766 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:05.790 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:05.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:05.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:05.892 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:05.918 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:05.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:05.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:06.020 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:06.041 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:06.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:06.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:06.143 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:06.165 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:06.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:06.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:06.267 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:06.294 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:06.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:06.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:06.396 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:06.420 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:06.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:06.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:06.522 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:06.547 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:06.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:06.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:06.612 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is unset 2024-09-13T14:44:06.621 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:06 smithi195 ceph-mon[104262]: pgmap v1004: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.6 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 314 op/s; 5286/129435 objects degraded (4.084%); 4724/129435 objects misplaced (3.650%); 2.3 MiB/s, 36 objects/s recovering 2024-09-13T14:44:06.622 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:06 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/870971619' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:44:06.628 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:44:06.628 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:44:06.649 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:06.672 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:06.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:06.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:06.733 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:06 smithi102 ceph-mon[110183]: pgmap v1004: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.6 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 314 op/s; 5286/129435 objects degraded (4.084%); 4724/129435 objects misplaced (3.650%); 2.3 MiB/s, 36 objects/s recovering 2024-09-13T14:44:06.733 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:06 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/870971619' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:44:06.734 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:44:06 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:44:06] "GET /metrics HTTP/1.1" 200 36549 "" "Prometheus/2.51.0" 2024-09-13T14:44:06.734 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:06 smithi102 ceph-mon[115269]: pgmap v1004: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.6 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 314 op/s; 5286/129435 objects degraded (4.084%); 4724/129435 objects misplaced (3.650%); 2.3 MiB/s, 36 objects/s recovering 2024-09-13T14:44:06.734 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:06 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/870971619' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:44:06.774 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:06.796 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:06.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:06.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:06.898 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:06.923 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:06.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:06.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:07.025 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:07.047 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:07.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:07.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:07.149 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:07.172 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:07.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:07.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:07.274 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:07.279 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T14:44:07.297 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:07.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:07.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:07.399 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:07.431 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:07.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:07.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:07.533 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:07.538 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:44:07.557 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:07.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:07.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:07.660 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:07.682 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:07.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:07.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:07.730 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:07 smithi102 ceph-mon[110183]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:44:07.730 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:07 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/870971619' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:44:07.730 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:07 smithi102 ceph-mon[110183]: osdmap e292: 8 total, 8 up, 6 in 2024-09-13T14:44:07.730 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:07 smithi102 ceph-mon[115269]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:44:07.730 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:07 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/870971619' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:44:07.730 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:07 smithi102 ceph-mon[115269]: osdmap e292: 8 total, 8 up, 6 in 2024-09-13T14:44:07.784 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:07.806 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:07.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:07.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:07.908 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:07.929 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:07.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:07.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:07.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:07 smithi195 ceph-mon[104262]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:44:07.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:07 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/870971619' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:44:07.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:07 smithi195 ceph-mon[104262]: osdmap e292: 8 total, 8 up, 6 in 2024-09-13T14:44:08.030 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:08.054 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:08.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:08.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:08.156 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:08.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:08.180 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:08.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:08.281 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:08.302 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:08.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:08.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:08.404 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:08.427 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:08.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:08.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:08.529 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:08.551 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:08.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:08.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:08.653 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:08.675 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:08.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:08.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:08.776 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:08.799 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:08.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:08.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:08.889 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:08 smithi102 ceph-mon[110183]: pgmap v1006: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.5 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 312 op/s; 5183/125031 objects degraded (4.145%); 4631/125031 objects misplaced (3.704%); 2.4 MiB/s, 38 objects/s recovering 2024-09-13T14:44:08.889 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:08 smithi102 ceph-mon[115269]: pgmap v1006: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.5 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 312 op/s; 5183/125031 objects degraded (4.145%); 4631/125031 objects misplaced (3.704%); 2.4 MiB/s, 38 objects/s recovering 2024-09-13T14:44:08.901 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:08.923 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:08.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:08.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:08.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:08 smithi195 ceph-mon[104262]: pgmap v1006: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.5 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 312 op/s; 5183/125031 objects degraded (4.145%); 4631/125031 objects misplaced (3.704%); 2.4 MiB/s, 38 objects/s recovering 2024-09-13T14:44:09.024 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:09.048 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:09.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:09.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:09.149 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:09.180 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:09.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:09.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:09.282 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:09.306 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:09.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:09.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:09.408 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:09.430 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:09.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:09.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:09.491 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:44:09.533 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:09.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:09.556 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:09.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:09.564 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:44:09.564 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:44:09.657 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:09.683 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:09.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:09.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:09.713 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:09 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 5183/125031 objects degraded (4.145%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:09.713 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:09 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 5183/125031 objects degraded (4.145%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:09.785 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:09.809 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:09.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:09.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:09.843 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:44:09.843 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:44:09.864 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:44:09.864 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:44:09.889 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:09 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 5183/125031 objects degraded (4.145%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:09.891 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:44:09.891 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:44:09.909 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:44:09.909 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:44:09.911 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:09.925 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:44:09.926 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:44:09.934 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:09.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:09.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:09.948 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:44:09.948 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:44:09.971 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:44:09.971 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:44:09.992 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:44:09.992 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:44:10.036 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:10.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:10.062 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:10.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:10.163 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:10.190 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:10.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:10.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:10.223 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T14:44:10.292 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:10.335 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:10.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:10.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:10.436 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:10.437 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:44:10.437 INFO:teuthology.orchestra.run.smithi102.stdout:{"fsid":"08cee502-71d9-11ef-bceb-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5183/124773 objects degraded (4.154%), 3 pgs degraded, 2 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1473,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":292,"num_osds":8,"num_up_osds":8,"osd_up_since":1726238267,"num_in_osds":6,"osd_in_since":1726238173,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":85},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering","count":1},{"state_name":"active+recovery_wait+degraded+remapped","count":1}],"num_pgs":89,"num_pools":5,"num_objects":41591,"data_bytes":2728597501,"bytes_used":13896593408,"bytes_avail":562073620480,"bytes_total":575970213888,"degraded_objects":5183,"degraded_total":124773,"degraded_ratio":0.041539435615076982,"misplaced_objects":4631,"misplaced_total":124773,"misplaced_ratio":0.037115401569249758,"recovering_objects_per_sec":38,"recovering_bytes_per_sec":2496321,"recovering_keys_per_sec":0,"num_objects_recovered":381,"num_bytes_recovered":24969216,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":312},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi102.njetxr","status":"up:active","gid":24521}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.102:8443/","prometheus":"http://172.21.15.102:9283/"}},"servicemap":{"epoch":28,"modified":"2024-09-13T14:40:21.476600+0000","services":{}},"progress_events":{}} 2024-09-13T14:44:10.466 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:10.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:10.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:10.520 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:44:10.569 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:10.597 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:10.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:10.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:10.699 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:10.725 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:10.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:10.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:10.826 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:10.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:10.852 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:10.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:10.888 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:10 smithi102 ceph-mon[110183]: pgmap v1007: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.5 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 312 op/s; 5183/124773 objects degraded (4.154%); 4631/124773 objects misplaced (3.712%); 2.4 MiB/s, 38 objects/s recovering 2024-09-13T14:44:10.888 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:10 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/680953363' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:44:10.888 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:44:10.888 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:10 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3640389279' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:44:10.888 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:10 smithi102 ceph-mon[115269]: pgmap v1007: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.5 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 312 op/s; 5183/124773 objects degraded (4.154%); 4631/124773 objects misplaced (3.712%); 2.4 MiB/s, 38 objects/s recovering 2024-09-13T14:44:10.888 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:10 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/680953363' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:44:10.888 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:44:10.889 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:10 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3640389279' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:44:10.935 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:10 smithi195 ceph-mon[104262]: pgmap v1007: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.5 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 312 op/s; 5183/124773 objects degraded (4.154%); 4631/124773 objects misplaced (3.712%); 2.4 MiB/s, 38 objects/s recovering 2024-09-13T14:44:10.935 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:10 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/680953363' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:44:10.935 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:44:10.935 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:10 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3640389279' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:44:10.954 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:11.054 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:11.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:11.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:11.156 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:11.178 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:11.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:11.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:11.280 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:11.302 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:11.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:11.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:11.404 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:11.427 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:11.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:11.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:11.529 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:11.550 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:11.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:11.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:11.652 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:11.674 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:11.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:11.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:11.776 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:11.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:11.806 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:11.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:11.907 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:11.934 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:11.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:11.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:12.036 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:12.062 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:12.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:12.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:12.125 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:12.135 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-13T14:44:12.135 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:44:12.164 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:12.187 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:12.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:12.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:12.289 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:12.313 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:12.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:12.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:12.352 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:44:12.352 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-09-13T14:44:12.352 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-09-13T14:44:12.352 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-13T14:44:12.352 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.352 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:198014 6.3 6:c6a15bb2:::benchmark_data_smithi102_194590_object87390:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-09-13T14:44:12.353 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:34:29.666548+0000", 2024-09-13T14:44:12.353 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 582.68263530399997, 2024-09-13T14:44:12.353 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.24791018200000001, 2024-09-13T14:44:12.353 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:44:12.353 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:12.353 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:44:12.353 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:44:12.353 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:44:12.353 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 198014 2024-09-13T14:44:12.353 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.353 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:44:12.354 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.354 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:44:12.354 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.666548+0000", 2024-09-13T14:44:12.354 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:44:12.354 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.354 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.354 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:44:12.354 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.666548+0000", 2024-09-13T14:44:12.354 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:44:12.354 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.354 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.354 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:44:12.354 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.666547+0000", 2024-09-13T14:44:12.355 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:44:12.355 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.355 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.355 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:44:12.355 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.666551+0000", 2024-09-13T14:44:12.355 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.1499999999999999e-06 2024-09-13T14:44:12.355 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.355 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.355 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:44:12.355 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.666560+0000", 2024-09-13T14:44:12.356 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.8630000000000003e-06 2024-09-13T14:44:12.356 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.356 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.356 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:44:12.356 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.666566+0000", 2024-09-13T14:44:12.356 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7409999999999998e-06 2024-09-13T14:44:12.356 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.356 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.356 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:44:12.356 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.677707+0000", 2024-09-13T14:44:12.356 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011140399 2024-09-13T14:44:12.356 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.357 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.357 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:44:12.357 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.677829+0000", 2024-09-13T14:44:12.357 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000122621 2024-09-13T14:44:12.357 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.357 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.357 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,6", 2024-09-13T14:44:12.357 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.677950+0000", 2024-09-13T14:44:12.357 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000120926 2024-09-13T14:44:12.357 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.357 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.357 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:12.357 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.694719+0000", 2024-09-13T14:44:12.358 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016769071 2024-09-13T14:44:12.358 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.358 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.358 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:44:12.358 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.694726+0000", 2024-09-13T14:44:12.358 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7870000000000004e-06 2024-09-13T14:44:12.358 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.358 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.358 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:12.358 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.914382+0000", 2024-09-13T14:44:12.358 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.219656197 2024-09-13T14:44:12.358 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.359 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.359 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:44:12.359 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.914419+0000", 2024-09-13T14:44:12.359 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6971000000000002e-05 2024-09-13T14:44:12.359 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.359 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.359 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:44:12.359 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.914458+0000", 2024-09-13T14:44:12.359 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9035000000000001e-05 2024-09-13T14:44:12.359 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:44:12.359 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:44:12.359 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:44:12.359 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.360 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.360 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35037.0:198032 6.3 6:cf2708b1:::benchmark_data_smithi102_194590_object87408:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-09-13T14:44:12.360 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:34:29.686356+0000", 2024-09-13T14:44:12.360 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 582.662826946, 2024-09-13T14:44:12.360 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.234390241, 2024-09-13T14:44:12.360 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:44:12.360 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:12.360 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:44:12.360 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35037", 2024-09-13T14:44:12.360 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3905825069", 2024-09-13T14:44:12.360 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 198032 2024-09-13T14:44:12.360 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.361 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:44:12.361 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.361 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:44:12.361 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.686356+0000", 2024-09-13T14:44:12.361 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:44:12.361 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.361 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.361 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:44:12.361 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.686356+0000", 2024-09-13T14:44:12.361 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:44:12.361 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.361 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.361 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:44:12.362 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.686355+0000", 2024-09-13T14:44:12.362 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:44:12.362 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.362 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.362 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:44:12.362 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.686360+0000", 2024-09-13T14:44:12.362 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1070000000000004e-06 2024-09-13T14:44:12.362 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.362 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.362 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:44:12.362 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.686374+0000", 2024-09-13T14:44:12.362 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3878e-05 2024-09-13T14:44:12.362 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.363 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.363 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:44:12.363 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.686381+0000", 2024-09-13T14:44:12.363 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8510000000000001e-06 2024-09-13T14:44:12.363 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.363 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.363 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:44:12.363 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.688802+0000", 2024-09-13T14:44:12.363 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0024206000000000002 2024-09-13T14:44:12.363 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.363 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.363 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:44:12.363 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.688928+0000", 2024-09-13T14:44:12.363 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012623499999999999 2024-09-13T14:44:12.364 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.364 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.364 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,6", 2024-09-13T14:44:12.364 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.690008+0000", 2024-09-13T14:44:12.364 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001080714 2024-09-13T14:44:12.364 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.364 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.364 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:44:12.364 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.694768+0000", 2024-09-13T14:44:12.364 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0047593569999999997 2024-09-13T14:44:12.364 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.364 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.364 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:12.365 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.695055+0000", 2024-09-13T14:44:12.365 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00028694099999999998 2024-09-13T14:44:12.365 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.365 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.365 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:12.365 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.920646+0000", 2024-09-13T14:44:12.365 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.225591033 2024-09-13T14:44:12.365 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.365 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.365 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:44:12.365 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.920712+0000", 2024-09-13T14:44:12.365 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5730999999999998e-05 2024-09-13T14:44:12.365 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.366 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.366 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:44:12.366 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:34:29.920747+0000", 2024-09-13T14:44:12.366 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5027000000000002e-05 2024-09-13T14:44:12.366 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:44:12.366 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:44:12.366 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:44:12.366 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.366 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.366 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113978 7.5 7:a1bc8a89:::benchmark_data_smithi102_227395_object1163:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-13T14:44:12.366 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:47.323151+0000", 2024-09-13T14:44:12.366 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 385.02603248100002, 2024-09-13T14:44:12.366 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.366077738, 2024-09-13T14:44:12.367 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:44:12.367 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:12.367 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:44:12.367 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:44:12.367 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:44:12.367 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113978 2024-09-13T14:44:12.367 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.367 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:44:12.367 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.367 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:44:12.368 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.323151+0000", 2024-09-13T14:44:12.368 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:44:12.368 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.368 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.368 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:44:12.369 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.323151+0000", 2024-09-13T14:44:12.369 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:44:12.369 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.369 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.369 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:44:12.369 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.323149+0000", 2024-09-13T14:44:12.369 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:44:12.369 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.369 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.369 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:44:12.369 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.323159+0000", 2024-09-13T14:44:12.369 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.7049999999999994e-06 2024-09-13T14:44:12.369 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.370 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.370 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:44:12.370 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.323176+0000", 2024-09-13T14:44:12.370 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7649999999999999e-05 2024-09-13T14:44:12.370 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.370 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.370 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:44:12.370 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.323184+0000", 2024-09-13T14:44:12.370 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7579999999999992e-06 2024-09-13T14:44:12.370 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.370 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.370 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:44:12.370 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.323232+0000", 2024-09-13T14:44:12.371 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8402999999999997e-05 2024-09-13T14:44:12.371 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.371 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.371 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:44:12.371 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.323241+0000", 2024-09-13T14:44:12.371 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1990000000000006e-06 2024-09-13T14:44:12.371 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.371 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.371 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:44:12.371 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.120037+0000", 2024-09-13T14:44:12.371 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.79679608999999996 2024-09-13T14:44:12.371 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.371 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.371 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:44:12.372 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.120050+0000", 2024-09-13T14:44:12.372 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2903e-05 2024-09-13T14:44:12.372 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.372 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.372 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:44:12.372 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.628936+0000", 2024-09-13T14:44:12.372 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.508886086 2024-09-13T14:44:12.372 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.372 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.372 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:44:12.372 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.629091+0000", 2024-09-13T14:44:12.372 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015541700000000001 2024-09-13T14:44:12.372 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.373 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.373 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:44:12.373 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.629244+0000", 2024-09-13T14:44:12.373 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015319199999999999 2024-09-13T14:44:12.373 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.373 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.373 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:44:12.373 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.635107+0000", 2024-09-13T14:44:12.373 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0058630619999999996 2024-09-13T14:44:12.373 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.373 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.373 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:12.374 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.643339+0000", 2024-09-13T14:44:12.374 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0082320020000000004 2024-09-13T14:44:12.374 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.374 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.374 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:12.374 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.685466+0000", 2024-09-13T14:44:12.374 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.042126602999999999 2024-09-13T14:44:12.374 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.374 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.374 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:44:12.374 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.685526+0000", 2024-09-13T14:44:12.374 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0146999999999999e-05 2024-09-13T14:44:12.374 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.375 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.375 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:44:12.375 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.689229+0000", 2024-09-13T14:44:12.375 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0037024200000000001 2024-09-13T14:44:12.375 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:44:12.375 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:44:12.375 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:44:12.375 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.375 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.375 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113996 7.5 7:a0c0436d:::benchmark_data_smithi102_227395_object1181:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-13T14:44:12.375 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:48.567765+0000", 2024-09-13T14:44:12.375 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 383.78141842799999, 2024-09-13T14:44:12.375 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1252143800000001, 2024-09-13T14:44:12.375 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:44:12.376 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:12.376 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:44:12.376 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:44:12.376 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:44:12.376 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113996 2024-09-13T14:44:12.376 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.376 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:44:12.376 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.376 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:44:12.376 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.567765+0000", 2024-09-13T14:44:12.376 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:44:12.376 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.377 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.377 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:44:12.377 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.567765+0000", 2024-09-13T14:44:12.377 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:44:12.377 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.377 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.377 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:44:12.377 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.567763+0000", 2024-09-13T14:44:12.377 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-09-13T14:44:12.377 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.381 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.381 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:44:12.381 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.567773+0000", 2024-09-13T14:44:12.381 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0569e-05 2024-09-13T14:44:12.381 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.381 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.381 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:44:12.381 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.567795+0000", 2024-09-13T14:44:12.382 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1739999999999999e-05 2024-09-13T14:44:12.382 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.382 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.382 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:44:12.382 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.567805+0000", 2024-09-13T14:44:12.382 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.8109999999999997e-06 2024-09-13T14:44:12.382 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.382 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.382 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:44:12.382 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.567869+0000", 2024-09-13T14:44:12.382 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4679999999999997e-05 2024-09-13T14:44:12.382 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.382 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.383 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:44:12.383 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.567883+0000", 2024-09-13T14:44:12.383 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3978999999999999e-05 2024-09-13T14:44:12.383 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.383 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.383 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:44:12.383 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.635139+0000", 2024-09-13T14:44:12.383 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0672552609999999 2024-09-13T14:44:12.383 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.383 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.383 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:44:12.383 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.635297+0000", 2024-09-13T14:44:12.383 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000158488 2024-09-13T14:44:12.383 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.384 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.384 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:44:12.384 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.635456+0000", 2024-09-13T14:44:12.384 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015900899999999999 2024-09-13T14:44:12.384 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.384 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.384 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:44:12.384 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.639619+0000", 2024-09-13T14:44:12.384 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0041628070000000001 2024-09-13T14:44:12.384 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.384 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.384 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:12.385 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.643415+0000", 2024-09-13T14:44:12.385 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003795625 2024-09-13T14:44:12.385 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.385 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.385 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:12.385 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.689289+0000", 2024-09-13T14:44:12.385 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.045873979000000002 2024-09-13T14:44:12.385 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.385 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.385 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:44:12.385 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.689334+0000", 2024-09-13T14:44:12.385 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5918000000000003e-05 2024-09-13T14:44:12.385 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.385 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.386 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:44:12.386 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.692979+0000", 2024-09-13T14:44:12.386 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0036447760000000002 2024-09-13T14:44:12.386 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:44:12.386 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:44:12.386 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:44:12.386 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.386 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.386 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:142079 7.5 7:a4bd193f:::benchmark_data_smithi102_227395_object29264:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e268)", 2024-09-13T14:44:12.386 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:40:54.899564+0000", 2024-09-13T14:44:12.386 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 197.44961964399999, 2024-09-13T14:44:12.386 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0097709749999999995, 2024-09-13T14:44:12.386 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:44:12.387 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:12.387 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:44:12.387 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:44:12.387 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:44:12.387 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 142079 2024-09-13T14:44:12.387 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.387 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:44:12.387 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.387 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:44:12.387 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:40:54.899564+0000", 2024-09-13T14:44:12.387 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:44:12.387 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.387 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.387 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:44:12.388 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:40:54.899564+0000", 2024-09-13T14:44:12.388 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:44:12.388 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.388 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.388 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:44:12.388 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:40:54.899562+0000", 2024-09-13T14:44:12.388 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:44:12.388 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.388 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.388 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:44:12.388 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:40:54.899569+0000", 2024-09-13T14:44:12.388 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.0260000000000002e-06 2024-09-13T14:44:12.388 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.389 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.389 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:44:12.389 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:40:54.899585+0000", 2024-09-13T14:44:12.389 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6138999999999999e-05 2024-09-13T14:44:12.389 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.389 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.389 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:44:12.389 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:40:54.899594+0000", 2024-09-13T14:44:12.389 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.6419999999999996e-06 2024-09-13T14:44:12.389 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.389 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.389 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:44:12.390 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:40:54.899644+0000", 2024-09-13T14:44:12.390 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0339e-05 2024-09-13T14:44:12.390 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.390 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.390 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:44:12.390 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:40:54.899783+0000", 2024-09-13T14:44:12.390 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013849399999999999 2024-09-13T14:44:12.390 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.390 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.390 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,6", 2024-09-13T14:44:12.390 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:40:54.899926+0000", 2024-09-13T14:44:12.390 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014349 2024-09-13T14:44:12.390 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.391 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.391 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:12.391 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:40:54.902885+0000", 2024-09-13T14:44:12.391 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0029591470000000001 2024-09-13T14:44:12.391 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.391 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.391 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:44:12.391 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:40:54.902963+0000", 2024-09-13T14:44:12.391 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7649000000000001e-05 2024-09-13T14:44:12.391 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.391 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.392 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:12.392 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:40:54.909239+0000", 2024-09-13T14:44:12.392 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.006275943 2024-09-13T14:44:12.392 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.392 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.392 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:44:12.392 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:40:54.909294+0000", 2024-09-13T14:44:12.392 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4984000000000001e-05 2024-09-13T14:44:12.392 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.392 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.392 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:44:12.392 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:40:54.909335+0000", 2024-09-13T14:44:12.392 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0617000000000001e-05 2024-09-13T14:44:12.393 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:44:12.393 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:44:12.393 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:44:12.393 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.393 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.393 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:142094 7.8 7:123bdd6c:::benchmark_data_smithi102_227395_object29279:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e268)", 2024-09-13T14:44:12.393 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:40:54.933692+0000", 2024-09-13T14:44:12.393 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 197.415491089, 2024-09-13T14:44:12.393 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010588589000000001, 2024-09-13T14:44:12.393 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:44:12.393 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:12.393 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:44:12.393 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:44:12.393 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:44:12.394 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 142094 2024-09-13T14:44:12.394 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.394 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:44:12.394 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.394 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:44:12.394 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:40:54.933692+0000", 2024-09-13T14:44:12.394 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:44:12.394 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.394 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.394 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:44:12.394 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:40:54.933692+0000", 2024-09-13T14:44:12.394 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:44:12.394 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.395 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.395 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:44:12.395 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:40:54.933691+0000", 2024-09-13T14:44:12.395 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:44:12.395 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.395 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.395 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:44:12.395 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:40:54.933697+0000", 2024-09-13T14:44:12.395 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3340000000000001e-06 2024-09-13T14:44:12.395 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.395 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.395 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:44:12.396 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:40:54.933708+0000", 2024-09-13T14:44:12.396 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1545e-05 2024-09-13T14:44:12.396 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.396 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.396 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:44:12.397 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:40:54.933715+0000", 2024-09-13T14:44:12.397 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.0879999999999997e-06 2024-09-13T14:44:12.397 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.397 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.397 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:44:12.397 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:40:54.936965+0000", 2024-09-13T14:44:12.397 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0032499270000000001 2024-09-13T14:44:12.397 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.397 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.397 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:44:12.397 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:40:54.937107+0000", 2024-09-13T14:44:12.397 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000142027 2024-09-13T14:44:12.397 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.398 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.398 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:44:12.398 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:40:54.937209+0000", 2024-09-13T14:44:12.398 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000101569 2024-09-13T14:44:12.398 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.398 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.398 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:12.398 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:40:54.940577+0000", 2024-09-13T14:44:12.398 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0033685690000000001 2024-09-13T14:44:12.398 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.398 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.398 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:12.398 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:40:54.944119+0000", 2024-09-13T14:44:12.399 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0035413039999999999 2024-09-13T14:44:12.399 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.399 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.399 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:44:12.399 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:40:54.944187+0000", 2024-09-13T14:44:12.399 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8329000000000003e-05 2024-09-13T14:44:12.399 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.399 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.399 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:44:12.399 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:40:54.944238+0000", 2024-09-13T14:44:12.399 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1075999999999997e-05 2024-09-13T14:44:12.399 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.399 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.399 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:44:12.400 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:40:54.944281+0000", 2024-09-13T14:44:12.400 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2627999999999999e-05 2024-09-13T14:44:12.400 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:44:12.400 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:44:12.400 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:44:12.400 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.400 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.400 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:148769 7.5 7:a1148f21:::benchmark_data_smithi102_227395_object35954:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-09-13T14:44:12.400 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:41:27.187333+0000", 2024-09-13T14:44:12.400 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 165.16185074500001, 2024-09-13T14:44:12.400 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010473068, 2024-09-13T14:44:12.400 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:44:12.400 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:12.401 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:44:12.401 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:44:12.401 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:44:12.401 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 148769 2024-09-13T14:44:12.401 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.401 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:44:12.401 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.401 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:44:12.401 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:27.187333+0000", 2024-09-13T14:44:12.401 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:44:12.401 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.401 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.401 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:44:12.402 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:27.187333+0000", 2024-09-13T14:44:12.402 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:44:12.402 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.402 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.402 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:44:12.402 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:27.187331+0000", 2024-09-13T14:44:12.402 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:44:12.402 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.403 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.403 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:44:12.403 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:27.187338+0000", 2024-09-13T14:44:12.403 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8970000000000002e-06 2024-09-13T14:44:12.403 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.403 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.403 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:44:12.403 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:27.187349+0000", 2024-09-13T14:44:12.403 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1678999999999999e-05 2024-09-13T14:44:12.403 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.403 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.403 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:44:12.403 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:27.187357+0000", 2024-09-13T14:44:12.404 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.52e-06 2024-09-13T14:44:12.404 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.404 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.404 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:44:12.404 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:27.187400+0000", 2024-09-13T14:44:12.404 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3239999999999999e-05 2024-09-13T14:44:12.404 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.404 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.404 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:44:12.404 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:27.187490+0000", 2024-09-13T14:44:12.404 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.0252000000000003e-05 2024-09-13T14:44:12.404 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.404 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.405 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,6", 2024-09-13T14:44:12.405 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:27.187559+0000", 2024-09-13T14:44:12.405 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8300000000000007e-05 2024-09-13T14:44:12.405 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.405 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.405 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:12.405 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:27.190066+0000", 2024-09-13T14:44:12.405 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0025069549999999999 2024-09-13T14:44:12.405 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.405 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.405 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:44:12.405 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:27.190121+0000", 2024-09-13T14:44:12.405 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4839000000000001e-05 2024-09-13T14:44:12.406 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.406 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.406 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:12.406 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:27.197745+0000", 2024-09-13T14:44:12.406 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0076240539999999999 2024-09-13T14:44:12.406 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.406 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.406 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:44:12.406 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:27.197779+0000", 2024-09-13T14:44:12.406 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4505999999999998e-05 2024-09-13T14:44:12.406 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.406 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.406 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:44:12.407 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:27.197806+0000", 2024-09-13T14:44:12.407 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.6483e-05 2024-09-13T14:44:12.407 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:44:12.407 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:44:12.407 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:44:12.407 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.407 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.407 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:149139 7.8 7:1997e210:::benchmark_data_smithi102_227395_object36324:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-09-13T14:44:12.407 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:41:28.695340+0000", 2024-09-13T14:44:12.407 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 163.653843467, 2024-09-13T14:44:12.407 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0098944770000000005, 2024-09-13T14:44:12.407 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:44:12.407 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:12.407 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:44:12.408 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:44:12.408 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:44:12.408 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 149139 2024-09-13T14:44:12.408 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.408 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:44:12.408 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.408 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:44:12.408 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:28.695340+0000", 2024-09-13T14:44:12.408 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:44:12.408 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.408 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.408 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:44:12.408 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:28.695340+0000", 2024-09-13T14:44:12.408 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:44:12.409 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.409 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.409 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:44:12.409 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:28.695339+0000", 2024-09-13T14:44:12.409 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:44:12.409 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.409 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.409 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:44:12.409 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:28.695346+0000", 2024-09-13T14:44:12.409 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.024e-06 2024-09-13T14:44:12.409 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.409 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.409 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:44:12.410 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:28.695357+0000", 2024-09-13T14:44:12.410 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1686e-05 2024-09-13T14:44:12.410 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.410 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.410 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:44:12.410 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:28.695364+0000", 2024-09-13T14:44:12.410 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3890000000000002e-06 2024-09-13T14:44:12.410 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.410 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.410 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:44:12.410 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:28.695408+0000", 2024-09-13T14:44:12.410 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4128000000000002e-05 2024-09-13T14:44:12.410 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.410 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.411 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:44:12.411 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:28.695530+0000", 2024-09-13T14:44:12.411 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000121966 2024-09-13T14:44:12.411 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.411 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.411 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:44:12.411 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:28.695659+0000", 2024-09-13T14:44:12.411 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012952999999999999 2024-09-13T14:44:12.411 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.411 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.411 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:12.411 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:28.698868+0000", 2024-09-13T14:44:12.411 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0032090230000000001 2024-09-13T14:44:12.412 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.412 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.412 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:44:12.412 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:28.698926+0000", 2024-09-13T14:44:12.412 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7710000000000001e-05 2024-09-13T14:44:12.412 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.412 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.412 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:12.412 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:28.705142+0000", 2024-09-13T14:44:12.412 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0062163169999999998 2024-09-13T14:44:12.412 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.412 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.412 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:44:12.413 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:28.705193+0000", 2024-09-13T14:44:12.413 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0337000000000003e-05 2024-09-13T14:44:12.413 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.413 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.413 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:44:12.413 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:28.705234+0000", 2024-09-13T14:44:12.413 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1514000000000003e-05 2024-09-13T14:44:12.413 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:44:12.413 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:44:12.413 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:44:12.413 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.413 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.413 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:149505 7.8 7:13a65793:::benchmark_data_smithi102_227395_object36690:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-09-13T14:44:12.414 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:41:30.509038+0000", 2024-09-13T14:44:12.414 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 161.84014506, 2024-09-13T14:44:12.414 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011262187999999999, 2024-09-13T14:44:12.414 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:44:12.414 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:12.414 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:44:12.414 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:44:12.414 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:44:12.414 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 149505 2024-09-13T14:44:12.414 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.414 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:44:12.414 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.414 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:44:12.415 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:30.509038+0000", 2024-09-13T14:44:12.415 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:44:12.415 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.415 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.415 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:44:12.415 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:30.509038+0000", 2024-09-13T14:44:12.415 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:44:12.415 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.415 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.415 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:44:12.415 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:30.509037+0000", 2024-09-13T14:44:12.415 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:44:12.415 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.415 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.416 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:44:12.416 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:30.509044+0000", 2024-09-13T14:44:12.416 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.0419999999999997e-06 2024-09-13T14:44:12.416 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.416 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.416 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:44:12.416 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:30.509058+0000", 2024-09-13T14:44:12.416 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4027e-05 2024-09-13T14:44:12.416 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.416 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.416 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:44:12.416 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:30.509065+0000", 2024-09-13T14:44:12.416 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3930000000000002e-06 2024-09-13T14:44:12.417 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.417 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.417 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:44:12.417 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:30.509115+0000", 2024-09-13T14:44:12.417 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9404999999999999e-05 2024-09-13T14:44:12.417 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.417 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.417 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:44:12.417 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:30.509253+0000", 2024-09-13T14:44:12.417 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013882799999999999 2024-09-13T14:44:12.417 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.417 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.417 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:44:12.418 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:30.509393+0000", 2024-09-13T14:44:12.418 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013977099999999999 2024-09-13T14:44:12.418 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.418 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.418 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:12.418 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:30.520090+0000", 2024-09-13T14:44:12.418 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010696639000000001 2024-09-13T14:44:12.418 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.418 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.418 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:44:12.418 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:30.520145+0000", 2024-09-13T14:44:12.418 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5560000000000003e-05 2024-09-13T14:44:12.418 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.419 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.419 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:12.419 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:30.520199+0000", 2024-09-13T14:44:12.419 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3764000000000002e-05 2024-09-13T14:44:12.419 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.419 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.419 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:44:12.419 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:30.520252+0000", 2024-09-13T14:44:12.419 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3214999999999998e-05 2024-09-13T14:44:12.419 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.419 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.419 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:44:12.419 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:30.520300+0000", 2024-09-13T14:44:12.420 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7963999999999997e-05 2024-09-13T14:44:12.420 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:44:12.420 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:44:12.420 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:44:12.420 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.420 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.420 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:152600 7.8 7:183ce7d5:::benchmark_data_smithi102_227395_object39785:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e273)", 2024-09-13T14:44:12.420 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:41:43.800528+0000", 2024-09-13T14:44:12.420 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 148.54865495199999, 2024-09-13T14:44:12.420 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013701814, 2024-09-13T14:44:12.420 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:44:12.420 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:12.420 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:44:12.421 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:44:12.421 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:44:12.421 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 152600 2024-09-13T14:44:12.421 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.421 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:44:12.421 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.421 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:44:12.421 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:43.800528+0000", 2024-09-13T14:44:12.421 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:44:12.421 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.421 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.421 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:44:12.421 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:43.800528+0000", 2024-09-13T14:44:12.422 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:44:12.422 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.422 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.422 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:44:12.422 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:43.800525+0000", 2024-09-13T14:44:12.422 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999971 2024-09-13T14:44:12.422 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.422 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.422 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:44:12.422 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:43.800537+0000", 2024-09-13T14:44:12.422 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2014999999999999e-05 2024-09-13T14:44:12.422 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.422 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.423 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:44:12.423 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:43.800562+0000", 2024-09-13T14:44:12.423 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4919999999999999e-05 2024-09-13T14:44:12.423 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.423 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.423 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:44:12.423 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:43.800572+0000", 2024-09-13T14:44:12.423 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0098000000000001e-05 2024-09-13T14:44:12.423 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.423 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.423 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:44:12.423 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:43.800627+0000", 2024-09-13T14:44:12.423 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4815999999999998e-05 2024-09-13T14:44:12.423 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.424 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.424 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:44:12.424 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:43.800769+0000", 2024-09-13T14:44:12.424 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000141724 2024-09-13T14:44:12.424 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.424 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.424 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:44:12.424 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:43.800893+0000", 2024-09-13T14:44:12.424 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012378899999999999 2024-09-13T14:44:12.424 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.424 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.424 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:44:12.424 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:43.804553+0000", 2024-09-13T14:44:12.425 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0036599200000000001 2024-09-13T14:44:12.425 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.425 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.425 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:12.425 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:43.806143+0000", 2024-09-13T14:44:12.425 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0015901240000000001 2024-09-13T14:44:12.425 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.425 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.425 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:12.425 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:43.814120+0000", 2024-09-13T14:44:12.425 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0079774530000000007 2024-09-13T14:44:12.425 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.425 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.426 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:44:12.426 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:43.814183+0000", 2024-09-13T14:44:12.426 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2430999999999999e-05 2024-09-13T14:44:12.426 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.426 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.426 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:44:12.426 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:43.814230+0000", 2024-09-13T14:44:12.426 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7533e-05 2024-09-13T14:44:12.426 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:44:12.426 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:44:12.426 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:44:12.426 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.426 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.426 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:152607 7.8 7:1dc8b6c2:::benchmark_data_smithi102_227395_object39792:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e273)", 2024-09-13T14:44:12.427 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:41:43.817031+0000", 2024-09-13T14:44:12.427 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 148.53215244500001, 2024-09-13T14:44:12.427 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010852700999999999, 2024-09-13T14:44:12.427 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:44:12.427 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:12.427 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:44:12.427 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:44:12.427 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:44:12.427 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 152607 2024-09-13T14:44:12.427 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.427 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:44:12.427 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.427 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:44:12.428 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:43.817031+0000", 2024-09-13T14:44:12.428 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:44:12.428 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.428 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.428 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:44:12.428 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:43.817031+0000", 2024-09-13T14:44:12.428 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:44:12.428 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.428 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.428 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:44:12.428 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:43.817029+0000", 2024-09-13T14:44:12.428 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:44:12.428 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.429 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.429 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:44:12.429 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:43.817038+0000", 2024-09-13T14:44:12.429 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1540000000000008e-06 2024-09-13T14:44:12.429 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.429 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.429 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:44:12.429 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:43.817053+0000", 2024-09-13T14:44:12.429 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.509e-05 2024-09-13T14:44:12.429 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.429 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.429 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:44:12.429 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:43.817062+0000", 2024-09-13T14:44:12.430 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.969e-06 2024-09-13T14:44:12.430 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.430 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.430 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:44:12.430 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:43.818531+0000", 2024-09-13T14:44:12.430 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0014684699999999999 2024-09-13T14:44:12.430 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.430 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.430 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:44:12.430 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:43.818655+0000", 2024-09-13T14:44:12.430 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012464299999999999 2024-09-13T14:44:12.430 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.430 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.430 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:44:12.431 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:43.818796+0000", 2024-09-13T14:44:12.431 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:12.432 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014099100000000001 2024-09-13T14:44:12.432 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.433 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.433 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:12.433 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:43.822678+0000", 2024-09-13T14:44:12.433 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0038821350000000001 2024-09-13T14:44:12.433 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.433 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.433 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:12.433 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:43.823385+0000", 2024-09-13T14:44:12.433 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00070617800000000001 2024-09-13T14:44:12.433 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.433 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.433 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:44:12.434 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:43.827801+0000", 2024-09-13T14:44:12.434 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.004416728 2024-09-13T14:44:12.434 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.434 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.434 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:44:12.434 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:43.827846+0000", 2024-09-13T14:44:12.434 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4784999999999999e-05 2024-09-13T14:44:12.434 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.434 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.434 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:44:12.434 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:43.827884+0000", 2024-09-13T14:44:12.434 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7481000000000002e-05 2024-09-13T14:44:12.434 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:44:12.435 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:44:12.435 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:44:12.435 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.435 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.435 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:153440 7.8 7:1af21cd8:::benchmark_data_smithi102_227395_object40625:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e273)", 2024-09-13T14:44:12.435 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:41:46.847972+0000", 2024-09-13T14:44:12.435 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 145.50121173400001, 2024-09-13T14:44:12.435 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0097207449999999994, 2024-09-13T14:44:12.435 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:44:12.435 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:12.435 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:44:12.435 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:44:12.435 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:44:12.436 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 153440 2024-09-13T14:44:12.436 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.436 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:44:12.436 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.436 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:44:12.436 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:46.847972+0000", 2024-09-13T14:44:12.436 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:44:12.436 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.436 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.436 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:44:12.436 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:46.847972+0000", 2024-09-13T14:44:12.436 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:44:12.436 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.437 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.437 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:44:12.437 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:46.847969+0000", 2024-09-13T14:44:12.437 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-09-13T14:44:12.437 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.437 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.437 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:44:12.437 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:46.847981+0000", 2024-09-13T14:44:12.437 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1588e-05 2024-09-13T14:44:12.437 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.437 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.437 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:44:12.437 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:46.848005+0000", 2024-09-13T14:44:12.437 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4182999999999999e-05 2024-09-13T14:44:12.438 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.438 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.438 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:44:12.438 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:46.848015+0000", 2024-09-13T14:44:12.438 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.9559999999999992e-06 2024-09-13T14:44:12.438 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.438 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.438 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:44:12.438 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:46.848074+0000", 2024-09-13T14:44:12.438 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.9346999999999999e-05 2024-09-13T14:44:12.438 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.438 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.438 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:44:12.439 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:46.848218+0000", 2024-09-13T14:44:12.439 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014387800000000001 2024-09-13T14:44:12.439 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.439 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.439 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:44:12.439 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:46.848349+0000", 2024-09-13T14:44:12.439 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013041999999999999 2024-09-13T14:44:12.439 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.439 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.439 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:12.439 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:46.855886+0000", 2024-09-13T14:44:12.439 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0075371320000000002 2024-09-13T14:44:12.439 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.439 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.440 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:12.440 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:46.857561+0000", 2024-09-13T14:44:12.440 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001675115 2024-09-13T14:44:12.440 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.440 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.440 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:44:12.440 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:46.857567+0000", 2024-09-13T14:44:12.440 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2740000000000004e-06 2024-09-13T14:44:12.440 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.440 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.440 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:44:12.440 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:46.857656+0000", 2024-09-13T14:44:12.440 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.9091000000000005e-05 2024-09-13T14:44:12.441 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.441 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.441 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:44:12.441 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:46.857692+0000", 2024-09-13T14:44:12.441 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.608e-05 2024-09-13T14:44:12.441 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:44:12.441 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:44:12.441 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:44:12.441 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.441 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.441 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:153441 7.8 7:19d36ace:::benchmark_data_smithi102_227395_object40626:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e273)", 2024-09-13T14:44:12.441 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:41:46.848031+0000", 2024-09-13T14:44:12.441 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 145.50115216699999, 2024-09-13T14:44:12.441 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0097625170000000001, 2024-09-13T14:44:12.442 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:44:12.442 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:12.442 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:44:12.442 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:44:12.442 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:44:12.442 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 153441 2024-09-13T14:44:12.442 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.442 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:44:12.442 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.442 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:44:12.442 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:46.848031+0000", 2024-09-13T14:44:12.442 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:44:12.442 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.442 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.443 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:44:12.443 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:46.848031+0000", 2024-09-13T14:44:12.443 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:44:12.443 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.443 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.443 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:44:12.443 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:46.848031+0000", 2024-09-13T14:44:12.443 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:44:12.443 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.443 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.443 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:44:12.443 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:46.848034+0000", 2024-09-13T14:44:12.443 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4860000000000002e-06 2024-09-13T14:44:12.443 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.444 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.444 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:44:12.444 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:46.848040+0000", 2024-09-13T14:44:12.444 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.6239999999999999e-06 2024-09-13T14:44:12.444 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.444 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.444 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:44:12.444 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:46.848044+0000", 2024-09-13T14:44:12.444 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1060000000000002e-06 2024-09-13T14:44:12.444 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.444 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.444 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:44:12.444 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:46.851887+0000", 2024-09-13T14:44:12.445 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0038429789999999998 2024-09-13T14:44:12.445 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.445 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.445 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:44:12.445 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:46.852012+0000", 2024-09-13T14:44:12.445 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012538800000000001 2024-09-13T14:44:12.445 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.445 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.445 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:44:12.445 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:46.852119+0000", 2024-09-13T14:44:12.445 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000106445 2024-09-13T14:44:12.445 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.445 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.445 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:12.446 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:46.857526+0000", 2024-09-13T14:44:12.446 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0054078499999999996 2024-09-13T14:44:12.446 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.446 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.446 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:44:12.446 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:46.857699+0000", 2024-09-13T14:44:12.446 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00017223600000000001 2024-09-13T14:44:12.446 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.446 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.446 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:12.446 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:46.857724+0000", 2024-09-13T14:44:12.446 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.586e-05 2024-09-13T14:44:12.446 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.447 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.447 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:44:12.447 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:46.857766+0000", 2024-09-13T14:44:12.447 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1828e-05 2024-09-13T14:44:12.447 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.447 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.447 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:44:12.447 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:46.857794+0000", 2024-09-13T14:44:12.447 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7293000000000001e-05 2024-09-13T14:44:12.447 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:44:12.447 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:44:12.447 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:44:12.447 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.448 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.448 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35379.0:153545 7.f e273/245 7:fe82b334:::benchmark_data_smithi102_227395_object40730:head v 273'9541, mlcod=273'9539)", 2024-09-13T14:44:12.448 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:41:47.300691+0000", 2024-09-13T14:44:12.448 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 145.048492356, 2024-09-13T14:44:12.448 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0099254270000000006, 2024-09-13T14:44:12.448 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:44:12.448 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:12.448 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:44:12.448 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.448 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:44:12.448 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:47.300691+0000", 2024-09-13T14:44:12.448 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:44:12.448 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.449 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.449 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:44:12.449 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:47.300691+0000", 2024-09-13T14:44:12.449 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:44:12.449 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.449 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.449 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:44:12.449 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:47.300689+0000", 2024-09-13T14:44:12.449 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:44:12.449 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.449 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.449 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:44:12.450 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:47.300700+0000", 2024-09-13T14:44:12.450 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0862e-05 2024-09-13T14:44:12.450 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.450 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.450 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:44:12.450 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:47.300719+0000", 2024-09-13T14:44:12.450 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8958e-05 2024-09-13T14:44:12.450 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.450 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.450 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:44:12.450 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:47.300728+0000", 2024-09-13T14:44:12.450 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.0110000000000006e-06 2024-09-13T14:44:12.450 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.450 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.451 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:44:12.451 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:47.300797+0000", 2024-09-13T14:44:12.451 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9061999999999997e-05 2024-09-13T14:44:12.451 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.451 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.451 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:44:12.451 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:47.300824+0000", 2024-09-13T14:44:12.451 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7019e-05 2024-09-13T14:44:12.451 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.451 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.451 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:44:12.451 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:47.310577+0000", 2024-09-13T14:44:12.452 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0097528110000000001 2024-09-13T14:44:12.452 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.452 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.452 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:44:12.452 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:47.310616+0000", 2024-09-13T14:44:12.452 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9113000000000003e-05 2024-09-13T14:44:12.452 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:44:12.452 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:44:12.452 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:44:12.452 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.452 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.452 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:158063 7.5 7:a01636a6:::benchmark_data_smithi102_227395_object45248:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-09-13T14:44:12.453 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:42:14.443387+0000", 2024-09-13T14:44:12.453 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 117.905796189, 2024-09-13T14:44:12.453 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0095531009999999996, 2024-09-13T14:44:12.453 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:44:12.453 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:12.453 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:44:12.453 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:44:12.453 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:44:12.453 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 158063 2024-09-13T14:44:12.453 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.453 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:44:12.453 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.453 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:44:12.454 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:42:14.443387+0000", 2024-09-13T14:44:12.454 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:44:12.454 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.454 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.454 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:44:12.454 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:42:14.443387+0000", 2024-09-13T14:44:12.454 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:44:12.454 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.454 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.454 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:44:12.454 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:42:14.443385+0000", 2024-09-13T14:44:12.454 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:44:12.454 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.454 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.455 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:44:12.455 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:42:14.443393+0000", 2024-09-13T14:44:12.455 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6199999999999999e-06 2024-09-13T14:44:12.455 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.455 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.455 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:44:12.455 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:42:14.443409+0000", 2024-09-13T14:44:12.455 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5744999999999999e-05 2024-09-13T14:44:12.456 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.456 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.456 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:44:12.456 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:42:14.443416+0000", 2024-09-13T14:44:12.456 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.0700000000000001e-06 2024-09-13T14:44:12.456 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.456 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.456 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:44:12.456 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:42:14.443489+0000", 2024-09-13T14:44:12.456 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2763999999999997e-05 2024-09-13T14:44:12.456 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.456 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.456 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:44:12.457 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:42:14.443619+0000", 2024-09-13T14:44:12.457 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013068800000000001 2024-09-13T14:44:12.457 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.457 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.457 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,6", 2024-09-13T14:44:12.457 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:42:14.444156+0000", 2024-09-13T14:44:12.457 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00053643899999999999 2024-09-13T14:44:12.457 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.457 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.457 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:12.457 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:42:14.452181+0000", 2024-09-13T14:44:12.457 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0080256519999999994 2024-09-13T14:44:12.457 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.457 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.458 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:44:12.458 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:42:14.452229+0000", 2024-09-13T14:44:12.458 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7613e-05 2024-09-13T14:44:12.458 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.458 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.458 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:12.458 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:42:14.452848+0000", 2024-09-13T14:44:12.458 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00061877099999999995 2024-09-13T14:44:12.458 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.458 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.458 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:44:12.458 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:42:14.452893+0000", 2024-09-13T14:44:12.458 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4963999999999998e-05 2024-09-13T14:44:12.459 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.459 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.459 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:44:12.459 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:42:14.452940+0000", 2024-09-13T14:44:12.459 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7389999999999999e-05 2024-09-13T14:44:12.459 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:44:12.459 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:44:12.459 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:44:12.459 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.459 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.459 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:158064 7.5 7:a9f6ec58:::benchmark_data_smithi102_227395_object45249:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-09-13T14:44:12.459 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:42:14.443430+0000", 2024-09-13T14:44:12.459 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 117.905752855, 2024-09-13T14:44:12.460 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0096224440000000008, 2024-09-13T14:44:12.460 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:44:12.460 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:12.460 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:44:12.460 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:44:12.460 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:44:12.460 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 158064 2024-09-13T14:44:12.460 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.460 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:44:12.460 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.460 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:44:12.460 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:42:14.443430+0000", 2024-09-13T14:44:12.460 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:44:12.461 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.461 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.461 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:44:12.461 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:42:14.443430+0000", 2024-09-13T14:44:12.461 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:44:12.461 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.461 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.461 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:44:12.461 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:42:14.443430+0000", 2024-09-13T14:44:12.461 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:44:12.461 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.461 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.462 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:44:12.462 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:42:14.443433+0000", 2024-09-13T14:44:12.462 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9100000000000001e-06 2024-09-13T14:44:12.462 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.462 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.462 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:44:12.462 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:42:14.443440+0000", 2024-09-13T14:44:12.462 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.542e-06 2024-09-13T14:44:12.462 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.462 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.462 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:44:12.462 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:42:14.443445+0000", 2024-09-13T14:44:12.462 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6319999999999999e-06 2024-09-13T14:44:12.463 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.463 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.463 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:44:12.463 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:42:14.448086+0000", 2024-09-13T14:44:12.463 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0046412800000000002 2024-09-13T14:44:12.463 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.463 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.463 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:44:12.463 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:42:14.448223+0000", 2024-09-13T14:44:12.463 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000136286 2024-09-13T14:44:12.463 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.463 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.464 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,6", 2024-09-13T14:44:12.464 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:42:14.448354+0000", 2024-09-13T14:44:12.464 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013150799999999999 2024-09-13T14:44:12.464 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.464 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.464 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:12.464 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:42:14.452221+0000", 2024-09-13T14:44:12.464 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0038664329999999998 2024-09-13T14:44:12.464 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.464 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.464 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:44:12.464 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:42:14.452234+0000", 2024-09-13T14:44:12.464 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3563e-05 2024-09-13T14:44:12.465 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.465 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.465 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:12.465 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:42:14.452986+0000", 2024-09-13T14:44:12.465 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00075153899999999996 2024-09-13T14:44:12.465 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.465 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.465 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:44:12.465 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:42:14.453023+0000", 2024-09-13T14:44:12.465 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6990000000000003e-05 2024-09-13T14:44:12.465 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.465 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.465 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:44:12.465 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:42:14.453053+0000", 2024-09-13T14:44:12.466 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0215000000000001e-05 2024-09-13T14:44:12.466 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:44:12.466 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:44:12.466 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:44:12.466 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.466 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.466 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:165613 7.8 7:18ac1a50:::benchmark_data_smithi102_227395_object52798:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e285)", 2024-09-13T14:44:12.466 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:42:57.455737+0000", 2024-09-13T14:44:12.466 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 74.893445904000004, 2024-09-13T14:44:12.466 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011263896000000001, 2024-09-13T14:44:12.466 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:44:12.466 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:12.466 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:44:12.467 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:44:12.467 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:44:12.467 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 165613 2024-09-13T14:44:12.467 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.467 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:44:12.467 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.467 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:44:12.467 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:42:57.455737+0000", 2024-09-13T14:44:12.467 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:44:12.467 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.467 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.467 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:44:12.467 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:42:57.455737+0000", 2024-09-13T14:44:12.468 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:44:12.468 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.468 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.468 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:44:12.468 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:42:57.455737+0000", 2024-09-13T14:44:12.468 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:44:12.468 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.468 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.468 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:44:12.468 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:42:57.455739+0000", 2024-09-13T14:44:12.468 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3130000000000001e-06 2024-09-13T14:44:12.468 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.468 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.469 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:44:12.469 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:42:57.455745+0000", 2024-09-13T14:44:12.469 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.665e-06 2024-09-13T14:44:12.469 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.469 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.469 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:44:12.469 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:42:57.455750+0000", 2024-09-13T14:44:12.469 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7999999999999998e-06 2024-09-13T14:44:12.469 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.469 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.469 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:44:12.469 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:42:57.458409+0000", 2024-09-13T14:44:12.469 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0026590089999999999 2024-09-13T14:44:12.470 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.470 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.470 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:44:12.470 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:42:57.458485+0000", 2024-09-13T14:44:12.470 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6339000000000002e-05 2024-09-13T14:44:12.470 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.470 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.470 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:44:12.470 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:42:57.458566+0000", 2024-09-13T14:44:12.470 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0988000000000005e-05 2024-09-13T14:44:12.470 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.470 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.470 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:12.470 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:42:57.460596+0000", 2024-09-13T14:44:12.471 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0020298849999999999 2024-09-13T14:44:12.471 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.471 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.471 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:44:12.471 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:42:57.460608+0000", 2024-09-13T14:44:12.471 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1528999999999999e-05 2024-09-13T14:44:12.471 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.471 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.471 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:12.471 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:42:57.466923+0000", 2024-09-13T14:44:12.471 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0063154429999999996 2024-09-13T14:44:12.471 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.471 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.472 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:44:12.472 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:42:57.466966+0000", 2024-09-13T14:44:12.472 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3000999999999997e-05 2024-09-13T14:44:12.472 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.472 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.472 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:44:12.472 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:42:57.467001+0000", 2024-09-13T14:44:12.472 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5269999999999999e-05 2024-09-13T14:44:12.472 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:44:12.472 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:44:12.472 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:44:12.472 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.472 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.473 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:174116 7.8 7:17ba3a5e:::benchmark_data_smithi102_227395_object61301:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-09-13T14:44:12.473 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:43:33.135945+0000", 2024-09-13T14:44:12.473 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 39.213238148000002, 2024-09-13T14:44:12.473 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013779288000000001, 2024-09-13T14:44:12.473 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:44:12.473 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:12.473 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:44:12.473 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:44:12.473 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:44:12.473 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 174116 2024-09-13T14:44:12.473 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.473 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:44:12.474 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.474 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:44:12.474 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:33.135945+0000", 2024-09-13T14:44:12.474 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:44:12.474 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.474 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.474 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:44:12.474 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:33.135945+0000", 2024-09-13T14:44:12.474 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:44:12.474 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.474 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.474 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:44:12.474 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:33.135945+0000", 2024-09-13T14:44:12.474 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:44:12.475 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.475 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.475 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:44:12.475 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:33.135948+0000", 2024-09-13T14:44:12.475 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.427e-06 2024-09-13T14:44:12.475 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.475 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.475 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:44:12.475 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:33.135958+0000", 2024-09-13T14:44:12.475 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0057e-05 2024-09-13T14:44:12.475 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.475 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.475 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:44:12.475 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:33.135963+0000", 2024-09-13T14:44:12.476 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2789999999999998e-06 2024-09-13T14:44:12.476 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.476 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.476 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:44:12.476 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:33.139806+0000", 2024-09-13T14:44:12.476 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0038431950000000002 2024-09-13T14:44:12.476 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.476 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.476 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:44:12.476 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:33.139959+0000", 2024-09-13T14:44:12.476 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000152187 2024-09-13T14:44:12.476 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.477 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.477 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:44:12.477 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:33.140096+0000", 2024-09-13T14:44:12.477 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013748499999999999 2024-09-13T14:44:12.477 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.477 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.477 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:12.477 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:33.143842+0000", 2024-09-13T14:44:12.477 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0037454720000000001 2024-09-13T14:44:12.477 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.477 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.477 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:44:12.477 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:33.144016+0000", 2024-09-13T14:44:12.478 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000174358 2024-09-13T14:44:12.478 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.478 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.478 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:12.478 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:33.149594+0000", 2024-09-13T14:44:12.478 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0055778440000000002 2024-09-13T14:44:12.478 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.478 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.478 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:44:12.478 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:33.149674+0000", 2024-09-13T14:44:12.478 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0240000000000004e-05 2024-09-13T14:44:12.478 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.478 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.479 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:44:12.479 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:33.149724+0000", 2024-09-13T14:44:12.479 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0321000000000002e-05 2024-09-13T14:44:12.479 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:44:12.479 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:44:12.479 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:44:12.479 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.479 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.479 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35379.0:179878 7.6 e290/254 7:66deb6ae:::benchmark_data_smithi102_227395_object67063:head v 290'11318, mlcod=0'0)", 2024-09-13T14:44:12.479 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:43:53.901082+0000", 2024-09-13T14:44:12.479 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 18.448101614999999, 2024-09-13T14:44:12.479 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.095177660999999997, 2024-09-13T14:44:12.479 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:44:12.480 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:12.480 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:44:12.480 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.480 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:44:12.480 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:53.901082+0000", 2024-09-13T14:44:12.480 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:44:12.480 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.480 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.480 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:44:12.480 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:53.901082+0000", 2024-09-13T14:44:12.480 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:44:12.480 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.480 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.481 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:44:12.481 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:53.901079+0000", 2024-09-13T14:44:12.481 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-09-13T14:44:12.481 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.481 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.481 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:44:12.481 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:53.901094+0000", 2024-09-13T14:44:12.481 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4562000000000001e-05 2024-09-13T14:44:12.482 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.482 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.482 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:44:12.482 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:53.993874+0000", 2024-09-13T14:44:12.482 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.092779529999999999 2024-09-13T14:44:12.482 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.482 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.482 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:44:12.482 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:53.993885+0000", 2024-09-13T14:44:12.482 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1620999999999999e-05 2024-09-13T14:44:12.482 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.482 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.482 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:44:12.483 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:53.993938+0000", 2024-09-13T14:44:12.483 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2377999999999999e-05 2024-09-13T14:44:12.483 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.483 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.483 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:44:12.483 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:53.993953+0000", 2024-09-13T14:44:12.483 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5244e-05 2024-09-13T14:44:12.483 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.483 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.483 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:44:12.483 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:53.996222+0000", 2024-09-13T14:44:12.483 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0022691709999999999 2024-09-13T14:44:12.483 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.483 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.484 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:44:12.484 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:53.996259+0000", 2024-09-13T14:44:12.484 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7354999999999997e-05 2024-09-13T14:44:12.484 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:44:12.484 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:44:12.484 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:44:12.484 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.484 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.484 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:179880 7.5 7:a9d4c919:::benchmark_data_smithi102_227395_object67065:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-13T14:44:12.484 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:43:53.930440+0000", 2024-09-13T14:44:12.484 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 18.418743461999998, 2024-09-13T14:44:12.484 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.067706946000000004, 2024-09-13T14:44:12.484 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:44:12.485 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:12.485 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:44:12.485 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:44:12.485 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:44:12.485 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 179880 2024-09-13T14:44:12.485 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.485 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:44:12.485 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.485 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:44:12.485 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:53.930440+0000", 2024-09-13T14:44:12.485 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:44:12.486 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.486 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.486 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:44:12.486 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:53.930440+0000", 2024-09-13T14:44:12.486 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:44:12.486 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.486 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.486 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:44:12.486 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:53.930438+0000", 2024-09-13T14:44:12.486 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:44:12.486 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.486 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.486 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:44:12.486 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:53.930447+0000", 2024-09-13T14:44:12.487 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8980000000000002e-06 2024-09-13T14:44:12.487 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.487 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.487 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:44:12.487 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:53.993881+0000", 2024-09-13T14:44:12.487 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.063434120999999996 2024-09-13T14:44:12.487 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.487 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.487 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:44:12.487 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:53.993896+0000", 2024-09-13T14:44:12.487 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4504e-05 2024-09-13T14:44:12.487 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.487 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.488 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:44:12.488 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:53.993948+0000", 2024-09-13T14:44:12.488 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2512999999999997e-05 2024-09-13T14:44:12.488 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.488 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.488 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:44:12.488 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:53.994076+0000", 2024-09-13T14:44:12.488 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000128033 2024-09-13T14:44:12.488 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.488 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.488 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,6", 2024-09-13T14:44:12.488 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:53.994199+0000", 2024-09-13T14:44:12.488 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012273199999999999 2024-09-13T14:44:12.489 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.489 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.489 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:12.489 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:53.997458+0000", 2024-09-13T14:44:12.489 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0032595620000000001 2024-09-13T14:44:12.489 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.489 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.489 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:44:12.489 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:53.997512+0000", 2024-09-13T14:44:12.489 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3834e-05 2024-09-13T14:44:12.489 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.489 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.489 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:12.490 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:53.998079+0000", 2024-09-13T14:44:12.490 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00056707800000000005 2024-09-13T14:44:12.490 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.490 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.490 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:44:12.490 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:53.998110+0000", 2024-09-13T14:44:12.490 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0870000000000001e-05 2024-09-13T14:44:12.490 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:12.490 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:12.490 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:44:12.490 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:53.998147+0000", 2024-09-13T14:44:12.490 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6606000000000002e-05 2024-09-13T14:44:12.490 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:44:12.490 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:44:12.491 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:44:12.491 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:44:12.491 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:44:12.491 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:44:12.513 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:12.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:12.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:12.615 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:12.638 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:12.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:12.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:12.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:12 smithi102 ceph-mon[110183]: pgmap v1008: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.5 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 312 op/s; 5183/124773 objects degraded (4.154%); 4631/124773 objects misplaced (3.712%); 2.4 MiB/s, 38 objects/s recovering 2024-09-13T14:44:12.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:12 smithi102 ceph-mon[115269]: pgmap v1008: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.5 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 312 op/s; 5183/124773 objects degraded (4.154%); 4631/124773 objects misplaced (3.712%); 2.4 MiB/s, 38 objects/s recovering 2024-09-13T14:44:12.740 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:12.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:12.766 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:12.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:12.867 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:12.883 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:12 smithi195 ceph-mon[104262]: pgmap v1008: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.5 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 312 op/s; 5183/124773 objects degraded (4.154%); 4631/124773 objects misplaced (3.712%); 2.4 MiB/s, 38 objects/s recovering 2024-09-13T14:44:12.891 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:12.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:12.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:12.915 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:44:12.993 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:12.994 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:13.015 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:13.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:13.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:13.117 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:13.140 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:13.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:13.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:13.242 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:13.263 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:13.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:13.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:13.360 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T14:44:13.364 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:13.425 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:13.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:13.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:13.527 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:13.550 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:13.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:13.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:13.599 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:44:13.652 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:13.674 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:13.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:13.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:13.776 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:13.800 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:13.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:13.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:13.902 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:13.928 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:13.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:13.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:14.029 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:14.053 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:14.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:14.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:14.155 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:14.183 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:14.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:14.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:14.285 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:14.311 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:14.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:14.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:14.413 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:14.436 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:14.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:14.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:14.538 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:14.565 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:14.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:14.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:14.667 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:14.690 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:14.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:14.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:14.791 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:14 smithi102 ceph-mon[110183]: pgmap v1009: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.4 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 325 op/s; 5074/119670 objects degraded (4.240%); 4533/119670 objects misplaced (3.788%); 2.5 MiB/s, 40 objects/s recovering 2024-09-13T14:44:14.791 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:14 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 5074/119670 objects degraded (4.240%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:14.791 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:14 smithi102 ceph-mon[115269]: pgmap v1009: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.4 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 325 op/s; 5074/119670 objects degraded (4.240%); 4533/119670 objects misplaced (3.788%); 2.5 MiB/s, 40 objects/s recovering 2024-09-13T14:44:14.791 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:14 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 5074/119670 objects degraded (4.240%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:14.791 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:14.813 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:14.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:14.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:14.831 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:14 smithi195 ceph-mon[104262]: pgmap v1009: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.4 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 325 op/s; 5074/119670 objects degraded (4.240%); 4533/119670 objects misplaced (3.788%); 2.5 MiB/s, 40 objects/s recovering 2024-09-13T14:44:14.831 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:14 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 5074/119670 objects degraded (4.240%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:14.914 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:14.939 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:14.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:14.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:15.040 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:15.067 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:15.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:15.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:15.169 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:15.195 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:15.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:15.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:15.297 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:15.325 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:15.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:15.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:15.427 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:15.444 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:44:15.445 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-13T14:44:15.445 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:15.445 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35379.0:186837 7.1 e292/289 7:8b1fadfb:::benchmark_data_smithi102_227395_object74022:head v 292'11888, mlcod=292'11886)", 2024-09-13T14:44:15.445 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:44:15.442374+0000", 2024-09-13T14:44:15.445 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.001329598, 2024-09-13T14:44:15.445 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001362025, 2024-09-13T14:44:15.445 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:44:15.445 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T14:44:15.445 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:44:15.445 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:15.446 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:44:15.446 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:44:15.442374+0000", 2024-09-13T14:44:15.446 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:44:15.446 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:15.446 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:15.446 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:44:15.446 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:44:15.442374+0000", 2024-09-13T14:44:15.446 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:44:15.446 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:15.447 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:15.447 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:44:15.447 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:44:15.442373+0000", 2024-09-13T14:44:15.447 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:44:15.447 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:15.447 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:15.447 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:44:15.447 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:44:15.442380+0000", 2024-09-13T14:44:15.447 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9840000000000004e-06 2024-09-13T14:44:15.447 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:15.447 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:15.447 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:44:15.447 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:44:15.442391+0000", 2024-09-13T14:44:15.447 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.111e-05 2024-09-13T14:44:15.448 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:15.448 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:15.448 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:44:15.448 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:44:15.442398+0000", 2024-09-13T14:44:15.448 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.0129999999999996e-06 2024-09-13T14:44:15.448 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:15.448 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:15.448 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:44:15.448 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:44:15.442465+0000", 2024-09-13T14:44:15.448 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6919000000000001e-05 2024-09-13T14:44:15.448 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:44:15.448 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:44:15.449 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:44:15.449 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:44:15.442487+0000", 2024-09-13T14:44:15.449 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1437000000000002e-05 2024-09-13T14:44:15.449 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:44:15.449 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:44:15.449 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:44:15.449 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:44:15.449 INFO:teuthology.orchestra.run.smithi195.stdout: ], 2024-09-13T14:44:15.449 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 1 2024-09-13T14:44:15.449 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:44:15.451 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:15.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:15.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:15.542 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:15.553 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:15.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:15.578 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:15.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:15.680 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:15.707 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:15.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:15.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:15.808 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:15.835 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:15.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:15.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:15.937 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:15.966 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:15.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:15.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:16.067 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:16.090 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:16.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:16.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:16.133 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T14:44:16.192 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:16.214 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:16.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:16.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:16.316 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:16.347 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:16.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:16.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:16.428 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:44:16.449 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:16.479 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:16.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:16.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:16.581 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:16.605 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:16.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:16.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:16.623 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:44:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:44:16] "GET /metrics HTTP/1.1" 200 36542 "" "Prometheus/2.51.0" 2024-09-13T14:44:16.631 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is unset 2024-09-13T14:44:16.708 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:16.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:16.731 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:16.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:16.833 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:16.856 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:16.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:16.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:16.906 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:16 smithi195 ceph-mon[104262]: pgmap v1010: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.4 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 325 op/s; 5074/119670 objects degraded (4.240%); 4533/119670 objects misplaced (3.788%); 2.5 MiB/s, 40 objects/s recovering 2024-09-13T14:44:16.906 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:16 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1353048623' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:44:16.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:16 smithi102 ceph-mon[110183]: pgmap v1010: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.4 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 325 op/s; 5074/119670 objects degraded (4.240%); 4533/119670 objects misplaced (3.788%); 2.5 MiB/s, 40 objects/s recovering 2024-09-13T14:44:16.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:16 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1353048623' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:44:16.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:16 smithi102 ceph-mon[115269]: pgmap v1010: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.4 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 325 op/s; 5074/119670 objects degraded (4.240%); 4533/119670 objects misplaced (3.788%); 2.5 MiB/s, 40 objects/s recovering 2024-09-13T14:44:16.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:16 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1353048623' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:44:16.958 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:16.981 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:16.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:16.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:17.083 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:17.110 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:17.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:17.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:17.212 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:17.236 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:17.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:17.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:17.338 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:17.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:17.360 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:17.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:17.461 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:17.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:17.485 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:17.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:17.586 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:17.608 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:17.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:17.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:17.695 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:17 smithi195 ceph-mon[104262]: from='client.45980 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:44:17.695 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:17 smithi195 ceph-mon[104262]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:44:17.695 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:17 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1353048623' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:44:17.695 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:17 smithi195 ceph-mon[104262]: osdmap e293: 8 total, 8 up, 6 in 2024-09-13T14:44:17.710 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:17.733 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:17.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:17.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:17.828 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:17 smithi102 ceph-mon[115269]: from='client.45980 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:44:17.828 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:17 smithi102 ceph-mon[115269]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:44:17.828 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:17 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1353048623' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:44:17.829 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:17 smithi102 ceph-mon[115269]: osdmap e293: 8 total, 8 up, 6 in 2024-09-13T14:44:17.829 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:17 smithi102 ceph-mon[110183]: from='client.45980 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:44:17.829 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:17 smithi102 ceph-mon[110183]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:44:17.829 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:17 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1353048623' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:44:17.829 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:17 smithi102 ceph-mon[110183]: osdmap e293: 8 total, 8 up, 6 in 2024-09-13T14:44:17.835 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:17.860 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:17.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:17.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:17.962 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:17.985 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:17.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:17.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:18.087 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:18.111 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:18.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:18.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:18.123 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:44:18.124 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:44:18.213 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:18.237 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:18.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:18.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:18.316 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:44:18.316 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-13T14:44:18.316 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-09-13T14:44:18.317 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-09-13T14:44:18.317 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:44:18.339 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:18.362 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:18.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:18.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:18.464 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:18.485 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:18.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:18.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:18.587 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:18.610 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:18.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:18.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:18.712 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:18.733 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:18.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:18.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:18.835 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:18.857 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:18.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:18.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:18.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:18 smithi102 ceph-mon[110183]: pgmap v1012: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.3 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 343 op/s; 4955/114729 objects degraded (4.319%); 4434/114729 objects misplaced (3.865%); 2.7 MiB/s, 42 objects/s recovering 2024-09-13T14:44:18.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:18 smithi102 ceph-mon[115269]: pgmap v1012: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.3 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 343 op/s; 4955/114729 objects degraded (4.319%); 4434/114729 objects misplaced (3.865%); 2.7 MiB/s, 42 objects/s recovering 2024-09-13T14:44:18.941 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":1010,"stamp":"2024-09-13T14:44:15.540070+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2617120765,"num_objects":39890,"num_object_clones":0,"num_object_copies":119670,"num_objects_missing_on_primary":82,"num_objects_missing":82,"num_objects_degraded":5074,"num_objects_misplaced":4533,"num_objects_unfound":0,"num_objects_dirty":39890,"num_whiteouts":0,"num_read":6377,"num_read_kb":13834,"num_write":186887,"num_write_kb":7235010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84608,"num_bytes_recovered":5416291469,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96169,"ondisk_log_size":96169,"up":267,"acting":265,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":299,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":13263268,"kb_used_data":9965300,"kb_used_omap":26,"kb_used_meta":3297893,"kb_avail":549207644,"statfs":{"total":575970213888,"available":562388627456,"internally_reserved":0,"allocated":10204467200,"data_stored":10195137015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27144,"internal_metadata":3377042936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-213319680,"num_objects":-3255,"num_object_clones":0,"num_object_copies":-9765,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-212,"num_objects_misplaced":-191,"num_objects_unfound":0,"num_objects_dirty":-3255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3255,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":26411008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002614"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":478,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.620296+0000","last_change":"2024-09-13T14:36:13.859927+0000","last_active":"2024-09-13T14:44:06.620296+0000","last_peered":"2024-09-13T14:44:06.620296+0000","last_clean":"2024-09-13T14:44:06.620296+0000","last_became_active":"2024-09-13T14:36:13.859505+0000","last_became_peered":"2024-09-13T14:36:13.859505+0000","last_unstale":"2024-09-13T14:44:06.620296+0000","last_undegraded":"2024-09-13T14:44:06.620296+0000","last_fullsized":"2024-09-13T14:44:06.620296+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:12:20.989627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":295,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.634815+0000","last_change":"2024-09-13T14:37:48.130968+0000","last_active":"2024-09-13T14:44:06.634815+0000","last_peered":"2024-09-13T14:44:06.634815+0000","last_clean":"2024-09-13T14:44:06.634815+0000","last_became_active":"2024-09-13T14:37:48.130603+0000","last_became_peered":"2024-09-13T14:37:48.130603+0000","last_unstale":"2024-09-13T14:44:06.634815+0000","last_undegraded":"2024-09-13T14:44:06.634815+0000","last_fullsized":"2024-09-13T14:44:06.634815+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:42.626386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":91,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-13T14:43:57.565066+0000","last_change":"2024-09-13T14:36:13.862181+0000","last_active":"2024-09-13T14:43:57.565066+0000","last_peered":"2024-09-13T14:43:57.565066+0000","last_clean":"2024-09-13T14:43:57.565066+0000","last_became_active":"2024-09-13T14:36:13.861873+0000","last_became_peered":"2024-09-13T14:36:13.861873+0000","last_unstale":"2024-09-13T14:43:57.565066+0000","last_undegraded":"2024-09-13T14:43:57.565066+0000","last_fullsized":"2024-09-13T14:43:57.565066+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:14.085700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":115,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.618620+0000","last_change":"2024-09-13T14:38:53.896672+0000","last_active":"2024-09-13T14:44:06.618620+0000","last_peered":"2024-09-13T14:44:06.618620+0000","last_clean":"2024-09-13T14:44:06.618620+0000","last_became_active":"2024-09-13T14:38:53.896346+0000","last_became_peered":"2024-09-13T14:38:53.896346+0000","last_unstale":"2024-09-13T14:44:06.618620+0000","last_undegraded":"2024-09-13T14:44:06.618620+0000","last_fullsized":"2024-09-13T14:44:06.618620+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:57.789686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":115,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.620555+0000","last_change":"2024-09-13T14:38:53.893021+0000","last_active":"2024-09-13T14:44:06.620555+0000","last_peered":"2024-09-13T14:44:06.620555+0000","last_clean":"2024-09-13T14:44:06.620555+0000","last_became_active":"2024-09-13T14:38:53.892668+0000","last_became_peered":"2024-09-13T14:38:53.892668+0000","last_unstale":"2024-09-13T14:44:06.620555+0000","last_undegraded":"2024-09-13T14:44:06.620555+0000","last_fullsized":"2024-09-13T14:44:06.620555+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:57.180738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":263,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.634587+0000","last_change":"2024-09-13T14:37:48.134642+0000","last_active":"2024-09-13T14:44:06.634587+0000","last_peered":"2024-09-13T14:44:06.634587+0000","last_clean":"2024-09-13T14:44:06.634587+0000","last_became_active":"2024-09-13T14:37:48.134402+0000","last_became_peered":"2024-09-13T14:37:48.134402+0000","last_unstale":"2024-09-13T14:44:06.634587+0000","last_undegraded":"2024-09-13T14:44:06.634587+0000","last_fullsized":"2024-09-13T14:44:06.634587+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:51.313663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":496,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.621364+0000","last_change":"2024-09-13T14:38:53.903022+0000","last_active":"2024-09-13T14:44:06.621364+0000","last_peered":"2024-09-13T14:44:06.621364+0000","last_clean":"2024-09-13T14:44:06.621364+0000","last_became_active":"2024-09-13T14:38:53.902448+0000","last_became_peered":"2024-09-13T14:38:53.902448+0000","last_unstale":"2024-09-13T14:44:06.621364+0000","last_undegraded":"2024-09-13T14:44:06.621364+0000","last_fullsized":"2024-09-13T14:44:06.621364+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:24.690401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":118,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.618195+0000","last_change":"2024-09-13T14:38:03.554500+0000","last_active":"2024-09-13T14:44:06.618195+0000","last_peered":"2024-09-13T14:44:06.618195+0000","last_clean":"2024-09-13T14:44:06.618195+0000","last_became_active":"2024-09-13T14:38:03.532910+0000","last_became_peered":"2024-09-13T14:38:03.532910+0000","last_unstale":"2024-09-13T14:44:06.618195+0000","last_undegraded":"2024-09-13T14:44:06.618195+0000","last_fullsized":"2024-09-13T14:44:06.618195+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:27:50.843498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"292'11465","reported_seq":22369,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:12.262776+0000","last_change":"2024-09-13T14:41:53.220083+0000","last_active":"2024-09-13T14:44:12.262776+0000","last_peered":"2024-09-13T14:44:12.262776+0000","last_clean":"2024-09-13T14:44:12.262776+0000","last_became_active":"2024-09-13T14:41:53.200084+0000","last_became_peered":"2024-09-13T14:41:53.200084+0000","last_unstale":"2024-09-13T14:44:12.262776+0000","last_undegraded":"2024-09-13T14:44:12.262776+0000","last_fullsized":"2024-09-13T14:44:12.262776+0000","mapping_epoch":275,"log_start":"219'5800","ondisk_log_start":"219'5800","created":205,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5665,"log_dups_size":0,"ondisk_log_size":5665,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:08.897524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158269440,"num_objects":2415,"num_object_clones":0,"num_object_copies":7245,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11465,"num_write_kb":444160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9152,"num_bytes_recovered":594608128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":55,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-13T14:43:57.574775+0000","last_change":"2024-09-13T14:38:53.940238+0000","last_active":"2024-09-13T14:43:57.574775+0000","last_peered":"2024-09-13T14:43:57.574775+0000","last_clean":"2024-09-13T14:43:57.574775+0000","last_became_active":"2024-09-13T14:38:53.892825+0000","last_became_peered":"2024-09-13T14:38:53.892825+0000","last_unstale":"2024-09-13T14:43:57.574775+0000","last_undegraded":"2024-09-13T14:43:57.574775+0000","last_fullsized":"2024-09-13T14:43:57.574775+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:46.666581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":111,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-13T14:43:57.564375+0000","last_change":"2024-09-13T14:38:53.898907+0000","last_active":"2024-09-13T14:43:57.564375+0000","last_peered":"2024-09-13T14:43:57.564375+0000","last_clean":"2024-09-13T14:43:57.564375+0000","last_became_active":"2024-09-13T14:38:53.898572+0000","last_became_peered":"2024-09-13T14:38:53.898572+0000","last_unstale":"2024-09-13T14:43:57.564375+0000","last_undegraded":"2024-09-13T14:43:57.564375+0000","last_fullsized":"2024-09-13T14:43:57.564375+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:29.222150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"292'11774","reported_seq":25969,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:12.407118+0000","last_change":"2024-09-13T14:38:57.930032+0000","last_active":"2024-09-13T14:44:12.407118+0000","last_peered":"2024-09-13T14:44:12.407118+0000","last_clean":"2024-09-13T14:44:12.407118+0000","last_became_active":"2024-09-13T14:38:57.921458+0000","last_became_peered":"2024-09-13T14:38:57.921458+0000","last_unstale":"2024-09-13T14:44:12.407118+0000","last_undegraded":"2024-09-13T14:44:12.407118+0000","last_fullsized":"2024-09-13T14:44:12.407118+0000","mapping_epoch":256,"log_start":"219'6039","ondisk_log_start":"219'6039","created":205,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5735,"log_dups_size":0,"ondisk_log_size":5735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:27:12.294038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161742848,"num_objects":2468,"num_object_clones":0,"num_object_copies":7404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11774,"num_write_kb":455744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6990,"num_bytes_recovered":458096640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":55,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-13T14:43:57.574615+0000","last_change":"2024-09-13T14:38:53.904885+0000","last_active":"2024-09-13T14:43:57.574615+0000","last_peered":"2024-09-13T14:43:57.574615+0000","last_clean":"2024-09-13T14:43:57.574615+0000","last_became_active":"2024-09-13T14:38:53.891726+0000","last_became_peered":"2024-09-13T14:38:53.891726+0000","last_unstale":"2024-09-13T14:43:57.574615+0000","last_undegraded":"2024-09-13T14:43:57.574615+0000","last_fullsized":"2024-09-13T14:43:57.574615+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:57.079205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":301,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.874133+0000","last_change":"2024-09-13T14:37:48.160504+0000","last_active":"2024-09-13T14:44:02.874133+0000","last_peered":"2024-09-13T14:44:02.874133+0000","last_clean":"2024-09-13T14:44:02.874133+0000","last_became_active":"2024-09-13T14:37:48.160181+0000","last_became_peered":"2024-09-13T14:37:48.160181+0000","last_unstale":"2024-09-13T14:44:02.874133+0000","last_undegraded":"2024-09-13T14:44:02.874133+0000","last_fullsized":"2024-09-13T14:44:02.874133+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:37.087008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"292'11633","reported_seq":13449,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:11.808286+0000","last_change":"2024-09-13T14:38:04.531914+0000","last_active":"2024-09-13T14:44:11.808286+0000","last_peered":"2024-09-13T14:44:11.808286+0000","last_clean":"2024-09-13T14:44:11.808286+0000","last_became_active":"2024-09-13T14:38:04.523448+0000","last_became_peered":"2024-09-13T14:38:04.523448+0000","last_unstale":"2024-09-13T14:44:11.808286+0000","last_undegraded":"2024-09-13T14:44:11.808286+0000","last_fullsized":"2024-09-13T14:44:11.808286+0000","mapping_epoch":245,"log_start":"219'5700","ondisk_log_start":"219'5700","created":205,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5933,"log_dups_size":0,"ondisk_log_size":5933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:06.536896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166526976,"num_objects":2541,"num_object_clones":0,"num_object_copies":7623,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11633,"num_write_kb":453568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":811,"num_bytes_recovered":53149696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":292,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-13T14:43:57.574445+0000","last_change":"2024-09-13T14:37:49.611774+0000","last_active":"2024-09-13T14:43:57.574445+0000","last_peered":"2024-09-13T14:43:57.574445+0000","last_clean":"2024-09-13T14:43:57.574445+0000","last_became_active":"2024-09-13T14:37:49.611088+0000","last_became_peered":"2024-09-13T14:37:49.611088+0000","last_unstale":"2024-09-13T14:43:57.574445+0000","last_undegraded":"2024-09-13T14:43:57.574445+0000","last_fullsized":"2024-09-13T14:43:57.574445+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:08.933213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":51,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-13T14:43:57.574093+0000","last_change":"2024-09-13T14:38:53.899211+0000","last_active":"2024-09-13T14:43:57.574093+0000","last_peered":"2024-09-13T14:43:57.574093+0000","last_clean":"2024-09-13T14:43:57.574093+0000","last_became_active":"2024-09-13T14:38:53.898927+0000","last_became_peered":"2024-09-13T14:38:53.898927+0000","last_unstale":"2024-09-13T14:43:57.574093+0000","last_undegraded":"2024-09-13T14:43:57.574093+0000","last_fullsized":"2024-09-13T14:43:57.574093+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:00:28.857611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"292'11509","reported_seq":11609,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:12.398816+0000","last_change":"2024-09-13T14:35:24.004651+0000","last_active":"2024-09-13T14:44:12.398816+0000","last_peered":"2024-09-13T14:44:12.398816+0000","last_clean":"2024-09-13T14:44:12.398816+0000","last_became_active":"2024-09-13T14:35:24.004358+0000","last_became_peered":"2024-09-13T14:35:24.004358+0000","last_unstale":"2024-09-13T14:44:12.398816+0000","last_undegraded":"2024-09-13T14:44:12.398816+0000","last_fullsized":"2024-09-13T14:44:12.398816+0000","mapping_epoch":205,"log_start":"219'5184","ondisk_log_start":"219'5184","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6325,"log_dups_size":0,"ondisk_log_size":6325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:57:38.023669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161415168,"num_objects":2463,"num_object_clones":0,"num_object_copies":7389,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11509,"num_write_kb":447104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":102,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-13T14:43:57.564974+0000","last_change":"2024-09-13T14:37:49.168003+0000","last_active":"2024-09-13T14:43:57.564974+0000","last_peered":"2024-09-13T14:43:57.564974+0000","last_clean":"2024-09-13T14:43:57.564974+0000","last_became_active":"2024-09-13T14:37:49.167638+0000","last_became_peered":"2024-09-13T14:37:49.167638+0000","last_unstale":"2024-09-13T14:43:57.564974+0000","last_undegraded":"2024-09-13T14:43:57.564974+0000","last_fullsized":"2024-09-13T14:43:57.564974+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:29.051981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":108,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.618987+0000","last_change":"2024-09-13T14:38:03.528381+0000","last_active":"2024-09-13T14:44:06.618987+0000","last_peered":"2024-09-13T14:44:06.618987+0000","last_clean":"2024-09-13T14:44:06.618987+0000","last_became_active":"2024-09-13T14:38:03.527282+0000","last_became_peered":"2024-09-13T14:38:03.527282+0000","last_unstale":"2024-09-13T14:44:06.618987+0000","last_undegraded":"2024-09-13T14:44:06.618987+0000","last_fullsized":"2024-09-13T14:44:06.618987+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:24.435067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.f","version":"292'11570","reported_seq":11881,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:11.844949+0000","last_change":"2024-09-13T14:38:16.371079+0000","last_active":"2024-09-13T14:44:11.844949+0000","last_peered":"2024-09-13T14:44:11.844949+0000","last_clean":"2024-09-13T14:44:11.844949+0000","last_became_active":"2024-09-13T14:38:12.859883+0000","last_became_peered":"2024-09-13T14:38:12.859883+0000","last_unstale":"2024-09-13T14:44:11.844949+0000","last_undegraded":"2024-09-13T14:44:11.844949+0000","last_fullsized":"2024-09-13T14:44:11.844949+0000","mapping_epoch":245,"log_start":"219'5663","ondisk_log_start":"219'5663","created":205,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5907,"log_dups_size":0,"ondisk_log_size":5907,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:08.436532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164888576,"num_objects":2516,"num_object_clones":0,"num_object_copies":7548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11570,"num_write_kb":450752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":3670016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":466,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.620435+0000","last_change":"2024-09-13T14:38:03.549595+0000","last_active":"2024-09-13T14:44:06.620435+0000","last_peered":"2024-09-13T14:44:06.620435+0000","last_clean":"2024-09-13T14:44:06.620435+0000","last_became_active":"2024-09-13T14:38:03.533331+0000","last_became_peered":"2024-09-13T14:38:03.533331+0000","last_unstale":"2024-09-13T14:44:06.620435+0000","last_undegraded":"2024-09-13T14:44:06.620435+0000","last_fullsized":"2024-09-13T14:44:06.620435+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:59:02.470343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":276,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-13T14:43:57.481008+0000","last_change":"2024-09-13T14:37:48.127600+0000","last_active":"2024-09-13T14:43:57.481008+0000","last_peered":"2024-09-13T14:43:57.481008+0000","last_clean":"2024-09-13T14:43:57.481008+0000","last_became_active":"2024-09-13T14:37:48.127349+0000","last_became_peered":"2024-09-13T14:37:48.127349+0000","last_unstale":"2024-09-13T14:43:57.481008+0000","last_undegraded":"2024-09-13T14:43:57.481008+0000","last_fullsized":"2024-09-13T14:43:57.481008+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:57.268374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"292'11579","reported_seq":21576,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:12.254507+0000","last_change":"2024-09-13T14:42:30.227259+0000","last_active":"2024-09-13T14:44:12.254507+0000","last_peered":"2024-09-13T14:44:12.254507+0000","last_clean":"2024-09-13T14:44:12.254507+0000","last_became_active":"2024-09-13T14:42:30.209102+0000","last_became_peered":"2024-09-13T14:42:30.209102+0000","last_unstale":"2024-09-13T14:44:12.254507+0000","last_undegraded":"2024-09-13T14:44:12.254507+0000","last_fullsized":"2024-09-13T14:44:12.254507+0000","mapping_epoch":281,"log_start":"219'5900","ondisk_log_start":"219'5900","created":205,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5679,"log_dups_size":0,"ondisk_log_size":5679,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:56:16.193287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159318016,"num_objects":2431,"num_object_clones":0,"num_object_copies":7293,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11579,"num_write_kb":448320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4926,"num_bytes_recovered":313851904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":209,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-13T14:43:57.479607+0000","last_change":"2024-09-13T14:37:48.127604+0000","last_active":"2024-09-13T14:43:57.479607+0000","last_peered":"2024-09-13T14:43:57.479607+0000","last_clean":"2024-09-13T14:43:57.479607+0000","last_became_active":"2024-09-13T14:37:48.127300+0000","last_became_peered":"2024-09-13T14:37:48.127300+0000","last_unstale":"2024-09-13T14:43:57.479607+0000","last_undegraded":"2024-09-13T14:43:57.479607+0000","last_fullsized":"2024-09-13T14:43:57.479607+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:51.999506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":329,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-13T14:43:57.479557+0000","last_change":"2024-09-13T14:38:03.529967+0000","last_active":"2024-09-13T14:43:57.479557+0000","last_peered":"2024-09-13T14:43:57.479557+0000","last_clean":"2024-09-13T14:43:57.479557+0000","last_became_active":"2024-09-13T14:38:03.529646+0000","last_became_peered":"2024-09-13T14:38:03.529646+0000","last_unstale":"2024-09-13T14:43:57.479557+0000","last_undegraded":"2024-09-13T14:43:57.479557+0000","last_fullsized":"2024-09-13T14:43:57.479557+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:52:05.193279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"292'11636","reported_seq":20822,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:11.844023+0000","last_change":"2024-09-13T14:40:34.583237+0000","last_active":"2024-09-13T14:44:11.844023+0000","last_peered":"2024-09-13T14:44:11.844023+0000","last_clean":"2024-09-13T14:44:11.844023+0000","last_became_active":"2024-09-13T14:40:34.563989+0000","last_became_peered":"2024-09-13T14:40:34.563989+0000","last_unstale":"2024-09-13T14:44:11.844023+0000","last_undegraded":"2024-09-13T14:44:11.844023+0000","last_fullsized":"2024-09-13T14:44:11.844023+0000","mapping_epoch":266,"log_start":"219'5700","ondisk_log_start":"219'5700","created":205,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5936,"log_dups_size":0,"ondisk_log_size":5936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:34:43.006051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164036632,"num_objects":2504,"num_object_clones":0,"num_object_copies":7512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2504,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11636,"num_write_kb":452417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4489,"num_bytes_recovered":289865752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":290,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-13T14:44:02.874165+0000","last_change":"2024-09-13T14:36:13.860367+0000","last_active":"2024-09-13T14:44:02.874165+0000","last_peered":"2024-09-13T14:44:02.874165+0000","last_clean":"2024-09-13T14:44:02.874165+0000","last_became_active":"2024-09-13T14:36:13.859982+0000","last_became_peered":"2024-09-13T14:36:13.859982+0000","last_unstale":"2024-09-13T14:44:02.874165+0000","last_undegraded":"2024-09-13T14:44:02.874165+0000","last_fullsized":"2024-09-13T14:44:02.874165+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:08:23.311350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":285,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-13T14:43:57.479765+0000","last_change":"2024-09-13T14:37:48.147360+0000","last_active":"2024-09-13T14:43:57.479765+0000","last_peered":"2024-09-13T14:43:57.479765+0000","last_clean":"2024-09-13T14:43:57.479765+0000","last_became_active":"2024-09-13T14:37:48.146885+0000","last_became_peered":"2024-09-13T14:37:48.146885+0000","last_unstale":"2024-09-13T14:43:57.479765+0000","last_undegraded":"2024-09-13T14:43:57.479765+0000","last_fullsized":"2024-09-13T14:43:57.479765+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:27.841810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"292'11590","reported_seq":20883,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:12.374338+0000","last_change":"2024-09-13T14:38:39.650332+0000","last_active":"2024-09-13T14:44:12.374338+0000","last_peered":"2024-09-13T14:44:12.374338+0000","last_clean":"2024-09-13T14:44:12.374338+0000","last_became_active":"2024-09-13T14:38:39.640835+0000","last_became_peered":"2024-09-13T14:38:39.640835+0000","last_unstale":"2024-09-13T14:44:12.374338+0000","last_undegraded":"2024-09-13T14:44:12.374338+0000","last_fullsized":"2024-09-13T14:44:12.374338+0000","mapping_epoch":251,"log_start":"219'5851","ondisk_log_start":"219'5851","created":205,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5739,"log_dups_size":0,"ondisk_log_size":5739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:48.176606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162791424,"num_objects":2484,"num_object_clones":0,"num_object_copies":7452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11590,"num_write_kb":450368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4544,"num_bytes_recovered":292945920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":484,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.619937+0000","last_change":"2024-09-13T14:38:53.900041+0000","last_active":"2024-09-13T14:44:06.619937+0000","last_peered":"2024-09-13T14:44:06.619937+0000","last_clean":"2024-09-13T14:44:06.619937+0000","last_became_active":"2024-09-13T14:38:53.899203+0000","last_became_peered":"2024-09-13T14:38:53.899203+0000","last_unstale":"2024-09-13T14:44:06.619937+0000","last_undegraded":"2024-09-13T14:44:06.619937+0000","last_fullsized":"2024-09-13T14:44:06.619937+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:28:07.883094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":416,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.621427+0000","last_change":"2024-09-13T14:26:15.823198+0000","last_active":"2024-09-13T14:44:06.621427+0000","last_peered":"2024-09-13T14:44:06.621427+0000","last_clean":"2024-09-13T14:44:06.621427+0000","last_became_active":"2024-09-13T14:26:15.822717+0000","last_became_peered":"2024-09-13T14:26:15.822717+0000","last_unstale":"2024-09-13T14:44:06.621427+0000","last_undegraded":"2024-09-13T14:44:06.621427+0000","last_fullsized":"2024-09-13T14:44:06.621427+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:22.471622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.6","version":"292'11681","reported_seq":20835,"reported_epoch":292,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-13T14:44:11.861410+0000","last_change":"2024-09-13T14:38:55.132683+0000","last_active":"2024-09-13T14:44:11.861410+0000","last_peered":"2024-09-13T14:44:11.861410+0000","last_clean":"2024-09-13T14:36:13.124368+0000","last_became_active":"2024-09-13T14:38:55.120883+0000","last_became_peered":"2024-09-13T14:38:55.120883+0000","last_unstale":"2024-09-13T14:44:11.861410+0000","last_undegraded":"2024-09-13T14:38:55.026610+0000","last_fullsized":"2024-09-13T14:44:11.861410+0000","mapping_epoch":254,"log_start":"219'5719","ondisk_log_start":"219'5719","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5962,"log_dups_size":0,"ondisk_log_size":5962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164954112,"num_objects":2517,"num_object_clones":0,"num_object_copies":7551,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":530,"num_objects_misplaced":1987,"num_objects_unfound":0,"num_objects_dirty":2517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11681,"num_write_kb":454336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4475,"num_bytes_recovered":282066944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,3,5],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3,4","objects":251},{"shards":"0,3,5","objects":2266}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":426,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-13T14:43:57.574117+0000","last_change":"2024-09-13T14:41:53.196835+0000","last_active":"2024-09-13T14:43:57.574117+0000","last_peered":"2024-09-13T14:43:57.574117+0000","last_clean":"2024-09-13T14:43:57.574117+0000","last_became_active":"2024-09-13T14:41:53.196387+0000","last_became_peered":"2024-09-13T14:41:53.196387+0000","last_unstale":"2024-09-13T14:43:57.574117+0000","last_undegraded":"2024-09-13T14:43:57.574117+0000","last_fullsized":"2024-09-13T14:43:57.574117+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:07:41.927740+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:49.972303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":531,"reported_epoch":291,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:43:57.564572+0000","last_change":"2024-09-13T14:38:54.904329+0000","last_active":"2024-09-13T14:43:57.564572+0000","last_peered":"2024-09-13T14:43:57.564572+0000","last_clean":"2024-09-13T14:38:39.648017+0000","last_became_active":"2024-09-13T14:38:54.903798+0000","last_became_peered":"2024-09-13T14:38:54.903798+0000","last_unstale":"2024-09-13T14:43:57.564572+0000","last_undegraded":"2024-09-13T14:38:54.890505+0000","last_fullsized":"2024-09-13T14:38:54.889701+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:18.120931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,3],"avail_no_missing":["5","3","6"],"object_location_counts":[{"shards":"3,5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":419,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.621582+0000","last_change":"2024-09-13T14:37:49.160719+0000","last_active":"2024-09-13T14:44:06.621582+0000","last_peered":"2024-09-13T14:44:06.621582+0000","last_clean":"2024-09-13T14:44:06.621582+0000","last_became_active":"2024-09-13T14:37:49.160401+0000","last_became_peered":"2024-09-13T14:37:49.160401+0000","last_unstale":"2024-09-13T14:44:06.621582+0000","last_undegraded":"2024-09-13T14:44:06.621582+0000","last_fullsized":"2024-09-13T14:44:06.621582+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:21.737945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":136,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-13T14:43:57.479473+0000","last_change":"2024-09-13T14:37:48.172940+0000","last_active":"2024-09-13T14:43:57.479473+0000","last_peered":"2024-09-13T14:43:57.479473+0000","last_clean":"2024-09-13T14:43:57.479473+0000","last_became_active":"2024-09-13T14:37:48.172060+0000","last_became_peered":"2024-09-13T14:37:48.172060+0000","last_unstale":"2024-09-13T14:43:57.479473+0000","last_undegraded":"2024-09-13T14:43:57.479473+0000","last_fullsized":"2024-09-13T14:43:57.479473+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:07.480645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.5","version":"292'11639","reported_seq":28015,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:12.376502+0000","last_change":"2024-09-13T14:39:34.446699+0000","last_active":"2024-09-13T14:44:12.376502+0000","last_peered":"2024-09-13T14:44:12.376502+0000","last_clean":"2024-09-13T14:44:12.376502+0000","last_became_active":"2024-09-13T14:38:04.600616+0000","last_became_peered":"2024-09-13T14:38:04.600616+0000","last_unstale":"2024-09-13T14:44:12.376502+0000","last_undegraded":"2024-09-13T14:44:12.376502+0000","last_fullsized":"2024-09-13T14:44:12.376502+0000","mapping_epoch":245,"log_start":"219'5316","ondisk_log_start":"219'5316","created":205,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6323,"log_dups_size":0,"ondisk_log_size":6323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:00:44.321678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164036608,"num_objects":2503,"num_object_clones":0,"num_object_copies":7509,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11639,"num_write_kb":452544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8120,"num_bytes_recovered":521076736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":509,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-13T14:43:57.574029+0000","last_change":"2024-09-13T14:26:15.835178+0000","last_active":"2024-09-13T14:43:57.574029+0000","last_peered":"2024-09-13T14:43:57.574029+0000","last_clean":"2024-09-13T14:43:57.574029+0000","last_became_active":"2024-09-13T14:26:15.820283+0000","last_became_peered":"2024-09-13T14:26:15.820283+0000","last_unstale":"2024-09-13T14:43:57.574029+0000","last_undegraded":"2024-09-13T14:43:57.574029+0000","last_fullsized":"2024-09-13T14:43:57.574029+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:30.340038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":108,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.618376+0000","last_change":"2024-09-13T14:38:03.525682+0000","last_active":"2024-09-13T14:44:06.618376+0000","last_peered":"2024-09-13T14:44:06.618376+0000","last_clean":"2024-09-13T14:44:06.618376+0000","last_became_active":"2024-09-13T14:38:03.525406+0000","last_became_peered":"2024-09-13T14:38:03.525406+0000","last_unstale":"2024-09-13T14:44:06.618376+0000","last_undegraded":"2024-09-13T14:44:06.618376+0000","last_fullsized":"2024-09-13T14:44:06.618376+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:41:28.970300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":115,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.620592+0000","last_change":"2024-09-13T14:38:53.991738+0000","last_active":"2024-09-13T14:44:06.620592+0000","last_peered":"2024-09-13T14:44:06.620592+0000","last_clean":"2024-09-13T14:44:06.620592+0000","last_became_active":"2024-09-13T14:38:53.991417+0000","last_became_peered":"2024-09-13T14:38:53.991417+0000","last_unstale":"2024-09-13T14:44:06.620592+0000","last_undegraded":"2024-09-13T14:44:06.620592+0000","last_fullsized":"2024-09-13T14:44:06.620592+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:50:38.559838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"292'11559","reported_seq":29502,"reported_epoch":292,"state":"active+recovering","last_fresh":"2024-09-13T14:44:11.841130+0000","last_change":"2024-09-13T14:43:50.028785+0000","last_active":"2024-09-13T14:44:11.841130+0000","last_peered":"2024-09-13T14:44:11.841130+0000","last_clean":"2024-09-13T14:37:46.290754+0000","last_became_active":"2024-09-13T14:38:17.381753+0000","last_became_peered":"2024-09-13T14:38:17.381753+0000","last_unstale":"2024-09-13T14:44:11.841130+0000","last_undegraded":"2024-09-13T14:44:11.841130+0000","last_fullsized":"2024-09-13T14:44:11.841130+0000","mapping_epoch":245,"log_start":"219'5659","ondisk_log_start":"219'5659","created":205,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5900,"log_dups_size":0,"ondisk_log_size":5900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166920192,"num_objects":2547,"num_object_clones":0,"num_object_copies":7641,"num_objects_missing_on_primary":82,"num_objects_missing":82,"num_objects_degraded":0,"num_objects_misplaced":2546,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11559,"num_write_kb":451392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8921,"num_bytes_recovered":584581120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"0,4,5","objects":2546},{"shards":"0,4,6","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":830,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.620179+0000","last_change":"2024-09-13T14:37:48.131799+0000","last_active":"2024-09-13T14:44:06.620179+0000","last_peered":"2024-09-13T14:44:06.620179+0000","last_clean":"2024-09-13T14:44:06.620179+0000","last_became_active":"2024-09-13T14:37:48.130837+0000","last_became_peered":"2024-09-13T14:37:48.130837+0000","last_unstale":"2024-09-13T14:44:06.620179+0000","last_undegraded":"2024-09-13T14:44:06.620179+0000","last_fullsized":"2024-09-13T14:44:06.620179+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:06:32.663770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":349,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.618439+0000","last_change":"2024-09-13T14:37:49.301531+0000","last_active":"2024-09-13T14:44:06.618439+0000","last_peered":"2024-09-13T14:44:06.618439+0000","last_clean":"2024-09-13T14:44:06.618439+0000","last_became_active":"2024-09-13T14:37:49.301249+0000","last_became_peered":"2024-09-13T14:37:49.301249+0000","last_unstale":"2024-09-13T14:44:06.618439+0000","last_undegraded":"2024-09-13T14:44:06.618439+0000","last_fullsized":"2024-09-13T14:44:06.618439+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:10.977710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":118,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-13T14:43:57.565999+0000","last_change":"2024-09-13T14:38:53.901427+0000","last_active":"2024-09-13T14:43:57.565999+0000","last_peered":"2024-09-13T14:43:57.565999+0000","last_clean":"2024-09-13T14:43:57.565999+0000","last_became_active":"2024-09-13T14:38:53.901157+0000","last_became_peered":"2024-09-13T14:38:53.901157+0000","last_unstale":"2024-09-13T14:43:57.565999+0000","last_undegraded":"2024-09-13T14:43:57.565999+0000","last_fullsized":"2024-09-13T14:43:57.565999+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:28:21.713025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"292'11525","reported_seq":34772,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:11.826839+0000","last_change":"2024-09-13T14:43:16.851222+0000","last_active":"2024-09-13T14:44:11.826839+0000","last_peered":"2024-09-13T14:44:11.826839+0000","last_clean":"2024-09-13T14:44:11.826839+0000","last_became_active":"2024-09-13T14:38:16.427510+0000","last_became_peered":"2024-09-13T14:38:16.427510+0000","last_unstale":"2024-09-13T14:44:11.826839+0000","last_undegraded":"2024-09-13T14:44:11.826839+0000","last_fullsized":"2024-09-13T14:44:11.826839+0000","mapping_epoch":245,"log_start":"219'5618","ondisk_log_start":"219'5618","created":205,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5907,"log_dups_size":0,"ondisk_log_size":5907,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:27:26.281938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162070528,"num_objects":2473,"num_object_clones":0,"num_object_copies":7419,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11525,"num_write_kb":447936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11571,"num_bytes_recovered":747372544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":1000,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-13T14:43:57.564626+0000","last_change":"2024-09-13T14:37:49.147498+0000","last_active":"2024-09-13T14:43:57.564626+0000","last_peered":"2024-09-13T14:43:57.564626+0000","last_clean":"2024-09-13T14:43:57.564626+0000","last_became_active":"2024-09-13T14:37:49.147028+0000","last_became_peered":"2024-09-13T14:37:49.147028+0000","last_unstale":"2024-09-13T14:43:57.564626+0000","last_undegraded":"2024-09-13T14:43:57.564626+0000","last_fullsized":"2024-09-13T14:43:57.564626+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:11:29.050178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":298,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-13T14:43:57.479898+0000","last_change":"2024-09-13T14:38:03.528133+0000","last_active":"2024-09-13T14:43:57.479898+0000","last_peered":"2024-09-13T14:43:57.479898+0000","last_clean":"2024-09-13T14:43:57.479898+0000","last_became_active":"2024-09-13T14:38:03.527829+0000","last_became_peered":"2024-09-13T14:38:03.527829+0000","last_unstale":"2024-09-13T14:43:57.479898+0000","last_undegraded":"2024-09-13T14:43:57.479898+0000","last_fullsized":"2024-09-13T14:43:57.479898+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:48.254169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":469,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.621142+0000","last_change":"2024-09-13T14:38:53.899755+0000","last_active":"2024-09-13T14:44:06.621142+0000","last_peered":"2024-09-13T14:44:06.621142+0000","last_clean":"2024-09-13T14:44:06.621142+0000","last_became_active":"2024-09-13T14:38:53.899448+0000","last_became_peered":"2024-09-13T14:38:53.899448+0000","last_unstale":"2024-09-13T14:44:06.621142+0000","last_undegraded":"2024-09-13T14:44:06.621142+0000","last_fullsized":"2024-09-13T14:44:06.621142+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:55.897035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"292'11830","reported_seq":20964,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:12.755585+0000","last_change":"2024-09-13T14:43:51.649951+0000","last_active":"2024-09-13T14:44:12.755585+0000","last_peered":"2024-09-13T14:44:12.755585+0000","last_clean":"2024-09-13T14:44:12.755585+0000","last_became_active":"2024-09-13T14:43:51.626628+0000","last_became_peered":"2024-09-13T14:43:51.626628+0000","last_unstale":"2024-09-13T14:44:12.755585+0000","last_undegraded":"2024-09-13T14:44:12.755585+0000","last_fullsized":"2024-09-13T14:44:12.755585+0000","mapping_epoch":289,"log_start":"216'4963","ondisk_log_start":"216'4963","created":205,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6867,"log_dups_size":0,"ondisk_log_size":6867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:37:06.795173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162004992,"num_objects":2472,"num_object_clones":0,"num_object_copies":7416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11830,"num_write_kb":457664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4502,"num_bytes_recovered":228851712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":102,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-13T14:43:57.564998+0000","last_change":"2024-09-13T14:37:49.149770+0000","last_active":"2024-09-13T14:43:57.564998+0000","last_peered":"2024-09-13T14:43:57.564998+0000","last_clean":"2024-09-13T14:43:57.564998+0000","last_became_active":"2024-09-13T14:37:49.148991+0000","last_became_peered":"2024-09-13T14:37:49.148991+0000","last_unstale":"2024-09-13T14:43:57.564998+0000","last_undegraded":"2024-09-13T14:43:57.564998+0000","last_fullsized":"2024-09-13T14:43:57.564998+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:04:04.193366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":1167,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.626688+0000","last_change":"2024-09-13T14:30:58.928302+0000","last_active":"2024-09-13T14:44:06.626688+0000","last_peered":"2024-09-13T14:44:06.626688+0000","last_clean":"2024-09-13T14:44:06.626688+0000","last_became_active":"2024-09-13T14:30:58.927718+0000","last_became_peered":"2024-09-13T14:30:58.927718+0000","last_unstale":"2024-09-13T14:44:06.626688+0000","last_undegraded":"2024-09-13T14:44:06.626688+0000","last_fullsized":"2024-09-13T14:44:06.626688+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:37.323036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":109,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-13T14:43:57.564731+0000","last_change":"2024-09-13T14:38:03.534628+0000","last_active":"2024-09-13T14:43:57.564731+0000","last_peered":"2024-09-13T14:43:57.564731+0000","last_clean":"2024-09-13T14:43:57.564731+0000","last_became_active":"2024-09-13T14:38:03.534262+0000","last_became_peered":"2024-09-13T14:38:03.534262+0000","last_unstale":"2024-09-13T14:43:57.564731+0000","last_undegraded":"2024-09-13T14:43:57.564731+0000","last_fullsized":"2024-09-13T14:43:57.564731+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:31.168783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"292'11588","reported_seq":25768,"reported_epoch":292,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:44:12.274181+0000","last_change":"2024-09-13T14:38:53.944111+0000","last_active":"2024-09-13T14:44:12.274181+0000","last_peered":"2024-09-13T14:44:12.274181+0000","last_clean":"2024-09-13T14:37:47.047672+0000","last_became_active":"2024-09-13T14:37:49.664294+0000","last_became_peered":"2024-09-13T14:37:49.664294+0000","last_unstale":"2024-09-13T14:44:12.274181+0000","last_undegraded":"2024-09-13T14:38:53.940822+0000","last_fullsized":"2024-09-13T14:37:49.126627+0000","mapping_epoch":240,"log_start":"219'5900","ondisk_log_start":"219'5900","created":205,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5688,"log_dups_size":0,"ondisk_log_size":5688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:11.766692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161480704,"num_objects":2464,"num_object_clones":0,"num_object_copies":7392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4535,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11588,"num_write_kb":449664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7009,"num_bytes_recovered":459341824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3],"avail_no_missing":["5","0","3"],"object_location_counts":[{"shards":"0,3,5","objects":2464},{"shards":"3,5","objects":-1996}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":121,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-13T14:43:57.565049+0000","last_change":"2024-09-13T14:38:03.534303+0000","last_active":"2024-09-13T14:43:57.565049+0000","last_peered":"2024-09-13T14:43:57.565049+0000","last_clean":"2024-09-13T14:43:57.565049+0000","last_became_active":"2024-09-13T14:38:03.533171+0000","last_became_peered":"2024-09-13T14:38:03.533171+0000","last_unstale":"2024-09-13T14:43:57.565049+0000","last_undegraded":"2024-09-13T14:43:57.565049+0000","last_fullsized":"2024-09-13T14:43:57.565049+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:00:16.553526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":814,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-13T14:43:57.574457+0000","last_change":"2024-09-13T14:38:04.596192+0000","last_active":"2024-09-13T14:43:57.574457+0000","last_peered":"2024-09-13T14:43:57.574457+0000","last_clean":"2024-09-13T14:43:57.574457+0000","last_became_active":"2024-09-13T14:38:04.593299+0000","last_became_peered":"2024-09-13T14:38:04.593299+0000","last_unstale":"2024-09-13T14:43:57.574457+0000","last_undegraded":"2024-09-13T14:43:57.574457+0000","last_fullsized":"2024-09-13T14:43:57.574457+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:34.241410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":108,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.619064+0000","last_change":"2024-09-13T14:38:03.533120+0000","last_active":"2024-09-13T14:44:06.619064+0000","last_peered":"2024-09-13T14:44:06.619064+0000","last_clean":"2024-09-13T14:44:06.619064+0000","last_became_active":"2024-09-13T14:38:03.532829+0000","last_became_peered":"2024-09-13T14:38:03.532829+0000","last_unstale":"2024-09-13T14:44:06.619064+0000","last_undegraded":"2024-09-13T14:44:06.619064+0000","last_fullsized":"2024-09-13T14:44:06.619064+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:24.514661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"292'11808","reported_seq":21081,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:12.274858+0000","last_change":"2024-09-13T14:42:04.660877+0000","last_active":"2024-09-13T14:44:12.274858+0000","last_peered":"2024-09-13T14:44:12.274858+0000","last_clean":"2024-09-13T14:44:12.274858+0000","last_became_active":"2024-09-13T14:42:04.641680+0000","last_became_peered":"2024-09-13T14:42:04.641680+0000","last_unstale":"2024-09-13T14:44:12.274858+0000","last_undegraded":"2024-09-13T14:44:12.274858+0000","last_fullsized":"2024-09-13T14:44:12.274858+0000","mapping_epoch":277,"log_start":"219'6118","ondisk_log_start":"219'6118","created":205,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5690,"log_dups_size":0,"ondisk_log_size":5690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:22:18.955428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169738240,"num_objects":2590,"num_object_clones":0,"num_object_copies":7770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11808,"num_write_kb":460736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4533,"num_bytes_recovered":297074688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":510,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.621479+0000","last_change":"2024-09-13T14:38:53.938227+0000","last_active":"2024-09-13T14:44:06.621479+0000","last_peered":"2024-09-13T14:44:06.621479+0000","last_clean":"2024-09-13T14:44:06.621479+0000","last_became_active":"2024-09-13T14:38:53.902867+0000","last_became_peered":"2024-09-13T14:38:53.902867+0000","last_unstale":"2024-09-13T14:44:06.621479+0000","last_undegraded":"2024-09-13T14:44:06.621479+0000","last_fullsized":"2024-09-13T14:44:06.621479+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:16.831756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":592,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-13T14:43:57.479836+0000","last_change":"2024-09-13T14:37:48.147677+0000","last_active":"2024-09-13T14:43:57.479836+0000","last_peered":"2024-09-13T14:43:57.479836+0000","last_clean":"2024-09-13T14:43:57.479836+0000","last_became_active":"2024-09-13T14:37:48.147001+0000","last_became_peered":"2024-09-13T14:37:48.147001+0000","last_unstale":"2024-09-13T14:43:57.479836+0000","last_undegraded":"2024-09-13T14:43:57.479836+0000","last_fullsized":"2024-09-13T14:43:57.479836+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:35:00.653576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":492,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.619819+0000","last_change":"2024-09-13T14:38:53.900107+0000","last_active":"2024-09-13T14:44:06.619819+0000","last_peered":"2024-09-13T14:44:06.619819+0000","last_clean":"2024-09-13T14:44:06.619819+0000","last_became_active":"2024-09-13T14:38:53.899556+0000","last_became_peered":"2024-09-13T14:38:53.899556+0000","last_unstale":"2024-09-13T14:44:06.619819+0000","last_undegraded":"2024-09-13T14:44:06.619819+0000","last_fullsized":"2024-09-13T14:44:06.619819+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:15:47.918163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"292'11378","reported_seq":20377,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:12.522930+0000","last_change":"2024-09-13T14:39:54.921239+0000","last_active":"2024-09-13T14:44:12.522930+0000","last_peered":"2024-09-13T14:44:12.522930+0000","last_clean":"2024-09-13T14:44:12.522930+0000","last_became_active":"2024-09-13T14:39:54.907894+0000","last_became_peered":"2024-09-13T14:39:54.907894+0000","last_unstale":"2024-09-13T14:44:12.522930+0000","last_undegraded":"2024-09-13T14:44:12.522930+0000","last_fullsized":"2024-09-13T14:44:12.522930+0000","mapping_epoch":263,"log_start":"219'5647","ondisk_log_start":"219'5647","created":205,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5731,"log_dups_size":0,"ondisk_log_size":5731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:52:04.619238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162136064,"num_objects":2474,"num_object_clones":0,"num_object_copies":7422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11378,"num_write_kb":443264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4395,"num_bytes_recovered":283770880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":123,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.619891+0000","last_change":"2024-09-13T14:38:53.901806+0000","last_active":"2024-09-13T14:44:06.619891+0000","last_peered":"2024-09-13T14:44:06.619891+0000","last_clean":"2024-09-13T14:44:06.619891+0000","last_became_active":"2024-09-13T14:38:53.891055+0000","last_became_peered":"2024-09-13T14:38:53.891055+0000","last_unstale":"2024-09-13T14:44:06.619891+0000","last_undegraded":"2024-09-13T14:44:06.619891+0000","last_fullsized":"2024-09-13T14:44:06.619891+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:03:00.416198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":67,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-13T14:43:57.574855+0000","last_change":"2024-09-13T14:43:16.854299+0000","last_active":"2024-09-13T14:43:57.574855+0000","last_peered":"2024-09-13T14:43:57.574855+0000","last_clean":"2024-09-13T14:43:57.574855+0000","last_became_active":"2024-09-13T14:38:53.900626+0000","last_became_peered":"2024-09-13T14:38:53.900626+0000","last_unstale":"2024-09-13T14:43:57.574855+0000","last_undegraded":"2024-09-13T14:43:57.574855+0000","last_fullsized":"2024-09-13T14:43:57.574855+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:37:21.715778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":282,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-13T14:43:57.479447+0000","last_change":"2024-09-13T14:37:49.172218+0000","last_active":"2024-09-13T14:43:57.479447+0000","last_peered":"2024-09-13T14:43:57.479447+0000","last_clean":"2024-09-13T14:43:57.479447+0000","last_became_active":"2024-09-13T14:37:49.171718+0000","last_became_peered":"2024-09-13T14:37:49.171718+0000","last_unstale":"2024-09-13T14:43:57.479447+0000","last_undegraded":"2024-09-13T14:43:57.479447+0000","last_fullsized":"2024-09-13T14:43:57.479447+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:46:54.626521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":305,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.616129+0000","last_change":"2024-09-13T14:37:49.230094+0000","last_active":"2024-09-13T14:44:06.616129+0000","last_peered":"2024-09-13T14:44:06.616129+0000","last_clean":"2024-09-13T14:44:06.616129+0000","last_became_active":"2024-09-13T14:37:49.229393+0000","last_became_peered":"2024-09-13T14:37:49.229393+0000","last_unstale":"2024-09-13T14:44:06.616129+0000","last_undegraded":"2024-09-13T14:44:06.616129+0000","last_fullsized":"2024-09-13T14:44:06.616129+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:02:28.690318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":284,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-13T14:43:57.481062+0000","last_change":"2024-09-13T14:37:48.128447+0000","last_active":"2024-09-13T14:43:57.481062+0000","last_peered":"2024-09-13T14:43:57.481062+0000","last_clean":"2024-09-13T14:43:57.481062+0000","last_became_active":"2024-09-13T14:37:48.128143+0000","last_became_peered":"2024-09-13T14:37:48.128143+0000","last_unstale":"2024-09-13T14:43:57.481062+0000","last_undegraded":"2024-09-13T14:43:57.481062+0000","last_fullsized":"2024-09-13T14:43:57.481062+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:22.204039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":108,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.618572+0000","last_change":"2024-09-13T14:38:03.531683+0000","last_active":"2024-09-13T14:44:06.618572+0000","last_peered":"2024-09-13T14:44:06.618572+0000","last_clean":"2024-09-13T14:44:06.618572+0000","last_became_active":"2024-09-13T14:38:03.531476+0000","last_became_peered":"2024-09-13T14:38:03.531476+0000","last_unstale":"2024-09-13T14:44:06.618572+0000","last_undegraded":"2024-09-13T14:44:06.618572+0000","last_fullsized":"2024-09-13T14:44:06.618572+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:41.557569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":57,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-13T14:43:57.574319+0000","last_change":"2024-09-13T14:38:53.954281+0000","last_active":"2024-09-13T14:43:57.574319+0000","last_peered":"2024-09-13T14:43:57.574319+0000","last_clean":"2024-09-13T14:43:57.574319+0000","last_became_active":"2024-09-13T14:38:53.900912+0000","last_became_peered":"2024-09-13T14:38:53.900912+0000","last_unstale":"2024-09-13T14:43:57.574319+0000","last_undegraded":"2024-09-13T14:43:57.574319+0000","last_fullsized":"2024-09-13T14:43:57.574319+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:47.876275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":109,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-13T14:43:57.565950+0000","last_change":"2024-09-13T14:38:54.000782+0000","last_active":"2024-09-13T14:43:57.565950+0000","last_peered":"2024-09-13T14:43:57.565950+0000","last_clean":"2024-09-13T14:43:57.565950+0000","last_became_active":"2024-09-13T14:38:54.000322+0000","last_became_peered":"2024-09-13T14:38:54.000322+0000","last_unstale":"2024-09-13T14:43:57.565950+0000","last_undegraded":"2024-09-13T14:43:57.565950+0000","last_fullsized":"2024-09-13T14:43:57.565950+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:09.057597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":434,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.619984+0000","last_change":"2024-09-13T14:38:53.899535+0000","last_active":"2024-09-13T14:44:06.619984+0000","last_peered":"2024-09-13T14:44:06.619984+0000","last_clean":"2024-09-13T14:44:06.619984+0000","last_became_active":"2024-09-13T14:38:53.899183+0000","last_became_peered":"2024-09-13T14:38:53.899183+0000","last_unstale":"2024-09-13T14:44:06.619984+0000","last_undegraded":"2024-09-13T14:44:06.619984+0000","last_fullsized":"2024-09-13T14:44:06.619984+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:33.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":108,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.619903+0000","last_change":"2024-09-13T14:38:03.543426+0000","last_active":"2024-09-13T14:44:06.619903+0000","last_peered":"2024-09-13T14:44:06.619903+0000","last_clean":"2024-09-13T14:44:06.619903+0000","last_became_active":"2024-09-13T14:38:03.542661+0000","last_became_peered":"2024-09-13T14:38:03.542661+0000","last_unstale":"2024-09-13T14:44:06.619903+0000","last_undegraded":"2024-09-13T14:44:06.619903+0000","last_fullsized":"2024-09-13T14:44:06.619903+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:54.050127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":463,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.619757+0000","last_change":"2024-09-13T14:37:48.131321+0000","last_active":"2024-09-13T14:44:06.619757+0000","last_peered":"2024-09-13T14:44:06.619757+0000","last_clean":"2024-09-13T14:44:06.619757+0000","last_became_active":"2024-09-13T14:37:48.130998+0000","last_became_peered":"2024-09-13T14:37:48.130998+0000","last_unstale":"2024-09-13T14:44:06.619757+0000","last_undegraded":"2024-09-13T14:44:06.619757+0000","last_fullsized":"2024-09-13T14:44:06.619757+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:33:26.597825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":121,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-13T14:43:57.565977+0000","last_change":"2024-09-13T14:38:03.535454+0000","last_active":"2024-09-13T14:43:57.565977+0000","last_peered":"2024-09-13T14:43:57.565977+0000","last_clean":"2024-09-13T14:43:57.565977+0000","last_became_active":"2024-09-13T14:38:03.535131+0000","last_became_peered":"2024-09-13T14:38:03.535131+0000","last_unstale":"2024-09-13T14:43:57.565977+0000","last_undegraded":"2024-09-13T14:43:57.565977+0000","last_fullsized":"2024-09-13T14:43:57.565977+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:24.477601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":487,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.619850+0000","last_change":"2024-09-13T14:38:03.524939+0000","last_active":"2024-09-13T14:44:06.619850+0000","last_peered":"2024-09-13T14:44:06.619850+0000","last_clean":"2024-09-13T14:44:06.619850+0000","last_became_active":"2024-09-13T14:38:03.524406+0000","last_became_peered":"2024-09-13T14:38:03.524406+0000","last_unstale":"2024-09-13T14:44:06.619850+0000","last_undegraded":"2024-09-13T14:44:06.619850+0000","last_fullsized":"2024-09-13T14:44:06.619850+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:19:45.257424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":115,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.619792+0000","last_change":"2024-09-13T14:38:53.892948+0000","last_active":"2024-09-13T14:44:06.619792+0000","last_peered":"2024-09-13T14:44:06.619792+0000","last_clean":"2024-09-13T14:44:06.619792+0000","last_became_active":"2024-09-13T14:38:53.892665+0000","last_became_peered":"2024-09-13T14:38:53.892665+0000","last_unstale":"2024-09-13T14:44:06.619792+0000","last_undegraded":"2024-09-13T14:44:06.619792+0000","last_fullsized":"2024-09-13T14:44:06.619792+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:08.292359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":436,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.620146+0000","last_change":"2024-09-13T14:37:48.139348+0000","last_active":"2024-09-13T14:44:06.620146+0000","last_peered":"2024-09-13T14:44:06.620146+0000","last_clean":"2024-09-13T14:44:06.620146+0000","last_became_active":"2024-09-13T14:37:48.139115+0000","last_became_peered":"2024-09-13T14:37:48.139115+0000","last_unstale":"2024-09-13T14:44:06.620146+0000","last_undegraded":"2024-09-13T14:44:06.620146+0000","last_fullsized":"2024-09-13T14:44:06.620146+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:32:20.413346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":476,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.620489+0000","last_change":"2024-09-13T14:37:48.172088+0000","last_active":"2024-09-13T14:44:06.620489+0000","last_peered":"2024-09-13T14:44:06.620489+0000","last_clean":"2024-09-13T14:44:06.620489+0000","last_became_active":"2024-09-13T14:37:48.171779+0000","last_became_peered":"2024-09-13T14:37:48.171779+0000","last_unstale":"2024-09-13T14:44:06.620489+0000","last_undegraded":"2024-09-13T14:44:06.620489+0000","last_fullsized":"2024-09-13T14:44:06.620489+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:02.852712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":349,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-13T14:43:57.871144+0000","last_change":"2024-09-13T14:38:53.890236+0000","last_active":"2024-09-13T14:43:57.871144+0000","last_peered":"2024-09-13T14:43:57.871144+0000","last_clean":"2024-09-13T14:43:57.871144+0000","last_became_active":"2024-09-13T14:38:53.889582+0000","last_became_peered":"2024-09-13T14:38:53.889582+0000","last_unstale":"2024-09-13T14:43:57.871144+0000","last_undegraded":"2024-09-13T14:43:57.871144+0000","last_fullsized":"2024-09-13T14:43:57.871144+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:14:29.334774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":462,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.620503+0000","last_change":"2024-09-13T14:38:03.534097+0000","last_active":"2024-09-13T14:44:06.620503+0000","last_peered":"2024-09-13T14:44:06.620503+0000","last_clean":"2024-09-13T14:44:06.620503+0000","last_became_active":"2024-09-13T14:38:03.533794+0000","last_became_peered":"2024-09-13T14:38:03.533794+0000","last_unstale":"2024-09-13T14:44:06.620503+0000","last_undegraded":"2024-09-13T14:44:06.620503+0000","last_fullsized":"2024-09-13T14:44:06.620503+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:23.272098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":108,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.618893+0000","last_change":"2024-09-13T14:38:03.527021+0000","last_active":"2024-09-13T14:44:06.618893+0000","last_peered":"2024-09-13T14:44:06.618893+0000","last_clean":"2024-09-13T14:44:06.618893+0000","last_became_active":"2024-09-13T14:38:03.526698+0000","last_became_peered":"2024-09-13T14:38:03.526698+0000","last_unstale":"2024-09-13T14:44:06.618893+0000","last_undegraded":"2024-09-13T14:44:06.618893+0000","last_fullsized":"2024-09-13T14:44:06.618893+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:01:35.734086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":293,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-13T14:43:57.871158+0000","last_change":"2024-09-13T14:27:12.834931+0000","last_active":"2024-09-13T14:43:57.871158+0000","last_peered":"2024-09-13T14:43:57.871158+0000","last_clean":"2024-09-13T14:43:57.871158+0000","last_became_active":"2024-09-13T14:27:12.834572+0000","last_became_peered":"2024-09-13T14:27:12.834572+0000","last_unstale":"2024-09-13T14:43:57.871158+0000","last_undegraded":"2024-09-13T14:43:57.871158+0000","last_fullsized":"2024-09-13T14:43:57.871158+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:54:21.155689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":106,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-13T14:43:57.565019+0000","last_change":"2024-09-13T14:37:49.176796+0000","last_active":"2024-09-13T14:43:57.565019+0000","last_peered":"2024-09-13T14:43:57.565019+0000","last_clean":"2024-09-13T14:43:57.565019+0000","last_became_active":"2024-09-13T14:37:49.176246+0000","last_became_peered":"2024-09-13T14:37:49.176246+0000","last_unstale":"2024-09-13T14:43:57.565019+0000","last_undegraded":"2024-09-13T14:43:57.565019+0000","last_fullsized":"2024-09-13T14:43:57.565019+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:29:24.936718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":304,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-13T14:43:57.871122+0000","last_change":"2024-09-13T14:38:03.526333+0000","last_active":"2024-09-13T14:43:57.871122+0000","last_peered":"2024-09-13T14:43:57.871122+0000","last_clean":"2024-09-13T14:43:57.871122+0000","last_became_active":"2024-09-13T14:38:03.526001+0000","last_became_peered":"2024-09-13T14:38:03.526001+0000","last_unstale":"2024-09-13T14:43:57.871122+0000","last_undegraded":"2024-09-13T14:43:57.871122+0000","last_fullsized":"2024-09-13T14:43:57.871122+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:25:25.181271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":106,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-13T14:43:57.564995+0000","last_change":"2024-09-13T14:37:49.149582+0000","last_active":"2024-09-13T14:43:57.564995+0000","last_peered":"2024-09-13T14:43:57.564995+0000","last_clean":"2024-09-13T14:43:57.564995+0000","last_became_active":"2024-09-13T14:37:49.148525+0000","last_became_peered":"2024-09-13T14:37:49.148525+0000","last_unstale":"2024-09-13T14:43:57.564995+0000","last_undegraded":"2024-09-13T14:43:57.564995+0000","last_fullsized":"2024-09-13T14:43:57.564995+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:48.655665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":508,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.620240+0000","last_change":"2024-09-13T14:38:03.535255+0000","last_active":"2024-09-13T14:44:06.620240+0000","last_peered":"2024-09-13T14:44:06.620240+0000","last_clean":"2024-09-13T14:44:06.620240+0000","last_became_active":"2024-09-13T14:38:03.533917+0000","last_became_peered":"2024-09-13T14:38:03.533917+0000","last_unstale":"2024-09-13T14:44:06.620240+0000","last_undegraded":"2024-09-13T14:44:06.620240+0000","last_fullsized":"2024-09-13T14:44:06.620240+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:33.559788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":472,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.621104+0000","last_change":"2024-09-13T14:37:48.146784+0000","last_active":"2024-09-13T14:44:06.621104+0000","last_peered":"2024-09-13T14:44:06.621104+0000","last_clean":"2024-09-13T14:44:06.621104+0000","last_became_active":"2024-09-13T14:37:48.146491+0000","last_became_peered":"2024-09-13T14:37:48.146491+0000","last_unstale":"2024-09-13T14:44:06.621104+0000","last_undegraded":"2024-09-13T14:44:06.621104+0000","last_fullsized":"2024-09-13T14:44:06.621104+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:30:54.772513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":284,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-13T14:43:57.479728+0000","last_change":"2024-09-13T14:37:48.127607+0000","last_active":"2024-09-13T14:43:57.479728+0000","last_peered":"2024-09-13T14:43:57.479728+0000","last_clean":"2024-09-13T14:43:57.479728+0000","last_became_active":"2024-09-13T14:37:48.127316+0000","last_became_peered":"2024-09-13T14:37:48.127316+0000","last_unstale":"2024-09-13T14:43:57.479728+0000","last_undegraded":"2024-09-13T14:43:57.479728+0000","last_fullsized":"2024-09-13T14:43:57.479728+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:53:37.678861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":114,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.619802+0000","last_change":"2024-09-13T14:38:03.563200+0000","last_active":"2024-09-13T14:44:06.619802+0000","last_peered":"2024-09-13T14:44:06.619802+0000","last_clean":"2024-09-13T14:44:06.619802+0000","last_became_active":"2024-09-13T14:38:03.534258+0000","last_became_peered":"2024-09-13T14:38:03.534258+0000","last_unstale":"2024-09-13T14:44:06.619802+0000","last_undegraded":"2024-09-13T14:44:06.619802+0000","last_fullsized":"2024-09-13T14:44:06.619802+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:49:07.814419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":2612330520,"num_objects":39862,"num_object_clones":0,"num_object_copies":119586,"num_objects_missing_on_primary":82,"num_objects_missing":82,"num_objects_degraded":5065,"num_objects_misplaced":4533,"num_objects_unfound":0,"num_objects_dirty":39862,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":185764,"num_write_kb":7219969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84494,"num_bytes_recovered":5410324504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10133647360,"data_stored":10133635144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94987,"ondisk_log_size":94987,"up":48,"acting":47,"num_store_stats":6},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1538,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":17390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25132,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16793600,"data_stored":16777292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":23,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2969600,"data_stored":2951840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234404,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1960024,"kb_used_data":1495760,"kb_used_omap":1,"kb_used_meta":464254,"kb_avail":91785128,"statfs":{"total":95995035648,"available":93987971072,"internally_reserved":0,"allocated":1531658240,"data_stored":1530097993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":475396555},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":239,"seq":1026497183822,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1592156,"kb_used_data":1305044,"kb_used_omap":8,"kb_used_meta":287095,"kb_avail":92152996,"statfs":{"total":95995035648,"available":94364667904,"internally_reserved":0,"allocated":1336365056,"data_stored":1334801284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8453,"internal_metadata":293986043},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757669,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1890760,"kb_used_data":1219776,"kb_used_omap":1,"kb_used_meta":670974,"kb_avail":91854392,"statfs":{"total":95995035648,"available":94058897408,"internally_reserved":0,"allocated":1249050624,"data_stored":1247501938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":687077830},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019313,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2031248,"kb_used_data":1418504,"kb_used_omap":1,"kb_used_meta":612734,"kb_avail":91713904,"statfs":{"total":95995035648,"available":93915037696,"internally_reserved":0,"allocated":1452548096,"data_stored":1450995485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":627440070},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435496,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2492868,"kb_used_data":1919740,"kb_used_omap":5,"kb_used_meta":573114,"kb_avail":91252284,"statfs":{"total":95995035648,"available":93442338816,"internally_reserved":0,"allocated":1965813760,"data_stored":1964268934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":586868862},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":446676599024,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":425201762551,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926074,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3296212,"kb_used_data":2606476,"kb_used_omap":7,"kb_used_meta":689720,"kb_avail":90448940,"statfs":{"total":95995035648,"available":92619714560,"internally_reserved":0,"allocated":2669031424,"data_stored":2667471381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":706273576},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6860,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2653556736,"data_stored":2653552664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1952714752,"data_stored":1952710680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1445789696,"data_stored":1445789696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1240727552,"data_stored":1240727552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1327763456,"data_stored":1327759384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1513095168,"data_stored":1513095168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:44:18.947 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:18 smithi195 ceph-mon[104262]: pgmap v1012: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.3 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 343 op/s; 4955/114729 objects degraded (4.319%); 4434/114729 objects misplaced (3.865%); 2.7 MiB/s, 42 objects/s recovering 2024-09-13T14:44:18.959 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:18.969 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T14:44:18.982 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:18.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:18.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:19.084 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:19.110 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:19.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:19.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:19.179 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:44:19.179 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:44:19.205 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:44:19.205 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:44:19.207 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:44:19.212 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:19.231 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:44:19.231 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:44:19.242 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:19.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:19.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:19.257 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:44:19.257 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:44:19.277 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:44:19.278 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:44:19.296 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:44:19.296 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:44:19.314 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:44:19.314 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:44:19.331 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:44:19.331 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:44:19.344 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:19.369 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:19.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:19.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:19.470 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:19.493 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:19.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:19.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:19.595 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:19.621 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:19.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:19.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:19.723 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:19.747 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:19.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:19.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:19.848 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:19.872 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:19.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:19.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:19.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:19 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 4955/114729 objects degraded (4.319%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:19.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:19 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/4144406739' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:44:19.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:19 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 4955/114729 objects degraded (4.319%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:19.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:19 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/4144406739' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:44:19.974 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:19.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:19 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 4955/114729 objects degraded (4.319%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:19.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:19 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/4144406739' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:44:20.000 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:20.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:20.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:20.102 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:20.124 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:20.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:20.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:20.225 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:20.246 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:20.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:20.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:20.348 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:20.372 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:20.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:20.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:20.473 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:20.498 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:20.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:20.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:20.599 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:20.623 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:20.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:20.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:20.725 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:20.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:20 smithi195 ceph-mon[104262]: pgmap v1013: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.3 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 346 op/s; 4955/114378 objects degraded (4.332%); 4434/114378 objects misplaced (3.877%); 2.7 MiB/s, 42 objects/s recovering 2024-09-13T14:44:20.748 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:20.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:20.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:20.850 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:20.873 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:20.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:20.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:20.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:20 smithi102 ceph-mon[110183]: pgmap v1013: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.3 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 346 op/s; 4955/114378 objects degraded (4.332%); 4434/114378 objects misplaced (3.877%); 2.7 MiB/s, 42 objects/s recovering 2024-09-13T14:44:20.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:20 smithi102 ceph-mon[115269]: pgmap v1013: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.3 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 346 op/s; 4955/114378 objects degraded (4.332%); 4434/114378 objects misplaced (3.877%); 2.7 MiB/s, 42 objects/s recovering 2024-09-13T14:44:20.975 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:20.986 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:44:20.986 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:44:20.996 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:20.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:20.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:21.098 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:21.124 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:21.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:21.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:21.226 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:21.252 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:21.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:21.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:21.354 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:21.377 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:21.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:21.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:21.479 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:21.502 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:21.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:21.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:21.603 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:21.621 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T14:44:21.625 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:21.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:21.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:21.727 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:21.749 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:21.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:21.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:21.850 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:21.855 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:21.872 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:21.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:21.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:21.946 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:44:21.973 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:21.995 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:44:21.996 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:21.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:21.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:22.097 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:22.119 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:22.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:22.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:22.221 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:22.242 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:22.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:22.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:22.344 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:22.366 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:22.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:22.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:22.468 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:22.490 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:22.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:22.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:22.591 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:22.618 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:22.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:22.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:22.632 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:22.642 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:22 smithi102 ceph-mon[110183]: pgmap v1014: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.3 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 346 op/s; 4955/114378 objects degraded (4.332%); 4434/114378 objects misplaced (3.877%); 2.7 MiB/s, 42 objects/s recovering 2024-09-13T14:44:22.720 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:22.745 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:22.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:22.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:22.757 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:44:22.847 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:22.872 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:22.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:22.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:22.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:22 smithi102 ceph-mon[115269]: pgmap v1014: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.3 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 346 op/s; 4955/114378 objects degraded (4.332%); 4434/114378 objects misplaced (3.877%); 2.7 MiB/s, 42 objects/s recovering 2024-09-13T14:44:22.974 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:22.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:22 smithi195 ceph-mon[104262]: pgmap v1014: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.3 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 346 op/s; 4955/114378 objects degraded (4.332%); 4434/114378 objects misplaced (3.877%); 2.7 MiB/s, 42 objects/s recovering 2024-09-13T14:44:22.998 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:22.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:22.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:23.100 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:23.121 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:23.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:23.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:23.223 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:23.251 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:23.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:23.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:23.353 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:23.377 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:23.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:23.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:23.459 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:23.479 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:23.502 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:23.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:23.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:23.604 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:23.626 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:23.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:23.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:23.728 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:23.755 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:23.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:23.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:23.857 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:23.880 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:23.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:23.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:23.981 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:24.005 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:24.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:24.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:24.107 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:24.132 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:24.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:24.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:24.234 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:24.267 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:24.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:24.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:24.369 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:24.396 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:24.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:24.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:24.497 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:24.530 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:24.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:24.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:24.631 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:24.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:24.658 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:24.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:24.759 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:24.784 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:24.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:24.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:24.886 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:24.908 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:24.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:24.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:24.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:24 smithi102 ceph-mon[110183]: pgmap v1015: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 343 op/s; 4847/109365 objects degraded (4.432%); 4328/109365 objects misplaced (3.957%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:44:24.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:24 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 4847/109365 objects degraded (4.432%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:24.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:24 smithi102 ceph-mon[115269]: pgmap v1015: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 343 op/s; 4847/109365 objects degraded (4.432%); 4328/109365 objects misplaced (3.957%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:44:24.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:24 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 4847/109365 objects degraded (4.432%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:24.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:24 smithi195 ceph-mon[104262]: pgmap v1015: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 343 op/s; 4847/109365 objects degraded (4.432%); 4328/109365 objects misplaced (3.957%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:44:24.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:24 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 4847/109365 objects degraded (4.432%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:25.010 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:25.032 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:25.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:25.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:25.134 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:25.156 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:25.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:25.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:25.258 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:25.278 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:25.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:25.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:25.380 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:25.402 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:25.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:25.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:25.504 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:25.527 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:25.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:25.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:25.629 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:25.653 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:25.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:25.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:25.754 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:25.779 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:25.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:25.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:25.878 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:44:25.878 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:44:25.880 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:25.905 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:25.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:25.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:26.007 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:26.029 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:26.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:26.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:26.130 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:26.170 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:26.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:26.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:26.272 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:26.296 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:26.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:26.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:26.398 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:26.421 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:26.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:26.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:26.522 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:26.545 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:26.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:26.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:26.643 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:44:26.643 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:44:26.643 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:26 smithi102 ceph-mon[110183]: pgmap v1016: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 343 op/s; 4847/109365 objects degraded (4.432%); 4328/109365 objects misplaced (3.957%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:44:26.644 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:44:26.644 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:44:26.644 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:26 smithi102 ceph-mon[115269]: pgmap v1016: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 343 op/s; 4847/109365 objects degraded (4.432%); 4328/109365 objects misplaced (3.957%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:44:26.644 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:44:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:44:26] "GET /metrics HTTP/1.1" 200 36547 "" "Prometheus/2.51.0" 2024-09-13T14:44:26.647 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:26.669 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:26.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:26.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:26.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:44:26.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:44:26.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:26 smithi195 ceph-mon[104262]: pgmap v1016: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 343 op/s; 4847/109365 objects degraded (4.432%); 4328/109365 objects misplaced (3.957%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:44:26.771 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:26.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:26.795 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:26.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:26.896 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:26.917 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:26.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:26.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:27.019 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:27.044 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:27.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:27.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:27.146 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:27.170 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:27.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:27.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:27.271 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:27.297 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:27.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:27.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:27.399 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:27.427 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:27.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:27.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:27.529 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:27.557 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:27.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:27.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:27.647 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T14:44:27.658 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:27.718 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:27.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:27.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:27.820 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:27.845 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:27.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:27.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:27.947 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:27.973 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:27.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:27.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:28.075 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:28.097 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:28.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:28.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:28.198 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:28.220 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:28.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:28.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:28.322 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:28.344 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:28.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:28.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:28.373 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:28.445 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:28.471 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:28.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:28.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:28.573 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:28.596 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:28.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:28.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:28.697 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:28.721 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:28.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:28.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:28.823 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:28.851 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:28.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:28.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:28.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:28 smithi102 ceph-mon[110183]: pgmap v1017: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.1 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 439 op/s; 4823/105300 objects degraded (4.580%); 4124/105300 objects misplaced (3.916%); 3.2 MiB/s, 51 objects/s recovering 2024-09-13T14:44:28.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:28 smithi102 ceph-mon[115269]: pgmap v1017: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.1 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 439 op/s; 4823/105300 objects degraded (4.580%); 4124/105300 objects misplaced (3.916%); 3.2 MiB/s, 51 objects/s recovering 2024-09-13T14:44:28.925 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:28.953 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:28.975 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:28.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:28.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:28.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:28 smithi195 ceph-mon[104262]: pgmap v1017: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.1 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 439 op/s; 4823/105300 objects degraded (4.580%); 4124/105300 objects misplaced (3.916%); 3.2 MiB/s, 51 objects/s recovering 2024-09-13T14:44:29.078 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:29.101 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:29.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:29.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:29.116 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:44:29.117 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:44:29.203 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:29.225 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:29.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:29.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:29.326 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:29.349 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:29.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:29.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:29.451 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:29.474 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:29.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:29.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:29.576 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:29.598 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:29.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:29.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:29.602 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is set 2024-09-13T14:44:29.700 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:29.723 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:29.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:29.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:29.790 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:29 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/446845916' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:44:29.790 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:29 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 4823/105300 objects degraded (4.580%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:29.790 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:29 smithi102 ceph-mon[110183]: from='client.36072 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:44:29.791 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:29 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/446845916' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:44:29.791 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:29 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 4823/105300 objects degraded (4.580%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:29.791 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:29 smithi102 ceph-mon[115269]: from='client.36072 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:44:29.824 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:29.849 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:29.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:29.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:29.898 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:29 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/446845916' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:44:29.898 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:29 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 4823/105300 objects degraded (4.580%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:29.898 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:29 smithi195 ceph-mon[104262]: from='client.36072 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:44:29.951 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:29.975 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:29.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:29.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:30.077 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:30.101 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:30.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:30.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:30.203 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:30.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:30.231 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:30.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:30.333 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:30.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:30.355 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:30.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:30.456 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:30.480 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:30.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:30.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:30.582 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:30.607 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:30.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:30.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:30.682 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:30 smithi195 ceph-mon[104262]: pgmap v1018: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.1 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 318 op/s; 4823/103263 objects degraded (4.671%); 4124/103263 objects misplaced (3.994%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:44:30.682 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:30 smithi195 ceph-mon[104262]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:44:30.682 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:30 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/446845916' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:44:30.682 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:30 smithi195 ceph-mon[104262]: osdmap e294: 8 total, 8 up, 6 in 2024-09-13T14:44:30.709 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:30.732 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:30.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:30.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:30.834 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:30.856 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:30.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:30.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:30.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:30 smithi102 ceph-mon[110183]: pgmap v1018: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.1 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 318 op/s; 4823/103263 objects degraded (4.671%); 4124/103263 objects misplaced (3.994%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:44:30.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:30 smithi102 ceph-mon[110183]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:44:30.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:30 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/446845916' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:44:30.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:30 smithi102 ceph-mon[110183]: osdmap e294: 8 total, 8 up, 6 in 2024-09-13T14:44:30.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:30 smithi102 ceph-mon[115269]: pgmap v1018: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.1 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 318 op/s; 4823/103263 objects degraded (4.671%); 4124/103263 objects misplaced (3.994%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:44:30.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:30 smithi102 ceph-mon[115269]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:44:30.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:30 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/446845916' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:44:30.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:30 smithi102 ceph-mon[115269]: osdmap e294: 8 total, 8 up, 6 in 2024-09-13T14:44:30.958 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:30.982 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:30.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:30.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:31.083 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:31.108 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:31.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:31.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:31.210 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:31.232 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:31.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:31.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:31.334 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:31.356 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:31.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:31.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:31.458 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:31.481 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:31.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:31.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:31.583 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:31.606 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:31.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:31.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:31.708 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:31.729 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:31.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:31.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:31.831 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:31.853 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:31.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:31.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:31.955 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:31.976 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:31.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:31.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:32.078 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:32.099 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:32.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:32.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:32.201 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:32.211 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":1017,"stamp":"2024-09-13T14:44:27.542933+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2303203325,"num_objects":35100,"num_object_clones":0,"num_object_copies":105300,"num_objects_missing_on_primary":80,"num_objects_missing":80,"num_objects_degraded":4823,"num_objects_misplaced":4124,"num_objects_unfound":0,"num_objects_dirty":35100,"num_whiteouts":0,"num_read":6377,"num_read_kb":13834,"num_write":191677,"num_write_kb":7235010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85171,"num_bytes_recovered":5453057165,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96159,"ondisk_log_size":96159,"up":267,"acting":265,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":299,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":12411756,"kb_used_data":9080060,"kb_used_omap":26,"kb_used_meta":3331621,"kb_avail":550059156,"statfs":{"total":575970213888,"available":563260575744,"internally_reserved":0,"allocated":9297981440,"data_stored":9288627111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27144,"internal_metadata":3411580408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-313917440,"num_objects":-4790,"num_object_clones":0,"num_object_copies":-14370,"num_objects_missing_on_primary":-2,"num_objects_missing":-2,"num_objects_degraded":-251,"num_objects_misplaced":-409,"num_objects_unfound":0,"num_objects_dirty":-4790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4790,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":563,"num_bytes_recovered":36765696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.911100"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":480,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:16.636937+0000","last_change":"2024-09-13T14:36:13.859927+0000","last_active":"2024-09-13T14:44:16.636937+0000","last_peered":"2024-09-13T14:44:16.636937+0000","last_clean":"2024-09-13T14:44:16.636937+0000","last_became_active":"2024-09-13T14:36:13.859505+0000","last_became_peered":"2024-09-13T14:36:13.859505+0000","last_unstale":"2024-09-13T14:44:16.636937+0000","last_undegraded":"2024-09-13T14:44:16.636937+0000","last_fullsized":"2024-09-13T14:44:16.636937+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:12:20.989627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":296,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:12.442297+0000","last_change":"2024-09-13T14:37:48.130968+0000","last_active":"2024-09-13T14:44:12.442297+0000","last_peered":"2024-09-13T14:44:12.442297+0000","last_clean":"2024-09-13T14:44:12.442297+0000","last_became_active":"2024-09-13T14:37:48.130603+0000","last_became_peered":"2024-09-13T14:37:48.130603+0000","last_unstale":"2024-09-13T14:44:12.442297+0000","last_undegraded":"2024-09-13T14:44:12.442297+0000","last_fullsized":"2024-09-13T14:44:12.442297+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:42.626386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":92,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.623871+0000","last_change":"2024-09-13T14:36:13.862181+0000","last_active":"2024-09-13T14:44:06.623871+0000","last_peered":"2024-09-13T14:44:06.623871+0000","last_clean":"2024-09-13T14:44:06.623871+0000","last_became_active":"2024-09-13T14:36:13.861873+0000","last_became_peered":"2024-09-13T14:36:13.861873+0000","last_unstale":"2024-09-13T14:44:06.623871+0000","last_undegraded":"2024-09-13T14:44:06.623871+0000","last_fullsized":"2024-09-13T14:44:06.623871+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:14.085700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":117,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:16.637183+0000","last_change":"2024-09-13T14:38:53.896672+0000","last_active":"2024-09-13T14:44:16.637183+0000","last_peered":"2024-09-13T14:44:16.637183+0000","last_clean":"2024-09-13T14:44:16.637183+0000","last_became_active":"2024-09-13T14:38:53.896346+0000","last_became_peered":"2024-09-13T14:38:53.896346+0000","last_unstale":"2024-09-13T14:44:16.637183+0000","last_undegraded":"2024-09-13T14:44:16.637183+0000","last_fullsized":"2024-09-13T14:44:16.637183+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:57.789686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":117,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:16.637498+0000","last_change":"2024-09-13T14:38:53.893021+0000","last_active":"2024-09-13T14:44:16.637498+0000","last_peered":"2024-09-13T14:44:16.637498+0000","last_clean":"2024-09-13T14:44:16.637498+0000","last_became_active":"2024-09-13T14:38:53.892668+0000","last_became_peered":"2024-09-13T14:38:53.892668+0000","last_unstale":"2024-09-13T14:44:16.637498+0000","last_undegraded":"2024-09-13T14:44:16.637498+0000","last_fullsized":"2024-09-13T14:44:16.637498+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:57.180738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":264,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:07.441718+0000","last_change":"2024-09-13T14:37:48.134642+0000","last_active":"2024-09-13T14:44:07.441718+0000","last_peered":"2024-09-13T14:44:07.441718+0000","last_clean":"2024-09-13T14:44:07.441718+0000","last_became_active":"2024-09-13T14:37:48.134402+0000","last_became_peered":"2024-09-13T14:37:48.134402+0000","last_unstale":"2024-09-13T14:44:07.441718+0000","last_undegraded":"2024-09-13T14:44:07.441718+0000","last_fullsized":"2024-09-13T14:44:07.441718+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:51.313663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":499,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:16.636088+0000","last_change":"2024-09-13T14:38:53.903022+0000","last_active":"2024-09-13T14:44:16.636088+0000","last_peered":"2024-09-13T14:44:16.636088+0000","last_clean":"2024-09-13T14:44:16.636088+0000","last_became_active":"2024-09-13T14:38:53.902448+0000","last_became_peered":"2024-09-13T14:38:53.902448+0000","last_unstale":"2024-09-13T14:44:16.636088+0000","last_undegraded":"2024-09-13T14:44:16.636088+0000","last_fullsized":"2024-09-13T14:44:16.636088+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:24.690401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":120,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:16.636791+0000","last_change":"2024-09-13T14:38:03.554500+0000","last_active":"2024-09-13T14:44:16.636791+0000","last_peered":"2024-09-13T14:44:16.636791+0000","last_clean":"2024-09-13T14:44:16.636791+0000","last_became_active":"2024-09-13T14:38:03.532910+0000","last_became_peered":"2024-09-13T14:38:03.532910+0000","last_unstale":"2024-09-13T14:44:16.636791+0000","last_undegraded":"2024-09-13T14:44:16.636791+0000","last_fullsized":"2024-09-13T14:44:16.636791+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:27:50.843498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"293'11806","reported_seq":22711,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:27.186388+0000","last_change":"2024-09-13T14:41:53.220083+0000","last_active":"2024-09-13T14:44:27.186388+0000","last_peered":"2024-09-13T14:44:27.186388+0000","last_clean":"2024-09-13T14:44:27.186388+0000","last_became_active":"2024-09-13T14:41:53.200084+0000","last_became_peered":"2024-09-13T14:41:53.200084+0000","last_unstale":"2024-09-13T14:44:27.186388+0000","last_undegraded":"2024-09-13T14:44:27.186388+0000","last_fullsized":"2024-09-13T14:44:27.186388+0000","mapping_epoch":275,"log_start":"219'6100","ondisk_log_start":"219'6100","created":205,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5706,"log_dups_size":0,"ondisk_log_size":5706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:08.897524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135921664,"num_objects":2074,"num_object_clones":0,"num_object_copies":6222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11806,"num_write_kb":444160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9152,"num_bytes_recovered":594608128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":56,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.635212+0000","last_change":"2024-09-13T14:38:53.940238+0000","last_active":"2024-09-13T14:44:06.635212+0000","last_peered":"2024-09-13T14:44:06.635212+0000","last_clean":"2024-09-13T14:44:06.635212+0000","last_became_active":"2024-09-13T14:38:53.892825+0000","last_became_peered":"2024-09-13T14:38:53.892825+0000","last_unstale":"2024-09-13T14:44:06.635212+0000","last_undegraded":"2024-09-13T14:44:06.635212+0000","last_fullsized":"2024-09-13T14:44:06.635212+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:46.666581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":112,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.623023+0000","last_change":"2024-09-13T14:38:53.898907+0000","last_active":"2024-09-13T14:44:06.623023+0000","last_peered":"2024-09-13T14:44:06.623023+0000","last_clean":"2024-09-13T14:44:06.623023+0000","last_became_active":"2024-09-13T14:38:53.898572+0000","last_became_peered":"2024-09-13T14:38:53.898572+0000","last_unstale":"2024-09-13T14:44:06.623023+0000","last_undegraded":"2024-09-13T14:44:06.623023+0000","last_fullsized":"2024-09-13T14:44:06.623023+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:29.222150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"293'11958","reported_seq":26155,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:22.508836+0000","last_change":"2024-09-13T14:38:57.930032+0000","last_active":"2024-09-13T14:44:22.508836+0000","last_peered":"2024-09-13T14:44:22.508836+0000","last_clean":"2024-09-13T14:44:22.508836+0000","last_became_active":"2024-09-13T14:38:57.921458+0000","last_became_peered":"2024-09-13T14:38:57.921458+0000","last_unstale":"2024-09-13T14:44:22.508836+0000","last_undegraded":"2024-09-13T14:44:22.508836+0000","last_fullsized":"2024-09-13T14:44:22.508836+0000","mapping_epoch":256,"log_start":"219'6239","ondisk_log_start":"219'6239","created":205,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5719,"log_dups_size":0,"ondisk_log_size":5719,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:27:12.294038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149684224,"num_objects":2284,"num_object_clones":0,"num_object_copies":6852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11958,"num_write_kb":455744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6990,"num_bytes_recovered":458096640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":56,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.635158+0000","last_change":"2024-09-13T14:38:53.904885+0000","last_active":"2024-09-13T14:44:06.635158+0000","last_peered":"2024-09-13T14:44:06.635158+0000","last_clean":"2024-09-13T14:44:06.635158+0000","last_became_active":"2024-09-13T14:38:53.891726+0000","last_became_peered":"2024-09-13T14:38:53.891726+0000","last_unstale":"2024-09-13T14:44:06.635158+0000","last_undegraded":"2024-09-13T14:44:06.635158+0000","last_fullsized":"2024-09-13T14:44:06.635158+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:57.079205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":302,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.616321+0000","last_change":"2024-09-13T14:37:48.160504+0000","last_active":"2024-09-13T14:44:06.616321+0000","last_peered":"2024-09-13T14:44:06.616321+0000","last_clean":"2024-09-13T14:44:06.616321+0000","last_became_active":"2024-09-13T14:37:48.160181+0000","last_became_peered":"2024-09-13T14:37:48.160181+0000","last_unstale":"2024-09-13T14:44:06.616321+0000","last_undegraded":"2024-09-13T14:44:06.616321+0000","last_fullsized":"2024-09-13T14:44:06.616321+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:37.087008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"293'11984","reported_seq":13802,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:26.865192+0000","last_change":"2024-09-13T14:38:04.531914+0000","last_active":"2024-09-13T14:44:26.865192+0000","last_peered":"2024-09-13T14:44:26.865192+0000","last_clean":"2024-09-13T14:44:26.865192+0000","last_became_active":"2024-09-13T14:38:04.523448+0000","last_became_peered":"2024-09-13T14:38:04.523448+0000","last_unstale":"2024-09-13T14:44:26.865192+0000","last_undegraded":"2024-09-13T14:44:26.865192+0000","last_fullsized":"2024-09-13T14:44:26.865192+0000","mapping_epoch":245,"log_start":"219'6100","ondisk_log_start":"219'6100","created":205,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5884,"log_dups_size":0,"ondisk_log_size":5884,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:06.536896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143523840,"num_objects":2190,"num_object_clones":0,"num_object_copies":6570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11984,"num_write_kb":453568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":811,"num_bytes_recovered":53149696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":293,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.635144+0000","last_change":"2024-09-13T14:37:49.611774+0000","last_active":"2024-09-13T14:44:06.635144+0000","last_peered":"2024-09-13T14:44:06.635144+0000","last_clean":"2024-09-13T14:44:06.635144+0000","last_became_active":"2024-09-13T14:37:49.611088+0000","last_became_peered":"2024-09-13T14:37:49.611088+0000","last_unstale":"2024-09-13T14:44:06.635144+0000","last_undegraded":"2024-09-13T14:44:06.635144+0000","last_fullsized":"2024-09-13T14:44:06.635144+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:08.933213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":52,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.634576+0000","last_change":"2024-09-13T14:38:53.899211+0000","last_active":"2024-09-13T14:44:06.634576+0000","last_peered":"2024-09-13T14:44:06.634576+0000","last_clean":"2024-09-13T14:44:06.634576+0000","last_became_active":"2024-09-13T14:38:53.898927+0000","last_became_peered":"2024-09-13T14:38:53.898927+0000","last_unstale":"2024-09-13T14:44:06.634576+0000","last_undegraded":"2024-09-13T14:44:06.634576+0000","last_fullsized":"2024-09-13T14:44:06.634576+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:00:28.857611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"293'11848","reported_seq":11949,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:27.401516+0000","last_change":"2024-09-13T14:35:24.004651+0000","last_active":"2024-09-13T14:44:27.401516+0000","last_peered":"2024-09-13T14:44:27.401516+0000","last_clean":"2024-09-13T14:44:27.401516+0000","last_became_active":"2024-09-13T14:35:24.004358+0000","last_became_peered":"2024-09-13T14:35:24.004358+0000","last_unstale":"2024-09-13T14:44:27.401516+0000","last_undegraded":"2024-09-13T14:44:27.401516+0000","last_fullsized":"2024-09-13T14:44:27.401516+0000","mapping_epoch":205,"log_start":"219'5584","ondisk_log_start":"219'5584","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6264,"log_dups_size":0,"ondisk_log_size":6264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:57:38.023669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139198464,"num_objects":2124,"num_object_clones":0,"num_object_copies":6372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11848,"num_write_kb":447104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":103,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.623495+0000","last_change":"2024-09-13T14:37:49.168003+0000","last_active":"2024-09-13T14:44:06.623495+0000","last_peered":"2024-09-13T14:44:06.623495+0000","last_clean":"2024-09-13T14:44:06.623495+0000","last_became_active":"2024-09-13T14:37:49.167638+0000","last_became_peered":"2024-09-13T14:37:49.167638+0000","last_unstale":"2024-09-13T14:44:06.623495+0000","last_undegraded":"2024-09-13T14:44:06.623495+0000","last_fullsized":"2024-09-13T14:44:06.623495+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:29.051981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":110,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:16.637444+0000","last_change":"2024-09-13T14:38:03.528381+0000","last_active":"2024-09-13T14:44:16.637444+0000","last_peered":"2024-09-13T14:44:16.637444+0000","last_clean":"2024-09-13T14:44:16.637444+0000","last_became_active":"2024-09-13T14:38:03.527282+0000","last_became_peered":"2024-09-13T14:38:03.527282+0000","last_unstale":"2024-09-13T14:44:16.637444+0000","last_undegraded":"2024-09-13T14:44:16.637444+0000","last_fullsized":"2024-09-13T14:44:16.637444+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:24.435067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.f","version":"293'11905","reported_seq":12218,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:26.872034+0000","last_change":"2024-09-13T14:38:16.371079+0000","last_active":"2024-09-13T14:44:26.872034+0000","last_peered":"2024-09-13T14:44:26.872034+0000","last_clean":"2024-09-13T14:44:26.872034+0000","last_became_active":"2024-09-13T14:38:12.859883+0000","last_became_peered":"2024-09-13T14:38:12.859883+0000","last_unstale":"2024-09-13T14:44:26.872034+0000","last_undegraded":"2024-09-13T14:44:26.872034+0000","last_fullsized":"2024-09-13T14:44:26.872034+0000","mapping_epoch":245,"log_start":"219'5963","ondisk_log_start":"219'5963","created":205,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5942,"log_dups_size":0,"ondisk_log_size":5942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:08.436532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142934016,"num_objects":2181,"num_object_clones":0,"num_object_copies":6543,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11905,"num_write_kb":450752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":3670016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":468,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:16.636695+0000","last_change":"2024-09-13T14:38:03.549595+0000","last_active":"2024-09-13T14:44:16.636695+0000","last_peered":"2024-09-13T14:44:16.636695+0000","last_clean":"2024-09-13T14:44:16.636695+0000","last_became_active":"2024-09-13T14:38:03.533331+0000","last_became_peered":"2024-09-13T14:38:03.533331+0000","last_unstale":"2024-09-13T14:44:16.636695+0000","last_undegraded":"2024-09-13T14:44:16.636695+0000","last_fullsized":"2024-09-13T14:44:16.636695+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:59:02.470343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":277,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.616102+0000","last_change":"2024-09-13T14:37:48.127600+0000","last_active":"2024-09-13T14:44:06.616102+0000","last_peered":"2024-09-13T14:44:06.616102+0000","last_clean":"2024-09-13T14:44:06.616102+0000","last_became_active":"2024-09-13T14:37:48.127349+0000","last_became_peered":"2024-09-13T14:37:48.127349+0000","last_unstale":"2024-09-13T14:44:06.616102+0000","last_undegraded":"2024-09-13T14:44:06.616102+0000","last_fullsized":"2024-09-13T14:44:06.616102+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:57.268374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"293'11900","reported_seq":21898,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:27.289089+0000","last_change":"2024-09-13T14:42:30.227259+0000","last_active":"2024-09-13T14:44:27.289089+0000","last_peered":"2024-09-13T14:44:27.289089+0000","last_clean":"2024-09-13T14:44:27.289089+0000","last_became_active":"2024-09-13T14:42:30.209102+0000","last_became_peered":"2024-09-13T14:42:30.209102+0000","last_unstale":"2024-09-13T14:44:27.289089+0000","last_undegraded":"2024-09-13T14:44:27.289089+0000","last_fullsized":"2024-09-13T14:44:27.289089+0000","mapping_epoch":281,"log_start":"220'6200","ondisk_log_start":"220'6200","created":205,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5700,"log_dups_size":0,"ondisk_log_size":5700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:56:16.193287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138280960,"num_objects":2110,"num_object_clones":0,"num_object_copies":6330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11900,"num_write_kb":448320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4926,"num_bytes_recovered":313851904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":210,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.616001+0000","last_change":"2024-09-13T14:37:48.127604+0000","last_active":"2024-09-13T14:44:06.616001+0000","last_peered":"2024-09-13T14:44:06.616001+0000","last_clean":"2024-09-13T14:44:06.616001+0000","last_became_active":"2024-09-13T14:37:48.127300+0000","last_became_peered":"2024-09-13T14:37:48.127300+0000","last_unstale":"2024-09-13T14:44:06.616001+0000","last_undegraded":"2024-09-13T14:44:06.616001+0000","last_fullsized":"2024-09-13T14:44:06.616001+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:51.999506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":330,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.616328+0000","last_change":"2024-09-13T14:38:03.529967+0000","last_active":"2024-09-13T14:44:06.616328+0000","last_peered":"2024-09-13T14:44:06.616328+0000","last_clean":"2024-09-13T14:44:06.616328+0000","last_became_active":"2024-09-13T14:38:03.529646+0000","last_became_peered":"2024-09-13T14:38:03.529646+0000","last_unstale":"2024-09-13T14:44:06.616328+0000","last_undegraded":"2024-09-13T14:44:06.616328+0000","last_fullsized":"2024-09-13T14:44:06.616328+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:52:05.193279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"293'11954","reported_seq":21142,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:26.839326+0000","last_change":"2024-09-13T14:40:34.583237+0000","last_active":"2024-09-13T14:44:26.839326+0000","last_peered":"2024-09-13T14:44:26.839326+0000","last_clean":"2024-09-13T14:44:26.839326+0000","last_became_active":"2024-09-13T14:40:34.563989+0000","last_became_peered":"2024-09-13T14:40:34.563989+0000","last_unstale":"2024-09-13T14:44:26.839326+0000","last_undegraded":"2024-09-13T14:44:26.839326+0000","last_fullsized":"2024-09-13T14:44:26.839326+0000","mapping_epoch":266,"log_start":"219'6000","ondisk_log_start":"219'6000","created":205,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5954,"log_dups_size":0,"ondisk_log_size":5954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:34:43.006051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143196184,"num_objects":2186,"num_object_clones":0,"num_object_copies":6558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2186,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11954,"num_write_kb":452417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4489,"num_bytes_recovered":289865752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":291,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.616522+0000","last_change":"2024-09-13T14:36:13.860367+0000","last_active":"2024-09-13T14:44:06.616522+0000","last_peered":"2024-09-13T14:44:06.616522+0000","last_clean":"2024-09-13T14:44:06.616522+0000","last_became_active":"2024-09-13T14:36:13.859982+0000","last_became_peered":"2024-09-13T14:36:13.859982+0000","last_unstale":"2024-09-13T14:44:06.616522+0000","last_undegraded":"2024-09-13T14:44:06.616522+0000","last_fullsized":"2024-09-13T14:44:06.616522+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:08:23.311350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":286,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.616220+0000","last_change":"2024-09-13T14:37:48.147360+0000","last_active":"2024-09-13T14:44:06.616220+0000","last_peered":"2024-09-13T14:44:06.616220+0000","last_clean":"2024-09-13T14:44:06.616220+0000","last_became_active":"2024-09-13T14:37:48.146885+0000","last_became_peered":"2024-09-13T14:37:48.146885+0000","last_unstale":"2024-09-13T14:44:06.616220+0000","last_undegraded":"2024-09-13T14:44:06.616220+0000","last_fullsized":"2024-09-13T14:44:06.616220+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:27.841810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"293'11812","reported_seq":21107,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:22.502333+0000","last_change":"2024-09-13T14:38:39.650332+0000","last_active":"2024-09-13T14:44:22.502333+0000","last_peered":"2024-09-13T14:44:22.502333+0000","last_clean":"2024-09-13T14:44:22.502333+0000","last_became_active":"2024-09-13T14:38:39.640835+0000","last_became_peered":"2024-09-13T14:38:39.640835+0000","last_unstale":"2024-09-13T14:44:22.502333+0000","last_undegraded":"2024-09-13T14:44:22.502333+0000","last_fullsized":"2024-09-13T14:44:22.502333+0000","mapping_epoch":251,"log_start":"219'6151","ondisk_log_start":"219'6151","created":205,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5661,"log_dups_size":0,"ondisk_log_size":5661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:48.176606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148242432,"num_objects":2262,"num_object_clones":0,"num_object_copies":6786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11812,"num_write_kb":450368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4544,"num_bytes_recovered":292945920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":487,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:16.636467+0000","last_change":"2024-09-13T14:38:53.900041+0000","last_active":"2024-09-13T14:44:16.636467+0000","last_peered":"2024-09-13T14:44:16.636467+0000","last_clean":"2024-09-13T14:44:16.636467+0000","last_became_active":"2024-09-13T14:38:53.899203+0000","last_became_peered":"2024-09-13T14:38:53.899203+0000","last_unstale":"2024-09-13T14:44:16.636467+0000","last_undegraded":"2024-09-13T14:44:16.636467+0000","last_fullsized":"2024-09-13T14:44:16.636467+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:28:07.883094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":419,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:16.636171+0000","last_change":"2024-09-13T14:26:15.823198+0000","last_active":"2024-09-13T14:44:16.636171+0000","last_peered":"2024-09-13T14:44:16.636171+0000","last_clean":"2024-09-13T14:44:16.636171+0000","last_became_active":"2024-09-13T14:26:15.822717+0000","last_became_peered":"2024-09-13T14:26:15.822717+0000","last_unstale":"2024-09-13T14:44:16.636171+0000","last_undegraded":"2024-09-13T14:44:16.636171+0000","last_fullsized":"2024-09-13T14:44:16.636171+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:22.471622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.6","version":"293'12031","reported_seq":21689,"reported_epoch":293,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-13T14:44:26.860884+0000","last_change":"2024-09-13T14:38:55.132683+0000","last_active":"2024-09-13T14:44:26.860884+0000","last_peered":"2024-09-13T14:44:26.860884+0000","last_clean":"2024-09-13T14:36:13.124368+0000","last_became_active":"2024-09-13T14:38:55.120883+0000","last_became_peered":"2024-09-13T14:38:55.120883+0000","last_unstale":"2024-09-13T14:44:26.860884+0000","last_undegraded":"2024-09-13T14:38:55.026610+0000","last_fullsized":"2024-09-13T14:44:26.860884+0000","mapping_epoch":254,"log_start":"219'6119","ondisk_log_start":"219'6119","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5912,"log_dups_size":0,"ondisk_log_size":5912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142016512,"num_objects":2167,"num_object_clones":0,"num_object_copies":6501,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":279,"num_objects_misplaced":1888,"num_objects_unfound":0,"num_objects_dirty":2167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12031,"num_write_kb":454336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4726,"num_bytes_recovered":298516480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,3,5],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3,5","objects":2167}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":427,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.634587+0000","last_change":"2024-09-13T14:41:53.196835+0000","last_active":"2024-09-13T14:44:06.634587+0000","last_peered":"2024-09-13T14:44:06.634587+0000","last_clean":"2024-09-13T14:44:06.634587+0000","last_became_active":"2024-09-13T14:41:53.196387+0000","last_became_peered":"2024-09-13T14:41:53.196387+0000","last_unstale":"2024-09-13T14:44:06.634587+0000","last_undegraded":"2024-09-13T14:44:06.634587+0000","last_fullsized":"2024-09-13T14:44:06.634587+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:07:41.927740+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:49.972303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":532,"reported_epoch":292,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:44:06.623445+0000","last_change":"2024-09-13T14:38:54.904329+0000","last_active":"2024-09-13T14:44:06.623445+0000","last_peered":"2024-09-13T14:44:06.623445+0000","last_clean":"2024-09-13T14:38:39.648017+0000","last_became_active":"2024-09-13T14:38:54.903798+0000","last_became_peered":"2024-09-13T14:38:54.903798+0000","last_unstale":"2024-09-13T14:44:06.623445+0000","last_undegraded":"2024-09-13T14:38:54.890505+0000","last_fullsized":"2024-09-13T14:38:54.889701+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:18.120931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,3],"avail_no_missing":["5","3","6"],"object_location_counts":[{"shards":"3,5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":421,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:16.636316+0000","last_change":"2024-09-13T14:37:49.160719+0000","last_active":"2024-09-13T14:44:16.636316+0000","last_peered":"2024-09-13T14:44:16.636316+0000","last_clean":"2024-09-13T14:44:16.636316+0000","last_became_active":"2024-09-13T14:37:49.160401+0000","last_became_peered":"2024-09-13T14:37:49.160401+0000","last_unstale":"2024-09-13T14:44:16.636316+0000","last_undegraded":"2024-09-13T14:44:16.636316+0000","last_fullsized":"2024-09-13T14:44:16.636316+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:21.737945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":138,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:16.633264+0000","last_change":"2024-09-13T14:37:48.172940+0000","last_active":"2024-09-13T14:44:16.633264+0000","last_peered":"2024-09-13T14:44:16.633264+0000","last_clean":"2024-09-13T14:44:16.633264+0000","last_became_active":"2024-09-13T14:37:48.172060+0000","last_became_peered":"2024-09-13T14:37:48.172060+0000","last_unstale":"2024-09-13T14:44:16.633264+0000","last_undegraded":"2024-09-13T14:44:16.633264+0000","last_fullsized":"2024-09-13T14:44:16.633264+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:07.480645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.5","version":"293'12010","reported_seq":28387,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:27.414162+0000","last_change":"2024-09-13T14:39:34.446699+0000","last_active":"2024-09-13T14:44:27.414162+0000","last_peered":"2024-09-13T14:44:27.414162+0000","last_clean":"2024-09-13T14:44:27.414162+0000","last_became_active":"2024-09-13T14:38:04.600616+0000","last_became_peered":"2024-09-13T14:38:04.600616+0000","last_unstale":"2024-09-13T14:44:27.414162+0000","last_undegraded":"2024-09-13T14:44:27.414162+0000","last_fullsized":"2024-09-13T14:44:27.414162+0000","mapping_epoch":245,"log_start":"219'5716","ondisk_log_start":"219'5716","created":205,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6294,"log_dups_size":0,"ondisk_log_size":6294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:00:44.321678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139722752,"num_objects":2132,"num_object_clones":0,"num_object_copies":6396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12010,"num_write_kb":452544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8120,"num_bytes_recovered":521076736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":511,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:16.636697+0000","last_change":"2024-09-13T14:26:15.835178+0000","last_active":"2024-09-13T14:44:16.636697+0000","last_peered":"2024-09-13T14:44:16.636697+0000","last_clean":"2024-09-13T14:44:16.636697+0000","last_became_active":"2024-09-13T14:26:15.820283+0000","last_became_peered":"2024-09-13T14:26:15.820283+0000","last_unstale":"2024-09-13T14:44:16.636697+0000","last_undegraded":"2024-09-13T14:44:16.636697+0000","last_fullsized":"2024-09-13T14:44:16.636697+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:30.340038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":111,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:16.636920+0000","last_change":"2024-09-13T14:38:03.525682+0000","last_active":"2024-09-13T14:44:16.636920+0000","last_peered":"2024-09-13T14:44:16.636920+0000","last_clean":"2024-09-13T14:44:16.636920+0000","last_became_active":"2024-09-13T14:38:03.525406+0000","last_became_peered":"2024-09-13T14:38:03.525406+0000","last_unstale":"2024-09-13T14:44:16.636920+0000","last_undegraded":"2024-09-13T14:44:16.636920+0000","last_fullsized":"2024-09-13T14:44:16.636920+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:41:28.970300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":117,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:16.637572+0000","last_change":"2024-09-13T14:38:53.991738+0000","last_active":"2024-09-13T14:44:16.637572+0000","last_peered":"2024-09-13T14:44:16.637572+0000","last_clean":"2024-09-13T14:44:16.637572+0000","last_became_active":"2024-09-13T14:38:53.991417+0000","last_became_peered":"2024-09-13T14:38:53.991417+0000","last_unstale":"2024-09-13T14:44:16.637572+0000","last_undegraded":"2024-09-13T14:44:16.637572+0000","last_fullsized":"2024-09-13T14:44:16.637572+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:50:38.559838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"293'11870","reported_seq":30441,"reported_epoch":293,"state":"active+recovering","last_fresh":"2024-09-13T14:44:26.973991+0000","last_change":"2024-09-13T14:43:50.028785+0000","last_active":"2024-09-13T14:44:26.973991+0000","last_peered":"2024-09-13T14:44:26.973991+0000","last_clean":"2024-09-13T14:37:46.290754+0000","last_became_active":"2024-09-13T14:38:17.381753+0000","last_became_peered":"2024-09-13T14:38:17.381753+0000","last_unstale":"2024-09-13T14:44:26.973991+0000","last_undegraded":"2024-09-13T14:44:26.973991+0000","last_fullsized":"2024-09-13T14:44:26.973991+0000","mapping_epoch":245,"log_start":"219'5959","ondisk_log_start":"219'5959","created":205,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5911,"log_dups_size":0,"ondisk_log_size":5911,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146538496,"num_objects":2236,"num_object_clones":0,"num_object_copies":6708,"num_objects_missing_on_primary":80,"num_objects_missing":80,"num_objects_degraded":0,"num_objects_misplaced":2236,"num_objects_unfound":0,"num_objects_dirty":2236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11870,"num_write_kb":451392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9233,"num_bytes_recovered":604897280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"0,4,5","objects":2236}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":832,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:16.636501+0000","last_change":"2024-09-13T14:37:48.131799+0000","last_active":"2024-09-13T14:44:16.636501+0000","last_peered":"2024-09-13T14:44:16.636501+0000","last_clean":"2024-09-13T14:44:16.636501+0000","last_became_active":"2024-09-13T14:37:48.130837+0000","last_became_peered":"2024-09-13T14:37:48.130837+0000","last_unstale":"2024-09-13T14:44:16.636501+0000","last_undegraded":"2024-09-13T14:44:16.636501+0000","last_fullsized":"2024-09-13T14:44:16.636501+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:06:32.663770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":351,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:16.637051+0000","last_change":"2024-09-13T14:37:49.301531+0000","last_active":"2024-09-13T14:44:16.637051+0000","last_peered":"2024-09-13T14:44:16.637051+0000","last_clean":"2024-09-13T14:44:16.637051+0000","last_became_active":"2024-09-13T14:37:49.301249+0000","last_became_peered":"2024-09-13T14:37:49.301249+0000","last_unstale":"2024-09-13T14:44:16.637051+0000","last_undegraded":"2024-09-13T14:44:16.637051+0000","last_fullsized":"2024-09-13T14:44:16.637051+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:10.977710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":119,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.624338+0000","last_change":"2024-09-13T14:38:53.901427+0000","last_active":"2024-09-13T14:44:06.624338+0000","last_peered":"2024-09-13T14:44:06.624338+0000","last_clean":"2024-09-13T14:44:06.624338+0000","last_became_active":"2024-09-13T14:38:53.901157+0000","last_became_peered":"2024-09-13T14:38:53.901157+0000","last_unstale":"2024-09-13T14:44:06.624338+0000","last_undegraded":"2024-09-13T14:44:06.624338+0000","last_fullsized":"2024-09-13T14:44:06.624338+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:28:21.713025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"293'11848","reported_seq":35097,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:26.883437+0000","last_change":"2024-09-13T14:43:16.851222+0000","last_active":"2024-09-13T14:44:26.883437+0000","last_peered":"2024-09-13T14:44:26.883437+0000","last_clean":"2024-09-13T14:44:26.883437+0000","last_became_active":"2024-09-13T14:38:16.427510+0000","last_became_peered":"2024-09-13T14:38:16.427510+0000","last_unstale":"2024-09-13T14:44:26.883437+0000","last_undegraded":"2024-09-13T14:44:26.883437+0000","last_fullsized":"2024-09-13T14:44:26.883437+0000","mapping_epoch":245,"log_start":"219'5918","ondisk_log_start":"219'5918","created":205,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5930,"log_dups_size":0,"ondisk_log_size":5930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:27:26.281938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140902400,"num_objects":2150,"num_object_clones":0,"num_object_copies":6450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11848,"num_write_kb":447936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11571,"num_bytes_recovered":747372544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":1001,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.623391+0000","last_change":"2024-09-13T14:37:49.147498+0000","last_active":"2024-09-13T14:44:06.623391+0000","last_peered":"2024-09-13T14:44:06.623391+0000","last_clean":"2024-09-13T14:44:06.623391+0000","last_became_active":"2024-09-13T14:37:49.147028+0000","last_became_peered":"2024-09-13T14:37:49.147028+0000","last_unstale":"2024-09-13T14:44:06.623391+0000","last_undegraded":"2024-09-13T14:44:06.623391+0000","last_fullsized":"2024-09-13T14:44:06.623391+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:11:29.050178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":299,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.616219+0000","last_change":"2024-09-13T14:38:03.528133+0000","last_active":"2024-09-13T14:44:06.616219+0000","last_peered":"2024-09-13T14:44:06.616219+0000","last_clean":"2024-09-13T14:44:06.616219+0000","last_became_active":"2024-09-13T14:38:03.527829+0000","last_became_peered":"2024-09-13T14:38:03.527829+0000","last_unstale":"2024-09-13T14:44:06.616219+0000","last_undegraded":"2024-09-13T14:44:06.616219+0000","last_fullsized":"2024-09-13T14:44:06.616219+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:48.254169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":471,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:16.636859+0000","last_change":"2024-09-13T14:38:53.899755+0000","last_active":"2024-09-13T14:44:16.636859+0000","last_peered":"2024-09-13T14:44:16.636859+0000","last_clean":"2024-09-13T14:44:16.636859+0000","last_became_active":"2024-09-13T14:38:53.899448+0000","last_became_peered":"2024-09-13T14:38:53.899448+0000","last_unstale":"2024-09-13T14:44:16.636859+0000","last_undegraded":"2024-09-13T14:44:16.636859+0000","last_fullsized":"2024-09-13T14:44:16.636859+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:55.897035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"293'12059","reported_seq":21194,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:22.853215+0000","last_change":"2024-09-13T14:43:51.649951+0000","last_active":"2024-09-13T14:44:22.853215+0000","last_peered":"2024-09-13T14:44:22.853215+0000","last_clean":"2024-09-13T14:44:22.853215+0000","last_became_active":"2024-09-13T14:43:51.626628+0000","last_became_peered":"2024-09-13T14:43:51.626628+0000","last_unstale":"2024-09-13T14:44:22.853215+0000","last_undegraded":"2024-09-13T14:44:22.853215+0000","last_fullsized":"2024-09-13T14:44:22.853215+0000","mapping_epoch":289,"log_start":"219'5163","ondisk_log_start":"219'5163","created":205,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6896,"log_dups_size":0,"ondisk_log_size":6896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:37:06.795173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146997248,"num_objects":2243,"num_object_clones":0,"num_object_copies":6729,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12059,"num_write_kb":457664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4502,"num_bytes_recovered":228851712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":103,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.623801+0000","last_change":"2024-09-13T14:37:49.149770+0000","last_active":"2024-09-13T14:44:06.623801+0000","last_peered":"2024-09-13T14:44:06.623801+0000","last_clean":"2024-09-13T14:44:06.623801+0000","last_became_active":"2024-09-13T14:37:49.148991+0000","last_became_peered":"2024-09-13T14:37:49.148991+0000","last_unstale":"2024-09-13T14:44:06.623801+0000","last_undegraded":"2024-09-13T14:44:06.623801+0000","last_fullsized":"2024-09-13T14:44:06.623801+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:04:04.193366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":1169,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:16.651729+0000","last_change":"2024-09-13T14:30:58.928302+0000","last_active":"2024-09-13T14:44:16.651729+0000","last_peered":"2024-09-13T14:44:16.651729+0000","last_clean":"2024-09-13T14:44:16.651729+0000","last_became_active":"2024-09-13T14:30:58.927718+0000","last_became_peered":"2024-09-13T14:30:58.927718+0000","last_unstale":"2024-09-13T14:44:16.651729+0000","last_undegraded":"2024-09-13T14:44:16.651729+0000","last_fullsized":"2024-09-13T14:44:16.651729+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:37.323036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":110,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.623313+0000","last_change":"2024-09-13T14:38:03.534628+0000","last_active":"2024-09-13T14:44:06.623313+0000","last_peered":"2024-09-13T14:44:06.623313+0000","last_clean":"2024-09-13T14:44:06.623313+0000","last_became_active":"2024-09-13T14:38:03.534262+0000","last_became_peered":"2024-09-13T14:38:03.534262+0000","last_unstale":"2024-09-13T14:44:06.623313+0000","last_undegraded":"2024-09-13T14:44:06.623313+0000","last_fullsized":"2024-09-13T14:44:06.623313+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:31.168783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"293'11925","reported_seq":26106,"reported_epoch":293,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:44:27.175331+0000","last_change":"2024-09-13T14:38:53.944111+0000","last_active":"2024-09-13T14:44:27.175331+0000","last_peered":"2024-09-13T14:44:27.175331+0000","last_clean":"2024-09-13T14:37:47.047672+0000","last_became_active":"2024-09-13T14:37:49.664294+0000","last_became_peered":"2024-09-13T14:37:49.664294+0000","last_unstale":"2024-09-13T14:44:27.175331+0000","last_undegraded":"2024-09-13T14:38:53.940822+0000","last_fullsized":"2024-09-13T14:37:49.126627+0000","mapping_epoch":240,"log_start":"219'6200","ondisk_log_start":"219'6200","created":205,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5725,"log_dups_size":0,"ondisk_log_size":5725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:11.766692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139395072,"num_objects":2127,"num_object_clones":0,"num_object_copies":6381,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4535,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11925,"num_write_kb":449664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7009,"num_bytes_recovered":459341824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3],"avail_no_missing":["5","0","3"],"object_location_counts":[{"shards":"0,3,5","objects":2127},{"shards":"3,5","objects":-2333}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":122,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.623654+0000","last_change":"2024-09-13T14:38:03.534303+0000","last_active":"2024-09-13T14:44:06.623654+0000","last_peered":"2024-09-13T14:44:06.623654+0000","last_clean":"2024-09-13T14:44:06.623654+0000","last_became_active":"2024-09-13T14:38:03.533171+0000","last_became_peered":"2024-09-13T14:38:03.533171+0000","last_unstale":"2024-09-13T14:44:06.623654+0000","last_undegraded":"2024-09-13T14:44:06.623654+0000","last_fullsized":"2024-09-13T14:44:06.623654+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:00:16.553526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":815,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.635169+0000","last_change":"2024-09-13T14:38:04.596192+0000","last_active":"2024-09-13T14:44:06.635169+0000","last_peered":"2024-09-13T14:44:06.635169+0000","last_clean":"2024-09-13T14:44:06.635169+0000","last_became_active":"2024-09-13T14:38:04.593299+0000","last_became_peered":"2024-09-13T14:38:04.593299+0000","last_unstale":"2024-09-13T14:44:06.635169+0000","last_undegraded":"2024-09-13T14:44:06.635169+0000","last_fullsized":"2024-09-13T14:44:06.635169+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:34.241410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":110,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:16.637523+0000","last_change":"2024-09-13T14:38:03.533120+0000","last_active":"2024-09-13T14:44:16.637523+0000","last_peered":"2024-09-13T14:44:16.637523+0000","last_clean":"2024-09-13T14:44:16.637523+0000","last_became_active":"2024-09-13T14:38:03.532829+0000","last_became_peered":"2024-09-13T14:38:03.532829+0000","last_unstale":"2024-09-13T14:44:16.637523+0000","last_undegraded":"2024-09-13T14:44:16.637523+0000","last_fullsized":"2024-09-13T14:44:16.637523+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:24.514661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"293'12043","reported_seq":21318,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:22.522806+0000","last_change":"2024-09-13T14:42:04.660877+0000","last_active":"2024-09-13T14:44:22.522806+0000","last_peered":"2024-09-13T14:44:22.522806+0000","last_clean":"2024-09-13T14:44:22.522806+0000","last_became_active":"2024-09-13T14:42:04.641680+0000","last_became_peered":"2024-09-13T14:42:04.641680+0000","last_unstale":"2024-09-13T14:44:22.522806+0000","last_undegraded":"2024-09-13T14:44:22.522806+0000","last_fullsized":"2024-09-13T14:44:22.522806+0000","mapping_epoch":277,"log_start":"219'6318","ondisk_log_start":"219'6318","created":205,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5725,"log_dups_size":0,"ondisk_log_size":5725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:22:18.955428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154337280,"num_objects":2355,"num_object_clones":0,"num_object_copies":7065,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12043,"num_write_kb":460736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4533,"num_bytes_recovered":297074688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":513,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:16.636276+0000","last_change":"2024-09-13T14:38:53.938227+0000","last_active":"2024-09-13T14:44:16.636276+0000","last_peered":"2024-09-13T14:44:16.636276+0000","last_clean":"2024-09-13T14:44:16.636276+0000","last_became_active":"2024-09-13T14:38:53.902867+0000","last_became_peered":"2024-09-13T14:38:53.902867+0000","last_unstale":"2024-09-13T14:44:16.636276+0000","last_undegraded":"2024-09-13T14:44:16.636276+0000","last_fullsized":"2024-09-13T14:44:16.636276+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:16.831756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":593,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.616334+0000","last_change":"2024-09-13T14:37:48.147677+0000","last_active":"2024-09-13T14:44:06.616334+0000","last_peered":"2024-09-13T14:44:06.616334+0000","last_clean":"2024-09-13T14:44:06.616334+0000","last_became_active":"2024-09-13T14:37:48.147001+0000","last_became_peered":"2024-09-13T14:37:48.147001+0000","last_unstale":"2024-09-13T14:44:06.616334+0000","last_undegraded":"2024-09-13T14:44:06.616334+0000","last_fullsized":"2024-09-13T14:44:06.616334+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:35:00.653576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":494,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:16.636262+0000","last_change":"2024-09-13T14:38:53.900107+0000","last_active":"2024-09-13T14:44:16.636262+0000","last_peered":"2024-09-13T14:44:16.636262+0000","last_clean":"2024-09-13T14:44:16.636262+0000","last_became_active":"2024-09-13T14:38:53.899556+0000","last_became_peered":"2024-09-13T14:38:53.899556+0000","last_unstale":"2024-09-13T14:44:16.636262+0000","last_undegraded":"2024-09-13T14:44:16.636262+0000","last_fullsized":"2024-09-13T14:44:16.636262+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:15:47.918163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"293'11601","reported_seq":20602,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:22.423729+0000","last_change":"2024-09-13T14:39:54.921239+0000","last_active":"2024-09-13T14:44:22.423729+0000","last_peered":"2024-09-13T14:44:22.423729+0000","last_clean":"2024-09-13T14:44:22.423729+0000","last_became_active":"2024-09-13T14:39:54.907894+0000","last_became_peered":"2024-09-13T14:39:54.907894+0000","last_unstale":"2024-09-13T14:44:22.423729+0000","last_undegraded":"2024-09-13T14:44:22.423729+0000","last_fullsized":"2024-09-13T14:44:22.423729+0000","mapping_epoch":263,"log_start":"219'5847","ondisk_log_start":"219'5847","created":205,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5754,"log_dups_size":0,"ondisk_log_size":5754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:52:04.619238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147521536,"num_objects":2251,"num_object_clones":0,"num_object_copies":6753,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11601,"num_write_kb":443264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4395,"num_bytes_recovered":283770880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":125,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:16.637414+0000","last_change":"2024-09-13T14:38:53.901806+0000","last_active":"2024-09-13T14:44:16.637414+0000","last_peered":"2024-09-13T14:44:16.637414+0000","last_clean":"2024-09-13T14:44:16.637414+0000","last_became_active":"2024-09-13T14:38:53.891055+0000","last_became_peered":"2024-09-13T14:38:53.891055+0000","last_unstale":"2024-09-13T14:44:16.637414+0000","last_undegraded":"2024-09-13T14:44:16.637414+0000","last_fullsized":"2024-09-13T14:44:16.637414+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:03:00.416198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":68,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.635243+0000","last_change":"2024-09-13T14:43:16.854299+0000","last_active":"2024-09-13T14:44:06.635243+0000","last_peered":"2024-09-13T14:44:06.635243+0000","last_clean":"2024-09-13T14:44:06.635243+0000","last_became_active":"2024-09-13T14:38:53.900626+0000","last_became_peered":"2024-09-13T14:38:53.900626+0000","last_unstale":"2024-09-13T14:44:06.635243+0000","last_undegraded":"2024-09-13T14:44:06.635243+0000","last_fullsized":"2024-09-13T14:44:06.635243+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:37:21.715778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":283,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.616539+0000","last_change":"2024-09-13T14:37:49.172218+0000","last_active":"2024-09-13T14:44:06.616539+0000","last_peered":"2024-09-13T14:44:06.616539+0000","last_clean":"2024-09-13T14:44:06.616539+0000","last_became_active":"2024-09-13T14:37:49.171718+0000","last_became_peered":"2024-09-13T14:37:49.171718+0000","last_unstale":"2024-09-13T14:44:06.616539+0000","last_undegraded":"2024-09-13T14:44:06.616539+0000","last_fullsized":"2024-09-13T14:44:06.616539+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:46:54.626521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":306,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:12.875118+0000","last_change":"2024-09-13T14:37:49.230094+0000","last_active":"2024-09-13T14:44:12.875118+0000","last_peered":"2024-09-13T14:44:12.875118+0000","last_clean":"2024-09-13T14:44:12.875118+0000","last_became_active":"2024-09-13T14:37:49.229393+0000","last_became_peered":"2024-09-13T14:37:49.229393+0000","last_unstale":"2024-09-13T14:44:12.875118+0000","last_undegraded":"2024-09-13T14:44:12.875118+0000","last_fullsized":"2024-09-13T14:44:12.875118+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:02:28.690318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":286,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:16.634741+0000","last_change":"2024-09-13T14:37:48.128447+0000","last_active":"2024-09-13T14:44:16.634741+0000","last_peered":"2024-09-13T14:44:16.634741+0000","last_clean":"2024-09-13T14:44:16.634741+0000","last_became_active":"2024-09-13T14:37:48.128143+0000","last_became_peered":"2024-09-13T14:37:48.128143+0000","last_unstale":"2024-09-13T14:44:16.634741+0000","last_undegraded":"2024-09-13T14:44:16.634741+0000","last_fullsized":"2024-09-13T14:44:16.634741+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:22.204039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":110,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:16.637102+0000","last_change":"2024-09-13T14:38:03.531683+0000","last_active":"2024-09-13T14:44:16.637102+0000","last_peered":"2024-09-13T14:44:16.637102+0000","last_clean":"2024-09-13T14:44:16.637102+0000","last_became_active":"2024-09-13T14:38:03.531476+0000","last_became_peered":"2024-09-13T14:38:03.531476+0000","last_unstale":"2024-09-13T14:44:16.637102+0000","last_undegraded":"2024-09-13T14:44:16.637102+0000","last_fullsized":"2024-09-13T14:44:16.637102+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:41.557569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":58,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.634967+0000","last_change":"2024-09-13T14:38:53.954281+0000","last_active":"2024-09-13T14:44:06.634967+0000","last_peered":"2024-09-13T14:44:06.634967+0000","last_clean":"2024-09-13T14:44:06.634967+0000","last_became_active":"2024-09-13T14:38:53.900912+0000","last_became_peered":"2024-09-13T14:38:53.900912+0000","last_unstale":"2024-09-13T14:44:06.634967+0000","last_undegraded":"2024-09-13T14:44:06.634967+0000","last_fullsized":"2024-09-13T14:44:06.634967+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:47.876275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":110,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.624146+0000","last_change":"2024-09-13T14:38:54.000782+0000","last_active":"2024-09-13T14:44:06.624146+0000","last_peered":"2024-09-13T14:44:06.624146+0000","last_clean":"2024-09-13T14:44:06.624146+0000","last_became_active":"2024-09-13T14:38:54.000322+0000","last_became_peered":"2024-09-13T14:38:54.000322+0000","last_unstale":"2024-09-13T14:44:06.624146+0000","last_undegraded":"2024-09-13T14:44:06.624146+0000","last_fullsized":"2024-09-13T14:44:06.624146+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:09.057597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":436,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:16.636202+0000","last_change":"2024-09-13T14:38:53.899535+0000","last_active":"2024-09-13T14:44:16.636202+0000","last_peered":"2024-09-13T14:44:16.636202+0000","last_clean":"2024-09-13T14:44:16.636202+0000","last_became_active":"2024-09-13T14:38:53.899183+0000","last_became_peered":"2024-09-13T14:38:53.899183+0000","last_unstale":"2024-09-13T14:44:16.636202+0000","last_undegraded":"2024-09-13T14:44:16.636202+0000","last_fullsized":"2024-09-13T14:44:16.636202+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:33.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":110,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:16.637896+0000","last_change":"2024-09-13T14:38:03.543426+0000","last_active":"2024-09-13T14:44:16.637896+0000","last_peered":"2024-09-13T14:44:16.637896+0000","last_clean":"2024-09-13T14:44:16.637896+0000","last_became_active":"2024-09-13T14:38:03.542661+0000","last_became_peered":"2024-09-13T14:38:03.542661+0000","last_unstale":"2024-09-13T14:44:16.637896+0000","last_undegraded":"2024-09-13T14:44:16.637896+0000","last_fullsized":"2024-09-13T14:44:16.637896+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:54.050127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":465,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:16.635895+0000","last_change":"2024-09-13T14:37:48.131321+0000","last_active":"2024-09-13T14:44:16.635895+0000","last_peered":"2024-09-13T14:44:16.635895+0000","last_clean":"2024-09-13T14:44:16.635895+0000","last_became_active":"2024-09-13T14:37:48.130998+0000","last_became_peered":"2024-09-13T14:37:48.130998+0000","last_unstale":"2024-09-13T14:44:16.635895+0000","last_undegraded":"2024-09-13T14:44:16.635895+0000","last_fullsized":"2024-09-13T14:44:16.635895+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:33:26.597825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":122,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.624302+0000","last_change":"2024-09-13T14:38:03.535454+0000","last_active":"2024-09-13T14:44:06.624302+0000","last_peered":"2024-09-13T14:44:06.624302+0000","last_clean":"2024-09-13T14:44:06.624302+0000","last_became_active":"2024-09-13T14:38:03.535131+0000","last_became_peered":"2024-09-13T14:38:03.535131+0000","last_unstale":"2024-09-13T14:44:06.624302+0000","last_undegraded":"2024-09-13T14:44:06.624302+0000","last_fullsized":"2024-09-13T14:44:06.624302+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:24.477601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":490,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:16.636387+0000","last_change":"2024-09-13T14:38:03.524939+0000","last_active":"2024-09-13T14:44:16.636387+0000","last_peered":"2024-09-13T14:44:16.636387+0000","last_clean":"2024-09-13T14:44:16.636387+0000","last_became_active":"2024-09-13T14:38:03.524406+0000","last_became_peered":"2024-09-13T14:38:03.524406+0000","last_unstale":"2024-09-13T14:44:16.636387+0000","last_undegraded":"2024-09-13T14:44:16.636387+0000","last_fullsized":"2024-09-13T14:44:16.636387+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:19:45.257424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":118,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:16.637369+0000","last_change":"2024-09-13T14:38:53.892948+0000","last_active":"2024-09-13T14:44:16.637369+0000","last_peered":"2024-09-13T14:44:16.637369+0000","last_clean":"2024-09-13T14:44:16.637369+0000","last_became_active":"2024-09-13T14:38:53.892665+0000","last_became_peered":"2024-09-13T14:38:53.892665+0000","last_unstale":"2024-09-13T14:44:16.637369+0000","last_undegraded":"2024-09-13T14:44:16.637369+0000","last_fullsized":"2024-09-13T14:44:16.637369+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:08.292359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":438,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:16.636404+0000","last_change":"2024-09-13T14:37:48.139348+0000","last_active":"2024-09-13T14:44:16.636404+0000","last_peered":"2024-09-13T14:44:16.636404+0000","last_clean":"2024-09-13T14:44:16.636404+0000","last_became_active":"2024-09-13T14:37:48.139115+0000","last_became_peered":"2024-09-13T14:37:48.139115+0000","last_unstale":"2024-09-13T14:44:16.636404+0000","last_undegraded":"2024-09-13T14:44:16.636404+0000","last_fullsized":"2024-09-13T14:44:16.636404+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:32:20.413346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":478,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:16.635929+0000","last_change":"2024-09-13T14:37:48.172088+0000","last_active":"2024-09-13T14:44:16.635929+0000","last_peered":"2024-09-13T14:44:16.635929+0000","last_clean":"2024-09-13T14:44:16.635929+0000","last_became_active":"2024-09-13T14:37:48.171779+0000","last_became_peered":"2024-09-13T14:37:48.171779+0000","last_unstale":"2024-09-13T14:44:16.635929+0000","last_undegraded":"2024-09-13T14:44:16.635929+0000","last_fullsized":"2024-09-13T14:44:16.635929+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:02.852712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":350,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.616089+0000","last_change":"2024-09-13T14:38:53.890236+0000","last_active":"2024-09-13T14:44:06.616089+0000","last_peered":"2024-09-13T14:44:06.616089+0000","last_clean":"2024-09-13T14:44:06.616089+0000","last_became_active":"2024-09-13T14:38:53.889582+0000","last_became_peered":"2024-09-13T14:38:53.889582+0000","last_unstale":"2024-09-13T14:44:06.616089+0000","last_undegraded":"2024-09-13T14:44:06.616089+0000","last_fullsized":"2024-09-13T14:44:06.616089+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:14:29.334774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":464,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:16.637111+0000","last_change":"2024-09-13T14:38:03.534097+0000","last_active":"2024-09-13T14:44:16.637111+0000","last_peered":"2024-09-13T14:44:16.637111+0000","last_clean":"2024-09-13T14:44:16.637111+0000","last_became_active":"2024-09-13T14:38:03.533794+0000","last_became_peered":"2024-09-13T14:38:03.533794+0000","last_unstale":"2024-09-13T14:44:16.637111+0000","last_undegraded":"2024-09-13T14:44:16.637111+0000","last_fullsized":"2024-09-13T14:44:16.637111+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:23.272098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":110,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:16.637345+0000","last_change":"2024-09-13T14:38:03.527021+0000","last_active":"2024-09-13T14:44:16.637345+0000","last_peered":"2024-09-13T14:44:16.637345+0000","last_clean":"2024-09-13T14:44:16.637345+0000","last_became_active":"2024-09-13T14:38:03.526698+0000","last_became_peered":"2024-09-13T14:38:03.526698+0000","last_unstale":"2024-09-13T14:44:16.637345+0000","last_undegraded":"2024-09-13T14:44:16.637345+0000","last_fullsized":"2024-09-13T14:44:16.637345+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:01:35.734086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":294,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.616058+0000","last_change":"2024-09-13T14:27:12.834931+0000","last_active":"2024-09-13T14:44:06.616058+0000","last_peered":"2024-09-13T14:44:06.616058+0000","last_clean":"2024-09-13T14:44:06.616058+0000","last_became_active":"2024-09-13T14:27:12.834572+0000","last_became_peered":"2024-09-13T14:27:12.834572+0000","last_unstale":"2024-09-13T14:44:06.616058+0000","last_undegraded":"2024-09-13T14:44:06.616058+0000","last_fullsized":"2024-09-13T14:44:06.616058+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:54:21.155689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":107,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.623574+0000","last_change":"2024-09-13T14:37:49.176796+0000","last_active":"2024-09-13T14:44:06.623574+0000","last_peered":"2024-09-13T14:44:06.623574+0000","last_clean":"2024-09-13T14:44:06.623574+0000","last_became_active":"2024-09-13T14:37:49.176246+0000","last_became_peered":"2024-09-13T14:37:49.176246+0000","last_unstale":"2024-09-13T14:44:06.623574+0000","last_undegraded":"2024-09-13T14:44:06.623574+0000","last_fullsized":"2024-09-13T14:44:06.623574+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:29:24.936718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":305,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.615900+0000","last_change":"2024-09-13T14:38:03.526333+0000","last_active":"2024-09-13T14:44:06.615900+0000","last_peered":"2024-09-13T14:44:06.615900+0000","last_clean":"2024-09-13T14:44:06.615900+0000","last_became_active":"2024-09-13T14:38:03.526001+0000","last_became_peered":"2024-09-13T14:38:03.526001+0000","last_unstale":"2024-09-13T14:44:06.615900+0000","last_undegraded":"2024-09-13T14:44:06.615900+0000","last_fullsized":"2024-09-13T14:44:06.615900+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:25:25.181271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":107,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.623806+0000","last_change":"2024-09-13T14:37:49.149582+0000","last_active":"2024-09-13T14:44:06.623806+0000","last_peered":"2024-09-13T14:44:06.623806+0000","last_clean":"2024-09-13T14:44:06.623806+0000","last_became_active":"2024-09-13T14:37:49.148525+0000","last_became_peered":"2024-09-13T14:37:49.148525+0000","last_unstale":"2024-09-13T14:44:06.623806+0000","last_undegraded":"2024-09-13T14:44:06.623806+0000","last_fullsized":"2024-09-13T14:44:06.623806+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:48.655665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":511,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:16.636629+0000","last_change":"2024-09-13T14:38:03.535255+0000","last_active":"2024-09-13T14:44:16.636629+0000","last_peered":"2024-09-13T14:44:16.636629+0000","last_clean":"2024-09-13T14:44:16.636629+0000","last_became_active":"2024-09-13T14:38:03.533917+0000","last_became_peered":"2024-09-13T14:38:03.533917+0000","last_unstale":"2024-09-13T14:44:16.636629+0000","last_undegraded":"2024-09-13T14:44:16.636629+0000","last_fullsized":"2024-09-13T14:44:16.636629+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:33.559788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":474,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:16.636542+0000","last_change":"2024-09-13T14:37:48.146784+0000","last_active":"2024-09-13T14:44:16.636542+0000","last_peered":"2024-09-13T14:44:16.636542+0000","last_clean":"2024-09-13T14:44:16.636542+0000","last_became_active":"2024-09-13T14:37:48.146491+0000","last_became_peered":"2024-09-13T14:37:48.146491+0000","last_unstale":"2024-09-13T14:44:16.636542+0000","last_undegraded":"2024-09-13T14:44:16.636542+0000","last_fullsized":"2024-09-13T14:44:16.636542+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:30:54.772513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":285,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-13T14:44:06.615655+0000","last_change":"2024-09-13T14:37:48.127607+0000","last_active":"2024-09-13T14:44:06.615655+0000","last_peered":"2024-09-13T14:44:06.615655+0000","last_clean":"2024-09-13T14:44:06.615655+0000","last_became_active":"2024-09-13T14:37:48.127316+0000","last_became_peered":"2024-09-13T14:37:48.127316+0000","last_unstale":"2024-09-13T14:44:06.615655+0000","last_undegraded":"2024-09-13T14:44:06.615655+0000","last_fullsized":"2024-09-13T14:44:06.615655+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:53:37.678861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":117,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:16.637839+0000","last_change":"2024-09-13T14:38:03.563200+0000","last_active":"2024-09-13T14:44:16.637839+0000","last_peered":"2024-09-13T14:44:16.637839+0000","last_clean":"2024-09-13T14:44:16.637839+0000","last_became_active":"2024-09-13T14:38:03.534258+0000","last_became_peered":"2024-09-13T14:38:03.534258+0000","last_unstale":"2024-09-13T14:44:16.637839+0000","last_undegraded":"2024-09-13T14:44:16.637839+0000","last_fullsized":"2024-09-13T14:44:16.637839+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:49:07.814419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":2298413080,"num_objects":35072,"num_object_clones":0,"num_object_copies":105216,"num_objects_missing_on_primary":80,"num_objects_missing":80,"num_objects_degraded":4814,"num_objects_misplaced":4124,"num_objects_unfound":0,"num_objects_dirty":35072,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190554,"num_write_kb":7219969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85057,"num_bytes_recovered":5447090200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9222696960,"data_stored":9222684744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94977,"ondisk_log_size":94977,"up":48,"acting":47,"num_store_stats":6},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1538,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":17390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25132,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16793600,"data_stored":16777292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":23,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2969600,"data_stored":2951840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234407,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1760676,"kb_used_data":1300508,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":91984476,"statfs":{"total":95995035648,"available":94192103424,"internally_reserved":0,"allocated":1331720192,"data_stored":1330155921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":471202251},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":239,"seq":1026497183825,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1450088,"kb_used_data":1130976,"kb_used_omap":8,"kb_used_meta":319095,"kb_avail":92295064,"statfs":{"total":95995035648,"available":94510145536,"internally_reserved":0,"allocated":1158119424,"data_stored":1156551628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8453,"internal_metadata":326754043},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757672,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1797460,"kb_used_data":1134668,"kb_used_omap":1,"kb_used_meta":662782,"kb_avail":91947692,"statfs":{"total":95995035648,"available":94154436608,"internally_reserved":0,"allocated":1161900032,"data_stored":1160347322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":678689222},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019315,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1928668,"kb_used_data":1324116,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":91816484,"statfs":{"total":95995035648,"available":94020079616,"internally_reserved":0,"allocated":1355894784,"data_stored":1354338149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":619051462},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435498,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2354192,"kb_used_data":1754696,"kb_used_omap":5,"kb_used_meta":599482,"kb_avail":91390960,"statfs":{"total":95995035648,"available":93584343040,"internally_reserved":0,"allocated":1796808704,"data_stored":1795259854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":613869694},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":446676599027,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":425201762553,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926077,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3120672,"kb_used_data":2435096,"kb_used_omap":7,"kb_used_meta":685560,"kb_avail":90624480,"statfs":{"total":95995035648,"available":92799467520,"internally_reserved":0,"allocated":2493538304,"data_stored":2491974237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":702013736},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6860,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2477723648,"data_stored":2477719576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1786712064,"data_stored":1786707992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1348075520,"data_stored":1348075520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1147273216,"data_stored":1147273216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1146884096,"data_stored":1146880024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1316028416,"data_stored":1316028416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:44:32.215 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph status --format=json 2024-09-13T14:44:32.225 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:32.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:32.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:32.327 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:32.354 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:32.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:32.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:32.456 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:32.478 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:32.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:32.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:32.580 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:32.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:32.605 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:32.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:32.706 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:32.731 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:32.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:32.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:32.833 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:32.859 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:32.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:32.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:32.873 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:32 smithi102 ceph-mon[110183]: pgmap v1020: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.1 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 370 op/s; 4823/103263 objects degraded (4.671%); 4124/103263 objects misplaced (3.994%); 2.1 MiB/s, 34 objects/s recovering 2024-09-13T14:44:32.873 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:32 smithi102 ceph-mon[115269]: pgmap v1020: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.1 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 370 op/s; 4823/103263 objects degraded (4.671%); 4124/103263 objects misplaced (3.994%); 2.1 MiB/s, 34 objects/s recovering 2024-09-13T14:44:32.961 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:32.982 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:32.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:32.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:32.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:32 smithi195 ceph-mon[104262]: pgmap v1020: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.1 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 370 op/s; 4823/103263 objects degraded (4.671%); 4124/103263 objects misplaced (3.994%); 2.1 MiB/s, 34 objects/s recovering 2024-09-13T14:44:33.084 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:33.107 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:33.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:33.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:33.209 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:33.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:33.231 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:33.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:33.332 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:33.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:33.357 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:33.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:33.458 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:33.480 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:33.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:33.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:33.582 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:33.605 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:33.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:33.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:33.707 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:33.729 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:33.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:33.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:33.831 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:33.852 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:33.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:33.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:33.954 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:33.975 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:33.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:33.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:34.076 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:34.099 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:34.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:34.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:34.201 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:34.223 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:34.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:34.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:34.324 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:34.351 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:34.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:34.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:34.389 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:44:34.389 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:44:34.407 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:44:34.407 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:44:34.427 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:44:34.427 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:44:34.444 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:44:34.444 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:44:34.452 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:34.461 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:44:34.461 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:44:34.476 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:34.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:34.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:34.479 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:44:34.479 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:44:34.497 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:44:34.498 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:44:34.514 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:44:34.514 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:44:34.577 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:34.601 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:34.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:34.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:34.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:34 smithi102 ceph-mon[110183]: pgmap v1021: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.0 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 364 op/s; 4823/98439 objects degraded (4.899%); 3901/98439 objects misplaced (3.963%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:44:34.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:34 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 4823/98439 objects degraded (4.899%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:34.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:34 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/814491134' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:44:34.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:34 smithi102 ceph-mon[115269]: pgmap v1021: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.0 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 364 op/s; 4823/98439 objects degraded (4.899%); 3901/98439 objects misplaced (3.963%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:44:34.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:34 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 4823/98439 objects degraded (4.899%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:34.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:34 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/814491134' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:44:34.703 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:34.724 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:34.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:34.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:34.825 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:34.847 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:34.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:34.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:34.948 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:34.970 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:34.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:34.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:34.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:34 smithi195 ceph-mon[104262]: pgmap v1021: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.0 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 364 op/s; 4823/98439 objects degraded (4.899%); 3901/98439 objects misplaced (3.963%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:44:34.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:34 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 4823/98439 objects degraded (4.899%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:34.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:34 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/814491134' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:44:35.072 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:35.097 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:35.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:35.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:35.198 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:35.220 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:35.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:35.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:35.231 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:44:35.232 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:35.322 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:35.343 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:35.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:35.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:35.445 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:35.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:35.469 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:35.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:35.570 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:35.593 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:35.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:35.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:35.694 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:35.716 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:35.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:35.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:35.817 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:35.841 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:35.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:35.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:35.943 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:35.966 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:35.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:35.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:36.068 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:36.097 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:36.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:36.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:36.198 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:36.220 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:36.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:36.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:36.322 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:36.349 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:36.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:36.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:36.451 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:36.481 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:36.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:36.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:36.583 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:36.594 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:36.606 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:36 smithi102 ceph-mon[110183]: pgmap v1022: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.0 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 364 op/s; 4823/98439 objects degraded (4.899%); 3901/98439 objects misplaced (3.963%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:44:36.606 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:44:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:44:36] "GET /metrics HTTP/1.1" 200 36547 "" "Prometheus/2.51.0" 2024-09-13T14:44:36.616 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:36.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:36.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:36.718 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:36.743 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:36.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:36.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:36.845 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:36.871 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:36.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:36.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:36.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:36 smithi102 ceph-mon[115269]: pgmap v1022: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.0 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 364 op/s; 4823/98439 objects degraded (4.899%); 3901/98439 objects misplaced (3.963%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:44:36.974 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:36.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:36 smithi195 ceph-mon[104262]: pgmap v1022: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 2.0 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 364 op/s; 4823/98439 objects degraded (4.899%); 3901/98439 objects misplaced (3.963%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:44:36.999 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:37.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:37.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:37.101 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:37.124 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:37.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:37.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:37.226 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:37.248 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:37.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:37.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:37.350 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:37.373 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:37.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:37.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:37.475 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:37.499 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:37.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:37.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:37.601 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:37.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:37.624 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:37.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:37.725 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:37.747 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:37.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:37.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:37.809 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:44:37.850 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:37.873 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:37.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:37.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:37.975 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:38.004 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:38.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:38.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:38.106 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:38.130 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:38.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:38.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:38.232 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:38.259 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:38.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:38.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:38.361 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:38.386 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:38.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:38.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:38.488 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:38.513 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:38.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:38.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:38.614 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:38.643 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:38.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:38.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:38.745 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:38.770 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:38.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:38.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:38.872 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:38.897 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:38.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:38.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:38.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:38 smithi102 ceph-mon[110183]: pgmap v1023: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.9 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 402 op/s; 4823/93210 objects degraded (5.174%); 3667/93210 objects misplaced (3.934%); 1.4 MiB/s, 23 objects/s recovering 2024-09-13T14:44:38.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:38 smithi102 ceph-mon[115269]: pgmap v1023: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.9 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 402 op/s; 4823/93210 objects degraded (5.174%); 3667/93210 objects misplaced (3.934%); 1.4 MiB/s, 23 objects/s recovering 2024-09-13T14:44:38.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:38 smithi195 ceph-mon[104262]: pgmap v1023: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.9 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 402 op/s; 4823/93210 objects degraded (5.174%); 3667/93210 objects misplaced (3.934%); 1.4 MiB/s, 23 objects/s recovering 2024-09-13T14:44:38.998 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:39.024 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:39.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:39.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:39.125 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:39.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:39.150 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:39.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:39.251 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:39.272 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:39.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:39.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:39.374 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:39.395 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:39.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:39.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:39.447 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:44:39.448 INFO:teuthology.orchestra.run.smithi102.stdout:{"fsid":"08cee502-71d9-11ef-bceb-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4823/93210 objects degraded (5.174%), 3 pgs degraded, 2 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1503,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":294,"num_osds":8,"num_up_osds":8,"osd_up_since":1726238267,"num_in_osds":6,"osd_in_since":1726238173,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":85},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering","count":1},{"state_name":"active+recovery_wait+degraded+remapped","count":1}],"num_pgs":89,"num_pools":5,"num_objects":31070,"data_bytes":2039093245,"bytes_used":11944579072,"bytes_avail":564025634816,"bytes_total":575970213888,"degraded_objects":4823,"degraded_total":93210,"degraded_ratio":0.051743375174337511,"misplaced_objects":3667,"misplaced_total":93210,"misplaced_ratio":0.039341272395665698,"recovering_objects_per_sec":23,"recovering_bytes_per_sec":1513494,"recovering_keys_per_sec":0,"num_objects_recovered":232,"num_bytes_recovered":15138816,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":402},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi102.njetxr","status":"up:active","gid":24521}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.102:8443/","prometheus":"http://172.21.15.102:9283/"}},"servicemap":{"epoch":28,"modified":"2024-09-13T14:40:21.476600+0000","services":{}},"progress_events":{}} 2024-09-13T14:44:39.497 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:39.522 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:39.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:39.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:39.623 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:39.649 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:39.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:39.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:39.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:39 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 4823/93210 objects degraded (5.174%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:39.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:39 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2842207348' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:44:39.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:39 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 4823/93210 objects degraded (5.174%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:39.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:39 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2842207348' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:44:39.686 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:44:39.687 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [], 2024-09-13T14:44:39.687 INFO:teuthology.orchestra.run.smithi102.stdout: "complaint_time": 30, 2024-09-13T14:44:39.687 INFO:teuthology.orchestra.run.smithi102.stdout: "num_blocked_ops": 0 2024-09-13T14:44:39.687 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:44:39.752 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:39.776 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:39.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:39.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:39.877 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:39.901 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:39.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:39.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:39.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:39 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 4823/93210 objects degraded (5.174%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:39.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:39 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2842207348' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:44:40.003 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:40.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:40.026 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:40.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:40.127 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:40.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:40.148 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:40.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:40.249 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:40.274 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:40.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:40.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:40.376 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:40.402 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:40.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:40.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:40.504 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:40.528 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:40.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:40.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:40.630 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:40.652 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:40.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:40.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:40.754 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:40.776 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:40.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:40.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:40.878 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:40.902 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:40.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:40.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:40.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:40 smithi102 ceph-mon[110183]: pgmap v1024: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.9 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 346 op/s; 4823/92877 objects degraded (5.193%); 3667/92877 objects misplaced (3.948%); 1.4 MiB/s, 23 objects/s recovering 2024-09-13T14:44:40.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:40 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:44:40.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:40 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:44:40.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:40 smithi102 ceph-mon[115269]: pgmap v1024: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.9 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 346 op/s; 4823/92877 objects degraded (5.193%); 3667/92877 objects misplaced (3.948%); 1.4 MiB/s, 23 objects/s recovering 2024-09-13T14:44:40.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:40 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:44:40.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:40 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:44:40.961 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:40 smithi195 ceph-mon[104262]: pgmap v1024: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.9 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 346 op/s; 4823/92877 objects degraded (5.193%); 3667/92877 objects misplaced (3.948%); 1.4 MiB/s, 23 objects/s recovering 2024-09-13T14:44:40.961 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:40 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:44:40.961 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:40 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:44:41.003 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:41.029 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:41.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:41.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:41.131 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:41.159 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:41.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:41.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:41.260 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:41.286 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:41.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:41.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:41.388 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:41.415 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:41.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:41.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:41.428 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-13T14:44:41.428 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:44:41.517 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:41.540 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:41.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:41.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:41.642 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:41.668 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:41.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:41.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:41.769 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:41.792 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:41.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:41.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:41.894 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:41.915 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:41.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:41.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:42.018 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:42.076 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T14:44:42.076 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:42.139 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:42.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:42.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:42.241 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:42.270 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:42.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:42.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:42.371 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:42.398 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:42.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:42.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:42.499 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:42.524 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:42.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:42.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:42.626 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:42.649 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:42.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:42.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:42.751 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:42.773 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:42.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:42.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:42.875 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:42.899 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:42.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:42.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:42.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:42 smithi102 ceph-mon[110183]: pgmap v1025: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.9 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 289 op/s; 4823/92877 objects degraded (5.193%); 3667/92877 objects misplaced (3.948%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T14:44:42.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:42 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:44:42.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:42 smithi102 ceph-mon[115269]: pgmap v1025: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.9 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 289 op/s; 4823/92877 objects degraded (5.193%); 3667/92877 objects misplaced (3.948%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T14:44:42.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:42 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:44:42.963 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:42 smithi195 ceph-mon[104262]: pgmap v1025: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.9 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 289 op/s; 4823/92877 objects degraded (5.193%); 3667/92877 objects misplaced (3.948%); 1.2 MiB/s, 19 objects/s recovering 2024-09-13T14:44:42.964 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:42 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:44:43.001 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:43.024 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:43.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:43.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:43.127 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:43.155 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:43.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:43.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:43.255 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:43.256 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:43.294 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:43.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:43.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:43.396 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:43.418 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:43.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:43.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:43.520 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:43.544 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:43.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:43.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:43.630 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is set 2024-09-13T14:44:43.646 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:43.669 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:43.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:43.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:43.770 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:43.793 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:43.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:43.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:43.894 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:43.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:43 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3818349485' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:44:43.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:43 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:44:43.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:43 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3818349485' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:44:43.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:43 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:44:43.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:43.919 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:43.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:43.960 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:43 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3818349485' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:44:43.961 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:43 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:44:44.020 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:44.045 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:44.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:44.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:44.146 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:44.172 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:44.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:44.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:44.274 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:44.296 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:44.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:44.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:44.398 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:44.419 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:44.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:44.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:44.521 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:44.542 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:44.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:44.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:44.644 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:44.666 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:44.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:44.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:44.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:44 smithi102 ceph-mon[110183]: pgmap v1026: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.8 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 463 op/s; 4823/86565 objects degraded (5.572%); 3410/86565 objects misplaced (3.939%); 1.8 MiB/s, 29 objects/s recovering 2024-09-13T14:44:44.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:44 smithi102 ceph-mon[110183]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:44:44.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:44 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:44:44.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:44 smithi102 ceph-mon[110183]: osdmap e295: 8 total, 8 up, 6 in 2024-09-13T14:44:44.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:44 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 4823/86565 objects degraded (5.572%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:44.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:44 smithi102 ceph-mon[115269]: pgmap v1026: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.8 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 463 op/s; 4823/86565 objects degraded (5.572%); 3410/86565 objects misplaced (3.939%); 1.8 MiB/s, 29 objects/s recovering 2024-09-13T14:44:44.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:44 smithi102 ceph-mon[115269]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:44:44.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:44 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:44:44.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:44 smithi102 ceph-mon[115269]: osdmap e295: 8 total, 8 up, 6 in 2024-09-13T14:44:44.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:44 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 4823/86565 objects degraded (5.572%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:44.769 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:44.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:44.792 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:44.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:44.893 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:44.916 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:44.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:44.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:44.983 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:44 smithi195 ceph-mon[104262]: pgmap v1026: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.8 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 463 op/s; 4823/86565 objects degraded (5.572%); 3410/86565 objects misplaced (3.939%); 1.8 MiB/s, 29 objects/s recovering 2024-09-13T14:44:44.983 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:44 smithi195 ceph-mon[104262]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:44:44.983 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:44 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:44:44.983 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:44 smithi195 ceph-mon[104262]: osdmap e295: 8 total, 8 up, 6 in 2024-09-13T14:44:44.983 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:44 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 4823/86565 objects degraded (5.572%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:45.019 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:45.041 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:45.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:45.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:45.143 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:45.168 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:45.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:45.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:45.231 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:44:45.231 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:44:45.231 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:44:45.231 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:44:45.231 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:44:45.231 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:44:45.231 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:44:45.231 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:44:45.232 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:44:45.232 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:44:45.232 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:44:45.232 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:44:45.232 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:44:45.232 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:44:45.232 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:44:45.232 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:44:45.269 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:45.297 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:45.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:45.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:45.399 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:45.422 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:45.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:45.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:45.524 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:45.549 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:45.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:45.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:45.651 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:45.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:45.677 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:45.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:45.778 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:45.800 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:45.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:45.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:45.901 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:45.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:45 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/4154653069' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:44:45.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:45 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/4154653069' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:44:45.924 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:45.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:45.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:45.978 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:45 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/4154653069' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:44:46.026 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:46.049 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:46.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:46.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:46.150 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:46.177 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:46.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:46.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:46.278 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:46.305 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:46.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:46.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:46.407 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:46.434 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:46.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:46.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:46.535 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:46.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:46.565 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:46.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:46.665 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:46.694 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:46.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:46.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:46.758 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:46 smithi102 ceph-mon[110183]: pgmap v1028: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.8 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 395 op/s; 4823/86565 objects degraded (5.572%); 3410/86565 objects misplaced (3.939%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:44:46.759 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:46 smithi102 ceph-mon[115269]: pgmap v1028: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.8 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 395 op/s; 4823/86565 objects degraded (5.572%); 3410/86565 objects misplaced (3.939%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:44:46.759 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:44:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:44:46] "GET /metrics HTTP/1.1" 200 36544 "" "Prometheus/2.51.0" 2024-09-13T14:44:46.795 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:46.821 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:46.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:46.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:46.879 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:46 smithi195 ceph-mon[104262]: pgmap v1028: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.8 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 395 op/s; 4823/86565 objects degraded (5.572%); 3410/86565 objects misplaced (3.939%); 1.5 MiB/s, 24 objects/s recovering 2024-09-13T14:44:46.923 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:46.947 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:46.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:46.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:47.049 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:47.073 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:47.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:47.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:47.175 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:47.200 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:47.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:47.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:47.302 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:47.328 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:47.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:47.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:47.430 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:47.450 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:47.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:47.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:47.504 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:47.553 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:47.574 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:47.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:47.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:47.676 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:47.699 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:47.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:47.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:47.801 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:47.823 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:47.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:47.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:47.928 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:47.953 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:47.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:47.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:48.026 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:44:48.055 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:48.077 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:48.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:48.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:48.179 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:48.200 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:48.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:48.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:48.302 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:48.313 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:48 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:44:48.313 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:48 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:44:48.313 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:48 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:44:48.313 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:48 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:44:48.328 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:48.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:48.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:48.430 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:48.453 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:48.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:48.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:48.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:48 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:44:48.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:48 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:44:48.555 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:48.577 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:48.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:48.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:48.679 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:48.705 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:48.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:48.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:48.729 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:44:48.730 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:44:48.788 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:44:48.807 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:48.828 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:48.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:48.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:48.930 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:48.955 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:48.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:48.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:49.058 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:49.080 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:49.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:49.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:49.182 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:49.205 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:49.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:49.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:49.307 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:49.330 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:49.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:49.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:49.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:49 smithi102 ceph-mon[110183]: pgmap v1029: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.7 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 402 op/s; 4823/81144 objects degraded (5.944%); 3171/81144 objects misplaced (3.908%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T14:44:49.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:49 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 4823/81144 objects degraded (5.944%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:49.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:49 smithi102 ceph-mon[115269]: pgmap v1029: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.7 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 402 op/s; 4823/81144 objects degraded (5.944%); 3171/81144 objects misplaced (3.908%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T14:44:49.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:49 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 4823/81144 objects degraded (5.944%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:49.432 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:49.457 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:49.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:49.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:49.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:49 smithi195 ceph-mon[104262]: pgmap v1029: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.7 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 402 op/s; 4823/81144 objects degraded (5.944%); 3171/81144 objects misplaced (3.908%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T14:44:49.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:49 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 4823/81144 objects degraded (5.944%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:49.560 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:49.586 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:49.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:49.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:49.688 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:49.709 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:49.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:49.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:49.811 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:49.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:49.833 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:49.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:49.934 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:49.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:49.957 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:49.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:50.058 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:50.082 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:50.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:50.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:50.131 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:50.184 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:50.213 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:50.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:50.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:50.315 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:50.344 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:50.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:50.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:50.446 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:50.469 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:50.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:50.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:50.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:50 smithi195 ceph-mon[104262]: from='client.36102 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:44:50.571 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:50.594 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:50.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:50.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:50.660 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:50 smithi102 ceph-mon[110183]: from='client.36102 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:44:50.660 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:50 smithi102 ceph-mon[115269]: from='client.36102 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:44:50.695 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:50.718 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:50.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:50.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:50.820 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:50.842 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:50.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:50.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:50.944 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:50.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:50.967 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:50.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:51.069 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:51.092 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:51.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:51.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:51.194 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:51.220 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:51.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:51.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:51.323 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:51.353 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:51.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:51.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:51.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:51 smithi102 ceph-mon[110183]: pgmap v1030: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.6 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 402 op/s; 4823/80793 objects degraded (5.970%); 3171/80793 objects misplaced (3.925%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T14:44:51.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:51 smithi102 ceph-mon[115269]: pgmap v1030: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.6 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 402 op/s; 4823/80793 objects degraded (5.970%); 3171/80793 objects misplaced (3.925%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T14:44:51.456 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:51.481 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:51.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:51.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:51.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:51 smithi195 ceph-mon[104262]: pgmap v1030: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.6 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 402 op/s; 4823/80793 objects degraded (5.970%); 3171/80793 objects misplaced (3.925%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T14:44:51.502 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":1029,"stamp":"2024-09-13T14:44:47.547711+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1775507453,"num_objects":27048,"num_object_clones":0,"num_object_copies":81144,"num_objects_missing_on_primary":78,"num_objects_missing":78,"num_objects_degraded":4823,"num_objects_misplaced":3171,"num_objects_unfound":0,"num_objects_dirty":27048,"num_whiteouts":0,"num_read":6377,"num_read_kb":13834,"num_write":199729,"num_write_kb":7235010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85656,"num_bytes_recovered":5484711053,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96111,"ondisk_log_size":96111,"up":267,"acting":265,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":299,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":10887984,"kb_used_data":7527680,"kb_used_omap":26,"kb_used_meta":3360165,"kb_avail":551582928,"statfs":{"total":575970213888,"available":564820918272,"internally_reserved":0,"allocated":7708344320,"data_stored":7698945727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27144,"internal_metadata":3440809464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,1,1,1,2,3,9,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-263585792,"num_objects":-4022,"num_object_clones":0,"num_object_copies":-12066,"num_objects_missing_on_primary":-1,"num_objects_missing":-1,"num_objects_degraded":0,"num_objects_misplaced":-496,"num_objects_unfound":0,"num_objects_dirty":-4022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4022,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":253,"num_bytes_recovered":16515072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002219"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":484,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.651158+0000","last_change":"2024-09-13T14:36:13.859927+0000","last_active":"2024-09-13T14:44:43.651158+0000","last_peered":"2024-09-13T14:44:43.651158+0000","last_clean":"2024-09-13T14:44:43.651158+0000","last_became_active":"2024-09-13T14:36:13.859505+0000","last_became_peered":"2024-09-13T14:36:13.859505+0000","last_unstale":"2024-09-13T14:44:43.651158+0000","last_undegraded":"2024-09-13T14:44:43.651158+0000","last_fullsized":"2024-09-13T14:44:43.651158+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:12:20.989627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":298,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-13T14:44:29.621591+0000","last_change":"2024-09-13T14:37:48.130968+0000","last_active":"2024-09-13T14:44:29.621591+0000","last_peered":"2024-09-13T14:44:29.621591+0000","last_clean":"2024-09-13T14:44:29.621591+0000","last_became_active":"2024-09-13T14:37:48.130603+0000","last_became_peered":"2024-09-13T14:37:48.130603+0000","last_unstale":"2024-09-13T14:44:29.621591+0000","last_undegraded":"2024-09-13T14:44:29.621591+0000","last_fullsized":"2024-09-13T14:44:29.621591+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:42.626386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":94,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-13T14:44:29.607304+0000","last_change":"2024-09-13T14:36:13.862181+0000","last_active":"2024-09-13T14:44:29.607304+0000","last_peered":"2024-09-13T14:44:29.607304+0000","last_clean":"2024-09-13T14:44:29.607304+0000","last_became_active":"2024-09-13T14:36:13.861873+0000","last_became_peered":"2024-09-13T14:36:13.861873+0000","last_unstale":"2024-09-13T14:44:29.607304+0000","last_undegraded":"2024-09-13T14:44:29.607304+0000","last_fullsized":"2024-09-13T14:44:29.607304+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:14.085700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":121,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.635908+0000","last_change":"2024-09-13T14:38:53.896672+0000","last_active":"2024-09-13T14:44:43.635908+0000","last_peered":"2024-09-13T14:44:43.635908+0000","last_clean":"2024-09-13T14:44:43.635908+0000","last_became_active":"2024-09-13T14:38:53.896346+0000","last_became_peered":"2024-09-13T14:38:53.896346+0000","last_unstale":"2024-09-13T14:44:43.635908+0000","last_undegraded":"2024-09-13T14:44:43.635908+0000","last_fullsized":"2024-09-13T14:44:43.635908+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:57.789686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":121,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.635760+0000","last_change":"2024-09-13T14:38:53.893021+0000","last_active":"2024-09-13T14:44:43.635760+0000","last_peered":"2024-09-13T14:44:43.635760+0000","last_clean":"2024-09-13T14:44:43.635760+0000","last_became_active":"2024-09-13T14:38:53.892668+0000","last_became_peered":"2024-09-13T14:38:53.892668+0000","last_unstale":"2024-09-13T14:44:43.635760+0000","last_undegraded":"2024-09-13T14:44:43.635760+0000","last_fullsized":"2024-09-13T14:44:43.635760+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:57.180738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":266,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-13T14:44:29.621153+0000","last_change":"2024-09-13T14:37:48.134642+0000","last_active":"2024-09-13T14:44:29.621153+0000","last_peered":"2024-09-13T14:44:29.621153+0000","last_clean":"2024-09-13T14:44:29.621153+0000","last_became_active":"2024-09-13T14:37:48.134402+0000","last_became_peered":"2024-09-13T14:37:48.134402+0000","last_unstale":"2024-09-13T14:44:29.621153+0000","last_undegraded":"2024-09-13T14:44:29.621153+0000","last_fullsized":"2024-09-13T14:44:29.621153+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:51.313663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":503,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.651198+0000","last_change":"2024-09-13T14:38:53.903022+0000","last_active":"2024-09-13T14:44:43.651198+0000","last_peered":"2024-09-13T14:44:43.651198+0000","last_clean":"2024-09-13T14:44:43.651198+0000","last_became_active":"2024-09-13T14:38:53.902448+0000","last_became_peered":"2024-09-13T14:38:53.902448+0000","last_unstale":"2024-09-13T14:44:43.651198+0000","last_undegraded":"2024-09-13T14:44:43.651198+0000","last_fullsized":"2024-09-13T14:44:43.651198+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:24.690401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":124,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.635070+0000","last_change":"2024-09-13T14:38:03.554500+0000","last_active":"2024-09-13T14:44:43.635070+0000","last_peered":"2024-09-13T14:44:43.635070+0000","last_clean":"2024-09-13T14:44:43.635070+0000","last_became_active":"2024-09-13T14:38:03.532910+0000","last_became_peered":"2024-09-13T14:38:03.532910+0000","last_unstale":"2024-09-13T14:44:43.635070+0000","last_undegraded":"2024-09-13T14:44:43.635070+0000","last_fullsized":"2024-09-13T14:44:43.635070+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:27:50.843498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"295'12256","reported_seq":23163,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:47.267508+0000","last_change":"2024-09-13T14:41:53.220083+0000","last_active":"2024-09-13T14:44:47.267508+0000","last_peered":"2024-09-13T14:44:47.267508+0000","last_clean":"2024-09-13T14:44:47.267508+0000","last_became_active":"2024-09-13T14:41:53.200084+0000","last_became_peered":"2024-09-13T14:41:53.200084+0000","last_unstale":"2024-09-13T14:44:47.267508+0000","last_undegraded":"2024-09-13T14:44:47.267508+0000","last_fullsized":"2024-09-13T14:44:47.267508+0000","mapping_epoch":275,"log_start":"220'6500","ondisk_log_start":"220'6500","created":205,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5756,"log_dups_size":0,"ondisk_log_size":5756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:08.897524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106430464,"num_objects":1624,"num_object_clones":0,"num_object_copies":4872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12256,"num_write_kb":444160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9152,"num_bytes_recovered":594608128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":58,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-13T14:44:29.621499+0000","last_change":"2024-09-13T14:38:53.940238+0000","last_active":"2024-09-13T14:44:29.621499+0000","last_peered":"2024-09-13T14:44:29.621499+0000","last_clean":"2024-09-13T14:44:29.621499+0000","last_became_active":"2024-09-13T14:38:53.892825+0000","last_became_peered":"2024-09-13T14:38:53.892825+0000","last_unstale":"2024-09-13T14:44:29.621499+0000","last_undegraded":"2024-09-13T14:44:29.621499+0000","last_fullsized":"2024-09-13T14:44:29.621499+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:46.666581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":114,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-13T14:44:29.606808+0000","last_change":"2024-09-13T14:38:53.898907+0000","last_active":"2024-09-13T14:44:29.606808+0000","last_peered":"2024-09-13T14:44:29.606808+0000","last_clean":"2024-09-13T14:44:29.606808+0000","last_became_active":"2024-09-13T14:38:53.898572+0000","last_became_peered":"2024-09-13T14:38:53.898572+0000","last_unstale":"2024-09-13T14:44:29.606808+0000","last_undegraded":"2024-09-13T14:44:29.606808+0000","last_fullsized":"2024-09-13T14:44:29.606808+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:29.222150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"295'12561","reported_seq":26762,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:47.383345+0000","last_change":"2024-09-13T14:38:57.930032+0000","last_active":"2024-09-13T14:44:47.383345+0000","last_peered":"2024-09-13T14:44:47.383345+0000","last_clean":"2024-09-13T14:44:47.383345+0000","last_became_active":"2024-09-13T14:38:57.921458+0000","last_became_peered":"2024-09-13T14:38:57.921458+0000","last_unstale":"2024-09-13T14:44:47.383345+0000","last_undegraded":"2024-09-13T14:44:47.383345+0000","last_fullsized":"2024-09-13T14:44:47.383345+0000","mapping_epoch":256,"log_start":"220'6839","ondisk_log_start":"220'6839","created":205,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5722,"log_dups_size":0,"ondisk_log_size":5722,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:27:12.294038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110166016,"num_objects":1681,"num_object_clones":0,"num_object_copies":5043,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12561,"num_write_kb":455744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6990,"num_bytes_recovered":458096640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":58,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-13T14:44:29.623029+0000","last_change":"2024-09-13T14:38:53.904885+0000","last_active":"2024-09-13T14:44:29.623029+0000","last_peered":"2024-09-13T14:44:29.623029+0000","last_clean":"2024-09-13T14:44:29.623029+0000","last_became_active":"2024-09-13T14:38:53.891726+0000","last_became_peered":"2024-09-13T14:38:53.891726+0000","last_unstale":"2024-09-13T14:44:29.623029+0000","last_undegraded":"2024-09-13T14:44:29.623029+0000","last_fullsized":"2024-09-13T14:44:29.623029+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:57.079205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":303,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:16.633707+0000","last_change":"2024-09-13T14:37:48.160504+0000","last_active":"2024-09-13T14:44:16.633707+0000","last_peered":"2024-09-13T14:44:16.633707+0000","last_clean":"2024-09-13T14:44:16.633707+0000","last_became_active":"2024-09-13T14:37:48.160181+0000","last_became_peered":"2024-09-13T14:37:48.160181+0000","last_unstale":"2024-09-13T14:44:16.633707+0000","last_undegraded":"2024-09-13T14:44:16.633707+0000","last_fullsized":"2024-09-13T14:44:16.633707+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:37.087008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"295'12482","reported_seq":14304,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:47.075132+0000","last_change":"2024-09-13T14:38:04.531914+0000","last_active":"2024-09-13T14:44:47.075132+0000","last_peered":"2024-09-13T14:44:47.075132+0000","last_clean":"2024-09-13T14:44:47.075132+0000","last_became_active":"2024-09-13T14:38:04.523448+0000","last_became_peered":"2024-09-13T14:38:04.523448+0000","last_unstale":"2024-09-13T14:44:47.075132+0000","last_undegraded":"2024-09-13T14:44:47.075132+0000","last_fullsized":"2024-09-13T14:44:47.075132+0000","mapping_epoch":245,"log_start":"220'6500","ondisk_log_start":"220'6500","created":205,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5982,"log_dups_size":0,"ondisk_log_size":5982,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:06.536896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110886912,"num_objects":1692,"num_object_clones":0,"num_object_copies":5076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12482,"num_write_kb":453568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":811,"num_bytes_recovered":53149696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":295,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-13T14:44:29.621414+0000","last_change":"2024-09-13T14:37:49.611774+0000","last_active":"2024-09-13T14:44:29.621414+0000","last_peered":"2024-09-13T14:44:29.621414+0000","last_clean":"2024-09-13T14:44:29.621414+0000","last_became_active":"2024-09-13T14:37:49.611088+0000","last_became_peered":"2024-09-13T14:37:49.611088+0000","last_unstale":"2024-09-13T14:44:29.621414+0000","last_undegraded":"2024-09-13T14:44:29.621414+0000","last_fullsized":"2024-09-13T14:44:29.621414+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:08.933213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":54,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-13T14:44:29.621136+0000","last_change":"2024-09-13T14:38:53.899211+0000","last_active":"2024-09-13T14:44:29.621136+0000","last_peered":"2024-09-13T14:44:29.621136+0000","last_clean":"2024-09-13T14:44:29.621136+0000","last_became_active":"2024-09-13T14:38:53.898927+0000","last_became_peered":"2024-09-13T14:38:53.898927+0000","last_unstale":"2024-09-13T14:44:29.621136+0000","last_undegraded":"2024-09-13T14:44:29.621136+0000","last_fullsized":"2024-09-13T14:44:29.621136+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:00:28.857611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"295'12338","reported_seq":12441,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:47.363471+0000","last_change":"2024-09-13T14:35:24.004651+0000","last_active":"2024-09-13T14:44:47.363471+0000","last_peered":"2024-09-13T14:44:47.363471+0000","last_clean":"2024-09-13T14:44:47.363471+0000","last_became_active":"2024-09-13T14:35:24.004358+0000","last_became_peered":"2024-09-13T14:35:24.004358+0000","last_unstale":"2024-09-13T14:44:47.363471+0000","last_undegraded":"2024-09-13T14:44:47.363471+0000","last_fullsized":"2024-09-13T14:44:47.363471+0000","mapping_epoch":205,"log_start":"219'6084","ondisk_log_start":"219'6084","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6254,"log_dups_size":0,"ondisk_log_size":6254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:57:38.023669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107085824,"num_objects":1634,"num_object_clones":0,"num_object_copies":4902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12338,"num_write_kb":447104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":105,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-13T14:44:29.607329+0000","last_change":"2024-09-13T14:37:49.168003+0000","last_active":"2024-09-13T14:44:29.607329+0000","last_peered":"2024-09-13T14:44:29.607329+0000","last_clean":"2024-09-13T14:44:29.607329+0000","last_became_active":"2024-09-13T14:37:49.167638+0000","last_became_peered":"2024-09-13T14:37:49.167638+0000","last_unstale":"2024-09-13T14:44:29.607329+0000","last_undegraded":"2024-09-13T14:44:29.607329+0000","last_fullsized":"2024-09-13T14:44:29.607329+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:29.051981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":114,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.635628+0000","last_change":"2024-09-13T14:38:03.528381+0000","last_active":"2024-09-13T14:44:43.635628+0000","last_peered":"2024-09-13T14:44:43.635628+0000","last_clean":"2024-09-13T14:44:43.635628+0000","last_became_active":"2024-09-13T14:38:03.527282+0000","last_became_peered":"2024-09-13T14:38:03.527282+0000","last_unstale":"2024-09-13T14:44:43.635628+0000","last_undegraded":"2024-09-13T14:44:43.635628+0000","last_fullsized":"2024-09-13T14:44:43.635628+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:24.435067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.f","version":"295'12357","reported_seq":12674,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:47.085432+0000","last_change":"2024-09-13T14:38:16.371079+0000","last_active":"2024-09-13T14:44:47.085432+0000","last_peered":"2024-09-13T14:44:47.085432+0000","last_clean":"2024-09-13T14:44:47.085432+0000","last_became_active":"2024-09-13T14:38:12.859883+0000","last_became_peered":"2024-09-13T14:38:12.859883+0000","last_unstale":"2024-09-13T14:44:47.085432+0000","last_undegraded":"2024-09-13T14:44:47.085432+0000","last_fullsized":"2024-09-13T14:44:47.085432+0000","mapping_epoch":245,"log_start":"220'6463","ondisk_log_start":"220'6463","created":205,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5894,"log_dups_size":0,"ondisk_log_size":5894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:08.436532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113311744,"num_objects":1729,"num_object_clones":0,"num_object_copies":5187,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12357,"num_write_kb":450752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":3670016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":472,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.650933+0000","last_change":"2024-09-13T14:38:03.549595+0000","last_active":"2024-09-13T14:44:43.650933+0000","last_peered":"2024-09-13T14:44:43.650933+0000","last_clean":"2024-09-13T14:44:43.650933+0000","last_became_active":"2024-09-13T14:38:03.533331+0000","last_became_peered":"2024-09-13T14:38:03.533331+0000","last_unstale":"2024-09-13T14:44:43.650933+0000","last_undegraded":"2024-09-13T14:44:43.650933+0000","last_fullsized":"2024-09-13T14:44:43.650933+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:59:02.470343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":278,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:16.634571+0000","last_change":"2024-09-13T14:37:48.127600+0000","last_active":"2024-09-13T14:44:16.634571+0000","last_peered":"2024-09-13T14:44:16.634571+0000","last_clean":"2024-09-13T14:44:16.634571+0000","last_became_active":"2024-09-13T14:37:48.127349+0000","last_became_peered":"2024-09-13T14:37:48.127349+0000","last_unstale":"2024-09-13T14:44:16.634571+0000","last_undegraded":"2024-09-13T14:44:16.634571+0000","last_fullsized":"2024-09-13T14:44:16.634571+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:57.268374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"295'12330","reported_seq":22330,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:47.297831+0000","last_change":"2024-09-13T14:42:30.227259+0000","last_active":"2024-09-13T14:44:47.297831+0000","last_peered":"2024-09-13T14:44:47.297831+0000","last_clean":"2024-09-13T14:44:47.297831+0000","last_became_active":"2024-09-13T14:42:30.209102+0000","last_became_peered":"2024-09-13T14:42:30.209102+0000","last_unstale":"2024-09-13T14:44:47.297831+0000","last_undegraded":"2024-09-13T14:44:47.297831+0000","last_fullsized":"2024-09-13T14:44:47.297831+0000","mapping_epoch":281,"log_start":"220'6600","ondisk_log_start":"220'6600","created":205,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5730,"log_dups_size":0,"ondisk_log_size":5730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:56:16.193287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110100480,"num_objects":1680,"num_object_clones":0,"num_object_copies":5040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12330,"num_write_kb":448320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4926,"num_bytes_recovered":313851904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":211,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:16.634242+0000","last_change":"2024-09-13T14:37:48.127604+0000","last_active":"2024-09-13T14:44:16.634242+0000","last_peered":"2024-09-13T14:44:16.634242+0000","last_clean":"2024-09-13T14:44:16.634242+0000","last_became_active":"2024-09-13T14:37:48.127300+0000","last_became_peered":"2024-09-13T14:37:48.127300+0000","last_unstale":"2024-09-13T14:44:16.634242+0000","last_undegraded":"2024-09-13T14:44:16.634242+0000","last_fullsized":"2024-09-13T14:44:16.634242+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:51.999506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":331,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:16.633297+0000","last_change":"2024-09-13T14:38:03.529967+0000","last_active":"2024-09-13T14:44:16.633297+0000","last_peered":"2024-09-13T14:44:16.633297+0000","last_clean":"2024-09-13T14:44:16.633297+0000","last_became_active":"2024-09-13T14:38:03.529646+0000","last_became_peered":"2024-09-13T14:38:03.529646+0000","last_unstale":"2024-09-13T14:44:16.633297+0000","last_undegraded":"2024-09-13T14:44:16.633297+0000","last_fullsized":"2024-09-13T14:44:16.633297+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:52:05.193279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"295'12407","reported_seq":21600,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:47.042396+0000","last_change":"2024-09-13T14:40:34.583237+0000","last_active":"2024-09-13T14:44:47.042396+0000","last_peered":"2024-09-13T14:44:47.042396+0000","last_clean":"2024-09-13T14:44:47.042396+0000","last_became_active":"2024-09-13T14:40:34.563989+0000","last_became_peered":"2024-09-13T14:40:34.563989+0000","last_unstale":"2024-09-13T14:44:47.042396+0000","last_undegraded":"2024-09-13T14:44:47.042396+0000","last_fullsized":"2024-09-13T14:44:47.042396+0000","mapping_epoch":266,"log_start":"220'6500","ondisk_log_start":"220'6500","created":205,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5907,"log_dups_size":0,"ondisk_log_size":5907,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:34:43.006051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113508376,"num_objects":1733,"num_object_clones":0,"num_object_copies":5199,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1733,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12407,"num_write_kb":452417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4489,"num_bytes_recovered":289865752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":292,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:16.633946+0000","last_change":"2024-09-13T14:36:13.860367+0000","last_active":"2024-09-13T14:44:16.633946+0000","last_peered":"2024-09-13T14:44:16.633946+0000","last_clean":"2024-09-13T14:44:16.633946+0000","last_became_active":"2024-09-13T14:36:13.859982+0000","last_became_peered":"2024-09-13T14:36:13.859982+0000","last_unstale":"2024-09-13T14:44:16.633946+0000","last_undegraded":"2024-09-13T14:44:16.633946+0000","last_fullsized":"2024-09-13T14:44:16.633946+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:08:23.311350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":287,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:16.633577+0000","last_change":"2024-09-13T14:37:48.147360+0000","last_active":"2024-09-13T14:44:16.633577+0000","last_peered":"2024-09-13T14:44:16.633577+0000","last_clean":"2024-09-13T14:44:16.633577+0000","last_became_active":"2024-09-13T14:37:48.146885+0000","last_became_peered":"2024-09-13T14:37:48.146885+0000","last_unstale":"2024-09-13T14:44:16.633577+0000","last_undegraded":"2024-09-13T14:44:16.633577+0000","last_fullsized":"2024-09-13T14:44:16.633577+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:27.841810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"295'12431","reported_seq":21731,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:47.434097+0000","last_change":"2024-09-13T14:38:39.650332+0000","last_active":"2024-09-13T14:44:47.434097+0000","last_peered":"2024-09-13T14:44:47.434097+0000","last_clean":"2024-09-13T14:44:47.434097+0000","last_became_active":"2024-09-13T14:38:39.640835+0000","last_became_peered":"2024-09-13T14:38:39.640835+0000","last_unstale":"2024-09-13T14:44:47.434097+0000","last_undegraded":"2024-09-13T14:44:47.434097+0000","last_fullsized":"2024-09-13T14:44:47.434097+0000","mapping_epoch":251,"log_start":"220'6751","ondisk_log_start":"220'6751","created":205,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5680,"log_dups_size":0,"ondisk_log_size":5680,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:48.176606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107675648,"num_objects":1643,"num_object_clones":0,"num_object_copies":4929,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12431,"num_write_kb":450368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4544,"num_bytes_recovered":292945920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":491,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.651087+0000","last_change":"2024-09-13T14:38:53.900041+0000","last_active":"2024-09-13T14:44:43.651087+0000","last_peered":"2024-09-13T14:44:43.651087+0000","last_clean":"2024-09-13T14:44:43.651087+0000","last_became_active":"2024-09-13T14:38:53.899203+0000","last_became_peered":"2024-09-13T14:38:53.899203+0000","last_unstale":"2024-09-13T14:44:43.651087+0000","last_undegraded":"2024-09-13T14:44:43.651087+0000","last_fullsized":"2024-09-13T14:44:43.651087+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:28:07.883094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":423,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.651375+0000","last_change":"2024-09-13T14:26:15.823198+0000","last_active":"2024-09-13T14:44:43.651375+0000","last_peered":"2024-09-13T14:44:43.651375+0000","last_clean":"2024-09-13T14:44:43.651375+0000","last_became_active":"2024-09-13T14:26:15.822717+0000","last_became_peered":"2024-09-13T14:26:15.822717+0000","last_unstale":"2024-09-13T14:44:43.651375+0000","last_undegraded":"2024-09-13T14:44:43.651375+0000","last_fullsized":"2024-09-13T14:44:43.651375+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:22.471622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.6","version":"295'12501","reported_seq":22163,"reported_epoch":295,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-13T14:44:47.126011+0000","last_change":"2024-09-13T14:38:55.132683+0000","last_active":"2024-09-13T14:44:47.126011+0000","last_peered":"2024-09-13T14:44:47.126011+0000","last_clean":"2024-09-13T14:36:13.124368+0000","last_became_active":"2024-09-13T14:38:55.120883+0000","last_became_peered":"2024-09-13T14:38:55.120883+0000","last_unstale":"2024-09-13T14:44:47.126011+0000","last_undegraded":"2024-09-13T14:38:55.026610+0000","last_fullsized":"2024-09-13T14:44:47.126011+0000","mapping_epoch":254,"log_start":"220'6519","ondisk_log_start":"220'6519","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5982,"log_dups_size":0,"ondisk_log_size":5982,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111214592,"num_objects":1697,"num_object_clones":0,"num_object_copies":5091,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":279,"num_objects_misplaced":1418,"num_objects_unfound":0,"num_objects_dirty":1697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12501,"num_write_kb":454336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4726,"num_bytes_recovered":298516480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,3,5],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3,5","objects":1697}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":429,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-13T14:44:29.621118+0000","last_change":"2024-09-13T14:41:53.196835+0000","last_active":"2024-09-13T14:44:29.621118+0000","last_peered":"2024-09-13T14:44:29.621118+0000","last_clean":"2024-09-13T14:44:29.621118+0000","last_became_active":"2024-09-13T14:41:53.196387+0000","last_became_peered":"2024-09-13T14:41:53.196387+0000","last_unstale":"2024-09-13T14:44:29.621118+0000","last_undegraded":"2024-09-13T14:44:29.621118+0000","last_fullsized":"2024-09-13T14:44:29.621118+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:07:41.927740+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:49.972303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":535,"reported_epoch":294,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:44:37.313124+0000","last_change":"2024-09-13T14:38:54.904329+0000","last_active":"2024-09-13T14:44:37.313124+0000","last_peered":"2024-09-13T14:44:37.313124+0000","last_clean":"2024-09-13T14:38:39.648017+0000","last_became_active":"2024-09-13T14:38:54.903798+0000","last_became_peered":"2024-09-13T14:38:54.903798+0000","last_unstale":"2024-09-13T14:44:37.313124+0000","last_undegraded":"2024-09-13T14:38:54.890505+0000","last_fullsized":"2024-09-13T14:38:54.889701+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:18.120931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,3],"avail_no_missing":["5","3","6"],"object_location_counts":[{"shards":"3,5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":425,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.651116+0000","last_change":"2024-09-13T14:37:49.160719+0000","last_active":"2024-09-13T14:44:43.651116+0000","last_peered":"2024-09-13T14:44:43.651116+0000","last_clean":"2024-09-13T14:44:43.651116+0000","last_became_active":"2024-09-13T14:37:49.160401+0000","last_became_peered":"2024-09-13T14:37:49.160401+0000","last_unstale":"2024-09-13T14:44:43.651116+0000","last_undegraded":"2024-09-13T14:44:43.651116+0000","last_fullsized":"2024-09-13T14:44:43.651116+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:21.737945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":139,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:22.876114+0000","last_change":"2024-09-13T14:37:48.172940+0000","last_active":"2024-09-13T14:44:22.876114+0000","last_peered":"2024-09-13T14:44:22.876114+0000","last_clean":"2024-09-13T14:44:22.876114+0000","last_became_active":"2024-09-13T14:37:48.172060+0000","last_became_peered":"2024-09-13T14:37:48.172060+0000","last_unstale":"2024-09-13T14:44:22.876114+0000","last_undegraded":"2024-09-13T14:44:22.876114+0000","last_fullsized":"2024-09-13T14:44:22.876114+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:07.480645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.5","version":"295'12482","reported_seq":28861,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:47.398629+0000","last_change":"2024-09-13T14:39:34.446699+0000","last_active":"2024-09-13T14:44:47.398629+0000","last_peered":"2024-09-13T14:44:47.398629+0000","last_clean":"2024-09-13T14:44:47.398629+0000","last_became_active":"2024-09-13T14:38:04.600616+0000","last_became_peered":"2024-09-13T14:38:04.600616+0000","last_unstale":"2024-09-13T14:44:47.398629+0000","last_undegraded":"2024-09-13T14:44:47.398629+0000","last_fullsized":"2024-09-13T14:44:47.398629+0000","mapping_epoch":245,"log_start":"219'6216","ondisk_log_start":"219'6216","created":205,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6266,"log_dups_size":0,"ondisk_log_size":6266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:00:44.321678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108789760,"num_objects":1660,"num_object_clones":0,"num_object_copies":4980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12482,"num_write_kb":452544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8120,"num_bytes_recovered":521076736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":513,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-13T14:44:29.621081+0000","last_change":"2024-09-13T14:26:15.835178+0000","last_active":"2024-09-13T14:44:29.621081+0000","last_peered":"2024-09-13T14:44:29.621081+0000","last_clean":"2024-09-13T14:44:29.621081+0000","last_became_active":"2024-09-13T14:26:15.820283+0000","last_became_peered":"2024-09-13T14:26:15.820283+0000","last_unstale":"2024-09-13T14:44:29.621081+0000","last_undegraded":"2024-09-13T14:44:29.621081+0000","last_fullsized":"2024-09-13T14:44:29.621081+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:30.340038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":115,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.635146+0000","last_change":"2024-09-13T14:38:03.525682+0000","last_active":"2024-09-13T14:44:43.635146+0000","last_peered":"2024-09-13T14:44:43.635146+0000","last_clean":"2024-09-13T14:44:43.635146+0000","last_became_active":"2024-09-13T14:38:03.525406+0000","last_became_peered":"2024-09-13T14:38:03.525406+0000","last_unstale":"2024-09-13T14:44:43.635146+0000","last_undegraded":"2024-09-13T14:44:43.635146+0000","last_fullsized":"2024-09-13T14:44:43.635146+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:41:28.970300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":122,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.635870+0000","last_change":"2024-09-13T14:38:53.991738+0000","last_active":"2024-09-13T14:44:43.635870+0000","last_peered":"2024-09-13T14:44:43.635870+0000","last_clean":"2024-09-13T14:44:43.635870+0000","last_became_active":"2024-09-13T14:38:53.991417+0000","last_became_peered":"2024-09-13T14:38:53.991417+0000","last_unstale":"2024-09-13T14:44:43.635870+0000","last_undegraded":"2024-09-13T14:44:43.635870+0000","last_fullsized":"2024-09-13T14:44:43.635870+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:50:38.559838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"295'12353","reported_seq":31899,"reported_epoch":295,"state":"active+recovering","last_fresh":"2024-09-13T14:44:47.126408+0000","last_change":"2024-09-13T14:43:50.028785+0000","last_active":"2024-09-13T14:44:47.126408+0000","last_peered":"2024-09-13T14:44:47.126408+0000","last_clean":"2024-09-13T14:37:46.290754+0000","last_became_active":"2024-09-13T14:38:17.381753+0000","last_became_peered":"2024-09-13T14:38:17.381753+0000","last_unstale":"2024-09-13T14:44:47.126408+0000","last_undegraded":"2024-09-13T14:44:47.126408+0000","last_fullsized":"2024-09-13T14:44:47.126408+0000","mapping_epoch":245,"log_start":"220'6459","ondisk_log_start":"220'6459","created":205,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5894,"log_dups_size":0,"ondisk_log_size":5894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114884608,"num_objects":1753,"num_object_clones":0,"num_object_copies":5259,"num_objects_missing_on_primary":78,"num_objects_missing":78,"num_objects_degraded":0,"num_objects_misplaced":1753,"num_objects_unfound":0,"num_objects_dirty":1753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12353,"num_write_kb":451392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9718,"num_bytes_recovered":636551168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"0,4,5","objects":1753}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":836,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.651119+0000","last_change":"2024-09-13T14:37:48.131799+0000","last_active":"2024-09-13T14:44:43.651119+0000","last_peered":"2024-09-13T14:44:43.651119+0000","last_clean":"2024-09-13T14:44:43.651119+0000","last_became_active":"2024-09-13T14:37:48.130837+0000","last_became_peered":"2024-09-13T14:37:48.130837+0000","last_unstale":"2024-09-13T14:44:43.651119+0000","last_undegraded":"2024-09-13T14:44:43.651119+0000","last_fullsized":"2024-09-13T14:44:43.651119+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:06:32.663770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":355,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.635250+0000","last_change":"2024-09-13T14:37:49.301531+0000","last_active":"2024-09-13T14:44:43.635250+0000","last_peered":"2024-09-13T14:44:43.635250+0000","last_clean":"2024-09-13T14:44:43.635250+0000","last_became_active":"2024-09-13T14:37:49.301249+0000","last_became_peered":"2024-09-13T14:37:49.301249+0000","last_unstale":"2024-09-13T14:44:43.635250+0000","last_undegraded":"2024-09-13T14:44:43.635250+0000","last_fullsized":"2024-09-13T14:44:43.635250+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:10.977710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":121,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-13T14:44:29.607535+0000","last_change":"2024-09-13T14:38:53.901427+0000","last_active":"2024-09-13T14:44:29.607535+0000","last_peered":"2024-09-13T14:44:29.607535+0000","last_clean":"2024-09-13T14:44:29.607535+0000","last_became_active":"2024-09-13T14:38:53.901157+0000","last_became_peered":"2024-09-13T14:38:53.901157+0000","last_unstale":"2024-09-13T14:44:29.607535+0000","last_undegraded":"2024-09-13T14:44:29.607535+0000","last_fullsized":"2024-09-13T14:44:29.607535+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:28:21.713025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"295'12323","reported_seq":35577,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:47.140312+0000","last_change":"2024-09-13T14:43:16.851222+0000","last_active":"2024-09-13T14:44:47.140312+0000","last_peered":"2024-09-13T14:44:47.140312+0000","last_clean":"2024-09-13T14:44:47.140312+0000","last_became_active":"2024-09-13T14:38:16.427510+0000","last_became_peered":"2024-09-13T14:38:16.427510+0000","last_unstale":"2024-09-13T14:44:47.140312+0000","last_undegraded":"2024-09-13T14:44:47.140312+0000","last_fullsized":"2024-09-13T14:44:47.140312+0000","mapping_epoch":245,"log_start":"220'6418","ondisk_log_start":"220'6418","created":205,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5905,"log_dups_size":0,"ondisk_log_size":5905,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:27:26.281938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109772800,"num_objects":1675,"num_object_clones":0,"num_object_copies":5025,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12323,"num_write_kb":447936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11571,"num_bytes_recovered":747372544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":1004,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-13T14:44:37.313082+0000","last_change":"2024-09-13T14:37:49.147498+0000","last_active":"2024-09-13T14:44:37.313082+0000","last_peered":"2024-09-13T14:44:37.313082+0000","last_clean":"2024-09-13T14:44:37.313082+0000","last_became_active":"2024-09-13T14:37:49.147028+0000","last_became_peered":"2024-09-13T14:37:49.147028+0000","last_unstale":"2024-09-13T14:44:37.313082+0000","last_undegraded":"2024-09-13T14:44:37.313082+0000","last_fullsized":"2024-09-13T14:44:37.313082+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:11:29.050178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":300,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:16.634514+0000","last_change":"2024-09-13T14:38:03.528133+0000","last_active":"2024-09-13T14:44:16.634514+0000","last_peered":"2024-09-13T14:44:16.634514+0000","last_clean":"2024-09-13T14:44:16.634514+0000","last_became_active":"2024-09-13T14:38:03.527829+0000","last_became_peered":"2024-09-13T14:38:03.527829+0000","last_unstale":"2024-09-13T14:44:16.634514+0000","last_undegraded":"2024-09-13T14:44:16.634514+0000","last_fullsized":"2024-09-13T14:44:16.634514+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:48.254169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":475,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.651031+0000","last_change":"2024-09-13T14:38:53.899755+0000","last_active":"2024-09-13T14:44:43.651031+0000","last_peered":"2024-09-13T14:44:43.651031+0000","last_clean":"2024-09-13T14:44:43.651031+0000","last_became_active":"2024-09-13T14:38:53.899448+0000","last_became_peered":"2024-09-13T14:38:53.899448+0000","last_unstale":"2024-09-13T14:44:43.651031+0000","last_undegraded":"2024-09-13T14:44:43.651031+0000","last_fullsized":"2024-09-13T14:44:43.651031+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:55.897035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"294'12539","reported_seq":21675,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-13T14:44:42.846579+0000","last_change":"2024-09-13T14:43:51.649951+0000","last_active":"2024-09-13T14:44:42.846579+0000","last_peered":"2024-09-13T14:44:42.846579+0000","last_clean":"2024-09-13T14:44:42.846579+0000","last_became_active":"2024-09-13T14:43:51.626628+0000","last_became_peered":"2024-09-13T14:43:51.626628+0000","last_unstale":"2024-09-13T14:44:42.846579+0000","last_undegraded":"2024-09-13T14:44:42.846579+0000","last_fullsized":"2024-09-13T14:44:42.846579+0000","mapping_epoch":289,"log_start":"219'5663","ondisk_log_start":"219'5663","created":205,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6876,"log_dups_size":0,"ondisk_log_size":6876,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:37:06.795173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115539968,"num_objects":1763,"num_object_clones":0,"num_object_copies":5289,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12539,"num_write_kb":457664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4502,"num_bytes_recovered":228851712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":106,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-13T14:44:42.313669+0000","last_change":"2024-09-13T14:37:49.149770+0000","last_active":"2024-09-13T14:44:42.313669+0000","last_peered":"2024-09-13T14:44:42.313669+0000","last_clean":"2024-09-13T14:44:42.313669+0000","last_became_active":"2024-09-13T14:37:49.148991+0000","last_became_peered":"2024-09-13T14:37:49.148991+0000","last_unstale":"2024-09-13T14:44:42.313669+0000","last_undegraded":"2024-09-13T14:44:42.313669+0000","last_fullsized":"2024-09-13T14:44:42.313669+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:04:04.193366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":1173,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.646479+0000","last_change":"2024-09-13T14:30:58.928302+0000","last_active":"2024-09-13T14:44:43.646479+0000","last_peered":"2024-09-13T14:44:43.646479+0000","last_clean":"2024-09-13T14:44:43.646479+0000","last_became_active":"2024-09-13T14:30:58.927718+0000","last_became_peered":"2024-09-13T14:30:58.927718+0000","last_unstale":"2024-09-13T14:44:43.646479+0000","last_undegraded":"2024-09-13T14:44:43.646479+0000","last_fullsized":"2024-09-13T14:44:43.646479+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:37.323036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":113,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-13T14:44:37.313105+0000","last_change":"2024-09-13T14:38:03.534628+0000","last_active":"2024-09-13T14:44:37.313105+0000","last_peered":"2024-09-13T14:44:37.313105+0000","last_clean":"2024-09-13T14:44:37.313105+0000","last_became_active":"2024-09-13T14:38:03.534262+0000","last_became_peered":"2024-09-13T14:38:03.534262+0000","last_unstale":"2024-09-13T14:44:37.313105+0000","last_undegraded":"2024-09-13T14:44:37.313105+0000","last_fullsized":"2024-09-13T14:44:37.313105+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:31.168783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"295'12372","reported_seq":26555,"reported_epoch":295,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:44:47.274078+0000","last_change":"2024-09-13T14:38:53.944111+0000","last_active":"2024-09-13T14:44:47.274078+0000","last_peered":"2024-09-13T14:44:47.274078+0000","last_clean":"2024-09-13T14:37:47.047672+0000","last_became_active":"2024-09-13T14:37:49.664294+0000","last_became_peered":"2024-09-13T14:37:49.664294+0000","last_unstale":"2024-09-13T14:44:47.274078+0000","last_undegraded":"2024-09-13T14:38:53.940822+0000","last_fullsized":"2024-09-13T14:37:49.126627+0000","mapping_epoch":240,"log_start":"220'6700","ondisk_log_start":"220'6700","created":205,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5672,"log_dups_size":0,"ondisk_log_size":5672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:11.766692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110100480,"num_objects":1680,"num_object_clones":0,"num_object_copies":5040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4535,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12372,"num_write_kb":449664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7009,"num_bytes_recovered":459341824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3],"avail_no_missing":["5","0","3"],"object_location_counts":[{"shards":"0,3,5","objects":1680},{"shards":"3,5","objects":-2780}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":125,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-13T14:44:42.313654+0000","last_change":"2024-09-13T14:38:03.534303+0000","last_active":"2024-09-13T14:44:42.313654+0000","last_peered":"2024-09-13T14:44:42.313654+0000","last_clean":"2024-09-13T14:44:42.313654+0000","last_became_active":"2024-09-13T14:38:03.533171+0000","last_became_peered":"2024-09-13T14:38:03.533171+0000","last_unstale":"2024-09-13T14:44:42.313654+0000","last_undegraded":"2024-09-13T14:44:42.313654+0000","last_fullsized":"2024-09-13T14:44:42.313654+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:00:16.553526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":817,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-13T14:44:29.621452+0000","last_change":"2024-09-13T14:38:04.596192+0000","last_active":"2024-09-13T14:44:29.621452+0000","last_peered":"2024-09-13T14:44:29.621452+0000","last_clean":"2024-09-13T14:44:29.621452+0000","last_became_active":"2024-09-13T14:38:04.593299+0000","last_became_peered":"2024-09-13T14:38:04.593299+0000","last_unstale":"2024-09-13T14:44:29.621452+0000","last_undegraded":"2024-09-13T14:44:29.621452+0000","last_fullsized":"2024-09-13T14:44:29.621452+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:34.241410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":115,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.635717+0000","last_change":"2024-09-13T14:38:03.533120+0000","last_active":"2024-09-13T14:44:43.635717+0000","last_peered":"2024-09-13T14:44:43.635717+0000","last_clean":"2024-09-13T14:44:43.635717+0000","last_became_active":"2024-09-13T14:38:03.532829+0000","last_became_peered":"2024-09-13T14:38:03.532829+0000","last_unstale":"2024-09-13T14:44:43.635717+0000","last_undegraded":"2024-09-13T14:44:43.635717+0000","last_fullsized":"2024-09-13T14:44:43.635717+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:24.514661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"295'12652","reported_seq":21931,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:47.410075+0000","last_change":"2024-09-13T14:42:04.660877+0000","last_active":"2024-09-13T14:44:47.410075+0000","last_peered":"2024-09-13T14:44:47.410075+0000","last_clean":"2024-09-13T14:44:47.410075+0000","last_became_active":"2024-09-13T14:42:04.641680+0000","last_became_peered":"2024-09-13T14:42:04.641680+0000","last_unstale":"2024-09-13T14:44:47.410075+0000","last_undegraded":"2024-09-13T14:44:47.410075+0000","last_fullsized":"2024-09-13T14:44:47.410075+0000","mapping_epoch":277,"log_start":"220'6918","ondisk_log_start":"220'6918","created":205,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5734,"log_dups_size":0,"ondisk_log_size":5734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:22:18.955428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114425856,"num_objects":1746,"num_object_clones":0,"num_object_copies":5238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12652,"num_write_kb":460736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4533,"num_bytes_recovered":297074688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":517,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.651463+0000","last_change":"2024-09-13T14:38:53.938227+0000","last_active":"2024-09-13T14:44:43.651463+0000","last_peered":"2024-09-13T14:44:43.651463+0000","last_clean":"2024-09-13T14:44:43.651463+0000","last_became_active":"2024-09-13T14:38:53.902867+0000","last_became_peered":"2024-09-13T14:38:53.902867+0000","last_unstale":"2024-09-13T14:44:43.651463+0000","last_undegraded":"2024-09-13T14:44:43.651463+0000","last_fullsized":"2024-09-13T14:44:43.651463+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:16.831756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":594,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:16.633748+0000","last_change":"2024-09-13T14:37:48.147677+0000","last_active":"2024-09-13T14:44:16.633748+0000","last_peered":"2024-09-13T14:44:16.633748+0000","last_clean":"2024-09-13T14:44:16.633748+0000","last_became_active":"2024-09-13T14:37:48.147001+0000","last_became_peered":"2024-09-13T14:37:48.147001+0000","last_unstale":"2024-09-13T14:44:16.633748+0000","last_undegraded":"2024-09-13T14:44:16.633748+0000","last_fullsized":"2024-09-13T14:44:16.633748+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:35:00.653576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":498,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.650843+0000","last_change":"2024-09-13T14:38:53.900107+0000","last_active":"2024-09-13T14:44:43.650843+0000","last_peered":"2024-09-13T14:44:43.650843+0000","last_clean":"2024-09-13T14:44:43.650843+0000","last_became_active":"2024-09-13T14:38:53.899556+0000","last_became_peered":"2024-09-13T14:38:53.899556+0000","last_unstale":"2024-09-13T14:44:43.650843+0000","last_undegraded":"2024-09-13T14:44:43.650843+0000","last_fullsized":"2024-09-13T14:44:43.650843+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:15:47.918163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"295'12222","reported_seq":21227,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:47.424554+0000","last_change":"2024-09-13T14:39:54.921239+0000","last_active":"2024-09-13T14:44:47.424554+0000","last_peered":"2024-09-13T14:44:47.424554+0000","last_clean":"2024-09-13T14:44:47.424554+0000","last_became_active":"2024-09-13T14:39:54.907894+0000","last_became_peered":"2024-09-13T14:39:54.907894+0000","last_unstale":"2024-09-13T14:44:47.424554+0000","last_undegraded":"2024-09-13T14:44:47.424554+0000","last_fullsized":"2024-09-13T14:44:47.424554+0000","mapping_epoch":263,"log_start":"220'6547","ondisk_log_start":"220'6547","created":205,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5675,"log_dups_size":0,"ondisk_log_size":5675,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:52:04.619238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106823680,"num_objects":1630,"num_object_clones":0,"num_object_copies":4890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12222,"num_write_kb":443264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4395,"num_bytes_recovered":283770880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":130,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.635637+0000","last_change":"2024-09-13T14:38:53.901806+0000","last_active":"2024-09-13T14:44:43.635637+0000","last_peered":"2024-09-13T14:44:43.635637+0000","last_clean":"2024-09-13T14:44:43.635637+0000","last_became_active":"2024-09-13T14:38:53.891055+0000","last_became_peered":"2024-09-13T14:38:53.891055+0000","last_unstale":"2024-09-13T14:44:43.635637+0000","last_undegraded":"2024-09-13T14:44:43.635637+0000","last_fullsized":"2024-09-13T14:44:43.635637+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:03:00.416198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":70,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-13T14:44:29.621572+0000","last_change":"2024-09-13T14:43:16.854299+0000","last_active":"2024-09-13T14:44:29.621572+0000","last_peered":"2024-09-13T14:44:29.621572+0000","last_clean":"2024-09-13T14:44:29.621572+0000","last_became_active":"2024-09-13T14:38:53.900626+0000","last_became_peered":"2024-09-13T14:38:53.900626+0000","last_unstale":"2024-09-13T14:44:29.621572+0000","last_undegraded":"2024-09-13T14:44:29.621572+0000","last_fullsized":"2024-09-13T14:44:29.621572+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:37:21.715778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":284,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:16.633825+0000","last_change":"2024-09-13T14:37:49.172218+0000","last_active":"2024-09-13T14:44:16.633825+0000","last_peered":"2024-09-13T14:44:16.633825+0000","last_clean":"2024-09-13T14:44:16.633825+0000","last_became_active":"2024-09-13T14:37:49.171718+0000","last_became_peered":"2024-09-13T14:37:49.171718+0000","last_unstale":"2024-09-13T14:44:16.633825+0000","last_undegraded":"2024-09-13T14:44:16.633825+0000","last_fullsized":"2024-09-13T14:44:16.633825+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:46:54.626521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":307,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:16.634339+0000","last_change":"2024-09-13T14:37:49.230094+0000","last_active":"2024-09-13T14:44:16.634339+0000","last_peered":"2024-09-13T14:44:16.634339+0000","last_clean":"2024-09-13T14:44:16.634339+0000","last_became_active":"2024-09-13T14:37:49.229393+0000","last_became_peered":"2024-09-13T14:37:49.229393+0000","last_unstale":"2024-09-13T14:44:16.634339+0000","last_undegraded":"2024-09-13T14:44:16.634339+0000","last_fullsized":"2024-09-13T14:44:16.634339+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:02:28.690318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":287,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:17.875535+0000","last_change":"2024-09-13T14:37:48.128447+0000","last_active":"2024-09-13T14:44:17.875535+0000","last_peered":"2024-09-13T14:44:17.875535+0000","last_clean":"2024-09-13T14:44:17.875535+0000","last_became_active":"2024-09-13T14:37:48.128143+0000","last_became_peered":"2024-09-13T14:37:48.128143+0000","last_unstale":"2024-09-13T14:44:17.875535+0000","last_undegraded":"2024-09-13T14:44:17.875535+0000","last_fullsized":"2024-09-13T14:44:17.875535+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:22.204039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":114,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.635802+0000","last_change":"2024-09-13T14:38:03.531683+0000","last_active":"2024-09-13T14:44:43.635802+0000","last_peered":"2024-09-13T14:44:43.635802+0000","last_clean":"2024-09-13T14:44:43.635802+0000","last_became_active":"2024-09-13T14:38:03.531476+0000","last_became_peered":"2024-09-13T14:38:03.531476+0000","last_unstale":"2024-09-13T14:44:43.635802+0000","last_undegraded":"2024-09-13T14:44:43.635802+0000","last_fullsized":"2024-09-13T14:44:43.635802+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:41.557569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":60,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-13T14:44:29.621294+0000","last_change":"2024-09-13T14:38:53.954281+0000","last_active":"2024-09-13T14:44:29.621294+0000","last_peered":"2024-09-13T14:44:29.621294+0000","last_clean":"2024-09-13T14:44:29.621294+0000","last_became_active":"2024-09-13T14:38:53.900912+0000","last_became_peered":"2024-09-13T14:38:53.900912+0000","last_unstale":"2024-09-13T14:44:29.621294+0000","last_undegraded":"2024-09-13T14:44:29.621294+0000","last_fullsized":"2024-09-13T14:44:29.621294+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:47.876275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":112,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-13T14:44:29.607335+0000","last_change":"2024-09-13T14:38:54.000782+0000","last_active":"2024-09-13T14:44:29.607335+0000","last_peered":"2024-09-13T14:44:29.607335+0000","last_clean":"2024-09-13T14:44:29.607335+0000","last_became_active":"2024-09-13T14:38:54.000322+0000","last_became_peered":"2024-09-13T14:38:54.000322+0000","last_unstale":"2024-09-13T14:44:29.607335+0000","last_undegraded":"2024-09-13T14:44:29.607335+0000","last_fullsized":"2024-09-13T14:44:29.607335+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:09.057597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":440,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.650713+0000","last_change":"2024-09-13T14:38:53.899535+0000","last_active":"2024-09-13T14:44:43.650713+0000","last_peered":"2024-09-13T14:44:43.650713+0000","last_clean":"2024-09-13T14:44:43.650713+0000","last_became_active":"2024-09-13T14:38:53.899183+0000","last_became_peered":"2024-09-13T14:38:53.899183+0000","last_unstale":"2024-09-13T14:44:43.650713+0000","last_undegraded":"2024-09-13T14:44:43.650713+0000","last_fullsized":"2024-09-13T14:44:43.650713+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:33.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":114,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.635841+0000","last_change":"2024-09-13T14:38:03.543426+0000","last_active":"2024-09-13T14:44:43.635841+0000","last_peered":"2024-09-13T14:44:43.635841+0000","last_clean":"2024-09-13T14:44:43.635841+0000","last_became_active":"2024-09-13T14:38:03.542661+0000","last_became_peered":"2024-09-13T14:38:03.542661+0000","last_unstale":"2024-09-13T14:44:43.635841+0000","last_undegraded":"2024-09-13T14:44:43.635841+0000","last_fullsized":"2024-09-13T14:44:43.635841+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:54.050127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":469,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.650759+0000","last_change":"2024-09-13T14:37:48.131321+0000","last_active":"2024-09-13T14:44:43.650759+0000","last_peered":"2024-09-13T14:44:43.650759+0000","last_clean":"2024-09-13T14:44:43.650759+0000","last_became_active":"2024-09-13T14:37:48.130998+0000","last_became_peered":"2024-09-13T14:37:48.130998+0000","last_unstale":"2024-09-13T14:44:43.650759+0000","last_undegraded":"2024-09-13T14:44:43.650759+0000","last_fullsized":"2024-09-13T14:44:43.650759+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:33:26.597825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":125,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-13T14:44:32.312614+0000","last_change":"2024-09-13T14:38:03.535454+0000","last_active":"2024-09-13T14:44:32.312614+0000","last_peered":"2024-09-13T14:44:32.312614+0000","last_clean":"2024-09-13T14:44:32.312614+0000","last_became_active":"2024-09-13T14:38:03.535131+0000","last_became_peered":"2024-09-13T14:38:03.535131+0000","last_unstale":"2024-09-13T14:44:32.312614+0000","last_undegraded":"2024-09-13T14:44:32.312614+0000","last_fullsized":"2024-09-13T14:44:32.312614+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:24.477601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":494,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.650998+0000","last_change":"2024-09-13T14:38:03.524939+0000","last_active":"2024-09-13T14:44:43.650998+0000","last_peered":"2024-09-13T14:44:43.650998+0000","last_clean":"2024-09-13T14:44:43.650998+0000","last_became_active":"2024-09-13T14:38:03.524406+0000","last_became_peered":"2024-09-13T14:38:03.524406+0000","last_unstale":"2024-09-13T14:44:43.650998+0000","last_undegraded":"2024-09-13T14:44:43.650998+0000","last_fullsized":"2024-09-13T14:44:43.650998+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:19:45.257424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":122,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.635516+0000","last_change":"2024-09-13T14:38:53.892948+0000","last_active":"2024-09-13T14:44:43.635516+0000","last_peered":"2024-09-13T14:44:43.635516+0000","last_clean":"2024-09-13T14:44:43.635516+0000","last_became_active":"2024-09-13T14:38:53.892665+0000","last_became_peered":"2024-09-13T14:38:53.892665+0000","last_unstale":"2024-09-13T14:44:43.635516+0000","last_undegraded":"2024-09-13T14:44:43.635516+0000","last_fullsized":"2024-09-13T14:44:43.635516+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:08.292359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":442,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.651017+0000","last_change":"2024-09-13T14:37:48.139348+0000","last_active":"2024-09-13T14:44:43.651017+0000","last_peered":"2024-09-13T14:44:43.651017+0000","last_clean":"2024-09-13T14:44:43.651017+0000","last_became_active":"2024-09-13T14:37:48.139115+0000","last_became_peered":"2024-09-13T14:37:48.139115+0000","last_unstale":"2024-09-13T14:44:43.651017+0000","last_undegraded":"2024-09-13T14:44:43.651017+0000","last_fullsized":"2024-09-13T14:44:43.651017+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:32:20.413346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":482,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.651242+0000","last_change":"2024-09-13T14:37:48.172088+0000","last_active":"2024-09-13T14:44:43.651242+0000","last_peered":"2024-09-13T14:44:43.651242+0000","last_clean":"2024-09-13T14:44:43.651242+0000","last_became_active":"2024-09-13T14:37:48.171779+0000","last_became_peered":"2024-09-13T14:37:48.171779+0000","last_unstale":"2024-09-13T14:44:43.651242+0000","last_undegraded":"2024-09-13T14:44:43.651242+0000","last_fullsized":"2024-09-13T14:44:43.651242+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:02.852712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":351,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:16.634332+0000","last_change":"2024-09-13T14:38:53.890236+0000","last_active":"2024-09-13T14:44:16.634332+0000","last_peered":"2024-09-13T14:44:16.634332+0000","last_clean":"2024-09-13T14:44:16.634332+0000","last_became_active":"2024-09-13T14:38:53.889582+0000","last_became_peered":"2024-09-13T14:38:53.889582+0000","last_unstale":"2024-09-13T14:44:16.634332+0000","last_undegraded":"2024-09-13T14:44:16.634332+0000","last_fullsized":"2024-09-13T14:44:16.634332+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:14:29.334774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":469,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.651423+0000","last_change":"2024-09-13T14:38:03.534097+0000","last_active":"2024-09-13T14:44:43.651423+0000","last_peered":"2024-09-13T14:44:43.651423+0000","last_clean":"2024-09-13T14:44:43.651423+0000","last_became_active":"2024-09-13T14:38:03.533794+0000","last_became_peered":"2024-09-13T14:38:03.533794+0000","last_unstale":"2024-09-13T14:44:43.651423+0000","last_undegraded":"2024-09-13T14:44:43.651423+0000","last_fullsized":"2024-09-13T14:44:43.651423+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:23.272098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":114,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.635518+0000","last_change":"2024-09-13T14:38:03.527021+0000","last_active":"2024-09-13T14:44:43.635518+0000","last_peered":"2024-09-13T14:44:43.635518+0000","last_clean":"2024-09-13T14:44:43.635518+0000","last_became_active":"2024-09-13T14:38:03.526698+0000","last_became_peered":"2024-09-13T14:38:03.526698+0000","last_unstale":"2024-09-13T14:44:43.635518+0000","last_undegraded":"2024-09-13T14:44:43.635518+0000","last_fullsized":"2024-09-13T14:44:43.635518+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:01:35.734086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":295,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:16.634259+0000","last_change":"2024-09-13T14:27:12.834931+0000","last_active":"2024-09-13T14:44:16.634259+0000","last_peered":"2024-09-13T14:44:16.634259+0000","last_clean":"2024-09-13T14:44:16.634259+0000","last_became_active":"2024-09-13T14:27:12.834572+0000","last_became_peered":"2024-09-13T14:27:12.834572+0000","last_unstale":"2024-09-13T14:44:16.634259+0000","last_undegraded":"2024-09-13T14:44:16.634259+0000","last_fullsized":"2024-09-13T14:44:16.634259+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:54:21.155689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":109,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-13T14:44:29.607171+0000","last_change":"2024-09-13T14:37:49.176796+0000","last_active":"2024-09-13T14:44:29.607171+0000","last_peered":"2024-09-13T14:44:29.607171+0000","last_clean":"2024-09-13T14:44:29.607171+0000","last_became_active":"2024-09-13T14:37:49.176246+0000","last_became_peered":"2024-09-13T14:37:49.176246+0000","last_unstale":"2024-09-13T14:44:29.607171+0000","last_undegraded":"2024-09-13T14:44:29.607171+0000","last_fullsized":"2024-09-13T14:44:29.607171+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:29:24.936718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":306,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:16.634156+0000","last_change":"2024-09-13T14:38:03.526333+0000","last_active":"2024-09-13T14:44:16.634156+0000","last_peered":"2024-09-13T14:44:16.634156+0000","last_clean":"2024-09-13T14:44:16.634156+0000","last_became_active":"2024-09-13T14:38:03.526001+0000","last_became_peered":"2024-09-13T14:38:03.526001+0000","last_unstale":"2024-09-13T14:44:16.634156+0000","last_undegraded":"2024-09-13T14:44:16.634156+0000","last_fullsized":"2024-09-13T14:44:16.634156+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:25:25.181271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":109,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-13T14:44:29.607206+0000","last_change":"2024-09-13T14:37:49.149582+0000","last_active":"2024-09-13T14:44:29.607206+0000","last_peered":"2024-09-13T14:44:29.607206+0000","last_clean":"2024-09-13T14:44:29.607206+0000","last_became_active":"2024-09-13T14:37:49.148525+0000","last_became_peered":"2024-09-13T14:37:49.148525+0000","last_unstale":"2024-09-13T14:44:29.607206+0000","last_undegraded":"2024-09-13T14:44:29.607206+0000","last_fullsized":"2024-09-13T14:44:29.607206+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:48.655665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":515,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.651216+0000","last_change":"2024-09-13T14:38:03.535255+0000","last_active":"2024-09-13T14:44:43.651216+0000","last_peered":"2024-09-13T14:44:43.651216+0000","last_clean":"2024-09-13T14:44:43.651216+0000","last_became_active":"2024-09-13T14:38:03.533917+0000","last_became_peered":"2024-09-13T14:38:03.533917+0000","last_unstale":"2024-09-13T14:44:43.651216+0000","last_undegraded":"2024-09-13T14:44:43.651216+0000","last_fullsized":"2024-09-13T14:44:43.651216+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:33.559788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":478,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.652702+0000","last_change":"2024-09-13T14:37:48.146784+0000","last_active":"2024-09-13T14:44:43.652702+0000","last_peered":"2024-09-13T14:44:43.652702+0000","last_clean":"2024-09-13T14:44:43.652702+0000","last_became_active":"2024-09-13T14:37:48.146491+0000","last_became_peered":"2024-09-13T14:37:48.146491+0000","last_unstale":"2024-09-13T14:44:43.652702+0000","last_undegraded":"2024-09-13T14:44:43.652702+0000","last_fullsized":"2024-09-13T14:44:43.652702+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:30:54.772513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":286,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-13T14:44:16.633663+0000","last_change":"2024-09-13T14:37:48.127607+0000","last_active":"2024-09-13T14:44:16.633663+0000","last_peered":"2024-09-13T14:44:16.633663+0000","last_clean":"2024-09-13T14:44:16.633663+0000","last_became_active":"2024-09-13T14:37:48.127316+0000","last_became_peered":"2024-09-13T14:37:48.127316+0000","last_unstale":"2024-09-13T14:44:16.633663+0000","last_undegraded":"2024-09-13T14:44:16.633663+0000","last_fullsized":"2024-09-13T14:44:16.633663+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:53:37.678861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":121,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.635785+0000","last_change":"2024-09-13T14:38:03.563200+0000","last_active":"2024-09-13T14:44:43.635785+0000","last_peered":"2024-09-13T14:44:43.635785+0000","last_clean":"2024-09-13T14:44:43.635785+0000","last_became_active":"2024-09-13T14:38:03.534258+0000","last_became_peered":"2024-09-13T14:38:03.534258+0000","last_unstale":"2024-09-13T14:44:43.635785+0000","last_undegraded":"2024-09-13T14:44:43.635785+0000","last_fullsized":"2024-09-13T14:44:43.635785+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:49:07.814419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":1770717208,"num_objects":27020,"num_object_clones":0,"num_object_copies":81060,"num_objects_missing_on_primary":78,"num_objects_missing":78,"num_objects_degraded":4814,"num_objects_misplaced":3171,"num_objects_unfound":0,"num_objects_dirty":27020,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":198606,"num_write_kb":7219969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85542,"num_bytes_recovered":5478744088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7632334848,"data_stored":7632322632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94929,"ondisk_log_size":94929,"up":48,"acting":47,"num_store_stats":6},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1538,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":17390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25132,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16793600,"data_stored":16777292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":23,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2969600,"data_stored":2951840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234411,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1514556,"kb_used_data":1042100,"kb_used_omap":1,"kb_used_meta":472446,"kb_avail":92230596,"statfs":{"total":95995035648,"available":94444130304,"internally_reserved":0,"allocated":1067110400,"data_stored":1065538081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":483785163},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":239,"seq":1026497183829,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1215872,"kb_used_data":892664,"kb_used_omap":8,"kb_used_meta":323191,"kb_avail":92529280,"statfs":{"total":95995035648,"available":94749982720,"internally_reserved":0,"allocated":914087936,"data_stored":912512092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8453,"internal_metadata":330948347},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757676,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1654636,"kb_used_data":983652,"kb_used_omap":1,"kb_used_meta":670974,"kb_avail":92090516,"statfs":{"total":95995035648,"available":94300688384,"internally_reserved":0,"allocated":1007259648,"data_stored":1005698890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":687077830},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019319,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1724072,"kb_used_data":1107232,"kb_used_omap":1,"kb_used_meta":616830,"kb_avail":92021080,"statfs":{"total":95995035648,"available":94229585920,"internally_reserved":0,"allocated":1133805568,"data_stored":1132244909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":631634374},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435503,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1893864,"kb_used_data":1298528,"kb_used_omap":5,"kb_used_meta":595322,"kb_avail":91851288,"statfs":{"total":95995035648,"available":94055718912,"internally_reserved":0,"allocated":1329692672,"data_stored":1328135774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":609609854},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":446676599031,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":425201762557,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926081,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2884984,"kb_used_data":2203504,"kb_used_omap":7,"kb_used_meta":681400,"kb_avail":90860168,"statfs":{"total":95995035648,"available":93040812032,"internally_reserved":0,"allocated":2256388096,"data_stored":2254815981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":697753896},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1,1,2,3,9,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6860,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2232029184,"data_stored":2232025112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1320226816,"data_stored":1320222744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1128267776,"data_stored":1128267776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":998965248,"data_stored":998965248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":904073216,"data_stored":904069144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1048772608,"data_stored":1048772608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:44:51.554 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:51.583 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:51.610 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:51.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:51.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:51.712 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:51.735 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:51.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:51.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:51.837 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:51.859 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:51.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:51.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:51.961 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:51.985 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:51.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:51.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:52.087 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:52.109 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:52.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:52.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:52.210 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:52.234 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:52.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:52.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:52.336 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:52.359 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:52.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:52.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:52.461 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:52.483 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:52.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:52.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:52.585 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:52.611 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:52.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:52.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:52.713 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:52.736 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:52.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:52.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:52.838 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:52.863 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:52.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:52.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:52.965 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:52.986 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:52.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:52.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:53.088 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:53.112 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:53.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:53.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:53.214 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:53.237 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:53 smithi102 ceph-mon[110183]: pgmap v1031: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.6 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 402 op/s; 4823/80793 objects degraded (5.970%); 3171/80793 objects misplaced (3.925%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T14:44:53.237 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:53 smithi102 ceph-mon[115269]: pgmap v1031: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.6 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 402 op/s; 4823/80793 objects degraded (5.970%); 3171/80793 objects misplaced (3.925%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T14:44:53.250 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:53.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:53.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:53.352 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:53.374 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:53.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:53.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:53.476 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:53.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:53 smithi195 ceph-mon[104262]: pgmap v1031: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.6 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 402 op/s; 4823/80793 objects degraded (5.970%); 3171/80793 objects misplaced (3.925%); 1.6 MiB/s, 25 objects/s recovering 2024-09-13T14:44:53.500 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:53.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:53.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:53.602 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:53.629 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:53.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:53.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:53.703 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:44:53.703 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:44:53.731 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:53.754 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:53.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:53.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:53.856 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:53.880 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:53.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:53.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:53.982 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:54.009 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:54.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:54.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:54.110 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:54.140 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:54.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:54.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:54.242 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:54.265 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:54.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:54.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:54.367 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:54.389 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:54.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:54.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:54.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:54 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 4823/80793 objects degraded (5.970%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:54.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:54 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 4823/80793 objects degraded (5.970%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:54.491 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:54.495 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T14:44:54.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:54 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 4823/80793 objects degraded (5.970%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:54.508 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:44:54.512 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:54.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:54.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:54.614 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:54.635 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:54.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:54.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:54.737 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:54.762 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:54.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:54.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:54.864 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:54.889 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:54.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:54.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:54.990 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:55.013 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:55.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:55.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:55.115 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:55.139 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:55.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:55.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:55.241 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:55.263 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:55.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:55.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:55.365 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:55.390 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:55.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:55.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:55.492 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:55.500 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:55 smithi195 ceph-mon[104262]: pgmap v1032: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.5 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 381 op/s; 4823/75108 objects degraded (6.421%); 2908/75108 objects misplaced (3.872%); 1.7 MiB/s, 26 objects/s recovering 2024-09-13T14:44:55.514 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:55.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:55.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:55.616 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:55.639 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:55.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:55.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:55.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:55 smithi102 ceph-mon[110183]: pgmap v1032: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.5 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 381 op/s; 4823/75108 objects degraded (6.421%); 2908/75108 objects misplaced (3.872%); 1.7 MiB/s, 26 objects/s recovering 2024-09-13T14:44:55.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:55 smithi102 ceph-mon[115269]: pgmap v1032: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.5 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 381 op/s; 4823/75108 objects degraded (6.421%); 2908/75108 objects misplaced (3.872%); 1.7 MiB/s, 26 objects/s recovering 2024-09-13T14:44:55.741 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:55.763 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:55.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:55.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:55.865 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:55.886 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:55.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:55.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:55.988 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:56.010 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:56.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:56.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:56.112 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:56.138 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:56.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:56.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:56.240 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:56.258 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:44:56.259 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:44:56.277 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:56.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:56.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:56.379 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:56.401 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:56.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:56.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:56.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:44:56.502 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:56.520 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:44:56 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:44:56] "GET /metrics HTTP/1.1" 200 36547 "" "Prometheus/2.51.0" 2024-09-13T14:44:56.536 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:56.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:56.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:56.638 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:56.659 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:56.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:56.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:56.761 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:56.788 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:56.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:56.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:56.804 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:44:56.804 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:44:56.813 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:56.815 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:44:56.815 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:44:56.847 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:44:56.847 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:44:56.870 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:56.880 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:44:56.880 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:44:56.890 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:56.907 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:44:56.907 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:44:56.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:56.921 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:56.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:56.931 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:44:56.931 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:44:56.953 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:44:56.954 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:44:56.974 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:44:56.974 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:44:57.022 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:57.047 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:57.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:57.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:57.149 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:57.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:57.177 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:57.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:57.214 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is unset 2024-09-13T14:44:57.278 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:57.305 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:57.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:57.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:57.373 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:57 smithi102 ceph-mon[110183]: pgmap v1033: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.5 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 320 op/s; 4823/75108 objects degraded (6.421%); 2908/75108 objects misplaced (3.872%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T14:44:57.373 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:57 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3174639544' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:44:57.373 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:57 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2593316392' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:44:57.374 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:57 smithi102 ceph-mon[115269]: pgmap v1033: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.5 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 320 op/s; 4823/75108 objects degraded (6.421%); 2908/75108 objects misplaced (3.872%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T14:44:57.374 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:57 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3174639544' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:44:57.374 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:57 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2593316392' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:44:57.407 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:57.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:57.442 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:57.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:57.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:57 smithi195 ceph-mon[104262]: pgmap v1033: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.5 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 320 op/s; 4823/75108 objects degraded (6.421%); 2908/75108 objects misplaced (3.872%); 1.4 MiB/s, 22 objects/s recovering 2024-09-13T14:44:57.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:57 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3174639544' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:44:57.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:57 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2593316392' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:44:57.545 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:57.573 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:57.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:57.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:57.675 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:57.706 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:57.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:57.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:57.808 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:57.829 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:57.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:57.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:57.931 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:57.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:57.953 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:57.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:58.054 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:58.076 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:58.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:58.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:58.177 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:58.202 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:58.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:58.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:58.304 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:58.330 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:58.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:58.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:58.432 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:58.453 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:58.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:58.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:58.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:58 smithi195 ceph-mon[104262]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:44:58.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:58 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3174639544' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:44:58.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:58 smithi195 ceph-mon[104262]: osdmap e296: 8 total, 8 up, 6 in 2024-09-13T14:44:58.556 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:58.578 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:58.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:58.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:58.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:58 smithi102 ceph-mon[110183]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:44:58.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:58 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3174639544' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:44:58.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:58 smithi102 ceph-mon[110183]: osdmap e296: 8 total, 8 up, 6 in 2024-09-13T14:44:58.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:58 smithi102 ceph-mon[115269]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:44:58.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:58 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3174639544' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:44:58.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:58 smithi102 ceph-mon[115269]: osdmap e296: 8 total, 8 up, 6 in 2024-09-13T14:44:58.680 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:58.704 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:58.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:58.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:58.806 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:58.828 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:58.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:58.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:58.930 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:58.952 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:58.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:58.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:59.053 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:59.074 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:59.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:59.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:59.176 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:59.202 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:59.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:59.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:59.304 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:59.331 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:59.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:59.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:59.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:59 smithi102 ceph-mon[110183]: pgmap v1035: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.4 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 360 op/s; 4823/70335 objects degraded (6.857%); 2602/70335 objects misplaced (3.699%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:44:59.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:44:59 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 4823/70335 objects degraded (6.857%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:59.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:59 smithi102 ceph-mon[115269]: pgmap v1035: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.4 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 360 op/s; 4823/70335 objects degraded (6.857%); 2602/70335 objects misplaced (3.699%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:44:59.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:44:59 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 4823/70335 objects degraded (6.857%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:59.433 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:59.459 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:44:59.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:59.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:59.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:59 smithi195 ceph-mon[104262]: pgmap v1035: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.4 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 360 op/s; 4823/70335 objects degraded (6.857%); 2602/70335 objects misplaced (3.699%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:44:59.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:44:59 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 4823/70335 objects degraded (6.857%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:44:59.562 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:59.586 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:59.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:59.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:59.688 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:59.716 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:59.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:59.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:59.818 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:59.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:59.845 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:59.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:59.946 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:59.969 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:44:59.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:59.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:00.071 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:00.100 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:00.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:00.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:00.202 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:00.229 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:00.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:00.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:00.330 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:00.356 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:00.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:00.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:00.458 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:00.480 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:00.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:00.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:00.582 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:00.607 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:00.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:00.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:00.709 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:00.730 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:00.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:00.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:00.832 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:00.855 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:00.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:00.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:00.957 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:00.982 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:00.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:00.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:01.084 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:01.106 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:01.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:01.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:01.207 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:01.233 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:01.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:01.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:01.335 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:01.357 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:01.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:01.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:01.459 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:01.486 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:01.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:01.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:01.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:01 smithi195 ceph-mon[104262]: pgmap v1036: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.4 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 421 op/s; 4823/68130 objects degraded (7.079%); 2602/68130 objects misplaced (3.819%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:45:01.588 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:01.611 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:01.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:01.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:01.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:01 smithi102 ceph-mon[110183]: pgmap v1036: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.4 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 421 op/s; 4823/68130 objects degraded (7.079%); 2602/68130 objects misplaced (3.819%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:45:01.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:01 smithi102 ceph-mon[115269]: pgmap v1036: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.4 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 421 op/s; 4823/68130 objects degraded (7.079%); 2602/68130 objects misplaced (3.819%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:45:01.713 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:01.737 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:01.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:01.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:01.839 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:01.866 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:01.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:01.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:01.968 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:01.992 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:01.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:01.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:02.094 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:02.117 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:02.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:02.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:02.219 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:02.244 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:02.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:02.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:02.346 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:02.373 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:02.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:02.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:02.474 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:02.499 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:02.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:02.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:02.601 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:02.626 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:02.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:02.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:02.729 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:02.757 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:02.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:02.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:02.804 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:45:02.858 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:02.880 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:02.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:02.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:02.982 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:03.010 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:03.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:03.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:03.111 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:03.137 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:03.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:03.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:03.239 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:03.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:03.267 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:03.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:03.367 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:03.389 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:03.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:03.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:03.490 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:03.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:03 smithi195 ceph-mon[104262]: pgmap v1037: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.4 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 421 op/s; 4823/68130 objects degraded (7.079%); 2602/68130 objects misplaced (3.819%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:45:03.506 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:03 smithi102 ceph-mon[110183]: pgmap v1037: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.4 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 421 op/s; 4823/68130 objects degraded (7.079%); 2602/68130 objects misplaced (3.819%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:45:03.506 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:03 smithi102 ceph-mon[115269]: pgmap v1037: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.4 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 421 op/s; 4823/68130 objects degraded (7.079%); 2602/68130 objects misplaced (3.819%); 1.8 MiB/s, 28 objects/s recovering 2024-09-13T14:45:03.514 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:03.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:03.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:03.616 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:03.638 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:03.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:03.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:03.740 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:03.768 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:03.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:03.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:03.870 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:03.894 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:03.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:03.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:03.996 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:04.021 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:04.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:04.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:04.123 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:04.151 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:04.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:04.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:04.253 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:04.275 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:04.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:04.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:04.347 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:04 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 4823/68130 objects degraded (7.079%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:04.347 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:04 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 4823/68130 objects degraded (7.079%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:04.377 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:04.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:04.407 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:04.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:04.480 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:45:04.481 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [], 2024-09-13T14:45:04.481 INFO:teuthology.orchestra.run.smithi102.stdout: "num_ops": 0 2024-09-13T14:45:04.481 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:45:04.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:04 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 4823/68130 objects degraded (7.079%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:04.508 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:04.537 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:04.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:04.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:04.623 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:45:04.624 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:45:04.639 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:04.669 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:04.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:04.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:04.723 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:45:04.774 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:04.801 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:04.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:04.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:04.903 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:04.929 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:04.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:04.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:05.031 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:05.055 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:05.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:05.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:05.157 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:05.183 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:05.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:05.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:05.285 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:05.306 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:05.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:05.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:05.407 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:05.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:05 smithi102 ceph-mon[110183]: pgmap v1038: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.2 GiB data, 9.1 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 479 op/s; 4823/60729 objects degraded (7.942%); 2309/60729 objects misplaced (3.802%); 1.8 MiB/s, 29 objects/s recovering 2024-09-13T14:45:05.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:05 smithi102 ceph-mon[115269]: pgmap v1038: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.2 GiB data, 9.1 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 479 op/s; 4823/60729 objects degraded (7.942%); 2309/60729 objects misplaced (3.802%); 1.8 MiB/s, 29 objects/s recovering 2024-09-13T14:45:05.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:05.436 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:05.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:05.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:05 smithi195 ceph-mon[104262]: pgmap v1038: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.2 GiB data, 9.1 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 479 op/s; 4823/60729 objects degraded (7.942%); 2309/60729 objects misplaced (3.802%); 1.8 MiB/s, 29 objects/s recovering 2024-09-13T14:45:05.538 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:05.564 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:05.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:05.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:05.608 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T14:45:05.665 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:05.690 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:05.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:05.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:05.791 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:05.816 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:05.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:05.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:05.918 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:05.947 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:05.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:05.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:06.049 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:06.073 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:06.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:06.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:06.175 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:06.196 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:06.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:06.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:06.298 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:06.319 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":1038,"stamp":"2024-09-13T14:45:03.551912+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1329534973,"num_objects":20243,"num_object_clones":0,"num_object_copies":60729,"num_objects_missing_on_primary":73,"num_objects_missing":73,"num_objects_degraded":4823,"num_objects_misplaced":2309,"num_objects_unfound":0,"num_objects_dirty":20243,"num_whiteouts":0,"num_read":6377,"num_read_kb":13834,"num_write":206534,"num_write_kb":7235010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86095,"num_bytes_recovered":5513153677,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96216,"ondisk_log_size":96216,"up":267,"acting":265,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":299,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":9553220,"kb_used_data":6178836,"kb_used_omap":26,"kb_used_meta":3374181,"kb_avail":552917692,"statfs":{"total":575970213888,"available":566187716608,"internally_reserved":0,"allocated":6327128064,"data_stored":6317701303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27144,"internal_metadata":3455161848},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,2,1,2,2,0,2,2,0,9],"upper_bound":1024},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-314114048,"num_objects":-4793,"num_object_clones":0,"num_object_copies":-14379,"num_objects_missing_on_primary":-3,"num_objects_missing":-3,"num_objects_degraded":0,"num_objects_misplaced":-599,"num_objects_unfound":0,"num_objects_dirty":-4793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4793,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":299,"num_bytes_recovered":19398656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002682"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":486,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-13T14:44:57.219719+0000","last_change":"2024-09-13T14:36:13.859927+0000","last_active":"2024-09-13T14:44:57.219719+0000","last_peered":"2024-09-13T14:44:57.219719+0000","last_clean":"2024-09-13T14:44:57.219719+0000","last_became_active":"2024-09-13T14:36:13.859505+0000","last_became_peered":"2024-09-13T14:36:13.859505+0000","last_unstale":"2024-09-13T14:44:57.219719+0000","last_undegraded":"2024-09-13T14:44:57.219719+0000","last_fullsized":"2024-09-13T14:44:57.219719+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:12:20.989627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":299,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.626248+0000","last_change":"2024-09-13T14:37:48.130968+0000","last_active":"2024-09-13T14:44:43.626248+0000","last_peered":"2024-09-13T14:44:43.626248+0000","last_clean":"2024-09-13T14:44:43.626248+0000","last_became_active":"2024-09-13T14:37:48.130603+0000","last_became_peered":"2024-09-13T14:37:48.130603+0000","last_unstale":"2024-09-13T14:44:43.626248+0000","last_undegraded":"2024-09-13T14:44:43.626248+0000","last_fullsized":"2024-09-13T14:44:43.626248+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:42.626386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":95,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.647388+0000","last_change":"2024-09-13T14:36:13.862181+0000","last_active":"2024-09-13T14:44:43.647388+0000","last_peered":"2024-09-13T14:44:43.647388+0000","last_clean":"2024-09-13T14:44:43.647388+0000","last_became_active":"2024-09-13T14:36:13.861873+0000","last_became_peered":"2024-09-13T14:36:13.861873+0000","last_unstale":"2024-09-13T14:44:43.647388+0000","last_undegraded":"2024-09-13T14:44:43.647388+0000","last_fullsized":"2024-09-13T14:44:43.647388+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:14.085700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":123,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-13T14:44:57.229819+0000","last_change":"2024-09-13T14:38:53.896672+0000","last_active":"2024-09-13T14:44:57.229819+0000","last_peered":"2024-09-13T14:44:57.229819+0000","last_clean":"2024-09-13T14:44:57.229819+0000","last_became_active":"2024-09-13T14:38:53.896346+0000","last_became_peered":"2024-09-13T14:38:53.896346+0000","last_unstale":"2024-09-13T14:44:57.229819+0000","last_undegraded":"2024-09-13T14:44:57.229819+0000","last_fullsized":"2024-09-13T14:44:57.229819+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:57.789686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":123,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-13T14:44:57.230062+0000","last_change":"2024-09-13T14:38:53.893021+0000","last_active":"2024-09-13T14:44:57.230062+0000","last_peered":"2024-09-13T14:44:57.230062+0000","last_clean":"2024-09-13T14:44:57.230062+0000","last_became_active":"2024-09-13T14:38:53.892668+0000","last_became_peered":"2024-09-13T14:38:53.892668+0000","last_unstale":"2024-09-13T14:44:57.230062+0000","last_undegraded":"2024-09-13T14:44:57.230062+0000","last_fullsized":"2024-09-13T14:44:57.230062+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:57.180738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":267,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.626096+0000","last_change":"2024-09-13T14:37:48.134642+0000","last_active":"2024-09-13T14:44:43.626096+0000","last_peered":"2024-09-13T14:44:43.626096+0000","last_clean":"2024-09-13T14:44:43.626096+0000","last_became_active":"2024-09-13T14:37:48.134402+0000","last_became_peered":"2024-09-13T14:37:48.134402+0000","last_unstale":"2024-09-13T14:44:43.626096+0000","last_undegraded":"2024-09-13T14:44:43.626096+0000","last_fullsized":"2024-09-13T14:44:43.626096+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:51.313663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":505,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-13T14:44:57.220128+0000","last_change":"2024-09-13T14:38:53.903022+0000","last_active":"2024-09-13T14:44:57.220128+0000","last_peered":"2024-09-13T14:44:57.220128+0000","last_clean":"2024-09-13T14:44:57.220128+0000","last_became_active":"2024-09-13T14:38:53.902448+0000","last_became_peered":"2024-09-13T14:38:53.902448+0000","last_unstale":"2024-09-13T14:44:57.220128+0000","last_undegraded":"2024-09-13T14:44:57.220128+0000","last_fullsized":"2024-09-13T14:44:57.220128+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:24.690401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":126,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-13T14:44:57.228981+0000","last_change":"2024-09-13T14:38:03.554500+0000","last_active":"2024-09-13T14:44:57.228981+0000","last_peered":"2024-09-13T14:44:57.228981+0000","last_clean":"2024-09-13T14:44:57.228981+0000","last_became_active":"2024-09-13T14:38:03.532910+0000","last_became_peered":"2024-09-13T14:38:03.532910+0000","last_unstale":"2024-09-13T14:44:57.228981+0000","last_undegraded":"2024-09-13T14:44:57.228981+0000","last_fullsized":"2024-09-13T14:44:57.228981+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:27:50.843498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"296'12630","reported_seq":23538,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-13T14:45:02.274746+0000","last_change":"2024-09-13T14:41:53.220083+0000","last_active":"2024-09-13T14:45:02.274746+0000","last_peered":"2024-09-13T14:45:02.274746+0000","last_clean":"2024-09-13T14:45:02.274746+0000","last_became_active":"2024-09-13T14:41:53.200084+0000","last_became_peered":"2024-09-13T14:41:53.200084+0000","last_unstale":"2024-09-13T14:45:02.274746+0000","last_undegraded":"2024-09-13T14:45:02.274746+0000","last_fullsized":"2024-09-13T14:45:02.274746+0000","mapping_epoch":275,"log_start":"220'6900","ondisk_log_start":"220'6900","created":205,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5730,"log_dups_size":0,"ondisk_log_size":5730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:08.897524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81920000,"num_objects":1250,"num_object_clones":0,"num_object_copies":3750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12630,"num_write_kb":444160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9152,"num_bytes_recovered":594608128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":59,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.626628+0000","last_change":"2024-09-13T14:38:53.940238+0000","last_active":"2024-09-13T14:44:43.626628+0000","last_peered":"2024-09-13T14:44:43.626628+0000","last_clean":"2024-09-13T14:44:43.626628+0000","last_became_active":"2024-09-13T14:38:53.892825+0000","last_became_peered":"2024-09-13T14:38:53.892825+0000","last_unstale":"2024-09-13T14:44:43.626628+0000","last_undegraded":"2024-09-13T14:44:43.626628+0000","last_fullsized":"2024-09-13T14:44:43.626628+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:46.666581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":116,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-13T14:44:57.218151+0000","last_change":"2024-09-13T14:38:53.898907+0000","last_active":"2024-09-13T14:44:57.218151+0000","last_peered":"2024-09-13T14:44:57.218151+0000","last_clean":"2024-09-13T14:44:57.218151+0000","last_became_active":"2024-09-13T14:38:53.898572+0000","last_became_peered":"2024-09-13T14:38:53.898572+0000","last_unstale":"2024-09-13T14:44:57.218151+0000","last_undegraded":"2024-09-13T14:44:57.218151+0000","last_fullsized":"2024-09-13T14:44:57.218151+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:29.222150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"296'12966","reported_seq":27169,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-13T14:45:02.517493+0000","last_change":"2024-09-13T14:38:57.930032+0000","last_active":"2024-09-13T14:45:02.517493+0000","last_peered":"2024-09-13T14:45:02.517493+0000","last_clean":"2024-09-13T14:45:02.517493+0000","last_became_active":"2024-09-13T14:38:57.921458+0000","last_became_peered":"2024-09-13T14:38:57.921458+0000","last_unstale":"2024-09-13T14:45:02.517493+0000","last_undegraded":"2024-09-13T14:45:02.517493+0000","last_fullsized":"2024-09-13T14:45:02.517493+0000","mapping_epoch":256,"log_start":"242'7239","ondisk_log_start":"242'7239","created":205,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5727,"log_dups_size":0,"ondisk_log_size":5727,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:27:12.294038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83623936,"num_objects":1276,"num_object_clones":0,"num_object_copies":3828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12966,"num_write_kb":455744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6990,"num_bytes_recovered":458096640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":59,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.627007+0000","last_change":"2024-09-13T14:38:53.904885+0000","last_active":"2024-09-13T14:44:43.627007+0000","last_peered":"2024-09-13T14:44:43.627007+0000","last_clean":"2024-09-13T14:44:43.627007+0000","last_became_active":"2024-09-13T14:38:53.891726+0000","last_became_peered":"2024-09-13T14:38:53.891726+0000","last_unstale":"2024-09-13T14:44:43.627007+0000","last_undegraded":"2024-09-13T14:44:43.627007+0000","last_fullsized":"2024-09-13T14:44:43.627007+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:57.079205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":305,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.650073+0000","last_change":"2024-09-13T14:37:48.160504+0000","last_active":"2024-09-13T14:44:43.650073+0000","last_peered":"2024-09-13T14:44:43.650073+0000","last_clean":"2024-09-13T14:44:43.650073+0000","last_became_active":"2024-09-13T14:37:48.160181+0000","last_became_peered":"2024-09-13T14:37:48.160181+0000","last_unstale":"2024-09-13T14:44:43.650073+0000","last_undegraded":"2024-09-13T14:44:43.650073+0000","last_fullsized":"2024-09-13T14:44:43.650073+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:37.087008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"296'12899","reported_seq":14724,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-13T14:45:02.154416+0000","last_change":"2024-09-13T14:38:04.531914+0000","last_active":"2024-09-13T14:45:02.154416+0000","last_peered":"2024-09-13T14:45:02.154416+0000","last_clean":"2024-09-13T14:45:02.154416+0000","last_became_active":"2024-09-13T14:38:04.523448+0000","last_became_peered":"2024-09-13T14:38:04.523448+0000","last_unstale":"2024-09-13T14:45:02.154416+0000","last_undegraded":"2024-09-13T14:45:02.154416+0000","last_fullsized":"2024-09-13T14:45:02.154416+0000","mapping_epoch":245,"log_start":"220'7000","ondisk_log_start":"220'7000","created":205,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5899,"log_dups_size":0,"ondisk_log_size":5899,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:06.536896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83558400,"num_objects":1275,"num_object_clones":0,"num_object_copies":3825,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":453568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":811,"num_bytes_recovered":53149696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":296,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.626504+0000","last_change":"2024-09-13T14:37:49.611774+0000","last_active":"2024-09-13T14:44:43.626504+0000","last_peered":"2024-09-13T14:44:43.626504+0000","last_clean":"2024-09-13T14:44:43.626504+0000","last_became_active":"2024-09-13T14:37:49.611088+0000","last_became_peered":"2024-09-13T14:37:49.611088+0000","last_unstale":"2024-09-13T14:44:43.626504+0000","last_undegraded":"2024-09-13T14:44:43.626504+0000","last_fullsized":"2024-09-13T14:44:43.626504+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:08.933213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":55,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.626138+0000","last_change":"2024-09-13T14:38:53.899211+0000","last_active":"2024-09-13T14:44:43.626138+0000","last_peered":"2024-09-13T14:44:43.626138+0000","last_clean":"2024-09-13T14:44:43.626138+0000","last_became_active":"2024-09-13T14:38:53.898927+0000","last_became_peered":"2024-09-13T14:38:53.898927+0000","last_unstale":"2024-09-13T14:44:43.626138+0000","last_undegraded":"2024-09-13T14:44:43.626138+0000","last_fullsized":"2024-09-13T14:44:43.626138+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:00:28.857611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"296'12745","reported_seq":12849,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-13T14:45:02.358012+0000","last_change":"2024-09-13T14:35:24.004651+0000","last_active":"2024-09-13T14:45:02.358012+0000","last_peered":"2024-09-13T14:45:02.358012+0000","last_clean":"2024-09-13T14:45:02.358012+0000","last_became_active":"2024-09-13T14:35:24.004358+0000","last_became_peered":"2024-09-13T14:35:24.004358+0000","last_unstale":"2024-09-13T14:45:02.358012+0000","last_undegraded":"2024-09-13T14:45:02.358012+0000","last_fullsized":"2024-09-13T14:45:02.358012+0000","mapping_epoch":205,"log_start":"220'6484","ondisk_log_start":"220'6484","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6261,"log_dups_size":0,"ondisk_log_size":6261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:57:38.023669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80412672,"num_objects":1227,"num_object_clones":0,"num_object_copies":3681,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12745,"num_write_kb":447104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":106,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.647109+0000","last_change":"2024-09-13T14:37:49.168003+0000","last_active":"2024-09-13T14:44:43.647109+0000","last_peered":"2024-09-13T14:44:43.647109+0000","last_clean":"2024-09-13T14:44:43.647109+0000","last_became_active":"2024-09-13T14:37:49.167638+0000","last_became_peered":"2024-09-13T14:37:49.167638+0000","last_unstale":"2024-09-13T14:44:43.647109+0000","last_undegraded":"2024-09-13T14:44:43.647109+0000","last_fullsized":"2024-09-13T14:44:43.647109+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:29.051981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":117,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-13T14:44:57.230126+0000","last_change":"2024-09-13T14:38:03.528381+0000","last_active":"2024-09-13T14:44:57.230126+0000","last_peered":"2024-09-13T14:44:57.230126+0000","last_clean":"2024-09-13T14:44:57.230126+0000","last_became_active":"2024-09-13T14:38:03.527282+0000","last_became_peered":"2024-09-13T14:38:03.527282+0000","last_unstale":"2024-09-13T14:44:57.230126+0000","last_undegraded":"2024-09-13T14:44:57.230126+0000","last_fullsized":"2024-09-13T14:44:57.230126+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:24.435067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.f","version":"296'12773","reported_seq":13092,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-13T14:45:02.211623+0000","last_change":"2024-09-13T14:38:16.371079+0000","last_active":"2024-09-13T14:45:02.211623+0000","last_peered":"2024-09-13T14:45:02.211623+0000","last_clean":"2024-09-13T14:45:02.211623+0000","last_became_active":"2024-09-13T14:38:12.859883+0000","last_became_peered":"2024-09-13T14:38:12.859883+0000","last_unstale":"2024-09-13T14:45:02.211623+0000","last_undegraded":"2024-09-13T14:45:02.211623+0000","last_fullsized":"2024-09-13T14:45:02.211623+0000","mapping_epoch":245,"log_start":"220'6863","ondisk_log_start":"220'6863","created":205,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5910,"log_dups_size":0,"ondisk_log_size":5910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:08.436532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86048768,"num_objects":1313,"num_object_clones":0,"num_object_copies":3939,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":450752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":3670016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":474,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-13T14:44:57.220382+0000","last_change":"2024-09-13T14:38:03.549595+0000","last_active":"2024-09-13T14:44:57.220382+0000","last_peered":"2024-09-13T14:44:57.220382+0000","last_clean":"2024-09-13T14:44:57.220382+0000","last_became_active":"2024-09-13T14:38:03.533331+0000","last_became_peered":"2024-09-13T14:38:03.533331+0000","last_unstale":"2024-09-13T14:44:57.220382+0000","last_undegraded":"2024-09-13T14:44:57.220382+0000","last_fullsized":"2024-09-13T14:44:57.220382+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:59:02.470343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":280,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.650433+0000","last_change":"2024-09-13T14:37:48.127600+0000","last_active":"2024-09-13T14:44:43.650433+0000","last_peered":"2024-09-13T14:44:43.650433+0000","last_clean":"2024-09-13T14:44:43.650433+0000","last_became_active":"2024-09-13T14:37:48.127349+0000","last_became_peered":"2024-09-13T14:37:48.127349+0000","last_unstale":"2024-09-13T14:44:43.650433+0000","last_undegraded":"2024-09-13T14:44:43.650433+0000","last_fullsized":"2024-09-13T14:44:43.650433+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:57.268374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"296'12727","reported_seq":22728,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-13T14:45:02.331258+0000","last_change":"2024-09-13T14:42:30.227259+0000","last_active":"2024-09-13T14:45:02.331258+0000","last_peered":"2024-09-13T14:45:02.331258+0000","last_clean":"2024-09-13T14:45:02.331258+0000","last_became_active":"2024-09-13T14:42:30.209102+0000","last_became_peered":"2024-09-13T14:42:30.209102+0000","last_unstale":"2024-09-13T14:45:02.331258+0000","last_undegraded":"2024-09-13T14:45:02.331258+0000","last_fullsized":"2024-09-13T14:45:02.331258+0000","mapping_epoch":281,"log_start":"220'7000","ondisk_log_start":"220'7000","created":205,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5727,"log_dups_size":0,"ondisk_log_size":5727,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:56:16.193287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84082688,"num_objects":1283,"num_object_clones":0,"num_object_copies":3849,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12727,"num_write_kb":448320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4926,"num_bytes_recovered":313851904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":214,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:47.880282+0000","last_change":"2024-09-13T14:37:48.127604+0000","last_active":"2024-09-13T14:44:47.880282+0000","last_peered":"2024-09-13T14:44:47.880282+0000","last_clean":"2024-09-13T14:44:47.880282+0000","last_became_active":"2024-09-13T14:37:48.127300+0000","last_became_peered":"2024-09-13T14:37:48.127300+0000","last_unstale":"2024-09-13T14:44:47.880282+0000","last_undegraded":"2024-09-13T14:44:47.880282+0000","last_fullsized":"2024-09-13T14:44:47.880282+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:51.999506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":333,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.650524+0000","last_change":"2024-09-13T14:38:03.529967+0000","last_active":"2024-09-13T14:44:43.650524+0000","last_peered":"2024-09-13T14:44:43.650524+0000","last_clean":"2024-09-13T14:44:43.650524+0000","last_became_active":"2024-09-13T14:38:03.529646+0000","last_became_peered":"2024-09-13T14:38:03.529646+0000","last_unstale":"2024-09-13T14:44:43.650524+0000","last_undegraded":"2024-09-13T14:44:43.650524+0000","last_fullsized":"2024-09-13T14:44:43.650524+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:52:05.193279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"296'12841","reported_seq":22036,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-13T14:45:02.211875+0000","last_change":"2024-09-13T14:40:34.583237+0000","last_active":"2024-09-13T14:45:02.211875+0000","last_peered":"2024-09-13T14:45:02.211875+0000","last_clean":"2024-09-13T14:45:02.211875+0000","last_became_active":"2024-09-13T14:40:34.563989+0000","last_became_peered":"2024-09-13T14:40:34.563989+0000","last_unstale":"2024-09-13T14:45:02.211875+0000","last_undegraded":"2024-09-13T14:45:02.211875+0000","last_fullsized":"2024-09-13T14:45:02.211875+0000","mapping_epoch":266,"log_start":"220'6900","ondisk_log_start":"220'6900","created":205,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5941,"log_dups_size":0,"ondisk_log_size":5941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:34:43.006051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85065752,"num_objects":1299,"num_object_clones":0,"num_object_copies":3897,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1299,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12841,"num_write_kb":452417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4489,"num_bytes_recovered":289865752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":294,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.650220+0000","last_change":"2024-09-13T14:36:13.860367+0000","last_active":"2024-09-13T14:44:43.650220+0000","last_peered":"2024-09-13T14:44:43.650220+0000","last_clean":"2024-09-13T14:44:43.650220+0000","last_became_active":"2024-09-13T14:36:13.859982+0000","last_became_peered":"2024-09-13T14:36:13.859982+0000","last_unstale":"2024-09-13T14:44:43.650220+0000","last_undegraded":"2024-09-13T14:44:43.650220+0000","last_fullsized":"2024-09-13T14:44:43.650220+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:08:23.311350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":289,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.650568+0000","last_change":"2024-09-13T14:37:48.147360+0000","last_active":"2024-09-13T14:44:43.650568+0000","last_peered":"2024-09-13T14:44:43.650568+0000","last_clean":"2024-09-13T14:44:43.650568+0000","last_became_active":"2024-09-13T14:37:48.146885+0000","last_became_peered":"2024-09-13T14:37:48.146885+0000","last_unstale":"2024-09-13T14:44:43.650568+0000","last_undegraded":"2024-09-13T14:44:43.650568+0000","last_fullsized":"2024-09-13T14:44:43.650568+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:27.841810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"296'12846","reported_seq":22148,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-13T14:45:02.477248+0000","last_change":"2024-09-13T14:38:39.650332+0000","last_active":"2024-09-13T14:45:02.477248+0000","last_peered":"2024-09-13T14:45:02.477248+0000","last_clean":"2024-09-13T14:45:02.477248+0000","last_became_active":"2024-09-13T14:38:39.640835+0000","last_became_peered":"2024-09-13T14:38:39.640835+0000","last_unstale":"2024-09-13T14:45:02.477248+0000","last_undegraded":"2024-09-13T14:45:02.477248+0000","last_fullsized":"2024-09-13T14:45:02.477248+0000","mapping_epoch":251,"log_start":"241'7151","ondisk_log_start":"241'7151","created":205,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5695,"log_dups_size":0,"ondisk_log_size":5695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:48.176606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80478208,"num_objects":1228,"num_object_clones":0,"num_object_copies":3684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12846,"num_write_kb":450368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4544,"num_bytes_recovered":292945920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":493,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-13T14:44:57.220937+0000","last_change":"2024-09-13T14:38:53.900041+0000","last_active":"2024-09-13T14:44:57.220937+0000","last_peered":"2024-09-13T14:44:57.220937+0000","last_clean":"2024-09-13T14:44:57.220937+0000","last_became_active":"2024-09-13T14:38:53.899203+0000","last_became_peered":"2024-09-13T14:38:53.899203+0000","last_unstale":"2024-09-13T14:44:57.220937+0000","last_undegraded":"2024-09-13T14:44:57.220937+0000","last_fullsized":"2024-09-13T14:44:57.220937+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:28:07.883094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":425,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-13T14:44:57.220264+0000","last_change":"2024-09-13T14:26:15.823198+0000","last_active":"2024-09-13T14:44:57.220264+0000","last_peered":"2024-09-13T14:44:57.220264+0000","last_clean":"2024-09-13T14:44:57.220264+0000","last_became_active":"2024-09-13T14:26:15.822717+0000","last_became_peered":"2024-09-13T14:26:15.822717+0000","last_unstale":"2024-09-13T14:44:57.220264+0000","last_undegraded":"2024-09-13T14:44:57.220264+0000","last_fullsized":"2024-09-13T14:44:57.220264+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:22.471622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.6","version":"296'12929","reported_seq":22593,"reported_epoch":296,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-13T14:45:02.011085+0000","last_change":"2024-09-13T14:38:55.132683+0000","last_active":"2024-09-13T14:45:02.011085+0000","last_peered":"2024-09-13T14:45:02.011085+0000","last_clean":"2024-09-13T14:36:13.124368+0000","last_became_active":"2024-09-13T14:38:55.120883+0000","last_became_peered":"2024-09-13T14:38:55.120883+0000","last_unstale":"2024-09-13T14:45:02.011085+0000","last_undegraded":"2024-09-13T14:38:55.026610+0000","last_fullsized":"2024-09-13T14:45:02.011085+0000","mapping_epoch":254,"log_start":"220'7019","ondisk_log_start":"220'7019","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5910,"log_dups_size":0,"ondisk_log_size":5910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83165184,"num_objects":1269,"num_object_clones":0,"num_object_copies":3807,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":279,"num_objects_misplaced":990,"num_objects_unfound":0,"num_objects_dirty":1269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12929,"num_write_kb":454336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4726,"num_bytes_recovered":298516480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,3,5],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3,5","objects":1269}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":430,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.627216+0000","last_change":"2024-09-13T14:41:53.196835+0000","last_active":"2024-09-13T14:44:43.627216+0000","last_peered":"2024-09-13T14:44:43.627216+0000","last_clean":"2024-09-13T14:44:43.627216+0000","last_became_active":"2024-09-13T14:41:53.196387+0000","last_became_peered":"2024-09-13T14:41:53.196387+0000","last_unstale":"2024-09-13T14:44:43.627216+0000","last_undegraded":"2024-09-13T14:44:43.627216+0000","last_fullsized":"2024-09-13T14:44:43.627216+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:07:41.927740+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:49.972303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":536,"reported_epoch":295,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:44:43.648581+0000","last_change":"2024-09-13T14:38:54.904329+0000","last_active":"2024-09-13T14:44:43.648581+0000","last_peered":"2024-09-13T14:44:43.648581+0000","last_clean":"2024-09-13T14:38:39.648017+0000","last_became_active":"2024-09-13T14:38:54.903798+0000","last_became_peered":"2024-09-13T14:38:54.903798+0000","last_unstale":"2024-09-13T14:44:43.648581+0000","last_undegraded":"2024-09-13T14:38:54.890505+0000","last_fullsized":"2024-09-13T14:38:54.889701+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:18.120931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,3],"avail_no_missing":["5","3","6"],"object_location_counts":[{"shards":"3,5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":427,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-13T14:44:57.219573+0000","last_change":"2024-09-13T14:37:49.160719+0000","last_active":"2024-09-13T14:44:57.219573+0000","last_peered":"2024-09-13T14:44:57.219573+0000","last_clean":"2024-09-13T14:44:57.219573+0000","last_became_active":"2024-09-13T14:37:49.160401+0000","last_became_peered":"2024-09-13T14:37:49.160401+0000","last_unstale":"2024-09-13T14:44:57.219573+0000","last_undegraded":"2024-09-13T14:44:57.219573+0000","last_fullsized":"2024-09-13T14:44:57.219573+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:21.737945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":141,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.650422+0000","last_change":"2024-09-13T14:37:48.172940+0000","last_active":"2024-09-13T14:44:43.650422+0000","last_peered":"2024-09-13T14:44:43.650422+0000","last_clean":"2024-09-13T14:44:43.650422+0000","last_became_active":"2024-09-13T14:37:48.172060+0000","last_became_peered":"2024-09-13T14:37:48.172060+0000","last_unstale":"2024-09-13T14:44:43.650422+0000","last_undegraded":"2024-09-13T14:44:43.650422+0000","last_fullsized":"2024-09-13T14:44:43.650422+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:07.480645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.5","version":"296'12923","reported_seq":29303,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-13T14:45:02.388736+0000","last_change":"2024-09-13T14:39:34.446699+0000","last_active":"2024-09-13T14:45:02.388736+0000","last_peered":"2024-09-13T14:45:02.388736+0000","last_clean":"2024-09-13T14:45:02.388736+0000","last_became_active":"2024-09-13T14:38:04.600616+0000","last_became_peered":"2024-09-13T14:38:04.600616+0000","last_unstale":"2024-09-13T14:45:02.388736+0000","last_undegraded":"2024-09-13T14:45:02.388736+0000","last_fullsized":"2024-09-13T14:45:02.388736+0000","mapping_epoch":245,"log_start":"220'6616","ondisk_log_start":"220'6616","created":205,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6307,"log_dups_size":0,"ondisk_log_size":6307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:00:44.321678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79888384,"num_objects":1219,"num_object_clones":0,"num_object_copies":3657,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12923,"num_write_kb":452544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8120,"num_bytes_recovered":521076736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":514,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.627201+0000","last_change":"2024-09-13T14:26:15.835178+0000","last_active":"2024-09-13T14:44:43.627201+0000","last_peered":"2024-09-13T14:44:43.627201+0000","last_clean":"2024-09-13T14:44:43.627201+0000","last_became_active":"2024-09-13T14:26:15.820283+0000","last_became_peered":"2024-09-13T14:26:15.820283+0000","last_unstale":"2024-09-13T14:44:43.627201+0000","last_undegraded":"2024-09-13T14:44:43.627201+0000","last_fullsized":"2024-09-13T14:44:43.627201+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:30.340038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":117,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-13T14:44:57.228914+0000","last_change":"2024-09-13T14:38:03.525682+0000","last_active":"2024-09-13T14:44:57.228914+0000","last_peered":"2024-09-13T14:44:57.228914+0000","last_clean":"2024-09-13T14:44:57.228914+0000","last_became_active":"2024-09-13T14:38:03.525406+0000","last_became_peered":"2024-09-13T14:38:03.525406+0000","last_unstale":"2024-09-13T14:44:57.228914+0000","last_undegraded":"2024-09-13T14:44:57.228914+0000","last_fullsized":"2024-09-13T14:44:57.228914+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:41:28.970300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":124,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-13T14:44:57.230146+0000","last_change":"2024-09-13T14:38:53.991738+0000","last_active":"2024-09-13T14:44:57.230146+0000","last_peered":"2024-09-13T14:44:57.230146+0000","last_clean":"2024-09-13T14:44:57.230146+0000","last_became_active":"2024-09-13T14:38:53.991417+0000","last_became_peered":"2024-09-13T14:38:53.991417+0000","last_unstale":"2024-09-13T14:44:57.230146+0000","last_undegraded":"2024-09-13T14:44:57.230146+0000","last_fullsized":"2024-09-13T14:44:57.230146+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:50:38.559838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"296'12787","reported_seq":33213,"reported_epoch":296,"state":"active+recovering","last_fresh":"2024-09-13T14:45:02.207203+0000","last_change":"2024-09-13T14:43:50.028785+0000","last_active":"2024-09-13T14:45:02.207203+0000","last_peered":"2024-09-13T14:45:02.207203+0000","last_clean":"2024-09-13T14:37:46.290754+0000","last_became_active":"2024-09-13T14:38:17.381753+0000","last_became_peered":"2024-09-13T14:38:17.381753+0000","last_unstale":"2024-09-13T14:45:02.207203+0000","last_undegraded":"2024-09-13T14:45:02.207203+0000","last_fullsized":"2024-09-13T14:45:02.207203+0000","mapping_epoch":245,"log_start":"220'6859","ondisk_log_start":"220'6859","created":205,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5928,"log_dups_size":0,"ondisk_log_size":5928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86441984,"num_objects":1319,"num_object_clones":0,"num_object_copies":3957,"num_objects_missing_on_primary":73,"num_objects_missing":73,"num_objects_degraded":0,"num_objects_misplaced":1319,"num_objects_unfound":0,"num_objects_dirty":1319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12787,"num_write_kb":451392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10157,"num_bytes_recovered":664993792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"0,4,5","objects":1319}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":838,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-13T14:44:57.220154+0000","last_change":"2024-09-13T14:37:48.131799+0000","last_active":"2024-09-13T14:44:57.220154+0000","last_peered":"2024-09-13T14:44:57.220154+0000","last_clean":"2024-09-13T14:44:57.220154+0000","last_became_active":"2024-09-13T14:37:48.130837+0000","last_became_peered":"2024-09-13T14:37:48.130837+0000","last_unstale":"2024-09-13T14:44:57.220154+0000","last_undegraded":"2024-09-13T14:44:57.220154+0000","last_fullsized":"2024-09-13T14:44:57.220154+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:06:32.663770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":357,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-13T14:44:57.229269+0000","last_change":"2024-09-13T14:37:49.301531+0000","last_active":"2024-09-13T14:44:57.229269+0000","last_peered":"2024-09-13T14:44:57.229269+0000","last_clean":"2024-09-13T14:44:57.229269+0000","last_became_active":"2024-09-13T14:37:49.301249+0000","last_became_peered":"2024-09-13T14:37:49.301249+0000","last_unstale":"2024-09-13T14:44:57.229269+0000","last_undegraded":"2024-09-13T14:44:57.229269+0000","last_fullsized":"2024-09-13T14:44:57.229269+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:10.977710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":123,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-13T14:44:57.218954+0000","last_change":"2024-09-13T14:38:53.901427+0000","last_active":"2024-09-13T14:44:57.218954+0000","last_peered":"2024-09-13T14:44:57.218954+0000","last_clean":"2024-09-13T14:44:57.218954+0000","last_became_active":"2024-09-13T14:38:53.901157+0000","last_became_peered":"2024-09-13T14:38:53.901157+0000","last_unstale":"2024-09-13T14:44:57.218954+0000","last_undegraded":"2024-09-13T14:44:57.218954+0000","last_fullsized":"2024-09-13T14:44:57.218954+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:28:21.713025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"296'12773","reported_seq":36029,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-13T14:45:02.120392+0000","last_change":"2024-09-13T14:43:16.851222+0000","last_active":"2024-09-13T14:45:02.120392+0000","last_peered":"2024-09-13T14:45:02.120392+0000","last_clean":"2024-09-13T14:45:02.120392+0000","last_became_active":"2024-09-13T14:38:16.427510+0000","last_became_peered":"2024-09-13T14:38:16.427510+0000","last_unstale":"2024-09-13T14:45:02.120392+0000","last_undegraded":"2024-09-13T14:45:02.120392+0000","last_fullsized":"2024-09-13T14:45:02.120392+0000","mapping_epoch":245,"log_start":"220'6818","ondisk_log_start":"220'6818","created":205,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5955,"log_dups_size":0,"ondisk_log_size":5955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:27:26.281938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80281600,"num_objects":1225,"num_object_clones":0,"num_object_copies":3675,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12773,"num_write_kb":447936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11571,"num_bytes_recovered":747372544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":1005,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.647299+0000","last_change":"2024-09-13T14:37:49.147498+0000","last_active":"2024-09-13T14:44:43.647299+0000","last_peered":"2024-09-13T14:44:43.647299+0000","last_clean":"2024-09-13T14:44:43.647299+0000","last_became_active":"2024-09-13T14:37:49.147028+0000","last_became_peered":"2024-09-13T14:37:49.147028+0000","last_unstale":"2024-09-13T14:44:43.647299+0000","last_undegraded":"2024-09-13T14:44:43.647299+0000","last_fullsized":"2024-09-13T14:44:43.647299+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:11:29.050178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":303,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:52.881207+0000","last_change":"2024-09-13T14:38:03.528133+0000","last_active":"2024-09-13T14:44:52.881207+0000","last_peered":"2024-09-13T14:44:52.881207+0000","last_clean":"2024-09-13T14:44:52.881207+0000","last_became_active":"2024-09-13T14:38:03.527829+0000","last_became_peered":"2024-09-13T14:38:03.527829+0000","last_unstale":"2024-09-13T14:44:52.881207+0000","last_undegraded":"2024-09-13T14:44:52.881207+0000","last_fullsized":"2024-09-13T14:44:52.881207+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:48.254169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":477,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-13T14:44:57.219569+0000","last_change":"2024-09-13T14:38:53.899755+0000","last_active":"2024-09-13T14:44:57.219569+0000","last_peered":"2024-09-13T14:44:57.219569+0000","last_clean":"2024-09-13T14:44:57.219569+0000","last_became_active":"2024-09-13T14:38:53.899448+0000","last_became_peered":"2024-09-13T14:38:53.899448+0000","last_unstale":"2024-09-13T14:44:57.219569+0000","last_undegraded":"2024-09-13T14:44:57.219569+0000","last_fullsized":"2024-09-13T14:44:57.219569+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:55.897035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"296'13077","reported_seq":22215,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-13T14:45:02.767309+0000","last_change":"2024-09-13T14:43:51.649951+0000","last_active":"2024-09-13T14:45:02.767309+0000","last_peered":"2024-09-13T14:45:02.767309+0000","last_clean":"2024-09-13T14:45:02.767309+0000","last_became_active":"2024-09-13T14:43:51.626628+0000","last_became_peered":"2024-09-13T14:43:51.626628+0000","last_unstale":"2024-09-13T14:45:02.767309+0000","last_undegraded":"2024-09-13T14:45:02.767309+0000","last_fullsized":"2024-09-13T14:45:02.767309+0000","mapping_epoch":289,"log_start":"219'6163","ondisk_log_start":"219'6163","created":205,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6914,"log_dups_size":0,"ondisk_log_size":6914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:37:06.795173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80281600,"num_objects":1225,"num_object_clones":0,"num_object_copies":3675,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13077,"num_write_kb":457664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4502,"num_bytes_recovered":228851712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":107,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.647246+0000","last_change":"2024-09-13T14:37:49.149770+0000","last_active":"2024-09-13T14:44:43.647246+0000","last_peered":"2024-09-13T14:44:43.647246+0000","last_clean":"2024-09-13T14:44:43.647246+0000","last_became_active":"2024-09-13T14:37:49.148991+0000","last_became_peered":"2024-09-13T14:37:49.148991+0000","last_unstale":"2024-09-13T14:44:43.647246+0000","last_undegraded":"2024-09-13T14:44:43.647246+0000","last_fullsized":"2024-09-13T14:44:43.647246+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:04:04.193366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":1175,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-13T14:44:57.239472+0000","last_change":"2024-09-13T14:30:58.928302+0000","last_active":"2024-09-13T14:44:57.239472+0000","last_peered":"2024-09-13T14:44:57.239472+0000","last_clean":"2024-09-13T14:44:57.239472+0000","last_became_active":"2024-09-13T14:30:58.927718+0000","last_became_peered":"2024-09-13T14:30:58.927718+0000","last_unstale":"2024-09-13T14:44:57.239472+0000","last_undegraded":"2024-09-13T14:44:57.239472+0000","last_fullsized":"2024-09-13T14:44:57.239472+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:37.323036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":114,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.647523+0000","last_change":"2024-09-13T14:38:03.534628+0000","last_active":"2024-09-13T14:44:43.647523+0000","last_peered":"2024-09-13T14:44:43.647523+0000","last_clean":"2024-09-13T14:44:43.647523+0000","last_became_active":"2024-09-13T14:38:03.534262+0000","last_became_peered":"2024-09-13T14:38:03.534262+0000","last_unstale":"2024-09-13T14:44:43.647523+0000","last_undegraded":"2024-09-13T14:44:43.647523+0000","last_fullsized":"2024-09-13T14:44:43.647523+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:31.168783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"296'12795","reported_seq":26979,"reported_epoch":296,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:45:02.315685+0000","last_change":"2024-09-13T14:38:53.944111+0000","last_active":"2024-09-13T14:45:02.315685+0000","last_peered":"2024-09-13T14:45:02.315685+0000","last_clean":"2024-09-13T14:37:47.047672+0000","last_became_active":"2024-09-13T14:37:49.664294+0000","last_became_peered":"2024-09-13T14:37:49.664294+0000","last_unstale":"2024-09-13T14:45:02.315685+0000","last_undegraded":"2024-09-13T14:38:53.940822+0000","last_fullsized":"2024-09-13T14:37:49.126627+0000","mapping_epoch":240,"log_start":"238'7100","ondisk_log_start":"238'7100","created":205,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5695,"log_dups_size":0,"ondisk_log_size":5695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:11.766692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82378752,"num_objects":1257,"num_object_clones":0,"num_object_copies":3771,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4535,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12795,"num_write_kb":449664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7009,"num_bytes_recovered":459341824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3],"avail_no_missing":["5","0","3"],"object_location_counts":[{"shards":"0,3,5","objects":1257},{"shards":"3,5","objects":-3203}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":126,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.647282+0000","last_change":"2024-09-13T14:38:03.534303+0000","last_active":"2024-09-13T14:44:43.647282+0000","last_peered":"2024-09-13T14:44:43.647282+0000","last_clean":"2024-09-13T14:44:43.647282+0000","last_became_active":"2024-09-13T14:38:03.533171+0000","last_became_peered":"2024-09-13T14:38:03.533171+0000","last_unstale":"2024-09-13T14:44:43.647282+0000","last_undegraded":"2024-09-13T14:44:43.647282+0000","last_fullsized":"2024-09-13T14:44:43.647282+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:00:16.553526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":818,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.626529+0000","last_change":"2024-09-13T14:38:04.596192+0000","last_active":"2024-09-13T14:44:43.626529+0000","last_peered":"2024-09-13T14:44:43.626529+0000","last_clean":"2024-09-13T14:44:43.626529+0000","last_became_active":"2024-09-13T14:38:04.593299+0000","last_became_peered":"2024-09-13T14:38:04.593299+0000","last_unstale":"2024-09-13T14:44:43.626529+0000","last_undegraded":"2024-09-13T14:44:43.626529+0000","last_fullsized":"2024-09-13T14:44:43.626529+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:34.241410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":117,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-13T14:44:57.230216+0000","last_change":"2024-09-13T14:38:03.533120+0000","last_active":"2024-09-13T14:44:57.230216+0000","last_peered":"2024-09-13T14:44:57.230216+0000","last_clean":"2024-09-13T14:44:57.230216+0000","last_became_active":"2024-09-13T14:38:03.532829+0000","last_became_peered":"2024-09-13T14:38:03.532829+0000","last_unstale":"2024-09-13T14:44:57.230216+0000","last_undegraded":"2024-09-13T14:44:57.230216+0000","last_fullsized":"2024-09-13T14:44:57.230216+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:24.514661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"296'13070","reported_seq":22351,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-13T14:45:02.512611+0000","last_change":"2024-09-13T14:42:04.660877+0000","last_active":"2024-09-13T14:45:02.512611+0000","last_peered":"2024-09-13T14:45:02.512611+0000","last_clean":"2024-09-13T14:45:02.512611+0000","last_became_active":"2024-09-13T14:42:04.641680+0000","last_became_peered":"2024-09-13T14:42:04.641680+0000","last_unstale":"2024-09-13T14:45:02.512611+0000","last_undegraded":"2024-09-13T14:45:02.512611+0000","last_fullsized":"2024-09-13T14:45:02.512611+0000","mapping_epoch":277,"log_start":"241'7318","ondisk_log_start":"241'7318","created":205,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5752,"log_dups_size":0,"ondisk_log_size":5752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:22:18.955428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87031808,"num_objects":1328,"num_object_clones":0,"num_object_copies":3984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13070,"num_write_kb":460736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4533,"num_bytes_recovered":297074688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":519,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-13T14:44:57.220348+0000","last_change":"2024-09-13T14:38:53.938227+0000","last_active":"2024-09-13T14:44:57.220348+0000","last_peered":"2024-09-13T14:44:57.220348+0000","last_clean":"2024-09-13T14:44:57.220348+0000","last_became_active":"2024-09-13T14:38:53.902867+0000","last_became_peered":"2024-09-13T14:38:53.902867+0000","last_unstale":"2024-09-13T14:44:57.220348+0000","last_undegraded":"2024-09-13T14:44:57.220348+0000","last_fullsized":"2024-09-13T14:44:57.220348+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:16.831756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":597,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:47.880359+0000","last_change":"2024-09-13T14:37:48.147677+0000","last_active":"2024-09-13T14:44:47.880359+0000","last_peered":"2024-09-13T14:44:47.880359+0000","last_clean":"2024-09-13T14:44:47.880359+0000","last_became_active":"2024-09-13T14:37:48.147001+0000","last_became_peered":"2024-09-13T14:37:48.147001+0000","last_unstale":"2024-09-13T14:44:47.880359+0000","last_undegraded":"2024-09-13T14:44:47.880359+0000","last_fullsized":"2024-09-13T14:44:47.880359+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:35:00.653576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":501,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-13T14:44:57.219873+0000","last_change":"2024-09-13T14:38:53.900107+0000","last_active":"2024-09-13T14:44:57.219873+0000","last_peered":"2024-09-13T14:44:57.219873+0000","last_clean":"2024-09-13T14:44:57.219873+0000","last_became_active":"2024-09-13T14:38:53.899556+0000","last_became_peered":"2024-09-13T14:38:53.899556+0000","last_unstale":"2024-09-13T14:44:57.219873+0000","last_undegraded":"2024-09-13T14:44:57.219873+0000","last_fullsized":"2024-09-13T14:44:57.219873+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:15:47.918163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"296'12630","reported_seq":21637,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-13T14:45:02.461039+0000","last_change":"2024-09-13T14:39:54.921239+0000","last_active":"2024-09-13T14:45:02.461039+0000","last_peered":"2024-09-13T14:45:02.461039+0000","last_clean":"2024-09-13T14:45:02.461039+0000","last_became_active":"2024-09-13T14:39:54.907894+0000","last_became_peered":"2024-09-13T14:39:54.907894+0000","last_unstale":"2024-09-13T14:45:02.461039+0000","last_undegraded":"2024-09-13T14:45:02.461039+0000","last_fullsized":"2024-09-13T14:45:02.461039+0000","mapping_epoch":263,"log_start":"233'6947","ondisk_log_start":"233'6947","created":205,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5683,"log_dups_size":0,"ondisk_log_size":5683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:52:04.619238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80084992,"num_objects":1222,"num_object_clones":0,"num_object_copies":3666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12630,"num_write_kb":443264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4395,"num_bytes_recovered":283770880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":132,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-13T14:44:57.229962+0000","last_change":"2024-09-13T14:38:53.901806+0000","last_active":"2024-09-13T14:44:57.229962+0000","last_peered":"2024-09-13T14:44:57.229962+0000","last_clean":"2024-09-13T14:44:57.229962+0000","last_became_active":"2024-09-13T14:38:53.891055+0000","last_became_peered":"2024-09-13T14:38:53.891055+0000","last_unstale":"2024-09-13T14:44:57.229962+0000","last_undegraded":"2024-09-13T14:44:57.229962+0000","last_fullsized":"2024-09-13T14:44:57.229962+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:03:00.416198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":71,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.626729+0000","last_change":"2024-09-13T14:43:16.854299+0000","last_active":"2024-09-13T14:44:43.626729+0000","last_peered":"2024-09-13T14:44:43.626729+0000","last_clean":"2024-09-13T14:44:43.626729+0000","last_became_active":"2024-09-13T14:38:53.900626+0000","last_became_peered":"2024-09-13T14:38:53.900626+0000","last_unstale":"2024-09-13T14:44:43.626729+0000","last_undegraded":"2024-09-13T14:44:43.626729+0000","last_fullsized":"2024-09-13T14:44:43.626729+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:37:21.715778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":287,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:47.880257+0000","last_change":"2024-09-13T14:37:49.172218+0000","last_active":"2024-09-13T14:44:47.880257+0000","last_peered":"2024-09-13T14:44:47.880257+0000","last_clean":"2024-09-13T14:44:47.880257+0000","last_became_active":"2024-09-13T14:37:49.171718+0000","last_became_peered":"2024-09-13T14:37:49.171718+0000","last_unstale":"2024-09-13T14:44:47.880257+0000","last_undegraded":"2024-09-13T14:44:47.880257+0000","last_fullsized":"2024-09-13T14:44:47.880257+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:46:54.626521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":309,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.650676+0000","last_change":"2024-09-13T14:37:49.230094+0000","last_active":"2024-09-13T14:44:43.650676+0000","last_peered":"2024-09-13T14:44:43.650676+0000","last_clean":"2024-09-13T14:44:43.650676+0000","last_became_active":"2024-09-13T14:37:49.229393+0000","last_became_peered":"2024-09-13T14:37:49.229393+0000","last_unstale":"2024-09-13T14:44:43.650676+0000","last_undegraded":"2024-09-13T14:44:43.650676+0000","last_fullsized":"2024-09-13T14:44:43.650676+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:02:28.690318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":289,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.650525+0000","last_change":"2024-09-13T14:37:48.128447+0000","last_active":"2024-09-13T14:44:43.650525+0000","last_peered":"2024-09-13T14:44:43.650525+0000","last_clean":"2024-09-13T14:44:43.650525+0000","last_became_active":"2024-09-13T14:37:48.128143+0000","last_became_peered":"2024-09-13T14:37:48.128143+0000","last_unstale":"2024-09-13T14:44:43.650525+0000","last_undegraded":"2024-09-13T14:44:43.650525+0000","last_fullsized":"2024-09-13T14:44:43.650525+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:22.204039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":116,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-13T14:44:57.230410+0000","last_change":"2024-09-13T14:38:03.531683+0000","last_active":"2024-09-13T14:44:57.230410+0000","last_peered":"2024-09-13T14:44:57.230410+0000","last_clean":"2024-09-13T14:44:57.230410+0000","last_became_active":"2024-09-13T14:38:03.531476+0000","last_became_peered":"2024-09-13T14:38:03.531476+0000","last_unstale":"2024-09-13T14:44:57.230410+0000","last_undegraded":"2024-09-13T14:44:57.230410+0000","last_fullsized":"2024-09-13T14:44:57.230410+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:41.557569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":61,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.626221+0000","last_change":"2024-09-13T14:38:53.954281+0000","last_active":"2024-09-13T14:44:43.626221+0000","last_peered":"2024-09-13T14:44:43.626221+0000","last_clean":"2024-09-13T14:44:43.626221+0000","last_became_active":"2024-09-13T14:38:53.900912+0000","last_became_peered":"2024-09-13T14:38:53.900912+0000","last_unstale":"2024-09-13T14:44:43.626221+0000","last_undegraded":"2024-09-13T14:44:43.626221+0000","last_fullsized":"2024-09-13T14:44:43.626221+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:47.876275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":113,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.647751+0000","last_change":"2024-09-13T14:38:54.000782+0000","last_active":"2024-09-13T14:44:43.647751+0000","last_peered":"2024-09-13T14:44:43.647751+0000","last_clean":"2024-09-13T14:44:43.647751+0000","last_became_active":"2024-09-13T14:38:54.000322+0000","last_became_peered":"2024-09-13T14:38:54.000322+0000","last_unstale":"2024-09-13T14:44:43.647751+0000","last_undegraded":"2024-09-13T14:44:43.647751+0000","last_fullsized":"2024-09-13T14:44:43.647751+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:09.057597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":442,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-13T14:44:57.219828+0000","last_change":"2024-09-13T14:38:53.899535+0000","last_active":"2024-09-13T14:44:57.219828+0000","last_peered":"2024-09-13T14:44:57.219828+0000","last_clean":"2024-09-13T14:44:57.219828+0000","last_became_active":"2024-09-13T14:38:53.899183+0000","last_became_peered":"2024-09-13T14:38:53.899183+0000","last_unstale":"2024-09-13T14:44:57.219828+0000","last_undegraded":"2024-09-13T14:44:57.219828+0000","last_fullsized":"2024-09-13T14:44:57.219828+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:33.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":116,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-13T14:44:57.230237+0000","last_change":"2024-09-13T14:38:03.543426+0000","last_active":"2024-09-13T14:44:57.230237+0000","last_peered":"2024-09-13T14:44:57.230237+0000","last_clean":"2024-09-13T14:44:57.230237+0000","last_became_active":"2024-09-13T14:38:03.542661+0000","last_became_peered":"2024-09-13T14:38:03.542661+0000","last_unstale":"2024-09-13T14:44:57.230237+0000","last_undegraded":"2024-09-13T14:44:57.230237+0000","last_fullsized":"2024-09-13T14:44:57.230237+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:54.050127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":471,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-13T14:44:57.219739+0000","last_change":"2024-09-13T14:37:48.131321+0000","last_active":"2024-09-13T14:44:57.219739+0000","last_peered":"2024-09-13T14:44:57.219739+0000","last_clean":"2024-09-13T14:44:57.219739+0000","last_became_active":"2024-09-13T14:37:48.130998+0000","last_became_peered":"2024-09-13T14:37:48.130998+0000","last_unstale":"2024-09-13T14:44:57.219739+0000","last_undegraded":"2024-09-13T14:44:57.219739+0000","last_fullsized":"2024-09-13T14:44:57.219739+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:33:26.597825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":126,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.647783+0000","last_change":"2024-09-13T14:38:03.535454+0000","last_active":"2024-09-13T14:44:43.647783+0000","last_peered":"2024-09-13T14:44:43.647783+0000","last_clean":"2024-09-13T14:44:43.647783+0000","last_became_active":"2024-09-13T14:38:03.535131+0000","last_became_peered":"2024-09-13T14:38:03.535131+0000","last_unstale":"2024-09-13T14:44:43.647783+0000","last_undegraded":"2024-09-13T14:44:43.647783+0000","last_fullsized":"2024-09-13T14:44:43.647783+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:24.477601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":496,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-13T14:44:57.220839+0000","last_change":"2024-09-13T14:38:03.524939+0000","last_active":"2024-09-13T14:44:57.220839+0000","last_peered":"2024-09-13T14:44:57.220839+0000","last_clean":"2024-09-13T14:44:57.220839+0000","last_became_active":"2024-09-13T14:38:03.524406+0000","last_became_peered":"2024-09-13T14:38:03.524406+0000","last_unstale":"2024-09-13T14:44:57.220839+0000","last_undegraded":"2024-09-13T14:44:57.220839+0000","last_fullsized":"2024-09-13T14:44:57.220839+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:19:45.257424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":124,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-13T14:44:57.229871+0000","last_change":"2024-09-13T14:38:53.892948+0000","last_active":"2024-09-13T14:44:57.229871+0000","last_peered":"2024-09-13T14:44:57.229871+0000","last_clean":"2024-09-13T14:44:57.229871+0000","last_became_active":"2024-09-13T14:38:53.892665+0000","last_became_peered":"2024-09-13T14:38:53.892665+0000","last_unstale":"2024-09-13T14:44:57.229871+0000","last_undegraded":"2024-09-13T14:44:57.229871+0000","last_fullsized":"2024-09-13T14:44:57.229871+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:08.292359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":444,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-13T14:44:57.220059+0000","last_change":"2024-09-13T14:37:48.139348+0000","last_active":"2024-09-13T14:44:57.220059+0000","last_peered":"2024-09-13T14:44:57.220059+0000","last_clean":"2024-09-13T14:44:57.220059+0000","last_became_active":"2024-09-13T14:37:48.139115+0000","last_became_peered":"2024-09-13T14:37:48.139115+0000","last_unstale":"2024-09-13T14:44:57.220059+0000","last_undegraded":"2024-09-13T14:44:57.220059+0000","last_fullsized":"2024-09-13T14:44:57.220059+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:32:20.413346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":485,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-13T14:44:57.219963+0000","last_change":"2024-09-13T14:37:48.172088+0000","last_active":"2024-09-13T14:44:57.219963+0000","last_peered":"2024-09-13T14:44:57.219963+0000","last_clean":"2024-09-13T14:44:57.219963+0000","last_became_active":"2024-09-13T14:37:48.171779+0000","last_became_peered":"2024-09-13T14:37:48.171779+0000","last_unstale":"2024-09-13T14:44:57.219963+0000","last_undegraded":"2024-09-13T14:44:57.219963+0000","last_fullsized":"2024-09-13T14:44:57.219963+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:02.852712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":353,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.650406+0000","last_change":"2024-09-13T14:38:53.890236+0000","last_active":"2024-09-13T14:44:43.650406+0000","last_peered":"2024-09-13T14:44:43.650406+0000","last_clean":"2024-09-13T14:44:43.650406+0000","last_became_active":"2024-09-13T14:38:53.889582+0000","last_became_peered":"2024-09-13T14:38:53.889582+0000","last_unstale":"2024-09-13T14:44:43.650406+0000","last_undegraded":"2024-09-13T14:44:43.650406+0000","last_fullsized":"2024-09-13T14:44:43.650406+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:14:29.334774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":471,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-13T14:44:57.219918+0000","last_change":"2024-09-13T14:38:03.534097+0000","last_active":"2024-09-13T14:44:57.219918+0000","last_peered":"2024-09-13T14:44:57.219918+0000","last_clean":"2024-09-13T14:44:57.219918+0000","last_became_active":"2024-09-13T14:38:03.533794+0000","last_became_peered":"2024-09-13T14:38:03.533794+0000","last_unstale":"2024-09-13T14:44:57.219918+0000","last_undegraded":"2024-09-13T14:44:57.219918+0000","last_fullsized":"2024-09-13T14:44:57.219918+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:23.272098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":116,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-13T14:44:57.229691+0000","last_change":"2024-09-13T14:38:03.527021+0000","last_active":"2024-09-13T14:44:57.229691+0000","last_peered":"2024-09-13T14:44:57.229691+0000","last_clean":"2024-09-13T14:44:57.229691+0000","last_became_active":"2024-09-13T14:38:03.526698+0000","last_became_peered":"2024-09-13T14:38:03.526698+0000","last_unstale":"2024-09-13T14:44:57.229691+0000","last_undegraded":"2024-09-13T14:44:57.229691+0000","last_fullsized":"2024-09-13T14:44:57.229691+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:01:35.734086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":297,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.650592+0000","last_change":"2024-09-13T14:27:12.834931+0000","last_active":"2024-09-13T14:44:43.650592+0000","last_peered":"2024-09-13T14:44:43.650592+0000","last_clean":"2024-09-13T14:44:43.650592+0000","last_became_active":"2024-09-13T14:27:12.834572+0000","last_became_peered":"2024-09-13T14:27:12.834572+0000","last_unstale":"2024-09-13T14:44:43.650592+0000","last_undegraded":"2024-09-13T14:44:43.650592+0000","last_fullsized":"2024-09-13T14:44:43.650592+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:54:21.155689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":110,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.647189+0000","last_change":"2024-09-13T14:37:49.176796+0000","last_active":"2024-09-13T14:44:43.647189+0000","last_peered":"2024-09-13T14:44:43.647189+0000","last_clean":"2024-09-13T14:44:43.647189+0000","last_became_active":"2024-09-13T14:37:49.176246+0000","last_became_peered":"2024-09-13T14:37:49.176246+0000","last_unstale":"2024-09-13T14:44:43.647189+0000","last_undegraded":"2024-09-13T14:44:43.647189+0000","last_fullsized":"2024-09-13T14:44:43.647189+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:29:24.936718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":308,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.650231+0000","last_change":"2024-09-13T14:38:03.526333+0000","last_active":"2024-09-13T14:44:43.650231+0000","last_peered":"2024-09-13T14:44:43.650231+0000","last_clean":"2024-09-13T14:44:43.650231+0000","last_became_active":"2024-09-13T14:38:03.526001+0000","last_became_peered":"2024-09-13T14:38:03.526001+0000","last_unstale":"2024-09-13T14:44:43.650231+0000","last_undegraded":"2024-09-13T14:44:43.650231+0000","last_fullsized":"2024-09-13T14:44:43.650231+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:25:25.181271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":110,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.647320+0000","last_change":"2024-09-13T14:37:49.149582+0000","last_active":"2024-09-13T14:44:43.647320+0000","last_peered":"2024-09-13T14:44:43.647320+0000","last_clean":"2024-09-13T14:44:43.647320+0000","last_became_active":"2024-09-13T14:37:49.148525+0000","last_became_peered":"2024-09-13T14:37:49.148525+0000","last_unstale":"2024-09-13T14:44:43.647320+0000","last_undegraded":"2024-09-13T14:44:43.647320+0000","last_fullsized":"2024-09-13T14:44:43.647320+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:48.655665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":517,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-13T14:44:57.220558+0000","last_change":"2024-09-13T14:38:03.535255+0000","last_active":"2024-09-13T14:44:57.220558+0000","last_peered":"2024-09-13T14:44:57.220558+0000","last_clean":"2024-09-13T14:44:57.220558+0000","last_became_active":"2024-09-13T14:38:03.533917+0000","last_became_peered":"2024-09-13T14:38:03.533917+0000","last_unstale":"2024-09-13T14:44:57.220558+0000","last_undegraded":"2024-09-13T14:44:57.220558+0000","last_fullsized":"2024-09-13T14:44:57.220558+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:33.559788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":481,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-13T14:44:57.221170+0000","last_change":"2024-09-13T14:37:48.146784+0000","last_active":"2024-09-13T14:44:57.221170+0000","last_peered":"2024-09-13T14:44:57.221170+0000","last_clean":"2024-09-13T14:44:57.221170+0000","last_became_active":"2024-09-13T14:37:48.146491+0000","last_became_peered":"2024-09-13T14:37:48.146491+0000","last_unstale":"2024-09-13T14:44:57.221170+0000","last_undegraded":"2024-09-13T14:44:57.221170+0000","last_fullsized":"2024-09-13T14:44:57.221170+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:30:54.772513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":288,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-09-13T14:44:43.650146+0000","last_change":"2024-09-13T14:37:48.127607+0000","last_active":"2024-09-13T14:44:43.650146+0000","last_peered":"2024-09-13T14:44:43.650146+0000","last_clean":"2024-09-13T14:44:43.650146+0000","last_became_active":"2024-09-13T14:37:48.127316+0000","last_became_peered":"2024-09-13T14:37:48.127316+0000","last_unstale":"2024-09-13T14:44:43.650146+0000","last_undegraded":"2024-09-13T14:44:43.650146+0000","last_fullsized":"2024-09-13T14:44:43.650146+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:53:37.678861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":123,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-09-13T14:44:57.230037+0000","last_change":"2024-09-13T14:38:03.563200+0000","last_active":"2024-09-13T14:44:57.230037+0000","last_peered":"2024-09-13T14:44:57.230037+0000","last_clean":"2024-09-13T14:44:57.230037+0000","last_became_active":"2024-09-13T14:38:03.534258+0000","last_became_peered":"2024-09-13T14:38:03.534258+0000","last_unstale":"2024-09-13T14:44:57.230037+0000","last_undegraded":"2024-09-13T14:44:57.230037+0000","last_fullsized":"2024-09-13T14:44:57.230037+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:49:07.814419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":1324744728,"num_objects":20215,"num_object_clones":0,"num_object_copies":60645,"num_objects_missing_on_primary":73,"num_objects_missing":73,"num_objects_degraded":4814,"num_objects_misplaced":2309,"num_objects_unfound":0,"num_objects_dirty":20215,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":205411,"num_write_kb":7219969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85981,"num_bytes_recovered":5507186712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6257192960,"data_stored":6257180744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95034,"ondisk_log_size":95034,"up":48,"acting":47,"num_store_stats":6},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1538,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":17390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25132,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16793600,"data_stored":16777292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":23,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2969600,"data_stored":2951840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234414,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1289864,"kb_used_data":821504,"kb_used_omap":1,"kb_used_meta":468350,"kb_avail":92455288,"statfs":{"total":95995035648,"available":94674214912,"internally_reserved":0,"allocated":841220096,"data_stored":839643753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":479590859},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":239,"seq":1026497183832,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1006988,"kb_used_data":679620,"kb_used_omap":8,"kb_used_meta":327287,"kb_avail":92738164,"statfs":{"total":95995035648,"available":94963879936,"internally_reserved":0,"allocated":695930880,"data_stored":694351012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8453,"internal_metadata":335142651},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757679,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1525240,"kb_used_data":862448,"kb_used_omap":1,"kb_used_meta":662782,"kb_avail":92219912,"statfs":{"total":95995035648,"available":94433189888,"internally_reserved":0,"allocated":883146752,"data_stored":881581970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":678689222},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019323,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1487232,"kb_used_data":870328,"kb_used_omap":1,"kb_used_meta":616830,"kb_avail":92257920,"statfs":{"total":95995035648,"available":94472110080,"internally_reserved":0,"allocated":891215872,"data_stored":889647165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":631634374},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435506,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1574132,"kb_used_data":974636,"kb_used_omap":5,"kb_used_meta":599482,"kb_avail":92171020,"statfs":{"total":95995035648,"available":94383124480,"internally_reserved":0,"allocated":998027264,"data_stored":996466342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":613869694},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":446676599034,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":425201762560,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926084,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2669764,"kb_used_data":1970300,"kb_used_omap":7,"kb_used_meta":699448,"kb_avail":91075388,"statfs":{"total":95995035648,"available":93261197312,"internally_reserved":0,"allocated":2017587200,"data_stored":2016011061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":716235048},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,0,2,2,0,9],"upper_bound":1024},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6860,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2008354816,"data_stored":2008350744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":988483584,"data_stored":988479512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":882114560,"data_stored":882114560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":866385920,"data_stored":866385920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":685445120,"data_stored":685441048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":826408960,"data_stored":826408960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:45:06.324 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph status --format=json 2024-09-13T14:45:06.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:06.325 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:06.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:06.426 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:06.451 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:06.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:06.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:06.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:06 smithi195 ceph-mon[104262]: from='client.36120 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:45:06.553 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:06.577 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:06.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:06.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:06.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:06 smithi102 ceph-mon[110183]: from='client.36120 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:45:06.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:06 smithi102 ceph-mon[115269]: from='client.36120 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:45:06.669 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:45:06 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:45:06] "GET /metrics HTTP/1.1" 200 36547 "" "Prometheus/2.51.0" 2024-09-13T14:45:06.679 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:06.705 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:06.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:06.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:06.807 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:06.810 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:06.829 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:06.843 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:06.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:06.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:06.848 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:06.946 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:06.974 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:06.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:06.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:07.075 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:07.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:07.102 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:07.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:07.203 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:07.226 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:07.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:07.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:07.328 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:07.356 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:07.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:07.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:07.457 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:07.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:07.485 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:07.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:07.546 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:07 smithi102 ceph-mon[110183]: pgmap v1039: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.2 GiB data, 9.1 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 479 op/s; 4823/60729 objects degraded (7.942%); 2309/60729 objects misplaced (3.802%); 1.8 MiB/s, 29 objects/s recovering 2024-09-13T14:45:07.546 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:07 smithi102 ceph-mon[115269]: pgmap v1039: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.2 GiB data, 9.1 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 479 op/s; 4823/60729 objects degraded (7.942%); 2309/60729 objects misplaced (3.802%); 1.8 MiB/s, 29 objects/s recovering 2024-09-13T14:45:07.587 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:07.608 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:07.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:07.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:07.710 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:07.736 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:07.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:07.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:07.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:07 smithi195 ceph-mon[104262]: pgmap v1039: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.2 GiB data, 9.1 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 479 op/s; 4823/60729 objects degraded (7.942%); 2309/60729 objects misplaced (3.802%); 1.8 MiB/s, 29 objects/s recovering 2024-09-13T14:45:07.837 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:07.864 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:07.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:07.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:07.966 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:07.996 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:07.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:07.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:08.098 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:08.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:08.133 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:08.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:08.234 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:08.271 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:08.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:08.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:08.373 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:08.396 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:08.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:08.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:08.498 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:08.522 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:08.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:08.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:08.624 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:08.649 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:08.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:08.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:08.750 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:08.778 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:08.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:08.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:08.880 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:08.904 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:08.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:08.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:09.007 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:09.031 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:09.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:09.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:09.133 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:09.157 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:09.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:09.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:09.259 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:09.288 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:09.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:09.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:09.390 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:09.416 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:09.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:09.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:09.518 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:09.544 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:09.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:09.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:09.636 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:09 smithi102 ceph-mon[115269]: pgmap v1040: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.1 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 573 op/s; 4823/54630 objects degraded (8.828%); 1943/54630 objects misplaced (3.557%); 1.9 MiB/s, 31 objects/s recovering 2024-09-13T14:45:09.637 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:09 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 4823/54630 objects degraded (8.828%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:09.637 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:09 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:45:09.637 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:09 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:45:09.637 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:09 smithi102 ceph-mon[110183]: pgmap v1040: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.1 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 573 op/s; 4823/54630 objects degraded (8.828%); 1943/54630 objects misplaced (3.557%); 1.9 MiB/s, 31 objects/s recovering 2024-09-13T14:45:09.637 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:09 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 4823/54630 objects degraded (8.828%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:09.637 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:09 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:45:09.637 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:09 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:45:09.646 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:09.675 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:09.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:09.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:09.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:09 smithi195 ceph-mon[104262]: pgmap v1040: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.1 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 573 op/s; 4823/54630 objects degraded (8.828%); 1943/54630 objects misplaced (3.557%); 1.9 MiB/s, 31 objects/s recovering 2024-09-13T14:45:09.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:09 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 4823/54630 objects degraded (8.828%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:09.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:09 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:45:09.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:09 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:45:09.777 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:09.799 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:09.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:09.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:09.900 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:09.924 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:09.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:09.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:10.026 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:10.051 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:10.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:10.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:10.152 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:10.178 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:10.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:10.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:10.280 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:10.307 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:10.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:10.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:10.394 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:45:10.395 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:45:10.408 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:10.421 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:45:10.421 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:45:10.437 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:10.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:10.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:10.439 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:45:10.439 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:45:10.463 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:45:10.464 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:45:10.485 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:45:10.485 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:45:10.509 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:45:10.509 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:45:10.509 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:45:10.510 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:45:10.510 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:45:10.510 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:45:10.510 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:45:10.510 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:45:10.535 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:45:10.535 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:45:10.539 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:10.559 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:45:10.559 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:45:10.570 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:10.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:10.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:10.601 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:45:10.601 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:45:10.601 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:45:10.672 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:10.700 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:10.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:10.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:10.802 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:10.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:10.829 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:10.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:10.930 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:10.956 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:10.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:10.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:11.057 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:11.086 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:11.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:11.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:11.187 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:11.214 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:11.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:11.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:11.316 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:11.339 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:11.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:11.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:11.441 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:11.467 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:11.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:11.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:11.568 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:11.600 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:11.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:11.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:11.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:11 smithi102 ceph-mon[110183]: pgmap v1041: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.1 GiB data, 8.6 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 507 op/s; 4823/52053 objects degraded (9.266%); 1943/52053 objects misplaced (3.733%); 1.7 MiB/s, 27 objects/s recovering 2024-09-13T14:45:11.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:11 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/502295460' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:11.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:11 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:45:11.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:11 smithi102 ceph-mon[115269]: pgmap v1041: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.1 GiB data, 8.6 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 507 op/s; 4823/52053 objects degraded (9.266%); 1943/52053 objects misplaced (3.733%); 1.7 MiB/s, 27 objects/s recovering 2024-09-13T14:45:11.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:11 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/502295460' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:11.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:11 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:45:11.702 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:11.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:11.734 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:11.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:11.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:11 smithi195 ceph-mon[104262]: pgmap v1041: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.1 GiB data, 8.6 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 507 op/s; 4823/52053 objects degraded (9.266%); 1943/52053 objects misplaced (3.733%); 1.7 MiB/s, 27 objects/s recovering 2024-09-13T14:45:11.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:11 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/502295460' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:11.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:11 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:45:11.835 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:11.858 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:11.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:11.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:11.960 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:11.991 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:11.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:11.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:12.093 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:12.116 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:12.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:12.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:12.218 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:12.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:12.246 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:12.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:12.347 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:12.373 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:12.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:12.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:12.475 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:12.500 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:12.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:12.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:12.602 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:12.627 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:12.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:12.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:12.729 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:12.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:12.785 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:12.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:12.887 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:12.911 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:12.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:12.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:13.013 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:13.041 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:13.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:13.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:13.126 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:13.144 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:13.171 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:13.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:13.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:13.273 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:13.299 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:13.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:13.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:13.401 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:13.410 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:13 smithi102 ceph-mon[110183]: pgmap v1042: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.1 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 446 op/s; 4823/52053 objects degraded (9.266%); 1943/52053 objects misplaced (3.733%); 1.7 MiB/s, 27 objects/s recovering 2024-09-13T14:45:13.410 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:13 smithi102 ceph-mon[115269]: pgmap v1042: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.1 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 446 op/s; 4823/52053 objects degraded (9.266%); 1943/52053 objects misplaced (3.733%); 1.7 MiB/s, 27 objects/s recovering 2024-09-13T14:45:13.429 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:13.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:13.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:13.530 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:13.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:13.556 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:13.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:13.657 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:13.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:13.690 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:13.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:13.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:13 smithi195 ceph-mon[104262]: pgmap v1042: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 1.1 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 446 op/s; 4823/52053 objects degraded (9.266%); 1943/52053 objects misplaced (3.733%); 1.7 MiB/s, 27 objects/s recovering 2024-09-13T14:45:13.791 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:13.815 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:13.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:13.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:13.882 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:45:13.917 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:13.940 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:13.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:13.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:14.041 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:14.063 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:14.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:14.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:14.164 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:14.187 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:14.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:14.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:14.289 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:14.318 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:14.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:14.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:14.386 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is unset 2024-09-13T14:45:14.420 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:14.443 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:14.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:14.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:14.546 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:14.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:14.568 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:14.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:14.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:14 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 4823/52053 objects degraded (9.266%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:14.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:14 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3659340734' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:45:14.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:14 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:45:14.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:14 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 4823/52053 objects degraded (9.266%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:14.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:14 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3659340734' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:45:14.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:14 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:45:14.669 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:14.695 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:14.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:14.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:14.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:14 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 4823/52053 objects degraded (9.266%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:14.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:14 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3659340734' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:45:14.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:14 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:45:14.797 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:14.821 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:14.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:14.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:14.923 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:14.953 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:14.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:14.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:15.055 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:15.080 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:15.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:15.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:15.182 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:15.209 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:15.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:15.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:15.311 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:15.337 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:15.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:15.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:15.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:15 smithi102 ceph-mon[110183]: pgmap v1043: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 901 MiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 694 op/s; 4823/43107 objects degraded (11.188%); 1590/43107 objects misplaced (3.688%); 2.7 MiB/s, 42 objects/s recovering 2024-09-13T14:45:15.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:15 smithi102 ceph-mon[110183]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:45:15.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:15 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:45:15.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:15 smithi102 ceph-mon[110183]: osdmap e297: 8 total, 8 up, 6 in 2024-09-13T14:45:15.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:15 smithi102 ceph-mon[115269]: pgmap v1043: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 901 MiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 694 op/s; 4823/43107 objects degraded (11.188%); 1590/43107 objects misplaced (3.688%); 2.7 MiB/s, 42 objects/s recovering 2024-09-13T14:45:15.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:15 smithi102 ceph-mon[115269]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:45:15.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:15 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:45:15.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:15 smithi102 ceph-mon[115269]: osdmap e297: 8 total, 8 up, 6 in 2024-09-13T14:45:15.439 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:15.465 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:15.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:15.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:15.567 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:15.595 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:15.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:15.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:15.697 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:15.721 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:15.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:15.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:15.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:15 smithi195 ceph-mon[104262]: pgmap v1043: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 901 MiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 694 op/s; 4823/43107 objects degraded (11.188%); 1590/43107 objects misplaced (3.688%); 2.7 MiB/s, 42 objects/s recovering 2024-09-13T14:45:15.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:15 smithi195 ceph-mon[104262]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:45:15.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:15 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:45:15.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:15 smithi195 ceph-mon[104262]: osdmap e297: 8 total, 8 up, 6 in 2024-09-13T14:45:15.823 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:15.838 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:45:15.838 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:45:15.850 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:15.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:15.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:15.952 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:15.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:15.976 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:15.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:16.076 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:16.102 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:16.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:16.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:16.204 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:16.231 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:16.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:16.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:16.333 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:16.357 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:16.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:16.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:16.459 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:16.486 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:16.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:16.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:16.587 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:16.616 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:16.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:16.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:16.718 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:16.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:16.750 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:16.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:16.851 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:16.878 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:16.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:16.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:16.918 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:45:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:45:16] "GET /metrics HTTP/1.1" 200 36542 "" "Prometheus/2.51.0" 2024-09-13T14:45:16.980 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:17.006 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:17.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:17.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:17.056 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T14:45:17.108 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:17.172 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:17.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:17.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:17.274 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:17.300 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:17.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:17.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:17.402 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:17.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:17 smithi102 ceph-mon[110183]: pgmap v1045: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 901 MiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 587 op/s; 4823/43107 objects degraded (11.188%); 1590/43107 objects misplaced (3.688%); 2.2 MiB/s, 36 objects/s recovering 2024-09-13T14:45:17.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:17 smithi102 ceph-mon[115269]: pgmap v1045: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 901 MiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 587 op/s; 4823/43107 objects degraded (11.188%); 1590/43107 objects misplaced (3.688%); 2.2 MiB/s, 36 objects/s recovering 2024-09-13T14:45:17.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:17.431 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:17.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:17.532 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:17.546 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:17.574 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:17.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:17.577 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:17.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:17.678 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:17.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:17.706 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:17.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:17.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:17 smithi195 ceph-mon[104262]: pgmap v1045: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 901 MiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 587 op/s; 4823/43107 objects degraded (11.188%); 1590/43107 objects misplaced (3.688%); 2.2 MiB/s, 36 objects/s recovering 2024-09-13T14:45:17.751 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:45:17.807 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:17.839 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:17.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:17.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:17.941 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:17.968 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:17.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:17.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:18.074 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:18.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:18.098 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:18.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:18.199 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:18.226 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:18.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:18.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:18.327 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:18.358 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:18.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:18.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:18.460 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:18.484 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:18.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:18.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:18.586 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:18.617 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:18.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:18.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:18.719 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:18.752 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:18.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:18.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:18.854 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:18.881 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:18.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:18.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:18.983 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:19.012 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:19.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:19.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:19.114 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:19.140 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:19.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:19.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:19.164 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:45:19.164 INFO:teuthology.orchestra.run.smithi102.stdout:{"fsid":"08cee502-71d9-11ef-bceb-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4823/36510 objects degraded (13.210%), 3 pgs degraded, 2 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1542,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":297,"num_osds":8,"num_up_osds":8,"osd_up_since":1726238267,"num_in_osds":6,"osd_in_since":1726238173,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":85},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering","count":1},{"state_name":"active+recovery_wait+degraded+remapped","count":1}],"num_pgs":89,"num_pools":5,"num_objects":12170,"data_bytes":800462845,"bytes_used":8270434304,"bytes_avail":567699779584,"bytes_total":575970213888,"degraded_objects":4823,"degraded_total":36510,"degraded_ratio":0.1321007943029307,"misplaced_objects":1173,"misplaced_total":36510,"misplaced_ratio":0.032128184059161874,"recovering_objects_per_sec":41,"recovering_bytes_per_sec":2679684,"recovering_keys_per_sec":0,"num_objects_recovered":413,"num_bytes_recovered":26804224,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":603},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi102.njetxr","status":"up:active","gid":24521}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.102:8443/","prometheus":"http://172.21.15.102:9283/"}},"servicemap":{"epoch":28,"modified":"2024-09-13T14:40:21.476600+0000","services":{}},"progress_events":{}} 2024-09-13T14:45:19.242 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:19.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:19.274 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:19.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:19.375 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:19.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:19.406 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:19.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:19.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:19 smithi102 ceph-mon[110183]: pgmap v1046: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 763 MiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 603 op/s; 4823/36510 objects degraded (13.210%); 1173/36510 objects misplaced (3.213%); 2.6 MiB/s, 41 objects/s recovering 2024-09-13T14:45:19.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:19 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 4823/36510 objects degraded (13.210%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:19.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:19 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3233188838' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:45:19.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:19 smithi102 ceph-mon[115269]: pgmap v1046: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 763 MiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 603 op/s; 4823/36510 objects degraded (13.210%); 1173/36510 objects misplaced (3.213%); 2.6 MiB/s, 41 objects/s recovering 2024-09-13T14:45:19.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:19 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 4823/36510 objects degraded (13.210%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:19.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:19 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3233188838' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:45:19.507 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:19.533 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:19.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:19.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:19.554 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:19.635 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:19.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:19.665 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:19.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:19.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:19 smithi195 ceph-mon[104262]: pgmap v1046: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 763 MiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 603 op/s; 4823/36510 objects degraded (13.210%); 1173/36510 objects misplaced (3.213%); 2.6 MiB/s, 41 objects/s recovering 2024-09-13T14:45:19.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:19 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 4823/36510 objects degraded (13.210%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:19.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:19 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3233188838' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:45:19.766 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:19.796 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:19.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:19.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:19.898 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:19.922 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:19.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:19.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:20.024 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:20.053 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:20.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:20.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:20.154 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:20.182 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:20.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:20.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:20.284 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:20.311 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:20.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:20.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:20.413 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:20.440 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:20.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:20.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:20.542 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:20.572 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:20.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:20.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:20.674 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:20.702 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:20.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:20.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:20.804 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:20.829 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:20.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:20.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:20.931 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:20.962 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:20.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:20.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:21.063 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:21.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:21.102 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:21.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:21.203 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:21.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:21.243 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:21.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:21.344 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:21.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:21.381 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:21.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:21.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:21 smithi102 ceph-mon[110183]: pgmap v1047: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 701 MiB data, 7.5 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 618 op/s; 4823/33498 objects degraded (14.398%); 1173/33498 objects misplaced (3.502%); 2.6 MiB/s, 41 objects/s recovering 2024-09-13T14:45:21.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:21 smithi102 ceph-mon[115269]: pgmap v1047: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 701 MiB data, 7.5 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 618 op/s; 4823/33498 objects degraded (14.398%); 1173/33498 objects misplaced (3.502%); 2.6 MiB/s, 41 objects/s recovering 2024-09-13T14:45:21.482 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:21.513 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:21.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:21.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:21.615 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:21.644 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:21.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:21.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:21.745 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:21.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:21 smithi195 ceph-mon[104262]: pgmap v1047: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 701 MiB data, 7.5 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 618 op/s; 4823/33498 objects degraded (14.398%); 1173/33498 objects misplaced (3.502%); 2.6 MiB/s, 41 objects/s recovering 2024-09-13T14:45:21.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:21.781 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:21.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:21.882 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:21.909 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:21.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:21.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:21.964 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:45:21.965 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:45:21.994 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:45:21.994 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:45:22.012 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:22.015 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:45:22.015 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:45:22.040 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:45:22.041 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:45:22.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:22.044 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:22.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:22.067 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:45:22.068 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:45:22.093 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:45:22.093 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:45:22.114 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:45:22.114 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:45:22.139 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:45:22.140 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:45:22.145 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:22.178 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:22.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:22.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:22.280 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:22.308 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:22.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:22.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:22.410 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:22.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:22 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/537279573' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:22.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:22 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/537279573' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:22.437 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:22.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:22.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:22.539 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:22.566 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:22.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:22.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:22.668 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:22.696 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:22.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:22.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:22.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:22 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/537279573' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:22.799 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:22.829 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:22.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:22.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:22.931 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:22.959 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:22.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:22.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:22.990 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-13T14:45:22.991 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:45:23.061 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:23.097 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:23.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:23.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:23.201 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:23.236 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:23.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:23.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:23.338 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:23.369 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:23.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:23.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:23.471 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:23.499 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:23.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:23.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:23.601 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:23.630 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:23.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:23.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:23.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:23 smithi102 ceph-mon[110183]: pgmap v1048: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 701 MiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 618 op/s; 4823/33498 objects degraded (14.398%); 1173/33498 objects misplaced (3.502%); 2.6 MiB/s, 41 objects/s recovering 2024-09-13T14:45:23.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:23 smithi102 ceph-mon[115269]: pgmap v1048: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 701 MiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 618 op/s; 4823/33498 objects degraded (14.398%); 1173/33498 objects misplaced (3.502%); 2.6 MiB/s, 41 objects/s recovering 2024-09-13T14:45:23.731 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:23.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:23 smithi195 ceph-mon[104262]: pgmap v1048: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 701 MiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 618 op/s; 4823/33498 objects degraded (14.398%); 1173/33498 objects misplaced (3.502%); 2.6 MiB/s, 41 objects/s recovering 2024-09-13T14:45:23.766 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:23.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:23.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:23.868 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:23.902 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:23.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:23.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:24.004 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:24.035 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:24.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:24.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:24.137 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:24.171 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:24.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:24.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:24.273 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:24.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:24.313 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:24.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:24.414 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:24.453 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:24.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:24.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:24.555 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:24.596 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:24.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:24.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:24.610 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:24 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 4823/33498 objects degraded (14.398%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:24.610 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:24 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 4823/33498 objects degraded (14.398%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:24.697 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:24.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:24.733 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:24.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:24.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:24 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 4823/33498 objects degraded (14.398%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:24.835 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:24.862 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:24.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:24.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:24.964 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:24.993 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:24.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:24.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:25.094 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:25.135 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:25.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:25.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:25.237 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:25.269 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:25.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:25.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:25.371 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:25.404 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:25.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:25.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:25.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:25 smithi102 ceph-mon[110183]: pgmap v1049: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 453 MiB data, 6.8 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 716 op/s; 4823/21612 objects degraded (22.316%); 679/21612 objects misplaced (3.142%); 3.1 MiB/s, 50 objects/s recovering 2024-09-13T14:45:25.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:25 smithi102 ceph-mon[115269]: pgmap v1049: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 453 MiB data, 6.8 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 716 op/s; 4823/21612 objects degraded (22.316%); 679/21612 objects misplaced (3.142%); 3.1 MiB/s, 50 objects/s recovering 2024-09-13T14:45:25.506 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:25.543 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:25.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:25.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:25.645 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:25.670 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:25.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:25.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:25.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:25 smithi195 ceph-mon[104262]: pgmap v1049: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 453 MiB data, 6.8 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 716 op/s; 4823/21612 objects degraded (22.316%); 679/21612 objects misplaced (3.142%); 3.1 MiB/s, 50 objects/s recovering 2024-09-13T14:45:25.772 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:25.807 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:25.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:25.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:25.909 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:25.955 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:25.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:25.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:26.057 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:26.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:26.099 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:26.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:26.200 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:26.231 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:26.237 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:45:26.237 INFO:teuthology.orchestra.run.smithi102.stdout: "size": 20, 2024-09-13T14:45:26.237 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 600, 2024-09-13T14:45:26.237 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [] 2024-09-13T14:45:26.237 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:45:26.246 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:26.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:26.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:26.348 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:26.383 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:26.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:26.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:26.446 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is set 2024-09-13T14:45:26.485 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:26.518 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:26.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:26.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:26.620 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:26.649 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:45:26.657 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:26.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:26.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:26.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:45:26.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:45:26.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:26 smithi195 ceph-mon[104262]: pgmap v1050: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 453 MiB data, 6.8 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 634 op/s; 4823/21612 objects degraded (22.316%); 679/21612 objects misplaced (3.142%); 2.7 MiB/s, 44 objects/s recovering 2024-09-13T14:45:26.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:26 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1142378829' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:45:26.758 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:26.793 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:26.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:26.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:26.894 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:26.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:45:26.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:45:26.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:26 smithi102 ceph-mon[110183]: pgmap v1050: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 453 MiB data, 6.8 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 634 op/s; 4823/21612 objects degraded (22.316%); 679/21612 objects misplaced (3.142%); 2.7 MiB/s, 44 objects/s recovering 2024-09-13T14:45:26.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:26 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1142378829' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:45:26.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:45:26.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:45:26.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:26 smithi102 ceph-mon[115269]: pgmap v1050: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 453 MiB data, 6.8 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 634 op/s; 4823/21612 objects degraded (22.316%); 679/21612 objects misplaced (3.142%); 2.7 MiB/s, 44 objects/s recovering 2024-09-13T14:45:26.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:26 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1142378829' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:45:26.919 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:45:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:45:26] "GET /metrics HTTP/1.1" 200 36537 "" "Prometheus/2.51.0" 2024-09-13T14:45:26.932 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:26.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:26.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:27.034 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:27.068 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:27.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:27.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:27.170 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:27.211 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:27.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:27.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:27.312 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:27.347 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:27.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:27.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:27.449 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:27.476 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:27.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:27.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:27.578 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:27.621 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:27.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:27.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:27.698 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:27 smithi102 ceph-mon[110183]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:45:27.698 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:27 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1142378829' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:45:27.698 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:27 smithi102 ceph-mon[110183]: osdmap e298: 8 total, 8 up, 6 in 2024-09-13T14:45:27.699 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:27 smithi102 ceph-mon[115269]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:45:27.699 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:27 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1142378829' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:45:27.699 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:27 smithi102 ceph-mon[115269]: osdmap e298: 8 total, 8 up, 6 in 2024-09-13T14:45:27.723 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:27.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:27 smithi195 ceph-mon[104262]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:45:27.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:27 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1142378829' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:45:27.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:27 smithi195 ceph-mon[104262]: osdmap e298: 8 total, 8 up, 6 in 2024-09-13T14:45:27.755 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:27.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:27.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:27.857 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:27.894 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:27.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:27.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:27.996 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:28.025 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:28.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:28.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:28.127 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:28.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:28.154 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:28.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:28.255 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:28.284 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:28.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:28.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:28.386 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:28.407 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:28.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:28.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:28.509 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:28.546 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:28.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:28.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:28.647 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:28.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:28 smithi102 ceph-mon[110183]: pgmap v1052: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 235 MiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 845 op/s; 4674/11148 objects degraded (41.927%); 132/11148 objects misplaced (1.184%); 3.8 MiB/s, 67 objects/s recovering 2024-09-13T14:45:28.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:28 smithi102 ceph-mon[115269]: pgmap v1052: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 235 MiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 845 op/s; 4674/11148 objects degraded (41.927%); 132/11148 objects misplaced (1.184%); 3.8 MiB/s, 67 objects/s recovering 2024-09-13T14:45:28.680 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:28.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:28.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:28.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:28 smithi195 ceph-mon[104262]: pgmap v1052: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 235 MiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 845 op/s; 4674/11148 objects degraded (41.927%); 132/11148 objects misplaced (1.184%); 3.8 MiB/s, 67 objects/s recovering 2024-09-13T14:45:28.783 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:28.811 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:28.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:28.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:28.913 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:28.953 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:28.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:28.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:29.054 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:29.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:29.087 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:29.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:29.188 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:29.215 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:29.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:29.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:29.317 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:29.357 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:29.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:29.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:29.448 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Removed 112812 objects 2024-09-13T14:45:29.449 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Clean up completed and total clean up time :481.396 2024-09-13T14:45:29.459 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:29.479 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:29.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:29.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:29.582 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:29.603 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:29.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:29.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:29.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:29 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 4674/11148 objects degraded (41.927%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:29.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:29 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 4674/11148 objects degraded (41.927%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:29.706 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:29.727 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:29.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:29.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:29.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:29 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 4674/11148 objects degraded (41.927%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:29.829 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:29.852 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:29.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:29.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:29.954 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:29.976 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:29.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:29.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:30.078 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:30.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:30.101 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:30.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:30.203 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:30.224 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:30.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:30.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:30.326 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:30.348 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:30.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:30.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:30.450 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:30.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:30.473 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:30.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:30.574 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:30.583 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T14:45:30.596 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:30.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:30.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:30.697 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:30.720 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:30.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:30.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:30.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:30 smithi195 ceph-mon[104262]: pgmap v1053: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 137 MiB data, 5.9 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 901 op/s; 4674/6459 objects degraded (72.364%); 132/6459 objects misplaced (2.044%); 3.8 MiB/s, 67 objects/s recovering 2024-09-13T14:45:30.798 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:30 smithi102 ceph-mon[110183]: pgmap v1053: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 137 MiB data, 5.9 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 901 op/s; 4674/6459 objects degraded (72.364%); 132/6459 objects misplaced (2.044%); 3.8 MiB/s, 67 objects/s recovering 2024-09-13T14:45:30.798 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:30 smithi102 ceph-mon[115269]: pgmap v1053: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 137 MiB data, 5.9 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 901 op/s; 4674/6459 objects degraded (72.364%); 132/6459 objects misplaced (2.044%); 3.8 MiB/s, 67 objects/s recovering 2024-09-13T14:45:30.823 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:30.844 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:30.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:30.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:30.946 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:30.968 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:30.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:30.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:31.034 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:31.070 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:31.092 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:31.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:31.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:31.194 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:31.215 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:31.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:31.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:31.317 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:31.338 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:31.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:31.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:31.440 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:31.461 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:31.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:31.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:31.564 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:31.585 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:31.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:31.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:31.687 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:31.709 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:31.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:31.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:31.811 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:31.832 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:31.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:31.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:31.887 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:45:31.908 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-09-13T14:45:31.908 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-09-13T14:45:31.934 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:31.997 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:31.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:31.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:32.099 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:32.121 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:32.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:32.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:32.222 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:32.245 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:32.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:32.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:32.346 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:32.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:32.370 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:32.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:32.471 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:32.493 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:32.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:32.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:32.594 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:32.618 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:32.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:32.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:32.720 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:32.724 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:32.742 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:32.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:32.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:32.844 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:32.866 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:32.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:32.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:32.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:32 smithi102 ceph-mon[110183]: pgmap v1054: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 137 MiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 901 op/s; 4674/6459 objects degraded (72.364%); 132/6459 objects misplaced (2.044%); 3.8 MiB/s, 67 objects/s recovering 2024-09-13T14:45:32.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:32 smithi102 ceph-mon[115269]: pgmap v1054: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 137 MiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 901 op/s; 4674/6459 objects degraded (72.364%); 132/6459 objects misplaced (2.044%); 3.8 MiB/s, 67 objects/s recovering 2024-09-13T14:45:32.968 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:32.990 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:32.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:32.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:32.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:32 smithi195 ceph-mon[104262]: pgmap v1054: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 85 active+clean; 137 MiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 901 op/s; 4674/6459 objects degraded (72.364%); 132/6459 objects misplaced (2.044%); 3.8 MiB/s, 67 objects/s recovering 2024-09-13T14:45:33.093 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:33.114 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:33.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:33.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:33.216 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:33.237 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:33.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:33.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:33.339 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:33.361 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:33.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:33.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:33.463 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:33.485 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:33.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:33.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:33.586 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:33.604 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:45:33.605 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:45:33.608 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:33.609 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:33.610 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:33.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:33.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:33.712 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:33.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:33.734 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:33.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:33.835 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:33.857 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:33.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:33.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:33.959 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:33.980 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:33.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:33.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:34.082 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:34.104 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:34.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:34.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:34.205 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:34.227 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:34.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:34.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:34.328 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:34.350 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:34.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:34.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:34.452 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:34.474 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:34.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:34.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:34.575 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:34.596 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:34.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:34.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:34.698 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:34.719 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:34.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:34.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:34.821 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:34.840 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":1055,"stamp":"2024-09-13T14:45:33.559840+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790245,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4199,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":6377,"num_read_kb":13834,"num_write":226749,"num_write_kb":7235010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87918,"num_bytes_recovered":5599595661,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96331,"ondisk_log_size":96331,"up":267,"acting":265,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":299,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":5741896,"kb_used_data":2295448,"kb_used_omap":26,"kb_used_meta":3446373,"kb_avail":556729016,"statfs":{"total":575970213888,"available":570090512384,"internally_reserved":0,"allocated":2350538752,"data_stored":2341075919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27144,"internal_metadata":3529086456},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-470220824,"num_objects":-7176,"num_object_clones":0,"num_object_copies":-21528,"num_objects_missing_on_primary":-52,"num_objects_missing":-52,"num_objects_degraded":-624,"num_objects_misplaced":-679,"num_objects_unfound":0,"num_objects_dirty":-7176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7176,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":955,"num_bytes_recovered":30932992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002565"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":490,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T14:45:26.455395+0000","last_change":"2024-09-13T14:36:13.859927+0000","last_active":"2024-09-13T14:45:26.455395+0000","last_peered":"2024-09-13T14:45:26.455395+0000","last_clean":"2024-09-13T14:45:26.455395+0000","last_became_active":"2024-09-13T14:36:13.859505+0000","last_became_peered":"2024-09-13T14:36:13.859505+0000","last_unstale":"2024-09-13T14:45:26.455395+0000","last_undegraded":"2024-09-13T14:45:26.455395+0000","last_fullsized":"2024-09-13T14:45:26.455395+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:12:20.989627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":301,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-13T14:45:14.271476+0000","last_change":"2024-09-13T14:37:48.130968+0000","last_active":"2024-09-13T14:45:14.271476+0000","last_peered":"2024-09-13T14:45:14.271476+0000","last_clean":"2024-09-13T14:45:14.271476+0000","last_became_active":"2024-09-13T14:37:48.130603+0000","last_became_peered":"2024-09-13T14:37:48.130603+0000","last_unstale":"2024-09-13T14:45:14.271476+0000","last_undegraded":"2024-09-13T14:45:14.271476+0000","last_fullsized":"2024-09-13T14:45:14.271476+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:42.626386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":97,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-13T14:45:14.306455+0000","last_change":"2024-09-13T14:36:13.862181+0000","last_active":"2024-09-13T14:45:14.306455+0000","last_peered":"2024-09-13T14:45:14.306455+0000","last_clean":"2024-09-13T14:45:14.306455+0000","last_became_active":"2024-09-13T14:36:13.861873+0000","last_became_peered":"2024-09-13T14:36:13.861873+0000","last_unstale":"2024-09-13T14:45:14.306455+0000","last_undegraded":"2024-09-13T14:45:14.306455+0000","last_fullsized":"2024-09-13T14:45:14.306455+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:14.085700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":128,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T14:45:26.450163+0000","last_change":"2024-09-13T14:38:53.896672+0000","last_active":"2024-09-13T14:45:26.450163+0000","last_peered":"2024-09-13T14:45:26.450163+0000","last_clean":"2024-09-13T14:45:26.450163+0000","last_became_active":"2024-09-13T14:38:53.896346+0000","last_became_peered":"2024-09-13T14:38:53.896346+0000","last_unstale":"2024-09-13T14:45:26.450163+0000","last_undegraded":"2024-09-13T14:45:26.450163+0000","last_fullsized":"2024-09-13T14:45:26.450163+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:57.789686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":128,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T14:45:26.452233+0000","last_change":"2024-09-13T14:38:53.893021+0000","last_active":"2024-09-13T14:45:26.452233+0000","last_peered":"2024-09-13T14:45:26.452233+0000","last_clean":"2024-09-13T14:45:26.452233+0000","last_became_active":"2024-09-13T14:38:53.892668+0000","last_became_peered":"2024-09-13T14:38:53.892668+0000","last_unstale":"2024-09-13T14:45:26.452233+0000","last_undegraded":"2024-09-13T14:45:26.452233+0000","last_fullsized":"2024-09-13T14:45:26.452233+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:57.180738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":269,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-13T14:45:14.271471+0000","last_change":"2024-09-13T14:37:48.134642+0000","last_active":"2024-09-13T14:45:14.271471+0000","last_peered":"2024-09-13T14:45:14.271471+0000","last_clean":"2024-09-13T14:45:14.271471+0000","last_became_active":"2024-09-13T14:37:48.134402+0000","last_became_peered":"2024-09-13T14:37:48.134402+0000","last_unstale":"2024-09-13T14:45:14.271471+0000","last_undegraded":"2024-09-13T14:45:14.271471+0000","last_fullsized":"2024-09-13T14:45:14.271471+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:51.313663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":509,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T14:45:26.455250+0000","last_change":"2024-09-13T14:38:53.903022+0000","last_active":"2024-09-13T14:45:26.455250+0000","last_peered":"2024-09-13T14:45:26.455250+0000","last_clean":"2024-09-13T14:45:26.455250+0000","last_became_active":"2024-09-13T14:38:53.902448+0000","last_became_peered":"2024-09-13T14:38:53.902448+0000","last_unstale":"2024-09-13T14:45:26.455250+0000","last_undegraded":"2024-09-13T14:45:26.455250+0000","last_fullsized":"2024-09-13T14:45:26.455250+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:24.690401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":131,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T14:45:26.448920+0000","last_change":"2024-09-13T14:38:03.554500+0000","last_active":"2024-09-13T14:45:26.448920+0000","last_peered":"2024-09-13T14:45:26.448920+0000","last_clean":"2024-09-13T14:45:26.448920+0000","last_became_active":"2024-09-13T14:38:03.532910+0000","last_became_peered":"2024-09-13T14:38:03.532910+0000","last_unstale":"2024-09-13T14:45:26.448920+0000","last_undegraded":"2024-09-13T14:45:26.448920+0000","last_fullsized":"2024-09-13T14:45:26.448920+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:27:50.843498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"298'13880","reported_seq":24790,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T14:45:29.425600+0000","last_change":"2024-09-13T14:41:53.220083+0000","last_active":"2024-09-13T14:45:29.425600+0000","last_peered":"2024-09-13T14:45:29.425600+0000","last_clean":"2024-09-13T14:45:29.425600+0000","last_became_active":"2024-09-13T14:41:53.200084+0000","last_became_peered":"2024-09-13T14:41:53.200084+0000","last_unstale":"2024-09-13T14:45:29.425600+0000","last_undegraded":"2024-09-13T14:45:29.425600+0000","last_fullsized":"2024-09-13T14:45:29.425600+0000","mapping_epoch":275,"log_start":"264'8200","ondisk_log_start":"264'8200","created":205,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5680,"log_dups_size":0,"ondisk_log_size":5680,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:08.897524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13880,"num_write_kb":444160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9152,"num_bytes_recovered":594608128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":61,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-13T14:45:14.271974+0000","last_change":"2024-09-13T14:38:53.940238+0000","last_active":"2024-09-13T14:45:14.271974+0000","last_peered":"2024-09-13T14:45:14.271974+0000","last_clean":"2024-09-13T14:45:14.271974+0000","last_became_active":"2024-09-13T14:38:53.892825+0000","last_became_peered":"2024-09-13T14:38:53.892825+0000","last_unstale":"2024-09-13T14:45:14.271974+0000","last_undegraded":"2024-09-13T14:45:14.271974+0000","last_fullsized":"2024-09-13T14:45:14.271974+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:46.666581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":118,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-13T14:45:14.305886+0000","last_change":"2024-09-13T14:38:53.898907+0000","last_active":"2024-09-13T14:45:14.305886+0000","last_peered":"2024-09-13T14:45:14.305886+0000","last_clean":"2024-09-13T14:45:14.305886+0000","last_became_active":"2024-09-13T14:38:53.898572+0000","last_became_peered":"2024-09-13T14:38:53.898572+0000","last_unstale":"2024-09-13T14:45:14.305886+0000","last_undegraded":"2024-09-13T14:45:14.305886+0000","last_fullsized":"2024-09-13T14:45:14.305886+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:29.222150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"298'14242","reported_seq":28449,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T14:45:29.422156+0000","last_change":"2024-09-13T14:38:57.930032+0000","last_active":"2024-09-13T14:45:29.422156+0000","last_peered":"2024-09-13T14:45:29.422156+0000","last_clean":"2024-09-13T14:45:29.422156+0000","last_became_active":"2024-09-13T14:38:57.921458+0000","last_became_peered":"2024-09-13T14:38:57.921458+0000","last_unstale":"2024-09-13T14:45:29.422156+0000","last_undegraded":"2024-09-13T14:45:29.422156+0000","last_fullsized":"2024-09-13T14:45:29.422156+0000","mapping_epoch":256,"log_start":"265'8539","ondisk_log_start":"265'8539","created":205,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5703,"log_dups_size":0,"ondisk_log_size":5703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:27:12.294038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14242,"num_write_kb":455744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6990,"num_bytes_recovered":458096640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":61,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-13T14:45:14.271841+0000","last_change":"2024-09-13T14:38:53.904885+0000","last_active":"2024-09-13T14:45:14.271841+0000","last_peered":"2024-09-13T14:45:14.271841+0000","last_clean":"2024-09-13T14:45:14.271841+0000","last_became_active":"2024-09-13T14:38:53.891726+0000","last_became_peered":"2024-09-13T14:38:53.891726+0000","last_unstale":"2024-09-13T14:45:14.271841+0000","last_undegraded":"2024-09-13T14:45:14.271841+0000","last_fullsized":"2024-09-13T14:45:14.271841+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:57.079205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":307,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-13T14:45:14.287874+0000","last_change":"2024-09-13T14:37:48.160504+0000","last_active":"2024-09-13T14:45:14.287874+0000","last_peered":"2024-09-13T14:45:14.287874+0000","last_clean":"2024-09-13T14:45:14.287874+0000","last_became_active":"2024-09-13T14:37:48.160181+0000","last_became_peered":"2024-09-13T14:37:48.160181+0000","last_unstale":"2024-09-13T14:45:14.287874+0000","last_undegraded":"2024-09-13T14:45:14.287874+0000","last_fullsized":"2024-09-13T14:45:14.287874+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:37.087008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"298'14174","reported_seq":16003,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T14:45:29.444986+0000","last_change":"2024-09-13T14:38:04.531914+0000","last_active":"2024-09-13T14:45:29.444986+0000","last_peered":"2024-09-13T14:45:29.444986+0000","last_clean":"2024-09-13T14:45:29.444986+0000","last_became_active":"2024-09-13T14:38:04.523448+0000","last_became_peered":"2024-09-13T14:38:04.523448+0000","last_unstale":"2024-09-13T14:45:29.444986+0000","last_undegraded":"2024-09-13T14:45:29.444986+0000","last_fullsized":"2024-09-13T14:45:29.444986+0000","mapping_epoch":245,"log_start":"263'8200","ondisk_log_start":"263'8200","created":205,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5974,"log_dups_size":0,"ondisk_log_size":5974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:06.536896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14174,"num_write_kb":453568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":811,"num_bytes_recovered":53149696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":299,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-13T14:45:17.453377+0000","last_change":"2024-09-13T14:37:49.611774+0000","last_active":"2024-09-13T14:45:17.453377+0000","last_peered":"2024-09-13T14:45:17.453377+0000","last_clean":"2024-09-13T14:45:17.453377+0000","last_became_active":"2024-09-13T14:37:49.611088+0000","last_became_peered":"2024-09-13T14:37:49.611088+0000","last_unstale":"2024-09-13T14:45:17.453377+0000","last_undegraded":"2024-09-13T14:45:17.453377+0000","last_fullsized":"2024-09-13T14:45:17.453377+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:08.933213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":58,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T14:45:26.457204+0000","last_change":"2024-09-13T14:38:53.899211+0000","last_active":"2024-09-13T14:45:26.457204+0000","last_peered":"2024-09-13T14:45:26.457204+0000","last_clean":"2024-09-13T14:45:26.457204+0000","last_became_active":"2024-09-13T14:38:53.898927+0000","last_became_peered":"2024-09-13T14:38:53.898927+0000","last_unstale":"2024-09-13T14:45:26.457204+0000","last_undegraded":"2024-09-13T14:45:26.457204+0000","last_fullsized":"2024-09-13T14:45:26.457204+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:00:28.857611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"298'13972","reported_seq":14078,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T14:45:29.441528+0000","last_change":"2024-09-13T14:35:24.004651+0000","last_active":"2024-09-13T14:45:29.441528+0000","last_peered":"2024-09-13T14:45:29.441528+0000","last_clean":"2024-09-13T14:45:29.441528+0000","last_became_active":"2024-09-13T14:35:24.004358+0000","last_became_peered":"2024-09-13T14:35:24.004358+0000","last_unstale":"2024-09-13T14:45:29.441528+0000","last_undegraded":"2024-09-13T14:45:29.441528+0000","last_fullsized":"2024-09-13T14:45:29.441528+0000","mapping_epoch":205,"log_start":"258'7684","ondisk_log_start":"258'7684","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6288,"log_dups_size":0,"ondisk_log_size":6288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:57:38.023669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13972,"num_write_kb":447104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":109,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-13T14:45:17.333847+0000","last_change":"2024-09-13T14:37:49.168003+0000","last_active":"2024-09-13T14:45:17.333847+0000","last_peered":"2024-09-13T14:45:17.333847+0000","last_clean":"2024-09-13T14:45:17.333847+0000","last_became_active":"2024-09-13T14:37:49.167638+0000","last_became_peered":"2024-09-13T14:37:49.167638+0000","last_unstale":"2024-09-13T14:45:17.333847+0000","last_undegraded":"2024-09-13T14:45:17.333847+0000","last_fullsized":"2024-09-13T14:45:17.333847+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:29.051981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":121,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T14:45:26.450559+0000","last_change":"2024-09-13T14:38:03.528381+0000","last_active":"2024-09-13T14:45:26.450559+0000","last_peered":"2024-09-13T14:45:26.450559+0000","last_clean":"2024-09-13T14:45:26.450559+0000","last_became_active":"2024-09-13T14:38:03.527282+0000","last_became_peered":"2024-09-13T14:38:03.527282+0000","last_unstale":"2024-09-13T14:45:26.450559+0000","last_undegraded":"2024-09-13T14:45:26.450559+0000","last_fullsized":"2024-09-13T14:45:26.450559+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:24.435067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.f","version":"298'14086","reported_seq":14410,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T14:45:29.438419+0000","last_change":"2024-09-13T14:38:16.371079+0000","last_active":"2024-09-13T14:45:29.438419+0000","last_peered":"2024-09-13T14:45:29.438419+0000","last_clean":"2024-09-13T14:45:29.438419+0000","last_became_active":"2024-09-13T14:38:12.859883+0000","last_became_peered":"2024-09-13T14:38:12.859883+0000","last_unstale":"2024-09-13T14:45:29.438419+0000","last_undegraded":"2024-09-13T14:45:29.438419+0000","last_fullsized":"2024-09-13T14:45:29.438419+0000","mapping_epoch":245,"log_start":"264'8163","ondisk_log_start":"264'8163","created":205,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5923,"log_dups_size":0,"ondisk_log_size":5923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:08.436532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14086,"num_write_kb":450752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":3670016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":479,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T14:45:26.455831+0000","last_change":"2024-09-13T14:38:03.549595+0000","last_active":"2024-09-13T14:45:26.455831+0000","last_peered":"2024-09-13T14:45:26.455831+0000","last_clean":"2024-09-13T14:45:26.455831+0000","last_became_active":"2024-09-13T14:38:03.533331+0000","last_became_peered":"2024-09-13T14:38:03.533331+0000","last_unstale":"2024-09-13T14:45:26.455831+0000","last_undegraded":"2024-09-13T14:45:26.455831+0000","last_fullsized":"2024-09-13T14:45:26.455831+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:59:02.470343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":283,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-13T14:45:14.288164+0000","last_change":"2024-09-13T14:37:48.127600+0000","last_active":"2024-09-13T14:45:14.288164+0000","last_peered":"2024-09-13T14:45:14.288164+0000","last_clean":"2024-09-13T14:45:14.288164+0000","last_became_active":"2024-09-13T14:37:48.127349+0000","last_became_peered":"2024-09-13T14:37:48.127349+0000","last_unstale":"2024-09-13T14:45:14.288164+0000","last_undegraded":"2024-09-13T14:45:14.288164+0000","last_fullsized":"2024-09-13T14:45:14.288164+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:57.268374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"298'14010","reported_seq":24013,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T14:45:29.445330+0000","last_change":"2024-09-13T14:42:30.227259+0000","last_active":"2024-09-13T14:45:29.445330+0000","last_peered":"2024-09-13T14:45:29.445330+0000","last_clean":"2024-09-13T14:45:29.445330+0000","last_became_active":"2024-09-13T14:42:30.209102+0000","last_became_peered":"2024-09-13T14:42:30.209102+0000","last_unstale":"2024-09-13T14:45:29.445330+0000","last_undegraded":"2024-09-13T14:45:29.445330+0000","last_fullsized":"2024-09-13T14:45:29.445330+0000","mapping_epoch":281,"log_start":"264'8300","ondisk_log_start":"264'8300","created":205,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5710,"log_dups_size":0,"ondisk_log_size":5710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:56:16.193287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14010,"num_write_kb":448320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4926,"num_bytes_recovered":313851904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":216,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-13T14:45:14.287925+0000","last_change":"2024-09-13T14:37:48.127604+0000","last_active":"2024-09-13T14:45:14.287925+0000","last_peered":"2024-09-13T14:45:14.287925+0000","last_clean":"2024-09-13T14:45:14.287925+0000","last_became_active":"2024-09-13T14:37:48.127300+0000","last_became_peered":"2024-09-13T14:37:48.127300+0000","last_unstale":"2024-09-13T14:45:14.287925+0000","last_undegraded":"2024-09-13T14:45:14.287925+0000","last_fullsized":"2024-09-13T14:45:14.287925+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:51.999506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":336,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-13T14:45:17.894128+0000","last_change":"2024-09-13T14:38:03.529967+0000","last_active":"2024-09-13T14:45:17.894128+0000","last_peered":"2024-09-13T14:45:17.894128+0000","last_clean":"2024-09-13T14:45:17.894128+0000","last_became_active":"2024-09-13T14:38:03.529646+0000","last_became_peered":"2024-09-13T14:38:03.529646+0000","last_unstale":"2024-09-13T14:45:17.894128+0000","last_undegraded":"2024-09-13T14:45:17.894128+0000","last_fullsized":"2024-09-13T14:45:17.894128+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:52:05.193279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"298'14140","reported_seq":23339,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T14:45:29.439449+0000","last_change":"2024-09-13T14:40:34.583237+0000","last_active":"2024-09-13T14:45:29.439449+0000","last_peered":"2024-09-13T14:45:29.439449+0000","last_clean":"2024-09-13T14:45:29.439449+0000","last_became_active":"2024-09-13T14:40:34.563989+0000","last_became_peered":"2024-09-13T14:40:34.563989+0000","last_unstale":"2024-09-13T14:45:29.439449+0000","last_undegraded":"2024-09-13T14:45:29.439449+0000","last_fullsized":"2024-09-13T14:45:29.439449+0000","mapping_epoch":266,"log_start":"263'8200","ondisk_log_start":"263'8200","created":205,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5940,"log_dups_size":0,"ondisk_log_size":5940,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:34:43.006051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14140,"num_write_kb":452417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4489,"num_bytes_recovered":289865752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":296,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-13T14:45:14.288036+0000","last_change":"2024-09-13T14:36:13.860367+0000","last_active":"2024-09-13T14:45:14.288036+0000","last_peered":"2024-09-13T14:45:14.288036+0000","last_clean":"2024-09-13T14:45:14.288036+0000","last_became_active":"2024-09-13T14:36:13.859982+0000","last_became_peered":"2024-09-13T14:36:13.859982+0000","last_unstale":"2024-09-13T14:45:14.288036+0000","last_undegraded":"2024-09-13T14:45:14.288036+0000","last_fullsized":"2024-09-13T14:45:14.288036+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:08:23.311350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":292,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-13T14:45:17.894144+0000","last_change":"2024-09-13T14:37:48.147360+0000","last_active":"2024-09-13T14:45:17.894144+0000","last_peered":"2024-09-13T14:45:17.894144+0000","last_clean":"2024-09-13T14:45:17.894144+0000","last_became_active":"2024-09-13T14:37:48.146885+0000","last_became_peered":"2024-09-13T14:37:48.146885+0000","last_unstale":"2024-09-13T14:45:17.894144+0000","last_undegraded":"2024-09-13T14:45:17.894144+0000","last_fullsized":"2024-09-13T14:45:17.894144+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:27.841810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"298'14074","reported_seq":23380,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T14:45:29.422113+0000","last_change":"2024-09-13T14:38:39.650332+0000","last_active":"2024-09-13T14:45:29.422113+0000","last_peered":"2024-09-13T14:45:29.422113+0000","last_clean":"2024-09-13T14:45:29.422113+0000","last_became_active":"2024-09-13T14:38:39.640835+0000","last_became_peered":"2024-09-13T14:38:39.640835+0000","last_unstale":"2024-09-13T14:45:29.422113+0000","last_undegraded":"2024-09-13T14:45:29.422113+0000","last_fullsized":"2024-09-13T14:45:29.422113+0000","mapping_epoch":251,"log_start":"264'8351","ondisk_log_start":"264'8351","created":205,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5723,"log_dups_size":0,"ondisk_log_size":5723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:48.176606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14074,"num_write_kb":450368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4544,"num_bytes_recovered":292945920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":497,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T14:45:26.455272+0000","last_change":"2024-09-13T14:38:53.900041+0000","last_active":"2024-09-13T14:45:26.455272+0000","last_peered":"2024-09-13T14:45:26.455272+0000","last_clean":"2024-09-13T14:45:26.455272+0000","last_became_active":"2024-09-13T14:38:53.899203+0000","last_became_peered":"2024-09-13T14:38:53.899203+0000","last_unstale":"2024-09-13T14:45:26.455272+0000","last_undegraded":"2024-09-13T14:45:26.455272+0000","last_fullsized":"2024-09-13T14:45:26.455272+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:28:07.883094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":429,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T14:45:26.455560+0000","last_change":"2024-09-13T14:26:15.823198+0000","last_active":"2024-09-13T14:45:26.455560+0000","last_peered":"2024-09-13T14:45:26.455560+0000","last_clean":"2024-09-13T14:45:26.455560+0000","last_became_active":"2024-09-13T14:26:15.822717+0000","last_became_peered":"2024-09-13T14:26:15.822717+0000","last_unstale":"2024-09-13T14:45:26.455560+0000","last_undegraded":"2024-09-13T14:45:26.455560+0000","last_fullsized":"2024-09-13T14:45:26.455560+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:22.471622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.6","version":"298'14198","reported_seq":23867,"reported_epoch":298,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-13T14:45:29.393241+0000","last_change":"2024-09-13T14:38:55.132683+0000","last_active":"2024-09-13T14:45:29.393241+0000","last_peered":"2024-09-13T14:45:29.393241+0000","last_clean":"2024-09-13T14:36:13.124368+0000","last_became_active":"2024-09-13T14:38:55.120883+0000","last_became_peered":"2024-09-13T14:38:55.120883+0000","last_unstale":"2024-09-13T14:45:29.393241+0000","last_undegraded":"2024-09-13T14:38:55.026610+0000","last_fullsized":"2024-09-13T14:45:29.393241+0000","mapping_epoch":254,"log_start":"261'8219","ondisk_log_start":"261'8219","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5979,"log_dups_size":0,"ondisk_log_size":5979,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14198,"num_write_kb":454336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4726,"num_bytes_recovered":298516480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,3,5],"avail_no_missing":["0","3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":432,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-13T14:45:14.271407+0000","last_change":"2024-09-13T14:41:53.196835+0000","last_active":"2024-09-13T14:45:14.271407+0000","last_peered":"2024-09-13T14:45:14.271407+0000","last_clean":"2024-09-13T14:45:14.271407+0000","last_became_active":"2024-09-13T14:41:53.196387+0000","last_became_peered":"2024-09-13T14:41:53.196387+0000","last_unstale":"2024-09-13T14:45:14.271407+0000","last_undegraded":"2024-09-13T14:45:14.271407+0000","last_fullsized":"2024-09-13T14:45:14.271407+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:07:41.927740+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:49.972303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":538,"reported_epoch":297,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:45:14.306001+0000","last_change":"2024-09-13T14:38:54.904329+0000","last_active":"2024-09-13T14:45:14.306001+0000","last_peered":"2024-09-13T14:45:14.306001+0000","last_clean":"2024-09-13T14:38:39.648017+0000","last_became_active":"2024-09-13T14:38:54.903798+0000","last_became_peered":"2024-09-13T14:38:54.903798+0000","last_unstale":"2024-09-13T14:45:14.306001+0000","last_undegraded":"2024-09-13T14:38:54.890505+0000","last_fullsized":"2024-09-13T14:38:54.889701+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:18.120931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,3],"avail_no_missing":["5","3","6"],"object_location_counts":[{"shards":"3,5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":431,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T14:45:26.457292+0000","last_change":"2024-09-13T14:37:49.160719+0000","last_active":"2024-09-13T14:45:26.457292+0000","last_peered":"2024-09-13T14:45:26.457292+0000","last_clean":"2024-09-13T14:45:26.457292+0000","last_became_active":"2024-09-13T14:37:49.160401+0000","last_became_peered":"2024-09-13T14:37:49.160401+0000","last_unstale":"2024-09-13T14:45:26.457292+0000","last_undegraded":"2024-09-13T14:45:26.457292+0000","last_fullsized":"2024-09-13T14:45:26.457292+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:21.737945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":143,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-13T14:45:14.288215+0000","last_change":"2024-09-13T14:37:48.172940+0000","last_active":"2024-09-13T14:45:14.288215+0000","last_peered":"2024-09-13T14:45:14.288215+0000","last_clean":"2024-09-13T14:45:14.288215+0000","last_became_active":"2024-09-13T14:37:48.172060+0000","last_became_peered":"2024-09-13T14:37:48.172060+0000","last_unstale":"2024-09-13T14:45:14.288215+0000","last_undegraded":"2024-09-13T14:45:14.288215+0000","last_fullsized":"2024-09-13T14:45:14.288215+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:07.480645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.5","version":"298'14142","reported_seq":30524,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T14:45:29.425417+0000","last_change":"2024-09-13T14:39:34.446699+0000","last_active":"2024-09-13T14:45:29.425417+0000","last_peered":"2024-09-13T14:45:29.425417+0000","last_clean":"2024-09-13T14:45:29.425417+0000","last_became_active":"2024-09-13T14:38:04.600616+0000","last_became_peered":"2024-09-13T14:38:04.600616+0000","last_unstale":"2024-09-13T14:45:29.425417+0000","last_undegraded":"2024-09-13T14:45:29.425417+0000","last_fullsized":"2024-09-13T14:45:29.425417+0000","mapping_epoch":245,"log_start":"259'7816","ondisk_log_start":"259'7816","created":205,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6326,"log_dups_size":0,"ondisk_log_size":6326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:00:44.321678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14142,"num_write_kb":452544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8120,"num_bytes_recovered":521076736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":516,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-13T14:45:14.271332+0000","last_change":"2024-09-13T14:26:15.835178+0000","last_active":"2024-09-13T14:45:14.271332+0000","last_peered":"2024-09-13T14:45:14.271332+0000","last_clean":"2024-09-13T14:45:14.271332+0000","last_became_active":"2024-09-13T14:26:15.820283+0000","last_became_peered":"2024-09-13T14:26:15.820283+0000","last_unstale":"2024-09-13T14:45:14.271332+0000","last_undegraded":"2024-09-13T14:45:14.271332+0000","last_fullsized":"2024-09-13T14:45:14.271332+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:30.340038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":121,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T14:45:26.449462+0000","last_change":"2024-09-13T14:38:03.525682+0000","last_active":"2024-09-13T14:45:26.449462+0000","last_peered":"2024-09-13T14:45:26.449462+0000","last_clean":"2024-09-13T14:45:26.449462+0000","last_became_active":"2024-09-13T14:38:03.525406+0000","last_became_peered":"2024-09-13T14:38:03.525406+0000","last_unstale":"2024-09-13T14:45:26.449462+0000","last_undegraded":"2024-09-13T14:45:26.449462+0000","last_fullsized":"2024-09-13T14:45:26.449462+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:41:28.970300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":128,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T14:45:26.452280+0000","last_change":"2024-09-13T14:38:53.991738+0000","last_active":"2024-09-13T14:45:26.452280+0000","last_peered":"2024-09-13T14:45:26.452280+0000","last_clean":"2024-09-13T14:45:26.452280+0000","last_became_active":"2024-09-13T14:38:53.991417+0000","last_became_peered":"2024-09-13T14:38:53.991417+0000","last_unstale":"2024-09-13T14:45:26.452280+0000","last_undegraded":"2024-09-13T14:45:26.452280+0000","last_fullsized":"2024-09-13T14:45:26.452280+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:50:38.559838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"298'14106","reported_seq":37496,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T14:45:29.575809+0000","last_change":"2024-09-13T14:45:29.575698+0000","last_active":"2024-09-13T14:45:29.575809+0000","last_peered":"2024-09-13T14:45:29.575809+0000","last_clean":"2024-09-13T14:45:29.575809+0000","last_became_active":"2024-09-13T14:38:17.381753+0000","last_became_peered":"2024-09-13T14:38:17.381753+0000","last_unstale":"2024-09-13T14:45:29.575809+0000","last_undegraded":"2024-09-13T14:45:29.575809+0000","last_fullsized":"2024-09-13T14:45:29.575809+0000","mapping_epoch":245,"log_start":"261'8159","ondisk_log_start":"261'8159","created":205,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5947,"log_dups_size":0,"ondisk_log_size":5947,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:34:24.585927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14106,"num_write_kb":451392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11635,"num_bytes_recovered":751435776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":842,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T14:45:26.455109+0000","last_change":"2024-09-13T14:37:48.131799+0000","last_active":"2024-09-13T14:45:26.455109+0000","last_peered":"2024-09-13T14:45:26.455109+0000","last_clean":"2024-09-13T14:45:26.455109+0000","last_became_active":"2024-09-13T14:37:48.130837+0000","last_became_peered":"2024-09-13T14:37:48.130837+0000","last_unstale":"2024-09-13T14:45:26.455109+0000","last_undegraded":"2024-09-13T14:45:26.455109+0000","last_fullsized":"2024-09-13T14:45:26.455109+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:06:32.663770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":361,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T14:45:26.449244+0000","last_change":"2024-09-13T14:37:49.301531+0000","last_active":"2024-09-13T14:45:26.449244+0000","last_peered":"2024-09-13T14:45:26.449244+0000","last_clean":"2024-09-13T14:45:26.449244+0000","last_became_active":"2024-09-13T14:37:49.301249+0000","last_became_peered":"2024-09-13T14:37:49.301249+0000","last_unstale":"2024-09-13T14:45:26.449244+0000","last_undegraded":"2024-09-13T14:45:26.449244+0000","last_fullsized":"2024-09-13T14:45:26.449244+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:10.977710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":125,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-13T14:45:14.306793+0000","last_change":"2024-09-13T14:38:53.901427+0000","last_active":"2024-09-13T14:45:14.306793+0000","last_peered":"2024-09-13T14:45:14.306793+0000","last_clean":"2024-09-13T14:45:14.306793+0000","last_became_active":"2024-09-13T14:38:53.901157+0000","last_became_peered":"2024-09-13T14:38:53.901157+0000","last_unstale":"2024-09-13T14:45:14.306793+0000","last_undegraded":"2024-09-13T14:45:14.306793+0000","last_fullsized":"2024-09-13T14:45:14.306793+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:28:21.713025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"298'13998","reported_seq":37258,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T14:45:29.423184+0000","last_change":"2024-09-13T14:43:16.851222+0000","last_active":"2024-09-13T14:45:29.423184+0000","last_peered":"2024-09-13T14:45:29.423184+0000","last_clean":"2024-09-13T14:45:29.423184+0000","last_became_active":"2024-09-13T14:38:16.427510+0000","last_became_peered":"2024-09-13T14:38:16.427510+0000","last_unstale":"2024-09-13T14:45:29.423184+0000","last_undegraded":"2024-09-13T14:45:29.423184+0000","last_fullsized":"2024-09-13T14:45:29.423184+0000","mapping_epoch":245,"log_start":"261'8018","ondisk_log_start":"261'8018","created":205,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5980,"log_dups_size":0,"ondisk_log_size":5980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:27:26.281938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13998,"num_write_kb":447936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11571,"num_bytes_recovered":747372544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":1007,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-13T14:45:14.306217+0000","last_change":"2024-09-13T14:37:49.147498+0000","last_active":"2024-09-13T14:45:14.306217+0000","last_peered":"2024-09-13T14:45:14.306217+0000","last_clean":"2024-09-13T14:45:14.306217+0000","last_became_active":"2024-09-13T14:37:49.147028+0000","last_became_peered":"2024-09-13T14:37:49.147028+0000","last_unstale":"2024-09-13T14:45:14.306217+0000","last_undegraded":"2024-09-13T14:45:14.306217+0000","last_fullsized":"2024-09-13T14:45:14.306217+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:11:29.050178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":305,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-13T14:45:14.288216+0000","last_change":"2024-09-13T14:38:03.528133+0000","last_active":"2024-09-13T14:45:14.288216+0000","last_peered":"2024-09-13T14:45:14.288216+0000","last_clean":"2024-09-13T14:45:14.288216+0000","last_became_active":"2024-09-13T14:38:03.527829+0000","last_became_peered":"2024-09-13T14:38:03.527829+0000","last_unstale":"2024-09-13T14:45:14.288216+0000","last_undegraded":"2024-09-13T14:45:14.288216+0000","last_fullsized":"2024-09-13T14:45:14.288216+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:48.254169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":481,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T14:45:26.455295+0000","last_change":"2024-09-13T14:38:53.899755+0000","last_active":"2024-09-13T14:45:26.455295+0000","last_peered":"2024-09-13T14:45:26.455295+0000","last_clean":"2024-09-13T14:45:26.455295+0000","last_became_active":"2024-09-13T14:38:53.899448+0000","last_became_peered":"2024-09-13T14:38:53.899448+0000","last_unstale":"2024-09-13T14:45:26.455295+0000","last_undegraded":"2024-09-13T14:45:26.455295+0000","last_fullsized":"2024-09-13T14:45:26.455295+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:55.897035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"298'14302","reported_seq":23442,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T14:45:29.417935+0000","last_change":"2024-09-13T14:43:51.649951+0000","last_active":"2024-09-13T14:45:29.417935+0000","last_peered":"2024-09-13T14:45:29.417935+0000","last_clean":"2024-09-13T14:45:29.417935+0000","last_became_active":"2024-09-13T14:43:51.626628+0000","last_became_peered":"2024-09-13T14:43:51.626628+0000","last_unstale":"2024-09-13T14:45:29.417935+0000","last_undegraded":"2024-09-13T14:45:29.417935+0000","last_fullsized":"2024-09-13T14:45:29.417935+0000","mapping_epoch":289,"log_start":"249'7463","ondisk_log_start":"249'7463","created":205,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":6839,"log_dups_size":0,"ondisk_log_size":6839,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:37:06.795173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14302,"num_write_kb":457664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4502,"num_bytes_recovered":228851712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":109,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-13T14:45:14.306352+0000","last_change":"2024-09-13T14:37:49.149770+0000","last_active":"2024-09-13T14:45:14.306352+0000","last_peered":"2024-09-13T14:45:14.306352+0000","last_clean":"2024-09-13T14:45:14.306352+0000","last_became_active":"2024-09-13T14:37:49.148991+0000","last_became_peered":"2024-09-13T14:37:49.148991+0000","last_unstale":"2024-09-13T14:45:14.306352+0000","last_undegraded":"2024-09-13T14:45:14.306352+0000","last_fullsized":"2024-09-13T14:45:14.306352+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:04:04.193366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":1180,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T14:45:26.456631+0000","last_change":"2024-09-13T14:30:58.928302+0000","last_active":"2024-09-13T14:45:26.456631+0000","last_peered":"2024-09-13T14:45:26.456631+0000","last_clean":"2024-09-13T14:45:26.456631+0000","last_became_active":"2024-09-13T14:30:58.927718+0000","last_became_peered":"2024-09-13T14:30:58.927718+0000","last_unstale":"2024-09-13T14:45:26.456631+0000","last_undegraded":"2024-09-13T14:45:26.456631+0000","last_fullsized":"2024-09-13T14:45:26.456631+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:37.323036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":116,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-13T14:45:14.306535+0000","last_change":"2024-09-13T14:38:03.534628+0000","last_active":"2024-09-13T14:45:14.306535+0000","last_peered":"2024-09-13T14:45:14.306535+0000","last_clean":"2024-09-13T14:45:14.306535+0000","last_became_active":"2024-09-13T14:38:03.534262+0000","last_became_peered":"2024-09-13T14:38:03.534262+0000","last_unstale":"2024-09-13T14:45:14.306535+0000","last_undegraded":"2024-09-13T14:45:14.306535+0000","last_fullsized":"2024-09-13T14:45:14.306535+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:31.168783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"298'14052","reported_seq":28929,"reported_epoch":298,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T14:45:32.338024+0000","last_change":"2024-09-13T14:45:29.575893+0000","last_active":"2024-09-13T14:45:32.338024+0000","last_peered":"2024-09-13T14:45:32.338024+0000","last_clean":"2024-09-13T14:37:47.047672+0000","last_became_active":"2024-09-13T14:37:49.664294+0000","last_became_peered":"2024-09-13T14:37:49.664294+0000","last_unstale":"2024-09-13T14:45:32.338024+0000","last_undegraded":"2024-09-13T14:38:53.940822+0000","last_fullsized":"2024-09-13T14:37:49.126627+0000","mapping_epoch":240,"log_start":"264'8300","ondisk_log_start":"264'8300","created":205,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5752,"log_dups_size":0,"ondisk_log_size":5752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:11.766692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14052,"num_write_kb":449664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7354,"num_bytes_recovered":459341824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3],"avail_no_missing":["5","0","3"],"object_location_counts":[{"shards":"3,5","objects":-4190}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":128,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-13T14:45:14.306259+0000","last_change":"2024-09-13T14:38:03.534303+0000","last_active":"2024-09-13T14:45:14.306259+0000","last_peered":"2024-09-13T14:45:14.306259+0000","last_clean":"2024-09-13T14:45:14.306259+0000","last_became_active":"2024-09-13T14:38:03.533171+0000","last_became_peered":"2024-09-13T14:38:03.533171+0000","last_unstale":"2024-09-13T14:45:14.306259+0000","last_undegraded":"2024-09-13T14:45:14.306259+0000","last_fullsized":"2024-09-13T14:45:14.306259+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:00:16.553526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":820,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-13T14:45:14.271779+0000","last_change":"2024-09-13T14:38:04.596192+0000","last_active":"2024-09-13T14:45:14.271779+0000","last_peered":"2024-09-13T14:45:14.271779+0000","last_clean":"2024-09-13T14:45:14.271779+0000","last_became_active":"2024-09-13T14:38:04.593299+0000","last_became_peered":"2024-09-13T14:38:04.593299+0000","last_unstale":"2024-09-13T14:45:14.271779+0000","last_undegraded":"2024-09-13T14:45:14.271779+0000","last_fullsized":"2024-09-13T14:45:14.271779+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:34.241410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":121,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T14:45:26.450645+0000","last_change":"2024-09-13T14:38:03.533120+0000","last_active":"2024-09-13T14:45:26.450645+0000","last_peered":"2024-09-13T14:45:26.450645+0000","last_clean":"2024-09-13T14:45:26.450645+0000","last_became_active":"2024-09-13T14:38:03.532829+0000","last_became_peered":"2024-09-13T14:38:03.532829+0000","last_unstale":"2024-09-13T14:45:26.450645+0000","last_undegraded":"2024-09-13T14:45:26.450645+0000","last_fullsized":"2024-09-13T14:45:26.450645+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:24.514661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"298'14398","reported_seq":23683,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T14:45:29.430160+0000","last_change":"2024-09-13T14:42:04.660877+0000","last_active":"2024-09-13T14:45:29.430160+0000","last_peered":"2024-09-13T14:45:29.430160+0000","last_clean":"2024-09-13T14:45:29.430160+0000","last_became_active":"2024-09-13T14:42:04.641680+0000","last_became_peered":"2024-09-13T14:42:04.641680+0000","last_unstale":"2024-09-13T14:45:29.430160+0000","last_undegraded":"2024-09-13T14:45:29.430160+0000","last_fullsized":"2024-09-13T14:45:29.430160+0000","mapping_epoch":277,"log_start":"265'8718","ondisk_log_start":"265'8718","created":205,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5680,"log_dups_size":0,"ondisk_log_size":5680,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:22:18.955428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14398,"num_write_kb":460736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4533,"num_bytes_recovered":297074688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":523,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T14:45:26.455396+0000","last_change":"2024-09-13T14:38:53.938227+0000","last_active":"2024-09-13T14:45:26.455396+0000","last_peered":"2024-09-13T14:45:26.455396+0000","last_clean":"2024-09-13T14:45:26.455396+0000","last_became_active":"2024-09-13T14:38:53.902867+0000","last_became_peered":"2024-09-13T14:38:53.902867+0000","last_unstale":"2024-09-13T14:45:26.455396+0000","last_undegraded":"2024-09-13T14:45:26.455396+0000","last_fullsized":"2024-09-13T14:45:26.455396+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:16.831756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":599,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-13T14:45:14.288581+0000","last_change":"2024-09-13T14:37:48.147677+0000","last_active":"2024-09-13T14:45:14.288581+0000","last_peered":"2024-09-13T14:45:14.288581+0000","last_clean":"2024-09-13T14:45:14.288581+0000","last_became_active":"2024-09-13T14:37:48.147001+0000","last_became_peered":"2024-09-13T14:37:48.147001+0000","last_unstale":"2024-09-13T14:45:14.288581+0000","last_undegraded":"2024-09-13T14:45:14.288581+0000","last_fullsized":"2024-09-13T14:45:14.288581+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:35:00.653576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":505,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T14:45:26.454887+0000","last_change":"2024-09-13T14:38:53.900107+0000","last_active":"2024-09-13T14:45:26.454887+0000","last_peered":"2024-09-13T14:45:26.454887+0000","last_clean":"2024-09-13T14:45:26.454887+0000","last_became_active":"2024-09-13T14:38:53.899556+0000","last_became_peered":"2024-09-13T14:38:53.899556+0000","last_unstale":"2024-09-13T14:45:26.454887+0000","last_undegraded":"2024-09-13T14:45:26.454887+0000","last_fullsized":"2024-09-13T14:45:26.454887+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:15:47.918163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"298'13852","reported_seq":22863,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T14:45:29.386021+0000","last_change":"2024-09-13T14:39:54.921239+0000","last_active":"2024-09-13T14:45:29.386021+0000","last_peered":"2024-09-13T14:45:29.386021+0000","last_clean":"2024-09-13T14:45:29.386021+0000","last_became_active":"2024-09-13T14:39:54.907894+0000","last_became_peered":"2024-09-13T14:39:54.907894+0000","last_unstale":"2024-09-13T14:45:29.386021+0000","last_undegraded":"2024-09-13T14:45:29.386021+0000","last_fullsized":"2024-09-13T14:45:29.386021+0000","mapping_epoch":263,"log_start":"264'8147","ondisk_log_start":"264'8147","created":205,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:35:22.986269+0000","last_clean_scrub_stamp":"2024-09-13T14:35:22.986269+0000","objects_scrubbed":0,"log_size":5705,"log_dups_size":0,"ondisk_log_size":5705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:52:04.619238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13852,"num_write_kb":443264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4395,"num_bytes_recovered":283770880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":136,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T14:45:26.452095+0000","last_change":"2024-09-13T14:38:53.901806+0000","last_active":"2024-09-13T14:45:26.452095+0000","last_peered":"2024-09-13T14:45:26.452095+0000","last_clean":"2024-09-13T14:45:26.452095+0000","last_became_active":"2024-09-13T14:38:53.891055+0000","last_became_peered":"2024-09-13T14:38:53.891055+0000","last_unstale":"2024-09-13T14:45:26.452095+0000","last_undegraded":"2024-09-13T14:45:26.452095+0000","last_fullsized":"2024-09-13T14:45:26.452095+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:03:00.416198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":74,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T14:45:26.457303+0000","last_change":"2024-09-13T14:43:16.854299+0000","last_active":"2024-09-13T14:45:26.457303+0000","last_peered":"2024-09-13T14:45:26.457303+0000","last_clean":"2024-09-13T14:45:26.457303+0000","last_became_active":"2024-09-13T14:38:53.900626+0000","last_became_peered":"2024-09-13T14:38:53.900626+0000","last_unstale":"2024-09-13T14:45:26.457303+0000","last_undegraded":"2024-09-13T14:45:26.457303+0000","last_fullsized":"2024-09-13T14:45:26.457303+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:37:21.715778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":289,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-13T14:45:14.288133+0000","last_change":"2024-09-13T14:37:49.172218+0000","last_active":"2024-09-13T14:45:14.288133+0000","last_peered":"2024-09-13T14:45:14.288133+0000","last_clean":"2024-09-13T14:45:14.288133+0000","last_became_active":"2024-09-13T14:37:49.171718+0000","last_became_peered":"2024-09-13T14:37:49.171718+0000","last_unstale":"2024-09-13T14:45:14.288133+0000","last_undegraded":"2024-09-13T14:45:14.288133+0000","last_fullsized":"2024-09-13T14:45:14.288133+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:46:54.626521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":311,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-13T14:45:14.288052+0000","last_change":"2024-09-13T14:37:49.230094+0000","last_active":"2024-09-13T14:45:14.288052+0000","last_peered":"2024-09-13T14:45:14.288052+0000","last_clean":"2024-09-13T14:45:14.288052+0000","last_became_active":"2024-09-13T14:37:49.229393+0000","last_became_peered":"2024-09-13T14:37:49.229393+0000","last_unstale":"2024-09-13T14:45:14.288052+0000","last_undegraded":"2024-09-13T14:45:14.288052+0000","last_fullsized":"2024-09-13T14:45:14.288052+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:02:28.690318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":291,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-13T14:45:14.288264+0000","last_change":"2024-09-13T14:37:48.128447+0000","last_active":"2024-09-13T14:45:14.288264+0000","last_peered":"2024-09-13T14:45:14.288264+0000","last_clean":"2024-09-13T14:45:14.288264+0000","last_became_active":"2024-09-13T14:37:48.128143+0000","last_became_peered":"2024-09-13T14:37:48.128143+0000","last_unstale":"2024-09-13T14:45:14.288264+0000","last_undegraded":"2024-09-13T14:45:14.288264+0000","last_fullsized":"2024-09-13T14:45:14.288264+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:22.204039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":121,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T14:45:26.450081+0000","last_change":"2024-09-13T14:38:03.531683+0000","last_active":"2024-09-13T14:45:26.450081+0000","last_peered":"2024-09-13T14:45:26.450081+0000","last_clean":"2024-09-13T14:45:26.450081+0000","last_became_active":"2024-09-13T14:38:03.531476+0000","last_became_peered":"2024-09-13T14:38:03.531476+0000","last_unstale":"2024-09-13T14:45:26.450081+0000","last_undegraded":"2024-09-13T14:45:26.450081+0000","last_fullsized":"2024-09-13T14:45:26.450081+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:41.557569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":63,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-13T14:45:14.271499+0000","last_change":"2024-09-13T14:38:53.954281+0000","last_active":"2024-09-13T14:45:14.271499+0000","last_peered":"2024-09-13T14:45:14.271499+0000","last_clean":"2024-09-13T14:45:14.271499+0000","last_became_active":"2024-09-13T14:38:53.900912+0000","last_became_peered":"2024-09-13T14:38:53.900912+0000","last_unstale":"2024-09-13T14:45:14.271499+0000","last_undegraded":"2024-09-13T14:45:14.271499+0000","last_fullsized":"2024-09-13T14:45:14.271499+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:47.876275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":115,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-13T14:45:14.306677+0000","last_change":"2024-09-13T14:38:54.000782+0000","last_active":"2024-09-13T14:45:14.306677+0000","last_peered":"2024-09-13T14:45:14.306677+0000","last_clean":"2024-09-13T14:45:14.306677+0000","last_became_active":"2024-09-13T14:38:54.000322+0000","last_became_peered":"2024-09-13T14:38:54.000322+0000","last_unstale":"2024-09-13T14:45:14.306677+0000","last_undegraded":"2024-09-13T14:45:14.306677+0000","last_fullsized":"2024-09-13T14:45:14.306677+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:09.057597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":446,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T14:45:26.454815+0000","last_change":"2024-09-13T14:38:53.899535+0000","last_active":"2024-09-13T14:45:26.454815+0000","last_peered":"2024-09-13T14:45:26.454815+0000","last_clean":"2024-09-13T14:45:26.454815+0000","last_became_active":"2024-09-13T14:38:53.899183+0000","last_became_peered":"2024-09-13T14:38:53.899183+0000","last_unstale":"2024-09-13T14:45:26.454815+0000","last_undegraded":"2024-09-13T14:45:26.454815+0000","last_fullsized":"2024-09-13T14:45:26.454815+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:33.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":120,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T14:45:26.453328+0000","last_change":"2024-09-13T14:38:03.543426+0000","last_active":"2024-09-13T14:45:26.453328+0000","last_peered":"2024-09-13T14:45:26.453328+0000","last_clean":"2024-09-13T14:45:26.453328+0000","last_became_active":"2024-09-13T14:38:03.542661+0000","last_became_peered":"2024-09-13T14:38:03.542661+0000","last_unstale":"2024-09-13T14:45:26.453328+0000","last_undegraded":"2024-09-13T14:45:26.453328+0000","last_fullsized":"2024-09-13T14:45:26.453328+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:54.050127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":475,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T14:45:26.454757+0000","last_change":"2024-09-13T14:37:48.131321+0000","last_active":"2024-09-13T14:45:26.454757+0000","last_peered":"2024-09-13T14:45:26.454757+0000","last_clean":"2024-09-13T14:45:26.454757+0000","last_became_active":"2024-09-13T14:37:48.130998+0000","last_became_peered":"2024-09-13T14:37:48.130998+0000","last_unstale":"2024-09-13T14:45:26.454757+0000","last_undegraded":"2024-09-13T14:45:26.454757+0000","last_fullsized":"2024-09-13T14:45:26.454757+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:33:26.597825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":128,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-13T14:45:14.306723+0000","last_change":"2024-09-13T14:38:03.535454+0000","last_active":"2024-09-13T14:45:14.306723+0000","last_peered":"2024-09-13T14:45:14.306723+0000","last_clean":"2024-09-13T14:45:14.306723+0000","last_became_active":"2024-09-13T14:38:03.535131+0000","last_became_peered":"2024-09-13T14:38:03.535131+0000","last_unstale":"2024-09-13T14:45:14.306723+0000","last_undegraded":"2024-09-13T14:45:14.306723+0000","last_fullsized":"2024-09-13T14:45:14.306723+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:24.477601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":500,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T14:45:26.454979+0000","last_change":"2024-09-13T14:38:03.524939+0000","last_active":"2024-09-13T14:45:26.454979+0000","last_peered":"2024-09-13T14:45:26.454979+0000","last_clean":"2024-09-13T14:45:26.454979+0000","last_became_active":"2024-09-13T14:38:03.524406+0000","last_became_peered":"2024-09-13T14:38:03.524406+0000","last_unstale":"2024-09-13T14:45:26.454979+0000","last_undegraded":"2024-09-13T14:45:26.454979+0000","last_fullsized":"2024-09-13T14:45:26.454979+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:19:45.257424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":128,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T14:45:26.452001+0000","last_change":"2024-09-13T14:38:53.892948+0000","last_active":"2024-09-13T14:45:26.452001+0000","last_peered":"2024-09-13T14:45:26.452001+0000","last_clean":"2024-09-13T14:45:26.452001+0000","last_became_active":"2024-09-13T14:38:53.892665+0000","last_became_peered":"2024-09-13T14:38:53.892665+0000","last_unstale":"2024-09-13T14:45:26.452001+0000","last_undegraded":"2024-09-13T14:45:26.452001+0000","last_fullsized":"2024-09-13T14:45:26.452001+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:08.292359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":448,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T14:45:26.455016+0000","last_change":"2024-09-13T14:37:48.139348+0000","last_active":"2024-09-13T14:45:26.455016+0000","last_peered":"2024-09-13T14:45:26.455016+0000","last_clean":"2024-09-13T14:45:26.455016+0000","last_became_active":"2024-09-13T14:37:48.139115+0000","last_became_peered":"2024-09-13T14:37:48.139115+0000","last_unstale":"2024-09-13T14:45:26.455016+0000","last_undegraded":"2024-09-13T14:45:26.455016+0000","last_fullsized":"2024-09-13T14:45:26.455016+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:32:20.413346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":489,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T14:45:26.454978+0000","last_change":"2024-09-13T14:37:48.172088+0000","last_active":"2024-09-13T14:45:26.454978+0000","last_peered":"2024-09-13T14:45:26.454978+0000","last_clean":"2024-09-13T14:45:26.454978+0000","last_became_active":"2024-09-13T14:37:48.171779+0000","last_became_peered":"2024-09-13T14:37:48.171779+0000","last_unstale":"2024-09-13T14:45:26.454978+0000","last_undegraded":"2024-09-13T14:45:26.454978+0000","last_fullsized":"2024-09-13T14:45:26.454978+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:02.852712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":355,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-13T14:45:14.287942+0000","last_change":"2024-09-13T14:38:53.890236+0000","last_active":"2024-09-13T14:45:14.287942+0000","last_peered":"2024-09-13T14:45:14.287942+0000","last_clean":"2024-09-13T14:45:14.287942+0000","last_became_active":"2024-09-13T14:38:53.889582+0000","last_became_peered":"2024-09-13T14:38:53.889582+0000","last_unstale":"2024-09-13T14:45:14.287942+0000","last_undegraded":"2024-09-13T14:45:14.287942+0000","last_fullsized":"2024-09-13T14:45:14.287942+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:14:29.334774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":475,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T14:45:26.455685+0000","last_change":"2024-09-13T14:38:03.534097+0000","last_active":"2024-09-13T14:45:26.455685+0000","last_peered":"2024-09-13T14:45:26.455685+0000","last_clean":"2024-09-13T14:45:26.455685+0000","last_became_active":"2024-09-13T14:38:03.533794+0000","last_became_peered":"2024-09-13T14:38:03.533794+0000","last_unstale":"2024-09-13T14:45:26.455685+0000","last_undegraded":"2024-09-13T14:45:26.455685+0000","last_fullsized":"2024-09-13T14:45:26.455685+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:23.272098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":120,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T14:45:26.450463+0000","last_change":"2024-09-13T14:38:03.527021+0000","last_active":"2024-09-13T14:45:26.450463+0000","last_peered":"2024-09-13T14:45:26.450463+0000","last_clean":"2024-09-13T14:45:26.450463+0000","last_became_active":"2024-09-13T14:38:03.526698+0000","last_became_peered":"2024-09-13T14:38:03.526698+0000","last_unstale":"2024-09-13T14:45:26.450463+0000","last_undegraded":"2024-09-13T14:45:26.450463+0000","last_fullsized":"2024-09-13T14:45:26.450463+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:01:35.734086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":299,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-13T14:45:14.288016+0000","last_change":"2024-09-13T14:27:12.834931+0000","last_active":"2024-09-13T14:45:14.288016+0000","last_peered":"2024-09-13T14:45:14.288016+0000","last_clean":"2024-09-13T14:45:14.288016+0000","last_became_active":"2024-09-13T14:27:12.834572+0000","last_became_peered":"2024-09-13T14:27:12.834572+0000","last_unstale":"2024-09-13T14:45:14.288016+0000","last_undegraded":"2024-09-13T14:45:14.288016+0000","last_fullsized":"2024-09-13T14:45:14.288016+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:54:21.155689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":112,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-13T14:45:14.306218+0000","last_change":"2024-09-13T14:37:49.176796+0000","last_active":"2024-09-13T14:45:14.306218+0000","last_peered":"2024-09-13T14:45:14.306218+0000","last_clean":"2024-09-13T14:45:14.306218+0000","last_became_active":"2024-09-13T14:37:49.176246+0000","last_became_peered":"2024-09-13T14:37:49.176246+0000","last_unstale":"2024-09-13T14:45:14.306218+0000","last_undegraded":"2024-09-13T14:45:14.306218+0000","last_fullsized":"2024-09-13T14:45:14.306218+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:29:24.936718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":310,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-13T14:45:14.287837+0000","last_change":"2024-09-13T14:38:03.526333+0000","last_active":"2024-09-13T14:45:14.287837+0000","last_peered":"2024-09-13T14:45:14.287837+0000","last_clean":"2024-09-13T14:45:14.287837+0000","last_became_active":"2024-09-13T14:38:03.526001+0000","last_became_peered":"2024-09-13T14:38:03.526001+0000","last_unstale":"2024-09-13T14:45:14.287837+0000","last_undegraded":"2024-09-13T14:45:14.287837+0000","last_fullsized":"2024-09-13T14:45:14.287837+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:25:25.181271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":112,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-13T14:45:14.306438+0000","last_change":"2024-09-13T14:37:49.149582+0000","last_active":"2024-09-13T14:45:14.306438+0000","last_peered":"2024-09-13T14:45:14.306438+0000","last_clean":"2024-09-13T14:45:14.306438+0000","last_became_active":"2024-09-13T14:37:49.148525+0000","last_became_peered":"2024-09-13T14:37:49.148525+0000","last_unstale":"2024-09-13T14:45:14.306438+0000","last_undegraded":"2024-09-13T14:45:14.306438+0000","last_fullsized":"2024-09-13T14:45:14.306438+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:48.655665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":521,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T14:45:26.455167+0000","last_change":"2024-09-13T14:38:03.535255+0000","last_active":"2024-09-13T14:45:26.455167+0000","last_peered":"2024-09-13T14:45:26.455167+0000","last_clean":"2024-09-13T14:45:26.455167+0000","last_became_active":"2024-09-13T14:38:03.533917+0000","last_became_peered":"2024-09-13T14:38:03.533917+0000","last_unstale":"2024-09-13T14:45:26.455167+0000","last_undegraded":"2024-09-13T14:45:26.455167+0000","last_fullsized":"2024-09-13T14:45:26.455167+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:33.559788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":485,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T14:45:26.457124+0000","last_change":"2024-09-13T14:37:48.146784+0000","last_active":"2024-09-13T14:45:26.457124+0000","last_peered":"2024-09-13T14:45:26.457124+0000","last_clean":"2024-09-13T14:45:26.457124+0000","last_became_active":"2024-09-13T14:37:48.146491+0000","last_became_peered":"2024-09-13T14:37:48.146491+0000","last_unstale":"2024-09-13T14:45:26.457124+0000","last_undegraded":"2024-09-13T14:45:26.457124+0000","last_fullsized":"2024-09-13T14:45:26.457124+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:30:54.772513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":291,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-09-13T14:45:22.895996+0000","last_change":"2024-09-13T14:37:48.127607+0000","last_active":"2024-09-13T14:45:22.895996+0000","last_peered":"2024-09-13T14:45:22.895996+0000","last_clean":"2024-09-13T14:45:22.895996+0000","last_became_active":"2024-09-13T14:37:48.127316+0000","last_became_peered":"2024-09-13T14:37:48.127316+0000","last_unstale":"2024-09-13T14:45:22.895996+0000","last_undegraded":"2024-09-13T14:45:22.895996+0000","last_fullsized":"2024-09-13T14:45:22.895996+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:53:37.678861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":127,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-13T14:45:26.453241+0000","last_change":"2024-09-13T14:38:03.563200+0000","last_active":"2024-09-13T14:45:26.453241+0000","last_peered":"2024-09-13T14:45:26.453241+0000","last_clean":"2024-09-13T14:45:26.453241+0000","last_became_active":"2024-09-13T14:38:03.534258+0000","last_became_peered":"2024-09-13T14:38:03.534258+0000","last_unstale":"2024-09-13T14:45:26.453241+0000","last_undegraded":"2024-09-13T14:45:26.453241+0000","last_fullsized":"2024-09-13T14:45:26.453241+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:49:07.814419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":225626,"num_write_kb":7219969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87804,"num_bytes_recovered":5593628696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2309357568,"data_stored":2309357568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95149,"ondisk_log_size":95149,"up":48,"acting":47,"num_store_stats":6},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1538,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":17390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25132,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16793600,"data_stored":16777292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":23,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2969600,"data_stored":2951840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234420,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":707872,"kb_used_data":221464,"kb_used_omap":1,"kb_used_meta":486398,"kb_avail":93037280,"statfs":{"total":95995035648,"available":95270174720,"internally_reserved":0,"allocated":226779136,"data_stored":225194745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":498072011},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":239,"seq":1026497183838,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":345568,"kb_used_data":8408,"kb_used_omap":8,"kb_used_meta":337143,"kb_avail":93399584,"statfs":{"total":95995035648,"available":95641174016,"internally_reserved":0,"allocated":8609792,"data_stored":7025948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8453,"internal_metadata":345235195},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757685,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1137424,"kb_used_data":448392,"kb_used_omap":1,"kb_used_meta":689022,"kb_avail":92607728,"statfs":{"total":95995035648,"available":94830313472,"internally_reserved":0,"allocated":459153408,"data_stored":457580578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":705558982},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019329,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":919192,"kb_used_data":302352,"kb_used_omap":1,"kb_used_meta":616830,"kb_avail":92825960,"statfs":{"total":95995035648,"available":95053783040,"internally_reserved":0,"allocated":309608448,"data_stored":308031693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":631634374},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435512,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":612936,"kb_used_data":3712,"kb_used_omap":5,"kb_used_meta":609210,"kb_avail":93132216,"statfs":{"total":95995035648,"available":95367389184,"internally_reserved":0,"allocated":3801088,"data_stored":2236190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":623831166},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":446676599040,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":425201762566,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926090,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2018904,"kb_used_data":1311120,"kb_used_omap":7,"kb_used_meta":707768,"kb_avail":91726248,"statfs":{"total":95995035648,"available":93927677952,"internally_reserved":0,"allocated":1342586880,"data_stored":1341006765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":724754728},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6860,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1333985280,"data_stored":1333985280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":305790976,"data_stored":305790976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":451149824,"data_stored":451149824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":218431488,"data_stored":218431488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:45:34.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:34.845 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:34.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:34.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:34 smithi102 ceph-mon[110183]: pgmap v1055: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.6 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 717 op/s; 4199/84 objects degraded (4998.810%); 2.9 MiB/s, 95 objects/s recovering 2024-09-13T14:45:34.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:34 smithi102 ceph-mon[110183]: from='client.46082 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:45:34.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:34 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 4199/84 objects degraded (4998.810%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:34.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:34 smithi102 ceph-mon[115269]: pgmap v1055: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.6 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 717 op/s; 4199/84 objects degraded (4998.810%); 2.9 MiB/s, 95 objects/s recovering 2024-09-13T14:45:34.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:34 smithi102 ceph-mon[115269]: from='client.46082 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:45:34.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:34 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 4199/84 objects degraded (4998.810%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:34.946 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:34.968 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:34.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:34.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:34.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:34 smithi195 ceph-mon[104262]: pgmap v1055: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.6 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 717 op/s; 4199/84 objects degraded (4998.810%); 2.9 MiB/s, 95 objects/s recovering 2024-09-13T14:45:34.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:34 smithi195 ceph-mon[104262]: from='client.46082 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:45:34.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:34 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 4199/84 objects degraded (4998.810%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:35.070 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:35.092 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:35.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:35.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:35.194 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:35.215 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:35.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:35.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:35.270 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:45:35.270 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:45:35.287 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:45:35.287 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:45:35.302 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:45:35.303 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:45:35.316 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:35.319 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:45:35.319 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:45:35.335 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:45:35.335 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:45:35.337 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:35.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:35.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:35.351 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:45:35.351 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:45:35.367 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:45:35.367 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:45:35.385 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:45:35.386 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:45:35.438 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:35.460 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:35.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:35.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:35.562 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:35.583 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:35.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:35.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:35.685 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:35.707 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:35.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:35.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:35.809 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:35.830 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:35.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:35.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:35.881 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:35 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1478023259' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:35.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:35 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1478023259' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:35.933 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:35.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:35.955 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:35.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:35.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:35 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1478023259' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:36.056 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:36.078 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:36.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:36.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:36.180 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:36.202 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:36.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:36.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:36.304 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:36.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:36.328 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:36.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:36.429 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:36.453 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:36.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:36.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:36.554 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:36.577 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:36.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:36.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:36.679 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:36.701 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:36.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:36.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:36.803 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:36.826 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:36.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:36.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:36.856 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:36 smithi102 ceph-mon[110183]: pgmap v1056: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.6 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 717 op/s; 4199/84 objects degraded (4998.810%); 2.9 MiB/s, 95 objects/s recovering 2024-09-13T14:45:36.857 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:36 smithi102 ceph-mon[115269]: pgmap v1056: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.6 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 717 op/s; 4199/84 objects degraded (4998.810%); 2.9 MiB/s, 95 objects/s recovering 2024-09-13T14:45:36.857 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:45:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:45:36] "GET /metrics HTTP/1.1" 200 36537 "" "Prometheus/2.51.0" 2024-09-13T14:45:36.928 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:36.949 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:36.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:36.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:36.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:36 smithi195 ceph-mon[104262]: pgmap v1056: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.6 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 717 op/s; 4199/84 objects degraded (4998.810%); 2.9 MiB/s, 95 objects/s recovering 2024-09-13T14:45:37.051 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:37.074 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:37.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:37.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:37.175 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:37.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:37.200 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:37.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:37.301 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:37.322 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:37.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:37.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:37.424 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:37.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:37.446 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:37.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:37.547 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:37.568 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:37.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:37.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:37.670 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:37.692 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:37.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:37.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:37.795 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:37.816 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:37.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:37.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:37.845 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:45:37.918 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:37.939 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:37.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:37.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:38.041 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:38.063 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:38.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:38.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:38.165 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:38.187 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:38.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:38.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:38.289 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:38.310 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:38.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:38.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:38.413 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:38.434 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:38.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:38.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:38.536 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:38.557 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:38.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:38.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:38.659 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:38.681 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:38.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:38.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:38.693 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:45:38.693 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:45:38.783 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:38.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:38.805 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:38.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:38.907 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:38.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:38 smithi102 ceph-mon[110183]: pgmap v1057: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.6 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 645 op/s; 3504/84 objects degraded (4171.429%); 2.7 MiB/s, 148 objects/s recovering 2024-09-13T14:45:38.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:38 smithi102 ceph-mon[115269]: pgmap v1057: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.6 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 645 op/s; 3504/84 objects degraded (4171.429%); 2.7 MiB/s, 148 objects/s recovering 2024-09-13T14:45:38.928 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:38.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:38.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:38.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:38 smithi195 ceph-mon[104262]: pgmap v1057: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.6 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 645 op/s; 3504/84 objects degraded (4171.429%); 2.7 MiB/s, 148 objects/s recovering 2024-09-13T14:45:39.029 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:39.051 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:39.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:39.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:39.153 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:39.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:39.175 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:39.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:39.276 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:39.298 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:39.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:39.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:39.400 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:39.423 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:39.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:39.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:39.525 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:39.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:39.548 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:39.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:39.649 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:39.670 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:39.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:39.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:39.701 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:45:39.772 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:39.794 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:39.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:39.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:39.816 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T14:45:39.896 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:39.904 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:39 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 3504/84 objects degraded (4171.429%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:39.904 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:39 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 3504/84 objects degraded (4171.429%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:39.917 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:39.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:39.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:39.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:39 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 3504/84 objects degraded (4171.429%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:45:40.019 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:40.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:40.041 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:40.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:40.142 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:40.163 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:40.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:40.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:40.265 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:40.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:40.287 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:40.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:40.388 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:40.410 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:40.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:40.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:40.512 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:40.533 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:40.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:40.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:40.635 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:40.656 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:40.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:40.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:40.758 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:40.779 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:40.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:40.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:40.881 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:40.902 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:40.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:40.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:40.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:40 smithi102 ceph-mon[115269]: pgmap v1058: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.6 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 307 op/s; 3504/84 objects degraded (4171.429%); 704 KiB/s, 104 objects/s recovering 2024-09-13T14:45:40.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:40 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:45:40.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:40 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:45:40.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:40 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3893502278' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:45:40.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:40 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:45:40.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:40 smithi102 ceph-mon[110183]: pgmap v1058: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.6 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 307 op/s; 3504/84 objects degraded (4171.429%); 704 KiB/s, 104 objects/s recovering 2024-09-13T14:45:40.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:40 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:45:40.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:40 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:45:40.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:40 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3893502278' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:45:40.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:40 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:45:40.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:40 smithi195 ceph-mon[104262]: pgmap v1058: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.6 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 307 op/s; 3504/84 objects degraded (4171.429%); 704 KiB/s, 104 objects/s recovering 2024-09-13T14:45:40.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:40 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:45:40.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:40 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:45:40.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:40 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3893502278' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:45:40.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:40 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:45:41.004 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:41.031 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:41.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:41.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:41.133 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:41.155 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:41.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:41.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:41.257 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:41.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:41.281 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:41.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:41.382 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:41.406 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:41.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:41.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:41.441 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is set 2024-09-13T14:45:41.507 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:41.534 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:41.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:41.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:41.635 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:41.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:41.657 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:41.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:41.758 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:41.766 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:41.766 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:41.766 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:41.782 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:41.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:41.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:41.883 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:41.892 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:41 smithi102 ceph-mon[110183]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:45:41.893 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:41 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:45:41.893 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:41 smithi102 ceph-mon[110183]: osdmap e299: 8 total, 8 up, 6 in 2024-09-13T14:45:41.893 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:41 smithi102 ceph-mon[115269]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:45:41.893 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:41 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:45:41.893 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:41 smithi102 ceph-mon[115269]: osdmap e299: 8 total, 8 up, 6 in 2024-09-13T14:45:41.905 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:41.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:41.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:41.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:41 smithi195 ceph-mon[104262]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:45:41.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:41 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:45:41.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:41 smithi195 ceph-mon[104262]: osdmap e299: 8 total, 8 up, 6 in 2024-09-13T14:45:42.007 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:42.029 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:42.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:42.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:42.130 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:42.155 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:42.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:42.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:42.257 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:42.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:42.279 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:42.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:42.380 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:42.403 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:42.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:42.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:42.505 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:42.527 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:42.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:42.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:42.629 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:42.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:42 smithi102 ceph-mon[115269]: pgmap v1060: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.6 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 212 op/s; 3504/84 objects degraded (4171.429%); 845 KiB/s, 125 objects/s recovering 2024-09-13T14:45:42.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:42 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2660929637' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:45:42.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:42 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:45:42.640 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'unique_pool_2' removed 2024-09-13T14:45:42.652 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:42.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:42.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:42.753 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:42.776 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:42.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:42.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:42.878 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:42.900 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:42.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:42.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:42.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:42 smithi102 ceph-mon[110183]: pgmap v1060: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.6 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 212 op/s; 3504/84 objects degraded (4171.429%); 845 KiB/s, 125 objects/s recovering 2024-09-13T14:45:42.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:42 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2660929637' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:45:42.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:42 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:45:42.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:42 smithi195 ceph-mon[104262]: pgmap v1060: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.6 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 212 op/s; 3504/84 objects degraded (4171.429%); 845 KiB/s, 125 objects/s recovering 2024-09-13T14:45:42.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:42 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2660929637' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:45:42.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:42 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:45:43.002 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:43.024 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:43.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:43.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:43.125 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:43.148 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:43.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:43.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:43.250 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:43.271 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:43.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:43.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:43.373 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:43.495 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:43.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:43.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:43.597 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:43.618 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:43.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:43.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:43.720 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:43.742 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:43.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:43.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:43.844 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:43.866 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:43.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:43.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:43.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:43 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:45:43.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:43 smithi102 ceph-mon[115269]: osdmap e300: 8 total, 8 up, 6 in 2024-09-13T14:45:43.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:43 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:45:43.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:43 smithi102 ceph-mon[110183]: osdmap e300: 8 total, 8 up, 6 in 2024-09-13T14:45:43.968 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:43.988 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-13T14:45:43.989 INFO:tasks.radosbench:Beginning radosbench... 2024-09-13T14:45:43.989 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-09-13T14:45:43.989 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_3 16 2024-09-13T14:45:43.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:43.993 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:43.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:43.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:43 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:45:43.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:43 smithi195 ceph-mon[104262]: osdmap e300: 8 total, 8 up, 6 in 2024-09-13T14:45:44.094 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:44.117 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:44.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:44.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:44.219 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:44.242 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:44.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:44.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:44.344 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:44.367 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:44.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:44.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:44.468 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:44.490 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:44.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:44.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:44.591 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:44.612 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:44.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:44.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:44.714 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:44.740 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:44.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:44.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:44.839 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:44 smithi102 ceph-mon[115269]: pgmap v1062: 73 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 4.6 MiB data, 5.4 GiB used, 531 GiB / 536 GiB avail; 9/84 objects degraded (10.714%) 2024-09-13T14:45:44.839 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:44 smithi102 ceph-mon[115269]: osdmap e301: 8 total, 8 up, 6 in 2024-09-13T14:45:44.839 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:44 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 9/84 objects degraded (10.714%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:45:44.840 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:44 smithi102 ceph-mon[110183]: pgmap v1062: 73 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 4.6 MiB data, 5.4 GiB used, 531 GiB / 536 GiB avail; 9/84 objects degraded (10.714%) 2024-09-13T14:45:44.840 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:44 smithi102 ceph-mon[110183]: osdmap e301: 8 total, 8 up, 6 in 2024-09-13T14:45:44.840 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:44 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 9/84 objects degraded (10.714%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:45:44.841 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:44.864 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:44.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:44.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:44.947 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:45:44.947 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:45:44.947 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:45:44.947 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:45:44.948 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:45:44.948 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:45:44.948 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:45:44.948 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:45:44.948 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:45:44.948 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:45:44.948 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:45:44.948 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:45:44.949 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:45:44.949 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:45:44.957 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:45:44.957 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:45:44.966 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:44.992 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:44.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:44.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:44.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:44 smithi195 ceph-mon[104262]: pgmap v1062: 73 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 4.6 MiB data, 5.4 GiB used, 531 GiB / 536 GiB avail; 9/84 objects degraded (10.714%) 2024-09-13T14:45:44.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:44 smithi195 ceph-mon[104262]: osdmap e301: 8 total, 8 up, 6 in 2024-09-13T14:45:44.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:44 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 9/84 objects degraded (10.714%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:45:45.095 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:45.116 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:45.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:45.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:45.218 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:45.244 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:45.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:45.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:45.346 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:45.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:45.372 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:45.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:45.474 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:45.499 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:45.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:45.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:45.601 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:45.623 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:45.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:45.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:45.725 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:45.751 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:45.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:45.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:45.852 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:45.877 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:45.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:45.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:45.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:45 smithi102 ceph-mon[115269]: osdmap e302: 8 total, 8 up, 6 in 2024-09-13T14:45:45.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:45 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1944180643' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:45.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:45 smithi102 ceph-mon[110183]: osdmap e302: 8 total, 8 up, 6 in 2024-09-13T14:45:45.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:45 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1944180643' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:45.979 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:45.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:45 smithi195 ceph-mon[104262]: osdmap e302: 8 total, 8 up, 6 in 2024-09-13T14:45:45.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:45 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1944180643' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:46.005 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:46.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:46.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:46.107 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:46.129 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:46.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:46.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:46.231 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:46.253 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:46.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:46.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:46.355 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:46.377 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:46.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:46.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:46.479 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:46.501 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:46.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:46.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:46.603 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:46.627 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:46.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:46.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:46.655 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:45:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:45:46] "GET /metrics HTTP/1.1" 200 36531 "" "Prometheus/2.51.0" 2024-09-13T14:45:46.730 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:46.750 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:46.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:46.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:46.852 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:46.878 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:46.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:46.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:46.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:46 smithi102 ceph-mon[110183]: pgmap v1065: 73 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 4.6 MiB data, 5.4 GiB used, 531 GiB / 536 GiB avail; 9/84 objects degraded (10.714%) 2024-09-13T14:45:46.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:46 smithi102 ceph-mon[115269]: pgmap v1065: 73 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 4.6 MiB data, 5.4 GiB used, 531 GiB / 536 GiB avail; 9/84 objects degraded (10.714%) 2024-09-13T14:45:46.980 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:46.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:46 smithi195 ceph-mon[104262]: pgmap v1065: 73 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 4.6 MiB data, 5.4 GiB used, 531 GiB / 536 GiB avail; 9/84 objects degraded (10.714%) 2024-09-13T14:45:47.002 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:47.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:47.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:47.104 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:47.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:47.126 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:47.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:47.227 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:47.252 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:47.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:47.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:47.354 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:47.377 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:47.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:47.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:47.479 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:47.505 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:47.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:47.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:47.607 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:47.631 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:47.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:47.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:47.733 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:47.758 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:47.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:47.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:47.859 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:47.882 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:47.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:47.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:47.967 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:45:47.984 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:47.989 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:47 smithi102 ceph-mon[115269]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 9/84 objects degraded (10.714%), 1 pg degraded, 1 pg undersized) 2024-09-13T14:45:47.989 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:47 smithi102 ceph-mon[110183]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 9/84 objects degraded (10.714%), 1 pg degraded, 1 pg undersized) 2024-09-13T14:45:47.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:47 smithi195 ceph-mon[104262]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 9/84 objects degraded (10.714%), 1 pg degraded, 1 pg undersized) 2024-09-13T14:45:48.009 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:48.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:48.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:48.111 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:48.132 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:48.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:48.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:48.234 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:48.255 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:48.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:48.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:48.347 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:45:48.347 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:45:48.357 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:48.383 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:48.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:48.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:48.485 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:48.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:48.512 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:48.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:48.614 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:48.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:48.641 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:48.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:48.742 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:48.765 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:48.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:48.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:48.867 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:48.888 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:48.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:48.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:48.990 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:48.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:48 smithi195 ceph-mon[104262]: pgmap v1066: 73 pgs: 73 active+clean; 4.6 MiB data, 5.0 GiB used, 531 GiB / 536 GiB avail 2024-09-13T14:45:48.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:48 smithi195 ceph-mon[104262]: from='client.36186 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:45:49.016 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:49.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:49.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:49.093 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:48 smithi102 ceph-mon[115269]: pgmap v1066: 73 pgs: 73 active+clean; 4.6 MiB data, 5.0 GiB used, 531 GiB / 536 GiB avail 2024-09-13T14:45:49.093 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:48 smithi102 ceph-mon[115269]: from='client.36186 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:45:49.094 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:48 smithi102 ceph-mon[110183]: pgmap v1066: 73 pgs: 73 active+clean; 4.6 MiB data, 5.0 GiB used, 531 GiB / 536 GiB avail 2024-09-13T14:45:49.094 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:48 smithi102 ceph-mon[110183]: from='client.36186 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:45:49.114 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":1066,"stamp":"2024-09-13T14:45:47.563416+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790245,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":6376,"num_read_kb":13833,"num_write":1123,"num_write_kb":15041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":5966965,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1182,"ondisk_log_size":1182,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":245,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":5208972,"kb_used_data":1774776,"kb_used_omap":26,"kb_used_meta":3434085,"kb_avail":557261940,"statfs":{"total":575970213888,"available":570636226560,"internally_reserved":0,"allocated":1817370624,"data_stored":1807823507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27144,"internal_metadata":3516503544},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3504,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-225626,"num_write_kb":-7219969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-88490,"num_bytes_recovered":-5593628696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001483"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":492,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T14:45:41.523294+0000","last_change":"2024-09-13T14:36:13.859927+0000","last_active":"2024-09-13T14:45:41.523294+0000","last_peered":"2024-09-13T14:45:41.523294+0000","last_clean":"2024-09-13T14:45:41.523294+0000","last_became_active":"2024-09-13T14:36:13.859505+0000","last_became_peered":"2024-09-13T14:36:13.859505+0000","last_unstale":"2024-09-13T14:45:41.523294+0000","last_undegraded":"2024-09-13T14:45:41.523294+0000","last_fullsized":"2024-09-13T14:45:41.523294+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:12:20.989627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":305,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-13T14:45:43.649469+0000","last_change":"2024-09-13T14:37:48.130968+0000","last_active":"2024-09-13T14:45:43.649469+0000","last_peered":"2024-09-13T14:45:43.649469+0000","last_clean":"2024-09-13T14:45:43.649469+0000","last_became_active":"2024-09-13T14:37:48.130603+0000","last_became_peered":"2024-09-13T14:37:48.130603+0000","last_unstale":"2024-09-13T14:45:43.649469+0000","last_undegraded":"2024-09-13T14:45:43.649469+0000","last_fullsized":"2024-09-13T14:45:43.649469+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:42.626386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":101,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-13T14:45:43.647294+0000","last_change":"2024-09-13T14:36:13.862181+0000","last_active":"2024-09-13T14:45:43.647294+0000","last_peered":"2024-09-13T14:45:43.647294+0000","last_clean":"2024-09-13T14:45:43.647294+0000","last_became_active":"2024-09-13T14:36:13.861873+0000","last_became_peered":"2024-09-13T14:36:13.861873+0000","last_unstale":"2024-09-13T14:45:43.647294+0000","last_undegraded":"2024-09-13T14:45:43.647294+0000","last_fullsized":"2024-09-13T14:45:43.647294+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:14.085700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":136,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-13T14:45:44.645253+0000","last_change":"2024-09-13T14:38:53.896672+0000","last_active":"2024-09-13T14:45:44.645253+0000","last_peered":"2024-09-13T14:45:44.645253+0000","last_clean":"2024-09-13T14:45:44.645253+0000","last_became_active":"2024-09-13T14:38:53.896346+0000","last_became_peered":"2024-09-13T14:38:53.896346+0000","last_unstale":"2024-09-13T14:45:44.645253+0000","last_undegraded":"2024-09-13T14:45:44.645253+0000","last_fullsized":"2024-09-13T14:45:44.645253+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:57.789686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":136,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-13T14:45:44.645810+0000","last_change":"2024-09-13T14:38:53.893021+0000","last_active":"2024-09-13T14:45:44.645810+0000","last_peered":"2024-09-13T14:45:44.645810+0000","last_clean":"2024-09-13T14:45:44.645810+0000","last_became_active":"2024-09-13T14:38:53.892668+0000","last_became_peered":"2024-09-13T14:38:53.892668+0000","last_unstale":"2024-09-13T14:45:44.645810+0000","last_undegraded":"2024-09-13T14:45:44.645810+0000","last_fullsized":"2024-09-13T14:45:44.645810+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:57.180738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":273,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-13T14:45:43.649423+0000","last_change":"2024-09-13T14:37:48.134642+0000","last_active":"2024-09-13T14:45:43.649423+0000","last_peered":"2024-09-13T14:45:43.649423+0000","last_clean":"2024-09-13T14:45:43.649423+0000","last_became_active":"2024-09-13T14:37:48.134402+0000","last_became_peered":"2024-09-13T14:37:48.134402+0000","last_unstale":"2024-09-13T14:45:43.649423+0000","last_undegraded":"2024-09-13T14:45:43.649423+0000","last_fullsized":"2024-09-13T14:45:43.649423+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:51.313663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":511,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T14:45:41.523694+0000","last_change":"2024-09-13T14:38:53.903022+0000","last_active":"2024-09-13T14:45:41.523694+0000","last_peered":"2024-09-13T14:45:41.523694+0000","last_clean":"2024-09-13T14:45:41.523694+0000","last_became_active":"2024-09-13T14:38:53.902448+0000","last_became_peered":"2024-09-13T14:38:53.902448+0000","last_unstale":"2024-09-13T14:45:41.523694+0000","last_undegraded":"2024-09-13T14:45:41.523694+0000","last_fullsized":"2024-09-13T14:45:41.523694+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:24.690401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":139,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-13T14:45:44.644861+0000","last_change":"2024-09-13T14:38:03.554500+0000","last_active":"2024-09-13T14:45:44.644861+0000","last_peered":"2024-09-13T14:45:44.644861+0000","last_clean":"2024-09-13T14:45:44.644861+0000","last_became_active":"2024-09-13T14:38:03.532910+0000","last_became_peered":"2024-09-13T14:38:03.532910+0000","last_unstale":"2024-09-13T14:45:44.644861+0000","last_undegraded":"2024-09-13T14:45:44.644861+0000","last_fullsized":"2024-09-13T14:45:44.644861+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:27:50.843498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":66,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-13T14:45:43.650162+0000","last_change":"2024-09-13T14:38:53.940238+0000","last_active":"2024-09-13T14:45:43.650162+0000","last_peered":"2024-09-13T14:45:43.650162+0000","last_clean":"2024-09-13T14:45:43.650162+0000","last_became_active":"2024-09-13T14:38:53.892825+0000","last_became_peered":"2024-09-13T14:38:53.892825+0000","last_unstale":"2024-09-13T14:45:43.650162+0000","last_undegraded":"2024-09-13T14:45:43.650162+0000","last_fullsized":"2024-09-13T14:45:43.650162+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:46.666581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":122,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-13T14:45:43.647028+0000","last_change":"2024-09-13T14:38:53.898907+0000","last_active":"2024-09-13T14:45:43.647028+0000","last_peered":"2024-09-13T14:45:43.647028+0000","last_clean":"2024-09-13T14:45:43.647028+0000","last_became_active":"2024-09-13T14:38:53.898572+0000","last_became_peered":"2024-09-13T14:38:53.898572+0000","last_unstale":"2024-09-13T14:45:43.647028+0000","last_undegraded":"2024-09-13T14:45:43.647028+0000","last_fullsized":"2024-09-13T14:45:43.647028+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:29.222150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":66,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-13T14:45:43.650003+0000","last_change":"2024-09-13T14:38:53.904885+0000","last_active":"2024-09-13T14:45:43.650003+0000","last_peered":"2024-09-13T14:45:43.650003+0000","last_clean":"2024-09-13T14:45:43.650003+0000","last_became_active":"2024-09-13T14:38:53.891726+0000","last_became_peered":"2024-09-13T14:38:53.891726+0000","last_unstale":"2024-09-13T14:45:43.650003+0000","last_undegraded":"2024-09-13T14:45:43.650003+0000","last_fullsized":"2024-09-13T14:45:43.650003+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:57.079205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":309,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T14:45:41.523097+0000","last_change":"2024-09-13T14:37:48.160504+0000","last_active":"2024-09-13T14:45:41.523097+0000","last_peered":"2024-09-13T14:45:41.523097+0000","last_clean":"2024-09-13T14:45:41.523097+0000","last_became_active":"2024-09-13T14:37:48.160181+0000","last_became_peered":"2024-09-13T14:37:48.160181+0000","last_unstale":"2024-09-13T14:45:41.523097+0000","last_undegraded":"2024-09-13T14:45:41.523097+0000","last_fullsized":"2024-09-13T14:45:41.523097+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:37.087008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":303,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-13T14:45:43.649809+0000","last_change":"2024-09-13T14:37:49.611774+0000","last_active":"2024-09-13T14:45:43.649809+0000","last_peered":"2024-09-13T14:45:43.649809+0000","last_clean":"2024-09-13T14:45:43.649809+0000","last_became_active":"2024-09-13T14:37:49.611088+0000","last_became_peered":"2024-09-13T14:37:49.611088+0000","last_unstale":"2024-09-13T14:45:43.649809+0000","last_undegraded":"2024-09-13T14:45:43.649809+0000","last_fullsized":"2024-09-13T14:45:43.649809+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:08.933213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":62,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-13T14:45:43.649310+0000","last_change":"2024-09-13T14:38:53.899211+0000","last_active":"2024-09-13T14:45:43.649310+0000","last_peered":"2024-09-13T14:45:43.649310+0000","last_clean":"2024-09-13T14:45:43.649310+0000","last_became_active":"2024-09-13T14:38:53.898927+0000","last_became_peered":"2024-09-13T14:38:53.898927+0000","last_unstale":"2024-09-13T14:45:43.649310+0000","last_undegraded":"2024-09-13T14:45:43.649310+0000","last_fullsized":"2024-09-13T14:45:43.649310+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:00:28.857611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":113,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-13T14:45:43.647111+0000","last_change":"2024-09-13T14:37:49.168003+0000","last_active":"2024-09-13T14:45:43.647111+0000","last_peered":"2024-09-13T14:45:43.647111+0000","last_clean":"2024-09-13T14:45:43.647111+0000","last_became_active":"2024-09-13T14:37:49.167638+0000","last_became_peered":"2024-09-13T14:37:49.167638+0000","last_unstale":"2024-09-13T14:45:43.647111+0000","last_undegraded":"2024-09-13T14:45:43.647111+0000","last_fullsized":"2024-09-13T14:45:43.647111+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:29.051981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":129,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-13T14:45:44.645260+0000","last_change":"2024-09-13T14:38:03.528381+0000","last_active":"2024-09-13T14:45:44.645260+0000","last_peered":"2024-09-13T14:45:44.645260+0000","last_clean":"2024-09-13T14:45:44.645260+0000","last_became_active":"2024-09-13T14:38:03.527282+0000","last_became_peered":"2024-09-13T14:38:03.527282+0000","last_unstale":"2024-09-13T14:45:44.645260+0000","last_undegraded":"2024-09-13T14:45:44.645260+0000","last_fullsized":"2024-09-13T14:45:44.645260+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:24.435067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":481,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T14:45:41.523228+0000","last_change":"2024-09-13T14:38:03.549595+0000","last_active":"2024-09-13T14:45:41.523228+0000","last_peered":"2024-09-13T14:45:41.523228+0000","last_clean":"2024-09-13T14:45:41.523228+0000","last_became_active":"2024-09-13T14:38:03.533331+0000","last_became_peered":"2024-09-13T14:38:03.533331+0000","last_unstale":"2024-09-13T14:45:41.523228+0000","last_undegraded":"2024-09-13T14:45:41.523228+0000","last_fullsized":"2024-09-13T14:45:41.523228+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:59:02.470343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":285,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T14:45:41.524129+0000","last_change":"2024-09-13T14:37:48.127600+0000","last_active":"2024-09-13T14:45:41.524129+0000","last_peered":"2024-09-13T14:45:41.524129+0000","last_clean":"2024-09-13T14:45:41.524129+0000","last_became_active":"2024-09-13T14:37:48.127349+0000","last_became_peered":"2024-09-13T14:37:48.127349+0000","last_unstale":"2024-09-13T14:45:41.524129+0000","last_undegraded":"2024-09-13T14:45:41.524129+0000","last_fullsized":"2024-09-13T14:45:41.524129+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:57.268374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":218,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T14:45:41.523595+0000","last_change":"2024-09-13T14:37:48.127604+0000","last_active":"2024-09-13T14:45:41.523595+0000","last_peered":"2024-09-13T14:45:41.523595+0000","last_clean":"2024-09-13T14:45:41.523595+0000","last_became_active":"2024-09-13T14:37:48.127300+0000","last_became_peered":"2024-09-13T14:37:48.127300+0000","last_unstale":"2024-09-13T14:45:41.523595+0000","last_undegraded":"2024-09-13T14:45:41.523595+0000","last_fullsized":"2024-09-13T14:45:41.523595+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:51.999506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":338,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T14:45:41.523872+0000","last_change":"2024-09-13T14:38:03.529967+0000","last_active":"2024-09-13T14:45:41.523872+0000","last_peered":"2024-09-13T14:45:41.523872+0000","last_clean":"2024-09-13T14:45:41.523872+0000","last_became_active":"2024-09-13T14:38:03.529646+0000","last_became_peered":"2024-09-13T14:38:03.529646+0000","last_unstale":"2024-09-13T14:45:41.523872+0000","last_undegraded":"2024-09-13T14:45:41.523872+0000","last_fullsized":"2024-09-13T14:45:41.523872+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:52:05.193279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":298,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T14:45:41.523391+0000","last_change":"2024-09-13T14:36:13.860367+0000","last_active":"2024-09-13T14:45:41.523391+0000","last_peered":"2024-09-13T14:45:41.523391+0000","last_clean":"2024-09-13T14:45:41.523391+0000","last_became_active":"2024-09-13T14:36:13.859982+0000","last_became_peered":"2024-09-13T14:36:13.859982+0000","last_unstale":"2024-09-13T14:45:41.523391+0000","last_undegraded":"2024-09-13T14:45:41.523391+0000","last_fullsized":"2024-09-13T14:45:41.523391+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:08:23.311350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":294,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T14:45:41.523065+0000","last_change":"2024-09-13T14:37:48.147360+0000","last_active":"2024-09-13T14:45:41.523065+0000","last_peered":"2024-09-13T14:45:41.523065+0000","last_clean":"2024-09-13T14:45:41.523065+0000","last_became_active":"2024-09-13T14:37:48.146885+0000","last_became_peered":"2024-09-13T14:37:48.146885+0000","last_unstale":"2024-09-13T14:45:41.523065+0000","last_undegraded":"2024-09-13T14:45:41.523065+0000","last_fullsized":"2024-09-13T14:45:41.523065+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:27.841810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":499,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T14:45:41.522947+0000","last_change":"2024-09-13T14:38:53.900041+0000","last_active":"2024-09-13T14:45:41.522947+0000","last_peered":"2024-09-13T14:45:41.522947+0000","last_clean":"2024-09-13T14:45:41.522947+0000","last_became_active":"2024-09-13T14:38:53.899203+0000","last_became_peered":"2024-09-13T14:38:53.899203+0000","last_unstale":"2024-09-13T14:45:41.522947+0000","last_undegraded":"2024-09-13T14:45:41.522947+0000","last_fullsized":"2024-09-13T14:45:41.522947+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:28:07.883094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":431,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T14:45:41.523825+0000","last_change":"2024-09-13T14:26:15.823198+0000","last_active":"2024-09-13T14:45:41.523825+0000","last_peered":"2024-09-13T14:45:41.523825+0000","last_clean":"2024-09-13T14:45:41.523825+0000","last_became_active":"2024-09-13T14:26:15.822717+0000","last_became_peered":"2024-09-13T14:26:15.822717+0000","last_unstale":"2024-09-13T14:45:41.523825+0000","last_undegraded":"2024-09-13T14:45:41.523825+0000","last_fullsized":"2024-09-13T14:45:41.523825+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:22.471622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":436,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-13T14:45:43.649084+0000","last_change":"2024-09-13T14:41:53.196835+0000","last_active":"2024-09-13T14:45:43.649084+0000","last_peered":"2024-09-13T14:45:43.649084+0000","last_clean":"2024-09-13T14:45:43.649084+0000","last_became_active":"2024-09-13T14:41:53.196387+0000","last_became_peered":"2024-09-13T14:41:53.196387+0000","last_unstale":"2024-09-13T14:45:43.649084+0000","last_undegraded":"2024-09-13T14:45:43.649084+0000","last_fullsized":"2024-09-13T14:45:43.649084+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:07:41.927740+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:49.972303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":571,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-13T14:45:44.649708+0000","last_change":"2024-09-13T14:45:44.649708+0000","last_active":"2024-09-13T14:45:44.649708+0000","last_peered":"2024-09-13T14:45:44.649708+0000","last_clean":"2024-09-13T14:45:44.649708+0000","last_became_active":"2024-09-13T14:45:44.649148+0000","last_became_peered":"2024-09-13T14:45:44.649148+0000","last_unstale":"2024-09-13T14:45:44.649708+0000","last_undegraded":"2024-09-13T14:45:44.649708+0000","last_fullsized":"2024-09-13T14:45:44.649708+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:57:36.951987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":434,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T14:45:41.523117+0000","last_change":"2024-09-13T14:37:49.160719+0000","last_active":"2024-09-13T14:45:41.523117+0000","last_peered":"2024-09-13T14:45:41.523117+0000","last_clean":"2024-09-13T14:45:41.523117+0000","last_became_active":"2024-09-13T14:37:49.160401+0000","last_became_peered":"2024-09-13T14:37:49.160401+0000","last_unstale":"2024-09-13T14:45:41.523117+0000","last_undegraded":"2024-09-13T14:45:41.523117+0000","last_fullsized":"2024-09-13T14:45:41.523117+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:21.737945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":145,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T14:45:41.523802+0000","last_change":"2024-09-13T14:37:48.172940+0000","last_active":"2024-09-13T14:45:41.523802+0000","last_peered":"2024-09-13T14:45:41.523802+0000","last_clean":"2024-09-13T14:45:41.523802+0000","last_became_active":"2024-09-13T14:37:48.172060+0000","last_became_peered":"2024-09-13T14:37:48.172060+0000","last_unstale":"2024-09-13T14:45:41.523802+0000","last_undegraded":"2024-09-13T14:45:41.523802+0000","last_fullsized":"2024-09-13T14:45:41.523802+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:07.480645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":520,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-13T14:45:43.649003+0000","last_change":"2024-09-13T14:26:15.835178+0000","last_active":"2024-09-13T14:45:43.649003+0000","last_peered":"2024-09-13T14:45:43.649003+0000","last_clean":"2024-09-13T14:45:43.649003+0000","last_became_active":"2024-09-13T14:26:15.820283+0000","last_became_peered":"2024-09-13T14:26:15.820283+0000","last_unstale":"2024-09-13T14:45:43.649003+0000","last_undegraded":"2024-09-13T14:45:43.649003+0000","last_fullsized":"2024-09-13T14:45:43.649003+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:30.340038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":129,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-13T14:45:44.644890+0000","last_change":"2024-09-13T14:38:03.525682+0000","last_active":"2024-09-13T14:45:44.644890+0000","last_peered":"2024-09-13T14:45:44.644890+0000","last_clean":"2024-09-13T14:45:44.644890+0000","last_became_active":"2024-09-13T14:38:03.525406+0000","last_became_peered":"2024-09-13T14:38:03.525406+0000","last_unstale":"2024-09-13T14:45:44.644890+0000","last_undegraded":"2024-09-13T14:45:44.644890+0000","last_fullsized":"2024-09-13T14:45:44.644890+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:41:28.970300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":136,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-13T14:45:44.645817+0000","last_change":"2024-09-13T14:38:53.991738+0000","last_active":"2024-09-13T14:45:44.645817+0000","last_peered":"2024-09-13T14:45:44.645817+0000","last_clean":"2024-09-13T14:45:44.645817+0000","last_became_active":"2024-09-13T14:38:53.991417+0000","last_became_peered":"2024-09-13T14:38:53.991417+0000","last_unstale":"2024-09-13T14:45:44.645817+0000","last_undegraded":"2024-09-13T14:45:44.645817+0000","last_fullsized":"2024-09-13T14:45:44.645817+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:50:38.559838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":844,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T14:45:41.523083+0000","last_change":"2024-09-13T14:37:48.131799+0000","last_active":"2024-09-13T14:45:41.523083+0000","last_peered":"2024-09-13T14:45:41.523083+0000","last_clean":"2024-09-13T14:45:41.523083+0000","last_became_active":"2024-09-13T14:37:48.130837+0000","last_became_peered":"2024-09-13T14:37:48.130837+0000","last_unstale":"2024-09-13T14:45:41.523083+0000","last_undegraded":"2024-09-13T14:45:41.523083+0000","last_fullsized":"2024-09-13T14:45:41.523083+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:06:32.663770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":369,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-13T14:45:44.645163+0000","last_change":"2024-09-13T14:37:49.301531+0000","last_active":"2024-09-13T14:45:44.645163+0000","last_peered":"2024-09-13T14:45:44.645163+0000","last_clean":"2024-09-13T14:45:44.645163+0000","last_became_active":"2024-09-13T14:37:49.301249+0000","last_became_peered":"2024-09-13T14:37:49.301249+0000","last_unstale":"2024-09-13T14:45:44.645163+0000","last_undegraded":"2024-09-13T14:45:44.645163+0000","last_fullsized":"2024-09-13T14:45:44.645163+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:10.977710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":129,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-13T14:45:43.647348+0000","last_change":"2024-09-13T14:38:53.901427+0000","last_active":"2024-09-13T14:45:43.647348+0000","last_peered":"2024-09-13T14:45:43.647348+0000","last_clean":"2024-09-13T14:45:43.647348+0000","last_became_active":"2024-09-13T14:38:53.901157+0000","last_became_peered":"2024-09-13T14:38:53.901157+0000","last_unstale":"2024-09-13T14:45:43.647348+0000","last_undegraded":"2024-09-13T14:45:43.647348+0000","last_fullsized":"2024-09-13T14:45:43.647348+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:28:21.713025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":1011,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-13T14:45:43.647019+0000","last_change":"2024-09-13T14:37:49.147498+0000","last_active":"2024-09-13T14:45:43.647019+0000","last_peered":"2024-09-13T14:45:43.647019+0000","last_clean":"2024-09-13T14:45:43.647019+0000","last_became_active":"2024-09-13T14:37:49.147028+0000","last_became_peered":"2024-09-13T14:37:49.147028+0000","last_unstale":"2024-09-13T14:45:43.647019+0000","last_undegraded":"2024-09-13T14:45:43.647019+0000","last_fullsized":"2024-09-13T14:45:43.647019+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:11:29.050178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":307,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T14:45:41.523688+0000","last_change":"2024-09-13T14:38:03.528133+0000","last_active":"2024-09-13T14:45:41.523688+0000","last_peered":"2024-09-13T14:45:41.523688+0000","last_clean":"2024-09-13T14:45:41.523688+0000","last_became_active":"2024-09-13T14:38:03.527829+0000","last_became_peered":"2024-09-13T14:38:03.527829+0000","last_unstale":"2024-09-13T14:45:41.523688+0000","last_undegraded":"2024-09-13T14:45:41.523688+0000","last_fullsized":"2024-09-13T14:45:41.523688+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:48.254169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":483,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T14:45:41.523290+0000","last_change":"2024-09-13T14:38:53.899755+0000","last_active":"2024-09-13T14:45:41.523290+0000","last_peered":"2024-09-13T14:45:41.523290+0000","last_clean":"2024-09-13T14:45:41.523290+0000","last_became_active":"2024-09-13T14:38:53.899448+0000","last_became_peered":"2024-09-13T14:38:53.899448+0000","last_unstale":"2024-09-13T14:45:41.523290+0000","last_undegraded":"2024-09-13T14:45:41.523290+0000","last_fullsized":"2024-09-13T14:45:41.523290+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:55.897035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":113,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-13T14:45:43.647200+0000","last_change":"2024-09-13T14:37:49.149770+0000","last_active":"2024-09-13T14:45:43.647200+0000","last_peered":"2024-09-13T14:45:43.647200+0000","last_clean":"2024-09-13T14:45:43.647200+0000","last_became_active":"2024-09-13T14:37:49.148991+0000","last_became_peered":"2024-09-13T14:37:49.148991+0000","last_unstale":"2024-09-13T14:45:43.647200+0000","last_undegraded":"2024-09-13T14:45:43.647200+0000","last_fullsized":"2024-09-13T14:45:43.647200+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:04:04.193366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":1188,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-13T14:45:44.645468+0000","last_change":"2024-09-13T14:30:58.928302+0000","last_active":"2024-09-13T14:45:44.645468+0000","last_peered":"2024-09-13T14:45:44.645468+0000","last_clean":"2024-09-13T14:45:44.645468+0000","last_became_active":"2024-09-13T14:30:58.927718+0000","last_became_peered":"2024-09-13T14:30:58.927718+0000","last_unstale":"2024-09-13T14:45:44.645468+0000","last_undegraded":"2024-09-13T14:45:44.645468+0000","last_fullsized":"2024-09-13T14:45:44.645468+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:37.323036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":120,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-13T14:45:43.647108+0000","last_change":"2024-09-13T14:38:03.534628+0000","last_active":"2024-09-13T14:45:43.647108+0000","last_peered":"2024-09-13T14:45:43.647108+0000","last_clean":"2024-09-13T14:45:43.647108+0000","last_became_active":"2024-09-13T14:38:03.534262+0000","last_became_peered":"2024-09-13T14:38:03.534262+0000","last_unstale":"2024-09-13T14:45:43.647108+0000","last_undegraded":"2024-09-13T14:45:43.647108+0000","last_fullsized":"2024-09-13T14:45:43.647108+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:31.168783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":132,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-13T14:45:43.647204+0000","last_change":"2024-09-13T14:38:03.534303+0000","last_active":"2024-09-13T14:45:43.647204+0000","last_peered":"2024-09-13T14:45:43.647204+0000","last_clean":"2024-09-13T14:45:43.647204+0000","last_became_active":"2024-09-13T14:38:03.533171+0000","last_became_peered":"2024-09-13T14:38:03.533171+0000","last_unstale":"2024-09-13T14:45:43.647204+0000","last_undegraded":"2024-09-13T14:45:43.647204+0000","last_fullsized":"2024-09-13T14:45:43.647204+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:00:16.553526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":825,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-13T14:45:44.648512+0000","last_change":"2024-09-13T14:38:04.596192+0000","last_active":"2024-09-13T14:45:44.648512+0000","last_peered":"2024-09-13T14:45:44.648512+0000","last_clean":"2024-09-13T14:45:44.648512+0000","last_became_active":"2024-09-13T14:38:04.593299+0000","last_became_peered":"2024-09-13T14:38:04.593299+0000","last_unstale":"2024-09-13T14:45:44.648512+0000","last_undegraded":"2024-09-13T14:45:44.648512+0000","last_fullsized":"2024-09-13T14:45:44.648512+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:34.241410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":129,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-13T14:45:44.645259+0000","last_change":"2024-09-13T14:38:03.533120+0000","last_active":"2024-09-13T14:45:44.645259+0000","last_peered":"2024-09-13T14:45:44.645259+0000","last_clean":"2024-09-13T14:45:44.645259+0000","last_became_active":"2024-09-13T14:38:03.532829+0000","last_became_peered":"2024-09-13T14:38:03.532829+0000","last_unstale":"2024-09-13T14:45:44.645259+0000","last_undegraded":"2024-09-13T14:45:44.645259+0000","last_fullsized":"2024-09-13T14:45:44.645259+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:24.514661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":525,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T14:45:41.523833+0000","last_change":"2024-09-13T14:38:53.938227+0000","last_active":"2024-09-13T14:45:41.523833+0000","last_peered":"2024-09-13T14:45:41.523833+0000","last_clean":"2024-09-13T14:45:41.523833+0000","last_became_active":"2024-09-13T14:38:53.902867+0000","last_became_peered":"2024-09-13T14:38:53.902867+0000","last_unstale":"2024-09-13T14:45:41.523833+0000","last_undegraded":"2024-09-13T14:45:41.523833+0000","last_fullsized":"2024-09-13T14:45:41.523833+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:16.831756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":601,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T14:45:41.523203+0000","last_change":"2024-09-13T14:37:48.147677+0000","last_active":"2024-09-13T14:45:41.523203+0000","last_peered":"2024-09-13T14:45:41.523203+0000","last_clean":"2024-09-13T14:45:41.523203+0000","last_became_active":"2024-09-13T14:37:48.147001+0000","last_became_peered":"2024-09-13T14:37:48.147001+0000","last_unstale":"2024-09-13T14:45:41.523203+0000","last_undegraded":"2024-09-13T14:45:41.523203+0000","last_fullsized":"2024-09-13T14:45:41.523203+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:35:00.653576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":507,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T14:45:41.522825+0000","last_change":"2024-09-13T14:38:53.900107+0000","last_active":"2024-09-13T14:45:41.522825+0000","last_peered":"2024-09-13T14:45:41.522825+0000","last_clean":"2024-09-13T14:45:41.522825+0000","last_became_active":"2024-09-13T14:38:53.899556+0000","last_became_peered":"2024-09-13T14:38:53.899556+0000","last_unstale":"2024-09-13T14:45:41.522825+0000","last_undegraded":"2024-09-13T14:45:41.522825+0000","last_fullsized":"2024-09-13T14:45:41.522825+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:15:47.918163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":144,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-13T14:45:44.645715+0000","last_change":"2024-09-13T14:38:53.901806+0000","last_active":"2024-09-13T14:45:44.645715+0000","last_peered":"2024-09-13T14:45:44.645715+0000","last_clean":"2024-09-13T14:45:44.645715+0000","last_became_active":"2024-09-13T14:38:53.891055+0000","last_became_peered":"2024-09-13T14:38:53.891055+0000","last_unstale":"2024-09-13T14:45:44.645715+0000","last_undegraded":"2024-09-13T14:45:44.645715+0000","last_fullsized":"2024-09-13T14:45:44.645715+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:03:00.416198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":78,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-13T14:45:43.650168+0000","last_change":"2024-09-13T14:43:16.854299+0000","last_active":"2024-09-13T14:45:43.650168+0000","last_peered":"2024-09-13T14:45:43.650168+0000","last_clean":"2024-09-13T14:45:43.650168+0000","last_became_active":"2024-09-13T14:38:53.900626+0000","last_became_peered":"2024-09-13T14:38:53.900626+0000","last_unstale":"2024-09-13T14:45:43.650168+0000","last_undegraded":"2024-09-13T14:45:43.650168+0000","last_fullsized":"2024-09-13T14:45:43.650168+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:37:21.715778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":291,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T14:45:41.523254+0000","last_change":"2024-09-13T14:37:49.172218+0000","last_active":"2024-09-13T14:45:41.523254+0000","last_peered":"2024-09-13T14:45:41.523254+0000","last_clean":"2024-09-13T14:45:41.523254+0000","last_became_active":"2024-09-13T14:37:49.171718+0000","last_became_peered":"2024-09-13T14:37:49.171718+0000","last_unstale":"2024-09-13T14:45:41.523254+0000","last_undegraded":"2024-09-13T14:45:41.523254+0000","last_fullsized":"2024-09-13T14:45:41.523254+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:46:54.626521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":313,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T14:45:41.523493+0000","last_change":"2024-09-13T14:37:49.230094+0000","last_active":"2024-09-13T14:45:41.523493+0000","last_peered":"2024-09-13T14:45:41.523493+0000","last_clean":"2024-09-13T14:45:41.523493+0000","last_became_active":"2024-09-13T14:37:49.229393+0000","last_became_peered":"2024-09-13T14:37:49.229393+0000","last_unstale":"2024-09-13T14:45:41.523493+0000","last_undegraded":"2024-09-13T14:45:41.523493+0000","last_fullsized":"2024-09-13T14:45:41.523493+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:02:28.690318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":293,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T14:45:41.524198+0000","last_change":"2024-09-13T14:37:48.128447+0000","last_active":"2024-09-13T14:45:41.524198+0000","last_peered":"2024-09-13T14:45:41.524198+0000","last_clean":"2024-09-13T14:45:41.524198+0000","last_became_active":"2024-09-13T14:37:48.128143+0000","last_became_peered":"2024-09-13T14:37:48.128143+0000","last_unstale":"2024-09-13T14:45:41.524198+0000","last_undegraded":"2024-09-13T14:45:41.524198+0000","last_fullsized":"2024-09-13T14:45:41.524198+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:22.204039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":129,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-13T14:45:44.645159+0000","last_change":"2024-09-13T14:38:03.531683+0000","last_active":"2024-09-13T14:45:44.645159+0000","last_peered":"2024-09-13T14:45:44.645159+0000","last_clean":"2024-09-13T14:45:44.645159+0000","last_became_active":"2024-09-13T14:38:03.531476+0000","last_became_peered":"2024-09-13T14:38:03.531476+0000","last_unstale":"2024-09-13T14:45:44.645159+0000","last_undegraded":"2024-09-13T14:45:44.645159+0000","last_fullsized":"2024-09-13T14:45:44.645159+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:41.557569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":67,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-13T14:45:43.649641+0000","last_change":"2024-09-13T14:38:53.954281+0000","last_active":"2024-09-13T14:45:43.649641+0000","last_peered":"2024-09-13T14:45:43.649641+0000","last_clean":"2024-09-13T14:45:43.649641+0000","last_became_active":"2024-09-13T14:38:53.900912+0000","last_became_peered":"2024-09-13T14:38:53.900912+0000","last_unstale":"2024-09-13T14:45:43.649641+0000","last_undegraded":"2024-09-13T14:45:43.649641+0000","last_fullsized":"2024-09-13T14:45:43.649641+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:47.876275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":119,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-13T14:45:43.647263+0000","last_change":"2024-09-13T14:38:54.000782+0000","last_active":"2024-09-13T14:45:43.647263+0000","last_peered":"2024-09-13T14:45:43.647263+0000","last_clean":"2024-09-13T14:45:43.647263+0000","last_became_active":"2024-09-13T14:38:54.000322+0000","last_became_peered":"2024-09-13T14:38:54.000322+0000","last_unstale":"2024-09-13T14:45:43.647263+0000","last_undegraded":"2024-09-13T14:45:43.647263+0000","last_fullsized":"2024-09-13T14:45:43.647263+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:09.057597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":448,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T14:45:41.522760+0000","last_change":"2024-09-13T14:38:53.899535+0000","last_active":"2024-09-13T14:45:41.522760+0000","last_peered":"2024-09-13T14:45:41.522760+0000","last_clean":"2024-09-13T14:45:41.522760+0000","last_became_active":"2024-09-13T14:38:53.899183+0000","last_became_peered":"2024-09-13T14:38:53.899183+0000","last_unstale":"2024-09-13T14:45:41.522760+0000","last_undegraded":"2024-09-13T14:45:41.522760+0000","last_fullsized":"2024-09-13T14:45:41.522760+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:33.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":129,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-13T14:45:44.645682+0000","last_change":"2024-09-13T14:38:03.543426+0000","last_active":"2024-09-13T14:45:44.645682+0000","last_peered":"2024-09-13T14:45:44.645682+0000","last_clean":"2024-09-13T14:45:44.645682+0000","last_became_active":"2024-09-13T14:38:03.542661+0000","last_became_peered":"2024-09-13T14:38:03.542661+0000","last_unstale":"2024-09-13T14:45:44.645682+0000","last_undegraded":"2024-09-13T14:45:44.645682+0000","last_fullsized":"2024-09-13T14:45:44.645682+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:54.050127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":477,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T14:45:41.522810+0000","last_change":"2024-09-13T14:37:48.131321+0000","last_active":"2024-09-13T14:45:41.522810+0000","last_peered":"2024-09-13T14:45:41.522810+0000","last_clean":"2024-09-13T14:45:41.522810+0000","last_became_active":"2024-09-13T14:37:48.130998+0000","last_became_peered":"2024-09-13T14:37:48.130998+0000","last_unstale":"2024-09-13T14:45:41.522810+0000","last_undegraded":"2024-09-13T14:45:41.522810+0000","last_fullsized":"2024-09-13T14:45:41.522810+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:33:26.597825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":132,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-13T14:45:43.647357+0000","last_change":"2024-09-13T14:38:03.535454+0000","last_active":"2024-09-13T14:45:43.647357+0000","last_peered":"2024-09-13T14:45:43.647357+0000","last_clean":"2024-09-13T14:45:43.647357+0000","last_became_active":"2024-09-13T14:38:03.535131+0000","last_became_peered":"2024-09-13T14:38:03.535131+0000","last_unstale":"2024-09-13T14:45:43.647357+0000","last_undegraded":"2024-09-13T14:45:43.647357+0000","last_fullsized":"2024-09-13T14:45:43.647357+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:24.477601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":502,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T14:45:41.522931+0000","last_change":"2024-09-13T14:38:03.524939+0000","last_active":"2024-09-13T14:45:41.522931+0000","last_peered":"2024-09-13T14:45:41.522931+0000","last_clean":"2024-09-13T14:45:41.522931+0000","last_became_active":"2024-09-13T14:38:03.524406+0000","last_became_peered":"2024-09-13T14:38:03.524406+0000","last_unstale":"2024-09-13T14:45:41.522931+0000","last_undegraded":"2024-09-13T14:45:41.522931+0000","last_fullsized":"2024-09-13T14:45:41.522931+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:19:45.257424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":136,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-13T14:45:44.645636+0000","last_change":"2024-09-13T14:38:53.892948+0000","last_active":"2024-09-13T14:45:44.645636+0000","last_peered":"2024-09-13T14:45:44.645636+0000","last_clean":"2024-09-13T14:45:44.645636+0000","last_became_active":"2024-09-13T14:38:53.892665+0000","last_became_peered":"2024-09-13T14:38:53.892665+0000","last_unstale":"2024-09-13T14:45:44.645636+0000","last_undegraded":"2024-09-13T14:45:44.645636+0000","last_fullsized":"2024-09-13T14:45:44.645636+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:08.292359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":451,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T14:45:41.522990+0000","last_change":"2024-09-13T14:37:48.139348+0000","last_active":"2024-09-13T14:45:41.522990+0000","last_peered":"2024-09-13T14:45:41.522990+0000","last_clean":"2024-09-13T14:45:41.522990+0000","last_became_active":"2024-09-13T14:37:48.139115+0000","last_became_peered":"2024-09-13T14:37:48.139115+0000","last_unstale":"2024-09-13T14:45:41.522990+0000","last_undegraded":"2024-09-13T14:45:41.522990+0000","last_fullsized":"2024-09-13T14:45:41.522990+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:32:20.413346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":491,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T14:45:41.523363+0000","last_change":"2024-09-13T14:37:48.172088+0000","last_active":"2024-09-13T14:45:41.523363+0000","last_peered":"2024-09-13T14:45:41.523363+0000","last_clean":"2024-09-13T14:45:41.523363+0000","last_became_active":"2024-09-13T14:37:48.171779+0000","last_became_peered":"2024-09-13T14:37:48.171779+0000","last_unstale":"2024-09-13T14:45:41.523363+0000","last_undegraded":"2024-09-13T14:45:41.523363+0000","last_fullsized":"2024-09-13T14:45:41.523363+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:02.852712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":357,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T14:45:41.523677+0000","last_change":"2024-09-13T14:38:53.890236+0000","last_active":"2024-09-13T14:45:41.523677+0000","last_peered":"2024-09-13T14:45:41.523677+0000","last_clean":"2024-09-13T14:45:41.523677+0000","last_became_active":"2024-09-13T14:38:53.889582+0000","last_became_peered":"2024-09-13T14:38:53.889582+0000","last_unstale":"2024-09-13T14:45:41.523677+0000","last_undegraded":"2024-09-13T14:45:41.523677+0000","last_fullsized":"2024-09-13T14:45:41.523677+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:14:29.334774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":477,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T14:45:41.523734+0000","last_change":"2024-09-13T14:38:03.534097+0000","last_active":"2024-09-13T14:45:41.523734+0000","last_peered":"2024-09-13T14:45:41.523734+0000","last_clean":"2024-09-13T14:45:41.523734+0000","last_became_active":"2024-09-13T14:38:03.533794+0000","last_became_peered":"2024-09-13T14:38:03.533794+0000","last_unstale":"2024-09-13T14:45:41.523734+0000","last_undegraded":"2024-09-13T14:45:41.523734+0000","last_fullsized":"2024-09-13T14:45:41.523734+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:23.272098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":129,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-13T14:45:44.645155+0000","last_change":"2024-09-13T14:38:03.527021+0000","last_active":"2024-09-13T14:45:44.645155+0000","last_peered":"2024-09-13T14:45:44.645155+0000","last_clean":"2024-09-13T14:45:44.645155+0000","last_became_active":"2024-09-13T14:38:03.526698+0000","last_became_peered":"2024-09-13T14:38:03.526698+0000","last_unstale":"2024-09-13T14:45:44.645155+0000","last_undegraded":"2024-09-13T14:45:44.645155+0000","last_fullsized":"2024-09-13T14:45:44.645155+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:01:35.734086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":301,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T14:45:41.523424+0000","last_change":"2024-09-13T14:27:12.834931+0000","last_active":"2024-09-13T14:45:41.523424+0000","last_peered":"2024-09-13T14:45:41.523424+0000","last_clean":"2024-09-13T14:45:41.523424+0000","last_became_active":"2024-09-13T14:27:12.834572+0000","last_became_peered":"2024-09-13T14:27:12.834572+0000","last_unstale":"2024-09-13T14:45:41.523424+0000","last_undegraded":"2024-09-13T14:45:41.523424+0000","last_fullsized":"2024-09-13T14:45:41.523424+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:54:21.155689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":116,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-13T14:45:43.647138+0000","last_change":"2024-09-13T14:37:49.176796+0000","last_active":"2024-09-13T14:45:43.647138+0000","last_peered":"2024-09-13T14:45:43.647138+0000","last_clean":"2024-09-13T14:45:43.647138+0000","last_became_active":"2024-09-13T14:37:49.176246+0000","last_became_peered":"2024-09-13T14:37:49.176246+0000","last_unstale":"2024-09-13T14:45:43.647138+0000","last_undegraded":"2024-09-13T14:45:43.647138+0000","last_fullsized":"2024-09-13T14:45:43.647138+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:29:24.936718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":312,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T14:45:41.523524+0000","last_change":"2024-09-13T14:38:03.526333+0000","last_active":"2024-09-13T14:45:41.523524+0000","last_peered":"2024-09-13T14:45:41.523524+0000","last_clean":"2024-09-13T14:45:41.523524+0000","last_became_active":"2024-09-13T14:38:03.526001+0000","last_became_peered":"2024-09-13T14:38:03.526001+0000","last_unstale":"2024-09-13T14:45:41.523524+0000","last_undegraded":"2024-09-13T14:45:41.523524+0000","last_fullsized":"2024-09-13T14:45:41.523524+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:25:25.181271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":116,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-13T14:45:43.647262+0000","last_change":"2024-09-13T14:37:49.149582+0000","last_active":"2024-09-13T14:45:43.647262+0000","last_peered":"2024-09-13T14:45:43.647262+0000","last_clean":"2024-09-13T14:45:43.647262+0000","last_became_active":"2024-09-13T14:37:49.148525+0000","last_became_peered":"2024-09-13T14:37:49.148525+0000","last_unstale":"2024-09-13T14:45:43.647262+0000","last_undegraded":"2024-09-13T14:45:43.647262+0000","last_fullsized":"2024-09-13T14:45:43.647262+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:48.655665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":523,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T14:45:41.523042+0000","last_change":"2024-09-13T14:38:03.535255+0000","last_active":"2024-09-13T14:45:41.523042+0000","last_peered":"2024-09-13T14:45:41.523042+0000","last_clean":"2024-09-13T14:45:41.523042+0000","last_became_active":"2024-09-13T14:38:03.533917+0000","last_became_peered":"2024-09-13T14:38:03.533917+0000","last_unstale":"2024-09-13T14:45:41.523042+0000","last_undegraded":"2024-09-13T14:45:41.523042+0000","last_fullsized":"2024-09-13T14:45:41.523042+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:33.559788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":487,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T14:45:41.523644+0000","last_change":"2024-09-13T14:37:48.146784+0000","last_active":"2024-09-13T14:45:41.523644+0000","last_peered":"2024-09-13T14:45:41.523644+0000","last_clean":"2024-09-13T14:45:41.523644+0000","last_became_active":"2024-09-13T14:37:48.146491+0000","last_became_peered":"2024-09-13T14:37:48.146491+0000","last_unstale":"2024-09-13T14:45:41.523644+0000","last_undegraded":"2024-09-13T14:45:41.523644+0000","last_fullsized":"2024-09-13T14:45:41.523644+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:30:54.772513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":293,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-13T14:45:41.522871+0000","last_change":"2024-09-13T14:37:48.127607+0000","last_active":"2024-09-13T14:45:41.522871+0000","last_peered":"2024-09-13T14:45:41.522871+0000","last_clean":"2024-09-13T14:45:41.522871+0000","last_became_active":"2024-09-13T14:37:48.127316+0000","last_became_peered":"2024-09-13T14:37:48.127316+0000","last_unstale":"2024-09-13T14:45:41.522871+0000","last_undegraded":"2024-09-13T14:45:41.522871+0000","last_fullsized":"2024-09-13T14:45:41.522871+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:53:37.678861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":135,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-13T14:45:44.645595+0000","last_change":"2024-09-13T14:38:03.563200+0000","last_active":"2024-09-13T14:45:44.645595+0000","last_peered":"2024-09-13T14:45:44.645595+0000","last_clean":"2024-09-13T14:45:44.645595+0000","last_became_active":"2024-09-13T14:38:03.534258+0000","last_became_peered":"2024-09-13T14:38:03.534258+0000","last_unstale":"2024-09-13T14:45:44.645595+0000","last_undegraded":"2024-09-13T14:45:44.645595+0000","last_fullsized":"2024-09-13T14:45:44.645595+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:49:07.814419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1538,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":17278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25132,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16793600,"data_stored":16777292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2969600,"data_stored":2951840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234423,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":650452,"kb_used_data":164044,"kb_used_omap":1,"kb_used_meta":486398,"kb_avail":93094700,"statfs":{"total":95995035648,"available":95328972800,"internally_reserved":0,"allocated":167981056,"data_stored":166375586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":498072011},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":239,"seq":1026497183841,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":345632,"kb_used_data":8460,"kb_used_omap":8,"kb_used_meta":337143,"kb_avail":93399520,"statfs":{"total":95995035648,"available":95641108480,"internally_reserved":0,"allocated":8663040,"data_stored":7058117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8453,"internal_metadata":345235195},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757688,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":684620,"kb_used_data":7864,"kb_used_omap":1,"kb_used_meta":676734,"kb_avail":93060532,"statfs":{"total":95995035648,"available":95293984768,"internally_reserved":0,"allocated":8052736,"data_stored":6462901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":692976070},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019331,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":919208,"kb_used_data":302364,"kb_used_omap":1,"kb_used_meta":616830,"kb_avail":92825944,"statfs":{"total":95995035648,"available":95053766656,"internally_reserved":0,"allocated":309620736,"data_stored":308039957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":631634374},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435514,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":612952,"kb_used_data":3724,"kb_used_omap":5,"kb_used_meta":609210,"kb_avail":93132200,"statfs":{"total":95995035648,"available":95367372800,"internally_reserved":0,"allocated":3813376,"data_stored":2244454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":623831166},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":446676599043,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":425201762569,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926093,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1996108,"kb_used_data":1288320,"kb_used_omap":7,"kb_used_meta":707768,"kb_avail":91749044,"statfs":{"total":95995035648,"available":93951021056,"internally_reserved":0,"allocated":1319239680,"data_stored":1317642492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":724754728},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6860,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:45:49.117 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-13T14:45:49.118 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:45:49.118 INFO:tasks.thrashosds.thrasher:Adding osd 2 2024-09-13T14:45:49.118 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd in 2 2024-09-13T14:45:49.119 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:49.144 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:49.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:49.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:49.246 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:49.270 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:49.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:49.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:49.372 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:49.395 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:49.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:49.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:49.415 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:45:49.415 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [], 2024-09-13T14:45:49.415 INFO:teuthology.orchestra.run.smithi102.stdout: "complaint_time": 30, 2024-09-13T14:45:49.415 INFO:teuthology.orchestra.run.smithi102.stdout: "num_blocked_ops": 0 2024-09-13T14:45:49.415 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:45:49.497 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:49.522 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:49.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:49.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:49.624 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:49.650 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:49.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:49.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:49.751 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:49.775 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:49.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:49.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:49.877 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:49.901 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:49.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:49.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:50.003 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:50.026 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:50.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:50.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:50.128 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:50.151 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:50.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:50.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:50.203 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T14:45:50.254 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:50.309 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:50.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:50.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:50.410 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:50.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:50.435 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:50.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:50.536 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:50.558 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:50.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:50.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:50.660 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:50.661 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:50.686 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:50.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:50.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:50.697 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:50.788 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:50.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:50.810 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:50.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:50.903 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:50 smithi102 ceph-mon[110183]: pgmap v1067: 73 pgs: 73 active+clean; 4.6 MiB data, 5.0 GiB used, 531 GiB / 536 GiB avail; 0 B/s, 1 objects/s recovering 2024-09-13T14:45:50.903 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:50 smithi102 ceph-mon[115269]: pgmap v1067: 73 pgs: 73 active+clean; 4.6 MiB data, 5.0 GiB used, 531 GiB / 536 GiB avail; 0 B/s, 1 objects/s recovering 2024-09-13T14:45:50.911 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:50.933 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:50.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:50.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:50.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:50 smithi195 ceph-mon[104262]: pgmap v1067: 73 pgs: 73 active+clean; 4.6 MiB data, 5.0 GiB used, 531 GiB / 536 GiB avail; 0 B/s, 1 objects/s recovering 2024-09-13T14:45:51.036 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:51.057 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:51.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:51.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:51.159 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:51.185 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:51.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:51.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:51.287 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:51.308 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:51.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:51.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:51.411 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:51.436 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:51.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:51.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:51.538 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:51.561 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:51.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:51.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:51.663 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:51.685 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:51.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:51.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:51.787 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:51.809 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:51.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:51.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:51.912 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:51.938 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:51.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:51.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:52.033 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:52.034 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:52.040 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:52.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:52.063 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:52.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:52.164 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:52.186 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:52.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:52.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:52.287 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:52.308 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:52.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:52.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:52.410 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:52.436 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:52.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:52.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:52.538 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:52.560 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:52.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:52.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:52.662 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:52.684 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:52.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:52.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:52.686 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:45:52.786 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:52.807 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:52.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:52.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:52.847 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:52 smithi102 ceph-mon[110183]: pgmap v1068: 73 pgs: 73 active+clean; 4.6 MiB data, 5.0 GiB used, 531 GiB / 536 GiB avail; 0 B/s, 1 objects/s recovering 2024-09-13T14:45:52.848 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:52 smithi102 ceph-mon[115269]: pgmap v1068: 73 pgs: 73 active+clean; 4.6 MiB data, 5.0 GiB used, 531 GiB / 536 GiB avail; 0 B/s, 1 objects/s recovering 2024-09-13T14:45:52.909 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:52.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:52.933 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:52.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:52.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:52 smithi195 ceph-mon[104262]: pgmap v1068: 73 pgs: 73 active+clean; 4.6 MiB data, 5.0 GiB used, 531 GiB / 536 GiB avail; 0 B/s, 1 objects/s recovering 2024-09-13T14:45:53.034 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:53.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:53.057 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:53.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:53.158 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:53.184 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:53.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:53.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:53.286 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:53.311 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:53.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:53.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:53.413 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:53.435 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:53.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:53.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:53.537 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:53.558 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:53.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:53.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:53.660 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:53.684 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:53.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:53.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:53.786 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:53.807 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:53.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:53.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:53.909 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:53.934 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:53.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:53.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:54.035 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:54.060 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:54.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:54.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:54.164 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:54.189 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:54.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:54.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:54.291 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:54.316 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:54.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:54.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:54.418 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:54.442 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:54.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:54.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:54.544 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:54.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:54.566 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:54.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:54.667 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:54.688 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:54.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:54.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:54.723 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is unset 2024-09-13T14:45:54.790 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:54.817 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:54.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:54.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:54.918 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:54.943 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:54.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:54.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:54.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:54 smithi195 ceph-mon[104262]: pgmap v1069: 73 pgs: 73 active+clean; 4.6 MiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-13T14:45:54.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:54 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2308045273' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:45:54.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:54 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:45:55.045 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:55.066 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:55.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:55.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:55.168 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:55.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:54 smithi102 ceph-mon[115269]: pgmap v1069: 73 pgs: 73 active+clean; 4.6 MiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-13T14:45:55.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:54 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2308045273' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:45:55.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:54 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:45:55.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:54 smithi102 ceph-mon[110183]: pgmap v1069: 73 pgs: 73 active+clean; 4.6 MiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-13T14:45:55.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:54 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2308045273' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:45:55.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:54 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:45:55.194 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:55.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:55.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:55.296 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:55.321 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:55.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:55.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:55.423 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:55.446 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:55.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:55.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:55.548 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:55.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:55.574 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:55.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:55.675 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:55.699 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:55.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:55.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:55.801 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:55.825 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:55.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:55.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:55.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:55 smithi102 ceph-mon[110183]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:45:55.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:55 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:45:55.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:55 smithi102 ceph-mon[110183]: osdmap e303: 8 total, 8 up, 6 in 2024-09-13T14:45:55.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:55 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:45:55.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:55 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:45:55.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:55 smithi102 ceph-mon[115269]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:45:55.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:55 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:45:55.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:55 smithi102 ceph-mon[115269]: osdmap e303: 8 total, 8 up, 6 in 2024-09-13T14:45:55.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:55 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:45:55.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:55 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:45:55.926 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:55.947 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:55.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:55.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:55.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:55 smithi195 ceph-mon[104262]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:45:55.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:55 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:45:55.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:55 smithi195 ceph-mon[104262]: osdmap e303: 8 total, 8 up, 6 in 2024-09-13T14:45:55.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:55 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:45:55.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:55 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:45:56.049 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:56.074 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:56.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:56.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:56.177 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:56.197 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:56.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:56.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:56.299 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:56.325 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:56.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:56.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:56.427 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:56.450 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:56.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:56.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:56.552 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:56.574 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:56.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:56.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:56.630 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:45:56 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:45:56] "GET /metrics HTTP/1.1" 200 34164 "" "Prometheus/2.51.0" 2024-09-13T14:45:56.676 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:56.701 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:56.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:56.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:56.803 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:56.828 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:56.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:56.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:56.831 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:45:56.831 INFO:teuthology.orchestra.run.smithi102.stdout: "size": 20, 2024-09-13T14:45:56.831 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 600, 2024-09-13T14:45:56.831 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [] 2024-09-13T14:45:56.831 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:45:56.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:56 smithi102 ceph-mon[110183]: pgmap v1071: 73 pgs: 73 active+clean; 4.6 MiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-13T14:45:56.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:56 smithi102 ceph-mon[115269]: pgmap v1071: 73 pgs: 73 active+clean; 4.6 MiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-13T14:45:56.930 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:56.954 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:56.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:56.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:56.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:56 smithi195 ceph-mon[104262]: pgmap v1071: 73 pgs: 73 active+clean; 4.6 MiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-13T14:45:57.056 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:57.080 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:57.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:57.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:57.181 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:57.205 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:57.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:57.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:57.306 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:57.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:57.330 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:57.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:57.431 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:57.453 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:57.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:57.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:57.554 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:57.576 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:45:57.580 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:57.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:57.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:57.683 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:57.704 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:57.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:57.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:57.806 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:57.813 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T14:45:57.833 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:57.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:57.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:57.934 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:57.959 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:57.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:57.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:58.061 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:58.083 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:58.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:58.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:58.114 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:58.126 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:58.185 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:58.210 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:58.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:58.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:58.312 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:58.337 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:58.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:58.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:58.439 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:58.460 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:58.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:58.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:58.562 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:58.588 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:58.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:58.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:58.690 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:58.716 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:58.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:58.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:58.737 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'unique_pool_3' created 2024-09-13T14:45:58.817 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:58.843 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:58.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:58.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:58.945 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:58.970 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:58.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:58.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:58.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:58 smithi195 ceph-mon[104262]: pgmap v1072: 73 pgs: 73 active+clean; 4.6 MiB data, 4.9 GiB used, 532 GiB / 536 GiB avail 2024-09-13T14:45:58.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:58 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/605228558' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-13T14:45:58.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:58 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-13T14:45:59.072 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:59.094 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:59.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:59.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:59.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:58 smithi102 ceph-mon[110183]: pgmap v1072: 73 pgs: 73 active+clean; 4.6 MiB data, 4.9 GiB used, 532 GiB / 536 GiB avail 2024-09-13T14:45:59.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:58 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/605228558' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-13T14:45:59.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:58 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-13T14:45:59.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:58 smithi102 ceph-mon[115269]: pgmap v1072: 73 pgs: 73 active+clean; 4.6 MiB data, 4.9 GiB used, 532 GiB / 536 GiB avail 2024-09-13T14:45:59.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:58 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/605228558' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-13T14:45:59.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:58 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-13T14:45:59.196 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:59.219 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:59.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:59.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:59.322 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:59.345 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:59.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:59.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:59.447 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:59.469 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:59.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:59.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:59.571 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:59.598 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:59.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:59.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:59.700 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:59.721 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:45:59.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:59.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:59.798 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:59 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-13T14:45:59.799 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:59 smithi102 ceph-mon[110183]: osdmap e304: 8 total, 8 up, 6 in 2024-09-13T14:45:59.799 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:59 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-13T14:45:59.799 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:45:59 smithi102 ceph-mon[115269]: osdmap e304: 8 total, 8 up, 6 in 2024-09-13T14:45:59.822 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:59.847 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:59.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:59.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:59.949 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:59.973 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:45:59.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:59.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:59.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:59 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-13T14:45:59.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:45:59 smithi195 ceph-mon[104262]: osdmap e304: 8 total, 8 up, 6 in 2024-09-13T14:46:00.075 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:00.099 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:00.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:00.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:00.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:45:59 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-a[110159]: 2024-09-13T14:45:59.796+0000 7fa3e7f42640 -1 mon.a@0(leader).osd e305 definitely_dead 0 2024-09-13T14:46:00.201 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:00.224 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:00.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:00.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:00.325 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:00.347 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:00.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:00.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:00.449 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:00.472 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:00.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:00.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:00.574 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:00.597 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:00.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:00.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:00.700 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-09-13T14:46:00.701 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:00.724 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:00.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:00.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:00.826 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:00.842 INFO:teuthology.orchestra.run.smithi102.stderr:marked in osd.2. 2024-09-13T14:46:00.848 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:00.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:00.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:00.950 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:00.971 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:00.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:00.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:00.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:00 smithi195 ceph-mon[104262]: pgmap v1074: 89 pgs: 16 unknown, 73 active+clean; 4.6 MiB data, 4.9 GiB used, 532 GiB / 536 GiB avail 2024-09-13T14:46:00.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:00 smithi195 ceph-mon[104262]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:46:00.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:00 smithi195 ceph-mon[104262]: osdmap e305: 8 total, 8 up, 6 in 2024-09-13T14:46:00.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:00 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/4147763974' entity='client.admin' cmd={"prefix": "osd in", "ids": ["2"]} : dispatch 2024-09-13T14:46:00.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:00 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["2"]} : dispatch 2024-09-13T14:46:01.073 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:01.096 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:01.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:01.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:01.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:00 smithi102 ceph-mon[110183]: pgmap v1074: 89 pgs: 16 unknown, 73 active+clean; 4.6 MiB data, 4.9 GiB used, 532 GiB / 536 GiB avail 2024-09-13T14:46:01.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:00 smithi102 ceph-mon[110183]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:46:01.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:00 smithi102 ceph-mon[110183]: osdmap e305: 8 total, 8 up, 6 in 2024-09-13T14:46:01.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:00 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/4147763974' entity='client.admin' cmd={"prefix": "osd in", "ids": ["2"]} : dispatch 2024-09-13T14:46:01.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:00 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["2"]} : dispatch 2024-09-13T14:46:01.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:00 smithi102 ceph-mon[115269]: pgmap v1074: 89 pgs: 16 unknown, 73 active+clean; 4.6 MiB data, 4.9 GiB used, 532 GiB / 536 GiB avail 2024-09-13T14:46:01.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:00 smithi102 ceph-mon[115269]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:46:01.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:00 smithi102 ceph-mon[115269]: osdmap e305: 8 total, 8 up, 6 in 2024-09-13T14:46:01.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:00 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/4147763974' entity='client.admin' cmd={"prefix": "osd in", "ids": ["2"]} : dispatch 2024-09-13T14:46:01.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:00 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["2"]} : dispatch 2024-09-13T14:46:01.198 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:01.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:01.220 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:01.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:01.321 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:01.342 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:01.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:01.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:01.444 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:01.465 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:01.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:01.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:01.567 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:01.588 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:01.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:01.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:01.609 INFO:tasks.thrashosds.thrasher:Added osd 2 2024-09-13T14:46:01.690 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:01.716 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:01.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:01.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:01.818 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:01.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:01.840 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:01.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:01.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:01 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-09-13T14:46:01.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:01 smithi102 ceph-mon[115269]: osdmap e306: 8 total, 8 up, 7 in 2024-09-13T14:46:01.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:01 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-09-13T14:46:01.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:01 smithi102 ceph-mon[110183]: osdmap e306: 8 total, 8 up, 7 in 2024-09-13T14:46:01.942 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:01.966 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:01.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:01.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:02.066 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:02.068 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:02.092 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:02.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:02.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:02.194 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:02.215 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:02.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:02.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:02.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:01 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-09-13T14:46:02.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:01 smithi195 ceph-mon[104262]: osdmap e306: 8 total, 8 up, 7 in 2024-09-13T14:46:02.317 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:02.343 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:02.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:02.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:02.444 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:02.467 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:02.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:02.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:02.569 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:02.591 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:02.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:02.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:02.693 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:02.716 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:02.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:02.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:02.818 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:02.840 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:02.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:02.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:02.942 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:02.964 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:02.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:02.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:03.005 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:02 smithi102 ceph-mon[110183]: pgmap v1077: 89 pgs: 6 peering, 16 unknown, 67 active+clean; 4.6 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-09-13T14:46:03.005 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:02 smithi102 ceph-mon[110183]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-13T14:46:03.005 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:02 smithi102 ceph-mon[110183]: osdmap e307: 8 total, 8 up, 7 in 2024-09-13T14:46:03.005 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:02 smithi102 ceph-mon[115269]: pgmap v1077: 89 pgs: 6 peering, 16 unknown, 67 active+clean; 4.6 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-09-13T14:46:03.005 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:02 smithi102 ceph-mon[115269]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-13T14:46:03.006 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:02 smithi102 ceph-mon[115269]: osdmap e307: 8 total, 8 up, 7 in 2024-09-13T14:46:03.066 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:03.087 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:03.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:03.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:03.189 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:03.213 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:03.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:03.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:03.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:02 smithi195 ceph-mon[104262]: pgmap v1077: 89 pgs: 6 peering, 16 unknown, 67 active+clean; 4.6 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-09-13T14:46:03.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:02 smithi195 ceph-mon[104262]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-13T14:46:03.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:02 smithi195 ceph-mon[104262]: osdmap e307: 8 total, 8 up, 7 in 2024-09-13T14:46:03.315 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:03.336 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:03.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:03.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:03.386 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:46:03.386 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [], 2024-09-13T14:46:03.386 INFO:teuthology.orchestra.run.smithi102.stdout: "num_ops": 0 2024-09-13T14:46:03.386 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:46:03.438 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:03.460 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:03.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:03.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:03.561 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:03.587 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:03.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:03.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:03.689 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:03.715 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:03.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:03.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:03.770 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is unset 2024-09-13T14:46:03.816 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:03.838 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:03.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:03.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:03.940 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:03.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:03.969 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:03.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:04.070 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:04.098 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:04.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:04.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:04.116 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:03 smithi102 ceph-mon[115269]: osdmap e308: 8 total, 8 up, 7 in 2024-09-13T14:46:04.117 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:03 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2773831245' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:46:04.117 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:03 smithi102 ceph-mon[110183]: osdmap e308: 8 total, 8 up, 7 in 2024-09-13T14:46:04.117 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:03 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2773831245' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:46:04.139 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T14:46:04.199 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:04.221 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:04.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:04.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:04.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:03 smithi195 ceph-mon[104262]: osdmap e308: 8 total, 8 up, 7 in 2024-09-13T14:46:04.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:03 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2773831245' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:46:04.323 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:04.344 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:04.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:04.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:04.446 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:04.467 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:04.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:04.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:04.569 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:04.589 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:04.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:04.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:04.691 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:04.713 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:04.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:04.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:04.744 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:04.814 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:04.836 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:04.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:04.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:04.937 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:04.958 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:04.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:04.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:05.060 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:05.086 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:05.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:05.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:05.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:04 smithi102 ceph-mon[110183]: pgmap v1080: 89 pgs: 6 peering, 3 remapped+peering, 80 active+clean; 4.6 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 153 B/s, 2 keys/s, 0 objects/s recovering 2024-09-13T14:46:05.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:04 smithi102 ceph-mon[110183]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:46:05.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:04 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2773831245' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:46:05.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:04 smithi102 ceph-mon[110183]: osdmap e309: 8 total, 8 up, 7 in 2024-09-13T14:46:05.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:04 smithi102 ceph-mon[115269]: pgmap v1080: 89 pgs: 6 peering, 3 remapped+peering, 80 active+clean; 4.6 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 153 B/s, 2 keys/s, 0 objects/s recovering 2024-09-13T14:46:05.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:04 smithi102 ceph-mon[115269]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:46:05.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:04 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2773831245' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:46:05.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:04 smithi102 ceph-mon[115269]: osdmap e309: 8 total, 8 up, 7 in 2024-09-13T14:46:05.189 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:05.216 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:05.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:05.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:05.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:04 smithi195 ceph-mon[104262]: pgmap v1080: 89 pgs: 6 peering, 3 remapped+peering, 80 active+clean; 4.6 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 153 B/s, 2 keys/s, 0 objects/s recovering 2024-09-13T14:46:05.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:04 smithi195 ceph-mon[104262]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:46:05.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:04 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2773831245' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:46:05.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:04 smithi195 ceph-mon[104262]: osdmap e309: 8 total, 8 up, 7 in 2024-09-13T14:46:05.318 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:05.342 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:05.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:05.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:05.444 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:05.468 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:05.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:05.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:05.569 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:05.593 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:05.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:05.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:05.694 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:05.716 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:05.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:05.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:05.817 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:05.838 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:05.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:05.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:05.940 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:05.963 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:05.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:05.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:06.064 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:06.085 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:06.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:06.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:06.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:05 smithi102 ceph-mon[110183]: osdmap e310: 8 total, 8 up, 7 in 2024-09-13T14:46:06.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:05 smithi102 ceph-mon[115269]: osdmap e310: 8 total, 8 up, 7 in 2024-09-13T14:46:06.187 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:06.207 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:06.210 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:06.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:06.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:06.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:05 smithi195 ceph-mon[104262]: osdmap e310: 8 total, 8 up, 7 in 2024-09-13T14:46:06.312 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:06.333 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:06.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:06.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:06.435 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:06.456 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:06.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:06.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:06.558 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:06.585 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:06.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:06.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:06.610 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 6, 7, 5, 2] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T14:46:06.610 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:46:06.686 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:06.712 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:06.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:06.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:06.720 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:46:06 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:46:06] "GET /metrics HTTP/1.1" 200 34164 "" "Prometheus/2.51.0" 2024-09-13T14:46:06.814 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:06.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:06.836 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:06.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:06.937 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:06.962 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:06.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:06.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:07.064 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:07.085 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:07.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:07.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:07.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:06 smithi102 ceph-mon[115269]: pgmap v1083: 89 pgs: 6 peering, 3 remapped+peering, 80 active+clean; 4.6 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 451 KiB/s rd, 311 KiB/s wr, 200 op/s; 153 B/s, 2 keys/s, 2 objects/s recovering 2024-09-13T14:46:07.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:06 smithi102 ceph-mon[110183]: pgmap v1083: 89 pgs: 6 peering, 3 remapped+peering, 80 active+clean; 4.6 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 451 KiB/s rd, 311 KiB/s wr, 200 op/s; 153 B/s, 2 keys/s, 2 objects/s recovering 2024-09-13T14:46:07.187 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:07.208 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:07.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:07.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:07.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:06 smithi195 ceph-mon[104262]: pgmap v1083: 89 pgs: 6 peering, 3 remapped+peering, 80 active+clean; 4.6 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 451 KiB/s rd, 311 KiB/s wr, 200 op/s; 153 B/s, 2 keys/s, 2 objects/s recovering 2024-09-13T14:46:07.296 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:46:07.310 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:07.360 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:07.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:07.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:07.461 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:07.487 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:07.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:07.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:07.589 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:07.614 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:07.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:07.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:07.716 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:07.717 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:46:07.717 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:46:07.717 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:46:07.718 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:46:07.732 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:46:07.732 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:46:07.739 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:07.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:07.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:07.750 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:46:07.750 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:46:07.766 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:46:07.766 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:46:07.782 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:46:07.782 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:46:07.800 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:46:07.800 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:46:07.816 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:46:07.817 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:46:07.841 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:07.867 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:07.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:07.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:07.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:07 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3217959847' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:46:07.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:07 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3217959847' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:46:07.969 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:07.994 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:07.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:07.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:08.096 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:08.119 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:08.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:08.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:08.220 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:08.242 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:08.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:08.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:08.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:07 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3217959847' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:46:08.345 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:08.370 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:08.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:08.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:08.472 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:08.493 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:08.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:08.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:08.595 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:08.620 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:08.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:08.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:08.721 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:08.743 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:08.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:08.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:08.810 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:08 smithi102 ceph-mon[110183]: pgmap v1084: 89 pgs: 89 active+clean; 4.6 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 310 KiB/s rd, 214 KiB/s wr, 137 op/s; 105 B/s, 1 keys/s, 7 objects/s recovering 2024-09-13T14:46:08.810 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:08 smithi102 ceph-mon[110183]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-09-13T14:46:08.810 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:08 smithi102 ceph-mon[115269]: pgmap v1084: 89 pgs: 89 active+clean; 4.6 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 310 KiB/s rd, 214 KiB/s wr, 137 op/s; 105 B/s, 1 keys/s, 7 objects/s recovering 2024-09-13T14:46:08.810 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:08 smithi102 ceph-mon[115269]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-09-13T14:46:08.845 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:08.865 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:08.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:08.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:08.967 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:08.988 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:08.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:08.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:09.090 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:09.116 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:09.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:09.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:09.218 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:09.242 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:09.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:09.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:09.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:08 smithi195 ceph-mon[104262]: pgmap v1084: 89 pgs: 89 active+clean; 4.6 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 310 KiB/s rd, 214 KiB/s wr, 137 op/s; 105 B/s, 1 keys/s, 7 objects/s recovering 2024-09-13T14:46:09.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:08 smithi195 ceph-mon[104262]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-09-13T14:46:09.344 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:09.365 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:09.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:09.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:09.467 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:09.492 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:09.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:09.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:09.594 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:09.615 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:09.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:09.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:09.717 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:09.738 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:09.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:09.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:09.840 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:09.861 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:09.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:09.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:09.963 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:09.987 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:09.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:09.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:10.088 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:10.109 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:10.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:10.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:10.211 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:10.232 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:10.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:10.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:10.334 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:10.355 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:10.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:10.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:10.457 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:10.479 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:10.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:10.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:10.581 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:10.602 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:10.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:10.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:10.704 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:10.724 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:10.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:10.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:10.802 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:46:10.802 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:46:10.827 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:10.836 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:10 smithi195 ceph-mon[104262]: pgmap v1085: 89 pgs: 89 active+clean; 4.6 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 0 B/s, 4 objects/s recovering 2024-09-13T14:46:10.836 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:46:10.836 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:46:10.836 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:46:10.849 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:10.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:10.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:10.950 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:10.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:10.973 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:10.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:11.074 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:11.096 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:11.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:11.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:11.102 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:10 smithi102 ceph-mon[110183]: pgmap v1085: 89 pgs: 89 active+clean; 4.6 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 0 B/s, 4 objects/s recovering 2024-09-13T14:46:11.103 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:46:11.103 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:46:11.103 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:46:11.103 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:10 smithi102 ceph-mon[115269]: pgmap v1085: 89 pgs: 89 active+clean; 4.6 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 0 B/s, 4 objects/s recovering 2024-09-13T14:46:11.103 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:46:11.103 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:46:11.103 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:46:11.200 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:11.221 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:11.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:11.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:11.324 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:11.345 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:11.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:11.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:11.447 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:11.468 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:11.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:11.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:11.532 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T14:46:11.570 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:11.583 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:11.599 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:11.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:11.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:11.700 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:11.722 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:11.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:11.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:11.796 INFO:teuthology.orchestra.run.smithi102.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-09-13T14:46:11.823 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:11.845 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:11.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:11.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:11.947 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:11.969 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:11.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:11.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:12.070 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:12.092 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:12.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:12.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:12.144 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:46:12.166 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:11 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/417001652' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:46:12.166 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:11 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/417001652' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:46:12.193 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:12.214 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:12.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:12.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:12.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:11 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/417001652' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:46:12.317 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:12.338 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:12.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:12.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:12.439 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:12.461 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:12.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:12.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:12.563 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:12.584 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:12.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:12.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:12.687 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:12.707 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:12.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:12.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:12.809 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:12.831 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:12.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:12.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:12.878 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:12 smithi102 ceph-mon[110183]: pgmap v1086: 89 pgs: 89 active+clean; 4.6 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 0 B/s, 4 objects/s recovering 2024-09-13T14:46:12.878 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:12 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/417001652' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:46:12.878 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:12 smithi102 ceph-mon[110183]: osdmap e311: 8 total, 8 up, 7 in 2024-09-13T14:46:12.879 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:12 smithi102 ceph-mon[115269]: pgmap v1086: 89 pgs: 89 active+clean; 4.6 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 0 B/s, 4 objects/s recovering 2024-09-13T14:46:12.879 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:12 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/417001652' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:46:12.879 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:12 smithi102 ceph-mon[115269]: osdmap e311: 8 total, 8 up, 7 in 2024-09-13T14:46:12.932 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:12.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:12.955 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:12.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:13.056 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:13.077 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:13.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:13.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:13.106 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:13.109 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:13.158 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:12 smithi195 ceph-mon[104262]: pgmap v1086: 89 pgs: 89 active+clean; 4.6 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 0 B/s, 4 objects/s recovering 2024-09-13T14:46:13.158 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:12 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/417001652' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:46:13.158 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:12 smithi195 ceph-mon[104262]: osdmap e311: 8 total, 8 up, 7 in 2024-09-13T14:46:13.180 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:13.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:13.202 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:13.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:13.303 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:13.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:13.325 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:13.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:13.426 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:13.447 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:13.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:13.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:13.549 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:13.569 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:13.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:13.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:13.606 DEBUG:teuthology.orchestra.run.smithi102:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-09-13T14:46:13.607 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-13T14:46:13.607 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:46:13.609 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-13T14:46:13.609 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-13T14:46:13.671 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:13.694 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:13.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:13.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:13.795 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:13.797 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:hints = 1 2024-09-13T14:46:13.797 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-13T14:46:13.797 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Object prefix: benchmark_data_smithi102_313275 2024-09-13T14:46:13.798 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:46:13.798 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 0 0 0 0 0 0 - 0 2024-09-13T14:46:13.830 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:13.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:13.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:13.932 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:13.966 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:13.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:13.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:14.068 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:14.104 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:14.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:14.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:14.214 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:13 smithi102 ceph-mon[110183]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:46:14.214 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:13 smithi102 ceph-mon[110183]: Cluster is now healthy 2024-09-13T14:46:14.214 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:13 smithi102 ceph-mon[115269]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:46:14.214 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:13 smithi102 ceph-mon[115269]: Cluster is now healthy 2024-09-13T14:46:14.240 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:13 smithi195 ceph-mon[104262]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:46:14.240 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:13 smithi195 ceph-mon[104262]: Cluster is now healthy 2024-09-13T14:46:14.315 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:14.347 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:14.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:14.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:14.449 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:14.482 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:14.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:14.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:14.584 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:14.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:14.627 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:14.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:14.728 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:14.763 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:14.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:14.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:14.798 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 1 16 4092 4076 254.734 254.75 0.00323982 0.00390927 2024-09-13T14:46:14.865 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:14.900 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:14.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:14.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:15.002 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:15.035 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:15.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:15.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:15.122 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:14 smithi102 ceph-mon[110183]: pgmap v1088: 89 pgs: 89 active+clean; 4.6 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 0 B/s, 3 objects/s recovering 2024-09-13T14:46:15.123 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:14 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:46:15.123 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:14 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:46:15.123 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:14 smithi102 ceph-mon[115269]: pgmap v1088: 89 pgs: 89 active+clean; 4.6 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 0 B/s, 3 objects/s recovering 2024-09-13T14:46:15.123 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:14 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:46:15.123 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:14 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:46:15.136 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:15.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:15.176 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:15.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:15.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:14 smithi195 ceph-mon[104262]: pgmap v1088: 89 pgs: 89 active+clean; 4.6 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 0 B/s, 3 objects/s recovering 2024-09-13T14:46:15.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:14 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:46:15.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:14 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:46:15.277 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:15.312 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:15.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:15.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:15.414 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:15.449 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:15.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:15.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:15.551 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:15.582 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:15.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:15.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:15.684 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:15.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:15.720 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:15.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:15.798 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 2 16 7710 7694 240.415 226.125 0.00138514 0.00413422 2024-09-13T14:46:15.821 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:15.842 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:15.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:15.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:15.944 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:15.977 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:15.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:15.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:16.079 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:16.101 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:16.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:16.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:16.203 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:16.238 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:16.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:16.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:16.340 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:16.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:16.377 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:16.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:16.478 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:16.511 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:16.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:16.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:16.613 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:16.649 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:16.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:16.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:16.751 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:16.789 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:16.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:16.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:16.798 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 3 16 10812 10796 224.897 193.875 0.00579992 0.0044395 2024-09-13T14:46:16.856 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:46:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:46:16] "GET /metrics HTTP/1.1" 200 36455 "" "Prometheus/2.51.0" 2024-09-13T14:46:16.891 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:16.925 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:16.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:16.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:17.027 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:17.065 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:17.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:17.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:17.121 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:16 smithi102 ceph-mon[110183]: pgmap v1089: 89 pgs: 89 active+clean; 4.6 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 0 B/s, 3 objects/s recovering 2024-09-13T14:46:17.121 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:16 smithi102 ceph-mon[115269]: pgmap v1089: 89 pgs: 89 active+clean; 4.6 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 0 B/s, 3 objects/s recovering 2024-09-13T14:46:17.167 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:17.198 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:17.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:17.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:17.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:16 smithi195 ceph-mon[104262]: pgmap v1089: 89 pgs: 89 active+clean; 4.6 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 0 B/s, 3 objects/s recovering 2024-09-13T14:46:17.300 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:17.331 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:17.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:17.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:17.433 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:17.467 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:17.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:17.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:17.569 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:17.600 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:17.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:17.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:17.702 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:17.740 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:17.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:17.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:17.798 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 4 16 14013 13997 218.682 200.062 0.00452165 0.00456576 2024-09-13T14:46:17.842 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:17.876 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:17.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:17.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:17.978 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:18.014 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:18.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:18.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:18.117 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:18.158 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:18.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:18.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:18.259 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:18.292 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:18.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:18.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:18.394 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:18.429 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:18.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:18.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:18.531 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:18.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:18.567 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:18.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:18.668 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:18.689 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:18.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:18.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:18.791 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:18.798 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 5 16 16848 16832 210.38 177.188 0.00388392 0.0047464 2024-09-13T14:46:18.826 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:18.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:18.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:18.865 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is set 2024-09-13T14:46:18.928 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:18.962 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:18.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:18.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:19.064 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:19.097 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:19.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:19.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:19.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:18 smithi102 ceph-mon[110183]: pgmap v1090: 89 pgs: 89 active+clean; 481 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 48 MiB/s wr, 762 op/s 2024-09-13T14:46:19.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:18 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/581354715' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:46:19.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:18 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:46:19.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:18 smithi102 ceph-mon[115269]: pgmap v1090: 89 pgs: 89 active+clean; 481 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 48 MiB/s wr, 762 op/s 2024-09-13T14:46:19.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:18 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/581354715' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:46:19.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:18 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:46:19.199 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:19.234 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:19.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:19.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:19.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:18 smithi195 ceph-mon[104262]: pgmap v1090: 89 pgs: 89 active+clean; 481 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 48 MiB/s wr, 762 op/s 2024-09-13T14:46:19.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:18 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/581354715' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:46:19.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:18 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:46:19.336 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:19.382 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:19.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:19.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:19.484 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:19.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:19.519 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:19.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:19.549 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:19.620 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:19.656 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:19.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:19.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:19.758 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:19.789 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:19.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:19.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:19.798 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 6 16 19838 19822 206.46 186.875 0.00460973 0.00483835 2024-09-13T14:46:19.891 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:19.926 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:19.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:19.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:19.979 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:19 smithi102 ceph-mon[110183]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:46:19.979 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:19 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:46:19.979 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:19 smithi102 ceph-mon[110183]: osdmap e312: 8 total, 8 up, 7 in 2024-09-13T14:46:19.980 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:19 smithi102 ceph-mon[115269]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:46:19.980 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:19 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:46:19.980 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:19 smithi102 ceph-mon[115269]: osdmap e312: 8 total, 8 up, 7 in 2024-09-13T14:46:20.028 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:20.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:20.062 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:20.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:20.163 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:20.198 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:20.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:20.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:20.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:19 smithi195 ceph-mon[104262]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:46:20.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:19 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:46:20.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:19 smithi195 ceph-mon[104262]: osdmap e312: 8 total, 8 up, 7 in 2024-09-13T14:46:20.301 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:20.338 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:20.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:20.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:20.440 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:20.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:20.475 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:20.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:20.576 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:20.608 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:20.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:20.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:20.713 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:20.758 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:20.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:20.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:20.798 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 7 16 22593 22577 201.562 172.188 0.00578335 0.00495622 2024-09-13T14:46:20.860 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:20.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:20.896 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:20.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:20.996 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:21.034 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:21.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:21.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:21.136 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:21.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:20 smithi102 ceph-mon[110183]: pgmap v1092: 89 pgs: 89 active+clean; 805 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-09-13T14:46:21.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:20 smithi102 ceph-mon[115269]: pgmap v1092: 89 pgs: 89 active+clean; 805 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-09-13T14:46:21.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:21.171 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:21.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:21.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:20 smithi195 ceph-mon[104262]: pgmap v1092: 89 pgs: 89 active+clean; 805 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-09-13T14:46:21.272 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:21.309 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:21.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:21.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:21.411 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:21.443 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:21.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:21.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:21.545 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:21.567 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:21.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:21.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:21.669 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:21.702 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:21.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:21.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:21.799 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 8 16 25090 25074 195.873 156.062 0.00597185 0.00509988 2024-09-13T14:46:21.803 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:21.837 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:21.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:21.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:21.941 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:21.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:21.963 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:21.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:22.016 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:46:22.064 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:22.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:22.105 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:22.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:22.206 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:22.245 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:22.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:22.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:22.347 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:22.382 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:22.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:22.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:22.484 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:22.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:22.518 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:22.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:22.619 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:22.652 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:22.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:22.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:22.754 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:22.788 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:22.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:22.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:22.798 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 9 16 27143 27127 188.365 128.312 0.00723617 0.0053034 2024-09-13T14:46:22.890 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:22.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:22.928 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:22.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:23.029 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:23.065 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:23.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:23.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:23.167 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:23.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:22 smithi102 ceph-mon[110183]: pgmap v1093: 89 pgs: 89 active+clean; 862 MiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 88 MiB/s wr, 1.40k op/s 2024-09-13T14:46:23.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:22 smithi102 ceph-mon[115269]: pgmap v1093: 89 pgs: 89 active+clean; 862 MiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 88 MiB/s wr, 1.40k op/s 2024-09-13T14:46:23.189 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:23.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:23.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:23.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:22 smithi195 ceph-mon[104262]: pgmap v1093: 89 pgs: 89 active+clean; 862 MiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 88 MiB/s wr, 1.40k op/s 2024-09-13T14:46:23.291 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:23.325 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:23.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:23.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:23.427 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:23.449 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:23.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:23.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:23.551 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:23.584 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:23.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:23.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:23.686 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:23.720 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:23.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:23.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:23.799 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 10 16 29100 29084 181.759 122.312 0.00147352 0.00546322 2024-09-13T14:46:23.822 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:23.848 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:23.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:23.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:23.950 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:23.981 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:23.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:23.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:24.083 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:24.117 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:24.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:24.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:24.219 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:24.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:24.256 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:24.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:24.357 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:24.378 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:24.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:24.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:24.480 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:24.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:24.516 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:24.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:24.618 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:24.650 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:24.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:24.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:24.752 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:24.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:24.788 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:24.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:24.799 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 11 15 30750 30735 174.615 103.188 0.010217 0.00572155 2024-09-13T14:46:24.890 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:24.927 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:24.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:24.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:25.029 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:25.052 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:25.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:25.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:25.154 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:25.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:24 smithi102 ceph-mon[115269]: pgmap v1094: 89 pgs: 89 active+clean; 1.6 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 164 MiB/s wr, 2.63k op/s 2024-09-13T14:46:25.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:24 smithi102 ceph-mon[110183]: pgmap v1094: 89 pgs: 89 active+clean; 1.6 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 164 MiB/s wr, 2.63k op/s 2024-09-13T14:46:25.188 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:25.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:25.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:25.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:24 smithi195 ceph-mon[104262]: pgmap v1094: 89 pgs: 89 active+clean; 1.6 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 164 MiB/s wr, 2.63k op/s 2024-09-13T14:46:25.290 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:25.321 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:25.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:25.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:25.424 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:25.459 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:25.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:25.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:25.561 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:25.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:25.582 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:25.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:25.683 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:25.716 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:25.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:25.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:25.799 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 12 16 32915 32899 171.334 135.25 0.00371741 0.00583094 2024-09-13T14:46:25.818 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:25.848 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:25.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:25.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:25.950 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:25.984 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:25.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:25.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:26.086 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:26.121 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:26.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:26.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:26.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:25 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [3, 2]} : dispatch 2024-09-13T14:46:26.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:25 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [3, 5]} : dispatch 2024-09-13T14:46:26.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:25 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:46:26.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:25 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:46:26.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:25 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [3, 2]} : dispatch 2024-09-13T14:46:26.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:25 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [3, 5]} : dispatch 2024-09-13T14:46:26.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:25 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:46:26.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:25 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:46:26.223 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:26.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:25 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [3, 2]} : dispatch 2024-09-13T14:46:26.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:25 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [3, 5]} : dispatch 2024-09-13T14:46:26.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:25 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:46:26.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:25 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:46:26.250 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:26.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:26.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:26.351 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:26.387 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:26.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:26.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:26.489 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:26.510 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:26.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:26.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:26.612 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:26.645 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:26.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:26.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:26.747 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:26.781 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:26.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:26.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:26.799 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 13 16 34580 34564 166.158 104.062 0.00294327 0.00601341 2024-09-13T14:46:26.883 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:26.918 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:46:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:46:26] "GET /metrics HTTP/1.1" 200 36521 "" "Prometheus/2.51.0" 2024-09-13T14:46:26.918 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:26.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:26.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:27.021 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:27.056 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:27.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:27.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:27.157 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:27.212 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:27.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:27.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:27.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:26 smithi195 ceph-mon[104262]: pgmap v1095: 89 pgs: 89 active+clean; 1.6 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 164 MiB/s wr, 2.63k op/s 2024-09-13T14:46:27.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [3, 2]}]': finished 2024-09-13T14:46:27.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [3, 5]}]': finished 2024-09-13T14:46:27.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:26 smithi195 ceph-mon[104262]: osdmap e313: 8 total, 8 up, 7 in 2024-09-13T14:46:27.315 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:27.335 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:27.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:27.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:27.372 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:46:27.372 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:46:27.373 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:26 smithi102 ceph-mon[110183]: pgmap v1095: 89 pgs: 89 active+clean; 1.6 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 164 MiB/s wr, 2.63k op/s 2024-09-13T14:46:27.373 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [3, 2]}]': finished 2024-09-13T14:46:27.373 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [3, 5]}]': finished 2024-09-13T14:46:27.373 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:26 smithi102 ceph-mon[110183]: osdmap e313: 8 total, 8 up, 7 in 2024-09-13T14:46:27.373 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:26 smithi102 ceph-mon[115269]: pgmap v1095: 89 pgs: 89 active+clean; 1.6 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 164 MiB/s wr, 2.63k op/s 2024-09-13T14:46:27.373 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [3, 2]}]': finished 2024-09-13T14:46:27.374 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [3, 5]}]': finished 2024-09-13T14:46:27.374 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:26 smithi102 ceph-mon[115269]: osdmap e313: 8 total, 8 up, 7 in 2024-09-13T14:46:27.437 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:27.475 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:27.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:27.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:27.577 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:27.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:27.617 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:27.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:27.718 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:27.761 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:27.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:27.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:27.799 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 14 16 36500 36484 162.86 120 0.00239566 0.00613494 2024-09-13T14:46:27.863 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:27.899 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:27.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:27.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:28.001 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:28.032 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:28.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:28.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:28.093 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:28.094 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T14:46:28.134 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:28.182 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:28.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:28.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:28.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:27 smithi195 ceph-mon[104262]: osdmap e314: 8 total, 8 up, 7 in 2024-09-13T14:46:28.262 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:27 smithi102 ceph-mon[110183]: osdmap e314: 8 total, 8 up, 7 in 2024-09-13T14:46:28.262 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:27 smithi102 ceph-mon[115269]: osdmap e314: 8 total, 8 up, 7 in 2024-09-13T14:46:28.284 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:28.319 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:28.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:28.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:28.421 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:28.440 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:46:28.463 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:28.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:28.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:28.564 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:28.600 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:28.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:28.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:28.701 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:28.734 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:28.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:28.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:28.799 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 15 16 38434 38418 160.061 120.875 0.00748522 0.00624251 2024-09-13T14:46:28.835 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:28.869 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:28.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:28.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:28.971 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:29.012 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:29.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:29.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:29.114 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:29.135 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:46:29.136 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:46:29.139 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:46:29.140 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:46:29.143 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:29.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:29.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:29.169 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:46:29.169 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:46:29.206 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:46:29.206 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:46:29.207 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:28 smithi102 ceph-mon[110183]: pgmap v1098: 89 pgs: 1 peering, 88 active+clean; 2.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 152 MiB/s wr, 2.42k op/s 2024-09-13T14:46:29.207 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:28 smithi102 ceph-mon[115269]: pgmap v1098: 89 pgs: 1 peering, 88 active+clean; 2.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 152 MiB/s wr, 2.42k op/s 2024-09-13T14:46:29.240 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:46:29.240 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:46:29.244 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:29.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:28 smithi195 ceph-mon[104262]: pgmap v1098: 89 pgs: 1 peering, 88 active+clean; 2.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 152 MiB/s wr, 2.42k op/s 2024-09-13T14:46:29.271 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:46:29.272 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:46:29.279 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:29.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:29.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:29.310 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:46:29.310 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:46:29.342 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:46:29.343 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:46:29.372 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:46:29.373 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:46:29.381 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:29.427 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:29.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:29.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:29.529 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:29.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:29.568 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:29.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:29.669 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:29.702 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:29.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:29.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:29.799 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 16 16 40395 40379 157.716 122.562 0.00534211 0.00633584 2024-09-13T14:46:29.804 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:29.840 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:29.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:29.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:29.941 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:29.980 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:29.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:29.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:30.082 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:30.112 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:30.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:30.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:30.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:30 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2990377742' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:46:30.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:30 smithi102 ceph-mon[110183]: from='client.55783 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:46:30.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:30 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2990377742' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:46:30.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:30 smithi102 ceph-mon[115269]: from='client.55783 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:46:30.214 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:30.241 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:30 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2990377742' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:46:30.241 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:30 smithi195 ceph-mon[104262]: from='client.55783 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:46:30.248 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:30.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:30.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:30.350 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:30.388 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:30.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:30.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:30.490 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:30.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:30.530 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:30.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:30.631 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:30.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:30.668 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:30.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:30.684 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:46:30.684 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-13T14:46:30.684 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:30.684 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.46157.0:42088 8.e e314/306 8:7936c3f2:::benchmark_data_smithi102_313275_object42087:head v 314'2610, mlcod=314'2608)", 2024-09-13T14:46:30.684 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:46:30.677517+0000", 2024-09-13T14:46:30.684 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.0041670789999999997, 2024-09-13T14:46:30.685 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0041953820000000001, 2024-09-13T14:46:30.685 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:46:30.685 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T14:46:30.685 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:46:30.685 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:30.685 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:46:30.685 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:30.677517+0000", 2024-09-13T14:46:30.685 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:30.685 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:30.685 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:30.685 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:46:30.685 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:30.677517+0000", 2024-09-13T14:46:30.685 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:30.686 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:30.686 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:30.686 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:46:30.686 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:30.677516+0000", 2024-09-13T14:46:30.686 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:46:30.686 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:30.686 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:30.686 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:46:30.686 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:30.677549+0000", 2024-09-13T14:46:30.686 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.3076e-05 2024-09-13T14:46:30.686 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:30.686 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:30.687 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:46:30.687 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:30.677565+0000", 2024-09-13T14:46:30.687 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6243999999999999e-05 2024-09-13T14:46:30.687 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:30.687 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:30.687 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:46:30.687 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:30.677572+0000", 2024-09-13T14:46:30.687 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4289999999999998e-06 2024-09-13T14:46:30.687 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:30.687 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:30.687 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:30.687 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:30.678333+0000", 2024-09-13T14:46:30.688 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00076137799999999995 2024-09-13T14:46:30.688 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:30.688 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:30.688 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:46:30.688 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:30.678351+0000", 2024-09-13T14:46:30.688 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8325000000000002e-05 2024-09-13T14:46:30.688 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:30.688 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:46:30.688 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:30.688 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:30.688 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:30.688 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.46157.0:42091 8.e e314/306 8:715b3827:::benchmark_data_smithi102_313275_object42090:head v 314'2611, mlcod=314'2608)", 2024-09-13T14:46:30.688 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:46:30.679555+0000", 2024-09-13T14:46:30.689 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.0021290990000000002, 2024-09-13T14:46:30.689 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0022182700000000001, 2024-09-13T14:46:30.689 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:46:30.689 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T14:46:30.689 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:46:30.689 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:30.689 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:46:30.689 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:30.679555+0000", 2024-09-13T14:46:30.689 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:30.689 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:30.689 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:30.689 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:46:30.690 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:30.679555+0000", 2024-09-13T14:46:30.690 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:30.690 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:30.690 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:30.690 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:46:30.690 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:30.679554+0000", 2024-09-13T14:46:30.690 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:46:30.690 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:30.690 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:30.690 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:46:30.690 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:30.679583+0000", 2024-09-13T14:46:30.690 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9289999999999999e-05 2024-09-13T14:46:30.691 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:30.691 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:30.691 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:46:30.691 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:30.679596+0000", 2024-09-13T14:46:30.691 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2132e-05 2024-09-13T14:46:30.691 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:30.691 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:30.691 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:46:30.691 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:30.679602+0000", 2024-09-13T14:46:30.691 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7460000000000003e-06 2024-09-13T14:46:30.691 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:30.691 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:30.691 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:30.691 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:30.680004+0000", 2024-09-13T14:46:30.692 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00040131800000000001 2024-09-13T14:46:30.692 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:30.692 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:30.692 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:46:30.692 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:30.680020+0000", 2024-09-13T14:46:30.692 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5977999999999998e-05 2024-09-13T14:46:30.692 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:30.692 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:46:30.692 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:30.692 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:30.692 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:30.692 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.46157.0:42092 8.4 e314/306 8:2d851693:::benchmark_data_smithi102_313275_object42091:head v 314'2608, mlcod=314'2606)", 2024-09-13T14:46:30.693 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:46:30.679321+0000", 2024-09-13T14:46:30.693 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.0023634049999999998, 2024-09-13T14:46:30.693 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0025053649999999998, 2024-09-13T14:46:30.693 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:46:30.693 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T14:46:30.693 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:46:30.693 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:30.693 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:46:30.693 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:30.679321+0000", 2024-09-13T14:46:30.693 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:30.693 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:30.693 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:30.693 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:46:30.694 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:30.679321+0000", 2024-09-13T14:46:30.694 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:30.694 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:30.694 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:30.694 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:46:30.694 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:30.679319+0000", 2024-09-13T14:46:30.694 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:46:30.694 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:30.694 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:30.694 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:46:30.694 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:30.679673+0000", 2024-09-13T14:46:30.694 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00035359399999999999 2024-09-13T14:46:30.694 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:30.694 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:30.695 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:46:30.695 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:30.679681+0000", 2024-09-13T14:46:30.695 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9810000000000003e-06 2024-09-13T14:46:30.695 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:30.695 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:30.695 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:46:30.695 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:30.679686+0000", 2024-09-13T14:46:30.695 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9309999999999999e-06 2024-09-13T14:46:30.695 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:30.695 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:30.695 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:30.695 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:30.679745+0000", 2024-09-13T14:46:30.695 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8919000000000003e-05 2024-09-13T14:46:30.695 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:30.696 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:30.696 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:46:30.696 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:30.679764+0000", 2024-09-13T14:46:30.696 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8703999999999999e-05 2024-09-13T14:46:30.696 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:30.696 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:46:30.696 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:30.696 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:30.696 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:30.696 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.46157.0:42095 8.b e314/306 8:d6a8fd63:::benchmark_data_smithi102_313275_object42094:head v 314'2596, mlcod=314'2594)", 2024-09-13T14:46:30.696 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:46:30.681201+0000", 2024-09-13T14:46:30.696 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.00048272000000000001, 2024-09-13T14:46:30.696 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00067640099999999998, 2024-09-13T14:46:30.697 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:46:30.697 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T14:46:30.697 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:46:30.697 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:30.697 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:46:30.697 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:30.681201+0000", 2024-09-13T14:46:30.697 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:30.697 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:30.697 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:30.697 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:46:30.697 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:30.681201+0000", 2024-09-13T14:46:30.697 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:30.698 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:30.698 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:30.698 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:46:30.698 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:30.681200+0000", 2024-09-13T14:46:30.698 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:46:30.698 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:30.698 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:30.698 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:46:30.698 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:30.681236+0000", 2024-09-13T14:46:30.698 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5408999999999999e-05 2024-09-13T14:46:30.698 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:30.698 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:30.698 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:46:30.699 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:30.681246+0000", 2024-09-13T14:46:30.699 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0370999999999999e-05 2024-09-13T14:46:30.699 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:30.699 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:30.699 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:46:30.699 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:30.681253+0000", 2024-09-13T14:46:30.699 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6239999999999996e-06 2024-09-13T14:46:30.699 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:30.699 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:30.699 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:30.699 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:30.681367+0000", 2024-09-13T14:46:30.699 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011414299999999999 2024-09-13T14:46:30.699 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:30.699 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:30.700 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:46:30.700 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:30.681387+0000", 2024-09-13T14:46:30.700 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9820999999999999e-05 2024-09-13T14:46:30.700 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:30.700 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:46:30.700 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:30.700 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:30.700 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:30.700 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.46157.0:42097 8.5 8:a9384055:::benchmark_data_smithi102_313275_object42096:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-09-13T14:46:30.700 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:46:30.681277+0000", 2024-09-13T14:46:30.700 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.00040733799999999998, 2024-09-13T14:46:30.700 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00067027000000000005, 2024-09-13T14:46:30.700 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:46:30.701 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:46:30.701 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:46:30.701 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.46157", 2024-09-13T14:46:30.701 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:46:30.701 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 42097 2024-09-13T14:46:30.701 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:30.701 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:46:30.701 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:30.701 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:46:30.701 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:30.681277+0000", 2024-09-13T14:46:30.701 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:30.701 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:30.701 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:30.702 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:46:30.702 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:30.681277+0000", 2024-09-13T14:46:30.702 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:30.702 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:30.702 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:30.702 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:46:30.702 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:30.681275+0000", 2024-09-13T14:46:30.702 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:46:30.702 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:30.702 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:30.702 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:46:30.702 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:30.681309+0000", 2024-09-13T14:46:30.702 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4332000000000003e-05 2024-09-13T14:46:30.703 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:30.703 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:30.703 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:46:30.703 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:30.681320+0000", 2024-09-13T14:46:30.703 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0348e-05 2024-09-13T14:46:30.703 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:30.703 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:30.703 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:46:30.703 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:30.681327+0000", 2024-09-13T14:46:30.703 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8719999999999996e-06 2024-09-13T14:46:30.703 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:30.703 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:30.703 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:30.703 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:30.681384+0000", 2024-09-13T14:46:30.704 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7158999999999999e-05 2024-09-13T14:46:30.704 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:30.704 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:30.704 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:46:30.704 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:30.681547+0000", 2024-09-13T14:46:30.704 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016291999999999999 2024-09-13T14:46:30.704 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:30.704 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:30.704 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:46:30.704 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:30.681665+0000", 2024-09-13T14:46:30.704 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011788100000000001 2024-09-13T14:46:30.704 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:30.704 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:46:30.704 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:30.705 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:30.705 INFO:teuthology.orchestra.run.smithi195.stdout: ], 2024-09-13T14:46:30.705 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 5 2024-09-13T14:46:30.705 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:46:30.769 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:30.801 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 17 16 42306 42290 155.448 119.438 0.008246 0.00642781 2024-09-13T14:46:30.814 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:30.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:30.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:30.916 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:30.949 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:30.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:30.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:31.050 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:31.085 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:31.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:31.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:31.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:31 smithi102 ceph-mon[110183]: pgmap v1099: 89 pgs: 1 peering, 88 active+clean; 2.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 172 MiB/s wr, 2.75k op/s 2024-09-13T14:46:31.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:31 smithi102 ceph-mon[115269]: pgmap v1099: 89 pgs: 1 peering, 88 active+clean; 2.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 172 MiB/s wr, 2.75k op/s 2024-09-13T14:46:31.187 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:31.219 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:31.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:31.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:31.321 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:31.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:31.361 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:31.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:31.462 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:31.491 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:31.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:31.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:31.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:31 smithi195 ceph-mon[104262]: pgmap v1099: 89 pgs: 1 peering, 88 active+clean; 2.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 172 MiB/s wr, 2.75k op/s 2024-09-13T14:46:31.593 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:31.628 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:31.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:31.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:31.730 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:31.766 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:31.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:31.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:31.801 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 18 16 44229 44213 153.488 120.188 0.00633044 0.00650905 2024-09-13T14:46:31.825 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":1098,"stamp":"2024-09-13T14:46:27.572748+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2115639269,"num_objects":32237,"num_object_clones":0,"num_object_copies":96711,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32237,"num_whiteouts":0,"num_read":6376,"num_read_kb":13833,"num_write":33332,"num_write_kb":2076417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":5967581,"num_keys_recovered":20,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":33391,"ondisk_log_size":33391,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":272,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":11202076,"kb_used_data":7535060,"kb_used_omap":34,"kb_used_meta":3666909,"kb_avail":645013988,"statfs":{"total":671965249536,"available":660494323712,"internally_reserved":0,"allocated":7715901440,"data_stored":7704433014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35154,"internal_metadata":3754915502},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1271922688,"num_objects":19408,"num_object_clones":0,"num_object_copies":58224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19408,"num_write_kb":1242112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002009"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":526,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-13T14:46:18.869919+0000","last_change":"2024-09-13T14:46:01.770112+0000","last_active":"2024-09-13T14:46:18.869919+0000","last_peered":"2024-09-13T14:46:18.869919+0000","last_clean":"2024-09-13T14:46:18.869919+0000","last_became_active":"2024-09-13T14:46:01.769495+0000","last_became_peered":"2024-09-13T14:46:01.769495+0000","last_unstale":"2024-09-13T14:46:18.869919+0000","last_undegraded":"2024-09-13T14:46:18.869919+0000","last_fullsized":"2024-09-13T14:46:18.869919+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:51:27.686377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":332,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.971173+0000","last_change":"2024-09-13T14:46:01.772811+0000","last_active":"2024-09-13T14:46:25.971173+0000","last_peered":"2024-09-13T14:46:25.971173+0000","last_clean":"2024-09-13T14:46:25.971173+0000","last_became_active":"2024-09-13T14:46:01.764775+0000","last_became_peered":"2024-09-13T14:46:01.764775+0000","last_unstale":"2024-09-13T14:46:25.971173+0000","last_undegraded":"2024-09-13T14:46:25.971173+0000","last_fullsized":"2024-09-13T14:46:25.971173+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:42:20.738421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":114,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.969749+0000","last_change":"2024-09-13T14:36:13.862181+0000","last_active":"2024-09-13T14:46:25.969749+0000","last_peered":"2024-09-13T14:46:25.969749+0000","last_clean":"2024-09-13T14:46:25.969749+0000","last_became_active":"2024-09-13T14:36:13.861873+0000","last_became_peered":"2024-09-13T14:36:13.861873+0000","last_unstale":"2024-09-13T14:46:25.969749+0000","last_undegraded":"2024-09-13T14:46:25.969749+0000","last_fullsized":"2024-09-13T14:46:25.969749+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:14.085700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":160,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:46:26.969883+0000","last_change":"2024-09-13T14:38:53.896672+0000","last_active":"2024-09-13T14:46:26.969883+0000","last_peered":"2024-09-13T14:46:26.969883+0000","last_clean":"2024-09-13T14:46:26.969883+0000","last_became_active":"2024-09-13T14:38:53.896346+0000","last_became_peered":"2024-09-13T14:38:53.896346+0000","last_unstale":"2024-09-13T14:46:26.969883+0000","last_undegraded":"2024-09-13T14:46:26.969883+0000","last_fullsized":"2024-09-13T14:46:26.969883+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:57.789686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":160,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:46:26.971336+0000","last_change":"2024-09-13T14:38:53.893021+0000","last_active":"2024-09-13T14:46:26.971336+0000","last_peered":"2024-09-13T14:46:26.971336+0000","last_clean":"2024-09-13T14:46:26.971336+0000","last_became_active":"2024-09-13T14:38:53.892668+0000","last_became_peered":"2024-09-13T14:38:53.892668+0000","last_unstale":"2024-09-13T14:46:26.971336+0000","last_undegraded":"2024-09-13T14:46:26.971336+0000","last_fullsized":"2024-09-13T14:46:26.971336+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:57.180738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":286,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:46:26.966416+0000","last_change":"2024-09-13T14:37:48.134642+0000","last_active":"2024-09-13T14:46:26.966416+0000","last_peered":"2024-09-13T14:46:26.966416+0000","last_clean":"2024-09-13T14:46:26.966416+0000","last_became_active":"2024-09-13T14:37:48.134402+0000","last_became_peered":"2024-09-13T14:37:48.134402+0000","last_unstale":"2024-09-13T14:46:26.966416+0000","last_undegraded":"2024-09-13T14:46:26.966416+0000","last_fullsized":"2024-09-13T14:46:26.966416+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:51.313663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":544,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-13T14:46:18.870044+0000","last_change":"2024-09-13T14:46:01.770090+0000","last_active":"2024-09-13T14:46:18.870044+0000","last_peered":"2024-09-13T14:46:18.870044+0000","last_clean":"2024-09-13T14:46:18.870044+0000","last_became_active":"2024-09-13T14:46:01.769671+0000","last_became_peered":"2024-09-13T14:46:01.769671+0000","last_unstale":"2024-09-13T14:46:18.870044+0000","last_undegraded":"2024-09-13T14:46:18.870044+0000","last_fullsized":"2024-09-13T14:46:18.870044+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:46.340346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":163,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:46:26.969502+0000","last_change":"2024-09-13T14:38:03.554500+0000","last_active":"2024-09-13T14:46:26.969502+0000","last_peered":"2024-09-13T14:46:26.969502+0000","last_clean":"2024-09-13T14:46:26.969502+0000","last_became_active":"2024-09-13T14:38:03.532910+0000","last_became_peered":"2024-09-13T14:38:03.532910+0000","last_unstale":"2024-09-13T14:46:26.969502+0000","last_undegraded":"2024-09-13T14:46:26.969502+0000","last_fullsized":"2024-09-13T14:46:26.969502+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:27:50.843498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"314'2185","reported_seq":2212,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.376764+0000","last_change":"2024-09-13T14:46:01.762751+0000","last_active":"2024-09-13T14:46:27.376764+0000","last_peered":"2024-09-13T14:46:27.376764+0000","last_clean":"2024-09-13T14:46:27.376764+0000","last_became_active":"2024-09-13T14:46:01.761929+0000","last_became_peered":"2024-09-13T14:46:01.761929+0000","last_unstale":"2024-09-13T14:46:27.376764+0000","last_undegraded":"2024-09-13T14:46:27.376764+0000","last_fullsized":"2024-09-13T14:46:27.376764+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":304,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":2185,"log_dups_size":0,"ondisk_log_size":2185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:20:57.325962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143196160,"num_objects":2185,"num_object_clones":0,"num_object_copies":6555,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2185,"num_write_kb":139840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":78,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.971311+0000","last_change":"2024-09-13T14:38:53.940238+0000","last_active":"2024-09-13T14:46:25.971311+0000","last_peered":"2024-09-13T14:46:25.971311+0000","last_clean":"2024-09-13T14:46:25.971311+0000","last_became_active":"2024-09-13T14:38:53.892825+0000","last_became_peered":"2024-09-13T14:38:53.892825+0000","last_unstale":"2024-09-13T14:46:25.971311+0000","last_undegraded":"2024-09-13T14:46:25.971311+0000","last_fullsized":"2024-09-13T14:46:25.971311+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:46.666581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":134,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.969298+0000","last_change":"2024-09-13T14:38:53.898907+0000","last_active":"2024-09-13T14:46:25.969298+0000","last_peered":"2024-09-13T14:46:25.969298+0000","last_clean":"2024-09-13T14:46:25.969298+0000","last_became_active":"2024-09-13T14:38:53.898572+0000","last_became_peered":"2024-09-13T14:38:53.898572+0000","last_unstale":"2024-09-13T14:46:25.969298+0000","last_undegraded":"2024-09-13T14:46:25.969298+0000","last_fullsized":"2024-09-13T14:46:25.969298+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:29.222150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.5","version":"312'1736","reported_seq":1753,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-13T14:46:22.910199+0000","last_change":"2024-09-13T14:46:00.028852+0000","last_active":"2024-09-13T14:46:22.910199+0000","last_peered":"2024-09-13T14:46:22.910199+0000","last_clean":"2024-09-13T14:46:22.910199+0000","last_became_active":"2024-09-13T14:46:00.028414+0000","last_became_peered":"2024-09-13T14:46:00.028414+0000","last_unstale":"2024-09-13T14:46:22.910199+0000","last_undegraded":"2024-09-13T14:46:22.910199+0000","last_fullsized":"2024-09-13T14:46:22.910199+0000","mapping_epoch":304,"log_start":"0'0","ondisk_log_start":"0'0","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":1736,"log_dups_size":0,"ondisk_log_size":1736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:03:28.278631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113770496,"num_objects":1736,"num_object_clones":0,"num_object_copies":5208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1736,"num_write_kb":111104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":78,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.971402+0000","last_change":"2024-09-13T14:38:53.904885+0000","last_active":"2024-09-13T14:46:25.971402+0000","last_peered":"2024-09-13T14:46:25.971402+0000","last_clean":"2024-09-13T14:46:25.971402+0000","last_became_active":"2024-09-13T14:38:53.891726+0000","last_became_peered":"2024-09-13T14:38:53.891726+0000","last_unstale":"2024-09-13T14:46:25.971402+0000","last_undegraded":"2024-09-13T14:46:25.971402+0000","last_fullsized":"2024-09-13T14:46:25.971402+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:57.079205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":322,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-13T14:46:12.917864+0000","last_change":"2024-09-13T14:37:48.160504+0000","last_active":"2024-09-13T14:46:12.917864+0000","last_peered":"2024-09-13T14:46:12.917864+0000","last_clean":"2024-09-13T14:46:12.917864+0000","last_became_active":"2024-09-13T14:37:48.160181+0000","last_became_peered":"2024-09-13T14:37:48.160181+0000","last_unstale":"2024-09-13T14:46:12.917864+0000","last_undegraded":"2024-09-13T14:46:12.917864+0000","last_fullsized":"2024-09-13T14:46:12.917864+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:37.087008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"314'2290","reported_seq":2321,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.289656+0000","last_change":"2024-09-13T14:46:00.027675+0000","last_active":"2024-09-13T14:46:27.289656+0000","last_peered":"2024-09-13T14:46:27.289656+0000","last_clean":"2024-09-13T14:46:27.289656+0000","last_became_active":"2024-09-13T14:46:00.027364+0000","last_became_peered":"2024-09-13T14:46:00.027364+0000","last_unstale":"2024-09-13T14:46:27.289656+0000","last_undegraded":"2024-09-13T14:46:27.289656+0000","last_fullsized":"2024-09-13T14:46:27.289656+0000","mapping_epoch":304,"log_start":"0'0","ondisk_log_start":"0'0","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":2290,"log_dups_size":0,"ondisk_log_size":2290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:42:37.193007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150077440,"num_objects":2290,"num_object_clones":0,"num_object_copies":6870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2290,"num_write_kb":146560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":315,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.971278+0000","last_change":"2024-09-13T14:37:49.611774+0000","last_active":"2024-09-13T14:46:25.971278+0000","last_peered":"2024-09-13T14:46:25.971278+0000","last_clean":"2024-09-13T14:46:25.971278+0000","last_became_active":"2024-09-13T14:37:49.611088+0000","last_became_peered":"2024-09-13T14:37:49.611088+0000","last_unstale":"2024-09-13T14:46:25.971278+0000","last_undegraded":"2024-09-13T14:46:25.971278+0000","last_fullsized":"2024-09-13T14:46:25.971278+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:08.933213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":74,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.970928+0000","last_change":"2024-09-13T14:38:53.899211+0000","last_active":"2024-09-13T14:46:25.970928+0000","last_peered":"2024-09-13T14:46:25.970928+0000","last_clean":"2024-09-13T14:46:25.970928+0000","last_became_active":"2024-09-13T14:38:53.898927+0000","last_became_peered":"2024-09-13T14:38:53.898927+0000","last_unstale":"2024-09-13T14:46:25.970928+0000","last_undegraded":"2024-09-13T14:46:25.970928+0000","last_fullsized":"2024-09-13T14:46:25.970928+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:00:28.857611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.7","version":"312'1730","reported_seq":1763,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-13T14:46:22.571601+0000","last_change":"2024-09-13T14:46:01.766640+0000","last_active":"2024-09-13T14:46:22.571601+0000","last_peered":"2024-09-13T14:46:22.571601+0000","last_clean":"2024-09-13T14:46:22.571601+0000","last_became_active":"2024-09-13T14:46:01.766349+0000","last_became_peered":"2024-09-13T14:46:01.766349+0000","last_unstale":"2024-09-13T14:46:22.571601+0000","last_undegraded":"2024-09-13T14:46:22.571601+0000","last_fullsized":"2024-09-13T14:46:22.571601+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":304,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":1730,"log_dups_size":0,"ondisk_log_size":1730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:16:28.600947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113377280,"num_objects":1730,"num_object_clones":0,"num_object_copies":5190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1730,"num_write_kb":110720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":125,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.969509+0000","last_change":"2024-09-13T14:37:49.168003+0000","last_active":"2024-09-13T14:46:25.969509+0000","last_peered":"2024-09-13T14:46:25.969509+0000","last_clean":"2024-09-13T14:46:25.969509+0000","last_became_active":"2024-09-13T14:37:49.167638+0000","last_became_peered":"2024-09-13T14:37:49.167638+0000","last_unstale":"2024-09-13T14:46:25.969509+0000","last_undegraded":"2024-09-13T14:46:25.969509+0000","last_fullsized":"2024-09-13T14:46:25.969509+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:29.051981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":153,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:46:26.970127+0000","last_change":"2024-09-13T14:38:03.528381+0000","last_active":"2024-09-13T14:46:26.970127+0000","last_peered":"2024-09-13T14:46:26.970127+0000","last_clean":"2024-09-13T14:46:26.970127+0000","last_became_active":"2024-09-13T14:38:03.527282+0000","last_became_peered":"2024-09-13T14:38:03.527282+0000","last_unstale":"2024-09-13T14:46:26.970127+0000","last_undegraded":"2024-09-13T14:46:26.970127+0000","last_fullsized":"2024-09-13T14:46:26.970127+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:24.435067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.0","version":"312'1662","reported_seq":1689,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-13T14:46:22.567128+0000","last_change":"2024-09-13T14:45:59.744591+0000","last_active":"2024-09-13T14:46:22.567128+0000","last_peered":"2024-09-13T14:46:22.567128+0000","last_clean":"2024-09-13T14:46:22.567128+0000","last_became_active":"2024-09-13T14:45:59.744297+0000","last_became_peered":"2024-09-13T14:45:59.744297+0000","last_unstale":"2024-09-13T14:46:22.567128+0000","last_undegraded":"2024-09-13T14:46:22.567128+0000","last_fullsized":"2024-09-13T14:46:22.567128+0000","mapping_epoch":304,"log_start":"0'0","ondisk_log_start":"0'0","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":1662,"log_dups_size":0,"ondisk_log_size":1662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:05:27.678824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108920832,"num_objects":1662,"num_object_clones":0,"num_object_copies":4986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1662,"num_write_kb":106368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":507,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-13T14:46:18.870038+0000","last_change":"2024-09-13T14:38:03.549595+0000","last_active":"2024-09-13T14:46:18.870038+0000","last_peered":"2024-09-13T14:46:18.870038+0000","last_clean":"2024-09-13T14:46:18.870038+0000","last_became_active":"2024-09-13T14:38:03.533331+0000","last_became_peered":"2024-09-13T14:38:03.533331+0000","last_unstale":"2024-09-13T14:46:18.870038+0000","last_undegraded":"2024-09-13T14:46:18.870038+0000","last_fullsized":"2024-09-13T14:46:18.870038+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:59:02.470343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":297,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-13T14:46:11.797508+0000","last_change":"2024-09-13T14:37:48.127600+0000","last_active":"2024-09-13T14:46:11.797508+0000","last_peered":"2024-09-13T14:46:11.797508+0000","last_clean":"2024-09-13T14:46:11.797508+0000","last_became_active":"2024-09-13T14:37:48.127349+0000","last_became_peered":"2024-09-13T14:37:48.127349+0000","last_unstale":"2024-09-13T14:46:11.797508+0000","last_undegraded":"2024-09-13T14:46:11.797508+0000","last_fullsized":"2024-09-13T14:46:11.797508+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:57.268374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"314'2174","reported_seq":2211,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.292239+0000","last_change":"2024-09-13T14:46:01.765500+0000","last_active":"2024-09-13T14:46:27.292239+0000","last_peered":"2024-09-13T14:46:27.292239+0000","last_clean":"2024-09-13T14:46:27.292239+0000","last_became_active":"2024-09-13T14:46:01.765187+0000","last_became_peered":"2024-09-13T14:46:01.765187+0000","last_unstale":"2024-09-13T14:46:27.292239+0000","last_undegraded":"2024-09-13T14:46:27.292239+0000","last_fullsized":"2024-09-13T14:46:27.292239+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":304,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":2174,"log_dups_size":0,"ondisk_log_size":2174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:15.232398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142475264,"num_objects":2174,"num_object_clones":0,"num_object_copies":6522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2174,"num_write_kb":139136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":230,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-13T14:46:11.797394+0000","last_change":"2024-09-13T14:37:48.127604+0000","last_active":"2024-09-13T14:46:11.797394+0000","last_peered":"2024-09-13T14:46:11.797394+0000","last_clean":"2024-09-13T14:46:11.797394+0000","last_became_active":"2024-09-13T14:37:48.127300+0000","last_became_peered":"2024-09-13T14:37:48.127300+0000","last_unstale":"2024-09-13T14:46:11.797394+0000","last_undegraded":"2024-09-13T14:46:11.797394+0000","last_fullsized":"2024-09-13T14:46:11.797394+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:51.999506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":359,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-13T14:46:11.797668+0000","last_change":"2024-09-13T14:46:01.761924+0000","last_active":"2024-09-13T14:46:11.797668+0000","last_peered":"2024-09-13T14:46:11.797668+0000","last_clean":"2024-09-13T14:46:11.797668+0000","last_became_active":"2024-09-13T14:46:01.761576+0000","last_became_peered":"2024-09-13T14:46:01.761576+0000","last_unstale":"2024-09-13T14:46:11.797668+0000","last_undegraded":"2024-09-13T14:46:11.797668+0000","last_fullsized":"2024-09-13T14:46:11.797668+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:02:34.186759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.2","version":"312'1663","reported_seq":1679,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-13T14:46:22.917483+0000","last_change":"2024-09-13T14:45:59.744581+0000","last_active":"2024-09-13T14:46:22.917483+0000","last_peered":"2024-09-13T14:46:22.917483+0000","last_clean":"2024-09-13T14:46:22.917483+0000","last_became_active":"2024-09-13T14:45:59.744231+0000","last_became_peered":"2024-09-13T14:45:59.744231+0000","last_unstale":"2024-09-13T14:46:22.917483+0000","last_undegraded":"2024-09-13T14:46:22.917483+0000","last_fullsized":"2024-09-13T14:46:22.917483+0000","mapping_epoch":304,"log_start":"0'0","ondisk_log_start":"0'0","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":1663,"log_dups_size":0,"ondisk_log_size":1663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:06:14.502840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108986368,"num_objects":1663,"num_object_clones":0,"num_object_copies":4989,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1663,"num_write_kb":106432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":320,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-13T14:46:12.917885+0000","last_change":"2024-09-13T14:46:01.762127+0000","last_active":"2024-09-13T14:46:12.917885+0000","last_peered":"2024-09-13T14:46:12.917885+0000","last_clean":"2024-09-13T14:46:12.917885+0000","last_became_active":"2024-09-13T14:46:01.761795+0000","last_became_peered":"2024-09-13T14:46:01.761795+0000","last_unstale":"2024-09-13T14:46:12.917885+0000","last_undegraded":"2024-09-13T14:46:12.917885+0000","last_fullsized":"2024-09-13T14:46:12.917885+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:41:15.626775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":306,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-13T14:46:11.797807+0000","last_change":"2024-09-13T14:37:48.147360+0000","last_active":"2024-09-13T14:46:11.797807+0000","last_peered":"2024-09-13T14:46:11.797807+0000","last_clean":"2024-09-13T14:46:11.797807+0000","last_became_active":"2024-09-13T14:37:48.146885+0000","last_became_peered":"2024-09-13T14:37:48.146885+0000","last_unstale":"2024-09-13T14:46:11.797807+0000","last_undegraded":"2024-09-13T14:46:11.797807+0000","last_fullsized":"2024-09-13T14:46:11.797807+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:27.841810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"314'2163","reported_seq":2191,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.284469+0000","last_change":"2024-09-13T14:46:01.764572+0000","last_active":"2024-09-13T14:46:27.284469+0000","last_peered":"2024-09-13T14:46:27.284469+0000","last_clean":"2024-09-13T14:46:27.284469+0000","last_became_active":"2024-09-13T14:46:01.764079+0000","last_became_peered":"2024-09-13T14:46:01.764079+0000","last_unstale":"2024-09-13T14:46:27.284469+0000","last_undegraded":"2024-09-13T14:46:27.284469+0000","last_fullsized":"2024-09-13T14:46:27.284469+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":304,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":2163,"log_dups_size":0,"ondisk_log_size":2163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:03:11.187551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":141754368,"num_objects":2163,"num_object_clones":0,"num_object_copies":6489,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2163,"num_write_kb":138432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":525,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-13T14:46:18.870895+0000","last_change":"2024-09-13T14:38:53.900041+0000","last_active":"2024-09-13T14:46:18.870895+0000","last_peered":"2024-09-13T14:46:18.870895+0000","last_clean":"2024-09-13T14:46:18.870895+0000","last_became_active":"2024-09-13T14:38:53.899203+0000","last_became_peered":"2024-09-13T14:38:53.899203+0000","last_unstale":"2024-09-13T14:46:18.870895+0000","last_undegraded":"2024-09-13T14:46:18.870895+0000","last_fullsized":"2024-09-13T14:46:18.870895+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:28:07.883094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":457,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-13T14:46:18.870327+0000","last_change":"2024-09-13T14:26:15.823198+0000","last_active":"2024-09-13T14:46:18.870327+0000","last_peered":"2024-09-13T14:46:18.870327+0000","last_clean":"2024-09-13T14:46:18.870327+0000","last_became_active":"2024-09-13T14:26:15.822717+0000","last_became_peered":"2024-09-13T14:26:15.822717+0000","last_unstale":"2024-09-13T14:46:18.870327+0000","last_undegraded":"2024-09-13T14:46:18.870327+0000","last_fullsized":"2024-09-13T14:46:18.870327+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:22.471622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"314'2289","reported_seq":2308,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.300062+0000","last_change":"2024-09-13T14:46:00.027751+0000","last_active":"2024-09-13T14:46:27.300062+0000","last_peered":"2024-09-13T14:46:27.300062+0000","last_clean":"2024-09-13T14:46:27.300062+0000","last_became_active":"2024-09-13T14:46:00.027429+0000","last_became_peered":"2024-09-13T14:46:00.027429+0000","last_unstale":"2024-09-13T14:46:27.300062+0000","last_undegraded":"2024-09-13T14:46:27.300062+0000","last_fullsized":"2024-09-13T14:46:27.300062+0000","mapping_epoch":304,"log_start":"0'0","ondisk_log_start":"0'0","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":2289,"log_dups_size":0,"ondisk_log_size":2289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:17:52.381240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150011904,"num_objects":2289,"num_object_clones":0,"num_object_copies":6867,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2289,"num_write_kb":146496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":449,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.971141+0000","last_change":"2024-09-13T14:41:53.196835+0000","last_active":"2024-09-13T14:46:25.971141+0000","last_peered":"2024-09-13T14:46:25.971141+0000","last_clean":"2024-09-13T14:46:25.971141+0000","last_became_active":"2024-09-13T14:41:53.196387+0000","last_became_peered":"2024-09-13T14:41:53.196387+0000","last_unstale":"2024-09-13T14:46:25.971141+0000","last_undegraded":"2024-09-13T14:46:25.971141+0000","last_fullsized":"2024-09-13T14:46:25.971141+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:07:41.927740+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:49.972303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":620,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.969455+0000","last_change":"2024-09-13T14:46:04.780718+0000","last_active":"2024-09-13T14:46:25.969455+0000","last_peered":"2024-09-13T14:46:25.969455+0000","last_clean":"2024-09-13T14:46:25.969455+0000","last_became_active":"2024-09-13T14:46:04.780120+0000","last_became_peered":"2024-09-13T14:46:04.780120+0000","last_unstale":"2024-09-13T14:46:25.969455+0000","last_undegraded":"2024-09-13T14:46:25.969455+0000","last_fullsized":"2024-09-13T14:46:25.969455+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:57:14.475888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":460,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-13T14:46:18.869840+0000","last_change":"2024-09-13T14:37:49.160719+0000","last_active":"2024-09-13T14:46:18.869840+0000","last_peered":"2024-09-13T14:46:18.869840+0000","last_clean":"2024-09-13T14:46:18.869840+0000","last_became_active":"2024-09-13T14:37:49.160401+0000","last_became_peered":"2024-09-13T14:37:49.160401+0000","last_unstale":"2024-09-13T14:46:18.869840+0000","last_undegraded":"2024-09-13T14:46:18.869840+0000","last_fullsized":"2024-09-13T14:46:18.869840+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:21.737945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":157,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-13T14:46:11.797640+0000","last_change":"2024-09-13T14:37:48.172940+0000","last_active":"2024-09-13T14:46:11.797640+0000","last_peered":"2024-09-13T14:46:11.797640+0000","last_clean":"2024-09-13T14:46:11.797640+0000","last_became_active":"2024-09-13T14:37:48.172060+0000","last_became_peered":"2024-09-13T14:37:48.172060+0000","last_unstale":"2024-09-13T14:46:11.797640+0000","last_undegraded":"2024-09-13T14:46:11.797640+0000","last_fullsized":"2024-09-13T14:46:11.797640+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:07.480645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.a","version":"313'2164","reported_seq":2189,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:46:26.359019+0000","last_change":"2024-09-13T14:46:01.767421+0000","last_active":"2024-09-13T14:46:26.359019+0000","last_peered":"2024-09-13T14:46:26.359019+0000","last_clean":"2024-09-13T14:46:26.359019+0000","last_became_active":"2024-09-13T14:46:01.767105+0000","last_became_peered":"2024-09-13T14:46:01.767105+0000","last_unstale":"2024-09-13T14:46:26.359019+0000","last_undegraded":"2024-09-13T14:46:26.359019+0000","last_fullsized":"2024-09-13T14:46:26.359019+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":304,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":2164,"log_dups_size":0,"ondisk_log_size":2164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:57:36.278404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":141819904,"num_objects":2164,"num_object_clones":0,"num_object_copies":6492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2164,"num_write_kb":138496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":533,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.971065+0000","last_change":"2024-09-13T14:26:15.835178+0000","last_active":"2024-09-13T14:46:25.971065+0000","last_peered":"2024-09-13T14:46:25.971065+0000","last_clean":"2024-09-13T14:46:25.971065+0000","last_became_active":"2024-09-13T14:26:15.820283+0000","last_became_peered":"2024-09-13T14:26:15.820283+0000","last_unstale":"2024-09-13T14:46:25.971065+0000","last_undegraded":"2024-09-13T14:46:25.971065+0000","last_fullsized":"2024-09-13T14:46:25.971065+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:30.340038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":154,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:46:26.970066+0000","last_change":"2024-09-13T14:38:03.525682+0000","last_active":"2024-09-13T14:46:26.970066+0000","last_peered":"2024-09-13T14:46:26.970066+0000","last_clean":"2024-09-13T14:46:26.970066+0000","last_became_active":"2024-09-13T14:38:03.525406+0000","last_became_peered":"2024-09-13T14:38:03.525406+0000","last_unstale":"2024-09-13T14:46:26.970066+0000","last_undegraded":"2024-09-13T14:46:26.970066+0000","last_fullsized":"2024-09-13T14:46:26.970066+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:41:28.970300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":167,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:46:26.972271+0000","last_change":"2024-09-13T14:46:01.769069+0000","last_active":"2024-09-13T14:46:26.972271+0000","last_peered":"2024-09-13T14:46:26.972271+0000","last_clean":"2024-09-13T14:46:26.972271+0000","last_became_active":"2024-09-13T14:46:01.768766+0000","last_became_peered":"2024-09-13T14:46:01.768766+0000","last_unstale":"2024-09-13T14:46:26.972271+0000","last_undegraded":"2024-09-13T14:46:26.972271+0000","last_fullsized":"2024-09-13T14:46:26.972271+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:22:47.154785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"312'1635","reported_seq":1668,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-13T14:46:22.555010+0000","last_change":"2024-09-13T14:46:01.766352+0000","last_active":"2024-09-13T14:46:22.555010+0000","last_peered":"2024-09-13T14:46:22.555010+0000","last_clean":"2024-09-13T14:46:22.555010+0000","last_became_active":"2024-09-13T14:46:01.766046+0000","last_became_peered":"2024-09-13T14:46:01.766046+0000","last_unstale":"2024-09-13T14:46:22.555010+0000","last_undegraded":"2024-09-13T14:46:22.555010+0000","last_fullsized":"2024-09-13T14:46:22.555010+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":304,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":1635,"log_dups_size":0,"ondisk_log_size":1635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:36:54.550417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107151360,"num_objects":1635,"num_object_clones":0,"num_object_copies":4905,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1635,"num_write_kb":104640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":446,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.972481+0000","last_change":"2024-09-13T14:46:04.782955+0000","last_active":"2024-09-13T14:46:25.972481+0000","last_peered":"2024-09-13T14:46:25.972481+0000","last_clean":"2024-09-13T14:46:25.972481+0000","last_became_active":"2024-09-13T14:46:04.782250+0000","last_became_peered":"2024-09-13T14:46:04.782250+0000","last_unstale":"2024-09-13T14:46:25.972481+0000","last_undegraded":"2024-09-13T14:46:25.972481+0000","last_fullsized":"2024-09-13T14:46:25.972481+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:24.596922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.971545+0000","last_change":"2024-09-13T14:46:01.767882+0000","last_active":"2024-09-13T14:46:25.971545+0000","last_peered":"2024-09-13T14:46:25.971545+0000","last_clean":"2024-09-13T14:46:25.971545+0000","last_became_active":"2024-09-13T14:46:01.767116+0000","last_became_peered":"2024-09-13T14:46:01.767116+0000","last_unstale":"2024-09-13T14:46:25.971545+0000","last_undegraded":"2024-09-13T14:46:25.971545+0000","last_fullsized":"2024-09-13T14:46:25.971545+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:24:48.755394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":141,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.969817+0000","last_change":"2024-09-13T14:38:53.901427+0000","last_active":"2024-09-13T14:46:25.969817+0000","last_peered":"2024-09-13T14:46:25.969817+0000","last_clean":"2024-09-13T14:46:25.969817+0000","last_became_active":"2024-09-13T14:38:53.901157+0000","last_became_peered":"2024-09-13T14:38:53.901157+0000","last_unstale":"2024-09-13T14:46:25.969817+0000","last_undegraded":"2024-09-13T14:46:25.969817+0000","last_fullsized":"2024-09-13T14:46:25.969817+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:28:21.713025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.8","version":"314'2234","reported_seq":2253,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.315115+0000","last_change":"2024-09-13T14:45:59.743468+0000","last_active":"2024-09-13T14:46:27.315115+0000","last_peered":"2024-09-13T14:46:27.315115+0000","last_clean":"2024-09-13T14:46:27.315115+0000","last_became_active":"2024-09-13T14:45:59.742168+0000","last_became_peered":"2024-09-13T14:45:59.742168+0000","last_unstale":"2024-09-13T14:46:27.315115+0000","last_undegraded":"2024-09-13T14:46:27.315115+0000","last_fullsized":"2024-09-13T14:46:27.315115+0000","mapping_epoch":304,"log_start":"0'0","ondisk_log_start":"0'0","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":2234,"log_dups_size":0,"ondisk_log_size":2234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:38:09.406586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146407424,"num_objects":2234,"num_object_clones":0,"num_object_copies":6702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2234,"num_write_kb":142976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":1023,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.969788+0000","last_change":"2024-09-13T14:37:49.147498+0000","last_active":"2024-09-13T14:46:25.969788+0000","last_peered":"2024-09-13T14:46:25.969788+0000","last_clean":"2024-09-13T14:46:25.969788+0000","last_became_active":"2024-09-13T14:37:49.147028+0000","last_became_peered":"2024-09-13T14:37:49.147028+0000","last_unstale":"2024-09-13T14:46:25.969788+0000","last_undegraded":"2024-09-13T14:46:25.969788+0000","last_fullsized":"2024-09-13T14:46:25.969788+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:11:29.050178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":319,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-13T14:46:11.797674+0000","last_change":"2024-09-13T14:38:03.528133+0000","last_active":"2024-09-13T14:46:11.797674+0000","last_peered":"2024-09-13T14:46:11.797674+0000","last_clean":"2024-09-13T14:46:11.797674+0000","last_became_active":"2024-09-13T14:38:03.527829+0000","last_became_peered":"2024-09-13T14:38:03.527829+0000","last_unstale":"2024-09-13T14:46:11.797674+0000","last_undegraded":"2024-09-13T14:46:11.797674+0000","last_fullsized":"2024-09-13T14:46:11.797674+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:48.254169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":24,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.973095+0000","last_change":"2024-09-13T14:46:01.768418+0000","last_active":"2024-09-13T14:46:25.973095+0000","last_peered":"2024-09-13T14:46:25.973095+0000","last_clean":"2024-09-13T14:46:25.973095+0000","last_became_active":"2024-09-13T14:46:01.767804+0000","last_became_peered":"2024-09-13T14:46:01.767804+0000","last_unstale":"2024-09-13T14:46:25.973095+0000","last_undegraded":"2024-09-13T14:46:25.973095+0000","last_fullsized":"2024-09-13T14:46:25.973095+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:05:10.365127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.e","version":"314'2223","reported_seq":2260,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.256283+0000","last_change":"2024-09-13T14:46:01.768936+0000","last_active":"2024-09-13T14:46:27.256283+0000","last_peered":"2024-09-13T14:46:27.256283+0000","last_clean":"2024-09-13T14:46:27.256283+0000","last_became_active":"2024-09-13T14:46:01.768354+0000","last_became_peered":"2024-09-13T14:46:01.768354+0000","last_unstale":"2024-09-13T14:46:27.256283+0000","last_undegraded":"2024-09-13T14:46:27.256283+0000","last_fullsized":"2024-09-13T14:46:27.256283+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":304,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":2223,"log_dups_size":0,"ondisk_log_size":2223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:01.762702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145686528,"num_objects":2223,"num_object_clones":0,"num_object_copies":6669,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2223,"num_write_kb":142272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":134,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.969668+0000","last_change":"2024-09-13T14:46:01.763906+0000","last_active":"2024-09-13T14:46:25.969668+0000","last_peered":"2024-09-13T14:46:25.969668+0000","last_clean":"2024-09-13T14:46:25.969668+0000","last_became_active":"2024-09-13T14:46:01.763542+0000","last_became_peered":"2024-09-13T14:46:01.763542+0000","last_unstale":"2024-09-13T14:46:25.969668+0000","last_undegraded":"2024-09-13T14:46:25.969668+0000","last_fullsized":"2024-09-13T14:46:25.969668+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:09:36.652767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":1248,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:46:26.969700+0000","last_change":"2024-09-13T14:46:04.780843+0000","last_active":"2024-09-13T14:46:26.969700+0000","last_peered":"2024-09-13T14:46:26.969700+0000","last_clean":"2024-09-13T14:46:26.969700+0000","last_became_active":"2024-09-13T14:46:04.780204+0000","last_became_peered":"2024-09-13T14:46:04.780204+0000","last_unstale":"2024-09-13T14:46:26.969700+0000","last_undegraded":"2024-09-13T14:46:26.969700+0000","last_fullsized":"2024-09-13T14:46:26.969700+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:20:41.074826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":132,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.969700+0000","last_change":"2024-09-13T14:38:03.534628+0000","last_active":"2024-09-13T14:46:25.969700+0000","last_peered":"2024-09-13T14:46:25.969700+0000","last_clean":"2024-09-13T14:46:25.969700+0000","last_became_active":"2024-09-13T14:38:03.534262+0000","last_became_peered":"2024-09-13T14:38:03.534262+0000","last_unstale":"2024-09-13T14:46:25.969700+0000","last_undegraded":"2024-09-13T14:46:25.969700+0000","last_fullsized":"2024-09-13T14:46:25.969700+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:31.168783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"312'1654","reported_seq":1687,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-13T14:46:22.580849+0000","last_change":"2024-09-13T14:46:01.769601+0000","last_active":"2024-09-13T14:46:22.580849+0000","last_peered":"2024-09-13T14:46:22.580849+0000","last_clean":"2024-09-13T14:46:22.580849+0000","last_became_active":"2024-09-13T14:46:01.769276+0000","last_became_peered":"2024-09-13T14:46:01.769276+0000","last_unstale":"2024-09-13T14:46:22.580849+0000","last_undegraded":"2024-09-13T14:46:22.580849+0000","last_fullsized":"2024-09-13T14:46:22.580849+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":304,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":1654,"log_dups_size":0,"ondisk_log_size":1654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:05:46.860485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108396544,"num_objects":1654,"num_object_clones":0,"num_object_copies":4962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1654,"num_write_kb":105856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.973000+0000","last_change":"2024-09-13T14:46:01.769791+0000","last_active":"2024-09-13T14:46:25.973000+0000","last_peered":"2024-09-13T14:46:25.973000+0000","last_clean":"2024-09-13T14:46:25.973000+0000","last_became_active":"2024-09-13T14:46:01.768801+0000","last_became_peered":"2024-09-13T14:46:01.768801+0000","last_unstale":"2024-09-13T14:46:25.973000+0000","last_undegraded":"2024-09-13T14:46:25.973000+0000","last_fullsized":"2024-09-13T14:46:25.973000+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:26:55.790375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":837,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.971312+0000","last_change":"2024-09-13T14:38:04.596192+0000","last_active":"2024-09-13T14:46:25.971312+0000","last_peered":"2024-09-13T14:46:25.971312+0000","last_clean":"2024-09-13T14:46:25.971312+0000","last_became_active":"2024-09-13T14:38:04.593299+0000","last_became_peered":"2024-09-13T14:38:04.593299+0000","last_unstale":"2024-09-13T14:46:25.971312+0000","last_undegraded":"2024-09-13T14:46:25.971312+0000","last_fullsized":"2024-09-13T14:46:25.971312+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:34.241410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":153,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:46:26.970208+0000","last_change":"2024-09-13T14:38:03.533120+0000","last_active":"2024-09-13T14:46:26.970208+0000","last_peered":"2024-09-13T14:46:26.970208+0000","last_clean":"2024-09-13T14:46:26.970208+0000","last_became_active":"2024-09-13T14:38:03.532829+0000","last_became_peered":"2024-09-13T14:38:03.532829+0000","last_unstale":"2024-09-13T14:46:26.970208+0000","last_undegraded":"2024-09-13T14:46:26.970208+0000","last_fullsized":"2024-09-13T14:46:26.970208+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:24.514661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"314'2177","reported_seq":2208,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.310944+0000","last_change":"2024-09-13T14:45:59.743204+0000","last_active":"2024-09-13T14:46:27.310944+0000","last_peered":"2024-09-13T14:46:27.310944+0000","last_clean":"2024-09-13T14:46:27.310944+0000","last_became_active":"2024-09-13T14:45:59.742910+0000","last_became_peered":"2024-09-13T14:45:59.742910+0000","last_unstale":"2024-09-13T14:46:27.310944+0000","last_undegraded":"2024-09-13T14:46:27.310944+0000","last_fullsized":"2024-09-13T14:46:27.310944+0000","mapping_epoch":304,"log_start":"0'0","ondisk_log_start":"0'0","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":2177,"log_dups_size":0,"ondisk_log_size":2177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:51.017907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142671872,"num_objects":2177,"num_object_clones":0,"num_object_copies":6531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2177,"num_write_kb":139328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":551,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-13T14:46:18.870369+0000","last_change":"2024-09-13T14:38:53.938227+0000","last_active":"2024-09-13T14:46:18.870369+0000","last_peered":"2024-09-13T14:46:18.870369+0000","last_clean":"2024-09-13T14:46:18.870369+0000","last_became_active":"2024-09-13T14:38:53.902867+0000","last_became_peered":"2024-09-13T14:38:53.902867+0000","last_unstale":"2024-09-13T14:46:18.870369+0000","last_undegraded":"2024-09-13T14:46:18.870369+0000","last_fullsized":"2024-09-13T14:46:18.870369+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:16.831756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":613,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-13T14:46:11.797920+0000","last_change":"2024-09-13T14:37:48.147677+0000","last_active":"2024-09-13T14:46:11.797920+0000","last_peered":"2024-09-13T14:46:11.797920+0000","last_clean":"2024-09-13T14:46:11.797920+0000","last_became_active":"2024-09-13T14:37:48.147001+0000","last_became_peered":"2024-09-13T14:37:48.147001+0000","last_unstale":"2024-09-13T14:46:11.797920+0000","last_undegraded":"2024-09-13T14:46:11.797920+0000","last_fullsized":"2024-09-13T14:46:11.797920+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:35:00.653576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":533,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-13T14:46:18.870443+0000","last_change":"2024-09-13T14:38:53.900107+0000","last_active":"2024-09-13T14:46:18.870443+0000","last_peered":"2024-09-13T14:46:18.870443+0000","last_clean":"2024-09-13T14:46:18.870443+0000","last_became_active":"2024-09-13T14:38:53.899556+0000","last_became_peered":"2024-09-13T14:38:53.899556+0000","last_unstale":"2024-09-13T14:46:18.870443+0000","last_undegraded":"2024-09-13T14:46:18.870443+0000","last_fullsized":"2024-09-13T14:46:18.870443+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:15:47.918163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"314'2230","reported_seq":2258,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:46:27.313128+0000","last_change":"2024-09-13T14:46:01.764360+0000","last_active":"2024-09-13T14:46:27.313128+0000","last_peered":"2024-09-13T14:46:27.313128+0000","last_clean":"2024-09-13T14:46:27.313128+0000","last_became_active":"2024-09-13T14:46:01.764029+0000","last_became_peered":"2024-09-13T14:46:01.764029+0000","last_unstale":"2024-09-13T14:46:27.313128+0000","last_undegraded":"2024-09-13T14:46:27.313128+0000","last_fullsized":"2024-09-13T14:46:27.313128+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":304,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":2230,"log_dups_size":0,"ondisk_log_size":2230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:02:23.092494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146145280,"num_objects":2230,"num_object_clones":0,"num_object_copies":6690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2230,"num_write_kb":142720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":168,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:46:26.971229+0000","last_change":"2024-09-13T14:38:53.901806+0000","last_active":"2024-09-13T14:46:26.971229+0000","last_peered":"2024-09-13T14:46:26.971229+0000","last_clean":"2024-09-13T14:46:26.971229+0000","last_became_active":"2024-09-13T14:38:53.891055+0000","last_became_peered":"2024-09-13T14:38:53.891055+0000","last_unstale":"2024-09-13T14:46:26.971229+0000","last_undegraded":"2024-09-13T14:46:26.971229+0000","last_fullsized":"2024-09-13T14:46:26.971229+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:03:00.416198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":115,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.971335+0000","last_change":"2024-09-13T14:46:01.781508+0000","last_active":"2024-09-13T14:46:25.971335+0000","last_peered":"2024-09-13T14:46:25.971335+0000","last_clean":"2024-09-13T14:46:25.971335+0000","last_became_active":"2024-09-13T14:46:01.767096+0000","last_became_peered":"2024-09-13T14:46:01.767096+0000","last_unstale":"2024-09-13T14:46:25.971335+0000","last_undegraded":"2024-09-13T14:46:25.971335+0000","last_fullsized":"2024-09-13T14:46:25.971335+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:20:01.119189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":303,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-13T14:46:11.797240+0000","last_change":"2024-09-13T14:37:49.172218+0000","last_active":"2024-09-13T14:46:11.797240+0000","last_peered":"2024-09-13T14:46:11.797240+0000","last_clean":"2024-09-13T14:46:11.797240+0000","last_became_active":"2024-09-13T14:37:49.171718+0000","last_became_peered":"2024-09-13T14:37:49.171718+0000","last_unstale":"2024-09-13T14:46:11.797240+0000","last_undegraded":"2024-09-13T14:46:11.797240+0000","last_fullsized":"2024-09-13T14:46:11.797240+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:46:54.626521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":325,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-13T14:46:11.797477+0000","last_change":"2024-09-13T14:37:49.230094+0000","last_active":"2024-09-13T14:46:11.797477+0000","last_peered":"2024-09-13T14:46:11.797477+0000","last_clean":"2024-09-13T14:46:11.797477+0000","last_became_active":"2024-09-13T14:37:49.229393+0000","last_became_peered":"2024-09-13T14:37:49.229393+0000","last_unstale":"2024-09-13T14:46:11.797477+0000","last_undegraded":"2024-09-13T14:46:11.797477+0000","last_fullsized":"2024-09-13T14:46:11.797477+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:02:28.690318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":305,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-13T14:46:11.797607+0000","last_change":"2024-09-13T14:37:48.128447+0000","last_active":"2024-09-13T14:46:11.797607+0000","last_peered":"2024-09-13T14:46:11.797607+0000","last_clean":"2024-09-13T14:46:11.797607+0000","last_became_active":"2024-09-13T14:37:48.128143+0000","last_became_peered":"2024-09-13T14:37:48.128143+0000","last_unstale":"2024-09-13T14:46:11.797607+0000","last_undegraded":"2024-09-13T14:46:11.797607+0000","last_fullsized":"2024-09-13T14:46:11.797607+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:22.204039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":24,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.972624+0000","last_change":"2024-09-13T14:46:01.768529+0000","last_active":"2024-09-13T14:46:25.972624+0000","last_peered":"2024-09-13T14:46:25.972624+0000","last_clean":"2024-09-13T14:46:25.972624+0000","last_became_active":"2024-09-13T14:46:01.768221+0000","last_became_peered":"2024-09-13T14:46:01.768221+0000","last_unstale":"2024-09-13T14:46:25.972624+0000","last_undegraded":"2024-09-13T14:46:25.972624+0000","last_fullsized":"2024-09-13T14:46:25.972624+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:16:00.647959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":80,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.971097+0000","last_change":"2024-09-13T14:38:53.954281+0000","last_active":"2024-09-13T14:46:25.971097+0000","last_peered":"2024-09-13T14:46:25.971097+0000","last_clean":"2024-09-13T14:46:25.971097+0000","last_became_active":"2024-09-13T14:38:53.900912+0000","last_became_peered":"2024-09-13T14:38:53.900912+0000","last_unstale":"2024-09-13T14:46:25.971097+0000","last_undegraded":"2024-09-13T14:46:25.971097+0000","last_fullsized":"2024-09-13T14:46:25.971097+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:47.876275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":132,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.969624+0000","last_change":"2024-09-13T14:38:54.000782+0000","last_active":"2024-09-13T14:46:25.969624+0000","last_peered":"2024-09-13T14:46:25.969624+0000","last_clean":"2024-09-13T14:46:25.969624+0000","last_became_active":"2024-09-13T14:38:54.000322+0000","last_became_peered":"2024-09-13T14:38:54.000322+0000","last_unstale":"2024-09-13T14:46:25.969624+0000","last_undegraded":"2024-09-13T14:46:25.969624+0000","last_fullsized":"2024-09-13T14:46:25.969624+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:09.057597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":475,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-13T14:46:18.869207+0000","last_change":"2024-09-13T14:38:53.899535+0000","last_active":"2024-09-13T14:46:18.869207+0000","last_peered":"2024-09-13T14:46:18.869207+0000","last_clean":"2024-09-13T14:46:18.869207+0000","last_became_active":"2024-09-13T14:38:53.899183+0000","last_became_peered":"2024-09-13T14:38:53.899183+0000","last_unstale":"2024-09-13T14:46:18.869207+0000","last_undegraded":"2024-09-13T14:46:18.869207+0000","last_fullsized":"2024-09-13T14:46:18.869207+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:33.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":153,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:46:26.970967+0000","last_change":"2024-09-13T14:38:03.543426+0000","last_active":"2024-09-13T14:46:26.970967+0000","last_peered":"2024-09-13T14:46:26.970967+0000","last_clean":"2024-09-13T14:46:26.970967+0000","last_became_active":"2024-09-13T14:38:03.542661+0000","last_became_peered":"2024-09-13T14:38:03.542661+0000","last_unstale":"2024-09-13T14:46:26.970967+0000","last_undegraded":"2024-09-13T14:46:26.970967+0000","last_fullsized":"2024-09-13T14:46:26.970967+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:54.050127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":504,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-13T14:46:18.870327+0000","last_change":"2024-09-13T14:37:48.131321+0000","last_active":"2024-09-13T14:46:18.870327+0000","last_peered":"2024-09-13T14:46:18.870327+0000","last_clean":"2024-09-13T14:46:18.870327+0000","last_became_active":"2024-09-13T14:37:48.130998+0000","last_became_peered":"2024-09-13T14:37:48.130998+0000","last_unstale":"2024-09-13T14:46:18.870327+0000","last_undegraded":"2024-09-13T14:46:18.870327+0000","last_fullsized":"2024-09-13T14:46:18.870327+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:33:26.597825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":144,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.969742+0000","last_change":"2024-09-13T14:38:03.535454+0000","last_active":"2024-09-13T14:46:25.969742+0000","last_peered":"2024-09-13T14:46:25.969742+0000","last_clean":"2024-09-13T14:46:25.969742+0000","last_became_active":"2024-09-13T14:38:03.535131+0000","last_became_peered":"2024-09-13T14:38:03.535131+0000","last_unstale":"2024-09-13T14:46:25.969742+0000","last_undegraded":"2024-09-13T14:46:25.969742+0000","last_fullsized":"2024-09-13T14:46:25.969742+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:24.477601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":535,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-13T14:46:18.870532+0000","last_change":"2024-09-13T14:46:01.766605+0000","last_active":"2024-09-13T14:46:18.870532+0000","last_peered":"2024-09-13T14:46:18.870532+0000","last_clean":"2024-09-13T14:46:18.870532+0000","last_became_active":"2024-09-13T14:46:01.766284+0000","last_became_peered":"2024-09-13T14:46:01.766284+0000","last_unstale":"2024-09-13T14:46:18.870532+0000","last_undegraded":"2024-09-13T14:46:18.870532+0000","last_fullsized":"2024-09-13T14:46:18.870532+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:53:48.686769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":161,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:46:26.971143+0000","last_change":"2024-09-13T14:38:53.892948+0000","last_active":"2024-09-13T14:46:26.971143+0000","last_peered":"2024-09-13T14:46:26.971143+0000","last_clean":"2024-09-13T14:46:26.971143+0000","last_became_active":"2024-09-13T14:38:53.892665+0000","last_became_peered":"2024-09-13T14:38:53.892665+0000","last_unstale":"2024-09-13T14:46:26.971143+0000","last_undegraded":"2024-09-13T14:46:26.971143+0000","last_fullsized":"2024-09-13T14:46:26.971143+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:08.292359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":477,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-13T14:46:18.869417+0000","last_change":"2024-09-13T14:37:48.139348+0000","last_active":"2024-09-13T14:46:18.869417+0000","last_peered":"2024-09-13T14:46:18.869417+0000","last_clean":"2024-09-13T14:46:18.869417+0000","last_became_active":"2024-09-13T14:37:48.139115+0000","last_became_peered":"2024-09-13T14:37:48.139115+0000","last_unstale":"2024-09-13T14:46:18.869417+0000","last_undegraded":"2024-09-13T14:46:18.869417+0000","last_fullsized":"2024-09-13T14:46:18.869417+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:32:20.413346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":517,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-13T14:46:18.873822+0000","last_change":"2024-09-13T14:37:48.172088+0000","last_active":"2024-09-13T14:46:18.873822+0000","last_peered":"2024-09-13T14:46:18.873822+0000","last_clean":"2024-09-13T14:46:18.873822+0000","last_became_active":"2024-09-13T14:37:48.171779+0000","last_became_peered":"2024-09-13T14:37:48.171779+0000","last_unstale":"2024-09-13T14:46:18.873822+0000","last_undegraded":"2024-09-13T14:46:18.873822+0000","last_fullsized":"2024-09-13T14:46:18.873822+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:02.852712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":378,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-13T14:46:11.797429+0000","last_change":"2024-09-13T14:46:01.762871+0000","last_active":"2024-09-13T14:46:11.797429+0000","last_peered":"2024-09-13T14:46:11.797429+0000","last_clean":"2024-09-13T14:46:11.797429+0000","last_became_active":"2024-09-13T14:46:01.762531+0000","last_became_peered":"2024-09-13T14:46:01.762531+0000","last_unstale":"2024-09-13T14:46:11.797429+0000","last_undegraded":"2024-09-13T14:46:11.797429+0000","last_fullsized":"2024-09-13T14:46:11.797429+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:09:16.982787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":23,"reported_epoch":312,"state":"peering","last_fresh":"2024-09-13T14:46:18.869736+0000","last_change":"2024-09-13T14:46:25.972484+0000","last_active":"2024-09-13T14:46:25.972484+0000","last_peered":"2024-09-13T14:46:18.869736+0000","last_clean":"2024-09-13T14:46:18.869736+0000","last_became_active":"2024-09-13T14:46:01.767398+0000","last_became_peered":"2024-09-13T14:46:01.767398+0000","last_unstale":"2024-09-13T14:46:18.869736+0000","last_undegraded":"2024-09-13T14:46:18.869736+0000","last_fullsized":"2024-09-13T14:46:18.869736+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,3,5],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":153,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:46:26.970037+0000","last_change":"2024-09-13T14:38:03.527021+0000","last_active":"2024-09-13T14:46:26.970037+0000","last_peered":"2024-09-13T14:46:26.970037+0000","last_clean":"2024-09-13T14:46:26.970037+0000","last_became_active":"2024-09-13T14:38:03.526698+0000","last_became_peered":"2024-09-13T14:38:03.526698+0000","last_unstale":"2024-09-13T14:46:26.970037+0000","last_undegraded":"2024-09-13T14:46:26.970037+0000","last_fullsized":"2024-09-13T14:46:26.970037+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:01:35.734086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":322,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-13T14:46:11.797427+0000","last_change":"2024-09-13T14:46:01.763375+0000","last_active":"2024-09-13T14:46:11.797427+0000","last_peered":"2024-09-13T14:46:11.797427+0000","last_clean":"2024-09-13T14:46:11.797427+0000","last_became_active":"2024-09-13T14:46:01.763045+0000","last_became_peered":"2024-09-13T14:46:01.763045+0000","last_unstale":"2024-09-13T14:46:11.797427+0000","last_undegraded":"2024-09-13T14:46:11.797427+0000","last_fullsized":"2024-09-13T14:46:11.797427+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:03:35.294598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":129,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.969606+0000","last_change":"2024-09-13T14:37:49.176796+0000","last_active":"2024-09-13T14:46:25.969606+0000","last_peered":"2024-09-13T14:46:25.969606+0000","last_clean":"2024-09-13T14:46:25.969606+0000","last_became_active":"2024-09-13T14:37:49.176246+0000","last_became_peered":"2024-09-13T14:37:49.176246+0000","last_unstale":"2024-09-13T14:46:25.969606+0000","last_undegraded":"2024-09-13T14:46:25.969606+0000","last_fullsized":"2024-09-13T14:46:25.969606+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:29:24.936718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":324,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-13T14:46:11.797337+0000","last_change":"2024-09-13T14:38:03.526333+0000","last_active":"2024-09-13T14:46:11.797337+0000","last_peered":"2024-09-13T14:46:11.797337+0000","last_clean":"2024-09-13T14:46:11.797337+0000","last_became_active":"2024-09-13T14:38:03.526001+0000","last_became_peered":"2024-09-13T14:38:03.526001+0000","last_unstale":"2024-09-13T14:46:11.797337+0000","last_undegraded":"2024-09-13T14:46:11.797337+0000","last_fullsized":"2024-09-13T14:46:11.797337+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:25:25.181271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":129,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-13T14:46:25.969703+0000","last_change":"2024-09-13T14:37:49.149582+0000","last_active":"2024-09-13T14:46:25.969703+0000","last_peered":"2024-09-13T14:46:25.969703+0000","last_clean":"2024-09-13T14:46:25.969703+0000","last_became_active":"2024-09-13T14:37:49.148525+0000","last_became_peered":"2024-09-13T14:37:49.148525+0000","last_unstale":"2024-09-13T14:46:25.969703+0000","last_undegraded":"2024-09-13T14:46:25.969703+0000","last_fullsized":"2024-09-13T14:46:25.969703+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:48.655665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":556,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-13T14:46:18.869563+0000","last_change":"2024-09-13T14:46:01.766580+0000","last_active":"2024-09-13T14:46:18.869563+0000","last_peered":"2024-09-13T14:46:18.869563+0000","last_clean":"2024-09-13T14:46:18.869563+0000","last_became_active":"2024-09-13T14:46:01.766257+0000","last_became_peered":"2024-09-13T14:46:01.766257+0000","last_unstale":"2024-09-13T14:46:18.869563+0000","last_undegraded":"2024-09-13T14:46:18.869563+0000","last_fullsized":"2024-09-13T14:46:18.869563+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:01.126109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":520,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-13T14:46:18.870595+0000","last_change":"2024-09-13T14:46:01.769647+0000","last_active":"2024-09-13T14:46:18.870595+0000","last_peered":"2024-09-13T14:46:18.870595+0000","last_clean":"2024-09-13T14:46:18.870595+0000","last_became_active":"2024-09-13T14:46:01.769228+0000","last_became_peered":"2024-09-13T14:46:01.769228+0000","last_unstale":"2024-09-13T14:46:18.870595+0000","last_undegraded":"2024-09-13T14:46:18.870595+0000","last_fullsized":"2024-09-13T14:46:18.870595+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:58.696024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":305,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-13T14:46:11.797221+0000","last_change":"2024-09-13T14:37:48.127607+0000","last_active":"2024-09-13T14:46:11.797221+0000","last_peered":"2024-09-13T14:46:11.797221+0000","last_clean":"2024-09-13T14:46:11.797221+0000","last_became_active":"2024-09-13T14:37:48.127316+0000","last_became_peered":"2024-09-13T14:37:48.127316+0000","last_unstale":"2024-09-13T14:46:11.797221+0000","last_undegraded":"2024-09-13T14:46:11.797221+0000","last_fullsized":"2024-09-13T14:46:11.797221+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:53:37.678861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":173,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-13T14:46:26.970913+0000","last_change":"2024-09-13T14:46:01.775237+0000","last_active":"2024-09-13T14:46:26.970913+0000","last_peered":"2024-09-13T14:46:26.970913+0000","last_clean":"2024-09-13T14:46:26.970913+0000","last_became_active":"2024-09-13T14:46:01.765025+0000","last_became_peered":"2024-09-13T14:46:01.765025+0000","last_unstale":"2024-09-13T14:46:26.970913+0000","last_undegraded":"2024-09-13T14:46:26.970913+0000","last_fullsized":"2024-09-13T14:46:26.970913+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:29:16.089042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1746,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":2110849024,"num_objects":32209,"num_object_clones":0,"num_object_copies":96627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32209,"num_write_kb":2061376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6412632064,"data_stored":6412632064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32209,"ondisk_log_size":32209,"up":48,"acting":48,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":2154,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25132,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16793600,"data_stored":16777292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2969600,"data_stored":2951840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234431,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1271140,"kb_used_data":788828,"kb_used_omap":1,"kb_used_meta":482302,"kb_avail":92474012,"statfs":{"total":95995035648,"available":94693388288,"internally_reserved":0,"allocated":807759872,"data_stored":806106775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":493877707},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":239,"seq":1026497183849,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1058068,"kb_used_data":680712,"kb_used_omap":8,"kb_used_meta":377335,"kb_avail":92687084,"statfs":{"total":95995035648,"available":94911574016,"internally_reserved":0,"allocated":697049088,"data_stored":695408155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8453,"internal_metadata":386391803},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757696,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2014168,"kb_used_data":1284936,"kb_used_omap":1,"kb_used_meta":729214,"kb_avail":91730984,"statfs":{"total":95995035648,"available":93932527616,"internally_reserved":0,"allocated":1315774464,"data_stored":1314137002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":746715590},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019339,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1601156,"kb_used_data":974460,"kb_used_omap":1,"kb_used_meta":626686,"kb_avail":92143996,"statfs":{"total":95995035648,"available":94355451904,"internally_reserved":0,"allocated":997847040,"data_stored":996209619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":641726918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435522,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1366836,"kb_used_data":731244,"kb_used_omap":5,"kb_used_meta":635578,"kb_avail":92378316,"statfs":{"total":95995035648,"available":94595395584,"internally_reserved":0,"allocated":748793856,"data_stored":747168292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":650831998},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":104,"seq":446676599051,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1259960,"kb_used_data":1165808,"kb_used_omap":7,"kb_used_meta":94136,"kb_avail":92485192,"statfs":{"total":95995035648,"available":94704836608,"internally_reserved":0,"allocated":1193787392,"data_stored":1192158258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8010,"internal_metadata":96395446},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":425201762577,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926101,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2630748,"kb_used_data":1909072,"kb_used_omap":7,"kb_used_meta":721656,"kb_avail":91114404,"statfs":{"total":95995035648,"available":93301149696,"internally_reserved":0,"allocated":1954889728,"data_stored":1953244913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":738976040},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6860,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":872677376,"data_stored":872677376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1247543296,"data_stored":1247543296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":766771200,"data_stored":766771200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":782172160,"data_stored":782172160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1308426240,"data_stored":1308426240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":699400192,"data_stored":699400192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":735641600,"data_stored":735641600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:46:31.829 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-13T14:46:31.829 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pg_num 11 2024-09-13T14:46:31.868 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:31.908 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:31.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:31.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:32.010 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:32.053 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:32.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:32.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:32.155 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:32.184 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:32.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:32.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:32.286 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:32.321 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:32.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:32.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:32.415 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:46:32.423 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:32.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:32.465 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:32.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:32.497 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:46:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=cleanup t=2024-09-13T14:46:32.213145483Z level=info msg="Completed cleanup jobs" duration=14.888758ms 2024-09-13T14:46:32.497 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:46:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=grafana.update.checker t=2024-09-13T14:46:32.346578532Z level=info msg="Update check succeeded" duration=70.457484ms 2024-09-13T14:46:32.497 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:46:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=plugins.update.checker t=2024-09-13T14:46:32.355452204Z level=info msg="Update check succeeded" duration=77.921545ms 2024-09-13T14:46:32.566 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:32.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:32.601 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:32.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:32.702 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:32.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:32.737 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:32.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:32.802 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 19 16 45927 45911 150.986 106.125 0.00573383 0.00661735 2024-09-13T14:46:32.838 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:32.859 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:32.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:32.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:32.961 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:32.991 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:32.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:32.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:33.093 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:33.126 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:33.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:33.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:33.154 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:33 smithi102 ceph-mon[110183]: pgmap v1100: 89 pgs: 89 active+clean; 2.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-13T14:46:33.155 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:33 smithi102 ceph-mon[115269]: pgmap v1100: 89 pgs: 89 active+clean; 2.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-13T14:46:33.228 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:33.262 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:33.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:33.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:33.266 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:33.364 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:33.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:33.404 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:33.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:33.419 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:33 smithi195 ceph-mon[104262]: pgmap v1100: 89 pgs: 89 active+clean; 2.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-13T14:46:33.505 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:33.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:33.527 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:33.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:33.628 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:33.665 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:33.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:33.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:33.767 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:33.802 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:33.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:33.803 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:2024-09-13T14:46:33.802774+0000 min lat: 0.00109072 max lat: 0.167201 avg lat: 0.00678526 2024-09-13T14:46:33.803 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:46:33.803 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 20 16 47155 47139 147.275 76.75 0.0139733 0.00678526 2024-09-13T14:46:33.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:33.904 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:33.941 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:33.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:33.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:34.014 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is set 2024-09-13T14:46:34.043 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:34.078 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:34.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:34.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:34.180 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:34.214 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:34.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:34.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:34.317 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:34.353 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:34.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:34.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:34.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:34 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/711802528' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:46:34.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:34 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/711802528' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:46:34.455 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:34.491 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:34.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:34.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:34.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:34 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/711802528' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:46:34.593 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:34.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:34.628 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:34.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:34.729 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:34.769 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:34.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:34.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:34.803 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 21 16 49025 49009 145.827 116.875 0.00716688 0.00685241 2024-09-13T14:46:34.871 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:34.900 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:34.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:34.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:34.964 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:35.002 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:35.042 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:35.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:35.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:35.144 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:35.179 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:35.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:35.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:35.281 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:35.314 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:35.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:35.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:35.416 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:35.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:35 smithi102 ceph-mon[110183]: pgmap v1101: 89 pgs: 89 active+clean; 2.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 148 MiB/s wr, 2.37k op/s 2024-09-13T14:46:35.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:35 smithi102 ceph-mon[110183]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:46:35.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:35 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/711802528' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:46:35.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:35 smithi102 ceph-mon[110183]: osdmap e315: 8 total, 8 up, 7 in 2024-09-13T14:46:35.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:35 smithi102 ceph-mon[115269]: pgmap v1101: 89 pgs: 89 active+clean; 2.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 148 MiB/s wr, 2.37k op/s 2024-09-13T14:46:35.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:35 smithi102 ceph-mon[115269]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:46:35.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:35 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/711802528' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:46:35.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:35 smithi102 ceph-mon[115269]: osdmap e315: 8 total, 8 up, 7 in 2024-09-13T14:46:35.451 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:35.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:35.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:35.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:35 smithi195 ceph-mon[104262]: pgmap v1101: 89 pgs: 89 active+clean; 2.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 148 MiB/s wr, 2.37k op/s 2024-09-13T14:46:35.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:35 smithi195 ceph-mon[104262]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:46:35.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:35 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/711802528' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:46:35.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:35 smithi195 ceph-mon[104262]: osdmap e315: 8 total, 8 up, 7 in 2024-09-13T14:46:35.554 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:35.586 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:35.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:35.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:35.688 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:35.720 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:35.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:35.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:35.803 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 22 16 50789 50773 144.209 110.25 0.0105812 0.00693 2024-09-13T14:46:35.821 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:35.860 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:35.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:35.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:35.961 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:35.993 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:35.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:35.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:36.095 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:36.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:36.135 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:36.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:36.236 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:36.267 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:36.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:36.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:36.343 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T14:46:36.369 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:36.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:36.432 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:36.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:36.533 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:36.559 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:36.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:36.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:36.610 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:46:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:46:36] "GET /metrics HTTP/1.1" 200 36521 "" "Prometheus/2.51.0" 2024-09-13T14:46:36.661 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:36.693 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:36.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:36.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:36.707 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:46:36.795 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:36.803 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 23 16 52351 52335 142.184 97.625 0.015173 0.00702518 2024-09-13T14:46:36.823 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:36.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:36.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:36.925 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:36.960 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:36.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:36.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:37.062 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:37.095 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:37.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:37.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:37.197 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:37.230 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:37.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:37.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:37.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:37 smithi195 ceph-mon[104262]: pgmap v1103: 89 pgs: 89 active+clean; 2.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 133 MiB/s wr, 2.13k op/s 2024-09-13T14:46:37.332 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:37.371 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:37.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:37.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:37.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:37 smithi102 ceph-mon[110183]: pgmap v1103: 89 pgs: 89 active+clean; 2.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 133 MiB/s wr, 2.13k op/s 2024-09-13T14:46:37.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:37 smithi102 ceph-mon[115269]: pgmap v1103: 89 pgs: 89 active+clean; 2.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 133 MiB/s wr, 2.13k op/s 2024-09-13T14:46:37.473 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:37.510 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:37.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:37.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:37.612 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:37.654 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:37.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:37.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:37.756 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:37.789 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:37.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:37.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:37.803 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 24 16 53968 53952 140.47 101.062 0.00763921 0.0071133 2024-09-13T14:46:37.891 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:37.922 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:37.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:37.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:38.025 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:38.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:38.059 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:38.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:38.092 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:46:38.161 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:38.204 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:38.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:38.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:38.306 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:38.327 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:38.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:38.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:38.429 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:38.479 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:38.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:38.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:38.581 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:38.614 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:38.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:38.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:38.716 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:38.749 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:38.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:38.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:38.803 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 25 16 55476 55460 138.621 94.25 0.013102 0.00720844 2024-09-13T14:46:38.851 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:38.875 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:38.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:38.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:38.977 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:39.003 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:39.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:39.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:39.048 INFO:teuthology.orchestra.run.smithi102.stderr:set pool 1 pg_num to 11 2024-09-13T14:46:39.106 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:39.126 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:39.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:39.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:39.228 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:39.249 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:39.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:39.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:39.350 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:39.382 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:39.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:39.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:39.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:39 smithi102 ceph-mon[110183]: pgmap v1104: 89 pgs: 89 active+clean; 3.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 114 MiB/s wr, 1.82k op/s 2024-09-13T14:46:39.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:39 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1635848300' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-13T14:46:39.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:39 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-13T14:46:39.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:39 smithi102 ceph-mon[115269]: pgmap v1104: 89 pgs: 89 active+clean; 3.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 114 MiB/s wr, 1.82k op/s 2024-09-13T14:46:39.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:39 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1635848300' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-13T14:46:39.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:39 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-13T14:46:39.484 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:39.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:39 smithi195 ceph-mon[104262]: pgmap v1104: 89 pgs: 89 active+clean; 3.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 114 MiB/s wr, 1.82k op/s 2024-09-13T14:46:39.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:39 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1635848300' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-13T14:46:39.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:39 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-13T14:46:39.507 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:39.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:39.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:39.609 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:39.630 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:39.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:39.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:39.732 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:39.752 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:39.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:39.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:39.803 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 26 16 56085 56069 134.754 38.0625 0.0084771 0.00739641 2024-09-13T14:46:39.854 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:39.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:39.900 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:39.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:40.001 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:40.022 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:40.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:40.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:40.024 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:46:40.024 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-13T14:46:40.024 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-09-13T14:46:40.024 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-09-13T14:46:40.024 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:46:40.124 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:40.154 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:40.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:40.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:40.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:40 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-13T14:46:40.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:40 smithi195 ceph-mon[104262]: osdmap e316: 8 total, 8 up, 7 in 2024-09-13T14:46:40.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:40 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-13T14:46:40.256 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:40.277 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:40.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:40.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:40.379 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:40.411 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:40.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:40.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:40.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:40 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-13T14:46:40.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:40 smithi102 ceph-mon[110183]: osdmap e316: 8 total, 8 up, 7 in 2024-09-13T14:46:40.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:40 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-13T14:46:40.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:40 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-13T14:46:40.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:40 smithi102 ceph-mon[115269]: osdmap e316: 8 total, 8 up, 7 in 2024-09-13T14:46:40.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:40 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-13T14:46:40.513 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:40.551 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:40.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:40.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:40.653 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:40.688 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:40.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:40.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:40.723 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T14:46:40.790 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:40.803 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 27 16 57320 57304 132.622 77.1875 0.00704563 0.00753479 2024-09-13T14:46:40.821 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:40.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:40.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:40.923 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:40.967 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:40.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:40.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:41.068 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:41.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:41.100 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:41.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:41.123 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:46:41.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:41 smithi102 ceph-mon[110183]: pgmap v1106: 89 pgs: 89 active+clean; 3.3 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 133 MiB/s wr, 2.13k op/s 2024-09-13T14:46:41.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:41 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-13T14:46:41.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:41 smithi102 ceph-mon[110183]: osdmap e317: 8 total, 8 up, 7 in 2024-09-13T14:46:41.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:41 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:46:41.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:41 smithi102 ceph-mon[115269]: pgmap v1106: 89 pgs: 89 active+clean; 3.3 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 133 MiB/s wr, 2.13k op/s 2024-09-13T14:46:41.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:41 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-13T14:46:41.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:41 smithi102 ceph-mon[115269]: osdmap e317: 8 total, 8 up, 7 in 2024-09-13T14:46:41.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:41 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:46:41.201 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:41.234 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:41.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:41.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:41.311 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:41 smithi195 ceph-mon[104262]: pgmap v1106: 89 pgs: 89 active+clean; 3.3 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 133 MiB/s wr, 2.13k op/s 2024-09-13T14:46:41.311 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:41 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-13T14:46:41.311 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:41 smithi195 ceph-mon[104262]: osdmap e317: 8 total, 8 up, 7 in 2024-09-13T14:46:41.311 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:41 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:46:41.336 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:41.375 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:41.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:41.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:41.477 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:41.513 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:41.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:41.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:41.614 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:41.646 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:41.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:41.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:41.748 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:41.780 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:41.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:41.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:41.803 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 28 16 58919 58903 131.455 99.9375 0.00517346 0.00760251 2024-09-13T14:46:41.882 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:41.916 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:41.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:41.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:42.018 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:42.054 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:42.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:42.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:42.156 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:42.189 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:42.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:42.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:42.291 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:42.325 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:42.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:42.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:42.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:42 smithi102 ceph-mon[115269]: osdmap e318: 8 total, 8 up, 7 in 2024-09-13T14:46:42.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:42 smithi102 ceph-mon[110183]: osdmap e318: 8 total, 8 up, 7 in 2024-09-13T14:46:42.419 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:42 smithi195 ceph-mon[104262]: osdmap e318: 8 total, 8 up, 7 in 2024-09-13T14:46:42.427 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:42.463 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:42.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:42.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:42.565 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:42.598 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:42.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:42.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:42.700 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:42.736 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:42.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:42.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:42.804 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 29 16 60593 60577 130.529 104.625 0.0129883 0.00765651 2024-09-13T14:46:42.838 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:42.861 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:42.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:42.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:42.962 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:42.997 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:42.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:42.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:43.098 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:43.127 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:43.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:43.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:43.158 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:46:43.158 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-09-13T14:46:43.158 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-09-13T14:46:43.158 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-13T14:46:43.158 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.158 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113979 7.4 7:2d0634a9:::benchmark_data_smithi102_227395_object1164:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-13T14:46:43.158 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:47.339159+0000", 2024-09-13T14:46:43.158 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 535.81308935599998, 2024-09-13T14:46:43.158 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.356370809, 2024-09-13T14:46:43.159 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:46:43.159 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:43.159 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:46:43.159 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:46:43.159 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:46:43.159 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113979 2024-09-13T14:46:43.159 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.159 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:46:43.159 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.159 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:46:43.159 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.339159+0000", 2024-09-13T14:46:43.160 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:43.160 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.160 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.160 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:46:43.160 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.339159+0000", 2024-09-13T14:46:43.160 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:43.160 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.160 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.160 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:46:43.161 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.339158+0000", 2024-09-13T14:46:43.161 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:46:43.161 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.161 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.161 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:46:43.161 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.339165+0000", 2024-09-13T14:46:43.161 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4719999999999998e-06 2024-09-13T14:46:43.161 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.161 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.161 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:46:43.161 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.339178+0000", 2024-09-13T14:46:43.161 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.306e-05 2024-09-13T14:46:43.161 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.162 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.162 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:46:43.162 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.339185+0000", 2024-09-13T14:46:43.162 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1509999999999998e-06 2024-09-13T14:46:43.162 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.162 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.162 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:43.162 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.339232+0000", 2024-09-13T14:46:43.162 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6953000000000003e-05 2024-09-13T14:46:43.162 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.162 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.162 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:46:43.162 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.339241+0000", 2024-09-13T14:46:43.162 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0900000000000005e-06 2024-09-13T14:46:43.162 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.163 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.163 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:43.163 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.112527+0000", 2024-09-13T14:46:43.163 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.77328639700000001 2024-09-13T14:46:43.163 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.163 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.163 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:46:43.163 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.112538+0000", 2024-09-13T14:46:43.163 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1090999999999999e-05 2024-09-13T14:46:43.163 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.163 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.163 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:43.163 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.664075+0000", 2024-09-13T14:46:43.163 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5515367659999999 2024-09-13T14:46:43.163 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.164 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.164 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:46:43.164 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.664233+0000", 2024-09-13T14:46:43.164 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015868300000000001 2024-09-13T14:46:43.164 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.164 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.164 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:46:43.164 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.664384+0000", 2024-09-13T14:46:43.164 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000150371 2024-09-13T14:46:43.164 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.164 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.165 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:43.165 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.673923+0000", 2024-09-13T14:46:43.165 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0095388030000000002 2024-09-13T14:46:43.165 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.165 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.165 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:46:43.165 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.677432+0000", 2024-09-13T14:46:43.165 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0035092560000000001 2024-09-13T14:46:43.165 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.165 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.165 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:43.165 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.692151+0000", 2024-09-13T14:46:43.165 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014718716999999999 2024-09-13T14:46:43.166 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.166 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.166 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:46:43.166 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.692201+0000", 2024-09-13T14:46:43.166 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0145000000000002e-05 2024-09-13T14:46:43.166 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.166 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.166 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:46:43.166 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.695530+0000", 2024-09-13T14:46:43.166 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0033293260000000001 2024-09-13T14:46:43.166 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:43.166 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:46:43.166 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:43.166 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.167 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.167 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113982 7.4 7:256588d9:::benchmark_data_smithi102_227395_object1167:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-13T14:46:43.167 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:47.404500+0000", 2024-09-13T14:46:43.167 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 535.74774857199998, 2024-09-13T14:46:43.168 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.296665693, 2024-09-13T14:46:43.168 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:46:43.168 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:43.168 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:46:43.168 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:46:43.168 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:46:43.168 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113982 2024-09-13T14:46:43.168 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.168 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:46:43.168 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.168 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:46:43.168 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.404500+0000", 2024-09-13T14:46:43.168 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:43.168 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.169 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.169 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:46:43.169 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.404500+0000", 2024-09-13T14:46:43.169 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:43.169 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.169 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.169 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:46:43.169 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.404499+0000", 2024-09-13T14:46:43.169 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:46:43.169 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.169 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.169 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:46:43.169 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.404505+0000", 2024-09-13T14:46:43.170 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.438e-06 2024-09-13T14:46:43.170 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.170 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.170 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:46:43.170 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.404517+0000", 2024-09-13T14:46:43.170 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2130000000000001e-05 2024-09-13T14:46:43.170 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.170 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.170 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:46:43.170 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.404522+0000", 2024-09-13T14:46:43.170 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2449999999999998e-06 2024-09-13T14:46:43.170 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.170 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.171 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:43.171 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.404570+0000", 2024-09-13T14:46:43.171 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7508e-05 2024-09-13T14:46:43.171 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.171 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.171 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:46:43.171 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.404578+0000", 2024-09-13T14:46:43.171 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2519999999999999e-06 2024-09-13T14:46:43.171 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.171 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.171 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:43.171 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.112562+0000", 2024-09-13T14:46:43.171 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.707983323 2024-09-13T14:46:43.171 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.172 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.172 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:46:43.172 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.112569+0000", 2024-09-13T14:46:43.172 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9379999999999992e-06 2024-09-13T14:46:43.172 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.172 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.172 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:43.172 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.670326+0000", 2024-09-13T14:46:43.172 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.557756707 2024-09-13T14:46:43.172 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.172 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.172 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:46:43.172 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.670465+0000", 2024-09-13T14:46:43.173 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013909099999999999 2024-09-13T14:46:43.173 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.173 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.173 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:46:43.173 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.670607+0000", 2024-09-13T14:46:43.173 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014218999999999999 2024-09-13T14:46:43.173 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.173 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.173 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:46:43.173 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.677438+0000", 2024-09-13T14:46:43.173 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.006830839 2024-09-13T14:46:43.173 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.173 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.174 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:43.174 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.678258+0000", 2024-09-13T14:46:43.174 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00081936699999999999 2024-09-13T14:46:43.174 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.174 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.174 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:43.174 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.695584+0000", 2024-09-13T14:46:43.174 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017326105000000001 2024-09-13T14:46:43.174 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.174 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.174 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:46:43.174 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.695638+0000", 2024-09-13T14:46:43.174 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3949000000000001e-05 2024-09-13T14:46:43.174 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.175 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.175 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:46:43.175 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.701166+0000", 2024-09-13T14:46:43.175 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0055280310000000001 2024-09-13T14:46:43.175 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:43.175 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:46:43.175 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:43.175 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.175 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.175 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113988 7.9 7:9e6eb1fd:::benchmark_data_smithi102_227395_object1173:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-13T14:46:43.175 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:47.878570+0000", 2024-09-13T14:46:43.175 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 535.27367818699997, 2024-09-13T14:46:43.175 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.34139049300000002, 2024-09-13T14:46:43.175 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:46:43.175 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:43.176 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:46:43.176 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:46:43.176 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:46:43.176 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113988 2024-09-13T14:46:43.176 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.176 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:46:43.176 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.176 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:46:43.176 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.878570+0000", 2024-09-13T14:46:43.176 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:43.176 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.176 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.176 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:46:43.177 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.878570+0000", 2024-09-13T14:46:43.177 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:43.177 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.177 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.177 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:46:43.177 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.878569+0000", 2024-09-13T14:46:43.177 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:46:43.177 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.177 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.177 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:46:43.177 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.878578+0000", 2024-09-13T14:46:43.177 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.8339999999999994e-06 2024-09-13T14:46:43.177 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.178 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.178 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:46:43.178 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.878595+0000", 2024-09-13T14:46:43.178 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7007000000000002e-05 2024-09-13T14:46:43.178 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.178 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.178 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:46:43.178 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.878623+0000", 2024-09-13T14:46:43.178 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7189999999999999e-05 2024-09-13T14:46:43.178 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.178 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.179 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:43.179 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.878685+0000", 2024-09-13T14:46:43.179 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2661000000000002e-05 2024-09-13T14:46:43.179 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.179 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.179 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:46:43.179 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.878698+0000", 2024-09-13T14:46:43.179 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3139000000000001e-05 2024-09-13T14:46:43.179 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.179 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.179 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:43.179 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.213485+0000", 2024-09-13T14:46:43.179 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.33478628199999999 2024-09-13T14:46:43.180 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.180 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.180 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:46:43.180 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.213573+0000", 2024-09-13T14:46:43.180 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8212000000000002e-05 2024-09-13T14:46:43.180 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.180 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.180 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,6,7", 2024-09-13T14:46:43.180 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.213671+0000", 2024-09-13T14:46:43.180 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.8344999999999994e-05 2024-09-13T14:46:43.180 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.180 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.180 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:43.180 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.216736+0000", 2024-09-13T14:46:43.181 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0030646760000000001 2024-09-13T14:46:43.181 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.181 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.181 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:46:43.181 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.216745+0000", 2024-09-13T14:46:43.181 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4220000000000006e-06 2024-09-13T14:46:43.181 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.181 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.181 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:43.181 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.216773+0000", 2024-09-13T14:46:43.181 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7970000000000002e-05 2024-09-13T14:46:43.181 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.181 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.182 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:43.182 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.219756+0000", 2024-09-13T14:46:43.182 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0029822120000000001 2024-09-13T14:46:43.182 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.182 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.182 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:46:43.182 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.219817+0000", 2024-09-13T14:46:43.182 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1235000000000006e-05 2024-09-13T14:46:43.182 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.182 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.182 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:46:43.182 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.219961+0000", 2024-09-13T14:46:43.182 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014409899999999999 2024-09-13T14:46:43.182 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:43.183 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:46:43.183 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:43.183 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.183 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.183 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113993 7.4 7:2f29f4ff:::benchmark_data_smithi102_227395_object1178:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-13T14:46:43.183 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:48.328273+0000", 2024-09-13T14:46:43.183 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 534.82397550300004, 2024-09-13T14:46:43.183 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3770136559999999, 2024-09-13T14:46:43.183 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:46:43.183 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:43.183 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:46:43.183 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:46:43.183 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:46:43.183 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113993 2024-09-13T14:46:43.183 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.183 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:46:43.184 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.184 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:46:43.184 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.328273+0000", 2024-09-13T14:46:43.184 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:43.184 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.184 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.184 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:46:43.184 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.328273+0000", 2024-09-13T14:46:43.184 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:43.184 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.184 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.184 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:46:43.184 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.328271+0000", 2024-09-13T14:46:43.184 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:46:43.185 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.185 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.185 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:46:43.185 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.328279+0000", 2024-09-13T14:46:43.185 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0609999999999996e-06 2024-09-13T14:46:43.185 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.185 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.185 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:46:43.185 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.328294+0000", 2024-09-13T14:46:43.185 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4405e-05 2024-09-13T14:46:43.185 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.185 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.185 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:46:43.185 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.328301+0000", 2024-09-13T14:46:43.186 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6750000000000002e-06 2024-09-13T14:46:43.186 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.186 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.186 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:43.186 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.328376+0000", 2024-09-13T14:46:43.186 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4040000000000003e-05 2024-09-13T14:46:43.186 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.186 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.186 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:46:43.186 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.328389+0000", 2024-09-13T14:46:43.186 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3478e-05 2024-09-13T14:46:43.186 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.186 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.187 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:43.187 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.673990+0000", 2024-09-13T14:46:43.187 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.345601324 2024-09-13T14:46:43.187 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.187 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.187 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:46:43.187 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.674152+0000", 2024-09-13T14:46:43.187 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016154299999999999 2024-09-13T14:46:43.188 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.188 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.188 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:46:43.188 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.674339+0000", 2024-09-13T14:46:43.188 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00018710999999999999 2024-09-13T14:46:43.188 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.189 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.189 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:46:43.189 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.678268+0000", 2024-09-13T14:46:43.189 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0039291450000000002 2024-09-13T14:46:43.189 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.189 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.189 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:43.189 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.678582+0000", 2024-09-13T14:46:43.189 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00031387 2024-09-13T14:46:43.189 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.189 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.189 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:43.189 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.701236+0000", 2024-09-13T14:46:43.190 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.022653889 2024-09-13T14:46:43.190 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.190 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.190 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:46:43.190 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.701392+0000", 2024-09-13T14:46:43.190 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015579999999999999 2024-09-13T14:46:43.190 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.190 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.190 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:46:43.190 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.705287+0000", 2024-09-13T14:46:43.190 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003895098 2024-09-13T14:46:43.190 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:43.190 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:46:43.190 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:43.190 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.190 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.191 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113998 7.1 7:8cc51271:::benchmark_data_smithi102_227395_object1183:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-13T14:46:43.191 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:48.829978+0000", 2024-09-13T14:46:43.191 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 534.32227047399999, 2024-09-13T14:46:43.191 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.86212481600000002, 2024-09-13T14:46:43.191 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:46:43.191 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:43.191 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:46:43.191 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:46:43.191 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:46:43.191 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113998 2024-09-13T14:46:43.191 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.191 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:46:43.191 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.191 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:46:43.191 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.829978+0000", 2024-09-13T14:46:43.192 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:43.192 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.192 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.192 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:46:43.192 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.829978+0000", 2024-09-13T14:46:43.192 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:43.192 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.192 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.192 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:46:43.192 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.829977+0000", 2024-09-13T14:46:43.192 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:46:43.192 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.192 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.192 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:46:43.193 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.829984+0000", 2024-09-13T14:46:43.193 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7479999999999996e-06 2024-09-13T14:46:43.193 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.193 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.193 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:46:43.193 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.829999+0000", 2024-09-13T14:46:43.193 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5233e-05 2024-09-13T14:46:43.193 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.193 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.193 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:46:43.193 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.830004+0000", 2024-09-13T14:46:43.193 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4519999999999996e-06 2024-09-13T14:46:43.193 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.193 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.193 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:43.194 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.830059+0000", 2024-09-13T14:46:43.194 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4829e-05 2024-09-13T14:46:43.194 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.194 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.194 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:46:43.194 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.830072+0000", 2024-09-13T14:46:43.194 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3410999999999999e-05 2024-09-13T14:46:43.194 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.194 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.194 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:43.195 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.675677+0000", 2024-09-13T14:46:43.195 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.84560498200000001 2024-09-13T14:46:43.195 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.195 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.195 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:46:43.195 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.675810+0000", 2024-09-13T14:46:43.195 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000132786 2024-09-13T14:46:43.195 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.195 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.195 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:46:43.195 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.675943+0000", 2024-09-13T14:46:43.195 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000132898 2024-09-13T14:46:43.195 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.195 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.195 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:43.196 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.681639+0000", 2024-09-13T14:46:43.196 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0056959269999999999 2024-09-13T14:46:43.196 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.196 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.196 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:46:43.196 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.681746+0000", 2024-09-13T14:46:43.196 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010723299999999999 2024-09-13T14:46:43.196 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.196 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.196 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:43.196 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.689715+0000", 2024-09-13T14:46:43.197 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0079683510000000003 2024-09-13T14:46:43.197 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.197 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.197 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:46:43.197 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.689761+0000", 2024-09-13T14:46:43.197 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5936999999999997e-05 2024-09-13T14:46:43.197 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.197 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.197 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:46:43.197 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:49.692103+0000", 2024-09-13T14:46:43.197 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002342392 2024-09-13T14:46:43.197 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:43.197 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:46:43.198 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:43.198 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.198 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.198 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:179128 7.1 7:8ae09e6f:::benchmark_data_smithi102_227395_object66313:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-09-13T14:46:43.198 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:43:51.034284+0000", 2024-09-13T14:46:43.198 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 172.11796451699999, 2024-09-13T14:46:43.198 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.61896007399999997, 2024-09-13T14:46:43.198 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:46:43.198 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:43.198 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:46:43.198 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:46:43.198 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:46:43.198 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 179128 2024-09-13T14:46:43.198 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.199 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:46:43.199 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.199 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:46:43.199 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.034284+0000", 2024-09-13T14:46:43.199 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:43.199 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.199 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.199 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:46:43.199 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.034284+0000", 2024-09-13T14:46:43.199 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:43.199 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.199 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.199 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:46:43.199 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.034283+0000", 2024-09-13T14:46:43.199 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:46:43.200 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.200 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.200 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:46:43.200 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.034290+0000", 2024-09-13T14:46:43.200 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9439999999999999e-06 2024-09-13T14:46:43.200 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.200 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.200 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:46:43.200 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.034300+0000", 2024-09-13T14:46:43.200 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.8810000000000008e-06 2024-09-13T14:46:43.200 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.200 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.200 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:46:43.200 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.034305+0000", 2024-09-13T14:46:43.201 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0610000000000003e-06 2024-09-13T14:46:43.201 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.201 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.201 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:43.201 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.034342+0000", 2024-09-13T14:46:43.201 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7280999999999998e-05 2024-09-13T14:46:43.201 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.201 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.201 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:46:43.201 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.034349+0000", 2024-09-13T14:46:43.201 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3180000000000001e-06 2024-09-13T14:46:43.201 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.201 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.201 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:43.202 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.646792+0000", 2024-09-13T14:46:43.202 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.61244286599999997 2024-09-13T14:46:43.202 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.202 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.202 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:46:43.202 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.646938+0000", 2024-09-13T14:46:43.202 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014614000000000001 2024-09-13T14:46:43.202 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.202 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.202 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:46:43.202 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.647012+0000", 2024-09-13T14:46:43.202 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3583000000000003e-05 2024-09-13T14:46:43.202 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.203 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.203 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:43.203 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.650051+0000", 2024-09-13T14:46:43.203 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0030396889999999999 2024-09-13T14:46:43.203 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.203 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.203 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:46:43.203 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.653152+0000", 2024-09-13T14:46:43.203 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003100364 2024-09-13T14:46:43.203 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.203 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.203 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:43.203 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.653187+0000", 2024-09-13T14:46:43.203 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5630000000000003e-05 2024-09-13T14:46:43.204 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.204 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.204 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:46:43.204 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.653220+0000", 2024-09-13T14:46:43.204 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2286999999999998e-05 2024-09-13T14:46:43.204 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.204 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.204 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:46:43.204 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.653244+0000", 2024-09-13T14:46:43.204 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4445000000000001e-05 2024-09-13T14:46:43.204 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:43.204 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:46:43.204 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:43.204 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.205 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.205 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:179175 7.1 7:82c4bfbf:::benchmark_data_smithi102_227395_object66360:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-09-13T14:46:43.205 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:43:51.202947+0000", 2024-09-13T14:46:43.205 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 171.94930131199999, 2024-09-13T14:46:43.205 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.45122264299999998, 2024-09-13T14:46:43.205 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:46:43.205 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:43.205 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:46:43.205 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:46:43.205 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:46:43.205 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 179175 2024-09-13T14:46:43.205 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.205 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:46:43.206 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.206 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:46:43.206 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.202947+0000", 2024-09-13T14:46:43.206 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:43.206 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.206 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.206 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:46:43.206 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.202947+0000", 2024-09-13T14:46:43.206 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:43.206 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.206 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.206 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:46:43.206 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.202945+0000", 2024-09-13T14:46:43.206 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-09-13T14:46:43.206 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.207 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.207 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:46:43.207 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.202955+0000", 2024-09-13T14:46:43.207 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.7240000000000004e-06 2024-09-13T14:46:43.207 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.207 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.207 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:46:43.207 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.202975+0000", 2024-09-13T14:46:43.207 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0346000000000001e-05 2024-09-13T14:46:43.207 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.207 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.207 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:46:43.207 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.202994+0000", 2024-09-13T14:46:43.207 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8763999999999999e-05 2024-09-13T14:46:43.207 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.208 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.208 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:43.208 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.203058+0000", 2024-09-13T14:46:43.208 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4003000000000004e-05 2024-09-13T14:46:43.208 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.208 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.208 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:46:43.208 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.203071+0000", 2024-09-13T14:46:43.208 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2992e-05 2024-09-13T14:46:43.208 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.208 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.208 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:43.208 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.650083+0000", 2024-09-13T14:46:43.209 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.44701179600000002 2024-09-13T14:46:43.209 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.209 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.209 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:46:43.209 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.650203+0000", 2024-09-13T14:46:43.209 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000120798 2024-09-13T14:46:43.209 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.209 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.209 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:46:43.209 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.650292+0000", 2024-09-13T14:46:43.209 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8274000000000002e-05 2024-09-13T14:46:43.209 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.209 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.210 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:43.210 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.653268+0000", 2024-09-13T14:46:43.210 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0029767320000000002 2024-09-13T14:46:43.210 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.210 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.210 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:46:43.210 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.653292+0000", 2024-09-13T14:46:43.210 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3147e-05 2024-09-13T14:46:43.210 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.210 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.210 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:43.210 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.654125+0000", 2024-09-13T14:46:43.210 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00083319100000000005 2024-09-13T14:46:43.210 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.211 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.211 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:46:43.211 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.654150+0000", 2024-09-13T14:46:43.211 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5219999999999999e-05 2024-09-13T14:46:43.211 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.211 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.211 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:46:43.211 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.654170+0000", 2024-09-13T14:46:43.211 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9921000000000001e-05 2024-09-13T14:46:43.211 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:43.211 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:46:43.211 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:43.211 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.211 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.212 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.46157.0:45931 8.0 e314/304 8:02787af3:::benchmark_data_smithi102_313275_object45930:head v 314'2879, mlcod=314'2877)", 2024-09-13T14:46:43.212 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:46:32.805701+0000", 2024-09-13T14:46:43.212 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 10.346547124000001, 2024-09-13T14:46:43.212 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.15992261799999999, 2024-09-13T14:46:43.212 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:46:43.212 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:43.212 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:46:43.212 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.212 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:46:43.212 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.805701+0000", 2024-09-13T14:46:43.212 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:43.212 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.212 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.212 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:46:43.213 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.805701+0000", 2024-09-13T14:46:43.213 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:43.213 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.213 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.213 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:46:43.213 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.805701+0000", 2024-09-13T14:46:43.213 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:46:43.213 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.213 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.213 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:46:43.213 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.805750+0000", 2024-09-13T14:46:43.213 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9530000000000002e-05 2024-09-13T14:46:43.213 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.214 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.214 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:46:43.214 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.805759+0000", 2024-09-13T14:46:43.214 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5029999999999997e-06 2024-09-13T14:46:43.214 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.214 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.214 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:46:43.214 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.805763+0000", 2024-09-13T14:46:43.214 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.267e-06 2024-09-13T14:46:43.214 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.214 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.214 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:43.214 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.805801+0000", 2024-09-13T14:46:43.214 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8019000000000003e-05 2024-09-13T14:46:43.215 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.215 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.215 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:46:43.215 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.805824+0000", 2024-09-13T14:46:43.215 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2903000000000001e-05 2024-09-13T14:46:43.215 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.215 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.215 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:46:43.215 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.808576+0000", 2024-09-13T14:46:43.215 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0027518109999999998 2024-09-13T14:46:43.215 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.215 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.215 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:46:43.216 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.965624+0000", 2024-09-13T14:46:43.216 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.15704854100000001 2024-09-13T14:46:43.216 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:43.216 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:46:43.216 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:43.216 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.216 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.216 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.46157.0:45932 8.0 e314/304 8:07864aaa:::benchmark_data_smithi102_313275_object45931:head v 314'2880, mlcod=314'2877)", 2024-09-13T14:46:43.216 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:46:32.806942+0000", 2024-09-13T14:46:43.216 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 10.345306559999999, 2024-09-13T14:46:43.216 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.15872799200000001, 2024-09-13T14:46:43.216 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:46:43.216 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:43.216 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:46:43.217 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.217 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:46:43.217 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.806942+0000", 2024-09-13T14:46:43.217 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:43.217 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.217 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.217 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:46:43.217 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.806942+0000", 2024-09-13T14:46:43.217 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:43.217 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.217 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.217 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:46:43.217 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.806941+0000", 2024-09-13T14:46:43.217 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:46:43.218 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.218 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.218 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:46:43.218 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.807015+0000", 2024-09-13T14:46:43.218 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3687000000000001e-05 2024-09-13T14:46:43.218 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.218 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.218 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:46:43.218 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.807026+0000", 2024-09-13T14:46:43.218 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0522e-05 2024-09-13T14:46:43.218 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.218 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.218 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:46:43.219 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.807032+0000", 2024-09-13T14:46:43.219 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6080000000000001e-06 2024-09-13T14:46:43.219 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.219 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.219 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:43.219 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.807142+0000", 2024-09-13T14:46:43.219 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000109551 2024-09-13T14:46:43.219 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.219 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.219 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:46:43.219 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.807161+0000", 2024-09-13T14:46:43.219 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9686000000000001e-05 2024-09-13T14:46:43.219 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.219 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.220 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:46:43.220 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.965640+0000", 2024-09-13T14:46:43.220 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.15847905700000001 2024-09-13T14:46:43.220 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.220 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.220 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:46:43.220 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.965670+0000", 2024-09-13T14:46:43.220 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9620000000000001e-05 2024-09-13T14:46:43.220 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:43.220 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:46:43.220 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:43.220 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.220 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.220 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.46157.0:45934 8.4 e314/306 8:2e076b64:::benchmark_data_smithi102_313275_object45933:head v 314'2828, mlcod=314'2825)", 2024-09-13T14:46:43.220 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:46:32.807706+0000", 2024-09-13T14:46:43.221 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 10.344542604000001, 2024-09-13T14:46:43.221 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.159780378, 2024-09-13T14:46:43.221 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:46:43.221 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:43.221 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:46:43.221 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.221 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:46:43.221 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.807706+0000", 2024-09-13T14:46:43.221 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:43.221 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.221 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.221 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:46:43.221 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.807706+0000", 2024-09-13T14:46:43.221 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:43.222 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.222 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.222 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:46:43.222 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.807705+0000", 2024-09-13T14:46:43.222 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:46:43.222 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.222 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.222 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:46:43.222 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.807729+0000", 2024-09-13T14:46:43.222 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3983000000000001e-05 2024-09-13T14:46:43.222 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.222 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.222 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:46:43.222 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.807740+0000", 2024-09-13T14:46:43.222 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0611000000000001e-05 2024-09-13T14:46:43.223 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.223 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.223 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:46:43.223 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.807746+0000", 2024-09-13T14:46:43.223 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0900000000000001e-06 2024-09-13T14:46:43.223 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.223 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.223 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:43.223 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.807805+0000", 2024-09-13T14:46:43.223 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.9122000000000001e-05 2024-09-13T14:46:43.223 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.223 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.223 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:46:43.224 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.807825+0000", 2024-09-13T14:46:43.224 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0035999999999999e-05 2024-09-13T14:46:43.224 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.224 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.224 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:46:43.224 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.967381+0000", 2024-09-13T14:46:43.224 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.159555902 2024-09-13T14:46:43.224 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.224 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.224 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:46:43.224 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.967486+0000", 2024-09-13T14:46:43.224 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000105312 2024-09-13T14:46:43.224 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:43.224 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:46:43.225 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:43.225 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.225 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.225 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.46157.0:45938 8.2 8:499f5c03:::benchmark_data_smithi102_313275_object45937:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-09-13T14:46:43.225 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:46:32.808980+0000", 2024-09-13T14:46:43.225 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 10.343268304, 2024-09-13T14:46:43.225 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.163339441, 2024-09-13T14:46:43.225 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:46:43.225 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:43.225 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:46:43.225 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.46157", 2024-09-13T14:46:43.225 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:46:43.225 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 45938 2024-09-13T14:46:43.225 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.226 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:46:43.226 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.226 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:46:43.226 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.808980+0000", 2024-09-13T14:46:43.226 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:43.226 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.226 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.226 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:46:43.226 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.808980+0000", 2024-09-13T14:46:43.226 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:43.226 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.226 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.226 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:46:43.226 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.808979+0000", 2024-09-13T14:46:43.227 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:46:43.227 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.227 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.227 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:46:43.227 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.809033+0000", 2024-09-13T14:46:43.227 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4489e-05 2024-09-13T14:46:43.227 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.227 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.227 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:46:43.227 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.965592+0000", 2024-09-13T14:46:43.227 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.156558744 2024-09-13T14:46:43.227 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.227 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.228 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:46:43.228 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.965605+0000", 2024-09-13T14:46:43.228 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2723e-05 2024-09-13T14:46:43.228 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.228 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.228 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:43.228 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.965674+0000", 2024-09-13T14:46:43.228 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9072999999999994e-05 2024-09-13T14:46:43.228 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.228 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.228 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:46:43.228 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.965811+0000", 2024-09-13T14:46:43.228 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013785800000000001 2024-09-13T14:46:43.229 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.229 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.229 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:46:43.229 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.965909+0000", 2024-09-13T14:46:43.229 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.7312999999999995e-05 2024-09-13T14:46:43.229 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.229 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.229 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:43.229 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.969910+0000", 2024-09-13T14:46:43.229 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0040011420000000001 2024-09-13T14:46:43.229 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.229 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.229 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:46:43.229 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.969932+0000", 2024-09-13T14:46:43.230 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2518999999999999e-05 2024-09-13T14:46:43.230 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:43.230 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.231 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.231 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:43.231 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.972243+0000", 2024-09-13T14:46:43.231 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0023110589999999999 2024-09-13T14:46:43.231 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.231 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.231 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:46:43.231 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.972288+0000", 2024-09-13T14:46:43.231 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4171000000000001e-05 2024-09-13T14:46:43.231 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.231 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.231 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:46:43.231 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.972320+0000", 2024-09-13T14:46:43.231 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2095999999999999e-05 2024-09-13T14:46:43.232 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:43.232 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:46:43.232 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:43.232 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.232 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.232 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.46157.0:45937 8.4 e314/306 8:2900bfef:::benchmark_data_smithi102_313275_object45936:head v 314'2829, mlcod=314'2826)", 2024-09-13T14:46:43.232 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:46:32.809611+0000", 2024-09-13T14:46:43.232 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 10.342638082000001, 2024-09-13T14:46:43.232 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.16101005099999999, 2024-09-13T14:46:43.232 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:46:43.232 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:43.232 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:46:43.232 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.232 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:46:43.233 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.809611+0000", 2024-09-13T14:46:43.233 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:43.233 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.233 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.233 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:46:43.233 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.809611+0000", 2024-09-13T14:46:43.233 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:43.233 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.233 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.233 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:46:43.233 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.809610+0000", 2024-09-13T14:46:43.233 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:46:43.233 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.233 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.234 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:46:43.234 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.809630+0000", 2024-09-13T14:46:43.234 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0068000000000002e-05 2024-09-13T14:46:43.234 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.234 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.234 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:46:43.234 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.965549+0000", 2024-09-13T14:46:43.234 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.155919481 2024-09-13T14:46:43.234 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.234 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.234 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:46:43.234 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.965588+0000", 2024-09-13T14:46:43.234 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9085999999999998e-05 2024-09-13T14:46:43.234 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.234 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.235 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:43.235 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.965647+0000", 2024-09-13T14:46:43.235 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8217999999999997e-05 2024-09-13T14:46:43.235 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.235 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.235 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:46:43.235 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.965674+0000", 2024-09-13T14:46:43.235 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7835e-05 2024-09-13T14:46:43.235 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.235 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.235 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:46:43.235 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.970579+0000", 2024-09-13T14:46:43.236 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0049042979999999996 2024-09-13T14:46:43.236 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.236 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.236 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:46:43.236 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.970621+0000", 2024-09-13T14:46:43.236 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1922999999999998e-05 2024-09-13T14:46:43.236 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:43.236 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:46:43.236 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:43.236 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.236 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.236 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.46157.0:46736 8.2 8:420125c2:::benchmark_data_smithi102_313275_object46735:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-09-13T14:46:43.237 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:46:33.497372+0000", 2024-09-13T14:46:43.237 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 9.6548767140000002, 2024-09-13T14:46:43.237 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.072670230000000002, 2024-09-13T14:46:43.237 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:46:43.237 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:43.237 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:46:43.237 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.46157", 2024-09-13T14:46:43.237 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:46:43.237 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 46736 2024-09-13T14:46:43.237 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.237 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:46:43.237 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.237 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:46:43.237 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:33.497372+0000", 2024-09-13T14:46:43.238 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:43.238 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.238 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.238 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:46:43.238 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:33.497372+0000", 2024-09-13T14:46:43.238 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:43.238 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.238 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.238 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:46:43.238 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:33.497371+0000", 2024-09-13T14:46:43.238 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:46:43.238 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.238 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.238 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:46:43.238 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:33.497404+0000", 2024-09-13T14:46:43.239 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2193999999999997e-05 2024-09-13T14:46:43.239 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.239 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.239 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:46:43.239 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:33.497413+0000", 2024-09-13T14:46:43.239 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.9609999999999994e-06 2024-09-13T14:46:43.239 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.239 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.239 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:46:43.239 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:33.497418+0000", 2024-09-13T14:46:43.239 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0699999999999997e-06 2024-09-13T14:46:43.239 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.239 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.240 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:43.240 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:33.497671+0000", 2024-09-13T14:46:43.240 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00025354300000000001 2024-09-13T14:46:43.240 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.240 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.240 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:46:43.240 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:33.497788+0000", 2024-09-13T14:46:43.240 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000116521 2024-09-13T14:46:43.240 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.240 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.240 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:46:43.240 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:33.497924+0000", 2024-09-13T14:46:43.240 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013596699999999999 2024-09-13T14:46:43.240 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.240 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.241 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:43.241 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:33.500991+0000", 2024-09-13T14:46:43.241 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0030670680000000001 2024-09-13T14:46:43.241 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.241 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.241 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:46:43.241 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:33.501821+0000", 2024-09-13T14:46:43.241 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00083022900000000004 2024-09-13T14:46:43.241 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.241 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.241 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:43.241 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:33.569937+0000", 2024-09-13T14:46:43.241 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.068115776000000003 2024-09-13T14:46:43.242 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.242 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.242 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:46:43.242 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:33.569999+0000", 2024-09-13T14:46:43.242 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2688999999999996e-05 2024-09-13T14:46:43.242 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.242 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.242 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:46:43.242 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:33.570042+0000", 2024-09-13T14:46:43.242 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2756000000000002e-05 2024-09-13T14:46:43.242 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:43.242 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:46:43.242 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:43.242 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.242 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.243 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.46157.0:50945 8.2 8:484d7ea7:::benchmark_data_smithi102_313275_object50944:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-13T14:46:43.243 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:46:35.885752+0000", 2024-09-13T14:46:43.243 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 7.2664967779999996, 2024-09-13T14:46:43.243 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.070647009999999996, 2024-09-13T14:46:43.243 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:46:43.243 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:43.243 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:46:43.243 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.46157", 2024-09-13T14:46:43.243 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:46:43.243 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 50945 2024-09-13T14:46:43.243 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.243 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:46:43.243 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.243 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:46:43.243 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:35.885752+0000", 2024-09-13T14:46:43.244 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:43.244 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.244 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.244 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:46:43.244 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:35.885752+0000", 2024-09-13T14:46:43.244 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:43.244 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.244 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:43 smithi195 ceph-mon[104262]: pgmap v1109: 99 pgs: 10 unknown, 89 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-09-13T14:46:43.245 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.245 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:46:43.245 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:35.885751+0000", 2024-09-13T14:46:43.245 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:46:43.245 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.245 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.245 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:46:43.245 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:35.885786+0000", 2024-09-13T14:46:43.245 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4459000000000003e-05 2024-09-13T14:46:43.245 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.246 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.246 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:46:43.246 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:35.885794+0000", 2024-09-13T14:46:43.246 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.4519999999999997e-06 2024-09-13T14:46:43.246 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.246 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.246 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:46:43.246 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:35.885800+0000", 2024-09-13T14:46:43.246 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.993e-06 2024-09-13T14:46:43.246 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.246 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.246 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:43.246 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:35.888087+0000", 2024-09-13T14:46:43.246 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0022864750000000001 2024-09-13T14:46:43.246 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.247 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.247 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:46:43.247 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:35.888238+0000", 2024-09-13T14:46:43.247 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015167300000000001 2024-09-13T14:46:43.247 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.247 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.247 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:46:43.247 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:35.888366+0000", 2024-09-13T14:46:43.247 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000127396 2024-09-13T14:46:43.247 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.247 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.247 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:46:43.247 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:35.894471+0000", 2024-09-13T14:46:43.247 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0061055650000000003 2024-09-13T14:46:43.248 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.248 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.248 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:43.248 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:35.894514+0000", 2024-09-13T14:46:43.248 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3031999999999997e-05 2024-09-13T14:46:43.248 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.248 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.248 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:43.248 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:35.956289+0000", 2024-09-13T14:46:43.248 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.061774849 2024-09-13T14:46:43.248 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.248 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.248 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:46:43.248 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:35.956338+0000", 2024-09-13T14:46:43.249 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9215999999999998e-05 2024-09-13T14:46:43.249 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.249 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.249 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:46:43.249 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:35.956399+0000", 2024-09-13T14:46:43.249 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0509e-05 2024-09-13T14:46:43.249 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:43.249 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:46:43.249 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:43.249 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.249 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.249 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.46157.0:50957 8.2 8:4f82215b:::benchmark_data_smithi102_313275_object50956:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-13T14:46:43.249 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:46:35.889880+0000", 2024-09-13T14:46:43.249 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 7.2623685939999998, 2024-09-13T14:46:43.250 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.066747652000000005, 2024-09-13T14:46:43.250 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:46:43.250 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:43.250 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:46:43.250 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.46157", 2024-09-13T14:46:43.250 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:46:43.250 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 50957 2024-09-13T14:46:43.250 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.250 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:46:43.250 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.250 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:46:43.250 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:35.889880+0000", 2024-09-13T14:46:43.250 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:43.251 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.251 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.251 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:46:43.251 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:35.889880+0000", 2024-09-13T14:46:43.251 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:43.251 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.251 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.251 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:46:43.251 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:35.889880+0000", 2024-09-13T14:46:43.251 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:46:43.251 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.251 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.251 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:46:43.251 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:35.889907+0000", 2024-09-13T14:46:43.252 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.6928e-05 2024-09-13T14:46:43.252 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.252 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.252 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:46:43.252 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:35.889921+0000", 2024-09-13T14:46:43.252 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4185000000000001e-05 2024-09-13T14:46:43.252 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.252 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.252 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:46:43.252 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:35.889926+0000", 2024-09-13T14:46:43.252 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.755e-06 2024-09-13T14:46:43.252 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.252 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.252 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:43.253 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:35.892037+0000", 2024-09-13T14:46:43.253 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0021105669999999998 2024-09-13T14:46:43.253 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.253 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.253 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:46:43.253 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:35.892147+0000", 2024-09-13T14:46:43.253 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000110444 2024-09-13T14:46:43.253 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.253 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.253 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:46:43.253 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:35.892270+0000", 2024-09-13T14:46:43.253 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012229299999999999 2024-09-13T14:46:43.253 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.253 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.253 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:46:43.254 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:35.894484+0000", 2024-09-13T14:46:43.254 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002214714 2024-09-13T14:46:43.254 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.254 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.254 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:43.254 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:35.895303+0000", 2024-09-13T14:46:43.254 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000818094 2024-09-13T14:46:43.254 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.254 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.254 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:43.254 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:35.956563+0000", 2024-09-13T14:46:43.254 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.061260862999999999 2024-09-13T14:46:43.254 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.254 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.255 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:46:43.255 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:35.956600+0000", 2024-09-13T14:46:43.255 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6310000000000003e-05 2024-09-13T14:46:43.255 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.255 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.255 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:46:43.255 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:35.956628+0000", 2024-09-13T14:46:43.255 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7932e-05 2024-09-13T14:46:43.255 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:43.255 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:46:43.255 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:43.255 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.255 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.255 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.46157.0:55619 8.5 8:a95080a8:::benchmark_data_smithi102_313275_object55618:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-13T14:46:43.256 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:46:38.946153+0000", 2024-09-13T14:46:43.256 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 4.2060955389999997, 2024-09-13T14:46:43.256 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.277270038, 2024-09-13T14:46:43.256 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:46:43.256 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:43.256 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:46:43.256 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.46157", 2024-09-13T14:46:43.256 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:46:43.256 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 55619 2024-09-13T14:46:43.256 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.256 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:46:43.256 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.256 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:46:43.256 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.946153+0000", 2024-09-13T14:46:43.256 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:43.256 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.257 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.257 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:46:43.257 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.946153+0000", 2024-09-13T14:46:43.257 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:43.257 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.257 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.257 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:46:43.257 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.946152+0000", 2024-09-13T14:46:43.257 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:46:43.257 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.257 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.257 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:46:43.257 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.946186+0000", 2024-09-13T14:46:43.257 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4739000000000001e-05 2024-09-13T14:46:43.258 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.258 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.258 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:46:43.258 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.946203+0000", 2024-09-13T14:46:43.258 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6977999999999999e-05 2024-09-13T14:46:43.258 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.258 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.258 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:46:43.258 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.946211+0000", 2024-09-13T14:46:43.258 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4410000000000004e-06 2024-09-13T14:46:43.258 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.258 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.258 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:43.258 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.946263+0000", 2024-09-13T14:46:43.259 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2694000000000001e-05 2024-09-13T14:46:43.259 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.259 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.259 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:46:43.259 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.946417+0000", 2024-09-13T14:46:43.259 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000153776 2024-09-13T14:46:43.259 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.259 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.259 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:46:43.259 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.946577+0000", 2024-09-13T14:46:43.259 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000159853 2024-09-13T14:46:43.259 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.259 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.259 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:46:43.260 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.948704+0000", 2024-09-13T14:46:43.260 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0021266850000000001 2024-09-13T14:46:43.260 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.260 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.260 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:43.260 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.952101+0000", 2024-09-13T14:46:43.260 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0033971349999999999 2024-09-13T14:46:43.260 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.260 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.260 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:43.260 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.223350+0000", 2024-09-13T14:46:43.260 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.27124903099999997 2024-09-13T14:46:43.260 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.260 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.261 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:46:43.261 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.223391+0000", 2024-09-13T14:46:43.261 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1205000000000003e-05 2024-09-13T14:46:43.261 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.261 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.261 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:46:43.261 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.223423+0000", 2024-09-13T14:46:43.261 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.1973e-05 2024-09-13T14:46:43.261 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:43.261 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:46:43.261 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:43.261 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.261 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.261 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.46157.0:55632 8.5 8:aee4ccb2:::benchmark_data_smithi102_313275_object55631:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-13T14:46:43.262 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:46:38.952877+0000", 2024-09-13T14:46:43.262 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 4.1993713880000003, 2024-09-13T14:46:43.262 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.27064351800000003, 2024-09-13T14:46:43.262 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:46:43.262 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:43.262 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:46:43.262 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.46157", 2024-09-13T14:46:43.262 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:46:43.262 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 55632 2024-09-13T14:46:43.262 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.262 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:46:43.262 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.262 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:46:43.262 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.952877+0000", 2024-09-13T14:46:43.262 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:43.263 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.263 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.263 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:46:43.263 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.952877+0000", 2024-09-13T14:46:43.263 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:43.263 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.263 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.263 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:46:43.263 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.952876+0000", 2024-09-13T14:46:43.263 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:46:43.263 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.263 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.263 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:46:43.263 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.952942+0000", 2024-09-13T14:46:43.263 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6246000000000002e-05 2024-09-13T14:46:43.264 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.264 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.264 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:46:43.264 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.952954+0000", 2024-09-13T14:46:43.264 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2326e-05 2024-09-13T14:46:43.264 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.264 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.264 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:46:43.264 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.952960+0000", 2024-09-13T14:46:43.264 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.524e-06 2024-09-13T14:46:43.264 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.264 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.264 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:43.264 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.952995+0000", 2024-09-13T14:46:43.264 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5221e-05 2024-09-13T14:46:43.265 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.265 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.265 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:46:43.265 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.953112+0000", 2024-09-13T14:46:43.265 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011763499999999999 2024-09-13T14:46:43.265 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.265 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.265 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:46:43.265 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.953267+0000", 2024-09-13T14:46:43.265 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015489099999999999 2024-09-13T14:46:43.265 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.265 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.265 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:46:43.265 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.955476+0000", 2024-09-13T14:46:43.266 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0022090769999999998 2024-09-13T14:46:43.266 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.266 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.266 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:43.266 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.220059+0000", 2024-09-13T14:46:43.266 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.26458301299999998 2024-09-13T14:46:43.266 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.266 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.266 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:43.266 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.223455+0000", 2024-09-13T14:46:43.266 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0033958970000000002 2024-09-13T14:46:43.266 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.266 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.266 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:46:43.267 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.223493+0000", 2024-09-13T14:46:43.267 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7351999999999997e-05 2024-09-13T14:46:43.267 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.267 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.267 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:46:43.267 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.223521+0000", 2024-09-13T14:46:43.267 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8059999999999999e-05 2024-09-13T14:46:43.267 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:43.267 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:46:43.267 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:43.267 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.267 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.267 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.46157.0:55645 8.5 8:ad2ba30b:::benchmark_data_smithi102_313275_object55644:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-13T14:46:43.267 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:46:38.961528+0000", 2024-09-13T14:46:43.267 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 4.1907206639999997, 2024-09-13T14:46:43.268 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.26211107900000002, 2024-09-13T14:46:43.268 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:46:43.268 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:43.268 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:46:43.268 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.46157", 2024-09-13T14:46:43.268 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:46:43.268 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 55645 2024-09-13T14:46:43.268 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.268 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:46:43.268 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.268 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:46:43.268 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.961528+0000", 2024-09-13T14:46:43.268 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:43.268 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.268 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.269 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:46:43.269 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.961528+0000", 2024-09-13T14:46:43.269 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:43.269 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.269 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.269 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:46:43.269 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.961527+0000", 2024-09-13T14:46:43.269 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:46:43.269 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.269 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.269 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:46:43.269 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.961580+0000", 2024-09-13T14:46:43.269 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3350000000000003e-05 2024-09-13T14:46:43.269 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.269 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.270 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:46:43.270 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.961591+0000", 2024-09-13T14:46:43.270 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0502e-05 2024-09-13T14:46:43.270 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.270 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.270 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:46:43.270 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.961596+0000", 2024-09-13T14:46:43.270 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9760000000000003e-06 2024-09-13T14:46:43.270 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.270 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.270 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:43.270 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.961628+0000", 2024-09-13T14:46:43.270 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2699e-05 2024-09-13T14:46:43.271 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.271 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.271 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:46:43.271 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.961730+0000", 2024-09-13T14:46:43.271 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010173700000000001 2024-09-13T14:46:43.271 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.271 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.271 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:46:43.271 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.961813+0000", 2024-09-13T14:46:43.271 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3072999999999996e-05 2024-09-13T14:46:43.271 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.271 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.271 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:46:43.271 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.963732+0000", 2024-09-13T14:46:43.272 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001919254 2024-09-13T14:46:43.272 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.272 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.272 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:43.272 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.223290+0000", 2024-09-13T14:46:43.272 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.25955769699999998 2024-09-13T14:46:43.272 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.272 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.272 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:43.272 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.223575+0000", 2024-09-13T14:46:43.272 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00028493300000000002 2024-09-13T14:46:43.272 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.272 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.273 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:46:43.273 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.223612+0000", 2024-09-13T14:46:43.273 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6838999999999997e-05 2024-09-13T14:46:43.273 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.273 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.273 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:46:43.273 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.223639+0000", 2024-09-13T14:46:43.273 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7206999999999999e-05 2024-09-13T14:46:43.273 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:43.273 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:46:43.273 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:43.273 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.273 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.273 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.46157.0:55650 8.5 8:a9a2aa34:::benchmark_data_smithi102_313275_object55649:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-13T14:46:43.274 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:46:38.964767+0000", 2024-09-13T14:46:43.274 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 4.1874813079999997, 2024-09-13T14:46:43.274 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.25903844199999998, 2024-09-13T14:46:43.274 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:46:43.274 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:43.274 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:46:43.274 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.46157", 2024-09-13T14:46:43.274 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:46:43.274 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 55650 2024-09-13T14:46:43.274 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.274 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:46:43.274 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.274 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:46:43.274 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.964767+0000", 2024-09-13T14:46:43.274 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:43.275 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.275 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.275 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:46:43.275 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.964767+0000", 2024-09-13T14:46:43.275 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:43.275 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.275 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.275 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:46:43.275 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.964766+0000", 2024-09-13T14:46:43.275 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:46:43.275 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.275 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.275 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:46:43.275 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.964853+0000", 2024-09-13T14:46:43.275 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.6948999999999997e-05 2024-09-13T14:46:43.276 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.276 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.276 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:46:43.276 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.964863+0000", 2024-09-13T14:46:43.276 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0084e-05 2024-09-13T14:46:43.276 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.276 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.276 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:46:43.276 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.964868+0000", 2024-09-13T14:46:43.276 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8550000000000001e-06 2024-09-13T14:46:43.276 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.276 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.276 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:43.276 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.964899+0000", 2024-09-13T14:46:43.277 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.1343000000000001e-05 2024-09-13T14:46:43.277 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.277 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.277 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:46:43.277 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.964997+0000", 2024-09-13T14:46:43.277 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.7309e-05 2024-09-13T14:46:43.277 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.277 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.277 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:46:43.277 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.965090+0000", 2024-09-13T14:46:43.277 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.3550999999999999e-05 2024-09-13T14:46:43.277 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.277 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.278 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:46:43.278 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.966569+0000", 2024-09-13T14:46:43.278 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0014783999999999999 2024-09-13T14:46:43.278 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.278 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.278 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:43.278 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.223550+0000", 2024-09-13T14:46:43.278 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.256981287 2024-09-13T14:46:43.278 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.278 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.278 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:43.278 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.223729+0000", 2024-09-13T14:46:43.278 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00017951800000000001 2024-09-13T14:46:43.278 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.278 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.279 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:46:43.279 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.223775+0000", 2024-09-13T14:46:43.279 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5061000000000001e-05 2024-09-13T14:46:43.279 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.279 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.279 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:46:43.279 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.223806+0000", 2024-09-13T14:46:43.279 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.1207999999999997e-05 2024-09-13T14:46:43.279 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:43.279 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:46:43.279 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:43.279 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.279 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.280 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.46157.0:56494 8.2 8:47377839:::benchmark_data_smithi102_313275_object56493:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e317)", 2024-09-13T14:46:43.280 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:46:40.202789+0000", 2024-09-13T14:46:43.280 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 2.9494591890000001, 2024-09-13T14:46:43.280 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.122189434, 2024-09-13T14:46:43.280 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:46:43.280 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:43.280 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:46:43.280 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.46157", 2024-09-13T14:46:43.280 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:46:43.280 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 56494 2024-09-13T14:46:43.280 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.280 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:46:43.280 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.280 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:46:43.281 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:40.202789+0000", 2024-09-13T14:46:43.281 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:43.281 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.281 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.281 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:46:43.281 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:40.202789+0000", 2024-09-13T14:46:43.281 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:43.281 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.281 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.281 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:46:43.281 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:40.202788+0000", 2024-09-13T14:46:43.281 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:46:43.281 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.282 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.282 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:46:43.282 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:40.202835+0000", 2024-09-13T14:46:43.282 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7049000000000003e-05 2024-09-13T14:46:43.282 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.282 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.282 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:46:43.282 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:40.202846+0000", 2024-09-13T14:46:43.282 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0902e-05 2024-09-13T14:46:43.282 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.282 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.282 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:46:43.282 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:40.202851+0000", 2024-09-13T14:46:43.283 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4890000000000003e-06 2024-09-13T14:46:43.283 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.283 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.283 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:43.283 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:40.202887+0000", 2024-09-13T14:46:43.283 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5729000000000003e-05 2024-09-13T14:46:43.283 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.283 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.283 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:46:43.283 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:40.202990+0000", 2024-09-13T14:46:43.283 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000103294 2024-09-13T14:46:43.283 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.283 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.283 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:46:43.284 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:40.203073+0000", 2024-09-13T14:46:43.284 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3287000000000004e-05 2024-09-13T14:46:43.284 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.284 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.284 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:46:43.284 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:40.204842+0000", 2024-09-13T14:46:43.284 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0017689909999999999 2024-09-13T14:46:43.284 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.284 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.284 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:43.284 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:40.205104+0000", 2024-09-13T14:46:43.284 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00026179099999999999 2024-09-13T14:46:43.285 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.285 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.285 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:43.285 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:40.324902+0000", 2024-09-13T14:46:43.285 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.119798214 2024-09-13T14:46:43.285 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.285 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.285 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:46:43.285 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:40.324954+0000", 2024-09-13T14:46:43.285 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1456999999999999e-05 2024-09-13T14:46:43.285 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:43.286 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:43.286 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:46:43.286 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:40.324979+0000", 2024-09-13T14:46:43.286 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5007999999999998e-05 2024-09-13T14:46:43.286 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:43.286 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:46:43.286 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:43.286 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:43.286 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:46:43.286 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:46:43.317 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:43.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:43.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:43.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:43 smithi102 ceph-mon[110183]: pgmap v1109: 99 pgs: 10 unknown, 89 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-09-13T14:46:43.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:43 smithi102 ceph-mon[115269]: pgmap v1109: 99 pgs: 10 unknown, 89 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-09-13T14:46:43.419 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:43.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:43.454 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:43.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:43.554 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:43.597 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:43.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:43.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:43.699 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:43.723 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:43.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:43.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:43.803 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 30 16 61889 61873 128.878 81 0.00821596 0.0077544 2024-09-13T14:46:43.825 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:43.857 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:43.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:43.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:43.959 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:43.997 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:43.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:43.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:44.098 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:44.130 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:44.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:44.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:44.232 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:44.270 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:44.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:44.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:44.297 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T14:46:44.367 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:44 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-13T14:46:44.372 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:44.407 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:44.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:44.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:44.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:44 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-13T14:46:44.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:44 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-13T14:46:44.509 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:44.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:44.548 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:44.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:44.649 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:44.664 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:46:44.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:44.691 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:44.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:44.791 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:44.804 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 31 16 63371 63355 127.705 92.625 0.0103061 0.00782578 2024-09-13T14:46:44.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:44.844 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:44.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:44.945 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:44.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:44.982 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:44.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:45.032 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:46:45.032 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:46:45.045 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:45.068 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:46:45.068 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:46:45.083 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:45.101 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:46:45.101 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:46:45.125 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:45.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:45.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:45.140 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:46:45.140 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:46:45.170 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:46:45.170 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:46:45.200 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:46:45.200 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:46:45.228 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:45.236 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:46:45.236 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:46:45.269 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:46:45.270 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:46:45.270 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:45.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:45.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:45.360 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:45 smithi102 ceph-mon[110183]: pgmap v1110: 99 pgs: 99 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-09-13T14:46:45.361 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:45 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-13T14:46:45.361 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:45 smithi102 ceph-mon[110183]: osdmap e319: 8 total, 8 up, 7 in 2024-09-13T14:46:45.361 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:45 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:46:45.361 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:45 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:46:45.361 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:45 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/658846980' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:46:45.361 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:45 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:46:45.361 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:45 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:46:45.361 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:45 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:46:45.361 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:45 smithi102 ceph-mon[115269]: pgmap v1110: 99 pgs: 99 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-09-13T14:46:45.361 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:45 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-13T14:46:45.362 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:45 smithi102 ceph-mon[115269]: osdmap e319: 8 total, 8 up, 7 in 2024-09-13T14:46:45.362 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:45 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:46:45.362 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:45 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:46:45.362 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:45 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/658846980' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:46:45.362 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:45 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:46:45.362 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:45 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:46:45.362 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:45 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:46:45.371 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:45.407 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:45.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:45.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:45.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:45 smithi195 ceph-mon[104262]: pgmap v1110: 99 pgs: 99 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-09-13T14:46:45.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:45 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-13T14:46:45.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:45 smithi195 ceph-mon[104262]: osdmap e319: 8 total, 8 up, 7 in 2024-09-13T14:46:45.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:45 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:46:45.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:45 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:46:45.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:45 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/658846980' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:46:45.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:45 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:46:45.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:45 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:46:45.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:45 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:46:45.509 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:45.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:45.549 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:45.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:45.652 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:45.688 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:45.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:45.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:45.790 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:45.804 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 32 16 64799 64783 126.503 89.25 0.0089075 0.00789994 2024-09-13T14:46:45.823 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:45.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:45.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:45.925 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:45.957 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:45.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:45.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:46.059 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:46.096 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:46.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:46.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:46.198 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:46.238 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:46.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:46.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:46.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:46 smithi195 ceph-mon[104262]: osdmap e320: 8 total, 8 up, 7 in 2024-09-13T14:46:46.341 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:46.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:46.387 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:46.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:46.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:46 smithi102 ceph-mon[110183]: osdmap e320: 8 total, 8 up, 7 in 2024-09-13T14:46:46.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:46 smithi102 ceph-mon[115269]: osdmap e320: 8 total, 8 up, 7 in 2024-09-13T14:46:46.488 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:46.525 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:46.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:46.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:46.626 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:46.658 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:46.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:46.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:46.726 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:46:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:46:46] "GET /metrics HTTP/1.1" 200 36523 "" "Prometheus/2.51.0" 2024-09-13T14:46:46.761 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:46.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:46.803 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:46.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:46.805 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 33 16 66274 66258 125.463 92.1875 0.0128698 0.00796509 2024-09-13T14:46:46.904 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:46.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:46.938 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:46.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:47.039 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:47.076 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:47.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:47.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:47.113 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 6, 7, 5, 2] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T14:46:47.113 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:46:47.113 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-09-13T14:46:47.113 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T14:46:47.113 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T14:46:47.113 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-09-13T14:46:47.178 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:47.217 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:47.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:47.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:47.319 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:47.365 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:47.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:47.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:47.375 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:46:47.376 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:46:47.389 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:47 smithi195 ceph-mon[104262]: pgmap v1113: 99 pgs: 99 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-09-13T14:46:47.389 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:47 smithi195 ceph-mon[104262]: osdmap e321: 8 total, 8 up, 7 in 2024-09-13T14:46:47.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:47 smithi102 ceph-mon[110183]: pgmap v1113: 99 pgs: 99 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-09-13T14:46:47.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:47 smithi102 ceph-mon[110183]: osdmap e321: 8 total, 8 up, 7 in 2024-09-13T14:46:47.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:47 smithi102 ceph-mon[115269]: pgmap v1113: 99 pgs: 99 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-09-13T14:46:47.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:47 smithi102 ceph-mon[115269]: osdmap e321: 8 total, 8 up, 7 in 2024-09-13T14:46:47.466 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:47.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:47.505 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:47.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:47.606 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:47.646 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:47.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:47.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:47.747 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:47.783 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:47.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:47.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:47.804 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 34 16 67723 67707 124.437 90.5625 0.0114549 0.00803155 2024-09-13T14:46:47.885 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:47.925 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:47.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:47.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:48.027 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:48.051 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T14:46:48.065 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:48.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:48.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:48.166 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:48.203 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:48.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:48.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:48.305 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:48.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:48.366 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:48.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:48.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:48 smithi102 ceph-mon[115269]: osdmap e322: 8 total, 8 up, 7 in 2024-09-13T14:46:48.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:48 smithi102 ceph-mon[115269]: osdmap e323: 8 total, 8 up, 7 in 2024-09-13T14:46:48.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:48 smithi102 ceph-mon[110183]: osdmap e322: 8 total, 8 up, 7 in 2024-09-13T14:46:48.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:48 smithi102 ceph-mon[110183]: osdmap e323: 8 total, 8 up, 7 in 2024-09-13T14:46:48.435 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:46:48.466 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:48.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:48 smithi195 ceph-mon[104262]: osdmap e322: 8 total, 8 up, 7 in 2024-09-13T14:46:48.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:48 smithi195 ceph-mon[104262]: osdmap e323: 8 total, 8 up, 7 in 2024-09-13T14:46:48.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:48.514 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:48.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:48.602 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:48.615 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:48.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:48.658 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:48.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:48.759 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:48.801 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:48.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:48.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:48.805 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 35 16 69034 69018 123.222 81.9375 0.00939177 0.00811043 2024-09-13T14:46:48.903 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:48.936 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:48.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:48.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:49.038 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:49.072 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:49.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:49.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:49.160 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is unset 2024-09-13T14:46:49.174 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:49.205 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:49.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:49.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:49.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:49 smithi195 ceph-mon[104262]: pgmap v1116: 99 pgs: 6 peering, 93 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s; 64 KiB/s, 0 objects/s recovering 2024-09-13T14:46:49.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:49 smithi195 ceph-mon[104262]: 1.7 scrub starts 2024-09-13T14:46:49.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:49 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/422900398' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:46:49.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:49 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:46:49.307 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:49.345 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:49.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:49.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:49.353 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:46:49.387 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:49 smithi102 ceph-mon[110183]: pgmap v1116: 99 pgs: 6 peering, 93 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s; 64 KiB/s, 0 objects/s recovering 2024-09-13T14:46:49.387 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:49 smithi102 ceph-mon[110183]: 1.7 scrub starts 2024-09-13T14:46:49.387 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:49 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/422900398' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:46:49.387 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:49 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:46:49.387 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:49 smithi102 ceph-mon[115269]: pgmap v1116: 99 pgs: 6 peering, 93 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s; 64 KiB/s, 0 objects/s recovering 2024-09-13T14:46:49.387 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:49 smithi102 ceph-mon[115269]: 1.7 scrub starts 2024-09-13T14:46:49.387 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:49 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/422900398' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:46:49.387 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:49 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:46:49.447 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:49.484 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:49.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:49.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:49.586 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:49.625 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:49.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:49.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:49.727 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:49.767 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:49.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:49.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:49.805 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 36 16 70458 70442 122.271 89 0.0183213 0.00817374 2024-09-13T14:46:49.869 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:49.899 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:49.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:49.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:50.001 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:50.040 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:50.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:50.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:50.142 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:50.183 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:50.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:50.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:50.285 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:50.286 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:50 smithi195 ceph-mon[104262]: 1.6 scrub starts 2024-09-13T14:46:50.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:50 smithi195 ceph-mon[104262]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:46:50.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:50 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:46:50.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:50 smithi195 ceph-mon[104262]: osdmap e324: 8 total, 8 up, 7 in 2024-09-13T14:46:50.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:50 smithi195 ceph-mon[104262]: 1.4 scrub starts 2024-09-13T14:46:50.287 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:50 smithi195 ceph-mon[104262]: 1.4 scrub ok 2024-09-13T14:46:50.319 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:50.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:50.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:50.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:50 smithi102 ceph-mon[110183]: 1.6 scrub starts 2024-09-13T14:46:50.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:50 smithi102 ceph-mon[110183]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:46:50.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:50 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:46:50.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:50 smithi102 ceph-mon[110183]: osdmap e324: 8 total, 8 up, 7 in 2024-09-13T14:46:50.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:50 smithi102 ceph-mon[110183]: 1.4 scrub starts 2024-09-13T14:46:50.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:50 smithi102 ceph-mon[110183]: 1.4 scrub ok 2024-09-13T14:46:50.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:50 smithi102 ceph-mon[115269]: 1.6 scrub starts 2024-09-13T14:46:50.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:50 smithi102 ceph-mon[115269]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:46:50.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:50 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:46:50.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:50 smithi102 ceph-mon[115269]: osdmap e324: 8 total, 8 up, 7 in 2024-09-13T14:46:50.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:50 smithi102 ceph-mon[115269]: 1.4 scrub starts 2024-09-13T14:46:50.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:50 smithi102 ceph-mon[115269]: 1.4 scrub ok 2024-09-13T14:46:50.422 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:50.455 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:46:50.456 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:46:50.460 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:50.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:50.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:50.561 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:50.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:50.605 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:50.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:50.706 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:50.744 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:50.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:50.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:50.805 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 37 16 71841 71825 121.302 86.4375 0.00877727 0.00823903 2024-09-13T14:46:50.846 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:50.880 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:50.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:50.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:50.982 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:51.021 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:51.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:51.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:51.123 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:51.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:51.166 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:51.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:51.266 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:51.303 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:51.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:51.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:51.405 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:51.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:51 smithi102 ceph-mon[110183]: pgmap v1119: 99 pgs: 10 peering, 89 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 112 MiB/s wr, 1.79k op/s; 144 KiB/s, 0 objects/s recovering 2024-09-13T14:46:51.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:51 smithi102 ceph-mon[110183]: 1.3 scrub starts 2024-09-13T14:46:51.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:51 smithi102 ceph-mon[110183]: 1.3 scrub ok 2024-09-13T14:46:51.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:51 smithi102 ceph-mon[115269]: pgmap v1119: 99 pgs: 10 peering, 89 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 112 MiB/s wr, 1.79k op/s; 144 KiB/s, 0 objects/s recovering 2024-09-13T14:46:51.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:51 smithi102 ceph-mon[115269]: 1.3 scrub starts 2024-09-13T14:46:51.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:51 smithi102 ceph-mon[115269]: 1.3 scrub ok 2024-09-13T14:46:51.439 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:51.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:51.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:51.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:51 smithi195 ceph-mon[104262]: pgmap v1119: 99 pgs: 10 peering, 89 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 112 MiB/s wr, 1.79k op/s; 144 KiB/s, 0 objects/s recovering 2024-09-13T14:46:51.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:51 smithi195 ceph-mon[104262]: 1.3 scrub starts 2024-09-13T14:46:51.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:51 smithi195 ceph-mon[104262]: 1.3 scrub ok 2024-09-13T14:46:51.541 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:51.562 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:51.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:51.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:51.664 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:51.690 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:51.705 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:51.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:51.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:51.725 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:46:51.725 INFO:teuthology.orchestra.run.smithi102.stdout: "success": "" 2024-09-13T14:46:51.725 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:46:51.805 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 38 16 73104 73088 120.187 78.9375 0.00690788 0.00831545 2024-09-13T14:46:51.807 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:51.840 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:51.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:51.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:51.942 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:51.978 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:51.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:51.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:52.079 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:52.112 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:52.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:52.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:52.213 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:52.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:52.246 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:52.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:52.347 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:52.376 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:52.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:52.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:52.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:52 smithi102 ceph-mon[110183]: 1.0 scrub starts 2024-09-13T14:46:52.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:52 smithi102 ceph-mon[110183]: 1.0 scrub ok 2024-09-13T14:46:52.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:52 smithi102 ceph-mon[115269]: 1.0 scrub starts 2024-09-13T14:46:52.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:52 smithi102 ceph-mon[115269]: 1.0 scrub ok 2024-09-13T14:46:52.478 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:52.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:52 smithi195 ceph-mon[104262]: 1.0 scrub starts 2024-09-13T14:46:52.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:52 smithi195 ceph-mon[104262]: 1.0 scrub ok 2024-09-13T14:46:52.518 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:52.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:52.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:52.620 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:52.653 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:52.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:52.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:52.754 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:52.790 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:52.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:52.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:52.808 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 39 16 74346 74330 119.096 77.625 0.00643166 0.00839183 2024-09-13T14:46:52.892 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:52.923 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:52.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:52.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:53.025 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:53.060 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:53.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:53.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:53.151 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:46:53.162 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:53.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:53.234 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:53.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:53.335 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:53.377 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:53.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:53.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:53.479 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:53.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:53 smithi195 ceph-mon[104262]: 1.2 scrub starts 2024-09-13T14:46:53.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:53 smithi195 ceph-mon[104262]: 1.2 scrub ok 2024-09-13T14:46:53.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:53 smithi195 ceph-mon[104262]: pgmap v1120: 99 pgs: 10 peering, 89 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 87 MiB/s wr, 1.39k op/s; 106 KiB/s, 0 objects/s recovering 2024-09-13T14:46:53.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:53 smithi195 ceph-mon[104262]: 1.a scrub starts 2024-09-13T14:46:53.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:53 smithi195 ceph-mon[104262]: 1.a scrub ok 2024-09-13T14:46:53.519 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:53.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:53.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:53.524 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:53 smithi102 ceph-mon[115269]: 1.2 scrub starts 2024-09-13T14:46:53.524 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:53 smithi102 ceph-mon[115269]: 1.2 scrub ok 2024-09-13T14:46:53.524 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:53 smithi102 ceph-mon[115269]: pgmap v1120: 99 pgs: 10 peering, 89 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 87 MiB/s wr, 1.39k op/s; 106 KiB/s, 0 objects/s recovering 2024-09-13T14:46:53.524 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:53 smithi102 ceph-mon[115269]: 1.a scrub starts 2024-09-13T14:46:53.524 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:53 smithi102 ceph-mon[115269]: 1.a scrub ok 2024-09-13T14:46:53.525 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:53 smithi102 ceph-mon[110183]: 1.2 scrub starts 2024-09-13T14:46:53.525 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:53 smithi102 ceph-mon[110183]: 1.2 scrub ok 2024-09-13T14:46:53.525 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:53 smithi102 ceph-mon[110183]: pgmap v1120: 99 pgs: 10 peering, 89 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 87 MiB/s wr, 1.39k op/s; 106 KiB/s, 0 objects/s recovering 2024-09-13T14:46:53.525 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:53 smithi102 ceph-mon[110183]: 1.a scrub starts 2024-09-13T14:46:53.525 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:53 smithi102 ceph-mon[110183]: 1.a scrub ok 2024-09-13T14:46:53.621 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:53.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:53.656 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:53.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:53.757 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:53.797 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:53.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:53.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:53.805 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:2024-09-13T14:46:53.805523+0000 min lat: 0.00109072 max lat: 0.310704 avg lat: 0.00844601 2024-09-13T14:46:53.805 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:46:53.805 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 40 16 75762 75746 118.331 88.5 0.0155762 0.00844601 2024-09-13T14:46:53.899 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:53.935 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:53.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:53.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:54.037 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:54.078 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:54.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:54.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:54.179 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:54.208 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T14:46:54.217 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:54.227 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:54.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:54.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:54.329 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:54.372 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:54.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:54.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:54.474 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:54.502 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:54.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:54.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:54.583 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:46:54.604 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:54.638 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:54.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:54.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:54.740 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:54.779 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:54.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:54.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:54.805 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 41 16 77101 77085 117.486 83.6875 0.0106813 0.00850679 2024-09-13T14:46:54.881 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:54.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:54.918 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:54.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:55.019 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:55.040 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:55.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:55.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:55.142 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:55.163 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:55.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:55.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:55.265 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:55.298 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:55.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:55.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:55.301 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:55 smithi102 ceph-mon[110183]: pgmap v1121: 99 pgs: 99 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 130 MiB/s wr, 2.06k op/s 2024-09-13T14:46:55.301 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:55 smithi102 ceph-mon[110183]: 1.6 scrub starts 2024-09-13T14:46:55.301 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:55 smithi102 ceph-mon[110183]: 1.6 scrub ok 2024-09-13T14:46:55.301 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:55 smithi102 ceph-mon[110183]: 1.7 scrub starts 2024-09-13T14:46:55.301 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:55 smithi102 ceph-mon[110183]: 1.7 scrub ok 2024-09-13T14:46:55.301 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:55 smithi102 ceph-mon[110183]: 1.8 scrub starts 2024-09-13T14:46:55.301 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:55 smithi102 ceph-mon[110183]: 1.8 scrub ok 2024-09-13T14:46:55.302 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:55 smithi102 ceph-mon[115269]: pgmap v1121: 99 pgs: 99 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 130 MiB/s wr, 2.06k op/s 2024-09-13T14:46:55.302 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:55 smithi102 ceph-mon[115269]: 1.6 scrub starts 2024-09-13T14:46:55.302 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:55 smithi102 ceph-mon[115269]: 1.6 scrub ok 2024-09-13T14:46:55.302 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:55 smithi102 ceph-mon[115269]: 1.7 scrub starts 2024-09-13T14:46:55.302 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:55 smithi102 ceph-mon[115269]: 1.7 scrub ok 2024-09-13T14:46:55.302 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:55 smithi102 ceph-mon[115269]: 1.8 scrub starts 2024-09-13T14:46:55.302 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:55 smithi102 ceph-mon[115269]: 1.8 scrub ok 2024-09-13T14:46:55.401 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:55.438 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:55.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:55.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:55.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:55 smithi195 ceph-mon[104262]: pgmap v1121: 99 pgs: 99 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 130 MiB/s wr, 2.06k op/s 2024-09-13T14:46:55.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:55 smithi195 ceph-mon[104262]: 1.6 scrub starts 2024-09-13T14:46:55.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:55 smithi195 ceph-mon[104262]: 1.6 scrub ok 2024-09-13T14:46:55.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:55 smithi195 ceph-mon[104262]: 1.7 scrub starts 2024-09-13T14:46:55.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:55 smithi195 ceph-mon[104262]: 1.7 scrub ok 2024-09-13T14:46:55.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:55 smithi195 ceph-mon[104262]: 1.8 scrub starts 2024-09-13T14:46:55.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:55 smithi195 ceph-mon[104262]: 1.8 scrub ok 2024-09-13T14:46:55.540 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:55.574 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:55.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:55.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:55.676 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:55.714 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:55.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:55.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:55.806 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 42 16 78174 78158 116.285 67.0625 0.0157593 0.00859457 2024-09-13T14:46:55.816 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:55.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:55.855 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:55.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:55.956 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:55.986 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:55.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:55.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:56.088 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:56.123 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:56.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:56.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:56.225 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:56.263 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:56.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:56.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:56.365 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:56.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:56.405 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:56.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:56.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:46:56.507 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:56.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:56.544 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:56.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:56.642 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:46:56.643 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:46:56.643 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:46:56 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:46:56] "GET /metrics HTTP/1.1" 200 36521 "" "Prometheus/2.51.0" 2024-09-13T14:46:56.645 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:56.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:56.682 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:56.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:56.783 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:56.806 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 43 16 79458 79442 115.446 80.25 0.00661505 0.00865681 2024-09-13T14:46:56.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:56.827 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:56.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:56.928 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:56.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:56.983 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:56.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:56.988 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:46:56.988 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:46:57.018 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:46:57.018 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:46:57.055 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:46:57.056 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:46:57.084 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:57.088 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:46:57.088 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:46:57.113 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:57.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:57.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:57.128 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:46:57.128 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:46:57.166 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:46:57.167 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:46:57.199 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:46:57.199 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:46:57.215 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:57.238 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:46:57.238 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:46:57.258 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:57.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:57.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:57.303 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:46:57.303 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-09-13T14:46:57.303 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-09-13T14:46:57.303 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-13T14:46:57.303 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.303 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:112933 7.e 7:71539828:::benchmark_data_smithi102_227395_object118:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-09-13T14:46:57.303 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:28.169308+0000", 2024-09-13T14:46:57.304 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 569.13003473200001, 2024-09-13T14:46:57.304 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.67251101, 2024-09-13T14:46:57.304 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:46:57.304 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:57.304 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:46:57.304 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:46:57.304 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:46:57.304 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 112933 2024-09-13T14:46:57.304 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.304 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:46:57.304 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.304 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:46:57.304 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.169308+0000", 2024-09-13T14:46:57.305 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:57.305 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.305 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.305 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:46:57.305 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.169308+0000", 2024-09-13T14:46:57.305 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:57.305 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.305 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.305 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:46:57.305 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.169306+0000", 2024-09-13T14:46:57.305 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:46:57.305 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.306 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.306 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:46:57.306 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.169314+0000", 2024-09-13T14:46:57.306 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5399999999999998e-06 2024-09-13T14:46:57.306 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.306 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.306 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:46:57.306 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.169327+0000", 2024-09-13T14:46:57.306 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3589e-05 2024-09-13T14:46:57.306 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.306 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.306 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:46:57.306 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.169336+0000", 2024-09-13T14:46:57.307 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1599999999999998e-06 2024-09-13T14:46:57.307 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.307 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.307 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.307 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.036088+0000", 2024-09-13T14:46:57.307 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.86675195199999999 2024-09-13T14:46:57.307 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.307 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.307 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:46:57.307 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.056928+0000", 2024-09-13T14:46:57.307 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.020840121 2024-09-13T14:46:57.307 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.307 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.308 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.308 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.228620+0000", 2024-09-13T14:46:57.308 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.17169189700000001 2024-09-13T14:46:57.308 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.308 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.308 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:46:57.308 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.256794+0000", 2024-09-13T14:46:57.308 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.028174637999999998 2024-09-13T14:46:57.308 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.308 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.308 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.308 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.612105+0000", 2024-09-13T14:46:57.308 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3553110849999999 2024-09-13T14:46:57.309 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.309 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.309 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:46:57.309 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.625533+0000", 2024-09-13T14:46:57.309 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013427308000000001 2024-09-13T14:46:57.309 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.309 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.309 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:46:57.309 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.650011+0000", 2024-09-13T14:46:57.309 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024478033999999999 2024-09-13T14:46:57.309 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.309 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.309 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:57.310 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.779644+0000", 2024-09-13T14:46:57.310 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.12963343899999999 2024-09-13T14:46:57.310 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.310 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.310 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:46:57.310 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.781386+0000", 2024-09-13T14:46:57.310 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0017419709999999999 2024-09-13T14:46:57.310 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.310 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.310 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:57.310 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.786230+0000", 2024-09-13T14:46:57.310 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0048441839999999996 2024-09-13T14:46:57.310 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.311 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.311 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:46:57.311 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.792473+0000", 2024-09-13T14:46:57.311 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0062430649999999999 2024-09-13T14:46:57.311 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.311 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.311 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:46:57.311 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.841819+0000", 2024-09-13T14:46:57.311 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.049345554999999999 2024-09-13T14:46:57.311 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:57.311 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:46:57.311 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:57.311 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.311 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.312 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:112953 7.e 7:72a6485e:::benchmark_data_smithi102_227395_object138:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-09-13T14:46:57.312 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:28.219817+0000", 2024-09-13T14:46:57.312 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 569.07952571700002, 2024-09-13T14:46:57.312 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.688959026, 2024-09-13T14:46:57.312 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:46:57.312 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:57.312 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:46:57.312 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:46:57.312 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:46:57.312 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 112953 2024-09-13T14:46:57.312 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.312 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:46:57.312 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.313 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:46:57.313 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.219817+0000", 2024-09-13T14:46:57.313 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:57.313 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.313 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.313 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:46:57.313 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.219817+0000", 2024-09-13T14:46:57.313 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:57.313 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.313 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.313 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:46:57.313 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.219815+0000", 2024-09-13T14:46:57.313 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:46:57.314 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.314 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.314 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:46:57.314 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.219824+0000", 2024-09-13T14:46:57.314 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.3449999999999995e-06 2024-09-13T14:46:57.314 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.314 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.314 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:46:57.314 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.219841+0000", 2024-09-13T14:46:57.314 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6330000000000001e-05 2024-09-13T14:46:57.314 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.314 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.314 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:46:57.314 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.219850+0000", 2024-09-13T14:46:57.315 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.2280000000000004e-06 2024-09-13T14:46:57.315 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.315 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.315 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.315 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.266772+0000", 2024-09-13T14:46:57.315 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.046922012 2024-09-13T14:46:57.315 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.315 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.315 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:46:57.315 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.295174+0000", 2024-09-13T14:46:57.315 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.028402107999999999 2024-09-13T14:46:57.315 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.315 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.316 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.316 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.443017+0000", 2024-09-13T14:46:57.316 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.14784280499999999 2024-09-13T14:46:57.316 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.316 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.316 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:46:57.316 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.465867+0000", 2024-09-13T14:46:57.316 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.022849813 2024-09-13T14:46:57.316 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.316 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.316 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.316 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.721401+0000", 2024-09-13T14:46:57.316 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.255534258 2024-09-13T14:46:57.317 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.317 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.317 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:46:57.317 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.733248+0000", 2024-09-13T14:46:57.317 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.01184754 2024-09-13T14:46:57.317 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.317 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.317 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:46:57.317 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.757733+0000", 2024-09-13T14:46:57.317 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024485033 2024-09-13T14:46:57.317 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.317 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.317 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:57.318 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.847315+0000", 2024-09-13T14:46:57.318 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.089581220000000003 2024-09-13T14:46:57.318 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.318 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.319 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:46:57.319 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.849721+0000", 2024-09-13T14:46:57.319 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0024068840000000002 2024-09-13T14:46:57.319 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.319 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.319 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:57.319 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.856740+0000", 2024-09-13T14:46:57.319 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0070188029999999997 2024-09-13T14:46:57.319 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.319 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.319 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:46:57.319 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.867725+0000", 2024-09-13T14:46:57.319 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010984910000000001 2024-09-13T14:46:57.320 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.320 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.320 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:46:57.320 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.908776+0000", 2024-09-13T14:46:57.320 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.041050775999999997 2024-09-13T14:46:57.320 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:57.320 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:46:57.320 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:57.320 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.320 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.320 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:112957 7.e 7:73c81a1b:::benchmark_data_smithi102_227395_object142:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-09-13T14:46:57.320 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:28.238422+0000", 2024-09-13T14:46:57.320 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 569.06092027199998, 2024-09-13T14:46:57.321 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9574894330000001, 2024-09-13T14:46:57.321 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:46:57.321 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:57.321 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:46:57.321 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:46:57.321 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:46:57.321 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 112957 2024-09-13T14:46:57.321 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.321 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:46:57.321 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.321 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:46:57.321 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.238422+0000", 2024-09-13T14:46:57.321 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:57.322 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.322 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.322 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:46:57.322 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.238422+0000", 2024-09-13T14:46:57.322 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:57.322 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.322 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.322 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:46:57.322 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.238421+0000", 2024-09-13T14:46:57.322 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:46:57.323 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.323 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.323 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:46:57.323 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.238429+0000", 2024-09-13T14:46:57.323 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9349999999999994e-06 2024-09-13T14:46:57.323 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.323 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.323 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:46:57.323 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.238442+0000", 2024-09-13T14:46:57.323 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3501e-05 2024-09-13T14:46:57.323 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.323 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.323 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:46:57.324 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.238448+0000", 2024-09-13T14:46:57.324 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3759999999999996e-06 2024-09-13T14:46:57.324 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.324 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.324 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.324 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.297744+0000", 2024-09-13T14:46:57.324 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.059295036 2024-09-13T14:46:57.324 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.324 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.324 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:46:57.324 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.319171+0000", 2024-09-13T14:46:57.324 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.021427883000000002 2024-09-13T14:46:57.325 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.325 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.325 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.325 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.554554+0000", 2024-09-13T14:46:57.325 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.23538229799999999 2024-09-13T14:46:57.325 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.325 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.325 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:46:57.325 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.579299+0000", 2024-09-13T14:46:57.325 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024745277 2024-09-13T14:46:57.325 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.325 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.325 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.325 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:31.992160+0000", 2024-09-13T14:46:57.326 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4128609559999998 2024-09-13T14:46:57.326 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.326 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.326 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:46:57.326 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.011466+0000", 2024-09-13T14:46:57.326 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.019306104000000001 2024-09-13T14:46:57.326 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.326 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.326 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:46:57.326 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.039599+0000", 2024-09-13T14:46:57.326 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.028132766999999999 2024-09-13T14:46:57.326 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.326 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.327 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:57.327 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.138317+0000", 2024-09-13T14:46:57.327 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.098718599000000004 2024-09-13T14:46:57.327 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.327 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.327 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:46:57.327 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.139902+0000", 2024-09-13T14:46:57.327 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001584184 2024-09-13T14:46:57.327 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.327 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.327 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:57.327 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.148487+0000", 2024-09-13T14:46:57.327 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0085852440000000006 2024-09-13T14:46:57.328 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.328 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.328 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:46:57.328 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.156727+0000", 2024-09-13T14:46:57.328 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0082399269999999993 2024-09-13T14:46:57.328 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.328 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.328 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:46:57.328 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.195912+0000", 2024-09-13T14:46:57.328 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.039185042000000003 2024-09-13T14:46:57.328 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:57.328 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:46:57.328 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:57.328 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.329 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.329 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:112968 7.e 7:7cd0abb8:::benchmark_data_smithi102_227395_object153:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-09-13T14:46:57.329 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:28.367834+0000", 2024-09-13T14:46:57.329 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 568.93150844199999, 2024-09-13T14:46:57.329 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1335673269999997, 2024-09-13T14:46:57.329 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:46:57.329 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:57.329 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:46:57.329 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:46:57.329 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:46:57.329 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 112968 2024-09-13T14:46:57.329 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.330 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:46:57.330 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.330 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:46:57.330 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.367834+0000", 2024-09-13T14:46:57.330 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:57.330 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.330 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.330 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:46:57.330 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.367834+0000", 2024-09-13T14:46:57.330 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:57.330 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.330 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.330 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:46:57.330 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.367832+0000", 2024-09-13T14:46:57.331 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:46:57.331 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.331 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.331 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:46:57.331 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.367841+0000", 2024-09-13T14:46:57.331 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7539999999999995e-06 2024-09-13T14:46:57.331 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.331 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.331 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:46:57.331 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.367856+0000", 2024-09-13T14:46:57.331 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4499e-05 2024-09-13T14:46:57.331 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.331 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.332 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:46:57.332 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.367863+0000", 2024-09-13T14:46:57.332 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6690000000000006e-06 2024-09-13T14:46:57.332 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.332 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.332 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.332 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.321756+0000", 2024-09-13T14:46:57.332 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.95389310000000005 2024-09-13T14:46:57.332 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.332 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.332 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:46:57.332 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.346608+0000", 2024-09-13T14:46:57.332 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024851504 2024-09-13T14:46:57.333 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.333 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.333 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.333 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.658373+0000", 2024-09-13T14:46:57.333 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.31176493100000002 2024-09-13T14:46:57.333 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.333 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.333 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:46:57.333 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.677721+0000", 2024-09-13T14:46:57.333 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.019348260999999999 2024-09-13T14:46:57.333 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.333 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.334 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.334 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.244806+0000", 2024-09-13T14:46:57.334 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.567085134 2024-09-13T14:46:57.334 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.334 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.334 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:46:57.334 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.260918+0000", 2024-09-13T14:46:57.334 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016111721999999998 2024-09-13T14:46:57.334 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.334 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.334 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:46:57.334 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.285995+0000", 2024-09-13T14:46:57.334 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.025077351000000001 2024-09-13T14:46:57.335 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.335 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.335 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:57.335 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.451788+0000", 2024-09-13T14:46:57.335 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.16579242799999999 2024-09-13T14:46:57.335 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.335 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.335 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:46:57.335 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.453689+0000", 2024-09-13T14:46:57.335 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0019015049999999999 2024-09-13T14:46:57.335 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.335 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.335 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:57.336 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.459427+0000", 2024-09-13T14:46:57.336 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0057378819999999997 2024-09-13T14:46:57.336 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.336 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.336 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:46:57.336 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.465958+0000", 2024-09-13T14:46:57.336 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0065305709999999998 2024-09-13T14:46:57.336 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.336 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.336 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:46:57.336 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.501402+0000", 2024-09-13T14:46:57.336 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.035443745999999998 2024-09-13T14:46:57.336 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:57.337 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:46:57.337 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:57.337 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.337 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.337 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113005 7.e 7:78e9e6f9:::benchmark_data_smithi102_227395_object190:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-09-13T14:46:57.337 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:28.544298+0000", 2024-09-13T14:46:57.337 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 568.75504490699996, 2024-09-13T14:46:57.337 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0162806550000001, 2024-09-13T14:46:57.337 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:46:57.337 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:57.337 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:46:57.337 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:46:57.337 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:46:57.338 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113005 2024-09-13T14:46:57.338 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.338 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:46:57.338 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.338 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:46:57.338 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.544298+0000", 2024-09-13T14:46:57.338 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:57.338 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.338 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.338 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:46:57.338 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.544298+0000", 2024-09-13T14:46:57.338 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:57.338 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.338 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.339 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:46:57.339 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.544296+0000", 2024-09-13T14:46:57.339 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:46:57.339 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.339 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.339 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:46:57.339 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.544305+0000", 2024-09-13T14:46:57.339 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.0310000000000004e-06 2024-09-13T14:46:57.339 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.339 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.339 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:46:57.339 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.544318+0000", 2024-09-13T14:46:57.339 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3301e-05 2024-09-13T14:46:57.340 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.340 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.340 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:46:57.340 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:28.544325+0000", 2024-09-13T14:46:57.340 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9380000000000003e-06 2024-09-13T14:46:57.340 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.340 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.340 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.340 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.679520+0000", 2024-09-13T14:46:57.340 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1351946159999999 2024-09-13T14:46:57.340 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.340 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.340 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:46:57.340 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.697460+0000", 2024-09-13T14:46:57.340 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017939749000000001 2024-09-13T14:46:57.341 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.341 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.341 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.341 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.780770+0000", 2024-09-13T14:46:57.341 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.083310181999999997 2024-09-13T14:46:57.341 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.341 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.341 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:46:57.341 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:29.798851+0000", 2024-09-13T14:46:57.341 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018080978000000001 2024-09-13T14:46:57.341 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.341 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.342 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.342 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.382351+0000", 2024-09-13T14:46:57.342 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5834998100000002 2024-09-13T14:46:57.342 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.342 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.342 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:46:57.342 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.398282+0000", 2024-09-13T14:46:57.342 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.015931325999999999 2024-09-13T14:46:57.342 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.342 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.342 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:46:57.342 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.426371+0000", 2024-09-13T14:46:57.342 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.028089050000000001 2024-09-13T14:46:57.343 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.343 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.343 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:46:57.343 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.503108+0000", 2024-09-13T14:46:57.343 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.076737284000000003 2024-09-13T14:46:57.343 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.343 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.343 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:57.343 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.507418+0000", 2024-09-13T14:46:57.343 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0043095690000000001 2024-09-13T14:46:57.343 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.343 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.343 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:57.344 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.514013+0000", 2024-09-13T14:46:57.344 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0065951639999999997 2024-09-13T14:46:57.344 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.344 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.344 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:46:57.344 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.521065+0000", 2024-09-13T14:46:57.344 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.007051753 2024-09-13T14:46:57.344 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.344 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.344 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:46:57.344 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:32.560578+0000", 2024-09-13T14:46:57.344 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.039513581999999998 2024-09-13T14:46:57.344 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:57.344 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:46:57.345 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:57.345 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.345 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.345 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113646 7.e 7:7fda021e:::benchmark_data_smithi102_227395_object831:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-13T14:46:57.345 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:39.650939+0000", 2024-09-13T14:46:57.345 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 557.64840316799996, 2024-09-13T14:46:57.345 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4310964820000001, 2024-09-13T14:46:57.345 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:46:57.345 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:57.345 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:46:57.345 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:46:57.345 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:46:57.345 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113646 2024-09-13T14:46:57.346 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.346 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:46:57.346 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.346 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:46:57.346 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:39.650939+0000", 2024-09-13T14:46:57.346 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:57.346 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.346 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.346 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:46:57.346 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:39.650939+0000", 2024-09-13T14:46:57.346 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:57.346 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.346 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.346 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:46:57.347 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:39.650938+0000", 2024-09-13T14:46:57.347 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:46:57.347 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.347 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.347 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:46:57.347 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:39.650945+0000", 2024-09-13T14:46:57.347 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4200000000000001e-06 2024-09-13T14:46:57.347 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.347 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.347 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:46:57.347 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:39.650960+0000", 2024-09-13T14:46:57.347 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5028000000000001e-05 2024-09-13T14:46:57.347 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.348 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.348 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:46:57.348 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:39.650969+0000", 2024-09-13T14:46:57.348 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.4319999999999999e-06 2024-09-13T14:46:57.348 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.348 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.348 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.348 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:39.791532+0000", 2024-09-13T14:46:57.348 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.140563199 2024-09-13T14:46:57.348 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.348 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.348 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:46:57.348 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:39.808048+0000", 2024-09-13T14:46:57.349 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016515914999999999 2024-09-13T14:46:57.349 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.349 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.349 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.349 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:39.937277+0000", 2024-09-13T14:46:57.349 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.129229236 2024-09-13T14:46:57.349 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.349 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.349 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:46:57.349 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:39.953949+0000", 2024-09-13T14:46:57.349 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016672356999999999 2024-09-13T14:46:57.349 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.349 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.350 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.350 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.812544+0000", 2024-09-13T14:46:57.350 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8585950149999997 2024-09-13T14:46:57.350 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.350 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.350 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:46:57.350 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.829480+0000", 2024-09-13T14:46:57.350 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016936072999999999 2024-09-13T14:46:57.350 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.350 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.350 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:46:57.350 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.857149+0000", 2024-09-13T14:46:57.350 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.027668132000000002 2024-09-13T14:46:57.351 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.351 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.351 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:57.351 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.013996+0000", 2024-09-13T14:46:57.351 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.15684716000000001 2024-09-13T14:46:57.351 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.351 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.351 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:57.351 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.021795+0000", 2024-09-13T14:46:57.351 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0077991079999999999 2024-09-13T14:46:57.351 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.351 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.351 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:46:57.351 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.023954+0000", 2024-09-13T14:46:57.352 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0021587970000000001 2024-09-13T14:46:57.352 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.352 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.352 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:46:57.352 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.034566+0000", 2024-09-13T14:46:57.352 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010612244 2024-09-13T14:46:57.352 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.352 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.352 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:46:57.352 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.082036+0000", 2024-09-13T14:46:57.352 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.047470036 2024-09-13T14:46:57.352 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:57.353 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:46:57.353 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:57.353 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.353 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.353 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113660 7.e 7:72e391fe:::benchmark_data_smithi102_227395_object845:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-13T14:46:57.353 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:39.881206+0000", 2024-09-13T14:46:57.353 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 557.41813650400002, 2024-09-13T14:46:57.353 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2702958029999998, 2024-09-13T14:46:57.353 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:46:57.353 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:57.353 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:46:57.353 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:46:57.353 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:46:57.354 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113660 2024-09-13T14:46:57.354 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.354 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:46:57.354 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.354 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:46:57.354 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:39.881206+0000", 2024-09-13T14:46:57.354 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:57.354 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.354 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.354 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:46:57.354 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:39.881206+0000", 2024-09-13T14:46:57.354 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:57.354 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.355 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.355 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:46:57.355 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:39.881205+0000", 2024-09-13T14:46:57.355 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:46:57.355 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.355 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.355 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:46:57.355 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:39.881212+0000", 2024-09-13T14:46:57.355 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.17e-06 2024-09-13T14:46:57.355 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.355 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.355 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:46:57.355 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:39.881223+0000", 2024-09-13T14:46:57.355 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1214e-05 2024-09-13T14:46:57.356 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.356 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.356 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:46:57.356 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:39.881229+0000", 2024-09-13T14:46:57.356 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5559999999999998e-06 2024-09-13T14:46:57.356 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.356 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.356 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.356 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.072810+0000", 2024-09-13T14:46:57.356 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.19158088300000001 2024-09-13T14:46:57.356 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.356 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.357 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:46:57.357 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.098533+0000", 2024-09-13T14:46:57.357 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.025723314000000001 2024-09-13T14:46:57.357 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.357 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.357 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.357 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.225733+0000", 2024-09-13T14:46:57.357 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.12720013499999999 2024-09-13T14:46:57.357 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.357 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.357 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:46:57.357 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.243239+0000", 2024-09-13T14:46:57.357 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017505427 2024-09-13T14:46:57.357 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.358 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.358 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.358 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.929831+0000", 2024-09-13T14:46:57.358 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6865921210000003 2024-09-13T14:46:57.358 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.358 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.358 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:46:57.358 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.951285+0000", 2024-09-13T14:46:57.358 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.021454606000000001 2024-09-13T14:46:57.358 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.358 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.358 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:46:57.358 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.980812+0000", 2024-09-13T14:46:57.359 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.029526611000000001 2024-09-13T14:46:57.359 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.359 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.359 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:46:57.359 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.084340+0000", 2024-09-13T14:46:57.359 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.10352797699999999 2024-09-13T14:46:57.359 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.359 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.359 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:57.359 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.090949+0000", 2024-09-13T14:46:57.359 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0066088229999999998 2024-09-13T14:46:57.359 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.359 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.360 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:57.360 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.095550+0000", 2024-09-13T14:46:57.360 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0046012689999999998 2024-09-13T14:46:57.360 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.360 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.360 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:46:57.360 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.104748+0000", 2024-09-13T14:46:57.360 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0091979999999999996 2024-09-13T14:46:57.360 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.361 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.361 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:46:57.361 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.151502+0000", 2024-09-13T14:46:57.361 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.046753905999999998 2024-09-13T14:46:57.361 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:57.361 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:46:57.361 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:57.361 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.362 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.362 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113667 7.e 7:7011123e:::benchmark_data_smithi102_227395_object852:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-13T14:46:57.362 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:40.145165+0000", 2024-09-13T14:46:57.362 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 557.15417753199995, 2024-09-13T14:46:57.362 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2681343839999997, 2024-09-13T14:46:57.362 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:46:57.362 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:57.362 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:46:57.362 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:46:57.362 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:46:57.362 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113667 2024-09-13T14:46:57.362 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.363 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:57.364 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:46:57.364 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.364 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:46:57.365 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.145165+0000", 2024-09-13T14:46:57.365 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:57.365 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.365 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.365 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:46:57.365 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.145165+0000", 2024-09-13T14:46:57.365 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:57.365 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.365 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.365 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:46:57.365 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.145164+0000", 2024-09-13T14:46:57.365 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:46:57.365 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.366 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.366 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:46:57.366 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.145170+0000", 2024-09-13T14:46:57.366 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8379999999999998e-06 2024-09-13T14:46:57.366 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.366 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.366 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:46:57.366 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.145181+0000", 2024-09-13T14:46:57.366 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1521e-05 2024-09-13T14:46:57.366 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.366 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.366 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:46:57.366 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.145186+0000", 2024-09-13T14:46:57.366 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0200000000000002e-06 2024-09-13T14:46:57.367 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.367 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.367 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.367 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.244931+0000", 2024-09-13T14:46:57.367 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.099744219999999995 2024-09-13T14:46:57.367 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.367 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.367 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:46:57.367 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.263033+0000", 2024-09-13T14:46:57.367 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018102917 2024-09-13T14:46:57.367 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.367 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.367 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.368 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.467278+0000", 2024-09-13T14:46:57.368 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.20424453100000001 2024-09-13T14:46:57.368 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.368 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.368 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:46:57.368 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.487283+0000", 2024-09-13T14:46:57.368 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.020004724000000002 2024-09-13T14:46:57.368 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.368 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.368 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.368 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.223527+0000", 2024-09-13T14:46:57.368 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7362443320000001 2024-09-13T14:46:57.368 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.368 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.369 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:46:57.369 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.237432+0000", 2024-09-13T14:46:57.369 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013905244000000001 2024-09-13T14:46:57.369 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.369 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.369 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:46:57.369 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.264564+0000", 2024-09-13T14:46:57.369 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.027132177 2024-09-13T14:46:57.369 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.369 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.369 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:57.369 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.369627+0000", 2024-09-13T14:46:57.369 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.10506285 2024-09-13T14:46:57.370 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.370 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.370 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:46:57.370 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.370616+0000", 2024-09-13T14:46:57.370 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00098851600000000005 2024-09-13T14:46:57.370 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.370 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.370 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:57.370 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.376243+0000", 2024-09-13T14:46:57.370 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0056276249999999998 2024-09-13T14:46:57.370 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.370 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.370 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:46:57.371 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.381212+0000", 2024-09-13T14:46:57.371 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0049688850000000001 2024-09-13T14:46:57.371 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.371 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.371 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:46:57.371 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.413299+0000", 2024-09-13T14:46:57.371 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.032087169999999998 2024-09-13T14:46:57.371 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:57.371 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:46:57.371 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:57.371 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.371 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.371 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113673 7.e 7:707c08ea:::benchmark_data_smithi102_227395_object858:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-13T14:46:57.372 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:40.852904+0000", 2024-09-13T14:46:57.372 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 556.44643905700002, 2024-09-13T14:46:57.372 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0436350230000002, 2024-09-13T14:46:57.372 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:46:57.372 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:57.372 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:46:57.372 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:46:57.372 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:46:57.372 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113673 2024-09-13T14:46:57.372 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.372 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:46:57.372 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.372 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:46:57.373 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.852904+0000", 2024-09-13T14:46:57.373 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:57.373 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.373 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.373 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:46:57.373 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.852904+0000", 2024-09-13T14:46:57.373 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:57.373 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.373 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.373 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:46:57.373 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.852902+0000", 2024-09-13T14:46:57.373 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:46:57.373 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.373 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.373 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:46:57.374 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.852909+0000", 2024-09-13T14:46:57.374 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8900000000000001e-06 2024-09-13T14:46:57.374 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.374 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.374 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:46:57.374 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.852927+0000", 2024-09-13T14:46:57.374 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8281999999999999e-05 2024-09-13T14:46:57.374 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.374 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.374 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:46:57.374 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.852933+0000", 2024-09-13T14:46:57.374 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0220000000000001e-06 2024-09-13T14:46:57.374 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.374 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.375 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.375 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.932659+0000", 2024-09-13T14:46:57.375 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.079725349000000001 2024-09-13T14:46:57.375 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.375 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.375 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:46:57.375 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:40.961885+0000", 2024-09-13T14:46:57.375 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.029226683999999999 2024-09-13T14:46:57.375 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.375 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.375 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.375 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.190540+0000", 2024-09-13T14:46:57.375 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.22865431899999999 2024-09-13T14:46:57.376 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.376 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.376 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:46:57.376 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.213124+0000", 2024-09-13T14:46:57.376 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.022584453000000001 2024-09-13T14:46:57.376 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.376 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.376 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.376 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.694534+0000", 2024-09-13T14:46:57.376 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4814095549999999 2024-09-13T14:46:57.376 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.376 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.376 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:46:57.377 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.708113+0000", 2024-09-13T14:46:57.377 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013578961000000001 2024-09-13T14:46:57.377 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.377 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.377 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:46:57.377 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.734818+0000", 2024-09-13T14:46:57.377 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.026704998000000001 2024-09-13T14:46:57.377 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.377 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.377 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:57.377 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.843296+0000", 2024-09-13T14:46:57.377 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.10847884000000001 2024-09-13T14:46:57.378 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.378 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.378 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:46:57.378 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.844461+0000", 2024-09-13T14:46:57.378 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001164111 2024-09-13T14:46:57.378 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.378 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.378 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:57.378 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.853549+0000", 2024-09-13T14:46:57.378 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0090885069999999991 2024-09-13T14:46:57.378 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.378 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.378 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:46:57.378 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.859381+0000", 2024-09-13T14:46:57.379 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.005832266 2024-09-13T14:46:57.379 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.379 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.379 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:46:57.379 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.896539+0000", 2024-09-13T14:46:57.379 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.037157218999999998 2024-09-13T14:46:57.379 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:57.379 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:46:57.379 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:57.379 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.379 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.379 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDPGPush(7.e 236/222 [PushOp(7:765a5df7:::benchmark_data_smithi102_227395_object908:head, version: 209'56, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:765a5df7:::benchmark_data_smithi102_227395_object908:head@209'56, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T14:46:57.379 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:41.327036+0000", 2024-09-13T14:46:57.379 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 555.97230632000003, 2024-09-13T14:46:57.380 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4911342149999998, 2024-09-13T14:46:57.380 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:46:57.380 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T14:46:57.380 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:46:57.380 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.380 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:46:57.380 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.327036+0000", 2024-09-13T14:46:57.380 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:57.380 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.380 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.380 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:46:57.380 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.327036+0000", 2024-09-13T14:46:57.380 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:57.380 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.381 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.381 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:46:57.381 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.327035+0000", 2024-09-13T14:46:57.381 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:46:57.381 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.381 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.381 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:46:57.381 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.327094+0000", 2024-09-13T14:46:57.381 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8471999999999999e-05 2024-09-13T14:46:57.381 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.381 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.381 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:46:57.381 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.327110+0000", 2024-09-13T14:46:57.382 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6333000000000001e-05 2024-09-13T14:46:57.382 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.382 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.382 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:46:57.382 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.327125+0000", 2024-09-13T14:46:57.382 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4766999999999999e-05 2024-09-13T14:46:57.382 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.382 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.382 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.382 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.756102+0000", 2024-09-13T14:46:57.382 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4289765430000001 2024-09-13T14:46:57.382 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.382 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.382 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:46:57.382 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.757954+0000", 2024-09-13T14:46:57.383 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0018523560000000001 2024-09-13T14:46:57.383 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.383 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.383 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:46:57.383 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.818171+0000", 2024-09-13T14:46:57.383 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.060216609999999997 2024-09-13T14:46:57.383 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:57.383 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:46:57.383 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:57.383 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.383 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.383 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113723 7.e 7:765a5df7:::benchmark_data_smithi102_227395_object908:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-13T14:46:57.383 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:41.735065+0000", 2024-09-13T14:46:57.383 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 555.56427760999998, 2024-09-13T14:46:57.383 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6293769879999997, 2024-09-13T14:46:57.384 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:46:57.384 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:57.384 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:46:57.384 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:46:57.384 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:46:57.384 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113723 2024-09-13T14:46:57.384 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.384 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:46:57.384 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.384 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:46:57.384 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.735065+0000", 2024-09-13T14:46:57.384 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:57.384 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.384 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.385 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:46:57.385 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.735065+0000", 2024-09-13T14:46:57.385 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:57.385 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.385 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.385 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:46:57.385 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.735064+0000", 2024-09-13T14:46:57.385 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:46:57.385 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.385 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.385 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:46:57.385 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.735071+0000", 2024-09-13T14:46:57.385 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2289999999999997e-06 2024-09-13T14:46:57.385 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.386 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.386 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:46:57.386 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.735085+0000", 2024-09-13T14:46:57.386 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4436e-05 2024-09-13T14:46:57.386 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.386 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.386 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:46:57.386 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.735092+0000", 2024-09-13T14:46:57.386 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.216e-06 2024-09-13T14:46:57.386 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.386 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.386 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.386 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.016651+0000", 2024-09-13T14:46:57.386 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.28155898499999998 2024-09-13T14:46:57.387 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.387 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.387 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:46:57.387 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.027127+0000", 2024-09-13T14:46:57.387 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010475241999999999 2024-09-13T14:46:57.387 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.387 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.387 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.387 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.819782+0000", 2024-09-13T14:46:57.387 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7926552459999998 2024-09-13T14:46:57.387 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.387 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.387 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:46:57.387 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.838097+0000", 2024-09-13T14:46:57.388 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018315084999999998 2024-09-13T14:46:57.388 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.388 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.388 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.388 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.103166+0000", 2024-09-13T14:46:57.388 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2650688400000001 2024-09-13T14:46:57.388 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.388 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.388 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:46:57.388 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.113423+0000", 2024-09-13T14:46:57.388 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010256919 2024-09-13T14:46:57.388 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.388 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.388 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:46:57.389 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.159715+0000", 2024-09-13T14:46:57.389 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.046292424999999998 2024-09-13T14:46:57.389 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.389 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.389 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:46:57.389 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.242694+0000", 2024-09-13T14:46:57.389 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.082978417999999998 2024-09-13T14:46:57.389 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.389 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.389 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:57.389 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.310555+0000", 2024-09-13T14:46:57.389 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.067861006000000001 2024-09-13T14:46:57.389 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.389 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.390 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:57.390 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.316591+0000", 2024-09-13T14:46:57.390 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0060365549999999999 2024-09-13T14:46:57.390 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.390 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.390 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:46:57.390 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.328028+0000", 2024-09-13T14:46:57.390 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.01143724 2024-09-13T14:46:57.390 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.390 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.390 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:46:57.390 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:48.364442+0000", 2024-09-13T14:46:57.391 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.036413661 2024-09-13T14:46:57.391 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:57.391 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:46:57.391 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:57.391 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.391 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.391 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113733 7.e 7:71f1f434:::benchmark_data_smithi102_227395_object918:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-13T14:46:57.391 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:41.768799+0000", 2024-09-13T14:46:57.391 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 555.53054407100001, 2024-09-13T14:46:57.391 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4105988570000001, 2024-09-13T14:46:57.391 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:46:57.391 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:57.391 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:46:57.391 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:46:57.392 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:46:57.392 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113733 2024-09-13T14:46:57.392 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.392 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:46:57.392 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.392 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:46:57.392 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.768799+0000", 2024-09-13T14:46:57.392 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:57.392 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.392 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.392 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:46:57.392 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.768799+0000", 2024-09-13T14:46:57.392 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:57.392 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.393 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.393 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:46:57.393 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.768797+0000", 2024-09-13T14:46:57.393 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:46:57.393 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.393 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.393 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:46:57.393 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.768805+0000", 2024-09-13T14:46:57.393 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1880000000000002e-06 2024-09-13T14:46:57.393 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.393 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.393 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:46:57.393 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.768818+0000", 2024-09-13T14:46:57.394 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3322999999999999e-05 2024-09-13T14:46:57.394 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.394 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.394 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:46:57.394 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.768826+0000", 2024-09-13T14:46:57.394 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5070000000000003e-06 2024-09-13T14:46:57.394 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.394 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.394 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.394 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.029999+0000", 2024-09-13T14:46:57.394 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.261173402 2024-09-13T14:46:57.394 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.394 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.394 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:46:57.395 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.071555+0000", 2024-09-13T14:46:57.395 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.041556211000000003 2024-09-13T14:46:57.395 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.395 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.395 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.395 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.202418+0000", 2024-09-13T14:46:57.395 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.13086251099999999 2024-09-13T14:46:57.395 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.395 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.395 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:46:57.395 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.224452+0000", 2024-09-13T14:46:57.395 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.022033740999999999 2024-09-13T14:46:57.395 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.396 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.396 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.396 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.962029+0000", 2024-09-13T14:46:57.396 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7375775129999997 2024-09-13T14:46:57.396 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.396 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.396 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:46:57.396 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:45.976076+0000", 2024-09-13T14:46:57.396 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014046856999999999 2024-09-13T14:46:57.396 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.396 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.396 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:46:57.396 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.000274+0000", 2024-09-13T14:46:57.396 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024197749000000001 2024-09-13T14:46:57.397 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.397 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.397 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:57.397 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.099261+0000", 2024-09-13T14:46:57.397 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.098987027000000005 2024-09-13T14:46:57.397 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.397 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.397 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:46:57.397 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.101138+0000", 2024-09-13T14:46:57.397 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0018768680000000001 2024-09-13T14:46:57.397 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.397 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.397 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:57.397 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.111923+0000", 2024-09-13T14:46:57.398 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010785444999999999 2024-09-13T14:46:57.398 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.398 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.398 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:46:57.398 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.124357+0000", 2024-09-13T14:46:57.398 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012433962999999999 2024-09-13T14:46:57.398 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.398 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.398 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:46:57.398 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.179397+0000", 2024-09-13T14:46:57.398 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.055040402000000002 2024-09-13T14:46:57.398 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:57.398 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:46:57.398 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:57.399 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.399 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.399 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113763 7.e 7:7a44c088:::benchmark_data_smithi102_227395_object948:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-13T14:46:57.399 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:41.987161+0000", 2024-09-13T14:46:57.399 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 555.31218185399996, 2024-09-13T14:46:57.399 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.450241471, 2024-09-13T14:46:57.399 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:46:57.399 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:57.399 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:46:57.399 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:46:57.399 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:46:57.399 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113763 2024-09-13T14:46:57.399 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.399 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:46:57.400 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.400 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:46:57.400 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.987161+0000", 2024-09-13T14:46:57.400 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:57.400 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.400 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.400 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:46:57.400 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.987161+0000", 2024-09-13T14:46:57.400 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:57.400 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.400 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.400 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:46:57.400 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.987159+0000", 2024-09-13T14:46:57.400 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:46:57.400 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.401 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.401 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:46:57.401 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.987166+0000", 2024-09-13T14:46:57.401 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5080000000000002e-06 2024-09-13T14:46:57.401 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.401 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.401 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:46:57.401 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.987178+0000", 2024-09-13T14:46:57.401 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2441e-05 2024-09-13T14:46:57.401 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.401 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.401 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:46:57.401 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:41.987184+0000", 2024-09-13T14:46:57.401 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5879999999999997e-06 2024-09-13T14:46:57.402 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.402 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.402 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.402 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.073662+0000", 2024-09-13T14:46:57.402 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.086478476999999998 2024-09-13T14:46:57.402 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.402 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.402 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:46:57.402 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.099105+0000", 2024-09-13T14:46:57.402 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.025442959000000001 2024-09-13T14:46:57.402 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.402 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.402 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.402 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.302626+0000", 2024-09-13T14:46:57.403 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.203520283 2024-09-13T14:46:57.403 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.403 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.403 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:46:57.403 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.324932+0000", 2024-09-13T14:46:57.403 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.022305898000000001 2024-09-13T14:46:57.403 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.403 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.403 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.403 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.224175+0000", 2024-09-13T14:46:57.403 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8992434930000002 2024-09-13T14:46:57.403 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.403 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.403 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:46:57.403 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.241799+0000", 2024-09-13T14:46:57.404 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017623608999999998 2024-09-13T14:46:57.404 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.404 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.404 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:46:57.404 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.274010+0000", 2024-09-13T14:46:57.404 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.032211245999999999 2024-09-13T14:46:57.404 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.404 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.404 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:57.404 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.364856+0000", 2024-09-13T14:46:57.404 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.090845578999999996 2024-09-13T14:46:57.404 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.404 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.404 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:57.405 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.372367+0000", 2024-09-13T14:46:57.405 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0075111149999999996 2024-09-13T14:46:57.405 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.405 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.405 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:46:57.405 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.374485+0000", 2024-09-13T14:46:57.405 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0021184810000000002 2024-09-13T14:46:57.405 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.405 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.405 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:46:57.405 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.386174+0000", 2024-09-13T14:46:57.405 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011689181999999999 2024-09-13T14:46:57.405 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.406 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.406 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:46:57.406 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.437402+0000", 2024-09-13T14:46:57.406 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.051227946000000003 2024-09-13T14:46:57.406 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:57.406 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:46:57.406 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:57.406 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.406 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.406 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113767 7.e 7:78506392:::benchmark_data_smithi102_227395_object952:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-13T14:46:57.406 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:42.006143+0000", 2024-09-13T14:46:57.406 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 555.29320001300005, 2024-09-13T14:46:57.406 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0048697080000002, 2024-09-13T14:46:57.406 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:46:57.407 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:57.407 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:46:57.407 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:46:57.407 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:46:57.407 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113767 2024-09-13T14:46:57.407 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.407 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:46:57.407 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.407 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:46:57.407 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.006143+0000", 2024-09-13T14:46:57.407 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:57.407 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.407 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.407 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:46:57.408 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.006143+0000", 2024-09-13T14:46:57.408 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:57.408 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.408 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.408 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:46:57.408 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.006141+0000", 2024-09-13T14:46:57.408 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:46:57.408 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.408 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.408 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:46:57.408 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.006149+0000", 2024-09-13T14:46:57.408 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8699999999999992e-06 2024-09-13T14:46:57.408 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.409 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.409 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:46:57.409 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.006161+0000", 2024-09-13T14:46:57.409 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1939999999999999e-05 2024-09-13T14:46:57.409 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.409 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.409 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:46:57.409 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.006168+0000", 2024-09-13T14:46:57.409 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4390000000000003e-06 2024-09-13T14:46:57.409 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.409 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.409 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.409 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.101455+0000", 2024-09-13T14:46:57.409 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.095286518000000001 2024-09-13T14:46:57.410 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.410 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.410 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:46:57.410 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.125347+0000", 2024-09-13T14:46:57.410 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.023891539999999999 2024-09-13T14:46:57.410 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.410 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.410 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.410 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.397577+0000", 2024-09-13T14:46:57.410 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.27223060100000002 2024-09-13T14:46:57.410 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.410 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.410 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:46:57.410 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.417123+0000", 2024-09-13T14:46:57.411 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.019546392999999999 2024-09-13T14:46:57.411 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.411 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.411 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.411 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.791931+0000", 2024-09-13T14:46:57.411 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3748072650000003 2024-09-13T14:46:57.411 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.411 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.411 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:46:57.411 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.809411+0000", 2024-09-13T14:46:57.411 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017480623000000001 2024-09-13T14:46:57.411 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.411 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.411 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:46:57.412 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.831143+0000", 2024-09-13T14:46:57.412 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.021731145 2024-09-13T14:46:57.412 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.412 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.412 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:57.412 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.959387+0000", 2024-09-13T14:46:57.412 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.12824418500000001 2024-09-13T14:46:57.412 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.412 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.412 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:57.412 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.964317+0000", 2024-09-13T14:46:57.412 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0049304350000000004 2024-09-13T14:46:57.412 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.412 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.413 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:46:57.413 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.966146+0000", 2024-09-13T14:46:57.413 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001829041 2024-09-13T14:46:57.413 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.413 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.413 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:46:57.413 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.972871+0000", 2024-09-13T14:46:57.413 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0067250080000000002 2024-09-13T14:46:57.413 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.413 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.413 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:46:57.413 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.011012+0000", 2024-09-13T14:46:57.413 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.038141126999999997 2024-09-13T14:46:57.413 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:57.414 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:46:57.414 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:57.414 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.414 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.414 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113770 7.e 7:7f1c5c68:::benchmark_data_smithi102_227395_object955:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-13T14:46:57.414 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:42.134010+0000", 2024-09-13T14:46:57.414 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 555.16533300599997, 2024-09-13T14:46:57.414 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9464087430000001, 2024-09-13T14:46:57.414 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:46:57.414 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:57.414 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:46:57.414 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:46:57.414 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:46:57.414 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113770 2024-09-13T14:46:57.415 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.415 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:46:57.415 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.415 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:46:57.415 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.134010+0000", 2024-09-13T14:46:57.415 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:57.415 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.415 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.415 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:46:57.415 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.134010+0000", 2024-09-13T14:46:57.415 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:57.415 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.415 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.416 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:46:57.416 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.134008+0000", 2024-09-13T14:46:57.416 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:46:57.416 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.416 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.416 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:46:57.416 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.134019+0000", 2024-09-13T14:46:57.416 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0458e-05 2024-09-13T14:46:57.416 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.416 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.416 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:46:57.416 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.134030+0000", 2024-09-13T14:46:57.417 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1311999999999999e-05 2024-09-13T14:46:57.417 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.417 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.417 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:46:57.417 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.134036+0000", 2024-09-13T14:46:57.417 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0229999999999998e-06 2024-09-13T14:46:57.417 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.417 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.417 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.417 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.419198+0000", 2024-09-13T14:46:57.417 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.28516167399999998 2024-09-13T14:46:57.417 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.417 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.417 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:46:57.418 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.440868+0000", 2024-09-13T14:46:57.418 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.021670779000000001 2024-09-13T14:46:57.418 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.418 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.418 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.418 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.554028+0000", 2024-09-13T14:46:57.418 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.113159653 2024-09-13T14:46:57.418 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.418 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.418 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:46:57.418 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.576302+0000", 2024-09-13T14:46:57.418 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.022274200000000001 2024-09-13T14:46:57.418 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.418 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.419 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.419 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.890592+0000", 2024-09-13T14:46:57.419 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3142900390000003 2024-09-13T14:46:57.420 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.421 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.421 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:46:57.421 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.904275+0000", 2024-09-13T14:46:57.421 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013682952 2024-09-13T14:46:57.421 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.421 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.421 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:46:57.421 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:46.931383+0000", 2024-09-13T14:46:57.421 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.027107368 2024-09-13T14:46:57.421 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.421 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.421 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:46:57.421 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.012801+0000", 2024-09-13T14:46:57.422 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.081418328999999998 2024-09-13T14:46:57.422 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.422 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.422 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:57.422 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.018242+0000", 2024-09-13T14:46:57.422 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0054409819999999996 2024-09-13T14:46:57.422 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.422 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.422 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:57.422 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.025406+0000", 2024-09-13T14:46:57.422 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0071640150000000001 2024-09-13T14:46:57.422 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.422 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.422 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:46:57.422 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.036309+0000", 2024-09-13T14:46:57.423 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010902949 2024-09-13T14:46:57.423 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.423 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.423 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:46:57.423 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.080418+0000", 2024-09-13T14:46:57.423 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.044109451000000001 2024-09-13T14:46:57.423 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:57.423 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:46:57.423 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:57.423 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.423 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.423 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113800 7.e 7:745cc90f:::benchmark_data_smithi102_227395_object985:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-13T14:46:57.423 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:42.931912+0000", 2024-09-13T14:46:57.424 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 554.36743086000001, 2024-09-13T14:46:57.424 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3762813339999997, 2024-09-13T14:46:57.424 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:46:57.424 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:57.424 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:46:57.424 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:46:57.424 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:46:57.424 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113800 2024-09-13T14:46:57.424 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.424 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:46:57.424 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.424 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:46:57.424 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.931912+0000", 2024-09-13T14:46:57.424 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:57.425 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.425 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.425 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:46:57.425 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.931912+0000", 2024-09-13T14:46:57.425 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:57.425 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.425 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.425 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:46:57.425 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.931910+0000", 2024-09-13T14:46:57.425 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:46:57.425 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.425 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.425 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:46:57.426 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.931924+0000", 2024-09-13T14:46:57.426 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3915e-05 2024-09-13T14:46:57.426 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.426 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.426 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:46:57.426 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.931937+0000", 2024-09-13T14:46:57.426 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2835e-05 2024-09-13T14:46:57.426 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.426 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.426 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:46:57.426 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:42.931942+0000", 2024-09-13T14:46:57.426 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.181e-06 2024-09-13T14:46:57.426 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.426 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.427 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.427 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:43.100264+0000", 2024-09-13T14:46:57.427 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.16832238599999999 2024-09-13T14:46:57.427 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.427 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.427 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:46:57.427 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:43.124629+0000", 2024-09-13T14:46:57.427 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024364495 2024-09-13T14:46:57.427 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.427 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.427 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.427 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:43.260475+0000", 2024-09-13T14:46:57.427 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.13584602100000001 2024-09-13T14:46:57.428 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.428 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.428 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:46:57.428 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:43.281800+0000", 2024-09-13T14:46:57.428 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.021324864999999998 2024-09-13T14:46:57.428 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.428 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.428 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.428 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.133937+0000", 2024-09-13T14:46:57.428 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8521372940000003 2024-09-13T14:46:57.428 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.428 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.428 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:46:57.429 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.152762+0000", 2024-09-13T14:46:57.429 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018825346 2024-09-13T14:46:57.429 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.429 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.429 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:46:57.429 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.167553+0000", 2024-09-13T14:46:57.429 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014790905 2024-09-13T14:46:57.429 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.429 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.429 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:46:57.429 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.272976+0000", 2024-09-13T14:46:57.429 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.105422497 2024-09-13T14:46:57.429 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.429 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.430 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:57.430 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.275989+0000", 2024-09-13T14:46:57.430 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0030131899999999998 2024-09-13T14:46:57.430 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.430 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.430 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:57.430 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.279778+0000", 2024-09-13T14:46:57.430 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0037891349999999999 2024-09-13T14:46:57.430 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.430 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.430 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:46:57.430 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.285756+0000", 2024-09-13T14:46:57.430 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0059774989999999998 2024-09-13T14:46:57.430 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.431 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.431 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:46:57.431 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.308193+0000", 2024-09-13T14:46:57.431 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.022437436000000002 2024-09-13T14:46:57.431 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:57.431 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:46:57.431 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:57.431 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.431 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.431 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113838 7.e 7:725b9897:::benchmark_data_smithi102_227395_object1023:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-13T14:46:57.431 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:43.215448+0000", 2024-09-13T14:46:57.431 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 554.08389416299997, 2024-09-13T14:46:57.431 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1232826060000001, 2024-09-13T14:46:57.432 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:46:57.432 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:57.432 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:46:57.432 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:46:57.432 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:46:57.432 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113838 2024-09-13T14:46:57.432 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.432 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:46:57.432 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.432 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:46:57.432 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:43.215448+0000", 2024-09-13T14:46:57.432 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:57.432 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.432 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.433 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:46:57.433 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:43.215448+0000", 2024-09-13T14:46:57.433 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:57.433 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.433 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.433 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:46:57.433 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:43.215447+0000", 2024-09-13T14:46:57.433 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:46:57.433 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.433 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.433 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:46:57.433 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:43.215455+0000", 2024-09-13T14:46:57.433 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8089999999999993e-06 2024-09-13T14:46:57.433 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.434 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.434 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:46:57.434 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:43.215469+0000", 2024-09-13T14:46:57.434 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4056000000000001e-05 2024-09-13T14:46:57.434 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.434 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.434 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:46:57.434 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:43.215476+0000", 2024-09-13T14:46:57.434 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3749999999999997e-06 2024-09-13T14:46:57.434 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.434 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.434 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.434 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:43.413654+0000", 2024-09-13T14:46:57.435 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.19817833800000001 2024-09-13T14:46:57.435 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.435 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.435 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:46:57.435 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:43.438154+0000", 2024-09-13T14:46:57.435 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024499441 2024-09-13T14:46:57.435 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.435 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.435 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.435 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:43.627551+0000", 2024-09-13T14:46:57.435 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.189397542 2024-09-13T14:46:57.435 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.435 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.436 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:46:57.436 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:43.651356+0000", 2024-09-13T14:46:57.436 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.023804698999999999 2024-09-13T14:46:57.436 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.436 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.436 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.436 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.210574+0000", 2024-09-13T14:46:57.436 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5592174459999999 2024-09-13T14:46:57.436 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.436 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.436 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:46:57.436 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.219313+0000", 2024-09-13T14:46:57.436 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0087393239999999997 2024-09-13T14:46:57.437 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.437 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.437 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:46:57.437 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.233331+0000", 2024-09-13T14:46:57.437 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014018256999999999 2024-09-13T14:46:57.437 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.437 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.437 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:46:57.437 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.309091+0000", 2024-09-13T14:46:57.437 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.075759601999999995 2024-09-13T14:46:57.437 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.437 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.438 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:57.438 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.311847+0000", 2024-09-13T14:46:57.438 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0027565770000000001 2024-09-13T14:46:57.438 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.438 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.438 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:57.438 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.315219+0000", 2024-09-13T14:46:57.438 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0033721850000000002 2024-09-13T14:46:57.439 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.439 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.439 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:46:57.439 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.319648+0000", 2024-09-13T14:46:57.439 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0044280630000000003 2024-09-13T14:46:57.439 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.439 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.439 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:46:57.439 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.338731+0000", 2024-09-13T14:46:57.439 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.019083547999999999 2024-09-13T14:46:57.439 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:57.439 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:46:57.440 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:57.440 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.440 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.440 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113868 7.e 7:729791b6:::benchmark_data_smithi102_227395_object1053:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-13T14:46:57.440 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:44.019108+0000", 2024-09-13T14:46:57.440 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 553.28023478499995, 2024-09-13T14:46:57.440 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6053882540000002, 2024-09-13T14:46:57.440 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:46:57.440 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:57.440 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:46:57.440 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:46:57.440 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:46:57.440 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113868 2024-09-13T14:46:57.441 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.441 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:46:57.441 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.441 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:46:57.441 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.019108+0000", 2024-09-13T14:46:57.441 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:57.441 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.441 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.441 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:46:57.441 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.019108+0000", 2024-09-13T14:46:57.441 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:57.441 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.442 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.442 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:46:57.442 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.019107+0000", 2024-09-13T14:46:57.442 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:46:57.442 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.442 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.442 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:46:57.442 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.019112+0000", 2024-09-13T14:46:57.442 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7239999999999997e-06 2024-09-13T14:46:57.442 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.442 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.442 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:46:57.442 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.019122+0000", 2024-09-13T14:46:57.442 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6539999999999994e-06 2024-09-13T14:46:57.443 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.443 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.443 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:46:57.443 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.019128+0000", 2024-09-13T14:46:57.443 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8749999999999997e-06 2024-09-13T14:46:57.443 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.443 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.443 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.443 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.114948+0000", 2024-09-13T14:46:57.443 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.095819748999999996 2024-09-13T14:46:57.443 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.443 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.443 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:46:57.444 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.144054+0000", 2024-09-13T14:46:57.444 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.029106476999999999 2024-09-13T14:46:57.444 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.444 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.444 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.444 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.443107+0000", 2024-09-13T14:46:57.444 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.29905287200000003 2024-09-13T14:46:57.444 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.444 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.444 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:46:57.444 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.470900+0000", 2024-09-13T14:46:57.444 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.027792595999999999 2024-09-13T14:46:57.444 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.445 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.445 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.445 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.424154+0000", 2024-09-13T14:46:57.445 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9532546819999999 2024-09-13T14:46:57.445 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.445 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.445 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:46:57.445 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.431994+0000", 2024-09-13T14:46:57.445 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0078397650000000003 2024-09-13T14:46:57.445 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.445 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.445 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:46:57.445 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.445713+0000", 2024-09-13T14:46:57.445 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013719080999999999 2024-09-13T14:46:57.445 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.446 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.446 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:57.446 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.463051+0000", 2024-09-13T14:46:57.446 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017337680000000001 2024-09-13T14:46:57.446 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.446 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.446 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:57.446 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.465649+0000", 2024-09-13T14:46:57.446 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002598684 2024-09-13T14:46:57.446 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.446 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.446 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:46:57.446 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.600753+0000", 2024-09-13T14:46:57.447 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.13510388200000001 2024-09-13T14:46:57.447 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.447 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.447 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:46:57.447 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.605058+0000", 2024-09-13T14:46:57.447 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0043048879999999998 2024-09-13T14:46:57.447 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.447 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.447 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:46:57.447 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.624496+0000", 2024-09-13T14:46:57.447 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.019437904999999998 2024-09-13T14:46:57.447 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:57.447 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:46:57.447 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:57.448 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.448 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.448 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113882 7.e 7:73c217af:::benchmark_data_smithi102_227395_object1067:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-13T14:46:57.448 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:44.061776+0000", 2024-09-13T14:46:57.448 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 553.23756645699996, 2024-09-13T14:46:57.448 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.59134807, 2024-09-13T14:46:57.448 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:46:57.448 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:57.448 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:46:57.448 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:46:57.448 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:46:57.448 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113882 2024-09-13T14:46:57.448 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.449 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:46:57.449 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.449 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:46:57.449 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.061776+0000", 2024-09-13T14:46:57.449 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:57.449 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.449 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.449 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:46:57.449 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.061776+0000", 2024-09-13T14:46:57.449 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:57.449 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.449 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.450 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:46:57.450 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.061775+0000", 2024-09-13T14:46:57.450 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:46:57.450 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.450 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.450 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:46:57.450 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.061781+0000", 2024-09-13T14:46:57.450 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0839999999999997e-06 2024-09-13T14:46:57.450 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.450 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.450 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:46:57.450 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.061793+0000", 2024-09-13T14:46:57.451 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1939e-05 2024-09-13T14:46:57.451 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.451 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.451 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:46:57.451 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.061801+0000", 2024-09-13T14:46:57.451 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2490000000000001e-06 2024-09-13T14:46:57.451 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.451 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.451 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.451 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.146275+0000", 2024-09-13T14:46:57.451 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.084473626999999996 2024-09-13T14:46:57.451 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.451 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.451 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:46:57.452 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.170652+0000", 2024-09-13T14:46:57.452 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024377374 2024-09-13T14:46:57.452 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.452 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.452 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.452 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.548899+0000", 2024-09-13T14:46:57.452 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.37824670900000001 2024-09-13T14:46:57.452 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.452 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.452 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:46:57.452 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.610957+0000", 2024-09-13T14:46:57.452 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.062058073999999998 2024-09-13T14:46:57.453 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.453 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.453 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.453 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.488800+0000", 2024-09-13T14:46:57.453 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8778428460000001 2024-09-13T14:46:57.453 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.453 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.453 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:46:57.453 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.496393+0000", 2024-09-13T14:46:57.453 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0075932910000000003 2024-09-13T14:46:57.453 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.453 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.453 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:46:57.454 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.509330+0000", 2024-09-13T14:46:57.454 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012937051999999999 2024-09-13T14:46:57.454 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.454 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.454 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:46:57.454 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.625334+0000", 2024-09-13T14:46:57.454 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.116003785 2024-09-13T14:46:57.454 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.454 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.454 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:57.454 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.627836+0000", 2024-09-13T14:46:57.454 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0025019109999999999 2024-09-13T14:46:57.454 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.455 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.455 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:57.455 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.630456+0000", 2024-09-13T14:46:57.455 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0026199159999999999 2024-09-13T14:46:57.455 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.455 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.455 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:46:57.455 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.634782+0000", 2024-09-13T14:46:57.455 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0043260019999999998 2024-09-13T14:46:57.455 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.455 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.455 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:46:57.456 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.653124+0000", 2024-09-13T14:46:57.456 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018342607 2024-09-13T14:46:57.456 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:57.456 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:46:57.456 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:57.456 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.456 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.456 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:113893 7.e 7:70855f2b:::benchmark_data_smithi102_227395_object1078:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-13T14:46:57.456 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:37:44.143011+0000", 2024-09-13T14:46:57.456 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 553.15633131100003, 2024-09-13T14:46:57.456 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.54156615, 2024-09-13T14:46:57.456 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:46:57.457 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:46:57.457 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:46:57.457 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:46:57.457 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:46:57.457 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113893 2024-09-13T14:46:57.457 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.457 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:46:57.457 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.457 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:46:57.457 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.143011+0000", 2024-09-13T14:46:57.457 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:57.457 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.458 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.458 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:46:57.458 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.143011+0000", 2024-09-13T14:46:57.458 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:46:57.458 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.458 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.458 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:46:57.458 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.143010+0000", 2024-09-13T14:46:57.458 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:46:57.458 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.458 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.458 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:46:57.459 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.143017+0000", 2024-09-13T14:46:57.459 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.0389999999999999e-06 2024-09-13T14:46:57.459 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.459 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.459 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:46:57.459 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.143028+0000", 2024-09-13T14:46:57.459 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0519999999999999e-05 2024-09-13T14:46:57.459 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.459 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.460 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:46:57.460 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.143033+0000", 2024-09-13T14:46:57.460 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5110000000000003e-06 2024-09-13T14:46:57.460 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.460 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.460 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.460 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.613507+0000", 2024-09-13T14:46:57.460 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.47047437800000003 2024-09-13T14:46:57.460 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.460 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.460 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:46:57.460 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.642684+0000", 2024-09-13T14:46:57.460 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.029176705000000001 2024-09-13T14:46:57.461 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.461 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.461 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.461 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.784620+0000", 2024-09-13T14:46:57.461 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.14193537000000001 2024-09-13T14:46:57.461 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.461 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.461 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-13T14:46:57.461 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:44.810347+0000", 2024-09-13T14:46:57.461 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.025727015999999998 2024-09-13T14:46:57.461 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.461 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.462 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:46:57.462 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.567851+0000", 2024-09-13T14:46:57.462 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7575047000000001 2024-09-13T14:46:57.462 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.462 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.462 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:46:57.462 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.575637+0000", 2024-09-13T14:46:57.462 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0077859319999999997 2024-09-13T14:46:57.462 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.462 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.462 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:46:57.462 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.590198+0000", 2024-09-13T14:46:57.462 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.01456061 2024-09-13T14:46:57.462 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.463 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.463 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:46:57.463 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.653927+0000", 2024-09-13T14:46:57.463 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.063728952000000005 2024-09-13T14:46:57.463 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.463 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.463 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:57.463 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.656466+0000", 2024-09-13T14:46:57.463 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0025390679999999998 2024-09-13T14:46:57.463 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.463 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.463 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:46:57.463 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.659062+0000", 2024-09-13T14:46:57.464 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0025957469999999998 2024-09-13T14:46:57.464 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.464 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.464 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:46:57.464 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.663437+0000", 2024-09-13T14:46:57.464 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0043751850000000002 2024-09-13T14:46:57.464 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:46:57.464 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:46:57.464 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:46:57.464 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:37:47.684577+0000", 2024-09-13T14:46:57.464 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.021140732999999998 2024-09-13T14:46:57.464 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:57.464 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:46:57.464 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:57.465 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:46:57.465 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:46:57.465 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:46:57.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:57 smithi195 ceph-mon[104262]: 1.9 scrub starts 2024-09-13T14:46:57.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:57 smithi195 ceph-mon[104262]: 1.9 scrub ok 2024-09-13T14:46:57.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:57 smithi195 ceph-mon[104262]: pgmap v1122: 99 pgs: 99 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-09-13T14:46:57.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:57 smithi195 ceph-mon[104262]: 1.1 scrub starts 2024-09-13T14:46:57.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:57 smithi195 ceph-mon[104262]: 1.1 scrub ok 2024-09-13T14:46:57.466 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:57 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3099680138' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:46:57.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:57.466 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:57.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:57.467 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:57 smithi102 ceph-mon[110183]: 1.9 scrub starts 2024-09-13T14:46:57.467 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:57 smithi102 ceph-mon[110183]: 1.9 scrub ok 2024-09-13T14:46:57.467 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:57 smithi102 ceph-mon[110183]: pgmap v1122: 99 pgs: 99 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-09-13T14:46:57.467 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:57 smithi102 ceph-mon[110183]: 1.1 scrub starts 2024-09-13T14:46:57.467 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:57 smithi102 ceph-mon[110183]: 1.1 scrub ok 2024-09-13T14:46:57.467 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:57 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3099680138' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:46:57.468 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:57 smithi102 ceph-mon[115269]: 1.9 scrub starts 2024-09-13T14:46:57.468 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:57 smithi102 ceph-mon[115269]: 1.9 scrub ok 2024-09-13T14:46:57.468 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:57 smithi102 ceph-mon[115269]: pgmap v1122: 99 pgs: 99 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-09-13T14:46:57.468 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:57 smithi102 ceph-mon[115269]: 1.1 scrub starts 2024-09-13T14:46:57.468 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:57 smithi102 ceph-mon[115269]: 1.1 scrub ok 2024-09-13T14:46:57.468 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:57 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3099680138' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:46:57.568 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:57.601 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:57.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:57.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:57.702 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:57.732 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:57.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:57.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:57.806 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 44 16 80635 80619 114.494 73.5625 0.0143678 0.00872935 2024-09-13T14:46:57.834 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:57.866 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:57.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:57.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:57.968 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:58.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:58.006 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:58.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:58.107 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:58.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:58.152 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:58.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:58.254 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:58.289 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:58.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:58.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:58.391 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:58.412 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 6, 7, 5, 2] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T14:46:58.412 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:46:58.413 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-09-13T14:46:58.413 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T14:46:58.413 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T14:46:58.413 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-09-13T14:46:58.428 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:58.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:58.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:58.477 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T14:46:58.532 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:58.608 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:58.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:58.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:58.710 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:58.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:58.737 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:58.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:58.806 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 45 16 81845 81829 113.631 75.625 0.0157603 0.00879442 2024-09-13T14:46:58.838 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:58.857 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:46:58.863 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:58.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:58.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:58.915 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:46:58.965 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:59.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:59.004 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:59.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:59.105 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:59.140 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:59.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:59.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:59.215 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is unset 2024-09-13T14:46:59.243 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:59.280 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:59.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:59.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:59.316 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:59 smithi195 ceph-mon[104262]: pgmap v1123: 99 pgs: 99 active+clean; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-13T14:46:59.316 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:46:59 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2645084446' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:46:59.382 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:59.408 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:59.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:59.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:59.510 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:59.547 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:59.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:59.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:59.619 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:59 smithi102 ceph-mon[110183]: pgmap v1123: 99 pgs: 99 active+clean; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-13T14:46:59.619 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:46:59 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2645084446' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:46:59.620 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:59 smithi102 ceph-mon[115269]: pgmap v1123: 99 pgs: 99 active+clean; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-13T14:46:59.620 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:46:59 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2645084446' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:46:59.649 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:59.680 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:59.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:59.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:59.782 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:59.806 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 46 16 83109 83093 112.878 79 0.00925562 0.00885423 2024-09-13T14:46:59.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:59.813 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:46:59.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:59.914 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:59.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:59.948 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:46:59.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:00.049 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:00.083 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:00.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:00.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:00.167 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:47:00.185 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:00.222 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:00.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:00.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:00.324 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:00.373 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:00.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:00.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:00.477 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:00.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:00 smithi195 ceph-mon[104262]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:47:00.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:00 smithi195 ceph-mon[104262]: Cluster is now healthy 2024-09-13T14:47:00.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:00 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2645084446' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:47:00.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:00 smithi195 ceph-mon[104262]: osdmap e325: 8 total, 8 up, 7 in 2024-09-13T14:47:00.512 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:00.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:00.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:00.515 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:00.614 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:00.655 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:00 smithi102 ceph-mon[110183]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:47:00.655 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:00 smithi102 ceph-mon[110183]: Cluster is now healthy 2024-09-13T14:47:00.655 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:00 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2645084446' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:47:00.655 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:00 smithi102 ceph-mon[110183]: osdmap e325: 8 total, 8 up, 7 in 2024-09-13T14:47:00.655 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:00 smithi102 ceph-mon[115269]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:47:00.656 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:00 smithi102 ceph-mon[115269]: Cluster is now healthy 2024-09-13T14:47:00.656 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:00 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2645084446' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:47:00.656 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:00 smithi102 ceph-mon[115269]: osdmap e325: 8 total, 8 up, 7 in 2024-09-13T14:47:00.658 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:00.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:00.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:00.760 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:00.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:00.797 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:00.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:00.806 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 47 16 84402 84386 112.195 80.8125 0.0121374 0.00890854 2024-09-13T14:47:00.899 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:00.936 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:00.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:00.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:01.037 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:01.072 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:01.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:01.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:01.173 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:01.209 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:01.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:01.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:01.310 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:01.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:01.345 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:01.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:01.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:01 smithi102 ceph-mon[110183]: pgmap v1125: 99 pgs: 99 active+clean; 4.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 81 MiB/s wr, 1.28k op/s 2024-09-13T14:47:01.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:01 smithi102 ceph-mon[115269]: pgmap v1125: 99 pgs: 99 active+clean; 4.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 81 MiB/s wr, 1.28k op/s 2024-09-13T14:47:01.446 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:01.480 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:01.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:01.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:01.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:01 smithi195 ceph-mon[104262]: pgmap v1125: 99 pgs: 99 active+clean; 4.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 81 MiB/s wr, 1.28k op/s 2024-09-13T14:47:01.582 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:01.616 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:01.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:01.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:01.718 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:01.759 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:01.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:01.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:01.806 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 48 16 85567 85551 111.375 72.8125 0.00300863 0.00896939 2024-09-13T14:47:01.862 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:01.882 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:01.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:01.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:01.984 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:02.008 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:47:02.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:02.015 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:02.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:02.116 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:02.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:02.155 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:02.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:02.256 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:02.288 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:02.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:02.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:02.390 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:02.421 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:02.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:02.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:02.523 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:02.555 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:02.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:02.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:02.657 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:02.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:02.691 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:02.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:02.792 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:02.806 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 49 16 86656 86640 110.491 68.0625 0.0111906 0.00904549 2024-09-13T14:47:02.834 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:02.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:02.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:02.936 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:02.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:02.966 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:02.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:03.067 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:03.116 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:03.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:03.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:03.217 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:03.255 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:03.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:03.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:03.264 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:03 smithi195 ceph-mon[104262]: pgmap v1126: 99 pgs: 99 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 81 MiB/s wr, 1.28k op/s 2024-09-13T14:47:03.357 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:03.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:03.387 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:03.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:03.488 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:03.519 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:03.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:03.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:03.599 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:47:03.599 INFO:teuthology.orchestra.run.smithi195.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-13T14:47:03.599 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:47:03.621 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:03.654 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:03.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:03.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:03.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:03 smithi102 ceph-mon[110183]: pgmap v1126: 99 pgs: 99 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 81 MiB/s wr, 1.28k op/s 2024-09-13T14:47:03.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:03 smithi102 ceph-mon[115269]: pgmap v1126: 99 pgs: 99 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 81 MiB/s wr, 1.28k op/s 2024-09-13T14:47:03.756 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:03.789 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:03.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:03.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:03.806 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 50 16 87765 87749 109.667 69.3125 0.0082984 0.00911376 2024-09-13T14:47:03.891 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:03.926 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:03.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:03.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:04.028 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:04.049 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:04.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:04.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:04.151 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:04.194 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:04.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:04.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:04.296 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:04.334 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:04.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:04.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:04.437 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:04.467 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:04.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:04.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:04.569 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:04.603 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:04.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:04.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:04.704 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:47:04.704 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-13T14:47:04.704 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:04.704 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.46157.0:88690 8.c e325/304 8:3353d8bd:::benchmark_data_smithi102_313275_object88689:head v 325'5434, mlcod=325'5432)", 2024-09-13T14:47:04.705 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:47:04.695196+0000", 2024-09-13T14:47:04.705 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.0073709709999999996, 2024-09-13T14:47:04.705 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0073988739999999997, 2024-09-13T14:47:04.705 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:47:04.705 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T14:47:04.705 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:47:04.705 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:04.705 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:47:04.705 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.695196+0000", 2024-09-13T14:47:04.705 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:47:04.705 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:04.705 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:04.705 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:47:04.706 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.695196+0000", 2024-09-13T14:47:04.706 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:47:04.706 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:04.706 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:04.706 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:47:04.706 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.695195+0000", 2024-09-13T14:47:04.706 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:47:04.706 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:04.706 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:04.706 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:47:04.706 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.695233+0000", 2024-09-13T14:47:04.706 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8488000000000001e-05 2024-09-13T14:47:04.706 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:04.707 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:04.707 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:47:04.707 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.695246+0000", 2024-09-13T14:47:04.707 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2608e-05 2024-09-13T14:47:04.707 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:04.707 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:04.707 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:47:04.707 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.695254+0000", 2024-09-13T14:47:04.707 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9310000000000008e-06 2024-09-13T14:47:04.707 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:04.707 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:04.707 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:47:04.707 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.695410+0000", 2024-09-13T14:47:04.708 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000156198 2024-09-13T14:47:04.708 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:04.708 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:04.708 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:47:04.708 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.695429+0000", 2024-09-13T14:47:04.708 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8681e-05 2024-09-13T14:47:04.708 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:47:04.708 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:47:04.708 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:47:04.708 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:04.708 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:04.708 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.46157.0:88696 8.5 e325/304 8:ae49a27b:::benchmark_data_smithi102_313275_object88695:head v 325'5499, mlcod=325'5496)", 2024-09-13T14:47:04.709 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:47:04.698167+0000", 2024-09-13T14:47:04.709 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.0043997619999999998, 2024-09-13T14:47:04.709 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0044908980000000001, 2024-09-13T14:47:04.709 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:47:04.709 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T14:47:04.709 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:47:04.709 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:04.709 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:47:04.709 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.698167+0000", 2024-09-13T14:47:04.709 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:47:04.709 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:04.709 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:04.710 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:47:04.710 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.698167+0000", 2024-09-13T14:47:04.710 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:47:04.710 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:04.710 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:04.710 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:47:04.710 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.698166+0000", 2024-09-13T14:47:04.710 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:47:04.710 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:04.710 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:04.710 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:47:04.710 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.698190+0000", 2024-09-13T14:47:04.710 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4409999999999998e-05 2024-09-13T14:47:04.710 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:04.711 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:04.711 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:47:04.711 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.698204+0000", 2024-09-13T14:47:04.711 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3030999999999999e-05 2024-09-13T14:47:04.711 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:04.711 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:04.711 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:47:04.711 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.698211+0000", 2024-09-13T14:47:04.711 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4379999999999998e-06 2024-09-13T14:47:04.711 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:04.711 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:04.711 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:47:04.711 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.698429+0000", 2024-09-13T14:47:04.712 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000218128 2024-09-13T14:47:04.712 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:04.712 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:04.712 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:47:04.712 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.698450+0000", 2024-09-13T14:47:04.712 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1376e-05 2024-09-13T14:47:04.712 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:47:04.712 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:47:04.712 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:47:04.712 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:04.712 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:04.712 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.46157.0:88697 8.c e325/304 8:32a28909:::benchmark_data_smithi102_313275_object88696:head v 325'5435, mlcod=325'5432)", 2024-09-13T14:47:04.712 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:47:04.698417+0000", 2024-09-13T14:47:04.713 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.0041496679999999996, 2024-09-13T14:47:04.713 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.004294043, 2024-09-13T14:47:04.713 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:47:04.713 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T14:47:04.713 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:47:04.713 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:04.713 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:47:04.713 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.698417+0000", 2024-09-13T14:47:04.713 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:47:04.713 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:04.713 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:04.713 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:47:04.714 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.698417+0000", 2024-09-13T14:47:04.714 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:47:04.714 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:04.714 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:04.714 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:47:04.714 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.698417+0000", 2024-09-13T14:47:04.714 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:47:04.714 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:04.714 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:04.714 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:47:04.714 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.698452+0000", 2024-09-13T14:47:04.714 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5150000000000001e-05 2024-09-13T14:47:04.714 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:04.714 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:04.715 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:47:04.715 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.698462+0000", 2024-09-13T14:47:04.715 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0146000000000001e-05 2024-09-13T14:47:04.715 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:04.715 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:04.715 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:47:04.715 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.698468+0000", 2024-09-13T14:47:04.715 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5270000000000004e-06 2024-09-13T14:47:04.715 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:04.715 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:04.715 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:47:04.715 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.699925+0000", 2024-09-13T14:47:04.715 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0014562760000000001 2024-09-13T14:47:04.716 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:04.716 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:04.716 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:47:04.716 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.699944+0000", 2024-09-13T14:47:04.716 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9351000000000002e-05 2024-09-13T14:47:04.716 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:47:04.716 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:47:04.716 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:47:04.716 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:04.716 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:04.716 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.46157.0:88703 8.6 e325/304 8:6d8adedf:::benchmark_data_smithi102_313275_object88702:head v 325'5576, mlcod=325'5574)", 2024-09-13T14:47:04.716 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:47:04.702187+0000", 2024-09-13T14:47:04.716 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.000380033, 2024-09-13T14:47:04.717 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00057663300000000001, 2024-09-13T14:47:04.717 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:47:04.717 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T14:47:04.717 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:47:04.717 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:04.717 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:47:04.717 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.702187+0000", 2024-09-13T14:47:04.717 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:47:04.717 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:04.717 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:04.717 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:47:04.717 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.702187+0000", 2024-09-13T14:47:04.717 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:47:04.718 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:04.718 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:04.718 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:47:04.718 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.702186+0000", 2024-09-13T14:47:04.718 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:47:04.718 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:04.718 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:04.718 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:47:04.718 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.702216+0000", 2024-09-13T14:47:04.718 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0419000000000001e-05 2024-09-13T14:47:04.718 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:04.718 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:04.718 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:47:04.719 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.702230+0000", 2024-09-13T14:47:04.719 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3767e-05 2024-09-13T14:47:04.719 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:04.719 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:04.719 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:47:04.719 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.702237+0000", 2024-09-13T14:47:04.719 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7889999999999997e-06 2024-09-13T14:47:04.719 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:04.719 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:04.719 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:47:04.719 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.702384+0000", 2024-09-13T14:47:04.719 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014716100000000001 2024-09-13T14:47:04.719 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:04.719 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:04.720 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:47:04.720 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.702435+0000", 2024-09-13T14:47:04.720 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1306e-05 2024-09-13T14:47:04.720 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:47:04.720 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:47:04.720 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:47:04.720 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:47:04.720 INFO:teuthology.orchestra.run.smithi195.stdout: ], 2024-09-13T14:47:04.720 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 4 2024-09-13T14:47:04.720 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:47:04.721 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:04.758 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:04.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:04.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:04.807 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 51 16 88829 88813 108.821 66.5 0.00450473 0.00918471 2024-09-13T14:47:04.860 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:04.891 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:04.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:04.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:04.993 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:05.032 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:05.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:05.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:05.134 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:05.147 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:05.151 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:47:05.151 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:47:05.180 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:05.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:05.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:05.193 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:47:05.193 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:47:05.226 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:47:05.227 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:47:05.264 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:47:05.264 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:47:05.283 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:05.297 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:47:05.297 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:47:05.318 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:05.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:05.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:05.329 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:47:05.329 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:47:05.365 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:47:05.365 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:47:05.391 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:47:05.391 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:47:05.407 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:05 smithi195 ceph-mon[104262]: pgmap v1127: 99 pgs: 99 active+clean; 5.3 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-13T14:47:05.407 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:05 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2309495398' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:47:05.420 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:05.448 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T14:47:05.463 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:05.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:05.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:05.564 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:05.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:05.608 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:05.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:05.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:05 smithi102 ceph-mon[110183]: pgmap v1127: 99 pgs: 99 active+clean; 5.3 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-13T14:47:05.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:05 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2309495398' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:47:05.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:05 smithi102 ceph-mon[115269]: pgmap v1127: 99 pgs: 99 active+clean; 5.3 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-13T14:47:05.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:05 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2309495398' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:47:05.709 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:05.740 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:05.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:05.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:05.807 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 52 16 89988 89972 108.121 72.4375 0.012797 0.00924351 2024-09-13T14:47:05.836 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:47:05.842 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:05.876 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:05.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:05.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:05.978 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:06.012 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:06.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:06.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:06.114 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:06.148 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:06.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:06.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:06.249 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:06.290 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:06.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:06.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:06.393 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:06.425 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:06.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:06.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:06.527 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:06.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:06.576 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:06.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:06.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:06 smithi102 ceph-mon[110183]: 1.5 scrub starts 2024-09-13T14:47:06.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:06 smithi102 ceph-mon[110183]: 1.5 scrub ok 2024-09-13T14:47:06.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:06 smithi102 ceph-mon[115269]: 1.5 scrub starts 2024-09-13T14:47:06.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:06 smithi102 ceph-mon[115269]: 1.5 scrub ok 2024-09-13T14:47:06.669 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:47:06 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:47:06] "GET /metrics HTTP/1.1" 200 36521 "" "Prometheus/2.51.0" 2024-09-13T14:47:06.677 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:06.711 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:06.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:06.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:06.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:06 smithi195 ceph-mon[104262]: 1.5 scrub starts 2024-09-13T14:47:06.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:06 smithi195 ceph-mon[104262]: 1.5 scrub ok 2024-09-13T14:47:06.807 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 53 16 91211 91195 107.523 76.4375 0.0208201 0.00929589 2024-09-13T14:47:06.813 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:06.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:06.844 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:06.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:06.944 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:06.977 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:06.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:06.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:07.079 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:07.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:07.114 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:07.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:07.215 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:07.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:07.249 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:07.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:07.350 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:07.380 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:07.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:07.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:07.482 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:07.491 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:07 smithi102 ceph-mon[110183]: pgmap v1128: 99 pgs: 99 active+clean; 5.3 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-13T14:47:07.491 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:07 smithi102 ceph-mon[115269]: pgmap v1128: 99 pgs: 99 active+clean; 5.3 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-13T14:47:07.520 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:07.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:07.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:07.621 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:07 smithi195 ceph-mon[104262]: pgmap v1128: 99 pgs: 99 active+clean; 5.3 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-13T14:47:07.622 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:07.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:07.672 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:07.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:07.773 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:07.804 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:07.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:07.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:07.807 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 54 16 92231 92215 106.712 63.75 0.00959028 0.00936582 2024-09-13T14:47:07.905 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:07.942 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:07.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:07.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:08.044 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:08.081 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:08.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:08.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:08.182 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:08.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:08.220 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:08.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:08.238 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:47:08.321 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:08.370 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:08.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:08.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:08.471 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:08.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:08.504 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:08.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:08.605 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:08.609 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:08.648 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:08.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:08.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:08.750 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:08.790 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:08.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:08.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:08.807 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 55 16 93377 93361 106.074 71.625 0.0272658 0.00942271 2024-09-13T14:47:08.892 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:08.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:08.939 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:08.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:09.040 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:09.068 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:09.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:09.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:09.130 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:47:09.130 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:47:09.169 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:09.200 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:09.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:09.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:09.284 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is set 2024-09-13T14:47:09.302 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:09.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:09.342 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:09.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:09.388 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 6, 7, 5, 2] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T14:47:09.388 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:47:09.388 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-13T14:47:09.388 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:47:09.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:09 smithi102 ceph-mon[115269]: pgmap v1129: 99 pgs: 99 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-09-13T14:47:09.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:09 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/65086541' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:47:09.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:09 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:47:09.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:09 smithi102 ceph-mon[110183]: pgmap v1129: 99 pgs: 99 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-09-13T14:47:09.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:09 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/65086541' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:47:09.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:09 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:47:09.443 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:09.475 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:09.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:09.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:09.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:09 smithi195 ceph-mon[104262]: pgmap v1129: 99 pgs: 99 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-09-13T14:47:09.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:09 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/65086541' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:47:09.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:09 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:47:09.577 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:09.620 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:09.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:09.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:09.722 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:09.747 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:09.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:09.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:09.807 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 56 16 94476 94460 105.407 68.6875 0.00889881 0.00948232 2024-09-13T14:47:09.849 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:09.887 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:09.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:09.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:09.935 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T14:47:09.988 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:10.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:10.036 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:10.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:10.138 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:10.193 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:10.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:10.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:10.294 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:10.317 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:47:10.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:10.334 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:10.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:10.435 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:10.470 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:10.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:10.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:10.549 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:10 smithi102 ceph-mon[110183]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:47:10.550 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:10 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:47:10.550 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:10 smithi102 ceph-mon[110183]: osdmap e326: 8 total, 8 up, 7 in 2024-09-13T14:47:10.550 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:10 smithi102 ceph-mon[115269]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:47:10.550 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:10 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:47:10.550 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:10 smithi102 ceph-mon[115269]: osdmap e326: 8 total, 8 up, 7 in 2024-09-13T14:47:10.571 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:10.603 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:10.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:10.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:10.615 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:10 smithi195 ceph-mon[104262]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:47:10.615 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:10 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:47:10.615 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:10 smithi195 ceph-mon[104262]: osdmap e326: 8 total, 8 up, 7 in 2024-09-13T14:47:10.704 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:10.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:10.742 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:10.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:10.807 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 57 16 95601 95585 104.791 70.3125 0.0050001 0.00953819 2024-09-13T14:47:10.843 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:10.875 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:10.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:10.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:10.977 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:11.016 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:11.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:11.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:11.118 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:11.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:11.148 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:11.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:11.249 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:11.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:11.279 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:11.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:11.380 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:11.422 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:11.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:11.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:11.524 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:11.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:11.557 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:11.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:11.658 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:11.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:11 smithi102 ceph-mon[110183]: pgmap v1131: 99 pgs: 99 active+clean; 5.6 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-09-13T14:47:11.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:11 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:47:11.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:11 smithi102 ceph-mon[115269]: pgmap v1131: 99 pgs: 99 active+clean; 5.6 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-09-13T14:47:11.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:11 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:47:11.690 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:11.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:11.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:11.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:11 smithi195 ceph-mon[104262]: pgmap v1131: 99 pgs: 99 active+clean; 5.6 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-09-13T14:47:11.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:11 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:47:11.792 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:11.807 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 58 16 96657 96641 104.122 66 0.0123369 0.00959912 2024-09-13T14:47:11.822 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:11.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:11.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:11.923 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:11.953 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:11.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:11.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:12.057 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:12.093 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:12.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:12.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:12.195 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:12.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:12.239 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:12.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:12.340 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:12.372 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:12.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:12.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:12.474 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:12.507 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:12.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:12.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:12.609 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:12.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:12.654 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:12.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:12.754 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:12.794 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:12.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:12.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:12.807 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 59 16 97806 97790 103.574 71.8125 0.0150942 0.00965073 2024-09-13T14:47:12.881 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:47:12.881 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:47:12.895 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:12.943 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:12.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:12.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:13.044 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:13.071 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:13.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:13.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:13.172 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:13.212 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:13.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:13.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:13.314 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:13.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:13.349 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:13.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:13.450 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:13.481 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:13.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:13.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:13.542 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:13 smithi102 ceph-mon[115269]: pgmap v1132: 99 pgs: 99 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 72 MiB/s wr, 1.16k op/s 2024-09-13T14:47:13.542 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:13 smithi102 ceph-mon[110183]: pgmap v1132: 99 pgs: 99 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 72 MiB/s wr, 1.16k op/s 2024-09-13T14:47:13.583 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:13.614 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:13.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:13.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:13.635 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:13 smithi195 ceph-mon[104262]: pgmap v1132: 99 pgs: 99 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 72 MiB/s wr, 1.16k op/s 2024-09-13T14:47:13.674 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T14:47:13.716 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:13.764 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:13.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:13.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:13.807 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:2024-09-13T14:47:13.807652+0000 min lat: 0.00109072 max lat: 0.310704 avg lat: 0.00970495 2024-09-13T14:47:13.807 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:47:13.807 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 60 15 98900 98885 102.989 68.4375 0.00724196 0.00970495 2024-09-13T14:47:13.866 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:13.913 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:13.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:13.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:13.971 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:14.016 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:14.065 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:47:14.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:14.074 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:14.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:14.175 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:14.208 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:14.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:14.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:14.310 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:14.343 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:14.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:14.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:14.445 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:14.477 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:14.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:14.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:14.578 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:14.630 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:14.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:14.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:14.733 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:14.754 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:14.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:14.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:14.808 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 61 16 99849 99833 102.272 59.25 0.00964704 0.00977324 2024-09-13T14:47:14.847 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:47:14.847 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:47:14.855 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:14.868 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:47:14.868 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:47:14.891 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:14.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:14.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:14.906 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:47:14.906 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:47:14.934 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:47:14.934 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:47:14.967 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:47:14.968 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:47:14.993 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:15.007 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:47:15.007 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:47:15.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:15.040 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:15.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:15.041 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:47:15.042 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:47:15.079 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:47:15.079 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:47:15.142 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:15.179 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:15.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:15.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:15.281 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:15.306 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:15.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:15.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:15.383 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:47:15.408 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:15.442 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:15.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:15.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:15.543 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:15.566 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:15 smithi195 ceph-mon[104262]: pgmap v1133: 99 pgs: 99 active+clean; 6.0 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-13T14:47:15.566 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:15 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/4081768303' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:47:15.609 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:15.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:15.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:15.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:15 smithi102 ceph-mon[110183]: pgmap v1133: 99 pgs: 99 active+clean; 6.0 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-13T14:47:15.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:15 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/4081768303' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:47:15.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:15 smithi102 ceph-mon[115269]: pgmap v1133: 99 pgs: 99 active+clean; 6.0 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-13T14:47:15.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:15 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/4081768303' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:47:15.712 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:15.746 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:15.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:15.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:15.807 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 62 16 100814 100798 101.595 60.3125 0.0179082 0.00983854 2024-09-13T14:47:15.848 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:15.869 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:15.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:15.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:15.970 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:16.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:16.007 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:16.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:16.108 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:16.136 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:16.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:16.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:16.238 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:16.295 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:16.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:16.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:16.397 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:16.435 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:16.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:16.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:16.537 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:16.571 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:16.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:16.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:16.594 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:47:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:47:16] "GET /metrics HTTP/1.1" 200 36513 "" "Prometheus/2.51.0" 2024-09-13T14:47:16.673 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:16.715 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:16.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:16.720 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:16.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:16.808 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 63 16 101906 101890 101.064 68.25 0.017269 0.00989025 2024-09-13T14:47:16.821 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:16.852 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:16.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:16.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:16.954 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:16.994 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:16.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:16.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:17.095 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:17.144 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:17.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:17.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:17.246 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:17.278 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:17.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:17.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:17.380 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:17.387 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:17 smithi195 ceph-mon[104262]: pgmap v1134: 99 pgs: 99 active+clean; 6.0 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-13T14:47:17.412 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:17.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:17.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:17.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:17 smithi102 ceph-mon[110183]: pgmap v1134: 99 pgs: 99 active+clean; 6.0 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-13T14:47:17.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:17 smithi102 ceph-mon[115269]: pgmap v1134: 99 pgs: 99 active+clean; 6.0 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-13T14:47:17.514 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:17.549 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:17.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:17.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:17.628 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:47:17.629 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-13T14:47:17.629 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.629 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.46157.0:102760 8.5 e326/304 8:a443fd55:::benchmark_data_smithi102_313275_object102759:head v 326'6379, mlcod=326'6372)", 2024-09-13T14:47:17.629 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:47:17.608082+0000", 2024-09-13T14:47:17.629 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.018912406999999999, 2024-09-13T14:47:17.629 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018940057, 2024-09-13T14:47:17.629 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:47:17.629 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T14:47:17.629 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:47:17.629 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.630 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:47:17.630 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:17.608082+0000", 2024-09-13T14:47:17.630 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:47:17.630 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:17.630 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.630 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:47:17.630 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:17.608082+0000", 2024-09-13T14:47:17.630 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:47:17.630 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:17.630 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.630 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:47:17.630 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:17.608080+0000", 2024-09-13T14:47:17.630 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:47:17.631 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:17.631 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.631 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:47:17.631 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:17.608111+0000", 2024-09-13T14:47:17.631 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0636999999999998e-05 2024-09-13T14:47:17.631 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:17.631 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.631 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:47:17.631 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:17.608148+0000", 2024-09-13T14:47:17.631 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6634000000000002e-05 2024-09-13T14:47:17.631 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:17.631 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.631 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:47:17.631 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:17.608155+0000", 2024-09-13T14:47:17.632 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6320000000000008e-06 2024-09-13T14:47:17.632 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:17.632 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.632 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:47:17.632 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:17.622602+0000", 2024-09-13T14:47:17.632 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014446509999999999 2024-09-13T14:47:17.632 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:17.632 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.632 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:47:17.632 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:17.622650+0000", 2024-09-13T14:47:17.632 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8412999999999999e-05 2024-09-13T14:47:17.632 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:47:17.633 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:47:17.633 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:47:17.633 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:17.633 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.633 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.46157.0:102761 8.5 e326/304 8:ae1c42ad:::benchmark_data_smithi102_313275_object102760:head v 326'6380, mlcod=326'6373)", 2024-09-13T14:47:17.633 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:47:17.611885+0000", 2024-09-13T14:47:17.633 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.01510886, 2024-09-13T14:47:17.633 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.015202268, 2024-09-13T14:47:17.633 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:47:17.633 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T14:47:17.633 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:47:17.633 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.633 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:47:17.634 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:17.611885+0000", 2024-09-13T14:47:17.634 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:47:17.634 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:17.634 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.634 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:47:17.634 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:17.611885+0000", 2024-09-13T14:47:17.634 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:47:17.634 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:17.634 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.634 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:47:17.634 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:17.611884+0000", 2024-09-13T14:47:17.634 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:47:17.635 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:17.635 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.635 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:47:17.635 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:17.611956+0000", 2024-09-13T14:47:17.635 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1396999999999994e-05 2024-09-13T14:47:17.635 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:17.635 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.635 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:47:17.635 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:17.611973+0000", 2024-09-13T14:47:17.635 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7864e-05 2024-09-13T14:47:17.635 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:17.635 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.635 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:47:17.636 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:17.611981+0000", 2024-09-13T14:47:17.636 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.294e-06 2024-09-13T14:47:17.636 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:17.636 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.636 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:47:17.636 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:17.626024+0000", 2024-09-13T14:47:17.636 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014042999 2024-09-13T14:47:17.636 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:17.636 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.636 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:47:17.636 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:17.626049+0000", 2024-09-13T14:47:17.636 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5411000000000001e-05 2024-09-13T14:47:17.636 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:47:17.637 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:47:17.637 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:47:17.637 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:17.637 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.637 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.46157.0:102768 8.5 e326/304)", 2024-09-13T14:47:17.637 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:47:17.617326+0000", 2024-09-13T14:47:17.637 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.0096682069999999998, 2024-09-13T14:47:17.637 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0098177270000000001, 2024-09-13T14:47:17.637 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:47:17.637 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "queued for pg", 2024-09-13T14:47:17.637 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:47:17.637 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.638 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:47:17.638 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:17.617326+0000", 2024-09-13T14:47:17.638 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:47:17.638 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:17.638 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.638 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:47:17.638 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:17.617326+0000", 2024-09-13T14:47:17.638 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:47:17.638 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:17.638 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.638 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:47:17.638 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:17.617325+0000", 2024-09-13T14:47:17.638 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:47:17.638 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:17.639 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.639 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:47:17.639 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:17.617349+0000", 2024-09-13T14:47:17.639 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4102999999999999e-05 2024-09-13T14:47:17.639 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:17.639 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.639 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:47:17.639 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:17.620703+0000", 2024-09-13T14:47:17.639 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0033536759999999999 2024-09-13T14:47:17.639 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:17.639 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.639 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:47:17.639 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:17.620711+0000", 2024-09-13T14:47:17.640 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5320000000000006e-06 2024-09-13T14:47:17.640 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:47:17.640 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:47:17.640 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:47:17.640 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:17.640 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.640 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.46157.0:102750 8.5 e326/304 8:aae99232:::benchmark_data_smithi102_313275_object102749:head v 326'6376, mlcod=326'6371)", 2024-09-13T14:47:17.640 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:47:17.597210+0000", 2024-09-13T14:47:17.640 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.02978426, 2024-09-13T14:47:17.640 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.029979780000000001, 2024-09-13T14:47:17.640 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:47:17.640 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T14:47:17.640 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:47:17.641 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.641 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:47:17.641 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:17.597210+0000", 2024-09-13T14:47:17.641 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:47:17.641 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:17.641 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.641 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:47:17.641 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:17.597210+0000", 2024-09-13T14:47:17.641 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:47:17.641 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:17.641 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.641 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:47:17.641 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:17.597209+0000", 2024-09-13T14:47:17.642 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:47:17.642 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:17.642 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.642 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:47:17.642 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:17.597230+0000", 2024-09-13T14:47:17.642 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0455e-05 2024-09-13T14:47:17.642 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:17.642 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.642 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:47:17.642 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:17.597236+0000", 2024-09-13T14:47:17.642 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.9139999999999996e-06 2024-09-13T14:47:17.642 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:17.642 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.643 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:47:17.643 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:17.597240+0000", 2024-09-13T14:47:17.643 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2309999999999998e-06 2024-09-13T14:47:17.643 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:17.643 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.643 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:47:17.643 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:17.600994+0000", 2024-09-13T14:47:17.643 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0037544459999999998 2024-09-13T14:47:17.643 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:17.643 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.643 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:47:17.643 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:17.601038+0000", 2024-09-13T14:47:17.643 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3390999999999998e-05 2024-09-13T14:47:17.644 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:47:17.644 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:47:17.644 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:47:17.644 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:17.644 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.644 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.46157.0:102751 8.5 e326/304 8:aefc480a:::benchmark_data_smithi102_313275_object102750:head v 326'6377, mlcod=326'6371)", 2024-09-13T14:47:17.644 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:47:17.598936+0000", 2024-09-13T14:47:17.644 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.028058319000000002, 2024-09-13T14:47:17.644 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.028313998, 2024-09-13T14:47:17.645 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:47:17.645 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T14:47:17.645 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:47:17.645 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.645 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:47:17.645 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:17.598936+0000", 2024-09-13T14:47:17.645 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:47:17.645 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:17.645 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.645 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:47:17.645 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:17.598936+0000", 2024-09-13T14:47:17.645 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:47:17.645 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:17.645 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.646 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:47:17.646 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:17.598935+0000", 2024-09-13T14:47:17.646 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:47:17.646 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:17.646 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.646 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:47:17.646 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:17.598958+0000", 2024-09-13T14:47:17.646 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2739999999999999e-05 2024-09-13T14:47:17.646 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:17.646 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.646 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:47:17.646 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:17.598970+0000", 2024-09-13T14:47:17.646 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2101e-05 2024-09-13T14:47:17.647 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:17.647 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.647 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:47:17.647 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:17.598976+0000", 2024-09-13T14:47:17.647 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.767e-06 2024-09-13T14:47:17.647 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:17.647 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.647 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:47:17.647 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:17.608310+0000", 2024-09-13T14:47:17.647 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.009334485 2024-09-13T14:47:17.647 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:17.647 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.647 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:47:17.648 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:17.608339+0000", 2024-09-13T14:47:17.648 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9105e-05 2024-09-13T14:47:17.648 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:47:17.648 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:47:17.648 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:47:17.648 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:17.648 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.648 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.46157.0:102758 8.5 e326/304 8:a48cb12d:::benchmark_data_smithi102_313275_object102757:head v 326'6378, mlcod=326'6372)", 2024-09-13T14:47:17.648 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:47:17.603899+0000", 2024-09-13T14:47:17.648 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.023095059000000001, 2024-09-13T14:47:17.648 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.023406663000000001, 2024-09-13T14:47:17.648 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:47:17.648 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T14:47:17.649 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:47:17.649 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.649 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:47:17.649 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:17.603899+0000", 2024-09-13T14:47:17.649 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:47:17.649 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:17.649 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.649 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:47:17.649 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:17.603899+0000", 2024-09-13T14:47:17.649 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:47:17.649 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:17.649 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.649 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:47:17.649 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:17.603898+0000", 2024-09-13T14:47:17.650 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:47:17.650 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:17.650 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.650 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:47:17.650 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:17.603930+0000", 2024-09-13T14:47:17.650 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2515999999999998e-05 2024-09-13T14:47:17.650 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:17.650 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.650 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:47:17.650 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:17.606949+0000", 2024-09-13T14:47:17.650 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0030183779999999999 2024-09-13T14:47:17.650 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:17.650 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.650 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:47:17.651 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:17.606957+0000", 2024-09-13T14:47:17.651 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8830000000000006e-06 2024-09-13T14:47:17.651 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:17.651 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.651 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:47:17.651 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:17.619090+0000", 2024-09-13T14:47:17.651 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012133823 2024-09-13T14:47:17.651 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:47:17.651 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:47:17.651 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:47:17.651 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:17.619148+0000", 2024-09-13T14:47:17.651 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7132999999999997e-05 2024-09-13T14:47:17.651 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:47:17.651 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:47:17.651 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:47:17.652 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:47:17.652 INFO:teuthology.orchestra.run.smithi195.stdout: ], 2024-09-13T14:47:17.652 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 6 2024-09-13T14:47:17.652 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:47:17.652 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:17.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:17.689 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:17.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:17.790 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:17.810 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 64 16 102972 102956 100.525 66.625 0.0173884 0.00994246 2024-09-13T14:47:17.834 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:17.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:17.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:17.936 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:17.941 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:47:17.945 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:47:17.982 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:17.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:17.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:18.084 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:18.121 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:18.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:18.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:18.222 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:18.254 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:47:18.257 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:18.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:18.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:18.359 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:18.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:18.405 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:18.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:18.503 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T14:47:18.506 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:18.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:18.592 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:18.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:18.692 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:18.753 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:18.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:18.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:18.809 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 65 15 104021 104006 99.988 65.625 0.0159172 0.00999679 2024-09-13T14:47:18.822 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":1135,"stamp":"2024-09-13T14:47:17.588914+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6607280101,"num_objects":100774,"num_object_clones":0,"num_object_copies":302322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100774,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":101654,"num_write_kb":6460369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":4196477,"num_keys_recovered":20,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103418,"ondisk_log_size":103418,"up":297,"acting":297,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":301,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":24189128,"kb_used_data":20191824,"kb_used_omap":34,"kb_used_meta":3997149,"kb_avail":632026936,"statfs":{"total":671965249536,"available":647195582464,"internally_reserved":0,"allocated":20676427776,"data_stored":20664549367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35154,"internal_metadata":4093081262},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":717762138,"num_objects":10953,"num_object_clones":0,"num_object_copies":32859,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10953,"num_whiteouts":0,"num_read":-34,"num_read_kb":-75,"num_write":10933,"num_write_kb":700772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":-161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004674"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":554,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:09.286689+0000","last_change":"2024-09-13T14:46:01.770112+0000","last_active":"2024-09-13T14:47:09.286689+0000","last_peered":"2024-09-13T14:47:09.286689+0000","last_clean":"2024-09-13T14:47:09.286689+0000","last_became_active":"2024-09-13T14:46:01.769495+0000","last_became_peered":"2024-09-13T14:46:01.769495+0000","last_unstale":"2024-09-13T14:47:09.286689+0000","last_undegraded":"2024-09-13T14:47:09.286689+0000","last_fullsized":"2024-09-13T14:47:09.286689+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:51:27.686377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":345,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-13T14:46:59.223712+0000","last_change":"2024-09-13T14:46:01.772811+0000","last_active":"2024-09-13T14:46:59.223712+0000","last_peered":"2024-09-13T14:46:59.223712+0000","last_clean":"2024-09-13T14:46:59.223712+0000","last_became_active":"2024-09-13T14:46:01.764775+0000","last_became_peered":"2024-09-13T14:46:01.764775+0000","last_unstale":"2024-09-13T14:46:59.223712+0000","last_undegraded":"2024-09-13T14:46:59.223712+0000","last_fullsized":"2024-09-13T14:46:59.223712+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:42:20.738421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":126,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-13T14:46:59.229572+0000","last_change":"2024-09-13T14:36:13.862181+0000","last_active":"2024-09-13T14:46:59.229572+0000","last_peered":"2024-09-13T14:46:59.229572+0000","last_clean":"2024-09-13T14:46:59.229572+0000","last_became_active":"2024-09-13T14:36:13.861873+0000","last_became_peered":"2024-09-13T14:36:13.861873+0000","last_unstale":"2024-09-13T14:46:59.229572+0000","last_undegraded":"2024-09-13T14:46:59.229572+0000","last_fullsized":"2024-09-13T14:46:59.229572+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:14.085700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":184,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:09.297820+0000","last_change":"2024-09-13T14:38:53.896672+0000","last_active":"2024-09-13T14:47:09.297820+0000","last_peered":"2024-09-13T14:47:09.297820+0000","last_clean":"2024-09-13T14:47:09.297820+0000","last_became_active":"2024-09-13T14:38:53.896346+0000","last_became_peered":"2024-09-13T14:38:53.896346+0000","last_unstale":"2024-09-13T14:47:09.297820+0000","last_undegraded":"2024-09-13T14:47:09.297820+0000","last_fullsized":"2024-09-13T14:47:09.297820+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:57.789686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":184,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:09.298738+0000","last_change":"2024-09-13T14:38:53.893021+0000","last_active":"2024-09-13T14:47:09.298738+0000","last_peered":"2024-09-13T14:47:09.298738+0000","last_clean":"2024-09-13T14:47:09.298738+0000","last_became_active":"2024-09-13T14:38:53.892668+0000","last_became_peered":"2024-09-13T14:38:53.892668+0000","last_unstale":"2024-09-13T14:47:09.298738+0000","last_undegraded":"2024-09-13T14:47:09.298738+0000","last_fullsized":"2024-09-13T14:47:09.298738+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:57.180738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":298,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-13T14:46:59.223471+0000","last_change":"2024-09-13T14:37:48.134642+0000","last_active":"2024-09-13T14:46:59.223471+0000","last_peered":"2024-09-13T14:46:59.223471+0000","last_clean":"2024-09-13T14:46:59.223471+0000","last_became_active":"2024-09-13T14:37:48.134402+0000","last_became_peered":"2024-09-13T14:37:48.134402+0000","last_unstale":"2024-09-13T14:46:59.223471+0000","last_undegraded":"2024-09-13T14:46:59.223471+0000","last_fullsized":"2024-09-13T14:46:59.223471+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:51.313663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":573,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:09.286787+0000","last_change":"2024-09-13T14:46:01.770090+0000","last_active":"2024-09-13T14:47:09.286787+0000","last_peered":"2024-09-13T14:47:09.286787+0000","last_clean":"2024-09-13T14:47:09.286787+0000","last_became_active":"2024-09-13T14:46:01.769671+0000","last_became_peered":"2024-09-13T14:46:01.769671+0000","last_unstale":"2024-09-13T14:47:09.286787+0000","last_undegraded":"2024-09-13T14:47:09.286787+0000","last_fullsized":"2024-09-13T14:47:09.286787+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:46.340346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":187,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:09.297261+0000","last_change":"2024-09-13T14:38:03.554500+0000","last_active":"2024-09-13T14:47:09.297261+0000","last_peered":"2024-09-13T14:47:09.297261+0000","last_clean":"2024-09-13T14:47:09.297261+0000","last_became_active":"2024-09-13T14:38:03.532910+0000","last_became_peered":"2024-09-13T14:38:03.532910+0000","last_unstale":"2024-09-13T14:47:09.297261+0000","last_undegraded":"2024-09-13T14:47:09.297261+0000","last_fullsized":"2024-09-13T14:47:09.297261+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:27:50.843498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"326'6421","reported_seq":6461,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:17.506372+0000","last_change":"2024-09-13T14:46:01.762751+0000","last_active":"2024-09-13T14:47:17.506372+0000","last_peered":"2024-09-13T14:47:17.506372+0000","last_clean":"2024-09-13T14:47:17.506372+0000","last_became_active":"2024-09-13T14:46:01.761929+0000","last_became_peered":"2024-09-13T14:46:01.761929+0000","last_unstale":"2024-09-13T14:47:17.506372+0000","last_undegraded":"2024-09-13T14:47:17.506372+0000","last_fullsized":"2024-09-13T14:47:17.506372+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":304,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":6421,"log_dups_size":0,"ondisk_log_size":6421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:20:57.325962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420806656,"num_objects":6421,"num_object_clones":0,"num_object_copies":19263,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6421,"num_write_kb":410944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":90,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-13T14:46:59.227227+0000","last_change":"2024-09-13T14:38:53.940238+0000","last_active":"2024-09-13T14:46:59.227227+0000","last_peered":"2024-09-13T14:46:59.227227+0000","last_clean":"2024-09-13T14:46:59.227227+0000","last_became_active":"2024-09-13T14:38:53.892825+0000","last_became_peered":"2024-09-13T14:38:53.892825+0000","last_unstale":"2024-09-13T14:46:59.227227+0000","last_undegraded":"2024-09-13T14:46:59.227227+0000","last_fullsized":"2024-09-13T14:46:59.227227+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:46.666581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":147,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-13T14:47:07.508596+0000","last_change":"2024-09-13T14:38:53.898907+0000","last_active":"2024-09-13T14:47:07.508596+0000","last_peered":"2024-09-13T14:47:07.508596+0000","last_clean":"2024-09-13T14:47:07.508596+0000","last_became_active":"2024-09-13T14:38:53.898572+0000","last_became_peered":"2024-09-13T14:38:53.898572+0000","last_unstale":"2024-09-13T14:47:07.508596+0000","last_undegraded":"2024-09-13T14:47:07.508596+0000","last_fullsized":"2024-09-13T14:47:07.508596+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:29.222150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.5","version":"326'6109","reported_seq":6138,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:13.013147+0000","last_change":"2024-09-13T14:46:00.028852+0000","last_active":"2024-09-13T14:47:13.013147+0000","last_peered":"2024-09-13T14:47:13.013147+0000","last_clean":"2024-09-13T14:47:13.013147+0000","last_became_active":"2024-09-13T14:46:00.028414+0000","last_became_peered":"2024-09-13T14:46:00.028414+0000","last_unstale":"2024-09-13T14:47:13.013147+0000","last_undegraded":"2024-09-13T14:47:13.013147+0000","last_fullsized":"2024-09-13T14:47:13.013147+0000","mapping_epoch":304,"log_start":"0'0","ondisk_log_start":"0'0","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":6109,"log_dups_size":0,"ondisk_log_size":6109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:03:28.278631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400359424,"num_objects":6109,"num_object_clones":0,"num_object_copies":18327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6109,"num_write_kb":390976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":90,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-13T14:46:59.237529+0000","last_change":"2024-09-13T14:38:53.904885+0000","last_active":"2024-09-13T14:46:59.237529+0000","last_peered":"2024-09-13T14:46:59.237529+0000","last_clean":"2024-09-13T14:46:59.237529+0000","last_became_active":"2024-09-13T14:38:53.891726+0000","last_became_peered":"2024-09-13T14:38:53.891726+0000","last_unstale":"2024-09-13T14:46:59.237529+0000","last_undegraded":"2024-09-13T14:46:59.237529+0000","last_fullsized":"2024-09-13T14:46:59.237529+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:57.079205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":336,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-13T14:46:59.234374+0000","last_change":"2024-09-13T14:37:48.160504+0000","last_active":"2024-09-13T14:46:59.234374+0000","last_peered":"2024-09-13T14:46:59.234374+0000","last_clean":"2024-09-13T14:46:59.234374+0000","last_became_active":"2024-09-13T14:37:48.160181+0000","last_became_peered":"2024-09-13T14:37:48.160181+0000","last_unstale":"2024-09-13T14:46:59.234374+0000","last_undegraded":"2024-09-13T14:46:59.234374+0000","last_fullsized":"2024-09-13T14:46:59.234374+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:37.087008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"326'6376","reported_seq":6430,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:17.362515+0000","last_change":"2024-09-13T14:46:00.027675+0000","last_active":"2024-09-13T14:47:17.362515+0000","last_peered":"2024-09-13T14:47:17.362515+0000","last_clean":"2024-09-13T14:47:17.362515+0000","last_became_active":"2024-09-13T14:46:00.027364+0000","last_became_peered":"2024-09-13T14:46:00.027364+0000","last_unstale":"2024-09-13T14:47:17.362515+0000","last_undegraded":"2024-09-13T14:47:17.362515+0000","last_fullsized":"2024-09-13T14:47:17.362515+0000","mapping_epoch":304,"log_start":"0'0","ondisk_log_start":"0'0","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":6376,"log_dups_size":0,"ondisk_log_size":6376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:42:37.193007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":417857536,"num_objects":6376,"num_object_clones":0,"num_object_copies":19128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6376,"num_write_kb":408064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":327,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-13T14:46:59.231160+0000","last_change":"2024-09-13T14:37:49.611774+0000","last_active":"2024-09-13T14:46:59.231160+0000","last_peered":"2024-09-13T14:46:59.231160+0000","last_clean":"2024-09-13T14:46:59.231160+0000","last_became_active":"2024-09-13T14:37:49.611088+0000","last_became_peered":"2024-09-13T14:37:49.611088+0000","last_unstale":"2024-09-13T14:46:59.231160+0000","last_undegraded":"2024-09-13T14:46:59.231160+0000","last_fullsized":"2024-09-13T14:46:59.231160+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:08.933213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":86,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-13T14:46:59.223241+0000","last_change":"2024-09-13T14:38:53.899211+0000","last_active":"2024-09-13T14:46:59.223241+0000","last_peered":"2024-09-13T14:46:59.223241+0000","last_clean":"2024-09-13T14:46:59.223241+0000","last_became_active":"2024-09-13T14:38:53.898927+0000","last_became_peered":"2024-09-13T14:38:53.898927+0000","last_unstale":"2024-09-13T14:46:59.223241+0000","last_undegraded":"2024-09-13T14:46:59.223241+0000","last_fullsized":"2024-09-13T14:46:59.223241+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:00:28.857611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.7","version":"326'6147","reported_seq":6208,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:12.575067+0000","last_change":"2024-09-13T14:46:01.766640+0000","last_active":"2024-09-13T14:47:12.575067+0000","last_peered":"2024-09-13T14:47:12.575067+0000","last_clean":"2024-09-13T14:47:12.575067+0000","last_became_active":"2024-09-13T14:46:01.766349+0000","last_became_peered":"2024-09-13T14:46:01.766349+0000","last_unstale":"2024-09-13T14:47:12.575067+0000","last_undegraded":"2024-09-13T14:47:12.575067+0000","last_fullsized":"2024-09-13T14:47:12.575067+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":304,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":6147,"log_dups_size":0,"ondisk_log_size":6147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:16:28.600947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402849792,"num_objects":6147,"num_object_clones":0,"num_object_copies":18441,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6147,"num_write_kb":393408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":137,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-13T14:46:59.229149+0000","last_change":"2024-09-13T14:37:49.168003+0000","last_active":"2024-09-13T14:46:59.229149+0000","last_peered":"2024-09-13T14:46:59.229149+0000","last_clean":"2024-09-13T14:46:59.229149+0000","last_became_active":"2024-09-13T14:37:49.167638+0000","last_became_peered":"2024-09-13T14:37:49.167638+0000","last_unstale":"2024-09-13T14:46:59.229149+0000","last_undegraded":"2024-09-13T14:46:59.229149+0000","last_fullsized":"2024-09-13T14:46:59.229149+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:29.051981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":178,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:09.298313+0000","last_change":"2024-09-13T14:38:03.528381+0000","last_active":"2024-09-13T14:47:09.298313+0000","last_peered":"2024-09-13T14:47:09.298313+0000","last_clean":"2024-09-13T14:47:09.298313+0000","last_became_active":"2024-09-13T14:38:03.527282+0000","last_became_peered":"2024-09-13T14:38:03.527282+0000","last_unstale":"2024-09-13T14:47:09.298313+0000","last_undegraded":"2024-09-13T14:47:09.298313+0000","last_fullsized":"2024-09-13T14:47:09.298313+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:24.435067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"90'149","reported_seq":500,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-13T14:46:59.223356+0000","last_change":"2024-09-13T14:46:55.355650+0000","last_active":"2024-09-13T14:46:59.223356+0000","last_peered":"2024-09-13T14:46:59.223356+0000","last_clean":"2024-09-13T14:46:59.223356+0000","last_became_active":"2024-09-13T14:46:48.150338+0000","last_became_peered":"2024-09-13T14:46:48.150338+0000","last_unstale":"2024-09-13T14:46:59.223356+0000","last_undegraded":"2024-09-13T14:46:59.223356+0000","last_fullsized":"2024-09-13T14:46:59.223356+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:46:55.355521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:46:55.355521+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:30:53.006278+0000","scrub_duration":0.002025448,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.0","version":"326'6087","reported_seq":6142,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:12.587054+0000","last_change":"2024-09-13T14:45:59.744591+0000","last_active":"2024-09-13T14:47:12.587054+0000","last_peered":"2024-09-13T14:47:12.587054+0000","last_clean":"2024-09-13T14:47:12.587054+0000","last_became_active":"2024-09-13T14:45:59.744297+0000","last_became_peered":"2024-09-13T14:45:59.744297+0000","last_unstale":"2024-09-13T14:47:12.587054+0000","last_undegraded":"2024-09-13T14:47:12.587054+0000","last_fullsized":"2024-09-13T14:47:12.587054+0000","mapping_epoch":304,"log_start":"0'0","ondisk_log_start":"0'0","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":6087,"log_dups_size":0,"ondisk_log_size":6087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:05:27.678824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398917632,"num_objects":6087,"num_object_clones":0,"num_object_copies":18261,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6087,"num_write_kb":389568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":535,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:09.286440+0000","last_change":"2024-09-13T14:38:03.549595+0000","last_active":"2024-09-13T14:47:09.286440+0000","last_peered":"2024-09-13T14:47:09.286440+0000","last_clean":"2024-09-13T14:47:09.286440+0000","last_became_active":"2024-09-13T14:38:03.533331+0000","last_became_peered":"2024-09-13T14:38:03.533331+0000","last_unstale":"2024-09-13T14:47:09.286440+0000","last_undegraded":"2024-09-13T14:47:09.286440+0000","last_fullsized":"2024-09-13T14:47:09.286440+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:59:02.470343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":320,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-13T14:46:59.235271+0000","last_change":"2024-09-13T14:46:26.980367+0000","last_active":"2024-09-13T14:46:59.235271+0000","last_peered":"2024-09-13T14:46:59.235271+0000","last_clean":"2024-09-13T14:46:59.235271+0000","last_became_active":"2024-09-13T14:46:26.979937+0000","last_became_peered":"2024-09-13T14:46:26.979937+0000","last_unstale":"2024-09-13T14:46:59.235271+0000","last_undegraded":"2024-09-13T14:46:59.235271+0000","last_fullsized":"2024-09-13T14:46:59.235271+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:56.830872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"90'149","reported_seq":402,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-13T14:46:59.228118+0000","last_change":"2024-09-13T14:46:54.393663+0000","last_active":"2024-09-13T14:46:59.228118+0000","last_peered":"2024-09-13T14:46:59.228118+0000","last_clean":"2024-09-13T14:46:59.228118+0000","last_became_active":"2024-09-13T14:46:48.149004+0000","last_became_peered":"2024-09-13T14:46:48.149004+0000","last_unstale":"2024-09-13T14:46:59.228118+0000","last_undegraded":"2024-09-13T14:46:59.228118+0000","last_fullsized":"2024-09-13T14:46:59.228118+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:46:54.393204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:46:54.393204+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:32:40.003989+0000","scrub_duration":0.015172223,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1","version":"326'6352","reported_seq":6413,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:17.335988+0000","last_change":"2024-09-13T14:46:01.765500+0000","last_active":"2024-09-13T14:47:17.335988+0000","last_peered":"2024-09-13T14:47:17.335988+0000","last_clean":"2024-09-13T14:47:17.335988+0000","last_became_active":"2024-09-13T14:46:01.765187+0000","last_became_peered":"2024-09-13T14:46:01.765187+0000","last_unstale":"2024-09-13T14:47:17.335988+0000","last_undegraded":"2024-09-13T14:47:17.335988+0000","last_fullsized":"2024-09-13T14:47:17.335988+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":304,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":6352,"log_dups_size":0,"ondisk_log_size":6352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:15.232398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":416284672,"num_objects":6352,"num_object_clones":0,"num_object_copies":19056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6352,"num_write_kb":406528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":245,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-13T14:46:59.239339+0000","last_change":"2024-09-13T14:37:48.127604+0000","last_active":"2024-09-13T14:46:59.239339+0000","last_peered":"2024-09-13T14:46:59.239339+0000","last_clean":"2024-09-13T14:46:59.239339+0000","last_became_active":"2024-09-13T14:37:48.127300+0000","last_became_peered":"2024-09-13T14:37:48.127300+0000","last_unstale":"2024-09-13T14:46:59.239339+0000","last_undegraded":"2024-09-13T14:46:59.239339+0000","last_fullsized":"2024-09-13T14:46:59.239339+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:51.999506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":373,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-13T14:46:59.235290+0000","last_change":"2024-09-13T14:46:01.761924+0000","last_active":"2024-09-13T14:46:59.235290+0000","last_peered":"2024-09-13T14:46:59.235290+0000","last_clean":"2024-09-13T14:46:59.235290+0000","last_became_active":"2024-09-13T14:46:01.761576+0000","last_became_peered":"2024-09-13T14:46:01.761576+0000","last_unstale":"2024-09-13T14:46:59.235290+0000","last_undegraded":"2024-09-13T14:46:59.235290+0000","last_fullsized":"2024-09-13T14:46:59.235290+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:02:34.186759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.2","version":"326'6084","reported_seq":6115,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:12.991095+0000","last_change":"2024-09-13T14:45:59.744581+0000","last_active":"2024-09-13T14:47:12.991095+0000","last_peered":"2024-09-13T14:47:12.991095+0000","last_clean":"2024-09-13T14:47:12.991095+0000","last_became_active":"2024-09-13T14:45:59.744231+0000","last_became_peered":"2024-09-13T14:45:59.744231+0000","last_unstale":"2024-09-13T14:47:12.991095+0000","last_undegraded":"2024-09-13T14:47:12.991095+0000","last_fullsized":"2024-09-13T14:47:12.991095+0000","mapping_epoch":304,"log_start":"0'0","ondisk_log_start":"0'0","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":6084,"log_dups_size":0,"ondisk_log_size":6084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:06:14.502840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398721024,"num_objects":6084,"num_object_clones":0,"num_object_copies":18252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6084,"num_write_kb":389376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":334,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-13T14:46:59.234590+0000","last_change":"2024-09-13T14:46:01.762127+0000","last_active":"2024-09-13T14:46:59.234590+0000","last_peered":"2024-09-13T14:46:59.234590+0000","last_clean":"2024-09-13T14:46:59.234590+0000","last_became_active":"2024-09-13T14:46:01.761795+0000","last_became_peered":"2024-09-13T14:46:01.761795+0000","last_unstale":"2024-09-13T14:46:59.234590+0000","last_undegraded":"2024-09-13T14:46:59.234590+0000","last_fullsized":"2024-09-13T14:46:59.234590+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:41:15.626775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":320,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-13T14:46:59.240055+0000","last_change":"2024-09-13T14:37:48.147360+0000","last_active":"2024-09-13T14:46:59.240055+0000","last_peered":"2024-09-13T14:46:59.240055+0000","last_clean":"2024-09-13T14:46:59.240055+0000","last_became_active":"2024-09-13T14:37:48.146885+0000","last_became_peered":"2024-09-13T14:37:48.146885+0000","last_unstale":"2024-09-13T14:46:59.240055+0000","last_undegraded":"2024-09-13T14:46:59.240055+0000","last_fullsized":"2024-09-13T14:46:59.240055+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:27.841810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"90'149","reported_seq":402,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-13T14:46:59.228419+0000","last_change":"2024-09-13T14:46:52.334715+0000","last_active":"2024-09-13T14:46:59.228419+0000","last_peered":"2024-09-13T14:46:59.228419+0000","last_clean":"2024-09-13T14:46:59.228419+0000","last_became_active":"2024-09-13T14:46:48.158685+0000","last_became_peered":"2024-09-13T14:46:48.158685+0000","last_unstale":"2024-09-13T14:46:59.228419+0000","last_undegraded":"2024-09-13T14:46:59.228419+0000","last_fullsized":"2024-09-13T14:46:59.228419+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:46:52.334301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:46:52.334301+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:46:03.579759+0000","scrub_duration":0.0050940330000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.3","version":"326'6352","reported_seq":6392,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:17.485729+0000","last_change":"2024-09-13T14:46:01.764572+0000","last_active":"2024-09-13T14:47:17.485729+0000","last_peered":"2024-09-13T14:47:17.485729+0000","last_clean":"2024-09-13T14:47:17.485729+0000","last_became_active":"2024-09-13T14:46:01.764079+0000","last_became_peered":"2024-09-13T14:46:01.764079+0000","last_unstale":"2024-09-13T14:47:17.485729+0000","last_undegraded":"2024-09-13T14:47:17.485729+0000","last_fullsized":"2024-09-13T14:47:17.485729+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":304,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":6352,"log_dups_size":0,"ondisk_log_size":6352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:03:11.187551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":416284672,"num_objects":6352,"num_object_clones":0,"num_object_copies":19056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6352,"num_write_kb":406528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":554,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:09.286460+0000","last_change":"2024-09-13T14:38:53.900041+0000","last_active":"2024-09-13T14:47:09.286460+0000","last_peered":"2024-09-13T14:47:09.286460+0000","last_clean":"2024-09-13T14:47:09.286460+0000","last_became_active":"2024-09-13T14:38:53.899203+0000","last_became_peered":"2024-09-13T14:38:53.899203+0000","last_unstale":"2024-09-13T14:47:09.286460+0000","last_undegraded":"2024-09-13T14:47:09.286460+0000","last_fullsized":"2024-09-13T14:47:09.286460+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:28:07.883094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":486,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:09.286895+0000","last_change":"2024-09-13T14:26:15.823198+0000","last_active":"2024-09-13T14:47:09.286895+0000","last_peered":"2024-09-13T14:47:09.286895+0000","last_clean":"2024-09-13T14:47:09.286895+0000","last_became_active":"2024-09-13T14:26:15.822717+0000","last_became_peered":"2024-09-13T14:26:15.822717+0000","last_unstale":"2024-09-13T14:47:09.286895+0000","last_undegraded":"2024-09-13T14:47:09.286895+0000","last_fullsized":"2024-09-13T14:47:09.286895+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:22.471622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"326'6541","reported_seq":6572,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:17.506815+0000","last_change":"2024-09-13T14:46:00.027751+0000","last_active":"2024-09-13T14:47:17.506815+0000","last_peered":"2024-09-13T14:47:17.506815+0000","last_clean":"2024-09-13T14:47:17.506815+0000","last_became_active":"2024-09-13T14:46:00.027429+0000","last_became_peered":"2024-09-13T14:46:00.027429+0000","last_unstale":"2024-09-13T14:47:17.506815+0000","last_undegraded":"2024-09-13T14:47:17.506815+0000","last_fullsized":"2024-09-13T14:47:17.506815+0000","mapping_epoch":304,"log_start":"0'0","ondisk_log_start":"0'0","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":6541,"log_dups_size":0,"ondisk_log_size":6541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:17:52.381240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":428670976,"num_objects":6541,"num_object_clones":0,"num_object_copies":19623,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6541,"num_write_kb":418624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":483,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-13T14:46:59.223323+0000","last_change":"2024-09-13T14:46:50.349014+0000","last_active":"2024-09-13T14:46:59.223323+0000","last_peered":"2024-09-13T14:46:59.223323+0000","last_clean":"2024-09-13T14:46:59.223323+0000","last_became_active":"2024-09-13T14:46:41.093256+0000","last_became_peered":"2024-09-13T14:46:41.093256+0000","last_unstale":"2024-09-13T14:46:59.223323+0000","last_undegraded":"2024-09-13T14:46:59.223323+0000","last_fullsized":"2024-09-13T14:46:59.223323+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:46:50.348873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:46:50.348873+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:47.678951+0000","scrub_duration":0.0039561329999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":633,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-13T14:46:59.230989+0000","last_change":"2024-09-13T14:46:04.780718+0000","last_active":"2024-09-13T14:46:59.230989+0000","last_peered":"2024-09-13T14:46:59.230989+0000","last_clean":"2024-09-13T14:46:59.230989+0000","last_became_active":"2024-09-13T14:46:04.780120+0000","last_became_peered":"2024-09-13T14:46:04.780120+0000","last_unstale":"2024-09-13T14:46:59.230989+0000","last_undegraded":"2024-09-13T14:46:59.230989+0000","last_fullsized":"2024-09-13T14:46:59.230989+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:57:14.475888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":488,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:09.286300+0000","last_change":"2024-09-13T14:37:49.160719+0000","last_active":"2024-09-13T14:47:09.286300+0000","last_peered":"2024-09-13T14:47:09.286300+0000","last_clean":"2024-09-13T14:47:09.286300+0000","last_became_active":"2024-09-13T14:37:49.160401+0000","last_became_peered":"2024-09-13T14:37:49.160401+0000","last_unstale":"2024-09-13T14:47:09.286300+0000","last_undegraded":"2024-09-13T14:47:09.286300+0000","last_fullsized":"2024-09-13T14:47:09.286300+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:21.737945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":172,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-13T14:46:59.235193+0000","last_change":"2024-09-13T14:37:48.172940+0000","last_active":"2024-09-13T14:46:59.235193+0000","last_peered":"2024-09-13T14:46:59.235193+0000","last_clean":"2024-09-13T14:46:59.235193+0000","last_became_active":"2024-09-13T14:37:48.172060+0000","last_became_peered":"2024-09-13T14:37:48.172060+0000","last_unstale":"2024-09-13T14:46:59.235193+0000","last_undegraded":"2024-09-13T14:46:59.235193+0000","last_fullsized":"2024-09-13T14:46:59.235193+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:07.480645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"90'149","reported_seq":402,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-13T14:46:59.234687+0000","last_change":"2024-09-13T14:46:50.622210+0000","last_active":"2024-09-13T14:46:59.234687+0000","last_peered":"2024-09-13T14:46:59.234687+0000","last_clean":"2024-09-13T14:46:59.234687+0000","last_became_active":"2024-09-13T14:46:48.169343+0000","last_became_peered":"2024-09-13T14:46:48.169343+0000","last_unstale":"2024-09-13T14:46:59.234687+0000","last_undegraded":"2024-09-13T14:46:59.234687+0000","last_fullsized":"2024-09-13T14:46:59.234687+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:46:50.621349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:46:50.621349+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:14:30.938920+0000","scrub_duration":0.0082913290000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.a","version":"326'6457","reported_seq":6508,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:16.384931+0000","last_change":"2024-09-13T14:46:01.767421+0000","last_active":"2024-09-13T14:47:16.384931+0000","last_peered":"2024-09-13T14:47:16.384931+0000","last_clean":"2024-09-13T14:47:16.384931+0000","last_became_active":"2024-09-13T14:46:01.767105+0000","last_became_peered":"2024-09-13T14:46:01.767105+0000","last_unstale":"2024-09-13T14:47:16.384931+0000","last_undegraded":"2024-09-13T14:47:16.384931+0000","last_fullsized":"2024-09-13T14:47:16.384931+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":304,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":6457,"log_dups_size":0,"ondisk_log_size":6457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:57:36.278404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":423165952,"num_objects":6457,"num_object_clones":0,"num_object_copies":19371,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6457,"num_write_kb":413248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":545,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-13T14:46:59.223224+0000","last_change":"2024-09-13T14:26:15.835178+0000","last_active":"2024-09-13T14:46:59.223224+0000","last_peered":"2024-09-13T14:46:59.223224+0000","last_clean":"2024-09-13T14:46:59.223224+0000","last_became_active":"2024-09-13T14:26:15.820283+0000","last_became_peered":"2024-09-13T14:26:15.820283+0000","last_unstale":"2024-09-13T14:46:59.223224+0000","last_undegraded":"2024-09-13T14:46:59.223224+0000","last_fullsized":"2024-09-13T14:46:59.223224+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:30.340038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":178,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:09.299323+0000","last_change":"2024-09-13T14:38:03.525682+0000","last_active":"2024-09-13T14:47:09.299323+0000","last_peered":"2024-09-13T14:47:09.299323+0000","last_clean":"2024-09-13T14:47:09.299323+0000","last_became_active":"2024-09-13T14:38:03.525406+0000","last_became_peered":"2024-09-13T14:38:03.525406+0000","last_unstale":"2024-09-13T14:47:09.299323+0000","last_undegraded":"2024-09-13T14:47:09.299323+0000","last_fullsized":"2024-09-13T14:47:09.299323+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:41:28.970300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":192,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:09.298811+0000","last_change":"2024-09-13T14:46:01.769069+0000","last_active":"2024-09-13T14:47:09.298811+0000","last_peered":"2024-09-13T14:47:09.298811+0000","last_clean":"2024-09-13T14:47:09.298811+0000","last_became_active":"2024-09-13T14:46:01.768766+0000","last_became_peered":"2024-09-13T14:46:01.768766+0000","last_unstale":"2024-09-13T14:47:09.298811+0000","last_undegraded":"2024-09-13T14:47:09.298811+0000","last_fullsized":"2024-09-13T14:47:09.298811+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:22:47.154785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"90'149","reported_seq":500,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-13T14:46:59.223112+0000","last_change":"2024-09-13T14:46:51.347800+0000","last_active":"2024-09-13T14:46:59.223112+0000","last_peered":"2024-09-13T14:46:59.223112+0000","last_clean":"2024-09-13T14:46:59.223112+0000","last_became_active":"2024-09-13T14:46:48.153059+0000","last_became_peered":"2024-09-13T14:46:48.153059+0000","last_unstale":"2024-09-13T14:46:59.223112+0000","last_undegraded":"2024-09-13T14:46:59.223112+0000","last_fullsized":"2024-09-13T14:46:59.223112+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:46:51.347652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:46:51.347652+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:22:09.525070+0000","scrub_duration":0.013466275,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"326'6128","reported_seq":6189,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:12.572472+0000","last_change":"2024-09-13T14:46:01.766352+0000","last_active":"2024-09-13T14:47:12.572472+0000","last_peered":"2024-09-13T14:47:12.572472+0000","last_clean":"2024-09-13T14:47:12.572472+0000","last_became_active":"2024-09-13T14:46:01.766046+0000","last_became_peered":"2024-09-13T14:46:01.766046+0000","last_unstale":"2024-09-13T14:47:12.572472+0000","last_undegraded":"2024-09-13T14:47:12.572472+0000","last_fullsized":"2024-09-13T14:47:12.572472+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":304,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":6128,"log_dups_size":0,"ondisk_log_size":6128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:36:54.550417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":401604608,"num_objects":6128,"num_object_clones":0,"num_object_copies":18384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6128,"num_write_kb":392192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":472,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:09.298287+0000","last_change":"2024-09-13T14:46:04.782955+0000","last_active":"2024-09-13T14:47:09.298287+0000","last_peered":"2024-09-13T14:47:09.298287+0000","last_clean":"2024-09-13T14:47:09.298287+0000","last_became_active":"2024-09-13T14:46:04.782250+0000","last_became_peered":"2024-09-13T14:46:04.782250+0000","last_unstale":"2024-09-13T14:47:09.298287+0000","last_undegraded":"2024-09-13T14:47:09.298287+0000","last_fullsized":"2024-09-13T14:47:09.298287+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:24.596922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":50,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:09.294412+0000","last_change":"2024-09-13T14:46:01.767882+0000","last_active":"2024-09-13T14:47:09.294412+0000","last_peered":"2024-09-13T14:47:09.294412+0000","last_clean":"2024-09-13T14:47:09.294412+0000","last_became_active":"2024-09-13T14:46:01.767116+0000","last_became_peered":"2024-09-13T14:46:01.767116+0000","last_unstale":"2024-09-13T14:47:09.294412+0000","last_undegraded":"2024-09-13T14:47:09.294412+0000","last_fullsized":"2024-09-13T14:47:09.294412+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:24:48.755394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":154,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-13T14:47:07.509341+0000","last_change":"2024-09-13T14:38:53.901427+0000","last_active":"2024-09-13T14:47:07.509341+0000","last_peered":"2024-09-13T14:47:07.509341+0000","last_clean":"2024-09-13T14:47:07.509341+0000","last_became_active":"2024-09-13T14:38:53.901157+0000","last_became_peered":"2024-09-13T14:38:53.901157+0000","last_unstale":"2024-09-13T14:47:07.509341+0000","last_undegraded":"2024-09-13T14:47:07.509341+0000","last_fullsized":"2024-09-13T14:47:07.509341+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:28:21.713025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"90'149","reported_seq":402,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-13T14:46:59.228221+0000","last_change":"2024-09-13T14:46:56.456623+0000","last_active":"2024-09-13T14:46:59.228221+0000","last_peered":"2024-09-13T14:46:59.228221+0000","last_clean":"2024-09-13T14:46:59.228221+0000","last_became_active":"2024-09-13T14:46:48.162544+0000","last_became_peered":"2024-09-13T14:46:48.162544+0000","last_unstale":"2024-09-13T14:46:59.228221+0000","last_undegraded":"2024-09-13T14:46:59.228221+0000","last_fullsized":"2024-09-13T14:46:59.228221+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:46:56.456458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:46:56.456458+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:24:41.205500+0000","scrub_duration":0.015876965,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.8","version":"326'6454","reported_seq":6485,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:17.514115+0000","last_change":"2024-09-13T14:45:59.743468+0000","last_active":"2024-09-13T14:47:17.514115+0000","last_peered":"2024-09-13T14:47:17.514115+0000","last_clean":"2024-09-13T14:47:17.514115+0000","last_became_active":"2024-09-13T14:45:59.742168+0000","last_became_peered":"2024-09-13T14:45:59.742168+0000","last_unstale":"2024-09-13T14:47:17.514115+0000","last_undegraded":"2024-09-13T14:47:17.514115+0000","last_fullsized":"2024-09-13T14:47:17.514115+0000","mapping_epoch":304,"log_start":"0'0","ondisk_log_start":"0'0","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":6454,"log_dups_size":0,"ondisk_log_size":6454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:38:09.406586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":422969344,"num_objects":6454,"num_object_clones":0,"num_object_copies":19362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6454,"num_write_kb":413056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":1036,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-13T14:46:59.231545+0000","last_change":"2024-09-13T14:37:49.147498+0000","last_active":"2024-09-13T14:46:59.231545+0000","last_peered":"2024-09-13T14:46:59.231545+0000","last_clean":"2024-09-13T14:46:59.231545+0000","last_became_active":"2024-09-13T14:37:49.147028+0000","last_became_peered":"2024-09-13T14:37:49.147028+0000","last_unstale":"2024-09-13T14:46:59.231545+0000","last_undegraded":"2024-09-13T14:46:59.231545+0000","last_fullsized":"2024-09-13T14:46:59.231545+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:11:29.050178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":334,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-13T14:46:59.237123+0000","last_change":"2024-09-13T14:38:03.528133+0000","last_active":"2024-09-13T14:46:59.237123+0000","last_peered":"2024-09-13T14:46:59.237123+0000","last_clean":"2024-09-13T14:46:59.237123+0000","last_became_active":"2024-09-13T14:38:03.527829+0000","last_became_peered":"2024-09-13T14:38:03.527829+0000","last_unstale":"2024-09-13T14:46:59.237123+0000","last_undegraded":"2024-09-13T14:46:59.237123+0000","last_fullsized":"2024-09-13T14:46:59.237123+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:48.254169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":50,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:09.298769+0000","last_change":"2024-09-13T14:46:01.768418+0000","last_active":"2024-09-13T14:47:09.298769+0000","last_peered":"2024-09-13T14:47:09.298769+0000","last_clean":"2024-09-13T14:47:09.298769+0000","last_became_active":"2024-09-13T14:46:01.767804+0000","last_became_peered":"2024-09-13T14:46:01.767804+0000","last_unstale":"2024-09-13T14:47:09.298769+0000","last_undegraded":"2024-09-13T14:47:09.298769+0000","last_fullsized":"2024-09-13T14:47:09.298769+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:05:10.365127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"90'149","reported_seq":405,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:09.286757+0000","last_change":"2024-09-13T14:46:54.148660+0000","last_active":"2024-09-13T14:47:09.286757+0000","last_peered":"2024-09-13T14:47:09.286757+0000","last_clean":"2024-09-13T14:47:09.286757+0000","last_became_active":"2024-09-13T14:46:48.155406+0000","last_became_peered":"2024-09-13T14:46:48.155406+0000","last_unstale":"2024-09-13T14:47:09.286757+0000","last_undegraded":"2024-09-13T14:47:09.286757+0000","last_fullsized":"2024-09-13T14:47:09.286757+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:46:54.148591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:46:54.148591+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:06:39.515283+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"326'6393","reported_seq":6454,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:17.336972+0000","last_change":"2024-09-13T14:46:01.768936+0000","last_active":"2024-09-13T14:47:17.336972+0000","last_peered":"2024-09-13T14:47:17.336972+0000","last_clean":"2024-09-13T14:47:17.336972+0000","last_became_active":"2024-09-13T14:46:01.768354+0000","last_became_peered":"2024-09-13T14:46:01.768354+0000","last_unstale":"2024-09-13T14:47:17.336972+0000","last_undegraded":"2024-09-13T14:47:17.336972+0000","last_fullsized":"2024-09-13T14:47:17.336972+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":304,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":6393,"log_dups_size":0,"ondisk_log_size":6393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:01.762702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":418971648,"num_objects":6393,"num_object_clones":0,"num_object_copies":19179,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6393,"num_write_kb":409152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":147,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-13T14:46:59.229434+0000","last_change":"2024-09-13T14:46:01.763906+0000","last_active":"2024-09-13T14:46:59.229434+0000","last_peered":"2024-09-13T14:46:59.229434+0000","last_clean":"2024-09-13T14:46:59.229434+0000","last_became_active":"2024-09-13T14:46:01.763542+0000","last_became_peered":"2024-09-13T14:46:01.763542+0000","last_unstale":"2024-09-13T14:46:59.229434+0000","last_undegraded":"2024-09-13T14:46:59.229434+0000","last_fullsized":"2024-09-13T14:46:59.229434+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:09:36.652767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":1273,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:09.297891+0000","last_change":"2024-09-13T14:46:04.780843+0000","last_active":"2024-09-13T14:47:09.297891+0000","last_peered":"2024-09-13T14:47:09.297891+0000","last_clean":"2024-09-13T14:47:09.297891+0000","last_became_active":"2024-09-13T14:46:04.780204+0000","last_became_peered":"2024-09-13T14:46:04.780204+0000","last_unstale":"2024-09-13T14:47:09.297891+0000","last_undegraded":"2024-09-13T14:47:09.297891+0000","last_fullsized":"2024-09-13T14:47:09.297891+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:20:41.074826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":145,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-13T14:46:59.231715+0000","last_change":"2024-09-13T14:38:03.534628+0000","last_active":"2024-09-13T14:46:59.231715+0000","last_peered":"2024-09-13T14:46:59.231715+0000","last_clean":"2024-09-13T14:46:59.231715+0000","last_became_active":"2024-09-13T14:38:03.534262+0000","last_became_peered":"2024-09-13T14:38:03.534262+0000","last_unstale":"2024-09-13T14:46:59.231715+0000","last_undegraded":"2024-09-13T14:46:59.231715+0000","last_fullsized":"2024-09-13T14:46:59.231715+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:31.168783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"90'149","reported_seq":411,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:09.298462+0000","last_change":"2024-09-13T14:46:53.895118+0000","last_active":"2024-09-13T14:47:09.298462+0000","last_peered":"2024-09-13T14:47:09.298462+0000","last_clean":"2024-09-13T14:47:09.298462+0000","last_became_active":"2024-09-13T14:46:48.155440+0000","last_became_peered":"2024-09-13T14:46:48.155440+0000","last_unstale":"2024-09-13T14:47:09.298462+0000","last_undegraded":"2024-09-13T14:47:09.298462+0000","last_fullsized":"2024-09-13T14:47:09.298462+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:46:53.895053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:46:53.895053+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:12:56.291840+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"326'6059","reported_seq":6120,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:12.565979+0000","last_change":"2024-09-13T14:46:01.769601+0000","last_active":"2024-09-13T14:47:12.565979+0000","last_peered":"2024-09-13T14:47:12.565979+0000","last_clean":"2024-09-13T14:47:12.565979+0000","last_became_active":"2024-09-13T14:46:01.769276+0000","last_became_peered":"2024-09-13T14:46:01.769276+0000","last_unstale":"2024-09-13T14:47:12.565979+0000","last_undegraded":"2024-09-13T14:47:12.565979+0000","last_fullsized":"2024-09-13T14:47:12.565979+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":304,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":6059,"log_dups_size":0,"ondisk_log_size":6059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:05:46.860485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":397082624,"num_objects":6059,"num_object_clones":0,"num_object_copies":18177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6059,"num_write_kb":387776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":50,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:09.298674+0000","last_change":"2024-09-13T14:46:01.769791+0000","last_active":"2024-09-13T14:47:09.298674+0000","last_peered":"2024-09-13T14:47:09.298674+0000","last_clean":"2024-09-13T14:47:09.298674+0000","last_became_active":"2024-09-13T14:46:01.768801+0000","last_became_peered":"2024-09-13T14:46:01.768801+0000","last_unstale":"2024-09-13T14:47:09.298674+0000","last_undegraded":"2024-09-13T14:47:09.298674+0000","last_fullsized":"2024-09-13T14:47:09.298674+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:26:55.790375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":849,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-13T14:46:59.231286+0000","last_change":"2024-09-13T14:38:04.596192+0000","last_active":"2024-09-13T14:46:59.231286+0000","last_peered":"2024-09-13T14:46:59.231286+0000","last_clean":"2024-09-13T14:46:59.231286+0000","last_became_active":"2024-09-13T14:38:04.593299+0000","last_became_peered":"2024-09-13T14:38:04.593299+0000","last_unstale":"2024-09-13T14:46:59.231286+0000","last_undegraded":"2024-09-13T14:46:59.231286+0000","last_fullsized":"2024-09-13T14:46:59.231286+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:34.241410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":178,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:09.298391+0000","last_change":"2024-09-13T14:38:03.533120+0000","last_active":"2024-09-13T14:47:09.298391+0000","last_peered":"2024-09-13T14:47:09.298391+0000","last_clean":"2024-09-13T14:47:09.298391+0000","last_became_active":"2024-09-13T14:38:03.532829+0000","last_became_peered":"2024-09-13T14:38:03.532829+0000","last_unstale":"2024-09-13T14:47:09.298391+0000","last_undegraded":"2024-09-13T14:47:09.298391+0000","last_fullsized":"2024-09-13T14:47:09.298391+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:24.514661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"90'149","reported_seq":402,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-13T14:47:05.452651+0000","last_change":"2024-09-13T14:47:05.452651+0000","last_active":"2024-09-13T14:47:05.452651+0000","last_peered":"2024-09-13T14:47:05.452651+0000","last_clean":"2024-09-13T14:47:05.452651+0000","last_became_active":"2024-09-13T14:46:48.157832+0000","last_became_peered":"2024-09-13T14:46:48.157832+0000","last_unstale":"2024-09-13T14:47:05.452651+0000","last_undegraded":"2024-09-13T14:47:05.452651+0000","last_fullsized":"2024-09-13T14:47:05.452651+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:05.452402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:05.452402+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:05:48.836648+0000","scrub_duration":0.0060199540000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.c","version":"326'6318","reported_seq":6373,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:17.330849+0000","last_change":"2024-09-13T14:45:59.743204+0000","last_active":"2024-09-13T14:47:17.330849+0000","last_peered":"2024-09-13T14:47:17.330849+0000","last_clean":"2024-09-13T14:47:17.330849+0000","last_became_active":"2024-09-13T14:45:59.742910+0000","last_became_peered":"2024-09-13T14:45:59.742910+0000","last_unstale":"2024-09-13T14:47:17.330849+0000","last_undegraded":"2024-09-13T14:47:17.330849+0000","last_fullsized":"2024-09-13T14:47:17.330849+0000","mapping_epoch":304,"log_start":"0'0","ondisk_log_start":"0'0","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":6318,"log_dups_size":0,"ondisk_log_size":6318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:51.017907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":414056448,"num_objects":6318,"num_object_clones":0,"num_object_copies":18954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6318,"num_write_kb":404352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":580,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:09.286942+0000","last_change":"2024-09-13T14:38:53.938227+0000","last_active":"2024-09-13T14:47:09.286942+0000","last_peered":"2024-09-13T14:47:09.286942+0000","last_clean":"2024-09-13T14:47:09.286942+0000","last_became_active":"2024-09-13T14:38:53.902867+0000","last_became_peered":"2024-09-13T14:38:53.902867+0000","last_unstale":"2024-09-13T14:47:09.286942+0000","last_undegraded":"2024-09-13T14:47:09.286942+0000","last_fullsized":"2024-09-13T14:47:09.286942+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:16.831756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":628,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-13T14:46:59.240191+0000","last_change":"2024-09-13T14:37:48.147677+0000","last_active":"2024-09-13T14:46:59.240191+0000","last_peered":"2024-09-13T14:46:59.240191+0000","last_clean":"2024-09-13T14:46:59.240191+0000","last_became_active":"2024-09-13T14:37:48.147001+0000","last_became_peered":"2024-09-13T14:37:48.147001+0000","last_unstale":"2024-09-13T14:46:59.240191+0000","last_undegraded":"2024-09-13T14:46:59.240191+0000","last_fullsized":"2024-09-13T14:46:59.240191+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:35:00.653576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":562,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:09.286240+0000","last_change":"2024-09-13T14:38:53.900107+0000","last_active":"2024-09-13T14:47:09.286240+0000","last_peered":"2024-09-13T14:47:09.286240+0000","last_clean":"2024-09-13T14:47:09.286240+0000","last_became_active":"2024-09-13T14:38:53.899556+0000","last_became_peered":"2024-09-13T14:38:53.899556+0000","last_unstale":"2024-09-13T14:47:09.286240+0000","last_undegraded":"2024-09-13T14:47:09.286240+0000","last_fullsized":"2024-09-13T14:47:09.286240+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:15:47.918163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"90'149","reported_seq":402,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:09.286306+0000","last_change":"2024-09-13T14:46:49.208148+0000","last_active":"2024-09-13T14:47:09.286306+0000","last_peered":"2024-09-13T14:47:09.286306+0000","last_clean":"2024-09-13T14:47:09.286306+0000","last_became_active":"2024-09-13T14:46:48.178209+0000","last_became_peered":"2024-09-13T14:46:48.178209+0000","last_unstale":"2024-09-13T14:47:09.286306+0000","last_undegraded":"2024-09-13T14:47:09.286306+0000","last_fullsized":"2024-09-13T14:47:09.286306+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:46:49.207941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:46:49.207941+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:32:00.690228+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"326'6468","reported_seq":6508,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:17.496346+0000","last_change":"2024-09-13T14:46:01.764360+0000","last_active":"2024-09-13T14:47:17.496346+0000","last_peered":"2024-09-13T14:47:17.496346+0000","last_clean":"2024-09-13T14:47:17.496346+0000","last_became_active":"2024-09-13T14:46:01.764029+0000","last_became_peered":"2024-09-13T14:46:01.764029+0000","last_unstale":"2024-09-13T14:47:17.496346+0000","last_undegraded":"2024-09-13T14:47:17.496346+0000","last_fullsized":"2024-09-13T14:47:17.496346+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":304,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":6468,"log_dups_size":0,"ondisk_log_size":6468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:02:23.092494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":423886848,"num_objects":6468,"num_object_clones":0,"num_object_copies":19404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6468,"num_write_kb":413952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":193,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:09.298712+0000","last_change":"2024-09-13T14:38:53.901806+0000","last_active":"2024-09-13T14:47:09.298712+0000","last_peered":"2024-09-13T14:47:09.298712+0000","last_clean":"2024-09-13T14:47:09.298712+0000","last_became_active":"2024-09-13T14:38:53.891055+0000","last_became_peered":"2024-09-13T14:38:53.891055+0000","last_unstale":"2024-09-13T14:47:09.298712+0000","last_undegraded":"2024-09-13T14:47:09.298712+0000","last_fullsized":"2024-09-13T14:47:09.298712+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:03:00.416198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":127,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-13T14:46:59.227358+0000","last_change":"2024-09-13T14:46:01.781508+0000","last_active":"2024-09-13T14:46:59.227358+0000","last_peered":"2024-09-13T14:46:59.227358+0000","last_clean":"2024-09-13T14:46:59.227358+0000","last_became_active":"2024-09-13T14:46:01.767096+0000","last_became_peered":"2024-09-13T14:46:01.767096+0000","last_unstale":"2024-09-13T14:46:59.227358+0000","last_undegraded":"2024-09-13T14:46:59.227358+0000","last_fullsized":"2024-09-13T14:46:59.227358+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:20:01.119189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":318,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-13T14:46:59.234484+0000","last_change":"2024-09-13T14:37:49.172218+0000","last_active":"2024-09-13T14:46:59.234484+0000","last_peered":"2024-09-13T14:46:59.234484+0000","last_clean":"2024-09-13T14:46:59.234484+0000","last_became_active":"2024-09-13T14:37:49.171718+0000","last_became_peered":"2024-09-13T14:37:49.171718+0000","last_unstale":"2024-09-13T14:46:59.234484+0000","last_undegraded":"2024-09-13T14:46:59.234484+0000","last_fullsized":"2024-09-13T14:46:59.234484+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:46:54.626521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":340,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-13T14:46:59.235794+0000","last_change":"2024-09-13T14:37:49.230094+0000","last_active":"2024-09-13T14:46:59.235794+0000","last_peered":"2024-09-13T14:46:59.235794+0000","last_clean":"2024-09-13T14:46:59.235794+0000","last_became_active":"2024-09-13T14:37:49.229393+0000","last_became_peered":"2024-09-13T14:37:49.229393+0000","last_unstale":"2024-09-13T14:46:59.235794+0000","last_undegraded":"2024-09-13T14:46:59.235794+0000","last_fullsized":"2024-09-13T14:46:59.235794+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:02:28.690318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":320,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-13T14:46:59.235202+0000","last_change":"2024-09-13T14:37:48.128447+0000","last_active":"2024-09-13T14:46:59.235202+0000","last_peered":"2024-09-13T14:46:59.235202+0000","last_clean":"2024-09-13T14:46:59.235202+0000","last_became_active":"2024-09-13T14:37:48.128143+0000","last_became_peered":"2024-09-13T14:37:48.128143+0000","last_unstale":"2024-09-13T14:46:59.235202+0000","last_undegraded":"2024-09-13T14:46:59.235202+0000","last_fullsized":"2024-09-13T14:46:59.235202+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:22.204039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":50,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:09.298973+0000","last_change":"2024-09-13T14:46:01.768529+0000","last_active":"2024-09-13T14:47:09.298973+0000","last_peered":"2024-09-13T14:47:09.298973+0000","last_clean":"2024-09-13T14:47:09.298973+0000","last_became_active":"2024-09-13T14:46:01.768221+0000","last_became_peered":"2024-09-13T14:46:01.768221+0000","last_unstale":"2024-09-13T14:47:09.298973+0000","last_undegraded":"2024-09-13T14:47:09.298973+0000","last_fullsized":"2024-09-13T14:47:09.298973+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:16:00.647959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":92,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-13T14:46:59.230907+0000","last_change":"2024-09-13T14:38:53.954281+0000","last_active":"2024-09-13T14:46:59.230907+0000","last_peered":"2024-09-13T14:46:59.230907+0000","last_clean":"2024-09-13T14:46:59.230907+0000","last_became_active":"2024-09-13T14:38:53.900912+0000","last_became_peered":"2024-09-13T14:38:53.900912+0000","last_unstale":"2024-09-13T14:46:59.230907+0000","last_undegraded":"2024-09-13T14:46:59.230907+0000","last_fullsized":"2024-09-13T14:46:59.230907+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:47.876275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":144,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-13T14:46:59.228147+0000","last_change":"2024-09-13T14:38:54.000782+0000","last_active":"2024-09-13T14:46:59.228147+0000","last_peered":"2024-09-13T14:46:59.228147+0000","last_clean":"2024-09-13T14:46:59.228147+0000","last_became_active":"2024-09-13T14:38:54.000322+0000","last_became_peered":"2024-09-13T14:38:54.000322+0000","last_unstale":"2024-09-13T14:46:59.228147+0000","last_undegraded":"2024-09-13T14:46:59.228147+0000","last_fullsized":"2024-09-13T14:46:59.228147+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:09.057597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":503,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:09.286146+0000","last_change":"2024-09-13T14:38:53.899535+0000","last_active":"2024-09-13T14:47:09.286146+0000","last_peered":"2024-09-13T14:47:09.286146+0000","last_clean":"2024-09-13T14:47:09.286146+0000","last_became_active":"2024-09-13T14:38:53.899183+0000","last_became_peered":"2024-09-13T14:38:53.899183+0000","last_unstale":"2024-09-13T14:47:09.286146+0000","last_undegraded":"2024-09-13T14:47:09.286146+0000","last_fullsized":"2024-09-13T14:47:09.286146+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:33.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":177,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:09.298185+0000","last_change":"2024-09-13T14:38:03.543426+0000","last_active":"2024-09-13T14:47:09.298185+0000","last_peered":"2024-09-13T14:47:09.298185+0000","last_clean":"2024-09-13T14:47:09.298185+0000","last_became_active":"2024-09-13T14:38:03.542661+0000","last_became_peered":"2024-09-13T14:38:03.542661+0000","last_unstale":"2024-09-13T14:47:09.298185+0000","last_undegraded":"2024-09-13T14:47:09.298185+0000","last_fullsized":"2024-09-13T14:47:09.298185+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:54.050127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":532,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:09.286225+0000","last_change":"2024-09-13T14:37:48.131321+0000","last_active":"2024-09-13T14:47:09.286225+0000","last_peered":"2024-09-13T14:47:09.286225+0000","last_clean":"2024-09-13T14:47:09.286225+0000","last_became_active":"2024-09-13T14:37:48.130998+0000","last_became_peered":"2024-09-13T14:37:48.130998+0000","last_unstale":"2024-09-13T14:47:09.286225+0000","last_undegraded":"2024-09-13T14:47:09.286225+0000","last_fullsized":"2024-09-13T14:47:09.286225+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:33:26.597825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":157,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-13T14:46:59.228356+0000","last_change":"2024-09-13T14:38:03.535454+0000","last_active":"2024-09-13T14:46:59.228356+0000","last_peered":"2024-09-13T14:46:59.228356+0000","last_clean":"2024-09-13T14:46:59.228356+0000","last_became_active":"2024-09-13T14:38:03.535131+0000","last_became_peered":"2024-09-13T14:38:03.535131+0000","last_unstale":"2024-09-13T14:46:59.228356+0000","last_undegraded":"2024-09-13T14:46:59.228356+0000","last_fullsized":"2024-09-13T14:46:59.228356+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:24.477601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":564,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:09.286345+0000","last_change":"2024-09-13T14:46:01.766605+0000","last_active":"2024-09-13T14:47:09.286345+0000","last_peered":"2024-09-13T14:47:09.286345+0000","last_clean":"2024-09-13T14:47:09.286345+0000","last_became_active":"2024-09-13T14:46:01.766284+0000","last_became_peered":"2024-09-13T14:46:01.766284+0000","last_unstale":"2024-09-13T14:47:09.286345+0000","last_undegraded":"2024-09-13T14:47:09.286345+0000","last_fullsized":"2024-09-13T14:47:09.286345+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:53:48.686769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":185,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:09.298656+0000","last_change":"2024-09-13T14:38:53.892948+0000","last_active":"2024-09-13T14:47:09.298656+0000","last_peered":"2024-09-13T14:47:09.298656+0000","last_clean":"2024-09-13T14:47:09.298656+0000","last_became_active":"2024-09-13T14:38:53.892665+0000","last_became_peered":"2024-09-13T14:38:53.892665+0000","last_unstale":"2024-09-13T14:47:09.298656+0000","last_undegraded":"2024-09-13T14:47:09.298656+0000","last_fullsized":"2024-09-13T14:47:09.298656+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:08.292359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":505,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:09.286261+0000","last_change":"2024-09-13T14:37:48.139348+0000","last_active":"2024-09-13T14:47:09.286261+0000","last_peered":"2024-09-13T14:47:09.286261+0000","last_clean":"2024-09-13T14:47:09.286261+0000","last_became_active":"2024-09-13T14:37:48.139115+0000","last_became_peered":"2024-09-13T14:37:48.139115+0000","last_unstale":"2024-09-13T14:47:09.286261+0000","last_undegraded":"2024-09-13T14:47:09.286261+0000","last_fullsized":"2024-09-13T14:47:09.286261+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:32:20.413346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":545,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:09.286470+0000","last_change":"2024-09-13T14:37:48.172088+0000","last_active":"2024-09-13T14:47:09.286470+0000","last_peered":"2024-09-13T14:47:09.286470+0000","last_clean":"2024-09-13T14:47:09.286470+0000","last_became_active":"2024-09-13T14:37:48.171779+0000","last_became_peered":"2024-09-13T14:37:48.171779+0000","last_unstale":"2024-09-13T14:47:09.286470+0000","last_undegraded":"2024-09-13T14:47:09.286470+0000","last_fullsized":"2024-09-13T14:47:09.286470+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:02.852712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":393,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-13T14:46:59.239424+0000","last_change":"2024-09-13T14:46:01.762871+0000","last_active":"2024-09-13T14:46:59.239424+0000","last_peered":"2024-09-13T14:46:59.239424+0000","last_clean":"2024-09-13T14:46:59.239424+0000","last_became_active":"2024-09-13T14:46:01.762531+0000","last_became_peered":"2024-09-13T14:46:01.762531+0000","last_unstale":"2024-09-13T14:46:59.239424+0000","last_undegraded":"2024-09-13T14:46:59.239424+0000","last_fullsized":"2024-09-13T14:46:59.239424+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:09:16.982787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":57,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:09.298862+0000","last_change":"2024-09-13T14:46:26.977621+0000","last_active":"2024-09-13T14:47:09.298862+0000","last_peered":"2024-09-13T14:47:09.298862+0000","last_clean":"2024-09-13T14:47:09.298862+0000","last_became_active":"2024-09-13T14:46:26.975127+0000","last_became_peered":"2024-09-13T14:46:26.975127+0000","last_unstale":"2024-09-13T14:47:09.298862+0000","last_undegraded":"2024-09-13T14:47:09.298862+0000","last_fullsized":"2024-09-13T14:47:09.298862+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:46:25.398406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":177,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:09.298223+0000","last_change":"2024-09-13T14:38:03.527021+0000","last_active":"2024-09-13T14:47:09.298223+0000","last_peered":"2024-09-13T14:47:09.298223+0000","last_clean":"2024-09-13T14:47:09.298223+0000","last_became_active":"2024-09-13T14:38:03.526698+0000","last_became_peered":"2024-09-13T14:38:03.526698+0000","last_unstale":"2024-09-13T14:47:09.298223+0000","last_undegraded":"2024-09-13T14:47:09.298223+0000","last_fullsized":"2024-09-13T14:47:09.298223+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:01:35.734086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":337,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-13T14:46:59.234986+0000","last_change":"2024-09-13T14:46:01.763375+0000","last_active":"2024-09-13T14:46:59.234986+0000","last_peered":"2024-09-13T14:46:59.234986+0000","last_clean":"2024-09-13T14:46:59.234986+0000","last_became_active":"2024-09-13T14:46:01.763045+0000","last_became_peered":"2024-09-13T14:46:01.763045+0000","last_unstale":"2024-09-13T14:46:59.234986+0000","last_undegraded":"2024-09-13T14:46:59.234986+0000","last_fullsized":"2024-09-13T14:46:59.234986+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:03:35.294598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":141,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-13T14:46:59.230447+0000","last_change":"2024-09-13T14:37:49.176796+0000","last_active":"2024-09-13T14:46:59.230447+0000","last_peered":"2024-09-13T14:46:59.230447+0000","last_clean":"2024-09-13T14:46:59.230447+0000","last_became_active":"2024-09-13T14:37:49.176246+0000","last_became_peered":"2024-09-13T14:37:49.176246+0000","last_unstale":"2024-09-13T14:46:59.230447+0000","last_undegraded":"2024-09-13T14:46:59.230447+0000","last_fullsized":"2024-09-13T14:46:59.230447+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:29:24.936718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":339,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-13T14:46:59.239266+0000","last_change":"2024-09-13T14:38:03.526333+0000","last_active":"2024-09-13T14:46:59.239266+0000","last_peered":"2024-09-13T14:46:59.239266+0000","last_clean":"2024-09-13T14:46:59.239266+0000","last_became_active":"2024-09-13T14:38:03.526001+0000","last_became_peered":"2024-09-13T14:38:03.526001+0000","last_unstale":"2024-09-13T14:46:59.239266+0000","last_undegraded":"2024-09-13T14:46:59.239266+0000","last_fullsized":"2024-09-13T14:46:59.239266+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:25:25.181271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":141,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-13T14:46:59.229505+0000","last_change":"2024-09-13T14:37:49.149582+0000","last_active":"2024-09-13T14:46:59.229505+0000","last_peered":"2024-09-13T14:46:59.229505+0000","last_clean":"2024-09-13T14:46:59.229505+0000","last_became_active":"2024-09-13T14:37:49.148525+0000","last_became_peered":"2024-09-13T14:37:49.148525+0000","last_unstale":"2024-09-13T14:46:59.229505+0000","last_undegraded":"2024-09-13T14:46:59.229505+0000","last_fullsized":"2024-09-13T14:46:59.229505+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:48.655665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":585,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:09.286359+0000","last_change":"2024-09-13T14:46:01.766580+0000","last_active":"2024-09-13T14:47:09.286359+0000","last_peered":"2024-09-13T14:47:09.286359+0000","last_clean":"2024-09-13T14:47:09.286359+0000","last_became_active":"2024-09-13T14:46:01.766257+0000","last_became_peered":"2024-09-13T14:46:01.766257+0000","last_unstale":"2024-09-13T14:47:09.286359+0000","last_undegraded":"2024-09-13T14:47:09.286359+0000","last_fullsized":"2024-09-13T14:47:09.286359+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:01.126109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":549,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:09.286773+0000","last_change":"2024-09-13T14:46:01.769647+0000","last_active":"2024-09-13T14:47:09.286773+0000","last_peered":"2024-09-13T14:47:09.286773+0000","last_clean":"2024-09-13T14:47:09.286773+0000","last_became_active":"2024-09-13T14:46:01.769228+0000","last_became_peered":"2024-09-13T14:46:01.769228+0000","last_unstale":"2024-09-13T14:47:09.286773+0000","last_undegraded":"2024-09-13T14:47:09.286773+0000","last_fullsized":"2024-09-13T14:47:09.286773+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:58.696024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":319,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-13T14:46:59.236263+0000","last_change":"2024-09-13T14:37:48.127607+0000","last_active":"2024-09-13T14:46:59.236263+0000","last_peered":"2024-09-13T14:46:59.236263+0000","last_clean":"2024-09-13T14:46:59.236263+0000","last_became_active":"2024-09-13T14:37:48.127316+0000","last_became_peered":"2024-09-13T14:37:48.127316+0000","last_unstale":"2024-09-13T14:46:59.236263+0000","last_undegraded":"2024-09-13T14:46:59.236263+0000","last_fullsized":"2024-09-13T14:46:59.236263+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:53:37.678861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":197,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-13T14:47:09.298097+0000","last_change":"2024-09-13T14:46:01.775237+0000","last_active":"2024-09-13T14:47:09.298097+0000","last_peered":"2024-09-13T14:47:09.298097+0000","last_clean":"2024-09-13T14:47:09.298097+0000","last_became_active":"2024-09-13T14:46:01.765025+0000","last_became_peered":"2024-09-13T14:46:01.765025+0000","last_unstale":"2024-09-13T14:47:09.298097+0000","last_undegraded":"2024-09-13T14:47:09.298097+0000","last_fullsized":"2024-09-13T14:47:09.298097+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:29:16.089042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1746,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":6602489856,"num_objects":100746,"num_object_clones":0,"num_object_copies":302238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100746,"num_write_kb":6447744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19831062528,"data_stored":19831062528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100746,"ondisk_log_size":100746,"up":48,"acting":48,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":2154,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25132,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":4194323,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16793600,"data_stored":16777292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1639,"ondisk_log_size":1639,"up":33,"acting":33,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234441,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2615224,"kb_used_data":2034480,"kb_used_omap":1,"kb_used_meta":580734,"kb_avail":91129928,"statfs":{"total":95995035648,"available":93317046272,"internally_reserved":0,"allocated":2083307520,"data_stored":2081603429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":594672075},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":239,"seq":1026497183859,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2429488,"kb_used_data":2034088,"kb_used_omap":8,"kb_used_meta":395383,"kb_avail":91315664,"statfs":{"total":95995035648,"available":93507239936,"internally_reserved":0,"allocated":2082906112,"data_stored":2081210144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8453,"internal_metadata":404872955},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757706,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4460460,"kb_used_data":3703332,"kb_used_omap":1,"kb_used_meta":757118,"kb_avail":89284692,"statfs":{"total":95995035648,"available":91427524608,"internally_reserved":0,"allocated":3792211968,"data_stored":3790516408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":775289286},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019349,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3529972,"kb_used_data":2854892,"kb_used_omap":1,"kb_used_meta":675070,"kb_avail":90215180,"statfs":{"total":95995035648,"available":92380344320,"internally_reserved":0,"allocated":2923409408,"data_stored":2921709401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":691272134},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435532,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3388648,"kb_used_data":2721120,"kb_used_omap":5,"kb_used_meta":667514,"kb_avail":90356504,"statfs":{"total":95995035648,"available":92525060096,"internally_reserved":0,"allocated":2786426880,"data_stored":2784735178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":683534462},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":104,"seq":446676599061,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3808684,"kb_used_data":3636772,"kb_used_omap":7,"kb_used_meta":171896,"kb_avail":89936468,"statfs":{"total":95995035648,"available":92094943232,"internally_reserved":0,"allocated":3724054528,"data_stored":3722359256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8010,"internal_metadata":176021686},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":425201762587,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926111,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3956652,"kb_used_data":3207140,"kb_used_omap":7,"kb_used_meta":749432,"kb_avail":89788500,"statfs":{"total":95995035648,"available":91943424000,"internally_reserved":0,"allocated":3284111360,"data_stored":3282415551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":767418664},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6860,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2512453632,"data_stored":2512453632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3745054720,"data_stored":3745054720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2801205248,"data_stored":2801205248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2804154368,"data_stored":2804154368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3794796544,"data_stored":3794796544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2076377088,"data_stored":2076377088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2097020928,"data_stored":2097020928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:47:18.827 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-13T14:47:18.827 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pg_num 21 2024-09-13T14:47:18.856 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:18.862 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:47:18.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:18.921 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:18.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:19.022 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:19.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:19.064 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:19.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:19.165 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:19.209 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:19.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:19.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:19.310 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:19.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:19.345 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:19.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:19.446 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:19.474 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:19.477 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:19.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:19.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:19.578 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:19.604 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:19 smithi102 ceph-mon[115269]: pgmap v1135: 99 pgs: 99 active+clean; 6.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-13T14:47:19.604 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:19 smithi102 ceph-mon[115269]: from='client.46220 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:47:19.606 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:19 smithi102 ceph-mon[110183]: pgmap v1135: 99 pgs: 99 active+clean; 6.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-13T14:47:19.606 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:19 smithi102 ceph-mon[110183]: from='client.46220 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:47:19.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:19.625 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:19.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:19.726 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:19.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:19 smithi195 ceph-mon[104262]: pgmap v1135: 99 pgs: 99 active+clean; 6.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-13T14:47:19.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:19 smithi195 ceph-mon[104262]: from='client.46220 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:47:19.772 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:19.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:19.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:19.811 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 66 15 105007 104992 99.4051 61.625 0.0160634 0.0100548 2024-09-13T14:47:19.873 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:19.910 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:19.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:19.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:20.011 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:20.053 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:20.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:20.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:20.155 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:20.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:20.187 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:20.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:20.287 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:20.323 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:20.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:20.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:20.425 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:20.464 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:20.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:20.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:20.566 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:20.599 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:20.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:20.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:20.701 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:20.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:20.738 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:20.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:20.812 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 67 16 106131 106115 98.969 70.1875 0.00990393 0.0100993 2024-09-13T14:47:20.839 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:20.871 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:20.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:20.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:20.881 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:20.973 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:21.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:21.011 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:21.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:21.112 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:21.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:21.150 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:21.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:21.251 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:21.291 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:21.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:21.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:21.323 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is set 2024-09-13T14:47:21.392 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:21.416 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:21 smithi195 ceph-mon[104262]: pgmap v1136: 99 pgs: 99 active+clean; 6.3 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-13T14:47:21.416 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:21 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3652027239' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:47:21.416 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:21 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:47:21.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:21.426 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:21.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:21.527 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:21.566 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:21.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:21.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:21.601 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:47:21.601 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-13T14:47:21.602 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-09-13T14:47:21.602 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-09-13T14:47:21.602 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:47:21.613 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:21 smithi102 ceph-mon[110183]: pgmap v1136: 99 pgs: 99 active+clean; 6.3 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-13T14:47:21.613 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:21 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3652027239' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:47:21.613 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:21 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:47:21.613 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:21 smithi102 ceph-mon[115269]: pgmap v1136: 99 pgs: 99 active+clean; 6.3 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-13T14:47:21.613 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:21 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3652027239' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:47:21.613 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:21 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:47:21.668 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:21.700 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:21.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:21.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:21.801 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:21.811 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 68 16 107155 107139 98.4543 64 0.0126994 0.0101516 2024-09-13T14:47:21.839 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:21.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:21.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:21.941 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:21.979 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:21.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:21.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:22.081 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:22.116 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:22.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:22.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:22.218 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:22.255 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:22.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:22.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:22.357 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:22.387 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T14:47:22.401 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:22.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:22.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:22.503 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:22.540 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:22.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:22.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:22.610 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:22 smithi102 ceph-mon[110183]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:47:22.611 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:22 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:47:22.611 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:22 smithi102 ceph-mon[110183]: osdmap e327: 8 total, 8 up, 7 in 2024-09-13T14:47:22.611 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:22 smithi102 ceph-mon[115269]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:47:22.611 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:22 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:47:22.611 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:22 smithi102 ceph-mon[115269]: osdmap e327: 8 total, 8 up, 7 in 2024-09-13T14:47:22.643 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:22.670 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:22.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:22.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:22.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:22 smithi195 ceph-mon[104262]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:47:22.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:22 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:47:22.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:22 smithi195 ceph-mon[104262]: osdmap e327: 8 total, 8 up, 7 in 2024-09-13T14:47:22.772 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:22.781 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:47:22.811 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 69 16 108226 108210 97.9975 66.9375 0.0171058 0.0101987 2024-09-13T14:47:22.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:22.841 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:22.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:22.942 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:22.979 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:22.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:22.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:23.070 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:47:23.070 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:47:23.081 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:23.102 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:47:23.102 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:47:23.116 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:23.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:23.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:23.131 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:47:23.132 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:47:23.160 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:47:23.160 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:47:23.199 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:47:23.200 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:47:23.218 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:23.235 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:47:23.236 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:47:23.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:23.253 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:23.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:23.261 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:47:23.262 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:47:23.291 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:47:23.291 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:47:23.354 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:23.388 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:23.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:23.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:23.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:23 smithi102 ceph-mon[110183]: pgmap v1138: 99 pgs: 99 active+clean; 6.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-09-13T14:47:23.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:23 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1221914661' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:47:23.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:23 smithi102 ceph-mon[115269]: pgmap v1138: 99 pgs: 99 active+clean; 6.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-09-13T14:47:23.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:23 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1221914661' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:47:23.490 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:23.528 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:23.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:23.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:23.630 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:23.666 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:23.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:23.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:23.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:23 smithi195 ceph-mon[104262]: pgmap v1138: 99 pgs: 99 active+clean; 6.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-09-13T14:47:23.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:23 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1221914661' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:47:23.768 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:23.810 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:23.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:23.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:23.811 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 70 16 109300 109284 97.5565 67.125 0.0175924 0.0102451 2024-09-13T14:47:23.912 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:23.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:23.943 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:23.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:24.044 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:24.074 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:24.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:24.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:24.176 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:24.209 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:24.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:24.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:24.311 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:24.347 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:24.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:24.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:24.449 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:24.484 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:24.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:24.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:24.586 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:24.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:24.627 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:24.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:24.648 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:47:24.728 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:24.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:24.773 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:24.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:24.811 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 71 16 110331 110315 97.09 64.4375 0.0074578 0.010295 2024-09-13T14:47:24.874 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:24.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:24.935 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:24.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:25.036 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:25.077 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:25.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:25.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:25.179 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:25.204 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:25.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:25.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:25.262 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:25.306 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:25.336 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:25.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:25.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:25.438 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:25.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:25 smithi102 ceph-mon[110183]: pgmap v1139: 99 pgs: 99 active+clean; 6.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-09-13T14:47:25.442 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:25 smithi102 ceph-mon[115269]: pgmap v1139: 99 pgs: 99 active+clean; 6.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-09-13T14:47:25.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:25.493 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:25.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:25.594 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:25.634 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:25.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:25.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:25.708 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:25 smithi195 ceph-mon[104262]: pgmap v1139: 99 pgs: 99 active+clean; 6.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-09-13T14:47:25.736 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:25.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:25.781 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:25.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:25.811 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 72 16 111265 111249 96.5522 58.375 0.00934995 0.0103521 2024-09-13T14:47:25.882 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:25.910 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:25.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:25.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:26.012 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:26.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:26.048 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:26.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:26.149 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:26.200 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:26.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:26.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:26.247 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:47:26.247 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:47:26.302 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:26.345 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:26.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:26.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:26.377 INFO:teuthology.orchestra.run.smithi102.stderr:set pool 1 pg_num to 21 2024-09-13T14:47:26.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:47:26.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:26 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2367296366' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-13T14:47:26.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:26 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-13T14:47:26.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:47:26.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:26 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2367296366' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-13T14:47:26.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:26 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-13T14:47:26.447 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:26.480 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:26.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:26.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:26.582 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:26.605 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:47:26.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:26.631 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:26.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:26.732 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:26.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:47:26.750 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:26 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2367296366' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-13T14:47:26.750 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:26 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-13T14:47:26.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:26.779 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:26.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:26.805 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:47:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:47:26] "GET /metrics HTTP/1.1" 200 36511 "" "Prometheus/2.51.0" 2024-09-13T14:47:26.811 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 73 16 112250 112234 96.0726 61.5625 0.022096 0.0104042 2024-09-13T14:47:26.880 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:26.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:26.919 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:26.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:27.014 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T14:47:27.019 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:27.115 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:27.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:27.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:27.216 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:27.249 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:27.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:27.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:27.351 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:27.391 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:27.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:27.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:27.493 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:27.527 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:27.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:27.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:27.629 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:27.633 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:27 smithi102 ceph-mon[110183]: pgmap v1140: 99 pgs: 99 active+clean; 6.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-09-13T14:47:27.633 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:27 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-13T14:47:27.633 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:27 smithi102 ceph-mon[110183]: osdmap e328: 8 total, 8 up, 7 in 2024-09-13T14:47:27.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:27 smithi102 ceph-mon[115269]: pgmap v1140: 99 pgs: 99 active+clean; 6.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-09-13T14:47:27.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:27 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-13T14:47:27.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:27 smithi102 ceph-mon[115269]: osdmap e328: 8 total, 8 up, 7 in 2024-09-13T14:47:27.657 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:27.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:27.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:27.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:27 smithi195 ceph-mon[104262]: pgmap v1140: 99 pgs: 99 active+clean; 6.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-09-13T14:47:27.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:27 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-13T14:47:27.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:27 smithi195 ceph-mon[104262]: osdmap e328: 8 total, 8 up, 7 in 2024-09-13T14:47:27.759 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:27.794 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:27.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:27.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:27.814 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 74 16 113284 113268 95.6447 64.625 0.0120218 0.0104506 2024-09-13T14:47:27.896 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:27.939 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:27.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:27.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:28.040 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:28.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:28.088 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:28.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:28.155 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:28.159 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:28.189 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:28.219 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:28.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:28.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:28.321 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:28.368 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:28.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:28.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:28.470 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:28.501 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:28.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:28.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:28.603 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:28.624 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:28 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-13T14:47:28.626 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:28 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-13T14:47:28.639 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:28.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:28.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:28.741 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:28.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:28 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-13T14:47:28.774 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:28.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:28.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:28.814 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 75 16 114326 114310 95.2378 65.125 0.00909792 0.010495 2024-09-13T14:47:28.876 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:28.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:28.912 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:28.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:29.013 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:29.046 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:29.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:29.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:29.148 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:29.178 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:29.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:29.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:29.279 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:29.316 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:29.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:29.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:29.396 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is unset 2024-09-13T14:47:29.418 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:29.452 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:29.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:29.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:29.554 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:29.590 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:29.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:29.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:29.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:29 smithi102 ceph-mon[115269]: pgmap v1142: 99 pgs: 99 active+clean; 6.8 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-13T14:47:29.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:29 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-13T14:47:29.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:29 smithi102 ceph-mon[115269]: osdmap e329: 8 total, 8 up, 7 in 2024-09-13T14:47:29.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:29 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/619260043' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:47:29.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:29 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:47:29.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:29 smithi102 ceph-mon[110183]: pgmap v1142: 99 pgs: 99 active+clean; 6.8 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-13T14:47:29.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:29 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-13T14:47:29.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:29 smithi102 ceph-mon[110183]: osdmap e329: 8 total, 8 up, 7 in 2024-09-13T14:47:29.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:29 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/619260043' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:47:29.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:29 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:47:29.692 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:29.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:29.730 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:29.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:29.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:29 smithi195 ceph-mon[104262]: pgmap v1142: 99 pgs: 99 active+clean; 6.8 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-13T14:47:29.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:29 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-13T14:47:29.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:29 smithi195 ceph-mon[104262]: osdmap e329: 8 total, 8 up, 7 in 2024-09-13T14:47:29.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:29 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/619260043' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:47:29.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:29 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:47:29.817 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 76 16 115468 115452 94.9237 71.375 0.0272994 0.0105296 2024-09-13T14:47:29.832 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:29.867 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:29.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:29.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:29.969 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:30.002 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:30.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:30.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:30.104 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:30.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:30.138 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:30.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:30.239 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:30.275 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:30.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:30.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:30.377 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:30.412 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:30.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:30.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:30.514 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:30.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:30.551 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:30.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:30.652 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:30.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:30 smithi102 ceph-mon[110183]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:47:30.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:30 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:47:30.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:30 smithi102 ceph-mon[110183]: osdmap e330: 8 total, 8 up, 7 in 2024-09-13T14:47:30.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:30 smithi102 ceph-mon[110183]: pgmap v1145: 109 pgs: 10 unknown, 99 active+clean; 6.9 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 45 MiB/s wr, 723 op/s 2024-09-13T14:47:30.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:30 smithi102 ceph-mon[110183]: 1.2 scrub starts 2024-09-13T14:47:30.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:30 smithi102 ceph-mon[110183]: 1.2 scrub ok 2024-09-13T14:47:30.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:30 smithi102 ceph-mon[115269]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:47:30.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:30 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:47:30.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:30 smithi102 ceph-mon[115269]: osdmap e330: 8 total, 8 up, 7 in 2024-09-13T14:47:30.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:30 smithi102 ceph-mon[115269]: pgmap v1145: 109 pgs: 10 unknown, 99 active+clean; 6.9 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 45 MiB/s wr, 723 op/s 2024-09-13T14:47:30.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:30 smithi102 ceph-mon[115269]: 1.2 scrub starts 2024-09-13T14:47:30.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:30 smithi102 ceph-mon[115269]: 1.2 scrub ok 2024-09-13T14:47:30.686 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:30.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:30.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:30.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:30 smithi195 ceph-mon[104262]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:47:30.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:30 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:47:30.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:30 smithi195 ceph-mon[104262]: osdmap e330: 8 total, 8 up, 7 in 2024-09-13T14:47:30.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:30 smithi195 ceph-mon[104262]: pgmap v1145: 109 pgs: 10 unknown, 99 active+clean; 6.9 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 45 MiB/s wr, 723 op/s 2024-09-13T14:47:30.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:30 smithi195 ceph-mon[104262]: 1.2 scrub starts 2024-09-13T14:47:30.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:30 smithi195 ceph-mon[104262]: 1.2 scrub ok 2024-09-13T14:47:30.787 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:30.814 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 77 16 116532 116516 94.5544 66.5 0.0109821 0.0105711 2024-09-13T14:47:30.822 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:30.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:30.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:30.924 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:30.959 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:30.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:30.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:31.061 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:31.096 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:31.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:31.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:31.198 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:31.242 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:31.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:31.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:31.344 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:31.375 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:31.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:31.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:31.477 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:31.505 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:31.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:31.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:31.607 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:31.625 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:31 smithi102 ceph-mon[110183]: 1.6 scrub starts 2024-09-13T14:47:31.626 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:31 smithi102 ceph-mon[110183]: 1.6 scrub ok 2024-09-13T14:47:31.626 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:31 smithi102 ceph-mon[110183]: 1.4 scrub starts 2024-09-13T14:47:31.626 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:31 smithi102 ceph-mon[110183]: 1.4 scrub ok 2024-09-13T14:47:31.626 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:31 smithi102 ceph-mon[110183]: 1.5 scrub starts 2024-09-13T14:47:31.626 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:31 smithi102 ceph-mon[110183]: 1.5 scrub ok 2024-09-13T14:47:31.626 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:31 smithi102 ceph-mon[115269]: 1.6 scrub starts 2024-09-13T14:47:31.626 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:31 smithi102 ceph-mon[115269]: 1.6 scrub ok 2024-09-13T14:47:31.626 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:31 smithi102 ceph-mon[115269]: 1.4 scrub starts 2024-09-13T14:47:31.626 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:31 smithi102 ceph-mon[115269]: 1.4 scrub ok 2024-09-13T14:47:31.626 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:31 smithi102 ceph-mon[115269]: 1.5 scrub starts 2024-09-13T14:47:31.626 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:31 smithi102 ceph-mon[115269]: 1.5 scrub ok 2024-09-13T14:47:31.642 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:31.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:31.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:31.744 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:31.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:31 smithi195 ceph-mon[104262]: 1.6 scrub starts 2024-09-13T14:47:31.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:31 smithi195 ceph-mon[104262]: 1.6 scrub ok 2024-09-13T14:47:31.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:31 smithi195 ceph-mon[104262]: 1.4 scrub starts 2024-09-13T14:47:31.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:31 smithi195 ceph-mon[104262]: 1.4 scrub ok 2024-09-13T14:47:31.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:31 smithi195 ceph-mon[104262]: 1.5 scrub starts 2024-09-13T14:47:31.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:31 smithi195 ceph-mon[104262]: 1.5 scrub ok 2024-09-13T14:47:31.773 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:31.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:31.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:31.814 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 78 16 117684 117668 94.2652 72 0.0113026 0.0106039 2024-09-13T14:47:31.875 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:31.915 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:31.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:31.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:32.016 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:32.056 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:32.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:32.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:32.158 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:32.191 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:32.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:32.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:32.293 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:32.314 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:32.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:32.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:32.416 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:32.450 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:32.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:32.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:32.548 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 6, 7, 5, 2] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T14:47:32.548 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:47:32.548 INFO:tasks.thrashosds.thrasher:Adding osd 1 2024-09-13T14:47:32.548 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd in 1 2024-09-13T14:47:32.552 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:32.585 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:32 smithi102 ceph-mon[110183]: 1.e scrub starts 2024-09-13T14:47:32.585 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:32 smithi102 ceph-mon[110183]: 1.e scrub ok 2024-09-13T14:47:32.585 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:32 smithi102 ceph-mon[110183]: 1.14 scrub starts 2024-09-13T14:47:32.585 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:32 smithi102 ceph-mon[110183]: 1.14 scrub ok 2024-09-13T14:47:32.585 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:32 smithi102 ceph-mon[110183]: pgmap v1146: 109 pgs: 4 unknown, 105 active+clean; 7.0 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-13T14:47:32.585 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:32 smithi102 ceph-mon[110183]: 1.0 scrub starts 2024-09-13T14:47:32.586 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:32 smithi102 ceph-mon[110183]: 1.0 scrub ok 2024-09-13T14:47:32.586 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:32 smithi102 ceph-mon[115269]: 1.e scrub starts 2024-09-13T14:47:32.586 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:32 smithi102 ceph-mon[115269]: 1.e scrub ok 2024-09-13T14:47:32.586 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:32 smithi102 ceph-mon[115269]: 1.14 scrub starts 2024-09-13T14:47:32.586 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:32 smithi102 ceph-mon[115269]: 1.14 scrub ok 2024-09-13T14:47:32.586 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:32 smithi102 ceph-mon[115269]: pgmap v1146: 109 pgs: 4 unknown, 105 active+clean; 7.0 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-13T14:47:32.586 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:32 smithi102 ceph-mon[115269]: 1.0 scrub starts 2024-09-13T14:47:32.586 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:32 smithi102 ceph-mon[115269]: 1.0 scrub ok 2024-09-13T14:47:32.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:32.645 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:32.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:32.745 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:32.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:32 smithi195 ceph-mon[104262]: 1.e scrub starts 2024-09-13T14:47:32.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:32 smithi195 ceph-mon[104262]: 1.e scrub ok 2024-09-13T14:47:32.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:32 smithi195 ceph-mon[104262]: 1.14 scrub starts 2024-09-13T14:47:32.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:32 smithi195 ceph-mon[104262]: 1.14 scrub ok 2024-09-13T14:47:32.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:32 smithi195 ceph-mon[104262]: pgmap v1146: 109 pgs: 4 unknown, 105 active+clean; 7.0 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-13T14:47:32.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:32 smithi195 ceph-mon[104262]: 1.0 scrub starts 2024-09-13T14:47:32.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:32 smithi195 ceph-mon[104262]: 1.0 scrub ok 2024-09-13T14:47:32.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:32.780 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:32.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:32.784 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:47:32.784 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:47:32.815 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 79 16 118737 118721 93.9049 65.8125 0.0151444 0.0106439 2024-09-13T14:47:32.822 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:47:32.823 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:47:32.840 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:47:32.840 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:47:32.853 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:47:32.853 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:47:32.881 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:32.882 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:47:32.882 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:47:32.913 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:47:32.913 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:47:32.917 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:32.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:32.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:32.945 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:47:32.945 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:47:32.981 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:47:32.982 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:47:33.016 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:47:33.016 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:47:33.019 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:33.064 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:33.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:33.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:33.165 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:33.203 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:33.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:33.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:33.305 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:33.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:33.335 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:33.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:33.436 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:33.473 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:33.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:33.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:33.575 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:33.608 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:33.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:33.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:33.660 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:47:33.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:33 smithi102 ceph-mon[110183]: 1.c scrub starts 2024-09-13T14:47:33.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:33 smithi102 ceph-mon[110183]: 1.c scrub ok 2024-09-13T14:47:33.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:33 smithi102 ceph-mon[110183]: 1.1 scrub starts 2024-09-13T14:47:33.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:33 smithi102 ceph-mon[110183]: 1.1 scrub ok 2024-09-13T14:47:33.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:33 smithi102 ceph-mon[110183]: 1.12 scrub starts 2024-09-13T14:47:33.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:33 smithi102 ceph-mon[110183]: 1.12 scrub ok 2024-09-13T14:47:33.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:33 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1746873429' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:47:33.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:33 smithi102 ceph-mon[110183]: 1.7 scrub starts 2024-09-13T14:47:33.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:33 smithi102 ceph-mon[110183]: 1.7 scrub ok 2024-09-13T14:47:33.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:33 smithi102 ceph-mon[115269]: 1.c scrub starts 2024-09-13T14:47:33.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:33 smithi102 ceph-mon[115269]: 1.c scrub ok 2024-09-13T14:47:33.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:33 smithi102 ceph-mon[115269]: 1.1 scrub starts 2024-09-13T14:47:33.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:33 smithi102 ceph-mon[115269]: 1.1 scrub ok 2024-09-13T14:47:33.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:33 smithi102 ceph-mon[115269]: 1.12 scrub starts 2024-09-13T14:47:33.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:33 smithi102 ceph-mon[115269]: 1.12 scrub ok 2024-09-13T14:47:33.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:33 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1746873429' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:47:33.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:33 smithi102 ceph-mon[115269]: 1.7 scrub starts 2024-09-13T14:47:33.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:33 smithi102 ceph-mon[115269]: 1.7 scrub ok 2024-09-13T14:47:33.710 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:33.737 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:33.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:33.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:33.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:33 smithi195 ceph-mon[104262]: 1.c scrub starts 2024-09-13T14:47:33.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:33 smithi195 ceph-mon[104262]: 1.c scrub ok 2024-09-13T14:47:33.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:33 smithi195 ceph-mon[104262]: 1.1 scrub starts 2024-09-13T14:47:33.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:33 smithi195 ceph-mon[104262]: 1.1 scrub ok 2024-09-13T14:47:33.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:33 smithi195 ceph-mon[104262]: 1.12 scrub starts 2024-09-13T14:47:33.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:33 smithi195 ceph-mon[104262]: 1.12 scrub ok 2024-09-13T14:47:33.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:33 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1746873429' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:47:33.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:33 smithi195 ceph-mon[104262]: 1.7 scrub starts 2024-09-13T14:47:33.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:33 smithi195 ceph-mon[104262]: 1.7 scrub ok 2024-09-13T14:47:33.815 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:2024-09-13T14:47:33.814956+0000 min lat: 0.00109072 max lat: 0.310704 avg lat: 0.010679 2024-09-13T14:47:33.815 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:47:33.815 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 80 16 119851 119835 93.6013 69.625 0.0141409 0.010679 2024-09-13T14:47:33.839 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:33.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:33.878 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:33.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:33.979 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:34.010 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:34.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:34.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:34.111 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:34.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:34.151 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:34.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:34.251 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:34.292 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:34.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:34.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:34.303 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T14:47:34.394 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:34.436 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:34.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:34.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:34.539 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:34.580 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:34.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:34.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:34.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:34 smithi102 ceph-mon[110183]: pgmap v1147: 109 pgs: 109 active+clean; 7.2 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-09-13T14:47:34.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:34 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-13T14:47:34.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:34 smithi102 ceph-mon[110183]: 1.f scrub starts 2024-09-13T14:47:34.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:34 smithi102 ceph-mon[110183]: 1.f scrub ok 2024-09-13T14:47:34.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:34 smithi102 ceph-mon[115269]: pgmap v1147: 109 pgs: 109 active+clean; 7.2 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-09-13T14:47:34.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:34 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-13T14:47:34.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:34 smithi102 ceph-mon[115269]: 1.f scrub starts 2024-09-13T14:47:34.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:34 smithi102 ceph-mon[115269]: 1.f scrub ok 2024-09-13T14:47:34.683 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:34.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:34.717 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:34.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:34.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:34 smithi195 ceph-mon[104262]: pgmap v1147: 109 pgs: 109 active+clean; 7.2 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-09-13T14:47:34.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:34 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-13T14:47:34.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:34 smithi195 ceph-mon[104262]: 1.f scrub starts 2024-09-13T14:47:34.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:34 smithi195 ceph-mon[104262]: 1.f scrub ok 2024-09-13T14:47:34.815 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 81 16 120974 120958 93.3122 70.1875 0.00653577 0.0107117 2024-09-13T14:47:34.818 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:34.855 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:34.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:34.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:34.957 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:34.991 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:34.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:34.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:35.093 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:35.123 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:35.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:35.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:35.175 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:35.199 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:35.225 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:35.235 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:35.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:35.286 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:35.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:35.387 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:35.419 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:35.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:35.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:35.521 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:35.556 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:35.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:35.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:35.658 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:35.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:35 smithi102 ceph-mon[110183]: 1.11 scrub starts 2024-09-13T14:47:35.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:35 smithi102 ceph-mon[110183]: 1.11 scrub ok 2024-09-13T14:47:35.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:35 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-13T14:47:35.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:35 smithi102 ceph-mon[110183]: osdmap e331: 8 total, 8 up, 7 in 2024-09-13T14:47:35.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:35 smithi102 ceph-mon[110183]: 1.3 scrub starts 2024-09-13T14:47:35.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:35 smithi102 ceph-mon[110183]: 1.3 scrub ok 2024-09-13T14:47:35.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:35 smithi102 ceph-mon[115269]: 1.11 scrub starts 2024-09-13T14:47:35.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:35 smithi102 ceph-mon[115269]: 1.11 scrub ok 2024-09-13T14:47:35.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:35 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-13T14:47:35.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:35 smithi102 ceph-mon[115269]: osdmap e331: 8 total, 8 up, 7 in 2024-09-13T14:47:35.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:35 smithi102 ceph-mon[115269]: 1.3 scrub starts 2024-09-13T14:47:35.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:35 smithi102 ceph-mon[115269]: 1.3 scrub ok 2024-09-13T14:47:35.691 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:35.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:35.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:35.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:35 smithi195 ceph-mon[104262]: 1.11 scrub starts 2024-09-13T14:47:35.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:35 smithi195 ceph-mon[104262]: 1.11 scrub ok 2024-09-13T14:47:35.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:35 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-13T14:47:35.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:35 smithi195 ceph-mon[104262]: osdmap e331: 8 total, 8 up, 7 in 2024-09-13T14:47:35.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:35 smithi195 ceph-mon[104262]: 1.3 scrub starts 2024-09-13T14:47:35.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:35 smithi195 ceph-mon[104262]: 1.3 scrub ok 2024-09-13T14:47:35.793 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:35.815 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 82 16 122090 122074 93.0248 69.75 0.0259322 0.010745 2024-09-13T14:47:35.826 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:35.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:35.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:35.928 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:35.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:35.965 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:35.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:36.066 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:36.102 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:36.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:36.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:36.204 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:36.238 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:36.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:36.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:36.340 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:36.371 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:36.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:36.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:36.473 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:36.506 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:36.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:36.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:36.607 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:36.638 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:36.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:36.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:36.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:36 smithi102 ceph-mon[110183]: osdmap e332: 8 total, 8 up, 7 in 2024-09-13T14:47:36.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:36 smithi102 ceph-mon[110183]: pgmap v1150: 109 pgs: 109 active+clean; 7.2 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 55 MiB/s wr, 873 op/s 2024-09-13T14:47:36.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:36 smithi102 ceph-mon[110183]: osdmap e333: 8 total, 8 up, 7 in 2024-09-13T14:47:36.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:36 smithi102 ceph-mon[115269]: osdmap e332: 8 total, 8 up, 7 in 2024-09-13T14:47:36.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:36 smithi102 ceph-mon[115269]: pgmap v1150: 109 pgs: 109 active+clean; 7.2 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 55 MiB/s wr, 873 op/s 2024-09-13T14:47:36.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:36 smithi102 ceph-mon[115269]: osdmap e333: 8 total, 8 up, 7 in 2024-09-13T14:47:36.669 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:47:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:47:36] "GET /metrics HTTP/1.1" 200 36511 "" "Prometheus/2.51.0" 2024-09-13T14:47:36.740 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:36.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:36 smithi195 ceph-mon[104262]: osdmap e332: 8 total, 8 up, 7 in 2024-09-13T14:47:36.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:36 smithi195 ceph-mon[104262]: pgmap v1150: 109 pgs: 109 active+clean; 7.2 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 55 MiB/s wr, 873 op/s 2024-09-13T14:47:36.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:36 smithi195 ceph-mon[104262]: osdmap e333: 8 total, 8 up, 7 in 2024-09-13T14:47:36.780 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:36.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:36.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:36.815 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 83 16 123249 123233 92.7768 72.4375 0.0130871 0.0107738 2024-09-13T14:47:36.882 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:36.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:36.917 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:36.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:37.018 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:37.051 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:37.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:37.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:37.086 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:47:37.153 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:37.185 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:37.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:37.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:37.286 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:37.322 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:37.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:37.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:37.424 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:37.455 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:37.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:37.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:37.556 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:37.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:37.589 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:37.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:37.691 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:37.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:37.727 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:37.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:37.815 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 84 16 124389 124373 92.5204 71.25 0.00579407 0.0108035 2024-09-13T14:47:37.828 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:37.876 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:37.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:37.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:37.977 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:38.013 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:38.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:38.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:38.114 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:38.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:38.150 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:38.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:38.251 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:38.281 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:38.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:38.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:38.383 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:38.418 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:38.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:38.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:38.520 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:38.557 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:38.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:38.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:38.659 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:38.697 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:38.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:38.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:38.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:38 smithi195 ceph-mon[104262]: osdmap e334: 8 total, 8 up, 7 in 2024-09-13T14:47:38.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:38 smithi195 ceph-mon[104262]: pgmap v1153: 109 pgs: 1 active+remapped, 4 active+undersized+remapped, 2 remapped+peering, 102 active+clean; 7.4 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 38 KiB/s rd, 65 MiB/s wr, 1.07k op/s; 79 KiB/s, 0 objects/s recovering 2024-09-13T14:47:38.800 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:38.815 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 85 16 125508 125492 92.2547 69.9375 0.00849026 0.0108344 2024-09-13T14:47:38.829 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:38.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:38.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:38.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:38 smithi102 ceph-mon[110183]: osdmap e334: 8 total, 8 up, 7 in 2024-09-13T14:47:38.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:38 smithi102 ceph-mon[110183]: pgmap v1153: 109 pgs: 1 active+remapped, 4 active+undersized+remapped, 2 remapped+peering, 102 active+clean; 7.4 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 38 KiB/s rd, 65 MiB/s wr, 1.07k op/s; 79 KiB/s, 0 objects/s recovering 2024-09-13T14:47:38.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:38 smithi102 ceph-mon[115269]: osdmap e334: 8 total, 8 up, 7 in 2024-09-13T14:47:38.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:38 smithi102 ceph-mon[115269]: pgmap v1153: 109 pgs: 1 active+remapped, 4 active+undersized+remapped, 2 remapped+peering, 102 active+clean; 7.4 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 38 KiB/s rd, 65 MiB/s wr, 1.07k op/s; 79 KiB/s, 0 objects/s recovering 2024-09-13T14:47:38.931 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:38.964 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:38.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:38.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:39.066 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:39.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:39.103 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:39.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:39.205 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:39.241 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:39.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:39.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:39.343 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:39.377 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:39.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:39.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:39.479 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:39.511 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:39.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:39.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:39.613 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:39.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:39.649 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:39.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:39.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:39 smithi195 ceph-mon[104262]: osdmap e335: 8 total, 8 up, 7 in 2024-09-13T14:47:39.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:39 smithi195 ceph-mon[104262]: 1.11 scrub starts 2024-09-13T14:47:39.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:39 smithi195 ceph-mon[104262]: 1.11 scrub ok 2024-09-13T14:47:39.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:39 smithi195 ceph-mon[104262]: 1.13 scrub starts 2024-09-13T14:47:39.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:39 smithi195 ceph-mon[104262]: 1.13 scrub ok 2024-09-13T14:47:39.750 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:39.792 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:39.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:39.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:39.815 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 86 16 126535 126519 91.9283 64.1875 0.0134962 0.0108735 2024-09-13T14:47:39.894 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:39.904 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:39 smithi102 ceph-mon[115269]: osdmap e335: 8 total, 8 up, 7 in 2024-09-13T14:47:39.904 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:39 smithi102 ceph-mon[115269]: 1.11 scrub starts 2024-09-13T14:47:39.904 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:39 smithi102 ceph-mon[115269]: 1.11 scrub ok 2024-09-13T14:47:39.904 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:39 smithi102 ceph-mon[115269]: 1.13 scrub starts 2024-09-13T14:47:39.905 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:39 smithi102 ceph-mon[115269]: 1.13 scrub ok 2024-09-13T14:47:39.905 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:39 smithi102 ceph-mon[110183]: osdmap e335: 8 total, 8 up, 7 in 2024-09-13T14:47:39.905 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:39 smithi102 ceph-mon[110183]: 1.11 scrub starts 2024-09-13T14:47:39.905 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:39 smithi102 ceph-mon[110183]: 1.11 scrub ok 2024-09-13T14:47:39.905 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:39 smithi102 ceph-mon[110183]: 1.13 scrub starts 2024-09-13T14:47:39.905 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:39 smithi102 ceph-mon[110183]: 1.13 scrub ok 2024-09-13T14:47:39.929 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:39.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:39.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:40.030 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:40.064 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:40.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:40.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:40.165 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:40.189 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:40.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:40.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:40.291 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:40.328 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:40.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:40.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:40.430 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:40.466 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:40.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:40.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:40.568 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:40.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:40.604 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:40.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:40.705 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:40.736 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:40.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:40.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:40.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:40 smithi195 ceph-mon[104262]: pgmap v1155: 109 pgs: 2 peering, 1 active+remapped, 4 active+undersized+remapped, 1 remapped+peering, 101 active+clean; 7.5 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 36 KiB/s rd, 83 MiB/s wr, 1.35k op/s; 151 KiB/s, 0 objects/s recovering 2024-09-13T14:47:40.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:40 smithi195 ceph-mon[104262]: 1.b scrub starts 2024-09-13T14:47:40.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:40 smithi195 ceph-mon[104262]: 1.b scrub ok 2024-09-13T14:47:40.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:40 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:47:40.815 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 87 16 127634 127618 91.6612 68.6875 0.0102654 0.0109048 2024-09-13T14:47:40.838 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:40.871 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:40.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:40.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:40.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:40 smithi102 ceph-mon[110183]: pgmap v1155: 109 pgs: 2 peering, 1 active+remapped, 4 active+undersized+remapped, 1 remapped+peering, 101 active+clean; 7.5 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 36 KiB/s rd, 83 MiB/s wr, 1.35k op/s; 151 KiB/s, 0 objects/s recovering 2024-09-13T14:47:40.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:40 smithi102 ceph-mon[110183]: 1.b scrub starts 2024-09-13T14:47:40.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:40 smithi102 ceph-mon[110183]: 1.b scrub ok 2024-09-13T14:47:40.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:40 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:47:40.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:40 smithi102 ceph-mon[115269]: pgmap v1155: 109 pgs: 2 peering, 1 active+remapped, 4 active+undersized+remapped, 1 remapped+peering, 101 active+clean; 7.5 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 36 KiB/s rd, 83 MiB/s wr, 1.35k op/s; 151 KiB/s, 0 objects/s recovering 2024-09-13T14:47:40.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:40 smithi102 ceph-mon[115269]: 1.b scrub starts 2024-09-13T14:47:40.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:40 smithi102 ceph-mon[115269]: 1.b scrub ok 2024-09-13T14:47:40.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:40 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:47:40.974 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:41.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:41.002 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:41.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:41.103 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:41.131 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:41.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:41.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:41.233 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:41.267 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:41.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:41.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:41.369 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:41.410 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:41.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:41.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:41.511 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:41.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:41.546 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:41.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:41.647 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:41.684 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:41.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:41.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:41.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:41 smithi195 ceph-mon[104262]: 1.d scrub starts 2024-09-13T14:47:41.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:41 smithi195 ceph-mon[104262]: 1.d scrub ok 2024-09-13T14:47:41.786 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:41.815 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 88 16 128692 128676 91.371 66.125 0.0171618 0.0109398 2024-09-13T14:47:41.819 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:41.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:41.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:41.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:41 smithi102 ceph-mon[110183]: 1.d scrub starts 2024-09-13T14:47:41.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:41 smithi102 ceph-mon[110183]: 1.d scrub ok 2024-09-13T14:47:41.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:41 smithi102 ceph-mon[115269]: 1.d scrub starts 2024-09-13T14:47:41.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:41 smithi102 ceph-mon[115269]: 1.d scrub ok 2024-09-13T14:47:41.921 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:41.957 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:41.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:41.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:42.059 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:42.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:42.090 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:42.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:42.191 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:42.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:42.230 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:42.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:42.331 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:42.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:42.353 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:42.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:42.454 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:42.488 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:42.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:42.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:42.590 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:42.627 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:42.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:42.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:42.729 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:42.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:42 smithi195 ceph-mon[104262]: 1.10 scrub starts 2024-09-13T14:47:42.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:42 smithi195 ceph-mon[104262]: 1.10 scrub ok 2024-09-13T14:47:42.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:42 smithi195 ceph-mon[104262]: pgmap v1156: 109 pgs: 2 peering, 1 active+undersized+remapped, 1 remapped+peering, 105 active+clean; 7.7 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 19 KiB/s rd, 87 MiB/s wr, 1.41k op/s; 93 KiB/s, 0 objects/s recovering 2024-09-13T14:47:42.750 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:42.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:42.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:42.816 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 89 16 129562 129546 90.9553 54.375 0.00351346 0.0109737 2024-09-13T14:47:42.851 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:42.886 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:42.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:42.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:42.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:42 smithi102 ceph-mon[115269]: 1.10 scrub starts 2024-09-13T14:47:42.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:42 smithi102 ceph-mon[115269]: 1.10 scrub ok 2024-09-13T14:47:42.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:42 smithi102 ceph-mon[115269]: pgmap v1156: 109 pgs: 2 peering, 1 active+undersized+remapped, 1 remapped+peering, 105 active+clean; 7.7 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 19 KiB/s rd, 87 MiB/s wr, 1.41k op/s; 93 KiB/s, 0 objects/s recovering 2024-09-13T14:47:42.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:42 smithi102 ceph-mon[110183]: 1.10 scrub starts 2024-09-13T14:47:42.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:42 smithi102 ceph-mon[110183]: 1.10 scrub ok 2024-09-13T14:47:42.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:42 smithi102 ceph-mon[110183]: pgmap v1156: 109 pgs: 2 peering, 1 active+undersized+remapped, 1 remapped+peering, 105 active+clean; 7.7 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 19 KiB/s rd, 87 MiB/s wr, 1.41k op/s; 93 KiB/s, 0 objects/s recovering 2024-09-13T14:47:42.987 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:43.022 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:43.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:43.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:43.124 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:43.156 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:43.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:43.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:43.258 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:43.295 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:43.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:43.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:43.397 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:43.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:43.430 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:43.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:43.532 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:43.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:43.565 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:43.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:43.666 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:43.700 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:43.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:43.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:43.802 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:43.816 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Total time run: 90.0089 2024-09-13T14:47:43.816 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Total writes made: 130584 2024-09-13T14:47:43.816 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Write size: 65536 2024-09-13T14:47:43.816 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Object size: 65536 2024-09-13T14:47:43.816 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Bandwidth (MB/sec): 90.6743 2024-09-13T14:47:43.816 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Stddev Bandwidth: 39.2524 2024-09-13T14:47:43.816 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Max bandwidth (MB/sec): 254.75 2024-09-13T14:47:43.816 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Min bandwidth (MB/sec): 38.0625 2024-09-13T14:47:43.816 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Average IOPS: 1450 2024-09-13T14:47:43.816 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Stddev IOPS: 628.038 2024-09-13T14:47:43.816 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Max IOPS: 4076 2024-09-13T14:47:43.817 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Min IOPS: 609 2024-09-13T14:47:43.817 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Average Latency(s): 0.0110243 2024-09-13T14:47:43.817 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Stddev Latency(s): 0.0100644 2024-09-13T14:47:43.817 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Max latency(s): 0.310704 2024-09-13T14:47:43.817 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Min latency(s): 0.00109072 2024-09-13T14:47:43.820 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Cleaning up (deleting benchmark objects) 2024-09-13T14:47:43.824 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:43.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:43.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:43.926 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:43.959 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:43.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:43.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:44.061 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:44.090 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:44.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:44.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:44.192 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:44.227 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:44.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:44.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:44.329 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:44.363 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:44.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:44.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:44.464 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:44.495 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:44.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:44.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:44.597 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:44.625 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:44.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:44.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:44.727 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:44.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:44.774 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:44.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:44.875 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:44.908 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:44.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:44.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:44.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:44 smithi102 ceph-mon[110183]: pgmap v1157: 109 pgs: 109 active+clean; 7.9 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 10 KiB/s rd, 74 MiB/s wr, 1.20k op/s; 66 KiB/s, 0 objects/s recovering 2024-09-13T14:47:44.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:44 smithi102 ceph-mon[115269]: pgmap v1157: 109 pgs: 109 active+clean; 7.9 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 10 KiB/s rd, 74 MiB/s wr, 1.20k op/s; 66 KiB/s, 0 objects/s recovering 2024-09-13T14:47:44.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:44 smithi195 ceph-mon[104262]: pgmap v1157: 109 pgs: 109 active+clean; 7.9 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 10 KiB/s rd, 74 MiB/s wr, 1.20k op/s; 66 KiB/s, 0 objects/s recovering 2024-09-13T14:47:45.001 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:45.010 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:45.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:45.079 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:45.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:45.181 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:45.212 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:45.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:45.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:45.314 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:45.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:45.357 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:45.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:45.458 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:45.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:45.522 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:45.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:45.624 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:45.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:45.667 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:45.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:45.768 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:45.821 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:45.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:45.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:45.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:45 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:47:45.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:45 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:47:45.923 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:45.960 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:45.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:45.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:45.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:45 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:47:46.065 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:46.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:46.102 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:46.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:46.203 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:46.242 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:46.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:46.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:46.395 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:46.438 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:46.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:46.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:46.540 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:46.586 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:46.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:46.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:46.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-a[110159]: 2024-09-13T14:47:46.221+0000 7fa3e7f42640 -1 mon.a@0(leader).osd e335 definitely_dead 0 2024-09-13T14:47:46.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:47:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:47:46] "GET /metrics HTTP/1.1" 200 36530 "" "Prometheus/2.51.0" 2024-09-13T14:47:46.688 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:46.723 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:46.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:46.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:46.824 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:46.846 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is unset 2024-09-13T14:47:46.864 INFO:teuthology.orchestra.run.smithi102.stderr:marked in osd.1. 2024-09-13T14:47:46.875 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:46.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:46.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:46.921 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:46 smithi102 ceph-mon[110183]: 1.12 scrub starts 2024-09-13T14:47:46.922 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:46 smithi102 ceph-mon[110183]: 1.12 scrub ok 2024-09-13T14:47:46.922 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:46 smithi102 ceph-mon[110183]: pgmap v1158: 109 pgs: 109 active+clean; 7.9 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 52 MiB/s wr, 833 op/s; 20 KiB/s, 0 objects/s recovering 2024-09-13T14:47:46.922 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:47:46.922 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:47:46.922 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:46 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2929366064' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:47:46.922 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:46 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/220695361' entity='client.admin' cmd={"prefix": "osd in", "ids": ["1"]} : dispatch 2024-09-13T14:47:46.922 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:46 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["1"]} : dispatch 2024-09-13T14:47:46.922 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:46 smithi102 ceph-mon[115269]: 1.12 scrub starts 2024-09-13T14:47:46.922 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:46 smithi102 ceph-mon[115269]: 1.12 scrub ok 2024-09-13T14:47:46.922 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:46 smithi102 ceph-mon[115269]: pgmap v1158: 109 pgs: 109 active+clean; 7.9 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 52 MiB/s wr, 833 op/s; 20 KiB/s, 0 objects/s recovering 2024-09-13T14:47:46.922 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:47:46.923 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:47:46.923 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:46 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2929366064' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:47:46.923 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:46 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/220695361' entity='client.admin' cmd={"prefix": "osd in", "ids": ["1"]} : dispatch 2024-09-13T14:47:46.923 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:46 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["1"]} : dispatch 2024-09-13T14:47:46.977 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:46.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:46 smithi195 ceph-mon[104262]: 1.12 scrub starts 2024-09-13T14:47:46.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:46 smithi195 ceph-mon[104262]: 1.12 scrub ok 2024-09-13T14:47:46.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:46 smithi195 ceph-mon[104262]: pgmap v1158: 109 pgs: 109 active+clean; 7.9 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 52 MiB/s wr, 833 op/s; 20 KiB/s, 0 objects/s recovering 2024-09-13T14:47:46.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:47:46.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:47:46.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:46 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2929366064' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:47:46.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:46 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/220695361' entity='client.admin' cmd={"prefix": "osd in", "ids": ["1"]} : dispatch 2024-09-13T14:47:46.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:46 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["1"]} : dispatch 2024-09-13T14:47:46.999 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:46.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:47.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:47.100 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:47.122 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:47.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:47.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:47.224 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:47.246 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:47.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:47.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:47.348 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:47.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:47.371 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:47.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:47.472 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:47.494 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:47.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:47.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:47.596 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:47.618 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:47.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:47.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:47.648 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:47:47.648 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [], 2024-09-13T14:47:47.648 INFO:teuthology.orchestra.run.smithi102.stdout: "complaint_time": 30, 2024-09-13T14:47:47.648 INFO:teuthology.orchestra.run.smithi102.stdout: "num_blocked_ops": 0 2024-09-13T14:47:47.648 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:47:47.720 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:47.741 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:47.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:47.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:47.843 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:47.882 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:47.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:47.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:47.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:47 smithi102 ceph-mon[110183]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:47:47.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:47 smithi102 ceph-mon[110183]: Cluster is now healthy 2024-09-13T14:47:47.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:47 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2929366064' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:47:47.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:47 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-09-13T14:47:47.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:47 smithi102 ceph-mon[110183]: osdmap e336: 8 total, 8 up, 8 in 2024-09-13T14:47:47.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:47 smithi102 ceph-mon[115269]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:47:47.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:47 smithi102 ceph-mon[115269]: Cluster is now healthy 2024-09-13T14:47:47.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:47 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2929366064' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:47:47.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:47 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-09-13T14:47:47.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:47 smithi102 ceph-mon[115269]: osdmap e336: 8 total, 8 up, 8 in 2024-09-13T14:47:47.984 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:47.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:47 smithi195 ceph-mon[104262]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:47:47.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:47 smithi195 ceph-mon[104262]: Cluster is now healthy 2024-09-13T14:47:47.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:47 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2929366064' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:47:47.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:47 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-09-13T14:47:47.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:47 smithi195 ceph-mon[104262]: osdmap e336: 8 total, 8 up, 8 in 2024-09-13T14:47:48.006 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:48.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:48.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:48.107 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:48.129 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:48.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:48.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:48.231 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:48.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:48.257 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:48.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:48.358 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:48.379 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:48.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:48.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:48.481 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:48.507 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:48.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:48.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:48.609 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:48.630 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:48.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:48.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:48.670 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T14:47:48.732 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:48.756 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:48.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:48.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:48.857 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:48.878 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:48.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:48.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:48.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:48 smithi102 ceph-mon[110183]: pgmap v1160: 109 pgs: 3 peering, 106 active+clean; 7.8 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 48 MiB/s wr, 974 op/s 2024-09-13T14:47:48.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:48 smithi102 ceph-mon[110183]: osdmap e337: 8 total, 8 up, 8 in 2024-09-13T14:47:48.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:48 smithi102 ceph-mon[115269]: pgmap v1160: 109 pgs: 3 peering, 106 active+clean; 7.8 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 48 MiB/s wr, 974 op/s 2024-09-13T14:47:48.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:48 smithi102 ceph-mon[115269]: osdmap e337: 8 total, 8 up, 8 in 2024-09-13T14:47:48.980 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:49.001 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:49.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:49.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:49.103 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:49.125 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:49.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:49.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:49.224 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:48 smithi195 ceph-mon[104262]: pgmap v1160: 109 pgs: 3 peering, 106 active+clean; 7.8 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 48 MiB/s wr, 974 op/s 2024-09-13T14:47:49.225 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:48 smithi195 ceph-mon[104262]: osdmap e337: 8 total, 8 up, 8 in 2024-09-13T14:47:49.227 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:49.264 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:49.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:49.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:49.365 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:49.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:49.401 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:49.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:49.502 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:49.537 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:49.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:49.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:49.639 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:49.681 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:49.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:49.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:49.692 INFO:tasks.thrashosds.thrasher:Added osd 1 2024-09-13T14:47:49.783 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:49.821 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:49.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:49.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:49.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:49 smithi102 ceph-mon[110183]: osdmap e338: 8 total, 8 up, 8 in 2024-09-13T14:47:49.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:49 smithi102 ceph-mon[110183]: 8.7 starting backfill to osd.1 from (0'0,0'0] MAX to 335'8418 2024-09-13T14:47:49.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:49 smithi102 ceph-mon[110183]: 8.3 starting backfill to osd.1 from (0'0,0'0] MAX to 335'8331 2024-09-13T14:47:49.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:49 smithi102 ceph-mon[110183]: 8.a starting backfill to osd.1 from (0'0,0'0] MAX to 335'8512 2024-09-13T14:47:49.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:49 smithi102 ceph-mon[110183]: 8.1 starting backfill to osd.1 from (0'0,0'0] MAX to 335'8284 2024-09-13T14:47:49.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:49 smithi102 ceph-mon[110183]: 8.b starting backfill to osd.1 from (0'0,0'0] MAX to 335'8392 2024-09-13T14:47:49.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:49 smithi102 ceph-mon[110183]: 8.f starting backfill to osd.1 from (0'0,0'0] MAX to 335'8391 2024-09-13T14:47:49.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:49 smithi102 ceph-mon[110183]: 8.0 starting backfill to osd.1 from (0'0,0'0] MAX to 335'8317 2024-09-13T14:47:49.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:49 smithi102 ceph-mon[115269]: osdmap e338: 8 total, 8 up, 8 in 2024-09-13T14:47:49.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:49 smithi102 ceph-mon[115269]: 8.7 starting backfill to osd.1 from (0'0,0'0] MAX to 335'8418 2024-09-13T14:47:49.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:49 smithi102 ceph-mon[115269]: 8.3 starting backfill to osd.1 from (0'0,0'0] MAX to 335'8331 2024-09-13T14:47:49.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:49 smithi102 ceph-mon[115269]: 8.a starting backfill to osd.1 from (0'0,0'0] MAX to 335'8512 2024-09-13T14:47:49.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:49 smithi102 ceph-mon[115269]: 8.1 starting backfill to osd.1 from (0'0,0'0] MAX to 335'8284 2024-09-13T14:47:49.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:49 smithi102 ceph-mon[115269]: 8.b starting backfill to osd.1 from (0'0,0'0] MAX to 335'8392 2024-09-13T14:47:49.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:49 smithi102 ceph-mon[115269]: 8.f starting backfill to osd.1 from (0'0,0'0] MAX to 335'8391 2024-09-13T14:47:49.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:49 smithi102 ceph-mon[115269]: 8.0 starting backfill to osd.1 from (0'0,0'0] MAX to 335'8317 2024-09-13T14:47:49.923 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:49.953 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:49.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:49.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:50.055 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:50.090 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:50.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:50.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:50.193 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:50.222 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:50.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:50.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:50.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:49 smithi195 ceph-mon[104262]: osdmap e338: 8 total, 8 up, 8 in 2024-09-13T14:47:50.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:49 smithi195 ceph-mon[104262]: 8.7 starting backfill to osd.1 from (0'0,0'0] MAX to 335'8418 2024-09-13T14:47:50.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:49 smithi195 ceph-mon[104262]: 8.3 starting backfill to osd.1 from (0'0,0'0] MAX to 335'8331 2024-09-13T14:47:50.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:49 smithi195 ceph-mon[104262]: 8.a starting backfill to osd.1 from (0'0,0'0] MAX to 335'8512 2024-09-13T14:47:50.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:49 smithi195 ceph-mon[104262]: 8.1 starting backfill to osd.1 from (0'0,0'0] MAX to 335'8284 2024-09-13T14:47:50.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:49 smithi195 ceph-mon[104262]: 8.b starting backfill to osd.1 from (0'0,0'0] MAX to 335'8392 2024-09-13T14:47:50.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:49 smithi195 ceph-mon[104262]: 8.f starting backfill to osd.1 from (0'0,0'0] MAX to 335'8391 2024-09-13T14:47:50.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:49 smithi195 ceph-mon[104262]: 8.0 starting backfill to osd.1 from (0'0,0'0] MAX to 335'8317 2024-09-13T14:47:50.324 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:50.359 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:50.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:50.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:50.461 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:50.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:50.524 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:50.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:50.625 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:50.653 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:50.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:50.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:50.755 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:50.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:50.794 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:50.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:50.895 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:50.931 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:50.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:50.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:51.033 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:51.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:51.073 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:51.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:51.111 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:50 smithi102 ceph-mon[110183]: pgmap v1163: 109 pgs: 12 peering, 97 active+clean; 7.8 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 18 MiB/s wr, 723 op/s 2024-09-13T14:47:51.111 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:50 smithi102 ceph-mon[110183]: osdmap e339: 8 total, 8 up, 8 in 2024-09-13T14:47:51.112 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:50 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:47:51.112 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:50 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:47:51.112 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:50 smithi102 ceph-mon[115269]: pgmap v1163: 109 pgs: 12 peering, 97 active+clean; 7.8 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 18 MiB/s wr, 723 op/s 2024-09-13T14:47:51.112 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:50 smithi102 ceph-mon[115269]: osdmap e339: 8 total, 8 up, 8 in 2024-09-13T14:47:51.112 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:50 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:47:51.112 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:50 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:47:51.174 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:51.202 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:51.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:51.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:51.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:50 smithi195 ceph-mon[104262]: pgmap v1163: 109 pgs: 12 peering, 97 active+clean; 7.8 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 18 MiB/s wr, 723 op/s 2024-09-13T14:47:51.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:50 smithi195 ceph-mon[104262]: osdmap e339: 8 total, 8 up, 8 in 2024-09-13T14:47:51.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:50 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:47:51.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:50 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:47:51.304 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:51.305 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:47:51.305 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:47:51.344 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:47:51.344 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:47:51.350 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:51.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:51.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:51.380 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:47:51.381 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:47:51.424 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:47:51.424 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:47:51.451 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:51.462 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:47:51.463 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:47:51.490 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:51.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:51.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:51.497 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:47:51.497 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:47:51.528 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:47:51.528 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:47:51.564 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:47:51.565 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:47:51.592 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:51.618 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:51.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:51.620 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:51.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:51.721 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:51.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:51.767 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:51.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:51.868 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:51.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:51.905 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:51.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:52.006 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:52.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:52.054 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:52.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:52.110 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:47:52.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:51 smithi102 ceph-mon[110183]: osdmap e340: 8 total, 8 up, 8 in 2024-09-13T14:47:52.151 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:51 smithi102 ceph-mon[110183]: 1.13 scrub starts 2024-09-13T14:47:52.151 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:51 smithi102 ceph-mon[110183]: 1.13 scrub ok 2024-09-13T14:47:52.151 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:51 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/231680323' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:47:52.151 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:51 smithi102 ceph-mon[115269]: osdmap e340: 8 total, 8 up, 8 in 2024-09-13T14:47:52.151 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:51 smithi102 ceph-mon[115269]: 1.13 scrub starts 2024-09-13T14:47:52.151 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:51 smithi102 ceph-mon[115269]: 1.13 scrub ok 2024-09-13T14:47:52.151 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:51 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/231680323' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:47:52.155 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:52.187 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:52.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:52.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:52.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:51 smithi195 ceph-mon[104262]: osdmap e340: 8 total, 8 up, 8 in 2024-09-13T14:47:52.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:51 smithi195 ceph-mon[104262]: 1.13 scrub starts 2024-09-13T14:47:52.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:51 smithi195 ceph-mon[104262]: 1.13 scrub ok 2024-09-13T14:47:52.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:51 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/231680323' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:47:52.289 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:52.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:52.342 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:52.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:52.442 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:52.484 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:52.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:52.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:52.585 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:52.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:52.625 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:52.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:52.726 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:52.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:52.760 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:52.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:52.862 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:52.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:52.900 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:52.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:53.002 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:53.037 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:53.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:53.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:53.120 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:52 smithi102 ceph-mon[110183]: pgmap v1166: 109 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 12 peering, 94 active+clean; 7.7 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 4.0 MiB/s wr, 636 op/s; 23550/378732 objects misplaced (6.218%); 1.6 MiB/s, 14 objects/s recovering 2024-09-13T14:47:53.120 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:52 smithi102 ceph-mon[115269]: pgmap v1166: 109 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 12 peering, 94 active+clean; 7.7 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 4.0 MiB/s wr, 636 op/s; 23550/378732 objects misplaced (6.218%); 1.6 MiB/s, 14 objects/s recovering 2024-09-13T14:47:53.139 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:53.162 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:53.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:53.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:53.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:52 smithi195 ceph-mon[104262]: pgmap v1166: 109 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 12 peering, 94 active+clean; 7.7 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 4.0 MiB/s wr, 636 op/s; 23550/378732 objects misplaced (6.218%); 1.6 MiB/s, 14 objects/s recovering 2024-09-13T14:47:53.264 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:53.297 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:53.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:53.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:53.399 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:53.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:53.445 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:53.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:53.546 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:53.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:53.583 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:53.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:53.684 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:53.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:53.736 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:53.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:53.837 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:53.869 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:53.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:53.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:53.971 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:54.007 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:54.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:54.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:54.109 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:54.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:54.144 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:54.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:54.246 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:54.279 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:54.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:54.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:54.304 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:54.381 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:54.424 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:54.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:54.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:54.526 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:54.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:54.566 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:54.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:54.668 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:54.693 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 6, 7, 5, 2, 1] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T14:47:54.693 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:47:54.693 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-09-13T14:47:54.693 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T14:47:54.693 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T14:47:54.694 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-09-13T14:47:54.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:54.703 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:54.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:54.804 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:54.853 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:47:54.856 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:54.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:54.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:54.904 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:54 smithi195 ceph-mon[104262]: pgmap v1167: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 7.6 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 1.6 MiB/s wr, 576 op/s; 54379/374544 objects misplaced (14.519%); 3.3 MiB/s, 46 objects/s recovering 2024-09-13T14:47:54.962 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:54.963 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:47:54.963 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:47:54.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:54.999 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:55.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:55.018 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:47:55.101 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:55.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:55.150 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:55.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:55.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:54 smithi102 ceph-mon[115269]: pgmap v1167: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 7.6 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 1.6 MiB/s wr, 576 op/s; 54379/374544 objects misplaced (14.519%); 3.3 MiB/s, 46 objects/s recovering 2024-09-13T14:47:55.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:54 smithi102 ceph-mon[110183]: pgmap v1167: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 7.6 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 1.6 MiB/s wr, 576 op/s; 54379/374544 objects misplaced (14.519%); 3.3 MiB/s, 46 objects/s recovering 2024-09-13T14:47:55.251 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:55.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:55.286 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:55.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:55.387 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:55.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:55.426 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:55.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:55.527 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:55.563 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:55.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:55.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:55.665 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:55.706 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:55.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:55.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:55.808 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:55.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:55.846 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:55.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:55.947 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:55.981 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:55.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:55.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:56.083 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:56.131 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:56.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:56.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:56.233 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:56.244 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T14:47:56.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:56.271 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:56.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:56.372 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:56.408 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:56.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:56.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:56.511 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:56.536 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:56.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:56.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:56.638 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:56.639 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:47:56.639 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:47:56.639 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:56 smithi195 ceph-mon[104262]: pgmap v1168: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 7.6 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 1.3 MiB/s wr, 490 op/s; 54379/374544 objects misplaced (14.519%); 2.9 MiB/s, 43 objects/s recovering 2024-09-13T14:47:56.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:47:56.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:47:56.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:56 smithi102 ceph-mon[110183]: pgmap v1168: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 7.6 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 1.3 MiB/s wr, 490 op/s; 54379/374544 objects misplaced (14.519%); 2.9 MiB/s, 43 objects/s recovering 2024-09-13T14:47:56.669 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:47:56 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:47:56] "GET /metrics HTTP/1.1" 200 36541 "" "Prometheus/2.51.0" 2024-09-13T14:47:56.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:47:56.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:47:56.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:56 smithi102 ceph-mon[115269]: pgmap v1168: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 7.6 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 1.3 MiB/s wr, 490 op/s; 54379/374544 objects misplaced (14.519%); 2.9 MiB/s, 43 objects/s recovering 2024-09-13T14:47:56.680 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:56.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:56.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:56.782 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:56.818 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:56.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:56.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:56.888 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:56.920 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:56.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:56.949 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:56.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:57.050 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:57.079 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:57.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:57.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:57.140 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:47:57.140 INFO:teuthology.orchestra.run.smithi195.stdout: "success": "" 2024-09-13T14:47:57.140 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:47:57.181 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:57.218 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:57.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:57.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:57.320 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:57.356 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:57.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:57.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:57.458 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:57.487 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:57.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:57.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:57.506 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:57.589 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:57.637 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:57.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:57.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:57.739 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:57.766 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:57.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:57.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:57.868 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:57.899 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:57.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:57.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:58.001 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:58.045 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:58.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:58.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:58.149 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:58.174 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:58.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:58.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:58.276 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:58.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:58.321 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:58.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:58.422 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:58.448 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:58.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:58.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:58.550 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:58.575 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:58.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:58.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:58.676 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:58.704 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:58.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:58.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:58.805 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:58.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:58.853 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:58.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:58.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:58 smithi102 ceph-mon[110183]: pgmap v1169: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 7.4 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.1 MiB/s wr, 880 op/s; 52781/363342 objects misplaced (14.527%); 6.2 MiB/s, 97 objects/s recovering 2024-09-13T14:47:58.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:58 smithi102 ceph-mon[115269]: pgmap v1169: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 7.4 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.1 MiB/s wr, 880 op/s; 52781/363342 objects misplaced (14.527%); 6.2 MiB/s, 97 objects/s recovering 2024-09-13T14:47:58.954 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:58.979 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:58.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:58.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:58.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:58 smithi195 ceph-mon[104262]: pgmap v1169: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 7.4 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.1 MiB/s wr, 880 op/s; 52781/363342 objects misplaced (14.527%); 6.2 MiB/s, 97 objects/s recovering 2024-09-13T14:47:59.081 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:59.122 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:59.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:59.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:59.224 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:59.250 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:59.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:59.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:59.352 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:59.383 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:59.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:59.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:59.485 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:59.516 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:59.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:59.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:59.618 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:59.675 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is set 2024-09-13T14:47:59.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:59.688 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:47:59.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:59.789 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:59.821 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:59.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:59.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:59.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:59 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/225022655' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:47:59.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:47:59 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:47:59.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:59 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/225022655' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:47:59.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:47:59 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:47:59.922 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:59.960 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:47:59.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:59.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:59.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:59 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/225022655' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:47:59.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:47:59 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:48:00.062 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:00.086 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:00.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:00.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:00.189 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:00.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:00.245 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:00.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:00.346 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:00.381 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:00.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:00.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:00.483 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:00.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:00.514 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:00.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:00.615 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:00.654 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:00.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:00.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:00.756 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:00.793 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:00.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:00.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:00.895 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:00.926 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:00.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:00.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:00.946 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:00 smithi102 ceph-mon[110183]: pgmap v1170: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 7.3 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 946 KiB/s wr, 811 op/s; 51928/360783 objects misplaced (14.393%); 5.1 MiB/s, 80 objects/s recovering 2024-09-13T14:48:00.946 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:00 smithi102 ceph-mon[110183]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:48:00.946 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:00 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:48:00.946 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:00 smithi102 ceph-mon[110183]: osdmap e341: 8 total, 8 up, 8 in 2024-09-13T14:48:00.947 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:00 smithi102 ceph-mon[115269]: pgmap v1170: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 7.3 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 946 KiB/s wr, 811 op/s; 51928/360783 objects misplaced (14.393%); 5.1 MiB/s, 80 objects/s recovering 2024-09-13T14:48:00.947 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:00 smithi102 ceph-mon[115269]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:48:00.947 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:00 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:48:00.947 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:00 smithi102 ceph-mon[115269]: osdmap e341: 8 total, 8 up, 8 in 2024-09-13T14:48:00.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:00 smithi195 ceph-mon[104262]: pgmap v1170: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 7.3 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 946 KiB/s wr, 811 op/s; 51928/360783 objects misplaced (14.393%); 5.1 MiB/s, 80 objects/s recovering 2024-09-13T14:48:00.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:00 smithi195 ceph-mon[104262]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:48:00.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:00 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:48:00.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:00 smithi195 ceph-mon[104262]: osdmap e341: 8 total, 8 up, 8 in 2024-09-13T14:48:01.028 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:01.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:01.072 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:01.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:01.172 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:01.211 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:01.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:01.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:01.313 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:01.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:01.354 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:01.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:01.455 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:01.487 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:01.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:01.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:01.589 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:01.617 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:01.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:01.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:01.719 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:01.746 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:01.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:01.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:01.848 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:01.875 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:01.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:01.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:01.977 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:02.008 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:02.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:02.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:02.110 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:02.181 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:02.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:02.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:02.283 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:02.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:02.319 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:02.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:02.420 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:02.463 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:02.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:02.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:02.566 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:02.606 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:02.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:02.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:02.708 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:02.737 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:02.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:02.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:02.781 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:02 smithi102 ceph-mon[110183]: pgmap v1172: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 7.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 614 KiB/s wr, 866 op/s; 50709/353295 objects misplaced (14.353%); 6.9 MiB/s, 113 objects/s recovering 2024-09-13T14:48:02.782 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:02 smithi102 ceph-mon[115269]: pgmap v1172: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 7.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 614 KiB/s wr, 866 op/s; 50709/353295 objects misplaced (14.353%); 6.9 MiB/s, 113 objects/s recovering 2024-09-13T14:48:02.839 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:02.848 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:48:02.849 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:48:02.865 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:02.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:02.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:02.910 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 6, 7, 5, 2, 1] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T14:48:02.910 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:48:02.910 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-09-13T14:48:02.910 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T14:48:02.910 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T14:48:02.911 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-09-13T14:48:02.967 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:02.988 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:02.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:02.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:02.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:02 smithi195 ceph-mon[104262]: pgmap v1172: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 7.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 614 KiB/s wr, 866 op/s; 50709/353295 objects misplaced (14.353%); 6.9 MiB/s, 113 objects/s recovering 2024-09-13T14:48:03.090 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:03.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:03.147 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:03.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:03.223 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:48:03.249 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:03.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:03.285 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:03.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:03.386 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:03.425 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:03.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:03.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:03.498 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:48:03 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=infra.usagestats t=2024-09-13T14:48:03.22123294Z level=info msg="Usage stats are ready to report" 2024-09-13T14:48:03.527 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:03.565 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:03.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:03.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:03.667 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:03.706 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:03.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:03.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:03.808 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:03.841 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:03.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:03.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:03.874 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:48:03.943 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:03.975 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:03.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:03.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:04.077 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:04.105 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:04.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:04.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:04.207 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:04.232 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:04.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:04.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:04.338 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:04.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:04.379 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:04.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:04.480 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:04.516 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:04.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:04.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:04.618 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:04.668 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:04.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:04.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:04.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:04 smithi195 ceph-mon[104262]: pgmap v1173: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 7.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 900 op/s; 49618/347523 objects misplaced (14.278%); 5.6 MiB/s, 92 objects/s recovering 2024-09-13T14:48:04.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:04 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3554807199' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:04.770 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:04.798 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:04.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:04.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:04.802 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:04 smithi102 ceph-mon[110183]: pgmap v1173: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 7.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 900 op/s; 49618/347523 objects misplaced (14.278%); 5.6 MiB/s, 92 objects/s recovering 2024-09-13T14:48:04.802 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:04 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3554807199' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:04.803 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:04 smithi102 ceph-mon[115269]: pgmap v1173: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 7.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 900 op/s; 49618/347523 objects misplaced (14.278%); 5.6 MiB/s, 92 objects/s recovering 2024-09-13T14:48:04.803 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:04 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3554807199' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:04.900 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:04.936 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:04.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:04.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:05.037 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:05.073 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:05.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:05.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:05.153 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T14:48:05.176 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:05.194 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:48:05.194 INFO:teuthology.orchestra.run.smithi195.stdout: "success": "" 2024-09-13T14:48:05.194 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:48:05.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:05.246 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:05.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:05.276 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:48:05.276 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:48:05.276 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:48:05.276 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:48:05.276 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:48:05.276 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:48:05.276 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:48:05.277 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:48:05.277 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:48:05.277 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:48:05.277 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:48:05.277 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:48:05.277 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:48:05.277 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:48:05.277 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:48:05.277 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:48:05.347 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:05.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:05.418 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:05.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:05.519 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:05.547 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:05.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:05.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:05.649 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:05.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:05.683 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:05.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:05.784 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:05.814 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:05.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:05.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:05.916 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:05.958 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:05.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:05.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:06.060 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:06.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:06.087 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:06.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:06.188 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:06.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:06.241 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:06.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:06.343 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:06.371 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:06.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:06.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:06.442 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:48:06 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:48:06] "GET /metrics HTTP/1.1" 200 36541 "" "Prometheus/2.51.0" 2024-09-13T14:48:06.472 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:06.505 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:06.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:06.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:06.607 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:06.658 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:06.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:06.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:06.695 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:06.700 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:06.760 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:06.805 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:06.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:06.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:06.906 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:06.933 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:06.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:06.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:06.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:06 smithi195 ceph-mon[104262]: pgmap v1174: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 7.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 900 op/s; 49618/347523 objects misplaced (14.278%); 5.6 MiB/s, 89 objects/s recovering 2024-09-13T14:48:06.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:06 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:48:07.035 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:07.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:07.062 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:07.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:07.079 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:06 smithi102 ceph-mon[110183]: pgmap v1174: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 7.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 900 op/s; 49618/347523 objects misplaced (14.278%); 5.6 MiB/s, 89 objects/s recovering 2024-09-13T14:48:07.080 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:06 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:48:07.080 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:06 smithi102 ceph-mon[115269]: pgmap v1174: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 7.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 900 op/s; 49618/347523 objects misplaced (14.278%); 5.6 MiB/s, 89 objects/s recovering 2024-09-13T14:48:07.080 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:06 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:48:07.164 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:07.195 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:07.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:07.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:07.297 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:07.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:07.358 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:07.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:07.459 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:07.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:07.487 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:07.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:07.588 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:07.619 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:07.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:07.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:07.721 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:07.767 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:07.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:07.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:07.869 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:07.901 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:07.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:07.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:07.932 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:48:07.979 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:07 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:48:07.980 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:07 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:48:07.980 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:07 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:48:07.980 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:07 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:48:07.980 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:07 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:48:07.981 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:07 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:48:07.981 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:07 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:48:07.981 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:07 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:48:07.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:07 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:48:07.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:07 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:48:07.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:07 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:48:07.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:07 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:48:08.003 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:08.039 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:08.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:08.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:08.141 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:08.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:08.200 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:08.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:08.301 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:08.338 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:08.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:08.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:08.440 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:08.474 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:08.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:08.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:08.576 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:08.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:08.617 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:08.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:08.718 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:08.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:08.761 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:08.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:08.862 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:08.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:08.909 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:08.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:08.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:08 smithi102 ceph-mon[110183]: pgmap v1175: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 6.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 881 op/s; 48172/336876 objects misplaced (14.300%); 5.2 MiB/s, 83 objects/s recovering 2024-09-13T14:48:08.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:08 smithi102 ceph-mon[115269]: pgmap v1175: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 6.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 881 op/s; 48172/336876 objects misplaced (14.300%); 5.2 MiB/s, 83 objects/s recovering 2024-09-13T14:48:08.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:08 smithi195 ceph-mon[104262]: pgmap v1175: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 6.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 881 op/s; 48172/336876 objects misplaced (14.300%); 5.2 MiB/s, 83 objects/s recovering 2024-09-13T14:48:09.010 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:09.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:09.041 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:09.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:09.142 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:09.175 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:09.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:09.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:09.277 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:09.310 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:09.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:09.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:09.412 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:09.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:09.451 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:09.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:09.553 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:09.581 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:09.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:09.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:09.683 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:09.720 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:09.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:09.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:09.822 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:09.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:09.854 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:09.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:09.955 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:09.992 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:09.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:09.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:10.094 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:10.129 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:10.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:10.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:10.231 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:10.266 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:10.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:10.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:10.368 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:10.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:10.415 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:10.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:10.516 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:10.548 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:10.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:10.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:10.650 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:10.690 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:10.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:10.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:10.792 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:10.838 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:10.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:10.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:10.940 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:10.979 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:10.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:10.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:11.000 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:10 smithi195 ceph-mon[104262]: pgmap v1176: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 6.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 873 op/s; 47405/334575 objects misplaced (14.169%); 5.2 MiB/s, 83 objects/s recovering 2024-09-13T14:48:11.001 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:48:11.001 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:48:11.081 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:11.110 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:11.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:11.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:11.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:10 smithi102 ceph-mon[110183]: pgmap v1176: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 6.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 873 op/s; 47405/334575 objects misplaced (14.169%); 5.2 MiB/s, 83 objects/s recovering 2024-09-13T14:48:11.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:48:11.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:48:11.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:10 smithi102 ceph-mon[115269]: pgmap v1176: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 6.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 873 op/s; 47405/334575 objects misplaced (14.169%); 5.2 MiB/s, 83 objects/s recovering 2024-09-13T14:48:11.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:48:11.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:48:11.201 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 6, 7, 5, 2, 1] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T14:48:11.202 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-13T14:48:11.212 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:11.289 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:11.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:11.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:11.391 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:11.419 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:11.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:11.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:11.522 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:11.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:11.566 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:11.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:11.667 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:11.706 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:11.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:11.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:11.807 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:11.856 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:11.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:11.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:11.958 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:11.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:11.994 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:11.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:12.095 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:12.134 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:12.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:12.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:12.236 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:12.270 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:12.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:12.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:12.372 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:12.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:12.405 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:12.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:12.506 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:12.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:12.543 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:12.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:12.644 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:12.678 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:12.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:12.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:12.780 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:12.810 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:12.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:12.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:12.911 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:12.944 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:12.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:12.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:12.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:12 smithi195 ceph-mon[104262]: pgmap v1177: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 6.7 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 933 op/s; 46269/327333 objects misplaced (14.135%); 6.2 MiB/s, 99 objects/s recovering 2024-09-13T14:48:13.047 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:13.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:13.082 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:13.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:13.154 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:12 smithi102 ceph-mon[110183]: pgmap v1177: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 6.7 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 933 op/s; 46269/327333 objects misplaced (14.135%); 6.2 MiB/s, 99 objects/s recovering 2024-09-13T14:48:13.155 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:12 smithi102 ceph-mon[115269]: pgmap v1177: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 6.7 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 933 op/s; 46269/327333 objects misplaced (14.135%); 6.2 MiB/s, 99 objects/s recovering 2024-09-13T14:48:13.182 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:13.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:13.225 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:13.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:13.326 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:13.347 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:13.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:13.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:13.448 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:13.484 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:13.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:13.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:13.586 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:13.621 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:13.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:13.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:13.722 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:13.748 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:13.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:13.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:13.850 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:13.883 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:13.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:13.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:13.985 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:14.006 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:14.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:14.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:14.107 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:14.147 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:14.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:14.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:14.249 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:14.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:14.285 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:14.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:14.386 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:14.426 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:14.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:14.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:14.528 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:14.556 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:14.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:14.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:14.657 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:14.692 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:14.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:14.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:14.794 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:14.821 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:14.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:14.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:14.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:14 smithi102 ceph-mon[110183]: pgmap v1178: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 6.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 875 op/s; 45200/321759 objects misplaced (14.048%); 4.1 MiB/s, 64 objects/s recovering 2024-09-13T14:48:14.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:14 smithi102 ceph-mon[115269]: pgmap v1178: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 6.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 875 op/s; 45200/321759 objects misplaced (14.048%); 4.1 MiB/s, 64 objects/s recovering 2024-09-13T14:48:14.923 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:14.955 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:14.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:14.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:14.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:14 smithi195 ceph-mon[104262]: pgmap v1178: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 6.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 875 op/s; 45200/321759 objects misplaced (14.048%); 4.1 MiB/s, 64 objects/s recovering 2024-09-13T14:48:15.057 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:15.080 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:15.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:15.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:15.182 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:15.225 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:15.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:15.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:15.327 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:15.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:15.363 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:15.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:15.464 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:15.506 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:15.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:15.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:15.608 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:15.634 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:15.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:15.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:15.736 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:15.771 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:15.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:15.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:15.873 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:15.907 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:15.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:15.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:16.009 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:16.041 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:16.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:16.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:16.143 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:16.174 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:16.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:16.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:16.276 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:16.323 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:16.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:16.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:16.426 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:16.467 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:16.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:16.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:16.569 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:16.610 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:16.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:16.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:16.712 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:16.741 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:48:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:48:16] "GET /metrics HTTP/1.1" 200 36534 "" "Prometheus/2.51.0" 2024-09-13T14:48:16.748 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:16.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:16.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:16.850 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:16.885 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:16.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:16.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:16.987 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:17.003 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:16 smithi195 ceph-mon[104262]: pgmap v1179: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 6.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 715 op/s; 45200/321759 objects misplaced (14.048%); 4.1 MiB/s, 64 objects/s recovering 2024-09-13T14:48:17.048 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:17.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:17.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:17.149 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:17.151 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:16 smithi102 ceph-mon[110183]: pgmap v1179: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 6.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 715 op/s; 45200/321759 objects misplaced (14.048%); 4.1 MiB/s, 64 objects/s recovering 2024-09-13T14:48:17.151 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:16 smithi102 ceph-mon[115269]: pgmap v1179: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 6.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 715 op/s; 45200/321759 objects misplaced (14.048%); 4.1 MiB/s, 64 objects/s recovering 2024-09-13T14:48:17.176 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:17.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:17.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:17.278 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:17.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:17.335 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:17.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:17.436 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:17.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:17.488 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:17.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:17.589 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:17.635 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:17.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:17.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:17.736 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:17.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:17.783 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:17.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:17.884 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:17.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:17.926 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:17.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:18.027 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:18.066 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:18.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:18.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:18.168 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:18.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:18.198 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:18.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:18.300 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:18.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:18.322 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:18.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:18.423 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:18.461 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:18.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:18.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:18.563 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:18.611 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:18.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:18.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:18.713 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:18.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:18.742 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:18.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:18.765 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is set 2024-09-13T14:48:18.843 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:18.854 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:18 smithi102 ceph-mon[110183]: pgmap v1180: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 997 op/s; 43828/311592 objects misplaced (14.066%); 6.0 MiB/s, 96 objects/s recovering 2024-09-13T14:48:18.854 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:18 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3022462093' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:48:18.854 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:18 smithi102 ceph-mon[115269]: pgmap v1180: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 997 op/s; 43828/311592 objects misplaced (14.066%); 6.0 MiB/s, 96 objects/s recovering 2024-09-13T14:48:18.854 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:18 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3022462093' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:48:18.867 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:18.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:18.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:18.919 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:48:18.919 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [], 2024-09-13T14:48:18.919 INFO:teuthology.orchestra.run.smithi102.stdout: "complaint_time": 30, 2024-09-13T14:48:18.919 INFO:teuthology.orchestra.run.smithi102.stdout: "num_blocked_ops": 0 2024-09-13T14:48:18.920 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:48:18.969 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:18.994 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:18.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:18.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:18.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:18 smithi195 ceph-mon[104262]: pgmap v1180: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 997 op/s; 43828/311592 objects misplaced (14.066%); 6.0 MiB/s, 96 objects/s recovering 2024-09-13T14:48:18.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:18 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3022462093' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:48:19.096 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:19.130 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:19.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:19.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:19.232 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:19.261 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:19.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:19.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:19.362 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:19.407 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:19.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:19.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:19.509 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:19.545 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:19.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:19.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:19.647 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:19.676 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:19.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:19.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:19.781 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:19.817 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:19.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:19.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:19.915 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:19.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:19 smithi102 ceph-mon[110183]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:48:19.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:19 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3022462093' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:48:19.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:19 smithi102 ceph-mon[110183]: osdmap e342: 8 total, 8 up, 8 in 2024-09-13T14:48:19.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:19 smithi102 ceph-mon[115269]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:48:19.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:19 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3022462093' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:48:19.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:19 smithi102 ceph-mon[115269]: osdmap e342: 8 total, 8 up, 8 in 2024-09-13T14:48:19.919 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:19.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:19.976 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:19.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:20.077 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:20.105 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:20.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:20.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:20.206 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:20.235 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:20.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:20.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:20.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:19 smithi195 ceph-mon[104262]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:48:20.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:19 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3022462093' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:48:20.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:19 smithi195 ceph-mon[104262]: osdmap e342: 8 total, 8 up, 8 in 2024-09-13T14:48:20.337 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:20.381 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:20.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:20.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:20.415 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T14:48:20.483 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:20.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:20.534 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:20.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:20.635 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:20.660 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:20.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:20.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:20.762 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:20.793 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:20.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:20.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:20.895 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:20.938 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:20.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:20.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:21.040 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:21.083 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:21.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:21.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:21.155 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:21.184 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:21.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:21.218 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:21.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:21.295 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:21 smithi102 ceph-mon[110183]: pgmap v1182: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 847 op/s; 43015/309153 objects misplaced (13.914%); 4.6 MiB/s, 73 objects/s recovering 2024-09-13T14:48:21.296 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:21 smithi102 ceph-mon[115269]: pgmap v1182: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 847 op/s; 43015/309153 objects misplaced (13.914%); 4.6 MiB/s, 73 objects/s recovering 2024-09-13T14:48:21.319 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:21.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:21.361 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:21.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:21.462 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:21.495 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:21.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:21.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:21.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:21 smithi195 ceph-mon[104262]: pgmap v1182: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 847 op/s; 43015/309153 objects misplaced (13.914%); 4.6 MiB/s, 73 objects/s recovering 2024-09-13T14:48:21.597 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:21.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:21.639 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:21.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:21.741 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:21.775 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:21.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:21.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:21.877 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:21.917 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:21.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:21.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:21.939 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:48:22.019 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:22.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:22.081 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:22.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:22.182 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:22.232 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:22.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:22.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:22.335 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:22.378 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:22.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:22.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:22.480 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:22.508 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:22.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:22.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:22.610 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:22.638 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:22.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:22.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:22.740 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:22.786 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:22.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:22.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:22.888 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:22.931 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:22.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:22.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:23.030 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:23.034 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:23.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:23.106 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:23.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:23.207 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:23.240 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:23.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:23.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:23.269 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:23 smithi102 ceph-mon[110183]: pgmap v1183: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 6.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 838 op/s; 41927/302169 objects misplaced (13.875%); 4.8 MiB/s, 76 objects/s recovering 2024-09-13T14:48:23.269 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:23 smithi102 ceph-mon[115269]: pgmap v1183: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 6.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 838 op/s; 41927/302169 objects misplaced (13.875%); 4.8 MiB/s, 76 objects/s recovering 2024-09-13T14:48:23.316 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:23 smithi195 ceph-mon[104262]: pgmap v1183: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 6.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 838 op/s; 41927/302169 objects misplaced (13.875%); 4.8 MiB/s, 76 objects/s recovering 2024-09-13T14:48:23.341 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:23.377 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:23.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:23.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:23.453 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:23.479 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:23.536 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:23.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:23.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:23.638 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:23.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:23.680 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:23.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:23.781 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:23.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:23.821 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:23.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:23.923 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:23.953 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:23.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:23.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:24.054 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:24.094 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:24.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:24.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:24.196 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:24.228 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:24.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:24.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:24.330 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:24.368 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:24.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:24.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:24.470 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:24.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:24.505 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:24.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:24.606 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:24.639 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:24.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:24.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:24.741 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:24.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:24.782 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:24.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:24.883 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:24.919 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:24.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:24.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:25.021 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:25.056 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:25.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:25.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:25.158 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:25.186 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:25.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:25.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:25.288 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:25.307 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:25 smithi102 ceph-mon[115269]: pgmap v1184: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 833 op/s; 40920/296727 objects misplaced (13.790%); 4.8 MiB/s, 76 objects/s recovering 2024-09-13T14:48:25.308 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:25 smithi102 ceph-mon[110183]: pgmap v1184: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 833 op/s; 40920/296727 objects misplaced (13.790%); 4.8 MiB/s, 76 objects/s recovering 2024-09-13T14:48:25.317 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:25.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:25.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:25.419 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:25.464 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:25.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:25.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:25.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:25 smithi195 ceph-mon[104262]: pgmap v1184: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 833 op/s; 40920/296727 objects misplaced (13.790%); 4.8 MiB/s, 76 objects/s recovering 2024-09-13T14:48:25.566 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:25.603 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:25.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:25.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:25.705 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:25.754 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:25.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:25.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:25.857 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:25.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:25.903 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:25.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:26.004 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:26.045 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:26.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:26.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:26.147 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:26.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:48:26.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:26 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1509754309' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:26.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:48:26.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:26 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1509754309' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:26.188 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:26.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:26.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:26.290 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:26.345 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:26.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:26.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:26.447 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:26.473 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:26.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:26.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:26.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:48:26.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:26 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1509754309' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:26.575 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:26.602 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:26.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:26.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:26.704 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:26.748 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:26.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:26.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:26.853 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:26.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:26.890 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:26.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:26.910 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:48:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:48:26] "GET /metrics HTTP/1.1" 200 36533 "" "Prometheus/2.51.0" 2024-09-13T14:48:26.991 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:27.047 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:27.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:27.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:27.050 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:48:27.051 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:48:27.051 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:48:27.051 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:48:27.051 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:48:27.051 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:48:27.051 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:48:27.051 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:48:27.051 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:48:27.051 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:48:27.051 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:48:27.051 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:48:27.052 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:48:27.052 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:48:27.052 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:48:27.052 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:48:27.075 INFO:teuthology.orchestra.run.smithi102.stderr:reweighted osd.0 to 1 (10000) 2024-09-13T14:48:27.150 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:27.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:27 smithi102 ceph-mon[110183]: pgmap v1185: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 833 op/s; 40920/296727 objects misplaced (13.790%); 4.8 MiB/s, 76 objects/s recovering 2024-09-13T14:48:27.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:27 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/4171809222' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:48:27.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:27 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:48:27.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:27 smithi102 ceph-mon[115269]: pgmap v1185: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 833 op/s; 40920/296727 objects misplaced (13.790%); 4.8 MiB/s, 76 objects/s recovering 2024-09-13T14:48:27.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:27 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/4171809222' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:48:27.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:27 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:48:27.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:27.192 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:27.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:27.293 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:27.317 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:27.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:27.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:27.419 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:27.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:27.467 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:27.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:27.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:27 smithi195 ceph-mon[104262]: pgmap v1185: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 833 op/s; 40920/296727 objects misplaced (13.790%); 4.8 MiB/s, 76 objects/s recovering 2024-09-13T14:48:27.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:27 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/4171809222' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:48:27.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:27 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:48:27.569 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:27.596 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:27.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:27.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:27.698 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:27.728 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:27.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:27.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:27.830 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:27.859 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:27.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:27.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:27.961 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:28.005 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:28.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:28.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:28.107 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:28.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:28.149 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:28.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:28.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:28 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T14:48:28.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:28 smithi102 ceph-mon[110183]: osdmap e343: 8 total, 8 up, 8 in 2024-09-13T14:48:28.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:28 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T14:48:28.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:28 smithi102 ceph-mon[115269]: osdmap e343: 8 total, 8 up, 8 in 2024-09-13T14:48:28.251 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:28.283 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:28.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:28.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:28.386 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:28.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:28.421 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:28.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:28.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:28 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T14:48:28.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:28 smithi195 ceph-mon[104262]: osdmap e343: 8 total, 8 up, 8 in 2024-09-13T14:48:28.523 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:28.548 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:28.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:28.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:28.649 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:28.674 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:28.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:28.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:28.776 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:28.819 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:28.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:28.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:28.922 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:28.952 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:28.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:28.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:29.054 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:29.091 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:29.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:29.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:29.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:29 smithi102 ceph-mon[110183]: pgmap v1187: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 727 op/s; 39890/289824 objects misplaced (13.764%); 5.3 MiB/s, 85 objects/s recovering 2024-09-13T14:48:29.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:29 smithi102 ceph-mon[115269]: pgmap v1187: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 727 op/s; 39890/289824 objects misplaced (13.764%); 5.3 MiB/s, 85 objects/s recovering 2024-09-13T14:48:29.193 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:29.231 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:29.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:29.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:29.333 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:29.364 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:29.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:29.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:29.466 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:29.480 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-13T14:48:29.488 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:29.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:29.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:29.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:29 smithi195 ceph-mon[104262]: pgmap v1187: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 727 op/s; 39890/289824 objects misplaced (13.764%); 5.3 MiB/s, 85 objects/s recovering 2024-09-13T14:48:29.591 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:29.648 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:29.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:29.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:29.750 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:29.777 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:29.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:29.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:29.879 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:29.921 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:29.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:29.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:30.024 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:30.054 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:30.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:30.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:30.156 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:30.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:30.188 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:30.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:30.289 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:30.326 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:30.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:30.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:30.428 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:30.455 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:30.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:30.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:30.557 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:30.582 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:30.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:30.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:30.684 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:30.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:30.726 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:30.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:30.827 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:30.849 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:30.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:30.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:30.951 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:30.990 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:30.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:30.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:31.039 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:48:31.039 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:48:31.092 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:31.127 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:31.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:31.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:31.229 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:31.264 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:31.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:31.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:31.366 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:31.387 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:31.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:31.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:31.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:31 smithi102 ceph-mon[110183]: pgmap v1188: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 820 op/s; 38863/284511 objects misplaced (13.660%); 4.7 MiB/s, 75 objects/s recovering 2024-09-13T14:48:31.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:31 smithi102 ceph-mon[115269]: pgmap v1188: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 820 op/s; 38863/284511 objects misplaced (13.660%); 4.7 MiB/s, 75 objects/s recovering 2024-09-13T14:48:31.489 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:31.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:31 smithi195 ceph-mon[104262]: pgmap v1188: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 820 op/s; 38863/284511 objects misplaced (13.660%); 4.7 MiB/s, 75 objects/s recovering 2024-09-13T14:48:31.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:31.524 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:31.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:31.625 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:31.647 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:31.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:31.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:31.748 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:31.776 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:31.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:31.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:31.878 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:31.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:31.921 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:31.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:32.005 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:48:32.022 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:32.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:32.050 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:32.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:32.151 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:32.172 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:32.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:32.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:32.274 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:32.301 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:32.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:32.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:32.403 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:32.437 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:32.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:32.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:32.539 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:32.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:32.575 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:32.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:32.676 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:32.704 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:32.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:32.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:32.806 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:32.842 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:32.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:32.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:32.944 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:32.976 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:32.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:32.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:33.078 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:33.101 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:33 smithi102 ceph-mon[110183]: pgmap v1189: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 5.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 814 op/s; 37779/277719 objects misplaced (13.603%); 4.9 MiB/s, 77 objects/s recovering 2024-09-13T14:48:33.119 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:33.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:33.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:33.221 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:33.263 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T14:48:33.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:33.267 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:33.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:33.369 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:33.381 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:33 smithi102 ceph-mon[115269]: pgmap v1189: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 5.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 814 op/s; 37779/277719 objects misplaced (13.603%); 4.9 MiB/s, 77 objects/s recovering 2024-09-13T14:48:33.413 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:33 smithi195 ceph-mon[104262]: pgmap v1189: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 5.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 814 op/s; 37779/277719 objects misplaced (13.603%); 4.9 MiB/s, 77 objects/s recovering 2024-09-13T14:48:33.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:33.431 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:33.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:33.532 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:33.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:33.605 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:33.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:33.706 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:33.750 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:33.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:33.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:33.852 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:33.908 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:33.924 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:33.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:33.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:33.947 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:33.971 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:34.026 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:34.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:34.072 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:34.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:34.173 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:34.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:34.220 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:34.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:34.321 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:34.355 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:34.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:34.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:34.457 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:34.495 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:34.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:34.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:34.597 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:34.626 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:34.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:34.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:34.728 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:34.769 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:34.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:34.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:34.871 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:34.908 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:34.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:34.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:35.009 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:35.038 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:35.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:35.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:35.117 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is unset 2024-09-13T14:48:35.140 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:35.167 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:35.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:35.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:35.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:35 smithi102 ceph-mon[110183]: pgmap v1190: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 5.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 803 op/s; 36816/272604 objects misplaced (13.505%); 4.9 MiB/s, 77 objects/s recovering 2024-09-13T14:48:35.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:35 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/31723683' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:48:35.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:35 smithi102 ceph-mon[115269]: pgmap v1190: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 5.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 803 op/s; 36816/272604 objects misplaced (13.505%); 4.9 MiB/s, 77 objects/s recovering 2024-09-13T14:48:35.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:35 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/31723683' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:48:35.269 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:35.300 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:35.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:35.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:35.402 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:35.435 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:35.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:35.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:35.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:35 smithi195 ceph-mon[104262]: pgmap v1190: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 5.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 803 op/s; 36816/272604 objects misplaced (13.505%); 4.9 MiB/s, 77 objects/s recovering 2024-09-13T14:48:35.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:35 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/31723683' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:48:35.538 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:35.574 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:35.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:35.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:35.675 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:35.705 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:35.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:35.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:35.806 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:35.842 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:35.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:35.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:35.944 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:35.990 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:35.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:35.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:36.092 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:36.132 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:36.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:36.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:36.234 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:36.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:36.273 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:36.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:36.359 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:36 smithi102 ceph-mon[115269]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:48:36.359 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:36 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/31723683' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:48:36.359 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:36 smithi102 ceph-mon[115269]: osdmap e344: 8 total, 8 up, 8 in 2024-09-13T14:48:36.359 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:36 smithi102 ceph-mon[110183]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:48:36.359 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:36 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/31723683' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:48:36.360 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:36 smithi102 ceph-mon[110183]: osdmap e344: 8 total, 8 up, 8 in 2024-09-13T14:48:36.374 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:36.411 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:36.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:36.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:36.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:36 smithi195 ceph-mon[104262]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:48:36.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:36 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/31723683' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:48:36.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:36 smithi195 ceph-mon[104262]: osdmap e344: 8 total, 8 up, 8 in 2024-09-13T14:48:36.513 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:36.553 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:36.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:36.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:36.654 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:48:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:48:36] "GET /metrics HTTP/1.1" 200 36533 "" "Prometheus/2.51.0" 2024-09-13T14:48:36.656 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:36.690 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:36.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:36.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:36.792 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:36.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:36.840 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:36.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:36.941 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:36.976 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:36.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:36.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:37.077 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:37.112 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:37.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:37.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:37.214 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:37.260 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:37.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:37.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:37.326 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:37 smithi102 ceph-mon[110183]: pgmap v1192: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 5.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 670 op/s; 36816/272604 objects misplaced (13.505%); 3.0 MiB/s, 47 objects/s recovering 2024-09-13T14:48:37.326 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:37 smithi102 ceph-mon[115269]: pgmap v1192: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 5.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 670 op/s; 36816/272604 objects misplaced (13.505%); 3.0 MiB/s, 47 objects/s recovering 2024-09-13T14:48:37.362 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:37.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:37.398 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:37.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:37.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:37 smithi195 ceph-mon[104262]: pgmap v1192: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 5.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 670 op/s; 36816/272604 objects misplaced (13.505%); 3.0 MiB/s, 47 objects/s recovering 2024-09-13T14:48:37.499 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:37.528 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:37.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:37.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:37.630 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:37.658 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:37.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:37.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:37.760 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:37.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:37.809 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:37.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:37.910 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:37.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:37.940 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:37.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:38.041 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:38.070 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:48:38.070 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:48:38.079 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:38.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:38.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:38.115 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:48:38.115 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:48:38.156 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:48:38.157 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:48:38.181 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:38.196 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:48:38.196 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:48:38.224 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:38.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:38.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:38.227 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:48:38.227 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:48:38.258 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:48:38.258 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:48:38.289 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:48:38.290 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:48:38.324 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:48:38.325 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:48:38.325 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:38 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3151164234' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:38.325 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:38 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3151164234' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:38.326 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:38.356 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:38.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:38.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:38.458 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:38.480 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:38.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:38.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:38.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:38 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3151164234' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:38.582 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:38.614 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:38.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:38.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:38.715 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:38.750 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:38.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:38.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:38.852 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:38.880 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:38.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:38.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:38.981 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:39.008 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:39.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:39.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:39.109 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:39.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:39.147 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:39.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:39.248 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:39.302 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:39 smithi102 ceph-mon[110183]: pgmap v1193: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 799 op/s; 35842/265824 objects misplaced (13.483%); 4.8 MiB/s, 76 objects/s recovering 2024-09-13T14:48:39.303 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:39 smithi102 ceph-mon[115269]: pgmap v1193: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 799 op/s; 35842/265824 objects misplaced (13.483%); 4.8 MiB/s, 76 objects/s recovering 2024-09-13T14:48:39.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:39.307 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:39.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:39.408 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:39.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:39.448 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:39.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:39.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:39 smithi195 ceph-mon[104262]: pgmap v1193: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 799 op/s; 35842/265824 objects misplaced (13.483%); 4.8 MiB/s, 76 objects/s recovering 2024-09-13T14:48:39.549 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:39.580 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:39.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:39.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:39.682 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:39.716 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:39.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:39.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:39.776 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:48:39.818 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:39.859 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:39.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:39.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:39.961 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:40.007 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:40.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:40.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:40.109 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:40.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:40.135 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:40.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:40.236 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:40.270 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:40.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:40.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:40.372 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:40.399 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:40.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:40.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:40.500 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:40.552 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:40.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:40.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:40.621 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:48:40.621 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [ 2024-09-13T14:48:40.621 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.621 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.46157.0:177405 8.b e344/337 8:d62f99a8:::benchmark_data_smithi102_313275_object46818:head v 344'11092, mlcod=344'11090)", 2024-09-13T14:48:40.621 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:48:40.561839+0000", 2024-09-13T14:48:40.621 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.024724597000000001, 2024-09-13T14:48:40.621 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.024747595000000001, 2024-09-13T14:48:40.621 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:48:40.622 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:48:40.622 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "started", 2024-09-13T14:48:40.622 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:48:40.622 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.622 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:48:40.622 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.561839+0000", 2024-09-13T14:48:40.622 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:48:40.622 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.622 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.622 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:48:40.622 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.561839+0000", 2024-09-13T14:48:40.622 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:48:40.623 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.623 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.623 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:48:40.623 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.561840+0000", 2024-09-13T14:48:40.623 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0899999999999999e-06 2024-09-13T14:48:40.623 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.623 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.623 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:48:40.623 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.561847+0000", 2024-09-13T14:48:40.623 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.0389999999999999e-06 2024-09-13T14:48:40.623 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.623 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.623 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:48:40.624 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.561875+0000", 2024-09-13T14:48:40.624 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.8062999999999999e-05 2024-09-13T14:48:40.624 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.624 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.624 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:48:40.624 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.561883+0000", 2024-09-13T14:48:40.624 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.9349999999999994e-06 2024-09-13T14:48:40.624 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.624 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.624 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:48:40.624 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.561976+0000", 2024-09-13T14:48:40.624 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.3527000000000001e-05 2024-09-13T14:48:40.625 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.625 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.625 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:48:40.625 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.561999+0000", 2024-09-13T14:48:40.625 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.2892e-05 2024-09-13T14:48:40.625 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:48:40.625 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:48:40.625 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:48:40.625 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.625 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.625 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.46157.0:177407 8.b e344/337 8:d00bdf27:::benchmark_data_smithi102_313275_object46820:head v 344'11093, mlcod=344'11090)", 2024-09-13T14:48:40.625 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:48:40.566667+0000", 2024-09-13T14:48:40.625 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.019895749000000001, 2024-09-13T14:48:40.626 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.019983088999999999, 2024-09-13T14:48:40.626 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:48:40.626 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:48:40.626 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "started", 2024-09-13T14:48:40.626 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:48:40.626 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.626 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:48:40.626 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.566667+0000", 2024-09-13T14:48:40.626 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:48:40.626 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.626 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.626 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:48:40.626 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.566667+0000", 2024-09-13T14:48:40.627 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:48:40.627 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.627 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.627 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:48:40.627 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.566668+0000", 2024-09-13T14:48:40.627 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.2399999999999997e-07 2024-09-13T14:48:40.627 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.627 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.627 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:48:40.627 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.566674+0000", 2024-09-13T14:48:40.627 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.1109999999999996e-06 2024-09-13T14:48:40.627 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.627 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.627 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:48:40.628 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.566685+0000", 2024-09-13T14:48:40.628 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0519e-05 2024-09-13T14:48:40.628 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.628 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.628 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:48:40.628 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.566691+0000", 2024-09-13T14:48:40.628 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.4069999999999998e-06 2024-09-13T14:48:40.628 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.628 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.628 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:48:40.628 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.584712+0000", 2024-09-13T14:48:40.629 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.018020878000000001 2024-09-13T14:48:40.629 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.629 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.629 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:48:40.629 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.584732+0000", 2024-09-13T14:48:40.629 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.0363000000000001e-05 2024-09-13T14:48:40.629 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:48:40.629 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:48:40.629 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:48:40.629 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.629 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.629 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:177409 8.7 8:e2a57147:::benchmark_data_smithi102_313275_object46822:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e344)", 2024-09-13T14:48:40.629 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:48:40.568973+0000", 2024-09-13T14:48:40.629 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.017590246, 2024-09-13T14:48:40.630 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.017728126, 2024-09-13T14:48:40.630 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:48:40.630 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:48:40.630 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:48:40.630 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:48:40.630 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:48:40.630 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:48:40.630 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 177409 2024-09-13T14:48:40.630 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.630 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:48:40.630 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.630 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:48:40.630 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.568973+0000", 2024-09-13T14:48:40.631 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:48:40.631 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.631 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.631 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:48:40.631 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.568973+0000", 2024-09-13T14:48:40.631 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:48:40.631 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.631 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.631 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:48:40.631 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.568975+0000", 2024-09-13T14:48:40.631 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.9759999999999998e-06 2024-09-13T14:48:40.631 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.631 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.632 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:48:40.632 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.568980+0000", 2024-09-13T14:48:40.632 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.0579999999999997e-06 2024-09-13T14:48:40.632 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.632 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.632 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:48:40.632 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.568997+0000", 2024-09-13T14:48:40.632 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.6622000000000001e-05 2024-09-13T14:48:40.632 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.632 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.632 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:48:40.632 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.569005+0000", 2024-09-13T14:48:40.632 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.5010000000000004e-06 2024-09-13T14:48:40.633 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.633 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.633 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:48:40.633 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.569511+0000", 2024-09-13T14:48:40.633 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00050561900000000001 2024-09-13T14:48:40.633 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.633 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.633 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:48:40.633 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.569627+0000", 2024-09-13T14:48:40.633 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000116638 2024-09-13T14:48:40.633 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.633 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.633 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,3,6", 2024-09-13T14:48:40.633 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.569741+0000", 2024-09-13T14:48:40.634 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000113375 2024-09-13T14:48:40.634 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.634 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.634 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:40.634 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.582906+0000", 2024-09-13T14:48:40.634 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.01316515 2024-09-13T14:48:40.634 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.634 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.634 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:48:40.634 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.582933+0000", 2024-09-13T14:48:40.634 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.6894999999999999e-05 2024-09-13T14:48:40.634 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.634 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.634 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:40.635 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.583248+0000", 2024-09-13T14:48:40.635 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00031516399999999999 2024-09-13T14:48:40.635 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:48:40.635 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:48:40.635 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:48:40.635 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.635 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.635 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:177411 8.7 8:eb9a0334:::benchmark_data_smithi102_313275_object46824:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e344)", 2024-09-13T14:48:40.635 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:48:40.569221+0000", 2024-09-13T14:48:40.636 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.017342416999999999, 2024-09-13T14:48:40.636 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.017553782, 2024-09-13T14:48:40.636 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:48:40.636 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:48:40.636 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:48:40.636 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:48:40.636 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:48:40.636 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:48:40.636 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 177411 2024-09-13T14:48:40.636 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.636 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:48:40.637 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.637 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:48:40.637 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.569221+0000", 2024-09-13T14:48:40.637 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:48:40.637 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.637 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.637 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:48:40.637 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.569221+0000", 2024-09-13T14:48:40.637 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:48:40.637 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.637 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.637 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:48:40.637 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.569222+0000", 2024-09-13T14:48:40.638 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.8199999999999998e-07 2024-09-13T14:48:40.638 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.638 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.638 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:48:40.638 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.569226+0000", 2024-09-13T14:48:40.638 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.0740000000000003e-06 2024-09-13T14:48:40.638 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.638 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.638 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:48:40.638 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.569241+0000", 2024-09-13T14:48:40.638 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4938e-05 2024-09-13T14:48:40.638 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.638 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.638 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:48:40.638 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.569248+0000", 2024-09-13T14:48:40.639 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.8669999999999994e-06 2024-09-13T14:48:40.639 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.639 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.639 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:48:40.639 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.583277+0000", 2024-09-13T14:48:40.639 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.014028132 2024-09-13T14:48:40.639 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.639 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.639 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:48:40.639 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.583624+0000", 2024-09-13T14:48:40.639 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00034705199999999998 2024-09-13T14:48:40.639 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.640 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.640 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,3,6", 2024-09-13T14:48:40.640 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.583737+0000", 2024-09-13T14:48:40.640 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00011349900000000001 2024-09-13T14:48:40.640 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:48:40.640 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:48:40.640 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:48:40.640 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.640 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.640 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.46157.0:177394 8.f e344/337 8:fb54261d:::benchmark_data_smithi102_313275_object46807:head v 344'11106, mlcod=344'11101)", 2024-09-13T14:48:40.640 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:48:40.571949+0000", 2024-09-13T14:48:40.640 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.014614383, 2024-09-13T14:48:40.641 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.014886681000000001, 2024-09-13T14:48:40.641 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:48:40.641 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:48:40.641 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "started", 2024-09-13T14:48:40.641 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:48:40.641 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.641 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:48:40.641 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.571949+0000", 2024-09-13T14:48:40.641 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:48:40.641 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.641 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.642 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:48:40.642 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.571949+0000", 2024-09-13T14:48:40.642 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:48:40.642 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.642 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.642 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:48:40.642 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.571950+0000", 2024-09-13T14:48:40.642 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.6000000000000002e-07 2024-09-13T14:48:40.642 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.642 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.642 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:48:40.642 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.571956+0000", 2024-09-13T14:48:40.642 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.6170000000000003e-06 2024-09-13T14:48:40.642 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.643 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.643 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:48:40.643 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.571968+0000", 2024-09-13T14:48:40.643 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2072000000000001e-05 2024-09-13T14:48:40.643 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.643 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.643 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:48:40.643 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.571975+0000", 2024-09-13T14:48:40.643 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.0990000000000001e-06 2024-09-13T14:48:40.643 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.643 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.643 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:48:40.643 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.572894+0000", 2024-09-13T14:48:40.643 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00091864900000000001 2024-09-13T14:48:40.644 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.644 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.644 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:48:40.644 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.572915+0000", 2024-09-13T14:48:40.644 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.0724999999999999e-05 2024-09-13T14:48:40.644 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:48:40.644 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:48:40.644 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:48:40.644 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.644 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.644 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:177416 8.3 8:cc0cb1dc:::benchmark_data_smithi102_313275_object46829:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e344)", 2024-09-13T14:48:40.644 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:48:40.576766+0000", 2024-09-13T14:48:40.644 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.0097967079999999995, 2024-09-13T14:48:40.645 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.010124311, 2024-09-13T14:48:40.645 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:48:40.645 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:48:40.645 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:48:40.645 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:48:40.645 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:48:40.645 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:48:40.645 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 177416 2024-09-13T14:48:40.645 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.645 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:48:40.645 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.645 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:48:40.645 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.576766+0000", 2024-09-13T14:48:40.645 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:48:40.646 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.646 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.646 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:48:40.646 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.576766+0000", 2024-09-13T14:48:40.646 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:48:40.646 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.646 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.646 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:48:40.646 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.576768+0000", 2024-09-13T14:48:40.646 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.469e-06 2024-09-13T14:48:40.646 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.646 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.646 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:48:40.647 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.576772+0000", 2024-09-13T14:48:40.647 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.8469999999999998e-06 2024-09-13T14:48:40.647 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.647 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.647 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:48:40.647 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.576783+0000", 2024-09-13T14:48:40.647 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1027e-05 2024-09-13T14:48:40.647 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.647 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.647 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:48:40.647 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.576790+0000", 2024-09-13T14:48:40.647 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.8759999999999999e-06 2024-09-13T14:48:40.647 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.648 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.648 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:48:40.648 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.576898+0000", 2024-09-13T14:48:40.648 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00010836900000000001 2024-09-13T14:48:40.648 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.648 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.648 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:48:40.648 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.577008+0000", 2024-09-13T14:48:40.648 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00011013700000000001 2024-09-13T14:48:40.648 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.648 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.648 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,5,6", 2024-09-13T14:48:40.649 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.577125+0000", 2024-09-13T14:48:40.649 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000117291 2024-09-13T14:48:40.649 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:48:40.649 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:48:40.649 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:48:40.649 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.649 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.649 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:177418 8.3 8.ca1f623 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e344)", 2024-09-13T14:48:40.649 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:48:40.576928+0000", 2024-09-13T14:48:40.649 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.0096355450000000006, 2024-09-13T14:48:40.649 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.010021832, 2024-09-13T14:48:40.649 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:48:40.649 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:48:40.649 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "queued for pg", 2024-09-13T14:48:40.650 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:48:40.650 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:48:40.650 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:48:40.650 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 177418 2024-09-13T14:48:40.650 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.650 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:48:40.650 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.650 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:48:40.650 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.576928+0000", 2024-09-13T14:48:40.650 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:48:40.650 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.650 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.650 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:48:40.651 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.576928+0000", 2024-09-13T14:48:40.651 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:48:40.651 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.651 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.651 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:48:40.651 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.576928+0000", 2024-09-13T14:48:40.651 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.5300000000000003e-07 2024-09-13T14:48:40.651 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.651 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.651 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:48:40.651 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.576933+0000", 2024-09-13T14:48:40.651 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.1749999999999998e-06 2024-09-13T14:48:40.651 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.652 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.652 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:48:40.652 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.576944+0000", 2024-09-13T14:48:40.652 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1579e-05 2024-09-13T14:48:40.652 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.652 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.652 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:48:40.652 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.576952+0000", 2024-09-13T14:48:40.652 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.3019999999999997e-06 2024-09-13T14:48:40.652 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:48:40.652 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:48:40.652 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:48:40.652 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.652 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.653 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.46157.0:177420 8.1 e344/337 8:81750884:::benchmark_data_smithi102_313275_object46833:head v 344'10978, mlcod=344'10976)", 2024-09-13T14:48:40.653 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:48:40.580795+0000", 2024-09-13T14:48:40.653 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.0057681640000000001, 2024-09-13T14:48:40.653 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0062056200000000002, 2024-09-13T14:48:40.653 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:48:40.653 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:48:40.653 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "started", 2024-09-13T14:48:40.653 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:48:40.653 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.653 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:48:40.653 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.580795+0000", 2024-09-13T14:48:40.653 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:48:40.653 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.654 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.654 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:48:40.654 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.580795+0000", 2024-09-13T14:48:40.654 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:48:40.654 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.654 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.654 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:48:40.654 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.580795+0000", 2024-09-13T14:48:40.654 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.7700000000000001e-07 2024-09-13T14:48:40.654 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.654 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.654 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:48:40.655 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.580800+0000", 2024-09-13T14:48:40.655 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.3730000000000003e-06 2024-09-13T14:48:40.655 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.655 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.655 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:48:40.655 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.580806+0000", 2024-09-13T14:48:40.655 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.9270000000000002e-06 2024-09-13T14:48:40.655 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.655 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.655 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:48:40.655 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.580810+0000", 2024-09-13T14:48:40.655 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.2560000000000004e-06 2024-09-13T14:48:40.655 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.655 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.656 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:48:40.656 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.584347+0000", 2024-09-13T14:48:40.656 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0035374019999999998 2024-09-13T14:48:40.656 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.656 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.656 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:48:40.656 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.584370+0000", 2024-09-13T14:48:40.656 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.2563000000000001e-05 2024-09-13T14:48:40.656 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:48:40.656 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:48:40.656 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:48:40.656 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.656 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.657 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:177419 8.a 8:521e248f:::benchmark_data_smithi102_313275_object46832:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e344)", 2024-09-13T14:48:40.657 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:48:40.580862+0000", 2024-09-13T14:48:40.657 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.0057011700000000002, 2024-09-13T14:48:40.657 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0061863300000000003, 2024-09-13T14:48:40.657 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:48:40.657 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:48:40.657 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:48:40.657 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:48:40.657 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:48:40.657 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:48:40.657 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 177419 2024-09-13T14:48:40.657 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.657 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:48:40.658 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.658 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:48:40.658 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.580862+0000", 2024-09-13T14:48:40.658 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:48:40.658 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.658 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.658 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:48:40.658 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.580862+0000", 2024-09-13T14:48:40.658 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:48:40.658 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.658 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.658 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:48:40.658 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.580863+0000", 2024-09-13T14:48:40.659 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.207e-06 2024-09-13T14:48:40.659 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.659 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.659 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:48:40.659 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.580867+0000", 2024-09-13T14:48:40.659 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.9740000000000004e-06 2024-09-13T14:48:40.659 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.659 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.659 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:48:40.659 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.580877+0000", 2024-09-13T14:48:40.659 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0193999999999999e-05 2024-09-13T14:48:40.659 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.659 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.660 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:48:40.660 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.580883+0000", 2024-09-13T14:48:40.660 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.5899999999999998e-06 2024-09-13T14:48:40.660 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.660 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.660 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:48:40.660 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.582033+0000", 2024-09-13T14:48:40.660 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.001150493 2024-09-13T14:48:40.660 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.660 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.660 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:48:40.660 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.582171+0000", 2024-09-13T14:48:40.660 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000137653 2024-09-13T14:48:40.661 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.661 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.661 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,3,5", 2024-09-13T14:48:40.661 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.582313+0000", 2024-09-13T14:48:40.661 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00014164900000000001 2024-09-13T14:48:40.661 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:48:40.661 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:48:40.661 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:48:40.661 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.661 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.661 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop_reply(client.46157.0:177416 8.3 e344/337)", 2024-09-13T14:48:40.661 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:48:40.582393+0000", 2024-09-13T14:48:40.661 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.0041703139999999996, 2024-09-13T14:48:40.661 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0047092540000000004, 2024-09-13T14:48:40.662 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:48:40.662 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:48:40.662 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "queued for pg", 2024-09-13T14:48:40.662 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:48:40.662 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.662 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:48:40.662 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.582393+0000", 2024-09-13T14:48:40.662 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:48:40.662 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.662 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.662 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:48:40.662 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.582393+0000", 2024-09-13T14:48:40.662 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:48:40.663 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.663 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.663 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:48:40.663 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.582394+0000", 2024-09-13T14:48:40.663 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.1399999999999996e-07 2024-09-13T14:48:40.663 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.663 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.663 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:48:40.663 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.582400+0000", 2024-09-13T14:48:40.663 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.63e-06 2024-09-13T14:48:40.663 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.663 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.663 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:48:40.664 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.582410+0000", 2024-09-13T14:48:40.664 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.91e-06 2024-09-13T14:48:40.664 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.664 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.664 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:48:40.664 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.582416+0000", 2024-09-13T14:48:40.664 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.0000000000000002e-06 2024-09-13T14:48:40.664 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:48:40.664 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:48:40.664 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:48:40.664 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.664 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.664 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "MOSDPGPushReply(8.7 344/337 [PushReplyOp(8:e9b1b60e:::benchmark_data_smithi102_313275_object47556:head)])", 2024-09-13T14:48:40.664 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:48:40.582458+0000", 2024-09-13T14:48:40.665 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.0041053679999999999, 2024-09-13T14:48:40.665 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0046803679999999999, 2024-09-13T14:48:40.665 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:48:40.665 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:48:40.665 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "queued for pg", 2024-09-13T14:48:40.665 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:48:40.665 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.665 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:48:40.665 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.582458+0000", 2024-09-13T14:48:40.665 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:48:40.665 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.665 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.665 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:48:40.665 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.582458+0000", 2024-09-13T14:48:40.666 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:48:40.666 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.666 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.666 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:48:40.666 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.582458+0000", 2024-09-13T14:48:40.666 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.0800000000000001e-07 2024-09-13T14:48:40.666 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.666 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.666 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:48:40.666 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.582461+0000", 2024-09-13T14:48:40.666 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.0860000000000002e-06 2024-09-13T14:48:40.666 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.666 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.667 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:48:40.667 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.582467+0000", 2024-09-13T14:48:40.667 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.421e-06 2024-09-13T14:48:40.667 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.667 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.667 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:48:40.667 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.582473+0000", 2024-09-13T14:48:40.667 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.0730000000000004e-06 2024-09-13T14:48:40.667 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:48:40.667 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:48:40.667 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:48:40.667 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.667 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.667 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop_reply(client.46157.0:177416 8.3 e344/337)", 2024-09-13T14:48:40.668 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:48:40.582729+0000", 2024-09-13T14:48:40.668 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.0038342760000000002, 2024-09-13T14:48:40.668 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0044433679999999996, 2024-09-13T14:48:40.668 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:48:40.668 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:48:40.668 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "queued for pg", 2024-09-13T14:48:40.668 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:48:40.668 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.668 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:48:40.668 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.582729+0000", 2024-09-13T14:48:40.668 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:48:40.668 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.669 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.669 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:48:40.669 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.582729+0000", 2024-09-13T14:48:40.669 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:48:40.669 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.669 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.669 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:48:40.669 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.582730+0000", 2024-09-13T14:48:40.669 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.3900000000000004e-07 2024-09-13T14:48:40.669 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.669 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.669 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:48:40.669 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.582733+0000", 2024-09-13T14:48:40.669 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.3000000000000002e-06 2024-09-13T14:48:40.670 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.670 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.670 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:48:40.670 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.582750+0000", 2024-09-13T14:48:40.670 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.6948000000000001e-05 2024-09-13T14:48:40.670 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:48:40.670 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:48:40.670 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:48:40.670 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:40.582755+0000", 2024-09-13T14:48:40.670 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.6339999999999998e-06 2024-09-13T14:48:40.670 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:48:40.670 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:48:40.670 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:48:40.670 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:48:40.671 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-09-13T14:48:40.671 INFO:teuthology.orchestra.run.smithi102.stdout: "num_ops": 12 2024-09-13T14:48:40.671 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:48:40.671 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:40.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:40.709 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:40.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:40.810 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:40.842 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:40.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:40.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:40.944 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:40.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:40.984 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:40.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:41.085 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:41.127 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:41.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:41.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:41.155 INFO:teuthology.orchestra.run.smithi102.stderr:reweighted osd.3 to 1 (10000) 2024-09-13T14:48:41.229 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:41.269 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:41.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:41.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:41.371 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:41.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:41 smithi102 ceph-mon[110183]: pgmap v1194: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 806 op/s; 34795/260307 objects misplaced (13.367%); 4.8 MiB/s, 76 objects/s recovering 2024-09-13T14:48:41.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:41 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:48:41.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:41 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1283635862' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:48:41.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:41 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:48:41.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:41 smithi102 ceph-mon[115269]: pgmap v1194: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 806 op/s; 34795/260307 objects misplaced (13.367%); 4.8 MiB/s, 76 objects/s recovering 2024-09-13T14:48:41.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:41 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:48:41.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:41 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1283635862' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:48:41.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:41 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:48:41.424 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:41.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:41.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:41.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:41 smithi195 ceph-mon[104262]: pgmap v1194: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 806 op/s; 34795/260307 objects misplaced (13.367%); 4.8 MiB/s, 76 objects/s recovering 2024-09-13T14:48:41.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:41 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:48:41.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:41 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1283635862' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:48:41.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:41 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:48:41.527 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:41.556 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:41.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:41.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:41.658 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:41.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:41.693 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:41.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:41.794 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:41.826 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:41.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:41.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:41.928 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:41.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:41.956 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:41.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:41.965 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T14:48:42.058 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:42.083 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:42.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:42.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:42.184 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:42.239 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:42.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:42.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:42.323 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:42 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T14:48:42.323 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:42 smithi102 ceph-mon[110183]: osdmap e345: 8 total, 8 up, 8 in 2024-09-13T14:48:42.323 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:42 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T14:48:42.323 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:42 smithi102 ceph-mon[115269]: osdmap e345: 8 total, 8 up, 8 in 2024-09-13T14:48:42.341 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:42.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:42.377 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:42.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:42.478 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:42.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:42 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T14:48:42.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:42 smithi195 ceph-mon[104262]: osdmap e345: 8 total, 8 up, 8 in 2024-09-13T14:48:42.510 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:42.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:42.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:42.612 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:42.654 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:42.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:42.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:42.756 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:42.782 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:42.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:42.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:42.785 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-13T14:48:42.884 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:42.913 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:42.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:42.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:42.956 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:42.957 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:43.015 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:43.038 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:43.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:43.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:43.140 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:43.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:43.185 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:43.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:43.286 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:43.327 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:43.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:43.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:43.363 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:43 smithi102 ceph-mon[110183]: pgmap v1196: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 810 op/s; 33823/253128 objects misplaced (13.362%); 5.5 MiB/s, 87 objects/s recovering 2024-09-13T14:48:43.363 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:43 smithi102 ceph-mon[115269]: pgmap v1196: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 810 op/s; 33823/253128 objects misplaced (13.362%); 5.5 MiB/s, 87 objects/s recovering 2024-09-13T14:48:43.434 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:43.470 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:43.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:43.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:43.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:43 smithi195 ceph-mon[104262]: pgmap v1196: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 810 op/s; 33823/253128 objects misplaced (13.362%); 5.5 MiB/s, 87 objects/s recovering 2024-09-13T14:48:43.564 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:48:43.572 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:43.599 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:43.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:43.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:43.701 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:43.733 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:43.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:43.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:43.834 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:43.858 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:43.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:43.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:43.959 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:43.984 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:43.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:43.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:44.086 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:44.112 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:44.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:44.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:44.215 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:44.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:44.253 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:44.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:44.354 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:44.387 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:44.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:44.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:44.489 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:44.517 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:44.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:44.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:44.619 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:44.648 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:44.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:44.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:44.750 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:44.789 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:44.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:44.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:44.891 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:44.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:44.924 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:44.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:45.025 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:45.086 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:45.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:45.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:45.187 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:45.240 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:45.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:45.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:45.342 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:45.343 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:45.366 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:45.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:45.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:45.468 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:45.492 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:45 smithi102 ceph-mon[110183]: pgmap v1197: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 969 op/s; 32820/247860 objects misplaced (13.241%); 5.2 MiB/s, 82 objects/s recovering 2024-09-13T14:48:45.493 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:45 smithi102 ceph-mon[115269]: pgmap v1197: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 969 op/s; 32820/247860 objects misplaced (13.241%); 5.2 MiB/s, 82 objects/s recovering 2024-09-13T14:48:45.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:45 smithi195 ceph-mon[104262]: pgmap v1197: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 969 op/s; 32820/247860 objects misplaced (13.241%); 5.2 MiB/s, 82 objects/s recovering 2024-09-13T14:48:45.498 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:45.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:45.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:45.600 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:45.632 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:45.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:45.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:45.734 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:45.761 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:45.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:45.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:45.863 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:45.902 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:45.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:45.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:46.004 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:46.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:46.031 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:46.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:46.132 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:46.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:46.178 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:46.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:46.279 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:46.316 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:46.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:46.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:46.418 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:46.454 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:46.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:46.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:46.556 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:46.576 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:46.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:46.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:46.678 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:46.724 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:46.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:46.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:46.826 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:46.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:46.848 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:46.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:46.919 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:48:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:48:46] "GET /metrics HTTP/1.1" 200 36535 "" "Prometheus/2.51.0" 2024-09-13T14:48:46.949 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:46.979 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:46.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:46.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:47.081 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:47.122 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:47.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:47.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:47.224 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:47.261 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:47.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:47.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:47.364 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:47.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:47.412 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:47.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:47.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:47 smithi195 ceph-mon[104262]: pgmap v1198: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 824 op/s; 32820/247860 objects misplaced (13.241%); 4.4 MiB/s, 70 objects/s recovering 2024-09-13T14:48:47.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:47 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2825274897' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:48:47.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:47 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:48:47.513 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:47.540 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:47.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:47.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:47.642 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:47.659 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is unset 2024-09-13T14:48:47.659 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:47 smithi102 ceph-mon[110183]: pgmap v1198: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 824 op/s; 32820/247860 objects misplaced (13.241%); 4.4 MiB/s, 70 objects/s recovering 2024-09-13T14:48:47.659 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:47 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2825274897' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:48:47.659 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:47 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:48:47.659 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:47 smithi102 ceph-mon[115269]: pgmap v1198: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 824 op/s; 32820/247860 objects misplaced (13.241%); 4.4 MiB/s, 70 objects/s recovering 2024-09-13T14:48:47.659 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:47 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2825274897' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:48:47.659 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:47 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:48:47.669 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:47.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:47.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:47.771 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:47.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:47.805 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:47.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:47.906 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:47.937 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:47.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:47.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:47.938 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:48.039 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:48.084 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:48.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:48.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:48.186 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:48.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:48.216 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:48.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:48.317 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:48.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:48.351 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:48.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:48.452 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:48.487 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:48.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:48.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:48.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:48 smithi195 ceph-mon[104262]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:48:48.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:48 smithi195 ceph-mon[104262]: Cluster is now healthy 2024-09-13T14:48:48.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:48 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:48:48.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:48 smithi195 ceph-mon[104262]: osdmap e346: 8 total, 8 up, 8 in 2024-09-13T14:48:48.589 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:48.623 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:48.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:48.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:48.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:48 smithi102 ceph-mon[110183]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:48:48.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:48 smithi102 ceph-mon[110183]: Cluster is now healthy 2024-09-13T14:48:48.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:48 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:48:48.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:48 smithi102 ceph-mon[110183]: osdmap e346: 8 total, 8 up, 8 in 2024-09-13T14:48:48.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:48 smithi102 ceph-mon[115269]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:48:48.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:48 smithi102 ceph-mon[115269]: Cluster is now healthy 2024-09-13T14:48:48.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:48 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:48:48.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:48 smithi102 ceph-mon[115269]: osdmap e346: 8 total, 8 up, 8 in 2024-09-13T14:48:48.726 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:48.760 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:48.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:48.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:48.862 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:48.895 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:48.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:48.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:48.997 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:49.047 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:49.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:49.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:49.148 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:49.174 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:49.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:49.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:49.276 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:49.310 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:49.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:49.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:49.407 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:49 smithi102 ceph-mon[110183]: pgmap v1200: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 800 op/s; 31866/241092 objects misplaced (13.217%); 5.3 MiB/s, 84 objects/s recovering 2024-09-13T14:48:49.407 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:49 smithi102 ceph-mon[115269]: pgmap v1200: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 800 op/s; 31866/241092 objects misplaced (13.217%); 5.3 MiB/s, 84 objects/s recovering 2024-09-13T14:48:49.412 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:49.469 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:49.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:49.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:49.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:49 smithi195 ceph-mon[104262]: pgmap v1200: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 800 op/s; 31866/241092 objects misplaced (13.217%); 5.3 MiB/s, 84 objects/s recovering 2024-09-13T14:48:49.575 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:49.606 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:49.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:49.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:49.709 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:49.749 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:49.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:49.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:49.758 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:48:49.758 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [], 2024-09-13T14:48:49.758 INFO:teuthology.orchestra.run.smithi102.stdout: "complaint_time": 30, 2024-09-13T14:48:49.758 INFO:teuthology.orchestra.run.smithi102.stdout: "num_blocked_ops": 0 2024-09-13T14:48:49.758 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:48:49.851 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:49.891 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:49.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:49.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:49.993 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:50.028 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:50.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:50.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:50.130 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:50.171 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:50.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:50.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:50.272 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:50.306 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:50.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:50.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:50.408 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:50.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:50.438 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:50.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:50.539 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:50.571 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:50.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:50.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:50.673 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:50.706 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:50.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:50.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:50.808 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:50.841 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:50.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:50.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:50.943 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:50.987 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:50.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:50.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:51.088 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:51.112 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:51.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:51.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:51.214 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:51.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:51.242 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:51.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:51.343 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:51.372 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:51.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:51.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:51.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:51 smithi102 ceph-mon[110183]: pgmap v1201: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 959 op/s; 30882/235902 objects misplaced (13.091%); 5.0 MiB/s, 79 objects/s recovering 2024-09-13T14:48:51.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:51 smithi102 ceph-mon[115269]: pgmap v1201: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 959 op/s; 30882/235902 objects misplaced (13.091%); 5.0 MiB/s, 79 objects/s recovering 2024-09-13T14:48:51.474 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:51.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:51 smithi195 ceph-mon[104262]: pgmap v1201: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 959 op/s; 30882/235902 objects misplaced (13.091%); 5.0 MiB/s, 79 objects/s recovering 2024-09-13T14:48:51.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:51.517 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:51.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:51.618 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:51.668 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T14:48:51.669 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:51.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:51.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:51.772 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:51.811 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:51.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:51.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:51.913 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:51.960 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:51.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:51.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:52.062 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:52.095 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:52.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:52.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:52.197 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:52.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:52.224 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:52.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:52.325 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:52.363 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:52.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:52.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:52.465 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:52.509 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:52.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:52.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:52.611 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:52.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:52.641 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:52.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:52.744 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:52.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:52.776 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:52.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:52.877 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:52.910 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:52.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:52.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:53.011 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:53.058 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:53.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:53.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:53.160 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:53.194 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:53.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:53.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:53.297 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:53.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:53.333 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:53.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:53.434 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:53.471 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:53.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:53.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:53.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:53 smithi195 ceph-mon[104262]: pgmap v1202: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 728 op/s; 30882/231264 objects misplaced (13.354%); 2.1 MiB/s, 33 objects/s recovering 2024-09-13T14:48:53.573 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:53.589 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:53 smithi102 ceph-mon[110183]: pgmap v1202: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 728 op/s; 30882/231264 objects misplaced (13.354%); 2.1 MiB/s, 33 objects/s recovering 2024-09-13T14:48:53.589 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:53 smithi102 ceph-mon[115269]: pgmap v1202: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 728 op/s; 30882/231264 objects misplaced (13.354%); 2.1 MiB/s, 33 objects/s recovering 2024-09-13T14:48:53.602 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:53.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:53.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:53.704 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:53.729 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:53.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:53.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:53.831 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:53.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:53.867 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:53.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:53.968 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:54.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:54.017 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:54.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:54.118 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:54.157 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:54.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:54.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:54.224 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:48:54.259 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:54.307 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:54.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:54.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:54.410 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:54.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:54.440 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:54.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:54.541 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:54.582 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:54.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:54.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:54.685 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:54.713 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:54.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:54.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:54.815 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:54.853 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:54.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:54.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:54.954 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:54.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:54.994 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:54.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:55.095 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:55.117 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:55.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:55.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:55.219 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:55.245 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:55.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:55.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:55.347 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:55.374 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:55.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:55.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:55.476 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:55.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:55 smithi195 ceph-mon[104262]: pgmap v1203: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 4.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 816 op/s; 28877/223350 objects misplaced (12.929%); 4.3 MiB/s, 69 objects/s recovering 2024-09-13T14:48:55.501 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:55.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:55.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:55.604 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:55.634 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:55.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:55.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:55.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:55 smithi102 ceph-mon[110183]: pgmap v1203: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 4.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 816 op/s; 28877/223350 objects misplaced (12.929%); 4.3 MiB/s, 69 objects/s recovering 2024-09-13T14:48:55.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:55 smithi102 ceph-mon[115269]: pgmap v1203: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 4.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 816 op/s; 28877/223350 objects misplaced (12.929%); 4.3 MiB/s, 69 objects/s recovering 2024-09-13T14:48:55.735 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:55.772 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:55.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:55.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:55.874 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:55.906 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:55.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:55.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:56.008 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:56.041 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:56.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:56.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:56.143 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:56.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:56.184 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:56.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:56.286 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:56.331 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:56.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:56.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:56.434 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:56.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:56.469 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:56.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:56.570 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:56.593 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:56.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:56.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:56.695 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:56.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:56.731 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:56.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:56.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:48:56.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:48:56.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:56 smithi195 ceph-mon[104262]: pgmap v1204: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 4.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 816 op/s; 28877/223350 objects misplaced (12.929%); 4.3 MiB/s, 69 objects/s recovering 2024-09-13T14:48:56.832 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:56.856 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:56.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:56.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:56.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:48:56.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:48:56.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:56 smithi102 ceph-mon[110183]: pgmap v1204: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 4.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 816 op/s; 28877/223350 objects misplaced (12.929%); 4.3 MiB/s, 69 objects/s recovering 2024-09-13T14:48:56.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:48:56.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:48:56.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:56 smithi102 ceph-mon[115269]: pgmap v1204: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 4.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 816 op/s; 28877/223350 objects misplaced (12.929%); 4.3 MiB/s, 69 objects/s recovering 2024-09-13T14:48:56.919 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:48:56 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:48:56] "GET /metrics HTTP/1.1" 200 36529 "" "Prometheus/2.51.0" 2024-09-13T14:48:56.958 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:56.983 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:56.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:56.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:57.085 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:57.111 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:57.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:57.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:57.213 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:57.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:57.241 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:57.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:57.342 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:57.371 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:57.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:57.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:57.473 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:57.496 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:57.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:57.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:57.598 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:57.622 INFO:teuthology.orchestra.run.smithi102.stderr:reweighted osd.4 to 1 (10000) 2024-09-13T14:48:57.626 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:57.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:57.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:57.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:57 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2552682854' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:48:57.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:57 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:48:57.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:57 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2552682854' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:48:57.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:57 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:48:57.728 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:57.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:57.762 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:57.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:57.863 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:57.912 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:57.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:57.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:57.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:57 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2552682854' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:48:57.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:57 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:48:58.014 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:58.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:58.045 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:58.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:58.146 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:58.183 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:58.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:58.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:58.209 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:58.213 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:58.285 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:58.320 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:58.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:58.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:58.423 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:58.461 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:58.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:58.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:58.562 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:58.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:58.589 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:58.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:58.690 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:58.717 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-13T14:48:58.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:58.738 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:58.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:58.840 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:58.887 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:58.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:58.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:58.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:58 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T14:48:58.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:58 smithi102 ceph-mon[110183]: osdmap e347: 8 total, 8 up, 8 in 2024-09-13T14:48:58.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:48:58 smithi102 ceph-mon[110183]: pgmap v1206: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 4.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 808 op/s; 27989/216822 objects misplaced (12.909%); 4.5 MiB/s, 72 objects/s recovering 2024-09-13T14:48:58.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:58 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T14:48:58.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:58 smithi102 ceph-mon[115269]: osdmap e347: 8 total, 8 up, 8 in 2024-09-13T14:48:58.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:48:58 smithi102 ceph-mon[115269]: pgmap v1206: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 4.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 808 op/s; 27989/216822 objects misplaced (12.909%); 4.5 MiB/s, 72 objects/s recovering 2024-09-13T14:48:58.991 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:58.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:58 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T14:48:58.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:58 smithi195 ceph-mon[104262]: osdmap e347: 8 total, 8 up, 8 in 2024-09-13T14:48:58.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:48:58 smithi195 ceph-mon[104262]: pgmap v1206: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 4.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 808 op/s; 27989/216822 objects misplaced (12.909%); 4.5 MiB/s, 72 objects/s recovering 2024-09-13T14:48:59.038 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:59.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:59.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:59.140 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:59.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:59.181 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:59.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:59.281 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:59.307 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:59.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:59.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:59.410 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:59.447 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:59.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:59.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:59.549 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:59.588 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:48:59.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:59.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:59.690 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:59.726 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:59.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:59.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:59.828 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:59.859 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:48:59.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:59.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:59.961 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:00.000 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:00.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:00.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:00.104 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:00.132 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:00.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:00.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:00.234 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:00.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:00.281 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:00.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:00.382 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:00.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:00.415 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:00.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:00.517 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:00.548 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:00.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:00.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:00.650 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:00.680 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:00.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:00.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:00.782 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:00.809 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:00.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:00.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:00.911 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:00.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:00 smithi102 ceph-mon[110183]: pgmap v1207: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 803 op/s; 27037/211770 objects misplaced (12.767%); 4.5 MiB/s, 72 objects/s recovering 2024-09-13T14:49:00.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:00 smithi102 ceph-mon[115269]: pgmap v1207: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 803 op/s; 27037/211770 objects misplaced (12.767%); 4.5 MiB/s, 72 objects/s recovering 2024-09-13T14:49:00.948 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:00.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:00.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:00.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:00 smithi195 ceph-mon[104262]: pgmap v1207: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 803 op/s; 27037/211770 objects misplaced (12.767%); 4.5 MiB/s, 72 objects/s recovering 2024-09-13T14:49:01.050 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:01.079 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:01.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:01.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:01.181 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:01.229 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:01.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:01.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:01.313 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:49:01.313 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:49:01.330 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:01.347 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:01.350 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:49:01.350 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:49:01.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:01.385 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:01.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:01.401 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:49:01.401 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:49:01.439 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:49:01.439 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:49:01.474 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:49:01.474 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:49:01.486 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:01.510 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:01.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:01.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:01.520 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:49:01.520 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:49:01.550 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:49:01.551 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:49:01.588 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:49:01.588 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:49:01.613 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:01.638 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:01.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:01.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:01.740 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:01.778 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:01.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:01.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:01.880 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:01.911 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:01.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:01.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:01.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:01 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3727841805' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:49:01.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:01 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3727841805' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:49:01.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:01 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3727841805' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:49:02.013 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:02.046 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:02.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:02.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:02.147 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:02.175 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:02.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:02.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:02.277 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:02.316 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:02.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:02.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:02.418 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:02.445 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:02.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:02.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:02.547 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:02.581 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:02.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:02.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:02.683 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:02.724 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:02.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:02.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:02.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:02 smithi195 ceph-mon[104262]: pgmap v1208: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 4.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 798 op/s; 27037/207285 objects misplaced (13.043%); 4.5 MiB/s, 72 objects/s recovering 2024-09-13T14:49:02.826 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:02.858 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:02.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:02.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:02.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:02 smithi102 ceph-mon[110183]: pgmap v1208: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 4.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 798 op/s; 27037/207285 objects misplaced (13.043%); 4.5 MiB/s, 72 objects/s recovering 2024-09-13T14:49:02.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:02 smithi102 ceph-mon[115269]: pgmap v1208: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 4.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 798 op/s; 27037/207285 objects misplaced (13.043%); 4.5 MiB/s, 72 objects/s recovering 2024-09-13T14:49:02.959 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:02.985 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:02.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:02.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:03.087 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:03.120 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:03.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:03.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:03.222 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:03.253 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:03.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:03.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:03.355 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:03.377 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:03.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:03.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:03.479 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:03.506 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:03.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:03.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:03.608 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:03.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:03.641 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:03.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:03.742 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:03.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:03.782 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:03.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:03.883 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:03.914 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:03.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:03.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:03.917 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:49:03.917 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:49:04.016 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:04.044 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:04.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:04.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:04.146 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:04.185 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:04.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:04.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:04.286 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:04.313 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:04.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:04.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:04.414 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:04.450 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:04.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:04.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:04.552 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:04.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:04.588 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:04.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:04.689 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:04.731 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:04.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:04.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:04.833 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:04.876 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:04.877 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T14:49:04.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:04.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:04.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:04 smithi102 ceph-mon[110183]: pgmap v1209: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 4.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 785 op/s; 25186/199761 objects misplaced (12.608%); 4.2 MiB/s, 67 objects/s recovering 2024-09-13T14:49:04.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:04 smithi102 ceph-mon[115269]: pgmap v1209: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 4.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 785 op/s; 25186/199761 objects misplaced (12.608%); 4.2 MiB/s, 67 objects/s recovering 2024-09-13T14:49:04.979 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:04.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:04 smithi195 ceph-mon[104262]: pgmap v1209: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 4.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 785 op/s; 25186/199761 objects misplaced (12.608%); 4.2 MiB/s, 67 objects/s recovering 2024-09-13T14:49:05.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:05.009 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:05.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:05.110 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:05.155 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:05.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:05.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:05.257 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:05.299 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:05.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:05.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:05.401 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:05.429 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:05.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:05.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:05.531 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:05.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:05.574 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:05.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:05.675 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:05.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:05.714 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:05.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:05.815 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:05.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:05.859 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:05.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:05.960 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:05.988 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:05.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:05.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:06.090 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:06.126 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:06.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:06.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:06.228 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:06.270 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:06.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:06.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:06.372 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:06.398 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:06.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:06.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:06.501 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:06.527 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:06.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:06.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:06.629 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:06.666 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:49:06 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:49:06] "GET /metrics HTTP/1.1" 200 36529 "" "Prometheus/2.51.0" 2024-09-13T14:49:06.667 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:06.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:06.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:06.770 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:06.804 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:06.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:06.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:06.905 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:06.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:06 smithi102 ceph-mon[110183]: pgmap v1210: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 4.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 785 op/s; 25186/199761 objects misplaced (12.608%); 4.2 MiB/s, 67 objects/s recovering 2024-09-13T14:49:06.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:06 smithi102 ceph-mon[115269]: pgmap v1210: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 4.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 785 op/s; 25186/199761 objects misplaced (12.608%); 4.2 MiB/s, 67 objects/s recovering 2024-09-13T14:49:06.932 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:06.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:06.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:06.977 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:06.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:06 smithi195 ceph-mon[104262]: pgmap v1210: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 4.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 785 op/s; 25186/199761 objects misplaced (12.608%); 4.2 MiB/s, 67 objects/s recovering 2024-09-13T14:49:07.034 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:07.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:07.057 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:07.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:07.158 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:07.179 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:07.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:07.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:07.281 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:07.303 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:07.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:07.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:07.405 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:07.428 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:07.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:07.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:07.530 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:07.554 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:07.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:07.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:07.656 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:07.683 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:07.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:07.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:07.786 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:07.793 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:49:07.812 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:07.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:07.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:07.832 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:07 smithi102 ceph-mon[110183]: osdmap e348: 8 total, 8 up, 8 in 2024-09-13T14:49:07.832 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:07 smithi102 ceph-mon[115269]: osdmap e348: 8 total, 8 up, 8 in 2024-09-13T14:49:07.914 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:07.950 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:07.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:07.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:07.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:07 smithi195 ceph-mon[104262]: osdmap e348: 8 total, 8 up, 8 in 2024-09-13T14:49:08.052 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:08.089 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:08.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:08.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:08.191 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:08.213 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:08.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:08.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:08.314 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:08.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:08.349 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:08.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:08.451 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:08.491 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:08.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:08.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:08.593 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:08.614 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:08.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:08.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:08.716 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:08.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:08.748 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:08.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:08.849 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:08.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:08.889 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:08.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:08.951 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:08 smithi195 ceph-mon[104262]: pgmap v1212: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 3.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 772 op/s; 24709/193650 objects misplaced (12.760%); 1.9 MiB/s, 30 objects/s recovering 2024-09-13T14:49:08.951 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:08 smithi195 ceph-mon[104262]: osdmap e349: 8 total, 8 up, 8 in 2024-09-13T14:49:08.951 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:08 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:49:08.951 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:08 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:49:08.951 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:08 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:49:08.990 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:09.033 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:09.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:09.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:09.093 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:08 smithi102 ceph-mon[110183]: pgmap v1212: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 3.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 772 op/s; 24709/193650 objects misplaced (12.760%); 1.9 MiB/s, 30 objects/s recovering 2024-09-13T14:49:09.093 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:08 smithi102 ceph-mon[110183]: osdmap e349: 8 total, 8 up, 8 in 2024-09-13T14:49:09.093 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:08 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:49:09.093 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:08 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:49:09.093 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:08 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:49:09.093 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:08 smithi102 ceph-mon[115269]: pgmap v1212: 109 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 102 active+clean; 3.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 772 op/s; 24709/193650 objects misplaced (12.760%); 1.9 MiB/s, 30 objects/s recovering 2024-09-13T14:49:09.093 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:08 smithi102 ceph-mon[115269]: osdmap e349: 8 total, 8 up, 8 in 2024-09-13T14:49:09.093 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:08 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:49:09.094 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:08 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:49:09.094 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:08 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:49:09.135 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:09.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:09.176 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:09.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:09.277 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:09.314 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:09.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:09.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:09.416 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:09.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:09.441 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:09.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:09.542 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:09.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:09.588 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:09.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:09.689 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:09.729 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:09.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:09.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:09.831 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:09.863 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:09.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:09.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:09.964 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:10.011 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:10.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:10.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:10.112 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:10.120 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:10.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:10.191 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:10.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:10.292 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:10.322 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:10.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:10.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:10.424 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:10.461 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:10.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:10.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:10.563 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:10.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:10.601 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:10.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:10.702 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:10.728 INFO:teuthology.orchestra.run.smithi102.stderr:reweighted osd.6 to 1 (10000) 2024-09-13T14:49:10.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:10.730 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:10.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:10.732 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is set 2024-09-13T14:49:10.831 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:10.858 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:10.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:10.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:10.932 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:10 smithi102 ceph-mon[110183]: pgmap v1214: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 3.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 804 op/s; 22924/187956 objects misplaced (12.196%); 9.3 MiB/s, 149 objects/s recovering 2024-09-13T14:49:10.932 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:10 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1330314337' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:49:10.932 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:10 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:49:10.932 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:10 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3792295662' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:49:10.932 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:10 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:49:10.932 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:49:10.933 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:10 smithi102 ceph-mon[115269]: pgmap v1214: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 3.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 804 op/s; 22924/187956 objects misplaced (12.196%); 9.3 MiB/s, 149 objects/s recovering 2024-09-13T14:49:10.933 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:10 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1330314337' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:49:10.933 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:10 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:49:10.933 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:10 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3792295662' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:49:10.933 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:10 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:49:10.933 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:49:10.961 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:10.969 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:10 smithi195 ceph-mon[104262]: pgmap v1214: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 3.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 804 op/s; 22924/187956 objects misplaced (12.196%); 9.3 MiB/s, 149 objects/s recovering 2024-09-13T14:49:10.969 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:10 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1330314337' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:49:10.969 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:10 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:49:10.969 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:10 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3792295662' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:49:10.969 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:10 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:49:10.969 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:49:11.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:11.001 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:11.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:11.103 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:11.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:11.140 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:11.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:11.241 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:11.274 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:11.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:11.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:11.376 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:11.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:11.415 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:11.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:11.516 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:11.564 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:11.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:11.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:11.665 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:11.699 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:11.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:11.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:11.801 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:11.828 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:11.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:11.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:11.930 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:11.970 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:11.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:11.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:11.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:11 smithi195 ceph-mon[104262]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:49:11.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:11 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T14:49:11.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:11 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:49:11.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:11 smithi195 ceph-mon[104262]: osdmap e350: 8 total, 8 up, 8 in 2024-09-13T14:49:12.072 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:12.089 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:11 smithi102 ceph-mon[110183]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:49:12.090 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:11 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T14:49:12.090 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:11 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:49:12.090 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:11 smithi102 ceph-mon[110183]: osdmap e350: 8 total, 8 up, 8 in 2024-09-13T14:49:12.091 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:11 smithi102 ceph-mon[115269]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:49:12.091 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:11 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T14:49:12.091 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:11 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:49:12.091 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:11 smithi102 ceph-mon[115269]: osdmap e350: 8 total, 8 up, 8 in 2024-09-13T14:49:12.107 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:12.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:12.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:12.209 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:12.245 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:12.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:12.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:12.347 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:12.378 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:12.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:12.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:12.480 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:12.514 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:12.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:12.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:12.616 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:12.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:12.640 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:12.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:12.741 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:12.778 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:12.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:12.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:12.880 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:12.936 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:12.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:12.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:12.992 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:12 smithi195 ceph-mon[104262]: pgmap v1216: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 3.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 861 op/s; 22924/184251 objects misplaced (12.442%); 9.2 MiB/s, 147 objects/s recovering 2024-09-13T14:49:13.039 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:13.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:13.076 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:13.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:13.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:12 smithi102 ceph-mon[110183]: pgmap v1216: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 3.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 861 op/s; 22924/184251 objects misplaced (12.442%); 9.2 MiB/s, 147 objects/s recovering 2024-09-13T14:49:13.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:12 smithi102 ceph-mon[115269]: pgmap v1216: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 3.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 861 op/s; 22924/184251 objects misplaced (12.442%); 9.2 MiB/s, 147 objects/s recovering 2024-09-13T14:49:13.177 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:13.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:13.206 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:13.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:13.307 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:13.331 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:13.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:13.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:13.433 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:13.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:13.481 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:13.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:13.582 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:13.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:13.624 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:13.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:13.725 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:13.745 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:13.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:13.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:13.847 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:13.886 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:13.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:13.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:13.988 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:14.029 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:14.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:14.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:14.131 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:14.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:14.166 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:14.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:14.267 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:14.292 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:14.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:14.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:14.394 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:14.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:14.425 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:14.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:14.526 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:14.557 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:14.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:14.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:14.659 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:14.687 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:14.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:14.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:14.789 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:14.827 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:14.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:14.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:14.875 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:14 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:49:14.875 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:14 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:49:14.875 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:14 smithi102 ceph-mon[110183]: pgmap v1217: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 3.6 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 818 op/s; 21187/177966 objects misplaced (11.905%); 12 MiB/s, 188 objects/s recovering 2024-09-13T14:49:14.875 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:14 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:49:14.875 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:14 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:49:14.875 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:14 smithi102 ceph-mon[115269]: pgmap v1217: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 3.6 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 818 op/s; 21187/177966 objects misplaced (11.905%); 12 MiB/s, 188 objects/s recovering 2024-09-13T14:49:14.909 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-13T14:49:14.928 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:14.956 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:14.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:14.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:14.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:14 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:49:14.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:14 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:49:14.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:14 smithi195 ceph-mon[104262]: pgmap v1217: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 3.6 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 818 op/s; 21187/177966 objects misplaced (11.905%); 12 MiB/s, 188 objects/s recovering 2024-09-13T14:49:15.058 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:15.094 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:15.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:15.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:15.195 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:15.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:15.237 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:15.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:15.338 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:15.381 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:15.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:15.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:15.483 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:15.518 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:15.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:15.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:15.619 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:15.646 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:15.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:15.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:15.747 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:15.779 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:15.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:15.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:15.880 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:15.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:15.941 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:15.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:16.042 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:16.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:16.081 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:16.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:16.182 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:16.192 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:49:16.193 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:49:16.222 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:16.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:16.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:16.324 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:16.356 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:16.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:16.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:16.458 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:16.473 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:49:16.473 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:49:16.485 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:16.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:16.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:16.506 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:49:16.506 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:49:16.544 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:49:16.544 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:49:16.581 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:49:16.581 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:49:16.586 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:16.610 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:49:16.611 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:49:16.622 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:16.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:16.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:16.642 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:49:16.642 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:49:16.676 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:49:16.677 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:49:16.701 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:49:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:49:16] "GET /metrics HTTP/1.1" 200 36535 "" "Prometheus/2.51.0" 2024-09-13T14:49:16.717 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:49:16.717 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:49:16.725 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:16.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:16.785 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:16.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:16.886 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:16.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:16.931 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:16.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:16.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:16 smithi195 ceph-mon[104262]: pgmap v1218: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 3.6 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 684 op/s; 21187/177216 objects misplaced (11.955%); 10 MiB/s, 164 objects/s recovering 2024-09-13T14:49:16.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:16 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/4100970120' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:49:17.032 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:17.063 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:17.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:17.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:17.164 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:17.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:16 smithi102 ceph-mon[110183]: pgmap v1218: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 3.6 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 684 op/s; 21187/177216 objects misplaced (11.955%); 10 MiB/s, 164 objects/s recovering 2024-09-13T14:49:17.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:16 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/4100970120' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:49:17.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:16 smithi102 ceph-mon[115269]: pgmap v1218: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 3.6 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 684 op/s; 21187/177216 objects misplaced (11.955%); 10 MiB/s, 164 objects/s recovering 2024-09-13T14:49:17.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:16 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/4100970120' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:49:17.194 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:17.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:17.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:17.279 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T14:49:17.295 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:17.328 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:17.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:17.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:17.430 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:17.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:17.477 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:17.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:17.578 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:17.606 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:17.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:17.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:17.707 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:17.717 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:49:17.733 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:17.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:17.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:17.835 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:17.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:17.871 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:17.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:17.972 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:18.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:18.001 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:18.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:18.102 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:18.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:18.125 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:18.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:18.226 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:18.265 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:18.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:18.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:18.367 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:18.411 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:18.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:18.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:18.564 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:18.586 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:18.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:18.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:18.688 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:18.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:18.727 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:18.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:18.828 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:18.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:18.868 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:18.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:18.969 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:18.992 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:18.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:18.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:18.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:18 smithi195 ceph-mon[104262]: pgmap v1219: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 3.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 757 op/s; 20685/170991 objects misplaced (12.097%); 8.3 MiB/s, 132 objects/s recovering 2024-09-13T14:49:19.093 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:19.115 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:19.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:19.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:19.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:18 smithi102 ceph-mon[110183]: pgmap v1219: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 3.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 757 op/s; 20685/170991 objects misplaced (12.097%); 8.3 MiB/s, 132 objects/s recovering 2024-09-13T14:49:19.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:18 smithi102 ceph-mon[115269]: pgmap v1219: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 3.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 757 op/s; 20685/170991 objects misplaced (12.097%); 8.3 MiB/s, 132 objects/s recovering 2024-09-13T14:49:19.217 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:19.249 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:19.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:19.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:19.350 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:19.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:19.398 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:19.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:19.499 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:19.500 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:19.525 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:19.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:19.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:19.626 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:19.665 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:19.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:19.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:19.767 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:19.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:19.801 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:19.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:19.902 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:19.932 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:19.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:19.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:20.034 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:20.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:20.069 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:20.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:20.169 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:20.206 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:20.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:20.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:20.308 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:20.330 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:20.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:20.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:20.432 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:20.468 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:20.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:20.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:20.570 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:20.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:20.603 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:20.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:20.704 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:20.731 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:20.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:20.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:20.833 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:20.870 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:20.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:20.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:20.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:20 smithi102 ceph-mon[110183]: pgmap v1220: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 3.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 773 op/s; 19358/164736 objects misplaced (11.751%); 4.9 MiB/s, 79 objects/s recovering 2024-09-13T14:49:20.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:20 smithi102 ceph-mon[115269]: pgmap v1220: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 3.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 773 op/s; 19358/164736 objects misplaced (11.751%); 4.9 MiB/s, 79 objects/s recovering 2024-09-13T14:49:20.972 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:20.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:20 smithi195 ceph-mon[104262]: pgmap v1220: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 3.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 773 op/s; 19358/164736 objects misplaced (11.751%); 4.9 MiB/s, 79 objects/s recovering 2024-09-13T14:49:20.999 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:21.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:21.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:21.102 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:21.146 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:21.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:21.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:21.249 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:21.280 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:21.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:21.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:21.382 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:21.410 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:21.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:21.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:21.479 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:49:21.512 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:21.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:21.576 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:21.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:21.678 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:21.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:21.719 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:21.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:21.820 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:21.841 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:21.847 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:21.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:21.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:21.861 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:21.949 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:21.975 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:21.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:21.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:22.077 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:22.118 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:22.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:22.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:22.220 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:22.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:22.262 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:22.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:22.363 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:22.403 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:22.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:22.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:22.506 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:22.558 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:22.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:22.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:22.659 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:22.690 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:22.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:22.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:22.792 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:22.836 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:22.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:22.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:22.938 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:22.975 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:22.975 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:22 smithi102 ceph-mon[110183]: pgmap v1221: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 3.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 840 op/s; 19358/160356 objects misplaced (12.072%); 4.5 MiB/s, 72 objects/s recovering 2024-09-13T14:49:22.976 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:22 smithi102 ceph-mon[115269]: pgmap v1221: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 3.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 840 op/s; 19358/160356 objects misplaced (12.072%); 4.5 MiB/s, 72 objects/s recovering 2024-09-13T14:49:22.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:22.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:22.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:22 smithi195 ceph-mon[104262]: pgmap v1221: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 3.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 840 op/s; 19358/160356 objects misplaced (12.072%); 4.5 MiB/s, 72 objects/s recovering 2024-09-13T14:49:23.077 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:23.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:23.121 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:23.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:23.222 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:23.259 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:23.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:23.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:23.361 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:23.389 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:23.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:23.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:23.491 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:23.538 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:23.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:23.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:23.640 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:23.681 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:23.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:23.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:23.783 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:23.809 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:23.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:23.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:23.911 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:23.945 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:23.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:23.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:24.047 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:24.079 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:24.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:24.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:24.181 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:24.224 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:24.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:24.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:24.326 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:24.357 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:24.363 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:24.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:24.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:24.464 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:24.513 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:24.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:24.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:24.615 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:24.656 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:24.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:24.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:24.758 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:24.787 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:24.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:24.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:24.875 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:24 smithi102 ceph-mon[110183]: pgmap v1222: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 3.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 846 op/s; 17531/153744 objects misplaced (11.403%); 6.5 MiB/s, 104 objects/s recovering 2024-09-13T14:49:24.875 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:24 smithi102 ceph-mon[115269]: pgmap v1222: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 3.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 846 op/s; 17531/153744 objects misplaced (11.403%); 6.5 MiB/s, 104 objects/s recovering 2024-09-13T14:49:24.889 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:24.918 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:24.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:24.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:24.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:24 smithi195 ceph-mon[104262]: pgmap v1222: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 3.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 846 op/s; 17531/153744 objects misplaced (11.403%); 6.5 MiB/s, 104 objects/s recovering 2024-09-13T14:49:25.021 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:25.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:25.060 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:25.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:25.161 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:25.201 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:25.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:25.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:25.303 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:25.340 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:25.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:25.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:25.442 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:25.464 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:25.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:25.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:25.566 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:25.592 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:25.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:25.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:25.695 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:25.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:25.726 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:25.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:25.749 INFO:teuthology.orchestra.run.smithi102.stderr:reweighted osd.7 to 1 (10000) 2024-09-13T14:49:25.827 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:25.853 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:25.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:25.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:25.955 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:25.978 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:25 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/688087209' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T14:49:25.978 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:25 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T14:49:25.978 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:25 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:49:25.978 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:25 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:49:25.978 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:25 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/688087209' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T14:49:25.978 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:25 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T14:49:25.978 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:25 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:49:25.978 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:25 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:49:25.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:25.993 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:25.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:25.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:25 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/688087209' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T14:49:25.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:25 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T14:49:25.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:25 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:49:25.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:25 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:49:26.094 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:26.119 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:26.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:26.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:26.221 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:26.245 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:26.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:26.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:26.347 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:26.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:26.384 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:26.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:26.486 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:26.516 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:26.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:26.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:26.553 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:49:26.554 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [], 2024-09-13T14:49:26.554 INFO:teuthology.orchestra.run.smithi102.stdout: "complaint_time": 30, 2024-09-13T14:49:26.554 INFO:teuthology.orchestra.run.smithi102.stdout: "num_blocked_ops": 0 2024-09-13T14:49:26.554 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:49:26.618 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:26.658 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:26.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:26.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:26.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:49:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:49:26] "GET /metrics HTTP/1.1" 200 36537 "" "Prometheus/2.51.0" 2024-09-13T14:49:26.760 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:26.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:26.801 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:26.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:26.903 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:26.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:26.935 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:26.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:27.036 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:27.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:27.080 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:27.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:27.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:26 smithi102 ceph-mon[110183]: pgmap v1223: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 3.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 695 op/s; 17531/152913 objects misplaced (11.465%); 4.3 MiB/s, 68 objects/s recovering 2024-09-13T14:49:27.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:26 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T14:49:27.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:26 smithi102 ceph-mon[110183]: osdmap e351: 8 total, 8 up, 8 in 2024-09-13T14:49:27.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:26 smithi102 ceph-mon[115269]: pgmap v1223: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 3.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 695 op/s; 17531/152913 objects misplaced (11.465%); 4.3 MiB/s, 68 objects/s recovering 2024-09-13T14:49:27.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:26 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T14:49:27.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:26 smithi102 ceph-mon[115269]: osdmap e351: 8 total, 8 up, 8 in 2024-09-13T14:49:27.184 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:27.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:27.213 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:27.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:27.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:26 smithi195 ceph-mon[104262]: pgmap v1223: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 3.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 695 op/s; 17531/152913 objects misplaced (11.465%); 4.3 MiB/s, 68 objects/s recovering 2024-09-13T14:49:27.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:26 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T14:49:27.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:26 smithi195 ceph-mon[104262]: osdmap e351: 8 total, 8 up, 8 in 2024-09-13T14:49:27.314 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:27.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:27.358 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:27.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:27.460 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:27.500 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:27.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:27.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:27.602 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:27.629 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:27.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:27.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:27.731 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:27.778 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:27.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:27.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:27.880 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:27.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:27.909 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:27.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:28.010 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:28.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:28.045 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:28.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:28.146 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:28.178 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:28.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:28.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:28.280 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:28.307 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:28.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:28.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:28.409 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:28.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:28.448 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:28.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:28.549 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:28.589 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:28.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:28.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:28.691 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:28.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:28.722 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:28.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:28.752 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T14:49:28.815 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:28 smithi102 ceph-mon[110183]: pgmap v1225: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 3.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 804 op/s; 16999/146835 objects misplaced (11.577%); 5.1 MiB/s, 82 objects/s recovering 2024-09-13T14:49:28.817 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:28 smithi102 ceph-mon[115269]: pgmap v1225: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 3.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 804 op/s; 16999/146835 objects misplaced (11.577%); 5.1 MiB/s, 82 objects/s recovering 2024-09-13T14:49:28.823 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:28.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:28.869 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:28.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:28.970 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:28.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:28.994 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:28.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:29.095 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:29.144 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:29.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:29.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:29.245 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:29.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:28 smithi195 ceph-mon[104262]: pgmap v1225: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 3.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 804 op/s; 16999/146835 objects misplaced (11.577%); 5.1 MiB/s, 82 objects/s recovering 2024-09-13T14:49:29.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:29.284 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:29.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:29.385 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:29.414 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:29.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:29.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:29.515 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:29.548 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:29.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:29.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:29.650 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:29.656 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-13T14:49:29.677 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:29.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:29.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:29.779 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:29.808 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:29.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:29.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:29.910 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:29.936 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:29.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:29.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:30.038 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:30.062 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:30.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:30.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:30.164 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:30.194 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:30.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:30.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:30.296 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:30.324 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:30.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:30.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:30.426 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:30.487 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:30.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:30.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:30.589 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:30.616 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:30.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:30.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:30.718 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:30.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:30.757 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:30.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:30.859 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:30.888 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:30.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:30.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:30.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:30 smithi102 ceph-mon[110183]: pgmap v1226: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 2.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 794 op/s; 15737/140898 objects misplaced (11.169%); 5.5 MiB/s, 87 objects/s recovering 2024-09-13T14:49:30.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:30 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2843521546' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:49:30.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:30 smithi102 ceph-mon[115269]: pgmap v1226: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 2.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 794 op/s; 15737/140898 objects misplaced (11.169%); 5.5 MiB/s, 87 objects/s recovering 2024-09-13T14:49:30.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:30 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2843521546' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:49:30.990 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:31.016 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:31.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:31.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:31.118 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:31.150 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:49:31.150 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:49:31.150 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:49:31.151 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:49:31.151 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:49:31.151 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:49:31.151 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:49:31.151 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:49:31.151 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:49:31.151 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:49:31.151 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:49:31.151 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:49:31.151 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:49:31.152 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:49:31.152 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:49:31.152 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:49:31.165 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:31.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:31.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:31.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:30 smithi195 ceph-mon[104262]: pgmap v1226: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 2.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 794 op/s; 15737/140898 objects misplaced (11.169%); 5.5 MiB/s, 87 objects/s recovering 2024-09-13T14:49:31.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:30 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2843521546' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:49:31.267 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:31.303 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:31.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:31.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:31.405 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:31.438 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:31.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:31.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:31.539 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:31.577 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:31.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:31.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:31.679 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:31.719 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:31.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:31.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:31.821 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:31.826 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is set 2024-09-13T14:49:31.857 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:31.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:31.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:31.958 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:32.000 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:32.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:32.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:32.102 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:32.136 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:32.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:32.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:32.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:31 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1923755895' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:49:32.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:31 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:49:32.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:31 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1923755895' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:49:32.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:31 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:49:32.238 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:32.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:31 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1923755895' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:49:32.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:31 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:49:32.268 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:32.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:32.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:32.370 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:32.404 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:32.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:32.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:32.506 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:32.556 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:32.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:32.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:32.658 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:32.685 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:32.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:32.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:32.787 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:32.812 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:32.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:32.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:32.914 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:32.952 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:32.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:32.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:33.054 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:33.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:33.085 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:33.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:33.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:32 smithi102 ceph-mon[110183]: pgmap v1227: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 2.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 797 op/s; 15737/136416 objects misplaced (11.536%); 5.5 MiB/s, 87 objects/s recovering 2024-09-13T14:49:33.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:32 smithi102 ceph-mon[110183]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:49:33.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:32 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:49:33.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:32 smithi102 ceph-mon[110183]: osdmap e352: 8 total, 8 up, 8 in 2024-09-13T14:49:33.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:32 smithi102 ceph-mon[115269]: pgmap v1227: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 2.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 797 op/s; 15737/136416 objects misplaced (11.536%); 5.5 MiB/s, 87 objects/s recovering 2024-09-13T14:49:33.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:32 smithi102 ceph-mon[115269]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:49:33.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:32 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:49:33.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:32 smithi102 ceph-mon[115269]: osdmap e352: 8 total, 8 up, 8 in 2024-09-13T14:49:33.186 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:33.218 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:33.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:33.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:33.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:32 smithi195 ceph-mon[104262]: pgmap v1227: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 2.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 797 op/s; 15737/136416 objects misplaced (11.536%); 5.5 MiB/s, 87 objects/s recovering 2024-09-13T14:49:33.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:32 smithi195 ceph-mon[104262]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:49:33.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:32 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:49:33.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:32 smithi195 ceph-mon[104262]: osdmap e352: 8 total, 8 up, 8 in 2024-09-13T14:49:33.320 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:33.356 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:33.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:33.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:33.458 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:33.497 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:33.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:33.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:33.599 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:33.640 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:33.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:33.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:33.742 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:33.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:33.776 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:33.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:33.877 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:33.891 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:49:33.900 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:33.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:33.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:34.002 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:34.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:34.045 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:34.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:34.148 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:34.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:34.202 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:34.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:34.303 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:34.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:34.370 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:34.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:34.412 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:34.432 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:34.451 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:34.471 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:34.518 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:34.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:34.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:34.619 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:34.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:34.654 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:34.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:34.755 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:34.776 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:34.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:34.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:34.878 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:34.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:34.909 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:34.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:34.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:34 smithi102 ceph-mon[110183]: pgmap v1229: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 2.6 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 977 op/s; 14028/129450 objects misplaced (10.837%); 6.0 MiB/s, 95 objects/s recovering 2024-09-13T14:49:34.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:34 smithi102 ceph-mon[115269]: pgmap v1229: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 2.6 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 977 op/s; 14028/129450 objects misplaced (10.837%); 6.0 MiB/s, 95 objects/s recovering 2024-09-13T14:49:34.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:34 smithi195 ceph-mon[104262]: pgmap v1229: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 2.6 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 977 op/s; 14028/129450 objects misplaced (10.837%); 6.0 MiB/s, 95 objects/s recovering 2024-09-13T14:49:35.010 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:35.040 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:35.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:35.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:35.142 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:35.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:35.194 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:35.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:35.295 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:35.330 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:35.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:35.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:35.432 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:35.457 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:35.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:35.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:35.559 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:35.598 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:35.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:35.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:35.700 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:35.726 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:35.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:35.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:35.828 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:35.856 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:35.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:35.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:35.958 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:36.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:36.003 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:36.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:36.104 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:36.132 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:36.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:36.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:36.234 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:36.272 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:36.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:36.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:36.348 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:49:36.374 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:36.420 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:36.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:36.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:36.522 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:36.548 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:36.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:36.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:36.650 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:36.669 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:49:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:49:36] "GET /metrics HTTP/1.1" 200 36537 "" "Prometheus/2.51.0" 2024-09-13T14:49:36.687 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:36.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:36.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:36.789 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:36.824 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:36.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:36.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:36.927 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:36.958 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:36.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:36.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:37.060 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:37.089 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:37.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:37.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:37.112 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:36 smithi102 ceph-mon[110183]: pgmap v1230: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 2.6 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 813 op/s; 14028/128733 objects misplaced (10.897%); 4.8 MiB/s, 77 objects/s recovering 2024-09-13T14:49:37.112 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:36 smithi102 ceph-mon[115269]: pgmap v1230: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 2.6 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 813 op/s; 14028/128733 objects misplaced (10.897%); 4.8 MiB/s, 77 objects/s recovering 2024-09-13T14:49:37.191 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:37.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:37.229 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:37.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:37.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:36 smithi195 ceph-mon[104262]: pgmap v1230: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 2.6 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 813 op/s; 14028/128733 objects misplaced (10.897%); 4.8 MiB/s, 77 objects/s recovering 2024-09-13T14:49:37.330 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:37.357 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:37.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:37.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:37.458 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:37.495 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:37.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:37.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:37.597 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:37.643 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:37.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:37.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:37.745 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:37.777 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:37.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:37.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:37.879 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:37.919 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:37.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:37.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:38.021 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:38.044 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:38.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:38.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:38.146 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:38.187 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:38.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:38.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:38.289 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:38.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:38.319 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:38.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:38.421 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:38.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:38.462 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:38.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:38.563 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:38.586 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:38.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:38.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:38.688 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:38.722 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:38.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:38.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:38.824 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:38.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:38.855 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:38.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:38.956 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:38.988 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:38.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:38.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:39.004 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:38 smithi102 ceph-mon[110183]: pgmap v1231: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 2.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 811 op/s; 13504/122490 objects misplaced (11.025%); 4.8 MiB/s, 76 objects/s recovering 2024-09-13T14:49:39.004 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:38 smithi102 ceph-mon[115269]: pgmap v1231: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 2.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 811 op/s; 13504/122490 objects misplaced (11.025%); 4.8 MiB/s, 76 objects/s recovering 2024-09-13T14:49:39.090 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:39.122 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:39.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:39.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:39.224 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:39.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:38 smithi195 ceph-mon[104262]: pgmap v1231: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 2.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 811 op/s; 13504/122490 objects misplaced (11.025%); 4.8 MiB/s, 76 objects/s recovering 2024-09-13T14:49:39.251 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:39.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:39.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:39.353 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:39.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:39.389 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:39.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:39.490 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:39.522 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:39.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:39.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:39.625 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:39.660 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:39.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:39.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:39.762 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:39.816 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:39.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:39.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:39.919 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:39.950 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:39.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:39.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:40.053 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:40.076 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:40.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:40.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:40.178 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:40.205 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:40.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:40.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:40.307 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:40.330 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:40.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:40.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:40.431 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:40.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:40.454 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:40.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:40.555 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:40.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:40.579 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:40.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:40.680 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:40.701 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:40.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:40.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:40.803 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:40.830 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:40.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:40.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:40.888 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:40 smithi102 ceph-mon[110183]: pgmap v1232: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 2.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 810 op/s; 12282/116568 objects misplaced (10.536%); 4.8 MiB/s, 77 objects/s recovering 2024-09-13T14:49:40.888 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:40 smithi102 ceph-mon[110183]: osdmap e353: 8 total, 8 up, 8 in 2024-09-13T14:49:40.888 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:40 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:49:40.888 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:40 smithi102 ceph-mon[115269]: pgmap v1232: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 2.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 810 op/s; 12282/116568 objects misplaced (10.536%); 4.8 MiB/s, 77 objects/s recovering 2024-09-13T14:49:40.888 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:40 smithi102 ceph-mon[115269]: osdmap e353: 8 total, 8 up, 8 in 2024-09-13T14:49:40.888 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:40 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:49:40.932 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:40.973 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:40.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:40.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:41.075 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:41.115 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:41.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:41.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:41.217 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:41.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:40 smithi195 ceph-mon[104262]: pgmap v1232: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 2.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 810 op/s; 12282/116568 objects misplaced (10.536%); 4.8 MiB/s, 77 objects/s recovering 2024-09-13T14:49:41.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:40 smithi195 ceph-mon[104262]: osdmap e353: 8 total, 8 up, 8 in 2024-09-13T14:49:41.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:40 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:49:41.255 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:41.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:41.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:41.357 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:41.383 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:41.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:41.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:41.485 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:41.517 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:41.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:41.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:41.619 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:41.647 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:41.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:41.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:41.749 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:41.794 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:41.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:41.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:41.895 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:41.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:41.933 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:41.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:42.034 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:42.063 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:42.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:42.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:42.165 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:42.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:41 smithi102 ceph-mon[115269]: osdmap e354: 8 total, 8 up, 8 in 2024-09-13T14:49:42.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:41 smithi102 ceph-mon[110183]: osdmap e354: 8 total, 8 up, 8 in 2024-09-13T14:49:42.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:42.203 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:42.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:42.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:41 smithi195 ceph-mon[104262]: osdmap e354: 8 total, 8 up, 8 in 2024-09-13T14:49:42.304 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:42.335 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:42.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:42.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:42.436 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:42.460 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:42.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:42.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:42.556 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:49:42.556 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:49:42.562 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:42.588 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:42.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:42.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:42.590 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:49:42.590 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:49:42.638 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:49:42.639 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:49:42.674 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:49:42.674 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:49:42.691 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:42.707 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:49:42.707 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:49:42.736 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:42.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:42.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:42.742 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:49:42.742 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:49:42.787 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:49:42.787 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:49:42.814 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:49:42.814 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:49:42.839 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:42.864 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:42.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:42.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:42.906 INFO:teuthology.orchestra.run.smithi102.stderr:reweighted osd.5 to 1 (10000) 2024-09-13T14:49:42.966 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:42.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:42.991 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:42.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:43.092 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:43.113 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:43.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:43.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:43.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:42 smithi102 ceph-mon[110183]: pgmap v1235: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 2.3 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 704 op/s; 12282/112533 objects misplaced (10.914%); 3.3 MiB/s, 52 objects/s recovering 2024-09-13T14:49:43.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:42 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2231240233' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:49:43.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:42 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:49:43.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:42 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3381027192' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:49:43.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:42 smithi102 ceph-mon[115269]: pgmap v1235: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 2.3 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 704 op/s; 12282/112533 objects misplaced (10.914%); 3.3 MiB/s, 52 objects/s recovering 2024-09-13T14:49:43.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:42 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2231240233' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:49:43.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:42 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:49:43.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:42 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3381027192' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:49:43.215 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:43.237 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:43.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:43.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:43.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:42 smithi195 ceph-mon[104262]: pgmap v1235: 109 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 103 active+clean; 2.3 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 704 op/s; 12282/112533 objects misplaced (10.914%); 3.3 MiB/s, 52 objects/s recovering 2024-09-13T14:49:43.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:42 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2231240233' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:49:43.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:42 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:49:43.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:42 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3381027192' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:49:43.339 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:43.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:43.379 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:43.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:43.480 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:43.516 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:43.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:43.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:43.618 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:43.656 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:43.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:43.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:43.758 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:43.804 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:43.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:43.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:43.906 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:43.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:43 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T14:49:43.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:43 smithi102 ceph-mon[110183]: osdmap e355: 8 total, 8 up, 8 in 2024-09-13T14:49:43.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:43 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T14:49:43.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:43 smithi102 ceph-mon[115269]: osdmap e355: 8 total, 8 up, 8 in 2024-09-13T14:49:43.937 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:43.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:43.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:44.038 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:44.075 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:44.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:44.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:44.177 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:44.204 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:44.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:44.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:44.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:43 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T14:49:44.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:43 smithi195 ceph-mon[104262]: osdmap e355: 8 total, 8 up, 8 in 2024-09-13T14:49:44.306 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:44.330 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:44.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:44.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:44.432 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:44.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:44.465 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:44.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:44.566 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:44.595 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:44.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:44.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:44.697 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:44.734 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:44.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:44.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:44.835 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:44.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:44.875 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:44.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:44.959 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-13T14:49:44.976 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:45.050 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:45.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:45.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:45.152 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:45.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:44 smithi102 ceph-mon[110183]: pgmap v1237: 109 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 104 active+clean; 2.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 882 op/s; 10519/106602 objects misplaced (9.868%); 12 MiB/s, 184 objects/s recovering 2024-09-13T14:49:45.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:44 smithi102 ceph-mon[115269]: pgmap v1237: 109 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 104 active+clean; 2.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 882 op/s; 10519/106602 objects misplaced (9.868%); 12 MiB/s, 184 objects/s recovering 2024-09-13T14:49:45.179 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:45.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:45.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:45.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:44 smithi195 ceph-mon[104262]: pgmap v1237: 109 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 104 active+clean; 2.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 882 op/s; 10519/106602 objects misplaced (9.868%); 12 MiB/s, 184 objects/s recovering 2024-09-13T14:49:45.282 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:45.319 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:45.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:45.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:45.421 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:45.462 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:45.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:45.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:45.564 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:45.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:45.596 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:45.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:45.618 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:45.696 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:45.727 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:45.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:45.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:45.828 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:45.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:45.868 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:45.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:45.970 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:46.013 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:46.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:46.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:46.115 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:46.143 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:46.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:46.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:46.245 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:46.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:46.311 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:46.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:46.411 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:46.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:46.469 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:46.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:46.571 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:46.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:46.633 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:46.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:46.636 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:49:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:49:46] "GET /metrics HTTP/1.1" 200 36531 "" "Prometheus/2.51.0" 2024-09-13T14:49:46.735 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:46.767 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:46.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:46.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:46.869 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:46.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:46.906 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:46.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:47.007 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:47.042 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:47.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:47.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:47.143 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:47.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:47.177 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:47.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:47.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:46 smithi195 ceph-mon[104262]: pgmap v1238: 109 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 104 active+clean; 2.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 585 op/s; 10519/106032 objects misplaced (9.921%); 7.1 MiB/s, 113 objects/s recovering 2024-09-13T14:49:47.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:49:47.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:49:47.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:49:47.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:49:47.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:49:47.277 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:47.307 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:47.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:47.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:47.345 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:46 smithi102 ceph-mon[110183]: pgmap v1238: 109 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 104 active+clean; 2.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 585 op/s; 10519/106032 objects misplaced (9.921%); 7.1 MiB/s, 113 objects/s recovering 2024-09-13T14:49:47.345 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:49:47.345 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:49:47.345 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:49:47.345 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:49:47.346 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:49:47.346 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:46 smithi102 ceph-mon[115269]: pgmap v1238: 109 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 104 active+clean; 2.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 585 op/s; 10519/106032 objects misplaced (9.921%); 7.1 MiB/s, 113 objects/s recovering 2024-09-13T14:49:47.346 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:49:47.346 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:49:47.346 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:49:47.346 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:49:47.346 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:49:47.409 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:47.435 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:47.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:47.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:47.537 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:47.543 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:49:47.544 INFO:teuthology.orchestra.run.smithi102.stdout: "size": 20, 2024-09-13T14:49:47.544 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 600, 2024-09-13T14:49:47.544 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [ 2024-09-13T14:49:47.544 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.544 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:55621 8.7 8:e7453055:::benchmark_data_smithi102_313275_object55620:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-13T14:49:47.544 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:46:38.945196+0000", 2024-09-13T14:49:47.544 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 188.589274512, 2024-09-13T14:49:47.544 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.27066489900000001, 2024-09-13T14:49:47.544 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:49:47.544 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:49:47.545 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:47.545 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:49:47.545 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:49:47.545 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:49:47.545 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 55621 2024-09-13T14:49:47.545 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.545 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:49:47.545 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.545 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:49:47.545 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:38.945196+0000", 2024-09-13T14:49:47.545 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:49:47.545 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.546 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.546 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:49:47.546 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:38.945196+0000", 2024-09-13T14:49:47.546 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:49:47.546 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.546 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.546 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:49:47.546 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:38.945197+0000", 2024-09-13T14:49:47.546 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.1500000000000004e-07 2024-09-13T14:49:47.546 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.546 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.546 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:49:47.547 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:38.945216+0000", 2024-09-13T14:49:47.547 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.9267999999999999e-05 2024-09-13T14:49:47.547 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.547 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.547 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:49:47.547 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:38.945227+0000", 2024-09-13T14:49:47.547 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0259e-05 2024-09-13T14:49:47.547 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.547 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.547 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:49:47.547 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:38.945233+0000", 2024-09-13T14:49:47.548 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.4509999999999997e-06 2024-09-13T14:49:47.548 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.548 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.548 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:49:47.548 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:38.945294+0000", 2024-09-13T14:49:47.548 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.0733999999999998e-05 2024-09-13T14:49:47.548 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.548 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.548 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:49:47.548 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:38.945416+0000", 2024-09-13T14:49:47.548 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000121749 2024-09-13T14:49:47.549 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.549 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.549 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:49:47.549 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:38.945533+0000", 2024-09-13T14:49:47.549 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000117531 2024-09-13T14:49:47.549 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.549 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.549 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:49:47.549 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:38.950132+0000", 2024-09-13T14:49:47.549 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0045992710000000003 2024-09-13T14:49:47.550 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.550 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.550 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:47.550 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:38.950171+0000", 2024-09-13T14:49:47.550 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.8825000000000002e-05 2024-09-13T14:49:47.550 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.550 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.550 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:47.550 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:39.215774+0000", 2024-09-13T14:49:47.550 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.26560235300000001 2024-09-13T14:49:47.550 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.551 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.551 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:49:47.551 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:39.215817+0000", 2024-09-13T14:49:47.551 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.3071000000000001e-05 2024-09-13T14:49:47.551 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.551 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.551 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:49:47.551 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:39.215861+0000", 2024-09-13T14:49:47.551 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.4672000000000002e-05 2024-09-13T14:49:47.551 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:49:47.551 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:49:47.551 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:49:47.552 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.552 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.552 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:55622 8.7 8:e4c8fb9b:::benchmark_data_smithi102_313275_object55621:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-13T14:49:47.552 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:46:38.945870+0000", 2024-09-13T14:49:47.552 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 188.588601049, 2024-09-13T14:49:47.552 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.31054596800000001, 2024-09-13T14:49:47.552 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:49:47.552 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:49:47.553 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:47.553 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:49:47.553 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:49:47.553 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:49:47.553 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 55622 2024-09-13T14:49:47.553 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.553 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:49:47.553 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.553 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:49:47.553 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:38.945870+0000", 2024-09-13T14:49:47.553 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:49:47.554 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.554 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.554 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:49:47.554 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:38.945870+0000", 2024-09-13T14:49:47.554 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:49:47.554 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.554 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.554 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:49:47.554 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:38.945871+0000", 2024-09-13T14:49:47.554 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.5899999999999996e-07 2024-09-13T14:49:47.554 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.554 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.555 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:49:47.555 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:38.945921+0000", 2024-09-13T14:49:47.555 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.0207000000000002e-05 2024-09-13T14:49:47.555 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.555 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.555 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:49:47.555 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:38.945931+0000", 2024-09-13T14:49:47.555 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.91e-06 2024-09-13T14:49:47.555 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.555 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.555 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:49:47.555 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:38.945937+0000", 2024-09-13T14:49:47.556 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.2879999999999998e-06 2024-09-13T14:49:47.556 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.556 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.556 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:49:47.556 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:38.947683+0000", 2024-09-13T14:49:47.556 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.001745592 2024-09-13T14:49:47.556 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.556 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.556 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:49:47.556 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:38.947812+0000", 2024-09-13T14:49:47.556 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00012924200000000001 2024-09-13T14:49:47.557 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.557 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.557 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:49:47.557 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:38.947966+0000", 2024-09-13T14:49:47.557 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00015401299999999999 2024-09-13T14:49:47.557 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.557 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.557 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:49:47.557 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:38.950379+0000", 2024-09-13T14:49:47.557 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0024132929999999999 2024-09-13T14:49:47.558 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.558 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.558 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:47.558 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:38.951387+0000", 2024-09-13T14:49:47.558 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0010074579999999999 2024-09-13T14:49:47.558 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.558 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.558 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:47.558 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:39.256327+0000", 2024-09-13T14:49:47.558 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.30494053999999998 2024-09-13T14:49:47.558 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.559 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.559 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:49:47.559 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:39.256379+0000", 2024-09-13T14:49:47.559 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.1705e-05 2024-09-13T14:49:47.559 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.559 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.559 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:49:47.559 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:39.256416+0000", 2024-09-13T14:49:47.559 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.7061000000000003e-05 2024-09-13T14:49:47.559 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:49:47.559 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:49:47.559 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:49:47.560 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.560 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.560 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:55641 8.7 8:eede7c29:::benchmark_data_smithi102_313275_object55640:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-13T14:49:47.560 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:46:38.958102+0000", 2024-09-13T14:49:47.560 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 188.57636871299999, 2024-09-13T14:49:47.560 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.29844512899999998, 2024-09-13T14:49:47.560 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:49:47.560 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:49:47.560 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:47.560 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:49:47.561 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:49:47.561 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:49:47.561 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 55641 2024-09-13T14:49:47.561 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.561 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:49:47.561 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.561 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:49:47.561 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:38.958102+0000", 2024-09-13T14:49:47.561 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:49:47.561 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.562 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.562 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:49:47.562 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:38.958102+0000", 2024-09-13T14:49:47.562 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:49:47.562 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.562 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.562 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:49:47.562 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:38.958103+0000", 2024-09-13T14:49:47.562 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.114e-06 2024-09-13T14:49:47.562 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.562 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.563 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:49:47.563 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:38.958124+0000", 2024-09-13T14:49:47.563 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.1007000000000001e-05 2024-09-13T14:49:47.563 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.563 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.563 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:49:47.563 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:38.958137+0000", 2024-09-13T14:49:47.563 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2849e-05 2024-09-13T14:49:47.563 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.563 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.563 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:49:47.563 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:38.958145+0000", 2024-09-13T14:49:47.564 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.9640000000000003e-06 2024-09-13T14:49:47.564 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.564 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.564 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:49:47.564 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:38.958241+0000", 2024-09-13T14:49:47.564 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.6045999999999995e-05 2024-09-13T14:49:47.564 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.565 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.565 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:49:47.565 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:38.958385+0000", 2024-09-13T14:49:47.565 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00014338400000000001 2024-09-13T14:49:47.565 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.565 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.565 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:49:47.565 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:38.958522+0000", 2024-09-13T14:49:47.565 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00013782900000000001 2024-09-13T14:49:47.565 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.565 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.566 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:49:47.566 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:38.960666+0000", 2024-09-13T14:49:47.566 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0021434929999999998 2024-09-13T14:49:47.566 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.566 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.566 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:47.566 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:38.961382+0000", 2024-09-13T14:49:47.566 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000716499 2024-09-13T14:49:47.566 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.566 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.567 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:47.567 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:39.256465+0000", 2024-09-13T14:49:47.567 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.29508249399999997 2024-09-13T14:49:47.567 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.567 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.567 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:49:47.567 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:39.256512+0000", 2024-09-13T14:49:47.567 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.7443e-05 2024-09-13T14:49:47.567 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.567 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.567 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:49:47.568 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:39.256547+0000", 2024-09-13T14:49:47.568 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.5006999999999999e-05 2024-09-13T14:49:47.568 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:49:47.568 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:49:47.568 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:49:47.568 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.568 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.568 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:55652 8.7 8:ebb5be44:::benchmark_data_smithi102_313275_object55651:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-13T14:49:47.568 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:46:38.965872+0000", 2024-09-13T14:49:47.568 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 188.56859930300001, 2024-09-13T14:49:47.568 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.29097279100000001, 2024-09-13T14:49:47.569 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:49:47.569 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:49:47.569 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:47.569 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:49:47.569 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:49:47.569 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:49:47.569 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 55652 2024-09-13T14:49:47.569 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.569 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:49:47.569 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.570 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:49:47.570 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:38.965872+0000", 2024-09-13T14:49:47.570 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:49:47.570 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.570 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.570 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:49:47.570 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:38.965872+0000", 2024-09-13T14:49:47.570 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:49:47.570 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.570 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.571 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:49:47.571 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:38.965872+0000", 2024-09-13T14:49:47.571 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.3499999999999996e-07 2024-09-13T14:49:47.571 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.571 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.571 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:49:47.571 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:38.965889+0000", 2024-09-13T14:49:47.571 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.7059e-05 2024-09-13T14:49:47.571 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.571 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.571 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:49:47.571 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:38.965897+0000", 2024-09-13T14:49:47.572 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.1709999999999996e-06 2024-09-13T14:49:47.572 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.572 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.572 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:49:47.572 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:38.965901+0000", 2024-09-13T14:49:47.572 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.2919999999999997e-06 2024-09-13T14:49:47.572 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.572 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.572 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:49:47.572 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:38.965949+0000", 2024-09-13T14:49:47.572 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.7691999999999997e-05 2024-09-13T14:49:47.573 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.573 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.573 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:49:47.573 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:38.966070+0000", 2024-09-13T14:49:47.573 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00012118299999999999 2024-09-13T14:49:47.573 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.573 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.573 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:49:47.573 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:38.966196+0000", 2024-09-13T14:49:47.573 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00012594599999999999 2024-09-13T14:49:47.573 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.574 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.574 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:49:47.574 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:38.974804+0000", 2024-09-13T14:49:47.574 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0086085299999999997 2024-09-13T14:49:47.574 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.574 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.574 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:47.574 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:38.975062+0000", 2024-09-13T14:49:47.574 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00025761600000000001 2024-09-13T14:49:47.574 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.574 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.574 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:47.575 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:39.256767+0000", 2024-09-13T14:49:47.575 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.28170509900000001 2024-09-13T14:49:47.575 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.575 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.575 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:49:47.575 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:39.256813+0000", 2024-09-13T14:49:47.575 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.5682000000000001e-05 2024-09-13T14:49:47.575 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.575 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.575 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:49:47.576 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:46:39.256844+0000", 2024-09-13T14:49:47.576 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.1885999999999999e-05 2024-09-13T14:49:47.576 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:49:47.576 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:49:47.576 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:49:47.576 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.576 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.576 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:133953 8.b 8:df7b1c1e:::benchmark_data_smithi102_313275_object3366:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-13T14:49:47.576 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:47:46.853533+0000", 2024-09-13T14:49:47.576 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 120.68093839700001, 2024-09-13T14:49:47.577 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.4499644360000001, 2024-09-13T14:49:47.577 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:49:47.577 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:49:47.577 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "reached pg", 2024-09-13T14:49:47.577 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:49:47.577 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:49:47.577 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:49:47.577 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 133953 2024-09-13T14:49:47.577 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.577 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:49:47.577 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.578 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:49:47.578 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.853533+0000", 2024-09-13T14:49:47.578 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:49:47.578 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.578 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.578 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:49:47.578 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.853533+0000", 2024-09-13T14:49:47.578 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:49:47.578 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.578 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.578 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:49:47.579 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.853534+0000", 2024-09-13T14:49:47.579 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.5689999999999999e-06 2024-09-13T14:49:47.579 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.579 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.579 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:49:47.579 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.853539+0000", 2024-09-13T14:49:47.579 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.5299999999999998e-06 2024-09-13T14:49:47.579 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.579 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.579 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:49:47.580 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.853553+0000", 2024-09-13T14:49:47.580 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4144e-05 2024-09-13T14:49:47.580 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.580 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.580 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:49:47.580 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.853561+0000", 2024-09-13T14:49:47.580 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.9759999999999995e-06 2024-09-13T14:49:47.580 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.580 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.580 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:49:47.581 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.853628+0000", 2024-09-13T14:49:47.581 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.7219999999999997e-05 2024-09-13T14:49:47.581 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.581 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.581 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:49:47.581 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.853747+0000", 2024-09-13T14:49:47.581 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00011941400000000001 2024-09-13T14:49:47.582 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.582 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.582 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 2,4", 2024-09-13T14:49:47.582 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.853867+0000", 2024-09-13T14:49:47.582 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000119722 2024-09-13T14:49:47.582 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.582 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.582 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:49:47.582 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.857965+0000", 2024-09-13T14:49:47.582 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0040982650000000002 2024-09-13T14:49:47.582 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.583 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.583 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:49:47.583 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.876714+0000", 2024-09-13T14:49:47.583 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.018748715999999999 2024-09-13T14:49:47.583 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.583 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.583 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:49:47.583 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.876739+0000", 2024-09-13T14:49:47.583 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.4368000000000001e-05 2024-09-13T14:49:47.583 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.583 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.584 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:49:47.584 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.857882+0000", 2024-09-13T14:49:47.584 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.981143667 2024-09-13T14:49:47.584 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.584 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.584 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:49:47.584 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.857895+0000", 2024-09-13T14:49:47.584 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2941e-05 2024-09-13T14:49:47.584 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.584 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.585 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:49:47.585 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.267370+0000", 2024-09-13T14:49:47.585 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.409474772 2024-09-13T14:49:47.585 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.585 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.585 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for missing object", 2024-09-13T14:49:47.585 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.267563+0000", 2024-09-13T14:49:47.585 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000192745 2024-09-13T14:49:47.585 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.585 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.585 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:49:47.586 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.303430+0000", 2024-09-13T14:49:47.586 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.035866902999999999 2024-09-13T14:49:47.586 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.586 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.586 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:49:47.586 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.303497+0000", 2024-09-13T14:49:47.586 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.7484000000000006e-05 2024-09-13T14:49:47.586 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:49:47.586 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:49:47.586 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:49:47.587 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.587 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.587 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:133954 8.f 8:f0515611:::benchmark_data_smithi102_313275_object3367:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-13T14:49:47.587 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:47:46.853764+0000", 2024-09-13T14:49:47.587 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 120.680706614, 2024-09-13T14:49:47.587 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.4361166660000002, 2024-09-13T14:49:47.587 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:49:47.587 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:49:47.587 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "reached pg", 2024-09-13T14:49:47.587 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:49:47.587 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:49:47.587 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:49:47.588 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 133954 2024-09-13T14:49:47.588 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.588 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:49:47.588 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.588 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:49:47.588 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.853764+0000", 2024-09-13T14:49:47.588 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:49:47.588 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.588 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.588 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:49:47.589 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.853764+0000", 2024-09-13T14:49:47.589 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:49:47.589 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.589 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.589 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:49:47.589 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.853765+0000", 2024-09-13T14:49:47.589 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1000000000000001e-06 2024-09-13T14:49:47.589 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.589 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.589 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:49:47.589 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.853771+0000", 2024-09-13T14:49:47.589 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.0499999999999999e-06 2024-09-13T14:49:47.590 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.590 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.590 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:49:47.590 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.853782+0000", 2024-09-13T14:49:47.590 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1902000000000001e-05 2024-09-13T14:49:47.590 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.590 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.590 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:49:47.590 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.853790+0000", 2024-09-13T14:49:47.590 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.3209999999999999e-06 2024-09-13T14:49:47.590 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.590 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.591 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:49:47.591 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.855981+0000", 2024-09-13T14:49:47.591 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0021912099999999999 2024-09-13T14:49:47.591 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.591 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.591 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:49:47.591 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.856123+0000", 2024-09-13T14:49:47.591 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000141629 2024-09-13T14:49:47.591 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.591 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.591 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 2,4", 2024-09-13T14:49:47.591 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.856267+0000", 2024-09-13T14:49:47.592 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00014467200000000001 2024-09-13T14:49:47.592 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.592 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.592 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:49:47.592 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.870325+0000", 2024-09-13T14:49:47.592 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.014058083000000001 2024-09-13T14:49:47.592 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.592 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.592 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:49:47.592 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.886947+0000", 2024-09-13T14:49:47.592 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.016621915000000001 2024-09-13T14:49:47.593 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.593 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.593 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:49:47.593 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.886960+0000", 2024-09-13T14:49:47.593 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3108999999999999e-05 2024-09-13T14:49:47.593 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.593 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.593 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:49:47.593 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.858729+0000", 2024-09-13T14:49:47.593 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.97176854300000004 2024-09-13T14:49:47.593 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.593 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.594 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:49:47.594 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.858739+0000", 2024-09-13T14:49:47.594 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0331e-05 2024-09-13T14:49:47.594 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.594 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.594 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:49:47.594 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.263570+0000", 2024-09-13T14:49:47.594 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4048303849999999 2024-09-13T14:49:47.594 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.594 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.594 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for missing object", 2024-09-13T14:49:47.594 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.263716+0000", 2024-09-13T14:49:47.595 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00014647799999999999 2024-09-13T14:49:47.595 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.595 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.595 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:49:47.595 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.289803+0000", 2024-09-13T14:49:47.595 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.02608655 2024-09-13T14:49:47.595 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.595 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.595 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:49:47.595 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.289881+0000", 2024-09-13T14:49:47.595 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.8387999999999996e-05 2024-09-13T14:49:47.596 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:49:47.596 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:49:47.596 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:49:47.596 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.596 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.597 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:133962 8.0 8:0dc20890:::benchmark_data_smithi102_313275_object3375:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-13T14:49:47.597 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:47:46.876677+0000", 2024-09-13T14:49:47.597 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 120.65779399199999, 2024-09-13T14:49:47.597 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.3932651090000001, 2024-09-13T14:49:47.597 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:49:47.597 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:49:47.597 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:47.597 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:49:47.598 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:49:47.598 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:49:47.598 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 133962 2024-09-13T14:49:47.598 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.598 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:49:47.598 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.598 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:49:47.598 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.876677+0000", 2024-09-13T14:49:47.598 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:49:47.598 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.598 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.599 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:49:47.599 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.876677+0000", 2024-09-13T14:49:47.599 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:49:47.599 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.599 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.599 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:49:47.599 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.876678+0000", 2024-09-13T14:49:47.599 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.364e-06 2024-09-13T14:49:47.599 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.599 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.599 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:49:47.600 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.876682+0000", 2024-09-13T14:49:47.600 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.3790000000000002e-06 2024-09-13T14:49:47.600 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.600 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.600 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:49:47.600 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.876690+0000", 2024-09-13T14:49:47.600 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.8199999999999997e-06 2024-09-13T14:49:47.600 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.600 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.600 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:49:47.600 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.876696+0000", 2024-09-13T14:49:47.600 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.8569999999999997e-06 2024-09-13T14:49:47.601 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.601 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.601 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:49:47.601 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.881714+0000", 2024-09-13T14:49:47.601 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0050180390000000002 2024-09-13T14:49:47.601 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.601 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.601 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:49:47.601 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.881726+0000", 2024-09-13T14:49:47.601 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1318000000000001e-05 2024-09-13T14:49:47.601 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.602 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.602 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:49:47.602 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.860611+0000", 2024-09-13T14:49:47.602 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.97888544700000002 2024-09-13T14:49:47.602 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.602 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.602 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:49:47.602 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.860621+0000", 2024-09-13T14:49:47.602 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.9429999999999995e-06 2024-09-13T14:49:47.602 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.603 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.603 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:49:47.603 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.248331+0000", 2024-09-13T14:49:47.603 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.387709729 2024-09-13T14:49:47.603 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.603 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.603 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:49:47.603 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.248463+0000", 2024-09-13T14:49:47.603 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000132435 2024-09-13T14:49:47.603 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.603 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.604 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,4,5", 2024-09-13T14:49:47.604 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.248579+0000", 2024-09-13T14:49:47.604 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00011574999999999999 2024-09-13T14:49:47.604 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.604 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.604 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:49:47.604 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.255787+0000", 2024-09-13T14:49:47.604 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.007207761 2024-09-13T14:49:47.604 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.604 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.604 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:47.605 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.255832+0000", 2024-09-13T14:49:47.605 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.5062999999999999e-05 2024-09-13T14:49:47.605 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.605 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.605 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:47.605 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.255923+0000", 2024-09-13T14:49:47.605 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.1244999999999998e-05 2024-09-13T14:49:47.605 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.605 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.605 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:47.605 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.269728+0000", 2024-09-13T14:49:47.606 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.013804902000000001 2024-09-13T14:49:47.606 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.606 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.606 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:49:47.606 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.269831+0000", 2024-09-13T14:49:47.606 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000102543 2024-09-13T14:49:47.606 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.606 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.606 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:49:47.606 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.269942+0000", 2024-09-13T14:49:47.606 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000111514 2024-09-13T14:49:47.606 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:49:47.607 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:49:47.607 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:49:47.607 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.607 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.607 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:133964 8.b 8:d2fb3f4d:::benchmark_data_smithi102_313275_object3377:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-13T14:49:47.607 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:47:46.882828+0000", 2024-09-13T14:49:47.607 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 120.651642622, 2024-09-13T14:49:47.607 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.412555024, 2024-09-13T14:49:47.607 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:49:47.607 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:49:47.607 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:47.607 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:49:47.608 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:49:47.608 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:49:47.608 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 133964 2024-09-13T14:49:47.608 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.608 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:49:47.608 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.608 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:49:47.608 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.882828+0000", 2024-09-13T14:49:47.608 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:49:47.608 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.608 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.609 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:49:47.609 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.882828+0000", 2024-09-13T14:49:47.609 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:49:47.609 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.609 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.609 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:49:47.609 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.882830+0000", 2024-09-13T14:49:47.609 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.305e-06 2024-09-13T14:49:47.609 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.609 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.609 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:49:47.609 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.882835+0000", 2024-09-13T14:49:47.610 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.0139999999999998e-06 2024-09-13T14:49:47.610 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.610 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.610 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:49:47.610 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.882845+0000", 2024-09-13T14:49:47.610 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0066e-05 2024-09-13T14:49:47.610 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.610 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.610 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:49:47.610 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.882866+0000", 2024-09-13T14:49:47.610 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.1614e-05 2024-09-13T14:49:47.611 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.611 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.611 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:49:47.611 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.884959+0000", 2024-09-13T14:49:47.611 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0020924619999999998 2024-09-13T14:49:47.611 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.611 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.611 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:49:47.611 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.884968+0000", 2024-09-13T14:49:47.611 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.2280000000000004e-06 2024-09-13T14:49:47.611 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.612 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.612 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:49:47.612 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.857921+0000", 2024-09-13T14:49:47.612 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.97295337900000001 2024-09-13T14:49:47.612 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.612 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.612 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:49:47.612 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.857928+0000", 2024-09-13T14:49:47.612 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.0199999999999997e-06 2024-09-13T14:49:47.612 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.612 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.613 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:49:47.613 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.267894+0000", 2024-09-13T14:49:47.613 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4099659170000001 2024-09-13T14:49:47.613 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.613 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.613 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:49:47.613 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.267994+0000", 2024-09-13T14:49:47.613 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.9600999999999997e-05 2024-09-13T14:49:47.613 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.613 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.613 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,2,4", 2024-09-13T14:49:47.614 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.268120+0000", 2024-09-13T14:49:47.614 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000126246 2024-09-13T14:49:47.614 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.614 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.614 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:47.614 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.271322+0000", 2024-09-13T14:49:47.614 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0032020120000000002 2024-09-13T14:49:47.614 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.614 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.614 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:49:47.614 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.279444+0000", 2024-09-13T14:49:47.614 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.008121975 2024-09-13T14:49:47.615 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.615 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.615 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:47.615 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.279643+0000", 2024-09-13T14:49:47.615 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00019856600000000001 2024-09-13T14:49:47.615 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.615 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.615 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:47.615 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.295203+0000", 2024-09-13T14:49:47.615 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.015560305999999999 2024-09-13T14:49:47.615 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.616 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.616 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:49:47.616 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.295264+0000", 2024-09-13T14:49:47.616 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.1435999999999999e-05 2024-09-13T14:49:47.616 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.616 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.616 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:49:47.616 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.295383+0000", 2024-09-13T14:49:47.616 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00011887699999999999 2024-09-13T14:49:47.616 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:49:47.616 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:49:47.616 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:49:47.617 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.617 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.617 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:133978 8.b 8:d6e40fd1:::benchmark_data_smithi102_313275_object3391:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-13T14:49:47.617 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:47:46.901533+0000", 2024-09-13T14:49:47.617 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 120.632937732, 2024-09-13T14:49:47.617 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.3940416419999999, 2024-09-13T14:49:47.617 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:49:47.617 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:49:47.617 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:47.617 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:49:47.617 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:49:47.617 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:49:47.618 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 133978 2024-09-13T14:49:47.618 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.618 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:49:47.618 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.618 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:49:47.618 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.901533+0000", 2024-09-13T14:49:47.618 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:49:47.618 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.618 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.618 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:49:47.618 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.901533+0000", 2024-09-13T14:49:47.618 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:49:47.619 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.619 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.619 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:49:47.619 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.901535+0000", 2024-09-13T14:49:47.619 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.1909999999999999e-06 2024-09-13T14:49:47.619 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.619 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.619 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:49:47.619 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.901544+0000", 2024-09-13T14:49:47.619 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.5769999999999994e-06 2024-09-13T14:49:47.619 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.620 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.620 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:49:47.620 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.901567+0000", 2024-09-13T14:49:47.620 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.2957999999999999e-05 2024-09-13T14:49:47.620 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.620 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.620 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:49:47.620 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.901577+0000", 2024-09-13T14:49:47.620 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.6090000000000007e-06 2024-09-13T14:49:47.620 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.620 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.621 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:49:47.621 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.901648+0000", 2024-09-13T14:49:47.621 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.1589999999999997e-05 2024-09-13T14:49:47.621 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.621 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.621 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:49:47.621 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.901659+0000", 2024-09-13T14:49:47.621 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1075e-05 2024-09-13T14:49:47.621 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.621 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.621 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:49:47.622 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.857950+0000", 2024-09-13T14:49:47.622 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.95629086699999999 2024-09-13T14:49:47.622 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.622 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.622 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:49:47.622 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.857958+0000", 2024-09-13T14:49:47.622 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.5349999999999999e-06 2024-09-13T14:49:47.622 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.622 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.622 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:49:47.622 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.271350+0000", 2024-09-13T14:49:47.623 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.413392398 2024-09-13T14:49:47.623 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.623 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.623 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:49:47.623 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.271467+0000", 2024-09-13T14:49:47.623 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000117148 2024-09-13T14:49:47.623 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.623 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.623 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,2,4", 2024-09-13T14:49:47.623 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.271585+0000", 2024-09-13T14:49:47.623 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000117323 2024-09-13T14:49:47.623 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.624 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.624 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:49:47.624 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.279595+0000", 2024-09-13T14:49:47.624 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0080109780000000002 2024-09-13T14:49:47.624 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.624 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.624 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:47.624 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.279691+0000", 2024-09-13T14:49:47.624 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.5951000000000003e-05 2024-09-13T14:49:47.624 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.624 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.624 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:47.625 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.287370+0000", 2024-09-13T14:49:47.625 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0076781009999999997 2024-09-13T14:49:47.625 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.625 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.625 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:47.625 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.295424+0000", 2024-09-13T14:49:47.625 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0080549490000000005 2024-09-13T14:49:47.625 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.625 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.625 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:49:47.625 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.295468+0000", 2024-09-13T14:49:47.625 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.3951e-05 2024-09-13T14:49:47.626 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.626 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.626 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:49:47.626 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.295575+0000", 2024-09-13T14:49:47.626 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00010644100000000001 2024-09-13T14:49:47.626 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:49:47.626 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:49:47.626 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:49:47.626 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.626 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.626 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:133980 8.f 8:f0429519:::benchmark_data_smithi102_313275_object3393:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-13T14:49:47.626 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:47:46.903410+0000", 2024-09-13T14:49:47.627 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 120.631060549, 2024-09-13T14:49:47.627 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.3834898679999998, 2024-09-13T14:49:47.627 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:49:47.627 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:49:47.627 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:47.627 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:49:47.627 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:49:47.627 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:49:47.627 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 133980 2024-09-13T14:49:47.627 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.627 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:49:47.628 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.628 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:49:47.628 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.903410+0000", 2024-09-13T14:49:47.628 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:49:47.628 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.628 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.628 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:49:47.628 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.903410+0000", 2024-09-13T14:49:47.628 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:49:47.628 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.628 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.628 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:49:47.629 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.903412+0000", 2024-09-13T14:49:47.629 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.722e-06 2024-09-13T14:49:47.629 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.629 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.629 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:49:47.629 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.903419+0000", 2024-09-13T14:49:47.629 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.1590000000000004e-06 2024-09-13T14:49:47.629 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.629 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.629 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:49:47.629 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.903438+0000", 2024-09-13T14:49:47.629 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.8751000000000001e-05 2024-09-13T14:49:47.630 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.630 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.630 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:49:47.630 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.903447+0000", 2024-09-13T14:49:47.630 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.4369999999999996e-06 2024-09-13T14:49:47.630 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.630 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.630 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:49:47.630 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.903523+0000", 2024-09-13T14:49:47.630 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.5556999999999995e-05 2024-09-13T14:49:47.631 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.631 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.631 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:49:47.631 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.903534+0000", 2024-09-13T14:49:47.631 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1046000000000001e-05 2024-09-13T14:49:47.631 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.631 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.631 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:49:47.631 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859417+0000", 2024-09-13T14:49:47.631 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.95588269000000003 2024-09-13T14:49:47.631 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.631 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.632 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:49:47.632 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859425+0000", 2024-09-13T14:49:47.632 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.6910000000000003e-06 2024-09-13T14:49:47.632 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.632 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.632 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:49:47.632 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.263904+0000", 2024-09-13T14:49:47.632 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4044789719999999 2024-09-13T14:49:47.632 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.632 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.632 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:49:47.633 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.264020+0000", 2024-09-13T14:49:47.633 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00011522 2024-09-13T14:49:47.633 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.633 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.633 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,2,4", 2024-09-13T14:49:47.633 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.264139+0000", 2024-09-13T14:49:47.633 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000119253 2024-09-13T14:49:47.633 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.633 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.634 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:47.634 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.267982+0000", 2024-09-13T14:49:47.634 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0038432119999999999 2024-09-13T14:49:47.634 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.634 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.634 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:49:47.634 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.268187+0000", 2024-09-13T14:49:47.634 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000204818 2024-09-13T14:49:47.634 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.634 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.634 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:47.634 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.270572+0000", 2024-09-13T14:49:47.635 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0023851269999999999 2024-09-13T14:49:47.635 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.635 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.635 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:47.635 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.286685+0000", 2024-09-13T14:49:47.635 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.016113245000000002 2024-09-13T14:49:47.635 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.635 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.635 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:49:47.635 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.286749+0000", 2024-09-13T14:49:47.635 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.3503000000000005e-05 2024-09-13T14:49:47.635 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.636 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.636 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:49:47.636 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.286900+0000", 2024-09-13T14:49:47.636 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00015146499999999999 2024-09-13T14:49:47.636 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:49:47.636 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:49:47.636 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:49:47.636 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.636 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.636 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:133983 8.b 8:dcf739ee:::benchmark_data_smithi102_313275_object3396:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-13T14:49:47.636 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:47:46.909053+0000", 2024-09-13T14:49:47.636 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 120.625418096, 2024-09-13T14:49:47.637 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.3940629379999998, 2024-09-13T14:49:47.637 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:49:47.637 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:49:47.637 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:47.637 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:49:47.637 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:49:47.637 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:49:47.637 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 133983 2024-09-13T14:49:47.637 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.637 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:49:47.637 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.638 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:49:47.638 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.909053+0000", 2024-09-13T14:49:47.638 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:49:47.638 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.638 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.638 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:49:47.638 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.909053+0000", 2024-09-13T14:49:47.638 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:49:47.638 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.638 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.638 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:49:47.639 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.909055+0000", 2024-09-13T14:49:47.639 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.795e-06 2024-09-13T14:49:47.639 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.639 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.639 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:49:47.639 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.909061+0000", 2024-09-13T14:49:47.639 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.1210000000000003e-06 2024-09-13T14:49:47.639 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.639 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.639 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:49:47.640 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.909077+0000", 2024-09-13T14:49:47.640 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.6328999999999999e-05 2024-09-13T14:49:47.640 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.640 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.640 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:49:47.640 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.909084+0000", 2024-09-13T14:49:47.640 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.9290000000000001e-06 2024-09-13T14:49:47.640 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.640 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.640 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:49:47.640 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.909154+0000", 2024-09-13T14:49:47.641 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.0185999999999995e-05 2024-09-13T14:49:47.641 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.641 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.641 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:49:47.641 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.909168+0000", 2024-09-13T14:49:47.641 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3423e-05 2024-09-13T14:49:47.641 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.641 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.641 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:49:47.641 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.857980+0000", 2024-09-13T14:49:47.642 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.94881206799999995 2024-09-13T14:49:47.642 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.642 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.642 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:49:47.642 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.857987+0000", 2024-09-13T14:49:47.642 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.6539999999999999e-06 2024-09-13T14:49:47.642 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.642 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.642 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:49:47.642 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.275587+0000", 2024-09-13T14:49:47.642 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4175995450000001 2024-09-13T14:49:47.643 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.643 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.643 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:49:47.643 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.275781+0000", 2024-09-13T14:49:47.643 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00019366199999999999 2024-09-13T14:49:47.643 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.643 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.643 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,2,4", 2024-09-13T14:49:47.643 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.276020+0000", 2024-09-13T14:49:47.643 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000238934 2024-09-13T14:49:47.643 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.644 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.644 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:49:47.644 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.279601+0000", 2024-09-13T14:49:47.644 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0035817750000000001 2024-09-13T14:49:47.644 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.644 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.644 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:47.644 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.283529+0000", 2024-09-13T14:49:47.644 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0039279320000000003 2024-09-13T14:49:47.644 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.644 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.645 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:47.645 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.296300+0000", 2024-09-13T14:49:47.645 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.012770399999999999 2024-09-13T14:49:47.645 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.645 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.645 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:47.645 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.302946+0000", 2024-09-13T14:49:47.645 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0066464829999999999 2024-09-13T14:49:47.645 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.645 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.645 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:49:47.646 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.302999+0000", 2024-09-13T14:49:47.646 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.2719000000000001e-05 2024-09-13T14:49:47.646 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.646 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.646 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:49:47.646 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.303116+0000", 2024-09-13T14:49:47.646 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000116983 2024-09-13T14:49:47.646 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:49:47.646 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:49:47.646 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:49:47.646 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.647 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.647 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:133984 8.0 8:09c1f50d:::benchmark_data_smithi102_313275_object3397:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-13T14:49:47.647 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:47:46.913150+0000", 2024-09-13T14:49:47.647 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 120.62132123799999, 2024-09-13T14:49:47.647 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.3576139669999998, 2024-09-13T14:49:47.647 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:49:47.647 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:49:47.647 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:47.647 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:49:47.647 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:49:47.647 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:49:47.647 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 133984 2024-09-13T14:49:47.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:47.649 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:47.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:47.650 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.650 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:49:47.650 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.650 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:49:47.650 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.913150+0000", 2024-09-13T14:49:47.650 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:49:47.651 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.651 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.651 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:49:47.651 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.913150+0000", 2024-09-13T14:49:47.651 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:49:47.651 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.651 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.651 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:49:47.651 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.913151+0000", 2024-09-13T14:49:47.651 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.3200000000000003e-07 2024-09-13T14:49:47.651 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.652 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.652 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:49:47.652 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.913155+0000", 2024-09-13T14:49:47.652 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.3370000000000001e-06 2024-09-13T14:49:47.652 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.652 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.652 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:49:47.652 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.913166+0000", 2024-09-13T14:49:47.652 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1399999999999999e-05 2024-09-13T14:49:47.652 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.652 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.653 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:49:47.653 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.913172+0000", 2024-09-13T14:49:47.653 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.9109999999999998e-06 2024-09-13T14:49:47.653 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.653 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.653 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:49:47.653 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.913224+0000", 2024-09-13T14:49:47.653 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.1277999999999999e-05 2024-09-13T14:49:47.653 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.653 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.653 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:49:47.654 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.913230+0000", 2024-09-13T14:49:47.654 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.5529999999999998e-06 2024-09-13T14:49:47.654 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.654 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.654 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:49:47.654 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.860646+0000", 2024-09-13T14:49:47.654 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.94741563299999998 2024-09-13T14:49:47.654 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.654 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.654 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:49:47.654 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.860653+0000", 2024-09-13T14:49:47.654 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.2150000000000004e-06 2024-09-13T14:49:47.655 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.655 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.655 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:49:47.655 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.252136+0000", 2024-09-13T14:49:47.655 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.39148323 2024-09-13T14:49:47.655 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.655 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.655 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:49:47.655 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.252259+0000", 2024-09-13T14:49:47.655 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00012318999999999999 2024-09-13T14:49:47.655 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.655 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.656 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,4,5", 2024-09-13T14:49:47.656 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.252375+0000", 2024-09-13T14:49:47.656 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000115986 2024-09-13T14:49:47.656 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.656 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.656 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:47.656 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.255884+0000", 2024-09-13T14:49:47.656 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.003508653 2024-09-13T14:49:47.656 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.656 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.656 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:49:47.656 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.255936+0000", 2024-09-13T14:49:47.657 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.2435999999999997e-05 2024-09-13T14:49:47.657 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.657 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.657 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:47.657 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.256005+0000", 2024-09-13T14:49:47.657 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.8221999999999998e-05 2024-09-13T14:49:47.657 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.657 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.657 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:47.657 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.270574+0000", 2024-09-13T14:49:47.657 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.014568965999999999 2024-09-13T14:49:47.658 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.658 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.658 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:49:47.658 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.270641+0000", 2024-09-13T14:49:47.658 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.7502999999999994e-05 2024-09-13T14:49:47.658 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.658 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.658 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:49:47.658 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.270764+0000", 2024-09-13T14:49:47.658 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00012252199999999999 2024-09-13T14:49:47.658 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:49:47.658 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:49:47.659 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:49:47.659 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.659 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.659 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:190172 8.f 8:f7b5cc0b:::benchmark_data_smithi102_313275_object59585:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-13T14:49:47.659 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:48:56.317141+0000", 2024-09-13T14:49:47.659 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 51.217329702000001, 2024-09-13T14:49:47.659 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.14350576100000001, 2024-09-13T14:49:47.659 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:49:47.659 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:49:47.659 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:47.659 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:49:47.659 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:49:47.660 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:49:47.660 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 190172 2024-09-13T14:49:47.660 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.660 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:49:47.660 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.660 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:49:47.660 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:56.317141+0000", 2024-09-13T14:49:47.660 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:49:47.660 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.660 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.660 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:49:47.660 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:56.317141+0000", 2024-09-13T14:49:47.661 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:49:47.661 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.661 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.661 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:49:47.661 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:56.317142+0000", 2024-09-13T14:49:47.661 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.2199999999999999e-07 2024-09-13T14:49:47.661 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.661 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.661 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:49:47.661 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:56.317144+0000", 2024-09-13T14:49:47.661 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.2259999999999999e-06 2024-09-13T14:49:47.661 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.662 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.662 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:49:47.662 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:56.317150+0000", 2024-09-13T14:49:47.662 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.3350000000000003e-06 2024-09-13T14:49:47.662 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.662 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.662 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:49:47.662 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:56.317155+0000", 2024-09-13T14:49:47.662 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.6500000000000004e-06 2024-09-13T14:49:47.662 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.662 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.662 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:49:47.663 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:56.328708+0000", 2024-09-13T14:49:47.663 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.011553308 2024-09-13T14:49:47.663 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.663 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.663 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:49:47.663 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:56.328826+0000", 2024-09-13T14:49:47.663 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000118068 2024-09-13T14:49:47.663 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.663 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.663 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,2,4", 2024-09-13T14:49:47.663 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:56.329083+0000", 2024-09-13T14:49:47.663 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00025675999999999997 2024-09-13T14:49:47.664 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.664 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.664 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:49:47.664 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:56.337321+0000", 2024-09-13T14:49:47.664 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0082382440000000005 2024-09-13T14:49:47.664 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.664 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.664 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:47.664 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:56.337362+0000", 2024-09-13T14:49:47.665 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.0618999999999999e-05 2024-09-13T14:49:47.665 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.665 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.665 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:47.665 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:56.338064+0000", 2024-09-13T14:49:47.665 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00070170400000000004 2024-09-13T14:49:47.665 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.665 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.665 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:47.665 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:56.460027+0000", 2024-09-13T14:49:47.665 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.121963659 2024-09-13T14:49:47.666 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.666 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.666 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:49:47.666 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:56.460302+0000", 2024-09-13T14:49:47.666 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00027505999999999999 2024-09-13T14:49:47.666 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.666 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.666 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:49:47.666 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:48:56.460647+0000", 2024-09-13T14:49:47.666 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00034470600000000002 2024-09-13T14:49:47.666 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:49:47.666 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:49:47.667 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:49:47.667 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.667 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.667 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:207722 8.b 8:d960ebc7:::benchmark_data_smithi102_313275_object77135:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e350)", 2024-09-13T14:49:47.667 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:49:18.937623+0000", 2024-09-13T14:49:47.667 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 28.596847734000001, 2024-09-13T14:49:47.667 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.28795287800000002, 2024-09-13T14:49:47.667 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:49:47.667 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:49:47.667 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:47.667 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:49:47.667 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:49:47.668 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:49:47.668 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 207722 2024-09-13T14:49:47.668 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.668 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:49:47.668 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.668 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:49:47.668 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.937623+0000", 2024-09-13T14:49:47.668 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:49:47.668 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.668 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.668 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:49:47.669 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.937623+0000", 2024-09-13T14:49:47.669 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:49:47.669 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.669 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.669 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:49:47.669 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.937625+0000", 2024-09-13T14:49:47.669 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.483e-06 2024-09-13T14:49:47.669 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.669 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.669 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:49:47.669 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.937630+0000", 2024-09-13T14:49:47.669 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.1279999999999999e-06 2024-09-13T14:49:47.670 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.670 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.670 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:49:47.670 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.937642+0000", 2024-09-13T14:49:47.670 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2407e-05 2024-09-13T14:49:47.670 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.670 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.670 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:49:47.670 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.937649+0000", 2024-09-13T14:49:47.670 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.9e-06 2024-09-13T14:49:47.670 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.670 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.671 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:49:47.671 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.937858+0000", 2024-09-13T14:49:47.671 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000209271 2024-09-13T14:49:47.671 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.671 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.671 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:49:47.671 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.938068+0000", 2024-09-13T14:49:47.671 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00020926399999999999 2024-09-13T14:49:47.671 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.671 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.671 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,2,4", 2024-09-13T14:49:47.671 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.938191+0000", 2024-09-13T14:49:47.671 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00012364900000000001 2024-09-13T14:49:47.672 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.672 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.672 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:49:47.672 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.942048+0000", 2024-09-13T14:49:47.672 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0038561770000000001 2024-09-13T14:49:47.672 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.672 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.672 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:47.672 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.943131+0000", 2024-09-13T14:49:47.672 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0010838270000000001 2024-09-13T14:49:47.673 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.673 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.673 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:47.673 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.943171+0000", 2024-09-13T14:49:47.673 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.0101000000000001e-05 2024-09-13T14:49:47.673 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.673 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.673 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:47.673 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:19.225389+0000", 2024-09-13T14:49:47.673 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.28221763900000002 2024-09-13T14:49:47.673 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.673 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.674 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:49:47.674 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:19.225453+0000", 2024-09-13T14:49:47.674 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.3874000000000005e-05 2024-09-13T14:49:47.674 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.674 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.674 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:49:47.674 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:19.225576+0000", 2024-09-13T14:49:47.674 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000123158 2024-09-13T14:49:47.674 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:49:47.674 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:49:47.674 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:49:47.675 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.675 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.675 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:207739 8.b 8:da1880e1:::benchmark_data_smithi102_313275_object77152:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e350)", 2024-09-13T14:49:47.675 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:49:18.961134+0000", 2024-09-13T14:49:47.675 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 28.573336769000001, 2024-09-13T14:49:47.675 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.27795650500000002, 2024-09-13T14:49:47.675 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:49:47.675 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:49:47.675 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:47.675 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:49:47.675 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:49:47.676 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:49:47.676 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 207739 2024-09-13T14:49:47.676 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.676 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:49:47.676 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.676 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:49:47.676 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.961134+0000", 2024-09-13T14:49:47.676 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:49:47.676 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.676 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.676 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:49:47.677 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.961134+0000", 2024-09-13T14:49:47.677 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:49:47.677 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.677 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.677 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:49:47.677 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.961136+0000", 2024-09-13T14:49:47.677 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.305e-06 2024-09-13T14:49:47.677 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.677 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.677 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:49:47.677 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.961138+0000", 2024-09-13T14:49:47.677 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.9840000000000001e-06 2024-09-13T14:49:47.678 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.678 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.678 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:49:47.678 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.961146+0000", 2024-09-13T14:49:47.678 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.1300000000000003e-06 2024-09-13T14:49:47.678 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.678 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.678 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:49:47.678 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.961152+0000", 2024-09-13T14:49:47.678 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.7870000000000004e-06 2024-09-13T14:49:47.678 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.679 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.679 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:49:47.679 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.961506+0000", 2024-09-13T14:49:47.679 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000353178 2024-09-13T14:49:47.679 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.679 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.679 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:49:47.679 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.961630+0000", 2024-09-13T14:49:47.679 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000124117 2024-09-13T14:49:47.679 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.679 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.679 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,2,4", 2024-09-13T14:49:47.679 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.961752+0000", 2024-09-13T14:49:47.680 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00012265099999999999 2024-09-13T14:49:47.680 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.680 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.680 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:49:47.680 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.965487+0000", 2024-09-13T14:49:47.680 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0037349560000000002 2024-09-13T14:49:47.680 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.680 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.680 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:47.680 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.965571+0000", 2024-09-13T14:49:47.680 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.3696999999999995e-05 2024-09-13T14:49:47.681 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.681 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.681 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:47.681 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.965702+0000", 2024-09-13T14:49:47.681 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000131018 2024-09-13T14:49:47.681 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.681 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.681 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:47.682 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:19.238906+0000", 2024-09-13T14:49:47.682 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.27320436300000001 2024-09-13T14:49:47.682 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.682 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.682 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:49:47.682 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:19.238965+0000", 2024-09-13T14:49:47.682 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.8947999999999999e-05 2024-09-13T14:49:47.682 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.682 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.682 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:49:47.682 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:19.239091+0000", 2024-09-13T14:49:47.682 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00012537099999999999 2024-09-13T14:49:47.682 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:49:47.683 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:49:47.683 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:49:47.683 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.683 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.683 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:207746 8.b 8:d822562c:::benchmark_data_smithi102_313275_object77159:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e350)", 2024-09-13T14:49:47.683 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:49:18.967458+0000", 2024-09-13T14:49:47.683 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 28.567013399, 2024-09-13T14:49:47.683 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.27182446599999999, 2024-09-13T14:49:47.683 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:49:47.683 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:49:47.683 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:47.683 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:49:47.684 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:49:47.684 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:49:47.684 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 207746 2024-09-13T14:49:47.684 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.684 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:49:47.684 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.684 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:49:47.684 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.967458+0000", 2024-09-13T14:49:47.684 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:49:47.684 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.684 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.685 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:49:47.685 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.967458+0000", 2024-09-13T14:49:47.685 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:49:47.685 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.685 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.685 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:49:47.685 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.967459+0000", 2024-09-13T14:49:47.685 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0589999999999999e-06 2024-09-13T14:49:47.685 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.685 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.685 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:49:47.685 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.967463+0000", 2024-09-13T14:49:47.686 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.1919999999999998e-06 2024-09-13T14:49:47.686 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.686 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.686 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:49:47.686 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.967473+0000", 2024-09-13T14:49:47.686 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0000000000000001e-05 2024-09-13T14:49:47.686 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.686 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.686 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:49:47.686 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.967478+0000", 2024-09-13T14:49:47.686 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.8210000000000001e-06 2024-09-13T14:49:47.686 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.687 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.687 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:49:47.687 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.967527+0000", 2024-09-13T14:49:47.687 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.8865000000000001e-05 2024-09-13T14:49:47.687 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.687 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.687 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:49:47.687 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.967603+0000", 2024-09-13T14:49:47.687 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.6511e-05 2024-09-13T14:49:47.687 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.687 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.687 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,2,4", 2024-09-13T14:49:47.687 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.967679+0000", 2024-09-13T14:49:47.688 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.6205999999999995e-05 2024-09-13T14:49:47.688 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.688 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.688 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:47.688 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.971174+0000", 2024-09-13T14:49:47.688 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0034949339999999999 2024-09-13T14:49:47.688 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.688 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.688 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:49:47.688 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.971260+0000", 2024-09-13T14:49:47.688 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.5541e-05 2024-09-13T14:49:47.689 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.689 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.689 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:47.689 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.971377+0000", 2024-09-13T14:49:47.689 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000117087 2024-09-13T14:49:47.689 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.689 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.689 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:47.689 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:19.239131+0000", 2024-09-13T14:49:47.689 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.26775398700000003 2024-09-13T14:49:47.689 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.690 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.690 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:49:47.690 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:19.239179+0000", 2024-09-13T14:49:47.690 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.7877999999999998e-05 2024-09-13T14:49:47.690 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.690 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.690 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:49:47.690 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:19.239282+0000", 2024-09-13T14:49:47.690 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000103385 2024-09-13T14:49:47.690 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:49:47.690 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:49:47.691 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:49:47.691 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.691 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.691 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:207752 8.b 8:d89d313a:::benchmark_data_smithi102_313275_object77165:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e350)", 2024-09-13T14:49:47.691 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:49:18.980188+0000", 2024-09-13T14:49:47.691 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 28.554283054999999, 2024-09-13T14:49:47.691 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.259614699, 2024-09-13T14:49:47.691 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:49:47.691 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:49:47.691 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:47.691 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:49:47.691 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:49:47.692 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:49:47.692 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 207752 2024-09-13T14:49:47.692 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.692 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:49:47.692 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.692 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:49:47.692 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.980188+0000", 2024-09-13T14:49:47.692 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:49:47.692 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.692 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.692 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:49:47.693 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.980188+0000", 2024-09-13T14:49:47.693 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:49:47.693 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.693 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.693 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:49:47.693 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.980189+0000", 2024-09-13T14:49:47.693 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3880000000000001e-06 2024-09-13T14:49:47.693 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.693 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.693 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:49:47.693 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.980195+0000", 2024-09-13T14:49:47.693 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.1670000000000004e-06 2024-09-13T14:49:47.694 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.694 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.694 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:49:47.694 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.980206+0000", 2024-09-13T14:49:47.694 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0468999999999999e-05 2024-09-13T14:49:47.694 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.694 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.694 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:49:47.694 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.980212+0000", 2024-09-13T14:49:47.694 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.8250000000000001e-06 2024-09-13T14:49:47.694 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.694 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.695 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:49:47.695 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.980278+0000", 2024-09-13T14:49:47.695 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.5840000000000007e-05 2024-09-13T14:49:47.695 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.695 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.695 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:49:47.695 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.980423+0000", 2024-09-13T14:49:47.695 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000145205 2024-09-13T14:49:47.695 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.695 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.696 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,2,4", 2024-09-13T14:49:47.696 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.980571+0000", 2024-09-13T14:49:47.696 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00014773799999999999 2024-09-13T14:49:47.696 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.696 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.696 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:47.696 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.983665+0000", 2024-09-13T14:49:47.696 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0030945209999999998 2024-09-13T14:49:47.696 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.696 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.696 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:49:47.696 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.983683+0000", 2024-09-13T14:49:47.697 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.8399000000000001e-05 2024-09-13T14:49:47.697 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.697 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.697 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:47.697 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.984129+0000", 2024-09-13T14:49:47.697 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00044559100000000001 2024-09-13T14:49:47.697 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.697 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.697 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:47.697 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:19.239643+0000", 2024-09-13T14:49:47.697 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.25551406700000001 2024-09-13T14:49:47.698 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.698 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.698 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:49:47.698 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:19.239699+0000", 2024-09-13T14:49:47.698 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.5659999999999999e-05 2024-09-13T14:49:47.698 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.698 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.698 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:49:47.698 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:19.239803+0000", 2024-09-13T14:49:47.698 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000103829 2024-09-13T14:49:47.698 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:49:47.699 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:49:47.699 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:49:47.699 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.699 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.699 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:226343 8.b 8:d626b72e:::benchmark_data_smithi102_313275_object95756:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-09-13T14:49:47.699 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:49:42.869990+0000", 2024-09-13T14:49:47.699 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 4.6644809939999998, 2024-09-13T14:49:47.699 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.44102997500000002, 2024-09-13T14:49:47.699 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:49:47.699 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:49:47.699 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:47.700 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:49:47.700 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:49:47.700 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:47.701 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:49:47.701 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 226343 2024-09-13T14:49:47.701 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.701 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:49:47.701 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.701 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:49:47.701 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.869990+0000", 2024-09-13T14:49:47.701 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:49:47.701 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.701 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.701 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:49:47.702 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.869990+0000", 2024-09-13T14:49:47.702 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:49:47.702 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.702 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.702 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:49:47.702 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.869991+0000", 2024-09-13T14:49:47.702 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.8800000000000001e-07 2024-09-13T14:49:47.702 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.702 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.702 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:49:47.703 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.869994+0000", 2024-09-13T14:49:47.703 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.4060000000000001e-06 2024-09-13T14:49:47.703 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.703 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.703 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:49:47.703 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.870003+0000", 2024-09-13T14:49:47.703 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.8939999999999999e-06 2024-09-13T14:49:47.703 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.703 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.703 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:49:47.703 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.870008+0000", 2024-09-13T14:49:47.704 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.9899999999999997e-06 2024-09-13T14:49:47.704 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.704 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.704 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:49:47.704 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.875941+0000", 2024-09-13T14:49:47.704 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0059329910000000003 2024-09-13T14:49:47.704 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.704 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.704 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:49:47.704 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.876064+0000", 2024-09-13T14:49:47.705 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00012314499999999999 2024-09-13T14:49:47.705 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.705 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.705 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,2,4", 2024-09-13T14:49:47.705 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.876201+0000", 2024-09-13T14:49:47.705 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00013661200000000001 2024-09-13T14:49:47.705 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.705 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.705 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:49:47.705 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.879974+0000", 2024-09-13T14:49:47.705 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00377357 2024-09-13T14:49:47.705 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.706 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.706 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:47.706 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.880284+0000", 2024-09-13T14:49:47.706 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00030931200000000001 2024-09-13T14:49:47.706 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.706 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.706 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:47.706 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.880582+0000", 2024-09-13T14:49:47.706 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00029824400000000003 2024-09-13T14:49:47.706 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.707 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.707 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:47.707 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.310836+0000", 2024-09-13T14:49:47.707 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.43025402000000001 2024-09-13T14:49:47.707 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.707 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.707 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:49:47.707 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.310909+0000", 2024-09-13T14:49:47.707 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.2825999999999997e-05 2024-09-13T14:49:47.707 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.707 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.707 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:49:47.708 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.311020+0000", 2024-09-13T14:49:47.708 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00011107699999999999 2024-09-13T14:49:47.708 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:49:47.708 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:49:47.708 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:49:47.708 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.708 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.708 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:226363 8.f 8:f9e9d9b3:::benchmark_data_smithi102_313275_object95776:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-09-13T14:49:47.708 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:49:42.906221+0000", 2024-09-13T14:49:47.708 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 4.6282499990000003, 2024-09-13T14:49:47.708 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.39981434999999999, 2024-09-13T14:49:47.709 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:49:47.709 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:49:47.709 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:47.709 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:49:47.709 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:49:47.709 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:49:47.709 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 226363 2024-09-13T14:49:47.709 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.709 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:49:47.709 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.709 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:49:47.710 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.906221+0000", 2024-09-13T14:49:47.710 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:49:47.710 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.710 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.710 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:49:47.710 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.906221+0000", 2024-09-13T14:49:47.710 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:49:47.710 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.710 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.710 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:49:47.710 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.906222+0000", 2024-09-13T14:49:47.710 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4929999999999999e-06 2024-09-13T14:49:47.711 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.712 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.712 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:49:47.712 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.906229+0000", 2024-09-13T14:49:47.712 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.7569999999999998e-06 2024-09-13T14:49:47.712 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.712 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.712 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:49:47.712 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.906245+0000", 2024-09-13T14:49:47.713 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.5761999999999999e-05 2024-09-13T14:49:47.713 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.713 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.713 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:49:47.713 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.906252+0000", 2024-09-13T14:49:47.713 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.1709999999999996e-06 2024-09-13T14:49:47.713 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.713 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.713 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:49:47.713 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.906322+0000", 2024-09-13T14:49:47.713 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.0079000000000004e-05 2024-09-13T14:49:47.714 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.714 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.714 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:49:47.714 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.906424+0000", 2024-09-13T14:49:47.714 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000101358 2024-09-13T14:49:47.714 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.714 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.714 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,2,4", 2024-09-13T14:49:47.714 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.906514+0000", 2024-09-13T14:49:47.714 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.0093999999999995e-05 2024-09-13T14:49:47.714 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.714 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.715 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:47.715 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.914894+0000", 2024-09-13T14:49:47.715 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0083803369999999999 2024-09-13T14:49:47.715 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.715 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.715 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:47.715 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.914929+0000", 2024-09-13T14:49:47.715 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.451e-05 2024-09-13T14:49:47.715 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.715 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.715 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:49:47.715 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.914938+0000", 2024-09-13T14:49:47.715 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.2099999999999999e-06 2024-09-13T14:49:47.716 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.716 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.716 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:47.716 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.305813+0000", 2024-09-13T14:49:47.716 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.39087482600000001 2024-09-13T14:49:47.716 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.716 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.716 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:49:47.716 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.305920+0000", 2024-09-13T14:49:47.716 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00010788700000000001 2024-09-13T14:49:47.716 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.717 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.717 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:49:47.717 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.306035+0000", 2024-09-13T14:49:47.717 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00011486599999999999 2024-09-13T14:49:47.717 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:49:47.717 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:49:47.717 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:49:47.717 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.717 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.717 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:226364 8.f 8:f43d0b70:::benchmark_data_smithi102_313275_object95777:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-09-13T14:49:47.717 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:49:42.907034+0000", 2024-09-13T14:49:47.717 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 4.6274370610000002, 2024-09-13T14:49:47.718 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.39922270900000001, 2024-09-13T14:49:47.718 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:49:47.718 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:49:47.718 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:49:47.718 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:49:47.718 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:49:47.718 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:49:47.718 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 226364 2024-09-13T14:49:47.718 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.718 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:49:47.718 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.719 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:49:47.719 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.907034+0000", 2024-09-13T14:49:47.719 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:49:47.719 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.719 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.719 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:49:47.719 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.907034+0000", 2024-09-13T14:49:47.719 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:49:47.719 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.719 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.719 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:49:47.719 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.907035+0000", 2024-09-13T14:49:47.720 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.457e-06 2024-09-13T14:49:47.720 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.720 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.720 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:49:47.720 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.907041+0000", 2024-09-13T14:49:47.720 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.5380000000000002e-06 2024-09-13T14:49:47.720 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.720 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.720 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:49:47.720 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.907053+0000", 2024-09-13T14:49:47.720 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2488e-05 2024-09-13T14:49:47.721 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.721 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.721 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:49:47.721 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.907062+0000", 2024-09-13T14:49:47.721 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.6549999999999993e-06 2024-09-13T14:49:47.721 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.721 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.721 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:49:47.721 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.911463+0000", 2024-09-13T14:49:47.721 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0044006729999999999 2024-09-13T14:49:47.721 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.721 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.722 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:49:47.722 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.911585+0000", 2024-09-13T14:49:47.722 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000121802 2024-09-13T14:49:47.722 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.722 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.722 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,2,4", 2024-09-13T14:49:47.722 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.911685+0000", 2024-09-13T14:49:47.722 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000100517 2024-09-13T14:49:47.722 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.722 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.722 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:49:47.722 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.914942+0000", 2024-09-13T14:49:47.723 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0032566980000000001 2024-09-13T14:49:47.723 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.723 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.723 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:47.723 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.914970+0000", 2024-09-13T14:49:47.723 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.8017999999999998e-05 2024-09-13T14:49:47.723 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.723 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.723 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:47.723 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.931986+0000", 2024-09-13T14:49:47.723 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.017015966 2024-09-13T14:49:47.723 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.724 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.724 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:47.724 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.306081+0000", 2024-09-13T14:49:47.724 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.37409518000000003 2024-09-13T14:49:47.724 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.724 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.724 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:49:47.724 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.306162+0000", 2024-09-13T14:49:47.724 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.0871999999999994e-05 2024-09-13T14:49:47.724 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:49:47.725 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:49:47.725 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:49:47.725 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.306257+0000", 2024-09-13T14:49:47.725 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.4845000000000004e-05 2024-09-13T14:49:47.725 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:49:47.725 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:49:47.725 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:49:47.725 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:49:47.725 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:49:47.725 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:49:47.726 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:49:47.750 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:47.776 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:47.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:47.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:47.878 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:47.902 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:47.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:47.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:48.004 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:48.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:48.064 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:48.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:48.165 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:48.190 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:48.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:48.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:48.292 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:48.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:48.328 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:48.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:48.429 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:48.463 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:48.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:48.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:48.565 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:48.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:48.598 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:48.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:48.699 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:48.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:48.731 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:48.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:48.832 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:48.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:48.874 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:48.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:48.975 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:49.001 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:49.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:49.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:49.103 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:49.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:49.158 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:49.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:49.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:48 smithi195 ceph-mon[104262]: pgmap v1239: 109 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 104 active+clean; 2.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 695 op/s; 9833/100260 objects misplaced (9.808%); 7.6 MiB/s, 121 objects/s recovering 2024-09-13T14:49:49.260 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:49.295 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:49.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:49.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:49.397 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:49.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:48 smithi102 ceph-mon[110183]: pgmap v1239: 109 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 104 active+clean; 2.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 695 op/s; 9833/100260 objects misplaced (9.808%); 7.6 MiB/s, 121 objects/s recovering 2024-09-13T14:49:49.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:48 smithi102 ceph-mon[115269]: pgmap v1239: 109 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 104 active+clean; 2.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 695 op/s; 9833/100260 objects misplaced (9.808%); 7.6 MiB/s, 121 objects/s recovering 2024-09-13T14:49:49.427 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:49.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:49.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:49.529 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:49.556 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:49.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:49.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:49.658 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:49.697 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:49.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:49.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:49.799 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:49.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:49.827 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:49.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:49.928 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:49.955 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:49.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:49.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:50.057 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:50.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:50.094 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:50.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:50.195 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:50.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:50.233 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:50.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:50.334 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:50.379 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:50.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:50.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:50.481 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:50.514 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:50.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:50.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:50.616 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:50.653 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:50.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:50.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:50.755 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:50.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:50.776 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:50.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:50.877 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:50.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:50.917 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:50.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:51.018 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:51.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:51.050 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:51.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:51.151 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:51.184 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:51.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:51.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:51.228 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:50 smithi102 ceph-mon[110183]: pgmap v1240: 109 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 104 active+clean; 1.9 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 828 op/s; 9120/94662 objects misplaced (9.634%); 6.7 MiB/s, 107 objects/s recovering 2024-09-13T14:49:51.229 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:50 smithi102 ceph-mon[115269]: pgmap v1240: 109 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 104 active+clean; 1.9 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 828 op/s; 9120/94662 objects misplaced (9.634%); 6.7 MiB/s, 107 objects/s recovering 2024-09-13T14:49:51.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:50 smithi195 ceph-mon[104262]: pgmap v1240: 109 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 104 active+clean; 1.9 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 828 op/s; 9120/94662 objects misplaced (9.634%); 6.7 MiB/s, 107 objects/s recovering 2024-09-13T14:49:51.286 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:51.317 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:51.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:51.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:51.381 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T14:49:51.419 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:51.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:51.457 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:51.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:51.559 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:51.606 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:51.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:51.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:51.708 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:51.717 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:49:51.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:51.749 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:51.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:51.791 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:51.851 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:51.896 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:51.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:51.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:51.998 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:52.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:52.023 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:52.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:52.124 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:52.162 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:52.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:52.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:52.263 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:52.307 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:52.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:52.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:52.409 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:52.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:52.438 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:52.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:52.540 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:52.566 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:52.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:52.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:52.668 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:52.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:52.715 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:52.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:52.816 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:52.853 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:52.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:52.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:52.955 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:52.976 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:52.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:52.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:53.023 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is unset 2024-09-13T14:49:53.041 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:52 smithi102 ceph-mon[110183]: pgmap v1241: 109 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 104 active+clean; 1.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 749 op/s; 9120/90033 objects misplaced (10.130%); 5.9 MiB/s, 95 objects/s recovering 2024-09-13T14:49:53.041 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:52 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3083121572' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:49:53.041 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:52 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:49:53.041 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:52 smithi102 ceph-mon[115269]: pgmap v1241: 109 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 104 active+clean; 1.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 749 op/s; 9120/90033 objects misplaced (10.130%); 5.9 MiB/s, 95 objects/s recovering 2024-09-13T14:49:53.042 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:52 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3083121572' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:49:53.042 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:52 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:49:53.077 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:53.121 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:53.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:53.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:53.223 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:53.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:52 smithi195 ceph-mon[104262]: pgmap v1241: 109 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 104 active+clean; 1.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 749 op/s; 9120/90033 objects misplaced (10.130%); 5.9 MiB/s, 95 objects/s recovering 2024-09-13T14:49:53.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:53 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3083121572' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:49:53.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:53 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:49:53.259 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:53.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:53.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:53.361 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:53.396 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:53.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:53.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:53.498 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:53.530 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:53.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:53.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:53.631 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:53.655 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:53.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:53.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:53.757 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:53.778 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:53.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:53.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:53.880 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:53.919 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:53.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:53.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:54.021 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:54.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:54.062 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:54.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:54.163 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:54.186 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:54.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:54.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:54.217 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:53 smithi195 ceph-mon[104262]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:49:54.217 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:53 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:49:54.218 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:53 smithi195 ceph-mon[104262]: osdmap e356: 8 total, 8 up, 8 in 2024-09-13T14:49:54.288 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:54.310 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:54.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:54.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:54.392 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:53 smithi102 ceph-mon[110183]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:49:54.393 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:53 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:49:54.393 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:53 smithi102 ceph-mon[110183]: osdmap e356: 8 total, 8 up, 8 in 2024-09-13T14:49:54.393 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:53 smithi102 ceph-mon[115269]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:49:54.393 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:53 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:49:54.393 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:53 smithi102 ceph-mon[115269]: osdmap e356: 8 total, 8 up, 8 in 2024-09-13T14:49:54.413 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:54.439 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:54.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:54.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:54.541 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:54.571 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:54.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:54.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:54.672 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:54.700 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:54.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:54.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:54.804 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:54.837 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:54.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:54.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:54.939 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:54.980 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:54.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:54.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:55.082 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:55.117 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:55.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:55.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:55.219 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:55.258 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:55.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:55.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:55.300 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:55 smithi195 ceph-mon[104262]: pgmap v1243: 109 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 104 active+clean; 1.7 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 783 op/s; 7561/83100 objects misplaced (9.099%); 4.1 MiB/s, 66 objects/s recovering 2024-09-13T14:49:55.360 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:55.402 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:55.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:55.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:55.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:55 smithi102 ceph-mon[110183]: pgmap v1243: 109 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 104 active+clean; 1.7 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 783 op/s; 7561/83100 objects misplaced (9.099%); 4.1 MiB/s, 66 objects/s recovering 2024-09-13T14:49:55.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:55 smithi102 ceph-mon[115269]: pgmap v1243: 109 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 104 active+clean; 1.7 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 783 op/s; 7561/83100 objects misplaced (9.099%); 4.1 MiB/s, 66 objects/s recovering 2024-09-13T14:49:55.504 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:55.544 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:55.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:55.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:55.646 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:55.685 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:55.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:55.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:55.735 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:49:55.736 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-13T14:49:55.736 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:49:55.736 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.46157.0:236369 8.2 8:491a7ed9:::benchmark_data_smithi102_313275_object105782:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e356)", 2024-09-13T14:49:55.736 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:49:55.728531+0000", 2024-09-13T14:49:55.736 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.0054260080000000004, 2024-09-13T14:49:55.736 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0054557390000000002, 2024-09-13T14:49:55.736 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:49:55.736 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:49:55.736 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:49:55.736 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.46157", 2024-09-13T14:49:55.737 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:49:55.737 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 236369 2024-09-13T14:49:55.737 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:49:55.737 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:49:55.737 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:49:55.737 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:49:55.737 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:55.728531+0000", 2024-09-13T14:49:55.737 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:49:55.737 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:49:55.737 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:49:55.737 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:49:55.737 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:55.728531+0000", 2024-09-13T14:49:55.738 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:49:55.738 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:49:55.738 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:49:55.738 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:49:55.738 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:55.728529+0000", 2024-09-13T14:49:55.738 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:49:55.738 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:49:55.738 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:49:55.738 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:49:55.738 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:55.728540+0000", 2024-09-13T14:49:55.738 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.08e-05 2024-09-13T14:49:55.738 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:49:55.738 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:49:55.739 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:49:55.739 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:55.728554+0000", 2024-09-13T14:49:55.739 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3672000000000001e-05 2024-09-13T14:49:55.739 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:49:55.739 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:49:55.739 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:49:55.739 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:55.728562+0000", 2024-09-13T14:49:55.739 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1140000000000006e-06 2024-09-13T14:49:55.739 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:49:55.739 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:49:55.739 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:49:55.739 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:55.728606+0000", 2024-09-13T14:49:55.739 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3676e-05 2024-09-13T14:49:55.740 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:49:55.740 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:49:55.740 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:49:55.740 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:55.728718+0000", 2024-09-13T14:49:55.740 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011219500000000001 2024-09-13T14:49:55.740 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:49:55.740 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:49:55.740 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:49:55.740 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:55.728826+0000", 2024-09-13T14:49:55.740 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010856799999999999 2024-09-13T14:49:55.740 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:49:55.740 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:49:55.741 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:49:55.741 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:55.732209+0000", 2024-09-13T14:49:55.741 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0033828529999999999 2024-09-13T14:49:55.741 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:49:55.741 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:49:55.741 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:55.741 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:55.733261+0000", 2024-09-13T14:49:55.741 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0010513569999999999 2024-09-13T14:49:55.741 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:49:55.741 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:49:55.741 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:49:55.741 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:49:55.742 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:49:55.742 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.46157.0:236373 8.f e356/337 8:fbcd4d0e:::benchmark_data_smithi102_313275_object105786:head v 356'14779, mlcod=356'14777)", 2024-09-13T14:49:55.742 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:49:55.732446+0000", 2024-09-13T14:49:55.742 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.0015111879999999999, 2024-09-13T14:49:55.742 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0016318089999999999, 2024-09-13T14:49:55.742 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:49:55.742 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T14:49:55.742 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:49:55.742 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:49:55.742 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:49:55.742 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:55.732446+0000", 2024-09-13T14:49:55.742 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:49:55.742 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:49:55.743 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:49:55.743 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:49:55.743 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:55.732446+0000", 2024-09-13T14:49:55.743 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:49:55.743 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:49:55.743 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:49:55.743 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:49:55.743 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:55.732445+0000", 2024-09-13T14:49:55.743 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:49:55.743 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:49:55.743 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:49:55.744 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:49:55.744 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:55.732453+0000", 2024-09-13T14:49:55.744 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3189999999999995e-06 2024-09-13T14:49:55.744 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:49:55.744 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:49:55.744 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:49:55.744 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:55.732466+0000", 2024-09-13T14:49:55.744 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3495e-05 2024-09-13T14:49:55.744 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:49:55.744 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:49:55.745 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:49:55.745 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:55.732474+0000", 2024-09-13T14:49:55.745 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1670000000000002e-06 2024-09-13T14:49:55.745 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:49:55.745 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:49:55.745 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:49:55.745 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:55.732542+0000", 2024-09-13T14:49:55.745 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7978999999999994e-05 2024-09-13T14:49:55.745 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:49:55.745 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:49:55.745 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:49:55.745 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:55.732567+0000", 2024-09-13T14:49:55.745 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5236e-05 2024-09-13T14:49:55.746 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:49:55.746 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:49:55.746 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:49:55.746 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:49:55.746 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:49:55.746 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.46157.0:236366 8.2 8:4c0a583f:::benchmark_data_smithi102_313275_object105779:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e356)", 2024-09-13T14:49:55.746 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:49:55.724301+0000", 2024-09-13T14:49:55.746 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.0096556090000000008, 2024-09-13T14:49:55.746 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0098399630000000002, 2024-09-13T14:49:55.746 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:49:55.746 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:49:55.746 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:49:55.746 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.46157", 2024-09-13T14:49:55.746 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:49:55.747 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 236366 2024-09-13T14:49:55.747 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:49:55.747 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:49:55.747 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:49:55.747 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:49:55.747 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:55.724301+0000", 2024-09-13T14:49:55.747 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:49:55.747 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:49:55.747 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:49:55.747 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:49:55.747 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:55.724301+0000", 2024-09-13T14:49:55.747 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:49:55.747 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:49:55.748 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:49:55.748 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:49:55.748 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:55.724300+0000", 2024-09-13T14:49:55.748 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:49:55.748 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:49:55.748 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:49:55.748 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:49:55.748 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:55.724306+0000", 2024-09-13T14:49:55.748 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2639999999999997e-06 2024-09-13T14:49:55.748 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:49:55.748 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:49:55.748 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:49:55.748 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:55.724318+0000", 2024-09-13T14:49:55.749 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1389000000000001e-05 2024-09-13T14:49:55.749 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:49:55.749 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:49:55.749 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:49:55.749 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:55.724324+0000", 2024-09-13T14:49:55.749 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.533e-06 2024-09-13T14:49:55.749 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:49:55.749 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:49:55.749 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:49:55.749 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:55.724363+0000", 2024-09-13T14:49:55.749 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8847000000000003e-05 2024-09-13T14:49:55.749 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:49:55.749 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:49:55.750 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:49:55.750 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:55.724447+0000", 2024-09-13T14:49:55.750 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3764000000000006e-05 2024-09-13T14:49:55.750 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:49:55.750 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:49:55.750 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:49:55.750 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:55.724522+0000", 2024-09-13T14:49:55.750 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5388000000000004e-05 2024-09-13T14:49:55.750 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:49:55.750 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:49:55.750 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:49:55.750 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:55.728003+0000", 2024-09-13T14:49:55.750 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0034813859999999999 2024-09-13T14:49:55.751 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:49:55.751 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:49:55.751 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:49:55.751 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:55.732745+0000", 2024-09-13T14:49:55.751 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0047413389999999998 2024-09-13T14:49:55.751 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:49:55.751 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:49:55.751 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:49:55.751 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:49:55.751 INFO:teuthology.orchestra.run.smithi195.stdout: ], 2024-09-13T14:49:55.751 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 3 2024-09-13T14:49:55.751 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:49:55.787 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:55.825 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:55.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:55.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:55.927 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:55.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:55.962 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:55.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:56.063 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:56.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:56.112 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:56.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:56.213 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:56.271 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:56.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:56.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:56.373 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:56.392 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:49:56.393 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:49:56.409 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:56.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:56.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:56.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:49:56.512 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:56.539 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:56.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:56.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:56.641 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:56.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:49:56 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:49:56] "GET /metrics HTTP/1.1" 200 36532 "" "Prometheus/2.51.0" 2024-09-13T14:49:56.679 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:56.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:56.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:56.781 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:56.807 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:56.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:56.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:56.908 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:56.937 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:56.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:56.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:57.015 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T14:49:57.040 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:57.076 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:57.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:57.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:57.178 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:57.208 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:57.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:57.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:57.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:57 smithi195 ceph-mon[104262]: pgmap v1244: 109 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 104 active+clean; 1.7 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 788 op/s; 7561/82365 objects misplaced (9.180%); 4.1 MiB/s, 66 objects/s recovering 2024-09-13T14:49:57.283 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:49:57.310 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:57.344 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:57 smithi102 ceph-mon[110183]: pgmap v1244: 109 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 104 active+clean; 1.7 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 788 op/s; 7561/82365 objects misplaced (9.180%); 4.1 MiB/s, 66 objects/s recovering 2024-09-13T14:49:57.344 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:57 smithi102 ceph-mon[115269]: pgmap v1244: 109 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 104 active+clean; 1.7 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 788 op/s; 7561/82365 objects misplaced (9.180%); 4.1 MiB/s, 66 objects/s recovering 2024-09-13T14:49:57.358 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:57.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:57.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:57.460 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:57.489 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:57.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:57.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:57.591 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:57.617 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:57.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:57.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:57.719 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:57.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:57.757 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:57.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:57.858 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:57.897 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:57.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:57.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:57.999 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:58.040 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:58.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:58.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:58.051 INFO:teuthology.orchestra.run.smithi102.stderr:reweighted osd.2 to 1 (10000) 2024-09-13T14:49:58.142 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:58.169 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:58.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:58.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:58.248 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:49:58.272 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:58.299 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:58 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1454957779' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:49:58.299 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:58 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:49:58.299 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:58 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1454957779' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:49:58.300 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:58 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:49:58.303 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:58.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:58.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:58.405 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:58.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:58.461 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:58.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:58.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:58 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1454957779' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:49:58.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:58 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:49:58.562 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:58.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:58.594 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:58.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:58.695 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:58.729 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:58.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:58.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:58.831 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:58.857 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-13T14:49:58.858 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:58.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:58.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:58.961 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:59.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:59.016 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:59.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:59.117 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:59.155 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:59 smithi102 ceph-mon[110183]: pgmap v1245: 109 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 104 active+clean; 1.6 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 789 op/s; 6797/76566 objects misplaced (8.877%); 5.4 MiB/s, 86 objects/s recovering 2024-09-13T14:49:59.155 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:59 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T14:49:59.156 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:49:59 smithi102 ceph-mon[110183]: osdmap e357: 8 total, 8 up, 8 in 2024-09-13T14:49:59.156 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:59 smithi102 ceph-mon[115269]: pgmap v1245: 109 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 104 active+clean; 1.6 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 789 op/s; 6797/76566 objects misplaced (8.877%); 5.4 MiB/s, 86 objects/s recovering 2024-09-13T14:49:59.156 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:59 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T14:49:59.156 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:49:59 smithi102 ceph-mon[115269]: osdmap e357: 8 total, 8 up, 8 in 2024-09-13T14:49:59.156 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:49:59.157 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:49:59.174 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:49:59.174 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-13T14:49:59.174 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-09-13T14:49:59.174 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-09-13T14:49:59.174 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:49:59.174 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:59 smithi195 ceph-mon[104262]: pgmap v1245: 109 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 104 active+clean; 1.6 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 789 op/s; 6797/76566 objects misplaced (8.877%); 5.4 MiB/s, 86 objects/s recovering 2024-09-13T14:49:59.174 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:59 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T14:49:59.175 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:49:59 smithi195 ceph-mon[104262]: osdmap e357: 8 total, 8 up, 8 in 2024-09-13T14:49:59.182 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:59.193 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:49:59.193 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:49:59.194 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:59.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:59.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:59.226 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:59.242 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:49:59.243 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:49:59.282 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:49:59.282 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:49:59.296 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:59.314 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:49:59.314 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:49:59.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:59.344 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:59.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:59.346 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:49:59.346 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:49:59.377 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:49:59.377 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:49:59.411 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:49:59.411 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:49:59.445 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:59.472 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:59.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:59.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:59.574 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:59.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:59.615 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:59.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:59.716 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:59.757 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:49:59.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:59.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:59.842 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T14:49:59.858 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:59.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:59.924 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:49:59.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:00.026 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:00.058 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:00.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:00.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:00.075 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:00 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2274796658' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:50:00.130 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:50:00.161 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:00.203 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:00.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:00.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:00.305 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:00.312 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:00 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2274796658' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:50:00.312 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:00 smithi102 ceph-mon[110183]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-09-13T14:50:00.312 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:00 smithi102 ceph-mon[110183]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-09-13T14:50:00.312 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:00 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2274796658' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:50:00.312 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:00 smithi102 ceph-mon[115269]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-09-13T14:50:00.312 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:00 smithi102 ceph-mon[115269]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-09-13T14:50:00.341 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:00.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:00.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:00.383 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:00 smithi195 ceph-mon[104262]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-09-13T14:50:00.383 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:00 smithi195 ceph-mon[104262]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-09-13T14:50:00.443 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:00.474 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:00.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:00.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:00.577 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:00.613 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:00.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:00.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:00.715 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:00.762 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:00.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:00.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:00.864 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:00.902 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:00.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:00.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:01.004 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:01.047 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:01.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:01.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:01.149 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:01.186 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:01.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:01.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:01.288 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:01.331 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:01.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:01.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:01.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:01 smithi102 ceph-mon[110183]: pgmap v1247: 109 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 104 active+clean; 1.4 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 795 op/s; 6098/70920 objects misplaced (8.598%); 6.7 MiB/s, 107 objects/s recovering 2024-09-13T14:50:01.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:01 smithi102 ceph-mon[115269]: pgmap v1247: 109 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 104 active+clean; 1.4 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 795 op/s; 6098/70920 objects misplaced (8.598%); 6.7 MiB/s, 107 objects/s recovering 2024-09-13T14:50:01.433 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:01.479 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:01.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:01.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:01.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:01 smithi195 ceph-mon[104262]: pgmap v1247: 109 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 104 active+clean; 1.4 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 795 op/s; 6098/70920 objects misplaced (8.598%); 6.7 MiB/s, 107 objects/s recovering 2024-09-13T14:50:01.581 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:01.615 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:01.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:01.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:01.718 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:01.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:01.749 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:01.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:01.850 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:01.887 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:01.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:01.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:01.989 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:02.016 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:02.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:02.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:02.118 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:02.149 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:02.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:02.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:02.251 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:02.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:02.281 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:02.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:02.382 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:02.406 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:50:02.407 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:50:02.411 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:02.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:02.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:02.514 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:02.540 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:02.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:02.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:02.641 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:02.662 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:02.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:02.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:02.754 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:50:02.764 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:02.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:02.787 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:02.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:02.888 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:02.909 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:02.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:02.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:03.011 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:03.032 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:03.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:03.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:03.123 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T14:50:03.134 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:03.160 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:03.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:03.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:03.262 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:03.296 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:03.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:03.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:03.398 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:03.418 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:50:03.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:03 smithi102 ceph-mon[110183]: pgmap v1248: 109 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 104 active+clean; 1.4 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 634 op/s; 6098/66642 objects misplaced (9.150%); 3.4 MiB/s, 54 objects/s recovering 2024-09-13T14:50:03.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:03 smithi102 ceph-mon[110183]: osdmap e358: 8 total, 8 up, 8 in 2024-09-13T14:50:03.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:03 smithi102 ceph-mon[115269]: pgmap v1248: 109 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 104 active+clean; 1.4 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 634 op/s; 6098/66642 objects misplaced (9.150%); 3.4 MiB/s, 54 objects/s recovering 2024-09-13T14:50:03.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:03 smithi102 ceph-mon[115269]: osdmap e358: 8 total, 8 up, 8 in 2024-09-13T14:50:03.446 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:03.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:03.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:03.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:03 smithi195 ceph-mon[104262]: pgmap v1248: 109 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 104 active+clean; 1.4 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 634 op/s; 6098/66642 objects misplaced (9.150%); 3.4 MiB/s, 54 objects/s recovering 2024-09-13T14:50:03.499 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:03 smithi195 ceph-mon[104262]: osdmap e358: 8 total, 8 up, 8 in 2024-09-13T14:50:03.548 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:03.580 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:03.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:03.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:03.682 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:03.712 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:03.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:03.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:03.813 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:03.850 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:03.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:03.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:03.952 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:03.984 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:03.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:03.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:04.086 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:04.108 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:04.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:04.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:04.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:04 smithi102 ceph-mon[110183]: osdmap e359: 8 total, 8 up, 8 in 2024-09-13T14:50:04.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:04 smithi102 ceph-mon[115269]: osdmap e359: 8 total, 8 up, 8 in 2024-09-13T14:50:04.210 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:04.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:04.232 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:04.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:04.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:04 smithi195 ceph-mon[104262]: osdmap e359: 8 total, 8 up, 8 in 2024-09-13T14:50:04.334 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:04.368 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:04.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:04.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:04.470 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:04.502 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:04.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:04.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:04.604 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:04.625 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:04.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:04.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:04.727 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:04.759 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:04.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:04.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:04.861 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:04.895 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:04.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:04.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:04.997 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:05.025 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:05.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:05.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:05.127 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:05.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:05.171 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:05.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:05.260 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:05 smithi102 ceph-mon[110183]: pgmap v1251: 109 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 3 active+remapped+backfill_wait, 104 active+clean; 1.2 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 907 op/s; 4595/60234 objects misplaced (7.629%); 5.5 MiB/s, 88 objects/s recovering 2024-09-13T14:50:05.261 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:05 smithi102 ceph-mon[115269]: pgmap v1251: 109 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 3 active+remapped+backfill_wait, 104 active+clean; 1.2 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 907 op/s; 4595/60234 objects misplaced (7.629%); 5.5 MiB/s, 88 objects/s recovering 2024-09-13T14:50:05.272 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:05.322 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:05.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:05.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:05.357 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:05 smithi195 ceph-mon[104262]: pgmap v1251: 109 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 3 active+remapped+backfill_wait, 104 active+clean; 1.2 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 907 op/s; 4595/60234 objects misplaced (7.629%); 5.5 MiB/s, 88 objects/s recovering 2024-09-13T14:50:05.395 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:50:05.395 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:50:05.423 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:05.463 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:05.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:05.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:05.565 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:05.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:05.616 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:05.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:05.717 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:05.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:05.758 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:05.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:05.859 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:05.905 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:05.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:05.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:06.007 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:06.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:06.046 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:06.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:06.133 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T14:50:06.147 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:06.194 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:06.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:06.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:06.296 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:06.337 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:06.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:06.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:06.420 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:50:06.439 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:06.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:06.475 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:06.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:06.509 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:06.576 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:06.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:06.611 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:06.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:06.701 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:50:06 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:50:06] "GET /metrics HTTP/1.1" 200 36532 "" "Prometheus/2.51.0" 2024-09-13T14:50:06.713 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:06.748 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:06.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:06.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:06.850 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:06.894 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:06.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:06.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:06.996 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:07.035 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:07.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:07.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:07.125 INFO:teuthology.orchestra.run.smithi102.stderr:reweighted osd.1 to 1 (10000) 2024-09-13T14:50:07.128 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is unset 2024-09-13T14:50:07.137 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:07.180 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:07.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:07.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:07.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:07 smithi195 ceph-mon[104262]: pgmap v1252: 109 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 105 active+clean; 1.2 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 538 op/s; 4595/59328 objects misplaced (7.745%); 4.3 MiB/s, 69 objects/s recovering 2024-09-13T14:50:07.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:07 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3209218227' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:50:07.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:07 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:50:07.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:07 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/919476026' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:50:07.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:07 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:50:07.282 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:07.322 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:07.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:07.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:07.332 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:07 smithi102 ceph-mon[110183]: pgmap v1252: 109 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 105 active+clean; 1.2 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 538 op/s; 4595/59328 objects misplaced (7.745%); 4.3 MiB/s, 69 objects/s recovering 2024-09-13T14:50:07.332 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:07 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3209218227' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:50:07.333 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:07 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:50:07.333 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:07 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/919476026' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:50:07.333 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:07 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:50:07.333 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:07 smithi102 ceph-mon[115269]: pgmap v1252: 109 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 105 active+clean; 1.2 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 538 op/s; 4595/59328 objects misplaced (7.745%); 4.3 MiB/s, 69 objects/s recovering 2024-09-13T14:50:07.333 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:07 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3209218227' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:50:07.333 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:07 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:50:07.333 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:07 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/919476026' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:50:07.334 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:07 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:50:07.424 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:07.452 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:07.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:07.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:07.554 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:07.580 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:07.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:07.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:07.682 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:07.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:07.715 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:07.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:07.816 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:07.855 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:07.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:07.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:07.957 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:07.998 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:07.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:07.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:08.100 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:08.138 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:08.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:08.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:08.240 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:08.275 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:08.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:08.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:08.377 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:08.385 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:08 smithi102 ceph-mon[110183]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:50:08.385 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:08 smithi102 ceph-mon[110183]: Cluster is now healthy 2024-09-13T14:50:08.385 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:08 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T14:50:08.385 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:08 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:50:08.385 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:08 smithi102 ceph-mon[110183]: osdmap e360: 8 total, 8 up, 8 in 2024-09-13T14:50:08.385 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:08 smithi102 ceph-mon[115269]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:50:08.385 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:08 smithi102 ceph-mon[115269]: Cluster is now healthy 2024-09-13T14:50:08.386 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:08 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T14:50:08.386 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:08 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:50:08.386 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:08 smithi102 ceph-mon[115269]: osdmap e360: 8 total, 8 up, 8 in 2024-09-13T14:50:08.414 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:08.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:08.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:08.422 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:08 smithi195 ceph-mon[104262]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:50:08.422 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:08 smithi195 ceph-mon[104262]: Cluster is now healthy 2024-09-13T14:50:08.423 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:08 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T14:50:08.423 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:08 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:50:08.423 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:08 smithi195 ceph-mon[104262]: osdmap e360: 8 total, 8 up, 8 in 2024-09-13T14:50:08.468 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:50:08.468 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:50:08.517 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:08.559 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:08.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:08.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:08.662 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:08.696 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:08.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:08.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:08.724 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-13T14:50:08.724 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:50:08.798 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:08.825 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:08.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:08.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:08.927 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:08.968 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:08.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:08.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:09.070 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:09.109 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:09.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:09.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:09.165 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T14:50:09.211 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:09.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:09.240 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:09.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:09.341 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:09.385 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:09.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:09.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:09.394 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:09 smithi102 ceph-mon[110183]: pgmap v1254: 109 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 105 active+clean; 1.1 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 644 op/s; 4388/55035 objects misplaced (7.973%); 5.5 MiB/s, 88 objects/s recovering 2024-09-13T14:50:09.395 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:09 smithi102 ceph-mon[115269]: pgmap v1254: 109 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 105 active+clean; 1.1 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 644 op/s; 4388/55035 objects misplaced (7.973%); 5.5 MiB/s, 88 objects/s recovering 2024-09-13T14:50:09.466 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:50:09.487 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:09.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:09 smithi195 ceph-mon[104262]: pgmap v1254: 109 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 105 active+clean; 1.1 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 644 op/s; 4388/55035 objects misplaced (7.973%); 5.5 MiB/s, 88 objects/s recovering 2024-09-13T14:50:09.519 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:09.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:09.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:09.590 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:09.621 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:09.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:09.649 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:09.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:09.750 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:09.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:09.797 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:09.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:09.897 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:50:09.898 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:50:09.898 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:09.932 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:50:09.932 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:50:09.938 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:09.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:09.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:09.970 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:50:09.970 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:50:10.008 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:50:10.008 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:50:10.038 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:50:10.038 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:50:10.041 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:10.073 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:50:10.073 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:50:10.076 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:10.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:10.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:10.086 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:50:10.111 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:50:10.111 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:50:10.157 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:50:10.157 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:50:10.178 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:10.204 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:10.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:10.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:10.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:10 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2762715730' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:50:10.306 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:10.340 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:10.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:10.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:10.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:10 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2762715730' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:50:10.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:10 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2762715730' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:50:10.443 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:10.473 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:10.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:10.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:10.575 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:10.602 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:10.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:10.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:10.704 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:10.735 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:10.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:10.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:10.837 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:10.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:10.883 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:10.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:10.984 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:11.019 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:11.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:11.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:11.120 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:11.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:11.160 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:11.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:11.262 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:11.298 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:11.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:11.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:11.400 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:11.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:11 smithi102 ceph-mon[110183]: pgmap v1255: 109 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 105 active+clean; 1014 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 731 op/s; 3344/48531 objects misplaced (6.890%); 6.3 MiB/s, 100 objects/s recovering 2024-09-13T14:50:11.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:11 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:50:11.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:11 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:50:11.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:11 smithi102 ceph-mon[115269]: pgmap v1255: 109 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 105 active+clean; 1014 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 731 op/s; 3344/48531 objects misplaced (6.890%); 6.3 MiB/s, 100 objects/s recovering 2024-09-13T14:50:11.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:11 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:50:11.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:11 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:50:11.435 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:11.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:11.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:11.452 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:11 smithi195 ceph-mon[104262]: pgmap v1255: 109 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 105 active+clean; 1014 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 731 op/s; 3344/48531 objects misplaced (6.890%); 6.3 MiB/s, 100 objects/s recovering 2024-09-13T14:50:11.452 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:11 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:50:11.452 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:11 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:50:11.512 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:50:11.512 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-09-13T14:50:11.512 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-09-13T14:50:11.512 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-13T14:50:11.512 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.513 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.46157.0:129241 8.9 8:93cac0a8:::benchmark_data_smithi102_313275_object129240:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-13T14:50:11.513 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:47:42.300714+0000", 2024-09-13T14:50:11.513 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 149.2077659, 2024-09-13T14:50:11.513 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.156294027, 2024-09-13T14:50:11.513 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:11.513 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:11.514 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:50:11.514 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.46157", 2024-09-13T14:50:11.514 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:50:11.514 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 129241 2024-09-13T14:50:11.514 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.514 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:11.514 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.514 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:11.514 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.300714+0000", 2024-09-13T14:50:11.514 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:11.514 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.514 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.515 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:11.515 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.300714+0000", 2024-09-13T14:50:11.515 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:11.515 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.515 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.515 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:11.515 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.300712+0000", 2024-09-13T14:50:11.515 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:50:11.515 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.515 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.515 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:11.515 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.300743+0000", 2024-09-13T14:50:11.515 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.1000000000000001e-05 2024-09-13T14:50:11.515 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.516 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.516 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:11.516 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.300755+0000", 2024-09-13T14:50:11.516 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2415000000000001e-05 2024-09-13T14:50:11.516 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.516 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.516 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:11.516 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.300761+0000", 2024-09-13T14:50:11.516 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3010000000000003e-06 2024-09-13T14:50:11.516 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.516 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.516 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:11.517 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.323853+0000", 2024-09-13T14:50:11.517 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.023091558000000002 2024-09-13T14:50:11.517 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.517 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.517 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:11.517 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.323981+0000", 2024-09-13T14:50:11.517 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012807899999999999 2024-09-13T14:50:11.517 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.517 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.517 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,6", 2024-09-13T14:50:11.517 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.324110+0000", 2024-09-13T14:50:11.517 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012860500000000001 2024-09-13T14:50:11.517 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.518 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.518 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:50:11.518 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.331027+0000", 2024-09-13T14:50:11.518 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0069175260000000002 2024-09-13T14:50:11.518 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.518 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.518 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:11.518 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.332691+0000", 2024-09-13T14:50:11.518 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001663487 2024-09-13T14:50:11.518 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.518 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.518 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:11.518 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.456924+0000", 2024-09-13T14:50:11.519 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.124233696 2024-09-13T14:50:11.519 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.519 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.519 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:11.519 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.456971+0000", 2024-09-13T14:50:11.519 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6971999999999997e-05 2024-09-13T14:50:11.519 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.519 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.519 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:11.519 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.457008+0000", 2024-09-13T14:50:11.519 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6294000000000002e-05 2024-09-13T14:50:11.519 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:11.519 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:11.520 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:11.520 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.520 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.520 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.46157.0:129254 8.9 8:940937c9:::benchmark_data_smithi102_313275_object129253:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-13T14:50:11.520 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:47:42.309981+0000", 2024-09-13T14:50:11.520 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 149.19849822399999, 2024-09-13T14:50:11.520 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.14721906700000001, 2024-09-13T14:50:11.520 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:11.520 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:11.520 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:50:11.520 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.46157", 2024-09-13T14:50:11.520 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:50:11.521 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 129254 2024-09-13T14:50:11.521 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.521 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:11.521 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.521 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:11.521 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.309981+0000", 2024-09-13T14:50:11.521 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:11.521 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.521 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.521 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:11.521 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.309981+0000", 2024-09-13T14:50:11.521 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:11.521 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.522 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.522 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:11.522 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.309980+0000", 2024-09-13T14:50:11.522 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:50:11.522 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.522 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.522 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:11.522 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.310010+0000", 2024-09-13T14:50:11.522 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0592000000000001e-05 2024-09-13T14:50:11.522 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.522 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.522 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:11.522 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.310025+0000", 2024-09-13T14:50:11.522 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4898e-05 2024-09-13T14:50:11.523 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.523 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.523 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:11.523 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.310032+0000", 2024-09-13T14:50:11.523 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1559999999999998e-06 2024-09-13T14:50:11.523 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.523 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.523 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:11.523 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.327734+0000", 2024-09-13T14:50:11.523 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017701449000000001 2024-09-13T14:50:11.523 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.523 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.523 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:11.524 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.327874+0000", 2024-09-13T14:50:11.524 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000140018 2024-09-13T14:50:11.524 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.524 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.524 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,6", 2024-09-13T14:50:11.524 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.328028+0000", 2024-09-13T14:50:11.524 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015407999999999999 2024-09-13T14:50:11.524 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.524 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.524 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:50:11.524 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.331098+0000", 2024-09-13T14:50:11.524 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003069989 2024-09-13T14:50:11.524 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.524 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.525 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:11.525 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.332820+0000", 2024-09-13T14:50:11.525 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0017219360000000001 2024-09-13T14:50:11.525 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.525 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.525 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:11.525 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.457120+0000", 2024-09-13T14:50:11.525 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.124300577 2024-09-13T14:50:11.525 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.525 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.525 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:11.525 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.457167+0000", 2024-09-13T14:50:11.525 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6232000000000001e-05 2024-09-13T14:50:11.526 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.526 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.526 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:11.526 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.457200+0000", 2024-09-13T14:50:11.526 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.3810000000000003e-05 2024-09-13T14:50:11.526 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:11.526 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:11.526 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:11.526 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.526 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.526 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.46157.0:129530 8.3 8:cc5c28ce:::benchmark_data_smithi102_313275_object129529:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-13T14:50:11.526 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:47:42.663378+0000", 2024-09-13T14:50:11.526 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 148.84510102499999, 2024-09-13T14:50:11.526 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.17495028300000001, 2024-09-13T14:50:11.527 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:11.527 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:11.527 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:50:11.527 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.46157", 2024-09-13T14:50:11.528 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:50:11.528 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 129530 2024-09-13T14:50:11.528 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.528 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:11.528 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.528 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:11.528 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.663378+0000", 2024-09-13T14:50:11.528 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:11.528 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.528 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.528 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:11.529 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.663378+0000", 2024-09-13T14:50:11.529 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:11.529 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.529 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.529 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:11.529 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.663378+0000", 2024-09-13T14:50:11.529 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:50:11.529 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.529 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.529 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:11.529 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.663408+0000", 2024-09-13T14:50:11.529 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0093000000000001e-05 2024-09-13T14:50:11.529 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.530 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.530 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:11.530 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.663417+0000", 2024-09-13T14:50:11.530 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.5049999999999996e-06 2024-09-13T14:50:11.530 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.530 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.530 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:11.530 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.663423+0000", 2024-09-13T14:50:11.530 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5439999999999998e-06 2024-09-13T14:50:11.530 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.530 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.530 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:11.530 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.663473+0000", 2024-09-13T14:50:11.531 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0112999999999999e-05 2024-09-13T14:50:11.531 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.531 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.531 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:11.531 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.663586+0000", 2024-09-13T14:50:11.531 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011284499999999999 2024-09-13T14:50:11.531 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.531 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.531 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:50:11.531 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.663699+0000", 2024-09-13T14:50:11.531 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000112914 2024-09-13T14:50:11.531 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.531 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.531 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:11.532 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.668120+0000", 2024-09-13T14:50:11.532 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0044209239999999997 2024-09-13T14:50:11.532 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.532 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.532 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:50:11.532 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.682704+0000", 2024-09-13T14:50:11.532 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014583921999999999 2024-09-13T14:50:11.532 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.532 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.532 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:11.532 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.838255+0000", 2024-09-13T14:50:11.532 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.15555179899999999 2024-09-13T14:50:11.532 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.533 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.533 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:11.533 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.838291+0000", 2024-09-13T14:50:11.533 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5386999999999998e-05 2024-09-13T14:50:11.533 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.533 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.533 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:11.533 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.838329+0000", 2024-09-13T14:50:11.533 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7876000000000001e-05 2024-09-13T14:50:11.533 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:11.533 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:11.533 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:11.533 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.534 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.534 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.46157.0:129539 8.3 8:ce1ec90c:::benchmark_data_smithi102_313275_object129538:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-13T14:50:11.534 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:47:42.668520+0000", 2024-09-13T14:50:11.534 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 148.83995963800001, 2024-09-13T14:50:11.534 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.177054238, 2024-09-13T14:50:11.534 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:11.534 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:11.534 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:50:11.534 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.46157", 2024-09-13T14:50:11.534 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:50:11.534 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 129539 2024-09-13T14:50:11.534 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.534 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:11.535 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.535 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:11.535 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.668520+0000", 2024-09-13T14:50:11.535 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:11.535 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.535 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.535 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:11.535 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.668520+0000", 2024-09-13T14:50:11.535 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:11.535 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.535 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.535 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:11.535 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.668519+0000", 2024-09-13T14:50:11.536 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:50:11.536 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.536 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.536 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:11.536 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.668572+0000", 2024-09-13T14:50:11.536 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2917000000000002e-05 2024-09-13T14:50:11.536 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.536 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.536 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:11.536 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.668585+0000", 2024-09-13T14:50:11.536 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2762000000000001e-05 2024-09-13T14:50:11.536 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.536 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.537 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:11.537 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.668591+0000", 2024-09-13T14:50:11.537 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6669999999999999e-06 2024-09-13T14:50:11.537 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.537 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.537 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:11.537 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.668657+0000", 2024-09-13T14:50:11.537 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5191999999999995e-05 2024-09-13T14:50:11.537 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.537 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.537 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:11.537 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.668791+0000", 2024-09-13T14:50:11.537 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000134633 2024-09-13T14:50:11.537 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.538 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.538 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:50:11.538 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.668994+0000", 2024-09-13T14:50:11.538 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000202926 2024-09-13T14:50:11.538 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.538 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.538 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:11.538 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.682740+0000", 2024-09-13T14:50:11.538 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.01374567 2024-09-13T14:50:11.538 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.538 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.538 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:50:11.539 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.682766+0000", 2024-09-13T14:50:11.539 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.6384999999999999e-05 2024-09-13T14:50:11.539 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.539 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.539 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:11.539 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.845494+0000", 2024-09-13T14:50:11.539 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.162727762 2024-09-13T14:50:11.539 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.539 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.539 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:11.539 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.845539+0000", 2024-09-13T14:50:11.539 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4586999999999998e-05 2024-09-13T14:50:11.539 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.539 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.540 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:11.540 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.845574+0000", 2024-09-13T14:50:11.540 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5502e-05 2024-09-13T14:50:11.540 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:11.540 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:11.540 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:11.540 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.540 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.540 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.46157.0:129541 8.3 8:c4013ac6:::benchmark_data_smithi102_313275_object129540:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-13T14:50:11.540 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:47:42.668679+0000", 2024-09-13T14:50:11.540 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 148.83979998500001, 2024-09-13T14:50:11.540 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.179840901, 2024-09-13T14:50:11.540 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:11.540 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:11.541 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:50:11.541 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.46157", 2024-09-13T14:50:11.541 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:50:11.541 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 129541 2024-09-13T14:50:11.541 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.541 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:11.541 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.541 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:11.541 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.668679+0000", 2024-09-13T14:50:11.541 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:11.541 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.541 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.541 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:11.541 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.668679+0000", 2024-09-13T14:50:11.542 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:11.542 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.542 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.542 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:11.542 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.668679+0000", 2024-09-13T14:50:11.542 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:50:11.542 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.542 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.542 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:11.542 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.668759+0000", 2024-09-13T14:50:11.542 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9896999999999998e-05 2024-09-13T14:50:11.542 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.542 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.543 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:11.543 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.668767+0000", 2024-09-13T14:50:11.543 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.79e-06 2024-09-13T14:50:11.543 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.543 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.543 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:11.543 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.668772+0000", 2024-09-13T14:50:11.543 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5589999999999996e-06 2024-09-13T14:50:11.543 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.543 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.543 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:11.543 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.674790+0000", 2024-09-13T14:50:11.544 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0060180040000000004 2024-09-13T14:50:11.544 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.544 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.544 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:11.544 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.674932+0000", 2024-09-13T14:50:11.544 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014179199999999999 2024-09-13T14:50:11.544 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.544 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.544 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:50:11.544 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.675064+0000", 2024-09-13T14:50:11.544 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013237899999999999 2024-09-13T14:50:11.544 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.544 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.545 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:11.545 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.682762+0000", 2024-09-13T14:50:11.545 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.007697822 2024-09-13T14:50:11.545 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.545 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.545 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:50:11.545 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.682769+0000", 2024-09-13T14:50:11.545 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6680000000000004e-06 2024-09-13T14:50:11.545 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.545 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.545 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:11.545 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.848411+0000", 2024-09-13T14:50:11.546 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.16564199700000001 2024-09-13T14:50:11.546 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.546 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.546 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:11.546 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.848482+0000", 2024-09-13T14:50:11.546 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.0765000000000004e-05 2024-09-13T14:50:11.546 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.546 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.546 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:11.546 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.848520+0000", 2024-09-13T14:50:11.546 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8841999999999998e-05 2024-09-13T14:50:11.547 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:11.547 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:11.547 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:11.547 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.547 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.547 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.46157.0:129542 8.3 8:c3472679:::benchmark_data_smithi102_313275_object129541:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-13T14:50:11.547 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:47:42.668778+0000", 2024-09-13T14:50:11.547 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 148.83970193900001, 2024-09-13T14:50:11.547 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.17986645100000001, 2024-09-13T14:50:11.547 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:11.547 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:11.547 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:50:11.547 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.46157", 2024-09-13T14:50:11.548 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:50:11.548 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 129542 2024-09-13T14:50:11.548 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.548 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:11.548 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.548 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:11.548 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.668778+0000", 2024-09-13T14:50:11.548 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:11.548 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.548 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.548 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:11.548 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.668778+0000", 2024-09-13T14:50:11.548 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:11.549 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.549 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.549 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:11.549 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.668777+0000", 2024-09-13T14:50:11.549 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:50:11.549 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.549 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.549 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:11.549 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.668809+0000", 2024-09-13T14:50:11.549 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2061999999999999e-05 2024-09-13T14:50:11.549 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.549 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.550 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:11.550 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.668817+0000", 2024-09-13T14:50:11.550 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3680000000000002e-06 2024-09-13T14:50:11.550 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.550 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.550 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:11.550 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.668823+0000", 2024-09-13T14:50:11.550 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1680000000000003e-06 2024-09-13T14:50:11.550 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.550 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.550 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:11.550 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.678824+0000", 2024-09-13T14:50:11.550 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010001358 2024-09-13T14:50:11.551 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.551 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.551 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:11.551 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.678967+0000", 2024-09-13T14:50:11.551 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014290700000000001 2024-09-13T14:50:11.551 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.551 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.551 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:50:11.551 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.679094+0000", 2024-09-13T14:50:11.551 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000127157 2024-09-13T14:50:11.551 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.551 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.551 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:50:11.552 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.682826+0000", 2024-09-13T14:50:11.552 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0037319879999999999 2024-09-13T14:50:11.552 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.552 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.552 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:11.552 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.684264+0000", 2024-09-13T14:50:11.552 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001438007 2024-09-13T14:50:11.552 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.552 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.552 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:11.553 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.848552+0000", 2024-09-13T14:50:11.553 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.164287821 2024-09-13T14:50:11.553 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.553 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.553 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:11.553 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.848613+0000", 2024-09-13T14:50:11.553 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0708999999999998e-05 2024-09-13T14:50:11.553 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.553 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.553 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:11.553 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.848644+0000", 2024-09-13T14:50:11.553 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.1386000000000001e-05 2024-09-13T14:50:11.554 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:11.554 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:11.554 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:11.554 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.554 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.554 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.46157.0:207733 8.4 8:283dea0f:::benchmark_data_smithi102_313275_object77146:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e350)", 2024-09-13T14:50:11.554 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:49:18.947147+0000", 2024-09-13T14:50:11.554 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 52.561332487000001, 2024-09-13T14:50:11.554 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.26735582400000002, 2024-09-13T14:50:11.554 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:11.554 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:11.554 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:50:11.555 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.46157", 2024-09-13T14:50:11.555 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:50:11.555 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 207733 2024-09-13T14:50:11.555 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.555 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:11.555 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.555 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:11.555 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:18.947147+0000", 2024-09-13T14:50:11.555 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:11.555 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.555 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.555 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:11.555 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:18.947147+0000", 2024-09-13T14:50:11.556 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:11.556 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.556 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.556 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:11.556 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:18.947146+0000", 2024-09-13T14:50:11.556 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:50:11.556 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.556 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.556 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:11.556 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:18.947151+0000", 2024-09-13T14:50:11.556 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.352e-06 2024-09-13T14:50:11.556 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.556 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.557 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:11.557 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:18.947159+0000", 2024-09-13T14:50:11.557 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2770000000000005e-06 2024-09-13T14:50:11.557 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.557 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.557 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:11.557 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:18.947165+0000", 2024-09-13T14:50:11.557 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1500000000000004e-06 2024-09-13T14:50:11.557 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.557 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.557 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:11.557 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:18.950351+0000", 2024-09-13T14:50:11.558 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0031856039999999999 2024-09-13T14:50:11.558 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.558 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.558 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:11.558 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:18.950465+0000", 2024-09-13T14:50:11.558 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000114548 2024-09-13T14:50:11.558 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.558 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.558 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 2,4", 2024-09-13T14:50:11.558 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:18.950573+0000", 2024-09-13T14:50:11.558 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000107748 2024-09-13T14:50:11.558 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.558 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.559 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:50:11.559 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:18.955659+0000", 2024-09-13T14:50:11.559 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0050855980000000002 2024-09-13T14:50:11.559 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.559 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.559 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:11.559 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:18.955857+0000", 2024-09-13T14:50:11.559 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00019875700000000001 2024-09-13T14:50:11.559 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.559 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.559 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:11.559 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:19.214389+0000", 2024-09-13T14:50:11.559 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.25853143699999998 2024-09-13T14:50:11.560 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.560 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.560 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:11.560 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:19.214457+0000", 2024-09-13T14:50:11.560 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7952000000000003e-05 2024-09-13T14:50:11.560 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.560 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.560 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:11.560 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:19.214503+0000", 2024-09-13T14:50:11.560 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6142e-05 2024-09-13T14:50:11.560 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:11.560 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:11.560 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:11.561 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.561 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.561 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.46157.0:226347 8.4 8:2531c967:::benchmark_data_smithi102_313275_object95760:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-09-13T14:50:11.561 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:49:42.875947+0000", 2024-09-13T14:50:11.561 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 28.632532023, 2024-09-13T14:50:11.561 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.44443308599999998, 2024-09-13T14:50:11.561 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:11.561 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:11.561 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:50:11.561 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.46157", 2024-09-13T14:50:11.561 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:50:11.561 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 226347 2024-09-13T14:50:11.562 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.562 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:11.562 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.562 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:11.562 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.875947+0000", 2024-09-13T14:50:11.562 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:11.562 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.562 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.562 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:11.562 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.875947+0000", 2024-09-13T14:50:11.562 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:11.562 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.562 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.563 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:11.563 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.875947+0000", 2024-09-13T14:50:11.563 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:50:11.563 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.563 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.563 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:11.563 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.875951+0000", 2024-09-13T14:50:11.563 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2479999999999998e-06 2024-09-13T14:50:11.563 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.563 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.563 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:11.563 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.875959+0000", 2024-09-13T14:50:11.564 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1659999999999994e-06 2024-09-13T14:50:11.564 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.564 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.564 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:11.564 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.875963+0000", 2024-09-13T14:50:11.564 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2440000000000004e-06 2024-09-13T14:50:11.564 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.564 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.564 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:11.564 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.876007+0000", 2024-09-13T14:50:11.564 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3640999999999998e-05 2024-09-13T14:50:11.564 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.564 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.565 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:11.565 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.876115+0000", 2024-09-13T14:50:11.565 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010817199999999999 2024-09-13T14:50:11.565 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.565 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.565 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 2,4", 2024-09-13T14:50:11.565 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.876222+0000", 2024-09-13T14:50:11.565 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000106699 2024-09-13T14:50:11.565 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.565 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.565 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:11.565 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.881552+0000", 2024-09-13T14:50:11.565 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0053302710000000001 2024-09-13T14:50:11.566 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.566 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.566 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:50:11.566 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.881591+0000", 2024-09-13T14:50:11.566 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8331999999999998e-05 2024-09-13T14:50:11.566 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.566 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.566 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:11.566 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:43.320287+0000", 2024-09-13T14:50:11.566 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.438696164 2024-09-13T14:50:11.566 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.566 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.566 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:11.567 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:43.320339+0000", 2024-09-13T14:50:11.567 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2484000000000001e-05 2024-09-13T14:50:11.567 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.567 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.567 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:11.567 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:43.320381+0000", 2024-09-13T14:50:11.567 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1322999999999997e-05 2024-09-13T14:50:11.567 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:11.567 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:11.567 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:11.567 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.567 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.567 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.46157.0:226351 8.4 8:23f709fc:::benchmark_data_smithi102_313275_object95764:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-09-13T14:50:11.568 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:49:42.883550+0000", 2024-09-13T14:50:11.568 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 28.624929380000001, 2024-09-13T14:50:11.568 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.43697799799999998, 2024-09-13T14:50:11.568 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:11.568 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:11.568 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:50:11.568 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.46157", 2024-09-13T14:50:11.568 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:50:11.568 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 226351 2024-09-13T14:50:11.568 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.568 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:11.568 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.568 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:11.569 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.883550+0000", 2024-09-13T14:50:11.569 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:11.569 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.569 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.569 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:11.569 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.883550+0000", 2024-09-13T14:50:11.569 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:11.569 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.569 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.569 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:11.569 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.883549+0000", 2024-09-13T14:50:11.569 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:50:11.569 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.570 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.570 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:11.570 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.883554+0000", 2024-09-13T14:50:11.570 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7899999999999999e-06 2024-09-13T14:50:11.570 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.570 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.570 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:11.570 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.883564+0000", 2024-09-13T14:50:11.570 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0305e-05 2024-09-13T14:50:11.570 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.570 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.570 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:11.570 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.883569+0000", 2024-09-13T14:50:11.570 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4939999999999997e-06 2024-09-13T14:50:11.571 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.571 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.571 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:11.571 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.883608+0000", 2024-09-13T14:50:11.571 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9635999999999998e-05 2024-09-13T14:50:11.571 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.571 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.571 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:11.571 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.883684+0000", 2024-09-13T14:50:11.571 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6062000000000005e-05 2024-09-13T14:50:11.571 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.571 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.571 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 2,4", 2024-09-13T14:50:11.572 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.883750+0000", 2024-09-13T14:50:11.572 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5398e-05 2024-09-13T14:50:11.572 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.572 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.572 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:50:11.572 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.886979+0000", 2024-09-13T14:50:11.572 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0032296360000000001 2024-09-13T14:50:11.572 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:11.573 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.574 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.574 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:11.574 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.887379+0000", 2024-09-13T14:50:11.574 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00039910400000000001 2024-09-13T14:50:11.574 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.574 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.574 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:11.574 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:43.320440+0000", 2024-09-13T14:50:11.574 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.43306097500000001 2024-09-13T14:50:11.574 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.574 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.574 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:11.575 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:43.320484+0000", 2024-09-13T14:50:11.575 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4311000000000003e-05 2024-09-13T14:50:11.575 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.575 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.575 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:11.575 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:43.320528+0000", 2024-09-13T14:50:11.575 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4246999999999998e-05 2024-09-13T14:50:11.575 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:11.575 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:11.575 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:11.575 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.575 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.575 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.46157.0:226367 8.4 8:2828d25f:::benchmark_data_smithi102_313275_object95780:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-09-13T14:50:11.575 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:49:42.967010+0000", 2024-09-13T14:50:11.576 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 28.541469432, 2024-09-13T14:50:11.576 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.353662376, 2024-09-13T14:50:11.576 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:11.576 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:11.576 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:50:11.576 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.46157", 2024-09-13T14:50:11.576 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:50:11.576 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 226367 2024-09-13T14:50:11.576 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.576 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:11.576 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.576 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:11.577 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.967010+0000", 2024-09-13T14:50:11.577 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:11.577 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.577 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.577 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:11.577 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.967010+0000", 2024-09-13T14:50:11.577 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:11.577 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.577 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.577 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:11.577 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.967010+0000", 2024-09-13T14:50:11.577 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:50:11.577 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.577 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.578 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:11.578 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.967013+0000", 2024-09-13T14:50:11.578 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.1329999999999999e-06 2024-09-13T14:50:11.578 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.578 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.578 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:11.578 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.967020+0000", 2024-09-13T14:50:11.578 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3200000000000002e-06 2024-09-13T14:50:11.578 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.578 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.578 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:11.578 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.967025+0000", 2024-09-13T14:50:11.578 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3070000000000002e-06 2024-09-13T14:50:11.579 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.579 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.579 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:11.579 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.967065+0000", 2024-09-13T14:50:11.579 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9372999999999997e-05 2024-09-13T14:50:11.579 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.579 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.579 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:11.579 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.967180+0000", 2024-09-13T14:50:11.579 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000115313 2024-09-13T14:50:11.579 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.579 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.579 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 2,4", 2024-09-13T14:50:11.580 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.967257+0000", 2024-09-13T14:50:11.580 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7218999999999994e-05 2024-09-13T14:50:11.580 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.580 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.580 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:11.580 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.973117+0000", 2024-09-13T14:50:11.580 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0058593530000000003 2024-09-13T14:50:11.580 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.580 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.580 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:50:11.580 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.973129+0000", 2024-09-13T14:50:11.580 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2286999999999999e-05 2024-09-13T14:50:11.580 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.581 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.581 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:11.581 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:43.320570+0000", 2024-09-13T14:50:11.581 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.34744124100000001 2024-09-13T14:50:11.581 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.581 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.581 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:11.581 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:43.320638+0000", 2024-09-13T14:50:11.581 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7707999999999996e-05 2024-09-13T14:50:11.581 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.581 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.581 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:11.581 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:43.320672+0000", 2024-09-13T14:50:11.581 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4576000000000002e-05 2024-09-13T14:50:11.582 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:11.582 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:11.582 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:11.582 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.582 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.582 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.46157.0:226370 8.d 8:be4049dc:::benchmark_data_smithi102_313275_object95783:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e355)", 2024-09-13T14:50:11.582 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:49:42.973234+0000", 2024-09-13T14:50:11.582 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 28.53524582, 2024-09-13T14:50:11.582 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.34722298600000001, 2024-09-13T14:50:11.582 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:11.582 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:11.582 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:50:11.582 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.46157", 2024-09-13T14:50:11.583 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:50:11.583 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 226370 2024-09-13T14:50:11.583 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.583 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:11.583 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.583 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:11.583 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.973234+0000", 2024-09-13T14:50:11.583 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:11.583 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.583 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.583 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:11.583 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.973234+0000", 2024-09-13T14:50:11.583 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:11.583 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.583 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.584 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:11.584 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.973232+0000", 2024-09-13T14:50:11.584 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:50:11.584 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.584 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.584 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:11.584 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.973239+0000", 2024-09-13T14:50:11.584 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2620000000000001e-06 2024-09-13T14:50:11.584 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.584 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.584 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:11.584 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.973251+0000", 2024-09-13T14:50:11.584 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1846e-05 2024-09-13T14:50:11.584 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.584 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.585 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:11.585 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.973258+0000", 2024-09-13T14:50:11.585 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.0210000000000002e-06 2024-09-13T14:50:11.585 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.585 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.585 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:11.585 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.973308+0000", 2024-09-13T14:50:11.585 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9370000000000003e-05 2024-09-13T14:50:11.585 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.585 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.585 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-09-13T14:50:11.585 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.973314+0000", 2024-09-13T14:50:11.585 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5209999999999999e-06 2024-09-13T14:50:11.586 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.586 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.586 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:11.586 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.974048+0000", 2024-09-13T14:50:11.586 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00073345599999999995 2024-09-13T14:50:11.586 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.586 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.586 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:11.586 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.974165+0000", 2024-09-13T14:50:11.586 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000117736 2024-09-13T14:50:11.586 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.586 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.586 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 2,7", 2024-09-13T14:50:11.586 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.974280+0000", 2024-09-13T14:50:11.586 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000114293 2024-09-13T14:50:11.586 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.587 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.587 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:50:11.587 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.978416+0000", 2024-09-13T14:50:11.587 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0041359980000000001 2024-09-13T14:50:11.587 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.587 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.587 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:11.587 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.978649+0000", 2024-09-13T14:50:11.587 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000233623 2024-09-13T14:50:11.587 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.587 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.587 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:11.587 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:43.320373+0000", 2024-09-13T14:50:11.587 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.34172394499999997 2024-09-13T14:50:11.588 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.588 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.588 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:11.588 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:43.320420+0000", 2024-09-13T14:50:11.588 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6966999999999999e-05 2024-09-13T14:50:11.588 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.588 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.588 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:11.588 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:43.320457+0000", 2024-09-13T14:50:11.588 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6557999999999998e-05 2024-09-13T14:50:11.588 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:11.588 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:11.588 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:11.588 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.589 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.589 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.46157.0:226371 8.4 8:2b4a5a64:::benchmark_data_smithi102_313275_object95784:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e355)", 2024-09-13T14:50:11.589 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:49:42.975194+0000", 2024-09-13T14:50:11.589 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 28.533285529, 2024-09-13T14:50:11.589 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.34566448399999999, 2024-09-13T14:50:11.589 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:11.589 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:11.589 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:50:11.589 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.46157", 2024-09-13T14:50:11.589 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:50:11.589 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 226371 2024-09-13T14:50:11.589 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.589 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:11.589 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.590 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:11.590 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.975194+0000", 2024-09-13T14:50:11.590 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:11.590 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.590 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.590 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:11.590 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.975194+0000", 2024-09-13T14:50:11.590 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:11.590 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.590 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.590 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:11.590 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.975193+0000", 2024-09-13T14:50:11.590 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:50:11.590 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.590 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.591 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:11.591 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.975198+0000", 2024-09-13T14:50:11.591 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4839999999999998e-06 2024-09-13T14:50:11.591 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.591 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.591 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:11.591 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.975206+0000", 2024-09-13T14:50:11.591 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.653e-06 2024-09-13T14:50:11.591 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.591 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.591 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:11.591 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.975211+0000", 2024-09-13T14:50:11.592 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7600000000000002e-06 2024-09-13T14:50:11.592 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.592 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.592 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:11.592 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.975247+0000", 2024-09-13T14:50:11.592 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5657000000000001e-05 2024-09-13T14:50:11.592 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.592 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.592 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:11.592 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.975350+0000", 2024-09-13T14:50:11.592 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010293099999999999 2024-09-13T14:50:11.592 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.592 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.593 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 2,4", 2024-09-13T14:50:11.593 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.975471+0000", 2024-09-13T14:50:11.593 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000121635 2024-09-13T14:50:11.593 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.593 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.593 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:50:11.593 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.978945+0000", 2024-09-13T14:50:11.593 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0034742430000000001 2024-09-13T14:50:11.593 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.593 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.593 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:11.593 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.979837+0000", 2024-09-13T14:50:11.593 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00089157400000000001 2024-09-13T14:50:11.593 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.593 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.594 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:11.594 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:43.320782+0000", 2024-09-13T14:50:11.594 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.34094479300000002 2024-09-13T14:50:11.594 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.594 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.594 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:11.594 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:43.320824+0000", 2024-09-13T14:50:11.594 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1943000000000001e-05 2024-09-13T14:50:11.594 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.594 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.594 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:11.594 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:43.320858+0000", 2024-09-13T14:50:11.594 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4650999999999997e-05 2024-09-13T14:50:11.595 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:11.595 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:11.595 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:11.595 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.595 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.595 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.46157.0:226372 8.d 8:b15e2373:::benchmark_data_smithi102_313275_object95785:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e355)", 2024-09-13T14:50:11.595 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:49:42.987543+0000", 2024-09-13T14:50:11.595 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 28.520936571, 2024-09-13T14:50:11.595 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.33302733699999998, 2024-09-13T14:50:11.595 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:11.595 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:11.596 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:50:11.596 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.46157", 2024-09-13T14:50:11.596 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:50:11.596 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 226372 2024-09-13T14:50:11.596 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.596 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:11.596 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.596 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:11.596 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.987543+0000", 2024-09-13T14:50:11.596 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:11.596 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.597 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.597 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:11.597 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.987543+0000", 2024-09-13T14:50:11.597 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:11.597 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.597 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.597 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:11.597 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.987541+0000", 2024-09-13T14:50:11.597 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:50:11.597 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.597 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.597 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:11.597 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.987549+0000", 2024-09-13T14:50:11.598 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6990000000000003e-06 2024-09-13T14:50:11.598 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.598 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.598 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:11.598 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.987564+0000", 2024-09-13T14:50:11.598 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4960000000000001e-05 2024-09-13T14:50:11.598 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.598 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.598 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:11.598 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.987572+0000", 2024-09-13T14:50:11.598 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2160000000000006e-06 2024-09-13T14:50:11.598 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.599 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.599 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:11.599 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.987616+0000", 2024-09-13T14:50:11.599 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3244000000000001e-05 2024-09-13T14:50:11.599 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.599 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.599 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:11.599 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.987715+0000", 2024-09-13T14:50:11.599 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.8982999999999997e-05 2024-09-13T14:50:11.599 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.599 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.599 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 2,7", 2024-09-13T14:50:11.600 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.987803+0000", 2024-09-13T14:50:11.600 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8109000000000006e-05 2024-09-13T14:50:11.600 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.600 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.600 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:50:11.600 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.994816+0000", 2024-09-13T14:50:11.600 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0070131660000000004 2024-09-13T14:50:11.600 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.600 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.600 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:11.600 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.995264+0000", 2024-09-13T14:50:11.600 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00044798100000000002 2024-09-13T14:50:11.600 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.601 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.601 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:11.601 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:43.320507+0000", 2024-09-13T14:50:11.601 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.32524275699999999 2024-09-13T14:50:11.601 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.601 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.601 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:11.601 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:43.320536+0000", 2024-09-13T14:50:11.601 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9416000000000001e-05 2024-09-13T14:50:11.601 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.601 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.601 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:11.601 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:43.320570+0000", 2024-09-13T14:50:11.602 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4310000000000002e-05 2024-09-13T14:50:11.602 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:11.602 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:11.602 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:11.602 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.602 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.602 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.46157.0:226373 8.d 8:b4dc162c:::benchmark_data_smithi102_313275_object95786:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e355)", 2024-09-13T14:50:11.602 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:49:42.987587+0000", 2024-09-13T14:50:11.602 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 28.520892626999998, 2024-09-13T14:50:11.602 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.33311505200000002, 2024-09-13T14:50:11.602 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:11.602 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:11.602 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:50:11.603 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.46157", 2024-09-13T14:50:11.603 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:50:11.603 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 226373 2024-09-13T14:50:11.603 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.603 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:11.603 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.603 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:11.603 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.987587+0000", 2024-09-13T14:50:11.603 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:11.603 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.603 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.603 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:11.603 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.987587+0000", 2024-09-13T14:50:11.604 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:11.604 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.604 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.604 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:11.604 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.987586+0000", 2024-09-13T14:50:11.604 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:50:11.604 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.604 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.604 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:11.604 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.987590+0000", 2024-09-13T14:50:11.604 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4290000000000001e-06 2024-09-13T14:50:11.604 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.604 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.605 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:11.605 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.987597+0000", 2024-09-13T14:50:11.605 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2880000000000004e-06 2024-09-13T14:50:11.605 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.605 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.605 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:11.605 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.987602+0000", 2024-09-13T14:50:11.605 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4199999999999998e-06 2024-09-13T14:50:11.605 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.605 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.605 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:11.605 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.991101+0000", 2024-09-13T14:50:11.606 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003498784 2024-09-13T14:50:11.606 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.606 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.606 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:11.606 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.991189+0000", 2024-09-13T14:50:11.606 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8026000000000001e-05 2024-09-13T14:50:11.606 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.606 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.606 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 2,7", 2024-09-13T14:50:11.606 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.991273+0000", 2024-09-13T14:50:11.606 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.4157000000000001e-05 2024-09-13T14:50:11.606 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.606 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.607 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:50:11.607 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.994886+0000", 2024-09-13T14:50:11.607 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0036127799999999999 2024-09-13T14:50:11.607 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.607 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.607 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:11.607 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:42.995332+0000", 2024-09-13T14:50:11.607 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00044604099999999999 2024-09-13T14:50:11.607 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.607 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.607 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:11.607 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:43.320636+0000", 2024-09-13T14:50:11.608 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.32530344500000002 2024-09-13T14:50:11.608 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.608 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.608 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:11.608 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:43.320671+0000", 2024-09-13T14:50:11.608 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5667999999999998e-05 2024-09-13T14:50:11.608 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.608 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.608 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:11.608 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:43.320702+0000", 2024-09-13T14:50:11.608 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0561000000000001e-05 2024-09-13T14:50:11.608 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:11.608 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:11.608 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:11.609 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.609 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.609 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.46157.0:232498 8.8 8:17ae06d9:::benchmark_data_smithi102_313275_object101911:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e355)", 2024-09-13T14:50:11.609 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:49:50.690959+0000", 2024-09-13T14:50:11.609 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 20.817520206000001, 2024-09-13T14:50:11.609 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.129849618, 2024-09-13T14:50:11.609 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:11.609 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:11.609 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:50:11.609 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.46157", 2024-09-13T14:50:11.609 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:50:11.609 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 232498 2024-09-13T14:50:11.609 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.610 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:11.610 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.610 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:11.610 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:50.690959+0000", 2024-09-13T14:50:11.610 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:11.610 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.610 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.610 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:11.610 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:50.690959+0000", 2024-09-13T14:50:11.610 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:11.610 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.610 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.611 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:11.611 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:50.690958+0000", 2024-09-13T14:50:11.611 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:50:11.611 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.611 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.611 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:11.611 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:50.690964+0000", 2024-09-13T14:50:11.611 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.728e-06 2024-09-13T14:50:11.611 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.611 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.611 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:11.611 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:50.690975+0000", 2024-09-13T14:50:11.611 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0643e-05 2024-09-13T14:50:11.611 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.612 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.612 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:11.612 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:50.690980+0000", 2024-09-13T14:50:11.612 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2650000000000004e-06 2024-09-13T14:50:11.612 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.612 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.612 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:11.612 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:50.691017+0000", 2024-09-13T14:50:11.612 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7586000000000002e-05 2024-09-13T14:50:11.612 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.612 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.612 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:11.612 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:50.691136+0000", 2024-09-13T14:50:11.612 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000118353 2024-09-13T14:50:11.613 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.613 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.613 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:50:11.613 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:50.691263+0000", 2024-09-13T14:50:11.613 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012721600000000001 2024-09-13T14:50:11.613 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.613 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.613 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:11.614 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:50.698460+0000", 2024-09-13T14:50:11.614 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0071973870000000004 2024-09-13T14:50:11.614 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.614 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.614 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:50:11.614 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:50.698471+0000", 2024-09-13T14:50:11.614 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.063e-05 2024-09-13T14:50:11.614 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.614 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.614 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:11.615 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:50.820722+0000", 2024-09-13T14:50:11.615 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.12225122500000001 2024-09-13T14:50:11.615 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.615 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.615 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:11.615 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:50.820768+0000", 2024-09-13T14:50:11.615 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6233000000000003e-05 2024-09-13T14:50:11.615 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.615 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.615 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:11.615 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:50.820809+0000", 2024-09-13T14:50:11.615 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0402999999999999e-05 2024-09-13T14:50:11.615 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:11.615 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:11.616 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:11.616 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.616 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.616 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.46157.0:232499 8.8 8:15219e9f:::benchmark_data_smithi102_313275_object101912:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e355)", 2024-09-13T14:50:11.616 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:49:50.691195+0000", 2024-09-13T14:50:11.616 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 20.817284804, 2024-09-13T14:50:11.616 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.12974743599999999, 2024-09-13T14:50:11.616 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:11.616 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:11.616 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:50:11.616 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.46157", 2024-09-13T14:50:11.616 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:50:11.617 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 232499 2024-09-13T14:50:11.617 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.617 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:11.617 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.617 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:11.617 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:50.691195+0000", 2024-09-13T14:50:11.617 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:11.617 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.617 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.617 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:11.617 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:50.691195+0000", 2024-09-13T14:50:11.617 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:11.617 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.617 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.618 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:11.618 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:50.691194+0000", 2024-09-13T14:50:11.618 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:50:11.618 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.618 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.618 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:11.618 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:50.691198+0000", 2024-09-13T14:50:11.618 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4850000000000003e-06 2024-09-13T14:50:11.618 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.618 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.618 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:11.618 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:50.691208+0000", 2024-09-13T14:50:11.619 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.2830000000000007e-06 2024-09-13T14:50:11.619 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.619 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.619 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:11.619 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:50.691213+0000", 2024-09-13T14:50:11.619 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0370000000000002e-06 2024-09-13T14:50:11.619 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.619 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.619 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:11.619 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:50.695106+0000", 2024-09-13T14:50:11.619 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0038928449999999998 2024-09-13T14:50:11.619 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.619 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.620 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:11.620 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:50.695236+0000", 2024-09-13T14:50:11.620 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000130887 2024-09-13T14:50:11.620 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.620 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.620 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:50:11.620 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:50.695359+0000", 2024-09-13T14:50:11.620 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012296099999999999 2024-09-13T14:50:11.620 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.620 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.620 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:50:11.620 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:50.698693+0000", 2024-09-13T14:50:11.620 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00333396 2024-09-13T14:50:11.621 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.621 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.621 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:11.621 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:50.699643+0000", 2024-09-13T14:50:11.621 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00094920499999999995 2024-09-13T14:50:11.621 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.621 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.621 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:11.621 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:50.820859+0000", 2024-09-13T14:50:11.621 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.121216531 2024-09-13T14:50:11.621 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.621 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.621 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:11.622 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:50.820902+0000", 2024-09-13T14:50:11.622 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2543999999999998e-05 2024-09-13T14:50:11.622 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.622 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.622 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:11.622 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:50.820942+0000", 2024-09-13T14:50:11.622 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0473999999999999e-05 2024-09-13T14:50:11.622 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:11.622 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:11.622 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:11.622 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.622 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.623 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.46157.0:235195 8.8 8:156a3806:::benchmark_data_smithi102_313275_object104608:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e356)", 2024-09-13T14:50:11.623 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:49:54.152856+0000", 2024-09-13T14:50:11.623 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 17.35562345, 2024-09-13T14:50:11.623 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.20061816599999999, 2024-09-13T14:50:11.623 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:11.623 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:11.623 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:50:11.623 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.46157", 2024-09-13T14:50:11.623 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:50:11.623 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 235195 2024-09-13T14:50:11.623 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.623 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:11.623 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.624 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:11.624 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.152856+0000", 2024-09-13T14:50:11.624 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:11.624 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.624 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.624 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:11.624 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.152856+0000", 2024-09-13T14:50:11.624 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:11.624 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.624 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.624 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:11.624 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.152854+0000", 2024-09-13T14:50:11.624 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:50:11.625 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.625 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.625 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:11.625 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.152862+0000", 2024-09-13T14:50:11.625 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4229999999999999e-06 2024-09-13T14:50:11.625 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.625 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.625 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:11.625 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.152876+0000", 2024-09-13T14:50:11.625 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4545999999999999e-05 2024-09-13T14:50:11.625 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.625 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.625 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:11.625 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.152884+0000", 2024-09-13T14:50:11.626 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7100000000000007e-06 2024-09-13T14:50:11.626 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.626 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.626 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:11.626 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.152934+0000", 2024-09-13T14:50:11.626 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9628e-05 2024-09-13T14:50:11.626 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.626 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.626 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:11.626 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.153044+0000", 2024-09-13T14:50:11.626 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000110389 2024-09-13T14:50:11.626 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.626 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.626 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:50:11.627 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.153156+0000", 2024-09-13T14:50:11.627 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000111883 2024-09-13T14:50:11.627 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.627 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.627 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:50:11.627 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.160667+0000", 2024-09-13T14:50:11.627 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0075110039999999999 2024-09-13T14:50:11.627 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.627 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.627 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:11.627 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.161063+0000", 2024-09-13T14:50:11.627 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00039602099999999998 2024-09-13T14:50:11.627 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.627 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.628 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:11.628 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.353360+0000", 2024-09-13T14:50:11.628 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.19229674999999999 2024-09-13T14:50:11.628 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.628 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.628 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:11.628 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.353429+0000", 2024-09-13T14:50:11.628 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9387999999999994e-05 2024-09-13T14:50:11.628 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.628 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.628 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:11.628 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.353474+0000", 2024-09-13T14:50:11.628 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5228999999999997e-05 2024-09-13T14:50:11.628 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:11.629 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:11.629 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:11.629 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.629 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.629 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.46157.0:235196 8.8 8:1667ae0c:::benchmark_data_smithi102_313275_object104609:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e356)", 2024-09-13T14:50:11.629 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:49:54.154824+0000", 2024-09-13T14:50:11.629 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 17.353655731, 2024-09-13T14:50:11.629 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.20063396999999999, 2024-09-13T14:50:11.629 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:11.629 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:11.629 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:50:11.629 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.46157", 2024-09-13T14:50:11.629 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:50:11.629 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 235196 2024-09-13T14:50:11.630 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.630 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:11.630 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.630 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:11.630 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.154824+0000", 2024-09-13T14:50:11.630 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:11.630 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.630 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.630 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:11.630 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.154824+0000", 2024-09-13T14:50:11.630 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:11.630 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.630 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.631 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:11.631 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.154823+0000", 2024-09-13T14:50:11.631 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:50:11.631 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.631 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.631 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:11.631 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.154828+0000", 2024-09-13T14:50:11.631 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5960000000000003e-06 2024-09-13T14:50:11.631 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.631 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.631 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:11.631 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.154840+0000", 2024-09-13T14:50:11.631 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1667e-05 2024-09-13T14:50:11.631 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.632 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.633 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:11.633 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.154846+0000", 2024-09-13T14:50:11.633 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1530000000000002e-06 2024-09-13T14:50:11.633 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.633 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.633 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:11.633 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.156930+0000", 2024-09-13T14:50:11.633 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0020839280000000001 2024-09-13T14:50:11.633 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.634 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.634 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:11.634 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.157032+0000", 2024-09-13T14:50:11.634 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000102248 2024-09-13T14:50:11.634 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.634 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.634 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:50:11.634 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.157136+0000", 2024-09-13T14:50:11.634 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000103361 2024-09-13T14:50:11.634 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.634 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.634 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:50:11.634 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.160836+0000", 2024-09-13T14:50:11.635 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0037003909999999999 2024-09-13T14:50:11.635 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.635 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.635 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:11.635 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.161162+0000", 2024-09-13T14:50:11.635 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00032633299999999999 2024-09-13T14:50:11.635 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.635 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.635 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:11.635 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.355375+0000", 2024-09-13T14:50:11.635 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.19421283 2024-09-13T14:50:11.635 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.635 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.636 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:11.636 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.355423+0000", 2024-09-13T14:50:11.636 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8003000000000001e-05 2024-09-13T14:50:11.636 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.636 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.636 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:11.636 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.355458+0000", 2024-09-13T14:50:11.636 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4453999999999999e-05 2024-09-13T14:50:11.636 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:11.636 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:11.636 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:11.636 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.636 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.636 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.46157.0:235203 8.8 8:1660597d:::benchmark_data_smithi102_313275_object104616:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e356)", 2024-09-13T14:50:11.637 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:49:54.165107+0000", 2024-09-13T14:50:11.637 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 17.343372725999998, 2024-09-13T14:50:11.637 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.20280396000000001, 2024-09-13T14:50:11.637 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:11.637 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:11.637 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:50:11.637 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.46157", 2024-09-13T14:50:11.637 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:50:11.637 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 235203 2024-09-13T14:50:11.637 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.637 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:11.637 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.637 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:11.638 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.165107+0000", 2024-09-13T14:50:11.638 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:11.638 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.638 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.638 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:11.638 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.165107+0000", 2024-09-13T14:50:11.638 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:11.638 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.638 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.638 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:11.638 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.165106+0000", 2024-09-13T14:50:11.638 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:50:11.638 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.639 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.639 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:11.639 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.165111+0000", 2024-09-13T14:50:11.639 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8150000000000002e-06 2024-09-13T14:50:11.639 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.639 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.639 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:11.639 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.165122+0000", 2024-09-13T14:50:11.639 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0413e-05 2024-09-13T14:50:11.639 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.639 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.639 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:11.639 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.165128+0000", 2024-09-13T14:50:11.639 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.818e-06 2024-09-13T14:50:11.640 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.640 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.640 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:11.640 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.165174+0000", 2024-09-13T14:50:11.640 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6749000000000003e-05 2024-09-13T14:50:11.640 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.640 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.640 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:11.640 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.165271+0000", 2024-09-13T14:50:11.640 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6327000000000001e-05 2024-09-13T14:50:11.640 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.640 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.640 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:50:11.640 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.165399+0000", 2024-09-13T14:50:11.641 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012813399999999999 2024-09-13T14:50:11.641 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.641 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.641 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:50:11.641 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.169559+0000", 2024-09-13T14:50:11.641 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0041598970000000001 2024-09-13T14:50:11.641 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.641 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.641 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:11.641 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.169770+0000", 2024-09-13T14:50:11.641 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00021152500000000001 2024-09-13T14:50:11.641 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.641 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.642 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:11.642 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.367854+0000", 2024-09-13T14:50:11.642 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.198084017 2024-09-13T14:50:11.642 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.642 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.642 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:11.642 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.367884+0000", 2024-09-13T14:50:11.642 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0168e-05 2024-09-13T14:50:11.642 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.642 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.642 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:11.642 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.367911+0000", 2024-09-13T14:50:11.642 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.6247000000000001e-05 2024-09-13T14:50:11.643 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:11.643 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:11.643 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:11.643 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.643 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.643 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.46157.0:235212 8.8 8:1c7e01d0:::benchmark_data_smithi102_313275_object104625:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e356)", 2024-09-13T14:50:11.643 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:49:54.200738+0000", 2024-09-13T14:50:11.643 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 17.307741954000001, 2024-09-13T14:50:11.643 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.17083188199999999, 2024-09-13T14:50:11.643 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:11.643 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:11.643 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:50:11.643 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.46157", 2024-09-13T14:50:11.644 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:50:11.644 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 235212 2024-09-13T14:50:11.644 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.644 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:11.644 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.644 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:11.644 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.200738+0000", 2024-09-13T14:50:11.644 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:11.644 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.644 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.644 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:11.645 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.200738+0000", 2024-09-13T14:50:11.645 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:11.645 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.645 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.645 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:11.645 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.200735+0000", 2024-09-13T14:50:11.645 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-09-13T14:50:11.645 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.645 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.645 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:11.645 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.200748+0000", 2024-09-13T14:50:11.645 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2435999999999999e-05 2024-09-13T14:50:11.645 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.646 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.646 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:11.646 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.200777+0000", 2024-09-13T14:50:11.646 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9294000000000001e-05 2024-09-13T14:50:11.646 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.646 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.646 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:11.646 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.200790+0000", 2024-09-13T14:50:11.646 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3278999999999999e-05 2024-09-13T14:50:11.647 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.647 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.647 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:11.647 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.200879+0000", 2024-09-13T14:50:11.647 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8269000000000005e-05 2024-09-13T14:50:11.647 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.647 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.647 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:11.647 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.201076+0000", 2024-09-13T14:50:11.647 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000197281 2024-09-13T14:50:11.647 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.647 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.647 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:50:11.647 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.201158+0000", 2024-09-13T14:50:11.648 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1613999999999995e-05 2024-09-13T14:50:11.648 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.648 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.648 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:50:11.648 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.204303+0000", 2024-09-13T14:50:11.648 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0031456470000000001 2024-09-13T14:50:11.648 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.648 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.648 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:11.648 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.204627+0000", 2024-09-13T14:50:11.648 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00032361400000000001 2024-09-13T14:50:11.648 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.649 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.649 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:11.649 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.371484+0000", 2024-09-13T14:50:11.649 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.16685703599999999 2024-09-13T14:50:11.649 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.649 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.649 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:11.649 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.371531+0000", 2024-09-13T14:50:11.649 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6962000000000002e-05 2024-09-13T14:50:11.649 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:11.649 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:11.649 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:11.649 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:49:54.371569+0000", 2024-09-13T14:50:11.650 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8622999999999999e-05 2024-09-13T14:50:11.650 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:11.650 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:11.650 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:11.650 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:11.650 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:11.650 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:50:11.687 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:11.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:11.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:11.789 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:11.826 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:11.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:11.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:11.928 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:11.965 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:11.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:11.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:12.067 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:12.089 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:12.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:12.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:12.191 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:12.227 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:12.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:12.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:12.329 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:12.366 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:12.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:12.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:12.468 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:12.501 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:12.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:12.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:12.603 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:12.627 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:12.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:12.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:12.659 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T14:50:12.729 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:12.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:12.766 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:12.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:12.867 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:12.905 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:12.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:12.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:12.983 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:50:13.007 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:13.039 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:13.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:13.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:13.141 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:13.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:13.178 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:13.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:13.279 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:13.317 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:13.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:13.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:13.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:13 smithi102 ceph-mon[110183]: pgmap v1256: 109 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 105 active+clean; 915 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 639 op/s; 3344/43800 objects misplaced (7.635%); 3.3 MiB/s, 52 objects/s recovering 2024-09-13T14:50:13.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:13 smithi102 ceph-mon[115269]: pgmap v1256: 109 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 105 active+clean; 915 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 639 op/s; 3344/43800 objects misplaced (7.635%); 3.3 MiB/s, 52 objects/s recovering 2024-09-13T14:50:13.419 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:13.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:13.449 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:13.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:13.458 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:13 smithi195 ceph-mon[104262]: pgmap v1256: 109 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 105 active+clean; 915 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 639 op/s; 3344/43800 objects misplaced (7.635%); 3.3 MiB/s, 52 objects/s recovering 2024-09-13T14:50:13.550 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:13.572 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:13.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:13.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:13.651 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:50:13.673 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:13.694 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:13.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:13.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:13.796 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:13.828 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:13.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:13.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:13.841 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:13.930 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:13.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:13.953 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:13.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:14.054 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:14.077 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:14.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:14.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:14.179 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:14.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:14.203 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:14.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:14.304 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:14.326 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:14.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:14.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:14.358 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:14 smithi102 ceph-mon[110183]: osdmap e361: 8 total, 8 up, 8 in 2024-09-13T14:50:14.359 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:14 smithi102 ceph-mon[115269]: osdmap e361: 8 total, 8 up, 8 in 2024-09-13T14:50:14.428 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:14.430 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:14 smithi195 ceph-mon[104262]: osdmap e361: 8 total, 8 up, 8 in 2024-09-13T14:50:14.451 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:14.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:14.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:14.553 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:14.575 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:14.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:14.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:14.677 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:14.718 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:14.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:14.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:14.820 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:14.826 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:50:14.826 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-13T14:50:14.826 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:14.826 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.46157.0:250243 8.5 e362/304 8:af056ac4:::benchmark_data_smithi102_313275_object119656:head v 362'15607, mlcod=362'15605)", 2024-09-13T14:50:14.826 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:50:14.822994+0000", 2024-09-13T14:50:14.826 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.0011231559999999999, 2024-09-13T14:50:14.826 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001149445, 2024-09-13T14:50:14.826 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:14.827 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T14:50:14.827 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:14.827 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:14.827 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:14.827 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:50:14.822994+0000", 2024-09-13T14:50:14.827 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:14.827 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:14.827 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:14.827 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:14.827 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:50:14.822994+0000", 2024-09-13T14:50:14.827 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:14.827 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:14.828 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:14.828 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:14.828 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:50:14.822993+0000", 2024-09-13T14:50:14.828 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:50:14.828 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:14.828 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:14.828 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:14.828 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:50:14.823001+0000", 2024-09-13T14:50:14.828 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0970000000000006e-06 2024-09-13T14:50:14.828 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:14.828 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:14.828 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:14.829 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:50:14.823017+0000", 2024-09-13T14:50:14.829 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5373999999999999e-05 2024-09-13T14:50:14.829 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:14.829 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:14.829 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:14.829 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:50:14.823024+0000", 2024-09-13T14:50:14.829 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7309999999999996e-06 2024-09-13T14:50:14.829 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:14.830 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:14.830 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:14.830 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:50:14.823106+0000", 2024-09-13T14:50:14.830 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2354999999999993e-05 2024-09-13T14:50:14.830 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:14.830 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:14.830 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:14.830 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:50:14.823126+0000", 2024-09-13T14:50:14.830 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0123e-05 2024-09-13T14:50:14.830 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:14.830 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:14.830 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:14.831 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:14.831 INFO:teuthology.orchestra.run.smithi195.stdout: ], 2024-09-13T14:50:14.831 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 1 2024-09-13T14:50:14.831 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:50:14.860 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:14.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:14.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:14.962 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:15.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:15.019 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:15.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:15.120 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:15.156 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:15.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:15.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:15.230 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:15 smithi102 ceph-mon[110183]: pgmap v1258: 109 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 105 active+clean; 783 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 911 op/s; 2175/37437 objects misplaced (5.810%); 6.4 MiB/s, 102 objects/s recovering 2024-09-13T14:50:15.231 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:15 smithi102 ceph-mon[110183]: osdmap e362: 8 total, 8 up, 8 in 2024-09-13T14:50:15.231 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:15 smithi102 ceph-mon[115269]: pgmap v1258: 109 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 105 active+clean; 783 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 911 op/s; 2175/37437 objects misplaced (5.810%); 6.4 MiB/s, 102 objects/s recovering 2024-09-13T14:50:15.231 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:15 smithi102 ceph-mon[115269]: osdmap e362: 8 total, 8 up, 8 in 2024-09-13T14:50:15.235 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:50:15.238 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:50:15.258 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:15.283 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:15.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:15.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:15.385 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:15.416 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:15.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:15.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:15.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:15 smithi195 ceph-mon[104262]: pgmap v1258: 109 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 105 active+clean; 783 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 911 op/s; 2175/37437 objects misplaced (5.810%); 6.4 MiB/s, 102 objects/s recovering 2024-09-13T14:50:15.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:15 smithi195 ceph-mon[104262]: osdmap e362: 8 total, 8 up, 8 in 2024-09-13T14:50:15.518 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:15.541 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:15.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:15.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:15.544 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T14:50:15.643 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:15.665 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:15.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:15.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:15.745 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:50:15.767 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:15.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:15.790 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:15.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:15.891 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:15.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:15.914 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:15.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:16.015 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:16.038 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:16.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:16.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:16.139 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:16.161 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:16.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:16.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:16.263 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:16.292 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:16.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:16.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:16.393 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:16.400 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:16.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:16 smithi102 ceph-mon[110183]: osdmap e363: 8 total, 8 up, 8 in 2024-09-13T14:50:16.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:16 smithi102 ceph-mon[110183]: from='client.36435 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:50:16.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:16 smithi102 ceph-mon[115269]: osdmap e363: 8 total, 8 up, 8 in 2024-09-13T14:50:16.421 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:16 smithi102 ceph-mon[115269]: from='client.36435 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:50:16.459 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:16.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:16.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:16.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:16 smithi195 ceph-mon[104262]: osdmap e363: 8 total, 8 up, 8 in 2024-09-13T14:50:16.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:16 smithi195 ceph-mon[104262]: from='client.36435 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:50:16.560 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:16.590 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:16.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:16.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:16.692 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:16.720 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:16.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:16.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:16.821 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:16.849 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:16.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:16.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:16.918 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:50:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:50:16] "GET /metrics HTTP/1.1" 200 36528 "" "Prometheus/2.51.0" 2024-09-13T14:50:16.951 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:16.959 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":1260,"stamp":"2024-09-13T14:50:15.196949+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":791091197,"num_objects":12027,"num_object_clones":0,"num_object_copies":36081,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2175,"num_objects_unfound":0,"num_objects_dirty":12027,"num_whiteouts":0,"num_read":5999,"num_read_kb":13012,"num_write":250079,"num_write_kb":8370002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12231,"num_bytes_recovered":797378704,"num_keys_recovered":20,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122370,"ondisk_log_size":122370,"up":327,"acting":327,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":332,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":7458160,"kb_used_data":2662248,"kb_used_omap":41,"kb_used_meta":4795350,"kb_avail":742503056,"statfs":{"total":767960285184,"available":760323129344,"internally_reserved":0,"allocated":2726141952,"data_stored":2711110991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42913,"internal_metadata":4910438495},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,1,5,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-414056448,"num_objects":-6318,"num_object_clones":0,"num_object_copies":-18954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2213,"num_objects_unfound":0,"num_objects_dirty":-6318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6318,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":818,"num_bytes_recovered":53608448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.528418"},"pg_stats":[{"pgid":"1.11","version":"90'149","reported_seq":461,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T14:50:07.117447+0000","last_change":"2024-09-13T14:47:38.688630+0000","last_active":"2024-09-13T14:50:07.117447+0000","last_peered":"2024-09-13T14:50:07.117447+0000","last_clean":"2024-09-13T14:50:07.117447+0000","last_became_active":"2024-09-13T14:47:38.460077+0000","last_became_peered":"2024-09-13T14:47:38.460077+0000","last_unstale":"2024-09-13T14:50:07.117447+0000","last_undegraded":"2024-09-13T14:50:07.117447+0000","last_fullsized":"2024-09-13T14:50:07.117447+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:38.688341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.688341+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:14:17.232921+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":630,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T14:50:07.145821+0000","last_change":"2024-09-13T14:47:47.868783+0000","last_active":"2024-09-13T14:50:07.145821+0000","last_peered":"2024-09-13T14:50:07.145821+0000","last_clean":"2024-09-13T14:50:07.145821+0000","last_became_active":"2024-09-13T14:47:47.868478+0000","last_became_peered":"2024-09-13T14:47:47.868478+0000","last_unstale":"2024-09-13T14:50:07.145821+0000","last_undegraded":"2024-09-13T14:50:07.145821+0000","last_fullsized":"2024-09-13T14:50:07.145821+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:27:01.784142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":380,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T14:50:03.102402+0000","last_change":"2024-09-13T14:46:01.772811+0000","last_active":"2024-09-13T14:50:03.102402+0000","last_peered":"2024-09-13T14:50:03.102402+0000","last_clean":"2024-09-13T14:50:03.102402+0000","last_became_active":"2024-09-13T14:46:01.764775+0000","last_became_peered":"2024-09-13T14:46:01.764775+0000","last_unstale":"2024-09-13T14:50:03.102402+0000","last_undegraded":"2024-09-13T14:50:03.102402+0000","last_fullsized":"2024-09-13T14:50:03.102402+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:42:20.738421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"90'149","reported_seq":425,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T14:50:03.104454+0000","last_change":"2024-09-13T14:47:41.482552+0000","last_active":"2024-09-13T14:50:03.104454+0000","last_peered":"2024-09-13T14:50:03.104454+0000","last_clean":"2024-09-13T14:50:03.104454+0000","last_became_active":"2024-09-13T14:47:38.466659+0000","last_became_peered":"2024-09-13T14:47:38.466659+0000","last_unstale":"2024-09-13T14:50:03.104454+0000","last_undegraded":"2024-09-13T14:50:03.104454+0000","last_fullsized":"2024-09-13T14:50:03.104454+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:41.482364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:41.482364+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:50:49.239389+0000","scrub_duration":0.027843855000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":161,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T14:50:03.105681+0000","last_change":"2024-09-13T14:36:13.862181+0000","last_active":"2024-09-13T14:50:03.105681+0000","last_peered":"2024-09-13T14:50:03.105681+0000","last_clean":"2024-09-13T14:50:03.105681+0000","last_became_active":"2024-09-13T14:36:13.861873+0000","last_became_peered":"2024-09-13T14:36:13.861873+0000","last_unstale":"2024-09-13T14:50:03.105681+0000","last_undegraded":"2024-09-13T14:50:03.105681+0000","last_fullsized":"2024-09-13T14:50:03.105681+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:14.085700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":254,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T14:50:07.117413+0000","last_change":"2024-09-13T14:38:53.896672+0000","last_active":"2024-09-13T14:50:07.117413+0000","last_peered":"2024-09-13T14:50:07.117413+0000","last_clean":"2024-09-13T14:50:07.117413+0000","last_became_active":"2024-09-13T14:38:53.896346+0000","last_became_peered":"2024-09-13T14:38:53.896346+0000","last_unstale":"2024-09-13T14:50:07.117413+0000","last_undegraded":"2024-09-13T14:50:07.117413+0000","last_fullsized":"2024-09-13T14:50:07.117413+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:57.789686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"90'149","reported_seq":485,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-09-13T14:50:14.173678+0000","last_change":"2024-09-13T14:47:51.226821+0000","last_active":"2024-09-13T14:50:14.173678+0000","last_peered":"2024-09-13T14:50:14.173678+0000","last_clean":"2024-09-13T14:50:14.173678+0000","last_became_active":"2024-09-13T14:47:50.881689+0000","last_became_peered":"2024-09-13T14:47:50.881689+0000","last_unstale":"2024-09-13T14:50:14.173678+0000","last_undegraded":"2024-09-13T14:50:14.173678+0000","last_fullsized":"2024-09-13T14:50:14.173678+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:51.226748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:51.226748+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:06:02.065990+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":253,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T14:50:07.119334+0000","last_change":"2024-09-13T14:38:53.893021+0000","last_active":"2024-09-13T14:50:07.119334+0000","last_peered":"2024-09-13T14:50:07.119334+0000","last_clean":"2024-09-13T14:50:07.119334+0000","last_became_active":"2024-09-13T14:38:53.892668+0000","last_became_peered":"2024-09-13T14:38:53.892668+0000","last_unstale":"2024-09-13T14:50:07.119334+0000","last_undegraded":"2024-09-13T14:50:07.119334+0000","last_fullsized":"2024-09-13T14:50:07.119334+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:57.180738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":333,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T14:50:03.102276+0000","last_change":"2024-09-13T14:37:48.134642+0000","last_active":"2024-09-13T14:50:03.102276+0000","last_peered":"2024-09-13T14:50:03.102276+0000","last_clean":"2024-09-13T14:50:03.102276+0000","last_became_active":"2024-09-13T14:37:48.134402+0000","last_became_peered":"2024-09-13T14:37:48.134402+0000","last_unstale":"2024-09-13T14:50:03.102276+0000","last_undegraded":"2024-09-13T14:50:03.102276+0000","last_fullsized":"2024-09-13T14:50:03.102276+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:51.313663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"90'149","reported_seq":442,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T14:50:03.105368+0000","last_change":"2024-09-13T14:47:50.892667+0000","last_active":"2024-09-13T14:50:03.105368+0000","last_peered":"2024-09-13T14:50:03.105368+0000","last_clean":"2024-09-13T14:50:03.105368+0000","last_became_active":"2024-09-13T14:47:50.883312+0000","last_became_peered":"2024-09-13T14:47:50.883312+0000","last_unstale":"2024-09-13T14:50:03.105368+0000","last_undegraded":"2024-09-13T14:50:03.105368+0000","last_fullsized":"2024-09-13T14:50:03.105368+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:45.441819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:45.441819+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:39.138573+0000","scrub_duration":0.021368558999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":649,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T14:50:07.146968+0000","last_change":"2024-09-13T14:47:47.867624+0000","last_active":"2024-09-13T14:50:07.146968+0000","last_peered":"2024-09-13T14:50:07.146968+0000","last_clean":"2024-09-13T14:50:07.146968+0000","last_became_active":"2024-09-13T14:47:47.867016+0000","last_became_peered":"2024-09-13T14:47:47.867016+0000","last_unstale":"2024-09-13T14:50:07.146968+0000","last_undegraded":"2024-09-13T14:50:07.146968+0000","last_fullsized":"2024-09-13T14:50:07.146968+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:55:45.344929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":257,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T14:50:07.116796+0000","last_change":"2024-09-13T14:38:03.554500+0000","last_active":"2024-09-13T14:50:07.116796+0000","last_peered":"2024-09-13T14:50:07.116796+0000","last_clean":"2024-09-13T14:50:07.116796+0000","last_became_active":"2024-09-13T14:38:03.532910+0000","last_became_peered":"2024-09-13T14:38:03.532910+0000","last_unstale":"2024-09-13T14:50:07.116796+0000","last_undegraded":"2024-09-13T14:50:07.116796+0000","last_fullsized":"2024-09-13T14:50:07.116796+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:27:50.843498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"90'149","reported_seq":444,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T14:50:03.102560+0000","last_change":"2024-09-13T14:47:50.887140+0000","last_active":"2024-09-13T14:50:03.102560+0000","last_peered":"2024-09-13T14:50:03.102560+0000","last_clean":"2024-09-13T14:50:03.102560+0000","last_became_active":"2024-09-13T14:47:50.886603+0000","last_became_peered":"2024-09-13T14:47:50.886603+0000","last_unstale":"2024-09-13T14:50:03.102560+0000","last_undegraded":"2024-09-13T14:50:03.102560+0000","last_fullsized":"2024-09-13T14:50:03.102560+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:40.786610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:40.786610+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:00.936651+0000","scrub_duration":0.013725185000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.4","version":"360'15393","reported_seq":15468,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T14:50:11.416449+0000","last_change":"2024-09-13T14:46:01.762751+0000","last_active":"2024-09-13T14:50:11.416449+0000","last_peered":"2024-09-13T14:50:11.416449+0000","last_clean":"2024-09-13T14:50:11.416449+0000","last_became_active":"2024-09-13T14:46:01.761929+0000","last_became_peered":"2024-09-13T14:46:01.761929+0000","last_unstale":"2024-09-13T14:50:11.416449+0000","last_undegraded":"2024-09-13T14:50:11.416449+0000","last_fullsized":"2024-09-13T14:50:11.416449+0000","mapping_epoch":306,"log_start":"332'7623","ondisk_log_start":"332'7623","created":304,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":7770,"log_dups_size":0,"ondisk_log_size":7770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:20:57.325962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49479680,"num_objects":755,"num_object_clones":0,"num_object_copies":2265,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15393,"num_write_kb":516736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":125,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T14:50:03.102853+0000","last_change":"2024-09-13T14:38:53.940238+0000","last_active":"2024-09-13T14:50:03.102853+0000","last_peered":"2024-09-13T14:50:03.102853+0000","last_clean":"2024-09-13T14:50:03.102853+0000","last_became_active":"2024-09-13T14:38:53.892825+0000","last_became_peered":"2024-09-13T14:38:53.892825+0000","last_unstale":"2024-09-13T14:50:03.102853+0000","last_undegraded":"2024-09-13T14:50:03.102853+0000","last_fullsized":"2024-09-13T14:50:03.102853+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:46.666581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":182,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T14:50:03.104677+0000","last_change":"2024-09-13T14:38:53.898907+0000","last_active":"2024-09-13T14:50:03.104677+0000","last_peered":"2024-09-13T14:50:03.104677+0000","last_clean":"2024-09-13T14:50:03.104677+0000","last_became_active":"2024-09-13T14:38:53.898572+0000","last_became_peered":"2024-09-13T14:38:53.898572+0000","last_unstale":"2024-09-13T14:50:03.104677+0000","last_undegraded":"2024-09-13T14:50:03.104677+0000","last_fullsized":"2024-09-13T14:50:03.104677+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:29.222150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"90'149","reported_seq":503,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T14:50:07.145682+0000","last_change":"2024-09-13T14:47:38.466101+0000","last_active":"2024-09-13T14:50:07.145682+0000","last_peered":"2024-09-13T14:50:07.145682+0000","last_clean":"2024-09-13T14:50:07.145682+0000","last_became_active":"2024-09-13T14:47:38.465607+0000","last_became_peered":"2024-09-13T14:47:38.465607+0000","last_unstale":"2024-09-13T14:50:07.145682+0000","last_undegraded":"2024-09-13T14:50:07.145682+0000","last_fullsized":"2024-09-13T14:50:07.145682+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:32.060817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:32.060817+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:55.362724+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.5","version":"360'15481","reported_seq":15547,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T14:50:11.255454+0000","last_change":"2024-09-13T14:46:00.028852+0000","last_active":"2024-09-13T14:50:11.255454+0000","last_peered":"2024-09-13T14:50:11.255454+0000","last_clean":"2024-09-13T14:50:11.255454+0000","last_became_active":"2024-09-13T14:46:00.028414+0000","last_became_peered":"2024-09-13T14:46:00.028414+0000","last_unstale":"2024-09-13T14:50:11.255454+0000","last_undegraded":"2024-09-13T14:50:11.255454+0000","last_fullsized":"2024-09-13T14:50:11.255454+0000","mapping_epoch":304,"log_start":"341'9197","ondisk_log_start":"341'9197","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":6284,"log_dups_size":0,"ondisk_log_size":6284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:03:28.278631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53542912,"num_objects":817,"num_object_clones":0,"num_object_copies":2451,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15481,"num_write_kb":521536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":138,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T14:50:03.102804+0000","last_change":"2024-09-13T14:47:47.878423+0000","last_active":"2024-09-13T14:50:03.102804+0000","last_peered":"2024-09-13T14:50:03.102804+0000","last_clean":"2024-09-13T14:50:03.102804+0000","last_became_active":"2024-09-13T14:47:47.866926+0000","last_became_peered":"2024-09-13T14:47:47.866926+0000","last_unstale":"2024-09-13T14:50:03.102804+0000","last_undegraded":"2024-09-13T14:50:03.102804+0000","last_fullsized":"2024-09-13T14:50:03.102804+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:11:07.534097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":380,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T14:50:03.184426+0000","last_change":"2024-09-13T14:47:47.860804+0000","last_active":"2024-09-13T14:50:03.184426+0000","last_peered":"2024-09-13T14:50:03.184426+0000","last_clean":"2024-09-13T14:50:03.184426+0000","last_became_active":"2024-09-13T14:47:47.860496+0000","last_became_peered":"2024-09-13T14:47:47.860496+0000","last_unstale":"2024-09-13T14:50:03.184426+0000","last_undegraded":"2024-09-13T14:50:03.184426+0000","last_fullsized":"2024-09-13T14:50:03.184426+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:17:57.542599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.f","version":"90'149","reported_seq":469,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T14:50:03.105639+0000","last_change":"2024-09-13T14:47:38.468287+0000","last_active":"2024-09-13T14:50:03.105639+0000","last_peered":"2024-09-13T14:50:03.105639+0000","last_clean":"2024-09-13T14:50:03.105639+0000","last_became_active":"2024-09-13T14:47:38.467941+0000","last_became_peered":"2024-09-13T14:47:38.467941+0000","last_unstale":"2024-09-13T14:50:03.105639+0000","last_undegraded":"2024-09-13T14:50:03.105639+0000","last_fullsized":"2024-09-13T14:50:03.105639+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:34.091782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:34.091782+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:34:20.519248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"360'15534","reported_seq":15656,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T14:50:12.981484+0000","last_change":"2024-09-13T14:46:00.027675+0000","last_active":"2024-09-13T14:50:12.981484+0000","last_peered":"2024-09-13T14:50:12.981484+0000","last_clean":"2024-09-13T14:50:12.981484+0000","last_became_active":"2024-09-13T14:46:00.027364+0000","last_became_peered":"2024-09-13T14:46:00.027364+0000","last_unstale":"2024-09-13T14:50:12.981484+0000","last_undegraded":"2024-09-13T14:50:12.981484+0000","last_fullsized":"2024-09-13T14:50:12.981484+0000","mapping_epoch":304,"log_start":"332'7578","ondisk_log_start":"332'7578","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":7956,"log_dups_size":0,"ondisk_log_size":7956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:42:37.193007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46661632,"num_objects":712,"num_object_clones":0,"num_object_copies":2136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15534,"num_write_kb":519872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":363,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T14:50:03.102874+0000","last_change":"2024-09-13T14:37:49.611774+0000","last_active":"2024-09-13T14:50:03.102874+0000","last_peered":"2024-09-13T14:50:03.102874+0000","last_clean":"2024-09-13T14:50:03.102874+0000","last_became_active":"2024-09-13T14:37:49.611088+0000","last_became_peered":"2024-09-13T14:37:49.611088+0000","last_unstale":"2024-09-13T14:50:03.102874+0000","last_undegraded":"2024-09-13T14:50:03.102874+0000","last_fullsized":"2024-09-13T14:50:03.102874+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:08.933213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":63,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-09-13T14:50:14.173513+0000","last_change":"2024-09-13T14:47:47.865523+0000","last_active":"2024-09-13T14:50:14.173513+0000","last_peered":"2024-09-13T14:50:14.173513+0000","last_clean":"2024-09-13T14:50:14.173513+0000","last_became_active":"2024-09-13T14:47:47.865180+0000","last_became_peered":"2024-09-13T14:47:47.865180+0000","last_unstale":"2024-09-13T14:50:14.173513+0000","last_undegraded":"2024-09-13T14:50:14.173513+0000","last_fullsized":"2024-09-13T14:50:14.173513+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:39:35.479251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.e","version":"90'149","reported_seq":475,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T14:50:03.185223+0000","last_change":"2024-09-13T14:47:38.459769+0000","last_active":"2024-09-13T14:50:03.185223+0000","last_peered":"2024-09-13T14:50:03.185223+0000","last_clean":"2024-09-13T14:50:03.185223+0000","last_became_active":"2024-09-13T14:47:38.459440+0000","last_became_peered":"2024-09-13T14:47:38.459440+0000","last_unstale":"2024-09-13T14:50:03.185223+0000","last_undegraded":"2024-09-13T14:50:03.185223+0000","last_fullsized":"2024-09-13T14:50:03.185223+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:30.526999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:30.526999+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:00.016329+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"361'15727","reported_seq":27245,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-09-13T14:50:14.174141+0000","last_change":"2024-09-13T14:49:07.729693+0000","last_active":"2024-09-13T14:50:14.174141+0000","last_peered":"2024-09-13T14:50:14.174141+0000","last_clean":"2024-09-13T14:50:14.174141+0000","last_became_active":"2024-09-13T14:49:07.685432+0000","last_became_peered":"2024-09-13T14:49:07.685432+0000","last_unstale":"2024-09-13T14:50:14.174141+0000","last_undegraded":"2024-09-13T14:50:14.174141+0000","last_fullsized":"2024-09-13T14:50:14.174141+0000","mapping_epoch":348,"log_start":"341'9274","ondisk_log_start":"341'9274","created":304,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":6453,"log_dups_size":0,"ondisk_log_size":6453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:32:31.253024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43450368,"num_objects":663,"num_object_clones":0,"num_object_copies":1989,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15727,"num_write_kb":524480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5685,"num_bytes_recovered":372572160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":173,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T14:50:03.105512+0000","last_change":"2024-09-13T14:37:49.168003+0000","last_active":"2024-09-13T14:50:03.105512+0000","last_peered":"2024-09-13T14:50:03.105512+0000","last_clean":"2024-09-13T14:50:03.105512+0000","last_became_active":"2024-09-13T14:37:49.167638+0000","last_became_peered":"2024-09-13T14:37:49.167638+0000","last_unstale":"2024-09-13T14:50:03.105512+0000","last_undegraded":"2024-09-13T14:50:03.105512+0000","last_fullsized":"2024-09-13T14:50:03.105512+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:29.051981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":254,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T14:50:07.118353+0000","last_change":"2024-09-13T14:47:47.868094+0000","last_active":"2024-09-13T14:50:07.118353+0000","last_peered":"2024-09-13T14:50:07.118353+0000","last_clean":"2024-09-13T14:50:07.118353+0000","last_became_active":"2024-09-13T14:47:47.867750+0000","last_became_peered":"2024-09-13T14:47:47.867750+0000","last_unstale":"2024-09-13T14:50:07.118353+0000","last_undegraded":"2024-09-13T14:50:07.118353+0000","last_fullsized":"2024-09-13T14:50:07.118353+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:55:38.203481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"90'149","reported_seq":535,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T14:50:03.102173+0000","last_change":"2024-09-13T14:46:55.355650+0000","last_active":"2024-09-13T14:50:03.102173+0000","last_peered":"2024-09-13T14:50:03.102173+0000","last_clean":"2024-09-13T14:50:03.102173+0000","last_became_active":"2024-09-13T14:46:48.150338+0000","last_became_peered":"2024-09-13T14:46:48.150338+0000","last_unstale":"2024-09-13T14:50:03.102173+0000","last_undegraded":"2024-09-13T14:50:03.102173+0000","last_fullsized":"2024-09-13T14:50:03.102173+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:46:55.355521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:46:55.355521+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:30:53.006278+0000","scrub_duration":0.002025448,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.0","version":"360'15536","reported_seq":21870,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T14:50:13.032866+0000","last_change":"2024-09-13T14:49:40.891281+0000","last_active":"2024-09-13T14:50:13.032866+0000","last_peered":"2024-09-13T14:50:13.032866+0000","last_clean":"2024-09-13T14:50:13.032866+0000","last_became_active":"2024-09-13T14:49:40.854627+0000","last_became_peered":"2024-09-13T14:49:40.854627+0000","last_unstale":"2024-09-13T14:50:13.032866+0000","last_undegraded":"2024-09-13T14:50:13.032866+0000","last_fullsized":"2024-09-13T14:50:13.032866+0000","mapping_epoch":353,"log_start":"340'8806","ondisk_log_start":"340'8806","created":304,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":6730,"log_dups_size":0,"ondisk_log_size":6730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:34:27.653199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44040192,"num_objects":672,"num_object_clones":0,"num_object_copies":2016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15536,"num_write_kb":518656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3095,"num_bytes_recovered":202833920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":605,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T14:50:07.145912+0000","last_change":"2024-09-13T14:38:03.549595+0000","last_active":"2024-09-13T14:50:07.145912+0000","last_peered":"2024-09-13T14:50:07.145912+0000","last_clean":"2024-09-13T14:50:07.145912+0000","last_became_active":"2024-09-13T14:38:03.533331+0000","last_became_peered":"2024-09-13T14:38:03.533331+0000","last_unstale":"2024-09-13T14:50:07.145912+0000","last_undegraded":"2024-09-13T14:50:07.145912+0000","last_fullsized":"2024-09-13T14:50:07.145912+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:59:02.470343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":356,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T14:50:03.187528+0000","last_change":"2024-09-13T14:46:26.980367+0000","last_active":"2024-09-13T14:50:03.187528+0000","last_peered":"2024-09-13T14:50:03.187528+0000","last_clean":"2024-09-13T14:50:03.187528+0000","last_became_active":"2024-09-13T14:46:26.979937+0000","last_became_peered":"2024-09-13T14:46:26.979937+0000","last_unstale":"2024-09-13T14:50:03.187528+0000","last_undegraded":"2024-09-13T14:50:03.187528+0000","last_fullsized":"2024-09-13T14:50:03.187528+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:56.830872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"90'149","reported_seq":437,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T14:50:03.104537+0000","last_change":"2024-09-13T14:46:54.393663+0000","last_active":"2024-09-13T14:50:03.104537+0000","last_peered":"2024-09-13T14:50:03.104537+0000","last_clean":"2024-09-13T14:50:03.104537+0000","last_became_active":"2024-09-13T14:46:48.149004+0000","last_became_peered":"2024-09-13T14:46:48.149004+0000","last_unstale":"2024-09-13T14:50:03.104537+0000","last_undegraded":"2024-09-13T14:50:03.104537+0000","last_fullsized":"2024-09-13T14:50:03.104537+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:46:54.393204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:46:54.393204+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:32:40.003989+0000","scrub_duration":0.015172223,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1","version":"360'15457","reported_seq":16109,"reported_epoch":360,"state":"active+remapped+backfilling","last_fresh":"2024-09-13T14:50:12.987681+0000","last_change":"2024-09-13T14:50:12.207736+0000","last_active":"2024-09-13T14:50:12.987681+0000","last_peered":"2024-09-13T14:50:12.987681+0000","last_clean":"2024-09-13T14:47:46.837670+0000","last_became_active":"2024-09-13T14:47:49.256170+0000","last_became_peered":"2024-09-13T14:47:49.256170+0000","last_unstale":"2024-09-13T14:50:12.987681+0000","last_undegraded":"2024-09-13T14:50:12.987681+0000","last_fullsized":"2024-09-13T14:50:12.987681+0000","mapping_epoch":337,"log_start":"332'7526","ondisk_log_start":"332'7526","created":304,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":7931,"log_dups_size":0,"ondisk_log_size":7931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49348608,"num_objects":753,"num_object_clones":0,"num_object_copies":2259,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":688,"num_objects_unfound":0,"num_objects_dirty":753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15457,"num_write_kb":518720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":257,"num_bytes_recovered":16842752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[0,2,5],"avail_no_missing":["0","2","5"],"object_location_counts":[{"shards":"0,2,5","objects":753}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":289,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T14:50:03.185755+0000","last_change":"2024-09-13T14:47:47.867571+0000","last_active":"2024-09-13T14:50:03.185755+0000","last_peered":"2024-09-13T14:50:03.185755+0000","last_clean":"2024-09-13T14:50:03.185755+0000","last_became_active":"2024-09-13T14:47:47.867269+0000","last_became_peered":"2024-09-13T14:47:47.867269+0000","last_unstale":"2024-09-13T14:50:03.185755+0000","last_undegraded":"2024-09-13T14:50:03.185755+0000","last_fullsized":"2024-09-13T14:50:03.185755+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:46.093600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":418,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T14:50:03.184904+0000","last_change":"2024-09-13T14:47:47.866925+0000","last_active":"2024-09-13T14:50:03.184904+0000","last_peered":"2024-09-13T14:50:03.184904+0000","last_clean":"2024-09-13T14:50:03.184904+0000","last_became_active":"2024-09-13T14:47:47.866596+0000","last_became_peered":"2024-09-13T14:47:47.866596+0000","last_unstale":"2024-09-13T14:50:03.184904+0000","last_undegraded":"2024-09-13T14:50:03.184904+0000","last_fullsized":"2024-09-13T14:50:03.184904+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:11:25.617891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"90'149","reported_seq":498,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-09-13T14:50:14.173445+0000","last_change":"2024-09-13T14:47:50.891055+0000","last_active":"2024-09-13T14:50:14.173445+0000","last_peered":"2024-09-13T14:50:14.173445+0000","last_clean":"2024-09-13T14:50:14.173445+0000","last_became_active":"2024-09-13T14:47:50.890368+0000","last_became_peered":"2024-09-13T14:47:50.890368+0000","last_unstale":"2024-09-13T14:50:14.173445+0000","last_undegraded":"2024-09-13T14:50:14.173445+0000","last_fullsized":"2024-09-13T14:50:14.173445+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:39.645908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:39.645908+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:29.800728+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"360'15418","reported_seq":15483,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T14:50:11.197377+0000","last_change":"2024-09-13T14:45:59.744581+0000","last_active":"2024-09-13T14:50:11.197377+0000","last_peered":"2024-09-13T14:50:11.197377+0000","last_clean":"2024-09-13T14:50:11.197377+0000","last_became_active":"2024-09-13T14:45:59.744231+0000","last_became_peered":"2024-09-13T14:45:59.744231+0000","last_unstale":"2024-09-13T14:50:11.197377+0000","last_undegraded":"2024-09-13T14:50:11.197377+0000","last_fullsized":"2024-09-13T14:50:11.197377+0000","mapping_epoch":304,"log_start":"341'9156","ondisk_log_start":"341'9156","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":6262,"log_dups_size":0,"ondisk_log_size":6262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:06:14.502840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56098816,"num_objects":856,"num_object_clones":0,"num_object_copies":2568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15418,"num_write_kb":520768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":369,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T14:50:03.184641+0000","last_change":"2024-09-13T14:46:01.762127+0000","last_active":"2024-09-13T14:50:03.184641+0000","last_peered":"2024-09-13T14:50:03.184641+0000","last_clean":"2024-09-13T14:50:03.184641+0000","last_became_active":"2024-09-13T14:46:01.761795+0000","last_became_peered":"2024-09-13T14:46:01.761795+0000","last_unstale":"2024-09-13T14:50:03.184641+0000","last_undegraded":"2024-09-13T14:50:03.184641+0000","last_fullsized":"2024-09-13T14:50:03.184641+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:41:15.626775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":365,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T14:50:03.185388+0000","last_change":"2024-09-13T14:47:47.861718+0000","last_active":"2024-09-13T14:50:03.185388+0000","last_peered":"2024-09-13T14:50:03.185388+0000","last_clean":"2024-09-13T14:50:03.185388+0000","last_became_active":"2024-09-13T14:47:47.861023+0000","last_became_peered":"2024-09-13T14:47:47.861023+0000","last_unstale":"2024-09-13T14:50:03.185388+0000","last_undegraded":"2024-09-13T14:50:03.185388+0000","last_fullsized":"2024-09-13T14:50:03.185388+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:12:13.274053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"90'149","reported_seq":437,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T14:50:03.104992+0000","last_change":"2024-09-13T14:46:52.334715+0000","last_active":"2024-09-13T14:50:03.104992+0000","last_peered":"2024-09-13T14:50:03.104992+0000","last_clean":"2024-09-13T14:50:03.104992+0000","last_became_active":"2024-09-13T14:46:48.158685+0000","last_became_peered":"2024-09-13T14:46:48.158685+0000","last_unstale":"2024-09-13T14:50:03.104992+0000","last_undegraded":"2024-09-13T14:50:03.104992+0000","last_fullsized":"2024-09-13T14:50:03.104992+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:46:52.334301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:46:52.334301+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:46:03.579759+0000","scrub_duration":0.0050940330000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.3","version":"361'15534","reported_seq":19480,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-09-13T14:50:14.173337+0000","last_change":"2024-09-13T14:50:03.175084+0000","last_active":"2024-09-13T14:50:14.173337+0000","last_peered":"2024-09-13T14:50:14.173337+0000","last_clean":"2024-09-13T14:50:14.173337+0000","last_became_active":"2024-09-13T14:50:03.105773+0000","last_became_peered":"2024-09-13T14:50:03.105773+0000","last_unstale":"2024-09-13T14:50:14.173337+0000","last_undegraded":"2024-09-13T14:50:14.173337+0000","last_fullsized":"2024-09-13T14:50:14.173337+0000","mapping_epoch":358,"log_start":"341'9091","ondisk_log_start":"341'9091","created":304,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":6443,"log_dups_size":0,"ondisk_log_size":6443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:22:19.749633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49283072,"num_objects":752,"num_object_clones":0,"num_object_copies":2256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15534,"num_write_kb":521152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1914,"num_bytes_recovered":125435904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":630,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T14:50:07.146671+0000","last_change":"2024-09-13T14:47:47.861249+0000","last_active":"2024-09-13T14:50:07.146671+0000","last_peered":"2024-09-13T14:50:07.146671+0000","last_clean":"2024-09-13T14:50:07.146671+0000","last_became_active":"2024-09-13T14:47:47.860873+0000","last_became_peered":"2024-09-13T14:47:47.860873+0000","last_unstale":"2024-09-13T14:50:07.146671+0000","last_undegraded":"2024-09-13T14:50:07.146671+0000","last_fullsized":"2024-09-13T14:50:07.146671+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:37:50.709437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":63,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-09-13T14:50:14.174168+0000","last_change":"2024-09-13T14:47:47.870185+0000","last_active":"2024-09-13T14:50:14.174168+0000","last_peered":"2024-09-13T14:50:14.174168+0000","last_clean":"2024-09-13T14:50:14.174168+0000","last_became_active":"2024-09-13T14:47:47.869862+0000","last_became_peered":"2024-09-13T14:47:47.869862+0000","last_unstale":"2024-09-13T14:50:14.174168+0000","last_undegraded":"2024-09-13T14:50:14.174168+0000","last_fullsized":"2024-09-13T14:50:14.174168+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:07:02.812306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"360'15737","reported_seq":15803,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T14:50:11.420347+0000","last_change":"2024-09-13T14:46:00.027751+0000","last_active":"2024-09-13T14:50:11.420347+0000","last_peered":"2024-09-13T14:50:11.420347+0000","last_clean":"2024-09-13T14:50:11.420347+0000","last_became_active":"2024-09-13T14:46:00.027429+0000","last_became_peered":"2024-09-13T14:46:00.027429+0000","last_unstale":"2024-09-13T14:50:11.420347+0000","last_undegraded":"2024-09-13T14:50:11.420347+0000","last_fullsized":"2024-09-13T14:50:11.420347+0000","mapping_epoch":304,"log_start":"335'7984","ondisk_log_start":"335'7984","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":7753,"log_dups_size":0,"ondisk_log_size":7753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:17:52.381240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53542912,"num_objects":817,"num_object_clones":0,"num_object_copies":2451,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15737,"num_write_kb":529728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":540,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T14:50:03.102449+0000","last_change":"2024-09-13T14:47:31.684319+0000","last_active":"2024-09-13T14:50:03.102449+0000","last_peered":"2024-09-13T14:50:03.102449+0000","last_clean":"2024-09-13T14:50:03.102449+0000","last_became_active":"2024-09-13T14:47:29.415658+0000","last_became_peered":"2024-09-13T14:47:29.415658+0000","last_unstale":"2024-09-13T14:50:03.102449+0000","last_undegraded":"2024-09-13T14:50:03.102449+0000","last_fullsized":"2024-09-13T14:50:03.102449+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:31.681433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:31.681433+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:29:34.663513+0000","scrub_duration":0.0059601139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":668,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T14:50:03.105328+0000","last_change":"2024-09-13T14:46:04.780718+0000","last_active":"2024-09-13T14:50:03.105328+0000","last_peered":"2024-09-13T14:50:03.105328+0000","last_clean":"2024-09-13T14:50:03.105328+0000","last_became_active":"2024-09-13T14:46:04.780120+0000","last_became_peered":"2024-09-13T14:46:04.780120+0000","last_unstale":"2024-09-13T14:50:03.105328+0000","last_undegraded":"2024-09-13T14:50:03.105328+0000","last_fullsized":"2024-09-13T14:50:03.105328+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:57:14.475888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":558,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T14:50:07.146035+0000","last_change":"2024-09-13T14:37:49.160719+0000","last_active":"2024-09-13T14:50:07.146035+0000","last_peered":"2024-09-13T14:50:07.146035+0000","last_clean":"2024-09-13T14:50:07.146035+0000","last_became_active":"2024-09-13T14:37:49.160401+0000","last_became_peered":"2024-09-13T14:37:49.160401+0000","last_unstale":"2024-09-13T14:50:07.146035+0000","last_undegraded":"2024-09-13T14:50:07.146035+0000","last_fullsized":"2024-09-13T14:50:07.146035+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:21.737945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":207,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T14:50:03.184831+0000","last_change":"2024-09-13T14:37:48.172940+0000","last_active":"2024-09-13T14:50:03.184831+0000","last_peered":"2024-09-13T14:50:03.184831+0000","last_clean":"2024-09-13T14:50:03.184831+0000","last_became_active":"2024-09-13T14:37:48.172060+0000","last_became_peered":"2024-09-13T14:37:48.172060+0000","last_unstale":"2024-09-13T14:50:03.184831+0000","last_undegraded":"2024-09-13T14:50:03.184831+0000","last_fullsized":"2024-09-13T14:50:03.184831+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:07.480645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"90'149","reported_seq":492,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-09-13T14:50:14.173519+0000","last_change":"2024-09-13T14:47:50.884721+0000","last_active":"2024-09-13T14:50:14.173519+0000","last_peered":"2024-09-13T14:50:14.173519+0000","last_clean":"2024-09-13T14:50:14.173519+0000","last_became_active":"2024-09-13T14:47:50.880185+0000","last_became_peered":"2024-09-13T14:47:50.880185+0000","last_unstale":"2024-09-13T14:50:14.173519+0000","last_undegraded":"2024-09-13T14:50:14.173519+0000","last_fullsized":"2024-09-13T14:50:14.173519+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:34.603316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:34.603316+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:38.321686+0000","scrub_duration":0.0057131120000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.a","version":"360'15791","reported_seq":15915,"reported_epoch":360,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:50:11.997346+0000","last_change":"2024-09-13T14:47:49.291273+0000","last_active":"2024-09-13T14:50:11.997346+0000","last_peered":"2024-09-13T14:50:11.997346+0000","last_clean":"2024-09-13T14:47:46.837555+0000","last_became_active":"2024-09-13T14:47:49.272934+0000","last_became_peered":"2024-09-13T14:47:49.272934+0000","last_unstale":"2024-09-13T14:50:11.997346+0000","last_undegraded":"2024-09-13T14:50:11.997346+0000","last_fullsized":"2024-09-13T14:50:11.997346+0000","mapping_epoch":337,"log_start":"326'6057","ondisk_log_start":"326'6057","created":304,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":9734,"log_dups_size":0,"ondisk_log_size":9734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51052544,"num_objects":779,"num_object_clones":0,"num_object_copies":2337,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":779,"num_objects_unfound":0,"num_objects_dirty":779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15791,"num_write_kb":530240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[2,5,3],"avail_no_missing":["2","3","5"],"object_location_counts":[{"shards":"2,3,5","objects":779}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":609,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T14:50:03.102402+0000","last_change":"2024-09-13T14:47:47.892973+0000","last_active":"2024-09-13T14:50:03.102402+0000","last_peered":"2024-09-13T14:50:03.102402+0000","last_clean":"2024-09-13T14:50:03.102402+0000","last_became_active":"2024-09-13T14:47:47.866986+0000","last_became_peered":"2024-09-13T14:47:47.866986+0000","last_unstale":"2024-09-13T14:50:03.102402+0000","last_undegraded":"2024-09-13T14:50:03.102402+0000","last_fullsized":"2024-09-13T14:50:03.102402+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:07.353290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":247,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T14:50:07.117359+0000","last_change":"2024-09-13T14:38:03.525682+0000","last_active":"2024-09-13T14:50:07.117359+0000","last_peered":"2024-09-13T14:50:07.117359+0000","last_clean":"2024-09-13T14:50:07.117359+0000","last_became_active":"2024-09-13T14:38:03.525406+0000","last_became_peered":"2024-09-13T14:38:03.525406+0000","last_unstale":"2024-09-13T14:50:07.117359+0000","last_undegraded":"2024-09-13T14:50:07.117359+0000","last_fullsized":"2024-09-13T14:50:07.117359+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:41:28.970300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":268,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T14:50:07.119412+0000","last_change":"2024-09-13T14:47:47.873233+0000","last_active":"2024-09-13T14:50:07.119412+0000","last_peered":"2024-09-13T14:50:07.119412+0000","last_clean":"2024-09-13T14:50:07.119412+0000","last_became_active":"2024-09-13T14:47:47.872788+0000","last_became_peered":"2024-09-13T14:47:47.872788+0000","last_unstale":"2024-09-13T14:50:07.119412+0000","last_undegraded":"2024-09-13T14:50:07.119412+0000","last_fullsized":"2024-09-13T14:50:07.119412+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:52.409108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"90'149","reported_seq":556,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T14:50:03.102209+0000","last_change":"2024-09-13T14:47:29.629488+0000","last_active":"2024-09-13T14:50:03.102209+0000","last_peered":"2024-09-13T14:50:03.102209+0000","last_clean":"2024-09-13T14:50:03.102209+0000","last_became_active":"2024-09-13T14:47:29.403279+0000","last_became_peered":"2024-09-13T14:47:29.403279+0000","last_unstale":"2024-09-13T14:50:03.102209+0000","last_undegraded":"2024-09-13T14:50:03.102209+0000","last_fullsized":"2024-09-13T14:50:03.102209+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:29.629342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:29.629342+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:14:55.249896+0000","scrub_duration":0.0099290990000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"360'15696","reported_seq":15837,"reported_epoch":360,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:50:12.985711+0000","last_change":"2024-09-13T14:47:49.380981+0000","last_active":"2024-09-13T14:50:12.985711+0000","last_peered":"2024-09-13T14:50:12.985711+0000","last_clean":"2024-09-13T14:47:46.840513+0000","last_became_active":"2024-09-13T14:47:49.259279+0000","last_became_peered":"2024-09-13T14:47:49.259279+0000","last_unstale":"2024-09-13T14:50:12.985711+0000","last_undegraded":"2024-09-13T14:50:12.985711+0000","last_fullsized":"2024-09-13T14:50:12.985711+0000","mapping_epoch":337,"log_start":"341'8977","ondisk_log_start":"341'8977","created":304,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":6719,"log_dups_size":0,"ondisk_log_size":6719,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46399488,"num_objects":708,"num_object_clones":0,"num_object_copies":2124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":708,"num_objects_unfound":0,"num_objects_dirty":708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15696,"num_write_kb":524928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,2],"avail_no_missing":["3","2","4"],"object_location_counts":[{"shards":"2,3,4","objects":708}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":542,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T14:50:07.134769+0000","last_change":"2024-09-13T14:46:04.782955+0000","last_active":"2024-09-13T14:50:07.134769+0000","last_peered":"2024-09-13T14:50:07.134769+0000","last_clean":"2024-09-13T14:50:07.134769+0000","last_became_active":"2024-09-13T14:46:04.782250+0000","last_became_peered":"2024-09-13T14:46:04.782250+0000","last_unstale":"2024-09-13T14:50:07.134769+0000","last_undegraded":"2024-09-13T14:50:07.134769+0000","last_fullsized":"2024-09-13T14:50:07.134769+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:24.596922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":63,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-09-13T14:50:14.173275+0000","last_change":"2024-09-13T14:47:47.866684+0000","last_active":"2024-09-13T14:50:14.173275+0000","last_peered":"2024-09-13T14:50:14.173275+0000","last_clean":"2024-09-13T14:50:14.173275+0000","last_became_active":"2024-09-13T14:47:47.866400+0000","last_became_peered":"2024-09-13T14:47:47.866400+0000","last_unstale":"2024-09-13T14:50:14.173275+0000","last_undegraded":"2024-09-13T14:50:14.173275+0000","last_fullsized":"2024-09-13T14:50:14.173275+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:08:34.289658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":63,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-09-13T14:50:14.174235+0000","last_change":"2024-09-13T14:47:47.870218+0000","last_active":"2024-09-13T14:50:14.174235+0000","last_peered":"2024-09-13T14:50:14.174235+0000","last_clean":"2024-09-13T14:50:14.174235+0000","last_became_active":"2024-09-13T14:47:47.869907+0000","last_became_peered":"2024-09-13T14:47:47.869907+0000","last_unstale":"2024-09-13T14:50:14.174235+0000","last_undegraded":"2024-09-13T14:50:14.174235+0000","last_fullsized":"2024-09-13T14:50:14.174235+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:41:49.141441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"90'149","reported_seq":458,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T14:50:03.105133+0000","last_change":"2024-09-13T14:47:32.275753+0000","last_active":"2024-09-13T14:50:03.105133+0000","last_peered":"2024-09-13T14:50:03.105133+0000","last_clean":"2024-09-13T14:50:03.105133+0000","last_became_active":"2024-09-13T14:47:29.401066+0000","last_became_peered":"2024-09-13T14:47:29.401066+0000","last_unstale":"2024-09-13T14:50:03.105133+0000","last_undegraded":"2024-09-13T14:50:03.105133+0000","last_fullsized":"2024-09-13T14:50:03.105133+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:32.275562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:32.275562+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:43:41.056775+0000","scrub_duration":0.0054674260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.8","version":"360'15537","reported_seq":15603,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T14:50:11.407275+0000","last_change":"2024-09-13T14:45:59.743468+0000","last_active":"2024-09-13T14:50:11.407275+0000","last_peered":"2024-09-13T14:50:11.407275+0000","last_clean":"2024-09-13T14:50:11.407275+0000","last_became_active":"2024-09-13T14:45:59.742168+0000","last_became_peered":"2024-09-13T14:45:59.742168+0000","last_unstale":"2024-09-13T14:50:11.407275+0000","last_undegraded":"2024-09-13T14:50:11.407275+0000","last_fullsized":"2024-09-13T14:50:11.407275+0000","mapping_epoch":304,"log_start":"334'7762","ondisk_log_start":"334'7762","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":7775,"log_dups_size":0,"ondisk_log_size":7775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:38:09.406586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50528256,"num_objects":771,"num_object_clones":0,"num_object_copies":2313,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15537,"num_write_kb":521856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":1113,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T14:50:03.105198+0000","last_change":"2024-09-13T14:47:50.892842+0000","last_active":"2024-09-13T14:50:03.105198+0000","last_peered":"2024-09-13T14:50:03.105198+0000","last_clean":"2024-09-13T14:50:03.105198+0000","last_became_active":"2024-09-13T14:47:50.892237+0000","last_became_peered":"2024-09-13T14:47:50.892237+0000","last_unstale":"2024-09-13T14:50:03.105198+0000","last_undegraded":"2024-09-13T14:50:03.105198+0000","last_fullsized":"2024-09-13T14:50:03.105198+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:20:05.796762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":8388646,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":369,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T14:50:03.185275+0000","last_change":"2024-09-13T14:38:03.528133+0000","last_active":"2024-09-13T14:50:03.185275+0000","last_peered":"2024-09-13T14:50:03.185275+0000","last_clean":"2024-09-13T14:50:03.185275+0000","last_became_active":"2024-09-13T14:38:03.527829+0000","last_became_peered":"2024-09-13T14:38:03.527829+0000","last_unstale":"2024-09-13T14:50:03.185275+0000","last_undegraded":"2024-09-13T14:50:03.185275+0000","last_fullsized":"2024-09-13T14:50:03.185275+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:48.254169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":63,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-09-13T14:50:14.173489+0000","last_change":"2024-09-13T14:47:47.877272+0000","last_active":"2024-09-13T14:50:14.173489+0000","last_peered":"2024-09-13T14:50:14.173489+0000","last_clean":"2024-09-13T14:50:14.173489+0000","last_became_active":"2024-09-13T14:47:47.877003+0000","last_became_peered":"2024-09-13T14:47:47.877003+0000","last_unstale":"2024-09-13T14:50:14.173489+0000","last_undegraded":"2024-09-13T14:50:14.173489+0000","last_fullsized":"2024-09-13T14:50:14.173489+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:47:23.172452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"90'149","reported_seq":491,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T14:50:07.146877+0000","last_change":"2024-09-13T14:47:33.067298+0000","last_active":"2024-09-13T14:50:07.146877+0000","last_peered":"2024-09-13T14:50:07.146877+0000","last_clean":"2024-09-13T14:50:07.146877+0000","last_became_active":"2024-09-13T14:47:29.419416+0000","last_became_peered":"2024-09-13T14:47:29.419416+0000","last_unstale":"2024-09-13T14:50:07.146877+0000","last_undegraded":"2024-09-13T14:50:07.146877+0000","last_fullsized":"2024-09-13T14:50:07.146877+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:33.066954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:33.066954+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:05:42.808060+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"360'15449","reported_seq":15579,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T14:50:12.902123+0000","last_change":"2024-09-13T14:46:01.768936+0000","last_active":"2024-09-13T14:50:12.902123+0000","last_peered":"2024-09-13T14:50:12.902123+0000","last_clean":"2024-09-13T14:50:12.902123+0000","last_became_active":"2024-09-13T14:46:01.768354+0000","last_became_peered":"2024-09-13T14:46:01.768354+0000","last_unstale":"2024-09-13T14:50:12.902123+0000","last_undegraded":"2024-09-13T14:50:12.902123+0000","last_fullsized":"2024-09-13T14:50:12.902123+0000","mapping_epoch":306,"log_start":"330'7462","ondisk_log_start":"330'7462","created":304,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":7987,"log_dups_size":0,"ondisk_log_size":7987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:01.762702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46596096,"num_objects":711,"num_object_clones":0,"num_object_copies":2133,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15449,"num_write_kb":517120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":182,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T14:50:03.105551+0000","last_change":"2024-09-13T14:46:01.763906+0000","last_active":"2024-09-13T14:50:03.105551+0000","last_peered":"2024-09-13T14:50:03.105551+0000","last_clean":"2024-09-13T14:50:03.105551+0000","last_became_active":"2024-09-13T14:46:01.763542+0000","last_became_peered":"2024-09-13T14:46:01.763542+0000","last_unstale":"2024-09-13T14:50:03.105551+0000","last_undegraded":"2024-09-13T14:50:03.105551+0000","last_fullsized":"2024-09-13T14:50:03.105551+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:09:36.652767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":1351,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-09-13T14:50:14.173520+0000","last_change":"2024-09-13T14:47:50.883365+0000","last_active":"2024-09-13T14:50:14.173520+0000","last_peered":"2024-09-13T14:50:14.173520+0000","last_clean":"2024-09-13T14:50:14.173520+0000","last_became_active":"2024-09-13T14:47:50.882609+0000","last_became_peered":"2024-09-13T14:47:50.882609+0000","last_unstale":"2024-09-13T14:50:14.173520+0000","last_undegraded":"2024-09-13T14:50:14.173520+0000","last_fullsized":"2024-09-13T14:50:14.173520+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:12:17.961575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":63,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-09-13T14:50:14.173998+0000","last_change":"2024-09-13T14:47:47.866979+0000","last_active":"2024-09-13T14:50:14.173998+0000","last_peered":"2024-09-13T14:50:14.173998+0000","last_clean":"2024-09-13T14:50:14.173998+0000","last_became_active":"2024-09-13T14:47:47.866634+0000","last_became_peered":"2024-09-13T14:47:47.866634+0000","last_unstale":"2024-09-13T14:50:14.173998+0000","last_undegraded":"2024-09-13T14:50:14.173998+0000","last_fullsized":"2024-09-13T14:50:14.173998+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:47:09.696139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"90'149","reported_seq":497,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T14:50:07.117688+0000","last_change":"2024-09-13T14:47:29.556225+0000","last_active":"2024-09-13T14:50:07.117688+0000","last_peered":"2024-09-13T14:50:07.117688+0000","last_clean":"2024-09-13T14:50:07.117688+0000","last_became_active":"2024-09-13T14:47:29.408774+0000","last_became_peered":"2024-09-13T14:47:29.408774+0000","last_unstale":"2024-09-13T14:50:07.117688+0000","last_undegraded":"2024-09-13T14:50:07.117688+0000","last_fullsized":"2024-09-13T14:50:07.117688+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:29.556149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:29.556149+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:35.691178+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"360'15637","reported_seq":17949,"reported_epoch":360,"state":"active+clean+remapped","last_fresh":"2024-09-13T14:50:13.034906+0000","last_change":"2024-09-13T14:50:12.207751+0000","last_active":"2024-09-13T14:50:13.034906+0000","last_peered":"2024-09-13T14:50:13.034906+0000","last_clean":"2024-09-13T14:50:13.034906+0000","last_became_active":"2024-09-13T14:47:49.256338+0000","last_became_peered":"2024-09-13T14:47:49.256338+0000","last_unstale":"2024-09-13T14:50:13.034906+0000","last_undegraded":"2024-09-13T14:50:13.034906+0000","last_fullsized":"2024-09-13T14:50:13.034906+0000","mapping_epoch":337,"log_start":"340'8941","ondisk_log_start":"340'8941","created":304,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":6696,"log_dups_size":0,"ondisk_log_size":6696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:49:24.776007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47513600,"num_objects":725,"num_object_clones":0,"num_object_copies":2175,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15637,"num_write_kb":523584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1084,"num_bytes_recovered":70975488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,4],"avail_no_missing":["3","1","2","4"],"object_location_counts":[{"shards":"2,3,4","objects":725}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":63,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-09-13T14:50:14.173732+0000","last_change":"2024-09-13T14:47:47.876562+0000","last_active":"2024-09-13T14:50:14.173732+0000","last_peered":"2024-09-13T14:50:14.173732+0000","last_clean":"2024-09-13T14:50:14.173732+0000","last_became_active":"2024-09-13T14:47:47.876297+0000","last_became_peered":"2024-09-13T14:47:47.876297+0000","last_unstale":"2024-09-13T14:50:14.173732+0000","last_undegraded":"2024-09-13T14:50:14.173732+0000","last_fullsized":"2024-09-13T14:50:14.173732+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:03:27.761394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":885,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T14:50:03.102874+0000","last_change":"2024-09-13T14:38:04.596192+0000","last_active":"2024-09-13T14:50:03.102874+0000","last_peered":"2024-09-13T14:50:03.102874+0000","last_clean":"2024-09-13T14:50:03.102874+0000","last_became_active":"2024-09-13T14:38:04.593299+0000","last_became_peered":"2024-09-13T14:38:04.593299+0000","last_unstale":"2024-09-13T14:50:03.102874+0000","last_undegraded":"2024-09-13T14:50:03.102874+0000","last_fullsized":"2024-09-13T14:50:03.102874+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:34.241410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":247,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T14:50:07.117888+0000","last_change":"2024-09-13T14:38:03.533120+0000","last_active":"2024-09-13T14:50:07.117888+0000","last_peered":"2024-09-13T14:50:07.117888+0000","last_clean":"2024-09-13T14:50:07.117888+0000","last_became_active":"2024-09-13T14:38:03.532829+0000","last_became_peered":"2024-09-13T14:38:03.532829+0000","last_unstale":"2024-09-13T14:50:07.117888+0000","last_undegraded":"2024-09-13T14:50:07.117888+0000","last_fullsized":"2024-09-13T14:50:07.117888+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:24.514661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"90'149","reported_seq":458,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T14:50:03.185171+0000","last_change":"2024-09-13T14:47:30.528588+0000","last_active":"2024-09-13T14:50:03.185171+0000","last_peered":"2024-09-13T14:50:03.185171+0000","last_clean":"2024-09-13T14:50:03.185171+0000","last_became_active":"2024-09-13T14:47:29.413621+0000","last_became_peered":"2024-09-13T14:47:29.413621+0000","last_unstale":"2024-09-13T14:50:03.185171+0000","last_undegraded":"2024-09-13T14:50:03.185171+0000","last_fullsized":"2024-09-13T14:50:03.185171+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:30.528112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:30.528112+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:30.462009+0000","scrub_duration":0.012751485,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.c","version":"360'15559","reported_seq":15683,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T14:50:12.924905+0000","last_change":"2024-09-13T14:45:59.743204+0000","last_active":"2024-09-13T14:50:12.924905+0000","last_peered":"2024-09-13T14:50:12.924905+0000","last_clean":"2024-09-13T14:50:12.924905+0000","last_became_active":"2024-09-13T14:45:59.742910+0000","last_became_peered":"2024-09-13T14:45:59.742910+0000","last_unstale":"2024-09-13T14:50:12.924905+0000","last_undegraded":"2024-09-13T14:50:12.924905+0000","last_fullsized":"2024-09-13T14:50:12.924905+0000","mapping_epoch":304,"log_start":"331'7584","ondisk_log_start":"331'7584","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":7975,"log_dups_size":0,"ondisk_log_size":7975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:51.017907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47251456,"num_objects":721,"num_object_clones":0,"num_object_copies":2163,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15559,"num_write_kb":520960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":649,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T14:50:07.147426+0000","last_change":"2024-09-13T14:38:53.938227+0000","last_active":"2024-09-13T14:50:07.147426+0000","last_peered":"2024-09-13T14:50:07.147426+0000","last_clean":"2024-09-13T14:50:07.147426+0000","last_became_active":"2024-09-13T14:38:53.902867+0000","last_became_peered":"2024-09-13T14:38:53.902867+0000","last_unstale":"2024-09-13T14:50:07.147426+0000","last_undegraded":"2024-09-13T14:50:07.147426+0000","last_fullsized":"2024-09-13T14:50:07.147426+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:16.831756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":533,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-09-13T14:50:14.174094+0000","last_change":"2024-09-13T14:47:50.884706+0000","last_active":"2024-09-13T14:50:14.174094+0000","last_peered":"2024-09-13T14:50:14.174094+0000","last_clean":"2024-09-13T14:50:14.174094+0000","last_became_active":"2024-09-13T14:47:50.879750+0000","last_became_peered":"2024-09-13T14:47:50.879750+0000","last_unstale":"2024-09-13T14:50:14.174094+0000","last_undegraded":"2024-09-13T14:50:14.174094+0000","last_fullsized":"2024-09-13T14:50:14.174094+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:34:32.941266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":631,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T14:50:07.146514+0000","last_change":"2024-09-13T14:38:53.900107+0000","last_active":"2024-09-13T14:50:07.146514+0000","last_peered":"2024-09-13T14:50:07.146514+0000","last_clean":"2024-09-13T14:50:07.146514+0000","last_became_active":"2024-09-13T14:38:53.899556+0000","last_became_peered":"2024-09-13T14:38:53.899556+0000","last_unstale":"2024-09-13T14:50:07.146514+0000","last_undegraded":"2024-09-13T14:50:07.146514+0000","last_fullsized":"2024-09-13T14:50:07.146514+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:15:47.918163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"90'149","reported_seq":488,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T14:50:07.145531+0000","last_change":"2024-09-13T14:47:29.999643+0000","last_active":"2024-09-13T14:50:07.145531+0000","last_peered":"2024-09-13T14:50:07.145531+0000","last_clean":"2024-09-13T14:50:07.145531+0000","last_became_active":"2024-09-13T14:47:29.410951+0000","last_became_peered":"2024-09-13T14:47:29.410951+0000","last_unstale":"2024-09-13T14:50:07.145531+0000","last_undegraded":"2024-09-13T14:50:07.145531+0000","last_fullsized":"2024-09-13T14:50:07.145531+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:29.999519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:29.999519+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:04:54.887884+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"360'15685","reported_seq":15761,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T14:50:11.355624+0000","last_change":"2024-09-13T14:46:01.764360+0000","last_active":"2024-09-13T14:50:11.355624+0000","last_peered":"2024-09-13T14:50:11.355624+0000","last_clean":"2024-09-13T14:50:11.355624+0000","last_became_active":"2024-09-13T14:46:01.764029+0000","last_became_peered":"2024-09-13T14:46:01.764029+0000","last_unstale":"2024-09-13T14:50:11.355624+0000","last_undegraded":"2024-09-13T14:50:11.355624+0000","last_fullsized":"2024-09-13T14:50:11.355624+0000","mapping_epoch":306,"log_start":"335'7945","ondisk_log_start":"335'7945","created":304,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":7740,"log_dups_size":0,"ondisk_log_size":7740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:02:23.092494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51511320,"num_objects":787,"num_object_clones":0,"num_object_copies":2361,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":787,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15685,"num_write_kb":527041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":273,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T14:50:07.119245+0000","last_change":"2024-09-13T14:47:47.883475+0000","last_active":"2024-09-13T14:50:07.119245+0000","last_peered":"2024-09-13T14:50:07.119245+0000","last_clean":"2024-09-13T14:50:07.119245+0000","last_became_active":"2024-09-13T14:47:47.871525+0000","last_became_peered":"2024-09-13T14:47:47.871525+0000","last_unstale":"2024-09-13T14:50:07.119245+0000","last_undegraded":"2024-09-13T14:50:07.119245+0000","last_fullsized":"2024-09-13T14:50:07.119245+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:33.903432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":163,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T14:50:03.102830+0000","last_change":"2024-09-13T14:46:01.781508+0000","last_active":"2024-09-13T14:50:03.102830+0000","last_peered":"2024-09-13T14:50:03.102830+0000","last_clean":"2024-09-13T14:50:03.102830+0000","last_became_active":"2024-09-13T14:46:01.767096+0000","last_became_peered":"2024-09-13T14:46:01.767096+0000","last_unstale":"2024-09-13T14:50:03.102830+0000","last_undegraded":"2024-09-13T14:50:03.102830+0000","last_fullsized":"2024-09-13T14:50:03.102830+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:20:01.119189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":353,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T14:50:03.184553+0000","last_change":"2024-09-13T14:37:49.172218+0000","last_active":"2024-09-13T14:50:03.184553+0000","last_peered":"2024-09-13T14:50:03.184553+0000","last_clean":"2024-09-13T14:50:03.184553+0000","last_became_active":"2024-09-13T14:37:49.171718+0000","last_became_peered":"2024-09-13T14:37:49.171718+0000","last_unstale":"2024-09-13T14:50:03.184553+0000","last_undegraded":"2024-09-13T14:50:03.184553+0000","last_fullsized":"2024-09-13T14:50:03.184553+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:46:54.626521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":384,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T14:50:03.184997+0000","last_change":"2024-09-13T14:47:47.868409+0000","last_active":"2024-09-13T14:50:03.184997+0000","last_peered":"2024-09-13T14:50:03.184997+0000","last_clean":"2024-09-13T14:50:03.184997+0000","last_became_active":"2024-09-13T14:47:47.867712+0000","last_became_peered":"2024-09-13T14:47:47.867712+0000","last_unstale":"2024-09-13T14:50:03.184997+0000","last_undegraded":"2024-09-13T14:50:03.184997+0000","last_fullsized":"2024-09-13T14:50:03.184997+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:57:14.647493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":355,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T14:50:03.187706+0000","last_change":"2024-09-13T14:37:48.128447+0000","last_active":"2024-09-13T14:50:03.187706+0000","last_peered":"2024-09-13T14:50:03.187706+0000","last_clean":"2024-09-13T14:50:03.187706+0000","last_became_active":"2024-09-13T14:37:48.128143+0000","last_became_peered":"2024-09-13T14:37:48.128143+0000","last_unstale":"2024-09-13T14:50:03.187706+0000","last_undegraded":"2024-09-13T14:50:03.187706+0000","last_fullsized":"2024-09-13T14:50:03.187706+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:22.204039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":63,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-09-13T14:50:14.173303+0000","last_change":"2024-09-13T14:47:47.867935+0000","last_active":"2024-09-13T14:50:14.173303+0000","last_peered":"2024-09-13T14:50:14.173303+0000","last_clean":"2024-09-13T14:50:14.173303+0000","last_became_active":"2024-09-13T14:47:47.867616+0000","last_became_peered":"2024-09-13T14:47:47.867616+0000","last_unstale":"2024-09-13T14:50:14.173303+0000","last_undegraded":"2024-09-13T14:50:14.173303+0000","last_fullsized":"2024-09-13T14:50:14.173303+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:52:38.619916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":127,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T14:50:03.102715+0000","last_change":"2024-09-13T14:38:53.954281+0000","last_active":"2024-09-13T14:50:03.102715+0000","last_peered":"2024-09-13T14:50:03.102715+0000","last_clean":"2024-09-13T14:50:03.102715+0000","last_became_active":"2024-09-13T14:38:53.900912+0000","last_became_peered":"2024-09-13T14:38:53.900912+0000","last_unstale":"2024-09-13T14:50:03.102715+0000","last_undegraded":"2024-09-13T14:50:03.102715+0000","last_fullsized":"2024-09-13T14:50:03.102715+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:47.876275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"90'149","reported_seq":492,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T14:50:07.127620+0000","last_change":"2024-09-13T14:47:38.468095+0000","last_active":"2024-09-13T14:50:07.127620+0000","last_peered":"2024-09-13T14:50:07.127620+0000","last_clean":"2024-09-13T14:50:07.127620+0000","last_became_active":"2024-09-13T14:47:38.467526+0000","last_became_peered":"2024-09-13T14:47:38.467526+0000","last_unstale":"2024-09-13T14:50:07.127620+0000","last_undegraded":"2024-09-13T14:50:07.127620+0000","last_fullsized":"2024-09-13T14:50:07.127620+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:31.013128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:31.013128+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:17:20.508142+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":188,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T14:50:03.105516+0000","last_change":"2024-09-13T14:47:47.861842+0000","last_active":"2024-09-13T14:50:03.105516+0000","last_peered":"2024-09-13T14:50:03.105516+0000","last_clean":"2024-09-13T14:50:03.105516+0000","last_became_active":"2024-09-13T14:47:47.861492+0000","last_became_peered":"2024-09-13T14:47:47.861492+0000","last_unstale":"2024-09-13T14:50:03.105516+0000","last_undegraded":"2024-09-13T14:50:03.105516+0000","last_fullsized":"2024-09-13T14:50:03.105516+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:17.279533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":572,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T14:50:07.144559+0000","last_change":"2024-09-13T14:38:53.899535+0000","last_active":"2024-09-13T14:50:07.144559+0000","last_peered":"2024-09-13T14:50:07.144559+0000","last_clean":"2024-09-13T14:50:07.144559+0000","last_became_active":"2024-09-13T14:38:53.899183+0000","last_became_peered":"2024-09-13T14:38:53.899183+0000","last_unstale":"2024-09-13T14:50:07.144559+0000","last_undegraded":"2024-09-13T14:50:07.144559+0000","last_fullsized":"2024-09-13T14:50:07.144559+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:33.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":246,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T14:50:07.118272+0000","last_change":"2024-09-13T14:38:03.543426+0000","last_active":"2024-09-13T14:50:07.118272+0000","last_peered":"2024-09-13T14:50:07.118272+0000","last_clean":"2024-09-13T14:50:07.118272+0000","last_became_active":"2024-09-13T14:38:03.542661+0000","last_became_peered":"2024-09-13T14:38:03.542661+0000","last_unstale":"2024-09-13T14:50:07.118272+0000","last_undegraded":"2024-09-13T14:50:07.118272+0000","last_fullsized":"2024-09-13T14:50:07.118272+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:54.050127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":608,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T14:50:07.146437+0000","last_change":"2024-09-13T14:47:47.868624+0000","last_active":"2024-09-13T14:50:07.146437+0000","last_peered":"2024-09-13T14:50:07.146437+0000","last_clean":"2024-09-13T14:50:07.146437+0000","last_became_active":"2024-09-13T14:47:47.868031+0000","last_became_peered":"2024-09-13T14:47:47.868031+0000","last_unstale":"2024-09-13T14:50:07.146437+0000","last_undegraded":"2024-09-13T14:50:07.146437+0000","last_fullsized":"2024-09-13T14:50:07.146437+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:05:25.365023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":201,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T14:50:03.105728+0000","last_change":"2024-09-13T14:47:47.859825+0000","last_active":"2024-09-13T14:50:03.105728+0000","last_peered":"2024-09-13T14:50:03.105728+0000","last_clean":"2024-09-13T14:50:03.105728+0000","last_became_active":"2024-09-13T14:47:47.859069+0000","last_became_peered":"2024-09-13T14:47:47.859069+0000","last_unstale":"2024-09-13T14:50:03.105728+0000","last_undegraded":"2024-09-13T14:50:03.105728+0000","last_fullsized":"2024-09-13T14:50:03.105728+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:18:36.452321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":640,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T14:50:07.146590+0000","last_change":"2024-09-13T14:47:47.868945+0000","last_active":"2024-09-13T14:50:07.146590+0000","last_peered":"2024-09-13T14:50:07.146590+0000","last_clean":"2024-09-13T14:50:07.146590+0000","last_became_active":"2024-09-13T14:47:47.868296+0000","last_became_peered":"2024-09-13T14:47:47.868296+0000","last_unstale":"2024-09-13T14:50:07.146590+0000","last_undegraded":"2024-09-13T14:50:07.146590+0000","last_fullsized":"2024-09-13T14:50:07.146590+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:22:24.778693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":254,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T14:50:07.119169+0000","last_change":"2024-09-13T14:38:53.892948+0000","last_active":"2024-09-13T14:50:07.119169+0000","last_peered":"2024-09-13T14:50:07.119169+0000","last_clean":"2024-09-13T14:50:07.119169+0000","last_became_active":"2024-09-13T14:38:53.892665+0000","last_became_peered":"2024-09-13T14:38:53.892665+0000","last_unstale":"2024-09-13T14:50:07.119169+0000","last_undegraded":"2024-09-13T14:50:07.119169+0000","last_fullsized":"2024-09-13T14:50:07.119169+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:08.292359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":63,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-09-13T14:50:14.173149+0000","last_change":"2024-09-13T14:47:47.871164+0000","last_active":"2024-09-13T14:50:14.173149+0000","last_peered":"2024-09-13T14:50:14.173149+0000","last_clean":"2024-09-13T14:50:14.173149+0000","last_became_active":"2024-09-13T14:47:47.870862+0000","last_became_peered":"2024-09-13T14:47:47.870862+0000","last_unstale":"2024-09-13T14:50:14.173149+0000","last_undegraded":"2024-09-13T14:50:14.173149+0000","last_fullsized":"2024-09-13T14:50:14.173149+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:41:34.392081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":622,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T14:50:07.145000+0000","last_change":"2024-09-13T14:47:47.860840+0000","last_active":"2024-09-13T14:50:07.145000+0000","last_peered":"2024-09-13T14:50:07.145000+0000","last_clean":"2024-09-13T14:50:07.145000+0000","last_became_active":"2024-09-13T14:47:47.860473+0000","last_became_peered":"2024-09-13T14:47:47.860473+0000","last_unstale":"2024-09-13T14:50:07.145000+0000","last_undegraded":"2024-09-13T14:50:07.145000+0000","last_fullsized":"2024-09-13T14:50:07.145000+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:31:39.442918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":437,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T14:50:03.185825+0000","last_change":"2024-09-13T14:47:47.868246+0000","last_active":"2024-09-13T14:50:03.185825+0000","last_peered":"2024-09-13T14:50:03.185825+0000","last_clean":"2024-09-13T14:50:03.185825+0000","last_became_active":"2024-09-13T14:47:47.867817+0000","last_became_peered":"2024-09-13T14:47:47.867817+0000","last_unstale":"2024-09-13T14:50:03.185825+0000","last_undegraded":"2024-09-13T14:50:03.185825+0000","last_fullsized":"2024-09-13T14:50:03.185825+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:07.612451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":133,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T14:50:07.127978+0000","last_change":"2024-09-13T14:47:47.876275+0000","last_active":"2024-09-13T14:50:07.127978+0000","last_peered":"2024-09-13T14:50:07.127978+0000","last_clean":"2024-09-13T14:50:07.127978+0000","last_became_active":"2024-09-13T14:47:47.875467+0000","last_became_peered":"2024-09-13T14:47:47.875467+0000","last_unstale":"2024-09-13T14:50:07.127978+0000","last_undegraded":"2024-09-13T14:50:07.127978+0000","last_fullsized":"2024-09-13T14:50:07.127978+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:12:36.717423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":246,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T14:50:07.117689+0000","last_change":"2024-09-13T14:38:03.527021+0000","last_active":"2024-09-13T14:50:07.117689+0000","last_peered":"2024-09-13T14:50:07.117689+0000","last_clean":"2024-09-13T14:50:07.117689+0000","last_became_active":"2024-09-13T14:38:03.526698+0000","last_became_peered":"2024-09-13T14:38:03.526698+0000","last_unstale":"2024-09-13T14:50:07.117689+0000","last_undegraded":"2024-09-13T14:50:07.117689+0000","last_fullsized":"2024-09-13T14:50:07.117689+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:01:35.734086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":381,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T14:50:03.184919+0000","last_change":"2024-09-13T14:47:47.866784+0000","last_active":"2024-09-13T14:50:03.184919+0000","last_peered":"2024-09-13T14:50:03.184919+0000","last_clean":"2024-09-13T14:50:03.184919+0000","last_became_active":"2024-09-13T14:47:47.866483+0000","last_became_peered":"2024-09-13T14:47:47.866483+0000","last_unstale":"2024-09-13T14:50:03.184919+0000","last_undegraded":"2024-09-13T14:50:03.184919+0000","last_fullsized":"2024-09-13T14:50:03.184919+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:15:53.119741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":189,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T14:50:03.105556+0000","last_change":"2024-09-13T14:47:47.880946+0000","last_active":"2024-09-13T14:50:03.105556+0000","last_peered":"2024-09-13T14:50:03.105556+0000","last_clean":"2024-09-13T14:50:03.105556+0000","last_became_active":"2024-09-13T14:47:47.866534+0000","last_became_peered":"2024-09-13T14:47:47.866534+0000","last_unstale":"2024-09-13T14:50:03.105556+0000","last_undegraded":"2024-09-13T14:50:03.105556+0000","last_fullsized":"2024-09-13T14:50:03.105556+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:01:01.384873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":374,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T14:50:03.185691+0000","last_change":"2024-09-13T14:38:03.526333+0000","last_active":"2024-09-13T14:50:03.185691+0000","last_peered":"2024-09-13T14:50:03.185691+0000","last_clean":"2024-09-13T14:50:03.185691+0000","last_became_active":"2024-09-13T14:38:03.526001+0000","last_became_peered":"2024-09-13T14:38:03.526001+0000","last_unstale":"2024-09-13T14:50:03.185691+0000","last_undegraded":"2024-09-13T14:50:03.185691+0000","last_fullsized":"2024-09-13T14:50:03.185691+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:25:25.181271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":176,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T14:50:03.105632+0000","last_change":"2024-09-13T14:37:49.149582+0000","last_active":"2024-09-13T14:50:03.105632+0000","last_peered":"2024-09-13T14:50:03.105632+0000","last_clean":"2024-09-13T14:50:03.105632+0000","last_became_active":"2024-09-13T14:37:49.148525+0000","last_became_peered":"2024-09-13T14:37:49.148525+0000","last_unstale":"2024-09-13T14:50:03.105632+0000","last_undegraded":"2024-09-13T14:50:03.105632+0000","last_fullsized":"2024-09-13T14:50:03.105632+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:48.655665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":654,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T14:50:07.144895+0000","last_change":"2024-09-13T14:46:01.766580+0000","last_active":"2024-09-13T14:50:07.144895+0000","last_peered":"2024-09-13T14:50:07.144895+0000","last_clean":"2024-09-13T14:50:07.144895+0000","last_became_active":"2024-09-13T14:46:01.766257+0000","last_became_peered":"2024-09-13T14:46:01.766257+0000","last_unstale":"2024-09-13T14:50:07.144895+0000","last_undegraded":"2024-09-13T14:50:07.144895+0000","last_fullsized":"2024-09-13T14:50:07.144895+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:01.126109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":618,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T14:50:07.145420+0000","last_change":"2024-09-13T14:46:01.769647+0000","last_active":"2024-09-13T14:50:07.145420+0000","last_peered":"2024-09-13T14:50:07.145420+0000","last_clean":"2024-09-13T14:50:07.145420+0000","last_became_active":"2024-09-13T14:46:01.769228+0000","last_became_peered":"2024-09-13T14:46:01.769228+0000","last_unstale":"2024-09-13T14:50:07.145420+0000","last_undegraded":"2024-09-13T14:50:07.145420+0000","last_fullsized":"2024-09-13T14:50:07.145420+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:58.696024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":363,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-13T14:50:03.184470+0000","last_change":"2024-09-13T14:47:47.860727+0000","last_active":"2024-09-13T14:50:03.184470+0000","last_peered":"2024-09-13T14:50:03.184470+0000","last_clean":"2024-09-13T14:50:03.184470+0000","last_became_active":"2024-09-13T14:47:47.860377+0000","last_became_peered":"2024-09-13T14:47:47.860377+0000","last_unstale":"2024-09-13T14:50:03.184470+0000","last_undegraded":"2024-09-13T14:50:03.184470+0000","last_fullsized":"2024-09-13T14:50:03.184470+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:27:00.868699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":266,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-13T14:50:07.118183+0000","last_change":"2024-09-13T14:46:01.775237+0000","last_active":"2024-09-13T14:50:07.118183+0000","last_peered":"2024-09-13T14:50:07.118183+0000","last_clean":"2024-09-13T14:50:07.118183+0000","last_became_active":"2024-09-13T14:46:01.765025+0000","last_became_peered":"2024-09-13T14:46:01.765025+0000","last_unstale":"2024-09-13T14:50:07.118183+0000","last_undegraded":"2024-09-13T14:50:07.118183+0000","last_fullsized":"2024-09-13T14:50:07.118183+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:29:16.089042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1746,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":786300952,"num_objects":11999,"num_object_clones":0,"num_object_copies":35997,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2175,"num_objects_unfound":0,"num_objects_dirty":11999,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":249171,"num_write_kb":8357377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12036,"num_bytes_recovered":788660224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2599038976,"data_stored":2599026760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118208,"ondisk_log_size":118208,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":2154,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25132,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":151,"num_bytes_recovered":8388646,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3129,"ondisk_log_size":3129,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234476,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":898568,"kb_used_data":263680,"kb_used_omap":1,"kb_used_meta":634878,"kb_avail":92846584,"statfs":{"total":95995035648,"available":95074902016,"internally_reserved":0,"allocated":270008320,"data_stored":268118640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":650115531},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":239,"seq":1026497183894,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":768444,"kb_used_data":286900,"kb_used_omap":8,"kb_used_meta":481527,"kb_avail":92976708,"statfs":{"total":95995035648,"available":95208148992,"internally_reserved":0,"allocated":293785600,"data_stored":291908115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8453,"internal_metadata":493084411},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757741,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1377148,"kb_used_data":479668,"kb_used_omap":1,"kb_used_meta":897406,"kb_avail":92368004,"statfs":{"total":95995035648,"available":94584836096,"internally_reserved":0,"allocated":491180032,"data_stored":489298883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":918944198},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019385,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1160448,"kb_used_data":417272,"kb_used_omap":1,"kb_used_meta":743166,"kb_avail":92584704,"statfs":{"total":95995035648,"available":94806736896,"internally_reserved":0,"allocated":427286528,"data_stored":425405004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":761002438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435568,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1089012,"kb_used_data":326892,"kb_used_omap":5,"kb_used_meta":761914,"kb_avail":92656140,"statfs":{"total":95995035648,"available":94879887360,"internally_reserved":0,"allocated":334737408,"data_stored":332864189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":780200062},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":104,"seq":446676599096,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":652540,"kb_used_data":359732,"kb_used_omap":7,"kb_used_meta":292664,"kb_avail":93092612,"statfs":{"total":95995035648,"available":95326834688,"internally_reserved":0,"allocated":368365568,"data_stored":366484707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8010,"internal_metadata":299688118},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":99,"seq":425201762623,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":370508,"kb_used_data":230524,"kb_used_omap":7,"kb_used_meta":139960,"kb_avail":93374644,"statfs":{"total":95995035648,"available":95615635456,"internally_reserved":0,"allocated":236056576,"data_stored":234182782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7759,"internal_metadata":143319473},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926146,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1141492,"kb_used_data":297580,"kb_used_omap":7,"kb_used_meta":843832,"kb_avail":92603660,"statfs":{"total":95995035648,"available":94826147840,"internally_reserved":0,"allocated":304721920,"data_stored":302848671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":864084264},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6860,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":287244288,"data_stored":287244288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":229834752,"data_stored":229834752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":349507584,"data_stored":349503512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":324403200,"data_stored":324403200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":403898368,"data_stored":403898368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":474288128,"data_stored":474284056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":266862592,"data_stored":266862592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":263000064,"data_stored":262995992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:50:16.965 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:50:16.995 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:16.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:16.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:17.098 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:17.146 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:17.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:17.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:17.248 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:17.307 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:17.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:17.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:17.409 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:17.434 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:17.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:17.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:17.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:17 smithi195 ceph-mon[104262]: pgmap v1261: 109 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 105 active+clean; 754 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 691 op/s; 2175/36081 objects misplaced (6.028%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:50:17.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:17 smithi195 ceph-mon[104262]: osdmap e364: 8 total, 8 up, 8 in 2024-09-13T14:50:17.537 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:17.558 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:17.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:17.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:17.563 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:17 smithi102 ceph-mon[110183]: pgmap v1261: 109 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 105 active+clean; 754 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 691 op/s; 2175/36081 objects misplaced (6.028%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:50:17.563 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:17 smithi102 ceph-mon[110183]: osdmap e364: 8 total, 8 up, 8 in 2024-09-13T14:50:17.563 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:17 smithi102 ceph-mon[115269]: pgmap v1261: 109 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 105 active+clean; 754 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 691 op/s; 2175/36081 objects misplaced (6.028%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:50:17.563 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:17 smithi102 ceph-mon[115269]: osdmap e364: 8 total, 8 up, 8 in 2024-09-13T14:50:17.659 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:17.680 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:17.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:17.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:17.782 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:17.805 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:17.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:17.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:17.907 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:17.929 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:17.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:17.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:18.031 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:18.054 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:18.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:18.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:18.144 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:50:18.144 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-13T14:50:18.144 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-09-13T14:50:18.145 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-09-13T14:50:18.145 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:50:18.155 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:18.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:18.178 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:18.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:18.264 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:18.278 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:18.345 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:18.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:18.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:18.395 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:18 smithi102 ceph-mon[110183]: osdmap e365: 8 total, 8 up, 8 in 2024-09-13T14:50:18.396 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:18 smithi102 ceph-mon[115269]: osdmap e365: 8 total, 8 up, 8 in 2024-09-13T14:50:18.447 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:18.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:18.487 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:18.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:18.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:18 smithi195 ceph-mon[104262]: osdmap e365: 8 total, 8 up, 8 in 2024-09-13T14:50:18.588 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:18.614 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:18.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:18.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:18.716 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:18.755 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:18.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:18.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:18.857 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:18.875 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T14:50:18.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:18.906 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:18.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:19.007 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:19.042 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:19.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:19.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:19.049 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is set 2024-09-13T14:50:19.144 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:19.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:19.172 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:19.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:19.180 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:50:19.221 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:19 smithi195 ceph-mon[104262]: pgmap v1264: 109 pgs: 1 active+clean+remapped, 2 active+remapped+backfilling, 1 active+remapped+backfill_wait, 105 active+clean; 689 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 375 op/s; 1997/32925 objects misplaced (6.065%); 448 KiB/s, 6 objects/s recovering 2024-09-13T14:50:19.221 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:19 smithi195 ceph-mon[104262]: osdmap e366: 8 total, 8 up, 8 in 2024-09-13T14:50:19.221 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:19 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3433081670' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:50:19.221 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:19 smithi195 ceph-mon[104262]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:50:19.221 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:19 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3433081670' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:50:19.222 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:19 smithi195 ceph-mon[104262]: osdmap e367: 8 total, 8 up, 8 in 2024-09-13T14:50:19.273 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:19.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:19.298 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:19.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:19.399 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:19.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:19 smithi102 ceph-mon[110183]: pgmap v1264: 109 pgs: 1 active+clean+remapped, 2 active+remapped+backfilling, 1 active+remapped+backfill_wait, 105 active+clean; 689 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 375 op/s; 1997/32925 objects misplaced (6.065%); 448 KiB/s, 6 objects/s recovering 2024-09-13T14:50:19.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:19 smithi102 ceph-mon[110183]: osdmap e366: 8 total, 8 up, 8 in 2024-09-13T14:50:19.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:19 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3433081670' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:50:19.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:19 smithi102 ceph-mon[110183]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:50:19.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:19 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3433081670' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:50:19.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:19 smithi102 ceph-mon[110183]: osdmap e367: 8 total, 8 up, 8 in 2024-09-13T14:50:19.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:19 smithi102 ceph-mon[115269]: pgmap v1264: 109 pgs: 1 active+clean+remapped, 2 active+remapped+backfilling, 1 active+remapped+backfill_wait, 105 active+clean; 689 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 375 op/s; 1997/32925 objects misplaced (6.065%); 448 KiB/s, 6 objects/s recovering 2024-09-13T14:50:19.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:19 smithi102 ceph-mon[115269]: osdmap e366: 8 total, 8 up, 8 in 2024-09-13T14:50:19.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:19 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3433081670' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:50:19.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:19 smithi102 ceph-mon[115269]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:50:19.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:19 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3433081670' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:50:19.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:19 smithi102 ceph-mon[115269]: osdmap e367: 8 total, 8 up, 8 in 2024-09-13T14:50:19.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:19.421 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:19.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:19.522 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:19.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:19.544 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:19.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:19.645 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:19.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:19.668 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:19.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:19.769 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:19.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:19.793 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:19.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:19.896 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:19.926 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:19.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:19.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:20.028 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:20.053 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:20.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:20.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:20.156 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:20.178 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:20.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:20.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:20.281 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:20.316 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:20.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:20.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:20.418 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:20.459 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:20.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:20.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:20.561 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:20.589 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:20.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:20.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:20.691 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:20.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:20.727 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:20.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:20.829 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:20.864 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:20.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:20.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:20.966 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:21.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:21.009 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:21.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:21.110 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:21.138 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:50:21.138 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-09-13T14:50:21.138 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-09-13T14:50:21.138 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-13T14:50:21.139 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.139 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35379.0:179878 7.6 e290/254 7:66deb6ae:::benchmark_data_smithi102_227395_object67063:head v 290'11318, mlcod=0'0)", 2024-09-13T14:50:21.139 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:43:53.901082+0000", 2024-09-13T14:50:21.139 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 387.23162429000001, 2024-09-13T14:50:21.139 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.095177660999999997, 2024-09-13T14:50:21.139 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:21.139 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:21.139 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:21.139 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.139 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:21.139 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:53.901082+0000", 2024-09-13T14:50:21.140 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:21.140 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.140 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.140 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:21.140 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:53.901082+0000", 2024-09-13T14:50:21.140 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:21.140 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.140 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.140 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:21.140 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:53.901079+0000", 2024-09-13T14:50:21.140 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-09-13T14:50:21.140 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.140 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.140 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:21.141 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:53.901094+0000", 2024-09-13T14:50:21.141 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4562000000000001e-05 2024-09-13T14:50:21.141 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.141 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.141 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:21.141 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:53.993874+0000", 2024-09-13T14:50:21.141 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.092779529999999999 2024-09-13T14:50:21.141 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.141 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.141 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:21.141 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:53.993885+0000", 2024-09-13T14:50:21.141 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1620999999999999e-05 2024-09-13T14:50:21.141 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.142 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.142 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:21.142 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:53.993938+0000", 2024-09-13T14:50:21.142 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2377999999999999e-05 2024-09-13T14:50:21.142 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.142 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.142 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:21.142 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:53.993953+0000", 2024-09-13T14:50:21.142 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5244e-05 2024-09-13T14:50:21.142 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.142 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.142 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:21.143 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:53.996222+0000", 2024-09-13T14:50:21.143 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0022691709999999999 2024-09-13T14:50:21.143 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.143 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.143 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:21.143 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:53.996259+0000", 2024-09-13T14:50:21.143 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7354999999999997e-05 2024-09-13T14:50:21.143 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:21.143 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:21.143 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:21.143 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.143 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.144 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:179880 7.5 7:a9d4c919:::benchmark_data_smithi102_227395_object67065:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-13T14:50:21.144 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:43:53.930440+0000", 2024-09-13T14:50:21.144 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 387.20226613699998, 2024-09-13T14:50:21.144 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.067706946000000004, 2024-09-13T14:50:21.144 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:21.144 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:21.144 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:50:21.144 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:50:21.145 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:50:21.145 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 179880 2024-09-13T14:50:21.145 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.145 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:21.145 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.145 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:21.145 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:53.930440+0000", 2024-09-13T14:50:21.145 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:21.146 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.146 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.146 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:21.146 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:53.930440+0000", 2024-09-13T14:50:21.146 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:21.146 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.146 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.146 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:21.146 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:53.930438+0000", 2024-09-13T14:50:21.146 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:50:21.146 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.146 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.147 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:21.147 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:53.930447+0000", 2024-09-13T14:50:21.147 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8980000000000002e-06 2024-09-13T14:50:21.147 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.147 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.147 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:21.147 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:53.993881+0000", 2024-09-13T14:50:21.147 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.063434120999999996 2024-09-13T14:50:21.147 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.147 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.147 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:21.147 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:53.993896+0000", 2024-09-13T14:50:21.147 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4504e-05 2024-09-13T14:50:21.147 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.148 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.148 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:21.148 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:53.993948+0000", 2024-09-13T14:50:21.148 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2512999999999997e-05 2024-09-13T14:50:21.148 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.148 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.148 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:21.148 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:53.994076+0000", 2024-09-13T14:50:21.148 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000128033 2024-09-13T14:50:21.148 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.148 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.149 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,6", 2024-09-13T14:50:21.149 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:53.994199+0000", 2024-09-13T14:50:21.149 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012273199999999999 2024-09-13T14:50:21.149 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.149 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.149 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:21.149 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:53.997458+0000", 2024-09-13T14:50:21.149 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0032595620000000001 2024-09-13T14:50:21.149 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.149 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.150 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:50:21.150 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:53.997512+0000", 2024-09-13T14:50:21.150 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3834e-05 2024-09-13T14:50:21.150 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.150 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.150 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:21.150 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:53.998079+0000", 2024-09-13T14:50:21.150 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00056707800000000005 2024-09-13T14:50:21.150 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.150 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.150 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:21.150 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:53.998110+0000", 2024-09-13T14:50:21.151 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0870000000000001e-05 2024-09-13T14:50:21.151 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.151 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.151 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:21.151 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:53.998147+0000", 2024-09-13T14:50:21.151 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6606000000000002e-05 2024-09-13T14:50:21.151 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:21.151 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:21.151 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:21.151 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.151 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.151 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:204687 7.8 7:171412e0:::benchmark_data_smithi102_227395_object91872:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-13T14:50:21.151 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:45:00.892104+0000", 2024-09-13T14:50:21.151 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 320.24060170600001, 2024-09-13T14:50:21.152 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.082416823, 2024-09-13T14:50:21.152 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:21.152 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:21.152 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:50:21.152 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:50:21.152 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:50:21.152 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 204687 2024-09-13T14:50:21.152 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.152 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:21.152 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.152 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:21.152 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:00.892104+0000", 2024-09-13T14:50:21.152 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:21.152 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.153 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.153 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:21.153 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:00.892104+0000", 2024-09-13T14:50:21.153 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:21.153 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.153 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.153 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:21.153 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:00.892102+0000", 2024-09-13T14:50:21.153 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-09-13T14:50:21.153 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.153 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.153 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:21.153 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:00.892112+0000", 2024-09-13T14:50:21.154 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0464e-05 2024-09-13T14:50:21.154 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.154 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.154 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:21.154 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:00.892130+0000", 2024-09-13T14:50:21.154 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7479e-05 2024-09-13T14:50:21.154 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.154 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.154 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:21.154 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:00.892138+0000", 2024-09-13T14:50:21.154 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5609999999999999e-06 2024-09-13T14:50:21.154 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.154 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.155 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:21.155 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:00.892187+0000", 2024-09-13T14:50:21.155 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9277999999999998e-05 2024-09-13T14:50:21.155 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.155 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.155 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:21.155 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:00.892276+0000", 2024-09-13T14:50:21.155 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8521000000000002e-05 2024-09-13T14:50:21.155 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.155 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.155 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:50:21.155 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:00.967434+0000", 2024-09-13T14:50:21.155 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.075158290000000003 2024-09-13T14:50:21.156 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.156 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.156 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:21.156 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:00.974371+0000", 2024-09-13T14:50:21.156 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0069367200000000004 2024-09-13T14:50:21.156 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.156 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.156 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:50:21.156 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:00.974380+0000", 2024-09-13T14:50:21.156 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8419999999999994e-06 2024-09-13T14:50:21.156 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.156 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.156 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:21.156 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:00.974445+0000", 2024-09-13T14:50:21.157 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4791999999999999e-05 2024-09-13T14:50:21.157 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.157 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.157 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:21.157 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:00.974488+0000", 2024-09-13T14:50:21.157 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3717000000000002e-05 2024-09-13T14:50:21.157 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.157 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.157 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:21.157 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:00.974521+0000", 2024-09-13T14:50:21.157 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2666999999999997e-05 2024-09-13T14:50:21.157 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:21.157 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:21.157 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:21.158 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.158 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.158 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:204688 7.5 7:a4d2a347:::benchmark_data_smithi102_227395_object91873:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-13T14:50:21.158 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:45:00.928189+0000", 2024-09-13T14:50:21.158 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 320.20451718800001, 2024-09-13T14:50:21.158 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.046066831000000003, 2024-09-13T14:50:21.158 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:21.158 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:21.158 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:50:21.158 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:50:21.158 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:50:21.158 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 204688 2024-09-13T14:50:21.159 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.159 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:21.159 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.159 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:21.159 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:00.928189+0000", 2024-09-13T14:50:21.159 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:21.159 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.159 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.159 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:21.159 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:00.928189+0000", 2024-09-13T14:50:21.159 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:21.159 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.159 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.159 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:21.160 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:00.928187+0000", 2024-09-13T14:50:21.160 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:50:21.160 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.160 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.160 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:21.160 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:00.928198+0000", 2024-09-13T14:50:21.160 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0855e-05 2024-09-13T14:50:21.160 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.160 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.160 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:21.160 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:00.967390+0000", 2024-09-13T14:50:21.160 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.039192347000000002 2024-09-13T14:50:21.160 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.160 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.161 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:21.161 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:00.967403+0000", 2024-09-13T14:50:21.161 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3344e-05 2024-09-13T14:50:21.161 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.161 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.161 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:21.161 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:00.967454+0000", 2024-09-13T14:50:21.161 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0290999999999997e-05 2024-09-13T14:50:21.161 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.161 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.161 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:21.161 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:00.967567+0000", 2024-09-13T14:50:21.161 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011315399999999999 2024-09-13T14:50:21.161 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.162 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.162 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,6", 2024-09-13T14:50:21.162 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:00.967683+0000", 2024-09-13T14:50:21.162 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000116222 2024-09-13T14:50:21.162 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.162 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.162 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:21.162 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:00.970791+0000", 2024-09-13T14:50:21.162 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0031079969999999999 2024-09-13T14:50:21.162 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.162 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.162 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:50:21.163 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:00.971376+0000", 2024-09-13T14:50:21.163 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00058513599999999999 2024-09-13T14:50:21.163 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.163 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.163 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:21.163 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:00.974165+0000", 2024-09-13T14:50:21.163 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0027884429999999998 2024-09-13T14:50:21.163 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.163 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.163 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:21.163 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:00.974213+0000", 2024-09-13T14:50:21.163 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8832999999999998e-05 2024-09-13T14:50:21.163 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.163 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.164 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:21.164 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:00.974256+0000", 2024-09-13T14:50:21.164 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2194000000000003e-05 2024-09-13T14:50:21.164 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:21.164 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:21.164 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:21.164 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.164 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.164 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:222814 7.5 7:a7950af7:::benchmark_data_smithi102_227395_object109999:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-13T14:50:21.164 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:45:26.701503+0000", 2024-09-13T14:50:21.164 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 294.431203412, 2024-09-13T14:50:21.164 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.039333630000000001, 2024-09-13T14:50:21.164 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:21.164 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:21.165 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:50:21.165 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:50:21.165 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:50:21.165 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 222814 2024-09-13T14:50:21.165 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.165 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:21.165 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.165 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:21.165 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.701503+0000", 2024-09-13T14:50:21.165 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:21.165 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.165 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.165 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:21.165 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.701503+0000", 2024-09-13T14:50:21.166 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:21.166 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.166 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.166 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:21.166 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.701500+0000", 2024-09-13T14:50:21.166 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:50:21.166 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.166 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.166 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:21.166 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.701507+0000", 2024-09-13T14:50:21.166 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.985e-06 2024-09-13T14:50:21.166 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.166 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.166 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:21.167 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.701520+0000", 2024-09-13T14:50:21.167 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2534999999999999e-05 2024-09-13T14:50:21.167 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.167 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.167 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:21.167 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.701527+0000", 2024-09-13T14:50:21.167 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6769999999999998e-06 2024-09-13T14:50:21.167 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.167 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.167 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:21.167 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.705897+0000", 2024-09-13T14:50:21.167 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0043699730000000001 2024-09-13T14:50:21.167 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.167 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.168 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:21.168 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.706007+0000", 2024-09-13T14:50:21.168 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000110814 2024-09-13T14:50:21.168 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.168 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.168 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,6", 2024-09-13T14:50:21.168 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.706121+0000", 2024-09-13T14:50:21.168 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000113237 2024-09-13T14:50:21.168 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.168 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.169 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:50:21.169 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.709513+0000", 2024-09-13T14:50:21.169 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0033923130000000001 2024-09-13T14:50:21.169 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.169 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.169 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:21.169 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.709685+0000", 2024-09-13T14:50:21.169 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000171974 2024-09-13T14:50:21.169 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.169 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.169 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:21.170 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.740621+0000", 2024-09-13T14:50:21.170 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.030935635 2024-09-13T14:50:21.170 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.170 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.170 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:21.170 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.740793+0000", 2024-09-13T14:50:21.170 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000172755 2024-09-13T14:50:21.170 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.170 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.170 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:21.170 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.740836+0000", 2024-09-13T14:50:21.170 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2870999999999997e-05 2024-09-13T14:50:21.171 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:21.171 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:21.171 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:21.171 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.171 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.171 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:222995 7.8 7:1efa15f1:::benchmark_data_smithi102_227395_object110180:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-13T14:50:21.171 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:45:26.920447+0000", 2024-09-13T14:50:21.171 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 294.21225921799999, 2024-09-13T14:50:21.171 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.041171419000000001, 2024-09-13T14:50:21.171 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:21.171 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:21.172 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:50:21.172 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:50:21.172 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:50:21.172 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 222995 2024-09-13T14:50:21.172 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.172 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:21.172 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.172 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:21.172 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.920447+0000", 2024-09-13T14:50:21.172 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:21.172 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.172 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.172 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:21.172 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.920447+0000", 2024-09-13T14:50:21.173 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:21.173 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.173 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.173 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:21.173 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.920445+0000", 2024-09-13T14:50:21.173 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:50:21.173 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.173 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.173 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:21.173 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.920451+0000", 2024-09-13T14:50:21.173 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.9839999999999998e-06 2024-09-13T14:50:21.173 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.173 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.173 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:21.174 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.920462+0000", 2024-09-13T14:50:21.174 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0594000000000001e-05 2024-09-13T14:50:21.174 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.174 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.174 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:21.174 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.920468+0000", 2024-09-13T14:50:21.174 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3690000000000003e-06 2024-09-13T14:50:21.174 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.174 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.174 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:21.174 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.921145+0000", 2024-09-13T14:50:21.174 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00067689400000000004 2024-09-13T14:50:21.174 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.175 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.175 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:21.175 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.924874+0000", 2024-09-13T14:50:21.175 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0037284530000000001 2024-09-13T14:50:21.175 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.175 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.175 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:50:21.175 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.925009+0000", 2024-09-13T14:50:21.175 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013554800000000001 2024-09-13T14:50:21.175 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.175 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.175 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:21.176 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.937345+0000", 2024-09-13T14:50:21.176 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012336089999999999 2024-09-13T14:50:21.176 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.176 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.176 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:50:21.176 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.937365+0000", 2024-09-13T14:50:21.176 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0214e-05 2024-09-13T14:50:21.176 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.176 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.176 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:21.176 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.961546+0000", 2024-09-13T14:50:21.176 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024180533000000001 2024-09-13T14:50:21.176 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.176 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.177 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:21.177 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.961581+0000", 2024-09-13T14:50:21.177 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4854000000000002e-05 2024-09-13T14:50:21.177 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.177 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.177 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:21.177 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.961618+0000", 2024-09-13T14:50:21.177 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7347e-05 2024-09-13T14:50:21.177 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:21.177 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:21.178 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:21.178 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.178 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.178 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:222997 7.8 7:1573d4fb:::benchmark_data_smithi102_227395_object110182:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-13T14:50:21.178 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:45:26.925710+0000", 2024-09-13T14:50:21.178 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 294.20699615699999, 2024-09-13T14:50:21.178 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.036431186999999997, 2024-09-13T14:50:21.178 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:21.178 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:21.178 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:50:21.178 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:50:21.179 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:50:21.179 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 222997 2024-09-13T14:50:21.179 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.179 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:21.179 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.179 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:21.179 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.925710+0000", 2024-09-13T14:50:21.179 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:21.179 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.179 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.179 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:21.179 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.925710+0000", 2024-09-13T14:50:21.179 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:21.179 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.180 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.180 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:21.180 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.925708+0000", 2024-09-13T14:50:21.180 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:50:21.180 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.180 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.180 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:21.180 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.925716+0000", 2024-09-13T14:50:21.180 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7379999999999994e-06 2024-09-13T14:50:21.180 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.180 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.180 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:21.180 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.925728+0000", 2024-09-13T14:50:21.180 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1921e-05 2024-09-13T14:50:21.181 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.181 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.181 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:21.181 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.925735+0000", 2024-09-13T14:50:21.181 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9149999999999999e-06 2024-09-13T14:50:21.181 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.181 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.181 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:21.181 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.930146+0000", 2024-09-13T14:50:21.181 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0044115810000000004 2024-09-13T14:50:21.181 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.181 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.181 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:21.181 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.930265+0000", 2024-09-13T14:50:21.181 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000118516 2024-09-13T14:50:21.182 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.182 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.182 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:50:21.182 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.930381+0000", 2024-09-13T14:50:21.182 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000115827 2024-09-13T14:50:21.182 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.182 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.182 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:50:21.182 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.937373+0000", 2024-09-13T14:50:21.182 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0069920659999999999 2024-09-13T14:50:21.182 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.182 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.182 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:21.182 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.961450+0000", 2024-09-13T14:50:21.183 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024077596 2024-09-13T14:50:21.183 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.183 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.183 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:21.183 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.961647+0000", 2024-09-13T14:50:21.183 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00019698799999999999 2024-09-13T14:50:21.183 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.183 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.183 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:21.183 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.961681+0000", 2024-09-13T14:50:21.183 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4048000000000003e-05 2024-09-13T14:50:21.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:21.185 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:21.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:21.186 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.186 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.186 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:21.186 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.962141+0000", 2024-09-13T14:50:21.186 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000459693 2024-09-13T14:50:21.186 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:21.186 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:21.186 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:21.186 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.187 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.187 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:223002 7.8 7:14961ccf:::benchmark_data_smithi102_227395_object110187:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-13T14:50:21.187 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:45:26.929088+0000", 2024-09-13T14:50:21.187 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 294.20361833800001, 2024-09-13T14:50:21.187 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.037645892, 2024-09-13T14:50:21.187 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:21.187 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:21.187 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:50:21.187 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:50:21.187 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:50:21.187 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 223002 2024-09-13T14:50:21.187 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.188 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:21.188 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.188 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:21.188 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.929088+0000", 2024-09-13T14:50:21.188 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:21.188 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.188 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.188 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:21.188 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.929088+0000", 2024-09-13T14:50:21.188 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:21.188 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.189 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.189 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:21.189 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.929086+0000", 2024-09-13T14:50:21.189 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:50:21.189 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.189 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.189 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:21.189 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.929091+0000", 2024-09-13T14:50:21.189 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8099999999999997e-06 2024-09-13T14:50:21.189 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.189 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.189 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:21.189 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.929099+0000", 2024-09-13T14:50:21.190 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2879999999999993e-06 2024-09-13T14:50:21.190 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.190 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.190 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:21.190 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.929105+0000", 2024-09-13T14:50:21.190 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4099999999999999e-06 2024-09-13T14:50:21.190 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.190 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.190 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:21.190 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.933676+0000", 2024-09-13T14:50:21.190 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0045716589999999996 2024-09-13T14:50:21.190 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.190 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.190 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:21.191 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.933792+0000", 2024-09-13T14:50:21.191 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011514999999999999 2024-09-13T14:50:21.191 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.191 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.191 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:50:21.191 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.933923+0000", 2024-09-13T14:50:21.191 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000131505 2024-09-13T14:50:21.191 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.191 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.191 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:50:21.191 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.937377+0000", 2024-09-13T14:50:21.191 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0034543519999999999 2024-09-13T14:50:21.191 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.191 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.192 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:21.192 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.961519+0000", 2024-09-13T14:50:21.192 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024141882999999999 2024-09-13T14:50:21.192 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.192 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.192 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:21.192 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.966612+0000", 2024-09-13T14:50:21.192 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0050923050000000001 2024-09-13T14:50:21.192 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.192 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.192 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:21.192 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.966695+0000", 2024-09-13T14:50:21.192 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3657000000000002e-05 2024-09-13T14:50:21.193 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.193 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.193 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:21.193 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.966733+0000", 2024-09-13T14:50:21.193 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8157000000000001e-05 2024-09-13T14:50:21.193 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:21.193 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:21.193 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:21.193 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.193 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.193 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:223313 7.5 7:ae451614:::benchmark_data_smithi102_227395_object110498:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-13T14:50:21.194 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:45:27.226588+0000", 2024-09-13T14:50:21.194 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 293.90611789399998, 2024-09-13T14:50:21.194 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.034797102000000003, 2024-09-13T14:50:21.194 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:21.194 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:21.194 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:50:21.194 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:50:21.194 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:50:21.194 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 223313 2024-09-13T14:50:21.194 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.194 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:21.194 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.194 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:21.195 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:27.226588+0000", 2024-09-13T14:50:21.195 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:21.195 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.195 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.195 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:21.195 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:27.226588+0000", 2024-09-13T14:50:21.195 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:21.195 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.195 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.195 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:21.195 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:27.226587+0000", 2024-09-13T14:50:21.195 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:50:21.196 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.196 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.196 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:21.196 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:27.226591+0000", 2024-09-13T14:50:21.196 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5429999999999998e-06 2024-09-13T14:50:21.196 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.196 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.196 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:21.196 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:27.226598+0000", 2024-09-13T14:50:21.196 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4780000000000002e-06 2024-09-13T14:50:21.196 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.196 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.196 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:21.196 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:27.226604+0000", 2024-09-13T14:50:21.197 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2440000000000001e-06 2024-09-13T14:50:21.197 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.197 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.197 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:21.197 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:27.232576+0000", 2024-09-13T14:50:21.197 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0059726830000000003 2024-09-13T14:50:21.197 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.197 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.197 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:21.197 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:27.232671+0000", 2024-09-13T14:50:21.197 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.5044999999999995e-05 2024-09-13T14:50:21.197 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.197 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.197 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,6", 2024-09-13T14:50:21.198 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:27.232745+0000", 2024-09-13T14:50:21.198 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3938999999999998e-05 2024-09-13T14:50:21.198 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.198 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.198 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:50:21.198 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:27.235688+0000", 2024-09-13T14:50:21.198 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0029423990000000001 2024-09-13T14:50:21.198 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.198 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.198 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:21.198 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:27.235729+0000", 2024-09-13T14:50:21.198 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1220000000000002e-05 2024-09-13T14:50:21.198 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.199 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.199 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:21.199 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:27.261291+0000", 2024-09-13T14:50:21.199 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.025562451 2024-09-13T14:50:21.199 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.199 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.199 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:21.199 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:27.261356+0000", 2024-09-13T14:50:21.199 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4147999999999996e-05 2024-09-13T14:50:21.199 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.199 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.199 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:21.199 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:27.261385+0000", 2024-09-13T14:50:21.199 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9618999999999999e-05 2024-09-13T14:50:21.200 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:21.200 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:21.200 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:21.200 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.200 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.200 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:224543 7.5 7:a479f1cf:::benchmark_data_smithi102_227395_object111728:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-13T14:50:21.200 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:45:28.382500+0000", 2024-09-13T14:50:21.200 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 292.75020577499998, 2024-09-13T14:50:21.200 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.055261112000000001, 2024-09-13T14:50:21.200 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:21.201 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:21.201 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:50:21.201 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:50:21.201 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:50:21.201 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 224543 2024-09-13T14:50:21.201 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.201 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:21.201 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.201 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:21.201 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:28.382500+0000", 2024-09-13T14:50:21.201 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:21.201 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.201 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.201 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:21.202 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:28.382500+0000", 2024-09-13T14:50:21.202 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:21.202 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.202 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.202 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:21.202 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:28.382499+0000", 2024-09-13T14:50:21.202 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:50:21.202 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.202 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.202 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:21.202 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:28.382505+0000", 2024-09-13T14:50:21.202 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8499999999999999e-06 2024-09-13T14:50:21.202 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.202 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.203 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:21.203 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:28.382516+0000", 2024-09-13T14:50:21.203 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1161999999999999e-05 2024-09-13T14:50:21.203 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.203 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.203 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:21.203 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:28.382523+0000", 2024-09-13T14:50:21.203 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8120000000000002e-06 2024-09-13T14:50:21.203 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.203 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.203 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:21.203 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:28.382560+0000", 2024-09-13T14:50:21.203 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7184999999999997e-05 2024-09-13T14:50:21.204 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.204 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.204 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:21.204 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:28.382658+0000", 2024-09-13T14:50:21.204 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.8493999999999996e-05 2024-09-13T14:50:21.204 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.204 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.204 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,6", 2024-09-13T14:50:21.204 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:28.382757+0000", 2024-09-13T14:50:21.204 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.9166999999999994e-05 2024-09-13T14:50:21.204 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.204 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.204 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:50:21.205 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:28.386034+0000", 2024-09-13T14:50:21.205 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0032761190000000001 2024-09-13T14:50:21.205 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.205 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.205 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:21.205 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:28.386253+0000", 2024-09-13T14:50:21.205 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00021993800000000001 2024-09-13T14:50:21.205 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.205 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.205 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:21.205 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:28.437667+0000", 2024-09-13T14:50:21.205 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.051413106 2024-09-13T14:50:21.205 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.206 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.206 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:21.206 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:28.437721+0000", 2024-09-13T14:50:21.206 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4846000000000003e-05 2024-09-13T14:50:21.206 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.206 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.206 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:21.206 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:28.437761+0000", 2024-09-13T14:50:21.206 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9826000000000001e-05 2024-09-13T14:50:21.206 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:21.206 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:21.206 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:21.206 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.207 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.207 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.46157.0:55632 8.5 e315/304 8:aee4ccb2:::benchmark_data_smithi102_313275_object55631:head v 315'3480, mlcod=315'3477)", 2024-09-13T14:50:21.207 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:46:38.953408+0000", 2024-09-13T14:50:21.207 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 222.17929772100001, 2024-09-13T14:50:21.207 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.26008881099999998, 2024-09-13T14:50:21.207 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:21.207 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:21.207 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:21.207 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.207 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:21.207 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.953408+0000", 2024-09-13T14:50:21.207 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:21.207 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.207 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.208 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:21.208 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.953408+0000", 2024-09-13T14:50:21.208 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:21.208 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.208 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.208 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:21.208 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.953408+0000", 2024-09-13T14:50:21.208 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:50:21.208 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.208 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.208 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:21.208 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.953431+0000", 2024-09-13T14:50:21.208 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3513999999999999e-05 2024-09-13T14:50:21.209 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.209 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.209 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:21.209 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.953441+0000", 2024-09-13T14:50:21.209 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0386e-05 2024-09-13T14:50:21.209 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.209 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.209 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:21.209 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.953447+0000", 2024-09-13T14:50:21.209 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.9789999999999998e-06 2024-09-13T14:50:21.209 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.209 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.209 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:21.209 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.953538+0000", 2024-09-13T14:50:21.210 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1011e-05 2024-09-13T14:50:21.210 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.210 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.210 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:21.210 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.953556+0000", 2024-09-13T14:50:21.210 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7305000000000001e-05 2024-09-13T14:50:21.210 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.210 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.210 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:21.210 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.956339+0000", 2024-09-13T14:50:21.210 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0027834330000000001 2024-09-13T14:50:21.210 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.210 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.210 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:21.211 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.213497+0000", 2024-09-13T14:50:21.211 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.25715786600000001 2024-09-13T14:50:21.211 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:21.211 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:21.211 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:21.211 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.211 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.211 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.46157.0:55639 8.c e315/304 8:38a94be8:::benchmark_data_smithi102_313275_object55638:head v 315'3407, mlcod=315'3405)", 2024-09-13T14:50:21.211 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:46:38.956296+0000", 2024-09-13T14:50:21.211 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 222.176409812, 2024-09-13T14:50:21.211 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.260222915, 2024-09-13T14:50:21.211 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:21.211 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:21.211 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:21.212 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.212 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:21.212 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.956296+0000", 2024-09-13T14:50:21.212 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:21.212 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.212 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.212 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:21.212 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.956296+0000", 2024-09-13T14:50:21.212 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:21.212 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.212 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.212 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:21.212 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.956295+0000", 2024-09-13T14:50:21.212 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:50:21.213 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.213 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.213 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:21.213 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.956339+0000", 2024-09-13T14:50:21.213 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4138000000000003e-05 2024-09-13T14:50:21.213 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.213 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.213 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:21.213 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.213081+0000", 2024-09-13T14:50:21.213 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.256742093 2024-09-13T14:50:21.213 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.213 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.214 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:21.214 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.213095+0000", 2024-09-13T14:50:21.214 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4015e-05 2024-09-13T14:50:21.214 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.214 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.214 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:21.214 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.213155+0000", 2024-09-13T14:50:21.214 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.9474e-05 2024-09-13T14:50:21.214 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.214 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.215 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:21.215 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.213178+0000", 2024-09-13T14:50:21.215 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3693000000000002e-05 2024-09-13T14:50:21.215 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.215 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.215 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:21.215 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.216477+0000", 2024-09-13T14:50:21.215 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003298203 2024-09-13T14:50:21.215 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.215 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.215 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:21.215 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.216519+0000", 2024-09-13T14:50:21.215 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2330999999999998e-05 2024-09-13T14:50:21.215 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:21.216 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:21.216 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:21.216 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.216 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.216 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.46157.0:55637 8.8 e315/304 8:1ed1dfd1:::benchmark_data_smithi102_313275_object55636:head v 315'3505, mlcod=315'3503)", 2024-09-13T14:50:21.216 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:46:38.956602+0000", 2024-09-13T14:50:21.216 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 222.17610357000001, 2024-09-13T14:50:21.216 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.26366437300000001, 2024-09-13T14:50:21.216 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:21.216 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:21.216 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:21.216 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.216 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:21.216 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.956602+0000", 2024-09-13T14:50:21.217 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:21.217 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.217 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.217 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:21.217 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.956602+0000", 2024-09-13T14:50:21.217 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:21.217 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.217 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.217 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:21.217 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.956602+0000", 2024-09-13T14:50:21.217 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:50:21.217 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.217 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.217 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:21.218 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.956625+0000", 2024-09-13T14:50:21.218 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3578e-05 2024-09-13T14:50:21.218 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.218 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.218 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:21.218 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.213443+0000", 2024-09-13T14:50:21.218 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.25681791900000001 2024-09-13T14:50:21.218 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.218 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.218 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:21.218 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.213452+0000", 2024-09-13T14:50:21.218 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.0489999999999993e-06 2024-09-13T14:50:21.218 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.218 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.219 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:21.219 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.213886+0000", 2024-09-13T14:50:21.219 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000433977 2024-09-13T14:50:21.219 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.219 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.219 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:21.219 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.213905+0000", 2024-09-13T14:50:21.219 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8598e-05 2024-09-13T14:50:21.219 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.219 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.219 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:21.219 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.219641+0000", 2024-09-13T14:50:21.219 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0057360010000000001 2024-09-13T14:50:21.219 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.220 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.220 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:21.220 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.220267+0000", 2024-09-13T14:50:21.220 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00062600700000000004 2024-09-13T14:50:21.220 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:21.220 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:21.220 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:21.220 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.220 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.220 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.46157.0:100262 8.5 e326/304 8:a13eef64:::benchmark_data_smithi102_313275_object100261:head v 326'6255, mlcod=326'6253)", 2024-09-13T14:50:21.220 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:47:15.183921+0000", 2024-09-13T14:50:21.220 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 185.94878465799999, 2024-09-13T14:50:21.220 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.10644295199999999, 2024-09-13T14:50:21.221 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:21.221 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:21.221 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:21.221 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.221 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:21.221 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:15.183921+0000", 2024-09-13T14:50:21.221 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:21.221 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.221 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.221 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:21.221 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:15.183921+0000", 2024-09-13T14:50:21.221 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:21.221 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.221 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.222 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:21.222 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:15.183920+0000", 2024-09-13T14:50:21.222 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:50:21.222 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.222 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.222 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:21.222 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:15.183944+0000", 2024-09-13T14:50:21.222 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.37e-05 2024-09-13T14:50:21.222 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.222 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.222 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:21.222 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:15.183955+0000", 2024-09-13T14:50:21.222 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0789e-05 2024-09-13T14:50:21.222 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.223 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.223 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:21.223 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:15.183961+0000", 2024-09-13T14:50:21.223 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5520000000000001e-06 2024-09-13T14:50:21.223 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.223 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.223 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:21.223 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:15.184018+0000", 2024-09-13T14:50:21.223 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.6907000000000003e-05 2024-09-13T14:50:21.223 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.223 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.223 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:21.223 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:15.184036+0000", 2024-09-13T14:50:21.224 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7929999999999999e-05 2024-09-13T14:50:21.224 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.224 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.224 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:21.224 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:15.194968+0000", 2024-09-13T14:50:21.224 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010931814 2024-09-13T14:50:21.224 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.224 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.224 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:21.224 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:15.290364+0000", 2024-09-13T14:50:21.224 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.095396264999999994 2024-09-13T14:50:21.224 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:21.224 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:21.224 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:21.225 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.225 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.225 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.46157.0:100271 8.8 e326/304 8:10320491:::benchmark_data_smithi102_313275_object100270:head v 326'6300, mlcod=326'6297)", 2024-09-13T14:50:21.225 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:47:15.191500+0000", 2024-09-13T14:50:21.225 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 185.941205623, 2024-09-13T14:50:21.225 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.099242359000000002, 2024-09-13T14:50:21.225 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:21.225 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:21.225 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:21.225 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.225 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:21.225 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:15.191500+0000", 2024-09-13T14:50:21.225 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:21.225 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.225 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.226 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:21.226 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:15.191500+0000", 2024-09-13T14:50:21.226 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:21.226 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.226 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.226 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:21.226 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:15.191499+0000", 2024-09-13T14:50:21.226 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:50:21.226 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.226 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.226 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:21.226 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:15.191522+0000", 2024-09-13T14:50:21.226 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3306e-05 2024-09-13T14:50:21.227 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.227 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.227 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:21.227 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:15.191534+0000", 2024-09-13T14:50:21.227 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.136e-05 2024-09-13T14:50:21.227 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.227 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.227 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:21.227 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:15.191540+0000", 2024-09-13T14:50:21.227 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5189999999999998e-06 2024-09-13T14:50:21.227 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.227 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.227 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:21.228 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:15.191598+0000", 2024-09-13T14:50:21.228 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7416000000000001e-05 2024-09-13T14:50:21.228 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.228 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.228 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:21.228 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:15.191616+0000", 2024-09-13T14:50:21.228 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8357000000000001e-05 2024-09-13T14:50:21.228 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.228 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.228 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:21.228 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:15.290702+0000", 2024-09-13T14:50:21.228 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.099085490999999998 2024-09-13T14:50:21.228 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.228 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.229 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:21.229 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:15.290743+0000", 2024-09-13T14:50:21.229 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1145e-05 2024-09-13T14:50:21.229 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:21.229 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:21.229 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:21.229 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.229 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.229 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.46157.0:100274 8.6 e326/304 8:689cd0a5:::benchmark_data_smithi102_313275_object100273:head v 326'6241, mlcod=326'6239)", 2024-09-13T14:50:21.229 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:47:15.192682+0000", 2024-09-13T14:50:21.229 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 185.94002384199999, 2024-09-13T14:50:21.230 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.117266067, 2024-09-13T14:50:21.230 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:21.230 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:21.230 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:21.230 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.230 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:21.230 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:15.192682+0000", 2024-09-13T14:50:21.230 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:21.230 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.231 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.231 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:21.231 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:15.192682+0000", 2024-09-13T14:50:21.231 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:21.231 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.231 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.231 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:21.231 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:15.192681+0000", 2024-09-13T14:50:21.231 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:50:21.231 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.232 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.232 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:21.232 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:15.193962+0000", 2024-09-13T14:50:21.232 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001281293 2024-09-13T14:50:21.232 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.232 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.232 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:21.232 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:15.193976+0000", 2024-09-13T14:50:21.232 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3747000000000001e-05 2024-09-13T14:50:21.233 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.233 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.233 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:21.233 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:15.193984+0000", 2024-09-13T14:50:21.233 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.323e-06 2024-09-13T14:50:21.233 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.234 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.234 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:21.234 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:15.194165+0000", 2024-09-13T14:50:21.234 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00018188199999999999 2024-09-13T14:50:21.234 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.234 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.234 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:21.234 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:15.194186+0000", 2024-09-13T14:50:21.234 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0253000000000001e-05 2024-09-13T14:50:21.234 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.234 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.235 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:21.235 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:15.309893+0000", 2024-09-13T14:50:21.235 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.115707185 2024-09-13T14:50:21.235 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.235 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.235 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:21.236 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:15.309948+0000", 2024-09-13T14:50:21.236 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5300000000000002e-05 2024-09-13T14:50:21.236 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:21.236 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:21.236 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:21.236 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.236 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.236 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.46157.0:100279 8.6 e326/304 8:65d8d305:::benchmark_data_smithi102_313275_object100278:head v 326'6242, mlcod=326'6239)", 2024-09-13T14:50:21.236 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:47:15.197732+0000", 2024-09-13T14:50:21.236 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 185.93497441900001, 2024-09-13T14:50:21.236 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.112242298, 2024-09-13T14:50:21.236 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:21.236 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:21.236 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:21.237 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.237 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:21.237 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:15.197732+0000", 2024-09-13T14:50:21.237 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:21.237 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.237 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.237 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:21.237 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:15.197732+0000", 2024-09-13T14:50:21.237 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:21.237 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.237 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.237 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:21.237 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:15.197730+0000", 2024-09-13T14:50:21.239 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:50:21.239 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.239 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.239 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:21.239 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:15.197852+0000", 2024-09-13T14:50:21.239 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000121715 2024-09-13T14:50:21.239 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.239 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.239 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:21.239 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:15.290426+0000", 2024-09-13T14:50:21.240 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.092573921000000003 2024-09-13T14:50:21.240 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.240 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.240 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:21.240 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:15.290438+0000", 2024-09-13T14:50:21.240 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2036e-05 2024-09-13T14:50:21.240 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.240 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.240 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:21.240 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:15.293690+0000", 2024-09-13T14:50:21.240 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0032515970000000002 2024-09-13T14:50:21.240 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.240 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.241 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:21.241 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:15.293712+0000", 2024-09-13T14:50:21.241 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2065999999999998e-05 2024-09-13T14:50:21.241 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.241 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.241 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:21.241 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:15.309953+0000", 2024-09-13T14:50:21.241 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016241537 2024-09-13T14:50:21.241 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.241 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.241 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:21.241 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:15.309974+0000", 2024-09-13T14:50:21.241 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0601000000000001e-05 2024-09-13T14:50:21.241 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:21.242 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:21.242 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:21.242 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.242 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.242 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.46157.0:100280 8.d e326/306 8:b30fa90c:::benchmark_data_smithi102_313275_object100279:head v 326'6314, mlcod=326'6312)", 2024-09-13T14:50:21.242 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:47:15.198355+0000", 2024-09-13T14:50:21.242 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 185.934350505, 2024-09-13T14:50:21.242 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.10109264900000001, 2024-09-13T14:50:21.242 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:21.242 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:21.242 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:21.242 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.242 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:21.242 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:15.198355+0000", 2024-09-13T14:50:21.243 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:21.243 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.243 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.243 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:21.243 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:15.198355+0000", 2024-09-13T14:50:21.243 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:21.243 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.243 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.243 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:21.243 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:15.198354+0000", 2024-09-13T14:50:21.243 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:50:21.243 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.243 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.243 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:21.244 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:15.198381+0000", 2024-09-13T14:50:21.244 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.6537999999999999e-05 2024-09-13T14:50:21.244 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.244 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.244 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:21.244 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:15.290767+0000", 2024-09-13T14:50:21.244 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.092386239999999994 2024-09-13T14:50:21.244 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.244 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.244 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:21.244 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:15.290775+0000", 2024-09-13T14:50:21.244 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4889999999999998e-06 2024-09-13T14:50:21.244 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.245 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.245 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:21.245 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:15.290822+0000", 2024-09-13T14:50:21.245 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7735000000000003e-05 2024-09-13T14:50:21.245 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.245 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.245 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:21.245 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:15.290841+0000", 2024-09-13T14:50:21.245 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9040000000000001e-05 2024-09-13T14:50:21.245 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.245 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.245 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:21.245 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:15.299404+0000", 2024-09-13T14:50:21.245 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0085629490000000003 2024-09-13T14:50:21.246 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.246 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.246 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:21.246 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:15.299448+0000", 2024-09-13T14:50:21.246 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3686000000000002e-05 2024-09-13T14:50:21.246 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:21.246 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:21.246 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:21.246 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.246 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.246 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.46157.0:159129 8.6 e341/304 8:6ab62934:::benchmark_data_smithi102_313275_object28542:head v 341'9942, mlcod=341'9939)", 2024-09-13T14:50:21.246 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:48:18.141255+0000", 2024-09-13T14:50:21.246 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 122.99145093999999, 2024-09-13T14:50:21.246 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.029821009999999998, 2024-09-13T14:50:21.247 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:21.247 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:21.247 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:21.247 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.247 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:21.247 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:48:18.141255+0000", 2024-09-13T14:50:21.247 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:21.247 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.247 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.247 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:21.247 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:48:18.141255+0000", 2024-09-13T14:50:21.247 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:21.247 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.247 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.248 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:21.248 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:48:18.141254+0000", 2024-09-13T14:50:21.248 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:50:21.248 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.248 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.248 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:21.248 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:48:18.141263+0000", 2024-09-13T14:50:21.248 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1779999999999992e-06 2024-09-13T14:50:21.248 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.248 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.248 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:21.248 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:48:18.156199+0000", 2024-09-13T14:50:21.248 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014935637999999999 2024-09-13T14:50:21.248 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.249 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.249 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:21.249 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:48:18.156214+0000", 2024-09-13T14:50:21.249 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5136999999999999e-05 2024-09-13T14:50:21.249 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.249 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.249 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:21.249 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:48:18.156277+0000", 2024-09-13T14:50:21.249 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3119000000000003e-05 2024-09-13T14:50:21.249 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.249 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.249 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:21.249 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:48:18.156297+0000", 2024-09-13T14:50:21.249 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9743999999999999e-05 2024-09-13T14:50:21.250 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.250 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.250 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:21.250 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:48:18.171013+0000", 2024-09-13T14:50:21.250 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014716121 2024-09-13T14:50:21.250 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.250 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.250 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:21.250 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:48:18.171076+0000", 2024-09-13T14:50:21.250 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3317000000000004e-05 2024-09-13T14:50:21.250 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:21.250 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:21.250 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:21.250 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.251 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.251 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.46157.0:162270 8.5 e342/304 8:a618063e:::benchmark_data_smithi102_313275_object31683:head v 342'10153, mlcod=342'10151)", 2024-09-13T14:50:21.251 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:48:21.881839+0000", 2024-09-13T14:50:21.251 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 119.250867328, 2024-09-13T14:50:21.251 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.030331548999999999, 2024-09-13T14:50:21.251 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:21.251 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:21.251 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:21.251 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.251 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:21.251 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:48:21.881839+0000", 2024-09-13T14:50:21.251 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:21.251 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.252 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.252 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:21.252 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:48:21.881839+0000", 2024-09-13T14:50:21.252 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:21.252 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.252 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.252 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:21.252 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:48:21.881838+0000", 2024-09-13T14:50:21.252 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:50:21.252 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.252 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.252 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:21.252 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:48:21.881854+0000", 2024-09-13T14:50:21.252 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6524e-05 2024-09-13T14:50:21.253 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.253 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.253 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:21.253 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:48:21.881864+0000", 2024-09-13T14:50:21.253 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.8320000000000001e-06 2024-09-13T14:50:21.253 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.253 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.253 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:21.253 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:48:21.881871+0000", 2024-09-13T14:50:21.253 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5069999999999997e-06 2024-09-13T14:50:21.253 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.253 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.253 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:21.253 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:48:21.894007+0000", 2024-09-13T14:50:21.254 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012135853 2024-09-13T14:50:21.254 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.254 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.254 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:21.254 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:48:21.894053+0000", 2024-09-13T14:50:21.254 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.668e-05 2024-09-13T14:50:21.254 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.254 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.254 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:21.254 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:48:21.912102+0000", 2024-09-13T14:50:21.254 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018048966 2024-09-13T14:50:21.254 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:21.254 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:21.254 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:21.255 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:48:21.912170+0000", 2024-09-13T14:50:21.255 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7915000000000003e-05 2024-09-13T14:50:21.255 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:21.255 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:21.255 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:21.255 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:21.255 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:21.255 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:50:21.287 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:21.288 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:21 smithi102 ceph-mon[110183]: pgmap v1267: 109 pgs: 2 peering, 107 active+clean; 607 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 589 op/s; 30 MiB/s, 480 objects/s recovering 2024-09-13T14:50:21.288 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:21 smithi102 ceph-mon[110183]: osdmap e368: 8 total, 8 up, 8 in 2024-09-13T14:50:21.288 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:21 smithi102 ceph-mon[115269]: pgmap v1267: 109 pgs: 2 peering, 107 active+clean; 607 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 589 op/s; 30 MiB/s, 480 objects/s recovering 2024-09-13T14:50:21.288 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:21 smithi102 ceph-mon[115269]: osdmap e368: 8 total, 8 up, 8 in 2024-09-13T14:50:21.317 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:21.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:21.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:21.419 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:21.446 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:21.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:21.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:21.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:21 smithi195 ceph-mon[104262]: pgmap v1267: 109 pgs: 2 peering, 107 active+clean; 607 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 589 op/s; 30 MiB/s, 480 objects/s recovering 2024-09-13T14:50:21.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:21 smithi195 ceph-mon[104262]: osdmap e368: 8 total, 8 up, 8 in 2024-09-13T14:50:21.547 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:21.578 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:21.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:21.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:21.681 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:21.713 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:21.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:21.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:21.815 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:21.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:21.849 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:21.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:21.949 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:21.987 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:21.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:21.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:22.032 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T14:50:22.088 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:22.121 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:22.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:22.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:22.223 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:22.268 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:22.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:22.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:22.345 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:50:22.370 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:22.407 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:22.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:22.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:22.449 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:50:22.450 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:50:22.486 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:50:22.487 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:50:22.509 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:22.522 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:50:22.522 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:50:22.546 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:22.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:22.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:22.559 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:50:22.559 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:50:22.590 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:50:22.590 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:50:22.648 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:22.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:22.670 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:22.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:22.681 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:50:22.682 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:50:22.717 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:50:22.717 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:50:22.750 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:50:22.750 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:50:22.771 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:22.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:22.800 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:22.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:22.901 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:22.929 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:22.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:22.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:23.031 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:23.069 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:23.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:23.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:23.172 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:23.204 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:23.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:23.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:23.306 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:23.353 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:23.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:23.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:23.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:23 smithi102 ceph-mon[110183]: pgmap v1269: 109 pgs: 2 peering, 107 active+clean; 538 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 540 op/s; 27 MiB/s, 424 objects/s recovering 2024-09-13T14:50:23.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:23 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3112152904' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:50:23.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:23 smithi102 ceph-mon[115269]: pgmap v1269: 109 pgs: 2 peering, 107 active+clean; 538 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 540 op/s; 27 MiB/s, 424 objects/s recovering 2024-09-13T14:50:23.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:23 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3112152904' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:50:23.455 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:23.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:23.492 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:23.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:23.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:23 smithi195 ceph-mon[104262]: pgmap v1269: 109 pgs: 2 peering, 107 active+clean; 538 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 540 op/s; 27 MiB/s, 424 objects/s recovering 2024-09-13T14:50:23.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:23 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3112152904' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:50:23.593 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:23.631 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:23.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:23.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:23.733 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:23.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:23.762 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:23.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:23.860 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:50:23.862 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:23.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:23.909 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:23.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:24.010 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:24.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:24.072 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:24.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:24.172 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:24.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:24.218 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:24.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:24.320 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:24.349 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:24.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:24.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:24.451 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:24.482 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:24.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:24.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:24.584 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:24.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:24.620 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:24.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:24.692 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:24.720 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:24.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:24.781 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:24.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:24.882 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:24.904 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:24.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:24.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:24.929 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:50:24.934 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:50:25.005 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:25.037 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:25.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:25.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:25.139 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:25.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:25 smithi102 ceph-mon[115269]: pgmap v1270: 109 pgs: 1 peering, 108 active+clean; 456 MiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 619 op/s; 20 MiB/s, 315 objects/s recovering 2024-09-13T14:50:25.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:25 smithi102 ceph-mon[110183]: pgmap v1270: 109 pgs: 1 peering, 108 active+clean; 456 MiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 619 op/s; 20 MiB/s, 315 objects/s recovering 2024-09-13T14:50:25.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:25.178 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:25.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:25.280 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:25.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:25.314 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:25.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:25.415 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:25.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:25.438 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:25.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:25.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:25 smithi195 ceph-mon[104262]: pgmap v1270: 109 pgs: 1 peering, 108 active+clean; 456 MiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 619 op/s; 20 MiB/s, 315 objects/s recovering 2024-09-13T14:50:25.539 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:25.569 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:25.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:25.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:25.671 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:25.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:25.709 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:25.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:25.810 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:25.837 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:25.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:25.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:25.939 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:25.975 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:25.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:25.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:26.077 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:26.111 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:26.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:26.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:26.137 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:50:26.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:26 smithi102 ceph-mon[115269]: from='client.36450 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:50:26.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:50:26.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:50:26.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]} : dispatch 2024-09-13T14:50:26.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [3, 5, 1, 0]} : dispatch 2024-09-13T14:50:26.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]} : dispatch 2024-09-13T14:50:26.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:26 smithi102 ceph-mon[110183]: from='client.36450 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:50:26.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:50:26.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:50:26.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]} : dispatch 2024-09-13T14:50:26.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [3, 5, 1, 0]} : dispatch 2024-09-13T14:50:26.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]} : dispatch 2024-09-13T14:50:26.213 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:26.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:26.250 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:26.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:26.353 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:26.398 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:26.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:26.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:26.422 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:26 smithi195 ceph-mon[104262]: from='client.36450 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:50:26.422 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:50:26.422 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:50:26.422 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]} : dispatch 2024-09-13T14:50:26.422 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [3, 5, 1, 0]} : dispatch 2024-09-13T14:50:26.422 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]} : dispatch 2024-09-13T14:50:26.500 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:26.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:26.543 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:26.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:26.644 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:26.680 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:26.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:26.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:26.768 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:50:26.768 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:50:26.783 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:26.827 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:26.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:26.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:26.878 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:50:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:50:26] "GET /metrics HTTP/1.1" 200 36531 "" "Prometheus/2.51.0" 2024-09-13T14:50:26.917 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":1270,"stamp":"2024-09-13T14:50:23.672914+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":478484477,"num_objects":7257,"num_object_clones":0,"num_object_copies":21771,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7257,"num_whiteouts":0,"num_read":5999,"num_read_kb":13012,"num_write":254849,"num_write_kb":8370002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14152,"num_bytes_recovered":923207824,"num_keys_recovered":20,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121823,"ondisk_log_size":121823,"up":327,"acting":327,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":329,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6400660,"kb_used_data":1617476,"kb_used_omap":41,"kb_used_meta":4782934,"kb_avail":743560556,"statfs":{"total":767960285184,"available":761406009344,"internally_reserved":0,"allocated":1656295424,"data_stored":1641021095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42913,"internal_metadata":4897724511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,2,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-243662848,"num_objects":-3718,"num_object_clones":0,"num_object_copies":-11154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1997,"num_objects_unfound":0,"num_objects_dirty":-3718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3718,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1893,"num_bytes_recovered":123994112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001728"},"pg_stats":[{"pgid":"1.11","version":"90'149","reported_seq":477,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T14:50:20.250100+0000","last_change":"2024-09-13T14:47:38.688630+0000","last_active":"2024-09-13T14:50:20.250100+0000","last_peered":"2024-09-13T14:50:20.250100+0000","last_clean":"2024-09-13T14:50:20.250100+0000","last_became_active":"2024-09-13T14:47:38.460077+0000","last_became_peered":"2024-09-13T14:47:38.460077+0000","last_unstale":"2024-09-13T14:50:20.250100+0000","last_undegraded":"2024-09-13T14:50:20.250100+0000","last_fullsized":"2024-09-13T14:50:20.250100+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:38.688341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.688341+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:14:17.232921+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":646,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T14:50:20.255838+0000","last_change":"2024-09-13T14:47:47.868783+0000","last_active":"2024-09-13T14:50:20.255838+0000","last_peered":"2024-09-13T14:50:20.255838+0000","last_clean":"2024-09-13T14:50:20.255838+0000","last_became_active":"2024-09-13T14:47:47.868478+0000","last_became_peered":"2024-09-13T14:47:47.868478+0000","last_unstale":"2024-09-13T14:50:20.255838+0000","last_undegraded":"2024-09-13T14:50:20.255838+0000","last_fullsized":"2024-09-13T14:50:20.255838+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:27:01.784142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":388,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.059740+0000","last_change":"2024-09-13T14:46:01.772811+0000","last_active":"2024-09-13T14:50:19.059740+0000","last_peered":"2024-09-13T14:50:19.059740+0000","last_clean":"2024-09-13T14:50:19.059740+0000","last_became_active":"2024-09-13T14:46:01.764775+0000","last_became_peered":"2024-09-13T14:46:01.764775+0000","last_unstale":"2024-09-13T14:50:19.059740+0000","last_undegraded":"2024-09-13T14:50:19.059740+0000","last_fullsized":"2024-09-13T14:50:19.059740+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:42:20.738421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"90'149","reported_seq":433,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.053368+0000","last_change":"2024-09-13T14:47:41.482552+0000","last_active":"2024-09-13T14:50:19.053368+0000","last_peered":"2024-09-13T14:50:19.053368+0000","last_clean":"2024-09-13T14:50:19.053368+0000","last_became_active":"2024-09-13T14:47:38.466659+0000","last_became_peered":"2024-09-13T14:47:38.466659+0000","last_unstale":"2024-09-13T14:50:19.053368+0000","last_undegraded":"2024-09-13T14:50:19.053368+0000","last_fullsized":"2024-09-13T14:50:19.053368+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:41.482364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:41.482364+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:50:49.239389+0000","scrub_duration":0.027843855000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":169,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.054004+0000","last_change":"2024-09-13T14:36:13.862181+0000","last_active":"2024-09-13T14:50:19.054004+0000","last_peered":"2024-09-13T14:50:19.054004+0000","last_clean":"2024-09-13T14:50:19.054004+0000","last_became_active":"2024-09-13T14:36:13.861873+0000","last_became_peered":"2024-09-13T14:36:13.861873+0000","last_unstale":"2024-09-13T14:50:19.054004+0000","last_undegraded":"2024-09-13T14:50:19.054004+0000","last_fullsized":"2024-09-13T14:50:19.054004+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:14.085700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":270,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T14:50:20.251030+0000","last_change":"2024-09-13T14:38:53.896672+0000","last_active":"2024-09-13T14:50:20.251030+0000","last_peered":"2024-09-13T14:50:20.251030+0000","last_clean":"2024-09-13T14:50:20.251030+0000","last_became_active":"2024-09-13T14:38:53.896346+0000","last_became_peered":"2024-09-13T14:38:53.896346+0000","last_unstale":"2024-09-13T14:50:20.251030+0000","last_undegraded":"2024-09-13T14:50:20.251030+0000","last_fullsized":"2024-09-13T14:50:20.251030+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:57.789686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"90'149","reported_seq":495,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.056565+0000","last_change":"2024-09-13T14:47:51.226821+0000","last_active":"2024-09-13T14:50:19.056565+0000","last_peered":"2024-09-13T14:50:19.056565+0000","last_clean":"2024-09-13T14:50:19.056565+0000","last_became_active":"2024-09-13T14:47:50.881689+0000","last_became_peered":"2024-09-13T14:47:50.881689+0000","last_unstale":"2024-09-13T14:50:19.056565+0000","last_undegraded":"2024-09-13T14:50:19.056565+0000","last_fullsized":"2024-09-13T14:50:19.056565+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:51.226748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:51.226748+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:06:02.065990+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":270,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T14:50:20.251172+0000","last_change":"2024-09-13T14:38:53.893021+0000","last_active":"2024-09-13T14:50:20.251172+0000","last_peered":"2024-09-13T14:50:20.251172+0000","last_clean":"2024-09-13T14:50:20.251172+0000","last_became_active":"2024-09-13T14:38:53.892668+0000","last_became_peered":"2024-09-13T14:38:53.892668+0000","last_unstale":"2024-09-13T14:50:20.251172+0000","last_undegraded":"2024-09-13T14:50:20.251172+0000","last_fullsized":"2024-09-13T14:50:20.251172+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:57.180738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":341,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.060275+0000","last_change":"2024-09-13T14:37:48.134642+0000","last_active":"2024-09-13T14:50:19.060275+0000","last_peered":"2024-09-13T14:50:19.060275+0000","last_clean":"2024-09-13T14:50:19.060275+0000","last_became_active":"2024-09-13T14:37:48.134402+0000","last_became_peered":"2024-09-13T14:37:48.134402+0000","last_unstale":"2024-09-13T14:50:19.060275+0000","last_undegraded":"2024-09-13T14:50:19.060275+0000","last_fullsized":"2024-09-13T14:50:19.060275+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:51.313663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"90'149","reported_seq":450,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.062652+0000","last_change":"2024-09-13T14:47:50.892667+0000","last_active":"2024-09-13T14:50:19.062652+0000","last_peered":"2024-09-13T14:50:19.062652+0000","last_clean":"2024-09-13T14:50:19.062652+0000","last_became_active":"2024-09-13T14:47:50.883312+0000","last_became_peered":"2024-09-13T14:47:50.883312+0000","last_unstale":"2024-09-13T14:50:19.062652+0000","last_undegraded":"2024-09-13T14:50:19.062652+0000","last_fullsized":"2024-09-13T14:50:19.062652+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:45.441819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:45.441819+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:39.138573+0000","scrub_duration":0.021368558999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":665,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T14:50:20.253870+0000","last_change":"2024-09-13T14:47:47.867624+0000","last_active":"2024-09-13T14:50:20.253870+0000","last_peered":"2024-09-13T14:50:20.253870+0000","last_clean":"2024-09-13T14:50:20.253870+0000","last_became_active":"2024-09-13T14:47:47.867016+0000","last_became_peered":"2024-09-13T14:47:47.867016+0000","last_unstale":"2024-09-13T14:50:20.253870+0000","last_undegraded":"2024-09-13T14:50:20.253870+0000","last_fullsized":"2024-09-13T14:50:20.253870+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:55:45.344929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":273,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T14:50:20.249731+0000","last_change":"2024-09-13T14:38:03.554500+0000","last_active":"2024-09-13T14:50:20.249731+0000","last_peered":"2024-09-13T14:50:20.249731+0000","last_clean":"2024-09-13T14:50:20.249731+0000","last_became_active":"2024-09-13T14:38:03.532910+0000","last_became_peered":"2024-09-13T14:38:03.532910+0000","last_unstale":"2024-09-13T14:50:20.249731+0000","last_undegraded":"2024-09-13T14:50:20.249731+0000","last_fullsized":"2024-09-13T14:50:20.249731+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:27:50.843498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"90'149","reported_seq":452,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.060204+0000","last_change":"2024-09-13T14:47:50.887140+0000","last_active":"2024-09-13T14:50:19.060204+0000","last_peered":"2024-09-13T14:50:19.060204+0000","last_clean":"2024-09-13T14:50:19.060204+0000","last_became_active":"2024-09-13T14:47:50.886603+0000","last_became_peered":"2024-09-13T14:47:50.886603+0000","last_unstale":"2024-09-13T14:50:19.060204+0000","last_undegraded":"2024-09-13T14:50:19.060204+0000","last_fullsized":"2024-09-13T14:50:19.060204+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:40.786610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:40.786610+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:00.936651+0000","scrub_duration":0.013725185000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.4","version":"368'15730","reported_seq":15811,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T14:50:21.449148+0000","last_change":"2024-09-13T14:46:01.762751+0000","last_active":"2024-09-13T14:50:21.449148+0000","last_peered":"2024-09-13T14:50:21.449148+0000","last_clean":"2024-09-13T14:50:21.449148+0000","last_became_active":"2024-09-13T14:46:01.761929+0000","last_became_peered":"2024-09-13T14:46:01.761929+0000","last_unstale":"2024-09-13T14:50:21.449148+0000","last_undegraded":"2024-09-13T14:50:21.449148+0000","last_fullsized":"2024-09-13T14:50:21.449148+0000","mapping_epoch":306,"log_start":"335'7823","ondisk_log_start":"335'7823","created":304,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":7907,"log_dups_size":0,"ondisk_log_size":7907,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:20:57.325962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27394048,"num_objects":418,"num_object_clones":0,"num_object_copies":1254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15730,"num_write_kb":516736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":134,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.059874+0000","last_change":"2024-09-13T14:38:53.940238+0000","last_active":"2024-09-13T14:50:19.059874+0000","last_peered":"2024-09-13T14:50:19.059874+0000","last_clean":"2024-09-13T14:50:19.059874+0000","last_became_active":"2024-09-13T14:38:53.892825+0000","last_became_peered":"2024-09-13T14:38:53.892825+0000","last_unstale":"2024-09-13T14:50:19.059874+0000","last_undegraded":"2024-09-13T14:50:19.059874+0000","last_fullsized":"2024-09-13T14:50:19.059874+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:46.666581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":190,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.053776+0000","last_change":"2024-09-13T14:38:53.898907+0000","last_active":"2024-09-13T14:50:19.053776+0000","last_peered":"2024-09-13T14:50:19.053776+0000","last_clean":"2024-09-13T14:50:19.053776+0000","last_became_active":"2024-09-13T14:38:53.898572+0000","last_became_peered":"2024-09-13T14:38:53.898572+0000","last_unstale":"2024-09-13T14:50:19.053776+0000","last_undegraded":"2024-09-13T14:50:19.053776+0000","last_fullsized":"2024-09-13T14:50:19.053776+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:29.222150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"90'149","reported_seq":519,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T14:50:20.255683+0000","last_change":"2024-09-13T14:47:38.466101+0000","last_active":"2024-09-13T14:50:20.255683+0000","last_peered":"2024-09-13T14:50:20.255683+0000","last_clean":"2024-09-13T14:50:20.255683+0000","last_became_active":"2024-09-13T14:47:38.465607+0000","last_became_peered":"2024-09-13T14:47:38.465607+0000","last_unstale":"2024-09-13T14:50:20.255683+0000","last_undegraded":"2024-09-13T14:50:20.255683+0000","last_fullsized":"2024-09-13T14:50:20.255683+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:32.060817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:32.060817+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:55.362724+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.5","version":"368'15813","reported_seq":15888,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T14:50:21.200045+0000","last_change":"2024-09-13T14:46:00.028852+0000","last_active":"2024-09-13T14:50:21.200045+0000","last_peered":"2024-09-13T14:50:21.200045+0000","last_clean":"2024-09-13T14:50:21.200045+0000","last_became_active":"2024-09-13T14:46:00.028414+0000","last_became_peered":"2024-09-13T14:46:00.028414+0000","last_unstale":"2024-09-13T14:50:21.200045+0000","last_undegraded":"2024-09-13T14:50:21.200045+0000","last_fullsized":"2024-09-13T14:50:21.200045+0000","mapping_epoch":304,"log_start":"341'9497","ondisk_log_start":"341'9497","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":6316,"log_dups_size":0,"ondisk_log_size":6316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:03:28.278631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31784960,"num_objects":485,"num_object_clones":0,"num_object_copies":1455,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15813,"num_write_kb":521536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":147,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.060161+0000","last_change":"2024-09-13T14:47:47.878423+0000","last_active":"2024-09-13T14:50:19.060161+0000","last_peered":"2024-09-13T14:50:19.060161+0000","last_clean":"2024-09-13T14:50:19.060161+0000","last_became_active":"2024-09-13T14:47:47.866926+0000","last_became_peered":"2024-09-13T14:47:47.866926+0000","last_unstale":"2024-09-13T14:50:19.060161+0000","last_undegraded":"2024-09-13T14:50:19.060161+0000","last_fullsized":"2024-09-13T14:50:19.060161+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:11:07.534097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":388,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.049714+0000","last_change":"2024-09-13T14:47:47.860804+0000","last_active":"2024-09-13T14:50:19.049714+0000","last_peered":"2024-09-13T14:50:19.049714+0000","last_clean":"2024-09-13T14:50:19.049714+0000","last_became_active":"2024-09-13T14:47:47.860496+0000","last_became_peered":"2024-09-13T14:47:47.860496+0000","last_unstale":"2024-09-13T14:50:19.049714+0000","last_undegraded":"2024-09-13T14:50:19.049714+0000","last_fullsized":"2024-09-13T14:50:19.049714+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:17:57.542599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.f","version":"90'149","reported_seq":477,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.053645+0000","last_change":"2024-09-13T14:47:38.468287+0000","last_active":"2024-09-13T14:50:19.053645+0000","last_peered":"2024-09-13T14:50:19.053645+0000","last_clean":"2024-09-13T14:50:19.053645+0000","last_became_active":"2024-09-13T14:47:38.467941+0000","last_became_peered":"2024-09-13T14:47:38.467941+0000","last_unstale":"2024-09-13T14:50:19.053645+0000","last_undegraded":"2024-09-13T14:50:19.053645+0000","last_fullsized":"2024-09-13T14:50:19.053645+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:34.091782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:34.091782+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:34:20.519248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"368'15870","reported_seq":16011,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T14:50:22.993586+0000","last_change":"2024-09-13T14:46:00.027675+0000","last_active":"2024-09-13T14:50:22.993586+0000","last_peered":"2024-09-13T14:50:22.993586+0000","last_clean":"2024-09-13T14:50:22.993586+0000","last_became_active":"2024-09-13T14:46:00.027364+0000","last_became_peered":"2024-09-13T14:46:00.027364+0000","last_unstale":"2024-09-13T14:50:22.993586+0000","last_undegraded":"2024-09-13T14:50:22.993586+0000","last_fullsized":"2024-09-13T14:50:22.993586+0000","mapping_epoch":304,"log_start":"335'7878","ondisk_log_start":"335'7878","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":7992,"log_dups_size":0,"ondisk_log_size":7992,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:42:37.193007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24641536,"num_objects":376,"num_object_clones":0,"num_object_copies":1128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15870,"num_write_kb":519872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":371,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.060910+0000","last_change":"2024-09-13T14:37:49.611774+0000","last_active":"2024-09-13T14:50:19.060910+0000","last_peered":"2024-09-13T14:50:19.060910+0000","last_clean":"2024-09-13T14:50:19.060910+0000","last_became_active":"2024-09-13T14:37:49.611088+0000","last_became_peered":"2024-09-13T14:37:49.611088+0000","last_unstale":"2024-09-13T14:50:19.060910+0000","last_undegraded":"2024-09-13T14:50:19.060910+0000","last_fullsized":"2024-09-13T14:50:19.060910+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:08.933213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":73,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.061677+0000","last_change":"2024-09-13T14:47:47.865523+0000","last_active":"2024-09-13T14:50:19.061677+0000","last_peered":"2024-09-13T14:50:19.061677+0000","last_clean":"2024-09-13T14:50:19.061677+0000","last_became_active":"2024-09-13T14:47:47.865180+0000","last_became_peered":"2024-09-13T14:47:47.865180+0000","last_unstale":"2024-09-13T14:50:19.061677+0000","last_undegraded":"2024-09-13T14:50:19.061677+0000","last_fullsized":"2024-09-13T14:50:19.061677+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:39:35.479251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.e","version":"90'149","reported_seq":483,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.050741+0000","last_change":"2024-09-13T14:47:38.459769+0000","last_active":"2024-09-13T14:50:19.050741+0000","last_peered":"2024-09-13T14:50:19.050741+0000","last_clean":"2024-09-13T14:50:19.050741+0000","last_became_active":"2024-09-13T14:47:38.459440+0000","last_became_peered":"2024-09-13T14:47:38.459440+0000","last_unstale":"2024-09-13T14:50:19.050741+0000","last_undegraded":"2024-09-13T14:50:19.050741+0000","last_fullsized":"2024-09-13T14:50:19.050741+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:30.526999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:30.526999+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:00.016329+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"367'15872","reported_seq":27400,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.262592+0000","last_change":"2024-09-13T14:49:07.729693+0000","last_active":"2024-09-13T14:50:19.262592+0000","last_peered":"2024-09-13T14:50:19.262592+0000","last_clean":"2024-09-13T14:50:19.262592+0000","last_became_active":"2024-09-13T14:49:07.685432+0000","last_became_peered":"2024-09-13T14:49:07.685432+0000","last_unstale":"2024-09-13T14:50:19.262592+0000","last_undegraded":"2024-09-13T14:50:19.262592+0000","last_fullsized":"2024-09-13T14:50:19.262592+0000","mapping_epoch":348,"log_start":"341'9474","ondisk_log_start":"341'9474","created":304,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":6398,"log_dups_size":0,"ondisk_log_size":6398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:32:31.253024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33947648,"num_objects":518,"num_object_clones":0,"num_object_copies":1554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15872,"num_write_kb":524480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5685,"num_bytes_recovered":372572160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":181,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.053621+0000","last_change":"2024-09-13T14:37:49.168003+0000","last_active":"2024-09-13T14:50:19.053621+0000","last_peered":"2024-09-13T14:50:19.053621+0000","last_clean":"2024-09-13T14:50:19.053621+0000","last_became_active":"2024-09-13T14:37:49.167638+0000","last_became_peered":"2024-09-13T14:37:49.167638+0000","last_unstale":"2024-09-13T14:50:19.053621+0000","last_undegraded":"2024-09-13T14:50:19.053621+0000","last_fullsized":"2024-09-13T14:50:19.053621+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:29.051981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":270,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T14:50:20.250570+0000","last_change":"2024-09-13T14:47:47.868094+0000","last_active":"2024-09-13T14:50:20.250570+0000","last_peered":"2024-09-13T14:50:20.250570+0000","last_clean":"2024-09-13T14:50:20.250570+0000","last_became_active":"2024-09-13T14:47:47.867750+0000","last_became_peered":"2024-09-13T14:47:47.867750+0000","last_unstale":"2024-09-13T14:50:20.250570+0000","last_undegraded":"2024-09-13T14:50:20.250570+0000","last_fullsized":"2024-09-13T14:50:20.250570+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:55:38.203481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"90'149","reported_seq":543,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.059352+0000","last_change":"2024-09-13T14:46:55.355650+0000","last_active":"2024-09-13T14:50:19.059352+0000","last_peered":"2024-09-13T14:50:19.059352+0000","last_clean":"2024-09-13T14:50:19.059352+0000","last_became_active":"2024-09-13T14:46:48.150338+0000","last_became_peered":"2024-09-13T14:46:48.150338+0000","last_unstale":"2024-09-13T14:50:19.059352+0000","last_undegraded":"2024-09-13T14:50:19.059352+0000","last_fullsized":"2024-09-13T14:50:19.059352+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:46:55.355521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:46:55.355521+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:30:53.006278+0000","scrub_duration":0.002025448,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.0","version":"368'15862","reported_seq":22212,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T14:50:22.994154+0000","last_change":"2024-09-13T14:49:40.891281+0000","last_active":"2024-09-13T14:50:22.994154+0000","last_peered":"2024-09-13T14:50:22.994154+0000","last_clean":"2024-09-13T14:50:22.994154+0000","last_became_active":"2024-09-13T14:49:40.854627+0000","last_became_peered":"2024-09-13T14:49:40.854627+0000","last_unstale":"2024-09-13T14:50:22.994154+0000","last_undegraded":"2024-09-13T14:50:22.994154+0000","last_fullsized":"2024-09-13T14:50:22.994154+0000","mapping_epoch":353,"log_start":"341'9006","ondisk_log_start":"341'9006","created":304,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":6856,"log_dups_size":0,"ondisk_log_size":6856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:34:27.653199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22675456,"num_objects":346,"num_object_clones":0,"num_object_copies":1038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15862,"num_write_kb":518656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3095,"num_bytes_recovered":202833920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":621,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T14:50:20.253199+0000","last_change":"2024-09-13T14:38:03.549595+0000","last_active":"2024-09-13T14:50:20.253199+0000","last_peered":"2024-09-13T14:50:20.253199+0000","last_clean":"2024-09-13T14:50:20.253199+0000","last_became_active":"2024-09-13T14:38:03.533331+0000","last_became_peered":"2024-09-13T14:38:03.533331+0000","last_unstale":"2024-09-13T14:50:20.253199+0000","last_undegraded":"2024-09-13T14:50:20.253199+0000","last_fullsized":"2024-09-13T14:50:20.253199+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:59:02.470343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":364,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.050896+0000","last_change":"2024-09-13T14:46:26.980367+0000","last_active":"2024-09-13T14:50:19.050896+0000","last_peered":"2024-09-13T14:50:19.050896+0000","last_clean":"2024-09-13T14:50:19.050896+0000","last_became_active":"2024-09-13T14:46:26.979937+0000","last_became_peered":"2024-09-13T14:46:26.979937+0000","last_unstale":"2024-09-13T14:50:19.050896+0000","last_undegraded":"2024-09-13T14:50:19.050896+0000","last_fullsized":"2024-09-13T14:50:19.050896+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:56.830872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"90'149","reported_seq":445,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.053432+0000","last_change":"2024-09-13T14:46:54.393663+0000","last_active":"2024-09-13T14:50:19.053432+0000","last_peered":"2024-09-13T14:50:19.053432+0000","last_clean":"2024-09-13T14:50:19.053432+0000","last_became_active":"2024-09-13T14:46:48.149004+0000","last_became_peered":"2024-09-13T14:46:48.149004+0000","last_unstale":"2024-09-13T14:50:19.053432+0000","last_undegraded":"2024-09-13T14:50:19.053432+0000","last_fullsized":"2024-09-13T14:50:19.053432+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:46:54.393204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:46:54.393204+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:32:40.003989+0000","scrub_duration":0.015172223,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1","version":"367'15629","reported_seq":17635,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.281269+0000","last_change":"2024-09-13T14:50:16.263279+0000","last_active":"2024-09-13T14:50:19.281269+0000","last_peered":"2024-09-13T14:50:19.281269+0000","last_clean":"2024-09-13T14:50:19.281269+0000","last_became_active":"2024-09-13T14:50:16.210911+0000","last_became_peered":"2024-09-13T14:50:16.210911+0000","last_unstale":"2024-09-13T14:50:19.281269+0000","last_undegraded":"2024-09-13T14:50:19.281269+0000","last_fullsized":"2024-09-13T14:50:19.281269+0000","mapping_epoch":363,"log_start":"341'9243","ondisk_log_start":"341'9243","created":304,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":6386,"log_dups_size":0,"ondisk_log_size":6386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:41:01.774596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38076416,"num_objects":581,"num_object_clones":0,"num_object_copies":1743,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15629,"num_write_kb":518720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":921,"num_bytes_recovered":60358656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":297,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.050042+0000","last_change":"2024-09-13T14:47:47.867571+0000","last_active":"2024-09-13T14:50:19.050042+0000","last_peered":"2024-09-13T14:50:19.050042+0000","last_clean":"2024-09-13T14:50:19.050042+0000","last_became_active":"2024-09-13T14:47:47.867269+0000","last_became_peered":"2024-09-13T14:47:47.867269+0000","last_unstale":"2024-09-13T14:50:19.050042+0000","last_undegraded":"2024-09-13T14:50:19.050042+0000","last_fullsized":"2024-09-13T14:50:19.050042+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:46.093600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":426,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.050373+0000","last_change":"2024-09-13T14:47:47.866925+0000","last_active":"2024-09-13T14:50:19.050373+0000","last_peered":"2024-09-13T14:50:19.050373+0000","last_clean":"2024-09-13T14:50:19.050373+0000","last_became_active":"2024-09-13T14:47:47.866596+0000","last_became_peered":"2024-09-13T14:47:47.866596+0000","last_unstale":"2024-09-13T14:50:19.050373+0000","last_undegraded":"2024-09-13T14:50:19.050373+0000","last_fullsized":"2024-09-13T14:50:19.050373+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:11:25.617891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"90'149","reported_seq":508,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.056412+0000","last_change":"2024-09-13T14:47:50.891055+0000","last_active":"2024-09-13T14:50:19.056412+0000","last_peered":"2024-09-13T14:50:19.056412+0000","last_clean":"2024-09-13T14:50:19.056412+0000","last_became_active":"2024-09-13T14:47:50.890368+0000","last_became_peered":"2024-09-13T14:47:50.890368+0000","last_unstale":"2024-09-13T14:50:19.056412+0000","last_undegraded":"2024-09-13T14:50:19.056412+0000","last_fullsized":"2024-09-13T14:50:19.056412+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:39.645908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:39.645908+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:29.800728+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"368'15768","reported_seq":15843,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T14:50:21.265891+0000","last_change":"2024-09-13T14:45:59.744581+0000","last_active":"2024-09-13T14:50:21.265891+0000","last_peered":"2024-09-13T14:50:21.265891+0000","last_clean":"2024-09-13T14:50:21.265891+0000","last_became_active":"2024-09-13T14:45:59.744231+0000","last_became_peered":"2024-09-13T14:45:59.744231+0000","last_unstale":"2024-09-13T14:50:21.265891+0000","last_undegraded":"2024-09-13T14:50:21.265891+0000","last_fullsized":"2024-09-13T14:50:21.265891+0000","mapping_epoch":304,"log_start":"341'9456","ondisk_log_start":"341'9456","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":6312,"log_dups_size":0,"ondisk_log_size":6312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:06:14.502840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33161216,"num_objects":506,"num_object_clones":0,"num_object_copies":1518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15768,"num_write_kb":520768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":377,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.050057+0000","last_change":"2024-09-13T14:46:01.762127+0000","last_active":"2024-09-13T14:50:19.050057+0000","last_peered":"2024-09-13T14:50:19.050057+0000","last_clean":"2024-09-13T14:50:19.050057+0000","last_became_active":"2024-09-13T14:46:01.761795+0000","last_became_peered":"2024-09-13T14:46:01.761795+0000","last_unstale":"2024-09-13T14:50:19.050057+0000","last_undegraded":"2024-09-13T14:50:19.050057+0000","last_fullsized":"2024-09-13T14:50:19.050057+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:41:15.626775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":373,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.050959+0000","last_change":"2024-09-13T14:47:47.861718+0000","last_active":"2024-09-13T14:50:19.050959+0000","last_peered":"2024-09-13T14:50:19.050959+0000","last_clean":"2024-09-13T14:50:19.050959+0000","last_became_active":"2024-09-13T14:47:47.861023+0000","last_became_peered":"2024-09-13T14:47:47.861023+0000","last_unstale":"2024-09-13T14:50:19.050959+0000","last_undegraded":"2024-09-13T14:50:19.050959+0000","last_fullsized":"2024-09-13T14:50:19.050959+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:12:13.274053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"90'149","reported_seq":445,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.059539+0000","last_change":"2024-09-13T14:46:52.334715+0000","last_active":"2024-09-13T14:50:19.059539+0000","last_peered":"2024-09-13T14:50:19.059539+0000","last_clean":"2024-09-13T14:50:19.059539+0000","last_became_active":"2024-09-13T14:46:48.158685+0000","last_became_peered":"2024-09-13T14:46:48.158685+0000","last_unstale":"2024-09-13T14:50:19.059539+0000","last_undegraded":"2024-09-13T14:50:19.059539+0000","last_fullsized":"2024-09-13T14:50:19.059539+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:46:52.334301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:46:52.334301+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:46:03.579759+0000","scrub_duration":0.0050940330000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.3","version":"367'15698","reported_seq":19654,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.232665+0000","last_change":"2024-09-13T14:50:03.175084+0000","last_active":"2024-09-13T14:50:19.232665+0000","last_peered":"2024-09-13T14:50:19.232665+0000","last_clean":"2024-09-13T14:50:19.232665+0000","last_became_active":"2024-09-13T14:50:03.105773+0000","last_became_peered":"2024-09-13T14:50:03.105773+0000","last_unstale":"2024-09-13T14:50:19.232665+0000","last_undegraded":"2024-09-13T14:50:19.232665+0000","last_fullsized":"2024-09-13T14:50:19.232665+0000","mapping_epoch":358,"log_start":"341'9291","ondisk_log_start":"341'9291","created":304,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":6407,"log_dups_size":0,"ondisk_log_size":6407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:22:19.749633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38535168,"num_objects":588,"num_object_clones":0,"num_object_copies":1764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15698,"num_write_kb":521152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1914,"num_bytes_recovered":125435904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":646,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T14:50:20.252507+0000","last_change":"2024-09-13T14:47:47.861249+0000","last_active":"2024-09-13T14:50:20.252507+0000","last_peered":"2024-09-13T14:50:20.252507+0000","last_clean":"2024-09-13T14:50:20.252507+0000","last_became_active":"2024-09-13T14:47:47.860873+0000","last_became_peered":"2024-09-13T14:47:47.860873+0000","last_unstale":"2024-09-13T14:50:20.252507+0000","last_undegraded":"2024-09-13T14:50:20.252507+0000","last_fullsized":"2024-09-13T14:50:20.252507+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:37:50.709437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":73,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.056304+0000","last_change":"2024-09-13T14:47:47.870185+0000","last_active":"2024-09-13T14:50:19.056304+0000","last_peered":"2024-09-13T14:50:19.056304+0000","last_clean":"2024-09-13T14:50:19.056304+0000","last_became_active":"2024-09-13T14:47:47.869862+0000","last_became_peered":"2024-09-13T14:47:47.869862+0000","last_unstale":"2024-09-13T14:50:19.056304+0000","last_undegraded":"2024-09-13T14:50:19.056304+0000","last_fullsized":"2024-09-13T14:50:19.056304+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:07:02.812306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"368'16083","reported_seq":16157,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T14:50:21.434910+0000","last_change":"2024-09-13T14:46:00.027751+0000","last_active":"2024-09-13T14:50:21.434910+0000","last_peered":"2024-09-13T14:50:21.434910+0000","last_clean":"2024-09-13T14:50:21.434910+0000","last_became_active":"2024-09-13T14:46:00.027429+0000","last_became_peered":"2024-09-13T14:46:00.027429+0000","last_unstale":"2024-09-13T14:50:21.434910+0000","last_undegraded":"2024-09-13T14:50:21.434910+0000","last_fullsized":"2024-09-13T14:50:21.434910+0000","mapping_epoch":304,"log_start":"335'8184","ondisk_log_start":"335'8184","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":7899,"log_dups_size":0,"ondisk_log_size":7899,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:17:52.381240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30867456,"num_objects":471,"num_object_clones":0,"num_object_copies":1413,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16083,"num_write_kb":529728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":548,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.059708+0000","last_change":"2024-09-13T14:47:31.684319+0000","last_active":"2024-09-13T14:50:19.059708+0000","last_peered":"2024-09-13T14:50:19.059708+0000","last_clean":"2024-09-13T14:50:19.059708+0000","last_became_active":"2024-09-13T14:47:29.415658+0000","last_became_peered":"2024-09-13T14:47:29.415658+0000","last_unstale":"2024-09-13T14:50:19.059708+0000","last_undegraded":"2024-09-13T14:50:19.059708+0000","last_fullsized":"2024-09-13T14:50:19.059708+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:31.681433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:31.681433+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:29:34.663513+0000","scrub_duration":0.0059601139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":676,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.053595+0000","last_change":"2024-09-13T14:46:04.780718+0000","last_active":"2024-09-13T14:50:19.053595+0000","last_peered":"2024-09-13T14:50:19.053595+0000","last_clean":"2024-09-13T14:50:19.053595+0000","last_became_active":"2024-09-13T14:46:04.780120+0000","last_became_peered":"2024-09-13T14:46:04.780120+0000","last_unstale":"2024-09-13T14:50:19.053595+0000","last_undegraded":"2024-09-13T14:50:19.053595+0000","last_fullsized":"2024-09-13T14:50:19.053595+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:57:14.475888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":574,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T14:50:20.253044+0000","last_change":"2024-09-13T14:37:49.160719+0000","last_active":"2024-09-13T14:50:20.253044+0000","last_peered":"2024-09-13T14:50:20.253044+0000","last_clean":"2024-09-13T14:50:20.253044+0000","last_became_active":"2024-09-13T14:37:49.160401+0000","last_became_peered":"2024-09-13T14:37:49.160401+0000","last_unstale":"2024-09-13T14:50:20.253044+0000","last_undegraded":"2024-09-13T14:50:20.253044+0000","last_fullsized":"2024-09-13T14:50:20.253044+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:21.737945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":215,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.050309+0000","last_change":"2024-09-13T14:37:48.172940+0000","last_active":"2024-09-13T14:50:19.050309+0000","last_peered":"2024-09-13T14:50:19.050309+0000","last_clean":"2024-09-13T14:50:19.050309+0000","last_became_active":"2024-09-13T14:37:48.172060+0000","last_became_peered":"2024-09-13T14:37:48.172060+0000","last_unstale":"2024-09-13T14:50:19.050309+0000","last_undegraded":"2024-09-13T14:50:19.050309+0000","last_fullsized":"2024-09-13T14:50:19.050309+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:07.480645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"90'149","reported_seq":502,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.056470+0000","last_change":"2024-09-13T14:47:50.884721+0000","last_active":"2024-09-13T14:50:19.056470+0000","last_peered":"2024-09-13T14:50:19.056470+0000","last_clean":"2024-09-13T14:50:19.056470+0000","last_became_active":"2024-09-13T14:47:50.880185+0000","last_became_peered":"2024-09-13T14:47:50.880185+0000","last_unstale":"2024-09-13T14:50:19.056470+0000","last_undegraded":"2024-09-13T14:50:19.056470+0000","last_fullsized":"2024-09-13T14:50:19.056470+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:34.603316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:34.603316+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:38.321686+0000","scrub_duration":0.0057131120000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.a","version":"366'16005","reported_seq":17378,"reported_epoch":367,"state":"peering","last_fresh":"2024-09-13T14:50:19.058776+0000","last_change":"2024-09-13T14:50:19.058776+0000","last_active":"2024-09-13T14:50:19.058776+0000","last_peered":"2024-09-13T14:50:19.013671+0000","last_clean":"2024-09-13T14:50:19.013671+0000","last_became_active":"2024-09-13T14:47:49.272934+0000","last_became_peered":"2024-09-13T14:47:49.272934+0000","last_unstale":"2024-09-13T14:50:19.058776+0000","last_undegraded":"2024-09-13T14:50:19.058776+0000","last_fullsized":"2024-09-13T14:50:19.058776+0000","mapping_epoch":367,"log_start":"326'6257","ondisk_log_start":"326'6257","created":304,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":9748,"log_dups_size":0,"ondisk_log_size":9748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37027840,"num_objects":565,"num_object_clones":0,"num_object_copies":1695,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16005,"num_write_kb":530240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":613,"num_bytes_recovered":40173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":617,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.059632+0000","last_change":"2024-09-13T14:47:47.892973+0000","last_active":"2024-09-13T14:50:19.059632+0000","last_peered":"2024-09-13T14:50:19.059632+0000","last_clean":"2024-09-13T14:50:19.059632+0000","last_became_active":"2024-09-13T14:47:47.866986+0000","last_became_peered":"2024-09-13T14:47:47.866986+0000","last_unstale":"2024-09-13T14:50:19.059632+0000","last_undegraded":"2024-09-13T14:50:19.059632+0000","last_fullsized":"2024-09-13T14:50:19.059632+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:07.353290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":263,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T14:50:20.250021+0000","last_change":"2024-09-13T14:38:03.525682+0000","last_active":"2024-09-13T14:50:20.250021+0000","last_peered":"2024-09-13T14:50:20.250021+0000","last_clean":"2024-09-13T14:50:20.250021+0000","last_became_active":"2024-09-13T14:38:03.525406+0000","last_became_peered":"2024-09-13T14:38:03.525406+0000","last_unstale":"2024-09-13T14:50:20.250021+0000","last_undegraded":"2024-09-13T14:50:20.250021+0000","last_fullsized":"2024-09-13T14:50:20.250021+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:41:28.970300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":284,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T14:50:20.251822+0000","last_change":"2024-09-13T14:47:47.873233+0000","last_active":"2024-09-13T14:50:20.251822+0000","last_peered":"2024-09-13T14:50:20.251822+0000","last_clean":"2024-09-13T14:50:20.251822+0000","last_became_active":"2024-09-13T14:47:47.872788+0000","last_became_peered":"2024-09-13T14:47:47.872788+0000","last_unstale":"2024-09-13T14:50:20.251822+0000","last_undegraded":"2024-09-13T14:50:20.251822+0000","last_fullsized":"2024-09-13T14:50:20.251822+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:52.409108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"90'149","reported_seq":564,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.059570+0000","last_change":"2024-09-13T14:47:29.629488+0000","last_active":"2024-09-13T14:50:19.059570+0000","last_peered":"2024-09-13T14:50:19.059570+0000","last_clean":"2024-09-13T14:50:19.059570+0000","last_became_active":"2024-09-13T14:47:29.403279+0000","last_became_peered":"2024-09-13T14:47:29.403279+0000","last_unstale":"2024-09-13T14:50:19.059570+0000","last_undegraded":"2024-09-13T14:50:19.059570+0000","last_fullsized":"2024-09-13T14:50:19.059570+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:29.629342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:29.629342+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:14:55.249896+0000","scrub_duration":0.0099290990000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"368'16052","reported_seq":17507,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T14:50:23.006540+0000","last_change":"2024-09-13T14:50:18.268858+0000","last_active":"2024-09-13T14:50:23.006540+0000","last_peered":"2024-09-13T14:50:23.006540+0000","last_clean":"2024-09-13T14:50:23.006540+0000","last_became_active":"2024-09-13T14:50:18.223597+0000","last_became_peered":"2024-09-13T14:50:18.223597+0000","last_unstale":"2024-09-13T14:50:23.006540+0000","last_undegraded":"2024-09-13T14:50:23.006540+0000","last_fullsized":"2024-09-13T14:50:23.006540+0000","mapping_epoch":365,"log_start":"341'9177","ondisk_log_start":"341'9177","created":304,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":6875,"log_dups_size":0,"ondisk_log_size":6875,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:05:01.758931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23068672,"num_objects":352,"num_object_clones":0,"num_object_copies":1056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16052,"num_write_kb":524928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":644,"num_bytes_recovered":42139648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":558,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T14:50:20.053024+0000","last_change":"2024-09-13T14:46:04.782955+0000","last_active":"2024-09-13T14:50:20.053024+0000","last_peered":"2024-09-13T14:50:20.053024+0000","last_clean":"2024-09-13T14:50:20.053024+0000","last_became_active":"2024-09-13T14:46:04.782250+0000","last_became_peered":"2024-09-13T14:46:04.782250+0000","last_unstale":"2024-09-13T14:50:20.053024+0000","last_undegraded":"2024-09-13T14:50:20.053024+0000","last_fullsized":"2024-09-13T14:50:20.053024+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:24.596922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":73,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.055890+0000","last_change":"2024-09-13T14:47:47.866684+0000","last_active":"2024-09-13T14:50:19.055890+0000","last_peered":"2024-09-13T14:50:19.055890+0000","last_clean":"2024-09-13T14:50:19.055890+0000","last_became_active":"2024-09-13T14:47:47.866400+0000","last_became_peered":"2024-09-13T14:47:47.866400+0000","last_unstale":"2024-09-13T14:50:19.055890+0000","last_undegraded":"2024-09-13T14:50:19.055890+0000","last_fullsized":"2024-09-13T14:50:19.055890+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:08:34.289658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":73,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.056387+0000","last_change":"2024-09-13T14:47:47.870218+0000","last_active":"2024-09-13T14:50:19.056387+0000","last_peered":"2024-09-13T14:50:19.056387+0000","last_clean":"2024-09-13T14:50:19.056387+0000","last_became_active":"2024-09-13T14:47:47.869907+0000","last_became_peered":"2024-09-13T14:47:47.869907+0000","last_unstale":"2024-09-13T14:50:19.056387+0000","last_undegraded":"2024-09-13T14:50:19.056387+0000","last_fullsized":"2024-09-13T14:50:19.056387+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:41:49.141441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"90'149","reported_seq":466,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.054001+0000","last_change":"2024-09-13T14:47:32.275753+0000","last_active":"2024-09-13T14:50:19.054001+0000","last_peered":"2024-09-13T14:50:19.054001+0000","last_clean":"2024-09-13T14:50:19.054001+0000","last_became_active":"2024-09-13T14:47:29.401066+0000","last_became_peered":"2024-09-13T14:47:29.401066+0000","last_unstale":"2024-09-13T14:50:19.054001+0000","last_undegraded":"2024-09-13T14:50:19.054001+0000","last_fullsized":"2024-09-13T14:50:19.054001+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:32.275562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:32.275562+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:43:41.056775+0000","scrub_duration":0.0054674260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.8","version":"368'15855","reported_seq":15927,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T14:50:21.375869+0000","last_change":"2024-09-13T14:45:59.743468+0000","last_active":"2024-09-13T14:50:21.375869+0000","last_peered":"2024-09-13T14:50:21.375869+0000","last_clean":"2024-09-13T14:50:21.375869+0000","last_became_active":"2024-09-13T14:45:59.742168+0000","last_became_peered":"2024-09-13T14:45:59.742168+0000","last_unstale":"2024-09-13T14:50:21.375869+0000","last_undegraded":"2024-09-13T14:50:21.375869+0000","last_fullsized":"2024-09-13T14:50:21.375869+0000","mapping_epoch":304,"log_start":"335'7962","ondisk_log_start":"335'7962","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":7893,"log_dups_size":0,"ondisk_log_size":7893,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:38:09.406586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29687808,"num_objects":453,"num_object_clones":0,"num_object_copies":1359,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15855,"num_write_kb":521856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":1121,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.062509+0000","last_change":"2024-09-13T14:47:50.892842+0000","last_active":"2024-09-13T14:50:19.062509+0000","last_peered":"2024-09-13T14:50:19.062509+0000","last_clean":"2024-09-13T14:50:19.062509+0000","last_became_active":"2024-09-13T14:47:50.892237+0000","last_became_peered":"2024-09-13T14:47:50.892237+0000","last_unstale":"2024-09-13T14:50:19.062509+0000","last_undegraded":"2024-09-13T14:50:19.062509+0000","last_fullsized":"2024-09-13T14:50:19.062509+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:20:05.796762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":8388646,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":377,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.050627+0000","last_change":"2024-09-13T14:38:03.528133+0000","last_active":"2024-09-13T14:50:19.050627+0000","last_peered":"2024-09-13T14:50:19.050627+0000","last_clean":"2024-09-13T14:50:19.050627+0000","last_became_active":"2024-09-13T14:38:03.527829+0000","last_became_peered":"2024-09-13T14:38:03.527829+0000","last_unstale":"2024-09-13T14:50:19.050627+0000","last_undegraded":"2024-09-13T14:50:19.050627+0000","last_fullsized":"2024-09-13T14:50:19.050627+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:48.254169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":73,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.056837+0000","last_change":"2024-09-13T14:47:47.877272+0000","last_active":"2024-09-13T14:50:19.056837+0000","last_peered":"2024-09-13T14:50:19.056837+0000","last_clean":"2024-09-13T14:50:19.056837+0000","last_became_active":"2024-09-13T14:47:47.877003+0000","last_became_peered":"2024-09-13T14:47:47.877003+0000","last_unstale":"2024-09-13T14:50:19.056837+0000","last_undegraded":"2024-09-13T14:50:19.056837+0000","last_fullsized":"2024-09-13T14:50:19.056837+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:47:23.172452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"90'149","reported_seq":507,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T14:50:20.253768+0000","last_change":"2024-09-13T14:47:33.067298+0000","last_active":"2024-09-13T14:50:20.253768+0000","last_peered":"2024-09-13T14:50:20.253768+0000","last_clean":"2024-09-13T14:50:20.253768+0000","last_became_active":"2024-09-13T14:47:29.419416+0000","last_became_peered":"2024-09-13T14:47:29.419416+0000","last_unstale":"2024-09-13T14:50:20.253768+0000","last_undegraded":"2024-09-13T14:50:20.253768+0000","last_fullsized":"2024-09-13T14:50:20.253768+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:33.066954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:33.066954+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:05:42.808060+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"368'15815","reported_seq":15961,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T14:50:22.996380+0000","last_change":"2024-09-13T14:46:01.768936+0000","last_active":"2024-09-13T14:50:22.996380+0000","last_peered":"2024-09-13T14:50:22.996380+0000","last_clean":"2024-09-13T14:50:22.996380+0000","last_became_active":"2024-09-13T14:46:01.768354+0000","last_became_peered":"2024-09-13T14:46:01.768354+0000","last_unstale":"2024-09-13T14:50:22.996380+0000","last_undegraded":"2024-09-13T14:50:22.996380+0000","last_fullsized":"2024-09-13T14:50:22.996380+0000","mapping_epoch":306,"log_start":"335'7862","ondisk_log_start":"335'7862","created":304,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":7953,"log_dups_size":0,"ondisk_log_size":7953,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:01.762702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22609920,"num_objects":345,"num_object_clones":0,"num_object_copies":1035,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15815,"num_write_kb":517120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":190,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.053928+0000","last_change":"2024-09-13T14:46:01.763906+0000","last_active":"2024-09-13T14:50:19.053928+0000","last_peered":"2024-09-13T14:50:19.053928+0000","last_clean":"2024-09-13T14:50:19.053928+0000","last_became_active":"2024-09-13T14:46:01.763542+0000","last_became_peered":"2024-09-13T14:46:01.763542+0000","last_unstale":"2024-09-13T14:50:19.053928+0000","last_undegraded":"2024-09-13T14:50:19.053928+0000","last_fullsized":"2024-09-13T14:50:19.053928+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:09:36.652767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":1361,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.056925+0000","last_change":"2024-09-13T14:47:50.883365+0000","last_active":"2024-09-13T14:50:19.056925+0000","last_peered":"2024-09-13T14:50:19.056925+0000","last_clean":"2024-09-13T14:50:19.056925+0000","last_became_active":"2024-09-13T14:47:50.882609+0000","last_became_peered":"2024-09-13T14:47:50.882609+0000","last_unstale":"2024-09-13T14:50:19.056925+0000","last_undegraded":"2024-09-13T14:50:19.056925+0000","last_fullsized":"2024-09-13T14:50:19.056925+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:12:17.961575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":73,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.062015+0000","last_change":"2024-09-13T14:47:47.866979+0000","last_active":"2024-09-13T14:50:19.062015+0000","last_peered":"2024-09-13T14:50:19.062015+0000","last_clean":"2024-09-13T14:50:19.062015+0000","last_became_active":"2024-09-13T14:47:47.866634+0000","last_became_peered":"2024-09-13T14:47:47.866634+0000","last_unstale":"2024-09-13T14:50:19.062015+0000","last_undegraded":"2024-09-13T14:50:19.062015+0000","last_fullsized":"2024-09-13T14:50:19.062015+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:47:09.696139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"90'149","reported_seq":513,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T14:50:20.250750+0000","last_change":"2024-09-13T14:47:29.556225+0000","last_active":"2024-09-13T14:50:20.250750+0000","last_peered":"2024-09-13T14:50:20.250750+0000","last_clean":"2024-09-13T14:50:20.250750+0000","last_became_active":"2024-09-13T14:47:29.408774+0000","last_became_peered":"2024-09-13T14:47:29.408774+0000","last_unstale":"2024-09-13T14:50:20.250750+0000","last_undegraded":"2024-09-13T14:50:20.250750+0000","last_fullsized":"2024-09-13T14:50:20.250750+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:29.556149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:29.556149+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:35.691178+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"368'15977","reported_seq":18315,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T14:50:23.025292+0000","last_change":"2024-09-13T14:50:14.673431+0000","last_active":"2024-09-13T14:50:23.025292+0000","last_peered":"2024-09-13T14:50:23.025292+0000","last_clean":"2024-09-13T14:50:23.025292+0000","last_became_active":"2024-09-13T14:50:14.568416+0000","last_became_peered":"2024-09-13T14:50:14.568416+0000","last_unstale":"2024-09-13T14:50:23.025292+0000","last_undegraded":"2024-09-13T14:50:23.025292+0000","last_fullsized":"2024-09-13T14:50:23.025292+0000","mapping_epoch":361,"log_start":"341'9141","ondisk_log_start":"341'9141","created":304,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":6836,"log_dups_size":0,"ondisk_log_size":6836,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:04.736862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25231360,"num_objects":385,"num_object_clones":0,"num_object_copies":1155,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15977,"num_write_kb":523584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1085,"num_bytes_recovered":70975488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":73,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.057004+0000","last_change":"2024-09-13T14:47:47.876562+0000","last_active":"2024-09-13T14:50:19.057004+0000","last_peered":"2024-09-13T14:50:19.057004+0000","last_clean":"2024-09-13T14:50:19.057004+0000","last_became_active":"2024-09-13T14:47:47.876297+0000","last_became_peered":"2024-09-13T14:47:47.876297+0000","last_unstale":"2024-09-13T14:50:19.057004+0000","last_undegraded":"2024-09-13T14:50:19.057004+0000","last_fullsized":"2024-09-13T14:50:19.057004+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:03:27.761394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":893,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.061027+0000","last_change":"2024-09-13T14:38:04.596192+0000","last_active":"2024-09-13T14:50:19.061027+0000","last_peered":"2024-09-13T14:50:19.061027+0000","last_clean":"2024-09-13T14:50:19.061027+0000","last_became_active":"2024-09-13T14:38:04.593299+0000","last_became_peered":"2024-09-13T14:38:04.593299+0000","last_unstale":"2024-09-13T14:50:19.061027+0000","last_undegraded":"2024-09-13T14:50:19.061027+0000","last_fullsized":"2024-09-13T14:50:19.061027+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:34.241410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":263,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T14:50:20.250660+0000","last_change":"2024-09-13T14:38:03.533120+0000","last_active":"2024-09-13T14:50:20.250660+0000","last_peered":"2024-09-13T14:50:20.250660+0000","last_clean":"2024-09-13T14:50:20.250660+0000","last_became_active":"2024-09-13T14:38:03.532829+0000","last_became_peered":"2024-09-13T14:38:03.532829+0000","last_unstale":"2024-09-13T14:50:20.250660+0000","last_undegraded":"2024-09-13T14:50:20.250660+0000","last_fullsized":"2024-09-13T14:50:20.250660+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:24.514661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"90'149","reported_seq":466,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.050517+0000","last_change":"2024-09-13T14:47:30.528588+0000","last_active":"2024-09-13T14:50:19.050517+0000","last_peered":"2024-09-13T14:50:19.050517+0000","last_clean":"2024-09-13T14:50:19.050517+0000","last_became_active":"2024-09-13T14:47:29.413621+0000","last_became_peered":"2024-09-13T14:47:29.413621+0000","last_unstale":"2024-09-13T14:50:19.050517+0000","last_undegraded":"2024-09-13T14:50:19.050517+0000","last_fullsized":"2024-09-13T14:50:19.050517+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:30.528112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:30.528112+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:30.462009+0000","scrub_duration":0.012751485,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.c","version":"368'15904","reported_seq":16044,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T14:50:22.996181+0000","last_change":"2024-09-13T14:45:59.743204+0000","last_active":"2024-09-13T14:50:22.996181+0000","last_peered":"2024-09-13T14:50:22.996181+0000","last_clean":"2024-09-13T14:50:22.996181+0000","last_became_active":"2024-09-13T14:45:59.742910+0000","last_became_peered":"2024-09-13T14:45:59.742910+0000","last_unstale":"2024-09-13T14:50:22.996181+0000","last_undegraded":"2024-09-13T14:50:22.996181+0000","last_fullsized":"2024-09-13T14:50:22.996181+0000","mapping_epoch":304,"log_start":"335'7984","ondisk_log_start":"335'7984","created":304,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":7920,"log_dups_size":0,"ondisk_log_size":7920,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:51.017907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24641536,"num_objects":376,"num_object_clones":0,"num_object_copies":1128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15904,"num_write_kb":520960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":665,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T14:50:20.254188+0000","last_change":"2024-09-13T14:38:53.938227+0000","last_active":"2024-09-13T14:50:20.254188+0000","last_peered":"2024-09-13T14:50:20.254188+0000","last_clean":"2024-09-13T14:50:20.254188+0000","last_became_active":"2024-09-13T14:38:53.902867+0000","last_became_peered":"2024-09-13T14:38:53.902867+0000","last_unstale":"2024-09-13T14:50:20.254188+0000","last_undegraded":"2024-09-13T14:50:20.254188+0000","last_fullsized":"2024-09-13T14:50:20.254188+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:16.831756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":543,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.056100+0000","last_change":"2024-09-13T14:47:50.884706+0000","last_active":"2024-09-13T14:50:19.056100+0000","last_peered":"2024-09-13T14:50:19.056100+0000","last_clean":"2024-09-13T14:50:19.056100+0000","last_became_active":"2024-09-13T14:47:50.879750+0000","last_became_peered":"2024-09-13T14:47:50.879750+0000","last_unstale":"2024-09-13T14:50:19.056100+0000","last_undegraded":"2024-09-13T14:50:19.056100+0000","last_fullsized":"2024-09-13T14:50:19.056100+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:34:32.941266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":647,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T14:50:20.252409+0000","last_change":"2024-09-13T14:38:53.900107+0000","last_active":"2024-09-13T14:50:20.252409+0000","last_peered":"2024-09-13T14:50:20.252409+0000","last_clean":"2024-09-13T14:50:20.252409+0000","last_became_active":"2024-09-13T14:38:53.899556+0000","last_became_peered":"2024-09-13T14:38:53.899556+0000","last_unstale":"2024-09-13T14:50:20.252409+0000","last_undegraded":"2024-09-13T14:50:20.252409+0000","last_fullsized":"2024-09-13T14:50:20.252409+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:15:47.918163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"90'149","reported_seq":504,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T14:50:20.254287+0000","last_change":"2024-09-13T14:47:29.999643+0000","last_active":"2024-09-13T14:50:20.254287+0000","last_peered":"2024-09-13T14:50:20.254287+0000","last_clean":"2024-09-13T14:50:20.254287+0000","last_became_active":"2024-09-13T14:47:29.410951+0000","last_became_peered":"2024-09-13T14:47:29.410951+0000","last_unstale":"2024-09-13T14:50:20.254287+0000","last_undegraded":"2024-09-13T14:50:20.254287+0000","last_fullsized":"2024-09-13T14:50:20.254287+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:29.999519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:29.999519+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:04:54.887884+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"368'16008","reported_seq":16092,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T14:50:21.452470+0000","last_change":"2024-09-13T14:46:01.764360+0000","last_active":"2024-09-13T14:50:21.452470+0000","last_peered":"2024-09-13T14:50:21.452470+0000","last_clean":"2024-09-13T14:50:21.452470+0000","last_became_active":"2024-09-13T14:46:01.764029+0000","last_became_peered":"2024-09-13T14:46:01.764029+0000","last_unstale":"2024-09-13T14:50:21.452470+0000","last_undegraded":"2024-09-13T14:50:21.452470+0000","last_fullsized":"2024-09-13T14:50:21.452470+0000","mapping_epoch":306,"log_start":"335'8045","ondisk_log_start":"335'8045","created":304,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:45:58.722152+0000","last_clean_scrub_stamp":"2024-09-13T14:45:58.722152+0000","objects_scrubbed":0,"log_size":7963,"log_dups_size":0,"ondisk_log_size":7963,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:02:23.092494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30343192,"num_objects":464,"num_object_clones":0,"num_object_copies":1392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":464,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16008,"num_write_kb":527041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":289,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T14:50:20.251263+0000","last_change":"2024-09-13T14:47:47.883475+0000","last_active":"2024-09-13T14:50:20.251263+0000","last_peered":"2024-09-13T14:50:20.251263+0000","last_clean":"2024-09-13T14:50:20.251263+0000","last_became_active":"2024-09-13T14:47:47.871525+0000","last_became_peered":"2024-09-13T14:47:47.871525+0000","last_unstale":"2024-09-13T14:50:20.251263+0000","last_undegraded":"2024-09-13T14:50:20.251263+0000","last_fullsized":"2024-09-13T14:50:20.251263+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:33.903432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":171,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.059927+0000","last_change":"2024-09-13T14:46:01.781508+0000","last_active":"2024-09-13T14:50:19.059927+0000","last_peered":"2024-09-13T14:50:19.059927+0000","last_clean":"2024-09-13T14:50:19.059927+0000","last_became_active":"2024-09-13T14:46:01.767096+0000","last_became_peered":"2024-09-13T14:46:01.767096+0000","last_unstale":"2024-09-13T14:50:19.059927+0000","last_undegraded":"2024-09-13T14:50:19.059927+0000","last_fullsized":"2024-09-13T14:50:19.059927+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:20:01.119189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":361,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.049838+0000","last_change":"2024-09-13T14:37:49.172218+0000","last_active":"2024-09-13T14:50:19.049838+0000","last_peered":"2024-09-13T14:50:19.049838+0000","last_clean":"2024-09-13T14:50:19.049838+0000","last_became_active":"2024-09-13T14:37:49.171718+0000","last_became_peered":"2024-09-13T14:37:49.171718+0000","last_unstale":"2024-09-13T14:50:19.049838+0000","last_undegraded":"2024-09-13T14:50:19.049838+0000","last_fullsized":"2024-09-13T14:50:19.049838+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:46:54.626521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":392,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.050368+0000","last_change":"2024-09-13T14:47:47.868409+0000","last_active":"2024-09-13T14:50:19.050368+0000","last_peered":"2024-09-13T14:50:19.050368+0000","last_clean":"2024-09-13T14:50:19.050368+0000","last_became_active":"2024-09-13T14:47:47.867712+0000","last_became_peered":"2024-09-13T14:47:47.867712+0000","last_unstale":"2024-09-13T14:50:19.050368+0000","last_undegraded":"2024-09-13T14:50:19.050368+0000","last_fullsized":"2024-09-13T14:50:19.050368+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:57:14.647493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":363,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.051053+0000","last_change":"2024-09-13T14:37:48.128447+0000","last_active":"2024-09-13T14:50:19.051053+0000","last_peered":"2024-09-13T14:50:19.051053+0000","last_clean":"2024-09-13T14:50:19.051053+0000","last_became_active":"2024-09-13T14:37:48.128143+0000","last_became_peered":"2024-09-13T14:37:48.128143+0000","last_unstale":"2024-09-13T14:50:19.051053+0000","last_undegraded":"2024-09-13T14:50:19.051053+0000","last_fullsized":"2024-09-13T14:50:19.051053+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:22.204039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":73,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.060939+0000","last_change":"2024-09-13T14:47:47.867935+0000","last_active":"2024-09-13T14:50:19.060939+0000","last_peered":"2024-09-13T14:50:19.060939+0000","last_clean":"2024-09-13T14:50:19.060939+0000","last_became_active":"2024-09-13T14:47:47.867616+0000","last_became_peered":"2024-09-13T14:47:47.867616+0000","last_unstale":"2024-09-13T14:50:19.060939+0000","last_undegraded":"2024-09-13T14:50:19.060939+0000","last_fullsized":"2024-09-13T14:50:19.060939+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:52:38.619916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":135,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.060750+0000","last_change":"2024-09-13T14:38:53.954281+0000","last_active":"2024-09-13T14:50:19.060750+0000","last_peered":"2024-09-13T14:50:19.060750+0000","last_clean":"2024-09-13T14:50:19.060750+0000","last_became_active":"2024-09-13T14:38:53.900912+0000","last_became_peered":"2024-09-13T14:38:53.900912+0000","last_unstale":"2024-09-13T14:50:19.060750+0000","last_undegraded":"2024-09-13T14:50:19.060750+0000","last_fullsized":"2024-09-13T14:50:19.060750+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:47.876275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"90'149","reported_seq":508,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T14:50:20.053037+0000","last_change":"2024-09-13T14:47:38.468095+0000","last_active":"2024-09-13T14:50:20.053037+0000","last_peered":"2024-09-13T14:50:20.053037+0000","last_clean":"2024-09-13T14:50:20.053037+0000","last_became_active":"2024-09-13T14:47:38.467526+0000","last_became_peered":"2024-09-13T14:47:38.467526+0000","last_unstale":"2024-09-13T14:50:20.053037+0000","last_undegraded":"2024-09-13T14:50:20.053037+0000","last_fullsized":"2024-09-13T14:50:20.053037+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:31.013128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:31.013128+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:17:20.508142+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":196,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.053767+0000","last_change":"2024-09-13T14:47:47.861842+0000","last_active":"2024-09-13T14:50:19.053767+0000","last_peered":"2024-09-13T14:50:19.053767+0000","last_clean":"2024-09-13T14:50:19.053767+0000","last_became_active":"2024-09-13T14:47:47.861492+0000","last_became_peered":"2024-09-13T14:47:47.861492+0000","last_unstale":"2024-09-13T14:50:19.053767+0000","last_undegraded":"2024-09-13T14:50:19.053767+0000","last_fullsized":"2024-09-13T14:50:19.053767+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:17.279533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":588,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T14:50:20.252356+0000","last_change":"2024-09-13T14:38:53.899535+0000","last_active":"2024-09-13T14:50:20.252356+0000","last_peered":"2024-09-13T14:50:20.252356+0000","last_clean":"2024-09-13T14:50:20.252356+0000","last_became_active":"2024-09-13T14:38:53.899183+0000","last_became_peered":"2024-09-13T14:38:53.899183+0000","last_unstale":"2024-09-13T14:50:20.252356+0000","last_undegraded":"2024-09-13T14:50:20.252356+0000","last_fullsized":"2024-09-13T14:50:20.252356+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:33.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":262,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T14:50:20.250605+0000","last_change":"2024-09-13T14:38:03.543426+0000","last_active":"2024-09-13T14:50:20.250605+0000","last_peered":"2024-09-13T14:50:20.250605+0000","last_clean":"2024-09-13T14:50:20.250605+0000","last_became_active":"2024-09-13T14:38:03.542661+0000","last_became_peered":"2024-09-13T14:38:03.542661+0000","last_unstale":"2024-09-13T14:50:20.250605+0000","last_undegraded":"2024-09-13T14:50:20.250605+0000","last_fullsized":"2024-09-13T14:50:20.250605+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:54.050127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":624,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T14:50:20.252321+0000","last_change":"2024-09-13T14:47:47.868624+0000","last_active":"2024-09-13T14:50:20.252321+0000","last_peered":"2024-09-13T14:50:20.252321+0000","last_clean":"2024-09-13T14:50:20.252321+0000","last_became_active":"2024-09-13T14:47:47.868031+0000","last_became_peered":"2024-09-13T14:47:47.868031+0000","last_unstale":"2024-09-13T14:50:20.252321+0000","last_undegraded":"2024-09-13T14:50:20.252321+0000","last_fullsized":"2024-09-13T14:50:20.252321+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:05:25.365023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":209,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.053732+0000","last_change":"2024-09-13T14:47:47.859825+0000","last_active":"2024-09-13T14:50:19.053732+0000","last_peered":"2024-09-13T14:50:19.053732+0000","last_clean":"2024-09-13T14:50:19.053732+0000","last_became_active":"2024-09-13T14:47:47.859069+0000","last_became_peered":"2024-09-13T14:47:47.859069+0000","last_unstale":"2024-09-13T14:50:19.053732+0000","last_undegraded":"2024-09-13T14:50:19.053732+0000","last_fullsized":"2024-09-13T14:50:19.053732+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:18:36.452321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":656,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T14:50:20.252414+0000","last_change":"2024-09-13T14:47:47.868945+0000","last_active":"2024-09-13T14:50:20.252414+0000","last_peered":"2024-09-13T14:50:20.252414+0000","last_clean":"2024-09-13T14:50:20.252414+0000","last_became_active":"2024-09-13T14:47:47.868296+0000","last_became_peered":"2024-09-13T14:47:47.868296+0000","last_unstale":"2024-09-13T14:50:20.252414+0000","last_undegraded":"2024-09-13T14:50:20.252414+0000","last_fullsized":"2024-09-13T14:50:20.252414+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:22:24.778693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":271,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T14:50:20.251094+0000","last_change":"2024-09-13T14:38:53.892948+0000","last_active":"2024-09-13T14:50:20.251094+0000","last_peered":"2024-09-13T14:50:20.251094+0000","last_clean":"2024-09-13T14:50:20.251094+0000","last_became_active":"2024-09-13T14:38:53.892665+0000","last_became_peered":"2024-09-13T14:38:53.892665+0000","last_unstale":"2024-09-13T14:50:20.251094+0000","last_undegraded":"2024-09-13T14:50:20.251094+0000","last_fullsized":"2024-09-13T14:50:20.251094+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:08.292359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":73,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.055996+0000","last_change":"2024-09-13T14:47:47.871164+0000","last_active":"2024-09-13T14:50:19.055996+0000","last_peered":"2024-09-13T14:50:19.055996+0000","last_clean":"2024-09-13T14:50:19.055996+0000","last_became_active":"2024-09-13T14:47:47.870862+0000","last_became_peered":"2024-09-13T14:47:47.870862+0000","last_unstale":"2024-09-13T14:50:19.055996+0000","last_undegraded":"2024-09-13T14:50:19.055996+0000","last_fullsized":"2024-09-13T14:50:19.055996+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:41:34.392081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":638,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T14:50:20.253299+0000","last_change":"2024-09-13T14:47:47.860840+0000","last_active":"2024-09-13T14:50:20.253299+0000","last_peered":"2024-09-13T14:50:20.253299+0000","last_clean":"2024-09-13T14:50:20.253299+0000","last_became_active":"2024-09-13T14:47:47.860473+0000","last_became_peered":"2024-09-13T14:47:47.860473+0000","last_unstale":"2024-09-13T14:50:20.253299+0000","last_undegraded":"2024-09-13T14:50:20.253299+0000","last_fullsized":"2024-09-13T14:50:20.253299+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:31:39.442918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":445,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.050120+0000","last_change":"2024-09-13T14:47:47.868246+0000","last_active":"2024-09-13T14:50:19.050120+0000","last_peered":"2024-09-13T14:50:19.050120+0000","last_clean":"2024-09-13T14:50:19.050120+0000","last_became_active":"2024-09-13T14:47:47.867817+0000","last_became_peered":"2024-09-13T14:47:47.867817+0000","last_unstale":"2024-09-13T14:50:19.050120+0000","last_undegraded":"2024-09-13T14:50:19.050120+0000","last_fullsized":"2024-09-13T14:50:19.050120+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:07.612451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":149,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T14:50:20.053254+0000","last_change":"2024-09-13T14:47:47.876275+0000","last_active":"2024-09-13T14:50:20.053254+0000","last_peered":"2024-09-13T14:50:20.053254+0000","last_clean":"2024-09-13T14:50:20.053254+0000","last_became_active":"2024-09-13T14:47:47.875467+0000","last_became_peered":"2024-09-13T14:47:47.875467+0000","last_unstale":"2024-09-13T14:50:20.053254+0000","last_undegraded":"2024-09-13T14:50:20.053254+0000","last_fullsized":"2024-09-13T14:50:20.053254+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:12:36.717423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":262,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T14:50:20.250441+0000","last_change":"2024-09-13T14:38:03.527021+0000","last_active":"2024-09-13T14:50:20.250441+0000","last_peered":"2024-09-13T14:50:20.250441+0000","last_clean":"2024-09-13T14:50:20.250441+0000","last_became_active":"2024-09-13T14:38:03.526698+0000","last_became_peered":"2024-09-13T14:38:03.526698+0000","last_unstale":"2024-09-13T14:50:20.250441+0000","last_undegraded":"2024-09-13T14:50:20.250441+0000","last_fullsized":"2024-09-13T14:50:20.250441+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:01:35.734086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":389,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.050297+0000","last_change":"2024-09-13T14:47:47.866784+0000","last_active":"2024-09-13T14:50:19.050297+0000","last_peered":"2024-09-13T14:50:19.050297+0000","last_clean":"2024-09-13T14:50:19.050297+0000","last_became_active":"2024-09-13T14:47:47.866483+0000","last_became_peered":"2024-09-13T14:47:47.866483+0000","last_unstale":"2024-09-13T14:50:19.050297+0000","last_undegraded":"2024-09-13T14:50:19.050297+0000","last_fullsized":"2024-09-13T14:50:19.050297+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:15:53.119741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":197,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.053771+0000","last_change":"2024-09-13T14:47:47.880946+0000","last_active":"2024-09-13T14:50:19.053771+0000","last_peered":"2024-09-13T14:50:19.053771+0000","last_clean":"2024-09-13T14:50:19.053771+0000","last_became_active":"2024-09-13T14:47:47.866534+0000","last_became_peered":"2024-09-13T14:47:47.866534+0000","last_unstale":"2024-09-13T14:50:19.053771+0000","last_undegraded":"2024-09-13T14:50:19.053771+0000","last_fullsized":"2024-09-13T14:50:19.053771+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:01:01.384873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":382,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.049968+0000","last_change":"2024-09-13T14:38:03.526333+0000","last_active":"2024-09-13T14:50:19.049968+0000","last_peered":"2024-09-13T14:50:19.049968+0000","last_clean":"2024-09-13T14:50:19.049968+0000","last_became_active":"2024-09-13T14:38:03.526001+0000","last_became_peered":"2024-09-13T14:38:03.526001+0000","last_unstale":"2024-09-13T14:50:19.049968+0000","last_undegraded":"2024-09-13T14:50:19.049968+0000","last_fullsized":"2024-09-13T14:50:19.049968+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:25:25.181271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":184,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.054086+0000","last_change":"2024-09-13T14:37:49.149582+0000","last_active":"2024-09-13T14:50:19.054086+0000","last_peered":"2024-09-13T14:50:19.054086+0000","last_clean":"2024-09-13T14:50:19.054086+0000","last_became_active":"2024-09-13T14:37:49.148525+0000","last_became_peered":"2024-09-13T14:37:49.148525+0000","last_unstale":"2024-09-13T14:50:19.054086+0000","last_undegraded":"2024-09-13T14:50:19.054086+0000","last_fullsized":"2024-09-13T14:50:19.054086+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:48.655665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":670,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T14:50:20.252756+0000","last_change":"2024-09-13T14:46:01.766580+0000","last_active":"2024-09-13T14:50:20.252756+0000","last_peered":"2024-09-13T14:50:20.252756+0000","last_clean":"2024-09-13T14:50:20.252756+0000","last_became_active":"2024-09-13T14:46:01.766257+0000","last_became_peered":"2024-09-13T14:46:01.766257+0000","last_unstale":"2024-09-13T14:50:20.252756+0000","last_undegraded":"2024-09-13T14:50:20.252756+0000","last_fullsized":"2024-09-13T14:50:20.252756+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:01.126109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":634,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T14:50:20.253575+0000","last_change":"2024-09-13T14:46:01.769647+0000","last_active":"2024-09-13T14:50:20.253575+0000","last_peered":"2024-09-13T14:50:20.253575+0000","last_clean":"2024-09-13T14:50:20.253575+0000","last_became_active":"2024-09-13T14:46:01.769228+0000","last_became_peered":"2024-09-13T14:46:01.769228+0000","last_unstale":"2024-09-13T14:50:20.253575+0000","last_undegraded":"2024-09-13T14:50:20.253575+0000","last_fullsized":"2024-09-13T14:50:20.253575+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:58.696024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":372,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-13T14:50:19.049989+0000","last_change":"2024-09-13T14:47:47.860727+0000","last_active":"2024-09-13T14:50:19.049989+0000","last_peered":"2024-09-13T14:50:19.049989+0000","last_clean":"2024-09-13T14:50:19.049989+0000","last_became_active":"2024-09-13T14:47:47.860377+0000","last_became_peered":"2024-09-13T14:47:47.860377+0000","last_unstale":"2024-09-13T14:50:19.049989+0000","last_undegraded":"2024-09-13T14:50:19.049989+0000","last_fullsized":"2024-09-13T14:50:19.049989+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:27:00.868699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":282,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-13T14:50:20.250566+0000","last_change":"2024-09-13T14:46:01.775237+0000","last_active":"2024-09-13T14:50:20.250566+0000","last_peered":"2024-09-13T14:50:20.250566+0000","last_clean":"2024-09-13T14:50:20.250566+0000","last_became_active":"2024-09-13T14:46:01.765025+0000","last_became_peered":"2024-09-13T14:46:01.765025+0000","last_unstale":"2024-09-13T14:50:20.250566+0000","last_undegraded":"2024-09-13T14:50:20.250566+0000","last_fullsized":"2024-09-13T14:50:20.250566+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:29:16.089042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1746,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":473694232,"num_objects":7229,"num_object_clones":0,"num_object_copies":21687,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7229,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":253941,"num_write_kb":8357377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13957,"num_bytes_recovered":914489344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1513566208,"data_stored":1513553992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117661,"ondisk_log_size":117661,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":2154,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25132,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":151,"num_bytes_recovered":8388646,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3129,"ondisk_log_size":3129,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234478,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":784300,"kb_used_data":161696,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":92960852,"statfs":{"total":95995035648,"available":95191912448,"internally_reserved":0,"allocated":165576704,"data_stored":163655112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":637532619},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":239,"seq":1026497183896,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":669276,"kb_used_data":179476,"kb_used_omap":8,"kb_used_meta":489719,"kb_avail":93075876,"statfs":{"total":95995035648,"available":95309697024,"internally_reserved":0,"allocated":183783424,"data_stored":181874027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8453,"internal_metadata":501473019},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757743,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1162588,"kb_used_data":256980,"kb_used_omap":1,"kb_used_meta":905598,"kb_avail":92582564,"statfs":{"total":95995035648,"available":94804545536,"internally_reserved":0,"allocated":263147520,"data_stored":261234459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":927332806},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019387,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1003236,"kb_used_data":268248,"kb_used_omap":1,"kb_used_meta":734974,"kb_avail":92741916,"statfs":{"total":95995035648,"available":94967721984,"internally_reserved":0,"allocated":274685952,"data_stored":272772516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":752613830},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435570,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":906772,"kb_used_data":149004,"kb_used_omap":5,"kb_used_meta":757754,"kb_avail":92838380,"statfs":{"total":95995035648,"available":95066501120,"internally_reserved":0,"allocated":152580096,"data_stored":150674965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":775940222},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":104,"seq":446676599098,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":468064,"kb_used_data":183700,"kb_used_omap":7,"kb_used_meta":284344,"kb_avail":93277088,"statfs":{"total":95995035648,"available":95515738112,"internally_reserved":0,"allocated":188108800,"data_stored":186196027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8010,"internal_metadata":291168438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":99,"seq":425201762624,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":388996,"kb_used_data":253176,"kb_used_omap":7,"kb_used_meta":135800,"kb_avail":93356156,"statfs":{"total":95995035648,"available":95596703744,"internally_reserved":0,"allocated":259252224,"data_stored":257358446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7759,"internal_metadata":139059633},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926148,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1017428,"kb_used_data":165196,"kb_used_omap":7,"kb_used_meta":852152,"kb_avail":92727724,"statfs":{"total":95995035648,"available":94953189376,"internally_reserved":0,"allocated":169160704,"data_stored":167255543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":872603944},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6860,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":148045824,"data_stored":148045824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":250347520,"data_stored":250347520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":166727680,"data_stored":166723608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":140836864,"data_stored":140836864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":252248064,"data_stored":252248064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":243142656,"data_stored":243138584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":154664960,"data_stored":154664960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":157552640,"data_stored":157548568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:50:26.922 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph status --format=json 2024-09-13T14:50:26.929 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:26.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:26.954 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:26.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:27.055 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:27.100 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:27.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:27.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:27.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:27 smithi102 ceph-mon[110183]: pgmap v1271: 109 pgs: 109 active+clean; 387 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 545 op/s; 10 MiB/s, 167 objects/s recovering 2024-09-13T14:50:27.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:27 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-09-13T14:50:27.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:27 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [3, 5, 1, 0]}]': finished 2024-09-13T14:50:27.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:27 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-09-13T14:50:27.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:27 smithi102 ceph-mon[110183]: osdmap e369: 8 total, 8 up, 8 in 2024-09-13T14:50:27.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:27 smithi102 ceph-mon[115269]: pgmap v1271: 109 pgs: 109 active+clean; 387 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 545 op/s; 10 MiB/s, 167 objects/s recovering 2024-09-13T14:50:27.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:27 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-09-13T14:50:27.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:27 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [3, 5, 1, 0]}]': finished 2024-09-13T14:50:27.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:27 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-09-13T14:50:27.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:27 smithi102 ceph-mon[115269]: osdmap e369: 8 total, 8 up, 8 in 2024-09-13T14:50:27.202 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:27.230 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:27.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:27.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:27.332 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:27.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:27.369 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:27.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:27.455 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:27 smithi195 ceph-mon[104262]: pgmap v1271: 109 pgs: 109 active+clean; 387 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 545 op/s; 10 MiB/s, 167 objects/s recovering 2024-09-13T14:50:27.455 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:27 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-09-13T14:50:27.455 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:27 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [3, 5, 1, 0]}]': finished 2024-09-13T14:50:27.455 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:27 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-09-13T14:50:27.455 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:27 smithi195 ceph-mon[104262]: osdmap e369: 8 total, 8 up, 8 in 2024-09-13T14:50:27.470 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:27.489 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T14:50:27.509 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:27.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:27.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:27.600 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:27.612 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:27.617 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:27.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:27.638 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:27.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:27.739 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:27.771 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:27.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:27.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:27.799 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:50:27.873 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:27.908 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:27.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:27.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:28.010 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:28.041 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:28.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:28.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:28.143 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:28.177 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:28.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:28.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:28.279 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:28.320 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:28.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:28.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:28.422 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:28.448 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:28.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:28.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:28.488 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is set 2024-09-13T14:50:28.550 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:28.583 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:28.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:28.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:28.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:28 smithi102 ceph-mon[110183]: osdmap e370: 8 total, 8 up, 8 in 2024-09-13T14:50:28.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:28 smithi102 ceph-mon[110183]: pgmap v1274: 109 pgs: 1 peering, 108 active+clean; 279 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 688 op/s 2024-09-13T14:50:28.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:28 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/4017978359' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:50:28.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:28 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:50:28.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:28 smithi102 ceph-mon[115269]: osdmap e370: 8 total, 8 up, 8 in 2024-09-13T14:50:28.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:28 smithi102 ceph-mon[115269]: pgmap v1274: 109 pgs: 1 peering, 108 active+clean; 279 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 688 op/s 2024-09-13T14:50:28.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:28 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/4017978359' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:50:28.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:28 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:50:28.685 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:28.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:28.719 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:28.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:28.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:28 smithi195 ceph-mon[104262]: osdmap e370: 8 total, 8 up, 8 in 2024-09-13T14:50:28.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:28 smithi195 ceph-mon[104262]: pgmap v1274: 109 pgs: 1 peering, 108 active+clean; 279 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 688 op/s 2024-09-13T14:50:28.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:28 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/4017978359' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:50:28.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:28 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:50:28.820 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:28.857 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:28.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:28.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:28.959 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:28.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:28.995 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:28.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:29.096 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:29.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:29.134 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:29.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:29.235 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:29.272 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:29.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:29.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:29.374 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:29.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:29.404 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:29.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:29.505 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:29.540 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:29.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:29.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:29.642 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:29.666 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Removed 130584 objects 2024-09-13T14:50:29.666 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Clean up completed and total clean up time :165.845 2024-09-13T14:50:29.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:29 smithi102 ceph-mon[110183]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:50:29.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:29 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:50:29.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:29 smithi102 ceph-mon[110183]: osdmap e371: 8 total, 8 up, 8 in 2024-09-13T14:50:29.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:29 smithi102 ceph-mon[115269]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:50:29.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:29 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:50:29.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:29 smithi102 ceph-mon[115269]: osdmap e371: 8 total, 8 up, 8 in 2024-09-13T14:50:29.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:29.673 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:29.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:29.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:29 smithi195 ceph-mon[104262]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:50:29.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:29 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:50:29.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:29 smithi195 ceph-mon[104262]: osdmap e371: 8 total, 8 up, 8 in 2024-09-13T14:50:29.774 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:29.784 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:50:29.785 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:50:29.797 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:29.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:29.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:29.877 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-09-13T14:50:29.878 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-09-13T14:50:29.899 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:29.920 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:29.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:29.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:30.021 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:30.043 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:30.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:30.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:30.145 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:30.168 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:30.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:30.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:30.270 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:30.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:30.293 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:30.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:30.395 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:30.416 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:30.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:30.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:30.485 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T14:50:30.517 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:30.549 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:30.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:30.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:30.651 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:30.673 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:30.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:30.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:30.686 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:50:30.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:30 smithi195 ceph-mon[104262]: pgmap v1276: 109 pgs: 1 peering, 108 active+clean; 108 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 928 op/s 2024-09-13T14:50:30.776 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:30.799 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:30.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:30.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:30.807 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:30 smithi102 ceph-mon[110183]: pgmap v1276: 109 pgs: 1 peering, 108 active+clean; 108 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 928 op/s 2024-09-13T14:50:30.807 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:30 smithi102 ceph-mon[115269]: pgmap v1276: 109 pgs: 1 peering, 108 active+clean; 108 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 928 op/s 2024-09-13T14:50:30.901 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:30.923 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:30.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:30.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:31.025 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:31.048 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:31.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:31.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:31.149 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:31.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:31.173 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:31.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:31.274 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:31.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:31.297 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:31.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:31.398 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:31.420 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:31.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:31.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:31.522 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:31.544 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:31.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:31.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:31.563 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:31.645 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:31.672 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:31.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:31.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:31.774 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:31.795 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:50:31.795 INFO:teuthology.orchestra.run.smithi102.stdout:{"fsid":"08cee502-71d9-11ef-bceb-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false}},"mutes":[]},"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1855,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":371,"num_osds":8,"num_up_osds":8,"osd_up_since":1726238267,"num_in_osds":8,"osd_in_since":1726238866,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":108},{"state_name":"peering","count":1}],"num_pgs":109,"num_pools":5,"num_objects":609,"data_bytes":42866661,"bytes_used":5241032704,"bytes_avail":762719252480,"bytes_total":767960285184,"inactive_pgs_ratio":0.0091743115335702896,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":922},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi102.njetxr","status":"up:active","gid":24521}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.102:8443/","prometheus":"http://172.21.15.102:9283/"}},"servicemap":{"epoch":30,"modified":"2024-09-13T14:48:41.626174+0000","services":{}},"progress_events":{}} 2024-09-13T14:50:31.796 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:31.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:31.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:31.898 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:31.919 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:31.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:31.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:32.020 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:32.047 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:32.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:32.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:32.148 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:32.171 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:32.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:32.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:32.193 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:50:32.194 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:50:32.201 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:50:32.201 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:50:32.218 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:50:32.218 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:50:32.234 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:50:32.235 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:50:32.251 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:50:32.251 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:50:32.267 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:50:32.267 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:50:32.272 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:32.284 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:50:32.284 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:50:32.296 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:32.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:32.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:32.301 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:50:32.301 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:50:32.398 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:32.420 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:32.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:32.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:32.522 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:32.543 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:32.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:32.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:32.595 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:50:32.645 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:32.667 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:32.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:32.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:32.769 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:32.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:32.791 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:32.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:32.892 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:32.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:32.916 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:32.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:32.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:32 smithi102 ceph-mon[110183]: pgmap v1277: 109 pgs: 1 peering, 108 active+clean; 41 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 922 op/s 2024-09-13T14:50:32.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:32 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/4003993882' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:50:32.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:32 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1728582818' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:50:32.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:32 smithi102 ceph-mon[115269]: pgmap v1277: 109 pgs: 1 peering, 108 active+clean; 41 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 922 op/s 2024-09-13T14:50:32.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:32 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/4003993882' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:50:32.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:32 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1728582818' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:50:32.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:32 smithi195 ceph-mon[104262]: pgmap v1277: 109 pgs: 1 peering, 108 active+clean; 41 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 922 op/s 2024-09-13T14:50:32.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:32 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/4003993882' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:50:32.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:32 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1728582818' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:50:33.017 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:33.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:33.040 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:33.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:33.141 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:33.164 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:33.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:33.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:33.267 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:33.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:33.290 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:33.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:33.391 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:33.416 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:33.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:33.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:33.517 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:33.540 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:33.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:33.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:33.641 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:50:33.641 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-09-13T14:50:33.641 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-09-13T14:50:33.641 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-13T14:50:33.641 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.641 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35379.0:148608 7.2 e272/263 7:44ad0776:::benchmark_data_smithi102_227395_object35793:head v 272'9111, mlcod=272'9109)", 2024-09-13T14:50:33.641 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:41:26.480081+0000", 2024-09-13T14:50:33.641 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 547.15863002000003, 2024-09-13T14:50:33.641 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.070670319999999995, 2024-09-13T14:50:33.641 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:33.641 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:33.641 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:33.642 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.642 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:33.642 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.480081+0000", 2024-09-13T14:50:33.642 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:33.642 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.642 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.642 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:33.642 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.480081+0000", 2024-09-13T14:50:33.642 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:33.642 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.642 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.642 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:33.642 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.480080+0000", 2024-09-13T14:50:33.644 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:50:33.644 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.644 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.644 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:33.644 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.480090+0000", 2024-09-13T14:50:33.644 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.7799999999999995e-06 2024-09-13T14:50:33.644 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.644 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.644 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:33.644 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.480106+0000", 2024-09-13T14:50:33.645 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5481999999999998e-05 2024-09-13T14:50:33.645 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.645 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.645 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:33.645 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.480113+0000", 2024-09-13T14:50:33.645 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9820000000000008e-06 2024-09-13T14:50:33.645 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.645 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.645 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:33.645 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.480156+0000", 2024-09-13T14:50:33.645 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.295e-05 2024-09-13T14:50:33.645 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.645 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.646 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:33.646 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.480177+0000", 2024-09-13T14:50:33.646 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0839000000000002e-05 2024-09-13T14:50:33.646 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.646 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.646 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:33.646 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.550706+0000", 2024-09-13T14:50:33.646 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.070529013000000002 2024-09-13T14:50:33.646 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.646 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.646 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:33.646 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.550752+0000", 2024-09-13T14:50:33.646 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5435999999999997e-05 2024-09-13T14:50:33.646 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:33.647 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:33.647 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:33.647 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.647 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.647 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDPGPush(7.b 272/240 [PushOp(7:d58d7887:::benchmark_data_smithi102_227395_object72243:head, version: 213'4482, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:d58d7887:::benchmark_data_smithi102_227395_object72243:head@213'4482, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T14:50:33.647 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:41:26.483641+0000", 2024-09-13T14:50:33.647 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 547.15507026299997, 2024-09-13T14:50:33.647 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.066958958999999998, 2024-09-13T14:50:33.647 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:33.647 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T14:50:33.647 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:33.647 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.647 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:33.647 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.483641+0000", 2024-09-13T14:50:33.648 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:33.648 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.648 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.648 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:33.648 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.483641+0000", 2024-09-13T14:50:33.648 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:33.648 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.648 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.648 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:33.649 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.483640+0000", 2024-09-13T14:50:33.649 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:50:33.649 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.649 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.649 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:33.649 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.483684+0000", 2024-09-13T14:50:33.649 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3866999999999999e-05 2024-09-13T14:50:33.649 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.649 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.649 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:33.650 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.548048+0000", 2024-09-13T14:50:33.650 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.064363715000000002 2024-09-13T14:50:33.650 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.650 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.650 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:33.650 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.548073+0000", 2024-09-13T14:50:33.650 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5522999999999999e-05 2024-09-13T14:50:33.650 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.650 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.650 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:33.650 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.548173+0000", 2024-09-13T14:50:33.650 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.9853999999999996e-05 2024-09-13T14:50:33.650 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.650 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.651 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:33.651 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.548187+0000", 2024-09-13T14:50:33.651 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4132e-05 2024-09-13T14:50:33.651 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.651 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.651 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:33.651 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.550600+0000", 2024-09-13T14:50:33.651 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0024130250000000001 2024-09-13T14:50:33.651 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:33.651 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:33.651 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:33.651 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.651 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.651 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35379.0:148609 7.c e272/251 7:313103c6:::benchmark_data_smithi102_227395_object35794:head v 272'9246, mlcod=272'9244)", 2024-09-13T14:50:33.652 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:41:26.488089+0000", 2024-09-13T14:50:33.652 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 547.15062278799996, 2024-09-13T14:50:33.652 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.066262520000000005, 2024-09-13T14:50:33.652 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:33.652 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:33.652 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:33.652 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.652 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:33.652 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.488089+0000", 2024-09-13T14:50:33.652 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:33.652 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.652 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.652 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:33.652 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.488089+0000", 2024-09-13T14:50:33.653 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:33.653 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.653 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.653 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:33.653 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.488088+0000", 2024-09-13T14:50:33.653 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:50:33.653 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.653 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.653 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:33.653 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.488096+0000", 2024-09-13T14:50:33.653 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3380000000000005e-06 2024-09-13T14:50:33.653 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.653 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.653 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:33.654 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.548070+0000", 2024-09-13T14:50:33.654 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.059973789999999999 2024-09-13T14:50:33.654 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.654 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.654 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:33.654 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.548100+0000", 2024-09-13T14:50:33.654 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0759000000000001e-05 2024-09-13T14:50:33.654 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.654 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.654 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:33.654 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.548263+0000", 2024-09-13T14:50:33.654 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000162399 2024-09-13T14:50:33.654 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.654 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.655 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:33.655 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.548286+0000", 2024-09-13T14:50:33.655 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3048e-05 2024-09-13T14:50:33.655 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.655 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.655 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:33.655 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.554312+0000", 2024-09-13T14:50:33.655 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0060261400000000001 2024-09-13T14:50:33.655 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.655 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.655 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:33.655 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:41:26.554351+0000", 2024-09-13T14:50:33.655 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9165000000000002e-05 2024-09-13T14:50:33.655 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:33.655 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:33.656 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:33.656 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.656 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.656 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35379.0:220849 7.0 e297/240 7:05c074bb:::benchmark_data_smithi102_227395_object108034:head v 297'13770, mlcod=210'2566)", 2024-09-13T14:50:33.656 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:45:24.786624+0000", 2024-09-13T14:50:33.656 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 308.85208719100001, 2024-09-13T14:50:33.656 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.054973017999999998, 2024-09-13T14:50:33.656 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:33.656 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:33.656 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:33.656 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.656 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:33.656 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:24.786624+0000", 2024-09-13T14:50:33.657 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:33.657 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.657 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.657 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:33.657 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:24.786624+0000", 2024-09-13T14:50:33.657 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:33.657 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.657 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.657 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:33.657 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:24.786624+0000", 2024-09-13T14:50:33.657 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967296 2024-09-13T14:50:33.657 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.657 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.657 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:33.658 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:24.786627+0000", 2024-09-13T14:50:33.658 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.3749999999999999e-06 2024-09-13T14:50:33.659 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.659 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.659 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:33.659 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:24.786632+0000", 2024-09-13T14:50:33.659 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9980000000000003e-06 2024-09-13T14:50:33.659 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.659 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.659 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:33.659 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:24.786636+0000", 2024-09-13T14:50:33.659 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.653e-06 2024-09-13T14:50:33.659 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.659 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.659 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:33.659 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:24.801344+0000", 2024-09-13T14:50:33.660 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014707872 2024-09-13T14:50:33.660 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.660 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.660 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:33.660 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:24.801396+0000", 2024-09-13T14:50:33.660 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1835000000000001e-05 2024-09-13T14:50:33.660 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.660 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.660 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:33.660 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:24.841243+0000", 2024-09-13T14:50:33.660 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.039847070999999998 2024-09-13T14:50:33.660 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.660 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.661 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:33.661 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:24.841597+0000", 2024-09-13T14:50:33.661 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00035439400000000001 2024-09-13T14:50:33.661 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:33.661 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:33.661 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:33.661 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.661 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.661 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35379.0:220868 7.0 e297/240 7:08b776e9:::benchmark_data_smithi102_227395_object108053:head v 297'13773, mlcod=210'2566)", 2024-09-13T14:50:33.661 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:45:24.809674+0000", 2024-09-13T14:50:33.661 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 308.82903727600001, 2024-09-13T14:50:33.661 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.054525718000000001, 2024-09-13T14:50:33.661 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:33.661 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:33.662 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:33.662 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.662 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:33.662 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:24.809674+0000", 2024-09-13T14:50:33.662 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:33.662 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.662 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.662 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:33.662 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:24.809674+0000", 2024-09-13T14:50:33.662 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:33.662 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.662 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.662 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:33.662 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:24.809673+0000", 2024-09-13T14:50:33.663 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:50:33.663 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.663 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.663 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:33.663 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:24.809680+0000", 2024-09-13T14:50:33.663 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1019999999999999e-06 2024-09-13T14:50:33.663 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.663 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.663 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:33.663 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:24.809691+0000", 2024-09-13T14:50:33.663 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.082e-05 2024-09-13T14:50:33.663 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.663 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.663 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:33.664 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:24.809698+0000", 2024-09-13T14:50:33.664 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7819999999999996e-06 2024-09-13T14:50:33.664 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.664 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.664 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:33.664 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:24.822664+0000", 2024-09-13T14:50:33.664 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012965714999999999 2024-09-13T14:50:33.664 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.664 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.664 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:33.664 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:24.822689+0000", 2024-09-13T14:50:33.664 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5049999999999999e-05 2024-09-13T14:50:33.664 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.665 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.665 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:33.665 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:24.864158+0000", 2024-09-13T14:50:33.665 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.041468813 2024-09-13T14:50:33.665 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.665 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.665 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:33.665 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:24.864200+0000", 2024-09-13T14:50:33.665 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2339000000000002e-05 2024-09-13T14:50:33.665 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:33.665 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:33.665 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:33.665 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.665 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.666 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35379.0:220893 7.0 e297/240 7:086883d2:::benchmark_data_smithi102_227395_object108078:head v 297'13776, mlcod=210'2566)", 2024-09-13T14:50:33.666 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:45:24.831653+0000", 2024-09-13T14:50:33.666 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 308.80705863100002, 2024-09-13T14:50:33.666 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.052369925999999997, 2024-09-13T14:50:33.666 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:33.666 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:33.666 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:33.666 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.666 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:33.666 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:24.831653+0000", 2024-09-13T14:50:33.666 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:33.666 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.666 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.666 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:33.667 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:24.831653+0000", 2024-09-13T14:50:33.667 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:33.667 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.667 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.667 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:33.667 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:24.831652+0000", 2024-09-13T14:50:33.667 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:50:33.667 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.667 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.667 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:33.667 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:24.831660+0000", 2024-09-13T14:50:33.667 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2109999999999998e-06 2024-09-13T14:50:33.667 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.667 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.668 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:33.668 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:24.831676+0000", 2024-09-13T14:50:33.668 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5560999999999999e-05 2024-09-13T14:50:33.668 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.668 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.668 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:33.668 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:24.831683+0000", 2024-09-13T14:50:33.668 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.0199999999999997e-06 2024-09-13T14:50:33.668 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.668 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.668 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:33.668 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:24.841685+0000", 2024-09-13T14:50:33.668 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010002426999999999 2024-09-13T14:50:33.668 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.669 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.669 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:33.669 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:24.841708+0000", 2024-09-13T14:50:33.669 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2776999999999999e-05 2024-09-13T14:50:33.669 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.669 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.669 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:33.669 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:24.883991+0000", 2024-09-13T14:50:33.669 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.042282921000000001 2024-09-13T14:50:33.669 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.669 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.669 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:33.669 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:24.884023+0000", 2024-09-13T14:50:33.669 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2067999999999999e-05 2024-09-13T14:50:33.670 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:33.670 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:33.670 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:33.670 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.670 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.670 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35379.0:222322 7.0 e297/240 7:0006c64a:::benchmark_data_smithi102_227395_object109507:head v 297'13852, mlcod=210'2566)", 2024-09-13T14:50:33.670 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:45:26.217319+0000", 2024-09-13T14:50:33.670 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 307.42139232, 2024-09-13T14:50:33.670 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.052771706000000002, 2024-09-13T14:50:33.670 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:33.670 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:33.670 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:33.670 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.671 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:33.672 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:33.672 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.217319+0000", 2024-09-13T14:50:33.672 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:33.672 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.672 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.673 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:33.673 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.217319+0000", 2024-09-13T14:50:33.673 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:33.673 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.673 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.673 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:33.673 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.217318+0000", 2024-09-13T14:50:33.673 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:50:33.673 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.673 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.673 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:33.673 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.217327+0000", 2024-09-13T14:50:33.673 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.9639999999999992e-06 2024-09-13T14:50:33.674 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.674 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.674 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:33.674 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.217340+0000", 2024-09-13T14:50:33.674 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3168e-05 2024-09-13T14:50:33.674 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.674 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.674 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:33.674 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.217347+0000", 2024-09-13T14:50:33.674 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6239999999999996e-06 2024-09-13T14:50:33.674 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.674 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.674 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:33.674 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.217416+0000", 2024-09-13T14:50:33.675 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8992000000000006e-05 2024-09-13T14:50:33.675 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.675 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.675 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:33.675 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.217444+0000", 2024-09-13T14:50:33.675 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8198e-05 2024-09-13T14:50:33.675 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.675 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.675 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:33.675 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.269901+0000", 2024-09-13T14:50:33.675 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.052457332000000002 2024-09-13T14:50:33.675 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.675 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.675 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:33.676 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:26.270091+0000", 2024-09-13T14:50:33.676 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00018934299999999999 2024-09-13T14:50:33.676 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:33.676 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:33.676 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:33.676 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.676 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.676 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35379.0:223232 7.0 e298/240 7:0612cd3d:::benchmark_data_smithi102_227395_object110417:head v 298'13900, mlcod=210'2566)", 2024-09-13T14:50:33.676 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:45:27.148403+0000", 2024-09-13T14:50:33.676 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 306.49030804, 2024-09-13T14:50:33.676 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.054078477, 2024-09-13T14:50:33.676 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:33.676 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:33.677 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:33.677 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.677 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:33.677 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:27.148403+0000", 2024-09-13T14:50:33.677 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:33.677 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.677 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.677 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:33.677 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:27.148403+0000", 2024-09-13T14:50:33.677 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:33.677 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.677 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.677 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:33.677 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:27.148403+0000", 2024-09-13T14:50:33.678 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:50:33.678 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.678 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.678 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:33.678 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:27.148408+0000", 2024-09-13T14:50:33.678 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3900000000000001e-06 2024-09-13T14:50:33.678 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.678 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.678 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:33.678 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:27.148417+0000", 2024-09-13T14:50:33.678 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3839999999999997e-06 2024-09-13T14:50:33.678 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.678 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.679 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:33.679 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:27.148421+0000", 2024-09-13T14:50:33.679 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.189e-06 2024-09-13T14:50:33.679 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.679 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.679 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:33.679 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:27.149151+0000", 2024-09-13T14:50:33.679 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00073037000000000004 2024-09-13T14:50:33.679 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.679 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.679 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:33.679 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:27.149174+0000", 2024-09-13T14:50:33.679 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2898999999999999e-05 2024-09-13T14:50:33.679 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.680 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.680 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:33.680 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:27.202370+0000", 2024-09-13T14:50:33.680 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.053196343 2024-09-13T14:50:33.680 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.680 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.680 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:33.680 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:27.202482+0000", 2024-09-13T14:50:33.680 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000111366 2024-09-13T14:50:33.680 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:33.680 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:33.680 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:33.680 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.680 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.681 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35379.0:223484 7.0 e298/240 7:062b9a7c:::benchmark_data_smithi102_227395_object110669:head v 298'13919, mlcod=210'2566)", 2024-09-13T14:50:33.681 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:45:27.408874+0000", 2024-09-13T14:50:33.681 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 306.22983771700001, 2024-09-13T14:50:33.681 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.051336930000000003, 2024-09-13T14:50:33.681 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:33.681 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:33.681 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:33.681 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.681 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:33.681 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:27.408874+0000", 2024-09-13T14:50:33.681 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:33.681 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.681 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.682 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:33.682 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:27.408874+0000", 2024-09-13T14:50:33.682 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:33.682 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.682 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.682 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:33.682 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:27.408873+0000", 2024-09-13T14:50:33.682 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:50:33.682 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.682 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.682 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:33.682 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:27.408880+0000", 2024-09-13T14:50:33.682 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3860000000000001e-06 2024-09-13T14:50:33.682 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.682 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.683 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:33.683 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:27.408896+0000", 2024-09-13T14:50:33.683 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5835999999999999e-05 2024-09-13T14:50:33.683 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.683 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.683 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:33.683 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:27.408903+0000", 2024-09-13T14:50:33.683 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.691e-06 2024-09-13T14:50:33.683 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.683 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.683 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:33.683 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:27.411854+0000", 2024-09-13T14:50:33.683 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0029512710000000001 2024-09-13T14:50:33.683 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.684 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.684 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:33.684 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:27.411922+0000", 2024-09-13T14:50:33.684 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8048999999999998e-05 2024-09-13T14:50:33.684 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.684 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.684 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:33.684 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:27.460093+0000", 2024-09-13T14:50:33.684 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.048170508000000001 2024-09-13T14:50:33.684 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.684 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.684 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:33.684 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:27.460211+0000", 2024-09-13T14:50:33.685 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000117929 2024-09-13T14:50:33.685 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:33.685 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:33.685 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:33.685 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.685 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.685 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35379.0:223673 7.0 e298/240 7:0c7839ff:::benchmark_data_smithi102_227395_object110858:head v 298'13934, mlcod=210'2566)", 2024-09-13T14:50:33.685 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:45:27.584939+0000", 2024-09-13T14:50:33.685 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 306.05377202199998, 2024-09-13T14:50:33.685 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.053583482000000002, 2024-09-13T14:50:33.685 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:33.685 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:33.685 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:33.685 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.686 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:33.686 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:27.584939+0000", 2024-09-13T14:50:33.686 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:33.686 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.686 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.686 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:33.686 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:27.584939+0000", 2024-09-13T14:50:33.686 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:33.686 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.686 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.686 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:33.686 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:27.584939+0000", 2024-09-13T14:50:33.686 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:50:33.686 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.687 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.687 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:33.687 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:27.584943+0000", 2024-09-13T14:50:33.687 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9049999999999999e-06 2024-09-13T14:50:33.687 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.687 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.687 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:33.687 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:27.584949+0000", 2024-09-13T14:50:33.687 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8080000000000001e-06 2024-09-13T14:50:33.687 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.687 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.687 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:33.687 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:27.584953+0000", 2024-09-13T14:50:33.688 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7759999999999999e-06 2024-09-13T14:50:33.688 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.688 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.688 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:33.688 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:27.587051+0000", 2024-09-13T14:50:33.688 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0020979750000000002 2024-09-13T14:50:33.688 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.688 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.688 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:33.688 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:27.587074+0000", 2024-09-13T14:50:33.688 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2993000000000002e-05 2024-09-13T14:50:33.688 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.688 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.688 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:33.689 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:27.638421+0000", 2024-09-13T14:50:33.689 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.051347272999999999 2024-09-13T14:50:33.689 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.689 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.689 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:33.689 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:27.638523+0000", 2024-09-13T14:50:33.689 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000102042 2024-09-13T14:50:33.689 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:33.689 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:33.689 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:33.689 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.689 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.689 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.35379.0:224924 7.0 e298/240 7:05f5fcb2:::benchmark_data_smithi102_227395_object112109:head v 298'14014, mlcod=210'2566)", 2024-09-13T14:50:33.689 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:45:28.781485+0000", 2024-09-13T14:50:33.690 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 304.857226733, 2024-09-13T14:50:33.690 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.060369177000000003, 2024-09-13T14:50:33.690 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:33.690 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:33.690 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:33.690 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.690 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:33.690 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:28.781485+0000", 2024-09-13T14:50:33.690 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:33.690 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.690 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.690 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:33.690 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:28.781485+0000", 2024-09-13T14:50:33.691 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:33.691 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.691 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.691 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:33.691 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:28.781484+0000", 2024-09-13T14:50:33.691 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967296 2024-09-13T14:50:33.691 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.691 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.691 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:33.691 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:28.781488+0000", 2024-09-13T14:50:33.691 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8929999999999998e-06 2024-09-13T14:50:33.691 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.691 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.691 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:33.692 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:28.781495+0000", 2024-09-13T14:50:33.692 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4509999999999997e-06 2024-09-13T14:50:33.692 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.692 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.692 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:33.692 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:28.781499+0000", 2024-09-13T14:50:33.692 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0849999999999999e-06 2024-09-13T14:50:33.692 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.692 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.692 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:33.692 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:28.798646+0000", 2024-09-13T14:50:33.692 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017147293000000001 2024-09-13T14:50:33.692 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.693 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.693 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:33.693 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:28.798670+0000", 2024-09-13T14:50:33.693 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4165999999999999e-05 2024-09-13T14:50:33.693 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.693 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.693 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:33.693 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:28.841781+0000", 2024-09-13T14:50:33.693 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.043110950000000002 2024-09-13T14:50:33.693 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.693 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.693 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:33.693 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:45:28.841854+0000", 2024-09-13T14:50:33.693 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2577000000000007e-05 2024-09-13T14:50:33.694 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:33.694 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:33.694 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:33.694 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.694 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.694 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.46157.0:55621 8.7 e315/306 8:e7453055:::benchmark_data_smithi102_313275_object55620:head v 315'3522, mlcod=315'3520)", 2024-09-13T14:50:33.694 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:46:38.945674+0000", 2024-09-13T14:50:33.694 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 234.69303702299999, 2024-09-13T14:50:33.694 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.26997093999999999, 2024-09-13T14:50:33.694 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:33.694 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:33.694 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:33.694 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.694 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:33.695 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.945674+0000", 2024-09-13T14:50:33.695 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:33.695 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.695 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.695 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:33.695 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.945674+0000", 2024-09-13T14:50:33.695 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:33.695 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.695 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.695 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:33.695 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.945673+0000", 2024-09-13T14:50:33.695 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:50:33.695 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.695 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.696 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:33.696 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.945717+0000", 2024-09-13T14:50:33.696 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3464000000000003e-05 2024-09-13T14:50:33.696 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.696 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.696 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:33.696 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.945728+0000", 2024-09-13T14:50:33.696 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0679999999999999e-05 2024-09-13T14:50:33.696 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.696 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.696 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:33.696 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.945734+0000", 2024-09-13T14:50:33.696 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6290000000000004e-06 2024-09-13T14:50:33.697 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.697 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.697 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:33.697 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.945795+0000", 2024-09-13T14:50:33.697 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0414000000000001e-05 2024-09-13T14:50:33.697 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.697 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.697 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:33.697 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.945813+0000", 2024-09-13T14:50:33.697 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7887e-05 2024-09-13T14:50:33.697 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.697 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.697 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:33.697 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.215303+0000", 2024-09-13T14:50:33.698 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.26949056300000002 2024-09-13T14:50:33.698 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.698 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.698 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:33.698 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.215645+0000", 2024-09-13T14:50:33.698 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00034222300000000001 2024-09-13T14:50:33.698 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:33.698 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:33.698 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:33.698 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.698 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.698 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.46157.0:55619 8.5 e315/304 8:a95080a8:::benchmark_data_smithi102_313275_object55618:head v 315'3479, mlcod=315'3477)", 2024-09-13T14:50:33.698 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:46:38.947670+0000", 2024-09-13T14:50:33.698 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 234.69104158100001, 2024-09-13T14:50:33.699 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.27381550500000001, 2024-09-13T14:50:33.699 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:33.699 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:33.699 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:33.699 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.699 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:33.699 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.947670+0000", 2024-09-13T14:50:33.699 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:33.699 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.699 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.699 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:33.699 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.947670+0000", 2024-09-13T14:50:33.699 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:33.699 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.700 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.700 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:33.700 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.947669+0000", 2024-09-13T14:50:33.700 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:50:33.700 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.700 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.700 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:33.700 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.947701+0000", 2024-09-13T14:50:33.700 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2079000000000002e-05 2024-09-13T14:50:33.700 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.700 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.700 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:33.700 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.947713+0000", 2024-09-13T14:50:33.701 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1843e-05 2024-09-13T14:50:33.701 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.701 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.701 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:33.701 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.947720+0000", 2024-09-13T14:50:33.701 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1450000000000002e-06 2024-09-13T14:50:33.701 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.701 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.701 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:33.701 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.213072+0000", 2024-09-13T14:50:33.701 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.26535172899999998 2024-09-13T14:50:33.701 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.701 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.701 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:33.702 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.213101+0000", 2024-09-13T14:50:33.702 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.915e-05 2024-09-13T14:50:33.702 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.702 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.702 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:33.702 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.221242+0000", 2024-09-13T14:50:33.702 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0081412930000000008 2024-09-13T14:50:33.702 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.702 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.702 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:33.702 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.221485+0000", 2024-09-13T14:50:33.702 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000243327 2024-09-13T14:50:33.702 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:33.702 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:33.703 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:33.703 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.703 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.703 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.46157.0:55622 8.7 e315/306 8:e4c8fb9b:::benchmark_data_smithi102_313275_object55621:head v 315'3523, mlcod=315'3520)", 2024-09-13T14:50:33.703 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:46:38.948123+0000", 2024-09-13T14:50:33.703 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 234.690588838, 2024-09-13T14:50:33.703 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.26951215099999998, 2024-09-13T14:50:33.703 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:33.703 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:33.703 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:33.703 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.703 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:33.703 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.948123+0000", 2024-09-13T14:50:33.704 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:33.704 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.704 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.704 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:33.704 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.948123+0000", 2024-09-13T14:50:33.704 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:33.704 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.704 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.704 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:33.704 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.948122+0000", 2024-09-13T14:50:33.704 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:50:33.704 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.704 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.704 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:33.705 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.948242+0000", 2024-09-13T14:50:33.705 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0001199 2024-09-13T14:50:33.705 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.705 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.705 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:33.705 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.213011+0000", 2024-09-13T14:50:33.705 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.26476898700000001 2024-09-13T14:50:33.705 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.705 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.705 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:33.705 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.213031+0000", 2024-09-13T14:50:33.705 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0506999999999999e-05 2024-09-13T14:50:33.705 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.706 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.706 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:33.706 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.213546+0000", 2024-09-13T14:50:33.706 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00051459499999999996 2024-09-13T14:50:33.706 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.706 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.706 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:33.706 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.213565+0000", 2024-09-13T14:50:33.706 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.863e-05 2024-09-13T14:50:33.706 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.706 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.706 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:33.706 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.217535+0000", 2024-09-13T14:50:33.706 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0039704470000000002 2024-09-13T14:50:33.707 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.707 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.707 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:33.707 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.217635+0000", 2024-09-13T14:50:33.707 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.9778000000000006e-05 2024-09-13T14:50:33.707 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:33.707 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:33.707 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:33.707 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.707 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.707 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.46157.0:55626 8.6 e315/304 8:68e51ae3:::benchmark_data_smithi102_313275_object55625:head v 315'3562, mlcod=315'3560)", 2024-09-13T14:50:33.707 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:46:38.948147+0000", 2024-09-13T14:50:33.707 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 234.69056462200001, 2024-09-13T14:50:33.707 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.26759492400000001, 2024-09-13T14:50:33.708 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:33.708 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:33.708 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:33.708 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.708 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:33.708 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.948147+0000", 2024-09-13T14:50:33.708 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:33.708 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.708 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.708 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:33.708 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.948147+0000", 2024-09-13T14:50:33.708 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:33.708 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.709 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.709 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:33.709 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.948147+0000", 2024-09-13T14:50:33.709 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:50:33.709 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.709 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.709 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:33.709 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.213234+0000", 2024-09-13T14:50:33.709 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.265087709 2024-09-13T14:50:33.709 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.709 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.709 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:33.709 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.213241+0000", 2024-09-13T14:50:33.709 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9750000000000001e-06 2024-09-13T14:50:33.709 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.710 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.710 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:33.710 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.213246+0000", 2024-09-13T14:50:33.710 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2769999999999999e-06 2024-09-13T14:50:33.710 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.710 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.710 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:33.710 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.213304+0000", 2024-09-13T14:50:33.710 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8257999999999997e-05 2024-09-13T14:50:33.710 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.710 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.710 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:33.710 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.213322+0000", 2024-09-13T14:50:33.710 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8159999999999999e-05 2024-09-13T14:50:33.711 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.711 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.711 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:33.711 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.215565+0000", 2024-09-13T14:50:33.711 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0022427419999999998 2024-09-13T14:50:33.711 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.711 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.711 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:33.711 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.215742+0000", 2024-09-13T14:50:33.711 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000177061 2024-09-13T14:50:33.711 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:33.711 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:33.711 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:33.711 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.712 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.712 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.46157.0:55627 8.3 e315/306 8:c1e2abda:::benchmark_data_smithi102_313275_object55626:head v 315'3421, mlcod=315'3419)", 2024-09-13T14:50:33.712 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:46:38.948838+0000", 2024-09-13T14:50:33.712 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 234.68987354800001, 2024-09-13T14:50:33.712 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.26959770900000002, 2024-09-13T14:50:33.712 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:33.712 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:33.712 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:33.712 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.712 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:33.712 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.948838+0000", 2024-09-13T14:50:33.712 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:33.712 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.713 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.713 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:33.713 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.948838+0000", 2024-09-13T14:50:33.713 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:33.713 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.713 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.713 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:33.713 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.948837+0000", 2024-09-13T14:50:33.713 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:50:33.713 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.713 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.713 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:33.713 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.948988+0000", 2024-09-13T14:50:33.713 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015049799999999999 2024-09-13T14:50:33.714 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.714 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.714 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:33.714 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.213438+0000", 2024-09-13T14:50:33.714 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.264450464 2024-09-13T14:50:33.714 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.714 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.714 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:33.714 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.213445+0000", 2024-09-13T14:50:33.714 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.939e-06 2024-09-13T14:50:33.714 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.714 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.714 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:33.714 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.213906+0000", 2024-09-13T14:50:33.715 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00046125400000000003 2024-09-13T14:50:33.715 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.715 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.715 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:33.715 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.213939+0000", 2024-09-13T14:50:33.715 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.3148000000000002e-05 2024-09-13T14:50:33.715 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.715 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.715 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:33.715 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.218295+0000", 2024-09-13T14:50:33.715 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0043552249999999999 2024-09-13T14:50:33.715 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.715 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.715 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:33.716 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.218436+0000", 2024-09-13T14:50:33.716 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000140896 2024-09-13T14:50:33.716 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:33.716 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:33.716 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:33.716 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.716 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.716 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.46157.0:135173 8.9 e339/304 8:97f3c262:::benchmark_data_smithi102_313275_object4586:head v 339'8557, mlcod=339'8555)", 2024-09-13T14:50:33.716 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:47:50.473760+0000", 2024-09-13T14:50:33.716 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 163.16495161700001, 2024-09-13T14:50:33.716 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.033006190999999997, 2024-09-13T14:50:33.716 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:33.716 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:33.717 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:33.717 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.717 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:33.717 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:50.473760+0000", 2024-09-13T14:50:33.717 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:33.717 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.717 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.717 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:33.717 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:50.473760+0000", 2024-09-13T14:50:33.717 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:33.717 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.717 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.717 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:33.717 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:50.473759+0000", 2024-09-13T14:50:33.718 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:50:33.718 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.718 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.718 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:33.718 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:50.473767+0000", 2024-09-13T14:50:33.718 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6990000000000003e-06 2024-09-13T14:50:33.718 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.718 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.718 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:33.718 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:50.473782+0000", 2024-09-13T14:50:33.718 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5586e-05 2024-09-13T14:50:33.718 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.718 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.718 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:33.719 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:50.473789+0000", 2024-09-13T14:50:33.719 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2019999999999998e-06 2024-09-13T14:50:33.719 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.719 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.719 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:33.719 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:50.473856+0000", 2024-09-13T14:50:33.719 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6309999999999994e-05 2024-09-13T14:50:33.719 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.719 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.719 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:33.719 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:50.473884+0000", 2024-09-13T14:50:33.719 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8714999999999999e-05 2024-09-13T14:50:33.719 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.719 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.720 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:33.720 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:50.506625+0000", 2024-09-13T14:50:33.720 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.032740865000000001 2024-09-13T14:50:33.720 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.720 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.720 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:33.720 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:50.506766+0000", 2024-09-13T14:50:33.720 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014074899999999999 2024-09-13T14:50:33.720 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:33.720 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:33.720 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:33.720 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.720 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.721 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.46157.0:186945 8.3 e346/337 8:c693f8b3:::benchmark_data_smithi102_313275_object56358:head v 346'11611, mlcod=346'11608)", 2024-09-13T14:50:33.721 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:48:52.310764+0000", 2024-09-13T14:50:33.721 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 101.327947783, 2024-09-13T14:50:33.721 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.028103357999999998, 2024-09-13T14:50:33.721 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:33.721 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:33.721 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:33.721 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.721 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:33.721 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:48:52.310764+0000", 2024-09-13T14:50:33.721 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:33.721 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.721 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.721 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:33.722 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:48:52.310764+0000", 2024-09-13T14:50:33.722 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:33.722 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.722 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.722 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:33.722 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:48:52.310763+0000", 2024-09-13T14:50:33.722 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:50:33.722 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.722 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.722 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:33.722 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:48:52.310772+0000", 2024-09-13T14:50:33.722 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.884e-06 2024-09-13T14:50:33.722 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.722 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.723 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:33.723 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:48:52.310791+0000", 2024-09-13T14:50:33.723 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9568e-05 2024-09-13T14:50:33.723 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.723 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.723 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:33.723 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:48:52.310799+0000", 2024-09-13T14:50:33.723 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9589999999999995e-06 2024-09-13T14:50:33.723 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.723 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.723 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:33.723 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:48:52.310868+0000", 2024-09-13T14:50:33.723 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8658999999999994e-05 2024-09-13T14:50:33.723 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.724 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.724 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:33.724 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:48:52.310900+0000", 2024-09-13T14:50:33.724 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2292999999999997e-05 2024-09-13T14:50:33.724 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.724 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.724 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:33.724 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:48:52.338757+0000", 2024-09-13T14:50:33.724 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.027857051000000001 2024-09-13T14:50:33.724 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.724 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.724 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:33.724 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:48:52.338867+0000", 2024-09-13T14:50:33.724 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000109944 2024-09-13T14:50:33.725 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:33.725 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:33.725 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:33.725 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.725 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.725 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.46157.0:251141 8.3 e364/358 8:cb8af7bf:::benchmark_data_smithi102_313275_object120554:head v 364'15613, mlcod=364'15610)", 2024-09-13T14:50:33.725 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:50:16.749936+0000", 2024-09-13T14:50:33.725 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 16.888775810999999, 2024-09-13T14:50:33.725 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.029279968, 2024-09-13T14:50:33.725 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:33.725 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:33.725 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:33.725 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.725 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:33.726 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:50:16.749936+0000", 2024-09-13T14:50:33.726 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:33.726 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.726 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.726 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:33.726 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:50:16.749936+0000", 2024-09-13T14:50:33.726 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:33.726 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.726 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.726 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:33.726 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:50:16.749935+0000", 2024-09-13T14:50:33.726 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:50:33.726 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.726 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.727 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:33.727 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:50:16.749940+0000", 2024-09-13T14:50:33.727 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9699999999999998e-06 2024-09-13T14:50:33.727 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.727 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.727 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:33.727 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:50:16.749949+0000", 2024-09-13T14:50:33.727 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.6969999999999999e-06 2024-09-13T14:50:33.727 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.727 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.727 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:33.727 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:50:16.749954+0000", 2024-09-13T14:50:33.727 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5099999999999998e-06 2024-09-13T14:50:33.727 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.728 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.728 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:33.728 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:50:16.751784+0000", 2024-09-13T14:50:33.728 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001829757 2024-09-13T14:50:33.728 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.728 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.728 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:33.728 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:50:16.751803+0000", 2024-09-13T14:50:33.728 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9059000000000002e-05 2024-09-13T14:50:33.728 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.728 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.728 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:33.728 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:50:16.779170+0000", 2024-09-13T14:50:33.728 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.027367446 2024-09-13T14:50:33.729 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.729 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.729 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:33.729 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:50:16.779216+0000", 2024-09-13T14:50:33.729 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5127999999999999e-05 2024-09-13T14:50:33.729 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:33.729 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:33.729 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:33.729 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.729 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.729 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.46157.0:251144 8.3 e364/358 8:c2c7f68b:::benchmark_data_smithi102_313275_object120557:head v 364'15614, mlcod=364'15610)", 2024-09-13T14:50:33.729 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:50:16.753389+0000", 2024-09-13T14:50:33.729 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 16.885322554999998, 2024-09-13T14:50:33.730 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.025855091, 2024-09-13T14:50:33.730 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:50:33.730 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:33.730 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:50:33.730 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.730 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:50:33.730 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:50:16.753389+0000", 2024-09-13T14:50:33.730 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:33.730 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.730 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.730 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:50:33.730 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:50:16.753389+0000", 2024-09-13T14:50:33.730 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:50:33.730 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.731 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.731 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:50:33.731 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:50:16.753388+0000", 2024-09-13T14:50:33.731 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:50:33.731 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.731 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.731 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:50:33.731 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:50:16.753395+0000", 2024-09-13T14:50:33.731 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8719999999999996e-06 2024-09-13T14:50:33.731 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.731 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.731 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:50:33.731 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:50:16.753405+0000", 2024-09-13T14:50:33.731 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0043e-05 2024-09-13T14:50:33.732 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.732 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.732 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:50:33.732 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:50:16.753411+0000", 2024-09-13T14:50:33.732 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8019999999999997e-06 2024-09-13T14:50:33.732 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.732 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.732 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:50:33.732 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:50:16.755203+0000", 2024-09-13T14:50:33.732 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001792402 2024-09-13T14:50:33.732 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.732 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.732 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:50:33.732 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:50:16.755222+0000", 2024-09-13T14:50:33.733 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8811e-05 2024-09-13T14:50:33.733 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.733 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.733 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:50:33.733 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:50:16.779220+0000", 2024-09-13T14:50:33.733 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.023998157999999999 2024-09-13T14:50:33.733 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:50:33.733 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:50:33.733 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:50:33.733 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:50:16.779244+0000", 2024-09-13T14:50:33.733 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3947999999999998e-05 2024-09-13T14:50:33.733 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:33.733 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:33.733 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:33.734 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:50:33.734 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:50:33.734 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:50:33.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:33.755 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:33.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:33.856 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:33.881 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:33.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:33.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:33.982 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:34.006 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:34.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:34.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:34.109 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:34.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:34.131 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:34.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:34.233 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:34.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:34.262 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:34.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:34.363 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:34.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:34.389 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:34.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:34.490 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:34.511 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:34.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:34.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:34.546 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T14:50:34.613 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:34.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:34.637 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:34.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:34.738 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:34.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:34.761 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:34.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:34.864 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:34.887 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:34.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:34.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:34.895 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:34 smithi102 ceph-mon[110183]: pgmap v1278: 109 pgs: 109 active+clean; 6.6 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 760 op/s 2024-09-13T14:50:34.896 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:34 smithi102 ceph-mon[115269]: pgmap v1278: 109 pgs: 109 active+clean; 6.6 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 760 op/s 2024-09-13T14:50:34.989 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:34.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:34 smithi195 ceph-mon[104262]: pgmap v1278: 109 pgs: 109 active+clean; 6.6 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 760 op/s 2024-09-13T14:50:35.010 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:35.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:35.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:35.113 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:35.135 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:35.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:35.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:35.237 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:35.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:35.260 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:35.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:35.362 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:35.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:35.384 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:35.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:35.485 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:35.507 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:35.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:35.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:35.608 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:35.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:35.631 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:35.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:35.701 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:35.723 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-13T14:50:35.723 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:50:35.733 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:35.736 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:35.739 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'unique_pool_3' removed 2024-09-13T14:50:35.820 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:35.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:35.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:35.922 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:35.944 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:35.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:35.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:35.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:35 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2128930343' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:50:36.046 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:36.068 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:36.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:36.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:36.140 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:35 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2128930343' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:50:36.141 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:35 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2128930343' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:50:36.169 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:36.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:36.191 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:36.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:36.292 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:36.314 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:36.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:36.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:36.416 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:36.437 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:36.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:36.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:36.539 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:36.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:36.562 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:36.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:36.588 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-13T14:50:36.589 INFO:tasks.radosbench:Beginning radosbench... 2024-09-13T14:50:36.589 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-09-13T14:50:36.589 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_4 16 2024-09-13T14:50:36.663 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:36.685 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:36.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:36.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:36.711 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:50:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:50:36] "GET /metrics HTTP/1.1" 200 36531 "" "Prometheus/2.51.0" 2024-09-13T14:50:36.788 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:36.809 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:36.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:36.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:36.911 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:36.934 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:36.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:36.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:36.979 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:36 smithi102 ceph-mon[110183]: pgmap v1279: 109 pgs: 109 active+clean; 4.6 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 535 op/s 2024-09-13T14:50:36.979 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:36 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2128930343' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:50:36.979 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:36 smithi102 ceph-mon[110183]: osdmap e372: 8 total, 8 up, 8 in 2024-09-13T14:50:36.980 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:36 smithi102 ceph-mon[115269]: pgmap v1279: 109 pgs: 109 active+clean; 4.6 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 535 op/s 2024-09-13T14:50:36.980 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:36 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2128930343' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:50:36.980 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:36 smithi102 ceph-mon[115269]: osdmap e372: 8 total, 8 up, 8 in 2024-09-13T14:50:36.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:36 smithi195 ceph-mon[104262]: pgmap v1279: 109 pgs: 109 active+clean; 4.6 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 535 op/s 2024-09-13T14:50:36.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:36 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2128930343' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:50:36.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:36 smithi195 ceph-mon[104262]: osdmap e372: 8 total, 8 up, 8 in 2024-09-13T14:50:37.036 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:37.059 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:37.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:37.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:37.161 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:37.183 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:37.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:37.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:37.285 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:37.302 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:50:37.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:37.309 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:37.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:37.409 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:37.431 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:37.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:37.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:37.533 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:37.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:37.556 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:37.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:37.657 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:37.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:37.680 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:37.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:37.781 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:37.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:37.804 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:37.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:37.905 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:37.927 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:37.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:37.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:38.029 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:38.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:38.054 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:38.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:38.155 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:38.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:38.178 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:38.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:38.279 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:38.300 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:38.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:38.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:38.402 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:38.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:38.426 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:38.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:38.527 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:38.549 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:38.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:38.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:38.650 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:38.672 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:38.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:38.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:38.773 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:38.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:38.798 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:38.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:38.899 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:38.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:38 smithi102 ceph-mon[110183]: pgmap v1281: 93 pgs: 93 active+clean; 4.6 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-13T14:50:38.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:38 smithi102 ceph-mon[115269]: pgmap v1281: 93 pgs: 93 active+clean; 4.6 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-13T14:50:38.920 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:38.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:38.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:38.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:38 smithi195 ceph-mon[104262]: pgmap v1281: 93 pgs: 93 active+clean; 4.6 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-13T14:50:39.022 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:39.048 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:39.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:39.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:39.150 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:39.174 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:39.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:39.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:39.275 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:39.301 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:39.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:39.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:39.403 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:39.427 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:39.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:39.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:39.529 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:39.551 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:39.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:39.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:39.653 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:39.679 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:39.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:39.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:39.781 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:39.807 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:39.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:39.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:39.909 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:39.934 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:39.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:39.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:40.036 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:40.061 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:40.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:40.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:40.163 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:40.190 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:40.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:40.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:40.292 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:40.316 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:40.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:40.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:40.418 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:40.446 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:40.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:40.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:40.548 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:40.574 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:40.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:40.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:40.676 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:40.701 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:40.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:40.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:40.803 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:40.830 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:40.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:40.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:40.932 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:40.959 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:40.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:40.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:40.996 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:40 smithi102 ceph-mon[110183]: pgmap v1282: 93 pgs: 93 active+clean; 4.6 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail 2024-09-13T14:50:40.996 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:40 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:50:40.996 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:40 smithi102 ceph-mon[115269]: pgmap v1282: 93 pgs: 93 active+clean; 4.6 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail 2024-09-13T14:50:40.996 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:40 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:50:40.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:40 smithi195 ceph-mon[104262]: pgmap v1282: 93 pgs: 93 active+clean; 4.6 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail 2024-09-13T14:50:40.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:40 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:50:41.043 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:41.061 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:41.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:41.089 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:41.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:41.190 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:41.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:41.213 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:41.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:41.314 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:41.315 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:41.340 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:41.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:41.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:41.442 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:41.464 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:41.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:41.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:41.566 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:41.588 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:41.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:41.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:41.690 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:41.716 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:41.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:41.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:41.817 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:41.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:41.843 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:41.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:41.874 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:41.943 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:41.967 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:41.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:41.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:41.976 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:50:41.976 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [], 2024-09-13T14:50:41.976 INFO:teuthology.orchestra.run.smithi102.stdout: "num_ops": 0 2024-09-13T14:50:41.976 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:50:42.068 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:42.092 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:42.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:42.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:42.194 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:42.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:42.216 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:42.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:42.317 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:42.343 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:42.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:42.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:42.445 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:42.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:42.473 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:42.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:42.574 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:42.601 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:42.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:42.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:42.703 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:42.728 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:42.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:42.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:42.753 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is unset 2024-09-13T14:50:42.829 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:42.851 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:42.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:42.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:42.895 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T14:50:42.954 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:42.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:42 smithi195 ceph-mon[104262]: pgmap v1283: 93 pgs: 93 active+clean; 4.6 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-13T14:50:42.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:42 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3853828285' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:50:43.000 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:43.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:43.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:43.102 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:43.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:43.124 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:43.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:43.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:42 smithi102 ceph-mon[110183]: pgmap v1283: 93 pgs: 93 active+clean; 4.6 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-13T14:50:43.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:42 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3853828285' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:50:43.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:42 smithi102 ceph-mon[115269]: pgmap v1283: 93 pgs: 93 active+clean; 4.6 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-13T14:50:43.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:42 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3853828285' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:50:43.224 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:43.251 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:43.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:43.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:43.352 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:43.378 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:43.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:43.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:43.480 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:43.504 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:43.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:43.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:43.606 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:43.630 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:43.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:43.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:43.732 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:43.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:43.757 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:43.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:43.858 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:43.879 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:43.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:43.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:43.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:43 smithi102 ceph-mon[110183]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:50:43.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:43 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3853828285' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:50:43.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:43 smithi102 ceph-mon[110183]: osdmap e373: 8 total, 8 up, 8 in 2024-09-13T14:50:43.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:43 smithi102 ceph-mon[115269]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:50:43.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:43 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3853828285' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:50:43.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:43 smithi102 ceph-mon[115269]: osdmap e373: 8 total, 8 up, 8 in 2024-09-13T14:50:43.982 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:44.003 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:44.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:44.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:44.104 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:44.130 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:44.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:44.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:44.232 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:44.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:43 smithi195 ceph-mon[104262]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:50:44.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:43 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3853828285' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:50:44.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:43 smithi195 ceph-mon[104262]: osdmap e373: 8 total, 8 up, 8 in 2024-09-13T14:50:44.253 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:44.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:44.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:44.355 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:44.382 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:44.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:44.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:44.483 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:44.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:44.506 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:44.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:44.607 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:44.628 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:44.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:44.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:44.732 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:44.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:44.754 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:44.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:44.855 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:44.876 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:44.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:44.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:44.978 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:45.000 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:45.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:45.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:45.101 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:45.128 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:45.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:45.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:45.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:44 smithi102 ceph-mon[110183]: pgmap v1285: 93 pgs: 93 active+clean; 4.6 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-13T14:50:45.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:44 smithi102 ceph-mon[115269]: pgmap v1285: 93 pgs: 93 active+clean; 4.6 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-13T14:50:45.230 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:45.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:44 smithi195 ceph-mon[104262]: pgmap v1285: 93 pgs: 93 active+clean; 4.6 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-13T14:50:45.255 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:45.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:45.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:45.357 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:45.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:45.383 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:45.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:45.484 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:45.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:45.507 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:45.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:45.609 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:45.630 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:45.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:45.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:45.732 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:45.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:45.753 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:45.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:45.855 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:45.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:45.883 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:45.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:45.984 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:46.010 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:46.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:46.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:46.103 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:50:46.112 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:46.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:46.170 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:46.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:46.271 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:46.298 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:46.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:46.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:46.400 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:46.408 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:50:46.408 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:50:46.421 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:46.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:46.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:46.425 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:50:46.426 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:50:46.446 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:50:46.446 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:50:46.463 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:50:46.463 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:50:46.480 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:50:46.480 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:50:46.497 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:50:46.497 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:50:46.514 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:50:46.514 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:50:46.523 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:46.531 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:50:46.531 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:50:46.549 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:46.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:46.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:46.651 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:46.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:50:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:50:46] "GET /metrics HTTP/1.1" 200 34169 "" "Prometheus/2.51.0" 2024-09-13T14:50:46.677 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:46.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:46.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:46.779 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:46.802 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:46.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:46.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:46.904 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:46.928 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:46.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:46.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:47.029 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:47.052 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:47.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:47.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:47.064 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:46 smithi102 ceph-mon[110183]: pgmap v1286: 93 pgs: 93 active+clean; 4.6 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-13T14:50:47.065 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:46 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/948625768' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:50:47.065 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:46 smithi102 ceph-mon[115269]: pgmap v1286: 93 pgs: 93 active+clean; 4.6 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-13T14:50:47.065 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:46 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/948625768' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:50:47.154 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:47.176 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:47.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:47.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:47.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:46 smithi195 ceph-mon[104262]: pgmap v1286: 93 pgs: 93 active+clean; 4.6 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-13T14:50:47.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:46 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/948625768' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:50:47.277 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:47.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:47.301 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:47.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:47.401 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:47.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:47.423 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:47.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:47.524 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:47.545 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:47.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:47.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:47.593 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:47.647 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:47.669 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:47.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:47.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:47.771 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:47.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:47.796 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:47.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:47.897 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:47.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:47.919 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:47.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:47.986 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:47 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:50:47.986 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:47 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:50:48.020 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:48.043 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:48.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:48.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:48.145 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:48.166 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:48.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:48.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:48.193 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:47 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:50:48.267 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:48.291 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:48.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:48.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:48.393 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:48.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:48.415 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:48.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:48.516 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:48.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:48.539 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:48.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:48.640 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:48.662 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:48.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:48.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:48.763 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:48.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:48.787 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:48.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:48.858 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:48 smithi102 ceph-mon[110183]: pgmap v1287: 93 pgs: 93 active+clean; 4.6 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-13T14:50:48.859 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:48 smithi102 ceph-mon[115269]: pgmap v1287: 93 pgs: 93 active+clean; 4.6 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-13T14:50:48.888 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:48.910 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:48.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:48.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:49.011 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:49.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:49.035 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:49.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:49.047 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:48 smithi195 ceph-mon[104262]: pgmap v1287: 93 pgs: 93 active+clean; 4.6 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-13T14:50:49.136 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:49.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:49.159 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:49.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:49.261 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:49.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:49.284 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:49.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:49.385 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:49.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:49.412 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:49.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:49.413 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:50:49.414 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:50:49.513 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:49.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:49.535 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:49.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:49.636 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:49.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:49.659 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:49.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:49.760 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:49.770 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'unique_pool_4' created 2024-09-13T14:50:49.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:49.787 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:49.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:49.888 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:49.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:49.910 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:49.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:50.000 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:49 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/542357329' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-13T14:50:50.000 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:49 smithi102 ceph-mon[110183]: from='client.46424 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:50:50.000 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:49 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/542357329' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-13T14:50:50.000 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:49 smithi102 ceph-mon[115269]: from='client.46424 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:50:50.007 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:50:50.011 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:50.032 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:50.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:50.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:50.135 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:50.158 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:50.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:50.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:50.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:49 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/542357329' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-13T14:50:50.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:49 smithi195 ceph-mon[104262]: from='client.46424 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:50:50.260 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:50.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:50.283 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:50.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:50.384 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:50.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:50.408 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:50.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:50.509 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:50.530 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:50.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:50.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:50.632 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:50.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:50.658 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:50.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:50.759 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:50.786 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:50.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:50.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:50.888 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:50.909 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:50.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:50.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:51.001 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":1287,"stamp":"2024-09-13T14:50:47.678780+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790245,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":908,"num_write_kb":12625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":195,"num_bytes_recovered":8718480,"num_keys_recovered":20,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4162,"ondisk_log_size":4162,"up":279,"acting":279,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":280,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":4937356,"kb_used_data":71212,"kb_used_omap":41,"kb_used_meta":4866006,"kb_avail":745023860,"statfs":{"total":767960285184,"available":762904432640,"internally_reserved":0,"allocated":72921088,"data_stored":57469262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42913,"internal_metadata":4982790239},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002453"},"pg_stats":[{"pgid":"1.11","version":"90'149","reported_seq":485,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.773704+0000","last_change":"2024-09-13T14:47:38.688630+0000","last_active":"2024-09-13T14:50:35.773704+0000","last_peered":"2024-09-13T14:50:35.773704+0000","last_clean":"2024-09-13T14:50:35.773704+0000","last_became_active":"2024-09-13T14:47:38.460077+0000","last_became_peered":"2024-09-13T14:47:38.460077+0000","last_unstale":"2024-09-13T14:50:35.773704+0000","last_undegraded":"2024-09-13T14:50:35.773704+0000","last_fullsized":"2024-09-13T14:50:35.773704+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:38.688341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.688341+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:14:17.232921+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":655,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.753265+0000","last_change":"2024-09-13T14:47:47.868783+0000","last_active":"2024-09-13T14:50:35.753265+0000","last_peered":"2024-09-13T14:50:35.753265+0000","last_clean":"2024-09-13T14:50:35.753265+0000","last_became_active":"2024-09-13T14:47:47.868478+0000","last_became_peered":"2024-09-13T14:47:47.868478+0000","last_unstale":"2024-09-13T14:50:35.753265+0000","last_undegraded":"2024-09-13T14:50:35.753265+0000","last_fullsized":"2024-09-13T14:50:35.753265+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:27:01.784142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":393,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.749127+0000","last_change":"2024-09-13T14:46:01.772811+0000","last_active":"2024-09-13T14:50:35.749127+0000","last_peered":"2024-09-13T14:50:35.749127+0000","last_clean":"2024-09-13T14:50:35.749127+0000","last_became_active":"2024-09-13T14:46:01.764775+0000","last_became_peered":"2024-09-13T14:46:01.764775+0000","last_unstale":"2024-09-13T14:50:35.749127+0000","last_undegraded":"2024-09-13T14:50:35.749127+0000","last_fullsized":"2024-09-13T14:50:35.749127+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:42:20.738421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"90'149","reported_seq":438,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.749289+0000","last_change":"2024-09-13T14:47:41.482552+0000","last_active":"2024-09-13T14:50:35.749289+0000","last_peered":"2024-09-13T14:50:35.749289+0000","last_clean":"2024-09-13T14:50:35.749289+0000","last_became_active":"2024-09-13T14:47:38.466659+0000","last_became_peered":"2024-09-13T14:47:38.466659+0000","last_unstale":"2024-09-13T14:50:35.749289+0000","last_undegraded":"2024-09-13T14:50:35.749289+0000","last_fullsized":"2024-09-13T14:50:35.749289+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:41.482364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:41.482364+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:50:49.239389+0000","scrub_duration":0.027843855000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":175,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:41.474256+0000","last_change":"2024-09-13T14:36:13.862181+0000","last_active":"2024-09-13T14:50:41.474256+0000","last_peered":"2024-09-13T14:50:41.474256+0000","last_clean":"2024-09-13T14:50:41.474256+0000","last_became_active":"2024-09-13T14:36:13.861873+0000","last_became_peered":"2024-09-13T14:36:13.861873+0000","last_unstale":"2024-09-13T14:50:41.474256+0000","last_undegraded":"2024-09-13T14:50:41.474256+0000","last_fullsized":"2024-09-13T14:50:41.474256+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:14.085700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":278,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.752378+0000","last_change":"2024-09-13T14:38:53.896672+0000","last_active":"2024-09-13T14:50:35.752378+0000","last_peered":"2024-09-13T14:50:35.752378+0000","last_clean":"2024-09-13T14:50:35.752378+0000","last_became_active":"2024-09-13T14:38:53.896346+0000","last_became_peered":"2024-09-13T14:38:53.896346+0000","last_unstale":"2024-09-13T14:50:35.752378+0000","last_undegraded":"2024-09-13T14:50:35.752378+0000","last_fullsized":"2024-09-13T14:50:35.752378+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:57.789686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"90'149","reported_seq":507,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-13T14:50:43.129908+0000","last_change":"2024-09-13T14:47:51.226821+0000","last_active":"2024-09-13T14:50:43.129908+0000","last_peered":"2024-09-13T14:50:43.129908+0000","last_clean":"2024-09-13T14:50:43.129908+0000","last_became_active":"2024-09-13T14:47:50.881689+0000","last_became_peered":"2024-09-13T14:47:50.881689+0000","last_unstale":"2024-09-13T14:50:43.129908+0000","last_undegraded":"2024-09-13T14:50:43.129908+0000","last_fullsized":"2024-09-13T14:50:43.129908+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:51.226748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:51.226748+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:06:02.065990+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":278,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.818657+0000","last_change":"2024-09-13T14:38:53.893021+0000","last_active":"2024-09-13T14:50:35.818657+0000","last_peered":"2024-09-13T14:50:35.818657+0000","last_clean":"2024-09-13T14:50:35.818657+0000","last_became_active":"2024-09-13T14:38:53.892668+0000","last_became_peered":"2024-09-13T14:38:53.892668+0000","last_unstale":"2024-09-13T14:50:35.818657+0000","last_undegraded":"2024-09-13T14:50:35.818657+0000","last_fullsized":"2024-09-13T14:50:35.818657+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:57.180738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":346,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.749214+0000","last_change":"2024-09-13T14:37:48.134642+0000","last_active":"2024-09-13T14:50:35.749214+0000","last_peered":"2024-09-13T14:50:35.749214+0000","last_clean":"2024-09-13T14:50:35.749214+0000","last_became_active":"2024-09-13T14:37:48.134402+0000","last_became_peered":"2024-09-13T14:37:48.134402+0000","last_unstale":"2024-09-13T14:50:35.749214+0000","last_undegraded":"2024-09-13T14:50:35.749214+0000","last_fullsized":"2024-09-13T14:50:35.749214+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:51.313663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"90'149","reported_seq":455,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.759583+0000","last_change":"2024-09-13T14:47:50.892667+0000","last_active":"2024-09-13T14:50:35.759583+0000","last_peered":"2024-09-13T14:50:35.759583+0000","last_clean":"2024-09-13T14:50:35.759583+0000","last_became_active":"2024-09-13T14:47:50.883312+0000","last_became_peered":"2024-09-13T14:47:50.883312+0000","last_unstale":"2024-09-13T14:50:35.759583+0000","last_undegraded":"2024-09-13T14:50:35.759583+0000","last_fullsized":"2024-09-13T14:50:35.759583+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:45.441819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:45.441819+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:39.138573+0000","scrub_duration":0.021368558999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":680,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.754061+0000","last_change":"2024-09-13T14:50:27.491166+0000","last_active":"2024-09-13T14:50:35.754061+0000","last_peered":"2024-09-13T14:50:35.754061+0000","last_clean":"2024-09-13T14:50:35.754061+0000","last_became_active":"2024-09-13T14:50:27.489905+0000","last_became_peered":"2024-09-13T14:50:27.489905+0000","last_unstale":"2024-09-13T14:50:35.754061+0000","last_undegraded":"2024-09-13T14:50:35.754061+0000","last_fullsized":"2024-09-13T14:50:35.754061+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:32.237904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":281,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.751220+0000","last_change":"2024-09-13T14:38:03.554500+0000","last_active":"2024-09-13T14:50:35.751220+0000","last_peered":"2024-09-13T14:50:35.751220+0000","last_clean":"2024-09-13T14:50:35.751220+0000","last_became_active":"2024-09-13T14:38:03.532910+0000","last_became_peered":"2024-09-13T14:38:03.532910+0000","last_unstale":"2024-09-13T14:50:35.751220+0000","last_undegraded":"2024-09-13T14:50:35.751220+0000","last_fullsized":"2024-09-13T14:50:35.751220+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:27:50.843498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"90'149","reported_seq":457,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.749406+0000","last_change":"2024-09-13T14:47:50.887140+0000","last_active":"2024-09-13T14:50:35.749406+0000","last_peered":"2024-09-13T14:50:35.749406+0000","last_clean":"2024-09-13T14:50:35.749406+0000","last_became_active":"2024-09-13T14:47:50.886603+0000","last_became_peered":"2024-09-13T14:47:50.886603+0000","last_unstale":"2024-09-13T14:50:35.749406+0000","last_undegraded":"2024-09-13T14:50:35.749406+0000","last_fullsized":"2024-09-13T14:50:35.749406+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:40.786610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:40.786610+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:00.936651+0000","scrub_duration":0.013725185000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":139,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.750288+0000","last_change":"2024-09-13T14:38:53.940238+0000","last_active":"2024-09-13T14:50:35.750288+0000","last_peered":"2024-09-13T14:50:35.750288+0000","last_clean":"2024-09-13T14:50:35.750288+0000","last_became_active":"2024-09-13T14:38:53.892825+0000","last_became_peered":"2024-09-13T14:38:53.892825+0000","last_unstale":"2024-09-13T14:50:35.750288+0000","last_undegraded":"2024-09-13T14:50:35.750288+0000","last_fullsized":"2024-09-13T14:50:35.750288+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:46.666581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":195,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.754545+0000","last_change":"2024-09-13T14:38:53.898907+0000","last_active":"2024-09-13T14:50:35.754545+0000","last_peered":"2024-09-13T14:50:35.754545+0000","last_clean":"2024-09-13T14:50:35.754545+0000","last_became_active":"2024-09-13T14:38:53.898572+0000","last_became_peered":"2024-09-13T14:38:53.898572+0000","last_unstale":"2024-09-13T14:50:35.754545+0000","last_undegraded":"2024-09-13T14:50:35.754545+0000","last_fullsized":"2024-09-13T14:50:35.754545+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:29.222150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"90'149","reported_seq":527,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.753099+0000","last_change":"2024-09-13T14:47:38.466101+0000","last_active":"2024-09-13T14:50:35.753099+0000","last_peered":"2024-09-13T14:50:35.753099+0000","last_clean":"2024-09-13T14:50:35.753099+0000","last_became_active":"2024-09-13T14:47:38.465607+0000","last_became_peered":"2024-09-13T14:47:38.465607+0000","last_unstale":"2024-09-13T14:50:35.753099+0000","last_undegraded":"2024-09-13T14:50:35.753099+0000","last_fullsized":"2024-09-13T14:50:35.753099+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:32.060817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:32.060817+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:55.362724+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":152,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.750703+0000","last_change":"2024-09-13T14:47:47.878423+0000","last_active":"2024-09-13T14:50:35.750703+0000","last_peered":"2024-09-13T14:50:35.750703+0000","last_clean":"2024-09-13T14:50:35.750703+0000","last_became_active":"2024-09-13T14:47:47.866926+0000","last_became_peered":"2024-09-13T14:47:47.866926+0000","last_unstale":"2024-09-13T14:50:35.750703+0000","last_undegraded":"2024-09-13T14:50:35.750703+0000","last_fullsized":"2024-09-13T14:50:35.750703+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:11:07.534097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":394,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.274788+0000","last_change":"2024-09-13T14:47:47.860804+0000","last_active":"2024-09-13T14:50:36.274788+0000","last_peered":"2024-09-13T14:50:36.274788+0000","last_clean":"2024-09-13T14:50:36.274788+0000","last_became_active":"2024-09-13T14:47:47.860496+0000","last_became_peered":"2024-09-13T14:47:47.860496+0000","last_unstale":"2024-09-13T14:50:36.274788+0000","last_undegraded":"2024-09-13T14:50:36.274788+0000","last_fullsized":"2024-09-13T14:50:36.274788+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:17:57.542599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.f","version":"90'149","reported_seq":482,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.750549+0000","last_change":"2024-09-13T14:47:38.468287+0000","last_active":"2024-09-13T14:50:35.750549+0000","last_peered":"2024-09-13T14:50:35.750549+0000","last_clean":"2024-09-13T14:50:35.750549+0000","last_became_active":"2024-09-13T14:47:38.467941+0000","last_became_peered":"2024-09-13T14:47:38.467941+0000","last_unstale":"2024-09-13T14:50:35.750549+0000","last_undegraded":"2024-09-13T14:50:35.750549+0000","last_fullsized":"2024-09-13T14:50:35.750549+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:34.091782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:34.091782+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:34:20.519248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":376,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.751630+0000","last_change":"2024-09-13T14:37:49.611774+0000","last_active":"2024-09-13T14:50:35.751630+0000","last_peered":"2024-09-13T14:50:35.751630+0000","last_clean":"2024-09-13T14:50:35.751630+0000","last_became_active":"2024-09-13T14:37:49.611088+0000","last_became_peered":"2024-09-13T14:37:49.611088+0000","last_unstale":"2024-09-13T14:50:35.751630+0000","last_undegraded":"2024-09-13T14:50:35.751630+0000","last_fullsized":"2024-09-13T14:50:35.751630+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:08.933213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":85,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-13T14:50:43.130952+0000","last_change":"2024-09-13T14:47:47.865523+0000","last_active":"2024-09-13T14:50:43.130952+0000","last_peered":"2024-09-13T14:50:43.130952+0000","last_clean":"2024-09-13T14:50:43.130952+0000","last_became_active":"2024-09-13T14:47:47.865180+0000","last_became_peered":"2024-09-13T14:47:47.865180+0000","last_unstale":"2024-09-13T14:50:43.130952+0000","last_undegraded":"2024-09-13T14:50:43.130952+0000","last_fullsized":"2024-09-13T14:50:43.130952+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:39:35.479251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.e","version":"90'149","reported_seq":488,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.738583+0000","last_change":"2024-09-13T14:47:38.459769+0000","last_active":"2024-09-13T14:50:35.738583+0000","last_peered":"2024-09-13T14:50:35.738583+0000","last_clean":"2024-09-13T14:50:35.738583+0000","last_became_active":"2024-09-13T14:47:38.459440+0000","last_became_peered":"2024-09-13T14:47:38.459440+0000","last_unstale":"2024-09-13T14:50:35.738583+0000","last_undegraded":"2024-09-13T14:50:35.738583+0000","last_fullsized":"2024-09-13T14:50:35.738583+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:30.526999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:30.526999+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:00.016329+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":186,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.759742+0000","last_change":"2024-09-13T14:37:49.168003+0000","last_active":"2024-09-13T14:50:35.759742+0000","last_peered":"2024-09-13T14:50:35.759742+0000","last_clean":"2024-09-13T14:50:35.759742+0000","last_became_active":"2024-09-13T14:37:49.167638+0000","last_became_peered":"2024-09-13T14:37:49.167638+0000","last_unstale":"2024-09-13T14:50:35.759742+0000","last_undegraded":"2024-09-13T14:50:35.759742+0000","last_fullsized":"2024-09-13T14:50:35.759742+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:29.051981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":278,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.751711+0000","last_change":"2024-09-13T14:47:47.868094+0000","last_active":"2024-09-13T14:50:35.751711+0000","last_peered":"2024-09-13T14:50:35.751711+0000","last_clean":"2024-09-13T14:50:35.751711+0000","last_became_active":"2024-09-13T14:47:47.867750+0000","last_became_peered":"2024-09-13T14:47:47.867750+0000","last_unstale":"2024-09-13T14:50:35.751711+0000","last_undegraded":"2024-09-13T14:50:35.751711+0000","last_fullsized":"2024-09-13T14:50:35.751711+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:55:38.203481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"90'149","reported_seq":548,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.749039+0000","last_change":"2024-09-13T14:46:55.355650+0000","last_active":"2024-09-13T14:50:35.749039+0000","last_peered":"2024-09-13T14:50:35.749039+0000","last_clean":"2024-09-13T14:50:35.749039+0000","last_became_active":"2024-09-13T14:46:48.150338+0000","last_became_peered":"2024-09-13T14:46:48.150338+0000","last_unstale":"2024-09-13T14:50:35.749039+0000","last_undegraded":"2024-09-13T14:50:35.749039+0000","last_fullsized":"2024-09-13T14:50:35.749039+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:46:55.355521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:46:55.355521+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:30:53.006278+0000","scrub_duration":0.002025448,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":629,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.752980+0000","last_change":"2024-09-13T14:38:03.549595+0000","last_active":"2024-09-13T14:50:35.752980+0000","last_peered":"2024-09-13T14:50:35.752980+0000","last_clean":"2024-09-13T14:50:35.752980+0000","last_became_active":"2024-09-13T14:38:03.533331+0000","last_became_peered":"2024-09-13T14:38:03.533331+0000","last_unstale":"2024-09-13T14:50:35.752980+0000","last_undegraded":"2024-09-13T14:50:35.752980+0000","last_fullsized":"2024-09-13T14:50:35.752980+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:59:02.470343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":369,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.738737+0000","last_change":"2024-09-13T14:46:26.980367+0000","last_active":"2024-09-13T14:50:35.738737+0000","last_peered":"2024-09-13T14:50:35.738737+0000","last_clean":"2024-09-13T14:50:35.738737+0000","last_became_active":"2024-09-13T14:46:26.979937+0000","last_became_peered":"2024-09-13T14:46:26.979937+0000","last_unstale":"2024-09-13T14:50:35.738737+0000","last_undegraded":"2024-09-13T14:50:35.738737+0000","last_fullsized":"2024-09-13T14:50:35.738737+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:56.830872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"90'149","reported_seq":450,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.752441+0000","last_change":"2024-09-13T14:46:54.393663+0000","last_active":"2024-09-13T14:50:35.752441+0000","last_peered":"2024-09-13T14:50:35.752441+0000","last_clean":"2024-09-13T14:50:35.752441+0000","last_became_active":"2024-09-13T14:46:48.149004+0000","last_became_peered":"2024-09-13T14:46:48.149004+0000","last_unstale":"2024-09-13T14:50:35.752441+0000","last_undegraded":"2024-09-13T14:50:35.752441+0000","last_fullsized":"2024-09-13T14:50:35.752441+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:46:54.393204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:46:54.393204+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:32:40.003989+0000","scrub_duration":0.015172223,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":302,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.738380+0000","last_change":"2024-09-13T14:47:47.867571+0000","last_active":"2024-09-13T14:50:35.738380+0000","last_peered":"2024-09-13T14:50:35.738380+0000","last_clean":"2024-09-13T14:50:35.738380+0000","last_became_active":"2024-09-13T14:47:47.867269+0000","last_became_peered":"2024-09-13T14:47:47.867269+0000","last_unstale":"2024-09-13T14:50:35.738380+0000","last_undegraded":"2024-09-13T14:50:35.738380+0000","last_fullsized":"2024-09-13T14:50:35.738380+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:46.093600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":431,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.739597+0000","last_change":"2024-09-13T14:47:47.866925+0000","last_active":"2024-09-13T14:50:35.739597+0000","last_peered":"2024-09-13T14:50:35.739597+0000","last_clean":"2024-09-13T14:50:35.739597+0000","last_became_active":"2024-09-13T14:47:47.866596+0000","last_became_peered":"2024-09-13T14:47:47.866596+0000","last_unstale":"2024-09-13T14:50:35.739597+0000","last_undegraded":"2024-09-13T14:50:35.739597+0000","last_fullsized":"2024-09-13T14:50:35.739597+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:11:25.617891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"90'149","reported_seq":520,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-13T14:50:43.129585+0000","last_change":"2024-09-13T14:47:50.891055+0000","last_active":"2024-09-13T14:50:43.129585+0000","last_peered":"2024-09-13T14:50:43.129585+0000","last_clean":"2024-09-13T14:50:43.129585+0000","last_became_active":"2024-09-13T14:47:50.890368+0000","last_became_peered":"2024-09-13T14:47:50.890368+0000","last_unstale":"2024-09-13T14:50:43.129585+0000","last_undegraded":"2024-09-13T14:50:43.129585+0000","last_fullsized":"2024-09-13T14:50:43.129585+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:39.645908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:39.645908+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:29.800728+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":383,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:36.274840+0000","last_change":"2024-09-13T14:46:01.762127+0000","last_active":"2024-09-13T14:50:36.274840+0000","last_peered":"2024-09-13T14:50:36.274840+0000","last_clean":"2024-09-13T14:50:36.274840+0000","last_became_active":"2024-09-13T14:46:01.761795+0000","last_became_peered":"2024-09-13T14:46:01.761795+0000","last_unstale":"2024-09-13T14:50:36.274840+0000","last_undegraded":"2024-09-13T14:50:36.274840+0000","last_fullsized":"2024-09-13T14:50:36.274840+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:41:15.626775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":378,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.738719+0000","last_change":"2024-09-13T14:47:47.861718+0000","last_active":"2024-09-13T14:50:35.738719+0000","last_peered":"2024-09-13T14:50:35.738719+0000","last_clean":"2024-09-13T14:50:35.738719+0000","last_became_active":"2024-09-13T14:47:47.861023+0000","last_became_peered":"2024-09-13T14:47:47.861023+0000","last_unstale":"2024-09-13T14:50:35.738719+0000","last_undegraded":"2024-09-13T14:50:35.738719+0000","last_fullsized":"2024-09-13T14:50:35.738719+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:12:13.274053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"90'149","reported_seq":450,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.758488+0000","last_change":"2024-09-13T14:46:52.334715+0000","last_active":"2024-09-13T14:50:35.758488+0000","last_peered":"2024-09-13T14:50:35.758488+0000","last_clean":"2024-09-13T14:50:35.758488+0000","last_became_active":"2024-09-13T14:46:48.158685+0000","last_became_peered":"2024-09-13T14:46:48.158685+0000","last_unstale":"2024-09-13T14:50:35.758488+0000","last_undegraded":"2024-09-13T14:50:35.758488+0000","last_fullsized":"2024-09-13T14:50:35.758488+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:46:52.334301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:46:52.334301+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:46:03.579759+0000","scrub_duration":0.0050940330000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":654,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.754574+0000","last_change":"2024-09-13T14:47:47.861249+0000","last_active":"2024-09-13T14:50:35.754574+0000","last_peered":"2024-09-13T14:50:35.754574+0000","last_clean":"2024-09-13T14:50:35.754574+0000","last_became_active":"2024-09-13T14:47:47.860873+0000","last_became_peered":"2024-09-13T14:47:47.860873+0000","last_unstale":"2024-09-13T14:50:35.754574+0000","last_undegraded":"2024-09-13T14:50:35.754574+0000","last_fullsized":"2024-09-13T14:50:35.754574+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:37:50.709437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":85,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-13T14:50:43.129721+0000","last_change":"2024-09-13T14:47:47.870185+0000","last_active":"2024-09-13T14:50:43.129721+0000","last_peered":"2024-09-13T14:50:43.129721+0000","last_clean":"2024-09-13T14:50:43.129721+0000","last_became_active":"2024-09-13T14:47:47.869862+0000","last_became_peered":"2024-09-13T14:47:47.869862+0000","last_unstale":"2024-09-13T14:50:43.129721+0000","last_undegraded":"2024-09-13T14:50:43.129721+0000","last_fullsized":"2024-09-13T14:50:43.129721+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:07:02.812306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":553,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.750880+0000","last_change":"2024-09-13T14:47:31.684319+0000","last_active":"2024-09-13T14:50:35.750880+0000","last_peered":"2024-09-13T14:50:35.750880+0000","last_clean":"2024-09-13T14:50:35.750880+0000","last_became_active":"2024-09-13T14:47:29.415658+0000","last_became_peered":"2024-09-13T14:47:29.415658+0000","last_unstale":"2024-09-13T14:50:35.750880+0000","last_undegraded":"2024-09-13T14:50:35.750880+0000","last_fullsized":"2024-09-13T14:50:35.750880+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:31.681433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:31.681433+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:29:34.663513+0000","scrub_duration":0.0059601139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":681,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.751307+0000","last_change":"2024-09-13T14:46:04.780718+0000","last_active":"2024-09-13T14:50:35.751307+0000","last_peered":"2024-09-13T14:50:35.751307+0000","last_clean":"2024-09-13T14:50:35.751307+0000","last_became_active":"2024-09-13T14:46:04.780120+0000","last_became_peered":"2024-09-13T14:46:04.780120+0000","last_unstale":"2024-09-13T14:50:35.751307+0000","last_undegraded":"2024-09-13T14:50:35.751307+0000","last_fullsized":"2024-09-13T14:50:35.751307+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:57:14.475888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":582,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.752311+0000","last_change":"2024-09-13T14:37:49.160719+0000","last_active":"2024-09-13T14:50:35.752311+0000","last_peered":"2024-09-13T14:50:35.752311+0000","last_clean":"2024-09-13T14:50:35.752311+0000","last_became_active":"2024-09-13T14:37:49.160401+0000","last_became_peered":"2024-09-13T14:37:49.160401+0000","last_unstale":"2024-09-13T14:50:35.752311+0000","last_undegraded":"2024-09-13T14:50:35.752311+0000","last_fullsized":"2024-09-13T14:50:35.752311+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:21.737945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":221,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:41.275684+0000","last_change":"2024-09-13T14:37:48.172940+0000","last_active":"2024-09-13T14:50:41.275684+0000","last_peered":"2024-09-13T14:50:41.275684+0000","last_clean":"2024-09-13T14:50:41.275684+0000","last_became_active":"2024-09-13T14:37:48.172060+0000","last_became_peered":"2024-09-13T14:37:48.172060+0000","last_unstale":"2024-09-13T14:50:41.275684+0000","last_undegraded":"2024-09-13T14:50:41.275684+0000","last_fullsized":"2024-09-13T14:50:41.275684+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:07.480645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"90'149","reported_seq":514,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-13T14:50:43.129746+0000","last_change":"2024-09-13T14:47:50.884721+0000","last_active":"2024-09-13T14:50:43.129746+0000","last_peered":"2024-09-13T14:50:43.129746+0000","last_clean":"2024-09-13T14:50:43.129746+0000","last_became_active":"2024-09-13T14:47:50.880185+0000","last_became_peered":"2024-09-13T14:47:50.880185+0000","last_unstale":"2024-09-13T14:50:43.129746+0000","last_undegraded":"2024-09-13T14:50:43.129746+0000","last_fullsized":"2024-09-13T14:50:43.129746+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:34.603316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:34.603316+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:38.321686+0000","scrub_duration":0.0057131120000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":623,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.750795+0000","last_change":"2024-09-13T14:47:47.892973+0000","last_active":"2024-09-13T14:50:35.750795+0000","last_peered":"2024-09-13T14:50:35.750795+0000","last_clean":"2024-09-13T14:50:35.750795+0000","last_became_active":"2024-09-13T14:47:47.866986+0000","last_became_peered":"2024-09-13T14:47:47.866986+0000","last_unstale":"2024-09-13T14:50:35.750795+0000","last_undegraded":"2024-09-13T14:50:35.750795+0000","last_fullsized":"2024-09-13T14:50:35.750795+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:07.353290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":272,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.773651+0000","last_change":"2024-09-13T14:38:03.525682+0000","last_active":"2024-09-13T14:50:35.773651+0000","last_peered":"2024-09-13T14:50:35.773651+0000","last_clean":"2024-09-13T14:50:35.773651+0000","last_became_active":"2024-09-13T14:38:03.525406+0000","last_became_peered":"2024-09-13T14:38:03.525406+0000","last_unstale":"2024-09-13T14:50:35.773651+0000","last_undegraded":"2024-09-13T14:50:35.773651+0000","last_fullsized":"2024-09-13T14:50:35.773651+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:41:28.970300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":292,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.819033+0000","last_change":"2024-09-13T14:47:47.873233+0000","last_active":"2024-09-13T14:50:35.819033+0000","last_peered":"2024-09-13T14:50:35.819033+0000","last_clean":"2024-09-13T14:50:35.819033+0000","last_became_active":"2024-09-13T14:47:47.872788+0000","last_became_peered":"2024-09-13T14:47:47.872788+0000","last_unstale":"2024-09-13T14:50:35.819033+0000","last_undegraded":"2024-09-13T14:50:35.819033+0000","last_fullsized":"2024-09-13T14:50:35.819033+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:52.409108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"90'149","reported_seq":569,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.749214+0000","last_change":"2024-09-13T14:47:29.629488+0000","last_active":"2024-09-13T14:50:35.749214+0000","last_peered":"2024-09-13T14:50:35.749214+0000","last_clean":"2024-09-13T14:50:35.749214+0000","last_became_active":"2024-09-13T14:47:29.403279+0000","last_became_peered":"2024-09-13T14:47:29.403279+0000","last_unstale":"2024-09-13T14:50:35.749214+0000","last_undegraded":"2024-09-13T14:50:35.749214+0000","last_fullsized":"2024-09-13T14:50:35.749214+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:29.629342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:29.629342+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:14:55.249896+0000","scrub_duration":0.0099290990000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":568,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-13T14:50:43.130652+0000","last_change":"2024-09-13T14:46:04.782955+0000","last_active":"2024-09-13T14:50:43.130652+0000","last_peered":"2024-09-13T14:50:43.130652+0000","last_clean":"2024-09-13T14:50:43.130652+0000","last_became_active":"2024-09-13T14:46:04.782250+0000","last_became_peered":"2024-09-13T14:46:04.782250+0000","last_unstale":"2024-09-13T14:50:43.130652+0000","last_undegraded":"2024-09-13T14:50:43.130652+0000","last_fullsized":"2024-09-13T14:50:43.130652+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:24.596922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":85,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-13T14:50:43.129782+0000","last_change":"2024-09-13T14:47:47.866684+0000","last_active":"2024-09-13T14:50:43.129782+0000","last_peered":"2024-09-13T14:50:43.129782+0000","last_clean":"2024-09-13T14:50:43.129782+0000","last_became_active":"2024-09-13T14:47:47.866400+0000","last_became_peered":"2024-09-13T14:47:47.866400+0000","last_unstale":"2024-09-13T14:50:43.129782+0000","last_undegraded":"2024-09-13T14:50:43.129782+0000","last_fullsized":"2024-09-13T14:50:43.129782+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:08:34.289658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":85,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-13T14:50:43.130181+0000","last_change":"2024-09-13T14:47:47.870218+0000","last_active":"2024-09-13T14:50:43.130181+0000","last_peered":"2024-09-13T14:50:43.130181+0000","last_clean":"2024-09-13T14:50:43.130181+0000","last_became_active":"2024-09-13T14:47:47.869907+0000","last_became_peered":"2024-09-13T14:47:47.869907+0000","last_unstale":"2024-09-13T14:50:43.130181+0000","last_undegraded":"2024-09-13T14:50:43.130181+0000","last_fullsized":"2024-09-13T14:50:43.130181+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:41:49.141441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"90'149","reported_seq":471,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.755839+0000","last_change":"2024-09-13T14:47:32.275753+0000","last_active":"2024-09-13T14:50:35.755839+0000","last_peered":"2024-09-13T14:50:35.755839+0000","last_clean":"2024-09-13T14:50:35.755839+0000","last_became_active":"2024-09-13T14:47:29.401066+0000","last_became_peered":"2024-09-13T14:47:29.401066+0000","last_unstale":"2024-09-13T14:50:35.755839+0000","last_undegraded":"2024-09-13T14:50:35.755839+0000","last_fullsized":"2024-09-13T14:50:35.755839+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:32.275562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:32.275562+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:43:41.056775+0000","scrub_duration":0.0054674260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":1126,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.759442+0000","last_change":"2024-09-13T14:47:50.892842+0000","last_active":"2024-09-13T14:50:35.759442+0000","last_peered":"2024-09-13T14:50:35.759442+0000","last_clean":"2024-09-13T14:50:35.759442+0000","last_became_active":"2024-09-13T14:47:50.892237+0000","last_became_peered":"2024-09-13T14:47:50.892237+0000","last_unstale":"2024-09-13T14:50:35.759442+0000","last_undegraded":"2024-09-13T14:50:35.759442+0000","last_fullsized":"2024-09-13T14:50:35.759442+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:20:05.796762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":8388646,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":382,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.740073+0000","last_change":"2024-09-13T14:38:03.528133+0000","last_active":"2024-09-13T14:50:35.740073+0000","last_peered":"2024-09-13T14:50:35.740073+0000","last_clean":"2024-09-13T14:50:35.740073+0000","last_became_active":"2024-09-13T14:38:03.527829+0000","last_became_peered":"2024-09-13T14:38:03.527829+0000","last_unstale":"2024-09-13T14:50:35.740073+0000","last_undegraded":"2024-09-13T14:50:35.740073+0000","last_fullsized":"2024-09-13T14:50:35.740073+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:48.254169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":85,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-13T14:50:43.129638+0000","last_change":"2024-09-13T14:47:47.877272+0000","last_active":"2024-09-13T14:50:43.129638+0000","last_peered":"2024-09-13T14:50:43.129638+0000","last_clean":"2024-09-13T14:50:43.129638+0000","last_became_active":"2024-09-13T14:47:47.877003+0000","last_became_peered":"2024-09-13T14:47:47.877003+0000","last_unstale":"2024-09-13T14:50:43.129638+0000","last_undegraded":"2024-09-13T14:50:43.129638+0000","last_fullsized":"2024-09-13T14:50:43.129638+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:47:23.172452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"90'149","reported_seq":515,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.753971+0000","last_change":"2024-09-13T14:47:33.067298+0000","last_active":"2024-09-13T14:50:35.753971+0000","last_peered":"2024-09-13T14:50:35.753971+0000","last_clean":"2024-09-13T14:50:35.753971+0000","last_became_active":"2024-09-13T14:47:29.419416+0000","last_became_peered":"2024-09-13T14:47:29.419416+0000","last_unstale":"2024-09-13T14:50:35.753971+0000","last_undegraded":"2024-09-13T14:50:35.753971+0000","last_fullsized":"2024-09-13T14:50:35.753971+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:33.066954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:33.066954+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:05:42.808060+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":195,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.759856+0000","last_change":"2024-09-13T14:46:01.763906+0000","last_active":"2024-09-13T14:50:35.759856+0000","last_peered":"2024-09-13T14:50:35.759856+0000","last_clean":"2024-09-13T14:50:35.759856+0000","last_became_active":"2024-09-13T14:46:01.763542+0000","last_became_peered":"2024-09-13T14:46:01.763542+0000","last_unstale":"2024-09-13T14:50:35.759856+0000","last_undegraded":"2024-09-13T14:50:35.759856+0000","last_fullsized":"2024-09-13T14:50:35.759856+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:09:36.652767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":1373,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-13T14:50:43.129742+0000","last_change":"2024-09-13T14:47:50.883365+0000","last_active":"2024-09-13T14:50:43.129742+0000","last_peered":"2024-09-13T14:50:43.129742+0000","last_clean":"2024-09-13T14:50:43.129742+0000","last_became_active":"2024-09-13T14:47:50.882609+0000","last_became_peered":"2024-09-13T14:47:50.882609+0000","last_unstale":"2024-09-13T14:50:43.129742+0000","last_undegraded":"2024-09-13T14:50:43.129742+0000","last_fullsized":"2024-09-13T14:50:43.129742+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:12:17.961575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":85,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-13T14:50:43.131093+0000","last_change":"2024-09-13T14:47:47.866979+0000","last_active":"2024-09-13T14:50:43.131093+0000","last_peered":"2024-09-13T14:50:43.131093+0000","last_clean":"2024-09-13T14:50:43.131093+0000","last_became_active":"2024-09-13T14:47:47.866634+0000","last_became_peered":"2024-09-13T14:47:47.866634+0000","last_unstale":"2024-09-13T14:50:43.131093+0000","last_undegraded":"2024-09-13T14:50:43.131093+0000","last_fullsized":"2024-09-13T14:50:43.131093+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:47:09.696139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"90'149","reported_seq":521,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.776935+0000","last_change":"2024-09-13T14:47:29.556225+0000","last_active":"2024-09-13T14:50:35.776935+0000","last_peered":"2024-09-13T14:50:35.776935+0000","last_clean":"2024-09-13T14:50:35.776935+0000","last_became_active":"2024-09-13T14:47:29.408774+0000","last_became_peered":"2024-09-13T14:47:29.408774+0000","last_unstale":"2024-09-13T14:50:35.776935+0000","last_undegraded":"2024-09-13T14:50:35.776935+0000","last_fullsized":"2024-09-13T14:50:35.776935+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:29.556149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:29.556149+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:35.691178+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":85,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-13T14:50:43.129848+0000","last_change":"2024-09-13T14:47:47.876562+0000","last_active":"2024-09-13T14:50:43.129848+0000","last_peered":"2024-09-13T14:50:43.129848+0000","last_clean":"2024-09-13T14:50:43.129848+0000","last_became_active":"2024-09-13T14:47:47.876297+0000","last_became_peered":"2024-09-13T14:47:47.876297+0000","last_unstale":"2024-09-13T14:50:43.129848+0000","last_undegraded":"2024-09-13T14:50:43.129848+0000","last_fullsized":"2024-09-13T14:50:43.129848+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:03:27.761394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":898,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.751712+0000","last_change":"2024-09-13T14:38:04.596192+0000","last_active":"2024-09-13T14:50:35.751712+0000","last_peered":"2024-09-13T14:50:35.751712+0000","last_clean":"2024-09-13T14:50:35.751712+0000","last_became_active":"2024-09-13T14:38:04.593299+0000","last_became_peered":"2024-09-13T14:38:04.593299+0000","last_unstale":"2024-09-13T14:50:35.751712+0000","last_undegraded":"2024-09-13T14:50:35.751712+0000","last_fullsized":"2024-09-13T14:50:35.751712+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:34.241410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":271,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.751788+0000","last_change":"2024-09-13T14:38:03.533120+0000","last_active":"2024-09-13T14:50:35.751788+0000","last_peered":"2024-09-13T14:50:35.751788+0000","last_clean":"2024-09-13T14:50:35.751788+0000","last_became_active":"2024-09-13T14:38:03.532829+0000","last_became_peered":"2024-09-13T14:38:03.532829+0000","last_unstale":"2024-09-13T14:50:35.751788+0000","last_undegraded":"2024-09-13T14:50:35.751788+0000","last_fullsized":"2024-09-13T14:50:35.751788+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:24.514661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"90'149","reported_seq":471,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.739806+0000","last_change":"2024-09-13T14:47:30.528588+0000","last_active":"2024-09-13T14:50:35.739806+0000","last_peered":"2024-09-13T14:50:35.739806+0000","last_clean":"2024-09-13T14:50:35.739806+0000","last_became_active":"2024-09-13T14:47:29.413621+0000","last_became_peered":"2024-09-13T14:47:29.413621+0000","last_unstale":"2024-09-13T14:50:35.739806+0000","last_undegraded":"2024-09-13T14:50:35.739806+0000","last_fullsized":"2024-09-13T14:50:35.739806+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:30.528112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:30.528112+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:30.462009+0000","scrub_duration":0.012751485,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":673,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.755064+0000","last_change":"2024-09-13T14:38:53.938227+0000","last_active":"2024-09-13T14:50:35.755064+0000","last_peered":"2024-09-13T14:50:35.755064+0000","last_clean":"2024-09-13T14:50:35.755064+0000","last_became_active":"2024-09-13T14:38:53.902867+0000","last_became_peered":"2024-09-13T14:38:53.902867+0000","last_unstale":"2024-09-13T14:50:35.755064+0000","last_undegraded":"2024-09-13T14:50:35.755064+0000","last_fullsized":"2024-09-13T14:50:35.755064+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:16.831756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":555,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-13T14:50:43.129981+0000","last_change":"2024-09-13T14:47:50.884706+0000","last_active":"2024-09-13T14:50:43.129981+0000","last_peered":"2024-09-13T14:50:43.129981+0000","last_clean":"2024-09-13T14:50:43.129981+0000","last_became_active":"2024-09-13T14:47:50.879750+0000","last_became_peered":"2024-09-13T14:47:50.879750+0000","last_unstale":"2024-09-13T14:50:43.129981+0000","last_undegraded":"2024-09-13T14:50:43.129981+0000","last_fullsized":"2024-09-13T14:50:43.129981+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:34:32.941266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":655,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.754329+0000","last_change":"2024-09-13T14:38:53.900107+0000","last_active":"2024-09-13T14:50:35.754329+0000","last_peered":"2024-09-13T14:50:35.754329+0000","last_clean":"2024-09-13T14:50:35.754329+0000","last_became_active":"2024-09-13T14:38:53.899556+0000","last_became_peered":"2024-09-13T14:38:53.899556+0000","last_unstale":"2024-09-13T14:50:35.754329+0000","last_undegraded":"2024-09-13T14:50:35.754329+0000","last_fullsized":"2024-09-13T14:50:35.754329+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:15:47.918163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"90'149","reported_seq":512,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.752252+0000","last_change":"2024-09-13T14:47:29.999643+0000","last_active":"2024-09-13T14:50:35.752252+0000","last_peered":"2024-09-13T14:50:35.752252+0000","last_clean":"2024-09-13T14:50:35.752252+0000","last_became_active":"2024-09-13T14:47:29.410951+0000","last_became_peered":"2024-09-13T14:47:29.410951+0000","last_unstale":"2024-09-13T14:50:35.752252+0000","last_undegraded":"2024-09-13T14:50:35.752252+0000","last_fullsized":"2024-09-13T14:50:35.752252+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:29.999519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:29.999519+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:04:54.887884+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":297,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.818245+0000","last_change":"2024-09-13T14:47:47.883475+0000","last_active":"2024-09-13T14:50:35.818245+0000","last_peered":"2024-09-13T14:50:35.818245+0000","last_clean":"2024-09-13T14:50:35.818245+0000","last_became_active":"2024-09-13T14:47:47.871525+0000","last_became_peered":"2024-09-13T14:47:47.871525+0000","last_unstale":"2024-09-13T14:50:35.818245+0000","last_undegraded":"2024-09-13T14:50:35.818245+0000","last_fullsized":"2024-09-13T14:50:35.818245+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:33.903432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":176,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.750365+0000","last_change":"2024-09-13T14:46:01.781508+0000","last_active":"2024-09-13T14:50:35.750365+0000","last_peered":"2024-09-13T14:50:35.750365+0000","last_clean":"2024-09-13T14:50:35.750365+0000","last_became_active":"2024-09-13T14:46:01.767096+0000","last_became_peered":"2024-09-13T14:46:01.767096+0000","last_unstale":"2024-09-13T14:50:35.750365+0000","last_undegraded":"2024-09-13T14:50:35.750365+0000","last_fullsized":"2024-09-13T14:50:35.750365+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:20:01.119189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":366,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.738211+0000","last_change":"2024-09-13T14:37:49.172218+0000","last_active":"2024-09-13T14:50:35.738211+0000","last_peered":"2024-09-13T14:50:35.738211+0000","last_clean":"2024-09-13T14:50:35.738211+0000","last_became_active":"2024-09-13T14:37:49.171718+0000","last_became_peered":"2024-09-13T14:37:49.171718+0000","last_unstale":"2024-09-13T14:50:35.738211+0000","last_undegraded":"2024-09-13T14:50:35.738211+0000","last_fullsized":"2024-09-13T14:50:35.738211+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:46:54.626521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":397,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.740313+0000","last_change":"2024-09-13T14:47:47.868409+0000","last_active":"2024-09-13T14:50:35.740313+0000","last_peered":"2024-09-13T14:50:35.740313+0000","last_clean":"2024-09-13T14:50:35.740313+0000","last_became_active":"2024-09-13T14:47:47.867712+0000","last_became_peered":"2024-09-13T14:47:47.867712+0000","last_unstale":"2024-09-13T14:50:35.740313+0000","last_undegraded":"2024-09-13T14:50:35.740313+0000","last_fullsized":"2024-09-13T14:50:35.740313+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:57:14.647493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":368,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.738909+0000","last_change":"2024-09-13T14:37:48.128447+0000","last_active":"2024-09-13T14:50:35.738909+0000","last_peered":"2024-09-13T14:50:35.738909+0000","last_clean":"2024-09-13T14:50:35.738909+0000","last_became_active":"2024-09-13T14:37:48.128143+0000","last_became_peered":"2024-09-13T14:37:48.128143+0000","last_unstale":"2024-09-13T14:50:35.738909+0000","last_undegraded":"2024-09-13T14:50:35.738909+0000","last_fullsized":"2024-09-13T14:50:35.738909+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:22.204039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":85,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-13T14:50:43.130055+0000","last_change":"2024-09-13T14:47:47.867935+0000","last_active":"2024-09-13T14:50:43.130055+0000","last_peered":"2024-09-13T14:50:43.130055+0000","last_clean":"2024-09-13T14:50:43.130055+0000","last_became_active":"2024-09-13T14:47:47.867616+0000","last_became_peered":"2024-09-13T14:47:47.867616+0000","last_unstale":"2024-09-13T14:50:43.130055+0000","last_undegraded":"2024-09-13T14:50:43.130055+0000","last_fullsized":"2024-09-13T14:50:43.130055+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:52:38.619916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":140,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.750969+0000","last_change":"2024-09-13T14:38:53.954281+0000","last_active":"2024-09-13T14:50:35.750969+0000","last_peered":"2024-09-13T14:50:35.750969+0000","last_clean":"2024-09-13T14:50:35.750969+0000","last_became_active":"2024-09-13T14:38:53.900912+0000","last_became_peered":"2024-09-13T14:38:53.900912+0000","last_unstale":"2024-09-13T14:50:35.750969+0000","last_undegraded":"2024-09-13T14:50:35.750969+0000","last_fullsized":"2024-09-13T14:50:35.750969+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:47.876275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"90'149","reported_seq":518,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-13T14:50:43.131000+0000","last_change":"2024-09-13T14:47:38.468095+0000","last_active":"2024-09-13T14:50:43.131000+0000","last_peered":"2024-09-13T14:50:43.131000+0000","last_clean":"2024-09-13T14:50:43.131000+0000","last_became_active":"2024-09-13T14:47:38.467526+0000","last_became_peered":"2024-09-13T14:47:38.467526+0000","last_unstale":"2024-09-13T14:50:43.131000+0000","last_undegraded":"2024-09-13T14:50:43.131000+0000","last_fullsized":"2024-09-13T14:50:43.131000+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:31.013128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:31.013128+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:17:20.508142+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":202,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-13T14:50:42.755437+0000","last_change":"2024-09-13T14:47:47.861842+0000","last_active":"2024-09-13T14:50:42.755437+0000","last_peered":"2024-09-13T14:50:42.755437+0000","last_clean":"2024-09-13T14:50:42.755437+0000","last_became_active":"2024-09-13T14:47:47.861492+0000","last_became_peered":"2024-09-13T14:47:47.861492+0000","last_unstale":"2024-09-13T14:50:42.755437+0000","last_undegraded":"2024-09-13T14:50:42.755437+0000","last_fullsized":"2024-09-13T14:50:42.755437+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:17.279533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":597,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.752261+0000","last_change":"2024-09-13T14:38:53.899535+0000","last_active":"2024-09-13T14:50:35.752261+0000","last_peered":"2024-09-13T14:50:35.752261+0000","last_clean":"2024-09-13T14:50:35.752261+0000","last_became_active":"2024-09-13T14:38:53.899183+0000","last_became_peered":"2024-09-13T14:38:53.899183+0000","last_unstale":"2024-09-13T14:50:35.752261+0000","last_undegraded":"2024-09-13T14:50:35.752261+0000","last_fullsized":"2024-09-13T14:50:35.752261+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:33.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":271,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.751118+0000","last_change":"2024-09-13T14:38:03.543426+0000","last_active":"2024-09-13T14:50:35.751118+0000","last_peered":"2024-09-13T14:50:35.751118+0000","last_clean":"2024-09-13T14:50:35.751118+0000","last_became_active":"2024-09-13T14:38:03.542661+0000","last_became_peered":"2024-09-13T14:38:03.542661+0000","last_unstale":"2024-09-13T14:50:35.751118+0000","last_undegraded":"2024-09-13T14:50:35.751118+0000","last_fullsized":"2024-09-13T14:50:35.751118+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:54.050127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":633,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.754243+0000","last_change":"2024-09-13T14:47:47.868624+0000","last_active":"2024-09-13T14:50:35.754243+0000","last_peered":"2024-09-13T14:50:35.754243+0000","last_clean":"2024-09-13T14:50:35.754243+0000","last_became_active":"2024-09-13T14:47:47.868031+0000","last_became_peered":"2024-09-13T14:47:47.868031+0000","last_unstale":"2024-09-13T14:50:35.754243+0000","last_undegraded":"2024-09-13T14:50:35.754243+0000","last_fullsized":"2024-09-13T14:50:35.754243+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:05:25.365023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":215,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-13T14:50:42.756014+0000","last_change":"2024-09-13T14:47:47.859825+0000","last_active":"2024-09-13T14:50:42.756014+0000","last_peered":"2024-09-13T14:50:42.756014+0000","last_clean":"2024-09-13T14:50:42.756014+0000","last_became_active":"2024-09-13T14:47:47.859069+0000","last_became_peered":"2024-09-13T14:47:47.859069+0000","last_unstale":"2024-09-13T14:50:42.756014+0000","last_undegraded":"2024-09-13T14:50:42.756014+0000","last_fullsized":"2024-09-13T14:50:42.756014+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:18:36.452321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":664,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.754485+0000","last_change":"2024-09-13T14:47:47.868945+0000","last_active":"2024-09-13T14:50:35.754485+0000","last_peered":"2024-09-13T14:50:35.754485+0000","last_clean":"2024-09-13T14:50:35.754485+0000","last_became_active":"2024-09-13T14:47:47.868296+0000","last_became_peered":"2024-09-13T14:47:47.868296+0000","last_unstale":"2024-09-13T14:50:35.754485+0000","last_undegraded":"2024-09-13T14:50:35.754485+0000","last_fullsized":"2024-09-13T14:50:35.754485+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:22:24.778693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":279,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.817799+0000","last_change":"2024-09-13T14:38:53.892948+0000","last_active":"2024-09-13T14:50:35.817799+0000","last_peered":"2024-09-13T14:50:35.817799+0000","last_clean":"2024-09-13T14:50:35.817799+0000","last_became_active":"2024-09-13T14:38:53.892665+0000","last_became_peered":"2024-09-13T14:38:53.892665+0000","last_unstale":"2024-09-13T14:50:35.817799+0000","last_undegraded":"2024-09-13T14:50:35.817799+0000","last_fullsized":"2024-09-13T14:50:35.817799+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:08.292359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":85,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-13T14:50:43.129504+0000","last_change":"2024-09-13T14:47:47.871164+0000","last_active":"2024-09-13T14:50:43.129504+0000","last_peered":"2024-09-13T14:50:43.129504+0000","last_clean":"2024-09-13T14:50:43.129504+0000","last_became_active":"2024-09-13T14:47:47.870862+0000","last_became_peered":"2024-09-13T14:47:47.870862+0000","last_unstale":"2024-09-13T14:50:43.129504+0000","last_undegraded":"2024-09-13T14:50:43.129504+0000","last_fullsized":"2024-09-13T14:50:43.129504+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:41:34.392081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":653,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.752251+0000","last_change":"2024-09-13T14:50:27.490554+0000","last_active":"2024-09-13T14:50:35.752251+0000","last_peered":"2024-09-13T14:50:35.752251+0000","last_clean":"2024-09-13T14:50:35.752251+0000","last_became_active":"2024-09-13T14:50:27.490211+0000","last_became_peered":"2024-09-13T14:50:27.490211+0000","last_unstale":"2024-09-13T14:50:35.752251+0000","last_undegraded":"2024-09-13T14:50:35.752251+0000","last_fullsized":"2024-09-13T14:50:35.752251+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:57:01.947741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":450,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.738581+0000","last_change":"2024-09-13T14:47:47.868246+0000","last_active":"2024-09-13T14:50:35.738581+0000","last_peered":"2024-09-13T14:50:35.738581+0000","last_clean":"2024-09-13T14:50:35.738581+0000","last_became_active":"2024-09-13T14:47:47.867817+0000","last_became_peered":"2024-09-13T14:47:47.867817+0000","last_unstale":"2024-09-13T14:50:35.738581+0000","last_undegraded":"2024-09-13T14:50:35.738581+0000","last_fullsized":"2024-09-13T14:50:35.738581+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:07.612451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":166,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-13T14:50:43.131144+0000","last_change":"2024-09-13T14:50:27.510133+0000","last_active":"2024-09-13T14:50:43.131144+0000","last_peered":"2024-09-13T14:50:43.131144+0000","last_clean":"2024-09-13T14:50:43.131144+0000","last_became_active":"2024-09-13T14:50:27.508774+0000","last_became_peered":"2024-09-13T14:50:27.508774+0000","last_unstale":"2024-09-13T14:50:43.131144+0000","last_undegraded":"2024-09-13T14:50:43.131144+0000","last_fullsized":"2024-09-13T14:50:43.131144+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:31:04.531746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":271,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.751667+0000","last_change":"2024-09-13T14:38:03.527021+0000","last_active":"2024-09-13T14:50:35.751667+0000","last_peered":"2024-09-13T14:50:35.751667+0000","last_clean":"2024-09-13T14:50:35.751667+0000","last_became_active":"2024-09-13T14:38:03.526698+0000","last_became_peered":"2024-09-13T14:38:03.526698+0000","last_unstale":"2024-09-13T14:50:35.751667+0000","last_undegraded":"2024-09-13T14:50:35.751667+0000","last_fullsized":"2024-09-13T14:50:35.751667+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:01:35.734086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":394,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.739453+0000","last_change":"2024-09-13T14:47:47.866784+0000","last_active":"2024-09-13T14:50:35.739453+0000","last_peered":"2024-09-13T14:50:35.739453+0000","last_clean":"2024-09-13T14:50:35.739453+0000","last_became_active":"2024-09-13T14:47:47.866483+0000","last_became_peered":"2024-09-13T14:47:47.866483+0000","last_unstale":"2024-09-13T14:50:35.739453+0000","last_undegraded":"2024-09-13T14:50:35.739453+0000","last_fullsized":"2024-09-13T14:50:35.739453+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:15:53.119741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":202,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.759871+0000","last_change":"2024-09-13T14:47:47.880946+0000","last_active":"2024-09-13T14:50:35.759871+0000","last_peered":"2024-09-13T14:50:35.759871+0000","last_clean":"2024-09-13T14:50:35.759871+0000","last_became_active":"2024-09-13T14:47:47.866534+0000","last_became_peered":"2024-09-13T14:47:47.866534+0000","last_unstale":"2024-09-13T14:50:35.759871+0000","last_undegraded":"2024-09-13T14:50:35.759871+0000","last_fullsized":"2024-09-13T14:50:35.759871+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:01:01.384873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":387,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.738309+0000","last_change":"2024-09-13T14:38:03.526333+0000","last_active":"2024-09-13T14:50:35.738309+0000","last_peered":"2024-09-13T14:50:35.738309+0000","last_clean":"2024-09-13T14:50:35.738309+0000","last_became_active":"2024-09-13T14:38:03.526001+0000","last_became_peered":"2024-09-13T14:38:03.526001+0000","last_unstale":"2024-09-13T14:50:35.738309+0000","last_undegraded":"2024-09-13T14:50:35.738309+0000","last_fullsized":"2024-09-13T14:50:35.738309+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:25:25.181271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":189,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.759940+0000","last_change":"2024-09-13T14:37:49.149582+0000","last_active":"2024-09-13T14:50:35.759940+0000","last_peered":"2024-09-13T14:50:35.759940+0000","last_clean":"2024-09-13T14:50:35.759940+0000","last_became_active":"2024-09-13T14:37:49.148525+0000","last_became_peered":"2024-09-13T14:37:49.148525+0000","last_unstale":"2024-09-13T14:50:35.759940+0000","last_undegraded":"2024-09-13T14:50:35.759940+0000","last_fullsized":"2024-09-13T14:50:35.759940+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:48.655665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":678,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.752677+0000","last_change":"2024-09-13T14:46:01.766580+0000","last_active":"2024-09-13T14:50:35.752677+0000","last_peered":"2024-09-13T14:50:35.752677+0000","last_clean":"2024-09-13T14:50:35.752677+0000","last_became_active":"2024-09-13T14:46:01.766257+0000","last_became_peered":"2024-09-13T14:46:01.766257+0000","last_unstale":"2024-09-13T14:50:35.752677+0000","last_undegraded":"2024-09-13T14:50:35.752677+0000","last_fullsized":"2024-09-13T14:50:35.752677+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:01.126109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":642,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.752724+0000","last_change":"2024-09-13T14:46:01.769647+0000","last_active":"2024-09-13T14:50:35.752724+0000","last_peered":"2024-09-13T14:50:35.752724+0000","last_clean":"2024-09-13T14:50:35.752724+0000","last_became_active":"2024-09-13T14:46:01.769228+0000","last_became_peered":"2024-09-13T14:46:01.769228+0000","last_unstale":"2024-09-13T14:50:35.752724+0000","last_undegraded":"2024-09-13T14:50:35.752724+0000","last_fullsized":"2024-09-13T14:50:35.752724+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:58.696024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":377,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.738113+0000","last_change":"2024-09-13T14:47:47.860727+0000","last_active":"2024-09-13T14:50:35.738113+0000","last_peered":"2024-09-13T14:50:35.738113+0000","last_clean":"2024-09-13T14:50:35.738113+0000","last_became_active":"2024-09-13T14:47:47.860377+0000","last_became_peered":"2024-09-13T14:47:47.860377+0000","last_unstale":"2024-09-13T14:50:35.738113+0000","last_undegraded":"2024-09-13T14:50:35.738113+0000","last_fullsized":"2024-09-13T14:50:35.738113+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:27:00.868699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":291,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-13T14:50:35.751071+0000","last_change":"2024-09-13T14:46:01.775237+0000","last_active":"2024-09-13T14:50:35.751071+0000","last_peered":"2024-09-13T14:50:35.751071+0000","last_clean":"2024-09-13T14:50:35.751071+0000","last_became_active":"2024-09-13T14:46:01.765025+0000","last_became_peered":"2024-09-13T14:46:01.765025+0000","last_unstale":"2024-09-13T14:50:35.751071+0000","last_undegraded":"2024-09-13T14:50:35.751071+0000","last_fullsized":"2024-09-13T14:50:35.751071+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:29:16.089042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1746,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":2154,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25132,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":151,"num_bytes_recovered":8388646,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3129,"ondisk_log_size":3129,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234483,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":639592,"kb_used_data":4700,"kb_used_omap":1,"kb_used_meta":634878,"kb_avail":93105560,"statfs":{"total":95995035648,"available":95340093440,"internally_reserved":0,"allocated":4812800,"data_stored":2870983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":650115531},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":239,"seq":1026497183901,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":508704,"kb_used_data":9108,"kb_used_omap":8,"kb_used_meta":499575,"kb_avail":93236448,"statfs":{"total":95995035648,"available":95474122752,"internally_reserved":0,"allocated":9326592,"data_stored":7392898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8453,"internal_metadata":511565563},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757748,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":914652,"kb_used_data":9040,"kb_used_omap":1,"kb_used_meta":905598,"kb_avail":92830500,"statfs":{"total":95995035648,"available":95058432000,"internally_reserved":0,"allocated":9256960,"data_stored":7323674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":927332806},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019392,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":767588,"kb_used_data":24408,"kb_used_omap":1,"kb_used_meta":743166,"kb_avail":92977564,"statfs":{"total":95995035648,"available":95209025536,"internally_reserved":0,"allocated":24993792,"data_stored":23056059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":761002438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435574,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":785228,"kb_used_data":5248,"kb_used_omap":5,"kb_used_meta":779962,"kb_avail":92959924,"statfs":{"total":95995035648,"available":95190962176,"internally_reserved":0,"allocated":5373952,"data_stored":3448880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":798681214},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":104,"seq":446676599103,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297628,"kb_used_data":4944,"kb_used_omap":7,"kb_used_meta":292664,"kb_avail":93447524,"statfs":{"total":95995035648,"available":95690264576,"internally_reserved":0,"allocated":5062656,"data_stored":3129658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8010,"internal_metadata":299688118},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":99,"seq":425201762629,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":175120,"kb_used_data":8772,"kb_used_omap":7,"kb_used_meta":166328,"kb_avail":93570032,"statfs":{"total":95995035648,"available":95815712768,"internally_reserved":0,"allocated":8982528,"data_stored":7060404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7759,"internal_metadata":170320305},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926152,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":848844,"kb_used_data":4992,"kb_used_omap":7,"kb_used_meta":843832,"kb_avail":92896308,"statfs":{"total":95995035648,"available":95125819392,"internally_reserved":0,"allocated":5111808,"data_stored":3186706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":864084264},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6860,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:50:51.011 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:51.033 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:51.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:51.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:51.135 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:51.156 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:51.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:51.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:51.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:50 smithi102 ceph-mon[110183]: pgmap v1288: 93 pgs: 93 active+clean; 4.6 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-13T14:50:51.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:50 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/542357329' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-13T14:50:51.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:50 smithi102 ceph-mon[110183]: osdmap e374: 8 total, 8 up, 8 in 2024-09-13T14:50:51.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:50 smithi102 ceph-mon[115269]: pgmap v1288: 93 pgs: 93 active+clean; 4.6 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-13T14:50:51.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:50 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/542357329' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-13T14:50:51.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:50 smithi102 ceph-mon[115269]: osdmap e374: 8 total, 8 up, 8 in 2024-09-13T14:50:51.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:50 smithi195 ceph-mon[104262]: pgmap v1288: 93 pgs: 93 active+clean; 4.6 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-13T14:50:51.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:50 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/542357329' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-13T14:50:51.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:50 smithi195 ceph-mon[104262]: osdmap e374: 8 total, 8 up, 8 in 2024-09-13T14:50:51.259 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:51.280 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:51.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:51.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:51.382 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:51.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:51.405 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:51.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:51.506 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:51.527 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:51.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:51.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:51.629 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:51.651 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:51.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:51.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:51.753 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:51.773 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:51.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:51.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:51.875 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:51.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:51.897 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:51.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:51.998 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:52.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:52.021 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:52.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:52.109 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:51 smithi195 ceph-mon[104262]: osdmap e375: 8 total, 8 up, 8 in 2024-09-13T14:50:52.109 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:51 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:50:52.109 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:51 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:50:52.109 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:51 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:50:52.109 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:51 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:50:52.123 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:52.144 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:52.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:52.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:52.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:51 smithi102 ceph-mon[110183]: osdmap e375: 8 total, 8 up, 8 in 2024-09-13T14:50:52.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:51 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:50:52.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:51 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:50:52.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:51 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:50:52.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:51 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:50:52.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:51 smithi102 ceph-mon[115269]: osdmap e375: 8 total, 8 up, 8 in 2024-09-13T14:50:52.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:51 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:50:52.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:51 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:50:52.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:51 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:50:52.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:51 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:50:52.246 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:52.264 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:52.269 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:52.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:52.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:52.371 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:52.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:52.394 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:52.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:52.495 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:52.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:52.518 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:52.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:52.619 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:52.642 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:52.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:52.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:52.656 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-09-13T14:50:52.744 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:52.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:52.766 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:52.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:52.867 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:52.890 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:52.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:52.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:52.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:52 smithi102 ceph-mon[110183]: pgmap v1291: 109 pgs: 9 unknown, 100 active+clean; 4.6 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-13T14:50:52.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:52 smithi102 ceph-mon[110183]: osdmap e376: 8 total, 8 up, 8 in 2024-09-13T14:50:52.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:52 smithi102 ceph-mon[110183]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:50:52.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:52 smithi102 ceph-mon[115269]: pgmap v1291: 109 pgs: 9 unknown, 100 active+clean; 4.6 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-13T14:50:52.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:52 smithi102 ceph-mon[115269]: osdmap e376: 8 total, 8 up, 8 in 2024-09-13T14:50:52.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:52 smithi102 ceph-mon[115269]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:50:52.992 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:52.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:52 smithi195 ceph-mon[104262]: pgmap v1291: 109 pgs: 9 unknown, 100 active+clean; 4.6 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-13T14:50:52.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:52 smithi195 ceph-mon[104262]: osdmap e376: 8 total, 8 up, 8 in 2024-09-13T14:50:52.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:52 smithi195 ceph-mon[104262]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:50:53.016 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:53.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:53.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:53.118 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:53.140 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:53.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:53.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:53.242 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:53.248 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:50:53.248 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:50:53.265 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:53.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:53.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:53.367 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:53.389 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:53.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:53.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:53.491 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:53.512 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:53.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:53.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:53.614 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:53.636 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:53.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:53.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:53.738 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:53.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:53.762 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:53.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:53.863 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:53.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:53.885 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:53.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:53.986 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:54.009 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:50:54.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:54.011 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:54.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:54.112 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:54.133 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:54.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:54.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:54.226 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T14:50:54.235 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:54.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:54.258 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:54.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:54.359 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:54.381 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:54.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:54.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:54.483 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:54.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:54.506 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:54.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:54.607 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:54.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:54.630 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:54.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:54.731 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:54.753 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:54.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:54.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:54.787 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:54.789 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:54.795 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:54 smithi102 ceph-mon[110183]: pgmap v1293: 109 pgs: 2 unknown, 107 active+clean; 4.6 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-13T14:50:54.841 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:54.856 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:54.879 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:54.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:54.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:54.981 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:55.003 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:55.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:55.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:55.105 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:55.108 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:54 smithi102 ceph-mon[115269]: pgmap v1293: 109 pgs: 2 unknown, 107 active+clean; 4.6 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-13T14:50:55.128 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:55.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:55.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:55.230 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:55.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:54 smithi195 ceph-mon[104262]: pgmap v1293: 109 pgs: 2 unknown, 107 active+clean; 4.6 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-13T14:50:55.255 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:55.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:55.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:55.357 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:55.379 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:55.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:55.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:55.481 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:55.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:55.504 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:55.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:55.605 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:55.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:55.628 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:55.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:55.730 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:55.751 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:55.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:55.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:55.853 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:55.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:55.876 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:55.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:55.977 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:55.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:55.998 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:55.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:56.099 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:56.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:56.127 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:56.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:56.228 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:56.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:56.250 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:56.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:56.351 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:56.365 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:56.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:56.374 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:56.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:56.447 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:50:56 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:50:56] "GET /metrics HTTP/1.1" 200 36443 "" "Prometheus/2.51.0" 2024-09-13T14:50:56.475 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:56.496 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:56.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:56.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:56.598 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:56.619 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:56.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:56.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:56.721 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:56.742 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:56.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:56.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:56.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:50:56.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:50:56.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:56 smithi195 ceph-mon[104262]: pgmap v1294: 109 pgs: 109 active+clean; 4.6 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-13T14:50:56.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:50:56.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:50:56.843 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:56.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:56.867 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:56.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:56.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:50:56.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:50:56.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:56 smithi102 ceph-mon[115269]: pgmap v1294: 109 pgs: 109 active+clean; 4.6 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-13T14:50:56.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:50:56.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:50:56.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:50:56.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:50:56.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:56 smithi102 ceph-mon[110183]: pgmap v1294: 109 pgs: 109 active+clean; 4.6 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-13T14:50:56.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:50:56.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:50:56.968 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:56.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:56.992 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:56.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:57.093 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:57.115 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:57.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:57.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:57.217 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:57.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:57.238 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:57.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:57.339 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:57.362 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:57.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:57.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:57.464 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:57.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:57.487 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:57.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:57.588 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:57.610 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:57.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:57.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:57.712 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:57.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:57.735 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:57.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:57.836 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:57.858 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:57.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:57.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:57.960 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:57.981 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:57.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:57.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:58.084 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:58.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:58.106 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:58.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:58.207 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:58.229 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:58.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:58.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:58.330 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:58.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:58.354 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:58.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:58.455 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:58.477 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:58.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:58.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:58.579 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:58.602 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:58.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:58.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:58.704 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:58.727 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:58.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:58.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:58.829 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:58.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:58.853 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:58.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:58.954 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:58.977 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:58.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:58.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:58.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:58 smithi195 ceph-mon[104262]: pgmap v1295: 109 pgs: 109 active+clean; 4.6 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-13T14:50:59.029 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:58 smithi102 ceph-mon[110183]: pgmap v1295: 109 pgs: 109 active+clean; 4.6 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-13T14:50:59.029 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:58 smithi102 ceph-mon[115269]: pgmap v1295: 109 pgs: 109 active+clean; 4.6 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-13T14:50:59.045 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:50:59.045 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:50:59.061 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:50:59.062 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:50:59.078 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:50:59.078 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:50:59.079 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:59.094 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:50:59.094 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:50:59.102 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:59.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:59.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:59.110 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:50:59.110 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:50:59.126 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:50:59.126 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:50:59.143 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:50:59.143 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:50:59.159 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:50:59.159 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:50:59.204 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:59.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:59.227 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:59.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:59.328 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:59.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:59.353 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:59.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:59.454 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:59.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:59.476 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:59.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:59.577 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:59.603 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:59.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:59.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:59.704 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:59.729 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:59.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:59.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:59.741 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is unset 2024-09-13T14:50:59.830 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:59.856 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:50:59.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:59.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:59.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:59 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/43485807' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:50:59.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:50:59 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1246077134' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:50:59.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:59 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/43485807' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:50:59.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:50:59 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1246077134' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:50:59.957 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:59.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:59.980 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:50:59.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:59.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:59 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/43485807' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:50:59.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:50:59 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1246077134' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:00.081 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:00.103 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:00.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:00.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:00.205 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:00.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:00.233 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:00.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:00.334 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:00.357 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:00.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:00.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:00.459 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:00.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:00.481 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:00.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:00.582 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:00.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:00.604 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:00.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:00.705 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:00.732 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:00.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:00.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:00.833 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:00.859 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:00.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:00.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:00.961 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:00.986 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:00.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:00.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:00.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:00 smithi195 ceph-mon[104262]: pgmap v1296: 109 pgs: 109 active+clean; 4.6 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-13T14:51:00.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:00 smithi195 ceph-mon[104262]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:51:00.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:00 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/43485807' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:51:00.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:00 smithi195 ceph-mon[104262]: osdmap e377: 8 total, 8 up, 8 in 2024-09-13T14:51:01.054 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:00 smithi102 ceph-mon[115269]: pgmap v1296: 109 pgs: 109 active+clean; 4.6 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-13T14:51:01.054 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:00 smithi102 ceph-mon[115269]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:51:01.054 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:00 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/43485807' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:51:01.055 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:00 smithi102 ceph-mon[115269]: osdmap e377: 8 total, 8 up, 8 in 2024-09-13T14:51:01.055 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:00 smithi102 ceph-mon[110183]: pgmap v1296: 109 pgs: 109 active+clean; 4.6 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-13T14:51:01.055 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:00 smithi102 ceph-mon[110183]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:51:01.055 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:00 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/43485807' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:51:01.055 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:00 smithi102 ceph-mon[110183]: osdmap e377: 8 total, 8 up, 8 in 2024-09-13T14:51:01.089 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:01.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:01.116 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:01.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:01.217 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:01.242 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:01.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:01.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:01.343 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:01.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:01.366 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:01.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:01.467 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:01.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:01.494 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:01.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:01.595 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:01.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:01.617 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:01.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:01.718 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:01.744 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:01.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:01.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:01.746 INFO:teuthology.orchestra.run.smithi102.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-09-13T14:51:01.846 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:01.872 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:01.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:01.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:01.974 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:01.991 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:01 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3701332414' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:51:01.991 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:01 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:51:01.991 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:01 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3701332414' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:51:01.992 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:01 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:51:01.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:01 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3701332414' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:51:01.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:01 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:51:02.001 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:02.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:02.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:02.102 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:02.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:02.125 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:02.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:02.227 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:02.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:02.250 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:02.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:02.351 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:02.373 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:02.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:02.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:02.475 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:02.502 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:02.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:02.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:02.518 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:51:02.519 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:51:02.604 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:02.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:02.626 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:02.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:02.727 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:02.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:02.754 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:02.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:02.855 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:02.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:02.878 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:02.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:02.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:02 smithi102 ceph-mon[110183]: pgmap v1298: 109 pgs: 109 active+clean; 4.6 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-13T14:51:02.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:02 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:51:02.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:02 smithi102 ceph-mon[110183]: osdmap e378: 8 total, 8 up, 8 in 2024-09-13T14:51:02.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:02 smithi102 ceph-mon[115269]: pgmap v1298: 109 pgs: 109 active+clean; 4.6 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-13T14:51:02.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:02 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:51:02.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:02 smithi102 ceph-mon[115269]: osdmap e378: 8 total, 8 up, 8 in 2024-09-13T14:51:02.979 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:02.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:02 smithi195 ceph-mon[104262]: pgmap v1298: 109 pgs: 109 active+clean; 4.6 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-13T14:51:02.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:02 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:51:02.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:02 smithi195 ceph-mon[104262]: osdmap e378: 8 total, 8 up, 8 in 2024-09-13T14:51:03.005 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:03.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:03.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:03.107 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:03.128 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:03.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:03.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:03.230 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:03.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:03.252 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:03.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:03.353 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:03.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:03.385 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:03.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:03.486 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:03.511 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:03.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:03.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:03.613 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:03.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:03.636 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:03.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:03.737 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:03.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:03.760 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:03.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:03.861 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:03.883 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:03.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:03.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:03.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:03 smithi102 ceph-mon[110183]: from='client.46448 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:51:03.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:03 smithi102 ceph-mon[115269]: from='client.46448 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:51:03.985 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:03.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:03 smithi195 ceph-mon[104262]: from='client.46448 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:51:04.007 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:04.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:04.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:04.028 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":1299,"stamp":"2024-09-13T14:51:01.744214+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790245,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":908,"num_write_kb":12625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":195,"num_bytes_recovered":8718480,"num_keys_recovered":20,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4162,"ondisk_log_size":4162,"up":327,"acting":327,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":327,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":4922180,"kb_used_data":51860,"kb_used_omap":41,"kb_used_meta":4870166,"kb_avail":745039036,"statfs":{"total":767960285184,"available":762919972864,"internally_reserved":0,"allocated":53104640,"data_stored":37537902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42913,"internal_metadata":4987050079},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.063828"},"pg_stats":[{"pgid":"1.11","version":"90'149","reported_seq":493,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.419336+0000","last_change":"2024-09-13T14:47:38.688630+0000","last_active":"2024-09-13T14:50:52.419336+0000","last_peered":"2024-09-13T14:50:52.419336+0000","last_clean":"2024-09-13T14:50:52.419336+0000","last_became_active":"2024-09-13T14:47:38.460077+0000","last_became_peered":"2024-09-13T14:47:38.460077+0000","last_unstale":"2024-09-13T14:50:52.419336+0000","last_undegraded":"2024-09-13T14:50:52.419336+0000","last_fullsized":"2024-09-13T14:50:52.419336+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:38.688341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.688341+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:14:17.232921+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":663,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:51.983060+0000","last_change":"2024-09-13T14:47:47.868783+0000","last_active":"2024-09-13T14:50:51.983060+0000","last_peered":"2024-09-13T14:50:51.983060+0000","last_clean":"2024-09-13T14:50:51.983060+0000","last_became_active":"2024-09-13T14:47:47.868478+0000","last_became_peered":"2024-09-13T14:47:47.868478+0000","last_unstale":"2024-09-13T14:50:51.983060+0000","last_undegraded":"2024-09-13T14:50:51.983060+0000","last_fullsized":"2024-09-13T14:50:51.983060+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:27:01.784142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":397,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.053371+0000","last_change":"2024-09-13T14:46:01.772811+0000","last_active":"2024-09-13T14:50:52.053371+0000","last_peered":"2024-09-13T14:50:52.053371+0000","last_clean":"2024-09-13T14:50:52.053371+0000","last_became_active":"2024-09-13T14:46:01.764775+0000","last_became_peered":"2024-09-13T14:46:01.764775+0000","last_unstale":"2024-09-13T14:50:52.053371+0000","last_undegraded":"2024-09-13T14:50:52.053371+0000","last_fullsized":"2024-09-13T14:50:52.053371+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:42:20.738421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"90'149","reported_seq":442,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.411544+0000","last_change":"2024-09-13T14:47:41.482552+0000","last_active":"2024-09-13T14:50:52.411544+0000","last_peered":"2024-09-13T14:50:52.411544+0000","last_clean":"2024-09-13T14:50:52.411544+0000","last_became_active":"2024-09-13T14:47:38.466659+0000","last_became_peered":"2024-09-13T14:47:38.466659+0000","last_unstale":"2024-09-13T14:50:52.411544+0000","last_undegraded":"2024-09-13T14:50:52.411544+0000","last_fullsized":"2024-09-13T14:50:52.411544+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:41.482364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:41.482364+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:50:49.239389+0000","scrub_duration":0.027843855000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":179,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.412119+0000","last_change":"2024-09-13T14:36:13.862181+0000","last_active":"2024-09-13T14:50:52.412119+0000","last_peered":"2024-09-13T14:50:52.412119+0000","last_clean":"2024-09-13T14:50:52.412119+0000","last_became_active":"2024-09-13T14:36:13.861873+0000","last_became_peered":"2024-09-13T14:36:13.861873+0000","last_unstale":"2024-09-13T14:50:52.412119+0000","last_undegraded":"2024-09-13T14:50:52.412119+0000","last_fullsized":"2024-09-13T14:50:52.412119+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:14.085700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":286,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.419181+0000","last_change":"2024-09-13T14:38:53.896672+0000","last_active":"2024-09-13T14:50:52.419181+0000","last_peered":"2024-09-13T14:50:52.419181+0000","last_clean":"2024-09-13T14:50:52.419181+0000","last_became_active":"2024-09-13T14:38:53.896346+0000","last_became_peered":"2024-09-13T14:38:53.896346+0000","last_unstale":"2024-09-13T14:50:52.419181+0000","last_undegraded":"2024-09-13T14:50:52.419181+0000","last_fullsized":"2024-09-13T14:50:52.419181+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:57.789686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"90'149","reported_seq":513,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.370223+0000","last_change":"2024-09-13T14:47:51.226821+0000","last_active":"2024-09-13T14:50:52.370223+0000","last_peered":"2024-09-13T14:50:52.370223+0000","last_clean":"2024-09-13T14:50:52.370223+0000","last_became_active":"2024-09-13T14:47:50.881689+0000","last_became_peered":"2024-09-13T14:47:50.881689+0000","last_unstale":"2024-09-13T14:50:52.370223+0000","last_undegraded":"2024-09-13T14:50:52.370223+0000","last_fullsized":"2024-09-13T14:50:52.370223+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:51.226748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:51.226748+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:06:02.065990+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":286,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.419588+0000","last_change":"2024-09-13T14:38:53.893021+0000","last_active":"2024-09-13T14:50:52.419588+0000","last_peered":"2024-09-13T14:50:52.419588+0000","last_clean":"2024-09-13T14:50:52.419588+0000","last_became_active":"2024-09-13T14:38:53.892668+0000","last_became_peered":"2024-09-13T14:38:53.892668+0000","last_unstale":"2024-09-13T14:50:52.419588+0000","last_undegraded":"2024-09-13T14:50:52.419588+0000","last_fullsized":"2024-09-13T14:50:52.419588+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:57.180738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":350,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.053196+0000","last_change":"2024-09-13T14:37:48.134642+0000","last_active":"2024-09-13T14:50:52.053196+0000","last_peered":"2024-09-13T14:50:52.053196+0000","last_clean":"2024-09-13T14:50:52.053196+0000","last_became_active":"2024-09-13T14:37:48.134402+0000","last_became_peered":"2024-09-13T14:37:48.134402+0000","last_unstale":"2024-09-13T14:50:52.053196+0000","last_undegraded":"2024-09-13T14:50:52.053196+0000","last_fullsized":"2024-09-13T14:50:52.053196+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:51.313663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"90'149","reported_seq":459,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.412036+0000","last_change":"2024-09-13T14:47:50.892667+0000","last_active":"2024-09-13T14:50:52.412036+0000","last_peered":"2024-09-13T14:50:52.412036+0000","last_clean":"2024-09-13T14:50:52.412036+0000","last_became_active":"2024-09-13T14:47:50.883312+0000","last_became_peered":"2024-09-13T14:47:50.883312+0000","last_unstale":"2024-09-13T14:50:52.412036+0000","last_undegraded":"2024-09-13T14:50:52.412036+0000","last_fullsized":"2024-09-13T14:50:52.412036+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:45.441819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:45.441819+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:39.138573+0000","scrub_duration":0.021368558999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":688,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:51.983234+0000","last_change":"2024-09-13T14:50:27.491166+0000","last_active":"2024-09-13T14:50:51.983234+0000","last_peered":"2024-09-13T14:50:51.983234+0000","last_clean":"2024-09-13T14:50:51.983234+0000","last_became_active":"2024-09-13T14:50:27.489905+0000","last_became_peered":"2024-09-13T14:50:27.489905+0000","last_unstale":"2024-09-13T14:50:51.983234+0000","last_undegraded":"2024-09-13T14:50:51.983234+0000","last_fullsized":"2024-09-13T14:50:51.983234+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:32.237904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":289,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.419050+0000","last_change":"2024-09-13T14:38:03.554500+0000","last_active":"2024-09-13T14:50:52.419050+0000","last_peered":"2024-09-13T14:50:52.419050+0000","last_clean":"2024-09-13T14:50:52.419050+0000","last_became_active":"2024-09-13T14:38:03.532910+0000","last_became_peered":"2024-09-13T14:38:03.532910+0000","last_unstale":"2024-09-13T14:50:52.419050+0000","last_undegraded":"2024-09-13T14:50:52.419050+0000","last_fullsized":"2024-09-13T14:50:52.419050+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:27:50.843498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":14,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.419500+0000","last_change":"2024-09-13T14:50:50.779688+0000","last_active":"2024-09-13T14:50:52.419500+0000","last_peered":"2024-09-13T14:50:52.419500+0000","last_clean":"2024-09-13T14:50:52.419500+0000","last_became_active":"2024-09-13T14:50:50.779411+0000","last_became_peered":"2024-09-13T14:50:50.779411+0000","last_unstale":"2024-09-13T14:50:52.419500+0000","last_undegraded":"2024-09-13T14:50:52.419500+0000","last_fullsized":"2024-09-13T14:50:52.419500+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:50:49.763471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:08:11.728619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"90'149","reported_seq":461,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.053277+0000","last_change":"2024-09-13T14:47:50.887140+0000","last_active":"2024-09-13T14:50:52.053277+0000","last_peered":"2024-09-13T14:50:52.053277+0000","last_clean":"2024-09-13T14:50:52.053277+0000","last_became_active":"2024-09-13T14:47:50.886603+0000","last_became_peered":"2024-09-13T14:47:50.886603+0000","last_unstale":"2024-09-13T14:50:52.053277+0000","last_undegraded":"2024-09-13T14:50:52.053277+0000","last_fullsized":"2024-09-13T14:50:52.053277+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:40.786610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:40.786610+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:00.936651+0000","scrub_duration":0.013725185000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":143,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.053549+0000","last_change":"2024-09-13T14:38:53.940238+0000","last_active":"2024-09-13T14:50:52.053549+0000","last_peered":"2024-09-13T14:50:52.053549+0000","last_clean":"2024-09-13T14:50:52.053549+0000","last_became_active":"2024-09-13T14:38:53.892825+0000","last_became_peered":"2024-09-13T14:38:53.892825+0000","last_unstale":"2024-09-13T14:50:52.053549+0000","last_undegraded":"2024-09-13T14:50:52.053549+0000","last_fullsized":"2024-09-13T14:50:52.053549+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:46.666581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":199,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.411719+0000","last_change":"2024-09-13T14:38:53.898907+0000","last_active":"2024-09-13T14:50:52.411719+0000","last_peered":"2024-09-13T14:50:52.411719+0000","last_clean":"2024-09-13T14:50:52.411719+0000","last_became_active":"2024-09-13T14:38:53.898572+0000","last_became_peered":"2024-09-13T14:38:53.898572+0000","last_unstale":"2024-09-13T14:50:52.411719+0000","last_undegraded":"2024-09-13T14:50:52.411719+0000","last_fullsized":"2024-09-13T14:50:52.411719+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:29.222150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":14,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.370418+0000","last_change":"2024-09-13T14:50:50.782298+0000","last_active":"2024-09-13T14:50:52.370418+0000","last_peered":"2024-09-13T14:50:52.370418+0000","last_clean":"2024-09-13T14:50:52.370418+0000","last_became_active":"2024-09-13T14:50:50.782029+0000","last_became_peered":"2024-09-13T14:50:50.782029+0000","last_unstale":"2024-09-13T14:50:52.370418+0000","last_undegraded":"2024-09-13T14:50:52.370418+0000","last_fullsized":"2024-09-13T14:50:52.370418+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:50:49.763471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:02:35.095008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"90'149","reported_seq":535,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:51.982998+0000","last_change":"2024-09-13T14:47:38.466101+0000","last_active":"2024-09-13T14:50:51.982998+0000","last_peered":"2024-09-13T14:50:51.982998+0000","last_clean":"2024-09-13T14:50:51.982998+0000","last_became_active":"2024-09-13T14:47:38.465607+0000","last_became_peered":"2024-09-13T14:47:38.465607+0000","last_unstale":"2024-09-13T14:50:51.982998+0000","last_undegraded":"2024-09-13T14:50:51.982998+0000","last_fullsized":"2024-09-13T14:50:51.982998+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:32.060817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:32.060817+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:55.362724+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":156,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.052885+0000","last_change":"2024-09-13T14:47:47.878423+0000","last_active":"2024-09-13T14:50:52.052885+0000","last_peered":"2024-09-13T14:50:52.052885+0000","last_clean":"2024-09-13T14:50:52.052885+0000","last_became_active":"2024-09-13T14:47:47.866926+0000","last_became_peered":"2024-09-13T14:47:47.866926+0000","last_unstale":"2024-09-13T14:50:52.052885+0000","last_undegraded":"2024-09-13T14:50:52.052885+0000","last_fullsized":"2024-09-13T14:50:52.052885+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:11:07.534097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":398,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:51.782006+0000","last_change":"2024-09-13T14:47:47.860804+0000","last_active":"2024-09-13T14:50:51.782006+0000","last_peered":"2024-09-13T14:50:51.782006+0000","last_clean":"2024-09-13T14:50:51.782006+0000","last_became_active":"2024-09-13T14:47:47.860496+0000","last_became_peered":"2024-09-13T14:47:47.860496+0000","last_unstale":"2024-09-13T14:50:51.782006+0000","last_undegraded":"2024-09-13T14:50:51.782006+0000","last_fullsized":"2024-09-13T14:50:51.782006+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:17:57.542599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":12,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.412243+0000","last_change":"2024-09-13T14:50:50.784134+0000","last_active":"2024-09-13T14:50:52.412243+0000","last_peered":"2024-09-13T14:50:52.412243+0000","last_clean":"2024-09-13T14:50:52.412243+0000","last_became_active":"2024-09-13T14:50:50.783762+0000","last_became_peered":"2024-09-13T14:50:50.783762+0000","last_unstale":"2024-09-13T14:50:52.412243+0000","last_undegraded":"2024-09-13T14:50:52.412243+0000","last_fullsized":"2024-09-13T14:50:52.412243+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:50:49.763471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:14:00.109438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"90'149","reported_seq":486,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.412224+0000","last_change":"2024-09-13T14:47:38.468287+0000","last_active":"2024-09-13T14:50:52.412224+0000","last_peered":"2024-09-13T14:50:52.412224+0000","last_clean":"2024-09-13T14:50:52.412224+0000","last_became_active":"2024-09-13T14:47:38.467941+0000","last_became_peered":"2024-09-13T14:47:38.467941+0000","last_unstale":"2024-09-13T14:50:52.412224+0000","last_undegraded":"2024-09-13T14:50:52.412224+0000","last_fullsized":"2024-09-13T14:50:52.412224+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:34.091782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:34.091782+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:34:20.519248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":380,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.053504+0000","last_change":"2024-09-13T14:37:49.611774+0000","last_active":"2024-09-13T14:50:52.053504+0000","last_peered":"2024-09-13T14:50:52.053504+0000","last_clean":"2024-09-13T14:50:52.053504+0000","last_became_active":"2024-09-13T14:37:49.611088+0000","last_became_peered":"2024-09-13T14:37:49.611088+0000","last_unstale":"2024-09-13T14:50:52.053504+0000","last_undegraded":"2024-09-13T14:50:52.053504+0000","last_fullsized":"2024-09-13T14:50:52.053504+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:08.933213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":91,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.370471+0000","last_change":"2024-09-13T14:47:47.865523+0000","last_active":"2024-09-13T14:50:52.370471+0000","last_peered":"2024-09-13T14:50:52.370471+0000","last_clean":"2024-09-13T14:50:52.370471+0000","last_became_active":"2024-09-13T14:47:47.865180+0000","last_became_peered":"2024-09-13T14:47:47.865180+0000","last_unstale":"2024-09-13T14:50:52.370471+0000","last_undegraded":"2024-09-13T14:50:52.370471+0000","last_fullsized":"2024-09-13T14:50:52.370471+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:39:35.479251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":12,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.411872+0000","last_change":"2024-09-13T14:50:51.090850+0000","last_active":"2024-09-13T14:50:52.411872+0000","last_peered":"2024-09-13T14:50:52.411872+0000","last_clean":"2024-09-13T14:50:52.411872+0000","last_became_active":"2024-09-13T14:50:51.090552+0000","last_became_peered":"2024-09-13T14:50:51.090552+0000","last_unstale":"2024-09-13T14:50:52.411872+0000","last_undegraded":"2024-09-13T14:50:52.411872+0000","last_fullsized":"2024-09-13T14:50:52.411872+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:50:49.763471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:44:01.059274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"90'149","reported_seq":492,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:51.782430+0000","last_change":"2024-09-13T14:47:38.459769+0000","last_active":"2024-09-13T14:50:51.782430+0000","last_peered":"2024-09-13T14:50:51.782430+0000","last_clean":"2024-09-13T14:50:51.782430+0000","last_became_active":"2024-09-13T14:47:38.459440+0000","last_became_peered":"2024-09-13T14:47:38.459440+0000","last_unstale":"2024-09-13T14:50:51.782430+0000","last_undegraded":"2024-09-13T14:50:51.782430+0000","last_fullsized":"2024-09-13T14:50:51.782430+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:30.526999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:30.526999+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:00.016329+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":190,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.411802+0000","last_change":"2024-09-13T14:37:49.168003+0000","last_active":"2024-09-13T14:50:52.411802+0000","last_peered":"2024-09-13T14:50:52.411802+0000","last_clean":"2024-09-13T14:50:52.411802+0000","last_became_active":"2024-09-13T14:37:49.167638+0000","last_became_peered":"2024-09-13T14:37:49.167638+0000","last_unstale":"2024-09-13T14:50:52.411802+0000","last_undegraded":"2024-09-13T14:50:52.411802+0000","last_fullsized":"2024-09-13T14:50:52.411802+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:29.051981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":286,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.418968+0000","last_change":"2024-09-13T14:47:47.868094+0000","last_active":"2024-09-13T14:50:52.418968+0000","last_peered":"2024-09-13T14:50:52.418968+0000","last_clean":"2024-09-13T14:50:52.418968+0000","last_became_active":"2024-09-13T14:47:47.867750+0000","last_became_peered":"2024-09-13T14:47:47.867750+0000","last_unstale":"2024-09-13T14:50:52.418968+0000","last_undegraded":"2024-09-13T14:50:52.418968+0000","last_fullsized":"2024-09-13T14:50:52.418968+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:55:38.203481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":14,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:51.982582+0000","last_change":"2024-09-13T14:50:51.091554+0000","last_active":"2024-09-13T14:50:51.982582+0000","last_peered":"2024-09-13T14:50:51.982582+0000","last_clean":"2024-09-13T14:50:51.982582+0000","last_became_active":"2024-09-13T14:50:51.091227+0000","last_became_peered":"2024-09-13T14:50:51.091227+0000","last_unstale":"2024-09-13T14:50:51.982582+0000","last_undegraded":"2024-09-13T14:50:51.982582+0000","last_fullsized":"2024-09-13T14:50:51.982582+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:50:49.763471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:57:32.951805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"90'149","reported_seq":552,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.053034+0000","last_change":"2024-09-13T14:46:55.355650+0000","last_active":"2024-09-13T14:50:52.053034+0000","last_peered":"2024-09-13T14:50:52.053034+0000","last_clean":"2024-09-13T14:50:52.053034+0000","last_became_active":"2024-09-13T14:46:48.150338+0000","last_became_peered":"2024-09-13T14:46:48.150338+0000","last_unstale":"2024-09-13T14:50:52.053034+0000","last_undegraded":"2024-09-13T14:50:52.053034+0000","last_fullsized":"2024-09-13T14:50:52.053034+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:46:55.355521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:46:55.355521+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:30:53.006278+0000","scrub_duration":0.002025448,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":637,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:51.982749+0000","last_change":"2024-09-13T14:38:03.549595+0000","last_active":"2024-09-13T14:50:51.982749+0000","last_peered":"2024-09-13T14:50:51.982749+0000","last_clean":"2024-09-13T14:50:51.982749+0000","last_became_active":"2024-09-13T14:38:03.533331+0000","last_became_peered":"2024-09-13T14:38:03.533331+0000","last_unstale":"2024-09-13T14:50:51.982749+0000","last_undegraded":"2024-09-13T14:50:51.982749+0000","last_fullsized":"2024-09-13T14:50:51.982749+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:59:02.470343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":373,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:51.782554+0000","last_change":"2024-09-13T14:46:26.980367+0000","last_active":"2024-09-13T14:50:51.782554+0000","last_peered":"2024-09-13T14:50:51.782554+0000","last_clean":"2024-09-13T14:50:51.782554+0000","last_became_active":"2024-09-13T14:46:26.979937+0000","last_became_peered":"2024-09-13T14:46:26.979937+0000","last_unstale":"2024-09-13T14:50:51.782554+0000","last_undegraded":"2024-09-13T14:50:51.782554+0000","last_fullsized":"2024-09-13T14:50:51.782554+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:56.830872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":14,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.370064+0000","last_change":"2024-09-13T14:50:51.092387+0000","last_active":"2024-09-13T14:50:52.370064+0000","last_peered":"2024-09-13T14:50:52.370064+0000","last_clean":"2024-09-13T14:50:52.370064+0000","last_became_active":"2024-09-13T14:50:51.092062+0000","last_became_peered":"2024-09-13T14:50:51.092062+0000","last_unstale":"2024-09-13T14:50:52.370064+0000","last_undegraded":"2024-09-13T14:50:52.370064+0000","last_fullsized":"2024-09-13T14:50:52.370064+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:50:49.763471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:56:42.246321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"90'149","reported_seq":454,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.411609+0000","last_change":"2024-09-13T14:46:54.393663+0000","last_active":"2024-09-13T14:50:52.411609+0000","last_peered":"2024-09-13T14:50:52.411609+0000","last_clean":"2024-09-13T14:50:52.411609+0000","last_became_active":"2024-09-13T14:46:48.149004+0000","last_became_peered":"2024-09-13T14:46:48.149004+0000","last_unstale":"2024-09-13T14:50:52.411609+0000","last_undegraded":"2024-09-13T14:50:52.411609+0000","last_fullsized":"2024-09-13T14:50:52.411609+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:46:54.393204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:46:54.393204+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:32:40.003989+0000","scrub_duration":0.015172223,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":306,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:51.782209+0000","last_change":"2024-09-13T14:47:47.867571+0000","last_active":"2024-09-13T14:50:51.782209+0000","last_peered":"2024-09-13T14:50:51.782209+0000","last_clean":"2024-09-13T14:50:51.782209+0000","last_became_active":"2024-09-13T14:47:47.867269+0000","last_became_peered":"2024-09-13T14:47:47.867269+0000","last_unstale":"2024-09-13T14:50:51.782209+0000","last_undegraded":"2024-09-13T14:50:51.782209+0000","last_fullsized":"2024-09-13T14:50:51.782209+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:46.093600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":435,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:51.782450+0000","last_change":"2024-09-13T14:47:47.866925+0000","last_active":"2024-09-13T14:50:51.782450+0000","last_peered":"2024-09-13T14:50:51.782450+0000","last_clean":"2024-09-13T14:50:51.782450+0000","last_became_active":"2024-09-13T14:47:47.866596+0000","last_became_peered":"2024-09-13T14:47:47.866596+0000","last_unstale":"2024-09-13T14:50:51.782450+0000","last_undegraded":"2024-09-13T14:50:51.782450+0000","last_fullsized":"2024-09-13T14:50:51.782450+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:11:25.617891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.053166+0000","last_change":"2024-09-13T14:50:50.782641+0000","last_active":"2024-09-13T14:50:52.053166+0000","last_peered":"2024-09-13T14:50:52.053166+0000","last_clean":"2024-09-13T14:50:52.053166+0000","last_became_active":"2024-09-13T14:50:50.782253+0000","last_became_peered":"2024-09-13T14:50:50.782253+0000","last_unstale":"2024-09-13T14:50:52.053166+0000","last_undegraded":"2024-09-13T14:50:52.053166+0000","last_fullsized":"2024-09-13T14:50:52.053166+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:50:49.763471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:29:58.217758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.b","version":"90'149","reported_seq":526,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.370068+0000","last_change":"2024-09-13T14:47:50.891055+0000","last_active":"2024-09-13T14:50:52.370068+0000","last_peered":"2024-09-13T14:50:52.370068+0000","last_clean":"2024-09-13T14:50:52.370068+0000","last_became_active":"2024-09-13T14:47:50.890368+0000","last_became_peered":"2024-09-13T14:47:50.890368+0000","last_unstale":"2024-09-13T14:50:52.370068+0000","last_undegraded":"2024-09-13T14:50:52.370068+0000","last_fullsized":"2024-09-13T14:50:52.370068+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:39.645908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:39.645908+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:29.800728+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":387,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:51.782210+0000","last_change":"2024-09-13T14:46:01.762127+0000","last_active":"2024-09-13T14:50:51.782210+0000","last_peered":"2024-09-13T14:50:51.782210+0000","last_clean":"2024-09-13T14:50:51.782210+0000","last_became_active":"2024-09-13T14:46:01.761795+0000","last_became_peered":"2024-09-13T14:46:01.761795+0000","last_unstale":"2024-09-13T14:50:51.782210+0000","last_undegraded":"2024-09-13T14:50:51.782210+0000","last_fullsized":"2024-09-13T14:50:51.782210+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:41:15.626775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":382,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:51.782624+0000","last_change":"2024-09-13T14:47:47.861718+0000","last_active":"2024-09-13T14:50:51.782624+0000","last_peered":"2024-09-13T14:50:51.782624+0000","last_clean":"2024-09-13T14:50:51.782624+0000","last_became_active":"2024-09-13T14:47:47.861023+0000","last_became_peered":"2024-09-13T14:47:47.861023+0000","last_unstale":"2024-09-13T14:50:51.782624+0000","last_undegraded":"2024-09-13T14:50:51.782624+0000","last_fullsized":"2024-09-13T14:50:51.782624+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:12:13.274053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":14,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:51.982836+0000","last_change":"2024-09-13T14:50:51.091067+0000","last_active":"2024-09-13T14:50:51.982836+0000","last_peered":"2024-09-13T14:50:51.982836+0000","last_clean":"2024-09-13T14:50:51.982836+0000","last_became_active":"2024-09-13T14:50:51.090690+0000","last_became_peered":"2024-09-13T14:50:51.090690+0000","last_unstale":"2024-09-13T14:50:51.982836+0000","last_undegraded":"2024-09-13T14:50:51.982836+0000","last_fullsized":"2024-09-13T14:50:51.982836+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:50:49.763471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:17.064482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"90'149","reported_seq":454,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.411800+0000","last_change":"2024-09-13T14:46:52.334715+0000","last_active":"2024-09-13T14:50:52.411800+0000","last_peered":"2024-09-13T14:50:52.411800+0000","last_clean":"2024-09-13T14:50:52.411800+0000","last_became_active":"2024-09-13T14:46:48.158685+0000","last_became_peered":"2024-09-13T14:46:48.158685+0000","last_unstale":"2024-09-13T14:50:52.411800+0000","last_undegraded":"2024-09-13T14:50:52.411800+0000","last_fullsized":"2024-09-13T14:50:52.411800+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:46:52.334301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:46:52.334301+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:46:03.579759+0000","scrub_duration":0.0050940330000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":663,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:51.982747+0000","last_change":"2024-09-13T14:47:47.861249+0000","last_active":"2024-09-13T14:50:51.982747+0000","last_peered":"2024-09-13T14:50:51.982747+0000","last_clean":"2024-09-13T14:50:51.982747+0000","last_became_active":"2024-09-13T14:47:47.860873+0000","last_became_peered":"2024-09-13T14:47:47.860873+0000","last_unstale":"2024-09-13T14:50:51.982747+0000","last_undegraded":"2024-09-13T14:50:51.982747+0000","last_fullsized":"2024-09-13T14:50:51.982747+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:37:50.709437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":91,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.369708+0000","last_change":"2024-09-13T14:47:47.870185+0000","last_active":"2024-09-13T14:50:52.369708+0000","last_peered":"2024-09-13T14:50:52.369708+0000","last_clean":"2024-09-13T14:50:52.369708+0000","last_became_active":"2024-09-13T14:47:47.869862+0000","last_became_peered":"2024-09-13T14:47:47.869862+0000","last_unstale":"2024-09-13T14:50:52.369708+0000","last_undegraded":"2024-09-13T14:50:52.369708+0000","last_fullsized":"2024-09-13T14:50:52.369708+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:07:02.812306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":14,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:51.982633+0000","last_change":"2024-09-13T14:50:50.781969+0000","last_active":"2024-09-13T14:50:51.982633+0000","last_peered":"2024-09-13T14:50:51.982633+0000","last_clean":"2024-09-13T14:50:51.982633+0000","last_became_active":"2024-09-13T14:50:50.781655+0000","last_became_peered":"2024-09-13T14:50:50.781655+0000","last_unstale":"2024-09-13T14:50:51.982633+0000","last_undegraded":"2024-09-13T14:50:51.982633+0000","last_fullsized":"2024-09-13T14:50:51.982633+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:50:49.763471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:34:21.057359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":557,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.053095+0000","last_change":"2024-09-13T14:47:31.684319+0000","last_active":"2024-09-13T14:50:52.053095+0000","last_peered":"2024-09-13T14:50:52.053095+0000","last_clean":"2024-09-13T14:50:52.053095+0000","last_became_active":"2024-09-13T14:47:29.415658+0000","last_became_peered":"2024-09-13T14:47:29.415658+0000","last_unstale":"2024-09-13T14:50:52.053095+0000","last_undegraded":"2024-09-13T14:50:52.053095+0000","last_fullsized":"2024-09-13T14:50:52.053095+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:31.681433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:31.681433+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:29:34.663513+0000","scrub_duration":0.0059601139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":686,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.411840+0000","last_change":"2024-09-13T14:46:04.780718+0000","last_active":"2024-09-13T14:50:52.411840+0000","last_peered":"2024-09-13T14:50:52.411840+0000","last_clean":"2024-09-13T14:50:52.411840+0000","last_became_active":"2024-09-13T14:46:04.780120+0000","last_became_peered":"2024-09-13T14:46:04.780120+0000","last_unstale":"2024-09-13T14:50:52.411840+0000","last_undegraded":"2024-09-13T14:50:52.411840+0000","last_fullsized":"2024-09-13T14:50:52.411840+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:57:14.475888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":590,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:51.982622+0000","last_change":"2024-09-13T14:37:49.160719+0000","last_active":"2024-09-13T14:50:51.982622+0000","last_peered":"2024-09-13T14:50:51.982622+0000","last_clean":"2024-09-13T14:50:51.982622+0000","last_became_active":"2024-09-13T14:37:49.160401+0000","last_became_peered":"2024-09-13T14:37:49.160401+0000","last_unstale":"2024-09-13T14:50:51.982622+0000","last_undegraded":"2024-09-13T14:50:51.982622+0000","last_fullsized":"2024-09-13T14:50:51.982622+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:21.737945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":225,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:51.782409+0000","last_change":"2024-09-13T14:37:48.172940+0000","last_active":"2024-09-13T14:50:51.782409+0000","last_peered":"2024-09-13T14:50:51.782409+0000","last_clean":"2024-09-13T14:50:51.782409+0000","last_became_active":"2024-09-13T14:37:48.172060+0000","last_became_peered":"2024-09-13T14:37:48.172060+0000","last_unstale":"2024-09-13T14:50:51.782409+0000","last_undegraded":"2024-09-13T14:50:51.782409+0000","last_fullsized":"2024-09-13T14:50:51.782409+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:07.480645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":14,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.419619+0000","last_change":"2024-09-13T14:50:51.091679+0000","last_active":"2024-09-13T14:50:52.419619+0000","last_peered":"2024-09-13T14:50:52.419619+0000","last_clean":"2024-09-13T14:50:52.419619+0000","last_became_active":"2024-09-13T14:50:51.091340+0000","last_became_peered":"2024-09-13T14:50:51.091340+0000","last_unstale":"2024-09-13T14:50:52.419619+0000","last_undegraded":"2024-09-13T14:50:52.419619+0000","last_fullsized":"2024-09-13T14:50:52.419619+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:50:49.763471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:33:38.187642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"90'149","reported_seq":520,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.370151+0000","last_change":"2024-09-13T14:47:50.884721+0000","last_active":"2024-09-13T14:50:52.370151+0000","last_peered":"2024-09-13T14:50:52.370151+0000","last_clean":"2024-09-13T14:50:52.370151+0000","last_became_active":"2024-09-13T14:47:50.880185+0000","last_became_peered":"2024-09-13T14:47:50.880185+0000","last_unstale":"2024-09-13T14:50:52.370151+0000","last_undegraded":"2024-09-13T14:50:52.370151+0000","last_fullsized":"2024-09-13T14:50:52.370151+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:34.603316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:34.603316+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:38.321686+0000","scrub_duration":0.0057131120000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":627,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.053033+0000","last_change":"2024-09-13T14:47:47.892973+0000","last_active":"2024-09-13T14:50:52.053033+0000","last_peered":"2024-09-13T14:50:52.053033+0000","last_clean":"2024-09-13T14:50:52.053033+0000","last_became_active":"2024-09-13T14:47:47.866986+0000","last_became_peered":"2024-09-13T14:47:47.866986+0000","last_unstale":"2024-09-13T14:50:52.053033+0000","last_undegraded":"2024-09-13T14:50:52.053033+0000","last_fullsized":"2024-09-13T14:50:52.053033+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:07.353290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":280,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.419160+0000","last_change":"2024-09-13T14:38:03.525682+0000","last_active":"2024-09-13T14:50:52.419160+0000","last_peered":"2024-09-13T14:50:52.419160+0000","last_clean":"2024-09-13T14:50:52.419160+0000","last_became_active":"2024-09-13T14:38:03.525406+0000","last_became_peered":"2024-09-13T14:38:03.525406+0000","last_unstale":"2024-09-13T14:50:52.419160+0000","last_undegraded":"2024-09-13T14:50:52.419160+0000","last_fullsized":"2024-09-13T14:50:52.419160+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:41:28.970300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":300,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.418980+0000","last_change":"2024-09-13T14:47:47.873233+0000","last_active":"2024-09-13T14:50:52.418980+0000","last_peered":"2024-09-13T14:50:52.418980+0000","last_clean":"2024-09-13T14:50:52.418980+0000","last_became_active":"2024-09-13T14:47:47.872788+0000","last_became_peered":"2024-09-13T14:47:47.872788+0000","last_unstale":"2024-09-13T14:50:52.418980+0000","last_undegraded":"2024-09-13T14:50:52.418980+0000","last_fullsized":"2024-09-13T14:50:52.418980+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:52.409108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":12,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.411661+0000","last_change":"2024-09-13T14:50:50.782599+0000","last_active":"2024-09-13T14:50:52.411661+0000","last_peered":"2024-09-13T14:50:52.411661+0000","last_clean":"2024-09-13T14:50:52.411661+0000","last_became_active":"2024-09-13T14:50:50.782273+0000","last_became_peered":"2024-09-13T14:50:50.782273+0000","last_unstale":"2024-09-13T14:50:52.411661+0000","last_undegraded":"2024-09-13T14:50:52.411661+0000","last_fullsized":"2024-09-13T14:50:52.411661+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:50:49.763471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:27:54.626945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"90'149","reported_seq":574,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.053037+0000","last_change":"2024-09-13T14:47:29.629488+0000","last_active":"2024-09-13T14:50:52.053037+0000","last_peered":"2024-09-13T14:50:52.053037+0000","last_clean":"2024-09-13T14:50:52.053037+0000","last_became_active":"2024-09-13T14:47:29.403279+0000","last_became_peered":"2024-09-13T14:47:29.403279+0000","last_unstale":"2024-09-13T14:50:52.053037+0000","last_undegraded":"2024-09-13T14:50:52.053037+0000","last_fullsized":"2024-09-13T14:50:52.053037+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:29.629342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:29.629342+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:14:55.249896+0000","scrub_duration":0.0099290990000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":574,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:51.784883+0000","last_change":"2024-09-13T14:46:04.782955+0000","last_active":"2024-09-13T14:50:51.784883+0000","last_peered":"2024-09-13T14:50:51.784883+0000","last_clean":"2024-09-13T14:50:51.784883+0000","last_became_active":"2024-09-13T14:46:04.782250+0000","last_became_peered":"2024-09-13T14:46:04.782250+0000","last_unstale":"2024-09-13T14:50:51.784883+0000","last_undegraded":"2024-09-13T14:50:51.784883+0000","last_fullsized":"2024-09-13T14:50:51.784883+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:24.596922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":91,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.369599+0000","last_change":"2024-09-13T14:47:47.866684+0000","last_active":"2024-09-13T14:50:52.369599+0000","last_peered":"2024-09-13T14:50:52.369599+0000","last_clean":"2024-09-13T14:50:52.369599+0000","last_became_active":"2024-09-13T14:47:47.866400+0000","last_became_peered":"2024-09-13T14:47:47.866400+0000","last_unstale":"2024-09-13T14:50:52.369599+0000","last_undegraded":"2024-09-13T14:50:52.369599+0000","last_fullsized":"2024-09-13T14:50:52.369599+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:08:34.289658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":91,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.370497+0000","last_change":"2024-09-13T14:47:47.870218+0000","last_active":"2024-09-13T14:50:52.370497+0000","last_peered":"2024-09-13T14:50:52.370497+0000","last_clean":"2024-09-13T14:50:52.370497+0000","last_became_active":"2024-09-13T14:47:47.869907+0000","last_became_peered":"2024-09-13T14:47:47.869907+0000","last_unstale":"2024-09-13T14:50:52.370497+0000","last_undegraded":"2024-09-13T14:50:52.370497+0000","last_fullsized":"2024-09-13T14:50:52.370497+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:41:49.141441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":14,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.419285+0000","last_change":"2024-09-13T14:50:50.779542+0000","last_active":"2024-09-13T14:50:52.419285+0000","last_peered":"2024-09-13T14:50:52.419285+0000","last_clean":"2024-09-13T14:50:52.419285+0000","last_became_active":"2024-09-13T14:50:50.779111+0000","last_became_peered":"2024-09-13T14:50:50.779111+0000","last_unstale":"2024-09-13T14:50:52.419285+0000","last_undegraded":"2024-09-13T14:50:52.419285+0000","last_fullsized":"2024-09-13T14:50:52.419285+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:50:49.763471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:38:15.985311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"90'149","reported_seq":476,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:56.476087+0000","last_change":"2024-09-13T14:47:32.275753+0000","last_active":"2024-09-13T14:50:56.476087+0000","last_peered":"2024-09-13T14:50:56.476087+0000","last_clean":"2024-09-13T14:50:56.476087+0000","last_became_active":"2024-09-13T14:47:29.401066+0000","last_became_peered":"2024-09-13T14:47:29.401066+0000","last_unstale":"2024-09-13T14:50:56.476087+0000","last_undegraded":"2024-09-13T14:50:56.476087+0000","last_fullsized":"2024-09-13T14:50:56.476087+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:32.275562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:32.275562+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:43:41.056775+0000","scrub_duration":0.0054674260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":1131,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.411950+0000","last_change":"2024-09-13T14:47:50.892842+0000","last_active":"2024-09-13T14:50:52.411950+0000","last_peered":"2024-09-13T14:50:52.411950+0000","last_clean":"2024-09-13T14:50:52.411950+0000","last_became_active":"2024-09-13T14:47:50.892237+0000","last_became_peered":"2024-09-13T14:47:50.892237+0000","last_unstale":"2024-09-13T14:50:52.411950+0000","last_undegraded":"2024-09-13T14:50:52.411950+0000","last_fullsized":"2024-09-13T14:50:52.411950+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:20:05.796762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":8388646,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":386,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:51.782518+0000","last_change":"2024-09-13T14:38:03.528133+0000","last_active":"2024-09-13T14:50:51.782518+0000","last_peered":"2024-09-13T14:50:51.782518+0000","last_clean":"2024-09-13T14:50:51.782518+0000","last_became_active":"2024-09-13T14:38:03.527829+0000","last_became_peered":"2024-09-13T14:38:03.527829+0000","last_unstale":"2024-09-13T14:50:51.782518+0000","last_undegraded":"2024-09-13T14:50:51.782518+0000","last_fullsized":"2024-09-13T14:50:51.782518+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:48.254169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":91,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.369707+0000","last_change":"2024-09-13T14:47:47.877272+0000","last_active":"2024-09-13T14:50:52.369707+0000","last_peered":"2024-09-13T14:50:52.369707+0000","last_clean":"2024-09-13T14:50:52.369707+0000","last_became_active":"2024-09-13T14:47:47.877003+0000","last_became_peered":"2024-09-13T14:47:47.877003+0000","last_unstale":"2024-09-13T14:50:52.369707+0000","last_undegraded":"2024-09-13T14:50:52.369707+0000","last_fullsized":"2024-09-13T14:50:52.369707+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:47:23.172452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":12,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.412152+0000","last_change":"2024-09-13T14:50:50.783992+0000","last_active":"2024-09-13T14:50:52.412152+0000","last_peered":"2024-09-13T14:50:52.412152+0000","last_clean":"2024-09-13T14:50:52.412152+0000","last_became_active":"2024-09-13T14:50:50.783687+0000","last_became_peered":"2024-09-13T14:50:50.783687+0000","last_unstale":"2024-09-13T14:50:52.412152+0000","last_undegraded":"2024-09-13T14:50:52.412152+0000","last_fullsized":"2024-09-13T14:50:52.412152+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:50:49.763471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:30.412734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"90'149","reported_seq":523,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:51.983229+0000","last_change":"2024-09-13T14:47:33.067298+0000","last_active":"2024-09-13T14:50:51.983229+0000","last_peered":"2024-09-13T14:50:51.983229+0000","last_clean":"2024-09-13T14:50:51.983229+0000","last_became_active":"2024-09-13T14:47:29.419416+0000","last_became_peered":"2024-09-13T14:47:29.419416+0000","last_unstale":"2024-09-13T14:50:51.983229+0000","last_undegraded":"2024-09-13T14:50:51.983229+0000","last_fullsized":"2024-09-13T14:50:51.983229+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:33.066954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:33.066954+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:05:42.808060+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":200,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T14:50:59.742535+0000","last_change":"2024-09-13T14:46:01.763906+0000","last_active":"2024-09-13T14:50:59.742535+0000","last_peered":"2024-09-13T14:50:59.742535+0000","last_clean":"2024-09-13T14:50:59.742535+0000","last_became_active":"2024-09-13T14:46:01.763542+0000","last_became_peered":"2024-09-13T14:46:01.763542+0000","last_unstale":"2024-09-13T14:50:59.742535+0000","last_undegraded":"2024-09-13T14:50:59.742535+0000","last_fullsized":"2024-09-13T14:50:59.742535+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:09:36.652767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":1379,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.370276+0000","last_change":"2024-09-13T14:47:50.883365+0000","last_active":"2024-09-13T14:50:52.370276+0000","last_peered":"2024-09-13T14:50:52.370276+0000","last_clean":"2024-09-13T14:50:52.370276+0000","last_became_active":"2024-09-13T14:47:50.882609+0000","last_became_peered":"2024-09-13T14:47:50.882609+0000","last_unstale":"2024-09-13T14:50:52.370276+0000","last_undegraded":"2024-09-13T14:50:52.370276+0000","last_fullsized":"2024-09-13T14:50:52.370276+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:12:17.961575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":91,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.369821+0000","last_change":"2024-09-13T14:47:47.866979+0000","last_active":"2024-09-13T14:50:52.369821+0000","last_peered":"2024-09-13T14:50:52.369821+0000","last_clean":"2024-09-13T14:50:52.369821+0000","last_became_active":"2024-09-13T14:47:47.866634+0000","last_became_peered":"2024-09-13T14:47:47.866634+0000","last_unstale":"2024-09-13T14:50:52.369821+0000","last_undegraded":"2024-09-13T14:50:52.369821+0000","last_fullsized":"2024-09-13T14:50:52.369821+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:47:09.696139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":12,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.053422+0000","last_change":"2024-09-13T14:50:50.782813+0000","last_active":"2024-09-13T14:50:52.053422+0000","last_peered":"2024-09-13T14:50:52.053422+0000","last_clean":"2024-09-13T14:50:52.053422+0000","last_became_active":"2024-09-13T14:50:50.782515+0000","last_became_peered":"2024-09-13T14:50:50.782515+0000","last_unstale":"2024-09-13T14:50:52.053422+0000","last_undegraded":"2024-09-13T14:50:52.053422+0000","last_fullsized":"2024-09-13T14:50:52.053422+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:50:49.763471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:20.624059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"90'149","reported_seq":529,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.419451+0000","last_change":"2024-09-13T14:47:29.556225+0000","last_active":"2024-09-13T14:50:52.419451+0000","last_peered":"2024-09-13T14:50:52.419451+0000","last_clean":"2024-09-13T14:50:52.419451+0000","last_became_active":"2024-09-13T14:47:29.408774+0000","last_became_peered":"2024-09-13T14:47:29.408774+0000","last_unstale":"2024-09-13T14:50:52.419451+0000","last_undegraded":"2024-09-13T14:50:52.419451+0000","last_fullsized":"2024-09-13T14:50:52.419451+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:29.556149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:29.556149+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:35.691178+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":91,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.369649+0000","last_change":"2024-09-13T14:47:47.876562+0000","last_active":"2024-09-13T14:50:52.369649+0000","last_peered":"2024-09-13T14:50:52.369649+0000","last_clean":"2024-09-13T14:50:52.369649+0000","last_became_active":"2024-09-13T14:47:47.876297+0000","last_became_peered":"2024-09-13T14:47:47.876297+0000","last_unstale":"2024-09-13T14:50:52.369649+0000","last_undegraded":"2024-09-13T14:50:52.369649+0000","last_fullsized":"2024-09-13T14:50:52.369649+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:03:27.761394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":902,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.053533+0000","last_change":"2024-09-13T14:38:04.596192+0000","last_active":"2024-09-13T14:50:52.053533+0000","last_peered":"2024-09-13T14:50:52.053533+0000","last_clean":"2024-09-13T14:50:52.053533+0000","last_became_active":"2024-09-13T14:38:04.593299+0000","last_became_peered":"2024-09-13T14:38:04.593299+0000","last_unstale":"2024-09-13T14:50:52.053533+0000","last_undegraded":"2024-09-13T14:50:52.053533+0000","last_fullsized":"2024-09-13T14:50:52.053533+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:34.241410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":280,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.419643+0000","last_change":"2024-09-13T14:38:03.533120+0000","last_active":"2024-09-13T14:50:52.419643+0000","last_peered":"2024-09-13T14:50:52.419643+0000","last_clean":"2024-09-13T14:50:52.419643+0000","last_became_active":"2024-09-13T14:38:03.532829+0000","last_became_peered":"2024-09-13T14:38:03.532829+0000","last_unstale":"2024-09-13T14:50:52.419643+0000","last_undegraded":"2024-09-13T14:50:52.419643+0000","last_fullsized":"2024-09-13T14:50:52.419643+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:24.514661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":12,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.053374+0000","last_change":"2024-09-13T14:50:50.782589+0000","last_active":"2024-09-13T14:50:52.053374+0000","last_peered":"2024-09-13T14:50:52.053374+0000","last_clean":"2024-09-13T14:50:52.053374+0000","last_became_active":"2024-09-13T14:50:50.782292+0000","last_became_peered":"2024-09-13T14:50:50.782292+0000","last_unstale":"2024-09-13T14:50:52.053374+0000","last_undegraded":"2024-09-13T14:50:52.053374+0000","last_fullsized":"2024-09-13T14:50:52.053374+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:50:49.763471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:01:30.753189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"90'149","reported_seq":476,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T14:51:00.392783+0000","last_change":"2024-09-13T14:47:30.528588+0000","last_active":"2024-09-13T14:51:00.392783+0000","last_peered":"2024-09-13T14:51:00.392783+0000","last_clean":"2024-09-13T14:51:00.392783+0000","last_became_active":"2024-09-13T14:47:29.413621+0000","last_became_peered":"2024-09-13T14:47:29.413621+0000","last_unstale":"2024-09-13T14:51:00.392783+0000","last_undegraded":"2024-09-13T14:51:00.392783+0000","last_fullsized":"2024-09-13T14:51:00.392783+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:30.528112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:30.528112+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:30.462009+0000","scrub_duration":0.012751485,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":682,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:51.983401+0000","last_change":"2024-09-13T14:38:53.938227+0000","last_active":"2024-09-13T14:50:51.983401+0000","last_peered":"2024-09-13T14:50:51.983401+0000","last_clean":"2024-09-13T14:50:51.983401+0000","last_became_active":"2024-09-13T14:38:53.902867+0000","last_became_peered":"2024-09-13T14:38:53.902867+0000","last_unstale":"2024-09-13T14:50:51.983401+0000","last_undegraded":"2024-09-13T14:50:51.983401+0000","last_fullsized":"2024-09-13T14:50:51.983401+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:16.831756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":561,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.370498+0000","last_change":"2024-09-13T14:47:50.884706+0000","last_active":"2024-09-13T14:50:52.370498+0000","last_peered":"2024-09-13T14:50:52.370498+0000","last_clean":"2024-09-13T14:50:52.370498+0000","last_became_active":"2024-09-13T14:47:50.879750+0000","last_became_peered":"2024-09-13T14:47:50.879750+0000","last_unstale":"2024-09-13T14:50:52.370498+0000","last_undegraded":"2024-09-13T14:50:52.370498+0000","last_fullsized":"2024-09-13T14:50:52.370498+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:34:32.941266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":664,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:51.982558+0000","last_change":"2024-09-13T14:38:53.900107+0000","last_active":"2024-09-13T14:50:51.982558+0000","last_peered":"2024-09-13T14:50:51.982558+0000","last_clean":"2024-09-13T14:50:51.982558+0000","last_became_active":"2024-09-13T14:38:53.899556+0000","last_became_peered":"2024-09-13T14:38:53.899556+0000","last_unstale":"2024-09-13T14:50:51.982558+0000","last_undegraded":"2024-09-13T14:50:51.982558+0000","last_fullsized":"2024-09-13T14:50:51.982558+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:15:47.918163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":14,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:51.982937+0000","last_change":"2024-09-13T14:50:50.776299+0000","last_active":"2024-09-13T14:50:51.982937+0000","last_peered":"2024-09-13T14:50:51.982937+0000","last_clean":"2024-09-13T14:50:51.982937+0000","last_became_active":"2024-09-13T14:50:50.776003+0000","last_became_peered":"2024-09-13T14:50:50.776003+0000","last_unstale":"2024-09-13T14:50:51.982937+0000","last_undegraded":"2024-09-13T14:50:51.982937+0000","last_fullsized":"2024-09-13T14:50:51.982937+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:50:49.763471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:13:05.387802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"90'149","reported_seq":521,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:51.982866+0000","last_change":"2024-09-13T14:47:29.999643+0000","last_active":"2024-09-13T14:50:51.982866+0000","last_peered":"2024-09-13T14:50:51.982866+0000","last_clean":"2024-09-13T14:50:51.982866+0000","last_became_active":"2024-09-13T14:47:29.410951+0000","last_became_peered":"2024-09-13T14:47:29.410951+0000","last_unstale":"2024-09-13T14:50:51.982866+0000","last_undegraded":"2024-09-13T14:50:51.982866+0000","last_fullsized":"2024-09-13T14:50:51.982866+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:29.999519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:29.999519+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:04:54.887884+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":305,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.418985+0000","last_change":"2024-09-13T14:47:47.883475+0000","last_active":"2024-09-13T14:50:52.418985+0000","last_peered":"2024-09-13T14:50:52.418985+0000","last_clean":"2024-09-13T14:50:52.418985+0000","last_became_active":"2024-09-13T14:47:47.871525+0000","last_became_peered":"2024-09-13T14:47:47.871525+0000","last_unstale":"2024-09-13T14:50:52.418985+0000","last_undegraded":"2024-09-13T14:50:52.418985+0000","last_fullsized":"2024-09-13T14:50:52.418985+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:33.903432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":180,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.053563+0000","last_change":"2024-09-13T14:46:01.781508+0000","last_active":"2024-09-13T14:50:52.053563+0000","last_peered":"2024-09-13T14:50:52.053563+0000","last_clean":"2024-09-13T14:50:52.053563+0000","last_became_active":"2024-09-13T14:46:01.767096+0000","last_became_peered":"2024-09-13T14:46:01.767096+0000","last_unstale":"2024-09-13T14:50:52.053563+0000","last_undegraded":"2024-09-13T14:50:52.053563+0000","last_fullsized":"2024-09-13T14:50:52.053563+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:20:01.119189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":371,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T14:51:00.391693+0000","last_change":"2024-09-13T14:37:49.172218+0000","last_active":"2024-09-13T14:51:00.391693+0000","last_peered":"2024-09-13T14:51:00.391693+0000","last_clean":"2024-09-13T14:51:00.391693+0000","last_became_active":"2024-09-13T14:37:49.171718+0000","last_became_peered":"2024-09-13T14:37:49.171718+0000","last_unstale":"2024-09-13T14:51:00.391693+0000","last_undegraded":"2024-09-13T14:51:00.391693+0000","last_fullsized":"2024-09-13T14:51:00.391693+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:46:54.626521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":401,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:51.782331+0000","last_change":"2024-09-13T14:47:47.868409+0000","last_active":"2024-09-13T14:50:51.782331+0000","last_peered":"2024-09-13T14:50:51.782331+0000","last_clean":"2024-09-13T14:50:51.782331+0000","last_became_active":"2024-09-13T14:47:47.867712+0000","last_became_peered":"2024-09-13T14:47:47.867712+0000","last_unstale":"2024-09-13T14:50:51.782331+0000","last_undegraded":"2024-09-13T14:50:51.782331+0000","last_fullsized":"2024-09-13T14:50:51.782331+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:57:14.647493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":372,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:51.782752+0000","last_change":"2024-09-13T14:37:48.128447+0000","last_active":"2024-09-13T14:50:51.782752+0000","last_peered":"2024-09-13T14:50:51.782752+0000","last_clean":"2024-09-13T14:50:51.782752+0000","last_became_active":"2024-09-13T14:37:48.128143+0000","last_became_peered":"2024-09-13T14:37:48.128143+0000","last_unstale":"2024-09-13T14:50:51.782752+0000","last_undegraded":"2024-09-13T14:50:51.782752+0000","last_fullsized":"2024-09-13T14:50:51.782752+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:22.204039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":91,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.370360+0000","last_change":"2024-09-13T14:47:47.867935+0000","last_active":"2024-09-13T14:50:52.370360+0000","last_peered":"2024-09-13T14:50:52.370360+0000","last_clean":"2024-09-13T14:50:52.370360+0000","last_became_active":"2024-09-13T14:47:47.867616+0000","last_became_peered":"2024-09-13T14:47:47.867616+0000","last_unstale":"2024-09-13T14:50:52.370360+0000","last_undegraded":"2024-09-13T14:50:52.370360+0000","last_fullsized":"2024-09-13T14:50:52.370360+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:52:38.619916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":145,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T14:50:59.743747+0000","last_change":"2024-09-13T14:38:53.954281+0000","last_active":"2024-09-13T14:50:59.743747+0000","last_peered":"2024-09-13T14:50:59.743747+0000","last_clean":"2024-09-13T14:50:59.743747+0000","last_became_active":"2024-09-13T14:38:53.900912+0000","last_became_peered":"2024-09-13T14:38:53.900912+0000","last_unstale":"2024-09-13T14:50:59.743747+0000","last_undegraded":"2024-09-13T14:50:59.743747+0000","last_fullsized":"2024-09-13T14:50:59.743747+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:47.876275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"90'149","reported_seq":524,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:51.785036+0000","last_change":"2024-09-13T14:47:38.468095+0000","last_active":"2024-09-13T14:50:51.785036+0000","last_peered":"2024-09-13T14:50:51.785036+0000","last_clean":"2024-09-13T14:50:51.785036+0000","last_became_active":"2024-09-13T14:47:38.467526+0000","last_became_peered":"2024-09-13T14:47:38.467526+0000","last_unstale":"2024-09-13T14:50:51.785036+0000","last_undegraded":"2024-09-13T14:50:51.785036+0000","last_fullsized":"2024-09-13T14:50:51.785036+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:31.013128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:31.013128+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:17:20.508142+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":206,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.412077+0000","last_change":"2024-09-13T14:47:47.861842+0000","last_active":"2024-09-13T14:50:52.412077+0000","last_peered":"2024-09-13T14:50:52.412077+0000","last_clean":"2024-09-13T14:50:52.412077+0000","last_became_active":"2024-09-13T14:47:47.861492+0000","last_became_peered":"2024-09-13T14:47:47.861492+0000","last_unstale":"2024-09-13T14:50:52.412077+0000","last_undegraded":"2024-09-13T14:50:52.412077+0000","last_fullsized":"2024-09-13T14:50:52.412077+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:17.279533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":605,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:51.982544+0000","last_change":"2024-09-13T14:38:53.899535+0000","last_active":"2024-09-13T14:50:51.982544+0000","last_peered":"2024-09-13T14:50:51.982544+0000","last_clean":"2024-09-13T14:50:51.982544+0000","last_became_active":"2024-09-13T14:38:53.899183+0000","last_became_peered":"2024-09-13T14:38:53.899183+0000","last_unstale":"2024-09-13T14:50:51.982544+0000","last_undegraded":"2024-09-13T14:50:51.982544+0000","last_fullsized":"2024-09-13T14:50:51.982544+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:33.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":279,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.419595+0000","last_change":"2024-09-13T14:38:03.543426+0000","last_active":"2024-09-13T14:50:52.419595+0000","last_peered":"2024-09-13T14:50:52.419595+0000","last_clean":"2024-09-13T14:50:52.419595+0000","last_became_active":"2024-09-13T14:38:03.542661+0000","last_became_peered":"2024-09-13T14:38:03.542661+0000","last_unstale":"2024-09-13T14:50:52.419595+0000","last_undegraded":"2024-09-13T14:50:52.419595+0000","last_fullsized":"2024-09-13T14:50:52.419595+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:54.050127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":641,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:51.982419+0000","last_change":"2024-09-13T14:47:47.868624+0000","last_active":"2024-09-13T14:50:51.982419+0000","last_peered":"2024-09-13T14:50:51.982419+0000","last_clean":"2024-09-13T14:50:51.982419+0000","last_became_active":"2024-09-13T14:47:47.868031+0000","last_became_peered":"2024-09-13T14:47:47.868031+0000","last_unstale":"2024-09-13T14:50:51.982419+0000","last_undegraded":"2024-09-13T14:50:51.982419+0000","last_fullsized":"2024-09-13T14:50:51.982419+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:05:25.365023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":219,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.412322+0000","last_change":"2024-09-13T14:47:47.859825+0000","last_active":"2024-09-13T14:50:52.412322+0000","last_peered":"2024-09-13T14:50:52.412322+0000","last_clean":"2024-09-13T14:50:52.412322+0000","last_became_active":"2024-09-13T14:47:47.859069+0000","last_became_peered":"2024-09-13T14:47:47.859069+0000","last_unstale":"2024-09-13T14:50:52.412322+0000","last_undegraded":"2024-09-13T14:50:52.412322+0000","last_fullsized":"2024-09-13T14:50:52.412322+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:18:36.452321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":672,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:51.982610+0000","last_change":"2024-09-13T14:47:47.868945+0000","last_active":"2024-09-13T14:50:51.982610+0000","last_peered":"2024-09-13T14:50:51.982610+0000","last_clean":"2024-09-13T14:50:51.982610+0000","last_became_active":"2024-09-13T14:47:47.868296+0000","last_became_peered":"2024-09-13T14:47:47.868296+0000","last_unstale":"2024-09-13T14:50:51.982610+0000","last_undegraded":"2024-09-13T14:50:51.982610+0000","last_fullsized":"2024-09-13T14:50:51.982610+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:22:24.778693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":287,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.419474+0000","last_change":"2024-09-13T14:38:53.892948+0000","last_active":"2024-09-13T14:50:52.419474+0000","last_peered":"2024-09-13T14:50:52.419474+0000","last_clean":"2024-09-13T14:50:52.419474+0000","last_became_active":"2024-09-13T14:38:53.892665+0000","last_became_peered":"2024-09-13T14:38:53.892665+0000","last_unstale":"2024-09-13T14:50:52.419474+0000","last_undegraded":"2024-09-13T14:50:52.419474+0000","last_fullsized":"2024-09-13T14:50:52.419474+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:08.292359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":91,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.369599+0000","last_change":"2024-09-13T14:47:47.871164+0000","last_active":"2024-09-13T14:50:52.369599+0000","last_peered":"2024-09-13T14:50:52.369599+0000","last_clean":"2024-09-13T14:50:52.369599+0000","last_became_active":"2024-09-13T14:47:47.870862+0000","last_became_peered":"2024-09-13T14:47:47.870862+0000","last_unstale":"2024-09-13T14:50:52.369599+0000","last_undegraded":"2024-09-13T14:50:52.369599+0000","last_fullsized":"2024-09-13T14:50:52.369599+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:41:34.392081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":661,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:51.982979+0000","last_change":"2024-09-13T14:50:27.490554+0000","last_active":"2024-09-13T14:50:51.982979+0000","last_peered":"2024-09-13T14:50:51.982979+0000","last_clean":"2024-09-13T14:50:51.982979+0000","last_became_active":"2024-09-13T14:50:27.490211+0000","last_became_peered":"2024-09-13T14:50:27.490211+0000","last_unstale":"2024-09-13T14:50:51.982979+0000","last_undegraded":"2024-09-13T14:50:51.982979+0000","last_fullsized":"2024-09-13T14:50:51.982979+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:57:01.947741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":454,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:51.782222+0000","last_change":"2024-09-13T14:47:47.868246+0000","last_active":"2024-09-13T14:50:51.782222+0000","last_peered":"2024-09-13T14:50:51.782222+0000","last_clean":"2024-09-13T14:50:51.782222+0000","last_became_active":"2024-09-13T14:47:47.867817+0000","last_became_peered":"2024-09-13T14:47:47.867817+0000","last_unstale":"2024-09-13T14:50:51.782222+0000","last_undegraded":"2024-09-13T14:50:51.782222+0000","last_fullsized":"2024-09-13T14:50:51.782222+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:07.612451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":172,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:51.785177+0000","last_change":"2024-09-13T14:50:27.510133+0000","last_active":"2024-09-13T14:50:51.785177+0000","last_peered":"2024-09-13T14:50:51.785177+0000","last_clean":"2024-09-13T14:50:51.785177+0000","last_became_active":"2024-09-13T14:50:27.508774+0000","last_became_peered":"2024-09-13T14:50:27.508774+0000","last_unstale":"2024-09-13T14:50:51.785177+0000","last_undegraded":"2024-09-13T14:50:51.785177+0000","last_fullsized":"2024-09-13T14:50:51.785177+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:31:04.531746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":279,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.419387+0000","last_change":"2024-09-13T14:38:03.527021+0000","last_active":"2024-09-13T14:50:52.419387+0000","last_peered":"2024-09-13T14:50:52.419387+0000","last_clean":"2024-09-13T14:50:52.419387+0000","last_became_active":"2024-09-13T14:38:03.526698+0000","last_became_peered":"2024-09-13T14:38:03.526698+0000","last_unstale":"2024-09-13T14:50:52.419387+0000","last_undegraded":"2024-09-13T14:50:52.419387+0000","last_fullsized":"2024-09-13T14:50:52.419387+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:01:35.734086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":398,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:51.782326+0000","last_change":"2024-09-13T14:47:47.866784+0000","last_active":"2024-09-13T14:50:51.782326+0000","last_peered":"2024-09-13T14:50:51.782326+0000","last_clean":"2024-09-13T14:50:51.782326+0000","last_became_active":"2024-09-13T14:47:47.866483+0000","last_became_peered":"2024-09-13T14:47:47.866483+0000","last_unstale":"2024-09-13T14:50:51.782326+0000","last_undegraded":"2024-09-13T14:50:51.782326+0000","last_fullsized":"2024-09-13T14:50:51.782326+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:15:53.119741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":206,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.411505+0000","last_change":"2024-09-13T14:47:47.880946+0000","last_active":"2024-09-13T14:50:52.411505+0000","last_peered":"2024-09-13T14:50:52.411505+0000","last_clean":"2024-09-13T14:50:52.411505+0000","last_became_active":"2024-09-13T14:47:47.866534+0000","last_became_peered":"2024-09-13T14:47:47.866534+0000","last_unstale":"2024-09-13T14:50:52.411505+0000","last_undegraded":"2024-09-13T14:50:52.411505+0000","last_fullsized":"2024-09-13T14:50:52.411505+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:01:01.384873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":391,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:51.782129+0000","last_change":"2024-09-13T14:38:03.526333+0000","last_active":"2024-09-13T14:50:51.782129+0000","last_peered":"2024-09-13T14:50:51.782129+0000","last_clean":"2024-09-13T14:50:51.782129+0000","last_became_active":"2024-09-13T14:38:03.526001+0000","last_became_peered":"2024-09-13T14:38:03.526001+0000","last_unstale":"2024-09-13T14:50:51.782129+0000","last_undegraded":"2024-09-13T14:50:51.782129+0000","last_fullsized":"2024-09-13T14:50:51.782129+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:25:25.181271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":193,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.412122+0000","last_change":"2024-09-13T14:37:49.149582+0000","last_active":"2024-09-13T14:50:52.412122+0000","last_peered":"2024-09-13T14:50:52.412122+0000","last_clean":"2024-09-13T14:50:52.412122+0000","last_became_active":"2024-09-13T14:37:49.148525+0000","last_became_peered":"2024-09-13T14:37:49.148525+0000","last_unstale":"2024-09-13T14:50:52.412122+0000","last_undegraded":"2024-09-13T14:50:52.412122+0000","last_fullsized":"2024-09-13T14:50:52.412122+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:48.655665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":686,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:51.982865+0000","last_change":"2024-09-13T14:46:01.766580+0000","last_active":"2024-09-13T14:50:51.982865+0000","last_peered":"2024-09-13T14:50:51.982865+0000","last_clean":"2024-09-13T14:50:51.982865+0000","last_became_active":"2024-09-13T14:46:01.766257+0000","last_became_peered":"2024-09-13T14:46:01.766257+0000","last_unstale":"2024-09-13T14:50:51.982865+0000","last_undegraded":"2024-09-13T14:50:51.982865+0000","last_fullsized":"2024-09-13T14:50:51.982865+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:01.126109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":651,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:51.983231+0000","last_change":"2024-09-13T14:46:01.769647+0000","last_active":"2024-09-13T14:50:51.983231+0000","last_peered":"2024-09-13T14:50:51.983231+0000","last_clean":"2024-09-13T14:50:51.983231+0000","last_became_active":"2024-09-13T14:46:01.769228+0000","last_became_peered":"2024-09-13T14:46:01.769228+0000","last_unstale":"2024-09-13T14:50:51.983231+0000","last_undegraded":"2024-09-13T14:50:51.983231+0000","last_fullsized":"2024-09-13T14:50:51.983231+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:58.696024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":381,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:51.781986+0000","last_change":"2024-09-13T14:47:47.860727+0000","last_active":"2024-09-13T14:50:51.781986+0000","last_peered":"2024-09-13T14:50:51.781986+0000","last_clean":"2024-09-13T14:50:51.781986+0000","last_became_active":"2024-09-13T14:47:47.860377+0000","last_became_peered":"2024-09-13T14:47:47.860377+0000","last_unstale":"2024-09-13T14:50:51.781986+0000","last_undegraded":"2024-09-13T14:50:51.781986+0000","last_fullsized":"2024-09-13T14:50:51.781986+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:27:00.868699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":299,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.419540+0000","last_change":"2024-09-13T14:46:01.775237+0000","last_active":"2024-09-13T14:50:52.419540+0000","last_peered":"2024-09-13T14:50:52.419540+0000","last_clean":"2024-09-13T14:50:52.419540+0000","last_became_active":"2024-09-13T14:46:01.765025+0000","last_became_peered":"2024-09-13T14:46:01.765025+0000","last_unstale":"2024-09-13T14:50:52.419540+0000","last_undegraded":"2024-09-13T14:50:52.419540+0000","last_fullsized":"2024-09-13T14:50:52.419540+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:29:16.089042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1746,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":2154,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25132,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":151,"num_bytes_recovered":8388646,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3129,"ondisk_log_size":3129,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234486,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":639640,"kb_used_data":4748,"kb_used_omap":1,"kb_used_meta":634878,"kb_avail":93105512,"statfs":{"total":95995035648,"available":95340044288,"internally_reserved":0,"allocated":4861952,"data_stored":2904858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":650115531},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":239,"seq":1026497183904,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":508752,"kb_used_data":9156,"kb_used_omap":8,"kb_used_meta":499575,"kb_avail":93236400,"statfs":{"total":95995035648,"available":95474073600,"internally_reserved":0,"allocated":9375744,"data_stored":7426773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8453,"internal_metadata":511565563},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757751,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":914700,"kb_used_data":9088,"kb_used_omap":1,"kb_used_meta":905598,"kb_avail":92830452,"statfs":{"total":95995035648,"available":95058382848,"internally_reserved":0,"allocated":9306112,"data_stored":7357549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":927332806},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019395,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":747924,"kb_used_data":4744,"kb_used_omap":1,"kb_used_meta":743166,"kb_avail":92997228,"statfs":{"total":95995035648,"available":95229161472,"internally_reserved":0,"allocated":4857856,"data_stored":2904846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":761002438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435577,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":789444,"kb_used_data":5296,"kb_used_omap":5,"kb_used_meta":784122,"kb_avail":92955708,"statfs":{"total":95995035648,"available":95186644992,"internally_reserved":0,"allocated":5423104,"data_stored":3482403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":802941054},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":104,"seq":446676599105,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297672,"kb_used_data":4980,"kb_used_omap":7,"kb_used_meta":292664,"kb_avail":93447480,"statfs":{"total":95995035648,"available":95690219520,"internally_reserved":0,"allocated":5099520,"data_stored":3155249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8010,"internal_metadata":299688118},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":99,"seq":425201762632,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":175156,"kb_used_data":8808,"kb_used_omap":7,"kb_used_meta":166328,"kb_avail":93569996,"statfs":{"total":95995035648,"available":95815675904,"internally_reserved":0,"allocated":9019392,"data_stored":7085995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7759,"internal_metadata":170320305},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926155,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":848892,"kb_used_data":5040,"kb_used_omap":7,"kb_used_meta":843832,"kb_avail":92896260,"statfs":{"total":95995035648,"available":95125770240,"internally_reserved":0,"allocated":5160960,"data_stored":3220229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":864084264},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6860,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:51:04.034 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-13T14:51:04.034 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:51:04.108 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:04.130 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:04.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:04.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:04.232 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:04.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:04.255 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:04.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:04.356 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:04.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:04.380 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:04.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:04.481 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:04.503 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:04.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:04.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:04.606 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:04.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:04.629 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:04.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:04.730 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:04.752 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:04.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:04.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:04.854 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:04.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:04.879 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:04.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:04.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:04 smithi102 ceph-mon[115269]: pgmap v1300: 109 pgs: 109 active+clean; 4.6 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-13T14:51:04.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:04 smithi102 ceph-mon[115269]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:51:04.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:04 smithi102 ceph-mon[115269]: Cluster is now healthy 2024-09-13T14:51:04.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:04 smithi102 ceph-mon[110183]: pgmap v1300: 109 pgs: 109 active+clean; 4.6 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-13T14:51:04.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:04 smithi102 ceph-mon[110183]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:51:04.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:04 smithi102 ceph-mon[110183]: Cluster is now healthy 2024-09-13T14:51:04.980 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:05.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:05.003 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:05.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:05.105 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:05.127 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:05.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:05.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:05.229 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:05.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:04 smithi195 ceph-mon[104262]: pgmap v1300: 109 pgs: 109 active+clean; 4.6 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-13T14:51:05.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:04 smithi195 ceph-mon[104262]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:51:05.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:04 smithi195 ceph-mon[104262]: Cluster is now healthy 2024-09-13T14:51:05.251 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:05.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:05.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:05.353 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:05.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:05.377 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:05.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:05.478 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:05.499 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:05.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:05.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:05.601 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:05.622 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:05.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:05.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:05.724 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:05.745 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:05.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:05.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:05.847 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:05.873 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:05.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:05.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:05.975 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:05.987 DEBUG:teuthology.orchestra.run.smithi102:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-09-13T14:51:05.988 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_4 2024-09-13T14:51:05.988 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:51:05.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:05.997 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:05.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:05.998 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-13T14:51:05.998 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-13T14:51:06.099 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:06.100 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:hints = 1 2024-09-13T14:51:06.100 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-13T14:51:06.100 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Object prefix: benchmark_data_smithi102_348563 2024-09-13T14:51:06.100 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:51:06.100 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 0 0 0 0 0 0 - 0 2024-09-13T14:51:06.139 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:06.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:06.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:06.241 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:06.255 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:51:06.255 INFO:teuthology.orchestra.run.smithi102.stdout: "size": 20, 2024-09-13T14:51:06.255 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 600, 2024-09-13T14:51:06.255 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [ 2024-09-13T14:51:06.255 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.255 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35379.0:160452 7.4 7:2d6ff74b:::benchmark_data_smithi102_227395_object47637:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-13T14:51:06.255 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:42:28.615406+0000", 2024-09-13T14:51:06.255 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 517.63538526900004, 2024-09-13T14:51:06.255 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.476235363, 2024-09-13T14:51:06.256 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:51:06.256 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:51:06.256 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:06.256 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:51:06.256 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35379", 2024-09-13T14:51:06.256 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:51:06.256 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 160452 2024-09-13T14:51:06.256 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.256 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:51:06.256 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.256 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:51:06.256 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:28.615406+0000", 2024-09-13T14:51:06.256 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:06.257 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.257 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.257 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:51:06.257 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:28.615406+0000", 2024-09-13T14:51:06.257 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:06.257 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.257 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.257 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:51:06.257 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:28.615407+0000", 2024-09-13T14:51:06.257 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1880000000000001e-06 2024-09-13T14:51:06.257 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.257 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.257 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:51:06.258 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:28.615411+0000", 2024-09-13T14:51:06.258 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.9260000000000002e-06 2024-09-13T14:51:06.258 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.258 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.258 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:51:06.258 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:28.615422+0000", 2024-09-13T14:51:06.258 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0713999999999999e-05 2024-09-13T14:51:06.258 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.258 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.258 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:51:06.258 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:28.615430+0000", 2024-09-13T14:51:06.258 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.0569999999999993e-06 2024-09-13T14:51:06.258 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.259 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.259 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:06.259 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:29.822728+0000", 2024-09-13T14:51:06.259 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2072981780000001 2024-09-13T14:51:06.259 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.259 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.259 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for degraded object", 2024-09-13T14:51:06.259 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:29.841233+0000", 2024-09-13T14:51:06.259 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.018505101999999999 2024-09-13T14:51:06.259 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.259 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.259 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:06.259 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:29.893934+0000", 2024-09-13T14:51:06.260 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.052700959999999998 2024-09-13T14:51:06.260 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.260 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.260 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:51:06.260 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:29.898776+0000", 2024-09-13T14:51:06.260 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0048412680000000001 2024-09-13T14:51:06.260 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.260 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.261 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:51:06.261 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:29.908502+0000", 2024-09-13T14:51:06.261 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0097266000000000002 2024-09-13T14:51:06.261 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.261 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.261 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:06.261 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:29.985608+0000", 2024-09-13T14:51:06.261 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.077105483000000002 2024-09-13T14:51:06.261 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.261 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.261 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:51:06.261 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:29.988077+0000", 2024-09-13T14:51:06.261 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0024695519999999999 2024-09-13T14:51:06.262 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.262 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.262 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:06.262 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:30.069674+0000", 2024-09-13T14:51:06.262 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.081596365000000004 2024-09-13T14:51:06.262 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.262 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.262 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:51:06.262 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:30.075578+0000", 2024-09-13T14:51:06.262 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0059048809999999998 2024-09-13T14:51:06.262 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.262 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.263 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:51:06.263 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:30.091642+0000", 2024-09-13T14:51:06.263 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.016063088999999999 2024-09-13T14:51:06.263 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:06.263 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:51:06.263 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:06.263 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.263 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.263 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35379.0:160523 7.4 7:261ac72b:::benchmark_data_smithi102_227395_object47708:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-13T14:51:06.263 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:42:28.922940+0000", 2024-09-13T14:51:06.264 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 517.32785162799996, 2024-09-13T14:51:06.264 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.491985248, 2024-09-13T14:51:06.264 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:51:06.264 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:51:06.264 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:06.264 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:51:06.264 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35379", 2024-09-13T14:51:06.264 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:51:06.264 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 160523 2024-09-13T14:51:06.264 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.264 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:51:06.265 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.265 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:51:06.265 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:28.922940+0000", 2024-09-13T14:51:06.265 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:06.265 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.265 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.265 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:51:06.265 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:28.922940+0000", 2024-09-13T14:51:06.265 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:06.265 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.266 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.266 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:51:06.266 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:28.922941+0000", 2024-09-13T14:51:06.266 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0210000000000001e-06 2024-09-13T14:51:06.266 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.266 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.266 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:51:06.266 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:28.922946+0000", 2024-09-13T14:51:06.266 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.9110000000000001e-06 2024-09-13T14:51:06.266 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.266 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.266 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:51:06.267 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:28.922957+0000", 2024-09-13T14:51:06.267 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1501000000000001e-05 2024-09-13T14:51:06.267 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.267 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.267 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:51:06.267 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:28.922963+0000", 2024-09-13T14:51:06.267 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.4040000000000003e-06 2024-09-13T14:51:06.268 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.268 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.268 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:06.268 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:30.135871+0000", 2024-09-13T14:51:06.268 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.212908259 2024-09-13T14:51:06.268 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.268 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.268 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for degraded object", 2024-09-13T14:51:06.268 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:30.149218+0000", 2024-09-13T14:51:06.268 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.013346761 2024-09-13T14:51:06.268 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.269 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.269 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:06.269 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:30.212828+0000", 2024-09-13T14:51:06.269 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.063610608999999999 2024-09-13T14:51:06.269 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.269 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.269 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:51:06.269 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:30.218028+0000", 2024-09-13T14:51:06.269 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0052001499999999997 2024-09-13T14:51:06.269 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.269 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.270 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:51:06.270 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:30.227858+0000", 2024-09-13T14:51:06.270 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0098296200000000007 2024-09-13T14:51:06.270 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.270 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.270 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:51:06.270 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:30.366015+0000", 2024-09-13T14:51:06.270 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.138156956 2024-09-13T14:51:06.270 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.270 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.270 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:06.270 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:30.394654+0000", 2024-09-13T14:51:06.271 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.028639270000000001 2024-09-13T14:51:06.271 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.271 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.271 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:06.271 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:30.397521+0000", 2024-09-13T14:51:06.271 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0028671159999999998 2024-09-13T14:51:06.271 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.271 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.271 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:51:06.271 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:30.401803+0000", 2024-09-13T14:51:06.271 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0042816499999999997 2024-09-13T14:51:06.272 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.272 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.273 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:51:06.273 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:30.414925+0000", 2024-09-13T14:51:06.273 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.01312202 2024-09-13T14:51:06.273 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:06.273 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:51:06.273 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:06.273 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.273 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.273 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35379.0:160547 7.4 7:2ff75aa6:::benchmark_data_smithi102_227395_object47732:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-13T14:51:06.274 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:42:29.003522+0000", 2024-09-13T14:51:06.274 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 517.24726958500003, 2024-09-13T14:51:06.274 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.453602882, 2024-09-13T14:51:06.274 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:51:06.274 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:51:06.274 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:06.274 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:51:06.274 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35379", 2024-09-13T14:51:06.274 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:51:06.274 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 160547 2024-09-13T14:51:06.274 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.274 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:51:06.274 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.275 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:51:06.275 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:29.003522+0000", 2024-09-13T14:51:06.275 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:06.275 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.275 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.275 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:51:06.275 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:29.003522+0000", 2024-09-13T14:51:06.275 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:06.275 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.275 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.275 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:51:06.275 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:29.003523+0000", 2024-09-13T14:51:06.275 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3480000000000001e-06 2024-09-13T14:51:06.276 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.276 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.276 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:51:06.276 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:29.003529+0000", 2024-09-13T14:51:06.276 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.913e-06 2024-09-13T14:51:06.276 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.276 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.276 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:51:06.276 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:29.003542+0000", 2024-09-13T14:51:06.276 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3287e-05 2024-09-13T14:51:06.276 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.276 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.276 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:51:06.277 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:29.003550+0000", 2024-09-13T14:51:06.277 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.4420000000000001e-06 2024-09-13T14:51:06.277 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.277 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.277 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:06.277 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:30.163428+0000", 2024-09-13T14:51:06.277 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.159878502 2024-09-13T14:51:06.277 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.277 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.277 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for degraded object", 2024-09-13T14:51:06.277 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:30.175776+0000", 2024-09-13T14:51:06.277 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.012347817000000001 2024-09-13T14:51:06.277 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.278 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.278 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:06.278 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:30.366829+0000", 2024-09-13T14:51:06.278 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.191053116 2024-09-13T14:51:06.278 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.278 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.278 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:51:06.278 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:30.372187+0000", 2024-09-13T14:51:06.278 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.005358051 2024-09-13T14:51:06.278 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.278 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.278 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:51:06.278 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:30.383321+0000", 2024-09-13T14:51:06.279 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.011133270000000001 2024-09-13T14:51:06.279 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.279 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.279 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:06.279 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:30.435259+0000", 2024-09-13T14:51:06.279 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.051938015999999997 2024-09-13T14:51:06.279 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.279 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.279 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:51:06.279 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:30.436252+0000", 2024-09-13T14:51:06.279 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00099365200000000003 2024-09-13T14:51:06.279 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.279 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.280 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:06.280 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:30.437867+0000", 2024-09-13T14:51:06.280 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0016151010000000001 2024-09-13T14:51:06.280 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.280 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.280 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:51:06.280 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:30.440803+0000", 2024-09-13T14:51:06.280 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0029359490000000002 2024-09-13T14:51:06.280 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.280 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.280 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:51:06.280 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:30.457125+0000", 2024-09-13T14:51:06.280 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.016321418000000001 2024-09-13T14:51:06.281 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:06.281 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:51:06.281 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:06.281 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.281 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.281 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35379.0:160762 7.4 7:275b9534:::benchmark_data_smithi102_227395_object47947:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-09-13T14:51:06.281 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:42:30.583268+0000", 2024-09-13T14:51:06.281 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 515.66752393599995, 2024-09-13T14:51:06.282 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4571130750000001, 2024-09-13T14:51:06.282 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:51:06.282 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:51:06.282 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:06.282 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:51:06.282 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35379", 2024-09-13T14:51:06.282 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:51:06.282 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 160762 2024-09-13T14:51:06.282 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.282 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:51:06.282 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.282 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:51:06.282 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:30.583268+0000", 2024-09-13T14:51:06.282 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:06.283 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.283 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.283 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:51:06.283 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:30.583268+0000", 2024-09-13T14:51:06.283 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:06.283 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.283 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.283 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:51:06.283 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:30.583270+0000", 2024-09-13T14:51:06.283 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.1689999999999999e-06 2024-09-13T14:51:06.283 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.283 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.283 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:51:06.284 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:30.583279+0000", 2024-09-13T14:51:06.284 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.4669999999999993e-06 2024-09-13T14:51:06.284 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.284 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.284 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:51:06.284 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:30.583296+0000", 2024-09-13T14:51:06.284 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.7062e-05 2024-09-13T14:51:06.284 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.284 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.284 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:51:06.284 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:30.583309+0000", 2024-09-13T14:51:06.284 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2604e-05 2024-09-13T14:51:06.284 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.285 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.285 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:06.285 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:31.824480+0000", 2024-09-13T14:51:06.285 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.241170758 2024-09-13T14:51:06.285 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.285 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.285 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for degraded object", 2024-09-13T14:51:06.285 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:31.838129+0000", 2024-09-13T14:51:06.285 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.013649415999999999 2024-09-13T14:51:06.285 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.285 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.285 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:06.285 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:31.897199+0000", 2024-09-13T14:51:06.285 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.059070081000000003 2024-09-13T14:51:06.286 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.286 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.286 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:51:06.286 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:31.902194+0000", 2024-09-13T14:51:06.286 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0049945659999999998 2024-09-13T14:51:06.286 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.286 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.286 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:51:06.286 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:31.912224+0000", 2024-09-13T14:51:06.286 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.01003064 2024-09-13T14:51:06.286 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.286 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.286 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:51:06.287 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:31.970818+0000", 2024-09-13T14:51:06.287 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.058593366000000001 2024-09-13T14:51:06.287 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.287 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.287 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:06.287 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:31.972930+0000", 2024-09-13T14:51:06.287 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0021125060000000001 2024-09-13T14:51:06.287 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.287 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.287 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:06.287 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:32.027244+0000", 2024-09-13T14:51:06.287 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.054314336999999997 2024-09-13T14:51:06.287 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.287 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.288 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:51:06.288 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:32.030230+0000", 2024-09-13T14:51:06.288 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.002985817 2024-09-13T14:51:06.288 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.288 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.288 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:51:06.288 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:32.040381+0000", 2024-09-13T14:51:06.288 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.010150286 2024-09-13T14:51:06.288 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:06.288 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:51:06.288 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:06.288 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.288 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.289 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35379.0:161619 7.4 7:26ab03ef:::benchmark_data_smithi102_227395_object48804:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e283)", 2024-09-13T14:51:06.289 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:42:35.724741+0000", 2024-09-13T14:51:06.289 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 510.52605060799999, 2024-09-13T14:51:06.289 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.45860086, 2024-09-13T14:51:06.289 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:51:06.289 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:51:06.289 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:06.289 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:51:06.289 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35379", 2024-09-13T14:51:06.289 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:51:06.289 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 161619 2024-09-13T14:51:06.289 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.289 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:51:06.290 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.290 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:51:06.290 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:35.724741+0000", 2024-09-13T14:51:06.290 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:06.290 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.290 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.290 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:51:06.290 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:35.724741+0000", 2024-09-13T14:51:06.290 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:06.290 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.290 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.290 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:51:06.290 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:35.724742+0000", 2024-09-13T14:51:06.291 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3790000000000001e-06 2024-09-13T14:51:06.291 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.291 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.291 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:51:06.291 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:35.724748+0000", 2024-09-13T14:51:06.291 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.8019999999999997e-06 2024-09-13T14:51:06.291 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.291 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.291 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:51:06.291 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:35.724760+0000", 2024-09-13T14:51:06.291 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2272e-05 2024-09-13T14:51:06.291 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.291 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.291 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:51:06.292 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:35.724767+0000", 2024-09-13T14:51:06.292 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.2709999999999998e-06 2024-09-13T14:51:06.292 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.292 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.292 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:06.292 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:36.883303+0000", 2024-09-13T14:51:06.292 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.158536523 2024-09-13T14:51:06.292 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.292 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.292 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for degraded object", 2024-09-13T14:51:06.292 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:36.901075+0000", 2024-09-13T14:51:06.292 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.017771431000000001 2024-09-13T14:51:06.293 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.293 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.293 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:06.293 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:36.984198+0000", 2024-09-13T14:51:06.293 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.083123064999999996 2024-09-13T14:51:06.293 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.293 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.293 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:51:06.293 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:36.991499+0000", 2024-09-13T14:51:06.293 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0073009470000000003 2024-09-13T14:51:06.293 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.293 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.293 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:51:06.294 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:37.005944+0000", 2024-09-13T14:51:06.294 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.014445698 2024-09-13T14:51:06.294 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.294 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.294 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:06.294 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:37.095796+0000", 2024-09-13T14:51:06.294 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.089851392000000002 2024-09-13T14:51:06.294 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.294 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.294 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:51:06.294 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:37.097020+0000", 2024-09-13T14:51:06.294 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0012245590000000001 2024-09-13T14:51:06.294 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.294 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.295 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:06.295 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:37.160346+0000", 2024-09-13T14:51:06.295 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.063326001000000007 2024-09-13T14:51:06.295 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.295 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.295 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:51:06.295 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:37.165154+0000", 2024-09-13T14:51:06.295 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.004807965 2024-09-13T14:51:06.295 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.295 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.295 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:51:06.295 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:37.183342+0000", 2024-09-13T14:51:06.295 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.018187555000000001 2024-09-13T14:51:06.296 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:06.296 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:51:06.296 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:06.296 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.296 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.296 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35379.0:161692 7.4 7:28db8956:::benchmark_data_smithi102_227395_object48877:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e283)", 2024-09-13T14:51:06.296 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:42:35.989050+0000", 2024-09-13T14:51:06.296 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 510.26174176400002, 2024-09-13T14:51:06.296 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.479639003, 2024-09-13T14:51:06.296 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:51:06.296 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:51:06.296 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:06.297 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:51:06.297 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35379", 2024-09-13T14:51:06.297 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:51:06.297 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 161692 2024-09-13T14:51:06.297 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.297 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:51:06.297 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.297 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:51:06.297 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:35.989050+0000", 2024-09-13T14:51:06.297 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:06.297 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.297 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.298 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:51:06.298 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:35.989050+0000", 2024-09-13T14:51:06.298 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:06.298 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.298 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.298 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:51:06.298 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:35.989052+0000", 2024-09-13T14:51:06.298 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.0379999999999998e-06 2024-09-13T14:51:06.298 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.298 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.298 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:51:06.298 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:35.989059+0000", 2024-09-13T14:51:06.298 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.6590000000000007e-06 2024-09-13T14:51:06.299 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.299 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.299 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:51:06.299 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:35.989074+0000", 2024-09-13T14:51:06.299 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4966e-05 2024-09-13T14:51:06.299 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.299 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.299 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:51:06.299 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:35.989083+0000", 2024-09-13T14:51:06.299 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.8400000000000001e-06 2024-09-13T14:51:06.299 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.299 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.299 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:06.299 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:37.240162+0000", 2024-09-13T14:51:06.300 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2510792180000001 2024-09-13T14:51:06.300 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.300 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.300 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for degraded object", 2024-09-13T14:51:06.300 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:37.255066+0000", 2024-09-13T14:51:06.300 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.014903602 2024-09-13T14:51:06.300 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.300 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.300 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:06.300 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:37.327243+0000", 2024-09-13T14:51:06.300 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.072176508 2024-09-13T14:51:06.300 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.300 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.301 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:51:06.301 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:37.333698+0000", 2024-09-13T14:51:06.301 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0064551749999999996 2024-09-13T14:51:06.301 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.301 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.301 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:51:06.301 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:37.344655+0000", 2024-09-13T14:51:06.301 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.010957289 2024-09-13T14:51:06.301 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.301 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.301 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:51:06.301 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:37.402218+0000", 2024-09-13T14:51:06.301 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.057562589999999997 2024-09-13T14:51:06.302 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.302 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.302 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:06.302 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:37.404428+0000", 2024-09-13T14:51:06.302 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0022100729999999999 2024-09-13T14:51:06.302 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.302 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.302 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:06.302 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:37.456227+0000", 2024-09-13T14:51:06.302 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.051799588000000001 2024-09-13T14:51:06.302 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.302 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.302 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:51:06.302 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:37.458985+0000", 2024-09-13T14:51:06.303 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0027575690000000001 2024-09-13T14:51:06.303 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.303 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.303 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:51:06.303 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:37.468689+0000", 2024-09-13T14:51:06.303 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0097038880000000008 2024-09-13T14:51:06.303 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:06.303 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:51:06.303 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:06.303 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.303 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.303 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35379.0:161695 7.4 7:270b99e4:::benchmark_data_smithi102_227395_object48880:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e283)", 2024-09-13T14:51:06.303 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:42:36.005113+0000", 2024-09-13T14:51:06.304 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 510.24567854700001, 2024-09-13T14:51:06.304 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.482191448, 2024-09-13T14:51:06.304 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:51:06.304 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:51:06.304 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:06.304 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:51:06.304 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35379", 2024-09-13T14:51:06.304 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:51:06.304 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 161695 2024-09-13T14:51:06.304 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.304 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:51:06.304 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.304 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:51:06.305 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:36.005113+0000", 2024-09-13T14:51:06.305 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:06.305 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.305 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.305 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:51:06.305 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:36.005113+0000", 2024-09-13T14:51:06.305 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:06.305 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.305 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.305 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:51:06.305 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:36.005115+0000", 2024-09-13T14:51:06.305 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.793e-06 2024-09-13T14:51:06.305 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.306 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.306 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:51:06.306 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:36.005119+0000", 2024-09-13T14:51:06.306 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.1450000000000001e-06 2024-09-13T14:51:06.306 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.306 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.306 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:51:06.306 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:36.005128+0000", 2024-09-13T14:51:06.306 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.6399999999999992e-06 2024-09-13T14:51:06.306 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.306 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.306 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:51:06.306 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:36.005134+0000", 2024-09-13T14:51:06.306 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.2340000000000002e-06 2024-09-13T14:51:06.307 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.307 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.307 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:06.307 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:37.255910+0000", 2024-09-13T14:51:06.307 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2507766929999999 2024-09-13T14:51:06.307 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.307 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.307 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for degraded object", 2024-09-13T14:51:06.307 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:37.280407+0000", 2024-09-13T14:51:06.307 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.024496626000000001 2024-09-13T14:51:06.307 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.307 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.307 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:06.308 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:37.379046+0000", 2024-09-13T14:51:06.308 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.098639284999999993 2024-09-13T14:51:06.308 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.308 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.308 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:51:06.308 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:37.383605+0000", 2024-09-13T14:51:06.308 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.004559072 2024-09-13T14:51:06.308 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.308 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.308 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:51:06.308 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:37.393378+0000", 2024-09-13T14:51:06.308 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0097726180000000003 2024-09-13T14:51:06.308 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.309 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.309 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:51:06.309 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:37.469167+0000", 2024-09-13T14:51:06.309 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.075789044999999999 2024-09-13T14:51:06.309 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.309 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.309 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:06.309 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:37.470617+0000", 2024-09-13T14:51:06.309 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0014499000000000001 2024-09-13T14:51:06.309 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.309 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.309 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:06.309 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:37.472608+0000", 2024-09-13T14:51:06.309 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0019906350000000001 2024-09-13T14:51:06.310 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.310 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.310 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:51:06.310 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:37.475505+0000", 2024-09-13T14:51:06.310 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0028970760000000002 2024-09-13T14:51:06.310 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.310 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.310 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:51:06.310 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:37.487304+0000", 2024-09-13T14:51:06.310 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.011799686 2024-09-13T14:51:06.310 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:06.310 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:51:06.310 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:06.311 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.311 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.311 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35379.0:161789 7.4 7:2a6883c4:::benchmark_data_smithi102_227395_object48974:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e283)", 2024-09-13T14:51:06.311 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:42:36.624721+0000", 2024-09-13T14:51:06.311 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 509.626070244, 2024-09-13T14:51:06.311 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.433910909, 2024-09-13T14:51:06.311 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:51:06.311 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:51:06.311 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:06.311 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:51:06.311 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35379", 2024-09-13T14:51:06.311 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:51:06.311 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 161789 2024-09-13T14:51:06.311 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.312 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:51:06.312 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.312 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:51:06.312 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:36.624721+0000", 2024-09-13T14:51:06.312 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:06.312 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.312 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.312 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:51:06.312 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:36.624721+0000", 2024-09-13T14:51:06.312 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:06.312 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.312 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.312 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:51:06.312 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:36.624723+0000", 2024-09-13T14:51:06.313 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.7150000000000001e-06 2024-09-13T14:51:06.313 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.313 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.313 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:51:06.313 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:36.624731+0000", 2024-09-13T14:51:06.313 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.2619999999999998e-06 2024-09-13T14:51:06.313 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.313 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.313 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:51:06.313 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:36.624745+0000", 2024-09-13T14:51:06.313 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3736e-05 2024-09-13T14:51:06.313 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.313 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.313 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:51:06.314 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:36.624755+0000", 2024-09-13T14:51:06.314 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0165e-05 2024-09-13T14:51:06.314 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.314 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.314 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:06.314 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:37.794732+0000", 2024-09-13T14:51:06.314 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.169976871 2024-09-13T14:51:06.314 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.314 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.314 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for degraded object", 2024-09-13T14:51:06.314 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:37.829926+0000", 2024-09-13T14:51:06.314 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.035194439000000001 2024-09-13T14:51:06.314 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.315 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.315 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:06.315 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:37.913395+0000", 2024-09-13T14:51:06.315 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.083468506999999997 2024-09-13T14:51:06.315 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.315 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.315 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:51:06.315 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:37.917947+0000", 2024-09-13T14:51:06.315 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0045517400000000003 2024-09-13T14:51:06.315 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.315 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.315 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:51:06.315 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:37.926967+0000", 2024-09-13T14:51:06.316 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0090199359999999992 2024-09-13T14:51:06.316 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.316 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.316 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:06.316 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:37.988371+0000", 2024-09-13T14:51:06.316 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.061404078000000001 2024-09-13T14:51:06.316 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.316 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.316 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:51:06.316 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:37.989384+0000", 2024-09-13T14:51:06.316 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0010131999999999999 2024-09-13T14:51:06.316 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.316 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:06.318 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:06.318 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:38.042738+0000", 2024-09-13T14:51:06.318 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.053353857999999997 2024-09-13T14:51:06.318 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.318 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.318 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:51:06.318 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:38.046790+0000", 2024-09-13T14:51:06.318 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0040519129999999999 2024-09-13T14:51:06.319 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.319 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.319 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:51:06.319 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:38.058632+0000", 2024-09-13T14:51:06.319 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.011842488999999999 2024-09-13T14:51:06.319 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:06.319 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:51:06.319 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:06.319 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.319 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.319 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35379.0:161828 7.4 7:2ef1b6e5:::benchmark_data_smithi102_227395_object49013:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e283)", 2024-09-13T14:51:06.319 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:42:36.876258+0000", 2024-09-13T14:51:06.319 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 509.37453378999999, 2024-09-13T14:51:06.319 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4574817790000001, 2024-09-13T14:51:06.320 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:51:06.320 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:51:06.320 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:06.320 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:51:06.320 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35379", 2024-09-13T14:51:06.320 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:51:06.320 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 161828 2024-09-13T14:51:06.320 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.320 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:51:06.320 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.320 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:51:06.320 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:36.876258+0000", 2024-09-13T14:51:06.321 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:06.321 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.321 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.321 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:51:06.321 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:36.876258+0000", 2024-09-13T14:51:06.321 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:06.321 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.321 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.321 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:51:06.321 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:36.876259+0000", 2024-09-13T14:51:06.321 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.508e-06 2024-09-13T14:51:06.321 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.321 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.321 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:51:06.322 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:36.876266+0000", 2024-09-13T14:51:06.322 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.6229999999999999e-06 2024-09-13T14:51:06.322 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.322 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.322 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:51:06.322 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:36.876280+0000", 2024-09-13T14:51:06.322 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3747999999999999e-05 2024-09-13T14:51:06.322 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.322 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.322 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:51:06.322 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:36.876288+0000", 2024-09-13T14:51:06.322 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.6649999999999992e-06 2024-09-13T14:51:06.322 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.323 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.323 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:06.323 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:38.105869+0000", 2024-09-13T14:51:06.323 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2295808420000001 2024-09-13T14:51:06.323 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.323 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.323 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for degraded object", 2024-09-13T14:51:06.323 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:38.129614+0000", 2024-09-13T14:51:06.323 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.023745219000000001 2024-09-13T14:51:06.323 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.323 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.323 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:06.323 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:38.194028+0000", 2024-09-13T14:51:06.324 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.064414100000000002 2024-09-13T14:51:06.324 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.324 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.324 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:51:06.324 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:38.198477+0000", 2024-09-13T14:51:06.324 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0044485319999999998 2024-09-13T14:51:06.324 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.324 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.324 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:51:06.324 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:38.206952+0000", 2024-09-13T14:51:06.324 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0084751429999999992 2024-09-13T14:51:06.324 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.324 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.325 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:06.325 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:38.263780+0000", 2024-09-13T14:51:06.325 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.056827673000000002 2024-09-13T14:51:06.325 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.325 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.325 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:51:06.325 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:38.264828+0000", 2024-09-13T14:51:06.325 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0010480839999999999 2024-09-13T14:51:06.325 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.325 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.325 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:06.325 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:38.319260+0000", 2024-09-13T14:51:06.325 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.054431978999999998 2024-09-13T14:51:06.325 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.326 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.326 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:51:06.326 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:38.322310+0000", 2024-09-13T14:51:06.326 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0030499899999999998 2024-09-13T14:51:06.326 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.326 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.326 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:51:06.326 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:38.333739+0000", 2024-09-13T14:51:06.326 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.011429673 2024-09-13T14:51:06.326 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:06.326 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:51:06.326 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:06.326 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.327 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.327 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35379.0:161835 7.4 7:2c52c14f:::benchmark_data_smithi102_227395_object49020:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e283)", 2024-09-13T14:51:06.327 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:42:36.889899+0000", 2024-09-13T14:51:06.327 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 509.36089197899997, 2024-09-13T14:51:06.327 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4604823760000001, 2024-09-13T14:51:06.327 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:51:06.327 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:51:06.327 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:06.327 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:51:06.327 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35379", 2024-09-13T14:51:06.327 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:51:06.327 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 161835 2024-09-13T14:51:06.327 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.327 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:51:06.328 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.328 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:51:06.328 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:36.889899+0000", 2024-09-13T14:51:06.328 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:06.328 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.328 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.328 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:51:06.328 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:36.889899+0000", 2024-09-13T14:51:06.328 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:06.328 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.328 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.328 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:51:06.328 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:36.889901+0000", 2024-09-13T14:51:06.329 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4529999999999999e-06 2024-09-13T14:51:06.329 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.329 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.329 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:51:06.329 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:36.889908+0000", 2024-09-13T14:51:06.329 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.3490000000000003e-06 2024-09-13T14:51:06.329 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.329 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.329 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:51:06.329 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:36.889921+0000", 2024-09-13T14:51:06.329 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2461e-05 2024-09-13T14:51:06.329 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.329 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.329 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:51:06.330 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:36.889929+0000", 2024-09-13T14:51:06.330 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.4789999999999996e-06 2024-09-13T14:51:06.330 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.330 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.330 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:06.330 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:38.130419+0000", 2024-09-13T14:51:06.330 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.240489527 2024-09-13T14:51:06.330 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.330 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.330 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for degraded object", 2024-09-13T14:51:06.330 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:38.151368+0000", 2024-09-13T14:51:06.330 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.020948947999999998 2024-09-13T14:51:06.330 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.330 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.331 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:06.331 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:38.239812+0000", 2024-09-13T14:51:06.331 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.088444048999999997 2024-09-13T14:51:06.331 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.331 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.331 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:51:06.331 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:38.245412+0000", 2024-09-13T14:51:06.331 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.005600407 2024-09-13T14:51:06.331 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.331 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.331 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:51:06.331 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:38.254843+0000", 2024-09-13T14:51:06.331 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0094312649999999994 2024-09-13T14:51:06.332 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.332 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.332 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:51:06.332 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:38.317709+0000", 2024-09-13T14:51:06.332 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.062865761000000006 2024-09-13T14:51:06.332 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.332 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.332 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:06.332 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:38.335357+0000", 2024-09-13T14:51:06.332 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.01764758 2024-09-13T14:51:06.332 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.332 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.332 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:06.333 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:38.337752+0000", 2024-09-13T14:51:06.333 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0023949840000000002 2024-09-13T14:51:06.333 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.333 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.333 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:51:06.333 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:38.340626+0000", 2024-09-13T14:51:06.333 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.002874064 2024-09-13T14:51:06.333 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.333 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.333 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:51:06.333 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:38.350382+0000", 2024-09-13T14:51:06.333 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0097560489999999993 2024-09-13T14:51:06.333 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:06.333 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:51:06.334 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:06.334 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.334 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.334 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35379.0:165621 7.4 7:23126f4f:::benchmark_data_smithi102_227395_object52806:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e285)", 2024-09-13T14:51:06.334 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:42:57.474275+0000", 2024-09-13T14:51:06.334 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 488.77651606799998, 2024-09-13T14:51:06.334 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.464794454, 2024-09-13T14:51:06.334 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:51:06.334 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:51:06.334 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:06.334 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:51:06.334 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35379", 2024-09-13T14:51:06.334 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:51:06.335 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 165621 2024-09-13T14:51:06.335 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.335 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:51:06.335 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.335 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:51:06.335 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:57.474275+0000", 2024-09-13T14:51:06.335 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:06.335 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.335 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.335 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:51:06.335 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:57.474275+0000", 2024-09-13T14:51:06.335 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:06.335 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.336 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.336 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:51:06.336 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:57.474277+0000", 2024-09-13T14:51:06.336 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4050000000000001e-06 2024-09-13T14:51:06.336 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.336 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.336 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:51:06.336 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:57.474283+0000", 2024-09-13T14:51:06.336 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.3620000000000002e-06 2024-09-13T14:51:06.336 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.336 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.336 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:51:06.336 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:57.474296+0000", 2024-09-13T14:51:06.337 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2588999999999999e-05 2024-09-13T14:51:06.337 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.337 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.337 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:51:06.337 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:57.474302+0000", 2024-09-13T14:51:06.337 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.567e-06 2024-09-13T14:51:06.337 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.337 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.337 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:06.337 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:58.703603+0000", 2024-09-13T14:51:06.337 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2293006829999999 2024-09-13T14:51:06.337 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.337 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.337 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for degraded object", 2024-09-13T14:51:06.338 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:58.728873+0000", 2024-09-13T14:51:06.338 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.025270200999999999 2024-09-13T14:51:06.338 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.338 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.338 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:06.338 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:58.814821+0000", 2024-09-13T14:51:06.338 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.085948076999999998 2024-09-13T14:51:06.338 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.338 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.338 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:51:06.338 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:58.818915+0000", 2024-09-13T14:51:06.338 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.004093625 2024-09-13T14:51:06.338 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.339 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.339 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:51:06.339 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:58.827487+0000", 2024-09-13T14:51:06.339 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0085716899999999999 2024-09-13T14:51:06.339 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.339 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.339 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:06.339 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:58.879813+0000", 2024-09-13T14:51:06.339 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.052326144999999998 2024-09-13T14:51:06.339 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.339 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.339 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:51:06.339 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:58.923960+0000", 2024-09-13T14:51:06.339 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.044147023000000001 2024-09-13T14:51:06.340 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.340 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.340 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:06.340 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:58.925420+0000", 2024-09-13T14:51:06.340 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0014603660000000001 2024-09-13T14:51:06.340 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.340 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.340 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:51:06.340 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:58.928408+0000", 2024-09-13T14:51:06.340 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0029878610000000001 2024-09-13T14:51:06.340 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.340 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.340 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:51:06.341 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:58.939070+0000", 2024-09-13T14:51:06.341 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.01066186 2024-09-13T14:51:06.341 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:06.341 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:51:06.341 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:06.341 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.341 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.341 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35379.0:165623 7.4 7:2d2b974f:::benchmark_data_smithi102_227395_object52808:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e285)", 2024-09-13T14:51:06.341 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:42:57.478178+0000", 2024-09-13T14:51:06.341 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 488.772613664, 2024-09-13T14:51:06.341 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4776153220000001, 2024-09-13T14:51:06.341 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:51:06.341 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:51:06.341 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:06.342 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:51:06.342 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35379", 2024-09-13T14:51:06.342 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:51:06.342 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 165623 2024-09-13T14:51:06.342 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.342 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:51:06.342 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.342 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:51:06.342 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:57.478178+0000", 2024-09-13T14:51:06.342 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:06.342 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.342 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.342 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:51:06.343 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:57.478178+0000", 2024-09-13T14:51:06.343 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:06.343 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.343 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.343 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:51:06.343 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:57.478179+0000", 2024-09-13T14:51:06.343 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1939999999999999e-06 2024-09-13T14:51:06.343 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.343 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.343 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:51:06.343 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:57.478183+0000", 2024-09-13T14:51:06.343 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.0099999999999997e-06 2024-09-13T14:51:06.343 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.343 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.344 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:51:06.344 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:57.478192+0000", 2024-09-13T14:51:06.344 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.1169999999999993e-06 2024-09-13T14:51:06.344 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.344 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.344 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:51:06.344 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:57.478197+0000", 2024-09-13T14:51:06.344 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.9150000000000004e-06 2024-09-13T14:51:06.344 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.344 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.344 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:06.344 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:58.730511+0000", 2024-09-13T14:51:06.344 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.252313622 2024-09-13T14:51:06.344 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.345 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.345 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for degraded object", 2024-09-13T14:51:06.345 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:58.753298+0000", 2024-09-13T14:51:06.345 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.022787714000000001 2024-09-13T14:51:06.345 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.345 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.345 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:06.345 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:58.857102+0000", 2024-09-13T14:51:06.345 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.103803768 2024-09-13T14:51:06.345 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.345 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.345 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:51:06.345 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:58.861167+0000", 2024-09-13T14:51:06.346 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0040650900000000004 2024-09-13T14:51:06.346 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.346 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.346 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:51:06.346 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:58.870724+0000", 2024-09-13T14:51:06.346 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0095567710000000004 2024-09-13T14:51:06.346 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.346 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.346 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:06.346 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:58.940512+0000", 2024-09-13T14:51:06.346 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.069787729000000007 2024-09-13T14:51:06.346 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.346 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.346 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:51:06.347 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:58.941417+0000", 2024-09-13T14:51:06.347 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00090505999999999996 2024-09-13T14:51:06.347 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.347 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.347 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:06.347 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:58.943211+0000", 2024-09-13T14:51:06.347 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.001793909 2024-09-13T14:51:06.347 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.347 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.347 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:51:06.347 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:58.946021+0000", 2024-09-13T14:51:06.347 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.002810037 2024-09-13T14:51:06.347 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.348 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.348 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:51:06.348 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:58.955793+0000", 2024-09-13T14:51:06.348 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0097723859999999992 2024-09-13T14:51:06.348 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:06.348 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:51:06.348 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:06.348 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.348 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.348 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35379.0:165658 7.4 7:275a1028:::benchmark_data_smithi102_227395_object52843:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e285)", 2024-09-13T14:51:06.348 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:42:57.717809+0000", 2024-09-13T14:51:06.348 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 488.53298207099999, 2024-09-13T14:51:06.348 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.449454987, 2024-09-13T14:51:06.349 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:51:06.349 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:51:06.349 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:06.349 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:51:06.349 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35379", 2024-09-13T14:51:06.349 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:51:06.349 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 165658 2024-09-13T14:51:06.349 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.349 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:51:06.349 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.349 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:51:06.349 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:57.717809+0000", 2024-09-13T14:51:06.349 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:06.349 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.350 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.350 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:51:06.350 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:57.717809+0000", 2024-09-13T14:51:06.350 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:06.350 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.350 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.350 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:51:06.350 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:57.717811+0000", 2024-09-13T14:51:06.350 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.5179999999999999e-06 2024-09-13T14:51:06.350 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.350 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.350 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:51:06.350 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:57.717815+0000", 2024-09-13T14:51:06.351 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.3209999999999998e-06 2024-09-13T14:51:06.351 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.351 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.351 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:51:06.351 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:57.717829+0000", 2024-09-13T14:51:06.351 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3652000000000001e-05 2024-09-13T14:51:06.351 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.351 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.351 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:51:06.351 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:57.717837+0000", 2024-09-13T14:51:06.351 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.8709999999999997e-06 2024-09-13T14:51:06.351 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.351 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.351 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:06.352 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:58.972394+0000", 2024-09-13T14:51:06.352 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.254557108 2024-09-13T14:51:06.352 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.352 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.352 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for degraded object", 2024-09-13T14:51:06.352 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:58.990714+0000", 2024-09-13T14:51:06.352 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.018319946 2024-09-13T14:51:06.352 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.352 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.352 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:06.352 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:59.040823+0000", 2024-09-13T14:51:06.352 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.050109378000000003 2024-09-13T14:51:06.352 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.353 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.353 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:51:06.353 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:59.045310+0000", 2024-09-13T14:51:06.353 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0044870480000000004 2024-09-13T14:51:06.353 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.353 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.353 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:51:06.353 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:59.053302+0000", 2024-09-13T14:51:06.353 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0079917960000000007 2024-09-13T14:51:06.353 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.353 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.353 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:51:06.353 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:59.079170+0000", 2024-09-13T14:51:06.354 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.025868215999999999 2024-09-13T14:51:06.354 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.354 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.354 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:06.354 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:59.099383+0000", 2024-09-13T14:51:06.354 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.020213041000000001 2024-09-13T14:51:06.354 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.354 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.354 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:06.354 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:59.149214+0000", 2024-09-13T14:51:06.354 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.049831015999999999 2024-09-13T14:51:06.354 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.354 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.354 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:51:06.355 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:59.153065+0000", 2024-09-13T14:51:06.355 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.003851087 2024-09-13T14:51:06.355 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.355 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.355 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:51:06.355 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:42:59.167264+0000", 2024-09-13T14:51:06.355 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.014198989 2024-09-13T14:51:06.355 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:06.355 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:51:06.355 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:06.355 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.355 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.355 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35379.0:179268 7.4 7:299d2bab:::benchmark_data_smithi102_227395_object66453:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-13T14:51:06.356 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:43:51.719911+0000", 2024-09-13T14:51:06.356 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 434.53088040400002, 2024-09-13T14:51:06.356 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.007595904, 2024-09-13T14:51:06.356 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:51:06.356 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:51:06.356 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:06.356 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:51:06.356 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35379", 2024-09-13T14:51:06.356 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:51:06.356 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 179268 2024-09-13T14:51:06.356 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.356 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:51:06.356 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.356 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:51:06.357 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:43:51.719911+0000", 2024-09-13T14:51:06.357 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:06.357 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.357 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.357 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:51:06.357 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:43:51.719911+0000", 2024-09-13T14:51:06.357 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:06.357 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.357 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.357 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:51:06.357 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:43:51.719912+0000", 2024-09-13T14:51:06.357 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.0999999999999997e-07 2024-09-13T14:51:06.357 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.358 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.358 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:51:06.358 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:43:51.719916+0000", 2024-09-13T14:51:06.358 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.031e-06 2024-09-13T14:51:06.358 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.358 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.358 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:51:06.358 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:43:51.719926+0000", 2024-09-13T14:51:06.358 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0488e-05 2024-09-13T14:51:06.358 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.358 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.358 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:51:06.358 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:43:51.719932+0000", 2024-09-13T14:51:06.358 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.5369999999999997e-06 2024-09-13T14:51:06.359 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.359 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.359 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:06.359 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:43:52.583619+0000", 2024-09-13T14:51:06.359 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.86368709799999999 2024-09-13T14:51:06.359 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.359 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.359 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for degraded object", 2024-09-13T14:51:06.359 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:43:52.593621+0000", 2024-09-13T14:51:06.359 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.010001741 2024-09-13T14:51:06.359 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.359 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.359 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:06.359 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:43:52.636860+0000", 2024-09-13T14:51:06.360 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.043239441000000003 2024-09-13T14:51:06.360 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.360 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.360 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:51:06.360 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:43:52.640048+0000", 2024-09-13T14:51:06.360 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0031872900000000002 2024-09-13T14:51:06.360 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.360 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.360 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:51:06.360 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:43:52.646467+0000", 2024-09-13T14:51:06.360 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0064196119999999999 2024-09-13T14:51:06.361 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:06.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:06.362 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.362 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.362 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:51:06.362 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:43:52.683779+0000", 2024-09-13T14:51:06.362 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.037311505000000002 2024-09-13T14:51:06.362 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.362 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.362 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:06.363 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:43:52.684971+0000", 2024-09-13T14:51:06.363 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.001191899 2024-09-13T14:51:06.363 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.363 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.363 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:06.363 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:43:52.718142+0000", 2024-09-13T14:51:06.363 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.033171042999999997 2024-09-13T14:51:06.363 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.363 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.363 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:51:06.363 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:43:52.720375+0000", 2024-09-13T14:51:06.363 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0022331870000000002 2024-09-13T14:51:06.363 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.364 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.364 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:51:06.364 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:43:52.727507+0000", 2024-09-13T14:51:06.364 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0071321220000000003 2024-09-13T14:51:06.364 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:06.364 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:51:06.364 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:06.364 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.364 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.364 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35379.0:179677 7.4 7:244b87fc:::benchmark_data_smithi102_227395_object66862:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-13T14:51:06.364 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:43:53.113420+0000", 2024-09-13T14:51:06.364 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 433.13737110699998, 2024-09-13T14:51:06.364 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.98978256399999998, 2024-09-13T14:51:06.365 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:51:06.365 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:51:06.365 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:06.365 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:51:06.365 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35379", 2024-09-13T14:51:06.365 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:51:06.365 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 179677 2024-09-13T14:51:06.365 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.365 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:51:06.365 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.365 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:51:06.365 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:43:53.113420+0000", 2024-09-13T14:51:06.365 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:06.366 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.366 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.366 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:51:06.366 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:43:53.113420+0000", 2024-09-13T14:51:06.366 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:06.366 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.366 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.366 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:51:06.366 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:43:53.113422+0000", 2024-09-13T14:51:06.366 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.248e-06 2024-09-13T14:51:06.366 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.366 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.366 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:51:06.367 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:43:53.113426+0000", 2024-09-13T14:51:06.367 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.566e-06 2024-09-13T14:51:06.367 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.367 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.367 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:51:06.367 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:43:53.113437+0000", 2024-09-13T14:51:06.367 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1041e-05 2024-09-13T14:51:06.367 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.367 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.367 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:51:06.367 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:43:53.113443+0000", 2024-09-13T14:51:06.367 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.8429999999999998e-06 2024-09-13T14:51:06.367 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.368 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.368 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:06.368 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:43:53.958976+0000", 2024-09-13T14:51:06.368 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.84553280799999997 2024-09-13T14:51:06.368 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.368 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.368 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for degraded object", 2024-09-13T14:51:06.368 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:43:53.968019+0000", 2024-09-13T14:51:06.368 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0090436000000000006 2024-09-13T14:51:06.368 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.368 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.368 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:06.368 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:43:54.003505+0000", 2024-09-13T14:51:06.369 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.035485213000000002 2024-09-13T14:51:06.369 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.369 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.369 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:51:06.369 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:43:54.007789+0000", 2024-09-13T14:51:06.369 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0042844429999999998 2024-09-13T14:51:06.369 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.369 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.369 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:51:06.369 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:43:54.015530+0000", 2024-09-13T14:51:06.369 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0077405549999999997 2024-09-13T14:51:06.369 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.369 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.370 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:51:06.370 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:43:54.040580+0000", 2024-09-13T14:51:06.370 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.025050735000000001 2024-09-13T14:51:06.370 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.370 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.370 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:06.370 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:43:54.060171+0000", 2024-09-13T14:51:06.370 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.019590152 2024-09-13T14:51:06.370 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.370 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.370 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:06.370 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:43:54.093929+0000", 2024-09-13T14:51:06.370 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.033757966 2024-09-13T14:51:06.370 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.371 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.371 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:51:06.371 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:43:54.096062+0000", 2024-09-13T14:51:06.371 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.002133551 2024-09-13T14:51:06.371 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.371 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.371 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:51:06.371 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:43:54.103203+0000", 2024-09-13T14:51:06.371 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.007140843 2024-09-13T14:51:06.371 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:06.371 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:51:06.371 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:06.371 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.372 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.372 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.35379.0:180633 7.4 7:2068e536:::benchmark_data_smithi102_227395_object67818:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-13T14:51:06.372 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:43:56.047845+0000", 2024-09-13T14:51:06.372 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 430.20294655200001, 2024-09-13T14:51:06.372 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.99494763900000005, 2024-09-13T14:51:06.372 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:51:06.372 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:51:06.372 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:06.372 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:51:06.372 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.35379", 2024-09-13T14:51:06.372 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:51:06.372 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 180633 2024-09-13T14:51:06.372 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.373 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:51:06.373 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.373 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:51:06.373 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:43:56.047845+0000", 2024-09-13T14:51:06.373 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:06.373 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.373 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.373 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:51:06.373 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:43:56.047845+0000", 2024-09-13T14:51:06.373 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:06.373 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.373 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.373 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:51:06.374 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:43:56.047846+0000", 2024-09-13T14:51:06.374 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.5300000000000003e-07 2024-09-13T14:51:06.374 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.374 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.374 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:51:06.374 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:43:56.047849+0000", 2024-09-13T14:51:06.374 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.7009999999999998e-06 2024-09-13T14:51:06.374 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.374 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.374 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:51:06.374 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:43:56.047864+0000", 2024-09-13T14:51:06.374 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.469e-05 2024-09-13T14:51:06.374 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.374 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.375 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:51:06.375 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:43:56.047869+0000", 2024-09-13T14:51:06.375 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.8740000000000003e-06 2024-09-13T14:51:06.375 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.375 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.375 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:06.375 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:43:56.901005+0000", 2024-09-13T14:51:06.375 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.85313610500000003 2024-09-13T14:51:06.375 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.375 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.375 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for degraded object", 2024-09-13T14:51:06.375 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:43:56.910388+0000", 2024-09-13T14:51:06.375 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0093827779999999996 2024-09-13T14:51:06.376 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.376 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.376 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:06.376 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:43:56.952270+0000", 2024-09-13T14:51:06.376 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.041882160000000002 2024-09-13T14:51:06.376 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.376 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.376 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:51:06.376 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:43:56.955421+0000", 2024-09-13T14:51:06.376 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0031513629999999999 2024-09-13T14:51:06.376 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.376 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.377 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:51:06.377 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:43:56.961726+0000", 2024-09-13T14:51:06.377 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0063045949999999996 2024-09-13T14:51:06.377 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.377 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.377 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:06.377 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:43:57.000279+0000", 2024-09-13T14:51:06.377 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.038552783 2024-09-13T14:51:06.377 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.377 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.377 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:51:06.377 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:43:57.000971+0000", 2024-09-13T14:51:06.377 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00069203699999999995 2024-09-13T14:51:06.378 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.378 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.378 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:06.378 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:43:57.032825+0000", 2024-09-13T14:51:06.378 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.031853846999999998 2024-09-13T14:51:06.378 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.378 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.378 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:51:06.378 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:43:57.035028+0000", 2024-09-13T14:51:06.378 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0022035570000000001 2024-09-13T14:51:06.378 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.378 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.378 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:51:06.379 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:43:57.042793+0000", 2024-09-13T14:51:06.379 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0077643959999999998 2024-09-13T14:51:06.379 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:06.379 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:51:06.379 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:06.379 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.379 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.379 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:133945 8.1 8:8839da07:::benchmark_data_smithi102_313275_object3358:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e337)", 2024-09-13T14:51:06.379 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:47:47.864515+0000", 2024-09-13T14:51:06.379 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 198.38627678200001, 2024-09-13T14:51:06.379 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4022485659999999, 2024-09-13T14:51:06.379 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:51:06.379 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:51:06.380 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "reached pg", 2024-09-13T14:51:06.380 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:51:06.380 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:51:06.380 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:51:06.380 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 133945 2024-09-13T14:51:06.380 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.380 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:51:06.380 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.380 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:51:06.380 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.864515+0000", 2024-09-13T14:51:06.380 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:06.380 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.380 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.381 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:51:06.381 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.864515+0000", 2024-09-13T14:51:06.381 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:06.381 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.381 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.381 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:51:06.381 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.864516+0000", 2024-09-13T14:51:06.381 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.584e-06 2024-09-13T14:51:06.381 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.381 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.381 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:51:06.381 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.864520+0000", 2024-09-13T14:51:06.381 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.3009999999999998e-06 2024-09-13T14:51:06.382 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.382 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.382 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:51:06.382 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.864529+0000", 2024-09-13T14:51:06.382 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.6660000000000003e-06 2024-09-13T14:51:06.382 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.382 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.382 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:51:06.382 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.864539+0000", 2024-09-13T14:51:06.382 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0220999999999999e-05 2024-09-13T14:51:06.382 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.382 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.382 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:06.383 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.865094+0000", 2024-09-13T14:51:06.383 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00055455200000000004 2024-09-13T14:51:06.383 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.383 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.383 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:51:06.383 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.865103+0000", 2024-09-13T14:51:06.383 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.1020000000000003e-06 2024-09-13T14:51:06.383 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.383 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.383 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:06.383 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.266596+0000", 2024-09-13T14:51:06.383 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4014933250000001 2024-09-13T14:51:06.383 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.384 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.384 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:51:06.384 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.266763+0000", 2024-09-13T14:51:06.384 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00016681500000000001 2024-09-13T14:51:06.384 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:06.384 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:51:06.384 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:06.384 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.384 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.384 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:133951 8.1 8:89dc2928:::benchmark_data_smithi102_313275_object3364:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e337)", 2024-09-13T14:51:06.384 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:47:47.864559+0000", 2024-09-13T14:51:06.384 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 198.38623294800001, 2024-09-13T14:51:06.384 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4184646949999999, 2024-09-13T14:51:06.385 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:51:06.385 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:51:06.385 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:06.385 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:51:06.385 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:51:06.385 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:51:06.385 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 133951 2024-09-13T14:51:06.385 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.385 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:51:06.385 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.385 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:51:06.385 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.864559+0000", 2024-09-13T14:51:06.385 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:06.386 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.386 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.386 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:51:06.386 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.864559+0000", 2024-09-13T14:51:06.386 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:06.386 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.386 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.386 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:51:06.386 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.864559+0000", 2024-09-13T14:51:06.386 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.9200000000000001e-07 2024-09-13T14:51:06.386 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.386 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.386 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:51:06.387 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.864562+0000", 2024-09-13T14:51:06.387 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.83e-06 2024-09-13T14:51:06.387 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.387 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.387 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:51:06.387 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.864569+0000", 2024-09-13T14:51:06.387 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.0890000000000002e-06 2024-09-13T14:51:06.387 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.387 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.387 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:51:06.387 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.864574+0000", 2024-09-13T14:51:06.387 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.4990000000000002e-06 2024-09-13T14:51:06.387 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.388 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.388 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:06.388 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.866053+0000", 2024-09-13T14:51:06.388 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00147857 2024-09-13T14:51:06.388 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.388 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.388 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:51:06.388 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.866061+0000", 2024-09-13T14:51:06.388 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.4540000000000007e-06 2024-09-13T14:51:06.388 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.388 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.388 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:06.388 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.266929+0000", 2024-09-13T14:51:06.388 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.400868011 2024-09-13T14:51:06.389 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.389 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.389 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:51:06.389 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.267042+0000", 2024-09-13T14:51:06.389 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000112576 2024-09-13T14:51:06.389 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.389 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.389 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,2,5", 2024-09-13T14:51:06.389 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.267161+0000", 2024-09-13T14:51:06.389 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000118644 2024-09-13T14:51:06.389 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.389 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.389 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:06.390 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.274891+0000", 2024-09-13T14:51:06.390 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0077304410000000002 2024-09-13T14:51:06.390 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.390 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.390 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:06.390 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.274935+0000", 2024-09-13T14:51:06.390 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.4326000000000002e-05 2024-09-13T14:51:06.390 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.390 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.390 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:06.390 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.282738+0000", 2024-09-13T14:51:06.390 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.007802308 2024-09-13T14:51:06.390 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.390 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.391 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:51:06.391 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.282792+0000", 2024-09-13T14:51:06.391 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.4347e-05 2024-09-13T14:51:06.391 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.391 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.391 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:51:06.391 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.282847+0000", 2024-09-13T14:51:06.391 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.4849999999999998e-05 2024-09-13T14:51:06.391 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.391 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.391 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:51:06.391 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.283023+0000", 2024-09-13T14:51:06.391 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00017625799999999999 2024-09-13T14:51:06.392 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:06.392 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:51:06.392 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:06.392 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.392 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.392 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:133952 8.1 8:899012f5:::benchmark_data_smithi102_313275_object3365:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e337)", 2024-09-13T14:51:06.392 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:47:47.864581+0000", 2024-09-13T14:51:06.392 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 198.38621028599999, 2024-09-13T14:51:06.392 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4316166779999999, 2024-09-13T14:51:06.392 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:51:06.392 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:51:06.392 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:06.392 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:51:06.393 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:51:06.393 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:51:06.393 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 133952 2024-09-13T14:51:06.393 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.393 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:51:06.393 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.393 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:51:06.393 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.864581+0000", 2024-09-13T14:51:06.393 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:06.393 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.393 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.393 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:51:06.393 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.864581+0000", 2024-09-13T14:51:06.393 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:06.394 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.394 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.394 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:51:06.394 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.864582+0000", 2024-09-13T14:51:06.394 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.0699999999999998e-07 2024-09-13T14:51:06.394 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.394 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.394 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:51:06.394 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.864584+0000", 2024-09-13T14:51:06.394 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.0619999999999999e-06 2024-09-13T14:51:06.394 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.394 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.394 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:51:06.395 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.864590+0000", 2024-09-13T14:51:06.395 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.3550000000000001e-06 2024-09-13T14:51:06.395 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.395 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.395 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:51:06.395 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.864595+0000", 2024-09-13T14:51:06.395 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.1719999999999998e-06 2024-09-13T14:51:06.395 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.395 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.395 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:06.395 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.866085+0000", 2024-09-13T14:51:06.395 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.001489986 2024-09-13T14:51:06.395 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.396 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.396 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:51:06.396 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.866092+0000", 2024-09-13T14:51:06.396 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.5429999999999999e-06 2024-09-13T14:51:06.396 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.396 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.396 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:06.396 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.274967+0000", 2024-09-13T14:51:06.396 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.408875093 2024-09-13T14:51:06.396 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.396 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.396 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:51:06.396 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.275076+0000", 2024-09-13T14:51:06.397 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000109224 2024-09-13T14:51:06.397 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.397 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.397 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,2,5", 2024-09-13T14:51:06.397 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.275189+0000", 2024-09-13T14:51:06.397 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00011342599999999999 2024-09-13T14:51:06.397 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.397 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.397 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:06.397 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.282674+0000", 2024-09-13T14:51:06.397 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0074845759999999997 2024-09-13T14:51:06.397 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.397 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.398 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:51:06.398 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.283031+0000", 2024-09-13T14:51:06.398 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00035696399999999998 2024-09-13T14:51:06.398 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.398 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.398 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:06.398 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.293749+0000", 2024-09-13T14:51:06.398 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.010717880000000001 2024-09-13T14:51:06.398 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.398 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.398 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:06.398 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.296031+0000", 2024-09-13T14:51:06.398 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0022818030000000002 2024-09-13T14:51:06.398 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.399 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.399 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:51:06.399 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.296090+0000", 2024-09-13T14:51:06.399 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.9215999999999997e-05 2024-09-13T14:51:06.399 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.399 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.399 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:51:06.399 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.296198+0000", 2024-09-13T14:51:06.399 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000107971 2024-09-13T14:51:06.399 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:06.399 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:51:06.399 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:06.399 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.400 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.400 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:133963 8.1 8:811019dd:::benchmark_data_smithi102_313275_object3376:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e337)", 2024-09-13T14:51:06.400 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:47:47.864601+0000", 2024-09-13T14:51:06.400 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 198.38619024299999, 2024-09-13T14:51:06.400 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4317937970000001, 2024-09-13T14:51:06.400 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:51:06.400 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:51:06.400 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:06.400 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:51:06.400 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:51:06.400 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:51:06.400 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 133963 2024-09-13T14:51:06.400 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.401 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:51:06.401 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.401 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:51:06.401 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.864601+0000", 2024-09-13T14:51:06.401 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:06.401 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.401 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.401 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:51:06.401 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.864601+0000", 2024-09-13T14:51:06.401 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:06.401 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.401 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.401 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:51:06.402 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.864602+0000", 2024-09-13T14:51:06.402 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.9299999999999999e-07 2024-09-13T14:51:06.402 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.402 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.402 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:51:06.402 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.864603+0000", 2024-09-13T14:51:06.402 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.778e-06 2024-09-13T14:51:06.402 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.402 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.402 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:51:06.402 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.864609+0000", 2024-09-13T14:51:06.402 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.1029999999999998e-06 2024-09-13T14:51:06.402 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.402 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.403 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:51:06.403 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.864614+0000", 2024-09-13T14:51:06.403 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.9579999999999998e-06 2024-09-13T14:51:06.403 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.403 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.403 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:06.403 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.866117+0000", 2024-09-13T14:51:06.403 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0015030110000000001 2024-09-13T14:51:06.403 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.403 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.403 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:51:06.403 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.866124+0000", 2024-09-13T14:51:06.403 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.9489999999999999e-06 2024-09-13T14:51:06.404 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.404 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.404 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:06.404 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.278477+0000", 2024-09-13T14:51:06.404 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4123528730000001 2024-09-13T14:51:06.404 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.404 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.404 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:51:06.404 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.278589+0000", 2024-09-13T14:51:06.404 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000111346 2024-09-13T14:51:06.404 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.404 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.404 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,2,5", 2024-09-13T14:51:06.405 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.278703+0000", 2024-09-13T14:51:06.405 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000114465 2024-09-13T14:51:06.405 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.405 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.405 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:06.405 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.282777+0000", 2024-09-13T14:51:06.405 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0040742479999999999 2024-09-13T14:51:06.405 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.405 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.405 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:51:06.405 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.283036+0000", 2024-09-13T14:51:06.405 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00025854300000000002 2024-09-13T14:51:06.405 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.406 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.406 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:06.406 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.295275+0000", 2024-09-13T14:51:06.406 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.012239243 2024-09-13T14:51:06.406 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.406 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.406 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:06.406 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.296237+0000", 2024-09-13T14:51:06.406 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00096223100000000002 2024-09-13T14:51:06.406 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.406 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.406 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:51:06.406 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.296292+0000", 2024-09-13T14:51:06.406 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.5109999999999999e-05 2024-09-13T14:51:06.407 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:06.407 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:06.407 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:51:06.407 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.296395+0000", 2024-09-13T14:51:06.407 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000102546 2024-09-13T14:51:06.407 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:06.407 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:51:06.407 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:06.407 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:06.407 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:51:06.407 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:51:06.462 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:06.495 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:06.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:06.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:06.559 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:51:06.597 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:06.598 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:51:06 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:51:06] "GET /metrics HTTP/1.1" 200 36443 "" "Prometheus/2.51.0" 2024-09-13T14:51:06.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:06.637 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:06.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:06.738 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:06.770 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:06.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:06.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:06.872 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:06.910 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:06.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:06.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:06.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:06 smithi102 ceph-mon[110183]: pgmap v1301: 109 pgs: 109 active+clean; 4.6 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-13T14:51:06.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:06 smithi102 ceph-mon[115269]: pgmap v1301: 109 pgs: 109 active+clean; 4.6 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-13T14:51:07.013 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:07.048 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:07.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:07.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:07.101 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 1 16 3627 3611 225.545 225.688 0.00610202 0.00441015 2024-09-13T14:51:07.151 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:07.188 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:07.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:07.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:07.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:06 smithi195 ceph-mon[104262]: pgmap v1301: 109 pgs: 109 active+clean; 4.6 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-13T14:51:07.290 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:07.325 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:07.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:07.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:07.427 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:07.467 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:07.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:07.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:07.569 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:07.605 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:07.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:07.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:07.707 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:07.736 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:51:07.739 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:07.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:07.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:07.842 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:07.874 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:07.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:07.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:07.976 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:08.007 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:08.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:08.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:08.101 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 2 15 7177 7162 223.732 221.938 0.00451543 0.00446122 2024-09-13T14:51:08.109 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:08.142 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:08.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:08.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:08.244 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:08.275 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:08.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:08.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:08.303 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T14:51:08.333 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:08.377 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:08.415 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:08.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:08.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:08.517 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:08.562 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:08.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:08.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:08.665 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:08.698 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:08.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:08.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:08.800 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:08.839 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:08.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:08.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:08.894 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:08 smithi102 ceph-mon[110183]: pgmap v1302: 109 pgs: 109 active+clean; 32 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 3.4 MiB/s wr, 54 op/s 2024-09-13T14:51:08.894 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:08 smithi102 ceph-mon[115269]: pgmap v1302: 109 pgs: 109 active+clean; 32 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 3.4 MiB/s wr, 54 op/s 2024-09-13T14:51:08.941 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:08.977 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:08.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:08.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:09.079 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:09.101 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 3 16 10318 10302 214.56 196.25 0.00401986 0.00465331 2024-09-13T14:51:09.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:09.117 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:09.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:09.218 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:09.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:08 smithi195 ceph-mon[104262]: pgmap v1302: 109 pgs: 109 active+clean; 32 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 3.4 MiB/s wr, 54 op/s 2024-09-13T14:51:09.258 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:09.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:09.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:09.360 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:09.394 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:09.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:09.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:09.496 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:09.530 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:09.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:09.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:09.632 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:09.669 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:09.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:09.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:09.772 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:09.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:09.802 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:09.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:09.804 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:09.817 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:09.902 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:09.934 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:09.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:09.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:10.036 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:10.067 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:10.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:10.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:10.101 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 4 16 13462 13446 210.042 196.5 0.00584257 0.00475434 2024-09-13T14:51:10.169 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:10.202 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:10.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:10.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:10.304 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:10.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:10.341 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:10.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:10.442 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:10.477 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:10.509 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:10.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:10.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:10.612 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:10.661 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:10.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:10.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:10.764 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:10.805 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:10.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:10.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:10.907 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:10.930 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:10.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:10.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:11.031 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:11.061 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:10 smithi102 ceph-mon[110183]: pgmap v1303: 109 pgs: 109 active+clean; 314 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 31 MiB/s wr, 498 op/s 2024-09-13T14:51:11.061 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:51:11.061 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:51:11.061 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:51:11.062 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:51:11.062 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:51:11.062 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:51:11.062 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:51:11.062 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:10 smithi102 ceph-mon[115269]: pgmap v1303: 109 pgs: 109 active+clean; 314 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 31 MiB/s wr, 498 op/s 2024-09-13T14:51:11.062 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:51:11.062 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:51:11.062 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:51:11.062 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:51:11.062 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:51:11.062 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:51:11.063 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:51:11.078 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:11.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:11.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:11.102 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 5 16 16183 16167 202.037 170.062 0.00758875 0.00494369 2024-09-13T14:51:11.181 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:11.219 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:11.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:11.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:11.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:10 smithi195 ceph-mon[104262]: pgmap v1303: 109 pgs: 109 active+clean; 314 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 31 MiB/s wr, 498 op/s 2024-09-13T14:51:11.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:51:11.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:51:11.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:51:11.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:51:11.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:51:11.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:51:11.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:51:11.321 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:11.363 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:11.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:11.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:11.465 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:11.498 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:11.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:11.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:11.600 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:11.634 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:11.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:11.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:11.736 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:11.769 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:11.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:11.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:11.871 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:11.907 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:11.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:11.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:12.009 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:12.048 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:12.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:12.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:12.102 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 6 16 19088 19072 198.623 181.562 0.00821092 0.00502899 2024-09-13T14:51:12.150 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:12.171 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:12.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:12.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:12.273 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:12.309 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:12.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:12.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:12.411 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:12.447 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:12.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:12.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:12.549 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:12.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:12.586 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:12.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:12.686 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:12.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:12.725 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:12.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:12.826 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:12.859 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:12.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:12.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:12.961 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:12.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:12.989 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:12.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:13.090 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:13.102 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 7 16 21522 21506 191.979 152.125 0.00721322 0.00520288 2024-09-13T14:51:13.125 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:13.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:13.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:13.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:12 smithi102 ceph-mon[110183]: pgmap v1304: 109 pgs: 109 active+clean; 752 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-13T14:51:13.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:12 smithi102 ceph-mon[115269]: pgmap v1304: 109 pgs: 109 active+clean; 752 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-13T14:51:13.228 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:13.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:12 smithi195 ceph-mon[104262]: pgmap v1304: 109 pgs: 109 active+clean; 752 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-13T14:51:13.255 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:13.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:13.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:13.358 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:13.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:13.394 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:13.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:13.495 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:13.518 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:13.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:13.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:13.619 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:13.656 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:13.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:13.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:13.758 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:13.788 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:13.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:13.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:13.890 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:13.923 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:13.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:13.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:14.025 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:14.057 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:14.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:14.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:14.102 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 8 16 23845 23829 186.128 145.188 0.00308293 0.00536325 2024-09-13T14:51:14.159 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:14.181 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:14.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:14.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:14.283 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:14.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:14.318 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:14.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:14.419 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:14.449 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:14.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:14.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:14.551 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:14.575 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:14.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:14.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:14.677 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:14.708 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:14.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:14.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:14.810 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:14.843 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:14.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:14.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:14.945 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:14.978 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:14.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:14.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:15.080 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:15.102 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 9 16 26068 26052 180.872 138.938 0.00806824 0.00552232 2024-09-13T14:51:15.114 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:15.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:15.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:15.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:14 smithi102 ceph-mon[110183]: pgmap v1305: 109 pgs: 109 active+clean; 1.1 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-09-13T14:51:15.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:14 smithi102 ceph-mon[115269]: pgmap v1305: 109 pgs: 109 active+clean; 1.1 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-09-13T14:51:15.216 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:15.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:14 smithi195 ceph-mon[104262]: pgmap v1305: 109 pgs: 109 active+clean; 1.1 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-09-13T14:51:15.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:15.252 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:15.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:15.353 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:15.390 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:15.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:15.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:15.492 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:15.529 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:15.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:15.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:15.631 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:15.667 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:15.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:15.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:15.769 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:15.803 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:15.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:15.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:15.905 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:15.939 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:15.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:15.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:16.041 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:16.069 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:16.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:16.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:16.103 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 10 16 28179 28163 175.978 131.938 0.00585581 0.00567733 2024-09-13T14:51:16.171 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:16.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:16.208 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:16.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:16.309 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:16.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:16.337 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:16.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:16.439 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:16.470 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:16.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:16.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:16.572 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:16.593 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:16.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:16.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:16.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:51:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:51:16] "GET /metrics HTTP/1.1" 200 36497 "" "Prometheus/2.51.0" 2024-09-13T14:51:16.696 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:16.733 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:16.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:16.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:16.817 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:51:16.820 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:51:16.835 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:16.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:16.866 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:16.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:16.962 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:16 smithi102 ceph-mon[110183]: pgmap v1306: 109 pgs: 109 active+clean; 1.2 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-09-13T14:51:16.962 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:16 smithi102 ceph-mon[110183]: from='client.36534 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:51:16.962 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:16 smithi102 ceph-mon[115269]: pgmap v1306: 109 pgs: 109 active+clean; 1.2 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-09-13T14:51:16.963 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:16 smithi102 ceph-mon[115269]: from='client.36534 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:51:16.967 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:17.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:17.004 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:17.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:17.104 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 11 16 30270 30254 171.86 130.688 0.00424512 0.00581409 2024-09-13T14:51:17.104 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:17.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:17.138 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:17.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:17.239 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:17.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:16 smithi195 ceph-mon[104262]: pgmap v1306: 109 pgs: 109 active+clean; 1.2 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-09-13T14:51:17.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:16 smithi195 ceph-mon[104262]: from='client.36534 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:51:17.274 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:17.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:17.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:17.376 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:17.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:17.412 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:17.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:17.514 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:17.547 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:17.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:17.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:17.651 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:17.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:17.672 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:17.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:17.773 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:17.794 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:17.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:17.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:17.896 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:17.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:17.934 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:17.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:18.035 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:18.065 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:18.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:18.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:18.103 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 12 16 31933 31917 166.2 103.938 0.0459848 0.00599539 2024-09-13T14:51:18.167 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:18.203 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:18.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:18.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:18.306 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:18.343 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:18.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:18.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:18.445 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:18.481 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:18.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:18.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:18.583 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:18.616 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:18.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:18.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:18.719 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:18.756 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:18.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:18.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:18.858 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:18.895 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:18.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:18.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:18.997 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:19.031 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:19.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:19.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:19.103 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 13 15 34241 34226 164.516 144.312 0.0113859 0.00607356 2024-09-13T14:51:19.134 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:19.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:18 smithi102 ceph-mon[110183]: pgmap v1307: 109 pgs: 109 active+clean; 1.5 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 131 MiB/s wr, 2.09k op/s 2024-09-13T14:51:19.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:18 smithi102 ceph-mon[115269]: pgmap v1307: 109 pgs: 109 active+clean; 1.5 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 131 MiB/s wr, 2.09k op/s 2024-09-13T14:51:19.171 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:19.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:19.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:19.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:18 smithi195 ceph-mon[104262]: pgmap v1307: 109 pgs: 109 active+clean; 1.5 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 131 MiB/s wr, 2.09k op/s 2024-09-13T14:51:19.273 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:19.310 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:19.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:19.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:19.411 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:19.445 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:19.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:19.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:19.547 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:19.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:19.580 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:19.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:19.681 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:19.722 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:19.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:19.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:19.824 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:19.859 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:19.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:19.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:19.961 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:19.995 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:19.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:19.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:20.097 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:20.103 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 14 16 36477 36461 162.741 139.688 0.00520845 0.00613854 2024-09-13T14:51:20.128 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:20.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:20.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:20.230 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:20.264 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:20.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:20.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:20.366 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:20.405 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:20.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:20.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:20.508 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:20.542 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:20.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:20.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:20.644 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:20.680 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:20.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:20.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:20.781 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:20.816 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:20.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:20.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:20.918 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:20.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:20.965 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:20.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:21.066 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:21.099 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:21.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:21.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:21.103 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 15 16 38596 38580 160.72 132.438 0.00445781 0.00621712 2024-09-13T14:51:21.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:20 smithi102 ceph-mon[110183]: pgmap v1308: 109 pgs: 109 active+clean; 1.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 159 MiB/s wr, 2.54k op/s 2024-09-13T14:51:21.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:20 smithi102 ceph-mon[115269]: pgmap v1308: 109 pgs: 109 active+clean; 1.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 159 MiB/s wr, 2.54k op/s 2024-09-13T14:51:21.201 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:21.239 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:21.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:21.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:21.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:20 smithi195 ceph-mon[104262]: pgmap v1308: 109 pgs: 109 active+clean; 1.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 159 MiB/s wr, 2.54k op/s 2024-09-13T14:51:21.341 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:21.412 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:21.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:21.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:21.514 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:21.549 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:21.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:21.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:21.651 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:21.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:21.686 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:21.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:21.787 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:21.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:21.822 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:21.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:21.923 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:21.959 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:21.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:21.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:22.061 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:22.094 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:22.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:22.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:22.103 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 16 16 40473 40457 158.007 117.312 0.00458516 0.00632397 2024-09-13T14:51:22.196 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:22.243 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:22.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:22.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:22.258 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":1306,"stamp":"2024-09-13T14:51:15.687913+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1313282021,"num_objects":19994,"num_object_clones":0,"num_object_copies":59982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19994,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":20874,"num_write_kb":1290449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":195,"num_bytes_recovered":8718480,"num_keys_recovered":20,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24128,"ondisk_log_size":24128,"up":327,"acting":327,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":327,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":8606500,"kb_used_data":3684644,"kb_used_omap":41,"kb_used_meta":4921750,"kb_avail":741354716,"statfs":{"total":767960285184,"available":759147229184,"internally_reserved":0,"allocated":3773075456,"data_stored":3757463494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42913,"internal_metadata":5039872095},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,3,7],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1308491776,"num_objects":19966,"num_object_clones":0,"num_object_copies":59898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19966,"num_write_kb":1277824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004505"},"pg_stats":[{"pgid":"1.11","version":"90'149","reported_seq":497,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:02.389354+0000","last_change":"2024-09-13T14:47:38.688630+0000","last_active":"2024-09-13T14:51:02.389354+0000","last_peered":"2024-09-13T14:51:02.389354+0000","last_clean":"2024-09-13T14:51:02.389354+0000","last_became_active":"2024-09-13T14:47:38.460077+0000","last_became_peered":"2024-09-13T14:47:38.460077+0000","last_unstale":"2024-09-13T14:51:02.389354+0000","last_undegraded":"2024-09-13T14:51:02.389354+0000","last_fullsized":"2024-09-13T14:51:02.389354+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:38.688341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.688341+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:14:17.232921+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":667,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:02.201807+0000","last_change":"2024-09-13T14:47:47.868783+0000","last_active":"2024-09-13T14:51:02.201807+0000","last_peered":"2024-09-13T14:51:02.201807+0000","last_clean":"2024-09-13T14:51:02.201807+0000","last_became_active":"2024-09-13T14:47:47.868478+0000","last_became_peered":"2024-09-13T14:47:47.868478+0000","last_unstale":"2024-09-13T14:51:02.201807+0000","last_undegraded":"2024-09-13T14:51:02.201807+0000","last_fullsized":"2024-09-13T14:51:02.201807+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:27:01.784142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":398,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T14:50:59.743543+0000","last_change":"2024-09-13T14:46:01.772811+0000","last_active":"2024-09-13T14:50:59.743543+0000","last_peered":"2024-09-13T14:50:59.743543+0000","last_clean":"2024-09-13T14:50:59.743543+0000","last_became_active":"2024-09-13T14:46:01.764775+0000","last_became_peered":"2024-09-13T14:46:01.764775+0000","last_unstale":"2024-09-13T14:50:59.743543+0000","last_undegraded":"2024-09-13T14:50:59.743543+0000","last_fullsized":"2024-09-13T14:50:59.743543+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:42:20.738421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"90'149","reported_seq":443,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T14:50:59.741957+0000","last_change":"2024-09-13T14:47:41.482552+0000","last_active":"2024-09-13T14:50:59.741957+0000","last_peered":"2024-09-13T14:50:59.741957+0000","last_clean":"2024-09-13T14:50:59.741957+0000","last_became_active":"2024-09-13T14:47:38.466659+0000","last_became_peered":"2024-09-13T14:47:38.466659+0000","last_unstale":"2024-09-13T14:50:59.741957+0000","last_undegraded":"2024-09-13T14:50:59.741957+0000","last_fullsized":"2024-09-13T14:50:59.741957+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:41.482364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:41.482364+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:50:49.239389+0000","scrub_duration":0.027843855000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":180,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T14:50:59.742675+0000","last_change":"2024-09-13T14:36:13.862181+0000","last_active":"2024-09-13T14:50:59.742675+0000","last_peered":"2024-09-13T14:50:59.742675+0000","last_clean":"2024-09-13T14:50:59.742675+0000","last_became_active":"2024-09-13T14:36:13.861873+0000","last_became_peered":"2024-09-13T14:36:13.861873+0000","last_unstale":"2024-09-13T14:50:59.742675+0000","last_undegraded":"2024-09-13T14:50:59.742675+0000","last_fullsized":"2024-09-13T14:50:59.742675+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:14.085700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":290,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:02.388926+0000","last_change":"2024-09-13T14:38:53.896672+0000","last_active":"2024-09-13T14:51:02.388926+0000","last_peered":"2024-09-13T14:51:02.388926+0000","last_clean":"2024-09-13T14:51:02.388926+0000","last_became_active":"2024-09-13T14:38:53.896346+0000","last_became_peered":"2024-09-13T14:38:53.896346+0000","last_unstale":"2024-09-13T14:51:02.388926+0000","last_undegraded":"2024-09-13T14:51:02.388926+0000","last_fullsized":"2024-09-13T14:51:02.388926+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:57.789686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"90'149","reported_seq":517,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:02.004814+0000","last_change":"2024-09-13T14:47:51.226821+0000","last_active":"2024-09-13T14:51:02.004814+0000","last_peered":"2024-09-13T14:51:02.004814+0000","last_clean":"2024-09-13T14:51:02.004814+0000","last_became_active":"2024-09-13T14:47:50.881689+0000","last_became_peered":"2024-09-13T14:47:50.881689+0000","last_unstale":"2024-09-13T14:51:02.004814+0000","last_undegraded":"2024-09-13T14:51:02.004814+0000","last_fullsized":"2024-09-13T14:51:02.004814+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:51.226748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:51.226748+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:06:02.065990+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":290,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:02.388967+0000","last_change":"2024-09-13T14:38:53.893021+0000","last_active":"2024-09-13T14:51:02.388967+0000","last_peered":"2024-09-13T14:51:02.388967+0000","last_clean":"2024-09-13T14:51:02.388967+0000","last_became_active":"2024-09-13T14:38:53.892668+0000","last_became_peered":"2024-09-13T14:38:53.892668+0000","last_unstale":"2024-09-13T14:51:02.388967+0000","last_undegraded":"2024-09-13T14:51:02.388967+0000","last_fullsized":"2024-09-13T14:51:02.388967+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:57.180738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":352,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:01.746164+0000","last_change":"2024-09-13T14:37:48.134642+0000","last_active":"2024-09-13T14:51:01.746164+0000","last_peered":"2024-09-13T14:51:01.746164+0000","last_clean":"2024-09-13T14:51:01.746164+0000","last_became_active":"2024-09-13T14:37:48.134402+0000","last_became_peered":"2024-09-13T14:37:48.134402+0000","last_unstale":"2024-09-13T14:51:01.746164+0000","last_undegraded":"2024-09-13T14:51:01.746164+0000","last_fullsized":"2024-09-13T14:51:01.746164+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:51.313663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"90'149","reported_seq":460,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T14:50:59.742280+0000","last_change":"2024-09-13T14:47:50.892667+0000","last_active":"2024-09-13T14:50:59.742280+0000","last_peered":"2024-09-13T14:50:59.742280+0000","last_clean":"2024-09-13T14:50:59.742280+0000","last_became_active":"2024-09-13T14:47:50.883312+0000","last_became_peered":"2024-09-13T14:47:50.883312+0000","last_unstale":"2024-09-13T14:50:59.742280+0000","last_undegraded":"2024-09-13T14:50:59.742280+0000","last_fullsized":"2024-09-13T14:50:59.742280+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:45.441819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:45.441819+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:39.138573+0000","scrub_duration":0.021368558999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":692,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:02.201935+0000","last_change":"2024-09-13T14:50:27.491166+0000","last_active":"2024-09-13T14:51:02.201935+0000","last_peered":"2024-09-13T14:51:02.201935+0000","last_clean":"2024-09-13T14:51:02.201935+0000","last_became_active":"2024-09-13T14:50:27.489905+0000","last_became_peered":"2024-09-13T14:50:27.489905+0000","last_unstale":"2024-09-13T14:51:02.201935+0000","last_undegraded":"2024-09-13T14:51:02.201935+0000","last_fullsized":"2024-09-13T14:51:02.201935+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:32.237904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":293,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:02.389085+0000","last_change":"2024-09-13T14:38:03.554500+0000","last_active":"2024-09-13T14:51:02.389085+0000","last_peered":"2024-09-13T14:51:02.389085+0000","last_clean":"2024-09-13T14:51:02.389085+0000","last_became_active":"2024-09-13T14:38:03.532910+0000","last_became_peered":"2024-09-13T14:38:03.532910+0000","last_unstale":"2024-09-13T14:51:02.389085+0000","last_undegraded":"2024-09-13T14:51:02.389085+0000","last_fullsized":"2024-09-13T14:51:02.389085+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:27:50.843498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.5","version":"378'1313","reported_seq":1332,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:12.927415+0000","last_change":"2024-09-13T14:50:50.779688+0000","last_active":"2024-09-13T14:51:12.927415+0000","last_peered":"2024-09-13T14:51:12.927415+0000","last_clean":"2024-09-13T14:51:12.927415+0000","last_became_active":"2024-09-13T14:50:50.779411+0000","last_became_peered":"2024-09-13T14:50:50.779411+0000","last_unstale":"2024-09-13T14:51:12.927415+0000","last_undegraded":"2024-09-13T14:51:12.927415+0000","last_fullsized":"2024-09-13T14:51:12.927415+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:50:49.763471+0000","objects_scrubbed":0,"log_size":1313,"log_dups_size":0,"ondisk_log_size":1313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:09:03.891969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86048768,"num_objects":1313,"num_object_clones":0,"num_object_copies":3939,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1313,"num_write_kb":84032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"90'149","reported_seq":462,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T14:50:59.743679+0000","last_change":"2024-09-13T14:47:50.887140+0000","last_active":"2024-09-13T14:50:59.743679+0000","last_peered":"2024-09-13T14:50:59.743679+0000","last_clean":"2024-09-13T14:50:59.743679+0000","last_became_active":"2024-09-13T14:47:50.886603+0000","last_became_peered":"2024-09-13T14:47:50.886603+0000","last_unstale":"2024-09-13T14:50:59.743679+0000","last_undegraded":"2024-09-13T14:50:59.743679+0000","last_fullsized":"2024-09-13T14:50:59.743679+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:40.786610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:40.786610+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:00.936651+0000","scrub_duration":0.013725185000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":144,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T14:50:59.744000+0000","last_change":"2024-09-13T14:38:53.940238+0000","last_active":"2024-09-13T14:50:59.744000+0000","last_peered":"2024-09-13T14:50:59.744000+0000","last_clean":"2024-09-13T14:50:59.744000+0000","last_became_active":"2024-09-13T14:38:53.892825+0000","last_became_peered":"2024-09-13T14:38:53.892825+0000","last_unstale":"2024-09-13T14:50:59.744000+0000","last_undegraded":"2024-09-13T14:50:59.744000+0000","last_fullsized":"2024-09-13T14:50:59.744000+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:46.666581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":200,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T14:50:59.742140+0000","last_change":"2024-09-13T14:38:53.898907+0000","last_active":"2024-09-13T14:50:59.742140+0000","last_peered":"2024-09-13T14:50:59.742140+0000","last_clean":"2024-09-13T14:50:59.742140+0000","last_became_active":"2024-09-13T14:38:53.898572+0000","last_became_peered":"2024-09-13T14:38:53.898572+0000","last_unstale":"2024-09-13T14:50:59.742140+0000","last_undegraded":"2024-09-13T14:50:59.742140+0000","last_fullsized":"2024-09-13T14:50:59.742140+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:29.222150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"378'1550","reported_seq":1569,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:14.522777+0000","last_change":"2024-09-13T14:50:50.782298+0000","last_active":"2024-09-13T14:51:14.522777+0000","last_peered":"2024-09-13T14:51:14.522777+0000","last_clean":"2024-09-13T14:51:14.522777+0000","last_became_active":"2024-09-13T14:50:50.782029+0000","last_became_peered":"2024-09-13T14:50:50.782029+0000","last_unstale":"2024-09-13T14:51:14.522777+0000","last_undegraded":"2024-09-13T14:51:14.522777+0000","last_fullsized":"2024-09-13T14:51:14.522777+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:50:49.763471+0000","objects_scrubbed":0,"log_size":1550,"log_dups_size":0,"ondisk_log_size":1550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:15:06.095221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101580800,"num_objects":1550,"num_object_clones":0,"num_object_copies":4650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1550,"num_write_kb":99200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"90'149","reported_seq":539,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:02.201693+0000","last_change":"2024-09-13T14:47:38.466101+0000","last_active":"2024-09-13T14:51:02.201693+0000","last_peered":"2024-09-13T14:51:02.201693+0000","last_clean":"2024-09-13T14:51:02.201693+0000","last_became_active":"2024-09-13T14:47:38.465607+0000","last_became_peered":"2024-09-13T14:47:38.465607+0000","last_unstale":"2024-09-13T14:51:02.201693+0000","last_undegraded":"2024-09-13T14:51:02.201693+0000","last_fullsized":"2024-09-13T14:51:02.201693+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:32.060817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:32.060817+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:55.362724+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":157,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T14:50:59.743981+0000","last_change":"2024-09-13T14:47:47.878423+0000","last_active":"2024-09-13T14:50:59.743981+0000","last_peered":"2024-09-13T14:50:59.743981+0000","last_clean":"2024-09-13T14:50:59.743981+0000","last_became_active":"2024-09-13T14:47:47.866926+0000","last_became_peered":"2024-09-13T14:47:47.866926+0000","last_unstale":"2024-09-13T14:50:59.743981+0000","last_undegraded":"2024-09-13T14:50:59.743981+0000","last_fullsized":"2024-09-13T14:50:59.743981+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:11:07.534097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":399,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T14:51:00.391274+0000","last_change":"2024-09-13T14:47:47.860804+0000","last_active":"2024-09-13T14:51:00.391274+0000","last_peered":"2024-09-13T14:51:00.391274+0000","last_clean":"2024-09-13T14:51:00.391274+0000","last_became_active":"2024-09-13T14:47:47.860496+0000","last_became_peered":"2024-09-13T14:47:47.860496+0000","last_unstale":"2024-09-13T14:51:00.391274+0000","last_undegraded":"2024-09-13T14:51:00.391274+0000","last_fullsized":"2024-09-13T14:51:00.391274+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:17:57.542599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"378'1147","reported_seq":1160,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:11.468429+0000","last_change":"2024-09-13T14:50:50.784134+0000","last_active":"2024-09-13T14:51:11.468429+0000","last_peered":"2024-09-13T14:51:11.468429+0000","last_clean":"2024-09-13T14:51:11.468429+0000","last_became_active":"2024-09-13T14:50:50.783762+0000","last_became_peered":"2024-09-13T14:50:50.783762+0000","last_unstale":"2024-09-13T14:51:11.468429+0000","last_undegraded":"2024-09-13T14:51:11.468429+0000","last_fullsized":"2024-09-13T14:51:11.468429+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:50:49.763471+0000","objects_scrubbed":0,"log_size":1147,"log_dups_size":0,"ondisk_log_size":1147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:14:00.109438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75169792,"num_objects":1147,"num_object_clones":0,"num_object_copies":3441,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1147,"num_write_kb":73408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"90'149","reported_seq":487,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T14:50:59.742765+0000","last_change":"2024-09-13T14:47:38.468287+0000","last_active":"2024-09-13T14:50:59.742765+0000","last_peered":"2024-09-13T14:50:59.742765+0000","last_clean":"2024-09-13T14:50:59.742765+0000","last_became_active":"2024-09-13T14:47:38.467941+0000","last_became_peered":"2024-09-13T14:47:38.467941+0000","last_unstale":"2024-09-13T14:50:59.742765+0000","last_undegraded":"2024-09-13T14:50:59.742765+0000","last_fullsized":"2024-09-13T14:50:59.742765+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:34.091782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:34.091782+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:34:20.519248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":381,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T14:50:59.743969+0000","last_change":"2024-09-13T14:37:49.611774+0000","last_active":"2024-09-13T14:50:59.743969+0000","last_peered":"2024-09-13T14:50:59.743969+0000","last_clean":"2024-09-13T14:50:59.743969+0000","last_became_active":"2024-09-13T14:37:49.611088+0000","last_became_peered":"2024-09-13T14:37:49.611088+0000","last_unstale":"2024-09-13T14:50:59.743969+0000","last_undegraded":"2024-09-13T14:50:59.743969+0000","last_fullsized":"2024-09-13T14:50:59.743969+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:08.933213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":95,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:02.004840+0000","last_change":"2024-09-13T14:47:47.865523+0000","last_active":"2024-09-13T14:51:02.004840+0000","last_peered":"2024-09-13T14:51:02.004840+0000","last_clean":"2024-09-13T14:51:02.004840+0000","last_became_active":"2024-09-13T14:47:47.865180+0000","last_became_peered":"2024-09-13T14:47:47.865180+0000","last_unstale":"2024-09-13T14:51:02.004840+0000","last_undegraded":"2024-09-13T14:51:02.004840+0000","last_fullsized":"2024-09-13T14:51:02.004840+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:39:35.479251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.6","version":"378'1095","reported_seq":1108,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:11.481379+0000","last_change":"2024-09-13T14:50:51.090850+0000","last_active":"2024-09-13T14:51:11.481379+0000","last_peered":"2024-09-13T14:51:11.481379+0000","last_clean":"2024-09-13T14:51:11.481379+0000","last_became_active":"2024-09-13T14:50:51.090552+0000","last_became_peered":"2024-09-13T14:50:51.090552+0000","last_unstale":"2024-09-13T14:51:11.481379+0000","last_undegraded":"2024-09-13T14:51:11.481379+0000","last_fullsized":"2024-09-13T14:51:11.481379+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:50:49.763471+0000","objects_scrubbed":0,"log_size":1095,"log_dups_size":0,"ondisk_log_size":1095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:44:01.059274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71761920,"num_objects":1095,"num_object_clones":0,"num_object_copies":3285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1095,"num_write_kb":70080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"90'149","reported_seq":493,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T14:51:00.392744+0000","last_change":"2024-09-13T14:47:38.459769+0000","last_active":"2024-09-13T14:51:00.392744+0000","last_peered":"2024-09-13T14:51:00.392744+0000","last_clean":"2024-09-13T14:51:00.392744+0000","last_became_active":"2024-09-13T14:47:38.459440+0000","last_became_peered":"2024-09-13T14:47:38.459440+0000","last_unstale":"2024-09-13T14:51:00.392744+0000","last_undegraded":"2024-09-13T14:51:00.392744+0000","last_fullsized":"2024-09-13T14:51:00.392744+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:30.526999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:30.526999+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:00.016329+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":191,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T14:50:59.742493+0000","last_change":"2024-09-13T14:37:49.168003+0000","last_active":"2024-09-13T14:50:59.742493+0000","last_peered":"2024-09-13T14:50:59.742493+0000","last_clean":"2024-09-13T14:50:59.742493+0000","last_became_active":"2024-09-13T14:37:49.167638+0000","last_became_peered":"2024-09-13T14:37:49.167638+0000","last_unstale":"2024-09-13T14:50:59.742493+0000","last_undegraded":"2024-09-13T14:50:59.742493+0000","last_fullsized":"2024-09-13T14:50:59.742493+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:29.051981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":290,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:02.389693+0000","last_change":"2024-09-13T14:47:47.868094+0000","last_active":"2024-09-13T14:51:02.389693+0000","last_peered":"2024-09-13T14:51:02.389693+0000","last_clean":"2024-09-13T14:51:02.389693+0000","last_became_active":"2024-09-13T14:47:47.867750+0000","last_became_peered":"2024-09-13T14:47:47.867750+0000","last_unstale":"2024-09-13T14:51:02.389693+0000","last_undegraded":"2024-09-13T14:51:02.389693+0000","last_fullsized":"2024-09-13T14:51:02.389693+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:55:38.203481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1","version":"378'1403","reported_seq":1422,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:13.070526+0000","last_change":"2024-09-13T14:50:51.091554+0000","last_active":"2024-09-13T14:51:13.070526+0000","last_peered":"2024-09-13T14:51:13.070526+0000","last_clean":"2024-09-13T14:51:13.070526+0000","last_became_active":"2024-09-13T14:50:51.091227+0000","last_became_peered":"2024-09-13T14:50:51.091227+0000","last_unstale":"2024-09-13T14:51:13.070526+0000","last_undegraded":"2024-09-13T14:51:13.070526+0000","last_fullsized":"2024-09-13T14:51:13.070526+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:50:49.763471+0000","objects_scrubbed":0,"log_size":1403,"log_dups_size":0,"ondisk_log_size":1403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:45:57.473254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91947008,"num_objects":1403,"num_object_clones":0,"num_object_copies":4209,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1403,"num_write_kb":89792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"90'149","reported_seq":554,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:01.746001+0000","last_change":"2024-09-13T14:46:55.355650+0000","last_active":"2024-09-13T14:51:01.746001+0000","last_peered":"2024-09-13T14:51:01.746001+0000","last_clean":"2024-09-13T14:51:01.746001+0000","last_became_active":"2024-09-13T14:46:48.150338+0000","last_became_peered":"2024-09-13T14:46:48.150338+0000","last_unstale":"2024-09-13T14:51:01.746001+0000","last_undegraded":"2024-09-13T14:51:01.746001+0000","last_fullsized":"2024-09-13T14:51:01.746001+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:46:55.355521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:46:55.355521+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:30:53.006278+0000","scrub_duration":0.002025448,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":641,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:02.201667+0000","last_change":"2024-09-13T14:38:03.549595+0000","last_active":"2024-09-13T14:51:02.201667+0000","last_peered":"2024-09-13T14:51:02.201667+0000","last_clean":"2024-09-13T14:51:02.201667+0000","last_became_active":"2024-09-13T14:38:03.533331+0000","last_became_peered":"2024-09-13T14:38:03.533331+0000","last_unstale":"2024-09-13T14:51:02.201667+0000","last_undegraded":"2024-09-13T14:51:02.201667+0000","last_fullsized":"2024-09-13T14:51:02.201667+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:59:02.470343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":374,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T14:51:00.392504+0000","last_change":"2024-09-13T14:46:26.980367+0000","last_active":"2024-09-13T14:51:00.392504+0000","last_peered":"2024-09-13T14:51:00.392504+0000","last_clean":"2024-09-13T14:51:00.392504+0000","last_became_active":"2024-09-13T14:46:26.979937+0000","last_became_peered":"2024-09-13T14:46:26.979937+0000","last_unstale":"2024-09-13T14:51:00.392504+0000","last_undegraded":"2024-09-13T14:51:00.392504+0000","last_fullsized":"2024-09-13T14:51:00.392504+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:56.830872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"378'1557","reported_seq":1576,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:14.528980+0000","last_change":"2024-09-13T14:50:51.092387+0000","last_active":"2024-09-13T14:51:14.528980+0000","last_peered":"2024-09-13T14:51:14.528980+0000","last_clean":"2024-09-13T14:51:14.528980+0000","last_became_active":"2024-09-13T14:50:51.092062+0000","last_became_peered":"2024-09-13T14:50:51.092062+0000","last_unstale":"2024-09-13T14:51:14.528980+0000","last_undegraded":"2024-09-13T14:51:14.528980+0000","last_fullsized":"2024-09-13T14:51:14.528980+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:50:49.763471+0000","objects_scrubbed":0,"log_size":1557,"log_dups_size":0,"ondisk_log_size":1557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:20:14.882212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102039552,"num_objects":1557,"num_object_clones":0,"num_object_copies":4671,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1557,"num_write_kb":99648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"90'149","reported_seq":456,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:01.744546+0000","last_change":"2024-09-13T14:46:54.393663+0000","last_active":"2024-09-13T14:51:01.744546+0000","last_peered":"2024-09-13T14:51:01.744546+0000","last_clean":"2024-09-13T14:51:01.744546+0000","last_became_active":"2024-09-13T14:46:48.149004+0000","last_became_peered":"2024-09-13T14:46:48.149004+0000","last_unstale":"2024-09-13T14:51:01.744546+0000","last_undegraded":"2024-09-13T14:51:01.744546+0000","last_fullsized":"2024-09-13T14:51:01.744546+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:46:54.393204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:46:54.393204+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:32:40.003989+0000","scrub_duration":0.015172223,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":307,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T14:51:00.391981+0000","last_change":"2024-09-13T14:47:47.867571+0000","last_active":"2024-09-13T14:51:00.391981+0000","last_peered":"2024-09-13T14:51:00.391981+0000","last_clean":"2024-09-13T14:51:00.391981+0000","last_became_active":"2024-09-13T14:47:47.867269+0000","last_became_peered":"2024-09-13T14:47:47.867269+0000","last_unstale":"2024-09-13T14:51:00.391981+0000","last_undegraded":"2024-09-13T14:51:00.391981+0000","last_fullsized":"2024-09-13T14:51:00.391981+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:46.093600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":436,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T14:51:00.392655+0000","last_change":"2024-09-13T14:47:47.866925+0000","last_active":"2024-09-13T14:51:00.392655+0000","last_peered":"2024-09-13T14:51:00.392655+0000","last_clean":"2024-09-13T14:51:00.392655+0000","last_became_active":"2024-09-13T14:47:47.866596+0000","last_became_peered":"2024-09-13T14:47:47.866596+0000","last_unstale":"2024-09-13T14:51:00.392655+0000","last_undegraded":"2024-09-13T14:51:00.392655+0000","last_fullsized":"2024-09-13T14:51:00.392655+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:11:25.617891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.3","version":"378'989","reported_seq":1002,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:11.226857+0000","last_change":"2024-09-13T14:50:50.782641+0000","last_active":"2024-09-13T14:51:11.226857+0000","last_peered":"2024-09-13T14:51:11.226857+0000","last_clean":"2024-09-13T14:51:11.226857+0000","last_became_active":"2024-09-13T14:50:50.782253+0000","last_became_peered":"2024-09-13T14:50:50.782253+0000","last_unstale":"2024-09-13T14:51:11.226857+0000","last_undegraded":"2024-09-13T14:51:11.226857+0000","last_fullsized":"2024-09-13T14:51:11.226857+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:50:49.763471+0000","objects_scrubbed":0,"log_size":989,"log_dups_size":0,"ondisk_log_size":989,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:29:58.217758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64815104,"num_objects":989,"num_object_clones":0,"num_object_copies":2967,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":989,"num_write_kb":63296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.b","version":"90'149","reported_seq":530,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:02.004583+0000","last_change":"2024-09-13T14:47:50.891055+0000","last_active":"2024-09-13T14:51:02.004583+0000","last_peered":"2024-09-13T14:51:02.004583+0000","last_clean":"2024-09-13T14:51:02.004583+0000","last_became_active":"2024-09-13T14:47:50.890368+0000","last_became_peered":"2024-09-13T14:47:50.890368+0000","last_unstale":"2024-09-13T14:51:02.004583+0000","last_undegraded":"2024-09-13T14:51:02.004583+0000","last_fullsized":"2024-09-13T14:51:02.004583+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:39.645908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:39.645908+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:29.800728+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":388,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T14:51:00.391803+0000","last_change":"2024-09-13T14:46:01.762127+0000","last_active":"2024-09-13T14:51:00.391803+0000","last_peered":"2024-09-13T14:51:00.391803+0000","last_clean":"2024-09-13T14:51:00.391803+0000","last_became_active":"2024-09-13T14:46:01.761795+0000","last_became_peered":"2024-09-13T14:46:01.761795+0000","last_unstale":"2024-09-13T14:51:00.391803+0000","last_undegraded":"2024-09-13T14:51:00.391803+0000","last_fullsized":"2024-09-13T14:51:00.391803+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:41:15.626775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":383,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T14:51:00.392846+0000","last_change":"2024-09-13T14:47:47.861718+0000","last_active":"2024-09-13T14:51:00.392846+0000","last_peered":"2024-09-13T14:51:00.392846+0000","last_clean":"2024-09-13T14:51:00.392846+0000","last_became_active":"2024-09-13T14:47:47.861023+0000","last_became_peered":"2024-09-13T14:47:47.861023+0000","last_unstale":"2024-09-13T14:51:00.392846+0000","last_undegraded":"2024-09-13T14:51:00.392846+0000","last_fullsized":"2024-09-13T14:51:00.392846+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:12:13.274053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"378'1309","reported_seq":1328,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:13.067791+0000","last_change":"2024-09-13T14:50:51.091067+0000","last_active":"2024-09-13T14:51:13.067791+0000","last_peered":"2024-09-13T14:51:13.067791+0000","last_clean":"2024-09-13T14:51:13.067791+0000","last_became_active":"2024-09-13T14:50:51.090690+0000","last_became_peered":"2024-09-13T14:50:51.090690+0000","last_unstale":"2024-09-13T14:51:13.067791+0000","last_undegraded":"2024-09-13T14:51:13.067791+0000","last_fullsized":"2024-09-13T14:51:13.067791+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:50:49.763471+0000","objects_scrubbed":0,"log_size":1309,"log_dups_size":0,"ondisk_log_size":1309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:41:24.042790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85786624,"num_objects":1309,"num_object_clones":0,"num_object_copies":3927,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1309,"num_write_kb":83776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"90'149","reported_seq":456,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:01.744659+0000","last_change":"2024-09-13T14:46:52.334715+0000","last_active":"2024-09-13T14:51:01.744659+0000","last_peered":"2024-09-13T14:51:01.744659+0000","last_clean":"2024-09-13T14:51:01.744659+0000","last_became_active":"2024-09-13T14:46:48.158685+0000","last_became_peered":"2024-09-13T14:46:48.158685+0000","last_unstale":"2024-09-13T14:51:01.744659+0000","last_undegraded":"2024-09-13T14:51:01.744659+0000","last_fullsized":"2024-09-13T14:51:01.744659+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:46:52.334301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:46:52.334301+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:46:03.579759+0000","scrub_duration":0.0050940330000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":667,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:02.201453+0000","last_change":"2024-09-13T14:47:47.861249+0000","last_active":"2024-09-13T14:51:02.201453+0000","last_peered":"2024-09-13T14:51:02.201453+0000","last_clean":"2024-09-13T14:51:02.201453+0000","last_became_active":"2024-09-13T14:47:47.860873+0000","last_became_peered":"2024-09-13T14:47:47.860873+0000","last_unstale":"2024-09-13T14:51:02.201453+0000","last_undegraded":"2024-09-13T14:51:02.201453+0000","last_fullsized":"2024-09-13T14:51:02.201453+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:37:50.709437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":95,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:02.004979+0000","last_change":"2024-09-13T14:47:47.870185+0000","last_active":"2024-09-13T14:51:02.004979+0000","last_peered":"2024-09-13T14:51:02.004979+0000","last_clean":"2024-09-13T14:51:02.004979+0000","last_became_active":"2024-09-13T14:47:47.869862+0000","last_became_peered":"2024-09-13T14:47:47.869862+0000","last_unstale":"2024-09-13T14:51:02.004979+0000","last_undegraded":"2024-09-13T14:51:02.004979+0000","last_fullsized":"2024-09-13T14:51:02.004979+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:07:02.812306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"378'1348","reported_seq":1367,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:13.066705+0000","last_change":"2024-09-13T14:50:50.781969+0000","last_active":"2024-09-13T14:51:13.066705+0000","last_peered":"2024-09-13T14:51:13.066705+0000","last_clean":"2024-09-13T14:51:13.066705+0000","last_became_active":"2024-09-13T14:50:50.781655+0000","last_became_peered":"2024-09-13T14:50:50.781655+0000","last_unstale":"2024-09-13T14:51:13.066705+0000","last_undegraded":"2024-09-13T14:51:13.066705+0000","last_fullsized":"2024-09-13T14:51:13.066705+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:50:49.763471+0000","objects_scrubbed":0,"log_size":1348,"log_dups_size":0,"ondisk_log_size":1348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:06:21.667297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88342528,"num_objects":1348,"num_object_clones":0,"num_object_copies":4044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1348,"num_write_kb":86272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":558,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T14:50:59.743284+0000","last_change":"2024-09-13T14:47:31.684319+0000","last_active":"2024-09-13T14:50:59.743284+0000","last_peered":"2024-09-13T14:50:59.743284+0000","last_clean":"2024-09-13T14:50:59.743284+0000","last_became_active":"2024-09-13T14:47:29.415658+0000","last_became_peered":"2024-09-13T14:47:29.415658+0000","last_unstale":"2024-09-13T14:50:59.743284+0000","last_undegraded":"2024-09-13T14:50:59.743284+0000","last_fullsized":"2024-09-13T14:50:59.743284+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:31.681433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:31.681433+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:29:34.663513+0000","scrub_duration":0.0059601139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":687,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T14:50:59.742324+0000","last_change":"2024-09-13T14:46:04.780718+0000","last_active":"2024-09-13T14:50:59.742324+0000","last_peered":"2024-09-13T14:50:59.742324+0000","last_clean":"2024-09-13T14:50:59.742324+0000","last_became_active":"2024-09-13T14:46:04.780120+0000","last_became_peered":"2024-09-13T14:46:04.780120+0000","last_unstale":"2024-09-13T14:50:59.742324+0000","last_undegraded":"2024-09-13T14:50:59.742324+0000","last_fullsized":"2024-09-13T14:50:59.742324+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:57:14.475888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":594,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:02.201340+0000","last_change":"2024-09-13T14:37:49.160719+0000","last_active":"2024-09-13T14:51:02.201340+0000","last_peered":"2024-09-13T14:51:02.201340+0000","last_clean":"2024-09-13T14:51:02.201340+0000","last_became_active":"2024-09-13T14:37:49.160401+0000","last_became_peered":"2024-09-13T14:37:49.160401+0000","last_unstale":"2024-09-13T14:51:02.201340+0000","last_undegraded":"2024-09-13T14:51:02.201340+0000","last_fullsized":"2024-09-13T14:51:02.201340+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:21.737945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":226,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T14:51:00.391420+0000","last_change":"2024-09-13T14:37:48.172940+0000","last_active":"2024-09-13T14:51:00.391420+0000","last_peered":"2024-09-13T14:51:00.391420+0000","last_clean":"2024-09-13T14:51:00.391420+0000","last_became_active":"2024-09-13T14:37:48.172060+0000","last_became_peered":"2024-09-13T14:37:48.172060+0000","last_unstale":"2024-09-13T14:51:00.391420+0000","last_undegraded":"2024-09-13T14:51:00.391420+0000","last_fullsized":"2024-09-13T14:51:00.391420+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:07.480645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.b","version":"378'1288","reported_seq":1307,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:12.965896+0000","last_change":"2024-09-13T14:50:51.091679+0000","last_active":"2024-09-13T14:51:12.965896+0000","last_peered":"2024-09-13T14:51:12.965896+0000","last_clean":"2024-09-13T14:51:12.965896+0000","last_became_active":"2024-09-13T14:50:51.091340+0000","last_became_peered":"2024-09-13T14:50:51.091340+0000","last_unstale":"2024-09-13T14:51:12.965896+0000","last_undegraded":"2024-09-13T14:51:12.965896+0000","last_fullsized":"2024-09-13T14:51:12.965896+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:50:49.763471+0000","objects_scrubbed":0,"log_size":1288,"log_dups_size":0,"ondisk_log_size":1288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:58:33.368611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84410368,"num_objects":1288,"num_object_clones":0,"num_object_copies":3864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1288,"num_write_kb":82432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"90'149","reported_seq":524,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:02.004689+0000","last_change":"2024-09-13T14:47:50.884721+0000","last_active":"2024-09-13T14:51:02.004689+0000","last_peered":"2024-09-13T14:51:02.004689+0000","last_clean":"2024-09-13T14:51:02.004689+0000","last_became_active":"2024-09-13T14:47:50.880185+0000","last_became_peered":"2024-09-13T14:47:50.880185+0000","last_unstale":"2024-09-13T14:51:02.004689+0000","last_undegraded":"2024-09-13T14:51:02.004689+0000","last_fullsized":"2024-09-13T14:51:02.004689+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:34.603316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:34.603316+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:38.321686+0000","scrub_duration":0.0057131120000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":628,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T14:50:59.743266+0000","last_change":"2024-09-13T14:47:47.892973+0000","last_active":"2024-09-13T14:50:59.743266+0000","last_peered":"2024-09-13T14:50:59.743266+0000","last_clean":"2024-09-13T14:50:59.743266+0000","last_became_active":"2024-09-13T14:47:47.866986+0000","last_became_peered":"2024-09-13T14:47:47.866986+0000","last_unstale":"2024-09-13T14:50:59.743266+0000","last_undegraded":"2024-09-13T14:50:59.743266+0000","last_fullsized":"2024-09-13T14:50:59.743266+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:07.353290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":284,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:02.389308+0000","last_change":"2024-09-13T14:38:03.525682+0000","last_active":"2024-09-13T14:51:02.389308+0000","last_peered":"2024-09-13T14:51:02.389308+0000","last_clean":"2024-09-13T14:51:02.389308+0000","last_became_active":"2024-09-13T14:38:03.525406+0000","last_became_peered":"2024-09-13T14:38:03.525406+0000","last_unstale":"2024-09-13T14:51:02.389308+0000","last_undegraded":"2024-09-13T14:51:02.389308+0000","last_fullsized":"2024-09-13T14:51:02.389308+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:41:28.970300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":305,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:02.389776+0000","last_change":"2024-09-13T14:47:47.873233+0000","last_active":"2024-09-13T14:51:02.389776+0000","last_peered":"2024-09-13T14:51:02.389776+0000","last_clean":"2024-09-13T14:51:02.389776+0000","last_became_active":"2024-09-13T14:47:47.872788+0000","last_became_peered":"2024-09-13T14:47:47.872788+0000","last_unstale":"2024-09-13T14:51:02.389776+0000","last_undegraded":"2024-09-13T14:51:02.389776+0000","last_fullsized":"2024-09-13T14:51:02.389776+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:52.409108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"378'1063","reported_seq":1075,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:11.476806+0000","last_change":"2024-09-13T14:50:50.782599+0000","last_active":"2024-09-13T14:51:11.476806+0000","last_peered":"2024-09-13T14:51:11.476806+0000","last_clean":"2024-09-13T14:51:11.476806+0000","last_became_active":"2024-09-13T14:50:50.782273+0000","last_became_peered":"2024-09-13T14:50:50.782273+0000","last_unstale":"2024-09-13T14:51:11.476806+0000","last_undegraded":"2024-09-13T14:51:11.476806+0000","last_fullsized":"2024-09-13T14:51:11.476806+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:50:49.763471+0000","objects_scrubbed":0,"log_size":1063,"log_dups_size":0,"ondisk_log_size":1063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:27:54.626945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69664768,"num_objects":1063,"num_object_clones":0,"num_object_copies":3189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1063,"num_write_kb":68032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"90'149","reported_seq":575,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T14:50:59.743361+0000","last_change":"2024-09-13T14:47:29.629488+0000","last_active":"2024-09-13T14:50:59.743361+0000","last_peered":"2024-09-13T14:50:59.743361+0000","last_clean":"2024-09-13T14:50:59.743361+0000","last_became_active":"2024-09-13T14:47:29.403279+0000","last_became_peered":"2024-09-13T14:47:29.403279+0000","last_unstale":"2024-09-13T14:50:59.743361+0000","last_undegraded":"2024-09-13T14:50:59.743361+0000","last_fullsized":"2024-09-13T14:50:59.743361+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:29.629342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:29.629342+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:14:55.249896+0000","scrub_duration":0.0099290990000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":578,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:02.595636+0000","last_change":"2024-09-13T14:46:04.782955+0000","last_active":"2024-09-13T14:51:02.595636+0000","last_peered":"2024-09-13T14:51:02.595636+0000","last_clean":"2024-09-13T14:51:02.595636+0000","last_became_active":"2024-09-13T14:46:04.782250+0000","last_became_peered":"2024-09-13T14:46:04.782250+0000","last_unstale":"2024-09-13T14:51:02.595636+0000","last_undegraded":"2024-09-13T14:51:02.595636+0000","last_fullsized":"2024-09-13T14:51:02.595636+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:24.596922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":95,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:02.004652+0000","last_change":"2024-09-13T14:47:47.866684+0000","last_active":"2024-09-13T14:51:02.004652+0000","last_peered":"2024-09-13T14:51:02.004652+0000","last_clean":"2024-09-13T14:51:02.004652+0000","last_became_active":"2024-09-13T14:47:47.866400+0000","last_became_peered":"2024-09-13T14:47:47.866400+0000","last_unstale":"2024-09-13T14:51:02.004652+0000","last_undegraded":"2024-09-13T14:51:02.004652+0000","last_fullsized":"2024-09-13T14:51:02.004652+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:08:34.289658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":95,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:02.005152+0000","last_change":"2024-09-13T14:47:47.870218+0000","last_active":"2024-09-13T14:51:02.005152+0000","last_peered":"2024-09-13T14:51:02.005152+0000","last_clean":"2024-09-13T14:51:02.005152+0000","last_became_active":"2024-09-13T14:47:47.869907+0000","last_became_peered":"2024-09-13T14:47:47.869907+0000","last_unstale":"2024-09-13T14:51:02.005152+0000","last_undegraded":"2024-09-13T14:51:02.005152+0000","last_fullsized":"2024-09-13T14:51:02.005152+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:41:49.141441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"378'1409","reported_seq":1428,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:12.973378+0000","last_change":"2024-09-13T14:50:50.779542+0000","last_active":"2024-09-13T14:51:12.973378+0000","last_peered":"2024-09-13T14:51:12.973378+0000","last_clean":"2024-09-13T14:51:12.973378+0000","last_became_active":"2024-09-13T14:50:50.779111+0000","last_became_peered":"2024-09-13T14:50:50.779111+0000","last_unstale":"2024-09-13T14:51:12.973378+0000","last_undegraded":"2024-09-13T14:51:12.973378+0000","last_fullsized":"2024-09-13T14:51:12.973378+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:50:49.763471+0000","objects_scrubbed":0,"log_size":1409,"log_dups_size":0,"ondisk_log_size":1409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:01:27.827997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92340224,"num_objects":1409,"num_object_clones":0,"num_object_copies":4227,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1409,"num_write_kb":90176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"90'149","reported_seq":477,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T14:50:59.742195+0000","last_change":"2024-09-13T14:47:32.275753+0000","last_active":"2024-09-13T14:50:59.742195+0000","last_peered":"2024-09-13T14:50:59.742195+0000","last_clean":"2024-09-13T14:50:59.742195+0000","last_became_active":"2024-09-13T14:47:29.401066+0000","last_became_peered":"2024-09-13T14:47:29.401066+0000","last_unstale":"2024-09-13T14:50:59.742195+0000","last_undegraded":"2024-09-13T14:50:59.742195+0000","last_fullsized":"2024-09-13T14:50:59.742195+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:32.275562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:32.275562+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:43:41.056775+0000","scrub_duration":0.0054674260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":1132,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T14:50:59.742152+0000","last_change":"2024-09-13T14:47:50.892842+0000","last_active":"2024-09-13T14:50:59.742152+0000","last_peered":"2024-09-13T14:50:59.742152+0000","last_clean":"2024-09-13T14:50:59.742152+0000","last_became_active":"2024-09-13T14:47:50.892237+0000","last_became_peered":"2024-09-13T14:47:50.892237+0000","last_unstale":"2024-09-13T14:50:59.742152+0000","last_undegraded":"2024-09-13T14:50:59.742152+0000","last_fullsized":"2024-09-13T14:50:59.742152+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:20:05.796762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":8388646,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":388,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:02.493229+0000","last_change":"2024-09-13T14:38:03.528133+0000","last_active":"2024-09-13T14:51:02.493229+0000","last_peered":"2024-09-13T14:51:02.493229+0000","last_clean":"2024-09-13T14:51:02.493229+0000","last_became_active":"2024-09-13T14:38:03.527829+0000","last_became_peered":"2024-09-13T14:38:03.527829+0000","last_unstale":"2024-09-13T14:51:02.493229+0000","last_undegraded":"2024-09-13T14:51:02.493229+0000","last_fullsized":"2024-09-13T14:51:02.493229+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:48.254169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":95,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:02.004925+0000","last_change":"2024-09-13T14:47:47.877272+0000","last_active":"2024-09-13T14:51:02.004925+0000","last_peered":"2024-09-13T14:51:02.004925+0000","last_clean":"2024-09-13T14:51:02.004925+0000","last_became_active":"2024-09-13T14:47:47.877003+0000","last_became_peered":"2024-09-13T14:47:47.877003+0000","last_unstale":"2024-09-13T14:51:02.004925+0000","last_undegraded":"2024-09-13T14:51:02.004925+0000","last_fullsized":"2024-09-13T14:51:02.004925+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:47:23.172452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.f","version":"378'1084","reported_seq":1096,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:11.484234+0000","last_change":"2024-09-13T14:50:50.783992+0000","last_active":"2024-09-13T14:51:11.484234+0000","last_peered":"2024-09-13T14:51:11.484234+0000","last_clean":"2024-09-13T14:51:11.484234+0000","last_became_active":"2024-09-13T14:50:50.783687+0000","last_became_peered":"2024-09-13T14:50:50.783687+0000","last_unstale":"2024-09-13T14:51:11.484234+0000","last_undegraded":"2024-09-13T14:51:11.484234+0000","last_fullsized":"2024-09-13T14:51:11.484234+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:50:49.763471+0000","objects_scrubbed":0,"log_size":1084,"log_dups_size":0,"ondisk_log_size":1084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:30.412734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71041024,"num_objects":1084,"num_object_clones":0,"num_object_copies":3252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1084,"num_write_kb":69376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"90'149","reported_seq":528,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:02.201944+0000","last_change":"2024-09-13T14:47:33.067298+0000","last_active":"2024-09-13T14:51:02.201944+0000","last_peered":"2024-09-13T14:51:02.201944+0000","last_clean":"2024-09-13T14:51:02.201944+0000","last_became_active":"2024-09-13T14:47:29.419416+0000","last_became_peered":"2024-09-13T14:47:29.419416+0000","last_unstale":"2024-09-13T14:51:02.201944+0000","last_undegraded":"2024-09-13T14:51:02.201944+0000","last_fullsized":"2024-09-13T14:51:02.201944+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:33.066954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:33.066954+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:05:42.808060+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":201,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T14:51:01.476752+0000","last_change":"2024-09-13T14:46:01.763906+0000","last_active":"2024-09-13T14:51:01.476752+0000","last_peered":"2024-09-13T14:51:01.476752+0000","last_clean":"2024-09-13T14:51:01.476752+0000","last_became_active":"2024-09-13T14:46:01.763542+0000","last_became_peered":"2024-09-13T14:46:01.763542+0000","last_unstale":"2024-09-13T14:51:01.476752+0000","last_undegraded":"2024-09-13T14:51:01.476752+0000","last_fullsized":"2024-09-13T14:51:01.476752+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:09:36.652767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":1383,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:02.004990+0000","last_change":"2024-09-13T14:47:50.883365+0000","last_active":"2024-09-13T14:51:02.004990+0000","last_peered":"2024-09-13T14:51:02.004990+0000","last_clean":"2024-09-13T14:51:02.004990+0000","last_became_active":"2024-09-13T14:47:50.882609+0000","last_became_peered":"2024-09-13T14:47:50.882609+0000","last_unstale":"2024-09-13T14:51:02.004990+0000","last_undegraded":"2024-09-13T14:51:02.004990+0000","last_fullsized":"2024-09-13T14:51:02.004990+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:12:17.961575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":95,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:02.004821+0000","last_change":"2024-09-13T14:47:47.866979+0000","last_active":"2024-09-13T14:51:02.004821+0000","last_peered":"2024-09-13T14:51:02.004821+0000","last_clean":"2024-09-13T14:51:02.004821+0000","last_became_active":"2024-09-13T14:47:47.866634+0000","last_became_peered":"2024-09-13T14:47:47.866634+0000","last_unstale":"2024-09-13T14:51:02.004821+0000","last_undegraded":"2024-09-13T14:51:02.004821+0000","last_fullsized":"2024-09-13T14:51:02.004821+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:47:09.696139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.e","version":"378'982","reported_seq":995,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:11.238801+0000","last_change":"2024-09-13T14:50:50.782813+0000","last_active":"2024-09-13T14:51:11.238801+0000","last_peered":"2024-09-13T14:51:11.238801+0000","last_clean":"2024-09-13T14:51:11.238801+0000","last_became_active":"2024-09-13T14:50:50.782515+0000","last_became_peered":"2024-09-13T14:50:50.782515+0000","last_unstale":"2024-09-13T14:51:11.238801+0000","last_undegraded":"2024-09-13T14:51:11.238801+0000","last_fullsized":"2024-09-13T14:51:11.238801+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:50:49.763471+0000","objects_scrubbed":0,"log_size":982,"log_dups_size":0,"ondisk_log_size":982,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:20.624059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64356352,"num_objects":982,"num_object_clones":0,"num_object_copies":2946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":982,"num_write_kb":62848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"90'149","reported_seq":534,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:02.389675+0000","last_change":"2024-09-13T14:47:29.556225+0000","last_active":"2024-09-13T14:51:02.389675+0000","last_peered":"2024-09-13T14:51:02.389675+0000","last_clean":"2024-09-13T14:51:02.389675+0000","last_became_active":"2024-09-13T14:47:29.408774+0000","last_became_peered":"2024-09-13T14:47:29.408774+0000","last_unstale":"2024-09-13T14:51:02.389675+0000","last_undegraded":"2024-09-13T14:51:02.389675+0000","last_fullsized":"2024-09-13T14:51:02.389675+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:29.556149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:29.556149+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:35.691178+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":95,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:02.005035+0000","last_change":"2024-09-13T14:47:47.876562+0000","last_active":"2024-09-13T14:51:02.005035+0000","last_peered":"2024-09-13T14:51:02.005035+0000","last_clean":"2024-09-13T14:51:02.005035+0000","last_became_active":"2024-09-13T14:47:47.876297+0000","last_became_peered":"2024-09-13T14:47:47.876297+0000","last_unstale":"2024-09-13T14:51:02.005035+0000","last_undegraded":"2024-09-13T14:51:02.005035+0000","last_fullsized":"2024-09-13T14:51:02.005035+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:03:27.761394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":903,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T14:50:59.743977+0000","last_change":"2024-09-13T14:38:04.596192+0000","last_active":"2024-09-13T14:50:59.743977+0000","last_peered":"2024-09-13T14:50:59.743977+0000","last_clean":"2024-09-13T14:50:59.743977+0000","last_became_active":"2024-09-13T14:38:04.593299+0000","last_became_peered":"2024-09-13T14:38:04.593299+0000","last_unstale":"2024-09-13T14:50:59.743977+0000","last_undegraded":"2024-09-13T14:50:59.743977+0000","last_fullsized":"2024-09-13T14:50:59.743977+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:34.241410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":284,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:02.389754+0000","last_change":"2024-09-13T14:38:03.533120+0000","last_active":"2024-09-13T14:51:02.389754+0000","last_peered":"2024-09-13T14:51:02.389754+0000","last_clean":"2024-09-13T14:51:02.389754+0000","last_became_active":"2024-09-13T14:38:03.532829+0000","last_became_peered":"2024-09-13T14:38:03.532829+0000","last_unstale":"2024-09-13T14:51:02.389754+0000","last_undegraded":"2024-09-13T14:51:02.389754+0000","last_fullsized":"2024-09-13T14:51:02.389754+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:24.514661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.d","version":"378'1080","reported_seq":1093,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:11.245130+0000","last_change":"2024-09-13T14:50:50.782589+0000","last_active":"2024-09-13T14:51:11.245130+0000","last_peered":"2024-09-13T14:51:11.245130+0000","last_clean":"2024-09-13T14:51:11.245130+0000","last_became_active":"2024-09-13T14:50:50.782292+0000","last_became_peered":"2024-09-13T14:50:50.782292+0000","last_unstale":"2024-09-13T14:51:11.245130+0000","last_undegraded":"2024-09-13T14:51:11.245130+0000","last_fullsized":"2024-09-13T14:51:11.245130+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:50:49.763471+0000","objects_scrubbed":0,"log_size":1080,"log_dups_size":0,"ondisk_log_size":1080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:01:30.753189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70778880,"num_objects":1080,"num_object_clones":0,"num_object_copies":3240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1080,"num_write_kb":69120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"90'149","reported_seq":477,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T14:51:01.277977+0000","last_change":"2024-09-13T14:47:30.528588+0000","last_active":"2024-09-13T14:51:01.277977+0000","last_peered":"2024-09-13T14:51:01.277977+0000","last_clean":"2024-09-13T14:51:01.277977+0000","last_became_active":"2024-09-13T14:47:29.413621+0000","last_became_peered":"2024-09-13T14:47:29.413621+0000","last_unstale":"2024-09-13T14:51:01.277977+0000","last_undegraded":"2024-09-13T14:51:01.277977+0000","last_fullsized":"2024-09-13T14:51:01.277977+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:30.528112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:30.528112+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:30.462009+0000","scrub_duration":0.012751485,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":686,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:02.201077+0000","last_change":"2024-09-13T14:38:53.938227+0000","last_active":"2024-09-13T14:51:02.201077+0000","last_peered":"2024-09-13T14:51:02.201077+0000","last_clean":"2024-09-13T14:51:02.201077+0000","last_became_active":"2024-09-13T14:38:53.902867+0000","last_became_peered":"2024-09-13T14:38:53.902867+0000","last_unstale":"2024-09-13T14:51:02.201077+0000","last_undegraded":"2024-09-13T14:51:02.201077+0000","last_fullsized":"2024-09-13T14:51:02.201077+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:16.831756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":565,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:02.005301+0000","last_change":"2024-09-13T14:47:50.884706+0000","last_active":"2024-09-13T14:51:02.005301+0000","last_peered":"2024-09-13T14:51:02.005301+0000","last_clean":"2024-09-13T14:51:02.005301+0000","last_became_active":"2024-09-13T14:47:50.879750+0000","last_became_peered":"2024-09-13T14:47:50.879750+0000","last_unstale":"2024-09-13T14:51:02.005301+0000","last_undegraded":"2024-09-13T14:51:02.005301+0000","last_fullsized":"2024-09-13T14:51:02.005301+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:34:32.941266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":668,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:02.201024+0000","last_change":"2024-09-13T14:38:53.900107+0000","last_active":"2024-09-13T14:51:02.201024+0000","last_peered":"2024-09-13T14:51:02.201024+0000","last_clean":"2024-09-13T14:51:02.201024+0000","last_became_active":"2024-09-13T14:38:53.899556+0000","last_became_peered":"2024-09-13T14:38:53.899556+0000","last_unstale":"2024-09-13T14:51:02.201024+0000","last_undegraded":"2024-09-13T14:51:02.201024+0000","last_fullsized":"2024-09-13T14:51:02.201024+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:15:47.918163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"378'1349","reported_seq":1368,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:13.066204+0000","last_change":"2024-09-13T14:50:50.776299+0000","last_active":"2024-09-13T14:51:13.066204+0000","last_peered":"2024-09-13T14:51:13.066204+0000","last_clean":"2024-09-13T14:51:13.066204+0000","last_became_active":"2024-09-13T14:50:50.776003+0000","last_became_peered":"2024-09-13T14:50:50.776003+0000","last_unstale":"2024-09-13T14:51:13.066204+0000","last_undegraded":"2024-09-13T14:51:13.066204+0000","last_fullsized":"2024-09-13T14:51:13.066204+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:50:49.763471+0000","objects_scrubbed":0,"log_size":1349,"log_dups_size":0,"ondisk_log_size":1349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:49.877589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88408064,"num_objects":1349,"num_object_clones":0,"num_object_copies":4047,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1349,"num_write_kb":86336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"90'149","reported_seq":525,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:02.201576+0000","last_change":"2024-09-13T14:47:29.999643+0000","last_active":"2024-09-13T14:51:02.201576+0000","last_peered":"2024-09-13T14:51:02.201576+0000","last_clean":"2024-09-13T14:51:02.201576+0000","last_became_active":"2024-09-13T14:47:29.410951+0000","last_became_peered":"2024-09-13T14:47:29.410951+0000","last_unstale":"2024-09-13T14:51:02.201576+0000","last_undegraded":"2024-09-13T14:51:02.201576+0000","last_fullsized":"2024-09-13T14:51:02.201576+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:29.999519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:29.999519+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:04:54.887884+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":310,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:02.389698+0000","last_change":"2024-09-13T14:47:47.883475+0000","last_active":"2024-09-13T14:51:02.389698+0000","last_peered":"2024-09-13T14:51:02.389698+0000","last_clean":"2024-09-13T14:51:02.389698+0000","last_became_active":"2024-09-13T14:47:47.871525+0000","last_became_peered":"2024-09-13T14:47:47.871525+0000","last_unstale":"2024-09-13T14:51:02.389698+0000","last_undegraded":"2024-09-13T14:51:02.389698+0000","last_fullsized":"2024-09-13T14:51:02.389698+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:33.903432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":181,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T14:50:59.744030+0000","last_change":"2024-09-13T14:46:01.781508+0000","last_active":"2024-09-13T14:50:59.744030+0000","last_peered":"2024-09-13T14:50:59.744030+0000","last_clean":"2024-09-13T14:50:59.744030+0000","last_became_active":"2024-09-13T14:46:01.767096+0000","last_became_peered":"2024-09-13T14:46:01.767096+0000","last_unstale":"2024-09-13T14:50:59.744030+0000","last_undegraded":"2024-09-13T14:50:59.744030+0000","last_fullsized":"2024-09-13T14:50:59.744030+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:20:01.119189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":372,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T14:51:01.277879+0000","last_change":"2024-09-13T14:37:49.172218+0000","last_active":"2024-09-13T14:51:01.277879+0000","last_peered":"2024-09-13T14:51:01.277879+0000","last_clean":"2024-09-13T14:51:01.277879+0000","last_became_active":"2024-09-13T14:37:49.171718+0000","last_became_peered":"2024-09-13T14:37:49.171718+0000","last_unstale":"2024-09-13T14:51:01.277879+0000","last_undegraded":"2024-09-13T14:51:01.277879+0000","last_fullsized":"2024-09-13T14:51:01.277879+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:46:54.626521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":402,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T14:51:00.392135+0000","last_change":"2024-09-13T14:47:47.868409+0000","last_active":"2024-09-13T14:51:00.392135+0000","last_peered":"2024-09-13T14:51:00.392135+0000","last_clean":"2024-09-13T14:51:00.392135+0000","last_became_active":"2024-09-13T14:47:47.867712+0000","last_became_peered":"2024-09-13T14:47:47.867712+0000","last_unstale":"2024-09-13T14:51:00.392135+0000","last_undegraded":"2024-09-13T14:51:00.392135+0000","last_fullsized":"2024-09-13T14:51:00.392135+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:57:14.647493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":373,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T14:51:00.391120+0000","last_change":"2024-09-13T14:37:48.128447+0000","last_active":"2024-09-13T14:51:00.391120+0000","last_peered":"2024-09-13T14:51:00.391120+0000","last_clean":"2024-09-13T14:51:00.391120+0000","last_became_active":"2024-09-13T14:37:48.128143+0000","last_became_peered":"2024-09-13T14:37:48.128143+0000","last_unstale":"2024-09-13T14:51:00.391120+0000","last_undegraded":"2024-09-13T14:51:00.391120+0000","last_fullsized":"2024-09-13T14:51:00.391120+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:22.204039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":95,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:02.004628+0000","last_change":"2024-09-13T14:47:47.867935+0000","last_active":"2024-09-13T14:51:02.004628+0000","last_peered":"2024-09-13T14:51:02.004628+0000","last_clean":"2024-09-13T14:51:02.004628+0000","last_became_active":"2024-09-13T14:47:47.867616+0000","last_became_peered":"2024-09-13T14:47:47.867616+0000","last_unstale":"2024-09-13T14:51:02.004628+0000","last_undegraded":"2024-09-13T14:51:02.004628+0000","last_fullsized":"2024-09-13T14:51:02.004628+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:52:38.619916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":146,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T14:51:01.245004+0000","last_change":"2024-09-13T14:38:53.954281+0000","last_active":"2024-09-13T14:51:01.245004+0000","last_peered":"2024-09-13T14:51:01.245004+0000","last_clean":"2024-09-13T14:51:01.245004+0000","last_became_active":"2024-09-13T14:38:53.900912+0000","last_became_peered":"2024-09-13T14:38:53.900912+0000","last_unstale":"2024-09-13T14:51:01.245004+0000","last_undegraded":"2024-09-13T14:51:01.245004+0000","last_fullsized":"2024-09-13T14:51:01.245004+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:47.876275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"90'149","reported_seq":528,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:02.595614+0000","last_change":"2024-09-13T14:47:38.468095+0000","last_active":"2024-09-13T14:51:02.595614+0000","last_peered":"2024-09-13T14:51:02.595614+0000","last_clean":"2024-09-13T14:51:02.595614+0000","last_became_active":"2024-09-13T14:47:38.467526+0000","last_became_peered":"2024-09-13T14:47:38.467526+0000","last_unstale":"2024-09-13T14:51:02.595614+0000","last_undegraded":"2024-09-13T14:51:02.595614+0000","last_fullsized":"2024-09-13T14:51:02.595614+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:31.013128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:31.013128+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:17:20.508142+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":207,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T14:50:59.742699+0000","last_change":"2024-09-13T14:47:47.861842+0000","last_active":"2024-09-13T14:50:59.742699+0000","last_peered":"2024-09-13T14:50:59.742699+0000","last_clean":"2024-09-13T14:50:59.742699+0000","last_became_active":"2024-09-13T14:47:47.861492+0000","last_became_peered":"2024-09-13T14:47:47.861492+0000","last_unstale":"2024-09-13T14:50:59.742699+0000","last_undegraded":"2024-09-13T14:50:59.742699+0000","last_fullsized":"2024-09-13T14:50:59.742699+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:17.279533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":609,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:02.201039+0000","last_change":"2024-09-13T14:38:53.899535+0000","last_active":"2024-09-13T14:51:02.201039+0000","last_peered":"2024-09-13T14:51:02.201039+0000","last_clean":"2024-09-13T14:51:02.201039+0000","last_became_active":"2024-09-13T14:38:53.899183+0000","last_became_peered":"2024-09-13T14:38:53.899183+0000","last_unstale":"2024-09-13T14:51:02.201039+0000","last_undegraded":"2024-09-13T14:51:02.201039+0000","last_fullsized":"2024-09-13T14:51:02.201039+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:33.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":283,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:02.389940+0000","last_change":"2024-09-13T14:38:03.543426+0000","last_active":"2024-09-13T14:51:02.389940+0000","last_peered":"2024-09-13T14:51:02.389940+0000","last_clean":"2024-09-13T14:51:02.389940+0000","last_became_active":"2024-09-13T14:38:03.542661+0000","last_became_peered":"2024-09-13T14:38:03.542661+0000","last_unstale":"2024-09-13T14:51:02.389940+0000","last_undegraded":"2024-09-13T14:51:02.389940+0000","last_fullsized":"2024-09-13T14:51:02.389940+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:54.050127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":645,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:02.201167+0000","last_change":"2024-09-13T14:47:47.868624+0000","last_active":"2024-09-13T14:51:02.201167+0000","last_peered":"2024-09-13T14:51:02.201167+0000","last_clean":"2024-09-13T14:51:02.201167+0000","last_became_active":"2024-09-13T14:47:47.868031+0000","last_became_peered":"2024-09-13T14:47:47.868031+0000","last_unstale":"2024-09-13T14:51:02.201167+0000","last_undegraded":"2024-09-13T14:51:02.201167+0000","last_fullsized":"2024-09-13T14:51:02.201167+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:05:25.365023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":220,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T14:50:59.742787+0000","last_change":"2024-09-13T14:47:47.859825+0000","last_active":"2024-09-13T14:50:59.742787+0000","last_peered":"2024-09-13T14:50:59.742787+0000","last_clean":"2024-09-13T14:50:59.742787+0000","last_became_active":"2024-09-13T14:47:47.859069+0000","last_became_peered":"2024-09-13T14:47:47.859069+0000","last_unstale":"2024-09-13T14:50:59.742787+0000","last_undegraded":"2024-09-13T14:50:59.742787+0000","last_fullsized":"2024-09-13T14:50:59.742787+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:18:36.452321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":676,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:02.201309+0000","last_change":"2024-09-13T14:47:47.868945+0000","last_active":"2024-09-13T14:51:02.201309+0000","last_peered":"2024-09-13T14:51:02.201309+0000","last_clean":"2024-09-13T14:51:02.201309+0000","last_became_active":"2024-09-13T14:47:47.868296+0000","last_became_peered":"2024-09-13T14:47:47.868296+0000","last_unstale":"2024-09-13T14:51:02.201309+0000","last_undegraded":"2024-09-13T14:51:02.201309+0000","last_fullsized":"2024-09-13T14:51:02.201309+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:22:24.778693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":291,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:02.388981+0000","last_change":"2024-09-13T14:38:53.892948+0000","last_active":"2024-09-13T14:51:02.388981+0000","last_peered":"2024-09-13T14:51:02.388981+0000","last_clean":"2024-09-13T14:51:02.388981+0000","last_became_active":"2024-09-13T14:38:53.892665+0000","last_became_peered":"2024-09-13T14:38:53.892665+0000","last_unstale":"2024-09-13T14:51:02.388981+0000","last_undegraded":"2024-09-13T14:51:02.388981+0000","last_fullsized":"2024-09-13T14:51:02.388981+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:08.292359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":95,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:02.004393+0000","last_change":"2024-09-13T14:47:47.871164+0000","last_active":"2024-09-13T14:51:02.004393+0000","last_peered":"2024-09-13T14:51:02.004393+0000","last_clean":"2024-09-13T14:51:02.004393+0000","last_became_active":"2024-09-13T14:47:47.870862+0000","last_became_peered":"2024-09-13T14:47:47.870862+0000","last_unstale":"2024-09-13T14:51:02.004393+0000","last_undegraded":"2024-09-13T14:51:02.004393+0000","last_fullsized":"2024-09-13T14:51:02.004393+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:41:34.392081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":665,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:02.201489+0000","last_change":"2024-09-13T14:50:27.490554+0000","last_active":"2024-09-13T14:51:02.201489+0000","last_peered":"2024-09-13T14:51:02.201489+0000","last_clean":"2024-09-13T14:51:02.201489+0000","last_became_active":"2024-09-13T14:50:27.490211+0000","last_became_peered":"2024-09-13T14:50:27.490211+0000","last_unstale":"2024-09-13T14:51:02.201489+0000","last_undegraded":"2024-09-13T14:51:02.201489+0000","last_fullsized":"2024-09-13T14:51:02.201489+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:57:01.947741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":455,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T14:51:00.392084+0000","last_change":"2024-09-13T14:47:47.868246+0000","last_active":"2024-09-13T14:51:00.392084+0000","last_peered":"2024-09-13T14:51:00.392084+0000","last_clean":"2024-09-13T14:51:00.392084+0000","last_became_active":"2024-09-13T14:47:47.867817+0000","last_became_peered":"2024-09-13T14:47:47.867817+0000","last_unstale":"2024-09-13T14:51:00.392084+0000","last_undegraded":"2024-09-13T14:51:00.392084+0000","last_fullsized":"2024-09-13T14:51:00.392084+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:07.612451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":177,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:02.595799+0000","last_change":"2024-09-13T14:50:27.510133+0000","last_active":"2024-09-13T14:51:02.595799+0000","last_peered":"2024-09-13T14:51:02.595799+0000","last_clean":"2024-09-13T14:51:02.595799+0000","last_became_active":"2024-09-13T14:50:27.508774+0000","last_became_peered":"2024-09-13T14:50:27.508774+0000","last_unstale":"2024-09-13T14:51:02.595799+0000","last_undegraded":"2024-09-13T14:51:02.595799+0000","last_fullsized":"2024-09-13T14:51:02.595799+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:31:04.531746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":283,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:02.389576+0000","last_change":"2024-09-13T14:38:03.527021+0000","last_active":"2024-09-13T14:51:02.389576+0000","last_peered":"2024-09-13T14:51:02.389576+0000","last_clean":"2024-09-13T14:51:02.389576+0000","last_became_active":"2024-09-13T14:38:03.526698+0000","last_became_peered":"2024-09-13T14:38:03.526698+0000","last_unstale":"2024-09-13T14:51:02.389576+0000","last_undegraded":"2024-09-13T14:51:02.389576+0000","last_fullsized":"2024-09-13T14:51:02.389576+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:01:35.734086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":399,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T14:51:00.392050+0000","last_change":"2024-09-13T14:47:47.866784+0000","last_active":"2024-09-13T14:51:00.392050+0000","last_peered":"2024-09-13T14:51:00.392050+0000","last_clean":"2024-09-13T14:51:00.392050+0000","last_became_active":"2024-09-13T14:47:47.866483+0000","last_became_peered":"2024-09-13T14:47:47.866483+0000","last_unstale":"2024-09-13T14:51:00.392050+0000","last_undegraded":"2024-09-13T14:51:00.392050+0000","last_fullsized":"2024-09-13T14:51:00.392050+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:15:53.119741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":207,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T14:50:59.742521+0000","last_change":"2024-09-13T14:47:47.880946+0000","last_active":"2024-09-13T14:50:59.742521+0000","last_peered":"2024-09-13T14:50:59.742521+0000","last_clean":"2024-09-13T14:50:59.742521+0000","last_became_active":"2024-09-13T14:47:47.866534+0000","last_became_peered":"2024-09-13T14:47:47.866534+0000","last_unstale":"2024-09-13T14:50:59.742521+0000","last_undegraded":"2024-09-13T14:50:59.742521+0000","last_fullsized":"2024-09-13T14:50:59.742521+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:01:01.384873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":392,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T14:51:00.391865+0000","last_change":"2024-09-13T14:38:03.526333+0000","last_active":"2024-09-13T14:51:00.391865+0000","last_peered":"2024-09-13T14:51:00.391865+0000","last_clean":"2024-09-13T14:51:00.391865+0000","last_became_active":"2024-09-13T14:38:03.526001+0000","last_became_peered":"2024-09-13T14:38:03.526001+0000","last_unstale":"2024-09-13T14:51:00.391865+0000","last_undegraded":"2024-09-13T14:51:00.391865+0000","last_fullsized":"2024-09-13T14:51:00.391865+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:25:25.181271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":194,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T14:50:59.742625+0000","last_change":"2024-09-13T14:37:49.149582+0000","last_active":"2024-09-13T14:50:59.742625+0000","last_peered":"2024-09-13T14:50:59.742625+0000","last_clean":"2024-09-13T14:50:59.742625+0000","last_became_active":"2024-09-13T14:37:49.148525+0000","last_became_peered":"2024-09-13T14:37:49.148525+0000","last_unstale":"2024-09-13T14:50:59.742625+0000","last_undegraded":"2024-09-13T14:50:59.742625+0000","last_fullsized":"2024-09-13T14:50:59.742625+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:48.655665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":690,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:02.201561+0000","last_change":"2024-09-13T14:46:01.766580+0000","last_active":"2024-09-13T14:51:02.201561+0000","last_peered":"2024-09-13T14:51:02.201561+0000","last_clean":"2024-09-13T14:51:02.201561+0000","last_became_active":"2024-09-13T14:46:01.766257+0000","last_became_peered":"2024-09-13T14:46:01.766257+0000","last_unstale":"2024-09-13T14:51:02.201561+0000","last_undegraded":"2024-09-13T14:51:02.201561+0000","last_fullsized":"2024-09-13T14:51:02.201561+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:01.126109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":655,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:02.201738+0000","last_change":"2024-09-13T14:46:01.769647+0000","last_active":"2024-09-13T14:51:02.201738+0000","last_peered":"2024-09-13T14:51:02.201738+0000","last_clean":"2024-09-13T14:51:02.201738+0000","last_became_active":"2024-09-13T14:46:01.769228+0000","last_became_peered":"2024-09-13T14:46:01.769228+0000","last_unstale":"2024-09-13T14:51:02.201738+0000","last_undegraded":"2024-09-13T14:51:02.201738+0000","last_fullsized":"2024-09-13T14:51:02.201738+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:58.696024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":382,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-09-13T14:51:00.391386+0000","last_change":"2024-09-13T14:47:47.860727+0000","last_active":"2024-09-13T14:51:00.391386+0000","last_peered":"2024-09-13T14:51:00.391386+0000","last_clean":"2024-09-13T14:51:00.391386+0000","last_became_active":"2024-09-13T14:47:47.860377+0000","last_became_peered":"2024-09-13T14:47:47.860377+0000","last_unstale":"2024-09-13T14:51:00.391386+0000","last_undegraded":"2024-09-13T14:51:00.391386+0000","last_fullsized":"2024-09-13T14:51:00.391386+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:27:00.868699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":303,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-13T14:51:02.389906+0000","last_change":"2024-09-13T14:46:01.775237+0000","last_active":"2024-09-13T14:51:02.389906+0000","last_peered":"2024-09-13T14:51:02.389906+0000","last_clean":"2024-09-13T14:51:02.389906+0000","last_became_active":"2024-09-13T14:46:01.765025+0000","last_became_peered":"2024-09-13T14:46:01.765025+0000","last_unstale":"2024-09-13T14:51:02.389906+0000","last_undegraded":"2024-09-13T14:51:02.389906+0000","last_fullsized":"2024-09-13T14:51:02.389906+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:29:16.089042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1746,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":1308491776,"num_objects":19966,"num_object_clones":0,"num_object_copies":59898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19966,"num_write_kb":1277824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3863347200,"data_stored":3863347200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19966,"ondisk_log_size":19966,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":2154,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25132,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":151,"num_bytes_recovered":8388646,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3129,"ondisk_log_size":3129,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234488,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1227616,"kb_used_data":596824,"kb_used_omap":1,"kb_used_meta":630782,"kb_avail":92517536,"statfs":{"total":95995035648,"available":94737956864,"internally_reserved":0,"allocated":611147776,"data_stored":609187031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":645921227},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":239,"seq":1026497183906,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":682328,"kb_used_data":186832,"kb_used_omap":8,"kb_used_meta":495479,"kb_avail":93062824,"statfs":{"total":95995035648,"available":95296331776,"internally_reserved":0,"allocated":191315968,"data_stored":189363346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8453,"internal_metadata":507371259},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757753,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1559572,"kb_used_data":635916,"kb_used_omap":1,"kb_used_meta":923646,"kb_avail":92185580,"statfs":{"total":95995035648,"available":94398033920,"internally_reserved":0,"allocated":651177984,"data_stored":649225770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":945813958},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019397,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":913116,"kb_used_data":174036,"kb_used_omap":1,"kb_used_meta":739070,"kb_avail":92832036,"statfs":{"total":95995035648,"available":95060004864,"internally_reserved":0,"allocated":178212864,"data_stored":176256203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":756808134},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435580,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1522128,"kb_used_data":688008,"kb_used_omap":5,"kb_used_meta":834106,"kb_avail":92223024,"statfs":{"total":95995035648,"available":94436376576,"internally_reserved":0,"allocated":704520192,"data_stored":702571836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":854124670},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":104,"seq":446676599108,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":517908,"kb_used_data":225228,"kb_used_omap":7,"kb_used_meta":292664,"kb_avail":93227244,"statfs":{"total":95995035648,"available":95464697856,"internally_reserved":0,"allocated":230633472,"data_stored":228681546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8010,"internal_metadata":299688118},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":99,"seq":425201762635,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1021128,"kb_used_data":863104,"kb_used_omap":7,"kb_used_meta":158008,"kb_avail":92724024,"statfs":{"total":95995035648,"available":94949400576,"internally_reserved":0,"allocated":883818496,"data_stored":881877444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7759,"internal_metadata":161800625},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926158,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1162704,"kb_used_data":314696,"kb_used_omap":7,"kb_used_meta":847992,"kb_avail":92582448,"statfs":{"total":95995035648,"available":94804426752,"internally_reserved":0,"allocated":322248704,"data_stored":320300318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":868344104},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6860,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":350814208,"data_stored":350814208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":902561792,"data_stored":902561792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":234356736,"data_stored":234356736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":705822720,"data_stored":705822720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":201523200,"data_stored":201523200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":643891200,"data_stored":643891200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":208535552,"data_stored":208535552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":615841792,"data_stored":615841792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:51:22.263 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-13T14:51:22.263 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_4 pg_num 26 2024-09-13T14:51:22.346 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:22.378 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:22.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:22.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:22.480 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:22.506 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:22.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:22.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:22.608 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:22.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:22.652 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:22.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:22.753 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:22.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:22.801 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:22.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:22.902 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:22.933 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:22.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:22.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:23.035 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:23.056 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:23.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:23.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:23.103 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 17 16 42096 42080 154.679 101.438 0.00827065 0.00645966 2024-09-13T14:51:23.158 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:23.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:22 smithi102 ceph-mon[110183]: pgmap v1309: 109 pgs: 109 active+clean; 2.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 159 MiB/s wr, 2.55k op/s 2024-09-13T14:51:23.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:22 smithi102 ceph-mon[115269]: pgmap v1309: 109 pgs: 109 active+clean; 2.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 159 MiB/s wr, 2.55k op/s 2024-09-13T14:51:23.197 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:23.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:23.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:23.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:22 smithi195 ceph-mon[104262]: pgmap v1309: 109 pgs: 109 active+clean; 2.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 159 MiB/s wr, 2.55k op/s 2024-09-13T14:51:23.299 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:23.334 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:23.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:23.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:23.436 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:23.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:23.477 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:23.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:23.578 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:23.616 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:23.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:23.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:23.718 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:23.752 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:23.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:23.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:23.854 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:23.884 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:23.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:23.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:23.986 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:24.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:24.009 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:24.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:24.103 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 18 16 43503 43487 150.971 87.9375 0.00364391 0.00661907 2024-09-13T14:51:24.110 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:24.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:24.147 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:24.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:24.248 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:24.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:24.284 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:24.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:24.385 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:24.420 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:24.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:24.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:24.522 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:24.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:24.557 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:24.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:24.658 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:24.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:24.694 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:24.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:24.795 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:24.837 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:24.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:24.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:24.890 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is set 2024-09-13T14:51:24.939 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:24.983 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:24.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:24.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:25.085 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:25.103 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 19 15 45142 45127 148.42 102.5 0.00587779 0.0067326 2024-09-13T14:51:25.126 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:24 smithi102 ceph-mon[115269]: pgmap v1310: 109 pgs: 109 active+clean; 2.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 146 MiB/s wr, 2.33k op/s 2024-09-13T14:51:25.127 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:24 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2609684851' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:51:25.127 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:24 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:51:25.127 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:24 smithi102 ceph-mon[110183]: pgmap v1310: 109 pgs: 109 active+clean; 2.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 146 MiB/s wr, 2.33k op/s 2024-09-13T14:51:25.127 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:24 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2609684851' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:51:25.127 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:24 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:51:25.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:25.135 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:25.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:25.195 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:51:25.196 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [ 2024-09-13T14:51:25.196 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:25.196 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.36528.0:45272 9.a e379/374 9:5e64cdc4:::benchmark_data_smithi102_348563_object45271:head v 379'2802, mlcod=379'2800)", 2024-09-13T14:51:25.196 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:51:25.185575+0000", 2024-09-13T14:51:25.196 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.007606285, 2024-09-13T14:51:25.196 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0076352199999999999, 2024-09-13T14:51:25.196 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:51:25.196 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:51:25.196 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "started", 2024-09-13T14:51:25.196 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:51:25.196 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:25.196 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:51:25.197 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:25.185575+0000", 2024-09-13T14:51:25.197 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:25.197 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:25.197 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:25.197 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:51:25.197 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:25.185575+0000", 2024-09-13T14:51:25.197 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:25.197 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:25.197 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:25.197 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:51:25.197 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:25.185576+0000", 2024-09-13T14:51:25.197 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.9700000000000002e-07 2024-09-13T14:51:25.197 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:25.198 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:25.198 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:51:25.198 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:25.185603+0000", 2024-09-13T14:51:25.198 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.7245000000000001e-05 2024-09-13T14:51:25.198 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:25.198 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:25.198 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:51:25.198 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:25.185610+0000", 2024-09-13T14:51:25.198 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.1949999999999997e-06 2024-09-13T14:51:25.198 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:25.198 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:25.198 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:51:25.198 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:25.185615+0000", 2024-09-13T14:51:25.199 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.7550000000000002e-06 2024-09-13T14:51:25.199 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:25.199 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:25.199 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:25.199 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:25.185674+0000", 2024-09-13T14:51:25.199 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.9312999999999999e-05 2024-09-13T14:51:25.199 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:25.199 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:25.199 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:51:25.199 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:25.185691+0000", 2024-09-13T14:51:25.199 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.7269999999999999e-05 2024-09-13T14:51:25.199 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:25.199 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:51:25.199 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:25.200 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:25.200 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:25.200 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.36528.0:45270 9.2 e379/374 9:4b546917:::benchmark_data_smithi102_348563_object45269:head v 379'2789, mlcod=379'2787)", 2024-09-13T14:51:25.200 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:51:25.187285+0000", 2024-09-13T14:51:25.200 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.00589618, 2024-09-13T14:51:25.200 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0059877680000000001, 2024-09-13T14:51:25.200 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:51:25.200 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:51:25.200 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "started", 2024-09-13T14:51:25.200 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:51:25.200 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:25.200 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:51:25.200 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:25.187285+0000", 2024-09-13T14:51:25.201 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:25.201 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:25.201 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:25.201 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:51:25.201 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:25.187285+0000", 2024-09-13T14:51:25.201 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:25.201 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:25.201 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:25.201 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:51:25.201 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:25.187286+0000", 2024-09-13T14:51:25.201 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.9899999999999999e-07 2024-09-13T14:51:25.201 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:25.201 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:25.201 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:51:25.202 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:25.187314+0000", 2024-09-13T14:51:25.202 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.8119999999999998e-05 2024-09-13T14:51:25.202 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:25.202 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:25.202 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:51:25.202 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:25.187325+0000", 2024-09-13T14:51:25.202 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1257000000000001e-05 2024-09-13T14:51:25.202 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:25.202 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:25.202 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:51:25.202 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:25.187332+0000", 2024-09-13T14:51:25.202 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.6030000000000001e-06 2024-09-13T14:51:25.202 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:25.202 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:25.203 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:25.203 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:25.189072+0000", 2024-09-13T14:51:25.203 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0017400810000000001 2024-09-13T14:51:25.203 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:25.203 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:25.203 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:51:25.203 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:25.189091+0000", 2024-09-13T14:51:25.203 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.9089999999999998e-05 2024-09-13T14:51:25.203 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:25.203 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:51:25.203 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:25.203 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:25.203 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:25.204 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.36528.0:45274 9.2 e379/374 9:4a955de5:::benchmark_data_smithi102_348563_object45273:head v 379'2790, mlcod=379'2787)", 2024-09-13T14:51:25.204 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:51:25.187352+0000", 2024-09-13T14:51:25.204 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.0058294820000000004, 2024-09-13T14:51:25.204 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0059685500000000004, 2024-09-13T14:51:25.204 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:51:25.204 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:51:25.204 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "started", 2024-09-13T14:51:25.204 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:51:25.204 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:25.204 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:51:25.204 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:25.187352+0000", 2024-09-13T14:51:25.204 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:25.204 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:25.205 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:25.205 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:51:25.205 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:25.187352+0000", 2024-09-13T14:51:25.205 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:25.205 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:25.205 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:25.205 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:51:25.205 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:25.187352+0000", 2024-09-13T14:51:25.205 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.2500000000000001e-07 2024-09-13T14:51:25.205 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:25.205 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:25.205 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:51:25.205 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:25.187373+0000", 2024-09-13T14:51:25.205 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.0890000000000002e-05 2024-09-13T14:51:25.206 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:25.206 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:25.206 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:51:25.206 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:25.187381+0000", 2024-09-13T14:51:25.206 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.1610000000000003e-06 2024-09-13T14:51:25.206 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:25.206 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:25.206 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:51:25.206 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:25.187387+0000", 2024-09-13T14:51:25.206 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.1109999999999999e-06 2024-09-13T14:51:25.206 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:25.206 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:25.206 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:25.206 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:25.190827+0000", 2024-09-13T14:51:25.207 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0034402090000000001 2024-09-13T14:51:25.207 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:25.207 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:25.207 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:51:25.207 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:25.190848+0000", 2024-09-13T14:51:25.207 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.075e-05 2024-09-13T14:51:25.207 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:25.207 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:51:25.207 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:25.207 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:25.207 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:25.207 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.36528.0:45281 9.3 e379/374 9:c0262bac:::benchmark_data_smithi102_348563_object45280:head v 379'2756, mlcod=379'2752)", 2024-09-13T14:51:25.207 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:51:25.189477+0000", 2024-09-13T14:51:25.207 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.0037047360000000001, 2024-09-13T14:51:25.208 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0038922570000000001, 2024-09-13T14:51:25.208 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:51:25.208 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:51:25.208 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "started", 2024-09-13T14:51:25.208 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:51:25.208 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:25.208 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:51:25.208 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:25.189477+0000", 2024-09-13T14:51:25.208 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:25.208 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:25.208 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:25.208 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:51:25.208 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:25.189477+0000", 2024-09-13T14:51:25.209 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:25.209 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:25.209 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:25.209 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:51:25.209 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:25.189478+0000", 2024-09-13T14:51:25.209 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.114e-06 2024-09-13T14:51:25.209 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:25.209 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:25.209 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:51:25.209 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:25.189510+0000", 2024-09-13T14:51:25.209 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.2450999999999998e-05 2024-09-13T14:51:25.209 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:25.209 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:25.209 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:51:25.210 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:25.189521+0000", 2024-09-13T14:51:25.210 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0854e-05 2024-09-13T14:51:25.210 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:25.210 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:25.210 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:51:25.210 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:25.189528+0000", 2024-09-13T14:51:25.210 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.37e-06 2024-09-13T14:51:25.210 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:25.210 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:25.210 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:25.210 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:25.191663+0000", 2024-09-13T14:51:25.210 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0021358890000000002 2024-09-13T14:51:25.210 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:25.211 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:25.211 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:51:25.211 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:25.191730+0000", 2024-09-13T14:51:25.211 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.6509999999999998e-05 2024-09-13T14:51:25.211 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:25.211 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:51:25.211 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:25.211 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:25.211 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:25.211 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.36528.0:45284 9.2 e379/374 9:40b9fbc4:::benchmark_data_smithi102_348563_object45283:head v 379'2791, mlcod=379'2787)", 2024-09-13T14:51:25.212 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:51:25.191509+0000", 2024-09-13T14:51:25.212 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.0016725609999999999, 2024-09-13T14:51:25.212 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.001904745, 2024-09-13T14:51:25.212 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:51:25.212 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:51:25.212 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "started", 2024-09-13T14:51:25.212 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:51:25.212 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:25.212 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:51:25.212 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:25.191509+0000", 2024-09-13T14:51:25.212 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:25.212 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:25.212 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:25.212 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:51:25.213 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:25.191509+0000", 2024-09-13T14:51:25.213 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:25.213 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:25.213 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:25.213 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:51:25.213 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:25.191510+0000", 2024-09-13T14:51:25.213 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.8899999999999998e-07 2024-09-13T14:51:25.213 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:25.213 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:25.213 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:51:25.213 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:25.191533+0000", 2024-09-13T14:51:25.213 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.3195999999999999e-05 2024-09-13T14:51:25.213 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:25.213 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:25.214 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:51:25.214 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:25.191544+0000", 2024-09-13T14:51:25.214 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.076e-05 2024-09-13T14:51:25.214 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:25.214 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:25.214 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:51:25.214 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:25.191550+0000", 2024-09-13T14:51:25.214 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.4230000000000002e-06 2024-09-13T14:51:25.214 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:25.214 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:25.214 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:25.214 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:25.193065+0000", 2024-09-13T14:51:25.214 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0015152060000000001 2024-09-13T14:51:25.215 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:25.215 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:25.215 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:51:25.215 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:25.193084+0000", 2024-09-13T14:51:25.215 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.8774999999999999e-05 2024-09-13T14:51:25.215 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:25.215 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:51:25.215 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:25.215 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:25.215 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-09-13T14:51:25.215 INFO:teuthology.orchestra.run.smithi102.stdout: "num_ops": 5 2024-09-13T14:51:25.215 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:51:25.236 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:25.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:24 smithi195 ceph-mon[104262]: pgmap v1310: 109 pgs: 109 active+clean; 2.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 146 MiB/s wr, 2.33k op/s 2024-09-13T14:51:25.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:24 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2609684851' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:51:25.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:24 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:51:25.272 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:25.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:25.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:25.374 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:25.413 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:25.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:25.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:25.514 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:25.549 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:25.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:25.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:25.651 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:25.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:25.686 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:25.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:25.787 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:25.823 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:25.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:25.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:25.925 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:25.965 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:25.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:25.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:26.067 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:26.103 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:2024-09-13T14:51:26.103564+0000 min lat: 0.00110201 max lat: 0.210662 avg lat: 0.00680362 2024-09-13T14:51:26.104 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:51:26.104 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 20 16 47020 47004 146.864 117.312 0.00555679 0.00680362 2024-09-13T14:51:26.109 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:26.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:26.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:26.155 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:26.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:25 smithi102 ceph-mon[110183]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:51:26.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:25 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:51:26.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:25 smithi102 ceph-mon[110183]: osdmap e379: 8 total, 8 up, 8 in 2024-09-13T14:51:26.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:25 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:51:26.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:25 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:51:26.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:25 smithi102 ceph-mon[115269]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:51:26.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:25 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:51:26.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:25 smithi102 ceph-mon[115269]: osdmap e379: 8 total, 8 up, 8 in 2024-09-13T14:51:26.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:25 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:51:26.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:25 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:51:26.212 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:26.242 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:26.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:26.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:26.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:25 smithi195 ceph-mon[104262]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:51:26.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:25 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:51:26.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:25 smithi195 ceph-mon[104262]: osdmap e379: 8 total, 8 up, 8 in 2024-09-13T14:51:26.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:25 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:51:26.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:25 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:51:26.344 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:26.376 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:26.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:26.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:26.478 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:26.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:26.519 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:26.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:26.620 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:26.655 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:26.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:26.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:26.669 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:51:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:51:26] "GET /metrics HTTP/1.1" 200 36508 "" "Prometheus/2.51.0" 2024-09-13T14:51:26.757 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:26.799 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:26.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:26.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:26.901 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:26.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:26.939 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:26.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:27.040 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:27.075 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:27.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:27.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:27.104 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 21 16 48749 48733 145.016 108.062 0.00374952 0.00689081 2024-09-13T14:51:27.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:26 smithi102 ceph-mon[110183]: pgmap v1312: 109 pgs: 109 active+clean; 2.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 132 MiB/s wr, 2.11k op/s 2024-09-13T14:51:27.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:26 smithi102 ceph-mon[115269]: pgmap v1312: 109 pgs: 109 active+clean; 2.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 132 MiB/s wr, 2.11k op/s 2024-09-13T14:51:27.177 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:27.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:27.213 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:27.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:27.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:26 smithi195 ceph-mon[104262]: pgmap v1312: 109 pgs: 109 active+clean; 2.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 132 MiB/s wr, 2.11k op/s 2024-09-13T14:51:27.314 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:27.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:27.347 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:27.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:27.448 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:27.484 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:27.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:27.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:27.586 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:27.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:27.624 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:27.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:27.628 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T14:51:27.725 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:27.758 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:27.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:27.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:27.859 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:27.881 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:27.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:27.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:27.954 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:51:27.983 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:28.039 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:28.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:28.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:28.104 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 22 16 50326 50310 142.904 98.5625 0.0146425 0.00699208 2024-09-13T14:51:28.141 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:28.184 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:28.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:28.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:28.286 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:28.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:28.321 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:28.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:28.422 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:28.488 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:28.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:28.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:28.590 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:28.623 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:28.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:28.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:28.725 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:28.763 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:28.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:28.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:28.803 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:51:28.804 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:51:28.848 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:51:28.849 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:51:28.865 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:28.883 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:51:28.883 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:51:28.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:28.894 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:28.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:28.917 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:51:28.917 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:51:28.950 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:51:28.950 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:51:28.989 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:51:28.989 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:51:28.990 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:28 smithi102 ceph-mon[115269]: pgmap v1313: 109 pgs: 109 active+clean; 2.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-09-13T14:51:28.990 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:28 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1799676889' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:28.990 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:28 smithi102 ceph-mon[110183]: pgmap v1313: 109 pgs: 109 active+clean; 2.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-09-13T14:51:28.990 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:28 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1799676889' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:28.996 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:29.021 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:51:29.021 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:51:29.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:29.034 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:29.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:29.056 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:51:29.056 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:51:29.104 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 23 16 51890 51874 140.941 97.75 0.0114283 0.00709013 2024-09-13T14:51:29.135 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:29.169 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:29.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:29.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:29.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:28 smithi195 ceph-mon[104262]: pgmap v1313: 109 pgs: 109 active+clean; 2.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-09-13T14:51:29.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:28 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1799676889' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:29.271 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:29.309 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:29.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:29.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:29.410 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:29.438 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:29.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:29.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:29.456 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:29.540 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:29.582 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:29.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:29.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:29.684 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:29.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:29.718 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:29.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:29.820 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:29.852 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:29.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:29.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:29.954 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:29.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:29.990 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:29.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:30.091 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:30.104 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 24 16 53623 53607 139.581 108.312 0.00782447 0.00715926 2024-09-13T14:51:30.129 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:30.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:30.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:30.231 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:30.266 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:30.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:30.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:30.368 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:30.405 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:30.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:30.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:30.507 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:30.541 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:30.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:30.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:30.643 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:30.671 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:30.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:30.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:30.773 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:30.811 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:30.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:30.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:30.913 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:30.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:30.949 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:30.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:31.050 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:31.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:31.088 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:31.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:31.104 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 25 16 55360 55344 138.34 108.562 0.00700304 0.00722414 2024-09-13T14:51:31.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:30 smithi102 ceph-mon[110183]: pgmap v1314: 109 pgs: 109 active+clean; 3.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-09-13T14:51:31.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:30 smithi102 ceph-mon[115269]: pgmap v1314: 109 pgs: 109 active+clean; 3.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-09-13T14:51:31.190 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:31.222 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:31.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:31.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:31.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:30 smithi195 ceph-mon[104262]: pgmap v1314: 109 pgs: 109 active+clean; 3.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-09-13T14:51:31.324 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:31.358 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:31.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:31.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:31.460 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:31.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:31.493 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:31.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:31.594 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:31.620 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:31.631 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:31.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:31.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:31.733 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:31.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:31.781 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:31.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:31.882 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:31.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:31.914 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:31.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:32.015 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:32.042 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:32.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:32.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:32.074 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:51:32.104 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 26 16 57010 56994 136.985 103.125 0.011452 0.00729442 2024-09-13T14:51:32.145 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:32.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:32.174 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:32.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:32.275 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:32.304 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:32.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:32.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:32.406 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:32.440 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:32.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:32.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:32.542 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:32.573 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:32.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:32.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:32.675 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:32.696 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:32.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:32.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:32.798 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:32.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:32.830 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:32.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:32.925 INFO:teuthology.orchestra.run.smithi102.stderr:set pool 9 pg_num to 26 2024-09-13T14:51:32.932 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:32.970 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:32.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:32.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:33.072 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:33.101 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:33.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:33.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:33.104 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 27 16 58288 58272 134.87 79.875 0.00607947 0.00740989 2024-09-13T14:51:33.202 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:33.235 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:33.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:33.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:33.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:32 smithi195 ceph-mon[104262]: pgmap v1315: 109 pgs: 109 active+clean; 3.2 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 110 MiB/s wr, 1.75k op/s 2024-09-13T14:51:33.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:32 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2766838871' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"} : dispatch 2024-09-13T14:51:33.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:32 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"} : dispatch 2024-09-13T14:51:33.337 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:33.360 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:33.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:33.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:33.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:32 smithi102 ceph-mon[110183]: pgmap v1315: 109 pgs: 109 active+clean; 3.2 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 110 MiB/s wr, 1.75k op/s 2024-09-13T14:51:33.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:32 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2766838871' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"} : dispatch 2024-09-13T14:51:33.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:32 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"} : dispatch 2024-09-13T14:51:33.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:32 smithi102 ceph-mon[115269]: pgmap v1315: 109 pgs: 109 active+clean; 3.2 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 110 MiB/s wr, 1.75k op/s 2024-09-13T14:51:33.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:32 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2766838871' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"} : dispatch 2024-09-13T14:51:33.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:32 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"} : dispatch 2024-09-13T14:51:33.463 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:33.502 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:33.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:33.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:33.604 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:33.625 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:33.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:33.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:33.727 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:33.757 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:33.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:33.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:33.859 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:33.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:33.892 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:33.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:33.993 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:34.059 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:34.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:34.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:34.104 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 28 16 59445 59429 132.635 72.3125 0.163088 0.00750142 2024-09-13T14:51:34.161 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:34.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:34.184 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:34.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:34.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:33 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-09-13T14:51:34.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:33 smithi195 ceph-mon[104262]: osdmap e380: 8 total, 8 up, 8 in 2024-09-13T14:51:34.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:33 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-13T14:51:34.284 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:34.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:34.306 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:34.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:34.407 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:34.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:33 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-09-13T14:51:34.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:33 smithi102 ceph-mon[110183]: osdmap e380: 8 total, 8 up, 8 in 2024-09-13T14:51:34.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:33 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-13T14:51:34.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:33 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-09-13T14:51:34.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:33 smithi102 ceph-mon[115269]: osdmap e380: 8 total, 8 up, 8 in 2024-09-13T14:51:34.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:33 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-13T14:51:34.430 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:34.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:34.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:34.532 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:34.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:34.554 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:34.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:34.655 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:34.677 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:34.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:34.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:34.779 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:34.804 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:34.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:34.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:34.905 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:34.928 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:34.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:34.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:35.030 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:35.066 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:35.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:35.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:35.104 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 29 16 59675 59659 128.557 14.375 0.00291816 0.00777381 2024-09-13T14:51:35.168 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:35.204 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:35.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:35.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:35.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:34 smithi195 ceph-mon[104262]: pgmap v1317: 109 pgs: 109 active+clean; 3.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 110 MiB/s wr, 1.76k op/s 2024-09-13T14:51:35.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:34 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-13T14:51:35.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:34 smithi195 ceph-mon[104262]: osdmap e381: 8 total, 8 up, 8 in 2024-09-13T14:51:35.306 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:35.340 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:35.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:35.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:35.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:34 smithi102 ceph-mon[110183]: pgmap v1317: 109 pgs: 109 active+clean; 3.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 110 MiB/s wr, 1.76k op/s 2024-09-13T14:51:35.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:34 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-13T14:51:35.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:34 smithi102 ceph-mon[110183]: osdmap e381: 8 total, 8 up, 8 in 2024-09-13T14:51:35.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:34 smithi102 ceph-mon[115269]: pgmap v1317: 109 pgs: 109 active+clean; 3.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 110 MiB/s wr, 1.76k op/s 2024-09-13T14:51:35.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:34 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-13T14:51:35.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:34 smithi102 ceph-mon[115269]: osdmap e381: 8 total, 8 up, 8 in 2024-09-13T14:51:35.442 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:35.478 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:35.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:35.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:35.580 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:35.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:35.616 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:35.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:35.718 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:35.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:35.752 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:35.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:35.852 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:35.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:35.890 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:35.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:35.991 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:36.019 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:35 smithi102 ceph-mon[110183]: osdmap e382: 8 total, 8 up, 8 in 2024-09-13T14:51:36.019 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:35 smithi102 ceph-mon[110183]: 9.0 scrub starts 2024-09-13T14:51:36.019 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:35 smithi102 ceph-mon[110183]: 9.9 scrub starts 2024-09-13T14:51:36.020 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:35 smithi102 ceph-mon[115269]: osdmap e382: 8 total, 8 up, 8 in 2024-09-13T14:51:36.020 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:35 smithi102 ceph-mon[115269]: 9.0 scrub starts 2024-09-13T14:51:36.020 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:35 smithi102 ceph-mon[115269]: 9.9 scrub starts 2024-09-13T14:51:36.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:36.027 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:36.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:36.104 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 30 16 61751 61735 128.597 129.75 0.00481958 0.00777193 2024-09-13T14:51:36.128 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:36.166 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:36.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:36.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:36.250 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:35 smithi195 ceph-mon[104262]: osdmap e382: 8 total, 8 up, 8 in 2024-09-13T14:51:36.250 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:35 smithi195 ceph-mon[104262]: 9.0 scrub starts 2024-09-13T14:51:36.250 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:35 smithi195 ceph-mon[104262]: 9.9 scrub starts 2024-09-13T14:51:36.267 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:36.302 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:36.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:36.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:36.404 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:36.439 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:36.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:36.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:36.541 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:36.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:36.572 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:36.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:36.577 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:36.669 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:51:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:51:36] "GET /metrics HTTP/1.1" 200 36508 "" "Prometheus/2.51.0" 2024-09-13T14:51:36.673 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:36.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:36.713 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:36.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:36.815 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:36.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:36.852 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:36.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:36.953 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:36.981 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is set 2024-09-13T14:51:36.982 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:36 smithi102 ceph-mon[110183]: pgmap v1320: 119 pgs: 2 peering, 10 unknown, 107 active+clean; 3.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-09-13T14:51:36.982 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:36 smithi102 ceph-mon[110183]: 9.1 scrub starts 2024-09-13T14:51:36.982 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:36 smithi102 ceph-mon[110183]: 9.4 scrub starts 2024-09-13T14:51:36.982 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:36 smithi102 ceph-mon[110183]: 9.5 scrub starts 2024-09-13T14:51:36.982 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:36 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1586291306' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:51:36.982 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:36 smithi102 ceph-mon[115269]: pgmap v1320: 119 pgs: 2 peering, 10 unknown, 107 active+clean; 3.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-09-13T14:51:36.983 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:36 smithi102 ceph-mon[115269]: 9.1 scrub starts 2024-09-13T14:51:36.983 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:36 smithi102 ceph-mon[115269]: 9.4 scrub starts 2024-09-13T14:51:36.983 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:36 smithi102 ceph-mon[115269]: 9.5 scrub starts 2024-09-13T14:51:36.983 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:36 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1586291306' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:51:36.988 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:36.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:36.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:37.090 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:37.105 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 31 16 63672 63656 128.321 120.062 0.00805752 0.00778883 2024-09-13T14:51:37.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:37.135 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:37.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:37.236 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:37.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:36 smithi195 ceph-mon[104262]: pgmap v1320: 119 pgs: 2 peering, 10 unknown, 107 active+clean; 3.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-09-13T14:51:37.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:36 smithi195 ceph-mon[104262]: 9.1 scrub starts 2024-09-13T14:51:37.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:36 smithi195 ceph-mon[104262]: 9.4 scrub starts 2024-09-13T14:51:37.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:36 smithi195 ceph-mon[104262]: 9.5 scrub starts 2024-09-13T14:51:37.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:36 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1586291306' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:51:37.265 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:37.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:37.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:37.367 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:37.401 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:37.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:37.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:37.503 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:37.539 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:37.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:37.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:37.641 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:37.680 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:37.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:37.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:37.783 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:37.818 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:37.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:37.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:37.920 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:37.954 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:37.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:37.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:38.055 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:38.093 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:38.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:38.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:38.105 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 32 15 65641 65626 128.158 123.125 0.00822925 0.00779834 2024-09-13T14:51:38.195 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:38.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:38.232 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:38.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:38.247 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:37 smithi102 ceph-mon[110183]: 9.2 scrub starts 2024-09-13T14:51:38.247 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:37 smithi102 ceph-mon[110183]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:51:38.248 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:37 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1586291306' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:51:38.248 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:37 smithi102 ceph-mon[110183]: osdmap e383: 8 total, 8 up, 8 in 2024-09-13T14:51:38.248 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:37 smithi102 ceph-mon[110183]: 9.14 scrub starts 2024-09-13T14:51:38.248 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:37 smithi102 ceph-mon[110183]: 9.19 scrub starts 2024-09-13T14:51:38.248 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:37 smithi102 ceph-mon[115269]: 9.2 scrub starts 2024-09-13T14:51:38.248 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:37 smithi102 ceph-mon[115269]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:51:38.248 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:37 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1586291306' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:51:38.248 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:37 smithi102 ceph-mon[115269]: osdmap e383: 8 total, 8 up, 8 in 2024-09-13T14:51:38.248 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:37 smithi102 ceph-mon[115269]: 9.14 scrub starts 2024-09-13T14:51:38.248 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:37 smithi102 ceph-mon[115269]: 9.19 scrub starts 2024-09-13T14:51:38.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:37 smithi195 ceph-mon[104262]: 9.2 scrub starts 2024-09-13T14:51:38.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:37 smithi195 ceph-mon[104262]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:51:38.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:37 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1586291306' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:51:38.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:37 smithi195 ceph-mon[104262]: osdmap e383: 8 total, 8 up, 8 in 2024-09-13T14:51:38.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:37 smithi195 ceph-mon[104262]: 9.14 scrub starts 2024-09-13T14:51:38.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:37 smithi195 ceph-mon[104262]: 9.19 scrub starts 2024-09-13T14:51:38.334 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:38.356 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:51:38.357 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:51:38.370 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:38.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:38.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:38.472 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:38.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:38.511 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:38.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:38.612 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:38.644 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:38.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:38.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:38.749 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:38.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:38.781 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:38.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:38.882 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:38.917 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:38.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:38.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:39.019 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:39.054 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:39.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:39.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:39.105 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 33 16 67632 67616 128.043 124.375 0.00518199 0.00780556 2024-09-13T14:51:39.156 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:39.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:38 smithi102 ceph-mon[110183]: pgmap v1322: 119 pgs: 2 peering, 7 unknown, 110 active+clean; 3.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 93 MiB/s wr, 1.48k op/s; 13 KiB/s, 0 objects/s recovering 2024-09-13T14:51:39.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:38 smithi102 ceph-mon[110183]: 9.8 scrub starts 2024-09-13T14:51:39.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:38 smithi102 ceph-mon[110183]: 9.10 scrub starts 2024-09-13T14:51:39.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:38 smithi102 ceph-mon[110183]: 9.15 scrub starts 2024-09-13T14:51:39.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:38 smithi102 ceph-mon[115269]: pgmap v1322: 119 pgs: 2 peering, 7 unknown, 110 active+clean; 3.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 93 MiB/s wr, 1.48k op/s; 13 KiB/s, 0 objects/s recovering 2024-09-13T14:51:39.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:38 smithi102 ceph-mon[115269]: 9.8 scrub starts 2024-09-13T14:51:39.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:38 smithi102 ceph-mon[115269]: 9.10 scrub starts 2024-09-13T14:51:39.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:38 smithi102 ceph-mon[115269]: 9.15 scrub starts 2024-09-13T14:51:39.191 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:39.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:39.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:39.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:38 smithi195 ceph-mon[104262]: pgmap v1322: 119 pgs: 2 peering, 7 unknown, 110 active+clean; 3.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 93 MiB/s wr, 1.48k op/s; 13 KiB/s, 0 objects/s recovering 2024-09-13T14:51:39.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:38 smithi195 ceph-mon[104262]: 9.8 scrub starts 2024-09-13T14:51:39.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:38 smithi195 ceph-mon[104262]: 9.10 scrub starts 2024-09-13T14:51:39.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:38 smithi195 ceph-mon[104262]: 9.15 scrub starts 2024-09-13T14:51:39.294 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:39.305 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T14:51:39.328 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:39.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:39.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:39.430 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:39.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:39.484 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:39.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:39.585 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:39.617 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:39.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:39.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:39.702 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:51:39.719 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:39.753 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:39.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:39.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:39.855 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:39.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:39.896 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:39.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:39.997 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:40.037 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:40.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:40.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:40.105 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 34 16 69460 69444 127.637 114.25 0.00239186 0.00782912 2024-09-13T14:51:40.139 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:40.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:40.174 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:40.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:40.275 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:40.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:40.298 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:40.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:40.365 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:40 smithi102 ceph-mon[110183]: 9.11 scrub starts 2024-09-13T14:51:40.366 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:40 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "17"} : dispatch 2024-09-13T14:51:40.366 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:40 smithi102 ceph-mon[115269]: 9.11 scrub starts 2024-09-13T14:51:40.366 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:40 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "17"} : dispatch 2024-09-13T14:51:40.399 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:40.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:40.423 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:40.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:40.440 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:40.492 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 6, 7, 5, 2, 1] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T14:51:40.492 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:51:40.492 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-09-13T14:51:40.492 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T14:51:40.493 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T14:51:40.493 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-09-13T14:51:40.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:40 smithi195 ceph-mon[104262]: 9.11 scrub starts 2024-09-13T14:51:40.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:40 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "17"} : dispatch 2024-09-13T14:51:40.524 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:40.546 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:40.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:40.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:40.648 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:40.669 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:40.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:40.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:40.771 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:40.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:40.794 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:40.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:40.814 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:51:40.897 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:40.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:40.922 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:40.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:41.022 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:41.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:41.044 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:41.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:41.105 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 35 16 69679 69663 124.382 13.6875 0.00298219 0.00781992 2024-09-13T14:51:41.145 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:41.167 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:41.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:41.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:41.236 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:41 smithi102 ceph-mon[110183]: pgmap v1323: 119 pgs: 119 active+clean; 3.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 83 MiB/s wr, 1.33k op/s; 21 KiB/s, 0 objects/s recovering 2024-09-13T14:51:41.237 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:41 smithi102 ceph-mon[110183]: 9.12 scrub starts 2024-09-13T14:51:41.237 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:41 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "17"}]': finished 2024-09-13T14:51:41.237 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:41 smithi102 ceph-mon[110183]: osdmap e384: 8 total, 8 up, 8 in 2024-09-13T14:51:41.237 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:41 smithi102 ceph-mon[110183]: 9.15 scrub starts 2024-09-13T14:51:41.237 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:41 smithi102 ceph-mon[110183]: 9.14 scrub starts 2024-09-13T14:51:41.238 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:41 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:51:41.238 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:41 smithi102 ceph-mon[110183]: osdmap e385: 8 total, 8 up, 8 in 2024-09-13T14:51:41.238 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:41 smithi102 ceph-mon[115269]: pgmap v1323: 119 pgs: 119 active+clean; 3.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 83 MiB/s wr, 1.33k op/s; 21 KiB/s, 0 objects/s recovering 2024-09-13T14:51:41.238 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:41 smithi102 ceph-mon[115269]: 9.12 scrub starts 2024-09-13T14:51:41.239 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:41 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "17"}]': finished 2024-09-13T14:51:41.239 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:41 smithi102 ceph-mon[115269]: osdmap e384: 8 total, 8 up, 8 in 2024-09-13T14:51:41.239 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:41 smithi102 ceph-mon[115269]: 9.15 scrub starts 2024-09-13T14:51:41.239 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:41 smithi102 ceph-mon[115269]: 9.14 scrub starts 2024-09-13T14:51:41.239 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:41 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:51:41.239 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:41 smithi102 ceph-mon[115269]: osdmap e385: 8 total, 8 up, 8 in 2024-09-13T14:51:41.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:41 smithi195 ceph-mon[104262]: pgmap v1323: 119 pgs: 119 active+clean; 3.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 83 MiB/s wr, 1.33k op/s; 21 KiB/s, 0 objects/s recovering 2024-09-13T14:51:41.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:41 smithi195 ceph-mon[104262]: 9.12 scrub starts 2024-09-13T14:51:41.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:41 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "17"}]': finished 2024-09-13T14:51:41.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:41 smithi195 ceph-mon[104262]: osdmap e384: 8 total, 8 up, 8 in 2024-09-13T14:51:41.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:41 smithi195 ceph-mon[104262]: 9.15 scrub starts 2024-09-13T14:51:41.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:41 smithi195 ceph-mon[104262]: 9.14 scrub starts 2024-09-13T14:51:41.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:41 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:51:41.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:41 smithi195 ceph-mon[104262]: osdmap e385: 8 total, 8 up, 8 in 2024-09-13T14:51:41.269 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:41.292 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:41.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:41.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:41.307 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:51:41.307 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:51:41.307 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:51:41.307 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:51:41.307 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:51:41.307 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:51:41.307 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:51:41.307 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:51:41.308 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:51:41.308 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:51:41.308 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:51:41.308 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:51:41.308 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:51:41.308 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:51:41.317 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:51:41.318 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:51:41.393 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:41.415 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:41.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:41.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:41.517 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:41.538 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:41.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:41.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:41.640 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:41.662 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:41.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:41.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:41.764 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:41.785 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:41.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:41.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:41.887 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:41.908 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:41.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:41.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:42.010 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:42.033 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:42.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:42.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:42.105 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 36 16 69679 69663 120.927 0 - 0.00781992 2024-09-13T14:51:42.114 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:42 smithi195 ceph-mon[104262]: 9.11 scrub starts 2024-09-13T14:51:42.114 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:42 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/268387278' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:42.114 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:42 smithi195 ceph-mon[104262]: 9.4 scrub starts 2024-09-13T14:51:42.115 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:42 smithi195 ceph-mon[104262]: 9.5 scrub starts 2024-09-13T14:51:42.115 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:42 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "18"} : dispatch 2024-09-13T14:51:42.135 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:42.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:42.175 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:42.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:42.275 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:42.305 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:42.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:42.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:42.406 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:42.407 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:42 smithi102 ceph-mon[110183]: 9.11 scrub starts 2024-09-13T14:51:42.408 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:42 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/268387278' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:42.408 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:42 smithi102 ceph-mon[110183]: 9.4 scrub starts 2024-09-13T14:51:42.408 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:42 smithi102 ceph-mon[110183]: 9.5 scrub starts 2024-09-13T14:51:42.408 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:42 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "18"} : dispatch 2024-09-13T14:51:42.408 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:42 smithi102 ceph-mon[115269]: 9.11 scrub starts 2024-09-13T14:51:42.408 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:42 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/268387278' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:42.408 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:42 smithi102 ceph-mon[115269]: 9.4 scrub starts 2024-09-13T14:51:42.408 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:42 smithi102 ceph-mon[115269]: 9.5 scrub starts 2024-09-13T14:51:42.408 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:42 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "18"} : dispatch 2024-09-13T14:51:42.435 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:42.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:42.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:42.537 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:42.560 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:42.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:42.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:42.662 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:42.683 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:42.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:42.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:42.785 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:42.807 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:42.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:42.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:42.909 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:42.930 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:42.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:42.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:43.032 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:43.041 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:43.075 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:43.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:43.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:43.105 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 37 16 70287 70271 118.685 19 0.0216348 0.00842078 2024-09-13T14:51:43.177 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:43.211 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:43.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:43.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:43.313 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:43.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:43.350 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:43.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:43.352 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:43 smithi195 ceph-mon[104262]: pgmap v1326: 119 pgs: 119 active+clean; 4.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 108 MiB/s wr, 1.72k op/s; 21 KiB/s, 0 objects/s recovering 2024-09-13T14:51:43.352 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:43 smithi195 ceph-mon[104262]: 9.1 scrub starts 2024-09-13T14:51:43.352 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:43 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "18"}]': finished 2024-09-13T14:51:43.352 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:43 smithi195 ceph-mon[104262]: osdmap e386: 8 total, 8 up, 8 in 2024-09-13T14:51:43.352 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:43 smithi195 ceph-mon[104262]: 9.14 scrub starts 2024-09-13T14:51:43.352 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:43 smithi195 ceph-mon[104262]: 9.9 scrub starts 2024-09-13T14:51:43.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:43 smithi102 ceph-mon[110183]: pgmap v1326: 119 pgs: 119 active+clean; 4.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 108 MiB/s wr, 1.72k op/s; 21 KiB/s, 0 objects/s recovering 2024-09-13T14:51:43.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:43 smithi102 ceph-mon[110183]: 9.1 scrub starts 2024-09-13T14:51:43.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:43 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "18"}]': finished 2024-09-13T14:51:43.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:43 smithi102 ceph-mon[110183]: osdmap e386: 8 total, 8 up, 8 in 2024-09-13T14:51:43.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:43 smithi102 ceph-mon[110183]: 9.14 scrub starts 2024-09-13T14:51:43.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:43 smithi102 ceph-mon[110183]: 9.9 scrub starts 2024-09-13T14:51:43.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:43 smithi102 ceph-mon[115269]: pgmap v1326: 119 pgs: 119 active+clean; 4.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 108 MiB/s wr, 1.72k op/s; 21 KiB/s, 0 objects/s recovering 2024-09-13T14:51:43.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:43 smithi102 ceph-mon[115269]: 9.1 scrub starts 2024-09-13T14:51:43.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:43 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "18"}]': finished 2024-09-13T14:51:43.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:43 smithi102 ceph-mon[115269]: osdmap e386: 8 total, 8 up, 8 in 2024-09-13T14:51:43.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:43 smithi102 ceph-mon[115269]: 9.14 scrub starts 2024-09-13T14:51:43.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:43 smithi102 ceph-mon[115269]: 9.9 scrub starts 2024-09-13T14:51:43.451 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:43.495 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:43.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:43.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:43.596 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:43.641 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:43.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:43.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:43.680 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:51:43.680 INFO:teuthology.orchestra.run.smithi195.stdout: "success": "" 2024-09-13T14:51:43.680 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:51:43.742 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:43.780 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:43.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:43.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:43.881 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:43.918 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:43.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:43.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:44.020 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:44.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:44.055 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:44.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:44.105 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 38 16 71893 71877 118.203 100.375 0.0034943 0.00845458 2024-09-13T14:51:44.156 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:44.160 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:44 smithi102 ceph-mon[110183]: 9.8 scrub starts 2024-09-13T14:51:44.161 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:44 smithi102 ceph-mon[110183]: osdmap e387: 8 total, 8 up, 8 in 2024-09-13T14:51:44.161 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:44 smithi102 ceph-mon[110183]: 9.19 scrub starts 2024-09-13T14:51:44.161 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:44 smithi102 ceph-mon[110183]: 9.4 scrub starts 2024-09-13T14:51:44.161 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:44 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "19"} : dispatch 2024-09-13T14:51:44.161 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:44 smithi102 ceph-mon[115269]: 9.8 scrub starts 2024-09-13T14:51:44.161 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:44 smithi102 ceph-mon[115269]: osdmap e387: 8 total, 8 up, 8 in 2024-09-13T14:51:44.161 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:44 smithi102 ceph-mon[115269]: 9.19 scrub starts 2024-09-13T14:51:44.161 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:44 smithi102 ceph-mon[115269]: 9.4 scrub starts 2024-09-13T14:51:44.161 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:44 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "19"} : dispatch 2024-09-13T14:51:44.187 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:44.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:44.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:44.289 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:44.314 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:44.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:44.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:44.416 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:44.440 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:51:44.441 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:44.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:44.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:44.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:44 smithi195 ceph-mon[104262]: 9.8 scrub starts 2024-09-13T14:51:44.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:44 smithi195 ceph-mon[104262]: osdmap e387: 8 total, 8 up, 8 in 2024-09-13T14:51:44.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:44 smithi195 ceph-mon[104262]: 9.19 scrub starts 2024-09-13T14:51:44.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:44 smithi195 ceph-mon[104262]: 9.4 scrub starts 2024-09-13T14:51:44.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:44 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "19"} : dispatch 2024-09-13T14:51:44.543 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:44.565 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:44.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:44.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:44.666 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:44.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:44.689 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:44.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:44.792 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:44.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:44.819 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:44.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:44.920 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:44.941 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:44.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:44.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:45.043 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:45.051 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is unset 2024-09-13T14:51:45.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:45.067 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:45.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:45.105 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 39 16 72296 72280 115.818 25.1875 0.00234068 0.00843619 2024-09-13T14:51:45.168 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:45.190 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:45.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:45.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:45.291 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:45.313 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:45.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:45.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:45.395 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:45 smithi102 ceph-mon[110183]: pgmap v1329: 119 pgs: 119 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-13T14:51:45.396 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:45 smithi102 ceph-mon[110183]: 9.18 scrub starts 2024-09-13T14:51:45.396 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:45 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "19"}]': finished 2024-09-13T14:51:45.396 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:45 smithi102 ceph-mon[110183]: osdmap e388: 8 total, 8 up, 8 in 2024-09-13T14:51:45.396 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:45 smithi102 ceph-mon[110183]: 9.14 scrub starts 2024-09-13T14:51:45.396 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:45 smithi102 ceph-mon[110183]: 9.9 scrub starts 2024-09-13T14:51:45.396 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:45 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1142142643' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:51:45.397 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:45 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:51:45.397 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:45 smithi102 ceph-mon[115269]: pgmap v1329: 119 pgs: 119 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-13T14:51:45.397 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:45 smithi102 ceph-mon[115269]: 9.18 scrub starts 2024-09-13T14:51:45.397 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:45 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "19"}]': finished 2024-09-13T14:51:45.397 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:45 smithi102 ceph-mon[115269]: osdmap e388: 8 total, 8 up, 8 in 2024-09-13T14:51:45.397 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:45 smithi102 ceph-mon[115269]: 9.14 scrub starts 2024-09-13T14:51:45.397 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:45 smithi102 ceph-mon[115269]: 9.9 scrub starts 2024-09-13T14:51:45.397 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:45 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1142142643' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:51:45.397 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:45 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:51:45.415 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:45.443 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:45.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:45.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:45.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:45 smithi195 ceph-mon[104262]: pgmap v1329: 119 pgs: 119 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-13T14:51:45.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:45 smithi195 ceph-mon[104262]: 9.18 scrub starts 2024-09-13T14:51:45.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:45 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "19"}]': finished 2024-09-13T14:51:45.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:45 smithi195 ceph-mon[104262]: osdmap e388: 8 total, 8 up, 8 in 2024-09-13T14:51:45.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:45 smithi195 ceph-mon[104262]: 9.14 scrub starts 2024-09-13T14:51:45.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:45 smithi195 ceph-mon[104262]: 9.9 scrub starts 2024-09-13T14:51:45.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:45 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1142142643' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:51:45.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:45 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:51:45.545 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:45.567 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:45.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:45.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:45.668 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:45.691 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:45.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:45.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:45.793 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:45.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:45.818 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:45.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:45.919 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:45.941 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:45.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:45.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:46.043 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:46.067 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:46.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:46.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:46.105 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:2024-09-13T14:51:46.105453+0000 min lat: 0.00110201 max lat: 2.08078 avg lat: 0.00843619 2024-09-13T14:51:46.105 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:51:46.106 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 40 16 72296 72280 112.923 0 - 0.00843619 2024-09-13T14:51:46.161 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:46 smithi102 ceph-mon[110183]: 9.2 scrub starts 2024-09-13T14:51:46.161 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:46 smithi102 ceph-mon[110183]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:51:46.161 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:46 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:51:46.161 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:46 smithi102 ceph-mon[110183]: osdmap e389: 8 total, 8 up, 8 in 2024-09-13T14:51:46.161 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:46 smithi102 ceph-mon[110183]: 9.4 scrub starts 2024-09-13T14:51:46.161 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:46 smithi102 ceph-mon[110183]: 9.19 scrub starts 2024-09-13T14:51:46.161 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:46 smithi102 ceph-mon[110183]: 9.4 scrub ok 2024-09-13T14:51:46.161 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "20"} : dispatch 2024-09-13T14:51:46.162 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:46 smithi102 ceph-mon[115269]: 9.2 scrub starts 2024-09-13T14:51:46.162 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:46 smithi102 ceph-mon[115269]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:51:46.162 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:46 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:51:46.162 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:46 smithi102 ceph-mon[115269]: osdmap e389: 8 total, 8 up, 8 in 2024-09-13T14:51:46.162 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:46 smithi102 ceph-mon[115269]: 9.4 scrub starts 2024-09-13T14:51:46.162 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:46 smithi102 ceph-mon[115269]: 9.19 scrub starts 2024-09-13T14:51:46.162 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:46 smithi102 ceph-mon[115269]: 9.4 scrub ok 2024-09-13T14:51:46.162 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "20"} : dispatch 2024-09-13T14:51:46.169 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:46.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:46.200 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:46.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:46.301 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:46.331 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:46.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:46.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:46.433 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:46.460 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:46.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:46.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:46.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:46 smithi195 ceph-mon[104262]: 9.2 scrub starts 2024-09-13T14:51:46.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:46 smithi195 ceph-mon[104262]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:51:46.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:46 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:51:46.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:46 smithi195 ceph-mon[104262]: osdmap e389: 8 total, 8 up, 8 in 2024-09-13T14:51:46.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:46 smithi195 ceph-mon[104262]: 9.4 scrub starts 2024-09-13T14:51:46.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:46 smithi195 ceph-mon[104262]: 9.19 scrub starts 2024-09-13T14:51:46.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:46 smithi195 ceph-mon[104262]: 9.4 scrub ok 2024-09-13T14:51:46.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "20"} : dispatch 2024-09-13T14:51:46.563 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:46.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:46.600 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:46.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:46.701 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:46.725 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:46.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:46.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:46.798 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:51:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:51:46] "GET /metrics HTTP/1.1" 200 36518 "" "Prometheus/2.51.0" 2024-09-13T14:51:46.808 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:51:46.809 INFO:teuthology.orchestra.run.smithi102.stdout: "size": 20, 2024-09-13T14:51:46.809 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 600, 2024-09-13T14:51:46.809 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [ 2024-09-13T14:51:46.809 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.809 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:252346 8.a 8:58700e01:::benchmark_data_smithi102_313275_object121759:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e367)", 2024-09-13T14:51:46.809 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:50:19.052204+0000", 2024-09-13T14:51:46.809 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 87.753272804000005, 2024-09-13T14:51:46.809 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.196151849, 2024-09-13T14:51:46.809 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:51:46.809 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:51:46.809 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "reached pg", 2024-09-13T14:51:46.810 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:51:46.810 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:51:46.810 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:51:46.810 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 252346 2024-09-13T14:51:46.810 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.810 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:51:46.810 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.810 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:51:46.810 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.052204+0000", 2024-09-13T14:51:46.810 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:46.810 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.810 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.810 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:51:46.810 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.052204+0000", 2024-09-13T14:51:46.811 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:46.811 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.811 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.811 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:51:46.811 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.052206+0000", 2024-09-13T14:51:46.811 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.736e-06 2024-09-13T14:51:46.811 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.811 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.811 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:51:46.811 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.052213+0000", 2024-09-13T14:51:46.811 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.0330000000000003e-06 2024-09-13T14:51:46.811 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.811 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.812 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:51:46.812 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.052233+0000", 2024-09-13T14:51:46.812 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.9701e-05 2024-09-13T14:51:46.812 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.812 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.812 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:51:46.812 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.052242+0000", 2024-09-13T14:51:46.812 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.5489999999999995e-06 2024-09-13T14:51:46.812 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.812 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.812 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:46.813 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.052480+0000", 2024-09-13T14:51:46.813 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00023812099999999999 2024-09-13T14:51:46.813 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.813 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.813 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op must wait for map", 2024-09-13T14:51:46.813 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.052502+0000", 2024-09-13T14:51:46.813 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.1322999999999999e-05 2024-09-13T14:51:46.813 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.813 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.813 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:46.813 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.062076+0000", 2024-09-13T14:51:46.813 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0095745269999999993 2024-09-13T14:51:46.813 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.814 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.814 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:51:46.814 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.062085+0000", 2024-09-13T14:51:46.814 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.6300000000000004e-06 2024-09-13T14:51:46.814 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.814 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.814 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:46.814 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.248270+0000", 2024-09-13T14:51:46.814 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.186185171 2024-09-13T14:51:46.814 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.814 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.814 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:51:46.814 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.248356+0000", 2024-09-13T14:51:46.815 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.6057999999999995e-05 2024-09-13T14:51:46.815 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:46.815 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:51:46.815 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:46.815 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.815 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.815 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:252348 8.a 8:5ec110d5:::benchmark_data_smithi102_313275_object121761:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e367)", 2024-09-13T14:51:46.815 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:50:19.052269+0000", 2024-09-13T14:51:46.815 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 87.753208025000006, 2024-09-13T14:51:46.815 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1965683030000001, 2024-09-13T14:51:46.815 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:51:46.815 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:51:46.815 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "reached pg", 2024-09-13T14:51:46.815 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:51:46.816 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:51:46.816 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:51:46.816 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 252348 2024-09-13T14:51:46.816 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.816 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:51:46.816 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.816 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:51:46.816 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.052269+0000", 2024-09-13T14:51:46.816 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:46.816 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.816 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.816 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:51:46.817 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.052269+0000", 2024-09-13T14:51:46.817 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:46.817 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.817 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.817 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:51:46.817 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.052270+0000", 2024-09-13T14:51:46.817 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.2e-07 2024-09-13T14:51:46.817 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.817 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.817 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:51:46.817 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.052274+0000", 2024-09-13T14:51:46.817 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.0450000000000002e-06 2024-09-13T14:51:46.817 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.818 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.818 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:51:46.818 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.052281+0000", 2024-09-13T14:51:46.818 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.8150000000000006e-06 2024-09-13T14:51:46.818 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.818 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.818 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:51:46.818 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.052288+0000", 2024-09-13T14:51:46.818 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.1979999999999998e-06 2024-09-13T14:51:46.818 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.818 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.818 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:46.818 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.052673+0000", 2024-09-13T14:51:46.818 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00038578199999999999 2024-09-13T14:51:46.819 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.819 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.819 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting_for_map not empty", 2024-09-13T14:51:46.819 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.052692+0000", 2024-09-13T14:51:46.819 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.8286000000000001e-05 2024-09-13T14:51:46.819 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.819 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.819 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:46.819 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.062112+0000", 2024-09-13T14:51:46.819 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0094207289999999992 2024-09-13T14:51:46.819 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.819 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.819 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:51:46.819 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.062118+0000", 2024-09-13T14:51:46.820 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.2689999999999999e-06 2024-09-13T14:51:46.820 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.820 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.820 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:46.820 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.248779+0000", 2024-09-13T14:51:46.820 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1866613610000001 2024-09-13T14:51:46.820 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.820 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.820 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:51:46.820 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.248837+0000", 2024-09-13T14:51:46.820 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.8298000000000003e-05 2024-09-13T14:51:46.820 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:46.820 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:51:46.821 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:46.821 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.821 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.821 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:252356 8.a 8:5578b6ec:::benchmark_data_smithi102_313275_object121769:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e367)", 2024-09-13T14:51:46.821 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:50:19.052296+0000", 2024-09-13T14:51:46.821 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 87.753181197000004, 2024-09-13T14:51:46.821 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1966574910000001, 2024-09-13T14:51:46.821 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:51:46.821 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:51:46.821 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "reached pg", 2024-09-13T14:51:46.821 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:51:46.821 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:51:46.821 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:51:46.822 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 252356 2024-09-13T14:51:46.822 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.822 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:51:46.822 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.822 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:51:46.822 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.052296+0000", 2024-09-13T14:51:46.822 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:46.822 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.822 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.822 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:51:46.822 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.052296+0000", 2024-09-13T14:51:46.822 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:46.822 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.822 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.823 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:51:46.823 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.052296+0000", 2024-09-13T14:51:46.823 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.1899999999999998e-07 2024-09-13T14:51:46.823 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.823 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.823 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:51:46.823 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.052298+0000", 2024-09-13T14:51:46.823 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.911e-06 2024-09-13T14:51:46.823 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.823 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.823 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:51:46.823 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.052304+0000", 2024-09-13T14:51:46.823 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.9769999999999997e-06 2024-09-13T14:51:46.824 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.824 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.824 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:51:46.824 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.052309+0000", 2024-09-13T14:51:46.824 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.9599999999999999e-06 2024-09-13T14:51:46.824 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.824 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.824 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:46.824 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.052855+0000", 2024-09-13T14:51:46.824 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00054586299999999995 2024-09-13T14:51:46.824 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.824 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.824 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting_for_map not empty", 2024-09-13T14:51:46.824 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.052876+0000", 2024-09-13T14:51:46.825 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.0845000000000001e-05 2024-09-13T14:51:46.825 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.825 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.825 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:46.825 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.062465+0000", 2024-09-13T14:51:46.825 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0095893200000000001 2024-09-13T14:51:46.825 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.825 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.825 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:51:46.825 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.062473+0000", 2024-09-13T14:51:46.825 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.5340000000000002e-06 2024-09-13T14:51:46.825 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.825 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.825 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:46.826 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.248884+0000", 2024-09-13T14:51:46.826 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.186410838 2024-09-13T14:51:46.826 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.826 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.826 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:51:46.826 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.248953+0000", 2024-09-13T14:51:46.826 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.9824000000000001e-05 2024-09-13T14:51:46.826 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:46.826 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:51:46.826 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:46.826 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.826 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.826 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:252383 8.a 8:5934c274:::benchmark_data_smithi102_313275_object121796:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e367)", 2024-09-13T14:51:46.826 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:50:19.076745+0000", 2024-09-13T14:51:46.827 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 87.728732355000005, 2024-09-13T14:51:46.827 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2015794909999999, 2024-09-13T14:51:46.827 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:51:46.827 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:51:46.827 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:46.827 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:51:46.827 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:51:46.827 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:51:46.827 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 252383 2024-09-13T14:51:46.827 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.827 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:51:46.827 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.827 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:51:46.828 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.076745+0000", 2024-09-13T14:51:46.828 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:46.828 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.828 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.828 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:51:46.828 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.076745+0000", 2024-09-13T14:51:46.828 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:46.828 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.828 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.828 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:51:46.828 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.076747+0000", 2024-09-13T14:51:46.828 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.782e-06 2024-09-13T14:51:46.829 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.829 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.829 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:51:46.829 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.076754+0000", 2024-09-13T14:51:46.829 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.0219999999999999e-06 2024-09-13T14:51:46.829 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.829 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.829 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:51:46.829 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.076772+0000", 2024-09-13T14:51:46.829 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.8465999999999999e-05 2024-09-13T14:51:46.829 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.829 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.829 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:51:46.829 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.076782+0000", 2024-09-13T14:51:46.830 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.7229999999999999e-06 2024-09-13T14:51:46.830 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.830 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.830 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:46.830 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.076850+0000", 2024-09-13T14:51:46.830 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.8052000000000005e-05 2024-09-13T14:51:46.830 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.830 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.830 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:51:46.830 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.076869+0000", 2024-09-13T14:51:46.830 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.9065000000000001e-05 2024-09-13T14:51:46.830 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.830 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.830 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:46.831 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.248980+0000", 2024-09-13T14:51:46.831 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.172110741 2024-09-13T14:51:46.831 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.831 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.831 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:51:46.831 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.249082+0000", 2024-09-13T14:51:46.831 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00010212 2024-09-13T14:51:46.831 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.831 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.831 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 2,5", 2024-09-13T14:51:46.831 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.249656+0000", 2024-09-13T14:51:46.831 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00057433400000000002 2024-09-13T14:51:46.831 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.831 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.832 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:51:46.832 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.278140+0000", 2024-09-13T14:51:46.832 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.028483952999999999 2024-09-13T14:51:46.832 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.832 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.832 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:46.832 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.278196+0000", 2024-09-13T14:51:46.832 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.5767000000000003e-05 2024-09-13T14:51:46.832 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.832 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.832 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:46.832 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.278239+0000", 2024-09-13T14:51:46.832 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.2852999999999998e-05 2024-09-13T14:51:46.832 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.833 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.833 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:51:46.833 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.278287+0000", 2024-09-13T14:51:46.833 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.8078000000000003e-05 2024-09-13T14:51:46.833 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.833 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.833 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:51:46.836 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.278324+0000", 2024-09-13T14:51:46.836 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.7534999999999999e-05 2024-09-13T14:51:46.836 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:46.836 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:51:46.836 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:46.836 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.837 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.837 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:252389 8.a 8:55e597ad:::benchmark_data_smithi102_313275_object121802:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e367)", 2024-09-13T14:51:46.837 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:50:19.083438+0000", 2024-09-13T14:51:46.837 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 87.722039176999999, 2024-09-13T14:51:46.837 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.202485394, 2024-09-13T14:51:46.837 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:51:46.837 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:51:46.837 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:46.837 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:51:46.837 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:51:46.837 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:51:46.837 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 252389 2024-09-13T14:51:46.837 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.837 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:51:46.838 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.838 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:51:46.838 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.083438+0000", 2024-09-13T14:51:46.838 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:46.838 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.838 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.838 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:51:46.838 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.083438+0000", 2024-09-13T14:51:46.838 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:46.838 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.838 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.838 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:51:46.838 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.083440+0000", 2024-09-13T14:51:46.838 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.7370000000000001e-06 2024-09-13T14:51:46.839 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.839 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.839 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:51:46.839 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.083447+0000", 2024-09-13T14:51:46.839 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.9500000000000004e-06 2024-09-13T14:51:46.839 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.839 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.839 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:51:46.839 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.083466+0000", 2024-09-13T14:51:46.839 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.9188999999999999e-05 2024-09-13T14:51:46.839 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.839 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.839 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:51:46.839 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.083474+0000", 2024-09-13T14:51:46.840 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.242e-06 2024-09-13T14:51:46.840 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.840 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.840 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:46.840 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.083647+0000", 2024-09-13T14:51:46.840 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00017294599999999999 2024-09-13T14:51:46.840 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.840 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.840 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:51:46.840 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:46.842 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.083662+0000", 2024-09-13T14:51:46.842 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.5308999999999999e-05 2024-09-13T14:51:46.842 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.842 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.842 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:46.843 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.275214+0000", 2024-09-13T14:51:46.843 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1915512640000001 2024-09-13T14:51:46.843 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.843 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.843 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:51:46.843 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.275286+0000", 2024-09-13T14:51:46.843 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.2293999999999996e-05 2024-09-13T14:51:46.843 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.843 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.843 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 2,5", 2024-09-13T14:51:46.843 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.275374+0000", 2024-09-13T14:51:46.843 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.7997000000000005e-05 2024-09-13T14:51:46.843 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.844 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.844 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:51:46.844 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.285678+0000", 2024-09-13T14:51:46.844 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.010303970000000001 2024-09-13T14:51:46.844 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.844 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.844 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:46.844 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.285716+0000", 2024-09-13T14:51:46.844 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.7945999999999999e-05 2024-09-13T14:51:46.844 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.844 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.844 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:46.844 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.285756+0000", 2024-09-13T14:51:46.845 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.0621999999999998e-05 2024-09-13T14:51:46.845 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.845 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.845 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:51:46.845 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.285884+0000", 2024-09-13T14:51:46.845 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000127816 2024-09-13T14:51:46.845 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.845 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.845 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:51:46.845 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.285923+0000", 2024-09-13T14:51:46.845 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.9112000000000001e-05 2024-09-13T14:51:46.845 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:46.845 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:51:46.845 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:46.846 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.846 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.846 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:252395 8.a 8:5d2ce78d:::benchmark_data_smithi102_313275_object121808:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e367)", 2024-09-13T14:51:46.846 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:50:19.088847+0000", 2024-09-13T14:51:46.846 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 87.716630447, 2024-09-13T14:51:46.846 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2060130390000001, 2024-09-13T14:51:46.846 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:51:46.846 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:51:46.846 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:46.846 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:51:46.846 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:51:46.846 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:51:46.846 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 252395 2024-09-13T14:51:46.846 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.847 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:51:46.847 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.847 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:51:46.847 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.088847+0000", 2024-09-13T14:51:46.847 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:46.847 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.847 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.847 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:51:46.847 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.088847+0000", 2024-09-13T14:51:46.847 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:46.847 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.847 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.847 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:51:46.848 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.088848+0000", 2024-09-13T14:51:46.848 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4100000000000001e-06 2024-09-13T14:51:46.848 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.848 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.848 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:51:46.848 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.088859+0000", 2024-09-13T14:51:46.848 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0547000000000001e-05 2024-09-13T14:51:46.848 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.848 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.848 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:51:46.848 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.088872+0000", 2024-09-13T14:51:46.848 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3115999999999999e-05 2024-09-13T14:51:46.848 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.848 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.849 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:51:46.849 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.088879+0000", 2024-09-13T14:51:46.849 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.374e-06 2024-09-13T14:51:46.849 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.849 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.849 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:46.849 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.089230+0000", 2024-09-13T14:51:46.849 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000350961 2024-09-13T14:51:46.849 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.849 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.849 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:51:46.849 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.089267+0000", 2024-09-13T14:51:46.849 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.7184000000000002e-05 2024-09-13T14:51:46.850 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.850 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.850 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:46.850 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.278358+0000", 2024-09-13T14:51:46.850 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1890905540000001 2024-09-13T14:51:46.850 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.850 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.850 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:51:46.850 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.278467+0000", 2024-09-13T14:51:46.850 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00010953700000000001 2024-09-13T14:51:46.850 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.850 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.850 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 2,5", 2024-09-13T14:51:46.850 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.278823+0000", 2024-09-13T14:51:46.851 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00035605599999999999 2024-09-13T14:51:46.851 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.851 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.851 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:51:46.851 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.285726+0000", 2024-09-13T14:51:46.851 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0069029249999999999 2024-09-13T14:51:46.851 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.851 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.851 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:46.851 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.285963+0000", 2024-09-13T14:51:46.851 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000236791 2024-09-13T14:51:46.851 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.851 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.852 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:46.852 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.294755+0000", 2024-09-13T14:51:46.852 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0087923540000000005 2024-09-13T14:51:46.852 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.852 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.852 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:51:46.852 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.294810+0000", 2024-09-13T14:51:46.852 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.4151999999999999e-05 2024-09-13T14:51:46.852 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.852 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.852 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:51:46.852 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.294860+0000", 2024-09-13T14:51:46.852 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.0077999999999997e-05 2024-09-13T14:51:46.852 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:46.853 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:51:46.853 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:46.853 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.853 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.853 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:252401 8.a 8:54313b1e:::benchmark_data_smithi102_313275_object121814:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e367)", 2024-09-13T14:51:46.853 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:50:19.095734+0000", 2024-09-13T14:51:46.853 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 87.709743360999994, 2024-09-13T14:51:46.853 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.199272672, 2024-09-13T14:51:46.853 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:51:46.853 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:51:46.853 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:46.853 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:51:46.853 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:51:46.853 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:51:46.854 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 252401 2024-09-13T14:51:46.854 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.854 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:51:46.854 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.854 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:51:46.854 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.095734+0000", 2024-09-13T14:51:46.854 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:46.854 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.854 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.854 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:51:46.854 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.095734+0000", 2024-09-13T14:51:46.854 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:46.854 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.854 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.855 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:51:46.855 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.095735+0000", 2024-09-13T14:51:46.855 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.421e-06 2024-09-13T14:51:46.855 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.855 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.855 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:51:46.855 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.095740+0000", 2024-09-13T14:51:46.855 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.5499999999999996e-06 2024-09-13T14:51:46.855 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.855 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.855 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:51:46.855 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.095753+0000", 2024-09-13T14:51:46.855 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3608000000000001e-05 2024-09-13T14:51:46.855 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.856 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.856 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:51:46.856 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.095760+0000", 2024-09-13T14:51:46.856 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.7490000000000001e-06 2024-09-13T14:51:46.856 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.856 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.856 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:46.856 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.095929+0000", 2024-09-13T14:51:46.856 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000169228 2024-09-13T14:51:46.856 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.856 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.856 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:51:46.856 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.095951+0000", 2024-09-13T14:51:46.856 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.1203e-05 2024-09-13T14:51:46.857 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.857 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.857 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:46.857 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.281896+0000", 2024-09-13T14:51:46.857 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.185945706 2024-09-13T14:51:46.857 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.857 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.857 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:51:46.857 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.282016+0000", 2024-09-13T14:51:46.857 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000120196 2024-09-13T14:51:46.857 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.857 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.857 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 2,5", 2024-09-13T14:51:46.858 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.282160+0000", 2024-09-13T14:51:46.858 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00014351699999999999 2024-09-13T14:51:46.858 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.858 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.858 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:51:46.858 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.285973+0000", 2024-09-13T14:51:46.858 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0038129700000000002 2024-09-13T14:51:46.858 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.858 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.858 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:46.858 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.286003+0000", 2024-09-13T14:51:46.858 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.9736000000000001e-05 2024-09-13T14:51:46.858 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.858 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.858 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:46.859 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.294904+0000", 2024-09-13T14:51:46.859 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0089009529999999996 2024-09-13T14:51:46.859 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.859 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.859 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:51:46.859 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.294945+0000", 2024-09-13T14:51:46.859 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.1899e-05 2024-09-13T14:51:46.859 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.859 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.859 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:51:46.859 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.295006+0000", 2024-09-13T14:51:46.859 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.0936000000000001e-05 2024-09-13T14:51:46.859 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:46.859 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:51:46.860 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:46.860 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.860 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.860 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:252407 8.a 8:5b55c829:::benchmark_data_smithi102_313275_object121820:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e367)", 2024-09-13T14:51:46.860 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:50:19.100510+0000", 2024-09-13T14:51:46.860 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 87.704967206000006, 2024-09-13T14:51:46.860 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.221289726, 2024-09-13T14:51:46.860 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:51:46.860 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:51:46.860 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:46.860 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:51:46.860 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:51:46.860 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:51:46.861 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 252407 2024-09-13T14:51:46.861 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.861 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:51:46.861 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.861 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:51:46.861 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.100510+0000", 2024-09-13T14:51:46.861 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:46.861 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.861 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.861 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:51:46.861 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.100510+0000", 2024-09-13T14:51:46.861 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:46.861 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.861 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.862 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:51:46.862 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.100511+0000", 2024-09-13T14:51:46.862 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.252e-06 2024-09-13T14:51:46.862 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.862 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.862 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:51:46.862 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.100516+0000", 2024-09-13T14:51:46.862 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.3070000000000002e-06 2024-09-13T14:51:46.862 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.862 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.862 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:51:46.862 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.100533+0000", 2024-09-13T14:51:46.862 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.7041000000000002e-05 2024-09-13T14:51:46.862 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.862 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.863 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:51:46.863 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.100542+0000", 2024-09-13T14:51:46.863 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.1389999999999995e-06 2024-09-13T14:51:46.863 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.863 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.863 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:46.863 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.100615+0000", 2024-09-13T14:51:46.863 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.3051000000000002e-05 2024-09-13T14:51:46.863 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.863 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.863 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:51:46.863 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.100623+0000", 2024-09-13T14:51:46.863 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.5739999999999996e-06 2024-09-13T14:51:46.863 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.864 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.864 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:46.864 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.286031+0000", 2024-09-13T14:51:46.864 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1854076570000001 2024-09-13T14:51:46.864 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.864 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.864 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:51:46.864 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.286129+0000", 2024-09-13T14:51:46.864 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.7870999999999999e-05 2024-09-13T14:51:46.864 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.864 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.864 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 2,5", 2024-09-13T14:51:46.864 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.286233+0000", 2024-09-13T14:51:46.865 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00010375200000000001 2024-09-13T14:51:46.865 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.865 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.865 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:46.865 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.295050+0000", 2024-09-13T14:51:46.865 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0088175110000000001 2024-09-13T14:51:46.865 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.865 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.865 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:46.865 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.295090+0000", 2024-09-13T14:51:46.865 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.9585000000000001e-05 2024-09-13T14:51:46.865 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.865 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.865 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:51:46.866 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.321672+0000", 2024-09-13T14:51:46.866 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.026581944 2024-09-13T14:51:46.866 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.866 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.866 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:51:46.866 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.321760+0000", 2024-09-13T14:51:46.866 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.8051000000000001e-05 2024-09-13T14:51:46.866 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.866 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.866 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:51:46.866 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.321800+0000", 2024-09-13T14:51:46.866 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.9990999999999997e-05 2024-09-13T14:51:46.866 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:46.866 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:51:46.867 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:46.867 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.867 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.867 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:252409 8.a 8:5f8882c3:::benchmark_data_smithi102_313275_object121822:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e367)", 2024-09-13T14:51:46.867 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:50:19.104221+0000", 2024-09-13T14:51:46.867 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 87.701255998999997, 2024-09-13T14:51:46.867 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.217702721, 2024-09-13T14:51:46.867 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:51:46.867 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:51:46.867 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:46.867 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:51:46.867 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:51:46.867 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:51:46.867 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 252409 2024-09-13T14:51:46.868 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.868 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:51:46.868 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.868 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:51:46.868 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.104221+0000", 2024-09-13T14:51:46.868 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:46.868 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.868 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.868 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:51:46.868 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.104221+0000", 2024-09-13T14:51:46.868 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:46.868 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.868 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.868 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:51:46.869 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.104223+0000", 2024-09-13T14:51:46.869 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.469e-06 2024-09-13T14:51:46.869 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.869 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.869 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:51:46.869 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.104227+0000", 2024-09-13T14:51:46.869 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.6099999999999999e-06 2024-09-13T14:51:46.869 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.869 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.869 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:51:46.869 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.104237+0000", 2024-09-13T14:51:46.869 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.6539999999999994e-06 2024-09-13T14:51:46.869 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.869 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.870 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:51:46.870 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.104243+0000", 2024-09-13T14:51:46.870 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.5409999999999997e-06 2024-09-13T14:51:46.870 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.870 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.870 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:46.870 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.104399+0000", 2024-09-13T14:51:46.870 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000155241 2024-09-13T14:51:46.870 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.870 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.870 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:51:46.870 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.104420+0000", 2024-09-13T14:51:46.870 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.1039999999999998e-05 2024-09-13T14:51:46.870 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.871 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.871 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:46.871 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.291076+0000", 2024-09-13T14:51:46.871 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.186656763 2024-09-13T14:51:46.871 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.871 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.871 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:51:46.871 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.291203+0000", 2024-09-13T14:51:46.871 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00012628800000000001 2024-09-13T14:51:46.871 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.871 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.871 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 2,5", 2024-09-13T14:51:46.871 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.291317+0000", 2024-09-13T14:51:46.872 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00011383 2024-09-13T14:51:46.872 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.872 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.872 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:46.872 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.305727+0000", 2024-09-13T14:51:46.872 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.014410378 2024-09-13T14:51:46.872 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.872 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.872 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:46.872 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.305765+0000", 2024-09-13T14:51:46.872 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.7793999999999998e-05 2024-09-13T14:51:46.872 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.872 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.872 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:51:46.873 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.321856+0000", 2024-09-13T14:51:46.873 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.016091759000000001 2024-09-13T14:51:46.873 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.873 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.873 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:51:46.873 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.321893+0000", 2024-09-13T14:51:46.873 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.6247e-05 2024-09-13T14:51:46.873 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.873 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.873 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:51:46.873 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.321924+0000", 2024-09-13T14:51:46.873 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.1106999999999999e-05 2024-09-13T14:51:46.873 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:46.874 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:51:46.874 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:46.874 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.874 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.874 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:252418 8.a 8:53b16a66:::benchmark_data_smithi102_313275_object121831:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e367)", 2024-09-13T14:51:46.874 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:50:19.111174+0000", 2024-09-13T14:51:46.874 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 87.694303425000001, 2024-09-13T14:51:46.874 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.210831445, 2024-09-13T14:51:46.874 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:51:46.874 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:51:46.874 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:46.874 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:51:46.874 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:51:46.874 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:51:46.875 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 252418 2024-09-13T14:51:46.875 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.875 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:51:46.875 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.875 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:51:46.875 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.111174+0000", 2024-09-13T14:51:46.875 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:46.875 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.875 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.875 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:51:46.875 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.111174+0000", 2024-09-13T14:51:46.875 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:46.875 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.875 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.876 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:51:46.876 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.111175+0000", 2024-09-13T14:51:46.876 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.468e-06 2024-09-13T14:51:46.876 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.876 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.876 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:51:46.876 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.111180+0000", 2024-09-13T14:51:46.876 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.0780000000000003e-06 2024-09-13T14:51:46.876 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.876 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.876 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:51:46.876 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.111192+0000", 2024-09-13T14:51:46.876 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1789e-05 2024-09-13T14:51:46.876 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.877 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.877 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:51:46.877 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.111199+0000", 2024-09-13T14:51:46.877 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.0999999999999998e-06 2024-09-13T14:51:46.877 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.877 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.877 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:46.877 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.111255+0000", 2024-09-13T14:51:46.877 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.6264000000000002e-05 2024-09-13T14:51:46.877 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.877 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.877 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:51:46.877 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.111264+0000", 2024-09-13T14:51:46.878 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.1850000000000004e-06 2024-09-13T14:51:46.878 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.878 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.878 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:46.878 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.295119+0000", 2024-09-13T14:51:46.878 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.183855814 2024-09-13T14:51:46.878 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.878 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.878 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:51:46.878 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.295250+0000", 2024-09-13T14:51:46.878 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000130302 2024-09-13T14:51:46.878 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.878 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.878 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 2,5", 2024-09-13T14:51:46.879 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.295395+0000", 2024-09-13T14:51:46.879 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000144945 2024-09-13T14:51:46.879 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.879 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.879 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:46.879 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.305800+0000", 2024-09-13T14:51:46.879 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.010405642999999999 2024-09-13T14:51:46.879 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.879 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.879 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:46.879 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.313168+0000", 2024-09-13T14:51:46.879 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.007367635 2024-09-13T14:51:46.879 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.879 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.880 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:51:46.880 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.321929+0000", 2024-09-13T14:51:46.880 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0087614579999999997 2024-09-13T14:51:46.880 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.880 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.880 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:51:46.880 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.321976+0000", 2024-09-13T14:51:46.880 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.6618e-05 2024-09-13T14:51:46.880 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.880 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.880 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:51:46.880 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.322005+0000", 2024-09-13T14:51:46.880 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.9145999999999998e-05 2024-09-13T14:51:46.881 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:46.881 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:51:46.881 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:46.881 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.881 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.881 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:252426 8.a 8:51e1e350:::benchmark_data_smithi102_313275_object121839:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e367)", 2024-09-13T14:51:46.881 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:50:19.120028+0000", 2024-09-13T14:51:46.881 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 87.685449183000003, 2024-09-13T14:51:46.881 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.202042093, 2024-09-13T14:51:46.881 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:51:46.881 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:51:46.881 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:46.881 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:51:46.881 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:51:46.882 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:51:46.882 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 252426 2024-09-13T14:51:46.882 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.882 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:51:46.882 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.882 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:51:46.882 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.120028+0000", 2024-09-13T14:51:46.882 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:46.882 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.882 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.882 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:51:46.882 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.120028+0000", 2024-09-13T14:51:46.882 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:46.882 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.883 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.883 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:51:46.883 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.120029+0000", 2024-09-13T14:51:46.883 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.238e-06 2024-09-13T14:51:46.883 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.883 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.883 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:51:46.883 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.120033+0000", 2024-09-13T14:51:46.883 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.2930000000000002e-06 2024-09-13T14:51:46.883 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.883 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.883 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:51:46.883 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.120044+0000", 2024-09-13T14:51:46.883 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0738e-05 2024-09-13T14:51:46.883 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.884 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.884 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:51:46.884 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.120051+0000", 2024-09-13T14:51:46.884 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.669e-06 2024-09-13T14:51:46.884 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.884 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.884 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:46.884 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.120110+0000", 2024-09-13T14:51:46.884 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.8900000000000002e-05 2024-09-13T14:51:46.884 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.884 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.884 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:51:46.884 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.120118+0000", 2024-09-13T14:51:46.885 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.2290000000000003e-06 2024-09-13T14:51:46.885 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.885 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.885 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:46.885 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.298589+0000", 2024-09-13T14:51:46.885 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.178470525 2024-09-13T14:51:46.885 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.885 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.885 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:51:46.885 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.298681+0000", 2024-09-13T14:51:46.885 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.2600000000000001e-05 2024-09-13T14:51:46.885 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.885 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.885 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 2,5", 2024-09-13T14:51:46.886 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.298781+0000", 2024-09-13T14:51:46.886 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00010007 2024-09-13T14:51:46.886 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.886 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.886 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:46.886 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.305839+0000", 2024-09-13T14:51:46.886 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0070574569999999996 2024-09-13T14:51:46.886 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.886 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.886 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:46.886 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.313206+0000", 2024-09-13T14:51:46.886 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0073676119999999999 2024-09-13T14:51:46.886 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.886 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.887 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:51:46.887 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.322010+0000", 2024-09-13T14:51:46.887 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.008803857 2024-09-13T14:51:46.887 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.887 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.887 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:51:46.887 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.322043+0000", 2024-09-13T14:51:46.887 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.2775999999999999e-05 2024-09-13T14:51:46.887 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.887 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.887 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:51:46.887 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.322070+0000", 2024-09-13T14:51:46.887 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.7129000000000001e-05 2024-09-13T14:51:46.888 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:46.888 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:51:46.888 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:46.888 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.888 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.888 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.36528.0:72018 9.12 9:4a33b163:::benchmark_data_smithi102_348563_object72017:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T14:51:46.888 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:51:45.055437+0000", 2024-09-13T14:51:46.888 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 1.7500402369999999, 2024-09-13T14:51:46.888 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1630659910000001, 2024-09-13T14:51:46.888 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:51:46.888 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:51:46.888 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:46.888 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:51:46.888 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.36528", 2024-09-13T14:51:46.889 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:51:46.889 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 72018 2024-09-13T14:51:46.889 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.889 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:51:46.889 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.889 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:51:46.889 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055437+0000", 2024-09-13T14:51:46.889 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:46.889 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.889 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.889 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:51:46.889 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055437+0000", 2024-09-13T14:51:46.889 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:46.889 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.890 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.890 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:51:46.890 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055437+0000", 2024-09-13T14:51:46.890 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.0200000000000002e-07 2024-09-13T14:51:46.890 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.890 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.890 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:51:46.890 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055480+0000", 2024-09-13T14:51:46.890 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.3099000000000002e-05 2024-09-13T14:51:46.890 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.890 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.890 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:51:46.890 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055488+0000", 2024-09-13T14:51:46.890 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.4590000000000001e-06 2024-09-13T14:51:46.891 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.891 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.891 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:51:46.891 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055493+0000", 2024-09-13T14:51:46.891 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.1830000000000002e-06 2024-09-13T14:51:46.891 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.891 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.891 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:46.891 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055544+0000", 2024-09-13T14:51:46.891 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.0797999999999997e-05 2024-09-13T14:51:46.891 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.891 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.891 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:51:46.891 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055554+0000", 2024-09-13T14:51:46.892 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.7399999999999999e-06 2024-09-13T14:51:46.892 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.892 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.892 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:46.892 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.197245+0000", 2024-09-13T14:51:46.892 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1416912830000001 2024-09-13T14:51:46.892 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.892 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.892 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:51:46.892 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.197357+0000", 2024-09-13T14:51:46.892 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000112503 2024-09-13T14:51:46.892 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.892 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.892 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:51:46.893 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.197453+0000", 2024-09-13T14:51:46.893 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.5779999999999994e-05 2024-09-13T14:51:46.893 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.893 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.893 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:51:46.893 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.200844+0000", 2024-09-13T14:51:46.893 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.003390924 2024-09-13T14:51:46.893 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.893 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.893 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:46.893 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.200894+0000", 2024-09-13T14:51:46.893 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.9375000000000001e-05 2024-09-13T14:51:46.893 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.893 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.894 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:46.894 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.215451+0000", 2024-09-13T14:51:46.894 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.014557455 2024-09-13T14:51:46.894 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.894 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.894 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:51:46.894 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.215582+0000", 2024-09-13T14:51:46.894 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000130655 2024-09-13T14:51:46.894 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.894 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.894 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:51:46.894 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.218503+0000", 2024-09-13T14:51:46.894 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0029212349999999999 2024-09-13T14:51:46.894 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:46.895 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:51:46.895 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:46.895 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.895 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.895 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.36528.0:72059 9.12 9:4c71233f:::benchmark_data_smithi102_348563_object72058:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T14:51:46.895 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:51:45.055516+0000", 2024-09-13T14:51:46.895 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 1.749961098, 2024-09-13T14:51:46.895 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.165950303, 2024-09-13T14:51:46.895 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:51:46.895 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:51:46.895 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:46.895 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:51:46.895 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.36528", 2024-09-13T14:51:46.895 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:51:46.895 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 72059 2024-09-13T14:51:46.896 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.896 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:51:46.896 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.896 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:51:46.896 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055516+0000", 2024-09-13T14:51:46.896 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:46.896 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.896 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.896 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:51:46.896 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055516+0000", 2024-09-13T14:51:46.896 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:46.896 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.896 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.896 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:51:46.897 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055517+0000", 2024-09-13T14:51:46.897 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.0400000000000005e-07 2024-09-13T14:51:46.897 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.897 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.897 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:51:46.897 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055570+0000", 2024-09-13T14:51:46.897 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.3514000000000003e-05 2024-09-13T14:51:46.897 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.897 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.897 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:51:46.897 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055584+0000", 2024-09-13T14:51:46.897 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3682999999999999e-05 2024-09-13T14:51:46.897 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.898 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.898 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:51:46.898 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055592+0000", 2024-09-13T14:51:46.898 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.0790000000000001e-06 2024-09-13T14:51:46.898 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.898 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.898 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:46.898 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055639+0000", 2024-09-13T14:51:46.898 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.7332e-05 2024-09-13T14:51:46.898 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.898 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.898 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:51:46.898 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055644+0000", 2024-09-13T14:51:46.898 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.8749999999999999e-06 2024-09-13T14:51:46.899 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.899 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.899 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:46.899 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.200933+0000", 2024-09-13T14:51:46.899 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1452891000000001 2024-09-13T14:51:46.899 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.899 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.899 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:51:46.899 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.201107+0000", 2024-09-13T14:51:46.899 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000173158 2024-09-13T14:51:46.899 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.899 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.899 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:51:46.899 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.201288+0000", 2024-09-13T14:51:46.900 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00018143 2024-09-13T14:51:46.900 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.900 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.900 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:51:46.900 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.205867+0000", 2024-09-13T14:51:46.900 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0045794649999999996 2024-09-13T14:51:46.900 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.900 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.900 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:46.900 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.207600+0000", 2024-09-13T14:51:46.900 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.001732716 2024-09-13T14:51:46.900 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.900 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.900 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:46.901 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.218907+0000", 2024-09-13T14:51:46.901 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.011306837 2024-09-13T14:51:46.901 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.901 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.901 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:51:46.901 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.219315+0000", 2024-09-13T14:51:46.901 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00040796499999999999 2024-09-13T14:51:46.901 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.901 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.901 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:51:46.901 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.221466+0000", 2024-09-13T14:51:46.901 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0021513449999999998 2024-09-13T14:51:46.901 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:46.901 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:51:46.902 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:46.902 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.902 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.902 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.36528.0:72094 9.12 9:4c58cbaa:::benchmark_data_smithi102_348563_object72093:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T14:51:46.902 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:51:45.055610+0000", 2024-09-13T14:51:46.902 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 1.7498672669999999, 2024-09-13T14:51:46.902 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.17144435, 2024-09-13T14:51:46.902 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:51:46.902 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:51:46.902 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:46.902 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:51:46.902 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.36528", 2024-09-13T14:51:46.902 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:51:46.903 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 72094 2024-09-13T14:51:46.903 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.903 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:51:46.903 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.903 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:51:46.903 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055610+0000", 2024-09-13T14:51:46.903 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:46.903 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.903 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.903 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:51:46.903 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055610+0000", 2024-09-13T14:51:46.903 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:46.903 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.904 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.904 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:51:46.904 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055610+0000", 2024-09-13T14:51:46.904 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.82e-07 2024-09-13T14:51:46.904 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.904 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.904 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:51:46.904 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055657+0000", 2024-09-13T14:51:46.904 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.6347000000000002e-05 2024-09-13T14:51:46.904 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.904 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.904 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:51:46.904 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055667+0000", 2024-09-13T14:51:46.905 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.7880000000000001e-06 2024-09-13T14:51:46.905 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.905 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.905 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:51:46.905 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055672+0000", 2024-09-13T14:51:46.905 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.8320000000000002e-06 2024-09-13T14:51:46.905 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.905 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.905 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:46.905 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055705+0000", 2024-09-13T14:51:46.905 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.3049000000000002e-05 2024-09-13T14:51:46.905 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.905 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.906 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:51:46.906 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055711+0000", 2024-09-13T14:51:46.906 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.8980000000000001e-06 2024-09-13T14:51:46.906 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.906 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.906 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:46.906 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.202961+0000", 2024-09-13T14:51:46.906 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.147249438 2024-09-13T14:51:46.906 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.906 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.906 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:51:46.906 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.203111+0000", 2024-09-13T14:51:46.906 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000150211 2024-09-13T14:51:46.906 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.907 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.907 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:51:46.907 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.203262+0000", 2024-09-13T14:51:46.907 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000151187 2024-09-13T14:51:46.907 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.907 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.907 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:46.907 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.207649+0000", 2024-09-13T14:51:46.907 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0043872900000000003 2024-09-13T14:51:46.907 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.907 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.907 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:51:46.907 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.211746+0000", 2024-09-13T14:51:46.907 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0040962230000000004 2024-09-13T14:51:46.908 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.908 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.908 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:46.908 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.224231+0000", 2024-09-13T14:51:46.908 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.012485489000000001 2024-09-13T14:51:46.908 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.908 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.908 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:51:46.908 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.224292+0000", 2024-09-13T14:51:46.908 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.0692000000000001e-05 2024-09-13T14:51:46.908 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.908 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.908 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:51:46.909 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.227054+0000", 2024-09-13T14:51:46.909 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.002762324 2024-09-13T14:51:46.909 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:46.909 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:51:46.909 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:46.909 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.909 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.909 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.36528.0:72114 9.12 9:48c2513d:::benchmark_data_smithi102_348563_object72113:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T14:51:46.909 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:51:45.055685+0000", 2024-09-13T14:51:46.909 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 1.7497924309999999, 2024-09-13T14:51:46.909 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.173506333, 2024-09-13T14:51:46.909 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:51:46.909 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:51:46.909 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:46.910 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:51:46.910 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.36528", 2024-09-13T14:51:46.910 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:51:46.910 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 72114 2024-09-13T14:51:46.910 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.910 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:51:46.910 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.910 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:51:46.910 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055685+0000", 2024-09-13T14:51:46.910 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:46.910 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.910 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.910 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:51:46.910 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055685+0000", 2024-09-13T14:51:46.911 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:46.911 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.911 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.911 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:51:46.911 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055685+0000", 2024-09-13T14:51:46.911 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.4099999999999999e-07 2024-09-13T14:51:46.911 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.911 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.913 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:51:46.913 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055729+0000", 2024-09-13T14:51:46.913 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.4044e-05 2024-09-13T14:51:46.913 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.913 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.913 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:51:46.913 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055738+0000", 2024-09-13T14:51:46.913 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.8300000000000002e-06 2024-09-13T14:51:46.913 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.914 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.914 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:51:46.914 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055743+0000", 2024-09-13T14:51:46.914 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.3229999999999997e-06 2024-09-13T14:51:46.914 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.914 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.914 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:46.914 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055785+0000", 2024-09-13T14:51:46.914 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.1734999999999999e-05 2024-09-13T14:51:46.914 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.914 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.914 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:51:46.914 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055792+0000", 2024-09-13T14:51:46.915 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.8770000000000004e-06 2024-09-13T14:51:46.915 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.915 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.915 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:46.915 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.205937+0000", 2024-09-13T14:51:46.915 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1501447659999999 2024-09-13T14:51:46.915 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.915 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.915 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:51:46.915 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.206070+0000", 2024-09-13T14:51:46.915 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000133132 2024-09-13T14:51:46.915 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.915 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.915 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:51:46.916 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.206221+0000", 2024-09-13T14:51:46.916 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00015074399999999999 2024-09-13T14:51:46.916 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.916 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.916 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:46.916 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.211639+0000", 2024-09-13T14:51:46.916 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0054186119999999997 2024-09-13T14:51:46.916 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.916 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.916 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:51:46.916 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.214424+0000", 2024-09-13T14:51:46.916 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0027847800000000002 2024-09-13T14:51:46.916 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.917 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.917 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:46.917 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.227212+0000", 2024-09-13T14:51:46.917 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.012788336000000001 2024-09-13T14:51:46.917 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.917 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.917 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:51:46.917 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.227288+0000", 2024-09-13T14:51:46.917 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.5174999999999998e-05 2024-09-13T14:51:46.917 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.917 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.917 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:51:46.917 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.229191+0000", 2024-09-13T14:51:46.918 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0019034379999999999 2024-09-13T14:51:46.918 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:46.918 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:51:46.918 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:46.918 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.918 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.918 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.36528.0:72115 9.12 9:48e0c73b:::benchmark_data_smithi102_348563_object72114:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T14:51:46.918 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:51:45.055755+0000", 2024-09-13T14:51:46.918 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 1.7497224579999999, 2024-09-13T14:51:46.918 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1751879940000001, 2024-09-13T14:51:46.918 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:51:46.918 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:51:46.918 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:46.918 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:51:46.919 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.36528", 2024-09-13T14:51:46.919 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:51:46.919 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 72115 2024-09-13T14:51:46.919 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.919 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:51:46.919 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.919 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:51:46.919 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055755+0000", 2024-09-13T14:51:46.919 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:46.919 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.919 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.919 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:51:46.919 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055755+0000", 2024-09-13T14:51:46.919 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:46.920 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.920 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.920 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:51:46.920 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055755+0000", 2024-09-13T14:51:46.920 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.6499999999999999e-07 2024-09-13T14:51:46.920 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.920 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.920 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:51:46.920 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055799+0000", 2024-09-13T14:51:46.920 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.3971999999999998e-05 2024-09-13T14:51:46.920 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.920 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.920 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:51:46.921 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055807+0000", 2024-09-13T14:51:46.921 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.2800000000000003e-06 2024-09-13T14:51:46.921 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.921 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.921 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:51:46.921 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055813+0000", 2024-09-13T14:51:46.921 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.49e-06 2024-09-13T14:51:46.921 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.921 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.921 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:46.921 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055839+0000", 2024-09-13T14:51:46.921 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.6492999999999999e-05 2024-09-13T14:51:46.921 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.921 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.922 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:51:46.922 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055843+0000", 2024-09-13T14:51:46.922 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.7929999999999999e-06 2024-09-13T14:51:46.922 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.922 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.922 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:46.922 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.207677+0000", 2024-09-13T14:51:46.922 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1518341400000001 2024-09-13T14:51:46.922 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.922 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.922 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:51:46.922 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.207846+0000", 2024-09-13T14:51:46.922 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000168908 2024-09-13T14:51:46.922 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.923 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.923 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:51:46.923 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.207989+0000", 2024-09-13T14:51:46.923 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00014232000000000001 2024-09-13T14:51:46.923 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.923 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.923 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:46.923 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.211724+0000", 2024-09-13T14:51:46.923 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0037355489999999999 2024-09-13T14:51:46.923 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.923 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.923 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:51:46.923 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.214439+0000", 2024-09-13T14:51:46.923 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0027154369999999998 2024-09-13T14:51:46.924 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.924 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.924 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:46.924 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.229240+0000", 2024-09-13T14:51:46.924 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.014800845 2024-09-13T14:51:46.924 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.924 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.924 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:51:46.924 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.229384+0000", 2024-09-13T14:51:46.924 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00014322799999999999 2024-09-13T14:51:46.924 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.924 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.924 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:51:46.924 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.230943+0000", 2024-09-13T14:51:46.925 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0015590739999999999 2024-09-13T14:51:46.925 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:46.925 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:51:46.925 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:46.925 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.925 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.925 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.36528.0:72120 9.12 9:49a2b7f2:::benchmark_data_smithi102_348563_object72119:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T14:51:46.925 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:51:45.055823+0000", 2024-09-13T14:51:46.925 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 1.749654051, 2024-09-13T14:51:46.925 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1773272690000001, 2024-09-13T14:51:46.925 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:51:46.925 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:51:46.925 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:46.925 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:51:46.926 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.36528", 2024-09-13T14:51:46.926 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:51:46.926 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 72120 2024-09-13T14:51:46.926 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.926 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:51:46.926 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.926 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:51:46.926 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055823+0000", 2024-09-13T14:51:46.926 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:46.926 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.926 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.926 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:51:46.926 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055823+0000", 2024-09-13T14:51:46.926 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:46.927 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.927 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.927 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:51:46.927 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055824+0000", 2024-09-13T14:51:46.927 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.7800000000000002e-07 2024-09-13T14:51:46.927 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.927 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.927 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:51:46.927 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055879+0000", 2024-09-13T14:51:46.927 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.5770999999999998e-05 2024-09-13T14:51:46.927 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.927 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.927 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:51:46.927 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055888+0000", 2024-09-13T14:51:46.928 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.0580000000000004e-06 2024-09-13T14:51:46.928 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.928 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.928 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:51:46.928 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055894+0000", 2024-09-13T14:51:46.928 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.4909999999999996e-06 2024-09-13T14:51:46.928 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.928 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.928 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:46.928 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055922+0000", 2024-09-13T14:51:46.928 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.7801e-05 2024-09-13T14:51:46.928 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.928 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.929 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:51:46.929 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055925+0000", 2024-09-13T14:51:46.929 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.6389999999999998e-06 2024-09-13T14:51:46.929 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.929 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.929 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:46.929 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.209252+0000", 2024-09-13T14:51:46.929 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1533266170000001 2024-09-13T14:51:46.929 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.929 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.929 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:51:46.929 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.209386+0000", 2024-09-13T14:51:46.929 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00013422299999999999 2024-09-13T14:51:46.929 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.930 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.930 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:51:46.930 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.209542+0000", 2024-09-13T14:51:46.930 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00015550500000000001 2024-09-13T14:51:46.930 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.930 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.930 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:46.930 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.214299+0000", 2024-09-13T14:51:46.930 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.004757102 2024-09-13T14:51:46.930 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.930 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.930 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:51:46.930 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.214450+0000", 2024-09-13T14:51:46.931 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00015120699999999999 2024-09-13T14:51:46.931 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.931 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.931 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:46.931 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.231578+0000", 2024-09-13T14:51:46.931 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.017128056999999999 2024-09-13T14:51:46.931 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.931 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.931 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:51:46.931 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.231635+0000", 2024-09-13T14:51:46.931 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.7043000000000003e-05 2024-09-13T14:51:46.931 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.931 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.932 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:51:46.932 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.233150+0000", 2024-09-13T14:51:46.932 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.001515277 2024-09-13T14:51:46.932 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:46.932 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:51:46.932 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:46.932 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.932 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.932 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.36528.0:72209 9.12 9:484a00ee:::benchmark_data_smithi102_348563_object72208:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T14:51:46.932 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:51:45.055905+0000", 2024-09-13T14:51:46.932 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 1.749572101, 2024-09-13T14:51:46.932 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1930136650000001, 2024-09-13T14:51:46.933 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:51:46.933 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:51:46.933 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:46.933 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:51:46.933 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.36528", 2024-09-13T14:51:46.933 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:51:46.933 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 72209 2024-09-13T14:51:46.933 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.933 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:51:46.933 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.933 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:51:46.933 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055905+0000", 2024-09-13T14:51:46.933 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:46.933 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.934 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.934 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:51:46.934 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055905+0000", 2024-09-13T14:51:46.934 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:46.934 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.934 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.934 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:51:46.934 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055906+0000", 2024-09-13T14:51:46.934 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.1699999999999998e-07 2024-09-13T14:51:46.934 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.934 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.934 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:51:46.934 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.056091+0000", 2024-09-13T14:51:46.934 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00018559800000000001 2024-09-13T14:51:46.935 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.935 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.935 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:51:46.935 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.056101+0000", 2024-09-13T14:51:46.935 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.7480000000000005e-06 2024-09-13T14:51:46.935 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.935 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.935 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:51:46.935 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.056107+0000", 2024-09-13T14:51:46.935 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.2090000000000002e-06 2024-09-13T14:51:46.935 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.935 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.935 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:46.935 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.056149+0000", 2024-09-13T14:51:46.936 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.1588000000000002e-05 2024-09-13T14:51:46.936 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.936 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.936 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:51:46.936 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.056155+0000", 2024-09-13T14:51:46.936 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.8159999999999996e-06 2024-09-13T14:51:46.936 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.936 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.936 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:46.936 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.233187+0000", 2024-09-13T14:51:46.936 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1770313290000001 2024-09-13T14:51:46.936 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.936 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.936 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:51:46.937 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.233321+0000", 2024-09-13T14:51:46.937 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00013375600000000001 2024-09-13T14:51:46.937 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.937 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.937 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:51:46.937 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.233472+0000", 2024-09-13T14:51:46.937 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00015193199999999999 2024-09-13T14:51:46.937 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.937 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.937 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:51:46.937 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.241018+0000", 2024-09-13T14:51:46.937 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0075451299999999997 2024-09-13T14:51:46.937 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.937 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.938 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:46.938 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.242760+0000", 2024-09-13T14:51:46.938 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.001742328 2024-09-13T14:51:46.938 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.938 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.938 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:46.938 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.247171+0000", 2024-09-13T14:51:46.938 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0044109700000000002 2024-09-13T14:51:46.938 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.938 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.938 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:51:46.938 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.247265+0000", 2024-09-13T14:51:46.938 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.4537000000000006e-05 2024-09-13T14:51:46.939 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.939 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.939 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:51:46.939 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.248919+0000", 2024-09-13T14:51:46.939 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0016532070000000001 2024-09-13T14:51:46.939 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:46.939 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:51:46.939 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:46.939 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.939 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.939 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.36528.0:72281 9.12 9:4e035cfa:::benchmark_data_smithi102_348563_object72280:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T14:51:46.939 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:51:45.056119+0000", 2024-09-13T14:51:46.939 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 1.749358084, 2024-09-13T14:51:46.939 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1953139319999999, 2024-09-13T14:51:46.940 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:51:46.940 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:51:46.940 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:46.940 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:51:46.940 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.36528", 2024-09-13T14:51:46.940 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:51:46.940 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 72281 2024-09-13T14:51:46.940 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.940 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:51:46.940 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.940 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:51:46.940 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.056119+0000", 2024-09-13T14:51:46.940 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:46.941 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.941 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.941 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:51:46.941 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.056119+0000", 2024-09-13T14:51:46.941 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:46.941 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.941 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.941 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:51:46.941 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.056120+0000", 2024-09-13T14:51:46.941 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.2600000000000002e-07 2024-09-13T14:51:46.941 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.941 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.941 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:51:46.942 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.056164+0000", 2024-09-13T14:51:46.942 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.4051999999999998e-05 2024-09-13T14:51:46.942 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.942 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.942 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:51:46.942 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.056173+0000", 2024-09-13T14:51:46.942 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.9299999999999992e-06 2024-09-13T14:51:46.942 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.942 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.942 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:51:46.942 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.056178+0000", 2024-09-13T14:51:46.942 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.716e-06 2024-09-13T14:51:46.942 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.942 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.943 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:46.943 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.056220+0000", 2024-09-13T14:51:46.943 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.2079000000000001e-05 2024-09-13T14:51:46.943 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.943 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.943 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:51:46.943 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.056228+0000", 2024-09-13T14:51:46.943 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.9789999999999993e-06 2024-09-13T14:51:46.943 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.943 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.943 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:46.943 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.235277+0000", 2024-09-13T14:51:46.943 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1790484210000001 2024-09-13T14:51:46.943 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.944 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.944 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:51:46.944 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.235422+0000", 2024-09-13T14:51:46.944 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000145518 2024-09-13T14:51:46.944 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.944 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.944 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:51:46.944 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.235596+0000", 2024-09-13T14:51:46.944 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00017405 2024-09-13T14:51:46.944 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.944 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.944 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:51:46.944 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.242515+0000", 2024-09-13T14:51:46.945 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.006918446 2024-09-13T14:51:46.945 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.945 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.945 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:46.945 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.243140+0000", 2024-09-13T14:51:46.945 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00062485299999999998 2024-09-13T14:51:46.945 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.945 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.945 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:46.945 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.249644+0000", 2024-09-13T14:51:46.945 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.006504625 2024-09-13T14:51:46.945 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.945 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.945 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:51:46.946 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.249962+0000", 2024-09-13T14:51:46.946 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00031751700000000002 2024-09-13T14:51:46.946 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.946 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.946 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:51:46.946 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.251433+0000", 2024-09-13T14:51:46.946 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0014712200000000001 2024-09-13T14:51:46.946 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:46.946 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:51:46.946 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:46.946 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.946 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.946 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.36528.0:72296 9.12 9:4ce750ff:::benchmark_data_smithi102_348563_object72295:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T14:51:46.946 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:51:45.056189+0000", 2024-09-13T14:51:46.947 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 1.7492879810000002, 2024-09-13T14:51:46.947 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1970618669999999, 2024-09-13T14:51:46.947 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:51:46.947 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:51:46.947 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:46.947 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:51:46.947 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.36528", 2024-09-13T14:51:46.947 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:51:46.947 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 72296 2024-09-13T14:51:46.947 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.947 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:51:46.947 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.947 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:51:46.947 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.056189+0000", 2024-09-13T14:51:46.948 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:46.948 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.948 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.948 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:51:46.948 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.056189+0000", 2024-09-13T14:51:46.948 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:51:46.948 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.948 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.948 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:51:46.948 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.056190+0000", 2024-09-13T14:51:46.948 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.4200000000000001e-07 2024-09-13T14:51:46.948 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.948 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.948 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:51:46.949 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.056233+0000", 2024-09-13T14:51:46.949 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.3047000000000003e-05 2024-09-13T14:51:46.949 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.949 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.949 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:51:46.949 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.056242+0000", 2024-09-13T14:51:46.949 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.5610000000000004e-06 2024-09-13T14:51:46.949 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.949 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.949 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:51:46.949 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.056248+0000", 2024-09-13T14:51:46.949 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.6690000000000003e-06 2024-09-13T14:51:46.949 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.949 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.950 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:46.950 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.056285+0000", 2024-09-13T14:51:46.950 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.6897999999999998e-05 2024-09-13T14:51:46.950 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.950 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.950 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:51:46.950 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.056291+0000", 2024-09-13T14:51:46.950 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.9039999999999997e-06 2024-09-13T14:51:46.950 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.950 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.950 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:51:46.950 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.237848+0000", 2024-09-13T14:51:46.950 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1815569530000001 2024-09-13T14:51:46.950 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.950 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.951 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:51:46.951 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.238023+0000", 2024-09-13T14:51:46.951 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00017584499999999999 2024-09-13T14:51:46.951 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.951 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.951 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:51:46.951 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.238186+0000", 2024-09-13T14:51:46.951 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00016267899999999999 2024-09-13T14:51:46.951 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.951 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.951 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:51:46.951 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.242556+0000", 2024-09-13T14:51:46.951 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0043702200000000002 2024-09-13T14:51:46.951 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.952 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.952 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:46.952 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.243176+0000", 2024-09-13T14:51:46.952 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00061982900000000002 2024-09-13T14:51:46.952 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.952 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.952 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:46.952 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.251472+0000", 2024-09-13T14:51:46.952 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0082953430000000002 2024-09-13T14:51:46.952 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.952 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.952 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:51:46.952 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.251506+0000", 2024-09-13T14:51:46.952 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.4180000000000001e-05 2024-09-13T14:51:46.953 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:51:46.953 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:51:46.953 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:51:46.953 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.253251+0000", 2024-09-13T14:51:46.953 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.001745297 2024-09-13T14:51:46.953 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:46.953 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:51:46.953 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:46.953 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:51:46.953 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:51:46.953 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:51:46.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:46.954 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:46.954 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:46.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:47.055 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:47.081 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:47.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:47.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:47.105 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 41 16 72821 72805 110.969 16.4062 0.00387946 0.0088645 2024-09-13T14:51:47.183 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:47.205 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:47.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:47.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:47.307 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:47.329 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:47.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:47.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:47.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:47 smithi102 ceph-mon[110183]: 9.19 scrub ok 2024-09-13T14:51:47.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:47 smithi102 ceph-mon[110183]: pgmap v1332: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 38 MiB/s wr, 601 op/s; 6207/210489 objects misplaced (2.949%); 2.0 MiB/s, 31 objects/s recovering 2024-09-13T14:51:47.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:47 smithi102 ceph-mon[110183]: 9.18 scrub starts 2024-09-13T14:51:47.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:47 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "20"}]': finished 2024-09-13T14:51:47.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:47 smithi102 ceph-mon[110183]: osdmap e390: 8 total, 8 up, 8 in 2024-09-13T14:51:47.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:47 smithi102 ceph-mon[110183]: 9.14 scrub starts 2024-09-13T14:51:47.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:47 smithi102 ceph-mon[110183]: 9.9 scrub starts 2024-09-13T14:51:47.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:47 smithi102 ceph-mon[110183]: 9.18 scrub ok 2024-09-13T14:51:47.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:47 smithi102 ceph-mon[115269]: 9.19 scrub ok 2024-09-13T14:51:47.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:47 smithi102 ceph-mon[115269]: pgmap v1332: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 38 MiB/s wr, 601 op/s; 6207/210489 objects misplaced (2.949%); 2.0 MiB/s, 31 objects/s recovering 2024-09-13T14:51:47.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:47 smithi102 ceph-mon[115269]: 9.18 scrub starts 2024-09-13T14:51:47.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:47 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "20"}]': finished 2024-09-13T14:51:47.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:47 smithi102 ceph-mon[115269]: osdmap e390: 8 total, 8 up, 8 in 2024-09-13T14:51:47.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:47 smithi102 ceph-mon[115269]: 9.14 scrub starts 2024-09-13T14:51:47.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:47 smithi102 ceph-mon[115269]: 9.9 scrub starts 2024-09-13T14:51:47.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:47 smithi102 ceph-mon[115269]: 9.18 scrub ok 2024-09-13T14:51:47.431 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:47.459 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:47.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:47.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:47.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:47 smithi195 ceph-mon[104262]: 9.19 scrub ok 2024-09-13T14:51:47.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:47 smithi195 ceph-mon[104262]: pgmap v1332: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 38 MiB/s wr, 601 op/s; 6207/210489 objects misplaced (2.949%); 2.0 MiB/s, 31 objects/s recovering 2024-09-13T14:51:47.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:47 smithi195 ceph-mon[104262]: 9.18 scrub starts 2024-09-13T14:51:47.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:47 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "20"}]': finished 2024-09-13T14:51:47.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:47 smithi195 ceph-mon[104262]: osdmap e390: 8 total, 8 up, 8 in 2024-09-13T14:51:47.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:47 smithi195 ceph-mon[104262]: 9.14 scrub starts 2024-09-13T14:51:47.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:47 smithi195 ceph-mon[104262]: 9.9 scrub starts 2024-09-13T14:51:47.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:47 smithi195 ceph-mon[104262]: 9.18 scrub ok 2024-09-13T14:51:47.561 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:47.583 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:47.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:47.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:47.684 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:47.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:47.707 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:47.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:47.808 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:47.829 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:47.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:47.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:47.931 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:47.953 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:47.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:47.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:47.972 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T14:51:48.054 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:48.086 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:48.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:48.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:48.106 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 42 16 72821 72805 108.327 0 - 0.0088645 2024-09-13T14:51:48.188 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:48.190 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:51:48.210 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:48.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:48.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:48.213 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:48 smithi102 ceph-mon[110183]: 9.14 scrub ok 2024-09-13T14:51:48.213 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:48 smithi102 ceph-mon[110183]: 9.9 scrub ok 2024-09-13T14:51:48.214 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:48 smithi102 ceph-mon[110183]: 9.1 scrub starts 2024-09-13T14:51:48.214 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:48 smithi102 ceph-mon[110183]: osdmap e391: 8 total, 8 up, 8 in 2024-09-13T14:51:48.214 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:48 smithi102 ceph-mon[110183]: 9.0 scrub starts 2024-09-13T14:51:48.214 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:48 smithi102 ceph-mon[110183]: 9.1 scrub ok 2024-09-13T14:51:48.214 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:48 smithi102 ceph-mon[110183]: 9.15 scrub starts 2024-09-13T14:51:48.214 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:48 smithi102 ceph-mon[110183]: 9.0 scrub ok 2024-09-13T14:51:48.214 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:48 smithi102 ceph-mon[115269]: 9.14 scrub ok 2024-09-13T14:51:48.214 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:48 smithi102 ceph-mon[115269]: 9.9 scrub ok 2024-09-13T14:51:48.214 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:48 smithi102 ceph-mon[115269]: 9.1 scrub starts 2024-09-13T14:51:48.214 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:48 smithi102 ceph-mon[115269]: osdmap e391: 8 total, 8 up, 8 in 2024-09-13T14:51:48.215 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:48 smithi102 ceph-mon[115269]: 9.0 scrub starts 2024-09-13T14:51:48.215 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:48 smithi102 ceph-mon[115269]: 9.1 scrub ok 2024-09-13T14:51:48.215 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:48 smithi102 ceph-mon[115269]: 9.15 scrub starts 2024-09-13T14:51:48.215 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:48 smithi102 ceph-mon[115269]: 9.0 scrub ok 2024-09-13T14:51:48.311 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:48.345 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:48.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:48.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:48.447 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:48.490 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:48.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:48.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:48.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:48 smithi195 ceph-mon[104262]: 9.14 scrub ok 2024-09-13T14:51:48.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:48 smithi195 ceph-mon[104262]: 9.9 scrub ok 2024-09-13T14:51:48.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:48 smithi195 ceph-mon[104262]: 9.1 scrub starts 2024-09-13T14:51:48.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:48 smithi195 ceph-mon[104262]: osdmap e391: 8 total, 8 up, 8 in 2024-09-13T14:51:48.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:48 smithi195 ceph-mon[104262]: 9.0 scrub starts 2024-09-13T14:51:48.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:48 smithi195 ceph-mon[104262]: 9.1 scrub ok 2024-09-13T14:51:48.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:48 smithi195 ceph-mon[104262]: 9.15 scrub starts 2024-09-13T14:51:48.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:48 smithi195 ceph-mon[104262]: 9.0 scrub ok 2024-09-13T14:51:48.592 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:48.631 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:48.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:48.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:48.733 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:48.761 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:48.768 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:48.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:48.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:48.871 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:48.910 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:48.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:48.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:49.012 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:49.050 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:49.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:49.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:49.106 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 43 15 73899 73884 107.376 33.7188 0.0174712 0.00930807 2024-09-13T14:51:49.152 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:49.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:49 smithi102 ceph-mon[110183]: 9.15 scrub ok 2024-09-13T14:51:49.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:49 smithi102 ceph-mon[110183]: pgmap v1335: 119 pgs: 1 peering, 1 active+recovering+undersized+remapped, 117 active+clean; 4.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 30 MiB/s wr, 474 op/s; 6207/214788 objects misplaced (2.890%); 2.0 MiB/s, 31 objects/s recovering 2024-09-13T14:51:49.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:49 smithi102 ceph-mon[110183]: 9.2 scrub starts 2024-09-13T14:51:49.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:49 smithi102 ceph-mon[110183]: osdmap e392: 8 total, 8 up, 8 in 2024-09-13T14:51:49.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:49 smithi102 ceph-mon[110183]: 9.2 scrub ok 2024-09-13T14:51:49.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:49 smithi102 ceph-mon[110183]: 9.5 scrub starts 2024-09-13T14:51:49.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:49 smithi102 ceph-mon[115269]: 9.15 scrub ok 2024-09-13T14:51:49.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:49 smithi102 ceph-mon[115269]: pgmap v1335: 119 pgs: 1 peering, 1 active+recovering+undersized+remapped, 117 active+clean; 4.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 30 MiB/s wr, 474 op/s; 6207/214788 objects misplaced (2.890%); 2.0 MiB/s, 31 objects/s recovering 2024-09-13T14:51:49.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:49 smithi102 ceph-mon[115269]: 9.2 scrub starts 2024-09-13T14:51:49.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:49 smithi102 ceph-mon[115269]: osdmap e392: 8 total, 8 up, 8 in 2024-09-13T14:51:49.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:49 smithi102 ceph-mon[115269]: 9.2 scrub ok 2024-09-13T14:51:49.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:49 smithi102 ceph-mon[115269]: 9.5 scrub starts 2024-09-13T14:51:49.174 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:49.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:49.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:49.276 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:49.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:49.320 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:49.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:49.421 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:49.460 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:49.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:49.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:49.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:49 smithi195 ceph-mon[104262]: 9.15 scrub ok 2024-09-13T14:51:49.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:49 smithi195 ceph-mon[104262]: pgmap v1335: 119 pgs: 1 peering, 1 active+recovering+undersized+remapped, 117 active+clean; 4.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 30 MiB/s wr, 474 op/s; 6207/214788 objects misplaced (2.890%); 2.0 MiB/s, 31 objects/s recovering 2024-09-13T14:51:49.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:49 smithi195 ceph-mon[104262]: 9.2 scrub starts 2024-09-13T14:51:49.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:49 smithi195 ceph-mon[104262]: osdmap e392: 8 total, 8 up, 8 in 2024-09-13T14:51:49.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:49 smithi195 ceph-mon[104262]: 9.2 scrub ok 2024-09-13T14:51:49.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:49 smithi195 ceph-mon[104262]: 9.5 scrub starts 2024-09-13T14:51:49.556 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 6, 7, 5, 2, 1] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T14:51:49.557 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-13T14:51:49.562 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:49.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:49.605 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:49.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:49.706 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:49.732 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:49.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:49.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:49.834 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:49.867 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:49.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:49.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:49.969 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:50.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:50.009 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:50.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:50.106 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 44 16 75193 75177 106.772 80.8125 0.00574452 0.00936109 2024-09-13T14:51:50.110 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:50.148 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:50.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:50.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:50.250 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:50.290 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:50.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:50.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:50.392 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:50.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:50 smithi102 ceph-mon[115269]: 9.5 scrub ok 2024-09-13T14:51:50.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:50 smithi102 ceph-mon[115269]: 9.8 scrub starts 2024-09-13T14:51:50.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:50 smithi102 ceph-mon[110183]: 9.5 scrub ok 2024-09-13T14:51:50.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:50 smithi102 ceph-mon[110183]: 9.8 scrub starts 2024-09-13T14:51:50.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:50.427 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:50.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:50.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:50 smithi195 ceph-mon[104262]: 9.5 scrub ok 2024-09-13T14:51:50.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:50 smithi195 ceph-mon[104262]: 9.8 scrub starts 2024-09-13T14:51:50.528 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:50.564 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:50.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:50.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:50.666 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:50.697 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:50.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:50.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:50.799 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:50.834 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:50.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:50.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:50.936 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:50.969 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:50.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:50.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:51.071 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:51.106 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 45 16 76641 76625 106.411 90.5 0.0144892 0.00939243 2024-09-13T14:51:51.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:51.107 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:51.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:51.209 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:51.241 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:51.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:51.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:51.343 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:51.383 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:51.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:51.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:51.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:51 smithi102 ceph-mon[110183]: pgmap v1337: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 1 peering, 115 active+clean; 4.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 4430/218985 objects degraded (2.023%); 5103/218985 objects misplaced (2.330%); 4.9 MiB/s, 79 objects/s recovering 2024-09-13T14:51:51.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:51 smithi102 ceph-mon[110183]: 9.8 scrub ok 2024-09-13T14:51:51.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:51 smithi102 ceph-mon[110183]: Health check failed: Degraded data redundancy: 4430/218985 objects degraded (2.023%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:51:51.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:51 smithi102 ceph-mon[115269]: pgmap v1337: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 1 peering, 115 active+clean; 4.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 4430/218985 objects degraded (2.023%); 5103/218985 objects misplaced (2.330%); 4.9 MiB/s, 79 objects/s recovering 2024-09-13T14:51:51.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:51 smithi102 ceph-mon[115269]: 9.8 scrub ok 2024-09-13T14:51:51.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:51 smithi102 ceph-mon[115269]: Health check failed: Degraded data redundancy: 4430/218985 objects degraded (2.023%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:51:51.485 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:51.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:51 smithi195 ceph-mon[104262]: pgmap v1337: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 1 peering, 115 active+clean; 4.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 4430/218985 objects degraded (2.023%); 5103/218985 objects misplaced (2.330%); 4.9 MiB/s, 79 objects/s recovering 2024-09-13T14:51:51.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:51 smithi195 ceph-mon[104262]: 9.8 scrub ok 2024-09-13T14:51:51.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:51 smithi195 ceph-mon[104262]: Health check failed: Degraded data redundancy: 4430/218985 objects degraded (2.023%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:51:51.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:51.523 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:51.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:51.624 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:51.661 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:51.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:51.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:51.763 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:51.804 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:51.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:51.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:51.909 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:51.943 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:51.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:51.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:52.046 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:52.068 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:52.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:52.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:52.106 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 46 16 77830 77814 105.713 74.3125 0.00505468 0.00942288 2024-09-13T14:51:52.169 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:52.206 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:52.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:52.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:52.309 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:52.344 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:52.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:52.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:52.446 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:52.480 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:52.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:52.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:52.582 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:52.618 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:52.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:52.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:52.720 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:52.760 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:52.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:52.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:52.862 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:52.902 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:52.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:52.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:53.004 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:53.038 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:53.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:53.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:53.106 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 47 16 79157 79141 105.228 82.9375 0.0234692 0.00949868 2024-09-13T14:51:53.141 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:53.165 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:53.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:53.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:53.268 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:53.299 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:53.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:53.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:53.401 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:53.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:53 smithi102 ceph-mon[110183]: pgmap v1338: 119 pgs: 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 115 active+clean; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 4430/225564 objects degraded (1.964%); 9061/225564 objects misplaced (4.017%); 5.4 MiB/s, 86 objects/s recovering 2024-09-13T14:51:53.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:53 smithi102 ceph-mon[115269]: pgmap v1338: 119 pgs: 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 115 active+clean; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 4430/225564 objects degraded (1.964%); 9061/225564 objects misplaced (4.017%); 5.4 MiB/s, 86 objects/s recovering 2024-09-13T14:51:53.433 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:53.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:53.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:53.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:53 smithi195 ceph-mon[104262]: pgmap v1338: 119 pgs: 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 115 active+clean; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 4430/225564 objects degraded (1.964%); 9061/225564 objects misplaced (4.017%); 5.4 MiB/s, 86 objects/s recovering 2024-09-13T14:51:53.535 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:53.567 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:53.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:53.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:53.669 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:53.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:53.715 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:53.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:53.816 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:53.852 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:53.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:53.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:53.953 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:53.981 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:53.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:53.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:54.083 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:54.106 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 48 16 80413 80397 104.671 78.5 0.00641961 0.00954933 2024-09-13T14:51:54.112 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:54.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:54.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:54.213 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:54.257 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:54.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:54.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:54.359 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:54.391 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:54.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:54.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:54.493 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:54.523 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:54.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:54.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:54.625 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:54.669 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:54.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:54.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:54.770 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:54.793 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:54.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:54.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:54.895 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:54.919 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:54.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:54.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:55.021 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:55.045 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:55.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:55.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:55.106 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 49 16 81155 81139 103.481 46.375 0.289458 0.00960536 2024-09-13T14:51:55.147 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:55.191 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:55.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:55.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:55.294 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:55.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:55.329 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:55.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:55.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:55 smithi102 ceph-mon[110183]: pgmap v1339: 119 pgs: 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 4430/232326 objects degraded (1.907%); 9061/232326 objects misplaced (3.900%); 4.2 MiB/s, 67 objects/s recovering 2024-09-13T14:51:55.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:55 smithi102 ceph-mon[115269]: pgmap v1339: 119 pgs: 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 4430/232326 objects degraded (1.907%); 9061/232326 objects misplaced (3.900%); 4.2 MiB/s, 67 objects/s recovering 2024-09-13T14:51:55.431 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:55.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:55.470 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:55.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:55.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:55 smithi195 ceph-mon[104262]: pgmap v1339: 119 pgs: 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 4430/232326 objects degraded (1.907%); 9061/232326 objects misplaced (3.900%); 4.2 MiB/s, 67 objects/s recovering 2024-09-13T14:51:55.571 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:55.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:55.614 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:55.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:55.715 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:55.737 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:55.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:55.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:55.839 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:55.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:55.888 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:55.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:55.989 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:56.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:56.028 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:56.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:56.106 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 50 16 82313 82297 102.859 72.375 0.0102952 0.00971752 2024-09-13T14:51:56.130 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:56.168 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:56.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:56.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:56.270 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:56.309 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:56.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:56.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:56.410 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:56.446 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:56.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:56.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:56.548 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:56.582 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:56.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:56.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:56.684 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:56.717 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:56.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:56.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:56.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:51:56.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:51:56.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:56 smithi195 ceph-mon[104262]: pgmap v1340: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 4430/235317 objects degraded (1.883%); 8455/235317 objects misplaced (3.593%); 5.2 MiB/s, 82 objects/s recovering 2024-09-13T14:51:56.819 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:56.857 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:56.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:56.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:56.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:51:56.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:51:56.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:56 smithi102 ceph-mon[110183]: pgmap v1340: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 4430/235317 objects degraded (1.883%); 8455/235317 objects misplaced (3.593%); 5.2 MiB/s, 82 objects/s recovering 2024-09-13T14:51:56.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:51:56.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:51:56.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:56 smithi102 ceph-mon[115269]: pgmap v1340: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 4430/235317 objects degraded (1.883%); 8455/235317 objects misplaced (3.593%); 5.2 MiB/s, 82 objects/s recovering 2024-09-13T14:51:56.920 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:51:56 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:51:56] "GET /metrics HTTP/1.1" 200 36531 "" "Prometheus/2.51.0" 2024-09-13T14:51:56.959 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:57.000 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:57.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:57.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:57.102 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:57.106 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 51 16 83612 83596 102.434 81.1875 0.00236839 0.00975332 2024-09-13T14:51:57.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:57.129 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:57.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:57.230 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:57.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:57.269 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:57.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:57.370 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:57.412 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:57.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:57.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:57.514 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:57.551 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:57.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:57.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:57.654 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:57.689 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:57.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:57.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:57.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:57 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 4430/235317 objects degraded (1.883%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:51:57.791 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:57.826 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:57.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:57.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:57.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:57 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 4430/235317 objects degraded (1.883%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:51:57.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:57 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 4430/235317 objects degraded (1.883%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:51:57.928 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:57.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:57.963 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:57.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:58.064 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:58.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:58.104 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:58.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:58.108 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 52 16 84985 84969 102.114 85.8125 0.00635878 0.00978813 2024-09-13T14:51:58.205 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:58.246 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:58.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:58.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:58.348 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:58.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:58.375 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:58.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:58.476 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:58.517 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:58.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:58.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:58.621 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:58.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:58.645 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:58.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:58.747 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:58.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:51:58 smithi195 ceph-mon[104262]: pgmap v1341: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 4430/243258 objects degraded (1.821%); 7837/243258 objects misplaced (3.222%); 6.4 MiB/s, 102 objects/s recovering 2024-09-13T14:51:58.788 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:58.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:58.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:58.890 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:58.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:51:58 smithi102 ceph-mon[110183]: pgmap v1341: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 4430/243258 objects degraded (1.821%); 7837/243258 objects misplaced (3.222%); 6.4 MiB/s, 102 objects/s recovering 2024-09-13T14:51:58.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:51:58 smithi102 ceph-mon[115269]: pgmap v1341: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 4430/243258 objects degraded (1.821%); 7837/243258 objects misplaced (3.222%); 6.4 MiB/s, 102 objects/s recovering 2024-09-13T14:51:58.925 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:58.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:58.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:59.030 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:59.062 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:59.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:59.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:59.107 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 53 16 86158 86142 101.571 73.3125 0.00323903 0.00983313 2024-09-13T14:51:59.163 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:59.196 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:59.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:59.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:59.298 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:59.337 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:59.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:59.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:59.439 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:59.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:59.476 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:59.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:59.577 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:59.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:59.610 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:51:59.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:59.712 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:59.749 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:59.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:59.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:59.851 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:59.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:59.887 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:51:59.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:59.988 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:00.021 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:00.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:00.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:00.107 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 54 16 87457 87441 101.193 81.1875 0.0105371 0.00987721 2024-09-13T14:52:00.123 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:00.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:00.165 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:00.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:00.266 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:00.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:00.301 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:00.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:00.328 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:00.402 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:00.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:00.447 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:00.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:00.548 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:01.033 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:01.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:01.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:01.107 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 55 16 88656 88640 100.715 74.9375 0.00786716 0.00992369 2024-09-13T14:52:01.136 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:01.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:00 smithi102 ceph-mon[115269]: pgmap v1342: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 5.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 4430/252516 objects degraded (1.754%); 7108/252516 objects misplaced (2.815%); 6.8 MiB/s, 109 objects/s recovering 2024-09-13T14:52:01.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:00 smithi102 ceph-mon[110183]: pgmap v1342: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 5.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 4430/252516 objects degraded (1.754%); 7108/252516 objects misplaced (2.815%); 6.8 MiB/s, 109 objects/s recovering 2024-09-13T14:52:01.171 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:01.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:01.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:01.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:00 smithi195 ceph-mon[104262]: pgmap v1342: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 5.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 4430/252516 objects degraded (1.754%); 7108/252516 objects misplaced (2.815%); 6.8 MiB/s, 109 objects/s recovering 2024-09-13T14:52:01.273 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:01.308 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:01.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:01.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:01.410 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:01.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:01.445 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:01.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:01.547 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:01.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:01.577 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:01.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:01.681 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:01.721 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:01.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:01.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:01.823 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:01.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:01.860 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:01.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:01.962 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:01.999 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:02.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:02.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:02.102 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:02.107 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 56 16 89712 89696 100.096 66 0.0101491 0.00998593 2024-09-13T14:52:02.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:02.136 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:02.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:02.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:01 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 4430/261606 objects degraded (1.693%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:52:02.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:01 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 4430/261606 objects degraded (1.693%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:52:02.237 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:02.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:01 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 4430/261606 objects degraded (1.693%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:52:02.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:02.271 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:02.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:02.372 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:02.402 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:02.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:02.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:02.504 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:02.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:02.537 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:02.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:02.638 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:02.670 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:02.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:02.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:02.772 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:02.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:02.807 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:02.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:02.908 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:02.948 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:02.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:02.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:03.050 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:03.084 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:03.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:03.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:03.107 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 57 16 90953 90937 99.7001 77.5625 0.0136981 0.0100251 2024-09-13T14:52:03.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:02 smithi102 ceph-mon[110183]: pgmap v1343: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 39 MiB/s wr, 622 op/s; 4430/261606 objects degraded (1.693%); 6468/261606 objects misplaced (2.472%); 6.4 MiB/s, 101 objects/s recovering 2024-09-13T14:52:03.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:02 smithi102 ceph-mon[115269]: pgmap v1343: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 39 MiB/s wr, 622 op/s; 4430/261606 objects degraded (1.693%); 6468/261606 objects misplaced (2.472%); 6.4 MiB/s, 101 objects/s recovering 2024-09-13T14:52:03.185 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:03.223 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:03.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:03.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:03.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:02 smithi195 ceph-mon[104262]: pgmap v1343: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 39 MiB/s wr, 622 op/s; 4430/261606 objects degraded (1.693%); 6468/261606 objects misplaced (2.472%); 6.4 MiB/s, 101 objects/s recovering 2024-09-13T14:52:03.326 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:03.363 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:03.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:03.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:03.465 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:03.495 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:03.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:03.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:03.597 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:03.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:03.631 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:03.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:03.732 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:03.770 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:03.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:03.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:03.872 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:03.907 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:03.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:03.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:04.009 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:04.052 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:04.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:04.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:04.107 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 58 16 92265 92249 99.3948 82 0.0220345 0.0100554 2024-09-13T14:52:04.153 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:04.185 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:04.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:04.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:04.287 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:04.319 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:04.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:04.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:04.420 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:04.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:04.450 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:04.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:04.551 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:04.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:04.590 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:04.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:04.692 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:04.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:04.731 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:04.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:04.832 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:04.866 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:04.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:04.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:04.968 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:04.999 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:05.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:05.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:05.101 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:05.107 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 59 16 93486 93470 99.0033 76.3125 0.00907867 0.0100948 2024-09-13T14:52:05.139 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:05.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:05.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:05.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:04 smithi102 ceph-mon[110183]: pgmap v1344: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 5.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 39 MiB/s wr, 622 op/s; 4430/268182 objects degraded (1.652%); 6468/268182 objects misplaced (2.412%); 5.6 MiB/s, 89 objects/s recovering 2024-09-13T14:52:05.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:04 smithi102 ceph-mon[115269]: pgmap v1344: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 5.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 39 MiB/s wr, 622 op/s; 4430/268182 objects degraded (1.652%); 6468/268182 objects misplaced (2.412%); 5.6 MiB/s, 89 objects/s recovering 2024-09-13T14:52:05.241 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:05.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:04 smithi195 ceph-mon[104262]: pgmap v1344: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 5.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 39 MiB/s wr, 622 op/s; 4430/268182 objects degraded (1.652%); 6468/268182 objects misplaced (2.412%); 5.6 MiB/s, 89 objects/s recovering 2024-09-13T14:52:05.262 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:05.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:05.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:05.364 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:05.397 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:05.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:05.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:05.500 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:05.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:05.537 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:05.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:05.638 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:05.674 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:05.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:05.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:05.775 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:05.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:05.810 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:05.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:05.911 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:05.950 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:05.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:05.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:06.052 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:06.089 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:06.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:06.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:06.107 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:2024-09-13T14:52:06.107339+0000 min lat: 0.00110201 max lat: 2.14011 avg lat: 0.0101369 2024-09-13T14:52:06.107 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:52:06.108 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 60 16 94685 94669 98.6021 74.9375 0.0137725 0.0101369 2024-09-13T14:52:06.191 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:06.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:06.235 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:06.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:06.336 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:06.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:06.363 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:06.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:06.464 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:06.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:06.497 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:06.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:06.598 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:06.632 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:06.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:06.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:06.735 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:06.772 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:06.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:06.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:06.829 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:52:06 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:52:06] "GET /metrics HTTP/1.1" 200 36531 "" "Prometheus/2.51.0" 2024-09-13T14:52:06.874 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:06.915 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:06.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:06.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:07.017 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:07.051 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:07.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:07.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:07.100 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:06 smithi102 ceph-mon[110183]: pgmap v1345: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 5.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 67 MiB/s wr, 1.08k op/s; 4430/271143 objects degraded (1.634%); 5736/271143 objects misplaced (2.115%); 6.9 MiB/s, 109 objects/s recovering 2024-09-13T14:52:07.101 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:06 smithi102 ceph-mon[115269]: pgmap v1345: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 5.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 67 MiB/s wr, 1.08k op/s; 4430/271143 objects degraded (1.634%); 5736/271143 objects misplaced (2.115%); 6.9 MiB/s, 109 objects/s recovering 2024-09-13T14:52:07.109 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 61 16 95871 95855 98.2007 74.125 0.0219573 0.0101783 2024-09-13T14:52:07.154 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:07.187 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:07.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:07.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:07.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:06 smithi195 ceph-mon[104262]: pgmap v1345: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 5.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 67 MiB/s wr, 1.08k op/s; 4430/271143 objects degraded (1.634%); 5736/271143 objects misplaced (2.115%); 6.9 MiB/s, 109 objects/s recovering 2024-09-13T14:52:07.289 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:07.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:07.313 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:07.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:07.414 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:07.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:07.449 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:07.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:07.550 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:07.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:07.573 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:07.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:07.674 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:07.712 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:07.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:07.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:07.814 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:07.850 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:07.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:07.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:07.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:07 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 4430/279678 objects degraded (1.584%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:52:07.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:07 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 4430/279678 objects degraded (1.584%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:52:07.952 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:07.983 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:07.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:07.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:08.085 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:08.107 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 62 16 97136 97120 97.8919 79.0625 0.00885982 0.0102107 2024-09-13T14:52:08.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:08.129 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:08.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:08.219 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:52:08.220 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:52:08.230 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:08.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:07 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 4430/279678 objects degraded (1.584%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:52:08.254 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:08.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:08.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:08.279 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:52:08.280 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:52:08.312 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:52:08.312 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:52:08.343 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:52:08.344 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:52:08.356 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:08.375 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:52:08.375 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:52:08.391 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:08.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:08.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:08.409 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:52:08.409 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:52:08.440 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:52:08.440 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:52:08.483 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:52:08.483 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:52:08.493 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:08.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:08.537 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:08.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:08.638 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:08.663 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:08.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:08.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:08.764 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:08.807 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:08.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:08.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:08.909 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:08.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:08.949 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:08.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:09.050 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:09.085 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:09.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:09.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:09.108 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 63 16 98299 98283 97.4917 72.6875 0.00933404 0.0102527 2024-09-13T14:52:09.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:08 smithi102 ceph-mon[110183]: pgmap v1346: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 5.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s; 4430/279678 objects degraded (1.584%); 5064/279678 objects misplaced (1.811%); 7.6 MiB/s, 120 objects/s recovering 2024-09-13T14:52:09.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:08 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/732533933' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:52:09.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:08 smithi102 ceph-mon[115269]: pgmap v1346: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 5.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s; 4430/279678 objects degraded (1.584%); 5064/279678 objects misplaced (1.811%); 7.6 MiB/s, 120 objects/s recovering 2024-09-13T14:52:09.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:08 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/732533933' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:52:09.186 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:09.221 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:09.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:09.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:09.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:08 smithi195 ceph-mon[104262]: pgmap v1346: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 5.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s; 4430/279678 objects degraded (1.584%); 5064/279678 objects misplaced (1.811%); 7.6 MiB/s, 120 objects/s recovering 2024-09-13T14:52:09.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:08 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/732533933' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:52:09.323 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:09.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:09.357 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:09.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:09.458 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:09.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:09.489 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:09.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:09.590 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:09.616 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:09.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:09.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:09.718 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:09.740 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:09.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:09.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:09.842 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:09.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:09.883 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:09.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:09.938 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:09.983 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:10.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:10.025 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:10.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:10.108 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 64 16 99446 99430 97.0884 71.6875 0.0060741 0.0102937 2024-09-13T14:52:10.126 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:10.157 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:10.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:10.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:10.259 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:10.295 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:10.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:10.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:10.397 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:10.432 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:10.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:10.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:10.534 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:10.573 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:10.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:10.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:10.675 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:10.711 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:10.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:10.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:10.813 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:10.856 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:10.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:10.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:10.958 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:11.000 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:11.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:11.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:11.095 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:10 smithi102 ceph-mon[115269]: pgmap v1347: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 5.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s; 4430/289278 objects degraded (1.531%); 4158/289278 objects misplaced (1.437%); 7.5 MiB/s, 120 objects/s recovering 2024-09-13T14:52:11.095 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:52:11.095 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:52:11.097 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:10 smithi102 ceph-mon[110183]: pgmap v1347: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 5.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s; 4430/289278 objects degraded (1.531%); 4158/289278 objects misplaced (1.437%); 7.5 MiB/s, 120 objects/s recovering 2024-09-13T14:52:11.097 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:52:11.097 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:52:11.102 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:11.109 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 65 16 100701 100685 96.8014 78.4375 0.0188176 0.0103255 2024-09-13T14:52:11.142 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:11.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:11.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:11.244 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:11.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:10 smithi195 ceph-mon[104262]: pgmap v1347: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 5.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s; 4430/289278 objects degraded (1.531%); 4158/289278 objects misplaced (1.437%); 7.5 MiB/s, 120 objects/s recovering 2024-09-13T14:52:11.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:52:11.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:52:11.271 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:11.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:11.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:11.373 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:11.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:11.417 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:11.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:11.518 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:11.576 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:11.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:11.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:11.678 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:11.723 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:11.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:11.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:11.825 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:11.856 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:11.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:11.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:11.957 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:12.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:12.006 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:12.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:12.107 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:12.108 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 66 16 101873 101857 96.4439 73.25 0.0118543 0.0103639 2024-09-13T14:52:12.150 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:12.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:12.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:12.252 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:12.282 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:12.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:12.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:12.384 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:12.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:12.421 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:12.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:12.523 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:12.568 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:12.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:12.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:12.669 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:12.701 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:12.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:12.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:12.798 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:52:12.803 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:12.888 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:12.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:12.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:12.990 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:13.027 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:13.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:13.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:13.108 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 67 16 102942 102926 96.0016 66.8125 0.00975336 0.0104118 2024-09-13T14:52:13.130 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:13.162 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:13.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:13.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:13.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:12 smithi102 ceph-mon[110183]: pgmap v1348: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s; 4430/297861 objects degraded (1.487%); 3434/297861 objects misplaced (1.153%); 8.1 MiB/s, 130 objects/s recovering 2024-09-13T14:52:13.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:12 smithi102 ceph-mon[115269]: pgmap v1348: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s; 4430/297861 objects degraded (1.487%); 3434/297861 objects misplaced (1.153%); 8.1 MiB/s, 130 objects/s recovering 2024-09-13T14:52:13.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:12 smithi195 ceph-mon[104262]: pgmap v1348: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s; 4430/297861 objects degraded (1.487%); 3434/297861 objects misplaced (1.153%); 8.1 MiB/s, 130 objects/s recovering 2024-09-13T14:52:13.265 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:13.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:13.309 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:13.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:13.410 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:13.440 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:13.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:13.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:13.542 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:13.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:13.579 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:13.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:13.680 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:13.721 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:13.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:13.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:13.823 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:13.854 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:52:13.854 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:52:13.868 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:13.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:13.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:13.970 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:14.003 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:14.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:14.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:14.104 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:14.108 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 68 16 104026 104010 95.5861 67.75 0.0168327 0.0104561 2024-09-13T14:52:14.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:14.154 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:14.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:14.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:13 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 4430/304398 objects degraded (1.455%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:52:14.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:13 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 4430/304398 objects degraded (1.455%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:52:14.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:13 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 4430/304398 objects degraded (1.455%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:52:14.256 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:14.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:14.305 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:14.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:14.406 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:14.435 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:14.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:14.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:14.537 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:14.568 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:14.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:14.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:14.670 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:14.698 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:14.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:14.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:14.800 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:14.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:14.841 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:14.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:14.943 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:14.981 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:14.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:14.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:15.083 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:15.108 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 69 16 105183 105167 95.2486 72.3125 0.0170361 0.0104934 2024-09-13T14:52:15.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:15.122 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:15.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:15.223 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:15.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:14 smithi195 ceph-mon[104262]: pgmap v1349: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 6.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 74 MiB/s wr, 1.19k op/s; 4430/304398 objects degraded (1.455%); 3434/304398 objects misplaced (1.128%); 6.5 MiB/s, 103 objects/s recovering 2024-09-13T14:52:15.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:15.261 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:15.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:15.287 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:14 smithi102 ceph-mon[110183]: pgmap v1349: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 6.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 74 MiB/s wr, 1.19k op/s; 4430/304398 objects degraded (1.455%); 3434/304398 objects misplaced (1.128%); 6.5 MiB/s, 103 objects/s recovering 2024-09-13T14:52:15.287 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:14 smithi102 ceph-mon[115269]: pgmap v1349: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 6.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 74 MiB/s wr, 1.19k op/s; 4430/304398 objects degraded (1.455%); 3434/304398 objects misplaced (1.128%); 6.5 MiB/s, 103 objects/s recovering 2024-09-13T14:52:15.363 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:15.411 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:15.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:15.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:15.513 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:15.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:15.567 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:15.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:15.668 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:15.697 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:15.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:15.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:15.767 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T14:52:15.800 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:15.834 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:15.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:15.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:15.936 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:15.974 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:15.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:15.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:15.978 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is unset 2024-09-13T14:52:15.999 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:15 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2847283050' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:52:16.000 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:15 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:52:16.000 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:15 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2847283050' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:52:16.000 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:15 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:52:16.076 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:16.109 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 70 16 106368 106352 94.9459 74.0625 0.0102381 0.0105276 2024-09-13T14:52:16.109 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:16.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:16.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:16.211 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:16.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:15 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2847283050' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:52:16.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:15 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:52:16.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:16.264 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:16.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:16.365 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:16.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:16.422 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:16.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:16.523 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:16.564 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:16.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:16.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:16.667 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:16.692 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:52:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:52:16] "GET /metrics HTTP/1.1" 200 36530 "" "Prometheus/2.51.0" 2024-09-13T14:52:16.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:16.704 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:16.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:16.805 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:16.856 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:16.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:16.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:16.958 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:16.972 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:16 smithi102 ceph-mon[115269]: pgmap v1350: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 6.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 67 MiB/s wr, 1.08k op/s; 4430/307023 objects degraded (1.443%); 2396/307023 objects misplaced (0.780%); 8.3 MiB/s, 132 objects/s recovering 2024-09-13T14:52:16.999 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:16.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:17.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:17.100 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:17.100 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:17.101 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:17.109 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 71 16 107560 107544 94.6575 74.5 0.0212833 0.0105594 2024-09-13T14:52:17.133 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:17.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:17.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:17.235 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:17.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:16 smithi195 ceph-mon[104262]: pgmap v1350: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 6.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 67 MiB/s wr, 1.08k op/s; 4430/307023 objects degraded (1.443%); 2396/307023 objects misplaced (0.780%); 8.3 MiB/s, 132 objects/s recovering 2024-09-13T14:52:17.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:16 smithi195 ceph-mon[104262]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:52:17.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:16 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:52:17.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:16 smithi195 ceph-mon[104262]: osdmap e393: 8 total, 8 up, 8 in 2024-09-13T14:52:17.259 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:16 smithi102 ceph-mon[110183]: pgmap v1350: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 6.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 67 MiB/s wr, 1.08k op/s; 4430/307023 objects degraded (1.443%); 2396/307023 objects misplaced (0.780%); 8.3 MiB/s, 132 objects/s recovering 2024-09-13T14:52:17.259 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:16 smithi102 ceph-mon[110183]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:52:17.259 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:16 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:52:17.260 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:16 smithi102 ceph-mon[110183]: osdmap e393: 8 total, 8 up, 8 in 2024-09-13T14:52:17.260 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:16 smithi102 ceph-mon[115269]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:52:17.260 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:16 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:52:17.260 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:16 smithi102 ceph-mon[115269]: osdmap e393: 8 total, 8 up, 8 in 2024-09-13T14:52:17.275 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:17.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:17.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:17.377 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:17.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:17.422 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:17.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:17.523 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:17.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:17.572 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:17.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:17.673 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:17.715 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:17.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:17.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:17.817 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:17.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:17.864 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:17.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:17.965 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:18.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:18.003 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:18.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:18.104 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:18.110 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 72 16 108715 108699 94.3454 72.1875 0.0110336 0.0105945 2024-09-13T14:52:18.139 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:18.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:18.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:18.241 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:18.276 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:18.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:18.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:18.305 INFO:teuthology.orchestra.run.smithi102.stderr:reweighted osd.0 to 1 (10000) 2024-09-13T14:52:18.378 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:18.412 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:18.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:18.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:18.514 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:18.545 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:18.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:18.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:18.647 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:18.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:18 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:52:18.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:18 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:52:18.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:18 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:52:18.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:18 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:52:18.674 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:18.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:18.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:18.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:18 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:52:18.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:18 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:52:18.776 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:18.815 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:18.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:18.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:18.918 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:18.955 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:18.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:18.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:19.052 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:52:19.057 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:19.090 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:19.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:19.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:19.109 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 73 15 109867 109852 94.0401 72.0625 0.0286714 0.010629 2024-09-13T14:52:19.192 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:19.222 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:19.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:19.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:19.324 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:19.339 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:19 smithi102 ceph-mon[110183]: pgmap v1352: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 73 MiB/s wr, 1.18k op/s; 4430/314961 objects degraded (1.407%); 1646/314961 objects misplaced (0.523%); 8.7 MiB/s, 138 objects/s recovering 2024-09-13T14:52:19.339 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:19 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/85629507' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:52:19.340 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:19 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:52:19.340 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:19 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T14:52:19.340 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:19 smithi102 ceph-mon[110183]: osdmap e394: 8 total, 8 up, 8 in 2024-09-13T14:52:19.340 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:19 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 4430/314961 objects degraded (1.407%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:52:19.340 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:19 smithi102 ceph-mon[110183]: osdmap e395: 8 total, 8 up, 8 in 2024-09-13T14:52:19.340 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:19 smithi102 ceph-mon[115269]: pgmap v1352: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 73 MiB/s wr, 1.18k op/s; 4430/314961 objects degraded (1.407%); 1646/314961 objects misplaced (0.523%); 8.7 MiB/s, 138 objects/s recovering 2024-09-13T14:52:19.340 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:19 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/85629507' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:52:19.340 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:19 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:52:19.340 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:19 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T14:52:19.340 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:19 smithi102 ceph-mon[115269]: osdmap e394: 8 total, 8 up, 8 in 2024-09-13T14:52:19.341 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:19 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 4430/314961 objects degraded (1.407%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:52:19.341 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:19 smithi102 ceph-mon[115269]: osdmap e395: 8 total, 8 up, 8 in 2024-09-13T14:52:19.352 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:19.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:19.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:19.454 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:19.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:19.482 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:19.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:19.583 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:19.605 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:19.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:19.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:19.707 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:19.728 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:19.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:19.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:19.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:19 smithi195 ceph-mon[104262]: pgmap v1352: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 73 MiB/s wr, 1.18k op/s; 4430/314961 objects degraded (1.407%); 1646/314961 objects misplaced (0.523%); 8.7 MiB/s, 138 objects/s recovering 2024-09-13T14:52:19.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:19 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/85629507' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:52:19.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:19 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:52:19.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:19 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T14:52:19.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:19 smithi195 ceph-mon[104262]: osdmap e394: 8 total, 8 up, 8 in 2024-09-13T14:52:19.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:19 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 4430/314961 objects degraded (1.407%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:52:19.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:19 smithi195 ceph-mon[104262]: osdmap e395: 8 total, 8 up, 8 in 2024-09-13T14:52:19.830 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:19.852 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:19.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:19.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:19.954 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:19.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:19.976 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:19.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:20.077 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:20.107 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:20.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:20.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:20.109 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 74 16 110361 110345 93.1856 30.8125 0.638081 0.010713 2024-09-13T14:52:20.209 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:20.244 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:20.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:20.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:20.347 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:20.384 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:20.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:20.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:20.487 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:20.518 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:20.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:20.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:20.620 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:20.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:20 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-13T14:52:20.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:20 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-13T14:52:20.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:20 smithi102 ceph-mon[115269]: osdmap e396: 8 total, 8 up, 8 in 2024-09-13T14:52:20.632 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:20 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-13T14:52:20.632 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:20 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-13T14:52:20.632 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:20 smithi102 ceph-mon[110183]: osdmap e396: 8 total, 8 up, 8 in 2024-09-13T14:52:20.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:20.647 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:20.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:20.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:20 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-13T14:52:20.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:20 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-13T14:52:20.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:20 smithi195 ceph-mon[104262]: osdmap e396: 8 total, 8 up, 8 in 2024-09-13T14:52:20.748 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:20.773 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:20.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:20.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:20.874 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:20.899 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:20.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:20.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:21.001 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:21.023 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:21.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:21.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:21.109 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 75 16 111014 110998 92.4873 40.8125 0.0118195 0.0107798 2024-09-13T14:52:21.125 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:21.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:21.167 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:21.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:21.268 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:21.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:21.302 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:21.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:21.403 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:21.435 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:21.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:21.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:21.536 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:21.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:21.573 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:21.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:21.590 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:21 smithi102 ceph-mon[110183]: pgmap v1355: 119 pgs: 3 active+recovering+undersized+remapped, 116 active+clean; 6.6 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 67 MiB/s wr, 1.08k op/s; 6010/323787 objects misplaced (1.856%); 7.9 MiB/s, 125 objects/s recovering 2024-09-13T14:52:21.590 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:21 smithi102 ceph-mon[110183]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4430/314961 objects degraded (1.407%), 1 pg degraded) 2024-09-13T14:52:21.590 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:21 smithi102 ceph-mon[110183]: Cluster is now healthy 2024-09-13T14:52:21.590 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:21 smithi102 ceph-mon[110183]: osdmap e397: 8 total, 8 up, 8 in 2024-09-13T14:52:21.590 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:21 smithi102 ceph-mon[115269]: pgmap v1355: 119 pgs: 3 active+recovering+undersized+remapped, 116 active+clean; 6.6 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 67 MiB/s wr, 1.08k op/s; 6010/323787 objects misplaced (1.856%); 7.9 MiB/s, 125 objects/s recovering 2024-09-13T14:52:21.590 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:21 smithi102 ceph-mon[115269]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4430/314961 objects degraded (1.407%), 1 pg degraded) 2024-09-13T14:52:21.590 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:21 smithi102 ceph-mon[115269]: Cluster is now healthy 2024-09-13T14:52:21.590 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:21 smithi102 ceph-mon[115269]: osdmap e397: 8 total, 8 up, 8 in 2024-09-13T14:52:21.674 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:21.710 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:21.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:21.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:21.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:21 smithi195 ceph-mon[104262]: pgmap v1355: 119 pgs: 3 active+recovering+undersized+remapped, 116 active+clean; 6.6 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 67 MiB/s wr, 1.08k op/s; 6010/323787 objects misplaced (1.856%); 7.9 MiB/s, 125 objects/s recovering 2024-09-13T14:52:21.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:21 smithi195 ceph-mon[104262]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4430/314961 objects degraded (1.407%), 1 pg degraded) 2024-09-13T14:52:21.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:21 smithi195 ceph-mon[104262]: Cluster is now healthy 2024-09-13T14:52:21.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:21 smithi195 ceph-mon[104262]: osdmap e397: 8 total, 8 up, 8 in 2024-09-13T14:52:21.812 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:21.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:21.852 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:21.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:21.953 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:21.985 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:21.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:21.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:22.087 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:22.109 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 76 16 112247 112231 92.2843 77.0625 0.00499284 0.0108313 2024-09-13T14:52:22.123 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:22.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:22.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:22.225 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:22.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:22.263 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:22.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:22.333 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-13T14:52:22.364 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:22.396 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:22.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:22.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:22.498 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:22.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:22.530 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:22.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:22.631 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:22.658 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:22.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:22.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:22.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:22 smithi102 ceph-mon[110183]: 9.3 scrub starts 2024-09-13T14:52:22.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:22 smithi102 ceph-mon[110183]: 9.3 scrub ok 2024-09-13T14:52:22.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:22 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "22"} : dispatch 2024-09-13T14:52:22.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:22 smithi102 ceph-mon[115269]: 9.3 scrub starts 2024-09-13T14:52:22.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:22 smithi102 ceph-mon[115269]: 9.3 scrub ok 2024-09-13T14:52:22.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:22 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "22"} : dispatch 2024-09-13T14:52:22.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:22 smithi195 ceph-mon[104262]: 9.3 scrub starts 2024-09-13T14:52:22.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:22 smithi195 ceph-mon[104262]: 9.3 scrub ok 2024-09-13T14:52:22.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:22 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "22"} : dispatch 2024-09-13T14:52:22.761 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:22.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:22.793 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:22.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:22.895 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:22.922 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:22.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:22.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:23.024 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:23.049 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:23.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:23.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:23.109 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 77 16 112991 112975 91.6896 46.5 0.0037317 0.0108295 2024-09-13T14:52:23.151 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:23.175 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:23.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:23.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:23.276 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:23.298 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:23.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:23.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:23.317 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:23.400 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:23.422 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:23.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:23.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:23.524 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:23.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:23.548 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:23.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:23.569 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:23 smithi102 ceph-mon[110183]: 9.11 deep-scrub starts 2024-09-13T14:52:23.569 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:23 smithi102 ceph-mon[110183]: pgmap v1358: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 30 MiB/s wr, 473 op/s; 4364/331200 objects misplaced (1.318%); 11 MiB/s, 168 objects/s recovering 2024-09-13T14:52:23.569 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:23 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "22"}]': finished 2024-09-13T14:52:23.569 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:23 smithi102 ceph-mon[110183]: osdmap e398: 8 total, 8 up, 8 in 2024-09-13T14:52:23.569 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:23 smithi102 ceph-mon[115269]: 9.11 deep-scrub starts 2024-09-13T14:52:23.570 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:23 smithi102 ceph-mon[115269]: pgmap v1358: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 30 MiB/s wr, 473 op/s; 4364/331200 objects misplaced (1.318%); 11 MiB/s, 168 objects/s recovering 2024-09-13T14:52:23.570 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:23 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "22"}]': finished 2024-09-13T14:52:23.570 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:23 smithi102 ceph-mon[115269]: osdmap e398: 8 total, 8 up, 8 in 2024-09-13T14:52:23.648 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:23 smithi195 ceph-mon[104262]: 9.11 deep-scrub starts 2024-09-13T14:52:23.649 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:23 smithi195 ceph-mon[104262]: pgmap v1358: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 30 MiB/s wr, 473 op/s; 4364/331200 objects misplaced (1.318%); 11 MiB/s, 168 objects/s recovering 2024-09-13T14:52:23.649 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:23 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "22"}]': finished 2024-09-13T14:52:23.649 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:23 smithi195 ceph-mon[104262]: osdmap e398: 8 total, 8 up, 8 in 2024-09-13T14:52:23.649 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:23.673 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:23.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:23.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:23.775 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:23.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:23.801 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:23.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:23.902 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:23.926 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:23.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:23.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:24.028 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:24.050 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:24.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:24.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:24.110 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 78 16 112999 112983 90.5205 0.5 0.00306048 0.0108292 2024-09-13T14:52:24.153 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:24.176 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:24.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:24.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:24.278 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:24.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:24.302 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:24.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:24.403 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:24.433 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:24.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:24.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:24.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:24 smithi102 ceph-mon[110183]: osdmap e399: 8 total, 8 up, 8 in 2024-09-13T14:52:24.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:24 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "23"} : dispatch 2024-09-13T14:52:24.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:24 smithi102 ceph-mon[115269]: osdmap e399: 8 total, 8 up, 8 in 2024-09-13T14:52:24.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:24 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "23"} : dispatch 2024-09-13T14:52:24.474 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:24 smithi195 ceph-mon[104262]: osdmap e399: 8 total, 8 up, 8 in 2024-09-13T14:52:24.474 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:24 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "23"} : dispatch 2024-09-13T14:52:24.488 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:52:24.488 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:52:24.535 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:24.566 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:24.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:24.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:24.668 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:24.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:24.701 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:24.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:24.802 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:24.835 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:24.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:24.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:24.937 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:24.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:24.967 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:24.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:25.068 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:25.097 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:25.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:25.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:25.110 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 79 16 113318 113302 89.6271 19.9375 0.00286853 0.0111085 2024-09-13T14:52:25.200 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:25.227 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:25.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:25.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:25.328 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:25.354 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:25.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:25.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:25.456 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:25.481 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:25.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:25.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:25.583 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:25.610 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:25.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:25.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:25.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:25 smithi102 ceph-mon[110183]: 9.13 scrub starts 2024-09-13T14:52:25.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:25 smithi102 ceph-mon[110183]: pgmap v1361: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 6.2 MiB/s wr, 99 op/s; 4364/335535 objects misplaced (1.301%); 10 MiB/s, 160 objects/s recovering 2024-09-13T14:52:25.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:25 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "23"}]': finished 2024-09-13T14:52:25.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:25 smithi102 ceph-mon[110183]: osdmap e400: 8 total, 8 up, 8 in 2024-09-13T14:52:25.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:25 smithi102 ceph-mon[115269]: 9.13 scrub starts 2024-09-13T14:52:25.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:25 smithi102 ceph-mon[115269]: pgmap v1361: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 6.2 MiB/s wr, 99 op/s; 4364/335535 objects misplaced (1.301%); 10 MiB/s, 160 objects/s recovering 2024-09-13T14:52:25.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:25 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "23"}]': finished 2024-09-13T14:52:25.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:25 smithi102 ceph-mon[115269]: osdmap e400: 8 total, 8 up, 8 in 2024-09-13T14:52:25.712 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:25.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:25.737 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:25.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:25.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:25 smithi195 ceph-mon[104262]: 9.13 scrub starts 2024-09-13T14:52:25.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:25 smithi195 ceph-mon[104262]: pgmap v1361: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 6.2 MiB/s wr, 99 op/s; 4364/335535 objects misplaced (1.301%); 10 MiB/s, 160 objects/s recovering 2024-09-13T14:52:25.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:25 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "23"}]': finished 2024-09-13T14:52:25.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:25 smithi195 ceph-mon[104262]: osdmap e400: 8 total, 8 up, 8 in 2024-09-13T14:52:25.838 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:25.861 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:25.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:25.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:25.963 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:25.985 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:25.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:25.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:26.087 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:26.110 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:2024-09-13T14:52:26.109751+0000 min lat: 0.00110201 max lat: 2.18409 avg lat: 0.011108 2024-09-13T14:52:26.110 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:52:26.110 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 80 16 113327 113311 88.5138 0.5625 0.0042219 0.011108 2024-09-13T14:52:26.114 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:26.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:26.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:26.216 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:26.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:26.239 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:26.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:26.242 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:52:26.242 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:52:26.243 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:26.278 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:52:26.278 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:52:26.311 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:52:26.311 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:52:26.337 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:52:26.337 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:52:26.341 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:26.362 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:52:26.362 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:52:26.365 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:26.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:26.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:26.392 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:52:26.393 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:52:26.417 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:52:26.417 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:52:26.440 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:52:26.441 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:52:26.467 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:26.507 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:26.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:26.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:26.556 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:52:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:52:26] "GET /metrics HTTP/1.1" 200 36526 "" "Prometheus/2.51.0" 2024-09-13T14:52:26.556 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:26 smithi102 ceph-mon[110183]: osdmap e401: 8 total, 8 up, 8 in 2024-09-13T14:52:26.556 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:52:26.556 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:52:26.557 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "24"} : dispatch 2024-09-13T14:52:26.557 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:26 smithi102 ceph-mon[110183]: 9.10 scrub starts 2024-09-13T14:52:26.557 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:26 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/874673251' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:52:26.557 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "24"}]': finished 2024-09-13T14:52:26.557 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:26 smithi102 ceph-mon[110183]: osdmap e402: 8 total, 8 up, 8 in 2024-09-13T14:52:26.557 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:26 smithi102 ceph-mon[115269]: osdmap e401: 8 total, 8 up, 8 in 2024-09-13T14:52:26.557 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:52:26.557 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:52:26.557 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "24"} : dispatch 2024-09-13T14:52:26.557 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:26 smithi102 ceph-mon[115269]: 9.10 scrub starts 2024-09-13T14:52:26.558 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:26 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/874673251' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:52:26.558 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "24"}]': finished 2024-09-13T14:52:26.558 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:26 smithi102 ceph-mon[115269]: osdmap e402: 8 total, 8 up, 8 in 2024-09-13T14:52:26.608 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:26.639 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:26.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:26.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:26.679 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:26 smithi195 ceph-mon[104262]: osdmap e401: 8 total, 8 up, 8 in 2024-09-13T14:52:26.680 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:52:26.680 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:52:26.680 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "24"} : dispatch 2024-09-13T14:52:26.680 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:26 smithi195 ceph-mon[104262]: 9.10 scrub starts 2024-09-13T14:52:26.680 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:26 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/874673251' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:52:26.680 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "24"}]': finished 2024-09-13T14:52:26.680 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:26 smithi195 ceph-mon[104262]: osdmap e402: 8 total, 8 up, 8 in 2024-09-13T14:52:26.741 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:26.774 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:26.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:26.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:26.876 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:26.903 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:26.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:26.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:27.005 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:27.032 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:27.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:27.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:27.110 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 81 16 113730 113714 87.732 25.1875 0.004628 0.0113508 2024-09-13T14:52:27.133 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:27.156 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:27.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:27.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:27.262 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:27.286 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:27.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:27.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:27.388 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:27.419 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:27.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:27.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:27.522 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:27.552 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:27.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:27.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:27.604 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:27 smithi102 ceph-mon[110183]: pgmap v1364: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 28 MiB/s wr, 449 op/s; 3512/336594 objects misplaced (1.043%); 6.7 MiB/s, 106 objects/s recovering 2024-09-13T14:52:27.605 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:27 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.3", "id": [7, 2]} : dispatch 2024-09-13T14:52:27.605 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:27 smithi102 ceph-mon[110183]: 9.10 scrub starts 2024-09-13T14:52:27.605 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:27 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.3", "id": [7, 2]}]': finished 2024-09-13T14:52:27.605 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:27 smithi102 ceph-mon[110183]: osdmap e403: 8 total, 8 up, 8 in 2024-09-13T14:52:27.605 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:27 smithi102 ceph-mon[115269]: pgmap v1364: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 28 MiB/s wr, 449 op/s; 3512/336594 objects misplaced (1.043%); 6.7 MiB/s, 106 objects/s recovering 2024-09-13T14:52:27.605 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:27 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.3", "id": [7, 2]} : dispatch 2024-09-13T14:52:27.605 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:27 smithi102 ceph-mon[115269]: 9.10 scrub starts 2024-09-13T14:52:27.605 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:27 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.3", "id": [7, 2]}]': finished 2024-09-13T14:52:27.605 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:27 smithi102 ceph-mon[115269]: osdmap e403: 8 total, 8 up, 8 in 2024-09-13T14:52:27.654 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:27.678 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:27.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:27.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:27.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:27 smithi195 ceph-mon[104262]: pgmap v1364: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 28 MiB/s wr, 449 op/s; 3512/336594 objects misplaced (1.043%); 6.7 MiB/s, 106 objects/s recovering 2024-09-13T14:52:27.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:27 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.3", "id": [7, 2]} : dispatch 2024-09-13T14:52:27.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:27 smithi195 ceph-mon[104262]: 9.10 scrub starts 2024-09-13T14:52:27.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:27 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.3", "id": [7, 2]}]': finished 2024-09-13T14:52:27.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:27 smithi195 ceph-mon[104262]: osdmap e403: 8 total, 8 up, 8 in 2024-09-13T14:52:27.780 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:27.806 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:27.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:27.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:27.907 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:27.930 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:27.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:27.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:28.032 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:28.055 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:28.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:28.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:28.110 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 82 16 113806 113790 86.7201 4.75 0.00644594 0.0113493 2024-09-13T14:52:28.157 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:28.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:28.185 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:28.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:28.286 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:28.310 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:28.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:28.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:28.336 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is set 2024-09-13T14:52:28.412 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:28.440 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:28.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:28.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:28.541 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:28.580 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:28.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:28.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:28.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:28 smithi102 ceph-mon[110183]: 9.10 scrub starts 2024-09-13T14:52:28.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:28 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2449567796' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:52:28.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:28 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:52:28.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:28 smithi102 ceph-mon[110183]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:52:28.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:28 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:52:28.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:28 smithi102 ceph-mon[110183]: osdmap e404: 8 total, 8 up, 8 in 2024-09-13T14:52:28.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:28 smithi102 ceph-mon[115269]: 9.10 scrub starts 2024-09-13T14:52:28.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:28 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2449567796' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:52:28.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:28 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:52:28.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:28 smithi102 ceph-mon[115269]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:52:28.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:28 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:52:28.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:28 smithi102 ceph-mon[115269]: osdmap e404: 8 total, 8 up, 8 in 2024-09-13T14:52:28.682 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:28.723 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:28.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:28.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:28.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:28 smithi195 ceph-mon[104262]: 9.10 scrub starts 2024-09-13T14:52:28.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:28 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2449567796' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:52:28.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:28 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:52:28.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:28 smithi195 ceph-mon[104262]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:52:28.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:28 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:52:28.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:28 smithi195 ceph-mon[104262]: osdmap e404: 8 total, 8 up, 8 in 2024-09-13T14:52:28.825 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:28.857 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:28.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:28.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:28.959 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:28.984 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:28.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:28.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:29.086 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:29.110 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 83 16 114148 114132 85.9328 21.375 0.00306793 0.0115808 2024-09-13T14:52:29.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:29.111 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:29.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:29.200 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T14:52:29.212 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:29.267 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:29.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:29.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:29.369 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:29.392 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:29.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:29.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:29.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:29 smithi102 ceph-mon[115269]: pgmap v1367: 119 pgs: 1 unknown, 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 114 active+clean; 6.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 22 MiB/s wr, 346 op/s; 10440/339690 objects misplaced (3.073%); 8.3 MiB/s, 132 objects/s recovering 2024-09-13T14:52:29.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:29 smithi102 ceph-mon[115269]: osdmap e405: 8 total, 8 up, 8 in 2024-09-13T14:52:29.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:29 smithi102 ceph-mon[110183]: pgmap v1367: 119 pgs: 1 unknown, 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 114 active+clean; 6.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 22 MiB/s wr, 346 op/s; 10440/339690 objects misplaced (3.073%); 8.3 MiB/s, 132 objects/s recovering 2024-09-13T14:52:29.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:29 smithi102 ceph-mon[110183]: osdmap e405: 8 total, 8 up, 8 in 2024-09-13T14:52:29.429 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:52:29.494 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:29.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:29.518 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:29.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:29.619 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:29.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:29.677 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:29.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:29.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:29 smithi195 ceph-mon[104262]: pgmap v1367: 119 pgs: 1 unknown, 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 114 active+clean; 6.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 22 MiB/s wr, 346 op/s; 10440/339690 objects misplaced (3.073%); 8.3 MiB/s, 132 objects/s recovering 2024-09-13T14:52:29.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:29 smithi195 ceph-mon[104262]: osdmap e405: 8 total, 8 up, 8 in 2024-09-13T14:52:29.778 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:29.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:29.822 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:29.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:29.923 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:29.967 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:29.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:29.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:30.069 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:30.106 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:30.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:30.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:30.110 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 84 16 114694 114678 85.3161 34.125 0.0118971 0.0117163 2024-09-13T14:52:30.208 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:30.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:30.242 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:30.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:30.344 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:30.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:30.388 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:30.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:30.489 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:30.522 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:30.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:30.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:30.624 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:30.664 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:30.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:30.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:30.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:30 smithi102 ceph-mon[110183]: 9.11 deep-scrub ok 2024-09-13T14:52:30.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:30 smithi102 ceph-mon[110183]: Health check failed: Degraded data redundancy: 7124/341313 objects degraded (2.087%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:52:30.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:30 smithi102 ceph-mon[115269]: 9.11 deep-scrub ok 2024-09-13T14:52:30.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:30 smithi102 ceph-mon[115269]: Health check failed: Degraded data redundancy: 7124/341313 objects degraded (2.087%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:52:30.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:30 smithi195 ceph-mon[104262]: 9.11 deep-scrub ok 2024-09-13T14:52:30.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:30 smithi195 ceph-mon[104262]: Health check failed: Degraded data redundancy: 7124/341313 objects degraded (2.087%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:52:30.767 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:30.802 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:30.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:30.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:30.904 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:30.936 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:30.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:30.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:31.039 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:31.075 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:31.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:31.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:31.110 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 85 16 115882 115866 85.1858 74.25 0.00740311 0.0117339 2024-09-13T14:52:31.178 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:31.214 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:31.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:31.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:31.315 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:31.357 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:31.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:31.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:31.459 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:31.493 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:31.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:31.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:31.595 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:31.624 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:31.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:31.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:31.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:31 smithi102 ceph-mon[110183]: pgmap v1370: 119 pgs: 1 unknown, 1 active+clean+scrubbing+deep, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 112 active+clean; 6.9 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 25 MiB/s wr, 393 op/s; 7124/341313 objects degraded (2.087%); 9372/341313 objects misplaced (2.746%); 10 MiB/s, 159 objects/s recovering 2024-09-13T14:52:31.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:31 smithi102 ceph-mon[115269]: pgmap v1370: 119 pgs: 1 unknown, 1 active+clean+scrubbing+deep, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 112 active+clean; 6.9 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 25 MiB/s wr, 393 op/s; 7124/341313 objects degraded (2.087%); 9372/341313 objects misplaced (2.746%); 10 MiB/s, 159 objects/s recovering 2024-09-13T14:52:31.726 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:31.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:31 smithi195 ceph-mon[104262]: pgmap v1370: 119 pgs: 1 unknown, 1 active+clean+scrubbing+deep, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 112 active+clean; 6.9 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 25 MiB/s wr, 393 op/s; 7124/341313 objects degraded (2.087%); 9372/341313 objects misplaced (2.746%); 10 MiB/s, 159 objects/s recovering 2024-09-13T14:52:31.764 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:31.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:31.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:31.866 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:31.897 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:31.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:31.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:31.998 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:32.072 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:32.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:32.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:32.112 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 86 16 117006 116990 85.0106 70.25 0.0108015 0.0117582 2024-09-13T14:52:32.175 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:32.197 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:32.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:32.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:32.299 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:32.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:32.335 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:32.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:32.436 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:32.470 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:32.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:32.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:32.571 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:32.593 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:32.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:32.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:32.695 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:32.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:32.726 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:32.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:32.827 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:32.863 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:32.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:32.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:32.965 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:33.001 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:33.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:33.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:33.103 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:33.111 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 87 16 117940 117924 84.7044 58.375 0.0106526 0.0118011 2024-09-13T14:52:33.127 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:33.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:33.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:33.229 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:33.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:33.268 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:33.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:33.369 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:33.399 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:33.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:33.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:33.501 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:33.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:33.524 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:33.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:33.625 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:33.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:33.666 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:33.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:33.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:33 smithi102 ceph-mon[110183]: pgmap v1371: 119 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 114 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 14232/345483 objects degraded (4.119%); 8927/345483 objects misplaced (2.584%); 10 MiB/s, 159 objects/s recovering 2024-09-13T14:52:33.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:33 smithi102 ceph-mon[115269]: pgmap v1371: 119 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 114 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 14232/345483 objects degraded (4.119%); 8927/345483 objects misplaced (2.584%); 10 MiB/s, 159 objects/s recovering 2024-09-13T14:52:33.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:33 smithi195 ceph-mon[104262]: pgmap v1371: 119 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 114 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 14232/345483 objects degraded (4.119%); 8927/345483 objects misplaced (2.584%); 10 MiB/s, 159 objects/s recovering 2024-09-13T14:52:33.767 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:33.803 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:33.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:33.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:33.908 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:33.943 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:33.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:33.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:34.045 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:34.083 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:34.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:34.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:34.113 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 88 16 118883 118867 84.4115 58.9375 0.0082001 0.0118418 2024-09-13T14:52:34.185 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:34.222 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:34.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:34.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:34.323 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:34.356 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:34.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:34.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:34.458 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:34.491 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:34.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:34.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:34.593 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:34.626 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:34.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:34.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:34.728 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:34.766 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:34.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:34.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:34.868 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:34.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:34.903 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:34.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:35.004 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:35.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:35.037 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:35.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:35.112 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 89 16 119992 119976 84.2418 69.3125 0.0793346 0.0118649 2024-09-13T14:52:35.138 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:35.177 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:35.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:35.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:35.278 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:35.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:35.313 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:35.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:35.414 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:35.449 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:35.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:35.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:35.551 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:35.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:35.583 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:35.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:35.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:35 smithi102 ceph-mon[110183]: pgmap v1372: 119 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 114 active+clean; 7.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 14232/349806 objects degraded (4.069%); 8927/349806 objects misplaced (2.552%); 7.4 MiB/s, 118 objects/s recovering 2024-09-13T14:52:35.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:35 smithi102 ceph-mon[115269]: pgmap v1372: 119 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 114 active+clean; 7.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 14232/349806 objects degraded (4.069%); 8927/349806 objects misplaced (2.552%); 7.4 MiB/s, 118 objects/s recovering 2024-09-13T14:52:35.684 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:35.729 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:35.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:35.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:35.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:35 smithi195 ceph-mon[104262]: pgmap v1372: 119 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 114 active+clean; 7.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 14232/349806 objects degraded (4.069%); 8927/349806 objects misplaced (2.552%); 7.4 MiB/s, 118 objects/s recovering 2024-09-13T14:52:35.831 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:35.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:35.876 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:35.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:35.977 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:36.014 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:36.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:36.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:36.112 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 90 4 121099 121095 84.0828 69.9375 0.0141688 0.0118881 2024-09-13T14:52:36.116 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:36.138 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:36.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:36.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:36.240 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:36.262 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:36.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:36.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:36.364 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:36.385 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:36.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:36.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:36.487 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:36.509 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:36.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:36.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:36.611 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:36.634 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:36.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:36.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:36.736 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:36.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:36 smithi195 ceph-mon[104262]: pgmap v1373: 119 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 114 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 6.1 MiB/s wr, 98 op/s; 14232/352782 objects degraded (4.034%); 8409/352782 objects misplaced (2.384%); 7.9 MiB/s, 126 objects/s recovering 2024-09-13T14:52:36.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:36 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 14232/352782 objects degraded (4.034%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:52:36.758 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:36.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:36.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:36.860 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:36.883 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:36.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:36.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:36.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:36 smithi102 ceph-mon[110183]: pgmap v1373: 119 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 114 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 6.1 MiB/s wr, 98 op/s; 14232/352782 objects degraded (4.034%); 8409/352782 objects misplaced (2.384%); 7.9 MiB/s, 126 objects/s recovering 2024-09-13T14:52:36.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:36 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 14232/352782 objects degraded (4.034%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:52:36.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:36 smithi102 ceph-mon[115269]: pgmap v1373: 119 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 114 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 6.1 MiB/s wr, 98 op/s; 14232/352782 objects degraded (4.034%); 8409/352782 objects misplaced (2.384%); 7.9 MiB/s, 126 objects/s recovering 2024-09-13T14:52:36.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:36 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 14232/352782 objects degraded (4.034%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:52:36.919 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:52:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:52:36] "GET /metrics HTTP/1.1" 200 36526 "" "Prometheus/2.51.0" 2024-09-13T14:52:36.984 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:37.007 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:37.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:37.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:37.108 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:37.112 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Total time run: 90.0143 2024-09-13T14:52:37.112 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Total writes made: 121099 2024-09-13T14:52:37.112 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Write size: 65536 2024-09-13T14:52:37.112 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Object size: 65536 2024-09-13T14:52:37.112 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Bandwidth (MB/sec): 84.0832 2024-09-13T14:52:37.113 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Stddev Bandwidth: 48.8228 2024-09-13T14:52:37.113 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Max bandwidth (MB/sec): 225.688 2024-09-13T14:52:37.113 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Min bandwidth (MB/sec): 0 2024-09-13T14:52:37.113 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Average IOPS: 1345 2024-09-13T14:52:37.113 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Stddev IOPS: 781.179 2024-09-13T14:52:37.113 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Max IOPS: 3611 2024-09-13T14:52:37.113 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Min IOPS: 0 2024-09-13T14:52:37.113 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Average Latency(s): 0.0118882 2024-09-13T14:52:37.113 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Stddev Latency(s): 0.0578597 2024-09-13T14:52:37.113 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Max latency(s): 2.18409 2024-09-13T14:52:37.113 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Min latency(s): 0.00110201 2024-09-13T14:52:37.117 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Cleaning up (deleting benchmark objects) 2024-09-13T14:52:37.144 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:37.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:37.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:37.246 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:37.290 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:37.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:37.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:37.348 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:37.392 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:37.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:37.430 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:37.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:37.531 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:37.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:37.559 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:37.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:37.660 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:37.697 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:37.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:37.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:37.799 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:37.834 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:37.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:37.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:37.936 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:37.973 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:37.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:37.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:38.075 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:38.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:38.115 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:38.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:38.216 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:38.255 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:38.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:38.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:38.357 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:38.390 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:38.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:38.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:38.491 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:38.514 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:38.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:38.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:38.617 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:38.651 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:38.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:38.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:38.753 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:38.791 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:38.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:38.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:38.893 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:38.927 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:38.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:38.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:39.029 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:39.061 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:39.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:39.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:39.163 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:39.168 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:52:39.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:38 smithi102 ceph-mon[110183]: pgmap v1374: 119 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 114 active+clean; 7.3 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 18 MiB/s wr, 292 op/s; 14232/359481 objects degraded (3.959%); 8126/359481 objects misplaced (2.260%); 7.7 MiB/s, 123 objects/s recovering 2024-09-13T14:52:39.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:38 smithi102 ceph-mon[115269]: pgmap v1374: 119 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 114 active+clean; 7.3 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 18 MiB/s wr, 292 op/s; 14232/359481 objects degraded (3.959%); 8126/359481 objects misplaced (2.260%); 7.7 MiB/s, 123 objects/s recovering 2024-09-13T14:52:39.185 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:39.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:39.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:39.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:38 smithi195 ceph-mon[104262]: pgmap v1374: 119 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 114 active+clean; 7.3 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 18 MiB/s wr, 292 op/s; 14232/359481 objects degraded (3.959%); 8126/359481 objects misplaced (2.260%); 7.7 MiB/s, 123 objects/s recovering 2024-09-13T14:52:39.287 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:39.334 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:39.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:39.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:39.436 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:39.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:39.479 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:39.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:39.580 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:39.602 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:39.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:39.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:39.704 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:39.725 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:39.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:39.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:39.827 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:39.867 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:39.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:39.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:39.970 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:40.000 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:40.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:40.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:40.102 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:40.134 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:40.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:40.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:40.236 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:40.258 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:40.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:40.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:40.360 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:40.397 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:40.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:40.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:40.499 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:40.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:40.521 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:40.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:40.622 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:40.658 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:40.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:40.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:40.761 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:40.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:40.797 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:40.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:40.898 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:40.919 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:40.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:40.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:41.021 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:41.061 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:41.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:41.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:41.103 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:40 smithi102 ceph-mon[110183]: pgmap v1375: 119 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 114 active+clean; 7.4 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 23 MiB/s wr, 410 op/s; 14232/361515 objects degraded (3.937%); 8126/361515 objects misplaced (2.248%); 7.0 MiB/s, 111 objects/s recovering 2024-09-13T14:52:41.103 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:40 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:52:41.103 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:40 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:52:41.104 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:40 smithi102 ceph-mon[115269]: pgmap v1375: 119 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 114 active+clean; 7.4 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 23 MiB/s wr, 410 op/s; 14232/361515 objects degraded (3.937%); 8126/361515 objects misplaced (2.248%); 7.0 MiB/s, 111 objects/s recovering 2024-09-13T14:52:41.104 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:40 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:52:41.104 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:40 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:52:41.163 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:41.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:41.203 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:41.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:41.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:40 smithi195 ceph-mon[104262]: pgmap v1375: 119 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 114 active+clean; 7.4 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 23 MiB/s wr, 410 op/s; 14232/361515 objects degraded (3.937%); 8126/361515 objects misplaced (2.248%); 7.0 MiB/s, 111 objects/s recovering 2024-09-13T14:52:41.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:40 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:52:41.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:40 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:52:41.305 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:41.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:41.344 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:41.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:41.445 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:41.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:41.481 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:41.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:41.587 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:41.634 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:41.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:41.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:41.736 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:41.757 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:41.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:41.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:41.859 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:41.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:41.898 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:41.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:41.999 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:42.037 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:42.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:42.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:42.139 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:42.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:41 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 13944/354669 objects degraded (3.932%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:52:42.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:41 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 13944/354669 objects degraded (3.932%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:52:42.177 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:42.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:42.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:42.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:41 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 13944/354669 objects degraded (3.932%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:52:42.279 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:42.312 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:42.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:42.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:42.414 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:42.454 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:42.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:42.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:42.556 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:42.597 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:42.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:42.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:42.699 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:42.738 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:42.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:42.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:42.840 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:42.852 INFO:teuthology.orchestra.run.smithi102.stderr:reweighted osd.3 to 1 (10000) 2024-09-13T14:52:42.894 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:42.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:42.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:42.996 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:43.030 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:43.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:43.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:43.132 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:43.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:43.165 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:43.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:43.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:42 smithi102 ceph-mon[110183]: pgmap v1376: 119 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 114 active+clean; 7.2 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 85 B/s rd, 20 MiB/s wr, 567 op/s; 13944/354669 objects degraded (3.932%); 6880/354669 objects misplaced (1.940%); 7.2 MiB/s, 115 objects/s recovering 2024-09-13T14:52:43.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:42 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2533533779' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:52:43.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:42 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:52:43.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:42 smithi102 ceph-mon[115269]: pgmap v1376: 119 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 114 active+clean; 7.2 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 85 B/s rd, 20 MiB/s wr, 567 op/s; 13944/354669 objects degraded (3.932%); 6880/354669 objects misplaced (1.940%); 7.2 MiB/s, 115 objects/s recovering 2024-09-13T14:52:43.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:42 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2533533779' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:52:43.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:42 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:52:43.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:42 smithi195 ceph-mon[104262]: pgmap v1376: 119 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 114 active+clean; 7.2 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 85 B/s rd, 20 MiB/s wr, 567 op/s; 13944/354669 objects degraded (3.932%); 6880/354669 objects misplaced (1.940%); 7.2 MiB/s, 115 objects/s recovering 2024-09-13T14:52:43.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:42 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2533533779' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:52:43.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:42 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:52:43.266 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:43.303 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:43.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:43.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:43.405 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:43.444 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:43.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:43.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:43.546 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:43.582 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:43.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:43.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:43.684 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:43.718 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:43.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:43.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:43.820 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:43.856 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:43.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:43.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:43.958 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:43.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:43.993 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:43.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:44.094 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:44.123 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:44.150 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:44.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:44.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:44.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:43 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T14:52:44.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:43 smithi102 ceph-mon[115269]: osdmap e406: 8 total, 8 up, 8 in 2024-09-13T14:52:44.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:43 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T14:52:44.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:43 smithi102 ceph-mon[110183]: osdmap e406: 8 total, 8 up, 8 in 2024-09-13T14:52:44.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:43 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T14:52:44.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:43 smithi195 ceph-mon[104262]: osdmap e406: 8 total, 8 up, 8 in 2024-09-13T14:52:44.252 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:44.301 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:44.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:44.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:44.403 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:44.438 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:44.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:44.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:44.540 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:44.586 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:44.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:44.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:44.688 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:44.725 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:44.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:44.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:44.827 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:44.861 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:44.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:44.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:44.893 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:52:44.893 INFO:teuthology.orchestra.run.smithi102.stdout: "size": 20, 2024-09-13T14:52:44.893 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 600, 2024-09-13T14:52:44.893 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [ 2024-09-13T14:52:44.893 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.893 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:133957 8.7 8:e8800331:::benchmark_data_smithi102_313275_object3370:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e337)", 2024-09-13T14:52:44.893 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:47:47.859677+0000", 2024-09-13T14:52:44.893 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 297.02929429199997, 2024-09-13T14:52:44.894 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4148750569999999, 2024-09-13T14:52:44.894 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:52:44.894 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:52:44.894 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:44.894 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:52:44.894 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:52:44.894 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:52:44.894 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 133957 2024-09-13T14:52:44.894 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.894 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:52:44.895 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.895 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:52:44.895 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859677+0000", 2024-09-13T14:52:44.895 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:52:44.895 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.895 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.895 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:52:44.895 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859677+0000", 2024-09-13T14:52:44.895 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:52:44.895 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.895 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.895 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:52:44.895 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859679+0000", 2024-09-13T14:52:44.896 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.5740000000000001e-06 2024-09-13T14:52:44.896 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.896 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.896 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:52:44.896 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859684+0000", 2024-09-13T14:52:44.896 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.6460000000000001e-06 2024-09-13T14:52:44.896 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.896 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.896 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:52:44.896 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859701+0000", 2024-09-13T14:52:44.896 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.6807e-05 2024-09-13T14:52:44.897 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.897 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.897 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:52:44.897 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859711+0000", 2024-09-13T14:52:44.897 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0729e-05 2024-09-13T14:52:44.897 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.897 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.897 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:52:44.897 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859782+0000", 2024-09-13T14:52:44.897 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.1051999999999997e-05 2024-09-13T14:52:44.897 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.897 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.897 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op must wait for map", 2024-09-13T14:52:44.898 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859789+0000", 2024-09-13T14:52:44.898 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.1380000000000001e-06 2024-09-13T14:52:44.898 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.898 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.898 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:52:44.898 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.863617+0000", 2024-09-13T14:52:44.898 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0038276289999999999 2024-09-13T14:52:44.898 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.898 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.898 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:52:44.898 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.863626+0000", 2024-09-13T14:52:44.898 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.8699999999999998e-06 2024-09-13T14:52:44.899 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.899 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.899 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:52:44.899 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.258622+0000", 2024-09-13T14:52:44.899 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3949958790000001 2024-09-13T14:52:44.899 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.899 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.899 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:52:44.899 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.261508+0000", 2024-09-13T14:52:44.900 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0028864149999999998 2024-09-13T14:52:44.900 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.900 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.900 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,3,6", 2024-09-13T14:52:44.900 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.261752+0000", 2024-09-13T14:52:44.900 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00024399600000000001 2024-09-13T14:52:44.900 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.900 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.900 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:52:44.900 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.271583+0000", 2024-09-13T14:52:44.900 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0098305049999999998 2024-09-13T14:52:44.900 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.900 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.901 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:44.901 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.271621+0000", 2024-09-13T14:52:44.901 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.7920999999999998e-05 2024-09-13T14:52:44.901 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.901 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.901 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:44.901 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.271687+0000", 2024-09-13T14:52:44.901 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.6130000000000006e-05 2024-09-13T14:52:44.901 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.901 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.901 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:44.901 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.274325+0000", 2024-09-13T14:52:44.901 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.002638404 2024-09-13T14:52:44.902 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.902 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.902 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:52:44.902 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.274389+0000", 2024-09-13T14:52:44.902 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.3415000000000002e-05 2024-09-13T14:52:44.902 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.902 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.902 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:52:44.903 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.274553+0000", 2024-09-13T14:52:44.903 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00016394700000000001 2024-09-13T14:52:44.903 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:52:44.903 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:52:44.903 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:52:44.903 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.903 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.903 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:133967 8.a 8:5f281cc1:::benchmark_data_smithi102_313275_object3380:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e337)", 2024-09-13T14:52:44.903 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:47:47.859736+0000", 2024-09-13T14:52:44.903 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 297.02923539800003, 2024-09-13T14:52:44.903 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4400547020000001, 2024-09-13T14:52:44.903 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:52:44.903 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:52:44.903 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:44.904 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:52:44.904 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:52:44.904 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:52:44.904 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 133967 2024-09-13T14:52:44.904 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.904 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:52:44.904 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.904 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:52:44.904 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859736+0000", 2024-09-13T14:52:44.904 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:52:44.904 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.904 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.904 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:52:44.905 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859736+0000", 2024-09-13T14:52:44.905 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:52:44.905 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.905 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.905 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:52:44.905 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859737+0000", 2024-09-13T14:52:44.905 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.6000000000000004e-07 2024-09-13T14:52:44.905 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.905 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.905 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:52:44.906 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859740+0000", 2024-09-13T14:52:44.906 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.6730000000000001e-06 2024-09-13T14:52:44.906 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.906 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.906 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:52:44.906 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859756+0000", 2024-09-13T14:52:44.906 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.6129000000000001e-05 2024-09-13T14:52:44.906 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.906 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.906 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:52:44.906 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859762+0000", 2024-09-13T14:52:44.906 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.1489999999999999e-06 2024-09-13T14:52:44.907 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.907 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.907 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:52:44.907 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859843+0000", 2024-09-13T14:52:44.907 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.1382000000000001e-05 2024-09-13T14:52:44.907 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.907 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.907 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op must wait for map", 2024-09-13T14:52:44.907 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859856+0000", 2024-09-13T14:52:44.907 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2598e-05 2024-09-13T14:52:44.907 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.907 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.907 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:52:44.908 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.862672+0000", 2024-09-13T14:52:44.908 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0028164650000000002 2024-09-13T14:52:44.908 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.908 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.908 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:52:44.908 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.862682+0000", 2024-09-13T14:52:44.908 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.4639999999999995e-06 2024-09-13T14:52:44.908 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.908 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.908 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:52:44.908 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.283265+0000", 2024-09-13T14:52:44.908 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.420583427 2024-09-13T14:52:44.908 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.909 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.909 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:52:44.909 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.283382+0000", 2024-09-13T14:52:44.909 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000116984 2024-09-13T14:52:44.909 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.910 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.910 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,3,5", 2024-09-13T14:52:44.910 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.283527+0000", 2024-09-13T14:52:44.910 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00014459600000000001 2024-09-13T14:52:44.910 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.910 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.910 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:44.911 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.291391+0000", 2024-09-13T14:52:44.911 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0078640450000000001 2024-09-13T14:52:44.911 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.911 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.911 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:44.911 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.291431+0000", 2024-09-13T14:52:44.911 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.0323e-05 2024-09-13T14:52:44.911 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.911 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.911 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:44.911 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.299553+0000", 2024-09-13T14:52:44.911 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0081221090000000006 2024-09-13T14:52:44.911 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.912 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.912 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:52:44.912 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.299567+0000", 2024-09-13T14:52:44.912 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4011e-05 2024-09-13T14:52:44.912 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.912 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.912 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:52:44.912 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.299679+0000", 2024-09-13T14:52:44.912 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00011214600000000001 2024-09-13T14:52:44.912 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.912 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.912 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:52:44.912 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.299791+0000", 2024-09-13T14:52:44.913 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000111641 2024-09-13T14:52:44.913 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:52:44.913 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:52:44.913 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:52:44.913 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.913 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.913 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:133969 8.7 8:e4b3c894:::benchmark_data_smithi102_313275_object3382:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e337)", 2024-09-13T14:52:44.913 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:47:47.859781+0000", 2024-09-13T14:52:44.913 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 297.02919091400003, 2024-09-13T14:52:44.913 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.430809029, 2024-09-13T14:52:44.913 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:52:44.913 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:52:44.913 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:44.914 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:52:44.914 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:52:44.914 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:52:44.914 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 133969 2024-09-13T14:52:44.914 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.914 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:52:44.914 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.914 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:52:44.914 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859781+0000", 2024-09-13T14:52:44.914 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:52:44.914 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.914 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.914 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:52:44.914 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859781+0000", 2024-09-13T14:52:44.915 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:52:44.915 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.915 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.915 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:52:44.915 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859781+0000", 2024-09-13T14:52:44.915 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.1399999999999997e-07 2024-09-13T14:52:44.915 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.915 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.915 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:52:44.915 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859784+0000", 2024-09-13T14:52:44.915 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.65e-06 2024-09-13T14:52:44.915 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.915 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.916 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:52:44.916 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859791+0000", 2024-09-13T14:52:44.916 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.3139999999999998e-06 2024-09-13T14:52:44.916 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.916 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.916 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:52:44.916 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859797+0000", 2024-09-13T14:52:44.916 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.4909999999999996e-06 2024-09-13T14:52:44.916 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.916 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.916 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:52:44.916 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859848+0000", 2024-09-13T14:52:44.916 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.1174999999999997e-05 2024-09-13T14:52:44.917 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.917 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.917 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting_for_map not empty", 2024-09-13T14:52:44.917 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859860+0000", 2024-09-13T14:52:44.917 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1762e-05 2024-09-13T14:52:44.917 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.917 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.917 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:52:44.917 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.863650+0000", 2024-09-13T14:52:44.917 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0037906820000000001 2024-09-13T14:52:44.917 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.917 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.917 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:52:44.918 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.863657+0000", 2024-09-13T14:52:44.918 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.8460000000000002e-06 2024-09-13T14:52:44.918 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.918 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.918 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:52:44.918 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.274593+0000", 2024-09-13T14:52:44.918 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.410936153 2024-09-13T14:52:44.918 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.918 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.918 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:52:44.918 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.274720+0000", 2024-09-13T14:52:44.918 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00012619899999999999 2024-09-13T14:52:44.919 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.919 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.919 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,3,6", 2024-09-13T14:52:44.919 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.274846+0000", 2024-09-13T14:52:44.919 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00012598399999999999 2024-09-13T14:52:44.919 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.919 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.920 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:44.920 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.281413+0000", 2024-09-13T14:52:44.920 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0065674490000000004 2024-09-13T14:52:44.920 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.920 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.920 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:52:44.920 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.281586+0000", 2024-09-13T14:52:44.920 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00017314899999999999 2024-09-13T14:52:44.920 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.921 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.921 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:44.921 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.289879+0000", 2024-09-13T14:52:44.921 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0082928470000000008 2024-09-13T14:52:44.921 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.921 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.921 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:44.921 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.290308+0000", 2024-09-13T14:52:44.921 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00042940799999999999 2024-09-13T14:52:44.921 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.921 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.921 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:52:44.921 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.290452+0000", 2024-09-13T14:52:44.922 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00014385299999999999 2024-09-13T14:52:44.922 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.922 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.922 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:52:44.922 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.290590+0000", 2024-09-13T14:52:44.922 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000137553 2024-09-13T14:52:44.922 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:52:44.922 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:52:44.922 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:52:44.922 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.922 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.922 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:133975 8.7 8:e7ad9da5:::benchmark_data_smithi102_313275_object3388:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e337)", 2024-09-13T14:52:44.922 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:47:47.859806+0000", 2024-09-13T14:52:44.923 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 297.02916588699998, 2024-09-13T14:52:44.923 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.433715885, 2024-09-13T14:52:44.923 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:52:44.923 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:52:44.923 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:44.923 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:52:44.923 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:52:44.923 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:52:44.923 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 133975 2024-09-13T14:52:44.923 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.923 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:52:44.923 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.923 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:52:44.923 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859806+0000", 2024-09-13T14:52:44.924 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:52:44.924 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.924 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.924 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:52:44.924 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859806+0000", 2024-09-13T14:52:44.924 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:52:44.924 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.924 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.924 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:52:44.924 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859806+0000", 2024-09-13T14:52:44.924 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.2800000000000002e-07 2024-09-13T14:52:44.924 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.924 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.925 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:52:44.925 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859808+0000", 2024-09-13T14:52:44.925 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.0719999999999998e-06 2024-09-13T14:52:44.925 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.925 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.925 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:52:44.925 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859815+0000", 2024-09-13T14:52:44.925 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.0330000000000003e-06 2024-09-13T14:52:44.925 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.925 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.925 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:52:44.925 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859823+0000", 2024-09-13T14:52:44.926 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.126e-06 2024-09-13T14:52:44.926 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.926 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.926 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:52:44.926 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859898+0000", 2024-09-13T14:52:44.926 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.5240999999999993e-05 2024-09-13T14:52:44.926 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.926 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.926 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting_for_map not empty", 2024-09-13T14:52:44.926 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859903+0000", 2024-09-13T14:52:44.926 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.1549999999999998e-06 2024-09-13T14:52:44.926 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.927 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.927 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:52:44.927 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.863680+0000", 2024-09-13T14:52:44.927 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0037772449999999998 2024-09-13T14:52:44.927 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.927 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.927 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:52:44.927 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.863687+0000", 2024-09-13T14:52:44.927 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.8000000000000001e-06 2024-09-13T14:52:44.927 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.927 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.927 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:52:44.927 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.278229+0000", 2024-09-13T14:52:44.928 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4145422000000001 2024-09-13T14:52:44.928 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.928 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.928 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:52:44.928 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.278360+0000", 2024-09-13T14:52:44.928 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000131254 2024-09-13T14:52:44.928 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.928 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.928 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,3,6", 2024-09-13T14:52:44.928 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.278491+0000", 2024-09-13T14:52:44.928 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00013044200000000001 2024-09-13T14:52:44.928 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.929 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.929 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:52:44.929 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.281818+0000", 2024-09-13T14:52:44.929 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0033266110000000002 2024-09-13T14:52:44.929 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.929 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.929 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:44.929 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.283640+0000", 2024-09-13T14:52:44.929 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.001822958 2024-09-13T14:52:44.929 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.930 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.930 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:44.930 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.293331+0000", 2024-09-13T14:52:44.930 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0096903600000000003 2024-09-13T14:52:44.930 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.930 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.930 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:44.930 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.293366+0000", 2024-09-13T14:52:44.930 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.5534000000000002e-05 2024-09-13T14:52:44.930 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.931 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.931 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:52:44.931 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.293418+0000", 2024-09-13T14:52:44.931 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.1622000000000001e-05 2024-09-13T14:52:44.931 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.931 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.931 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:52:44.931 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.293522+0000", 2024-09-13T14:52:44.931 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000103804 2024-09-13T14:52:44.931 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:52:44.931 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:52:44.931 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:52:44.931 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.932 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.932 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:207726 8.3 8:ca67a4e5:::benchmark_data_smithi102_313275_object77139:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e350)", 2024-09-13T14:52:44.932 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:49:18.942178+0000", 2024-09-13T14:52:44.932 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 205.94679370599999, 2024-09-13T14:52:44.932 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.29576781299999999, 2024-09-13T14:52:44.932 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:52:44.932 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:52:44.932 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:44.932 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:52:44.932 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:52:44.932 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:52:44.932 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 207726 2024-09-13T14:52:44.933 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.933 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:52:44.933 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.933 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:52:44.933 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.942178+0000", 2024-09-13T14:52:44.933 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:52:44.933 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.933 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.933 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:52:44.933 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.942178+0000", 2024-09-13T14:52:44.933 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:52:44.933 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.933 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.933 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:52:44.934 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.942179+0000", 2024-09-13T14:52:44.934 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2300000000000001e-06 2024-09-13T14:52:44.934 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.934 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.934 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:52:44.934 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.942183+0000", 2024-09-13T14:52:44.934 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.2540000000000001e-06 2024-09-13T14:52:44.934 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.934 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.934 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:52:44.934 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.942191+0000", 2024-09-13T14:52:44.934 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.7709999999999995e-06 2024-09-13T14:52:44.934 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.935 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.935 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:52:44.935 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.942197+0000", 2024-09-13T14:52:44.935 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.1869999999999996e-06 2024-09-13T14:52:44.935 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.935 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.935 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:52:44.935 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.946805+0000", 2024-09-13T14:52:44.935 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00460832 2024-09-13T14:52:44.935 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.935 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.935 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:52:44.935 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.946928+0000", 2024-09-13T14:52:44.936 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00012341199999999999 2024-09-13T14:52:44.936 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.936 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.936 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,5,6", 2024-09-13T14:52:44.936 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:18.947084+0000", 2024-09-13T14:52:44.936 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00015620599999999999 2024-09-13T14:52:44.936 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.936 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.936 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:44.936 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:19.214008+0000", 2024-09-13T14:52:44.936 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.26692375800000001 2024-09-13T14:52:44.936 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.936 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.937 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:44.937 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:19.214054+0000", 2024-09-13T14:52:44.937 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.5303999999999999e-05 2024-09-13T14:52:44.937 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.937 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.937 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:44.937 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:19.214178+0000", 2024-09-13T14:52:44.937 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000124368 2024-09-13T14:52:44.937 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.937 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.937 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:52:44.937 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:19.237770+0000", 2024-09-13T14:52:44.938 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.023592248999999999 2024-09-13T14:52:44.938 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.938 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.938 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:52:44.938 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:19.237827+0000", 2024-09-13T14:52:44.938 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.6629000000000003e-05 2024-09-13T14:52:44.938 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.938 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.938 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:52:44.938 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:19.237946+0000", 2024-09-13T14:52:44.938 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000119125 2024-09-13T14:52:44.938 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:52:44.938 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:52:44.939 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:52:44.939 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.939 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.939 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.46157.0:226347 8.4 e354/306 8:2531c967:::benchmark_data_smithi102_313275_object95760:head v 354'14056, mlcod=354'14054)", 2024-09-13T14:52:44.939 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:49:42.876357+0000", 2024-09-13T14:52:44.939 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 182.012615138, 2024-09-13T14:52:44.939 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.44227170199999999, 2024-09-13T14:52:44.939 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:52:44.939 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:52:44.939 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:44.939 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:52:44.939 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.939 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:52:44.940 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.876357+0000", 2024-09-13T14:52:44.940 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:52:44.940 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.940 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.940 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:52:44.940 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.876357+0000", 2024-09-13T14:52:44.940 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:52:44.940 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.940 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.940 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:52:44.940 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.876357+0000", 2024-09-13T14:52:44.940 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.0100000000000004e-07 2024-09-13T14:52:44.940 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.941 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.941 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:52:44.941 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.876366+0000", 2024-09-13T14:52:44.941 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.0909999999999993e-06 2024-09-13T14:52:44.941 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.941 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.941 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:52:44.941 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966171+0000", 2024-09-13T14:52:44.941 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.089805323000000006 2024-09-13T14:52:44.941 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.941 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.941 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:52:44.942 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966184+0000", 2024-09-13T14:52:44.942 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2666999999999999e-05 2024-09-13T14:52:44.942 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.942 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.942 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:52:44.942 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966365+0000", 2024-09-13T14:52:44.942 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000181962 2024-09-13T14:52:44.942 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.942 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.942 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:52:44.942 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966383+0000", 2024-09-13T14:52:44.942 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.7958e-05 2024-09-13T14:52:44.942 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.942 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.943 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:52:44.943 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.318583+0000", 2024-09-13T14:52:44.943 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.35219920900000001 2024-09-13T14:52:44.943 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.943 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.943 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:52:44.943 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.318628+0000", 2024-09-13T14:52:44.943 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.5691e-05 2024-09-13T14:52:44.943 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:52:44.943 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:52:44.943 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:52:44.943 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.943 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.944 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.46157.0:226343 8.b e354/337 8:d626b72e:::benchmark_data_smithi102_313275_object95756:head v 354'14208, mlcod=354'14206)", 2024-09-13T14:52:44.944 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:49:42.878012+0000", 2024-09-13T14:52:44.944 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 182.01095936900001, 2024-09-13T14:52:44.944 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.43247816100000003, 2024-09-13T14:52:44.944 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:52:44.944 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:52:44.944 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:44.944 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:52:44.944 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.944 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:52:44.944 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.878012+0000", 2024-09-13T14:52:44.944 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:52:44.944 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.945 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.945 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:52:44.945 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.878012+0000", 2024-09-13T14:52:44.945 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:52:44.945 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.945 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.945 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:52:44.945 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.878013+0000", 2024-09-13T14:52:44.945 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.051e-06 2024-09-13T14:52:44.945 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.945 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.945 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:52:44.945 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.878020+0000", 2024-09-13T14:52:44.946 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.9299999999999997e-06 2024-09-13T14:52:44.946 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.946 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.946 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:52:44.946 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966393+0000", 2024-09-13T14:52:44.946 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.088372493999999996 2024-09-13T14:52:44.946 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.946 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.946 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:52:44.946 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966403+0000", 2024-09-13T14:52:44.946 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.8190000000000003e-06 2024-09-13T14:52:44.947 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.947 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.947 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:52:44.947 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966450+0000", 2024-09-13T14:52:44.947 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.7005000000000001e-05 2024-09-13T14:52:44.947 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.947 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.948 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:52:44.948 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966471+0000", 2024-09-13T14:52:44.948 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.1560999999999999e-05 2024-09-13T14:52:44.948 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.948 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.948 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:52:44.948 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.310310+0000", 2024-09-13T14:52:44.948 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.34383856499999998 2024-09-13T14:52:44.949 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.949 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.949 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:52:44.949 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.310491+0000", 2024-09-13T14:52:44.949 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000180736 2024-09-13T14:52:44.949 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:52:44.949 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:52:44.949 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:52:44.949 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.949 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.949 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.46157.0:226348 8.e e354/306 8:75119f7b:::benchmark_data_smithi102_313275_object95761:head v 354'14072, mlcod=354'14070)", 2024-09-13T14:52:44.950 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:49:42.966225+0000", 2024-09-13T14:52:44.950 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 181.92274680599999, 2024-09-13T14:52:44.950 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.33910425399999999, 2024-09-13T14:52:44.950 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:52:44.950 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:52:44.950 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:44.950 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:52:44.950 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.950 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:52:44.950 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966225+0000", 2024-09-13T14:52:44.950 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:52:44.950 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.951 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.951 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:52:44.951 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966225+0000", 2024-09-13T14:52:44.951 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:52:44.951 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.951 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.951 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:52:44.951 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966226+0000", 2024-09-13T14:52:44.951 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.4499999999999997e-07 2024-09-13T14:52:44.951 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.951 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.952 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:52:44.952 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966232+0000", 2024-09-13T14:52:44.952 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.5490000000000003e-06 2024-09-13T14:52:44.952 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.952 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.952 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:52:44.952 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966239+0000", 2024-09-13T14:52:44.952 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.2309999999999999e-06 2024-09-13T14:52:44.952 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.952 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.952 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:52:44.952 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966244+0000", 2024-09-13T14:52:44.952 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.1960000000000001e-06 2024-09-13T14:52:44.953 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.953 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.953 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:52:44.953 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966776+0000", 2024-09-13T14:52:44.953 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00053200199999999995 2024-09-13T14:52:44.953 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.953 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.953 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:52:44.953 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966793+0000", 2024-09-13T14:52:44.953 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.749e-05 2024-09-13T14:52:44.953 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.953 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.954 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:52:44.954 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.305157+0000", 2024-09-13T14:52:44.954 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.33836434900000001 2024-09-13T14:52:44.954 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.954 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.954 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:52:44.954 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.305329+0000", 2024-09-13T14:52:44.954 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00017179200000000001 2024-09-13T14:52:44.954 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:52:44.954 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:52:44.954 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:52:44.954 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.954 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.954 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.46157.0:226333 8.1 e354/337 8:8840a5fd:::benchmark_data_smithi102_313275_object95746:head v 354'14039, mlcod=354'14033)", 2024-09-13T14:52:44.955 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:49:42.966259+0000", 2024-09-13T14:52:44.955 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 181.922713265, 2024-09-13T14:52:44.955 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.339067756, 2024-09-13T14:52:44.955 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:52:44.955 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:52:44.955 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:44.955 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:52:44.955 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.955 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:52:44.955 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966259+0000", 2024-09-13T14:52:44.955 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:52:44.955 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.956 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.956 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:52:44.956 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966259+0000", 2024-09-13T14:52:44.956 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:52:44.956 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.956 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.956 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:52:44.956 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966259+0000", 2024-09-13T14:52:44.956 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.6800000000000001e-07 2024-09-13T14:52:44.956 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.956 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.956 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:52:44.957 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966262+0000", 2024-09-13T14:52:44.957 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.129e-06 2024-09-13T14:52:44.957 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.957 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.957 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:52:44.957 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966267+0000", 2024-09-13T14:52:44.957 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.843e-06 2024-09-13T14:52:44.957 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.958 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.958 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:52:44.959 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966272+0000", 2024-09-13T14:52:44.959 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.7509999999999999e-06 2024-09-13T14:52:44.959 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.959 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.959 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:52:44.959 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.972016+0000", 2024-09-13T14:52:44.959 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0057441410000000004 2024-09-13T14:52:44.959 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.959 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.959 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:52:44.959 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.972038+0000", 2024-09-13T14:52:44.959 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.1860000000000001e-05 2024-09-13T14:52:44.959 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.960 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.960 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:52:44.960 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.305223+0000", 2024-09-13T14:52:44.960 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.33318497499999999 2024-09-13T14:52:44.960 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.960 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.960 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:52:44.960 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.305326+0000", 2024-09-13T14:52:44.960 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000103689 2024-09-13T14:52:44.960 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:52:44.960 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:52:44.960 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:52:44.960 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.961 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.961 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.46157.0:226351 8.4 e354/306 8:23f709fc:::benchmark_data_smithi102_313275_object95764:head v 354'14057, mlcod=354'14054)", 2024-09-13T14:52:44.961 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:49:42.966260+0000", 2024-09-13T14:52:44.961 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 181.922711451, 2024-09-13T14:52:44.961 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.352393028, 2024-09-13T14:52:44.961 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:52:44.961 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:52:44.961 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:44.961 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:52:44.961 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.961 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:52:44.961 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966260+0000", 2024-09-13T14:52:44.961 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:52:44.961 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.962 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.962 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:52:44.962 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966260+0000", 2024-09-13T14:52:44.962 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:52:44.962 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.962 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.962 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:52:44.962 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966261+0000", 2024-09-13T14:52:44.962 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.4400000000000001e-07 2024-09-13T14:52:44.962 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.962 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.962 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:52:44.962 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966265+0000", 2024-09-13T14:52:44.963 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.0239999999999999e-06 2024-09-13T14:52:44.963 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.963 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.963 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:52:44.963 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966271+0000", 2024-09-13T14:52:44.963 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.9370000000000001e-06 2024-09-13T14:52:44.963 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.963 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.963 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:52:44.963 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966274+0000", 2024-09-13T14:52:44.963 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.7639999999999999e-06 2024-09-13T14:52:44.963 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.963 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.963 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:52:44.964 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.970109+0000", 2024-09-13T14:52:44.964 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0038344640000000001 2024-09-13T14:52:44.964 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.964 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.964 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:52:44.964 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.970127+0000", 2024-09-13T14:52:44.964 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.8082000000000001e-05 2024-09-13T14:52:44.964 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.964 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.964 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:52:44.964 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.318632+0000", 2024-09-13T14:52:44.964 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.348505116 2024-09-13T14:52:44.965 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.965 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.965 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:52:44.965 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.318653+0000", 2024-09-13T14:52:44.965 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.1297e-05 2024-09-13T14:52:44.965 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:52:44.965 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:52:44.965 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:52:44.965 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.965 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.965 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.46157.0:226357 8.e e354/306 8:7912c2d7:::benchmark_data_smithi102_313275_object95770:head v 354'14073, mlcod=354'14070)", 2024-09-13T14:52:44.965 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:49:42.966277+0000", 2024-09-13T14:52:44.965 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 181.92269483600001, 2024-09-13T14:52:44.966 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.33908791799999999, 2024-09-13T14:52:44.966 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:52:44.966 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:52:44.966 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:44.966 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:52:44.966 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.966 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:52:44.966 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966277+0000", 2024-09-13T14:52:44.966 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:52:44.966 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.966 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.966 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:52:44.966 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966277+0000", 2024-09-13T14:52:44.966 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:52:44.967 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.967 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.967 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:52:44.967 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966277+0000", 2024-09-13T14:52:44.967 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.35e-07 2024-09-13T14:52:44.967 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.967 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.967 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:52:44.967 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966284+0000", 2024-09-13T14:52:44.967 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.5640000000000002e-06 2024-09-13T14:52:44.967 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.967 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.968 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:52:44.968 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966288+0000", 2024-09-13T14:52:44.968 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.7029999999999997e-06 2024-09-13T14:52:44.968 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.968 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.968 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:52:44.968 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966292+0000", 2024-09-13T14:52:44.968 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.456e-06 2024-09-13T14:52:44.968 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.968 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.968 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:52:44.968 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.969737+0000", 2024-09-13T14:52:44.969 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.003445591 2024-09-13T14:52:44.969 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.969 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.969 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:52:44.969 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.969751+0000", 2024-09-13T14:52:44.969 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3529999999999999e-05 2024-09-13T14:52:44.969 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.969 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.969 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:52:44.969 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.305333+0000", 2024-09-13T14:52:44.970 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.33558170999999998 2024-09-13T14:52:44.970 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.970 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.970 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:52:44.970 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.305365+0000", 2024-09-13T14:52:44.970 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.2128999999999997e-05 2024-09-13T14:52:44.970 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:52:44.970 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:52:44.970 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:52:44.970 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.970 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.970 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.46157.0:226363 8.f e354/337 8:f9e9d9b3:::benchmark_data_smithi102_313275_object95776:head v 354'14122, mlcod=354'14120)", 2024-09-13T14:52:44.970 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:49:42.966470+0000", 2024-09-13T14:52:44.971 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 181.922502017, 2024-09-13T14:52:44.971 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.33875966800000001, 2024-09-13T14:52:44.971 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:52:44.971 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:52:44.971 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:44.971 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:52:44.971 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.971 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:52:44.971 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966470+0000", 2024-09-13T14:52:44.971 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:52:44.971 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.971 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.971 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:52:44.972 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966470+0000", 2024-09-13T14:52:44.972 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:52:44.972 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.972 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.972 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:52:44.972 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966470+0000", 2024-09-13T14:52:44.972 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.9499999999999998e-07 2024-09-13T14:52:44.972 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.972 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.972 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:52:44.972 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966475+0000", 2024-09-13T14:52:44.972 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.4009999999999999e-06 2024-09-13T14:52:44.972 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.973 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.973 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:52:44.973 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966481+0000", 2024-09-13T14:52:44.973 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.5729999999999996e-06 2024-09-13T14:52:44.973 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.973 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.973 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:52:44.973 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966486+0000", 2024-09-13T14:52:44.973 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.498e-06 2024-09-13T14:52:44.973 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.973 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.973 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:52:44.973 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966552+0000", 2024-09-13T14:52:44.974 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.6180999999999995e-05 2024-09-13T14:52:44.974 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.974 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.974 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:52:44.974 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966572+0000", 2024-09-13T14:52:44.974 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.0339999999999998e-05 2024-09-13T14:52:44.974 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.974 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.974 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:52:44.974 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.304984+0000", 2024-09-13T14:52:44.974 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.33841160399999998 2024-09-13T14:52:44.974 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.974 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.974 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:52:44.975 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.305229+0000", 2024-09-13T14:52:44.975 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00024567600000000001 2024-09-13T14:52:44.975 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:52:44.975 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:52:44.975 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:52:44.975 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.975 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.975 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.46157.0:226364 8.f e354/337 8:f43d0b70:::benchmark_data_smithi102_313275_object95777:head v 354'14123, mlcod=354'14120)", 2024-09-13T14:52:44.975 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:49:42.966511+0000", 2024-09-13T14:52:44.975 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 181.922460473, 2024-09-13T14:52:44.975 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.33875917500000002, 2024-09-13T14:52:44.975 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:52:44.975 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:52:44.976 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:44.976 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:52:44.976 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.976 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:52:44.976 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966511+0000", 2024-09-13T14:52:44.976 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:52:44.976 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.976 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.976 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:52:44.976 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966511+0000", 2024-09-13T14:52:44.976 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:52:44.976 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.976 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.977 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:52:44.977 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966512+0000", 2024-09-13T14:52:44.977 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.2600000000000001e-07 2024-09-13T14:52:44.977 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.977 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.977 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:52:44.977 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966515+0000", 2024-09-13T14:52:44.977 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.2059999999999999e-06 2024-09-13T14:52:44.977 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.977 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.977 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:52:44.977 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966520+0000", 2024-09-13T14:52:44.977 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.2209999999999996e-06 2024-09-13T14:52:44.978 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.978 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.978 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:52:44.978 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966524+0000", 2024-09-13T14:52:44.978 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.1409999999999998e-06 2024-09-13T14:52:44.978 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.978 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.978 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:52:44.978 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.969884+0000", 2024-09-13T14:52:44.978 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0033597480000000001 2024-09-13T14:52:44.978 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.978 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.978 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:52:44.978 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.969904+0000", 2024-09-13T14:52:44.979 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.0265e-05 2024-09-13T14:52:44.979 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.979 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.979 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:52:44.979 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.305234+0000", 2024-09-13T14:52:44.979 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.33532999800000002 2024-09-13T14:52:44.979 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.979 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.979 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:52:44.979 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.305270+0000", 2024-09-13T14:52:44.979 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.6369999999999999e-05 2024-09-13T14:52:44.979 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:52:44.980 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:52:44.980 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:52:44.980 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.980 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.980 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:226353 8.a 8:56f6bbd7:::benchmark_data_smithi102_313275_object95766:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-09-13T14:52:44.980 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:49:42.966566+0000", 2024-09-13T14:52:44.980 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 181.92240554, 2024-09-13T14:52:44.980 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.33901846299999999, 2024-09-13T14:52:44.980 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:52:44.980 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:52:44.980 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:44.980 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:52:44.980 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:52:44.981 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:52:44.981 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 226353 2024-09-13T14:52:44.981 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.981 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:52:44.981 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.981 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:52:44.981 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966566+0000", 2024-09-13T14:52:44.981 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:52:44.981 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.981 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.981 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:52:44.981 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966566+0000", 2024-09-13T14:52:44.981 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:52:44.982 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.982 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.982 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:52:44.982 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966567+0000", 2024-09-13T14:52:44.982 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2020000000000001e-06 2024-09-13T14:52:44.982 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.982 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.982 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:52:44.982 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966571+0000", 2024-09-13T14:52:44.982 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.7060000000000002e-06 2024-09-13T14:52:44.982 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.982 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.982 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:52:44.982 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966581+0000", 2024-09-13T14:52:44.983 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.5899999999999997e-06 2024-09-13T14:52:44.983 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.983 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.983 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:52:44.983 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966587+0000", 2024-09-13T14:52:44.983 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.9100000000000002e-06 2024-09-13T14:52:44.983 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.983 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.983 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:52:44.983 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.967254+0000", 2024-09-13T14:52:44.983 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00066735600000000003 2024-09-13T14:52:44.983 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.983 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.983 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:52:44.984 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.967370+0000", 2024-09-13T14:52:44.984 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000116449 2024-09-13T14:52:44.984 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.984 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.984 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,3,5", 2024-09-13T14:52:44.984 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.967496+0000", 2024-09-13T14:52:44.984 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000125597 2024-09-13T14:52:44.984 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.984 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.984 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:52:44.984 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.972217+0000", 2024-09-13T14:52:44.984 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0047206050000000001 2024-09-13T14:52:44.984 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.985 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.985 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:44.985 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.304933+0000", 2024-09-13T14:52:44.985 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.33271657399999999 2024-09-13T14:52:44.985 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.985 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.985 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:44.985 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.305240+0000", 2024-09-13T14:52:44.985 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000307084 2024-09-13T14:52:44.985 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.985 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.985 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:44.985 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.305417+0000", 2024-09-13T14:52:44.986 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00017644700000000001 2024-09-13T14:52:44.986 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.986 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.986 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:52:44.986 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.305474+0000", 2024-09-13T14:52:44.986 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.6988999999999999e-05 2024-09-13T14:52:44.986 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.986 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.986 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:52:44.986 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.305585+0000", 2024-09-13T14:52:44.986 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000110954 2024-09-13T14:52:44.986 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:52:44.986 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:52:44.987 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:52:44.987 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.987 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.987 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:226356 8.3 8:cdb7d1cd:::benchmark_data_smithi102_313275_object95769:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-09-13T14:52:44.987 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:49:42.966599+0000", 2024-09-13T14:52:44.987 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 181.92237234199999, 2024-09-13T14:52:44.987 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.347066559, 2024-09-13T14:52:44.987 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:52:44.987 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:52:44.987 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:44.987 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:52:44.987 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:52:44.987 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:52:44.987 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 226356 2024-09-13T14:52:44.988 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.988 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:52:44.988 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.988 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:52:44.988 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966599+0000", 2024-09-13T14:52:44.988 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:52:44.988 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.988 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.988 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:52:44.988 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966599+0000", 2024-09-13T14:52:44.988 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:52:44.988 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.989 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.989 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:52:44.989 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966600+0000", 2024-09-13T14:52:44.989 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.2899999999999999e-07 2024-09-13T14:52:44.989 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.989 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.989 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:52:44.989 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966602+0000", 2024-09-13T14:52:44.989 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.3599999999999999e-06 2024-09-13T14:52:44.989 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.989 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.989 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:52:44.989 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966609+0000", 2024-09-13T14:52:44.990 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.3720000000000001e-06 2024-09-13T14:52:44.990 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.990 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.990 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:52:44.990 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966614+0000", 2024-09-13T14:52:44.990 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.4589999999999997e-06 2024-09-13T14:52:44.990 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.990 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.990 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:52:44.990 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.967705+0000", 2024-09-13T14:52:44.990 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0010911919999999999 2024-09-13T14:52:44.990 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.990 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.990 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:52:44.991 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.967826+0000", 2024-09-13T14:52:44.991 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000120355 2024-09-13T14:52:44.991 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.991 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.991 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,5,6", 2024-09-13T14:52:44.991 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.967955+0000", 2024-09-13T14:52:44.991 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00012938199999999999 2024-09-13T14:52:44.991 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.991 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.991 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:44.991 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.311754+0000", 2024-09-13T14:52:44.991 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.34379863500000002 2024-09-13T14:52:44.991 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.992 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.992 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:44.992 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.311792+0000", 2024-09-13T14:52:44.992 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.8247000000000002e-05 2024-09-13T14:52:44.992 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.992 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.992 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:44.992 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.311827+0000", 2024-09-13T14:52:44.992 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.5454000000000003e-05 2024-09-13T14:52:44.992 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.992 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.992 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:52:44.992 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.313159+0000", 2024-09-13T14:52:44.993 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0013320140000000001 2024-09-13T14:52:44.993 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.993 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.993 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:52:44.993 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.313312+0000", 2024-09-13T14:52:44.993 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000152318 2024-09-13T14:52:44.993 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.993 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.993 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:52:44.993 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.313666+0000", 2024-09-13T14:52:44.993 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000354342 2024-09-13T14:52:44.993 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:52:44.994 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:52:44.994 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:52:44.994 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.994 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.994 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:226359 8.3 8:c15c2493:::benchmark_data_smithi102_313275_object95772:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-09-13T14:52:44.994 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:49:42.966622+0000", 2024-09-13T14:52:44.994 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 181.92234993400001, 2024-09-13T14:52:44.994 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.35298271799999997, 2024-09-13T14:52:44.994 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:52:44.994 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:52:44.994 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:44.994 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:52:44.994 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:52:44.995 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:52:44.995 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 226359 2024-09-13T14:52:44.995 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.995 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:52:44.995 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.995 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:52:44.995 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966622+0000", 2024-09-13T14:52:44.995 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:52:44.995 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.995 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.995 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:52:44.995 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966622+0000", 2024-09-13T14:52:44.995 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:52:44.995 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.996 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.996 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:52:44.996 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966622+0000", 2024-09-13T14:52:44.996 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.9900000000000001e-07 2024-09-13T14:52:44.996 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.996 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.996 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:52:44.996 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966624+0000", 2024-09-13T14:52:44.996 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.2369999999999999e-06 2024-09-13T14:52:44.996 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.996 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.996 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:52:44.996 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966630+0000", 2024-09-13T14:52:44.997 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.3750000000000002e-06 2024-09-13T14:52:44.997 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.997 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.997 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:52:44.997 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966635+0000", 2024-09-13T14:52:44.997 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.7820000000000001e-06 2024-09-13T14:52:44.997 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.997 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.997 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:52:44.997 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.304872+0000", 2024-09-13T14:52:44.997 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.338237383 2024-09-13T14:52:44.997 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.998 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.998 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:52:44.998 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.305028+0000", 2024-09-13T14:52:44.998 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00015607300000000001 2024-09-13T14:52:44.998 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.998 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.998 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,5,6", 2024-09-13T14:52:44.998 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.305180+0000", 2024-09-13T14:52:44.998 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000151848 2024-09-13T14:52:44.998 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.998 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.998 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:44.998 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.313810+0000", 2024-09-13T14:52:44.998 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0086300419999999992 2024-09-13T14:52:44.999 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.999 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.999 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:52:44.999 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.313874+0000", 2024-09-13T14:52:44.999 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.4339999999999997e-05 2024-09-13T14:52:44.999 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.999 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.999 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:44.999 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.314061+0000", 2024-09-13T14:52:44.999 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000186653 2024-09-13T14:52:44.999 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:44.999 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:44.999 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:52:44.999 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.314236+0000", 2024-09-13T14:52:45.000 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00017469999999999999 2024-09-13T14:52:45.000 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:45.000 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:45.000 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:52:45.000 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.314451+0000", 2024-09-13T14:52:45.000 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00021489 2024-09-13T14:52:45.000 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:45.000 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:45.000 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:52:45.000 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.319605+0000", 2024-09-13T14:52:45.000 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0051539960000000001 2024-09-13T14:52:45.000 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:52:45.000 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:52:45.001 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:52:45.001 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:45.001 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:45.001 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.46157.0:226367 8.4 e354/306 8:2828d25f:::benchmark_data_smithi102_313275_object95780:head v 354'14058, mlcod=354'14054)", 2024-09-13T14:52:45.001 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:49:42.967323+0000", 2024-09-13T14:52:45.001 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 181.92164862300001, 2024-09-13T14:52:45.001 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.35143049599999998, 2024-09-13T14:52:45.001 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:52:45.001 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:52:45.001 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:52:45.001 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:52:45.001 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:45.001 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:52:45.002 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.967323+0000", 2024-09-13T14:52:45.002 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:52:45.002 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:45.002 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:45.002 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:52:45.002 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.967323+0000", 2024-09-13T14:52:45.002 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:52:45.002 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:45.002 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:45.002 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:52:45.002 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.967324+0000", 2024-09-13T14:52:45.002 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.1099999999999995e-07 2024-09-13T14:52:45.002 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:45.002 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:45.003 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:52:45.003 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.967329+0000", 2024-09-13T14:52:45.003 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.0390000000000004e-06 2024-09-13T14:52:45.003 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:45.003 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:45.003 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:52:45.003 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.967339+0000", 2024-09-13T14:52:45.003 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.8779999999999993e-06 2024-09-13T14:52:45.003 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:45.003 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:45.003 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:52:45.003 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.967344+0000", 2024-09-13T14:52:45.003 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.7219999999999996e-06 2024-09-13T14:52:45.004 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:45.004 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:45.004 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:52:45.004 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.304997+0000", 2024-09-13T14:52:45.004 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.33765234100000002 2024-09-13T14:52:45.004 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:45.004 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:45.004 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:52:45.004 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.305019+0000", 2024-09-13T14:52:45.004 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.1977e-05 2024-09-13T14:52:45.004 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:45.004 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:45.004 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:52:45.005 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.318658+0000", 2024-09-13T14:52:45.005 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.013639709 2024-09-13T14:52:45.005 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:45.005 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:45.005 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:52:45.005 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.318754+0000", 2024-09-13T14:52:45.005 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.5218999999999998e-05 2024-09-13T14:52:45.005 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:52:45.005 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:52:45.005 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:52:45.005 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:45.005 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:45.005 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop_reply(client.46157.0:226356 8.3 e355/337 ondisk, result = 0)", 2024-09-13T14:52:45.006 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:49:42.973291+0000", 2024-09-13T14:52:45.006 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 181.91568123299999, 2024-09-13T14:52:45.006 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.33847084700000002, 2024-09-13T14:52:45.006 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:52:45.006 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:52:45.006 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "started", 2024-09-13T14:52:45.006 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:52:45.006 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:45.006 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:52:45.006 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.973291+0000", 2024-09-13T14:52:45.006 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:52:45.006 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:45.006 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:45.007 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:52:45.007 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.973291+0000", 2024-09-13T14:52:45.007 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:52:45.007 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:45.007 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:45.007 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:52:45.007 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.973292+0000", 2024-09-13T14:52:45.007 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.8109999999999999e-06 2024-09-13T14:52:45.007 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:45.007 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:45.007 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:52:45.007 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.973298+0000", 2024-09-13T14:52:45.007 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.49e-06 2024-09-13T14:52:45.008 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:45.008 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:45.008 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:52:45.008 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.304829+0000", 2024-09-13T14:52:45.008 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.33153068099999999 2024-09-13T14:52:45.008 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:45.008 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:45.008 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:52:45.008 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.304844+0000", 2024-09-13T14:52:45.008 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.5858999999999998e-05 2024-09-13T14:52:45.008 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:45.008 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:45.008 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:52:45.009 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.311739+0000", 2024-09-13T14:52:45.009 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.006894664 2024-09-13T14:52:45.009 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:45.009 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:45.009 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:52:45.009 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.311747+0000", 2024-09-13T14:52:45.009 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.9910000000000002e-06 2024-09-13T14:52:45.009 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:45.009 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:45.009 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:52:45.009 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.311761+0000", 2024-09-13T14:52:45.009 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4351e-05 2024-09-13T14:52:45.009 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:52:45.010 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:52:45.010 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:52:45.010 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:45.010 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:45.010 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop_reply(client.46157.0:226353 8.a e355/337 ondisk, result = 0)", 2024-09-13T14:52:45.010 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:49:42.974862+0000", 2024-09-13T14:52:45.010 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 181.91411008700001, 2024-09-13T14:52:45.010 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.33008076200000003, 2024-09-13T14:52:45.010 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:52:45.010 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:52:45.010 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "started", 2024-09-13T14:52:45.010 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:52:45.010 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:45.011 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:52:45.011 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.974862+0000", 2024-09-13T14:52:45.011 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:52:45.011 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:45.011 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:45.011 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:52:45.011 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.974862+0000", 2024-09-13T14:52:45.011 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:52:45.011 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:45.011 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:45.011 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:52:45.011 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.974863+0000", 2024-09-13T14:52:45.011 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.4600000000000003e-07 2024-09-13T14:52:45.011 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:45.012 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:45.012 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:52:45.012 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.974868+0000", 2024-09-13T14:52:45.012 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.3430000000000003e-06 2024-09-13T14:52:45.012 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:45.012 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:45.012 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:52:45.012 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.304836+0000", 2024-09-13T14:52:45.012 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.32996769799999998 2024-09-13T14:52:45.012 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:45.012 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:45.012 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:52:45.012 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.304846+0000", 2024-09-13T14:52:45.013 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0651e-05 2024-09-13T14:52:45.013 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:45.013 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:45.013 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:52:45.013 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.304916+0000", 2024-09-13T14:52:45.013 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.9331000000000004e-05 2024-09-13T14:52:45.013 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:45.013 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:45.013 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:52:45.013 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.304926+0000", 2024-09-13T14:52:45.013 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0317e-05 2024-09-13T14:52:45.013 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:45.013 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:45.013 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:52:45.014 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.304942+0000", 2024-09-13T14:52:45.014 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.6475999999999999e-05 2024-09-13T14:52:45.014 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:52:45.014 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:52:45.014 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:52:45.014 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:45.014 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:45.014 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.36528.0:113806 9.3 9.95e516a3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e403)", 2024-09-13T14:52:45.014 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:52:27.688526+0000", 2024-09-13T14:52:45.014 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 17.200445464000001, 2024-09-13T14:52:45.014 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.64616753500000002, 2024-09-13T14:52:45.014 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:52:45.014 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:52:45.014 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "reached pg", 2024-09-13T14:52:45.015 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:52:45.015 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.36528", 2024-09-13T14:52:45.015 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:52:45.015 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 113806 2024-09-13T14:52:45.015 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:45.015 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:52:45.015 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:45.015 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:52:45.015 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:52:27.688526+0000", 2024-09-13T14:52:45.015 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:52:45.015 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:45.015 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:45.015 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:52:45.015 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:52:27.688526+0000", 2024-09-13T14:52:45.016 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:52:45.016 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:45.016 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:45.016 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:52:45.016 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:52:27.688528+0000", 2024-09-13T14:52:45.016 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.9360000000000002e-06 2024-09-13T14:52:45.016 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:45.016 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:45.016 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:52:45.016 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:52:27.688562+0000", 2024-09-13T14:52:45.016 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.4106000000000002e-05 2024-09-13T14:52:45.016 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:45.016 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:45.017 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:52:45.017 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:52:27.688581+0000", 2024-09-13T14:52:45.017 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.8448999999999999e-05 2024-09-13T14:52:45.017 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:45.017 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:45.017 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:52:45.017 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:52:27.688591+0000", 2024-09-13T14:52:45.017 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0638999999999999e-05 2024-09-13T14:52:45.017 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:45.017 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:45.017 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:52:45.017 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:52:27.688701+0000", 2024-09-13T14:52:45.017 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000109889 2024-09-13T14:52:45.017 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:45.018 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:45.018 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:52:45.018 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:52:27.688719+0000", 2024-09-13T14:52:45.018 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.7706999999999998e-05 2024-09-13T14:52:45.018 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:45.018 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:45.018 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:52:45.018 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:52:28.334684+0000", 2024-09-13T14:52:45.018 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.64596481900000002 2024-09-13T14:52:45.018 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:52:45.018 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:52:45.018 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:52:45.018 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:52:28.334694+0000", 2024-09-13T14:52:45.019 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.9899999999999992e-06 2024-09-13T14:52:45.019 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:52:45.019 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:52:45.019 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:52:45.019 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:52:45.019 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:52:45.019 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:52:45.019 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:45.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:45.055 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:45.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:45.156 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:45.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:44 smithi102 ceph-mon[115269]: pgmap v1378: 119 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 114 active+clean; 7.1 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 29 MiB/s wr, 929 op/s; 13944/349443 objects degraded (3.990%); 6880/349443 objects misplaced (1.969%); 7.3 MiB/s, 116 objects/s recovering 2024-09-13T14:52:45.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:44 smithi102 ceph-mon[110183]: pgmap v1378: 119 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 114 active+clean; 7.1 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 29 MiB/s wr, 929 op/s; 13944/349443 objects degraded (3.990%); 6880/349443 objects misplaced (1.969%); 7.3 MiB/s, 116 objects/s recovering 2024-09-13T14:52:45.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:45.194 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:45.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:45.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:44 smithi195 ceph-mon[104262]: pgmap v1378: 119 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 114 active+clean; 7.1 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 29 MiB/s wr, 929 op/s; 13944/349443 objects degraded (3.990%); 6880/349443 objects misplaced (1.969%); 7.3 MiB/s, 116 objects/s recovering 2024-09-13T14:52:45.295 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:45.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:45.330 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:45.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:45.407 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-13T14:52:45.431 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:45.464 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:45.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:45.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:45.566 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:45.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:45.609 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:45.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:45.710 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:45.716 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:45.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:45.748 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:45.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:45.849 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:45.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:45.903 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:45.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:46.004 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:46.038 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:46.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:46.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:46.140 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:46.177 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:46.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:46.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:46.279 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:46.315 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:46.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:46.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:46.417 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:46.427 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T14:52:46.458 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:46.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:46.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:46.560 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:46.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:46.600 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:46.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:46.671 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:52:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:52:46] "GET /metrics HTTP/1.1" 200 36536 "" "Prometheus/2.51.0" 2024-09-13T14:52:46.701 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:46.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:46.730 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:46.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:46.831 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:46.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:46.863 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:46.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:46.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:46 smithi102 ceph-mon[115269]: pgmap v1379: 119 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 114 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 22 MiB/s wr, 933 op/s; 13944/346335 objects degraded (4.026%); 6157/346335 objects misplaced (1.778%); 7.9 MiB/s, 126 objects/s recovering 2024-09-13T14:52:46.944 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:46 smithi102 ceph-mon[110183]: pgmap v1379: 119 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 114 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 22 MiB/s wr, 933 op/s; 13944/346335 objects degraded (4.026%); 6157/346335 objects misplaced (1.778%); 7.9 MiB/s, 126 objects/s recovering 2024-09-13T14:52:46.964 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:47.007 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:47.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:47.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:47.109 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:47.143 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:47.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:47.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:47.245 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:47.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:46 smithi195 ceph-mon[104262]: pgmap v1379: 119 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 114 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 22 MiB/s wr, 933 op/s; 13944/346335 objects degraded (4.026%); 6157/346335 objects misplaced (1.778%); 7.9 MiB/s, 126 objects/s recovering 2024-09-13T14:52:47.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:47.276 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:47.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:47.377 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:47.403 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:47.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:47.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:47.505 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:47.534 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:47.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:47.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:47.636 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:47.660 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:47.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:47.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:47.762 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:47.784 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:47.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:47.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:47.886 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:47.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:47.916 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:47.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:48.017 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:48.048 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:48.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:48.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:48.150 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:48.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:47 smithi102 ceph-mon[110183]: osdmap e407: 8 total, 8 up, 8 in 2024-09-13T14:52:48.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:47 smithi102 ceph-mon[115269]: osdmap e407: 8 total, 8 up, 8 in 2024-09-13T14:52:48.177 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:48.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:48.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:48.249 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:47 smithi195 ceph-mon[104262]: osdmap e407: 8 total, 8 up, 8 in 2024-09-13T14:52:48.280 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:48.318 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:48.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:48.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:48.420 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:48.454 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:48.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:48.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:48.556 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:48.593 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:48.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:48.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:48.695 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:48.730 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:48.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:48.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:48.832 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:48.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:48.867 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:48.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:48.968 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:49.006 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:49.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:49.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:49.107 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:49.139 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:49.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:49.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:49.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:48 smithi102 ceph-mon[110183]: pgmap v1381: 119 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 114 active+clean; 6.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 127 B/s rd, 2.0 MiB/s wr, 1.03k op/s; 13582/338262 objects degraded (4.015%); 5845/338262 objects misplaced (1.728%); 11 MiB/s, 183 objects/s recovering 2024-09-13T14:52:49.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:48 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 13582/338262 objects degraded (4.015%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:52:49.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:48 smithi102 ceph-mon[110183]: osdmap e408: 8 total, 8 up, 8 in 2024-09-13T14:52:49.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:48 smithi102 ceph-mon[115269]: pgmap v1381: 119 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 114 active+clean; 6.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 127 B/s rd, 2.0 MiB/s wr, 1.03k op/s; 13582/338262 objects degraded (4.015%); 5845/338262 objects misplaced (1.728%); 11 MiB/s, 183 objects/s recovering 2024-09-13T14:52:49.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:48 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 13582/338262 objects degraded (4.015%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:52:49.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:48 smithi102 ceph-mon[115269]: osdmap e408: 8 total, 8 up, 8 in 2024-09-13T14:52:49.240 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:49.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:48 smithi195 ceph-mon[104262]: pgmap v1381: 119 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 114 active+clean; 6.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 127 B/s rd, 2.0 MiB/s wr, 1.03k op/s; 13582/338262 objects degraded (4.015%); 5845/338262 objects misplaced (1.728%); 11 MiB/s, 183 objects/s recovering 2024-09-13T14:52:49.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:48 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 13582/338262 objects degraded (4.015%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:52:49.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:48 smithi195 ceph-mon[104262]: osdmap e408: 8 total, 8 up, 8 in 2024-09-13T14:52:49.274 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:49.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:49.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:49.376 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:49.416 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:49.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:49.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:49.518 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:49.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:49.558 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:49.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:49.659 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:49.696 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:49.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:49.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:49.798 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:49.832 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:49.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:49.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:49.934 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:49.969 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:49.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:49.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:50.069 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:50.071 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:50.100 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:50.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:50.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:50.203 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:50.253 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:50.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:50.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:50.355 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:50.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:50.404 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:50.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:50.505 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:50.545 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:50.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:50.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:50.647 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:50.673 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:50.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:50.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:50.775 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:50.809 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:50.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:50.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:50.911 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:50.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:50.954 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:50.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:51.055 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:51.089 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:51.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:51.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:51.191 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:51.212 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:52:51.212 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:52:51.212 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:52:51.212 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:52:51.212 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:52:51.212 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:52:51.212 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:52:51.212 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:52:51.213 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:52:51.213 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:52:51.213 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:52:51.213 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:52:51.213 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:52:51.213 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:52:51.221 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:52:51.222 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:52:51.239 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:51.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:51.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:51.341 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:51.375 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:51.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:51.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:51.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:51 smithi102 ceph-mon[110183]: pgmap v1383: 119 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 114 active+clean; 6.8 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s; 13582/333030 objects degraded (4.078%); 5845/333030 objects misplaced (1.755%); 6.3 MiB/s, 101 objects/s recovering 2024-09-13T14:52:51.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:51 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:52:51.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:51 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:52:51.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:51 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:52:51.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:51 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:52:51.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:51 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:52:51.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:51 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1793334525' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:52:51.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:51 smithi102 ceph-mon[115269]: pgmap v1383: 119 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 114 active+clean; 6.8 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s; 13582/333030 objects degraded (4.078%); 5845/333030 objects misplaced (1.755%); 6.3 MiB/s, 101 objects/s recovering 2024-09-13T14:52:51.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:51 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:52:51.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:51 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:52:51.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:51 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:52:51.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:51 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:52:51.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:51 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:52:51.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:51 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1793334525' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:52:51.477 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:51.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:51 smithi195 ceph-mon[104262]: pgmap v1383: 119 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 114 active+clean; 6.8 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s; 13582/333030 objects degraded (4.078%); 5845/333030 objects misplaced (1.755%); 6.3 MiB/s, 101 objects/s recovering 2024-09-13T14:52:51.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:51 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:52:51.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:51 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:52:51.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:51 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:52:51.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:51 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:52:51.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:51 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:52:51.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:51 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1793334525' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:52:51.514 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:51.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:51.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:51.615 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:51.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:51.657 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:51.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:51.758 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:51.791 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:51.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:51.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:51.893 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:51.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:51.926 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:51.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:52.027 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:52.062 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:52.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:52.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:52.164 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:52.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:52.203 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:52.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:52.304 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:52.328 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:52.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:52.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:52.430 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:52.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:52.463 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:52.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:52.564 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:52.598 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:52.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:52.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:52.625 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:52.701 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:52.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:52.738 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:52.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:52.839 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:52.873 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:52.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:52.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:52.976 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:53.012 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:53.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:53.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:53.114 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:53.148 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:53.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:53.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:53.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:53 smithi102 ceph-mon[110183]: pgmap v1384: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s; 10023/323214 objects degraded (3.101%); 8182/323214 objects misplaced (2.531%); 11 MiB/s, 175 objects/s recovering 2024-09-13T14:52:53.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:53 smithi102 ceph-mon[115269]: pgmap v1384: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s; 10023/323214 objects degraded (3.101%); 8182/323214 objects misplaced (2.531%); 11 MiB/s, 175 objects/s recovering 2024-09-13T14:52:53.250 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:53.285 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:53.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:53.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:53.387 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:53.416 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:53.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:53.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:53.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:53 smithi195 ceph-mon[104262]: pgmap v1384: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s; 10023/323214 objects degraded (3.101%); 8182/323214 objects misplaced (2.531%); 11 MiB/s, 175 objects/s recovering 2024-09-13T14:52:53.518 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:53.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:53.559 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:53.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:53.659 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:53.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:53.694 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:53.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:53.795 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:53.832 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:53.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:53.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:53.934 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:53.968 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:53.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:53.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:54.070 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:54.105 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:54.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:54.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:54.206 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:54.239 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:54.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:54.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:54.341 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:54.388 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:54.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:54.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:54.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:54 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 10023/323214 objects degraded (3.101%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:52:54.401 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:54 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 10023/323214 objects degraded (3.101%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:52:54.490 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:54.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:54 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 10023/323214 objects degraded (3.101%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:52:54.531 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:54.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:54.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:54.634 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:54.677 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:54.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:54.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:54.779 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:54.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:54.813 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:54.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:54.914 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:54.951 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:54.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:54.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:55.053 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:55.085 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:55.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:55.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:55.181 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:55 smithi102 ceph-mon[110183]: pgmap v1385: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 1.20k op/s; 10023/317574 objects degraded (3.156%); 8182/317574 objects misplaced (2.576%); 8.2 MiB/s, 130 objects/s recovering 2024-09-13T14:52:55.182 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:55 smithi102 ceph-mon[115269]: pgmap v1385: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 1.20k op/s; 10023/317574 objects degraded (3.156%); 8182/317574 objects misplaced (2.576%); 8.2 MiB/s, 130 objects/s recovering 2024-09-13T14:52:55.187 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:55.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:55.238 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:55.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:55.339 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:55.375 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:55.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:55.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:55.477 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:55.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:55 smithi195 ceph-mon[104262]: pgmap v1385: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 1.20k op/s; 10023/317574 objects degraded (3.156%); 8182/317574 objects misplaced (2.576%); 8.2 MiB/s, 130 objects/s recovering 2024-09-13T14:52:55.520 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:55.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:55.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:55.624 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:55.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:55.666 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:55.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:55.767 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:55.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:55.806 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:55.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:55.907 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:55.947 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:55.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:55.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:56.049 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:56.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:56.079 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:56.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:56.180 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:56.212 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:56.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:56.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:56.314 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:56.345 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:56.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:56.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:56.357 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:52:56.358 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:52:56.446 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:56.477 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:56.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:56.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:56.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:52:56.580 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:56.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:56.615 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:56.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:56.653 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:52:56.669 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:52:56 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:52:56] "GET /metrics HTTP/1.1" 200 36542 "" "Prometheus/2.51.0" 2024-09-13T14:52:56.716 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:56.754 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:56.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:56.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:56.856 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:56.908 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:56.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:56.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:57.010 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:57.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:57.061 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:57.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:57.111 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is set 2024-09-13T14:52:57.163 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:57.188 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:57.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:57.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:57.290 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:57.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:57.326 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:57.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:57.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:57 smithi102 ceph-mon[110183]: pgmap v1386: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 873 op/s; 10023/315072 objects degraded (3.181%); 7938/315072 objects misplaced (2.519%); 6.7 MiB/s, 107 objects/s recovering 2024-09-13T14:52:57.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:57 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1366474903' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:52:57.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:57 smithi102 ceph-mon[110183]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:52:57.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:57 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1366474903' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:52:57.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:57 smithi102 ceph-mon[115269]: pgmap v1386: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 873 op/s; 10023/315072 objects degraded (3.181%); 7938/315072 objects misplaced (2.519%); 6.7 MiB/s, 107 objects/s recovering 2024-09-13T14:52:57.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:57 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1366474903' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:52:57.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:57 smithi102 ceph-mon[115269]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:52:57.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:57 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1366474903' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:52:57.427 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:57.448 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:57.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:57.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:57.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:57 smithi195 ceph-mon[104262]: pgmap v1386: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 873 op/s; 10023/315072 objects degraded (3.181%); 7938/315072 objects misplaced (2.519%); 6.7 MiB/s, 107 objects/s recovering 2024-09-13T14:52:57.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:57 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1366474903' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:52:57.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:57 smithi195 ceph-mon[104262]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:52:57.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:57 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1366474903' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:52:57.550 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:57.584 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:57.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:57.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:57.686 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:57.720 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:57.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:57.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:57.821 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:57.850 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:57.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:57.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:57.953 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:57.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:57.990 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:57.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:58.091 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:58.121 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:58.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:58.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:58.142 INFO:teuthology.orchestra.run.smithi102.stderr:reweighted osd.4 to 1 (10000) 2024-09-13T14:52:58.223 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:58.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:58.261 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:58.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:58.362 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:58.394 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:58.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:58.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:58.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:58 smithi102 ceph-mon[110183]: osdmap e409: 8 total, 8 up, 8 in 2024-09-13T14:52:58.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:58 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1982853293' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:52:58.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:58 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:52:58.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:58 smithi102 ceph-mon[115269]: osdmap e409: 8 total, 8 up, 8 in 2024-09-13T14:52:58.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:58 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1982853293' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:52:58.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:58 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:52:58.496 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:58.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:58 smithi195 ceph-mon[104262]: osdmap e409: 8 total, 8 up, 8 in 2024-09-13T14:52:58.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:58 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1982853293' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:52:58.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:58 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:52:58.529 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:58.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:58.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:58.631 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:58.653 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:58.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:58.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:58.755 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:58.792 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:58.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:58.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:58.894 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:58.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:58.929 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:58.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:59.030 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:59.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:59.066 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:59.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:59.167 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:59.206 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:59.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:59.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:59.308 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:59.347 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:59.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:59.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:59.396 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:59 smithi102 ceph-mon[110183]: pgmap v1388: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 6.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s; 9687/307119 objects degraded (3.154%); 7588/307119 objects misplaced (2.471%); 8.2 MiB/s, 131 objects/s recovering 2024-09-13T14:52:59.396 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:59 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T14:52:59.396 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:59 smithi102 ceph-mon[110183]: osdmap e410: 8 total, 8 up, 8 in 2024-09-13T14:52:59.396 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:52:59 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 9687/307119 objects degraded (3.154%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:52:59.396 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:59 smithi102 ceph-mon[115269]: pgmap v1388: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 6.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s; 9687/307119 objects degraded (3.154%); 7588/307119 objects misplaced (2.471%); 8.2 MiB/s, 131 objects/s recovering 2024-09-13T14:52:59.396 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:59 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T14:52:59.396 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:59 smithi102 ceph-mon[115269]: osdmap e410: 8 total, 8 up, 8 in 2024-09-13T14:52:59.397 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:52:59 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 9687/307119 objects degraded (3.154%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:52:59.449 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:59.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:59.479 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:59.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:59.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:59 smithi195 ceph-mon[104262]: pgmap v1388: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 6.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s; 9687/307119 objects degraded (3.154%); 7588/307119 objects misplaced (2.471%); 8.2 MiB/s, 131 objects/s recovering 2024-09-13T14:52:59.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:59 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T14:52:59.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:59 smithi195 ceph-mon[104262]: osdmap e410: 8 total, 8 up, 8 in 2024-09-13T14:52:59.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:52:59 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 9687/307119 objects degraded (3.154%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:52:59.580 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:59.619 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:59.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:59.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:59.720 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:59.755 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:52:59.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:59.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:59.857 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:59.889 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:52:59.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:59.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:59.991 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:00.025 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:00.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:00.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:00.127 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:00.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:00.167 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:00.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:00.268 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:00.301 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:00.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:00.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:00.404 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:00.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:00.436 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:00.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:00.537 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:00.567 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:00.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:00.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:00.669 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:00.701 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:00.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:00.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:00.803 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:00.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:00.834 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:00.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:00.935 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:00.970 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:00.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:00.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:01.072 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:01.110 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:01.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:01.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:01.212 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:01.249 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:01.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:01.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:01.351 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:01.386 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:01.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:01.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:01.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:01 smithi102 ceph-mon[110183]: pgmap v1390: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 6.1 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 898 op/s; 9687/301632 objects degraded (3.212%); 7588/301632 objects misplaced (2.516%); 4.6 MiB/s, 73 objects/s recovering 2024-09-13T14:53:01.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:01 smithi102 ceph-mon[115269]: pgmap v1390: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 6.1 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 898 op/s; 9687/301632 objects degraded (3.212%); 7588/301632 objects misplaced (2.516%); 4.6 MiB/s, 73 objects/s recovering 2024-09-13T14:53:01.488 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:01.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:01 smithi195 ceph-mon[104262]: pgmap v1390: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 6.1 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 898 op/s; 9687/301632 objects degraded (3.212%); 7588/301632 objects misplaced (2.516%); 4.6 MiB/s, 73 objects/s recovering 2024-09-13T14:53:01.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:01.523 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:01.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:01.624 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:01.656 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:01.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:01.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:01.758 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:01.798 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:01.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:01.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:01.900 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:01.934 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:01.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:01.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:02.036 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:02.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:02.076 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:02.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:02.177 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:02.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:02.213 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:02.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:02.314 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:02.349 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:02.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:02.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:02.451 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:02.641 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:02.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:02.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:02.743 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:02.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:02.782 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:02.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:02.883 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:02.908 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:53:02.908 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:53:02.917 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:02.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:02.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:03.019 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:03.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:03.059 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:03.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:03.160 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:03.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:03 smithi102 ceph-mon[110183]: pgmap v1391: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 9381/291192 objects degraded (3.222%); 7020/291192 objects misplaced (2.411%); 8.9 MiB/s, 141 objects/s recovering 2024-09-13T14:53:03.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:03 smithi102 ceph-mon[115269]: pgmap v1391: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 9381/291192 objects degraded (3.222%); 7020/291192 objects misplaced (2.411%); 8.9 MiB/s, 141 objects/s recovering 2024-09-13T14:53:03.196 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:03.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:03.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:03.298 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:03.331 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:03.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:03.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:03.433 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:03.464 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:03.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:03.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:03.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:03 smithi195 ceph-mon[104262]: pgmap v1391: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 9381/291192 objects degraded (3.222%); 7020/291192 objects misplaced (2.411%); 8.9 MiB/s, 141 objects/s recovering 2024-09-13T14:53:03.566 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:03.604 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:03.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:03.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:03.706 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:03.742 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:03.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:03.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:03.844 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:03.878 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:03.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:03.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:03.979 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:04.020 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:04.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:04.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:04.091 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T14:53:04.122 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:04.154 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:04.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:04.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:04.256 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:04.300 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:04.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:04.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:04.402 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:04.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:04 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 9381/291192 objects degraded (3.222%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:53:04.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:04 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 9381/291192 objects degraded (3.222%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:53:04.439 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:04.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:04.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:04.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:04 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 9381/291192 objects degraded (3.222%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:53:04.541 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:04.574 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:04.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:04.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:04.676 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:04.740 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:04.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:04.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:04.749 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:04.842 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:04.879 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:04.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:04.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:04.982 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:05.016 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:05.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:05.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:05.118 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:05.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:05.156 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:05.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:05.199 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-13T14:53:05.244 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:05.257 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:05.300 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:05.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:05.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:05.402 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:05.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:05 smithi102 ceph-mon[110183]: pgmap v1392: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 5.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.23k op/s; 9381/285528 objects degraded (3.285%); 7020/285528 objects misplaced (2.459%); 7.0 MiB/s, 111 objects/s recovering 2024-09-13T14:53:05.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:05 smithi102 ceph-mon[115269]: pgmap v1392: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 5.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.23k op/s; 9381/285528 objects degraded (3.285%); 7020/285528 objects misplaced (2.459%); 7.0 MiB/s, 111 objects/s recovering 2024-09-13T14:53:05.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:05.449 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:05.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:05.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:05 smithi195 ceph-mon[104262]: pgmap v1392: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 5.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.23k op/s; 9381/285528 objects degraded (3.285%); 7020/285528 objects misplaced (2.459%); 7.0 MiB/s, 111 objects/s recovering 2024-09-13T14:53:05.550 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:05.578 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:05.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:05.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:05.680 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:05.720 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:05.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:05.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:05.822 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:05.858 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:05.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:05.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:05.960 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:05.997 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:05.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:05.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:06.099 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:06.134 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:06.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:06.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:06.236 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:06.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:06.274 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:06.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:06.375 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:06.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:06.409 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:06.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:06.511 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:06.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:06.548 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:06.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:06.649 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:06.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:53:06 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:53:06] "GET /metrics HTTP/1.1" 200 36542 "" "Prometheus/2.51.0" 2024-09-13T14:53:06.685 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:06.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:06.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:06.708 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:53:06.788 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:06.840 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:06.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:06.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:06.941 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:06.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:06.982 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:06.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:07.083 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:07.124 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:07.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:07.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:07.226 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:07.262 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:07.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:07.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:07.364 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:07.400 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:07.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:07.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:07.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:07 smithi195 ceph-mon[104262]: pgmap v1393: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 5.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 939 op/s; 9381/282840 objects degraded (3.317%); 6773/282840 objects misplaced (2.395%); 5.8 MiB/s, 92 objects/s recovering 2024-09-13T14:53:07.502 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:07.540 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:07 smithi102 ceph-mon[110183]: pgmap v1393: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 5.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 939 op/s; 9381/282840 objects degraded (3.317%); 6773/282840 objects misplaced (2.395%); 5.8 MiB/s, 92 objects/s recovering 2024-09-13T14:53:07.541 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:07 smithi102 ceph-mon[115269]: pgmap v1393: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 5.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 939 op/s; 9381/282840 objects degraded (3.317%); 6773/282840 objects misplaced (2.395%); 5.8 MiB/s, 92 objects/s recovering 2024-09-13T14:53:07.548 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:07.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:07.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:07.650 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:07.683 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:07.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:07.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:07.785 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:07.824 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:07.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:07.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:07.925 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:07.961 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:07.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:07.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:08.063 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:08.081 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:08.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:08.091 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:08.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:08.193 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:08.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:08.239 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:08.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:08.340 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:08.370 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:08.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:08.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:08.472 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:08.506 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:08.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:08.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:08.608 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:08.644 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:08.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:08.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:08.745 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:08.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:08.783 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:08.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:08.884 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:08.915 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:08.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:08.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:09.018 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:09.056 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:09.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:09.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:09.159 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:09.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:09.190 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:09.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:09.291 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:09.316 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:09.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:09.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:09.418 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:09.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:09.453 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:09.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:09.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:09 smithi195 ceph-mon[104262]: pgmap v1394: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s; 9013/274512 objects degraded (3.283%); 6413/274512 objects misplaced (2.336%); 7.2 MiB/s, 115 objects/s recovering 2024-09-13T14:53:09.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:09 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 9013/274512 objects degraded (3.283%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:53:09.507 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:09 smithi102 ceph-mon[110183]: pgmap v1394: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s; 9013/274512 objects degraded (3.283%); 6413/274512 objects misplaced (2.336%); 7.2 MiB/s, 115 objects/s recovering 2024-09-13T14:53:09.507 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:09 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 9013/274512 objects degraded (3.283%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:53:09.507 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:09 smithi102 ceph-mon[115269]: pgmap v1394: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s; 9013/274512 objects degraded (3.283%); 6413/274512 objects misplaced (2.336%); 7.2 MiB/s, 115 objects/s recovering 2024-09-13T14:53:09.508 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:09 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 9013/274512 objects degraded (3.283%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:53:09.555 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:09.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:09.589 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:09.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:09.690 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:09.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:09.722 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:09.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:09.790 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:53:09.790 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:53:09.823 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:09.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:09.856 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:09.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:09.958 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:09.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:09.992 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:09.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:10.093 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:10.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:10.128 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:10.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:10.229 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:10.258 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:10.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:10.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:10.353 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:10.360 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:10.385 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:10.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:10.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:10.398 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:53:10.399 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:53:10.433 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:53:10.433 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:53:10.468 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:53:10.469 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:53:10.487 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:10.503 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:53:10.503 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:53:10.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:10.533 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:10.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:10.535 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:53:10.535 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:53:10.564 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:53:10.565 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:53:10.594 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:53:10.594 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:53:10.634 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:10.635 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:53:10.635 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:53:10.673 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:10.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:10.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:10.776 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:10.811 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:10.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:10.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:10.913 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:10.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:10.956 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:10.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:10.970 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T14:53:11.057 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:11.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:11.092 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:11.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:11.194 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:11.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:11.253 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:11.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:11.353 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:11.394 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:11.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:11.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:11.448 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:11 smithi102 ceph-mon[110183]: pgmap v1395: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 5.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 9013/269046 objects degraded (3.350%); 6413/269046 objects misplaced (2.384%); 6.3 MiB/s, 100 objects/s recovering 2024-09-13T14:53:11.448 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:11 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1251902974' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:53:11.448 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:11 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:53:11.448 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:11 smithi102 ceph-mon[115269]: pgmap v1395: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 5.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 9013/269046 objects degraded (3.350%); 6413/269046 objects misplaced (2.384%); 6.3 MiB/s, 100 objects/s recovering 2024-09-13T14:53:11.448 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:11 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1251902974' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:53:11.448 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:11 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:53:11.496 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:11.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:11 smithi195 ceph-mon[104262]: pgmap v1395: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 5.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 9013/269046 objects degraded (3.350%); 6413/269046 objects misplaced (2.384%); 6.3 MiB/s, 100 objects/s recovering 2024-09-13T14:53:11.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:11 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1251902974' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:53:11.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:11 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:53:11.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:11.531 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:11.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:11.632 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:11.667 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:11.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:11.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:11.769 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:11.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:11.809 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:11.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:11.910 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:11.946 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:11.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:11.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:12.048 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:12.085 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:12.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:12.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:12.187 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:12.231 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:12.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:12.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:12.333 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:12.372 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:12.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:12.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:12.474 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:12.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:12.513 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:12.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:12.614 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:12.648 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:12.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:12.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:12.750 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:12.783 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:12.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:12.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:12.885 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:12.921 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:12.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:12.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:13.022 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:13.057 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:13.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:13.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:13.159 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:13.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:13.196 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:13.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:13.298 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:13.332 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:13.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:13.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:13.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:13 smithi102 ceph-mon[110183]: pgmap v1396: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 5.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.20k op/s; 8687/258555 objects degraded (3.360%); 5774/258555 objects misplaced (2.233%); 9.2 MiB/s, 147 objects/s recovering 2024-09-13T14:53:13.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:13 smithi102 ceph-mon[115269]: pgmap v1396: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 5.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.20k op/s; 8687/258555 objects degraded (3.360%); 5774/258555 objects misplaced (2.233%); 9.2 MiB/s, 147 objects/s recovering 2024-09-13T14:53:13.434 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:13.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:13.471 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:13.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:13.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:13 smithi195 ceph-mon[104262]: pgmap v1396: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 5.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.20k op/s; 8687/258555 objects degraded (3.360%); 5774/258555 objects misplaced (2.233%); 9.2 MiB/s, 147 objects/s recovering 2024-09-13T14:53:13.572 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:13.608 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:13.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:13.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:13.710 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:13.756 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:13.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:13.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:13.858 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:13.871 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:53:13.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:13.891 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:13.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:13.992 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:14.015 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:14.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:14.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:14.117 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:14.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:14.152 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:14.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:14.253 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:14.281 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:14.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:14.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:14.383 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:14.417 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:14.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:14.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:14.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:14 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 8687/258555 objects degraded (3.360%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:53:14.519 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:14.554 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:14.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:14.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:14.656 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:14.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:14 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 8687/258555 objects degraded (3.360%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:53:14.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:14 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 8687/258555 objects degraded (3.360%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:53:14.690 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:14.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:14.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:14.792 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:14.823 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:14.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:14.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:14.925 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:14.964 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:14.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:14.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:15.066 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:15.107 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:15.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:15.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:15.209 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:15.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:15.262 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:15.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:15.363 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:15.395 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:15.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:15.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:15.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:15 smithi102 ceph-mon[110183]: pgmap v1397: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 8687/252960 objects degraded (3.434%); 5774/252960 objects misplaced (2.283%); 6.4 MiB/s, 101 objects/s recovering 2024-09-13T14:53:15.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:15 smithi102 ceph-mon[115269]: pgmap v1397: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 8687/252960 objects degraded (3.434%); 5774/252960 objects misplaced (2.283%); 6.4 MiB/s, 101 objects/s recovering 2024-09-13T14:53:15.498 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:15.531 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:15.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:15.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:15.633 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:15.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:15.670 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:15.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:15.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:15 smithi195 ceph-mon[104262]: pgmap v1397: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 8687/252960 objects degraded (3.434%); 5774/252960 objects misplaced (2.283%); 6.4 MiB/s, 101 objects/s recovering 2024-09-13T14:53:15.771 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:15.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:15.808 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:15.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:15.909 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:15.942 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:15.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:15.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:16.044 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:16.065 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:16.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:16.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:16.166 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:16.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:16.203 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:16.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:16.304 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:16.335 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:16.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:16.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:16.437 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:16.476 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:16.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:16.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:16.578 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:16.609 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:16.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:16.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:16.711 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:16.749 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:16.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:16.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:16.852 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:16.873 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:16.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:16.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:16.919 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:53:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:53:16] "GET /metrics HTTP/1.1" 200 36539 "" "Prometheus/2.51.0" 2024-09-13T14:53:16.976 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:17.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:17.018 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:17.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:17.119 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:17.151 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:17.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:17.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:17.253 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:17.294 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:17.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:17.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:17.396 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:17.426 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:17.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:17.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:17.527 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:17.562 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:17.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:17.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:17.663 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:17.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:17 smithi102 ceph-mon[110183]: pgmap v1398: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 973 op/s; 8687/250458 objects degraded (3.468%); 5503/250458 objects misplaced (2.197%); 7.8 MiB/s, 124 objects/s recovering 2024-09-13T14:53:17.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:17 smithi102 ceph-mon[115269]: pgmap v1398: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 973 op/s; 8687/250458 objects degraded (3.468%); 5503/250458 objects misplaced (2.197%); 7.8 MiB/s, 124 objects/s recovering 2024-09-13T14:53:17.686 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:17.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:17.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:17.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:17 smithi195 ceph-mon[104262]: pgmap v1398: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 973 op/s; 8687/250458 objects degraded (3.468%); 5503/250458 objects misplaced (2.197%); 7.8 MiB/s, 124 objects/s recovering 2024-09-13T14:53:17.787 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:17.823 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:17.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:17.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:17.925 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:17.965 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:17.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:17.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:18.067 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:18.107 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:18.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:18.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:18.209 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:18.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:18.249 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:18.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:18.350 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:18.404 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:18.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:18.408 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:18.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:18.410 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:18.510 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:18.563 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:18.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:18.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:18.665 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:18.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:18.699 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:18.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:18.800 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:18.830 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:18.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:18.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:18.931 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:18.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:18.965 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:18.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:19.066 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:19.103 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:19.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:19.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:19.205 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:19.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:19.237 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:19.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:19.338 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:19.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:19.381 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:19.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:19.482 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:19.514 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:19.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:19.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:19.567 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:19 smithi102 ceph-mon[110183]: pgmap v1399: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 4.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s; 8343/241596 objects degraded (3.453%); 5119/241596 objects misplaced (2.119%); 8.4 MiB/s, 134 objects/s recovering 2024-09-13T14:53:19.567 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:19 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 8343/241596 objects degraded (3.453%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:53:19.567 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:19 smithi102 ceph-mon[115269]: pgmap v1399: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 4.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s; 8343/241596 objects degraded (3.453%); 5119/241596 objects misplaced (2.119%); 8.4 MiB/s, 134 objects/s recovering 2024-09-13T14:53:19.567 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:19 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 8343/241596 objects degraded (3.453%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:53:19.617 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:19.650 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:19.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:19.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:19.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:19 smithi195 ceph-mon[104262]: pgmap v1399: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 4.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s; 8343/241596 objects degraded (3.453%); 5119/241596 objects misplaced (2.119%); 8.4 MiB/s, 134 objects/s recovering 2024-09-13T14:53:19.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:19 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 8343/241596 objects degraded (3.453%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:53:19.753 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:19.783 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:19.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:19.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:19.885 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:19.927 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:19.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:19.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:20.029 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:20.067 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:20.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:20.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:20.171 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:20.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:20.203 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:20.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:20.304 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:20.324 INFO:teuthology.orchestra.run.smithi102.stderr:reweighted osd.6 to 1 (10000) 2024-09-13T14:53:20.349 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:20.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:20.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:20.452 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:20.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:20.482 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:20.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:20.584 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:20.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:20.615 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:20.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:20.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:20 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/788180519' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:53:20.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:20 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:53:20.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:20 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/788180519' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:53:20.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:20 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:53:20.716 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:20.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:20 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/788180519' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:53:20.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:20 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:53:20.752 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:20.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:20.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:20.854 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:20.892 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:20.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:20.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:20.994 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:21.034 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:21.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:21.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:21.135 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:21.170 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:21.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:21.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:21.272 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:21.311 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:21.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:21.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:21.334 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is unset 2024-09-13T14:53:21.412 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:21.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:21.450 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:21.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:21.551 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:21.583 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:21.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:21.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:21.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:21 smithi102 ceph-mon[110183]: pgmap v1400: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 4.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 990 op/s; 8343/238842 objects degraded (3.493%); 5119/238842 objects misplaced (2.143%); 6.5 MiB/s, 103 objects/s recovering 2024-09-13T14:53:21.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:21 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T14:53:21.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:21 smithi102 ceph-mon[110183]: osdmap e411: 8 total, 8 up, 8 in 2024-09-13T14:53:21.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:21 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3741543725' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:53:21.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:21 smithi102 ceph-mon[115269]: pgmap v1400: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 4.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 990 op/s; 8343/238842 objects degraded (3.493%); 5119/238842 objects misplaced (2.143%); 6.5 MiB/s, 103 objects/s recovering 2024-09-13T14:53:21.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:21 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T14:53:21.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:21 smithi102 ceph-mon[115269]: osdmap e411: 8 total, 8 up, 8 in 2024-09-13T14:53:21.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:21 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3741543725' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:53:21.686 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:21.727 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:21.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:21.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:21.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:21 smithi195 ceph-mon[104262]: pgmap v1400: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 4.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 990 op/s; 8343/238842 objects degraded (3.493%); 5119/238842 objects misplaced (2.143%); 6.5 MiB/s, 103 objects/s recovering 2024-09-13T14:53:21.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:21 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T14:53:21.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:21 smithi195 ceph-mon[104262]: osdmap e411: 8 total, 8 up, 8 in 2024-09-13T14:53:21.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:21 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3741543725' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:53:21.765 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-13T14:53:21.830 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:21.877 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:21.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:21.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:21.980 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:22.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:22.013 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:22.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:22.114 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:22.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:22.156 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:22.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:22.257 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:22.290 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:22.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:22.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:22.392 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:22.429 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:22.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:22.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:22.531 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:22.544 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:22 smithi102 ceph-mon[110183]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:53:22.544 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:22 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3741543725' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:53:22.544 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:22 smithi102 ceph-mon[110183]: osdmap e412: 8 total, 8 up, 8 in 2024-09-13T14:53:22.545 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:22 smithi102 ceph-mon[115269]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:53:22.545 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:22 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3741543725' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:53:22.545 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:22 smithi102 ceph-mon[115269]: osdmap e412: 8 total, 8 up, 8 in 2024-09-13T14:53:22.568 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:22.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:22.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:22.671 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:22.718 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:22.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:22.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:22.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:22 smithi195 ceph-mon[104262]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:53:22.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:22 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3741543725' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:53:22.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:22 smithi195 ceph-mon[104262]: osdmap e412: 8 total, 8 up, 8 in 2024-09-13T14:53:22.820 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:22.853 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:22.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:22.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:22.955 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:23.002 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:23.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:23.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:23.104 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:23.141 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:23.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:23.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:23.243 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:23.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:23.277 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:23.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:23.378 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:23.418 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:23.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:23.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:23.520 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:23.559 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:23.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:23.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:23.648 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:23 smithi195 ceph-mon[104262]: pgmap v1403: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 4.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 7959/226515 objects degraded (3.514%); 4415/226515 objects misplaced (1.949%); 10 MiB/s, 165 objects/s recovering 2024-09-13T14:53:23.649 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:23 smithi195 ceph-mon[104262]: 9.7 scrub starts 2024-09-13T14:53:23.655 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:23 smithi102 ceph-mon[110183]: pgmap v1403: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 4.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 7959/226515 objects degraded (3.514%); 4415/226515 objects misplaced (1.949%); 10 MiB/s, 165 objects/s recovering 2024-09-13T14:53:23.655 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:23 smithi102 ceph-mon[110183]: 9.7 scrub starts 2024-09-13T14:53:23.655 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:23 smithi102 ceph-mon[115269]: pgmap v1403: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 4.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 7959/226515 objects degraded (3.514%); 4415/226515 objects misplaced (1.949%); 10 MiB/s, 165 objects/s recovering 2024-09-13T14:53:23.656 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:23 smithi102 ceph-mon[115269]: 9.7 scrub starts 2024-09-13T14:53:23.661 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:23.703 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:23.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:23.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:23.805 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:23.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:23.841 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:23.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:23.942 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:23.970 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:23.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:23.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:24.072 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:24.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:24.106 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:24.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:24.207 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:24.247 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:24.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:24.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:24.349 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:24.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:24.388 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:24.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:24.489 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:24.527 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:24.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:24.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:24.549 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:24 smithi102 ceph-mon[110183]: 9.6 scrub starts 2024-09-13T14:53:24.549 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:24 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 7959/226515 objects degraded (3.514%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:53:24.549 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:24 smithi102 ceph-mon[115269]: 9.6 scrub starts 2024-09-13T14:53:24.549 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:24 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 7959/226515 objects degraded (3.514%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:53:24.629 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:24.659 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:24.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:24.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:24.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:24 smithi195 ceph-mon[104262]: 9.6 scrub starts 2024-09-13T14:53:24.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:24 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 7959/226515 objects degraded (3.514%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:53:24.761 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:24.810 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:24.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:24.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:24.863 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:53:24.863 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:53:24.863 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:53:24.863 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:53:24.863 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:53:24.863 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:53:24.863 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:53:24.864 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:53:24.864 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:53:24.864 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:53:24.864 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:53:24.864 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:53:24.864 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:53:24.864 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:53:24.864 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:53:24.864 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:53:24.913 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:24.944 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:24.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:24.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:25.006 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:53:25.046 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:25.079 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:25.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:25.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:25.181 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:25.259 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:25.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:25.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:25.361 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:25.395 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:25.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:25.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:25.457 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:25 smithi102 ceph-mon[110183]: pgmap v1404: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 4.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s; 7959/223584 objects degraded (3.560%); 4415/223584 objects misplaced (1.975%); 8.2 MiB/s, 131 objects/s recovering 2024-09-13T14:53:25.457 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:25 smithi102 ceph-mon[110183]: 9.6 scrub ok 2024-09-13T14:53:25.458 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:25 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2400077101' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:53:25.458 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:25 smithi102 ceph-mon[115269]: pgmap v1404: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 4.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s; 7959/223584 objects degraded (3.560%); 4415/223584 objects misplaced (1.975%); 8.2 MiB/s, 131 objects/s recovering 2024-09-13T14:53:25.458 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:25 smithi102 ceph-mon[115269]: 9.6 scrub ok 2024-09-13T14:53:25.458 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:25 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2400077101' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:53:25.469 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:25.496 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:25.505 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:53:25.506 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:53:25.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:25.541 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:25.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:25.642 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:25.679 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:25.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:25.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:25.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:25 smithi195 ceph-mon[104262]: pgmap v1404: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 4.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s; 7959/223584 objects degraded (3.560%); 4415/223584 objects misplaced (1.975%); 8.2 MiB/s, 131 objects/s recovering 2024-09-13T14:53:25.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:25 smithi195 ceph-mon[104262]: 9.6 scrub ok 2024-09-13T14:53:25.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:25 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2400077101' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:53:25.781 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:25.816 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:25.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:25.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:25.918 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:25.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:25.961 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:25.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:26.063 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:26.105 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:26.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:26.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:26.206 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:26.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:26.245 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:26.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:26.346 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:26.373 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:26.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:26.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:26.475 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:26.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:26.514 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:26.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:26.615 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:26.651 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:26.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:26.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:26.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:26 smithi102 ceph-mon[110183]: 9.12 scrub starts 2024-09-13T14:53:26.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:53:26.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:26 smithi102 ceph-mon[115269]: 9.12 scrub starts 2024-09-13T14:53:26.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:53:26.669 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:53:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:53:26] "GET /metrics HTTP/1.1" 200 36535 "" "Prometheus/2.51.0" 2024-09-13T14:53:26.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:26 smithi195 ceph-mon[104262]: 9.12 scrub starts 2024-09-13T14:53:26.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:53:26.753 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:26.798 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:26.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:26.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:26.900 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:26.938 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:26.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:26.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:27.040 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:27.085 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:27.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:27.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:27.107 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T14:53:27.187 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:27.224 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:27.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:27.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:27.326 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:27.357 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:27.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:27.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:27.458 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:27.464 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:53:27.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:27 smithi195 ceph-mon[104262]: pgmap v1405: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 4.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 848 op/s; 7959/221223 objects degraded (3.598%); 4123/221223 objects misplaced (1.864%); 7.7 MiB/s, 122 objects/s recovering 2024-09-13T14:53:27.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:27 smithi195 ceph-mon[104262]: 9.12 scrub ok 2024-09-13T14:53:27.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:27.518 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:27.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:27.588 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:27.619 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:27.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:27 smithi102 ceph-mon[110183]: pgmap v1405: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 4.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 848 op/s; 7959/221223 objects degraded (3.598%); 4123/221223 objects misplaced (1.864%); 7.7 MiB/s, 122 objects/s recovering 2024-09-13T14:53:27.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:27 smithi102 ceph-mon[110183]: 9.12 scrub ok 2024-09-13T14:53:27.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:27 smithi102 ceph-mon[115269]: pgmap v1405: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 4.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 848 op/s; 7959/221223 objects degraded (3.598%); 4123/221223 objects misplaced (1.864%); 7.7 MiB/s, 122 objects/s recovering 2024-09-13T14:53:27.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:27 smithi102 ceph-mon[115269]: 9.12 scrub ok 2024-09-13T14:53:27.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:27.674 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:27.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:27.775 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:27.814 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:27.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:27.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:27.916 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:27.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:27.952 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:27.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:28.053 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:28.075 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:53:28.085 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:28.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:28.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:28.186 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:28.219 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:28.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:28.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:28.320 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:28.361 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:28.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:28.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:28.463 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:28.510 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:28.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:28.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:28.612 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:28.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:28.652 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:28.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:28.753 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:28.795 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:28.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:28.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:28.897 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:28.958 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:28.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:28.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:29.060 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:29.097 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:29.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:29.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:29.199 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:29.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:29.241 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:29.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:29.342 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:29.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:29.388 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:29.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:29.464 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:29 smithi195 ceph-mon[104262]: pgmap v1406: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 4.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 7653/210921 objects degraded (3.628%); 3715/210921 objects misplaced (1.761%) 2024-09-13T14:53:29.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:29 smithi195 ceph-mon[104262]: 9.7 scrub starts 2024-09-13T14:53:29.465 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:29 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 7653/210921 objects degraded (3.628%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:53:29.490 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:29.502 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:29 smithi102 ceph-mon[115269]: pgmap v1406: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 4.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 7653/210921 objects degraded (3.628%); 3715/210921 objects misplaced (1.761%) 2024-09-13T14:53:29.503 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:29 smithi102 ceph-mon[115269]: 9.7 scrub starts 2024-09-13T14:53:29.503 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:29 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 7653/210921 objects degraded (3.628%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:53:29.503 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:29 smithi102 ceph-mon[110183]: pgmap v1406: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 4.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 7653/210921 objects degraded (3.628%); 3715/210921 objects misplaced (1.761%) 2024-09-13T14:53:29.503 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:29 smithi102 ceph-mon[110183]: 9.7 scrub starts 2024-09-13T14:53:29.503 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:29 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 7653/210921 objects degraded (3.628%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:53:29.531 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:29.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:29.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:29.628 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:53:29.629 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:53:29.633 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:29.669 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:29.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:29.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:29.771 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:29.806 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:29.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:29.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:29.908 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:29.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:29.950 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:29.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:30.051 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:30.078 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:30.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:30.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:30.180 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:30.243 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:30.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:30.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:30.321 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T14:53:30.346 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:30.384 INFO:teuthology.orchestra.run.smithi102.stderr:reweighted osd.7 to 1 (10000) 2024-09-13T14:53:30.387 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:30.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:30.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:30.489 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:30.523 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:30.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:30.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:30.625 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:30.657 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:30.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:30.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:30.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:30 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1028049242' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T14:53:30.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:30 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T14:53:30.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:30 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1028049242' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T14:53:30.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:30 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T14:53:30.670 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:53:30.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:30 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1028049242' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T14:53:30.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:30 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T14:53:30.759 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:30.799 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:30.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:30.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:30.901 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:30.931 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:30.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:30.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:31.033 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:31.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:31.075 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:31.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:31.177 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:31.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:31.217 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:31.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:31.318 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:31.355 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:31.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:31.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:31.424 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is unset 2024-09-13T14:53:31.457 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:31.502 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:31.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:31.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:31.604 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:31.641 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:31.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:31.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:31.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:31 smithi102 ceph-mon[110183]: pgmap v1407: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 4.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 7653/208146 objects degraded (3.677%); 3715/208146 objects misplaced (1.785%) 2024-09-13T14:53:31.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:31 smithi102 ceph-mon[110183]: 9.7 scrub starts 2024-09-13T14:53:31.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:31 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T14:53:31.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:31 smithi102 ceph-mon[110183]: osdmap e413: 8 total, 8 up, 8 in 2024-09-13T14:53:31.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:31 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2223630700' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:53:31.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:31 smithi102 ceph-mon[115269]: pgmap v1407: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 4.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 7653/208146 objects degraded (3.677%); 3715/208146 objects misplaced (1.785%) 2024-09-13T14:53:31.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:31 smithi102 ceph-mon[115269]: 9.7 scrub starts 2024-09-13T14:53:31.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:31 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T14:53:31.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:31 smithi102 ceph-mon[115269]: osdmap e413: 8 total, 8 up, 8 in 2024-09-13T14:53:31.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:31 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2223630700' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:53:31.743 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:31.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:31 smithi195 ceph-mon[104262]: pgmap v1407: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 4.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 7653/208146 objects degraded (3.677%); 3715/208146 objects misplaced (1.785%) 2024-09-13T14:53:31.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:31 smithi195 ceph-mon[104262]: 9.7 scrub starts 2024-09-13T14:53:31.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:31 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T14:53:31.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:31 smithi195 ceph-mon[104262]: osdmap e413: 8 total, 8 up, 8 in 2024-09-13T14:53:31.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:31 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2223630700' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:53:31.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:31.782 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:31.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:31.856 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:31.884 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:31.888 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-13T14:53:31.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:31.914 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:31.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:32.016 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:32.055 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:32.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:32.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:32.157 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:32.189 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:32.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:32.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:32.291 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:32.312 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:32.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:32.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:32.414 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:32.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:32.450 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:32.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:32.458 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:32 smithi102 ceph-mon[110183]: 9.7 scrub starts 2024-09-13T14:53:32.458 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:32 smithi102 ceph-mon[110183]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:53:32.458 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:32 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2223630700' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:53:32.458 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:32 smithi102 ceph-mon[110183]: osdmap e414: 8 total, 8 up, 8 in 2024-09-13T14:53:32.458 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:32 smithi102 ceph-mon[115269]: 9.7 scrub starts 2024-09-13T14:53:32.458 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:32 smithi102 ceph-mon[115269]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:53:32.458 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:32 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2223630700' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:53:32.458 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:32 smithi102 ceph-mon[115269]: osdmap e414: 8 total, 8 up, 8 in 2024-09-13T14:53:32.552 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:32.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:32.587 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:32.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:32.688 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:32.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:32.710 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:32.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:32.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:32 smithi195 ceph-mon[104262]: 9.7 scrub starts 2024-09-13T14:53:32.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:32 smithi195 ceph-mon[104262]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:53:32.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:32 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2223630700' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:53:32.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:32 smithi195 ceph-mon[104262]: osdmap e414: 8 total, 8 up, 8 in 2024-09-13T14:53:32.811 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:32.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:32.849 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:32.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:32.950 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:32.987 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:32.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:32.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:33.089 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:33.123 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:33.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:33.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:33.225 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:33.257 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:33.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:33.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:33.359 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:33.381 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:33.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:33.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:33.483 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:33.514 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:33.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:33.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:33.616 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:33.652 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:33.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:33.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:33.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:33 smithi102 ceph-mon[110183]: pgmap v1410: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 4.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 7341/196128 objects degraded (3.743%); 2979/196128 objects misplaced (1.519%) 2024-09-13T14:53:33.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:33 smithi102 ceph-mon[110183]: 9.7 deep-scrub starts 2024-09-13T14:53:33.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:33 smithi102 ceph-mon[115269]: pgmap v1410: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 4.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 7341/196128 objects degraded (3.743%); 2979/196128 objects misplaced (1.519%) 2024-09-13T14:53:33.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:33 smithi102 ceph-mon[115269]: 9.7 deep-scrub starts 2024-09-13T14:53:33.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:33 smithi195 ceph-mon[104262]: pgmap v1410: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 4.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 7341/196128 objects degraded (3.743%); 2979/196128 objects misplaced (1.519%) 2024-09-13T14:53:33.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:33 smithi195 ceph-mon[104262]: 9.7 deep-scrub starts 2024-09-13T14:53:33.754 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:33.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:33.795 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:33.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:33.865 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:33.896 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:34.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:34.024 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:34.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:34.125 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:34.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:34.161 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:34.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:34.262 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:34.296 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:34.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:34.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:34.398 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:34.431 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:34.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:34.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:34.533 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:34.559 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:34.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:34.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:34.661 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:34.697 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:34.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:34.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:34.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:34 smithi195 ceph-mon[104262]: 9.7 scrub starts 2024-09-13T14:53:34.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:34 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 7341/196128 objects degraded (3.743%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:53:34.799 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:34.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:34.836 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:34.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:34.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:34 smithi102 ceph-mon[110183]: 9.7 scrub starts 2024-09-13T14:53:34.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:34 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 7341/196128 objects degraded (3.743%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:53:34.899 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:34 smithi102 ceph-mon[115269]: 9.7 scrub starts 2024-09-13T14:53:34.899 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:34 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 7341/196128 objects degraded (3.743%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:53:34.937 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:34.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:34.990 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:34.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:35.091 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:35.127 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:35.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:35.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:35.229 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:35.266 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:35.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:35.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:35.278 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:53:35.278 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:53:35.278 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:53:35.278 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:53:35.292 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:53:35.292 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:53:35.332 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:53:35.333 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:53:35.359 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:53:35.364 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:53:35.364 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:53:35.368 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:35.402 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:53:35.402 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:53:35.413 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:35.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:35.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:35.433 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:53:35.434 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:53:35.467 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:53:35.467 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:53:35.515 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:35.552 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:35.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:35.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:35.625 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:35 smithi195 ceph-mon[104262]: pgmap v1411: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 3.9 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 7341/193398 objects degraded (3.796%); 2979/193398 objects misplaced (1.540%) 2024-09-13T14:53:35.625 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:35 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/958738648' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:53:35.654 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:35.688 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:35.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:35.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:35.790 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:35.827 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:35.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:35.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:35.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:35 smithi102 ceph-mon[110183]: pgmap v1411: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 3.9 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 7341/193398 objects degraded (3.796%); 2979/193398 objects misplaced (1.540%) 2024-09-13T14:53:35.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:35 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/958738648' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:53:35.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:35 smithi102 ceph-mon[115269]: pgmap v1411: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 3.9 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 7341/193398 objects degraded (3.796%); 2979/193398 objects misplaced (1.540%) 2024-09-13T14:53:35.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:35 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/958738648' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:53:35.928 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:35.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:35.963 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:35.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:36.065 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:36.086 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:53:36.087 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-13T14:53:36.087 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-09-13T14:53:36.087 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-09-13T14:53:36.087 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:53:36.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:36.096 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:36.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:36.197 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:36.235 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:36.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:36.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:36.337 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:36.379 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:36.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:36.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:36.480 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:36.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:36.522 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:36.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:36.623 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:36.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:36.662 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:36.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:36.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:36 smithi195 ceph-mon[104262]: 9.7 scrub starts 2024-09-13T14:53:36.763 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:36.809 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:36.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:36.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:36.828 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:36 smithi102 ceph-mon[110183]: 9.7 scrub starts 2024-09-13T14:53:36.828 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:36 smithi102 ceph-mon[115269]: 9.7 scrub starts 2024-09-13T14:53:36.829 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:53:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:53:36] "GET /metrics HTTP/1.1" 200 36535 "" "Prometheus/2.51.0" 2024-09-13T14:53:36.911 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:36.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:36.953 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:36.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:37.025 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T14:53:37.054 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:37.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:37.116 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:37.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:37.217 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:37.264 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:37.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:37.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:37.366 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:37.386 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:53:37.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:37.394 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:37.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:37.495 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:37.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:37.544 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:37.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:37.645 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:37.687 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:37.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:37.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:37.711 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:37 smithi195 ceph-mon[104262]: pgmap v1412: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 3.9 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 825 op/s; 7341/191088 objects degraded (3.842%); 2625/191088 objects misplaced (1.374%); 8.0 MiB/s, 128 objects/s recovering 2024-09-13T14:53:37.789 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:37.811 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:37 smithi102 ceph-mon[110183]: pgmap v1412: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 3.9 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 825 op/s; 7341/191088 objects degraded (3.842%); 2625/191088 objects misplaced (1.374%); 8.0 MiB/s, 128 objects/s recovering 2024-09-13T14:53:37.811 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:37 smithi102 ceph-mon[115269]: pgmap v1412: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 3.9 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 825 op/s; 7341/191088 objects degraded (3.842%); 2625/191088 objects misplaced (1.374%); 8.0 MiB/s, 128 objects/s recovering 2024-09-13T14:53:37.840 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:37.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:37.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:37.942 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:37.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:37.978 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:37.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:38.079 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:38.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:38.115 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:38.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:38.195 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:38.216 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:38.265 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:38.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:38.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:38.369 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:38.405 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:38.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:38.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:38.506 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:38.540 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:38.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:38.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:38.643 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:38.678 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:38.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:38.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:38.736 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:38 smithi102 ceph-mon[115269]: pgmap v1413: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s; 7054/181392 objects degraded (3.889%); 2165/181392 objects misplaced (1.194%); 11 MiB/s, 175 objects/s recovering 2024-09-13T14:53:38.737 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:38 smithi102 ceph-mon[110183]: pgmap v1413: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s; 7054/181392 objects degraded (3.889%); 2165/181392 objects misplaced (1.194%); 11 MiB/s, 175 objects/s recovering 2024-09-13T14:53:38.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:38 smithi195 ceph-mon[104262]: pgmap v1413: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s; 7054/181392 objects degraded (3.889%); 2165/181392 objects misplaced (1.194%); 11 MiB/s, 175 objects/s recovering 2024-09-13T14:53:38.780 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:38.818 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:38.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:38.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:38.920 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:38.960 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:38.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:38.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:39.062 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:39.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:39.106 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:39.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:39.208 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:39.254 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:39.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:39.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:39.356 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:39.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:39.406 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:39.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:39.465 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:53:39.466 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-09-13T14:53:39.466 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-09-13T14:53:39.466 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-13T14:53:39.466 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.466 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:179128 7.1 7:8ae09e6f:::benchmark_data_smithi102_227395_object66313:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-09-13T14:53:39.466 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:43:51.034284+0000", 2024-09-13T14:53:39.466 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 588.42757371799996, 2024-09-13T14:53:39.466 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.61896007399999997, 2024-09-13T14:53:39.466 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:53:39.466 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:39.466 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:53:39.466 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:53:39.467 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:53:39.467 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 179128 2024-09-13T14:53:39.467 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.467 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:53:39.467 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.467 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:53:39.467 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.034284+0000", 2024-09-13T14:53:39.467 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:39.467 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.467 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.467 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:53:39.467 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.034284+0000", 2024-09-13T14:53:39.467 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:39.468 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.468 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.468 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:53:39.468 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.034283+0000", 2024-09-13T14:53:39.468 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:53:39.468 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.468 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.468 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:53:39.468 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.034290+0000", 2024-09-13T14:53:39.468 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9439999999999999e-06 2024-09-13T14:53:39.468 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.468 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.468 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:53:39.469 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.034300+0000", 2024-09-13T14:53:39.469 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.8810000000000008e-06 2024-09-13T14:53:39.469 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.469 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.469 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:53:39.469 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.034305+0000", 2024-09-13T14:53:39.469 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0610000000000003e-06 2024-09-13T14:53:39.469 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.469 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.469 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:53:39.469 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.034342+0000", 2024-09-13T14:53:39.469 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7280999999999998e-05 2024-09-13T14:53:39.469 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.470 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.470 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:53:39.470 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.034349+0000", 2024-09-13T14:53:39.470 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3180000000000001e-06 2024-09-13T14:53:39.470 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.470 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.470 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:53:39.470 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.646792+0000", 2024-09-13T14:53:39.470 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.61244286599999997 2024-09-13T14:53:39.470 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.470 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.470 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:53:39.470 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.646938+0000", 2024-09-13T14:53:39.471 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014614000000000001 2024-09-13T14:53:39.471 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.471 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.471 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:53:39.471 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.647012+0000", 2024-09-13T14:53:39.471 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3583000000000003e-05 2024-09-13T14:53:39.471 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.471 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.471 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:39.471 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.650051+0000", 2024-09-13T14:53:39.471 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0030396889999999999 2024-09-13T14:53:39.471 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.472 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.472 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:53:39.472 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.653152+0000", 2024-09-13T14:53:39.472 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003100364 2024-09-13T14:53:39.472 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.472 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.472 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:39.472 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.653187+0000", 2024-09-13T14:53:39.472 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5630000000000003e-05 2024-09-13T14:53:39.472 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.472 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.472 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:53:39.472 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.653220+0000", 2024-09-13T14:53:39.472 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2286999999999998e-05 2024-09-13T14:53:39.473 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.473 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.473 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:53:39.473 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.653244+0000", 2024-09-13T14:53:39.473 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4445000000000001e-05 2024-09-13T14:53:39.473 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:39.473 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:53:39.473 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:39.473 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.473 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.474 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.35379.0:179175 7.1 7:82c4bfbf:::benchmark_data_smithi102_227395_object66360:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-09-13T14:53:39.474 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:43:51.202947+0000", 2024-09-13T14:53:39.474 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 588.25891051300005, 2024-09-13T14:53:39.474 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.45122264299999998, 2024-09-13T14:53:39.474 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:53:39.474 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:39.474 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:53:39.474 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.35379", 2024-09-13T14:53:39.474 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/3928700042", 2024-09-13T14:53:39.475 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 179175 2024-09-13T14:53:39.475 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.475 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:53:39.475 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.475 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:53:39.475 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.202947+0000", 2024-09-13T14:53:39.475 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:39.475 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.476 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.476 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:53:39.476 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.202947+0000", 2024-09-13T14:53:39.476 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:39.476 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.476 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.476 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:53:39.476 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.202945+0000", 2024-09-13T14:53:39.476 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-09-13T14:53:39.476 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.476 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.476 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:53:39.476 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.202955+0000", 2024-09-13T14:53:39.477 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.7240000000000004e-06 2024-09-13T14:53:39.477 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.477 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.477 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:53:39.477 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.202975+0000", 2024-09-13T14:53:39.477 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0346000000000001e-05 2024-09-13T14:53:39.477 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.477 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.477 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:53:39.477 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.202994+0000", 2024-09-13T14:53:39.477 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8763999999999999e-05 2024-09-13T14:53:39.477 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.477 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.477 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:53:39.478 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.203058+0000", 2024-09-13T14:53:39.478 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4003000000000004e-05 2024-09-13T14:53:39.478 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.478 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.478 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:53:39.478 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.203071+0000", 2024-09-13T14:53:39.478 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2992e-05 2024-09-13T14:53:39.478 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.478 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.478 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:53:39.478 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.650083+0000", 2024-09-13T14:53:39.479 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.44701179600000002 2024-09-13T14:53:39.479 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.479 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.479 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:53:39.479 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.650203+0000", 2024-09-13T14:53:39.479 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000120798 2024-09-13T14:53:39.479 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.479 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.479 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:53:39.479 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.650292+0000", 2024-09-13T14:53:39.480 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8274000000000002e-05 2024-09-13T14:53:39.480 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.480 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.480 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:39.481 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.653268+0000", 2024-09-13T14:53:39.481 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0029767320000000002 2024-09-13T14:53:39.481 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.481 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.481 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:53:39.481 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.653292+0000", 2024-09-13T14:53:39.481 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3147e-05 2024-09-13T14:53:39.481 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.481 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.481 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:39.481 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.654125+0000", 2024-09-13T14:53:39.481 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00083319100000000005 2024-09-13T14:53:39.482 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.482 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.482 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:53:39.482 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.654150+0000", 2024-09-13T14:53:39.482 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5219999999999999e-05 2024-09-13T14:53:39.482 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.482 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.482 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:53:39.483 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:43:51.654170+0000", 2024-09-13T14:53:39.483 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9921000000000001e-05 2024-09-13T14:53:39.483 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:39.483 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:53:39.483 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:39.483 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.483 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.483 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.46157.0:45931 8.0 e314/304 8:02787af3:::benchmark_data_smithi102_313275_object45930:head v 314'2879, mlcod=314'2877)", 2024-09-13T14:53:39.484 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:46:32.805701+0000", 2024-09-13T14:53:39.484 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 426.65615632499998, 2024-09-13T14:53:39.484 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.15992261799999999, 2024-09-13T14:53:39.484 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:53:39.484 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:39.484 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:53:39.484 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.484 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:53:39.484 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.805701+0000", 2024-09-13T14:53:39.484 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:39.484 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.484 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.484 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:53:39.485 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.805701+0000", 2024-09-13T14:53:39.485 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:39.485 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.485 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.485 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:53:39.485 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.805701+0000", 2024-09-13T14:53:39.485 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:53:39.485 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.485 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.485 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:53:39.485 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.805750+0000", 2024-09-13T14:53:39.485 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9530000000000002e-05 2024-09-13T14:53:39.486 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.486 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.486 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:53:39.486 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.805759+0000", 2024-09-13T14:53:39.486 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5029999999999997e-06 2024-09-13T14:53:39.486 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.486 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.486 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:53:39.486 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.805763+0000", 2024-09-13T14:53:39.486 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.267e-06 2024-09-13T14:53:39.486 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.487 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.487 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:53:39.487 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.805801+0000", 2024-09-13T14:53:39.487 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8019000000000003e-05 2024-09-13T14:53:39.487 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.487 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.487 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:53:39.487 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.805824+0000", 2024-09-13T14:53:39.487 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2903000000000001e-05 2024-09-13T14:53:39.487 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.487 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.487 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:53:39.487 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.808576+0000", 2024-09-13T14:53:39.488 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0027518109999999998 2024-09-13T14:53:39.488 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.488 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.488 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:53:39.488 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.965624+0000", 2024-09-13T14:53:39.488 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.15704854100000001 2024-09-13T14:53:39.488 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:39.488 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:53:39.488 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:39.488 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.488 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.488 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.46157.0:45932 8.0 e314/304 8:07864aaa:::benchmark_data_smithi102_313275_object45931:head v 314'2880, mlcod=314'2877)", 2024-09-13T14:53:39.488 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:46:32.806942+0000", 2024-09-13T14:53:39.489 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 426.65491576099998, 2024-09-13T14:53:39.489 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.15872799200000001, 2024-09-13T14:53:39.489 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:53:39.489 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:39.489 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:53:39.489 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.489 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:53:39.489 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.806942+0000", 2024-09-13T14:53:39.489 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:39.490 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.490 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.490 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:53:39.490 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.806942+0000", 2024-09-13T14:53:39.490 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:39.490 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.490 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.490 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:53:39.490 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.806941+0000", 2024-09-13T14:53:39.490 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:53:39.490 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.490 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.490 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:53:39.491 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.807015+0000", 2024-09-13T14:53:39.491 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3687000000000001e-05 2024-09-13T14:53:39.491 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.491 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.491 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:53:39.491 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.807026+0000", 2024-09-13T14:53:39.491 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0522e-05 2024-09-13T14:53:39.491 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.491 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.491 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:53:39.491 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.807032+0000", 2024-09-13T14:53:39.491 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6080000000000001e-06 2024-09-13T14:53:39.491 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.491 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.492 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:53:39.492 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.807142+0000", 2024-09-13T14:53:39.492 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000109551 2024-09-13T14:53:39.492 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.492 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.492 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:53:39.492 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.807161+0000", 2024-09-13T14:53:39.492 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9686000000000001e-05 2024-09-13T14:53:39.492 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.492 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.492 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:53:39.492 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.965640+0000", 2024-09-13T14:53:39.492 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.15847905700000001 2024-09-13T14:53:39.493 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.493 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.493 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:53:39.493 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.965670+0000", 2024-09-13T14:53:39.493 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9620000000000001e-05 2024-09-13T14:53:39.493 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:39.493 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:53:39.493 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:39.493 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.493 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.493 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.46157.0:45934 8.4 e314/306 8:2e076b64:::benchmark_data_smithi102_313275_object45933:head v 314'2828, mlcod=314'2825)", 2024-09-13T14:53:39.493 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:46:32.807706+0000", 2024-09-13T14:53:39.493 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 426.65415180500003, 2024-09-13T14:53:39.493 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.159780378, 2024-09-13T14:53:39.494 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:53:39.494 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:39.494 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:53:39.494 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.494 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:53:39.494 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.807706+0000", 2024-09-13T14:53:39.494 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:39.494 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.494 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.494 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:53:39.494 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.807706+0000", 2024-09-13T14:53:39.494 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:39.495 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.495 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.495 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:53:39.495 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.807705+0000", 2024-09-13T14:53:39.495 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:53:39.495 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.495 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.495 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:53:39.495 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.807729+0000", 2024-09-13T14:53:39.495 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3983000000000001e-05 2024-09-13T14:53:39.495 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.495 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.495 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:53:39.495 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.807740+0000", 2024-09-13T14:53:39.496 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0611000000000001e-05 2024-09-13T14:53:39.496 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.496 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.496 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:53:39.496 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.807746+0000", 2024-09-13T14:53:39.496 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0900000000000001e-06 2024-09-13T14:53:39.496 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.496 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.496 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:53:39.496 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.807805+0000", 2024-09-13T14:53:39.496 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.9122000000000001e-05 2024-09-13T14:53:39.496 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.496 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.497 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:53:39.497 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.807825+0000", 2024-09-13T14:53:39.497 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0035999999999999e-05 2024-09-13T14:53:39.497 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.497 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.497 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:53:39.497 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.967381+0000", 2024-09-13T14:53:39.497 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.159555902 2024-09-13T14:53:39.497 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.497 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.497 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:53:39.497 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.967486+0000", 2024-09-13T14:53:39.497 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000105312 2024-09-13T14:53:39.497 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:39.498 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:53:39.498 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:39.498 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.498 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.498 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.46157.0:45938 8.2 8:499f5c03:::benchmark_data_smithi102_313275_object45937:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-09-13T14:53:39.498 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:46:32.808980+0000", 2024-09-13T14:53:39.498 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 426.65287750499999, 2024-09-13T14:53:39.498 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.163339441, 2024-09-13T14:53:39.498 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:53:39.498 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:39.498 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:53:39.498 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.46157", 2024-09-13T14:53:39.498 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:53:39.499 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 45938 2024-09-13T14:53:39.499 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.499 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:53:39.499 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.499 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:53:39.499 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.808980+0000", 2024-09-13T14:53:39.499 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:39.499 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.499 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.499 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:53:39.499 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.808980+0000", 2024-09-13T14:53:39.499 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:39.500 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.500 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.500 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:53:39.500 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.808979+0000", 2024-09-13T14:53:39.500 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:53:39.500 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.500 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.500 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:53:39.500 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.809033+0000", 2024-09-13T14:53:39.500 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4489e-05 2024-09-13T14:53:39.500 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.500 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.500 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:53:39.500 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.965592+0000", 2024-09-13T14:53:39.501 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.156558744 2024-09-13T14:53:39.501 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.501 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.501 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:53:39.501 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.965605+0000", 2024-09-13T14:53:39.501 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2723e-05 2024-09-13T14:53:39.501 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.501 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.501 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:53:39.501 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.965674+0000", 2024-09-13T14:53:39.501 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9072999999999994e-05 2024-09-13T14:53:39.501 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.501 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.502 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:53:39.502 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.965811+0000", 2024-09-13T14:53:39.502 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013785800000000001 2024-09-13T14:53:39.502 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.502 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.502 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:53:39.502 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.965909+0000", 2024-09-13T14:53:39.502 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.7312999999999995e-05 2024-09-13T14:53:39.502 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.502 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.502 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:39.502 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.969910+0000", 2024-09-13T14:53:39.502 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0040011420000000001 2024-09-13T14:53:39.503 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.503 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.503 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:53:39.503 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.969932+0000", 2024-09-13T14:53:39.503 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2518999999999999e-05 2024-09-13T14:53:39.503 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.503 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.503 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:39.503 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.972243+0000", 2024-09-13T14:53:39.503 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0023110589999999999 2024-09-13T14:53:39.503 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.503 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.503 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:53:39.503 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.972288+0000", 2024-09-13T14:53:39.504 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4171000000000001e-05 2024-09-13T14:53:39.504 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.504 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.504 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:53:39.504 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.972320+0000", 2024-09-13T14:53:39.504 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2095999999999999e-05 2024-09-13T14:53:39.504 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:39.504 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:53:39.504 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:39.504 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.504 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.504 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.46157.0:45937 8.4 e314/306 8:2900bfef:::benchmark_data_smithi102_313275_object45936:head v 314'2829, mlcod=314'2826)", 2024-09-13T14:53:39.505 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:46:32.809611+0000", 2024-09-13T14:53:39.505 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 426.65224728300001, 2024-09-13T14:53:39.505 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.16101005099999999, 2024-09-13T14:53:39.505 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:53:39.505 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:39.505 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:53:39.505 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.505 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:53:39.505 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.809611+0000", 2024-09-13T14:53:39.505 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:39.505 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.505 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.505 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:53:39.505 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.809611+0000", 2024-09-13T14:53:39.506 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:39.506 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.506 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.506 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:53:39.506 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.809610+0000", 2024-09-13T14:53:39.506 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:53:39.506 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.506 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.506 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:53:39.506 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.809630+0000", 2024-09-13T14:53:39.506 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0068000000000002e-05 2024-09-13T14:53:39.506 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.506 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.507 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:53:39.507 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.965549+0000", 2024-09-13T14:53:39.507 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.155919481 2024-09-13T14:53:39.507 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.507 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.507 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:53:39.507 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.965588+0000", 2024-09-13T14:53:39.507 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9085999999999998e-05 2024-09-13T14:53:39.507 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.508 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.508 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:53:39.508 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.965647+0000", 2024-09-13T14:53:39.508 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8217999999999997e-05 2024-09-13T14:53:39.508 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.508 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.508 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:53:39.508 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.965674+0000", 2024-09-13T14:53:39.508 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7835e-05 2024-09-13T14:53:39.508 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.508 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.508 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:53:39.508 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.970579+0000", 2024-09-13T14:53:39.509 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0049042979999999996 2024-09-13T14:53:39.509 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.509 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.509 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:53:39.509 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:32.970621+0000", 2024-09-13T14:53:39.509 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1922999999999998e-05 2024-09-13T14:53:39.509 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:39.509 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:53:39.509 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:39.509 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.509 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.509 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.46157.0:55619 8.5 8:a95080a8:::benchmark_data_smithi102_313275_object55618:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-13T14:53:39.509 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:46:38.946153+0000", 2024-09-13T14:53:39.509 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 420.51570473999999, 2024-09-13T14:53:39.510 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.277270038, 2024-09-13T14:53:39.510 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:53:39.510 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:39.510 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:53:39.510 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.46157", 2024-09-13T14:53:39.510 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:53:39.510 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 55619 2024-09-13T14:53:39.510 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.510 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:53:39.510 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.510 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:53:39.510 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.946153+0000", 2024-09-13T14:53:39.510 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:39.511 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.511 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.511 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:53:39.511 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.946153+0000", 2024-09-13T14:53:39.511 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:39.511 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.511 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.511 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:53:39.511 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.946152+0000", 2024-09-13T14:53:39.511 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:53:39.511 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.511 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.511 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:53:39.511 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.946186+0000", 2024-09-13T14:53:39.512 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4739000000000001e-05 2024-09-13T14:53:39.512 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.512 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.512 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:53:39.512 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.946203+0000", 2024-09-13T14:53:39.512 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6977999999999999e-05 2024-09-13T14:53:39.512 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.512 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.512 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:53:39.512 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.946211+0000", 2024-09-13T14:53:39.512 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4410000000000004e-06 2024-09-13T14:53:39.512 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.512 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.512 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:53:39.513 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.946263+0000", 2024-09-13T14:53:39.513 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2694000000000001e-05 2024-09-13T14:53:39.513 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.513 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.513 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:53:39.513 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.946417+0000", 2024-09-13T14:53:39.513 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000153776 2024-09-13T14:53:39.513 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.513 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.513 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:53:39.513 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.946577+0000", 2024-09-13T14:53:39.513 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000159853 2024-09-13T14:53:39.514 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.514 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.514 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:53:39.514 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.948704+0000", 2024-09-13T14:53:39.514 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0021266850000000001 2024-09-13T14:53:39.514 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.514 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.514 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:39.514 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.952101+0000", 2024-09-13T14:53:39.514 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0033971349999999999 2024-09-13T14:53:39.514 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.514 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.514 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:39.514 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.223350+0000", 2024-09-13T14:53:39.515 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.27124903099999997 2024-09-13T14:53:39.515 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.515 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.515 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:53:39.515 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.223391+0000", 2024-09-13T14:53:39.515 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1205000000000003e-05 2024-09-13T14:53:39.515 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.515 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.515 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:53:39.515 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.223423+0000", 2024-09-13T14:53:39.515 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.1973e-05 2024-09-13T14:53:39.515 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:39.515 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:53:39.515 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:39.516 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.516 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.516 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.46157.0:55632 8.5 8:aee4ccb2:::benchmark_data_smithi102_313275_object55631:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-13T14:53:39.516 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:46:38.952877+0000", 2024-09-13T14:53:39.516 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 420.50898058899998, 2024-09-13T14:53:39.516 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.27064351800000003, 2024-09-13T14:53:39.516 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:53:39.516 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:39.516 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:53:39.516 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.46157", 2024-09-13T14:53:39.516 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:53:39.516 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 55632 2024-09-13T14:53:39.516 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.517 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:53:39.517 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.517 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:53:39.517 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.952877+0000", 2024-09-13T14:53:39.517 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:39.517 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.517 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.517 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:53:39.517 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.952877+0000", 2024-09-13T14:53:39.517 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:39.517 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.517 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.517 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:53:39.518 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.952876+0000", 2024-09-13T14:53:39.518 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:53:39.518 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.518 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.518 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:53:39.518 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.952942+0000", 2024-09-13T14:53:39.518 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6246000000000002e-05 2024-09-13T14:53:39.518 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.518 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.518 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:53:39.518 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.952954+0000", 2024-09-13T14:53:39.518 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2326e-05 2024-09-13T14:53:39.518 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.518 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.519 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:53:39.519 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.952960+0000", 2024-09-13T14:53:39.519 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.524e-06 2024-09-13T14:53:39.519 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.519 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.519 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:53:39.519 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.952995+0000", 2024-09-13T14:53:39.519 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5221e-05 2024-09-13T14:53:39.519 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.519 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.519 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:53:39.519 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.953112+0000", 2024-09-13T14:53:39.519 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011763499999999999 2024-09-13T14:53:39.519 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.520 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.520 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:53:39.520 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.953267+0000", 2024-09-13T14:53:39.520 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015489099999999999 2024-09-13T14:53:39.520 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.520 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.520 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:53:39.520 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.955476+0000", 2024-09-13T14:53:39.520 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0022090769999999998 2024-09-13T14:53:39.520 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.520 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.520 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:39.520 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.220059+0000", 2024-09-13T14:53:39.521 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.26458301299999998 2024-09-13T14:53:39.521 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.521 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.521 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:39.521 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.223455+0000", 2024-09-13T14:53:39.521 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0033958970000000002 2024-09-13T14:53:39.521 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.521 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.521 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:53:39.521 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.223493+0000", 2024-09-13T14:53:39.521 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7351999999999997e-05 2024-09-13T14:53:39.521 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.521 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.522 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:53:39.522 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.223521+0000", 2024-09-13T14:53:39.522 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8059999999999999e-05 2024-09-13T14:53:39.522 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:39.522 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:53:39.522 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:39.522 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.522 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.522 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.46157.0:55645 8.5 8:ad2ba30b:::benchmark_data_smithi102_313275_object55644:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-13T14:53:39.522 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:46:38.961528+0000", 2024-09-13T14:53:39.522 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 420.50032986500003, 2024-09-13T14:53:39.522 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.26211107900000002, 2024-09-13T14:53:39.522 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:53:39.523 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:39.523 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:53:39.523 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.46157", 2024-09-13T14:53:39.523 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:53:39.523 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 55645 2024-09-13T14:53:39.523 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.523 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:53:39.523 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.523 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:53:39.523 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.961528+0000", 2024-09-13T14:53:39.523 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:39.523 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.523 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.523 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:53:39.524 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.961528+0000", 2024-09-13T14:53:39.524 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:39.524 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.524 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.524 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:53:39.524 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.961527+0000", 2024-09-13T14:53:39.524 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:53:39.524 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.524 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.524 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:53:39.524 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.961580+0000", 2024-09-13T14:53:39.524 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3350000000000003e-05 2024-09-13T14:53:39.524 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.524 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.525 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:53:39.525 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.961591+0000", 2024-09-13T14:53:39.525 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0502e-05 2024-09-13T14:53:39.525 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.525 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.525 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:53:39.525 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.961596+0000", 2024-09-13T14:53:39.525 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9760000000000003e-06 2024-09-13T14:53:39.525 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.525 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.525 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:53:39.525 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.961628+0000", 2024-09-13T14:53:39.525 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2699e-05 2024-09-13T14:53:39.525 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.526 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.526 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:53:39.526 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.961730+0000", 2024-09-13T14:53:39.526 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010173700000000001 2024-09-13T14:53:39.526 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.526 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.526 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:53:39.526 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.961813+0000", 2024-09-13T14:53:39.526 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3072999999999996e-05 2024-09-13T14:53:39.526 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.526 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.526 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:53:39.526 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.963732+0000", 2024-09-13T14:53:39.527 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:39.528 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001919254 2024-09-13T14:53:39.528 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.528 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.528 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:39.528 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.223290+0000", 2024-09-13T14:53:39.528 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.25955769699999998 2024-09-13T14:53:39.528 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.529 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.529 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:39.529 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.223575+0000", 2024-09-13T14:53:39.529 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00028493300000000002 2024-09-13T14:53:39.529 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.529 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.529 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:53:39.529 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.223612+0000", 2024-09-13T14:53:39.529 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6838999999999997e-05 2024-09-13T14:53:39.529 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.529 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.529 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:53:39.529 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.223639+0000", 2024-09-13T14:53:39.530 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7206999999999999e-05 2024-09-13T14:53:39.530 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:39.530 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:53:39.530 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:39.530 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.530 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.530 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.46157.0:55650 8.5 8:a9a2aa34:::benchmark_data_smithi102_313275_object55649:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-13T14:53:39.530 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:46:38.964767+0000", 2024-09-13T14:53:39.530 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 420.49709050899997, 2024-09-13T14:53:39.530 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.25903844199999998, 2024-09-13T14:53:39.530 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:53:39.530 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:39.530 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:53:39.530 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.46157", 2024-09-13T14:53:39.531 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:53:39.531 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 55650 2024-09-13T14:53:39.531 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.531 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:53:39.531 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.531 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:53:39.531 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.964767+0000", 2024-09-13T14:53:39.531 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:39.531 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.531 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.531 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:53:39.531 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.964767+0000", 2024-09-13T14:53:39.531 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:39.532 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.532 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.532 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:53:39.532 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.964766+0000", 2024-09-13T14:53:39.532 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:53:39.532 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.532 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.532 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:53:39.532 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.964853+0000", 2024-09-13T14:53:39.532 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.6948999999999997e-05 2024-09-13T14:53:39.532 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.532 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.532 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:53:39.533 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.964863+0000", 2024-09-13T14:53:39.533 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0084e-05 2024-09-13T14:53:39.533 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.533 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.533 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:53:39.533 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.964868+0000", 2024-09-13T14:53:39.533 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8550000000000001e-06 2024-09-13T14:53:39.533 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.533 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.533 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:53:39.533 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.964899+0000", 2024-09-13T14:53:39.533 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.1343000000000001e-05 2024-09-13T14:53:39.533 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.534 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.534 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:53:39.534 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.964997+0000", 2024-09-13T14:53:39.534 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.7309e-05 2024-09-13T14:53:39.534 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.534 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.534 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:53:39.534 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.965090+0000", 2024-09-13T14:53:39.534 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.3550999999999999e-05 2024-09-13T14:53:39.534 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.534 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.534 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:53:39.534 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:38.966569+0000", 2024-09-13T14:53:39.534 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0014783999999999999 2024-09-13T14:53:39.535 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.535 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.535 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:39.535 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.223550+0000", 2024-09-13T14:53:39.535 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.256981287 2024-09-13T14:53:39.535 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.535 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.535 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:39.535 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.223729+0000", 2024-09-13T14:53:39.535 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00017951800000000001 2024-09-13T14:53:39.535 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.535 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.535 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:53:39.536 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.223775+0000", 2024-09-13T14:53:39.536 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5061000000000001e-05 2024-09-13T14:53:39.536 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.536 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.536 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:53:39.536 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:46:39.223806+0000", 2024-09-13T14:53:39.536 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.1207999999999997e-05 2024-09-13T14:53:39.536 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:39.536 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:53:39.536 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:39.536 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.536 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.536 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.46157.0:88008 8.2 8:401fd16f:::benchmark_data_smithi102_313275_object88007:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-09-13T14:53:39.536 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:47:03.994740+0000", 2024-09-13T14:53:39.537 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 395.46711822999998, 2024-09-13T14:53:39.537 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.161786597, 2024-09-13T14:53:39.537 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:53:39.537 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:39.537 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:53:39.537 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.46157", 2024-09-13T14:53:39.537 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:53:39.537 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 88008 2024-09-13T14:53:39.537 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.537 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:53:39.537 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.537 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:53:39.537 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:03.994740+0000", 2024-09-13T14:53:39.538 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:39.538 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.538 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.538 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:53:39.538 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:03.994740+0000", 2024-09-13T14:53:39.538 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:39.538 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.538 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.538 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:53:39.538 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:03.994738+0000", 2024-09-13T14:53:39.538 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:53:39.538 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.538 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.538 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:53:39.539 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:03.994771+0000", 2024-09-13T14:53:39.539 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.3178e-05 2024-09-13T14:53:39.539 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.539 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.539 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:53:39.539 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:03.994784+0000", 2024-09-13T14:53:39.539 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2935999999999999e-05 2024-09-13T14:53:39.539 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.539 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.539 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:53:39.539 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:03.994791+0000", 2024-09-13T14:53:39.539 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.652e-06 2024-09-13T14:53:39.539 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.539 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.540 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:53:39.540 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:03.994826+0000", 2024-09-13T14:53:39.540 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5413999999999997e-05 2024-09-13T14:53:39.540 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.540 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.540 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:53:39.540 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:03.994971+0000", 2024-09-13T14:53:39.540 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000144697 2024-09-13T14:53:39.540 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.540 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.540 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:53:39.540 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:03.995108+0000", 2024-09-13T14:53:39.540 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013675199999999999 2024-09-13T14:53:39.541 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.541 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.541 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:53:39.541 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:03.998907+0000", 2024-09-13T14:53:39.541 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0037994280000000001 2024-09-13T14:53:39.541 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.541 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.541 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:39.541 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.000247+0000", 2024-09-13T14:53:39.541 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001339841 2024-09-13T14:53:39.541 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.541 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.541 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:39.541 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.156448+0000", 2024-09-13T14:53:39.542 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.156200697 2024-09-13T14:53:39.542 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.542 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.542 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:53:39.542 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.156483+0000", 2024-09-13T14:53:39.542 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4989e-05 2024-09-13T14:53:39.542 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.542 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.542 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:53:39.542 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.156526+0000", 2024-09-13T14:53:39.542 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3674999999999998e-05 2024-09-13T14:53:39.542 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:39.542 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:53:39.543 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:39.543 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.543 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.543 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.46157.0:88022 8.2 8:4f0b62b0:::benchmark_data_smithi102_313275_object88021:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-09-13T14:53:39.543 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:47:04.007617+0000", 2024-09-13T14:53:39.543 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 395.454241185, 2024-09-13T14:53:39.543 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.15374996299999999, 2024-09-13T14:53:39.543 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:53:39.543 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:39.543 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:53:39.543 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.46157", 2024-09-13T14:53:39.543 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:53:39.543 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 88022 2024-09-13T14:53:39.543 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.544 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:53:39.544 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.544 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:53:39.544 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.007617+0000", 2024-09-13T14:53:39.544 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:39.544 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.544 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.544 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:53:39.544 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.007617+0000", 2024-09-13T14:53:39.544 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:39.544 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.544 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.544 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:53:39.545 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.007607+0000", 2024-09-13T14:53:39.545 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999909 2024-09-13T14:53:39.545 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.545 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.545 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:53:39.545 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.007640+0000", 2024-09-13T14:53:39.545 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2767e-05 2024-09-13T14:53:39.545 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.545 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.545 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:53:39.545 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.007650+0000", 2024-09-13T14:53:39.545 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.5599999999999999e-06 2024-09-13T14:53:39.545 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.546 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.546 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:53:39.546 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.007655+0000", 2024-09-13T14:53:39.546 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1800000000000004e-06 2024-09-13T14:53:39.546 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.546 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.546 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:53:39.546 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.007702+0000", 2024-09-13T14:53:39.546 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7472999999999998e-05 2024-09-13T14:53:39.546 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.546 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.546 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:53:39.546 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.007832+0000", 2024-09-13T14:53:39.546 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012959500000000001 2024-09-13T14:53:39.547 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.547 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.547 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:53:39.547 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.007966+0000", 2024-09-13T14:53:39.547 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013418699999999999 2024-09-13T14:53:39.547 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.547 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.547 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:53:39.547 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.011111+0000", 2024-09-13T14:53:39.547 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0031446069999999998 2024-09-13T14:53:39.547 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.547 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.547 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:39.547 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.014155+0000", 2024-09-13T14:53:39.548 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0030444090000000001 2024-09-13T14:53:39.548 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.548 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.548 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:39.548 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.161272+0000", 2024-09-13T14:53:39.548 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.14711637899999999 2024-09-13T14:53:39.548 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.548 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.548 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:53:39.548 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.161325+0000", 2024-09-13T14:53:39.548 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3829000000000002e-05 2024-09-13T14:53:39.548 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.548 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.549 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:53:39.549 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.161367+0000", 2024-09-13T14:53:39.549 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1114e-05 2024-09-13T14:53:39.549 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:39.549 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:53:39.549 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:39.549 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.549 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.549 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.46157.0:88026 8.2 8:41e4b3c0:::benchmark_data_smithi102_313275_object88025:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-09-13T14:53:39.549 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:47:04.011206+0000", 2024-09-13T14:53:39.549 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 395.45065150200003, 2024-09-13T14:53:39.549 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.15024669500000001, 2024-09-13T14:53:39.549 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:53:39.550 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:39.550 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:53:39.550 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.46157", 2024-09-13T14:53:39.550 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:53:39.550 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 88026 2024-09-13T14:53:39.550 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.550 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:53:39.550 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.550 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:53:39.550 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.011206+0000", 2024-09-13T14:53:39.550 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:39.550 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.550 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.550 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:53:39.551 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.011206+0000", 2024-09-13T14:53:39.551 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:39.551 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.551 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.551 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:53:39.551 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.011205+0000", 2024-09-13T14:53:39.551 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:53:39.551 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.551 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.551 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:53:39.551 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.011264+0000", 2024-09-13T14:53:39.551 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8860999999999998e-05 2024-09-13T14:53:39.551 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.552 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.552 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:53:39.552 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.011273+0000", 2024-09-13T14:53:39.552 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.9560000000000003e-06 2024-09-13T14:53:39.552 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.552 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.552 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:53:39.552 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.011280+0000", 2024-09-13T14:53:39.552 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.0129999999999996e-06 2024-09-13T14:53:39.552 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.552 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.552 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:53:39.552 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.011321+0000", 2024-09-13T14:53:39.552 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0802e-05 2024-09-13T14:53:39.553 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.553 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.553 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:53:39.553 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.011465+0000", 2024-09-13T14:53:39.553 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014436200000000001 2024-09-13T14:53:39.553 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.553 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.553 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:53:39.553 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.011614+0000", 2024-09-13T14:53:39.553 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014850099999999999 2024-09-13T14:53:39.553 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.553 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.553 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:53:39.553 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.014218+0000", 2024-09-13T14:53:39.554 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0026037759999999999 2024-09-13T14:53:39.554 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.554 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.554 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:39.554 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.014776+0000", 2024-09-13T14:53:39.554 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00055851700000000002 2024-09-13T14:53:39.554 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.554 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.554 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:39.554 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.161403+0000", 2024-09-13T14:53:39.554 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.146626532 2024-09-13T14:53:39.554 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.554 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.554 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:53:39.555 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.161433+0000", 2024-09-13T14:53:39.555 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0011000000000001e-05 2024-09-13T14:53:39.555 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.555 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.555 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:53:39.555 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:04.161453+0000", 2024-09-13T14:53:39.555 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0287000000000001e-05 2024-09-13T14:53:39.555 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:39.555 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:53:39.555 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:39.555 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.555 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.555 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.46157.0:129271 8.2 8:4bee72dd:::benchmark_data_smithi102_313275_object129270:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-13T14:53:39.556 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:47:42.324161+0000", 2024-09-13T14:53:39.556 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 357.13769670800002, 2024-09-13T14:53:39.556 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.12509938800000001, 2024-09-13T14:53:39.556 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:53:39.556 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:39.556 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:53:39.556 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.46157", 2024-09-13T14:53:39.556 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:53:39.556 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 129271 2024-09-13T14:53:39.556 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.556 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:53:39.556 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.556 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:53:39.557 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.324161+0000", 2024-09-13T14:53:39.557 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:39.557 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.557 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.557 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:53:39.557 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.324161+0000", 2024-09-13T14:53:39.557 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:39.557 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.557 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.557 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:53:39.557 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.324160+0000", 2024-09-13T14:53:39.558 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:53:39.558 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.558 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.558 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:53:39.558 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.324218+0000", 2024-09-13T14:53:39.558 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7559000000000002e-05 2024-09-13T14:53:39.558 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.558 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.558 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:53:39.558 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.324234+0000", 2024-09-13T14:53:39.558 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.641e-05 2024-09-13T14:53:39.558 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.558 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.559 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:53:39.559 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.324241+0000", 2024-09-13T14:53:39.559 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4619999999999999e-06 2024-09-13T14:53:39.559 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.559 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.559 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:53:39.559 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.327028+0000", 2024-09-13T14:53:39.559 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0027863060000000001 2024-09-13T14:53:39.559 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.559 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.559 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:53:39.559 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.327185+0000", 2024-09-13T14:53:39.559 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015714799999999999 2024-09-13T14:53:39.560 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.560 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.560 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:53:39.560 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.327341+0000", 2024-09-13T14:53:39.560 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015649899999999999 2024-09-13T14:53:39.560 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.560 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.560 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:39.560 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.330402+0000", 2024-09-13T14:53:39.560 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0030606100000000001 2024-09-13T14:53:39.560 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.560 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.560 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:53:39.561 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.330420+0000", 2024-09-13T14:53:39.561 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7666e-05 2024-09-13T14:53:39.561 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.561 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.561 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:39.561 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.449141+0000", 2024-09-13T14:53:39.561 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.118721302 2024-09-13T14:53:39.561 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.561 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.561 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:53:39.561 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.449199+0000", 2024-09-13T14:53:39.561 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8304000000000003e-05 2024-09-13T14:53:39.561 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.562 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.562 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:53:39.562 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:47:42.449260+0000", 2024-09-13T14:53:39.562 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1290000000000004e-05 2024-09-13T14:53:39.562 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:39.562 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:53:39.562 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:39.562 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.562 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.562 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:117374 9.10 9:0b249139:::benchmark_data_smithi102_348563_object117373:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e405)", 2024-09-13T14:53:39.562 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:52:32.510939+0000", 2024-09-13T14:53:39.562 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 66.950918983999998, 2024-09-13T14:53:39.562 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.108565514, 2024-09-13T14:53:39.562 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:53:39.563 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:39.563 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:53:39.563 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:53:39.563 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:53:39.563 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 117374 2024-09-13T14:53:39.563 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.563 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:53:39.563 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.564 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:53:39.564 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:32.510939+0000", 2024-09-13T14:53:39.564 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:39.564 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.564 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.564 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:53:39.564 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:32.510939+0000", 2024-09-13T14:53:39.564 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:39.564 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.564 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.564 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:53:39.564 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:32.510937+0000", 2024-09-13T14:53:39.564 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:53:39.565 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.565 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.565 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:53:39.565 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:32.510972+0000", 2024-09-13T14:53:39.565 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5333999999999997e-05 2024-09-13T14:53:39.565 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.565 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.565 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:53:39.565 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:32.510995+0000", 2024-09-13T14:53:39.565 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2875e-05 2024-09-13T14:53:39.565 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.565 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.565 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:53:39.566 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:32.511004+0000", 2024-09-13T14:53:39.566 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3999999999999992e-06 2024-09-13T14:53:39.566 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.566 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.566 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:53:39.566 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:32.511085+0000", 2024-09-13T14:53:39.566 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1502000000000007e-05 2024-09-13T14:53:39.566 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.566 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.566 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:53:39.566 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:32.511226+0000", 2024-09-13T14:53:39.566 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000140576 2024-09-13T14:53:39.566 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.566 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.567 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:53:39.567 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:32.511354+0000", 2024-09-13T14:53:39.567 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000128472 2024-09-13T14:53:39.567 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.567 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.567 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:53:39.567 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:32.514542+0000", 2024-09-13T14:53:39.567 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0031878119999999999 2024-09-13T14:53:39.567 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.567 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.567 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:39.567 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:32.514944+0000", 2024-09-13T14:53:39.567 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00040232100000000003 2024-09-13T14:53:39.567 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.568 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.568 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:39.568 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:32.619383+0000", 2024-09-13T14:53:39.568 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.104438662 2024-09-13T14:53:39.568 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.568 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.568 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:53:39.568 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:32.619462+0000", 2024-09-13T14:53:39.568 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8714999999999994e-05 2024-09-13T14:53:39.568 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.568 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.568 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:53:39.568 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:32.619504+0000", 2024-09-13T14:53:39.569 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2667999999999999e-05 2024-09-13T14:53:39.569 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:39.569 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:53:39.569 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:39.569 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.569 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.569 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:117378 9.10 9:0bf10505:::benchmark_data_smithi102_348563_object117377:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e405)", 2024-09-13T14:53:39.569 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:52:32.514803+0000", 2024-09-13T14:53:39.569 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 66.947054492999996, 2024-09-13T14:53:39.569 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.104810984, 2024-09-13T14:53:39.569 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:53:39.569 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:39.569 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:53:39.569 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:53:39.570 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:53:39.570 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 117378 2024-09-13T14:53:39.570 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.570 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:53:39.570 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.570 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:53:39.570 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:32.514803+0000", 2024-09-13T14:53:39.570 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:39.570 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.570 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.570 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:53:39.570 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:32.514803+0000", 2024-09-13T14:53:39.570 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:39.570 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.571 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.571 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:53:39.571 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:32.514802+0000", 2024-09-13T14:53:39.571 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:53:39.571 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.571 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.571 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:53:39.571 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:32.514853+0000", 2024-09-13T14:53:39.571 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0927000000000002e-05 2024-09-13T14:53:39.571 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.571 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.571 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:53:39.571 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:32.514864+0000", 2024-09-13T14:53:39.572 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1806e-05 2024-09-13T14:53:39.572 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.572 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.572 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:53:39.572 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:32.514871+0000", 2024-09-13T14:53:39.572 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.883e-06 2024-09-13T14:53:39.572 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.572 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.572 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:53:39.572 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:32.514965+0000", 2024-09-13T14:53:39.572 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4010000000000003e-05 2024-09-13T14:53:39.572 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.572 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.573 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:53:39.573 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:32.515104+0000", 2024-09-13T14:53:39.573 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013868800000000001 2024-09-13T14:53:39.573 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.573 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.573 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:53:39.573 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:32.515235+0000", 2024-09-13T14:53:39.573 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013111700000000001 2024-09-13T14:53:39.573 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.573 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.573 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:53:39.573 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:32.520430+0000", 2024-09-13T14:53:39.573 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0051949780000000003 2024-09-13T14:53:39.573 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.574 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.574 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:39.574 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:32.526794+0000", 2024-09-13T14:53:39.574 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0063642229999999996 2024-09-13T14:53:39.574 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.574 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.574 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:39.574 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:32.619542+0000", 2024-09-13T14:53:39.574 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.092748112999999993 2024-09-13T14:53:39.574 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.574 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.574 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:53:39.574 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:32.619583+0000", 2024-09-13T14:53:39.575 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1022000000000001e-05 2024-09-13T14:53:39.575 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.575 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.575 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:53:39.575 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:32.619614+0000", 2024-09-13T14:53:39.575 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0891e-05 2024-09-13T14:53:39.575 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:39.575 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:53:39.576 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:39.576 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.576 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.576 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:117381 9.10 9:0b487471:::benchmark_data_smithi102_348563_object117380:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e405)", 2024-09-13T14:53:39.576 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:52:32.517250+0000", 2024-09-13T14:53:39.576 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 66.944607321000007, 2024-09-13T14:53:39.576 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.10247227, 2024-09-13T14:53:39.576 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:53:39.576 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:39.576 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:53:39.576 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:53:39.576 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:53:39.576 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 117381 2024-09-13T14:53:39.577 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.577 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:53:39.577 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.577 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:53:39.577 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:32.517250+0000", 2024-09-13T14:53:39.577 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:39.577 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.577 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.577 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:53:39.577 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:32.517250+0000", 2024-09-13T14:53:39.577 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:39.577 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.577 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.578 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:53:39.578 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:32.517249+0000", 2024-09-13T14:53:39.578 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:53:39.578 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.578 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.578 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:53:39.578 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:32.517282+0000", 2024-09-13T14:53:39.578 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2910999999999997e-05 2024-09-13T14:53:39.578 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.578 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.578 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:53:39.578 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:32.517292+0000", 2024-09-13T14:53:39.578 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0601999999999999e-05 2024-09-13T14:53:39.579 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.579 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.579 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:53:39.579 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:32.517298+0000", 2024-09-13T14:53:39.579 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.9800000000000003e-06 2024-09-13T14:53:39.579 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.579 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.579 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:53:39.579 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:32.517649+0000", 2024-09-13T14:53:39.579 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00035059499999999999 2024-09-13T14:53:39.579 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.580 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.580 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:53:39.580 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:32.517774+0000", 2024-09-13T14:53:39.580 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000125275 2024-09-13T14:53:39.580 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.580 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.580 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:53:39.580 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:32.517900+0000", 2024-09-13T14:53:39.580 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012579799999999999 2024-09-13T14:53:39.580 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.580 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.580 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:53:39.580 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:32.522897+0000", 2024-09-13T14:53:39.580 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.004997235 2024-09-13T14:53:39.581 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.581 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.581 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:39.581 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:32.526911+0000", 2024-09-13T14:53:39.581 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0040143749999999997 2024-09-13T14:53:39.581 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.581 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.581 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:39.581 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:32.619655+0000", 2024-09-13T14:53:39.581 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.092743949000000006 2024-09-13T14:53:39.581 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.581 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.581 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:53:39.582 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:32.619695+0000", 2024-09-13T14:53:39.582 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9804e-05 2024-09-13T14:53:39.582 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.582 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.582 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:53:39.582 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:32.619723+0000", 2024-09-13T14:53:39.582 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7528000000000001e-05 2024-09-13T14:53:39.582 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:39.582 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:53:39.582 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:39.582 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.582 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.582 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:122639 9.10 9:0a9d71f0:::benchmark_data_smithi102_348563_object1537:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e405)", 2024-09-13T14:53:39.582 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:52:38.486294+0000", 2024-09-13T14:53:39.583 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 60.975563444000002, 2024-09-13T14:53:39.583 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.10430621499999999, 2024-09-13T14:53:39.583 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:53:39.583 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:39.583 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:53:39.583 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:53:39.583 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:53:39.583 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 122639 2024-09-13T14:53:39.583 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.583 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:53:39.583 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.583 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:53:39.583 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:38.486294+0000", 2024-09-13T14:53:39.583 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:39.584 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.584 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.584 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:53:39.584 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:38.486294+0000", 2024-09-13T14:53:39.584 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:39.584 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.584 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.584 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:53:39.584 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:38.486293+0000", 2024-09-13T14:53:39.584 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:53:39.584 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.584 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.584 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:53:39.585 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:38.486299+0000", 2024-09-13T14:53:39.585 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.9689999999999999e-06 2024-09-13T14:53:39.585 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.585 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.585 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:53:39.585 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:38.486307+0000", 2024-09-13T14:53:39.585 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.4039999999999995e-06 2024-09-13T14:53:39.585 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.585 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.585 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:53:39.586 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:38.486314+0000", 2024-09-13T14:53:39.586 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4150000000000004e-06 2024-09-13T14:53:39.586 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.586 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.587 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:53:39.587 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:38.486348+0000", 2024-09-13T14:53:39.587 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4276000000000002e-05 2024-09-13T14:53:39.587 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.587 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.587 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:53:39.587 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:38.486456+0000", 2024-09-13T14:53:39.587 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010802900000000001 2024-09-13T14:53:39.587 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.587 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.587 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:53:39.587 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:38.486556+0000", 2024-09-13T14:53:39.587 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.9691000000000005e-05 2024-09-13T14:53:39.587 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.588 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.588 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:53:39.588 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:38.488994+0000", 2024-09-13T14:53:39.588 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0024383849999999999 2024-09-13T14:53:39.588 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.588 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.588 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:39.588 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:38.491183+0000", 2024-09-13T14:53:39.588 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0021889079999999998 2024-09-13T14:53:39.588 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.588 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.588 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:39.588 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:38.590527+0000", 2024-09-13T14:53:39.589 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.099344210000000002 2024-09-13T14:53:39.589 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.589 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.589 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:53:39.589 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:38.590561+0000", 2024-09-13T14:53:39.589 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.3615000000000003e-05 2024-09-13T14:53:39.589 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.589 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.589 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:53:39.589 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:38.590601+0000", 2024-09-13T14:53:39.589 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9848999999999997e-05 2024-09-13T14:53:39.589 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:39.589 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:53:39.590 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:39.590 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.590 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.590 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:161761 9.10 9:0f800c21:::benchmark_data_smithi102_348563_object40659:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e410)", 2024-09-13T14:53:39.590 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:53:15.534471+0000", 2024-09-13T14:53:39.590 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 23.927386803000001, 2024-09-13T14:53:39.590 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.108700159, 2024-09-13T14:53:39.590 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:53:39.590 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:39.590 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:53:39.590 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:53:39.590 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:53:39.590 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 161761 2024-09-13T14:53:39.591 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.591 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:53:39.591 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.591 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:53:39.591 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:15.534471+0000", 2024-09-13T14:53:39.591 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:39.591 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.591 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.591 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:53:39.591 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:15.534471+0000", 2024-09-13T14:53:39.591 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:39.591 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.591 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.592 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:53:39.592 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:15.534469+0000", 2024-09-13T14:53:39.592 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:53:39.592 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.592 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.592 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:53:39.592 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:15.534477+0000", 2024-09-13T14:53:39.592 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0350000000000001e-06 2024-09-13T14:53:39.592 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.592 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.592 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:53:39.592 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:15.534493+0000", 2024-09-13T14:53:39.592 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6195e-05 2024-09-13T14:53:39.592 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.593 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.593 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:53:39.593 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:15.534501+0000", 2024-09-13T14:53:39.593 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2339999999999997e-06 2024-09-13T14:53:39.593 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.593 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.593 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:53:39.593 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:15.534549+0000", 2024-09-13T14:53:39.593 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8149000000000003e-05 2024-09-13T14:53:39.593 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.593 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.593 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:53:39.593 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:15.534673+0000", 2024-09-13T14:53:39.593 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012479599999999999 2024-09-13T14:53:39.594 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.594 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.594 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:53:39.594 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:15.535111+0000", 2024-09-13T14:53:39.594 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00043772 2024-09-13T14:53:39.594 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.594 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.594 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:53:39.594 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:15.537809+0000", 2024-09-13T14:53:39.594 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002697886 2024-09-13T14:53:39.594 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.594 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.594 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:39.595 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:15.540163+0000", 2024-09-13T14:53:39.595 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0023535230000000002 2024-09-13T14:53:39.595 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.595 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.595 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:39.595 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:15.643063+0000", 2024-09-13T14:53:39.595 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.10290042200000001 2024-09-13T14:53:39.595 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.595 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.595 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:53:39.595 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:15.643137+0000", 2024-09-13T14:53:39.595 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4064000000000001e-05 2024-09-13T14:53:39.595 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:39.595 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:39.596 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:53:39.596 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:15.643171+0000", 2024-09-13T14:53:39.596 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4063999999999998e-05 2024-09-13T14:53:39.596 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:39.596 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:53:39.596 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:39.596 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:39.596 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:53:39.596 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:53:39.596 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:39 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 7054/181392 objects degraded (3.889%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:53:39.622 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:39.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:39.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:39.656 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:39 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 7054/181392 objects degraded (3.889%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:53:39.657 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:39 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 7054/181392 objects degraded (3.889%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:53:39.724 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:39.754 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:39.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:39.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:39.856 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:39.890 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:39.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:39.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:39.992 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:40.031 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:40.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:40.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:40.133 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:40.171 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:40.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:40.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:40.273 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:40.309 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:40.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:40.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:40.411 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:40.449 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:40.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:40.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:40.502 INFO:teuthology.orchestra.run.smithi102.stderr:reweighted osd.5 to 1 (10000) 2024-09-13T14:53:40.550 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:40.584 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:40.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:40.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:40.649 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T14:53:40.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:40 smithi102 ceph-mon[110183]: 9.7 scrub starts 2024-09-13T14:53:40.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:40 smithi102 ceph-mon[110183]: pgmap v1414: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 3.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 970 op/s; 7054/178629 objects degraded (3.949%); 2165/178629 objects misplaced (1.212%); 7.3 MiB/s, 117 objects/s recovering 2024-09-13T14:53:40.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:40 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2423182904' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:53:40.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:40 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:53:40.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:40 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:53:40.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:40 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:53:40.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:40 smithi102 ceph-mon[115269]: 9.7 scrub starts 2024-09-13T14:53:40.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:40 smithi102 ceph-mon[115269]: pgmap v1414: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 3.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 970 op/s; 7054/178629 objects degraded (3.949%); 2165/178629 objects misplaced (1.212%); 7.3 MiB/s, 117 objects/s recovering 2024-09-13T14:53:40.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:40 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2423182904' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:53:40.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:40 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:53:40.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:40 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:53:40.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:40 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:53:40.686 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:40.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:40.728 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:40.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:40.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:40 smithi195 ceph-mon[104262]: 9.7 scrub starts 2024-09-13T14:53:40.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:40 smithi195 ceph-mon[104262]: pgmap v1414: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 3.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 970 op/s; 7054/178629 objects degraded (3.949%); 2165/178629 objects misplaced (1.212%); 7.3 MiB/s, 117 objects/s recovering 2024-09-13T14:53:40.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:40 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2423182904' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:53:40.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:40 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:53:40.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:40 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:53:40.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:40 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:53:40.772 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:53:40.829 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:40.880 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:40.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:40.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:40.984 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:40.991 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:53:41.032 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:41.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:41.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:41.133 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:41.177 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:41.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:41.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:41.279 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:41.311 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:41.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:41.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:41.413 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:41.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:41.443 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:41.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:41.544 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:41.565 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:41.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:41.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:41.667 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:41.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:41 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T14:53:41.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:41 smithi102 ceph-mon[110183]: osdmap e415: 8 total, 8 up, 8 in 2024-09-13T14:53:41.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:41 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T14:53:41.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:41 smithi102 ceph-mon[115269]: osdmap e415: 8 total, 8 up, 8 in 2024-09-13T14:53:41.704 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:41.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:41.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:41.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:41 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T14:53:41.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:41 smithi195 ceph-mon[104262]: osdmap e415: 8 total, 8 up, 8 in 2024-09-13T14:53:41.806 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:41.844 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:41.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:41.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:41.946 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:41.968 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:41.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:41.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:42.070 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:42.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:42.099 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:42.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:42.200 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:42.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:42.234 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:42.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:42.335 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:42.369 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:42.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:42.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:42.470 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:42.506 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:42.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:42.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:42.608 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:42.648 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:42.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:42.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:42.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:42 smithi195 ceph-mon[104262]: pgmap v1416: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 3.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 993 op/s; 6781/166308 objects degraded (4.077%); 1307/166308 objects misplaced (0.786%); 9.2 MiB/s, 147 objects/s recovering 2024-09-13T14:53:42.749 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:42.782 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:42.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:42.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:42.885 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:42.914 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:42.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:42.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:42.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:42 smithi102 ceph-mon[110183]: pgmap v1416: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 3.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 993 op/s; 6781/166308 objects degraded (4.077%); 1307/166308 objects misplaced (0.786%); 9.2 MiB/s, 147 objects/s recovering 2024-09-13T14:53:42.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:42 smithi102 ceph-mon[115269]: pgmap v1416: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 3.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 993 op/s; 6781/166308 objects degraded (4.077%); 1307/166308 objects misplaced (0.786%); 9.2 MiB/s, 147 objects/s recovering 2024-09-13T14:53:43.016 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:43.052 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:43.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:43.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:43.154 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:43.197 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:43.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:43.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:43.299 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:43.320 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:43.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:43.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:43.421 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:43.462 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:43.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:43.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:43.564 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:43.599 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:43.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:43.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:43.701 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:43.743 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:43.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:43.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:43.845 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:43.883 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:43.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:43.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:43.890 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:53:43.890 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-13T14:53:43.890 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.891 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:189880 9.12 9:4defa47f:::benchmark_data_smithi102_348563_object68778:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-09-13T14:53:43.891 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:53:43.881194+0000", 2024-09-13T14:53:43.891 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.0070842220000000003, 2024-09-13T14:53:43.891 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.007114286, 2024-09-13T14:53:43.891 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:53:43.891 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:53:43.891 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:53:43.891 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:53:43.891 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:53:43.891 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 189880 2024-09-13T14:53:43.891 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.891 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:53:43.891 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.892 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:53:43.892 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.881194+0000", 2024-09-13T14:53:43.892 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:43.892 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.892 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.892 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:53:43.892 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.881194+0000", 2024-09-13T14:53:43.892 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:43.892 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.892 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.892 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:53:43.892 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.881192+0000", 2024-09-13T14:53:43.892 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:53:43.892 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.893 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.893 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:53:43.893 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.881199+0000", 2024-09-13T14:53:43.893 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7780000000000002e-06 2024-09-13T14:53:43.893 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.893 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.893 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:53:43.893 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.881213+0000", 2024-09-13T14:53:43.893 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3518e-05 2024-09-13T14:53:43.893 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.893 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.893 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:53:43.894 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.881220+0000", 2024-09-13T14:53:43.894 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6939999999999995e-06 2024-09-13T14:53:43.894 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.894 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.894 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:53:43.894 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.881255+0000", 2024-09-13T14:53:43.894 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4854999999999997e-05 2024-09-13T14:53:43.894 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.894 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.894 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:53:43.894 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.881407+0000", 2024-09-13T14:53:43.894 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015158700000000001 2024-09-13T14:53:43.894 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.895 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.895 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:53:43.895 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.881996+0000", 2024-09-13T14:53:43.895 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00058920000000000001 2024-09-13T14:53:43.895 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:43.895 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:53:43.895 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:43.895 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.895 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.895 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:189881 9.12 9:4eba38a6:::benchmark_data_smithi102_348563_object68779:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-09-13T14:53:43.895 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:53:43.881234+0000", 2024-09-13T14:53:43.895 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.0070442129999999997, 2024-09-13T14:53:43.896 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0071605899999999997, 2024-09-13T14:53:43.896 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:53:43.896 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:53:43.896 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:53:43.896 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:53:43.896 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:53:43.896 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 189881 2024-09-13T14:53:43.896 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.896 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:53:43.896 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.896 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:53:43.896 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.881234+0000", 2024-09-13T14:53:43.896 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:43.897 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.897 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.897 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:53:43.897 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.881234+0000", 2024-09-13T14:53:43.897 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:43.897 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.897 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.897 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:53:43.897 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.881233+0000", 2024-09-13T14:53:43.897 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:53:43.897 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.897 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.897 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:53:43.898 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.881236+0000", 2024-09-13T14:53:43.898 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.976e-06 2024-09-13T14:53:43.898 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.898 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.898 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:53:43.898 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.881243+0000", 2024-09-13T14:53:43.898 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1790000000000002e-06 2024-09-13T14:53:43.898 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.898 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.898 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:53:43.898 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.881249+0000", 2024-09-13T14:53:43.898 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.879e-06 2024-09-13T14:53:43.898 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.898 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.899 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:53:43.899 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.887591+0000", 2024-09-13T14:53:43.899 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0063419080000000003 2024-09-13T14:53:43.899 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.899 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.899 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:53:43.899 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.887715+0000", 2024-09-13T14:53:43.899 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012417999999999999 2024-09-13T14:53:43.899 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.899 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.899 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:53:43.899 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.888272+0000", 2024-09-13T14:53:43.899 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00055677400000000001 2024-09-13T14:53:43.899 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:43.900 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:53:43.900 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:43.900 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.900 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.900 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36528.0:189879 9.b e415/374)", 2024-09-13T14:53:43.900 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:53:43.881729+0000", 2024-09-13T14:53:43.900 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.0065491619999999999, 2024-09-13T14:53:43.900 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0067358729999999999, 2024-09-13T14:53:43.900 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:53:43.900 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "queued for pg", 2024-09-13T14:53:43.900 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:53:43.900 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.900 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:53:43.901 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.881729+0000", 2024-09-13T14:53:43.901 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:43.901 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.901 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.901 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:53:43.901 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.881729+0000", 2024-09-13T14:53:43.901 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:43.901 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.901 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.901 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:53:43.901 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.881728+0000", 2024-09-13T14:53:43.901 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:53:43.901 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.902 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.902 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:53:43.902 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.881735+0000", 2024-09-13T14:53:43.902 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.0310000000000001e-06 2024-09-13T14:53:43.902 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.902 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.902 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:53:43.902 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.881745+0000", 2024-09-13T14:53:43.902 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.01e-05 2024-09-13T14:53:43.902 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.902 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.902 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:53:43.902 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.881752+0000", 2024-09-13T14:53:43.902 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4640000000000003e-06 2024-09-13T14:53:43.903 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:43.903 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:53:43.903 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:43.903 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.903 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.903 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36528.0:189886 9.d e415/374 9:b1edcdbc:::benchmark_data_smithi102_348563_object68784:head v 415'11931, mlcod=415'11929)", 2024-09-13T14:53:43.903 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:53:43.887470+0000", 2024-09-13T14:53:43.903 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.000807806, 2024-09-13T14:53:43.903 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0010401570000000001, 2024-09-13T14:53:43.903 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:53:43.903 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T14:53:43.903 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:53:43.903 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.903 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:53:43.904 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.887470+0000", 2024-09-13T14:53:43.904 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:43.904 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.904 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.904 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:53:43.904 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.887470+0000", 2024-09-13T14:53:43.904 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:43.904 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.904 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.904 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:53:43.904 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.887469+0000", 2024-09-13T14:53:43.904 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:53:43.904 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.905 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.905 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:53:43.905 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.887477+0000", 2024-09-13T14:53:43.905 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5560000000000002e-06 2024-09-13T14:53:43.905 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.905 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.905 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:53:43.906 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.887490+0000", 2024-09-13T14:53:43.906 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3309000000000001e-05 2024-09-13T14:53:43.906 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.906 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.906 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:53:43.906 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.887497+0000", 2024-09-13T14:53:43.906 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.0990000000000001e-06 2024-09-13T14:53:43.906 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.906 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.906 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:53:43.906 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.887651+0000", 2024-09-13T14:53:43.906 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015416099999999999 2024-09-13T14:53:43.907 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.907 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.907 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:53:43.907 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.887670+0000", 2024-09-13T14:53:43.907 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9091999999999999e-05 2024-09-13T14:53:43.907 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:43.907 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:53:43.907 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:43.907 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.907 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.907 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36528.0:189882 9.13 e415/396)", 2024-09-13T14:53:43.907 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:53:43.887511+0000", 2024-09-13T14:53:43.907 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.00076672899999999996, 2024-09-13T14:53:43.907 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0010517440000000001, 2024-09-13T14:53:43.908 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:53:43.908 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "queued for pg", 2024-09-13T14:53:43.908 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:53:43.908 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.908 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:53:43.908 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.887511+0000", 2024-09-13T14:53:43.908 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:43.908 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.908 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.908 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:53:43.908 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.887511+0000", 2024-09-13T14:53:43.908 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:43.908 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.909 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.909 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:53:43.909 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.887511+0000", 2024-09-13T14:53:43.909 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:53:43.909 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.909 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.909 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:53:43.910 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.887515+0000", 2024-09-13T14:53:43.910 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8789999999999996e-06 2024-09-13T14:53:43.910 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.910 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.910 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:53:43.910 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.887522+0000", 2024-09-13T14:53:43.910 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2690000000000002e-06 2024-09-13T14:53:43.910 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.910 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.910 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:53:43.910 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.887526+0000", 2024-09-13T14:53:43.910 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3760000000000001e-06 2024-09-13T14:53:43.910 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:43.911 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:53:43.911 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:43.911 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.911 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.911 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop_reply(client.36528.0:189880 9.12 e415/407)", 2024-09-13T14:53:43.911 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:53:43.887893+0000", 2024-09-13T14:53:43.911 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.000385287, 2024-09-13T14:53:43.911 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00071025099999999996, 2024-09-13T14:53:43.911 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:53:43.911 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "queued for pg", 2024-09-13T14:53:43.911 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:53:43.911 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.911 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:53:43.911 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.887893+0000", 2024-09-13T14:53:43.912 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:43.912 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.912 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.912 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:53:43.912 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.887893+0000", 2024-09-13T14:53:43.912 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:43.912 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.912 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.912 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:53:43.912 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.887892+0000", 2024-09-13T14:53:43.912 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:53:43.912 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.912 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.912 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:53:43.913 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.887906+0000", 2024-09-13T14:53:43.913 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3567000000000001e-05 2024-09-13T14:53:43.913 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.913 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.913 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:53:43.913 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.887926+0000", 2024-09-13T14:53:43.913 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0531e-05 2024-09-13T14:53:43.913 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.913 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.913 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:53:43.913 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.887932+0000", 2024-09-13T14:53:43.913 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.801e-06 2024-09-13T14:53:43.913 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:43.914 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:53:43.914 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:43.914 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.914 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.914 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:189888 9.6 9:638dbd4d:::benchmark_data_smithi102_348563_object68786:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-09-13T14:53:43.914 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:53:43.888308+0000", 2024-09-13T14:53:43.914 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 4294967295.9999704, 2024-09-13T14:53:43.914 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000345872, 2024-09-13T14:53:43.914 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:53:43.914 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-13T14:53:43.914 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:53:43.914 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:53:43.914 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:53:43.914 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 189888 2024-09-13T14:53:43.915 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.915 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:53:43.915 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.915 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:53:43.915 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.888308+0000", 2024-09-13T14:53:43.915 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:43.915 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.915 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.915 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:53:43.915 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.888308+0000", 2024-09-13T14:53:43.915 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:43.915 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.915 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.915 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:53:43.916 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.888306+0000", 2024-09-13T14:53:43.916 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:53:43.916 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.916 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.916 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:53:43.916 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.888312+0000", 2024-09-13T14:53:43.916 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0619999999999997e-06 2024-09-13T14:53:43.916 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.916 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.916 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:53:43.916 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.888324+0000", 2024-09-13T14:53:43.916 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1105e-05 2024-09-13T14:53:43.916 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.916 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.917 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:53:43.917 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.888330+0000", 2024-09-13T14:53:43.917 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3319999999999997e-06 2024-09-13T14:53:43.917 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.917 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.917 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:53:43.917 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.888375+0000", 2024-09-13T14:53:43.917 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4879999999999997e-05 2024-09-13T14:53:43.917 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:43.917 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:53:43.917 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:43.917 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.917 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.917 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:189889 9.6 9.323687e6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-09-13T14:53:43.918 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:53:43.888488+0000", 2024-09-13T14:53:43.918 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 4294967295.9997902, 2024-09-13T14:53:43.918 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000228478, 2024-09-13T14:53:43.918 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:53:43.918 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "queued for pg", 2024-09-13T14:53:43.918 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:53:43.918 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:53:43.918 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:53:43.918 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 189889 2024-09-13T14:53:43.918 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.918 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:53:43.918 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.918 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:53:43.919 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.888488+0000", 2024-09-13T14:53:43.919 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:43.919 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.919 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.919 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:53:43.919 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.888488+0000", 2024-09-13T14:53:43.919 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:43.919 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.919 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.919 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:53:43.919 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.888487+0000", 2024-09-13T14:53:43.919 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:53:43.919 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.919 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.920 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:53:43.920 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.888491+0000", 2024-09-13T14:53:43.920 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8809999999999998e-06 2024-09-13T14:53:43.920 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.920 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.920 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:53:43.920 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.888501+0000", 2024-09-13T14:53:43.920 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0206e-05 2024-09-13T14:53:43.920 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.920 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.920 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:53:43.920 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.888507+0000", 2024-09-13T14:53:43.920 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.6629999999999998e-06 2024-09-13T14:53:43.920 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:43.921 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:53:43.921 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:43.921 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.921 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.921 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36528.0:189863 9.13 e415/396 9:ca5f552d:::benchmark_data_smithi102_348563_object68761:head v 415'7552, mlcod=415'7550)", 2024-09-13T14:53:43.921 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:53:43.864466+0000", 2024-09-13T14:53:43.921 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.023811577, 2024-09-13T14:53:43.921 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024301909, 2024-09-13T14:53:43.921 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:53:43.921 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T14:53:43.921 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:53:43.921 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.921 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:53:43.922 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.864466+0000", 2024-09-13T14:53:43.922 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:43.922 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.922 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.922 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:53:43.923 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.864466+0000", 2024-09-13T14:53:43.923 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:43.923 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.923 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.923 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:53:43.923 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.864466+0000", 2024-09-13T14:53:43.923 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967296 2024-09-13T14:53:43.923 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.923 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.923 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:53:43.924 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.864470+0000", 2024-09-13T14:53:43.924 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.456e-06 2024-09-13T14:53:43.924 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.924 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.924 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:53:43.924 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.864475+0000", 2024-09-13T14:53:43.924 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4650000000000002e-06 2024-09-13T14:53:43.924 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.924 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.924 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:53:43.924 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.864479+0000", 2024-09-13T14:53:43.924 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8369999999999999e-06 2024-09-13T14:53:43.924 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.925 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.925 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:53:43.925 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.864814+0000", 2024-09-13T14:53:43.925 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00033505200000000002 2024-09-13T14:53:43.925 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.925 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.925 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:53:43.925 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.864857+0000", 2024-09-13T14:53:43.925 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2673999999999998e-05 2024-09-13T14:53:43.925 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:43.925 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:53:43.925 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:43.925 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.925 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.926 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36528.0:189862 9.b e415/374 9:de74320a:::benchmark_data_smithi102_348563_object68760:head v 415'11736, mlcod=415'11733)", 2024-09-13T14:53:43.926 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:53:43.865537+0000", 2024-09-13T14:53:43.926 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.022740586, 2024-09-13T14:53:43.926 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.023283397000000001, 2024-09-13T14:53:43.926 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:53:43.926 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T14:53:43.926 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:53:43.926 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.926 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:53:43.926 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.865537+0000", 2024-09-13T14:53:43.926 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:43.926 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.926 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.926 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:53:43.927 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.865537+0000", 2024-09-13T14:53:43.927 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:43.927 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.927 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.927 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:53:43.927 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.865537+0000", 2024-09-13T14:53:43.927 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:53:43.927 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.927 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.927 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:53:43.927 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.865542+0000", 2024-09-13T14:53:43.927 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5390000000000001e-06 2024-09-13T14:53:43.927 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.927 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.928 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:53:43.928 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.865548+0000", 2024-09-13T14:53:43.928 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7680000000000003e-06 2024-09-13T14:53:43.928 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.928 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.928 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:53:43.928 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.865553+0000", 2024-09-13T14:53:43.928 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0209999999999999e-06 2024-09-13T14:53:43.928 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.928 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.928 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:53:43.928 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.868556+0000", 2024-09-13T14:53:43.928 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0030023329999999998 2024-09-13T14:53:43.928 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.929 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.929 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:53:43.929 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.868576+0000", 2024-09-13T14:53:43.929 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0458e-05 2024-09-13T14:53:43.929 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:43.929 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:53:43.929 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:43.929 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.929 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.929 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36528.0:189871 9.b e415/374 9:dc63c17c:::benchmark_data_smithi102_348563_object68769:head v 415'11737, mlcod=415'11733)", 2024-09-13T14:53:43.929 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:53:43.873576+0000", 2024-09-13T14:53:43.929 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.01470198, 2024-09-13T14:53:43.929 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.015297115999999999, 2024-09-13T14:53:43.929 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:53:43.930 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T14:53:43.930 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:53:43.930 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.930 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:53:43.930 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.873576+0000", 2024-09-13T14:53:43.930 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:43.930 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.930 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.930 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:53:43.930 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.873576+0000", 2024-09-13T14:53:43.930 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:53:43.930 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.930 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.930 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:53:43.931 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.873575+0000", 2024-09-13T14:53:43.931 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:53:43.931 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.931 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.931 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:53:43.931 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.873583+0000", 2024-09-13T14:53:43.931 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1410000000000005e-06 2024-09-13T14:53:43.931 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.931 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.931 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:53:43.931 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.873596+0000", 2024-09-13T14:53:43.931 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.273e-05 2024-09-13T14:53:43.931 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.931 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.932 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:53:43.932 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.873603+0000", 2024-09-13T14:53:43.932 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2110000000000001e-06 2024-09-13T14:53:43.932 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.932 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.932 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:53:43.932 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.878902+0000", 2024-09-13T14:53:43.932 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0052985460000000003 2024-09-13T14:53:43.932 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:53:43.932 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:53:43.932 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:53:43.932 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:43.878967+0000", 2024-09-13T14:53:43.932 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5592999999999994e-05 2024-09-13T14:53:43.932 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:43.933 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:53:43.933 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:43.933 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:53:43.933 INFO:teuthology.orchestra.run.smithi195.stdout: ], 2024-09-13T14:53:43.933 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 11 2024-09-13T14:53:43.933 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:53:43.985 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:44.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:44.022 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:44.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:44.123 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:44.157 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:44.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:44.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:44.259 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:44.297 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:44.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:44.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:44.399 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:44.437 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:44.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:44.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:44.461 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-13T14:53:44.539 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:44.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:44.576 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:44.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:44.677 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:44.713 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:44.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:44.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:44.717 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T14:53:44.815 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:44.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:44.881 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:44.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:44.982 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:44.999 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:44 smithi195 ceph-mon[104262]: pgmap v1417: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 3.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 989 op/s; 6781/163689 objects degraded (4.143%); 1307/163689 objects misplaced (0.798%); 9.2 MiB/s, 147 objects/s recovering 2024-09-13T14:53:44.999 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:44 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 6781/163689 objects degraded (4.143%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:53:45.017 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:45.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:45.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:45.087 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:53:45.118 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:45.150 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:45.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:45.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:45.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:44 smithi102 ceph-mon[110183]: pgmap v1417: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 3.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 989 op/s; 6781/163689 objects degraded (4.143%); 1307/163689 objects misplaced (0.798%); 9.2 MiB/s, 147 objects/s recovering 2024-09-13T14:53:45.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:44 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 6781/163689 objects degraded (4.143%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:53:45.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:44 smithi102 ceph-mon[115269]: pgmap v1417: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 3.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 989 op/s; 6781/163689 objects degraded (4.143%); 1307/163689 objects misplaced (0.798%); 9.2 MiB/s, 147 objects/s recovering 2024-09-13T14:53:45.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:44 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 6781/163689 objects degraded (4.143%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:53:45.252 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:45.279 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:45.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:45.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:45.381 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:45.412 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:45.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:45.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:45.514 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:45.552 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:45.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:45.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:45.654 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:45.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:45.685 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:45.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:45.786 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:45.821 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:45.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:45.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:45.923 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:45.954 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:45.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:45.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:46.057 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:46.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:46.087 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:46.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:46.188 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:46.218 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:46.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:46.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:46.320 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:46.342 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:46.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:46.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:46.444 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:46.467 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:46.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:46.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:46.570 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:46.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:46.594 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:46.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:46.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:53:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:53:46] "GET /metrics HTTP/1.1" 200 36534 "" "Prometheus/2.51.0" 2024-09-13T14:53:46.695 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:46.717 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:46.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:46.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:46.819 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:46.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:46.845 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:46.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:46.946 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:46.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:46.979 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:46.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:47.080 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:47.104 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:46 smithi195 ceph-mon[104262]: pgmap v1418: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 3.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 985 op/s; 6483/161523 objects degraded (4.014%); 1126/161523 objects misplaced (0.697%); 8.1 MiB/s, 159 objects/s recovering 2024-09-13T14:53:47.104 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:46 smithi195 ceph-mon[104262]: osdmap e416: 8 total, 8 up, 8 in 2024-09-13T14:53:47.113 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:47.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:47.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:47.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:46 smithi102 ceph-mon[110183]: pgmap v1418: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 3.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 985 op/s; 6483/161523 objects degraded (4.014%); 1126/161523 objects misplaced (0.697%); 8.1 MiB/s, 159 objects/s recovering 2024-09-13T14:53:47.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:46 smithi102 ceph-mon[110183]: osdmap e416: 8 total, 8 up, 8 in 2024-09-13T14:53:47.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:46 smithi102 ceph-mon[115269]: pgmap v1418: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 3.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 985 op/s; 6483/161523 objects degraded (4.014%); 1126/161523 objects misplaced (0.697%); 8.1 MiB/s, 159 objects/s recovering 2024-09-13T14:53:47.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:46 smithi102 ceph-mon[115269]: osdmap e416: 8 total, 8 up, 8 in 2024-09-13T14:53:47.215 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:47.248 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:47.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:47.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:47.265 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:53:47.265 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-13T14:53:47.265 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-09-13T14:53:47.265 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-09-13T14:53:47.265 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:53:47.351 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:47.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:47.402 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:47.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:47.503 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:47.542 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:47.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:47.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:47.643 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:47.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:47.679 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:47.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:47.780 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:47.801 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:47.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:47.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:47.813 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:47.870 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:47.903 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:47.933 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:47.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:47.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:48.035 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:48.043 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:47 smithi102 ceph-mon[110183]: 9.10 scrub starts 2024-09-13T14:53:48.043 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:47 smithi102 ceph-mon[110183]: osdmap e417: 8 total, 8 up, 8 in 2024-09-13T14:53:48.043 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:47 smithi102 ceph-mon[115269]: 9.10 scrub starts 2024-09-13T14:53:48.043 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:47 smithi102 ceph-mon[115269]: osdmap e417: 8 total, 8 up, 8 in 2024-09-13T14:53:48.067 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:48.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:48.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:48.169 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:48.206 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:48.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:48.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:48.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:47 smithi195 ceph-mon[104262]: 9.10 scrub starts 2024-09-13T14:53:48.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:47 smithi195 ceph-mon[104262]: osdmap e417: 8 total, 8 up, 8 in 2024-09-13T14:53:48.308 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:48.340 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:48.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:48.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:48.442 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:48.481 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:48.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:48.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:48.583 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:48.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:48.625 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:48.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:48.726 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:48.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:48.767 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:48.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:48.833 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is set 2024-09-13T14:53:48.869 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:48.909 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:48.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:48.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:48.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:48 smithi102 ceph-mon[110183]: pgmap v1421: 119 pgs: 1 active+recovering+undersized+remapped, 1 creating+peering, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 3.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s; 6216/152838 objects degraded (4.067%); 306/152838 objects misplaced (0.200%); 9.5 MiB/s, 194 objects/s recovering 2024-09-13T14:53:48.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:48 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/607223699' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:53:48.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:48 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:53:48.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:48 smithi102 ceph-mon[115269]: pgmap v1421: 119 pgs: 1 active+recovering+undersized+remapped, 1 creating+peering, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 3.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s; 6216/152838 objects degraded (4.067%); 306/152838 objects misplaced (0.200%); 9.5 MiB/s, 194 objects/s recovering 2024-09-13T14:53:48.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:48 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/607223699' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:53:48.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:48 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:53:49.012 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:49.037 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:49.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:49.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:49.139 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:49.166 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:49.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:49.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:49.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:48 smithi195 ceph-mon[104262]: pgmap v1421: 119 pgs: 1 active+recovering+undersized+remapped, 1 creating+peering, 2 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 3.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s; 6216/152838 objects degraded (4.067%); 306/152838 objects misplaced (0.200%); 9.5 MiB/s, 194 objects/s recovering 2024-09-13T14:53:49.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:48 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/607223699' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:53:49.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:48 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:53:49.268 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:49.293 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:49.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:49.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:49.395 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:49.418 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:49.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:49.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:49.520 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:49.544 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:49.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:49.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:49.646 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:49.668 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:49.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:49.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:49.770 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:49.792 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:49.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:49.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:49.895 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:49.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:49.924 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:49.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:50.025 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:50.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:50.056 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:50.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:50.156 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:50.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:49 smithi102 ceph-mon[110183]: 9.16 scrub starts 2024-09-13T14:53:50.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:49 smithi102 ceph-mon[110183]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:53:50.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:49 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:53:50.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:49 smithi102 ceph-mon[110183]: osdmap e418: 8 total, 8 up, 8 in 2024-09-13T14:53:50.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:49 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 6216/152838 objects degraded (4.067%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:53:50.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:49 smithi102 ceph-mon[110183]: osdmap e419: 8 total, 8 up, 8 in 2024-09-13T14:53:50.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:49 smithi102 ceph-mon[115269]: 9.16 scrub starts 2024-09-13T14:53:50.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:49 smithi102 ceph-mon[115269]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:53:50.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:49 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:53:50.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:49 smithi102 ceph-mon[115269]: osdmap e418: 8 total, 8 up, 8 in 2024-09-13T14:53:50.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:49 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 6216/152838 objects degraded (4.067%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:53:50.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:49 smithi102 ceph-mon[115269]: osdmap e419: 8 total, 8 up, 8 in 2024-09-13T14:53:50.183 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:50.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:50.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:50.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:49 smithi195 ceph-mon[104262]: 9.16 scrub starts 2024-09-13T14:53:50.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:49 smithi195 ceph-mon[104262]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:53:50.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:49 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:53:50.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:49 smithi195 ceph-mon[104262]: osdmap e418: 8 total, 8 up, 8 in 2024-09-13T14:53:50.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:49 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 6216/152838 objects degraded (4.067%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:53:50.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:49 smithi195 ceph-mon[104262]: osdmap e419: 8 total, 8 up, 8 in 2024-09-13T14:53:50.284 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:50.319 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:50.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:50.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:50.420 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:50.459 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:50.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:50.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:50.561 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:50.597 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:50.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:50.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:50.699 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:50.723 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:50.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:50.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:50.825 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:50.856 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:50.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:50.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:50.958 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:50.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:50.991 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:50.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:51.092 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:51.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:51.136 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:51.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:51.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:50 smithi102 ceph-mon[115269]: pgmap v1423: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 3.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 810 op/s; 4918/149103 objects degraded (3.298%); 1567/149103 objects misplaced (1.051%); 7.9 MiB/s, 177 objects/s recovering 2024-09-13T14:53:51.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:50 smithi102 ceph-mon[110183]: pgmap v1423: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 3.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 810 op/s; 4918/149103 objects degraded (3.298%); 1567/149103 objects misplaced (1.051%); 7.9 MiB/s, 177 objects/s recovering 2024-09-13T14:53:51.237 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:51.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:50 smithi195 ceph-mon[104262]: pgmap v1423: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 3.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 810 op/s; 4918/149103 objects degraded (3.298%); 1567/149103 objects misplaced (1.051%); 7.9 MiB/s, 177 objects/s recovering 2024-09-13T14:53:51.261 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:51.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:51.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:51.363 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:51.401 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:51.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:51.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:51.502 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:51.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:51.524 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:51.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:51.625 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:51.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:51.660 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:51.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:51.761 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:51.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:51.792 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:51.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:51.893 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:51.928 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:51.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:51.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:52.030 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:52.070 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:52.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:52.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:52.165 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:51 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:53:52.166 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:51 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:53:52.172 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:52.238 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:52.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:52.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:52.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:51 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:53:52.341 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:52.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:52.373 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:52.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:52.474 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:52.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:52.513 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:52.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:52.614 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:52.638 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:52.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:52.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:52.740 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:52.770 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:52.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:52.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:52.872 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:52.906 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T14:53:52.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:52.914 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:52.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:53.015 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:53.021 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:53:53.021 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:53:53.052 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:53:53.052 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:53:53.059 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:53.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:53.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:53.081 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:53:53.081 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:53:53.109 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:53:53.109 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:53:53.109 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:52 smithi102 ceph-mon[115269]: pgmap v1425: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 117 active+clean; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s; 2282/141159 objects degraded (1.617%); 3403/141159 objects misplaced (2.411%); 10 MiB/s, 166 objects/s recovering 2024-09-13T14:53:53.110 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:52 smithi102 ceph-mon[110183]: pgmap v1425: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 117 active+clean; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s; 2282/141159 objects degraded (1.617%); 3403/141159 objects misplaced (2.411%); 10 MiB/s, 166 objects/s recovering 2024-09-13T14:53:53.151 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:52 smithi195 ceph-mon[104262]: pgmap v1425: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 117 active+clean; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s; 2282/141159 objects degraded (1.617%); 3403/141159 objects misplaced (2.411%); 10 MiB/s, 166 objects/s recovering 2024-09-13T14:53:53.151 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:53:53.151 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:53:53.161 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:53.193 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:53:53.193 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:53:53.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:53.219 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:53.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:53.220 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:53:53.220 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:53:53.254 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:53:53.254 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:53:53.320 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:53.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:53.366 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:53.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:53.466 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:53.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:53.529 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:53.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:53.630 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:53.635 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:53:53.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:53.665 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:53.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:53.766 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:53.799 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:53.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:53.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:53.903 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:53.942 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:53.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:53.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:54.037 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:53 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2438530342' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:53:54.037 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:53 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2438530342' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:53:54.044 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:54.081 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:54.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:54.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:54.183 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:54.218 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:54.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:54.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:54.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:53 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2438530342' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:53:54.319 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:54.353 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:54.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:54.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:54.455 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:54.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:54.500 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:54.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:54.601 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:54.632 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:54.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:54.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:54.733 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:54.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:54.766 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:54.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:54.867 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:54.923 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:54.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:54.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:54.975 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:54 smithi195 ceph-mon[104262]: pgmap v1426: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 117 active+clean; 2.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 669 op/s; 2282/138915 objects degraded (1.643%); 3403/138915 objects misplaced (2.450%); 3.9 MiB/s, 63 objects/s recovering 2024-09-13T14:53:54.975 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:54 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 2282/138915 objects degraded (1.643%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:53:55.025 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:55.069 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:55.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:55.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:55.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:54 smithi102 ceph-mon[115269]: pgmap v1426: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 117 active+clean; 2.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 669 op/s; 2282/138915 objects degraded (1.643%); 3403/138915 objects misplaced (2.450%); 3.9 MiB/s, 63 objects/s recovering 2024-09-13T14:53:55.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:54 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 2282/138915 objects degraded (1.643%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:53:55.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:54 smithi102 ceph-mon[110183]: pgmap v1426: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 117 active+clean; 2.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 669 op/s; 2282/138915 objects degraded (1.643%); 3403/138915 objects misplaced (2.450%); 3.9 MiB/s, 63 objects/s recovering 2024-09-13T14:53:55.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:54 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 2282/138915 objects degraded (1.643%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:53:55.171 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:55.211 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:55.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:55.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:55.313 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:55.347 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:55.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:55.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:55.400 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:53:55.450 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:55.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:55.479 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:55.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:55.580 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:55.634 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:55.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:55.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:55.736 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:55.784 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:55.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:55.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:55.886 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:55.904 INFO:teuthology.orchestra.run.smithi102.stderr:reweighted osd.2 to 1 (10000) 2024-09-13T14:53:55.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:55.941 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:55.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:56.042 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:56.077 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:56.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:56.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:56.080 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:53:56.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:55 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3672315064' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:53:56.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:55 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:53:56.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:55 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:53:56.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:55 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:53:56.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:55 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3672315064' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:53:56.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:55 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:53:56.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:55 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:53:56.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:55 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:53:56.181 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:56.226 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:56.227 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:55 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3672315064' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:53:56.227 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:55 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:53:56.227 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:55 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:53:56.227 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:55 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:53:56.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:56.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:56.329 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:56.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:56.378 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:56.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:56.479 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:56.525 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:56.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:56.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:56.627 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:56.650 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:56.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:56.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:56.751 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:56.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:56.774 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:56.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:56.875 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:56.891 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:53:56 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:53:56] "GET /metrics HTTP/1.1" 200 36535 "" "Prometheus/2.51.0" 2024-09-13T14:53:56.906 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:56.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:56.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:56.933 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-13T14:53:57.007 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:57.030 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:57.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:57.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:57.076 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:57.132 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:57.162 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:57.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:56 smithi102 ceph-mon[110183]: pgmap v1427: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 2.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 648 op/s; 2282/137268 objects degraded (1.662%); 3082/137268 objects misplaced (2.245%); 5.9 MiB/s, 94 objects/s recovering 2024-09-13T14:53:57.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:56 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T14:53:57.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:56 smithi102 ceph-mon[110183]: osdmap e420: 8 total, 8 up, 8 in 2024-09-13T14:53:57.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:56 smithi102 ceph-mon[115269]: pgmap v1427: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 2.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 648 op/s; 2282/137268 objects degraded (1.662%); 3082/137268 objects misplaced (2.245%); 5.9 MiB/s, 94 objects/s recovering 2024-09-13T14:53:57.172 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:56 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T14:53:57.172 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:56 smithi102 ceph-mon[115269]: osdmap e420: 8 total, 8 up, 8 in 2024-09-13T14:53:57.209 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:57.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:57.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:57.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:56 smithi195 ceph-mon[104262]: pgmap v1427: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 2.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 648 op/s; 2282/137268 objects degraded (1.662%); 3082/137268 objects misplaced (2.245%); 5.9 MiB/s, 94 objects/s recovering 2024-09-13T14:53:57.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:56 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T14:53:57.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:56 smithi195 ceph-mon[104262]: osdmap e420: 8 total, 8 up, 8 in 2024-09-13T14:53:57.310 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:57.362 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:57.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:57.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:57.464 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:57.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:57.512 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:57.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:57.613 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:57.656 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:57.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:57.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:57.758 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:57.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:57.807 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:57.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:57.867 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:53:57.867 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:53:57.908 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:57.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:57.955 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:57.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:58.056 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:58.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:58.098 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:58.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:58.199 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:58.233 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:58.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:58.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:58.335 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:58.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:58.372 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:58.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:58.473 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:58.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:58.506 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:58.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:58.608 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:58.622 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T14:53:58.650 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:58.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:58.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:58.751 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:58.787 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:58.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:58.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:58.889 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:58.927 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:58.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:58.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:59.029 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:59.068 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:59.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:59.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:59.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:58 smithi102 ceph-mon[110183]: pgmap v1429: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 117 active+clean; 2.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 885 op/s; 2282/127845 objects degraded (1.785%); 2588/127845 objects misplaced (2.024%); 7.6 MiB/s, 120 objects/s recovering 2024-09-13T14:53:59.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:58 smithi102 ceph-mon[115269]: pgmap v1429: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 117 active+clean; 2.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 885 op/s; 2282/127845 objects degraded (1.785%); 2588/127845 objects misplaced (2.024%); 7.6 MiB/s, 120 objects/s recovering 2024-09-13T14:53:59.170 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:59.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:59.222 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:59.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:59.222 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:58 smithi195 ceph-mon[104262]: pgmap v1429: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 117 active+clean; 2.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 885 op/s; 2282/127845 objects degraded (1.785%); 2588/127845 objects misplaced (2.024%); 7.6 MiB/s, 120 objects/s recovering 2024-09-13T14:53:59.323 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:59.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:59.361 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:59.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:59.462 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:59.507 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:59.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:59.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:59.588 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:53:59.609 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:59.639 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:59.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:59.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:59.741 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:59.788 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:53:59.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:59.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:59.891 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:59.928 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:53:59.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:59.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:59.993 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:53:59 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 2282/127845 objects degraded (1.785%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:54:00.030 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:00.079 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:00.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:00.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:00.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:53:59 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 2282/127845 objects degraded (1.785%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:54:00.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:53:59 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 2282/127845 objects degraded (1.785%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:54:00.183 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:00.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:00.220 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:00.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:00.321 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:00.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:00.361 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:00.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:00.462 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:00.496 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:00.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:00.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:00.598 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:00.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:00.633 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:00.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:00.734 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:00.765 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:00.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:00.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:00.867 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:00.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:00.930 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:00.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:01.020 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:54:01.020 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:54:01.031 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:01.087 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:00 smithi102 ceph-mon[110183]: pgmap v1430: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 2.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 853 op/s; 2282/123705 objects degraded (1.845%); 2328/123705 objects misplaced (1.882%); 7.7 MiB/s, 123 objects/s recovering 2024-09-13T14:54:01.087 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:00 smithi102 ceph-mon[115269]: pgmap v1430: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 2.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 853 op/s; 2282/123705 objects degraded (1.845%); 2328/123705 objects misplaced (1.882%); 7.7 MiB/s, 123 objects/s recovering 2024-09-13T14:54:01.089 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:54:01.089 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:54:01.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:01.090 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:01.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:01.135 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:54:01.136 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:54:01.182 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:54:01.182 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:54:01.191 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:01.217 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:01.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:01.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:01.223 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:54:01.223 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:54:01.242 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:00 smithi195 ceph-mon[104262]: pgmap v1430: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 2.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 853 op/s; 2282/123705 objects degraded (1.845%); 2328/123705 objects misplaced (1.882%); 7.7 MiB/s, 123 objects/s recovering 2024-09-13T14:54:01.262 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:54:01.262 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:54:01.272 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:01.295 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:54:01.296 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:54:01.319 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:01.328 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:54:01.328 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:54:01.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:01.377 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:01.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:01.478 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:01.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:01.527 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:01.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:01.628 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:01.694 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:01.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:01.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:01.797 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:01.837 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:01.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:01.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:01.932 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is set 2024-09-13T14:54:01.939 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:01.984 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:01.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:01.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:02.085 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:02.120 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:02.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:02.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:02.222 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:02.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:01 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3461824269' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:54:02.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:01 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/192268132' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:54:02.263 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:02.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:02.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:02.365 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:02.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:02.403 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:02.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:02.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:01 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3461824269' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:54:02.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:01 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/192268132' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:54:02.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:01 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3461824269' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:54:02.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:01 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/192268132' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:54:02.504 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:02.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:02.545 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:02.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:02.646 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:02.681 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:02.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:02.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:02.782 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:02.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:02.826 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:02.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:02.927 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:02.960 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:02.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:02.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:03.062 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:03.103 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:03.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:03.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:03.140 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:02 smithi102 ceph-mon[110183]: pgmap v1431: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 117 active+clean; 2.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 907 op/s; 2282/113922 objects degraded (2.003%); 1858/113922 objects misplaced (1.631%); 6.6 MiB/s, 106 objects/s recovering 2024-09-13T14:54:03.141 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:02 smithi102 ceph-mon[110183]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:54:03.141 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:02 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/192268132' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:54:03.141 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:02 smithi102 ceph-mon[110183]: osdmap e421: 8 total, 8 up, 8 in 2024-09-13T14:54:03.141 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:02 smithi102 ceph-mon[115269]: pgmap v1431: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 117 active+clean; 2.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 907 op/s; 2282/113922 objects degraded (2.003%); 1858/113922 objects misplaced (1.631%); 6.6 MiB/s, 106 objects/s recovering 2024-09-13T14:54:03.141 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:02 smithi102 ceph-mon[115269]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:54:03.141 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:02 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/192268132' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:54:03.141 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:02 smithi102 ceph-mon[115269]: osdmap e421: 8 total, 8 up, 8 in 2024-09-13T14:54:03.205 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:03.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:03.238 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:03.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:03.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:02 smithi195 ceph-mon[104262]: pgmap v1431: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 117 active+clean; 2.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 907 op/s; 2282/113922 objects degraded (2.003%); 1858/113922 objects misplaced (1.631%); 6.6 MiB/s, 106 objects/s recovering 2024-09-13T14:54:03.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:02 smithi195 ceph-mon[104262]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:54:03.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:02 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/192268132' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:54:03.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:02 smithi195 ceph-mon[104262]: osdmap e421: 8 total, 8 up, 8 in 2024-09-13T14:54:03.339 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:03.372 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:03.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:03.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:03.474 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:03.508 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:03.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:03.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:03.610 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:03.641 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:03.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:03.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:03.743 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:03.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:03.780 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:03.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:03.807 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:54:03.807 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-13T14:54:03.807 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.807 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:206731 9.d 9:b4567c5f:::benchmark_data_smithi102_348563_object85629:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e421)", 2024-09-13T14:54:03.807 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:54:03.027892+0000", 2024-09-13T14:54:03.807 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.021642399999999999, 2024-09-13T14:54:03.807 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.021673589, 2024-09-13T14:54:03.807 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:54:03.808 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:54:03.808 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:54:03.808 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:54:03.808 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:54:03.808 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 206731 2024-09-13T14:54:03.808 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.808 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:54:03.808 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.808 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:54:03.808 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.027892+0000", 2024-09-13T14:54:03.809 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:03.809 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.809 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.809 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:54:03.809 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.027892+0000", 2024-09-13T14:54:03.809 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:03.809 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.809 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.809 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:54:03.809 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.027891+0000", 2024-09-13T14:54:03.809 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:54:03.810 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.810 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.810 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:54:03.810 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.027894+0000", 2024-09-13T14:54:03.810 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4619999999999999e-06 2024-09-13T14:54:03.810 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.810 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.810 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:54:03.810 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.027899+0000", 2024-09-13T14:54:03.810 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4920000000000001e-06 2024-09-13T14:54:03.810 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.810 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.811 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:54:03.811 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.027903+0000", 2024-09-13T14:54:03.811 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3669999999999999e-06 2024-09-13T14:54:03.811 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.811 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.811 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:03.811 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.028054+0000", 2024-09-13T14:54:03.811 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015026300000000001 2024-09-13T14:54:03.811 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.811 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.812 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:54:03.812 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.028183+0000", 2024-09-13T14:54:03.812 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012945400000000001 2024-09-13T14:54:03.812 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.812 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.812 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1,5", 2024-09-13T14:54:03.812 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.028303+0000", 2024-09-13T14:54:03.812 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011955699999999999 2024-09-13T14:54:03.812 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.812 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.813 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:03.813 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.035959+0000", 2024-09-13T14:54:03.813 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0076564449999999996 2024-09-13T14:54:03.813 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.813 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.813 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:54:03.813 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.035968+0000", 2024-09-13T14:54:03.813 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1970000000000002e-06 2024-09-13T14:54:03.813 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:03.814 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:54:03.814 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:03.814 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.814 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.814 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:206746 9.17 9.d3ba9a37 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e421)", 2024-09-13T14:54:03.814 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:54:03.044679+0000", 2024-09-13T14:54:03.814 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.0048552719999999999, 2024-09-13T14:54:03.814 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0049714540000000002, 2024-09-13T14:54:03.814 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:54:03.814 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "queued for pg", 2024-09-13T14:54:03.814 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:54:03.815 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:54:03.815 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:54:03.815 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 206746 2024-09-13T14:54:03.815 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.815 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:54:03.815 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.815 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:54:03.815 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.044679+0000", 2024-09-13T14:54:03.815 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:03.815 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.816 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.816 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:54:03.816 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.044679+0000", 2024-09-13T14:54:03.816 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:03.816 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.816 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.816 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:54:03.816 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.044677+0000", 2024-09-13T14:54:03.816 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:54:03.816 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.817 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.817 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:54:03.817 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.044683+0000", 2024-09-13T14:54:03.817 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5990000000000001e-06 2024-09-13T14:54:03.817 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.817 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.817 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:54:03.817 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.044695+0000", 2024-09-13T14:54:03.817 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1606e-05 2024-09-13T14:54:03.817 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.818 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.818 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:54:03.818 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.044701+0000", 2024-09-13T14:54:03.818 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6139999999999997e-06 2024-09-13T14:54:03.818 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:03.818 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:54:03.818 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:03.818 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.818 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.818 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop_reply(client.36528.0:206734 9.17 e421/403 ondisk, result = 0)", 2024-09-13T14:54:03.819 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:54:03.045334+0000", 2024-09-13T14:54:03.819 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.004199698, 2024-09-13T14:54:03.819 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.004750298, 2024-09-13T14:54:03.819 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:54:03.819 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T14:54:03.819 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:54:03.819 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.819 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:54:03.819 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.045334+0000", 2024-09-13T14:54:03.819 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:03.819 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.820 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.820 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:54:03.820 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.045334+0000", 2024-09-13T14:54:03.820 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:03.820 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.820 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.820 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:54:03.820 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.045334+0000", 2024-09-13T14:54:03.820 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:54:03.820 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.820 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.821 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:54:03.821 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.045339+0000", 2024-09-13T14:54:03.821 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3029999999999999e-06 2024-09-13T14:54:03.821 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.821 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.821 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:54:03.821 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.045348+0000", 2024-09-13T14:54:03.821 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7190000000000007e-06 2024-09-13T14:54:03.821 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.821 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.821 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:54:03.822 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.045353+0000", 2024-09-13T14:54:03.822 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3340000000000001e-06 2024-09-13T14:54:03.822 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.822 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.822 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:03.822 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.049171+0000", 2024-09-13T14:54:03.822 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0038176859999999998 2024-09-13T14:54:03.822 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.822 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.822 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:54:03.822 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.049178+0000", 2024-09-13T14:54:03.823 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9419999999999998e-06 2024-09-13T14:54:03.823 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:03.824 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:54:03.824 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:03.824 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.824 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.824 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36528.0:206744 9.5 e421/381 9:a1515a8e:::benchmark_data_smithi102_348563_object85642:head v 421'8339, mlcod=421'8337)", 2024-09-13T14:54:03.824 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:54:03.045358+0000", 2024-09-13T14:54:03.824 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.0041756229999999998, 2024-09-13T14:54:03.824 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0047832780000000002, 2024-09-13T14:54:03.824 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:54:03.825 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T14:54:03.825 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:54:03.825 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.825 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:54:03.825 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.045358+0000", 2024-09-13T14:54:03.825 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:03.825 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.825 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.825 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:54:03.825 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.045358+0000", 2024-09-13T14:54:03.826 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:03.826 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.826 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.826 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:54:03.826 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.045358+0000", 2024-09-13T14:54:03.826 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:54:03.826 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.826 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.826 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:54:03.826 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.045362+0000", 2024-09-13T14:54:03.826 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0080000000000004e-06 2024-09-13T14:54:03.827 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.827 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.827 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:54:03.827 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.045367+0000", 2024-09-13T14:54:03.827 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0119999999999996e-06 2024-09-13T14:54:03.827 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.827 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.827 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:54:03.827 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.045371+0000", 2024-09-13T14:54:03.827 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7299999999999999e-06 2024-09-13T14:54:03.827 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.827 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.827 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:03.828 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.045413+0000", 2024-09-13T14:54:03.828 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1831e-05 2024-09-13T14:54:03.828 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.828 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.828 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:54:03.828 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.045429+0000", 2024-09-13T14:54:03.828 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6303999999999998e-05 2024-09-13T14:54:03.828 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:03.828 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:54:03.829 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:03.829 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.829 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.829 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36528.0:206747 9.c e421/374 9:3ae40bef:::benchmark_data_smithi102_348563_object85645:head v 421'12832, mlcod=421'12829)", 2024-09-13T14:54:03.829 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:54:03.045447+0000", 2024-09-13T14:54:03.829 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.0040869679999999999, 2024-09-13T14:54:03.829 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0047539000000000001, 2024-09-13T14:54:03.829 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:54:03.829 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T14:54:03.829 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:54:03.829 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.829 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:54:03.829 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.045447+0000", 2024-09-13T14:54:03.830 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:03.830 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.830 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.830 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:54:03.830 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.045447+0000", 2024-09-13T14:54:03.830 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:03.830 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.830 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.830 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:54:03.830 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.045446+0000", 2024-09-13T14:54:03.830 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:54:03.830 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.830 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.831 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:54:03.831 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.045452+0000", 2024-09-13T14:54:03.831 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4389999999999999e-06 2024-09-13T14:54:03.831 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.831 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.831 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:54:03.831 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.045461+0000", 2024-09-13T14:54:03.831 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6690000000000001e-06 2024-09-13T14:54:03.831 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.831 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.831 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:54:03.831 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.045467+0000", 2024-09-13T14:54:03.831 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3399999999999997e-06 2024-09-13T14:54:03.832 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.832 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.832 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:03.832 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.045499+0000", 2024-09-13T14:54:03.832 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.1828000000000001e-05 2024-09-13T14:54:03.832 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.832 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.832 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:54:03.832 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.045514+0000", 2024-09-13T14:54:03.832 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5255e-05 2024-09-13T14:54:03.832 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:03.832 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:54:03.833 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:03.833 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.833 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.833 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36528.0:206749 9.9 e421/381 9:9269ca29:::benchmark_data_smithi102_348563_object85647:head v 421'8462, mlcod=421'8460)", 2024-09-13T14:54:03.833 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:54:03.046625+0000", 2024-09-13T14:54:03.833 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.002908487, 2024-09-13T14:54:03.833 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0036284780000000001, 2024-09-13T14:54:03.833 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:54:03.833 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T14:54:03.833 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:54:03.833 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.833 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:54:03.834 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.046625+0000", 2024-09-13T14:54:03.834 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:03.834 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.834 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.834 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:54:03.834 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.046625+0000", 2024-09-13T14:54:03.834 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:03.834 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.834 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.834 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:54:03.834 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.046625+0000", 2024-09-13T14:54:03.834 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:54:03.834 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.834 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.835 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:54:03.835 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.046631+0000", 2024-09-13T14:54:03.835 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.669e-06 2024-09-13T14:54:03.835 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.835 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.835 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:54:03.835 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.046643+0000", 2024-09-13T14:54:03.835 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1873e-05 2024-09-13T14:54:03.835 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.835 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.835 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:54:03.835 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.046650+0000", 2024-09-13T14:54:03.835 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8959999999999997e-06 2024-09-13T14:54:03.835 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.836 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.836 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:03.836 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.046708+0000", 2024-09-13T14:54:03.836 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8281e-05 2024-09-13T14:54:03.836 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.836 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.836 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:54:03.836 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.046728+0000", 2024-09-13T14:54:03.836 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0041999999999999e-05 2024-09-13T14:54:03.836 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:03.836 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:54:03.836 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:03.836 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.836 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.837 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:206734 9.17 9:ecdf2583:::benchmark_data_smithi102_348563_object85632:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e421)", 2024-09-13T14:54:03.837 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:54:03.031827+0000", 2024-09-13T14:54:03.837 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.017706599, 2024-09-13T14:54:03.837 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018481958999999999, 2024-09-13T14:54:03.837 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:54:03.837 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:03.837 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:54:03.837 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:54:03.837 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:54:03.837 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 206734 2024-09-13T14:54:03.837 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.837 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:54:03.837 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.838 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:54:03.838 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.031827+0000", 2024-09-13T14:54:03.838 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:03.838 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.838 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.838 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:54:03.838 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.031827+0000", 2024-09-13T14:54:03.838 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:03.838 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.838 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.838 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:54:03.838 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.031827+0000", 2024-09-13T14:54:03.839 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:54:03.839 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.839 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.839 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:54:03.839 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.031829+0000", 2024-09-13T14:54:03.839 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1380000000000001e-06 2024-09-13T14:54:03.839 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.840 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.840 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:54:03.840 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.031835+0000", 2024-09-13T14:54:03.840 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.6609999999999997e-06 2024-09-13T14:54:03.840 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.840 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.840 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:54:03.840 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.031839+0000", 2024-09-13T14:54:03.840 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4630000000000003e-06 2024-09-13T14:54:03.840 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.840 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.841 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:03.841 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.036292+0000", 2024-09-13T14:54:03.841 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0044524050000000004 2024-09-13T14:54:03.841 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.841 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.841 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:54:03.841 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.036416+0000", 2024-09-13T14:54:03.841 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000124688 2024-09-13T14:54:03.841 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.841 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.841 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:54:03.842 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.038038+0000", 2024-09-13T14:54:03.842 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0016213460000000001 2024-09-13T14:54:03.842 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.842 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.842 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:54:03.842 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.041651+0000", 2024-09-13T14:54:03.842 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0036132389999999999 2024-09-13T14:54:03.842 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.842 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.842 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:03.842 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.049147+0000", 2024-09-13T14:54:03.843 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0074963549999999997 2024-09-13T14:54:03.843 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.843 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.843 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:03.843 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.049183+0000", 2024-09-13T14:54:03.843 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5712e-05 2024-09-13T14:54:03.843 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.843 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.843 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:54:03.843 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.049326+0000", 2024-09-13T14:54:03.844 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000142664 2024-09-13T14:54:03.844 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:03.844 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:54:03.844 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:03.844 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.844 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.844 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:206752 9.17 9.93be19b7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e421)", 2024-09-13T14:54:03.844 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:54:03.049611+0000", 2024-09-13T14:54:03.844 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 4294967295.9999228, 2024-09-13T14:54:03.844 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00078606100000000001, 2024-09-13T14:54:03.844 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:54:03.845 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "queued for pg", 2024-09-13T14:54:03.845 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:54:03.845 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:54:03.845 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:54:03.845 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 206752 2024-09-13T14:54:03.845 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.845 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:54:03.845 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.845 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:54:03.845 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.049611+0000", 2024-09-13T14:54:03.845 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:03.846 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.846 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.846 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:54:03.846 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.049611+0000", 2024-09-13T14:54:03.846 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:03.846 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.846 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.846 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:54:03.846 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.049610+0000", 2024-09-13T14:54:03.846 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:54:03.846 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.847 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.847 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:54:03.847 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.049616+0000", 2024-09-13T14:54:03.847 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3579999999999999e-06 2024-09-13T14:54:03.847 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.847 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.847 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:54:03.847 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.049629+0000", 2024-09-13T14:54:03.847 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2808e-05 2024-09-13T14:54:03.847 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.847 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.848 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:54:03.848 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.049636+0000", 2024-09-13T14:54:03.848 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1720000000000001e-06 2024-09-13T14:54:03.848 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:03.848 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:54:03.848 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:03.848 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.848 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.848 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop_reply(client.36528.0:206738 9.13 e421/396)", 2024-09-13T14:54:03.848 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:54:03.049917+0000", 2024-09-13T14:54:03.848 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 4294967295.9996166, 2024-09-13T14:54:03.849 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00052646300000000002, 2024-09-13T14:54:03.849 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:54:03.849 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "queued for pg", 2024-09-13T14:54:03.849 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:54:03.849 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.849 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:54:03.849 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.049917+0000", 2024-09-13T14:54:03.849 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:03.849 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.849 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.849 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:54:03.849 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.049917+0000", 2024-09-13T14:54:03.850 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:03.850 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.850 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.850 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:54:03.850 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.049916+0000", 2024-09-13T14:54:03.850 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:54:03.850 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.850 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.850 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:54:03.850 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.049921+0000", 2024-09-13T14:54:03.850 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0289999999999996e-06 2024-09-13T14:54:03.851 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.851 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.851 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:54:03.851 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.049931+0000", 2024-09-13T14:54:03.851 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6949999999999995e-06 2024-09-13T14:54:03.851 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.851 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.851 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:54:03.851 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.049937+0000", 2024-09-13T14:54:03.851 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8340000000000004e-06 2024-09-13T14:54:03.851 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:03.851 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:54:03.852 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:03.852 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.852 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.852 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:206738 9.13 9:cc5fdcbb:::benchmark_data_smithi102_348563_object85636:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e421)", 2024-09-13T14:54:03.852 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:54:03.035713+0000", 2024-09-13T14:54:03.852 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.013820918999999999, 2024-09-13T14:54:03.852 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014777696999999999, 2024-09-13T14:54:03.852 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:54:03.852 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:54:03.852 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:54:03.852 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:54:03.853 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:54:03.853 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 206738 2024-09-13T14:54:03.853 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.853 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:54:03.853 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.853 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:54:03.853 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.035713+0000", 2024-09-13T14:54:03.853 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:03.853 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.853 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.854 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:54:03.854 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.035713+0000", 2024-09-13T14:54:03.854 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:03.854 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.854 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.854 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:54:03.854 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.035712+0000", 2024-09-13T14:54:03.854 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:54:03.854 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.854 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.854 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:54:03.854 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.035716+0000", 2024-09-13T14:54:03.855 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9809999999999997e-06 2024-09-13T14:54:03.855 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.855 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.855 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:54:03.855 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.035725+0000", 2024-09-13T14:54:03.855 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.0280000000000006e-06 2024-09-13T14:54:03.855 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.855 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.855 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:54:03.855 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.035731+0000", 2024-09-13T14:54:03.855 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3339999999999998e-06 2024-09-13T14:54:03.855 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.855 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.856 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:03.856 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.037647+0000", 2024-09-13T14:54:03.856 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0019159349999999999 2024-09-13T14:54:03.856 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.856 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.856 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:54:03.856 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.037775+0000", 2024-09-13T14:54:03.856 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000127212 2024-09-13T14:54:03.856 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.856 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.857 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:54:03.857 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.038211+0000", 2024-09-13T14:54:03.857 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000436577 2024-09-13T14:54:03.857 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.857 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.857 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:54:03.857 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.041047+0000", 2024-09-13T14:54:03.857 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0028360770000000002 2024-09-13T14:54:03.857 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:03.857 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:03.857 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:03.857 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:54:03.047144+0000", 2024-09-13T14:54:03.857 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0060968070000000001 2024-09-13T14:54:03.857 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:03.858 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:54:03.858 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:03.858 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:03.858 INFO:teuthology.orchestra.run.smithi195.stdout: ], 2024-09-13T14:54:03.858 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 10 2024-09-13T14:54:03.858 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:54:03.881 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:03.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:03.928 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:03.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:04.029 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:04.066 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:04.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:04.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:04.168 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:04.204 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:04.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:04.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:04.306 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:04.342 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:04.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:04.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:04.444 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:04.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:04.494 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:04.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:04.501 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:54:04.595 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:04.654 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:04.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:04.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:04.756 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:04.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:04.809 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:04.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:04.910 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:04.939 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:04.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:04.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:05.041 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:05.072 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:05.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:05.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:05.078 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:04 smithi102 ceph-mon[110183]: pgmap v1433: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 117 active+clean; 2.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s; 2282/111072 objects degraded (2.055%); 1858/111072 objects misplaced (1.673%); 5.8 MiB/s, 92 objects/s recovering 2024-09-13T14:54:05.078 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:04 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 2282/111072 objects degraded (2.055%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:54:05.078 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:04 smithi102 ceph-mon[115269]: pgmap v1433: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 117 active+clean; 2.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s; 2282/111072 objects degraded (2.055%); 1858/111072 objects misplaced (1.673%); 5.8 MiB/s, 92 objects/s recovering 2024-09-13T14:54:05.078 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:04 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 2282/111072 objects degraded (2.055%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:54:05.173 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:54:05.178 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:05.214 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:05.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:05.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:05.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:04 smithi195 ceph-mon[104262]: pgmap v1433: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 117 active+clean; 2.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s; 2282/111072 objects degraded (2.055%); 1858/111072 objects misplaced (1.673%); 5.8 MiB/s, 92 objects/s recovering 2024-09-13T14:54:05.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:04 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 2282/111072 objects degraded (2.055%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:54:05.316 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:05.374 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:05.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:05.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:05.454 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T14:54:05.476 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:05.501 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:05.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:05.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:05.602 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:05.633 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:05.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:05.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:05.735 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:05.771 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:05.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:05.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:05.792 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:05.873 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:05.917 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:05.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:05.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:05.949 INFO:teuthology.orchestra.run.smithi102.stderr:reweighted osd.1 to 1 (10000) 2024-09-13T14:54:06.019 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:06.059 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:06.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:06.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:06.124 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:05 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1857327059' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:54:06.161 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:06.192 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:06.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:06.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:06.294 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:06.333 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:06.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:06.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:06.362 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:54:06.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:05 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1857327059' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:54:06.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:05 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1857327059' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:54:06.435 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:06.487 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:06.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:06.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:06.590 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:06.617 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:06.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:06.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:06.719 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:06.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:06.760 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:06.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:06.834 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:54:06 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:54:06] "GET /metrics HTTP/1.1" 200 36535 "" "Prometheus/2.51.0" 2024-09-13T14:54:06.861 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:06.895 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:06.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:06.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:06.997 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:07.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:07.031 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:07.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:07.132 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:07.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:06 smithi102 ceph-mon[110183]: pgmap v1434: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 2.2 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 947 op/s; 2282/109269 objects degraded (2.088%); 1571/109269 objects misplaced (1.438%); 6.5 MiB/s, 104 objects/s recovering 2024-09-13T14:54:07.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:06 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1857327059' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T14:54:07.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:06 smithi102 ceph-mon[110183]: osdmap e422: 8 total, 8 up, 8 in 2024-09-13T14:54:07.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:06 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:54:07.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:06 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:54:07.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:06 smithi102 ceph-mon[115269]: pgmap v1434: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 2.2 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 947 op/s; 2282/109269 objects degraded (2.088%); 1571/109269 objects misplaced (1.438%); 6.5 MiB/s, 104 objects/s recovering 2024-09-13T14:54:07.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:06 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1857327059' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T14:54:07.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:06 smithi102 ceph-mon[115269]: osdmap e422: 8 total, 8 up, 8 in 2024-09-13T14:54:07.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:06 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:54:07.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:06 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:54:07.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:07.172 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:07.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:07.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:06 smithi195 ceph-mon[104262]: pgmap v1434: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 2.2 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 947 op/s; 2282/109269 objects degraded (2.088%); 1571/109269 objects misplaced (1.438%); 6.5 MiB/s, 104 objects/s recovering 2024-09-13T14:54:07.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:06 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1857327059' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T14:54:07.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:06 smithi195 ceph-mon[104262]: osdmap e422: 8 total, 8 up, 8 in 2024-09-13T14:54:07.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:06 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:54:07.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:06 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:54:07.274 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:07.278 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-13T14:54:07.278 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:54:07.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:07.320 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:07.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:07.421 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:07.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:07.485 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:07.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:07.586 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:07.637 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:07.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:07.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:07.739 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:07.783 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:07.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:07.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:07.802 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:07.886 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:07.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:07.929 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:07.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:08.030 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:08.069 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:08.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:08.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:08.171 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:08.212 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:08.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:08.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:08.314 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:08.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:08.360 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:08.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:08.461 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:08.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:08.508 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:08.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:08.609 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:08.644 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:08.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:08.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:08.746 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:08.779 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:08.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:08.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:08.849 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:54:08.849 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-13T14:54:08.849 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-09-13T14:54:08.849 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-09-13T14:54:08.849 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:54:08.885 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:08.918 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:08.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:08.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:09.020 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:09.068 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:09.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:09.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:09.170 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:09.205 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:09.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:09.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:09.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:08 smithi195 ceph-mon[104262]: pgmap v1436: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 117 active+clean; 2.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 2282/99156 objects degraded (2.301%); 1029/99156 objects misplaced (1.038%); 6.2 MiB/s, 99 objects/s recovering 2024-09-13T14:54:09.307 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:09.350 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:09.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:09.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:09.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:08 smithi102 ceph-mon[110183]: pgmap v1436: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 117 active+clean; 2.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 2282/99156 objects degraded (2.301%); 1029/99156 objects misplaced (1.038%); 6.2 MiB/s, 99 objects/s recovering 2024-09-13T14:54:09.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:08 smithi102 ceph-mon[115269]: pgmap v1436: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 117 active+clean; 2.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 2282/99156 objects degraded (2.301%); 1029/99156 objects misplaced (1.038%); 6.2 MiB/s, 99 objects/s recovering 2024-09-13T14:54:09.451 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:09.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:09.496 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:09.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:09.562 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T14:54:09.597 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:09.656 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:09.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:09.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:09.758 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:09.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:09.809 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:09.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:09.891 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:54:09.910 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:09.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:09.962 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:09.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:10.063 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:10.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:10.098 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:10.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:10.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:09 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 2282/99156 objects degraded (2.301%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:54:10.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:09 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 2282/99156 objects degraded (2.301%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:54:10.198 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:10.232 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:10.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:10.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:10.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:09 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 2282/99156 objects degraded (2.301%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:54:10.334 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:10.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:10.373 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:10.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:10.474 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:10.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:10.521 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:10.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:10.622 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:10.651 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:10.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:10.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:10.753 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:10.788 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:10.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:10.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:10.890 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:10.929 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:10.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:10.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:11.040 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:11.086 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:11.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:11.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:11.188 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:11.206 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:10 smithi195 ceph-mon[104262]: pgmap v1437: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 1.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 795 op/s; 2282/94815 objects degraded (2.407%); 739/94815 objects misplaced (0.779%); 6.6 MiB/s, 105 objects/s recovering 2024-09-13T14:54:11.206 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:54:11.208 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:11.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:11.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:11.310 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:11.346 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:11.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:11.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:11.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:10 smithi102 ceph-mon[110183]: pgmap v1437: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 1.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 795 op/s; 2282/94815 objects degraded (2.407%); 739/94815 objects misplaced (0.779%); 6.6 MiB/s, 105 objects/s recovering 2024-09-13T14:54:11.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:54:11.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:10 smithi102 ceph-mon[115269]: pgmap v1437: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 1.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 795 op/s; 2282/94815 objects degraded (2.407%); 739/94815 objects misplaced (0.779%); 6.6 MiB/s, 105 objects/s recovering 2024-09-13T14:54:11.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:54:11.448 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:11.470 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:11.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:11.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:11.572 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:11.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:11.613 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:11.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:11.714 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:11.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:11.757 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:11.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:11.858 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:11.895 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:11.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:11.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:11.959 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:54:11.959 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-09-13T14:54:11.959 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-09-13T14:54:11.959 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-13T14:54:11.959 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.959 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:72299 9.13 9:c8621b29:::benchmark_data_smithi102_348563_object72298:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-09-13T14:54:11.959 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:51:46.193942+0000", 2024-09-13T14:54:11.959 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 145.75996578499999, 2024-09-13T14:54:11.959 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1183224979999999, 2024-09-13T14:54:11.960 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:54:11.960 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:11.960 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:54:11.960 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:54:11.960 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:54:11.960 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 72299 2024-09-13T14:54:11.960 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.960 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:54:11.960 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.960 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:54:11.960 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.193942+0000", 2024-09-13T14:54:11.960 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:11.961 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.961 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.961 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:54:11.961 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.193942+0000", 2024-09-13T14:54:11.961 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:11.961 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.961 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.961 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:54:11.961 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.193940+0000", 2024-09-13T14:54:11.961 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:54:11.962 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.962 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.962 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:54:11.962 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.193990+0000", 2024-09-13T14:54:11.963 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9901999999999998e-05 2024-09-13T14:54:11.963 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.963 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.963 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:54:11.964 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.194003+0000", 2024-09-13T14:54:11.964 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3222e-05 2024-09-13T14:54:11.964 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.964 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.964 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:54:11.964 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.194009+0000", 2024-09-13T14:54:11.964 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7710000000000003e-06 2024-09-13T14:54:11.964 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.964 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.964 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:11.964 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.194063+0000", 2024-09-13T14:54:11.965 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3977999999999997e-05 2024-09-13T14:54:11.965 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.965 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.965 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:54:11.965 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.194076+0000", 2024-09-13T14:54:11.965 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3288000000000001e-05 2024-09-13T14:54:11.966 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.966 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.966 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:11.966 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.056600+0000", 2024-09-13T14:54:11.966 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.86252373199999999 2024-09-13T14:54:11.966 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.966 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.966 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:54:11.966 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.056614+0000", 2024-09-13T14:54:11.966 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3764999999999999e-05 2024-09-13T14:54:11.967 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.967 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.967 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:11.967 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.302885+0000", 2024-09-13T14:54:11.967 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.246271474 2024-09-13T14:54:11.967 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.967 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.967 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:54:11.967 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.303015+0000", 2024-09-13T14:54:11.967 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012967 2024-09-13T14:54:11.967 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.967 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.967 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:54:11.968 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.303117+0000", 2024-09-13T14:54:11.968 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010189800000000001 2024-09-13T14:54:11.968 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.968 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.968 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:54:11.968 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.307981+0000", 2024-09-13T14:54:11.968 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0048640360000000004 2024-09-13T14:54:11.968 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.968 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.968 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:11.968 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.308022+0000", 2024-09-13T14:54:11.969 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1356999999999997e-05 2024-09-13T14:54:11.969 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.969 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.969 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:11.969 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.310751+0000", 2024-09-13T14:54:11.969 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0027289879999999999 2024-09-13T14:54:11.969 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.969 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.969 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:54:11.969 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.310808+0000", 2024-09-13T14:54:11.969 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.6246000000000003e-05 2024-09-13T14:54:11.970 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.970 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.970 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:54:11.970 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.312264+0000", 2024-09-13T14:54:11.970 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001456511 2024-09-13T14:54:11.970 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:11.971 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:54:11.971 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:11.971 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.971 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.971 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:72416 9.13 9:ca62faaf:::benchmark_data_smithi102_348563_object72415:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-09-13T14:54:11.971 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:51:46.289815+0000", 2024-09-13T14:54:11.971 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 145.66409182800001, 2024-09-13T14:54:11.971 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0320571169999999, 2024-09-13T14:54:11.971 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:54:11.971 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:11.971 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:54:11.972 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:54:11.972 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:54:11.972 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 72416 2024-09-13T14:54:11.972 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.972 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:54:11.972 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.972 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:54:11.972 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.289815+0000", 2024-09-13T14:54:11.972 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:11.972 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.972 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.973 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:54:11.973 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.289815+0000", 2024-09-13T14:54:11.973 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:11.973 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.973 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.973 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:54:11.974 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.289809+0000", 2024-09-13T14:54:11.974 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999938 2024-09-13T14:54:11.974 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.974 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.974 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:54:11.974 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.289859+0000", 2024-09-13T14:54:11.974 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9295000000000002e-05 2024-09-13T14:54:11.974 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.974 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.974 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:54:11.974 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.289870+0000", 2024-09-13T14:54:11.974 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1575e-05 2024-09-13T14:54:11.974 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.975 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.975 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:54:11.975 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.289877+0000", 2024-09-13T14:54:11.975 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.0670000000000003e-06 2024-09-13T14:54:11.975 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.975 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.975 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:11.975 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.289939+0000", 2024-09-13T14:54:11.975 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1532999999999995e-05 2024-09-13T14:54:11.975 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.975 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.975 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:54:11.975 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.289956+0000", 2024-09-13T14:54:11.975 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7453999999999999e-05 2024-09-13T14:54:11.976 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.976 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.976 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:11.976 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.056739+0000", 2024-09-13T14:54:11.976 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.76678289799999999 2024-09-13T14:54:11.976 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.976 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.976 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:54:11.976 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.056751+0000", 2024-09-13T14:54:11.976 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1839e-05 2024-09-13T14:54:11.976 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.976 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.976 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:11.977 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.305540+0000", 2024-09-13T14:54:11.977 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2487891710000001 2024-09-13T14:54:11.977 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.977 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.977 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:54:11.977 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.305705+0000", 2024-09-13T14:54:11.977 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000164547 2024-09-13T14:54:11.977 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.977 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.977 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:54:11.977 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.305889+0000", 2024-09-13T14:54:11.977 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0001839 2024-09-13T14:54:11.977 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.978 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.978 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:54:11.978 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.308075+0000", 2024-09-13T14:54:11.978 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002185977 2024-09-13T14:54:11.978 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.978 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.978 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:11.978 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.312336+0000", 2024-09-13T14:54:11.978 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0042612129999999998 2024-09-13T14:54:11.978 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.978 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.978 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:11.978 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.320281+0000", 2024-09-13T14:54:11.979 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0079447949999999993 2024-09-13T14:54:11.979 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.979 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.979 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:54:11.979 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.320333+0000", 2024-09-13T14:54:11.979 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1956999999999997e-05 2024-09-13T14:54:11.979 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.979 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.979 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:54:11.979 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.321873+0000", 2024-09-13T14:54:11.979 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001540076 2024-09-13T14:54:11.980 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:11.980 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:54:11.980 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:11.980 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.980 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.980 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:72422 9.13 9:cc800bcd:::benchmark_data_smithi102_348563_object72421:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-09-13T14:54:11.980 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:51:46.291786+0000", 2024-09-13T14:54:11.980 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 145.662120902, 2024-09-13T14:54:11.980 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0314244129999999, 2024-09-13T14:54:11.981 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:54:11.981 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:11.981 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:54:11.981 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:54:11.981 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:54:11.981 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 72422 2024-09-13T14:54:11.981 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.981 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:54:11.981 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.981 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:54:11.981 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.291786+0000", 2024-09-13T14:54:11.981 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:11.981 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.982 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.982 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:54:11.982 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.291786+0000", 2024-09-13T14:54:11.982 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:11.982 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.982 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.982 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:54:11.982 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.291786+0000", 2024-09-13T14:54:11.982 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:54:11.982 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.982 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.982 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:54:11.983 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.291867+0000", 2024-09-13T14:54:11.983 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1588000000000006e-05 2024-09-13T14:54:11.983 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.983 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.983 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:54:11.983 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.291881+0000", 2024-09-13T14:54:11.983 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3638000000000001e-05 2024-09-13T14:54:11.983 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.983 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.983 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:54:11.983 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.291888+0000", 2024-09-13T14:54:11.983 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4610000000000002e-06 2024-09-13T14:54:11.983 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.984 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.984 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:11.984 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.291997+0000", 2024-09-13T14:54:11.984 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010846800000000001 2024-09-13T14:54:11.984 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.984 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.984 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:54:11.984 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.292005+0000", 2024-09-13T14:54:11.984 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9459999999999998e-06 2024-09-13T14:54:11.984 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.984 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.984 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:11.985 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.056783+0000", 2024-09-13T14:54:11.985 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.76477843000000001 2024-09-13T14:54:11.985 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.985 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.985 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:54:11.985 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.056792+0000", 2024-09-13T14:54:11.985 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8200000000000003e-06 2024-09-13T14:54:11.985 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.985 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.985 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:11.985 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.306918+0000", 2024-09-13T14:54:11.985 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2501263090000001 2024-09-13T14:54:11.986 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.986 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.986 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:54:11.986 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.307041+0000", 2024-09-13T14:54:11.986 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012237399999999999 2024-09-13T14:54:11.986 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.986 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.986 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:54:11.986 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.307146+0000", 2024-09-13T14:54:11.986 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000105288 2024-09-13T14:54:11.986 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.986 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.986 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:54:11.987 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.313885+0000", 2024-09-13T14:54:11.987 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0067393590000000003 2024-09-13T14:54:11.987 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.987 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.987 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:11.987 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.315634+0000", 2024-09-13T14:54:11.987 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001748318 2024-09-13T14:54:11.987 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.987 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.987 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:11.987 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.321938+0000", 2024-09-13T14:54:11.987 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0063039960000000001 2024-09-13T14:54:11.987 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.988 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.988 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:54:11.988 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.321991+0000", 2024-09-13T14:54:11.988 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3810000000000001e-05 2024-09-13T14:54:11.988 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.988 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.988 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:54:11.988 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.323211+0000", 2024-09-13T14:54:11.988 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0012194949999999999 2024-09-13T14:54:11.988 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:11.989 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:54:11.989 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:11.989 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.989 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.989 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:72445 9.13 9:c8139549:::benchmark_data_smithi102_348563_object72444:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-09-13T14:54:11.989 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:51:46.307279+0000", 2024-09-13T14:54:11.989 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 145.64662840700001, 2024-09-13T14:54:11.989 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0180606839999999, 2024-09-13T14:54:11.989 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:54:11.989 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:11.989 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:54:11.990 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:54:11.990 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:54:11.990 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 72445 2024-09-13T14:54:11.990 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.990 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:54:11.990 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.990 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:54:11.990 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.307279+0000", 2024-09-13T14:54:11.990 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:11.990 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.991 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.991 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:54:11.991 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.307279+0000", 2024-09-13T14:54:11.991 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:11.991 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.991 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.991 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:54:11.991 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.307278+0000", 2024-09-13T14:54:11.991 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:54:11.991 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.991 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.992 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:54:11.992 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.307327+0000", 2024-09-13T14:54:11.992 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8435e-05 2024-09-13T14:54:11.992 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.992 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.992 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:54:11.992 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.307333+0000", 2024-09-13T14:54:11.992 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5989999999999998e-06 2024-09-13T14:54:11.992 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.992 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.992 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:54:11.992 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.307338+0000", 2024-09-13T14:54:11.993 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0300000000000001e-06 2024-09-13T14:54:11.993 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.993 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.993 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:11.993 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.307385+0000", 2024-09-13T14:54:11.993 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6740000000000003e-05 2024-09-13T14:54:11.993 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.993 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.993 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:54:11.993 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.307397+0000", 2024-09-13T14:54:11.993 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1565e-05 2024-09-13T14:54:11.993 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.994 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.994 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:11.995 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.056816+0000", 2024-09-13T14:54:11.995 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.74941926000000003 2024-09-13T14:54:11.995 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.995 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.995 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:54:11.995 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.056825+0000", 2024-09-13T14:54:11.996 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.6719999999999993e-06 2024-09-13T14:54:11.996 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.996 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.996 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:11.996 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.308111+0000", 2024-09-13T14:54:11.997 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.251286127 2024-09-13T14:54:11.997 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.997 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.998 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:54:11.998 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.308256+0000", 2024-09-13T14:54:11.998 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014530000000000001 2024-09-13T14:54:11.998 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.998 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.998 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:54:11.998 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.308423+0000", 2024-09-13T14:54:11.999 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016677200000000001 2024-09-13T14:54:11.999 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.999 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.999 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:54:11.999 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.313898+0000", 2024-09-13T14:54:11.999 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0054755699999999999 2024-09-13T14:54:11.999 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.999 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:11.999 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:11.999 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.315748+0000", 2024-09-13T14:54:11.999 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0018497710000000001 2024-09-13T14:54:11.999 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:11.999 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.000 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:12.000 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.323271+0000", 2024-09-13T14:54:12.000 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0075229169999999996 2024-09-13T14:54:12.000 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.000 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.000 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:54:12.000 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.323324+0000", 2024-09-13T14:54:12.000 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2889000000000001e-05 2024-09-13T14:54:12.000 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.000 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.001 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:54:12.001 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.325340+0000", 2024-09-13T14:54:12.001 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0020155580000000002 2024-09-13T14:54:12.001 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:12.001 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:54:12.001 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:12.001 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.001 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.001 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:72478 9.13 9:cd39ee78:::benchmark_data_smithi102_348563_object72477:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-09-13T14:54:12.001 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:51:46.334058+0000", 2024-09-13T14:54:12.001 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 145.61984915100001, 2024-09-13T14:54:12.001 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.016141556, 2024-09-13T14:54:12.001 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:54:12.002 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:12.002 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:54:12.002 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:54:12.002 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:54:12.002 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 72478 2024-09-13T14:54:12.002 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.002 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:54:12.002 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.002 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:54:12.002 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.334058+0000", 2024-09-13T14:54:12.002 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:12.003 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.003 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.003 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:54:12.003 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.334058+0000", 2024-09-13T14:54:12.003 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:12.003 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.003 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.003 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:54:12.003 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.334057+0000", 2024-09-13T14:54:12.003 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:54:12.003 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.004 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.004 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:54:12.004 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.334087+0000", 2024-09-13T14:54:12.004 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0477999999999999e-05 2024-09-13T14:54:12.004 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.004 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.004 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:54:12.004 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.334098+0000", 2024-09-13T14:54:12.004 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0684e-05 2024-09-13T14:54:12.004 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.004 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.004 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:54:12.005 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.334105+0000", 2024-09-13T14:54:12.005 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6080000000000001e-06 2024-09-13T14:54:12.005 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.005 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.005 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:12.005 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.334197+0000", 2024-09-13T14:54:12.005 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.2443999999999998e-05 2024-09-13T14:54:12.005 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.005 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.005 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:54:12.005 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.334207+0000", 2024-09-13T14:54:12.005 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0013000000000001e-05 2024-09-13T14:54:12.006 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.006 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.006 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:12.006 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.056863+0000", 2024-09-13T14:54:12.006 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.72265576499999995 2024-09-13T14:54:12.006 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.006 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.006 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:54:12.006 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.056870+0000", 2024-09-13T14:54:12.006 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8569999999999997e-06 2024-09-13T14:54:12.006 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.006 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.007 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:12.007 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.309442+0000", 2024-09-13T14:54:12.007 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2525720090000001 2024-09-13T14:54:12.007 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.007 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.007 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:54:12.007 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.309592+0000", 2024-09-13T14:54:12.007 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015071600000000001 2024-09-13T14:54:12.007 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.007 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.007 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:54:12.007 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.309748+0000", 2024-09-13T14:54:12.008 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015559800000000001 2024-09-13T14:54:12.008 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.008 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.008 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:54:12.008 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.313905+0000", 2024-09-13T14:54:12.008 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0041569320000000003 2024-09-13T14:54:12.008 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.008 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.008 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:12.008 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.325405+0000", 2024-09-13T14:54:12.008 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011499578999999999 2024-09-13T14:54:12.008 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.009 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.009 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:12.009 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.348507+0000", 2024-09-13T14:54:12.009 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.023102192000000001 2024-09-13T14:54:12.009 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.009 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.009 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:54:12.009 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.348573+0000", 2024-09-13T14:54:12.009 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6265000000000003e-05 2024-09-13T14:54:12.009 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.010 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.010 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:54:12.010 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.350200+0000", 2024-09-13T14:54:12.010 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0016267650000000001 2024-09-13T14:54:12.010 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:12.010 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:54:12.010 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:12.010 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.010 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.010 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:72481 9.13 9:c872827f:::benchmark_data_smithi102_348563_object72480:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-09-13T14:54:12.010 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:51:46.336298+0000", 2024-09-13T14:54:12.010 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 145.61760885499999, 2024-09-13T14:54:12.011 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0149789880000002, 2024-09-13T14:54:12.011 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:54:12.011 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:12.011 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:54:12.011 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:54:12.011 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:54:12.011 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 72481 2024-09-13T14:54:12.011 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.011 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:54:12.011 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.011 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:54:12.011 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.336298+0000", 2024-09-13T14:54:12.011 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:12.012 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.012 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.012 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:54:12.012 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.336298+0000", 2024-09-13T14:54:12.012 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:12.012 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.012 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.012 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:54:12.012 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.336297+0000", 2024-09-13T14:54:12.012 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:54:12.012 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.012 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.013 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:54:12.013 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.336328+0000", 2024-09-13T14:54:12.013 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0834000000000003e-05 2024-09-13T14:54:12.013 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.013 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.013 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:54:12.013 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.336340+0000", 2024-09-13T14:54:12.013 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2451e-05 2024-09-13T14:54:12.013 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.014 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.014 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:54:12.014 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.336347+0000", 2024-09-13T14:54:12.014 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1339999999999998e-06 2024-09-13T14:54:12.014 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.014 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.014 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:12.014 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.336403+0000", 2024-09-13T14:54:12.014 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5198999999999998e-05 2024-09-13T14:54:12.014 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.014 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.014 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:54:12.014 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.336416+0000", 2024-09-13T14:54:12.014 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3451999999999999e-05 2024-09-13T14:54:12.015 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.015 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.015 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:12.015 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.056988+0000", 2024-09-13T14:54:12.015 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.72057172000000003 2024-09-13T14:54:12.015 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.015 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.015 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:54:12.015 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.056993+0000", 2024-09-13T14:54:12.016 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5690000000000004e-06 2024-09-13T14:54:12.016 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.016 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.016 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:12.016 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.312379+0000", 2024-09-13T14:54:12.016 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.255385564 2024-09-13T14:54:12.016 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.016 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.016 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:54:12.016 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.312524+0000", 2024-09-13T14:54:12.016 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014541200000000001 2024-09-13T14:54:12.017 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.017 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.017 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:54:12.017 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.312690+0000", 2024-09-13T14:54:12.017 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016532 2024-09-13T14:54:12.017 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.017 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.017 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:54:12.017 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.318889+0000", 2024-09-13T14:54:12.017 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.006199202 2024-09-13T14:54:12.017 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.017 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.018 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:12.018 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.325439+0000", 2024-09-13T14:54:12.018 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0065503610000000002 2024-09-13T14:54:12.018 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.018 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.018 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:12.018 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.350254+0000", 2024-09-13T14:54:12.018 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024814994999999999 2024-09-13T14:54:12.018 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.018 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.018 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:54:12.018 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.350321+0000", 2024-09-13T14:54:12.018 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7285000000000004e-05 2024-09-13T14:54:12.019 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.019 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.019 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:54:12.019 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.351277+0000", 2024-09-13T14:54:12.019 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000955977 2024-09-13T14:54:12.019 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:12.019 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:54:12.019 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:12.019 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.019 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.019 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:72521 9.13 9:cd0cc28e:::benchmark_data_smithi102_348563_object72520:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-09-13T14:54:12.019 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:51:46.367757+0000", 2024-09-13T14:54:12.019 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 145.58615066499999, 2024-09-13T14:54:12.020 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.984496971, 2024-09-13T14:54:12.020 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:54:12.020 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:12.020 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:54:12.020 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:54:12.020 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:54:12.020 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 72521 2024-09-13T14:54:12.020 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.021 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:54:12.021 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.021 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:54:12.021 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.367757+0000", 2024-09-13T14:54:12.021 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:12.021 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.021 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.021 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:54:12.021 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.367757+0000", 2024-09-13T14:54:12.021 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:12.021 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.021 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.021 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:54:12.022 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.367755+0000", 2024-09-13T14:54:12.022 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:54:12.022 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.022 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.022 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:54:12.022 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.367856+0000", 2024-09-13T14:54:12.022 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000100467 2024-09-13T14:54:12.022 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.022 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.022 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:54:12.023 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.367865+0000", 2024-09-13T14:54:12.023 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.833e-06 2024-09-13T14:54:12.023 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.023 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.023 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:54:12.023 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.367871+0000", 2024-09-13T14:54:12.023 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3180000000000003e-06 2024-09-13T14:54:12.023 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.023 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.023 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:12.023 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.368022+0000", 2024-09-13T14:54:12.023 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015103 2024-09-13T14:54:12.023 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.023 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.024 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:54:12.024 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.368032+0000", 2024-09-13T14:54:12.024 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.024e-05 2024-09-13T14:54:12.024 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.024 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.024 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:12.024 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.057022+0000", 2024-09-13T14:54:12.024 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.68899013099999995 2024-09-13T14:54:12.024 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.024 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.024 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:54:12.024 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.057033+0000", 2024-09-13T14:54:12.025 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0335e-05 2024-09-13T14:54:12.025 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.025 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.025 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:12.025 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.313958+0000", 2024-09-13T14:54:12.025 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.256924937 2024-09-13T14:54:12.025 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.026 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.026 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:54:12.026 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.314126+0000", 2024-09-13T14:54:12.026 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000167978 2024-09-13T14:54:12.026 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.026 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.026 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:54:12.027 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.314336+0000", 2024-09-13T14:54:12.027 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000209855 2024-09-13T14:54:12.027 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.027 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.027 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:54:12.027 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.320183+0000", 2024-09-13T14:54:12.027 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0058473869999999999 2024-09-13T14:54:12.028 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.028 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.028 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:12.028 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.325473+0000", 2024-09-13T14:54:12.028 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0052903059999999998 2024-09-13T14:54:12.029 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.029 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.029 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:12.029 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.351404+0000", 2024-09-13T14:54:12.029 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.025930947999999999 2024-09-13T14:54:12.029 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.029 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.029 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:54:12.030 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.351459+0000", 2024-09-13T14:54:12.030 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5000999999999997e-05 2024-09-13T14:54:12.030 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.030 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.030 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:54:12.031 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.352254+0000", 2024-09-13T14:54:12.031 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00079445299999999998 2024-09-13T14:54:12.031 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:12.031 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:54:12.031 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:12.032 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.032 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.032 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:72531 9.13 9:cd5ddef9:::benchmark_data_smithi102_348563_object72530:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-09-13T14:54:12.032 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:51:46.381783+0000", 2024-09-13T14:54:12.032 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 145.57212433399999, 2024-09-13T14:54:12.032 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.971096263, 2024-09-13T14:54:12.033 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:54:12.033 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:12.033 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:54:12.033 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:54:12.034 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:54:12.034 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 72531 2024-09-13T14:54:12.034 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.034 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:54:12.034 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.034 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:54:12.035 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.381783+0000", 2024-09-13T14:54:12.035 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:12.035 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.035 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.035 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:54:12.035 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.381783+0000", 2024-09-13T14:54:12.035 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:12.035 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.035 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.035 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:54:12.035 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.381781+0000", 2024-09-13T14:54:12.035 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:54:12.036 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.036 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.036 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:54:12.036 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.381839+0000", 2024-09-13T14:54:12.036 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8542999999999998e-05 2024-09-13T14:54:12.036 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.036 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.036 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:54:12.036 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.381853+0000", 2024-09-13T14:54:12.036 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3885e-05 2024-09-13T14:54:12.036 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.036 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.036 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:54:12.037 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.381861+0000", 2024-09-13T14:54:12.037 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7260000000000002e-06 2024-09-13T14:54:12.037 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.037 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.037 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:12.037 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.381922+0000", 2024-09-13T14:54:12.037 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0841000000000002e-05 2024-09-13T14:54:12.037 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.037 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.037 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:54:12.037 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.381931+0000", 2024-09-13T14:54:12.037 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.3510000000000008e-06 2024-09-13T14:54:12.037 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.038 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.038 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:12.038 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.057056+0000", 2024-09-13T14:54:12.038 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.67512471100000004 2024-09-13T14:54:12.038 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.038 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.038 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:54:12.038 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.057061+0000", 2024-09-13T14:54:12.038 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6009999999999997e-06 2024-09-13T14:54:12.038 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.038 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.038 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:12.038 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.315777+0000", 2024-09-13T14:54:12.038 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2587159479999999 2024-09-13T14:54:12.039 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.039 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.039 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:54:12.039 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.315929+0000", 2024-09-13T14:54:12.039 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015216700000000001 2024-09-13T14:54:12.039 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.039 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.039 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:54:12.039 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.316179+0000", 2024-09-13T14:54:12.039 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000250709 2024-09-13T14:54:12.039 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.039 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.039 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:54:12.040 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.320189+0000", 2024-09-13T14:54:12.040 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0040100420000000001 2024-09-13T14:54:12.040 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.040 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.040 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:12.040 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.333905+0000", 2024-09-13T14:54:12.040 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013715283999999999 2024-09-13T14:54:12.040 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.040 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.040 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:12.040 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.352333+0000", 2024-09-13T14:54:12.040 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018428402999999999 2024-09-13T14:54:12.041 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.041 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.041 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:54:12.041 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.352367+0000", 2024-09-13T14:54:12.041 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.3874999999999997e-05 2024-09-13T14:54:12.041 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.041 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.042 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:54:12.042 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.352879+0000", 2024-09-13T14:54:12.042 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00051223200000000005 2024-09-13T14:54:12.042 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:12.042 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:54:12.042 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:12.042 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.042 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.042 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:72556 9.13 9:cf932f14:::benchmark_data_smithi102_348563_object72555:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-09-13T14:54:12.042 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:51:46.403040+0000", 2024-09-13T14:54:12.042 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 145.55086711199999, 2024-09-13T14:54:12.042 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9505509889999999, 2024-09-13T14:54:12.042 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:54:12.043 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:12.043 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:54:12.043 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:54:12.043 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:54:12.043 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 72556 2024-09-13T14:54:12.043 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.043 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:54:12.043 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.043 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:54:12.043 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.403040+0000", 2024-09-13T14:54:12.043 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:12.043 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.043 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.044 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:54:12.044 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.403040+0000", 2024-09-13T14:54:12.044 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:12.044 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.044 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.044 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:54:12.044 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.403039+0000", 2024-09-13T14:54:12.044 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:54:12.044 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.044 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.044 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:54:12.044 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.403067+0000", 2024-09-13T14:54:12.044 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7789000000000001e-05 2024-09-13T14:54:12.044 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.045 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.045 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:54:12.045 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.403086+0000", 2024-09-13T14:54:12.045 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9351000000000002e-05 2024-09-13T14:54:12.045 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.045 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.045 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:54:12.045 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.403094+0000", 2024-09-13T14:54:12.045 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5050000000000001e-06 2024-09-13T14:54:12.045 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.045 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.045 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:12.045 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.403146+0000", 2024-09-13T14:54:12.046 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2153e-05 2024-09-13T14:54:12.046 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.046 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.046 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:54:12.046 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.403155+0000", 2024-09-13T14:54:12.046 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.3000000000000007e-06 2024-09-13T14:54:12.046 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.046 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.046 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:12.046 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.057087+0000", 2024-09-13T14:54:12.046 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.65393121300000001 2024-09-13T14:54:12.046 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.046 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.046 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:54:12.047 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.057095+0000", 2024-09-13T14:54:12.047 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8310000000000007e-06 2024-09-13T14:54:12.047 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.047 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.047 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:12.047 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.317544+0000", 2024-09-13T14:54:12.047 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.260448537 2024-09-13T14:54:12.047 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.047 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.047 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:54:12.047 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.317701+0000", 2024-09-13T14:54:12.047 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000157405 2024-09-13T14:54:12.047 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.047 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.048 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:54:12.048 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.317872+0000", 2024-09-13T14:54:12.048 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0001707 2024-09-13T14:54:12.048 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.048 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.048 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:54:12.048 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.320245+0000", 2024-09-13T14:54:12.048 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0023728360000000001 2024-09-13T14:54:12.048 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.048 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.048 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:12.048 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.351338+0000", 2024-09-13T14:54:12.049 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:12.050 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.031092813 2024-09-13T14:54:12.050 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.050 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.050 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:12.050 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.352986+0000", 2024-09-13T14:54:12.050 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0016485619999999999 2024-09-13T14:54:12.051 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.051 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.051 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:54:12.051 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.353017+0000", 2024-09-13T14:54:12.051 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0880999999999998e-05 2024-09-13T14:54:12.051 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.051 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.051 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:54:12.051 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.353591+0000", 2024-09-13T14:54:12.051 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00057408700000000003 2024-09-13T14:54:12.051 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:12.051 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:54:12.051 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:12.052 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.052 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.052 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:72572 9.13 9:ce9461e5:::benchmark_data_smithi102_348563_object72571:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-09-13T14:54:12.052 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:51:46.420582+0000", 2024-09-13T14:54:12.052 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 145.53332490299999, 2024-09-13T14:54:12.052 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9336864380000001, 2024-09-13T14:54:12.052 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:54:12.052 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:12.052 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:54:12.052 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:54:12.052 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:54:12.052 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 72572 2024-09-13T14:54:12.052 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.053 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:54:12.053 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.053 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:54:12.053 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.420582+0000", 2024-09-13T14:54:12.053 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:12.053 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.053 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.053 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:54:12.053 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.420582+0000", 2024-09-13T14:54:12.053 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:12.053 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.053 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.054 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:54:12.054 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.420581+0000", 2024-09-13T14:54:12.054 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:54:12.054 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.054 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.054 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:54:12.054 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.420633+0000", 2024-09-13T14:54:12.054 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2237999999999997e-05 2024-09-13T14:54:12.054 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.054 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.054 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:54:12.054 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.420644+0000", 2024-09-13T14:54:12.054 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1249e-05 2024-09-13T14:54:12.054 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.055 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.055 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:54:12.055 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.420650+0000", 2024-09-13T14:54:12.055 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4500000000000003e-06 2024-09-13T14:54:12.055 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.055 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.055 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:12.055 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.420696+0000", 2024-09-13T14:54:12.055 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5902999999999997e-05 2024-09-13T14:54:12.055 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.055 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.055 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:54:12.055 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.420704+0000", 2024-09-13T14:54:12.056 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.6559999999999998e-06 2024-09-13T14:54:12.056 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.056 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.056 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:12.056 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.057119+0000", 2024-09-13T14:54:12.056 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.63641496200000003 2024-09-13T14:54:12.056 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.056 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.056 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:54:12.056 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.057124+0000", 2024-09-13T14:54:12.056 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5920000000000002e-06 2024-09-13T14:54:12.056 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.056 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.057 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:12.057 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.318936+0000", 2024-09-13T14:54:12.057 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2618115219999999 2024-09-13T14:54:12.057 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.057 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.057 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:54:12.057 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.319072+0000", 2024-09-13T14:54:12.057 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013614700000000001 2024-09-13T14:54:12.057 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.057 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.057 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:54:12.057 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.319195+0000", 2024-09-13T14:54:12.057 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012294199999999999 2024-09-13T14:54:12.057 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.058 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.058 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:54:12.058 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.323229+0000", 2024-09-13T14:54:12.058 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0040348149999999998 2024-09-13T14:54:12.058 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.058 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.058 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:12.058 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.352299+0000", 2024-09-13T14:54:12.058 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.029069787 2024-09-13T14:54:12.058 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.058 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.058 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:12.058 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.353679+0000", 2024-09-13T14:54:12.059 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0013798129999999999 2024-09-13T14:54:12.059 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.059 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.059 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:54:12.059 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.353708+0000", 2024-09-13T14:54:12.059 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9454e-05 2024-09-13T14:54:12.059 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.059 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.059 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:54:12.059 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.354269+0000", 2024-09-13T14:54:12.059 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00056036699999999998 2024-09-13T14:54:12.059 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:12.059 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:54:12.059 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:12.060 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.060 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.060 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:72733 9.13 9:cc33e58b:::benchmark_data_smithi102_348563_object72732:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-09-13T14:54:12.060 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:51:46.588502+0000", 2024-09-13T14:54:12.060 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 145.365405224, 2024-09-13T14:54:12.060 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.766632268, 2024-09-13T14:54:12.060 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:54:12.060 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:12.060 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:54:12.060 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:54:12.060 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:54:12.060 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 72733 2024-09-13T14:54:12.060 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.061 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:54:12.061 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.061 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:54:12.061 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.588502+0000", 2024-09-13T14:54:12.061 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:12.061 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.061 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.061 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:54:12.061 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.588502+0000", 2024-09-13T14:54:12.061 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:12.061 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.061 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.062 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:54:12.062 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.588501+0000", 2024-09-13T14:54:12.062 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:54:12.062 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.062 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.062 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:54:12.062 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.588596+0000", 2024-09-13T14:54:12.062 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4958999999999997e-05 2024-09-13T14:54:12.062 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.062 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.062 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:54:12.062 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.588607+0000", 2024-09-13T14:54:12.062 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1015e-05 2024-09-13T14:54:12.062 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.063 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.063 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:54:12.063 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.588614+0000", 2024-09-13T14:54:12.063 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9750000000000001e-06 2024-09-13T14:54:12.063 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.063 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.063 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:12.063 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.588658+0000", 2024-09-13T14:54:12.063 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4016e-05 2024-09-13T14:54:12.063 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.063 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.063 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:54:12.063 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.588670+0000", 2024-09-13T14:54:12.063 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2163e-05 2024-09-13T14:54:12.064 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.064 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.064 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:12.064 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.057156+0000", 2024-09-13T14:54:12.064 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.46848595900000001 2024-09-13T14:54:12.064 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.064 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.064 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:54:12.064 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.057161+0000", 2024-09-13T14:54:12.064 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2650000000000004e-06 2024-09-13T14:54:12.064 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.064 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.065 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:12.065 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.325495+0000", 2024-09-13T14:54:12.065 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.268334128 2024-09-13T14:54:12.065 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.065 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.065 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:54:12.065 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.325653+0000", 2024-09-13T14:54:12.065 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015764899999999999 2024-09-13T14:54:12.065 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.065 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.065 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:54:12.065 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.325811+0000", 2024-09-13T14:54:12.065 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015845200000000001 2024-09-13T14:54:12.065 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.066 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.066 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:54:12.066 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.332343+0000", 2024-09-13T14:54:12.066 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0065321010000000002 2024-09-13T14:54:12.066 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.066 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.066 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:12.066 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.352949+0000", 2024-09-13T14:54:12.066 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.020605677999999999 2024-09-13T14:54:12.066 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.066 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.066 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:12.066 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.354358+0000", 2024-09-13T14:54:12.067 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001408921 2024-09-13T14:54:12.067 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.067 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.067 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:54:12.067 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.354425+0000", 2024-09-13T14:54:12.067 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7328000000000003e-05 2024-09-13T14:54:12.067 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.067 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.067 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:54:12.067 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.355134+0000", 2024-09-13T14:54:12.067 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00070911900000000003 2024-09-13T14:54:12.067 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:12.067 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:54:12.067 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:12.068 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.068 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.068 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:72760 9.13 9:cea85915:::benchmark_data_smithi102_348563_object72759:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-09-13T14:54:12.068 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:51:46.620001+0000", 2024-09-13T14:54:12.068 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 145.333905843, 2024-09-13T14:54:12.068 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.736355243, 2024-09-13T14:54:12.068 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:54:12.068 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:12.068 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:54:12.068 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:54:12.068 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:54:12.068 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 72760 2024-09-13T14:54:12.068 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.069 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:54:12.069 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.069 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:54:12.069 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.620001+0000", 2024-09-13T14:54:12.069 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:12.069 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.069 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.069 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:54:12.069 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.620001+0000", 2024-09-13T14:54:12.069 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:12.069 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.069 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.069 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:54:12.070 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.620000+0000", 2024-09-13T14:54:12.070 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:54:12.070 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.070 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.070 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:54:12.070 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.620100+0000", 2024-09-13T14:54:12.070 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.9945999999999995e-05 2024-09-13T14:54:12.070 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.070 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.070 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:54:12.071 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.620110+0000", 2024-09-13T14:54:12.071 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6800000000000005e-06 2024-09-13T14:54:12.071 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.071 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.071 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:54:12.071 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.620116+0000", 2024-09-13T14:54:12.071 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4289999999999998e-06 2024-09-13T14:54:12.071 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.071 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.071 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:12.071 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.620177+0000", 2024-09-13T14:54:12.071 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0121000000000003e-05 2024-09-13T14:54:12.071 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.072 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.072 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:54:12.072 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.620189+0000", 2024-09-13T14:54:12.072 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2911000000000001e-05 2024-09-13T14:54:12.072 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.072 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.072 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:12.072 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.057184+0000", 2024-09-13T14:54:12.072 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.43699416800000002 2024-09-13T14:54:12.072 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.072 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.072 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:54:12.072 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.057190+0000", 2024-09-13T14:54:12.072 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7179999999999999e-06 2024-09-13T14:54:12.073 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.073 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.073 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:12.073 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.327133+0000", 2024-09-13T14:54:12.073 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2699431109999999 2024-09-13T14:54:12.073 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.073 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.073 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:54:12.073 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.327311+0000", 2024-09-13T14:54:12.073 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00017787099999999999 2024-09-13T14:54:12.073 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.073 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.073 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:54:12.074 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.327509+0000", 2024-09-13T14:54:12.074 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000197345 2024-09-13T14:54:12.074 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.074 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.074 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:54:12.074 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.335945+0000", 2024-09-13T14:54:12.074 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0084361149999999992 2024-09-13T14:54:12.074 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.074 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.074 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:12.074 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.353645+0000", 2024-09-13T14:54:12.074 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017700583999999998 2024-09-13T14:54:12.074 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.074 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.075 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:12.075 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.355228+0000", 2024-09-13T14:54:12.075 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0015828120000000001 2024-09-13T14:54:12.075 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.075 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.075 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:54:12.075 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.355297+0000", 2024-09-13T14:54:12.075 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8486999999999997e-05 2024-09-13T14:54:12.075 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.075 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.075 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:54:12.075 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.356357+0000", 2024-09-13T14:54:12.075 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001060011 2024-09-13T14:54:12.075 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:12.076 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:54:12.076 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:12.076 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.076 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.076 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:113330 9.17 9:e8bce8ce:::benchmark_data_smithi102_348563_object113329:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e402)", 2024-09-13T14:54:12.076 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:52:26.477514+0000", 2024-09-13T14:54:12.076 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 105.476393393, 2024-09-13T14:54:12.076 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0168906770000001, 2024-09-13T14:54:12.076 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:54:12.076 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:12.076 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:54:12.076 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:54:12.076 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:54:12.077 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113330 2024-09-13T14:54:12.077 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.077 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:54:12.077 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.077 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:54:12.077 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.477514+0000", 2024-09-13T14:54:12.077 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:12.077 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.077 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.077 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:54:12.077 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.477514+0000", 2024-09-13T14:54:12.077 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:12.077 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.077 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.078 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:54:12.078 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.477512+0000", 2024-09-13T14:54:12.078 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:54:12.078 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.078 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.078 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:54:12.078 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.477630+0000", 2024-09-13T14:54:12.078 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011756000000000001 2024-09-13T14:54:12.078 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.078 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.078 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:54:12.078 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.477642+0000", 2024-09-13T14:54:12.078 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2492e-05 2024-09-13T14:54:12.079 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.079 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.079 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:54:12.079 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.477648+0000", 2024-09-13T14:54:12.079 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5230000000000003e-06 2024-09-13T14:54:12.079 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.079 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.079 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:12.079 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.477703+0000", 2024-09-13T14:54:12.079 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5500000000000001e-05 2024-09-13T14:54:12.079 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.079 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.079 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:54:12.080 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.477715+0000", 2024-09-13T14:54:12.080 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1083e-05 2024-09-13T14:54:12.080 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.080 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.080 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:12.080 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:27.337497+0000", 2024-09-13T14:54:12.080 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.85978237800000001 2024-09-13T14:54:12.080 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.080 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.080 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:54:12.080 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:27.337508+0000", 2024-09-13T14:54:12.080 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1004e-05 2024-09-13T14:54:12.080 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.081 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.081 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:12.081 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.485507+0000", 2024-09-13T14:54:12.081 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.147998759 2024-09-13T14:54:12.081 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.081 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.081 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:54:12.081 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.485653+0000", 2024-09-13T14:54:12.081 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014649300000000001 2024-09-13T14:54:12.081 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.081 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.081 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:54:12.081 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.485818+0000", 2024-09-13T14:54:12.081 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016457 2024-09-13T14:54:12.082 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.082 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.082 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:12.082 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.492578+0000", 2024-09-13T14:54:12.082 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0067599959999999999 2024-09-13T14:54:12.082 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.082 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.082 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:12.082 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.492640+0000", 2024-09-13T14:54:12.082 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2607999999999994e-05 2024-09-13T14:54:12.082 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.082 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.082 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:54:12.083 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.492652+0000", 2024-09-13T14:54:12.083 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2021000000000001e-05 2024-09-13T14:54:12.083 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.083 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.083 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:54:12.083 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.492691+0000", 2024-09-13T14:54:12.083 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8656999999999999e-05 2024-09-13T14:54:12.083 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.083 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.083 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:54:12.083 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.494405+0000", 2024-09-13T14:54:12.083 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0017135500000000001 2024-09-13T14:54:12.083 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:12.084 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:54:12.084 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:12.084 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.084 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.084 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:113341 9.17 9:ead480fa:::benchmark_data_smithi102_348563_object113340:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e402)", 2024-09-13T14:54:12.084 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:52:26.502483+0000", 2024-09-13T14:54:12.084 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 105.451424555, 2024-09-13T14:54:12.084 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0150210039999998, 2024-09-13T14:54:12.084 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:54:12.084 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:12.084 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:54:12.084 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:54:12.084 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:54:12.085 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113341 2024-09-13T14:54:12.085 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.085 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:54:12.085 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.085 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:54:12.085 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.502483+0000", 2024-09-13T14:54:12.085 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:12.085 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.085 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.085 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:54:12.085 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.502483+0000", 2024-09-13T14:54:12.085 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:12.086 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.086 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.086 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:54:12.086 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.502481+0000", 2024-09-13T14:54:12.086 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:54:12.086 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.086 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.086 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:54:12.086 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.502555+0000", 2024-09-13T14:54:12.086 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4099999999999999e-05 2024-09-13T14:54:12.086 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.086 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.086 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:54:12.086 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.502567+0000", 2024-09-13T14:54:12.087 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1698999999999999e-05 2024-09-13T14:54:12.087 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.087 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.087 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:54:12.087 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.502574+0000", 2024-09-13T14:54:12.087 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2570000000000001e-06 2024-09-13T14:54:12.087 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.087 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.087 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:12.087 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.502625+0000", 2024-09-13T14:54:12.087 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0603000000000003e-05 2024-09-13T14:54:12.087 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.087 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.087 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:54:12.088 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.502636+0000", 2024-09-13T14:54:12.088 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.082e-05 2024-09-13T14:54:12.088 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.088 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.088 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:12.088 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:27.337554+0000", 2024-09-13T14:54:12.088 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.83491843899999996 2024-09-13T14:54:12.088 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.088 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.088 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:54:12.088 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:27.337562+0000", 2024-09-13T14:54:12.088 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1960000000000008e-06 2024-09-13T14:54:12.088 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.089 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.089 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:12.089 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.489118+0000", 2024-09-13T14:54:12.089 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1515562509999999 2024-09-13T14:54:12.089 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.089 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.089 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:54:12.089 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.489230+0000", 2024-09-13T14:54:12.089 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011199100000000001 2024-09-13T14:54:12.089 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.089 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.089 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:54:12.089 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.489333+0000", 2024-09-13T14:54:12.090 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000102605 2024-09-13T14:54:12.090 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.090 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.090 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:54:12.090 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.498423+0000", 2024-09-13T14:54:12.090 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0090894700000000005 2024-09-13T14:54:12.090 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.090 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.090 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:12.090 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.498475+0000", 2024-09-13T14:54:12.090 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2296999999999998e-05 2024-09-13T14:54:12.090 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.090 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.091 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:12.091 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.515297+0000", 2024-09-13T14:54:12.091 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016822522999999999 2024-09-13T14:54:12.091 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.091 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.091 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:54:12.091 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.515353+0000", 2024-09-13T14:54:12.091 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5451000000000002e-05 2024-09-13T14:54:12.091 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.091 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.091 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:54:12.091 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.517504+0000", 2024-09-13T14:54:12.091 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0021509900000000002 2024-09-13T14:54:12.092 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:12.092 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:54:12.092 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:12.092 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.092 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.092 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:113388 9.17 9:ededae97:::benchmark_data_smithi102_348563_object113387:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e402)", 2024-09-13T14:54:12.092 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:52:26.574318+0000", 2024-09-13T14:54:12.092 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 105.379589072, 2024-09-13T14:54:12.092 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.944988825, 2024-09-13T14:54:12.092 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:54:12.092 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:12.092 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:54:12.092 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:54:12.092 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:54:12.093 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113388 2024-09-13T14:54:12.093 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.093 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:54:12.093 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.093 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:54:12.093 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.574318+0000", 2024-09-13T14:54:12.093 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:12.093 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.093 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.093 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:54:12.093 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.574318+0000", 2024-09-13T14:54:12.093 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:12.093 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.094 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.094 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:54:12.094 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.574316+0000", 2024-09-13T14:54:12.094 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:54:12.094 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.094 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.094 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:54:12.094 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.574361+0000", 2024-09-13T14:54:12.094 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5012000000000002e-05 2024-09-13T14:54:12.094 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.094 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.094 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:54:12.094 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.574373+0000", 2024-09-13T14:54:12.095 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1810999999999999e-05 2024-09-13T14:54:12.095 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.095 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.095 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:54:12.095 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.574379+0000", 2024-09-13T14:54:12.095 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.348e-06 2024-09-13T14:54:12.095 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.095 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.095 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:12.095 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.574431+0000", 2024-09-13T14:54:12.095 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1205000000000002e-05 2024-09-13T14:54:12.095 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.095 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.095 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:54:12.096 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.574443+0000", 2024-09-13T14:54:12.096 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2842e-05 2024-09-13T14:54:12.096 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.096 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.096 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:12.096 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:27.337587+0000", 2024-09-13T14:54:12.096 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.76314326399999999 2024-09-13T14:54:12.096 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.096 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.096 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:54:12.096 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:27.337595+0000", 2024-09-13T14:54:12.096 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.4600000000000003e-06 2024-09-13T14:54:12.097 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.097 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.097 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:12.097 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.490785+0000", 2024-09-13T14:54:12.097 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1531900020000001 2024-09-13T14:54:12.097 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.097 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.097 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:54:12.097 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.490897+0000", 2024-09-13T14:54:12.097 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011162699999999999 2024-09-13T14:54:12.097 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.097 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.097 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:54:12.097 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.491023+0000", 2024-09-13T14:54:12.098 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012667900000000001 2024-09-13T14:54:12.098 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.098 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.098 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:54:12.098 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.498430+0000", 2024-09-13T14:54:12.098 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00740619 2024-09-13T14:54:12.098 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.098 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.098 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:12.098 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.498510+0000", 2024-09-13T14:54:12.098 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0636999999999994e-05 2024-09-13T14:54:12.098 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.098 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.099 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:12.099 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.517640+0000", 2024-09-13T14:54:12.099 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.019130004999999999 2024-09-13T14:54:12.099 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.099 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.099 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:54:12.099 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.517695+0000", 2024-09-13T14:54:12.099 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4718e-05 2024-09-13T14:54:12.099 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.099 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.099 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:54:12.099 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.519307+0000", 2024-09-13T14:54:12.099 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0016120279999999999 2024-09-13T14:54:12.100 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:12.100 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:54:12.100 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:12.100 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.100 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.100 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:113411 9.17 9:e864c0f1:::benchmark_data_smithi102_348563_object113410:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e402)", 2024-09-13T14:54:12.100 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:52:26.602093+0000", 2024-09-13T14:54:12.100 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 105.351814003, 2024-09-13T14:54:12.100 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.921966007, 2024-09-13T14:54:12.100 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:54:12.100 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:12.100 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:54:12.100 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:54:12.100 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:54:12.101 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113411 2024-09-13T14:54:12.101 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.101 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:54:12.101 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.101 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:54:12.101 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.602093+0000", 2024-09-13T14:54:12.101 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:12.101 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.101 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.101 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:54:12.101 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.602093+0000", 2024-09-13T14:54:12.101 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:12.101 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.102 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.102 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:54:12.102 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.602092+0000", 2024-09-13T14:54:12.102 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:54:12.102 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.102 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.102 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:54:12.102 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.602155+0000", 2024-09-13T14:54:12.102 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3108999999999995e-05 2024-09-13T14:54:12.102 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.102 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.102 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:54:12.102 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.602169+0000", 2024-09-13T14:54:12.102 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3198e-05 2024-09-13T14:54:12.103 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.103 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.103 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:54:12.103 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.602176+0000", 2024-09-13T14:54:12.103 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.238e-06 2024-09-13T14:54:12.103 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.103 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.103 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:12.103 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.602227+0000", 2024-09-13T14:54:12.103 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1411e-05 2024-09-13T14:54:12.103 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.103 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.103 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:54:12.103 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.602236+0000", 2024-09-13T14:54:12.104 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1579999999999994e-06 2024-09-13T14:54:12.104 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.104 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.104 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:12.104 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:27.337620+0000", 2024-09-13T14:54:12.104 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.73538361699999999 2024-09-13T14:54:12.104 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.104 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.104 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:54:12.104 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:27.337628+0000", 2024-09-13T14:54:12.104 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8730000000000007e-06 2024-09-13T14:54:12.104 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.104 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.104 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:12.105 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.494470+0000", 2024-09-13T14:54:12.105 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.15684242 2024-09-13T14:54:12.105 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.105 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.105 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:54:12.105 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.494630+0000", 2024-09-13T14:54:12.105 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015944 2024-09-13T14:54:12.105 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.105 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.105 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:54:12.105 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.494786+0000", 2024-09-13T14:54:12.105 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000155909 2024-09-13T14:54:12.105 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.105 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.106 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:54:12.106 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.498518+0000", 2024-09-13T14:54:12.106 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0037322029999999999 2024-09-13T14:54:12.106 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.106 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.106 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:12.106 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.509421+0000", 2024-09-13T14:54:12.106 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010903256 2024-09-13T14:54:12.106 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.106 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.106 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:12.106 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.521680+0000", 2024-09-13T14:54:12.106 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012258929 2024-09-13T14:54:12.106 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.107 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.107 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:54:12.107 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.521736+0000", 2024-09-13T14:54:12.107 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5813999999999998e-05 2024-09-13T14:54:12.107 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.107 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.107 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:54:12.107 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.524059+0000", 2024-09-13T14:54:12.107 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0023235220000000002 2024-09-13T14:54:12.107 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:12.107 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:54:12.107 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:12.107 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.108 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.108 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:113426 9.17 9:ee9b3220:::benchmark_data_smithi102_348563_object113425:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e402)", 2024-09-13T14:54:12.108 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:52:26.616755+0000", 2024-09-13T14:54:12.108 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 105.33715249399999, 2024-09-13T14:54:12.108 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.909422927, 2024-09-13T14:54:12.108 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:54:12.108 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:12.108 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:54:12.108 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:54:12.108 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:54:12.108 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113426 2024-09-13T14:54:12.108 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.108 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:54:12.108 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.109 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:54:12.109 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.616755+0000", 2024-09-13T14:54:12.109 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:12.109 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.109 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.109 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:54:12.109 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.616755+0000", 2024-09-13T14:54:12.109 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:12.109 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.109 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.109 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:54:12.109 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.616754+0000", 2024-09-13T14:54:12.109 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:54:12.110 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.110 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.110 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:54:12.110 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.616878+0000", 2024-09-13T14:54:12.110 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000124043 2024-09-13T14:54:12.110 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.110 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.110 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:54:12.110 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.616888+0000", 2024-09-13T14:54:12.110 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0301e-05 2024-09-13T14:54:12.110 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.110 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.110 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:54:12.110 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.616895+0000", 2024-09-13T14:54:12.111 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6649999999999997e-06 2024-09-13T14:54:12.111 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.111 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.111 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:12.111 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.617105+0000", 2024-09-13T14:54:12.111 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00021071100000000001 2024-09-13T14:54:12.111 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.111 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.111 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:54:12.111 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.617115+0000", 2024-09-13T14:54:12.111 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.8849999999999994e-06 2024-09-13T14:54:12.111 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.111 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.111 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:12.112 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:27.337655+0000", 2024-09-13T14:54:12.112 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.72054000100000004 2024-09-13T14:54:12.112 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.112 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.112 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:54:12.112 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:27.337663+0000", 2024-09-13T14:54:12.112 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7230000000000004e-06 2024-09-13T14:54:12.112 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.112 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.112 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:12.112 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.496533+0000", 2024-09-13T14:54:12.112 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.158870515 2024-09-13T14:54:12.112 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.113 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.113 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:54:12.113 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.496848+0000", 2024-09-13T14:54:12.113 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00031459100000000002 2024-09-13T14:54:12.113 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.113 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.113 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:54:12.113 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.496952+0000", 2024-09-13T14:54:12.113 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000104147 2024-09-13T14:54:12.113 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.114 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.114 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:54:12.114 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.503972+0000", 2024-09-13T14:54:12.114 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0070199650000000004 2024-09-13T14:54:12.114 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.114 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.114 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:12.114 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.517572+0000", 2024-09-13T14:54:12.114 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013599546000000001 2024-09-13T14:54:12.114 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.114 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.114 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:12.114 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.524126+0000", 2024-09-13T14:54:12.115 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0065545020000000002 2024-09-13T14:54:12.115 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.115 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.115 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:54:12.115 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.524182+0000", 2024-09-13T14:54:12.115 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5952999999999998e-05 2024-09-13T14:54:12.115 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.115 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.115 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:54:12.115 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.526178+0000", 2024-09-13T14:54:12.115 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001995564 2024-09-13T14:54:12.115 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:12.115 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:54:12.116 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:12.116 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.116 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.116 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:113445 9.17 9:eec04f73:::benchmark_data_smithi102_348563_object113444:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e402)", 2024-09-13T14:54:12.116 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:52:26.657492+0000", 2024-09-13T14:54:12.116 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 105.296415626, 2024-09-13T14:54:12.116 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8791318000000001, 2024-09-13T14:54:12.116 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:54:12.116 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:12.116 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:54:12.116 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:54:12.116 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:54:12.116 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113445 2024-09-13T14:54:12.116 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.117 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:54:12.117 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.117 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:54:12.117 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.657492+0000", 2024-09-13T14:54:12.117 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:12.117 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.117 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.117 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:54:12.117 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.657492+0000", 2024-09-13T14:54:12.117 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:12.117 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.117 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.117 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:54:12.117 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.657491+0000", 2024-09-13T14:54:12.118 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:54:12.118 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.118 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.118 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:54:12.118 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.657520+0000", 2024-09-13T14:54:12.118 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9128999999999998e-05 2024-09-13T14:54:12.118 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.118 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.118 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:54:12.118 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.657532+0000", 2024-09-13T14:54:12.118 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1413999999999999e-05 2024-09-13T14:54:12.118 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.118 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.118 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:54:12.119 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.657538+0000", 2024-09-13T14:54:12.119 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1789999999999996e-06 2024-09-13T14:54:12.119 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.119 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.119 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:12.119 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.657589+0000", 2024-09-13T14:54:12.119 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0759e-05 2024-09-13T14:54:12.119 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.119 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.119 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:54:12.119 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.657597+0000", 2024-09-13T14:54:12.119 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2449999999999998e-06 2024-09-13T14:54:12.119 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.119 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.120 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:12.120 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:27.337687+0000", 2024-09-13T14:54:12.120 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.68009004100000003 2024-09-13T14:54:12.120 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.120 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.120 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:54:12.120 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:27.337703+0000", 2024-09-13T14:54:12.120 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6435000000000001e-05 2024-09-13T14:54:12.120 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.120 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.120 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:12.120 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.498540+0000", 2024-09-13T14:54:12.120 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.160837068 2024-09-13T14:54:12.121 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.121 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.121 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:54:12.121 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.498656+0000", 2024-09-13T14:54:12.121 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000115621 2024-09-13T14:54:12.121 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.121 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.121 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:54:12.121 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.498755+0000", 2024-09-13T14:54:12.121 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.9049999999999995e-05 2024-09-13T14:54:12.121 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.121 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.121 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:54:12.121 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.503980+0000", 2024-09-13T14:54:12.122 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0052250810000000003 2024-09-13T14:54:12.122 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.122 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.122 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:12.122 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.526576+0000", 2024-09-13T14:54:12.122 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.022596247 2024-09-13T14:54:12.122 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.122 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.122 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:12.122 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.533884+0000", 2024-09-13T14:54:12.122 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0073076959999999998 2024-09-13T14:54:12.122 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.122 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.122 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:54:12.123 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.533947+0000", 2024-09-13T14:54:12.123 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3268000000000005e-05 2024-09-13T14:54:12.123 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.123 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.123 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:54:12.123 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.536623+0000", 2024-09-13T14:54:12.123 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0026760989999999999 2024-09-13T14:54:12.123 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:12.123 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:54:12.123 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:12.123 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.123 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.123 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:113451 9.17 9:eeade24e:::benchmark_data_smithi102_348563_object113450:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e402)", 2024-09-13T14:54:12.123 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:52:26.661456+0000", 2024-09-13T14:54:12.124 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 105.29245173299999, 2024-09-13T14:54:12.124 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.877630669, 2024-09-13T14:54:12.124 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:54:12.124 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:12.124 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:54:12.124 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:54:12.124 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:54:12.124 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113451 2024-09-13T14:54:12.124 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.124 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:54:12.124 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.124 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:54:12.124 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.661456+0000", 2024-09-13T14:54:12.125 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:12.125 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.125 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.125 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:54:12.125 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.661456+0000", 2024-09-13T14:54:12.125 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:12.125 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.125 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.125 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:54:12.125 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.661455+0000", 2024-09-13T14:54:12.125 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:54:12.125 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.125 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.126 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:54:12.126 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.661508+0000", 2024-09-13T14:54:12.126 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3758000000000003e-05 2024-09-13T14:54:12.126 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.126 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.126 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:54:12.126 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.661517+0000", 2024-09-13T14:54:12.126 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.02e-06 2024-09-13T14:54:12.126 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.126 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.126 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:54:12.126 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.661525+0000", 2024-09-13T14:54:12.126 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7449999999999995e-06 2024-09-13T14:54:12.126 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.127 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.127 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:12.127 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.661595+0000", 2024-09-13T14:54:12.127 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9338999999999995e-05 2024-09-13T14:54:12.127 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.127 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.127 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:54:12.127 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.661668+0000", 2024-09-13T14:54:12.127 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3152999999999996e-05 2024-09-13T14:54:12.127 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.127 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.127 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:12.127 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:27.337741+0000", 2024-09-13T14:54:12.127 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.676073692 2024-09-13T14:54:12.128 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.128 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.128 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:54:12.128 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:27.337750+0000", 2024-09-13T14:54:12.128 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.054e-06 2024-09-13T14:54:12.128 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.128 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.128 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:12.128 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.500434+0000", 2024-09-13T14:54:12.128 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.162683624 2024-09-13T14:54:12.128 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.128 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.128 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:54:12.129 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.500554+0000", 2024-09-13T14:54:12.129 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012008 2024-09-13T14:54:12.129 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.129 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.129 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:54:12.129 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.500668+0000", 2024-09-13T14:54:12.129 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000114251 2024-09-13T14:54:12.129 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.129 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.129 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:54:12.129 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.509372+0000", 2024-09-13T14:54:12.129 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0087033370000000002 2024-09-13T14:54:12.129 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.129 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.130 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:12.130 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.526625+0000", 2024-09-13T14:54:12.130 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017252989 2024-09-13T14:54:12.130 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.130 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.130 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:12.130 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.536677+0000", 2024-09-13T14:54:12.130 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010052613 2024-09-13T14:54:12.130 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.130 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.130 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:54:12.130 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.536731+0000", 2024-09-13T14:54:12.130 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3430000000000002e-05 2024-09-13T14:54:12.130 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.131 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.131 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:54:12.131 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.539086+0000", 2024-09-13T14:54:12.131 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0023555130000000001 2024-09-13T14:54:12.131 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:12.131 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:54:12.131 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:12.131 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.131 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.131 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:113463 9.17 9:e9efb8ae:::benchmark_data_smithi102_348563_object113462:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e402)", 2024-09-13T14:54:12.131 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:52:26.677091+0000", 2024-09-13T14:54:12.131 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 105.27681656199999, 2024-09-13T14:54:12.131 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.864105189, 2024-09-13T14:54:12.131 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:54:12.132 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:12.132 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:54:12.132 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:54:12.132 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:54:12.132 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113463 2024-09-13T14:54:12.132 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.132 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:54:12.132 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.132 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:54:12.132 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.677091+0000", 2024-09-13T14:54:12.132 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:12.132 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.132 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.132 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:54:12.133 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.677091+0000", 2024-09-13T14:54:12.133 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:54:12.133 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.133 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.133 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:54:12.133 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.677089+0000", 2024-09-13T14:54:12.133 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:54:12.133 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.133 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.133 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:54:12.133 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.677163+0000", 2024-09-13T14:54:12.133 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3646000000000006e-05 2024-09-13T14:54:12.133 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.134 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.134 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:54:12.134 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.677174+0000", 2024-09-13T14:54:12.134 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0671e-05 2024-09-13T14:54:12.134 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.134 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.134 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:54:12.134 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.677181+0000", 2024-09-13T14:54:12.134 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1319999999999996e-06 2024-09-13T14:54:12.134 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.134 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.134 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:12.134 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.677585+0000", 2024-09-13T14:54:12.134 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00040452100000000003 2024-09-13T14:54:12.135 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.135 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.135 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:54:12.136 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.677595+0000", 2024-09-13T14:54:12.136 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0202e-05 2024-09-13T14:54:12.136 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.136 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.136 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:12.136 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:27.337777+0000", 2024-09-13T14:54:12.136 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.66018160299999995 2024-09-13T14:54:12.136 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.136 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.136 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:54:12.136 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:27.337785+0000", 2024-09-13T14:54:12.136 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6790000000000005e-06 2024-09-13T14:54:12.136 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.136 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.137 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:54:12.137 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.502270+0000", 2024-09-13T14:54:12.137 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1644854650000001 2024-09-13T14:54:12.137 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.137 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.137 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:54:12.137 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.502417+0000", 2024-09-13T14:54:12.137 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000147229 2024-09-13T14:54:12.137 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.137 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.137 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:54:12.137 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.502598+0000", 2024-09-13T14:54:12.137 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00018033400000000001 2024-09-13T14:54:12.138 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.138 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.138 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:54:12.138 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.511383+0000", 2024-09-13T14:54:12.138 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0087850340000000006 2024-09-13T14:54:12.138 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.138 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.138 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:12.138 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.526671+0000", 2024-09-13T14:54:12.138 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.015287953999999999 2024-09-13T14:54:12.138 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.138 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.138 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:12.138 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.539136+0000", 2024-09-13T14:54:12.139 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012465072000000001 2024-09-13T14:54:12.139 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.139 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.139 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:54:12.139 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.539189+0000", 2024-09-13T14:54:12.139 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3069000000000003e-05 2024-09-13T14:54:12.139 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:54:12.139 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:54:12.139 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:54:12.139 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.541196+0000", 2024-09-13T14:54:12.139 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0020070930000000002 2024-09-13T14:54:12.139 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:12.139 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:54:12.139 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:12.140 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:54:12.140 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:54:12.140 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:54:12.156 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:12.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:12.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:12.257 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:12.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:12.294 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:12.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:12.395 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:12.417 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:12.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:12.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:12.519 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:12.554 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:12.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:12.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:12.656 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:12.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:12.686 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:12.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:12.787 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:12.813 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:12.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:12.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:12.915 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:12.939 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:12.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:12.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:13.041 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:13.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:13.078 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:13.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:13.179 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:13.212 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:13.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:13.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:13.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:12 smithi195 ceph-mon[104262]: pgmap v1438: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 117 active+clean; 1.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 995 op/s; 2282/84603 objects degraded (2.697%); 123/84603 objects misplaced (0.145%); 7.4 MiB/s, 117 objects/s recovering 2024-09-13T14:54:13.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:12 smithi195 ceph-mon[104262]: osdmap e423: 8 total, 8 up, 8 in 2024-09-13T14:54:13.314 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:13.354 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:13.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:13.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:13.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:12 smithi102 ceph-mon[110183]: pgmap v1438: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 117 active+clean; 1.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 995 op/s; 2282/84603 objects degraded (2.697%); 123/84603 objects misplaced (0.145%); 7.4 MiB/s, 117 objects/s recovering 2024-09-13T14:54:13.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:12 smithi102 ceph-mon[110183]: osdmap e423: 8 total, 8 up, 8 in 2024-09-13T14:54:13.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:12 smithi102 ceph-mon[115269]: pgmap v1438: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 117 active+clean; 1.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 995 op/s; 2282/84603 objects degraded (2.697%); 123/84603 objects misplaced (0.145%); 7.4 MiB/s, 117 objects/s recovering 2024-09-13T14:54:13.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:12 smithi102 ceph-mon[115269]: osdmap e423: 8 total, 8 up, 8 in 2024-09-13T14:54:13.457 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:13.499 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T14:54:13.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:13.509 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:13.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:13.610 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:13.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:13.655 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:13.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:13.756 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:13.781 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:13.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:13.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:13.833 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:54:13.884 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:13.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:13.923 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:13.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:14.023 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:14.061 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:14.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:14.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:14.163 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:14.186 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:14.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:14.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:14.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:13 smithi195 ceph-mon[104262]: osdmap e424: 8 total, 8 up, 8 in 2024-09-13T14:54:14.289 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:14.324 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:14.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:14.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:14.333 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:13 smithi102 ceph-mon[110183]: osdmap e424: 8 total, 8 up, 8 in 2024-09-13T14:54:14.333 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:13 smithi102 ceph-mon[115269]: osdmap e424: 8 total, 8 up, 8 in 2024-09-13T14:54:14.426 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:14.453 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:14.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:14.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:14.554 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:14.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:14.595 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:14.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:14.697 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:14.738 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:14.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:14.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:14.840 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:14.862 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:14.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:14.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:14.964 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:15.000 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:15.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:15.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:15.102 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:15.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:15.134 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:15.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:15.189 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:14 smithi195 ceph-mon[104262]: pgmap v1441: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 117 active+clean; 1.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.17k op/s; 2282/81810 objects degraded (2.789%); 123/81810 objects misplaced (0.150%); 7.0 MiB/s, 111 objects/s recovering 2024-09-13T14:54:15.189 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:14 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 2282/81810 objects degraded (2.789%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:54:15.189 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:14 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1154428626' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:54:15.235 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:15.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:15.275 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:15.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:15.358 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:14 smithi102 ceph-mon[110183]: pgmap v1441: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 117 active+clean; 1.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.17k op/s; 2282/81810 objects degraded (2.789%); 123/81810 objects misplaced (0.150%); 7.0 MiB/s, 111 objects/s recovering 2024-09-13T14:54:15.358 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:14 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 2282/81810 objects degraded (2.789%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:54:15.358 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:14 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1154428626' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:54:15.358 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:14 smithi102 ceph-mon[115269]: pgmap v1441: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 117 active+clean; 1.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.17k op/s; 2282/81810 objects degraded (2.789%); 123/81810 objects misplaced (0.150%); 7.0 MiB/s, 111 objects/s recovering 2024-09-13T14:54:15.358 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:14 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 2282/81810 objects degraded (2.789%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:54:15.359 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:14 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1154428626' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:54:15.359 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:54:15.359 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:54:15.359 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:54:15.359 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:54:15.359 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:54:15.359 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:54:15.359 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:54:15.359 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:54:15.360 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:54:15.360 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:54:15.360 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:54:15.360 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:54:15.360 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:54:15.360 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:54:15.360 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:54:15.360 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:54:15.376 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:15.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:15.414 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:15.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:15.515 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:15.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:15.550 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:15.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:15.653 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:15.675 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:15.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:15.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:15.777 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:15.813 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:15.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:15.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:15.893 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:54:15.893 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:54:15.916 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:15.960 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:15.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:15.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:16.064 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:16.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:16.118 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:16.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:16.219 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:16.248 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:16.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:16.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:16.350 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:16.385 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:16.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:16.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:16.487 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:16.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:16.510 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:16.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:16.612 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:16.643 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:16.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:16.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:16.660 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T14:54:16.744 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:16.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:16.778 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:16.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:16.879 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:16.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:16.912 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:16.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:16.918 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:54:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:54:16] "GET /metrics HTTP/1.1" 200 36530 "" "Prometheus/2.51.0" 2024-09-13T14:54:16.996 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:54:17.013 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:17.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:17.071 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:17.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:17.172 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:17.206 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:17.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:17.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:17.293 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:16 smithi195 ceph-mon[104262]: pgmap v1442: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 1.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 801 op/s; 2002/79920 objects degraded (2.505%); 20/79920 objects misplaced (0.025%); 5.5 MiB/s, 122 objects/s recovering 2024-09-13T14:54:17.307 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:17.341 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:17.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:17.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:17.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:16 smithi102 ceph-mon[110183]: pgmap v1442: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 1.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 801 op/s; 2002/79920 objects degraded (2.505%); 20/79920 objects misplaced (0.025%); 5.5 MiB/s, 122 objects/s recovering 2024-09-13T14:54:17.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:17 smithi102 ceph-mon[115269]: pgmap v1442: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 117 active+clean; 1.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 801 op/s; 2002/79920 objects degraded (2.505%); 20/79920 objects misplaced (0.025%); 5.5 MiB/s, 122 objects/s recovering 2024-09-13T14:54:17.443 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:17.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:17.484 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:17.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:17.585 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:17.617 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:17.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:17.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:17.719 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:17.757 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:17.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:17.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:17.859 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:17.898 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:17.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:17.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:18.000 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:18.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:18.036 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:18.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:18.137 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:18.178 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:18.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:18.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:18.280 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:18.301 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:18.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:18.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:18.403 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:18.443 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:18.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:18.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:18.545 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:18.585 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:18.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:18.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:18.687 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:18.730 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:18.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:18.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:18.832 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:18.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:18.863 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:18.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:18.964 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:19.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:19.001 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:19.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:19.102 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:19.128 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:19.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:19.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:19.230 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:19.231 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:19 smithi195 ceph-mon[104262]: pgmap v1443: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 2002/70293 objects degraded (2.848%); 3.3 MiB/s, 87 objects/s recovering 2024-09-13T14:54:19.231 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:19 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/25937507' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:54:19.267 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:19.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:19.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:19.279 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:54:19.279 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:54:19.369 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:19.395 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:19.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:19.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:19.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:19 smithi102 ceph-mon[110183]: pgmap v1443: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 2002/70293 objects degraded (2.848%); 3.3 MiB/s, 87 objects/s recovering 2024-09-13T14:54:19.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:19 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/25937507' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:54:19.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:19 smithi102 ceph-mon[115269]: pgmap v1443: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 2002/70293 objects degraded (2.848%); 3.3 MiB/s, 87 objects/s recovering 2024-09-13T14:54:19.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:19 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/25937507' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:54:19.497 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:19.525 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is unset 2024-09-13T14:54:19.535 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:19.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:19.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:19.638 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:19.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:19.672 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:19.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:19.773 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:19.803 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:19.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:19.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:19.905 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:19.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:19.928 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:19.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:20.029 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:20.034 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T14:54:20.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:20.064 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:20.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:20.165 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:20.213 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:20.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:20.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:20.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:20 smithi195 ceph-mon[104262]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:54:20.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:20 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/25937507' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:54:20.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:20 smithi195 ceph-mon[104262]: osdmap e425: 8 total, 8 up, 8 in 2024-09-13T14:54:20.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:20 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 2002/70293 objects degraded (2.848%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:54:20.315 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:20.344 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:20.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:20.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:20.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:20 smithi102 ceph-mon[110183]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:54:20.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:20 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/25937507' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:54:20.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:20 smithi102 ceph-mon[110183]: osdmap e425: 8 total, 8 up, 8 in 2024-09-13T14:54:20.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:20 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 2002/70293 objects degraded (2.848%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:54:20.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:20 smithi102 ceph-mon[115269]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:54:20.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:20 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/25937507' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:54:20.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:20 smithi102 ceph-mon[115269]: osdmap e425: 8 total, 8 up, 8 in 2024-09-13T14:54:20.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:20 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 2002/70293 objects degraded (2.848%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:54:20.447 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:20.456 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:54:20.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:20.485 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:20.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:20.591 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:20.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:20.629 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:20.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:20.730 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:20.757 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:20.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:20.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:20.859 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:20.891 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:20.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:20.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:20.993 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:21.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:21.022 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:21.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:21.124 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:21.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:21.169 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:21.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:21.270 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:21.302 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:21.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:21.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:21.405 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:21.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:21 smithi102 ceph-mon[115269]: pgmap v1445: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.3 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 815 op/s; 1624/65574 objects degraded (2.477%); 930 KiB/s, 99 objects/s recovering 2024-09-13T14:54:21.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:21 smithi102 ceph-mon[110183]: pgmap v1445: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.3 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 815 op/s; 1624/65574 objects degraded (2.477%); 930 KiB/s, 99 objects/s recovering 2024-09-13T14:54:21.440 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:21.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:21.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:21.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:21 smithi195 ceph-mon[104262]: pgmap v1445: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.3 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 815 op/s; 1624/65574 objects degraded (2.477%); 930 KiB/s, 99 objects/s recovering 2024-09-13T14:54:21.545 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:21.568 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:21.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:21.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:21.671 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:21.714 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:21.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:21.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:21.816 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:21.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:21.854 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:21.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:21.955 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:21.984 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:21.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:21.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:22.086 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:22.120 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:22.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:22.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:22.221 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:22.243 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:22.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:22.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:22.345 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:22.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:22.390 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:22.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:22.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:22 smithi102 ceph-mon[115269]: 9.10 scrub starts 2024-09-13T14:54:22.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:22 smithi102 ceph-mon[115269]: 9.10 scrub ok 2024-09-13T14:54:22.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:22 smithi102 ceph-mon[115269]: 9.7 scrub starts 2024-09-13T14:54:22.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:22 smithi102 ceph-mon[115269]: 9.7 scrub ok 2024-09-13T14:54:22.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:22 smithi102 ceph-mon[110183]: 9.10 scrub starts 2024-09-13T14:54:22.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:22 smithi102 ceph-mon[110183]: 9.10 scrub ok 2024-09-13T14:54:22.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:22 smithi102 ceph-mon[110183]: 9.7 scrub starts 2024-09-13T14:54:22.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:22 smithi102 ceph-mon[110183]: 9.7 scrub ok 2024-09-13T14:54:22.491 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:22.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:22 smithi195 ceph-mon[104262]: 9.10 scrub starts 2024-09-13T14:54:22.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:22 smithi195 ceph-mon[104262]: 9.10 scrub ok 2024-09-13T14:54:22.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:22 smithi195 ceph-mon[104262]: 9.7 scrub starts 2024-09-13T14:54:22.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:22 smithi195 ceph-mon[104262]: 9.7 scrub ok 2024-09-13T14:54:22.529 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:22.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:22.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:22.630 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:22.668 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:22.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:22.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:22.770 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:22.805 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:22.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:22.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:22.907 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:22.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:22.958 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:22.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:23.059 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:23.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:23.091 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:23.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:23.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:23 smithi102 ceph-mon[115269]: pgmap v1446: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1624/58992 objects degraded (2.753%) 2024-09-13T14:54:23.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:23 smithi102 ceph-mon[115269]: 9.16 scrub starts 2024-09-13T14:54:23.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:23 smithi102 ceph-mon[115269]: 9.16 scrub ok 2024-09-13T14:54:23.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:23 smithi102 ceph-mon[110183]: pgmap v1446: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1624/58992 objects degraded (2.753%) 2024-09-13T14:54:23.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:23 smithi102 ceph-mon[110183]: 9.16 scrub starts 2024-09-13T14:54:23.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:23 smithi102 ceph-mon[110183]: 9.16 scrub ok 2024-09-13T14:54:23.192 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:23.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:23.228 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:23.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:23.228 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:23 smithi195 ceph-mon[104262]: pgmap v1446: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1624/58992 objects degraded (2.753%) 2024-09-13T14:54:23.229 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:23 smithi195 ceph-mon[104262]: 9.16 scrub starts 2024-09-13T14:54:23.229 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:23 smithi195 ceph-mon[104262]: 9.16 scrub ok 2024-09-13T14:54:23.268 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:54:23.268 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:54:23.329 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:23.365 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:23.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:23.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:23.467 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:23.507 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:23.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:23.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:23.527 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:23.609 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:23.646 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:23.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:23.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:23.748 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:23.784 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:23.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:23.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:23.886 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:23.924 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:23.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:23.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:24.026 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:24.059 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:24.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:24.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:24.079 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T14:54:24.145 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:24 smithi102 ceph-mon[110183]: 9.13 scrub starts 2024-09-13T14:54:24.145 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:24 smithi102 ceph-mon[110183]: 9.13 scrub ok 2024-09-13T14:54:24.145 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:24 smithi102 ceph-mon[115269]: 9.13 scrub starts 2024-09-13T14:54:24.145 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:24 smithi102 ceph-mon[115269]: 9.13 scrub ok 2024-09-13T14:54:24.161 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:24.188 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:24.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:24.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:24.290 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:24.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:24.333 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:24.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:24.434 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:24.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:24.472 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:24.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:24.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:24 smithi195 ceph-mon[104262]: 9.13 scrub starts 2024-09-13T14:54:24.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:24 smithi195 ceph-mon[104262]: 9.13 scrub ok 2024-09-13T14:54:24.573 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:24.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:24.608 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:24.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:24.709 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:24.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:24.744 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:24.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:24.845 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:24.877 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:24.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:24.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:24.902 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:54:24.980 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:25.022 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:25.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:25.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:25.124 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:25.148 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:25.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:25.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:25.250 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:25.278 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:25.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:25.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:25.319 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:25 smithi102 ceph-mon[115269]: pgmap v1447: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.1 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1624/52308 objects degraded (3.105%) 2024-09-13T14:54:25.319 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:25 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 1624/52308 objects degraded (3.105%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:54:25.320 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:25 smithi102 ceph-mon[110183]: pgmap v1447: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.1 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1624/52308 objects degraded (3.105%) 2024-09-13T14:54:25.320 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:25 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 1624/52308 objects degraded (3.105%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:54:25.381 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:25.419 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:25.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:25.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:25.430 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:25 smithi195 ceph-mon[104262]: pgmap v1447: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.1 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1624/52308 objects degraded (3.105%) 2024-09-13T14:54:25.430 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:25 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 1624/52308 objects degraded (3.105%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:54:25.521 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:25.554 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:54:25.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:25.560 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:25.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:25.661 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:25.726 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:25.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:25.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:25.828 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:25.852 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:25.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:25.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:25.954 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:25.990 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:25.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:25.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:26.092 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:26.129 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:26.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:26.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:26.231 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:26.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:26.267 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:26.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:26.368 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:26.409 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:26.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:26.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:26.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:26 smithi102 ceph-mon[110183]: 9.17 scrub starts 2024-09-13T14:54:26.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:26 smithi102 ceph-mon[110183]: 9.17 scrub ok 2024-09-13T14:54:26.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:54:26.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:54:26.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:54:26.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:26 smithi102 ceph-mon[115269]: 9.17 scrub starts 2024-09-13T14:54:26.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:26 smithi102 ceph-mon[115269]: 9.17 scrub ok 2024-09-13T14:54:26.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:54:26.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:54:26.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:54:26.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:26 smithi195 ceph-mon[104262]: 9.17 scrub starts 2024-09-13T14:54:26.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:26 smithi195 ceph-mon[104262]: 9.17 scrub ok 2024-09-13T14:54:26.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:54:26.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:54:26.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:54:26.512 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:26.537 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:26.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:26.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:26.600 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:26.618 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:26.640 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:26.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:26.699 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:26.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:26.713 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:54:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:54:26] "GET /metrics HTTP/1.1" 200 36513 "" "Prometheus/2.51.0" 2024-09-13T14:54:26.801 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:26.827 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:26.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:26.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:26.930 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:26.958 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:26.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:26.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:27.060 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:27.101 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:27.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:27.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:27.202 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:27.231 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:27.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:27.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:27.247 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:27 smithi102 ceph-mon[115269]: pgmap v1448: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.0 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1175/50424 objects degraded (2.330%) 2024-09-13T14:54:27.247 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:27 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:54:27.247 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:27 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:54:27.248 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:27 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:54:27.248 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:27 smithi102 ceph-mon[110183]: pgmap v1448: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.0 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1175/50424 objects degraded (2.330%) 2024-09-13T14:54:27.248 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:27 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:54:27.248 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:27 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:54:27.248 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:27 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:54:27.333 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:27.362 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:27.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:27.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:27.464 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:27.492 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:27.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:27.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:27.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:27 smithi195 ceph-mon[104262]: pgmap v1448: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.0 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1175/50424 objects degraded (2.330%) 2024-09-13T14:54:27.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:27 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:54:27.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:27 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:54:27.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:27 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:54:27.594 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:27.639 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:27.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:27.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:27.742 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:27.798 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:27.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:27.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:27.900 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:27.951 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:27.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:27.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:28.053 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:28.080 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:28.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:28.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:28.182 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:28.218 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:28.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:28.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:28.320 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:28.357 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:28.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:28.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:28.459 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:28.496 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:28.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:28.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:28.598 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:28.635 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:28.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:28.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:28.737 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:28.773 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:28.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:28.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:28.875 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:28.910 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:28.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:28.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:29.013 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:29.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:29.049 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:29.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:29.150 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:29.183 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:29.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:29.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:29.285 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:29.329 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:29.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:29.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:29.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:29 smithi102 ceph-mon[115269]: pgmap v1449: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 837 MiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1175/40047 objects degraded (2.934%) 2024-09-13T14:54:29.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:29 smithi102 ceph-mon[115269]: from='client.36672 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:54:29.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:29 smithi102 ceph-mon[110183]: pgmap v1449: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 837 MiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1175/40047 objects degraded (2.934%) 2024-09-13T14:54:29.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:29 smithi102 ceph-mon[110183]: from='client.36672 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:54:29.431 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:29.470 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:29.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:29.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:29.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:29 smithi195 ceph-mon[104262]: pgmap v1449: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 837 MiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1175/40047 objects degraded (2.934%) 2024-09-13T14:54:29.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:29 smithi195 ceph-mon[104262]: from='client.36672 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:54:29.572 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:29.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:29.594 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:29.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:29.695 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:29.730 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:29.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:29.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:29.832 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:29.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:29.874 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:29.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:29.891 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:54:29.895 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:54:29.907 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:29.975 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:30.016 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:30.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:30.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:30.118 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:30.151 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:30.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:30.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:30.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:30 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 1175/40047 objects degraded (2.934%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:54:30.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:30 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 1175/40047 objects degraded (2.934%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:54:30.253 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:30.286 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:30.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:30.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:30.388 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:30.421 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:30.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:30.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:30.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:30 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 1175/40047 objects degraded (2.934%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:54:30.523 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:30.553 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:30.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:30.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:30.655 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:30.698 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:30.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:30.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:30.801 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:30.834 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:30.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:30.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:30.936 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:30.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:30.971 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:30.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:31.072 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:31.125 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:31.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:31.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:31.227 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:31.256 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:31.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:31.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:31.358 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:31.369 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:31 smithi102 ceph-mon[110183]: pgmap v1450: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 741 MiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 750/35421 objects degraded (2.117%) 2024-09-13T14:54:31.371 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:31 smithi102 ceph-mon[115269]: pgmap v1450: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 741 MiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 750/35421 objects degraded (2.117%) 2024-09-13T14:54:31.392 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:31.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:31.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:31.494 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:31.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:31 smithi195 ceph-mon[104262]: pgmap v1450: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 741 MiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 750/35421 objects degraded (2.117%) 2024-09-13T14:54:31.525 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:31.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:31.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:31.628 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:31.660 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:31.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:31.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:31.762 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:31.793 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:31.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:31.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:31.895 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:31.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:31.932 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:31.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:32.033 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:32.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:32.070 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:32.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:32.172 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:32.203 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:32.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:32.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:32.304 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:32.332 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:32.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:32.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:32.434 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:32.461 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:32.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:32.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:32.563 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:32.600 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:32.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:32.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:32.702 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:32.736 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:32.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:32.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:32.838 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:32.847 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:54:32.847 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:54:32.880 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:32.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:32.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:32.982 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:33.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:33.021 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:33.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:33.122 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:33.162 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:33.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:33.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:33.264 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:33.293 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:33.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:33.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:33.395 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:33.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:33 smithi102 ceph-mon[110183]: pgmap v1451: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 603 MiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 750/28797 objects degraded (2.604%) 2024-09-13T14:54:33.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:33 smithi102 ceph-mon[115269]: pgmap v1451: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 603 MiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 750/28797 objects degraded (2.604%) 2024-09-13T14:54:33.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:33.435 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:33.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:33.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:33 smithi195 ceph-mon[104262]: pgmap v1451: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 603 MiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 750/28797 objects degraded (2.604%) 2024-09-13T14:54:33.536 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:33.545 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":1449,"stamp":"2024-09-13T14:54:27.756514+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":877729789,"num_objects":13349,"num_object_clones":0,"num_object_copies":40047,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1175,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13349,"num_whiteouts":0,"num_read":5999,"num_read_kb":13012,"num_write":151727,"num_write_kb":4010962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6167,"num_bytes_recovered":327551120,"num_keys_recovered":20,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":184736,"ondisk_log_size":184736,"up":357,"acting":356,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":361,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":9623920,"kb_used_data":3449520,"kb_used_omap":41,"kb_used_meta":6174166,"kb_avail":740337296,"statfs":{"total":767960285184,"available":758105391104,"internally_reserved":0,"allocated":3532308480,"data_stored":3514821014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42913,"internal_metadata":6322346079},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-660701184,"num_objects":-10082,"num_object_clones":0,"num_object_copies":-30246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-827,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-25222,"num_write_kb":-1210592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-10597,"num_bytes_recovered":-728301568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003288"},"pg_stats":[{"pgid":"9.19","version":"425'9147","reported_seq":9315,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:26.095202+0000","last_change":"2024-09-13T14:51:45.626030+0000","last_active":"2024-09-13T14:54:26.095202+0000","last_peered":"2024-09-13T14:54:26.095202+0000","last_clean":"2024-09-13T14:54:26.095202+0000","last_became_active":"2024-09-13T14:51:34.977608+0000","last_became_peered":"2024-09-13T14:51:34.977608+0000","last_unstale":"2024-09-13T14:54:26.095202+0000","last_undegraded":"2024-09-13T14:54:26.095202+0000","last_fullsized":"2024-09-13T14:54:26.095202+0000","mapping_epoch":381,"log_start":"378'624","ondisk_log_start":"378'624","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":5,"last_scrub":"388'4219","last_scrub_stamp":"2024-09-13T14:51:45.625947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:51:45.625947+0000","objects_scrubbed":2358,"log_size":8523,"log_dups_size":0,"ondisk_log_size":8523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:18:22.817526+0000","scrub_duration":306,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27131904,"num_objects":414,"num_object_clones":0,"num_object_copies":1242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4928,"num_write_kb":95488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.11","version":"90'149","reported_seq":592,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.012646+0000","last_change":"2024-09-13T14:47:38.688630+0000","last_active":"2024-09-13T14:54:19.012646+0000","last_peered":"2024-09-13T14:54:19.012646+0000","last_clean":"2024-09-13T14:54:19.012646+0000","last_became_active":"2024-09-13T14:47:38.460077+0000","last_became_peered":"2024-09-13T14:47:38.460077+0000","last_unstale":"2024-09-13T14:54:19.012646+0000","last_undegraded":"2024-09-13T14:54:19.012646+0000","last_fullsized":"2024-09-13T14:54:19.012646+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:38.688341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.688341+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:14:17.232921+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":762,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.017104+0000","last_change":"2024-09-13T14:47:47.868783+0000","last_active":"2024-09-13T14:54:19.017104+0000","last_peered":"2024-09-13T14:54:19.017104+0000","last_clean":"2024-09-13T14:54:19.017104+0000","last_became_active":"2024-09-13T14:47:47.868478+0000","last_became_peered":"2024-09-13T14:47:47.868478+0000","last_unstale":"2024-09-13T14:54:19.017104+0000","last_undegraded":"2024-09-13T14:54:19.017104+0000","last_fullsized":"2024-09-13T14:54:19.017104+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:27:01.784142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":447,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-13T14:54:12.982327+0000","last_change":"2024-09-13T14:46:01.772811+0000","last_active":"2024-09-13T14:54:12.982327+0000","last_peered":"2024-09-13T14:54:12.982327+0000","last_clean":"2024-09-13T14:54:12.982327+0000","last_became_active":"2024-09-13T14:46:01.764775+0000","last_became_peered":"2024-09-13T14:46:01.764775+0000","last_unstale":"2024-09-13T14:54:12.982327+0000","last_undegraded":"2024-09-13T14:54:12.982327+0000","last_fullsized":"2024-09-13T14:54:12.982327+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:42:20.738421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.18","version":"425'9010","reported_seq":9173,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:23.480607+0000","last_change":"2024-09-13T14:51:46.427526+0000","last_active":"2024-09-13T14:54:23.480607+0000","last_peered":"2024-09-13T14:54:23.480607+0000","last_clean":"2024-09-13T14:54:23.480607+0000","last_became_active":"2024-09-13T14:51:34.984056+0000","last_became_peered":"2024-09-13T14:51:34.984056+0000","last_unstale":"2024-09-13T14:54:23.480607+0000","last_undegraded":"2024-09-13T14:54:23.480607+0000","last_fullsized":"2024-09-13T14:54:23.480607+0000","mapping_epoch":381,"log_start":"378'2335","ondisk_log_start":"378'2335","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":5,"last_scrub":"390'4133","last_scrub_stamp":"2024-09-13T14:51:46.427440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:51:46.427440+0000","objects_scrubbed":2314,"log_size":6675,"log_dups_size":0,"ondisk_log_size":6675,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:27.127606+0000","scrub_duration":404,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32964608,"num_objects":503,"num_object_clones":0,"num_object_copies":1509,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4881,"num_write_kb":98240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.10","version":"90'149","reported_seq":491,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-13T14:54:12.986800+0000","last_change":"2024-09-13T14:47:41.482552+0000","last_active":"2024-09-13T14:54:12.986800+0000","last_peered":"2024-09-13T14:54:12.986800+0000","last_clean":"2024-09-13T14:54:12.986800+0000","last_became_active":"2024-09-13T14:47:38.466659+0000","last_became_peered":"2024-09-13T14:47:38.466659+0000","last_unstale":"2024-09-13T14:54:12.986800+0000","last_undegraded":"2024-09-13T14:54:12.986800+0000","last_fullsized":"2024-09-13T14:54:12.986800+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:41.482364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:41.482364+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:50:49.239389+0000","scrub_duration":0.027843855000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":228,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-13T14:54:12.987361+0000","last_change":"2024-09-13T14:36:13.862181+0000","last_active":"2024-09-13T14:54:12.987361+0000","last_peered":"2024-09-13T14:54:12.987361+0000","last_clean":"2024-09-13T14:54:12.987361+0000","last_became_active":"2024-09-13T14:36:13.861873+0000","last_became_peered":"2024-09-13T14:36:13.861873+0000","last_unstale":"2024-09-13T14:54:12.987361+0000","last_undegraded":"2024-09-13T14:54:12.987361+0000","last_fullsized":"2024-09-13T14:54:12.987361+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:14.085700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":385,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.010596+0000","last_change":"2024-09-13T14:38:53.896672+0000","last_active":"2024-09-13T14:54:19.010596+0000","last_peered":"2024-09-13T14:54:19.010596+0000","last_clean":"2024-09-13T14:54:19.010596+0000","last_became_active":"2024-09-13T14:38:53.896346+0000","last_became_peered":"2024-09-13T14:38:53.896346+0000","last_unstale":"2024-09-13T14:54:19.010596+0000","last_undegraded":"2024-09-13T14:54:19.010596+0000","last_fullsized":"2024-09-13T14:54:19.010596+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:57.789686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"90'149","reported_seq":612,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.022480+0000","last_change":"2024-09-13T14:47:51.226821+0000","last_active":"2024-09-13T14:54:19.022480+0000","last_peered":"2024-09-13T14:54:19.022480+0000","last_clean":"2024-09-13T14:54:19.022480+0000","last_became_active":"2024-09-13T14:47:50.881689+0000","last_became_peered":"2024-09-13T14:47:50.881689+0000","last_unstale":"2024-09-13T14:54:19.022480+0000","last_undegraded":"2024-09-13T14:54:19.022480+0000","last_fullsized":"2024-09-13T14:54:19.022480+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:51.226748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:51.226748+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:06:02.065990+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":385,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.011641+0000","last_change":"2024-09-13T14:38:53.893021+0000","last_active":"2024-09-13T14:54:19.011641+0000","last_peered":"2024-09-13T14:54:19.011641+0000","last_clean":"2024-09-13T14:54:19.011641+0000","last_became_active":"2024-09-13T14:38:53.892668+0000","last_became_peered":"2024-09-13T14:38:53.892668+0000","last_unstale":"2024-09-13T14:54:19.011641+0000","last_undegraded":"2024-09-13T14:54:19.011641+0000","last_fullsized":"2024-09-13T14:54:19.011641+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:57.180738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":400,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-13T14:54:12.981935+0000","last_change":"2024-09-13T14:37:48.134642+0000","last_active":"2024-09-13T14:54:12.981935+0000","last_peered":"2024-09-13T14:54:12.981935+0000","last_clean":"2024-09-13T14:54:12.981935+0000","last_became_active":"2024-09-13T14:37:48.134402+0000","last_became_peered":"2024-09-13T14:37:48.134402+0000","last_unstale":"2024-09-13T14:54:12.981935+0000","last_undegraded":"2024-09-13T14:54:12.981935+0000","last_fullsized":"2024-09-13T14:54:12.981935+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:51.313663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"90'149","reported_seq":508,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-13T14:54:12.987504+0000","last_change":"2024-09-13T14:47:50.892667+0000","last_active":"2024-09-13T14:54:12.987504+0000","last_peered":"2024-09-13T14:54:12.987504+0000","last_clean":"2024-09-13T14:54:12.987504+0000","last_became_active":"2024-09-13T14:47:50.883312+0000","last_became_peered":"2024-09-13T14:47:50.883312+0000","last_unstale":"2024-09-13T14:54:12.987504+0000","last_undegraded":"2024-09-13T14:54:12.987504+0000","last_fullsized":"2024-09-13T14:54:12.987504+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:45.441819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:45.441819+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:39.138573+0000","scrub_duration":0.021368558999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":788,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.022738+0000","last_change":"2024-09-13T14:50:27.491166+0000","last_active":"2024-09-13T14:54:19.022738+0000","last_peered":"2024-09-13T14:54:19.022738+0000","last_clean":"2024-09-13T14:54:19.022738+0000","last_became_active":"2024-09-13T14:50:27.489905+0000","last_became_peered":"2024-09-13T14:50:27.489905+0000","last_unstale":"2024-09-13T14:54:19.022738+0000","last_undegraded":"2024-09-13T14:54:19.022738+0000","last_fullsized":"2024-09-13T14:54:19.022738+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:32.237904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":389,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.012374+0000","last_change":"2024-09-13T14:38:03.554500+0000","last_active":"2024-09-13T14:54:19.012374+0000","last_peered":"2024-09-13T14:54:19.012374+0000","last_clean":"2024-09-13T14:54:19.012374+0000","last_became_active":"2024-09-13T14:38:03.532910+0000","last_became_peered":"2024-09-13T14:38:03.532910+0000","last_unstale":"2024-09-13T14:54:19.012374+0000","last_undegraded":"2024-09-13T14:54:19.012374+0000","last_fullsized":"2024-09-13T14:54:19.012374+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:27:50.843498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.5","version":"425'9070","reported_seq":9241,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:26.068213+0000","last_change":"2024-09-13T14:51:49.029752+0000","last_active":"2024-09-13T14:54:26.068213+0000","last_peered":"2024-09-13T14:54:26.068213+0000","last_clean":"2024-09-13T14:54:26.068213+0000","last_became_active":"2024-09-13T14:51:34.969970+0000","last_became_peered":"2024-09-13T14:51:34.969970+0000","last_unstale":"2024-09-13T14:54:26.068213+0000","last_undegraded":"2024-09-13T14:54:26.068213+0000","last_fullsized":"2024-09-13T14:54:26.068213+0000","mapping_epoch":374,"log_start":"378'469","ondisk_log_start":"378'469","created":374,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"392'4147","last_scrub_stamp":"2024-09-13T14:51:49.029682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.029682+0000","objects_scrubbed":2265,"log_size":8601,"log_dups_size":0,"ondisk_log_size":8601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:38:39.412951+0000","scrub_duration":625,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26869760,"num_objects":410,"num_object_clones":0,"num_object_copies":1230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4937,"num_write_kb":98624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"90'149","reported_seq":511,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.011680+0000","last_change":"2024-09-13T14:47:50.887140+0000","last_active":"2024-09-13T14:54:19.011680+0000","last_peered":"2024-09-13T14:54:19.011680+0000","last_clean":"2024-09-13T14:54:19.011680+0000","last_became_active":"2024-09-13T14:47:50.886603+0000","last_became_peered":"2024-09-13T14:47:50.886603+0000","last_unstale":"2024-09-13T14:54:19.011680+0000","last_undegraded":"2024-09-13T14:54:19.011680+0000","last_fullsized":"2024-09-13T14:54:19.011680+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:40.786610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:40.786610+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:00.936651+0000","scrub_duration":0.013725185000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":192,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-13T14:54:12.982233+0000","last_change":"2024-09-13T14:38:53.940238+0000","last_active":"2024-09-13T14:54:12.982233+0000","last_peered":"2024-09-13T14:54:12.982233+0000","last_clean":"2024-09-13T14:54:12.982233+0000","last_became_active":"2024-09-13T14:38:53.892825+0000","last_became_peered":"2024-09-13T14:38:53.892825+0000","last_unstale":"2024-09-13T14:54:12.982233+0000","last_undegraded":"2024-09-13T14:54:12.982233+0000","last_fullsized":"2024-09-13T14:54:12.982233+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:46.666581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":249,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-13T14:54:12.986947+0000","last_change":"2024-09-13T14:38:53.898907+0000","last_active":"2024-09-13T14:54:12.986947+0000","last_peered":"2024-09-13T14:54:12.986947+0000","last_clean":"2024-09-13T14:54:12.986947+0000","last_became_active":"2024-09-13T14:38:53.898572+0000","last_became_peered":"2024-09-13T14:38:53.898572+0000","last_unstale":"2024-09-13T14:54:12.986947+0000","last_undegraded":"2024-09-13T14:54:12.986947+0000","last_fullsized":"2024-09-13T14:54:12.986947+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:29.222150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"425'8940","reported_seq":9110,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:24.274255+0000","last_change":"2024-09-13T14:51:45.505943+0000","last_active":"2024-09-13T14:54:24.274255+0000","last_peered":"2024-09-13T14:54:24.274255+0000","last_clean":"2024-09-13T14:54:24.274255+0000","last_became_active":"2024-09-13T14:51:34.959707+0000","last_became_peered":"2024-09-13T14:51:34.959707+0000","last_unstale":"2024-09-13T14:54:24.274255+0000","last_undegraded":"2024-09-13T14:54:24.274255+0000","last_fullsized":"2024-09-13T14:54:24.274255+0000","mapping_epoch":374,"log_start":"378'2815","ondisk_log_start":"378'2815","created":374,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'4140","last_scrub_stamp":"2024-09-13T14:51:45.505871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:51:45.505871+0000","objects_scrubbed":2238,"log_size":6125,"log_dups_size":0,"ondisk_log_size":6125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:47:04.944570+0000","scrub_duration":274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28442624,"num_objects":434,"num_object_clones":0,"num_object_copies":1302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4800,"num_write_kb":95872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"90'149","reported_seq":635,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.015129+0000","last_change":"2024-09-13T14:47:38.466101+0000","last_active":"2024-09-13T14:54:19.015129+0000","last_peered":"2024-09-13T14:54:19.015129+0000","last_clean":"2024-09-13T14:54:19.015129+0000","last_became_active":"2024-09-13T14:47:38.465607+0000","last_became_peered":"2024-09-13T14:47:38.465607+0000","last_unstale":"2024-09-13T14:54:19.015129+0000","last_undegraded":"2024-09-13T14:54:19.015129+0000","last_fullsized":"2024-09-13T14:54:19.015129+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:32.060817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:32.060817+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:55.362724+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":205,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-13T14:54:12.982226+0000","last_change":"2024-09-13T14:47:47.878423+0000","last_active":"2024-09-13T14:54:12.982226+0000","last_peered":"2024-09-13T14:54:12.982226+0000","last_clean":"2024-09-13T14:54:12.982226+0000","last_became_active":"2024-09-13T14:47:47.866926+0000","last_became_peered":"2024-09-13T14:47:47.866926+0000","last_unstale":"2024-09-13T14:54:12.982226+0000","last_undegraded":"2024-09-13T14:54:12.982226+0000","last_fullsized":"2024-09-13T14:54:12.982226+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:11:07.534097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":447,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-13T14:54:12.979300+0000","last_change":"2024-09-13T14:47:47.860804+0000","last_active":"2024-09-13T14:54:12.979300+0000","last_peered":"2024-09-13T14:54:12.979300+0000","last_clean":"2024-09-13T14:54:12.979300+0000","last_became_active":"2024-09-13T14:47:47.860496+0000","last_became_peered":"2024-09-13T14:47:47.860496+0000","last_unstale":"2024-09-13T14:54:12.979300+0000","last_undegraded":"2024-09-13T14:54:12.979300+0000","last_fullsized":"2024-09-13T14:54:12.979300+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:17:57.542599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"425'9067","reported_seq":9166,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:26.676739+0000","last_change":"2024-09-13T14:54:21.707772+0000","last_active":"2024-09-13T14:54:26.676739+0000","last_peered":"2024-09-13T14:54:26.676739+0000","last_clean":"2024-09-13T14:54:26.676739+0000","last_became_active":"2024-09-13T14:51:34.964068+0000","last_became_peered":"2024-09-13T14:51:34.964068+0000","last_unstale":"2024-09-13T14:54:26.676739+0000","last_undegraded":"2024-09-13T14:54:26.676739+0000","last_fullsized":"2024-09-13T14:54:26.676739+0000","mapping_epoch":374,"log_start":"378'362","ondisk_log_start":"378'362","created":374,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"425'8922","last_scrub_stamp":"2024-09-13T14:54:21.707642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:54:21.707642+0000","objects_scrubbed":508,"log_size":8705,"log_dups_size":0,"ondisk_log_size":8705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:57:25.924695+0000","scrub_duration":0.15450330700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23658496,"num_objects":361,"num_object_clones":0,"num_object_copies":1083,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"90'149","reported_seq":536,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.018035+0000","last_change":"2024-09-13T14:47:38.468287+0000","last_active":"2024-09-13T14:54:19.018035+0000","last_peered":"2024-09-13T14:54:19.018035+0000","last_clean":"2024-09-13T14:54:19.018035+0000","last_became_active":"2024-09-13T14:47:38.467941+0000","last_became_peered":"2024-09-13T14:47:38.467941+0000","last_unstale":"2024-09-13T14:54:19.018035+0000","last_undegraded":"2024-09-13T14:54:19.018035+0000","last_fullsized":"2024-09-13T14:54:19.018035+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:34.091782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:34.091782+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:34:20.519248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":430,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-13T14:54:16.754451+0000","last_change":"2024-09-13T14:37:49.611774+0000","last_active":"2024-09-13T14:54:16.754451+0000","last_peered":"2024-09-13T14:54:16.754451+0000","last_clean":"2024-09-13T14:54:16.754451+0000","last_became_active":"2024-09-13T14:37:49.611088+0000","last_became_peered":"2024-09-13T14:37:49.611088+0000","last_unstale":"2024-09-13T14:54:16.754451+0000","last_undegraded":"2024-09-13T14:54:16.754451+0000","last_fullsized":"2024-09-13T14:54:16.754451+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:08.933213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":191,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.022634+0000","last_change":"2024-09-13T14:47:47.865523+0000","last_active":"2024-09-13T14:54:19.022634+0000","last_peered":"2024-09-13T14:54:19.022634+0000","last_clean":"2024-09-13T14:54:19.022634+0000","last_became_active":"2024-09-13T14:47:47.865180+0000","last_became_peered":"2024-09-13T14:47:47.865180+0000","last_unstale":"2024-09-13T14:54:19.022634+0000","last_undegraded":"2024-09-13T14:54:19.022634+0000","last_fullsized":"2024-09-13T14:54:19.022634+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:39:35.479251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.6","version":"425'9174","reported_seq":9301,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:26.661375+0000","last_change":"2024-09-13T14:53:24.014642+0000","last_active":"2024-09-13T14:54:26.661375+0000","last_peered":"2024-09-13T14:54:26.661375+0000","last_clean":"2024-09-13T14:54:26.661375+0000","last_became_active":"2024-09-13T14:51:34.950991+0000","last_became_peered":"2024-09-13T14:51:34.950991+0000","last_unstale":"2024-09-13T14:54:26.661375+0000","last_undegraded":"2024-09-13T14:54:26.661375+0000","last_fullsized":"2024-09-13T14:54:26.661375+0000","mapping_epoch":374,"log_start":"378'689","ondisk_log_start":"378'689","created":374,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'7316","last_scrub_stamp":"2024-09-13T14:53:24.014329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:53:24.014329+0000","objects_scrubbed":2274,"log_size":8485,"log_dups_size":0,"ondisk_log_size":8485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:59:56.818703+0000","scrub_duration":0.64986771499999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26607616,"num_objects":406,"num_object_clones":0,"num_object_copies":1218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1868,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"90'149","reported_seq":542,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.009697+0000","last_change":"2024-09-13T14:47:38.459769+0000","last_active":"2024-09-13T14:54:19.009697+0000","last_peered":"2024-09-13T14:54:19.009697+0000","last_clean":"2024-09-13T14:54:19.009697+0000","last_became_active":"2024-09-13T14:47:38.459440+0000","last_became_peered":"2024-09-13T14:47:38.459440+0000","last_unstale":"2024-09-13T14:54:19.009697+0000","last_undegraded":"2024-09-13T14:54:19.009697+0000","last_fullsized":"2024-09-13T14:54:19.009697+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:30.526999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:30.526999+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:00.016329+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":240,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-13T14:54:16.702431+0000","last_change":"2024-09-13T14:37:49.168003+0000","last_active":"2024-09-13T14:54:16.702431+0000","last_peered":"2024-09-13T14:54:16.702431+0000","last_clean":"2024-09-13T14:54:16.702431+0000","last_became_active":"2024-09-13T14:37:49.167638+0000","last_became_peered":"2024-09-13T14:37:49.167638+0000","last_unstale":"2024-09-13T14:54:16.702431+0000","last_undegraded":"2024-09-13T14:54:16.702431+0000","last_fullsized":"2024-09-13T14:54:16.702431+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:29.051981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":386,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.012188+0000","last_change":"2024-09-13T14:47:47.868094+0000","last_active":"2024-09-13T14:54:19.012188+0000","last_peered":"2024-09-13T14:54:19.012188+0000","last_clean":"2024-09-13T14:54:19.012188+0000","last_became_active":"2024-09-13T14:47:47.867750+0000","last_became_peered":"2024-09-13T14:47:47.867750+0000","last_unstale":"2024-09-13T14:54:19.012188+0000","last_undegraded":"2024-09-13T14:54:19.012188+0000","last_fullsized":"2024-09-13T14:54:19.012188+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:55:38.203481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1","version":"425'8972","reported_seq":9138,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:23.463604+0000","last_change":"2024-09-13T14:51:47.325893+0000","last_active":"2024-09-13T14:54:23.463604+0000","last_peered":"2024-09-13T14:54:23.463604+0000","last_clean":"2024-09-13T14:54:23.463604+0000","last_became_active":"2024-09-13T14:51:34.949258+0000","last_became_peered":"2024-09-13T14:51:34.949258+0000","last_unstale":"2024-09-13T14:54:23.463604+0000","last_undegraded":"2024-09-13T14:54:23.463604+0000","last_fullsized":"2024-09-13T14:54:23.463604+0000","mapping_epoch":374,"log_start":"378'2213","ondisk_log_start":"378'2213","created":374,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'4149","last_scrub_stamp":"2024-09-13T14:51:47.325649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:51:47.325649+0000","objects_scrubbed":2277,"log_size":6759,"log_dups_size":0,"ondisk_log_size":6759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:00:35.037975+0000","scrub_duration":278,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32243712,"num_objects":492,"num_object_clones":0,"num_object_copies":1476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4823,"num_write_kb":97216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"90'149","reported_seq":602,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-13T14:54:12.981616+0000","last_change":"2024-09-13T14:46:55.355650+0000","last_active":"2024-09-13T14:54:12.981616+0000","last_peered":"2024-09-13T14:54:12.981616+0000","last_clean":"2024-09-13T14:54:12.981616+0000","last_became_active":"2024-09-13T14:46:48.150338+0000","last_became_peered":"2024-09-13T14:46:48.150338+0000","last_unstale":"2024-09-13T14:54:12.981616+0000","last_undegraded":"2024-09-13T14:54:12.981616+0000","last_fullsized":"2024-09-13T14:54:12.981616+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:46:55.355521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:46:55.355521+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:30:53.006278+0000","scrub_duration":0.002025448,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":737,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.014740+0000","last_change":"2024-09-13T14:38:03.549595+0000","last_active":"2024-09-13T14:54:19.014740+0000","last_peered":"2024-09-13T14:54:19.014740+0000","last_clean":"2024-09-13T14:54:19.014740+0000","last_became_active":"2024-09-13T14:38:03.533331+0000","last_became_peered":"2024-09-13T14:38:03.533331+0000","last_unstale":"2024-09-13T14:54:19.014740+0000","last_undegraded":"2024-09-13T14:54:19.014740+0000","last_fullsized":"2024-09-13T14:54:19.014740+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:59:02.470343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":423,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-13T14:54:12.979691+0000","last_change":"2024-09-13T14:46:26.980367+0000","last_active":"2024-09-13T14:54:12.979691+0000","last_peered":"2024-09-13T14:54:12.979691+0000","last_clean":"2024-09-13T14:54:12.979691+0000","last_became_active":"2024-09-13T14:46:26.979937+0000","last_became_peered":"2024-09-13T14:46:26.979937+0000","last_unstale":"2024-09-13T14:54:12.979691+0000","last_undegraded":"2024-09-13T14:54:12.979691+0000","last_fullsized":"2024-09-13T14:54:12.979691+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:56.830872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"425'8963","reported_seq":9127,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:24.280689+0000","last_change":"2024-09-13T14:51:47.574877+0000","last_active":"2024-09-13T14:54:24.280689+0000","last_peered":"2024-09-13T14:54:24.280689+0000","last_clean":"2024-09-13T14:54:24.280689+0000","last_became_active":"2024-09-13T14:51:34.954215+0000","last_became_peered":"2024-09-13T14:51:34.954215+0000","last_unstale":"2024-09-13T14:54:24.280689+0000","last_undegraded":"2024-09-13T14:54:24.280689+0000","last_fullsized":"2024-09-13T14:54:24.280689+0000","mapping_epoch":374,"log_start":"378'2833","ondisk_log_start":"378'2833","created":374,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'4146","last_scrub_stamp":"2024-09-13T14:51:47.574706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:51:47.574706+0000","objects_scrubbed":2281,"log_size":6130,"log_dups_size":0,"ondisk_log_size":6130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:43:23.368120+0000","scrub_duration":297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28704768,"num_objects":438,"num_object_clones":0,"num_object_copies":1314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4817,"num_write_kb":95168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"90'149","reported_seq":504,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-13T14:54:12.986819+0000","last_change":"2024-09-13T14:46:54.393663+0000","last_active":"2024-09-13T14:54:12.986819+0000","last_peered":"2024-09-13T14:54:12.986819+0000","last_clean":"2024-09-13T14:54:12.986819+0000","last_became_active":"2024-09-13T14:46:48.149004+0000","last_became_peered":"2024-09-13T14:46:48.149004+0000","last_unstale":"2024-09-13T14:54:12.986819+0000","last_undegraded":"2024-09-13T14:54:12.986819+0000","last_fullsized":"2024-09-13T14:54:12.986819+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:46:54.393204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:46:54.393204+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:32:40.003989+0000","scrub_duration":0.015172223,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":356,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-13T14:54:12.979478+0000","last_change":"2024-09-13T14:47:47.867571+0000","last_active":"2024-09-13T14:54:12.979478+0000","last_peered":"2024-09-13T14:54:12.979478+0000","last_clean":"2024-09-13T14:54:12.979478+0000","last_became_active":"2024-09-13T14:47:47.867269+0000","last_became_peered":"2024-09-13T14:47:47.867269+0000","last_unstale":"2024-09-13T14:54:12.979478+0000","last_undegraded":"2024-09-13T14:54:12.979478+0000","last_fullsized":"2024-09-13T14:54:12.979478+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:46.093600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":485,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-13T14:54:16.364859+0000","last_change":"2024-09-13T14:47:47.866925+0000","last_active":"2024-09-13T14:54:16.364859+0000","last_peered":"2024-09-13T14:54:16.364859+0000","last_clean":"2024-09-13T14:54:16.364859+0000","last_became_active":"2024-09-13T14:47:47.866596+0000","last_became_peered":"2024-09-13T14:47:47.866596+0000","last_unstale":"2024-09-13T14:54:16.364859+0000","last_undegraded":"2024-09-13T14:54:16.364859+0000","last_fullsized":"2024-09-13T14:54:16.364859+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:11:25.617891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.3","version":"425'8867","reported_seq":13851,"reported_epoch":425,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T14:54:24.319829+0000","last_change":"2024-09-13T14:53:48.447497+0000","last_active":"2024-09-13T14:54:24.319829+0000","last_peered":"2024-09-13T14:54:24.319829+0000","last_clean":"2024-09-13T14:52:26.840083+0000","last_became_active":"2024-09-13T14:52:29.531227+0000","last_became_peered":"2024-09-13T14:52:29.531227+0000","last_unstale":"2024-09-13T14:54:24.319829+0000","last_undegraded":"2024-09-13T14:52:29.352309+0000","last_fullsized":"2024-09-13T14:52:29.332897+0000","mapping_epoch":404,"log_start":"379'2804","ondisk_log_start":"379'2804","created":374,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'5251","last_scrub_stamp":"2024-09-13T14:52:21.162564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:52:21.162564+0000","objects_scrubbed":3517,"log_size":6063,"log_dups_size":0,"ondisk_log_size":6063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":0.45406527499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27459584,"num_objects":419,"num_object_clones":0,"num_object_copies":1257,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1175,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3618,"num_write_kb":16640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2405,"num_bytes_recovered":85065728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[1,3],"avail_no_missing":["1","3","7"],"object_location_counts":[{"shards":"1,3","objects":-756}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"90'149","reported_seq":625,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.022173+0000","last_change":"2024-09-13T14:47:50.891055+0000","last_active":"2024-09-13T14:54:19.022173+0000","last_peered":"2024-09-13T14:54:19.022173+0000","last_clean":"2024-09-13T14:54:19.022173+0000","last_became_active":"2024-09-13T14:47:50.890368+0000","last_became_peered":"2024-09-13T14:47:50.890368+0000","last_unstale":"2024-09-13T14:54:19.022173+0000","last_undegraded":"2024-09-13T14:54:19.022173+0000","last_fullsized":"2024-09-13T14:54:19.022173+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:39.645908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:39.645908+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:29.800728+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":436,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-13T14:54:12.979502+0000","last_change":"2024-09-13T14:46:01.762127+0000","last_active":"2024-09-13T14:54:12.979502+0000","last_peered":"2024-09-13T14:54:12.979502+0000","last_clean":"2024-09-13T14:54:12.979502+0000","last_became_active":"2024-09-13T14:46:01.761795+0000","last_became_peered":"2024-09-13T14:46:01.761795+0000","last_unstale":"2024-09-13T14:54:12.979502+0000","last_undegraded":"2024-09-13T14:54:12.979502+0000","last_fullsized":"2024-09-13T14:54:12.979502+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:41:15.626775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":432,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.009944+0000","last_change":"2024-09-13T14:47:47.861718+0000","last_active":"2024-09-13T14:54:19.009944+0000","last_peered":"2024-09-13T14:54:19.009944+0000","last_clean":"2024-09-13T14:54:19.009944+0000","last_became_active":"2024-09-13T14:47:47.861023+0000","last_became_peered":"2024-09-13T14:47:47.861023+0000","last_unstale":"2024-09-13T14:54:19.009944+0000","last_undegraded":"2024-09-13T14:54:19.009944+0000","last_fullsized":"2024-09-13T14:54:19.009944+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:12:13.274053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"425'8719","reported_seq":8885,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:23.442684+0000","last_change":"2024-09-13T14:51:48.316128+0000","last_active":"2024-09-13T14:54:23.442684+0000","last_peered":"2024-09-13T14:54:23.442684+0000","last_clean":"2024-09-13T14:54:23.442684+0000","last_became_active":"2024-09-13T14:51:34.954425+0000","last_became_peered":"2024-09-13T14:51:34.954425+0000","last_unstale":"2024-09-13T14:54:23.442684+0000","last_undegraded":"2024-09-13T14:54:23.442684+0000","last_fullsized":"2024-09-13T14:54:23.442684+0000","mapping_epoch":374,"log_start":"378'1753","ondisk_log_start":"378'1753","created":374,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'4030","last_scrub_stamp":"2024-09-13T14:51:48.315933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:51:48.315933+0000","objects_scrubbed":2195,"log_size":6966,"log_dups_size":0,"ondisk_log_size":6966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:56:58.693839+0000","scrub_duration":269,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29884416,"num_objects":456,"num_object_clones":0,"num_object_copies":1368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4689,"num_write_kb":94400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"90'149","reported_seq":504,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-13T14:54:12.986875+0000","last_change":"2024-09-13T14:46:52.334715+0000","last_active":"2024-09-13T14:54:12.986875+0000","last_peered":"2024-09-13T14:54:12.986875+0000","last_clean":"2024-09-13T14:54:12.986875+0000","last_became_active":"2024-09-13T14:46:48.158685+0000","last_became_peered":"2024-09-13T14:46:48.158685+0000","last_unstale":"2024-09-13T14:54:12.986875+0000","last_undegraded":"2024-09-13T14:54:12.986875+0000","last_fullsized":"2024-09-13T14:54:12.986875+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:46:52.334301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:46:52.334301+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:46:03.579759+0000","scrub_duration":0.0050940330000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":762,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.010439+0000","last_change":"2024-09-13T14:47:47.861249+0000","last_active":"2024-09-13T14:54:19.010439+0000","last_peered":"2024-09-13T14:54:19.010439+0000","last_clean":"2024-09-13T14:54:19.010439+0000","last_became_active":"2024-09-13T14:47:47.860873+0000","last_became_peered":"2024-09-13T14:47:47.860873+0000","last_unstale":"2024-09-13T14:54:19.010439+0000","last_undegraded":"2024-09-13T14:54:19.010439+0000","last_fullsized":"2024-09-13T14:54:19.010439+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:37:50.709437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":190,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.064501+0000","last_change":"2024-09-13T14:47:47.870185+0000","last_active":"2024-09-13T14:54:19.064501+0000","last_peered":"2024-09-13T14:54:19.064501+0000","last_clean":"2024-09-13T14:54:19.064501+0000","last_became_active":"2024-09-13T14:47:47.869862+0000","last_became_peered":"2024-09-13T14:47:47.869862+0000","last_unstale":"2024-09-13T14:54:19.064501+0000","last_undegraded":"2024-09-13T14:54:19.064501+0000","last_fullsized":"2024-09-13T14:54:19.064501+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:07:02.812306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"425'8866","reported_seq":9032,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:23.484784+0000","last_change":"2024-09-13T14:51:49.726190+0000","last_active":"2024-09-13T14:54:23.484784+0000","last_peered":"2024-09-13T14:54:23.484784+0000","last_clean":"2024-09-13T14:54:23.484784+0000","last_became_active":"2024-09-13T14:51:34.965667+0000","last_became_peered":"2024-09-13T14:51:34.965667+0000","last_unstale":"2024-09-13T14:54:23.484784+0000","last_undegraded":"2024-09-13T14:54:23.484784+0000","last_fullsized":"2024-09-13T14:54:23.484784+0000","mapping_epoch":374,"log_start":"378'2021","ondisk_log_start":"378'2021","created":374,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"392'4161","last_scrub_stamp":"2024-09-13T14:51:49.725920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.725920+0000","objects_scrubbed":2275,"log_size":6845,"log_dups_size":0,"ondisk_log_size":6845,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:28:01.500768+0000","scrub_duration":638,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32636928,"num_objects":498,"num_object_clones":0,"num_object_copies":1494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4715,"num_write_kb":94016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":607,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-13T14:54:12.981548+0000","last_change":"2024-09-13T14:47:31.684319+0000","last_active":"2024-09-13T14:54:12.981548+0000","last_peered":"2024-09-13T14:54:12.981548+0000","last_clean":"2024-09-13T14:54:12.981548+0000","last_became_active":"2024-09-13T14:47:29.415658+0000","last_became_peered":"2024-09-13T14:47:29.415658+0000","last_unstale":"2024-09-13T14:54:12.981548+0000","last_undegraded":"2024-09-13T14:54:12.981548+0000","last_fullsized":"2024-09-13T14:54:12.981548+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:31.681433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:31.681433+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:29:34.663513+0000","scrub_duration":0.0059601139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":735,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-13T14:54:12.987146+0000","last_change":"2024-09-13T14:46:04.780718+0000","last_active":"2024-09-13T14:54:12.987146+0000","last_peered":"2024-09-13T14:54:12.987146+0000","last_clean":"2024-09-13T14:54:12.987146+0000","last_became_active":"2024-09-13T14:46:04.780120+0000","last_became_peered":"2024-09-13T14:46:04.780120+0000","last_unstale":"2024-09-13T14:54:12.987146+0000","last_undegraded":"2024-09-13T14:54:12.987146+0000","last_fullsized":"2024-09-13T14:54:12.987146+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:57:14.475888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":690,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.013436+0000","last_change":"2024-09-13T14:37:49.160719+0000","last_active":"2024-09-13T14:54:19.013436+0000","last_peered":"2024-09-13T14:54:19.013436+0000","last_clean":"2024-09-13T14:54:19.013436+0000","last_became_active":"2024-09-13T14:37:49.160401+0000","last_became_peered":"2024-09-13T14:37:49.160401+0000","last_unstale":"2024-09-13T14:54:19.013436+0000","last_undegraded":"2024-09-13T14:54:19.013436+0000","last_fullsized":"2024-09-13T14:54:19.013436+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:21.737945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":274,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-13T14:54:12.979695+0000","last_change":"2024-09-13T14:37:48.172940+0000","last_active":"2024-09-13T14:54:12.979695+0000","last_peered":"2024-09-13T14:54:12.979695+0000","last_clean":"2024-09-13T14:54:12.979695+0000","last_became_active":"2024-09-13T14:37:48.172060+0000","last_became_peered":"2024-09-13T14:37:48.172060+0000","last_unstale":"2024-09-13T14:54:12.979695+0000","last_undegraded":"2024-09-13T14:54:12.979695+0000","last_fullsized":"2024-09-13T14:54:12.979695+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:07.480645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.b","version":"425'14156","reported_seq":14280,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:26.044111+0000","last_change":"2024-09-13T14:50:51.091679+0000","last_active":"2024-09-13T14:54:26.044111+0000","last_peered":"2024-09-13T14:54:26.044111+0000","last_clean":"2024-09-13T14:54:26.044111+0000","last_became_active":"2024-09-13T14:50:51.091340+0000","last_became_peered":"2024-09-13T14:50:51.091340+0000","last_unstale":"2024-09-13T14:54:26.044111+0000","last_undegraded":"2024-09-13T14:54:26.044111+0000","last_fullsized":"2024-09-13T14:54:26.044111+0000","mapping_epoch":374,"log_start":"405'7100","ondisk_log_start":"405'7100","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:50:49.763471+0000","objects_scrubbed":0,"log_size":7056,"log_dups_size":0,"ondisk_log_size":7056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:45:57.926434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51904512,"num_objects":792,"num_object_clones":0,"num_object_copies":2376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14156,"num_write_kb":478336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"90'149","reported_seq":620,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.022327+0000","last_change":"2024-09-13T14:47:50.884721+0000","last_active":"2024-09-13T14:54:19.022327+0000","last_peered":"2024-09-13T14:54:19.022327+0000","last_clean":"2024-09-13T14:54:19.022327+0000","last_became_active":"2024-09-13T14:47:50.880185+0000","last_became_peered":"2024-09-13T14:47:50.880185+0000","last_unstale":"2024-09-13T14:54:19.022327+0000","last_undegraded":"2024-09-13T14:54:19.022327+0000","last_fullsized":"2024-09-13T14:54:19.022327+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:34.603316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:34.603316+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:38.321686+0000","scrub_duration":0.0057131120000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":677,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.011238+0000","last_change":"2024-09-13T14:47:47.892973+0000","last_active":"2024-09-13T14:54:19.011238+0000","last_peered":"2024-09-13T14:54:19.011238+0000","last_clean":"2024-09-13T14:54:19.011238+0000","last_became_active":"2024-09-13T14:47:47.866986+0000","last_became_peered":"2024-09-13T14:47:47.866986+0000","last_unstale":"2024-09-13T14:54:19.011238+0000","last_undegraded":"2024-09-13T14:54:19.011238+0000","last_fullsized":"2024-09-13T14:54:19.011238+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:07.353290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":379,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.012461+0000","last_change":"2024-09-13T14:38:03.525682+0000","last_active":"2024-09-13T14:54:19.012461+0000","last_peered":"2024-09-13T14:54:19.012461+0000","last_clean":"2024-09-13T14:54:19.012461+0000","last_became_active":"2024-09-13T14:38:03.525406+0000","last_became_peered":"2024-09-13T14:38:03.525406+0000","last_unstale":"2024-09-13T14:54:19.012461+0000","last_undegraded":"2024-09-13T14:54:19.012461+0000","last_fullsized":"2024-09-13T14:54:19.012461+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:41:28.970300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":400,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.011701+0000","last_change":"2024-09-13T14:47:47.873233+0000","last_active":"2024-09-13T14:54:19.011701+0000","last_peered":"2024-09-13T14:54:19.011701+0000","last_clean":"2024-09-13T14:54:19.011701+0000","last_became_active":"2024-09-13T14:47:47.872788+0000","last_became_peered":"2024-09-13T14:47:47.872788+0000","last_unstale":"2024-09-13T14:54:19.011701+0000","last_undegraded":"2024-09-13T14:54:19.011701+0000","last_fullsized":"2024-09-13T14:54:19.011701+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:52.409108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"425'14259","reported_seq":14320,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:26.701924+0000","last_change":"2024-09-13T14:50:50.782599+0000","last_active":"2024-09-13T14:54:26.701924+0000","last_peered":"2024-09-13T14:54:26.701924+0000","last_clean":"2024-09-13T14:54:26.701924+0000","last_became_active":"2024-09-13T14:50:50.782273+0000","last_became_peered":"2024-09-13T14:50:50.782273+0000","last_unstale":"2024-09-13T14:54:26.701924+0000","last_undegraded":"2024-09-13T14:54:26.701924+0000","last_fullsized":"2024-09-13T14:54:26.701924+0000","mapping_epoch":374,"log_start":"405'7200","ondisk_log_start":"405'7200","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:50:49.763471+0000","objects_scrubbed":0,"log_size":7059,"log_dups_size":0,"ondisk_log_size":7059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:27:54.626945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51707904,"num_objects":789,"num_object_clones":0,"num_object_copies":2367,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14259,"num_write_kb":481536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"90'149","reported_seq":623,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-13T14:54:12.982156+0000","last_change":"2024-09-13T14:47:29.629488+0000","last_active":"2024-09-13T14:54:12.982156+0000","last_peered":"2024-09-13T14:54:12.982156+0000","last_clean":"2024-09-13T14:54:12.982156+0000","last_became_active":"2024-09-13T14:47:29.403279+0000","last_became_peered":"2024-09-13T14:47:29.403279+0000","last_unstale":"2024-09-13T14:54:12.982156+0000","last_undegraded":"2024-09-13T14:54:12.982156+0000","last_fullsized":"2024-09-13T14:54:12.982156+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:29.629342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:29.629342+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:14:55.249896+0000","scrub_duration":0.0099290990000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":674,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.025124+0000","last_change":"2024-09-13T14:46:04.782955+0000","last_active":"2024-09-13T14:54:19.025124+0000","last_peered":"2024-09-13T14:54:19.025124+0000","last_clean":"2024-09-13T14:54:19.025124+0000","last_became_active":"2024-09-13T14:46:04.782250+0000","last_became_peered":"2024-09-13T14:46:04.782250+0000","last_unstale":"2024-09-13T14:54:19.025124+0000","last_undegraded":"2024-09-13T14:54:19.025124+0000","last_fullsized":"2024-09-13T14:54:19.025124+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:24.596922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":191,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.020017+0000","last_change":"2024-09-13T14:47:47.866684+0000","last_active":"2024-09-13T14:54:19.020017+0000","last_peered":"2024-09-13T14:54:19.020017+0000","last_clean":"2024-09-13T14:54:19.020017+0000","last_became_active":"2024-09-13T14:47:47.866400+0000","last_became_peered":"2024-09-13T14:47:47.866400+0000","last_unstale":"2024-09-13T14:54:19.020017+0000","last_undegraded":"2024-09-13T14:54:19.020017+0000","last_fullsized":"2024-09-13T14:54:19.020017+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:08:34.289658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":191,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.064589+0000","last_change":"2024-09-13T14:47:47.870218+0000","last_active":"2024-09-13T14:54:19.064589+0000","last_peered":"2024-09-13T14:54:19.064589+0000","last_clean":"2024-09-13T14:54:19.064589+0000","last_became_active":"2024-09-13T14:47:47.869907+0000","last_became_peered":"2024-09-13T14:47:47.869907+0000","last_unstale":"2024-09-13T14:54:19.064589+0000","last_undegraded":"2024-09-13T14:54:19.064589+0000","last_fullsized":"2024-09-13T14:54:19.064589+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:41:49.141441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"425'9146","reported_seq":9317,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:25.976648+0000","last_change":"2024-09-13T14:51:46.774586+0000","last_active":"2024-09-13T14:54:25.976648+0000","last_peered":"2024-09-13T14:54:25.976648+0000","last_clean":"2024-09-13T14:54:25.976648+0000","last_became_active":"2024-09-13T14:51:34.952323+0000","last_became_peered":"2024-09-13T14:51:34.952323+0000","last_unstale":"2024-09-13T14:54:25.976648+0000","last_undegraded":"2024-09-13T14:54:25.976648+0000","last_fullsized":"2024-09-13T14:54:25.976648+0000","mapping_epoch":374,"log_start":"378'484","ondisk_log_start":"378'484","created":374,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'4242","last_scrub_stamp":"2024-09-13T14:51:46.774489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:51:46.774489+0000","objects_scrubbed":2307,"log_size":8662,"log_dups_size":0,"ondisk_log_size":8662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:32:50.482249+0000","scrub_duration":408,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27721728,"num_objects":423,"num_object_clones":0,"num_object_copies":1269,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4908,"num_write_kb":96768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"90'149","reported_seq":525,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-13T14:54:12.987036+0000","last_change":"2024-09-13T14:47:32.275753+0000","last_active":"2024-09-13T14:54:12.987036+0000","last_peered":"2024-09-13T14:54:12.987036+0000","last_clean":"2024-09-13T14:54:12.987036+0000","last_became_active":"2024-09-13T14:47:29.401066+0000","last_became_peered":"2024-09-13T14:47:29.401066+0000","last_unstale":"2024-09-13T14:54:12.987036+0000","last_undegraded":"2024-09-13T14:54:12.987036+0000","last_fullsized":"2024-09-13T14:54:12.987036+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:32.275562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:32.275562+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:43:41.056775+0000","scrub_duration":0.0054674260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":1180,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-13T14:54:12.987004+0000","last_change":"2024-09-13T14:47:50.892842+0000","last_active":"2024-09-13T14:54:12.987004+0000","last_peered":"2024-09-13T14:54:12.987004+0000","last_clean":"2024-09-13T14:54:12.987004+0000","last_became_active":"2024-09-13T14:47:50.892237+0000","last_became_peered":"2024-09-13T14:47:50.892237+0000","last_unstale":"2024-09-13T14:54:12.987004+0000","last_undegraded":"2024-09-13T14:54:12.987004+0000","last_fullsized":"2024-09-13T14:54:12.987004+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:20:05.796762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":8388646,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":436,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-13T14:54:12.979748+0000","last_change":"2024-09-13T14:38:03.528133+0000","last_active":"2024-09-13T14:54:12.979748+0000","last_peered":"2024-09-13T14:54:12.979748+0000","last_clean":"2024-09-13T14:54:12.979748+0000","last_became_active":"2024-09-13T14:38:03.527829+0000","last_became_peered":"2024-09-13T14:38:03.527829+0000","last_unstale":"2024-09-13T14:54:12.979748+0000","last_undegraded":"2024-09-13T14:54:12.979748+0000","last_fullsized":"2024-09-13T14:54:12.979748+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:48.254169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":191,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.022241+0000","last_change":"2024-09-13T14:47:47.877272+0000","last_active":"2024-09-13T14:54:19.022241+0000","last_peered":"2024-09-13T14:54:19.022241+0000","last_clean":"2024-09-13T14:54:19.022241+0000","last_became_active":"2024-09-13T14:47:47.877003+0000","last_became_peered":"2024-09-13T14:47:47.877003+0000","last_unstale":"2024-09-13T14:54:19.022241+0000","last_undegraded":"2024-09-13T14:54:19.022241+0000","last_fullsized":"2024-09-13T14:54:19.022241+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:47:23.172452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.f","version":"425'14684","reported_seq":14745,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:26.677889+0000","last_change":"2024-09-13T14:50:50.783992+0000","last_active":"2024-09-13T14:54:26.677889+0000","last_peered":"2024-09-13T14:54:26.677889+0000","last_clean":"2024-09-13T14:54:26.677889+0000","last_became_active":"2024-09-13T14:50:50.783687+0000","last_became_peered":"2024-09-13T14:50:50.783687+0000","last_unstale":"2024-09-13T14:54:26.677889+0000","last_undegraded":"2024-09-13T14:54:26.677889+0000","last_fullsized":"2024-09-13T14:54:26.677889+0000","mapping_epoch":374,"log_start":"405'7700","ondisk_log_start":"405'7700","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:50:49.763471+0000","objects_scrubbed":0,"log_size":6984,"log_dups_size":0,"ondisk_log_size":6984,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:30.412734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51380224,"num_objects":784,"num_object_clones":0,"num_object_copies":2352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14684,"num_write_kb":494976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"90'149","reported_seq":623,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.021665+0000","last_change":"2024-09-13T14:47:33.067298+0000","last_active":"2024-09-13T14:54:19.021665+0000","last_peered":"2024-09-13T14:54:19.021665+0000","last_clean":"2024-09-13T14:54:19.021665+0000","last_became_active":"2024-09-13T14:47:29.419416+0000","last_became_peered":"2024-09-13T14:47:29.419416+0000","last_unstale":"2024-09-13T14:54:19.021665+0000","last_undegraded":"2024-09-13T14:54:19.021665+0000","last_fullsized":"2024-09-13T14:54:19.021665+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:33.066954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:33.066954+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:05:42.808060+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":249,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-13T14:54:12.987253+0000","last_change":"2024-09-13T14:46:01.763906+0000","last_active":"2024-09-13T14:54:12.987253+0000","last_peered":"2024-09-13T14:54:12.987253+0000","last_clean":"2024-09-13T14:54:12.987253+0000","last_became_active":"2024-09-13T14:46:01.763542+0000","last_became_peered":"2024-09-13T14:46:01.763542+0000","last_unstale":"2024-09-13T14:54:12.987253+0000","last_undegraded":"2024-09-13T14:54:12.987253+0000","last_fullsized":"2024-09-13T14:54:12.987253+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:09:36.652767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":1479,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.022309+0000","last_change":"2024-09-13T14:47:50.883365+0000","last_active":"2024-09-13T14:54:19.022309+0000","last_peered":"2024-09-13T14:54:19.022309+0000","last_clean":"2024-09-13T14:54:19.022309+0000","last_became_active":"2024-09-13T14:47:50.882609+0000","last_became_peered":"2024-09-13T14:47:50.882609+0000","last_unstale":"2024-09-13T14:54:19.022309+0000","last_undegraded":"2024-09-13T14:54:19.022309+0000","last_fullsized":"2024-09-13T14:54:19.022309+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:12:17.961575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":191,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.022732+0000","last_change":"2024-09-13T14:47:47.866979+0000","last_active":"2024-09-13T14:54:19.022732+0000","last_peered":"2024-09-13T14:54:19.022732+0000","last_clean":"2024-09-13T14:54:19.022732+0000","last_became_active":"2024-09-13T14:47:47.866634+0000","last_became_peered":"2024-09-13T14:47:47.866634+0000","last_unstale":"2024-09-13T14:54:19.022732+0000","last_undegraded":"2024-09-13T14:54:19.022732+0000","last_fullsized":"2024-09-13T14:54:19.022732+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:47:09.696139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.e","version":"425'14250","reported_seq":14311,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:26.753707+0000","last_change":"2024-09-13T14:50:50.782813+0000","last_active":"2024-09-13T14:54:26.753707+0000","last_peered":"2024-09-13T14:54:26.753707+0000","last_clean":"2024-09-13T14:54:26.753707+0000","last_became_active":"2024-09-13T14:50:50.782515+0000","last_became_peered":"2024-09-13T14:50:50.782515+0000","last_unstale":"2024-09-13T14:54:26.753707+0000","last_undegraded":"2024-09-13T14:54:26.753707+0000","last_fullsized":"2024-09-13T14:54:26.753707+0000","mapping_epoch":374,"log_start":"408'8500","ondisk_log_start":"408'8500","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:50:49.763471+0000","objects_scrubbed":0,"log_size":5750,"log_dups_size":0,"ondisk_log_size":5750,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:20.624059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331648,"num_objects":768,"num_object_clones":0,"num_object_copies":2304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14250,"num_write_kb":480576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"90'149","reported_seq":629,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.010874+0000","last_change":"2024-09-13T14:47:29.556225+0000","last_active":"2024-09-13T14:54:19.010874+0000","last_peered":"2024-09-13T14:54:19.010874+0000","last_clean":"2024-09-13T14:54:19.010874+0000","last_became_active":"2024-09-13T14:47:29.408774+0000","last_became_peered":"2024-09-13T14:47:29.408774+0000","last_unstale":"2024-09-13T14:54:19.010874+0000","last_undegraded":"2024-09-13T14:54:19.010874+0000","last_fullsized":"2024-09-13T14:54:19.010874+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:29.556149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:29.556149+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:35.691178+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":191,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.022399+0000","last_change":"2024-09-13T14:47:47.876562+0000","last_active":"2024-09-13T14:54:19.022399+0000","last_peered":"2024-09-13T14:54:19.022399+0000","last_clean":"2024-09-13T14:54:19.022399+0000","last_became_active":"2024-09-13T14:47:47.876297+0000","last_became_peered":"2024-09-13T14:47:47.876297+0000","last_unstale":"2024-09-13T14:54:19.022399+0000","last_undegraded":"2024-09-13T14:54:19.022399+0000","last_fullsized":"2024-09-13T14:54:19.022399+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:03:27.761394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":952,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-13T14:54:12.982255+0000","last_change":"2024-09-13T14:38:04.596192+0000","last_active":"2024-09-13T14:54:12.982255+0000","last_peered":"2024-09-13T14:54:12.982255+0000","last_clean":"2024-09-13T14:54:12.982255+0000","last_became_active":"2024-09-13T14:38:04.593299+0000","last_became_peered":"2024-09-13T14:38:04.593299+0000","last_unstale":"2024-09-13T14:54:12.982255+0000","last_undegraded":"2024-09-13T14:54:12.982255+0000","last_fullsized":"2024-09-13T14:54:12.982255+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:34.241410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":379,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.012274+0000","last_change":"2024-09-13T14:38:03.533120+0000","last_active":"2024-09-13T14:54:19.012274+0000","last_peered":"2024-09-13T14:54:19.012274+0000","last_clean":"2024-09-13T14:54:19.012274+0000","last_became_active":"2024-09-13T14:38:03.532829+0000","last_became_peered":"2024-09-13T14:38:03.532829+0000","last_unstale":"2024-09-13T14:54:19.012274+0000","last_undegraded":"2024-09-13T14:54:19.012274+0000","last_fullsized":"2024-09-13T14:54:19.012274+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:24.514661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.d","version":"425'14445","reported_seq":14507,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:26.706544+0000","last_change":"2024-09-13T14:50:50.782589+0000","last_active":"2024-09-13T14:54:26.706544+0000","last_peered":"2024-09-13T14:54:26.706544+0000","last_clean":"2024-09-13T14:54:26.706544+0000","last_became_active":"2024-09-13T14:50:50.782292+0000","last_became_peered":"2024-09-13T14:50:50.782292+0000","last_unstale":"2024-09-13T14:54:26.706544+0000","last_undegraded":"2024-09-13T14:54:26.706544+0000","last_fullsized":"2024-09-13T14:54:26.706544+0000","mapping_epoch":374,"log_start":"408'8700","ondisk_log_start":"408'8700","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:50:49.763471+0000","objects_scrubbed":0,"log_size":5745,"log_dups_size":0,"ondisk_log_size":5745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:01:30.753189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49807384,"num_objects":761,"num_object_clones":0,"num_object_copies":2283,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":761,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14445,"num_write_kb":486529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"90'149","reported_seq":525,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-13T14:54:12.979897+0000","last_change":"2024-09-13T14:47:30.528588+0000","last_active":"2024-09-13T14:54:12.979897+0000","last_peered":"2024-09-13T14:54:12.979897+0000","last_clean":"2024-09-13T14:54:12.979897+0000","last_became_active":"2024-09-13T14:47:29.413621+0000","last_became_peered":"2024-09-13T14:47:29.413621+0000","last_unstale":"2024-09-13T14:54:12.979897+0000","last_undegraded":"2024-09-13T14:54:12.979897+0000","last_fullsized":"2024-09-13T14:54:12.979897+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:30.528112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:30.528112+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:30.462009+0000","scrub_duration":0.012751485,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":781,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.022951+0000","last_change":"2024-09-13T14:38:53.938227+0000","last_active":"2024-09-13T14:54:19.022951+0000","last_peered":"2024-09-13T14:54:19.022951+0000","last_clean":"2024-09-13T14:54:19.022951+0000","last_became_active":"2024-09-13T14:38:53.902867+0000","last_became_peered":"2024-09-13T14:38:53.902867+0000","last_unstale":"2024-09-13T14:54:19.022951+0000","last_undegraded":"2024-09-13T14:54:19.022951+0000","last_fullsized":"2024-09-13T14:54:19.022951+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:16.831756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":661,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.064951+0000","last_change":"2024-09-13T14:47:50.884706+0000","last_active":"2024-09-13T14:54:19.064951+0000","last_peered":"2024-09-13T14:54:19.064951+0000","last_clean":"2024-09-13T14:54:19.064951+0000","last_became_active":"2024-09-13T14:47:50.879750+0000","last_became_peered":"2024-09-13T14:47:50.879750+0000","last_unstale":"2024-09-13T14:54:19.064951+0000","last_undegraded":"2024-09-13T14:54:19.064951+0000","last_fullsized":"2024-09-13T14:54:19.064951+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:34:32.941266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":763,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.008440+0000","last_change":"2024-09-13T14:38:53.900107+0000","last_active":"2024-09-13T14:54:19.008440+0000","last_peered":"2024-09-13T14:54:19.008440+0000","last_clean":"2024-09-13T14:54:19.008440+0000","last_became_active":"2024-09-13T14:38:53.899556+0000","last_became_peered":"2024-09-13T14:38:53.899556+0000","last_unstale":"2024-09-13T14:54:19.008440+0000","last_undegraded":"2024-09-13T14:54:19.008440+0000","last_fullsized":"2024-09-13T14:54:19.008440+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:15:47.918163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"425'14140","reported_seq":14263,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:23.505805+0000","last_change":"2024-09-13T14:50:50.776299+0000","last_active":"2024-09-13T14:54:23.505805+0000","last_peered":"2024-09-13T14:54:23.505805+0000","last_clean":"2024-09-13T14:54:23.505805+0000","last_became_active":"2024-09-13T14:50:50.776003+0000","last_became_peered":"2024-09-13T14:50:50.776003+0000","last_unstale":"2024-09-13T14:54:23.505805+0000","last_undegraded":"2024-09-13T14:54:23.505805+0000","last_fullsized":"2024-09-13T14:54:23.505805+0000","mapping_epoch":374,"log_start":"406'8075","ondisk_log_start":"406'8075","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:50:49.763471+0000","objects_scrubbed":0,"log_size":6065,"log_dups_size":0,"ondisk_log_size":6065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:26.038913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61210624,"num_objects":934,"num_object_clones":0,"num_object_copies":2802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14140,"num_write_kb":482368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"90'149","reported_seq":620,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.014814+0000","last_change":"2024-09-13T14:47:29.999643+0000","last_active":"2024-09-13T14:54:19.014814+0000","last_peered":"2024-09-13T14:54:19.014814+0000","last_clean":"2024-09-13T14:54:19.014814+0000","last_became_active":"2024-09-13T14:47:29.410951+0000","last_became_peered":"2024-09-13T14:47:29.410951+0000","last_unstale":"2024-09-13T14:54:19.014814+0000","last_undegraded":"2024-09-13T14:54:19.014814+0000","last_fullsized":"2024-09-13T14:54:19.014814+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:29.999519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:29.999519+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:04:54.887884+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":405,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.011176+0000","last_change":"2024-09-13T14:47:47.883475+0000","last_active":"2024-09-13T14:54:19.011176+0000","last_peered":"2024-09-13T14:54:19.011176+0000","last_clean":"2024-09-13T14:54:19.011176+0000","last_became_active":"2024-09-13T14:47:47.871525+0000","last_became_peered":"2024-09-13T14:47:47.871525+0000","last_unstale":"2024-09-13T14:54:19.011176+0000","last_undegraded":"2024-09-13T14:54:19.011176+0000","last_fullsized":"2024-09-13T14:54:19.011176+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:33.903432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":230,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-13T14:54:12.982271+0000","last_change":"2024-09-13T14:46:01.781508+0000","last_active":"2024-09-13T14:54:12.982271+0000","last_peered":"2024-09-13T14:54:12.982271+0000","last_clean":"2024-09-13T14:54:12.982271+0000","last_became_active":"2024-09-13T14:46:01.767096+0000","last_became_peered":"2024-09-13T14:46:01.767096+0000","last_unstale":"2024-09-13T14:54:12.982271+0000","last_undegraded":"2024-09-13T14:54:12.982271+0000","last_fullsized":"2024-09-13T14:54:12.982271+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:20:01.119189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":420,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-13T14:54:12.979394+0000","last_change":"2024-09-13T14:37:49.172218+0000","last_active":"2024-09-13T14:54:12.979394+0000","last_peered":"2024-09-13T14:54:12.979394+0000","last_clean":"2024-09-13T14:54:12.979394+0000","last_became_active":"2024-09-13T14:37:49.171718+0000","last_became_peered":"2024-09-13T14:37:49.171718+0000","last_unstale":"2024-09-13T14:54:12.979394+0000","last_undegraded":"2024-09-13T14:54:12.979394+0000","last_fullsized":"2024-09-13T14:54:12.979394+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:46:54.626521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":451,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-13T14:54:12.979642+0000","last_change":"2024-09-13T14:47:47.868409+0000","last_active":"2024-09-13T14:54:12.979642+0000","last_peered":"2024-09-13T14:54:12.979642+0000","last_clean":"2024-09-13T14:54:12.979642+0000","last_became_active":"2024-09-13T14:47:47.867712+0000","last_became_peered":"2024-09-13T14:47:47.867712+0000","last_unstale":"2024-09-13T14:54:12.979642+0000","last_undegraded":"2024-09-13T14:54:12.979642+0000","last_fullsized":"2024-09-13T14:54:12.979642+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:57:14.647493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":422,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-13T14:54:12.979923+0000","last_change":"2024-09-13T14:37:48.128447+0000","last_active":"2024-09-13T14:54:12.979923+0000","last_peered":"2024-09-13T14:54:12.979923+0000","last_clean":"2024-09-13T14:54:12.979923+0000","last_became_active":"2024-09-13T14:37:48.128143+0000","last_became_peered":"2024-09-13T14:37:48.128143+0000","last_unstale":"2024-09-13T14:54:12.979923+0000","last_undegraded":"2024-09-13T14:54:12.979923+0000","last_fullsized":"2024-09-13T14:54:12.979923+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:22.204039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":190,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.022090+0000","last_change":"2024-09-13T14:47:47.867935+0000","last_active":"2024-09-13T14:54:19.022090+0000","last_peered":"2024-09-13T14:54:19.022090+0000","last_clean":"2024-09-13T14:54:19.022090+0000","last_became_active":"2024-09-13T14:47:47.867616+0000","last_became_peered":"2024-09-13T14:47:47.867616+0000","last_unstale":"2024-09-13T14:54:19.022090+0000","last_undegraded":"2024-09-13T14:54:19.022090+0000","last_fullsized":"2024-09-13T14:54:19.022090+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:52:38.619916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":194,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-13T14:54:12.982088+0000","last_change":"2024-09-13T14:38:53.954281+0000","last_active":"2024-09-13T14:54:12.982088+0000","last_peered":"2024-09-13T14:54:12.982088+0000","last_clean":"2024-09-13T14:54:12.982088+0000","last_became_active":"2024-09-13T14:38:53.900912+0000","last_became_peered":"2024-09-13T14:38:53.900912+0000","last_unstale":"2024-09-13T14:54:12.982088+0000","last_undegraded":"2024-09-13T14:54:12.982088+0000","last_fullsized":"2024-09-13T14:54:12.982088+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:47.876275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"90'149","reported_seq":623,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.025324+0000","last_change":"2024-09-13T14:47:38.468095+0000","last_active":"2024-09-13T14:54:19.025324+0000","last_peered":"2024-09-13T14:54:19.025324+0000","last_clean":"2024-09-13T14:54:19.025324+0000","last_became_active":"2024-09-13T14:47:38.467526+0000","last_became_peered":"2024-09-13T14:47:38.467526+0000","last_unstale":"2024-09-13T14:54:19.025324+0000","last_undegraded":"2024-09-13T14:54:19.025324+0000","last_fullsized":"2024-09-13T14:54:19.025324+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:31.013128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:31.013128+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:17:20.508142+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":255,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-13T14:54:12.987390+0000","last_change":"2024-09-13T14:47:47.861842+0000","last_active":"2024-09-13T14:54:12.987390+0000","last_peered":"2024-09-13T14:54:12.987390+0000","last_clean":"2024-09-13T14:54:12.987390+0000","last_became_active":"2024-09-13T14:47:47.861492+0000","last_became_peered":"2024-09-13T14:47:47.861492+0000","last_unstale":"2024-09-13T14:54:12.987390+0000","last_undegraded":"2024-09-13T14:54:12.987390+0000","last_fullsized":"2024-09-13T14:54:12.987390+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:17.279533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":704,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.013441+0000","last_change":"2024-09-13T14:38:53.899535+0000","last_active":"2024-09-13T14:54:19.013441+0000","last_peered":"2024-09-13T14:54:19.013441+0000","last_clean":"2024-09-13T14:54:19.013441+0000","last_became_active":"2024-09-13T14:38:53.899183+0000","last_became_peered":"2024-09-13T14:38:53.899183+0000","last_unstale":"2024-09-13T14:54:19.013441+0000","last_undegraded":"2024-09-13T14:54:19.013441+0000","last_fullsized":"2024-09-13T14:54:19.013441+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:33.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":378,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.011558+0000","last_change":"2024-09-13T14:38:03.543426+0000","last_active":"2024-09-13T14:54:19.011558+0000","last_peered":"2024-09-13T14:54:19.011558+0000","last_clean":"2024-09-13T14:54:19.011558+0000","last_became_active":"2024-09-13T14:38:03.542661+0000","last_became_peered":"2024-09-13T14:38:03.542661+0000","last_unstale":"2024-09-13T14:54:19.011558+0000","last_undegraded":"2024-09-13T14:54:19.011558+0000","last_fullsized":"2024-09-13T14:54:19.011558+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:54.050127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":740,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.007927+0000","last_change":"2024-09-13T14:47:47.868624+0000","last_active":"2024-09-13T14:54:19.007927+0000","last_peered":"2024-09-13T14:54:19.007927+0000","last_clean":"2024-09-13T14:54:19.007927+0000","last_became_active":"2024-09-13T14:47:47.868031+0000","last_became_peered":"2024-09-13T14:47:47.868031+0000","last_unstale":"2024-09-13T14:54:19.007927+0000","last_undegraded":"2024-09-13T14:54:19.007927+0000","last_fullsized":"2024-09-13T14:54:19.007927+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:05:25.365023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.12","version":"425'8850","reported_seq":15671,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:26.702044+0000","last_change":"2024-09-13T14:53:25.946730+0000","last_active":"2024-09-13T14:54:26.702044+0000","last_peered":"2024-09-13T14:54:26.702044+0000","last_clean":"2024-09-13T14:54:26.702044+0000","last_became_active":"2024-09-13T14:52:47.881391+0000","last_became_peered":"2024-09-13T14:52:47.881391+0000","last_unstale":"2024-09-13T14:54:26.702044+0000","last_undegraded":"2024-09-13T14:54:26.702044+0000","last_fullsized":"2024-09-13T14:54:26.702044+0000","mapping_epoch":407,"log_start":"378'159","ondisk_log_start":"378'159","created":381,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":5,"last_scrub":"412'7108","last_scrub_stamp":"2024-09-13T14:53:25.946507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:53:25.946507+0000","objects_scrubbed":2170,"log_size":8691,"log_dups_size":0,"ondisk_log_size":8691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:13:57.596649+0000","scrub_duration":0.56883067700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27590656,"num_objects":421,"num_object_clones":0,"num_object_copies":1263,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1749,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":268,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-13T14:54:12.989022+0000","last_change":"2024-09-13T14:47:47.859825+0000","last_active":"2024-09-13T14:54:12.989022+0000","last_peered":"2024-09-13T14:54:12.989022+0000","last_clean":"2024-09-13T14:54:12.989022+0000","last_became_active":"2024-09-13T14:47:47.859069+0000","last_became_peered":"2024-09-13T14:47:47.859069+0000","last_unstale":"2024-09-13T14:54:12.989022+0000","last_undegraded":"2024-09-13T14:54:12.989022+0000","last_fullsized":"2024-09-13T14:54:12.989022+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:18:36.452321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":772,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.010328+0000","last_change":"2024-09-13T14:47:47.868945+0000","last_active":"2024-09-13T14:54:19.010328+0000","last_peered":"2024-09-13T14:54:19.010328+0000","last_clean":"2024-09-13T14:54:19.010328+0000","last_became_active":"2024-09-13T14:47:47.868296+0000","last_became_peered":"2024-09-13T14:47:47.868296+0000","last_unstale":"2024-09-13T14:54:19.010328+0000","last_undegraded":"2024-09-13T14:54:19.010328+0000","last_fullsized":"2024-09-13T14:54:19.010328+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:22:24.778693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"425'8781","reported_seq":15278,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:26.659627+0000","last_change":"2024-09-13T14:54:23.022769+0000","last_active":"2024-09-13T14:54:26.659627+0000","last_peered":"2024-09-13T14:54:26.659627+0000","last_clean":"2024-09-13T14:54:26.659627+0000","last_became_active":"2024-09-13T14:52:21.079272+0000","last_became_peered":"2024-09-13T14:52:21.079272+0000","last_unstale":"2024-09-13T14:54:26.659627+0000","last_undegraded":"2024-09-13T14:54:26.659627+0000","last_fullsized":"2024-09-13T14:54:26.659627+0000","mapping_epoch":396,"log_start":"378'2604","ondisk_log_start":"378'2604","created":381,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":5,"last_scrub":"425'8665","last_scrub_stamp":"2024-09-13T14:54:23.022501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:54:23.022501+0000","objects_scrubbed":482,"log_size":6177,"log_dups_size":0,"ondisk_log_size":6177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:00.219372+0000","scrub_duration":0.160633102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23724032,"num_objects":362,"num_object_clones":0,"num_object_copies":1086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":387,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.011116+0000","last_change":"2024-09-13T14:38:53.892948+0000","last_active":"2024-09-13T14:54:19.011116+0000","last_peered":"2024-09-13T14:54:19.011116+0000","last_clean":"2024-09-13T14:54:19.011116+0000","last_became_active":"2024-09-13T14:38:53.892665+0000","last_became_peered":"2024-09-13T14:38:53.892665+0000","last_unstale":"2024-09-13T14:54:19.011116+0000","last_undegraded":"2024-09-13T14:54:19.011116+0000","last_fullsized":"2024-09-13T14:54:19.011116+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:08.292359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":190,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.019637+0000","last_change":"2024-09-13T14:47:47.871164+0000","last_active":"2024-09-13T14:54:19.019637+0000","last_peered":"2024-09-13T14:54:19.019637+0000","last_clean":"2024-09-13T14:54:19.019637+0000","last_became_active":"2024-09-13T14:47:47.870862+0000","last_became_peered":"2024-09-13T14:47:47.870862+0000","last_unstale":"2024-09-13T14:54:19.019637+0000","last_undegraded":"2024-09-13T14:54:19.019637+0000","last_fullsized":"2024-09-13T14:54:19.019637+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:41:34.392081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"425'9048","reported_seq":17971,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:26.343300+0000","last_change":"2024-09-13T14:54:21.112968+0000","last_active":"2024-09-13T14:54:26.343300+0000","last_peered":"2024-09-13T14:54:26.343300+0000","last_clean":"2024-09-13T14:54:26.343300+0000","last_became_active":"2024-09-13T14:52:20.077449+0000","last_became_peered":"2024-09-13T14:52:20.077449+0000","last_unstale":"2024-09-13T14:54:26.343300+0000","last_undegraded":"2024-09-13T14:54:26.343300+0000","last_fullsized":"2024-09-13T14:54:26.343300+0000","mapping_epoch":395,"log_start":"378'1628","ondisk_log_start":"378'1628","created":381,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":5,"last_scrub":"425'8899","last_scrub_stamp":"2024-09-13T14:54:21.112680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:54:21.112680+0000","objects_scrubbed":575,"log_size":7420,"log_dups_size":0,"ondisk_log_size":7420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:57:25.961389+0000","scrub_duration":0.15945490500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27459584,"num_objects":419,"num_object_clones":0,"num_object_copies":1257,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":760,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.014482+0000","last_change":"2024-09-13T14:50:27.490554+0000","last_active":"2024-09-13T14:54:19.014482+0000","last_peered":"2024-09-13T14:54:19.014482+0000","last_clean":"2024-09-13T14:54:19.014482+0000","last_became_active":"2024-09-13T14:50:27.490211+0000","last_became_peered":"2024-09-13T14:50:27.490211+0000","last_unstale":"2024-09-13T14:54:19.014482+0000","last_undegraded":"2024-09-13T14:54:19.014482+0000","last_fullsized":"2024-09-13T14:54:19.014482+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:57:01.947741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":504,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-13T14:54:12.979510+0000","last_change":"2024-09-13T14:47:47.868246+0000","last_active":"2024-09-13T14:54:12.979510+0000","last_peered":"2024-09-13T14:54:12.979510+0000","last_clean":"2024-09-13T14:54:12.979510+0000","last_became_active":"2024-09-13T14:47:47.867817+0000","last_became_peered":"2024-09-13T14:47:47.867817+0000","last_unstale":"2024-09-13T14:54:12.979510+0000","last_undegraded":"2024-09-13T14:54:12.979510+0000","last_fullsized":"2024-09-13T14:54:12.979510+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:07.612451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.11","version":"425'11702","reported_seq":12092,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:26.756075+0000","last_change":"2024-09-13T14:52:29.869039+0000","last_active":"2024-09-13T14:54:26.756075+0000","last_peered":"2024-09-13T14:54:26.756075+0000","last_clean":"2024-09-13T14:54:26.756075+0000","last_became_active":"2024-09-13T14:51:43.012842+0000","last_became_peered":"2024-09-13T14:51:43.012842+0000","last_unstale":"2024-09-13T14:54:26.756075+0000","last_undegraded":"2024-09-13T14:54:26.756075+0000","last_fullsized":"2024-09-13T14:54:26.756075+0000","mapping_epoch":386,"log_start":"398'5956","ondisk_log_start":"398'5956","created":381,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":5,"last_scrub":"405'8081","last_scrub_stamp":"2024-09-13T14:52:29.868545+0000","last_deep_scrub":"405'8081","last_deep_scrub_stamp":"2024-09-13T14:52:29.868545+0000","last_clean_scrub_stamp":"2024-09-13T14:52:29.868545+0000","objects_scrubbed":3578,"log_size":5746,"log_dups_size":0,"ondisk_log_size":5746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":9,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:54:13.210442+0000","scrub_duration":8.1822236159999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25427968,"num_objects":388,"num_object_clones":0,"num_object_copies":1164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3660,"num_write_kb":15040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":272,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.025472+0000","last_change":"2024-09-13T14:50:27.510133+0000","last_active":"2024-09-13T14:54:19.025472+0000","last_peered":"2024-09-13T14:54:19.025472+0000","last_clean":"2024-09-13T14:54:19.025472+0000","last_became_active":"2024-09-13T14:50:27.508774+0000","last_became_peered":"2024-09-13T14:50:27.508774+0000","last_unstale":"2024-09-13T14:54:19.025472+0000","last_undegraded":"2024-09-13T14:54:19.025472+0000","last_fullsized":"2024-09-13T14:54:19.025472+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:31:04.531746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":378,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.011989+0000","last_change":"2024-09-13T14:38:03.527021+0000","last_active":"2024-09-13T14:54:19.011989+0000","last_peered":"2024-09-13T14:54:19.011989+0000","last_clean":"2024-09-13T14:54:19.011989+0000","last_became_active":"2024-09-13T14:38:03.526698+0000","last_became_peered":"2024-09-13T14:38:03.526698+0000","last_unstale":"2024-09-13T14:54:19.011989+0000","last_undegraded":"2024-09-13T14:54:19.011989+0000","last_fullsized":"2024-09-13T14:54:19.011989+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:01:35.734086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.16","version":"425'9145","reported_seq":16376,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:26.296048+0000","last_change":"2024-09-13T14:54:22.118979+0000","last_active":"2024-09-13T14:54:26.296048+0000","last_peered":"2024-09-13T14:54:26.296048+0000","last_clean":"2024-09-13T14:54:26.296048+0000","last_became_active":"2024-09-13T14:53:46.817869+0000","last_became_peered":"2024-09-13T14:53:46.817869+0000","last_unstale":"2024-09-13T14:54:26.296048+0000","last_undegraded":"2024-09-13T14:54:26.296048+0000","last_fullsized":"2024-09-13T14:54:26.296048+0000","mapping_epoch":416,"log_start":"378'1716","ondisk_log_start":"378'1716","created":381,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":5,"last_scrub":"425'9006","last_scrub_stamp":"2024-09-13T14:54:22.118689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:54:22.118689+0000","objects_scrubbed":538,"log_size":7429,"log_dups_size":0,"ondisk_log_size":7429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:48:22.280753+0000","scrub_duration":0.190029279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26148864,"num_objects":399,"num_object_clones":0,"num_object_copies":1197,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":139,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":448,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-13T14:54:12.979607+0000","last_change":"2024-09-13T14:47:47.866784+0000","last_active":"2024-09-13T14:54:12.979607+0000","last_peered":"2024-09-13T14:54:12.979607+0000","last_clean":"2024-09-13T14:54:12.979607+0000","last_became_active":"2024-09-13T14:47:47.866483+0000","last_became_peered":"2024-09-13T14:47:47.866483+0000","last_unstale":"2024-09-13T14:54:12.979607+0000","last_undegraded":"2024-09-13T14:54:12.979607+0000","last_fullsized":"2024-09-13T14:54:12.979607+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:15:53.119741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":256,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-13T14:54:12.987005+0000","last_change":"2024-09-13T14:47:47.880946+0000","last_active":"2024-09-13T14:54:12.987005+0000","last_peered":"2024-09-13T14:54:12.987005+0000","last_clean":"2024-09-13T14:54:12.987005+0000","last_became_active":"2024-09-13T14:47:47.866534+0000","last_became_peered":"2024-09-13T14:47:47.866534+0000","last_unstale":"2024-09-13T14:54:12.987005+0000","last_undegraded":"2024-09-13T14:54:12.987005+0000","last_fullsized":"2024-09-13T14:54:12.987005+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:01:01.384873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.17","version":"425'9097","reported_seq":19864,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:26.693067+0000","last_change":"2024-09-13T14:54:25.015288+0000","last_active":"2024-09-13T14:54:26.693067+0000","last_peered":"2024-09-13T14:54:26.693067+0000","last_clean":"2024-09-13T14:54:26.693067+0000","last_became_active":"2024-09-13T14:54:12.986075+0000","last_became_peered":"2024-09-13T14:54:12.986075+0000","last_unstale":"2024-09-13T14:54:26.693067+0000","last_undegraded":"2024-09-13T14:54:26.693067+0000","last_fullsized":"2024-09-13T14:54:26.693067+0000","mapping_epoch":423,"log_start":"379'3070","ondisk_log_start":"379'3070","created":381,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":5,"last_scrub":"425'9051","last_scrub_stamp":"2024-09-13T14:54:25.015010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:54:25.015010+0000","objects_scrubbed":425,"log_size":6027,"log_dups_size":0,"ondisk_log_size":6027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:21.055686+0000","scrub_duration":0.14909209200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24707072,"num_objects":377,"num_object_clones":0,"num_object_copies":1131,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":441,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-13T14:54:12.979421+0000","last_change":"2024-09-13T14:38:03.526333+0000","last_active":"2024-09-13T14:54:12.979421+0000","last_peered":"2024-09-13T14:54:12.979421+0000","last_clean":"2024-09-13T14:54:12.979421+0000","last_became_active":"2024-09-13T14:38:03.526001+0000","last_became_peered":"2024-09-13T14:38:03.526001+0000","last_unstale":"2024-09-13T14:54:12.979421+0000","last_undegraded":"2024-09-13T14:54:12.979421+0000","last_fullsized":"2024-09-13T14:54:12.979421+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:25:25.181271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":243,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-13T14:54:12.987420+0000","last_change":"2024-09-13T14:37:49.149582+0000","last_active":"2024-09-13T14:54:12.987420+0000","last_peered":"2024-09-13T14:54:12.987420+0000","last_clean":"2024-09-13T14:54:12.987420+0000","last_became_active":"2024-09-13T14:37:49.148525+0000","last_became_peered":"2024-09-13T14:37:49.148525+0000","last_unstale":"2024-09-13T14:54:12.987420+0000","last_undegraded":"2024-09-13T14:54:12.987420+0000","last_fullsized":"2024-09-13T14:54:12.987420+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:48.655665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.14","version":"425'9049","reported_seq":9223,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:24.267147+0000","last_change":"2024-09-13T14:51:46.732408+0000","last_active":"2024-09-13T14:54:24.267147+0000","last_peered":"2024-09-13T14:54:24.267147+0000","last_clean":"2024-09-13T14:54:24.267147+0000","last_became_active":"2024-09-13T14:51:34.975041+0000","last_became_peered":"2024-09-13T14:51:34.975041+0000","last_unstale":"2024-09-13T14:54:24.267147+0000","last_undegraded":"2024-09-13T14:54:24.267147+0000","last_fullsized":"2024-09-13T14:54:24.267147+0000","mapping_epoch":381,"log_start":"379'3052","ondisk_log_start":"379'3052","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":5,"last_scrub":"390'4143","last_scrub_stamp":"2024-09-13T14:51:46.732277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:51:46.732277+0000","objects_scrubbed":2317,"log_size":5997,"log_dups_size":0,"ondisk_log_size":5997,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:17.426329+0000","scrub_duration":474,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32768000,"num_objects":500,"num_object_clones":0,"num_object_copies":1500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4913,"num_write_kb":99072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":786,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.014050+0000","last_change":"2024-09-13T14:46:01.766580+0000","last_active":"2024-09-13T14:54:19.014050+0000","last_peered":"2024-09-13T14:54:19.014050+0000","last_clean":"2024-09-13T14:54:19.014050+0000","last_became_active":"2024-09-13T14:46:01.766257+0000","last_became_peered":"2024-09-13T14:46:01.766257+0000","last_unstale":"2024-09-13T14:54:19.014050+0000","last_undegraded":"2024-09-13T14:54:19.014050+0000","last_fullsized":"2024-09-13T14:54:19.014050+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:01.126109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":750,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.015468+0000","last_change":"2024-09-13T14:46:01.769647+0000","last_active":"2024-09-13T14:54:19.015468+0000","last_peered":"2024-09-13T14:54:19.015468+0000","last_clean":"2024-09-13T14:54:19.015468+0000","last_became_active":"2024-09-13T14:46:01.769228+0000","last_became_peered":"2024-09-13T14:46:01.769228+0000","last_unstale":"2024-09-13T14:54:19.015468+0000","last_undegraded":"2024-09-13T14:54:19.015468+0000","last_fullsized":"2024-09-13T14:54:19.015468+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:58.696024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.15","version":"425'9080","reported_seq":19931,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:26.672078+0000","last_change":"2024-09-13T14:53:49.842329+0000","last_active":"2024-09-13T14:54:26.672078+0000","last_peered":"2024-09-13T14:54:26.672078+0000","last_clean":"2024-09-13T14:54:26.672078+0000","last_became_active":"2024-09-13T14:53:49.830376+0000","last_became_peered":"2024-09-13T14:53:49.830376+0000","last_unstale":"2024-09-13T14:54:26.672078+0000","last_undegraded":"2024-09-13T14:54:26.672078+0000","last_fullsized":"2024-09-13T14:54:26.672078+0000","mapping_epoch":418,"log_start":"379'3191","ondisk_log_start":"379'3191","created":381,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":5,"last_scrub":"390'4111","last_scrub_stamp":"2024-09-13T14:51:47.635938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:51:47.635938+0000","objects_scrubbed":2296,"log_size":5889,"log_dups_size":0,"ondisk_log_size":5889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:12:42.736570+0000","scrub_duration":278,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24444928,"num_objects":373,"num_object_clones":0,"num_object_copies":1119,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4969,"num_write_kb":97472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3567,"num_bytes_recovered":233766912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":430,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-13T14:54:12.979301+0000","last_change":"2024-09-13T14:47:47.860727+0000","last_active":"2024-09-13T14:54:12.979301+0000","last_peered":"2024-09-13T14:54:12.979301+0000","last_clean":"2024-09-13T14:54:12.979301+0000","last_became_active":"2024-09-13T14:47:47.860377+0000","last_became_peered":"2024-09-13T14:47:47.860377+0000","last_unstale":"2024-09-13T14:54:12.979301+0000","last_undegraded":"2024-09-13T14:54:12.979301+0000","last_fullsized":"2024-09-13T14:54:12.979301+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:27:00.868699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":398,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.011464+0000","last_change":"2024-09-13T14:46:01.775237+0000","last_active":"2024-09-13T14:54:19.011464+0000","last_peered":"2024-09-13T14:54:19.011464+0000","last_clean":"2024-09-13T14:54:19.011464+0000","last_became_active":"2024-09-13T14:46:01.765025+0000","last_became_peered":"2024-09-13T14:46:01.765025+0000","last_unstale":"2024-09-13T14:54:19.011464+0000","last_undegraded":"2024-09-13T14:54:19.011464+0000","last_fullsized":"2024-09-13T14:54:19.011464+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:29:16.089042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1746,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":26,"stat_sum":{"num_bytes":872939544,"num_objects":13321,"num_object_clones":0,"num_object_copies":39963,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1175,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13321,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150819,"num_write_kb":3998337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5972,"num_bytes_recovered":318832640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3383242752,"data_stored":3383230536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":180574,"ondisk_log_size":180574,"up":78,"acting":77,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":2154,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25132,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":151,"num_bytes_recovered":8388646,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3129,"ondisk_log_size":3129,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234527,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1570124,"kb_used_data":704708,"kb_used_omap":1,"kb_used_meta":865406,"kb_avail":92175028,"statfs":{"total":95995035648,"available":94387228672,"internally_reserved":0,"allocated":721620992,"data_stored":719423264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":886176203},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":239,"seq":1026497183945,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":745280,"kb_used_data":165240,"kb_used_omap":8,"kb_used_meta":579959,"kb_avail":92999872,"statfs":{"total":95995035648,"available":95231868928,"internally_reserved":0,"allocated":169205760,"data_stored":167020227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8453,"internal_metadata":593878779},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757792,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1561152,"kb_used_data":435832,"kb_used_omap":1,"kb_used_meta":1125310,"kb_avail":92184000,"statfs":{"total":95995035648,"available":94396416000,"internally_reserved":0,"allocated":446291968,"data_stored":444102771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1152317894},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019436,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1098628,"kb_used_data":215164,"kb_used_omap":1,"kb_used_meta":883454,"kb_avail":92646524,"statfs":{"total":95995035648,"available":94870040576,"internally_reserved":0,"allocated":220327936,"data_stored":218138364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":904657350},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435618,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1416632,"kb_used_data":442288,"kb_used_omap":5,"kb_used_meta":974330,"kb_avail":92328520,"statfs":{"total":95995035648,"available":94544404480,"internally_reserved":0,"allocated":452902912,"data_stored":450721645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":997714046},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":104,"seq":446676599147,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":490300,"kb_used_data":129588,"kb_used_omap":7,"kb_used_meta":360696,"kb_avail":93254852,"statfs":{"total":95995035648,"available":95492968448,"internally_reserved":0,"allocated":132698112,"data_stored":130513275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8010,"internal_metadata":369352886},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":99,"seq":425201762673,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1302836,"kb_used_data":910188,"kb_used_omap":7,"kb_used_meta":392632,"kb_avail":92442316,"statfs":{"total":95995035648,"available":94660931584,"internally_reserved":0,"allocated":932032512,"data_stored":929854477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7759,"internal_metadata":402055601},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926197,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1438968,"kb_used_data":446512,"kb_used_omap":7,"kb_used_meta":992376,"kb_avail":92306184,"statfs":{"total":95995035648,"available":94521532416,"internally_reserved":0,"allocated":457228288,"data_stored":455046991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":1016193320},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6860,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":431226880,"data_stored":431226880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":904138752,"data_stored":904134680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126222336,"data_stored":126222336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":438632448,"data_stored":438632448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":207093760,"data_stored":207093760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":436015104,"data_stored":436011032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":152174592,"data_stored":152174592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":687738880,"data_stored":687734808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:54:33.552 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:54:33.575 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:33.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:33.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:33.677 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:33.721 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:33.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:33.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:33.822 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:33.857 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:33.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:33.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:33.958 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:33.992 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:33.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:33.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:34.094 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:34.124 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:34.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:34.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:34.226 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:34.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:34.269 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:34.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:34.370 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:34.412 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:34.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:34.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:34.514 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:34.538 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:34.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:34.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:34.640 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:34.670 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:34.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:34.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:34.772 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:34.810 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:34.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:34.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:34.912 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:34.935 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:34.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:34.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:35.037 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:35.080 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:35.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:35.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:35.182 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:35.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:35.222 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:35.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:35.324 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:35.360 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:35.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:35.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:35.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:35 smithi102 ceph-mon[110183]: pgmap v1452: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 460 MiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 750/21930 objects degraded (3.420%) 2024-09-13T14:54:35.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:35 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 750/21930 objects degraded (3.420%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:54:35.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:35 smithi102 ceph-mon[115269]: pgmap v1452: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 460 MiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 750/21930 objects degraded (3.420%) 2024-09-13T14:54:35.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:35 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 750/21930 objects degraded (3.420%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:54:35.462 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:35.496 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:35.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:35.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:35.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:35 smithi195 ceph-mon[104262]: pgmap v1452: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 460 MiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 750/21930 objects degraded (3.420%) 2024-09-13T14:54:35.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:35 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 750/21930 objects degraded (3.420%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:54:35.598 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:35.631 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:35.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:35.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:35.732 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:35.762 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:35.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:35.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:35.864 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:35.903 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:35.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:35.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:36.005 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:36.037 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:36.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:36.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:36.139 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:36.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:36.839 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:36.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:36.877 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:54:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:54:36] "GET /metrics HTTP/1.1" 200 36513 "" "Prometheus/2.51.0" 2024-09-13T14:54:36.897 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T14:54:36.940 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:36.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:36.964 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:36.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:37.066 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:37.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:37.090 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:37.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:37.139 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:37 smithi102 ceph-mon[110183]: pgmap v1453: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 418 MiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 170/19932 objects degraded (0.853%) 2024-09-13T14:54:37.140 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:37 smithi102 ceph-mon[110183]: osdmap e426: 8 total, 8 up, 8 in 2024-09-13T14:54:37.140 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:37 smithi102 ceph-mon[115269]: pgmap v1453: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 418 MiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 170/19932 objects degraded (0.853%) 2024-09-13T14:54:37.140 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:37 smithi102 ceph-mon[115269]: osdmap e426: 8 total, 8 up, 8 in 2024-09-13T14:54:37.191 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:37.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:37.227 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:37.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:37.329 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:37.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:37.367 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:37.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:37.468 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:37.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:37 smithi195 ceph-mon[104262]: pgmap v1453: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 418 MiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 170/19932 objects degraded (0.853%) 2024-09-13T14:54:37.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:37 smithi195 ceph-mon[104262]: osdmap e426: 8 total, 8 up, 8 in 2024-09-13T14:54:37.509 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:37.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:37.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:37.612 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:37.655 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:37.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:37.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:37.758 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:37.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:37.798 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:37.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:37.899 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:37.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:37.938 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:37.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:38.040 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:38.075 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:38.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:38.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:38.177 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:38.189 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:38.194 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:38.212 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:54:38.212 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:54:38.259 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:54:38.260 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:54:38.260 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:38.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:38.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:38.295 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:54:38.295 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:54:38.328 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:54:38.328 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:54:38.339 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:38 smithi102 ceph-mon[110183]: osdmap e427: 8 total, 8 up, 8 in 2024-09-13T14:54:38.340 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:38 smithi102 ceph-mon[115269]: osdmap e427: 8 total, 8 up, 8 in 2024-09-13T14:54:38.362 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:38.365 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:54:38.366 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:54:38.400 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:54:38.400 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:54:38.401 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:38.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:38.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:38.431 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:54:38.431 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:54:38.473 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:54:38.473 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:54:38.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:38 smithi195 ceph-mon[104262]: osdmap e427: 8 total, 8 up, 8 in 2024-09-13T14:54:38.505 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:38.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:38.553 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:38.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:38.654 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:38.691 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:38.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:38.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:38.793 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:38.821 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Removed 121099 objects 2024-09-13T14:54:38.822 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Clean up completed and total clean up time :121.705 2024-09-13T14:54:38.824 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:38.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:38.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:38.926 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:38.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:38.948 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:38.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:39.049 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:39.070 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:39.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:39.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:39.146 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is unset 2024-09-13T14:54:39.172 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:39.193 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:39.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:39.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:39.238 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:39 smithi102 ceph-mon[115269]: pgmap v1456: 119 pgs: 1 creating+peering, 118 active+clean; 180 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s; 0 B/s, 93 objects/s recovering 2024-09-13T14:54:39.238 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:39 smithi102 ceph-mon[115269]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 170/19932 objects degraded (0.853%), 1 pg degraded, 1 pg undersized) 2024-09-13T14:54:39.238 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:39 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1898178882' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:54:39.238 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:39 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3346071322' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:54:39.239 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:39 smithi102 ceph-mon[110183]: pgmap v1456: 119 pgs: 1 creating+peering, 118 active+clean; 180 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s; 0 B/s, 93 objects/s recovering 2024-09-13T14:54:39.239 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:39 smithi102 ceph-mon[110183]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 170/19932 objects degraded (0.853%), 1 pg degraded, 1 pg undersized) 2024-09-13T14:54:39.239 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:39 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1898178882' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:54:39.239 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:39 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3346071322' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:54:39.295 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:39.319 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:39.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:39.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:39.421 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:39.443 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:39.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:39.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:39.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:39 smithi195 ceph-mon[104262]: pgmap v1456: 119 pgs: 1 creating+peering, 118 active+clean; 180 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s; 0 B/s, 93 objects/s recovering 2024-09-13T14:54:39.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:39 smithi195 ceph-mon[104262]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 170/19932 objects degraded (0.853%), 1 pg degraded, 1 pg undersized) 2024-09-13T14:54:39.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:39 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1898178882' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:54:39.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:39 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3346071322' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:54:39.545 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:39.566 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:39.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:39.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:39.627 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-09-13T14:54:39.628 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-09-13T14:54:39.669 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:39.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:39.693 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:39.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:39.794 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:39.815 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:39.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:39.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:39.917 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:39.940 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:39.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:39.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:40.044 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:40.065 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:40.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:40.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:40.167 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:40.189 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:40.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:40.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:40.290 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:40.312 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:40.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:40.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:40.316 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:40 smithi102 ceph-mon[110183]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:54:40.316 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:40 smithi102 ceph-mon[110183]: Cluster is now healthy 2024-09-13T14:54:40.316 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:40 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3346071322' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:54:40.316 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:40 smithi102 ceph-mon[110183]: osdmap e428: 8 total, 8 up, 8 in 2024-09-13T14:54:40.316 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:40 smithi102 ceph-mon[115269]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:54:40.316 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:40 smithi102 ceph-mon[115269]: Cluster is now healthy 2024-09-13T14:54:40.316 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:40 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3346071322' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:54:40.317 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:40 smithi102 ceph-mon[115269]: osdmap e428: 8 total, 8 up, 8 in 2024-09-13T14:54:40.414 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:40.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:40.436 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:40.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:40.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:40 smithi195 ceph-mon[104262]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:54:40.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:40 smithi195 ceph-mon[104262]: Cluster is now healthy 2024-09-13T14:54:40.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:40 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3346071322' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:54:40.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:40 smithi195 ceph-mon[104262]: osdmap e428: 8 total, 8 up, 8 in 2024-09-13T14:54:40.537 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:40.563 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:40.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:40.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:40.665 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:40.691 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:40.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:40.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:40.793 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:40.819 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:40.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:40.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:40.920 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:40.942 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:40.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:40.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:41.043 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:41.069 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:41.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:41.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:41.171 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:41.196 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:41.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:41.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:41.298 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:41.324 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:41.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:41.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:41.426 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:41.449 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:41.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:41.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:41.550 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:41.572 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:41.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:41.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:41.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:41 smithi102 ceph-mon[110183]: pgmap v1458: 119 pgs: 1 creating+peering, 118 active+clean; 98 MiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 965 op/s; 0 B/s, 124 objects/s recovering 2024-09-13T14:54:41.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:41 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:54:41.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:41 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:54:41.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:41 smithi102 ceph-mon[115269]: pgmap v1458: 119 pgs: 1 creating+peering, 118 active+clean; 98 MiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 965 op/s; 0 B/s, 124 objects/s recovering 2024-09-13T14:54:41.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:41 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:54:41.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:41 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:54:41.674 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:41.700 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:41.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:41.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:41.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:41 smithi195 ceph-mon[104262]: pgmap v1458: 119 pgs: 1 creating+peering, 118 active+clean; 98 MiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 965 op/s; 0 B/s, 124 objects/s recovering 2024-09-13T14:54:41.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:41 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:54:41.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:41 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:54:41.802 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:41.826 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:41.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:41.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:41.928 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:41.955 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:41.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:41.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:41.981 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:54:42.057 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:42.083 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:42.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:42.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:42.185 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:42.207 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:42.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:42.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:42.309 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:42.335 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:42.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:42.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:42.437 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:42.459 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:42.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:42.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:42.561 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:42.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:42.588 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:42.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:42.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:42 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "25"} : dispatch 2024-09-13T14:54:42.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:42 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "25"} : dispatch 2024-09-13T14:54:42.671 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:54:42.672 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [], 2024-09-13T14:54:42.672 INFO:teuthology.orchestra.run.smithi102.stdout: "complaint_time": 30, 2024-09-13T14:54:42.672 INFO:teuthology.orchestra.run.smithi102.stdout: "num_blocked_ops": 0 2024-09-13T14:54:42.672 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:54:42.689 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:42.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:42.715 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:42.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:42.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:42 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "25"} : dispatch 2024-09-13T14:54:42.816 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:42.842 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:42.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:42.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:42.880 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:42.944 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:42.966 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:42.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:42.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:43.068 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:43.094 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:43.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:43.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:43.196 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:43.220 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:43.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:43.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:43.322 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:43.344 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:43.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:43.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:43.445 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:43.467 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:43.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:43.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:43.568 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:43.594 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:43.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:43.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:43.681 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:43 smithi102 ceph-mon[110183]: pgmap v1459: 119 pgs: 119 active+clean; 36 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 0 B/s, 28 objects/s recovering 2024-09-13T14:54:43.681 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:43 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "25"}]': finished 2024-09-13T14:54:43.682 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:43 smithi102 ceph-mon[110183]: osdmap e429: 8 total, 8 up, 8 in 2024-09-13T14:54:43.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:43 smithi102 ceph-mon[115269]: pgmap v1459: 119 pgs: 119 active+clean; 36 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 0 B/s, 28 objects/s recovering 2024-09-13T14:54:43.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:43 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "25"}]': finished 2024-09-13T14:54:43.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:43 smithi102 ceph-mon[115269]: osdmap e429: 8 total, 8 up, 8 in 2024-09-13T14:54:43.695 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:43.719 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:43.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:43.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:43.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:43 smithi195 ceph-mon[104262]: pgmap v1459: 119 pgs: 119 active+clean; 36 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 0 B/s, 28 objects/s recovering 2024-09-13T14:54:43.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:43 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "25"}]': finished 2024-09-13T14:54:43.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:43 smithi195 ceph-mon[104262]: osdmap e429: 8 total, 8 up, 8 in 2024-09-13T14:54:43.821 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:43.843 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:43.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:43.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:43.945 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:43.967 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:43.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:43.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:44.068 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:44.095 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T14:54:44.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:44.096 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:44.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:44.197 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:44.211 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:54:44.212 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:54:44.223 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:44.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:44.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:44.325 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:44.346 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:44.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:44.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:44.448 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:44.474 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:44.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:44.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:44.576 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:44.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:44.598 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:44.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:44.699 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:44.720 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:44.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:44.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:44.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:44 smithi195 ceph-mon[104262]: from='client.46637 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:54:44.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:44 smithi195 ceph-mon[104262]: osdmap e430: 8 total, 8 up, 8 in 2024-09-13T14:54:44.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:44 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-13T14:54:44.822 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:44.843 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:44.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:44.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:44.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:44 smithi102 ceph-mon[110183]: from='client.46637 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:54:44.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:44 smithi102 ceph-mon[110183]: osdmap e430: 8 total, 8 up, 8 in 2024-09-13T14:54:44.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:44 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-13T14:54:44.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:44 smithi102 ceph-mon[115269]: from='client.46637 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:54:44.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:44 smithi102 ceph-mon[115269]: osdmap e430: 8 total, 8 up, 8 in 2024-09-13T14:54:44.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:44 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-13T14:54:44.945 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:44.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:44.968 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:44.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:45.050 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":1460,"stamp":"2024-09-13T14:54:42.460107+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7804901,"num_objects":74,"num_object_clones":0,"num_object_copies":222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":0,"num_read":5999,"num_read_kb":13012,"num_write":165002,"num_write_kb":4010962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7342,"num_bytes_recovered":327551120,"num_keys_recovered":20,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":177683,"ondisk_log_size":177683,"up":357,"acting":357,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":359,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6663312,"kb_used_data":429100,"kb_used_omap":41,"kb_used_meta":6234070,"kb_avail":743297904,"statfs":{"total":767960285184,"available":761137053696,"internally_reserved":0,"allocated":439398400,"data_stored":421836282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42913,"internal_metadata":6383687775},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,0,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-430506008,"num_objects":-6570,"num_object_clones":0,"num_object_copies":-19710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6570,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.350055"},"pg_stats":[{"pgid":"9.19","version":"427'9561","reported_seq":9735,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-13T14:54:39.698051+0000","last_change":"2024-09-13T14:51:45.626030+0000","last_active":"2024-09-13T14:54:39.698051+0000","last_peered":"2024-09-13T14:54:39.698051+0000","last_clean":"2024-09-13T14:54:39.698051+0000","last_became_active":"2024-09-13T14:51:34.977608+0000","last_became_peered":"2024-09-13T14:51:34.977608+0000","last_unstale":"2024-09-13T14:54:39.698051+0000","last_undegraded":"2024-09-13T14:54:39.698051+0000","last_fullsized":"2024-09-13T14:54:39.698051+0000","mapping_epoch":381,"log_start":"378'1438","ondisk_log_start":"378'1438","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":5,"last_scrub":"388'4219","last_scrub_stamp":"2024-09-13T14:51:45.625947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:51:45.625947+0000","objects_scrubbed":2358,"log_size":8123,"log_dups_size":0,"ondisk_log_size":8123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:18:22.817526+0000","scrub_duration":306,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5342,"num_write_kb":95488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.11","version":"90'149","reported_seq":599,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-13T14:54:39.698129+0000","last_change":"2024-09-13T14:47:38.688630+0000","last_active":"2024-09-13T14:54:39.698129+0000","last_peered":"2024-09-13T14:54:39.698129+0000","last_clean":"2024-09-13T14:54:39.698129+0000","last_became_active":"2024-09-13T14:47:38.460077+0000","last_became_peered":"2024-09-13T14:47:38.460077+0000","last_unstale":"2024-09-13T14:54:39.698129+0000","last_undegraded":"2024-09-13T14:54:39.698129+0000","last_fullsized":"2024-09-13T14:54:39.698129+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:38.688341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.688341+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:14:17.232921+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":766,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.107929+0000","last_change":"2024-09-13T14:47:47.868783+0000","last_active":"2024-09-13T14:54:37.107929+0000","last_peered":"2024-09-13T14:54:37.107929+0000","last_clean":"2024-09-13T14:54:37.107929+0000","last_became_active":"2024-09-13T14:47:47.868478+0000","last_became_peered":"2024-09-13T14:47:47.868478+0000","last_unstale":"2024-09-13T14:54:37.107929+0000","last_undegraded":"2024-09-13T14:54:37.107929+0000","last_fullsized":"2024-09-13T14:54:37.107929+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:27:01.784142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":450,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.104360+0000","last_change":"2024-09-13T14:46:01.772811+0000","last_active":"2024-09-13T14:54:37.104360+0000","last_peered":"2024-09-13T14:54:37.104360+0000","last_clean":"2024-09-13T14:54:37.104360+0000","last_became_active":"2024-09-13T14:46:01.764775+0000","last_became_peered":"2024-09-13T14:46:01.764775+0000","last_unstale":"2024-09-13T14:54:37.104360+0000","last_undegraded":"2024-09-13T14:54:37.104360+0000","last_fullsized":"2024-09-13T14:54:37.104360+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:42:20.738421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.18","version":"427'9503","reported_seq":9670,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:38.512266+0000","last_change":"2024-09-13T14:51:46.427526+0000","last_active":"2024-09-13T14:54:38.512266+0000","last_peered":"2024-09-13T14:54:38.512266+0000","last_clean":"2024-09-13T14:54:38.512266+0000","last_became_active":"2024-09-13T14:51:34.984056+0000","last_became_peered":"2024-09-13T14:51:34.984056+0000","last_unstale":"2024-09-13T14:54:38.512266+0000","last_undegraded":"2024-09-13T14:54:38.512266+0000","last_fullsized":"2024-09-13T14:54:38.512266+0000","mapping_epoch":381,"log_start":"379'3293","ondisk_log_start":"379'3293","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":5,"last_scrub":"390'4133","last_scrub_stamp":"2024-09-13T14:51:46.427440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:51:46.427440+0000","objects_scrubbed":2314,"log_size":6210,"log_dups_size":0,"ondisk_log_size":6210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:27.127606+0000","scrub_duration":404,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":655360,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5374,"num_write_kb":98240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.10","version":"90'149","reported_seq":495,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-13T14:54:39.697441+0000","last_change":"2024-09-13T14:47:41.482552+0000","last_active":"2024-09-13T14:54:39.697441+0000","last_peered":"2024-09-13T14:54:39.697441+0000","last_clean":"2024-09-13T14:54:39.697441+0000","last_became_active":"2024-09-13T14:47:38.466659+0000","last_became_peered":"2024-09-13T14:47:38.466659+0000","last_unstale":"2024-09-13T14:54:39.697441+0000","last_undegraded":"2024-09-13T14:54:39.697441+0000","last_fullsized":"2024-09-13T14:54:39.697441+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:41.482364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:41.482364+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:50:49.239389+0000","scrub_duration":0.027843855000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":231,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.142001+0000","last_change":"2024-09-13T14:36:13.862181+0000","last_active":"2024-09-13T14:54:37.142001+0000","last_peered":"2024-09-13T14:54:37.142001+0000","last_clean":"2024-09-13T14:54:37.142001+0000","last_became_active":"2024-09-13T14:36:13.861873+0000","last_became_peered":"2024-09-13T14:36:13.861873+0000","last_unstale":"2024-09-13T14:54:37.142001+0000","last_undegraded":"2024-09-13T14:54:37.142001+0000","last_fullsized":"2024-09-13T14:54:37.142001+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:14.085700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":392,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-13T14:54:39.698274+0000","last_change":"2024-09-13T14:38:53.896672+0000","last_active":"2024-09-13T14:54:39.698274+0000","last_peered":"2024-09-13T14:54:39.698274+0000","last_clean":"2024-09-13T14:54:39.698274+0000","last_became_active":"2024-09-13T14:38:53.896346+0000","last_became_peered":"2024-09-13T14:38:53.896346+0000","last_unstale":"2024-09-13T14:54:39.698274+0000","last_undegraded":"2024-09-13T14:54:39.698274+0000","last_fullsized":"2024-09-13T14:54:39.698274+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:57.789686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"90'149","reported_seq":618,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-13T14:54:39.152850+0000","last_change":"2024-09-13T14:47:51.226821+0000","last_active":"2024-09-13T14:54:39.152850+0000","last_peered":"2024-09-13T14:54:39.152850+0000","last_clean":"2024-09-13T14:54:39.152850+0000","last_became_active":"2024-09-13T14:47:50.881689+0000","last_became_peered":"2024-09-13T14:47:50.881689+0000","last_unstale":"2024-09-13T14:54:39.152850+0000","last_undegraded":"2024-09-13T14:54:39.152850+0000","last_fullsized":"2024-09-13T14:54:39.152850+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:51.226748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:51.226748+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:06:02.065990+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":391,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-13T14:54:39.698577+0000","last_change":"2024-09-13T14:38:53.893021+0000","last_active":"2024-09-13T14:54:39.698577+0000","last_peered":"2024-09-13T14:54:39.698577+0000","last_clean":"2024-09-13T14:54:39.698577+0000","last_became_active":"2024-09-13T14:38:53.892668+0000","last_became_peered":"2024-09-13T14:38:53.892668+0000","last_unstale":"2024-09-13T14:54:39.698577+0000","last_undegraded":"2024-09-13T14:54:39.698577+0000","last_fullsized":"2024-09-13T14:54:39.698577+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:57.180738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":403,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.104244+0000","last_change":"2024-09-13T14:37:48.134642+0000","last_active":"2024-09-13T14:54:37.104244+0000","last_peered":"2024-09-13T14:54:37.104244+0000","last_clean":"2024-09-13T14:54:37.104244+0000","last_became_active":"2024-09-13T14:37:48.134402+0000","last_became_peered":"2024-09-13T14:37:48.134402+0000","last_unstale":"2024-09-13T14:54:37.104244+0000","last_undegraded":"2024-09-13T14:54:37.104244+0000","last_fullsized":"2024-09-13T14:54:37.104244+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:51.313663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"90'149","reported_seq":512,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-13T14:54:39.698846+0000","last_change":"2024-09-13T14:47:50.892667+0000","last_active":"2024-09-13T14:54:39.698846+0000","last_peered":"2024-09-13T14:54:39.698846+0000","last_clean":"2024-09-13T14:54:39.698846+0000","last_became_active":"2024-09-13T14:47:50.883312+0000","last_became_peered":"2024-09-13T14:47:50.883312+0000","last_unstale":"2024-09-13T14:54:39.698846+0000","last_undegraded":"2024-09-13T14:54:39.698846+0000","last_fullsized":"2024-09-13T14:54:39.698846+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:45.441819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:45.441819+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:39.138573+0000","scrub_duration":0.021368558999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":792,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.108140+0000","last_change":"2024-09-13T14:50:27.491166+0000","last_active":"2024-09-13T14:54:37.108140+0000","last_peered":"2024-09-13T14:54:37.108140+0000","last_clean":"2024-09-13T14:54:37.108140+0000","last_became_active":"2024-09-13T14:50:27.489905+0000","last_became_peered":"2024-09-13T14:50:27.489905+0000","last_unstale":"2024-09-13T14:54:37.108140+0000","last_undegraded":"2024-09-13T14:54:37.108140+0000","last_fullsized":"2024-09-13T14:54:37.108140+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:32.237904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":395,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-13T14:54:39.697371+0000","last_change":"2024-09-13T14:38:03.554500+0000","last_active":"2024-09-13T14:54:39.697371+0000","last_peered":"2024-09-13T14:54:39.697371+0000","last_clean":"2024-09-13T14:54:39.697371+0000","last_became_active":"2024-09-13T14:38:03.532910+0000","last_became_peered":"2024-09-13T14:38:03.532910+0000","last_unstale":"2024-09-13T14:54:39.697371+0000","last_undegraded":"2024-09-13T14:54:39.697371+0000","last_fullsized":"2024-09-13T14:54:39.697371+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:27:50.843498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.5","version":"427'9480","reported_seq":9657,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-13T14:54:39.698476+0000","last_change":"2024-09-13T14:51:49.029752+0000","last_active":"2024-09-13T14:54:39.698476+0000","last_peered":"2024-09-13T14:54:39.698476+0000","last_clean":"2024-09-13T14:54:39.698476+0000","last_became_active":"2024-09-13T14:51:34.969970+0000","last_became_peered":"2024-09-13T14:51:34.969970+0000","last_unstale":"2024-09-13T14:54:39.698476+0000","last_undegraded":"2024-09-13T14:54:39.698476+0000","last_fullsized":"2024-09-13T14:54:39.698476+0000","mapping_epoch":374,"log_start":"378'1290","ondisk_log_start":"378'1290","created":374,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"392'4147","last_scrub_stamp":"2024-09-13T14:51:49.029682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.029682+0000","objects_scrubbed":2265,"log_size":8190,"log_dups_size":0,"ondisk_log_size":8190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:38:39.412951+0000","scrub_duration":625,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5347,"num_write_kb":98624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"90'149","reported_seq":514,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.104444+0000","last_change":"2024-09-13T14:47:50.887140+0000","last_active":"2024-09-13T14:54:37.104444+0000","last_peered":"2024-09-13T14:54:37.104444+0000","last_clean":"2024-09-13T14:54:37.104444+0000","last_became_active":"2024-09-13T14:47:50.886603+0000","last_became_peered":"2024-09-13T14:47:50.886603+0000","last_unstale":"2024-09-13T14:54:37.104444+0000","last_undegraded":"2024-09-13T14:54:37.104444+0000","last_fullsized":"2024-09-13T14:54:37.104444+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:40.786610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:40.786610+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:00.936651+0000","scrub_duration":0.013725185000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":195,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.104731+0000","last_change":"2024-09-13T14:38:53.940238+0000","last_active":"2024-09-13T14:54:37.104731+0000","last_peered":"2024-09-13T14:54:37.104731+0000","last_clean":"2024-09-13T14:54:37.104731+0000","last_became_active":"2024-09-13T14:38:53.892825+0000","last_became_peered":"2024-09-13T14:38:53.892825+0000","last_unstale":"2024-09-13T14:54:37.104731+0000","last_undegraded":"2024-09-13T14:54:37.104731+0000","last_fullsized":"2024-09-13T14:54:37.104731+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:46.666581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":252,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.141499+0000","last_change":"2024-09-13T14:38:53.898907+0000","last_active":"2024-09-13T14:54:37.141499+0000","last_peered":"2024-09-13T14:54:37.141499+0000","last_clean":"2024-09-13T14:54:37.141499+0000","last_became_active":"2024-09-13T14:38:53.898572+0000","last_became_peered":"2024-09-13T14:38:53.898572+0000","last_unstale":"2024-09-13T14:54:37.141499+0000","last_undegraded":"2024-09-13T14:54:37.141499+0000","last_fullsized":"2024-09-13T14:54:37.141499+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:29.222150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"427'9374","reported_seq":9550,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-13T14:54:39.153005+0000","last_change":"2024-09-13T14:51:45.505943+0000","last_active":"2024-09-13T14:54:39.153005+0000","last_peered":"2024-09-13T14:54:39.153005+0000","last_clean":"2024-09-13T14:54:39.153005+0000","last_became_active":"2024-09-13T14:51:34.959707+0000","last_became_peered":"2024-09-13T14:51:34.959707+0000","last_unstale":"2024-09-13T14:54:39.153005+0000","last_undegraded":"2024-09-13T14:54:39.153005+0000","last_fullsized":"2024-09-13T14:54:39.153005+0000","mapping_epoch":374,"log_start":"379'3520","ondisk_log_start":"379'3520","created":374,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'4140","last_scrub_stamp":"2024-09-13T14:51:45.505871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:51:45.505871+0000","objects_scrubbed":2238,"log_size":5854,"log_dups_size":0,"ondisk_log_size":5854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:47:04.944570+0000","scrub_duration":274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5234,"num_write_kb":95872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"90'149","reported_seq":639,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.107750+0000","last_change":"2024-09-13T14:47:38.466101+0000","last_active":"2024-09-13T14:54:37.107750+0000","last_peered":"2024-09-13T14:54:37.107750+0000","last_clean":"2024-09-13T14:54:37.107750+0000","last_became_active":"2024-09-13T14:47:38.465607+0000","last_became_peered":"2024-09-13T14:47:38.465607+0000","last_unstale":"2024-09-13T14:54:37.107750+0000","last_undegraded":"2024-09-13T14:54:37.107750+0000","last_fullsized":"2024-09-13T14:54:37.107750+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:32.060817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:32.060817+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:55.362724+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":208,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.104663+0000","last_change":"2024-09-13T14:47:47.878423+0000","last_active":"2024-09-13T14:54:37.104663+0000","last_peered":"2024-09-13T14:54:37.104663+0000","last_clean":"2024-09-13T14:54:37.104663+0000","last_became_active":"2024-09-13T14:47:47.866926+0000","last_became_peered":"2024-09-13T14:47:47.866926+0000","last_unstale":"2024-09-13T14:54:37.104663+0000","last_undegraded":"2024-09-13T14:54:37.104663+0000","last_fullsized":"2024-09-13T14:54:37.104663+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:11:07.534097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":450,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.150070+0000","last_change":"2024-09-13T14:47:47.860804+0000","last_active":"2024-09-13T14:54:37.150070+0000","last_peered":"2024-09-13T14:54:37.150070+0000","last_clean":"2024-09-13T14:54:37.150070+0000","last_became_active":"2024-09-13T14:47:47.860496+0000","last_became_peered":"2024-09-13T14:47:47.860496+0000","last_unstale":"2024-09-13T14:54:37.150070+0000","last_undegraded":"2024-09-13T14:54:37.150070+0000","last_fullsized":"2024-09-13T14:54:37.150070+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:17:57.542599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"427'9428","reported_seq":9530,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:38.799780+0000","last_change":"2024-09-13T14:54:21.707772+0000","last_active":"2024-09-13T14:54:38.799780+0000","last_peered":"2024-09-13T14:54:38.799780+0000","last_clean":"2024-09-13T14:54:38.799780+0000","last_became_active":"2024-09-13T14:51:34.964068+0000","last_became_peered":"2024-09-13T14:51:34.964068+0000","last_unstale":"2024-09-13T14:54:38.799780+0000","last_undegraded":"2024-09-13T14:54:38.799780+0000","last_fullsized":"2024-09-13T14:54:38.799780+0000","mapping_epoch":374,"log_start":"378'1098","ondisk_log_start":"378'1098","created":374,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"425'8922","last_scrub_stamp":"2024-09-13T14:54:21.707642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:54:21.707642+0000","objects_scrubbed":508,"log_size":8330,"log_dups_size":0,"ondisk_log_size":8330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:57:25.924695+0000","scrub_duration":0.15450330700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":508,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"90'149","reported_seq":539,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.142274+0000","last_change":"2024-09-13T14:47:38.468287+0000","last_active":"2024-09-13T14:54:37.142274+0000","last_peered":"2024-09-13T14:54:37.142274+0000","last_clean":"2024-09-13T14:54:37.142274+0000","last_became_active":"2024-09-13T14:47:38.467941+0000","last_became_peered":"2024-09-13T14:47:38.467941+0000","last_unstale":"2024-09-13T14:54:37.142274+0000","last_undegraded":"2024-09-13T14:54:37.142274+0000","last_fullsized":"2024-09-13T14:54:37.142274+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:34.091782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:34.091782+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:34:20.519248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":433,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.104656+0000","last_change":"2024-09-13T14:37:49.611774+0000","last_active":"2024-09-13T14:54:37.104656+0000","last_peered":"2024-09-13T14:54:37.104656+0000","last_clean":"2024-09-13T14:54:37.104656+0000","last_became_active":"2024-09-13T14:37:49.611088+0000","last_became_peered":"2024-09-13T14:37:49.611088+0000","last_unstale":"2024-09-13T14:54:37.104656+0000","last_undegraded":"2024-09-13T14:54:37.104656+0000","last_fullsized":"2024-09-13T14:54:37.104656+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:08.933213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":197,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-13T14:54:39.152900+0000","last_change":"2024-09-13T14:47:47.865523+0000","last_active":"2024-09-13T14:54:39.152900+0000","last_peered":"2024-09-13T14:54:39.152900+0000","last_clean":"2024-09-13T14:54:39.152900+0000","last_became_active":"2024-09-13T14:47:47.865180+0000","last_became_peered":"2024-09-13T14:47:47.865180+0000","last_unstale":"2024-09-13T14:54:39.152900+0000","last_undegraded":"2024-09-13T14:54:39.152900+0000","last_fullsized":"2024-09-13T14:54:39.152900+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:39:35.479251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.6","version":"427'9580","reported_seq":9710,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:38.811198+0000","last_change":"2024-09-13T14:53:24.014642+0000","last_active":"2024-09-13T14:54:38.811198+0000","last_peered":"2024-09-13T14:54:38.811198+0000","last_clean":"2024-09-13T14:54:38.811198+0000","last_became_active":"2024-09-13T14:51:34.950991+0000","last_became_peered":"2024-09-13T14:51:34.950991+0000","last_unstale":"2024-09-13T14:54:38.811198+0000","last_undegraded":"2024-09-13T14:54:38.811198+0000","last_fullsized":"2024-09-13T14:54:38.811198+0000","mapping_epoch":374,"log_start":"378'1507","ondisk_log_start":"378'1507","created":374,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'7316","last_scrub_stamp":"2024-09-13T14:53:24.014329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:53:24.014329+0000","objects_scrubbed":2274,"log_size":8073,"log_dups_size":0,"ondisk_log_size":8073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:59:56.818703+0000","scrub_duration":0.64986771499999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2274,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"90'149","reported_seq":545,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.150704+0000","last_change":"2024-09-13T14:47:38.459769+0000","last_active":"2024-09-13T14:54:37.150704+0000","last_peered":"2024-09-13T14:54:37.150704+0000","last_clean":"2024-09-13T14:54:37.150704+0000","last_became_active":"2024-09-13T14:47:38.459440+0000","last_became_peered":"2024-09-13T14:47:38.459440+0000","last_unstale":"2024-09-13T14:54:37.150704+0000","last_undegraded":"2024-09-13T14:54:37.150704+0000","last_fullsized":"2024-09-13T14:54:37.150704+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:30.526999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:30.526999+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:02:00.016329+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":243,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.141581+0000","last_change":"2024-09-13T14:37:49.168003+0000","last_active":"2024-09-13T14:54:37.141581+0000","last_peered":"2024-09-13T14:54:37.141581+0000","last_clean":"2024-09-13T14:54:37.141581+0000","last_became_active":"2024-09-13T14:37:49.167638+0000","last_became_peered":"2024-09-13T14:37:49.167638+0000","last_unstale":"2024-09-13T14:54:37.141581+0000","last_undegraded":"2024-09-13T14:54:37.141581+0000","last_fullsized":"2024-09-13T14:54:37.141581+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:29.051981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":392,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-13T14:54:39.698787+0000","last_change":"2024-09-13T14:47:47.868094+0000","last_active":"2024-09-13T14:54:39.698787+0000","last_peered":"2024-09-13T14:54:39.698787+0000","last_clean":"2024-09-13T14:54:39.698787+0000","last_became_active":"2024-09-13T14:47:47.867750+0000","last_became_peered":"2024-09-13T14:47:47.867750+0000","last_unstale":"2024-09-13T14:54:39.698787+0000","last_undegraded":"2024-09-13T14:54:39.698787+0000","last_fullsized":"2024-09-13T14:54:39.698787+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:55:38.203481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1","version":"427'9453","reported_seq":9623,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:38.468125+0000","last_change":"2024-09-13T14:51:47.325893+0000","last_active":"2024-09-13T14:54:38.468125+0000","last_peered":"2024-09-13T14:54:38.468125+0000","last_clean":"2024-09-13T14:54:38.468125+0000","last_became_active":"2024-09-13T14:51:34.949258+0000","last_became_peered":"2024-09-13T14:51:34.949258+0000","last_unstale":"2024-09-13T14:54:38.468125+0000","last_undegraded":"2024-09-13T14:54:38.468125+0000","last_fullsized":"2024-09-13T14:54:38.468125+0000","mapping_epoch":374,"log_start":"379'3135","ondisk_log_start":"379'3135","created":374,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'4149","last_scrub_stamp":"2024-09-13T14:51:47.325649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:51:47.325649+0000","objects_scrubbed":2277,"log_size":6318,"log_dups_size":0,"ondisk_log_size":6318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:00:35.037975+0000","scrub_duration":278,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":720896,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5304,"num_write_kb":97216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"90'149","reported_seq":605,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.104142+0000","last_change":"2024-09-13T14:46:55.355650+0000","last_active":"2024-09-13T14:54:37.104142+0000","last_peered":"2024-09-13T14:54:37.104142+0000","last_clean":"2024-09-13T14:54:37.104142+0000","last_became_active":"2024-09-13T14:46:48.150338+0000","last_became_peered":"2024-09-13T14:46:48.150338+0000","last_unstale":"2024-09-13T14:54:37.104142+0000","last_undegraded":"2024-09-13T14:54:37.104142+0000","last_fullsized":"2024-09-13T14:54:37.104142+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:46:55.355521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:46:55.355521+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:30:53.006278+0000","scrub_duration":0.002025448,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":741,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.107482+0000","last_change":"2024-09-13T14:38:03.549595+0000","last_active":"2024-09-13T14:54:37.107482+0000","last_peered":"2024-09-13T14:54:37.107482+0000","last_clean":"2024-09-13T14:54:37.107482+0000","last_became_active":"2024-09-13T14:38:03.533331+0000","last_became_peered":"2024-09-13T14:38:03.533331+0000","last_unstale":"2024-09-13T14:54:37.107482+0000","last_undegraded":"2024-09-13T14:54:37.107482+0000","last_fullsized":"2024-09-13T14:54:37.107482+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:59:02.470343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":426,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.151998+0000","last_change":"2024-09-13T14:46:26.980367+0000","last_active":"2024-09-13T14:54:37.151998+0000","last_peered":"2024-09-13T14:54:37.151998+0000","last_clean":"2024-09-13T14:54:37.151998+0000","last_became_active":"2024-09-13T14:46:26.979937+0000","last_became_peered":"2024-09-13T14:46:26.979937+0000","last_unstale":"2024-09-13T14:54:37.151998+0000","last_undegraded":"2024-09-13T14:54:37.151998+0000","last_fullsized":"2024-09-13T14:54:37.151998+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:56.830872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"427'9401","reported_seq":9571,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-13T14:54:39.152324+0000","last_change":"2024-09-13T14:51:47.574877+0000","last_active":"2024-09-13T14:54:39.152324+0000","last_peered":"2024-09-13T14:54:39.152324+0000","last_clean":"2024-09-13T14:54:39.152324+0000","last_became_active":"2024-09-13T14:51:34.954215+0000","last_became_peered":"2024-09-13T14:51:34.954215+0000","last_unstale":"2024-09-13T14:54:39.152324+0000","last_undegraded":"2024-09-13T14:54:39.152324+0000","last_fullsized":"2024-09-13T14:54:39.152324+0000","mapping_epoch":374,"log_start":"379'3577","ondisk_log_start":"379'3577","created":374,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'4146","last_scrub_stamp":"2024-09-13T14:51:47.574706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:51:47.574706+0000","objects_scrubbed":2281,"log_size":5824,"log_dups_size":0,"ondisk_log_size":5824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:43:23.368120+0000","scrub_duration":297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5255,"num_write_kb":95168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"90'149","reported_seq":507,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.141416+0000","last_change":"2024-09-13T14:46:54.393663+0000","last_active":"2024-09-13T14:54:37.141416+0000","last_peered":"2024-09-13T14:54:37.141416+0000","last_clean":"2024-09-13T14:54:37.141416+0000","last_became_active":"2024-09-13T14:46:48.149004+0000","last_became_peered":"2024-09-13T14:46:48.149004+0000","last_unstale":"2024-09-13T14:54:37.141416+0000","last_undegraded":"2024-09-13T14:54:37.141416+0000","last_fullsized":"2024-09-13T14:54:37.141416+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:46:54.393204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:46:54.393204+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:32:40.003989+0000","scrub_duration":0.015172223,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":359,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.150394+0000","last_change":"2024-09-13T14:47:47.867571+0000","last_active":"2024-09-13T14:54:37.150394+0000","last_peered":"2024-09-13T14:54:37.150394+0000","last_clean":"2024-09-13T14:54:37.150394+0000","last_became_active":"2024-09-13T14:47:47.867269+0000","last_became_peered":"2024-09-13T14:47:47.867269+0000","last_unstale":"2024-09-13T14:54:37.150394+0000","last_undegraded":"2024-09-13T14:54:37.150394+0000","last_fullsized":"2024-09-13T14:54:37.150394+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:46.093600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":488,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.150701+0000","last_change":"2024-09-13T14:47:47.866925+0000","last_active":"2024-09-13T14:54:37.150701+0000","last_peered":"2024-09-13T14:54:37.150701+0000","last_clean":"2024-09-13T14:54:37.150701+0000","last_became_active":"2024-09-13T14:47:47.866596+0000","last_became_peered":"2024-09-13T14:47:47.866596+0000","last_unstale":"2024-09-13T14:54:37.150701+0000","last_undegraded":"2024-09-13T14:54:37.150701+0000","last_fullsized":"2024-09-13T14:54:37.150701+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:11:25.617891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.3","version":"427'9286","reported_seq":16638,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-13T14:54:39.699474+0000","last_change":"2024-09-13T14:54:37.133507+0000","last_active":"2024-09-13T14:54:39.699474+0000","last_peered":"2024-09-13T14:54:39.699474+0000","last_clean":"2024-09-13T14:54:39.699474+0000","last_became_active":"2024-09-13T14:54:37.108089+0000","last_became_peered":"2024-09-13T14:54:37.108089+0000","last_unstale":"2024-09-13T14:54:39.699474+0000","last_undegraded":"2024-09-13T14:54:39.699474+0000","last_fullsized":"2024-09-13T14:54:39.699474+0000","mapping_epoch":426,"log_start":"379'3461","ondisk_log_start":"379'3461","created":374,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'5251","last_scrub_stamp":"2024-09-13T14:52:21.162564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:52:21.162564+0000","objects_scrubbed":3517,"log_size":5825,"log_dups_size":0,"ondisk_log_size":5825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:13:42.266873+0000","scrub_duration":0.45406527499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":16640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3580,"num_bytes_recovered":85065728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"90'149","reported_seq":632,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-13T14:54:39.152736+0000","last_change":"2024-09-13T14:47:50.891055+0000","last_active":"2024-09-13T14:54:39.152736+0000","last_peered":"2024-09-13T14:54:39.152736+0000","last_clean":"2024-09-13T14:54:39.152736+0000","last_became_active":"2024-09-13T14:47:50.890368+0000","last_became_peered":"2024-09-13T14:47:50.890368+0000","last_unstale":"2024-09-13T14:54:39.152736+0000","last_undegraded":"2024-09-13T14:54:39.152736+0000","last_fullsized":"2024-09-13T14:54:39.152736+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:39.645908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:39.645908+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:29.800728+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":439,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.150260+0000","last_change":"2024-09-13T14:46:01.762127+0000","last_active":"2024-09-13T14:54:37.150260+0000","last_peered":"2024-09-13T14:54:37.150260+0000","last_clean":"2024-09-13T14:54:37.150260+0000","last_became_active":"2024-09-13T14:46:01.761795+0000","last_became_peered":"2024-09-13T14:46:01.761795+0000","last_unstale":"2024-09-13T14:54:37.150260+0000","last_undegraded":"2024-09-13T14:54:37.150260+0000","last_fullsized":"2024-09-13T14:54:37.150260+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:41:15.626775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":435,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.150841+0000","last_change":"2024-09-13T14:47:47.861718+0000","last_active":"2024-09-13T14:54:37.150841+0000","last_peered":"2024-09-13T14:54:37.150841+0000","last_clean":"2024-09-13T14:54:37.150841+0000","last_became_active":"2024-09-13T14:47:47.861023+0000","last_became_peered":"2024-09-13T14:47:47.861023+0000","last_unstale":"2024-09-13T14:54:37.150841+0000","last_undegraded":"2024-09-13T14:54:37.150841+0000","last_fullsized":"2024-09-13T14:54:37.150841+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:12:13.274053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"427'9172","reported_seq":9342,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:38.487113+0000","last_change":"2024-09-13T14:51:48.316128+0000","last_active":"2024-09-13T14:54:38.487113+0000","last_peered":"2024-09-13T14:54:38.487113+0000","last_clean":"2024-09-13T14:54:38.487113+0000","last_became_active":"2024-09-13T14:51:34.954425+0000","last_became_peered":"2024-09-13T14:51:34.954425+0000","last_unstale":"2024-09-13T14:54:38.487113+0000","last_undegraded":"2024-09-13T14:54:38.487113+0000","last_fullsized":"2024-09-13T14:54:38.487113+0000","mapping_epoch":374,"log_start":"378'2661","ondisk_log_start":"378'2661","created":374,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'4030","last_scrub_stamp":"2024-09-13T14:51:48.315933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:51:48.315933+0000","objects_scrubbed":2195,"log_size":6511,"log_dups_size":0,"ondisk_log_size":6511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:56:58.693839+0000","scrub_duration":269,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5142,"num_write_kb":94400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"90'149","reported_seq":507,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.141493+0000","last_change":"2024-09-13T14:46:52.334715+0000","last_active":"2024-09-13T14:54:37.141493+0000","last_peered":"2024-09-13T14:54:37.141493+0000","last_clean":"2024-09-13T14:54:37.141493+0000","last_became_active":"2024-09-13T14:46:48.158685+0000","last_became_peered":"2024-09-13T14:46:48.158685+0000","last_unstale":"2024-09-13T14:54:37.141493+0000","last_undegraded":"2024-09-13T14:54:37.141493+0000","last_fullsized":"2024-09-13T14:54:37.141493+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:46:52.334301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:46:52.334301+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:46:03.579759+0000","scrub_duration":0.0050940330000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":766,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.107272+0000","last_change":"2024-09-13T14:47:47.861249+0000","last_active":"2024-09-13T14:54:37.107272+0000","last_peered":"2024-09-13T14:54:37.107272+0000","last_clean":"2024-09-13T14:54:37.107272+0000","last_became_active":"2024-09-13T14:47:47.860873+0000","last_became_peered":"2024-09-13T14:47:47.860873+0000","last_unstale":"2024-09-13T14:54:37.107272+0000","last_undegraded":"2024-09-13T14:54:37.107272+0000","last_fullsized":"2024-09-13T14:54:37.107272+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:37:50.709437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":197,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-13T14:54:39.153577+0000","last_change":"2024-09-13T14:47:47.870185+0000","last_active":"2024-09-13T14:54:39.153577+0000","last_peered":"2024-09-13T14:54:39.153577+0000","last_clean":"2024-09-13T14:54:39.153577+0000","last_became_active":"2024-09-13T14:47:47.869862+0000","last_became_peered":"2024-09-13T14:47:47.869862+0000","last_unstale":"2024-09-13T14:54:39.153577+0000","last_undegraded":"2024-09-13T14:54:39.153577+0000","last_fullsized":"2024-09-13T14:54:39.153577+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:07:02.812306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"427'9357","reported_seq":9527,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:38.550515+0000","last_change":"2024-09-13T14:51:49.726190+0000","last_active":"2024-09-13T14:54:38.550515+0000","last_peered":"2024-09-13T14:54:38.550515+0000","last_clean":"2024-09-13T14:54:38.550515+0000","last_became_active":"2024-09-13T14:51:34.965667+0000","last_became_peered":"2024-09-13T14:51:34.965667+0000","last_unstale":"2024-09-13T14:54:38.550515+0000","last_undegraded":"2024-09-13T14:54:38.550515+0000","last_fullsized":"2024-09-13T14:54:38.550515+0000","mapping_epoch":374,"log_start":"379'3028","ondisk_log_start":"379'3028","created":374,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"392'4161","last_scrub_stamp":"2024-09-13T14:51:49.725920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:51:49.725920+0000","objects_scrubbed":2275,"log_size":6329,"log_dups_size":0,"ondisk_log_size":6329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:28:01.500768+0000","scrub_duration":638,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":458752,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5206,"num_write_kb":94016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":610,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.104054+0000","last_change":"2024-09-13T14:47:31.684319+0000","last_active":"2024-09-13T14:54:37.104054+0000","last_peered":"2024-09-13T14:54:37.104054+0000","last_clean":"2024-09-13T14:54:37.104054+0000","last_became_active":"2024-09-13T14:47:29.415658+0000","last_became_peered":"2024-09-13T14:47:29.415658+0000","last_unstale":"2024-09-13T14:54:37.104054+0000","last_undegraded":"2024-09-13T14:54:37.104054+0000","last_fullsized":"2024-09-13T14:54:37.104054+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:31.681433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:31.681433+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:29:34.663513+0000","scrub_duration":0.0059601139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":738,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.144962+0000","last_change":"2024-09-13T14:46:04.780718+0000","last_active":"2024-09-13T14:54:37.144962+0000","last_peered":"2024-09-13T14:54:37.144962+0000","last_clean":"2024-09-13T14:54:37.144962+0000","last_became_active":"2024-09-13T14:46:04.780120+0000","last_became_peered":"2024-09-13T14:46:04.780120+0000","last_unstale":"2024-09-13T14:54:37.144962+0000","last_undegraded":"2024-09-13T14:54:37.144962+0000","last_fullsized":"2024-09-13T14:54:37.144962+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:57:14.475888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":694,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.107344+0000","last_change":"2024-09-13T14:37:49.160719+0000","last_active":"2024-09-13T14:54:37.107344+0000","last_peered":"2024-09-13T14:54:37.107344+0000","last_clean":"2024-09-13T14:54:37.107344+0000","last_became_active":"2024-09-13T14:37:49.160401+0000","last_became_peered":"2024-09-13T14:37:49.160401+0000","last_unstale":"2024-09-13T14:54:37.107344+0000","last_undegraded":"2024-09-13T14:54:37.107344+0000","last_fullsized":"2024-09-13T14:54:37.107344+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:21.737945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":277,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.150699+0000","last_change":"2024-09-13T14:37:48.172940+0000","last_active":"2024-09-13T14:54:37.150699+0000","last_peered":"2024-09-13T14:54:37.150699+0000","last_clean":"2024-09-13T14:54:37.150699+0000","last_became_active":"2024-09-13T14:37:48.172060+0000","last_became_peered":"2024-09-13T14:37:48.172060+0000","last_unstale":"2024-09-13T14:54:37.150699+0000","last_undegraded":"2024-09-13T14:54:37.150699+0000","last_fullsized":"2024-09-13T14:54:37.150699+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:07.480645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.b","version":"427'14948","reported_seq":15078,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-13T14:54:39.698681+0000","last_change":"2024-09-13T14:50:51.091679+0000","last_active":"2024-09-13T14:54:39.698681+0000","last_peered":"2024-09-13T14:54:39.698681+0000","last_clean":"2024-09-13T14:54:39.698681+0000","last_became_active":"2024-09-13T14:50:51.091340+0000","last_became_peered":"2024-09-13T14:50:51.091340+0000","last_unstale":"2024-09-13T14:54:39.698681+0000","last_undegraded":"2024-09-13T14:54:39.698681+0000","last_fullsized":"2024-09-13T14:54:39.698681+0000","mapping_epoch":374,"log_start":"406'7900","ondisk_log_start":"406'7900","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:50:49.763471+0000","objects_scrubbed":0,"log_size":7048,"log_dups_size":0,"ondisk_log_size":7048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:45:57.926434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14948,"num_write_kb":478336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"90'149","reported_seq":626,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-13T14:54:39.152800+0000","last_change":"2024-09-13T14:47:50.884721+0000","last_active":"2024-09-13T14:54:39.152800+0000","last_peered":"2024-09-13T14:54:39.152800+0000","last_clean":"2024-09-13T14:54:39.152800+0000","last_became_active":"2024-09-13T14:47:50.880185+0000","last_became_peered":"2024-09-13T14:47:50.880185+0000","last_unstale":"2024-09-13T14:54:39.152800+0000","last_undegraded":"2024-09-13T14:54:39.152800+0000","last_fullsized":"2024-09-13T14:54:39.152800+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:34.603316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:34.603316+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:38.321686+0000","scrub_duration":0.0057131120000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":680,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.104028+0000","last_change":"2024-09-13T14:47:47.892973+0000","last_active":"2024-09-13T14:54:37.104028+0000","last_peered":"2024-09-13T14:54:37.104028+0000","last_clean":"2024-09-13T14:54:37.104028+0000","last_became_active":"2024-09-13T14:47:47.866986+0000","last_became_peered":"2024-09-13T14:47:47.866986+0000","last_unstale":"2024-09-13T14:54:37.104028+0000","last_undegraded":"2024-09-13T14:54:37.104028+0000","last_fullsized":"2024-09-13T14:54:37.104028+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:07.353290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":385,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-13T14:54:39.698022+0000","last_change":"2024-09-13T14:38:03.525682+0000","last_active":"2024-09-13T14:54:39.698022+0000","last_peered":"2024-09-13T14:54:39.698022+0000","last_clean":"2024-09-13T14:54:39.698022+0000","last_became_active":"2024-09-13T14:38:03.525406+0000","last_became_peered":"2024-09-13T14:38:03.525406+0000","last_unstale":"2024-09-13T14:54:39.698022+0000","last_undegraded":"2024-09-13T14:54:39.698022+0000","last_fullsized":"2024-09-13T14:54:39.698022+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:41:28.970300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":406,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-13T14:54:39.698609+0000","last_change":"2024-09-13T14:47:47.873233+0000","last_active":"2024-09-13T14:54:39.698609+0000","last_peered":"2024-09-13T14:54:39.698609+0000","last_clean":"2024-09-13T14:54:39.698609+0000","last_became_active":"2024-09-13T14:47:47.872788+0000","last_became_peered":"2024-09-13T14:47:47.872788+0000","last_unstale":"2024-09-13T14:54:39.698609+0000","last_undegraded":"2024-09-13T14:54:39.698609+0000","last_fullsized":"2024-09-13T14:54:39.698609+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:52.409108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"427'15048","reported_seq":15112,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:38.796409+0000","last_change":"2024-09-13T14:50:50.782599+0000","last_active":"2024-09-13T14:54:38.796409+0000","last_peered":"2024-09-13T14:54:38.796409+0000","last_clean":"2024-09-13T14:54:38.796409+0000","last_became_active":"2024-09-13T14:50:50.782273+0000","last_became_peered":"2024-09-13T14:50:50.782273+0000","last_unstale":"2024-09-13T14:54:38.796409+0000","last_undegraded":"2024-09-13T14:54:38.796409+0000","last_fullsized":"2024-09-13T14:54:38.796409+0000","mapping_epoch":374,"log_start":"406'8000","ondisk_log_start":"406'8000","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:50:49.763471+0000","objects_scrubbed":0,"log_size":7048,"log_dups_size":0,"ondisk_log_size":7048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:27:54.626945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15048,"num_write_kb":481536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"90'149","reported_seq":626,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.104059+0000","last_change":"2024-09-13T14:47:29.629488+0000","last_active":"2024-09-13T14:54:37.104059+0000","last_peered":"2024-09-13T14:54:37.104059+0000","last_clean":"2024-09-13T14:54:37.104059+0000","last_became_active":"2024-09-13T14:47:29.403279+0000","last_became_peered":"2024-09-13T14:47:29.403279+0000","last_unstale":"2024-09-13T14:54:37.104059+0000","last_undegraded":"2024-09-13T14:54:37.104059+0000","last_fullsized":"2024-09-13T14:54:37.104059+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:29.629342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:29.629342+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:14:55.249896+0000","scrub_duration":0.0099290990000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":680,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-13T14:54:39.699526+0000","last_change":"2024-09-13T14:46:04.782955+0000","last_active":"2024-09-13T14:54:39.699526+0000","last_peered":"2024-09-13T14:54:39.699526+0000","last_clean":"2024-09-13T14:54:39.699526+0000","last_became_active":"2024-09-13T14:46:04.782250+0000","last_became_peered":"2024-09-13T14:46:04.782250+0000","last_unstale":"2024-09-13T14:54:39.699526+0000","last_undegraded":"2024-09-13T14:54:39.699526+0000","last_fullsized":"2024-09-13T14:54:39.699526+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:24.596922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":197,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-13T14:54:39.152527+0000","last_change":"2024-09-13T14:47:47.866684+0000","last_active":"2024-09-13T14:54:39.152527+0000","last_peered":"2024-09-13T14:54:39.152527+0000","last_clean":"2024-09-13T14:54:39.152527+0000","last_became_active":"2024-09-13T14:47:47.866400+0000","last_became_peered":"2024-09-13T14:47:47.866400+0000","last_unstale":"2024-09-13T14:54:39.152527+0000","last_undegraded":"2024-09-13T14:54:39.152527+0000","last_fullsized":"2024-09-13T14:54:39.152527+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:08:34.289658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":197,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-13T14:54:39.153619+0000","last_change":"2024-09-13T14:47:47.870218+0000","last_active":"2024-09-13T14:54:39.153619+0000","last_peered":"2024-09-13T14:54:39.153619+0000","last_clean":"2024-09-13T14:54:39.153619+0000","last_became_active":"2024-09-13T14:47:47.869907+0000","last_became_peered":"2024-09-13T14:47:47.869907+0000","last_unstale":"2024-09-13T14:54:39.153619+0000","last_undegraded":"2024-09-13T14:54:39.153619+0000","last_fullsized":"2024-09-13T14:54:39.153619+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:41:49.141441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"427'9569","reported_seq":9746,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-13T14:54:39.697251+0000","last_change":"2024-09-13T14:51:46.774586+0000","last_active":"2024-09-13T14:54:39.697251+0000","last_peered":"2024-09-13T14:54:39.697251+0000","last_clean":"2024-09-13T14:54:39.697251+0000","last_became_active":"2024-09-13T14:51:34.952323+0000","last_became_peered":"2024-09-13T14:51:34.952323+0000","last_unstale":"2024-09-13T14:54:39.697251+0000","last_undegraded":"2024-09-13T14:54:39.697251+0000","last_fullsized":"2024-09-13T14:54:39.697251+0000","mapping_epoch":374,"log_start":"378'1321","ondisk_log_start":"378'1321","created":374,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'4242","last_scrub_stamp":"2024-09-13T14:51:46.774489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:51:46.774489+0000","objects_scrubbed":2307,"log_size":8248,"log_dups_size":0,"ondisk_log_size":8248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:32:50.482249+0000","scrub_duration":408,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5331,"num_write_kb":96768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"90'149","reported_seq":528,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.141927+0000","last_change":"2024-09-13T14:47:32.275753+0000","last_active":"2024-09-13T14:54:37.141927+0000","last_peered":"2024-09-13T14:54:37.141927+0000","last_clean":"2024-09-13T14:54:37.141927+0000","last_became_active":"2024-09-13T14:47:29.401066+0000","last_became_peered":"2024-09-13T14:47:29.401066+0000","last_unstale":"2024-09-13T14:54:37.141927+0000","last_undegraded":"2024-09-13T14:54:37.141927+0000","last_fullsized":"2024-09-13T14:54:37.141927+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:32.275562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:32.275562+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:43:41.056775+0000","scrub_duration":0.0054674260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":1183,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.141573+0000","last_change":"2024-09-13T14:47:50.892842+0000","last_active":"2024-09-13T14:54:37.141573+0000","last_peered":"2024-09-13T14:54:37.141573+0000","last_clean":"2024-09-13T14:54:37.141573+0000","last_became_active":"2024-09-13T14:47:50.892237+0000","last_became_peered":"2024-09-13T14:47:50.892237+0000","last_unstale":"2024-09-13T14:54:37.141573+0000","last_undegraded":"2024-09-13T14:54:37.141573+0000","last_fullsized":"2024-09-13T14:54:37.141573+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:20:05.796762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":8388646,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":439,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.150746+0000","last_change":"2024-09-13T14:38:03.528133+0000","last_active":"2024-09-13T14:54:37.150746+0000","last_peered":"2024-09-13T14:54:37.150746+0000","last_clean":"2024-09-13T14:54:37.150746+0000","last_became_active":"2024-09-13T14:38:03.527829+0000","last_became_peered":"2024-09-13T14:38:03.527829+0000","last_unstale":"2024-09-13T14:54:37.150746+0000","last_undegraded":"2024-09-13T14:54:37.150746+0000","last_fullsized":"2024-09-13T14:54:37.150746+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:48.254169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":197,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-13T14:54:39.153109+0000","last_change":"2024-09-13T14:47:47.877272+0000","last_active":"2024-09-13T14:54:39.153109+0000","last_peered":"2024-09-13T14:54:39.153109+0000","last_clean":"2024-09-13T14:54:39.153109+0000","last_became_active":"2024-09-13T14:47:47.877003+0000","last_became_peered":"2024-09-13T14:47:47.877003+0000","last_unstale":"2024-09-13T14:54:39.153109+0000","last_undegraded":"2024-09-13T14:54:39.153109+0000","last_fullsized":"2024-09-13T14:54:39.153109+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:47:23.172452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.f","version":"427'15468","reported_seq":15532,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:38.777354+0000","last_change":"2024-09-13T14:50:50.783992+0000","last_active":"2024-09-13T14:54:38.777354+0000","last_peered":"2024-09-13T14:54:38.777354+0000","last_clean":"2024-09-13T14:54:38.777354+0000","last_became_active":"2024-09-13T14:50:50.783687+0000","last_became_peered":"2024-09-13T14:50:50.783687+0000","last_unstale":"2024-09-13T14:54:38.777354+0000","last_undegraded":"2024-09-13T14:54:38.777354+0000","last_fullsized":"2024-09-13T14:54:38.777354+0000","mapping_epoch":374,"log_start":"406'8400","ondisk_log_start":"406'8400","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:50:49.763471+0000","objects_scrubbed":0,"log_size":7068,"log_dups_size":0,"ondisk_log_size":7068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:30.412734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15468,"num_write_kb":494976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"90'149","reported_seq":627,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.108108+0000","last_change":"2024-09-13T14:47:33.067298+0000","last_active":"2024-09-13T14:54:37.108108+0000","last_peered":"2024-09-13T14:54:37.108108+0000","last_clean":"2024-09-13T14:54:37.108108+0000","last_became_active":"2024-09-13T14:47:29.419416+0000","last_became_peered":"2024-09-13T14:47:29.419416+0000","last_unstale":"2024-09-13T14:54:37.108108+0000","last_undegraded":"2024-09-13T14:54:37.108108+0000","last_fullsized":"2024-09-13T14:54:37.108108+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:33.066954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:33.066954+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:05:42.808060+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":252,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.141686+0000","last_change":"2024-09-13T14:46:01.763906+0000","last_active":"2024-09-13T14:54:37.141686+0000","last_peered":"2024-09-13T14:54:37.141686+0000","last_clean":"2024-09-13T14:54:37.141686+0000","last_became_active":"2024-09-13T14:46:01.763542+0000","last_became_peered":"2024-09-13T14:46:01.763542+0000","last_unstale":"2024-09-13T14:54:37.141686+0000","last_undegraded":"2024-09-13T14:54:37.141686+0000","last_fullsized":"2024-09-13T14:54:37.141686+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:09:36.652767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":1485,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-13T14:54:39.153190+0000","last_change":"2024-09-13T14:47:50.883365+0000","last_active":"2024-09-13T14:54:39.153190+0000","last_peered":"2024-09-13T14:54:39.153190+0000","last_clean":"2024-09-13T14:54:39.153190+0000","last_became_active":"2024-09-13T14:47:50.882609+0000","last_became_peered":"2024-09-13T14:47:50.882609+0000","last_unstale":"2024-09-13T14:54:39.153190+0000","last_undegraded":"2024-09-13T14:54:39.153190+0000","last_fullsized":"2024-09-13T14:54:39.153190+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:12:17.961575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":197,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-13T14:54:39.152990+0000","last_change":"2024-09-13T14:47:47.866979+0000","last_active":"2024-09-13T14:54:39.152990+0000","last_peered":"2024-09-13T14:54:39.152990+0000","last_clean":"2024-09-13T14:54:39.152990+0000","last_became_active":"2024-09-13T14:47:47.866634+0000","last_became_peered":"2024-09-13T14:47:47.866634+0000","last_unstale":"2024-09-13T14:54:39.152990+0000","last_undegraded":"2024-09-13T14:54:39.152990+0000","last_fullsized":"2024-09-13T14:54:39.152990+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:47:09.696139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.e","version":"427'15018","reported_seq":15082,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:38.813737+0000","last_change":"2024-09-13T14:50:50.782813+0000","last_active":"2024-09-13T14:54:38.813737+0000","last_peered":"2024-09-13T14:54:38.813737+0000","last_clean":"2024-09-13T14:54:38.813737+0000","last_became_active":"2024-09-13T14:50:50.782515+0000","last_became_peered":"2024-09-13T14:50:50.782515+0000","last_unstale":"2024-09-13T14:54:38.813737+0000","last_undegraded":"2024-09-13T14:54:38.813737+0000","last_fullsized":"2024-09-13T14:54:38.813737+0000","mapping_epoch":374,"log_start":"410'9300","ondisk_log_start":"410'9300","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:50:49.763471+0000","objects_scrubbed":0,"log_size":5718,"log_dups_size":0,"ondisk_log_size":5718,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:20.624059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15018,"num_write_kb":480576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"90'149","reported_seq":635,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-13T14:54:39.698403+0000","last_change":"2024-09-13T14:47:29.556225+0000","last_active":"2024-09-13T14:54:39.698403+0000","last_peered":"2024-09-13T14:54:39.698403+0000","last_clean":"2024-09-13T14:54:39.698403+0000","last_became_active":"2024-09-13T14:47:29.408774+0000","last_became_peered":"2024-09-13T14:47:29.408774+0000","last_unstale":"2024-09-13T14:54:39.698403+0000","last_undegraded":"2024-09-13T14:54:39.698403+0000","last_fullsized":"2024-09-13T14:54:39.698403+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:29.556149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:29.556149+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:35.691178+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":197,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-13T14:54:39.153279+0000","last_change":"2024-09-13T14:47:47.876562+0000","last_active":"2024-09-13T14:54:39.153279+0000","last_peered":"2024-09-13T14:54:39.153279+0000","last_clean":"2024-09-13T14:54:39.153279+0000","last_became_active":"2024-09-13T14:47:47.876297+0000","last_became_peered":"2024-09-13T14:47:47.876297+0000","last_unstale":"2024-09-13T14:54:39.153279+0000","last_undegraded":"2024-09-13T14:54:39.153279+0000","last_fullsized":"2024-09-13T14:54:39.153279+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:03:27.761394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":955,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.104688+0000","last_change":"2024-09-13T14:38:04.596192+0000","last_active":"2024-09-13T14:54:37.104688+0000","last_peered":"2024-09-13T14:54:37.104688+0000","last_clean":"2024-09-13T14:54:37.104688+0000","last_became_active":"2024-09-13T14:38:04.593299+0000","last_became_peered":"2024-09-13T14:38:04.593299+0000","last_unstale":"2024-09-13T14:54:37.104688+0000","last_undegraded":"2024-09-13T14:54:37.104688+0000","last_fullsized":"2024-09-13T14:54:37.104688+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:34.241410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":385,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-13T14:54:39.698878+0000","last_change":"2024-09-13T14:38:03.533120+0000","last_active":"2024-09-13T14:54:39.698878+0000","last_peered":"2024-09-13T14:54:39.698878+0000","last_clean":"2024-09-13T14:54:39.698878+0000","last_became_active":"2024-09-13T14:38:03.532829+0000","last_became_peered":"2024-09-13T14:38:03.532829+0000","last_unstale":"2024-09-13T14:54:39.698878+0000","last_undegraded":"2024-09-13T14:54:39.698878+0000","last_fullsized":"2024-09-13T14:54:39.698878+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:24.514661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.d","version":"427'15206","reported_seq":15271,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:38.824167+0000","last_change":"2024-09-13T14:50:50.782589+0000","last_active":"2024-09-13T14:54:38.824167+0000","last_peered":"2024-09-13T14:54:38.824167+0000","last_clean":"2024-09-13T14:54:38.824167+0000","last_became_active":"2024-09-13T14:50:50.782292+0000","last_became_peered":"2024-09-13T14:50:50.782292+0000","last_unstale":"2024-09-13T14:54:38.824167+0000","last_undegraded":"2024-09-13T14:54:38.824167+0000","last_fullsized":"2024-09-13T14:54:38.824167+0000","mapping_epoch":374,"log_start":"410'9500","ondisk_log_start":"410'9500","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:50:49.763471+0000","objects_scrubbed":0,"log_size":5706,"log_dups_size":0,"ondisk_log_size":5706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:01:30.753189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15206,"num_write_kb":486529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"90'149","reported_seq":528,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.150832+0000","last_change":"2024-09-13T14:47:30.528588+0000","last_active":"2024-09-13T14:54:37.150832+0000","last_peered":"2024-09-13T14:54:37.150832+0000","last_clean":"2024-09-13T14:54:37.150832+0000","last_became_active":"2024-09-13T14:47:29.413621+0000","last_became_peered":"2024-09-13T14:47:29.413621+0000","last_unstale":"2024-09-13T14:54:37.150832+0000","last_undegraded":"2024-09-13T14:54:37.150832+0000","last_fullsized":"2024-09-13T14:54:37.150832+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:30.528112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:30.528112+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:30.462009+0000","scrub_duration":0.012751485,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":785,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.108241+0000","last_change":"2024-09-13T14:38:53.938227+0000","last_active":"2024-09-13T14:54:37.108241+0000","last_peered":"2024-09-13T14:54:37.108241+0000","last_clean":"2024-09-13T14:54:37.108241+0000","last_became_active":"2024-09-13T14:38:53.902867+0000","last_became_peered":"2024-09-13T14:38:53.902867+0000","last_unstale":"2024-09-13T14:54:37.108241+0000","last_undegraded":"2024-09-13T14:54:37.108241+0000","last_fullsized":"2024-09-13T14:54:37.108241+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:16.831756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":667,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-13T14:54:39.153561+0000","last_change":"2024-09-13T14:47:50.884706+0000","last_active":"2024-09-13T14:54:39.153561+0000","last_peered":"2024-09-13T14:54:39.153561+0000","last_clean":"2024-09-13T14:54:39.153561+0000","last_became_active":"2024-09-13T14:47:50.879750+0000","last_became_peered":"2024-09-13T14:47:50.879750+0000","last_unstale":"2024-09-13T14:54:39.153561+0000","last_undegraded":"2024-09-13T14:54:39.153561+0000","last_fullsized":"2024-09-13T14:54:39.153561+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:34:32.941266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":767,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.107164+0000","last_change":"2024-09-13T14:38:53.900107+0000","last_active":"2024-09-13T14:54:37.107164+0000","last_peered":"2024-09-13T14:54:37.107164+0000","last_clean":"2024-09-13T14:54:37.107164+0000","last_became_active":"2024-09-13T14:38:53.899556+0000","last_became_peered":"2024-09-13T14:38:53.899556+0000","last_unstale":"2024-09-13T14:54:37.107164+0000","last_undegraded":"2024-09-13T14:54:37.107164+0000","last_fullsized":"2024-09-13T14:54:37.107164+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:15:47.918163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"427'15059","reported_seq":15186,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:38.525939+0000","last_change":"2024-09-13T14:50:50.776299+0000","last_active":"2024-09-13T14:54:38.525939+0000","last_peered":"2024-09-13T14:54:38.525939+0000","last_clean":"2024-09-13T14:54:38.525939+0000","last_became_active":"2024-09-13T14:50:50.776003+0000","last_became_peered":"2024-09-13T14:50:50.776003+0000","last_unstale":"2024-09-13T14:54:38.525939+0000","last_undegraded":"2024-09-13T14:54:38.525939+0000","last_fullsized":"2024-09-13T14:54:38.525939+0000","mapping_epoch":374,"log_start":"410'8975","ondisk_log_start":"410'8975","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:50:49.763471+0000","objects_scrubbed":0,"log_size":6084,"log_dups_size":0,"ondisk_log_size":6084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:26.038913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":983040,"num_objects":15,"num_object_clones":0,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15059,"num_write_kb":482368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"90'149","reported_seq":624,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.107763+0000","last_change":"2024-09-13T14:47:29.999643+0000","last_active":"2024-09-13T14:54:37.107763+0000","last_peered":"2024-09-13T14:54:37.107763+0000","last_clean":"2024-09-13T14:54:37.107763+0000","last_became_active":"2024-09-13T14:47:29.410951+0000","last_became_peered":"2024-09-13T14:47:29.410951+0000","last_unstale":"2024-09-13T14:54:37.107763+0000","last_undegraded":"2024-09-13T14:54:37.107763+0000","last_fullsized":"2024-09-13T14:54:37.107763+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:29.999519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:29.999519+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:04:54.887884+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":411,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-13T14:54:39.698504+0000","last_change":"2024-09-13T14:47:47.883475+0000","last_active":"2024-09-13T14:54:39.698504+0000","last_peered":"2024-09-13T14:54:39.698504+0000","last_clean":"2024-09-13T14:54:39.698504+0000","last_became_active":"2024-09-13T14:47:47.871525+0000","last_became_peered":"2024-09-13T14:47:47.871525+0000","last_unstale":"2024-09-13T14:54:39.698504+0000","last_undegraded":"2024-09-13T14:54:39.698504+0000","last_fullsized":"2024-09-13T14:54:39.698504+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:33.903432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":233,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.104709+0000","last_change":"2024-09-13T14:46:01.781508+0000","last_active":"2024-09-13T14:54:37.104709+0000","last_peered":"2024-09-13T14:54:37.104709+0000","last_clean":"2024-09-13T14:54:37.104709+0000","last_became_active":"2024-09-13T14:46:01.767096+0000","last_became_peered":"2024-09-13T14:46:01.767096+0000","last_unstale":"2024-09-13T14:54:37.104709+0000","last_undegraded":"2024-09-13T14:54:37.104709+0000","last_fullsized":"2024-09-13T14:54:37.104709+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:20:01.119189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":423,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.150174+0000","last_change":"2024-09-13T14:37:49.172218+0000","last_active":"2024-09-13T14:54:37.150174+0000","last_peered":"2024-09-13T14:54:37.150174+0000","last_clean":"2024-09-13T14:54:37.150174+0000","last_became_active":"2024-09-13T14:37:49.171718+0000","last_became_peered":"2024-09-13T14:37:49.171718+0000","last_unstale":"2024-09-13T14:54:37.150174+0000","last_undegraded":"2024-09-13T14:54:37.150174+0000","last_fullsized":"2024-09-13T14:54:37.150174+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:46:54.626521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":454,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.150608+0000","last_change":"2024-09-13T14:47:47.868409+0000","last_active":"2024-09-13T14:54:37.150608+0000","last_peered":"2024-09-13T14:54:37.150608+0000","last_clean":"2024-09-13T14:54:37.150608+0000","last_became_active":"2024-09-13T14:47:47.867712+0000","last_became_peered":"2024-09-13T14:47:47.867712+0000","last_unstale":"2024-09-13T14:54:37.150608+0000","last_undegraded":"2024-09-13T14:54:37.150608+0000","last_fullsized":"2024-09-13T14:54:37.150608+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:57:14.647493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":425,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.152079+0000","last_change":"2024-09-13T14:37:48.128447+0000","last_active":"2024-09-13T14:54:37.152079+0000","last_peered":"2024-09-13T14:54:37.152079+0000","last_clean":"2024-09-13T14:54:37.152079+0000","last_became_active":"2024-09-13T14:37:48.128143+0000","last_became_peered":"2024-09-13T14:37:48.128143+0000","last_unstale":"2024-09-13T14:54:37.152079+0000","last_undegraded":"2024-09-13T14:54:37.152079+0000","last_fullsized":"2024-09-13T14:54:37.152079+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:22.204039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":196,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-13T14:54:39.152755+0000","last_change":"2024-09-13T14:47:47.867935+0000","last_active":"2024-09-13T14:54:39.152755+0000","last_peered":"2024-09-13T14:54:39.152755+0000","last_clean":"2024-09-13T14:54:39.152755+0000","last_became_active":"2024-09-13T14:47:47.867616+0000","last_became_peered":"2024-09-13T14:47:47.867616+0000","last_unstale":"2024-09-13T14:54:39.152755+0000","last_undegraded":"2024-09-13T14:54:39.152755+0000","last_fullsized":"2024-09-13T14:54:39.152755+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:52:38.619916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":197,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.104502+0000","last_change":"2024-09-13T14:38:53.954281+0000","last_active":"2024-09-13T14:54:37.104502+0000","last_peered":"2024-09-13T14:54:37.104502+0000","last_clean":"2024-09-13T14:54:37.104502+0000","last_became_active":"2024-09-13T14:38:53.900912+0000","last_became_peered":"2024-09-13T14:38:53.900912+0000","last_unstale":"2024-09-13T14:54:37.104502+0000","last_undegraded":"2024-09-13T14:54:37.104502+0000","last_fullsized":"2024-09-13T14:54:37.104502+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:47.876275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"90'149","reported_seq":629,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-13T14:54:39.697474+0000","last_change":"2024-09-13T14:47:38.468095+0000","last_active":"2024-09-13T14:54:39.697474+0000","last_peered":"2024-09-13T14:54:39.697474+0000","last_clean":"2024-09-13T14:54:39.697474+0000","last_became_active":"2024-09-13T14:47:38.467526+0000","last_became_peered":"2024-09-13T14:47:38.467526+0000","last_unstale":"2024-09-13T14:54:39.697474+0000","last_undegraded":"2024-09-13T14:54:39.697474+0000","last_fullsized":"2024-09-13T14:54:39.697474+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:31.013128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:31.013128+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:17:20.508142+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":258,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.142035+0000","last_change":"2024-09-13T14:47:47.861842+0000","last_active":"2024-09-13T14:54:37.142035+0000","last_peered":"2024-09-13T14:54:37.142035+0000","last_clean":"2024-09-13T14:54:37.142035+0000","last_became_active":"2024-09-13T14:47:47.861492+0000","last_became_peered":"2024-09-13T14:47:47.861492+0000","last_unstale":"2024-09-13T14:54:37.142035+0000","last_undegraded":"2024-09-13T14:54:37.142035+0000","last_fullsized":"2024-09-13T14:54:37.142035+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:17.279533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":708,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.107293+0000","last_change":"2024-09-13T14:38:53.899535+0000","last_active":"2024-09-13T14:54:37.107293+0000","last_peered":"2024-09-13T14:54:37.107293+0000","last_clean":"2024-09-13T14:54:37.107293+0000","last_became_active":"2024-09-13T14:38:53.899183+0000","last_became_peered":"2024-09-13T14:38:53.899183+0000","last_unstale":"2024-09-13T14:54:37.107293+0000","last_undegraded":"2024-09-13T14:54:37.107293+0000","last_fullsized":"2024-09-13T14:54:37.107293+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:33.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":384,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-13T14:54:39.698716+0000","last_change":"2024-09-13T14:38:03.543426+0000","last_active":"2024-09-13T14:54:39.698716+0000","last_peered":"2024-09-13T14:54:39.698716+0000","last_clean":"2024-09-13T14:54:39.698716+0000","last_became_active":"2024-09-13T14:38:03.542661+0000","last_became_peered":"2024-09-13T14:38:03.542661+0000","last_unstale":"2024-09-13T14:54:39.698716+0000","last_undegraded":"2024-09-13T14:54:39.698716+0000","last_fullsized":"2024-09-13T14:54:39.698716+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:54.050127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":744,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.107066+0000","last_change":"2024-09-13T14:47:47.868624+0000","last_active":"2024-09-13T14:54:37.107066+0000","last_peered":"2024-09-13T14:54:37.107066+0000","last_clean":"2024-09-13T14:54:37.107066+0000","last_became_active":"2024-09-13T14:47:47.868031+0000","last_became_peered":"2024-09-13T14:47:47.868031+0000","last_unstale":"2024-09-13T14:54:37.107066+0000","last_undegraded":"2024-09-13T14:54:37.107066+0000","last_fullsized":"2024-09-13T14:54:37.107066+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:05:25.365023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.12","version":"427'9271","reported_seq":16095,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:38.810297+0000","last_change":"2024-09-13T14:53:25.946730+0000","last_active":"2024-09-13T14:54:38.810297+0000","last_peered":"2024-09-13T14:54:38.810297+0000","last_clean":"2024-09-13T14:54:38.810297+0000","last_became_active":"2024-09-13T14:52:47.881391+0000","last_became_peered":"2024-09-13T14:52:47.881391+0000","last_unstale":"2024-09-13T14:54:38.810297+0000","last_undegraded":"2024-09-13T14:54:38.810297+0000","last_fullsized":"2024-09-13T14:54:38.810297+0000","mapping_epoch":407,"log_start":"378'1004","ondisk_log_start":"378'1004","created":381,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":5,"last_scrub":"412'7108","last_scrub_stamp":"2024-09-13T14:53:25.946507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:53:25.946507+0000","objects_scrubbed":2170,"log_size":8267,"log_dups_size":0,"ondisk_log_size":8267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:13:57.596649+0000","scrub_duration":0.56883067700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2170,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":271,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.142343+0000","last_change":"2024-09-13T14:47:47.859825+0000","last_active":"2024-09-13T14:54:37.142343+0000","last_peered":"2024-09-13T14:54:37.142343+0000","last_clean":"2024-09-13T14:54:37.142343+0000","last_became_active":"2024-09-13T14:47:47.859069+0000","last_became_peered":"2024-09-13T14:47:47.859069+0000","last_unstale":"2024-09-13T14:54:37.142343+0000","last_undegraded":"2024-09-13T14:54:37.142343+0000","last_fullsized":"2024-09-13T14:54:37.142343+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:18:36.452321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":776,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.107237+0000","last_change":"2024-09-13T14:47:47.868945+0000","last_active":"2024-09-13T14:54:37.107237+0000","last_peered":"2024-09-13T14:54:37.107237+0000","last_clean":"2024-09-13T14:54:37.107237+0000","last_became_active":"2024-09-13T14:47:47.868296+0000","last_became_peered":"2024-09-13T14:47:47.868296+0000","last_unstale":"2024-09-13T14:54:37.107237+0000","last_undegraded":"2024-09-13T14:54:37.107237+0000","last_fullsized":"2024-09-13T14:54:37.107237+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:22:24.778693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"427'9143","reported_seq":15643,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:38.810215+0000","last_change":"2024-09-13T14:54:23.022769+0000","last_active":"2024-09-13T14:54:38.810215+0000","last_peered":"2024-09-13T14:54:38.810215+0000","last_clean":"2024-09-13T14:54:38.810215+0000","last_became_active":"2024-09-13T14:52:21.079272+0000","last_became_peered":"2024-09-13T14:52:21.079272+0000","last_unstale":"2024-09-13T14:54:38.810215+0000","last_undegraded":"2024-09-13T14:54:38.810215+0000","last_fullsized":"2024-09-13T14:54:38.810215+0000","mapping_epoch":396,"log_start":"379'3369","ondisk_log_start":"379'3369","created":381,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":5,"last_scrub":"425'8665","last_scrub_stamp":"2024-09-13T14:54:23.022501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:54:23.022501+0000","objects_scrubbed":482,"log_size":5774,"log_dups_size":0,"ondisk_log_size":5774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:00.219372+0000","scrub_duration":0.160633102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":482,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":393,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-13T14:54:39.698478+0000","last_change":"2024-09-13T14:38:53.892948+0000","last_active":"2024-09-13T14:54:39.698478+0000","last_peered":"2024-09-13T14:54:39.698478+0000","last_clean":"2024-09-13T14:54:39.698478+0000","last_became_active":"2024-09-13T14:38:53.892665+0000","last_became_peered":"2024-09-13T14:38:53.892665+0000","last_unstale":"2024-09-13T14:54:39.698478+0000","last_undegraded":"2024-09-13T14:54:39.698478+0000","last_fullsized":"2024-09-13T14:54:39.698478+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:08.292359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":196,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-13T14:54:39.152391+0000","last_change":"2024-09-13T14:47:47.871164+0000","last_active":"2024-09-13T14:54:39.152391+0000","last_peered":"2024-09-13T14:54:39.152391+0000","last_clean":"2024-09-13T14:54:39.152391+0000","last_became_active":"2024-09-13T14:47:47.870862+0000","last_became_peered":"2024-09-13T14:47:47.870862+0000","last_unstale":"2024-09-13T14:54:39.152391+0000","last_undegraded":"2024-09-13T14:54:39.152391+0000","last_fullsized":"2024-09-13T14:54:39.152391+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:41:34.392081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"427'9467","reported_seq":18394,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-13T14:54:39.697837+0000","last_change":"2024-09-13T14:54:21.112968+0000","last_active":"2024-09-13T14:54:39.697837+0000","last_peered":"2024-09-13T14:54:39.697837+0000","last_clean":"2024-09-13T14:54:39.697837+0000","last_became_active":"2024-09-13T14:52:20.077449+0000","last_became_peered":"2024-09-13T14:52:20.077449+0000","last_unstale":"2024-09-13T14:54:39.697837+0000","last_undegraded":"2024-09-13T14:54:39.697837+0000","last_fullsized":"2024-09-13T14:54:39.697837+0000","mapping_epoch":395,"log_start":"378'2471","ondisk_log_start":"378'2471","created":381,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":5,"last_scrub":"425'8899","last_scrub_stamp":"2024-09-13T14:54:21.112680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:54:21.112680+0000","objects_scrubbed":575,"log_size":6996,"log_dups_size":0,"ondisk_log_size":6996,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:57:25.961389+0000","scrub_duration":0.15945490500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":575,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":764,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.107887+0000","last_change":"2024-09-13T14:50:27.490554+0000","last_active":"2024-09-13T14:54:37.107887+0000","last_peered":"2024-09-13T14:54:37.107887+0000","last_clean":"2024-09-13T14:54:37.107887+0000","last_became_active":"2024-09-13T14:50:27.490211+0000","last_became_peered":"2024-09-13T14:50:27.490211+0000","last_unstale":"2024-09-13T14:54:37.107887+0000","last_undegraded":"2024-09-13T14:54:37.107887+0000","last_fullsized":"2024-09-13T14:54:37.107887+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:57:01.947741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":507,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.150448+0000","last_change":"2024-09-13T14:47:47.868246+0000","last_active":"2024-09-13T14:54:37.150448+0000","last_peered":"2024-09-13T14:54:37.150448+0000","last_clean":"2024-09-13T14:54:37.150448+0000","last_became_active":"2024-09-13T14:47:47.867817+0000","last_became_peered":"2024-09-13T14:47:47.867817+0000","last_unstale":"2024-09-13T14:54:37.150448+0000","last_undegraded":"2024-09-13T14:54:37.150448+0000","last_fullsized":"2024-09-13T14:54:37.150448+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:07.612451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.11","version":"427'12090","reported_seq":12483,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:38.803444+0000","last_change":"2024-09-13T14:52:29.869039+0000","last_active":"2024-09-13T14:54:38.803444+0000","last_peered":"2024-09-13T14:54:38.803444+0000","last_clean":"2024-09-13T14:54:38.803444+0000","last_became_active":"2024-09-13T14:51:43.012842+0000","last_became_peered":"2024-09-13T14:51:43.012842+0000","last_unstale":"2024-09-13T14:54:38.803444+0000","last_undegraded":"2024-09-13T14:54:38.803444+0000","last_fullsized":"2024-09-13T14:54:38.803444+0000","mapping_epoch":386,"log_start":"399'6356","ondisk_log_start":"399'6356","created":381,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":5,"last_scrub":"405'8081","last_scrub_stamp":"2024-09-13T14:52:29.868545+0000","last_deep_scrub":"405'8081","last_deep_scrub_stamp":"2024-09-13T14:52:29.868545+0000","last_clean_scrub_stamp":"2024-09-13T14:52:29.868545+0000","objects_scrubbed":3578,"log_size":5734,"log_dups_size":0,"ondisk_log_size":5734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":9,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:54:13.210442+0000","scrub_duration":8.1822236159999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4048,"num_write_kb":15040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":278,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-13T14:54:39.697714+0000","last_change":"2024-09-13T14:50:27.510133+0000","last_active":"2024-09-13T14:54:39.697714+0000","last_peered":"2024-09-13T14:54:39.697714+0000","last_clean":"2024-09-13T14:54:39.697714+0000","last_became_active":"2024-09-13T14:50:27.508774+0000","last_became_peered":"2024-09-13T14:50:27.508774+0000","last_unstale":"2024-09-13T14:54:39.697714+0000","last_undegraded":"2024-09-13T14:54:39.697714+0000","last_fullsized":"2024-09-13T14:54:39.697714+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:31:04.531746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":384,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-13T14:54:39.698479+0000","last_change":"2024-09-13T14:38:03.527021+0000","last_active":"2024-09-13T14:54:39.698479+0000","last_peered":"2024-09-13T14:54:39.698479+0000","last_clean":"2024-09-13T14:54:39.698479+0000","last_became_active":"2024-09-13T14:38:03.526698+0000","last_became_peered":"2024-09-13T14:38:03.526698+0000","last_unstale":"2024-09-13T14:54:39.698479+0000","last_undegraded":"2024-09-13T14:54:39.698479+0000","last_fullsized":"2024-09-13T14:54:39.698479+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:01:35.734086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.16","version":"427'9544","reported_seq":16778,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:38.770589+0000","last_change":"2024-09-13T14:54:22.118979+0000","last_active":"2024-09-13T14:54:38.770589+0000","last_peered":"2024-09-13T14:54:38.770589+0000","last_clean":"2024-09-13T14:54:38.770589+0000","last_became_active":"2024-09-13T14:53:46.817869+0000","last_became_peered":"2024-09-13T14:53:46.817869+0000","last_unstale":"2024-09-13T14:54:38.770589+0000","last_undegraded":"2024-09-13T14:54:38.770589+0000","last_fullsized":"2024-09-13T14:54:38.770589+0000","mapping_epoch":416,"log_start":"378'2554","ondisk_log_start":"378'2554","created":381,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":5,"last_scrub":"425'9006","last_scrub_stamp":"2024-09-13T14:54:22.118689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:54:22.118689+0000","objects_scrubbed":538,"log_size":6990,"log_dups_size":0,"ondisk_log_size":6990,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:48:22.280753+0000","scrub_duration":0.190029279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":538,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":451,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.150603+0000","last_change":"2024-09-13T14:47:47.866784+0000","last_active":"2024-09-13T14:54:37.150603+0000","last_peered":"2024-09-13T14:54:37.150603+0000","last_clean":"2024-09-13T14:54:37.150603+0000","last_became_active":"2024-09-13T14:47:47.866483+0000","last_became_peered":"2024-09-13T14:47:47.866483+0000","last_unstale":"2024-09-13T14:54:37.150603+0000","last_undegraded":"2024-09-13T14:54:37.150603+0000","last_fullsized":"2024-09-13T14:54:37.150603+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:15:53.119741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":259,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.141717+0000","last_change":"2024-09-13T14:47:47.880946+0000","last_active":"2024-09-13T14:54:37.141717+0000","last_peered":"2024-09-13T14:54:37.141717+0000","last_clean":"2024-09-13T14:54:37.141717+0000","last_became_active":"2024-09-13T14:47:47.866534+0000","last_became_peered":"2024-09-13T14:47:47.866534+0000","last_unstale":"2024-09-13T14:54:37.141717+0000","last_undegraded":"2024-09-13T14:54:37.141717+0000","last_fullsized":"2024-09-13T14:54:37.141717+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:01:01.384873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.17","version":"427'9474","reported_seq":20244,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:38.782770+0000","last_change":"2024-09-13T14:54:25.015288+0000","last_active":"2024-09-13T14:54:38.782770+0000","last_peered":"2024-09-13T14:54:38.782770+0000","last_clean":"2024-09-13T14:54:38.782770+0000","last_became_active":"2024-09-13T14:54:12.986075+0000","last_became_peered":"2024-09-13T14:54:12.986075+0000","last_unstale":"2024-09-13T14:54:38.782770+0000","last_undegraded":"2024-09-13T14:54:38.782770+0000","last_fullsized":"2024-09-13T14:54:38.782770+0000","mapping_epoch":423,"log_start":"382'3796","ondisk_log_start":"382'3796","created":381,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":5,"last_scrub":"425'9051","last_scrub_stamp":"2024-09-13T14:54:25.015010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:54:25.015010+0000","objects_scrubbed":425,"log_size":5678,"log_dups_size":0,"ondisk_log_size":5678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:21.055686+0000","scrub_duration":0.14909209200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":425,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":444,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.150361+0000","last_change":"2024-09-13T14:38:03.526333+0000","last_active":"2024-09-13T14:54:37.150361+0000","last_peered":"2024-09-13T14:54:37.150361+0000","last_clean":"2024-09-13T14:54:37.150361+0000","last_became_active":"2024-09-13T14:38:03.526001+0000","last_became_peered":"2024-09-13T14:38:03.526001+0000","last_unstale":"2024-09-13T14:54:37.150361+0000","last_undegraded":"2024-09-13T14:54:37.150361+0000","last_fullsized":"2024-09-13T14:54:37.150361+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:25:25.181271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":246,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.141903+0000","last_change":"2024-09-13T14:37:49.149582+0000","last_active":"2024-09-13T14:54:37.141903+0000","last_peered":"2024-09-13T14:54:37.141903+0000","last_clean":"2024-09-13T14:54:37.141903+0000","last_became_active":"2024-09-13T14:37:49.148525+0000","last_became_peered":"2024-09-13T14:37:49.148525+0000","last_unstale":"2024-09-13T14:54:37.141903+0000","last_undegraded":"2024-09-13T14:54:37.141903+0000","last_fullsized":"2024-09-13T14:54:37.141903+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:48.655665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.14","version":"427'9549","reported_seq":9729,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-13T14:54:39.152737+0000","last_change":"2024-09-13T14:51:46.732408+0000","last_active":"2024-09-13T14:54:39.152737+0000","last_peered":"2024-09-13T14:54:39.152737+0000","last_clean":"2024-09-13T14:54:39.152737+0000","last_became_active":"2024-09-13T14:51:34.975041+0000","last_became_peered":"2024-09-13T14:51:34.975041+0000","last_unstale":"2024-09-13T14:54:39.152737+0000","last_undegraded":"2024-09-13T14:54:39.152737+0000","last_fullsized":"2024-09-13T14:54:39.152737+0000","mapping_epoch":381,"log_start":"380'3708","ondisk_log_start":"380'3708","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":5,"last_scrub":"390'4143","last_scrub_stamp":"2024-09-13T14:51:46.732277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:51:46.732277+0000","objects_scrubbed":2317,"log_size":5841,"log_dups_size":0,"ondisk_log_size":5841,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:17.426329+0000","scrub_duration":474,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5413,"num_write_kb":99072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":790,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.107675+0000","last_change":"2024-09-13T14:46:01.766580+0000","last_active":"2024-09-13T14:54:37.107675+0000","last_peered":"2024-09-13T14:54:37.107675+0000","last_clean":"2024-09-13T14:54:37.107675+0000","last_became_active":"2024-09-13T14:46:01.766257+0000","last_became_peered":"2024-09-13T14:46:01.766257+0000","last_unstale":"2024-09-13T14:54:37.107675+0000","last_undegraded":"2024-09-13T14:54:37.107675+0000","last_fullsized":"2024-09-13T14:54:37.107675+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:01.126109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":754,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.108464+0000","last_change":"2024-09-13T14:46:01.769647+0000","last_active":"2024-09-13T14:54:37.108464+0000","last_peered":"2024-09-13T14:54:37.108464+0000","last_clean":"2024-09-13T14:54:37.108464+0000","last_became_active":"2024-09-13T14:46:01.769228+0000","last_became_peered":"2024-09-13T14:46:01.769228+0000","last_unstale":"2024-09-13T14:54:37.108464+0000","last_undegraded":"2024-09-13T14:54:37.108464+0000","last_fullsized":"2024-09-13T14:54:37.108464+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:58.696024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.15","version":"427'9453","reported_seq":20307,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:38.818542+0000","last_change":"2024-09-13T14:53:49.842329+0000","last_active":"2024-09-13T14:54:38.818542+0000","last_peered":"2024-09-13T14:54:38.818542+0000","last_clean":"2024-09-13T14:54:38.818542+0000","last_became_active":"2024-09-13T14:53:49.830376+0000","last_became_peered":"2024-09-13T14:53:49.830376+0000","last_unstale":"2024-09-13T14:54:38.818542+0000","last_undegraded":"2024-09-13T14:54:38.818542+0000","last_fullsized":"2024-09-13T14:54:38.818542+0000","mapping_epoch":418,"log_start":"382'3719","ondisk_log_start":"382'3719","created":381,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":5,"last_scrub":"390'4111","last_scrub_stamp":"2024-09-13T14:51:47.635938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:50:49.763471+0000","last_clean_scrub_stamp":"2024-09-13T14:51:47.635938+0000","objects_scrubbed":2296,"log_size":5734,"log_dups_size":0,"ondisk_log_size":5734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:12:42.736570+0000","scrub_duration":278,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5342,"num_write_kb":97472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3567,"num_bytes_recovered":233766912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":433,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-13T14:54:37.150080+0000","last_change":"2024-09-13T14:47:47.860727+0000","last_active":"2024-09-13T14:54:37.150080+0000","last_peered":"2024-09-13T14:54:37.150080+0000","last_clean":"2024-09-13T14:54:37.150080+0000","last_became_active":"2024-09-13T14:47:47.860377+0000","last_became_peered":"2024-09-13T14:47:47.860377+0000","last_unstale":"2024-09-13T14:54:37.150080+0000","last_undegraded":"2024-09-13T14:54:37.150080+0000","last_fullsized":"2024-09-13T14:54:37.150080+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:27:00.868699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":404,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-13T14:54:39.698717+0000","last_change":"2024-09-13T14:46:01.775237+0000","last_active":"2024-09-13T14:54:39.698717+0000","last_peered":"2024-09-13T14:54:39.698717+0000","last_clean":"2024-09-13T14:54:39.698717+0000","last_became_active":"2024-09-13T14:46:01.765025+0000","last_became_peered":"2024-09-13T14:46:01.765025+0000","last_unstale":"2024-09-13T14:54:39.698717+0000","last_undegraded":"2024-09-13T14:54:39.698717+0000","last_fullsized":"2024-09-13T14:54:39.698717+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:29:16.089042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1746,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":26,"stat_sum":{"num_bytes":3014656,"num_objects":46,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164094,"num_write_kb":3998337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7147,"num_bytes_recovered":318832640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":364969984,"data_stored":364969984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":173521,"ondisk_log_size":173521,"up":78,"acting":78,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":2154,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25132,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":151,"num_bytes_recovered":8388646,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3129,"ondisk_log_size":3129,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234530,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1084144,"kb_used_data":226916,"kb_used_omap":1,"kb_used_meta":857214,"kb_avail":92661008,"statfs":{"total":95995035648,"available":94884872192,"internally_reserved":0,"allocated":232361984,"data_stored":230156479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":877787595},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":239,"seq":1026497183948,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":581612,"kb_used_data":9820,"kb_used_omap":8,"kb_used_meta":571767,"kb_avail":93163540,"statfs":{"total":95995035648,"available":95399464960,"internally_reserved":0,"allocated":10055680,"data_stored":7858298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8453,"internal_metadata":585490171},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757795,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1161316,"kb_used_data":9752,"kb_used_omap":1,"kb_used_meta":1151550,"kb_avail":92583836,"statfs":{"total":95995035648,"available":94805848064,"internally_reserved":0,"allocated":9986048,"data_stored":7789074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1179187654},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019439,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":884784,"kb_used_data":5408,"kb_used_omap":1,"kb_used_meta":879358,"kb_avail":92860368,"statfs":{"total":95995035648,"available":95089016832,"internally_reserved":0,"allocated":5537792,"data_stored":3336371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":900463046},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435621,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1022480,"kb_used_data":25928,"kb_used_omap":5,"kb_used_meta":996538,"kb_avail":92722672,"statfs":{"total":95995035648,"available":94948016128,"internally_reserved":0,"allocated":26550272,"data_stored":24361118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":1020455038},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":104,"seq":446676599150,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":384420,"kb_used_data":5656,"kb_used_omap":7,"kb_used_meta":378744,"kb_avail":93360732,"statfs":{"total":95995035648,"available":95601389568,"internally_reserved":0,"allocated":5791744,"data_stored":3595058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8010,"internal_metadata":387834038},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":99,"seq":425201762676,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":416012,"kb_used_data":9472,"kb_used_omap":7,"kb_used_meta":406520,"kb_avail":93329140,"statfs":{"total":95995035648,"available":95569039360,"internally_reserved":0,"allocated":9699328,"data_stored":7517478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7759,"internal_metadata":416276913},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,0,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926200,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1128544,"kb_used_data":136148,"kb_used_omap":7,"kb_used_meta":992376,"kb_avail":92616608,"statfs":{"total":95995035648,"available":94839406592,"internally_reserved":0,"allocated":139415552,"data_stored":137222406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":1016193320},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6860,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":131596288,"data_stored":131596288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8519680,"data_stored":8519680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":224854016,"data_stored":224854016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:54:45.056 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-13T14:54:45.056 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:54:45.057 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-09-13T14:54:45.057 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T14:54:45.057 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T14:54:45.057 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-09-13T14:54:45.068 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:45.124 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:45.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:45.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:45.225 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:45.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:45.248 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:45.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:45.261 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:54:45.348 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:45.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:45.371 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:45.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:45.441 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:54:45.472 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:45.505 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:45.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:45.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:45.607 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:45.629 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:45.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:45.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:45.635 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:45 smithi195 ceph-mon[104262]: pgmap v1462: 119 pgs: 119 active+clean; 5.2 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 467 op/s 2024-09-13T14:54:45.635 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:45 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-13T14:54:45.635 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:45 smithi195 ceph-mon[104262]: 9.18 starting backfill to osd.6 from (0'0,0'0] MAX to 427'9513 2024-09-13T14:54:45.635 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:45 smithi195 ceph-mon[104262]: osdmap e431: 8 total, 8 up, 8 in 2024-09-13T14:54:45.635 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:45 smithi195 ceph-mon[104262]: 9.18 starting backfill to osd.7 from (0'0,0'0] MAX to 427'9513 2024-09-13T14:54:45.731 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:45.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:45.753 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:45.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:45.854 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:45.875 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:45.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:45.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:45.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:45 smithi102 ceph-mon[115269]: pgmap v1462: 119 pgs: 119 active+clean; 5.2 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 467 op/s 2024-09-13T14:54:45.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:45 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-13T14:54:45.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:45 smithi102 ceph-mon[115269]: 9.18 starting backfill to osd.6 from (0'0,0'0] MAX to 427'9513 2024-09-13T14:54:45.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:45 smithi102 ceph-mon[115269]: osdmap e431: 8 total, 8 up, 8 in 2024-09-13T14:54:45.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:45 smithi102 ceph-mon[115269]: 9.18 starting backfill to osd.7 from (0'0,0'0] MAX to 427'9513 2024-09-13T14:54:45.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:45 smithi102 ceph-mon[110183]: pgmap v1462: 119 pgs: 119 active+clean; 5.2 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 467 op/s 2024-09-13T14:54:45.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:45 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-13T14:54:45.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:45 smithi102 ceph-mon[110183]: 9.18 starting backfill to osd.6 from (0'0,0'0] MAX to 427'9513 2024-09-13T14:54:45.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:45 smithi102 ceph-mon[110183]: osdmap e431: 8 total, 8 up, 8 in 2024-09-13T14:54:45.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:45 smithi102 ceph-mon[110183]: 9.18 starting backfill to osd.7 from (0'0,0'0] MAX to 427'9513 2024-09-13T14:54:45.977 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:45.998 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:45.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:45.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:46.100 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:46.121 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:46.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:46.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:46.223 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:46.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:46.245 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:46.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:46.346 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:46.368 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:46.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:46.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:46.470 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:46.494 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:46.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:46.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:46.596 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:46.617 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:46.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:46.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:46.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:46 smithi102 ceph-mon[110183]: osdmap e432: 8 total, 8 up, 8 in 2024-09-13T14:54:46.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:46 smithi102 ceph-mon[110183]: pgmap v1465: 119 pgs: 119 active+clean; 5.2 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 122 op/s 2024-09-13T14:54:46.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:46 smithi102 ceph-mon[115269]: osdmap e432: 8 total, 8 up, 8 in 2024-09-13T14:54:46.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:46 smithi102 ceph-mon[115269]: pgmap v1465: 119 pgs: 119 active+clean; 5.2 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 122 op/s 2024-09-13T14:54:46.669 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:54:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:54:46] "GET /metrics HTTP/1.1" 200 36506 "" "Prometheus/2.51.0" 2024-09-13T14:54:46.719 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:46.741 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:46.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:46.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:46.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:46 smithi195 ceph-mon[104262]: osdmap e432: 8 total, 8 up, 8 in 2024-09-13T14:54:46.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:46 smithi195 ceph-mon[104262]: pgmap v1465: 119 pgs: 119 active+clean; 5.2 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 122 op/s 2024-09-13T14:54:46.843 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:46.865 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:46.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:46.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:46.958 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:46.958 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:46.966 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:46.987 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:46.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:46.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:47.001 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:47.050 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:54:47.050 INFO:teuthology.orchestra.run.smithi195.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-13T14:54:47.050 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:54:47.089 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:47.111 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:47.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:47.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:47.213 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:47.234 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:47.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:47.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:47.337 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:47.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:47.359 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:47.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:47.460 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:47.482 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:47.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:47.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:47.494 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'unique_pool_4' removed 2024-09-13T14:54:47.504 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:47 smithi102 ceph-mon[110183]: 9.19 starting backfill to osd.3 from (0'0,0'0] MAX to 427'9561 2024-09-13T14:54:47.505 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:47 smithi102 ceph-mon[110183]: osdmap e433: 8 total, 8 up, 8 in 2024-09-13T14:54:47.505 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:47 smithi102 ceph-mon[110183]: 9.19 starting backfill to osd.6 from (0'0,0'0] MAX to 427'9561 2024-09-13T14:54:47.505 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:47 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/4165154956' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:54:47.505 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:47 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:54:47.505 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:47 smithi102 ceph-mon[115269]: 9.19 starting backfill to osd.3 from (0'0,0'0] MAX to 427'9561 2024-09-13T14:54:47.505 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:47 smithi102 ceph-mon[115269]: osdmap e433: 8 total, 8 up, 8 in 2024-09-13T14:54:47.505 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:47 smithi102 ceph-mon[115269]: 9.19 starting backfill to osd.6 from (0'0,0'0] MAX to 427'9561 2024-09-13T14:54:47.505 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:47 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/4165154956' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:54:47.506 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:47 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:54:47.583 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:47.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:47.605 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:47.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:47.709 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:47.731 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:47.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:47.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:47.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:47 smithi195 ceph-mon[104262]: 9.19 starting backfill to osd.3 from (0'0,0'0] MAX to 427'9561 2024-09-13T14:54:47.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:47 smithi195 ceph-mon[104262]: osdmap e433: 8 total, 8 up, 8 in 2024-09-13T14:54:47.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:47 smithi195 ceph-mon[104262]: 9.19 starting backfill to osd.6 from (0'0,0'0] MAX to 427'9561 2024-09-13T14:54:47.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:47 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/4165154956' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:54:47.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:47 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:54:47.832 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:47.866 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:47.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:47.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:47.968 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:47.989 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:47.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:47.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:48.091 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:48.113 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:48.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:48.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:48.215 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:48.236 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:48.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:48.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:48.332 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-13T14:54:48.333 INFO:tasks.radosbench:Beginning radosbench... 2024-09-13T14:54:48.333 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_5 2024-09-13T14:54:48.333 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_5 16 2024-09-13T14:54:48.338 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:48.398 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:48.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:48.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:48.500 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:48.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:48.522 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:48.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:48.623 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:48.644 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:48.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:48.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:48.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:48 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:54:48.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:48 smithi102 ceph-mon[110183]: osdmap e434: 8 total, 8 up, 8 in 2024-09-13T14:54:48.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:48 smithi102 ceph-mon[110183]: pgmap v1468: 93 pgs: 93 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:54:48.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:48 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:54:48.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:48 smithi102 ceph-mon[115269]: osdmap e434: 8 total, 8 up, 8 in 2024-09-13T14:54:48.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:48 smithi102 ceph-mon[115269]: pgmap v1468: 93 pgs: 93 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:54:48.746 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:48.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:48 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:54:48.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:48 smithi195 ceph-mon[104262]: osdmap e434: 8 total, 8 up, 8 in 2024-09-13T14:54:48.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:48 smithi195 ceph-mon[104262]: pgmap v1468: 93 pgs: 93 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:54:48.769 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:48.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:48.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:48.871 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:48.892 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:48.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:48.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:48.994 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:49.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:49.016 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:49.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:49.117 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:49.139 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:49.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:49.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:49.241 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:49.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:49.267 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:49.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:49.369 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:49.391 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:49.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:49.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:49.493 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:49.514 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:49.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:49.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:49.615 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:49.641 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:49.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:49.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:49.742 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:49.768 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:49.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:49.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:49.870 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:49.896 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:49.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:49.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:49.998 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:50.024 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:50.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:50.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:50.126 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:50.147 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:50.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:50.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:50.249 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:50.270 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:50.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:50.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:50.372 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:50.394 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:50.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:50.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:50.495 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:50.516 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:50.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:50.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:50.618 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:50.645 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:50.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:50.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:50.746 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:50.772 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:50.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:50.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:50.874 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:50.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:50.896 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:50.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:50.997 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:51.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:51.020 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:51.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:51.112 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:50 smithi102 ceph-mon[110183]: pgmap v1469: 93 pgs: 93 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:54:51.112 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:50 smithi102 ceph-mon[115269]: pgmap v1469: 93 pgs: 93 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:54:51.122 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:51.143 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:51.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:51.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:51.244 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:51.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:50 smithi195 ceph-mon[104262]: pgmap v1469: 93 pgs: 93 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:54:51.266 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:51.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:51.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:51.369 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:51.394 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:51.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:51.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:51.496 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:51.521 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:51.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:51.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:51.623 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:51.649 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:51.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:51.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:51.751 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:51.776 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:51.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:51.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:51.856 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:54:51.856 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:54:51.865 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:54:51.865 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:54:51.878 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:51.882 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:54:51.883 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:54:51.899 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:54:51.899 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:54:51.904 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:51.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:51.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:51.916 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:54:51.916 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:54:51.932 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:54:51.932 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:54:51.949 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:54:51.949 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:54:51.966 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:54:51.966 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:54:52.005 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:52.030 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:52.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:52.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:52.131 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:52.155 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:52.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:52.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:52.257 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:52.280 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:52.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:52.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:52.382 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:52.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:52.406 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:52.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:52.507 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:52.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:52.530 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:52.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:52.631 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:52.652 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:52.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:52.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:52.754 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:52.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:52.776 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:52.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:52.824 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:52.877 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:52.899 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:52.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:52.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:52.915 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 6, 7, 5, 2, 1] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T14:54:52.915 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:54:52.915 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-09-13T14:54:52.916 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T14:54:52.916 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T14:54:52.916 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.0 config set filestore_inject_stall 3 2024-09-13T14:54:52.964 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:52 smithi102 ceph-mon[115269]: pgmap v1470: 93 pgs: 93 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:54:52.964 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:52 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2654152375' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:54:52.964 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:52 smithi102 ceph-mon[110183]: pgmap v1470: 93 pgs: 93 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:54:52.964 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:52 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2654152375' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:54:53.001 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:53.027 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:53.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:53.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:53.129 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:53.150 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:53.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:53.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:53.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:52 smithi195 ceph-mon[104262]: pgmap v1470: 93 pgs: 93 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:54:53.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:52 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2654152375' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:54:53.252 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:53.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:53.274 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:53.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:53.350 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:54:53.350 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:54:53.375 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:53.396 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:53.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:53.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:53.498 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:53.520 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:53.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:53.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:53.621 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:53.643 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:53.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:53.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:53.745 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:53.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:53.773 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:53.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:53.874 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:53.897 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:53.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:53.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:53.999 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:54.022 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:54.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:54.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:54.069 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T14:54:54.124 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:54.145 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:54.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:54.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:54.247 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:54.269 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:54.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:54.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:54.370 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:54.393 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:54.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:54.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:54.494 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:54.516 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:54.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:54.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:54.619 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:54.640 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:54.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:54.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:54.741 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:54.762 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:54.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:54.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:54.864 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:54.885 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:54.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:54.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:54.987 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:55.010 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:55.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:55.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:55.109 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:54 smithi102 ceph-mon[115269]: pgmap v1471: 93 pgs: 93 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:54:55.109 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:54 smithi102 ceph-mon[110183]: pgmap v1471: 93 pgs: 93 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:54:55.112 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:55.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:55.135 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:55.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:55.236 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:55.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:54 smithi195 ceph-mon[104262]: pgmap v1471: 93 pgs: 93 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:54:55.256 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:55.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:55.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:55.358 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:55.381 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:55.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:55.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:55.482 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:55.503 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:55.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:55.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:55.605 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:55.627 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:55.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:55.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:55.729 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:55.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:55.751 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:55.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:55.852 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:55.874 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:55.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:55.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:55.976 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:55.997 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:55.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:55.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:56.070 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:55 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3477507514' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:54:56.070 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:55 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:54:56.071 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:55 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3477507514' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:54:56.071 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:55 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:54:56.099 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:56.121 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:56.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:56.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:56.223 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:56.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:56.247 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:56.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:56.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:55 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3477507514' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:54:56.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:55 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:54:56.348 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:56.372 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:56.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:56.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:56.431 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is set 2024-09-13T14:54:56.474 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:56.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:56.496 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:56.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:56.525 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:54:56.597 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:56.621 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:56.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:56.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:56.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:54:56 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:54:56] "GET /metrics HTTP/1.1" 200 34158 "" "Prometheus/2.51.0" 2024-09-13T14:54:56.723 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:56.746 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:56.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:56.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:56.836 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'unique_pool_5' created 2024-09-13T14:54:56.848 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:56.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:56.873 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:56.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:56.974 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:56.997 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:56.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:56.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:57.099 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:57.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:57.123 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:57.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:57.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:56 smithi102 ceph-mon[110183]: pgmap v1472: 93 pgs: 93 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:54:57.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:56 smithi102 ceph-mon[110183]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:54:57.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:56 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3477507514' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:54:57.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:56 smithi102 ceph-mon[110183]: osdmap e435: 8 total, 8 up, 8 in 2024-09-13T14:54:57.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:56 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1290363920' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-09-13T14:54:57.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:56 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-09-13T14:54:57.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:56 smithi102 ceph-mon[115269]: pgmap v1472: 93 pgs: 93 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:54:57.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:56 smithi102 ceph-mon[115269]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:54:57.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:56 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3477507514' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:54:57.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:56 smithi102 ceph-mon[115269]: osdmap e435: 8 total, 8 up, 8 in 2024-09-13T14:54:57.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:56 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1290363920' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-09-13T14:54:57.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:56 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-09-13T14:54:57.224 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:57.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:56 smithi195 ceph-mon[104262]: pgmap v1472: 93 pgs: 93 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:54:57.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:56 smithi195 ceph-mon[104262]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:54:57.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:56 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3477507514' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:54:57.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:56 smithi195 ceph-mon[104262]: osdmap e435: 8 total, 8 up, 8 in 2024-09-13T14:54:57.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:56 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1290363920' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-09-13T14:54:57.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:56 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-09-13T14:54:57.249 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:57.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:57.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:57.279 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:57.279 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:57.325 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:57.350 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:57.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:57.377 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:57.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:57.478 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:57.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:57.501 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:57.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:57.602 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:57.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:57.629 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:57.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:57.730 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:57.756 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:57.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:57.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:57.858 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:57.880 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:57.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:57.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:57.983 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:58.004 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:58.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:58.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:58.103 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:57 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-09-13T14:54:58.103 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:57 smithi102 ceph-mon[110183]: osdmap e436: 8 total, 8 up, 8 in 2024-09-13T14:54:58.103 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:57 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-09-13T14:54:58.103 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:57 smithi102 ceph-mon[115269]: osdmap e436: 8 total, 8 up, 8 in 2024-09-13T14:54:58.106 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:58.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:58.133 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:58.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:58.234 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:58.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:57 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-09-13T14:54:58.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:57 smithi195 ceph-mon[104262]: osdmap e436: 8 total, 8 up, 8 in 2024-09-13T14:54:58.259 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:58.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:58.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:58.361 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:58.384 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:58.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:58.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:58.486 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:58.513 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:58.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:58.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:58.616 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:58.641 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:58.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:58.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:58.743 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:58.767 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:58.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:58.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:58.869 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:58.890 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:58.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:58.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:58.992 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:59.013 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:59.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:59.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:59.115 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:59.136 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:59.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:59.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:59.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:58 smithi102 ceph-mon[110183]: pgmap v1475: 109 pgs: 16 unknown, 93 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:54:59.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:58 smithi102 ceph-mon[110183]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:54:59.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:58 smithi102 ceph-mon[110183]: osdmap e437: 8 total, 8 up, 8 in 2024-09-13T14:54:59.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:58 smithi102 ceph-mon[115269]: pgmap v1475: 109 pgs: 16 unknown, 93 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:54:59.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:58 smithi102 ceph-mon[115269]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:54:59.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:58 smithi102 ceph-mon[115269]: osdmap e437: 8 total, 8 up, 8 in 2024-09-13T14:54:59.238 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:59.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:58 smithi195 ceph-mon[104262]: pgmap v1475: 109 pgs: 16 unknown, 93 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:54:59.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:58 smithi195 ceph-mon[104262]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:54:59.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:58 smithi195 ceph-mon[104262]: osdmap e437: 8 total, 8 up, 8 in 2024-09-13T14:54:59.259 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:59.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:59.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:59.305 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:54:59.361 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:59.387 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:59.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:59.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:59.489 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:59.511 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:54:59.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:59.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:59.612 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:59.638 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:59.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:59.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:59.739 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:59.742 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_5 rados --yes-i-really-mean-it || true 2024-09-13T14:54:59.765 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:59.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:59.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:59.866 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:59.892 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:54:59.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:59.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:59.994 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:00.019 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:00.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:00.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:00.121 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:00.145 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:00.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:00.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:00.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:54:59 smithi102 ceph-mon[110183]: osdmap e438: 8 total, 8 up, 8 in 2024-09-13T14:55:00.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:54:59 smithi102 ceph-mon[115269]: osdmap e438: 8 total, 8 up, 8 in 2024-09-13T14:55:00.247 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:00.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:54:59 smithi195 ceph-mon[104262]: osdmap e438: 8 total, 8 up, 8 in 2024-09-13T14:55:00.269 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:00.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:00.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:00.371 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:00.395 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:00.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:00.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:00.497 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:00.518 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:00.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:00.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:00.620 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:00.641 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:00.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:00.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:00.743 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:00.769 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:00.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:00.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:00.871 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:00.897 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:00.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:00.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:00.999 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:01.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:01.021 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:01.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:01.124 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:01.147 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:00 smithi102 ceph-mon[110183]: pgmap v1478: 109 pgs: 2 creating+activating, 11 unknown, 96 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:55:01.147 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:00 smithi102 ceph-mon[115269]: pgmap v1478: 109 pgs: 2 creating+activating, 11 unknown, 96 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:55:01.150 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:01.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:01.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:01.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:00 smithi195 ceph-mon[104262]: pgmap v1478: 109 pgs: 2 creating+activating, 11 unknown, 96 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:55:01.252 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:01.274 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:01.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:01.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:01.367 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:55:01.368 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:55:01.377 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:01.402 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:01.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:01.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:01.504 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:01.527 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:01.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:01.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:01.629 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:01.651 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:01.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:01.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:01.755 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:01.781 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:01.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:01.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:01.805 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:55:01.805 INFO:teuthology.orchestra.run.smithi102.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-13T14:55:01.806 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:55:01.883 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:01.904 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:01.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:01.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:02.006 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:02.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:02.034 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:02.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:02.135 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:02.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:02.162 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:02.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:02.263 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:02.285 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:02.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:02.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:02.388 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:02.414 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:02.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:02.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:02.517 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:02.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:02.544 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:02.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:02.645 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:02.667 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:02.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:02.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:02.769 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:02.794 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:02.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:02.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:02.896 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:02.918 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:02.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:02.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:03.020 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:03.035 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T14:55:03.047 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:03.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:03.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:03.140 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:02 smithi102 ceph-mon[110183]: pgmap v1479: 109 pgs: 2 creating+activating, 1 unknown, 106 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:55:03.140 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:02 smithi102 ceph-mon[115269]: pgmap v1479: 109 pgs: 2 creating+activating, 1 unknown, 106 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:55:03.149 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:03.173 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:03.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:03.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:03.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:02 smithi195 ceph-mon[104262]: pgmap v1479: 109 pgs: 2 creating+activating, 1 unknown, 106 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:55:03.275 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:03.296 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:03.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:03.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:03.398 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:03.419 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:03.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:03.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:03.520 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:03.546 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:03.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:03.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:03.648 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:03.675 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:03.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:03.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:03.776 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:03.802 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:03.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:03.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:03.904 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:03.929 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:03.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:03.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:04.031 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:04.053 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:04.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:04.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:04.154 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:04.176 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:04.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:04.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:04.278 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:04.304 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:04.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:04.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:04.398 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:55:04.398 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:55:04.399 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:55:04.399 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:55:04.406 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:04.414 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:55:04.414 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:55:04.431 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:04.432 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:55:04.433 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:55:04.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:04.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:04.447 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:04.452 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:04.454 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:55:04.454 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:55:04.458 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:04.472 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:55:04.473 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:55:04.492 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:55:04.492 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:55:04.509 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:55:04.509 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:55:04.534 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:04.557 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:04.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:04.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:04.659 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:04.680 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:04.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:04.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:04.782 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:04.807 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:04.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:04.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:04.908 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:04.933 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:04.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:04.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:05.016 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:04 smithi102 ceph-mon[115269]: pgmap v1480: 109 pgs: 109 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:55:05.016 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:04 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2706790712' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:55:05.017 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:04 smithi102 ceph-mon[110183]: pgmap v1480: 109 pgs: 109 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:55:05.017 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:04 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2706790712' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:55:05.034 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:05.055 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:05.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:05.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:05.157 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:05.178 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:05.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:05.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:05.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:04 smithi195 ceph-mon[104262]: pgmap v1480: 109 pgs: 109 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:55:05.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:04 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2706790712' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:55:05.280 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:05.303 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:05.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:05.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:05.405 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:05.426 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:05.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:05.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:05.528 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:05.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:05.549 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:05.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:05.650 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:05.671 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:05.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:05.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:05.773 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:05.798 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:05.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:05.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:05.900 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:05.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:05.923 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:05.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:06.024 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:06.050 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:06.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:06.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:06.152 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:06.173 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:06.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:06.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:06.275 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:06.297 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:06.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:06.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:06.399 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:06.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:06.422 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:06.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:06.522 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:06.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:06.545 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:06.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:06.646 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:06.668 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:06.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:55:06 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:55:06] "GET /metrics HTTP/1.1" 200 34158 "" "Prometheus/2.51.0" 2024-09-13T14:55:06.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:06.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:06.770 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:06.791 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:06.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:06.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:06.894 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:06.916 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:06.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:06.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:07.018 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:07.045 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:07.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:07.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:07.146 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:07.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:06 smithi102 ceph-mon[110183]: pgmap v1481: 109 pgs: 109 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:55:07.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:06 smithi102 ceph-mon[115269]: pgmap v1481: 109 pgs: 109 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:55:07.171 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:07.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:07.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:07.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:06 smithi195 ceph-mon[104262]: pgmap v1481: 109 pgs: 109 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:55:07.273 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:07.302 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:07.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:07.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:07.404 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:07.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:07.427 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:07.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:07.516 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 6, 7, 5, 2, 1] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T14:55:07.516 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:55:07.516 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 3, 4, 6, 7, 5, 2, 1] 2024-09-13T14:55:07.517 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd out 4 2024-09-13T14:55:07.529 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:07.554 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:07.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:07.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:07.656 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:07.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:07.686 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:07.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:07.787 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:07.811 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:07.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:07.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:07.913 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:07.934 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:07.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:07.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:08.036 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:08.062 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:08.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:08.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:08.164 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:08.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:08.187 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:08.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:08.288 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:08.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:08.310 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:08.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:08.317 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:55:08.410 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:08.435 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:08.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:08.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:08.537 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:08.560 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:08.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:08.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:08.662 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:08.685 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:08.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:08.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:08.787 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:08.809 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:08.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:08.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:08.850 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:55:08.850 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [], 2024-09-13T14:55:08.850 INFO:teuthology.orchestra.run.smithi102.stdout: "complaint_time": 30, 2024-09-13T14:55:08.850 INFO:teuthology.orchestra.run.smithi102.stdout: "num_blocked_ops": 0 2024-09-13T14:55:08.850 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:55:08.911 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:08.935 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:08.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:08.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:09.037 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:09.061 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:09.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:09.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:09.159 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:08 smithi102 ceph-mon[110183]: pgmap v1482: 109 pgs: 109 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:55:09.160 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:08 smithi102 ceph-mon[115269]: pgmap v1482: 109 pgs: 109 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:55:09.163 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:09.187 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:09.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:09.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:09.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:08 smithi195 ceph-mon[104262]: pgmap v1482: 109 pgs: 109 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:55:09.289 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:09.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:09.314 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:09.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:09.415 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:09.437 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:09.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:09.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:09.491 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T14:55:09.538 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:09.559 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:09.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:09.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:09.661 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:09.683 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:09.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:09.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:09.785 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:09.811 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:09.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:09.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:09.913 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:09.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:09.936 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:09.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:10.037 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:10.063 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:10.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:10.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:10.165 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:10.190 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:10.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:10.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:10.292 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:10.317 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:10.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:10.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:10.419 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:10.443 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:10.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:10.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:10.500 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:10.539 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:10.541 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:10.545 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:10.569 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:10.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:10.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:10.671 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:10.696 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:10.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:10.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:10.798 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:10.823 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:10.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:10.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:10.873 INFO:teuthology.orchestra.run.smithi102.stderr:enabled application 'rados' on pool 'unique_pool_5' 2024-09-13T14:55:10.925 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:10.950 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:10.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:10.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:11.052 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:11.074 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:11.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:11.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:11.080 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:10 smithi102 ceph-mon[110183]: pgmap v1483: 109 pgs: 109 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:55:11.081 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:10 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3228490725' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:55:11.081 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:10 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:55:11.081 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:55:11.081 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:55:11.081 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:10 smithi102 ceph-mon[115269]: pgmap v1483: 109 pgs: 109 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:55:11.081 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:10 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3228490725' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:55:11.081 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:10 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:55:11.081 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:55:11.081 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:55:11.176 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:11.199 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:11.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:11.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:11.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:10 smithi195 ceph-mon[104262]: pgmap v1483: 109 pgs: 109 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:55:11.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:10 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3228490725' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:55:11.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:10 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:55:11.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:55:11.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:55:11.300 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:11.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:11.322 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:11.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:11.423 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:11.449 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:11.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:11.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:11.551 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:11.577 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:11.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:11.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:11.679 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:11.701 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:11.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:11.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:11.802 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:11.828 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:11.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:11.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:11.930 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:11.952 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:11.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:11.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:12.054 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:12.076 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:12.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:12.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:12.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:11 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:55:12.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:11 smithi102 ceph-mon[110183]: osdmap e439: 8 total, 8 up, 8 in 2024-09-13T14:55:12.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:11 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:55:12.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:11 smithi102 ceph-mon[115269]: osdmap e439: 8 total, 8 up, 8 in 2024-09-13T14:55:12.177 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:12.204 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:12.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:12.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:12.253 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:11 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:55:12.253 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:11 smithi195 ceph-mon[104262]: osdmap e439: 8 total, 8 up, 8 in 2024-09-13T14:55:12.307 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:12.329 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:12.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:12.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:12.431 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:12.453 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:12.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:12.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:12.555 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:12.576 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:12.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:12.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:12.678 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:12.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:12.705 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:12.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:12.806 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:12.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:12.829 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:12.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:12.931 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:12.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:12.955 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:12.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:13.055 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:13.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:13.084 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:13.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:13.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:12 smithi102 ceph-mon[110183]: pgmap v1485: 109 pgs: 109 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:55:13.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:12 smithi102 ceph-mon[110183]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:55:13.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:12 smithi102 ceph-mon[115269]: pgmap v1485: 109 pgs: 109 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:55:13.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:12 smithi102 ceph-mon[115269]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:55:13.185 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:13.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:13.210 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:13.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:13.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:12 smithi195 ceph-mon[104262]: pgmap v1485: 109 pgs: 109 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:55:13.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:12 smithi195 ceph-mon[104262]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:55:13.311 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:13.337 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:13.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:13.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:13.438 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:13.461 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:13.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:13.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:13.563 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:13.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:13.585 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:13.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:13.686 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:13.708 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:13.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:13.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:13.809 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:13.836 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:13.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:13.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:13.938 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:13.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:13.961 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:13.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:14.048 DEBUG:teuthology.orchestra.run.smithi102:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_5 bench 90 write' 2024-09-13T14:55:14.052 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-13T14:55:14.052 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-13T14:55:14.062 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:14.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:14.090 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:14.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:14.102 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:hints = 1 2024-09-13T14:55:14.103 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-13T14:55:14.103 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Object prefix: benchmark_data_smithi102_377304 2024-09-13T14:55:14.103 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:55:14.103 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 0 0 0 0 0 0 - 0 2024-09-13T14:55:14.190 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:14.224 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:14.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:14.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:14.330 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:14.370 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:14.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:14.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:14.472 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:14.510 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:14.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:14.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:14.613 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:14.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:14.654 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:14.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:14.755 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:14.789 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:14.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:14.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:14.794 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:55:14.794 INFO:teuthology.orchestra.run.smithi102.stdout: "size": 20, 2024-09-13T14:55:14.794 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 600, 2024-09-13T14:55:14.795 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [ 2024-09-13T14:55:14.795 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.795 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:252346 8.a 8:58700e01:::benchmark_data_smithi102_313275_object121759:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e367)", 2024-09-13T14:55:14.795 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:50:19.052204+0000", 2024-09-13T14:55:14.795 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 295.573992428, 2024-09-13T14:55:14.795 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.196151849, 2024-09-13T14:55:14.795 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:55:14.795 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:55:14.796 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "reached pg", 2024-09-13T14:55:14.796 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:55:14.796 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:55:14.796 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:55:14.796 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 252346 2024-09-13T14:55:14.796 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.796 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:55:14.796 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.797 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:55:14.797 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.052204+0000", 2024-09-13T14:55:14.797 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:14.797 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.797 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.797 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:55:14.797 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.052204+0000", 2024-09-13T14:55:14.797 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:14.797 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.797 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.797 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:55:14.797 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.052206+0000", 2024-09-13T14:55:14.797 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.736e-06 2024-09-13T14:55:14.797 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.798 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.798 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:55:14.798 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.052213+0000", 2024-09-13T14:55:14.798 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.0330000000000003e-06 2024-09-13T14:55:14.798 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.798 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.798 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:55:14.798 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.052233+0000", 2024-09-13T14:55:14.798 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.9701e-05 2024-09-13T14:55:14.798 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.798 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.798 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:55:14.798 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.052242+0000", 2024-09-13T14:55:14.799 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.5489999999999995e-06 2024-09-13T14:55:14.799 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.799 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.799 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:14.799 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.052480+0000", 2024-09-13T14:55:14.799 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00023812099999999999 2024-09-13T14:55:14.799 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.799 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.799 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op must wait for map", 2024-09-13T14:55:14.799 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.052502+0000", 2024-09-13T14:55:14.799 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.1322999999999999e-05 2024-09-13T14:55:14.799 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.800 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.800 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:14.800 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.062076+0000", 2024-09-13T14:55:14.800 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0095745269999999993 2024-09-13T14:55:14.800 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.800 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.800 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:55:14.800 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.062085+0000", 2024-09-13T14:55:14.800 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.6300000000000004e-06 2024-09-13T14:55:14.800 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.800 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.800 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:14.800 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.248270+0000", 2024-09-13T14:55:14.801 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.186185171 2024-09-13T14:55:14.801 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.801 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.801 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:55:14.801 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.248356+0000", 2024-09-13T14:55:14.801 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.6057999999999995e-05 2024-09-13T14:55:14.801 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:14.801 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:55:14.801 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:14.801 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.801 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.801 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:252348 8.a 8:5ec110d5:::benchmark_data_smithi102_313275_object121761:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e367)", 2024-09-13T14:55:14.801 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:50:19.052269+0000", 2024-09-13T14:55:14.802 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 295.57392764899998, 2024-09-13T14:55:14.802 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1965683030000001, 2024-09-13T14:55:14.802 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:55:14.802 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:55:14.802 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "reached pg", 2024-09-13T14:55:14.802 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:55:14.802 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:55:14.802 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:55:14.802 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 252348 2024-09-13T14:55:14.802 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.802 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:55:14.802 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.802 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:55:14.803 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.052269+0000", 2024-09-13T14:55:14.803 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:14.803 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.803 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.803 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:55:14.803 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.052269+0000", 2024-09-13T14:55:14.803 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:14.803 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.803 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.803 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:55:14.803 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.052270+0000", 2024-09-13T14:55:14.803 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.2e-07 2024-09-13T14:55:14.803 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.804 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.804 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:55:14.804 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.052274+0000", 2024-09-13T14:55:14.804 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.0450000000000002e-06 2024-09-13T14:55:14.804 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.804 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.804 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:55:14.804 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.052281+0000", 2024-09-13T14:55:14.804 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.8150000000000006e-06 2024-09-13T14:55:14.804 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.804 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.804 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:55:14.804 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.052288+0000", 2024-09-13T14:55:14.805 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.1979999999999998e-06 2024-09-13T14:55:14.805 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.805 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.805 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:14.805 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.052673+0000", 2024-09-13T14:55:14.805 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00038578199999999999 2024-09-13T14:55:14.805 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.805 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.805 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting_for_map not empty", 2024-09-13T14:55:14.805 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.052692+0000", 2024-09-13T14:55:14.805 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.8286000000000001e-05 2024-09-13T14:55:14.805 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.805 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.806 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:14.806 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.062112+0000", 2024-09-13T14:55:14.806 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0094207289999999992 2024-09-13T14:55:14.806 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.806 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.806 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:55:14.806 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.062118+0000", 2024-09-13T14:55:14.806 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.2689999999999999e-06 2024-09-13T14:55:14.806 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.806 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.806 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:14.806 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.248779+0000", 2024-09-13T14:55:14.806 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1866613610000001 2024-09-13T14:55:14.806 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.807 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.807 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:55:14.807 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.248837+0000", 2024-09-13T14:55:14.807 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.8298000000000003e-05 2024-09-13T14:55:14.807 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:14.807 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:55:14.807 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:14.807 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.807 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.807 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:252356 8.a 8:5578b6ec:::benchmark_data_smithi102_313275_object121769:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e367)", 2024-09-13T14:55:14.807 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:50:19.052296+0000", 2024-09-13T14:55:14.807 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 295.573900821, 2024-09-13T14:55:14.807 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1966574910000001, 2024-09-13T14:55:14.808 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:55:14.808 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:55:14.808 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "reached pg", 2024-09-13T14:55:14.808 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:55:14.808 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:55:14.808 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:55:14.808 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 252356 2024-09-13T14:55:14.808 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.808 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:55:14.808 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.808 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:55:14.808 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.052296+0000", 2024-09-13T14:55:14.808 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:14.809 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.809 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.809 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:55:14.809 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.052296+0000", 2024-09-13T14:55:14.809 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:14.809 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.809 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.809 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:55:14.809 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.052296+0000", 2024-09-13T14:55:14.809 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.1899999999999998e-07 2024-09-13T14:55:14.809 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.809 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.809 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:55:14.810 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.052298+0000", 2024-09-13T14:55:14.810 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.911e-06 2024-09-13T14:55:14.810 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.810 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.810 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:55:14.810 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.052304+0000", 2024-09-13T14:55:14.810 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.9769999999999997e-06 2024-09-13T14:55:14.810 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.810 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.811 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:55:14.811 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.052309+0000", 2024-09-13T14:55:14.811 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.9599999999999999e-06 2024-09-13T14:55:14.811 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.811 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.811 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:14.811 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.052855+0000", 2024-09-13T14:55:14.811 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00054586299999999995 2024-09-13T14:55:14.811 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.811 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.811 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting_for_map not empty", 2024-09-13T14:55:14.811 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.052876+0000", 2024-09-13T14:55:14.812 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.0845000000000001e-05 2024-09-13T14:55:14.812 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.812 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.812 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:14.812 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.062465+0000", 2024-09-13T14:55:14.812 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0095893200000000001 2024-09-13T14:55:14.812 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.812 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.812 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:55:14.812 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.062473+0000", 2024-09-13T14:55:14.812 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.5340000000000002e-06 2024-09-13T14:55:14.812 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.812 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.813 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:14.813 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.248884+0000", 2024-09-13T14:55:14.813 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.186410838 2024-09-13T14:55:14.813 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.813 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.813 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:55:14.813 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.248953+0000", 2024-09-13T14:55:14.813 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.9824000000000001e-05 2024-09-13T14:55:14.813 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:14.813 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:55:14.813 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:14.814 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.814 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.814 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:252383 8.a 8:5934c274:::benchmark_data_smithi102_313275_object121796:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e367)", 2024-09-13T14:55:14.814 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:50:19.076745+0000", 2024-09-13T14:55:14.814 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 295.54945197900003, 2024-09-13T14:55:14.814 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2015794909999999, 2024-09-13T14:55:14.814 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:55:14.814 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:55:14.814 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:14.814 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:55:14.814 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:55:14.814 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:55:14.815 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 252383 2024-09-13T14:55:14.815 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.815 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:55:14.815 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.815 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:55:14.815 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.076745+0000", 2024-09-13T14:55:14.815 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:14.815 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.815 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.815 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:55:14.815 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.076745+0000", 2024-09-13T14:55:14.815 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:14.815 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.816 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.816 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:55:14.816 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.076747+0000", 2024-09-13T14:55:14.816 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.782e-06 2024-09-13T14:55:14.816 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.816 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.816 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:55:14.816 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.076754+0000", 2024-09-13T14:55:14.816 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.0219999999999999e-06 2024-09-13T14:55:14.816 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.816 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.816 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:55:14.816 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.076772+0000", 2024-09-13T14:55:14.817 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.8465999999999999e-05 2024-09-13T14:55:14.817 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.817 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.817 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:55:14.817 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.076782+0000", 2024-09-13T14:55:14.817 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.7229999999999999e-06 2024-09-13T14:55:14.817 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.817 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.817 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:14.817 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.076850+0000", 2024-09-13T14:55:14.817 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.8052000000000005e-05 2024-09-13T14:55:14.817 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.817 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.818 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:55:14.818 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.076869+0000", 2024-09-13T14:55:14.818 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.9065000000000001e-05 2024-09-13T14:55:14.818 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.818 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.818 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:14.818 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.248980+0000", 2024-09-13T14:55:14.818 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.172110741 2024-09-13T14:55:14.818 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.818 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.818 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:55:14.818 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.249082+0000", 2024-09-13T14:55:14.818 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00010212 2024-09-13T14:55:14.819 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.819 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.819 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 2,5", 2024-09-13T14:55:14.819 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.249656+0000", 2024-09-13T14:55:14.819 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00057433400000000002 2024-09-13T14:55:14.819 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.819 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.819 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:55:14.819 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.278140+0000", 2024-09-13T14:55:14.819 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.028483952999999999 2024-09-13T14:55:14.819 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.819 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.819 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:14.820 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.278196+0000", 2024-09-13T14:55:14.820 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.5767000000000003e-05 2024-09-13T14:55:14.820 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.820 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.820 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:14.820 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.278239+0000", 2024-09-13T14:55:14.820 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.2852999999999998e-05 2024-09-13T14:55:14.820 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.820 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.820 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:55:14.820 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.278287+0000", 2024-09-13T14:55:14.820 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.8078000000000003e-05 2024-09-13T14:55:14.820 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.821 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.821 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:55:14.821 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.278324+0000", 2024-09-13T14:55:14.821 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.7534999999999999e-05 2024-09-13T14:55:14.821 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:14.821 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:55:14.821 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:14.821 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.821 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.821 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:252389 8.a 8:55e597ad:::benchmark_data_smithi102_313275_object121802:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e367)", 2024-09-13T14:55:14.821 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:50:19.083438+0000", 2024-09-13T14:55:14.821 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 295.54275880099999, 2024-09-13T14:55:14.821 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.202485394, 2024-09-13T14:55:14.822 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:55:14.822 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:55:14.822 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:14.822 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:55:14.822 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:55:14.822 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:55:14.822 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 252389 2024-09-13T14:55:14.822 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.822 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:55:14.822 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.822 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:55:14.822 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.083438+0000", 2024-09-13T14:55:14.822 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:14.822 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.823 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.823 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:55:14.823 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.083438+0000", 2024-09-13T14:55:14.823 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:14.823 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.823 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.823 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:55:14.823 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.083440+0000", 2024-09-13T14:55:14.823 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.7370000000000001e-06 2024-09-13T14:55:14.823 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.823 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.823 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:55:14.823 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.083447+0000", 2024-09-13T14:55:14.824 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.9500000000000004e-06 2024-09-13T14:55:14.824 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.824 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.824 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:55:14.824 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.083466+0000", 2024-09-13T14:55:14.824 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.9188999999999999e-05 2024-09-13T14:55:14.824 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.824 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.824 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:55:14.824 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.083474+0000", 2024-09-13T14:55:14.824 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.242e-06 2024-09-13T14:55:14.824 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.824 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.824 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:14.825 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.083647+0000", 2024-09-13T14:55:14.825 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00017294599999999999 2024-09-13T14:55:14.825 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.825 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.825 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:55:14.825 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.083662+0000", 2024-09-13T14:55:14.825 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.5308999999999999e-05 2024-09-13T14:55:14.825 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.825 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.825 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:14.825 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.275214+0000", 2024-09-13T14:55:14.825 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1915512640000001 2024-09-13T14:55:14.826 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.826 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.826 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:55:14.826 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.275286+0000", 2024-09-13T14:55:14.826 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.2293999999999996e-05 2024-09-13T14:55:14.826 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.826 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.826 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 2,5", 2024-09-13T14:55:14.826 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.275374+0000", 2024-09-13T14:55:14.826 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.7997000000000005e-05 2024-09-13T14:55:14.826 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.826 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.826 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:55:14.826 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.285678+0000", 2024-09-13T14:55:14.827 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.010303970000000001 2024-09-13T14:55:14.827 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.827 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.827 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:14.827 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.285716+0000", 2024-09-13T14:55:14.827 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.7945999999999999e-05 2024-09-13T14:55:14.827 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.827 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.827 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:14.827 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.285756+0000", 2024-09-13T14:55:14.827 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.0621999999999998e-05 2024-09-13T14:55:14.827 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.827 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.828 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:55:14.828 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.285884+0000", 2024-09-13T14:55:14.828 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000127816 2024-09-13T14:55:14.828 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.828 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.828 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:55:14.828 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.285923+0000", 2024-09-13T14:55:14.828 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.9112000000000001e-05 2024-09-13T14:55:14.828 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:14.828 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:55:14.828 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:14.828 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.828 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.829 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:252395 8.a 8:5d2ce78d:::benchmark_data_smithi102_313275_object121808:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e367)", 2024-09-13T14:55:14.829 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:50:19.088847+0000", 2024-09-13T14:55:14.829 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 295.53735007099999, 2024-09-13T14:55:14.829 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2060130390000001, 2024-09-13T14:55:14.829 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:55:14.829 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:55:14.829 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:14.829 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:55:14.829 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:55:14.829 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:55:14.829 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 252395 2024-09-13T14:55:14.829 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.829 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:55:14.830 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.830 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:55:14.830 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.088847+0000", 2024-09-13T14:55:14.830 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:14.830 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.830 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.830 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:55:14.830 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.088847+0000", 2024-09-13T14:55:14.830 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:14.830 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.830 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.830 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:55:14.830 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.088848+0000", 2024-09-13T14:55:14.831 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4100000000000001e-06 2024-09-13T14:55:14.831 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.831 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.831 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:55:14.831 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.088859+0000", 2024-09-13T14:55:14.831 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0547000000000001e-05 2024-09-13T14:55:14.831 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.831 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.831 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:55:14.831 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.088872+0000", 2024-09-13T14:55:14.831 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3115999999999999e-05 2024-09-13T14:55:14.831 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.831 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.831 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:55:14.832 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.088879+0000", 2024-09-13T14:55:14.832 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.374e-06 2024-09-13T14:55:14.832 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.832 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.832 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:14.832 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.089230+0000", 2024-09-13T14:55:14.832 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000350961 2024-09-13T14:55:14.832 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.832 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.832 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:55:14.832 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.089267+0000", 2024-09-13T14:55:14.832 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.7184000000000002e-05 2024-09-13T14:55:14.832 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.833 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.833 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:14.833 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.278358+0000", 2024-09-13T14:55:14.833 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1890905540000001 2024-09-13T14:55:14.833 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.833 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.833 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:55:14.833 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.278467+0000", 2024-09-13T14:55:14.833 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00010953700000000001 2024-09-13T14:55:14.833 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.833 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.833 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 2,5", 2024-09-13T14:55:14.833 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.278823+0000", 2024-09-13T14:55:14.833 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00035605599999999999 2024-09-13T14:55:14.834 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.834 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.834 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:55:14.834 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.285726+0000", 2024-09-13T14:55:14.834 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0069029249999999999 2024-09-13T14:55:14.834 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.834 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.834 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:14.834 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.285963+0000", 2024-09-13T14:55:14.834 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000236791 2024-09-13T14:55:14.834 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.834 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.834 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:14.834 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.294755+0000", 2024-09-13T14:55:14.835 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0087923540000000005 2024-09-13T14:55:14.835 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.835 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.835 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:55:14.835 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.294810+0000", 2024-09-13T14:55:14.835 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.4151999999999999e-05 2024-09-13T14:55:14.835 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.835 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.835 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:55:14.835 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.294860+0000", 2024-09-13T14:55:14.835 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.0077999999999997e-05 2024-09-13T14:55:14.835 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:14.835 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:55:14.836 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:14.836 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.836 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.836 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:252401 8.a 8:54313b1e:::benchmark_data_smithi102_313275_object121814:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e367)", 2024-09-13T14:55:14.836 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:50:19.095734+0000", 2024-09-13T14:55:14.836 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 295.53046298499999, 2024-09-13T14:55:14.836 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.199272672, 2024-09-13T14:55:14.836 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:55:14.836 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:55:14.836 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:14.836 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:55:14.836 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:55:14.836 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:55:14.837 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 252401 2024-09-13T14:55:14.837 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.837 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:55:14.837 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.837 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:55:14.837 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.095734+0000", 2024-09-13T14:55:14.837 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:14.837 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.837 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.837 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:55:14.837 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.095734+0000", 2024-09-13T14:55:14.837 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:14.837 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.837 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.838 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:55:14.838 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.095735+0000", 2024-09-13T14:55:14.838 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.421e-06 2024-09-13T14:55:14.838 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.838 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.838 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:55:14.838 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.095740+0000", 2024-09-13T14:55:14.838 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.5499999999999996e-06 2024-09-13T14:55:14.838 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.838 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.838 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:55:14.838 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.095753+0000", 2024-09-13T14:55:14.838 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3608000000000001e-05 2024-09-13T14:55:14.838 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.839 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.839 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:55:14.839 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.095760+0000", 2024-09-13T14:55:14.839 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.7490000000000001e-06 2024-09-13T14:55:14.839 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.839 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.839 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:14.839 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.095929+0000", 2024-09-13T14:55:14.839 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000169228 2024-09-13T14:55:14.839 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.839 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.839 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:55:14.839 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.095951+0000", 2024-09-13T14:55:14.840 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.1203e-05 2024-09-13T14:55:14.840 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.840 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.840 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:14.840 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.281896+0000", 2024-09-13T14:55:14.840 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.185945706 2024-09-13T14:55:14.840 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.840 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.840 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:55:14.840 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.282016+0000", 2024-09-13T14:55:14.840 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000120196 2024-09-13T14:55:14.840 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.840 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.841 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 2,5", 2024-09-13T14:55:14.841 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.282160+0000", 2024-09-13T14:55:14.841 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00014351699999999999 2024-09-13T14:55:14.841 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.841 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.841 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:55:14.841 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.285973+0000", 2024-09-13T14:55:14.841 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0038129700000000002 2024-09-13T14:55:14.841 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.841 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.841 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:14.841 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.286003+0000", 2024-09-13T14:55:14.841 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.9736000000000001e-05 2024-09-13T14:55:14.842 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.842 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.842 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:14.844 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.294904+0000", 2024-09-13T14:55:14.844 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0089009529999999996 2024-09-13T14:55:14.844 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.845 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.845 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:55:14.845 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.294945+0000", 2024-09-13T14:55:14.845 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.1899e-05 2024-09-13T14:55:14.845 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.845 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.845 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:55:14.845 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.295006+0000", 2024-09-13T14:55:14.845 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.0936000000000001e-05 2024-09-13T14:55:14.845 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:14.845 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:55:14.845 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:14.845 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.845 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.846 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:252407 8.a 8:5b55c829:::benchmark_data_smithi102_313275_object121820:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e367)", 2024-09-13T14:55:14.846 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:50:19.100510+0000", 2024-09-13T14:55:14.846 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 295.52568682999998, 2024-09-13T14:55:14.846 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.221289726, 2024-09-13T14:55:14.846 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:55:14.846 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:55:14.846 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:14.846 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:55:14.846 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:55:14.846 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:55:14.846 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 252407 2024-09-13T14:55:14.846 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.846 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:55:14.847 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.847 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:55:14.847 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.100510+0000", 2024-09-13T14:55:14.847 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:14.847 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.847 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.847 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:55:14.847 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.100510+0000", 2024-09-13T14:55:14.847 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:14.847 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.847 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.847 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:55:14.847 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.100511+0000", 2024-09-13T14:55:14.848 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.252e-06 2024-09-13T14:55:14.848 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.848 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.848 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:55:14.848 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.100516+0000", 2024-09-13T14:55:14.848 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.3070000000000002e-06 2024-09-13T14:55:14.848 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.848 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.848 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:55:14.848 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.100533+0000", 2024-09-13T14:55:14.848 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.7041000000000002e-05 2024-09-13T14:55:14.848 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.848 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.849 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:55:14.849 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.100542+0000", 2024-09-13T14:55:14.849 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.1389999999999995e-06 2024-09-13T14:55:14.849 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.849 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.849 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:14.849 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.100615+0000", 2024-09-13T14:55:14.849 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.3051000000000002e-05 2024-09-13T14:55:14.849 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.849 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.849 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:55:14.849 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.100623+0000", 2024-09-13T14:55:14.849 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.5739999999999996e-06 2024-09-13T14:55:14.850 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.850 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.850 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:14.850 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.286031+0000", 2024-09-13T14:55:14.850 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1854076570000001 2024-09-13T14:55:14.850 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.850 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.850 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:55:14.850 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.286129+0000", 2024-09-13T14:55:14.850 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.7870999999999999e-05 2024-09-13T14:55:14.850 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.850 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.850 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 2,5", 2024-09-13T14:55:14.850 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.286233+0000", 2024-09-13T14:55:14.851 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00010375200000000001 2024-09-13T14:55:14.851 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.851 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.851 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:14.851 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.295050+0000", 2024-09-13T14:55:14.851 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0088175110000000001 2024-09-13T14:55:14.851 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.851 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.851 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:14.851 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.295090+0000", 2024-09-13T14:55:14.851 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.9585000000000001e-05 2024-09-13T14:55:14.851 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.851 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.851 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:55:14.852 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.321672+0000", 2024-09-13T14:55:14.852 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.026581944 2024-09-13T14:55:14.852 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.852 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.852 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:55:14.852 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.321760+0000", 2024-09-13T14:55:14.852 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.8051000000000001e-05 2024-09-13T14:55:14.852 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.852 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.852 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:55:14.852 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.321800+0000", 2024-09-13T14:55:14.852 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.9990999999999997e-05 2024-09-13T14:55:14.852 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:14.853 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:55:14.853 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:14.853 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.853 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.853 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:252409 8.a 8:5f8882c3:::benchmark_data_smithi102_313275_object121822:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e367)", 2024-09-13T14:55:14.853 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:50:19.104221+0000", 2024-09-13T14:55:14.853 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 295.521975623, 2024-09-13T14:55:14.853 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.217702721, 2024-09-13T14:55:14.853 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:55:14.853 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:55:14.853 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:14.853 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:55:14.853 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:55:14.853 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:55:14.854 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 252409 2024-09-13T14:55:14.854 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.854 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:55:14.854 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.854 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:55:14.854 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.104221+0000", 2024-09-13T14:55:14.854 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:14.854 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.854 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.854 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:55:14.854 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.104221+0000", 2024-09-13T14:55:14.854 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:14.854 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.855 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.855 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:55:14.855 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.104223+0000", 2024-09-13T14:55:14.855 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.469e-06 2024-09-13T14:55:14.855 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.855 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.855 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:55:14.855 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.104227+0000", 2024-09-13T14:55:14.855 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.6099999999999999e-06 2024-09-13T14:55:14.855 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.855 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.855 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:55:14.855 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.104237+0000", 2024-09-13T14:55:14.855 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.6539999999999994e-06 2024-09-13T14:55:14.856 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.856 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.856 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:55:14.856 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.104243+0000", 2024-09-13T14:55:14.856 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.5409999999999997e-06 2024-09-13T14:55:14.856 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.856 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.856 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:14.856 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.104399+0000", 2024-09-13T14:55:14.856 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000155241 2024-09-13T14:55:14.856 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.856 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.856 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:55:14.857 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.104420+0000", 2024-09-13T14:55:14.857 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.1039999999999998e-05 2024-09-13T14:55:14.857 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.857 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.857 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:14.857 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.291076+0000", 2024-09-13T14:55:14.857 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.186656763 2024-09-13T14:55:14.857 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.857 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.857 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:55:14.857 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.291203+0000", 2024-09-13T14:55:14.857 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00012628800000000001 2024-09-13T14:55:14.857 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.858 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.858 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 2,5", 2024-09-13T14:55:14.858 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.291317+0000", 2024-09-13T14:55:14.858 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00011383 2024-09-13T14:55:14.858 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.858 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.858 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:14.858 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.305727+0000", 2024-09-13T14:55:14.858 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.014410378 2024-09-13T14:55:14.858 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.858 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.858 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:14.858 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.305765+0000", 2024-09-13T14:55:14.858 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.7793999999999998e-05 2024-09-13T14:55:14.859 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.859 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.859 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:55:14.859 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.321856+0000", 2024-09-13T14:55:14.859 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.016091759000000001 2024-09-13T14:55:14.859 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.859 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.859 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:55:14.859 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.321893+0000", 2024-09-13T14:55:14.859 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.6247e-05 2024-09-13T14:55:14.859 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.859 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.859 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:55:14.860 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.321924+0000", 2024-09-13T14:55:14.860 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.1106999999999999e-05 2024-09-13T14:55:14.860 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:14.860 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:55:14.860 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:14.860 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.860 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.860 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:252418 8.a 8:53b16a66:::benchmark_data_smithi102_313275_object121831:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e367)", 2024-09-13T14:55:14.860 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:50:19.111174+0000", 2024-09-13T14:55:14.860 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 295.51502304899998, 2024-09-13T14:55:14.860 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.210831445, 2024-09-13T14:55:14.860 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:55:14.860 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:55:14.860 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:14.861 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:55:14.861 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:55:14.861 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:55:14.861 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 252418 2024-09-13T14:55:14.861 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.861 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:55:14.861 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.861 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:55:14.861 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.111174+0000", 2024-09-13T14:55:14.861 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:14.861 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.861 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.861 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:55:14.862 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.111174+0000", 2024-09-13T14:55:14.862 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:14.862 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.862 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.862 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:55:14.862 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.111175+0000", 2024-09-13T14:55:14.862 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.468e-06 2024-09-13T14:55:14.862 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.862 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.862 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:55:14.862 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.111180+0000", 2024-09-13T14:55:14.862 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.0780000000000003e-06 2024-09-13T14:55:14.862 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.862 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.863 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:55:14.863 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.111192+0000", 2024-09-13T14:55:14.863 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1789e-05 2024-09-13T14:55:14.863 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.863 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.863 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:55:14.863 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.111199+0000", 2024-09-13T14:55:14.863 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.0999999999999998e-06 2024-09-13T14:55:14.863 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.863 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.863 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:14.863 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.111255+0000", 2024-09-13T14:55:14.863 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.6264000000000002e-05 2024-09-13T14:55:14.863 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.864 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.864 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:55:14.864 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.111264+0000", 2024-09-13T14:55:14.864 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.1850000000000004e-06 2024-09-13T14:55:14.864 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.864 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.864 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:14.864 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.295119+0000", 2024-09-13T14:55:14.864 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.183855814 2024-09-13T14:55:14.864 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.864 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.864 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:55:14.864 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.295250+0000", 2024-09-13T14:55:14.865 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000130302 2024-09-13T14:55:14.865 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.865 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.865 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 2,5", 2024-09-13T14:55:14.865 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.295395+0000", 2024-09-13T14:55:14.865 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000144945 2024-09-13T14:55:14.865 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.865 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.865 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:14.865 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.305800+0000", 2024-09-13T14:55:14.865 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.010405642999999999 2024-09-13T14:55:14.865 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.865 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.865 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:14.866 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.313168+0000", 2024-09-13T14:55:14.866 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.007367635 2024-09-13T14:55:14.866 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.866 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.866 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:55:14.866 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.321929+0000", 2024-09-13T14:55:14.866 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0087614579999999997 2024-09-13T14:55:14.866 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.866 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.866 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:55:14.866 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.321976+0000", 2024-09-13T14:55:14.866 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.6618e-05 2024-09-13T14:55:14.866 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.866 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.867 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:55:14.867 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.322005+0000", 2024-09-13T14:55:14.867 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.9145999999999998e-05 2024-09-13T14:55:14.867 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:14.867 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:55:14.867 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:14.867 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.867 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.867 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:252426 8.a 8:51e1e350:::benchmark_data_smithi102_313275_object121839:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e367)", 2024-09-13T14:55:14.867 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:50:19.120028+0000", 2024-09-13T14:55:14.867 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 295.50616880699999, 2024-09-13T14:55:14.867 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.202042093, 2024-09-13T14:55:14.867 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:55:14.868 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:55:14.868 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:14.868 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:55:14.868 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:55:14.868 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:55:14.868 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 252426 2024-09-13T14:55:14.868 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.868 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:55:14.868 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.868 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:55:14.868 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.120028+0000", 2024-09-13T14:55:14.868 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:14.868 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.869 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.869 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:55:14.869 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.120028+0000", 2024-09-13T14:55:14.869 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:14.869 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.869 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.869 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:55:14.869 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.120029+0000", 2024-09-13T14:55:14.869 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.238e-06 2024-09-13T14:55:14.869 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.869 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.869 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:55:14.869 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.120033+0000", 2024-09-13T14:55:14.869 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.2930000000000002e-06 2024-09-13T14:55:14.870 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.870 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.870 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:55:14.870 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.120044+0000", 2024-09-13T14:55:14.870 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0738e-05 2024-09-13T14:55:14.870 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.870 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.870 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:55:14.870 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.120051+0000", 2024-09-13T14:55:14.870 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.669e-06 2024-09-13T14:55:14.870 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.870 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.871 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:14.871 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.120110+0000", 2024-09-13T14:55:14.871 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.8900000000000002e-05 2024-09-13T14:55:14.871 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.871 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.871 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:55:14.871 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:19.120118+0000", 2024-09-13T14:55:14.871 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.2290000000000003e-06 2024-09-13T14:55:14.871 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.871 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.871 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:14.871 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.298589+0000", 2024-09-13T14:55:14.871 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.178470525 2024-09-13T14:55:14.872 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.872 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.872 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:55:14.872 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.298681+0000", 2024-09-13T14:55:14.872 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.2600000000000001e-05 2024-09-13T14:55:14.872 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.872 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.872 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 2,5", 2024-09-13T14:55:14.872 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.298781+0000", 2024-09-13T14:55:14.872 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00010007 2024-09-13T14:55:14.872 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.872 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.872 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:14.873 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.305839+0000", 2024-09-13T14:55:14.873 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0070574569999999996 2024-09-13T14:55:14.873 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.873 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.873 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:14.873 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.313206+0000", 2024-09-13T14:55:14.873 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0073676119999999999 2024-09-13T14:55:14.873 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.873 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.873 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:55:14.873 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.322010+0000", 2024-09-13T14:55:14.873 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.008803857 2024-09-13T14:55:14.873 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.874 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.874 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:55:14.874 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.322043+0000", 2024-09-13T14:55:14.874 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.2775999999999999e-05 2024-09-13T14:55:14.874 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.874 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.874 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:55:14.874 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:20.322070+0000", 2024-09-13T14:55:14.874 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.7129000000000001e-05 2024-09-13T14:55:14.874 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:14.874 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:55:14.874 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:14.874 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.875 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.875 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.36528.0:72059 9.12 9:4c71233f:::benchmark_data_smithi102_348563_object72058:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T14:55:14.875 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:51:45.055516+0000", 2024-09-13T14:55:14.875 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 209.57068072199999, 2024-09-13T14:55:14.875 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.165950303, 2024-09-13T14:55:14.875 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:55:14.875 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:55:14.875 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:14.875 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:55:14.875 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.36528", 2024-09-13T14:55:14.875 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:55:14.875 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 72059 2024-09-13T14:55:14.875 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.876 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:55:14.876 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.876 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:55:14.876 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055516+0000", 2024-09-13T14:55:14.876 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:14.876 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.876 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.876 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:55:14.876 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055516+0000", 2024-09-13T14:55:14.876 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:14.876 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.876 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.876 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:55:14.877 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055517+0000", 2024-09-13T14:55:14.877 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.0400000000000005e-07 2024-09-13T14:55:14.877 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.877 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.877 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:55:14.877 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055570+0000", 2024-09-13T14:55:14.877 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.3514000000000003e-05 2024-09-13T14:55:14.877 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.877 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.877 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:55:14.877 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055584+0000", 2024-09-13T14:55:14.877 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3682999999999999e-05 2024-09-13T14:55:14.877 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.878 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.878 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:55:14.878 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055592+0000", 2024-09-13T14:55:14.878 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.0790000000000001e-06 2024-09-13T14:55:14.878 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.878 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.878 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:14.878 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055639+0000", 2024-09-13T14:55:14.878 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.7332e-05 2024-09-13T14:55:14.878 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.878 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.878 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:55:14.878 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055644+0000", 2024-09-13T14:55:14.879 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.8749999999999999e-06 2024-09-13T14:55:14.879 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.879 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.879 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:14.879 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.200933+0000", 2024-09-13T14:55:14.879 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1452891000000001 2024-09-13T14:55:14.879 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.879 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.879 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:55:14.879 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.201107+0000", 2024-09-13T14:55:14.880 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000173158 2024-09-13T14:55:14.880 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.880 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.880 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:55:14.880 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.201288+0000", 2024-09-13T14:55:14.880 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00018143 2024-09-13T14:55:14.880 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.880 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.880 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:55:14.880 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.205867+0000", 2024-09-13T14:55:14.880 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0045794649999999996 2024-09-13T14:55:14.880 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.880 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.881 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:14.881 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.207600+0000", 2024-09-13T14:55:14.881 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.001732716 2024-09-13T14:55:14.881 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.881 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.881 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:14.881 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.218907+0000", 2024-09-13T14:55:14.881 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.011306837 2024-09-13T14:55:14.881 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.881 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.881 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:55:14.881 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.219315+0000", 2024-09-13T14:55:14.881 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00040796499999999999 2024-09-13T14:55:14.882 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.882 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.882 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:55:14.882 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.221466+0000", 2024-09-13T14:55:14.882 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0021513449999999998 2024-09-13T14:55:14.882 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:14.882 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:55:14.882 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:14.882 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.882 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.882 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.36528.0:72094 9.12 9:4c58cbaa:::benchmark_data_smithi102_348563_object72093:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T14:55:14.882 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:51:45.055610+0000", 2024-09-13T14:55:14.882 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 209.570586891, 2024-09-13T14:55:14.883 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.17144435, 2024-09-13T14:55:14.883 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:55:14.883 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:55:14.883 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:14.883 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:55:14.883 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.36528", 2024-09-13T14:55:14.883 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:55:14.883 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 72094 2024-09-13T14:55:14.883 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.883 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:55:14.883 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.883 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:55:14.883 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055610+0000", 2024-09-13T14:55:14.884 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:14.884 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.884 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.884 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:55:14.884 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055610+0000", 2024-09-13T14:55:14.884 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:14.884 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.884 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.884 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:55:14.884 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055610+0000", 2024-09-13T14:55:14.884 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.82e-07 2024-09-13T14:55:14.884 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.884 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.885 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:55:14.885 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055657+0000", 2024-09-13T14:55:14.885 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.6347000000000002e-05 2024-09-13T14:55:14.885 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.885 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.885 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:55:14.885 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055667+0000", 2024-09-13T14:55:14.885 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.7880000000000001e-06 2024-09-13T14:55:14.885 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.885 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.885 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:55:14.885 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055672+0000", 2024-09-13T14:55:14.885 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.8320000000000002e-06 2024-09-13T14:55:14.886 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.886 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.886 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:14.886 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055705+0000", 2024-09-13T14:55:14.886 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.3049000000000002e-05 2024-09-13T14:55:14.886 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.886 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.886 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:55:14.886 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055711+0000", 2024-09-13T14:55:14.886 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.8980000000000001e-06 2024-09-13T14:55:14.886 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.886 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.886 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:14.887 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.202961+0000", 2024-09-13T14:55:14.887 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.147249438 2024-09-13T14:55:14.887 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.887 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.887 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:55:14.887 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.203111+0000", 2024-09-13T14:55:14.887 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000150211 2024-09-13T14:55:14.887 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.887 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.887 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:55:14.887 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.203262+0000", 2024-09-13T14:55:14.887 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000151187 2024-09-13T14:55:14.888 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.888 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.888 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:14.888 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.207649+0000", 2024-09-13T14:55:14.888 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0043872900000000003 2024-09-13T14:55:14.888 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.888 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.888 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:55:14.888 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.211746+0000", 2024-09-13T14:55:14.888 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0040962230000000004 2024-09-13T14:55:14.888 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.888 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.889 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:14.889 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.224231+0000", 2024-09-13T14:55:14.889 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.012485489000000001 2024-09-13T14:55:14.889 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.889 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.889 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:55:14.889 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.224292+0000", 2024-09-13T14:55:14.889 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.0692000000000001e-05 2024-09-13T14:55:14.889 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.889 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.889 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:55:14.889 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.227054+0000", 2024-09-13T14:55:14.889 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.002762324 2024-09-13T14:55:14.889 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:14.890 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:55:14.890 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:14.890 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.890 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.890 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.36528.0:72114 9.12 9:48c2513d:::benchmark_data_smithi102_348563_object72113:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T14:55:14.890 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:51:45.055685+0000", 2024-09-13T14:55:14.890 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 209.57051205499999, 2024-09-13T14:55:14.890 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.173506333, 2024-09-13T14:55:14.890 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:55:14.890 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:55:14.890 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:14.890 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:55:14.890 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.36528", 2024-09-13T14:55:14.891 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:55:14.891 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 72114 2024-09-13T14:55:14.891 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.891 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:55:14.891 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.891 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:55:14.891 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055685+0000", 2024-09-13T14:55:14.891 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:14.891 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.891 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.891 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:55:14.891 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055685+0000", 2024-09-13T14:55:14.892 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:14.892 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.892 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.892 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:55:14.892 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055685+0000", 2024-09-13T14:55:14.892 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.4099999999999999e-07 2024-09-13T14:55:14.892 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.892 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.892 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:55:14.892 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055729+0000", 2024-09-13T14:55:14.892 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.4044e-05 2024-09-13T14:55:14.892 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.892 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.893 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:55:14.893 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055738+0000", 2024-09-13T14:55:14.893 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.8300000000000002e-06 2024-09-13T14:55:14.893 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.893 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.893 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:55:14.893 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055743+0000", 2024-09-13T14:55:14.893 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.3229999999999997e-06 2024-09-13T14:55:14.893 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.893 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.893 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:14.893 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055785+0000", 2024-09-13T14:55:14.893 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.1734999999999999e-05 2024-09-13T14:55:14.894 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.894 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.894 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:55:14.894 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055792+0000", 2024-09-13T14:55:14.894 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.8770000000000004e-06 2024-09-13T14:55:14.894 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.894 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.894 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:14.894 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.205937+0000", 2024-09-13T14:55:14.894 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1501447659999999 2024-09-13T14:55:14.894 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.894 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.894 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:55:14.895 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.206070+0000", 2024-09-13T14:55:14.895 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000133132 2024-09-13T14:55:14.895 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.895 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.895 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:55:14.895 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.206221+0000", 2024-09-13T14:55:14.895 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00015074399999999999 2024-09-13T14:55:14.895 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.895 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.895 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:14.895 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.211639+0000", 2024-09-13T14:55:14.895 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0054186119999999997 2024-09-13T14:55:14.895 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.896 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.896 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:55:14.896 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.214424+0000", 2024-09-13T14:55:14.896 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0027847800000000002 2024-09-13T14:55:14.896 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.896 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.896 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:14.896 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.227212+0000", 2024-09-13T14:55:14.896 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.012788336000000001 2024-09-13T14:55:14.896 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.896 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.896 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:55:14.896 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.227288+0000", 2024-09-13T14:55:14.897 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.5174999999999998e-05 2024-09-13T14:55:14.897 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.897 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.897 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:55:14.897 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.229191+0000", 2024-09-13T14:55:14.897 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0019034379999999999 2024-09-13T14:55:14.897 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:14.897 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:55:14.897 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:14.897 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.897 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.897 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.36528.0:72115 9.12 9:48e0c73b:::benchmark_data_smithi102_348563_object72114:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T14:55:14.897 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:51:45.055755+0000", 2024-09-13T14:55:14.898 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 209.570442082, 2024-09-13T14:55:14.898 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1751879940000001, 2024-09-13T14:55:14.898 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:55:14.898 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:55:14.898 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:14.898 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:55:14.898 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.36528", 2024-09-13T14:55:14.898 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:55:14.898 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 72115 2024-09-13T14:55:14.898 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.898 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:55:14.898 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.898 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:55:14.899 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055755+0000", 2024-09-13T14:55:14.899 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:14.899 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.899 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.899 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:55:14.899 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055755+0000", 2024-09-13T14:55:14.899 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:14.899 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.899 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.899 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:55:14.899 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055755+0000", 2024-09-13T14:55:14.899 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.6499999999999999e-07 2024-09-13T14:55:14.899 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.899 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.900 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:55:14.900 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055799+0000", 2024-09-13T14:55:14.900 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.3971999999999998e-05 2024-09-13T14:55:14.900 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.900 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.900 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:55:14.900 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055807+0000", 2024-09-13T14:55:14.900 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.2800000000000003e-06 2024-09-13T14:55:14.900 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.900 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.900 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:55:14.900 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055813+0000", 2024-09-13T14:55:14.900 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.49e-06 2024-09-13T14:55:14.901 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.901 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.901 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:14.901 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055839+0000", 2024-09-13T14:55:14.901 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.6492999999999999e-05 2024-09-13T14:55:14.901 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.901 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.901 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:55:14.901 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055843+0000", 2024-09-13T14:55:14.901 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.7929999999999999e-06 2024-09-13T14:55:14.901 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.901 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.902 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:14.902 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.207677+0000", 2024-09-13T14:55:14.902 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1518341400000001 2024-09-13T14:55:14.902 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.902 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.902 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:55:14.902 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.207846+0000", 2024-09-13T14:55:14.902 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000168908 2024-09-13T14:55:14.902 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.902 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.902 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:55:14.902 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.207989+0000", 2024-09-13T14:55:14.902 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00014232000000000001 2024-09-13T14:55:14.903 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.903 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.903 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:14.903 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.211724+0000", 2024-09-13T14:55:14.903 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0037355489999999999 2024-09-13T14:55:14.903 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.903 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.903 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:55:14.903 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.214439+0000", 2024-09-13T14:55:14.903 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0027154369999999998 2024-09-13T14:55:14.903 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.903 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.903 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:14.903 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.229240+0000", 2024-09-13T14:55:14.904 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.014800845 2024-09-13T14:55:14.904 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.904 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.904 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:55:14.904 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.229384+0000", 2024-09-13T14:55:14.904 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00014322799999999999 2024-09-13T14:55:14.904 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.904 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.904 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:55:14.904 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.230943+0000", 2024-09-13T14:55:14.904 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0015590739999999999 2024-09-13T14:55:14.904 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:14.904 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:55:14.905 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:14.905 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.905 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.905 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.36528.0:72120 9.12 9:49a2b7f2:::benchmark_data_smithi102_348563_object72119:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T14:55:14.905 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:51:45.055823+0000", 2024-09-13T14:55:14.905 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 209.57037367500001, 2024-09-13T14:55:14.905 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1773272690000001, 2024-09-13T14:55:14.905 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:55:14.905 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:55:14.905 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:14.905 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:55:14.905 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.36528", 2024-09-13T14:55:14.905 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:55:14.906 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 72120 2024-09-13T14:55:14.906 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.906 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:55:14.906 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.906 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:55:14.906 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055823+0000", 2024-09-13T14:55:14.906 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:14.906 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.906 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.906 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:55:14.906 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055823+0000", 2024-09-13T14:55:14.906 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:14.906 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.906 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.907 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:55:14.907 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055824+0000", 2024-09-13T14:55:14.907 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.7800000000000002e-07 2024-09-13T14:55:14.907 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.907 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.907 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:55:14.907 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055879+0000", 2024-09-13T14:55:14.907 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.5770999999999998e-05 2024-09-13T14:55:14.907 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.907 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.907 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:55:14.907 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055888+0000", 2024-09-13T14:55:14.907 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.0580000000000004e-06 2024-09-13T14:55:14.908 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.908 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.908 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:55:14.908 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055894+0000", 2024-09-13T14:55:14.908 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.4909999999999996e-06 2024-09-13T14:55:14.908 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.908 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.908 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:14.908 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055922+0000", 2024-09-13T14:55:14.908 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.7801e-05 2024-09-13T14:55:14.908 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.908 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.908 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:55:14.909 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055925+0000", 2024-09-13T14:55:14.909 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.6389999999999998e-06 2024-09-13T14:55:14.909 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.909 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.909 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:14.909 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.209252+0000", 2024-09-13T14:55:14.909 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1533266170000001 2024-09-13T14:55:14.909 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.909 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.909 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:55:14.909 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.209386+0000", 2024-09-13T14:55:14.909 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00013422299999999999 2024-09-13T14:55:14.910 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.910 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.910 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:55:14.910 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.209542+0000", 2024-09-13T14:55:14.910 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00015550500000000001 2024-09-13T14:55:14.910 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.910 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.910 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:14.910 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.214299+0000", 2024-09-13T14:55:14.910 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.004757102 2024-09-13T14:55:14.911 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.911 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.911 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:55:14.911 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.214450+0000", 2024-09-13T14:55:14.911 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00015120699999999999 2024-09-13T14:55:14.911 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.911 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.911 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:14.911 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.231578+0000", 2024-09-13T14:55:14.911 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.017128056999999999 2024-09-13T14:55:14.911 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.911 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.911 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:55:14.911 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.231635+0000", 2024-09-13T14:55:14.912 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.7043000000000003e-05 2024-09-13T14:55:14.912 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.912 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.912 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:55:14.912 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.233150+0000", 2024-09-13T14:55:14.912 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.001515277 2024-09-13T14:55:14.912 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:14.912 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:55:14.912 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:14.912 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.912 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.912 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.36528.0:72209 9.12 9:484a00ee:::benchmark_data_smithi102_348563_object72208:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T14:55:14.912 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:51:45.055905+0000", 2024-09-13T14:55:14.913 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 209.570291725, 2024-09-13T14:55:14.913 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1930136650000001, 2024-09-13T14:55:14.913 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:55:14.913 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:55:14.913 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:14.913 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:55:14.913 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.36528", 2024-09-13T14:55:14.913 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:55:14.913 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 72209 2024-09-13T14:55:14.913 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.913 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:55:14.913 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.913 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:55:14.913 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055905+0000", 2024-09-13T14:55:14.914 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:14.914 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.914 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.914 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:55:14.914 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055905+0000", 2024-09-13T14:55:14.914 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:14.914 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.914 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.914 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:55:14.914 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.055906+0000", 2024-09-13T14:55:14.914 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.1699999999999998e-07 2024-09-13T14:55:14.914 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.914 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.915 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:55:14.915 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.056091+0000", 2024-09-13T14:55:14.915 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00018559800000000001 2024-09-13T14:55:14.915 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.915 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.915 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:55:14.915 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.056101+0000", 2024-09-13T14:55:14.915 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.7480000000000005e-06 2024-09-13T14:55:14.915 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.915 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.915 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:55:14.915 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.056107+0000", 2024-09-13T14:55:14.915 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.2090000000000002e-06 2024-09-13T14:55:14.915 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.916 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.916 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:14.916 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.056149+0000", 2024-09-13T14:55:14.916 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.1588000000000002e-05 2024-09-13T14:55:14.916 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.916 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.916 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:55:14.916 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.056155+0000", 2024-09-13T14:55:14.916 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.8159999999999996e-06 2024-09-13T14:55:14.916 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.916 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.916 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:14.916 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.233187+0000", 2024-09-13T14:55:14.917 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1770313290000001 2024-09-13T14:55:14.917 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.917 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.917 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:55:14.917 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.233321+0000", 2024-09-13T14:55:14.917 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00013375600000000001 2024-09-13T14:55:14.917 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.917 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.917 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:55:14.917 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.233472+0000", 2024-09-13T14:55:14.917 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00015193199999999999 2024-09-13T14:55:14.917 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.917 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.918 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:55:14.918 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.241018+0000", 2024-09-13T14:55:14.918 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0075451299999999997 2024-09-13T14:55:14.918 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.918 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.918 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:14.918 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.242760+0000", 2024-09-13T14:55:14.918 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.001742328 2024-09-13T14:55:14.918 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.918 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.918 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:14.918 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.247171+0000", 2024-09-13T14:55:14.918 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0044109700000000002 2024-09-13T14:55:14.918 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.919 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.919 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:55:14.919 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.247265+0000", 2024-09-13T14:55:14.919 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.4537000000000006e-05 2024-09-13T14:55:14.919 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.919 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.919 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:55:14.919 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.248919+0000", 2024-09-13T14:55:14.919 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0016532070000000001 2024-09-13T14:55:14.919 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:14.919 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:55:14.919 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:14.919 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.920 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.920 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.36528.0:72281 9.12 9:4e035cfa:::benchmark_data_smithi102_348563_object72280:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T14:55:14.920 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:51:45.056119+0000", 2024-09-13T14:55:14.920 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 209.57007770800001, 2024-09-13T14:55:14.920 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1953139319999999, 2024-09-13T14:55:14.920 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:55:14.920 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:55:14.920 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:14.920 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:55:14.920 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.36528", 2024-09-13T14:55:14.920 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:55:14.920 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 72281 2024-09-13T14:55:14.920 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.921 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:55:14.921 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.921 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:55:14.921 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.056119+0000", 2024-09-13T14:55:14.921 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:14.921 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.921 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.921 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:55:14.921 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.056119+0000", 2024-09-13T14:55:14.921 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:14.921 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.921 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.921 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:55:14.921 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.056120+0000", 2024-09-13T14:55:14.922 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.2600000000000002e-07 2024-09-13T14:55:14.922 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.922 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.922 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:55:14.922 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.056164+0000", 2024-09-13T14:55:14.922 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.4051999999999998e-05 2024-09-13T14:55:14.922 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.922 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.922 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:55:14.922 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.056173+0000", 2024-09-13T14:55:14.922 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.9299999999999992e-06 2024-09-13T14:55:14.922 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.922 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.923 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:55:14.923 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.056178+0000", 2024-09-13T14:55:14.923 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.716e-06 2024-09-13T14:55:14.923 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.923 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.923 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:14.923 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.056220+0000", 2024-09-13T14:55:14.923 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.2079000000000001e-05 2024-09-13T14:55:14.923 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.923 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.923 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:55:14.923 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.056228+0000", 2024-09-13T14:55:14.923 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.9789999999999993e-06 2024-09-13T14:55:14.923 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.924 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.924 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:14.924 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.235277+0000", 2024-09-13T14:55:14.924 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1790484210000001 2024-09-13T14:55:14.924 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.924 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.924 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:55:14.924 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.235422+0000", 2024-09-13T14:55:14.924 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000145518 2024-09-13T14:55:14.924 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.924 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.924 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:55:14.924 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.235596+0000", 2024-09-13T14:55:14.925 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00017405 2024-09-13T14:55:14.925 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.925 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.925 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:55:14.925 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.242515+0000", 2024-09-13T14:55:14.925 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.006918446 2024-09-13T14:55:14.925 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.925 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.925 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:14.925 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.243140+0000", 2024-09-13T14:55:14.925 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00062485299999999998 2024-09-13T14:55:14.925 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.925 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.926 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:14.926 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.249644+0000", 2024-09-13T14:55:14.926 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.006504625 2024-09-13T14:55:14.926 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.926 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.926 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:55:14.926 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.249962+0000", 2024-09-13T14:55:14.926 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00031751700000000002 2024-09-13T14:55:14.926 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.926 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.926 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:55:14.926 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.251433+0000", 2024-09-13T14:55:14.926 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0014712200000000001 2024-09-13T14:55:14.926 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:14.927 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:55:14.927 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:14.927 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.927 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.927 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.36528.0:72296 9.12 9:4ce750ff:::benchmark_data_smithi102_348563_object72295:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e389)", 2024-09-13T14:55:14.927 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:51:45.056189+0000", 2024-09-13T14:55:14.927 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 209.570007605, 2024-09-13T14:55:14.927 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1970618669999999, 2024-09-13T14:55:14.927 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:55:14.927 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:55:14.927 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:14.927 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:55:14.927 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.36528", 2024-09-13T14:55:14.928 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:55:14.928 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 72296 2024-09-13T14:55:14.928 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.928 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:55:14.928 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.928 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:55:14.928 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.056189+0000", 2024-09-13T14:55:14.928 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:14.928 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.928 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.928 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:55:14.928 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.056189+0000", 2024-09-13T14:55:14.929 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:14.929 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.929 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.929 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:55:14.929 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.056190+0000", 2024-09-13T14:55:14.929 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.4200000000000001e-07 2024-09-13T14:55:14.929 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.929 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.929 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:55:14.929 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.056233+0000", 2024-09-13T14:55:14.929 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.3047000000000003e-05 2024-09-13T14:55:14.929 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.929 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.929 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:55:14.930 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.056242+0000", 2024-09-13T14:55:14.930 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.5610000000000004e-06 2024-09-13T14:55:14.930 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.930 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.930 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:55:14.930 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.056248+0000", 2024-09-13T14:55:14.930 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.6690000000000003e-06 2024-09-13T14:55:14.930 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.930 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.930 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:14.930 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.056285+0000", 2024-09-13T14:55:14.930 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.6897999999999998e-05 2024-09-13T14:55:14.930 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.930 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.931 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:55:14.931 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:45.056291+0000", 2024-09-13T14:55:14.931 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.9039999999999997e-06 2024-09-13T14:55:14.931 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.931 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.931 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:14.931 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.237848+0000", 2024-09-13T14:55:14.931 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1815569530000001 2024-09-13T14:55:14.931 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.931 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.931 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:55:14.931 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.238023+0000", 2024-09-13T14:55:14.931 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00017584499999999999 2024-09-13T14:55:14.932 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.932 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.932 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 0,5", 2024-09-13T14:55:14.932 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.238186+0000", 2024-09-13T14:55:14.932 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00016267899999999999 2024-09-13T14:55:14.932 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.932 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.932 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:55:14.932 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.242556+0000", 2024-09-13T14:55:14.932 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0043702200000000002 2024-09-13T14:55:14.932 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.932 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.932 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:14.933 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.243176+0000", 2024-09-13T14:55:14.933 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00061982900000000002 2024-09-13T14:55:14.933 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.933 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.933 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:14.933 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.251472+0000", 2024-09-13T14:55:14.933 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0082953430000000002 2024-09-13T14:55:14.933 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.933 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.933 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:55:14.933 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.251506+0000", 2024-09-13T14:55:14.933 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.4180000000000001e-05 2024-09-13T14:55:14.933 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.934 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.934 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:55:14.934 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:51:46.253251+0000", 2024-09-13T14:55:14.934 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.001745297 2024-09-13T14:55:14.934 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:14.934 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:55:14.934 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:14.934 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.934 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.934 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.36528.0:113806 9.3 9:c568a7a9:::benchmark_data_smithi102_348563_object113805:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e404)", 2024-09-13T14:55:14.934 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:52:28.335435+0000", 2024-09-13T14:55:14.934 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 166.29076136099999, 2024-09-13T14:55:14.934 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2129686470000001, 2024-09-13T14:55:14.935 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:55:14.935 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:55:14.935 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:14.935 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:55:14.935 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.36528", 2024-09-13T14:55:14.935 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:55:14.935 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 113806 2024-09-13T14:55:14.935 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.935 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:55:14.935 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.935 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:55:14.935 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:52:28.335435+0000", 2024-09-13T14:55:14.935 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:14.935 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.936 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.936 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:55:14.936 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:52:28.335435+0000", 2024-09-13T14:55:14.936 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:14.936 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.936 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.936 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:55:14.936 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:52:28.335437+0000", 2024-09-13T14:55:14.936 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.5540000000000001e-06 2024-09-13T14:55:14.936 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.936 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.936 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:55:14.936 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:52:28.335462+0000", 2024-09-13T14:55:14.937 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.5210000000000001e-05 2024-09-13T14:55:14.937 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.937 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.937 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:55:14.937 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:52:28.335480+0000", 2024-09-13T14:55:14.937 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.7410000000000001e-05 2024-09-13T14:55:14.937 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.937 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.937 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:55:14.937 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:52:28.335489+0000", 2024-09-13T14:55:14.937 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.3680000000000008e-06 2024-09-13T14:55:14.937 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.937 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.937 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:14.938 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:52:28.335572+0000", 2024-09-13T14:55:14.938 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.3277999999999998e-05 2024-09-13T14:55:14.938 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.938 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.938 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op must wait for map", 2024-09-13T14:55:14.938 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:52:28.335579+0000", 2024-09-13T14:55:14.938 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.9859999999999997e-06 2024-09-13T14:55:14.938 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.938 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.938 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:14.938 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:52:28.338326+0000", 2024-09-13T14:55:14.938 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.002746353 2024-09-13T14:55:14.938 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.938 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.939 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:55:14.939 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:52:28.338337+0000", 2024-09-13T14:55:14.939 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1303e-05 2024-09-13T14:55:14.939 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.939 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.939 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:14.939 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:52:29.536152+0000", 2024-09-13T14:55:14.939 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.197815512 2024-09-13T14:55:14.939 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.939 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.939 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:55:14.939 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:52:29.536330+0000", 2024-09-13T14:55:14.939 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00017795199999999999 2024-09-13T14:55:14.940 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.940 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.940 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:55:14.940 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:52:29.536529+0000", 2024-09-13T14:55:14.940 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0001983 2024-09-13T14:55:14.940 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.940 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.940 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:14.940 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:52:29.542436+0000", 2024-09-13T14:55:14.940 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0059073449999999996 2024-09-13T14:55:14.940 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.940 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.940 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:55:14.941 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:52:29.542450+0000", 2024-09-13T14:55:14.941 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4453999999999999e-05 2024-09-13T14:55:14.941 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.941 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.941 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:14.941 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:52:29.546938+0000", 2024-09-13T14:55:14.941 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0044880479999999997 2024-09-13T14:55:14.941 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.941 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.941 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:55:14.941 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:52:29.547042+0000", 2024-09-13T14:55:14.941 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000103448 2024-09-13T14:55:14.941 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:14.941 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:14.942 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:55:14.942 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:52:29.548404+0000", 2024-09-13T14:55:14.942 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.001362126 2024-09-13T14:55:14.942 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:14.942 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:55:14.942 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:14.942 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:14.942 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:55:14.942 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:55:14.942 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:14.943 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is set 2024-09-13T14:55:14.973 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:14.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:14.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:15.075 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:15.103 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 1 16 3673 3657 228.55 228.562 0.00578212 0.0043629 2024-09-13T14:55:15.107 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:15.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:15.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:15.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:14 smithi102 ceph-mon[110183]: pgmap v1486: 109 pgs: 109 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:55:15.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:14 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/785102690' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:55:15.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:14 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:55:15.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:14 smithi102 ceph-mon[115269]: pgmap v1486: 109 pgs: 109 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:55:15.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:14 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/785102690' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:55:15.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:14 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:55:15.209 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:15.246 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:15.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:15.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:15.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:14 smithi195 ceph-mon[104262]: pgmap v1486: 109 pgs: 109 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail 2024-09-13T14:55:15.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:14 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/785102690' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:55:15.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:14 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:55:15.348 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:15.382 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:15.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:15.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:15.484 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:15.520 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:15.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:15.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:15.622 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:15.656 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:15.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:15.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:15.762 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:15.794 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:15.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:15.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:15.896 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:15.932 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:15.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:15.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:16.035 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:16.066 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:16.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:16.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:16.103 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 2 16 7213 7197 224.89 221.25 0.00357817 0.0044365 2024-09-13T14:55:16.168 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:16.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:15 smithi102 ceph-mon[110183]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:55:16.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:15 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:55:16.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:15 smithi102 ceph-mon[110183]: osdmap e440: 8 total, 8 up, 8 in 2024-09-13T14:55:16.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:15 smithi102 ceph-mon[115269]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:55:16.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:15 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:55:16.170 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:15 smithi102 ceph-mon[115269]: osdmap e440: 8 total, 8 up, 8 in 2024-09-13T14:55:16.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:16.207 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:16.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:16.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:15 smithi195 ceph-mon[104262]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:55:16.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:15 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:55:16.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:15 smithi195 ceph-mon[104262]: osdmap e440: 8 total, 8 up, 8 in 2024-09-13T14:55:16.308 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:16.342 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:16.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:16.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:16.444 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:16.481 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:16.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:16.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:16.539 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-a[110159]: 2024-09-13T14:55:16.276+0000 7fa3e7f42640 -1 mon.a@0(leader).osd e440 definitely_dead 0 2024-09-13T14:55:16.540 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:55:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:55:16] "GET /metrics HTTP/1.1" 200 36495 "" "Prometheus/2.51.0" 2024-09-13T14:55:16.582 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:16.609 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:16.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:16.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:16.711 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:16.744 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:16.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:16.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:16.846 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:16.880 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:16.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:16.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:16.908 INFO:teuthology.orchestra.run.smithi102.stderr:marked out osd.4. 2024-09-13T14:55:16.982 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:17.004 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:17.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:17.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:17.103 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 3 16 9820 9804 204.229 162.938 0.00164463 0.00459698 2024-09-13T14:55:17.106 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:17.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:17.128 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:17.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:17.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:16 smithi102 ceph-mon[110183]: pgmap v1488: 109 pgs: 109 active+clean; 14 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 MiB/s wr, 19 op/s 2024-09-13T14:55:17.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:16 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1194152084' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-13T14:55:17.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:16 smithi102 ceph-mon[110183]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-13T14:55:17.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:16 smithi102 ceph-mon[115269]: pgmap v1488: 109 pgs: 109 active+clean; 14 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 MiB/s wr, 19 op/s 2024-09-13T14:55:17.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:16 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1194152084' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-13T14:55:17.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:16 smithi102 ceph-mon[115269]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-13T14:55:17.229 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:17.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:16 smithi195 ceph-mon[104262]: pgmap v1488: 109 pgs: 109 active+clean; 14 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 MiB/s wr, 19 op/s 2024-09-13T14:55:17.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:16 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1194152084' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-13T14:55:17.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:16 smithi195 ceph-mon[104262]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-13T14:55:17.251 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:17.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:17.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:17.353 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:17.375 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:17.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:17.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:17.476 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:17.498 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:17.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:17.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:17.600 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:17.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:17.622 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:17.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:17.723 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:17.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:17.745 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:17.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:17.846 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:17.867 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:17.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:17.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:17.969 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:17.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:17.991 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:17.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:18.030 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:17 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1194152084' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-13T14:55:18.030 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:17 smithi102 ceph-mon[115269]: osdmap e441: 8 total, 8 up, 7 in 2024-09-13T14:55:18.030 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:17 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1194152084' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-13T14:55:18.030 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:17 smithi102 ceph-mon[110183]: osdmap e441: 8 total, 8 up, 7 in 2024-09-13T14:55:18.093 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:18.103 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 4 16 9820 9804 153.173 0 - 0.00459698 2024-09-13T14:55:18.116 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:18.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:18.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:18.218 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:18.239 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:18.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:18.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:18.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:17 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1194152084' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-13T14:55:18.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:17 smithi195 ceph-mon[104262]: osdmap e441: 8 total, 8 up, 7 in 2024-09-13T14:55:18.340 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:18.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:18.363 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:18.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:18.464 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:18.486 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:18.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:18.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:18.587 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:18.612 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:18.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:18.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:18.714 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:18.738 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:18.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:18.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:18.840 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:18.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:18.865 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:18.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:18.966 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:18.994 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:18.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:18.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:19.096 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:19.103 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 5 16 9896 9880 123.489 2.375 0.00144259 0.0072823 2024-09-13T14:55:19.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:19.121 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:19.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:19.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:18 smithi102 ceph-mon[110183]: pgmap v1490: 109 pgs: 109 active+clean; 367 MiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 52 MiB/s wr, 839 op/s 2024-09-13T14:55:19.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:18 smithi102 ceph-mon[110183]: osdmap e442: 8 total, 8 up, 7 in 2024-09-13T14:55:19.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:18 smithi102 ceph-mon[115269]: pgmap v1490: 109 pgs: 109 active+clean; 367 MiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 52 MiB/s wr, 839 op/s 2024-09-13T14:55:19.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:18 smithi102 ceph-mon[115269]: osdmap e442: 8 total, 8 up, 7 in 2024-09-13T14:55:19.222 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:19.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:18 smithi195 ceph-mon[104262]: pgmap v1490: 109 pgs: 109 active+clean; 367 MiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 52 MiB/s wr, 839 op/s 2024-09-13T14:55:19.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:18 smithi195 ceph-mon[104262]: osdmap e442: 8 total, 8 up, 7 in 2024-09-13T14:55:19.254 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:19.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:19.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:19.355 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:19.384 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:19.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:19.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:19.486 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:19.527 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:55:19.527 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:55:19.530 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:19.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:19.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:19.569 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:55:19.570 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:55:19.609 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:55:19.609 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:55:19.633 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:19.647 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:55:19.648 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:55:19.659 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:19.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:19.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:19.680 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:55:19.680 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:55:19.712 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:55:19.713 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:55:19.748 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:55:19.749 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:55:19.761 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:19.782 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:55:19.782 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:55:19.790 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:19.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:19.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:19.892 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:19.921 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:19.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:19.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:20.024 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:20.050 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:55:20.053 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:20.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:20.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:20.091 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:19 smithi102 ceph-mon[110183]: osdmap e443: 8 total, 8 up, 7 in 2024-09-13T14:55:20.091 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:19 smithi102 ceph-mon[110183]: osdmap e444: 8 total, 8 up, 7 in 2024-09-13T14:55:20.091 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:19 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2762461565' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:55:20.091 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:19 smithi102 ceph-mon[115269]: osdmap e443: 8 total, 8 up, 7 in 2024-09-13T14:55:20.091 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:19 smithi102 ceph-mon[115269]: osdmap e444: 8 total, 8 up, 7 in 2024-09-13T14:55:20.091 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:19 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2762461565' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:55:20.103 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 6 16 11936 11920 124.152 127.5 0.0039028 0.00804571 2024-09-13T14:55:20.156 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:20.183 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:20.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:20.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:20.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:19 smithi195 ceph-mon[104262]: osdmap e443: 8 total, 8 up, 7 in 2024-09-13T14:55:20.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:19 smithi195 ceph-mon[104262]: osdmap e444: 8 total, 8 up, 7 in 2024-09-13T14:55:20.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:19 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2762461565' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:55:20.286 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:20.314 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:20.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:20.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:20.416 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:20.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:20.463 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:20.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:20.564 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:20.604 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:20.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:20.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:20.706 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:20.733 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:20.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:20.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:20.835 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:20.856 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:20.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:20.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:20.958 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:20.987 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:20.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:20.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:21.088 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:21.103 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 7 16 13968 13952 124.558 127 0.00732256 0.00801974 2024-09-13T14:55:21.116 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:21.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:21.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:21.218 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:21.247 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:21.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:21.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:21.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:20 smithi195 ceph-mon[104262]: pgmap v1494: 109 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 peering, 104 active+clean; 551 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 134 MiB/s wr, 2.15k op/s; 1228/26316 objects degraded (4.666%); 32 KiB/s, 0 objects/s recovering 2024-09-13T14:55:21.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:20 smithi195 ceph-mon[104262]: Health check failed: Degraded data redundancy: 1228/26316 objects degraded (4.666%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:55:21.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:20 smithi195 ceph-mon[104262]: osdmap e445: 8 total, 8 up, 7 in 2024-09-13T14:55:21.349 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:21.388 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:21.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:21.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:21.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:20 smithi102 ceph-mon[110183]: pgmap v1494: 109 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 peering, 104 active+clean; 551 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 134 MiB/s wr, 2.15k op/s; 1228/26316 objects degraded (4.666%); 32 KiB/s, 0 objects/s recovering 2024-09-13T14:55:21.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:20 smithi102 ceph-mon[110183]: Health check failed: Degraded data redundancy: 1228/26316 objects degraded (4.666%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:55:21.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:20 smithi102 ceph-mon[110183]: osdmap e445: 8 total, 8 up, 7 in 2024-09-13T14:55:21.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:20 smithi102 ceph-mon[115269]: pgmap v1494: 109 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 peering, 104 active+clean; 551 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 134 MiB/s wr, 2.15k op/s; 1228/26316 objects degraded (4.666%); 32 KiB/s, 0 objects/s recovering 2024-09-13T14:55:21.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:20 smithi102 ceph-mon[115269]: Health check failed: Degraded data redundancy: 1228/26316 objects degraded (4.666%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:55:21.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:20 smithi102 ceph-mon[115269]: osdmap e445: 8 total, 8 up, 7 in 2024-09-13T14:55:21.490 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:21.527 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:21.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:21.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:21.629 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:21.657 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:21.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:21.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:21.759 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:21.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:21.788 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:21.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:21.889 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:21.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:21.911 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:21.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:22.013 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:22.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:22.035 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:22.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:22.103 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 8 16 15799 15783 123.291 114.438 0.00194292 0.00801727 2024-09-13T14:55:22.136 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:22.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:22.158 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:22.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:22.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:21 smithi195 ceph-mon[104262]: 1.5 scrub starts 2024-09-13T14:55:22.259 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:22.280 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:22.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:22.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:22.382 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:22.404 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:22.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:22.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:22.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:21 smithi102 ceph-mon[110183]: 1.5 scrub starts 2024-09-13T14:55:22.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:21 smithi102 ceph-mon[115269]: 1.5 scrub starts 2024-09-13T14:55:22.509 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:22.530 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:22.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:22.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:22.632 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:22.653 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:22.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:22.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:22.755 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:22.776 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:22.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:22.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:22.878 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:22.899 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:22.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:22.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:23.002 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:23.023 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:23.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:23.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:23.103 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 9 16 15957 15941 110.69 9.875 0.00218482 0.00896134 2024-09-13T14:55:23.125 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:23.147 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:23.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:23.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:23.166 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T14:55:23.230 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:22 smithi102 ceph-mon[110183]: pgmap v1496: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 2 peering, 100 active+clean; 776 MiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 9.2 KiB/s rd, 102 MiB/s wr, 1.65k op/s; 2421/37134 objects degraded (6.520%); 290/37134 objects misplaced (0.781%); 15 MiB/s, 240 objects/s recovering 2024-09-13T14:55:23.230 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:22 smithi102 ceph-mon[110183]: osdmap e446: 8 total, 8 up, 7 in 2024-09-13T14:55:23.230 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:22 smithi102 ceph-mon[115269]: pgmap v1496: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 2 peering, 100 active+clean; 776 MiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 9.2 KiB/s rd, 102 MiB/s wr, 1.65k op/s; 2421/37134 objects degraded (6.520%); 290/37134 objects misplaced (0.781%); 15 MiB/s, 240 objects/s recovering 2024-09-13T14:55:23.230 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:22 smithi102 ceph-mon[115269]: osdmap e446: 8 total, 8 up, 7 in 2024-09-13T14:55:23.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:22 smithi195 ceph-mon[104262]: pgmap v1496: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 2 peering, 100 active+clean; 776 MiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 9.2 KiB/s rd, 102 MiB/s wr, 1.65k op/s; 2421/37134 objects degraded (6.520%); 290/37134 objects misplaced (0.781%); 15 MiB/s, 240 objects/s recovering 2024-09-13T14:55:23.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:22 smithi195 ceph-mon[104262]: osdmap e446: 8 total, 8 up, 7 in 2024-09-13T14:55:23.249 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:23.270 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:23.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:23.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:23.372 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:23.393 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:23.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:23.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:23.495 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:23.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:23.518 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:23.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:23.619 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:23.641 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:23.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:23.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:23.743 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:23.765 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:23.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:23.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:23.867 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:23.887 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:23.891 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:23.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:23.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:23.993 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:24.017 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:24.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:24.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:24.103 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 10 16 16161 16145 100.896 12.75 0.00202035 0.00982669 2024-09-13T14:55:24.120 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:24.142 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:24.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:24.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:24.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:23 smithi102 ceph-mon[110183]: osdmap e447: 8 total, 8 up, 7 in 2024-09-13T14:55:24.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:23 smithi102 ceph-mon[115269]: osdmap e447: 8 total, 8 up, 7 in 2024-09-13T14:55:24.243 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:24.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:23 smithi195 ceph-mon[104262]: osdmap e447: 8 total, 8 up, 7 in 2024-09-13T14:55:24.264 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:24.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:24.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:24.365 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:24.389 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:24.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:24.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:24.491 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:24.512 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:24.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:24.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:24.614 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:24.636 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:24.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:24.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:24.738 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:24.761 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:24.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:24.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:24.862 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:24.884 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:24.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:24.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:24.986 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:25.016 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:25.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:25.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:25.052 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:25.076 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:24 smithi102 ceph-mon[110183]: pgmap v1499: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 2 peering, 98 active+clean; 879 MiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 7.9 KiB/s rd, 94 MiB/s wr, 1.51k op/s; 3719/42066 objects degraded (8.841%); 290/42066 objects misplaced (0.689%); 13 MiB/s, 2 keys/s, 210 objects/s recovering 2024-09-13T14:55:25.077 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:24 smithi102 ceph-mon[110183]: osdmap e448: 8 total, 8 up, 7 in 2024-09-13T14:55:25.077 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:24 smithi102 ceph-mon[115269]: pgmap v1499: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 2 peering, 98 active+clean; 879 MiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 7.9 KiB/s rd, 94 MiB/s wr, 1.51k op/s; 3719/42066 objects degraded (8.841%); 290/42066 objects misplaced (0.689%); 13 MiB/s, 2 keys/s, 210 objects/s recovering 2024-09-13T14:55:25.077 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:24 smithi102 ceph-mon[115269]: osdmap e448: 8 total, 8 up, 7 in 2024-09-13T14:55:25.105 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 11 16 16435 16419 93.2751 17.125 0.00546432 0.0107124 2024-09-13T14:55:25.118 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:25.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:25.156 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:25.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:25.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:24 smithi195 ceph-mon[104262]: pgmap v1499: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 2 peering, 98 active+clean; 879 MiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 7.9 KiB/s rd, 94 MiB/s wr, 1.51k op/s; 3719/42066 objects degraded (8.841%); 290/42066 objects misplaced (0.689%); 13 MiB/s, 2 keys/s, 210 objects/s recovering 2024-09-13T14:55:25.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:24 smithi195 ceph-mon[104262]: osdmap e448: 8 total, 8 up, 7 in 2024-09-13T14:55:25.257 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:25.294 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:25.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:25.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:25.396 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:25.423 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:25.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:25.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:25.525 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:25.563 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:25.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:25.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:25.665 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:25.693 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:25.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:25.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:25.795 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:25.823 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:25.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:25.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:25.840 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:55:25.925 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:25.952 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:25.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:25.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:26.053 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:26.101 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:26.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:26.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:26.104 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 12 16 18548 18532 96.5042 132.062 0.00576688 0.0103555 2024-09-13T14:55:26.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:25 smithi102 ceph-mon[110183]: osdmap e449: 8 total, 8 up, 7 in 2024-09-13T14:55:26.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:25 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:55:26.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:25 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:55:26.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:25 smithi102 ceph-mon[115269]: osdmap e449: 8 total, 8 up, 7 in 2024-09-13T14:55:26.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:25 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:55:26.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:25 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:55:26.202 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:26.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:26.240 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:26.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:26.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:25 smithi195 ceph-mon[104262]: osdmap e449: 8 total, 8 up, 7 in 2024-09-13T14:55:26.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:25 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:55:26.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:25 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:55:26.341 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:26.382 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:26.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:26.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:26.485 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:26.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:26.515 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:26.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:26.616 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:26.640 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:26.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:26.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:26.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:55:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:55:26] "GET /metrics HTTP/1.1" 200 36607 "" "Prometheus/2.51.0" 2024-09-13T14:55:26.742 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:26.763 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:26.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:26.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:26.865 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:26.887 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:26.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:26.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:26.989 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:27.011 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:27.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:27.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:27.105 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 13 16 19620 19604 94.2344 67 0.00159016 0.0101521 2024-09-13T14:55:27.113 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:27.129 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:26 smithi102 ceph-mon[110183]: pgmap v1502: 109 pgs: 1 peering, 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 100 active+clean; 949 MiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 43 MiB/s wr, 688 op/s; 3065/45393 objects degraded (6.752%); 290/45393 objects misplaced (0.639%); 10 MiB/s, 2 keys/s, 168 objects/s recovering 2024-09-13T14:55:27.129 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:26 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 3065/45393 objects degraded (6.752%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:55:27.129 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:26 smithi102 ceph-mon[110183]: osdmap e450: 8 total, 8 up, 7 in 2024-09-13T14:55:27.129 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:55:27.129 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:26 smithi102 ceph-mon[115269]: pgmap v1502: 109 pgs: 1 peering, 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 100 active+clean; 949 MiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 43 MiB/s wr, 688 op/s; 3065/45393 objects degraded (6.752%); 290/45393 objects misplaced (0.639%); 10 MiB/s, 2 keys/s, 168 objects/s recovering 2024-09-13T14:55:27.130 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:26 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 3065/45393 objects degraded (6.752%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:55:27.130 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:26 smithi102 ceph-mon[115269]: osdmap e450: 8 total, 8 up, 7 in 2024-09-13T14:55:27.130 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:55:27.137 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:27.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:27.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:27.239 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:27.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:26 smithi195 ceph-mon[104262]: pgmap v1502: 109 pgs: 1 peering, 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 100 active+clean; 949 MiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 43 MiB/s wr, 688 op/s; 3065/45393 objects degraded (6.752%); 290/45393 objects misplaced (0.639%); 10 MiB/s, 2 keys/s, 168 objects/s recovering 2024-09-13T14:55:27.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:26 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 3065/45393 objects degraded (6.752%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:55:27.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:26 smithi195 ceph-mon[104262]: osdmap e450: 8 total, 8 up, 7 in 2024-09-13T14:55:27.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:55:27.260 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:27.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:27.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:27.362 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:27.383 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:27.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:27.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:27.455 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 6, 7, 5, 2, 1] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T14:55:27.455 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:55:27.455 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-09-13T14:55:27.455 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T14:55:27.455 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T14:55:27.455 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-09-13T14:55:27.485 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:27.510 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:27.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:27.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:27.612 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:27.633 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:27.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:27.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:27.735 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:27.756 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:27.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:27.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:27.858 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:27.879 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:27.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:27.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:27.968 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:27 smithi102 ceph-mon[110183]: 1.5 scrub starts 2024-09-13T14:55:27.968 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:27 smithi102 ceph-mon[110183]: osdmap e451: 8 total, 8 up, 7 in 2024-09-13T14:55:27.981 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:27.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:27 smithi195 ceph-mon[104262]: 1.5 scrub starts 2024-09-13T14:55:27.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:27 smithi195 ceph-mon[104262]: osdmap e451: 8 total, 8 up, 7 in 2024-09-13T14:55:28.003 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:28.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:28.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:28.105 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 14 16 19783 19767 88.2315 10.1875 0.0016239 0.0108795 2024-09-13T14:55:28.105 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:28.126 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:28.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:28.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:28.142 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:55:28.142 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:55:28.229 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:28.251 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:28.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:28.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:28.353 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:28.377 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:28.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:28.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:28.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:27 smithi102 ceph-mon[115269]: 1.5 scrub starts 2024-09-13T14:55:28.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:27 smithi102 ceph-mon[115269]: osdmap e451: 8 total, 8 up, 7 in 2024-09-13T14:55:28.479 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:28.515 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:28.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:28.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:28.616 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:28.645 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:28.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:28.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:28.747 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:28.783 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:28.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:28.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:28.885 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:28.923 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:28.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:28.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:28.974 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T14:55:29.025 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:29.068 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:29.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:29.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:29.105 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 15 15 20936 20921 87.1557 72.125 0.0142064 0.0114662 2024-09-13T14:55:29.172 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:29.201 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:29.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:29.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:29.302 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:29.356 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:29.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:29.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:29.458 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:29.506 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:29.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:29.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:29.597 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:29 smithi102 ceph-mon[110183]: pgmap v1505: 109 pgs: 1 active+recovering+undersized+remapped, 1 peering, 1 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 102 active+clean; 1.1 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s; 1861/54615 objects degraded (3.407%); 404/54615 objects misplaced (0.740%); 27 MiB/s, 435 objects/s recovering 2024-09-13T14:55:29.597 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:29 smithi102 ceph-mon[110183]: osdmap e452: 8 total, 8 up, 7 in 2024-09-13T14:55:29.597 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:29 smithi102 ceph-mon[115269]: pgmap v1505: 109 pgs: 1 active+recovering+undersized+remapped, 1 peering, 1 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 102 active+clean; 1.1 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s; 1861/54615 objects degraded (3.407%); 404/54615 objects misplaced (0.740%); 27 MiB/s, 435 objects/s recovering 2024-09-13T14:55:29.597 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:29 smithi102 ceph-mon[115269]: osdmap e452: 8 total, 8 up, 7 in 2024-09-13T14:55:29.608 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:29.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:29.648 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:29.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:29.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:29 smithi195 ceph-mon[104262]: pgmap v1505: 109 pgs: 1 active+recovering+undersized+remapped, 1 peering, 1 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 102 active+clean; 1.1 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s; 1861/54615 objects degraded (3.407%); 404/54615 objects misplaced (0.740%); 27 MiB/s, 435 objects/s recovering 2024-09-13T14:55:29.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:29 smithi195 ceph-mon[104262]: osdmap e452: 8 total, 8 up, 7 in 2024-09-13T14:55:29.750 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:29.763 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:29.769 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:29.774 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:29.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:29.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:29.777 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:29.876 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:29.902 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:29.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:29.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:30.004 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:30.048 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:30.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:30.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:30.105 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 16 16 22582 22566 88.1336 102.812 0.00695063 0.01134 2024-09-13T14:55:30.151 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:30.176 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:30.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:30.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:30.278 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:30.312 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:30.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:30.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:30.414 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:30.455 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:30.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:30.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:30.510 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is unset 2024-09-13T14:55:30.557 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:30.585 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:30.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:30.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:30.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:30 smithi102 ceph-mon[110183]: osdmap e453: 8 total, 8 up, 7 in 2024-09-13T14:55:30.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:30 smithi102 ceph-mon[110183]: pgmap v1508: 109 pgs: 1 active+recovering+undersized+remapped, 1 peering, 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 103 active+clean; 1.2 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s; 1850/58818 objects degraded (3.145%); 404/58818 objects misplaced (0.687%); 17 MiB/s, 275 objects/s recovering 2024-09-13T14:55:30.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:30 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/4181803988' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:55:30.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:30 smithi102 ceph-mon[115269]: osdmap e453: 8 total, 8 up, 7 in 2024-09-13T14:55:30.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:30 smithi102 ceph-mon[115269]: pgmap v1508: 109 pgs: 1 active+recovering+undersized+remapped, 1 peering, 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 103 active+clean; 1.2 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s; 1850/58818 objects degraded (3.145%); 404/58818 objects misplaced (0.687%); 17 MiB/s, 275 objects/s recovering 2024-09-13T14:55:30.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:30 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/4181803988' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:55:30.687 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:30.726 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:30.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:30.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:30.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:30 smithi195 ceph-mon[104262]: osdmap e453: 8 total, 8 up, 7 in 2024-09-13T14:55:30.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:30 smithi195 ceph-mon[104262]: pgmap v1508: 109 pgs: 1 active+recovering+undersized+remapped, 1 peering, 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 103 active+clean; 1.2 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s; 1850/58818 objects degraded (3.145%); 404/58818 objects misplaced (0.687%); 17 MiB/s, 275 objects/s recovering 2024-09-13T14:55:30.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:30 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/4181803988' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:55:30.828 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:30.862 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:30.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:30.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:30.964 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:31.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:31.014 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:31.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:31.105 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 17 16 24565 24549 90.2389 123.938 0.00771124 0.0110736 2024-09-13T14:55:31.116 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:31.144 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:31.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:31.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:31.246 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:31.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:31.270 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:31.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:31.371 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:31.398 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:31.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:31.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:31.500 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:31.527 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:31.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:31.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:31.629 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:31.650 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:31.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:31.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:31.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:31 smithi102 ceph-mon[115269]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:55:31.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:31 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/4181803988' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:55:31.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:31 smithi102 ceph-mon[115269]: osdmap e454: 8 total, 8 up, 7 in 2024-09-13T14:55:31.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:31 smithi102 ceph-mon[110183]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:55:31.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:31 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/4181803988' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:55:31.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:31 smithi102 ceph-mon[110183]: osdmap e454: 8 total, 8 up, 7 in 2024-09-13T14:55:31.753 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:31.779 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:31.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:31.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:31.881 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:31.905 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:31.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:31.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:31.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:31 smithi195 ceph-mon[104262]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:55:31.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:31 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/4181803988' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:55:31.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:31 smithi195 ceph-mon[104262]: osdmap e454: 8 total, 8 up, 7 in 2024-09-13T14:55:32.007 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:32.029 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:32.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:32.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:32.105 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 18 16 25553 25537 88.656 61.75 0.00173302 0.0110153 2024-09-13T14:55:32.131 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:32.152 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:32.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:32.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:32.253 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:32.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:32.275 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:32.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:32.376 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:32.398 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:32.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:32.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:32.500 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:32.521 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:32.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:32.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:32.624 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:32.663 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:32.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:32.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:32.765 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:32.802 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:32.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:32.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:32.904 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:32.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:32 smithi102 ceph-mon[110183]: osdmap e455: 8 total, 8 up, 7 in 2024-09-13T14:55:32.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:32 smithi102 ceph-mon[110183]: pgmap v1511: 109 pgs: 1 active+recovering+undersized+remapped, 1 peering, 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 104 active+clean; 1.4 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 74 MiB/s wr, 1.18k op/s; 1229/68820 objects degraded (1.786%); 404/68820 objects misplaced (0.587%); 9.7 MiB/s, 157 objects/s recovering 2024-09-13T14:55:32.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:32 smithi102 ceph-mon[115269]: osdmap e455: 8 total, 8 up, 7 in 2024-09-13T14:55:32.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:32 smithi102 ceph-mon[115269]: pgmap v1511: 109 pgs: 1 active+recovering+undersized+remapped, 1 peering, 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 104 active+clean; 1.4 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 74 MiB/s wr, 1.18k op/s; 1229/68820 objects degraded (1.786%); 404/68820 objects misplaced (0.587%); 9.7 MiB/s, 157 objects/s recovering 2024-09-13T14:55:32.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:32.944 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:32.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:32.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:32 smithi195 ceph-mon[104262]: osdmap e455: 8 total, 8 up, 7 in 2024-09-13T14:55:32.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:32 smithi195 ceph-mon[104262]: pgmap v1511: 109 pgs: 1 active+recovering+undersized+remapped, 1 peering, 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 104 active+clean; 1.4 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 74 MiB/s wr, 1.18k op/s; 1229/68820 objects degraded (1.786%); 404/68820 objects misplaced (0.587%); 9.7 MiB/s, 157 objects/s recovering 2024-09-13T14:55:33.045 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:33.074 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:33.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:33.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:33.105 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 19 16 26565 26549 87.3187 63.25 0.016576 0.0114465 2024-09-13T14:55:33.176 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:33.207 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:33.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:33.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:33.309 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:33.339 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:33.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:33.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:33.441 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:33.461 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:33.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:33.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:33.563 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:33.597 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:33.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:33.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:33.699 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:33.735 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:33.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:33.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:33.837 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:33.872 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:33.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:33.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:33.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:33 smithi102 ceph-mon[110183]: Health check update: Degraded data redundancy: 1229/68820 objects degraded (1.786%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:55:33.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:33 smithi102 ceph-mon[110183]: osdmap e456: 8 total, 8 up, 7 in 2024-09-13T14:55:33.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:33 smithi102 ceph-mon[110183]: 1.5 scrub starts 2024-09-13T14:55:33.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:33 smithi102 ceph-mon[110183]: 1.5 scrub ok 2024-09-13T14:55:33.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:33 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:55:33.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:33 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:55:33.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:33 smithi102 ceph-mon[115269]: Health check update: Degraded data redundancy: 1229/68820 objects degraded (1.786%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:55:33.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:33 smithi102 ceph-mon[115269]: osdmap e456: 8 total, 8 up, 7 in 2024-09-13T14:55:33.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:33 smithi102 ceph-mon[115269]: 1.5 scrub starts 2024-09-13T14:55:33.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:33 smithi102 ceph-mon[115269]: 1.5 scrub ok 2024-09-13T14:55:33.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:33 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:55:33.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:33 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:55:33.974 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:33.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:33 smithi195 ceph-mon[104262]: Health check update: Degraded data redundancy: 1229/68820 objects degraded (1.786%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:55:33.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:33 smithi195 ceph-mon[104262]: osdmap e456: 8 total, 8 up, 7 in 2024-09-13T14:55:33.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:33 smithi195 ceph-mon[104262]: 1.5 scrub starts 2024-09-13T14:55:33.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:33 smithi195 ceph-mon[104262]: 1.5 scrub ok 2024-09-13T14:55:33.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:33 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:55:33.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:33 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:55:34.014 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:34.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:34.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:34.106 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:2024-09-13T14:55:34.105537+0000 min lat: 0.00130255 max lat: 2.27933 avg lat: 0.0113665 2024-09-13T14:55:34.106 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:55:34.106 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 20 16 28158 28142 87.9304 99.5625 0.00580403 0.0113665 2024-09-13T14:55:34.116 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:34.158 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:34.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:34.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:34.260 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:34.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:34.285 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:34.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:34.386 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:34.408 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:34.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:34.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:34.510 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:34.531 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:34.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:34.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:34.633 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:34.654 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:34.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:34.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:34.756 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:34.776 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:34.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:34.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:34.878 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:34.900 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:34.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:34.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:35.001 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:35.023 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:35.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:35.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:35.106 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 21 16 28434 28418 84.5648 17.25 0.001915 0.0113222 2024-09-13T14:55:35.125 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:35.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:35.162 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:35.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:35.263 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:35.288 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:35.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:35.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:35.348 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:35 smithi102 ceph-mon[110183]: pgmap v1513: 109 pgs: 1 active+recovering+undersized+remapped, 1 peering, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 1.5 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 73 MiB/s wr, 1.16k op/s; 563/73743 objects degraded (0.763%); 534/73743 objects misplaced (0.724%); 17 MiB/s, 270 objects/s recovering 2024-09-13T14:55:35.348 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:35 smithi102 ceph-mon[110183]: osdmap e457: 8 total, 8 up, 7 in 2024-09-13T14:55:35.348 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:35 smithi102 ceph-mon[115269]: pgmap v1513: 109 pgs: 1 active+recovering+undersized+remapped, 1 peering, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 1.5 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 73 MiB/s wr, 1.16k op/s; 563/73743 objects degraded (0.763%); 534/73743 objects misplaced (0.724%); 17 MiB/s, 270 objects/s recovering 2024-09-13T14:55:35.349 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:35 smithi102 ceph-mon[115269]: osdmap e457: 8 total, 8 up, 7 in 2024-09-13T14:55:35.390 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:35.432 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:35.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:35.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:35.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:35 smithi195 ceph-mon[104262]: pgmap v1513: 109 pgs: 1 active+recovering+undersized+remapped, 1 peering, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 1.5 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 73 MiB/s wr, 1.16k op/s; 563/73743 objects degraded (0.763%); 534/73743 objects misplaced (0.724%); 17 MiB/s, 270 objects/s recovering 2024-09-13T14:55:35.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:35 smithi195 ceph-mon[104262]: osdmap e457: 8 total, 8 up, 7 in 2024-09-13T14:55:35.534 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:35.573 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:35.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:35.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:35.675 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:35.726 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:35.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:35.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:35.828 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:35.858 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:35.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:35.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:35.960 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:36.004 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:36.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:36.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:36.106 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 22 16 30288 30272 85.9854 115.875 0.00845752 0.0116243 2024-09-13T14:55:36.107 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:36.149 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:36.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:36.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:36.251 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:36.276 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:36.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:36.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:36.360 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:55:36.360 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:55:36.377 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:36.398 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:36.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:36.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:36.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:36 smithi102 ceph-mon[110183]: osdmap e458: 8 total, 8 up, 7 in 2024-09-13T14:55:36.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:36 smithi102 ceph-mon[115269]: osdmap e458: 8 total, 8 up, 7 in 2024-09-13T14:55:36.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:36 smithi195 ceph-mon[104262]: osdmap e458: 8 total, 8 up, 7 in 2024-09-13T14:55:36.501 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:36.522 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:36.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:36.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:36.624 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:36.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:36.645 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:36.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:36.746 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:36.756 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:55:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:55:36] "GET /metrics HTTP/1.1" 200 36607 "" "Prometheus/2.51.0" 2024-09-13T14:55:36.770 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:36.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:36.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:36.871 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:36.892 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:36.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:36.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:36.994 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:37.016 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:37.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:37.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:37.106 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 23 16 30634 30618 83.1873 21.625 0.0018787 0.0115612 2024-09-13T14:55:37.118 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:37.139 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:37.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:37.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:37.241 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:37.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:37.280 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:37.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:37.381 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:37.409 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:37.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:37.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:37.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:37 smithi102 ceph-mon[110183]: pgmap v1516: 109 pgs: 2 active+recovering+undersized+remapped, 1 active+undersized+remapped, 106 active+clean; 1.6 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 55 MiB/s wr, 875 op/s; 1025/77790 objects misplaced (1.318%); 8.9 MiB/s, 143 objects/s recovering 2024-09-13T14:55:37.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:37 smithi102 ceph-mon[110183]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 563/73743 objects degraded (0.763%), 1 pg degraded) 2024-09-13T14:55:37.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:37 smithi102 ceph-mon[110183]: osdmap e459: 8 total, 8 up, 7 in 2024-09-13T14:55:37.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:37 smithi102 ceph-mon[115269]: pgmap v1516: 109 pgs: 2 active+recovering+undersized+remapped, 1 active+undersized+remapped, 106 active+clean; 1.6 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 55 MiB/s wr, 875 op/s; 1025/77790 objects misplaced (1.318%); 8.9 MiB/s, 143 objects/s recovering 2024-09-13T14:55:37.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:37 smithi102 ceph-mon[115269]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 563/73743 objects degraded (0.763%), 1 pg degraded) 2024-09-13T14:55:37.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:37 smithi102 ceph-mon[115269]: osdmap e459: 8 total, 8 up, 7 in 2024-09-13T14:55:37.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:37 smithi195 ceph-mon[104262]: pgmap v1516: 109 pgs: 2 active+recovering+undersized+remapped, 1 active+undersized+remapped, 106 active+clean; 1.6 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 55 MiB/s wr, 875 op/s; 1025/77790 objects misplaced (1.318%); 8.9 MiB/s, 143 objects/s recovering 2024-09-13T14:55:37.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:37 smithi195 ceph-mon[104262]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 563/73743 objects degraded (0.763%), 1 pg degraded) 2024-09-13T14:55:37.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:37 smithi195 ceph-mon[104262]: osdmap e459: 8 total, 8 up, 7 in 2024-09-13T14:55:37.511 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:37.546 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:37.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:37.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:37.648 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:37.674 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:37.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:37.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:37.776 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:37.802 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:37.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:37.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:37.903 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:37.931 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:37.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:37.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:38.032 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:38.038 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T14:55:38.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:38.122 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:38.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:38.123 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 24 16 32533 32517 84.6659 118.688 0.00516546 0.0118038 2024-09-13T14:55:38.223 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:38.251 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:38.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:38.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:38.303 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:38 smithi102 ceph-mon[110183]: osdmap e460: 8 total, 8 up, 7 in 2024-09-13T14:55:38.303 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:38 smithi102 ceph-mon[115269]: osdmap e460: 8 total, 8 up, 7 in 2024-09-13T14:55:38.352 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:38.399 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:38.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:38.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:38.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:38 smithi195 ceph-mon[104262]: osdmap e460: 8 total, 8 up, 7 in 2024-09-13T14:55:38.501 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:38.527 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:38.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:38.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:38.629 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:38.666 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:38.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:38.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:38.701 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:55:38.701 INFO:teuthology.orchestra.run.smithi102.stdout: "success": "" 2024-09-13T14:55:38.701 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:55:38.768 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:38.782 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:55:38.811 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:38.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:38.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:38.914 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:38.940 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:38.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:38.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:38.984 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:55:38.985 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:55:39.032 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:55:39.032 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:55:39.041 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:39.064 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:55:39.064 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:55:39.084 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:39.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:39.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:39.097 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:55:39.097 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:55:39.108 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 25 16 34197 34181 85.4341 104 0.00781339 0.0116994 2024-09-13T14:55:39.131 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:55:39.131 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:55:39.162 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:55:39.163 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:55:39.186 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:39.194 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:55:39.194 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:55:39.227 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:55:39.227 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:55:39.231 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:39.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:39.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:39.244 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:39 smithi102 ceph-mon[110183]: pgmap v1519: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 1.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 80 MiB/s wr, 1.28k op/s; 491/89184 objects misplaced (0.551%); 9.4 MiB/s, 151 objects/s recovering 2024-09-13T14:55:39.244 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:39 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2436182576' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:55:39.245 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:39 smithi102 ceph-mon[115269]: pgmap v1519: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 1.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 80 MiB/s wr, 1.28k op/s; 491/89184 objects misplaced (0.551%); 9.4 MiB/s, 151 objects/s recovering 2024-09-13T14:55:39.245 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:39 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2436182576' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:55:39.333 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:39.372 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:39.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:39.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:39.474 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:39.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:39 smithi195 ceph-mon[104262]: pgmap v1519: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 1.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 80 MiB/s wr, 1.28k op/s; 491/89184 objects misplaced (0.551%); 9.4 MiB/s, 151 objects/s recovering 2024-09-13T14:55:39.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:39 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2436182576' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:55:39.502 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:39.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:39.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:39.603 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:39.631 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:39.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:39.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:39.733 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:39.759 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:39.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:39.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:39.860 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:39.887 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:39.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:39.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:39.988 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:40.014 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:40.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:40.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:40.108 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 26 16 35898 35882 86.2367 106.312 0.0109263 0.0115899 2024-09-13T14:55:40.115 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:40.143 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:40.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:40.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:40.245 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:40.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:40.271 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:40.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:40.372 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:40.406 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:40.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:40.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:40.508 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:40.529 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:40.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:40.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:40.631 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:40.659 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:40.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:40.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:40.761 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:40.789 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:40.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:40.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:40.891 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:40.924 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:40.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:40.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:41.026 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:41.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:41.067 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:41.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:41.108 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 27 16 37460 37444 86.6581 97.625 0.00477457 0.0115341 2024-09-13T14:55:41.169 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:41.198 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:41.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:41.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:41.300 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:41.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:41.337 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:41.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:41.438 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:41.475 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:41.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:41.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:41.577 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:41.614 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:41.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:41.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:41.716 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:41.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:41.745 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:41.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:41.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:41 smithi195 ceph-mon[104262]: pgmap v1520: 109 pgs: 109 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 78 MiB/s wr, 1.24k op/s; 12 MiB/s, 194 objects/s recovering 2024-09-13T14:55:41.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:41 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:55:41.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:41 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:55:41.846 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:41.884 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:41.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:41.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:41.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:41 smithi102 ceph-mon[110183]: pgmap v1520: 109 pgs: 109 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 78 MiB/s wr, 1.24k op/s; 12 MiB/s, 194 objects/s recovering 2024-09-13T14:55:41.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:41 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:55:41.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:41 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:55:41.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:41 smithi102 ceph-mon[115269]: pgmap v1520: 109 pgs: 109 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 78 MiB/s wr, 1.24k op/s; 12 MiB/s, 194 objects/s recovering 2024-09-13T14:55:41.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:41 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:55:41.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:41 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:55:41.986 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:42.013 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:42.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:42.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:42.108 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 28 16 39160 39144 87.3574 106.25 0.0108545 0.0114423 2024-09-13T14:55:42.115 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:42.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:42.156 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:42.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:42.257 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:42.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:42.295 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:42.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:42.396 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:42.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:42.418 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:42.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:42.519 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:42.558 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:42.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:42.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:42.660 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:42.689 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:42.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:42.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:42.793 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:42.830 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:42.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:42.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:42.932 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:42.959 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:42.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:42.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:43.061 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:43.101 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:43.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:43.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:43.108 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 29 16 40555 40539 87.3513 87.1875 0.0143736 0.011442 2024-09-13T14:55:43.203 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:43.229 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:43.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:43.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:43.331 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:43.358 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:43.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:43.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:43.459 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:43.486 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:43.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:43.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:43.588 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:43.625 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:43.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:43.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:43.727 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:43.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:43 smithi195 ceph-mon[104262]: pgmap v1521: 109 pgs: 109 active+clean; 2.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 93 MiB/s wr, 1.48k op/s; 9.6 MiB/s, 154 objects/s recovering 2024-09-13T14:55:43.770 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:43.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:43.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:43.871 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:43.879 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:43 smithi102 ceph-mon[110183]: pgmap v1521: 109 pgs: 109 active+clean; 2.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 93 MiB/s wr, 1.48k op/s; 9.6 MiB/s, 154 objects/s recovering 2024-09-13T14:55:43.879 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:43 smithi102 ceph-mon[115269]: pgmap v1521: 109 pgs: 109 active+clean; 2.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 93 MiB/s wr, 1.48k op/s; 9.6 MiB/s, 154 objects/s recovering 2024-09-13T14:55:43.908 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:43.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:43.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:44.009 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:44.036 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:44.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:44.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:44.109 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 30 16 42167 42151 87.796 100.75 0.0100187 0.0113841 2024-09-13T14:55:44.138 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:44.159 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:44.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:44.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:44.261 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:44.288 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:44.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:44.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:44.390 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:44.419 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:44.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:44.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:44.521 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:44.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:44.558 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:44.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:44.659 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:44.695 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:44.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:44.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:44.797 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:44.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:44.838 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:44.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:44.939 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:44.968 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:44.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:44.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:45.069 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:45.090 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:45.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:45.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:45.109 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 31 16 43675 43659 88.0038 94.25 0.00201234 0.0113356 2024-09-13T14:55:45.192 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:45.219 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:45.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:45.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:45.321 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:45.346 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:45.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:45.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:45.448 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:45.488 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:45.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:45.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:45.590 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:45.636 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:45.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:45.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:45.738 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:45.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:45 smithi195 ceph-mon[104262]: pgmap v1522: 109 pgs: 109 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 92 MiB/s wr, 1.48k op/s; 8.0 MiB/s, 128 objects/s recovering 2024-09-13T14:55:45.764 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:45.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:45.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:45.866 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:45.904 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:45.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:45.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:45.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:45 smithi102 ceph-mon[110183]: pgmap v1522: 109 pgs: 109 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 92 MiB/s wr, 1.48k op/s; 8.0 MiB/s, 128 objects/s recovering 2024-09-13T14:55:45.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:45 smithi102 ceph-mon[115269]: pgmap v1522: 109 pgs: 109 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 92 MiB/s wr, 1.48k op/s; 8.0 MiB/s, 128 objects/s recovering 2024-09-13T14:55:46.006 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:46.074 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:46.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:46.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:46.109 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 32 16 45274 45258 88.3764 99.9375 0.0126019 0.0113099 2024-09-13T14:55:46.177 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:46.201 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:46.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:46.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:46.221 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:46.259 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:46.303 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:46.353 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:46.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:46.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:46.454 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:46.496 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:46.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:46.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:46.598 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:46.622 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:46.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:46.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:46.634 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:55:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:55:46] "GET /metrics HTTP/1.1" 200 36581 "" "Prometheus/2.51.0" 2024-09-13T14:55:46.724 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:46.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:46.774 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:46.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:46.875 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:46.911 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:46.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:46.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:47.013 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:47.041 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:47.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:47.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:47.109 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 33 16 46784 46768 88.5579 94.375 0.00755039 0.011286 2024-09-13T14:55:47.143 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:47.181 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:47.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:47.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:47.283 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:47.310 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:47.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:47.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:47.411 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:47.440 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:47.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:47.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:47.542 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:47.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:47.569 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:47.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:47.670 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:47.694 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:47.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:47.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:47.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:47 smithi195 ceph-mon[104262]: pgmap v1523: 109 pgs: 109 active+clean; 2.4 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 77 MiB/s wr, 1.24k op/s; 6.6 MiB/s, 106 objects/s recovering 2024-09-13T14:55:47.796 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:47.818 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:47.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:47.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:47.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:47 smithi102 ceph-mon[110183]: pgmap v1523: 109 pgs: 109 active+clean; 2.4 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 77 MiB/s wr, 1.24k op/s; 6.6 MiB/s, 106 objects/s recovering 2024-09-13T14:55:47.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:47 smithi102 ceph-mon[115269]: pgmap v1523: 109 pgs: 109 active+clean; 2.4 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 77 MiB/s wr, 1.24k op/s; 6.6 MiB/s, 106 objects/s recovering 2024-09-13T14:55:47.919 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:47.953 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:47.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:47.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:47.998 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:55:48.055 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:48.080 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:48.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:48.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:48.109 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 34 16 48236 48220 88.6222 90.75 0.00519722 0.0112786 2024-09-13T14:55:48.182 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:48.226 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:48.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:48.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:48.328 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:48.354 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:48.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:48.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:48.456 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:48.482 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:48.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:48.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:48.584 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:48.624 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:48.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:48.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:48.726 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:48.761 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:48.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:48.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:48.863 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:48.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:48.902 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:48.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:49.003 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:49.029 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:49.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:49.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:49.109 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 35 16 49752 49736 88.797 94.75 0.0104266 0.0112564 2024-09-13T14:55:49.131 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:49.158 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:49.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:49.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:49.260 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:49.300 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:49.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:49.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:49.403 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:49.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:49.437 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:49.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:49.538 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:49.578 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:49.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:49.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:49.679 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:49.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:49.705 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:49.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:49.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:49 smithi195 ceph-mon[104262]: pgmap v1524: 109 pgs: 109 active+clean; 2.7 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 87 MiB/s wr, 1.40k op/s; 2.9 MiB/s, 46 objects/s recovering 2024-09-13T14:55:49.806 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:49.852 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:49.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:49.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:49.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:49 smithi102 ceph-mon[110183]: pgmap v1524: 109 pgs: 109 active+clean; 2.7 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 87 MiB/s wr, 1.40k op/s; 2.9 MiB/s, 46 objects/s recovering 2024-09-13T14:55:49.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:49 smithi102 ceph-mon[115269]: pgmap v1524: 109 pgs: 109 active+clean; 2.7 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 87 MiB/s wr, 1.40k op/s; 2.9 MiB/s, 46 objects/s recovering 2024-09-13T14:55:49.954 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:49.982 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:49.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:49.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:50.084 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:50.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:50.109 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:50.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:50.110 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 36 16 51166 51150 88.7851 88.375 0.0122347 0.0112565 2024-09-13T14:55:50.210 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:50.246 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:50.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:50.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:50.348 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:50.390 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:50.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:50.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:50.492 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:50.532 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:50.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:50.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:50.634 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:50.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:50.674 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:50.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:50.775 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:50.813 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:50.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:50.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:50.915 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:50.940 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:50.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:50.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:51.042 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:51.063 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:51.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:51.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:51.109 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 37 16 52471 52455 88.5897 81.5625 0.0146841 0.0112826 2024-09-13T14:55:51.165 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:51.171 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 6, 7, 5, 2, 1] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T14:55:51.171 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:55:51.171 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-09-13T14:55:51.171 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T14:55:51.171 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T14:55:51.171 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-09-13T14:55:51.200 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:51.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:51.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:51.302 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:51.344 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:51.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:51.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:51.429 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:55:51.446 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:51.478 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:51.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:51.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:51.580 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:51.609 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:51.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:51.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:51.711 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:51.735 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:51 smithi195 ceph-mon[104262]: pgmap v1525: 109 pgs: 109 active+clean; 2.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 89 MiB/s wr, 1.43k op/s; 2.6 MiB/s, 40 objects/s recovering 2024-09-13T14:55:51.746 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:51.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:51.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:51.848 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:51.872 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:51.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:51.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:51.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:51 smithi102 ceph-mon[110183]: pgmap v1525: 109 pgs: 109 active+clean; 2.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 89 MiB/s wr, 1.43k op/s; 2.6 MiB/s, 40 objects/s recovering 2024-09-13T14:55:51.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:51 smithi102 ceph-mon[115269]: pgmap v1525: 109 pgs: 109 active+clean; 2.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 89 MiB/s wr, 1.43k op/s; 2.6 MiB/s, 40 objects/s recovering 2024-09-13T14:55:51.974 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:51.999 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:52.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:52.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:52.101 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:52.110 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 38 16 53639 53623 88.1793 73 0.0109708 0.0113344 2024-09-13T14:55:52.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:52.142 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:52.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:52.243 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:52.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:52.279 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:52.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:52.380 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:52.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:52.417 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:52.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:52.518 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:52.539 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:52.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:52.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:52.641 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:52.667 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:52.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:52.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:52.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:52 smithi195 ceph-mon[104262]: pgmap v1526: 109 pgs: 109 active+clean; 3.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-09-13T14:55:52.769 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:52.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:52.808 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:52.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:52.909 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:52.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:52 smithi102 ceph-mon[110183]: pgmap v1526: 109 pgs: 109 active+clean; 3.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-09-13T14:55:52.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:52 smithi102 ceph-mon[115269]: pgmap v1526: 109 pgs: 109 active+clean; 3.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-09-13T14:55:52.940 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:52.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:52.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:53.042 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:53.081 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:53.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:53.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:53.112 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 39 16 55028 55012 88.144 86.8125 0.00961426 0.0113392 2024-09-13T14:55:53.183 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:53.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:53.205 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:53.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:53.306 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:53.331 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:53.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:53.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:53.433 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:53.454 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:53.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:53.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:53.481 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:55:53.481 INFO:teuthology.orchestra.run.smithi195.stdout: "success": "" 2024-09-13T14:55:53.481 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:55:53.556 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:53.576 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:53.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:53.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:53.678 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:53.711 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:53.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:53.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:53.813 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:53.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:53.850 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:53.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:53.951 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:53.990 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:53.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:53.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:54.092 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:54.110 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:2024-09-13T14:55:54.109774+0000 min lat: 0.00130255 max lat: 2.27933 avg lat: 0.011472 2024-09-13T14:55:54.110 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:55:54.110 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 40 15 55782 55767 87.1201 47.1875 0.013191 0.011472 2024-09-13T14:55:54.123 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:54.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:54.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:54.226 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:54.269 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:54.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:54.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:54.370 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:54.429 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:54.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:54.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:54.532 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:54.560 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:54.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:54.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:54.662 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:54.699 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:54.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:54.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:54.801 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:54.830 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:54.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:54.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:54.932 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:54.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:54.971 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:54.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:55.072 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:55.101 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:55.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:55.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:55.110 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 41 16 57195 57179 87.1473 88.25 0.0102651 0.0114686 2024-09-13T14:55:55.139 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:54 smithi102 ceph-mon[115269]: pgmap v1527: 109 pgs: 109 active+clean; 3.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-13T14:55:55.139 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:54 smithi102 ceph-mon[110183]: pgmap v1527: 109 pgs: 109 active+clean; 3.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-13T14:55:55.141 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:54 smithi195 ceph-mon[104262]: pgmap v1527: 109 pgs: 109 active+clean; 3.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-13T14:55:55.203 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:55.252 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:55.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:55.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:55.354 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:55.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:55.404 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:55.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:55.505 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:55.530 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:55.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:55.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:55.632 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:55.673 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:55.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:55.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:55.775 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:55.801 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:55.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:55.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:55.903 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:55.930 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:55.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:55.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:56.004 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:55 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:55:56.005 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:55 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:55:56.032 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:56.074 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:56.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:56.082 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:56.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:56.112 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 42 16 58491 58475 86.9975 81 0.0147003 0.0114872 2024-09-13T14:55:56.184 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:56.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:56.229 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:56.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:56.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:55 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:55:56.330 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:56.372 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:56.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:56.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:56.474 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:56.520 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:56.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:56.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:56.598 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:55:56 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:55:56] "GET /metrics HTTP/1.1" 200 36564 "" "Prometheus/2.51.0" 2024-09-13T14:55:56.621 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:56.667 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:56.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:56.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:56.769 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:56.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:56.803 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:56.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:56.904 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:56.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:56.948 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:56.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:57.049 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:57.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:57.075 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:57.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:57.112 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 43 16 59633 59617 86.6341 71.375 0.0138599 0.0115357 2024-09-13T14:55:57.139 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:56 smithi102 ceph-mon[110183]: pgmap v1528: 109 pgs: 109 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-09-13T14:55:57.139 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:56 smithi102 ceph-mon[115269]: pgmap v1528: 109 pgs: 109 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-09-13T14:55:57.176 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:57.201 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:57.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:57.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:57.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:56 smithi195 ceph-mon[104262]: pgmap v1528: 109 pgs: 109 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-09-13T14:55:57.304 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:57.344 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:57.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:57.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:57.446 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:57.483 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:57.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:57.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:57.585 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:57.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:57.608 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:57.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:57.710 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:57.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:57.748 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:57.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:57.849 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:57.873 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:57.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:57.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:57.975 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:58.017 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:58.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:58.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:58.112 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 44 16 60789 60773 86.3071 72.25 0.00810942 0.0115811 2024-09-13T14:55:58.118 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:58.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:58.148 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:58.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:58.249 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:58.286 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:58.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:58.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:58.388 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:58.425 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:58.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:58.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:58.527 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:58.565 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:58.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:58.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:58.667 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:58.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:58.702 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:58.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:58.804 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:58.841 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:58.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:58.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:58.943 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:58.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:58.987 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:58.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:59.064 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:55:58 smithi102 ceph-mon[110183]: pgmap v1529: 109 pgs: 109 active+clean; 3.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-09-13T14:55:59.064 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:55:58 smithi102 ceph-mon[115269]: pgmap v1529: 109 pgs: 109 active+clean; 3.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-09-13T14:55:59.065 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:55:59.065 INFO:teuthology.orchestra.run.smithi102.stdout: "size": 20, 2024-09-13T14:55:59.065 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 600, 2024-09-13T14:55:59.065 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [ 2024-09-13T14:55:59.065 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.065 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:133957 8.7 8:e8800331:::benchmark_data_smithi102_313275_object3370:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e337)", 2024-09-13T14:55:59.065 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:47:47.859677+0000", 2024-09-13T14:55:59.065 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 491.20157211999998, 2024-09-13T14:55:59.065 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4148750569999999, 2024-09-13T14:55:59.065 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:55:59.066 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:55:59.066 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:59.066 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:55:59.066 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:55:59.066 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:55:59.066 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 133957 2024-09-13T14:55:59.066 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.066 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:55:59.066 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.066 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:55:59.066 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859677+0000", 2024-09-13T14:55:59.066 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:59.066 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.066 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.067 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:55:59.067 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859677+0000", 2024-09-13T14:55:59.067 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:59.067 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.067 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.067 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:55:59.067 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859679+0000", 2024-09-13T14:55:59.067 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.5740000000000001e-06 2024-09-13T14:55:59.067 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.067 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.067 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:55:59.067 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859684+0000", 2024-09-13T14:55:59.067 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.6460000000000001e-06 2024-09-13T14:55:59.068 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.068 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.068 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:55:59.068 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859701+0000", 2024-09-13T14:55:59.068 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.6807e-05 2024-09-13T14:55:59.068 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.068 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.068 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:55:59.068 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859711+0000", 2024-09-13T14:55:59.068 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0729e-05 2024-09-13T14:55:59.068 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.068 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.069 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:59.069 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859782+0000", 2024-09-13T14:55:59.069 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.1051999999999997e-05 2024-09-13T14:55:59.069 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.069 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.069 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op must wait for map", 2024-09-13T14:55:59.069 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859789+0000", 2024-09-13T14:55:59.069 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.1380000000000001e-06 2024-09-13T14:55:59.069 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.069 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.069 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:59.069 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.863617+0000", 2024-09-13T14:55:59.069 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0038276289999999999 2024-09-13T14:55:59.070 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.070 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.070 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:55:59.070 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.863626+0000", 2024-09-13T14:55:59.070 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.8699999999999998e-06 2024-09-13T14:55:59.070 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.070 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.070 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:59.070 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.258622+0000", 2024-09-13T14:55:59.070 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3949958790000001 2024-09-13T14:55:59.070 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.070 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.071 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:55:59.071 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.261508+0000", 2024-09-13T14:55:59.071 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0028864149999999998 2024-09-13T14:55:59.071 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.071 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.071 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,3,6", 2024-09-13T14:55:59.071 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.261752+0000", 2024-09-13T14:55:59.071 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00024399600000000001 2024-09-13T14:55:59.071 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.071 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.071 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:55:59.071 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.271583+0000", 2024-09-13T14:55:59.071 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0098305049999999998 2024-09-13T14:55:59.072 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.072 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.072 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:59.072 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.271621+0000", 2024-09-13T14:55:59.072 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.7920999999999998e-05 2024-09-13T14:55:59.072 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.072 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.072 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:59.072 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.271687+0000", 2024-09-13T14:55:59.072 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.6130000000000006e-05 2024-09-13T14:55:59.072 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.072 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.073 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:59.073 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.274325+0000", 2024-09-13T14:55:59.073 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.002638404 2024-09-13T14:55:59.073 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.073 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.073 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:55:59.073 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.274389+0000", 2024-09-13T14:55:59.073 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.3415000000000002e-05 2024-09-13T14:55:59.073 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.073 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.073 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:55:59.073 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.274553+0000", 2024-09-13T14:55:59.073 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00016394700000000001 2024-09-13T14:55:59.074 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:59.074 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:55:59.074 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:59.074 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.074 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.074 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:133967 8.a 8:5f281cc1:::benchmark_data_smithi102_313275_object3380:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e337)", 2024-09-13T14:55:59.074 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:47:47.859736+0000", 2024-09-13T14:55:59.074 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 491.20151322599997, 2024-09-13T14:55:59.074 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4400547020000001, 2024-09-13T14:55:59.074 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:55:59.074 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:55:59.074 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:59.074 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:55:59.074 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:55:59.074 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:55:59.075 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 133967 2024-09-13T14:55:59.075 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.075 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:55:59.075 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.075 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:55:59.075 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859736+0000", 2024-09-13T14:55:59.075 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:59.075 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.075 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.075 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:55:59.075 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859736+0000", 2024-09-13T14:55:59.075 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:59.075 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.075 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.076 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:55:59.076 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859737+0000", 2024-09-13T14:55:59.076 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.6000000000000004e-07 2024-09-13T14:55:59.076 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.076 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.076 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:55:59.076 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859740+0000", 2024-09-13T14:55:59.076 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.6730000000000001e-06 2024-09-13T14:55:59.076 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.076 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.076 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:55:59.076 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859756+0000", 2024-09-13T14:55:59.076 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.6129000000000001e-05 2024-09-13T14:55:59.077 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.077 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.077 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:55:59.077 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859762+0000", 2024-09-13T14:55:59.077 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.1489999999999999e-06 2024-09-13T14:55:59.077 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.077 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.077 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:59.077 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859843+0000", 2024-09-13T14:55:59.077 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.1382000000000001e-05 2024-09-13T14:55:59.077 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.077 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.077 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op must wait for map", 2024-09-13T14:55:59.078 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859856+0000", 2024-09-13T14:55:59.078 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2598e-05 2024-09-13T14:55:59.078 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.078 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.078 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:59.078 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.862672+0000", 2024-09-13T14:55:59.078 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0028164650000000002 2024-09-13T14:55:59.078 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.078 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.078 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:55:59.078 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.862682+0000", 2024-09-13T14:55:59.078 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.4639999999999995e-06 2024-09-13T14:55:59.078 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.078 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.079 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:59.079 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.283265+0000", 2024-09-13T14:55:59.079 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.420583427 2024-09-13T14:55:59.079 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.079 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.079 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:55:59.079 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.283382+0000", 2024-09-13T14:55:59.079 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000116984 2024-09-13T14:55:59.079 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.080 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.080 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,3,5", 2024-09-13T14:55:59.080 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.283527+0000", 2024-09-13T14:55:59.080 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00014459600000000001 2024-09-13T14:55:59.080 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.080 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.080 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:59.080 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.291391+0000", 2024-09-13T14:55:59.081 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0078640450000000001 2024-09-13T14:55:59.081 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.081 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.081 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:59.081 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.291431+0000", 2024-09-13T14:55:59.081 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.0323e-05 2024-09-13T14:55:59.081 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.081 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.081 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:59.081 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.299553+0000", 2024-09-13T14:55:59.081 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0081221090000000006 2024-09-13T14:55:59.081 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.081 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.082 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:55:59.082 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.299567+0000", 2024-09-13T14:55:59.082 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4011e-05 2024-09-13T14:55:59.082 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.082 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.082 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:55:59.082 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.299679+0000", 2024-09-13T14:55:59.082 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00011214600000000001 2024-09-13T14:55:59.082 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.082 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.082 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:55:59.082 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.299791+0000", 2024-09-13T14:55:59.083 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000111641 2024-09-13T14:55:59.083 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:59.083 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:55:59.083 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:59.083 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.083 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.083 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:133969 8.7 8:e4b3c894:::benchmark_data_smithi102_313275_object3382:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e337)", 2024-09-13T14:55:59.084 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:47:47.859781+0000", 2024-09-13T14:55:59.084 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 491.20146874199997, 2024-09-13T14:55:59.084 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.430809029, 2024-09-13T14:55:59.084 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:55:59.084 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:55:59.084 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:59.084 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:55:59.084 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:55:59.084 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:55:59.084 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 133969 2024-09-13T14:55:59.084 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.084 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:55:59.084 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.085 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:55:59.085 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859781+0000", 2024-09-13T14:55:59.085 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:59.085 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.085 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.085 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:55:59.085 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859781+0000", 2024-09-13T14:55:59.085 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:59.086 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.086 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.086 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:55:59.086 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859781+0000", 2024-09-13T14:55:59.086 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.1399999999999997e-07 2024-09-13T14:55:59.086 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.086 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.086 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:55:59.086 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859784+0000", 2024-09-13T14:55:59.086 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.65e-06 2024-09-13T14:55:59.086 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.086 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.086 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:55:59.087 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859791+0000", 2024-09-13T14:55:59.087 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.3139999999999998e-06 2024-09-13T14:55:59.087 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.087 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.087 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:55:59.087 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859797+0000", 2024-09-13T14:55:59.087 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.4909999999999996e-06 2024-09-13T14:55:59.087 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.087 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.087 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:59.087 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859848+0000", 2024-09-13T14:55:59.087 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.1174999999999997e-05 2024-09-13T14:55:59.087 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.088 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.088 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting_for_map not empty", 2024-09-13T14:55:59.088 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859860+0000", 2024-09-13T14:55:59.088 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1762e-05 2024-09-13T14:55:59.088 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.088 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.088 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:59.088 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.863650+0000", 2024-09-13T14:55:59.088 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0037906820000000001 2024-09-13T14:55:59.088 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.088 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.088 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:55:59.089 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.863657+0000", 2024-09-13T14:55:59.089 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.8460000000000002e-06 2024-09-13T14:55:59.089 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.089 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.089 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:59.089 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.274593+0000", 2024-09-13T14:55:59.089 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.410936153 2024-09-13T14:55:59.089 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.089 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.089 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:55:59.089 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.274720+0000", 2024-09-13T14:55:59.089 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00012619899999999999 2024-09-13T14:55:59.089 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.090 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.090 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,3,6", 2024-09-13T14:55:59.090 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.274846+0000", 2024-09-13T14:55:59.090 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00012598399999999999 2024-09-13T14:55:59.090 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.090 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.090 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:59.090 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.281413+0000", 2024-09-13T14:55:59.090 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0065674490000000004 2024-09-13T14:55:59.090 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.090 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.090 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:55:59.090 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.281586+0000", 2024-09-13T14:55:59.091 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00017314899999999999 2024-09-13T14:55:59.091 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.091 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.091 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:59.091 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.289879+0000", 2024-09-13T14:55:59.091 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0082928470000000008 2024-09-13T14:55:59.091 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.091 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.091 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:59.091 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.290308+0000", 2024-09-13T14:55:59.091 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00042940799999999999 2024-09-13T14:55:59.091 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.091 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.091 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:55:59.092 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.290452+0000", 2024-09-13T14:55:59.092 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00014385299999999999 2024-09-13T14:55:59.092 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.092 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.092 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:55:59.092 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.290590+0000", 2024-09-13T14:55:59.092 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000137553 2024-09-13T14:55:59.092 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:59.092 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:55:59.092 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:59.092 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.092 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.092 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:133975 8.7 8:e7ad9da5:::benchmark_data_smithi102_313275_object3388:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e337)", 2024-09-13T14:55:59.093 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:47:47.859806+0000", 2024-09-13T14:55:59.093 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 491.20144371499998, 2024-09-13T14:55:59.093 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.433715885, 2024-09-13T14:55:59.093 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:55:59.093 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:55:59.093 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:59.093 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:55:59.093 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:55:59.093 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:55:59.093 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 133975 2024-09-13T14:55:59.093 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.093 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:55:59.093 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.093 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:55:59.094 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859806+0000", 2024-09-13T14:55:59.094 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:59.094 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.094 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.094 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:55:59.094 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859806+0000", 2024-09-13T14:55:59.094 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:59.094 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.094 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.094 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:55:59.094 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859806+0000", 2024-09-13T14:55:59.094 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.2800000000000002e-07 2024-09-13T14:55:59.094 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.094 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.095 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:55:59.095 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859808+0000", 2024-09-13T14:55:59.095 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.0719999999999998e-06 2024-09-13T14:55:59.095 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.095 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.095 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:55:59.095 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859815+0000", 2024-09-13T14:55:59.095 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.0330000000000003e-06 2024-09-13T14:55:59.095 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.095 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.095 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:55:59.095 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859823+0000", 2024-09-13T14:55:59.095 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.126e-06 2024-09-13T14:55:59.096 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.096 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.096 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:59.096 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859898+0000", 2024-09-13T14:55:59.096 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.5240999999999993e-05 2024-09-13T14:55:59.096 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.096 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.096 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting_for_map not empty", 2024-09-13T14:55:59.096 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859903+0000", 2024-09-13T14:55:59.096 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.1549999999999998e-06 2024-09-13T14:55:59.096 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.096 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.096 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:59.097 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.863680+0000", 2024-09-13T14:55:59.097 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0037772449999999998 2024-09-13T14:55:59.097 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.097 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.097 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:55:59.097 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.863687+0000", 2024-09-13T14:55:59.097 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.8000000000000001e-06 2024-09-13T14:55:59.097 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.097 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.097 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:59.097 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.278229+0000", 2024-09-13T14:55:59.097 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4145422000000001 2024-09-13T14:55:59.097 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.097 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.098 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:55:59.098 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.278360+0000", 2024-09-13T14:55:59.098 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000131254 2024-09-13T14:55:59.098 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.098 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.098 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,3,6", 2024-09-13T14:55:59.098 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.278491+0000", 2024-09-13T14:55:59.098 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00013044200000000001 2024-09-13T14:55:59.098 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.098 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.098 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:55:59.098 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.281818+0000", 2024-09-13T14:55:59.098 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0033266110000000002 2024-09-13T14:55:59.098 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.099 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.099 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:59.099 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.283640+0000", 2024-09-13T14:55:59.099 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.001822958 2024-09-13T14:55:59.099 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.099 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.099 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:59.099 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.293331+0000", 2024-09-13T14:55:59.099 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0096903600000000003 2024-09-13T14:55:59.099 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.099 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.099 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:59.099 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.293366+0000", 2024-09-13T14:55:59.100 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.5534000000000002e-05 2024-09-13T14:55:59.100 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.100 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.100 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:55:59.100 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.293418+0000", 2024-09-13T14:55:59.100 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.1622000000000001e-05 2024-09-13T14:55:59.100 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.100 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.100 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:55:59.100 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.293522+0000", 2024-09-13T14:55:59.100 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000103804 2024-09-13T14:55:59.100 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:59.100 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:55:59.101 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:59.101 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.101 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.101 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.46157.0:226347 8.4 e354/306 8:2531c967:::benchmark_data_smithi102_313275_object95760:head v 354'14056, mlcod=354'14054)", 2024-09-13T14:55:59.101 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:49:42.876357+0000", 2024-09-13T14:55:59.101 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 376.18489296600001, 2024-09-13T14:55:59.101 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.44227170199999999, 2024-09-13T14:55:59.101 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:55:59.101 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:55:59.101 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:59.101 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:55:59.101 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.101 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:55:59.101 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.876357+0000", 2024-09-13T14:55:59.102 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:59.102 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.102 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.102 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:55:59.102 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.876357+0000", 2024-09-13T14:55:59.102 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:59.102 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.102 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.102 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:55:59.102 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.876357+0000", 2024-09-13T14:55:59.102 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.0100000000000004e-07 2024-09-13T14:55:59.102 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.102 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.103 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:55:59.103 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.876366+0000", 2024-09-13T14:55:59.103 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.0909999999999993e-06 2024-09-13T14:55:59.103 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.103 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.103 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:55:59.103 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966171+0000", 2024-09-13T14:55:59.103 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.089805323000000006 2024-09-13T14:55:59.103 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.103 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.103 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:55:59.103 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966184+0000", 2024-09-13T14:55:59.103 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2666999999999999e-05 2024-09-13T14:55:59.103 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.104 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.104 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:59.104 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966365+0000", 2024-09-13T14:55:59.104 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000181962 2024-09-13T14:55:59.104 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.104 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.104 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:55:59.104 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966383+0000", 2024-09-13T14:55:59.104 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.7958e-05 2024-09-13T14:55:59.104 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.104 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.104 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:55:59.104 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.318583+0000", 2024-09-13T14:55:59.105 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.35219920900000001 2024-09-13T14:55:59.105 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.105 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.105 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:55:59.105 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.318628+0000", 2024-09-13T14:55:59.105 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.5691e-05 2024-09-13T14:55:59.105 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:59.105 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:55:59.105 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:59.105 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.105 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.105 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.46157.0:226343 8.b e354/337 8:d626b72e:::benchmark_data_smithi102_313275_object95756:head v 354'14208, mlcod=354'14206)", 2024-09-13T14:55:59.105 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:49:42.878012+0000", 2024-09-13T14:55:59.105 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 376.18323719699998, 2024-09-13T14:55:59.106 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.43247816100000003, 2024-09-13T14:55:59.106 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:55:59.106 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:55:59.106 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:59.106 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:55:59.106 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.106 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:55:59.106 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.878012+0000", 2024-09-13T14:55:59.106 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:59.106 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.106 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.106 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:55:59.106 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.878012+0000", 2024-09-13T14:55:59.106 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:59.107 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.107 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.107 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:55:59.107 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.878013+0000", 2024-09-13T14:55:59.107 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.051e-06 2024-09-13T14:55:59.107 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.107 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.107 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:55:59.107 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.878020+0000", 2024-09-13T14:55:59.107 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.9299999999999997e-06 2024-09-13T14:55:59.107 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.107 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.107 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:55:59.108 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966393+0000", 2024-09-13T14:55:59.108 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.088372493999999996 2024-09-13T14:55:59.108 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.108 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.108 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:55:59.108 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966403+0000", 2024-09-13T14:55:59.108 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.8190000000000003e-06 2024-09-13T14:55:59.108 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.108 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.108 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:59.108 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966450+0000", 2024-09-13T14:55:59.108 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.7005000000000001e-05 2024-09-13T14:55:59.109 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.109 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.109 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:55:59.109 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966471+0000", 2024-09-13T14:55:59.109 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.1560999999999999e-05 2024-09-13T14:55:59.109 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.109 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.109 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:55:59.109 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.310310+0000", 2024-09-13T14:55:59.109 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.34383856499999998 2024-09-13T14:55:59.109 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.109 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.109 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:55:59.109 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.310491+0000", 2024-09-13T14:55:59.110 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:59.111 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000180736 2024-09-13T14:55:59.111 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:59.111 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:55:59.111 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:59.111 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.111 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.111 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.46157.0:226348 8.e e354/306 8:75119f7b:::benchmark_data_smithi102_313275_object95761:head v 354'14072, mlcod=354'14070)", 2024-09-13T14:55:59.111 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:49:42.966225+0000", 2024-09-13T14:55:59.111 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 376.09502463400003, 2024-09-13T14:55:59.111 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.33910425399999999, 2024-09-13T14:55:59.112 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:55:59.112 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:55:59.112 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:59.112 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:55:59.112 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.112 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:55:59.112 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966225+0000", 2024-09-13T14:55:59.112 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:59.112 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.112 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.112 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:55:59.112 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966225+0000", 2024-09-13T14:55:59.112 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:59.113 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.113 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.113 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:55:59.113 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966226+0000", 2024-09-13T14:55:59.113 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.4499999999999997e-07 2024-09-13T14:55:59.113 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.113 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.113 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:55:59.113 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966232+0000", 2024-09-13T14:55:59.113 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.5490000000000003e-06 2024-09-13T14:55:59.113 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.113 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.114 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:55:59.114 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966239+0000", 2024-09-13T14:55:59.114 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.2309999999999999e-06 2024-09-13T14:55:59.114 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.114 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.114 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:55:59.114 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966244+0000", 2024-09-13T14:55:59.114 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.1960000000000001e-06 2024-09-13T14:55:59.114 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.114 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.114 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:59.114 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966776+0000", 2024-09-13T14:55:59.114 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00053200199999999995 2024-09-13T14:55:59.115 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.115 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.115 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:55:59.115 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966793+0000", 2024-09-13T14:55:59.115 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.749e-05 2024-09-13T14:55:59.115 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.115 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.115 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:55:59.115 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.305157+0000", 2024-09-13T14:55:59.115 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.33836434900000001 2024-09-13T14:55:59.115 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.115 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.115 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:55:59.116 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.305329+0000", 2024-09-13T14:55:59.116 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00017179200000000001 2024-09-13T14:55:59.116 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:59.116 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:55:59.116 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:59.116 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.116 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.116 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.46157.0:226333 8.1 e354/337 8:8840a5fd:::benchmark_data_smithi102_313275_object95746:head v 354'14039, mlcod=354'14033)", 2024-09-13T14:55:59.116 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:49:42.966259+0000", 2024-09-13T14:55:59.116 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 376.09499109299998, 2024-09-13T14:55:59.116 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.339067756, 2024-09-13T14:55:59.116 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:55:59.117 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:55:59.117 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:59.117 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:55:59.117 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.117 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:55:59.117 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966259+0000", 2024-09-13T14:55:59.117 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:59.117 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.117 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.117 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:55:59.117 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966259+0000", 2024-09-13T14:55:59.117 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:59.117 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.118 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.118 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:55:59.118 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966259+0000", 2024-09-13T14:55:59.118 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.6800000000000001e-07 2024-09-13T14:55:59.118 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.118 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.118 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:55:59.118 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966262+0000", 2024-09-13T14:55:59.118 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.129e-06 2024-09-13T14:55:59.118 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.118 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.118 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:55:59.119 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966267+0000", 2024-09-13T14:55:59.119 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.843e-06 2024-09-13T14:55:59.119 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.119 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.119 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:55:59.119 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966272+0000", 2024-09-13T14:55:59.119 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.7509999999999999e-06 2024-09-13T14:55:59.119 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.119 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.119 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:59.119 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.972016+0000", 2024-09-13T14:55:59.119 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0057441410000000004 2024-09-13T14:55:59.119 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.119 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.120 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:55:59.120 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.972038+0000", 2024-09-13T14:55:59.120 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.1860000000000001e-05 2024-09-13T14:55:59.120 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.120 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.120 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:55:59.120 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.305223+0000", 2024-09-13T14:55:59.120 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.33318497499999999 2024-09-13T14:55:59.120 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.120 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.120 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:55:59.120 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.305326+0000", 2024-09-13T14:55:59.120 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000103689 2024-09-13T14:55:59.120 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:59.121 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:55:59.121 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:59.121 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.121 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.121 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.46157.0:226351 8.4 e354/306 8:23f709fc:::benchmark_data_smithi102_313275_object95764:head v 354'14057, mlcod=354'14054)", 2024-09-13T14:55:59.121 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:49:42.966260+0000", 2024-09-13T14:55:59.121 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 376.094989279, 2024-09-13T14:55:59.121 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.352393028, 2024-09-13T14:55:59.121 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:55:59.121 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:55:59.121 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:59.121 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:55:59.121 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.121 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:55:59.122 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966260+0000", 2024-09-13T14:55:59.122 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:59.122 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.122 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.122 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:55:59.122 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966260+0000", 2024-09-13T14:55:59.122 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:59.122 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.122 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.122 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:55:59.122 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966261+0000", 2024-09-13T14:55:59.122 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.4400000000000001e-07 2024-09-13T14:55:59.122 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.122 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.123 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:55:59.123 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966265+0000", 2024-09-13T14:55:59.123 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.0239999999999999e-06 2024-09-13T14:55:59.123 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.123 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.123 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:55:59.123 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966271+0000", 2024-09-13T14:55:59.123 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.9370000000000001e-06 2024-09-13T14:55:59.123 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.123 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.123 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:55:59.123 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966274+0000", 2024-09-13T14:55:59.123 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.7639999999999999e-06 2024-09-13T14:55:59.124 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.124 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.124 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:59.124 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.970109+0000", 2024-09-13T14:55:59.124 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0038344640000000001 2024-09-13T14:55:59.124 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.124 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.124 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:55:59.124 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.970127+0000", 2024-09-13T14:55:59.124 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.8082000000000001e-05 2024-09-13T14:55:59.124 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.124 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.124 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:55:59.124 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.318632+0000", 2024-09-13T14:55:59.125 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.348505116 2024-09-13T14:55:59.125 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.125 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.125 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:55:59.125 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.318653+0000", 2024-09-13T14:55:59.125 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.1297e-05 2024-09-13T14:55:59.125 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:59.125 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:55:59.125 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:59.125 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.125 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.125 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.46157.0:226357 8.e e354/306 8:7912c2d7:::benchmark_data_smithi102_313275_object95770:head v 354'14073, mlcod=354'14070)", 2024-09-13T14:55:59.126 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:49:42.966277+0000", 2024-09-13T14:55:59.126 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 376.09497266400001, 2024-09-13T14:55:59.126 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.33908791799999999, 2024-09-13T14:55:59.126 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:55:59.126 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:55:59.126 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:59.126 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:55:59.126 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.126 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:55:59.126 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966277+0000", 2024-09-13T14:55:59.126 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:59.126 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.127 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.127 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:55:59.127 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966277+0000", 2024-09-13T14:55:59.127 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:59.127 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.127 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.127 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:55:59.127 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966277+0000", 2024-09-13T14:55:59.127 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.35e-07 2024-09-13T14:55:59.127 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.127 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.127 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:55:59.127 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966284+0000", 2024-09-13T14:55:59.127 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.5640000000000002e-06 2024-09-13T14:55:59.128 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.128 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.128 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:55:59.128 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966288+0000", 2024-09-13T14:55:59.128 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.7029999999999997e-06 2024-09-13T14:55:59.128 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.128 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.128 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:55:59.128 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966292+0000", 2024-09-13T14:55:59.128 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.456e-06 2024-09-13T14:55:59.128 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.128 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.129 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:59.129 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.969737+0000", 2024-09-13T14:55:59.129 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.003445591 2024-09-13T14:55:59.129 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.129 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.129 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:55:59.129 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.969751+0000", 2024-09-13T14:55:59.129 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3529999999999999e-05 2024-09-13T14:55:59.129 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.129 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.129 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:55:59.129 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.305333+0000", 2024-09-13T14:55:59.130 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.33558170999999998 2024-09-13T14:55:59.130 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.130 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.130 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:55:59.130 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.305365+0000", 2024-09-13T14:55:59.130 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.2128999999999997e-05 2024-09-13T14:55:59.130 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:59.130 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:55:59.130 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:59.130 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.130 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.130 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.46157.0:226363 8.f e354/337 8:f9e9d9b3:::benchmark_data_smithi102_313275_object95776:head v 354'14122, mlcod=354'14120)", 2024-09-13T14:55:59.130 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:49:42.966470+0000", 2024-09-13T14:55:59.131 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 376.09477984500001, 2024-09-13T14:55:59.131 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.33875966800000001, 2024-09-13T14:55:59.131 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:55:59.131 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:55:59.131 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:59.131 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:55:59.131 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.131 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:55:59.131 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966470+0000", 2024-09-13T14:55:59.131 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:59.131 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.131 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.131 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:55:59.132 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966470+0000", 2024-09-13T14:55:59.132 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:59.132 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.132 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.132 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:55:59.132 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966470+0000", 2024-09-13T14:55:59.132 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.9499999999999998e-07 2024-09-13T14:55:59.132 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.132 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.132 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:55:59.132 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966475+0000", 2024-09-13T14:55:59.132 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.4009999999999999e-06 2024-09-13T14:55:59.133 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.133 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.133 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:55:59.133 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966481+0000", 2024-09-13T14:55:59.133 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.5729999999999996e-06 2024-09-13T14:55:59.133 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.133 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.133 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:55:59.133 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966486+0000", 2024-09-13T14:55:59.133 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.498e-06 2024-09-13T14:55:59.133 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.133 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.133 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:59.134 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966552+0000", 2024-09-13T14:55:59.134 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.6180999999999995e-05 2024-09-13T14:55:59.134 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.134 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.134 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:55:59.134 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966572+0000", 2024-09-13T14:55:59.134 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.0339999999999998e-05 2024-09-13T14:55:59.134 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.134 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.134 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:55:59.134 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.304984+0000", 2024-09-13T14:55:59.134 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.33841160399999998 2024-09-13T14:55:59.135 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.135 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.135 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:55:59.135 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.305229+0000", 2024-09-13T14:55:59.135 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00024567600000000001 2024-09-13T14:55:59.135 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:59.135 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:55:59.135 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:59.135 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.135 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.135 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.46157.0:226364 8.f e354/337 8:f43d0b70:::benchmark_data_smithi102_313275_object95777:head v 354'14123, mlcod=354'14120)", 2024-09-13T14:55:59.135 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:49:42.966511+0000", 2024-09-13T14:55:59.135 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 376.09473830100001, 2024-09-13T14:55:59.136 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.33875917500000002, 2024-09-13T14:55:59.136 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:55:59.136 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:55:59.136 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:59.136 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:55:59.136 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.136 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:55:59.136 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966511+0000", 2024-09-13T14:55:59.136 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:59.136 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.136 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.136 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:55:59.137 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966511+0000", 2024-09-13T14:55:59.137 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:59.137 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.137 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.137 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:55:59.137 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966512+0000", 2024-09-13T14:55:59.137 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.2600000000000001e-07 2024-09-13T14:55:59.137 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.137 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.137 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:55:59.138 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966515+0000", 2024-09-13T14:55:59.138 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.2059999999999999e-06 2024-09-13T14:55:59.139 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.139 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.139 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:55:59.139 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966520+0000", 2024-09-13T14:55:59.139 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.2209999999999996e-06 2024-09-13T14:55:59.139 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.139 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.139 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:55:59.139 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966524+0000", 2024-09-13T14:55:59.139 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.1409999999999998e-06 2024-09-13T14:55:59.139 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.140 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.140 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:59.140 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.969884+0000", 2024-09-13T14:55:59.140 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0033597480000000001 2024-09-13T14:55:59.140 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.140 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.140 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:55:59.140 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.969904+0000", 2024-09-13T14:55:59.140 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.0265e-05 2024-09-13T14:55:59.140 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.140 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.140 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:55:59.141 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.305234+0000", 2024-09-13T14:55:59.141 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.33532999800000002 2024-09-13T14:55:59.141 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.141 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.141 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:55:59.141 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.305270+0000", 2024-09-13T14:55:59.141 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.6369999999999999e-05 2024-09-13T14:55:59.141 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:59.141 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:55:59.141 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:59.141 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.141 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.141 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:226353 8.a 8:56f6bbd7:::benchmark_data_smithi102_313275_object95766:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-09-13T14:55:59.141 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:49:42.966566+0000", 2024-09-13T14:55:59.142 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 376.09468336800001, 2024-09-13T14:55:59.142 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.33901846299999999, 2024-09-13T14:55:59.142 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:55:59.142 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:55:59.142 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:59.142 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:55:59.142 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:55:59.142 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:55:59.142 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 226353 2024-09-13T14:55:59.142 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.142 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:55:59.142 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.142 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:55:59.143 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966566+0000", 2024-09-13T14:55:59.143 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:59.143 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.143 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.143 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:55:59.143 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966566+0000", 2024-09-13T14:55:59.143 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:59.143 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.143 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.143 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:55:59.143 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966567+0000", 2024-09-13T14:55:59.143 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2020000000000001e-06 2024-09-13T14:55:59.143 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.143 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.144 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:55:59.144 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966571+0000", 2024-09-13T14:55:59.144 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.7060000000000002e-06 2024-09-13T14:55:59.144 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.144 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.144 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:55:59.144 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966581+0000", 2024-09-13T14:55:59.144 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.5899999999999997e-06 2024-09-13T14:55:59.144 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.144 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.145 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:55:59.145 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966587+0000", 2024-09-13T14:55:59.145 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.9100000000000002e-06 2024-09-13T14:55:59.145 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.145 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.145 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:59.145 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.967254+0000", 2024-09-13T14:55:59.145 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00066735600000000003 2024-09-13T14:55:59.145 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.145 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.145 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:55:59.145 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.967370+0000", 2024-09-13T14:55:59.145 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000116449 2024-09-13T14:55:59.145 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.146 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.146 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,3,5", 2024-09-13T14:55:59.146 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.967496+0000", 2024-09-13T14:55:59.146 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000125597 2024-09-13T14:55:59.146 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.146 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.146 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:55:59.146 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.972217+0000", 2024-09-13T14:55:59.146 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0047206050000000001 2024-09-13T14:55:59.146 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.146 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.146 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:59.146 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.304933+0000", 2024-09-13T14:55:59.147 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.33271657399999999 2024-09-13T14:55:59.147 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.147 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.147 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:59.147 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.305240+0000", 2024-09-13T14:55:59.147 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000307084 2024-09-13T14:55:59.147 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.147 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.147 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:59.147 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.305417+0000", 2024-09-13T14:55:59.147 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00017644700000000001 2024-09-13T14:55:59.147 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.147 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.147 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:55:59.148 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.305474+0000", 2024-09-13T14:55:59.148 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.6988999999999999e-05 2024-09-13T14:55:59.148 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.148 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.148 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:55:59.148 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.305585+0000", 2024-09-13T14:55:59.148 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000110954 2024-09-13T14:55:59.148 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:59.148 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:55:59.148 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:59.148 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.148 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.148 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:226356 8.3 8:cdb7d1cd:::benchmark_data_smithi102_313275_object95769:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-09-13T14:55:59.148 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:49:42.966599+0000", 2024-09-13T14:55:59.149 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 376.09465017000002, 2024-09-13T14:55:59.149 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.347066559, 2024-09-13T14:55:59.149 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:55:59.149 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:55:59.149 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:59.149 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:55:59.149 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:55:59.149 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:55:59.149 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 226356 2024-09-13T14:55:59.149 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.149 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:55:59.149 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.149 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:55:59.149 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966599+0000", 2024-09-13T14:55:59.150 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:59.150 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.150 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.150 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:55:59.150 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966599+0000", 2024-09-13T14:55:59.150 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:59.150 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.150 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.150 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:55:59.150 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966600+0000", 2024-09-13T14:55:59.150 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.2899999999999999e-07 2024-09-13T14:55:59.150 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.150 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.151 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:55:59.151 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966602+0000", 2024-09-13T14:55:59.151 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.3599999999999999e-06 2024-09-13T14:55:59.151 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.151 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.151 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:55:59.151 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966609+0000", 2024-09-13T14:55:59.151 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.3720000000000001e-06 2024-09-13T14:55:59.151 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.151 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.151 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:55:59.151 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966614+0000", 2024-09-13T14:55:59.151 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.4589999999999997e-06 2024-09-13T14:55:59.151 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.152 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.152 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:59.152 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.967705+0000", 2024-09-13T14:55:59.152 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0010911919999999999 2024-09-13T14:55:59.152 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.152 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.152 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:55:59.152 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.967826+0000", 2024-09-13T14:55:59.152 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000120355 2024-09-13T14:55:59.152 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.152 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.152 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,5,6", 2024-09-13T14:55:59.153 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.967955+0000", 2024-09-13T14:55:59.153 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00012938199999999999 2024-09-13T14:55:59.153 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.153 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.153 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:59.153 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.311754+0000", 2024-09-13T14:55:59.153 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.34379863500000002 2024-09-13T14:55:59.153 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.153 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.153 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:59.153 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.311792+0000", 2024-09-13T14:55:59.153 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.8247000000000002e-05 2024-09-13T14:55:59.153 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.153 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.154 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:59.154 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.311827+0000", 2024-09-13T14:55:59.154 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.5454000000000003e-05 2024-09-13T14:55:59.154 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.154 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.154 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:55:59.154 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.313159+0000", 2024-09-13T14:55:59.154 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0013320140000000001 2024-09-13T14:55:59.154 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.154 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.155 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:55:59.155 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.313312+0000", 2024-09-13T14:55:59.155 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000152318 2024-09-13T14:55:59.155 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.155 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.155 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:55:59.155 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.313666+0000", 2024-09-13T14:55:59.155 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000354342 2024-09-13T14:55:59.155 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:59.155 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:55:59.155 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:59.155 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.155 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.156 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:226359 8.3 8:c15c2493:::benchmark_data_smithi102_313275_object95772:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-09-13T14:55:59.156 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:49:42.966622+0000", 2024-09-13T14:55:59.156 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 376.09462776200002, 2024-09-13T14:55:59.156 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.35298271799999997, 2024-09-13T14:55:59.156 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:55:59.156 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:55:59.156 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:59.156 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:55:59.156 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:55:59.156 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:55:59.156 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 226359 2024-09-13T14:55:59.156 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.156 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:55:59.157 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.157 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:55:59.157 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966622+0000", 2024-09-13T14:55:59.158 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 45 16 62070 62054 86.1657 80.0625 0.0129235 0.0115999 2024-09-13T14:55:59.158 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:59.158 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.158 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.158 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:55:59.159 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966622+0000", 2024-09-13T14:55:59.159 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:59.159 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.159 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.159 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:55:59.159 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966622+0000", 2024-09-13T14:55:59.159 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.9900000000000001e-07 2024-09-13T14:55:59.159 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.159 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.159 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:55:59.159 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966624+0000", 2024-09-13T14:55:59.159 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.2369999999999999e-06 2024-09-13T14:55:59.159 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.160 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.160 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:55:59.160 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966630+0000", 2024-09-13T14:55:59.160 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.3750000000000002e-06 2024-09-13T14:55:59.160 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.160 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.160 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:55:59.160 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.966635+0000", 2024-09-13T14:55:59.160 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.7820000000000001e-06 2024-09-13T14:55:59.160 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.160 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.160 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:59.160 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.304872+0000", 2024-09-13T14:55:59.161 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.338237383 2024-09-13T14:55:59.161 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.161 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.161 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:55:59.161 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.305028+0000", 2024-09-13T14:55:59.161 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00015607300000000001 2024-09-13T14:55:59.161 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.161 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.161 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,5,6", 2024-09-13T14:55:59.161 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.305180+0000", 2024-09-13T14:55:59.161 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000151848 2024-09-13T14:55:59.161 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.161 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.162 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:59.162 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.313810+0000", 2024-09-13T14:55:59.162 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0086300419999999992 2024-09-13T14:55:59.162 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.162 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.162 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:55:59.162 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.313874+0000", 2024-09-13T14:55:59.162 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.4339999999999997e-05 2024-09-13T14:55:59.162 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.162 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.162 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:59.162 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.314061+0000", 2024-09-13T14:55:59.162 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000186653 2024-09-13T14:55:59.163 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.163 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.163 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:59.163 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.314236+0000", 2024-09-13T14:55:59.163 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00017469999999999999 2024-09-13T14:55:59.163 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.163 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.163 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:55:59.163 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.314451+0000", 2024-09-13T14:55:59.163 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00021489 2024-09-13T14:55:59.163 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.163 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.163 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:55:59.164 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.319605+0000", 2024-09-13T14:55:59.164 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0051539960000000001 2024-09-13T14:55:59.164 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:59.164 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:55:59.164 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:59.164 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.164 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.164 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.46157.0:226367 8.4 e354/306 8:2828d25f:::benchmark_data_smithi102_313275_object95780:head v 354'14058, mlcod=354'14054)", 2024-09-13T14:55:59.164 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:49:42.967323+0000", 2024-09-13T14:55:59.164 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 376.09392645100002, 2024-09-13T14:55:59.165 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.35143049599999998, 2024-09-13T14:55:59.165 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:55:59.165 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:55:59.165 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:59.165 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:55:59.165 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.165 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:55:59.165 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.967323+0000", 2024-09-13T14:55:59.165 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:59.165 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.165 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.165 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:55:59.165 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.967323+0000", 2024-09-13T14:55:59.166 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:59.166 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.166 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.166 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:55:59.166 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.967324+0000", 2024-09-13T14:55:59.166 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.1099999999999995e-07 2024-09-13T14:55:59.166 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.166 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.166 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:55:59.166 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.967329+0000", 2024-09-13T14:55:59.166 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.0390000000000004e-06 2024-09-13T14:55:59.166 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.166 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.167 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:55:59.167 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.967339+0000", 2024-09-13T14:55:59.167 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.8779999999999993e-06 2024-09-13T14:55:59.167 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.167 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.167 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:55:59.167 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.967344+0000", 2024-09-13T14:55:59.167 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.7219999999999996e-06 2024-09-13T14:55:59.167 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.167 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.167 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:59.167 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.304997+0000", 2024-09-13T14:55:59.167 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.33765234100000002 2024-09-13T14:55:59.167 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.168 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.168 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:55:59.168 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.305019+0000", 2024-09-13T14:55:59.168 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.1977e-05 2024-09-13T14:55:59.168 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.168 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.168 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:55:59.168 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.318658+0000", 2024-09-13T14:55:59.168 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.013639709 2024-09-13T14:55:59.168 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.168 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.168 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:55:59.168 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.318754+0000", 2024-09-13T14:55:59.169 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.5218999999999998e-05 2024-09-13T14:55:59.169 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:59.169 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:55:59.169 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:59.169 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.169 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.169 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop_reply(client.46157.0:226356 8.3 e355/337 ondisk, result = 0)", 2024-09-13T14:55:59.169 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:49:42.973291+0000", 2024-09-13T14:55:59.169 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 376.08795906099999, 2024-09-13T14:55:59.169 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.33847084700000002, 2024-09-13T14:55:59.169 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:55:59.169 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:55:59.169 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "started", 2024-09-13T14:55:59.170 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:55:59.170 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.170 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:55:59.170 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.973291+0000", 2024-09-13T14:55:59.170 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:59.170 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.170 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.170 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:55:59.170 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.973291+0000", 2024-09-13T14:55:59.170 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:59.170 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.170 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.170 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:55:59.170 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.973292+0000", 2024-09-13T14:55:59.170 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.8109999999999999e-06 2024-09-13T14:55:59.171 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.171 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.171 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:55:59.171 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.973298+0000", 2024-09-13T14:55:59.171 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.49e-06 2024-09-13T14:55:59.171 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.171 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.171 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:55:59.171 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.304829+0000", 2024-09-13T14:55:59.171 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.33153068099999999 2024-09-13T14:55:59.171 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.171 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.171 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:55:59.171 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.304844+0000", 2024-09-13T14:55:59.171 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.5858999999999998e-05 2024-09-13T14:55:59.172 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.172 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.172 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:59.172 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.311739+0000", 2024-09-13T14:55:59.172 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.006894664 2024-09-13T14:55:59.172 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.172 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.172 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:55:59.172 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.311747+0000", 2024-09-13T14:55:59.172 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.9910000000000002e-06 2024-09-13T14:55:59.172 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.172 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.172 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:55:59.172 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.311761+0000", 2024-09-13T14:55:59.173 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4351e-05 2024-09-13T14:55:59.173 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:59.173 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:55:59.173 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:59.173 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.173 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.173 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop_reply(client.46157.0:226353 8.a e355/337 ondisk, result = 0)", 2024-09-13T14:55:59.173 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:49:42.974862+0000", 2024-09-13T14:55:59.173 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 376.08638791499999, 2024-09-13T14:55:59.173 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.33008076200000003, 2024-09-13T14:55:59.173 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:55:59.173 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:55:59.173 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "started", 2024-09-13T14:55:59.174 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:55:59.174 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.174 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:55:59.174 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.974862+0000", 2024-09-13T14:55:59.174 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:59.174 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.174 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.174 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:55:59.174 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.974862+0000", 2024-09-13T14:55:59.174 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:59.174 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.174 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.174 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:55:59.175 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.974863+0000", 2024-09-13T14:55:59.175 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.4600000000000003e-07 2024-09-13T14:55:59.175 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.175 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.175 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:55:59.175 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:42.974868+0000", 2024-09-13T14:55:59.175 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.3430000000000003e-06 2024-09-13T14:55:59.175 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.175 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.175 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:55:59.175 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.304836+0000", 2024-09-13T14:55:59.175 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.32996769799999998 2024-09-13T14:55:59.175 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.175 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.176 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:55:59.176 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.304846+0000", 2024-09-13T14:55:59.176 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0651e-05 2024-09-13T14:55:59.176 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.176 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.176 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:59.176 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.304916+0000", 2024-09-13T14:55:59.176 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.9331000000000004e-05 2024-09-13T14:55:59.176 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.176 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.176 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:55:59.176 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.304926+0000", 2024-09-13T14:55:59.176 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0317e-05 2024-09-13T14:55:59.177 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.177 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.177 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:55:59.177 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:49:43.304942+0000", 2024-09-13T14:55:59.177 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.6475999999999999e-05 2024-09-13T14:55:59.177 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:59.177 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:55:59.177 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:59.177 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.177 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.177 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.36528.0:113806 9.3 9.95e516a3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e403)", 2024-09-13T14:55:59.177 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:52:27.688526+0000", 2024-09-13T14:55:59.177 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 211.37272329199999, 2024-09-13T14:55:59.177 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.64616753500000002, 2024-09-13T14:55:59.178 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:55:59.178 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:55:59.178 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "reached pg", 2024-09-13T14:55:59.178 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:55:59.178 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.36528", 2024-09-13T14:55:59.178 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:55:59.178 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 113806 2024-09-13T14:55:59.178 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.178 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:55:59.178 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.178 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:55:59.178 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:52:27.688526+0000", 2024-09-13T14:55:59.178 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:59.179 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.179 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.179 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:55:59.179 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:52:27.688526+0000", 2024-09-13T14:55:59.179 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:59.179 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.179 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.179 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:55:59.179 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:52:27.688528+0000", 2024-09-13T14:55:59.179 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.9360000000000002e-06 2024-09-13T14:55:59.179 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.179 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.179 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:55:59.180 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:52:27.688562+0000", 2024-09-13T14:55:59.180 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.4106000000000002e-05 2024-09-13T14:55:59.180 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.180 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.180 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:55:59.180 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:52:27.688581+0000", 2024-09-13T14:55:59.180 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.8448999999999999e-05 2024-09-13T14:55:59.180 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.180 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.180 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:55:59.180 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:52:27.688591+0000", 2024-09-13T14:55:59.180 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0638999999999999e-05 2024-09-13T14:55:59.180 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.180 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.181 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:59.181 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:52:27.688701+0000", 2024-09-13T14:55:59.181 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000109889 2024-09-13T14:55:59.181 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.181 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.181 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:55:59.181 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:52:27.688719+0000", 2024-09-13T14:55:59.181 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.7706999999999998e-05 2024-09-13T14:55:59.181 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.181 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.181 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:59.181 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:52:28.334684+0000", 2024-09-13T14:55:59.182 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.64596481900000002 2024-09-13T14:55:59.182 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.182 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.182 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:55:59.182 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:52:28.334694+0000", 2024-09-13T14:55:59.182 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.9899999999999992e-06 2024-09-13T14:55:59.182 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:59.182 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:55:59.182 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:59.182 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.182 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.182 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.36738.0:55144 10.5 10:ae712df7:::benchmark_data_smithi102_377304_object55143:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e460)", 2024-09-13T14:55:59.183 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:55:53.281276+0000", 2024-09-13T14:55:59.183 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 5.7799735800000001, 2024-09-13T14:55:59.183 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.29904046000000001, 2024-09-13T14:55:59.183 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:55:59.183 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:55:59.183 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:59.183 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:55:59.183 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.36738", 2024-09-13T14:55:59.183 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/2915890700", 2024-09-13T14:55:59.183 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 55144 2024-09-13T14:55:59.183 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.183 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:55:59.183 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.184 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:55:59.184 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:55:53.281276+0000", 2024-09-13T14:55:59.184 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:59.184 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.184 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.184 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:55:59.184 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:55:53.281276+0000", 2024-09-13T14:55:59.184 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:55:59.184 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.184 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.184 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:55:59.184 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:55:53.281277+0000", 2024-09-13T14:55:59.184 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.4000000000000001e-07 2024-09-13T14:55:59.184 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.185 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.185 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:55:59.185 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:55:53.281298+0000", 2024-09-13T14:55:59.185 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.1444e-05 2024-09-13T14:55:59.185 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.185 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.185 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:55:59.185 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:55:53.281306+0000", 2024-09-13T14:55:59.185 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.4619999999999999e-06 2024-09-13T14:55:59.185 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.185 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.185 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:55:59.186 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:55:53.281311+0000", 2024-09-13T14:55:59.186 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.9919999999999998e-06 2024-09-13T14:55:59.186 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.186 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.186 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:55:59.186 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:55:53.283564+0000", 2024-09-13T14:55:59.186 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.002253255 2024-09-13T14:55:59.186 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.186 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.186 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:55:59.186 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:55:53.283680+0000", 2024-09-13T14:55:59.186 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000116517 2024-09-13T14:55:59.186 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.186 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.187 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 0,3", 2024-09-13T14:55:59.187 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:55:53.283800+0000", 2024-09-13T14:55:59.187 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000119512 2024-09-13T14:55:59.187 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.187 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.187 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:55:59.187 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:55:53.286342+0000", 2024-09-13T14:55:59.187 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0025419840000000002 2024-09-13T14:55:59.187 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.187 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.187 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:59.187 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:55:53.295204+0000", 2024-09-13T14:55:59.187 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0088626 2024-09-13T14:55:59.187 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.188 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.188 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:59.188 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:55:53.302749+0000", 2024-09-13T14:55:59.188 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0075447939999999996 2024-09-13T14:55:59.188 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.188 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.188 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:55:59.188 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:55:53.580267+0000", 2024-09-13T14:55:59.188 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.27751806200000001 2024-09-13T14:55:59.188 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:55:59.188 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:55:59.188 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:55:59.189 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:55:53.580316+0000", 2024-09-13T14:55:59.189 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.9197999999999999e-05 2024-09-13T14:55:59.189 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:59.189 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:55:59.189 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:59.189 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:55:59.189 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:55:59.189 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:55:59.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:59.191 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:59.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:59.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:55:58 smithi195 ceph-mon[104262]: pgmap v1529: 109 pgs: 109 active+clean; 3.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-09-13T14:55:59.292 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:59.318 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 6, 7, 5, 2, 1] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T14:55:59.318 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:55:59.319 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-13T14:55:59.319 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 0.000000 2024-09-13T14:55:59.319 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 0 0 2024-09-13T14:55:59.335 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:59.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:59.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:59.438 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:59.477 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:59.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:59.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:59.579 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:59.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:59.626 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:55:59.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:59.727 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:59.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:59.755 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:59.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:59.856 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:59.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:59.883 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:55:59.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:59.984 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:00.010 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:00.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:00.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:00.112 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:00.113 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 46 16 63349 63333 86.0301 79.9375 0.0271208 0.0116164 2024-09-13T14:56:00.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:00.173 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:00.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:00.274 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:00.317 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:00.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:00.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:00.419 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:00.454 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:00.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:00.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:00.556 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:00.584 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:00.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:00.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:00.686 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:00.713 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:00.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:00.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:00.748 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T14:56:00.815 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:00.872 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:00.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:00.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:00.974 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:01.029 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:01.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:01.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:01.114 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 47 16 64571 64555 85.8242 76.375 0.0176002 0.0116467 2024-09-13T14:56:01.132 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:01.157 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:01.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:01.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:01.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:00 smithi102 ceph-mon[115269]: pgmap v1530: 109 pgs: 109 active+clean; 3.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-09-13T14:56:01.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:00 smithi102 ceph-mon[110183]: pgmap v1530: 109 pgs: 109 active+clean; 3.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-09-13T14:56:01.242 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:00 smithi195 ceph-mon[104262]: pgmap v1530: 109 pgs: 109 active+clean; 3.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-09-13T14:56:01.258 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:01.283 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:01.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:01.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:01.384 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:01.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:01.429 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:01.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:01.530 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:01.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:01.580 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:01.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:01.680 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:01.708 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:01.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:01.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:01.810 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:01.862 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:01.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:01.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:01.883 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is unset 2024-09-13T14:56:01.965 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:01.990 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:01.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:01.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:02.092 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:02.117 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 48 16 65673 65657 85.4653 68.875 0.0140053 0.0116941 2024-09-13T14:56:02.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:02.158 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:02.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:02.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:01 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3852810818' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:56:02.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:01 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:56:02.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:01 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:56:02.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:01 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:56:02.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:01 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3852810818' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:56:02.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:01 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:56:02.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:01 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:56:02.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:01 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:56:02.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:01 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3852810818' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:56:02.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:01 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:56:02.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:01 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:56:02.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:01 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:56:02.259 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:02.284 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:02.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:02.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:02.386 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:02.430 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:02.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:02.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:02.532 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:02.582 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:02.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:02.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:02.683 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:02.707 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:02.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:02.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:02.809 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:02.834 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:02.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:02.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:02.935 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:02.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:02.978 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:02.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:03.079 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:03.117 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 49 16 66785 66769 85.1388 69.5 0.00687068 0.0117403 2024-09-13T14:56:03.124 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:03.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:03.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:03.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:02 smithi102 ceph-mon[115269]: pgmap v1531: 109 pgs: 109 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-09-13T14:56:03.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:02 smithi102 ceph-mon[115269]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:56:03.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:02 smithi102 ceph-mon[115269]: Cluster is now healthy 2024-09-13T14:56:03.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:02 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:56:03.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:02 smithi102 ceph-mon[115269]: osdmap e461: 8 total, 8 up, 7 in 2024-09-13T14:56:03.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:02 smithi102 ceph-mon[110183]: pgmap v1531: 109 pgs: 109 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-09-13T14:56:03.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:02 smithi102 ceph-mon[110183]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:56:03.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:02 smithi102 ceph-mon[110183]: Cluster is now healthy 2024-09-13T14:56:03.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:02 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:56:03.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:02 smithi102 ceph-mon[110183]: osdmap e461: 8 total, 8 up, 7 in 2024-09-13T14:56:03.227 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:03.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:02 smithi195 ceph-mon[104262]: pgmap v1531: 109 pgs: 109 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-09-13T14:56:03.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:02 smithi195 ceph-mon[104262]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:56:03.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:02 smithi195 ceph-mon[104262]: Cluster is now healthy 2024-09-13T14:56:03.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:02 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:56:03.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:02 smithi195 ceph-mon[104262]: osdmap e461: 8 total, 8 up, 7 in 2024-09-13T14:56:03.251 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:03.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:03.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:03.353 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:03.381 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:03.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:03.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:03.483 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:03.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:03.529 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:03.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:03.630 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:03.658 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:03.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:03.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:03.761 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:03.789 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:03.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:03.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:03.891 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:03.912 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:03.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:03.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:04.014 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:04.041 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:04.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:04.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:04.118 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 50 16 67777 67761 84.6759 62 0.0126551 0.0118036 2024-09-13T14:56:04.143 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:04.190 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:04.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:04.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:04.293 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:04.336 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:04.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:04.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:04.438 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:04.484 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:04.485 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:04.523 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:04.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:04.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:04.575 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:56:04.576 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:56:04.612 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:56:04.613 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:56:04.625 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:04.649 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:56:04.649 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:56:04.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:04.669 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:04.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:04.681 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:56:04.682 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:56:04.715 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:56:04.715 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:56:04.750 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:56:04.750 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:56:04.770 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:04.783 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:56:04.784 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:56:04.814 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:04.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:04.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:04.819 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:56:04.819 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:56:04.916 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:04.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:04.960 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:04.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:05.061 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:05.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:05.100 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:05.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:05.118 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 51 16 68819 68803 84.2925 65.125 0.00784742 0.011858 2024-09-13T14:56:05.201 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:05.232 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:05.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:05.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:05.281 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:05 smithi102 ceph-mon[110183]: pgmap v1533: 109 pgs: 109 active+clean; 4.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-13T14:56:05.281 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:05 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:56:05.281 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:05 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:56:05.281 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:05 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/542057231' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:56:05.281 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:05 smithi102 ceph-mon[115269]: pgmap v1533: 109 pgs: 109 active+clean; 4.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-13T14:56:05.281 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:05 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:56:05.281 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:05 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:56:05.282 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:05 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/542057231' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:56:05.334 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:05.376 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:05.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:05.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:05.477 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:05.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:05 smithi195 ceph-mon[104262]: pgmap v1533: 109 pgs: 109 active+clean; 4.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-13T14:56:05.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:05 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:56:05.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:05 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:56:05.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:05 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/542057231' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:56:05.503 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:05.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:05.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:05.605 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:05.632 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:05.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:05.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:05.734 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:05.761 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:05.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:05.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:05.863 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:05.888 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:05.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:05.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:05.990 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:06.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:06.031 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:06.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:06.118 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 52 16 70042 70026 84.1414 76.4375 0.0120435 0.0118799 2024-09-13T14:56:06.132 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:06.173 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:06.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:06.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:06.238 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:56:06.275 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:06.323 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:06.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:06.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:06.425 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:06.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:06.479 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:06.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:06.580 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:06.616 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:06.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:06.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:06.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:56:06 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:56:06] "GET /metrics HTTP/1.1" 200 36564 "" "Prometheus/2.51.0" 2024-09-13T14:56:06.718 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:06.743 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:06.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:06.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:06.845 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:06.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:06.895 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:06.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:06.996 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:07.039 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:07.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:07.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:07.118 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 53 16 71196 71180 83.9146 72.125 0.0105763 0.0119116 2024-09-13T14:56:07.141 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:07.180 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:07.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:07.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:07.282 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:07.323 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:07.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:07.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:07.397 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:07 smithi102 ceph-mon[110183]: pgmap v1534: 109 pgs: 109 active+clean; 4.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-13T14:56:07.397 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:07 smithi102 ceph-mon[115269]: pgmap v1534: 109 pgs: 109 active+clean; 4.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-13T14:56:07.425 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:07.451 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:07.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:07.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:07.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:07 smithi195 ceph-mon[104262]: pgmap v1534: 109 pgs: 109 active+clean; 4.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-13T14:56:07.553 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:07.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:07.591 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:07.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:07.692 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:07.718 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:07.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:07.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:07.820 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:07.847 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:07.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:07.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:07.949 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:07.974 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:07.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:07.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:08.075 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:08.117 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:08.118 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 54 16 72427 72411 83.7851 76.9375 0.0120431 0.0119289 2024-09-13T14:56:08.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:08.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:08.220 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:08.246 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:08.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:08.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:08.348 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:08.397 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:08.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:08.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:08.499 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:08.536 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:08.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:08.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:08.638 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:08.663 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:08.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:08.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:08.765 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:08.791 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:08.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:08.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:08.910 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:08.937 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:08.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:08.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:09.039 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:09.048 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:56:09.049 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:56:09.064 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:09.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:09.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:09.118 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 55 16 73577 73561 83.5685 71.875 0.00996393 0.0119608 2024-09-13T14:56:09.166 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:09.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:09.193 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:09.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:09.294 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:09.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:09.334 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:09.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:09.399 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:56:09.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:09 smithi102 ceph-mon[110183]: pgmap v1535: 109 pgs: 109 active+clean; 4.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-13T14:56:09.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:09 smithi102 ceph-mon[115269]: pgmap v1535: 109 pgs: 109 active+clean; 4.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-13T14:56:09.435 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:09.461 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:09.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:09.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:09.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:09 smithi195 ceph-mon[104262]: pgmap v1535: 109 pgs: 109 active+clean; 4.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-13T14:56:09.563 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:09.587 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:09.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:09.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:09.689 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:09.717 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:09.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:09.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:09.818 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:09.862 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:09.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:09.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:09.964 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:10.009 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:10.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:10.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:10.111 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:10.118 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 56 16 74733 74717 83.3663 72.25 0.0136292 0.01199 2024-09-13T14:56:10.135 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:10.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:10.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:10.188 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T14:56:10.237 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:10.261 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:10.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:10.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:10.363 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:10.402 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:10.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:10.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:10.504 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:10.542 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:10.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:10.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:10.644 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:10.686 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:10.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:10.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:10.788 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:10.829 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:10.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:10.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:10.858 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:10.931 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:10.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:10.958 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:10.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:11.059 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:11.088 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:11.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:11.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:11.119 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 57 16 75766 75750 83.0354 64.5625 0.019416 0.0120385 2024-09-13T14:56:11.190 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:11.216 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:11.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:11.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:11.318 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:11.346 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:11.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:11.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:11.447 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:11.467 INFO:teuthology.orchestra.run.smithi102.stderr:set osd.0 primary-affinity to 0 (802) 2024-09-13T14:56:11.493 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:11.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:11.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:11.595 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:11.617 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:11.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:11.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:11.657 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:11 smithi102 ceph-mon[110183]: pgmap v1536: 109 pgs: 109 active+clean; 4.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 72 MiB/s wr, 1.16k op/s 2024-09-13T14:56:11.657 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:11 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:56:11.657 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:11 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:56:11.657 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:11 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1284359688' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 0.0} : dispatch 2024-09-13T14:56:11.658 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:11 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 0.0} : dispatch 2024-09-13T14:56:11.658 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:11 smithi102 ceph-mon[115269]: pgmap v1536: 109 pgs: 109 active+clean; 4.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 72 MiB/s wr, 1.16k op/s 2024-09-13T14:56:11.658 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:11 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:56:11.658 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:11 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:56:11.658 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:11 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1284359688' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 0.0} : dispatch 2024-09-13T14:56:11.658 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:11 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 0.0} : dispatch 2024-09-13T14:56:11.719 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:11.740 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:11.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:11.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:11.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:11 smithi195 ceph-mon[104262]: pgmap v1536: 109 pgs: 109 active+clean; 4.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 72 MiB/s wr, 1.16k op/s 2024-09-13T14:56:11.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:11 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:56:11.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:11 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:56:11.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:11 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1284359688' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 0.0} : dispatch 2024-09-13T14:56:11.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:11 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 0.0} : dispatch 2024-09-13T14:56:11.842 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:11.864 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:11.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:11.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:11.965 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:11.991 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:11.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:11.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:12.093 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:12.114 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:12.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:12.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:12.119 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 58 16 76196 76180 82.0672 26.875 0.0039753 0.0120474 2024-09-13T14:56:12.216 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:12.238 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:12.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:12.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:12.340 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:12.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:12.363 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:12.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:12.464 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:12.489 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:12.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:12.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:12.591 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:12.617 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:12.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:12.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:12.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:12 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]': finished 2024-09-13T14:56:12.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:12 smithi102 ceph-mon[110183]: osdmap e462: 8 total, 8 up, 7 in 2024-09-13T14:56:12.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:12 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]': finished 2024-09-13T14:56:12.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:12 smithi102 ceph-mon[115269]: osdmap e462: 8 total, 8 up, 7 in 2024-09-13T14:56:12.718 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:12.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:12 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]': finished 2024-09-13T14:56:12.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:12 smithi195 ceph-mon[104262]: osdmap e462: 8 total, 8 up, 7 in 2024-09-13T14:56:12.760 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:12.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:12.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:12.862 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:12.887 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:12.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:12.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:12.989 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:13.014 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:13.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:13.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:13.116 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:13.119 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 59 16 76942 76926 81.4665 46.625 0.0132417 0.0122702 2024-09-13T14:56:13.156 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:13.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:13.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:13.258 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:13.284 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:13.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:13.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:13.386 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:13.410 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:13.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:13.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:13.512 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:13.538 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:13.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:13.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:13.639 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:13.676 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:13.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:13.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:13.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:13 smithi195 ceph-mon[104262]: pgmap v1538: 109 pgs: 6 peering, 103 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-13T14:56:13.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:13 smithi195 ceph-mon[104262]: Health check failed: Reduced data availability: 2 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-09-13T14:56:13.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:13 smithi195 ceph-mon[104262]: osdmap e463: 8 total, 8 up, 7 in 2024-09-13T14:56:13.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:13 smithi195 ceph-mon[104262]: 1.6 scrub starts 2024-09-13T14:56:13.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:13 smithi195 ceph-mon[104262]: 1.6 scrub ok 2024-09-13T14:56:13.777 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:13.802 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:13.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:13.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:13.904 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:13.914 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:13 smithi102 ceph-mon[110183]: pgmap v1538: 109 pgs: 6 peering, 103 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-13T14:56:13.914 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:13 smithi102 ceph-mon[110183]: Health check failed: Reduced data availability: 2 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-09-13T14:56:13.914 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:13 smithi102 ceph-mon[110183]: osdmap e463: 8 total, 8 up, 7 in 2024-09-13T14:56:13.914 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:13 smithi102 ceph-mon[110183]: 1.6 scrub starts 2024-09-13T14:56:13.914 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:13 smithi102 ceph-mon[110183]: 1.6 scrub ok 2024-09-13T14:56:13.915 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:13 smithi102 ceph-mon[115269]: pgmap v1538: 109 pgs: 6 peering, 103 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-13T14:56:13.915 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:13 smithi102 ceph-mon[115269]: Health check failed: Reduced data availability: 2 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-09-13T14:56:13.915 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:13 smithi102 ceph-mon[115269]: osdmap e463: 8 total, 8 up, 7 in 2024-09-13T14:56:13.915 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:13 smithi102 ceph-mon[115269]: 1.6 scrub starts 2024-09-13T14:56:13.915 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:13 smithi102 ceph-mon[115269]: 1.6 scrub ok 2024-09-13T14:56:13.949 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:13.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:13.965 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:13.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:14.008 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:14.066 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:14.091 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:14.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:14.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:14.120 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:2024-09-13T14:56:14.119862+0000 min lat: 0.00130255 max lat: 2.27933 avg lat: 0.0123047 2024-09-13T14:56:14.120 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:56:14.120 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 60 16 78023 78007 81.2338 67.5625 0.0123084 0.0123047 2024-09-13T14:56:14.193 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:14.221 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:14.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:14.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:14.323 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:14.370 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:14.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:14.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:14.472 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:14.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:14.520 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:14.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:14.621 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:14.659 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:14.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:14.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:14.760 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:14.804 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:14.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:14.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:14.906 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:14.932 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:14.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:14.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:15.034 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:15.059 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:15.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:15.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:15.120 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 61 16 79051 79035 80.9553 64.25 0.00709981 0.0123462 2024-09-13T14:56:15.162 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:15.201 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:15.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:15.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:15.303 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:15.327 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:15.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:15.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:15.429 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:15.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:15.457 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:15.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:15.500 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is set 2024-09-13T14:56:15.557 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:15.598 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:15.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:15.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:15.602 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:15 smithi102 ceph-mon[110183]: pgmap v1540: 109 pgs: 9 peering, 100 active+clean; 4.6 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 80 MiB/s wr, 1.27k op/s 2024-09-13T14:56:15.602 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:15 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/914697331' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:56:15.603 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:15 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:56:15.603 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:15 smithi102 ceph-mon[115269]: pgmap v1540: 109 pgs: 9 peering, 100 active+clean; 4.6 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 80 MiB/s wr, 1.27k op/s 2024-09-13T14:56:15.603 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:15 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/914697331' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:56:15.603 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:15 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:56:15.701 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:15.728 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:15.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:15.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:15.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:15 smithi195 ceph-mon[104262]: pgmap v1540: 109 pgs: 9 peering, 100 active+clean; 4.6 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 80 MiB/s wr, 1.27k op/s 2024-09-13T14:56:15.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:15 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/914697331' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:56:15.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:15 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:56:15.830 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:15.856 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:15.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:15.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:15.958 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:15.983 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:15.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:15.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:16.084 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:16.112 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:16.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:16.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:16.120 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 62 16 80070 80054 80.6768 63.6875 0.021048 0.0123895 2024-09-13T14:56:16.213 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:16.255 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:16.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:16.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:16.357 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:16.379 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:16.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:16.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:16.481 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:16.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:16.529 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:16.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:16.630 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:16.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:16 smithi102 ceph-mon[110183]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:56:16.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:16 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:56:16.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:16 smithi102 ceph-mon[110183]: osdmap e464: 8 total, 8 up, 7 in 2024-09-13T14:56:16.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:16 smithi102 ceph-mon[115269]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:56:16.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:16 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:56:16.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:16 smithi102 ceph-mon[115269]: osdmap e464: 8 total, 8 up, 7 in 2024-09-13T14:56:16.670 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:56:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:56:16] "GET /metrics HTTP/1.1" 200 36566 "" "Prometheus/2.51.0" 2024-09-13T14:56:16.671 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:16.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:16.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:16.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:16 smithi195 ceph-mon[104262]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:56:16.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:16 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:56:16.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:16 smithi195 ceph-mon[104262]: osdmap e464: 8 total, 8 up, 7 in 2024-09-13T14:56:16.773 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:16.815 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:16.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:16.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:16.917 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:16.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:16.944 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:16.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:17.045 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:17.080 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:17.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:17.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:17.120 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 63 16 81175 81159 80.4923 69.0625 0.0173435 0.0124179 2024-09-13T14:56:17.182 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:17.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:17.219 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:17.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:17.320 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:17.343 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:17.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:17.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:17.444 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:17.474 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:17.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:17.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:17.575 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:17.604 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:17.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:17.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:17.705 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:17.743 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:17.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:17.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:17.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:17 smithi195 ceph-mon[104262]: pgmap v1542: 109 pgs: 9 peering, 100 active+clean; 4.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 56 MiB/s wr, 899 op/s 2024-09-13T14:56:17.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:17 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:56:17.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:17 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:56:17.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:17 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:56:17.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:17 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:56:17.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:17 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:56:17.758 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 6, 7, 5, 2, 1] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T14:56:17.758 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:56:17.759 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_5 2024-09-13T14:56:17.759 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:56:17.829 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:17 smithi102 ceph-mon[110183]: pgmap v1542: 109 pgs: 9 peering, 100 active+clean; 4.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 56 MiB/s wr, 899 op/s 2024-09-13T14:56:17.829 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:17 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:56:17.829 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:17 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:56:17.829 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:17 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:56:17.830 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:17 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:56:17.830 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:17 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:56:17.830 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:17 smithi102 ceph-mon[115269]: pgmap v1542: 109 pgs: 9 peering, 100 active+clean; 4.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 56 MiB/s wr, 899 op/s 2024-09-13T14:56:17.830 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:17 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:56:17.830 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:17 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:56:17.830 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:17 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:56:17.830 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:17 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:56:17.830 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:17 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:56:17.845 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:17.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:17.870 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:17.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:17.971 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:18.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:18.018 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:18.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:18.120 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:18.120 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 64 16 82214 82198 80.2492 64.9375 0.0172081 0.0124558 2024-09-13T14:56:18.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:18.145 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:18.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:18.246 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:18.283 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:18.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:18.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:18.385 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:18.419 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:18.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:18.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:18.521 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:18.558 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:18.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:18.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:18.659 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:18.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:18.711 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:18.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:18.812 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:18.833 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:18 smithi102 ceph-mon[110183]: pgmap v1543: 109 pgs: 109 active+clean; 4.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 54 MiB/s wr, 868 op/s; 10 KiB/s, 0 objects/s recovering 2024-09-13T14:56:18.834 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:18 smithi102 ceph-mon[115269]: pgmap v1543: 109 pgs: 109 active+clean; 4.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 54 MiB/s wr, 868 op/s; 10 KiB/s, 0 objects/s recovering 2024-09-13T14:56:18.855 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:18.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:18.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:18.957 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:18.982 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:18.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:18.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:18.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:18 smithi195 ceph-mon[104262]: pgmap v1543: 109 pgs: 109 active+clean; 4.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 54 MiB/s wr, 868 op/s; 10 KiB/s, 0 objects/s recovering 2024-09-13T14:56:19.084 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:19.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:19.112 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:19.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:19.121 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 65 16 83291 83275 80.0501 67.3125 0.0124527 0.0124865 2024-09-13T14:56:19.212 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:19.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:19.248 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:19.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:19.349 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:19.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:19.384 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:19.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:19.485 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:19.529 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:19.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:19.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:19.549 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:56:19.631 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:19.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:19.666 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:19.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:19.768 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:19.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:19.818 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:19.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:19.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:19 smithi102 ceph-mon[110183]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 8 pgs peering) 2024-09-13T14:56:19.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:19 smithi102 ceph-mon[115269]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 8 pgs peering) 2024-09-13T14:56:19.919 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:19.961 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:19.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:19.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:19.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:19 smithi195 ceph-mon[104262]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 8 pgs peering) 2024-09-13T14:56:20.063 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:20.084 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:20.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:20.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:20.121 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 66 16 84221 84205 79.7178 58.125 0.00985325 0.012521 2024-09-13T14:56:20.186 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:20.212 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:20.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:20.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:20.314 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:20.340 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:20.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:20.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:20.442 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:20.462 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:20.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:20.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:20.564 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:20.592 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:20.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:20.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:20.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:20 smithi102 ceph-mon[110183]: pgmap v1544: 109 pgs: 109 active+clean; 5.0 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 56 MiB/s wr, 889 op/s; 8.0 KiB/s, 0 objects/s recovering 2024-09-13T14:56:20.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:20 smithi102 ceph-mon[115269]: pgmap v1544: 109 pgs: 109 active+clean; 5.0 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 56 MiB/s wr, 889 op/s; 8.0 KiB/s, 0 objects/s recovering 2024-09-13T14:56:20.693 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:20.718 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:20.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:20.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:20.820 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:20.844 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:20.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:20.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:20.946 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:20.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:20.971 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:20.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:20.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:20 smithi195 ceph-mon[104262]: pgmap v1544: 109 pgs: 109 active+clean; 5.0 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 56 MiB/s wr, 889 op/s; 8.0 KiB/s, 0 objects/s recovering 2024-09-13T14:56:21.071 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:21.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:21.111 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:21.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:21.121 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 67 16 85097 85081 79.3452 54.75 0.0120535 0.0125971 2024-09-13T14:56:21.213 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:21.239 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:21.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:21.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:21.341 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:21.364 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:21.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:21.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:21.466 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:21.502 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:21.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:21.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:21.603 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:21.629 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:21.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:21.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:21.730 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:21.759 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:21.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:21.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:21.861 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:21.884 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:21.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:21.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:21.986 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:22.023 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:22.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:22.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:22.121 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 68 16 86153 86137 79.1488 66 0.0144566 0.0126271 2024-09-13T14:56:22.124 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:22.145 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:22.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:22.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:22.247 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:22.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:22.286 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:22.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:22.387 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:22.410 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:22.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:22.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:22.512 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:22.539 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:22.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:22.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:22.641 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:22.664 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:22.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:22.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:22.766 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:22.807 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:22.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:22.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:22.909 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:22.938 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:22.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:22.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:22.999 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:56:22.999 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:56:22.999 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:56:22.999 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:56:22.999 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:56:22.999 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:56:22.999 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:56:23.000 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:56:23.000 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:56:23.000 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:56:23.000 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:56:23.000 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:56:23.000 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:56:23.000 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:56:23.006 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:56:23.006 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:56:23.010 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:23.040 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:23.085 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:23.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:23.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:23.121 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 69 16 87057 87041 78.8204 56.5 0.0182046 0.0126821 2024-09-13T14:56:23.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:22 smithi102 ceph-mon[110183]: pgmap v1545: 109 pgs: 109 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 58 MiB/s wr, 921 op/s; 6.9 KiB/s, 0 objects/s recovering 2024-09-13T14:56:23.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:22 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3819817001' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:56:23.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:22 smithi102 ceph-mon[115269]: pgmap v1545: 109 pgs: 109 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 58 MiB/s wr, 921 op/s; 6.9 KiB/s, 0 objects/s recovering 2024-09-13T14:56:23.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:22 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3819817001' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:56:23.187 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:23.216 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:23.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:23.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:23.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:22 smithi195 ceph-mon[104262]: pgmap v1545: 109 pgs: 109 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 58 MiB/s wr, 921 op/s; 6.9 KiB/s, 0 objects/s recovering 2024-09-13T14:56:23.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:22 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3819817001' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:56:23.318 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:23.358 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:23.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:23.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:23.458 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:56:23.458 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [], 2024-09-13T14:56:23.458 INFO:teuthology.orchestra.run.smithi102.stdout: "complaint_time": 30, 2024-09-13T14:56:23.458 INFO:teuthology.orchestra.run.smithi102.stdout: "num_blocked_ops": 0 2024-09-13T14:56:23.459 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:56:23.460 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:23.510 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:23.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:23.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:23.612 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:23.637 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:23.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:23.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:23.739 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:23.760 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:23.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:23.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:23.862 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:23.895 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:23.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:23.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:23.997 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:24.023 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:24.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:24.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:24.121 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 70 16 88005 87989 78.5408 59.25 0.0106265 0.0127264 2024-09-13T14:56:24.125 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:24.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:24.161 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:24.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:24.262 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:24.304 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:24.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:24.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:24.406 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:24.448 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:24.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:24.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:24.550 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:24.581 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:24.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:24.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:24.683 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:24.720 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:24.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:24.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:24.821 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:24.849 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:24.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:24.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:24.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:24 smithi102 ceph-mon[110183]: pgmap v1546: 109 pgs: 109 active+clean; 5.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 61 MiB/s wr, 971 op/s; 6.4 KiB/s, 0 objects/s recovering 2024-09-13T14:56:24.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:24 smithi102 ceph-mon[115269]: pgmap v1546: 109 pgs: 109 active+clean; 5.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 61 MiB/s wr, 971 op/s; 6.4 KiB/s, 0 objects/s recovering 2024-09-13T14:56:24.951 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:24.990 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:24.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:24.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:25.092 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:25.120 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:25.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:25.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:25.122 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 71 15 89061 89046 78.365 66.0625 0.00993789 0.0127553 2024-09-13T14:56:25.142 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T14:56:25.222 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:25.247 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:25.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:25.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:25.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:24 smithi195 ceph-mon[104262]: pgmap v1546: 109 pgs: 109 active+clean; 5.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 61 MiB/s wr, 971 op/s; 6.4 KiB/s, 0 objects/s recovering 2024-09-13T14:56:25.349 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:25.374 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:25.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:25.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:25.476 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:25.499 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:25.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:25.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:25.601 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:25.639 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:25.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:25.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:25.740 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:25.776 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:25.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:25.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:25.878 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:25.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:25.917 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:25.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:26.018 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:26.040 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:26.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:26.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:26.121 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 72 16 89840 89824 77.9519 48.625 0.00316623 0.0128003 2024-09-13T14:56:26.142 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:26.163 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:26.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:26.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:26.264 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:26.289 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:26.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:26.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:26.391 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:26.418 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:26.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:26.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:26.520 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:26.559 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:26.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:26.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:26.661 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:26.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:26.692 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:26.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:26.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:56:26.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:56:26.793 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:26.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:26.834 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:26.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:26.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:56:26.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:56:26.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:56:26.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:56:26.919 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:56:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:56:26] "GET /metrics HTTP/1.1" 200 36575 "" "Prometheus/2.51.0" 2024-09-13T14:56:26.934 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:26.956 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:26.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:26.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:27.058 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:27.082 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:27.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:27.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:27.121 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 73 16 90513 90497 77.4603 42.0625 0.0119404 0.0129034 2024-09-13T14:56:27.184 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:27.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:27.225 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:27.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:27.326 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:27.354 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:27.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:27.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:27.456 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:27.482 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:27.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:27.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:27.583 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:27.622 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:27.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:27.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:27.724 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:27.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:27 smithi195 ceph-mon[104262]: pgmap v1547: 109 pgs: 109 active+clean; 5.3 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 59 MiB/s wr, 951 op/s; 6.2 KiB/s, 0 objects/s recovering 2024-09-13T14:56:27.751 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:27.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:27.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:27.853 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:27.877 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:27.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:27.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:27.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:27 smithi102 ceph-mon[110183]: pgmap v1547: 109 pgs: 109 active+clean; 5.3 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 59 MiB/s wr, 951 op/s; 6.2 KiB/s, 0 objects/s recovering 2024-09-13T14:56:27.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:27 smithi102 ceph-mon[115269]: pgmap v1547: 109 pgs: 109 active+clean; 5.3 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 59 MiB/s wr, 951 op/s; 6.2 KiB/s, 0 objects/s recovering 2024-09-13T14:56:27.979 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:28.015 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:28.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:28.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:28.117 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:28.123 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 74 16 91489 91473 77.2376 61 0.0125179 0.0129421 2024-09-13T14:56:28.159 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:28.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:28.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:28.261 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:28.300 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:28.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:28.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:28.403 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:28.444 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:28.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:28.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:28.546 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:28.583 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:28.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:28.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:28.685 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:28.709 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:28.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:28.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:28.811 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:28.832 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:28.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:28.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:28.933 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:28.954 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:28.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:28.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:29.056 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:29.100 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:29.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:29.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:29.122 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 75 16 92289 92273 76.8745 50 0.00622737 0.0130034 2024-09-13T14:56:29.201 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:29.222 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:29.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:29.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:29.324 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:29.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:29.376 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:29.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:29.476 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:29.511 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:29.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:29.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:29.612 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:29.648 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:29.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:29.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:29.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:29 smithi195 ceph-mon[104262]: pgmap v1548: 109 pgs: 109 active+clean; 5.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 64 MiB/s wr, 1.02k op/s; 5.3 KiB/s, 0 objects/s recovering 2024-09-13T14:56:29.750 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:29.795 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:29.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:29.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:29.898 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:29.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:29 smithi102 ceph-mon[110183]: pgmap v1548: 109 pgs: 109 active+clean; 5.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 64 MiB/s wr, 1.02k op/s; 5.3 KiB/s, 0 objects/s recovering 2024-09-13T14:56:29.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:29 smithi102 ceph-mon[115269]: pgmap v1548: 109 pgs: 109 active+clean; 5.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 64 MiB/s wr, 1.02k op/s; 5.3 KiB/s, 0 objects/s recovering 2024-09-13T14:56:29.926 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:29.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:29.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:30.028 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:30.054 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:30.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:30.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:30.124 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 76 16 93196 93180 76.607 56.6875 0.0143543 0.0130482 2024-09-13T14:56:30.156 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:30.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:30.183 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:30.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:30.285 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:30.310 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:30.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:30.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:30.412 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:30.446 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:30.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:30.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:30.548 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:30.587 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:30.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:30.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:30.689 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:30.689 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:30.693 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:30.712 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:30.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:30.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:30.813 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:30.840 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:30.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:30.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:30.942 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:30.982 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:30.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:30.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:31.083 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:31.124 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:31.124 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 77 16 94161 94145 76.3953 60.3125 0.0146273 0.0130844 2024-09-13T14:56:31.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:31.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:31.226 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:31.257 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:31.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:31.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:31.359 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:31.384 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:31.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:31.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:31.486 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:31.529 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:31.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:31.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:31.631 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:31.664 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:31.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:31.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:31.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:31 smithi195 ceph-mon[104262]: pgmap v1549: 109 pgs: 109 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 60 MiB/s wr, 967 op/s 2024-09-13T14:56:31.767 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:31.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:31.802 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:31.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:31.903 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:31.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:31 smithi102 ceph-mon[110183]: pgmap v1549: 109 pgs: 109 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 60 MiB/s wr, 967 op/s 2024-09-13T14:56:31.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:31 smithi102 ceph-mon[115269]: pgmap v1549: 109 pgs: 109 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 60 MiB/s wr, 967 op/s 2024-09-13T14:56:31.929 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:31.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:31.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:32.013 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:56:32.031 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:32.055 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:32.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:32.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:32.124 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 78 16 95140 95124 76.2003 61.1875 0.034766 0.013117 2024-09-13T14:56:32.157 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:32.204 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:32.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:32.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:32.306 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:32.332 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:32.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:32.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:32.434 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:32.473 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:32.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:32.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:32.477 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:56:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=cleanup t=2024-09-13T14:56:32.213383839Z level=info msg="Completed cleanup jobs" duration=16.838606ms 2024-09-13T14:56:32.477 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:56:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=plugins.update.checker t=2024-09-13T14:56:32.352082169Z level=info msg="Update check succeeded" duration=74.363983ms 2024-09-13T14:56:32.575 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:32.615 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:32.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:32.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:32.717 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:32.738 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:32.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:32.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:32.747 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 14:56:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=grafana.update.checker t=2024-09-13T14:56:32.476394723Z level=info msg="Update check succeeded" duration=200.865491ms 2024-09-13T14:56:32.840 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:32.860 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:32.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:32.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:32.962 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:33.004 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:33.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:33.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:33.106 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:33.124 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 79 16 95882 95866 75.8228 46.375 0.00621833 0.0131837 2024-09-13T14:56:33.131 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:33.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:33.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:33.233 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:33.254 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:33.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:33.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:33.356 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:33.380 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:33.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:33.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:33.482 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:33.516 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:33.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:33.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:33.618 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:33.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:33.645 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:33.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:33.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:33 smithi102 ceph-mon[110183]: pgmap v1550: 109 pgs: 109 active+clean; 5.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 59 MiB/s wr, 941 op/s 2024-09-13T14:56:33.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:33 smithi102 ceph-mon[115269]: pgmap v1550: 109 pgs: 109 active+clean; 5.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 59 MiB/s wr, 941 op/s 2024-09-13T14:56:33.746 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:33.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:33 smithi195 ceph-mon[104262]: pgmap v1550: 109 pgs: 109 active+clean; 5.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 59 MiB/s wr, 941 op/s 2024-09-13T14:56:33.774 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:33.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:33.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:33.876 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:33.902 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:33.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:33.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:34.003 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:34.027 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:34.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:34.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:34.124 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:2024-09-13T14:56:34.124022+0000 min lat: 0.00130255 max lat: 2.27933 avg lat: 0.0132343 2024-09-13T14:56:34.124 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:56:34.124 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 80 16 96721 96705 75.5305 52.4375 0.0048808 0.0132343 2024-09-13T14:56:34.129 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:34.167 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:34.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:34.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:34.269 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:34.295 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:34.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:34.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:34.397 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:34.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:34.450 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:34.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:34.551 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:34.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:34.577 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:34.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:34.678 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:34.702 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:34.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:34.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:34.804 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:34.867 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:34.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:34.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:34.874 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:56:34.881 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:56:34.969 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:35.007 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:35.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:35.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:35.109 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:35.127 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 81 16 97594 97578 75.2709 54.5625 0.0117252 0.0132794 2024-09-13T14:56:35.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:35.147 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:35.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:35.248 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:35.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:35.285 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:35.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:35.386 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:35.416 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:35.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:35.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:35.518 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:35.541 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:35.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:35.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:35.643 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:35.668 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:35.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:35.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:35.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:35 smithi195 ceph-mon[104262]: pgmap v1551: 109 pgs: 109 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 56 MiB/s wr, 901 op/s 2024-09-13T14:56:35.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:35 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3403037356' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:56:35.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:35 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:56:35.770 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:35.827 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:35.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:35.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:35.888 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is set 2024-09-13T14:56:35.888 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:35 smithi102 ceph-mon[110183]: pgmap v1551: 109 pgs: 109 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 56 MiB/s wr, 901 op/s 2024-09-13T14:56:35.888 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:35 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3403037356' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:56:35.889 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:35 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:56:35.889 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:35 smithi102 ceph-mon[115269]: pgmap v1551: 109 pgs: 109 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 56 MiB/s wr, 901 op/s 2024-09-13T14:56:35.889 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:35 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3403037356' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:56:35.889 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:35 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:56:35.928 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:35.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:35.957 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:35.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:36.058 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:36.101 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:36.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:36.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:36.125 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 82 16 98524 98508 75.0618 58.125 0.0147632 0.0133172 2024-09-13T14:56:36.203 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:36.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:36.256 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:36.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:36.357 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:36.394 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:36.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:36.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:36.492 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:56:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:56:36] "GET /metrics HTTP/1.1" 200 36575 "" "Prometheus/2.51.0" 2024-09-13T14:56:36.495 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:36.521 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:36.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:36.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:36.623 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:36.653 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:36.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:36.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:36.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:36 smithi195 ceph-mon[104262]: from='client.36783 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:56:36.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:36 smithi195 ceph-mon[104262]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:56:36.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:36 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:56:36.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:36 smithi195 ceph-mon[104262]: osdmap e465: 8 total, 8 up, 7 in 2024-09-13T14:56:36.755 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:36.782 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:36.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:36.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:36.854 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:36 smithi102 ceph-mon[110183]: from='client.36783 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:56:36.855 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:36 smithi102 ceph-mon[110183]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:56:36.855 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:36 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:56:36.855 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:36 smithi102 ceph-mon[110183]: osdmap e465: 8 total, 8 up, 7 in 2024-09-13T14:56:36.855 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:36 smithi102 ceph-mon[115269]: from='client.36783 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:56:36.855 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:36 smithi102 ceph-mon[115269]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:56:36.855 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:36 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:56:36.855 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:36 smithi102 ceph-mon[115269]: osdmap e465: 8 total, 8 up, 7 in 2024-09-13T14:56:36.885 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:36.911 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:36.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:36.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:37.012 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:37.037 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:37.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:37.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:37.125 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 83 16 99468 99452 74.8682 59 0.0159715 0.0133515 2024-09-13T14:56:37.138 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:37.163 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:37.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:37.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:37.265 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:37.305 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:37.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:37.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:37.407 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:37.432 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:37.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:37.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:37.534 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:37.570 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:37.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:37.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:37.671 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:37.683 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":1551,"stamp":"2024-09-13T14:56:33.793728+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6207313893,"num_objects":94671,"num_object_clones":0,"num_object_copies":284013,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94671,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":95551,"num_write_kb":6069777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5178,"num_bytes_recovered":333584752,"num_keys_recovered":30,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98205,"ondisk_log_size":98205,"up":327,"acting":327,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":328,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":23954972,"kb_used_data":18104548,"kb_used_omap":40,"kb_used_meta":5850327,"kb_avail":632261092,"statfs":{"total":671965249536,"available":647435358208,"internally_reserved":0,"allocated":18539057152,"data_stored":18522426896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41319,"internal_metadata":5990735513},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,3,1,9,6],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":709492736,"num_objects":10826,"num_object_clones":0,"num_object_copies":32478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10826,"num_write_kb":692864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003232"},"pg_stats":[{"pgid":"1.11","version":"90'149","reported_seq":439,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-13T14:56:12.492033+0000","last_change":"2024-09-13T14:56:12.490567+0000","last_active":"2024-09-13T14:56:12.492033+0000","last_peered":"2024-09-13T14:56:12.492033+0000","last_clean":"2024-09-13T14:56:12.492033+0000","last_became_active":"2024-09-13T14:56:12.490253+0000","last_became_peered":"2024-09-13T14:56:12.490253+0000","last_unstale":"2024-09-13T14:56:12.492033+0000","last_undegraded":"2024-09-13T14:56:12.492033+0000","last_fullsized":"2024-09-13T14:56:12.492033+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:38.688341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.688341+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:20.151605+0000","scrub_duration":0.0054331650000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":841,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.525240+0000","last_change":"2024-09-13T14:47:47.868783+0000","last_active":"2024-09-13T14:56:15.525240+0000","last_peered":"2024-09-13T14:56:15.525240+0000","last_clean":"2024-09-13T14:56:15.525240+0000","last_became_active":"2024-09-13T14:47:47.868478+0000","last_became_peered":"2024-09-13T14:47:47.868478+0000","last_unstale":"2024-09-13T14:56:15.525240+0000","last_undegraded":"2024-09-13T14:56:15.525240+0000","last_fullsized":"2024-09-13T14:56:15.525240+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:27:01.784142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":502,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-13T14:56:12.484304+0000","last_change":"2024-09-13T14:55:17.943558+0000","last_active":"2024-09-13T14:56:12.484304+0000","last_peered":"2024-09-13T14:56:12.484304+0000","last_clean":"2024-09-13T14:56:12.484304+0000","last_became_active":"2024-09-13T14:55:17.927162+0000","last_became_peered":"2024-09-13T14:55:17.927162+0000","last_unstale":"2024-09-13T14:56:12.484304+0000","last_undegraded":"2024-09-13T14:56:12.484304+0000","last_fullsized":"2024-09-13T14:56:12.484304+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:37:44.820559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":102,"num_keys_recovered":30,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"90'149","reported_seq":531,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-13T14:56:12.481484+0000","last_change":"2024-09-13T14:47:41.482552+0000","last_active":"2024-09-13T14:56:12.481484+0000","last_peered":"2024-09-13T14:56:12.481484+0000","last_clean":"2024-09-13T14:56:12.481484+0000","last_became_active":"2024-09-13T14:47:38.466659+0000","last_became_peered":"2024-09-13T14:47:38.466659+0000","last_unstale":"2024-09-13T14:56:12.481484+0000","last_undegraded":"2024-09-13T14:56:12.481484+0000","last_fullsized":"2024-09-13T14:56:12.481484+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:41.482364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:41.482364+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:50:49.239389+0000","scrub_duration":0.027843855000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":268,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-13T14:56:12.482143+0000","last_change":"2024-09-13T14:36:13.862181+0000","last_active":"2024-09-13T14:56:12.482143+0000","last_peered":"2024-09-13T14:56:12.482143+0000","last_clean":"2024-09-13T14:56:12.482143+0000","last_became_active":"2024-09-13T14:36:13.861873+0000","last_became_peered":"2024-09-13T14:36:13.861873+0000","last_unstale":"2024-09-13T14:56:12.482143+0000","last_undegraded":"2024-09-13T14:56:12.482143+0000","last_fullsized":"2024-09-13T14:56:12.482143+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:14.085700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":40,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-13T14:56:12.491402+0000","last_change":"2024-09-13T14:56:12.489695+0000","last_active":"2024-09-13T14:56:12.491402+0000","last_peered":"2024-09-13T14:56:12.491402+0000","last_clean":"2024-09-13T14:56:12.491402+0000","last_became_active":"2024-09-13T14:56:12.489302+0000","last_became_peered":"2024-09-13T14:56:12.489302+0000","last_unstale":"2024-09-13T14:56:12.491402+0000","last_undegraded":"2024-09-13T14:56:12.491402+0000","last_fullsized":"2024-09-13T14:56:12.491402+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:28:53.275229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"90'149","reported_seq":701,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.527879+0000","last_change":"2024-09-13T14:55:20.150382+0000","last_active":"2024-09-13T14:56:15.527879+0000","last_peered":"2024-09-13T14:56:15.527879+0000","last_clean":"2024-09-13T14:56:15.527879+0000","last_became_active":"2024-09-13T14:55:20.149422+0000","last_became_peered":"2024-09-13T14:55:20.149422+0000","last_unstale":"2024-09-13T14:56:15.527879+0000","last_undegraded":"2024-09-13T14:56:15.527879+0000","last_fullsized":"2024-09-13T14:56:15.527879+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:51.226748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:51.226748+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:10.410483+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":353,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.522837+0000","last_change":"2024-09-13T14:56:12.490265+0000","last_active":"2024-09-13T14:56:15.522837+0000","last_peered":"2024-09-13T14:56:15.522837+0000","last_clean":"2024-09-13T14:56:15.522837+0000","last_became_active":"2024-09-13T14:56:12.489937+0000","last_became_peered":"2024-09-13T14:56:12.489937+0000","last_unstale":"2024-09-13T14:56:15.522837+0000","last_undegraded":"2024-09-13T14:56:15.522837+0000","last_fullsized":"2024-09-13T14:56:15.522837+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:49.461532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":448,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-13T14:56:12.484327+0000","last_change":"2024-09-13T14:55:17.934695+0000","last_active":"2024-09-13T14:56:12.484327+0000","last_peered":"2024-09-13T14:56:12.484327+0000","last_clean":"2024-09-13T14:56:12.484327+0000","last_became_active":"2024-09-13T14:55:17.934165+0000","last_became_peered":"2024-09-13T14:55:17.934165+0000","last_unstale":"2024-09-13T14:56:12.484327+0000","last_undegraded":"2024-09-13T14:56:12.484327+0000","last_fullsized":"2024-09-13T14:56:12.484327+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:36:25.739706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"90'149","reported_seq":548,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-13T14:56:12.482123+0000","last_change":"2024-09-13T14:47:50.892667+0000","last_active":"2024-09-13T14:56:12.482123+0000","last_peered":"2024-09-13T14:56:12.482123+0000","last_clean":"2024-09-13T14:56:12.482123+0000","last_became_active":"2024-09-13T14:47:50.883312+0000","last_became_peered":"2024-09-13T14:47:50.883312+0000","last_unstale":"2024-09-13T14:56:12.482123+0000","last_undegraded":"2024-09-13T14:56:12.482123+0000","last_fullsized":"2024-09-13T14:56:12.482123+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:45.441819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:45.441819+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:39.138573+0000","scrub_duration":0.021368558999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":867,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.519709+0000","last_change":"2024-09-13T14:50:27.491166+0000","last_active":"2024-09-13T14:56:15.519709+0000","last_peered":"2024-09-13T14:56:15.519709+0000","last_clean":"2024-09-13T14:56:15.519709+0000","last_became_active":"2024-09-13T14:50:27.489905+0000","last_became_peered":"2024-09-13T14:50:27.489905+0000","last_unstale":"2024-09-13T14:56:15.519709+0000","last_undegraded":"2024-09-13T14:56:15.519709+0000","last_fullsized":"2024-09-13T14:56:15.519709+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:32.237904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":133,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-13T14:56:12.490548+0000","last_change":"2024-09-13T14:56:12.490197+0000","last_active":"2024-09-13T14:56:12.490548+0000","last_peered":"2024-09-13T14:56:12.490548+0000","last_clean":"2024-09-13T14:56:12.490548+0000","last_became_active":"2024-09-13T14:56:12.489554+0000","last_became_peered":"2024-09-13T14:56:12.489554+0000","last_unstale":"2024-09-13T14:56:12.490548+0000","last_undegraded":"2024-09-13T14:56:12.490548+0000","last_fullsized":"2024-09-13T14:56:12.490548+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:06.773211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"464'5838","reported_seq":7062,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:31.724657+0000","last_change":"2024-09-13T14:55:22.944506+0000","last_active":"2024-09-13T14:56:31.724657+0000","last_peered":"2024-09-13T14:56:31.724657+0000","last_clean":"2024-09-13T14:56:31.724657+0000","last_became_active":"2024-09-13T14:55:22.942014+0000","last_became_peered":"2024-09-13T14:55:22.942014+0000","last_unstale":"2024-09-13T14:56:31.724657+0000","last_undegraded":"2024-09-13T14:56:31.724657+0000","last_fullsized":"2024-09-13T14:56:31.724657+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":5838,"log_dups_size":0,"ondisk_log_size":5838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:00:33.167429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382599168,"num_objects":5838,"num_object_clones":0,"num_object_copies":17514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5838,"num_write_kb":373632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":584,"num_bytes_recovered":38273024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"90'149","reported_seq":567,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-13T14:56:12.484712+0000","last_change":"2024-09-13T14:55:20.147437+0000","last_active":"2024-09-13T14:56:12.484712+0000","last_peered":"2024-09-13T14:56:12.484712+0000","last_clean":"2024-09-13T14:56:12.484712+0000","last_became_active":"2024-09-13T14:55:20.147197+0000","last_became_peered":"2024-09-13T14:55:20.147197+0000","last_unstale":"2024-09-13T14:56:12.484712+0000","last_undegraded":"2024-09-13T14:56:12.484712+0000","last_fullsized":"2024-09-13T14:56:12.484712+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:40.786610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:40.786610+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:36.008713+0000","scrub_duration":0.013725185000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":245,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-13T14:56:12.485797+0000","last_change":"2024-09-13T14:55:17.947508+0000","last_active":"2024-09-13T14:56:12.485797+0000","last_peered":"2024-09-13T14:56:12.485797+0000","last_clean":"2024-09-13T14:56:12.485797+0000","last_became_active":"2024-09-13T14:55:17.932604+0000","last_became_peered":"2024-09-13T14:55:17.932604+0000","last_unstale":"2024-09-13T14:56:12.485797+0000","last_undegraded":"2024-09-13T14:56:12.485797+0000","last_fullsized":"2024-09-13T14:56:12.485797+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:55.084612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":132,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":289,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-13T14:56:12.481597+0000","last_change":"2024-09-13T14:38:53.898907+0000","last_active":"2024-09-13T14:56:12.481597+0000","last_peered":"2024-09-13T14:56:12.481597+0000","last_clean":"2024-09-13T14:56:12.481597+0000","last_became_active":"2024-09-13T14:38:53.898572+0000","last_became_peered":"2024-09-13T14:38:53.898572+0000","last_unstale":"2024-09-13T14:56:12.481597+0000","last_undegraded":"2024-09-13T14:56:12.481597+0000","last_fullsized":"2024-09-13T14:56:12.481597+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:29.222150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"464'5957","reported_seq":6024,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:33.319743+0000","last_change":"2024-09-13T14:54:58.384717+0000","last_active":"2024-09-13T14:56:33.319743+0000","last_peered":"2024-09-13T14:56:33.319743+0000","last_clean":"2024-09-13T14:56:33.319743+0000","last_became_active":"2024-09-13T14:54:58.384225+0000","last_became_peered":"2024-09-13T14:54:58.384225+0000","last_unstale":"2024-09-13T14:56:33.319743+0000","last_undegraded":"2024-09-13T14:56:33.319743+0000","last_fullsized":"2024-09-13T14:56:33.319743+0000","mapping_epoch":436,"log_start":"439'100","ondisk_log_start":"439'100","created":436,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":5857,"log_dups_size":0,"ondisk_log_size":5857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:57.584308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390397952,"num_objects":5957,"num_object_clones":0,"num_object_copies":17871,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5957,"num_write_kb":381248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"90'149","reported_seq":714,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.525153+0000","last_change":"2024-09-13T14:47:38.466101+0000","last_active":"2024-09-13T14:56:15.525153+0000","last_peered":"2024-09-13T14:56:15.525153+0000","last_clean":"2024-09-13T14:56:15.525153+0000","last_became_active":"2024-09-13T14:47:38.465607+0000","last_became_peered":"2024-09-13T14:47:38.465607+0000","last_unstale":"2024-09-13T14:56:15.525153+0000","last_undegraded":"2024-09-13T14:56:15.525153+0000","last_fullsized":"2024-09-13T14:56:15.525153+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:32.060817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:32.060817+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:55.362724+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":258,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-13T14:56:12.485699+0000","last_change":"2024-09-13T14:55:17.949010+0000","last_active":"2024-09-13T14:56:12.485699+0000","last_peered":"2024-09-13T14:56:12.485699+0000","last_clean":"2024-09-13T14:56:12.485699+0000","last_became_active":"2024-09-13T14:55:17.932712+0000","last_became_peered":"2024-09-13T14:55:17.932712+0000","last_unstale":"2024-09-13T14:56:12.485699+0000","last_undegraded":"2024-09-13T14:56:12.485699+0000","last_fullsized":"2024-09-13T14:56:12.485699+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:46:43.390958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":404,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.530393+0000","last_change":"2024-09-13T14:55:17.929723+0000","last_active":"2024-09-13T14:56:15.530393+0000","last_peered":"2024-09-13T14:56:15.530393+0000","last_clean":"2024-09-13T14:56:15.530393+0000","last_became_active":"2024-09-13T14:55:17.929131+0000","last_became_peered":"2024-09-13T14:55:17.929131+0000","last_unstale":"2024-09-13T14:56:15.530393+0000","last_undegraded":"2024-09-13T14:56:15.530393+0000","last_fullsized":"2024-09-13T14:56:15.530393+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:27:15.828973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.4","version":"464'6019","reported_seq":7342,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:33.118900+0000","last_change":"2024-09-13T14:56:12.495568+0000","last_active":"2024-09-13T14:56:33.118900+0000","last_peered":"2024-09-13T14:56:33.118900+0000","last_clean":"2024-09-13T14:56:33.118900+0000","last_became_active":"2024-09-13T14:56:12.490664+0000","last_became_peered":"2024-09-13T14:56:12.490664+0000","last_unstale":"2024-09-13T14:56:33.118900+0000","last_undegraded":"2024-09-13T14:56:33.118900+0000","last_fullsized":"2024-09-13T14:56:33.118900+0000","mapping_epoch":462,"log_start":"439'200","ondisk_log_start":"439'200","created":436,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":5819,"log_dups_size":0,"ondisk_log_size":5819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:23:39.927054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394461184,"num_objects":6019,"num_object_clones":0,"num_object_copies":18057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6019,"num_write_kb":385216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":623,"num_bytes_recovered":40828928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.f","version":"90'149","reported_seq":592,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-13T14:56:12.482970+0000","last_change":"2024-09-13T14:55:20.140396+0000","last_active":"2024-09-13T14:56:12.482970+0000","last_peered":"2024-09-13T14:56:12.482970+0000","last_clean":"2024-09-13T14:56:12.482970+0000","last_became_active":"2024-09-13T14:55:20.140087+0000","last_became_peered":"2024-09-13T14:55:20.140087+0000","last_unstale":"2024-09-13T14:56:12.482970+0000","last_undegraded":"2024-09-13T14:56:12.482970+0000","last_fullsized":"2024-09-13T14:56:12.482970+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:34.091782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:34.091782+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:46:05.540133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":470,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.517081+0000","last_change":"2024-09-13T14:37:49.611774+0000","last_active":"2024-09-13T14:56:15.517081+0000","last_peered":"2024-09-13T14:56:15.517081+0000","last_clean":"2024-09-13T14:56:15.517081+0000","last_became_active":"2024-09-13T14:37:49.611088+0000","last_became_peered":"2024-09-13T14:37:49.611088+0000","last_unstale":"2024-09-13T14:56:15.517081+0000","last_undegraded":"2024-09-13T14:56:15.517081+0000","last_fullsized":"2024-09-13T14:56:15.517081+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:08.933213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":269,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.525705+0000","last_change":"2024-09-13T14:47:47.865523+0000","last_active":"2024-09-13T14:56:15.525705+0000","last_peered":"2024-09-13T14:56:15.525705+0000","last_clean":"2024-09-13T14:56:15.525705+0000","last_became_active":"2024-09-13T14:47:47.865180+0000","last_became_peered":"2024-09-13T14:47:47.865180+0000","last_unstale":"2024-09-13T14:56:15.525705+0000","last_undegraded":"2024-09-13T14:56:15.525705+0000","last_fullsized":"2024-09-13T14:56:15.525705+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:39:35.479251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.5","version":"464'5786","reported_seq":5853,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:31.444195+0000","last_change":"2024-09-13T14:54:57.846849+0000","last_active":"2024-09-13T14:56:31.444195+0000","last_peered":"2024-09-13T14:56:31.444195+0000","last_clean":"2024-09-13T14:56:31.444195+0000","last_became_active":"2024-09-13T14:54:57.846557+0000","last_became_peered":"2024-09-13T14:54:57.846557+0000","last_unstale":"2024-09-13T14:56:31.444195+0000","last_undegraded":"2024-09-13T14:56:31.444195+0000","last_fullsized":"2024-09-13T14:56:31.444195+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":5786,"log_dups_size":0,"ondisk_log_size":5786,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:33:07.871445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":379191296,"num_objects":5786,"num_object_clones":0,"num_object_copies":17358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5786,"num_write_kb":370304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"90'149","reported_seq":478,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-13T14:56:12.485214+0000","last_change":"2024-09-13T14:55:20.145243+0000","last_active":"2024-09-13T14:56:12.485214+0000","last_peered":"2024-09-13T14:56:12.485214+0000","last_clean":"2024-09-13T14:56:12.485214+0000","last_became_active":"2024-09-13T14:55:20.145001+0000","last_became_peered":"2024-09-13T14:55:20.145001+0000","last_unstale":"2024-09-13T14:56:12.485214+0000","last_undegraded":"2024-09-13T14:56:12.485214+0000","last_fullsized":"2024-09-13T14:56:12.485214+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:30.526999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:30.526999+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:11:56.231618+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":280,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.499622+0000","last_change":"2024-09-13T14:37:49.168003+0000","last_active":"2024-09-13T14:56:15.499622+0000","last_peered":"2024-09-13T14:56:15.499622+0000","last_clean":"2024-09-13T14:56:15.499622+0000","last_became_active":"2024-09-13T14:37:49.167638+0000","last_became_peered":"2024-09-13T14:37:49.167638+0000","last_unstale":"2024-09-13T14:56:15.499622+0000","last_undegraded":"2024-09-13T14:56:15.499622+0000","last_fullsized":"2024-09-13T14:56:15.499622+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:29.051981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":432,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.527622+0000","last_change":"2024-09-13T14:56:12.491208+0000","last_active":"2024-09-13T14:56:15.527622+0000","last_peered":"2024-09-13T14:56:15.527622+0000","last_clean":"2024-09-13T14:56:15.527622+0000","last_became_active":"2024-09-13T14:56:12.490943+0000","last_became_peered":"2024-09-13T14:56:12.490943+0000","last_unstale":"2024-09-13T14:56:15.527622+0000","last_undegraded":"2024-09-13T14:56:15.527622+0000","last_fullsized":"2024-09-13T14:56:15.527622+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:31:57.892194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"464'5770","reported_seq":6978,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:29.451436+0000","last_change":"2024-09-13T14:55:37.154924+0000","last_active":"2024-09-13T14:56:29.451436+0000","last_peered":"2024-09-13T14:56:29.451436+0000","last_clean":"2024-09-13T14:56:29.451436+0000","last_became_active":"2024-09-13T14:55:37.153166+0000","last_became_peered":"2024-09-13T14:55:37.153166+0000","last_unstale":"2024-09-13T14:56:29.451436+0000","last_undegraded":"2024-09-13T14:56:29.451436+0000","last_fullsized":"2024-09-13T14:56:29.451436+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":5770,"log_dups_size":0,"ondisk_log_size":5770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:06:19.253339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":378142720,"num_objects":5770,"num_object_clones":0,"num_object_copies":17310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5770,"num_write_kb":369280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":565,"num_bytes_recovered":37027840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"90'149","reported_seq":641,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-13T14:56:12.484156+0000","last_change":"2024-09-13T14:46:55.355650+0000","last_active":"2024-09-13T14:56:12.484156+0000","last_peered":"2024-09-13T14:56:12.484156+0000","last_clean":"2024-09-13T14:56:12.484156+0000","last_became_active":"2024-09-13T14:46:48.150338+0000","last_became_peered":"2024-09-13T14:46:48.150338+0000","last_unstale":"2024-09-13T14:56:12.484156+0000","last_undegraded":"2024-09-13T14:56:12.484156+0000","last_fullsized":"2024-09-13T14:56:12.484156+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:46:55.355521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:46:55.355521+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:30:53.006278+0000","scrub_duration":0.002025448,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":827,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.520284+0000","last_change":"2024-09-13T14:55:17.944596+0000","last_active":"2024-09-13T14:56:15.520284+0000","last_peered":"2024-09-13T14:56:15.520284+0000","last_clean":"2024-09-13T14:56:15.520284+0000","last_became_active":"2024-09-13T14:55:17.935767+0000","last_became_peered":"2024-09-13T14:55:17.935767+0000","last_unstale":"2024-09-13T14:56:15.520284+0000","last_undegraded":"2024-09-13T14:56:15.520284+0000","last_fullsized":"2024-09-13T14:56:15.520284+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:22:19.619639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":356,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.510578+0000","last_change":"2024-09-13T14:55:17.933005+0000","last_active":"2024-09-13T14:56:15.510578+0000","last_peered":"2024-09-13T14:56:15.510578+0000","last_clean":"2024-09-13T14:56:15.510578+0000","last_became_active":"2024-09-13T14:55:17.931575+0000","last_became_peered":"2024-09-13T14:55:17.931575+0000","last_unstale":"2024-09-13T14:56:15.510578+0000","last_undegraded":"2024-09-13T14:56:15.510578+0000","last_fullsized":"2024-09-13T14:56:15.510578+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:50:20.352970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"464'5920","reported_seq":5986,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:31.487332+0000","last_change":"2024-09-13T14:54:57.846161+0000","last_active":"2024-09-13T14:56:31.487332+0000","last_peered":"2024-09-13T14:56:31.487332+0000","last_clean":"2024-09-13T14:56:31.487332+0000","last_became_active":"2024-09-13T14:54:57.845348+0000","last_became_peered":"2024-09-13T14:54:57.845348+0000","last_unstale":"2024-09-13T14:56:31.487332+0000","last_undegraded":"2024-09-13T14:56:31.487332+0000","last_fullsized":"2024-09-13T14:56:31.487332+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":5920,"log_dups_size":0,"ondisk_log_size":5920,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:28:09.809813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":387973120,"num_objects":5920,"num_object_clones":0,"num_object_copies":17760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5920,"num_write_kb":378880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"90'149","reported_seq":544,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-13T14:56:12.481484+0000","last_change":"2024-09-13T14:46:54.393663+0000","last_active":"2024-09-13T14:56:12.481484+0000","last_peered":"2024-09-13T14:56:12.481484+0000","last_clean":"2024-09-13T14:56:12.481484+0000","last_became_active":"2024-09-13T14:46:48.149004+0000","last_became_peered":"2024-09-13T14:46:48.149004+0000","last_unstale":"2024-09-13T14:56:12.481484+0000","last_undegraded":"2024-09-13T14:56:12.481484+0000","last_fullsized":"2024-09-13T14:56:12.481484+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:46:54.393204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:46:54.393204+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:32:40.003989+0000","scrub_duration":0.015172223,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":313,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.525532+0000","last_change":"2024-09-13T14:55:17.935369+0000","last_active":"2024-09-13T14:56:15.525532+0000","last_peered":"2024-09-13T14:56:15.525532+0000","last_clean":"2024-09-13T14:56:15.525532+0000","last_became_active":"2024-09-13T14:55:17.930815+0000","last_became_peered":"2024-09-13T14:55:17.930815+0000","last_unstale":"2024-09-13T14:56:15.525532+0000","last_undegraded":"2024-09-13T14:56:15.525532+0000","last_fullsized":"2024-09-13T14:56:15.525532+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:59:31.247160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":442,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.510643+0000","last_change":"2024-09-13T14:55:17.932243+0000","last_active":"2024-09-13T14:56:15.510643+0000","last_peered":"2024-09-13T14:56:15.510643+0000","last_clean":"2024-09-13T14:56:15.510643+0000","last_became_active":"2024-09-13T14:55:17.931961+0000","last_became_peered":"2024-09-13T14:55:17.931961+0000","last_unstale":"2024-09-13T14:56:15.510643+0000","last_undegraded":"2024-09-13T14:56:15.510643+0000","last_fullsized":"2024-09-13T14:56:15.510643+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:23:52.680942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.0","version":"464'5744","reported_seq":7131,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:29.445858+0000","last_change":"2024-09-13T14:55:24.959843+0000","last_active":"2024-09-13T14:56:29.445858+0000","last_peered":"2024-09-13T14:56:29.445858+0000","last_clean":"2024-09-13T14:56:29.445858+0000","last_became_active":"2024-09-13T14:55:24.957491+0000","last_became_peered":"2024-09-13T14:55:24.957491+0000","last_unstale":"2024-09-13T14:56:29.445858+0000","last_undegraded":"2024-09-13T14:56:29.445858+0000","last_fullsized":"2024-09-13T14:56:29.445858+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":5744,"log_dups_size":0,"ondisk_log_size":5744,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:49:46.140123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376438784,"num_objects":5744,"num_object_clones":0,"num_object_copies":17232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5744,"num_write_kb":367616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":654,"num_bytes_recovered":42860544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"90'149","reported_seq":704,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.527540+0000","last_change":"2024-09-13T14:47:50.891055+0000","last_active":"2024-09-13T14:56:15.527540+0000","last_peered":"2024-09-13T14:56:15.527540+0000","last_clean":"2024-09-13T14:56:15.527540+0000","last_became_active":"2024-09-13T14:47:50.890368+0000","last_became_peered":"2024-09-13T14:47:50.890368+0000","last_unstale":"2024-09-13T14:56:15.527540+0000","last_undegraded":"2024-09-13T14:56:15.527540+0000","last_fullsized":"2024-09-13T14:56:15.527540+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:39.645908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:39.645908+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:29.800728+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":362,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.510208+0000","last_change":"2024-09-13T14:55:17.932199+0000","last_active":"2024-09-13T14:56:15.510208+0000","last_peered":"2024-09-13T14:56:15.510208+0000","last_clean":"2024-09-13T14:56:15.510208+0000","last_became_active":"2024-09-13T14:55:17.931873+0000","last_became_peered":"2024-09-13T14:55:17.931873+0000","last_unstale":"2024-09-13T14:56:15.510208+0000","last_undegraded":"2024-09-13T14:56:15.510208+0000","last_fullsized":"2024-09-13T14:56:15.510208+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:02:18.004113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":389,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.522584+0000","last_change":"2024-09-13T14:55:17.931269+0000","last_active":"2024-09-13T14:56:15.522584+0000","last_peered":"2024-09-13T14:56:15.522584+0000","last_clean":"2024-09-13T14:56:15.522584+0000","last_became_active":"2024-09-13T14:55:17.930909+0000","last_became_peered":"2024-09-13T14:55:17.930909+0000","last_unstale":"2024-09-13T14:56:15.522584+0000","last_undegraded":"2024-09-13T14:56:15.522584+0000","last_fullsized":"2024-09-13T14:56:15.522584+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:47.592526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"464'6162","reported_seq":7553,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:31.771192+0000","last_change":"2024-09-13T14:55:35.136618+0000","last_active":"2024-09-13T14:56:31.771192+0000","last_peered":"2024-09-13T14:56:31.771192+0000","last_clean":"2024-09-13T14:56:31.771192+0000","last_became_active":"2024-09-13T14:55:35.134043+0000","last_became_peered":"2024-09-13T14:55:35.134043+0000","last_unstale":"2024-09-13T14:56:31.771192+0000","last_undegraded":"2024-09-13T14:56:31.771192+0000","last_fullsized":"2024-09-13T14:56:31.771192+0000","mapping_epoch":457,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":6162,"log_dups_size":0,"ondisk_log_size":6162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:55.445002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403832832,"num_objects":6162,"num_object_clones":0,"num_object_copies":18486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6162,"num_write_kb":394368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":667,"num_bytes_recovered":43712512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"90'149","reported_seq":544,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-13T14:56:12.481838+0000","last_change":"2024-09-13T14:46:52.334715+0000","last_active":"2024-09-13T14:56:12.481838+0000","last_peered":"2024-09-13T14:56:12.481838+0000","last_clean":"2024-09-13T14:56:12.481838+0000","last_became_active":"2024-09-13T14:46:48.158685+0000","last_became_peered":"2024-09-13T14:46:48.158685+0000","last_unstale":"2024-09-13T14:56:12.481838+0000","last_undegraded":"2024-09-13T14:56:12.481838+0000","last_fullsized":"2024-09-13T14:56:12.481838+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:46:52.334301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:46:52.334301+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:46:03.579759+0000","scrub_duration":0.0050940330000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":841,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.520278+0000","last_change":"2024-09-13T14:47:47.861249+0000","last_active":"2024-09-13T14:56:15.520278+0000","last_peered":"2024-09-13T14:56:15.520278+0000","last_clean":"2024-09-13T14:56:15.520278+0000","last_became_active":"2024-09-13T14:47:47.860873+0000","last_became_peered":"2024-09-13T14:47:47.860873+0000","last_unstale":"2024-09-13T14:56:15.520278+0000","last_undegraded":"2024-09-13T14:56:15.520278+0000","last_fullsized":"2024-09-13T14:56:15.520278+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:37:50.709437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":276,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.520905+0000","last_change":"2024-09-13T14:55:17.933070+0000","last_active":"2024-09-13T14:56:15.520905+0000","last_peered":"2024-09-13T14:56:15.520905+0000","last_clean":"2024-09-13T14:56:15.520905+0000","last_became_active":"2024-09-13T14:55:17.931232+0000","last_became_peered":"2024-09-13T14:55:17.931232+0000","last_unstale":"2024-09-13T14:56:15.520905+0000","last_undegraded":"2024-09-13T14:56:15.520905+0000","last_fullsized":"2024-09-13T14:56:15.520905+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:22:46.719483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"464'5725","reported_seq":5792,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:29.450682+0000","last_change":"2024-09-13T14:54:58.384133+0000","last_active":"2024-09-13T14:56:29.450682+0000","last_peered":"2024-09-13T14:56:29.450682+0000","last_clean":"2024-09-13T14:56:29.450682+0000","last_became_active":"2024-09-13T14:54:58.383823+0000","last_became_peered":"2024-09-13T14:54:58.383823+0000","last_unstale":"2024-09-13T14:56:29.450682+0000","last_undegraded":"2024-09-13T14:56:29.450682+0000","last_fullsized":"2024-09-13T14:56:29.450682+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":5725,"log_dups_size":0,"ondisk_log_size":5725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:23.557883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":375193600,"num_objects":5725,"num_object_clones":0,"num_object_copies":17175,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5725,"num_write_kb":366400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":647,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-13T14:56:12.484951+0000","last_change":"2024-09-13T14:47:31.684319+0000","last_active":"2024-09-13T14:56:12.484951+0000","last_peered":"2024-09-13T14:56:12.484951+0000","last_clean":"2024-09-13T14:56:12.484951+0000","last_became_active":"2024-09-13T14:47:29.415658+0000","last_became_peered":"2024-09-13T14:47:29.415658+0000","last_unstale":"2024-09-13T14:56:12.484951+0000","last_undegraded":"2024-09-13T14:56:12.484951+0000","last_fullsized":"2024-09-13T14:56:12.484951+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:31.681433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:31.681433+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:29:34.663513+0000","scrub_duration":0.0059601139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":775,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-13T14:56:12.482809+0000","last_change":"2024-09-13T14:46:04.780718+0000","last_active":"2024-09-13T14:56:12.482809+0000","last_peered":"2024-09-13T14:56:12.482809+0000","last_clean":"2024-09-13T14:56:12.482809+0000","last_became_active":"2024-09-13T14:46:04.780120+0000","last_became_peered":"2024-09-13T14:46:04.780120+0000","last_unstale":"2024-09-13T14:56:12.482809+0000","last_undegraded":"2024-09-13T14:56:12.482809+0000","last_fullsized":"2024-09-13T14:56:12.482809+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:57:14.475888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":769,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.520819+0000","last_change":"2024-09-13T14:37:49.160719+0000","last_active":"2024-09-13T14:56:15.520819+0000","last_peered":"2024-09-13T14:56:15.520819+0000","last_clean":"2024-09-13T14:56:15.520819+0000","last_became_active":"2024-09-13T14:37:49.160401+0000","last_became_peered":"2024-09-13T14:37:49.160401+0000","last_unstale":"2024-09-13T14:56:15.520819+0000","last_undegraded":"2024-09-13T14:56:15.520819+0000","last_fullsized":"2024-09-13T14:56:15.520819+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:21.737945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":106,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-13T14:56:12.484492+0000","last_change":"2024-09-13T14:55:17.933431+0000","last_active":"2024-09-13T14:56:12.484492+0000","last_peered":"2024-09-13T14:56:12.484492+0000","last_clean":"2024-09-13T14:56:12.484492+0000","last_became_active":"2024-09-13T14:55:17.932711+0000","last_became_peered":"2024-09-13T14:55:17.932711+0000","last_unstale":"2024-09-13T14:56:12.484492+0000","last_undegraded":"2024-09-13T14:56:12.484492+0000","last_fullsized":"2024-09-13T14:56:12.484492+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:47.712045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"464'5777","reported_seq":5855,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:31.470700+0000","last_change":"2024-09-13T14:56:12.516271+0000","last_active":"2024-09-13T14:56:31.470700+0000","last_peered":"2024-09-13T14:56:31.470700+0000","last_clean":"2024-09-13T14:56:31.470700+0000","last_became_active":"2024-09-13T14:56:12.494537+0000","last_became_peered":"2024-09-13T14:56:12.494537+0000","last_unstale":"2024-09-13T14:56:31.470700+0000","last_undegraded":"2024-09-13T14:56:31.470700+0000","last_fullsized":"2024-09-13T14:56:31.470700+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":5777,"log_dups_size":0,"ondisk_log_size":5777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:41:12.203533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":378601472,"num_objects":5777,"num_object_clones":0,"num_object_copies":17331,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5777,"num_write_kb":369728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"90'149","reported_seq":709,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.527712+0000","last_change":"2024-09-13T14:55:20.148734+0000","last_active":"2024-09-13T14:56:15.527712+0000","last_peered":"2024-09-13T14:56:15.527712+0000","last_clean":"2024-09-13T14:56:15.527712+0000","last_became_active":"2024-09-13T14:55:20.147918+0000","last_became_peered":"2024-09-13T14:55:20.147918+0000","last_unstale":"2024-09-13T14:56:15.527712+0000","last_undegraded":"2024-09-13T14:56:15.527712+0000","last_fullsized":"2024-09-13T14:56:15.527712+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:34.603316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:34.603316+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:57.974517+0000","scrub_duration":0.0057131120000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":717,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.516660+0000","last_change":"2024-09-13T14:47:47.892973+0000","last_active":"2024-09-13T14:56:15.516660+0000","last_peered":"2024-09-13T14:56:15.516660+0000","last_clean":"2024-09-13T14:56:15.516660+0000","last_became_active":"2024-09-13T14:47:47.866986+0000","last_became_peered":"2024-09-13T14:47:47.866986+0000","last_unstale":"2024-09-13T14:56:15.516660+0000","last_undegraded":"2024-09-13T14:56:15.516660+0000","last_fullsized":"2024-09-13T14:56:15.516660+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:07.353290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":426,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.521673+0000","last_change":"2024-09-13T14:56:12.490871+0000","last_active":"2024-09-13T14:56:15.521673+0000","last_peered":"2024-09-13T14:56:15.521673+0000","last_clean":"2024-09-13T14:56:15.521673+0000","last_became_active":"2024-09-13T14:56:12.490535+0000","last_became_peered":"2024-09-13T14:56:12.490535+0000","last_unstale":"2024-09-13T14:56:15.521673+0000","last_undegraded":"2024-09-13T14:56:15.521673+0000","last_fullsized":"2024-09-13T14:56:15.521673+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:49.429179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":226,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.522345+0000","last_change":"2024-09-13T14:56:12.490229+0000","last_active":"2024-09-13T14:56:15.522345+0000","last_peered":"2024-09-13T14:56:15.522345+0000","last_clean":"2024-09-13T14:56:15.522345+0000","last_became_active":"2024-09-13T14:56:12.489512+0000","last_became_peered":"2024-09-13T14:56:12.489512+0000","last_unstale":"2024-09-13T14:56:15.522345+0000","last_undegraded":"2024-09-13T14:56:15.522345+0000","last_fullsized":"2024-09-13T14:56:15.522345+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:26:10.594549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.9","version":"464'6014","reported_seq":6051,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:31.690391+0000","last_change":"2024-09-13T14:54:57.844803+0000","last_active":"2024-09-13T14:56:31.690391+0000","last_peered":"2024-09-13T14:56:31.690391+0000","last_clean":"2024-09-13T14:56:31.690391+0000","last_became_active":"2024-09-13T14:54:57.844499+0000","last_became_peered":"2024-09-13T14:54:57.844499+0000","last_unstale":"2024-09-13T14:56:31.690391+0000","last_undegraded":"2024-09-13T14:56:31.690391+0000","last_fullsized":"2024-09-13T14:56:31.690391+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":6014,"log_dups_size":0,"ondisk_log_size":6014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:46.617173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394133504,"num_objects":6014,"num_object_clones":0,"num_object_copies":18042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6014,"num_write_kb":384896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"90'149","reported_seq":680,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-13T14:56:12.484063+0000","last_change":"2024-09-13T14:55:20.134578+0000","last_active":"2024-09-13T14:56:12.484063+0000","last_peered":"2024-09-13T14:56:12.484063+0000","last_clean":"2024-09-13T14:56:12.484063+0000","last_became_active":"2024-09-13T14:55:20.134120+0000","last_became_peered":"2024-09-13T14:55:20.134120+0000","last_unstale":"2024-09-13T14:56:12.484063+0000","last_undegraded":"2024-09-13T14:56:12.484063+0000","last_fullsized":"2024-09-13T14:56:12.484063+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:29.629342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:29.629342+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:38:13.399024+0000","scrub_duration":0.0099290990000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":753,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.510164+0000","last_change":"2024-09-13T14:46:04.782955+0000","last_active":"2024-09-13T14:56:15.510164+0000","last_peered":"2024-09-13T14:56:15.510164+0000","last_clean":"2024-09-13T14:56:15.510164+0000","last_became_active":"2024-09-13T14:46:04.782250+0000","last_became_peered":"2024-09-13T14:46:04.782250+0000","last_unstale":"2024-09-13T14:56:15.510164+0000","last_undegraded":"2024-09-13T14:56:15.510164+0000","last_fullsized":"2024-09-13T14:56:15.510164+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:24.596922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":270,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.521345+0000","last_change":"2024-09-13T14:47:47.866684+0000","last_active":"2024-09-13T14:56:15.521345+0000","last_peered":"2024-09-13T14:56:15.521345+0000","last_clean":"2024-09-13T14:56:15.521345+0000","last_became_active":"2024-09-13T14:47:47.866400+0000","last_became_peered":"2024-09-13T14:47:47.866400+0000","last_unstale":"2024-09-13T14:56:15.521345+0000","last_undegraded":"2024-09-13T14:56:15.521345+0000","last_fullsized":"2024-09-13T14:56:15.521345+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:08:34.289658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":270,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.522216+0000","last_change":"2024-09-13T14:47:47.870218+0000","last_active":"2024-09-13T14:56:15.522216+0000","last_peered":"2024-09-13T14:56:15.522216+0000","last_clean":"2024-09-13T14:56:15.522216+0000","last_became_active":"2024-09-13T14:47:47.869907+0000","last_became_peered":"2024-09-13T14:47:47.869907+0000","last_unstale":"2024-09-13T14:56:15.522216+0000","last_undegraded":"2024-09-13T14:56:15.522216+0000","last_fullsized":"2024-09-13T14:56:15.522216+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:41:49.141441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"464'6061","reported_seq":6133,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:31.709611+0000","last_change":"2024-09-13T14:56:12.497755+0000","last_active":"2024-09-13T14:56:31.709611+0000","last_peered":"2024-09-13T14:56:31.709611+0000","last_clean":"2024-09-13T14:56:31.709611+0000","last_became_active":"2024-09-13T14:56:12.489758+0000","last_became_peered":"2024-09-13T14:56:12.489758+0000","last_unstale":"2024-09-13T14:56:31.709611+0000","last_undegraded":"2024-09-13T14:56:31.709611+0000","last_fullsized":"2024-09-13T14:56:31.709611+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":6061,"log_dups_size":0,"ondisk_log_size":6061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:22:53.808045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":397213696,"num_objects":6061,"num_object_clones":0,"num_object_copies":18183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6061,"num_write_kb":387904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"90'149","reported_seq":565,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-13T14:56:12.481721+0000","last_change":"2024-09-13T14:47:32.275753+0000","last_active":"2024-09-13T14:56:12.481721+0000","last_peered":"2024-09-13T14:56:12.481721+0000","last_clean":"2024-09-13T14:56:12.481721+0000","last_became_active":"2024-09-13T14:47:29.401066+0000","last_became_peered":"2024-09-13T14:47:29.401066+0000","last_unstale":"2024-09-13T14:56:12.481721+0000","last_undegraded":"2024-09-13T14:56:12.481721+0000","last_fullsized":"2024-09-13T14:56:12.481721+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:32.275562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:32.275562+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:43:41.056775+0000","scrub_duration":0.0054674260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":1220,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-13T14:56:12.481943+0000","last_change":"2024-09-13T14:47:50.892842+0000","last_active":"2024-09-13T14:56:12.481943+0000","last_peered":"2024-09-13T14:56:12.481943+0000","last_clean":"2024-09-13T14:56:12.481943+0000","last_became_active":"2024-09-13T14:47:50.892237+0000","last_became_peered":"2024-09-13T14:47:50.892237+0000","last_unstale":"2024-09-13T14:56:12.481943+0000","last_undegraded":"2024-09-13T14:56:12.481943+0000","last_fullsized":"2024-09-13T14:56:12.481943+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:20:05.796762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":8388646,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":256,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.527157+0000","last_change":"2024-09-13T14:56:12.491483+0000","last_active":"2024-09-13T14:56:15.527157+0000","last_peered":"2024-09-13T14:56:15.527157+0000","last_clean":"2024-09-13T14:56:15.527157+0000","last_became_active":"2024-09-13T14:56:12.491191+0000","last_became_peered":"2024-09-13T14:56:12.491191+0000","last_unstale":"2024-09-13T14:56:15.527157+0000","last_undegraded":"2024-09-13T14:56:15.527157+0000","last_fullsized":"2024-09-13T14:56:15.527157+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:54:11.122787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":270,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.524692+0000","last_change":"2024-09-13T14:47:47.877272+0000","last_active":"2024-09-13T14:56:15.524692+0000","last_peered":"2024-09-13T14:56:15.524692+0000","last_clean":"2024-09-13T14:56:15.524692+0000","last_became_active":"2024-09-13T14:47:47.877003+0000","last_became_peered":"2024-09-13T14:47:47.877003+0000","last_unstale":"2024-09-13T14:56:15.524692+0000","last_undegraded":"2024-09-13T14:56:15.524692+0000","last_fullsized":"2024-09-13T14:56:15.524692+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:47:23.172452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.c","version":"464'5957","reported_seq":7277,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:31.749704+0000","last_change":"2024-09-13T14:55:28.486018+0000","last_active":"2024-09-13T14:56:31.749704+0000","last_peered":"2024-09-13T14:56:31.749704+0000","last_clean":"2024-09-13T14:56:31.749704+0000","last_became_active":"2024-09-13T14:55:28.483879+0000","last_became_peered":"2024-09-13T14:55:28.483879+0000","last_unstale":"2024-09-13T14:56:31.749704+0000","last_undegraded":"2024-09-13T14:56:31.749704+0000","last_fullsized":"2024-09-13T14:56:31.749704+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":5957,"log_dups_size":0,"ondisk_log_size":5957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:45:38.042233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390397952,"num_objects":5957,"num_object_clones":0,"num_object_copies":17871,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5957,"num_write_kb":381248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":632,"num_bytes_recovered":41418752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"90'149","reported_seq":702,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.519579+0000","last_change":"2024-09-13T14:47:33.067298+0000","last_active":"2024-09-13T14:56:15.519579+0000","last_peered":"2024-09-13T14:56:15.519579+0000","last_clean":"2024-09-13T14:56:15.519579+0000","last_became_active":"2024-09-13T14:47:29.419416+0000","last_became_peered":"2024-09-13T14:47:29.419416+0000","last_unstale":"2024-09-13T14:56:15.519579+0000","last_undegraded":"2024-09-13T14:56:15.519579+0000","last_fullsized":"2024-09-13T14:56:15.519579+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:33.066954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:33.066954+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:05:42.808060+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":289,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-13T14:56:12.481976+0000","last_change":"2024-09-13T14:46:01.763906+0000","last_active":"2024-09-13T14:56:12.481976+0000","last_peered":"2024-09-13T14:56:12.481976+0000","last_clean":"2024-09-13T14:56:12.481976+0000","last_became_active":"2024-09-13T14:46:01.763542+0000","last_became_peered":"2024-09-13T14:46:01.763542+0000","last_unstale":"2024-09-13T14:56:12.481976+0000","last_undegraded":"2024-09-13T14:56:12.481976+0000","last_fullsized":"2024-09-13T14:56:12.481976+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:09:36.652767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":1558,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.524774+0000","last_change":"2024-09-13T14:47:50.883365+0000","last_active":"2024-09-13T14:56:15.524774+0000","last_peered":"2024-09-13T14:56:15.524774+0000","last_clean":"2024-09-13T14:56:15.524774+0000","last_became_active":"2024-09-13T14:47:50.882609+0000","last_became_peered":"2024-09-13T14:47:50.882609+0000","last_unstale":"2024-09-13T14:56:15.524774+0000","last_undegraded":"2024-09-13T14:56:15.524774+0000","last_fullsized":"2024-09-13T14:56:15.524774+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:12:17.961575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":277,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.525844+0000","last_change":"2024-09-13T14:55:17.937000+0000","last_active":"2024-09-13T14:56:15.525844+0000","last_peered":"2024-09-13T14:56:15.525844+0000","last_clean":"2024-09-13T14:56:15.525844+0000","last_became_active":"2024-09-13T14:55:17.936557+0000","last_became_peered":"2024-09-13T14:55:17.936557+0000","last_unstale":"2024-09-13T14:56:15.525844+0000","last_undegraded":"2024-09-13T14:56:15.525844+0000","last_fullsized":"2024-09-13T14:56:15.525844+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:55:25.879096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"464'5944","reported_seq":7165,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:33.095588+0000","last_change":"2024-09-13T14:56:12.498406+0000","last_active":"2024-09-13T14:56:33.095588+0000","last_peered":"2024-09-13T14:56:33.095588+0000","last_clean":"2024-09-13T14:56:33.095588+0000","last_became_active":"2024-09-13T14:56:12.490622+0000","last_became_peered":"2024-09-13T14:56:12.490622+0000","last_unstale":"2024-09-13T14:56:33.095588+0000","last_undegraded":"2024-09-13T14:56:33.095588+0000","last_fullsized":"2024-09-13T14:56:33.095588+0000","mapping_epoch":462,"log_start":"439'100","ondisk_log_start":"439'100","created":436,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":5844,"log_dups_size":0,"ondisk_log_size":5844,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:19:06.141428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":389545984,"num_objects":5944,"num_object_clones":0,"num_object_copies":17832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5944,"num_write_kb":380416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":572,"num_bytes_recovered":37486592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"90'149","reported_seq":398,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.522603+0000","last_change":"2024-09-13T14:56:12.709187+0000","last_active":"2024-09-13T14:56:15.522603+0000","last_peered":"2024-09-13T14:56:15.522603+0000","last_clean":"2024-09-13T14:56:15.522603+0000","last_became_active":"2024-09-13T14:56:12.491639+0000","last_became_peered":"2024-09-13T14:56:12.491639+0000","last_unstale":"2024-09-13T14:56:15.522603+0000","last_undegraded":"2024-09-13T14:56:15.522603+0000","last_fullsized":"2024-09-13T14:56:15.522603+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:56:12.709114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:56:12.709114+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:39:58.475573+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":270,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.524861+0000","last_change":"2024-09-13T14:47:47.876562+0000","last_active":"2024-09-13T14:56:15.524861+0000","last_peered":"2024-09-13T14:56:15.524861+0000","last_clean":"2024-09-13T14:56:15.524861+0000","last_became_active":"2024-09-13T14:47:47.876297+0000","last_became_peered":"2024-09-13T14:47:47.876297+0000","last_unstale":"2024-09-13T14:56:15.524861+0000","last_undegraded":"2024-09-13T14:56:15.524861+0000","last_fullsized":"2024-09-13T14:56:15.524861+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:03:27.761394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":1029,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.517153+0000","last_change":"2024-09-13T14:55:20.147689+0000","last_active":"2024-09-13T14:56:15.517153+0000","last_peered":"2024-09-13T14:56:15.517153+0000","last_clean":"2024-09-13T14:56:15.517153+0000","last_became_active":"2024-09-13T14:55:20.147241+0000","last_became_peered":"2024-09-13T14:55:20.147241+0000","last_unstale":"2024-09-13T14:56:15.517153+0000","last_undegraded":"2024-09-13T14:56:15.517153+0000","last_fullsized":"2024-09-13T14:56:15.517153+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:43:36.224330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":320,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-13T14:56:12.491970+0000","last_change":"2024-09-13T14:56:12.491490+0000","last_active":"2024-09-13T14:56:12.491970+0000","last_peered":"2024-09-13T14:56:12.491970+0000","last_clean":"2024-09-13T14:56:12.491970+0000","last_became_active":"2024-09-13T14:56:12.490650+0000","last_became_peered":"2024-09-13T14:56:12.490650+0000","last_unstale":"2024-09-13T14:56:12.491970+0000","last_undegraded":"2024-09-13T14:56:12.491970+0000","last_fullsized":"2024-09-13T14:56:12.491970+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:55:33.313465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"464'5982","reported_seq":7372,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:31.453966+0000","last_change":"2024-09-13T14:55:23.955183+0000","last_active":"2024-09-13T14:56:31.453966+0000","last_peered":"2024-09-13T14:56:31.453966+0000","last_clean":"2024-09-13T14:56:31.453966+0000","last_became_active":"2024-09-13T14:55:23.952692+0000","last_became_peered":"2024-09-13T14:55:23.952692+0000","last_unstale":"2024-09-13T14:56:31.453966+0000","last_undegraded":"2024-09-13T14:56:31.453966+0000","last_fullsized":"2024-09-13T14:56:31.453966+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":5982,"log_dups_size":0,"ondisk_log_size":5982,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:56:02.111293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":392036352,"num_objects":5982,"num_object_clones":0,"num_object_copies":17946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5982,"num_write_kb":382848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":659,"num_bytes_recovered":43188224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"90'149","reported_seq":448,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.510510+0000","last_change":"2024-09-13T14:55:32.717170+0000","last_active":"2024-09-13T14:56:15.510510+0000","last_peered":"2024-09-13T14:56:15.510510+0000","last_clean":"2024-09-13T14:56:15.510510+0000","last_became_active":"2024-09-13T14:55:20.143587+0000","last_became_peered":"2024-09-13T14:55:20.143587+0000","last_unstale":"2024-09-13T14:56:15.510510+0000","last_undegraded":"2024-09-13T14:56:15.510510+0000","last_fullsized":"2024-09-13T14:56:15.510510+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:55:32.716911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:55:32.716911+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:01:10.628613+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":860,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.528132+0000","last_change":"2024-09-13T14:38:53.938227+0000","last_active":"2024-09-13T14:56:15.528132+0000","last_peered":"2024-09-13T14:56:15.528132+0000","last_clean":"2024-09-13T14:56:15.528132+0000","last_became_active":"2024-09-13T14:38:53.902867+0000","last_became_peered":"2024-09-13T14:38:53.902867+0000","last_unstale":"2024-09-13T14:56:15.528132+0000","last_undegraded":"2024-09-13T14:56:15.528132+0000","last_fullsized":"2024-09-13T14:56:15.528132+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:16.831756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":773,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.522715+0000","last_change":"2024-09-13T14:55:29.510841+0000","last_active":"2024-09-13T14:56:15.522715+0000","last_peered":"2024-09-13T14:56:15.522715+0000","last_clean":"2024-09-13T14:56:15.522715+0000","last_became_active":"2024-09-13T14:55:29.510252+0000","last_became_peered":"2024-09-13T14:55:29.510252+0000","last_unstale":"2024-09-13T14:56:15.522715+0000","last_undegraded":"2024-09-13T14:56:15.522715+0000","last_fullsized":"2024-09-13T14:56:15.522715+0000","mapping_epoch":452,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":453,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:14:27.135796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":842,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.519480+0000","last_change":"2024-09-13T14:38:53.900107+0000","last_active":"2024-09-13T14:56:15.519480+0000","last_peered":"2024-09-13T14:56:15.519480+0000","last_clean":"2024-09-13T14:56:15.519480+0000","last_became_active":"2024-09-13T14:38:53.899556+0000","last_became_peered":"2024-09-13T14:38:53.899556+0000","last_unstale":"2024-09-13T14:56:15.519480+0000","last_undegraded":"2024-09-13T14:56:15.519480+0000","last_fullsized":"2024-09-13T14:56:15.519480+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:15:47.918163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"464'5987","reported_seq":6054,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:33.303875+0000","last_change":"2024-09-13T14:54:58.382957+0000","last_active":"2024-09-13T14:56:33.303875+0000","last_peered":"2024-09-13T14:56:33.303875+0000","last_clean":"2024-09-13T14:56:33.303875+0000","last_became_active":"2024-09-13T14:54:58.382609+0000","last_became_peered":"2024-09-13T14:54:58.382609+0000","last_unstale":"2024-09-13T14:56:33.303875+0000","last_undegraded":"2024-09-13T14:56:33.303875+0000","last_fullsized":"2024-09-13T14:56:33.303875+0000","mapping_epoch":436,"log_start":"440'200","ondisk_log_start":"440'200","created":436,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":5787,"log_dups_size":0,"ondisk_log_size":5787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:10:56.760499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":392364032,"num_objects":5987,"num_object_clones":0,"num_object_copies":17961,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5987,"num_write_kb":383168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"90'149","reported_seq":699,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.525031+0000","last_change":"2024-09-13T14:47:29.999643+0000","last_active":"2024-09-13T14:56:15.525031+0000","last_peered":"2024-09-13T14:56:15.525031+0000","last_clean":"2024-09-13T14:56:15.525031+0000","last_became_active":"2024-09-13T14:47:29.410951+0000","last_became_peered":"2024-09-13T14:47:29.410951+0000","last_unstale":"2024-09-13T14:56:15.525031+0000","last_undegraded":"2024-09-13T14:56:15.525031+0000","last_fullsized":"2024-09-13T14:56:15.525031+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:29.999519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:29.999519+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:04:54.887884+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":227,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.522429+0000","last_change":"2024-09-13T14:56:12.493198+0000","last_active":"2024-09-13T14:56:15.522429+0000","last_peered":"2024-09-13T14:56:15.522429+0000","last_clean":"2024-09-13T14:56:15.522429+0000","last_became_active":"2024-09-13T14:56:12.490423+0000","last_became_peered":"2024-09-13T14:56:12.490423+0000","last_unstale":"2024-09-13T14:56:15.522429+0000","last_undegraded":"2024-09-13T14:56:15.522429+0000","last_fullsized":"2024-09-13T14:56:15.522429+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:29.984031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":270,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.517632+0000","last_change":"2024-09-13T14:46:01.781508+0000","last_active":"2024-09-13T14:56:15.517632+0000","last_peered":"2024-09-13T14:56:15.517632+0000","last_clean":"2024-09-13T14:56:15.517632+0000","last_became_active":"2024-09-13T14:46:01.767096+0000","last_became_peered":"2024-09-13T14:46:01.767096+0000","last_unstale":"2024-09-13T14:56:15.517632+0000","last_undegraded":"2024-09-13T14:56:15.517632+0000","last_fullsized":"2024-09-13T14:56:15.517632+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:20:01.119189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":253,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-13T14:56:12.481808+0000","last_change":"2024-09-13T14:55:17.932691+0000","last_active":"2024-09-13T14:56:12.481808+0000","last_peered":"2024-09-13T14:56:12.481808+0000","last_clean":"2024-09-13T14:56:12.481808+0000","last_became_active":"2024-09-13T14:55:17.932391+0000","last_became_peered":"2024-09-13T14:55:17.932391+0000","last_unstale":"2024-09-13T14:56:12.481808+0000","last_undegraded":"2024-09-13T14:56:12.481808+0000","last_fullsized":"2024-09-13T14:56:12.481808+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:03:22.734037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":408,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.527465+0000","last_change":"2024-09-13T14:55:17.930074+0000","last_active":"2024-09-13T14:56:15.527465+0000","last_peered":"2024-09-13T14:56:15.527465+0000","last_clean":"2024-09-13T14:56:15.527465+0000","last_became_active":"2024-09-13T14:55:17.928411+0000","last_became_peered":"2024-09-13T14:55:17.928411+0000","last_unstale":"2024-09-13T14:56:15.527465+0000","last_undegraded":"2024-09-13T14:56:15.527465+0000","last_fullsized":"2024-09-13T14:56:15.527465+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:11:35.302734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":100,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-13T14:56:12.484965+0000","last_change":"2024-09-13T14:55:17.939375+0000","last_active":"2024-09-13T14:56:12.484965+0000","last_peered":"2024-09-13T14:56:12.484965+0000","last_clean":"2024-09-13T14:56:12.484965+0000","last_became_active":"2024-09-13T14:55:17.926420+0000","last_became_peered":"2024-09-13T14:55:17.926420+0000","last_unstale":"2024-09-13T14:56:12.484965+0000","last_undegraded":"2024-09-13T14:56:12.484965+0000","last_fullsized":"2024-09-13T14:56:12.484965+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:38:13.401866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4756,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":269,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.525623+0000","last_change":"2024-09-13T14:47:47.867935+0000","last_active":"2024-09-13T14:56:15.525623+0000","last_peered":"2024-09-13T14:56:15.525623+0000","last_clean":"2024-09-13T14:56:15.525623+0000","last_became_active":"2024-09-13T14:47:47.867616+0000","last_became_peered":"2024-09-13T14:47:47.867616+0000","last_unstale":"2024-09-13T14:56:15.525623+0000","last_undegraded":"2024-09-13T14:56:15.525623+0000","last_fullsized":"2024-09-13T14:56:15.525623+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:52:38.619916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":249,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-13T14:56:12.484348+0000","last_change":"2024-09-13T14:55:17.951412+0000","last_active":"2024-09-13T14:56:12.484348+0000","last_peered":"2024-09-13T14:56:12.484348+0000","last_clean":"2024-09-13T14:56:12.484348+0000","last_became_active":"2024-09-13T14:55:17.937147+0000","last_became_peered":"2024-09-13T14:55:17.937147+0000","last_unstale":"2024-09-13T14:56:12.484348+0000","last_undegraded":"2024-09-13T14:56:12.484348+0000","last_fullsized":"2024-09-13T14:56:12.484348+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:43.786806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"90'149","reported_seq":712,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.510489+0000","last_change":"2024-09-13T14:55:20.149252+0000","last_active":"2024-09-13T14:56:15.510489+0000","last_peered":"2024-09-13T14:56:15.510489+0000","last_clean":"2024-09-13T14:56:15.510489+0000","last_became_active":"2024-09-13T14:55:20.148898+0000","last_became_peered":"2024-09-13T14:55:20.148898+0000","last_unstale":"2024-09-13T14:56:15.510489+0000","last_undegraded":"2024-09-13T14:56:15.510489+0000","last_fullsized":"2024-09-13T14:56:15.510489+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:31.013128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:31.013128+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:36:08.933590+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":295,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-13T14:56:12.482600+0000","last_change":"2024-09-13T14:47:47.861842+0000","last_active":"2024-09-13T14:56:12.482600+0000","last_peered":"2024-09-13T14:56:12.482600+0000","last_clean":"2024-09-13T14:56:12.482600+0000","last_became_active":"2024-09-13T14:47:47.861492+0000","last_became_peered":"2024-09-13T14:47:47.861492+0000","last_unstale":"2024-09-13T14:56:12.482600+0000","last_undegraded":"2024-09-13T14:56:12.482600+0000","last_fullsized":"2024-09-13T14:56:12.482600+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:17.279533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":783,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.521497+0000","last_change":"2024-09-13T14:38:53.899535+0000","last_active":"2024-09-13T14:56:15.521497+0000","last_peered":"2024-09-13T14:56:15.521497+0000","last_clean":"2024-09-13T14:56:15.521497+0000","last_became_active":"2024-09-13T14:38:53.899183+0000","last_became_peered":"2024-09-13T14:38:53.899183+0000","last_unstale":"2024-09-13T14:56:15.521497+0000","last_undegraded":"2024-09-13T14:56:15.521497+0000","last_fullsized":"2024-09-13T14:56:15.521497+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:33.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":344,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-13T14:56:12.492321+0000","last_change":"2024-09-13T14:56:12.488981+0000","last_active":"2024-09-13T14:56:12.492321+0000","last_peered":"2024-09-13T14:56:12.492321+0000","last_clean":"2024-09-13T14:56:12.492321+0000","last_became_active":"2024-09-13T14:56:12.488552+0000","last_became_peered":"2024-09-13T14:56:12.488552+0000","last_unstale":"2024-09-13T14:56:12.492321+0000","last_undegraded":"2024-09-13T14:56:12.492321+0000","last_fullsized":"2024-09-13T14:56:12.492321+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:02:23.732498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":819,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.519401+0000","last_change":"2024-09-13T14:47:47.868624+0000","last_active":"2024-09-13T14:56:15.519401+0000","last_peered":"2024-09-13T14:56:15.519401+0000","last_clean":"2024-09-13T14:56:15.519401+0000","last_became_active":"2024-09-13T14:47:47.868031+0000","last_became_peered":"2024-09-13T14:47:47.868031+0000","last_unstale":"2024-09-13T14:56:15.519401+0000","last_undegraded":"2024-09-13T14:56:15.519401+0000","last_fullsized":"2024-09-13T14:56:15.519401+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:05:25.365023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":308,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-13T14:56:12.483047+0000","last_change":"2024-09-13T14:47:47.859825+0000","last_active":"2024-09-13T14:56:12.483047+0000","last_peered":"2024-09-13T14:56:12.483047+0000","last_clean":"2024-09-13T14:56:12.483047+0000","last_became_active":"2024-09-13T14:47:47.859069+0000","last_became_peered":"2024-09-13T14:47:47.859069+0000","last_unstale":"2024-09-13T14:56:12.483047+0000","last_undegraded":"2024-09-13T14:56:12.483047+0000","last_fullsized":"2024-09-13T14:56:12.483047+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:18:36.452321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":851,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.523045+0000","last_change":"2024-09-13T14:47:47.868945+0000","last_active":"2024-09-13T14:56:15.523045+0000","last_peered":"2024-09-13T14:56:15.523045+0000","last_clean":"2024-09-13T14:56:15.523045+0000","last_became_active":"2024-09-13T14:47:47.868296+0000","last_became_peered":"2024-09-13T14:47:47.868296+0000","last_unstale":"2024-09-13T14:56:15.523045+0000","last_undegraded":"2024-09-13T14:56:15.523045+0000","last_fullsized":"2024-09-13T14:56:15.523045+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:22:24.778693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":40,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-13T14:56:12.491737+0000","last_change":"2024-09-13T14:56:12.491111+0000","last_active":"2024-09-13T14:56:12.491737+0000","last_peered":"2024-09-13T14:56:12.491737+0000","last_clean":"2024-09-13T14:56:12.491737+0000","last_became_active":"2024-09-13T14:56:12.490691+0000","last_became_peered":"2024-09-13T14:56:12.490691+0000","last_unstale":"2024-09-13T14:56:12.491737+0000","last_undegraded":"2024-09-13T14:56:12.491737+0000","last_fullsized":"2024-09-13T14:56:12.491737+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:01.439884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":276,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.530306+0000","last_change":"2024-09-13T14:55:17.929446+0000","last_active":"2024-09-13T14:56:15.530306+0000","last_peered":"2024-09-13T14:56:15.530306+0000","last_clean":"2024-09-13T14:56:15.530306+0000","last_became_active":"2024-09-13T14:55:17.928598+0000","last_became_peered":"2024-09-13T14:55:17.928598+0000","last_unstale":"2024-09-13T14:56:15.530306+0000","last_undegraded":"2024-09-13T14:56:15.530306+0000","last_fullsized":"2024-09-13T14:56:15.530306+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:27.446138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":846,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.520099+0000","last_change":"2024-09-13T14:55:17.933267+0000","last_active":"2024-09-13T14:56:15.520099+0000","last_peered":"2024-09-13T14:56:15.520099+0000","last_clean":"2024-09-13T14:56:15.520099+0000","last_became_active":"2024-09-13T14:55:17.932890+0000","last_became_peered":"2024-09-13T14:55:17.932890+0000","last_unstale":"2024-09-13T14:56:15.520099+0000","last_undegraded":"2024-09-13T14:56:15.520099+0000","last_fullsized":"2024-09-13T14:56:15.520099+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:48:59.113533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":461,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.525421+0000","last_change":"2024-09-13T14:55:17.937105+0000","last_active":"2024-09-13T14:56:15.525421+0000","last_peered":"2024-09-13T14:56:15.525421+0000","last_clean":"2024-09-13T14:56:15.525421+0000","last_became_active":"2024-09-13T14:55:17.936621+0000","last_became_peered":"2024-09-13T14:55:17.936621+0000","last_unstale":"2024-09-13T14:56:15.525421+0000","last_undegraded":"2024-09-13T14:56:15.525421+0000","last_fullsized":"2024-09-13T14:56:15.525421+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:37:28.134499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":351,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.510742+0000","last_change":"2024-09-13T14:50:27.510133+0000","last_active":"2024-09-13T14:56:15.510742+0000","last_peered":"2024-09-13T14:56:15.510742+0000","last_clean":"2024-09-13T14:56:15.510742+0000","last_became_active":"2024-09-13T14:50:27.508774+0000","last_became_peered":"2024-09-13T14:50:27.508774+0000","last_unstale":"2024-09-13T14:56:15.510742+0000","last_undegraded":"2024-09-13T14:56:15.510742+0000","last_fullsized":"2024-09-13T14:56:15.510742+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:31:04.531746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":13,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-13T14:56:12.490361+0000","last_change":"2024-09-13T14:56:12.489663+0000","last_active":"2024-09-13T14:56:12.490361+0000","last_peered":"2024-09-13T14:56:12.490361+0000","last_clean":"2024-09-13T14:56:12.490361+0000","last_became_active":"2024-09-13T14:56:12.489317+0000","last_became_peered":"2024-09-13T14:56:12.489317+0000","last_unstale":"2024-09-13T14:56:12.490361+0000","last_undegraded":"2024-09-13T14:56:12.490361+0000","last_fullsized":"2024-09-13T14:56:12.490361+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:08:44.565862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":405,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.520160+0000","last_change":"2024-09-13T14:55:17.932405+0000","last_active":"2024-09-13T14:56:15.520160+0000","last_peered":"2024-09-13T14:56:15.520160+0000","last_clean":"2024-09-13T14:56:15.520160+0000","last_became_active":"2024-09-13T14:55:17.932058+0000","last_became_peered":"2024-09-13T14:55:17.932058+0000","last_unstale":"2024-09-13T14:56:15.520160+0000","last_undegraded":"2024-09-13T14:56:15.520160+0000","last_fullsized":"2024-09-13T14:56:15.520160+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:05:01.826514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":309,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.499750+0000","last_change":"2024-09-13T14:55:17.946147+0000","last_active":"2024-09-13T14:56:15.499750+0000","last_peered":"2024-09-13T14:56:15.499750+0000","last_clean":"2024-09-13T14:56:15.499750+0000","last_became_active":"2024-09-13T14:55:17.931009+0000","last_became_peered":"2024-09-13T14:55:17.931009+0000","last_unstale":"2024-09-13T14:56:15.499750+0000","last_undegraded":"2024-09-13T14:56:15.499750+0000","last_fullsized":"2024-09-13T14:56:15.499750+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:08:26.717928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":302,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.521024+0000","last_change":"2024-09-13T14:55:17.936194+0000","last_active":"2024-09-13T14:56:15.521024+0000","last_peered":"2024-09-13T14:56:15.521024+0000","last_clean":"2024-09-13T14:56:15.521024+0000","last_became_active":"2024-09-13T14:55:17.935832+0000","last_became_peered":"2024-09-13T14:55:17.935832+0000","last_unstale":"2024-09-13T14:56:15.521024+0000","last_undegraded":"2024-09-13T14:56:15.521024+0000","last_fullsized":"2024-09-13T14:56:15.521024+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:40:52.745439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":296,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.500066+0000","last_change":"2024-09-13T14:55:17.941129+0000","last_active":"2024-09-13T14:56:15.500066+0000","last_peered":"2024-09-13T14:56:15.500066+0000","last_clean":"2024-09-13T14:56:15.500066+0000","last_became_active":"2024-09-13T14:55:17.930922+0000","last_became_peered":"2024-09-13T14:55:17.930922+0000","last_unstale":"2024-09-13T14:56:15.500066+0000","last_undegraded":"2024-09-13T14:56:15.500066+0000","last_fullsized":"2024-09-13T14:56:15.500066+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:06:29.610560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":865,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.523150+0000","last_change":"2024-09-13T14:46:01.766580+0000","last_active":"2024-09-13T14:56:15.523150+0000","last_peered":"2024-09-13T14:56:15.523150+0000","last_clean":"2024-09-13T14:56:15.523150+0000","last_became_active":"2024-09-13T14:46:01.766257+0000","last_became_peered":"2024-09-13T14:46:01.766257+0000","last_unstale":"2024-09-13T14:56:15.523150+0000","last_undegraded":"2024-09-13T14:56:15.523150+0000","last_fullsized":"2024-09-13T14:56:15.523150+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:01.126109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":829,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.522937+0000","last_change":"2024-09-13T14:46:01.769647+0000","last_active":"2024-09-13T14:56:15.522937+0000","last_peered":"2024-09-13T14:56:15.522937+0000","last_clean":"2024-09-13T14:56:15.522937+0000","last_became_active":"2024-09-13T14:46:01.769228+0000","last_became_peered":"2024-09-13T14:46:01.769228+0000","last_unstale":"2024-09-13T14:56:15.522937+0000","last_undegraded":"2024-09-13T14:56:15.522937+0000","last_fullsized":"2024-09-13T14:56:15.522937+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:58.696024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":387,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-13T14:56:15.521261+0000","last_change":"2024-09-13T14:55:17.929205+0000","last_active":"2024-09-13T14:56:15.521261+0000","last_peered":"2024-09-13T14:56:15.521261+0000","last_clean":"2024-09-13T14:56:15.521261+0000","last_became_active":"2024-09-13T14:55:17.928883+0000","last_became_peered":"2024-09-13T14:55:17.928883+0000","last_unstale":"2024-09-13T14:56:15.521261+0000","last_undegraded":"2024-09-13T14:56:15.521261+0000","last_fullsized":"2024-09-13T14:56:15.521261+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:27:31.570502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":367,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-13T14:56:12.492416+0000","last_change":"2024-09-13T14:56:12.492223+0000","last_active":"2024-09-13T14:56:12.492416+0000","last_peered":"2024-09-13T14:56:12.492416+0000","last_clean":"2024-09-13T14:56:12.492416+0000","last_became_active":"2024-09-13T14:56:12.490578+0000","last_became_peered":"2024-09-13T14:56:12.490578+0000","last_unstale":"2024-09-13T14:56:12.492416+0000","last_undegraded":"2024-09-13T14:56:12.492416+0000","last_fullsized":"2024-09-13T14:56:12.492416+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:11:13.623995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":6202523648,"num_objects":94643,"num_object_clones":0,"num_object_copies":283929,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94643,"num_write_kb":6057152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4957,"num_bytes_recovered":324861952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18590466048,"data_stored":18590466048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94043,"ondisk_log_size":94043,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":6474,"num_keys_recovered":30,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":21564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25132,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":171,"num_bytes_recovered":8388646,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3129,"ondisk_log_size":3129,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234552,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2828676,"kb_used_data":1913212,"kb_used_omap":1,"kb_used_meta":915454,"kb_avail":90916476,"statfs":{"total":95995035648,"available":93098471424,"internally_reserved":0,"allocated":1959129088,"data_stored":1956746160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":937425355},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":239,"seq":1026497183970,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3252544,"kb_used_data":2626616,"kb_used_omap":8,"kb_used_meta":625911,"kb_avail":90492608,"statfs":{"total":95995035648,"available":92664430592,"internally_reserved":0,"allocated":2689654784,"data_stored":2687275917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8453,"internal_metadata":640933627},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757817,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3491384,"kb_used_data":2289776,"kb_used_omap":1,"kb_used_meta":1201598,"kb_avail":90253768,"statfs":{"total":95995035648,"available":92419858432,"internally_reserved":0,"allocated":2344730624,"data_stored":2342356227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1230436806},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":128849019461,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435644,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4156276,"kb_used_data":3055596,"kb_used_omap":5,"kb_used_meta":1100666,"kb_avail":89588876,"statfs":{"total":95995035648,"available":91739009024,"internally_reserved":0,"allocated":3128930304,"data_stored":3126559765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":1127082110},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":104,"seq":446676599172,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2672004,"kb_used_data":2248828,"kb_used_omap":7,"kb_used_meta":423160,"kb_avail":91073148,"statfs":{"total":95995035648,"available":93258903552,"internally_reserved":0,"allocated":2302799872,"data_stored":2300418231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8010,"internal_metadata":433316022},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":99,"seq":425201762698,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2700848,"kb_used_data":2231848,"kb_used_omap":7,"kb_used_meta":468984,"kb_avail":91044304,"statfs":{"total":95995035648,"available":93229367296,"internally_reserved":0,"allocated":2285412352,"data_stored":2283045043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7759,"internal_metadata":480240049},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926222,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4853240,"kb_used_data":3738672,"kb_used_omap":7,"kb_used_meta":1114552,"kb_avail":88891912,"statfs":{"total":95995035648,"available":91025317888,"internally_reserved":0,"allocated":3828400128,"data_stored":3826025553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":1141301544},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6860,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3861118976,"data_stored":3861118976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2277441536,"data_stored":2277441536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2316238848,"data_stored":2316238848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3131899904,"data_stored":3131899904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":524288,"data_stored":524288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2341797888,"data_stored":2341797888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2708209664,"data_stored":2708209664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1953234944,"data_stored":1953234944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:56:37.689 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_5 pgp_num 16 2024-09-13T14:56:37.722 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:37.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:37.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:37.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:37 smithi195 ceph-mon[104262]: pgmap v1553: 109 pgs: 109 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 56 MiB/s wr, 888 op/s 2024-09-13T14:56:37.761 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:37 smithi102 ceph-mon[110183]: pgmap v1553: 109 pgs: 109 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 56 MiB/s wr, 888 op/s 2024-09-13T14:56:37.761 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:37 smithi102 ceph-mon[115269]: pgmap v1553: 109 pgs: 109 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 56 MiB/s wr, 888 op/s 2024-09-13T14:56:37.824 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:37.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:37.876 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:37.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:37.977 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:38.015 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:38.032 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:38.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:38.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:38.042 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:56:38.042 INFO:teuthology.orchestra.run.smithi102.stdout: "size": 20, 2024-09-13T14:56:38.042 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 600, 2024-09-13T14:56:38.042 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [ 2024-09-13T14:56:38.042 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.042 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:133953 8.b 8:df7b1c1e:::benchmark_data_smithi102_313275_object3366:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-13T14:56:38.042 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:47:46.853533+0000", 2024-09-13T14:56:38.042 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 531.182328998, 2024-09-13T14:56:38.043 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.4499644360000001, 2024-09-13T14:56:38.043 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:56:38.043 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:56:38.043 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "reached pg", 2024-09-13T14:56:38.043 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:56:38.043 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:56:38.043 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:56:38.043 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 133953 2024-09-13T14:56:38.043 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.043 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:56:38.043 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.043 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:56:38.043 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.853533+0000", 2024-09-13T14:56:38.044 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:56:38.044 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.044 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.044 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:56:38.044 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.853533+0000", 2024-09-13T14:56:38.044 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:56:38.044 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.044 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.044 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:56:38.044 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.853534+0000", 2024-09-13T14:56:38.044 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.5689999999999999e-06 2024-09-13T14:56:38.044 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.044 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.044 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:56:38.045 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.853539+0000", 2024-09-13T14:56:38.045 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.5299999999999998e-06 2024-09-13T14:56:38.045 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.045 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.045 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:56:38.045 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.853553+0000", 2024-09-13T14:56:38.045 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4144e-05 2024-09-13T14:56:38.045 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.045 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.045 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:56:38.046 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.853561+0000", 2024-09-13T14:56:38.046 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.9759999999999995e-06 2024-09-13T14:56:38.046 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.046 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.046 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.046 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.853628+0000", 2024-09-13T14:56:38.046 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.7219999999999997e-05 2024-09-13T14:56:38.046 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.046 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.046 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:56:38.046 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.853747+0000", 2024-09-13T14:56:38.047 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00011941400000000001 2024-09-13T14:56:38.047 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.047 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.047 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 2,4", 2024-09-13T14:56:38.047 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.853867+0000", 2024-09-13T14:56:38.047 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000119722 2024-09-13T14:56:38.047 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.047 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.047 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:56:38.047 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.857965+0000", 2024-09-13T14:56:38.047 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0040982650000000002 2024-09-13T14:56:38.047 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.047 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.047 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.048 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.876714+0000", 2024-09-13T14:56:38.048 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.018748715999999999 2024-09-13T14:56:38.048 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.048 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.048 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:56:38.048 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.876739+0000", 2024-09-13T14:56:38.048 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.4368000000000001e-05 2024-09-13T14:56:38.048 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.048 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.048 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.048 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.857882+0000", 2024-09-13T14:56:38.048 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.981143667 2024-09-13T14:56:38.048 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.049 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.049 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:56:38.049 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.857895+0000", 2024-09-13T14:56:38.049 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2941e-05 2024-09-13T14:56:38.049 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.049 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.049 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.049 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.267370+0000", 2024-09-13T14:56:38.049 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.409474772 2024-09-13T14:56:38.049 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.049 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.049 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for missing object", 2024-09-13T14:56:38.049 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.267563+0000", 2024-09-13T14:56:38.049 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000192745 2024-09-13T14:56:38.050 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.050 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.050 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.050 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.303430+0000", 2024-09-13T14:56:38.050 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.035866902999999999 2024-09-13T14:56:38.050 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.050 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.050 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:56:38.050 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.303497+0000", 2024-09-13T14:56:38.050 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.7484000000000006e-05 2024-09-13T14:56:38.050 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:56:38.050 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:56:38.050 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:56:38.051 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.051 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.051 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:133954 8.f 8:f0515611:::benchmark_data_smithi102_313275_object3367:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-13T14:56:38.051 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:47:46.853764+0000", 2024-09-13T14:56:38.051 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 531.182097215, 2024-09-13T14:56:38.051 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.4361166660000002, 2024-09-13T14:56:38.051 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:56:38.051 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:56:38.051 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "reached pg", 2024-09-13T14:56:38.051 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:56:38.051 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:56:38.051 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:56:38.051 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 133954 2024-09-13T14:56:38.052 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.052 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:56:38.052 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.052 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:56:38.052 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.853764+0000", 2024-09-13T14:56:38.052 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:56:38.052 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.052 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.052 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:56:38.052 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.853764+0000", 2024-09-13T14:56:38.052 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:56:38.052 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.052 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.053 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:56:38.053 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.853765+0000", 2024-09-13T14:56:38.053 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1000000000000001e-06 2024-09-13T14:56:38.053 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.053 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.053 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:56:38.053 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.853771+0000", 2024-09-13T14:56:38.053 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.0499999999999999e-06 2024-09-13T14:56:38.053 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.053 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.053 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:56:38.053 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.853782+0000", 2024-09-13T14:56:38.053 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1902000000000001e-05 2024-09-13T14:56:38.053 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.054 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.054 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:56:38.054 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.853790+0000", 2024-09-13T14:56:38.054 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.3209999999999999e-06 2024-09-13T14:56:38.054 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.054 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.054 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.054 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.855981+0000", 2024-09-13T14:56:38.054 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0021912099999999999 2024-09-13T14:56:38.054 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.054 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.054 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:56:38.054 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.856123+0000", 2024-09-13T14:56:38.054 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000141629 2024-09-13T14:56:38.054 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.055 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.055 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 2,4", 2024-09-13T14:56:38.055 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.856267+0000", 2024-09-13T14:56:38.055 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00014467200000000001 2024-09-13T14:56:38.055 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.055 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.055 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:56:38.055 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.870325+0000", 2024-09-13T14:56:38.055 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.014058083000000001 2024-09-13T14:56:38.055 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.055 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.055 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.055 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.886947+0000", 2024-09-13T14:56:38.056 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.016621915000000001 2024-09-13T14:56:38.056 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.056 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.056 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:56:38.056 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.886960+0000", 2024-09-13T14:56:38.056 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3108999999999999e-05 2024-09-13T14:56:38.056 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.056 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.056 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.056 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.858729+0000", 2024-09-13T14:56:38.056 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.97176854300000004 2024-09-13T14:56:38.056 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.056 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.057 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:56:38.057 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.858739+0000", 2024-09-13T14:56:38.057 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0331e-05 2024-09-13T14:56:38.057 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.057 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.057 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.057 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.263570+0000", 2024-09-13T14:56:38.057 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4048303849999999 2024-09-13T14:56:38.057 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.057 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.057 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for missing object", 2024-09-13T14:56:38.057 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.263716+0000", 2024-09-13T14:56:38.057 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00014647799999999999 2024-09-13T14:56:38.058 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.058 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.058 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.058 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.289803+0000", 2024-09-13T14:56:38.058 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.02608655 2024-09-13T14:56:38.058 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.058 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.058 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:56:38.058 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.289881+0000", 2024-09-13T14:56:38.058 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.8387999999999996e-05 2024-09-13T14:56:38.058 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:56:38.058 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:56:38.059 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:56:38.059 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.059 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.059 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:133962 8.0 8:0dc20890:::benchmark_data_smithi102_313275_object3375:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-13T14:56:38.059 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:47:46.876677+0000", 2024-09-13T14:56:38.059 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 531.15918459299996, 2024-09-13T14:56:38.059 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.3932651090000001, 2024-09-13T14:56:38.059 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:56:38.059 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:56:38.059 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:56:38.059 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:56:38.059 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:56:38.059 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:56:38.059 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 133962 2024-09-13T14:56:38.060 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.060 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:56:38.060 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.060 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:56:38.060 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.876677+0000", 2024-09-13T14:56:38.060 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:56:38.060 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.060 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.060 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:56:38.060 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.876677+0000", 2024-09-13T14:56:38.060 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:56:38.060 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.061 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.061 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:56:38.061 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.876678+0000", 2024-09-13T14:56:38.061 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.364e-06 2024-09-13T14:56:38.061 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.061 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.061 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:56:38.061 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.876682+0000", 2024-09-13T14:56:38.061 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.3790000000000002e-06 2024-09-13T14:56:38.061 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.061 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.061 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:56:38.062 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.876690+0000", 2024-09-13T14:56:38.062 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.8199999999999997e-06 2024-09-13T14:56:38.062 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.062 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.062 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:56:38.062 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.876696+0000", 2024-09-13T14:56:38.062 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.8569999999999997e-06 2024-09-13T14:56:38.062 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.062 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.062 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.062 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.881714+0000", 2024-09-13T14:56:38.062 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0050180390000000002 2024-09-13T14:56:38.062 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.062 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.063 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:56:38.063 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.881726+0000", 2024-09-13T14:56:38.063 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1318000000000001e-05 2024-09-13T14:56:38.063 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.063 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.063 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.063 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.860611+0000", 2024-09-13T14:56:38.063 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.97888544700000002 2024-09-13T14:56:38.063 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.063 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.063 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:56:38.063 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.860621+0000", 2024-09-13T14:56:38.063 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.9429999999999995e-06 2024-09-13T14:56:38.063 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.064 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.064 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.064 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.248331+0000", 2024-09-13T14:56:38.064 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.387709729 2024-09-13T14:56:38.064 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.064 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.064 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:56:38.064 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.248463+0000", 2024-09-13T14:56:38.064 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000132435 2024-09-13T14:56:38.064 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.064 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.064 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,4,5", 2024-09-13T14:56:38.064 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.248579+0000", 2024-09-13T14:56:38.065 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00011574999999999999 2024-09-13T14:56:38.065 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.065 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.065 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:56:38.065 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.255787+0000", 2024-09-13T14:56:38.065 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.007207761 2024-09-13T14:56:38.065 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.065 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.065 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.065 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.255832+0000", 2024-09-13T14:56:38.065 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.5062999999999999e-05 2024-09-13T14:56:38.065 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.065 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.066 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.066 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.255923+0000", 2024-09-13T14:56:38.066 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.1244999999999998e-05 2024-09-13T14:56:38.066 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.066 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.066 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.066 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.269728+0000", 2024-09-13T14:56:38.066 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.013804902000000001 2024-09-13T14:56:38.066 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.066 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.066 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:56:38.066 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.269831+0000", 2024-09-13T14:56:38.066 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000102543 2024-09-13T14:56:38.066 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.067 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.067 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:56:38.067 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.269942+0000", 2024-09-13T14:56:38.067 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000111514 2024-09-13T14:56:38.067 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:56:38.067 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:56:38.067 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:56:38.067 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.067 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.067 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:133964 8.b 8:d2fb3f4d:::benchmark_data_smithi102_313275_object3377:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-13T14:56:38.067 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:47:46.882828+0000", 2024-09-13T14:56:38.067 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 531.15303322299997, 2024-09-13T14:56:38.067 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.412555024, 2024-09-13T14:56:38.067 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:56:38.067 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:56:38.068 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:56:38.068 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:56:38.068 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:56:38.068 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:56:38.068 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 133964 2024-09-13T14:56:38.068 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.068 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:56:38.068 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.068 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:56:38.068 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.882828+0000", 2024-09-13T14:56:38.068 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:56:38.068 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.068 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.069 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:56:38.069 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.882828+0000", 2024-09-13T14:56:38.069 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:56:38.069 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.069 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.069 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:56:38.069 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.882830+0000", 2024-09-13T14:56:38.069 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.305e-06 2024-09-13T14:56:38.069 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.069 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.069 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:56:38.069 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.882835+0000", 2024-09-13T14:56:38.069 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.0139999999999998e-06 2024-09-13T14:56:38.069 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.070 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.070 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:56:38.070 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.882845+0000", 2024-09-13T14:56:38.070 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0066e-05 2024-09-13T14:56:38.070 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.070 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.070 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:56:38.070 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.882866+0000", 2024-09-13T14:56:38.070 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.1614e-05 2024-09-13T14:56:38.071 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.071 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.071 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.071 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.884959+0000", 2024-09-13T14:56:38.071 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0020924619999999998 2024-09-13T14:56:38.071 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.071 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.071 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:56:38.071 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.884968+0000", 2024-09-13T14:56:38.072 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.2280000000000004e-06 2024-09-13T14:56:38.072 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.073 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.073 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.073 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.857921+0000", 2024-09-13T14:56:38.073 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.97295337900000001 2024-09-13T14:56:38.073 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.073 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.073 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:56:38.073 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.857928+0000", 2024-09-13T14:56:38.073 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.0199999999999997e-06 2024-09-13T14:56:38.074 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.074 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.074 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.074 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.267894+0000", 2024-09-13T14:56:38.074 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4099659170000001 2024-09-13T14:56:38.074 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.074 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.074 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:56:38.074 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.267994+0000", 2024-09-13T14:56:38.074 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.9600999999999997e-05 2024-09-13T14:56:38.074 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.074 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.075 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,2,4", 2024-09-13T14:56:38.075 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.268120+0000", 2024-09-13T14:56:38.075 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000126246 2024-09-13T14:56:38.075 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.075 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.075 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.075 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.271322+0000", 2024-09-13T14:56:38.075 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0032020120000000002 2024-09-13T14:56:38.075 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.075 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.075 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:56:38.075 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.279444+0000", 2024-09-13T14:56:38.075 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.008121975 2024-09-13T14:56:38.076 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.076 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.076 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.076 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.279643+0000", 2024-09-13T14:56:38.076 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00019856600000000001 2024-09-13T14:56:38.076 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.076 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.076 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.076 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.295203+0000", 2024-09-13T14:56:38.076 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.015560305999999999 2024-09-13T14:56:38.076 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.076 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.076 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:56:38.076 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.295264+0000", 2024-09-13T14:56:38.077 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.1435999999999999e-05 2024-09-13T14:56:38.077 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.077 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.077 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:56:38.077 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.295383+0000", 2024-09-13T14:56:38.077 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00011887699999999999 2024-09-13T14:56:38.077 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:56:38.077 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:56:38.077 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:56:38.077 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.077 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.077 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:133978 8.b 8:d6e40fd1:::benchmark_data_smithi102_313275_object3391:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-13T14:56:38.078 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:47:46.901533+0000", 2024-09-13T14:56:38.078 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 531.13432833299998, 2024-09-13T14:56:38.078 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.3940416419999999, 2024-09-13T14:56:38.078 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:56:38.078 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:56:38.078 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:56:38.078 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:56:38.078 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:56:38.078 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:56:38.078 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 133978 2024-09-13T14:56:38.078 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.078 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:56:38.078 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.079 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:56:38.079 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.901533+0000", 2024-09-13T14:56:38.079 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:56:38.079 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.079 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.079 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:56:38.079 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.901533+0000", 2024-09-13T14:56:38.079 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:56:38.079 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.079 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.079 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:56:38.079 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.901535+0000", 2024-09-13T14:56:38.079 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.1909999999999999e-06 2024-09-13T14:56:38.079 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.080 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.080 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:56:38.080 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.901544+0000", 2024-09-13T14:56:38.080 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.5769999999999994e-06 2024-09-13T14:56:38.080 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.080 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.080 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:56:38.080 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.901567+0000", 2024-09-13T14:56:38.080 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.2957999999999999e-05 2024-09-13T14:56:38.080 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.080 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.080 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:56:38.080 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.901577+0000", 2024-09-13T14:56:38.080 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.6090000000000007e-06 2024-09-13T14:56:38.081 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.081 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.081 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.081 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.901648+0000", 2024-09-13T14:56:38.081 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.1589999999999997e-05 2024-09-13T14:56:38.081 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.081 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.081 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:56:38.081 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.901659+0000", 2024-09-13T14:56:38.081 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1075e-05 2024-09-13T14:56:38.081 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.081 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.082 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.082 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.857950+0000", 2024-09-13T14:56:38.082 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.95629086699999999 2024-09-13T14:56:38.082 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.082 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.082 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:56:38.082 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.857958+0000", 2024-09-13T14:56:38.082 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.5349999999999999e-06 2024-09-13T14:56:38.082 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.082 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.082 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.082 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.271350+0000", 2024-09-13T14:56:38.082 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.413392398 2024-09-13T14:56:38.082 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.083 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.083 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:56:38.083 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.271467+0000", 2024-09-13T14:56:38.083 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000117148 2024-09-13T14:56:38.083 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.083 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.083 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,2,4", 2024-09-13T14:56:38.083 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.271585+0000", 2024-09-13T14:56:38.083 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000117323 2024-09-13T14:56:38.083 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.083 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.083 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:56:38.083 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.279595+0000", 2024-09-13T14:56:38.083 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0080109780000000002 2024-09-13T14:56:38.084 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.084 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.084 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.084 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.279691+0000", 2024-09-13T14:56:38.084 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.5951000000000003e-05 2024-09-13T14:56:38.084 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.084 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.084 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.084 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.287370+0000", 2024-09-13T14:56:38.084 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0076781009999999997 2024-09-13T14:56:38.084 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.084 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.084 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.085 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.295424+0000", 2024-09-13T14:56:38.085 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0080549490000000005 2024-09-13T14:56:38.085 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.085 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.085 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:56:38.085 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.295468+0000", 2024-09-13T14:56:38.085 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.3951e-05 2024-09-13T14:56:38.085 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.085 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.085 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:56:38.085 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.295575+0000", 2024-09-13T14:56:38.085 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00010644100000000001 2024-09-13T14:56:38.085 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:56:38.085 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:56:38.086 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:56:38.086 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.086 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.086 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:133980 8.f 8:f0429519:::benchmark_data_smithi102_313275_object3393:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-13T14:56:38.086 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:47:46.903410+0000", 2024-09-13T14:56:38.086 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 531.13245114999995, 2024-09-13T14:56:38.086 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.3834898679999998, 2024-09-13T14:56:38.086 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:56:38.086 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:56:38.086 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:56:38.086 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:56:38.086 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:56:38.086 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:56:38.086 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 133980 2024-09-13T14:56:38.087 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.087 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:56:38.087 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.087 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:56:38.087 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.903410+0000", 2024-09-13T14:56:38.087 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:56:38.087 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.087 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.087 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:56:38.087 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.903410+0000", 2024-09-13T14:56:38.087 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:56:38.087 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.087 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.087 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:56:38.088 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.903412+0000", 2024-09-13T14:56:38.088 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.722e-06 2024-09-13T14:56:38.088 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.088 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.088 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:56:38.088 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.903419+0000", 2024-09-13T14:56:38.088 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.1590000000000004e-06 2024-09-13T14:56:38.088 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.088 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.088 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:56:38.088 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.903438+0000", 2024-09-13T14:56:38.088 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.8751000000000001e-05 2024-09-13T14:56:38.089 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.089 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.089 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:56:38.089 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.903447+0000", 2024-09-13T14:56:38.089 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.4369999999999996e-06 2024-09-13T14:56:38.089 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.089 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.089 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.089 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.903523+0000", 2024-09-13T14:56:38.089 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.5556999999999995e-05 2024-09-13T14:56:38.089 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.089 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.089 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:56:38.090 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.903534+0000", 2024-09-13T14:56:38.090 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1046000000000001e-05 2024-09-13T14:56:38.090 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.090 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.090 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.090 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859417+0000", 2024-09-13T14:56:38.090 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.95588269000000003 2024-09-13T14:56:38.090 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.090 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.090 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:56:38.090 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.859425+0000", 2024-09-13T14:56:38.090 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.6910000000000003e-06 2024-09-13T14:56:38.091 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.091 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.091 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.091 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.263904+0000", 2024-09-13T14:56:38.091 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4044789719999999 2024-09-13T14:56:38.091 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.091 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.091 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:56:38.091 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.264020+0000", 2024-09-13T14:56:38.091 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00011522 2024-09-13T14:56:38.091 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.091 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.091 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,2,4", 2024-09-13T14:56:38.091 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.264139+0000", 2024-09-13T14:56:38.092 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000119253 2024-09-13T14:56:38.092 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.092 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.092 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.092 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.267982+0000", 2024-09-13T14:56:38.092 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0038432119999999999 2024-09-13T14:56:38.092 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.092 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.092 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:56:38.092 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.268187+0000", 2024-09-13T14:56:38.093 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000204818 2024-09-13T14:56:38.093 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.093 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.093 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.093 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.270572+0000", 2024-09-13T14:56:38.093 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0023851269999999999 2024-09-13T14:56:38.093 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.093 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.093 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.093 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.286685+0000", 2024-09-13T14:56:38.093 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.016113245000000002 2024-09-13T14:56:38.093 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.093 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.094 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:56:38.094 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.286749+0000", 2024-09-13T14:56:38.094 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.3503000000000005e-05 2024-09-13T14:56:38.094 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.094 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.094 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:56:38.094 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.286900+0000", 2024-09-13T14:56:38.094 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00015146499999999999 2024-09-13T14:56:38.094 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:56:38.094 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:56:38.094 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:56:38.094 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.094 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.094 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:133983 8.b 8:dcf739ee:::benchmark_data_smithi102_313275_object3396:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-13T14:56:38.095 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:47:46.909053+0000", 2024-09-13T14:56:38.095 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 531.126808697, 2024-09-13T14:56:38.095 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.3940629379999998, 2024-09-13T14:56:38.095 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:56:38.095 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:56:38.095 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:56:38.095 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:56:38.095 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:56:38.095 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:56:38.095 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 133983 2024-09-13T14:56:38.095 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.095 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:56:38.096 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.096 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:56:38.096 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.909053+0000", 2024-09-13T14:56:38.096 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:56:38.096 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.096 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.096 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:56:38.096 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.909053+0000", 2024-09-13T14:56:38.096 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:56:38.096 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.096 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.096 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:56:38.096 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.909055+0000", 2024-09-13T14:56:38.097 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.795e-06 2024-09-13T14:56:38.097 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.097 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.097 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:56:38.097 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.909061+0000", 2024-09-13T14:56:38.097 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.1210000000000003e-06 2024-09-13T14:56:38.097 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.097 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.097 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:56:38.097 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.909077+0000", 2024-09-13T14:56:38.097 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.6328999999999999e-05 2024-09-13T14:56:38.097 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.098 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.098 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:56:38.098 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.909084+0000", 2024-09-13T14:56:38.098 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.9290000000000001e-06 2024-09-13T14:56:38.098 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.098 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.098 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.098 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.909154+0000", 2024-09-13T14:56:38.098 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.0185999999999995e-05 2024-09-13T14:56:38.098 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.098 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.098 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:56:38.098 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.909168+0000", 2024-09-13T14:56:38.098 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3423e-05 2024-09-13T14:56:38.099 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.099 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.099 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.099 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.857980+0000", 2024-09-13T14:56:38.099 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.94881206799999995 2024-09-13T14:56:38.099 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.099 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.099 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:56:38.099 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.857987+0000", 2024-09-13T14:56:38.099 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.6539999999999999e-06 2024-09-13T14:56:38.099 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.099 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.100 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.100 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.275587+0000", 2024-09-13T14:56:38.100 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4175995450000001 2024-09-13T14:56:38.100 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.100 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.100 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:56:38.100 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.275781+0000", 2024-09-13T14:56:38.100 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00019366199999999999 2024-09-13T14:56:38.100 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.100 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.100 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,2,4", 2024-09-13T14:56:38.100 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.276020+0000", 2024-09-13T14:56:38.100 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000238934 2024-09-13T14:56:38.101 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.101 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.101 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:56:38.101 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.279601+0000", 2024-09-13T14:56:38.101 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0035817750000000001 2024-09-13T14:56:38.101 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.101 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.101 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.101 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.283529+0000", 2024-09-13T14:56:38.101 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0039279320000000003 2024-09-13T14:56:38.101 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.101 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.101 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.101 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.296300+0000", 2024-09-13T14:56:38.101 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.012770399999999999 2024-09-13T14:56:38.102 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.102 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.102 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.102 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.302946+0000", 2024-09-13T14:56:38.102 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0066464829999999999 2024-09-13T14:56:38.102 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.102 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.102 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:56:38.103 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.302999+0000", 2024-09-13T14:56:38.103 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.2719000000000001e-05 2024-09-13T14:56:38.103 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.103 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.103 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:56:38.103 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.303116+0000", 2024-09-13T14:56:38.103 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000116983 2024-09-13T14:56:38.103 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:56:38.103 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:56:38.103 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:56:38.103 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.103 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.103 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:133984 8.0 8:09c1f50d:::benchmark_data_smithi102_313275_object3397:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-13T14:56:38.104 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:47:46.913150+0000", 2024-09-13T14:56:38.104 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 531.12271183899998, 2024-09-13T14:56:38.104 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.3576139669999998, 2024-09-13T14:56:38.104 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:56:38.104 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:56:38.104 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:56:38.104 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:56:38.104 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:56:38.104 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:56:38.104 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 133984 2024-09-13T14:56:38.104 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.104 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:56:38.105 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.105 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:56:38.105 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.913150+0000", 2024-09-13T14:56:38.105 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:56:38.105 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.105 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.105 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:56:38.105 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.913150+0000", 2024-09-13T14:56:38.105 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:56:38.105 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.105 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.105 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:56:38.105 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.913151+0000", 2024-09-13T14:56:38.105 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.3200000000000003e-07 2024-09-13T14:56:38.106 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.106 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.106 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:56:38.106 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.913155+0000", 2024-09-13T14:56:38.106 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.3370000000000001e-06 2024-09-13T14:56:38.106 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.106 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.106 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:56:38.106 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.913166+0000", 2024-09-13T14:56:38.106 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1399999999999999e-05 2024-09-13T14:56:38.106 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.106 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.106 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:56:38.106 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.913172+0000", 2024-09-13T14:56:38.107 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.9109999999999998e-06 2024-09-13T14:56:38.107 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.107 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.107 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.107 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.913224+0000", 2024-09-13T14:56:38.107 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.1277999999999999e-05 2024-09-13T14:56:38.107 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.107 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.107 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:56:38.107 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:46.913230+0000", 2024-09-13T14:56:38.107 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.5529999999999998e-06 2024-09-13T14:56:38.107 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.108 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.108 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.108 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.860646+0000", 2024-09-13T14:56:38.108 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.94741563299999998 2024-09-13T14:56:38.108 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.108 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.108 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:56:38.108 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:47.860653+0000", 2024-09-13T14:56:38.108 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.2150000000000004e-06 2024-09-13T14:56:38.108 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.108 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.108 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.108 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.252136+0000", 2024-09-13T14:56:38.109 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.39148323 2024-09-13T14:56:38.109 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.109 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.109 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:56:38.109 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.252259+0000", 2024-09-13T14:56:38.109 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00012318999999999999 2024-09-13T14:56:38.109 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.109 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.109 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,4,5", 2024-09-13T14:56:38.109 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.252375+0000", 2024-09-13T14:56:38.109 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000115986 2024-09-13T14:56:38.109 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.109 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.109 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.109 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.255884+0000", 2024-09-13T14:56:38.110 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.003508653 2024-09-13T14:56:38.110 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.110 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.110 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:56:38.110 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.255936+0000", 2024-09-13T14:56:38.110 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.2435999999999997e-05 2024-09-13T14:56:38.110 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.110 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.110 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.110 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.256005+0000", 2024-09-13T14:56:38.110 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.8221999999999998e-05 2024-09-13T14:56:38.110 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.110 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.110 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.111 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.270574+0000", 2024-09-13T14:56:38.111 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.014568965999999999 2024-09-13T14:56:38.111 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.111 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.111 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:56:38.111 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.270641+0000", 2024-09-13T14:56:38.111 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.7502999999999994e-05 2024-09-13T14:56:38.111 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.111 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.111 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:56:38.111 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:47:49.270764+0000", 2024-09-13T14:56:38.111 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00012252199999999999 2024-09-13T14:56:38.111 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:56:38.111 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:56:38.111 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:56:38.112 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.112 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.112 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:249803 8.f 8:f189cad3:::benchmark_data_smithi102_313275_object119216:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-09-13T14:56:38.112 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:50:13.153033+0000", 2024-09-13T14:56:38.112 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 384.88282857000002, 2024-09-13T14:56:38.112 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4494403890000001, 2024-09-13T14:56:38.112 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:56:38.112 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:56:38.112 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "reached pg", 2024-09-13T14:56:38.112 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:56:38.112 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:56:38.112 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:56:38.112 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 249803 2024-09-13T14:56:38.112 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.113 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:56:38.113 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.113 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:56:38.113 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.153033+0000", 2024-09-13T14:56:38.113 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:56:38.113 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.113 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.113 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:56:38.113 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.153033+0000", 2024-09-13T14:56:38.113 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:56:38.113 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.113 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.113 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:56:38.113 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.153034+0000", 2024-09-13T14:56:38.114 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3030000000000001e-06 2024-09-13T14:56:38.114 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.114 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.114 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:56:38.114 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.153038+0000", 2024-09-13T14:56:38.114 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.5880000000000002e-06 2024-09-13T14:56:38.114 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.114 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.114 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:56:38.114 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.153046+0000", 2024-09-13T14:56:38.114 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.5180000000000004e-06 2024-09-13T14:56:38.114 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.114 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.114 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:56:38.115 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.153052+0000", 2024-09-13T14:56:38.115 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.2800000000000003e-06 2024-09-13T14:56:38.115 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.115 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.115 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.115 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.154378+0000", 2024-09-13T14:56:38.115 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0013263299999999999 2024-09-13T14:56:38.115 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.115 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.115 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:56:38.115 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.154654+0000", 2024-09-13T14:56:38.115 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00027594000000000002 2024-09-13T14:56:38.115 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.116 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.116 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,2,4", 2024-09-13T14:56:38.116 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.155042+0000", 2024-09-13T14:56:38.116 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00038806700000000002 2024-09-13T14:56:38.116 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.116 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.116 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.116 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.170794+0000", 2024-09-13T14:56:38.117 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.015751654 2024-09-13T14:56:38.117 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.117 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.117 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.117 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.191965+0000", 2024-09-13T14:56:38.117 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.021171713000000002 2024-09-13T14:56:38.117 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.117 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.117 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.118 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.198860+0000", 2024-09-13T14:56:38.118 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0068942059999999999 2024-09-13T14:56:38.118 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.118 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.118 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:56:38.118 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.198873+0000", 2024-09-13T14:56:38.118 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3499000000000001e-05 2024-09-13T14:56:38.118 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.118 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.118 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.118 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.602392+0000", 2024-09-13T14:56:38.118 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4035188350000001 2024-09-13T14:56:38.118 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.118 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.119 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:56:38.119 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.602473+0000", 2024-09-13T14:56:38.119 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.1456000000000001e-05 2024-09-13T14:56:38.119 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:56:38.119 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:56:38.119 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:56:38.119 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.119 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.119 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:249810 8.f 8:f6698050:::benchmark_data_smithi102_313275_object119223:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-09-13T14:56:38.119 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:50:13.159508+0000", 2024-09-13T14:56:38.119 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 384.87635353899998, 2024-09-13T14:56:38.119 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4504869890000001, 2024-09-13T14:56:38.119 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:56:38.119 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:56:38.120 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "reached pg", 2024-09-13T14:56:38.120 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:56:38.120 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:56:38.120 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:56:38.120 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 249810 2024-09-13T14:56:38.120 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.120 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:56:38.120 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.120 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:56:38.120 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.159508+0000", 2024-09-13T14:56:38.120 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:56:38.120 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.120 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.120 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:56:38.121 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.159508+0000", 2024-09-13T14:56:38.121 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:56:38.121 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.121 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.121 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:56:38.121 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.159510+0000", 2024-09-13T14:56:38.121 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.6390000000000001e-06 2024-09-13T14:56:38.121 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.121 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.121 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:56:38.121 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.159515+0000", 2024-09-13T14:56:38.121 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.8860000000000003e-06 2024-09-13T14:56:38.121 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.121 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.122 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:56:38.122 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.159528+0000", 2024-09-13T14:56:38.122 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3200999999999999e-05 2024-09-13T14:56:38.122 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.122 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.122 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:56:38.122 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.159535+0000", 2024-09-13T14:56:38.122 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.5290000000000003e-06 2024-09-13T14:56:38.122 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.122 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.122 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.122 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.166935+0000", 2024-09-13T14:56:38.122 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.007399554 2024-09-13T14:56:38.122 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.123 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.123 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:56:38.123 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.167038+0000", 2024-09-13T14:56:38.123 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000102719 2024-09-13T14:56:38.123 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.123 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.123 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,2,4", 2024-09-13T14:56:38.123 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.167147+0000", 2024-09-13T14:56:38.123 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000109601 2024-09-13T14:56:38.123 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.123 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.123 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.123 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.204880+0000", 2024-09-13T14:56:38.124 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.037732849999999998 2024-09-13T14:56:38.124 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.124 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.124 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:56:38.124 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.204889+0000", 2024-09-13T14:56:38.124 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.0000000000000002e-06 2024-09-13T14:56:38.124 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.124 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.124 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.124 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.602703+0000", 2024-09-13T14:56:38.124 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3978135190000001 2024-09-13T14:56:38.124 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.124 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.124 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for missing object", 2024-09-13T14:56:38.125 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.602771+0000", 2024-09-13T14:56:38.125 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.8176999999999995e-05 2024-09-13T14:56:38.125 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.125 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.125 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.125 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.609884+0000", 2024-09-13T14:56:38.125 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0071134850000000001 2024-09-13T14:56:38.125 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.125 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.125 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:56:38.125 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.609995+0000", 2024-09-13T14:56:38.125 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00011082900000000001 2024-09-13T14:56:38.125 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:56:38.125 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:56:38.126 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:56:38.126 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.126 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.126 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:249881 8.f 8:f3194c62:::benchmark_data_smithi102_313275_object119294:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-13T14:56:38.126 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:50:13.261774+0000", 2024-09-13T14:56:38.126 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 384.77408793799998, 2024-09-13T14:56:38.126 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3524746409999999, 2024-09-13T14:56:38.126 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:56:38.126 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:56:38.126 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:56:38.126 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:56:38.126 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:56:38.127 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:56:38.127 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 249881 2024-09-13T14:56:38.127 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.127 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:56:38.127 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.127 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:56:38.127 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.261774+0000", 2024-09-13T14:56:38.127 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:56:38.127 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.127 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.127 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:56:38.127 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.261774+0000", 2024-09-13T14:56:38.127 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:56:38.127 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.128 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.128 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:56:38.128 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.261775+0000", 2024-09-13T14:56:38.128 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.708e-06 2024-09-13T14:56:38.128 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.128 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.128 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:56:38.128 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.261780+0000", 2024-09-13T14:56:38.128 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.5959999999999997e-06 2024-09-13T14:56:38.128 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.128 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.128 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:56:38.128 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.261796+0000", 2024-09-13T14:56:38.128 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.6143000000000001e-05 2024-09-13T14:56:38.129 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.129 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.129 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:56:38.129 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.261807+0000", 2024-09-13T14:56:38.129 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1022000000000001e-05 2024-09-13T14:56:38.129 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.129 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.129 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.129 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.262190+0000", 2024-09-13T14:56:38.129 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000383032 2024-09-13T14:56:38.129 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.129 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.129 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:56:38.129 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.262238+0000", 2024-09-13T14:56:38.130 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.8004000000000003e-05 2024-09-13T14:56:38.130 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.130 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.130 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.130 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.602809+0000", 2024-09-13T14:56:38.130 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3405708009999999 2024-09-13T14:56:38.130 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.130 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.130 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:56:38.130 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.602925+0000", 2024-09-13T14:56:38.130 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000116081 2024-09-13T14:56:38.130 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.130 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.130 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:56:38.130 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.603013+0000", 2024-09-13T14:56:38.131 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.8063e-05 2024-09-13T14:56:38.131 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.131 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.131 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.131 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.609463+0000", 2024-09-13T14:56:38.131 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0064497260000000002 2024-09-13T14:56:38.131 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.131 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.131 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:56:38.131 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.609994+0000", 2024-09-13T14:56:38.131 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00053112500000000002 2024-09-13T14:56:38.131 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.131 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.132 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.132 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.614049+0000", 2024-09-13T14:56:38.132 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0040555239999999996 2024-09-13T14:56:38.132 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.132 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.132 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:56:38.132 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.614151+0000", 2024-09-13T14:56:38.132 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00010197800000000001 2024-09-13T14:56:38.132 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.133 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.133 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:56:38.133 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.614248+0000", 2024-09-13T14:56:38.133 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.6837999999999997e-05 2024-09-13T14:56:38.133 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:56:38.133 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:56:38.133 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:56:38.133 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.133 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.133 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:249908 8.f 8:f0b20de0:::benchmark_data_smithi102_313275_object119321:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-13T14:56:38.133 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:50:13.297325+0000", 2024-09-13T14:56:38.133 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 384.73853679899997, 2024-09-13T14:56:38.133 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3251422660000001, 2024-09-13T14:56:38.133 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:56:38.134 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:56:38.134 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:56:38.134 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:56:38.134 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:56:38.134 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:56:38.134 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 249908 2024-09-13T14:56:38.134 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.134 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:56:38.134 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.134 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:56:38.134 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.297325+0000", 2024-09-13T14:56:38.134 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:56:38.135 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.135 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.135 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:56:38.135 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.297325+0000", 2024-09-13T14:56:38.135 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:56:38.135 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.135 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.135 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:56:38.135 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.297326+0000", 2024-09-13T14:56:38.135 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.697e-06 2024-09-13T14:56:38.135 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.135 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.135 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:56:38.135 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.297333+0000", 2024-09-13T14:56:38.136 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.1410000000000002e-06 2024-09-13T14:56:38.136 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.136 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.136 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:56:38.136 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.297351+0000", 2024-09-13T14:56:38.136 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.8131999999999999e-05 2024-09-13T14:56:38.137 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.137 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.137 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:56:38.137 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.297359+0000", 2024-09-13T14:56:38.137 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.2849999999999995e-06 2024-09-13T14:56:38.137 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.137 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.137 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.137 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.297616+0000", 2024-09-13T14:56:38.137 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00025743000000000001 2024-09-13T14:56:38.137 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.137 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.137 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:56:38.137 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.297627+0000", 2024-09-13T14:56:38.138 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0546e-05 2024-09-13T14:56:38.138 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.138 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.138 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.138 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.606160+0000", 2024-09-13T14:56:38.138 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3085327330000001 2024-09-13T14:56:38.138 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.138 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.138 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:56:38.138 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.606273+0000", 2024-09-13T14:56:38.138 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00011299200000000001 2024-09-13T14:56:38.138 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.138 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.138 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:56:38.139 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.606350+0000", 2024-09-13T14:56:38.139 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.7202999999999999e-05 2024-09-13T14:56:38.139 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.139 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.139 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:56:38.139 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.613913+0000", 2024-09-13T14:56:38.139 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0075633840000000003 2024-09-13T14:56:38.139 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.139 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.139 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.139 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.622195+0000", 2024-09-13T14:56:38.139 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0082820189999999998 2024-09-13T14:56:38.139 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.139 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.140 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.140 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.622300+0000", 2024-09-13T14:56:38.140 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000104709 2024-09-13T14:56:38.140 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.140 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.140 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:56:38.140 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.622356+0000", 2024-09-13T14:56:38.140 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.6020999999999998e-05 2024-09-13T14:56:38.140 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.140 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.140 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:56:38.140 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.622467+0000", 2024-09-13T14:56:38.141 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000110974 2024-09-13T14:56:38.141 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:56:38.141 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:56:38.141 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:56:38.141 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.141 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.141 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:249937 8.f 8:fe425234:::benchmark_data_smithi102_313275_object119350:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-13T14:56:38.141 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:50:13.331885+0000", 2024-09-13T14:56:38.141 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 384.703976533, 2024-09-13T14:56:38.141 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.294601433, 2024-09-13T14:56:38.141 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:56:38.141 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:56:38.141 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:56:38.141 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:56:38.142 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:56:38.142 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:56:38.142 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 249937 2024-09-13T14:56:38.142 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.142 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:56:38.142 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.142 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:56:38.142 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.331885+0000", 2024-09-13T14:56:38.142 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:56:38.142 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.142 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.142 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:56:38.142 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.331885+0000", 2024-09-13T14:56:38.142 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:56:38.143 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.143 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.143 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:56:38.143 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.331886+0000", 2024-09-13T14:56:38.143 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.8800000000000003e-07 2024-09-13T14:56:38.143 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.143 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.143 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:56:38.143 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.331888+0000", 2024-09-13T14:56:38.143 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.2340000000000001e-06 2024-09-13T14:56:38.143 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.143 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.143 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:56:38.143 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.331895+0000", 2024-09-13T14:56:38.144 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.5989999999999996e-06 2024-09-13T14:56:38.144 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.144 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.144 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:56:38.144 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.331901+0000", 2024-09-13T14:56:38.144 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.4639999999999997e-06 2024-09-13T14:56:38.144 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.144 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.144 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.144 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.332688+0000", 2024-09-13T14:56:38.144 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00078735199999999995 2024-09-13T14:56:38.144 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.144 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.144 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:56:38.145 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.332752+0000", 2024-09-13T14:56:38.145 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.3461999999999996e-05 2024-09-13T14:56:38.145 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.145 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.145 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.145 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.610018+0000", 2024-09-13T14:56:38.145 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2772668039999999 2024-09-13T14:56:38.145 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.145 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.145 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:56:38.145 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.610119+0000", 2024-09-13T14:56:38.145 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00010053800000000001 2024-09-13T14:56:38.145 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.145 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.146 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:56:38.146 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.610263+0000", 2024-09-13T14:56:38.146 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00014417 2024-09-13T14:56:38.146 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.146 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.146 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:56:38.146 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.622144+0000", 2024-09-13T14:56:38.146 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.011880559000000001 2024-09-13T14:56:38.146 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.146 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.146 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.146 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.622243+0000", 2024-09-13T14:56:38.146 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.9465999999999999e-05 2024-09-13T14:56:38.146 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.147 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.147 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.147 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.626338+0000", 2024-09-13T14:56:38.147 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0040946330000000003 2024-09-13T14:56:38.147 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.147 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.147 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:56:38.147 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.626390+0000", 2024-09-13T14:56:38.147 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.2049000000000003e-05 2024-09-13T14:56:38.147 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.147 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.147 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:56:38.147 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.626486+0000", 2024-09-13T14:56:38.147 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.6614999999999996e-05 2024-09-13T14:56:38.148 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:56:38.148 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:56:38.148 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:56:38.148 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.148 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.148 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:249947 8.f 8:fc361518:::benchmark_data_smithi102_313275_object119360:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-13T14:56:38.148 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:50:13.340670+0000", 2024-09-13T14:56:38.148 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 384.69519198, 2024-09-13T14:56:38.148 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2950710439999999, 2024-09-13T14:56:38.148 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:56:38.148 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:56:38.148 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:56:38.148 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:56:38.148 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:56:38.149 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:56:38.149 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 249947 2024-09-13T14:56:38.149 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.149 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:56:38.149 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.149 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:56:38.149 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.340670+0000", 2024-09-13T14:56:38.149 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:56:38.149 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.149 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.149 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:56:38.149 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.340670+0000", 2024-09-13T14:56:38.149 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:56:38.149 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.150 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.150 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:56:38.150 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.340671+0000", 2024-09-13T14:56:38.150 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.5740000000000001e-06 2024-09-13T14:56:38.150 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.150 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.150 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:56:38.150 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.340676+0000", 2024-09-13T14:56:38.150 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.2989999999999996e-06 2024-09-13T14:56:38.150 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.150 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.150 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:56:38.150 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.340692+0000", 2024-09-13T14:56:38.150 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.5838999999999999e-05 2024-09-13T14:56:38.151 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.151 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.151 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:56:38.151 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.340699+0000", 2024-09-13T14:56:38.151 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.8990000000000004e-06 2024-09-13T14:56:38.151 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.151 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.151 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.151 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.340745+0000", 2024-09-13T14:56:38.151 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.5435999999999997e-05 2024-09-13T14:56:38.151 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.151 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.151 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:56:38.151 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.340753+0000", 2024-09-13T14:56:38.152 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.4630000000000001e-06 2024-09-13T14:56:38.152 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.152 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.152 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.152 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.614282+0000", 2024-09-13T14:56:38.152 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.273528888 2024-09-13T14:56:38.152 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.152 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.152 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:56:38.152 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.614401+0000", 2024-09-13T14:56:38.152 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000119292 2024-09-13T14:56:38.152 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.152 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.152 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:56:38.153 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.614528+0000", 2024-09-13T14:56:38.153 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00012681899999999999 2024-09-13T14:56:38.153 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.153 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.153 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:56:38.153 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.622480+0000", 2024-09-13T14:56:38.153 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0079516880000000002 2024-09-13T14:56:38.153 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.153 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.153 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.153 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.626648+0000", 2024-09-13T14:56:38.153 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0041681260000000003 2024-09-13T14:56:38.153 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.154 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.154 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.154 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.635542+0000", 2024-09-13T14:56:38.154 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0088941709999999993 2024-09-13T14:56:38.154 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.154 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.154 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:56:38.154 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.635610+0000", 2024-09-13T14:56:38.154 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.8392999999999994e-05 2024-09-13T14:56:38.154 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.154 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.154 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:56:38.154 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.635741+0000", 2024-09-13T14:56:38.154 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00013015699999999999 2024-09-13T14:56:38.155 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:56:38.155 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:56:38.155 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:56:38.155 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.155 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.156 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:249954 8.f 8:feebf84b:::benchmark_data_smithi102_313275_object119367:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-13T14:56:38.156 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:50:13.350386+0000", 2024-09-13T14:56:38.156 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 384.68547601099999, 2024-09-13T14:56:38.156 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.285591291, 2024-09-13T14:56:38.156 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:56:38.156 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:56:38.156 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:56:38.156 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:56:38.156 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:56:38.156 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:56:38.156 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 249954 2024-09-13T14:56:38.156 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.156 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:56:38.156 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.157 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:56:38.157 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.350386+0000", 2024-09-13T14:56:38.157 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:56:38.157 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.157 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.157 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:56:38.157 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.350386+0000", 2024-09-13T14:56:38.157 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:56:38.157 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.157 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.157 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:56:38.157 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.350387+0000", 2024-09-13T14:56:38.157 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.443e-06 2024-09-13T14:56:38.158 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.158 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.158 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:56:38.158 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.350392+0000", 2024-09-13T14:56:38.158 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.3279999999999996e-06 2024-09-13T14:56:38.158 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.158 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.158 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:56:38.158 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.350402+0000", 2024-09-13T14:56:38.158 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.7280000000000007e-06 2024-09-13T14:56:38.158 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.158 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.158 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:56:38.158 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.350409+0000", 2024-09-13T14:56:38.159 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.1180000000000003e-06 2024-09-13T14:56:38.159 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.159 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.159 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.159 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.350501+0000", 2024-09-13T14:56:38.159 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.1643000000000003e-05 2024-09-13T14:56:38.159 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.159 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.159 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:56:38.159 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.350509+0000", 2024-09-13T14:56:38.159 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.6370000000000005e-06 2024-09-13T14:56:38.159 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.159 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.159 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.160 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.618359+0000", 2024-09-13T14:56:38.160 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.267849821 2024-09-13T14:56:38.160 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.160 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.160 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:56:38.160 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:38.161 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.618466+0000", 2024-09-13T14:56:38.161 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000106984 2024-09-13T14:56:38.161 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.161 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.161 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:56:38.161 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.618581+0000", 2024-09-13T14:56:38.161 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00011464300000000001 2024-09-13T14:56:38.162 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.162 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.162 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:56:38.162 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.626300+0000", 2024-09-13T14:56:38.162 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0077186609999999999 2024-09-13T14:56:38.162 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.162 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.162 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.162 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.626695+0000", 2024-09-13T14:56:38.162 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00039502299999999999 2024-09-13T14:56:38.162 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.162 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.162 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.163 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.635799+0000", 2024-09-13T14:56:38.163 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0091048840000000006 2024-09-13T14:56:38.163 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.163 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.163 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:56:38.163 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.635874+0000", 2024-09-13T14:56:38.163 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.4433999999999999e-05 2024-09-13T14:56:38.163 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.163 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.163 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:56:38.163 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.635977+0000", 2024-09-13T14:56:38.163 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000102944 2024-09-13T14:56:38.163 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:56:38.164 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:56:38.164 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:56:38.164 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.164 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.164 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:249959 8.f 8:f42b2733:::benchmark_data_smithi102_313275_object119372:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-13T14:56:38.164 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:50:13.351208+0000", 2024-09-13T14:56:38.164 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 384.684653099, 2024-09-13T14:56:38.164 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.28519165, 2024-09-13T14:56:38.164 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:56:38.164 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:56:38.164 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:56:38.165 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:56:38.165 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:56:38.165 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:56:38.165 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 249959 2024-09-13T14:56:38.165 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.165 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:56:38.165 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.165 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:56:38.165 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.351208+0000", 2024-09-13T14:56:38.165 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:56:38.166 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.166 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.166 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:56:38.166 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.351208+0000", 2024-09-13T14:56:38.166 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:56:38.166 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.166 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.166 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:56:38.166 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.351209+0000", 2024-09-13T14:56:38.166 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.3099999999999996e-07 2024-09-13T14:56:38.166 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.166 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.166 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:56:38.166 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.351214+0000", 2024-09-13T14:56:38.167 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.622e-06 2024-09-13T14:56:38.167 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.167 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.167 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:56:38.167 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.351228+0000", 2024-09-13T14:56:38.167 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4280999999999999e-05 2024-09-13T14:56:38.167 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.167 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.167 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:56:38.167 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.351236+0000", 2024-09-13T14:56:38.167 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.6190000000000002e-06 2024-09-13T14:56:38.167 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.167 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.168 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.168 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.351309+0000", 2024-09-13T14:56:38.168 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.2606000000000002e-05 2024-09-13T14:56:38.168 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.168 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.168 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:56:38.168 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.351316+0000", 2024-09-13T14:56:38.168 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.8189999999999992e-06 2024-09-13T14:56:38.168 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.168 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.168 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.168 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.622504+0000", 2024-09-13T14:56:38.168 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.271187625 2024-09-13T14:56:38.169 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.169 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.169 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:56:38.169 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.622633+0000", 2024-09-13T14:56:38.169 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000129474 2024-09-13T14:56:38.169 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.169 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.169 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:56:38.169 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.622746+0000", 2024-09-13T14:56:38.169 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00011304299999999999 2024-09-13T14:56:38.169 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.169 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.169 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:56:38.169 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.635757+0000", 2024-09-13T14:56:38.170 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.013010649000000001 2024-09-13T14:56:38.170 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.170 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.170 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.170 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.636075+0000", 2024-09-13T14:56:38.170 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000317764 2024-09-13T14:56:38.170 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.170 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.170 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.170 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.636159+0000", 2024-09-13T14:56:38.170 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.3968999999999995e-05 2024-09-13T14:56:38.170 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.170 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.171 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:56:38.171 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.636306+0000", 2024-09-13T14:56:38.171 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00014711999999999999 2024-09-13T14:56:38.171 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.171 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.171 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:56:38.171 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.636400+0000", 2024-09-13T14:56:38.171 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.4128000000000004e-05 2024-09-13T14:56:38.171 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:56:38.171 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:56:38.171 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:56:38.171 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.171 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.171 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:249975 8.f 8:f1516336:::benchmark_data_smithi102_313275_object119388:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-13T14:56:38.172 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:50:13.368418+0000", 2024-09-13T14:56:38.172 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 384.66744340000002, 2024-09-13T14:56:38.172 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.275804943, 2024-09-13T14:56:38.172 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:56:38.172 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:56:38.172 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:56:38.172 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:56:38.172 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:56:38.172 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:56:38.172 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 249975 2024-09-13T14:56:38.172 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.172 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:56:38.172 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.173 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:56:38.173 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.368418+0000", 2024-09-13T14:56:38.173 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:56:38.173 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.173 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.173 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:56:38.173 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.368418+0000", 2024-09-13T14:56:38.173 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:56:38.173 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.173 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.173 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:56:38.173 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.368420+0000", 2024-09-13T14:56:38.173 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.5269999999999999e-06 2024-09-13T14:56:38.173 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.174 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.174 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:56:38.174 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.368426+0000", 2024-09-13T14:56:38.174 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.2600000000000002e-06 2024-09-13T14:56:38.174 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.174 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.174 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:56:38.174 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.368440+0000", 2024-09-13T14:56:38.174 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4518999999999999e-05 2024-09-13T14:56:38.174 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.174 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.174 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:56:38.174 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.368452+0000", 2024-09-13T14:56:38.175 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2017e-05 2024-09-13T14:56:38.175 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.175 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.175 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.175 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.368534+0000", 2024-09-13T14:56:38.175 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.1895000000000001e-05 2024-09-13T14:56:38.175 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.175 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.175 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:56:38.175 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.368546+0000", 2024-09-13T14:56:38.175 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1505000000000001e-05 2024-09-13T14:56:38.175 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.175 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.176 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.176 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.626725+0000", 2024-09-13T14:56:38.176 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 84 16 100362 100346 74.6388 55.875 0.0311098 0.013392 2024-09-13T14:56:38.176 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.258179167 2024-09-13T14:56:38.177 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.177 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.177 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:56:38.177 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.628700+0000", 2024-09-13T14:56:38.177 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0019751130000000001 2024-09-13T14:56:38.177 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.177 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.177 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:56:38.177 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.628828+0000", 2024-09-13T14:56:38.177 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00012799999999999999 2024-09-13T14:56:38.177 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.177 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.178 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:56:38.178 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.636037+0000", 2024-09-13T14:56:38.178 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0072084949999999997 2024-09-13T14:56:38.178 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.178 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.178 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.178 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.636115+0000", 2024-09-13T14:56:38.178 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.7855999999999995e-05 2024-09-13T14:56:38.178 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.178 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.178 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.178 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.644016+0000", 2024-09-13T14:56:38.178 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0079011629999999992 2024-09-13T14:56:38.179 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.179 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.179 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:56:38.179 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.644121+0000", 2024-09-13T14:56:38.179 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00010556399999999999 2024-09-13T14:56:38.179 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.179 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.179 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:56:38.179 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.644223+0000", 2024-09-13T14:56:38.179 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00010186199999999999 2024-09-13T14:56:38.179 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:56:38.179 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:56:38.179 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:56:38.180 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.180 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.180 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:250009 8.f 8:f41ba2b5:::benchmark_data_smithi102_313275_object119422:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-13T14:56:38.180 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:50:13.408634+0000", 2024-09-13T14:56:38.180 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 384.62722791800002, 2024-09-13T14:56:38.180 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.235791938, 2024-09-13T14:56:38.180 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:56:38.180 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:56:38.180 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:56:38.180 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:56:38.180 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:56:38.180 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:56:38.180 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 250009 2024-09-13T14:56:38.180 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.181 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:56:38.181 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.181 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:56:38.181 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.408634+0000", 2024-09-13T14:56:38.181 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:56:38.181 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.181 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.181 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:56:38.182 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.408634+0000", 2024-09-13T14:56:38.182 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:56:38.182 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.182 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.182 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:56:38.182 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.408635+0000", 2024-09-13T14:56:38.182 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0610000000000001e-06 2024-09-13T14:56:38.182 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.182 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.182 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:56:38.182 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.408639+0000", 2024-09-13T14:56:38.182 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.5299999999999998e-06 2024-09-13T14:56:38.182 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.182 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.183 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:56:38.183 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.408654+0000", 2024-09-13T14:56:38.183 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.5174000000000001e-05 2024-09-13T14:56:38.183 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.183 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.183 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:56:38.183 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.408662+0000", 2024-09-13T14:56:38.183 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.3039999999999999e-06 2024-09-13T14:56:38.183 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.183 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.183 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.183 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.408995+0000", 2024-09-13T14:56:38.183 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00033359699999999998 2024-09-13T14:56:38.184 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.184 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.184 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:56:38.184 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.409004+0000", 2024-09-13T14:56:38.184 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.3969999999999994e-06 2024-09-13T14:56:38.184 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.184 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.184 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.184 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.632581+0000", 2024-09-13T14:56:38.184 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.223577205 2024-09-13T14:56:38.184 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.184 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.184 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:56:38.184 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.632695+0000", 2024-09-13T14:56:38.185 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000114461 2024-09-13T14:56:38.185 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.185 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.185 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:56:38.185 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.632781+0000", 2024-09-13T14:56:38.185 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.5303999999999995e-05 2024-09-13T14:56:38.185 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.185 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.185 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:56:38.185 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.640128+0000", 2024-09-13T14:56:38.185 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0073468739999999998 2024-09-13T14:56:38.185 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.186 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.186 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.186 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.643973+0000", 2024-09-13T14:56:38.186 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0038452460000000001 2024-09-13T14:56:38.186 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.186 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.186 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.186 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.644260+0000", 2024-09-13T14:56:38.186 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00028718999999999999 2024-09-13T14:56:38.186 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.186 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.186 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:56:38.187 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.644340+0000", 2024-09-13T14:56:38.187 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.9756e-05 2024-09-13T14:56:38.187 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.187 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.187 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:56:38.187 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.644426+0000", 2024-09-13T14:56:38.187 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.5839000000000003e-05 2024-09-13T14:56:38.187 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:56:38.187 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:56:38.187 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:56:38.187 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.187 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.187 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:250012 8.f 8:f74e2086:::benchmark_data_smithi102_313275_object119425:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-13T14:56:38.188 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:50:13.413092+0000", 2024-09-13T14:56:38.188 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 384.62276967000003, 2024-09-13T14:56:38.188 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2354481820000001, 2024-09-13T14:56:38.188 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:56:38.188 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:56:38.188 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:56:38.188 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:56:38.188 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:56:38.188 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:56:38.188 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 250012 2024-09-13T14:56:38.188 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.188 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:56:38.188 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.188 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:56:38.189 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.413092+0000", 2024-09-13T14:56:38.189 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:56:38.189 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.189 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.189 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:56:38.189 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.413092+0000", 2024-09-13T14:56:38.189 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:56:38.189 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.189 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.189 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:56:38.189 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.413094+0000", 2024-09-13T14:56:38.189 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.8819999999999999e-06 2024-09-13T14:56:38.189 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.189 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.190 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:56:38.190 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.413099+0000", 2024-09-13T14:56:38.190 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.7790000000000003e-06 2024-09-13T14:56:38.190 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.190 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.190 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:56:38.190 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.413109+0000", 2024-09-13T14:56:38.190 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0096e-05 2024-09-13T14:56:38.190 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.190 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.190 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:56:38.190 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.413117+0000", 2024-09-13T14:56:38.190 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.9030000000000004e-06 2024-09-13T14:56:38.191 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.191 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.191 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.191 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.413195+0000", 2024-09-13T14:56:38.191 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.8024999999999999e-05 2024-09-13T14:56:38.191 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.191 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.191 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:56:38.191 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.413203+0000", 2024-09-13T14:56:38.191 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.7530000000000007e-06 2024-09-13T14:56:38.191 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.191 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.191 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.191 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.636436+0000", 2024-09-13T14:56:38.192 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2232331329999999 2024-09-13T14:56:38.192 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.192 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.192 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:56:38.192 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.636568+0000", 2024-09-13T14:56:38.192 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000131144 2024-09-13T14:56:38.192 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.192 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.192 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:56:38.192 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.636679+0000", 2024-09-13T14:56:38.192 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000111508 2024-09-13T14:56:38.192 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.192 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.193 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.193 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.644461+0000", 2024-09-13T14:56:38.193 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0077816170000000002 2024-09-13T14:56:38.193 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.193 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.193 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.193 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.648104+0000", 2024-09-13T14:56:38.193 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0036430130000000001 2024-09-13T14:56:38.193 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.193 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.193 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:56:38.193 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.648133+0000", 2024-09-13T14:56:38.193 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.8815000000000001e-05 2024-09-13T14:56:38.193 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.194 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.194 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:56:38.194 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.648439+0000", 2024-09-13T14:56:38.194 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000306915 2024-09-13T14:56:38.194 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.194 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.194 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:56:38.194 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.648540+0000", 2024-09-13T14:56:38.194 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000100599 2024-09-13T14:56:38.194 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:56:38.194 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:56:38.194 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:56:38.194 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.194 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.195 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:250019 8.f 8:f4c1da74:::benchmark_data_smithi102_313275_object119432:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-13T14:56:38.195 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:50:13.421510+0000", 2024-09-13T14:56:38.195 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 384.61435189299999, 2024-09-13T14:56:38.195 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.242516763, 2024-09-13T14:56:38.195 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:56:38.195 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:56:38.195 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:56:38.195 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:56:38.195 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:56:38.195 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:56:38.195 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 250019 2024-09-13T14:56:38.195 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.195 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:56:38.196 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.196 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:56:38.196 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.421510+0000", 2024-09-13T14:56:38.196 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:56:38.196 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.196 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.196 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:56:38.196 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.421510+0000", 2024-09-13T14:56:38.196 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:56:38.196 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.196 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.196 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:56:38.196 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.421511+0000", 2024-09-13T14:56:38.196 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.8599999999999997e-07 2024-09-13T14:56:38.197 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.197 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.197 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:56:38.197 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.421515+0000", 2024-09-13T14:56:38.197 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.9609999999999999e-06 2024-09-13T14:56:38.197 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.197 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.197 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:56:38.197 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.421525+0000", 2024-09-13T14:56:38.197 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0052e-05 2024-09-13T14:56:38.197 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.197 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.197 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:56:38.197 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.421530+0000", 2024-09-13T14:56:38.198 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.8560000000000003e-06 2024-09-13T14:56:38.198 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.198 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.198 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.198 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.421588+0000", 2024-09-13T14:56:38.198 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.7553999999999998e-05 2024-09-13T14:56:38.198 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.198 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.198 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:56:38.198 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.421596+0000", 2024-09-13T14:56:38.198 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.6669999999999996e-06 2024-09-13T14:56:38.198 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.198 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.198 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:56:38.199 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.640161+0000", 2024-09-13T14:56:38.199 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2185656380000001 2024-09-13T14:56:38.199 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.199 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.199 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:56:38.199 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.640277+0000", 2024-09-13T14:56:38.199 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000116193 2024-09-13T14:56:38.199 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.199 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.199 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:56:38.199 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.640398+0000", 2024-09-13T14:56:38.200 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00012017 2024-09-13T14:56:38.200 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.200 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.200 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:56:38.200 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.652552+0000", 2024-09-13T14:56:38.200 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.012154094000000001 2024-09-13T14:56:38.200 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.200 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.200 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.200 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.652630+0000", 2024-09-13T14:56:38.200 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.8067999999999999e-05 2024-09-13T14:56:38.200 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.201 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.201 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:38.201 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.663828+0000", 2024-09-13T14:56:38.201 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.011198143000000001 2024-09-13T14:56:38.201 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.201 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.201 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:56:38.201 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.663934+0000", 2024-09-13T14:56:38.201 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00010572400000000001 2024-09-13T14:56:38.201 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:56:38.201 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:56:38.201 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:56:38.201 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.664026+0000", 2024-09-13T14:56:38.201 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.2757000000000007e-05 2024-09-13T14:56:38.202 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:56:38.202 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:56:38.202 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:56:38.202 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:56:38.202 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:56:38.202 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:56:38.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:38.244 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:38.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:38.345 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:38.371 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:38.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:38.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:38.473 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:38.514 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:38.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:38.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:38.574 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:38 smithi102 ceph-mon[110183]: pgmap v1554: 109 pgs: 109 active+clean; 6.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 55 MiB/s wr, 882 op/s 2024-09-13T14:56:38.574 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:38 smithi102 ceph-mon[115269]: pgmap v1554: 109 pgs: 109 active+clean; 6.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 55 MiB/s wr, 882 op/s 2024-09-13T14:56:38.616 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:38.642 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:38.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:38.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:38.733 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:56:38.744 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:38.765 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:38.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:38.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:38.869 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:38.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:38.914 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:38.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:38.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:38 smithi195 ceph-mon[104262]: pgmap v1554: 109 pgs: 109 active+clean; 6.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 55 MiB/s wr, 882 op/s 2024-09-13T14:56:39.015 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:39.041 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:39.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:39.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:39.129 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 85 16 101146 101130 74.3372 49 0.00886314 0.0134469 2024-09-13T14:56:39.143 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:39.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:39.194 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:39.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:39.295 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:39.330 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:39.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:39.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:39.432 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:39.465 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:39.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:39.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:39.567 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:39.594 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:39.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:39.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:39.696 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:39.738 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:39.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:39.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:39.840 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:39.854 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T14:56:39.865 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:39.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:39.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:39.968 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:40.004 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:40.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:40.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:40.106 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:40.130 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 86 16 102111 102095 74.1733 60.3125 0.00974825 0.0134763 2024-09-13T14:56:40.142 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:40.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:40.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:40.225 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:40.245 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:40.286 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:40.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:40.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:40.388 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:40.433 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:56:40.436 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:40.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:40.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:40.538 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:40.583 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:40.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:40.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:40.685 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:40.718 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:40.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:40.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:40.820 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:40.854 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:40.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:40.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:40.956 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:41.003 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:41.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:41.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:41.105 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:41.132 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 87 16 103058 103042 73.9994 59.1875 0.0155888 0.0135079 2024-09-13T14:56:41.145 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:41.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:41.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:41.248 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:41.280 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:41.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:41.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:41.318 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:41 smithi102 ceph-mon[110183]: pgmap v1555: 109 pgs: 109 active+clean; 6.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 56 MiB/s wr, 889 op/s 2024-09-13T14:56:41.318 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:41 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:56:41.318 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:41 smithi102 ceph-mon[115269]: pgmap v1555: 109 pgs: 109 active+clean; 6.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 56 MiB/s wr, 889 op/s 2024-09-13T14:56:41.318 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:41 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:56:41.382 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:41.405 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:41 smithi195 ceph-mon[104262]: pgmap v1555: 109 pgs: 109 active+clean; 6.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 56 MiB/s wr, 889 op/s 2024-09-13T14:56:41.406 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:41 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:56:41.410 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:41.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:41.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:41.512 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:41.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:41.541 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:41.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:41.642 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:41.670 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:41.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:41.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:41.772 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:41.811 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:41.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:41.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:41.913 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:41.952 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:41.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:41.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:42.054 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:42.095 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:42.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:42.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:42.132 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 88 16 103997 103981 73.8254 58.6875 0.0135058 0.0135396 2024-09-13T14:56:42.197 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:42.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:42.238 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:42.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:42.339 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:42.368 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:42.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:42.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:42.470 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:42.484 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:56:42.484 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:56:42.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:42.508 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:42.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:42.551 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:42.609 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:42.635 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:42.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:42.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:42.705 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:56:42.705 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:56:42.705 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:56:42.705 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:56:42.705 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:56:42.705 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:56:42.705 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:56:42.706 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:56:42.706 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:56:42.706 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:56:42.706 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:56:42.706 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:56:42.706 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:56:42.706 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:56:42.706 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:56:42.706 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:56:42.737 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:42.775 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:42.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:42.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:42.877 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:42.919 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:42.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:42.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:43.021 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:43.046 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:43.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:43.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:43.132 INFO:tasks.radosbench.radosbench.0.smithi102.stdout: 89 16 104917 104901 73.642 57.5 0.0241394 0.0135738 2024-09-13T14:56:43.149 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:43.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:43.184 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:43.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:43.286 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:43.302 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T14:56:43.320 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:43.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:43.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:43.393 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:43 smithi102 ceph-mon[110183]: pgmap v1556: 109 pgs: 109 active+clean; 6.2 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 55 MiB/s wr, 886 op/s 2024-09-13T14:56:43.393 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:43 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3513578337' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:56:43.394 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:43 smithi102 ceph-mon[115269]: pgmap v1556: 109 pgs: 109 active+clean; 6.2 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 55 MiB/s wr, 886 op/s 2024-09-13T14:56:43.394 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:43 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3513578337' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:56:43.423 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:43.466 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:43.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:43.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:43.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:43 smithi195 ceph-mon[104262]: pgmap v1556: 109 pgs: 109 active+clean; 6.2 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 55 MiB/s wr, 886 op/s 2024-09-13T14:56:43.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:43 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3513578337' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:56:43.567 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:56:43.568 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:43.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:43.607 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:43.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:43.708 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:43.733 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:43.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:43.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:43.835 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:43.876 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:43.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:43.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:43.978 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:44.018 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:44.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:44.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:44.120 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:44.132 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Total time run: 90.0107 2024-09-13T14:56:44.132 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Total writes made: 105839 2024-09-13T14:56:44.132 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Write size: 65536 2024-09-13T14:56:44.132 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Object size: 65536 2024-09-13T14:56:44.133 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Bandwidth (MB/sec): 73.4906 2024-09-13T14:56:44.133 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Stddev Bandwidth: 37.5522 2024-09-13T14:56:44.133 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Max bandwidth (MB/sec): 228.562 2024-09-13T14:56:44.133 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Min bandwidth (MB/sec): 0 2024-09-13T14:56:44.133 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Average IOPS: 1175 2024-09-13T14:56:44.133 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Stddev IOPS: 600.834 2024-09-13T14:56:44.133 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Max IOPS: 3657 2024-09-13T14:56:44.133 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Min IOPS: 0 2024-09-13T14:56:44.133 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Average Latency(s): 0.0136026 2024-09-13T14:56:44.133 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Stddev Latency(s): 0.0445385 2024-09-13T14:56:44.133 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Max latency(s): 2.27933 2024-09-13T14:56:44.134 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Min latency(s): 0.00130255 2024-09-13T14:56:44.139 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Cleaning up (deleting benchmark objects) 2024-09-13T14:56:44.143 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:44.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:44.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:44.245 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:44.270 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:44.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:44.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:44.372 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:44.410 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:44.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:44.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:44.512 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:44.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:44.550 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:44.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:44.651 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:44.677 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:44.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:44.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:44.778 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:44.819 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:44.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:44.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:44.922 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:44.947 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:44.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:44.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:45.049 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:45.074 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:45.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:45.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:45.176 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:45.210 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:45.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:45.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:45.312 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:45.355 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:45.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:45.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:45.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:45 smithi102 ceph-mon[110183]: pgmap v1557: 109 pgs: 109 active+clean; 6.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 56 MiB/s wr, 889 op/s 2024-09-13T14:56:45.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:45 smithi102 ceph-mon[115269]: pgmap v1557: 109 pgs: 109 active+clean; 6.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 56 MiB/s wr, 889 op/s 2024-09-13T14:56:45.431 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:56:45.457 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:45.480 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:45.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:45.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:45.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:45 smithi195 ceph-mon[104262]: pgmap v1557: 109 pgs: 109 active+clean; 6.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 56 MiB/s wr, 889 op/s 2024-09-13T14:56:45.582 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:45.614 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:45.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:45.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:45.716 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:45.742 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:45.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:45.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:45.844 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:45.871 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:45.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:45.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:45.973 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:45.997 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:45.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:45.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:46.040 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:46.100 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:46.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:46.150 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:46.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:46.166 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:56:46.167 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:56:46.176 INFO:teuthology.orchestra.run.smithi102.stderr:set pool 10 pgp_num to 16 2024-09-13T14:56:46.252 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:46.279 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:46.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:46.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:46.381 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:46.408 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:46.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:46.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:46.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:46 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/4017124911' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"} : dispatch 2024-09-13T14:56:46.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:46 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"} : dispatch 2024-09-13T14:56:46.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:46 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/4017124911' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"} : dispatch 2024-09-13T14:56:46.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:46 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"} : dispatch 2024-09-13T14:56:46.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:46 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/4017124911' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"} : dispatch 2024-09-13T14:56:46.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:46 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"} : dispatch 2024-09-13T14:56:46.510 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:46.553 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:46.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:46.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:46.655 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:46.701 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:46.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:46.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:46.779 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:56:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:56:46] "GET /metrics HTTP/1.1" 200 36566 "" "Prometheus/2.51.0" 2024-09-13T14:56:46.804 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:46.827 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:46.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:46.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:46.885 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T14:56:46.929 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:46.953 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:46.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:46.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:47.055 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:47.087 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:47.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:47.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:47.156 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:56:47.189 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:47.192 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is unset 2024-09-13T14:56:47.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:47.236 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:47.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:47.337 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:47.371 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:47.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:47.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:47.453 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:47 smithi195 ceph-mon[104262]: pgmap v1558: 109 pgs: 109 active+clean; 6.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 54 MiB/s wr, 869 op/s 2024-09-13T14:56:47.453 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:47 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"}]': finished 2024-09-13T14:56:47.453 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:47 smithi195 ceph-mon[104262]: osdmap e466: 8 total, 8 up, 7 in 2024-09-13T14:56:47.453 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:47 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3605062295' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:56:47.454 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:47 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:56:47.473 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:47.506 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:47.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:47.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:47.608 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:47.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:47.642 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:47.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:47.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:47 smithi102 ceph-mon[110183]: pgmap v1558: 109 pgs: 109 active+clean; 6.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 54 MiB/s wr, 869 op/s 2024-09-13T14:56:47.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:47 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"}]': finished 2024-09-13T14:56:47.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:47 smithi102 ceph-mon[110183]: osdmap e466: 8 total, 8 up, 7 in 2024-09-13T14:56:47.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:47 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3605062295' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:56:47.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:47 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:56:47.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:47 smithi102 ceph-mon[115269]: pgmap v1558: 109 pgs: 109 active+clean; 6.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 54 MiB/s wr, 869 op/s 2024-09-13T14:56:47.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:47 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"}]': finished 2024-09-13T14:56:47.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:47 smithi102 ceph-mon[115269]: osdmap e466: 8 total, 8 up, 7 in 2024-09-13T14:56:47.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:47 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3605062295' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:56:47.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:47 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:56:47.743 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:47.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:47.784 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:47.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:47.885 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:47.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:47.912 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:47.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:48.013 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:48.054 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:48.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:48.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:48.156 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:48.181 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:48.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:48.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:48.283 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:48.309 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:48.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:48.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:48.410 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:48 smithi195 ceph-mon[104262]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:56:48.410 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:48 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:56:48.410 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:48 smithi195 ceph-mon[104262]: osdmap e467: 8 total, 8 up, 7 in 2024-09-13T14:56:48.411 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:48.439 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:48.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:48.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:48.541 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:48.574 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:48.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:48.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:48.582 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:48 smithi102 ceph-mon[110183]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:56:48.582 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:48 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:56:48.582 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:48 smithi102 ceph-mon[110183]: osdmap e467: 8 total, 8 up, 7 in 2024-09-13T14:56:48.583 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:48 smithi102 ceph-mon[115269]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:56:48.583 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:48 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:56:48.583 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:48 smithi102 ceph-mon[115269]: osdmap e467: 8 total, 8 up, 7 in 2024-09-13T14:56:48.676 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:48.712 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:48.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:48.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:48.814 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:48.856 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:48.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:48.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:48.958 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:48.991 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:48.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:48.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:49.022 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:56:49.022 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:56:49.093 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:49.133 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:49.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:49.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:49.234 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:49.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:49 smithi195 ceph-mon[104262]: pgmap v1561: 109 pgs: 109 active+clean; 6.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 46 MiB/s wr, 908 op/s 2024-09-13T14:56:49.264 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:49.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:49.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:49.366 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:49.407 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:49.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:49.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:49.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:49 smithi102 ceph-mon[110183]: pgmap v1561: 109 pgs: 109 active+clean; 6.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 46 MiB/s wr, 908 op/s 2024-09-13T14:56:49.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:49 smithi102 ceph-mon[115269]: pgmap v1561: 109 pgs: 109 active+clean; 6.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 46 MiB/s wr, 908 op/s 2024-09-13T14:56:49.509 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:49.546 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:49.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:49.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:49.648 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:49.672 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:49.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:49.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:49.742 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T14:56:49.783 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:49.826 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:49.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:49.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:49.929 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:49.958 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:49.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:49.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:50.048 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:56:50.060 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:50.093 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:50.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:50.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:50.195 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:50.241 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:50.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:50.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:50.313 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:56:50.314 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:56:50.343 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:50.350 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:56:50.350 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:56:50.372 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:50.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:50.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:50.384 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:56:50.384 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:56:50.418 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:56:50.419 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:56:50.449 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:56:50.449 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:56:50.473 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:50.478 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:56:50.478 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:56:50.496 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:50.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:50.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:50.517 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:56:50.517 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:56:50.549 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:56:50.549 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:56:50.598 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:50.634 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:50.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:50.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:50.736 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:50.771 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:50.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:50.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:50.873 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:50.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:50.899 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:50.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:51.000 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:51.034 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:51.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:51.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:51.082 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:56:51.135 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:51.161 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:51.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:51.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:51.263 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:51.288 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:51.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:51.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:51.390 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:51.416 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:51.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:51.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:51.474 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:51.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:51 smithi195 ceph-mon[104262]: pgmap v1562: 109 pgs: 109 active+clean; 6.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 127 B/s rd, 31 MiB/s wr, 917 op/s 2024-09-13T14:56:51.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:51 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2040010365' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:56:51.518 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:51.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:51.567 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:51.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:51.608 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:51 smithi102 ceph-mon[110183]: pgmap v1562: 109 pgs: 109 active+clean; 6.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 127 B/s rd, 31 MiB/s wr, 917 op/s 2024-09-13T14:56:51.608 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:51 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2040010365' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:56:51.609 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:51 smithi102 ceph-mon[115269]: pgmap v1562: 109 pgs: 109 active+clean; 6.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 127 B/s rd, 31 MiB/s wr, 917 op/s 2024-09-13T14:56:51.609 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:51 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2040010365' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:56:51.668 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:51.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:51.692 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:51.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:51.793 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:51.827 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:51.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:51.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:51.929 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:51.953 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:51.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:51.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:52.055 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:52.081 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:52.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:52.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:52.182 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:52.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:52.225 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:52.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:52.325 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:52.361 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:52.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:52.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:52.463 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:52.501 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:52.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:52.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:52.603 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:52.628 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:52.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:52.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:52.730 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:52.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:52.763 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:52.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:52.864 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:52.892 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:52.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:52.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:52.994 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:53.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:53.020 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:53.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:53.058 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:56:53.058 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-13T14:56:53.058 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.058 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36738.0:114452 10.9 10:9380fa50:::benchmark_data_smithi102_377304_object8610:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e467)", 2024-09-13T14:56:53.058 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:56:53.050841+0000", 2024-09-13T14:56:53.058 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.004278156, 2024-09-13T14:56:53.058 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0043066720000000001, 2024-09-13T14:56:53.059 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:56:53.059 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:56:53.059 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:56:53.059 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36738", 2024-09-13T14:56:53.059 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/2915890700", 2024-09-13T14:56:53.059 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 114452 2024-09-13T14:56:53.059 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.059 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:56:53.059 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.059 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:56:53.059 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.050841+0000", 2024-09-13T14:56:53.059 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:56:53.059 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.059 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.060 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:56:53.060 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.050841+0000", 2024-09-13T14:56:53.060 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:56:53.060 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.060 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.060 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:56:53.060 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.050839+0000", 2024-09-13T14:56:53.060 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:56:53.060 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.060 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.060 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:56:53.060 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.050844+0000", 2024-09-13T14:56:53.060 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9849999999999997e-06 2024-09-13T14:56:53.061 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.061 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.061 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:56:53.061 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.050852+0000", 2024-09-13T14:56:53.061 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2230000000000001e-06 2024-09-13T14:56:53.061 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.061 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.061 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:56:53.061 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.050861+0000", 2024-09-13T14:56:53.061 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6900000000000004e-06 2024-09-13T14:56:53.061 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.061 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.061 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:56:53.062 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.051106+0000", 2024-09-13T14:56:53.062 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00024436999999999999 2024-09-13T14:56:53.062 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.062 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.062 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:56:53.062 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.051215+0000", 2024-09-13T14:56:53.062 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000109706 2024-09-13T14:56:53.062 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.062 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.062 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 2,7", 2024-09-13T14:56:53.062 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.051318+0000", 2024-09-13T14:56:53.062 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010252899999999999 2024-09-13T14:56:53.062 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:56:53.063 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:56:53.063 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:56:53.063 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.063 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.063 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36738.0:114453 10.a 10:5ae50816:::benchmark_data_smithi102_377304_object8611:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e467)", 2024-09-13T14:56:53.063 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:56:53.050877+0000", 2024-09-13T14:56:53.063 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.0042416449999999996, 2024-09-13T14:56:53.063 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0043460740000000001, 2024-09-13T14:56:53.063 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:56:53.063 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:56:53.063 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:56:53.063 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36738", 2024-09-13T14:56:53.063 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/2915890700", 2024-09-13T14:56:53.063 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 114453 2024-09-13T14:56:53.064 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.064 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:56:53.064 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.064 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:56:53.064 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.050877+0000", 2024-09-13T14:56:53.064 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:56:53.064 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.064 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.064 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:56:53.064 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.050877+0000", 2024-09-13T14:56:53.064 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:56:53.064 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.064 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.065 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:56:53.065 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.050877+0000", 2024-09-13T14:56:53.065 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:56:53.065 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.065 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.065 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:56:53.065 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.050879+0000", 2024-09-13T14:56:53.065 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.6060000000000001e-06 2024-09-13T14:56:53.065 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.065 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.065 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:56:53.065 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.050887+0000", 2024-09-13T14:56:53.065 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2179999999999999e-06 2024-09-13T14:56:53.066 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.066 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.066 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:56:53.066 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.050892+0000", 2024-09-13T14:56:53.066 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5349999999999998e-06 2024-09-13T14:56:53.066 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.066 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.066 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:56:53.066 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.053978+0000", 2024-09-13T14:56:53.066 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0030863090000000002 2024-09-13T14:56:53.066 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.066 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.066 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:56:53.066 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.054177+0000", 2024-09-13T14:56:53.067 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000198269 2024-09-13T14:56:53.067 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.067 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.067 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,6", 2024-09-13T14:56:53.067 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.054281+0000", 2024-09-13T14:56:53.067 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000104227 2024-09-13T14:56:53.067 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:56:53.067 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:56:53.067 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:56:53.067 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.068 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.068 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36738.0:114454 10.9 10:90506dbd:::benchmark_data_smithi102_377304_object8612:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e467)", 2024-09-13T14:56:53.068 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:56:53.051056+0000", 2024-09-13T14:56:53.068 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.0040627600000000003, 2024-09-13T14:56:53.068 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0042340540000000001, 2024-09-13T14:56:53.068 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:56:53.068 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T14:56:53.068 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:56:53.068 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36738", 2024-09-13T14:56:53.068 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/2915890700", 2024-09-13T14:56:53.068 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 114454 2024-09-13T14:56:53.068 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.069 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:56:53.069 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.069 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:56:53.069 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.051056+0000", 2024-09-13T14:56:53.069 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:56:53.069 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.069 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.069 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:56:53.069 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.051056+0000", 2024-09-13T14:56:53.069 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:56:53.069 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.069 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.069 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:56:53.069 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.051055+0000", 2024-09-13T14:56:53.070 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:56:53.070 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.070 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.070 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:56:53.070 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.051059+0000", 2024-09-13T14:56:53.070 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0459999999999999e-06 2024-09-13T14:56:53.070 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.070 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.070 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:56:53.070 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.051067+0000", 2024-09-13T14:56:53.070 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9130000000000003e-06 2024-09-13T14:56:53.070 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.070 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.071 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:56:53.071 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.051072+0000", 2024-09-13T14:56:53.071 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2000000000000002e-06 2024-09-13T14:56:53.071 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.071 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.071 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:56:53.071 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.055139+0000", 2024-09-13T14:56:53.071 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0040662340000000002 2024-09-13T14:56:53.071 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.071 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.071 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:56:53.071 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.055244+0000", 2024-09-13T14:56:53.071 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010569999999999999 2024-09-13T14:56:53.071 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:56:53.072 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:56:53.072 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:56:53.072 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.072 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.072 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36738.0:114456 10.e e467/447 10:79640cb0:::benchmark_data_smithi102_377304_object8614:head v 467'7220, mlcod=467'7217)", 2024-09-13T14:56:53.072 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:56:53.053038+0000", 2024-09-13T14:56:53.072 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.0020806039999999998, 2024-09-13T14:56:53.072 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0023107510000000002, 2024-09-13T14:56:53.072 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:56:53.072 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T14:56:53.072 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:56:53.072 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.072 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:56:53.073 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.053038+0000", 2024-09-13T14:56:53.073 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:56:53.073 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.073 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.073 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:56:53.074 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.053038+0000", 2024-09-13T14:56:53.074 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:56:53.074 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.074 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.074 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:56:53.074 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.053037+0000", 2024-09-13T14:56:53.074 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:56:53.074 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.074 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.074 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:56:53.074 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.053043+0000", 2024-09-13T14:56:53.074 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.331e-06 2024-09-13T14:56:53.074 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.075 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.075 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:56:53.075 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.053057+0000", 2024-09-13T14:56:53.075 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3308e-05 2024-09-13T14:56:53.075 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.075 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.075 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:56:53.075 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.053063+0000", 2024-09-13T14:56:53.075 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7070000000000003e-06 2024-09-13T14:56:53.076 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.076 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.076 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:56:53.076 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.053719+0000", 2024-09-13T14:56:53.076 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00065554199999999995 2024-09-13T14:56:53.076 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.076 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.076 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:56:53.076 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.053735+0000", 2024-09-13T14:56:53.076 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5786000000000001e-05 2024-09-13T14:56:53.076 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:56:53.076 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:56:53.076 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:56:53.077 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.077 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.077 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36738.0:114459 10.6 10.47090896 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e467)", 2024-09-13T14:56:53.077 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:56:53.054015+0000", 2024-09-13T14:56:53.077 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.001103618, 2024-09-13T14:56:53.077 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0013860350000000001, 2024-09-13T14:56:53.077 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:56:53.077 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "queued for pg", 2024-09-13T14:56:53.077 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:56:53.077 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36738", 2024-09-13T14:56:53.077 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/2915890700", 2024-09-13T14:56:53.077 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 114459 2024-09-13T14:56:53.078 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.078 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:56:53.078 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.078 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:56:53.078 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.054015+0000", 2024-09-13T14:56:53.078 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:56:53.078 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.078 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.078 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:56:53.078 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.054015+0000", 2024-09-13T14:56:53.078 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:56:53.078 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.078 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.078 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:56:53.079 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.054014+0000", 2024-09-13T14:56:53.079 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:56:53.079 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.079 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.079 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:56:53.079 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.054019+0000", 2024-09-13T14:56:53.079 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1399999999999999e-06 2024-09-13T14:56:53.079 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.079 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.079 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:56:53.079 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.054027+0000", 2024-09-13T14:56:53.079 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8320000000000006e-06 2024-09-13T14:56:53.079 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.080 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.080 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:56:53.080 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.054033+0000", 2024-09-13T14:56:53.080 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.9150000000000001e-06 2024-09-13T14:56:53.080 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:56:53.080 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:56:53.080 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:56:53.080 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.080 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.080 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36738.0:114441 10.a 10:501d63e8:::benchmark_data_smithi102_377304_object8599:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e467)", 2024-09-13T14:56:53.080 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:56:53.039338+0000", 2024-09-13T14:56:53.080 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.015780531, 2024-09-13T14:56:53.080 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016113688000000001, 2024-09-13T14:56:53.080 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:56:53.081 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:56:53.081 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:56:53.081 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36738", 2024-09-13T14:56:53.081 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/2915890700", 2024-09-13T14:56:53.081 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 114441 2024-09-13T14:56:53.081 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.081 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:56:53.081 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.081 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:56:53.081 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.039338+0000", 2024-09-13T14:56:53.081 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:56:53.081 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.081 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.082 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:56:53.082 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.039338+0000", 2024-09-13T14:56:53.082 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:56:53.082 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.082 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.082 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:56:53.082 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.039337+0000", 2024-09-13T14:56:53.082 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:56:53.082 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.082 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.082 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:56:53.082 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.039341+0000", 2024-09-13T14:56:53.082 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.484e-06 2024-09-13T14:56:53.082 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.083 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.083 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:56:53.083 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.039349+0000", 2024-09-13T14:56:53.083 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0099999999999995e-06 2024-09-13T14:56:53.083 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.083 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.083 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:56:53.083 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.039355+0000", 2024-09-13T14:56:53.083 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7180000000000001e-06 2024-09-13T14:56:53.083 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.083 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.083 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:56:53.083 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.039682+0000", 2024-09-13T14:56:53.083 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00032687800000000001 2024-09-13T14:56:53.084 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.084 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.084 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:56:53.084 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.039781+0000", 2024-09-13T14:56:53.084 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.9382000000000005e-05 2024-09-13T14:56:53.084 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.084 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.084 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,6", 2024-09-13T14:56:53.084 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.040104+0000", 2024-09-13T14:56:53.084 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00032283799999999999 2024-09-13T14:56:53.084 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.084 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.084 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:56:53.084 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.049853+0000", 2024-09-13T14:56:53.085 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0097494609999999992 2024-09-13T14:56:53.085 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.085 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.085 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:53.085 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.050467+0000", 2024-09-13T14:56:53.085 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000613741 2024-09-13T14:56:53.085 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:56:53.085 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:56:53.085 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:56:53.085 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.085 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.085 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36738.0:114443 10.9 10:935ba4d4:::benchmark_data_smithi102_377304_object8601:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e467)", 2024-09-13T14:56:53.085 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:56:53.039441+0000", 2024-09-13T14:56:53.086 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.015678067, 2024-09-13T14:56:53.086 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016086000999999999, 2024-09-13T14:56:53.086 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:56:53.086 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:56:53.086 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:56:53.086 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36738", 2024-09-13T14:56:53.086 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/2915890700", 2024-09-13T14:56:53.086 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 114443 2024-09-13T14:56:53.086 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.086 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:56:53.086 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.086 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:56:53.086 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.039441+0000", 2024-09-13T14:56:53.086 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:56:53.087 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.087 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.087 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:56:53.087 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.039441+0000", 2024-09-13T14:56:53.087 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:56:53.087 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.087 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.087 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:56:53.087 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.039440+0000", 2024-09-13T14:56:53.087 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:56:53.088 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.088 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.088 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:56:53.088 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.039443+0000", 2024-09-13T14:56:53.088 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.1640000000000001e-06 2024-09-13T14:56:53.088 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.088 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.088 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:56:53.088 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.039450+0000", 2024-09-13T14:56:53.088 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8619999999999997e-06 2024-09-13T14:56:53.088 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.088 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.088 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:56:53.089 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.039456+0000", 2024-09-13T14:56:53.089 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3890000000000004e-06 2024-09-13T14:56:53.089 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.089 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.089 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:56:53.089 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.044215+0000", 2024-09-13T14:56:53.089 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0047592980000000003 2024-09-13T14:56:53.089 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.089 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.089 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:56:53.089 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.044335+0000", 2024-09-13T14:56:53.089 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012001 2024-09-13T14:56:53.089 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.090 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.090 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 2,7", 2024-09-13T14:56:53.090 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.044441+0000", 2024-09-13T14:56:53.090 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000105987 2024-09-13T14:56:53.090 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.090 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.090 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:56:53.090 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.050381+0000", 2024-09-13T14:56:53.090 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0059404770000000004 2024-09-13T14:56:53.090 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.090 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.090 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:56:53.090 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.051013+0000", 2024-09-13T14:56:53.090 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000631582 2024-09-13T14:56:53.091 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:56:53.091 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:56:53.091 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:56:53.091 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.091 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.091 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36738.0:114449 10.e e467/447 10:719fb15d:::benchmark_data_smithi102_377304_object8607:head v 467'7219, mlcod=467'7217)", 2024-09-13T14:56:53.091 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:56:53.044506+0000", 2024-09-13T14:56:53.091 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.010612632, 2024-09-13T14:56:53.091 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011098591999999999, 2024-09-13T14:56:53.091 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:56:53.091 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T14:56:53.091 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:56:53.091 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.091 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:56:53.092 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.044506+0000", 2024-09-13T14:56:53.092 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:56:53.092 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.092 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.092 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:56:53.092 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.044506+0000", 2024-09-13T14:56:53.092 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:56:53.092 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.092 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.092 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:56:53.092 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.044506+0000", 2024-09-13T14:56:53.092 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:56:53.092 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.093 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.093 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:56:53.093 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.044511+0000", 2024-09-13T14:56:53.093 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7810000000000002e-06 2024-09-13T14:56:53.093 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.093 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.093 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:56:53.093 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.044518+0000", 2024-09-13T14:56:53.093 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.0500000000000003e-06 2024-09-13T14:56:53.093 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.093 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.093 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:56:53.093 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.044524+0000", 2024-09-13T14:56:53.093 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.6450000000000002e-06 2024-09-13T14:56:53.094 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.094 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.094 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:56:53.094 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.050131+0000", 2024-09-13T14:56:53.094 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0056067119999999998 2024-09-13T14:56:53.094 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.094 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.094 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:56:53.094 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.050147+0000", 2024-09-13T14:56:53.094 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5917e-05 2024-09-13T14:56:53.094 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:56:53.094 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:56:53.094 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:56:53.095 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.095 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.095 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36738.0:114448 10.7 e467/436 10:eff61ebf:::benchmark_data_smithi102_377304_object8606:head v 467'7092, mlcod=467'7088)", 2024-09-13T14:56:53.095 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:56:53.049999+0000", 2024-09-13T14:56:53.095 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.0051193929999999999, 2024-09-13T14:56:53.095 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0056575940000000002, 2024-09-13T14:56:53.095 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:56:53.095 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T14:56:53.095 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:56:53.095 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.095 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:56:53.095 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.049999+0000", 2024-09-13T14:56:53.095 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:56:53.096 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.096 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.096 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:56:53.096 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.049999+0000", 2024-09-13T14:56:53.096 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:56:53.096 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.096 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.096 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:56:53.096 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.049998+0000", 2024-09-13T14:56:53.096 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:56:53.096 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.096 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.097 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:56:53.097 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.050005+0000", 2024-09-13T14:56:53.097 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1750000000000002e-06 2024-09-13T14:56:53.097 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.097 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.097 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:56:53.097 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.050016+0000", 2024-09-13T14:56:53.097 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1256e-05 2024-09-13T14:56:53.097 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.097 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.097 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:56:53.097 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.050023+0000", 2024-09-13T14:56:53.097 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.985e-06 2024-09-13T14:56:53.098 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.098 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.098 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:56:53.098 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.050379+0000", 2024-09-13T14:56:53.098 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000356308 2024-09-13T14:56:53.098 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:56:53.098 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:56:53.098 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:56:53.098 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:53.050400+0000", 2024-09-13T14:56:53.098 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0982999999999999e-05 2024-09-13T14:56:53.098 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:56:53.098 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:56:53.098 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:56:53.099 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:56:53.099 INFO:teuthology.orchestra.run.smithi195.stdout: ], 2024-09-13T14:56:53.099 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 9 2024-09-13T14:56:53.099 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:56:53.125 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:53.163 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:53.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:53.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:53.264 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:53.265 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:56:53.275 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 6, 7, 5, 2, 1] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T14:56:53.275 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:56:53.275 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-09-13T14:56:53.275 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T14:56:53.275 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T14:56:53.275 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-09-13T14:56:53.293 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:53.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:53.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:53.320 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:53 smithi195 ceph-mon[104262]: pgmap v1563: 109 pgs: 109 active+clean; 6.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 127 B/s rd, 18 MiB/s wr, 974 op/s 2024-09-13T14:56:53.395 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:53.462 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:53.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:53.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:53.564 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:53.592 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:53.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:53.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:53.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:53 smithi102 ceph-mon[110183]: pgmap v1563: 109 pgs: 109 active+clean; 6.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 127 B/s rd, 18 MiB/s wr, 974 op/s 2024-09-13T14:56:53.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:53 smithi102 ceph-mon[115269]: pgmap v1563: 109 pgs: 109 active+clean; 6.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 127 B/s rd, 18 MiB/s wr, 974 op/s 2024-09-13T14:56:53.695 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:53.719 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:53.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:53.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:53.822 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:53.850 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:53.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:53.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:53.952 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:53.959 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T14:56:53.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:53.974 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:53.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:54.076 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:54.077 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:56:54.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:54.133 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:54.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:54.234 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:54.254 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:54.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:54.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:54.356 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:54.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:54.393 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:54.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:54.494 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:54.538 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:54.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:54.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:54.640 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:54.679 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:54.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:54.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:54.781 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:54.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:54.810 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:54.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:54.911 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:54.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:54.948 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:54.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:55.049 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:55.088 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:55.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:55.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:55.190 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:55.219 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:55.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:55.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:55.321 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:55.362 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:55.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:55.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:55.445 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:55 smithi195 ceph-mon[104262]: pgmap v1564: 109 pgs: 109 active+clean; 6.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 127 B/s rd, 12 MiB/s wr, 1.09k op/s 2024-09-13T14:56:55.463 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:55.488 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:55.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:55.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:55.522 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:56:55.591 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:55.613 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:55.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:55.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:55.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:55 smithi102 ceph-mon[110183]: pgmap v1564: 109 pgs: 109 active+clean; 6.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 127 B/s rd, 12 MiB/s wr, 1.09k op/s 2024-09-13T14:56:55.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:55 smithi102 ceph-mon[115269]: pgmap v1564: 109 pgs: 109 active+clean; 6.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 127 B/s rd, 12 MiB/s wr, 1.09k op/s 2024-09-13T14:56:55.715 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:55.742 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:55.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:55.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:55.844 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:55.878 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:55.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:55.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:55.980 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:56.007 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:56.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:56.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:56.109 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:56.151 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:56.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:56.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:56.253 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:56.288 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:56:56.288 INFO:teuthology.orchestra.run.smithi195.stdout: "success": "" 2024-09-13T14:56:56.288 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:56:56.290 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:56.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:56.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:56.392 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:56.417 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:56:56.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:56.439 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:56.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:56.469 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:56:56.469 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:56:56.469 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:56:56 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:56:56] "GET /metrics HTTP/1.1" 200 36567 "" "Prometheus/2.51.0" 2024-09-13T14:56:56.540 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:56.576 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:56.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:56.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:56.678 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:56.733 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:56.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:56.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:56.836 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:56.859 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:56.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:56.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:56.961 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:56.998 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:56.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:56.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:57.100 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:57.128 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:57.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:57.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:57.230 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:57.253 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:57.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:57.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:57.258 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:57.355 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:57.383 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:57.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:57.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:57.451 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:57 smithi102 ceph-mon[110183]: pgmap v1565: 109 pgs: 109 active+clean; 6.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 106 B/s rd, 9.6 MiB/s wr, 995 op/s 2024-09-13T14:56:57.451 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:57 smithi102 ceph-mon[115269]: pgmap v1565: 109 pgs: 109 active+clean; 6.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 106 B/s rd, 9.6 MiB/s wr, 995 op/s 2024-09-13T14:56:57.485 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:57.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:57 smithi195 ceph-mon[104262]: pgmap v1565: 109 pgs: 109 active+clean; 6.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 106 B/s rd, 9.6 MiB/s wr, 995 op/s 2024-09-13T14:56:57.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:57.522 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:57.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:57.623 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:57.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:57.658 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:57.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:57.759 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:57.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:57.804 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:57.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:57.905 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:57.945 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:57.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:57.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:58.047 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:58.071 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:58.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:58.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:58.173 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:58.203 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:58.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:58.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:58.305 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:58.352 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:58.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:58.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:58.454 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:58.488 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:58.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:58.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:58.590 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:58.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:58.638 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:58.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:58.713 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:56:58.713 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-13T14:56:58.713 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-09-13T14:56:58.713 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-09-13T14:56:58.713 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:56:58.740 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:58.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:58.791 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:58.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:58.894 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:58.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:58.933 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:58.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:59.034 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:59.064 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:59.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:59.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:59.166 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:59.191 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:59.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:59.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:59.293 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:59.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:59.321 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:59.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:59.422 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:59.424 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:56:59 smithi195 ceph-mon[104262]: pgmap v1566: 109 pgs: 109 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 96 B/s rd, 989 KiB/s wr, 890 op/s 2024-09-13T14:56:59.451 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T14:56:59.461 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:59.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:59.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:59.563 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:59.603 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:56:59.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:59.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:59.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:56:59 smithi102 ceph-mon[110183]: pgmap v1566: 109 pgs: 109 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 96 B/s rd, 989 KiB/s wr, 890 op/s 2024-09-13T14:56:59.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:56:59 smithi102 ceph-mon[115269]: pgmap v1566: 109 pgs: 109 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 96 B/s rd, 989 KiB/s wr, 890 op/s 2024-09-13T14:56:59.696 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:56:59.704 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:59.736 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:59.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:59.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:59.838 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:59.860 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:56:59.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:59.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:59.929 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:56:59.929 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:56:59.962 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:59.968 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:56:59.968 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:57:00.003 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:57:00.003 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:57:00.006 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:00.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:00.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:00.036 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:57:00.036 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:57:00.071 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:57:00.072 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:57:00.104 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:57:00.105 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:57:00.107 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:00.136 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:57:00.136 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:57:00.141 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:00.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:00.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:00.167 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:57:00.168 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:57:00.243 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:00.270 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is unset 2024-09-13T14:57:00.289 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:00.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:00.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:00.391 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:00.419 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:00.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:00.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:00.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:00 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2504683411' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:57:00.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:00 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:57:00.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:00 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3940330852' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:57:00.521 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:00.551 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:00.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:00.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:00.653 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:00.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:00 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2504683411' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:57:00.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:00 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:57:00.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:00 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3940330852' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:57:00.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:00 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2504683411' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:57:00.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:00 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:57:00.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:00 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3940330852' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:57:00.688 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:00.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:00.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:00.790 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:00.831 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:00.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:00.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:00.933 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:00.957 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:00.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:00.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:01.059 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:01.083 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:01.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:01.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:01.185 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:01.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:01.220 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:01.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:01.322 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:01.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:01.356 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:01.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:01.457 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:01.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:01.484 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:01.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:01.585 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:01.624 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:01.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:01.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:01.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:01 smithi102 ceph-mon[110183]: pgmap v1567: 109 pgs: 109 active+clean; 5.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 85 B/s rd, 874 KiB/s wr, 959 op/s 2024-09-13T14:57:01.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:01 smithi102 ceph-mon[110183]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:57:01.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:01 smithi102 ceph-mon[110183]: Cluster is now healthy 2024-09-13T14:57:01.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:01 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:57:01.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:01 smithi102 ceph-mon[110183]: osdmap e468: 8 total, 8 up, 7 in 2024-09-13T14:57:01.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:01 smithi102 ceph-mon[115269]: pgmap v1567: 109 pgs: 109 active+clean; 5.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 85 B/s rd, 874 KiB/s wr, 959 op/s 2024-09-13T14:57:01.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:01 smithi102 ceph-mon[115269]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:57:01.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:01 smithi102 ceph-mon[115269]: Cluster is now healthy 2024-09-13T14:57:01.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:01 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:57:01.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:01 smithi102 ceph-mon[115269]: osdmap e468: 8 total, 8 up, 7 in 2024-09-13T14:57:01.726 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:01.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:01 smithi195 ceph-mon[104262]: pgmap v1567: 109 pgs: 109 active+clean; 5.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 85 B/s rd, 874 KiB/s wr, 959 op/s 2024-09-13T14:57:01.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:01 smithi195 ceph-mon[104262]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:57:01.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:01 smithi195 ceph-mon[104262]: Cluster is now healthy 2024-09-13T14:57:01.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:01 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:57:01.749 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:01 smithi195 ceph-mon[104262]: osdmap e468: 8 total, 8 up, 7 in 2024-09-13T14:57:01.755 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:01.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:01.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:01.857 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:01.894 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:01.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:01.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:01.996 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:02.022 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:02.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:02.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:02.123 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:02.156 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:02.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:02.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:02.257 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:02.292 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:02.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:02.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:02.394 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:02.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:02.424 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:02.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:02.525 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:02.550 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:02.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:02.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:02.581 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:57:02.581 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:57:02.641 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 6, 7, 5, 2, 1] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T14:57:02.641 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:57:02.641 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-13T14:57:02.642 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:57:02.652 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:02.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:02.681 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:02.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:02.782 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:02.809 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:02.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:02.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:02.911 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:02.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:02.969 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:02.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:03.011 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:03.070 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:03.093 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:03.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:03.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:03.195 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:03.239 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:03.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:03.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:03.303 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:57:03.308 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T14:57:03.340 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:03.408 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:03.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:03.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:03.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:03 smithi102 ceph-mon[110183]: pgmap v1569: 109 pgs: 109 active+clean; 5.6 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 931 op/s 2024-09-13T14:57:03.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:03 smithi102 ceph-mon[115269]: pgmap v1569: 109 pgs: 109 active+clean; 5.6 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 931 op/s 2024-09-13T14:57:03.510 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:03.536 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:03.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:03.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:03.559 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:57:03.638 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:03.662 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:03.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:03.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:03.687 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:03 smithi195 ceph-mon[104262]: pgmap v1569: 109 pgs: 109 active+clean; 5.6 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 931 op/s 2024-09-13T14:57:03.764 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:03.793 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:03.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:03.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:03.861 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:57:03.896 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:03.963 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:03.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:03.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:04.065 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:04.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:04.092 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:04.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:04.193 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:04.217 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:04.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:04.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:04.319 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:04.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:04.359 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:04.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:04.461 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:04.507 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:04.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:04.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:04.588 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:04.608 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:04.657 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:04.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:04.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:04.758 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:04.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:04.789 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:04.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:04.890 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:04.929 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:04.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:04.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:05.030 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:05.055 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:05.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:05.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:05.157 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:05.185 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:05.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:05.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:05.287 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:05.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:05.335 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:05.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:05.436 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:05.451 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:05 smithi195 ceph-mon[104262]: pgmap v1570: 109 pgs: 109 active+clean; 5.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 917 op/s 2024-09-13T14:57:05.472 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:05.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:05.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:05.481 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:57:05.482 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:57:05.557 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:05 smithi102 ceph-mon[110183]: pgmap v1570: 109 pgs: 109 active+clean; 5.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 917 op/s 2024-09-13T14:57:05.557 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:05 smithi102 ceph-mon[115269]: pgmap v1570: 109 pgs: 109 active+clean; 5.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 917 op/s 2024-09-13T14:57:05.574 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:05.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:05.610 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:05.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:05.711 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:05.736 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:05.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:05.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:05.775 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:57:05.777 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:57:05.837 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:05.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:05.864 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:05.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:05.965 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:05.990 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:05.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:05.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:06.055 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:06.093 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:06.145 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:06.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:06.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:06.247 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:06.279 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:06.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:06.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:06.281 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T14:57:06.380 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:06.420 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:06.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:06.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:06.522 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:06.549 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:06.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:06.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:06.592 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:57:06.651 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:06.701 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:06.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:06.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:06.722 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:57:06 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:57:06] "GET /metrics HTTP/1.1" 200 36567 "" "Prometheus/2.51.0" 2024-09-13T14:57:06.804 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:06.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:06.855 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:06.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:06.956 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:07.001 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:07.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:07.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:07.085 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":1570,"stamp":"2024-09-13T14:57:03.802506+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5861742589,"num_objects":89399,"num_object_clones":0,"num_object_copies":268197,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89399,"num_whiteouts":0,"num_read":5999,"num_read_kb":13012,"num_write":123217,"num_write_kb":6786322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5178,"num_bytes_recovered":333584752,"num_keys_recovered":30,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111371,"ondisk_log_size":111371,"up":327,"acting":327,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":327,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":23315836,"kb_used_data":17336948,"kb_used_omap":40,"kb_used_meta":5978647,"kb_avail":632900228,"statfs":{"total":671965249536,"available":648089833472,"internally_reserved":0,"allocated":17753034752,"data_stored":17736286903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41319,"internal_metadata":6122135193},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,4,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-601751552,"num_objects":-9182,"num_object_clones":0,"num_object_copies":-27546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9182,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003109"},"pg_stats":[{"pgid":"1.11","version":"90'149","reported_seq":443,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.208446+0000","last_change":"2024-09-13T14:56:12.490567+0000","last_active":"2024-09-13T14:56:47.208446+0000","last_peered":"2024-09-13T14:56:47.208446+0000","last_clean":"2024-09-13T14:56:47.208446+0000","last_became_active":"2024-09-13T14:56:12.490253+0000","last_became_peered":"2024-09-13T14:56:12.490253+0000","last_unstale":"2024-09-13T14:56:47.208446+0000","last_undegraded":"2024-09-13T14:56:47.208446+0000","last_fullsized":"2024-09-13T14:56:47.208446+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:38.688341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.688341+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:20.151605+0000","scrub_duration":0.0054331650000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":849,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-13T14:57:00.291691+0000","last_change":"2024-09-13T14:47:47.868783+0000","last_active":"2024-09-13T14:57:00.291691+0000","last_peered":"2024-09-13T14:57:00.291691+0000","last_clean":"2024-09-13T14:57:00.291691+0000","last_became_active":"2024-09-13T14:47:47.868478+0000","last_became_peered":"2024-09-13T14:47:47.868478+0000","last_unstale":"2024-09-13T14:57:00.291691+0000","last_undegraded":"2024-09-13T14:57:00.291691+0000","last_fullsized":"2024-09-13T14:57:00.291691+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:27:01.784142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":506,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.206524+0000","last_change":"2024-09-13T14:55:17.943558+0000","last_active":"2024-09-13T14:56:47.206524+0000","last_peered":"2024-09-13T14:56:47.206524+0000","last_clean":"2024-09-13T14:56:47.206524+0000","last_became_active":"2024-09-13T14:55:17.927162+0000","last_became_peered":"2024-09-13T14:55:17.927162+0000","last_unstale":"2024-09-13T14:56:47.206524+0000","last_undegraded":"2024-09-13T14:56:47.206524+0000","last_fullsized":"2024-09-13T14:56:47.206524+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:37:44.820559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":102,"num_keys_recovered":30,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"90'149","reported_seq":536,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-13T14:57:00.266114+0000","last_change":"2024-09-13T14:47:41.482552+0000","last_active":"2024-09-13T14:57:00.266114+0000","last_peered":"2024-09-13T14:57:00.266114+0000","last_clean":"2024-09-13T14:57:00.266114+0000","last_became_active":"2024-09-13T14:47:38.466659+0000","last_became_peered":"2024-09-13T14:47:38.466659+0000","last_unstale":"2024-09-13T14:57:00.266114+0000","last_undegraded":"2024-09-13T14:57:00.266114+0000","last_fullsized":"2024-09-13T14:57:00.266114+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:41.482364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:41.482364+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:50:49.239389+0000","scrub_duration":0.027843855000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":272,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.206700+0000","last_change":"2024-09-13T14:36:13.862181+0000","last_active":"2024-09-13T14:56:47.206700+0000","last_peered":"2024-09-13T14:56:47.206700+0000","last_clean":"2024-09-13T14:56:47.206700+0000","last_became_active":"2024-09-13T14:36:13.861873+0000","last_became_peered":"2024-09-13T14:36:13.861873+0000","last_unstale":"2024-09-13T14:56:47.206700+0000","last_undegraded":"2024-09-13T14:56:47.206700+0000","last_fullsized":"2024-09-13T14:56:47.206700+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:14.085700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":44,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.206224+0000","last_change":"2024-09-13T14:56:12.489695+0000","last_active":"2024-09-13T14:56:47.206224+0000","last_peered":"2024-09-13T14:56:47.206224+0000","last_clean":"2024-09-13T14:56:47.206224+0000","last_became_active":"2024-09-13T14:56:12.489302+0000","last_became_peered":"2024-09-13T14:56:12.489302+0000","last_unstale":"2024-09-13T14:56:47.206224+0000","last_undegraded":"2024-09-13T14:56:47.206224+0000","last_fullsized":"2024-09-13T14:56:47.206224+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:28:53.275229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"90'149","reported_seq":707,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.196392+0000","last_change":"2024-09-13T14:55:20.150382+0000","last_active":"2024-09-13T14:56:47.196392+0000","last_peered":"2024-09-13T14:56:47.196392+0000","last_clean":"2024-09-13T14:56:47.196392+0000","last_became_active":"2024-09-13T14:55:20.149422+0000","last_became_peered":"2024-09-13T14:55:20.149422+0000","last_unstale":"2024-09-13T14:56:47.196392+0000","last_undegraded":"2024-09-13T14:56:47.196392+0000","last_fullsized":"2024-09-13T14:56:47.196392+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:51.226748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:51.226748+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:10.410483+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":361,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-13T14:57:00.272709+0000","last_change":"2024-09-13T14:56:12.490265+0000","last_active":"2024-09-13T14:57:00.272709+0000","last_peered":"2024-09-13T14:57:00.272709+0000","last_clean":"2024-09-13T14:57:00.272709+0000","last_became_active":"2024-09-13T14:56:12.489937+0000","last_became_peered":"2024-09-13T14:56:12.489937+0000","last_unstale":"2024-09-13T14:57:00.272709+0000","last_undegraded":"2024-09-13T14:57:00.272709+0000","last_fullsized":"2024-09-13T14:57:00.272709+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:49.461532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":452,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.208175+0000","last_change":"2024-09-13T14:55:17.934695+0000","last_active":"2024-09-13T14:56:47.208175+0000","last_peered":"2024-09-13T14:56:47.208175+0000","last_clean":"2024-09-13T14:56:47.208175+0000","last_became_active":"2024-09-13T14:55:17.934165+0000","last_became_peered":"2024-09-13T14:55:17.934165+0000","last_unstale":"2024-09-13T14:56:47.208175+0000","last_undegraded":"2024-09-13T14:56:47.208175+0000","last_fullsized":"2024-09-13T14:56:47.208175+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:36:25.739706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"90'149","reported_seq":553,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-13T14:57:00.266465+0000","last_change":"2024-09-13T14:47:50.892667+0000","last_active":"2024-09-13T14:57:00.266465+0000","last_peered":"2024-09-13T14:57:00.266465+0000","last_clean":"2024-09-13T14:57:00.266465+0000","last_became_active":"2024-09-13T14:47:50.883312+0000","last_became_peered":"2024-09-13T14:47:50.883312+0000","last_unstale":"2024-09-13T14:57:00.266465+0000","last_undegraded":"2024-09-13T14:57:00.266465+0000","last_fullsized":"2024-09-13T14:57:00.266465+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:45.441819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:45.441819+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:39.138573+0000","scrub_duration":0.021368558999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":875,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-13T14:57:00.273897+0000","last_change":"2024-09-13T14:50:27.491166+0000","last_active":"2024-09-13T14:57:00.273897+0000","last_peered":"2024-09-13T14:57:00.273897+0000","last_clean":"2024-09-13T14:57:00.273897+0000","last_became_active":"2024-09-13T14:50:27.489905+0000","last_became_peered":"2024-09-13T14:50:27.489905+0000","last_unstale":"2024-09-13T14:57:00.273897+0000","last_undegraded":"2024-09-13T14:57:00.273897+0000","last_fullsized":"2024-09-13T14:57:00.273897+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:32.237904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":137,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.205693+0000","last_change":"2024-09-13T14:56:12.490197+0000","last_active":"2024-09-13T14:56:47.205693+0000","last_peered":"2024-09-13T14:56:47.205693+0000","last_clean":"2024-09-13T14:56:47.205693+0000","last_became_active":"2024-09-13T14:56:12.489554+0000","last_became_peered":"2024-09-13T14:56:12.489554+0000","last_unstale":"2024-09-13T14:56:47.205693+0000","last_undegraded":"2024-09-13T14:56:47.205693+0000","last_fullsized":"2024-09-13T14:56:47.205693+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:06.773211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"468'7561","reported_seq":8789,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-13T14:57:01.720133+0000","last_change":"2024-09-13T14:55:22.944506+0000","last_active":"2024-09-13T14:57:01.720133+0000","last_peered":"2024-09-13T14:57:01.720133+0000","last_clean":"2024-09-13T14:57:01.720133+0000","last_became_active":"2024-09-13T14:55:22.942014+0000","last_became_peered":"2024-09-13T14:55:22.942014+0000","last_unstale":"2024-09-13T14:57:01.720133+0000","last_undegraded":"2024-09-13T14:57:01.720133+0000","last_fullsized":"2024-09-13T14:57:01.720133+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":7561,"log_dups_size":0,"ondisk_log_size":7561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:00:33.167429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360775680,"num_objects":5505,"num_object_clones":0,"num_object_copies":16515,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7561,"num_write_kb":418112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":584,"num_bytes_recovered":38273024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"90'149","reported_seq":572,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.206777+0000","last_change":"2024-09-13T14:55:20.147437+0000","last_active":"2024-09-13T14:56:47.206777+0000","last_peered":"2024-09-13T14:56:47.206777+0000","last_clean":"2024-09-13T14:56:47.206777+0000","last_became_active":"2024-09-13T14:55:20.147197+0000","last_became_peered":"2024-09-13T14:55:20.147197+0000","last_unstale":"2024-09-13T14:56:47.206777+0000","last_undegraded":"2024-09-13T14:56:47.206777+0000","last_fullsized":"2024-09-13T14:56:47.206777+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:40.786610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:40.786610+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:36.008713+0000","scrub_duration":0.013725185000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":250,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-13T14:57:00.258095+0000","last_change":"2024-09-13T14:55:17.947508+0000","last_active":"2024-09-13T14:57:00.258095+0000","last_peered":"2024-09-13T14:57:00.258095+0000","last_clean":"2024-09-13T14:57:00.258095+0000","last_became_active":"2024-09-13T14:55:17.932604+0000","last_became_peered":"2024-09-13T14:55:17.932604+0000","last_unstale":"2024-09-13T14:57:00.258095+0000","last_undegraded":"2024-09-13T14:57:00.258095+0000","last_fullsized":"2024-09-13T14:57:00.258095+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:55.084612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":132,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":293,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.204950+0000","last_change":"2024-09-13T14:38:53.898907+0000","last_active":"2024-09-13T14:56:47.204950+0000","last_peered":"2024-09-13T14:56:47.204950+0000","last_clean":"2024-09-13T14:56:47.204950+0000","last_became_active":"2024-09-13T14:38:53.898572+0000","last_became_peered":"2024-09-13T14:38:53.898572+0000","last_unstale":"2024-09-13T14:56:47.204950+0000","last_undegraded":"2024-09-13T14:56:47.204950+0000","last_fullsized":"2024-09-13T14:56:47.204950+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:29.222150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"468'7655","reported_seq":7731,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-13T14:57:03.351354+0000","last_change":"2024-09-13T14:54:58.384717+0000","last_active":"2024-09-13T14:57:03.351354+0000","last_peered":"2024-09-13T14:57:03.351354+0000","last_clean":"2024-09-13T14:57:03.351354+0000","last_became_active":"2024-09-13T14:54:58.384225+0000","last_became_peered":"2024-09-13T14:54:58.384225+0000","last_unstale":"2024-09-13T14:57:03.351354+0000","last_undegraded":"2024-09-13T14:57:03.351354+0000","last_fullsized":"2024-09-13T14:57:03.351354+0000","mapping_epoch":436,"log_start":"458'1800","ondisk_log_start":"458'1800","created":436,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":5855,"log_dups_size":0,"ondisk_log_size":5855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:25:22.716676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":357498880,"num_objects":5455,"num_object_clones":0,"num_object_copies":16365,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7655,"num_write_kb":419520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"90'149","reported_seq":722,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-13T14:57:00.291550+0000","last_change":"2024-09-13T14:47:38.466101+0000","last_active":"2024-09-13T14:57:00.291550+0000","last_peered":"2024-09-13T14:57:00.291550+0000","last_clean":"2024-09-13T14:57:00.291550+0000","last_became_active":"2024-09-13T14:47:38.465607+0000","last_became_peered":"2024-09-13T14:47:38.465607+0000","last_unstale":"2024-09-13T14:57:00.291550+0000","last_undegraded":"2024-09-13T14:57:00.291550+0000","last_fullsized":"2024-09-13T14:57:00.291550+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:32.060817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:32.060817+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:55.362724+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":263,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-13T14:57:00.257818+0000","last_change":"2024-09-13T14:55:17.949010+0000","last_active":"2024-09-13T14:57:00.257818+0000","last_peered":"2024-09-13T14:57:00.257818+0000","last_clean":"2024-09-13T14:57:00.257818+0000","last_became_active":"2024-09-13T14:55:17.932712+0000","last_became_peered":"2024-09-13T14:55:17.932712+0000","last_unstale":"2024-09-13T14:57:00.257818+0000","last_undegraded":"2024-09-13T14:57:00.257818+0000","last_fullsized":"2024-09-13T14:57:00.257818+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:46:43.390958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":410,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.191631+0000","last_change":"2024-09-13T14:55:17.929723+0000","last_active":"2024-09-13T14:56:47.191631+0000","last_peered":"2024-09-13T14:56:47.191631+0000","last_clean":"2024-09-13T14:56:47.191631+0000","last_became_active":"2024-09-13T14:55:17.929131+0000","last_became_peered":"2024-09-13T14:55:17.929131+0000","last_unstale":"2024-09-13T14:56:47.191631+0000","last_undegraded":"2024-09-13T14:56:47.191631+0000","last_fullsized":"2024-09-13T14:56:47.191631+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:27:15.828973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.4","version":"468'7754","reported_seq":9090,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-13T14:57:03.408355+0000","last_change":"2024-09-13T14:56:12.495568+0000","last_active":"2024-09-13T14:57:03.408355+0000","last_peered":"2024-09-13T14:57:03.408355+0000","last_clean":"2024-09-13T14:57:03.408355+0000","last_became_active":"2024-09-13T14:56:12.490664+0000","last_became_peered":"2024-09-13T14:56:12.490664+0000","last_unstale":"2024-09-13T14:57:03.408355+0000","last_undegraded":"2024-09-13T14:57:03.408355+0000","last_fullsized":"2024-09-13T14:57:03.408355+0000","mapping_epoch":462,"log_start":"458'1900","ondisk_log_start":"458'1900","created":436,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":5854,"log_dups_size":0,"ondisk_log_size":5854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:17:29.226407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":361234432,"num_objects":5512,"num_object_clones":0,"num_object_copies":16536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7754,"num_write_kb":424512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":623,"num_bytes_recovered":40828928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.f","version":"90'149","reported_seq":597,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.206908+0000","last_change":"2024-09-13T14:55:20.140396+0000","last_active":"2024-09-13T14:56:47.206908+0000","last_peered":"2024-09-13T14:56:47.206908+0000","last_clean":"2024-09-13T14:56:47.206908+0000","last_became_active":"2024-09-13T14:55:20.140087+0000","last_became_peered":"2024-09-13T14:55:20.140087+0000","last_unstale":"2024-09-13T14:56:47.206908+0000","last_undegraded":"2024-09-13T14:56:47.206908+0000","last_fullsized":"2024-09-13T14:56:47.206908+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:34.091782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:34.091782+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:46:05.540133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":474,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.207047+0000","last_change":"2024-09-13T14:37:49.611774+0000","last_active":"2024-09-13T14:56:47.207047+0000","last_peered":"2024-09-13T14:56:47.207047+0000","last_clean":"2024-09-13T14:56:47.207047+0000","last_became_active":"2024-09-13T14:37:49.611088+0000","last_became_peered":"2024-09-13T14:37:49.611088+0000","last_unstale":"2024-09-13T14:56:47.207047+0000","last_undegraded":"2024-09-13T14:56:47.207047+0000","last_fullsized":"2024-09-13T14:56:47.207047+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:08.933213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":276,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.193648+0000","last_change":"2024-09-13T14:47:47.865523+0000","last_active":"2024-09-13T14:56:47.193648+0000","last_peered":"2024-09-13T14:56:47.193648+0000","last_clean":"2024-09-13T14:56:47.193648+0000","last_became_active":"2024-09-13T14:47:47.865180+0000","last_became_peered":"2024-09-13T14:47:47.865180+0000","last_unstale":"2024-09-13T14:56:47.193648+0000","last_undegraded":"2024-09-13T14:56:47.193648+0000","last_fullsized":"2024-09-13T14:56:47.193648+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:39:35.479251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.5","version":"468'7465","reported_seq":7540,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-13T14:57:01.570150+0000","last_change":"2024-09-13T14:54:57.846849+0000","last_active":"2024-09-13T14:57:01.570150+0000","last_peered":"2024-09-13T14:57:01.570150+0000","last_clean":"2024-09-13T14:57:01.570150+0000","last_became_active":"2024-09-13T14:54:57.846557+0000","last_became_peered":"2024-09-13T14:54:57.846557+0000","last_unstale":"2024-09-13T14:57:01.570150+0000","last_undegraded":"2024-09-13T14:57:01.570150+0000","last_fullsized":"2024-09-13T14:57:01.570150+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":7465,"log_dups_size":0,"ondisk_log_size":7465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:49.590084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360513536,"num_objects":5501,"num_object_clones":0,"num_object_copies":16503,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7465,"num_write_kb":414912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"90'149","reported_seq":482,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.207092+0000","last_change":"2024-09-13T14:55:20.145243+0000","last_active":"2024-09-13T14:56:47.207092+0000","last_peered":"2024-09-13T14:56:47.207092+0000","last_clean":"2024-09-13T14:56:47.207092+0000","last_became_active":"2024-09-13T14:55:20.145001+0000","last_became_peered":"2024-09-13T14:55:20.145001+0000","last_unstale":"2024-09-13T14:56:47.207092+0000","last_undegraded":"2024-09-13T14:56:47.207092+0000","last_fullsized":"2024-09-13T14:56:47.207092+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:30.526999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:30.526999+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:11:56.231618+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":284,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.205591+0000","last_change":"2024-09-13T14:37:49.168003+0000","last_active":"2024-09-13T14:56:47.205591+0000","last_peered":"2024-09-13T14:56:47.205591+0000","last_clean":"2024-09-13T14:56:47.205591+0000","last_became_active":"2024-09-13T14:37:49.167638+0000","last_became_peered":"2024-09-13T14:37:49.167638+0000","last_unstale":"2024-09-13T14:56:47.205591+0000","last_undegraded":"2024-09-13T14:56:47.205591+0000","last_fullsized":"2024-09-13T14:56:47.205591+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:29.051981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":438,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.196179+0000","last_change":"2024-09-13T14:56:12.491208+0000","last_active":"2024-09-13T14:56:47.196179+0000","last_peered":"2024-09-13T14:56:47.196179+0000","last_clean":"2024-09-13T14:56:47.196179+0000","last_became_active":"2024-09-13T14:56:12.490943+0000","last_became_peered":"2024-09-13T14:56:12.490943+0000","last_unstale":"2024-09-13T14:56:47.196179+0000","last_undegraded":"2024-09-13T14:56:47.196179+0000","last_fullsized":"2024-09-13T14:56:47.196179+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:31:57.892194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"467'7426","reported_seq":8641,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:59.478735+0000","last_change":"2024-09-13T14:55:37.154924+0000","last_active":"2024-09-13T14:56:59.478735+0000","last_peered":"2024-09-13T14:56:59.478735+0000","last_clean":"2024-09-13T14:56:59.478735+0000","last_became_active":"2024-09-13T14:55:37.153166+0000","last_became_peered":"2024-09-13T14:55:37.153166+0000","last_unstale":"2024-09-13T14:56:59.478735+0000","last_undegraded":"2024-09-13T14:56:59.478735+0000","last_fullsized":"2024-09-13T14:56:59.478735+0000","mapping_epoch":459,"log_start":"456'1600","ondisk_log_start":"456'1600","created":436,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":5826,"log_dups_size":0,"ondisk_log_size":5826,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:48:15.748240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":374210560,"num_objects":5710,"num_object_clones":0,"num_object_copies":17130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7426,"num_write_kb":420352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":565,"num_bytes_recovered":37027840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"90'149","reported_seq":645,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.208064+0000","last_change":"2024-09-13T14:46:55.355650+0000","last_active":"2024-09-13T14:56:47.208064+0000","last_peered":"2024-09-13T14:56:47.208064+0000","last_clean":"2024-09-13T14:56:47.208064+0000","last_became_active":"2024-09-13T14:46:48.150338+0000","last_became_peered":"2024-09-13T14:46:48.150338+0000","last_unstale":"2024-09-13T14:56:47.208064+0000","last_undegraded":"2024-09-13T14:56:47.208064+0000","last_fullsized":"2024-09-13T14:56:47.208064+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:46:55.355521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:46:55.355521+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:30:53.006278+0000","scrub_duration":0.002025448,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":835,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-13T14:57:00.272152+0000","last_change":"2024-09-13T14:55:17.944596+0000","last_active":"2024-09-13T14:57:00.272152+0000","last_peered":"2024-09-13T14:57:00.272152+0000","last_clean":"2024-09-13T14:57:00.272152+0000","last_became_active":"2024-09-13T14:55:17.935767+0000","last_became_peered":"2024-09-13T14:55:17.935767+0000","last_unstale":"2024-09-13T14:57:00.272152+0000","last_undegraded":"2024-09-13T14:57:00.272152+0000","last_fullsized":"2024-09-13T14:57:00.272152+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:22:19.619639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":364,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-13T14:57:00.274978+0000","last_change":"2024-09-13T14:55:17.933005+0000","last_active":"2024-09-13T14:57:00.274978+0000","last_peered":"2024-09-13T14:57:00.274978+0000","last_clean":"2024-09-13T14:57:00.274978+0000","last_became_active":"2024-09-13T14:55:17.931575+0000","last_became_peered":"2024-09-13T14:55:17.931575+0000","last_unstale":"2024-09-13T14:57:00.274978+0000","last_undegraded":"2024-09-13T14:57:00.274978+0000","last_fullsized":"2024-09-13T14:57:00.274978+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:50:20.352970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"468'7682","reported_seq":7758,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-13T14:57:01.538999+0000","last_change":"2024-09-13T14:54:57.846161+0000","last_active":"2024-09-13T14:57:01.538999+0000","last_peered":"2024-09-13T14:57:01.538999+0000","last_clean":"2024-09-13T14:57:01.538999+0000","last_became_active":"2024-09-13T14:54:57.845348+0000","last_became_peered":"2024-09-13T14:54:57.845348+0000","last_unstale":"2024-09-13T14:57:01.538999+0000","last_undegraded":"2024-09-13T14:57:01.538999+0000","last_fullsized":"2024-09-13T14:57:01.538999+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":7682,"log_dups_size":0,"ondisk_log_size":7682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:07:01.751407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":366739456,"num_objects":5596,"num_object_clones":0,"num_object_copies":16788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7682,"num_write_kb":424896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"90'149","reported_seq":548,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.204750+0000","last_change":"2024-09-13T14:46:54.393663+0000","last_active":"2024-09-13T14:56:47.204750+0000","last_peered":"2024-09-13T14:56:47.204750+0000","last_clean":"2024-09-13T14:56:47.204750+0000","last_became_active":"2024-09-13T14:46:48.149004+0000","last_became_peered":"2024-09-13T14:46:48.149004+0000","last_unstale":"2024-09-13T14:56:47.204750+0000","last_undegraded":"2024-09-13T14:56:47.204750+0000","last_fullsized":"2024-09-13T14:56:47.204750+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:46:54.393204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:46:54.393204+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:32:40.003989+0000","scrub_duration":0.015172223,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":319,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.193507+0000","last_change":"2024-09-13T14:55:17.935369+0000","last_active":"2024-09-13T14:56:47.193507+0000","last_peered":"2024-09-13T14:56:47.193507+0000","last_clean":"2024-09-13T14:56:47.193507+0000","last_became_active":"2024-09-13T14:55:17.930815+0000","last_became_peered":"2024-09-13T14:55:17.930815+0000","last_unstale":"2024-09-13T14:56:47.193507+0000","last_undegraded":"2024-09-13T14:56:47.193507+0000","last_fullsized":"2024-09-13T14:56:47.193507+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:59:31.247160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":450,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-13T14:57:00.266394+0000","last_change":"2024-09-13T14:55:17.932243+0000","last_active":"2024-09-13T14:57:00.266394+0000","last_peered":"2024-09-13T14:57:00.266394+0000","last_clean":"2024-09-13T14:57:00.266394+0000","last_became_active":"2024-09-13T14:55:17.931961+0000","last_became_peered":"2024-09-13T14:55:17.931961+0000","last_unstale":"2024-09-13T14:57:00.266394+0000","last_undegraded":"2024-09-13T14:57:00.266394+0000","last_fullsized":"2024-09-13T14:57:00.266394+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:23:52.680942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.0","version":"467'7548","reported_seq":8943,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:59.466490+0000","last_change":"2024-09-13T14:55:24.959843+0000","last_active":"2024-09-13T14:56:59.466490+0000","last_peered":"2024-09-13T14:56:59.466490+0000","last_clean":"2024-09-13T14:56:59.466490+0000","last_became_active":"2024-09-13T14:55:24.957491+0000","last_became_peered":"2024-09-13T14:55:24.957491+0000","last_unstale":"2024-09-13T14:56:59.466490+0000","last_undegraded":"2024-09-13T14:56:59.466490+0000","last_fullsized":"2024-09-13T14:56:59.466490+0000","mapping_epoch":448,"log_start":"456'1700","ondisk_log_start":"456'1700","created":436,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":5848,"log_dups_size":0,"ondisk_log_size":5848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:09:19.929604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368181248,"num_objects":5618,"num_object_clones":0,"num_object_copies":16854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7548,"num_write_kb":421312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":654,"num_bytes_recovered":42860544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"90'149","reported_seq":711,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.196134+0000","last_change":"2024-09-13T14:47:50.891055+0000","last_active":"2024-09-13T14:56:47.196134+0000","last_peered":"2024-09-13T14:56:47.196134+0000","last_clean":"2024-09-13T14:56:47.196134+0000","last_became_active":"2024-09-13T14:47:50.890368+0000","last_became_peered":"2024-09-13T14:47:50.890368+0000","last_unstale":"2024-09-13T14:56:47.196134+0000","last_undegraded":"2024-09-13T14:56:47.196134+0000","last_fullsized":"2024-09-13T14:56:47.196134+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:39.645908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:39.645908+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:29.800728+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":370,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-13T14:57:00.266452+0000","last_change":"2024-09-13T14:55:17.932199+0000","last_active":"2024-09-13T14:57:00.266452+0000","last_peered":"2024-09-13T14:57:00.266452+0000","last_clean":"2024-09-13T14:57:00.266452+0000","last_became_active":"2024-09-13T14:55:17.931873+0000","last_became_peered":"2024-09-13T14:55:17.931873+0000","last_unstale":"2024-09-13T14:57:00.266452+0000","last_undegraded":"2024-09-13T14:57:00.266452+0000","last_fullsized":"2024-09-13T14:57:00.266452+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:02:18.004113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":395,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.203661+0000","last_change":"2024-09-13T14:55:17.931269+0000","last_active":"2024-09-13T14:56:47.203661+0000","last_peered":"2024-09-13T14:56:47.203661+0000","last_clean":"2024-09-13T14:56:47.203661+0000","last_became_active":"2024-09-13T14:55:17.930909+0000","last_became_peered":"2024-09-13T14:55:17.930909+0000","last_unstale":"2024-09-13T14:56:47.203661+0000","last_undegraded":"2024-09-13T14:56:47.203661+0000","last_fullsized":"2024-09-13T14:56:47.203661+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:47.592526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"468'7968","reported_seq":9363,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-13T14:57:01.789145+0000","last_change":"2024-09-13T14:55:35.136618+0000","last_active":"2024-09-13T14:57:01.789145+0000","last_peered":"2024-09-13T14:57:01.789145+0000","last_clean":"2024-09-13T14:57:01.789145+0000","last_became_active":"2024-09-13T14:55:35.134043+0000","last_became_peered":"2024-09-13T14:55:35.134043+0000","last_unstale":"2024-09-13T14:57:01.789145+0000","last_undegraded":"2024-09-13T14:57:01.789145+0000","last_fullsized":"2024-09-13T14:57:01.789145+0000","mapping_epoch":457,"log_start":"450'1300","ondisk_log_start":"450'1300","created":436,"last_epoch_clean":458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":6668,"log_dups_size":0,"ondisk_log_size":6668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:55.445002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376569856,"num_objects":5746,"num_object_clones":0,"num_object_copies":17238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7968,"num_write_kb":438848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":667,"num_bytes_recovered":43712512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"90'149","reported_seq":548,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.205443+0000","last_change":"2024-09-13T14:46:52.334715+0000","last_active":"2024-09-13T14:56:47.205443+0000","last_peered":"2024-09-13T14:56:47.205443+0000","last_clean":"2024-09-13T14:56:47.205443+0000","last_became_active":"2024-09-13T14:46:48.158685+0000","last_became_peered":"2024-09-13T14:46:48.158685+0000","last_unstale":"2024-09-13T14:56:47.205443+0000","last_undegraded":"2024-09-13T14:56:47.205443+0000","last_fullsized":"2024-09-13T14:56:47.205443+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:46:52.334301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:46:52.334301+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:46:03.579759+0000","scrub_duration":0.0050940330000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":849,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-13T14:57:00.271984+0000","last_change":"2024-09-13T14:47:47.861249+0000","last_active":"2024-09-13T14:57:00.271984+0000","last_peered":"2024-09-13T14:57:00.271984+0000","last_clean":"2024-09-13T14:57:00.271984+0000","last_became_active":"2024-09-13T14:47:47.860873+0000","last_became_peered":"2024-09-13T14:47:47.860873+0000","last_unstale":"2024-09-13T14:57:00.271984+0000","last_undegraded":"2024-09-13T14:57:00.271984+0000","last_fullsized":"2024-09-13T14:57:00.271984+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:37:50.709437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":283,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.191941+0000","last_change":"2024-09-13T14:55:17.933070+0000","last_active":"2024-09-13T14:56:47.191941+0000","last_peered":"2024-09-13T14:56:47.191941+0000","last_clean":"2024-09-13T14:56:47.191941+0000","last_became_active":"2024-09-13T14:55:17.931232+0000","last_became_peered":"2024-09-13T14:55:17.931232+0000","last_unstale":"2024-09-13T14:56:47.191941+0000","last_undegraded":"2024-09-13T14:56:47.191941+0000","last_fullsized":"2024-09-13T14:56:47.191941+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:22:46.719483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"467'7453","reported_seq":7526,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:59.506333+0000","last_change":"2024-09-13T14:54:58.384133+0000","last_active":"2024-09-13T14:56:59.506333+0000","last_peered":"2024-09-13T14:56:59.506333+0000","last_clean":"2024-09-13T14:56:59.506333+0000","last_became_active":"2024-09-13T14:54:58.383823+0000","last_became_peered":"2024-09-13T14:54:58.383823+0000","last_unstale":"2024-09-13T14:56:59.506333+0000","last_undegraded":"2024-09-13T14:56:59.506333+0000","last_fullsized":"2024-09-13T14:56:59.506333+0000","mapping_epoch":436,"log_start":"456'1600","ondisk_log_start":"456'1600","created":436,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":5853,"log_dups_size":0,"ondisk_log_size":5853,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:30.724982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369164288,"num_objects":5633,"num_object_clones":0,"num_object_copies":16899,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7453,"num_write_kb":418752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":651,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.206796+0000","last_change":"2024-09-13T14:47:31.684319+0000","last_active":"2024-09-13T14:56:47.206796+0000","last_peered":"2024-09-13T14:56:47.206796+0000","last_clean":"2024-09-13T14:56:47.206796+0000","last_became_active":"2024-09-13T14:47:29.415658+0000","last_became_peered":"2024-09-13T14:47:29.415658+0000","last_unstale":"2024-09-13T14:56:47.206796+0000","last_undegraded":"2024-09-13T14:56:47.206796+0000","last_fullsized":"2024-09-13T14:56:47.206796+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:31.681433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:31.681433+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:29:34.663513+0000","scrub_duration":0.0059601139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":779,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.205648+0000","last_change":"2024-09-13T14:46:04.780718+0000","last_active":"2024-09-13T14:56:47.205648+0000","last_peered":"2024-09-13T14:56:47.205648+0000","last_clean":"2024-09-13T14:56:47.205648+0000","last_became_active":"2024-09-13T14:46:04.780120+0000","last_became_peered":"2024-09-13T14:46:04.780120+0000","last_unstale":"2024-09-13T14:56:47.205648+0000","last_undegraded":"2024-09-13T14:56:47.205648+0000","last_fullsized":"2024-09-13T14:56:47.205648+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:57:14.475888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":777,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-13T14:57:00.273206+0000","last_change":"2024-09-13T14:37:49.160719+0000","last_active":"2024-09-13T14:57:00.273206+0000","last_peered":"2024-09-13T14:57:00.273206+0000","last_clean":"2024-09-13T14:57:00.273206+0000","last_became_active":"2024-09-13T14:37:49.160401+0000","last_became_peered":"2024-09-13T14:37:49.160401+0000","last_unstale":"2024-09-13T14:57:00.273206+0000","last_undegraded":"2024-09-13T14:57:00.273206+0000","last_fullsized":"2024-09-13T14:57:00.273206+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:21.737945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":110,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.206967+0000","last_change":"2024-09-13T14:55:17.933431+0000","last_active":"2024-09-13T14:56:47.206967+0000","last_peered":"2024-09-13T14:56:47.206967+0000","last_clean":"2024-09-13T14:56:47.206967+0000","last_became_active":"2024-09-13T14:55:17.932711+0000","last_became_peered":"2024-09-13T14:55:17.932711+0000","last_unstale":"2024-09-13T14:56:47.206967+0000","last_undegraded":"2024-09-13T14:56:47.206967+0000","last_fullsized":"2024-09-13T14:56:47.206967+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:47.712045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"468'7496","reported_seq":7583,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-13T14:57:01.511280+0000","last_change":"2024-09-13T14:56:12.516271+0000","last_active":"2024-09-13T14:57:01.511280+0000","last_peered":"2024-09-13T14:57:01.511280+0000","last_clean":"2024-09-13T14:57:01.511280+0000","last_became_active":"2024-09-13T14:56:12.494537+0000","last_became_peered":"2024-09-13T14:56:12.494537+0000","last_unstale":"2024-09-13T14:57:01.511280+0000","last_undegraded":"2024-09-13T14:57:01.511280+0000","last_fullsized":"2024-09-13T14:57:01.511280+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":7496,"log_dups_size":0,"ondisk_log_size":7496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:07:34.989938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362020864,"num_objects":5524,"num_object_clones":0,"num_object_copies":16572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7496,"num_write_kb":416640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"90'149","reported_seq":715,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.196254+0000","last_change":"2024-09-13T14:55:20.148734+0000","last_active":"2024-09-13T14:56:47.196254+0000","last_peered":"2024-09-13T14:56:47.196254+0000","last_clean":"2024-09-13T14:56:47.196254+0000","last_became_active":"2024-09-13T14:55:20.147918+0000","last_became_peered":"2024-09-13T14:55:20.147918+0000","last_unstale":"2024-09-13T14:56:47.196254+0000","last_undegraded":"2024-09-13T14:56:47.196254+0000","last_fullsized":"2024-09-13T14:56:47.196254+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:34.603316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:34.603316+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:57.974517+0000","scrub_duration":0.0057131120000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":721,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.206716+0000","last_change":"2024-09-13T14:47:47.892973+0000","last_active":"2024-09-13T14:56:47.206716+0000","last_peered":"2024-09-13T14:56:47.206716+0000","last_clean":"2024-09-13T14:56:47.206716+0000","last_became_active":"2024-09-13T14:47:47.866986+0000","last_became_peered":"2024-09-13T14:47:47.866986+0000","last_unstale":"2024-09-13T14:56:47.206716+0000","last_undegraded":"2024-09-13T14:56:47.206716+0000","last_fullsized":"2024-09-13T14:56:47.206716+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:07.353290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":434,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-13T14:57:00.272209+0000","last_change":"2024-09-13T14:56:12.490871+0000","last_active":"2024-09-13T14:57:00.272209+0000","last_peered":"2024-09-13T14:57:00.272209+0000","last_clean":"2024-09-13T14:57:00.272209+0000","last_became_active":"2024-09-13T14:56:12.490535+0000","last_became_peered":"2024-09-13T14:56:12.490535+0000","last_unstale":"2024-09-13T14:57:00.272209+0000","last_undegraded":"2024-09-13T14:57:00.272209+0000","last_fullsized":"2024-09-13T14:57:00.272209+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:49.429179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":232,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.203346+0000","last_change":"2024-09-13T14:56:12.490229+0000","last_active":"2024-09-13T14:56:47.203346+0000","last_peered":"2024-09-13T14:56:47.203346+0000","last_clean":"2024-09-13T14:56:47.203346+0000","last_became_active":"2024-09-13T14:56:12.489512+0000","last_became_peered":"2024-09-13T14:56:12.489512+0000","last_unstale":"2024-09-13T14:56:47.203346+0000","last_undegraded":"2024-09-13T14:56:47.203346+0000","last_fullsized":"2024-09-13T14:56:47.203346+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:26:10.594549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.9","version":"468'7717","reported_seq":7758,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-13T14:57:01.672169+0000","last_change":"2024-09-13T14:54:57.844803+0000","last_active":"2024-09-13T14:57:01.672169+0000","last_peered":"2024-09-13T14:57:01.672169+0000","last_clean":"2024-09-13T14:57:01.672169+0000","last_became_active":"2024-09-13T14:54:57.844499+0000","last_became_peered":"2024-09-13T14:54:57.844499+0000","last_unstale":"2024-09-13T14:57:01.672169+0000","last_undegraded":"2024-09-13T14:57:01.672169+0000","last_fullsized":"2024-09-13T14:57:01.672169+0000","mapping_epoch":436,"log_start":"440'200","ondisk_log_start":"440'200","created":436,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":7517,"log_dups_size":0,"ondisk_log_size":7517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:46.617173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370737152,"num_objects":5657,"num_object_clones":0,"num_object_copies":16971,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7717,"num_write_kb":427968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"90'149","reported_seq":684,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.206143+0000","last_change":"2024-09-13T14:55:20.134578+0000","last_active":"2024-09-13T14:56:47.206143+0000","last_peered":"2024-09-13T14:56:47.206143+0000","last_clean":"2024-09-13T14:56:47.206143+0000","last_became_active":"2024-09-13T14:55:20.134120+0000","last_became_peered":"2024-09-13T14:55:20.134120+0000","last_unstale":"2024-09-13T14:56:47.206143+0000","last_undegraded":"2024-09-13T14:56:47.206143+0000","last_fullsized":"2024-09-13T14:56:47.206143+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:29.629342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:29.629342+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:38:13.399024+0000","scrub_duration":0.0099290990000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":761,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-13T14:57:00.266357+0000","last_change":"2024-09-13T14:46:04.782955+0000","last_active":"2024-09-13T14:57:00.266357+0000","last_peered":"2024-09-13T14:57:00.266357+0000","last_clean":"2024-09-13T14:57:00.266357+0000","last_became_active":"2024-09-13T14:46:04.782250+0000","last_became_peered":"2024-09-13T14:46:04.782250+0000","last_unstale":"2024-09-13T14:57:00.266357+0000","last_undegraded":"2024-09-13T14:57:00.266357+0000","last_fullsized":"2024-09-13T14:57:00.266357+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:24.596922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":276,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.192328+0000","last_change":"2024-09-13T14:47:47.866684+0000","last_active":"2024-09-13T14:56:47.192328+0000","last_peered":"2024-09-13T14:56:47.192328+0000","last_clean":"2024-09-13T14:56:47.192328+0000","last_became_active":"2024-09-13T14:47:47.866400+0000","last_became_peered":"2024-09-13T14:47:47.866400+0000","last_unstale":"2024-09-13T14:56:47.192328+0000","last_undegraded":"2024-09-13T14:56:47.192328+0000","last_fullsized":"2024-09-13T14:56:47.192328+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:08:34.289658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":276,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.191929+0000","last_change":"2024-09-13T14:47:47.870218+0000","last_active":"2024-09-13T14:56:47.191929+0000","last_peered":"2024-09-13T14:56:47.191929+0000","last_clean":"2024-09-13T14:56:47.191929+0000","last_became_active":"2024-09-13T14:47:47.869907+0000","last_became_peered":"2024-09-13T14:47:47.869907+0000","last_unstale":"2024-09-13T14:56:47.191929+0000","last_undegraded":"2024-09-13T14:56:47.191929+0000","last_fullsized":"2024-09-13T14:56:47.191929+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:41:49.141441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"468'7835","reported_seq":7911,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-13T14:57:01.703888+0000","last_change":"2024-09-13T14:56:12.497755+0000","last_active":"2024-09-13T14:57:01.703888+0000","last_peered":"2024-09-13T14:57:01.703888+0000","last_clean":"2024-09-13T14:57:01.703888+0000","last_became_active":"2024-09-13T14:56:12.489758+0000","last_became_peered":"2024-09-13T14:56:12.489758+0000","last_unstale":"2024-09-13T14:57:01.703888+0000","last_undegraded":"2024-09-13T14:57:01.703888+0000","last_fullsized":"2024-09-13T14:57:01.703888+0000","mapping_epoch":462,"log_start":"440'300","ondisk_log_start":"440'300","created":436,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":7535,"log_dups_size":0,"ondisk_log_size":7535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:22:53.808045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":379387904,"num_objects":5789,"num_object_clones":0,"num_object_copies":17367,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7835,"num_write_kb":435968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"90'149","reported_seq":569,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.208504+0000","last_change":"2024-09-13T14:47:32.275753+0000","last_active":"2024-09-13T14:56:47.208504+0000","last_peered":"2024-09-13T14:56:47.208504+0000","last_clean":"2024-09-13T14:56:47.208504+0000","last_became_active":"2024-09-13T14:47:29.401066+0000","last_became_peered":"2024-09-13T14:47:29.401066+0000","last_unstale":"2024-09-13T14:56:47.208504+0000","last_undegraded":"2024-09-13T14:56:47.208504+0000","last_fullsized":"2024-09-13T14:56:47.208504+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:32.275562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:32.275562+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:43:41.056775+0000","scrub_duration":0.0054674260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":1224,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.206351+0000","last_change":"2024-09-13T14:47:50.892842+0000","last_active":"2024-09-13T14:56:47.206351+0000","last_peered":"2024-09-13T14:56:47.206351+0000","last_clean":"2024-09-13T14:56:47.206351+0000","last_became_active":"2024-09-13T14:47:50.892237+0000","last_became_peered":"2024-09-13T14:47:50.892237+0000","last_unstale":"2024-09-13T14:56:47.206351+0000","last_undegraded":"2024-09-13T14:56:47.206351+0000","last_fullsized":"2024-09-13T14:56:47.206351+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:20:05.796762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":8388646,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":262,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.195180+0000","last_change":"2024-09-13T14:56:12.491483+0000","last_active":"2024-09-13T14:56:47.195180+0000","last_peered":"2024-09-13T14:56:47.195180+0000","last_clean":"2024-09-13T14:56:47.195180+0000","last_became_active":"2024-09-13T14:56:12.491191+0000","last_became_peered":"2024-09-13T14:56:12.491191+0000","last_unstale":"2024-09-13T14:56:47.195180+0000","last_undegraded":"2024-09-13T14:56:47.195180+0000","last_fullsized":"2024-09-13T14:56:47.195180+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:54:11.122787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":276,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.196164+0000","last_change":"2024-09-13T14:47:47.877272+0000","last_active":"2024-09-13T14:56:47.196164+0000","last_peered":"2024-09-13T14:56:47.196164+0000","last_clean":"2024-09-13T14:56:47.196164+0000","last_became_active":"2024-09-13T14:47:47.877003+0000","last_became_peered":"2024-09-13T14:47:47.877003+0000","last_unstale":"2024-09-13T14:56:47.196164+0000","last_undegraded":"2024-09-13T14:56:47.196164+0000","last_fullsized":"2024-09-13T14:56:47.196164+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:47:23.172452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.c","version":"468'7703","reported_seq":9027,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-13T14:57:01.768574+0000","last_change":"2024-09-13T14:55:28.486018+0000","last_active":"2024-09-13T14:57:01.768574+0000","last_peered":"2024-09-13T14:57:01.768574+0000","last_clean":"2024-09-13T14:57:01.768574+0000","last_became_active":"2024-09-13T14:55:28.483879+0000","last_became_peered":"2024-09-13T14:55:28.483879+0000","last_unstale":"2024-09-13T14:57:01.768574+0000","last_undegraded":"2024-09-13T14:57:01.768574+0000","last_fullsized":"2024-09-13T14:57:01.768574+0000","mapping_epoch":451,"log_start":"445'1000","ondisk_log_start":"445'1000","created":436,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":6703,"log_dups_size":0,"ondisk_log_size":6703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:45:38.042233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365101056,"num_objects":5571,"num_object_clones":0,"num_object_copies":16713,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7703,"num_write_kb":424768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":632,"num_bytes_recovered":41418752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"90'149","reported_seq":710,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-13T14:57:00.273407+0000","last_change":"2024-09-13T14:47:33.067298+0000","last_active":"2024-09-13T14:57:00.273407+0000","last_peered":"2024-09-13T14:57:00.273407+0000","last_clean":"2024-09-13T14:57:00.273407+0000","last_became_active":"2024-09-13T14:47:29.419416+0000","last_became_peered":"2024-09-13T14:47:29.419416+0000","last_unstale":"2024-09-13T14:57:00.273407+0000","last_undegraded":"2024-09-13T14:57:00.273407+0000","last_fullsized":"2024-09-13T14:57:00.273407+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:33.066954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:33.066954+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:05:42.808060+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":293,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.206569+0000","last_change":"2024-09-13T14:46:01.763906+0000","last_active":"2024-09-13T14:56:47.206569+0000","last_peered":"2024-09-13T14:56:47.206569+0000","last_clean":"2024-09-13T14:56:47.206569+0000","last_became_active":"2024-09-13T14:46:01.763542+0000","last_became_peered":"2024-09-13T14:46:01.763542+0000","last_unstale":"2024-09-13T14:56:47.206569+0000","last_undegraded":"2024-09-13T14:56:47.206569+0000","last_fullsized":"2024-09-13T14:56:47.206569+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:09:36.652767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":1564,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.208292+0000","last_change":"2024-09-13T14:47:50.883365+0000","last_active":"2024-09-13T14:56:47.208292+0000","last_peered":"2024-09-13T14:56:47.208292+0000","last_clean":"2024-09-13T14:56:47.208292+0000","last_became_active":"2024-09-13T14:47:50.882609+0000","last_became_peered":"2024-09-13T14:47:50.882609+0000","last_unstale":"2024-09-13T14:56:47.208292+0000","last_undegraded":"2024-09-13T14:56:47.208292+0000","last_fullsized":"2024-09-13T14:56:47.208292+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:12:17.961575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":283,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.193767+0000","last_change":"2024-09-13T14:55:17.937000+0000","last_active":"2024-09-13T14:56:47.193767+0000","last_peered":"2024-09-13T14:56:47.193767+0000","last_clean":"2024-09-13T14:56:47.193767+0000","last_became_active":"2024-09-13T14:55:17.936557+0000","last_became_peered":"2024-09-13T14:55:17.936557+0000","last_unstale":"2024-09-13T14:56:47.193767+0000","last_undegraded":"2024-09-13T14:56:47.193767+0000","last_fullsized":"2024-09-13T14:56:47.193767+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:55:25.879096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"468'7637","reported_seq":8866,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-13T14:57:03.407572+0000","last_change":"2024-09-13T14:56:12.498406+0000","last_active":"2024-09-13T14:57:03.407572+0000","last_peered":"2024-09-13T14:57:03.407572+0000","last_clean":"2024-09-13T14:57:03.407572+0000","last_became_active":"2024-09-13T14:56:12.490622+0000","last_became_peered":"2024-09-13T14:56:12.490622+0000","last_unstale":"2024-09-13T14:57:03.407572+0000","last_undegraded":"2024-09-13T14:57:03.407572+0000","last_fullsized":"2024-09-13T14:57:03.407572+0000","mapping_epoch":462,"log_start":"458'1800","ondisk_log_start":"458'1800","created":436,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":5837,"log_dups_size":0,"ondisk_log_size":5837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:07:04.577772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358613016,"num_objects":5473,"num_object_clones":0,"num_object_copies":16419,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5473,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7637,"num_write_kb":419457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":572,"num_bytes_recovered":37486592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"90'149","reported_seq":406,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-13T14:57:00.272577+0000","last_change":"2024-09-13T14:56:12.709187+0000","last_active":"2024-09-13T14:57:00.272577+0000","last_peered":"2024-09-13T14:57:00.272577+0000","last_clean":"2024-09-13T14:57:00.272577+0000","last_became_active":"2024-09-13T14:56:12.491639+0000","last_became_peered":"2024-09-13T14:56:12.491639+0000","last_unstale":"2024-09-13T14:57:00.272577+0000","last_undegraded":"2024-09-13T14:57:00.272577+0000","last_fullsized":"2024-09-13T14:57:00.272577+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:56:12.709114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:56:12.709114+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:39:58.475573+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":276,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.196526+0000","last_change":"2024-09-13T14:47:47.876562+0000","last_active":"2024-09-13T14:56:47.196526+0000","last_peered":"2024-09-13T14:56:47.196526+0000","last_clean":"2024-09-13T14:56:47.196526+0000","last_became_active":"2024-09-13T14:47:47.876297+0000","last_became_peered":"2024-09-13T14:47:47.876297+0000","last_unstale":"2024-09-13T14:56:47.196526+0000","last_undegraded":"2024-09-13T14:56:47.196526+0000","last_fullsized":"2024-09-13T14:56:47.196526+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:03:27.761394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":1033,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.207126+0000","last_change":"2024-09-13T14:55:20.147689+0000","last_active":"2024-09-13T14:56:47.207126+0000","last_peered":"2024-09-13T14:56:47.207126+0000","last_clean":"2024-09-13T14:56:47.207126+0000","last_became_active":"2024-09-13T14:55:20.147241+0000","last_became_peered":"2024-09-13T14:55:20.147241+0000","last_unstale":"2024-09-13T14:56:47.207126+0000","last_undegraded":"2024-09-13T14:56:47.207126+0000","last_fullsized":"2024-09-13T14:56:47.207126+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:43:36.224330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":324,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.205615+0000","last_change":"2024-09-13T14:56:12.491490+0000","last_active":"2024-09-13T14:56:47.205615+0000","last_peered":"2024-09-13T14:56:47.205615+0000","last_clean":"2024-09-13T14:56:47.205615+0000","last_became_active":"2024-09-13T14:56:12.490650+0000","last_became_peered":"2024-09-13T14:56:12.490650+0000","last_unstale":"2024-09-13T14:56:47.205615+0000","last_undegraded":"2024-09-13T14:56:47.205615+0000","last_fullsized":"2024-09-13T14:56:47.205615+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:55:33.313465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"468'7706","reported_seq":9105,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-13T14:57:01.536917+0000","last_change":"2024-09-13T14:55:23.955183+0000","last_active":"2024-09-13T14:57:01.536917+0000","last_peered":"2024-09-13T14:57:01.536917+0000","last_clean":"2024-09-13T14:57:01.536917+0000","last_became_active":"2024-09-13T14:55:23.952692+0000","last_became_peered":"2024-09-13T14:55:23.952692+0000","last_unstale":"2024-09-13T14:57:01.536917+0000","last_undegraded":"2024-09-13T14:57:01.536917+0000","last_fullsized":"2024-09-13T14:57:01.536917+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":7706,"log_dups_size":0,"ondisk_log_size":7706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:48:52.915452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365166592,"num_objects":5572,"num_object_clones":0,"num_object_copies":16716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7706,"num_write_kb":424896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":659,"num_bytes_recovered":43188224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"90'149","reported_seq":456,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-13T14:57:00.266247+0000","last_change":"2024-09-13T14:55:32.717170+0000","last_active":"2024-09-13T14:57:00.266247+0000","last_peered":"2024-09-13T14:57:00.266247+0000","last_clean":"2024-09-13T14:57:00.266247+0000","last_became_active":"2024-09-13T14:55:20.143587+0000","last_became_peered":"2024-09-13T14:55:20.143587+0000","last_unstale":"2024-09-13T14:57:00.266247+0000","last_undegraded":"2024-09-13T14:57:00.266247+0000","last_fullsized":"2024-09-13T14:57:00.266247+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:55:32.716911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:55:32.716911+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:01:10.628613+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":868,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-13T14:57:00.273695+0000","last_change":"2024-09-13T14:38:53.938227+0000","last_active":"2024-09-13T14:57:00.273695+0000","last_peered":"2024-09-13T14:57:00.273695+0000","last_clean":"2024-09-13T14:57:00.273695+0000","last_became_active":"2024-09-13T14:38:53.902867+0000","last_became_peered":"2024-09-13T14:38:53.902867+0000","last_unstale":"2024-09-13T14:57:00.273695+0000","last_undegraded":"2024-09-13T14:57:00.273695+0000","last_fullsized":"2024-09-13T14:57:00.273695+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:16.831756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":780,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.203782+0000","last_change":"2024-09-13T14:55:29.510841+0000","last_active":"2024-09-13T14:56:47.203782+0000","last_peered":"2024-09-13T14:56:47.203782+0000","last_clean":"2024-09-13T14:56:47.203782+0000","last_became_active":"2024-09-13T14:55:29.510252+0000","last_became_peered":"2024-09-13T14:55:29.510252+0000","last_unstale":"2024-09-13T14:56:47.203782+0000","last_undegraded":"2024-09-13T14:56:47.203782+0000","last_fullsized":"2024-09-13T14:56:47.203782+0000","mapping_epoch":452,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":453,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:14:27.135796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":851,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-13T14:57:00.271846+0000","last_change":"2024-09-13T14:38:53.900107+0000","last_active":"2024-09-13T14:57:00.271846+0000","last_peered":"2024-09-13T14:57:00.271846+0000","last_clean":"2024-09-13T14:57:00.271846+0000","last_became_active":"2024-09-13T14:38:53.899556+0000","last_became_peered":"2024-09-13T14:38:53.899556+0000","last_unstale":"2024-09-13T14:57:00.271846+0000","last_undegraded":"2024-09-13T14:57:00.271846+0000","last_fullsized":"2024-09-13T14:57:00.271846+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:15:47.918163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"468'7703","reported_seq":7779,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-13T14:57:03.413808+0000","last_change":"2024-09-13T14:54:58.382957+0000","last_active":"2024-09-13T14:57:03.413808+0000","last_peered":"2024-09-13T14:57:03.413808+0000","last_clean":"2024-09-13T14:57:03.413808+0000","last_became_active":"2024-09-13T14:54:58.382609+0000","last_became_peered":"2024-09-13T14:54:58.382609+0000","last_unstale":"2024-09-13T14:57:03.413808+0000","last_undegraded":"2024-09-13T14:57:03.413808+0000","last_fullsized":"2024-09-13T14:57:03.413808+0000","mapping_epoch":436,"log_start":"460'1900","ondisk_log_start":"460'1900","created":436,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":5803,"log_dups_size":0,"ondisk_log_size":5803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:55:46.466387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":361037824,"num_objects":5509,"num_object_clones":0,"num_object_copies":16527,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7703,"num_write_kb":422784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"90'149","reported_seq":707,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-13T14:57:00.291434+0000","last_change":"2024-09-13T14:47:29.999643+0000","last_active":"2024-09-13T14:57:00.291434+0000","last_peered":"2024-09-13T14:57:00.291434+0000","last_clean":"2024-09-13T14:57:00.291434+0000","last_became_active":"2024-09-13T14:47:29.410951+0000","last_became_peered":"2024-09-13T14:47:29.410951+0000","last_unstale":"2024-09-13T14:57:00.291434+0000","last_undegraded":"2024-09-13T14:57:00.291434+0000","last_fullsized":"2024-09-13T14:57:00.291434+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:29.999519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:29.999519+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:04:54.887884+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":233,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.203515+0000","last_change":"2024-09-13T14:56:12.493198+0000","last_active":"2024-09-13T14:56:47.203515+0000","last_peered":"2024-09-13T14:56:47.203515+0000","last_clean":"2024-09-13T14:56:47.203515+0000","last_became_active":"2024-09-13T14:56:12.490423+0000","last_became_peered":"2024-09-13T14:56:12.490423+0000","last_unstale":"2024-09-13T14:56:47.203515+0000","last_undegraded":"2024-09-13T14:56:47.203515+0000","last_fullsized":"2024-09-13T14:56:47.203515+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:29.984031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":274,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.207631+0000","last_change":"2024-09-13T14:46:01.781508+0000","last_active":"2024-09-13T14:56:47.207631+0000","last_peered":"2024-09-13T14:56:47.207631+0000","last_clean":"2024-09-13T14:56:47.207631+0000","last_became_active":"2024-09-13T14:46:01.767096+0000","last_became_peered":"2024-09-13T14:46:01.767096+0000","last_unstale":"2024-09-13T14:56:47.207631+0000","last_undegraded":"2024-09-13T14:56:47.207631+0000","last_fullsized":"2024-09-13T14:56:47.207631+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:20:01.119189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":257,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.208513+0000","last_change":"2024-09-13T14:55:17.932691+0000","last_active":"2024-09-13T14:56:47.208513+0000","last_peered":"2024-09-13T14:56:47.208513+0000","last_clean":"2024-09-13T14:56:47.208513+0000","last_became_active":"2024-09-13T14:55:17.932391+0000","last_became_peered":"2024-09-13T14:55:17.932391+0000","last_unstale":"2024-09-13T14:56:47.208513+0000","last_undegraded":"2024-09-13T14:56:47.208513+0000","last_fullsized":"2024-09-13T14:56:47.208513+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:03:22.734037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":414,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.196010+0000","last_change":"2024-09-13T14:55:17.930074+0000","last_active":"2024-09-13T14:56:47.196010+0000","last_peered":"2024-09-13T14:56:47.196010+0000","last_clean":"2024-09-13T14:56:47.196010+0000","last_became_active":"2024-09-13T14:55:17.928411+0000","last_became_peered":"2024-09-13T14:55:17.928411+0000","last_unstale":"2024-09-13T14:56:47.196010+0000","last_undegraded":"2024-09-13T14:56:47.196010+0000","last_fullsized":"2024-09-13T14:56:47.196010+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:11:35.302734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":104,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.206738+0000","last_change":"2024-09-13T14:55:17.939375+0000","last_active":"2024-09-13T14:56:47.206738+0000","last_peered":"2024-09-13T14:56:47.206738+0000","last_clean":"2024-09-13T14:56:47.206738+0000","last_became_active":"2024-09-13T14:55:17.926420+0000","last_became_peered":"2024-09-13T14:55:17.926420+0000","last_unstale":"2024-09-13T14:56:47.206738+0000","last_undegraded":"2024-09-13T14:56:47.206738+0000","last_fullsized":"2024-09-13T14:56:47.206738+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:38:13.401866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4756,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":275,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.193579+0000","last_change":"2024-09-13T14:47:47.867935+0000","last_active":"2024-09-13T14:56:47.193579+0000","last_peered":"2024-09-13T14:56:47.193579+0000","last_clean":"2024-09-13T14:56:47.193579+0000","last_became_active":"2024-09-13T14:47:47.867616+0000","last_became_peered":"2024-09-13T14:47:47.867616+0000","last_unstale":"2024-09-13T14:56:47.193579+0000","last_undegraded":"2024-09-13T14:56:47.193579+0000","last_fullsized":"2024-09-13T14:56:47.193579+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:52:38.619916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":253,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.206837+0000","last_change":"2024-09-13T14:55:17.951412+0000","last_active":"2024-09-13T14:56:47.206837+0000","last_peered":"2024-09-13T14:56:47.206837+0000","last_clean":"2024-09-13T14:56:47.206837+0000","last_became_active":"2024-09-13T14:55:17.937147+0000","last_became_peered":"2024-09-13T14:55:17.937147+0000","last_unstale":"2024-09-13T14:56:47.206837+0000","last_undegraded":"2024-09-13T14:56:47.206837+0000","last_fullsized":"2024-09-13T14:56:47.206837+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:43.786806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"90'149","reported_seq":720,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-13T14:57:00.274877+0000","last_change":"2024-09-13T14:55:20.149252+0000","last_active":"2024-09-13T14:57:00.274877+0000","last_peered":"2024-09-13T14:57:00.274877+0000","last_clean":"2024-09-13T14:57:00.274877+0000","last_became_active":"2024-09-13T14:55:20.148898+0000","last_became_peered":"2024-09-13T14:55:20.148898+0000","last_unstale":"2024-09-13T14:57:00.274877+0000","last_undegraded":"2024-09-13T14:57:00.274877+0000","last_fullsized":"2024-09-13T14:57:00.274877+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:31.013128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:31.013128+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:36:08.933590+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":299,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.206787+0000","last_change":"2024-09-13T14:47:47.861842+0000","last_active":"2024-09-13T14:56:47.206787+0000","last_peered":"2024-09-13T14:56:47.206787+0000","last_clean":"2024-09-13T14:56:47.206787+0000","last_became_active":"2024-09-13T14:47:47.861492+0000","last_became_peered":"2024-09-13T14:47:47.861492+0000","last_unstale":"2024-09-13T14:56:47.206787+0000","last_undegraded":"2024-09-13T14:56:47.206787+0000","last_fullsized":"2024-09-13T14:56:47.206787+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:17.279533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":791,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-13T14:57:00.272132+0000","last_change":"2024-09-13T14:38:53.899535+0000","last_active":"2024-09-13T14:57:00.272132+0000","last_peered":"2024-09-13T14:57:00.272132+0000","last_clean":"2024-09-13T14:57:00.272132+0000","last_became_active":"2024-09-13T14:38:53.899183+0000","last_became_peered":"2024-09-13T14:38:53.899183+0000","last_unstale":"2024-09-13T14:57:00.272132+0000","last_undegraded":"2024-09-13T14:57:00.272132+0000","last_fullsized":"2024-09-13T14:57:00.272132+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:33.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":348,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.206881+0000","last_change":"2024-09-13T14:56:12.488981+0000","last_active":"2024-09-13T14:56:47.206881+0000","last_peered":"2024-09-13T14:56:47.206881+0000","last_clean":"2024-09-13T14:56:47.206881+0000","last_became_active":"2024-09-13T14:56:12.488552+0000","last_became_peered":"2024-09-13T14:56:12.488552+0000","last_unstale":"2024-09-13T14:56:47.206881+0000","last_undegraded":"2024-09-13T14:56:47.206881+0000","last_fullsized":"2024-09-13T14:56:47.206881+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:02:23.732498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":827,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-13T14:57:00.271735+0000","last_change":"2024-09-13T14:47:47.868624+0000","last_active":"2024-09-13T14:57:00.271735+0000","last_peered":"2024-09-13T14:57:00.271735+0000","last_clean":"2024-09-13T14:57:00.271735+0000","last_became_active":"2024-09-13T14:47:47.868031+0000","last_became_peered":"2024-09-13T14:47:47.868031+0000","last_unstale":"2024-09-13T14:57:00.271735+0000","last_undegraded":"2024-09-13T14:57:00.271735+0000","last_fullsized":"2024-09-13T14:57:00.271735+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:05:25.365023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":313,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-13T14:57:00.266824+0000","last_change":"2024-09-13T14:47:47.859825+0000","last_active":"2024-09-13T14:57:00.266824+0000","last_peered":"2024-09-13T14:57:00.266824+0000","last_clean":"2024-09-13T14:57:00.266824+0000","last_became_active":"2024-09-13T14:47:47.859069+0000","last_became_peered":"2024-09-13T14:47:47.859069+0000","last_unstale":"2024-09-13T14:57:00.266824+0000","last_undegraded":"2024-09-13T14:57:00.266824+0000","last_fullsized":"2024-09-13T14:57:00.266824+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:18:36.452321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":859,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-13T14:57:00.271899+0000","last_change":"2024-09-13T14:47:47.868945+0000","last_active":"2024-09-13T14:57:00.271899+0000","last_peered":"2024-09-13T14:57:00.271899+0000","last_clean":"2024-09-13T14:57:00.271899+0000","last_became_active":"2024-09-13T14:47:47.868296+0000","last_became_peered":"2024-09-13T14:47:47.868296+0000","last_unstale":"2024-09-13T14:57:00.271899+0000","last_undegraded":"2024-09-13T14:57:00.271899+0000","last_fullsized":"2024-09-13T14:57:00.271899+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:22:24.778693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":44,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.207159+0000","last_change":"2024-09-13T14:56:12.491111+0000","last_active":"2024-09-13T14:56:47.207159+0000","last_peered":"2024-09-13T14:56:47.207159+0000","last_clean":"2024-09-13T14:56:47.207159+0000","last_became_active":"2024-09-13T14:56:12.490691+0000","last_became_peered":"2024-09-13T14:56:12.490691+0000","last_unstale":"2024-09-13T14:56:47.207159+0000","last_undegraded":"2024-09-13T14:56:47.207159+0000","last_fullsized":"2024-09-13T14:56:47.207159+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:01.439884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":282,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.191532+0000","last_change":"2024-09-13T14:55:17.929446+0000","last_active":"2024-09-13T14:56:47.191532+0000","last_peered":"2024-09-13T14:56:47.191532+0000","last_clean":"2024-09-13T14:56:47.191532+0000","last_became_active":"2024-09-13T14:55:17.928598+0000","last_became_peered":"2024-09-13T14:55:17.928598+0000","last_unstale":"2024-09-13T14:56:47.191532+0000","last_undegraded":"2024-09-13T14:56:47.191532+0000","last_fullsized":"2024-09-13T14:56:47.191532+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:27.446138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":854,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-13T14:57:00.288212+0000","last_change":"2024-09-13T14:55:17.933267+0000","last_active":"2024-09-13T14:57:00.288212+0000","last_peered":"2024-09-13T14:57:00.288212+0000","last_clean":"2024-09-13T14:57:00.288212+0000","last_became_active":"2024-09-13T14:55:17.932890+0000","last_became_peered":"2024-09-13T14:55:17.932890+0000","last_unstale":"2024-09-13T14:57:00.288212+0000","last_undegraded":"2024-09-13T14:57:00.288212+0000","last_fullsized":"2024-09-13T14:57:00.288212+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:48:59.113533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":467,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.193419+0000","last_change":"2024-09-13T14:55:17.937105+0000","last_active":"2024-09-13T14:56:47.193419+0000","last_peered":"2024-09-13T14:56:47.193419+0000","last_clean":"2024-09-13T14:56:47.193419+0000","last_became_active":"2024-09-13T14:55:17.936621+0000","last_became_peered":"2024-09-13T14:55:17.936621+0000","last_unstale":"2024-09-13T14:56:47.193419+0000","last_undegraded":"2024-09-13T14:56:47.193419+0000","last_fullsized":"2024-09-13T14:56:47.193419+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:37:28.134499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":359,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-13T14:57:00.275162+0000","last_change":"2024-09-13T14:50:27.510133+0000","last_active":"2024-09-13T14:57:00.275162+0000","last_peered":"2024-09-13T14:57:00.275162+0000","last_clean":"2024-09-13T14:57:00.275162+0000","last_became_active":"2024-09-13T14:50:27.508774+0000","last_became_peered":"2024-09-13T14:50:27.508774+0000","last_unstale":"2024-09-13T14:57:00.275162+0000","last_undegraded":"2024-09-13T14:57:00.275162+0000","last_fullsized":"2024-09-13T14:57:00.275162+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:31:04.531746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":17,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.206449+0000","last_change":"2024-09-13T14:56:12.489663+0000","last_active":"2024-09-13T14:56:47.206449+0000","last_peered":"2024-09-13T14:56:47.206449+0000","last_clean":"2024-09-13T14:56:47.206449+0000","last_became_active":"2024-09-13T14:56:12.489317+0000","last_became_peered":"2024-09-13T14:56:12.489317+0000","last_unstale":"2024-09-13T14:56:47.206449+0000","last_undegraded":"2024-09-13T14:56:47.206449+0000","last_fullsized":"2024-09-13T14:56:47.206449+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:08:44.565862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":413,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-13T14:57:00.272056+0000","last_change":"2024-09-13T14:55:17.932405+0000","last_active":"2024-09-13T14:57:00.272056+0000","last_peered":"2024-09-13T14:57:00.272056+0000","last_clean":"2024-09-13T14:57:00.272056+0000","last_became_active":"2024-09-13T14:55:17.932058+0000","last_became_peered":"2024-09-13T14:55:17.932058+0000","last_unstale":"2024-09-13T14:57:00.272056+0000","last_undegraded":"2024-09-13T14:57:00.272056+0000","last_fullsized":"2024-09-13T14:57:00.272056+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:05:01.826514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":313,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.207058+0000","last_change":"2024-09-13T14:55:17.946147+0000","last_active":"2024-09-13T14:56:47.207058+0000","last_peered":"2024-09-13T14:56:47.207058+0000","last_clean":"2024-09-13T14:56:47.207058+0000","last_became_active":"2024-09-13T14:55:17.931009+0000","last_became_peered":"2024-09-13T14:55:17.931009+0000","last_unstale":"2024-09-13T14:56:47.207058+0000","last_undegraded":"2024-09-13T14:56:47.207058+0000","last_fullsized":"2024-09-13T14:56:47.207058+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:08:26.717928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":310,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-13T14:57:00.276129+0000","last_change":"2024-09-13T14:55:17.936194+0000","last_active":"2024-09-13T14:57:00.276129+0000","last_peered":"2024-09-13T14:57:00.276129+0000","last_clean":"2024-09-13T14:57:00.276129+0000","last_became_active":"2024-09-13T14:55:17.935832+0000","last_became_peered":"2024-09-13T14:55:17.935832+0000","last_unstale":"2024-09-13T14:57:00.276129+0000","last_undegraded":"2024-09-13T14:57:00.276129+0000","last_fullsized":"2024-09-13T14:57:00.276129+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:40:52.745439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":300,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.206653+0000","last_change":"2024-09-13T14:55:17.941129+0000","last_active":"2024-09-13T14:56:47.206653+0000","last_peered":"2024-09-13T14:56:47.206653+0000","last_clean":"2024-09-13T14:56:47.206653+0000","last_became_active":"2024-09-13T14:55:17.930922+0000","last_became_peered":"2024-09-13T14:55:17.930922+0000","last_unstale":"2024-09-13T14:56:47.206653+0000","last_undegraded":"2024-09-13T14:56:47.206653+0000","last_fullsized":"2024-09-13T14:56:47.206653+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:06:29.610560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":873,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-13T14:57:00.272594+0000","last_change":"2024-09-13T14:46:01.766580+0000","last_active":"2024-09-13T14:57:00.272594+0000","last_peered":"2024-09-13T14:57:00.272594+0000","last_clean":"2024-09-13T14:57:00.272594+0000","last_became_active":"2024-09-13T14:46:01.766257+0000","last_became_peered":"2024-09-13T14:46:01.766257+0000","last_unstale":"2024-09-13T14:57:00.272594+0000","last_undegraded":"2024-09-13T14:57:00.272594+0000","last_fullsized":"2024-09-13T14:57:00.272594+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:01.126109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":838,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-13T14:57:00.272839+0000","last_change":"2024-09-13T14:46:01.769647+0000","last_active":"2024-09-13T14:57:00.272839+0000","last_peered":"2024-09-13T14:57:00.272839+0000","last_clean":"2024-09-13T14:57:00.272839+0000","last_became_active":"2024-09-13T14:46:01.769228+0000","last_became_peered":"2024-09-13T14:46:01.769228+0000","last_unstale":"2024-09-13T14:57:00.272839+0000","last_undegraded":"2024-09-13T14:57:00.272839+0000","last_fullsized":"2024-09-13T14:57:00.272839+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:58.696024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":393,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.192245+0000","last_change":"2024-09-13T14:55:17.929205+0000","last_active":"2024-09-13T14:56:47.192245+0000","last_peered":"2024-09-13T14:56:47.192245+0000","last_clean":"2024-09-13T14:56:47.192245+0000","last_became_active":"2024-09-13T14:55:17.928883+0000","last_became_peered":"2024-09-13T14:55:17.928883+0000","last_unstale":"2024-09-13T14:56:47.192245+0000","last_undegraded":"2024-09-13T14:56:47.192245+0000","last_fullsized":"2024-09-13T14:56:47.192245+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:27:31.570502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":371,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-13T14:56:47.206805+0000","last_change":"2024-09-13T14:56:12.492223+0000","last_active":"2024-09-13T14:56:47.206805+0000","last_peered":"2024-09-13T14:56:47.206805+0000","last_clean":"2024-09-13T14:56:47.206805+0000","last_became_active":"2024-09-13T14:56:12.490578+0000","last_became_peered":"2024-09-13T14:56:12.490578+0000","last_unstale":"2024-09-13T14:56:47.206805+0000","last_undegraded":"2024-09-13T14:56:47.206805+0000","last_fullsized":"2024-09-13T14:56:47.206805+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:11:13.623995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":5856952344,"num_objects":89371,"num_object_clones":0,"num_object_copies":268113,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89371,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122309,"num_write_kb":6773697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4957,"num_bytes_recovered":324861952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17594331136,"data_stored":17594318920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107209,"ondisk_log_size":107209,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":6474,"num_keys_recovered":30,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":21564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25132,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":171,"num_bytes_recovered":8388646,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3129,"ondisk_log_size":3129,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234558,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2726392,"kb_used_data":1806828,"kb_used_omap":1,"kb_used_meta":919550,"kb_avail":91018760,"statfs":{"total":95995035648,"available":93203210240,"internally_reserved":0,"allocated":1850191872,"data_stored":1847793320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":941619659},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":239,"seq":1026497183976,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3164404,"kb_used_data":2520428,"kb_used_omap":8,"kb_used_meta":643959,"kb_avail":90580748,"statfs":{"total":95995035648,"available":92754685952,"internally_reserved":0,"allocated":2580918272,"data_stored":2578519709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8453,"internal_metadata":659414779},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757823,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3380588,"kb_used_data":2174880,"kb_used_omap":1,"kb_used_meta":1205694,"kb_avail":90364564,"statfs":{"total":95995035648,"available":92533313536,"internally_reserved":0,"allocated":2227077120,"data_stored":2224687099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1234631110},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":128849019467,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435650,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3965480,"kb_used_data":2826592,"kb_used_omap":5,"kb_used_meta":1138874,"kb_avail":89779672,"statfs":{"total":95995035648,"available":91934384128,"internally_reserved":0,"allocated":2894430208,"data_stored":2892039973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":1166207102},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":104,"seq":446676599178,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2621368,"kb_used_data":2156460,"kb_used_omap":7,"kb_used_meta":464824,"kb_avail":91123784,"statfs":{"total":95995035648,"available":93310754816,"internally_reserved":0,"allocated":2208215040,"data_stored":2205817775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8010,"internal_metadata":475979958},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":99,"seq":425201762704,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2689044,"kb_used_data":2219980,"kb_used_omap":7,"kb_used_meta":468984,"kb_avail":91056108,"statfs":{"total":95995035648,"available":93241454592,"internally_reserved":0,"allocated":2273259520,"data_stored":2270880578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7759,"internal_metadata":480240049},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926228,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4768560,"kb_used_data":3631780,"kb_used_omap":7,"kb_used_meta":1136760,"kb_avail":88976592,"statfs":{"total":95995035648,"available":91112030208,"internally_reserved":0,"allocated":3718942720,"data_stored":3716548449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":1164042536},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6860,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3679457280,"data_stored":3679453208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2242510848,"data_stored":2242510848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2194604032,"data_stored":2194604032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2870480896,"data_stored":2870476824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":524288,"data_stored":524288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2205089792,"data_stored":2205089792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2569211904,"data_stored":2569207832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1832452096,"data_stored":1832452096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:57:07.091 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-13T14:57:07.092 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pg_num 31 2024-09-13T14:57:07.103 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:07.128 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:07.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:07.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:07.230 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:07.282 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:07.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:07.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:07.384 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:07.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:07.452 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:07.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:07.553 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:07.557 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:07 smithi102 ceph-mon[110183]: from='client.46778 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:57:07.558 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:07 smithi102 ceph-mon[110183]: pgmap v1571: 109 pgs: 109 active+clean; 5.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 911 op/s 2024-09-13T14:57:07.558 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:07 smithi102 ceph-mon[115269]: from='client.46778 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:57:07.558 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:07 smithi102 ceph-mon[115269]: pgmap v1571: 109 pgs: 109 active+clean; 5.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 911 op/s 2024-09-13T14:57:07.580 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:07.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:07.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:07.624 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:07 smithi195 ceph-mon[104262]: from='client.46778 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:57:07.624 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:07 smithi195 ceph-mon[104262]: pgmap v1571: 109 pgs: 109 active+clean; 5.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 911 op/s 2024-09-13T14:57:07.682 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:07.706 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:07.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:07.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:07.807 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:07.850 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:07.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:07.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:07.951 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:07.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:07.978 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:07.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:08.079 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:08.103 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:08.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:08.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:08.204 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:08.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:08.245 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:08.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:08.347 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:08.372 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:08.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:08.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:08.400 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:08.474 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:08.504 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:08.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:08.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:08.604 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-13T14:57:08.604 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:57:08.606 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:08.634 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:08.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:08.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:08.735 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:08.757 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:08.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:08.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:08.859 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:08.904 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:08.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:08.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:09.006 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:09.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:09.058 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:09.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:09.159 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:09.184 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:09.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:09.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:09.286 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:09.325 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T14:57:09.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:09.326 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:09.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:09.330 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is set 2024-09-13T14:57:09.428 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:09.460 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:09.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:09.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:09.493 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:09 smithi102 ceph-mon[115269]: pgmap v1572: 109 pgs: 109 active+clean; 5.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 916 op/s 2024-09-13T14:57:09.493 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:09 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1994709239' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:57:09.493 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:09 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:57:09.494 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:09 smithi102 ceph-mon[110183]: pgmap v1572: 109 pgs: 109 active+clean; 5.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 916 op/s 2024-09-13T14:57:09.494 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:09 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1994709239' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:57:09.494 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:09 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:57:09.563 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:09.588 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:57:09.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:09.604 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:09.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:09.706 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:09.719 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:09 smithi195 ceph-mon[104262]: pgmap v1572: 109 pgs: 109 active+clean; 5.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 916 op/s 2024-09-13T14:57:09.719 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:09 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1994709239' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:57:09.719 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:09 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:57:09.729 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:09.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:09.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:09.831 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:09.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:09.856 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:09.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:09.957 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:09.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:09.982 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:09.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:10.083 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:10.110 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:10.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:10.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:10.212 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:10.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:10.300 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:10.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:10.402 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:10.436 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:10.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:10.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:10.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:10 smithi195 ceph-mon[104262]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:57:10.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:10 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:57:10.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:10 smithi195 ceph-mon[104262]: osdmap e469: 8 total, 8 up, 7 in 2024-09-13T14:57:10.538 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:10.568 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:10.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:10.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:10.645 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:10 smithi102 ceph-mon[110183]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:57:10.646 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:10 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:57:10.646 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:10 smithi102 ceph-mon[110183]: osdmap e469: 8 total, 8 up, 7 in 2024-09-13T14:57:10.646 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:10 smithi102 ceph-mon[115269]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:57:10.646 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:10 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:57:10.646 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:10 smithi102 ceph-mon[115269]: osdmap e469: 8 total, 8 up, 7 in 2024-09-13T14:57:10.670 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:10.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:10.724 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:10.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:10.825 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:10.833 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:57:10.833 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:57:10.851 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:10.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:10.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:10.858 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:57:10.859 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:57:10.902 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:57:10.903 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:57:10.942 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:57:10.942 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:57:10.953 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:10.976 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:57:10.977 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:57:10.977 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:10.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:10.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:11.023 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:57:11.024 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:57:11.062 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:57:11.062 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:57:11.079 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:11.091 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:57:11.091 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:57:11.108 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:11.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:11.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:11.210 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:11.235 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:11.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:11.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:11.340 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:11.367 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:11.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:11.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:11.433 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:11 smithi195 ceph-mon[104262]: pgmap v1574: 109 pgs: 109 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 938 op/s 2024-09-13T14:57:11.433 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:11 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:57:11.433 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:11 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2248609280' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:57:11.470 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:11.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:11.508 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:11.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:11.609 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:11.641 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:11.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:11.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:11.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:11 smithi102 ceph-mon[110183]: pgmap v1574: 109 pgs: 109 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 938 op/s 2024-09-13T14:57:11.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:11 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:57:11.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:11 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2248609280' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:57:11.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:11 smithi102 ceph-mon[115269]: pgmap v1574: 109 pgs: 109 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 938 op/s 2024-09-13T14:57:11.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:11 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:57:11.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:11 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2248609280' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:57:11.712 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:57:11.712 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-09-13T14:57:11.712 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-09-13T14:57:11.712 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-13T14:57:11.712 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.712 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:72299 9.13 9:c8621b29:::benchmark_data_smithi102_348563_object72298:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-09-13T14:57:11.712 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:51:46.193942+0000", 2024-09-13T14:57:11.712 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 325.51444934800003, 2024-09-13T14:57:11.712 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1183224979999999, 2024-09-13T14:57:11.712 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:57:11.713 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:11.713 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:57:11.713 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:57:11.713 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:57:11.713 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 72299 2024-09-13T14:57:11.713 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.713 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:57:11.713 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.713 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:57:11.713 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.193942+0000", 2024-09-13T14:57:11.713 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:11.713 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.714 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.714 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:57:11.714 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.193942+0000", 2024-09-13T14:57:11.714 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:11.714 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.714 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.714 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:57:11.714 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.193940+0000", 2024-09-13T14:57:11.714 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:57:11.714 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.714 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.714 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:57:11.715 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.193990+0000", 2024-09-13T14:57:11.715 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9901999999999998e-05 2024-09-13T14:57:11.715 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.715 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.715 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:57:11.715 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.194003+0000", 2024-09-13T14:57:11.715 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3222e-05 2024-09-13T14:57:11.715 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.715 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.715 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:57:11.715 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.194009+0000", 2024-09-13T14:57:11.715 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7710000000000003e-06 2024-09-13T14:57:11.715 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.716 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.716 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.716 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.194063+0000", 2024-09-13T14:57:11.716 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3977999999999997e-05 2024-09-13T14:57:11.716 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.716 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.716 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.716 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.194076+0000", 2024-09-13T14:57:11.716 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3288000000000001e-05 2024-09-13T14:57:11.716 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.717 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.717 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.717 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.056600+0000", 2024-09-13T14:57:11.717 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.86252373199999999 2024-09-13T14:57:11.717 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.717 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.717 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.717 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.056614+0000", 2024-09-13T14:57:11.717 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3764999999999999e-05 2024-09-13T14:57:11.717 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.717 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.717 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.718 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.302885+0000", 2024-09-13T14:57:11.718 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.246271474 2024-09-13T14:57:11.718 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.718 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.718 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:57:11.719 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.303015+0000", 2024-09-13T14:57:11.719 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012967 2024-09-13T14:57:11.719 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.719 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.719 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:57:11.719 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.303117+0000", 2024-09-13T14:57:11.719 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010189800000000001 2024-09-13T14:57:11.719 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.719 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.719 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:57:11.719 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.307981+0000", 2024-09-13T14:57:11.719 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0048640360000000004 2024-09-13T14:57:11.719 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.720 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.720 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.720 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.308022+0000", 2024-09-13T14:57:11.720 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1356999999999997e-05 2024-09-13T14:57:11.720 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.720 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.720 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.720 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.310751+0000", 2024-09-13T14:57:11.720 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0027289879999999999 2024-09-13T14:57:11.720 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.720 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.720 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:57:11.720 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.310808+0000", 2024-09-13T14:57:11.721 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.6246000000000003e-05 2024-09-13T14:57:11.721 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.721 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.721 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:57:11.721 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.312264+0000", 2024-09-13T14:57:11.721 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001456511 2024-09-13T14:57:11.721 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:11.721 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:57:11.721 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:11.721 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.721 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.721 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:72416 9.13 9:ca62faaf:::benchmark_data_smithi102_348563_object72415:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-09-13T14:57:11.721 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:51:46.289815+0000", 2024-09-13T14:57:11.722 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 325.41857539099999, 2024-09-13T14:57:11.722 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0320571169999999, 2024-09-13T14:57:11.722 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:57:11.722 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:11.722 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:57:11.722 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:57:11.722 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:57:11.722 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 72416 2024-09-13T14:57:11.722 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.722 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:57:11.722 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.722 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:57:11.722 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.289815+0000", 2024-09-13T14:57:11.722 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:11.723 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.723 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.723 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:57:11.723 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.289815+0000", 2024-09-13T14:57:11.723 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:11.723 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.723 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.723 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:57:11.723 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.289809+0000", 2024-09-13T14:57:11.723 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999938 2024-09-13T14:57:11.723 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.723 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.723 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:57:11.724 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.289859+0000", 2024-09-13T14:57:11.724 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9295000000000002e-05 2024-09-13T14:57:11.724 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.724 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.724 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:57:11.724 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.289870+0000", 2024-09-13T14:57:11.724 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1575e-05 2024-09-13T14:57:11.724 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.724 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.724 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:57:11.724 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.289877+0000", 2024-09-13T14:57:11.724 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.0670000000000003e-06 2024-09-13T14:57:11.724 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.724 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.725 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.725 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.289939+0000", 2024-09-13T14:57:11.725 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1532999999999995e-05 2024-09-13T14:57:11.725 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.725 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.725 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.725 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.289956+0000", 2024-09-13T14:57:11.725 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7453999999999999e-05 2024-09-13T14:57:11.725 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.726 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.726 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.726 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.056739+0000", 2024-09-13T14:57:11.726 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.76678289799999999 2024-09-13T14:57:11.726 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.726 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.726 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.726 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.056751+0000", 2024-09-13T14:57:11.726 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1839e-05 2024-09-13T14:57:11.726 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.726 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.726 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.726 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.305540+0000", 2024-09-13T14:57:11.726 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2487891710000001 2024-09-13T14:57:11.727 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.727 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.727 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:57:11.727 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.305705+0000", 2024-09-13T14:57:11.727 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000164547 2024-09-13T14:57:11.727 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.727 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.727 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:57:11.727 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.305889+0000", 2024-09-13T14:57:11.727 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0001839 2024-09-13T14:57:11.727 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.727 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.727 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:57:11.728 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.308075+0000", 2024-09-13T14:57:11.728 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002185977 2024-09-13T14:57:11.728 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.728 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.728 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.728 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.312336+0000", 2024-09-13T14:57:11.728 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0042612129999999998 2024-09-13T14:57:11.728 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.728 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.728 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.728 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.320281+0000", 2024-09-13T14:57:11.728 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0079447949999999993 2024-09-13T14:57:11.728 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.729 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.729 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:57:11.729 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.320333+0000", 2024-09-13T14:57:11.729 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1956999999999997e-05 2024-09-13T14:57:11.729 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.729 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.729 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:57:11.729 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.321873+0000", 2024-09-13T14:57:11.729 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001540076 2024-09-13T14:57:11.729 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:11.729 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:57:11.729 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:11.729 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.730 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.730 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:72422 9.13 9:cc800bcd:::benchmark_data_smithi102_348563_object72421:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-09-13T14:57:11.730 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:51:46.291786+0000", 2024-09-13T14:57:11.730 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 325.41660446499998, 2024-09-13T14:57:11.730 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0314244129999999, 2024-09-13T14:57:11.730 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:57:11.730 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:11.730 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:57:11.730 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:57:11.730 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:57:11.730 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 72422 2024-09-13T14:57:11.730 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.730 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:57:11.731 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.731 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:57:11.731 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.291786+0000", 2024-09-13T14:57:11.731 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:11.731 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.731 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.731 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:57:11.731 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.291786+0000", 2024-09-13T14:57:11.731 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:11.732 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.732 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.732 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:57:11.732 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.291786+0000", 2024-09-13T14:57:11.732 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:57:11.732 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.732 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.732 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:57:11.732 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.291867+0000", 2024-09-13T14:57:11.732 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1588000000000006e-05 2024-09-13T14:57:11.732 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.732 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.732 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:57:11.733 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.291881+0000", 2024-09-13T14:57:11.733 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3638000000000001e-05 2024-09-13T14:57:11.733 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.733 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.733 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:57:11.733 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.291888+0000", 2024-09-13T14:57:11.733 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4610000000000002e-06 2024-09-13T14:57:11.733 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.733 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.733 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.733 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.291997+0000", 2024-09-13T14:57:11.733 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010846800000000001 2024-09-13T14:57:11.733 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.734 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.734 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.734 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.292005+0000", 2024-09-13T14:57:11.734 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9459999999999998e-06 2024-09-13T14:57:11.734 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.734 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.734 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.734 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.056783+0000", 2024-09-13T14:57:11.734 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.76477843000000001 2024-09-13T14:57:11.734 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.734 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.734 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.734 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.056792+0000", 2024-09-13T14:57:11.735 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8200000000000003e-06 2024-09-13T14:57:11.735 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.735 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.735 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.735 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.306918+0000", 2024-09-13T14:57:11.735 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2501263090000001 2024-09-13T14:57:11.735 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.735 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.735 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:57:11.735 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.307041+0000", 2024-09-13T14:57:11.735 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012237399999999999 2024-09-13T14:57:11.735 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.735 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.735 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:57:11.736 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.307146+0000", 2024-09-13T14:57:11.736 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000105288 2024-09-13T14:57:11.736 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.736 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.736 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:57:11.736 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.313885+0000", 2024-09-13T14:57:11.736 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0067393590000000003 2024-09-13T14:57:11.736 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.736 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.736 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.736 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.315634+0000", 2024-09-13T14:57:11.736 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001748318 2024-09-13T14:57:11.736 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.737 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.737 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.737 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.321938+0000", 2024-09-13T14:57:11.737 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0063039960000000001 2024-09-13T14:57:11.737 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.737 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.737 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:57:11.737 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.321991+0000", 2024-09-13T14:57:11.737 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3810000000000001e-05 2024-09-13T14:57:11.737 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.737 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.737 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:57:11.737 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.323211+0000", 2024-09-13T14:57:11.737 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0012194949999999999 2024-09-13T14:57:11.738 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:11.738 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:57:11.738 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:11.738 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.738 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.738 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:72445 9.13 9:c8139549:::benchmark_data_smithi102_348563_object72444:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-09-13T14:57:11.738 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:51:46.307279+0000", 2024-09-13T14:57:11.738 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 325.40111196999999, 2024-09-13T14:57:11.738 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0180606839999999, 2024-09-13T14:57:11.738 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:57:11.738 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:11.738 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:57:11.738 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:57:11.738 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:57:11.739 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 72445 2024-09-13T14:57:11.739 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.739 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:57:11.739 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.739 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:57:11.739 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.307279+0000", 2024-09-13T14:57:11.739 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:11.739 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.739 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.739 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:57:11.739 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.307279+0000", 2024-09-13T14:57:11.739 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:11.739 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.739 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.740 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:57:11.740 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.307278+0000", 2024-09-13T14:57:11.740 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:57:11.740 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.740 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.740 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:57:11.740 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.307327+0000", 2024-09-13T14:57:11.740 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8435e-05 2024-09-13T14:57:11.740 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.740 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.740 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:57:11.740 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.307333+0000", 2024-09-13T14:57:11.741 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5989999999999998e-06 2024-09-13T14:57:11.741 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.741 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.741 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:57:11.741 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.307338+0000", 2024-09-13T14:57:11.741 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0300000000000001e-06 2024-09-13T14:57:11.741 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.741 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.741 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.742 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.307385+0000", 2024-09-13T14:57:11.742 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6740000000000003e-05 2024-09-13T14:57:11.742 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.742 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.742 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.742 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.307397+0000", 2024-09-13T14:57:11.742 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1565e-05 2024-09-13T14:57:11.742 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.742 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.742 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:11.744 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.744 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.056816+0000", 2024-09-13T14:57:11.744 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.74941926000000003 2024-09-13T14:57:11.744 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.744 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.744 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.744 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.056825+0000", 2024-09-13T14:57:11.744 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.6719999999999993e-06 2024-09-13T14:57:11.745 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.745 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.748 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.748 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.308111+0000", 2024-09-13T14:57:11.749 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.251286127 2024-09-13T14:57:11.749 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.749 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.749 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:57:11.749 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.308256+0000", 2024-09-13T14:57:11.749 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014530000000000001 2024-09-13T14:57:11.749 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.749 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.749 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:57:11.749 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.308423+0000", 2024-09-13T14:57:11.749 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016677200000000001 2024-09-13T14:57:11.749 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.750 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.750 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:57:11.750 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.313898+0000", 2024-09-13T14:57:11.750 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0054755699999999999 2024-09-13T14:57:11.750 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.750 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.750 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.750 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.315748+0000", 2024-09-13T14:57:11.750 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0018497710000000001 2024-09-13T14:57:11.750 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.750 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.750 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.750 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.323271+0000", 2024-09-13T14:57:11.751 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0075229169999999996 2024-09-13T14:57:11.751 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.751 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.751 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:57:11.751 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.323324+0000", 2024-09-13T14:57:11.751 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2889000000000001e-05 2024-09-13T14:57:11.751 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.751 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.751 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:57:11.751 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.325340+0000", 2024-09-13T14:57:11.751 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0020155580000000002 2024-09-13T14:57:11.751 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:11.751 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:57:11.752 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:11.752 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.752 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.752 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:72478 9.13 9:cd39ee78:::benchmark_data_smithi102_348563_object72477:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-09-13T14:57:11.752 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:51:46.334058+0000", 2024-09-13T14:57:11.752 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 325.37433271399999, 2024-09-13T14:57:11.752 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.016141556, 2024-09-13T14:57:11.752 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:57:11.752 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:11.752 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:57:11.752 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:57:11.752 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:57:11.752 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 72478 2024-09-13T14:57:11.753 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.753 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:57:11.753 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.753 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:57:11.753 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.334058+0000", 2024-09-13T14:57:11.753 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:11.753 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.753 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.753 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:57:11.753 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.334058+0000", 2024-09-13T14:57:11.753 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:11.753 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.754 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.754 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:57:11.754 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.334057+0000", 2024-09-13T14:57:11.754 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:57:11.754 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.754 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.754 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:57:11.754 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.334087+0000", 2024-09-13T14:57:11.754 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0477999999999999e-05 2024-09-13T14:57:11.754 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.754 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.754 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:57:11.754 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.334098+0000", 2024-09-13T14:57:11.755 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0684e-05 2024-09-13T14:57:11.755 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.755 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.755 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:57:11.755 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.334105+0000", 2024-09-13T14:57:11.755 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6080000000000001e-06 2024-09-13T14:57:11.755 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.755 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.755 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.755 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.334197+0000", 2024-09-13T14:57:11.755 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.2443999999999998e-05 2024-09-13T14:57:11.755 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.755 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.756 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.756 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.334207+0000", 2024-09-13T14:57:11.756 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0013000000000001e-05 2024-09-13T14:57:11.756 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.756 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.756 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.756 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.056863+0000", 2024-09-13T14:57:11.756 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.72265576499999995 2024-09-13T14:57:11.756 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.757 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.757 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.757 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.056870+0000", 2024-09-13T14:57:11.757 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8569999999999997e-06 2024-09-13T14:57:11.757 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.757 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.757 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.757 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.309442+0000", 2024-09-13T14:57:11.757 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2525720090000001 2024-09-13T14:57:11.757 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.757 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.757 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:57:11.757 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.309592+0000", 2024-09-13T14:57:11.758 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015071600000000001 2024-09-13T14:57:11.758 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.758 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.758 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:57:11.758 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.309748+0000", 2024-09-13T14:57:11.758 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015559800000000001 2024-09-13T14:57:11.758 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.758 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.758 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:57:11.758 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.313905+0000", 2024-09-13T14:57:11.758 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0041569320000000003 2024-09-13T14:57:11.758 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.758 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.758 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.759 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.325405+0000", 2024-09-13T14:57:11.759 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011499578999999999 2024-09-13T14:57:11.759 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.759 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.759 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.759 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.348507+0000", 2024-09-13T14:57:11.759 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.023102192000000001 2024-09-13T14:57:11.759 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.759 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.759 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:57:11.759 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.348573+0000", 2024-09-13T14:57:11.759 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6265000000000003e-05 2024-09-13T14:57:11.759 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.760 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.760 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:57:11.760 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.350200+0000", 2024-09-13T14:57:11.760 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0016267650000000001 2024-09-13T14:57:11.760 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:11.760 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:57:11.760 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:11.760 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.760 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.760 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:72481 9.13 9:c872827f:::benchmark_data_smithi102_348563_object72480:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-09-13T14:57:11.760 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:51:46.336298+0000", 2024-09-13T14:57:11.760 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 325.37209241800002, 2024-09-13T14:57:11.760 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0149789880000002, 2024-09-13T14:57:11.761 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:57:11.761 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:11.761 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:57:11.761 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:57:11.761 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:57:11.761 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 72481 2024-09-13T14:57:11.761 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.761 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:57:11.761 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.761 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:57:11.761 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.336298+0000", 2024-09-13T14:57:11.761 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:11.761 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.762 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.762 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:57:11.762 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.336298+0000", 2024-09-13T14:57:11.762 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:11.762 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.762 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.762 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:57:11.762 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.336297+0000", 2024-09-13T14:57:11.762 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:57:11.762 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.762 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.762 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:57:11.763 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.336328+0000", 2024-09-13T14:57:11.763 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0834000000000003e-05 2024-09-13T14:57:11.763 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.763 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.763 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:57:11.763 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.336340+0000", 2024-09-13T14:57:11.763 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2451e-05 2024-09-13T14:57:11.763 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.763 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.763 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:57:11.763 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.336347+0000", 2024-09-13T14:57:11.763 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1339999999999998e-06 2024-09-13T14:57:11.763 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.764 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.764 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.764 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.336403+0000", 2024-09-13T14:57:11.764 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5198999999999998e-05 2024-09-13T14:57:11.764 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.764 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.764 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.764 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.336416+0000", 2024-09-13T14:57:11.764 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3451999999999999e-05 2024-09-13T14:57:11.764 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.764 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.765 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.765 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.056988+0000", 2024-09-13T14:57:11.765 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.72057172000000003 2024-09-13T14:57:11.765 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.765 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.765 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.765 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.056993+0000", 2024-09-13T14:57:11.765 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5690000000000004e-06 2024-09-13T14:57:11.765 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.765 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.765 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.765 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.312379+0000", 2024-09-13T14:57:11.765 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.255385564 2024-09-13T14:57:11.766 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.766 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.766 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:57:11.766 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.312524+0000", 2024-09-13T14:57:11.766 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014541200000000001 2024-09-13T14:57:11.766 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.766 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.766 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:57:11.766 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.312690+0000", 2024-09-13T14:57:11.766 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016532 2024-09-13T14:57:11.766 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.766 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.766 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:57:11.767 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.318889+0000", 2024-09-13T14:57:11.767 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.006199202 2024-09-13T14:57:11.767 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.767 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.767 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.767 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.325439+0000", 2024-09-13T14:57:11.767 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0065503610000000002 2024-09-13T14:57:11.767 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.767 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.767 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.767 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.350254+0000", 2024-09-13T14:57:11.767 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024814994999999999 2024-09-13T14:57:11.768 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.768 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.768 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:57:11.768 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.350321+0000", 2024-09-13T14:57:11.768 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7285000000000004e-05 2024-09-13T14:57:11.768 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.768 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.768 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:57:11.768 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.351277+0000", 2024-09-13T14:57:11.768 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000955977 2024-09-13T14:57:11.768 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:11.768 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:57:11.768 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:11.769 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.769 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.769 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:72521 9.13 9:cd0cc28e:::benchmark_data_smithi102_348563_object72520:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-09-13T14:57:11.769 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:51:46.367757+0000", 2024-09-13T14:57:11.769 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 325.340634228, 2024-09-13T14:57:11.769 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.984496971, 2024-09-13T14:57:11.769 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:57:11.769 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:11.769 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:57:11.769 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:57:11.769 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:57:11.769 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 72521 2024-09-13T14:57:11.769 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.770 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:57:11.770 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.770 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:57:11.770 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.367757+0000", 2024-09-13T14:57:11.770 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:11.770 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.770 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.770 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:57:11.770 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.367757+0000", 2024-09-13T14:57:11.770 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:11.770 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.770 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.770 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:57:11.771 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.367755+0000", 2024-09-13T14:57:11.771 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:57:11.771 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.771 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.771 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:57:11.771 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.367856+0000", 2024-09-13T14:57:11.771 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000100467 2024-09-13T14:57:11.771 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.771 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.771 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:57:11.771 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.367865+0000", 2024-09-13T14:57:11.771 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.833e-06 2024-09-13T14:57:11.771 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.772 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.772 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:57:11.772 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.367871+0000", 2024-09-13T14:57:11.772 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3180000000000003e-06 2024-09-13T14:57:11.772 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.772 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.772 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.772 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.368022+0000", 2024-09-13T14:57:11.772 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015103 2024-09-13T14:57:11.772 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.772 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.773 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.773 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.368032+0000", 2024-09-13T14:57:11.773 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.024e-05 2024-09-13T14:57:11.773 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.773 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.773 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.773 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.057022+0000", 2024-09-13T14:57:11.773 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.68899013099999995 2024-09-13T14:57:11.773 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.773 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.773 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.773 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.057033+0000", 2024-09-13T14:57:11.774 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0335e-05 2024-09-13T14:57:11.774 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.774 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.774 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.774 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.313958+0000", 2024-09-13T14:57:11.774 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.256924937 2024-09-13T14:57:11.774 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.774 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.774 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:57:11.774 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.314126+0000", 2024-09-13T14:57:11.774 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000167978 2024-09-13T14:57:11.774 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.774 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.774 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:57:11.775 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.314336+0000", 2024-09-13T14:57:11.775 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000209855 2024-09-13T14:57:11.775 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.775 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.775 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:57:11.775 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.320183+0000", 2024-09-13T14:57:11.775 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0058473869999999999 2024-09-13T14:57:11.775 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.775 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.775 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.775 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.325473+0000", 2024-09-13T14:57:11.775 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0052903059999999998 2024-09-13T14:57:11.776 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.776 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.776 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.776 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.351404+0000", 2024-09-13T14:57:11.776 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.025930947999999999 2024-09-13T14:57:11.776 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.776 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.776 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:57:11.776 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.351459+0000", 2024-09-13T14:57:11.776 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5000999999999997e-05 2024-09-13T14:57:11.776 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.776 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.776 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:57:11.777 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.352254+0000", 2024-09-13T14:57:11.777 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00079445299999999998 2024-09-13T14:57:11.777 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:11.777 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:57:11.777 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:11.777 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.777 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.777 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:72531 9.13 9:cd5ddef9:::benchmark_data_smithi102_348563_object72530:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-09-13T14:57:11.777 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:51:46.381783+0000", 2024-09-13T14:57:11.777 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 325.32660789699997, 2024-09-13T14:57:11.777 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.971096263, 2024-09-13T14:57:11.777 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:57:11.778 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:11.778 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:57:11.778 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:57:11.778 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:57:11.778 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 72531 2024-09-13T14:57:11.778 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.778 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:57:11.778 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.778 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:57:11.778 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.381783+0000", 2024-09-13T14:57:11.778 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:11.778 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.778 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.779 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:57:11.779 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.381783+0000", 2024-09-13T14:57:11.779 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:11.779 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.779 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.779 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:57:11.779 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.381781+0000", 2024-09-13T14:57:11.779 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:57:11.779 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.779 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.779 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:57:11.780 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.381839+0000", 2024-09-13T14:57:11.780 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8542999999999998e-05 2024-09-13T14:57:11.780 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.780 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.780 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:57:11.780 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.381853+0000", 2024-09-13T14:57:11.780 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3885e-05 2024-09-13T14:57:11.780 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.780 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.780 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:57:11.780 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.381861+0000", 2024-09-13T14:57:11.780 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7260000000000002e-06 2024-09-13T14:57:11.781 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.781 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.781 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.781 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.381922+0000", 2024-09-13T14:57:11.781 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0841000000000002e-05 2024-09-13T14:57:11.781 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.781 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.781 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.781 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.381931+0000", 2024-09-13T14:57:11.781 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.3510000000000008e-06 2024-09-13T14:57:11.781 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.781 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.782 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.782 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.057056+0000", 2024-09-13T14:57:11.782 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.67512471100000004 2024-09-13T14:57:11.782 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.782 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.782 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.782 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.057061+0000", 2024-09-13T14:57:11.782 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6009999999999997e-06 2024-09-13T14:57:11.782 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.782 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.782 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.782 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.315777+0000", 2024-09-13T14:57:11.782 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2587159479999999 2024-09-13T14:57:11.783 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.783 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.783 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:57:11.783 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.315929+0000", 2024-09-13T14:57:11.783 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015216700000000001 2024-09-13T14:57:11.783 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.783 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.783 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:57:11.783 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.316179+0000", 2024-09-13T14:57:11.783 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000250709 2024-09-13T14:57:11.783 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.783 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.783 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:57:11.784 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.320189+0000", 2024-09-13T14:57:11.784 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0040100420000000001 2024-09-13T14:57:11.784 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.784 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.784 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.784 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.333905+0000", 2024-09-13T14:57:11.784 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013715283999999999 2024-09-13T14:57:11.784 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.784 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.784 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.784 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.352333+0000", 2024-09-13T14:57:11.784 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018428402999999999 2024-09-13T14:57:11.785 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.785 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.785 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:57:11.785 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.352367+0000", 2024-09-13T14:57:11.785 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.3874999999999997e-05 2024-09-13T14:57:11.785 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.785 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.785 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:57:11.785 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.352879+0000", 2024-09-13T14:57:11.785 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00051223200000000005 2024-09-13T14:57:11.785 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:11.785 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:57:11.785 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:11.786 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.786 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.786 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:72556 9.13 9:cf932f14:::benchmark_data_smithi102_348563_object72555:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-09-13T14:57:11.786 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:51:46.403040+0000", 2024-09-13T14:57:11.786 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 325.305350675, 2024-09-13T14:57:11.786 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9505509889999999, 2024-09-13T14:57:11.786 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:57:11.786 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:11.786 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:57:11.786 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:57:11.786 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:57:11.786 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 72556 2024-09-13T14:57:11.786 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.786 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:57:11.787 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.787 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:57:11.787 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.403040+0000", 2024-09-13T14:57:11.787 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:11.787 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.787 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.787 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:57:11.787 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.403040+0000", 2024-09-13T14:57:11.787 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:11.787 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.787 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.787 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:57:11.787 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.403039+0000", 2024-09-13T14:57:11.788 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:57:11.788 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.788 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.788 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:57:11.788 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.403067+0000", 2024-09-13T14:57:11.788 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7789000000000001e-05 2024-09-13T14:57:11.788 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.788 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.788 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:57:11.788 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.403086+0000", 2024-09-13T14:57:11.788 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9351000000000002e-05 2024-09-13T14:57:11.788 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.788 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.789 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:57:11.789 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.403094+0000", 2024-09-13T14:57:11.789 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5050000000000001e-06 2024-09-13T14:57:11.789 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.789 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.789 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.789 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.403146+0000", 2024-09-13T14:57:11.789 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2153e-05 2024-09-13T14:57:11.789 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.789 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.789 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.789 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.403155+0000", 2024-09-13T14:57:11.789 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.3000000000000007e-06 2024-09-13T14:57:11.789 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.790 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.790 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.790 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.057087+0000", 2024-09-13T14:57:11.790 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.65393121300000001 2024-09-13T14:57:11.790 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.790 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.790 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.790 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.057095+0000", 2024-09-13T14:57:11.790 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8310000000000007e-06 2024-09-13T14:57:11.790 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.790 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.790 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.790 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.317544+0000", 2024-09-13T14:57:11.791 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.260448537 2024-09-13T14:57:11.791 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.791 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.791 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:57:11.791 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.317701+0000", 2024-09-13T14:57:11.791 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000157405 2024-09-13T14:57:11.791 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.791 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.791 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:57:11.791 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.317872+0000", 2024-09-13T14:57:11.791 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0001707 2024-09-13T14:57:11.791 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.791 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.792 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:57:11.792 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.320245+0000", 2024-09-13T14:57:11.792 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0023728360000000001 2024-09-13T14:57:11.792 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.792 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.792 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.792 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.351338+0000", 2024-09-13T14:57:11.792 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.031092813 2024-09-13T14:57:11.792 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.792 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.792 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.792 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.352986+0000", 2024-09-13T14:57:11.793 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0016485619999999999 2024-09-13T14:57:11.793 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.793 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.793 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:57:11.793 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.353017+0000", 2024-09-13T14:57:11.793 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0880999999999998e-05 2024-09-13T14:57:11.793 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.793 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.793 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:57:11.793 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.353591+0000", 2024-09-13T14:57:11.793 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00057408700000000003 2024-09-13T14:57:11.793 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:11.793 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:57:11.794 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:11.794 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.794 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.794 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:72572 9.13 9:ce9461e5:::benchmark_data_smithi102_348563_object72571:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-09-13T14:57:11.794 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:51:46.420582+0000", 2024-09-13T14:57:11.794 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 325.287808466, 2024-09-13T14:57:11.794 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9336864380000001, 2024-09-13T14:57:11.794 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:57:11.794 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:11.794 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:57:11.794 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:57:11.794 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:57:11.794 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 72572 2024-09-13T14:57:11.795 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.795 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:57:11.795 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.795 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:57:11.795 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.420582+0000", 2024-09-13T14:57:11.795 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:11.795 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.795 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.795 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:57:11.795 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.420582+0000", 2024-09-13T14:57:11.795 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:11.795 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.795 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.795 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:57:11.796 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.420581+0000", 2024-09-13T14:57:11.796 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:57:11.796 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.796 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.796 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:57:11.796 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.420633+0000", 2024-09-13T14:57:11.796 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2237999999999997e-05 2024-09-13T14:57:11.796 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.796 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.796 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:57:11.796 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.420644+0000", 2024-09-13T14:57:11.796 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1249e-05 2024-09-13T14:57:11.796 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.796 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.797 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:57:11.797 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.420650+0000", 2024-09-13T14:57:11.797 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4500000000000003e-06 2024-09-13T14:57:11.797 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.797 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.797 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.797 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.420696+0000", 2024-09-13T14:57:11.797 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5902999999999997e-05 2024-09-13T14:57:11.797 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.797 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.797 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.797 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.420704+0000", 2024-09-13T14:57:11.797 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.6559999999999998e-06 2024-09-13T14:57:11.798 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.798 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.798 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.798 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.057119+0000", 2024-09-13T14:57:11.798 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.63641496200000003 2024-09-13T14:57:11.798 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.798 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.798 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.798 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.057124+0000", 2024-09-13T14:57:11.798 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5920000000000002e-06 2024-09-13T14:57:11.798 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.798 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.798 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.799 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.318936+0000", 2024-09-13T14:57:11.799 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2618115219999999 2024-09-13T14:57:11.799 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.799 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.799 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:57:11.799 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.319072+0000", 2024-09-13T14:57:11.799 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013614700000000001 2024-09-13T14:57:11.799 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.799 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.799 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:57:11.799 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.319195+0000", 2024-09-13T14:57:11.799 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012294199999999999 2024-09-13T14:57:11.799 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.800 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.800 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:57:11.800 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.323229+0000", 2024-09-13T14:57:11.800 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0040348149999999998 2024-09-13T14:57:11.800 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.800 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.800 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.800 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.352299+0000", 2024-09-13T14:57:11.800 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.029069787 2024-09-13T14:57:11.800 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.800 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.800 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.801 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.353679+0000", 2024-09-13T14:57:11.801 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0013798129999999999 2024-09-13T14:57:11.801 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.801 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.801 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:57:11.801 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.353708+0000", 2024-09-13T14:57:11.801 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9454e-05 2024-09-13T14:57:11.801 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.801 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.801 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:57:11.801 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.354269+0000", 2024-09-13T14:57:11.801 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00056036699999999998 2024-09-13T14:57:11.801 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:11.802 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:57:11.802 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:11.802 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.802 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.802 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:113330 9.17 9:e8bce8ce:::benchmark_data_smithi102_348563_object113329:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e402)", 2024-09-13T14:57:11.802 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:52:26.477514+0000", 2024-09-13T14:57:11.802 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 285.23087695599997, 2024-09-13T14:57:11.802 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0168906770000001, 2024-09-13T14:57:11.802 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:57:11.802 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:11.802 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:57:11.803 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:57:11.803 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:57:11.803 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113330 2024-09-13T14:57:11.803 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.803 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:57:11.803 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.803 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:57:11.803 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.477514+0000", 2024-09-13T14:57:11.803 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:11.803 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.803 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.803 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:57:11.803 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.477514+0000", 2024-09-13T14:57:11.804 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:11.804 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.804 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.804 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:57:11.804 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.477512+0000", 2024-09-13T14:57:11.804 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:57:11.804 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.804 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.804 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:57:11.804 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.477630+0000", 2024-09-13T14:57:11.804 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011756000000000001 2024-09-13T14:57:11.804 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.805 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.805 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:57:11.805 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.477642+0000", 2024-09-13T14:57:11.805 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2492e-05 2024-09-13T14:57:11.805 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.805 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.805 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:57:11.805 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.477648+0000", 2024-09-13T14:57:11.805 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5230000000000003e-06 2024-09-13T14:57:11.805 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.805 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.805 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.805 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.477703+0000", 2024-09-13T14:57:11.806 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5500000000000001e-05 2024-09-13T14:57:11.806 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.806 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.806 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.806 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.477715+0000", 2024-09-13T14:57:11.806 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1083e-05 2024-09-13T14:57:11.806 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.806 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.806 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.806 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:27.337497+0000", 2024-09-13T14:57:11.806 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.85978237800000001 2024-09-13T14:57:11.806 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.807 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.807 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.807 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:27.337508+0000", 2024-09-13T14:57:11.807 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1004e-05 2024-09-13T14:57:11.807 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.807 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.807 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.807 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.485507+0000", 2024-09-13T14:57:11.807 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.147998759 2024-09-13T14:57:11.807 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.807 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.807 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:57:11.807 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.485653+0000", 2024-09-13T14:57:11.808 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014649300000000001 2024-09-13T14:57:11.808 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.808 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.808 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:57:11.808 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.485818+0000", 2024-09-13T14:57:11.808 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016457 2024-09-13T14:57:11.808 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.808 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.808 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.808 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.492578+0000", 2024-09-13T14:57:11.808 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0067599959999999999 2024-09-13T14:57:11.808 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.808 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.808 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.809 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.492640+0000", 2024-09-13T14:57:11.809 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2607999999999994e-05 2024-09-13T14:57:11.809 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.809 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.809 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:57:11.809 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.492652+0000", 2024-09-13T14:57:11.809 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2021000000000001e-05 2024-09-13T14:57:11.809 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.809 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.809 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:57:11.809 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.492691+0000", 2024-09-13T14:57:11.809 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8656999999999999e-05 2024-09-13T14:57:11.810 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.810 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.810 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:57:11.810 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.494405+0000", 2024-09-13T14:57:11.810 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0017135500000000001 2024-09-13T14:57:11.810 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:11.810 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:57:11.810 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:11.810 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.810 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.810 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:113341 9.17 9:ead480fa:::benchmark_data_smithi102_348563_object113340:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e402)", 2024-09-13T14:57:11.810 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:52:26.502483+0000", 2024-09-13T14:57:11.811 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 285.20590811800002, 2024-09-13T14:57:11.811 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0150210039999998, 2024-09-13T14:57:11.811 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:57:11.811 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:11.811 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:57:11.811 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:57:11.811 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:57:11.811 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113341 2024-09-13T14:57:11.811 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.811 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:57:11.811 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.811 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:57:11.811 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.502483+0000", 2024-09-13T14:57:11.812 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:11.812 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.812 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.812 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:57:11.812 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.502483+0000", 2024-09-13T14:57:11.812 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:11.812 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.812 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.812 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:57:11.812 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.502481+0000", 2024-09-13T14:57:11.812 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:57:11.812 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.813 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.813 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:57:11.813 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.502555+0000", 2024-09-13T14:57:11.813 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4099999999999999e-05 2024-09-13T14:57:11.813 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.813 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.813 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:57:11.813 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.502567+0000", 2024-09-13T14:57:11.813 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1698999999999999e-05 2024-09-13T14:57:11.813 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.813 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.813 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:57:11.813 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.502574+0000", 2024-09-13T14:57:11.814 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2570000000000001e-06 2024-09-13T14:57:11.814 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.814 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.814 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.814 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.502625+0000", 2024-09-13T14:57:11.814 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0603000000000003e-05 2024-09-13T14:57:11.814 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.814 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.814 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.814 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.502636+0000", 2024-09-13T14:57:11.814 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.082e-05 2024-09-13T14:57:11.814 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.814 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.814 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.815 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:27.337554+0000", 2024-09-13T14:57:11.815 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.83491843899999996 2024-09-13T14:57:11.815 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.815 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.815 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.815 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:27.337562+0000", 2024-09-13T14:57:11.815 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1960000000000008e-06 2024-09-13T14:57:11.815 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.815 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.815 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.815 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.489118+0000", 2024-09-13T14:57:11.815 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1515562509999999 2024-09-13T14:57:11.815 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.816 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.816 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:57:11.816 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.489230+0000", 2024-09-13T14:57:11.816 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011199100000000001 2024-09-13T14:57:11.816 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.816 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.816 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:57:11.816 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.489333+0000", 2024-09-13T14:57:11.816 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000102605 2024-09-13T14:57:11.816 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.816 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.816 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:57:11.816 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.498423+0000", 2024-09-13T14:57:11.816 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0090894700000000005 2024-09-13T14:57:11.817 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.817 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.817 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.817 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.498475+0000", 2024-09-13T14:57:11.817 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2296999999999998e-05 2024-09-13T14:57:11.817 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.817 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.817 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.817 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.515297+0000", 2024-09-13T14:57:11.817 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016822522999999999 2024-09-13T14:57:11.817 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.817 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.817 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:57:11.818 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.515353+0000", 2024-09-13T14:57:11.818 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5451000000000002e-05 2024-09-13T14:57:11.818 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.818 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.818 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:57:11.818 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.517504+0000", 2024-09-13T14:57:11.818 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0021509900000000002 2024-09-13T14:57:11.818 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:11.818 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:57:11.818 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:11.818 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.818 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.818 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:113388 9.17 9:ededae97:::benchmark_data_smithi102_348563_object113387:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e402)", 2024-09-13T14:57:11.819 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:52:26.574318+0000", 2024-09-13T14:57:11.819 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 285.134072635, 2024-09-13T14:57:11.819 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.944988825, 2024-09-13T14:57:11.819 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:57:11.819 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:11.819 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:57:11.819 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:57:11.819 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:57:11.819 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113388 2024-09-13T14:57:11.819 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.819 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:57:11.819 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.819 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:57:11.820 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.574318+0000", 2024-09-13T14:57:11.820 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:11.820 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.820 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.820 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:57:11.820 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.574318+0000", 2024-09-13T14:57:11.820 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:11.820 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.820 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.820 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:57:11.820 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.574316+0000", 2024-09-13T14:57:11.820 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:57:11.821 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.821 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.821 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:57:11.821 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.574361+0000", 2024-09-13T14:57:11.821 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5012000000000002e-05 2024-09-13T14:57:11.821 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.821 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.821 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:57:11.821 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.574373+0000", 2024-09-13T14:57:11.823 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1810999999999999e-05 2024-09-13T14:57:11.823 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.823 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.823 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:57:11.823 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.574379+0000", 2024-09-13T14:57:11.823 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.348e-06 2024-09-13T14:57:11.823 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.823 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.823 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.823 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.574431+0000", 2024-09-13T14:57:11.823 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1205000000000002e-05 2024-09-13T14:57:11.824 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.824 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.824 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.824 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.574443+0000", 2024-09-13T14:57:11.824 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2842e-05 2024-09-13T14:57:11.824 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.824 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.824 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.824 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:27.337587+0000", 2024-09-13T14:57:11.824 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.76314326399999999 2024-09-13T14:57:11.824 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.824 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.824 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.824 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:27.337595+0000", 2024-09-13T14:57:11.825 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.4600000000000003e-06 2024-09-13T14:57:11.825 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.825 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.825 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.825 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.490785+0000", 2024-09-13T14:57:11.825 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1531900020000001 2024-09-13T14:57:11.825 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.825 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.825 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:57:11.825 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.490897+0000", 2024-09-13T14:57:11.825 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011162699999999999 2024-09-13T14:57:11.825 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.825 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.826 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:57:11.826 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.491023+0000", 2024-09-13T14:57:11.826 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012667900000000001 2024-09-13T14:57:11.826 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.826 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.826 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:57:11.826 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.498430+0000", 2024-09-13T14:57:11.826 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00740619 2024-09-13T14:57:11.826 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.826 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.826 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.826 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.498510+0000", 2024-09-13T14:57:11.826 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0636999999999994e-05 2024-09-13T14:57:11.827 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.827 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.827 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.827 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.517640+0000", 2024-09-13T14:57:11.827 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.019130004999999999 2024-09-13T14:57:11.827 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.827 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.827 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:57:11.827 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.517695+0000", 2024-09-13T14:57:11.827 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4718e-05 2024-09-13T14:57:11.827 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.827 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.827 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:57:11.827 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.519307+0000", 2024-09-13T14:57:11.828 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0016120279999999999 2024-09-13T14:57:11.828 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:11.828 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:57:11.828 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:11.828 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.828 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.828 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:113411 9.17 9:e864c0f1:::benchmark_data_smithi102_348563_object113410:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e402)", 2024-09-13T14:57:11.828 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:52:26.602093+0000", 2024-09-13T14:57:11.828 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 285.10629756600002, 2024-09-13T14:57:11.828 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.921966007, 2024-09-13T14:57:11.828 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:57:11.828 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:11.828 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:57:11.829 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:57:11.829 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:57:11.829 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113411 2024-09-13T14:57:11.829 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.829 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:57:11.829 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.829 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:57:11.829 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.602093+0000", 2024-09-13T14:57:11.829 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:11.829 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.829 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.829 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:57:11.829 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.602093+0000", 2024-09-13T14:57:11.829 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:11.830 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.830 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.830 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:57:11.830 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.602092+0000", 2024-09-13T14:57:11.830 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:57:11.830 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.830 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.830 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:57:11.830 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.602155+0000", 2024-09-13T14:57:11.830 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3108999999999995e-05 2024-09-13T14:57:11.830 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.830 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.830 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:57:11.830 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.602169+0000", 2024-09-13T14:57:11.831 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3198e-05 2024-09-13T14:57:11.831 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.831 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.831 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:57:11.831 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.602176+0000", 2024-09-13T14:57:11.831 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.238e-06 2024-09-13T14:57:11.831 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.831 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.831 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.831 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.602227+0000", 2024-09-13T14:57:11.831 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1411e-05 2024-09-13T14:57:11.831 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.831 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.832 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.832 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.602236+0000", 2024-09-13T14:57:11.832 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1579999999999994e-06 2024-09-13T14:57:11.832 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.832 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.832 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.832 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:27.337620+0000", 2024-09-13T14:57:11.832 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.73538361699999999 2024-09-13T14:57:11.832 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.832 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.832 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.832 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:27.337628+0000", 2024-09-13T14:57:11.832 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8730000000000007e-06 2024-09-13T14:57:11.832 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.833 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.833 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.833 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.494470+0000", 2024-09-13T14:57:11.833 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.15684242 2024-09-13T14:57:11.833 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.833 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.833 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:57:11.833 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.494630+0000", 2024-09-13T14:57:11.833 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015944 2024-09-13T14:57:11.833 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.833 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.833 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:57:11.833 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.494786+0000", 2024-09-13T14:57:11.834 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000155909 2024-09-13T14:57:11.834 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.834 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.834 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:57:11.834 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.498518+0000", 2024-09-13T14:57:11.834 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0037322029999999999 2024-09-13T14:57:11.834 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.834 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.834 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.834 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.509421+0000", 2024-09-13T14:57:11.834 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010903256 2024-09-13T14:57:11.834 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.834 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.834 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.835 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.521680+0000", 2024-09-13T14:57:11.835 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012258929 2024-09-13T14:57:11.835 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.835 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.835 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:57:11.835 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.521736+0000", 2024-09-13T14:57:11.835 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5813999999999998e-05 2024-09-13T14:57:11.835 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.835 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.835 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:57:11.835 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.524059+0000", 2024-09-13T14:57:11.835 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0023235220000000002 2024-09-13T14:57:11.835 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:11.836 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:57:11.836 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:11.836 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.836 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.836 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:113426 9.17 9:ee9b3220:::benchmark_data_smithi102_348563_object113425:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e402)", 2024-09-13T14:57:11.836 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:52:26.616755+0000", 2024-09-13T14:57:11.836 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 285.09163605700002, 2024-09-13T14:57:11.836 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.909422927, 2024-09-13T14:57:11.836 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:57:11.836 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:11.836 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:57:11.836 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:57:11.836 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:57:11.837 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113426 2024-09-13T14:57:11.837 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.837 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:57:11.837 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.837 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:57:11.837 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.616755+0000", 2024-09-13T14:57:11.837 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:11.837 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.837 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.837 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:57:11.837 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.616755+0000", 2024-09-13T14:57:11.837 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:11.837 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.838 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.838 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:57:11.838 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.616754+0000", 2024-09-13T14:57:11.838 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:57:11.838 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.838 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.838 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:57:11.838 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.616878+0000", 2024-09-13T14:57:11.838 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000124043 2024-09-13T14:57:11.838 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.838 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.838 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:57:11.838 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.616888+0000", 2024-09-13T14:57:11.838 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0301e-05 2024-09-13T14:57:11.839 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.839 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.839 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:57:11.839 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.616895+0000", 2024-09-13T14:57:11.839 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6649999999999997e-06 2024-09-13T14:57:11.839 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.839 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.839 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.839 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.617105+0000", 2024-09-13T14:57:11.839 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00021071100000000001 2024-09-13T14:57:11.839 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.839 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.839 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.840 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.617115+0000", 2024-09-13T14:57:11.840 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.8849999999999994e-06 2024-09-13T14:57:11.840 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.840 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.840 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.840 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:27.337655+0000", 2024-09-13T14:57:11.840 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.72054000100000004 2024-09-13T14:57:11.840 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.840 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.840 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.840 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:27.337663+0000", 2024-09-13T14:57:11.840 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7230000000000004e-06 2024-09-13T14:57:11.841 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.841 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.841 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.841 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.496533+0000", 2024-09-13T14:57:11.841 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.158870515 2024-09-13T14:57:11.841 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.841 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.841 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:57:11.841 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.496848+0000", 2024-09-13T14:57:11.841 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00031459100000000002 2024-09-13T14:57:11.841 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.841 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.842 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:57:11.842 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.496952+0000", 2024-09-13T14:57:11.842 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000104147 2024-09-13T14:57:11.842 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.842 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.842 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:57:11.842 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.503972+0000", 2024-09-13T14:57:11.842 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0070199650000000004 2024-09-13T14:57:11.842 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.843 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.843 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.843 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.517572+0000", 2024-09-13T14:57:11.843 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013599546000000001 2024-09-13T14:57:11.843 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.843 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.843 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.843 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.524126+0000", 2024-09-13T14:57:11.843 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0065545020000000002 2024-09-13T14:57:11.843 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.843 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.843 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:57:11.843 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.524182+0000", 2024-09-13T14:57:11.844 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5952999999999998e-05 2024-09-13T14:57:11.844 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.844 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.844 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:57:11.844 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.526178+0000", 2024-09-13T14:57:11.844 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001995564 2024-09-13T14:57:11.844 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:11.844 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:57:11.844 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:11.844 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.844 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.845 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36738.0:9805 10.1 10:87853b1e:::benchmark_data_smithi102_377304_object9804:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e441)", 2024-09-13T14:57:11.845 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:55:16.922955+0000", 2024-09-13T14:57:11.845 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 114.785436299, 2024-09-13T14:57:11.845 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.271878337, 2024-09-13T14:57:11.845 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:57:11.845 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-13T14:57:11.845 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:57:11.845 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36738", 2024-09-13T14:57:11.845 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/2915890700", 2024-09-13T14:57:11.845 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 9805 2024-09-13T14:57:11.846 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.846 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:57:11.846 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.846 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:57:11.846 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.922955+0000", 2024-09-13T14:57:11.846 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:11.846 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.846 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.846 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:57:11.846 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.922955+0000", 2024-09-13T14:57:11.846 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:11.846 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.846 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.846 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:57:11.847 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.922952+0000", 2024-09-13T14:57:11.847 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999971 2024-09-13T14:57:11.847 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.847 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.847 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:57:11.847 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.922989+0000", 2024-09-13T14:57:11.847 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7221000000000002e-05 2024-09-13T14:57:11.847 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.847 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.847 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:57:11.847 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928405+0000", 2024-09-13T14:57:11.847 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0054158330000000001 2024-09-13T14:57:11.847 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.848 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.848 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:57:11.848 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928415+0000", 2024-09-13T14:57:11.848 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.7200000000000001e-06 2024-09-13T14:57:11.848 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.848 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.848 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.848 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928621+0000", 2024-09-13T14:57:11.848 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000206126 2024-09-13T14:57:11.848 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.848 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.849 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.849 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928636+0000", 2024-09-13T14:57:11.849 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5101e-05 2024-09-13T14:57:11.849 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.849 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.849 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.849 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:17.919116+0000", 2024-09-13T14:57:11.849 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.99048049599999999 2024-09-13T14:57:11.849 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.849 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.849 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.849 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:17.919126+0000", 2024-09-13T14:57:11.850 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6029999999999994e-06 2024-09-13T14:57:11.850 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.850 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.850 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.850 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:19.173075+0000", 2024-09-13T14:57:11.850 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.253948692 2024-09-13T14:57:11.850 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.850 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.850 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:57:11.850 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:19.174363+0000", 2024-09-13T14:57:11.850 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001287898 2024-09-13T14:57:11.850 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.850 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.851 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.851 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:19.178513+0000", 2024-09-13T14:57:11.851 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.004150089 2024-09-13T14:57:11.851 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.851 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.851 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for degraded object", 2024-09-13T14:57:11.851 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:19.178580+0000", 2024-09-13T14:57:11.851 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7539000000000005e-05 2024-09-13T14:57:11.851 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.851 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.851 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.851 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:19.194762+0000", 2024-09-13T14:57:11.852 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016181389000000001 2024-09-13T14:57:11.852 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.852 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.852 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:57:11.852 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:19.194833+0000", 2024-09-13T14:57:11.852 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1254999999999995e-05 2024-09-13T14:57:11.852 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:11.852 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:57:11.852 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:11.852 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.852 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.852 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36738.0:9808 10.c 10:318791bf:::benchmark_data_smithi102_377304_object9807:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e441)", 2024-09-13T14:57:11.853 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:55:16.928442+0000", 2024-09-13T14:57:11.853 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 114.779949092, 2024-09-13T14:57:11.853 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.037536926, 2024-09-13T14:57:11.853 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:57:11.853 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:11.853 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:57:11.853 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36738", 2024-09-13T14:57:11.853 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/2915890700", 2024-09-13T14:57:11.853 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 9808 2024-09-13T14:57:11.853 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.853 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:57:11.853 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.854 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:57:11.854 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928442+0000", 2024-09-13T14:57:11.854 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:11.854 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.854 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.854 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:57:11.854 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928442+0000", 2024-09-13T14:57:11.854 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:11.854 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.854 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.854 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:57:11.855 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928440+0000", 2024-09-13T14:57:11.855 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:57:11.855 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.855 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.855 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:57:11.855 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928475+0000", 2024-09-13T14:57:11.855 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4659999999999997e-05 2024-09-13T14:57:11.855 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.855 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.855 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:57:11.855 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928485+0000", 2024-09-13T14:57:11.855 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.5189999999999998e-06 2024-09-13T14:57:11.855 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.856 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.856 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:57:11.856 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928489+0000", 2024-09-13T14:57:11.856 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8339999999999998e-06 2024-09-13T14:57:11.856 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.856 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.856 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.856 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.929346+0000", 2024-09-13T14:57:11.856 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00085621199999999997 2024-09-13T14:57:11.856 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.856 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.856 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.856 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.929356+0000", 2024-09-13T14:57:11.857 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.049e-05 2024-09-13T14:57:11.857 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.857 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.857 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.857 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:17.918693+0000", 2024-09-13T14:57:11.857 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.98933645100000001 2024-09-13T14:57:11.857 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.857 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.857 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.857 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:17.918705+0000", 2024-09-13T14:57:11.857 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2299e-05 2024-09-13T14:57:11.857 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.857 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.858 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.858 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:18.957681+0000", 2024-09-13T14:57:11.858 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0389758419999999 2024-09-13T14:57:11.858 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.858 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.858 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:57:11.858 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:18.957841+0000", 2024-09-13T14:57:11.858 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016063199999999999 2024-09-13T14:57:11.858 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.858 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.858 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,3", 2024-09-13T14:57:11.858 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:18.958022+0000", 2024-09-13T14:57:11.858 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00018070299999999999 2024-09-13T14:57:11.858 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.859 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.859 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:57:11.859 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:18.959663+0000", 2024-09-13T14:57:11.859 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001641188 2024-09-13T14:57:11.859 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.859 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.859 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.859 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:18.963832+0000", 2024-09-13T14:57:11.859 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0041683049999999998 2024-09-13T14:57:11.859 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.859 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.859 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.860 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:18.965594+0000", 2024-09-13T14:57:11.860 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0017620800000000001 2024-09-13T14:57:11.860 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.860 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.860 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:57:11.860 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:18.965641+0000", 2024-09-13T14:57:11.860 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6956999999999998e-05 2024-09-13T14:57:11.860 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.860 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.860 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:57:11.860 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:18.965979+0000", 2024-09-13T14:57:11.860 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000338098 2024-09-13T14:57:11.860 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:11.861 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:57:11.861 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:11.861 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.861 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.861 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36738.0:9810 10.1 10:8c8c1ace:::benchmark_data_smithi102_377304_object9809:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e441)", 2024-09-13T14:57:11.861 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:55:16.928494+0000", 2024-09-13T14:57:11.861 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 114.77989696100001, 2024-09-13T14:57:11.861 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2640140639999999, 2024-09-13T14:57:11.861 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:57:11.861 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:11.861 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:57:11.861 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36738", 2024-09-13T14:57:11.862 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/2915890700", 2024-09-13T14:57:11.862 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 9810 2024-09-13T14:57:11.862 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.862 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:57:11.862 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.862 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:57:11.862 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928494+0000", 2024-09-13T14:57:11.862 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:11.862 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.862 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.862 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:57:11.862 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928494+0000", 2024-09-13T14:57:11.862 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:11.863 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.863 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.863 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:57:11.863 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928493+0000", 2024-09-13T14:57:11.863 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:57:11.863 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.863 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.863 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:57:11.863 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928527+0000", 2024-09-13T14:57:11.863 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.3821e-05 2024-09-13T14:57:11.863 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.863 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.863 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:57:11.864 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928533+0000", 2024-09-13T14:57:11.864 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0889999999999996e-06 2024-09-13T14:57:11.864 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.864 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.864 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:57:11.864 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928538+0000", 2024-09-13T14:57:11.864 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2989999999999998e-06 2024-09-13T14:57:11.864 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.864 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.864 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.864 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928660+0000", 2024-09-13T14:57:11.864 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000122012 2024-09-13T14:57:11.864 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.865 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.865 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.865 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928669+0000", 2024-09-13T14:57:11.865 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1339999999999993e-06 2024-09-13T14:57:11.865 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.865 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.865 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.865 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:17.919149+0000", 2024-09-13T14:57:11.865 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.99048049699999996 2024-09-13T14:57:11.865 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.865 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.865 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.865 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:17.919157+0000", 2024-09-13T14:57:11.866 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9629999999999998e-06 2024-09-13T14:57:11.866 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.866 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.866 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.866 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:19.178662+0000", 2024-09-13T14:57:11.866 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2595047319999999 2024-09-13T14:57:11.866 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.866 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.866 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:57:11.866 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:19.178786+0000", 2024-09-13T14:57:11.866 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000124051 2024-09-13T14:57:11.866 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.866 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.867 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:57:11.867 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:19.178891+0000", 2024-09-13T14:57:11.867 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000105219 2024-09-13T14:57:11.867 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.867 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.867 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:57:11.867 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:19.181396+0000", 2024-09-13T14:57:11.867 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0025047469999999999 2024-09-13T14:57:11.867 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.867 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.867 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.867 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:19.182895+0000", 2024-09-13T14:57:11.867 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001499055 2024-09-13T14:57:11.868 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.868 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.868 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.868 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:19.191955+0000", 2024-09-13T14:57:11.868 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0090602289999999995 2024-09-13T14:57:11.868 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.868 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.868 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:57:11.868 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:19.191991+0000", 2024-09-13T14:57:11.868 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6177999999999998e-05 2024-09-13T14:57:11.868 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.868 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.868 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:57:11.868 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:19.192508+0000", 2024-09-13T14:57:11.869 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00051648799999999995 2024-09-13T14:57:11.869 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:11.869 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:57:11.869 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:11.869 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.869 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.869 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36738.0:9811 10.1 10:8943192a:::benchmark_data_smithi102_377304_object9810:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e441)", 2024-09-13T14:57:11.869 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:55:16.928542+0000", 2024-09-13T14:57:11.869 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 114.779848848, 2024-09-13T14:57:11.869 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.264276787, 2024-09-13T14:57:11.869 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:57:11.869 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:11.869 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:57:11.870 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36738", 2024-09-13T14:57:11.870 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/2915890700", 2024-09-13T14:57:11.870 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 9811 2024-09-13T14:57:11.870 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.870 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:57:11.870 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.870 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:57:11.870 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928542+0000", 2024-09-13T14:57:11.870 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:11.870 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.870 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.870 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:57:11.870 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928542+0000", 2024-09-13T14:57:11.870 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:11.871 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.871 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.871 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:57:11.871 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928542+0000", 2024-09-13T14:57:11.871 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:57:11.871 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.871 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.871 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:57:11.871 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928568+0000", 2024-09-13T14:57:11.871 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5936999999999999e-05 2024-09-13T14:57:11.871 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.871 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.871 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:57:11.872 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928573+0000", 2024-09-13T14:57:11.872 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4210000000000003e-06 2024-09-13T14:57:11.872 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.872 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.872 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:57:11.872 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928577+0000", 2024-09-13T14:57:11.872 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.799e-06 2024-09-13T14:57:11.872 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.872 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.872 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.872 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928692+0000", 2024-09-13T14:57:11.872 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000115659 2024-09-13T14:57:11.872 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.873 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.873 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.873 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928703+0000", 2024-09-13T14:57:11.873 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0062e-05 2024-09-13T14:57:11.873 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.873 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.873 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.873 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:17.919520+0000", 2024-09-13T14:57:11.873 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.99081732 2024-09-13T14:57:11.873 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.873 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.873 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.873 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:17.919531+0000", 2024-09-13T14:57:11.873 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1484999999999999e-05 2024-09-13T14:57:11.874 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.874 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.874 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.874 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:19.179311+0000", 2024-09-13T14:57:11.874 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2597794689999999 2024-09-13T14:57:11.874 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.874 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.874 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:57:11.874 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:19.179395+0000", 2024-09-13T14:57:11.874 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.4116000000000006e-05 2024-09-13T14:57:11.874 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.874 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.874 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:57:11.875 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:19.179496+0000", 2024-09-13T14:57:11.875 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000101041 2024-09-13T14:57:11.875 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.875 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.875 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:57:11.875 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:19.181404+0000", 2024-09-13T14:57:11.875 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001908332 2024-09-13T14:57:11.875 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.875 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.875 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.875 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:19.182937+0000", 2024-09-13T14:57:11.875 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001533005 2024-09-13T14:57:11.875 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.875 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.876 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.876 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:19.192548+0000", 2024-09-13T14:57:11.876 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0096108989999999991 2024-09-13T14:57:11.876 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.876 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.876 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:57:11.876 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:19.192581+0000", 2024-09-13T14:57:11.876 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.311e-05 2024-09-13T14:57:11.876 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.876 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.876 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:57:11.876 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:19.192819+0000", 2024-09-13T14:57:11.876 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00023754399999999999 2024-09-13T14:57:11.877 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:11.877 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:57:11.877 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:11.877 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.877 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.877 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36738.0:9817 10.c 10:385a7ba4:::benchmark_data_smithi102_377304_object9816:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e441)", 2024-09-13T14:57:11.877 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:55:16.937249+0000", 2024-09-13T14:57:11.877 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 114.77114230799999, 2024-09-13T14:57:11.877 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0291470550000001, 2024-09-13T14:57:11.877 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:57:11.877 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:11.877 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:57:11.877 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36738", 2024-09-13T14:57:11.878 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/2915890700", 2024-09-13T14:57:11.878 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 9817 2024-09-13T14:57:11.878 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.878 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:57:11.878 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.878 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:57:11.878 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.937249+0000", 2024-09-13T14:57:11.878 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:11.878 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.878 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.878 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:57:11.878 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.937249+0000", 2024-09-13T14:57:11.878 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:11.878 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.879 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.879 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:57:11.879 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.937247+0000", 2024-09-13T14:57:11.879 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:57:11.879 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.879 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.879 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:57:11.879 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.937301+0000", 2024-09-13T14:57:11.879 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3604999999999999e-05 2024-09-13T14:57:11.879 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.879 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.879 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:57:11.879 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.937312+0000", 2024-09-13T14:57:11.879 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1126e-05 2024-09-13T14:57:11.880 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.880 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.880 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:57:11.880 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.937317+0000", 2024-09-13T14:57:11.880 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5380000000000002e-06 2024-09-13T14:57:11.880 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.880 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.880 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.880 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.937352+0000", 2024-09-13T14:57:11.880 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5055000000000002e-05 2024-09-13T14:57:11.880 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.880 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.880 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.881 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.937360+0000", 2024-09-13T14:57:11.881 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8080000000000005e-06 2024-09-13T14:57:11.881 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.881 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.881 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.881 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:17.918729+0000", 2024-09-13T14:57:11.881 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.98136912700000001 2024-09-13T14:57:11.881 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.881 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.881 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:57:11.881 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:17.918738+0000", 2024-09-13T14:57:11.881 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.4430000000000003e-06 2024-09-13T14:57:11.881 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.882 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.882 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:11.882 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:18.959065+0000", 2024-09-13T14:57:11.882 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0403277209999999 2024-09-13T14:57:11.882 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.882 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.882 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:57:11.882 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:18.959204+0000", 2024-09-13T14:57:11.882 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013865000000000001 2024-09-13T14:57:11.882 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.882 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.882 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,3", 2024-09-13T14:57:11.882 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:18.959318+0000", 2024-09-13T14:57:11.883 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000114129 2024-09-13T14:57:11.883 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.883 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.883 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:57:11.883 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:18.959805+0000", 2024-09-13T14:57:11.883 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00048650000000000001 2024-09-13T14:57:11.883 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.883 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.883 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.883 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:18.964080+0000", 2024-09-13T14:57:11.883 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0042753269999999998 2024-09-13T14:57:11.883 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.883 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.884 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:11.884 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:18.966033+0000", 2024-09-13T14:57:11.884 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0019524519999999999 2024-09-13T14:57:11.884 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.884 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.884 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:57:11.884 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:18.966088+0000", 2024-09-13T14:57:11.884 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5105999999999997e-05 2024-09-13T14:57:11.884 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:11.884 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:11.884 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:57:11.884 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:18.966396+0000", 2024-09-13T14:57:11.885 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000307972 2024-09-13T14:57:11.885 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:11.885 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:57:11.885 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:11.885 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:11.885 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:57:11.885 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:57:11.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:11.887 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:11.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:11.988 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:12.017 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:12.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:12.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:12.119 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:12.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:12.144 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:12.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:12.246 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:12.270 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:12.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:12.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:12.373 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:12.397 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:12.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:12.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:12.499 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:12.535 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:12.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:12.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:12.637 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:12.667 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:12.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:12.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:12.769 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:12.799 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:12.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:12.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:12.901 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:12.926 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:12.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:12.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:13.028 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:13.073 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:13.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:13.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:13.175 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:13.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:13.208 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:13.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:13.233 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T14:57:13.309 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:13.346 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:13.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:13.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:13.448 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:13.478 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:13.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:13.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:13.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:13 smithi195 ceph-mon[104262]: pgmap v1575: 109 pgs: 109 active+clean; 5.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 887 op/s 2024-09-13T14:57:13.540 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:57:13.575 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:13 smithi102 ceph-mon[110183]: pgmap v1575: 109 pgs: 109 active+clean; 5.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 887 op/s 2024-09-13T14:57:13.575 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:13 smithi102 ceph-mon[115269]: pgmap v1575: 109 pgs: 109 active+clean; 5.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 887 op/s 2024-09-13T14:57:13.580 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:13.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:13.608 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:13.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:13.709 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:13.736 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:13.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:13.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:13.838 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:13.866 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:13.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:13.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:13.968 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:13.993 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:13.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:13.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:14.095 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:14.119 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:14.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:14.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:14.221 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:14.246 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:14.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:14.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:14.348 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:14.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:14.375 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:14.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:14.476 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:14.513 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:14.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:14.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:14.574 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:57:14.615 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:14.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:14.658 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:14.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:14.759 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:14.817 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:14.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:14.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:14.919 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:14.949 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:14.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:14.981 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:14.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:15.083 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:15.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:15.110 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:15.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:15.211 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:15.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:15.238 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:15.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:15.333 INFO:teuthology.orchestra.run.smithi102.stderr:set pool 1 pg_num to 31 2024-09-13T14:57:15.338 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:15.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:15.387 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:15.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:15.388 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:15 smithi195 ceph-mon[104262]: pgmap v1576: 109 pgs: 109 active+clean; 4.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 898 op/s 2024-09-13T14:57:15.388 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:15 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3046617140' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"} : dispatch 2024-09-13T14:57:15.418 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:57:15.418 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-13T14:57:15.418 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:15.418 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36738.0:134720 10.e e470/447 10:7920aed6:::benchmark_data_smithi102_377304_object28878:head v 470'8505, mlcod=469'8501)", 2024-09-13T14:57:15.419 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:57:15.413717+0000", 2024-09-13T14:57:15.419 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.0030111040000000001, 2024-09-13T14:57:15.419 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003034943, 2024-09-13T14:57:15.419 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:57:15.419 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T14:57:15.419 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:57:15.419 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:15.419 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:57:15.419 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:15.413717+0000", 2024-09-13T14:57:15.419 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:15.419 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:15.419 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:15.419 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:57:15.420 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:15.413717+0000", 2024-09-13T14:57:15.420 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:15.420 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:15.420 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:15.420 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:57:15.420 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:15.413716+0000", 2024-09-13T14:57:15.420 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:57:15.420 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:15.420 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:15.420 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:57:15.420 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:15.413723+0000", 2024-09-13T14:57:15.420 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6429999999999998e-06 2024-09-13T14:57:15.420 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:15.420 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:15.421 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:57:15.421 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:15.413734+0000", 2024-09-13T14:57:15.421 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1729000000000001e-05 2024-09-13T14:57:15.421 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:15.421 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:15.421 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:57:15.421 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:15.413741+0000", 2024-09-13T14:57:15.421 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3459999999999999e-06 2024-09-13T14:57:15.421 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:15.421 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:15.421 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:15.421 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:15.413793+0000", 2024-09-13T14:57:15.421 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.198e-05 2024-09-13T14:57:15.422 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:15.422 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:15.422 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:57:15.422 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:15.413816+0000", 2024-09-13T14:57:15.422 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2940000000000001e-05 2024-09-13T14:57:15.423 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:15.423 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:57:15.423 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:15.423 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:15.423 INFO:teuthology.orchestra.run.smithi195.stdout: ], 2024-09-13T14:57:15.423 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 1 2024-09-13T14:57:15.423 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:57:15.489 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:15.518 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:15.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:15.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:15.577 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:57:15.618 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:15 smithi102 ceph-mon[110183]: pgmap v1576: 109 pgs: 109 active+clean; 4.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 898 op/s 2024-09-13T14:57:15.618 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:15 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3046617140' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"} : dispatch 2024-09-13T14:57:15.619 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:15 smithi102 ceph-mon[115269]: pgmap v1576: 109 pgs: 109 active+clean; 4.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 898 op/s 2024-09-13T14:57:15.619 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:15 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3046617140' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"} : dispatch 2024-09-13T14:57:15.620 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:15.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:15.673 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:15.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:15.774 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:15.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:15.802 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:15.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:15.904 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:15.933 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:15.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:15.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:16.035 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:16.073 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:16.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:16.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:16.156 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T14:57:16.175 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:16.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:16.223 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:16.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:16.325 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:16.359 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:16.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:16.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:16.430 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:57:16.462 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:16.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:16.488 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:16.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:16.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:16 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3046617140' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-09-13T14:57:16.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:16 smithi195 ceph-mon[104262]: osdmap e470: 8 total, 8 up, 7 in 2024-09-13T14:57:16.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:16 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"} : dispatch 2024-09-13T14:57:16.590 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:16.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:16.634 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:16.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:16.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:16 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3046617140' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-09-13T14:57:16.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:16 smithi102 ceph-mon[110183]: osdmap e470: 8 total, 8 up, 7 in 2024-09-13T14:57:16.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:16 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"} : dispatch 2024-09-13T14:57:16.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:16 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3046617140' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-09-13T14:57:16.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:16 smithi102 ceph-mon[115269]: osdmap e470: 8 total, 8 up, 7 in 2024-09-13T14:57:16.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:16 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"} : dispatch 2024-09-13T14:57:16.669 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:57:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:57:16] "GET /metrics HTTP/1.1" 200 36567 "" "Prometheus/2.51.0" 2024-09-13T14:57:16.735 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:16.766 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:16.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:16.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:16.868 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:16.907 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:16.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:16.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:17.009 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:17.035 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:17.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:17.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:17.136 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:17.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:17.187 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:17.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:17.288 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:17.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:17.330 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:17.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:17.431 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:17.455 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:17.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:17.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:17.557 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:17.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:17.581 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:17.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:17.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:17 smithi102 ceph-mon[110183]: pgmap v1578: 109 pgs: 109 active+clean; 4.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 807 op/s 2024-09-13T14:57:17.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:17 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-09-13T14:57:17.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:17 smithi102 ceph-mon[110183]: osdmap e471: 8 total, 8 up, 7 in 2024-09-13T14:57:17.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:17 smithi102 ceph-mon[110183]: osdmap e472: 8 total, 8 up, 7 in 2024-09-13T14:57:17.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:17 smithi102 ceph-mon[115269]: pgmap v1578: 109 pgs: 109 active+clean; 4.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 807 op/s 2024-09-13T14:57:17.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:17 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-09-13T14:57:17.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:17 smithi102 ceph-mon[115269]: osdmap e471: 8 total, 8 up, 7 in 2024-09-13T14:57:17.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:17 smithi102 ceph-mon[115269]: osdmap e472: 8 total, 8 up, 7 in 2024-09-13T14:57:17.682 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:17.717 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:17.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:17.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:17.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:17 smithi195 ceph-mon[104262]: pgmap v1578: 109 pgs: 109 active+clean; 4.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 807 op/s 2024-09-13T14:57:17.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:17 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-09-13T14:57:17.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:17 smithi195 ceph-mon[104262]: osdmap e471: 8 total, 8 up, 7 in 2024-09-13T14:57:17.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:17 smithi195 ceph-mon[104262]: osdmap e472: 8 total, 8 up, 7 in 2024-09-13T14:57:17.820 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:17.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:17.844 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:17.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:17.945 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:17.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:17.986 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:17.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:18.087 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:18.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:18.119 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:18.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:18.220 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:18.254 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:18.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:18.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:18.355 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:18.382 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:18.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:18.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:18.484 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:18.521 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:18.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:18.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:18.623 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:18.650 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:18.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:18.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:18.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:18 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:57:18.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:18 smithi102 ceph-mon[110183]: 1.b scrub starts 2024-09-13T14:57:18.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:18 smithi102 ceph-mon[110183]: 1.7 scrub starts 2024-09-13T14:57:18.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:18 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:57:18.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:18 smithi102 ceph-mon[115269]: 1.b scrub starts 2024-09-13T14:57:18.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:18 smithi102 ceph-mon[115269]: 1.7 scrub starts 2024-09-13T14:57:18.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:18 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:57:18.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:18 smithi195 ceph-mon[104262]: 1.b scrub starts 2024-09-13T14:57:18.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:18 smithi195 ceph-mon[104262]: 1.7 scrub starts 2024-09-13T14:57:18.752 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:18.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:18.796 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:18.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:18.897 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:18.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:18.936 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:18.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:19.037 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:19.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:19.078 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:19.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:19.179 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:19.212 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:19.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:19.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:19.314 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:19.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:19.351 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:19.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:19.451 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:19.479 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:19.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:19.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:19.582 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:19.608 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:19.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:19.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:19.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:19 smithi102 ceph-mon[110183]: pgmap v1581: 119 pgs: 6 peering, 10 unknown, 103 active+clean; 4.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 882 op/s 2024-09-13T14:57:19.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:19 smithi102 ceph-mon[110183]: 1.5 scrub starts 2024-09-13T14:57:19.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:19 smithi102 ceph-mon[110183]: 1.1b scrub starts 2024-09-13T14:57:19.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:19 smithi102 ceph-mon[110183]: 1.17 scrub starts 2024-09-13T14:57:19.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:19 smithi102 ceph-mon[115269]: pgmap v1581: 119 pgs: 6 peering, 10 unknown, 103 active+clean; 4.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 882 op/s 2024-09-13T14:57:19.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:19 smithi102 ceph-mon[115269]: 1.5 scrub starts 2024-09-13T14:57:19.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:19 smithi102 ceph-mon[115269]: 1.1b scrub starts 2024-09-13T14:57:19.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:19 smithi102 ceph-mon[115269]: 1.17 scrub starts 2024-09-13T14:57:19.710 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:19.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:19 smithi195 ceph-mon[104262]: pgmap v1581: 119 pgs: 6 peering, 10 unknown, 103 active+clean; 4.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 882 op/s 2024-09-13T14:57:19.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:19 smithi195 ceph-mon[104262]: 1.5 scrub starts 2024-09-13T14:57:19.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:19 smithi195 ceph-mon[104262]: 1.1b scrub starts 2024-09-13T14:57:19.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:19 smithi195 ceph-mon[104262]: 1.17 scrub starts 2024-09-13T14:57:19.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:19.751 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:19.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:19.852 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:19.879 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:19.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:19.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:19.981 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:20.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:20.014 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:20.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:20.115 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:20.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:20.142 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:20.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:20.243 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:20.267 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:20.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:20.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:20.369 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:20.392 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:20.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:20.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:20.494 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:20.519 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:20.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:20.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:20.621 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:20.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:20.655 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:20.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:20.663 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:20 smithi195 ceph-mon[104262]: 1.15 scrub starts 2024-09-13T14:57:20.663 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:20 smithi195 ceph-mon[104262]: 1.6 scrub starts 2024-09-13T14:57:20.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:20 smithi102 ceph-mon[110183]: 1.15 scrub starts 2024-09-13T14:57:20.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:20 smithi102 ceph-mon[110183]: 1.6 scrub starts 2024-09-13T14:57:20.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:20 smithi102 ceph-mon[115269]: 1.15 scrub starts 2024-09-13T14:57:20.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:20 smithi102 ceph-mon[115269]: 1.6 scrub starts 2024-09-13T14:57:20.756 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:20.785 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:20.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:20.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:20.887 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:20.922 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:57:20.922 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-13T14:57:20.922 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-09-13T14:57:20.922 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-09-13T14:57:20.922 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:57:20.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:20.931 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:20.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:21.032 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:21.073 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:21.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:21.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:21.175 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:21.203 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:21.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:21.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:21.305 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:21.349 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:21.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:21.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:21.451 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:21.486 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:21.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:21.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:21.588 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:21.616 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:21.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:21.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:21.624 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:21 smithi195 ceph-mon[104262]: pgmap v1582: 119 pgs: 2 active+clean+scrubbing, 6 peering, 6 unknown, 105 active+clean; 4.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 925 op/s 2024-09-13T14:57:21.624 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:21 smithi195 ceph-mon[104262]: 1.16 scrub starts 2024-09-13T14:57:21.656 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T14:57:21.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:21 smithi102 ceph-mon[110183]: pgmap v1582: 119 pgs: 2 active+clean+scrubbing, 6 peering, 6 unknown, 105 active+clean; 4.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 925 op/s 2024-09-13T14:57:21.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:21 smithi102 ceph-mon[110183]: 1.16 scrub starts 2024-09-13T14:57:21.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:21 smithi102 ceph-mon[115269]: pgmap v1582: 119 pgs: 2 active+clean+scrubbing, 6 peering, 6 unknown, 105 active+clean; 4.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 925 op/s 2024-09-13T14:57:21.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:21 smithi102 ceph-mon[115269]: 1.16 scrub starts 2024-09-13T14:57:21.717 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:21.771 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:21.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:21.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:21.872 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:21.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:21.908 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:21.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:22.009 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:22.045 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:22.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:22.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:22.147 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:22.171 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:22.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:22.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:22.273 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:22.297 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:22.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:22.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:22.399 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:22.429 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:22.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:22.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:22.431 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:57:22.532 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:22.555 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:22.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:22.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:22.657 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:22.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:22 smithi102 ceph-mon[110183]: 1.1c scrub starts 2024-09-13T14:57:22.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:22 smithi102 ceph-mon[115269]: 1.1c scrub starts 2024-09-13T14:57:22.685 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:22.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:22.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:22.693 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:22 smithi195 ceph-mon[104262]: 1.1c scrub starts 2024-09-13T14:57:22.788 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:22.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:22.813 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:22.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:22.914 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:22.937 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:22.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:22.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:22.949 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:23.039 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:23.101 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:23.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:23.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:23.202 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:23.230 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:23.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:23.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:23.332 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:23.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:23.377 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:23.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:23.432 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is set 2024-09-13T14:57:23.479 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:23.531 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:23.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:23.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:23.633 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:23.667 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:23.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:23.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:23.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:23 smithi195 ceph-mon[104262]: pgmap v1583: 119 pgs: 4 active+clean+scrubbing, 3 peering, 3 unknown, 109 active+clean; 4.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 1.07k op/s 2024-09-13T14:57:23.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:23 smithi195 ceph-mon[104262]: 1.c scrub starts 2024-09-13T14:57:23.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:23 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2489977144' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:57:23.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:23 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:57:23.769 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:23.794 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:23.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:23.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:23.833 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:23 smithi102 ceph-mon[110183]: pgmap v1583: 119 pgs: 4 active+clean+scrubbing, 3 peering, 3 unknown, 109 active+clean; 4.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 1.07k op/s 2024-09-13T14:57:23.833 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:23 smithi102 ceph-mon[110183]: 1.c scrub starts 2024-09-13T14:57:23.833 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:23 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2489977144' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:57:23.833 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:23 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:57:23.834 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:23 smithi102 ceph-mon[115269]: pgmap v1583: 119 pgs: 4 active+clean+scrubbing, 3 peering, 3 unknown, 109 active+clean; 4.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 1.07k op/s 2024-09-13T14:57:23.834 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:23 smithi102 ceph-mon[115269]: 1.c scrub starts 2024-09-13T14:57:23.834 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:23 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2489977144' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:57:23.834 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:23 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:57:23.895 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:23.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:23.933 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:23.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:24.034 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:24.069 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:24.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:24.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:24.171 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:24.198 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:24.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:24.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:24.299 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:24.327 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:24.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:24.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:24.429 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:24.461 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:24.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:24.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:24.563 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:24.604 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:24.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:24.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:24.706 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:24.722 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:24 smithi195 ceph-mon[104262]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:57:24.722 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:24 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:57:24.722 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:24 smithi195 ceph-mon[104262]: osdmap e473: 8 total, 8 up, 7 in 2024-09-13T14:57:24.722 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:24 smithi195 ceph-mon[104262]: 1.b scrub starts 2024-09-13T14:57:24.722 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:24 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"} : dispatch 2024-09-13T14:57:24.722 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:24 smithi195 ceph-mon[104262]: 1.7 scrub starts 2024-09-13T14:57:24.735 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:24.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:24.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:24.778 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:24 smithi102 ceph-mon[110183]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:57:24.779 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:24 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:57:24.779 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:24 smithi102 ceph-mon[110183]: osdmap e473: 8 total, 8 up, 7 in 2024-09-13T14:57:24.779 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:24 smithi102 ceph-mon[110183]: 1.b scrub starts 2024-09-13T14:57:24.779 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:24 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"} : dispatch 2024-09-13T14:57:24.779 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:24 smithi102 ceph-mon[110183]: 1.7 scrub starts 2024-09-13T14:57:24.779 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:24 smithi102 ceph-mon[115269]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:57:24.779 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:24 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:57:24.779 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:24 smithi102 ceph-mon[115269]: osdmap e473: 8 total, 8 up, 7 in 2024-09-13T14:57:24.779 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:24 smithi102 ceph-mon[115269]: 1.b scrub starts 2024-09-13T14:57:24.779 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:24 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"} : dispatch 2024-09-13T14:57:24.780 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:24 smithi102 ceph-mon[115269]: 1.7 scrub starts 2024-09-13T14:57:24.837 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:24.880 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:24.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:24.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:24.982 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:25.013 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:25.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:25.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:25.115 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:25.149 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:25.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:25.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:25.252 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:25.288 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:25.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:25.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:25.365 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:57:25.365 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-09-13T14:57:25.365 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-09-13T14:57:25.365 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-13T14:57:25.365 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.365 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDPGPush(9.13 392/391 [PushOp(9:cce1574f:::benchmark_data_smithi102_348563_object40483:head, version: 378'2457, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(9:cce1574f:::benchmark_data_smithi102_348563_object40483:head@378'2457, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T14:57:25.366 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:52:06.936874+0000", 2024-09-13T14:57:25.366 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 318.42460986700002, 2024-09-13T14:57:25.366 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.059475924999999999, 2024-09-13T14:57:25.366 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:57:25.366 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T14:57:25.366 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:57:25.366 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.366 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:57:25.366 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:06.936874+0000", 2024-09-13T14:57:25.366 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:25.366 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.366 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.366 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:57:25.367 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:06.936874+0000", 2024-09-13T14:57:25.367 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:25.367 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.367 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.367 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:57:25.367 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:06.936873+0000", 2024-09-13T14:57:25.367 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:57:25.367 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.367 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.367 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:57:25.367 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:06.936936+0000", 2024-09-13T14:57:25.367 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3254000000000001e-05 2024-09-13T14:57:25.367 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.368 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.368 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:57:25.368 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:06.936959+0000", 2024-09-13T14:57:25.368 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3076000000000001e-05 2024-09-13T14:57:25.368 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.368 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.368 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:57:25.368 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:06.936974+0000", 2024-09-13T14:57:25.368 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4769e-05 2024-09-13T14:57:25.368 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.368 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.368 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:25.368 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:06.994303+0000", 2024-09-13T14:57:25.369 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.057328955000000001 2024-09-13T14:57:25.369 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.369 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.369 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:57:25.369 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:06.994315+0000", 2024-09-13T14:57:25.369 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.201e-05 2024-09-13T14:57:25.369 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.369 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.369 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:57:25.369 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:06.996350+0000", 2024-09-13T14:57:25.369 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0020354599999999998 2024-09-13T14:57:25.370 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:25.370 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:57:25.370 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:25.370 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.370 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.370 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36528.0:95656 9.11 e392/386 9:892a0af5:::benchmark_data_smithi102_348563_object95655:head v 392'4898, mlcod=392'4896)", 2024-09-13T14:57:25.370 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:52:06.940813+0000", 2024-09-13T14:57:25.370 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 318.42067108999998, 2024-09-13T14:57:25.370 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.05923722, 2024-09-13T14:57:25.370 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:57:25.370 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:25.370 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:57:25.370 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.371 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:57:25.371 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:06.940813+0000", 2024-09-13T14:57:25.371 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:25.371 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.371 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.371 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:57:25.371 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:06.940813+0000", 2024-09-13T14:57:25.371 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:25.371 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.371 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.372 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:57:25.372 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:06.940812+0000", 2024-09-13T14:57:25.372 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:57:25.372 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.372 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.372 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:57:25.372 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:06.940861+0000", 2024-09-13T14:57:25.372 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8776999999999997e-05 2024-09-13T14:57:25.372 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.372 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.372 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:57:25.372 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:06.994165+0000", 2024-09-13T14:57:25.373 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.053304451000000003 2024-09-13T14:57:25.373 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.373 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.373 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:57:25.373 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:06.994181+0000", 2024-09-13T14:57:25.373 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6205000000000002e-05 2024-09-13T14:57:25.373 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.373 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.373 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:25.373 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:06.994231+0000", 2024-09-13T14:57:25.373 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9947999999999997e-05 2024-09-13T14:57:25.373 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.373 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.374 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:57:25.374 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:06.994254+0000", 2024-09-13T14:57:25.374 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2249e-05 2024-09-13T14:57:25.374 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.374 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.374 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:57:25.374 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:07.000018+0000", 2024-09-13T14:57:25.374 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.005764384 2024-09-13T14:57:25.374 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.374 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.374 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:57:25.374 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:07.000050+0000", 2024-09-13T14:57:25.374 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2515999999999998e-05 2024-09-13T14:57:25.374 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:25.375 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:57:25.375 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:25.375 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.375 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.375 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36528.0:95655 9.1 e392/381 9:8571fba6:::benchmark_data_smithi102_348563_object95654:head v 392'4899, mlcod=392'4897)", 2024-09-13T14:57:25.375 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:52:06.940990+0000", 2024-09-13T14:57:25.375 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 318.42049467200002, 2024-09-13T14:57:25.375 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.063504271000000001, 2024-09-13T14:57:25.375 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:57:25.375 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:25.375 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:57:25.375 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.375 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:57:25.375 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:06.940990+0000", 2024-09-13T14:57:25.376 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:25.376 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.376 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.376 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:57:25.376 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:06.940990+0000", 2024-09-13T14:57:25.376 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:25.376 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.376 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.376 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:57:25.376 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:06.940989+0000", 2024-09-13T14:57:25.376 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:57:25.376 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.376 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.377 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:57:25.377 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:06.941184+0000", 2024-09-13T14:57:25.377 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00019578 2024-09-13T14:57:25.377 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.377 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.377 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:57:25.377 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:06.994827+0000", 2024-09-13T14:57:25.377 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.053642626999999998 2024-09-13T14:57:25.377 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.377 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.377 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:57:25.377 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:06.994837+0000", 2024-09-13T14:57:25.377 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4299999999999995e-06 2024-09-13T14:57:25.378 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.378 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.378 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:25.378 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:06.995800+0000", 2024-09-13T14:57:25.378 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00096365299999999997 2024-09-13T14:57:25.378 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.378 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.378 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:57:25.378 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:06.995817+0000", 2024-09-13T14:57:25.378 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7218999999999999e-05 2024-09-13T14:57:25.378 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.378 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.378 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:57:25.379 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:07.004453+0000", 2024-09-13T14:57:25.379 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0086354520000000001 2024-09-13T14:57:25.379 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.379 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.379 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:57:25.379 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:07.004494+0000", 2024-09-13T14:57:25.379 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1103000000000003e-05 2024-09-13T14:57:25.379 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:25.379 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:57:25.379 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:25.379 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.379 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.380 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36528.0:188021 9.1 e415/381 9:804798bb:::benchmark_data_smithi102_348563_object66919:head v 415'7743, mlcod=415'7741)", 2024-09-13T14:57:25.380 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:53:41.828489+0000", 2024-09-13T14:57:25.380 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 223.53299567900001, 2024-09-13T14:57:25.381 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.20997781600000001, 2024-09-13T14:57:25.381 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:57:25.381 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:25.381 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:57:25.381 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.381 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:57:25.381 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:41.828489+0000", 2024-09-13T14:57:25.381 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:25.381 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.381 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.381 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:57:25.381 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:41.828489+0000", 2024-09-13T14:57:25.381 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:25.382 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.382 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.382 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:57:25.382 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:41.828488+0000", 2024-09-13T14:57:25.382 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:57:25.382 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.382 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.382 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:57:25.382 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:41.828498+0000", 2024-09-13T14:57:25.382 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0414999999999999e-05 2024-09-13T14:57:25.382 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.382 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.382 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:57:25.383 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:41.828512+0000", 2024-09-13T14:57:25.383 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3583e-05 2024-09-13T14:57:25.383 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.383 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.383 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:57:25.383 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:41.828519+0000", 2024-09-13T14:57:25.383 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3679999999999996e-06 2024-09-13T14:57:25.383 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.383 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.383 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:25.383 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:41.828579+0000", 2024-09-13T14:57:25.383 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0560999999999998e-05 2024-09-13T14:57:25.383 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.383 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.384 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:57:25.384 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:42.028489+0000", 2024-09-13T14:57:25.384 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.19990945600000001 2024-09-13T14:57:25.384 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.384 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.384 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:57:25.384 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:42.038410+0000", 2024-09-13T14:57:25.384 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0099208209999999998 2024-09-13T14:57:25.384 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.384 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.384 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:57:25.384 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:42.038466+0000", 2024-09-13T14:57:25.384 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.6771000000000002e-05 2024-09-13T14:57:25.384 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:25.385 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:57:25.385 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:25.385 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.385 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.385 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36528.0:188024 9.13 e415/396 9:cc7fd1e7:::benchmark_data_smithi102_348563_object66922:head v 415'7487, mlcod=415'7484)", 2024-09-13T14:57:25.385 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:53:41.830705+0000", 2024-09-13T14:57:25.385 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 223.53077945300001, 2024-09-13T14:57:25.385 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.21054535599999999, 2024-09-13T14:57:25.385 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:57:25.385 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:25.385 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:57:25.385 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.385 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:57:25.386 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:41.830705+0000", 2024-09-13T14:57:25.386 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:25.386 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.386 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.386 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:57:25.386 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:41.830705+0000", 2024-09-13T14:57:25.386 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:25.386 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.386 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.386 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:57:25.386 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:41.830704+0000", 2024-09-13T14:57:25.386 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:57:25.386 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.387 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.387 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:57:25.387 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:41.830713+0000", 2024-09-13T14:57:25.387 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.02e-06 2024-09-13T14:57:25.387 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.387 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.387 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:57:25.387 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:42.028380+0000", 2024-09-13T14:57:25.387 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.19766724199999999 2024-09-13T14:57:25.387 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.387 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.387 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:57:25.387 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:42.028396+0000", 2024-09-13T14:57:25.388 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6498999999999999e-05 2024-09-13T14:57:25.388 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.388 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.388 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:25.388 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:42.029221+0000", 2024-09-13T14:57:25.388 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00082415500000000003 2024-09-13T14:57:25.388 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.388 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.388 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:57:25.388 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:42.029239+0000", 2024-09-13T14:57:25.388 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8774999999999999e-05 2024-09-13T14:57:25.388 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.388 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.389 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:57:25.389 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:42.041023+0000", 2024-09-13T14:57:25.389 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.01178389 2024-09-13T14:57:25.389 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.389 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.389 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:57:25.389 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:42.041250+0000", 2024-09-13T14:57:25.389 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00022703600000000001 2024-09-13T14:57:25.389 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:25.389 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:57:25.389 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:25.389 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.389 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.389 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36528.0:188032 9.b e415/374 9:d64c5cf6:::benchmark_data_smithi102_348563_object66930:head v 415'11623, mlcod=415'11621)", 2024-09-13T14:57:25.390 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:53:41.837840+0000", 2024-09-13T14:57:25.390 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 223.523644641, 2024-09-13T14:57:25.390 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.210339263, 2024-09-13T14:57:25.390 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:57:25.390 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:25.390 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:57:25.390 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.390 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:57:25.390 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:41.837840+0000", 2024-09-13T14:57:25.390 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:25.390 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.390 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.391 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:57:25.391 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:41.837840+0000", 2024-09-13T14:57:25.391 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:25.391 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.391 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.391 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:57:25.391 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:41.837839+0000", 2024-09-13T14:57:25.391 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:57:25.391 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.391 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.391 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:57:25.391 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:41.837849+0000", 2024-09-13T14:57:25.391 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0169999999999999e-05 2024-09-13T14:57:25.391 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.392 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.392 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:57:25.392 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:42.028406+0000", 2024-09-13T14:57:25.392 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.190557227 2024-09-13T14:57:25.392 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.392 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.392 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:57:25.392 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:42.028417+0000", 2024-09-13T14:57:25.392 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1284000000000001e-05 2024-09-13T14:57:25.392 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.392 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.392 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:25.392 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:42.031718+0000", 2024-09-13T14:57:25.392 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0033011519999999999 2024-09-13T14:57:25.393 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.393 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.393 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:57:25.393 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:42.031740+0000", 2024-09-13T14:57:25.393 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1200999999999999e-05 2024-09-13T14:57:25.393 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.393 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.393 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:57:25.393 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:42.048103+0000", 2024-09-13T14:57:25.393 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016363773000000002 2024-09-13T14:57:25.393 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.393 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.393 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:57:25.394 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:42.048179+0000", 2024-09-13T14:57:25.394 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5655999999999995e-05 2024-09-13T14:57:25.394 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:25.394 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:57:25.394 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:25.394 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.394 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.394 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36528.0:188050 9.0 e415/381 9:03605f91:::benchmark_data_smithi102_348563_object66948:head v 415'7737, mlcod=415'7735)", 2024-09-13T14:57:25.394 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:53:41.856396+0000", 2024-09-13T14:57:25.394 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 223.505088449, 2024-09-13T14:57:25.394 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.17779988899999999, 2024-09-13T14:57:25.394 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:57:25.394 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:25.395 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:57:25.395 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.395 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:57:25.395 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:41.856396+0000", 2024-09-13T14:57:25.395 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:25.395 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.395 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.395 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:57:25.396 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:41.856396+0000", 2024-09-13T14:57:25.396 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:25.396 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.396 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.396 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:57:25.396 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:41.856395+0000", 2024-09-13T14:57:25.396 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:57:25.396 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.396 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.396 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:57:25.396 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:41.856403+0000", 2024-09-13T14:57:25.396 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.4330000000000004e-06 2024-09-13T14:57:25.397 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.397 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.397 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:57:25.397 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:42.028441+0000", 2024-09-13T14:57:25.397 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.17203776800000001 2024-09-13T14:57:25.397 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.397 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.397 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:57:25.397 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:42.028451+0000", 2024-09-13T14:57:25.397 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6989999999999998e-06 2024-09-13T14:57:25.397 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.397 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.397 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:25.398 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:42.029191+0000", 2024-09-13T14:57:25.398 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00074086200000000001 2024-09-13T14:57:25.398 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.398 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.398 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:57:25.398 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:42.029217+0000", 2024-09-13T14:57:25.398 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5358999999999999e-05 2024-09-13T14:57:25.398 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.398 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.398 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:57:25.398 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:42.034162+0000", 2024-09-13T14:57:25.398 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0049453759999999996 2024-09-13T14:57:25.398 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.398 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.399 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:57:25.399 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:42.034196+0000", 2024-09-13T14:57:25.399 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.3577000000000001e-05 2024-09-13T14:57:25.399 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:25.399 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:57:25.399 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:25.399 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.399 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.399 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36738.0:43668 10.e e460/447 10:7207ca7d:::benchmark_data_smithi102_377304_object43667:head v 460'2826, mlcod=460'2824)", 2024-09-13T14:57:25.399 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:55:45.045034+0000", 2024-09-13T14:57:25.399 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 100.316450422, 2024-09-13T14:57:25.399 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.104706725, 2024-09-13T14:57:25.399 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:57:25.399 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:25.400 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:57:25.400 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.400 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:57:25.400 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045034+0000", 2024-09-13T14:57:25.400 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:25.400 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.400 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.400 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:57:25.400 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045034+0000", 2024-09-13T14:57:25.400 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:25.400 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.400 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.400 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:57:25.401 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045033+0000", 2024-09-13T14:57:25.401 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:57:25.401 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.401 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.401 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:57:25.401 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045093+0000", 2024-09-13T14:57:25.401 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.9845e-05 2024-09-13T14:57:25.401 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.401 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.401 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:57:25.401 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045101+0000", 2024-09-13T14:57:25.401 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0730000000000005e-06 2024-09-13T14:57:25.401 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.401 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.402 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:57:25.402 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045106+0000", 2024-09-13T14:57:25.402 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3089999999999997e-06 2024-09-13T14:57:25.402 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.402 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.402 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:25.402 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045136+0000", 2024-09-13T14:57:25.402 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0877999999999998e-05 2024-09-13T14:57:25.402 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.402 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.402 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:57:25.402 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045149+0000", 2024-09-13T14:57:25.403 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2941e-05 2024-09-13T14:57:25.403 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.403 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.403 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:57:25.403 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.149711+0000", 2024-09-13T14:57:25.403 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.104562057 2024-09-13T14:57:25.403 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.403 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.403 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:57:25.403 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.149741+0000", 2024-09-13T14:57:25.403 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9220000000000001e-05 2024-09-13T14:57:25.403 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:25.403 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:57:25.403 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:25.404 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.404 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.404 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36738.0:43659 10.b e460/436 10:d26becbe:::benchmark_data_smithi102_377304_object43658:head v 460'2698, mlcod=460'2695)", 2024-09-13T14:57:25.404 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:55:45.045201+0000", 2024-09-13T14:57:25.404 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 100.31628295199999, 2024-09-13T14:57:25.404 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.10350997200000001, 2024-09-13T14:57:25.404 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:57:25.404 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:25.404 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:57:25.404 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.404 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:57:25.404 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045201+0000", 2024-09-13T14:57:25.405 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:25.405 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.405 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.405 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:57:25.405 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045201+0000", 2024-09-13T14:57:25.405 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:25.405 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.405 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.405 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:57:25.405 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045201+0000", 2024-09-13T14:57:25.405 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:57:25.405 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.405 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.405 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:57:25.406 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045242+0000", 2024-09-13T14:57:25.406 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1594999999999997e-05 2024-09-13T14:57:25.406 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.406 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.406 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:57:25.406 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045250+0000", 2024-09-13T14:57:25.406 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5310000000000004e-06 2024-09-13T14:57:25.406 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.406 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.406 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:57:25.406 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045254+0000", 2024-09-13T14:57:25.406 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.194e-06 2024-09-13T14:57:25.406 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.406 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.407 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:25.407 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045279+0000", 2024-09-13T14:57:25.407 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4769999999999998e-05 2024-09-13T14:57:25.407 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.407 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.407 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:57:25.407 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045291+0000", 2024-09-13T14:57:25.407 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2408999999999999e-05 2024-09-13T14:57:25.407 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.407 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.407 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:57:25.407 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.148688+0000", 2024-09-13T14:57:25.407 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.103396548 2024-09-13T14:57:25.407 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.408 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.408 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:57:25.408 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.148711+0000", 2024-09-13T14:57:25.408 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.374e-05 2024-09-13T14:57:25.408 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:25.408 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:57:25.408 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:25.408 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.408 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.408 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36738.0:43663 10.0 e460/448 10:0eb12d1f:::benchmark_data_smithi102_377304_object43662:head v 460'2781, mlcod=460'2779)", 2024-09-13T14:57:25.408 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:55:45.045654+0000", 2024-09-13T14:57:25.408 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 100.315830633, 2024-09-13T14:57:25.408 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.10277876499999999, 2024-09-13T14:57:25.409 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:57:25.409 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:25.409 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:57:25.409 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.409 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:57:25.409 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045654+0000", 2024-09-13T14:57:25.409 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:25.409 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.409 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.409 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:57:25.409 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045654+0000", 2024-09-13T14:57:25.409 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:25.409 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.410 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.410 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:57:25.410 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045653+0000", 2024-09-13T14:57:25.410 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:57:25.410 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.410 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.410 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:57:25.410 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045713+0000", 2024-09-13T14:57:25.410 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.9783e-05 2024-09-13T14:57:25.410 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.410 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.410 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:57:25.410 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045720+0000", 2024-09-13T14:57:25.411 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1459999999999999e-06 2024-09-13T14:57:25.411 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.411 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.411 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:57:25.411 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045723+0000", 2024-09-13T14:57:25.411 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.1930000000000002e-06 2024-09-13T14:57:25.411 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.411 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.411 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:25.411 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045752+0000", 2024-09-13T14:57:25.411 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8223999999999999e-05 2024-09-13T14:57:25.411 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.411 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.411 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:57:25.412 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045763+0000", 2024-09-13T14:57:25.412 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.181e-05 2024-09-13T14:57:25.412 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.412 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.412 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:57:25.412 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.148387+0000", 2024-09-13T14:57:25.412 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.102623805 2024-09-13T14:57:25.412 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.412 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.412 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:57:25.412 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.148432+0000", 2024-09-13T14:57:25.413 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5262000000000001e-05 2024-09-13T14:57:25.413 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:25.413 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:57:25.413 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:25.413 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.413 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.413 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36738.0:43667 10.a e460/436 10:51b79b08:::benchmark_data_smithi102_377304_object43666:head v 460'2804, mlcod=460'2802)", 2024-09-13T14:57:25.413 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:55:45.045813+0000", 2024-09-13T14:57:25.413 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 100.315671071, 2024-09-13T14:57:25.413 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.10402655199999999, 2024-09-13T14:57:25.413 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:57:25.413 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:25.413 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:57:25.413 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.414 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:57:25.414 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045813+0000", 2024-09-13T14:57:25.414 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:25.414 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.414 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.414 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:57:25.414 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045813+0000", 2024-09-13T14:57:25.414 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:25.414 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.414 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.414 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:57:25.414 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045813+0000", 2024-09-13T14:57:25.415 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:57:25.415 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.415 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.415 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:57:25.415 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045875+0000", 2024-09-13T14:57:25.415 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2755000000000005e-05 2024-09-13T14:57:25.415 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.415 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.415 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:57:25.415 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045886+0000", 2024-09-13T14:57:25.415 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0549e-05 2024-09-13T14:57:25.415 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.415 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.416 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:57:25.416 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045892+0000", 2024-09-13T14:57:25.416 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2589999999999997e-06 2024-09-13T14:57:25.416 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.416 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.416 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:25.416 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045931+0000", 2024-09-13T14:57:25.416 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8822999999999997e-05 2024-09-13T14:57:25.416 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.416 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.416 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:57:25.416 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045947+0000", 2024-09-13T14:57:25.416 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6147e-05 2024-09-13T14:57:25.417 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.417 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.417 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:57:25.417 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.149817+0000", 2024-09-13T14:57:25.417 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.103869609 2024-09-13T14:57:25.417 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.417 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.417 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:57:25.417 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.149840+0000", 2024-09-13T14:57:25.417 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3149000000000001e-05 2024-09-13T14:57:25.417 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:25.417 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:57:25.417 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:25.418 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.418 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.418 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36738.0:43669 10.f e460/436 10:f48161ab:::benchmark_data_smithi102_377304_object43668:head v 460'2691, mlcod=460'2688)", 2024-09-13T14:57:25.418 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:55:45.046758+0000", 2024-09-13T14:57:25.418 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 100.314726507, 2024-09-13T14:57:25.418 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.103328541, 2024-09-13T14:57:25.418 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:57:25.418 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:25.419 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:57:25.419 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.419 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:57:25.419 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.046758+0000", 2024-09-13T14:57:25.419 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:25.419 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.419 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.419 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:57:25.419 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.046758+0000", 2024-09-13T14:57:25.419 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:25.419 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.419 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.420 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:57:25.420 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.046757+0000", 2024-09-13T14:57:25.420 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:57:25.420 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.420 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.420 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:57:25.420 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.046823+0000", 2024-09-13T14:57:25.420 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5493999999999993e-05 2024-09-13T14:57:25.420 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.420 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.420 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:57:25.420 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.148448+0000", 2024-09-13T14:57:25.421 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.101625698 2024-09-13T14:57:25.421 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.421 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.421 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:57:25.421 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.148455+0000", 2024-09-13T14:57:25.421 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0850000000000002e-06 2024-09-13T14:57:25.421 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.421 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.421 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:25.421 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.148496+0000", 2024-09-13T14:57:25.421 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1149999999999997e-05 2024-09-13T14:57:25.421 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.421 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.421 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:57:25.422 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.148514+0000", 2024-09-13T14:57:25.422 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8425000000000001e-05 2024-09-13T14:57:25.422 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.422 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.422 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:57:25.422 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.150063+0000", 2024-09-13T14:57:25.422 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001549337 2024-09-13T14:57:25.422 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.422 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.422 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:57:25.422 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.150086+0000", 2024-09-13T14:57:25.422 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2938e-05 2024-09-13T14:57:25.422 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:25.423 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:57:25.423 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:25.423 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.423 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.423 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36738.0:43666 10.b e460/436 10:d20c5cea:::benchmark_data_smithi102_377304_object43665:head v 460'2699, mlcod=460'2696)", 2024-09-13T14:57:25.423 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:55:45.047252+0000", 2024-09-13T14:57:25.423 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 100.314232656, 2024-09-13T14:57:25.423 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.104257054, 2024-09-13T14:57:25.423 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:57:25.423 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:25.423 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:57:25.423 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.423 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:57:25.423 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.047252+0000", 2024-09-13T14:57:25.424 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:25.424 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.424 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.424 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:57:25.424 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.047252+0000", 2024-09-13T14:57:25.424 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:25.424 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.424 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.424 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:57:25.424 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.047251+0000", 2024-09-13T14:57:25.424 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:57:25.424 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.425 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.425 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:57:25.425 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.047306+0000", 2024-09-13T14:57:25.425 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5194e-05 2024-09-13T14:57:25.425 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.425 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.425 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:57:25.425 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.149545+0000", 2024-09-13T14:57:25.425 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.10223884599999999 2024-09-13T14:57:25.425 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.425 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:25.426 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.427 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:57:25.427 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.149553+0000", 2024-09-13T14:57:25.427 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9060000000000002e-06 2024-09-13T14:57:25.427 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.427 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.427 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:25.427 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.149592+0000", 2024-09-13T14:57:25.427 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9258999999999998e-05 2024-09-13T14:57:25.427 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.427 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.427 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:57:25.427 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.149612+0000", 2024-09-13T14:57:25.428 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.925e-05 2024-09-13T14:57:25.428 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.428 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.428 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:57:25.428 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.151482+0000", 2024-09-13T14:57:25.428 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0018704520000000001 2024-09-13T14:57:25.428 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.428 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.428 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:57:25.428 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.151509+0000", 2024-09-13T14:57:25.428 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.6667000000000001e-05 2024-09-13T14:57:25.428 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:25.429 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:57:25.429 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:25.429 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.429 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.429 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36738.0:43670 10.e e460/447 10:7cbfdc04:::benchmark_data_smithi102_377304_object43669:head v 460'2827, mlcod=460'2824)", 2024-09-13T14:57:25.429 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:55:45.047844+0000", 2024-09-13T14:57:25.429 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 100.31364016000001, 2024-09-13T14:57:25.429 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.102086207, 2024-09-13T14:57:25.429 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:57:25.429 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:25.429 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:57:25.429 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.429 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:57:25.430 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.047844+0000", 2024-09-13T14:57:25.430 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:25.430 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.430 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.430 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:57:25.430 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.047844+0000", 2024-09-13T14:57:25.430 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:25.430 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.430 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.430 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:57:25.430 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.047844+0000", 2024-09-13T14:57:25.430 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:57:25.430 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.431 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.431 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:57:25.431 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.047901+0000", 2024-09-13T14:57:25.431 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7136000000000003e-05 2024-09-13T14:57:25.431 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.431 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.431 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:57:25.431 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.148379+0000", 2024-09-13T14:57:25.431 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.100478291 2024-09-13T14:57:25.431 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.431 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.431 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:57:25.431 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.148389+0000", 2024-09-13T14:57:25.432 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.9699999999999994e-06 2024-09-13T14:57:25.432 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.432 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.432 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:25.432 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.148529+0000", 2024-09-13T14:57:25.432 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000139896 2024-09-13T14:57:25.432 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.432 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.432 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:57:25.432 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.148546+0000", 2024-09-13T14:57:25.432 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7204e-05 2024-09-13T14:57:25.432 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.432 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.432 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:57:25.433 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.149896+0000", 2024-09-13T14:57:25.433 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0013502379999999999 2024-09-13T14:57:25.433 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.433 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.433 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:57:25.433 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.149930+0000", 2024-09-13T14:57:25.433 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4066999999999998e-05 2024-09-13T14:57:25.433 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:25.433 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:57:25.433 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:25.433 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.433 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.433 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36738.0:63691 10.d e460/450 10:bd07241a:::benchmark_data_smithi102_377304_object63690:head v 460'3897, mlcod=460'3895)", 2024-09-13T14:57:25.434 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:56:00.383770+0000", 2024-09-13T14:57:25.434 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 84.977714641000006, 2024-09-13T14:57:25.434 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.033853341000000002, 2024-09-13T14:57:25.434 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:57:25.434 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:25.434 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:57:25.434 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.434 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:57:25.434 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:00.383770+0000", 2024-09-13T14:57:25.434 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:25.434 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.434 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.434 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:57:25.435 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:00.383770+0000", 2024-09-13T14:57:25.435 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:25.435 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.435 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.435 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:57:25.435 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:00.383769+0000", 2024-09-13T14:57:25.435 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:57:25.435 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.435 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.435 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:57:25.435 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:00.383826+0000", 2024-09-13T14:57:25.435 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.6845000000000002e-05 2024-09-13T14:57:25.435 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.435 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.436 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:57:25.436 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:00.383835+0000", 2024-09-13T14:57:25.436 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.2629999999999992e-06 2024-09-13T14:57:25.436 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.436 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.436 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:57:25.436 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:00.383840+0000", 2024-09-13T14:57:25.436 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7060000000000003e-06 2024-09-13T14:57:25.436 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.436 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.436 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:25.436 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:00.383873+0000", 2024-09-13T14:57:25.436 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.3408999999999998e-05 2024-09-13T14:57:25.437 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.437 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.437 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:57:25.437 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:00.383886+0000", 2024-09-13T14:57:25.437 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2979000000000001e-05 2024-09-13T14:57:25.437 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.437 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.437 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:57:25.437 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:00.417592+0000", 2024-09-13T14:57:25.437 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.033705450999999997 2024-09-13T14:57:25.437 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.437 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.437 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:57:25.437 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:56:00.417623+0000", 2024-09-13T14:57:25.438 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.1273999999999999e-05 2024-09-13T14:57:25.438 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:25.438 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:57:25.438 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:25.438 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.438 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.438 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36738.0:135600 10.e e471/447 10:74acff9b:::benchmark_data_smithi102_377304_object29758:head v 471'8568, mlcod=470'8566)", 2024-09-13T14:57:25.438 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:57:16.408426+0000", 2024-09-13T14:57:25.438 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 8.9530578779999992, 2024-09-13T14:57:25.438 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.022052784999999998, 2024-09-13T14:57:25.438 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:57:25.438 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:25.438 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:57:25.438 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.439 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:57:25.439 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:16.408426+0000", 2024-09-13T14:57:25.439 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:25.439 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.439 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.439 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:57:25.439 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:16.408426+0000", 2024-09-13T14:57:25.439 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:25.439 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.439 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.439 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:57:25.439 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:16.408426+0000", 2024-09-13T14:57:25.439 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:57:25.440 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.440 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.440 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:57:25.440 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:16.408436+0000", 2024-09-13T14:57:25.440 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0373999999999999e-05 2024-09-13T14:57:25.440 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.440 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.440 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:57:25.440 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:16.408448+0000", 2024-09-13T14:57:25.440 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1857e-05 2024-09-13T14:57:25.440 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.440 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.440 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:57:25.441 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:16.408454+0000", 2024-09-13T14:57:25.441 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4670000000000001e-06 2024-09-13T14:57:25.441 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.441 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.441 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:25.441 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:16.408496+0000", 2024-09-13T14:57:25.441 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1365000000000001e-05 2024-09-13T14:57:25.441 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.441 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.441 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:57:25.441 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:16.408517+0000", 2024-09-13T14:57:25.441 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1545000000000001e-05 2024-09-13T14:57:25.442 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.442 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.442 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:57:25.442 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:16.430446+0000", 2024-09-13T14:57:25.442 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.021928828000000001 2024-09-13T14:57:25.442 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.442 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.442 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:57:25.442 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:16.430479+0000", 2024-09-13T14:57:25.442 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.3120999999999997e-05 2024-09-13T14:57:25.442 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:25.442 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:57:25.442 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:25.442 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.443 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.443 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36738.0:135603 10.e e471/447 10:75484547:::benchmark_data_smithi102_377304_object29761:head v 471'8569, mlcod=470'8566)", 2024-09-13T14:57:25.443 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:57:16.408470+0000", 2024-09-13T14:57:25.443 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 8.9530145379999997, 2024-09-13T14:57:25.443 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.022033978999999999, 2024-09-13T14:57:25.443 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:57:25.443 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:25.443 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:57:25.443 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.443 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:57:25.443 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:16.408470+0000", 2024-09-13T14:57:25.443 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:25.443 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.444 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.444 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:57:25.444 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:16.408470+0000", 2024-09-13T14:57:25.444 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:25.444 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.444 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.444 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:57:25.444 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:16.408470+0000", 2024-09-13T14:57:25.444 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:57:25.444 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.444 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.444 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:57:25.444 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:16.408473+0000", 2024-09-13T14:57:25.445 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4640000000000002e-06 2024-09-13T14:57:25.445 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.445 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.445 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:57:25.445 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:16.408479+0000", 2024-09-13T14:57:25.445 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4729999999999997e-06 2024-09-13T14:57:25.445 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.445 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.445 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:57:25.445 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:16.408484+0000", 2024-09-13T14:57:25.445 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8049999999999997e-06 2024-09-13T14:57:25.445 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.445 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.446 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:25.446 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:16.414858+0000", 2024-09-13T14:57:25.446 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0063738880000000003 2024-09-13T14:57:25.446 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.446 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.446 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:57:25.446 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:16.414879+0000", 2024-09-13T14:57:25.446 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0750999999999998e-05 2024-09-13T14:57:25.446 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.446 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.446 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:57:25.446 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:16.430482+0000", 2024-09-13T14:57:25.446 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.015603044 2024-09-13T14:57:25.446 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.447 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.447 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:57:25.447 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:16.430504+0000", 2024-09-13T14:57:25.447 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1806000000000001e-05 2024-09-13T14:57:25.447 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:25.447 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:57:25.447 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:25.447 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.447 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.447 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36738.0:138799 10.a e472/462 10:5af6bd1d:::benchmark_data_smithi102_377304_object32957:head v 472'8912, mlcod=472'8910)", 2024-09-13T14:57:25.447 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:57:19.575596+0000", 2024-09-13T14:57:25.447 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 5.7858886439999999, 2024-09-13T14:57:25.447 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.022517081000000001, 2024-09-13T14:57:25.448 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:57:25.448 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:25.448 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:57:25.448 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.448 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:57:25.448 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:19.575596+0000", 2024-09-13T14:57:25.448 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:25.448 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.448 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.448 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:57:25.448 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:19.575596+0000", 2024-09-13T14:57:25.448 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:25.448 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.448 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.449 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:57:25.449 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:19.575595+0000", 2024-09-13T14:57:25.449 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:57:25.449 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.449 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.449 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:57:25.449 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:19.575600+0000", 2024-09-13T14:57:25.449 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0529999999999997e-06 2024-09-13T14:57:25.449 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.449 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.449 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:57:25.449 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:19.575609+0000", 2024-09-13T14:57:25.449 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5399999999999996e-06 2024-09-13T14:57:25.450 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.450 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.450 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:57:25.450 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:19.575614+0000", 2024-09-13T14:57:25.450 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0579999999999997e-06 2024-09-13T14:57:25.450 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.450 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.450 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:25.450 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:19.575677+0000", 2024-09-13T14:57:25.450 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3312000000000006e-05 2024-09-13T14:57:25.450 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.450 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.450 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:57:25.450 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:19.575696+0000", 2024-09-13T14:57:25.451 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9230000000000001e-05 2024-09-13T14:57:25.451 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.451 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.451 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:57:25.451 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:19.598076+0000", 2024-09-13T14:57:25.451 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.022379966000000001 2024-09-13T14:57:25.451 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.451 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.451 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:57:25.451 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:19.598113+0000", 2024-09-13T14:57:25.451 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6514999999999999e-05 2024-09-13T14:57:25.451 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:25.452 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:57:25.452 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:25.452 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.452 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.452 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36738.0:138798 10.e e472/447 10:7df769ba:::benchmark_data_smithi102_377304_object32956:head v 472'8816, mlcod=472'8811)", 2024-09-13T14:57:25.452 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:57:19.578482+0000", 2024-09-13T14:57:25.452 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 5.7830021269999996, 2024-09-13T14:57:25.452 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.027489350999999999, 2024-09-13T14:57:25.452 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:57:25.452 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:25.452 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:57:25.452 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.452 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:57:25.453 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:19.578482+0000", 2024-09-13T14:57:25.453 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:25.453 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.453 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.453 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:57:25.453 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:19.578482+0000", 2024-09-13T14:57:25.453 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:25.453 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.453 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.453 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:57:25.453 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:19.578481+0000", 2024-09-13T14:57:25.453 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:57:25.453 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.454 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.454 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:57:25.454 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:19.578489+0000", 2024-09-13T14:57:25.454 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3150000000000003e-06 2024-09-13T14:57:25.454 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.454 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.454 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:57:25.455 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:19.578499+0000", 2024-09-13T14:57:25.455 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0108999999999999e-05 2024-09-13T14:57:25.456 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.456 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.456 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:57:25.456 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:19.578505+0000", 2024-09-13T14:57:25.456 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3609999999999997e-06 2024-09-13T14:57:25.456 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.456 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.456 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:25.456 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:19.578551+0000", 2024-09-13T14:57:25.456 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6273000000000002e-05 2024-09-13T14:57:25.456 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.456 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.456 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:57:25.457 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:19.578569+0000", 2024-09-13T14:57:25.457 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7764000000000001e-05 2024-09-13T14:57:25.457 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.457 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.457 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:57:25.457 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:19.605886+0000", 2024-09-13T14:57:25.457 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.027316557000000002 2024-09-13T14:57:25.457 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.457 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.457 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:57:25.457 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:19.605972+0000", 2024-09-13T14:57:25.457 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5802000000000003e-05 2024-09-13T14:57:25.457 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:25.457 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:57:25.458 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:25.458 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.458 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.458 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36738.0:138808 10.e e472/447 10:74bae73b:::benchmark_data_smithi102_377304_object32966:head v 472'8817, mlcod=472'8811)", 2024-09-13T14:57:25.458 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:57:19.584644+0000", 2024-09-13T14:57:25.458 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 5.776840387, 2024-09-13T14:57:25.458 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.021407579999999999, 2024-09-13T14:57:25.458 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:57:25.458 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:25.458 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:57:25.458 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.458 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:57:25.458 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:19.584644+0000", 2024-09-13T14:57:25.458 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:25.459 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.459 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.459 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:57:25.459 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:19.584644+0000", 2024-09-13T14:57:25.459 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:57:25.459 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.459 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.459 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:57:25.459 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:19.584643+0000", 2024-09-13T14:57:25.459 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:57:25.459 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.459 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.459 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:57:25.460 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:19.584652+0000", 2024-09-13T14:57:25.460 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.0389999999999993e-06 2024-09-13T14:57:25.460 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.460 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.460 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:57:25.460 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:19.594615+0000", 2024-09-13T14:57:25.460 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0099633489999999998 2024-09-13T14:57:25.460 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.460 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.460 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:57:25.460 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:19.594621+0000", 2024-09-13T14:57:25.460 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7729999999999996e-06 2024-09-13T14:57:25.460 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.460 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.460 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:57:25.461 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:19.599094+0000", 2024-09-13T14:57:25.461 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0044735119999999998 2024-09-13T14:57:25.461 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.461 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.461 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:57:25.461 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:19.599115+0000", 2024-09-13T14:57:25.461 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.003e-05 2024-09-13T14:57:25.461 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.461 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.461 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:57:25.461 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:19.605974+0000", 2024-09-13T14:57:25.461 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.006859853 2024-09-13T14:57:25.461 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:57:25.462 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:57:25.462 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:57:25.462 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:19.606052+0000", 2024-09-13T14:57:25.462 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7183999999999998e-05 2024-09-13T14:57:25.462 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:25.462 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:57:25.462 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:25.462 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:57:25.462 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:57:25.462 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:57:25.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:25 smithi195 ceph-mon[104262]: pgmap v1585: 119 pgs: 3 active+clean+scrubbing, 116 active+clean; 4.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 1.18k op/s 2024-09-13T14:57:25.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:25 smithi195 ceph-mon[104262]: 1.5 scrub starts 2024-09-13T14:57:25.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:25 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-09-13T14:57:25.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:25 smithi195 ceph-mon[104262]: osdmap e474: 8 total, 8 up, 7 in 2024-09-13T14:57:25.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:25 smithi195 ceph-mon[104262]: 1.b scrub starts 2024-09-13T14:57:25.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:25 smithi195 ceph-mon[104262]: 1.c scrub starts 2024-09-13T14:57:25.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:25.513 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:25.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:25.614 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:25.634 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:25 smithi102 ceph-mon[110183]: pgmap v1585: 119 pgs: 3 active+clean+scrubbing, 116 active+clean; 4.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 1.18k op/s 2024-09-13T14:57:25.634 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:25 smithi102 ceph-mon[110183]: 1.5 scrub starts 2024-09-13T14:57:25.634 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:25 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-09-13T14:57:25.635 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:25 smithi102 ceph-mon[110183]: osdmap e474: 8 total, 8 up, 7 in 2024-09-13T14:57:25.635 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:25 smithi102 ceph-mon[110183]: 1.b scrub starts 2024-09-13T14:57:25.635 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:25 smithi102 ceph-mon[110183]: 1.c scrub starts 2024-09-13T14:57:25.635 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:25 smithi102 ceph-mon[115269]: pgmap v1585: 119 pgs: 3 active+clean+scrubbing, 116 active+clean; 4.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 1.18k op/s 2024-09-13T14:57:25.635 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:25 smithi102 ceph-mon[115269]: 1.5 scrub starts 2024-09-13T14:57:25.635 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:25 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-09-13T14:57:25.635 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:25 smithi102 ceph-mon[115269]: osdmap e474: 8 total, 8 up, 7 in 2024-09-13T14:57:25.635 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:25 smithi102 ceph-mon[115269]: 1.b scrub starts 2024-09-13T14:57:25.635 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:25 smithi102 ceph-mon[115269]: 1.c scrub starts 2024-09-13T14:57:25.641 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:25.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:25.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:25.742 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:25.790 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:25.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:25.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:25.891 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:25.917 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:25.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:25.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:26.019 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:26.056 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:26.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:26.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:26.158 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:26.180 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:26.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:26.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:26.282 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:26.311 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:26.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:26.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:26.366 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:57:26.366 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:57:26.383 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:57:26.383 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:57:26.406 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:57:26.406 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:57:26.413 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:26.439 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:26.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:26.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:26.441 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:57:26.441 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:57:26.481 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:57:26.481 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:57:26.516 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:57:26.517 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:57:26.517 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:26 smithi102 ceph-mon[110183]: 1.5 scrub starts 2024-09-13T14:57:26.517 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:26 smithi102 ceph-mon[110183]: osdmap e475: 8 total, 8 up, 7 in 2024-09-13T14:57:26.517 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:57:26.517 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:26 smithi102 ceph-mon[110183]: 1.6 scrub starts 2024-09-13T14:57:26.517 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:26 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1634428320' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:57:26.518 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:57:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:57:26] "GET /metrics HTTP/1.1" 200 36570 "" "Prometheus/2.51.0" 2024-09-13T14:57:26.518 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:26 smithi102 ceph-mon[115269]: 1.5 scrub starts 2024-09-13T14:57:26.518 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:26 smithi102 ceph-mon[115269]: osdmap e475: 8 total, 8 up, 7 in 2024-09-13T14:57:26.518 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:57:26.518 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:26 smithi102 ceph-mon[115269]: 1.6 scrub starts 2024-09-13T14:57:26.518 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:26 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1634428320' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:57:26.541 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:26.559 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:57:26.560 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:57:26.564 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:26.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:26.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:26.590 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:57:26.591 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:57:26.666 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:26.710 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:26.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:26.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:26.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:26 smithi195 ceph-mon[104262]: 1.5 scrub starts 2024-09-13T14:57:26.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:26 smithi195 ceph-mon[104262]: osdmap e475: 8 total, 8 up, 7 in 2024-09-13T14:57:26.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:57:26.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:26 smithi195 ceph-mon[104262]: 1.6 scrub starts 2024-09-13T14:57:26.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:26 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1634428320' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:57:26.811 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:26.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:26.844 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:26.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:26.945 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:26.974 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:26.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:26.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:27.076 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:27.110 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:27.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:27.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:27.125 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:57:27.211 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:27.237 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:27.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:27.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:27.305 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T14:57:27.339 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:27.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:27.411 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:27.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:27.512 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:27.537 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:27.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:27.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:27.639 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:27.666 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:27 smithi195 ceph-mon[104262]: pgmap v1588: 119 pgs: 2 unknown, 2 active+clean+scrubbing, 115 active+clean; 4.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 847 op/s 2024-09-13T14:57:27.666 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:27 smithi195 ceph-mon[104262]: osdmap e476: 8 total, 8 up, 7 in 2024-09-13T14:57:27.666 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:27 smithi195 ceph-mon[104262]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-09-13T14:57:27.666 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:27 smithi195 ceph-mon[104262]: 1.7 scrub starts 2024-09-13T14:57:27.679 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:27.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:27.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:27.688 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 6, 7, 5, 2, 1] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T14:57:27.688 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:57:27.689 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-13T14:57:27.689 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:57:27.761 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:27 smithi102 ceph-mon[110183]: pgmap v1588: 119 pgs: 2 unknown, 2 active+clean+scrubbing, 115 active+clean; 4.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 847 op/s 2024-09-13T14:57:27.762 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:27 smithi102 ceph-mon[110183]: osdmap e476: 8 total, 8 up, 7 in 2024-09-13T14:57:27.762 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:27 smithi102 ceph-mon[110183]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-09-13T14:57:27.762 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:27 smithi102 ceph-mon[110183]: 1.7 scrub starts 2024-09-13T14:57:27.762 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:27 smithi102 ceph-mon[115269]: pgmap v1588: 119 pgs: 2 unknown, 2 active+clean+scrubbing, 115 active+clean; 4.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 847 op/s 2024-09-13T14:57:27.762 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:27 smithi102 ceph-mon[115269]: osdmap e476: 8 total, 8 up, 7 in 2024-09-13T14:57:27.762 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:27 smithi102 ceph-mon[115269]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-09-13T14:57:27.762 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:27 smithi102 ceph-mon[115269]: 1.7 scrub starts 2024-09-13T14:57:27.781 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:27.831 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:27.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:27.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:27.933 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:28.027 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:28.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:28.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:28.083 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:28.111 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:28.130 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:28.131 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:28.154 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:28.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:28.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:28.256 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:28.299 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:28.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:28.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:28.401 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:28.426 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:28.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:28.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:28.528 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:28.560 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:28.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:28.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:28.662 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:28.687 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:28.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:28.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:28.789 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:28.791 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:28 smithi102 ceph-mon[110183]: osdmap e477: 8 total, 8 up, 7 in 2024-09-13T14:57:28.791 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:28 smithi102 ceph-mon[110183]: osdmap e478: 8 total, 8 up, 7 in 2024-09-13T14:57:28.791 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:28 smithi102 ceph-mon[115269]: osdmap e477: 8 total, 8 up, 7 in 2024-09-13T14:57:28.791 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:28 smithi102 ceph-mon[115269]: osdmap e478: 8 total, 8 up, 7 in 2024-09-13T14:57:28.813 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:28.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:28.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:28.915 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:28.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:28.954 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:28.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:28.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:28 smithi195 ceph-mon[104262]: osdmap e477: 8 total, 8 up, 7 in 2024-09-13T14:57:28.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:28 smithi195 ceph-mon[104262]: osdmap e478: 8 total, 8 up, 7 in 2024-09-13T14:57:29.055 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:29.091 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:29.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:29.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:29.193 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:29.218 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:29.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:29.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:29.322 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:29.349 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:29.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:29.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:29.451 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:29.476 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:29.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:29.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:29.578 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:29.618 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:29.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:29.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:29.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:29 smithi102 ceph-mon[110183]: pgmap v1591: 119 pgs: 4 active+undersized+remapped, 2 remapped+peering, 1 unknown, 112 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 12 KiB/s rd, 37 KiB/s wr, 914 op/s; 50 KiB/s, 0 objects/s recovering 2024-09-13T14:57:29.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:29 smithi102 ceph-mon[110183]: 1.1d scrub starts 2024-09-13T14:57:29.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:29 smithi102 ceph-mon[110183]: 1.1e scrub starts 2024-09-13T14:57:29.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:29 smithi102 ceph-mon[115269]: pgmap v1591: 119 pgs: 4 active+undersized+remapped, 2 remapped+peering, 1 unknown, 112 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 12 KiB/s rd, 37 KiB/s wr, 914 op/s; 50 KiB/s, 0 objects/s recovering 2024-09-13T14:57:29.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:29 smithi102 ceph-mon[115269]: 1.1d scrub starts 2024-09-13T14:57:29.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:29 smithi102 ceph-mon[115269]: 1.1e scrub starts 2024-09-13T14:57:29.702 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:29 smithi195 ceph-mon[104262]: pgmap v1591: 119 pgs: 4 active+undersized+remapped, 2 remapped+peering, 1 unknown, 112 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 12 KiB/s rd, 37 KiB/s wr, 914 op/s; 50 KiB/s, 0 objects/s recovering 2024-09-13T14:57:29.702 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:29 smithi195 ceph-mon[104262]: 1.1d scrub starts 2024-09-13T14:57:29.702 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:29 smithi195 ceph-mon[104262]: 1.1e scrub starts 2024-09-13T14:57:29.719 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:29.762 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:29.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:29.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:29.864 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:29.898 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:29.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:29.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:29.974 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:57:30.000 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:30.028 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:30.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:30.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:30.129 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:30.163 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:30.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:30.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:30.265 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:30.309 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:30.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:30.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:30.411 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:30.454 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:30.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:30.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:30.556 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:30.592 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:30.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:30.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:30.693 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:30.730 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:30.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:30.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:30.832 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:30.854 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:30.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:30.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:30.956 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:30.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:30.989 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:30.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:31.090 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:31.130 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:31.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:31.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:31.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:31 smithi102 ceph-mon[110183]: 1.b scrub starts 2024-09-13T14:57:31.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:31 smithi102 ceph-mon[110183]: pgmap v1593: 119 pgs: 1 active+clean+scrubbing, 2 peering, 4 active+undersized+remapped, 1 remapped+peering, 111 active+clean; 4.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 24 KiB/s rd, 72 KiB/s wr, 1.10k op/s; 73 KiB/s, 0 objects/s recovering 2024-09-13T14:57:31.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:31 smithi102 ceph-mon[110183]: 1.c scrub starts 2024-09-13T14:57:31.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:31 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:57:31.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:31 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:57:31.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:31 smithi102 ceph-mon[115269]: 1.b scrub starts 2024-09-13T14:57:31.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:31 smithi102 ceph-mon[115269]: pgmap v1593: 119 pgs: 1 active+clean+scrubbing, 2 peering, 4 active+undersized+remapped, 1 remapped+peering, 111 active+clean; 4.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 24 KiB/s rd, 72 KiB/s wr, 1.10k op/s; 73 KiB/s, 0 objects/s recovering 2024-09-13T14:57:31.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:31 smithi102 ceph-mon[115269]: 1.c scrub starts 2024-09-13T14:57:31.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:31 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:57:31.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:31 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:57:31.232 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:31.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:31.269 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:31.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:31.370 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:31.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:31.395 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:31.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:31.496 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:31.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:31 smithi195 ceph-mon[104262]: 1.b scrub starts 2024-09-13T14:57:31.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:31 smithi195 ceph-mon[104262]: pgmap v1593: 119 pgs: 1 active+clean+scrubbing, 2 peering, 4 active+undersized+remapped, 1 remapped+peering, 111 active+clean; 4.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 24 KiB/s rd, 72 KiB/s wr, 1.10k op/s; 73 KiB/s, 0 objects/s recovering 2024-09-13T14:57:31.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:31 smithi195 ceph-mon[104262]: 1.c scrub starts 2024-09-13T14:57:31.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:31 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:57:31.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:31 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:57:31.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:31.537 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:31.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:31.638 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:31.662 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:31.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:31.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:31.764 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:31.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:31.803 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:31.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:31.904 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:31.951 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:31.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:31.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:32.053 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:32.073 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:32 smithi102 ceph-mon[110183]: 1.16 scrub starts 2024-09-13T14:57:32.073 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:32 smithi102 ceph-mon[110183]: 1.6 scrub starts 2024-09-13T14:57:32.073 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:32 smithi102 ceph-mon[110183]: 1.5 scrub starts 2024-09-13T14:57:32.073 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:32 smithi102 ceph-mon[115269]: 1.16 scrub starts 2024-09-13T14:57:32.073 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:32 smithi102 ceph-mon[115269]: 1.6 scrub starts 2024-09-13T14:57:32.073 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:32 smithi102 ceph-mon[115269]: 1.5 scrub starts 2024-09-13T14:57:32.076 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:32.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:32.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:32.178 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:32.202 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:32.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:32.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:32.303 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:32.350 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:32.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:32.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:32.452 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:32.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:32.491 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:32.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:32.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:32 smithi195 ceph-mon[104262]: 1.16 scrub starts 2024-09-13T14:57:32.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:32 smithi195 ceph-mon[104262]: 1.6 scrub starts 2024-09-13T14:57:32.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:32 smithi195 ceph-mon[104262]: 1.5 scrub starts 2024-09-13T14:57:32.592 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:32.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:32.617 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:32.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:32.718 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:32.740 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:57:32.740 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:57:32.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:32.784 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:32.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:32.885 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:32.936 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:32.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:32.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:33.038 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:33.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:33.068 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:33.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:33.169 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:33.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:33.199 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:33.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:33.300 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:33.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:33.336 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:33.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:33.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:33 smithi102 ceph-mon[110183]: pgmap v1594: 119 pgs: 2 active+clean+scrubbing, 2 peering, 2 active+undersized+remapped, 1 remapped+peering, 112 active+clean; 3.8 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 18 KiB/s rd, 52 KiB/s wr, 1.16k op/s; 53 KiB/s, 0 objects/s recovering 2024-09-13T14:57:33.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:33 smithi102 ceph-mon[110183]: 1.7 scrub starts 2024-09-13T14:57:33.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:33 smithi102 ceph-mon[115269]: pgmap v1594: 119 pgs: 2 active+clean+scrubbing, 2 peering, 2 active+undersized+remapped, 1 remapped+peering, 112 active+clean; 3.8 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 18 KiB/s rd, 52 KiB/s wr, 1.16k op/s; 53 KiB/s, 0 objects/s recovering 2024-09-13T14:57:33.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:33 smithi102 ceph-mon[115269]: 1.7 scrub starts 2024-09-13T14:57:33.437 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:33.461 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:33.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:33.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:33.483 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:57:33.486 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:57:33.495 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:33.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:33 smithi195 ceph-mon[104262]: pgmap v1594: 119 pgs: 2 active+clean+scrubbing, 2 peering, 2 active+undersized+remapped, 1 remapped+peering, 112 active+clean; 3.8 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 18 KiB/s rd, 52 KiB/s wr, 1.16k op/s; 53 KiB/s, 0 objects/s recovering 2024-09-13T14:57:33.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:33 smithi195 ceph-mon[104262]: 1.7 scrub starts 2024-09-13T14:57:33.563 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:33.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:33.594 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:33.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:33.695 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:33.745 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:33.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:33.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:33.847 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:33.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:33.892 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:33.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:33.993 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:34.034 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:34.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:34.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:34.136 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:34.174 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:34.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:34.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:34.276 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:34.309 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:34.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:34.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:34.408 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:34 smithi102 ceph-mon[110183]: from='client.46808 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:57:34.408 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:34 smithi102 ceph-mon[110183]: 1.19 scrub starts 2024-09-13T14:57:34.408 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:34 smithi102 ceph-mon[115269]: from='client.46808 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:57:34.408 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:34 smithi102 ceph-mon[115269]: 1.19 scrub starts 2024-09-13T14:57:34.410 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:34.447 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":1594,"stamp":"2024-09-13T14:57:31.813690+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4124655870,"num_objects":62892,"num_object_clones":0,"num_object_copies":188676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62892,"num_whiteouts":0,"num_read":6048,"num_read_kb":13118,"num_write":149753,"num_write_kb":6786634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5180,"num_bytes_recovered":333911698,"num_keys_recovered":30,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100919,"ondisk_log_size":100919,"up":357,"acting":354,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":357,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":18199128,"kb_used_data":12130508,"kb_used_omap":40,"kb_used_meta":6068439,"kb_avail":638016936,"statfs":{"total":671965249536,"available":653329342464,"internally_reserved":0,"allocated":12421640192,"data_stored":12404587728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41319,"internal_metadata":6214082201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,4,6,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-451008255,"num_objects":-6883,"num_object_clones":0,"num_object_copies":-20649,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6883,"num_whiteouts":0,"num_read":49,"num_read_kb":106,"num_write":6912,"num_write_kb":312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":326946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.004092"},"pg_stats":[{"pgid":"1.11","version":"90'149","reported_seq":453,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T14:57:27.501514+0000","last_change":"2024-09-13T14:56:12.490567+0000","last_active":"2024-09-13T14:57:27.501514+0000","last_peered":"2024-09-13T14:57:27.501514+0000","last_clean":"2024-09-13T14:57:27.501514+0000","last_became_active":"2024-09-13T14:56:12.490253+0000","last_became_peered":"2024-09-13T14:56:12.490253+0000","last_unstale":"2024-09-13T14:57:27.501514+0000","last_undegraded":"2024-09-13T14:57:27.501514+0000","last_fullsized":"2024-09-13T14:57:27.501514+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:38.688341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.688341+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:20.151605+0000","scrub_duration":0.0054331650000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":867,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T14:57:27.510170+0000","last_change":"2024-09-13T14:47:47.868783+0000","last_active":"2024-09-13T14:57:27.510170+0000","last_peered":"2024-09-13T14:57:27.510170+0000","last_clean":"2024-09-13T14:57:27.510170+0000","last_became_active":"2024-09-13T14:47:47.868478+0000","last_became_peered":"2024-09-13T14:47:47.868478+0000","last_unstale":"2024-09-13T14:57:27.510170+0000","last_undegraded":"2024-09-13T14:57:27.510170+0000","last_fullsized":"2024-09-13T14:57:27.510170+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:27:01.784142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":514,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-09-13T14:57:25.458355+0000","last_change":"2024-09-13T14:55:17.943558+0000","last_active":"2024-09-13T14:57:25.458355+0000","last_peered":"2024-09-13T14:57:25.458355+0000","last_clean":"2024-09-13T14:57:25.458355+0000","last_became_active":"2024-09-13T14:55:17.927162+0000","last_became_peered":"2024-09-13T14:55:17.927162+0000","last_unstale":"2024-09-13T14:57:25.458355+0000","last_undegraded":"2024-09-13T14:57:25.458355+0000","last_fullsized":"2024-09-13T14:57:25.458355+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:37:44.820559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":102,"num_keys_recovered":30,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"90'149","reported_seq":546,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T14:57:27.505901+0000","last_change":"2024-09-13T14:47:41.482552+0000","last_active":"2024-09-13T14:57:27.505901+0000","last_peered":"2024-09-13T14:57:27.505901+0000","last_clean":"2024-09-13T14:57:27.505901+0000","last_became_active":"2024-09-13T14:47:38.466659+0000","last_became_peered":"2024-09-13T14:47:38.466659+0000","last_unstale":"2024-09-13T14:57:27.505901+0000","last_undegraded":"2024-09-13T14:57:27.505901+0000","last_fullsized":"2024-09-13T14:57:27.505901+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:41.482364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:41.482364+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:50:49.239389+0000","scrub_duration":0.027843855000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":282,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T14:57:27.502540+0000","last_change":"2024-09-13T14:36:13.862181+0000","last_active":"2024-09-13T14:57:27.502540+0000","last_peered":"2024-09-13T14:57:27.502540+0000","last_clean":"2024-09-13T14:57:27.502540+0000","last_became_active":"2024-09-13T14:36:13.861873+0000","last_became_peered":"2024-09-13T14:36:13.861873+0000","last_unstale":"2024-09-13T14:57:27.502540+0000","last_undegraded":"2024-09-13T14:57:27.502540+0000","last_fullsized":"2024-09-13T14:57:27.502540+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:14.085700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":52,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-09-13T14:57:25.457654+0000","last_change":"2024-09-13T14:56:12.489695+0000","last_active":"2024-09-13T14:57:25.457654+0000","last_peered":"2024-09-13T14:57:25.457654+0000","last_clean":"2024-09-13T14:57:25.457654+0000","last_became_active":"2024-09-13T14:56:12.489302+0000","last_became_peered":"2024-09-13T14:56:12.489302+0000","last_unstale":"2024-09-13T14:57:25.457654+0000","last_undegraded":"2024-09-13T14:57:25.457654+0000","last_fullsized":"2024-09-13T14:57:25.457654+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:28:53.275229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"90'149","reported_seq":733,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T14:57:28.509224+0000","last_change":"2024-09-13T14:55:20.150382+0000","last_active":"2024-09-13T14:57:28.509224+0000","last_peered":"2024-09-13T14:57:28.509224+0000","last_clean":"2024-09-13T14:57:28.509224+0000","last_became_active":"2024-09-13T14:55:20.149422+0000","last_became_peered":"2024-09-13T14:55:20.149422+0000","last_unstale":"2024-09-13T14:57:28.509224+0000","last_undegraded":"2024-09-13T14:57:28.509224+0000","last_fullsized":"2024-09-13T14:57:28.509224+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:51.226748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:51.226748+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:51:07.379350+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":379,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T14:57:27.507532+0000","last_change":"2024-09-13T14:56:12.490265+0000","last_active":"2024-09-13T14:57:27.507532+0000","last_peered":"2024-09-13T14:57:27.507532+0000","last_clean":"2024-09-13T14:57:27.507532+0000","last_became_active":"2024-09-13T14:56:12.489937+0000","last_became_peered":"2024-09-13T14:56:12.489937+0000","last_unstale":"2024-09-13T14:57:27.507532+0000","last_undegraded":"2024-09-13T14:57:27.507532+0000","last_fullsized":"2024-09-13T14:57:27.507532+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:49.461532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":460,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-09-13T14:57:25.457888+0000","last_change":"2024-09-13T14:55:17.934695+0000","last_active":"2024-09-13T14:57:25.457888+0000","last_peered":"2024-09-13T14:57:25.457888+0000","last_clean":"2024-09-13T14:57:25.457888+0000","last_became_active":"2024-09-13T14:55:17.934165+0000","last_became_peered":"2024-09-13T14:55:17.934165+0000","last_unstale":"2024-09-13T14:57:25.457888+0000","last_undegraded":"2024-09-13T14:57:25.457888+0000","last_fullsized":"2024-09-13T14:57:25.457888+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:36:25.739706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"90'149","reported_seq":563,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T14:57:27.502738+0000","last_change":"2024-09-13T14:47:50.892667+0000","last_active":"2024-09-13T14:57:27.502738+0000","last_peered":"2024-09-13T14:57:27.502738+0000","last_clean":"2024-09-13T14:57:27.502738+0000","last_became_active":"2024-09-13T14:47:50.883312+0000","last_became_peered":"2024-09-13T14:47:50.883312+0000","last_unstale":"2024-09-13T14:57:27.502738+0000","last_undegraded":"2024-09-13T14:57:27.502738+0000","last_fullsized":"2024-09-13T14:57:27.502738+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:45.441819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:45.441819+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:39.138573+0000","scrub_duration":0.021368558999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":893,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T14:57:27.511135+0000","last_change":"2024-09-13T14:50:27.491166+0000","last_active":"2024-09-13T14:57:27.511135+0000","last_peered":"2024-09-13T14:57:27.511135+0000","last_clean":"2024-09-13T14:57:27.511135+0000","last_became_active":"2024-09-13T14:50:27.489905+0000","last_became_peered":"2024-09-13T14:50:27.489905+0000","last_unstale":"2024-09-13T14:57:27.511135+0000","last_undegraded":"2024-09-13T14:57:27.511135+0000","last_fullsized":"2024-09-13T14:57:27.511135+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:32.237904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":147,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T14:57:27.502706+0000","last_change":"2024-09-13T14:56:12.490197+0000","last_active":"2024-09-13T14:57:27.502706+0000","last_peered":"2024-09-13T14:57:27.502706+0000","last_clean":"2024-09-13T14:57:27.502706+0000","last_became_active":"2024-09-13T14:56:12.489554+0000","last_became_peered":"2024-09-13T14:56:12.489554+0000","last_unstale":"2024-09-13T14:57:27.502706+0000","last_undegraded":"2024-09-13T14:57:27.502706+0000","last_fullsized":"2024-09-13T14:57:27.502706+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:06.773211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"478'9350","reported_seq":10589,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T14:57:31.729791+0000","last_change":"2024-09-13T14:55:22.944506+0000","last_active":"2024-09-13T14:57:31.729791+0000","last_peered":"2024-09-13T14:57:31.729791+0000","last_clean":"2024-09-13T14:57:31.729791+0000","last_became_active":"2024-09-13T14:55:22.942014+0000","last_became_peered":"2024-09-13T14:55:22.942014+0000","last_unstale":"2024-09-13T14:57:31.729791+0000","last_undegraded":"2024-09-13T14:57:31.729791+0000","last_fullsized":"2024-09-13T14:57:31.729791+0000","mapping_epoch":446,"log_start":"460'2611","ondisk_log_start":"460'2611","created":436,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":6739,"log_dups_size":0,"ondisk_log_size":6739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:00:33.167429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243531776,"num_objects":3716,"num_object_clones":0,"num_object_copies":11148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9350,"num_write_kb":418112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":584,"num_bytes_recovered":38273024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"90'149","reported_seq":589,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-09-13T14:57:25.458450+0000","last_change":"2024-09-13T14:57:17.394510+0000","last_active":"2024-09-13T14:57:25.458450+0000","last_peered":"2024-09-13T14:57:25.458450+0000","last_clean":"2024-09-13T14:57:25.458450+0000","last_became_active":"2024-09-13T14:57:17.394134+0000","last_became_peered":"2024-09-13T14:57:17.394134+0000","last_unstale":"2024-09-13T14:57:25.458450+0000","last_undegraded":"2024-09-13T14:57:25.458450+0000","last_fullsized":"2024-09-13T14:57:25.458450+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:40.786610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:40.786610+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013725185000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":258,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-09-13T14:57:25.459396+0000","last_change":"2024-09-13T14:55:17.947508+0000","last_active":"2024-09-13T14:57:25.459396+0000","last_peered":"2024-09-13T14:57:25.459396+0000","last_clean":"2024-09-13T14:57:25.459396+0000","last_became_active":"2024-09-13T14:55:17.932604+0000","last_became_peered":"2024-09-13T14:55:17.932604+0000","last_unstale":"2024-09-13T14:57:25.459396+0000","last_undegraded":"2024-09-13T14:57:25.459396+0000","last_fullsized":"2024-09-13T14:57:25.459396+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:55.084612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":132,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":303,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T14:57:27.505993+0000","last_change":"2024-09-13T14:38:53.898907+0000","last_active":"2024-09-13T14:57:27.505993+0000","last_peered":"2024-09-13T14:57:27.505993+0000","last_clean":"2024-09-13T14:57:27.505993+0000","last_became_active":"2024-09-13T14:38:53.898572+0000","last_became_peered":"2024-09-13T14:38:53.898572+0000","last_unstale":"2024-09-13T14:57:27.505993+0000","last_undegraded":"2024-09-13T14:57:27.505993+0000","last_fullsized":"2024-09-13T14:57:27.505993+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:29.222150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"477'9045","reported_seq":9140,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T14:57:28.463820+0000","last_change":"2024-09-13T14:54:58.384717+0000","last_active":"2024-09-13T14:57:28.463820+0000","last_peered":"2024-09-13T14:57:28.463820+0000","last_clean":"2024-09-13T14:57:28.463820+0000","last_became_active":"2024-09-13T14:54:58.384225+0000","last_became_peered":"2024-09-13T14:54:58.384225+0000","last_unstale":"2024-09-13T14:57:28.463820+0000","last_undegraded":"2024-09-13T14:57:28.463820+0000","last_fullsized":"2024-09-13T14:57:28.463820+0000","mapping_epoch":436,"log_start":"460'3957","ondisk_log_start":"460'3957","created":436,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":5088,"log_dups_size":0,"ondisk_log_size":5088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:25:22.716676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266403840,"num_objects":4065,"num_object_clones":0,"num_object_copies":12195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9045,"num_write_kb":419520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"90'149","reported_seq":756,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T14:57:27.510029+0000","last_change":"2024-09-13T14:57:25.475570+0000","last_active":"2024-09-13T14:57:27.510029+0000","last_peered":"2024-09-13T14:57:27.510029+0000","last_clean":"2024-09-13T14:57:27.510029+0000","last_became_active":"2024-09-13T14:57:17.404299+0000","last_became_peered":"2024-09-13T14:57:17.404299+0000","last_unstale":"2024-09-13T14:57:27.510029+0000","last_undegraded":"2024-09-13T14:57:27.510029+0000","last_fullsized":"2024-09-13T14:57:27.510029+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:32.060817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:32.060817+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T14:57:29.920002+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163840,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":271,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-09-13T14:57:25.459240+0000","last_change":"2024-09-13T14:55:17.949010+0000","last_active":"2024-09-13T14:57:25.459240+0000","last_peered":"2024-09-13T14:57:25.459240+0000","last_clean":"2024-09-13T14:57:25.459240+0000","last_became_active":"2024-09-13T14:55:17.932712+0000","last_became_peered":"2024-09-13T14:55:17.932712+0000","last_unstale":"2024-09-13T14:57:25.459240+0000","last_undegraded":"2024-09-13T14:57:25.459240+0000","last_fullsized":"2024-09-13T14:57:25.459240+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:46:43.390958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":433,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T14:57:28.507981+0000","last_change":"2024-09-13T14:55:17.929723+0000","last_active":"2024-09-13T14:57:28.507981+0000","last_peered":"2024-09-13T14:57:28.507981+0000","last_clean":"2024-09-13T14:57:28.507981+0000","last_became_active":"2024-09-13T14:55:17.929131+0000","last_became_peered":"2024-09-13T14:55:17.929131+0000","last_unstale":"2024-09-13T14:57:28.507981+0000","last_undegraded":"2024-09-13T14:57:28.507981+0000","last_fullsized":"2024-09-13T14:57:28.507981+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:27:15.828973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.4","version":"477'9222","reported_seq":10574,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T14:57:28.477113+0000","last_change":"2024-09-13T14:56:12.495568+0000","last_active":"2024-09-13T14:57:28.477113+0000","last_peered":"2024-09-13T14:57:28.477113+0000","last_clean":"2024-09-13T14:57:28.477113+0000","last_became_active":"2024-09-13T14:56:12.490664+0000","last_became_peered":"2024-09-13T14:56:12.490664+0000","last_unstale":"2024-09-13T14:57:28.477113+0000","last_undegraded":"2024-09-13T14:57:28.477113+0000","last_fullsized":"2024-09-13T14:57:28.477113+0000","mapping_epoch":462,"log_start":"461'4131","ondisk_log_start":"461'4131","created":436,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":5091,"log_dups_size":0,"ondisk_log_size":5091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:17:29.226407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265027584,"num_objects":4044,"num_object_clones":0,"num_object_copies":12132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9222,"num_write_kb":424512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":623,"num_bytes_recovered":40828928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.f","version":"90'149","reported_seq":607,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T14:57:27.507128+0000","last_change":"2024-09-13T14:55:20.140396+0000","last_active":"2024-09-13T14:57:27.507128+0000","last_peered":"2024-09-13T14:57:27.507128+0000","last_clean":"2024-09-13T14:57:27.507128+0000","last_became_active":"2024-09-13T14:55:20.140087+0000","last_became_peered":"2024-09-13T14:55:20.140087+0000","last_unstale":"2024-09-13T14:57:27.507128+0000","last_undegraded":"2024-09-13T14:57:27.507128+0000","last_fullsized":"2024-09-13T14:57:27.507128+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:34.091782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:34.091782+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:46:05.540133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":482,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-09-13T14:57:25.458786+0000","last_change":"2024-09-13T14:37:49.611774+0000","last_active":"2024-09-13T14:57:25.458786+0000","last_peered":"2024-09-13T14:57:25.458786+0000","last_clean":"2024-09-13T14:57:25.458786+0000","last_became_active":"2024-09-13T14:37:49.611088+0000","last_became_peered":"2024-09-13T14:37:49.611088+0000","last_unstale":"2024-09-13T14:57:25.458786+0000","last_undegraded":"2024-09-13T14:57:25.458786+0000","last_fullsized":"2024-09-13T14:57:25.458786+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:08.933213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":298,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T14:57:28.513152+0000","last_change":"2024-09-13T14:47:47.865523+0000","last_active":"2024-09-13T14:57:28.513152+0000","last_peered":"2024-09-13T14:57:28.513152+0000","last_clean":"2024-09-13T14:57:28.513152+0000","last_became_active":"2024-09-13T14:47:47.865180+0000","last_became_peered":"2024-09-13T14:47:47.865180+0000","last_unstale":"2024-09-13T14:57:28.513152+0000","last_undegraded":"2024-09-13T14:57:28.513152+0000","last_fullsized":"2024-09-13T14:57:28.513152+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:39:35.479251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.5","version":"478'9146","reported_seq":9242,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T14:57:31.621574+0000","last_change":"2024-09-13T14:54:57.846849+0000","last_active":"2024-09-13T14:57:31.621574+0000","last_peered":"2024-09-13T14:57:31.621574+0000","last_clean":"2024-09-13T14:57:31.621574+0000","last_became_active":"2024-09-13T14:54:57.846557+0000","last_became_peered":"2024-09-13T14:54:57.846557+0000","last_unstale":"2024-09-13T14:57:31.621574+0000","last_undegraded":"2024-09-13T14:57:31.621574+0000","last_fullsized":"2024-09-13T14:57:31.621574+0000","mapping_epoch":436,"log_start":"460'2291","ondisk_log_start":"460'2291","created":436,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":6855,"log_dups_size":0,"ondisk_log_size":6855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:49.590084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250347520,"num_objects":3820,"num_object_clones":0,"num_object_copies":11460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9146,"num_write_kb":414912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"90'149","reported_seq":499,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-09-13T14:57:25.458903+0000","last_change":"2024-09-13T14:57:17.385260+0000","last_active":"2024-09-13T14:57:25.458903+0000","last_peered":"2024-09-13T14:57:25.458903+0000","last_clean":"2024-09-13T14:57:25.458903+0000","last_became_active":"2024-09-13T14:57:17.384951+0000","last_became_peered":"2024-09-13T14:57:17.384951+0000","last_unstale":"2024-09-13T14:57:25.458903+0000","last_undegraded":"2024-09-13T14:57:25.458903+0000","last_fullsized":"2024-09-13T14:57:25.458903+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:30.526999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:30.526999+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":294,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T14:57:27.505475+0000","last_change":"2024-09-13T14:37:49.168003+0000","last_active":"2024-09-13T14:57:27.505475+0000","last_peered":"2024-09-13T14:57:27.505475+0000","last_clean":"2024-09-13T14:57:27.505475+0000","last_became_active":"2024-09-13T14:37:49.167638+0000","last_became_peered":"2024-09-13T14:37:49.167638+0000","last_unstale":"2024-09-13T14:57:27.505475+0000","last_undegraded":"2024-09-13T14:57:27.505475+0000","last_fullsized":"2024-09-13T14:57:27.505475+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:29.051981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":460,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T14:57:28.509023+0000","last_change":"2024-09-13T14:56:12.491208+0000","last_active":"2024-09-13T14:57:28.509023+0000","last_peered":"2024-09-13T14:57:28.509023+0000","last_clean":"2024-09-13T14:57:28.509023+0000","last_became_active":"2024-09-13T14:56:12.490943+0000","last_became_peered":"2024-09-13T14:56:12.490943+0000","last_unstale":"2024-09-13T14:57:28.509023+0000","last_undegraded":"2024-09-13T14:57:28.509023+0000","last_fullsized":"2024-09-13T14:57:28.509023+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:31:57.892194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"478'9183","reported_seq":10421,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T14:57:29.522850+0000","last_change":"2024-09-13T14:55:37.154924+0000","last_active":"2024-09-13T14:57:29.522850+0000","last_peered":"2024-09-13T14:57:29.522850+0000","last_clean":"2024-09-13T14:57:29.522850+0000","last_became_active":"2024-09-13T14:55:37.153166+0000","last_became_peered":"2024-09-13T14:55:37.153166+0000","last_unstale":"2024-09-13T14:57:29.522850+0000","last_undegraded":"2024-09-13T14:57:29.522850+0000","last_fullsized":"2024-09-13T14:57:29.522850+0000","mapping_epoch":459,"log_start":"460'3922","ondisk_log_start":"460'3922","created":436,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":5261,"log_dups_size":0,"ondisk_log_size":5261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:48:15.748240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":259063808,"num_objects":3953,"num_object_clones":0,"num_object_copies":11859,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9183,"num_write_kb":420352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":565,"num_bytes_recovered":37027840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"90'149","reported_seq":662,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-09-13T14:57:25.457736+0000","last_change":"2024-09-13T14:57:17.381130+0000","last_active":"2024-09-13T14:57:25.457736+0000","last_peered":"2024-09-13T14:57:25.457736+0000","last_clean":"2024-09-13T14:57:25.457736+0000","last_became_active":"2024-09-13T14:57:17.380843+0000","last_became_peered":"2024-09-13T14:57:17.380843+0000","last_unstale":"2024-09-13T14:57:25.457736+0000","last_undegraded":"2024-09-13T14:57:25.457736+0000","last_fullsized":"2024-09-13T14:57:25.457736+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:46:55.355521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:46:55.355521+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002025448,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":853,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T14:57:27.506390+0000","last_change":"2024-09-13T14:55:17.944596+0000","last_active":"2024-09-13T14:57:27.506390+0000","last_peered":"2024-09-13T14:57:27.506390+0000","last_clean":"2024-09-13T14:57:27.506390+0000","last_became_active":"2024-09-13T14:55:17.935767+0000","last_became_peered":"2024-09-13T14:55:17.935767+0000","last_unstale":"2024-09-13T14:57:27.506390+0000","last_undegraded":"2024-09-13T14:57:27.506390+0000","last_fullsized":"2024-09-13T14:57:27.506390+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:22:19.619639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":384,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T14:57:28.502217+0000","last_change":"2024-09-13T14:55:17.933005+0000","last_active":"2024-09-13T14:57:28.502217+0000","last_peered":"2024-09-13T14:57:28.502217+0000","last_clean":"2024-09-13T14:57:28.502217+0000","last_became_active":"2024-09-13T14:55:17.931575+0000","last_became_peered":"2024-09-13T14:55:17.931575+0000","last_unstale":"2024-09-13T14:57:28.502217+0000","last_undegraded":"2024-09-13T14:57:28.502217+0000","last_fullsized":"2024-09-13T14:57:28.502217+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:50:20.352970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"478'9462","reported_seq":9559,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T14:57:31.579421+0000","last_change":"2024-09-13T14:54:57.846161+0000","last_active":"2024-09-13T14:57:31.579421+0000","last_peered":"2024-09-13T14:57:31.579421+0000","last_clean":"2024-09-13T14:57:31.579421+0000","last_became_active":"2024-09-13T14:54:57.845348+0000","last_became_peered":"2024-09-13T14:54:57.845348+0000","last_unstale":"2024-09-13T14:57:31.579421+0000","last_undegraded":"2024-09-13T14:57:31.579421+0000","last_fullsized":"2024-09-13T14:57:31.579421+0000","mapping_epoch":436,"log_start":"460'2576","ondisk_log_start":"460'2576","created":436,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":6886,"log_dups_size":0,"ondisk_log_size":6886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:07:01.751407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250085376,"num_objects":3816,"num_object_clones":0,"num_object_copies":11448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9462,"num_write_kb":424896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"90'149","reported_seq":567,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T14:57:27.502628+0000","last_change":"2024-09-13T14:57:17.383979+0000","last_active":"2024-09-13T14:57:27.502628+0000","last_peered":"2024-09-13T14:57:27.502628+0000","last_clean":"2024-09-13T14:57:27.502628+0000","last_became_active":"2024-09-13T14:57:17.383307+0000","last_became_peered":"2024-09-13T14:57:17.383307+0000","last_unstale":"2024-09-13T14:57:27.502628+0000","last_undegraded":"2024-09-13T14:57:27.502628+0000","last_fullsized":"2024-09-13T14:57:27.502628+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:46:54.393204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:46:54.393204+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015172223,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131344,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":342,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T14:57:28.513019+0000","last_change":"2024-09-13T14:55:17.935369+0000","last_active":"2024-09-13T14:57:28.513019+0000","last_peered":"2024-09-13T14:57:28.513019+0000","last_clean":"2024-09-13T14:57:28.513019+0000","last_became_active":"2024-09-13T14:55:17.930815+0000","last_became_peered":"2024-09-13T14:55:17.930815+0000","last_unstale":"2024-09-13T14:57:28.513019+0000","last_undegraded":"2024-09-13T14:57:28.513019+0000","last_fullsized":"2024-09-13T14:57:28.513019+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:59:31.247160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":470,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T14:57:28.510583+0000","last_change":"2024-09-13T14:55:17.932243+0000","last_active":"2024-09-13T14:57:28.510583+0000","last_peered":"2024-09-13T14:57:28.510583+0000","last_clean":"2024-09-13T14:57:28.510583+0000","last_became_active":"2024-09-13T14:55:17.931961+0000","last_became_peered":"2024-09-13T14:55:17.931961+0000","last_unstale":"2024-09-13T14:57:28.510583+0000","last_undegraded":"2024-09-13T14:57:28.510583+0000","last_fullsized":"2024-09-13T14:57:28.510583+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:23:52.680942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.0","version":"478'9290","reported_seq":10707,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T14:57:29.541365+0000","last_change":"2024-09-13T14:55:24.959843+0000","last_active":"2024-09-13T14:57:29.541365+0000","last_peered":"2024-09-13T14:57:29.541365+0000","last_clean":"2024-09-13T14:57:29.541365+0000","last_became_active":"2024-09-13T14:55:24.957491+0000","last_became_peered":"2024-09-13T14:55:24.957491+0000","last_unstale":"2024-09-13T14:57:29.541365+0000","last_undegraded":"2024-09-13T14:57:29.541365+0000","last_fullsized":"2024-09-13T14:57:29.541365+0000","mapping_epoch":448,"log_start":"460'4038","ondisk_log_start":"460'4038","created":436,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":5252,"log_dups_size":0,"ondisk_log_size":5252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:09:19.929604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":254017536,"num_objects":3876,"num_object_clones":0,"num_object_copies":11628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9290,"num_write_kb":421312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":654,"num_bytes_recovered":42860544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"90'149","reported_seq":751,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T14:57:28.508942+0000","last_change":"2024-09-13T14:57:25.475374+0000","last_active":"2024-09-13T14:57:28.508942+0000","last_peered":"2024-09-13T14:57:28.508942+0000","last_clean":"2024-09-13T14:57:28.508942+0000","last_became_active":"2024-09-13T14:57:17.389370+0000","last_became_peered":"2024-09-13T14:57:17.389370+0000","last_unstale":"2024-09-13T14:57:28.508942+0000","last_undegraded":"2024-09-13T14:57:28.508942+0000","last_fullsized":"2024-09-13T14:57:28.508942+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:39.645908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:39.645908+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T14:57:29.631237+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":391,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T14:57:28.502495+0000","last_change":"2024-09-13T14:55:17.932199+0000","last_active":"2024-09-13T14:57:28.502495+0000","last_peered":"2024-09-13T14:57:28.502495+0000","last_clean":"2024-09-13T14:57:28.502495+0000","last_became_active":"2024-09-13T14:55:17.931873+0000","last_became_peered":"2024-09-13T14:55:17.931873+0000","last_unstale":"2024-09-13T14:57:28.502495+0000","last_undegraded":"2024-09-13T14:57:28.502495+0000","last_fullsized":"2024-09-13T14:57:28.502495+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:02:18.004113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":417,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T14:57:28.513434+0000","last_change":"2024-09-13T14:55:17.931269+0000","last_active":"2024-09-13T14:57:28.513434+0000","last_peered":"2024-09-13T14:57:28.513434+0000","last_clean":"2024-09-13T14:57:28.513434+0000","last_became_active":"2024-09-13T14:55:17.930909+0000","last_became_peered":"2024-09-13T14:55:17.930909+0000","last_unstale":"2024-09-13T14:57:28.513434+0000","last_undegraded":"2024-09-13T14:57:28.513434+0000","last_fullsized":"2024-09-13T14:57:28.513434+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:47.592526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"476'9475","reported_seq":10879,"reported_epoch":476,"state":"active+clean","last_fresh":"2024-09-13T14:57:26.823170+0000","last_change":"2024-09-13T14:55:35.136618+0000","last_active":"2024-09-13T14:57:26.823170+0000","last_peered":"2024-09-13T14:57:26.823170+0000","last_clean":"2024-09-13T14:57:26.823170+0000","last_became_active":"2024-09-13T14:55:35.134043+0000","last_became_peered":"2024-09-13T14:55:35.134043+0000","last_unstale":"2024-09-13T14:57:26.823170+0000","last_undegraded":"2024-09-13T14:57:26.823170+0000","last_fullsized":"2024-09-13T14:57:26.823170+0000","mapping_epoch":457,"log_start":"460'3797","ondisk_log_start":"460'3797","created":436,"last_epoch_clean":458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":5678,"log_dups_size":0,"ondisk_log_size":5678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:55.445002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":277807104,"num_objects":4239,"num_object_clones":0,"num_object_copies":12717,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9475,"num_write_kb":438848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":667,"num_bytes_recovered":43712512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"90'149","reported_seq":567,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T14:57:27.501828+0000","last_change":"2024-09-13T14:57:17.378924+0000","last_active":"2024-09-13T14:57:27.501828+0000","last_peered":"2024-09-13T14:57:27.501828+0000","last_clean":"2024-09-13T14:57:27.501828+0000","last_became_active":"2024-09-13T14:57:17.378580+0000","last_became_peered":"2024-09-13T14:57:17.378580+0000","last_unstale":"2024-09-13T14:57:27.501828+0000","last_undegraded":"2024-09-13T14:57:27.501828+0000","last_fullsized":"2024-09-13T14:57:27.501828+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:46:52.334301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:46:52.334301+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050940330000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":868,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T14:57:27.505980+0000","last_change":"2024-09-13T14:47:47.861249+0000","last_active":"2024-09-13T14:57:27.505980+0000","last_peered":"2024-09-13T14:57:27.505980+0000","last_clean":"2024-09-13T14:57:27.505980+0000","last_became_active":"2024-09-13T14:47:47.860873+0000","last_became_peered":"2024-09-13T14:47:47.860873+0000","last_unstale":"2024-09-13T14:57:27.505980+0000","last_undegraded":"2024-09-13T14:57:27.505980+0000","last_fullsized":"2024-09-13T14:57:27.505980+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:37:50.709437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":305,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T14:57:28.507373+0000","last_change":"2024-09-13T14:55:17.933070+0000","last_active":"2024-09-13T14:57:28.507373+0000","last_peered":"2024-09-13T14:57:28.507373+0000","last_clean":"2024-09-13T14:57:28.507373+0000","last_became_active":"2024-09-13T14:55:17.931232+0000","last_became_peered":"2024-09-13T14:55:17.931232+0000","last_unstale":"2024-09-13T14:57:28.507373+0000","last_undegraded":"2024-09-13T14:57:28.507373+0000","last_fullsized":"2024-09-13T14:57:28.507373+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:22:46.719483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"478'9184","reported_seq":9280,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T14:57:29.530267+0000","last_change":"2024-09-13T14:54:58.384133+0000","last_active":"2024-09-13T14:57:29.530267+0000","last_peered":"2024-09-13T14:57:29.530267+0000","last_clean":"2024-09-13T14:57:29.530267+0000","last_became_active":"2024-09-13T14:54:58.383823+0000","last_became_peered":"2024-09-13T14:54:58.383823+0000","last_unstale":"2024-09-13T14:57:29.530267+0000","last_undegraded":"2024-09-13T14:57:29.530267+0000","last_fullsized":"2024-09-13T14:57:29.530267+0000","mapping_epoch":436,"log_start":"460'3922","ondisk_log_start":"460'3922","created":436,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":5262,"log_dups_size":0,"ondisk_log_size":5262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:30.724982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255721472,"num_objects":3902,"num_object_clones":0,"num_object_copies":11706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9184,"num_write_kb":418752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":659,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-09-13T14:57:25.457657+0000","last_change":"2024-09-13T14:47:31.684319+0000","last_active":"2024-09-13T14:57:25.457657+0000","last_peered":"2024-09-13T14:57:25.457657+0000","last_clean":"2024-09-13T14:57:25.457657+0000","last_became_active":"2024-09-13T14:47:29.415658+0000","last_became_peered":"2024-09-13T14:47:29.415658+0000","last_unstale":"2024-09-13T14:57:25.457657+0000","last_undegraded":"2024-09-13T14:57:25.457657+0000","last_fullsized":"2024-09-13T14:57:25.457657+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:31.681433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:31.681433+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:29:34.663513+0000","scrub_duration":0.0059601139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":790,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T14:57:27.506225+0000","last_change":"2024-09-13T14:46:04.780718+0000","last_active":"2024-09-13T14:57:27.506225+0000","last_peered":"2024-09-13T14:57:27.506225+0000","last_clean":"2024-09-13T14:57:27.506225+0000","last_became_active":"2024-09-13T14:46:04.780120+0000","last_became_peered":"2024-09-13T14:46:04.780120+0000","last_unstale":"2024-09-13T14:57:27.506225+0000","last_undegraded":"2024-09-13T14:57:27.506225+0000","last_fullsized":"2024-09-13T14:57:27.506225+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:57:14.475888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":795,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T14:57:27.507932+0000","last_change":"2024-09-13T14:37:49.160719+0000","last_active":"2024-09-13T14:57:27.507932+0000","last_peered":"2024-09-13T14:57:27.507932+0000","last_clean":"2024-09-13T14:57:27.507932+0000","last_became_active":"2024-09-13T14:37:49.160401+0000","last_became_peered":"2024-09-13T14:37:49.160401+0000","last_unstale":"2024-09-13T14:57:27.507932+0000","last_undegraded":"2024-09-13T14:57:27.507932+0000","last_fullsized":"2024-09-13T14:57:27.507932+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:21.737945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":119,"reported_epoch":476,"state":"active+clean","last_fresh":"2024-09-13T14:57:26.465343+0000","last_change":"2024-09-13T14:55:17.933431+0000","last_active":"2024-09-13T14:57:26.465343+0000","last_peered":"2024-09-13T14:57:26.465343+0000","last_clean":"2024-09-13T14:57:26.465343+0000","last_became_active":"2024-09-13T14:55:17.932711+0000","last_became_peered":"2024-09-13T14:55:17.932711+0000","last_unstale":"2024-09-13T14:57:26.465343+0000","last_undegraded":"2024-09-13T14:57:26.465343+0000","last_fullsized":"2024-09-13T14:57:26.465343+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:47.712045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"478'9247","reported_seq":9354,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T14:57:31.560936+0000","last_change":"2024-09-13T14:56:12.516271+0000","last_active":"2024-09-13T14:57:31.560936+0000","last_peered":"2024-09-13T14:57:31.560936+0000","last_clean":"2024-09-13T14:57:31.560936+0000","last_became_active":"2024-09-13T14:56:12.494537+0000","last_became_peered":"2024-09-13T14:56:12.494537+0000","last_unstale":"2024-09-13T14:57:31.560936+0000","last_undegraded":"2024-09-13T14:57:31.560936+0000","last_fullsized":"2024-09-13T14:57:31.560936+0000","mapping_epoch":462,"log_start":"460'2359","ondisk_log_start":"460'2359","created":436,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":6888,"log_dups_size":0,"ondisk_log_size":6888,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:07:34.989938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247267328,"num_objects":3773,"num_object_clones":0,"num_object_copies":11319,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9247,"num_write_kb":416640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"90'149","reported_seq":740,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T14:57:28.509096+0000","last_change":"2024-09-13T14:55:20.148734+0000","last_active":"2024-09-13T14:57:28.509096+0000","last_peered":"2024-09-13T14:57:28.509096+0000","last_clean":"2024-09-13T14:57:28.509096+0000","last_became_active":"2024-09-13T14:55:20.147918+0000","last_became_peered":"2024-09-13T14:55:20.147918+0000","last_unstale":"2024-09-13T14:57:28.509096+0000","last_undegraded":"2024-09-13T14:57:28.509096+0000","last_fullsized":"2024-09-13T14:57:28.509096+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:34.603316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:34.603316+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:27.859151+0000","scrub_duration":0.0057131120000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":729,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-09-13T14:57:25.457653+0000","last_change":"2024-09-13T14:47:47.892973+0000","last_active":"2024-09-13T14:57:25.457653+0000","last_peered":"2024-09-13T14:57:25.457653+0000","last_clean":"2024-09-13T14:57:25.457653+0000","last_became_active":"2024-09-13T14:47:47.866986+0000","last_became_peered":"2024-09-13T14:47:47.866986+0000","last_unstale":"2024-09-13T14:57:25.457653+0000","last_undegraded":"2024-09-13T14:57:25.457653+0000","last_fullsized":"2024-09-13T14:57:25.457653+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:07.353290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":452,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T14:57:27.505617+0000","last_change":"2024-09-13T14:56:12.490871+0000","last_active":"2024-09-13T14:57:27.505617+0000","last_peered":"2024-09-13T14:57:27.505617+0000","last_clean":"2024-09-13T14:57:27.505617+0000","last_became_active":"2024-09-13T14:56:12.490535+0000","last_became_peered":"2024-09-13T14:56:12.490535+0000","last_unstale":"2024-09-13T14:57:27.505617+0000","last_undegraded":"2024-09-13T14:57:27.505617+0000","last_fullsized":"2024-09-13T14:57:27.505617+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:49.429179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":254,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T14:57:28.513152+0000","last_change":"2024-09-13T14:56:12.490229+0000","last_active":"2024-09-13T14:57:28.513152+0000","last_peered":"2024-09-13T14:57:28.513152+0000","last_clean":"2024-09-13T14:57:28.513152+0000","last_became_active":"2024-09-13T14:56:12.489512+0000","last_became_peered":"2024-09-13T14:56:12.489512+0000","last_unstale":"2024-09-13T14:57:28.513152+0000","last_undegraded":"2024-09-13T14:57:28.513152+0000","last_fullsized":"2024-09-13T14:57:28.513152+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:26:10.594549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.9","version":"478'9531","reported_seq":9583,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T14:57:31.741563+0000","last_change":"2024-09-13T14:54:57.844803+0000","last_active":"2024-09-13T14:57:31.741563+0000","last_peered":"2024-09-13T14:57:31.741563+0000","last_clean":"2024-09-13T14:57:31.741563+0000","last_became_active":"2024-09-13T14:54:57.844499+0000","last_became_peered":"2024-09-13T14:54:57.844499+0000","last_unstale":"2024-09-13T14:57:31.741563+0000","last_undegraded":"2024-09-13T14:57:31.741563+0000","last_fullsized":"2024-09-13T14:57:31.741563+0000","mapping_epoch":436,"log_start":"460'2835","ondisk_log_start":"460'2835","created":436,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":6696,"log_dups_size":0,"ondisk_log_size":6696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:46.617173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251854848,"num_objects":3843,"num_object_clones":0,"num_object_copies":11529,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9531,"num_write_kb":427968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"90'149","reported_seq":693,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-09-13T14:57:25.457446+0000","last_change":"2024-09-13T14:55:20.134578+0000","last_active":"2024-09-13T14:57:25.457446+0000","last_peered":"2024-09-13T14:57:25.457446+0000","last_clean":"2024-09-13T14:57:25.457446+0000","last_became_active":"2024-09-13T14:55:20.134120+0000","last_became_peered":"2024-09-13T14:55:20.134120+0000","last_unstale":"2024-09-13T14:57:25.457446+0000","last_undegraded":"2024-09-13T14:57:25.457446+0000","last_fullsized":"2024-09-13T14:57:25.457446+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:29.629342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:29.629342+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:38:13.399024+0000","scrub_duration":0.0099290990000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":781,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T14:57:28.502344+0000","last_change":"2024-09-13T14:46:04.782955+0000","last_active":"2024-09-13T14:57:28.502344+0000","last_peered":"2024-09-13T14:57:28.502344+0000","last_clean":"2024-09-13T14:57:28.502344+0000","last_became_active":"2024-09-13T14:46:04.782250+0000","last_became_peered":"2024-09-13T14:46:04.782250+0000","last_unstale":"2024-09-13T14:57:28.502344+0000","last_undegraded":"2024-09-13T14:57:28.502344+0000","last_fullsized":"2024-09-13T14:57:28.502344+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:24.596922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":298,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T14:57:28.508342+0000","last_change":"2024-09-13T14:47:47.866684+0000","last_active":"2024-09-13T14:57:28.508342+0000","last_peered":"2024-09-13T14:57:28.508342+0000","last_clean":"2024-09-13T14:57:28.508342+0000","last_became_active":"2024-09-13T14:47:47.866400+0000","last_became_peered":"2024-09-13T14:47:47.866400+0000","last_unstale":"2024-09-13T14:57:28.508342+0000","last_undegraded":"2024-09-13T14:57:28.508342+0000","last_fullsized":"2024-09-13T14:57:28.508342+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:08:34.289658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":298,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T14:57:28.507465+0000","last_change":"2024-09-13T14:47:47.870218+0000","last_active":"2024-09-13T14:57:28.507465+0000","last_peered":"2024-09-13T14:57:28.507465+0000","last_clean":"2024-09-13T14:57:28.507465+0000","last_became_active":"2024-09-13T14:47:47.869907+0000","last_became_peered":"2024-09-13T14:47:47.869907+0000","last_unstale":"2024-09-13T14:57:28.507465+0000","last_undegraded":"2024-09-13T14:57:28.507465+0000","last_fullsized":"2024-09-13T14:57:28.507465+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:41:49.141441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"478'9702","reported_seq":9788,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T14:57:31.766753+0000","last_change":"2024-09-13T14:56:12.497755+0000","last_active":"2024-09-13T14:57:31.766753+0000","last_peered":"2024-09-13T14:57:31.766753+0000","last_clean":"2024-09-13T14:57:31.766753+0000","last_became_active":"2024-09-13T14:56:12.489758+0000","last_became_peered":"2024-09-13T14:56:12.489758+0000","last_unstale":"2024-09-13T14:57:31.766753+0000","last_undegraded":"2024-09-13T14:57:31.766753+0000","last_fullsized":"2024-09-13T14:57:31.766753+0000","mapping_epoch":462,"log_start":"460'2969","ondisk_log_start":"460'2969","created":436,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":6733,"log_dups_size":0,"ondisk_log_size":6733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:22:53.808045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":257032192,"num_objects":3922,"num_object_clones":0,"num_object_copies":11766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9702,"num_write_kb":435968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"90'149","reported_seq":580,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T14:57:27.501627+0000","last_change":"2024-09-13T14:47:32.275753+0000","last_active":"2024-09-13T14:57:27.501627+0000","last_peered":"2024-09-13T14:57:27.501627+0000","last_clean":"2024-09-13T14:57:27.501627+0000","last_became_active":"2024-09-13T14:47:29.401066+0000","last_became_peered":"2024-09-13T14:47:29.401066+0000","last_unstale":"2024-09-13T14:57:27.501627+0000","last_undegraded":"2024-09-13T14:57:27.501627+0000","last_fullsized":"2024-09-13T14:57:27.501627+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:32.275562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:32.275562+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:43:41.056775+0000","scrub_duration":0.0054674260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":1235,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T14:57:27.502029+0000","last_change":"2024-09-13T14:47:50.892842+0000","last_active":"2024-09-13T14:57:27.502029+0000","last_peered":"2024-09-13T14:57:27.502029+0000","last_clean":"2024-09-13T14:57:27.502029+0000","last_became_active":"2024-09-13T14:47:50.892237+0000","last_became_peered":"2024-09-13T14:47:50.892237+0000","last_unstale":"2024-09-13T14:57:27.502029+0000","last_undegraded":"2024-09-13T14:57:27.502029+0000","last_fullsized":"2024-09-13T14:57:27.502029+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:20:05.796762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":8388646,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":284,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T14:57:28.508592+0000","last_change":"2024-09-13T14:56:12.491483+0000","last_active":"2024-09-13T14:57:28.508592+0000","last_peered":"2024-09-13T14:57:28.508592+0000","last_clean":"2024-09-13T14:57:28.508592+0000","last_became_active":"2024-09-13T14:56:12.491191+0000","last_became_peered":"2024-09-13T14:56:12.491191+0000","last_unstale":"2024-09-13T14:57:28.508592+0000","last_undegraded":"2024-09-13T14:57:28.508592+0000","last_fullsized":"2024-09-13T14:57:28.508592+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:54:11.122787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":298,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T14:57:28.507447+0000","last_change":"2024-09-13T14:47:47.877272+0000","last_active":"2024-09-13T14:57:28.507447+0000","last_peered":"2024-09-13T14:57:28.507447+0000","last_clean":"2024-09-13T14:57:28.507447+0000","last_became_active":"2024-09-13T14:47:47.877003+0000","last_became_peered":"2024-09-13T14:47:47.877003+0000","last_unstale":"2024-09-13T14:57:28.507447+0000","last_undegraded":"2024-09-13T14:57:28.507447+0000","last_fullsized":"2024-09-13T14:57:28.507447+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:47:23.172452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.c","version":"476'9203","reported_seq":10536,"reported_epoch":476,"state":"active+clean","last_fresh":"2024-09-13T14:57:26.781265+0000","last_change":"2024-09-13T14:55:28.486018+0000","last_active":"2024-09-13T14:57:26.781265+0000","last_peered":"2024-09-13T14:57:26.781265+0000","last_clean":"2024-09-13T14:57:26.781265+0000","last_became_active":"2024-09-13T14:55:28.483879+0000","last_became_peered":"2024-09-13T14:55:28.483879+0000","last_unstale":"2024-09-13T14:57:26.781265+0000","last_undegraded":"2024-09-13T14:57:26.781265+0000","last_fullsized":"2024-09-13T14:57:26.781265+0000","mapping_epoch":451,"log_start":"460'3530","ondisk_log_start":"460'3530","created":436,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":5673,"log_dups_size":0,"ondisk_log_size":5673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:45:38.042233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266797056,"num_objects":4071,"num_object_clones":0,"num_object_copies":12213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9203,"num_write_kb":424768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":632,"num_bytes_recovered":41418752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"90'149","reported_seq":746,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T14:57:27.511014+0000","last_change":"2024-09-13T14:57:27.511014+0000","last_active":"2024-09-13T14:57:27.511014+0000","last_peered":"2024-09-13T14:57:27.511014+0000","last_clean":"2024-09-13T14:57:27.511014+0000","last_became_active":"2024-09-13T14:57:17.385457+0000","last_became_peered":"2024-09-13T14:57:17.385457+0000","last_unstale":"2024-09-13T14:57:27.511014+0000","last_undegraded":"2024-09-13T14:57:27.511014+0000","last_fullsized":"2024-09-13T14:57:27.511014+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:33.066954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:33.066954+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T14:57:31.892945+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":304,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T14:57:27.502338+0000","last_change":"2024-09-13T14:46:01.763906+0000","last_active":"2024-09-13T14:57:27.502338+0000","last_peered":"2024-09-13T14:57:27.502338+0000","last_clean":"2024-09-13T14:57:27.502338+0000","last_became_active":"2024-09-13T14:46:01.763542+0000","last_became_peered":"2024-09-13T14:46:01.763542+0000","last_unstale":"2024-09-13T14:57:27.502338+0000","last_undegraded":"2024-09-13T14:57:27.502338+0000","last_fullsized":"2024-09-13T14:57:27.502338+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:09:36.652767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":1586,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T14:57:28.507528+0000","last_change":"2024-09-13T14:47:50.883365+0000","last_active":"2024-09-13T14:57:28.507528+0000","last_peered":"2024-09-13T14:57:28.507528+0000","last_clean":"2024-09-13T14:57:28.507528+0000","last_became_active":"2024-09-13T14:47:50.882609+0000","last_became_peered":"2024-09-13T14:47:50.882609+0000","last_unstale":"2024-09-13T14:57:28.507528+0000","last_undegraded":"2024-09-13T14:57:28.507528+0000","last_fullsized":"2024-09-13T14:57:28.507528+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:12:17.961575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":305,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T14:57:28.513396+0000","last_change":"2024-09-13T14:55:17.937000+0000","last_active":"2024-09-13T14:57:28.513396+0000","last_peered":"2024-09-13T14:57:28.513396+0000","last_clean":"2024-09-13T14:57:28.513396+0000","last_became_active":"2024-09-13T14:55:17.936557+0000","last_became_peered":"2024-09-13T14:55:17.936557+0000","last_unstale":"2024-09-13T14:57:28.513396+0000","last_undegraded":"2024-09-13T14:57:28.513396+0000","last_fullsized":"2024-09-13T14:57:28.513396+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:55:25.879096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"477'9063","reported_seq":10312,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T14:57:28.488108+0000","last_change":"2024-09-13T14:56:12.498406+0000","last_active":"2024-09-13T14:57:28.488108+0000","last_peered":"2024-09-13T14:57:28.488108+0000","last_clean":"2024-09-13T14:57:28.488108+0000","last_became_active":"2024-09-13T14:56:12.490622+0000","last_became_peered":"2024-09-13T14:56:12.490622+0000","last_unstale":"2024-09-13T14:57:28.488108+0000","last_undegraded":"2024-09-13T14:57:28.488108+0000","last_fullsized":"2024-09-13T14:57:28.488108+0000","mapping_epoch":462,"log_start":"460'3879","ondisk_log_start":"460'3879","created":436,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":5184,"log_dups_size":0,"ondisk_log_size":5184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:07:04.577772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265158680,"num_objects":4047,"num_object_clones":0,"num_object_copies":12141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4047,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9063,"num_write_kb":419457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":572,"num_bytes_recovered":37486592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"90'149","reported_seq":439,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T14:57:27.507381+0000","last_change":"2024-09-13T14:57:26.480810+0000","last_active":"2024-09-13T14:57:27.507381+0000","last_peered":"2024-09-13T14:57:27.507381+0000","last_clean":"2024-09-13T14:57:27.507381+0000","last_became_active":"2024-09-13T14:57:17.390576+0000","last_became_peered":"2024-09-13T14:57:17.390576+0000","last_unstale":"2024-09-13T14:57:27.507381+0000","last_undegraded":"2024-09-13T14:57:27.507381+0000","last_fullsized":"2024-09-13T14:57:27.507381+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:56:12.709114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:56:12.709114+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T14:57:30.891869+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":298,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T14:57:28.507571+0000","last_change":"2024-09-13T14:47:47.876562+0000","last_active":"2024-09-13T14:57:28.507571+0000","last_peered":"2024-09-13T14:57:28.507571+0000","last_clean":"2024-09-13T14:57:28.507571+0000","last_became_active":"2024-09-13T14:47:47.876297+0000","last_became_peered":"2024-09-13T14:47:47.876297+0000","last_unstale":"2024-09-13T14:57:28.507571+0000","last_undegraded":"2024-09-13T14:57:28.507571+0000","last_fullsized":"2024-09-13T14:57:28.507571+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:03:27.761394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":1041,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-09-13T14:57:25.459330+0000","last_change":"2024-09-13T14:55:20.147689+0000","last_active":"2024-09-13T14:57:25.459330+0000","last_peered":"2024-09-13T14:57:25.459330+0000","last_clean":"2024-09-13T14:57:25.459330+0000","last_became_active":"2024-09-13T14:55:20.147241+0000","last_became_peered":"2024-09-13T14:55:20.147241+0000","last_unstale":"2024-09-13T14:57:25.459330+0000","last_undegraded":"2024-09-13T14:57:25.459330+0000","last_fullsized":"2024-09-13T14:57:25.459330+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:43:36.224330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":334,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T14:57:27.506146+0000","last_change":"2024-09-13T14:56:12.491490+0000","last_active":"2024-09-13T14:57:27.506146+0000","last_peered":"2024-09-13T14:57:27.506146+0000","last_clean":"2024-09-13T14:57:27.506146+0000","last_became_active":"2024-09-13T14:56:12.490650+0000","last_became_peered":"2024-09-13T14:56:12.490650+0000","last_unstale":"2024-09-13T14:57:27.506146+0000","last_undegraded":"2024-09-13T14:57:27.506146+0000","last_fullsized":"2024-09-13T14:57:27.506146+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:55:33.313465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"478'9553","reported_seq":10972,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T14:57:31.622223+0000","last_change":"2024-09-13T14:55:23.955183+0000","last_active":"2024-09-13T14:57:31.622223+0000","last_peered":"2024-09-13T14:57:31.622223+0000","last_clean":"2024-09-13T14:57:31.622223+0000","last_became_active":"2024-09-13T14:55:23.952692+0000","last_became_peered":"2024-09-13T14:55:23.952692+0000","last_unstale":"2024-09-13T14:57:31.622223+0000","last_undegraded":"2024-09-13T14:57:31.622223+0000","last_fullsized":"2024-09-13T14:57:31.622223+0000","mapping_epoch":447,"log_start":"460'2668","ondisk_log_start":"460'2668","created":436,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":6885,"log_dups_size":0,"ondisk_log_size":6885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:48:52.915452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244121600,"num_objects":3725,"num_object_clones":0,"num_object_copies":11175,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9553,"num_write_kb":424896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":659,"num_bytes_recovered":43188224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"90'149","reported_seq":498,"reported_epoch":478,"state":"active+clean+scrubbing","last_fresh":"2024-09-13T14:57:31.099864+0000","last_change":"2024-09-13T14:57:31.099695+0000","last_active":"2024-09-13T14:57:31.099864+0000","last_peered":"2024-09-13T14:57:31.099864+0000","last_clean":"2024-09-13T14:57:31.099864+0000","last_became_active":"2024-09-13T14:57:17.385011+0000","last_became_peered":"2024-09-13T14:57:17.385011+0000","last_unstale":"2024-09-13T14:57:31.099864+0000","last_undegraded":"2024-09-13T14:57:31.099864+0000","last_fullsized":"2024-09-13T14:57:31.099864+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:55:32.716911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:55:32.716911+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":887,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T14:57:27.511333+0000","last_change":"2024-09-13T14:38:53.938227+0000","last_active":"2024-09-13T14:57:27.511333+0000","last_peered":"2024-09-13T14:57:27.511333+0000","last_clean":"2024-09-13T14:57:27.511333+0000","last_became_active":"2024-09-13T14:38:53.902867+0000","last_became_peered":"2024-09-13T14:38:53.902867+0000","last_unstale":"2024-09-13T14:57:27.511333+0000","last_undegraded":"2024-09-13T14:57:27.511333+0000","last_fullsized":"2024-09-13T14:57:27.511333+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:16.831756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":802,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T14:57:28.513572+0000","last_change":"2024-09-13T14:55:29.510841+0000","last_active":"2024-09-13T14:57:28.513572+0000","last_peered":"2024-09-13T14:57:28.513572+0000","last_clean":"2024-09-13T14:57:28.513572+0000","last_became_active":"2024-09-13T14:55:29.510252+0000","last_became_peered":"2024-09-13T14:55:29.510252+0000","last_unstale":"2024-09-13T14:57:28.513572+0000","last_undegraded":"2024-09-13T14:57:28.513572+0000","last_fullsized":"2024-09-13T14:57:28.513572+0000","mapping_epoch":452,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":453,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:14:27.135796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":869,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T14:57:27.505406+0000","last_change":"2024-09-13T14:38:53.900107+0000","last_active":"2024-09-13T14:57:27.505406+0000","last_peered":"2024-09-13T14:57:27.505406+0000","last_clean":"2024-09-13T14:57:27.505406+0000","last_became_active":"2024-09-13T14:38:53.899556+0000","last_became_peered":"2024-09-13T14:38:53.899556+0000","last_unstale":"2024-09-13T14:57:27.505406+0000","last_undegraded":"2024-09-13T14:57:27.505406+0000","last_fullsized":"2024-09-13T14:57:27.505406+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:15:47.918163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"477'9160","reported_seq":9255,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T14:57:28.463442+0000","last_change":"2024-09-13T14:54:58.382957+0000","last_active":"2024-09-13T14:57:28.463442+0000","last_peered":"2024-09-13T14:57:28.463442+0000","last_clean":"2024-09-13T14:57:28.463442+0000","last_became_active":"2024-09-13T14:54:58.382609+0000","last_became_peered":"2024-09-13T14:54:58.382609+0000","last_unstale":"2024-09-13T14:57:28.463442+0000","last_undegraded":"2024-09-13T14:57:28.463442+0000","last_fullsized":"2024-09-13T14:57:28.463442+0000","mapping_epoch":436,"log_start":"461'4064","ondisk_log_start":"461'4064","created":436,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":5096,"log_dups_size":0,"ondisk_log_size":5096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:55:46.466387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265551872,"num_objects":4052,"num_object_clones":0,"num_object_copies":12156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9160,"num_write_kb":422784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"90'149","reported_seq":729,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T14:57:27.509878+0000","last_change":"2024-09-13T14:47:29.999643+0000","last_active":"2024-09-13T14:57:27.509878+0000","last_peered":"2024-09-13T14:57:27.509878+0000","last_clean":"2024-09-13T14:57:27.509878+0000","last_became_active":"2024-09-13T14:47:29.410951+0000","last_became_peered":"2024-09-13T14:47:29.410951+0000","last_unstale":"2024-09-13T14:57:27.509878+0000","last_undegraded":"2024-09-13T14:57:27.509878+0000","last_fullsized":"2024-09-13T14:57:27.509878+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:29.999519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:29.999519+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:45:08.824756+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":255,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T14:57:28.513251+0000","last_change":"2024-09-13T14:56:12.493198+0000","last_active":"2024-09-13T14:57:28.513251+0000","last_peered":"2024-09-13T14:57:28.513251+0000","last_clean":"2024-09-13T14:57:28.513251+0000","last_became_active":"2024-09-13T14:56:12.490423+0000","last_became_peered":"2024-09-13T14:56:12.490423+0000","last_unstale":"2024-09-13T14:57:28.513251+0000","last_undegraded":"2024-09-13T14:57:28.513251+0000","last_fullsized":"2024-09-13T14:57:28.513251+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:29.984031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":282,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-09-13T14:57:25.459435+0000","last_change":"2024-09-13T14:46:01.781508+0000","last_active":"2024-09-13T14:57:25.459435+0000","last_peered":"2024-09-13T14:57:25.459435+0000","last_clean":"2024-09-13T14:57:25.459435+0000","last_became_active":"2024-09-13T14:46:01.767096+0000","last_became_peered":"2024-09-13T14:46:01.767096+0000","last_unstale":"2024-09-13T14:57:25.459435+0000","last_undegraded":"2024-09-13T14:57:25.459435+0000","last_fullsized":"2024-09-13T14:57:25.459435+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:20:01.119189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":268,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T14:57:27.501721+0000","last_change":"2024-09-13T14:55:17.932691+0000","last_active":"2024-09-13T14:57:27.501721+0000","last_peered":"2024-09-13T14:57:27.501721+0000","last_clean":"2024-09-13T14:57:27.501721+0000","last_became_active":"2024-09-13T14:55:17.932391+0000","last_became_peered":"2024-09-13T14:55:17.932391+0000","last_unstale":"2024-09-13T14:57:27.501721+0000","last_undegraded":"2024-09-13T14:57:27.501721+0000","last_fullsized":"2024-09-13T14:57:27.501721+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:03:22.734037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.16","version":"90'149","reported_seq":717,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T14:57:28.532530+0000","last_change":"2024-09-13T14:57:28.532442+0000","last_active":"2024-09-13T14:57:28.532530+0000","last_peered":"2024-09-13T14:57:28.532530+0000","last_clean":"2024-09-13T14:57:28.532530+0000","last_became_active":"2024-09-13T14:57:28.528351+0000","last_became_peered":"2024-09-13T14:57:28.528351+0000","last_unstale":"2024-09-13T14:57:28.532530+0000","last_undegraded":"2024-09-13T14:57:28.532530+0000","last_fullsized":"2024-09-13T14:57:28.532530+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":471,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:56:12.709114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:56:12.709114+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":436,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T14:57:28.508868+0000","last_change":"2024-09-13T14:55:17.930074+0000","last_active":"2024-09-13T14:57:28.508868+0000","last_peered":"2024-09-13T14:57:28.508868+0000","last_clean":"2024-09-13T14:57:28.508868+0000","last_became_active":"2024-09-13T14:55:17.928411+0000","last_became_peered":"2024-09-13T14:55:17.928411+0000","last_unstale":"2024-09-13T14:57:28.508868+0000","last_undegraded":"2024-09-13T14:57:28.508868+0000","last_fullsized":"2024-09-13T14:57:28.508868+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:11:35.302734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":112,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-09-13T14:57:25.458455+0000","last_change":"2024-09-13T14:55:17.939375+0000","last_active":"2024-09-13T14:57:25.458455+0000","last_peered":"2024-09-13T14:57:25.458455+0000","last_clean":"2024-09-13T14:57:25.458455+0000","last_became_active":"2024-09-13T14:55:17.926420+0000","last_became_peered":"2024-09-13T14:55:17.926420+0000","last_unstale":"2024-09-13T14:57:25.458455+0000","last_undegraded":"2024-09-13T14:57:25.458455+0000","last_fullsized":"2024-09-13T14:57:25.458455+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:38:13.401866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4756,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.17","version":"90'149","reported_seq":745,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T14:57:28.527388+0000","last_change":"2024-09-13T14:57:28.527121+0000","last_active":"2024-09-13T14:57:28.527388+0000","last_peered":"2024-09-13T14:57:28.527388+0000","last_clean":"2024-09-13T14:57:28.527388+0000","last_became_active":"2024-09-13T14:57:28.526594+0000","last_became_peered":"2024-09-13T14:57:28.526594+0000","last_unstale":"2024-09-13T14:57:28.527388+0000","last_undegraded":"2024-09-13T14:57:28.527388+0000","last_fullsized":"2024-09-13T14:57:28.527388+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":471,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:33.066954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:33.066954+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":298,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T14:57:28.513061+0000","last_change":"2024-09-13T14:47:47.867935+0000","last_active":"2024-09-13T14:57:28.513061+0000","last_peered":"2024-09-13T14:57:28.513061+0000","last_clean":"2024-09-13T14:57:28.513061+0000","last_became_active":"2024-09-13T14:47:47.867616+0000","last_became_peered":"2024-09-13T14:47:47.867616+0000","last_unstale":"2024-09-13T14:57:28.513061+0000","last_undegraded":"2024-09-13T14:57:28.513061+0000","last_fullsized":"2024-09-13T14:57:28.513061+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:52:38.619916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":261,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-09-13T14:57:25.458563+0000","last_change":"2024-09-13T14:55:17.951412+0000","last_active":"2024-09-13T14:57:25.458563+0000","last_peered":"2024-09-13T14:57:25.458563+0000","last_clean":"2024-09-13T14:57:25.458563+0000","last_became_active":"2024-09-13T14:55:17.937147+0000","last_became_peered":"2024-09-13T14:55:17.937147+0000","last_unstale":"2024-09-13T14:57:25.458563+0000","last_undegraded":"2024-09-13T14:57:25.458563+0000","last_fullsized":"2024-09-13T14:57:25.458563+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:43.786806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"90'149","reported_seq":744,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T14:57:28.502117+0000","last_change":"2024-09-13T14:55:20.149252+0000","last_active":"2024-09-13T14:57:28.502117+0000","last_peered":"2024-09-13T14:57:28.502117+0000","last_clean":"2024-09-13T14:57:28.502117+0000","last_became_active":"2024-09-13T14:55:20.148898+0000","last_became_peered":"2024-09-13T14:55:20.148898+0000","last_unstale":"2024-09-13T14:57:28.502117+0000","last_undegraded":"2024-09-13T14:57:28.502117+0000","last_fullsized":"2024-09-13T14:57:28.502117+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:31.013128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:31.013128+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:11.985087+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":309,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T14:57:27.506017+0000","last_change":"2024-09-13T14:47:47.861842+0000","last_active":"2024-09-13T14:57:27.506017+0000","last_peered":"2024-09-13T14:57:27.506017+0000","last_clean":"2024-09-13T14:57:27.506017+0000","last_became_active":"2024-09-13T14:47:47.861492+0000","last_became_peered":"2024-09-13T14:47:47.861492+0000","last_unstale":"2024-09-13T14:57:27.506017+0000","last_undegraded":"2024-09-13T14:57:27.506017+0000","last_fullsized":"2024-09-13T14:57:27.506017+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:17.279533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":809,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T14:57:27.505565+0000","last_change":"2024-09-13T14:38:53.899535+0000","last_active":"2024-09-13T14:57:27.505565+0000","last_peered":"2024-09-13T14:57:27.505565+0000","last_clean":"2024-09-13T14:57:27.505565+0000","last_became_active":"2024-09-13T14:38:53.899183+0000","last_became_peered":"2024-09-13T14:38:53.899183+0000","last_unstale":"2024-09-13T14:57:27.505565+0000","last_undegraded":"2024-09-13T14:57:27.505565+0000","last_fullsized":"2024-09-13T14:57:27.505565+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:33.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"90'149","reported_seq":486,"reported_epoch":476,"state":"active+undersized+remapped","last_fresh":"2024-09-13T14:57:26.514740+0000","last_change":"2024-09-13T14:57:26.514604+0000","last_active":"2024-09-13T14:57:26.514740+0000","last_peered":"2024-09-13T14:57:26.514740+0000","last_clean":"2024-09-13T14:57:23.433905+0000","last_became_active":"2024-09-13T14:57:26.514604+0000","last_became_peered":"2024-09-13T14:57:26.514604+0000","last_unstale":"2024-09-13T14:57:26.514740+0000","last_undegraded":"2024-09-13T14:57:26.514740+0000","last_fullsized":"2024-09-13T14:57:26.484101+0000","mapping_epoch":475,"log_start":"0'0","ondisk_log_start":"0'0","created":471,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:55:32.716911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:55:32.716911+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[2,5],"avail_no_missing":["2","1","5","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":358,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T14:57:27.507051+0000","last_change":"2024-09-13T14:56:12.488981+0000","last_active":"2024-09-13T14:57:27.507051+0000","last_peered":"2024-09-13T14:57:27.507051+0000","last_clean":"2024-09-13T14:57:27.507051+0000","last_became_active":"2024-09-13T14:56:12.488552+0000","last_became_peered":"2024-09-13T14:56:12.488552+0000","last_unstale":"2024-09-13T14:57:27.507051+0000","last_undegraded":"2024-09-13T14:57:27.507051+0000","last_fullsized":"2024-09-13T14:57:27.507051+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:02:23.732498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":845,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T14:57:27.505307+0000","last_change":"2024-09-13T14:47:47.868624+0000","last_active":"2024-09-13T14:57:27.505307+0000","last_peered":"2024-09-13T14:57:27.505307+0000","last_clean":"2024-09-13T14:57:27.505307+0000","last_became_active":"2024-09-13T14:47:47.868031+0000","last_became_peered":"2024-09-13T14:47:47.868031+0000","last_unstale":"2024-09-13T14:57:27.505307+0000","last_undegraded":"2024-09-13T14:57:27.505307+0000","last_fullsized":"2024-09-13T14:57:27.505307+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:05:25.365023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1a","version":"90'149","reported_seq":385,"reported_epoch":476,"state":"active+undersized+remapped","last_fresh":"2024-09-13T14:57:26.507655+0000","last_change":"2024-09-13T14:57:26.507389+0000","last_active":"2024-09-13T14:57:26.507655+0000","last_peered":"2024-09-13T14:57:26.507655+0000","last_clean":"2024-09-13T14:17:27.401592+0000","last_became_active":"2024-09-13T14:57:26.507389+0000","last_became_peered":"2024-09-13T14:57:26.507389+0000","last_unstale":"2024-09-13T14:57:26.507655+0000","last_undegraded":"2024-09-13T14:57:26.507655+0000","last_fullsized":"2024-09-13T14:57:26.476646+0000","mapping_epoch":475,"log_start":"0'0","ondisk_log_start":"0'0","created":471,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:46:52.334301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:46:52.334301+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":17,"num_read_kb":37,"num_write":10,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80505,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[2,5],"avail_no_missing":["2","1","5","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":323,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T14:57:27.507193+0000","last_change":"2024-09-13T14:47:47.859825+0000","last_active":"2024-09-13T14:57:27.507193+0000","last_peered":"2024-09-13T14:57:27.507193+0000","last_clean":"2024-09-13T14:57:27.507193+0000","last_became_active":"2024-09-13T14:47:47.859069+0000","last_became_peered":"2024-09-13T14:47:47.859069+0000","last_unstale":"2024-09-13T14:57:27.507193+0000","last_undegraded":"2024-09-13T14:57:27.507193+0000","last_fullsized":"2024-09-13T14:57:27.507193+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:18:36.452321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":877,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T14:57:27.505494+0000","last_change":"2024-09-13T14:47:47.868945+0000","last_active":"2024-09-13T14:57:27.505494+0000","last_peered":"2024-09-13T14:57:27.505494+0000","last_clean":"2024-09-13T14:57:27.505494+0000","last_became_active":"2024-09-13T14:47:47.868296+0000","last_became_peered":"2024-09-13T14:47:47.868296+0000","last_unstale":"2024-09-13T14:57:27.505494+0000","last_undegraded":"2024-09-13T14:57:27.505494+0000","last_fullsized":"2024-09-13T14:57:27.505494+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:22:24.778693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"90'149","reported_seq":397,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T14:57:28.527976+0000","last_change":"2024-09-13T14:57:28.527976+0000","last_active":"2024-09-13T14:57:28.527976+0000","last_peered":"2024-09-13T14:57:28.527976+0000","last_clean":"2024-09-13T14:57:28.527976+0000","last_became_active":"2024-09-13T14:57:28.527605+0000","last_became_peered":"2024-09-13T14:57:28.527605+0000","last_unstale":"2024-09-13T14:57:28.527976+0000","last_undegraded":"2024-09-13T14:57:28.527976+0000","last_fullsized":"2024-09-13T14:57:28.527976+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":471,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:39.645908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:39.645908+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8945,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":13,"num_write":4,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":52,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-09-13T14:57:25.459141+0000","last_change":"2024-09-13T14:56:12.491111+0000","last_active":"2024-09-13T14:57:25.459141+0000","last_peered":"2024-09-13T14:57:25.459141+0000","last_clean":"2024-09-13T14:57:25.459141+0000","last_became_active":"2024-09-13T14:56:12.490691+0000","last_became_peered":"2024-09-13T14:56:12.490691+0000","last_unstale":"2024-09-13T14:57:25.459141+0000","last_undegraded":"2024-09-13T14:57:25.459141+0000","last_fullsized":"2024-09-13T14:57:25.459141+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:01.439884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":304,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T14:57:28.507830+0000","last_change":"2024-09-13T14:55:17.929446+0000","last_active":"2024-09-13T14:57:28.507830+0000","last_peered":"2024-09-13T14:57:28.507830+0000","last_clean":"2024-09-13T14:57:28.507830+0000","last_became_active":"2024-09-13T14:55:17.928598+0000","last_became_peered":"2024-09-13T14:55:17.928598+0000","last_unstale":"2024-09-13T14:57:28.507830+0000","last_undegraded":"2024-09-13T14:57:28.507830+0000","last_fullsized":"2024-09-13T14:57:28.507830+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:27.446138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.18","version":"90'149","reported_seq":589,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T14:57:28.526180+0000","last_change":"2024-09-13T14:57:28.526180+0000","last_active":"2024-09-13T14:57:28.526180+0000","last_peered":"2024-09-13T14:57:28.526180+0000","last_clean":"2024-09-13T14:57:28.526180+0000","last_became_active":"2024-09-13T14:57:28.525726+0000","last_became_peered":"2024-09-13T14:57:28.525726+0000","last_unstale":"2024-09-13T14:57:28.526180+0000","last_undegraded":"2024-09-13T14:57:28.526180+0000","last_fullsized":"2024-09-13T14:57:28.526180+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":471,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:46:54.393204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:46:54.393204+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015172223,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131344,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":873,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T14:57:27.507046+0000","last_change":"2024-09-13T14:55:17.933267+0000","last_active":"2024-09-13T14:57:27.507046+0000","last_peered":"2024-09-13T14:57:27.507046+0000","last_clean":"2024-09-13T14:57:27.507046+0000","last_became_active":"2024-09-13T14:55:17.932890+0000","last_became_peered":"2024-09-13T14:55:17.932890+0000","last_unstale":"2024-09-13T14:57:27.507046+0000","last_undegraded":"2024-09-13T14:57:27.507046+0000","last_fullsized":"2024-09-13T14:57:27.507046+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:48:59.113533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":489,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T14:57:28.512885+0000","last_change":"2024-09-13T14:55:17.937105+0000","last_active":"2024-09-13T14:57:28.512885+0000","last_peered":"2024-09-13T14:57:28.512885+0000","last_clean":"2024-09-13T14:57:28.512885+0000","last_became_active":"2024-09-13T14:55:17.936621+0000","last_became_peered":"2024-09-13T14:55:17.936621+0000","last_unstale":"2024-09-13T14:57:28.512885+0000","last_undegraded":"2024-09-13T14:57:28.512885+0000","last_fullsized":"2024-09-13T14:57:28.512885+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:37:28.134499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.19","version":"90'149","reported_seq":479,"reported_epoch":477,"state":"peering","last_fresh":"2024-09-13T14:57:27.505201+0000","last_change":"2024-09-13T14:57:27.505201+0000","last_active":"2024-09-13T14:57:27.505201+0000","last_peered":"2024-09-13T14:57:26.498745+0000","last_clean":"2024-09-13T14:46:42.476444+0000","last_became_active":"2024-09-13T14:57:26.494547+0000","last_became_peered":"2024-09-13T14:57:26.494547+0000","last_unstale":"2024-09-13T14:57:27.505201+0000","last_undegraded":"2024-09-13T14:57:27.505201+0000","last_fullsized":"2024-09-13T14:57:27.505201+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":471,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:46:55.355521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:46:55.355521+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":17,"num_read_kb":37,"num_write":10,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80505,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":379,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T14:57:28.502406+0000","last_change":"2024-09-13T14:50:27.510133+0000","last_active":"2024-09-13T14:57:28.502406+0000","last_peered":"2024-09-13T14:57:28.502406+0000","last_clean":"2024-09-13T14:57:28.502406+0000","last_became_active":"2024-09-13T14:50:27.508774+0000","last_became_peered":"2024-09-13T14:50:27.508774+0000","last_unstale":"2024-09-13T14:57:28.502406+0000","last_undegraded":"2024-09-13T14:57:28.502406+0000","last_fullsized":"2024-09-13T14:57:28.502406+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:31:04.531746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":25,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-09-13T14:57:25.458318+0000","last_change":"2024-09-13T14:56:12.489663+0000","last_active":"2024-09-13T14:57:25.458318+0000","last_peered":"2024-09-13T14:57:25.458318+0000","last_clean":"2024-09-13T14:57:25.458318+0000","last_became_active":"2024-09-13T14:56:12.489317+0000","last_became_peered":"2024-09-13T14:56:12.489317+0000","last_unstale":"2024-09-13T14:57:25.458318+0000","last_undegraded":"2024-09-13T14:57:25.458318+0000","last_fullsized":"2024-09-13T14:57:25.458318+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:08:44.565862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1e","version":"90'149","reported_seq":387,"reported_epoch":477,"state":"peering","last_fresh":"2024-09-13T14:57:27.506556+0000","last_change":"2024-09-13T14:57:27.506556+0000","last_active":"2024-09-13T14:57:27.506556+0000","last_peered":"2024-09-13T14:57:26.486770+0000","last_clean":"2024-09-13T14:17:27.401592+0000","last_became_active":"2024-09-13T14:57:26.486505+0000","last_became_peered":"2024-09-13T14:57:26.486505+0000","last_unstale":"2024-09-13T14:57:27.506556+0000","last_undegraded":"2024-09-13T14:57:27.506556+0000","last_fullsized":"2024-09-13T14:57:27.506556+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":471,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:30.526999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:30.526999+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":19,"num_write":5,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":40253,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":431,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T14:57:27.506298+0000","last_change":"2024-09-13T14:55:17.932405+0000","last_active":"2024-09-13T14:57:27.506298+0000","last_peered":"2024-09-13T14:57:27.506298+0000","last_clean":"2024-09-13T14:57:27.506298+0000","last_became_active":"2024-09-13T14:55:17.932058+0000","last_became_peered":"2024-09-13T14:55:17.932058+0000","last_unstale":"2024-09-13T14:57:27.506298+0000","last_undegraded":"2024-09-13T14:57:27.506298+0000","last_fullsized":"2024-09-13T14:57:27.506298+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:05:01.826514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":323,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T14:57:27.505625+0000","last_change":"2024-09-13T14:55:17.946147+0000","last_active":"2024-09-13T14:57:27.505625+0000","last_peered":"2024-09-13T14:57:27.505625+0000","last_clean":"2024-09-13T14:57:27.505625+0000","last_became_active":"2024-09-13T14:55:17.931009+0000","last_became_peered":"2024-09-13T14:55:17.931009+0000","last_unstale":"2024-09-13T14:57:27.505625+0000","last_undegraded":"2024-09-13T14:57:27.505625+0000","last_fullsized":"2024-09-13T14:57:27.505625+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:08:26.717928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":328,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T14:57:27.508070+0000","last_change":"2024-09-13T14:55:17.936194+0000","last_active":"2024-09-13T14:57:27.508070+0000","last_peered":"2024-09-13T14:57:27.508070+0000","last_clean":"2024-09-13T14:57:27.508070+0000","last_became_active":"2024-09-13T14:55:17.935832+0000","last_became_peered":"2024-09-13T14:55:17.935832+0000","last_unstale":"2024-09-13T14:57:27.508070+0000","last_undegraded":"2024-09-13T14:57:27.508070+0000","last_fullsized":"2024-09-13T14:57:27.508070+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:40:52.745439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":310,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T14:57:27.502473+0000","last_change":"2024-09-13T14:55:17.941129+0000","last_active":"2024-09-13T14:57:27.502473+0000","last_peered":"2024-09-13T14:57:27.502473+0000","last_clean":"2024-09-13T14:57:27.502473+0000","last_became_active":"2024-09-13T14:55:17.930922+0000","last_became_peered":"2024-09-13T14:55:17.930922+0000","last_unstale":"2024-09-13T14:57:27.502473+0000","last_undegraded":"2024-09-13T14:57:27.502473+0000","last_fullsized":"2024-09-13T14:57:27.502473+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:06:29.610560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1c","version":"90'149","reported_seq":432,"reported_epoch":475,"state":"remapped+peering","last_fresh":"2024-09-13T14:57:25.475072+0000","last_change":"2024-09-13T14:47:35.429446+0000","last_active":"2024-09-13T14:47:34.418893+0000","last_peered":"2024-09-13T14:47:32.679161+0000","last_clean":"2024-09-13T14:47:32.679161+0000","last_became_active":"2024-09-13T14:47:29.414467+0000","last_became_peered":"2024-09-13T14:47:29.414467+0000","last_unstale":"2024-09-13T14:57:25.475072+0000","last_undegraded":"2024-09-13T14:57:25.475072+0000","last_fullsized":"2024-09-13T14:57:25.475072+0000","mapping_epoch":475,"log_start":"0'0","ondisk_log_start":"0'0","created":471,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:32.060817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:32.060817+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163840,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":891,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T14:57:27.505858+0000","last_change":"2024-09-13T14:46:01.766580+0000","last_active":"2024-09-13T14:57:27.505858+0000","last_peered":"2024-09-13T14:57:27.505858+0000","last_clean":"2024-09-13T14:57:27.505858+0000","last_became_active":"2024-09-13T14:46:01.766257+0000","last_became_peered":"2024-09-13T14:46:01.766257+0000","last_unstale":"2024-09-13T14:57:27.505858+0000","last_undegraded":"2024-09-13T14:57:27.505858+0000","last_fullsized":"2024-09-13T14:57:27.505858+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:01.126109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":856,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T14:57:27.507611+0000","last_change":"2024-09-13T14:46:01.769647+0000","last_active":"2024-09-13T14:57:27.507611+0000","last_peered":"2024-09-13T14:57:27.507611+0000","last_clean":"2024-09-13T14:57:27.507611+0000","last_became_active":"2024-09-13T14:46:01.769228+0000","last_became_peered":"2024-09-13T14:46:01.769228+0000","last_unstale":"2024-09-13T14:57:27.507611+0000","last_undegraded":"2024-09-13T14:57:27.507611+0000","last_fullsized":"2024-09-13T14:57:27.507611+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:58.696024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1d","version":"90'149","reported_seq":430,"reported_epoch":478,"state":"active+clean+scrubbing","last_fresh":"2024-09-13T14:57:28.657055+0000","last_change":"2024-09-13T14:57:28.656865+0000","last_active":"2024-09-13T14:57:28.657055+0000","last_peered":"2024-09-13T14:57:28.657055+0000","last_clean":"2024-09-13T14:57:28.657055+0000","last_became_active":"2024-09-13T14:57:28.529976+0000","last_became_peered":"2024-09-13T14:57:28.529976+0000","last_unstale":"2024-09-13T14:57:28.657055+0000","last_undegraded":"2024-09-13T14:57:28.657055+0000","last_fullsized":"2024-09-13T14:57:28.657055+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":471,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:40.786610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:40.786610+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":0.013725185000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":415,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-13T14:57:28.508268+0000","last_change":"2024-09-13T14:55:17.929205+0000","last_active":"2024-09-13T14:57:28.508268+0000","last_peered":"2024-09-13T14:57:28.508268+0000","last_clean":"2024-09-13T14:57:28.508268+0000","last_became_active":"2024-09-13T14:55:17.928883+0000","last_became_peered":"2024-09-13T14:55:17.928883+0000","last_unstale":"2024-09-13T14:57:28.508268+0000","last_undegraded":"2024-09-13T14:57:28.508268+0000","last_fullsized":"2024-09-13T14:57:28.508268+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:27:31.570502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":381,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-13T14:57:27.506942+0000","last_change":"2024-09-13T14:56:12.492223+0000","last_active":"2024-09-13T14:57:27.506942+0000","last_peered":"2024-09-13T14:57:27.506942+0000","last_clean":"2024-09-13T14:57:27.506942+0000","last_became_active":"2024-09-13T14:56:12.490578+0000","last_became_peered":"2024-09-13T14:56:12.490578+0000","last_unstale":"2024-09-13T14:57:27.506942+0000","last_undegraded":"2024-09-13T14:57:27.506942+0000","last_fullsized":"2024-09-13T14:57:27.506942+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:11:13.623995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":4119789592,"num_objects":62864,"num_object_clones":0,"num_object_copies":188592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62864,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148816,"num_write_kb":6773697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4957,"num_bytes_recovered":324861952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12265009152,"data_stored":12264996936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95267,"ondisk_log_size":95267,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":6474,"num_keys_recovered":30,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":21564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25132,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":171,"num_bytes_recovered":8388646,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":666401,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":49,"num_read_kb":106,"num_write":29,"num_write_kb":312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":654626,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2048000,"data_stored":2033792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4619,"ondisk_log_size":4619,"up":93,"acting":90,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234563,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2266444,"kb_used_data":1332932,"kb_used_omap":1,"kb_used_meta":933502,"kb_avail":91478708,"statfs":{"total":95995035648,"available":93674196992,"internally_reserved":0,"allocated":1364922368,"data_stored":1362493553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":955906507},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":239,"seq":1026497183982,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2388212,"kb_used_data":1730284,"kb_used_omap":8,"kb_used_meta":657911,"kb_avail":91356940,"statfs":{"total":95995035648,"available":93549506560,"internally_reserved":0,"allocated":1771810816,"data_stored":1769366533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8453,"internal_metadata":673701627},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757829,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2716776,"kb_used_data":1497120,"kb_used_omap":1,"kb_used_meta":1219646,"kb_avail":91028376,"statfs":{"total":95995035648,"available":93213057024,"internally_reserved":0,"allocated":1533050880,"data_stored":1530615139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1248917958},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":128849019473,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435655,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3231208,"kb_used_data":2083924,"kb_used_omap":5,"kb_used_meta":1147194,"kb_avail":90513944,"statfs":{"total":95995035648,"available":92686278656,"internally_reserved":0,"allocated":2133938176,"data_stored":2131506180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":1174726782},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":104,"seq":446676599184,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1967220,"kb_used_data":1480172,"kb_used_omap":7,"kb_used_meta":487032,"kb_avail":91777932,"statfs":{"total":95995035648,"available":93980602368,"internally_reserved":0,"allocated":1515696128,"data_stored":1513253143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8010,"internal_metadata":498720950},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":99,"seq":425201762710,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2005288,"kb_used_data":1544600,"kb_used_omap":7,"kb_used_meta":460664,"kb_avail":91739864,"statfs":{"total":95995035648,"available":93941620736,"internally_reserved":0,"allocated":1581670400,"data_stored":1579241747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7759,"internal_metadata":471720369},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926234,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3623980,"kb_used_data":2461476,"kb_used_omap":7,"kb_used_meta":1162488,"kb_avail":90121172,"statfs":{"total":95995035648,"available":92284080128,"internally_reserved":0,"allocated":2520551424,"data_stored":2518111433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":1190388008},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6860,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2511736832,"data_stored":2511732760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1546911744,"data_stored":1546911744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1495269376,"data_stored":1495269376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2110263296,"data_stored":2110259224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":524288,"data_stored":524288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1498611712,"data_stored":1498611712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1753878528,"data_stored":1753874456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1347813376,"data_stored":1347813376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:57:34.452 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-13T14:57:34.453 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pg_num 41 2024-09-13T14:57:34.465 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:34.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:34.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:34.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:34 smithi195 ceph-mon[104262]: from='client.46808 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:57:34.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:34 smithi195 ceph-mon[104262]: 1.19 scrub starts 2024-09-13T14:57:34.567 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:34.590 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:34.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:34.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:34.692 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:34.719 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:34.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:34.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:34.821 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:34.849 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:34.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:34.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:34.951 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:34.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:34.976 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:34.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:35.077 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:35.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:35.114 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:35.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:35.215 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:35.225 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:35 smithi102 ceph-mon[110183]: 1.1d scrub starts 2024-09-13T14:57:35.225 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:35 smithi102 ceph-mon[110183]: pgmap v1595: 119 pgs: 7 active+clean+scrubbing, 112 active+clean; 3.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 14 KiB/s rd, 42 KiB/s wr, 1.21k op/s; 65 KiB/s, 0 objects/s recovering 2024-09-13T14:57:35.225 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:35 smithi102 ceph-mon[110183]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-13T14:57:35.225 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:35 smithi102 ceph-mon[110183]: 1.1e scrub starts 2024-09-13T14:57:35.226 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:35 smithi102 ceph-mon[115269]: 1.1d scrub starts 2024-09-13T14:57:35.226 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:35 smithi102 ceph-mon[115269]: pgmap v1595: 119 pgs: 7 active+clean+scrubbing, 112 active+clean; 3.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 14 KiB/s rd, 42 KiB/s wr, 1.21k op/s; 65 KiB/s, 0 objects/s recovering 2024-09-13T14:57:35.226 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:35 smithi102 ceph-mon[115269]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-13T14:57:35.226 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:35 smithi102 ceph-mon[115269]: 1.1e scrub starts 2024-09-13T14:57:35.240 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:35.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:35.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:35.342 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:35.369 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:35.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:35.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:35.470 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:35.492 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:35.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:35.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:35.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:35 smithi195 ceph-mon[104262]: 1.1d scrub starts 2024-09-13T14:57:35.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:35 smithi195 ceph-mon[104262]: pgmap v1595: 119 pgs: 7 active+clean+scrubbing, 112 active+clean; 3.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 14 KiB/s rd, 42 KiB/s wr, 1.21k op/s; 65 KiB/s, 0 objects/s recovering 2024-09-13T14:57:35.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:35 smithi195 ceph-mon[104262]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-13T14:57:35.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:35 smithi195 ceph-mon[104262]: 1.1e scrub starts 2024-09-13T14:57:35.594 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:35.617 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:35.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:35.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:35.719 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:35.744 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:35.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:35.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:35.846 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:35.869 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:35.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:35.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:35.971 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:35.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:35.993 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:35.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:36.040 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is unset 2024-09-13T14:57:36.094 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:36.121 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:36.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:36.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:36.223 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:36.249 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:36.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:36.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:36.351 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:36.377 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:36.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:36.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:36.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:36 smithi102 ceph-mon[110183]: 1.b scrub starts 2024-09-13T14:57:36.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:36 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1689437486' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:57:36.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:36 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:57:36.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:36 smithi102 ceph-mon[110183]: 1.c scrub starts 2024-09-13T14:57:36.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:36 smithi102 ceph-mon[115269]: 1.b scrub starts 2024-09-13T14:57:36.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:36 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1689437486' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:57:36.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:36 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:57:36.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:36 smithi102 ceph-mon[115269]: 1.c scrub starts 2024-09-13T14:57:36.479 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:36.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:36 smithi195 ceph-mon[104262]: 1.b scrub starts 2024-09-13T14:57:36.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:36 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1689437486' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:57:36.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:36 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:57:36.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:36 smithi195 ceph-mon[104262]: 1.c scrub starts 2024-09-13T14:57:36.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:36.504 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:36.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:36.558 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T14:57:36.605 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:36.629 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:36.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:36.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:36.695 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:57:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:57:36] "GET /metrics HTTP/1.1" 200 36570 "" "Prometheus/2.51.0" 2024-09-13T14:57:36.731 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:36.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:36.757 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:36.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:36.858 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:36.884 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:36.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:36.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:36.985 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:37.025 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:37.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:37.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:37.127 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:37.149 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:37.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:37.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:37.251 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:37.277 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:37.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:37.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:37.379 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:37.405 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:37.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:37.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:37.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:37 smithi102 ceph-mon[110183]: 1.16 scrub starts 2024-09-13T14:57:37.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:37 smithi102 ceph-mon[110183]: pgmap v1596: 119 pgs: 9 active+clean+scrubbing, 110 active+clean; 3.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 6.7 KiB/s rd, 20 KiB/s wr, 850 op/s; 14 KiB/s, 0 objects/s recovering 2024-09-13T14:57:37.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:37 smithi102 ceph-mon[110183]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:57:37.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:37 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:57:37.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:37 smithi102 ceph-mon[110183]: 1.6 scrub starts 2024-09-13T14:57:37.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:37 smithi102 ceph-mon[110183]: osdmap e479: 8 total, 8 up, 7 in 2024-09-13T14:57:37.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:37 smithi102 ceph-mon[115269]: 1.16 scrub starts 2024-09-13T14:57:37.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:37 smithi102 ceph-mon[115269]: pgmap v1596: 119 pgs: 9 active+clean+scrubbing, 110 active+clean; 3.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 6.7 KiB/s rd, 20 KiB/s wr, 850 op/s; 14 KiB/s, 0 objects/s recovering 2024-09-13T14:57:37.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:37 smithi102 ceph-mon[115269]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:57:37.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:37 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:57:37.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:37 smithi102 ceph-mon[115269]: 1.6 scrub starts 2024-09-13T14:57:37.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:37 smithi102 ceph-mon[115269]: osdmap e479: 8 total, 8 up, 7 in 2024-09-13T14:57:37.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:37 smithi195 ceph-mon[104262]: 1.16 scrub starts 2024-09-13T14:57:37.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:37 smithi195 ceph-mon[104262]: pgmap v1596: 119 pgs: 9 active+clean+scrubbing, 110 active+clean; 3.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 6.7 KiB/s rd, 20 KiB/s wr, 850 op/s; 14 KiB/s, 0 objects/s recovering 2024-09-13T14:57:37.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:37 smithi195 ceph-mon[104262]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:57:37.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:37 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:57:37.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:37 smithi195 ceph-mon[104262]: 1.6 scrub starts 2024-09-13T14:57:37.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:37 smithi195 ceph-mon[104262]: osdmap e479: 8 total, 8 up, 7 in 2024-09-13T14:57:37.507 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:37.531 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:37.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:37.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:37.633 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:37.689 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:37.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:37.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:37.791 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:37.832 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:37.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:37.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:37.934 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:37.974 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:37.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:37.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:38.076 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:38.109 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:38.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:38.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:38.211 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:38.236 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:38.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:38.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:38.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:38 smithi195 ceph-mon[104262]: 1.7 scrub starts 2024-09-13T14:57:38.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:38 smithi195 ceph-mon[104262]: 1.7 scrub ok 2024-09-13T14:57:38.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:38 smithi195 ceph-mon[104262]: 1.5 scrub starts 2024-09-13T14:57:38.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:38 smithi195 ceph-mon[104262]: 1.5 scrub ok 2024-09-13T14:57:38.326 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:38 smithi102 ceph-mon[110183]: 1.7 scrub starts 2024-09-13T14:57:38.326 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:38 smithi102 ceph-mon[110183]: 1.7 scrub ok 2024-09-13T14:57:38.326 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:38 smithi102 ceph-mon[110183]: 1.5 scrub starts 2024-09-13T14:57:38.326 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:38 smithi102 ceph-mon[110183]: 1.5 scrub ok 2024-09-13T14:57:38.327 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:38 smithi102 ceph-mon[115269]: 1.7 scrub starts 2024-09-13T14:57:38.327 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:38 smithi102 ceph-mon[115269]: 1.7 scrub ok 2024-09-13T14:57:38.327 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:38 smithi102 ceph-mon[115269]: 1.5 scrub starts 2024-09-13T14:57:38.327 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:38 smithi102 ceph-mon[115269]: 1.5 scrub ok 2024-09-13T14:57:38.338 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:38.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:38.381 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:38.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:38.482 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:38.508 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:38.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:38.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:38.610 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:38.654 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:38.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:38.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:38.756 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:38.780 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:38.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:38.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:38.882 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:38.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:38.923 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:38.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:39.024 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:39.048 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:39.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:39.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:39.150 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:39.175 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:39.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:39.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:39.277 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:39.326 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:39.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:39.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:39.377 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:39 smithi102 ceph-mon[115269]: pgmap v1598: 119 pgs: 8 active+clean+scrubbing, 111 active+clean; 3.5 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 6.0 KiB/s rd, 18 KiB/s wr, 1.05k op/s; 13 KiB/s, 0 objects/s recovering 2024-09-13T14:57:39.377 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:39 smithi102 ceph-mon[115269]: 1.e scrub starts 2024-09-13T14:57:39.377 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:39 smithi102 ceph-mon[115269]: 1.19 scrub starts 2024-09-13T14:57:39.377 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:39 smithi102 ceph-mon[115269]: 1.e scrub ok 2024-09-13T14:57:39.377 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:39 smithi102 ceph-mon[115269]: 1.19 scrub ok 2024-09-13T14:57:39.377 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:39 smithi102 ceph-mon[115269]: 1.a scrub starts 2024-09-13T14:57:39.377 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:39 smithi102 ceph-mon[115269]: 1.a scrub ok 2024-09-13T14:57:39.379 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:39 smithi102 ceph-mon[110183]: pgmap v1598: 119 pgs: 8 active+clean+scrubbing, 111 active+clean; 3.5 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 6.0 KiB/s rd, 18 KiB/s wr, 1.05k op/s; 13 KiB/s, 0 objects/s recovering 2024-09-13T14:57:39.379 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:39 smithi102 ceph-mon[110183]: 1.e scrub starts 2024-09-13T14:57:39.379 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:39 smithi102 ceph-mon[110183]: 1.19 scrub starts 2024-09-13T14:57:39.379 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:39 smithi102 ceph-mon[110183]: 1.e scrub ok 2024-09-13T14:57:39.379 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:39 smithi102 ceph-mon[110183]: 1.19 scrub ok 2024-09-13T14:57:39.379 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:39 smithi102 ceph-mon[110183]: 1.a scrub starts 2024-09-13T14:57:39.379 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:39 smithi102 ceph-mon[110183]: 1.a scrub ok 2024-09-13T14:57:39.429 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:39.453 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:39.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:39.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:39.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:39 smithi195 ceph-mon[104262]: pgmap v1598: 119 pgs: 8 active+clean+scrubbing, 111 active+clean; 3.5 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 6.0 KiB/s rd, 18 KiB/s wr, 1.05k op/s; 13 KiB/s, 0 objects/s recovering 2024-09-13T14:57:39.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:39 smithi195 ceph-mon[104262]: 1.e scrub starts 2024-09-13T14:57:39.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:39 smithi195 ceph-mon[104262]: 1.19 scrub starts 2024-09-13T14:57:39.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:39 smithi195 ceph-mon[104262]: 1.e scrub ok 2024-09-13T14:57:39.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:39 smithi195 ceph-mon[104262]: 1.19 scrub ok 2024-09-13T14:57:39.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:39 smithi195 ceph-mon[104262]: 1.a scrub starts 2024-09-13T14:57:39.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:39 smithi195 ceph-mon[104262]: 1.a scrub ok 2024-09-13T14:57:39.555 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:39.593 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:39.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:39.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:39.695 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:39.719 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:39.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:39.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:39.766 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:39.821 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:39.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:39.865 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:39.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:39.966 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:40.013 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:40.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:40.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:40.115 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:40.163 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:40.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:40.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:40.265 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:40.321 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:40.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:40.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:40.386 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:40 smithi102 ceph-mon[110183]: 1.1d scrub starts 2024-09-13T14:57:40.386 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:40 smithi102 ceph-mon[110183]: 1.1d scrub ok 2024-09-13T14:57:40.386 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:40 smithi102 ceph-mon[110183]: 1.d scrub starts 2024-09-13T14:57:40.386 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:40 smithi102 ceph-mon[110183]: 1.d scrub ok 2024-09-13T14:57:40.386 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:40 smithi102 ceph-mon[110183]: 1.8 scrub starts 2024-09-13T14:57:40.386 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:40 smithi102 ceph-mon[110183]: 1.8 scrub ok 2024-09-13T14:57:40.387 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:40 smithi102 ceph-mon[115269]: 1.1d scrub starts 2024-09-13T14:57:40.387 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:40 smithi102 ceph-mon[115269]: 1.1d scrub ok 2024-09-13T14:57:40.387 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:40 smithi102 ceph-mon[115269]: 1.d scrub starts 2024-09-13T14:57:40.387 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:40 smithi102 ceph-mon[115269]: 1.d scrub ok 2024-09-13T14:57:40.387 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:40 smithi102 ceph-mon[115269]: 1.8 scrub starts 2024-09-13T14:57:40.387 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:40 smithi102 ceph-mon[115269]: 1.8 scrub ok 2024-09-13T14:57:40.423 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:40.437 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:40 smithi195 ceph-mon[104262]: 1.1d scrub starts 2024-09-13T14:57:40.437 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:40 smithi195 ceph-mon[104262]: 1.1d scrub ok 2024-09-13T14:57:40.437 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:40 smithi195 ceph-mon[104262]: 1.d scrub starts 2024-09-13T14:57:40.437 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:40 smithi195 ceph-mon[104262]: 1.d scrub ok 2024-09-13T14:57:40.437 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:40 smithi195 ceph-mon[104262]: 1.8 scrub starts 2024-09-13T14:57:40.437 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:40 smithi195 ceph-mon[104262]: 1.8 scrub ok 2024-09-13T14:57:40.454 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:40.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:40.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:40.556 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:40.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:40.596 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:40.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:40.697 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:40.722 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:40.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:40.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:40.777 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:57:40.812 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:57:40.812 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:57:40.824 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:40.859 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:57:40.859 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:57:40.895 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:57:40.895 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:57:40.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:40.903 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:40.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:40.930 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:57:40.930 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:57:40.973 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:57:40.974 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:57:41.004 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:41.010 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:57:41.010 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:57:41.044 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:57:41.045 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:57:41.060 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:41.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:41.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:41.073 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:57:41.073 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:57:41.162 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:41.191 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:41.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:41.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:41.331 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:41.365 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:41.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:41.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:41.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:41 smithi102 ceph-mon[110183]: 1.b scrub starts 2024-09-13T14:57:41.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:41 smithi102 ceph-mon[110183]: 1.b scrub ok 2024-09-13T14:57:41.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:41 smithi102 ceph-mon[110183]: pgmap v1599: 119 pgs: 119 active+clean; 3.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 983 op/s 2024-09-13T14:57:41.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:41 smithi102 ceph-mon[110183]: 1.1e scrub starts 2024-09-13T14:57:41.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:41 smithi102 ceph-mon[110183]: 1.1e scrub ok 2024-09-13T14:57:41.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:41 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:57:41.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:41 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2944530802' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:57:41.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:41 smithi102 ceph-mon[115269]: 1.b scrub starts 2024-09-13T14:57:41.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:41 smithi102 ceph-mon[115269]: 1.b scrub ok 2024-09-13T14:57:41.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:41 smithi102 ceph-mon[115269]: pgmap v1599: 119 pgs: 119 active+clean; 3.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 983 op/s 2024-09-13T14:57:41.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:41 smithi102 ceph-mon[115269]: 1.1e scrub starts 2024-09-13T14:57:41.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:41 smithi102 ceph-mon[115269]: 1.1e scrub ok 2024-09-13T14:57:41.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:41 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:57:41.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:41 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2944530802' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:57:41.468 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:41.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:41 smithi195 ceph-mon[104262]: 1.b scrub starts 2024-09-13T14:57:41.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:41 smithi195 ceph-mon[104262]: 1.b scrub ok 2024-09-13T14:57:41.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:41 smithi195 ceph-mon[104262]: pgmap v1599: 119 pgs: 119 active+clean; 3.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 983 op/s 2024-09-13T14:57:41.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:41 smithi195 ceph-mon[104262]: 1.1e scrub starts 2024-09-13T14:57:41.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:41 smithi195 ceph-mon[104262]: 1.1e scrub ok 2024-09-13T14:57:41.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:41 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:57:41.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:41 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2944530802' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:57:41.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:41.504 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:41.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:41.605 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:41.641 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:41.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:41.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:41.742 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:41.773 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:41.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:41.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:41.875 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:41.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:41.906 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:41.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:42.007 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:42.033 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:42.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:42.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:42.135 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:42.173 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:42.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:42.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:42.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:42 smithi195 ceph-mon[104262]: 1.16 scrub starts 2024-09-13T14:57:42.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:42 smithi195 ceph-mon[104262]: 1.16 scrub ok 2024-09-13T14:57:42.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:42 smithi195 ceph-mon[104262]: 1.c scrub starts 2024-09-13T14:57:42.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:42 smithi195 ceph-mon[104262]: 1.c scrub ok 2024-09-13T14:57:42.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:42 smithi195 ceph-mon[104262]: 1.18 scrub starts 2024-09-13T14:57:42.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:42 smithi195 ceph-mon[104262]: 1.18 scrub ok 2024-09-13T14:57:42.274 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:42.297 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:42.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:42.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:42.399 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:42.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:42 smithi102 ceph-mon[110183]: 1.16 scrub starts 2024-09-13T14:57:42.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:42 smithi102 ceph-mon[110183]: 1.16 scrub ok 2024-09-13T14:57:42.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:42 smithi102 ceph-mon[110183]: 1.c scrub starts 2024-09-13T14:57:42.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:42 smithi102 ceph-mon[110183]: 1.c scrub ok 2024-09-13T14:57:42.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:42 smithi102 ceph-mon[110183]: 1.18 scrub starts 2024-09-13T14:57:42.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:42 smithi102 ceph-mon[110183]: 1.18 scrub ok 2024-09-13T14:57:42.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:42 smithi102 ceph-mon[115269]: 1.16 scrub starts 2024-09-13T14:57:42.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:42 smithi102 ceph-mon[115269]: 1.16 scrub ok 2024-09-13T14:57:42.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:42 smithi102 ceph-mon[115269]: 1.c scrub starts 2024-09-13T14:57:42.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:42 smithi102 ceph-mon[115269]: 1.c scrub ok 2024-09-13T14:57:42.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:42 smithi102 ceph-mon[115269]: 1.18 scrub starts 2024-09-13T14:57:42.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:42 smithi102 ceph-mon[115269]: 1.18 scrub ok 2024-09-13T14:57:42.432 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:42.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:42.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:42.534 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:42.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:42.574 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:42.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:42.675 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:42.703 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:42.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:42.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:42.805 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:42.831 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:42.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:42.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:42.933 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:42.976 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:42.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:42.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:43.078 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:43.103 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:43.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:43.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:43.206 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:43.246 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:43.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:43.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:43.349 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:43.377 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:43.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:43.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:43.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:43 smithi102 ceph-mon[110183]: pgmap v1600: 119 pgs: 119 active+clean; 3.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 976 op/s 2024-09-13T14:57:43.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:43 smithi102 ceph-mon[110183]: 1.9 scrub starts 2024-09-13T14:57:43.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:43 smithi102 ceph-mon[110183]: 1.9 scrub ok 2024-09-13T14:57:43.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:43 smithi102 ceph-mon[110183]: 1.6 scrub starts 2024-09-13T14:57:43.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:43 smithi102 ceph-mon[110183]: 1.6 scrub ok 2024-09-13T14:57:43.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:43 smithi102 ceph-mon[110183]: 1.1b scrub starts 2024-09-13T14:57:43.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:43 smithi102 ceph-mon[110183]: 1.1b scrub ok 2024-09-13T14:57:43.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:43 smithi102 ceph-mon[115269]: pgmap v1600: 119 pgs: 119 active+clean; 3.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 976 op/s 2024-09-13T14:57:43.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:43 smithi102 ceph-mon[115269]: 1.9 scrub starts 2024-09-13T14:57:43.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:43 smithi102 ceph-mon[115269]: 1.9 scrub ok 2024-09-13T14:57:43.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:43 smithi102 ceph-mon[115269]: 1.6 scrub starts 2024-09-13T14:57:43.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:43 smithi102 ceph-mon[115269]: 1.6 scrub ok 2024-09-13T14:57:43.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:43 smithi102 ceph-mon[115269]: 1.1b scrub starts 2024-09-13T14:57:43.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:43 smithi102 ceph-mon[115269]: 1.1b scrub ok 2024-09-13T14:57:43.479 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:43.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:43 smithi195 ceph-mon[104262]: pgmap v1600: 119 pgs: 119 active+clean; 3.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 976 op/s 2024-09-13T14:57:43.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:43 smithi195 ceph-mon[104262]: 1.9 scrub starts 2024-09-13T14:57:43.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:43 smithi195 ceph-mon[104262]: 1.9 scrub ok 2024-09-13T14:57:43.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:43 smithi195 ceph-mon[104262]: 1.6 scrub starts 2024-09-13T14:57:43.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:43 smithi195 ceph-mon[104262]: 1.6 scrub ok 2024-09-13T14:57:43.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:43 smithi195 ceph-mon[104262]: 1.1b scrub starts 2024-09-13T14:57:43.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:43 smithi195 ceph-mon[104262]: 1.1b scrub ok 2024-09-13T14:57:43.515 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:43.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:43.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:43.617 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:43.641 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:43.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:43.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:43.743 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:43.782 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:43.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:43.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:43.883 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:43.908 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:43.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:43.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:44.010 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:44.050 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:44.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:44.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:44.152 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:44.186 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:44.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:44.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:44.288 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:44.321 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:44.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:44.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:44.423 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:44.468 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:44.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:44.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:44.570 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:44.618 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:44.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:44.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:44.720 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:44.786 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:44.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:44.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:44.888 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:44.930 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:44.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:44.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:45.032 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:45.056 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:45.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:45.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:45.158 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:45.198 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:45.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:45.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:45.300 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:45.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:45.326 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:45.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:45.427 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:45.451 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:45.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:45.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:45.519 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:45 smithi102 ceph-mon[110183]: pgmap v1601: 119 pgs: 119 active+clean; 3.1 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 959 op/s 2024-09-13T14:57:45.519 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:45 smithi102 ceph-mon[110183]: 1.1a scrub starts 2024-09-13T14:57:45.519 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:45 smithi102 ceph-mon[110183]: 1.1a scrub ok 2024-09-13T14:57:45.519 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:45 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:57:45.520 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:45 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:57:45.520 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:45 smithi102 ceph-mon[115269]: pgmap v1601: 119 pgs: 119 active+clean; 3.1 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 959 op/s 2024-09-13T14:57:45.520 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:45 smithi102 ceph-mon[115269]: 1.1a scrub starts 2024-09-13T14:57:45.520 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:45 smithi102 ceph-mon[115269]: 1.1a scrub ok 2024-09-13T14:57:45.520 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:45 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:57:45.520 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:45 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:57:45.553 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:45.577 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:45.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:45.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:45.679 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:45.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:45.707 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:45.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:45.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:45 smithi195 ceph-mon[104262]: pgmap v1601: 119 pgs: 119 active+clean; 3.1 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 959 op/s 2024-09-13T14:57:45.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:45 smithi195 ceph-mon[104262]: 1.1a scrub starts 2024-09-13T14:57:45.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:45 smithi195 ceph-mon[104262]: 1.1a scrub ok 2024-09-13T14:57:45.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:45 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:57:45.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:45 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:57:45.808 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:45.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:45.849 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:45.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:45.950 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:45.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:45.977 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:45.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:46.078 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:46.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:46.103 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:46.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:46.204 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:46.231 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:46.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:46.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:46.333 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:46.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:46.360 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:46.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:46.437 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:46.462 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:46.485 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:46.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:46.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:46.529 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:46.586 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:46.615 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:46.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:46.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:46.657 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:46 smithi102 ceph-mon[110183]: 1.1c scrub starts 2024-09-13T14:57:46.657 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:46 smithi102 ceph-mon[110183]: 1.1c scrub ok 2024-09-13T14:57:46.657 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:57:46.657 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:57:46.657 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:57:46.657 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:57:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:57:46] "GET /metrics HTTP/1.1" 200 36579 "" "Prometheus/2.51.0" 2024-09-13T14:57:46.658 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:46 smithi102 ceph-mon[115269]: 1.1c scrub starts 2024-09-13T14:57:46.658 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:46 smithi102 ceph-mon[115269]: 1.1c scrub ok 2024-09-13T14:57:46.658 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:57:46.658 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:57:46.658 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:57:46.717 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:46.744 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:46.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:46.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:46.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:46 smithi195 ceph-mon[104262]: 1.1c scrub starts 2024-09-13T14:57:46.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:46 smithi195 ceph-mon[104262]: 1.1c scrub ok 2024-09-13T14:57:46.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:57:46.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:57:46.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:57:46.846 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:46.869 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:46.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:46.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:46.971 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:46.995 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:46.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:46.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:47.097 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:47.123 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:47.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:47.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:47.225 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:47.267 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:47.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:47.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:47.296 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:57:47.369 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:47.397 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:47.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:47.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:47.499 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:47.521 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:47.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:47.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:47.623 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:47.661 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:47.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:47.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:47.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:47 smithi102 ceph-mon[110183]: pgmap v1602: 119 pgs: 119 active+clean; 3.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 964 op/s 2024-09-13T14:57:47.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:47 smithi102 ceph-mon[115269]: pgmap v1602: 119 pgs: 119 active+clean; 3.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 964 op/s 2024-09-13T14:57:47.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:47 smithi195 ceph-mon[104262]: pgmap v1602: 119 pgs: 119 active+clean; 3.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 964 op/s 2024-09-13T14:57:47.763 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:47.806 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:47.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:47.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:47.908 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:47.948 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:47.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:47.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:48.050 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:48.086 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:48.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:48.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:48.187 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:48.213 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:48.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:48.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:48.315 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:48.350 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:48.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:48.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:48.452 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:48.477 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:48.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:48.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:48.579 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:48.607 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:48.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:48.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:48.709 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:48.733 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:48.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:48.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:48.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:48 smithi195 ceph-mon[104262]: 1.15 scrub starts 2024-09-13T14:57:48.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:48 smithi195 ceph-mon[104262]: 1.15 scrub ok 2024-09-13T14:57:48.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:48 smithi195 ceph-mon[104262]: 1.17 scrub starts 2024-09-13T14:57:48.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:48 smithi195 ceph-mon[104262]: 1.17 scrub ok 2024-09-13T14:57:48.835 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:48.858 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:48.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:48.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:48.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:48 smithi102 ceph-mon[110183]: 1.15 scrub starts 2024-09-13T14:57:48.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:48 smithi102 ceph-mon[110183]: 1.15 scrub ok 2024-09-13T14:57:48.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:48 smithi102 ceph-mon[110183]: 1.17 scrub starts 2024-09-13T14:57:48.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:48 smithi102 ceph-mon[110183]: 1.17 scrub ok 2024-09-13T14:57:48.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:48 smithi102 ceph-mon[115269]: 1.15 scrub starts 2024-09-13T14:57:48.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:48 smithi102 ceph-mon[115269]: 1.15 scrub ok 2024-09-13T14:57:48.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:48 smithi102 ceph-mon[115269]: 1.17 scrub starts 2024-09-13T14:57:48.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:48 smithi102 ceph-mon[115269]: 1.17 scrub ok 2024-09-13T14:57:48.960 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:48.981 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:48.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:48.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:49.083 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:49.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:49.121 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:49.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:49.222 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:49.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:49.281 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:49.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:49.382 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:49.410 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:49.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:49.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:49.512 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:49.547 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:49.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:49.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:49.649 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:49.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:49 smithi102 ceph-mon[110183]: pgmap v1603: 119 pgs: 119 active+clean; 2.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-13T14:57:49.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:49 smithi102 ceph-mon[115269]: pgmap v1603: 119 pgs: 119 active+clean; 2.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-13T14:57:49.672 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:49.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:49.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:49.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:49 smithi195 ceph-mon[104262]: pgmap v1603: 119 pgs: 119 active+clean; 2.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-13T14:57:49.774 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:49.799 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:49.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:49.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:49.901 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:49.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:49.943 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:49.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:50.044 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:50.096 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:50.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:50.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:50.199 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:50.240 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:50.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:50.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:50.342 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:50.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:50.366 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:50.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:50.467 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:50.490 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:50.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:50.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:50.592 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:50.616 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:50.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:50.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:50.718 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:50.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:50.754 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:50.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:50.856 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:50.883 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:50.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:50.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:50.964 INFO:teuthology.orchestra.run.smithi102.stderr:Error EINVAL: specified pg_num 41 < pg_num_max 32 2024-09-13T14:57:50.986 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:51.023 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:51.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:51.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:51.125 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:51.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:51.151 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:51.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:51.253 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:51.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:51.297 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:51.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:51.398 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:51.422 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:51.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:51.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:51.524 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:51.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:51.547 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:51.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:51.585 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:51 smithi102 ceph-mon[110183]: pgmap v1604: 119 pgs: 119 active+clean; 2.8 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 984 op/s 2024-09-13T14:57:51.585 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:51 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2605794765' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"} : dispatch 2024-09-13T14:57:51.585 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:51 smithi102 ceph-mon[115269]: pgmap v1604: 119 pgs: 119 active+clean; 2.8 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 984 op/s 2024-09-13T14:57:51.586 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:51 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2605794765' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"} : dispatch 2024-09-13T14:57:51.648 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:51.672 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:51.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:51.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:51.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:51 smithi195 ceph-mon[104262]: pgmap v1604: 119 pgs: 119 active+clean; 2.8 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 984 op/s 2024-09-13T14:57:51.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:51 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2605794765' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"} : dispatch 2024-09-13T14:57:51.774 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:51.797 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:51.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:51.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:51.898 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:51.922 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:51.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:51.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:52.024 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:52.065 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:52.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:52.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:52.167 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:52.192 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:52.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:52.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:52.294 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:52.335 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:52.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:52.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:52.408 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T14:57:52.437 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:52.464 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:52.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:52.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:52.567 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:52.610 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:52.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:52.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:52.712 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:52.754 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:52.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:52.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:52.856 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:52.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:52.908 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:52.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:53.009 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:53.057 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:53.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:53.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:53.088 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:57:53.088 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [], 2024-09-13T14:57:53.088 INFO:teuthology.orchestra.run.smithi102.stdout: "complaint_time": 30, 2024-09-13T14:57:53.088 INFO:teuthology.orchestra.run.smithi102.stdout: "num_blocked_ops": 0 2024-09-13T14:57:53.088 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:57:53.159 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:53.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:53.188 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:53.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:53.289 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:53.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:53.331 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:53.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:53.432 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:53.475 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:53.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:53.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:53.577 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:53.624 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:53.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:53.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:53.638 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:53 smithi102 ceph-mon[110183]: pgmap v1605: 119 pgs: 119 active+clean; 2.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 981 op/s 2024-09-13T14:57:53.638 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:53 smithi102 ceph-mon[115269]: pgmap v1605: 119 pgs: 119 active+clean; 2.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 981 op/s 2024-09-13T14:57:53.727 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:53.749 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:53.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:53.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:53.851 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:53.897 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:53.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:53.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:53.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:53 smithi195 ceph-mon[104262]: pgmap v1605: 119 pgs: 119 active+clean; 2.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 981 op/s 2024-09-13T14:57:53.999 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:54.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:54.038 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:54.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:54.139 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:54.164 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:54.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:54.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:54.266 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:54.305 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:54.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:54.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:54.340 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:54.341 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T14:57:54.407 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:54.431 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:54.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:54.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:54.520 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is unset 2024-09-13T14:57:54.533 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:54.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:54.602 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:54.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:54.671 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:54 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1882061248' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:57:54.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:54 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1882061248' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:57:54.703 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:54.748 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:54.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:54.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:54.850 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:54.876 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:54.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:54.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:54.977 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:54.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:54 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1882061248' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:57:55.016 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:55.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:55.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:55.118 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:55.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:55.145 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:55.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:55.246 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:55.279 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:55.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:55.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:55.381 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:55.405 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:55.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:55.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:55.507 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:55.528 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:55.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:55.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:55.629 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:55.655 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:55.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:55.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:55.757 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:55.779 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:55.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:55.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:55.784 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:55 smithi102 ceph-mon[110183]: pgmap v1606: 119 pgs: 119 active+clean; 2.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 952 op/s 2024-09-13T14:57:55.784 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:55 smithi102 ceph-mon[110183]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:57:55.784 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:55 smithi102 ceph-mon[110183]: Cluster is now healthy 2024-09-13T14:57:55.784 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:55 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1882061248' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:57:55.784 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:55 smithi102 ceph-mon[110183]: osdmap e480: 8 total, 8 up, 7 in 2024-09-13T14:57:55.784 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:55 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:57:55.784 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:55 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:57:55.784 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:55 smithi102 ceph-mon[115269]: pgmap v1606: 119 pgs: 119 active+clean; 2.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 952 op/s 2024-09-13T14:57:55.785 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:55 smithi102 ceph-mon[115269]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:57:55.785 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:55 smithi102 ceph-mon[115269]: Cluster is now healthy 2024-09-13T14:57:55.785 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:55 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1882061248' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:57:55.785 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:55 smithi102 ceph-mon[115269]: osdmap e480: 8 total, 8 up, 7 in 2024-09-13T14:57:55.785 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:55 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:57:55.785 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:55 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:57:55.882 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:55.906 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:55.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:55.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:56.001 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:55 smithi195 ceph-mon[104262]: pgmap v1606: 119 pgs: 119 active+clean; 2.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 952 op/s 2024-09-13T14:57:56.001 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:55 smithi195 ceph-mon[104262]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:57:56.001 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:55 smithi195 ceph-mon[104262]: Cluster is now healthy 2024-09-13T14:57:56.001 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:55 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1882061248' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:57:56.001 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:55 smithi195 ceph-mon[104262]: osdmap e480: 8 total, 8 up, 7 in 2024-09-13T14:57:56.001 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:55 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:57:56.001 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:55 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:57:56.008 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:56.032 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:56.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:56.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:56.134 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:56.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:56.156 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:56.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:56.258 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:56.283 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:56.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:56.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:56.385 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:56.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:56.419 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:56.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:56.520 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:56.543 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:56.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:56.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:56.645 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:56.684 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:56.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:56.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:56.786 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:56.821 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:56.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:56.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:56.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:56 smithi102 ceph-mon[110183]: pgmap v1608: 119 pgs: 119 active+clean; 2.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 957 op/s 2024-09-13T14:57:56.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:56 smithi102 ceph-mon[115269]: pgmap v1608: 119 pgs: 119 active+clean; 2.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 957 op/s 2024-09-13T14:57:56.919 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:57:56 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:57:56] "GET /metrics HTTP/1.1" 200 36568 "" "Prometheus/2.51.0" 2024-09-13T14:57:56.923 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:56.967 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:56.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:56.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:56.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:56 smithi195 ceph-mon[104262]: pgmap v1608: 119 pgs: 119 active+clean; 2.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 957 op/s 2024-09-13T14:57:57.069 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:57.111 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:57.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:57.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:57.213 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:57.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:57.238 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:57.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:57.340 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:57.364 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:57.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:57.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:57.409 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 6, 7, 5, 2, 1] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T14:57:57.410 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:57:57.410 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-13T14:57:57.410 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:57:57.465 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:57.532 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:57.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:57.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:57.634 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:57.658 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:57.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:57.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:57.760 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:57.787 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:57.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:57.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:57.889 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:57.916 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:57.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:57.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:58.018 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:58.052 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:58.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:58.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:58.154 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:58.178 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:58.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:58.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:58.195 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:58.279 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:58.325 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:58.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:58.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:58.427 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:58.470 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:58.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:58.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:58.573 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:58.597 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:58.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:58.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:58.699 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:58.724 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:58.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:58.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:58.826 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:58.868 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:58.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:58.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:58.970 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:58.994 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:58.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:58.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:59.048 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:57:58 smithi102 ceph-mon[110183]: pgmap v1609: 119 pgs: 119 active+clean; 2.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 948 op/s 2024-09-13T14:57:59.048 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:57:58 smithi102 ceph-mon[115269]: pgmap v1609: 119 pgs: 119 active+clean; 2.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 948 op/s 2024-09-13T14:57:59.096 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:59.135 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:59.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:59.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:59.237 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:59.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:57:58 smithi195 ceph-mon[104262]: pgmap v1609: 119 pgs: 119 active+clean; 2.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 948 op/s 2024-09-13T14:57:59.270 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:57:59.270 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:57:59.276 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:59.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:59.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:59.314 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:57:59.315 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:57:59.347 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:57:59.347 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:57:59.378 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:59.379 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:57:59.379 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:57:59.413 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:57:59.413 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:57:59.419 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:59.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:59.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:59.443 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:57:59.443 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:57:59.472 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:57:59.472 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:57:59.503 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:57:59.507 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:57:59.507 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:57:59.521 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:59.545 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:59.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:59.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:59.646 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:59.671 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:59.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:59.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:59.773 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:59.809 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:57:59.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:59.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:59.911 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:59.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:59.938 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:57:59.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:00.039 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:00.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:00.069 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:00.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:00.170 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:00.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:00.193 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:00.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:00.294 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:00.331 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:00.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:00.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:00.433 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:00.457 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:00.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:00.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:00.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:00 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1471997725' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:00.505 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:00 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1471997725' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:00.506 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:00 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1471997725' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:00.559 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:00.600 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:00.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:00.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:00.702 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:00.729 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:00.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:00.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:00.832 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:00.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:00.875 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:00.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:00.976 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:01.019 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:01.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:01.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:01.121 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:01.146 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:01.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:01.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:01.176 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:01.248 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:01.275 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:01.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:01.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:01.377 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:01.402 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:01.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:01.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:01.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:01 smithi195 ceph-mon[104262]: pgmap v1610: 119 pgs: 119 active+clean; 2.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 942 op/s 2024-09-13T14:58:01.504 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:01.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:01.529 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:01.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:01.630 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:01.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:01 smithi102 ceph-mon[110183]: pgmap v1610: 119 pgs: 119 active+clean; 2.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 942 op/s 2024-09-13T14:58:01.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:01 smithi102 ceph-mon[115269]: pgmap v1610: 119 pgs: 119 active+clean; 2.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 942 op/s 2024-09-13T14:58:01.669 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:01.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:01.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:01.773 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:01.799 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:01.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:01.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:01.901 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:01.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:01.926 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:01.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:02.027 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:02.063 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:02.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:02.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:02.166 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:02.193 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:02.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:02.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:02.295 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:02.328 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:02.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:02.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:02.430 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:02.472 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:02.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:02.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:02.574 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:02.606 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:02.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:02.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:02.708 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:02.757 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:02.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:02.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:02.859 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:02.884 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:02.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:02.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:02.986 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:03.030 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:03.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:03.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:03.134 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:03.173 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:03.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:03.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:03.275 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:03.300 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:03.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:03.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:03.402 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:03.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:03 smithi102 ceph-mon[110183]: pgmap v1611: 119 pgs: 119 active+clean; 2.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 940 op/s 2024-09-13T14:58:03.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:03 smithi102 ceph-mon[115269]: pgmap v1611: 119 pgs: 119 active+clean; 2.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 940 op/s 2024-09-13T14:58:03.441 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:03.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:03.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:03.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:03 smithi195 ceph-mon[104262]: pgmap v1611: 119 pgs: 119 active+clean; 2.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 940 op/s 2024-09-13T14:58:03.543 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:03.577 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:03.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:03.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:03.679 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:03.713 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:03.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:03.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:03.815 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:03.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:03.861 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:03.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:03.867 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:58:03.867 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:58:03.963 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:03.989 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:03.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:03.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:04.090 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:04.115 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:04.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:04.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:04.125 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:58:04.217 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:04.241 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:04.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:04.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:04.343 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:04.386 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:04.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:04.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:04.488 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:04.512 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:04.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:04.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:04.614 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:04.639 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:04.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:04.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:04.666 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T14:58:04.741 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:04.768 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:04.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:04.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:04.870 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:04.895 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:04.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:04.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:04.998 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:05.041 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:05.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:05.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:05.143 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:05.169 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:05.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:05.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:05.170 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:05.271 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:05.296 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:05.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:05.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:05.398 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:05.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:05 smithi102 ceph-mon[110183]: pgmap v1612: 119 pgs: 119 active+clean; 2.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 948 op/s 2024-09-13T14:58:05.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:05 smithi102 ceph-mon[115269]: pgmap v1612: 119 pgs: 119 active+clean; 2.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 948 op/s 2024-09-13T14:58:05.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:05.426 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:05.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:05.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:05 smithi195 ceph-mon[104262]: pgmap v1612: 119 pgs: 119 active+clean; 2.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 948 op/s 2024-09-13T14:58:05.527 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:05.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:05.582 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:05.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:05.683 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:05.716 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:05.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:05.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:05.818 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:05.841 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:05.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:05.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:05.942 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:05.968 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:05.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:05.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:06.070 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:06.101 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:06.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:06.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:06.203 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:06.237 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:06.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:06.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:06.342 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:06.378 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:06.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:06.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:06.480 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:06.521 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:06.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:06.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:06.623 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:06.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:06.650 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:06.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:06.751 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:06.762 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:58:06 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:58:06] "GET /metrics HTTP/1.1" 200 36568 "" "Prometheus/2.51.0" 2024-09-13T14:58:06.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:06.802 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:06.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:06.903 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:06.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:06.941 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:06.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:06.977 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:58:06.979 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:58:07.042 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:07.069 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:07.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:07.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:07.072 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:07.170 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:07.207 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:07.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:07.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:07.308 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:07.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:07.335 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:07.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:07.368 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:07 smithi102 ceph-mon[110183]: pgmap v1613: 119 pgs: 119 active+clean; 1.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 924 op/s 2024-09-13T14:58:07.369 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:07 smithi102 ceph-mon[115269]: pgmap v1613: 119 pgs: 119 active+clean; 1.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 924 op/s 2024-09-13T14:58:07.436 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:07.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:07.460 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:07.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:07.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:07 smithi195 ceph-mon[104262]: pgmap v1613: 119 pgs: 119 active+clean; 1.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 924 op/s 2024-09-13T14:58:07.561 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:07.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:07.597 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:07.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:07.698 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:07.743 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:07.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:07.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:07.845 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:07.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:07.871 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:07.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:07.972 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:08.007 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:08.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:08.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:08.109 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:08.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:08.153 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:08.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:08.254 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:08.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:08.294 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:08.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:08.389 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":1613,"stamp":"2024-09-13T14:58:05.824703+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2048661501,"num_objects":31216,"num_object_clones":0,"num_object_copies":93648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31216,"num_whiteouts":0,"num_read":5999,"num_read_kb":13012,"num_write":181400,"num_write_kb":6786322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5177,"num_bytes_recovered":333257072,"num_keys_recovered":30,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105095,"ondisk_log_size":105095,"up":357,"acting":357,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":357,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":12412000,"kb_used_data":6197800,"kb_used_omap":40,"kb_used_meta":6214103,"kb_avail":643804064,"statfs":{"total":671965249536,"available":659255361536,"internally_reserved":0,"allocated":6346547200,"data_stored":6329419446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41319,"internal_metadata":6363242137},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-684851200,"num_objects":-10450,"num_object_clones":0,"num_object_copies":-31350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10450,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.304710"},"pg_stats":[{"pgid":"1.11","version":"90'149","reported_seq":455,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T14:57:36.047218+0000","last_change":"2024-09-13T14:56:12.490567+0000","last_active":"2024-09-13T14:57:36.047218+0000","last_peered":"2024-09-13T14:57:36.047218+0000","last_clean":"2024-09-13T14:57:36.047218+0000","last_became_active":"2024-09-13T14:56:12.490253+0000","last_became_peered":"2024-09-13T14:56:12.490253+0000","last_unstale":"2024-09-13T14:57:36.047218+0000","last_undegraded":"2024-09-13T14:57:36.047218+0000","last_fullsized":"2024-09-13T14:57:36.047218+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:38.688341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.688341+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:20.151605+0000","scrub_duration":0.0054331650000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":874,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.531876+0000","last_change":"2024-09-13T14:47:47.868783+0000","last_active":"2024-09-13T14:57:54.531876+0000","last_peered":"2024-09-13T14:57:54.531876+0000","last_clean":"2024-09-13T14:57:54.531876+0000","last_became_active":"2024-09-13T14:47:47.868478+0000","last_became_peered":"2024-09-13T14:47:47.868478+0000","last_unstale":"2024-09-13T14:57:54.531876+0000","last_undegraded":"2024-09-13T14:57:54.531876+0000","last_fullsized":"2024-09-13T14:57:54.531876+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:27:01.784142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":518,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T14:57:36.048241+0000","last_change":"2024-09-13T14:55:17.943558+0000","last_active":"2024-09-13T14:57:36.048241+0000","last_peered":"2024-09-13T14:57:36.048241+0000","last_clean":"2024-09-13T14:57:36.048241+0000","last_became_active":"2024-09-13T14:55:17.927162+0000","last_became_peered":"2024-09-13T14:55:17.927162+0000","last_unstale":"2024-09-13T14:57:36.048241+0000","last_undegraded":"2024-09-13T14:57:36.048241+0000","last_fullsized":"2024-09-13T14:57:36.048241+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:37:44.820559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":102,"num_keys_recovered":30,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"90'149","reported_seq":548,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T14:57:36.047116+0000","last_change":"2024-09-13T14:47:41.482552+0000","last_active":"2024-09-13T14:57:36.047116+0000","last_peered":"2024-09-13T14:57:36.047116+0000","last_clean":"2024-09-13T14:57:36.047116+0000","last_became_active":"2024-09-13T14:47:38.466659+0000","last_became_peered":"2024-09-13T14:47:38.466659+0000","last_unstale":"2024-09-13T14:57:36.047116+0000","last_undegraded":"2024-09-13T14:57:36.047116+0000","last_fullsized":"2024-09-13T14:57:36.047116+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:41.482364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:41.482364+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:50:49.239389+0000","scrub_duration":0.027843855000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":285,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.519673+0000","last_change":"2024-09-13T14:36:13.862181+0000","last_active":"2024-09-13T14:57:54.519673+0000","last_peered":"2024-09-13T14:57:54.519673+0000","last_clean":"2024-09-13T14:57:54.519673+0000","last_became_active":"2024-09-13T14:36:13.861873+0000","last_became_peered":"2024-09-13T14:36:13.861873+0000","last_unstale":"2024-09-13T14:57:54.519673+0000","last_undegraded":"2024-09-13T14:57:54.519673+0000","last_fullsized":"2024-09-13T14:57:54.519673+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:14.085700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":56,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T14:57:36.048337+0000","last_change":"2024-09-13T14:56:12.489695+0000","last_active":"2024-09-13T14:57:36.048337+0000","last_peered":"2024-09-13T14:57:36.048337+0000","last_clean":"2024-09-13T14:57:36.048337+0000","last_became_active":"2024-09-13T14:56:12.489302+0000","last_became_peered":"2024-09-13T14:56:12.489302+0000","last_unstale":"2024-09-13T14:57:36.048337+0000","last_undegraded":"2024-09-13T14:57:36.048337+0000","last_fullsized":"2024-09-13T14:57:36.048337+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:28:53.275229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"90'149","reported_seq":737,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.552582+0000","last_change":"2024-09-13T14:55:20.150382+0000","last_active":"2024-09-13T14:57:54.552582+0000","last_peered":"2024-09-13T14:57:54.552582+0000","last_clean":"2024-09-13T14:57:54.552582+0000","last_became_active":"2024-09-13T14:55:20.149422+0000","last_became_peered":"2024-09-13T14:55:20.149422+0000","last_unstale":"2024-09-13T14:57:54.552582+0000","last_undegraded":"2024-09-13T14:57:54.552582+0000","last_fullsized":"2024-09-13T14:57:54.552582+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:51.226748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:51.226748+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:51:07.379350+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":385,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.530192+0000","last_change":"2024-09-13T14:56:12.490265+0000","last_active":"2024-09-13T14:57:54.530192+0000","last_peered":"2024-09-13T14:57:54.530192+0000","last_clean":"2024-09-13T14:57:54.530192+0000","last_became_active":"2024-09-13T14:56:12.489937+0000","last_became_peered":"2024-09-13T14:56:12.489937+0000","last_unstale":"2024-09-13T14:57:54.530192+0000","last_undegraded":"2024-09-13T14:57:54.530192+0000","last_fullsized":"2024-09-13T14:57:54.530192+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:49.461532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":464,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T14:57:36.049053+0000","last_change":"2024-09-13T14:55:17.934695+0000","last_active":"2024-09-13T14:57:36.049053+0000","last_peered":"2024-09-13T14:57:36.049053+0000","last_clean":"2024-09-13T14:57:36.049053+0000","last_became_active":"2024-09-13T14:55:17.934165+0000","last_became_peered":"2024-09-13T14:55:17.934165+0000","last_unstale":"2024-09-13T14:57:36.049053+0000","last_undegraded":"2024-09-13T14:57:36.049053+0000","last_fullsized":"2024-09-13T14:57:36.049053+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:36:25.739706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"90'149","reported_seq":565,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T14:57:36.047610+0000","last_change":"2024-09-13T14:47:50.892667+0000","last_active":"2024-09-13T14:57:36.047610+0000","last_peered":"2024-09-13T14:57:36.047610+0000","last_clean":"2024-09-13T14:57:36.047610+0000","last_became_active":"2024-09-13T14:47:50.883312+0000","last_became_peered":"2024-09-13T14:47:50.883312+0000","last_unstale":"2024-09-13T14:57:36.047610+0000","last_undegraded":"2024-09-13T14:57:36.047610+0000","last_fullsized":"2024-09-13T14:57:36.047610+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:45.441819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:45.441819+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:39.138573+0000","scrub_duration":0.021368558999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":899,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.533139+0000","last_change":"2024-09-13T14:50:27.491166+0000","last_active":"2024-09-13T14:57:54.533139+0000","last_peered":"2024-09-13T14:57:54.533139+0000","last_clean":"2024-09-13T14:57:54.533139+0000","last_became_active":"2024-09-13T14:50:27.489905+0000","last_became_peered":"2024-09-13T14:50:27.489905+0000","last_unstale":"2024-09-13T14:57:54.533139+0000","last_undegraded":"2024-09-13T14:57:54.533139+0000","last_fullsized":"2024-09-13T14:57:54.533139+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:32.237904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":149,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T14:57:36.047226+0000","last_change":"2024-09-13T14:56:12.490197+0000","last_active":"2024-09-13T14:57:36.047226+0000","last_peered":"2024-09-13T14:57:36.047226+0000","last_clean":"2024-09-13T14:57:36.047226+0000","last_became_active":"2024-09-13T14:56:12.489554+0000","last_became_peered":"2024-09-13T14:56:12.489554+0000","last_unstale":"2024-09-13T14:57:36.047226+0000","last_undegraded":"2024-09-13T14:57:36.047226+0000","last_fullsized":"2024-09-13T14:57:36.047226+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:06.773211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"480'11133","reported_seq":12374,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:58:01.780180+0000","last_change":"2024-09-13T14:55:22.944506+0000","last_active":"2024-09-13T14:58:01.780180+0000","last_peered":"2024-09-13T14:58:01.780180+0000","last_clean":"2024-09-13T14:58:01.780180+0000","last_became_active":"2024-09-13T14:55:22.942014+0000","last_became_peered":"2024-09-13T14:55:22.942014+0000","last_unstale":"2024-09-13T14:58:01.780180+0000","last_undegraded":"2024-09-13T14:58:01.780180+0000","last_fullsized":"2024-09-13T14:58:01.780180+0000","mapping_epoch":446,"log_start":"461'4411","ondisk_log_start":"461'4411","created":436,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":6722,"log_dups_size":0,"ondisk_log_size":6722,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:00:33.167429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":126681088,"num_objects":1933,"num_object_clones":0,"num_object_copies":5799,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11133,"num_write_kb":418112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":584,"num_bytes_recovered":38273024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"90'149","reported_seq":605,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T14:57:39.030647+0000","last_change":"2024-09-13T14:57:39.030647+0000","last_active":"2024-09-13T14:57:39.030647+0000","last_peered":"2024-09-13T14:57:39.030647+0000","last_clean":"2024-09-13T14:57:39.030647+0000","last_became_active":"2024-09-13T14:57:17.394134+0000","last_became_peered":"2024-09-13T14:57:17.394134+0000","last_unstale":"2024-09-13T14:57:39.030647+0000","last_undegraded":"2024-09-13T14:57:39.030647+0000","last_fullsized":"2024-09-13T14:57:39.030647+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:39.030538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:39.030538+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:54:06.634791+0000","scrub_duration":0.014515093999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":262,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T14:57:36.049119+0000","last_change":"2024-09-13T14:55:17.947508+0000","last_active":"2024-09-13T14:57:36.049119+0000","last_peered":"2024-09-13T14:57:36.049119+0000","last_clean":"2024-09-13T14:57:36.049119+0000","last_became_active":"2024-09-13T14:55:17.932604+0000","last_became_peered":"2024-09-13T14:55:17.932604+0000","last_unstale":"2024-09-13T14:57:36.049119+0000","last_undegraded":"2024-09-13T14:57:36.049119+0000","last_fullsized":"2024-09-13T14:57:36.049119+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:55.084612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":132,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":306,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.519208+0000","last_change":"2024-09-13T14:38:53.898907+0000","last_active":"2024-09-13T14:57:54.519208+0000","last_peered":"2024-09-13T14:57:54.519208+0000","last_clean":"2024-09-13T14:57:54.519208+0000","last_became_active":"2024-09-13T14:38:53.898572+0000","last_became_peered":"2024-09-13T14:38:53.898572+0000","last_unstale":"2024-09-13T14:57:54.519208+0000","last_undegraded":"2024-09-13T14:57:54.519208+0000","last_fullsized":"2024-09-13T14:57:54.519208+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:29.222150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"480'11191","reported_seq":11292,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:58:03.550423+0000","last_change":"2024-09-13T14:54:58.384717+0000","last_active":"2024-09-13T14:58:03.550423+0000","last_peered":"2024-09-13T14:58:03.550423+0000","last_clean":"2024-09-13T14:58:03.550423+0000","last_became_active":"2024-09-13T14:54:58.384225+0000","last_became_peered":"2024-09-13T14:54:58.384225+0000","last_unstale":"2024-09-13T14:58:03.550423+0000","last_undegraded":"2024-09-13T14:58:03.550423+0000","last_fullsized":"2024-09-13T14:58:03.550423+0000","mapping_epoch":436,"log_start":"464'5757","ondisk_log_start":"464'5757","created":436,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":5434,"log_dups_size":0,"ondisk_log_size":5434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:25:22.716676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125763584,"num_objects":1919,"num_object_clones":0,"num_object_copies":5757,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11191,"num_write_kb":419520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"90'149","reported_seq":776,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.531735+0000","last_change":"2024-09-13T14:57:40.996705+0000","last_active":"2024-09-13T14:57:54.531735+0000","last_peered":"2024-09-13T14:57:54.531735+0000","last_clean":"2024-09-13T14:57:54.531735+0000","last_became_active":"2024-09-13T14:57:17.404299+0000","last_became_peered":"2024-09-13T14:57:17.404299+0000","last_unstale":"2024-09-13T14:57:54.531735+0000","last_undegraded":"2024-09-13T14:57:54.531735+0000","last_fullsized":"2024-09-13T14:57:54.531735+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:40.996402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:40.996402+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:04.912532+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":275,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T14:57:36.048928+0000","last_change":"2024-09-13T14:55:17.949010+0000","last_active":"2024-09-13T14:57:36.048928+0000","last_peered":"2024-09-13T14:57:36.048928+0000","last_clean":"2024-09-13T14:57:36.048928+0000","last_became_active":"2024-09-13T14:55:17.932712+0000","last_became_peered":"2024-09-13T14:55:17.932712+0000","last_unstale":"2024-09-13T14:57:36.048928+0000","last_undegraded":"2024-09-13T14:57:36.048928+0000","last_fullsized":"2024-09-13T14:57:36.048928+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:46:43.390958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":437,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.545151+0000","last_change":"2024-09-13T14:55:17.929723+0000","last_active":"2024-09-13T14:57:54.545151+0000","last_peered":"2024-09-13T14:57:54.545151+0000","last_clean":"2024-09-13T14:57:54.545151+0000","last_became_active":"2024-09-13T14:55:17.929131+0000","last_became_peered":"2024-09-13T14:55:17.929131+0000","last_unstale":"2024-09-13T14:57:54.545151+0000","last_undegraded":"2024-09-13T14:57:54.545151+0000","last_fullsized":"2024-09-13T14:57:54.545151+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:27:15.828973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.4","version":"480'11316","reported_seq":12676,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:58:03.532236+0000","last_change":"2024-09-13T14:56:12.495568+0000","last_active":"2024-09-13T14:58:03.532236+0000","last_peered":"2024-09-13T14:58:03.532236+0000","last_clean":"2024-09-13T14:58:03.532236+0000","last_became_active":"2024-09-13T14:56:12.490664+0000","last_became_peered":"2024-09-13T14:56:12.490664+0000","last_unstale":"2024-09-13T14:58:03.532236+0000","last_undegraded":"2024-09-13T14:58:03.532236+0000","last_fullsized":"2024-09-13T14:58:03.532236+0000","mapping_epoch":462,"log_start":"464'5931","ondisk_log_start":"464'5931","created":436,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":5385,"log_dups_size":0,"ondisk_log_size":5385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:17:29.226407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127795200,"num_objects":1950,"num_object_clones":0,"num_object_copies":5850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11316,"num_write_kb":424512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":623,"num_bytes_recovered":40828928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.f","version":"90'149","reported_seq":609,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T14:57:36.048170+0000","last_change":"2024-09-13T14:55:20.140396+0000","last_active":"2024-09-13T14:57:36.048170+0000","last_peered":"2024-09-13T14:57:36.048170+0000","last_clean":"2024-09-13T14:57:36.048170+0000","last_became_active":"2024-09-13T14:55:20.140087+0000","last_became_peered":"2024-09-13T14:55:20.140087+0000","last_unstale":"2024-09-13T14:57:36.048170+0000","last_undegraded":"2024-09-13T14:57:36.048170+0000","last_fullsized":"2024-09-13T14:57:36.048170+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:34.091782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:34.091782+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:46:05.540133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":486,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T14:57:36.048526+0000","last_change":"2024-09-13T14:37:49.611774+0000","last_active":"2024-09-13T14:57:36.048526+0000","last_peered":"2024-09-13T14:57:36.048526+0000","last_clean":"2024-09-13T14:57:36.048526+0000","last_became_active":"2024-09-13T14:37:49.611088+0000","last_became_peered":"2024-09-13T14:37:49.611088+0000","last_unstale":"2024-09-13T14:57:36.048526+0000","last_undegraded":"2024-09-13T14:57:36.048526+0000","last_fullsized":"2024-09-13T14:57:36.048526+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:08.933213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":302,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.549619+0000","last_change":"2024-09-13T14:47:47.865523+0000","last_active":"2024-09-13T14:57:54.549619+0000","last_peered":"2024-09-13T14:57:54.549619+0000","last_clean":"2024-09-13T14:57:54.549619+0000","last_became_active":"2024-09-13T14:47:47.865180+0000","last_became_peered":"2024-09-13T14:47:47.865180+0000","last_unstale":"2024-09-13T14:57:54.549619+0000","last_undegraded":"2024-09-13T14:57:54.549619+0000","last_fullsized":"2024-09-13T14:57:54.549619+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:39:35.479251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.5","version":"480'10969","reported_seq":11070,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:58:01.689404+0000","last_change":"2024-09-13T14:54:57.846849+0000","last_active":"2024-09-13T14:58:01.689404+0000","last_peered":"2024-09-13T14:58:01.689404+0000","last_clean":"2024-09-13T14:58:01.689404+0000","last_became_active":"2024-09-13T14:54:57.846557+0000","last_became_peered":"2024-09-13T14:54:57.846557+0000","last_unstale":"2024-09-13T14:58:01.689404+0000","last_undegraded":"2024-09-13T14:58:01.689404+0000","last_fullsized":"2024-09-13T14:58:01.689404+0000","mapping_epoch":436,"log_start":"460'3591","ondisk_log_start":"460'3591","created":436,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":7378,"log_dups_size":0,"ondisk_log_size":7378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:49.590084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130875392,"num_objects":1997,"num_object_clones":0,"num_object_copies":5991,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10969,"num_write_kb":414912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"90'149","reported_seq":516,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T14:57:38.048899+0000","last_change":"2024-09-13T14:57:38.048899+0000","last_active":"2024-09-13T14:57:38.048899+0000","last_peered":"2024-09-13T14:57:38.048899+0000","last_clean":"2024-09-13T14:57:38.048899+0000","last_became_active":"2024-09-13T14:57:17.384951+0000","last_became_peered":"2024-09-13T14:57:17.384951+0000","last_unstale":"2024-09-13T14:57:38.048899+0000","last_undegraded":"2024-09-13T14:57:38.048899+0000","last_fullsized":"2024-09-13T14:57:38.048899+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:38.048812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:38.048812+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:37:42.701104+0000","scrub_duration":0.004943551,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":296,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T14:57:36.047535+0000","last_change":"2024-09-13T14:37:49.168003+0000","last_active":"2024-09-13T14:57:36.047535+0000","last_peered":"2024-09-13T14:57:36.047535+0000","last_clean":"2024-09-13T14:57:36.047535+0000","last_became_active":"2024-09-13T14:37:49.167638+0000","last_became_peered":"2024-09-13T14:37:49.167638+0000","last_unstale":"2024-09-13T14:57:36.047535+0000","last_undegraded":"2024-09-13T14:57:36.047535+0000","last_fullsized":"2024-09-13T14:57:36.047535+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:29.051981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":464,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.557232+0000","last_change":"2024-09-13T14:56:12.491208+0000","last_active":"2024-09-13T14:57:54.557232+0000","last_peered":"2024-09-13T14:57:54.557232+0000","last_clean":"2024-09-13T14:57:54.557232+0000","last_became_active":"2024-09-13T14:56:12.490943+0000","last_became_peered":"2024-09-13T14:56:12.490943+0000","last_unstale":"2024-09-13T14:57:54.557232+0000","last_undegraded":"2024-09-13T14:57:54.557232+0000","last_fullsized":"2024-09-13T14:57:54.557232+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:31:57.892194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"480'11330","reported_seq":12572,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:58:04.565149+0000","last_change":"2024-09-13T14:55:37.154924+0000","last_active":"2024-09-13T14:58:04.565149+0000","last_peered":"2024-09-13T14:58:04.565149+0000","last_clean":"2024-09-13T14:58:04.565149+0000","last_became_active":"2024-09-13T14:55:37.153166+0000","last_became_peered":"2024-09-13T14:55:37.153166+0000","last_unstale":"2024-09-13T14:58:04.565149+0000","last_undegraded":"2024-09-13T14:58:04.565149+0000","last_fullsized":"2024-09-13T14:58:04.565149+0000","mapping_epoch":459,"log_start":"464'6022","ondisk_log_start":"464'6022","created":436,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":5308,"log_dups_size":0,"ondisk_log_size":5308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:48:15.748240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118358016,"num_objects":1806,"num_object_clones":0,"num_object_copies":5418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11330,"num_write_kb":420352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":565,"num_bytes_recovered":37027840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"90'149","reported_seq":679,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T14:57:41.926732+0000","last_change":"2024-09-13T14:57:41.926732+0000","last_active":"2024-09-13T14:57:41.926732+0000","last_peered":"2024-09-13T14:57:41.926732+0000","last_clean":"2024-09-13T14:57:41.926732+0000","last_became_active":"2024-09-13T14:57:17.380843+0000","last_became_peered":"2024-09-13T14:57:17.380843+0000","last_unstale":"2024-09-13T14:57:41.926732+0000","last_undegraded":"2024-09-13T14:57:41.926732+0000","last_fullsized":"2024-09-13T14:57:41.926732+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:41.926626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:41.926626+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:08:57.137123+0000","scrub_duration":0.0019476179999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":859,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.532281+0000","last_change":"2024-09-13T14:55:17.944596+0000","last_active":"2024-09-13T14:57:54.532281+0000","last_peered":"2024-09-13T14:57:54.532281+0000","last_clean":"2024-09-13T14:57:54.532281+0000","last_became_active":"2024-09-13T14:55:17.935767+0000","last_became_peered":"2024-09-13T14:55:17.935767+0000","last_unstale":"2024-09-13T14:57:54.532281+0000","last_undegraded":"2024-09-13T14:57:54.532281+0000","last_fullsized":"2024-09-13T14:57:54.532281+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:22:19.619639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":389,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.524615+0000","last_change":"2024-09-13T14:55:17.933005+0000","last_active":"2024-09-13T14:57:54.524615+0000","last_peered":"2024-09-13T14:57:54.524615+0000","last_clean":"2024-09-13T14:57:54.524615+0000","last_became_active":"2024-09-13T14:55:17.931575+0000","last_became_peered":"2024-09-13T14:55:17.931575+0000","last_unstale":"2024-09-13T14:57:54.524615+0000","last_undegraded":"2024-09-13T14:57:54.524615+0000","last_fullsized":"2024-09-13T14:57:54.524615+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:50:20.352970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"480'11210","reported_seq":11311,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:58:01.657169+0000","last_change":"2024-09-13T14:54:57.846161+0000","last_active":"2024-09-13T14:58:01.657169+0000","last_peered":"2024-09-13T14:58:01.657169+0000","last_clean":"2024-09-13T14:58:01.657169+0000","last_became_active":"2024-09-13T14:54:57.845348+0000","last_became_peered":"2024-09-13T14:54:57.845348+0000","last_unstale":"2024-09-13T14:58:01.657169+0000","last_undegraded":"2024-09-13T14:58:01.657169+0000","last_fullsized":"2024-09-13T14:58:01.657169+0000","mapping_epoch":436,"log_start":"460'3876","ondisk_log_start":"460'3876","created":436,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":7334,"log_dups_size":0,"ondisk_log_size":7334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:07:01.751407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135528448,"num_objects":2068,"num_object_clones":0,"num_object_copies":6204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11210,"num_write_kb":424896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"90'149","reported_seq":582,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T14:57:39.812785+0000","last_change":"2024-09-13T14:57:39.812785+0000","last_active":"2024-09-13T14:57:39.812785+0000","last_peered":"2024-09-13T14:57:39.812785+0000","last_clean":"2024-09-13T14:57:39.812785+0000","last_became_active":"2024-09-13T14:57:17.383307+0000","last_became_peered":"2024-09-13T14:57:17.383307+0000","last_unstale":"2024-09-13T14:57:39.812785+0000","last_undegraded":"2024-09-13T14:57:39.812785+0000","last_fullsized":"2024-09-13T14:57:39.812785+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:39.812688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:39.812688+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:18:59.726465+0000","scrub_duration":0.010496065000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":346,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.549373+0000","last_change":"2024-09-13T14:55:17.935369+0000","last_active":"2024-09-13T14:57:54.549373+0000","last_peered":"2024-09-13T14:57:54.549373+0000","last_clean":"2024-09-13T14:57:54.549373+0000","last_became_active":"2024-09-13T14:55:17.930815+0000","last_became_peered":"2024-09-13T14:55:17.930815+0000","last_unstale":"2024-09-13T14:57:54.549373+0000","last_undegraded":"2024-09-13T14:57:54.549373+0000","last_fullsized":"2024-09-13T14:57:54.549373+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:59:31.247160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":475,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.528707+0000","last_change":"2024-09-13T14:55:17.932243+0000","last_active":"2024-09-13T14:57:54.528707+0000","last_peered":"2024-09-13T14:57:54.528707+0000","last_clean":"2024-09-13T14:57:54.528707+0000","last_became_active":"2024-09-13T14:55:17.931961+0000","last_became_peered":"2024-09-13T14:55:17.931961+0000","last_unstale":"2024-09-13T14:57:54.528707+0000","last_undegraded":"2024-09-13T14:57:54.528707+0000","last_fullsized":"2024-09-13T14:57:54.528707+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:23:52.680942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.0","version":"480'11395","reported_seq":12816,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:58:04.549361+0000","last_change":"2024-09-13T14:55:24.959843+0000","last_active":"2024-09-13T14:58:04.549361+0000","last_peered":"2024-09-13T14:58:04.549361+0000","last_clean":"2024-09-13T14:58:04.549361+0000","last_became_active":"2024-09-13T14:55:24.957491+0000","last_became_peered":"2024-09-13T14:55:24.957491+0000","last_unstale":"2024-09-13T14:58:04.549361+0000","last_undegraded":"2024-09-13T14:58:04.549361+0000","last_fullsized":"2024-09-13T14:58:04.549361+0000","mapping_epoch":448,"log_start":"464'6038","ondisk_log_start":"464'6038","created":436,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":5357,"log_dups_size":0,"ondisk_log_size":5357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:09:19.929604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116064256,"num_objects":1771,"num_object_clones":0,"num_object_copies":5313,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11395,"num_write_kb":421312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":654,"num_bytes_recovered":42860544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"90'149","reported_seq":769,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.544881+0000","last_change":"2024-09-13T14:57:39.766173+0000","last_active":"2024-09-13T14:57:54.544881+0000","last_peered":"2024-09-13T14:57:54.544881+0000","last_clean":"2024-09-13T14:57:54.544881+0000","last_became_active":"2024-09-13T14:57:17.389370+0000","last_became_peered":"2024-09-13T14:57:17.389370+0000","last_unstale":"2024-09-13T14:57:54.544881+0000","last_undegraded":"2024-09-13T14:57:54.544881+0000","last_fullsized":"2024-09-13T14:57:54.544881+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:39.765978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:39.765978+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:39.980756+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":395,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.527505+0000","last_change":"2024-09-13T14:55:17.932199+0000","last_active":"2024-09-13T14:57:54.527505+0000","last_peered":"2024-09-13T14:57:54.527505+0000","last_clean":"2024-09-13T14:57:54.527505+0000","last_became_active":"2024-09-13T14:55:17.931873+0000","last_became_peered":"2024-09-13T14:55:17.931873+0000","last_unstale":"2024-09-13T14:57:54.527505+0000","last_undegraded":"2024-09-13T14:57:54.527505+0000","last_fullsized":"2024-09-13T14:57:54.527505+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:02:18.004113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":422,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.545713+0000","last_change":"2024-09-13T14:55:17.931269+0000","last_active":"2024-09-13T14:57:54.545713+0000","last_peered":"2024-09-13T14:57:54.545713+0000","last_clean":"2024-09-13T14:57:54.545713+0000","last_became_active":"2024-09-13T14:55:17.930909+0000","last_became_peered":"2024-09-13T14:55:17.930909+0000","last_unstale":"2024-09-13T14:57:54.545713+0000","last_undegraded":"2024-09-13T14:57:54.545713+0000","last_fullsized":"2024-09-13T14:57:54.545713+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:47.592526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"480'11641","reported_seq":13049,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:58:01.808280+0000","last_change":"2024-09-13T14:55:35.136618+0000","last_active":"2024-09-13T14:58:01.808280+0000","last_peered":"2024-09-13T14:58:01.808280+0000","last_clean":"2024-09-13T14:58:01.808280+0000","last_became_active":"2024-09-13T14:55:35.134043+0000","last_became_peered":"2024-09-13T14:55:35.134043+0000","last_unstale":"2024-09-13T14:58:01.808280+0000","last_undegraded":"2024-09-13T14:58:01.808280+0000","last_fullsized":"2024-09-13T14:58:01.808280+0000","mapping_epoch":457,"log_start":"464'5597","ondisk_log_start":"464'5597","created":436,"last_epoch_clean":458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":6044,"log_dups_size":0,"ondisk_log_size":6044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:55.445002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135856128,"num_objects":2073,"num_object_clones":0,"num_object_copies":6219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11641,"num_write_kb":438848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":667,"num_bytes_recovered":43712512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"90'149","reported_seq":582,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T14:57:41.774703+0000","last_change":"2024-09-13T14:57:38.849773+0000","last_active":"2024-09-13T14:57:41.774703+0000","last_peered":"2024-09-13T14:57:41.774703+0000","last_clean":"2024-09-13T14:57:41.774703+0000","last_became_active":"2024-09-13T14:57:17.378580+0000","last_became_peered":"2024-09-13T14:57:17.378580+0000","last_unstale":"2024-09-13T14:57:41.774703+0000","last_undegraded":"2024-09-13T14:57:41.774703+0000","last_fullsized":"2024-09-13T14:57:41.774703+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:38.849610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:38.849610+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:22:24.723639+0000","scrub_duration":0.0034255890000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":874,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.527835+0000","last_change":"2024-09-13T14:47:47.861249+0000","last_active":"2024-09-13T14:57:54.527835+0000","last_peered":"2024-09-13T14:57:54.527835+0000","last_clean":"2024-09-13T14:57:54.527835+0000","last_became_active":"2024-09-13T14:47:47.860873+0000","last_became_peered":"2024-09-13T14:47:47.860873+0000","last_unstale":"2024-09-13T14:57:54.527835+0000","last_undegraded":"2024-09-13T14:57:54.527835+0000","last_fullsized":"2024-09-13T14:57:54.527835+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:37:50.709437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":309,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.545785+0000","last_change":"2024-09-13T14:55:17.933070+0000","last_active":"2024-09-13T14:57:54.545785+0000","last_peered":"2024-09-13T14:57:54.545785+0000","last_clean":"2024-09-13T14:57:54.545785+0000","last_became_active":"2024-09-13T14:55:17.931232+0000","last_became_peered":"2024-09-13T14:55:17.931232+0000","last_unstale":"2024-09-13T14:57:54.545785+0000","last_undegraded":"2024-09-13T14:57:54.545785+0000","last_fullsized":"2024-09-13T14:57:54.545785+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:22:46.719483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"480'11251","reported_seq":11351,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:58:04.587960+0000","last_change":"2024-09-13T14:54:58.384133+0000","last_active":"2024-09-13T14:58:04.587960+0000","last_peered":"2024-09-13T14:58:04.587960+0000","last_clean":"2024-09-13T14:58:04.587960+0000","last_became_active":"2024-09-13T14:54:58.383823+0000","last_became_peered":"2024-09-13T14:54:58.383823+0000","last_unstale":"2024-09-13T14:58:04.587960+0000","last_undegraded":"2024-09-13T14:58:04.587960+0000","last_fullsized":"2024-09-13T14:58:04.587960+0000","mapping_epoch":436,"log_start":"464'5922","ondisk_log_start":"464'5922","created":436,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":5329,"log_dups_size":0,"ondisk_log_size":5329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:30.724982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120258560,"num_objects":1835,"num_object_clones":0,"num_object_copies":5505,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11251,"num_write_kb":418752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":664,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.521141+0000","last_change":"2024-09-13T14:47:31.684319+0000","last_active":"2024-09-13T14:57:54.521141+0000","last_peered":"2024-09-13T14:57:54.521141+0000","last_clean":"2024-09-13T14:57:54.521141+0000","last_became_active":"2024-09-13T14:47:29.415658+0000","last_became_peered":"2024-09-13T14:47:29.415658+0000","last_unstale":"2024-09-13T14:57:54.521141+0000","last_undegraded":"2024-09-13T14:57:54.521141+0000","last_fullsized":"2024-09-13T14:57:54.521141+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:31.681433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:31.681433+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:29:34.663513+0000","scrub_duration":0.0059601139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":792,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T14:57:36.047686+0000","last_change":"2024-09-13T14:46:04.780718+0000","last_active":"2024-09-13T14:57:36.047686+0000","last_peered":"2024-09-13T14:57:36.047686+0000","last_clean":"2024-09-13T14:57:36.047686+0000","last_became_active":"2024-09-13T14:46:04.780120+0000","last_became_peered":"2024-09-13T14:46:04.780120+0000","last_unstale":"2024-09-13T14:57:36.047686+0000","last_undegraded":"2024-09-13T14:57:36.047686+0000","last_fullsized":"2024-09-13T14:57:36.047686+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:57:14.475888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":801,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.531092+0000","last_change":"2024-09-13T14:37:49.160719+0000","last_active":"2024-09-13T14:57:54.531092+0000","last_peered":"2024-09-13T14:57:54.531092+0000","last_clean":"2024-09-13T14:57:54.531092+0000","last_became_active":"2024-09-13T14:37:49.160401+0000","last_became_peered":"2024-09-13T14:37:49.160401+0000","last_unstale":"2024-09-13T14:57:54.531092+0000","last_undegraded":"2024-09-13T14:57:54.531092+0000","last_fullsized":"2024-09-13T14:57:54.531092+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:21.737945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":123,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T14:57:36.048441+0000","last_change":"2024-09-13T14:55:17.933431+0000","last_active":"2024-09-13T14:57:36.048441+0000","last_peered":"2024-09-13T14:57:36.048441+0000","last_clean":"2024-09-13T14:57:36.048441+0000","last_became_active":"2024-09-13T14:55:17.932711+0000","last_became_peered":"2024-09-13T14:55:17.932711+0000","last_unstale":"2024-09-13T14:57:36.048441+0000","last_undegraded":"2024-09-13T14:57:36.048441+0000","last_fullsized":"2024-09-13T14:57:36.048441+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:47.712045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"480'10993","reported_seq":11104,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:58:01.704602+0000","last_change":"2024-09-13T14:56:12.516271+0000","last_active":"2024-09-13T14:58:01.704602+0000","last_peered":"2024-09-13T14:58:01.704602+0000","last_clean":"2024-09-13T14:58:01.704602+0000","last_became_active":"2024-09-13T14:56:12.494537+0000","last_became_peered":"2024-09-13T14:56:12.494537+0000","last_unstale":"2024-09-13T14:58:01.704602+0000","last_undegraded":"2024-09-13T14:58:01.704602+0000","last_fullsized":"2024-09-13T14:58:01.704602+0000","mapping_epoch":462,"log_start":"460'3659","ondisk_log_start":"460'3659","created":436,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":7334,"log_dups_size":0,"ondisk_log_size":7334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:07:34.989938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132841472,"num_objects":2027,"num_object_clones":0,"num_object_copies":6081,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10993,"num_write_kb":416640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"90'149","reported_seq":744,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.552426+0000","last_change":"2024-09-13T14:55:20.148734+0000","last_active":"2024-09-13T14:57:54.552426+0000","last_peered":"2024-09-13T14:57:54.552426+0000","last_clean":"2024-09-13T14:57:54.552426+0000","last_became_active":"2024-09-13T14:55:20.147918+0000","last_became_peered":"2024-09-13T14:55:20.147918+0000","last_unstale":"2024-09-13T14:57:54.552426+0000","last_undegraded":"2024-09-13T14:57:54.552426+0000","last_fullsized":"2024-09-13T14:57:54.552426+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:34.603316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:34.603316+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:27.859151+0000","scrub_duration":0.0057131120000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":733,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T14:57:36.051399+0000","last_change":"2024-09-13T14:47:47.892973+0000","last_active":"2024-09-13T14:57:36.051399+0000","last_peered":"2024-09-13T14:57:36.051399+0000","last_clean":"2024-09-13T14:57:36.051399+0000","last_became_active":"2024-09-13T14:47:47.866986+0000","last_became_peered":"2024-09-13T14:47:47.866986+0000","last_unstale":"2024-09-13T14:57:36.051399+0000","last_undegraded":"2024-09-13T14:57:36.051399+0000","last_fullsized":"2024-09-13T14:57:36.051399+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:07.353290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":458,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.530190+0000","last_change":"2024-09-13T14:56:12.490871+0000","last_active":"2024-09-13T14:57:54.530190+0000","last_peered":"2024-09-13T14:57:54.530190+0000","last_clean":"2024-09-13T14:57:54.530190+0000","last_became_active":"2024-09-13T14:56:12.490535+0000","last_became_peered":"2024-09-13T14:56:12.490535+0000","last_unstale":"2024-09-13T14:57:54.530190+0000","last_undegraded":"2024-09-13T14:57:54.530190+0000","last_fullsized":"2024-09-13T14:57:54.530190+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:49.429179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":258,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.545361+0000","last_change":"2024-09-13T14:56:12.490229+0000","last_active":"2024-09-13T14:57:54.545361+0000","last_peered":"2024-09-13T14:57:54.545361+0000","last_clean":"2024-09-13T14:57:54.545361+0000","last_became_active":"2024-09-13T14:56:12.489512+0000","last_became_peered":"2024-09-13T14:56:12.489512+0000","last_unstale":"2024-09-13T14:57:54.545361+0000","last_undegraded":"2024-09-13T14:57:54.545361+0000","last_fullsized":"2024-09-13T14:57:54.545361+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:26:10.594549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.9","version":"480'11401","reported_seq":11455,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:58:01.731621+0000","last_change":"2024-09-13T14:54:57.844803+0000","last_active":"2024-09-13T14:58:01.731621+0000","last_peered":"2024-09-13T14:58:01.731621+0000","last_clean":"2024-09-13T14:58:01.731621+0000","last_became_active":"2024-09-13T14:54:57.844499+0000","last_became_peered":"2024-09-13T14:54:57.844499+0000","last_unstale":"2024-09-13T14:58:01.731621+0000","last_undegraded":"2024-09-13T14:58:01.731621+0000","last_fullsized":"2024-09-13T14:58:01.731621+0000","mapping_epoch":436,"log_start":"461'4635","ondisk_log_start":"461'4635","created":436,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":6766,"log_dups_size":0,"ondisk_log_size":6766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:46.617173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129302528,"num_objects":1973,"num_object_clones":0,"num_object_copies":5919,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11401,"num_write_kb":427968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"90'149","reported_seq":697,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T14:57:36.048037+0000","last_change":"2024-09-13T14:55:20.134578+0000","last_active":"2024-09-13T14:57:36.048037+0000","last_peered":"2024-09-13T14:57:36.048037+0000","last_clean":"2024-09-13T14:57:36.048037+0000","last_became_active":"2024-09-13T14:55:20.134120+0000","last_became_peered":"2024-09-13T14:55:20.134120+0000","last_unstale":"2024-09-13T14:57:36.048037+0000","last_undegraded":"2024-09-13T14:57:36.048037+0000","last_fullsized":"2024-09-13T14:57:36.048037+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:29.629342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:29.629342+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:38:13.399024+0000","scrub_duration":0.0099290990000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":785,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.527408+0000","last_change":"2024-09-13T14:46:04.782955+0000","last_active":"2024-09-13T14:57:54.527408+0000","last_peered":"2024-09-13T14:57:54.527408+0000","last_clean":"2024-09-13T14:57:54.527408+0000","last_became_active":"2024-09-13T14:46:04.782250+0000","last_became_peered":"2024-09-13T14:46:04.782250+0000","last_unstale":"2024-09-13T14:57:54.527408+0000","last_undegraded":"2024-09-13T14:57:54.527408+0000","last_fullsized":"2024-09-13T14:57:54.527408+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:24.596922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":303,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.544802+0000","last_change":"2024-09-13T14:47:47.866684+0000","last_active":"2024-09-13T14:57:54.544802+0000","last_peered":"2024-09-13T14:57:54.544802+0000","last_clean":"2024-09-13T14:57:54.544802+0000","last_became_active":"2024-09-13T14:47:47.866400+0000","last_became_peered":"2024-09-13T14:47:47.866400+0000","last_unstale":"2024-09-13T14:57:54.544802+0000","last_undegraded":"2024-09-13T14:57:54.544802+0000","last_fullsized":"2024-09-13T14:57:54.544802+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:08:34.289658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":302,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.545880+0000","last_change":"2024-09-13T14:47:47.870218+0000","last_active":"2024-09-13T14:57:54.545880+0000","last_peered":"2024-09-13T14:57:54.545880+0000","last_clean":"2024-09-13T14:57:54.545880+0000","last_became_active":"2024-09-13T14:47:47.869907+0000","last_became_peered":"2024-09-13T14:47:47.869907+0000","last_unstale":"2024-09-13T14:57:54.545880+0000","last_undegraded":"2024-09-13T14:57:54.545880+0000","last_fullsized":"2024-09-13T14:57:54.545880+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:41:49.141441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"480'11528","reported_seq":11616,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:58:01.753448+0000","last_change":"2024-09-13T14:56:12.497755+0000","last_active":"2024-09-13T14:58:01.753448+0000","last_peered":"2024-09-13T14:58:01.753448+0000","last_clean":"2024-09-13T14:58:01.753448+0000","last_became_active":"2024-09-13T14:56:12.489758+0000","last_became_peered":"2024-09-13T14:56:12.489758+0000","last_unstale":"2024-09-13T14:58:01.753448+0000","last_undegraded":"2024-09-13T14:58:01.753448+0000","last_fullsized":"2024-09-13T14:58:01.753448+0000","mapping_epoch":462,"log_start":"461'4769","ondisk_log_start":"461'4769","created":436,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":6759,"log_dups_size":0,"ondisk_log_size":6759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:22:53.808045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137363456,"num_objects":2096,"num_object_clones":0,"num_object_copies":6288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11528,"num_write_kb":435968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"90'149","reported_seq":582,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T14:57:36.047276+0000","last_change":"2024-09-13T14:47:32.275753+0000","last_active":"2024-09-13T14:57:36.047276+0000","last_peered":"2024-09-13T14:57:36.047276+0000","last_clean":"2024-09-13T14:57:36.047276+0000","last_became_active":"2024-09-13T14:47:29.401066+0000","last_became_peered":"2024-09-13T14:47:29.401066+0000","last_unstale":"2024-09-13T14:57:36.047276+0000","last_undegraded":"2024-09-13T14:57:36.047276+0000","last_fullsized":"2024-09-13T14:57:36.047276+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:32.275562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:32.275562+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:43:41.056775+0000","scrub_duration":0.0054674260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":1237,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T14:57:36.047407+0000","last_change":"2024-09-13T14:47:50.892842+0000","last_active":"2024-09-13T14:57:36.047407+0000","last_peered":"2024-09-13T14:57:36.047407+0000","last_clean":"2024-09-13T14:57:36.047407+0000","last_became_active":"2024-09-13T14:47:50.892237+0000","last_became_peered":"2024-09-13T14:47:50.892237+0000","last_unstale":"2024-09-13T14:57:36.047407+0000","last_undegraded":"2024-09-13T14:57:36.047407+0000","last_fullsized":"2024-09-13T14:57:36.047407+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:20:05.796762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":8388646,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":288,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.544467+0000","last_change":"2024-09-13T14:56:12.491483+0000","last_active":"2024-09-13T14:57:54.544467+0000","last_peered":"2024-09-13T14:57:54.544467+0000","last_clean":"2024-09-13T14:57:54.544467+0000","last_became_active":"2024-09-13T14:56:12.491191+0000","last_became_peered":"2024-09-13T14:56:12.491191+0000","last_unstale":"2024-09-13T14:57:54.544467+0000","last_undegraded":"2024-09-13T14:57:54.544467+0000","last_fullsized":"2024-09-13T14:57:54.544467+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:54:11.122787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":302,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.545473+0000","last_change":"2024-09-13T14:47:47.877272+0000","last_active":"2024-09-13T14:57:54.545473+0000","last_peered":"2024-09-13T14:57:54.545473+0000","last_clean":"2024-09-13T14:57:54.545473+0000","last_became_active":"2024-09-13T14:47:47.877003+0000","last_became_peered":"2024-09-13T14:47:47.877003+0000","last_unstale":"2024-09-13T14:57:54.545473+0000","last_undegraded":"2024-09-13T14:57:54.545473+0000","last_fullsized":"2024-09-13T14:57:54.545473+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:47:23.172452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.c","version":"480'11291","reported_seq":12628,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:58:01.806038+0000","last_change":"2024-09-13T14:55:28.486018+0000","last_active":"2024-09-13T14:58:01.806038+0000","last_peered":"2024-09-13T14:58:01.806038+0000","last_clean":"2024-09-13T14:58:01.806038+0000","last_became_active":"2024-09-13T14:55:28.483879+0000","last_became_peered":"2024-09-13T14:55:28.483879+0000","last_unstale":"2024-09-13T14:58:01.806038+0000","last_undegraded":"2024-09-13T14:58:01.806038+0000","last_fullsized":"2024-09-13T14:58:01.806038+0000","mapping_epoch":451,"log_start":"464'5230","ondisk_log_start":"464'5230","created":436,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":6061,"log_dups_size":0,"ondisk_log_size":6061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:45:38.042233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129957888,"num_objects":1983,"num_object_clones":0,"num_object_copies":5949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11291,"num_write_kb":424768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":632,"num_bytes_recovered":41418752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"90'149","reported_seq":764,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.533044+0000","last_change":"2024-09-13T14:57:37.019594+0000","last_active":"2024-09-13T14:57:54.533044+0000","last_peered":"2024-09-13T14:57:54.533044+0000","last_clean":"2024-09-13T14:57:54.533044+0000","last_became_active":"2024-09-13T14:57:17.385457+0000","last_became_peered":"2024-09-13T14:57:17.385457+0000","last_unstale":"2024-09-13T14:57:54.533044+0000","last_undegraded":"2024-09-13T14:57:54.533044+0000","last_fullsized":"2024-09-13T14:57:54.533044+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:37.019510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:37.019510+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:54:21.275369+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":306,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T14:57:36.047793+0000","last_change":"2024-09-13T14:46:01.763906+0000","last_active":"2024-09-13T14:57:36.047793+0000","last_peered":"2024-09-13T14:57:36.047793+0000","last_clean":"2024-09-13T14:57:36.047793+0000","last_became_active":"2024-09-13T14:46:01.763542+0000","last_became_peered":"2024-09-13T14:46:01.763542+0000","last_unstale":"2024-09-13T14:57:36.047793+0000","last_undegraded":"2024-09-13T14:57:36.047793+0000","last_fullsized":"2024-09-13T14:57:36.047793+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:09:36.652767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":1590,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.545561+0000","last_change":"2024-09-13T14:47:50.883365+0000","last_active":"2024-09-13T14:57:54.545561+0000","last_peered":"2024-09-13T14:57:54.545561+0000","last_clean":"2024-09-13T14:57:54.545561+0000","last_became_active":"2024-09-13T14:47:50.882609+0000","last_became_peered":"2024-09-13T14:47:50.882609+0000","last_unstale":"2024-09-13T14:57:54.545561+0000","last_undegraded":"2024-09-13T14:57:54.545561+0000","last_fullsized":"2024-09-13T14:57:54.545561+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:12:17.961575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":309,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.549764+0000","last_change":"2024-09-13T14:55:17.937000+0000","last_active":"2024-09-13T14:57:54.549764+0000","last_peered":"2024-09-13T14:57:54.549764+0000","last_clean":"2024-09-13T14:57:54.549764+0000","last_became_active":"2024-09-13T14:55:17.936557+0000","last_became_peered":"2024-09-13T14:55:17.936557+0000","last_unstale":"2024-09-13T14:57:54.549764+0000","last_undegraded":"2024-09-13T14:57:54.549764+0000","last_fullsized":"2024-09-13T14:57:54.549764+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:55:25.879096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"480'11193","reported_seq":12448,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:58:03.493345+0000","last_change":"2024-09-13T14:56:12.498406+0000","last_active":"2024-09-13T14:58:03.493345+0000","last_peered":"2024-09-13T14:58:03.493345+0000","last_clean":"2024-09-13T14:58:03.493345+0000","last_became_active":"2024-09-13T14:56:12.490622+0000","last_became_peered":"2024-09-13T14:56:12.490622+0000","last_unstale":"2024-09-13T14:58:03.493345+0000","last_undegraded":"2024-09-13T14:58:03.493345+0000","last_fullsized":"2024-09-13T14:58:03.493345+0000","mapping_epoch":462,"log_start":"464'5779","ondisk_log_start":"464'5779","created":436,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":5414,"log_dups_size":0,"ondisk_log_size":5414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:07:04.577772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125567000,"num_objects":1917,"num_object_clones":0,"num_object_copies":5751,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1917,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11193,"num_write_kb":419457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":572,"num_bytes_recovered":37486592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"90'149","reported_seq":459,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.529972+0000","last_change":"2024-09-13T14:57:41.951221+0000","last_active":"2024-09-13T14:57:54.529972+0000","last_peered":"2024-09-13T14:57:54.529972+0000","last_clean":"2024-09-13T14:57:54.529972+0000","last_became_active":"2024-09-13T14:57:17.390576+0000","last_became_peered":"2024-09-13T14:57:17.390576+0000","last_unstale":"2024-09-13T14:57:54.529972+0000","last_undegraded":"2024-09-13T14:57:54.529972+0000","last_fullsized":"2024-09-13T14:57:54.529972+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:41.951149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:41.951149+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:49:55.865705+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":302,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.545651+0000","last_change":"2024-09-13T14:47:47.876562+0000","last_active":"2024-09-13T14:57:54.545651+0000","last_peered":"2024-09-13T14:57:54.545651+0000","last_clean":"2024-09-13T14:57:54.545651+0000","last_became_active":"2024-09-13T14:47:47.876297+0000","last_became_peered":"2024-09-13T14:47:47.876297+0000","last_unstale":"2024-09-13T14:57:54.545651+0000","last_undegraded":"2024-09-13T14:57:54.545651+0000","last_fullsized":"2024-09-13T14:57:54.545651+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:03:27.761394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":1045,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T14:57:36.048709+0000","last_change":"2024-09-13T14:55:20.147689+0000","last_active":"2024-09-13T14:57:36.048709+0000","last_peered":"2024-09-13T14:57:36.048709+0000","last_clean":"2024-09-13T14:57:36.048709+0000","last_became_active":"2024-09-13T14:55:20.147241+0000","last_became_peered":"2024-09-13T14:55:20.147241+0000","last_unstale":"2024-09-13T14:57:36.048709+0000","last_undegraded":"2024-09-13T14:57:36.048709+0000","last_fullsized":"2024-09-13T14:57:36.048709+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:43:36.224330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":336,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T14:57:36.047605+0000","last_change":"2024-09-13T14:56:12.491490+0000","last_active":"2024-09-13T14:57:36.047605+0000","last_peered":"2024-09-13T14:57:36.047605+0000","last_clean":"2024-09-13T14:57:36.047605+0000","last_became_active":"2024-09-13T14:56:12.490650+0000","last_became_peered":"2024-09-13T14:56:12.490650+0000","last_unstale":"2024-09-13T14:57:36.047605+0000","last_undegraded":"2024-09-13T14:57:36.047605+0000","last_fullsized":"2024-09-13T14:57:36.047605+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:55:33.313465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"480'11342","reported_seq":12766,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:58:01.732049+0000","last_change":"2024-09-13T14:55:23.955183+0000","last_active":"2024-09-13T14:58:01.732049+0000","last_peered":"2024-09-13T14:58:01.732049+0000","last_clean":"2024-09-13T14:58:01.732049+0000","last_became_active":"2024-09-13T14:55:23.952692+0000","last_became_peered":"2024-09-13T14:55:23.952692+0000","last_unstale":"2024-09-13T14:58:01.732049+0000","last_undegraded":"2024-09-13T14:58:01.732049+0000","last_fullsized":"2024-09-13T14:58:01.732049+0000","mapping_epoch":447,"log_start":"460'3968","ondisk_log_start":"460'3968","created":436,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":7374,"log_dups_size":0,"ondisk_log_size":7374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:48:52.915452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":126877696,"num_objects":1936,"num_object_clones":0,"num_object_copies":5808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11342,"num_write_kb":424896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":659,"num_bytes_recovered":43188224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"90'149","reported_seq":510,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.528568+0000","last_change":"2024-09-13T14:57:37.063675+0000","last_active":"2024-09-13T14:57:54.528568+0000","last_peered":"2024-09-13T14:57:54.528568+0000","last_clean":"2024-09-13T14:57:54.528568+0000","last_became_active":"2024-09-13T14:57:17.385011+0000","last_became_peered":"2024-09-13T14:57:17.385011+0000","last_unstale":"2024-09-13T14:57:54.528568+0000","last_undegraded":"2024-09-13T14:57:54.528568+0000","last_fullsized":"2024-09-13T14:57:54.528568+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:37.063494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:37.063494+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:33.681893+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":893,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.533345+0000","last_change":"2024-09-13T14:38:53.938227+0000","last_active":"2024-09-13T14:57:54.533345+0000","last_peered":"2024-09-13T14:57:54.533345+0000","last_clean":"2024-09-13T14:57:54.533345+0000","last_became_active":"2024-09-13T14:38:53.902867+0000","last_became_peered":"2024-09-13T14:38:53.902867+0000","last_unstale":"2024-09-13T14:57:54.533345+0000","last_undegraded":"2024-09-13T14:57:54.533345+0000","last_fullsized":"2024-09-13T14:57:54.533345+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:16.831756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":806,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.545834+0000","last_change":"2024-09-13T14:55:29.510841+0000","last_active":"2024-09-13T14:57:54.545834+0000","last_peered":"2024-09-13T14:57:54.545834+0000","last_clean":"2024-09-13T14:57:54.545834+0000","last_became_active":"2024-09-13T14:55:29.510252+0000","last_became_peered":"2024-09-13T14:55:29.510252+0000","last_unstale":"2024-09-13T14:57:54.545834+0000","last_undegraded":"2024-09-13T14:57:54.545834+0000","last_fullsized":"2024-09-13T14:57:54.545834+0000","mapping_epoch":452,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":453,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:14:27.135796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":875,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.527676+0000","last_change":"2024-09-13T14:38:53.900107+0000","last_active":"2024-09-13T14:57:54.527676+0000","last_peered":"2024-09-13T14:57:54.527676+0000","last_clean":"2024-09-13T14:57:54.527676+0000","last_became_active":"2024-09-13T14:38:53.899556+0000","last_became_peered":"2024-09-13T14:38:53.899556+0000","last_unstale":"2024-09-13T14:57:54.527676+0000","last_undegraded":"2024-09-13T14:57:54.527676+0000","last_fullsized":"2024-09-13T14:57:54.527676+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:15:47.918163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"480'11308","reported_seq":11409,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:58:03.552412+0000","last_change":"2024-09-13T14:54:58.382957+0000","last_active":"2024-09-13T14:58:03.552412+0000","last_peered":"2024-09-13T14:58:03.552412+0000","last_clean":"2024-09-13T14:58:03.552412+0000","last_became_active":"2024-09-13T14:54:58.382609+0000","last_became_peered":"2024-09-13T14:54:58.382609+0000","last_unstale":"2024-09-13T14:58:03.552412+0000","last_undegraded":"2024-09-13T14:58:03.552412+0000","last_fullsized":"2024-09-13T14:58:03.552412+0000","mapping_epoch":436,"log_start":"464'5864","ondisk_log_start":"464'5864","created":436,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":5444,"log_dups_size":0,"ondisk_log_size":5444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:55:46.466387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124780544,"num_objects":1904,"num_object_clones":0,"num_object_copies":5712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11308,"num_write_kb":422784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"90'149","reported_seq":735,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.531646+0000","last_change":"2024-09-13T14:47:29.999643+0000","last_active":"2024-09-13T14:57:54.531646+0000","last_peered":"2024-09-13T14:57:54.531646+0000","last_clean":"2024-09-13T14:57:54.531646+0000","last_became_active":"2024-09-13T14:47:29.410951+0000","last_became_peered":"2024-09-13T14:47:29.410951+0000","last_unstale":"2024-09-13T14:57:54.531646+0000","last_undegraded":"2024-09-13T14:57:54.531646+0000","last_fullsized":"2024-09-13T14:57:54.531646+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:29.999519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:29.999519+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:45:08.824756+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":259,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.545441+0000","last_change":"2024-09-13T14:56:12.493198+0000","last_active":"2024-09-13T14:57:54.545441+0000","last_peered":"2024-09-13T14:57:54.545441+0000","last_clean":"2024-09-13T14:57:54.545441+0000","last_became_active":"2024-09-13T14:56:12.490423+0000","last_became_peered":"2024-09-13T14:56:12.490423+0000","last_unstale":"2024-09-13T14:57:54.545441+0000","last_undegraded":"2024-09-13T14:57:54.545441+0000","last_fullsized":"2024-09-13T14:57:54.545441+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:29.984031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":286,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T14:57:36.049119+0000","last_change":"2024-09-13T14:46:01.781508+0000","last_active":"2024-09-13T14:57:36.049119+0000","last_peered":"2024-09-13T14:57:36.049119+0000","last_clean":"2024-09-13T14:57:36.049119+0000","last_became_active":"2024-09-13T14:46:01.767096+0000","last_became_peered":"2024-09-13T14:46:01.767096+0000","last_unstale":"2024-09-13T14:57:36.049119+0000","last_undegraded":"2024-09-13T14:57:36.049119+0000","last_fullsized":"2024-09-13T14:57:36.049119+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:20:01.119189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":270,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T14:57:36.047298+0000","last_change":"2024-09-13T14:55:17.932691+0000","last_active":"2024-09-13T14:57:36.047298+0000","last_peered":"2024-09-13T14:57:36.047298+0000","last_clean":"2024-09-13T14:57:36.047298+0000","last_became_active":"2024-09-13T14:55:17.932391+0000","last_became_peered":"2024-09-13T14:55:17.932391+0000","last_unstale":"2024-09-13T14:57:36.047298+0000","last_undegraded":"2024-09-13T14:57:36.047298+0000","last_fullsized":"2024-09-13T14:57:36.047298+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:03:22.734037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.16","version":"90'149","reported_seq":735,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.545628+0000","last_change":"2024-09-13T14:57:40.798149+0000","last_active":"2024-09-13T14:57:54.545628+0000","last_peered":"2024-09-13T14:57:54.545628+0000","last_clean":"2024-09-13T14:57:54.545628+0000","last_became_active":"2024-09-13T14:57:28.528351+0000","last_became_peered":"2024-09-13T14:57:28.528351+0000","last_unstale":"2024-09-13T14:57:54.545628+0000","last_undegraded":"2024-09-13T14:57:54.545628+0000","last_fullsized":"2024-09-13T14:57:54.545628+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":471,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:40.797796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:40.797796+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:09.769513+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":441,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.544788+0000","last_change":"2024-09-13T14:55:17.930074+0000","last_active":"2024-09-13T14:57:54.544788+0000","last_peered":"2024-09-13T14:57:54.544788+0000","last_clean":"2024-09-13T14:57:54.544788+0000","last_became_active":"2024-09-13T14:55:17.928411+0000","last_became_peered":"2024-09-13T14:55:17.928411+0000","last_unstale":"2024-09-13T14:57:54.544788+0000","last_undegraded":"2024-09-13T14:57:54.544788+0000","last_fullsized":"2024-09-13T14:57:54.544788+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:11:35.302734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":117,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T14:57:41.831537+0000","last_change":"2024-09-13T14:55:17.939375+0000","last_active":"2024-09-13T14:57:41.831537+0000","last_peered":"2024-09-13T14:57:41.831537+0000","last_clean":"2024-09-13T14:57:41.831537+0000","last_became_active":"2024-09-13T14:55:17.926420+0000","last_became_peered":"2024-09-13T14:55:17.926420+0000","last_unstale":"2024-09-13T14:57:41.831537+0000","last_undegraded":"2024-09-13T14:57:41.831537+0000","last_fullsized":"2024-09-13T14:57:41.831537+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:38:13.401866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4756,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.17","version":"90'149","reported_seq":759,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T14:57:47.798716+0000","last_change":"2024-09-13T14:57:47.798716+0000","last_active":"2024-09-13T14:57:47.798716+0000","last_peered":"2024-09-13T14:57:47.798716+0000","last_clean":"2024-09-13T14:57:47.798716+0000","last_became_active":"2024-09-13T14:57:28.526594+0000","last_became_peered":"2024-09-13T14:57:28.526594+0000","last_unstale":"2024-09-13T14:57:47.798716+0000","last_undegraded":"2024-09-13T14:57:47.798716+0000","last_fullsized":"2024-09-13T14:57:47.798716+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":471,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:47.798627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:47.798627+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:30:00.799937+0000","scrub_duration":0.0039598259999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":302,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.549464+0000","last_change":"2024-09-13T14:47:47.867935+0000","last_active":"2024-09-13T14:57:54.549464+0000","last_peered":"2024-09-13T14:57:54.549464+0000","last_clean":"2024-09-13T14:57:54.549464+0000","last_became_active":"2024-09-13T14:47:47.867616+0000","last_became_peered":"2024-09-13T14:47:47.867616+0000","last_unstale":"2024-09-13T14:57:54.549464+0000","last_undegraded":"2024-09-13T14:57:54.549464+0000","last_fullsized":"2024-09-13T14:57:54.549464+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:52:38.619916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":265,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T14:57:36.048297+0000","last_change":"2024-09-13T14:55:17.951412+0000","last_active":"2024-09-13T14:57:36.048297+0000","last_peered":"2024-09-13T14:57:36.048297+0000","last_clean":"2024-09-13T14:57:36.048297+0000","last_became_active":"2024-09-13T14:55:17.937147+0000","last_became_peered":"2024-09-13T14:55:17.937147+0000","last_unstale":"2024-09-13T14:57:36.048297+0000","last_undegraded":"2024-09-13T14:57:36.048297+0000","last_fullsized":"2024-09-13T14:57:36.048297+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:43.786806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"90'149","reported_seq":748,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.524499+0000","last_change":"2024-09-13T14:55:20.149252+0000","last_active":"2024-09-13T14:57:54.524499+0000","last_peered":"2024-09-13T14:57:54.524499+0000","last_clean":"2024-09-13T14:57:54.524499+0000","last_became_active":"2024-09-13T14:55:20.148898+0000","last_became_peered":"2024-09-13T14:55:20.148898+0000","last_unstale":"2024-09-13T14:57:54.524499+0000","last_undegraded":"2024-09-13T14:57:54.524499+0000","last_fullsized":"2024-09-13T14:57:54.524499+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:31.013128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:31.013128+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:11.985087+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":312,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.519755+0000","last_change":"2024-09-13T14:47:47.861842+0000","last_active":"2024-09-13T14:57:54.519755+0000","last_peered":"2024-09-13T14:57:54.519755+0000","last_clean":"2024-09-13T14:57:54.519755+0000","last_became_active":"2024-09-13T14:47:47.861492+0000","last_became_peered":"2024-09-13T14:47:47.861492+0000","last_unstale":"2024-09-13T14:57:54.519755+0000","last_undegraded":"2024-09-13T14:57:54.519755+0000","last_fullsized":"2024-09-13T14:57:54.519755+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:17.279533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":816,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.530106+0000","last_change":"2024-09-13T14:38:53.899535+0000","last_active":"2024-09-13T14:57:54.530106+0000","last_peered":"2024-09-13T14:57:54.530106+0000","last_clean":"2024-09-13T14:57:54.530106+0000","last_became_active":"2024-09-13T14:38:53.899183+0000","last_became_peered":"2024-09-13T14:38:53.899183+0000","last_unstale":"2024-09-13T14:57:54.530106+0000","last_undegraded":"2024-09-13T14:57:54.530106+0000","last_fullsized":"2024-09-13T14:57:54.530106+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:33.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"90'149","reported_seq":505,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T14:57:46.923759+0000","last_change":"2024-09-13T14:57:46.923759+0000","last_active":"2024-09-13T14:57:46.923759+0000","last_peered":"2024-09-13T14:57:46.923759+0000","last_clean":"2024-09-13T14:57:46.923759+0000","last_became_active":"2024-09-13T14:57:28.519678+0000","last_became_peered":"2024-09-13T14:57:28.519678+0000","last_unstale":"2024-09-13T14:57:46.923759+0000","last_undegraded":"2024-09-13T14:57:46.923759+0000","last_fullsized":"2024-09-13T14:57:46.923759+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":471,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:46.923638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:46.923638+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:59.709420+0000","scrub_duration":0.002484039,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":360,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T14:57:36.048094+0000","last_change":"2024-09-13T14:56:12.488981+0000","last_active":"2024-09-13T14:57:36.048094+0000","last_peered":"2024-09-13T14:57:36.048094+0000","last_clean":"2024-09-13T14:57:36.048094+0000","last_became_active":"2024-09-13T14:56:12.488552+0000","last_became_peered":"2024-09-13T14:56:12.488552+0000","last_unstale":"2024-09-13T14:57:36.048094+0000","last_undegraded":"2024-09-13T14:57:36.048094+0000","last_fullsized":"2024-09-13T14:57:36.048094+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:02:23.732498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":852,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.527592+0000","last_change":"2024-09-13T14:47:47.868624+0000","last_active":"2024-09-13T14:57:54.527592+0000","last_peered":"2024-09-13T14:57:54.527592+0000","last_clean":"2024-09-13T14:57:54.527592+0000","last_became_active":"2024-09-13T14:47:47.868031+0000","last_became_peered":"2024-09-13T14:47:47.868031+0000","last_unstale":"2024-09-13T14:57:54.527592+0000","last_undegraded":"2024-09-13T14:57:54.527592+0000","last_fullsized":"2024-09-13T14:57:54.527592+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:05:25.365023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1a","version":"90'149","reported_seq":403,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T14:57:43.982775+0000","last_change":"2024-09-13T14:57:43.982775+0000","last_active":"2024-09-13T14:57:43.982775+0000","last_peered":"2024-09-13T14:57:43.982775+0000","last_clean":"2024-09-13T14:57:43.982775+0000","last_became_active":"2024-09-13T14:57:28.517342+0000","last_became_peered":"2024-09-13T14:57:28.517342+0000","last_unstale":"2024-09-13T14:57:43.982775+0000","last_undegraded":"2024-09-13T14:57:43.982775+0000","last_fullsized":"2024-09-13T14:57:43.982775+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":471,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:43.982684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:43.982684+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:22:40.998656+0000","scrub_duration":0.0032555290000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":325,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T14:57:36.048243+0000","last_change":"2024-09-13T14:47:47.859825+0000","last_active":"2024-09-13T14:57:36.048243+0000","last_peered":"2024-09-13T14:57:36.048243+0000","last_clean":"2024-09-13T14:57:36.048243+0000","last_became_active":"2024-09-13T14:47:47.859069+0000","last_became_peered":"2024-09-13T14:47:47.859069+0000","last_unstale":"2024-09-13T14:57:36.048243+0000","last_undegraded":"2024-09-13T14:57:36.048243+0000","last_fullsized":"2024-09-13T14:57:36.048243+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:18:36.452321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":883,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.527753+0000","last_change":"2024-09-13T14:47:47.868945+0000","last_active":"2024-09-13T14:57:54.527753+0000","last_peered":"2024-09-13T14:57:54.527753+0000","last_clean":"2024-09-13T14:57:54.527753+0000","last_became_active":"2024-09-13T14:47:47.868296+0000","last_became_peered":"2024-09-13T14:47:47.868296+0000","last_unstale":"2024-09-13T14:57:54.527753+0000","last_undegraded":"2024-09-13T14:57:54.527753+0000","last_fullsized":"2024-09-13T14:57:54.527753+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:22:24.778693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"90'149","reported_seq":411,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T14:57:42.789225+0000","last_change":"2024-09-13T14:57:42.789225+0000","last_active":"2024-09-13T14:57:42.789225+0000","last_peered":"2024-09-13T14:57:42.789225+0000","last_clean":"2024-09-13T14:57:42.789225+0000","last_became_active":"2024-09-13T14:57:28.527605+0000","last_became_peered":"2024-09-13T14:57:28.527605+0000","last_unstale":"2024-09-13T14:57:42.789225+0000","last_undegraded":"2024-09-13T14:57:42.789225+0000","last_fullsized":"2024-09-13T14:57:42.789225+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":471,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:42.789102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:42.789102+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:00:41.558911+0000","scrub_duration":0.0083861109999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":56,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T14:57:36.048874+0000","last_change":"2024-09-13T14:56:12.491111+0000","last_active":"2024-09-13T14:57:36.048874+0000","last_peered":"2024-09-13T14:57:36.048874+0000","last_clean":"2024-09-13T14:57:36.048874+0000","last_became_active":"2024-09-13T14:56:12.490691+0000","last_became_peered":"2024-09-13T14:56:12.490691+0000","last_unstale":"2024-09-13T14:57:36.048874+0000","last_undegraded":"2024-09-13T14:57:36.048874+0000","last_fullsized":"2024-09-13T14:57:36.048874+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:01.439884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":309,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.545057+0000","last_change":"2024-09-13T14:55:17.929446+0000","last_active":"2024-09-13T14:57:54.545057+0000","last_peered":"2024-09-13T14:57:54.545057+0000","last_clean":"2024-09-13T14:57:54.545057+0000","last_became_active":"2024-09-13T14:55:17.928598+0000","last_became_peered":"2024-09-13T14:55:17.928598+0000","last_unstale":"2024-09-13T14:57:54.545057+0000","last_undegraded":"2024-09-13T14:57:54.545057+0000","last_fullsized":"2024-09-13T14:57:54.545057+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:27.446138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.18","version":"90'149","reported_seq":603,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T14:57:41.793060+0000","last_change":"2024-09-13T14:57:41.793060+0000","last_active":"2024-09-13T14:57:41.793060+0000","last_peered":"2024-09-13T14:57:41.793060+0000","last_clean":"2024-09-13T14:57:41.793060+0000","last_became_active":"2024-09-13T14:57:28.525726+0000","last_became_peered":"2024-09-13T14:57:28.525726+0000","last_unstale":"2024-09-13T14:57:41.793060+0000","last_undegraded":"2024-09-13T14:57:41.793060+0000","last_fullsized":"2024-09-13T14:57:41.793060+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":471,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:41.792934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:41.792934+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:18.870805+0000","scrub_duration":0.0079393269999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":879,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.532469+0000","last_change":"2024-09-13T14:55:17.933267+0000","last_active":"2024-09-13T14:57:54.532469+0000","last_peered":"2024-09-13T14:57:54.532469+0000","last_clean":"2024-09-13T14:57:54.532469+0000","last_became_active":"2024-09-13T14:55:17.932890+0000","last_became_peered":"2024-09-13T14:55:17.932890+0000","last_unstale":"2024-09-13T14:57:54.532469+0000","last_undegraded":"2024-09-13T14:57:54.532469+0000","last_fullsized":"2024-09-13T14:57:54.532469+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:48:59.113533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":494,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.549263+0000","last_change":"2024-09-13T14:55:17.937105+0000","last_active":"2024-09-13T14:57:54.549263+0000","last_peered":"2024-09-13T14:57:54.549263+0000","last_clean":"2024-09-13T14:57:54.549263+0000","last_became_active":"2024-09-13T14:55:17.936621+0000","last_became_peered":"2024-09-13T14:55:17.936621+0000","last_unstale":"2024-09-13T14:57:54.549263+0000","last_undegraded":"2024-09-13T14:57:54.549263+0000","last_fullsized":"2024-09-13T14:57:54.549263+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:37:28.134499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.19","version":"90'149","reported_seq":504,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.529944+0000","last_change":"2024-09-13T14:57:38.050038+0000","last_active":"2024-09-13T14:57:54.529944+0000","last_peered":"2024-09-13T14:57:54.529944+0000","last_clean":"2024-09-13T14:57:54.529944+0000","last_became_active":"2024-09-13T14:57:28.530372+0000","last_became_peered":"2024-09-13T14:57:28.530372+0000","last_unstale":"2024-09-13T14:57:54.529944+0000","last_undegraded":"2024-09-13T14:57:54.529944+0000","last_fullsized":"2024-09-13T14:57:54.529944+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":471,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:38.049970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:38.049970+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:10.878382+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":383,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.524816+0000","last_change":"2024-09-13T14:50:27.510133+0000","last_active":"2024-09-13T14:57:54.524816+0000","last_peered":"2024-09-13T14:57:54.524816+0000","last_clean":"2024-09-13T14:57:54.524816+0000","last_became_active":"2024-09-13T14:50:27.508774+0000","last_became_peered":"2024-09-13T14:50:27.508774+0000","last_unstale":"2024-09-13T14:57:54.524816+0000","last_undegraded":"2024-09-13T14:57:54.524816+0000","last_fullsized":"2024-09-13T14:57:54.524816+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:31:04.531746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":29,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T14:57:36.048314+0000","last_change":"2024-09-13T14:56:12.489663+0000","last_active":"2024-09-13T14:57:36.048314+0000","last_peered":"2024-09-13T14:57:36.048314+0000","last_clean":"2024-09-13T14:57:36.048314+0000","last_became_active":"2024-09-13T14:56:12.489317+0000","last_became_peered":"2024-09-13T14:56:12.489317+0000","last_unstale":"2024-09-13T14:57:36.048314+0000","last_undegraded":"2024-09-13T14:57:36.048314+0000","last_fullsized":"2024-09-13T14:57:36.048314+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:08:44.565862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1e","version":"90'149","reported_seq":415,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.530108+0000","last_change":"2024-09-13T14:57:39.990337+0000","last_active":"2024-09-13T14:57:54.530108+0000","last_peered":"2024-09-13T14:57:54.530108+0000","last_clean":"2024-09-13T14:57:54.530108+0000","last_became_active":"2024-09-13T14:57:28.525160+0000","last_became_peered":"2024-09-13T14:57:28.525160+0000","last_unstale":"2024-09-13T14:57:54.530108+0000","last_undegraded":"2024-09-13T14:57:54.530108+0000","last_fullsized":"2024-09-13T14:57:54.530108+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":471,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:39.990271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:39.990271+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:55.404672+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":438,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.532194+0000","last_change":"2024-09-13T14:55:17.932405+0000","last_active":"2024-09-13T14:57:54.532194+0000","last_peered":"2024-09-13T14:57:54.532194+0000","last_clean":"2024-09-13T14:57:54.532194+0000","last_became_active":"2024-09-13T14:55:17.932058+0000","last_became_peered":"2024-09-13T14:55:17.932058+0000","last_unstale":"2024-09-13T14:57:54.532194+0000","last_undegraded":"2024-09-13T14:57:54.532194+0000","last_fullsized":"2024-09-13T14:57:54.532194+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:05:01.826514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":325,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T14:57:36.047678+0000","last_change":"2024-09-13T14:55:17.946147+0000","last_active":"2024-09-13T14:57:36.047678+0000","last_peered":"2024-09-13T14:57:36.047678+0000","last_clean":"2024-09-13T14:57:36.047678+0000","last_became_active":"2024-09-13T14:55:17.931009+0000","last_became_peered":"2024-09-13T14:55:17.931009+0000","last_unstale":"2024-09-13T14:57:36.047678+0000","last_undegraded":"2024-09-13T14:57:36.047678+0000","last_fullsized":"2024-09-13T14:57:36.047678+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:08:26.717928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":335,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.531238+0000","last_change":"2024-09-13T14:55:17.936194+0000","last_active":"2024-09-13T14:57:54.531238+0000","last_peered":"2024-09-13T14:57:54.531238+0000","last_clean":"2024-09-13T14:57:54.531238+0000","last_became_active":"2024-09-13T14:55:17.935832+0000","last_became_peered":"2024-09-13T14:55:17.935832+0000","last_unstale":"2024-09-13T14:57:54.531238+0000","last_undegraded":"2024-09-13T14:57:54.531238+0000","last_fullsized":"2024-09-13T14:57:54.531238+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:40:52.745439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":312,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T14:57:36.047907+0000","last_change":"2024-09-13T14:55:17.941129+0000","last_active":"2024-09-13T14:57:36.047907+0000","last_peered":"2024-09-13T14:57:36.047907+0000","last_clean":"2024-09-13T14:57:36.047907+0000","last_became_active":"2024-09-13T14:55:17.930922+0000","last_became_peered":"2024-09-13T14:55:17.930922+0000","last_unstale":"2024-09-13T14:57:36.047907+0000","last_undegraded":"2024-09-13T14:57:36.047907+0000","last_fullsized":"2024-09-13T14:57:36.047907+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:06:29.610560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1c","version":"90'149","reported_seq":459,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T14:57:44.980830+0000","last_change":"2024-09-13T14:57:44.980830+0000","last_active":"2024-09-13T14:57:44.980830+0000","last_peered":"2024-09-13T14:57:44.980830+0000","last_clean":"2024-09-13T14:57:44.980830+0000","last_became_active":"2024-09-13T14:57:28.507948+0000","last_became_peered":"2024-09-13T14:57:28.507948+0000","last_unstale":"2024-09-13T14:57:44.980830+0000","last_undegraded":"2024-09-13T14:57:44.980830+0000","last_fullsized":"2024-09-13T14:57:44.980830+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":471,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:44.980740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:44.980740+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:52:20.504558+0000","scrub_duration":0.012428020999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":897,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.530487+0000","last_change":"2024-09-13T14:46:01.766580+0000","last_active":"2024-09-13T14:57:54.530487+0000","last_peered":"2024-09-13T14:57:54.530487+0000","last_clean":"2024-09-13T14:57:54.530487+0000","last_became_active":"2024-09-13T14:46:01.766257+0000","last_became_peered":"2024-09-13T14:46:01.766257+0000","last_unstale":"2024-09-13T14:57:54.530487+0000","last_undegraded":"2024-09-13T14:57:54.530487+0000","last_fullsized":"2024-09-13T14:57:54.530487+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:01.126109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":862,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.530271+0000","last_change":"2024-09-13T14:46:01.769647+0000","last_active":"2024-09-13T14:57:54.530271+0000","last_peered":"2024-09-13T14:57:54.530271+0000","last_clean":"2024-09-13T14:57:54.530271+0000","last_became_active":"2024-09-13T14:46:01.769228+0000","last_became_peered":"2024-09-13T14:46:01.769228+0000","last_unstale":"2024-09-13T14:57:54.530271+0000","last_undegraded":"2024-09-13T14:57:54.530271+0000","last_fullsized":"2024-09-13T14:57:54.530271+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:58.696024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1d","version":"90'149","reported_seq":445,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.544702+0000","last_change":"2024-09-13T14:57:38.761047+0000","last_active":"2024-09-13T14:57:54.544702+0000","last_peered":"2024-09-13T14:57:54.544702+0000","last_clean":"2024-09-13T14:57:54.544702+0000","last_became_active":"2024-09-13T14:57:28.529976+0000","last_became_peered":"2024-09-13T14:57:28.529976+0000","last_unstale":"2024-09-13T14:57:54.544702+0000","last_undegraded":"2024-09-13T14:57:54.544702+0000","last_fullsized":"2024-09-13T14:57:54.544702+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":471,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:38.760963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:38.760963+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:24.902108+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":420,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-13T14:57:54.544678+0000","last_change":"2024-09-13T14:55:17.929205+0000","last_active":"2024-09-13T14:57:54.544678+0000","last_peered":"2024-09-13T14:57:54.544678+0000","last_clean":"2024-09-13T14:57:54.544678+0000","last_became_active":"2024-09-13T14:55:17.928883+0000","last_became_peered":"2024-09-13T14:55:17.928883+0000","last_unstale":"2024-09-13T14:57:54.544678+0000","last_undegraded":"2024-09-13T14:57:54.544678+0000","last_fullsized":"2024-09-13T14:57:54.544678+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:27:31.570502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":383,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-13T14:57:36.048021+0000","last_change":"2024-09-13T14:56:12.492223+0000","last_active":"2024-09-13T14:57:36.048021+0000","last_peered":"2024-09-13T14:57:36.048021+0000","last_clean":"2024-09-13T14:57:36.048021+0000","last_became_active":"2024-09-13T14:56:12.490578+0000","last_became_peered":"2024-09-13T14:56:12.490578+0000","last_unstale":"2024-09-13T14:57:36.048021+0000","last_undegraded":"2024-09-13T14:57:36.048021+0000","last_fullsized":"2024-09-13T14:57:36.048021+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:11:13.623995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":2043871256,"num_objects":31188,"num_object_clones":0,"num_object_copies":93564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31188,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180492,"num_write_kb":6773697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4957,"num_bytes_recovered":324861952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6208237568,"data_stored":6208225352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99443,"ondisk_log_size":99443,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":6474,"num_keys_recovered":30,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":21564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25132,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":171,"num_bytes_recovered":8388646,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4619,"ondisk_log_size":4619,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234570,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1623376,"kb_used_data":663624,"kb_used_omap":1,"kb_used_meta":959742,"kb_avail":92121776,"statfs":{"total":95995035648,"available":94332698624,"internally_reserved":0,"allocated":679550976,"data_stored":677095170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":982776267},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":239,"seq":1026497183988,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1582604,"kb_used_data":920580,"kb_used_omap":8,"kb_used_meta":662007,"kb_avail":92162548,"statfs":{"total":95995035648,"available":94374449152,"internally_reserved":0,"allocated":942673920,"data_stored":940221655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8453,"internal_metadata":677895931},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757835,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2011328,"kb_used_data":781816,"kb_used_omap":1,"kb_used_meta":1229502,"kb_avail":91733824,"statfs":{"total":95995035648,"available":93935435776,"internally_reserved":0,"allocated":800579584,"data_stored":798135861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1259010502},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":128849019479,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435662,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2171324,"kb_used_data":997748,"kb_used_omap":5,"kb_used_meta":1173562,"kb_avail":91573828,"statfs":{"total":95995035648,"available":93771599872,"internally_reserved":0,"allocated":1021693952,"data_stored":1019253567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":1201727614},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":104,"seq":446676599190,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1279628,"kb_used_data":788420,"kb_used_omap":7,"kb_used_meta":491192,"kb_avail":92465524,"statfs":{"total":95995035648,"available":94684696576,"internally_reserved":0,"allocated":807342080,"data_stored":804891113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8010,"internal_metadata":502980790},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":99,"seq":425201762717,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1240888,"kb_used_data":731632,"kb_used_omap":7,"kb_used_meta":509240,"kb_avail":92504264,"statfs":{"total":95995035648,"available":94724366336,"internally_reserved":0,"allocated":749191168,"data_stored":746754533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7759,"internal_metadata":521462193},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926240,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2502852,"kb_used_data":1313980,"kb_used_omap":7,"kb_used_meta":1188856,"kb_avail":91242300,"statfs":{"total":95995035648,"available":93432115200,"internally_reserved":0,"allocated":1345515520,"data_stored":1343067547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":1217388840},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6860,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1332678656,"data_stored":1332674584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":718471168,"data_stored":718471168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":786497536,"data_stored":786497536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1001525248,"data_stored":1001521176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":524288,"data_stored":524288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":788463616,"data_stored":788463616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":926093312,"data_stored":926089240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":653983744,"data_stored":653983744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:58:08.395 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pgp_num 41 2024-09-13T14:58:08.396 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:08.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:08.462 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:08.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:08.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:08 smithi195 ceph-mon[104262]: from='client.46835 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:58:08.546 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:08 smithi102 ceph-mon[110183]: from='client.46835 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:58:08.546 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:08 smithi102 ceph-mon[115269]: from='client.46835 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:58:08.563 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:08.587 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:08.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:08.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:08.690 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:08.737 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:08.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:08.738 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:08.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:08.839 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:08.863 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:08.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:08.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:08.965 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:08.993 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:08.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:08.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:09.095 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:09.117 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:09.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:09.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:09.219 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:09.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:09.267 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:09.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:09.368 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:09.392 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:09.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:09.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:09.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:09 smithi102 ceph-mon[110183]: pgmap v1614: 119 pgs: 119 active+clean; 1.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 1.03k op/s 2024-09-13T14:58:09.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:09 smithi102 ceph-mon[115269]: pgmap v1614: 119 pgs: 119 active+clean; 1.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 1.03k op/s 2024-09-13T14:58:09.494 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:09.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:09 smithi195 ceph-mon[104262]: pgmap v1614: 119 pgs: 119 active+clean; 1.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 1.03k op/s 2024-09-13T14:58:09.539 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:09.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:09.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:09.641 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:09.670 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:09.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:09.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:09.772 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:09.798 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:09.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:09.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:09.900 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:09.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:09.951 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:09.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:10.052 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:10.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:10.093 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:10.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:10.194 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:10.222 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:10.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:10.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:10.324 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:10.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:10.358 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:10.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:10.459 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:10.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:10.507 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:10.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:10.608 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:10.635 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:10.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:10.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:10.737 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:10.780 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:10.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:10.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:10.881 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:10.907 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:10.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:10.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:11.009 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:11.047 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:11.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:11.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:11.149 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:11.171 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:11.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:11.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:11.274 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:11.301 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:11.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:11.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:11.338 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:58:11.339 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:58:11.380 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:58:11.380 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:58:11.404 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:11.424 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:58:11.425 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:58:11.445 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:11.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:11.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:11.460 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:58:11.460 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:58:11.493 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:58:11.493 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:58:11.493 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:11 smithi102 ceph-mon[110183]: pgmap v1615: 119 pgs: 119 active+clean; 1.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 976 op/s 2024-09-13T14:58:11.494 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:11 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:58:11.494 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:11 smithi102 ceph-mon[115269]: pgmap v1615: 119 pgs: 119 active+clean; 1.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 976 op/s 2024-09-13T14:58:11.494 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:11 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:58:11.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:11 smithi195 ceph-mon[104262]: pgmap v1615: 119 pgs: 119 active+clean; 1.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 976 op/s 2024-09-13T14:58:11.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:11 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:58:11.541 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:58:11.541 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:58:11.547 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:11.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:11.573 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:11.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:11.574 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:58:11.574 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:58:11.610 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:58:11.611 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:58:11.674 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:11.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:11.701 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:11.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:11.803 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:11.828 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:11.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:11.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:11.930 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:11.998 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:11.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:11.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:12.037 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:12.100 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:12.151 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:12.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:12.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:12.253 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:12.254 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is set 2024-09-13T14:58:12.291 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:12.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:12.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:12.393 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:12.407 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:12 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/662646862' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:12.407 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:12 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/4228045675' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:58:12.407 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:12 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:58:12.408 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:12 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/662646862' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:12.408 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:12 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/4228045675' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:58:12.408 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:12 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:58:12.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:12.438 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:12.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:12.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:12 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/662646862' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:12.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:12 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/4228045675' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:58:12.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:12 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:58:12.539 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:12.565 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:12.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:12.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:12.667 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:12.696 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:12.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:12.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:12.801 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:12.835 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:12.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:12.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:12.937 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:12.973 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:12.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:12.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:13.076 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:13.105 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:13.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:13.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:13.208 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:13.254 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:13.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:13.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:13.356 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:13.400 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:13.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:13.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:13.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:13 smithi195 ceph-mon[104262]: pgmap v1616: 119 pgs: 119 active+clean; 1.5 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 977 op/s 2024-09-13T14:58:13.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:13 smithi195 ceph-mon[104262]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:58:13.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:13 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:58:13.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:13 smithi195 ceph-mon[104262]: osdmap e481: 8 total, 8 up, 7 in 2024-09-13T14:58:13.501 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:13.526 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:13.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:13.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:13.628 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:13.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:13.659 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:13.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:13.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:13 smithi102 ceph-mon[110183]: pgmap v1616: 119 pgs: 119 active+clean; 1.5 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 977 op/s 2024-09-13T14:58:13.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:13 smithi102 ceph-mon[110183]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:58:13.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:13 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:58:13.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:13 smithi102 ceph-mon[110183]: osdmap e481: 8 total, 8 up, 7 in 2024-09-13T14:58:13.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:13 smithi102 ceph-mon[115269]: pgmap v1616: 119 pgs: 119 active+clean; 1.5 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 977 op/s 2024-09-13T14:58:13.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:13 smithi102 ceph-mon[115269]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:58:13.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:13 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:58:13.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:13 smithi102 ceph-mon[115269]: osdmap e481: 8 total, 8 up, 7 in 2024-09-13T14:58:13.760 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:13.784 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:13.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:13.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:13.887 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:13.921 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:13.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:13.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:14.023 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:14.056 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:14.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:14.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:14.158 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:14.180 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:14.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:14.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:14.282 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:14.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:14.334 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:14.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:14.435 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:14.474 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:14.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:14.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:14.575 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:14.602 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:14.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:14.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:14.704 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:14.730 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:14.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:14.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:14.832 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:14.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:14.857 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:14.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:14.958 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:14.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:14.983 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:14.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:15.008 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:58:15.008 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [ 2024-09-13T14:58:15.008 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.008 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.36738.0:192258 10.b 10:da772957:::benchmark_data_smithi102_377304_object86416:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e481)", 2024-09-13T14:58:15.009 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:58:14.988090+0000", 2024-09-13T14:58:15.009 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.018006271000000001, 2024-09-13T14:58:15.009 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.018032135000000001, 2024-09-13T14:58:15.009 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:58:15.009 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:58:15.009 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:58:15.009 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:58:15.009 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.36738", 2024-09-13T14:58:15.009 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/2915890700", 2024-09-13T14:58:15.009 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 192258 2024-09-13T14:58:15.009 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.009 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:58:15.009 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.010 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:58:15.010 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:14.988090+0000", 2024-09-13T14:58:15.010 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:58:15.010 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.010 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.010 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:58:15.010 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:14.988090+0000", 2024-09-13T14:58:15.010 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:58:15.010 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.010 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.010 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:58:15.010 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:14.988092+0000", 2024-09-13T14:58:15.011 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.57e-06 2024-09-13T14:58:15.011 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.011 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.011 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:58:15.011 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:14.988097+0000", 2024-09-13T14:58:15.011 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.3360000000000002e-06 2024-09-13T14:58:15.011 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.011 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.011 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:58:15.011 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:14.988113+0000", 2024-09-13T14:58:15.011 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.6563999999999999e-05 2024-09-13T14:58:15.011 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.012 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.012 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:58:15.012 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:14.988121+0000", 2024-09-13T14:58:15.012 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.7300000000000005e-06 2024-09-13T14:58:15.012 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.012 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.012 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:58:15.012 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:14.990521+0000", 2024-09-13T14:58:15.012 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0023998019999999999 2024-09-13T14:58:15.012 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.012 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.012 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:58:15.012 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:14.990625+0000", 2024-09-13T14:58:15.013 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00010396 2024-09-13T14:58:15.013 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.013 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.013 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 0,6", 2024-09-13T14:58:15.013 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:14.990727+0000", 2024-09-13T14:58:15.013 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000101656 2024-09-13T14:58:15.013 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.013 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.013 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:15.013 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:14.996086+0000", 2024-09-13T14:58:15.013 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0053596750000000004 2024-09-13T14:58:15.013 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.014 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.014 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:15.014 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:14.996483+0000", 2024-09-13T14:58:15.014 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00039629200000000002 2024-09-13T14:58:15.014 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:58:15.014 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:58:15.014 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:58:15.014 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.014 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.014 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.36738.0:192252 10.4 e481/462 10:2359f4c0:::benchmark_data_smithi102_377304_object86410:head v 481'12022, mlcod=481'12020)", 2024-09-13T14:58:15.014 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:58:14.988651+0000", 2024-09-13T14:58:15.014 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.017444905, 2024-09-13T14:58:15.014 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.017545104999999998, 2024-09-13T14:58:15.015 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:58:15.015 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:58:15.015 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "started", 2024-09-13T14:58:15.015 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:58:15.015 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.015 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:58:15.015 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:14.988651+0000", 2024-09-13T14:58:15.015 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:58:15.015 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.015 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.015 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:58:15.015 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:14.988651+0000", 2024-09-13T14:58:15.016 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:58:15.016 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.016 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.016 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:58:15.016 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:14.988652+0000", 2024-09-13T14:58:15.016 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.8500000000000001e-07 2024-09-13T14:58:15.016 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.016 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.016 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:58:15.016 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:14.988657+0000", 2024-09-13T14:58:15.016 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.0900000000000004e-06 2024-09-13T14:58:15.016 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.016 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.017 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:58:15.017 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:14.988666+0000", 2024-09-13T14:58:15.017 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.1409999999999994e-06 2024-09-13T14:58:15.017 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.017 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.017 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:58:15.017 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:14.988672+0000", 2024-09-13T14:58:15.017 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.0569999999999998e-06 2024-09-13T14:58:15.017 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.017 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.017 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:58:15.017 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:14.989043+0000", 2024-09-13T14:58:15.018 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00037090000000000002 2024-09-13T14:58:15.018 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.018 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.018 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:58:15.018 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:14.989062+0000", 2024-09-13T14:58:15.018 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.8374000000000001e-05 2024-09-13T14:58:15.018 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:58:15.018 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:58:15.018 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:58:15.018 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.018 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.018 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.36738.0:192266 10.2 10:4ed9489c:::benchmark_data_smithi102_377304_object86424:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e481)", 2024-09-13T14:58:15.018 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:58:14.997304+0000", 2024-09-13T14:58:15.019 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.0087924930000000002, 2024-09-13T14:58:15.019 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0089421259999999999, 2024-09-13T14:58:15.019 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:58:15.019 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:58:15.019 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:58:15.019 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:58:15.019 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.36738", 2024-09-13T14:58:15.019 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/2915890700", 2024-09-13T14:58:15.019 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 192266 2024-09-13T14:58:15.019 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.019 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:58:15.019 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.019 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:58:15.020 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:14.997304+0000", 2024-09-13T14:58:15.020 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:58:15.020 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.020 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.020 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:58:15.020 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:14.997304+0000", 2024-09-13T14:58:15.020 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:58:15.020 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.020 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.020 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:58:15.020 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:14.997305+0000", 2024-09-13T14:58:15.020 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.677e-06 2024-09-13T14:58:15.021 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.021 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.021 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:58:15.021 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:14.997310+0000", 2024-09-13T14:58:15.021 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.87e-06 2024-09-13T14:58:15.021 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.021 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.021 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:58:15.021 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:14.997323+0000", 2024-09-13T14:58:15.021 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2359e-05 2024-09-13T14:58:15.021 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.021 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.022 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:58:15.022 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:14.997330+0000", 2024-09-13T14:58:15.022 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.9949999999999999e-06 2024-09-13T14:58:15.022 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.022 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.022 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:58:15.022 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:14.997377+0000", 2024-09-13T14:58:15.022 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.7524000000000001e-05 2024-09-13T14:58:15.022 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.022 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.022 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:58:15.022 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:14.997448+0000", 2024-09-13T14:58:15.023 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.0945999999999995e-05 2024-09-13T14:58:15.023 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.023 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.023 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 0,3", 2024-09-13T14:58:15.023 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:14.997513+0000", 2024-09-13T14:58:15.023 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.5191999999999995e-05 2024-09-13T14:58:15.023 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.023 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.023 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:15.023 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:15.001341+0000", 2024-09-13T14:58:15.023 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.003827235 2024-09-13T14:58:15.023 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:58:15.023 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:58:15.024 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:58:15.024 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.024 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.024 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.36738.0:192254 10.4 e481/462 10:26452430:::benchmark_data_smithi102_377304_object86412:head v 481'12023, mlcod=481'12020)", 2024-09-13T14:58:15.024 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:58:14.997358+0000", 2024-09-13T14:58:15.024 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.0087381899999999998, 2024-09-13T14:58:15.025 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.008956393, 2024-09-13T14:58:15.025 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:58:15.025 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:58:15.025 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "started", 2024-09-13T14:58:15.025 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:58:15.025 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.025 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:58:15.025 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:14.997358+0000", 2024-09-13T14:58:15.025 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:58:15.025 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.025 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.025 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:58:15.025 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:14.997358+0000", 2024-09-13T14:58:15.026 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:58:15.026 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.026 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.026 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:58:15.026 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:14.997359+0000", 2024-09-13T14:58:15.026 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.06e-07 2024-09-13T14:58:15.026 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.026 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.026 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:58:15.026 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:14.997363+0000", 2024-09-13T14:58:15.026 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.9269999999999998e-06 2024-09-13T14:58:15.026 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.026 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.027 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:58:15.027 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:14.997369+0000", 2024-09-13T14:58:15.027 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.584e-06 2024-09-13T14:58:15.027 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.027 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.027 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:58:15.027 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:14.997374+0000", 2024-09-13T14:58:15.027 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.6990000000000002e-06 2024-09-13T14:58:15.027 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.027 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.027 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:58:15.027 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:14.997455+0000", 2024-09-13T14:58:15.027 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.1038999999999994e-05 2024-09-13T14:58:15.028 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.028 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.028 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:58:15.028 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:14.997472+0000", 2024-09-13T14:58:15.028 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.7651000000000001e-05 2024-09-13T14:58:15.028 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:58:15.028 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:58:15.028 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:58:15.028 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.028 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.028 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.36738.0:192261 10.4 e481/462 10:274ebe2b:::benchmark_data_smithi102_377304_object86419:head v 481'12024, mlcod=481'12020)", 2024-09-13T14:58:15.028 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:58:14.997388+0000", 2024-09-13T14:58:15.028 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.0087080969999999997, 2024-09-13T14:58:15.029 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0089715330000000003, 2024-09-13T14:58:15.029 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:58:15.029 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:58:15.029 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "started", 2024-09-13T14:58:15.029 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:58:15.029 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.029 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:58:15.029 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:14.997388+0000", 2024-09-13T14:58:15.029 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:58:15.029 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.029 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.029 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:58:15.029 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:14.997388+0000", 2024-09-13T14:58:15.030 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:58:15.030 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.030 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.030 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:58:15.030 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:14.997388+0000", 2024-09-13T14:58:15.030 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.2000000000000001e-07 2024-09-13T14:58:15.030 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.030 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.030 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:58:15.030 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:14.997391+0000", 2024-09-13T14:58:15.030 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.8600000000000001e-06 2024-09-13T14:58:15.031 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.031 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.031 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:58:15.031 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:14.997396+0000", 2024-09-13T14:58:15.031 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.0660000000000003e-06 2024-09-13T14:58:15.031 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.031 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.031 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:58:15.031 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:14.997400+0000", 2024-09-13T14:58:15.031 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.6899999999999998e-06 2024-09-13T14:58:15.031 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.031 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.031 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:58:15.032 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:15.002564+0000", 2024-09-13T14:58:15.032 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.005163819 2024-09-13T14:58:15.032 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.032 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.032 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:58:15.032 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:15.002583+0000", 2024-09-13T14:58:15.032 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.8984e-05 2024-09-13T14:58:15.032 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:58:15.032 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:58:15.032 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:58:15.032 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.032 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.032 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.36738.0:192264 10.4 e481/462 10:24a948f9:::benchmark_data_smithi102_377304_object86422:head v 481'12025, mlcod=481'12020)", 2024-09-13T14:58:15.033 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:58:15.001532+0000", 2024-09-13T14:58:15.033 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 0.0045644329999999997, 2024-09-13T14:58:15.033 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.004872983, 2024-09-13T14:58:15.033 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:58:15.033 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:58:15.033 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "started", 2024-09-13T14:58:15.033 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:58:15.033 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.033 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:58:15.033 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:15.001532+0000", 2024-09-13T14:58:15.033 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:58:15.033 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.033 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.033 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:58:15.034 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:15.001532+0000", 2024-09-13T14:58:15.034 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:58:15.034 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.034 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.034 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:58:15.034 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:15.001533+0000", 2024-09-13T14:58:15.034 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.6000000000000003e-07 2024-09-13T14:58:15.034 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.034 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.034 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:58:15.034 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:15.001538+0000", 2024-09-13T14:58:15.034 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.7429999999999999e-06 2024-09-13T14:58:15.034 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.034 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.035 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:58:15.035 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:15.001550+0000", 2024-09-13T14:58:15.035 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1557e-05 2024-09-13T14:58:15.035 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.035 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.035 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:58:15.035 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:15.001556+0000", 2024-09-13T14:58:15.035 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.3539999999999996e-06 2024-09-13T14:58:15.035 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.035 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.035 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:58:15.035 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:15.005883+0000", 2024-09-13T14:58:15.035 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0043272140000000002 2024-09-13T14:58:15.035 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:58:15.036 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:58:15.036 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:58:15.036 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:15.005901+0000", 2024-09-13T14:58:15.036 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.7192000000000001e-05 2024-09-13T14:58:15.036 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:58:15.036 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:58:15.036 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:58:15.036 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:58:15.036 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-09-13T14:58:15.036 INFO:teuthology.orchestra.run.smithi102.stdout: "num_ops": 6 2024-09-13T14:58:15.036 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:58:15.084 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:15.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:15.112 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:15.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:15.213 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:15.220 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:58:15.237 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:15.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:15.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:15.339 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:15.365 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:15.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:15.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:15.467 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:15.494 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:15.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:15.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:15.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:15 smithi195 ceph-mon[104262]: pgmap v1618: 119 pgs: 119 active+clean; 1.4 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 966 op/s 2024-09-13T14:58:15.596 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:15.622 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:15.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:15.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:15.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:15 smithi102 ceph-mon[110183]: pgmap v1618: 119 pgs: 119 active+clean; 1.4 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 966 op/s 2024-09-13T14:58:15.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:15 smithi102 ceph-mon[115269]: pgmap v1618: 119 pgs: 119 active+clean; 1.4 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 966 op/s 2024-09-13T14:58:15.707 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:58:15.723 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:15.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:15.813 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:15.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:15.914 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:15.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:15.967 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:15.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:16.068 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:16.093 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:16.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:16.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:16.195 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:16.199 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T14:58:16.219 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:16.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:16.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:16.321 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:16.366 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:16.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:16.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:16.468 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:16.495 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:16.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:16.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:16.597 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:16.641 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:16.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:16.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:16.716 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:16.743 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:16.753 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:16.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:16.820 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:16.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:16.861 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:58:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:58:16] "GET /metrics HTTP/1.1" 200 36565 "" "Prometheus/2.51.0" 2024-09-13T14:58:16.921 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:16.942 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:16.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:16.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:17.044 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:17.078 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:17.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:17.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:17.180 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:17.201 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:17.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:17.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:17.303 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:17.337 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:17.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:17.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:17.439 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:17.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:17.466 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:17.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:17.567 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:17.609 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:17.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:17.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:17.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:17 smithi102 ceph-mon[110183]: pgmap v1619: 119 pgs: 119 active+clean; 1.3 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 958 op/s 2024-09-13T14:58:17.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:17 smithi102 ceph-mon[115269]: pgmap v1619: 119 pgs: 119 active+clean; 1.3 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 958 op/s 2024-09-13T14:58:17.712 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:17.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:17.738 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:17.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:17.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:17 smithi195 ceph-mon[104262]: pgmap v1619: 119 pgs: 119 active+clean; 1.3 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 958 op/s 2024-09-13T14:58:17.839 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:17.886 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:17.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:17.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:17.988 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:18.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:18.015 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:18.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:18.116 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:18.148 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:18.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:18.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:18.251 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:18.291 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:18.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:18.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:18.393 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:18.435 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:18.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:18.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:18.537 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:18.563 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:18.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:18.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:18.665 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:18.690 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:18.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:18.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:18.747 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:18.793 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:18.817 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:18.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:18.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:18.919 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:18.971 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:18.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:18.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:19.073 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:19.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:19.101 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:19.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:19.202 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:19.225 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:19.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:19.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:19.327 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:19.351 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:19.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:19.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:19.453 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:19.491 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:19.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:19.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:19.521 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:19 smithi102 ceph-mon[110183]: pgmap v1620: 119 pgs: 119 active+clean; 1.2 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 956 op/s 2024-09-13T14:58:19.521 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:19 smithi102 ceph-mon[115269]: pgmap v1620: 119 pgs: 119 active+clean; 1.2 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 956 op/s 2024-09-13T14:58:19.593 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:19.626 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:19.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:19.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:19.728 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:19.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:19 smithi195 ceph-mon[104262]: pgmap v1620: 119 pgs: 119 active+clean; 1.2 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 956 op/s 2024-09-13T14:58:19.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:19.754 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:19.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:19.855 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:19.882 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:19.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:19.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:19.984 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:20.012 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:20.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:20.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:20.113 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:20.155 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:20.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:20.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:20.257 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:20.296 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:20.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:20.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:20.398 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:20.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:20.438 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:20.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:20.539 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:20.580 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:20.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:20.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:20.682 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:20.709 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:20.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:20.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:20.811 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:20.846 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:20.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:20.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:20.948 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:20.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:20.999 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:20.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:21.100 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:21.144 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:21.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:21.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:21.246 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:21.288 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:21.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:21.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:21.390 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:21.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:21.415 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:21.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:21.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:21 smithi102 ceph-mon[110183]: pgmap v1621: 119 pgs: 119 active+clean; 1.0 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 955 op/s 2024-09-13T14:58:21.420 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:21 smithi102 ceph-mon[115269]: pgmap v1621: 119 pgs: 119 active+clean; 1.0 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 955 op/s 2024-09-13T14:58:21.516 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:21.554 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:21.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:21.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:21.655 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:21.681 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:21.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:21.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:21.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:21 smithi195 ceph-mon[104262]: pgmap v1621: 119 pgs: 119 active+clean; 1.0 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 955 op/s 2024-09-13T14:58:21.783 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:21.806 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:21.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:21.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:21.908 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:21.931 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:21.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:21.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:22.033 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:22.079 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:22.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:22.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:22.181 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:22.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:22.203 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:22.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:22.304 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:22.332 INFO:teuthology.orchestra.run.smithi102.stderr:Error EINVAL: specified pgp_num 41 > pg_num 31 2024-09-13T14:58:22.354 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:22.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:22.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:22.456 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:22.480 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:22.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:22.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:22.582 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:22.621 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:22.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:22.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:22.723 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:22.746 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:22.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:22.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:22.848 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:22.887 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:22.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:22.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:22.989 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:23.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:23.024 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:23.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:23.125 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:23.149 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:23.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:23.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:23.251 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:23.275 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is set 2024-09-13T14:58:23.286 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:23.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:23.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:23.388 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:23.413 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:23.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:23.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:23.514 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:23.530 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:23 smithi102 ceph-mon[110183]: pgmap v1622: 119 pgs: 119 active+clean; 929 MiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 954 op/s 2024-09-13T14:58:23.531 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:23 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3651436566' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "41"} : dispatch 2024-09-13T14:58:23.531 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:23 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/4108730978' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:58:23.531 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:23 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:58:23.531 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:23 smithi102 ceph-mon[115269]: pgmap v1622: 119 pgs: 119 active+clean; 929 MiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 954 op/s 2024-09-13T14:58:23.531 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:23 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3651436566' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "41"} : dispatch 2024-09-13T14:58:23.531 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:23 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/4108730978' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:58:23.531 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:23 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:58:23.548 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:23 smithi195 ceph-mon[104262]: pgmap v1622: 119 pgs: 119 active+clean; 929 MiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 954 op/s 2024-09-13T14:58:23.548 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:23 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3651436566' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "41"} : dispatch 2024-09-13T14:58:23.549 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:23 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/4108730978' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:58:23.549 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:23 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:58:23.571 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:23.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:23.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:23.673 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:23.697 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:23.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:23.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:23.799 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:23.838 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:23.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:23.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:23.940 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:23.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:23.975 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:23.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:24.076 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:24.122 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:24.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:24.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:24.225 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:24.251 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:24.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:24.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:24.353 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:24.399 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:24.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:24.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:24.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:24 smithi102 ceph-mon[110183]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:58:24.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:24 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:58:24.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:24 smithi102 ceph-mon[110183]: osdmap e482: 8 total, 8 up, 7 in 2024-09-13T14:58:24.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:24 smithi102 ceph-mon[115269]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:58:24.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:24 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:58:24.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:24 smithi102 ceph-mon[115269]: osdmap e482: 8 total, 8 up, 7 in 2024-09-13T14:58:24.501 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:24.535 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:24.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:24.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:24.637 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:24.661 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:24.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:24.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:24.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:24 smithi195 ceph-mon[104262]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:58:24.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:24 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:58:24.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:24 smithi195 ceph-mon[104262]: osdmap e482: 8 total, 8 up, 7 in 2024-09-13T14:58:24.763 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:24.796 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:24.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:24.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:24.898 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:24.924 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:24.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:24.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:25.026 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:25.065 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:25.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:25.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:25.166 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:25.206 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:25.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:25.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:25.308 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:25.332 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:25.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:25.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:25.434 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:25.477 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:25.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:25.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:25.557 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:25 smithi102 ceph-mon[110183]: pgmap v1624: 119 pgs: 119 active+clean; 819 MiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 944 op/s 2024-09-13T14:58:25.557 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:25 smithi102 ceph-mon[115269]: pgmap v1624: 119 pgs: 119 active+clean; 819 MiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 944 op/s 2024-09-13T14:58:25.578 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:25.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:25.626 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:25.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:25.728 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:25.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:25 smithi195 ceph-mon[104262]: pgmap v1624: 119 pgs: 119 active+clean; 819 MiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 944 op/s 2024-09-13T14:58:25.755 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:25.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:25.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:25.857 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:25.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:25.896 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:25.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:25.909 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:58:25.910 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:58:25.946 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:58:25.947 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:58:25.978 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:58:25.978 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:58:25.998 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:26.017 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:58:26.017 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:58:26.023 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:26.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:26.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:26.048 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:58:26.048 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:58:26.082 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:58:26.082 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:58:26.112 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:58:26.112 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:58:26.124 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:26.142 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:58:26.143 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:58:26.150 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:26.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:26.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:26.252 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:26.292 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:26.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:26.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:26.394 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:26.407 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T14:58:26.436 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:26.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:26.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:26.538 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:26.570 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:26.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:26.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:26.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:58:26.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:26 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:58:26.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:26 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1385162719' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:26.653 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:58:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:58:26] "GET /metrics HTTP/1.1" 200 36568 "" "Prometheus/2.51.0" 2024-09-13T14:58:26.654 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:58:26.654 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:26 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:58:26.654 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:26 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1385162719' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:26.672 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:26.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:26.707 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:26.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:26.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:58:26.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:26 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:58:26.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:26 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1385162719' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:26.807 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:26.840 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:26.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:26.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:26.942 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:26.953 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:58:26.953 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:58:26.966 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:26.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:26.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:27.068 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:27.106 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:27.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:27.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:27.208 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:27.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:27.244 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:27.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:27.346 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:27.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:27.371 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:27.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:27.473 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:27.508 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:27.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:27.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:27.594 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:58:27.610 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:27.681 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:27.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:27.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:27.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:27 smithi195 ceph-mon[104262]: pgmap v1625: 119 pgs: 119 active+clean; 769 MiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 936 op/s 2024-09-13T14:58:27.783 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:27.805 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:27.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:27.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:27.907 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:27.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:27 smithi102 ceph-mon[110183]: pgmap v1625: 119 pgs: 119 active+clean; 769 MiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 936 op/s 2024-09-13T14:58:27.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:27 smithi102 ceph-mon[115269]: pgmap v1625: 119 pgs: 119 active+clean; 769 MiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 936 op/s 2024-09-13T14:58:27.930 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:27.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:27.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:28.032 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:28.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:28.069 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:28.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:28.170 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:28.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:28.203 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:28.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:28.304 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:28.330 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:28.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:28.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:28.432 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:28.464 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:28.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:28.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:28.566 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:28.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:28.608 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:28.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:28.709 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:28.734 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:28.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:28.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:28.835 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:28.862 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:28.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:28.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:28.964 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:28.986 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:28.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:28.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:29.088 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:29.112 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:29.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:29.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:29.215 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:29.242 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:29.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:29.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:29.344 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:29.369 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:29.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:29.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:29.471 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:29.496 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:29.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:29.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:29.598 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:29.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:29.629 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:29.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:29.730 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:29.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:29 smithi195 ceph-mon[104262]: pgmap v1626: 119 pgs: 119 active+clean; 614 MiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 916 op/s 2024-09-13T14:58:29.756 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:29.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:29.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:29.857 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:29.884 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:29.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:29.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:29.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:29 smithi102 ceph-mon[110183]: pgmap v1626: 119 pgs: 119 active+clean; 614 MiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 916 op/s 2024-09-13T14:58:29.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:29 smithi102 ceph-mon[115269]: pgmap v1626: 119 pgs: 119 active+clean; 614 MiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 916 op/s 2024-09-13T14:58:29.986 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:30.023 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:30.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:30.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:30.125 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:30.160 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:30.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:30.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:30.262 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:30.296 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:30.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:30.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:30.398 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:30.430 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:30.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:30.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:30.532 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:30.572 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:30.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:30.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:30.674 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:30.698 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:30.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:30.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:30.800 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:30.838 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:30.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:30.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:30.940 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:30.967 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:30.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:30.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:31.069 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:31.099 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:31.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:31.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:31.202 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:31.241 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:31.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:31.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:31.344 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T14:58:31.345 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:31.397 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:31.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:31.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:31.409 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 6, 7, 5, 2, 1] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T14:58:31.409 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:58:31.409 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-13T14:58:31.409 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:58:31.448 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:31.499 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:31.569 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:31.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:31.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:31.613 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:31 smithi102 ceph-mon[110183]: pgmap v1627: 119 pgs: 119 active+clean; 484 MiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 918 op/s 2024-09-13T14:58:31.613 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:31 smithi102 ceph-mon[115269]: pgmap v1627: 119 pgs: 119 active+clean; 484 MiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 918 op/s 2024-09-13T14:58:31.672 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:31.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:31.696 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:31.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:31.797 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:31.865 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:31.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:31.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:31.971 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:31.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:31.995 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:31.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:31.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:31 smithi195 ceph-mon[104262]: pgmap v1627: 119 pgs: 119 active+clean; 484 MiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 918 op/s 2024-09-13T14:58:32.097 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:32.120 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:32.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:32.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:32.222 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:32.263 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:32.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:32.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:32.365 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:32.380 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:32.386 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:32.417 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:32.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:32.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:32.519 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:32.541 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:32.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:32.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:32.548 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:58:32.643 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:32.668 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:32.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:32.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:32.770 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:32.795 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:32.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:32.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:32.897 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:32.923 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:32.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:32.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:33.025 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:33.054 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:33.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:33.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:33.157 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:33.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:33.204 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:33.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:33.305 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:33.334 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:33.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:33.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:33.436 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:33.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:33.476 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:33.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:33.577 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:33.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:33.601 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:33.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:33.703 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:33.743 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:33.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:33.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:33.845 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:33.888 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:33.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:33.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:33.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:33 smithi102 ceph-mon[110183]: pgmap v1628: 119 pgs: 119 active+clean; 355 MiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 918 op/s 2024-09-13T14:58:33.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:33 smithi102 ceph-mon[115269]: pgmap v1628: 119 pgs: 119 active+clean; 355 MiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 918 op/s 2024-09-13T14:58:33.991 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:33.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:33 smithi195 ceph-mon[104262]: pgmap v1628: 119 pgs: 119 active+clean; 355 MiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 918 op/s 2024-09-13T14:58:34.017 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:34.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:34.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:34.119 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:34.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:34.173 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:34.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:34.274 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:34.315 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:34.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:34.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:34.418 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:34.456 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:34.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:34.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:34.558 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:34.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:34.603 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:34.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:34.704 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:34.727 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:34.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:34.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:34.830 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:34.855 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:34.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:34.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:34.956 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:34.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:34.998 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:34.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:35.099 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:35.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:35.148 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:35.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:35.250 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:35.277 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:35.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:35.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:35.379 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:35.420 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:35.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:35.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:35.522 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:35.547 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:35.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:35.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:35.649 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:35.691 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:35.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:35.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:35.750 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Removed 105839 objects 2024-09-13T14:58:35.750 INFO:tasks.radosbench.radosbench.0.smithi102.stdout:Clean up completed and total clean up time :111.611 2024-09-13T14:58:35.777 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:35 smithi102 ceph-mon[110183]: pgmap v1629: 119 pgs: 119 active+clean; 241 MiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 984 op/s 2024-09-13T14:58:35.778 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:35 smithi102 ceph-mon[115269]: pgmap v1629: 119 pgs: 119 active+clean; 241 MiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 984 op/s 2024-09-13T14:58:35.793 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:35.814 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:35.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:35.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:35.916 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:35.938 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:35.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:35.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:35.992 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:35.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:35 smithi195 ceph-mon[104262]: pgmap v1629: 119 pgs: 119 active+clean; 241 MiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 984 op/s 2024-09-13T14:58:36.040 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:36.060 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:36.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:36.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:36.162 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:36.183 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:36.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:36.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:36.285 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:36.307 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:36.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:36.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:36.409 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:36.430 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:36.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:36.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:36.464 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:58:36.464 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:58:36.517 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is unset 2024-09-13T14:58:36.532 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:36.554 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:36.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:36.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:36.656 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:36.678 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:36.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:36.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:36.708 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:36 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/4221176957' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:58:36.708 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:36 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:58:36.708 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:36 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/4221176957' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:58:36.708 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:36 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:58:36.709 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:58:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:58:36] "GET /metrics HTTP/1.1" 200 36568 "" "Prometheus/2.51.0" 2024-09-13T14:58:36.779 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:36.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:36.803 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:36.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:36.904 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:36.927 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:36.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:36.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:36.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:36 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/4221176957' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:58:36.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:36 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:58:37.029 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:37.050 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:37.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:37.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:37.152 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:37.173 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:37.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:37.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:37.275 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:37.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:37.297 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:37.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:37.398 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:37.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:37.421 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:37.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:37.523 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:37.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:37.549 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:37.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:37.650 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:37.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:37 smithi102 ceph-mon[110183]: pgmap v1630: 119 pgs: 119 active+clean; 184 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 846 op/s 2024-09-13T14:58:37.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:37 smithi102 ceph-mon[110183]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:58:37.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:37 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:58:37.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:37 smithi102 ceph-mon[110183]: osdmap e483: 8 total, 8 up, 7 in 2024-09-13T14:58:37.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:37 smithi102 ceph-mon[115269]: pgmap v1630: 119 pgs: 119 active+clean; 184 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 846 op/s 2024-09-13T14:58:37.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:37 smithi102 ceph-mon[115269]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:58:37.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:37 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:58:37.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:37 smithi102 ceph-mon[115269]: osdmap e483: 8 total, 8 up, 7 in 2024-09-13T14:58:37.676 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:37.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:37.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:37.777 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:37.795 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T14:58:37.802 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:37.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:37.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:37.904 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:37.930 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:37.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:37.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:37.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:37 smithi195 ceph-mon[104262]: pgmap v1630: 119 pgs: 119 active+clean; 184 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 846 op/s 2024-09-13T14:58:37.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:37 smithi195 ceph-mon[104262]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:58:37.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:37 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:58:37.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:37 smithi195 ceph-mon[104262]: osdmap e483: 8 total, 8 up, 7 in 2024-09-13T14:58:38.031 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:38.056 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:38.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:38.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:38.158 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:38.182 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:38.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:38.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:38.284 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:38.306 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:38.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:38.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:38.408 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:38.429 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:38.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:38.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:38.531 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:38.558 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:38.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:38.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:38.660 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:38.682 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:38.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:38.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:38.784 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:38.810 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:38.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:38.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:38.912 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:38.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:38.939 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:38.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:39.040 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:39.065 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:39.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:39.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:39.167 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:39.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:39.191 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:39.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:39.292 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:39.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:39.314 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:39.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:39.416 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:39.427 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:58:39.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:39.437 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:39.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:39.539 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:39.560 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:39.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:39.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:39.664 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:39.685 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:39.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:39.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:39.787 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:39.808 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:39.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:39.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:39.910 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:39.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:39 smithi102 ceph-mon[110183]: pgmap v1632: 119 pgs: 119 active+clean; 48 MiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 905 op/s 2024-09-13T14:58:39.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:39 smithi102 ceph-mon[115269]: pgmap v1632: 119 pgs: 119 active+clean; 48 MiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 905 op/s 2024-09-13T14:58:39.937 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:39.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:39.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:39.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:39 smithi195 ceph-mon[104262]: pgmap v1632: 119 pgs: 119 active+clean; 48 MiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 905 op/s 2024-09-13T14:58:40.039 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:40.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:40.062 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:40.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:40.163 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:40.189 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:40.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:40.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:40.291 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:40.315 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:40.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:40.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:40.417 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:40.443 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:40.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:40.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:40.545 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:40.568 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:40.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:40.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:40.670 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:40.696 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:40.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:40.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:40.798 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:40.824 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:40.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:40.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:40.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:40 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:58:40.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:40 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:58:40.925 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:40.951 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:40.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:40.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:40.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:40 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:58:41.053 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:41.079 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:41.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:41.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:41.181 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:41.207 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:41.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:41.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:41.308 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:41.334 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:41.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:41.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:41.436 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:41.460 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:41.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:41.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:41.562 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:41.587 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:41.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:41.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:41.689 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:41.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:41.717 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:41.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:41.818 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:41.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:41.841 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:41.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:41.872 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:41 smithi102 ceph-mon[110183]: pgmap v1633: 119 pgs: 119 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 767 op/s 2024-09-13T14:58:41.873 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:41 smithi102 ceph-mon[115269]: pgmap v1633: 119 pgs: 119 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 767 op/s 2024-09-13T14:58:41.942 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:41.966 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:41.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:41.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:41.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:41 smithi195 ceph-mon[104262]: pgmap v1633: 119 pgs: 119 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 767 op/s 2024-09-13T14:58:42.067 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:42.093 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:42.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:42.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:42.195 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:42.219 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:42.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:42.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:42.321 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:42.346 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:42.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:42.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:42.448 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:42.471 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:42.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:42.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:42.573 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:42.598 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:42.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:42.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:42.700 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:42.719 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T14:58:42.721 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T14:58:42.726 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:42.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:42.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:42.795 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:58:42.796 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:58:42.806 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:58:42.807 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:58:42.823 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:58:42.823 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:58:42.827 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:42.839 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:58:42.839 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:58:42.852 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:42.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:42.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:42.855 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:58:42.855 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:58:42.871 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:58:42.871 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:58:42.888 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:58:42.888 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:58:42.904 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:58:42.904 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:58:42.954 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:42.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:42.976 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:42.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:43.077 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:43.099 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:43.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:43.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:43.201 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:43.226 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:43.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:43.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:43.329 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:43.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:43.357 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:43.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:43.458 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:43.479 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:43.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:43.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:43.581 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:43.604 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":1634,"stamp":"2024-09-13T14:58:41.837486+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790245,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":5999,"num_read_kb":13012,"num_write":212588,"num_write_kb":6786322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5177,"num_bytes_recovered":333257072,"num_keys_recovered":30,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105183,"ondisk_log_size":105183,"up":357,"acting":357,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":357,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":6255836,"kb_used_data":56780,"kb_used_omap":40,"kb_used_meta":6198935,"kb_avail":649960228,"statfs":{"total":671965249536,"available":665559273472,"internally_reserved":0,"allocated":58142720,"data_stored":40947362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41319,"internal_metadata":6347710105},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-367329304,"num_objects":-5606,"num_object_clones":0,"num_object_copies":-16818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5606,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002940"},"pg_stats":[{"pgid":"1.11","version":"90'149","reported_seq":459,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:31.801029+0000","last_change":"2024-09-13T14:56:12.490567+0000","last_active":"2024-09-13T14:58:31.801029+0000","last_peered":"2024-09-13T14:58:31.801029+0000","last_clean":"2024-09-13T14:58:31.801029+0000","last_became_active":"2024-09-13T14:56:12.490253+0000","last_became_peered":"2024-09-13T14:56:12.490253+0000","last_unstale":"2024-09-13T14:58:31.801029+0000","last_undegraded":"2024-09-13T14:58:31.801029+0000","last_fullsized":"2024-09-13T14:58:31.801029+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:38.688341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.688341+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:20.151605+0000","scrub_duration":0.0054331650000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":880,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.649981+0000","last_change":"2024-09-13T14:47:47.868783+0000","last_active":"2024-09-13T14:58:36.649981+0000","last_peered":"2024-09-13T14:58:36.649981+0000","last_clean":"2024-09-13T14:58:36.649981+0000","last_became_active":"2024-09-13T14:47:47.868478+0000","last_became_peered":"2024-09-13T14:47:47.868478+0000","last_unstale":"2024-09-13T14:58:36.649981+0000","last_undegraded":"2024-09-13T14:58:36.649981+0000","last_fullsized":"2024-09-13T14:58:36.649981+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:27:01.784142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":522,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.283019+0000","last_change":"2024-09-13T14:55:17.943558+0000","last_active":"2024-09-13T14:58:23.283019+0000","last_peered":"2024-09-13T14:58:23.283019+0000","last_clean":"2024-09-13T14:58:23.283019+0000","last_became_active":"2024-09-13T14:55:17.927162+0000","last_became_peered":"2024-09-13T14:55:17.927162+0000","last_unstale":"2024-09-13T14:58:23.283019+0000","last_undegraded":"2024-09-13T14:58:23.283019+0000","last_fullsized":"2024-09-13T14:58:23.283019+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:37:44.820559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":102,"num_keys_recovered":30,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"90'149","reported_seq":551,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.273567+0000","last_change":"2024-09-13T14:47:41.482552+0000","last_active":"2024-09-13T14:58:23.273567+0000","last_peered":"2024-09-13T14:58:23.273567+0000","last_clean":"2024-09-13T14:58:23.273567+0000","last_became_active":"2024-09-13T14:47:38.466659+0000","last_became_peered":"2024-09-13T14:47:38.466659+0000","last_unstale":"2024-09-13T14:58:23.273567+0000","last_undegraded":"2024-09-13T14:58:23.273567+0000","last_fullsized":"2024-09-13T14:58:23.273567+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:41.482364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:41.482364+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:50:49.239389+0000","scrub_duration":0.027843855000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":288,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.274304+0000","last_change":"2024-09-13T14:36:13.862181+0000","last_active":"2024-09-13T14:58:23.274304+0000","last_peered":"2024-09-13T14:58:23.274304+0000","last_clean":"2024-09-13T14:58:23.274304+0000","last_became_active":"2024-09-13T14:36:13.861873+0000","last_became_peered":"2024-09-13T14:36:13.861873+0000","last_unstale":"2024-09-13T14:58:23.274304+0000","last_undegraded":"2024-09-13T14:58:23.274304+0000","last_fullsized":"2024-09-13T14:58:23.274304+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:14.085700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":60,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:31.846635+0000","last_change":"2024-09-13T14:56:12.489695+0000","last_active":"2024-09-13T14:58:31.846635+0000","last_peered":"2024-09-13T14:58:31.846635+0000","last_clean":"2024-09-13T14:58:31.846635+0000","last_became_active":"2024-09-13T14:56:12.489302+0000","last_became_peered":"2024-09-13T14:56:12.489302+0000","last_unstale":"2024-09-13T14:58:31.846635+0000","last_undegraded":"2024-09-13T14:58:31.846635+0000","last_fullsized":"2024-09-13T14:58:31.846635+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:28:53.275229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"90'149","reported_seq":743,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.515998+0000","last_change":"2024-09-13T14:55:20.150382+0000","last_active":"2024-09-13T14:58:36.515998+0000","last_peered":"2024-09-13T14:58:36.515998+0000","last_clean":"2024-09-13T14:58:36.515998+0000","last_became_active":"2024-09-13T14:55:20.149422+0000","last_became_peered":"2024-09-13T14:55:20.149422+0000","last_unstale":"2024-09-13T14:58:36.515998+0000","last_undegraded":"2024-09-13T14:58:36.515998+0000","last_fullsized":"2024-09-13T14:58:36.515998+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:51.226748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:51.226748+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:51:07.379350+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":392,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.648702+0000","last_change":"2024-09-13T14:56:12.490265+0000","last_active":"2024-09-13T14:58:36.648702+0000","last_peered":"2024-09-13T14:58:36.648702+0000","last_clean":"2024-09-13T14:58:36.648702+0000","last_became_active":"2024-09-13T14:56:12.489937+0000","last_became_peered":"2024-09-13T14:56:12.489937+0000","last_unstale":"2024-09-13T14:58:36.648702+0000","last_undegraded":"2024-09-13T14:58:36.648702+0000","last_fullsized":"2024-09-13T14:58:36.648702+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:49.461532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":468,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.285018+0000","last_change":"2024-09-13T14:55:17.934695+0000","last_active":"2024-09-13T14:58:23.285018+0000","last_peered":"2024-09-13T14:58:23.285018+0000","last_clean":"2024-09-13T14:58:23.285018+0000","last_became_active":"2024-09-13T14:55:17.934165+0000","last_became_peered":"2024-09-13T14:55:17.934165+0000","last_unstale":"2024-09-13T14:58:23.285018+0000","last_undegraded":"2024-09-13T14:58:23.285018+0000","last_fullsized":"2024-09-13T14:58:23.285018+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:36:25.739706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"90'149","reported_seq":568,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.273899+0000","last_change":"2024-09-13T14:47:50.892667+0000","last_active":"2024-09-13T14:58:23.273899+0000","last_peered":"2024-09-13T14:58:23.273899+0000","last_clean":"2024-09-13T14:58:23.273899+0000","last_became_active":"2024-09-13T14:47:50.883312+0000","last_became_peered":"2024-09-13T14:47:50.883312+0000","last_unstale":"2024-09-13T14:58:23.273899+0000","last_undegraded":"2024-09-13T14:58:23.273899+0000","last_fullsized":"2024-09-13T14:58:23.273899+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:45.441819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:45.441819+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:39.138573+0000","scrub_duration":0.021368558999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":905,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.650235+0000","last_change":"2024-09-13T14:50:27.491166+0000","last_active":"2024-09-13T14:58:36.650235+0000","last_peered":"2024-09-13T14:58:36.650235+0000","last_clean":"2024-09-13T14:58:36.650235+0000","last_became_active":"2024-09-13T14:50:27.489905+0000","last_became_peered":"2024-09-13T14:50:27.489905+0000","last_unstale":"2024-09-13T14:58:36.650235+0000","last_undegraded":"2024-09-13T14:58:36.650235+0000","last_fullsized":"2024-09-13T14:58:36.650235+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:32.237904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":153,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:31.800996+0000","last_change":"2024-09-13T14:56:12.490197+0000","last_active":"2024-09-13T14:58:31.800996+0000","last_peered":"2024-09-13T14:58:31.800996+0000","last_clean":"2024-09-13T14:58:31.800996+0000","last_became_active":"2024-09-13T14:56:12.489554+0000","last_became_peered":"2024-09-13T14:56:12.489554+0000","last_unstale":"2024-09-13T14:58:31.800996+0000","last_undegraded":"2024-09-13T14:58:31.800996+0000","last_fullsized":"2024-09-13T14:58:31.800996+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:06.773211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"482'13066","reported_seq":14310,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:35.742591+0000","last_change":"2024-09-13T14:55:22.944506+0000","last_active":"2024-09-13T14:58:35.742591+0000","last_peered":"2024-09-13T14:58:35.742591+0000","last_clean":"2024-09-13T14:58:35.742591+0000","last_became_active":"2024-09-13T14:55:22.942014+0000","last_became_peered":"2024-09-13T14:55:22.942014+0000","last_unstale":"2024-09-13T14:58:35.742591+0000","last_undegraded":"2024-09-13T14:58:35.742591+0000","last_fullsized":"2024-09-13T14:58:35.742591+0000","mapping_epoch":446,"log_start":"465'6311","ondisk_log_start":"465'6311","created":436,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":6755,"log_dups_size":0,"ondisk_log_size":6755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:00:33.167429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13066,"num_write_kb":418112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":584,"num_bytes_recovered":38273024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"90'149","reported_seq":608,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.283193+0000","last_change":"2024-09-13T14:57:39.030647+0000","last_active":"2024-09-13T14:58:23.283193+0000","last_peered":"2024-09-13T14:58:23.283193+0000","last_clean":"2024-09-13T14:58:23.283193+0000","last_became_active":"2024-09-13T14:57:17.394134+0000","last_became_peered":"2024-09-13T14:57:17.394134+0000","last_unstale":"2024-09-13T14:58:23.283193+0000","last_undegraded":"2024-09-13T14:58:23.283193+0000","last_fullsized":"2024-09-13T14:58:23.283193+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:39.030538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:39.030538+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:54:06.634791+0000","scrub_duration":0.014515093999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":265,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.283756+0000","last_change":"2024-09-13T14:55:17.947508+0000","last_active":"2024-09-13T14:58:23.283756+0000","last_peered":"2024-09-13T14:58:23.283756+0000","last_clean":"2024-09-13T14:58:23.283756+0000","last_became_active":"2024-09-13T14:55:17.932604+0000","last_became_peered":"2024-09-13T14:55:17.932604+0000","last_unstale":"2024-09-13T14:58:23.283756+0000","last_undegraded":"2024-09-13T14:58:23.283756+0000","last_fullsized":"2024-09-13T14:58:23.283756+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:55.084612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":132,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":309,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.273741+0000","last_change":"2024-09-13T14:38:53.898907+0000","last_active":"2024-09-13T14:58:23.273741+0000","last_peered":"2024-09-13T14:58:23.273741+0000","last_clean":"2024-09-13T14:58:23.273741+0000","last_became_active":"2024-09-13T14:38:53.898572+0000","last_became_peered":"2024-09-13T14:38:53.898572+0000","last_unstale":"2024-09-13T14:58:23.273741+0000","last_undegraded":"2024-09-13T14:58:23.273741+0000","last_fullsized":"2024-09-13T14:58:23.273741+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:29.222150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"482'13110","reported_seq":13217,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.650127+0000","last_change":"2024-09-13T14:54:58.384717+0000","last_active":"2024-09-13T14:58:36.650127+0000","last_peered":"2024-09-13T14:58:36.650127+0000","last_clean":"2024-09-13T14:58:36.650127+0000","last_became_active":"2024-09-13T14:54:58.384225+0000","last_became_peered":"2024-09-13T14:54:58.384225+0000","last_unstale":"2024-09-13T14:58:36.650127+0000","last_undegraded":"2024-09-13T14:58:36.650127+0000","last_fullsized":"2024-09-13T14:58:36.650127+0000","mapping_epoch":436,"log_start":"468'7657","ondisk_log_start":"468'7657","created":436,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":5453,"log_dups_size":0,"ondisk_log_size":5453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:25:22.716676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13110,"num_write_kb":419520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"90'149","reported_seq":782,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.649793+0000","last_change":"2024-09-13T14:57:40.996705+0000","last_active":"2024-09-13T14:58:36.649793+0000","last_peered":"2024-09-13T14:58:36.649793+0000","last_clean":"2024-09-13T14:58:36.649793+0000","last_became_active":"2024-09-13T14:57:17.404299+0000","last_became_peered":"2024-09-13T14:57:17.404299+0000","last_unstale":"2024-09-13T14:58:36.649793+0000","last_undegraded":"2024-09-13T14:58:36.649793+0000","last_fullsized":"2024-09-13T14:58:36.649793+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:40.996402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:40.996402+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:04.912532+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":278,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.283757+0000","last_change":"2024-09-13T14:55:17.949010+0000","last_active":"2024-09-13T14:58:23.283757+0000","last_peered":"2024-09-13T14:58:23.283757+0000","last_clean":"2024-09-13T14:58:23.283757+0000","last_became_active":"2024-09-13T14:55:17.932712+0000","last_became_peered":"2024-09-13T14:55:17.932712+0000","last_unstale":"2024-09-13T14:58:23.283757+0000","last_undegraded":"2024-09-13T14:58:23.283757+0000","last_fullsized":"2024-09-13T14:58:23.283757+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:46:43.390958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":443,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.514981+0000","last_change":"2024-09-13T14:55:17.929723+0000","last_active":"2024-09-13T14:58:36.514981+0000","last_peered":"2024-09-13T14:58:36.514981+0000","last_clean":"2024-09-13T14:58:36.514981+0000","last_became_active":"2024-09-13T14:55:17.929131+0000","last_became_peered":"2024-09-13T14:55:17.929131+0000","last_unstale":"2024-09-13T14:58:36.514981+0000","last_undegraded":"2024-09-13T14:58:36.514981+0000","last_fullsized":"2024-09-13T14:58:36.514981+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:27:15.828973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.4","version":"482'13266","reported_seq":14633,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.648623+0000","last_change":"2024-09-13T14:56:12.495568+0000","last_active":"2024-09-13T14:58:36.648623+0000","last_peered":"2024-09-13T14:58:36.648623+0000","last_clean":"2024-09-13T14:58:36.648623+0000","last_became_active":"2024-09-13T14:56:12.490664+0000","last_became_peered":"2024-09-13T14:56:12.490664+0000","last_unstale":"2024-09-13T14:58:36.648623+0000","last_undegraded":"2024-09-13T14:58:36.648623+0000","last_fullsized":"2024-09-13T14:58:36.648623+0000","mapping_epoch":462,"log_start":"468'7831","ondisk_log_start":"468'7831","created":436,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":5435,"log_dups_size":0,"ondisk_log_size":5435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:17:29.226407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13266,"num_write_kb":424512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":623,"num_bytes_recovered":40828928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.f","version":"90'149","reported_seq":612,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.274303+0000","last_change":"2024-09-13T14:55:20.140396+0000","last_active":"2024-09-13T14:58:23.274303+0000","last_peered":"2024-09-13T14:58:23.274303+0000","last_clean":"2024-09-13T14:58:23.274303+0000","last_became_active":"2024-09-13T14:55:20.140087+0000","last_became_peered":"2024-09-13T14:55:20.140087+0000","last_unstale":"2024-09-13T14:58:23.274303+0000","last_undegraded":"2024-09-13T14:58:23.274303+0000","last_fullsized":"2024-09-13T14:58:23.274303+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:34.091782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:34.091782+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:46:05.540133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":489,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.283341+0000","last_change":"2024-09-13T14:37:49.611774+0000","last_active":"2024-09-13T14:58:23.283341+0000","last_peered":"2024-09-13T14:58:23.283341+0000","last_clean":"2024-09-13T14:58:23.283341+0000","last_became_active":"2024-09-13T14:37:49.611088+0000","last_became_peered":"2024-09-13T14:37:49.611088+0000","last_unstale":"2024-09-13T14:58:23.283341+0000","last_undegraded":"2024-09-13T14:58:23.283341+0000","last_fullsized":"2024-09-13T14:58:23.283341+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:08.933213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":308,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.515780+0000","last_change":"2024-09-13T14:47:47.865523+0000","last_active":"2024-09-13T14:58:36.515780+0000","last_peered":"2024-09-13T14:58:36.515780+0000","last_clean":"2024-09-13T14:58:36.515780+0000","last_became_active":"2024-09-13T14:47:47.865180+0000","last_became_peered":"2024-09-13T14:47:47.865180+0000","last_unstale":"2024-09-13T14:58:36.515780+0000","last_undegraded":"2024-09-13T14:58:36.515780+0000","last_fullsized":"2024-09-13T14:58:36.515780+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:39:35.479251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.5","version":"482'12966","reported_seq":13073,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.649635+0000","last_change":"2024-09-13T14:54:57.846849+0000","last_active":"2024-09-13T14:58:36.649635+0000","last_peered":"2024-09-13T14:58:36.649635+0000","last_clean":"2024-09-13T14:58:36.649635+0000","last_became_active":"2024-09-13T14:54:57.846557+0000","last_became_peered":"2024-09-13T14:54:57.846557+0000","last_unstale":"2024-09-13T14:58:36.649635+0000","last_undegraded":"2024-09-13T14:58:36.649635+0000","last_fullsized":"2024-09-13T14:58:36.649635+0000","mapping_epoch":436,"log_start":"464'5591","ondisk_log_start":"464'5591","created":436,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":7375,"log_dups_size":0,"ondisk_log_size":7375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:49.590084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12966,"num_write_kb":414912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"90'149","reported_seq":519,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.283672+0000","last_change":"2024-09-13T14:57:38.048899+0000","last_active":"2024-09-13T14:58:23.283672+0000","last_peered":"2024-09-13T14:58:23.283672+0000","last_clean":"2024-09-13T14:58:23.283672+0000","last_became_active":"2024-09-13T14:57:17.384951+0000","last_became_peered":"2024-09-13T14:57:17.384951+0000","last_unstale":"2024-09-13T14:58:23.283672+0000","last_undegraded":"2024-09-13T14:58:23.283672+0000","last_fullsized":"2024-09-13T14:58:23.283672+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:38.048812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:38.048812+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:37:42.701104+0000","scrub_duration":0.004943551,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":299,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.273960+0000","last_change":"2024-09-13T14:37:49.168003+0000","last_active":"2024-09-13T14:58:23.273960+0000","last_peered":"2024-09-13T14:58:23.273960+0000","last_clean":"2024-09-13T14:58:23.273960+0000","last_became_active":"2024-09-13T14:37:49.167638+0000","last_became_peered":"2024-09-13T14:37:49.167638+0000","last_unstale":"2024-09-13T14:58:23.273960+0000","last_undegraded":"2024-09-13T14:58:23.273960+0000","last_fullsized":"2024-09-13T14:58:23.273960+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:29.051981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":471,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.515795+0000","last_change":"2024-09-13T14:56:12.491208+0000","last_active":"2024-09-13T14:58:36.515795+0000","last_peered":"2024-09-13T14:58:36.515795+0000","last_clean":"2024-09-13T14:58:36.515795+0000","last_became_active":"2024-09-13T14:56:12.490943+0000","last_became_peered":"2024-09-13T14:56:12.490943+0000","last_unstale":"2024-09-13T14:58:36.515795+0000","last_undegraded":"2024-09-13T14:58:36.515795+0000","last_fullsized":"2024-09-13T14:58:36.515795+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:31:57.892194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"482'13136","reported_seq":14384,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.514927+0000","last_change":"2024-09-13T14:55:37.154924+0000","last_active":"2024-09-13T14:58:36.514927+0000","last_peered":"2024-09-13T14:58:36.514927+0000","last_clean":"2024-09-13T14:58:36.514927+0000","last_became_active":"2024-09-13T14:55:37.153166+0000","last_became_peered":"2024-09-13T14:55:37.153166+0000","last_unstale":"2024-09-13T14:58:36.514927+0000","last_undegraded":"2024-09-13T14:58:36.514927+0000","last_fullsized":"2024-09-13T14:58:36.514927+0000","mapping_epoch":459,"log_start":"468'7822","ondisk_log_start":"468'7822","created":436,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":5314,"log_dups_size":0,"ondisk_log_size":5314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:48:15.748240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13136,"num_write_kb":420352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":565,"num_bytes_recovered":37027840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"90'149","reported_seq":682,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.284931+0000","last_change":"2024-09-13T14:57:41.926732+0000","last_active":"2024-09-13T14:58:23.284931+0000","last_peered":"2024-09-13T14:58:23.284931+0000","last_clean":"2024-09-13T14:58:23.284931+0000","last_became_active":"2024-09-13T14:57:17.380843+0000","last_became_peered":"2024-09-13T14:57:17.380843+0000","last_unstale":"2024-09-13T14:58:23.284931+0000","last_undegraded":"2024-09-13T14:58:23.284931+0000","last_fullsized":"2024-09-13T14:58:23.284931+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:41.926626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:41.926626+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:08:57.137123+0000","scrub_duration":0.0019476179999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":866,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.649531+0000","last_change":"2024-09-13T14:55:17.944596+0000","last_active":"2024-09-13T14:58:36.649531+0000","last_peered":"2024-09-13T14:58:36.649531+0000","last_clean":"2024-09-13T14:58:36.649531+0000","last_became_active":"2024-09-13T14:55:17.935767+0000","last_became_peered":"2024-09-13T14:55:17.935767+0000","last_unstale":"2024-09-13T14:58:36.649531+0000","last_undegraded":"2024-09-13T14:58:36.649531+0000","last_fullsized":"2024-09-13T14:58:36.649531+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:22:19.619639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":395,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.649557+0000","last_change":"2024-09-13T14:55:17.933005+0000","last_active":"2024-09-13T14:58:36.649557+0000","last_peered":"2024-09-13T14:58:36.649557+0000","last_clean":"2024-09-13T14:58:36.649557+0000","last_became_active":"2024-09-13T14:55:17.931575+0000","last_became_peered":"2024-09-13T14:55:17.931575+0000","last_unstale":"2024-09-13T14:58:36.649557+0000","last_undegraded":"2024-09-13T14:58:36.649557+0000","last_fullsized":"2024-09-13T14:58:36.649557+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:50:20.352970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"482'13278","reported_seq":13385,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.649301+0000","last_change":"2024-09-13T14:54:57.846161+0000","last_active":"2024-09-13T14:58:36.649301+0000","last_peered":"2024-09-13T14:58:36.649301+0000","last_clean":"2024-09-13T14:58:36.649301+0000","last_became_active":"2024-09-13T14:54:57.845348+0000","last_became_peered":"2024-09-13T14:54:57.845348+0000","last_unstale":"2024-09-13T14:58:36.649301+0000","last_undegraded":"2024-09-13T14:58:36.649301+0000","last_fullsized":"2024-09-13T14:58:36.649301+0000","mapping_epoch":436,"log_start":"464'5876","ondisk_log_start":"464'5876","created":436,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":7402,"log_dups_size":0,"ondisk_log_size":7402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:07:01.751407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13278,"num_write_kb":424896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"90'149","reported_seq":585,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.273568+0000","last_change":"2024-09-13T14:57:39.812785+0000","last_active":"2024-09-13T14:58:23.273568+0000","last_peered":"2024-09-13T14:58:23.273568+0000","last_clean":"2024-09-13T14:58:23.273568+0000","last_became_active":"2024-09-13T14:57:17.383307+0000","last_became_peered":"2024-09-13T14:57:17.383307+0000","last_unstale":"2024-09-13T14:58:23.273568+0000","last_undegraded":"2024-09-13T14:58:23.273568+0000","last_fullsized":"2024-09-13T14:58:23.273568+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:39.812688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:39.812688+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:18:59.726465+0000","scrub_duration":0.010496065000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":352,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.515241+0000","last_change":"2024-09-13T14:55:17.935369+0000","last_active":"2024-09-13T14:58:36.515241+0000","last_peered":"2024-09-13T14:58:36.515241+0000","last_clean":"2024-09-13T14:58:36.515241+0000","last_became_active":"2024-09-13T14:55:17.930815+0000","last_became_peered":"2024-09-13T14:55:17.930815+0000","last_unstale":"2024-09-13T14:58:36.515241+0000","last_undegraded":"2024-09-13T14:58:36.515241+0000","last_fullsized":"2024-09-13T14:58:36.515241+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:59:31.247160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":481,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.649979+0000","last_change":"2024-09-13T14:55:17.932243+0000","last_active":"2024-09-13T14:58:36.649979+0000","last_peered":"2024-09-13T14:58:36.649979+0000","last_clean":"2024-09-13T14:58:36.649979+0000","last_became_active":"2024-09-13T14:55:17.931961+0000","last_became_peered":"2024-09-13T14:55:17.931961+0000","last_unstale":"2024-09-13T14:58:36.649979+0000","last_undegraded":"2024-09-13T14:58:36.649979+0000","last_fullsized":"2024-09-13T14:58:36.649979+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:23:52.680942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.0","version":"482'13166","reported_seq":14593,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.514164+0000","last_change":"2024-09-13T14:55:24.959843+0000","last_active":"2024-09-13T14:58:36.514164+0000","last_peered":"2024-09-13T14:58:36.514164+0000","last_clean":"2024-09-13T14:58:36.514164+0000","last_became_active":"2024-09-13T14:55:24.957491+0000","last_became_peered":"2024-09-13T14:55:24.957491+0000","last_unstale":"2024-09-13T14:58:36.514164+0000","last_undegraded":"2024-09-13T14:58:36.514164+0000","last_fullsized":"2024-09-13T14:58:36.514164+0000","mapping_epoch":448,"log_start":"468'7838","ondisk_log_start":"468'7838","created":436,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":5328,"log_dups_size":0,"ondisk_log_size":5328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:09:19.929604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13166,"num_write_kb":421312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":654,"num_bytes_recovered":42860544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"90'149","reported_seq":775,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.515707+0000","last_change":"2024-09-13T14:57:39.766173+0000","last_active":"2024-09-13T14:58:36.515707+0000","last_peered":"2024-09-13T14:58:36.515707+0000","last_clean":"2024-09-13T14:58:36.515707+0000","last_became_active":"2024-09-13T14:57:17.389370+0000","last_became_peered":"2024-09-13T14:57:17.389370+0000","last_unstale":"2024-09-13T14:58:36.515707+0000","last_undegraded":"2024-09-13T14:58:36.515707+0000","last_fullsized":"2024-09-13T14:58:36.515707+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:39.765978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:39.765978+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:39.980756+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":401,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.649382+0000","last_change":"2024-09-13T14:55:17.932199+0000","last_active":"2024-09-13T14:58:36.649382+0000","last_peered":"2024-09-13T14:58:36.649382+0000","last_clean":"2024-09-13T14:58:36.649382+0000","last_became_active":"2024-09-13T14:55:17.931873+0000","last_became_peered":"2024-09-13T14:55:17.931873+0000","last_unstale":"2024-09-13T14:58:36.649382+0000","last_undegraded":"2024-09-13T14:58:36.649382+0000","last_fullsized":"2024-09-13T14:58:36.649382+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:02:18.004113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":428,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.515508+0000","last_change":"2024-09-13T14:55:17.931269+0000","last_active":"2024-09-13T14:58:36.515508+0000","last_peered":"2024-09-13T14:58:36.515508+0000","last_clean":"2024-09-13T14:58:36.515508+0000","last_became_active":"2024-09-13T14:55:17.930909+0000","last_became_peered":"2024-09-13T14:55:17.930909+0000","last_unstale":"2024-09-13T14:58:36.515508+0000","last_undegraded":"2024-09-13T14:58:36.515508+0000","last_fullsized":"2024-09-13T14:58:36.515508+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:47.592526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"482'13714","reported_seq":15125,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:35.720415+0000","last_change":"2024-09-13T14:55:35.136618+0000","last_active":"2024-09-13T14:58:35.720415+0000","last_peered":"2024-09-13T14:58:35.720415+0000","last_clean":"2024-09-13T14:58:35.720415+0000","last_became_active":"2024-09-13T14:55:35.134043+0000","last_became_peered":"2024-09-13T14:55:35.134043+0000","last_unstale":"2024-09-13T14:58:35.720415+0000","last_undegraded":"2024-09-13T14:58:35.720415+0000","last_fullsized":"2024-09-13T14:58:35.720415+0000","mapping_epoch":457,"log_start":"467'7697","ondisk_log_start":"467'7697","created":436,"last_epoch_clean":458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":6017,"log_dups_size":0,"ondisk_log_size":6017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:55.445002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13714,"num_write_kb":438848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":667,"num_bytes_recovered":43712512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"90'149","reported_seq":585,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.273671+0000","last_change":"2024-09-13T14:57:38.849773+0000","last_active":"2024-09-13T14:58:23.273671+0000","last_peered":"2024-09-13T14:58:23.273671+0000","last_clean":"2024-09-13T14:58:23.273671+0000","last_became_active":"2024-09-13T14:57:17.378580+0000","last_became_peered":"2024-09-13T14:57:17.378580+0000","last_unstale":"2024-09-13T14:58:23.273671+0000","last_undegraded":"2024-09-13T14:58:23.273671+0000","last_fullsized":"2024-09-13T14:58:23.273671+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:38.849610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:38.849610+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:22:24.723639+0000","scrub_duration":0.0034255890000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":880,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.649052+0000","last_change":"2024-09-13T14:47:47.861249+0000","last_active":"2024-09-13T14:58:36.649052+0000","last_peered":"2024-09-13T14:58:36.649052+0000","last_clean":"2024-09-13T14:58:36.649052+0000","last_became_active":"2024-09-13T14:47:47.860873+0000","last_became_peered":"2024-09-13T14:47:47.860873+0000","last_unstale":"2024-09-13T14:58:36.649052+0000","last_undegraded":"2024-09-13T14:58:36.649052+0000","last_fullsized":"2024-09-13T14:58:36.649052+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:37:50.709437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":315,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.515139+0000","last_change":"2024-09-13T14:55:17.933070+0000","last_active":"2024-09-13T14:58:36.515139+0000","last_peered":"2024-09-13T14:58:36.515139+0000","last_clean":"2024-09-13T14:58:36.515139+0000","last_became_active":"2024-09-13T14:55:17.931232+0000","last_became_peered":"2024-09-13T14:55:17.931232+0000","last_unstale":"2024-09-13T14:58:36.515139+0000","last_undegraded":"2024-09-13T14:58:36.515139+0000","last_fullsized":"2024-09-13T14:58:36.515139+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:22:46.719483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"482'13086","reported_seq":13193,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.515596+0000","last_change":"2024-09-13T14:54:58.384133+0000","last_active":"2024-09-13T14:58:36.515596+0000","last_peered":"2024-09-13T14:58:36.515596+0000","last_clean":"2024-09-13T14:58:36.515596+0000","last_became_active":"2024-09-13T14:54:58.383823+0000","last_became_peered":"2024-09-13T14:54:58.383823+0000","last_unstale":"2024-09-13T14:58:36.515596+0000","last_undegraded":"2024-09-13T14:58:36.515596+0000","last_fullsized":"2024-09-13T14:58:36.515596+0000","mapping_epoch":436,"log_start":"468'7822","ondisk_log_start":"468'7822","created":436,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":5264,"log_dups_size":0,"ondisk_log_size":5264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:30.724982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13086,"num_write_kb":418752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":667,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.282802+0000","last_change":"2024-09-13T14:47:31.684319+0000","last_active":"2024-09-13T14:58:23.282802+0000","last_peered":"2024-09-13T14:58:23.282802+0000","last_clean":"2024-09-13T14:58:23.282802+0000","last_became_active":"2024-09-13T14:47:29.415658+0000","last_became_peered":"2024-09-13T14:47:29.415658+0000","last_unstale":"2024-09-13T14:58:23.282802+0000","last_undegraded":"2024-09-13T14:58:23.282802+0000","last_fullsized":"2024-09-13T14:58:23.282802+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:31.681433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:31.681433+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:29:34.663513+0000","scrub_duration":0.0059601139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":795,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.274265+0000","last_change":"2024-09-13T14:46:04.780718+0000","last_active":"2024-09-13T14:58:23.274265+0000","last_peered":"2024-09-13T14:58:23.274265+0000","last_clean":"2024-09-13T14:58:23.274265+0000","last_became_active":"2024-09-13T14:46:04.780120+0000","last_became_peered":"2024-09-13T14:46:04.780120+0000","last_unstale":"2024-09-13T14:58:23.274265+0000","last_undegraded":"2024-09-13T14:58:23.274265+0000","last_fullsized":"2024-09-13T14:58:23.274265+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:57:14.475888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":808,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.648930+0000","last_change":"2024-09-13T14:37:49.160719+0000","last_active":"2024-09-13T14:58:36.648930+0000","last_peered":"2024-09-13T14:58:36.648930+0000","last_clean":"2024-09-13T14:58:36.648930+0000","last_became_active":"2024-09-13T14:37:49.160401+0000","last_became_peered":"2024-09-13T14:37:49.160401+0000","last_unstale":"2024-09-13T14:58:36.648930+0000","last_undegraded":"2024-09-13T14:58:36.648930+0000","last_fullsized":"2024-09-13T14:58:36.648930+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:21.737945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":126,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.283335+0000","last_change":"2024-09-13T14:55:17.933431+0000","last_active":"2024-09-13T14:58:23.283335+0000","last_peered":"2024-09-13T14:58:23.283335+0000","last_clean":"2024-09-13T14:58:23.283335+0000","last_became_active":"2024-09-13T14:55:17.932711+0000","last_became_peered":"2024-09-13T14:55:17.932711+0000","last_unstale":"2024-09-13T14:58:23.283335+0000","last_undegraded":"2024-09-13T14:58:23.283335+0000","last_fullsized":"2024-09-13T14:58:23.283335+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:47.712045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"482'13020","reported_seq":13138,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.648475+0000","last_change":"2024-09-13T14:56:12.516271+0000","last_active":"2024-09-13T14:58:36.648475+0000","last_peered":"2024-09-13T14:58:36.648475+0000","last_clean":"2024-09-13T14:58:36.648475+0000","last_became_active":"2024-09-13T14:56:12.494537+0000","last_became_peered":"2024-09-13T14:56:12.494537+0000","last_unstale":"2024-09-13T14:58:36.648475+0000","last_undegraded":"2024-09-13T14:58:36.648475+0000","last_fullsized":"2024-09-13T14:58:36.648475+0000","mapping_epoch":462,"log_start":"464'5659","ondisk_log_start":"464'5659","created":436,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":7361,"log_dups_size":0,"ondisk_log_size":7361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:07:34.989938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13020,"num_write_kb":416640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"90'149","reported_seq":751,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.515828+0000","last_change":"2024-09-13T14:55:20.148734+0000","last_active":"2024-09-13T14:58:36.515828+0000","last_peered":"2024-09-13T14:58:36.515828+0000","last_clean":"2024-09-13T14:58:36.515828+0000","last_became_active":"2024-09-13T14:55:20.147918+0000","last_became_peered":"2024-09-13T14:55:20.147918+0000","last_unstale":"2024-09-13T14:58:36.515828+0000","last_undegraded":"2024-09-13T14:58:36.515828+0000","last_fullsized":"2024-09-13T14:58:36.515828+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:34.603316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:34.603316+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:27.859151+0000","scrub_duration":0.0057131120000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":737,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:26.846093+0000","last_change":"2024-09-13T14:47:47.892973+0000","last_active":"2024-09-13T14:58:26.846093+0000","last_peered":"2024-09-13T14:58:26.846093+0000","last_clean":"2024-09-13T14:58:26.846093+0000","last_became_active":"2024-09-13T14:47:47.866986+0000","last_became_peered":"2024-09-13T14:47:47.866986+0000","last_unstale":"2024-09-13T14:58:26.846093+0000","last_undegraded":"2024-09-13T14:58:26.846093+0000","last_fullsized":"2024-09-13T14:58:26.846093+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:07.353290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":465,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.648490+0000","last_change":"2024-09-13T14:56:12.490871+0000","last_active":"2024-09-13T14:58:36.648490+0000","last_peered":"2024-09-13T14:58:36.648490+0000","last_clean":"2024-09-13T14:58:36.648490+0000","last_became_active":"2024-09-13T14:56:12.490535+0000","last_became_peered":"2024-09-13T14:56:12.490535+0000","last_unstale":"2024-09-13T14:58:36.648490+0000","last_undegraded":"2024-09-13T14:58:36.648490+0000","last_fullsized":"2024-09-13T14:58:36.648490+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:49.429179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":265,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.515148+0000","last_change":"2024-09-13T14:56:12.490229+0000","last_active":"2024-09-13T14:58:36.515148+0000","last_peered":"2024-09-13T14:58:36.515148+0000","last_clean":"2024-09-13T14:58:36.515148+0000","last_became_active":"2024-09-13T14:56:12.489512+0000","last_became_peered":"2024-09-13T14:56:12.489512+0000","last_unstale":"2024-09-13T14:58:36.515148+0000","last_undegraded":"2024-09-13T14:58:36.515148+0000","last_fullsized":"2024-09-13T14:58:36.515148+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:26:10.594549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.9","version":"482'13374","reported_seq":13431,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:35.744320+0000","last_change":"2024-09-13T14:54:57.844803+0000","last_active":"2024-09-13T14:58:35.744320+0000","last_peered":"2024-09-13T14:58:35.744320+0000","last_clean":"2024-09-13T14:58:35.744320+0000","last_became_active":"2024-09-13T14:54:57.844499+0000","last_became_peered":"2024-09-13T14:54:57.844499+0000","last_unstale":"2024-09-13T14:58:35.744320+0000","last_undegraded":"2024-09-13T14:58:35.744320+0000","last_fullsized":"2024-09-13T14:58:35.744320+0000","mapping_epoch":436,"log_start":"465'6635","ondisk_log_start":"465'6635","created":436,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":6739,"log_dups_size":0,"ondisk_log_size":6739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:46.617173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13374,"num_write_kb":427968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"90'149","reported_seq":700,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.282603+0000","last_change":"2024-09-13T14:55:20.134578+0000","last_active":"2024-09-13T14:58:23.282603+0000","last_peered":"2024-09-13T14:58:23.282603+0000","last_clean":"2024-09-13T14:58:23.282603+0000","last_became_active":"2024-09-13T14:55:20.134120+0000","last_became_peered":"2024-09-13T14:55:20.134120+0000","last_unstale":"2024-09-13T14:58:23.282603+0000","last_undegraded":"2024-09-13T14:58:23.282603+0000","last_fullsized":"2024-09-13T14:58:23.282603+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:29.629342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:29.629342+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:38:13.399024+0000","scrub_duration":0.0099290990000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":792,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.649266+0000","last_change":"2024-09-13T14:46:04.782955+0000","last_active":"2024-09-13T14:58:36.649266+0000","last_peered":"2024-09-13T14:58:36.649266+0000","last_clean":"2024-09-13T14:58:36.649266+0000","last_became_active":"2024-09-13T14:46:04.782250+0000","last_became_peered":"2024-09-13T14:46:04.782250+0000","last_unstale":"2024-09-13T14:58:36.649266+0000","last_undegraded":"2024-09-13T14:58:36.649266+0000","last_fullsized":"2024-09-13T14:58:36.649266+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:24.596922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":309,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.514626+0000","last_change":"2024-09-13T14:47:47.866684+0000","last_active":"2024-09-13T14:58:36.514626+0000","last_peered":"2024-09-13T14:58:36.514626+0000","last_clean":"2024-09-13T14:58:36.514626+0000","last_became_active":"2024-09-13T14:47:47.866400+0000","last_became_peered":"2024-09-13T14:47:47.866400+0000","last_unstale":"2024-09-13T14:58:36.514626+0000","last_undegraded":"2024-09-13T14:58:36.514626+0000","last_fullsized":"2024-09-13T14:58:36.514626+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:08:34.289658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":309,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.515291+0000","last_change":"2024-09-13T14:47:47.870218+0000","last_active":"2024-09-13T14:58:36.515291+0000","last_peered":"2024-09-13T14:58:36.515291+0000","last_clean":"2024-09-13T14:58:36.515291+0000","last_became_active":"2024-09-13T14:47:47.869907+0000","last_became_peered":"2024-09-13T14:47:47.869907+0000","last_unstale":"2024-09-13T14:58:36.515291+0000","last_undegraded":"2024-09-13T14:58:36.515291+0000","last_fullsized":"2024-09-13T14:58:36.515291+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:41:49.141441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"482'13624","reported_seq":13716,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:35.734835+0000","last_change":"2024-09-13T14:56:12.497755+0000","last_active":"2024-09-13T14:58:35.734835+0000","last_peered":"2024-09-13T14:58:35.734835+0000","last_clean":"2024-09-13T14:58:35.734835+0000","last_became_active":"2024-09-13T14:56:12.489758+0000","last_became_peered":"2024-09-13T14:56:12.489758+0000","last_unstale":"2024-09-13T14:58:35.734835+0000","last_undegraded":"2024-09-13T14:58:35.734835+0000","last_fullsized":"2024-09-13T14:58:35.734835+0000","mapping_epoch":462,"log_start":"465'6869","ondisk_log_start":"465'6869","created":436,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":6755,"log_dups_size":0,"ondisk_log_size":6755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:22:53.808045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13624,"num_write_kb":435968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"90'149","reported_seq":585,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.273699+0000","last_change":"2024-09-13T14:47:32.275753+0000","last_active":"2024-09-13T14:58:23.273699+0000","last_peered":"2024-09-13T14:58:23.273699+0000","last_clean":"2024-09-13T14:58:23.273699+0000","last_became_active":"2024-09-13T14:47:29.401066+0000","last_became_peered":"2024-09-13T14:47:29.401066+0000","last_unstale":"2024-09-13T14:58:23.273699+0000","last_undegraded":"2024-09-13T14:58:23.273699+0000","last_fullsized":"2024-09-13T14:58:23.273699+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:32.275562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:32.275562+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:43:41.056775+0000","scrub_duration":0.0054674260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":1240,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.273802+0000","last_change":"2024-09-13T14:47:50.892842+0000","last_active":"2024-09-13T14:58:23.273802+0000","last_peered":"2024-09-13T14:58:23.273802+0000","last_clean":"2024-09-13T14:58:23.273802+0000","last_became_active":"2024-09-13T14:47:50.892237+0000","last_became_peered":"2024-09-13T14:47:50.892237+0000","last_unstale":"2024-09-13T14:58:23.273802+0000","last_undegraded":"2024-09-13T14:58:23.273802+0000","last_fullsized":"2024-09-13T14:58:23.273802+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:20:05.796762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":8388646,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":295,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.515378+0000","last_change":"2024-09-13T14:56:12.491483+0000","last_active":"2024-09-13T14:58:36.515378+0000","last_peered":"2024-09-13T14:58:36.515378+0000","last_clean":"2024-09-13T14:58:36.515378+0000","last_became_active":"2024-09-13T14:56:12.491191+0000","last_became_peered":"2024-09-13T14:56:12.491191+0000","last_unstale":"2024-09-13T14:58:36.515378+0000","last_undegraded":"2024-09-13T14:58:36.515378+0000","last_fullsized":"2024-09-13T14:58:36.515378+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:54:11.122787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":309,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.515861+0000","last_change":"2024-09-13T14:47:47.877272+0000","last_active":"2024-09-13T14:58:36.515861+0000","last_peered":"2024-09-13T14:58:36.515861+0000","last_clean":"2024-09-13T14:58:36.515861+0000","last_became_active":"2024-09-13T14:47:47.877003+0000","last_became_peered":"2024-09-13T14:47:47.877003+0000","last_unstale":"2024-09-13T14:58:36.515861+0000","last_undegraded":"2024-09-13T14:58:36.515861+0000","last_fullsized":"2024-09-13T14:58:36.515861+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:47:23.172452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.c","version":"482'13274","reported_seq":14614,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:35.724195+0000","last_change":"2024-09-13T14:55:28.486018+0000","last_active":"2024-09-13T14:58:35.724195+0000","last_peered":"2024-09-13T14:58:35.724195+0000","last_clean":"2024-09-13T14:58:35.724195+0000","last_became_active":"2024-09-13T14:55:28.483879+0000","last_became_peered":"2024-09-13T14:55:28.483879+0000","last_unstale":"2024-09-13T14:58:35.724195+0000","last_undegraded":"2024-09-13T14:58:35.724195+0000","last_fullsized":"2024-09-13T14:58:35.724195+0000","mapping_epoch":451,"log_start":"467'7230","ondisk_log_start":"467'7230","created":436,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":6044,"log_dups_size":0,"ondisk_log_size":6044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:45:38.042233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13274,"num_write_kb":424768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":632,"num_bytes_recovered":41418752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"90'149","reported_seq":770,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.650241+0000","last_change":"2024-09-13T14:57:37.019594+0000","last_active":"2024-09-13T14:58:36.650241+0000","last_peered":"2024-09-13T14:58:36.650241+0000","last_clean":"2024-09-13T14:58:36.650241+0000","last_became_active":"2024-09-13T14:57:17.385457+0000","last_became_peered":"2024-09-13T14:57:17.385457+0000","last_unstale":"2024-09-13T14:58:36.650241+0000","last_undegraded":"2024-09-13T14:58:36.650241+0000","last_fullsized":"2024-09-13T14:58:36.650241+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:37.019510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:37.019510+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:54:21.275369+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":309,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.274147+0000","last_change":"2024-09-13T14:46:01.763906+0000","last_active":"2024-09-13T14:58:23.274147+0000","last_peered":"2024-09-13T14:58:23.274147+0000","last_clean":"2024-09-13T14:58:23.274147+0000","last_became_active":"2024-09-13T14:46:01.763542+0000","last_became_peered":"2024-09-13T14:46:01.763542+0000","last_unstale":"2024-09-13T14:58:23.274147+0000","last_undegraded":"2024-09-13T14:58:23.274147+0000","last_fullsized":"2024-09-13T14:58:23.274147+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:09:36.652767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":1597,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.515950+0000","last_change":"2024-09-13T14:47:50.883365+0000","last_active":"2024-09-13T14:58:36.515950+0000","last_peered":"2024-09-13T14:58:36.515950+0000","last_clean":"2024-09-13T14:58:36.515950+0000","last_became_active":"2024-09-13T14:47:50.882609+0000","last_became_peered":"2024-09-13T14:47:50.882609+0000","last_unstale":"2024-09-13T14:58:36.515950+0000","last_undegraded":"2024-09-13T14:58:36.515950+0000","last_fullsized":"2024-09-13T14:58:36.515950+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:12:17.961575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":316,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.515939+0000","last_change":"2024-09-13T14:55:17.937000+0000","last_active":"2024-09-13T14:58:36.515939+0000","last_peered":"2024-09-13T14:58:36.515939+0000","last_clean":"2024-09-13T14:58:36.515939+0000","last_became_active":"2024-09-13T14:55:17.936557+0000","last_became_peered":"2024-09-13T14:55:17.936557+0000","last_unstale":"2024-09-13T14:58:36.515939+0000","last_undegraded":"2024-09-13T14:58:36.515939+0000","last_fullsized":"2024-09-13T14:58:36.515939+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:55:25.879096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"482'13110","reported_seq":14372,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.648643+0000","last_change":"2024-09-13T14:56:12.498406+0000","last_active":"2024-09-13T14:58:36.648643+0000","last_peered":"2024-09-13T14:58:36.648643+0000","last_clean":"2024-09-13T14:58:36.648643+0000","last_became_active":"2024-09-13T14:56:12.490622+0000","last_became_peered":"2024-09-13T14:56:12.490622+0000","last_unstale":"2024-09-13T14:58:36.648643+0000","last_undegraded":"2024-09-13T14:58:36.648643+0000","last_fullsized":"2024-09-13T14:58:36.648643+0000","mapping_epoch":462,"log_start":"468'7679","ondisk_log_start":"468'7679","created":436,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":5431,"log_dups_size":0,"ondisk_log_size":5431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:07:04.577772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13110,"num_write_kb":419457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":572,"num_bytes_recovered":37486592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"90'149","reported_seq":466,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.650165+0000","last_change":"2024-09-13T14:57:41.951221+0000","last_active":"2024-09-13T14:58:36.650165+0000","last_peered":"2024-09-13T14:58:36.650165+0000","last_clean":"2024-09-13T14:58:36.650165+0000","last_became_active":"2024-09-13T14:57:17.390576+0000","last_became_peered":"2024-09-13T14:57:17.390576+0000","last_unstale":"2024-09-13T14:58:36.650165+0000","last_undegraded":"2024-09-13T14:58:36.650165+0000","last_fullsized":"2024-09-13T14:58:36.650165+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:41.951149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:41.951149+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:49:55.865705+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":309,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.516012+0000","last_change":"2024-09-13T14:47:47.876562+0000","last_active":"2024-09-13T14:58:36.516012+0000","last_peered":"2024-09-13T14:58:36.516012+0000","last_clean":"2024-09-13T14:58:36.516012+0000","last_became_active":"2024-09-13T14:47:47.876297+0000","last_became_peered":"2024-09-13T14:47:47.876297+0000","last_unstale":"2024-09-13T14:58:36.516012+0000","last_undegraded":"2024-09-13T14:58:36.516012+0000","last_fullsized":"2024-09-13T14:58:36.516012+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:03:27.761394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":1049,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.283525+0000","last_change":"2024-09-13T14:55:20.147689+0000","last_active":"2024-09-13T14:58:23.283525+0000","last_peered":"2024-09-13T14:58:23.283525+0000","last_clean":"2024-09-13T14:58:23.283525+0000","last_became_active":"2024-09-13T14:55:20.147241+0000","last_became_peered":"2024-09-13T14:55:20.147241+0000","last_unstale":"2024-09-13T14:58:23.283525+0000","last_undegraded":"2024-09-13T14:58:23.283525+0000","last_fullsized":"2024-09-13T14:58:23.283525+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:43:36.224330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":340,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.274342+0000","last_change":"2024-09-13T14:56:12.491490+0000","last_active":"2024-09-13T14:58:23.274342+0000","last_peered":"2024-09-13T14:58:23.274342+0000","last_clean":"2024-09-13T14:58:23.274342+0000","last_became_active":"2024-09-13T14:56:12.490650+0000","last_became_peered":"2024-09-13T14:56:12.490650+0000","last_unstale":"2024-09-13T14:58:23.274342+0000","last_undegraded":"2024-09-13T14:58:23.274342+0000","last_fullsized":"2024-09-13T14:58:23.274342+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:55:33.313465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"482'13278","reported_seq":14708,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.649672+0000","last_change":"2024-09-13T14:55:23.955183+0000","last_active":"2024-09-13T14:58:36.649672+0000","last_peered":"2024-09-13T14:58:36.649672+0000","last_clean":"2024-09-13T14:58:36.649672+0000","last_became_active":"2024-09-13T14:55:23.952692+0000","last_became_peered":"2024-09-13T14:55:23.952692+0000","last_unstale":"2024-09-13T14:58:36.649672+0000","last_undegraded":"2024-09-13T14:58:36.649672+0000","last_fullsized":"2024-09-13T14:58:36.649672+0000","mapping_epoch":447,"log_start":"464'5868","ondisk_log_start":"464'5868","created":436,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":7410,"log_dups_size":0,"ondisk_log_size":7410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:48:52.915452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13278,"num_write_kb":424896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":659,"num_bytes_recovered":43188224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"90'149","reported_seq":516,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.649901+0000","last_change":"2024-09-13T14:57:37.063675+0000","last_active":"2024-09-13T14:58:36.649901+0000","last_peered":"2024-09-13T14:58:36.649901+0000","last_clean":"2024-09-13T14:58:36.649901+0000","last_became_active":"2024-09-13T14:57:17.385011+0000","last_became_peered":"2024-09-13T14:57:17.385011+0000","last_unstale":"2024-09-13T14:58:36.649901+0000","last_undegraded":"2024-09-13T14:58:36.649901+0000","last_fullsized":"2024-09-13T14:58:36.649901+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:37.063494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:37.063494+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:33.681893+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":899,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.650388+0000","last_change":"2024-09-13T14:38:53.938227+0000","last_active":"2024-09-13T14:58:36.650388+0000","last_peered":"2024-09-13T14:58:36.650388+0000","last_clean":"2024-09-13T14:58:36.650388+0000","last_became_active":"2024-09-13T14:38:53.902867+0000","last_became_peered":"2024-09-13T14:38:53.902867+0000","last_unstale":"2024-09-13T14:58:36.650388+0000","last_undegraded":"2024-09-13T14:58:36.650388+0000","last_fullsized":"2024-09-13T14:58:36.650388+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:16.831756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":812,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.515677+0000","last_change":"2024-09-13T14:55:29.510841+0000","last_active":"2024-09-13T14:58:36.515677+0000","last_peered":"2024-09-13T14:58:36.515677+0000","last_clean":"2024-09-13T14:58:36.515677+0000","last_became_active":"2024-09-13T14:55:29.510252+0000","last_became_peered":"2024-09-13T14:55:29.510252+0000","last_unstale":"2024-09-13T14:58:36.515677+0000","last_undegraded":"2024-09-13T14:58:36.515677+0000","last_fullsized":"2024-09-13T14:58:36.515677+0000","mapping_epoch":452,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":453,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:14:27.135796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":881,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.648876+0000","last_change":"2024-09-13T14:38:53.900107+0000","last_active":"2024-09-13T14:58:36.648876+0000","last_peered":"2024-09-13T14:58:36.648876+0000","last_clean":"2024-09-13T14:58:36.648876+0000","last_became_active":"2024-09-13T14:38:53.899556+0000","last_became_peered":"2024-09-13T14:38:53.899556+0000","last_unstale":"2024-09-13T14:58:36.648876+0000","last_undegraded":"2024-09-13T14:58:36.648876+0000","last_fullsized":"2024-09-13T14:58:36.648876+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:15:47.918163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"482'13212","reported_seq":13319,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.650120+0000","last_change":"2024-09-13T14:54:58.382957+0000","last_active":"2024-09-13T14:58:36.650120+0000","last_peered":"2024-09-13T14:58:36.650120+0000","last_clean":"2024-09-13T14:58:36.650120+0000","last_became_active":"2024-09-13T14:54:58.382609+0000","last_became_peered":"2024-09-13T14:54:58.382609+0000","last_unstale":"2024-09-13T14:58:36.650120+0000","last_undegraded":"2024-09-13T14:58:36.650120+0000","last_fullsized":"2024-09-13T14:58:36.650120+0000","mapping_epoch":436,"log_start":"468'7764","ondisk_log_start":"468'7764","created":436,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:56.829219+0000","last_clean_scrub_stamp":"2024-09-13T14:54:56.829219+0000","objects_scrubbed":0,"log_size":5448,"log_dups_size":0,"ondisk_log_size":5448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:55:46.466387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13212,"num_write_kb":422784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"90'149","reported_seq":741,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.649698+0000","last_change":"2024-09-13T14:47:29.999643+0000","last_active":"2024-09-13T14:58:36.649698+0000","last_peered":"2024-09-13T14:58:36.649698+0000","last_clean":"2024-09-13T14:58:36.649698+0000","last_became_active":"2024-09-13T14:47:29.410951+0000","last_became_peered":"2024-09-13T14:47:29.410951+0000","last_unstale":"2024-09-13T14:58:36.649698+0000","last_undegraded":"2024-09-13T14:58:36.649698+0000","last_fullsized":"2024-09-13T14:58:36.649698+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:29.999519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:29.999519+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:45:08.824756+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":266,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.515231+0000","last_change":"2024-09-13T14:56:12.493198+0000","last_active":"2024-09-13T14:58:36.515231+0000","last_peered":"2024-09-13T14:58:36.515231+0000","last_clean":"2024-09-13T14:58:36.515231+0000","last_became_active":"2024-09-13T14:56:12.490423+0000","last_became_peered":"2024-09-13T14:56:12.490423+0000","last_unstale":"2024-09-13T14:58:36.515231+0000","last_undegraded":"2024-09-13T14:58:36.515231+0000","last_fullsized":"2024-09-13T14:58:36.515231+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:29.984031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":290,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:26.846244+0000","last_change":"2024-09-13T14:46:01.781508+0000","last_active":"2024-09-13T14:58:26.846244+0000","last_peered":"2024-09-13T14:58:26.846244+0000","last_clean":"2024-09-13T14:58:26.846244+0000","last_became_active":"2024-09-13T14:46:01.767096+0000","last_became_peered":"2024-09-13T14:46:01.767096+0000","last_unstale":"2024-09-13T14:58:26.846244+0000","last_undegraded":"2024-09-13T14:58:26.846244+0000","last_fullsized":"2024-09-13T14:58:26.846244+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:20:01.119189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":273,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.273750+0000","last_change":"2024-09-13T14:55:17.932691+0000","last_active":"2024-09-13T14:58:23.273750+0000","last_peered":"2024-09-13T14:58:23.273750+0000","last_clean":"2024-09-13T14:58:23.273750+0000","last_became_active":"2024-09-13T14:55:17.932391+0000","last_became_peered":"2024-09-13T14:55:17.932391+0000","last_unstale":"2024-09-13T14:58:23.273750+0000","last_undegraded":"2024-09-13T14:58:23.273750+0000","last_fullsized":"2024-09-13T14:58:23.273750+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:03:22.734037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.16","version":"90'149","reported_seq":741,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.515411+0000","last_change":"2024-09-13T14:57:40.798149+0000","last_active":"2024-09-13T14:58:36.515411+0000","last_peered":"2024-09-13T14:58:36.515411+0000","last_clean":"2024-09-13T14:58:36.515411+0000","last_became_active":"2024-09-13T14:57:28.528351+0000","last_became_peered":"2024-09-13T14:57:28.528351+0000","last_unstale":"2024-09-13T14:58:36.515411+0000","last_undegraded":"2024-09-13T14:58:36.515411+0000","last_fullsized":"2024-09-13T14:58:36.515411+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":471,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:40.797796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:40.797796+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:09.769513+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":447,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.515670+0000","last_change":"2024-09-13T14:55:17.930074+0000","last_active":"2024-09-13T14:58:36.515670+0000","last_peered":"2024-09-13T14:58:36.515670+0000","last_clean":"2024-09-13T14:58:36.515670+0000","last_became_active":"2024-09-13T14:55:17.928411+0000","last_became_peered":"2024-09-13T14:55:17.928411+0000","last_unstale":"2024-09-13T14:58:36.515670+0000","last_undegraded":"2024-09-13T14:58:36.515670+0000","last_fullsized":"2024-09-13T14:58:36.515670+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:11:35.302734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":120,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.283352+0000","last_change":"2024-09-13T14:55:17.939375+0000","last_active":"2024-09-13T14:58:23.283352+0000","last_peered":"2024-09-13T14:58:23.283352+0000","last_clean":"2024-09-13T14:58:23.283352+0000","last_became_active":"2024-09-13T14:55:17.926420+0000","last_became_peered":"2024-09-13T14:55:17.926420+0000","last_unstale":"2024-09-13T14:58:23.283352+0000","last_undegraded":"2024-09-13T14:58:23.283352+0000","last_fullsized":"2024-09-13T14:58:23.283352+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:38:13.401866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4756,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.17","version":"90'149","reported_seq":762,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.274027+0000","last_change":"2024-09-13T14:57:47.798716+0000","last_active":"2024-09-13T14:58:23.274027+0000","last_peered":"2024-09-13T14:58:23.274027+0000","last_clean":"2024-09-13T14:58:23.274027+0000","last_became_active":"2024-09-13T14:57:28.526594+0000","last_became_peered":"2024-09-13T14:57:28.526594+0000","last_unstale":"2024-09-13T14:58:23.274027+0000","last_undegraded":"2024-09-13T14:58:23.274027+0000","last_fullsized":"2024-09-13T14:58:23.274027+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":471,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:47.798627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:47.798627+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:30:00.799937+0000","scrub_duration":0.0039598259999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":308,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.515609+0000","last_change":"2024-09-13T14:47:47.867935+0000","last_active":"2024-09-13T14:58:36.515609+0000","last_peered":"2024-09-13T14:58:36.515609+0000","last_clean":"2024-09-13T14:58:36.515609+0000","last_became_active":"2024-09-13T14:47:47.867616+0000","last_became_peered":"2024-09-13T14:47:47.867616+0000","last_unstale":"2024-09-13T14:58:36.515609+0000","last_undegraded":"2024-09-13T14:58:36.515609+0000","last_fullsized":"2024-09-13T14:58:36.515609+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:52:38.619916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":269,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.283247+0000","last_change":"2024-09-13T14:55:17.951412+0000","last_active":"2024-09-13T14:58:23.283247+0000","last_peered":"2024-09-13T14:58:23.283247+0000","last_clean":"2024-09-13T14:58:23.283247+0000","last_became_active":"2024-09-13T14:55:17.937147+0000","last_became_peered":"2024-09-13T14:55:17.937147+0000","last_unstale":"2024-09-13T14:58:23.283247+0000","last_undegraded":"2024-09-13T14:58:23.283247+0000","last_fullsized":"2024-09-13T14:58:23.283247+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:43.786806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"90'149","reported_seq":754,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.649492+0000","last_change":"2024-09-13T14:55:20.149252+0000","last_active":"2024-09-13T14:58:36.649492+0000","last_peered":"2024-09-13T14:58:36.649492+0000","last_clean":"2024-09-13T14:58:36.649492+0000","last_became_active":"2024-09-13T14:55:20.148898+0000","last_became_peered":"2024-09-13T14:55:20.148898+0000","last_unstale":"2024-09-13T14:58:36.649492+0000","last_undegraded":"2024-09-13T14:58:36.649492+0000","last_fullsized":"2024-09-13T14:58:36.649492+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:31.013128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:31.013128+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:11.985087+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":315,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.274219+0000","last_change":"2024-09-13T14:47:47.861842+0000","last_active":"2024-09-13T14:58:23.274219+0000","last_peered":"2024-09-13T14:58:23.274219+0000","last_clean":"2024-09-13T14:58:23.274219+0000","last_became_active":"2024-09-13T14:47:47.861492+0000","last_became_peered":"2024-09-13T14:47:47.861492+0000","last_unstale":"2024-09-13T14:58:23.274219+0000","last_undegraded":"2024-09-13T14:58:23.274219+0000","last_fullsized":"2024-09-13T14:58:23.274219+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:17.279533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":822,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.648844+0000","last_change":"2024-09-13T14:38:53.899535+0000","last_active":"2024-09-13T14:58:36.648844+0000","last_peered":"2024-09-13T14:58:36.648844+0000","last_clean":"2024-09-13T14:58:36.648844+0000","last_became_active":"2024-09-13T14:38:53.899183+0000","last_became_peered":"2024-09-13T14:38:53.899183+0000","last_unstale":"2024-09-13T14:58:36.648844+0000","last_undegraded":"2024-09-13T14:58:36.648844+0000","last_fullsized":"2024-09-13T14:58:36.648844+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:33.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"90'149","reported_seq":508,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.283425+0000","last_change":"2024-09-13T14:57:46.923759+0000","last_active":"2024-09-13T14:58:23.283425+0000","last_peered":"2024-09-13T14:58:23.283425+0000","last_clean":"2024-09-13T14:58:23.283425+0000","last_became_active":"2024-09-13T14:57:28.519678+0000","last_became_peered":"2024-09-13T14:57:28.519678+0000","last_unstale":"2024-09-13T14:58:23.283425+0000","last_undegraded":"2024-09-13T14:58:23.283425+0000","last_fullsized":"2024-09-13T14:58:23.283425+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":471,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:46.923638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:46.923638+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:59.709420+0000","scrub_duration":0.002484039,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":364,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.650782+0000","last_change":"2024-09-13T14:56:12.488981+0000","last_active":"2024-09-13T14:58:36.650782+0000","last_peered":"2024-09-13T14:58:36.650782+0000","last_clean":"2024-09-13T14:58:36.650782+0000","last_became_active":"2024-09-13T14:56:12.488552+0000","last_became_peered":"2024-09-13T14:56:12.488552+0000","last_unstale":"2024-09-13T14:58:36.650782+0000","last_undegraded":"2024-09-13T14:58:36.650782+0000","last_fullsized":"2024-09-13T14:58:36.650782+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:02:23.732498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":858,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.648830+0000","last_change":"2024-09-13T14:47:47.868624+0000","last_active":"2024-09-13T14:58:36.648830+0000","last_peered":"2024-09-13T14:58:36.648830+0000","last_clean":"2024-09-13T14:58:36.648830+0000","last_became_active":"2024-09-13T14:47:47.868031+0000","last_became_peered":"2024-09-13T14:47:47.868031+0000","last_unstale":"2024-09-13T14:58:36.648830+0000","last_undegraded":"2024-09-13T14:58:36.648830+0000","last_fullsized":"2024-09-13T14:58:36.648830+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:05:25.365023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1a","version":"90'149","reported_seq":406,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.282775+0000","last_change":"2024-09-13T14:57:43.982775+0000","last_active":"2024-09-13T14:58:23.282775+0000","last_peered":"2024-09-13T14:58:23.282775+0000","last_clean":"2024-09-13T14:58:23.282775+0000","last_became_active":"2024-09-13T14:57:28.517342+0000","last_became_peered":"2024-09-13T14:57:28.517342+0000","last_unstale":"2024-09-13T14:58:23.282775+0000","last_undegraded":"2024-09-13T14:58:23.282775+0000","last_fullsized":"2024-09-13T14:58:23.282775+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":471,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:43.982684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:43.982684+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:22:40.998656+0000","scrub_duration":0.0032555290000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":328,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.274323+0000","last_change":"2024-09-13T14:47:47.859825+0000","last_active":"2024-09-13T14:58:23.274323+0000","last_peered":"2024-09-13T14:58:23.274323+0000","last_clean":"2024-09-13T14:58:23.274323+0000","last_became_active":"2024-09-13T14:47:47.859069+0000","last_became_peered":"2024-09-13T14:47:47.859069+0000","last_unstale":"2024-09-13T14:58:23.274323+0000","last_undegraded":"2024-09-13T14:58:23.274323+0000","last_fullsized":"2024-09-13T14:58:23.274323+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:18:36.452321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":889,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.648935+0000","last_change":"2024-09-13T14:47:47.868945+0000","last_active":"2024-09-13T14:58:36.648935+0000","last_peered":"2024-09-13T14:58:36.648935+0000","last_clean":"2024-09-13T14:58:36.648935+0000","last_became_active":"2024-09-13T14:47:47.868296+0000","last_became_peered":"2024-09-13T14:47:47.868296+0000","last_unstale":"2024-09-13T14:58:36.648935+0000","last_undegraded":"2024-09-13T14:58:36.648935+0000","last_fullsized":"2024-09-13T14:58:36.648935+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:22:24.778693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"90'149","reported_seq":414,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.273851+0000","last_change":"2024-09-13T14:57:42.789225+0000","last_active":"2024-09-13T14:58:23.273851+0000","last_peered":"2024-09-13T14:58:23.273851+0000","last_clean":"2024-09-13T14:58:23.273851+0000","last_became_active":"2024-09-13T14:57:28.527605+0000","last_became_peered":"2024-09-13T14:57:28.527605+0000","last_unstale":"2024-09-13T14:58:23.273851+0000","last_undegraded":"2024-09-13T14:58:23.273851+0000","last_fullsized":"2024-09-13T14:58:23.273851+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":471,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:42.789102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:42.789102+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:00:41.558911+0000","scrub_duration":0.0083861109999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":60,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.283703+0000","last_change":"2024-09-13T14:56:12.491111+0000","last_active":"2024-09-13T14:58:23.283703+0000","last_peered":"2024-09-13T14:58:23.283703+0000","last_clean":"2024-09-13T14:58:23.283703+0000","last_became_active":"2024-09-13T14:56:12.490691+0000","last_became_peered":"2024-09-13T14:56:12.490691+0000","last_unstale":"2024-09-13T14:58:23.283703+0000","last_undegraded":"2024-09-13T14:58:23.283703+0000","last_fullsized":"2024-09-13T14:58:23.283703+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:01.439884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":315,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.514867+0000","last_change":"2024-09-13T14:55:17.929446+0000","last_active":"2024-09-13T14:58:36.514867+0000","last_peered":"2024-09-13T14:58:36.514867+0000","last_clean":"2024-09-13T14:58:36.514867+0000","last_became_active":"2024-09-13T14:55:17.928598+0000","last_became_peered":"2024-09-13T14:55:17.928598+0000","last_unstale":"2024-09-13T14:58:36.514867+0000","last_undegraded":"2024-09-13T14:58:36.514867+0000","last_fullsized":"2024-09-13T14:58:36.514867+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:27.446138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.18","version":"90'149","reported_seq":606,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.273692+0000","last_change":"2024-09-13T14:57:41.793060+0000","last_active":"2024-09-13T14:58:23.273692+0000","last_peered":"2024-09-13T14:58:23.273692+0000","last_clean":"2024-09-13T14:58:23.273692+0000","last_became_active":"2024-09-13T14:57:28.525726+0000","last_became_peered":"2024-09-13T14:57:28.525726+0000","last_unstale":"2024-09-13T14:58:23.273692+0000","last_undegraded":"2024-09-13T14:58:23.273692+0000","last_fullsized":"2024-09-13T14:58:23.273692+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":471,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:41.792934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:41.792934+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:18.870805+0000","scrub_duration":0.0079393269999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":885,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.650207+0000","last_change":"2024-09-13T14:55:17.933267+0000","last_active":"2024-09-13T14:58:36.650207+0000","last_peered":"2024-09-13T14:58:36.650207+0000","last_clean":"2024-09-13T14:58:36.650207+0000","last_became_active":"2024-09-13T14:55:17.932890+0000","last_became_peered":"2024-09-13T14:55:17.932890+0000","last_unstale":"2024-09-13T14:58:36.650207+0000","last_undegraded":"2024-09-13T14:58:36.650207+0000","last_fullsized":"2024-09-13T14:58:36.650207+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:48:59.113533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":500,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.515100+0000","last_change":"2024-09-13T14:55:17.937105+0000","last_active":"2024-09-13T14:58:36.515100+0000","last_peered":"2024-09-13T14:58:36.515100+0000","last_clean":"2024-09-13T14:58:36.515100+0000","last_became_active":"2024-09-13T14:55:17.936621+0000","last_became_peered":"2024-09-13T14:55:17.936621+0000","last_unstale":"2024-09-13T14:58:36.515100+0000","last_undegraded":"2024-09-13T14:58:36.515100+0000","last_fullsized":"2024-09-13T14:58:36.515100+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:37:28.134499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.19","version":"90'149","reported_seq":510,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.648996+0000","last_change":"2024-09-13T14:57:38.050038+0000","last_active":"2024-09-13T14:58:36.648996+0000","last_peered":"2024-09-13T14:58:36.648996+0000","last_clean":"2024-09-13T14:58:36.648996+0000","last_became_active":"2024-09-13T14:57:28.530372+0000","last_became_peered":"2024-09-13T14:57:28.530372+0000","last_unstale":"2024-09-13T14:58:36.648996+0000","last_undegraded":"2024-09-13T14:58:36.648996+0000","last_fullsized":"2024-09-13T14:58:36.648996+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":471,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:38.049970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:38.049970+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:10.878382+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":390,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.649714+0000","last_change":"2024-09-13T14:50:27.510133+0000","last_active":"2024-09-13T14:58:36.649714+0000","last_peered":"2024-09-13T14:58:36.649714+0000","last_clean":"2024-09-13T14:58:36.649714+0000","last_became_active":"2024-09-13T14:50:27.508774+0000","last_became_peered":"2024-09-13T14:50:27.508774+0000","last_unstale":"2024-09-13T14:58:36.649714+0000","last_undegraded":"2024-09-13T14:58:36.649714+0000","last_fullsized":"2024-09-13T14:58:36.649714+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:31:04.531746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":32,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.283004+0000","last_change":"2024-09-13T14:56:12.489663+0000","last_active":"2024-09-13T14:58:23.283004+0000","last_peered":"2024-09-13T14:58:23.283004+0000","last_clean":"2024-09-13T14:58:23.283004+0000","last_became_active":"2024-09-13T14:56:12.489317+0000","last_became_peered":"2024-09-13T14:56:12.489317+0000","last_unstale":"2024-09-13T14:58:23.283004+0000","last_undegraded":"2024-09-13T14:58:23.283004+0000","last_fullsized":"2024-09-13T14:58:23.283004+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:08:44.565862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1e","version":"90'149","reported_seq":421,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.650282+0000","last_change":"2024-09-13T14:57:39.990337+0000","last_active":"2024-09-13T14:58:36.650282+0000","last_peered":"2024-09-13T14:58:36.650282+0000","last_clean":"2024-09-13T14:58:36.650282+0000","last_became_active":"2024-09-13T14:57:28.525160+0000","last_became_peered":"2024-09-13T14:57:28.525160+0000","last_unstale":"2024-09-13T14:58:36.650282+0000","last_undegraded":"2024-09-13T14:58:36.650282+0000","last_fullsized":"2024-09-13T14:58:36.650282+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":471,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:39.990271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:39.990271+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:55.404672+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":444,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.649362+0000","last_change":"2024-09-13T14:55:17.932405+0000","last_active":"2024-09-13T14:58:36.649362+0000","last_peered":"2024-09-13T14:58:36.649362+0000","last_clean":"2024-09-13T14:58:36.649362+0000","last_became_active":"2024-09-13T14:55:17.932058+0000","last_became_peered":"2024-09-13T14:55:17.932058+0000","last_unstale":"2024-09-13T14:58:36.649362+0000","last_undegraded":"2024-09-13T14:58:36.649362+0000","last_fullsized":"2024-09-13T14:58:36.649362+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:05:01.826514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":328,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.274102+0000","last_change":"2024-09-13T14:55:17.946147+0000","last_active":"2024-09-13T14:58:23.274102+0000","last_peered":"2024-09-13T14:58:23.274102+0000","last_clean":"2024-09-13T14:58:23.274102+0000","last_became_active":"2024-09-13T14:55:17.931009+0000","last_became_peered":"2024-09-13T14:55:17.931009+0000","last_unstale":"2024-09-13T14:58:23.274102+0000","last_undegraded":"2024-09-13T14:58:23.274102+0000","last_fullsized":"2024-09-13T14:58:23.274102+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:08:26.717928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":341,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.649392+0000","last_change":"2024-09-13T14:55:17.936194+0000","last_active":"2024-09-13T14:58:36.649392+0000","last_peered":"2024-09-13T14:58:36.649392+0000","last_clean":"2024-09-13T14:58:36.649392+0000","last_became_active":"2024-09-13T14:55:17.935832+0000","last_became_peered":"2024-09-13T14:55:17.935832+0000","last_unstale":"2024-09-13T14:58:36.649392+0000","last_undegraded":"2024-09-13T14:58:36.649392+0000","last_fullsized":"2024-09-13T14:58:36.649392+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:40:52.745439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":315,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.274282+0000","last_change":"2024-09-13T14:55:17.941129+0000","last_active":"2024-09-13T14:58:23.274282+0000","last_peered":"2024-09-13T14:58:23.274282+0000","last_clean":"2024-09-13T14:58:23.274282+0000","last_became_active":"2024-09-13T14:55:17.930922+0000","last_became_peered":"2024-09-13T14:55:17.930922+0000","last_unstale":"2024-09-13T14:58:23.274282+0000","last_undegraded":"2024-09-13T14:58:23.274282+0000","last_fullsized":"2024-09-13T14:58:23.274282+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:06:29.610560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1c","version":"90'149","reported_seq":462,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.283228+0000","last_change":"2024-09-13T14:57:44.980830+0000","last_active":"2024-09-13T14:58:23.283228+0000","last_peered":"2024-09-13T14:58:23.283228+0000","last_clean":"2024-09-13T14:58:23.283228+0000","last_became_active":"2024-09-13T14:57:28.507948+0000","last_became_peered":"2024-09-13T14:57:28.507948+0000","last_unstale":"2024-09-13T14:58:23.283228+0000","last_undegraded":"2024-09-13T14:58:23.283228+0000","last_fullsized":"2024-09-13T14:58:23.283228+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":471,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:44.980740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:44.980740+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:52:20.504558+0000","scrub_duration":0.012428020999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":903,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.649265+0000","last_change":"2024-09-13T14:46:01.766580+0000","last_active":"2024-09-13T14:58:36.649265+0000","last_peered":"2024-09-13T14:58:36.649265+0000","last_clean":"2024-09-13T14:58:36.649265+0000","last_became_active":"2024-09-13T14:46:01.766257+0000","last_became_peered":"2024-09-13T14:46:01.766257+0000","last_unstale":"2024-09-13T14:58:36.649265+0000","last_undegraded":"2024-09-13T14:58:36.649265+0000","last_fullsized":"2024-09-13T14:58:36.649265+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:01.126109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":868,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.650341+0000","last_change":"2024-09-13T14:46:01.769647+0000","last_active":"2024-09-13T14:58:36.650341+0000","last_peered":"2024-09-13T14:58:36.650341+0000","last_clean":"2024-09-13T14:58:36.650341+0000","last_became_active":"2024-09-13T14:46:01.769228+0000","last_became_peered":"2024-09-13T14:46:01.769228+0000","last_unstale":"2024-09-13T14:58:36.650341+0000","last_undegraded":"2024-09-13T14:58:36.650341+0000","last_fullsized":"2024-09-13T14:58:36.650341+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:58.696024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1d","version":"90'149","reported_seq":451,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.514534+0000","last_change":"2024-09-13T14:57:38.761047+0000","last_active":"2024-09-13T14:58:36.514534+0000","last_peered":"2024-09-13T14:58:36.514534+0000","last_clean":"2024-09-13T14:58:36.514534+0000","last_became_active":"2024-09-13T14:57:28.529976+0000","last_became_peered":"2024-09-13T14:57:28.529976+0000","last_unstale":"2024-09-13T14:58:36.514534+0000","last_undegraded":"2024-09-13T14:58:36.514534+0000","last_fullsized":"2024-09-13T14:58:36.514534+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":471,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:38.760963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:38.760963+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:24.902108+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":426,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-13T14:58:36.514492+0000","last_change":"2024-09-13T14:55:17.929205+0000","last_active":"2024-09-13T14:58:36.514492+0000","last_peered":"2024-09-13T14:58:36.514492+0000","last_clean":"2024-09-13T14:58:36.514492+0000","last_became_active":"2024-09-13T14:55:17.928883+0000","last_became_peered":"2024-09-13T14:55:17.928883+0000","last_unstale":"2024-09-13T14:58:36.514492+0000","last_undegraded":"2024-09-13T14:58:36.514492+0000","last_fullsized":"2024-09-13T14:58:36.514492+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:27:31.570502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":387,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-13T14:58:23.274218+0000","last_change":"2024-09-13T14:56:12.492223+0000","last_active":"2024-09-13T14:58:23.274218+0000","last_peered":"2024-09-13T14:58:23.274218+0000","last_clean":"2024-09-13T14:58:23.274218+0000","last_became_active":"2024-09-13T14:56:12.490578+0000","last_became_peered":"2024-09-13T14:56:12.490578+0000","last_unstale":"2024-09-13T14:58:23.274218+0000","last_undegraded":"2024-09-13T14:58:23.274218+0000","last_fullsized":"2024-09-13T14:58:23.274218+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:11:13.623995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":211680,"num_write_kb":6773697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4957,"num_bytes_recovered":324861952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":524288,"data_stored":524288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99531,"ondisk_log_size":99531,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":6474,"num_keys_recovered":30,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":21564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25132,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":171,"num_bytes_recovered":8388646,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4619,"ondisk_log_size":4619,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234577,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":957928,"kb_used_data":6368,"kb_used_omap":1,"kb_used_meta":951550,"kb_avail":92787224,"statfs":{"total":95995035648,"available":95014117376,"internally_reserved":0,"allocated":6520832,"data_stored":4057044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":974387659},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":239,"seq":1026497183996,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":694704,"kb_used_data":10532,"kb_used_omap":8,"kb_used_meta":684151,"kb_avail":93050448,"statfs":{"total":95995035648,"available":95283658752,"internally_reserved":0,"allocated":10784768,"data_stored":8324602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8453,"internal_metadata":700571387},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757843,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1239976,"kb_used_data":10460,"kb_used_omap":1,"kb_used_meta":1229502,"kb_avail":92505176,"statfs":{"total":95995035648,"available":94725300224,"internally_reserved":0,"allocated":10711040,"data_stored":8255344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1259010502},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":128849019487,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435669,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1163360,"kb_used_data":6420,"kb_used_omap":5,"kb_used_meta":1156922,"kb_avail":92581792,"statfs":{"total":95995035648,"available":94803755008,"internally_reserved":0,"allocated":6574080,"data_stored":4125794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":1184688254},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":104,"seq":446676599198,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":493428,"kb_used_data":6376,"kb_used_omap":7,"kb_used_meta":487032,"kb_avail":93251724,"statfs":{"total":95995035648,"available":95489765376,"internally_reserved":0,"allocated":6529024,"data_stored":4066084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8010,"internal_metadata":498720950},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":99,"seq":425201762724,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":511136,"kb_used_data":10196,"kb_used_omap":7,"kb_used_meta":500920,"kb_avail":93234016,"statfs":{"total":95995035648,"available":95471632384,"internally_reserved":0,"allocated":10440704,"data_stored":7992096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7759,"internal_metadata":512942513},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926248,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1195304,"kb_used_data":6428,"kb_used_omap":7,"kb_used_meta":1188856,"kb_avail":92549848,"statfs":{"total":95995035648,"available":94771044352,"internally_reserved":0,"allocated":6582272,"data_stored":4126398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":1217388840},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6860,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":524288,"data_stored":524288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:58:43.610 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pgp_num 41 2024-09-13T14:58:43.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:43.611 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:43.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:43.712 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:43.737 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:43.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:43.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:43.839 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:43.840 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:43.841 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:43.864 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:43.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:43.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:43.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:43 smithi102 ceph-mon[115269]: pgmap v1634: 119 pgs: 119 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 560 op/s 2024-09-13T14:58:43.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:43 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1540655600' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:43.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:43 smithi102 ceph-mon[110183]: pgmap v1634: 119 pgs: 119 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 560 op/s 2024-09-13T14:58:43.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:43 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1540655600' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:43.967 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:43.993 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:43.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:43.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:43.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:43 smithi195 ceph-mon[104262]: pgmap v1634: 119 pgs: 119 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 560 op/s 2024-09-13T14:58:43.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:43 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1540655600' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:44.095 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:44.121 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:44.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:44.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:44.222 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:44.248 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:44.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:44.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:44.348 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:44.350 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:44.377 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:44.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:44.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:44.481 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:44.506 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:44.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:44.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:44.596 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:44 smithi102 ceph-mon[110183]: from='client.46871 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:58:44.597 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:44 smithi102 ceph-mon[115269]: from='client.46871 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:58:44.608 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:44.630 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:44.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:44.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:44.732 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:44.758 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:44.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:44.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:44.860 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:44.884 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:44.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:44.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:44.988 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:44.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:44 smithi195 ceph-mon[104262]: from='client.46871 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:58:45.011 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:45.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:45.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:45.114 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:45.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:45.138 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:45.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:45.239 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:45.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:45.260 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:45.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:45.361 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:45.383 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:45.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:45.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:45.485 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:45.506 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:45.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:45.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:45.608 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:45.629 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:45.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:45.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:45.658 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:45 smithi102 ceph-mon[110183]: pgmap v1635: 119 pgs: 119 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 378 op/s 2024-09-13T14:58:45.658 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:45 smithi102 ceph-mon[115269]: pgmap v1635: 119 pgs: 119 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 378 op/s 2024-09-13T14:58:45.732 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:45.755 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:45.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:45.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:45.856 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:45.878 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:45.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:45.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:45.979 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:45.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:45 smithi195 ceph-mon[104262]: pgmap v1635: 119 pgs: 119 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 378 op/s 2024-09-13T14:58:46.001 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:46.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:46.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:46.103 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:46.127 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:46.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:46.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:46.229 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:46.251 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:46.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:46.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:46.322 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:58:46.353 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:46.386 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:46.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:46.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:46.488 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:46.509 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:46.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:46.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:46.611 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:46.633 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:46.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:46.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:46.735 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:46.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:58:46.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:58:46.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:46 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:58:46.756 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:46.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:46.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:46.849 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:58:46.849 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:58:46.849 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:46 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:58:46.849 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:58:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:58:46] "GET /metrics HTTP/1.1" 200 36550 "" "Prometheus/2.51.0" 2024-09-13T14:58:46.850 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:58:46.850 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:58:46.850 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:46 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:58:46.858 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:46.880 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:46.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:46.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:46.982 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:47.004 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:47.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:47.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:47.105 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:47.128 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:47.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:47.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:47.230 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:47.253 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:47.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:47.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:47.355 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:47.377 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:47.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:47.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:47.479 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:47.502 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:47.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:47.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:47.604 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:47.626 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:47.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:47.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:47.727 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:47.749 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:47.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:47.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:47.851 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:47.872 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:47.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:47.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:47.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:47 smithi102 ceph-mon[110183]: pgmap v1636: 119 pgs: 119 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 287 op/s 2024-09-13T14:58:47.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:47 smithi102 ceph-mon[115269]: pgmap v1636: 119 pgs: 119 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 287 op/s 2024-09-13T14:58:47.974 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:47.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:47 smithi195 ceph-mon[104262]: pgmap v1636: 119 pgs: 119 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 287 op/s 2024-09-13T14:58:47.998 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:47.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:47.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:48.100 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:48.121 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:48.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:48.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:48.223 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:48.245 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:48.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:48.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:48.346 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:48.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:48.368 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:48.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:48.469 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:48.490 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:48.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:48.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:48.592 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:48.614 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:48.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:48.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:48.716 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:48.737 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:48.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:48.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:48.839 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:48.861 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:48.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:48.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:48.905 INFO:teuthology.orchestra.run.smithi102.stderr:Error EINVAL: specified pgp_num 41 > pg_num 31 2024-09-13T14:58:48.964 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:48.985 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:48.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:48.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:49.087 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:49.109 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:49.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:49.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:49.211 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:49.232 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:49.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:49.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:49.333 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:49.354 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:49.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:49.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:49.456 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:49.477 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:49.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:49.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:49.579 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:49.600 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:49.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:49.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:49.677 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:58:49.678 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [], 2024-09-13T14:58:49.678 INFO:teuthology.orchestra.run.smithi102.stdout: "num_ops": 0 2024-09-13T14:58:49.678 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:58:49.702 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:49.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:49.724 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:49.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:49.825 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:49.846 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:49.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:49.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:49.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:49 smithi102 ceph-mon[110183]: pgmap v1637: 119 pgs: 119 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 253 op/s 2024-09-13T14:58:49.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:49 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3325809769' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "41"} : dispatch 2024-09-13T14:58:49.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:49 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "41"} : dispatch 2024-09-13T14:58:49.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:49 smithi102 ceph-mon[115269]: pgmap v1637: 119 pgs: 119 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 253 op/s 2024-09-13T14:58:49.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:49 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3325809769' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "41"} : dispatch 2024-09-13T14:58:49.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:49 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "41"} : dispatch 2024-09-13T14:58:49.948 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:49.969 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:49.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:49.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:49.982 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:49 smithi195 ceph-mon[104262]: pgmap v1637: 119 pgs: 119 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 253 op/s 2024-09-13T14:58:49.983 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:49 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3325809769' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "41"} : dispatch 2024-09-13T14:58:49.983 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:49 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "41"} : dispatch 2024-09-13T14:58:50.071 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:50.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:50.095 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:50.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:50.196 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:50.217 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:50.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:50.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:50.319 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:50.340 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:50.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:50.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:50.439 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T14:58:50.442 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:50.464 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:50.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:50.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:50.555 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is unset 2024-09-13T14:58:50.566 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:50.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:50.588 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:50.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:50.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:50 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2335220354' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:58:50.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:50 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:58:50.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:50 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:58:50.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:50 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:58:50.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:50 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2335220354' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:58:50.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:50 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:58:50.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:50 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:58:50.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:50 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:58:50.689 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:50.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:50.712 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:50.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:50.813 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:50.834 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:50.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:50.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:50.871 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T14:58:50.871 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_5 2024-09-13T14:58:50.871 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_5 unique_pool_5 --yes-i-really-really-mean-it 2024-09-13T14:58:50.936 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:50.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:50.959 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:50.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:50.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:50 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2335220354' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:58:50.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:50 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:58:50.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:50 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:58:50.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:50 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:58:51.060 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:51.068 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:51.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:51.086 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:51.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:51.187 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:51.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:51.211 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:51.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:51.312 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:51.343 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:51.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:51.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:51.445 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:51.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:51.468 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:51.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:51.569 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:51.592 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:51.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:51.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:51.694 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:51.716 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:51.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:51.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:51.736 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:51 smithi102 ceph-mon[110183]: pgmap v1638: 119 pgs: 119 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 57 op/s 2024-09-13T14:58:51.737 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:51 smithi102 ceph-mon[110183]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:58:51.737 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:51 smithi102 ceph-mon[110183]: Cluster is now healthy 2024-09-13T14:58:51.737 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:51 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:58:51.737 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:51 smithi102 ceph-mon[110183]: osdmap e484: 8 total, 8 up, 7 in 2024-09-13T14:58:51.737 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:51 smithi102 ceph-mon[115269]: pgmap v1638: 119 pgs: 119 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 57 op/s 2024-09-13T14:58:51.737 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:51 smithi102 ceph-mon[115269]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:58:51.737 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:51 smithi102 ceph-mon[115269]: Cluster is now healthy 2024-09-13T14:58:51.737 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:51 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:58:51.737 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:51 smithi102 ceph-mon[115269]: osdmap e484: 8 total, 8 up, 7 in 2024-09-13T14:58:51.818 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:51.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:51.841 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:51.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:51.942 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:51.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:51.965 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:51.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:51.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:51 smithi195 ceph-mon[104262]: pgmap v1638: 119 pgs: 119 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 57 op/s 2024-09-13T14:58:51.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:51 smithi195 ceph-mon[104262]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:58:51.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:51 smithi195 ceph-mon[104262]: Cluster is now healthy 2024-09-13T14:58:51.999 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:51 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:58:51.999 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:51 smithi195 ceph-mon[104262]: osdmap e484: 8 total, 8 up, 7 in 2024-09-13T14:58:52.067 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:52.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:52.090 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:52.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:52.191 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:52.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:52.213 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:52.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:52.314 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:52.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:52.336 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:52.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:52.437 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:52.463 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:52.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:52.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:52.568 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:52.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:52.591 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:52.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:52.693 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:52.714 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:52.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:52.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:52.816 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:52.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:52.837 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:52.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:52.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:52 smithi102 ceph-mon[115269]: pgmap v1640: 119 pgs: 119 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:58:52.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:52 smithi102 ceph-mon[110183]: pgmap v1640: 119 pgs: 119 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:58:52.939 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:52.960 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:52.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:52.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:52.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:52 smithi195 ceph-mon[104262]: pgmap v1640: 119 pgs: 119 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:58:53.062 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:53.083 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:53.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:53.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:53.185 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:53.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:53.207 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:53.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:53.308 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:53.335 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:53.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:53.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:53.437 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:53.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:53.460 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:53.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:53.561 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:53.587 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:53.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:53.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:53.596 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:53.596 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:53.688 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:53.714 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:53.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:53.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:53.816 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:53.841 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:53.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:53.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:53.943 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:53.966 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:53.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:53.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:54.068 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:54.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:54.090 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:54.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:54.188 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:58:54.191 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:54.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:54.253 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:54.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:54.354 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:54.376 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:54.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:54.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:54.478 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:54.505 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:54.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:54.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:54.606 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:54.631 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:54.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:54.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:54.733 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:54.754 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:54.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:54.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:54.856 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:54.878 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:54.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:54.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:54.979 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:55.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:55.001 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:55.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:55.102 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:55.129 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:55.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:55.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:55.159 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:54 smithi102 ceph-mon[110183]: pgmap v1641: 119 pgs: 119 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:58:55.159 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:54 smithi102 ceph-mon[115269]: pgmap v1641: 119 pgs: 119 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:58:55.231 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:55.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:54 smithi195 ceph-mon[104262]: pgmap v1641: 119 pgs: 119 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:58:55.257 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:55.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:55.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:55.359 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:55.382 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:55.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:55.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:55.483 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:55.506 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:55.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:55.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:55.608 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:55.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:55.634 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:55.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:55.735 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:55.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:55.758 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:55.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:55.859 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:55.872 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 6, 7, 5, 2, 1] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T14:58:55.872 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:58:55.872 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-09-13T14:58:55.872 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.0 reset_purged_snaps_last 2024-09-13T14:58:55.880 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:55.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:55.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:55.982 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:56.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:56.004 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:56.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:56.105 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:56.126 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:56.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:56.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:56.228 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:56.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:56.251 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:56.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:56.352 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:56.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:56.375 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:56.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:56.477 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:56.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:56.501 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:56.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:56.602 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:56.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:56.625 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:56.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:56.657 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:58:56.657 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:58:56.658 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:58:56.658 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:58:56.658 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:58:56 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:58:56] "GET /metrics HTTP/1.1" 200 36545 "" "Prometheus/2.51.0" 2024-09-13T14:58:56.674 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:58:56.674 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [], 2024-09-13T14:58:56.674 INFO:teuthology.orchestra.run.smithi102.stdout: "complaint_time": 30, 2024-09-13T14:58:56.675 INFO:teuthology.orchestra.run.smithi102.stdout: "num_blocked_ops": 0 2024-09-13T14:58:56.675 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:58:56.726 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:56.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:58:56.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:58:56.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:56.749 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:56.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:56.850 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:56.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:56.873 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:56.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:56.974 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:56.996 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:56.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:56.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:57.098 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:57.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:57.120 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:57.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:57.221 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:57.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:57.245 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:57.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:58:57.346 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:57.368 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:57.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:57.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:57.470 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:57.491 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:57.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:57.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:57.593 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:57.615 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:57.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:57.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:57.645 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:57 smithi102 ceph-mon[110183]: pgmap v1642: 119 pgs: 119 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:58:57.646 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:57 smithi102 ceph-mon[115269]: pgmap v1642: 119 pgs: 119 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:58:57.667 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T14:58:57.717 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:57.738 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:57.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:57.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:57.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:57 smithi195 ceph-mon[104262]: pgmap v1642: 119 pgs: 119 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:58:57.840 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:57.862 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:57.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:57.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:57.964 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:57.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:57.986 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:57.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:58.088 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:58.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:58.110 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:58.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:58.211 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:58.233 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:58.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:58.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:58.267 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:58:58.267 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:58:58.268 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:58:58.268 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:58:58.281 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:58:58.281 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:58:58.303 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:58:58.303 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:58:58.327 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:58:58.327 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:58:58.334 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:58.348 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:58.348 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:58.349 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:58.351 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:58:58.351 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:58:58.367 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:58.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:58.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:58.380 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:58:58.380 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:58:58.398 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:58:58.398 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:58:58.469 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:58.492 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'unique_pool_5' removed 2024-09-13T14:58:58.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:58.497 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:58.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:58.507 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:58 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1750398518' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:58:58.507 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:58 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:58:58.507 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:58 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2173817108' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:58.507 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:58 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1750398518' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:58:58.508 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:58 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:58:58.508 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:58 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2173817108' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:58.598 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:58.619 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:58.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:58.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:58.721 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:58.742 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:58.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:58.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:58.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:58 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1750398518' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:58:58.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:58 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:58:58.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:58 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2173817108' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:58.844 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:58.866 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:58.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:58.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:58.969 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:58.992 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:58.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:58.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:59.094 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:59.115 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:59.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:59.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:59.217 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:59.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:59.239 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:59.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:59.340 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:59.363 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:59.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:59.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:59.465 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:59.487 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:59.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:59.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:59.589 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:59.611 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:59.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:59.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:59.713 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:59.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:59.735 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:59.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:59.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:59 smithi195 ceph-mon[104262]: pgmap v1643: 119 pgs: 119 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:58:59.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:59 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:58:59.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:58:59 smithi195 ceph-mon[104262]: osdmap e485: 8 total, 8 up, 7 in 2024-09-13T14:58:59.836 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:59.844 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:59 smithi102 ceph-mon[110183]: pgmap v1643: 119 pgs: 119 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:58:59.844 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:59 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:58:59.844 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:58:59 smithi102 ceph-mon[110183]: osdmap e485: 8 total, 8 up, 7 in 2024-09-13T14:58:59.844 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:59 smithi102 ceph-mon[115269]: pgmap v1643: 119 pgs: 119 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:58:59.844 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:59 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:58:59.844 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:58:59 smithi102 ceph-mon[115269]: osdmap e485: 8 total, 8 up, 7 in 2024-09-13T14:58:59.857 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:58:59.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:59.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:59.959 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:58:59.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:59.981 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:58:59.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:00.082 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:00.104 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:00.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:00.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:00.206 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:00.230 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:00.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:00.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:00.332 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:00.353 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:00.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:00.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:00.455 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:00.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:00.477 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:00.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:00.578 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:00.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:00.601 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:00.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:00.702 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:00.724 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:00.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:00.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:00.805 INFO:teuthology.task.sequential:In sequential, running task print... 2024-09-13T14:59:00.805 INFO:teuthology.task.print:**** done end radosbench.yaml 2024-09-13T14:59:00.805 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-09-13T14:59:00.806 INFO:tasks.workunit:Pulling workunits from ref reef 2024-09-13T14:59:00.806 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-13T14:59:00.807 DEBUG:teuthology.orchestra.run.smithi102:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-13T14:59:00.826 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:00.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:00.833 INFO:teuthology.orchestra.run.smithi102.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-13T14:59:00.833 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-13T14:59:00.848 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:00.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:00.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:00.907 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-13T14:59:00.907 DEBUG:teuthology.orchestra.run.smithi102:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-13T14:59:00.949 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:00.962 INFO:tasks.workunit:timeout=3h 2024-09-13T14:59:00.962 INFO:tasks.workunit:cleanup=True 2024-09-13T14:59:00.963 DEBUG:teuthology.orchestra.run.smithi102:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone --depth 1 --branch reef https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 2024-09-13T14:59:00.970 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:00.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:00.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:00.989 INFO:tasks.workunit.client.0.smithi102.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-13T14:59:01.071 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:01.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:01.095 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:01.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:01.196 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:01.217 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:01.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:01.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:01.319 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:01.343 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:01.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:01.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:01.445 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:01.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:01.468 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:01.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:01.569 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:01.590 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:01.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:01.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:01.692 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:01.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:01.719 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:01.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:01.745 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:01 smithi102 ceph-mon[110183]: pgmap v1645: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:01.746 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:01 smithi102 ceph-mon[115269]: pgmap v1645: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:01.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:01 smithi195 ceph-mon[104262]: pgmap v1645: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:01.821 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:01.846 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:01.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:01.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:01.949 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:01.973 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:01.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:01.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:02.075 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:02.100 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:02.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:02.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:02.202 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:02.224 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:02.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:02.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:02.236 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:59:02.327 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:02.354 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:02.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:02.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:02.456 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:02.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:02.480 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:02.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:02.581 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:02.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:02.603 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:02.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:02.704 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:02.731 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:02.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:02.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:02.833 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:02.860 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:02.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:02.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:02.961 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:02.983 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:02.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:02.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:03.085 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:03.111 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:03.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:03.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:03.253 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:03.281 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:03.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:03.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:03.300 INFO:teuthology.orchestra.run.smithi102.stdout:ERROR: (22) Invalid argument 2024-09-13T14:59:03.300 INFO:teuthology.orchestra.run.smithi102.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:59:03.382 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:03.407 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:03.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:03.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:03.509 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:03.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:03.533 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:03.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:03.634 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:03.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:03.656 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:03.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:03.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:03 smithi195 ceph-mon[104262]: pgmap v1646: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:03.758 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:03.784 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:03.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:03.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:03.886 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:03.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:03.907 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:03.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:03.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:03 smithi102 ceph-mon[110183]: pgmap v1646: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:03.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:03 smithi102 ceph-mon[115269]: pgmap v1646: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:04.008 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:04.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:04.037 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:04.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:04.138 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:04.164 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:04.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:04.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:04.266 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:04.286 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:04.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:04.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:04.388 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:04.414 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:04.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:04.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:04.516 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:04.537 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:04.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:04.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:04.639 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:04.665 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:04.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:04.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:04.767 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:04.789 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:04.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:04.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:04.891 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:04.917 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:04.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:04.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:05.019 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:05.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:05.041 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:05.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:05.142 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:05.163 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:05.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:05.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:05.268 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:05.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:05.291 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:05.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:05.392 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:05.419 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:05.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:05.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:05.521 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:05.544 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:05.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:05.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:05.646 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:05.667 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:05.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:05.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:05.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:05 smithi195 ceph-mon[104262]: pgmap v1647: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:05.769 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:05.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:05.790 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:05.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:05.882 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:05 smithi102 ceph-mon[110183]: pgmap v1647: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:05.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:05 smithi102 ceph-mon[115269]: pgmap v1647: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:05.892 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:05.905 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T14:59:05.914 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:05.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:05.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:06.016 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:06.038 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:06.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:06.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:06.140 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:06.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:06.162 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:06.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:06.263 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:06.284 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:06.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:06.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:06.300 DEBUG:teuthology.orchestra.run.smithi102:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-13T14:59:06.372 INFO:tasks.workunit.client.0.smithi102.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-13T14:59:06.374 INFO:tasks.workunit.client.0.smithi102.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-13T14:59:06.374 INFO:tasks.workunit.client.0.smithi102.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-13T14:59:06.389 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:06.410 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:06.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:06.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:06.512 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:06.534 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:06.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:06.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:06.579 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:59:06 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:59:06] "GET /metrics HTTP/1.1" 200 36545 "" "Prometheus/2.51.0" 2024-09-13T14:59:06.636 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:06.657 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:06.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:06.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:06.665 INFO:tasks.workunit.client.0.smithi102.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-13T14:59:06.716 INFO:tasks.workunit.client.0.smithi102.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-13T14:59:06.755 INFO:tasks.workunit.client.0.smithi102.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-13T14:59:06.757 INFO:tasks.workunit.client.0.smithi102.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-13T14:59:06.757 INFO:tasks.workunit.client.0.smithi102.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-13T14:59:06.759 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:06.781 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:06.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:06.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:06.795 INFO:tasks.workunit.client.0.smithi102.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-13T14:59:06.832 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-09-13T14:59:06.833 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-13T14:59:06.883 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:06.907 INFO:tasks.workunit:Running workunits matching cls/test_cls_rbd.sh on client.0... 2024-09-13T14:59:06.908 INFO:tasks.workunit:Running workunit cls/test_cls_rbd.sh... 2024-09-13T14:59:06.908 DEBUG:teuthology.orchestra.run.smithi102:workunit test cls/test_cls_rbd.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=reef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh 2024-09-13T14:59:06.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:06.910 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:06.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:07.011 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:07.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:07.034 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:07.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:07.135 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:07.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:07.157 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:07.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:07.196 INFO:tasks.workunit.client.0.smithi102.stdout:Running main() from gmock_main.cc 2024-09-13T14:59:07.196 INFO:tasks.workunit.client.0.smithi102.stdout:[==========] Running 70 tests from 1 test suite. 2024-09-13T14:59:07.196 INFO:tasks.workunit.client.0.smithi102.stdout:[----------] Global test environment set-up. 2024-09-13T14:59:07.196 INFO:tasks.workunit.client.0.smithi102.stdout:[----------] 70 tests from TestClsRbd 2024-09-13T14:59:07.258 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:07.280 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:07.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:07.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:07.382 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:07.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:07.405 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:07.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:07.506 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:07.527 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:07.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:07.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:07.629 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:07.652 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:07.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:07.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:07.745 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:07 smithi102 ceph-mon[110183]: pgmap v1648: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:07.746 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:07 smithi102 ceph-mon[115269]: pgmap v1648: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:07.754 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:07.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:07.776 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:07.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:07.877 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:07.902 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:07.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:07.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:07.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:07 smithi195 ceph-mon[104262]: pgmap v1648: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:08.003 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:08.026 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:08.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:08.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:08.127 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:08.153 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:08.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:08.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:08.255 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:08.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:08.277 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:08.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:08.307 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:08.378 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:08.404 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:08.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:08.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:08.501 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is set 2024-09-13T14:59:08.501 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.get_all_features 2024-09-13T14:59:08.505 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:08.535 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.get_all_features (35 ms) 2024-09-13T14:59:08.535 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.copyup 2024-09-13T14:59:08.540 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:08.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:08.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:08.642 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:08.664 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:08.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:08.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:08.725 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.copyup (190 ms) 2024-09-13T14:59:08.726 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.sparse_copyup 2024-09-13T14:59:08.738 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.sparse_copyup (12 ms) 2024-09-13T14:59:08.738 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.get_and_set_id 2024-09-13T14:59:08.744 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.get_and_set_id (7 ms) 2024-09-13T14:59:08.745 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.add_remove_child 2024-09-13T14:59:08.752 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:08 smithi102 ceph-mon[110183]: osdmap e486: 8 total, 8 up, 7 in 2024-09-13T14:59:08.753 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:08 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3675832364' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-407841-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-13T14:59:08.753 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:08 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-407841-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-13T14:59:08.753 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:08 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2773193294' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:59:08.753 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:08 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:59:08.753 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:08 smithi102 ceph-mon[110183]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:59:08.753 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:08 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-407841-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-13T14:59:08.754 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:08 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:59:08.754 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:08 smithi102 ceph-mon[110183]: osdmap e487: 8 total, 8 up, 7 in 2024-09-13T14:59:08.754 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:08 smithi102 ceph-mon[115269]: osdmap e486: 8 total, 8 up, 7 in 2024-09-13T14:59:08.754 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:08 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3675832364' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-407841-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-13T14:59:08.755 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:08 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-407841-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-13T14:59:08.755 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:08 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2773193294' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:59:08.755 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:08 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:59:08.755 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:08 smithi102 ceph-mon[115269]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:59:08.755 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:08 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-407841-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-13T14:59:08.755 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:08 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:59:08.755 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:08 smithi102 ceph-mon[115269]: osdmap e487: 8 total, 8 up, 7 in 2024-09-13T14:59:08.756 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.add_remove_child (10 ms) 2024-09-13T14:59:08.756 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.directory_methods 2024-09-13T14:59:08.767 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:08.783 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.directory_methods (29 ms) 2024-09-13T14:59:08.783 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.create 2024-09-13T14:59:08.789 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:08.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:08.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:08.812 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.create (29 ms) 2024-09-13T14:59:08.812 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.get_features 2024-09-13T14:59:08.814 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.get_features (1 ms) 2024-09-13T14:59:08.814 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.get_object_prefix 2024-09-13T14:59:08.816 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.get_object_prefix (2 ms) 2024-09-13T14:59:08.817 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.get_create_timestamp 2024-09-13T14:59:08.817 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.get_create_timestamp (1 ms) 2024-09-13T14:59:08.817 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.get_access_timestamp 2024-09-13T14:59:08.818 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.get_access_timestamp (2 ms) 2024-09-13T14:59:08.818 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.get_modify_timestamp 2024-09-13T14:59:08.819 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.get_modify_timestamp (1 ms) 2024-09-13T14:59:08.820 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.get_data_pool 2024-09-13T14:59:08.822 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.get_data_pool (3 ms) 2024-09-13T14:59:08.822 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.get_size 2024-09-13T14:59:08.826 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.get_size (3 ms) 2024-09-13T14:59:08.826 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.set_size 2024-09-13T14:59:08.831 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.set_size (5 ms) 2024-09-13T14:59:08.831 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.protection_status 2024-09-13T14:59:08.848 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.protection_status (17 ms) 2024-09-13T14:59:08.848 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.snapshot_limits 2024-09-13T14:59:08.854 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.snapshot_limits (7 ms) 2024-09-13T14:59:08.854 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.parents_v1 2024-09-13T14:59:08.891 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:08.894 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.parents_v1 (39 ms) 2024-09-13T14:59:08.894 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.parents_v2 2024-09-13T14:59:08.912 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:08.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:08.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:08.920 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.parents_v2 (27 ms) 2024-09-13T14:59:08.920 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.snapshots 2024-09-13T14:59:08.939 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.snapshots (18 ms) 2024-09-13T14:59:08.939 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.snapid_race 2024-09-13T14:59:08.942 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.snapid_race (4 ms) 2024-09-13T14:59:08.942 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.stripingv2 2024-09-13T14:59:08.952 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.stripingv2 (8 ms) 2024-09-13T14:59:08.953 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.object_map_save 2024-09-13T14:59:08.953 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.object_map_save (2 ms) 2024-09-13T14:59:08.953 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.object_map_resize 2024-09-13T14:59:08.957 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.object_map_resize (4 ms) 2024-09-13T14:59:08.957 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.object_map_update 2024-09-13T14:59:08.964 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.object_map_update (8 ms) 2024-09-13T14:59:08.964 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.object_map_load_enoent 2024-09-13T14:59:08.964 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.object_map_load_enoent (0 ms) 2024-09-13T14:59:08.964 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.object_map_snap_add 2024-09-13T14:59:08.970 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.object_map_snap_add (5 ms) 2024-09-13T14:59:08.970 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.object_map_snap_remove 2024-09-13T14:59:08.975 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.object_map_snap_remove (5 ms) 2024-09-13T14:59:08.975 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.flags 2024-09-13T14:59:08.979 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.flags (5 ms) 2024-09-13T14:59:08.979 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.metadata 2024-09-13T14:59:08.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:08 smithi195 ceph-mon[104262]: osdmap e486: 8 total, 8 up, 7 in 2024-09-13T14:59:08.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:08 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3675832364' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-407841-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-13T14:59:08.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:08 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-407841-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-13T14:59:08.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:08 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2773193294' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:59:08.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:08 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:59:08.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:08 smithi195 ceph-mon[104262]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:59:08.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:08 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-407841-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-13T14:59:08.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:08 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:59:08.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:08 smithi195 ceph-mon[104262]: osdmap e487: 8 total, 8 up, 7 in 2024-09-13T14:59:09.003 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.metadata (24 ms) 2024-09-13T14:59:09.003 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.set_features 2024-09-13T14:59:09.010 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.set_features (7 ms) 2024-09-13T14:59:09.010 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.mirror 2024-09-13T14:59:09.014 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:09.039 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:09.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:09.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:09.045 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.mirror (35 ms) 2024-09-13T14:59:09.045 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.mirror_image 2024-09-13T14:59:09.060 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.mirror_image (15 ms) 2024-09-13T14:59:09.060 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.mirror_image_status 2024-09-13T14:59:09.141 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:09.163 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:09.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:09.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:09.264 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:09.289 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:09.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:09.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:09.391 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:09.412 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:09.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:09.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:09.514 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:09.540 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:09.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:09.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:09.642 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:09.666 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:09.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:09.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:09.769 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:09.794 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:09.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:09.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:09.896 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:09.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:09 smithi102 ceph-mon[110183]: pgmap v1650: 135 pgs: 32 unknown, 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:09.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:09 smithi102 ceph-mon[110183]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:59:09.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:09 smithi102 ceph-mon[110183]: osdmap e488: 8 total, 8 up, 7 in 2024-09-13T14:59:09.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:09 smithi102 ceph-mon[115269]: pgmap v1650: 135 pgs: 32 unknown, 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:09.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:09 smithi102 ceph-mon[115269]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:59:09.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:09 smithi102 ceph-mon[115269]: osdmap e488: 8 total, 8 up, 7 in 2024-09-13T14:59:09.920 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:09.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:09.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:09.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:09 smithi195 ceph-mon[104262]: pgmap v1650: 135 pgs: 32 unknown, 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:09.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:09 smithi195 ceph-mon[104262]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:59:09.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:09 smithi195 ceph-mon[104262]: osdmap e488: 8 total, 8 up, 7 in 2024-09-13T14:59:10.022 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:10.049 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:10.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:10.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:10.151 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:10.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:10.180 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:10.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:10.282 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:10.308 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:10.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:10.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:10.411 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:10.436 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:10.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:10.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:10.538 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:10.563 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:10.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:10.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:10.665 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:10.688 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:10.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:10.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:10.790 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:10.813 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:10.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:10.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:10.915 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:10.937 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:10.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:10.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:11.039 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:11.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:11.068 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:11.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:11.169 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:11.194 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:11.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:11.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:11.296 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:11.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:11.319 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:11.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:11.421 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:11.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:11.444 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:11.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:11.493 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:11.545 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:11.568 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:11.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:11.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:11.670 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:11.692 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:11.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:11.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:11.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:11 smithi195 ceph-mon[104262]: pgmap v1653: 135 pgs: 23 unknown, 112 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 13 KiB/s rd, 7.0 KiB/s wr, 13 op/s 2024-09-13T14:59:11.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:11 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:59:11.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:11 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:59:11.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:11 smithi195 ceph-mon[104262]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:59:11.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:11 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:59:11.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:11 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:59:11.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:11 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:59:11.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:11 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:59:11.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:11 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:59:11.794 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:11.806 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.mirror_image_status (2745 ms) 2024-09-13T14:59:11.806 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.mirror_image_map 2024-09-13T14:59:11.817 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:11.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:11.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:11.891 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.mirror_image_map (85 ms) 2024-09-13T14:59:11.891 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.mirror_instances 2024-09-13T14:59:11.898 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.mirror_instances (8 ms) 2024-09-13T14:59:11.898 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.mirror_snapshot 2024-09-13T14:59:11.911 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.mirror_snapshot (12 ms) 2024-09-13T14:59:11.911 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.group_dir_list 2024-09-13T14:59:11.913 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.group_dir_list (3 ms) 2024-09-13T14:59:11.913 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.group_dir_add 2024-09-13T14:59:11.915 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.group_dir_add (2 ms) 2024-09-13T14:59:11.915 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.dir_add_already_existing 2024-09-13T14:59:11.918 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.dir_add_already_existing (2 ms) 2024-09-13T14:59:11.918 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.group_dir_rename 2024-09-13T14:59:11.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:11 smithi102 ceph-mon[110183]: pgmap v1653: 135 pgs: 23 unknown, 112 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 13 KiB/s rd, 7.0 KiB/s wr, 13 op/s 2024-09-13T14:59:11.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:11 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:59:11.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:11 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:59:11.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:11 smithi102 ceph-mon[110183]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:59:11.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:11 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:59:11.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:11 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:59:11.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:11 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:59:11.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:11 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:59:11.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:11 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:59:11.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:11 smithi102 ceph-mon[115269]: pgmap v1653: 135 pgs: 23 unknown, 112 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 13 KiB/s rd, 7.0 KiB/s wr, 13 op/s 2024-09-13T14:59:11.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:11 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:59:11.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:11 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:59:11.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:11 smithi102 ceph-mon[115269]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:59:11.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:11 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:59:11.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:11 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:59:11.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:11 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:59:11.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:11 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:59:11.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:11 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:59:11.920 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:11.925 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.group_dir_rename (7 ms) 2024-09-13T14:59:11.925 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.group_dir_remove 2024-09-13T14:59:11.928 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.group_dir_remove (4 ms) 2024-09-13T14:59:11.928 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.group_dir_remove_missing 2024-09-13T14:59:11.933 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.group_dir_remove_missing (4 ms) 2024-09-13T14:59:11.933 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.group_image_add 2024-09-13T14:59:11.935 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.group_image_add (2 ms) 2024-09-13T14:59:11.935 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.group_image_remove 2024-09-13T14:59:11.938 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.group_image_remove (3 ms) 2024-09-13T14:59:11.938 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.group_image_list 2024-09-13T14:59:11.940 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.group_image_list (3 ms) 2024-09-13T14:59:11.940 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.group_image_clean 2024-09-13T14:59:11.942 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:11.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:11.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:11.944 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.group_image_clean (4 ms) 2024-09-13T14:59:11.944 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.image_group_add 2024-09-13T14:59:11.946 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.image_group_add (2 ms) 2024-09-13T14:59:11.947 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.image_group_remove 2024-09-13T14:59:11.950 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.image_group_remove (3 ms) 2024-09-13T14:59:11.950 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.image_group_get 2024-09-13T14:59:11.952 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.image_group_get (2 ms) 2024-09-13T14:59:11.952 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.group_snap_set_empty_name 2024-09-13T14:59:11.953 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.group_snap_set_empty_name (2 ms) 2024-09-13T14:59:11.953 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.group_snap_set_empty_id 2024-09-13T14:59:11.955 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.group_snap_set_empty_id (2 ms) 2024-09-13T14:59:11.956 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.group_snap_set_duplicate_id 2024-09-13T14:59:11.959 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.group_snap_set_duplicate_id (4 ms) 2024-09-13T14:59:11.959 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.group_snap_set_duplicate_name 2024-09-13T14:59:11.963 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.group_snap_set_duplicate_name (4 ms) 2024-09-13T14:59:11.964 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.group_snap_set 2024-09-13T14:59:11.966 INFO:tasks.workunit.client.0.smithi102.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-843-g2c01458a/rpm/el9/BUILD/ceph-18.2.4-843-g2c01458a/src/test/cls_rbd/test_cls_rbd.cc:2681: Failure 2024-09-13T14:59:11.966 INFO:tasks.workunit.client.0.smithi102.stdout:Expected equality of these values: 2024-09-13T14:59:11.966 INFO:tasks.workunit.client.0.smithi102.stdout: 1U 2024-09-13T14:59:11.966 INFO:tasks.workunit.client.0.smithi102.stdout: Which is: 1 2024-09-13T14:59:11.966 INFO:tasks.workunit.client.0.smithi102.stdout: keys.size() 2024-09-13T14:59:11.966 INFO:tasks.workunit.client.0.smithi102.stdout: Which is: 3 2024-09-13T14:59:11.966 INFO:tasks.workunit.client.0.smithi102.stdout:[ FAILED ] TestClsRbd.group_snap_set (3 ms) 2024-09-13T14:59:11.967 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.group_snap_list 2024-09-13T14:59:11.971 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.group_snap_list (4 ms) 2024-09-13T14:59:11.971 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.group_snap_list_max_return 2024-09-13T14:59:11.986 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.group_snap_list_max_return (16 ms) 2024-09-13T14:59:11.986 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.group_snap_list_max_read 2024-09-13T14:59:12.044 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:12.065 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:12.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:12.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:12.167 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:12.189 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.group_snap_list_max_read (202 ms) 2024-09-13T14:59:12.189 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.group_snap_remove 2024-09-13T14:59:12.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:12.189 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:12.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:12.191 INFO:tasks.workunit.client.0.smithi102.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-843-g2c01458a/rpm/el9/BUILD/ceph-18.2.4-843-g2c01458a/src/test/cls_rbd/test_cls_rbd.cc:2792: Failure 2024-09-13T14:59:12.191 INFO:tasks.workunit.client.0.smithi102.stdout:Expected equality of these values: 2024-09-13T14:59:12.191 INFO:tasks.workunit.client.0.smithi102.stdout: 1U 2024-09-13T14:59:12.192 INFO:tasks.workunit.client.0.smithi102.stdout: Which is: 1 2024-09-13T14:59:12.192 INFO:tasks.workunit.client.0.smithi102.stdout: keys.size() 2024-09-13T14:59:12.192 INFO:tasks.workunit.client.0.smithi102.stdout: Which is: 3 2024-09-13T14:59:12.192 INFO:tasks.workunit.client.0.smithi102.stdout:[ FAILED ] TestClsRbd.group_snap_remove (3 ms) 2024-09-13T14:59:12.192 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.group_snap_get_by_id 2024-09-13T14:59:12.194 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.group_snap_get_by_id (3 ms) 2024-09-13T14:59:12.194 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.trash_methods 2024-09-13T14:59:12.202 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.trash_methods (8 ms) 2024-09-13T14:59:12.202 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.op_features 2024-09-13T14:59:12.209 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.op_features (7 ms) 2024-09-13T14:59:12.210 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.clone_parent 2024-09-13T14:59:12.229 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.clone_parent (19 ms) 2024-09-13T14:59:12.229 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.clone_parent_ns 2024-09-13T14:59:12.238 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.clone_parent_ns (9 ms) 2024-09-13T14:59:12.238 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.clone_child 2024-09-13T14:59:12.248 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.clone_child (11 ms) 2024-09-13T14:59:12.248 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.namespace_methods 2024-09-13T14:59:12.254 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.namespace_methods (6 ms) 2024-09-13T14:59:12.255 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.migration 2024-09-13T14:59:12.264 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.migration (9 ms) 2024-09-13T14:59:12.264 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.migration_v1 2024-09-13T14:59:12.272 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.migration_v1 (9 ms) 2024-09-13T14:59:12.272 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.assert_snapc_seq 2024-09-13T14:59:12.290 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:12.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:12.312 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:12.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:12.413 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:12.434 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:12.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:12.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:12.536 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:12.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:12.559 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:12.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:12.644 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.3 reset_purged_snaps_last 2024-09-13T14:59:12.660 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:12.681 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:12.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:12.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:12.783 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:12.804 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:12.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:12.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:12.906 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:12.928 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:12.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:12.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:13.030 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:13.051 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:13.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:13.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:13.153 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:13.176 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:13.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:13.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:13.278 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:13.300 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:13.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:13.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:13.402 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:13.424 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:13.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:13.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:13.480 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.assert_snapc_seq (1208 ms) 2024-09-13T14:59:13.481 INFO:tasks.workunit.client.0.smithi102.stdout:[ RUN ] TestClsRbd.sparsify 2024-09-13T14:59:13.510 INFO:tasks.workunit.client.0.smithi102.stdout:[ OK ] TestClsRbd.sparsify (29 ms) 2024-09-13T14:59:13.526 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:13.539 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0: {} 2024-09-13T14:59:13.539 INFO:teuthology.orchestra.run.smithi102.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:59:13.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:13.550 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:13.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:13.557 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1: {} 2024-09-13T14:59:13.557 INFO:teuthology.orchestra.run.smithi102.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:59:13.573 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:13.574 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2: {} 2024-09-13T14:59:13.574 INFO:teuthology.orchestra.run.smithi102.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:59:13.591 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3: {} 2024-09-13T14:59:13.592 INFO:teuthology.orchestra.run.smithi102.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:59:13.609 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4: {} 2024-09-13T14:59:13.609 INFO:teuthology.orchestra.run.smithi102.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:59:13.625 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5: {} 2024-09-13T14:59:13.625 INFO:teuthology.orchestra.run.smithi102.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:59:13.642 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6: {} 2024-09-13T14:59:13.643 INFO:teuthology.orchestra.run.smithi102.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:59:13.651 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:13.659 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7: {} 2024-09-13T14:59:13.659 INFO:teuthology.orchestra.run.smithi102.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:59:13.673 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:13.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:13.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:13.693 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:13 smithi102 ceph-mon[110183]: pgmap v1654: 135 pgs: 11 unknown, 124 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1.4 MiB/s rd, 538 KiB/s wr, 2.02k op/s 2024-09-13T14:59:13.693 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:13 smithi102 ceph-mon[110183]: osdmap e489: 8 total, 8 up, 7 in 2024-09-13T14:59:13.693 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:13 smithi102 ceph-mon[115269]: pgmap v1654: 135 pgs: 11 unknown, 124 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1.4 MiB/s rd, 538 KiB/s wr, 2.02k op/s 2024-09-13T14:59:13.693 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:13 smithi102 ceph-mon[115269]: osdmap e489: 8 total, 8 up, 7 in 2024-09-13T14:59:13.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:13 smithi195 ceph-mon[104262]: pgmap v1654: 135 pgs: 11 unknown, 124 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1.4 MiB/s rd, 538 KiB/s wr, 2.02k op/s 2024-09-13T14:59:13.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:13 smithi195 ceph-mon[104262]: osdmap e489: 8 total, 8 up, 7 in 2024-09-13T14:59:13.775 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:13.797 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:13.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:13.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:13.899 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:13.920 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:13.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:13.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:13.987 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:59:14.022 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:14.045 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:14.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:14.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:14.146 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:14.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:14.169 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:14.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:14.270 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:14.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:14.292 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:14.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:14.393 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:14.414 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:14.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:14.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:14.498 INFO:tasks.workunit.client.0.smithi102.stdout:[----------] 70 tests from TestClsRbd (5009 ms total) 2024-09-13T14:59:14.498 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-09-13T14:59:14.498 INFO:tasks.workunit.client.0.smithi102.stdout:[----------] Global test environment tear-down 2024-09-13T14:59:14.498 INFO:tasks.workunit.client.0.smithi102.stdout:[==========] 70 tests from 1 test suite ran. (7301 ms total) 2024-09-13T14:59:14.498 INFO:tasks.workunit.client.0.smithi102.stdout:[ PASSED ] 68 tests. 2024-09-13T14:59:14.498 INFO:tasks.workunit.client.0.smithi102.stdout:[ FAILED ] 2 tests, listed below: 2024-09-13T14:59:14.498 INFO:tasks.workunit.client.0.smithi102.stdout:[ FAILED ] TestClsRbd.group_snap_set 2024-09-13T14:59:14.498 INFO:tasks.workunit.client.0.smithi102.stdout:[ FAILED ] TestClsRbd.group_snap_remove 2024-09-13T14:59:14.498 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-09-13T14:59:14.498 INFO:tasks.workunit.client.0.smithi102.stdout: 2 FAILED TESTS 2024-09-13T14:59:14.517 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:14.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:14.525 INFO:tasks.workunit:Stopping ['cls/test_cls_rbd.sh'] on client.0... 2024-09-13T14:59:14.525 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-13T14:59:14.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:14.539 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:14.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:14.558 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:14 smithi102 ceph-mon[110183]: osdmap e490: 8 total, 8 up, 7 in 2024-09-13T14:59:14.559 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:14 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1006251694' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:59:14.559 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:14 smithi102 ceph-mon[115269]: osdmap e490: 8 total, 8 up, 7 in 2024-09-13T14:59:14.559 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:14 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1006251694' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:59:14.640 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:14.661 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:14.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:14.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:14.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:14 smithi195 ceph-mon[104262]: osdmap e490: 8 total, 8 up, 7 in 2024-09-13T14:59:14.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:14 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1006251694' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:59:14.763 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:14.786 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:14.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:14.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:14.889 DEBUG:teuthology.orchestra.run.smithi195:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:14.915 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: docker: command not found 2024-09-13T14:59:14.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:14.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi195 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:15.017 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:15.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:15.039 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:15.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:15.140 DEBUG:teuthology.orchestra.run.smithi102:> sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:59:15.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:15.162 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: docker: command not found 2024-09-13T14:59:15.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi102 with status 1: 'sudo docker kill -s 1 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:59:15.257 INFO:tasks.thrashosds:joining thrashosds 2024-09-13T14:59:15.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:15 smithi102 ceph-mon[110183]: pgmap v1657: 135 pgs: 135 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 3.5 MiB/s rd, 1.4 MiB/s wr, 2.58k op/s 2024-09-13T14:59:15.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:15 smithi102 ceph-mon[110183]: osdmap e491: 8 total, 8 up, 7 in 2024-09-13T14:59:15.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:15 smithi102 ceph-mon[115269]: pgmap v1657: 135 pgs: 135 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 3.5 MiB/s rd, 1.4 MiB/s wr, 2.58k op/s 2024-09-13T14:59:15.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:15 smithi102 ceph-mon[115269]: osdmap e491: 8 total, 8 up, 7 in 2024-09-13T14:59:15.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:15 smithi195 ceph-mon[104262]: pgmap v1657: 135 pgs: 135 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 3.5 MiB/s rd, 1.4 MiB/s wr, 2.58k op/s 2024-09-13T14:59:15.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:15 smithi195 ceph-mon[104262]: osdmap e491: 8 total, 8 up, 7 in 2024-09-13T14:59:16.755 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:59:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:59:16] "GET /metrics HTTP/1.1" 200 36553 "" "Prometheus/2.51.0" 2024-09-13T14:59:17.326 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:17.342 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:59:17.342 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [], 2024-09-13T14:59:17.344 INFO:teuthology.orchestra.run.smithi102.stdout: "num_ops": 0 2024-09-13T14:59:17.345 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:59:17.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:17 smithi102 ceph-mon[110183]: pgmap v1659: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:17.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:17 smithi102 ceph-mon[115269]: pgmap v1659: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:17.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:17 smithi195 ceph-mon[104262]: pgmap v1659: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:18.641 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T14:59:19.069 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.6 reset_purged_snaps_last 2024-09-13T14:59:19.918 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:19 smithi102 ceph-mon[115269]: pgmap v1660: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:19.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:19 smithi102 ceph-mon[110183]: pgmap v1660: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:19.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:19 smithi195 ceph-mon[104262]: pgmap v1660: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:20.650 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:20.683 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:21.512 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is set 2024-09-13T14:59:21.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:21 smithi102 ceph-mon[110183]: pgmap v1661: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:21.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:21 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3619384417' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:21.654 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:21 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:21.654 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:21 smithi102 ceph-mon[115269]: pgmap v1661: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:21.654 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:21 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3619384417' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:21.654 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:21 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:21.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:21 smithi195 ceph-mon[104262]: pgmap v1661: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:21.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:21 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3619384417' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:21.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:21 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:22.554 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:22 smithi102 ceph-mon[110183]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:59:22.555 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:22 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:59:22.555 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:22 smithi102 ceph-mon[110183]: osdmap e492: 8 total, 8 up, 7 in 2024-09-13T14:59:22.555 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:22 smithi102 ceph-mon[115269]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:59:22.555 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:22 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:59:22.555 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:22 smithi102 ceph-mon[115269]: osdmap e492: 8 total, 8 up, 7 in 2024-09-13T14:59:22.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:22 smithi195 ceph-mon[104262]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:59:22.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:22 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:59:22.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:22 smithi195 ceph-mon[104262]: osdmap e492: 8 total, 8 up, 7 in 2024-09-13T14:59:23.767 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:23 smithi102 ceph-mon[110183]: pgmap v1663: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:23.767 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:23 smithi102 ceph-mon[115269]: pgmap v1663: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:23.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:23 smithi195 ceph-mon[104262]: pgmap v1663: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:24.588 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.7 reset_purged_snaps_last 2024-09-13T14:59:24.958 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:59:25.286 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:59:25.286 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [], 2024-09-13T14:59:25.287 INFO:teuthology.orchestra.run.smithi102.stdout: "complaint_time": 30, 2024-09-13T14:59:25.287 INFO:teuthology.orchestra.run.smithi102.stdout: "num_blocked_ops": 0 2024-09-13T14:59:25.287 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:59:25.800 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:25 smithi195 ceph-mon[104262]: pgmap v1664: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:25.800 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:25 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:59:25.800 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:25 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:59:25.836 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:25.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:25 smithi102 ceph-mon[110183]: pgmap v1664: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:25.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:25 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:59:25.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:25 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:59:25.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:25 smithi102 ceph-mon[115269]: pgmap v1664: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:25.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:25 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:59:25.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:25 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:59:26.428 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:26.450 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T14:59:26.682 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:59:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:59:26] "GET /metrics HTTP/1.1" 200 34149 "" "Prometheus/2.51.0" 2024-09-13T14:59:27.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:27 smithi102 ceph-mon[110183]: pgmap v1665: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:27.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:27 smithi102 ceph-mon[115269]: pgmap v1665: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:27.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:27 smithi195 ceph-mon[104262]: pgmap v1665: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:28.326 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:29.872 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:29 smithi102 ceph-mon[110183]: pgmap v1666: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:29.872 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:29 smithi102 ceph-mon[115269]: pgmap v1666: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:29.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:29 smithi195 ceph-mon[104262]: pgmap v1666: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:30.531 INFO:teuthology.orchestra.run.smithi102.stderr:noscrub is unset 2024-09-13T14:59:30.872 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:30 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2167864791' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:59:30.872 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:30 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:59:30.872 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:30 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2167864791' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:59:30.873 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:30 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:59:30.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:30 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2167864791' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:59:30.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:30 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:59:31.875 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:31 smithi102 ceph-mon[115269]: pgmap v1667: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:31.875 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:31 smithi102 ceph-mon[115269]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:59:31.875 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:31 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:59:31.876 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:31 smithi102 ceph-mon[115269]: osdmap e493: 8 total, 8 up, 7 in 2024-09-13T14:59:31.876 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:31 smithi102 ceph-mon[110183]: pgmap v1667: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:31.876 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:31 smithi102 ceph-mon[110183]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:59:31.876 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:31 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:59:31.876 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:31 smithi102 ceph-mon[110183]: osdmap e493: 8 total, 8 up, 7 in 2024-09-13T14:59:31.904 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:59:31.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:31 smithi195 ceph-mon[104262]: pgmap v1667: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:31.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:31 smithi195 ceph-mon[104262]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:59:31.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:31 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:59:31.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:31 smithi195 ceph-mon[104262]: osdmap e493: 8 total, 8 up, 7 in 2024-09-13T14:59:32.816 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-09-13T14:59:32.816 INFO:teuthology.orchestra.run.smithi102.stdout: "size": 20, 2024-09-13T14:59:32.816 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 600, 2024-09-13T14:59:32.816 INFO:teuthology.orchestra.run.smithi102.stdout: "ops": [ 2024-09-13T14:59:32.816 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.817 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:249803 8.f 8:f189cad3:::benchmark_data_smithi102_313275_object119216:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-09-13T14:59:32.817 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:50:13.153033+0000", 2024-09-13T14:59:32.817 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 559.66044249499998, 2024-09-13T14:59:32.817 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4494403890000001, 2024-09-13T14:59:32.817 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:59:32.817 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:59:32.817 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "reached pg", 2024-09-13T14:59:32.817 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:59:32.817 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:59:32.817 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:59:32.817 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 249803 2024-09-13T14:59:32.817 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.818 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:59:32.818 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.818 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:59:32.818 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.153033+0000", 2024-09-13T14:59:32.818 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:59:32.818 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.818 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.818 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:59:32.818 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.153033+0000", 2024-09-13T14:59:32.818 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:59:32.818 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.818 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.818 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:59:32.818 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.153034+0000", 2024-09-13T14:59:32.819 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3030000000000001e-06 2024-09-13T14:59:32.819 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.819 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.819 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:59:32.819 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.153038+0000", 2024-09-13T14:59:32.819 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.5880000000000002e-06 2024-09-13T14:59:32.819 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.819 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.819 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:59:32.819 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.153046+0000", 2024-09-13T14:59:32.819 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.5180000000000004e-06 2024-09-13T14:59:32.819 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.820 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.820 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:59:32.820 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.153052+0000", 2024-09-13T14:59:32.820 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.2800000000000003e-06 2024-09-13T14:59:32.820 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.820 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.820 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:59:32.820 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.154378+0000", 2024-09-13T14:59:32.820 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0013263299999999999 2024-09-13T14:59:32.820 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.820 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.820 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:59:32.820 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.154654+0000", 2024-09-13T14:59:32.821 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00027594000000000002 2024-09-13T14:59:32.821 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.821 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.821 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,2,4", 2024-09-13T14:59:32.821 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.155042+0000", 2024-09-13T14:59:32.821 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00038806700000000002 2024-09-13T14:59:32.821 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.821 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.821 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:32.821 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.170794+0000", 2024-09-13T14:59:32.821 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.015751654 2024-09-13T14:59:32.821 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.822 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.822 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:32.822 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.191965+0000", 2024-09-13T14:59:32.822 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.021171713000000002 2024-09-13T14:59:32.822 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.822 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.822 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:59:32.822 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.198860+0000", 2024-09-13T14:59:32.822 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0068942059999999999 2024-09-13T14:59:32.822 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.822 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.822 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:59:32.822 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.198873+0000", 2024-09-13T14:59:32.823 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3499000000000001e-05 2024-09-13T14:59:32.823 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.823 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.823 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:59:32.823 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.602392+0000", 2024-09-13T14:59:32.823 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4035188350000001 2024-09-13T14:59:32.823 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.823 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.823 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:59:32.823 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.602473+0000", 2024-09-13T14:59:32.823 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.1456000000000001e-05 2024-09-13T14:59:32.823 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:59:32.823 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:59:32.824 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:59:32.824 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.824 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.824 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:249810 8.f 8:f6698050:::benchmark_data_smithi102_313275_object119223:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-09-13T14:59:32.824 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:50:13.159508+0000", 2024-09-13T14:59:32.824 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 559.65396746399995, 2024-09-13T14:59:32.824 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4504869890000001, 2024-09-13T14:59:32.824 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:59:32.824 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:59:32.824 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "reached pg", 2024-09-13T14:59:32.824 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:59:32.824 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:59:32.824 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:59:32.824 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 249810 2024-09-13T14:59:32.825 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.825 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:59:32.825 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.825 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:59:32.825 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.159508+0000", 2024-09-13T14:59:32.825 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:59:32.825 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.825 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.825 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:59:32.825 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.159508+0000", 2024-09-13T14:59:32.825 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:59:32.825 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.825 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.826 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:59:32.826 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.159510+0000", 2024-09-13T14:59:32.826 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.6390000000000001e-06 2024-09-13T14:59:32.826 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.826 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.826 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:59:32.826 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.159515+0000", 2024-09-13T14:59:32.826 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.8860000000000003e-06 2024-09-13T14:59:32.826 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.826 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.826 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:59:32.827 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.159528+0000", 2024-09-13T14:59:32.827 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3200999999999999e-05 2024-09-13T14:59:32.827 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.827 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.827 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:59:32.827 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.159535+0000", 2024-09-13T14:59:32.827 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.5290000000000003e-06 2024-09-13T14:59:32.827 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.827 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.827 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:59:32.827 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.166935+0000", 2024-09-13T14:59:32.828 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.007399554 2024-09-13T14:59:32.828 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.828 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.828 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:59:32.828 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.167038+0000", 2024-09-13T14:59:32.828 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000102719 2024-09-13T14:59:32.828 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.828 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.828 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,2,4", 2024-09-13T14:59:32.828 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.167147+0000", 2024-09-13T14:59:32.828 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000109601 2024-09-13T14:59:32.828 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.828 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.829 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:59:32.829 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.204880+0000", 2024-09-13T14:59:32.829 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.037732849999999998 2024-09-13T14:59:32.829 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.829 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.829 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:59:32.829 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.204889+0000", 2024-09-13T14:59:32.829 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.0000000000000002e-06 2024-09-13T14:59:32.829 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.829 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.829 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:59:32.829 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.602703+0000", 2024-09-13T14:59:32.829 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3978135190000001 2024-09-13T14:59:32.830 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.830 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.830 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for missing object", 2024-09-13T14:59:32.830 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.602771+0000", 2024-09-13T14:59:32.830 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.8176999999999995e-05 2024-09-13T14:59:32.830 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.830 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.830 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:59:32.830 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.609884+0000", 2024-09-13T14:59:32.830 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0071134850000000001 2024-09-13T14:59:32.830 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.830 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.830 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:59:32.830 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.609995+0000", 2024-09-13T14:59:32.831 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00011082900000000001 2024-09-13T14:59:32.831 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:59:32.831 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:59:32.831 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:59:32.831 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.831 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.831 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:249881 8.f 8:f3194c62:::benchmark_data_smithi102_313275_object119294:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-13T14:59:32.831 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:50:13.261774+0000", 2024-09-13T14:59:32.831 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 559.55170186299995, 2024-09-13T14:59:32.831 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3524746409999999, 2024-09-13T14:59:32.831 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:59:32.831 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:59:32.831 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:32.832 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:59:32.832 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:59:32.832 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:59:32.832 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 249881 2024-09-13T14:59:32.832 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.832 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:59:32.832 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.832 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:59:32.832 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.261774+0000", 2024-09-13T14:59:32.832 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:59:32.832 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.832 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.832 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:59:32.833 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.261774+0000", 2024-09-13T14:59:32.833 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:59:32.833 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.833 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.833 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:59:32.833 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.261775+0000", 2024-09-13T14:59:32.833 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.708e-06 2024-09-13T14:59:32.833 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.833 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.833 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:59:32.833 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.261780+0000", 2024-09-13T14:59:32.833 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.5959999999999997e-06 2024-09-13T14:59:32.833 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.833 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.834 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:59:32.834 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.261796+0000", 2024-09-13T14:59:32.834 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.6143000000000001e-05 2024-09-13T14:59:32.834 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.834 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.834 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:59:32.834 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.261807+0000", 2024-09-13T14:59:32.834 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1022000000000001e-05 2024-09-13T14:59:32.834 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.834 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.834 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:59:32.834 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.262190+0000", 2024-09-13T14:59:32.834 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000383032 2024-09-13T14:59:32.835 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.835 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.835 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:59:32.835 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.262238+0000", 2024-09-13T14:59:32.835 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.8004000000000003e-05 2024-09-13T14:59:32.835 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.835 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.835 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:59:32.835 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.602809+0000", 2024-09-13T14:59:32.835 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3405708009999999 2024-09-13T14:59:32.835 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.835 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.835 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:59:32.835 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.602925+0000", 2024-09-13T14:59:32.836 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000116081 2024-09-13T14:59:32.836 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.836 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.836 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:59:32.836 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.603013+0000", 2024-09-13T14:59:32.836 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.8063e-05 2024-09-13T14:59:32.836 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.836 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.836 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:32.836 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.609463+0000", 2024-09-13T14:59:32.836 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0064497260000000002 2024-09-13T14:59:32.836 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.837 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.837 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:59:32.837 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.609994+0000", 2024-09-13T14:59:32.837 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00053112500000000002 2024-09-13T14:59:32.837 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.837 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.837 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:32.837 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.614049+0000", 2024-09-13T14:59:32.837 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0040555239999999996 2024-09-13T14:59:32.837 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.838 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.838 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:59:32.838 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.614151+0000", 2024-09-13T14:59:32.838 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00010197800000000001 2024-09-13T14:59:32.838 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.838 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.838 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:59:32.838 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.614248+0000", 2024-09-13T14:59:32.838 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.6837999999999997e-05 2024-09-13T14:59:32.838 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:59:32.838 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:59:32.838 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:59:32.838 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.839 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.839 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:249908 8.f 8:f0b20de0:::benchmark_data_smithi102_313275_object119321:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-13T14:59:32.840 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:50:13.297325+0000", 2024-09-13T14:59:32.840 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 559.516150724, 2024-09-13T14:59:32.840 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3251422660000001, 2024-09-13T14:59:32.840 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:59:32.840 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:59:32.840 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:32.840 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:59:32.841 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:59:32.841 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:59:32.841 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 249908 2024-09-13T14:59:32.841 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.841 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:59:32.841 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.841 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:59:32.842 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.297325+0000", 2024-09-13T14:59:32.842 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:59:32.842 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.842 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.842 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:59:32.842 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.297325+0000", 2024-09-13T14:59:32.842 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:59:32.842 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.843 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.843 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:59:32.843 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.297326+0000", 2024-09-13T14:59:32.843 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.697e-06 2024-09-13T14:59:32.843 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.843 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.843 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:59:32.843 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.297333+0000", 2024-09-13T14:59:32.844 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.1410000000000002e-06 2024-09-13T14:59:32.844 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.844 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.844 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:59:32.844 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.297351+0000", 2024-09-13T14:59:32.845 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.8131999999999999e-05 2024-09-13T14:59:32.845 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.845 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.845 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:59:32.845 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.297359+0000", 2024-09-13T14:59:32.845 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.2849999999999995e-06 2024-09-13T14:59:32.845 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.845 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.845 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:59:32.846 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.297616+0000", 2024-09-13T14:59:32.846 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00025743000000000001 2024-09-13T14:59:32.846 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.846 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.846 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:59:32.846 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.297627+0000", 2024-09-13T14:59:32.846 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0546e-05 2024-09-13T14:59:32.846 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.846 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.846 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:59:32.846 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.606160+0000", 2024-09-13T14:59:32.846 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.3085327330000001 2024-09-13T14:59:32.846 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.846 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.847 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:59:32.847 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.606273+0000", 2024-09-13T14:59:32.847 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00011299200000000001 2024-09-13T14:59:32.847 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.847 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.847 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:59:32.847 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.606350+0000", 2024-09-13T14:59:32.847 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.7202999999999999e-05 2024-09-13T14:59:32.847 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.847 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.847 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:59:32.847 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.613913+0000", 2024-09-13T14:59:32.847 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0075633840000000003 2024-09-13T14:59:32.847 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.848 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.848 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:32.848 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.622195+0000", 2024-09-13T14:59:32.848 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0082820189999999998 2024-09-13T14:59:32.848 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.848 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.848 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:32.848 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.622300+0000", 2024-09-13T14:59:32.848 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000104709 2024-09-13T14:59:32.848 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.848 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.848 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:59:32.848 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.622356+0000", 2024-09-13T14:59:32.849 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.6020999999999998e-05 2024-09-13T14:59:32.849 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.849 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.849 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:59:32.849 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.622467+0000", 2024-09-13T14:59:32.849 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000110974 2024-09-13T14:59:32.849 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:59:32.849 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:59:32.849 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:59:32.849 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.849 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.851 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:249937 8.f 8:fe425234:::benchmark_data_smithi102_313275_object119350:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-13T14:59:32.851 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:50:13.331885+0000", 2024-09-13T14:59:32.851 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 559.48159045800003, 2024-09-13T14:59:32.851 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.294601433, 2024-09-13T14:59:32.851 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:59:32.851 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:59:32.851 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:32.852 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:59:32.852 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:59:32.852 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:59:32.852 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 249937 2024-09-13T14:59:32.852 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.852 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:59:32.852 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.852 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:59:32.852 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.331885+0000", 2024-09-13T14:59:32.852 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:59:32.852 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.852 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.853 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:59:32.853 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.331885+0000", 2024-09-13T14:59:32.853 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:59:32.853 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.853 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.853 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:59:32.853 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.331886+0000", 2024-09-13T14:59:32.853 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.8800000000000003e-07 2024-09-13T14:59:32.853 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.854 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.854 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:59:32.854 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.331888+0000", 2024-09-13T14:59:32.854 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.2340000000000001e-06 2024-09-13T14:59:32.854 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.854 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.854 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:59:32.854 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.331895+0000", 2024-09-13T14:59:32.854 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.5989999999999996e-06 2024-09-13T14:59:32.854 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.854 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.854 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:59:32.854 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.331901+0000", 2024-09-13T14:59:32.855 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.4639999999999997e-06 2024-09-13T14:59:32.855 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.855 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.855 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:59:32.855 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.332688+0000", 2024-09-13T14:59:32.855 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00078735199999999995 2024-09-13T14:59:32.855 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.855 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.855 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:59:32.855 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.332752+0000", 2024-09-13T14:59:32.855 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.3461999999999996e-05 2024-09-13T14:59:32.855 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.855 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.856 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:59:32.856 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.610018+0000", 2024-09-13T14:59:32.856 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2772668039999999 2024-09-13T14:59:32.856 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.856 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.856 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:59:32.856 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.610119+0000", 2024-09-13T14:59:32.856 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00010053800000000001 2024-09-13T14:59:32.857 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.857 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.857 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:59:32.857 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.610263+0000", 2024-09-13T14:59:32.857 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00014417 2024-09-13T14:59:32.857 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.858 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.858 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:59:32.858 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.622144+0000", 2024-09-13T14:59:32.858 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.011880559000000001 2024-09-13T14:59:32.859 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.859 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.859 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:32.859 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.622243+0000", 2024-09-13T14:59:32.859 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.9465999999999999e-05 2024-09-13T14:59:32.859 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.860 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.860 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:32.860 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.626338+0000", 2024-09-13T14:59:32.860 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0040946330000000003 2024-09-13T14:59:32.860 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.860 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.860 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:59:32.861 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.626390+0000", 2024-09-13T14:59:32.861 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.2049000000000003e-05 2024-09-13T14:59:32.861 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.861 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.861 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:59:32.861 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.626486+0000", 2024-09-13T14:59:32.861 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.6614999999999996e-05 2024-09-13T14:59:32.862 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:59:32.862 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:59:32.862 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:59:32.862 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.862 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.862 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:249947 8.f 8:fc361518:::benchmark_data_smithi102_313275_object119360:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-13T14:59:32.862 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:50:13.340670+0000", 2024-09-13T14:59:32.863 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 559.47280590499997, 2024-09-13T14:59:32.863 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2950710439999999, 2024-09-13T14:59:32.863 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:59:32.863 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:59:32.863 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:32.863 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:59:32.863 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:59:32.864 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:59:32.864 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 249947 2024-09-13T14:59:32.864 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.864 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:59:32.864 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.865 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:59:32.865 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.340670+0000", 2024-09-13T14:59:32.865 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:59:32.865 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.865 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.865 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:59:32.865 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.340670+0000", 2024-09-13T14:59:32.865 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:59:32.866 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.866 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.866 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:59:32.866 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.340671+0000", 2024-09-13T14:59:32.866 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.5740000000000001e-06 2024-09-13T14:59:32.866 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.866 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.866 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:59:32.866 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.340676+0000", 2024-09-13T14:59:32.866 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.2989999999999996e-06 2024-09-13T14:59:32.866 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.866 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.866 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:59:32.867 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.340692+0000", 2024-09-13T14:59:32.867 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.5838999999999999e-05 2024-09-13T14:59:32.867 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.867 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.867 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:59:32.867 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.340699+0000", 2024-09-13T14:59:32.867 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.8990000000000004e-06 2024-09-13T14:59:32.867 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.867 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.867 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:59:32.867 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.340745+0000", 2024-09-13T14:59:32.867 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.5435999999999997e-05 2024-09-13T14:59:32.867 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.867 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.868 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:59:32.868 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.340753+0000", 2024-09-13T14:59:32.868 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.4630000000000001e-06 2024-09-13T14:59:32.868 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.868 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.868 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:59:32.868 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.614282+0000", 2024-09-13T14:59:32.868 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.273528888 2024-09-13T14:59:32.868 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.868 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.868 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:59:32.868 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.614401+0000", 2024-09-13T14:59:32.868 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000119292 2024-09-13T14:59:32.868 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.869 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.869 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:59:32.869 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.614528+0000", 2024-09-13T14:59:32.869 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00012681899999999999 2024-09-13T14:59:32.869 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.869 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.869 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:59:32.869 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.622480+0000", 2024-09-13T14:59:32.869 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0079516880000000002 2024-09-13T14:59:32.869 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.869 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.869 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:32.870 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.626648+0000", 2024-09-13T14:59:32.870 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0041681260000000003 2024-09-13T14:59:32.870 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.870 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.870 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:32.870 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.635542+0000", 2024-09-13T14:59:32.870 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0088941709999999993 2024-09-13T14:59:32.870 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.870 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.870 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:59:32.870 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.635610+0000", 2024-09-13T14:59:32.871 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.8392999999999994e-05 2024-09-13T14:59:32.871 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.871 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.871 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:59:32.871 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.635741+0000", 2024-09-13T14:59:32.871 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00013015699999999999 2024-09-13T14:59:32.871 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:59:32.871 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:59:32.871 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:59:32.871 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.871 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.871 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:249954 8.f 8:feebf84b:::benchmark_data_smithi102_313275_object119367:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-13T14:59:32.871 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:50:13.350386+0000", 2024-09-13T14:59:32.872 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 559.46308993599996, 2024-09-13T14:59:32.872 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.285591291, 2024-09-13T14:59:32.872 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:59:32.872 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:59:32.872 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:32.872 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:59:32.872 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:59:32.872 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:59:32.872 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 249954 2024-09-13T14:59:32.872 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.872 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:59:32.872 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.872 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:59:32.873 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.350386+0000", 2024-09-13T14:59:32.873 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:59:32.873 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.873 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.873 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:59:32.873 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.350386+0000", 2024-09-13T14:59:32.873 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:59:32.873 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.873 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.873 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:59:32.873 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.350387+0000", 2024-09-13T14:59:32.873 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.443e-06 2024-09-13T14:59:32.873 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.874 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.874 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:59:32.874 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.350392+0000", 2024-09-13T14:59:32.874 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.3279999999999996e-06 2024-09-13T14:59:32.874 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.874 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.874 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:59:32.874 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.350402+0000", 2024-09-13T14:59:32.874 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.7280000000000007e-06 2024-09-13T14:59:32.874 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.874 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.874 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:59:32.874 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.350409+0000", 2024-09-13T14:59:32.875 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.1180000000000003e-06 2024-09-13T14:59:32.875 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.875 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.875 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:59:32.875 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.350501+0000", 2024-09-13T14:59:32.875 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.1643000000000003e-05 2024-09-13T14:59:32.875 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.875 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.875 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:59:32.875 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.350509+0000", 2024-09-13T14:59:32.875 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.6370000000000005e-06 2024-09-13T14:59:32.875 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.875 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.875 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:59:32.876 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.618359+0000", 2024-09-13T14:59:32.876 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.267849821 2024-09-13T14:59:32.876 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.876 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.876 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:59:32.876 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.618466+0000", 2024-09-13T14:59:32.876 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000106984 2024-09-13T14:59:32.876 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.876 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.876 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:59:32.876 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.618581+0000", 2024-09-13T14:59:32.876 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00011464300000000001 2024-09-13T14:59:32.876 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.877 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.877 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:59:32.877 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.626300+0000", 2024-09-13T14:59:32.877 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0077186609999999999 2024-09-13T14:59:32.877 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.877 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.877 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:32.877 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.626695+0000", 2024-09-13T14:59:32.877 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00039502299999999999 2024-09-13T14:59:32.877 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.877 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.877 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:32.877 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.635799+0000", 2024-09-13T14:59:32.878 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0091048840000000006 2024-09-13T14:59:32.878 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.878 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.878 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:59:32.878 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.635874+0000", 2024-09-13T14:59:32.878 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.4433999999999999e-05 2024-09-13T14:59:32.878 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.878 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.878 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:59:32.878 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.635977+0000", 2024-09-13T14:59:32.878 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000102944 2024-09-13T14:59:32.878 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:59:32.878 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:59:32.878 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:59:32.879 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.879 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.879 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:249959 8.f 8:f42b2733:::benchmark_data_smithi102_313275_object119372:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-13T14:59:32.879 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:50:13.351208+0000", 2024-09-13T14:59:32.879 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 559.46226702399997, 2024-09-13T14:59:32.879 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.28519165, 2024-09-13T14:59:32.879 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:59:32.879 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:59:32.879 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:32.879 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:59:32.879 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:59:32.879 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:59:32.879 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 249959 2024-09-13T14:59:32.880 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.880 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:59:32.880 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.880 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:59:32.880 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.351208+0000", 2024-09-13T14:59:32.880 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:59:32.880 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.880 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.881 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:59:32.881 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.351208+0000", 2024-09-13T14:59:32.881 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:59:32.881 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.881 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.881 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:59:32.881 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.351209+0000", 2024-09-13T14:59:32.881 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.3099999999999996e-07 2024-09-13T14:59:32.882 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.882 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.882 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:59:32.882 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.351214+0000", 2024-09-13T14:59:32.882 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.622e-06 2024-09-13T14:59:32.882 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.882 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.882 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:59:32.882 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.351228+0000", 2024-09-13T14:59:32.883 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4280999999999999e-05 2024-09-13T14:59:32.883 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.883 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.883 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:59:32.883 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.351236+0000", 2024-09-13T14:59:32.883 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.6190000000000002e-06 2024-09-13T14:59:32.883 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.883 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.884 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:59:32.884 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.351309+0000", 2024-09-13T14:59:32.884 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.2606000000000002e-05 2024-09-13T14:59:32.884 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.884 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.884 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:59:32.884 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.351316+0000", 2024-09-13T14:59:32.884 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.8189999999999992e-06 2024-09-13T14:59:32.885 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.885 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.885 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:59:32.885 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.622504+0000", 2024-09-13T14:59:32.885 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.271187625 2024-09-13T14:59:32.885 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.885 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.885 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:59:32.885 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.622633+0000", 2024-09-13T14:59:32.886 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000129474 2024-09-13T14:59:32.886 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.886 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.886 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:59:32.886 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.622746+0000", 2024-09-13T14:59:32.886 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00011304299999999999 2024-09-13T14:59:32.886 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.886 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.887 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:59:32.887 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.635757+0000", 2024-09-13T14:59:32.887 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.013010649000000001 2024-09-13T14:59:32.887 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.887 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.887 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:32.887 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.636075+0000", 2024-09-13T14:59:32.887 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000317764 2024-09-13T14:59:32.888 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.888 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.888 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:32.888 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.636159+0000", 2024-09-13T14:59:32.888 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.3968999999999995e-05 2024-09-13T14:59:32.888 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.888 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.888 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:59:32.888 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.636306+0000", 2024-09-13T14:59:32.889 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00014711999999999999 2024-09-13T14:59:32.889 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.889 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.889 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:59:32.889 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.636400+0000", 2024-09-13T14:59:32.889 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.4128000000000004e-05 2024-09-13T14:59:32.889 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:59:32.889 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:59:32.890 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:59:32.890 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.890 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.890 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:249975 8.f 8:f1516336:::benchmark_data_smithi102_313275_object119388:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-13T14:59:32.890 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:50:13.368418+0000", 2024-09-13T14:59:32.890 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 559.44505732499999, 2024-09-13T14:59:32.890 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.275804943, 2024-09-13T14:59:32.890 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:59:32.891 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:59:32.891 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:32.891 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:59:32.891 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:59:32.891 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:59:32.891 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 249975 2024-09-13T14:59:32.891 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.891 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:59:32.891 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.892 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:59:32.892 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.368418+0000", 2024-09-13T14:59:32.892 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:59:32.892 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.892 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.892 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:59:32.892 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.368418+0000", 2024-09-13T14:59:32.892 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:59:32.893 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.893 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.893 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:59:32.893 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.368420+0000", 2024-09-13T14:59:32.893 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.5269999999999999e-06 2024-09-13T14:59:32.893 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.893 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.893 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:59:32.894 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.368426+0000", 2024-09-13T14:59:32.894 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.2600000000000002e-06 2024-09-13T14:59:32.894 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.894 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.894 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:59:32.895 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.368440+0000", 2024-09-13T14:59:32.895 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4518999999999999e-05 2024-09-13T14:59:32.895 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.895 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.895 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:59:32.895 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.368452+0000", 2024-09-13T14:59:32.896 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2017e-05 2024-09-13T14:59:32.896 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.896 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.896 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:59:32.896 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.368534+0000", 2024-09-13T14:59:32.897 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.1895000000000001e-05 2024-09-13T14:59:32.897 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.897 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.897 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:59:32.897 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.368546+0000", 2024-09-13T14:59:32.897 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1505000000000001e-05 2024-09-13T14:59:32.898 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.898 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.898 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:59:32.898 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.626725+0000", 2024-09-13T14:59:32.898 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.258179167 2024-09-13T14:59:32.898 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.899 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.899 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:59:32.899 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.628700+0000", 2024-09-13T14:59:32.899 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0019751130000000001 2024-09-13T14:59:32.899 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.900 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.900 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:59:32.900 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.628828+0000", 2024-09-13T14:59:32.900 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00012799999999999999 2024-09-13T14:59:32.900 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.900 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.901 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:59:32.901 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.636037+0000", 2024-09-13T14:59:32.901 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0072084949999999997 2024-09-13T14:59:32.901 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.901 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.901 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:32.901 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.636115+0000", 2024-09-13T14:59:32.901 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.7855999999999995e-05 2024-09-13T14:59:32.901 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.901 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.902 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:32.902 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.644016+0000", 2024-09-13T14:59:32.902 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0079011629999999992 2024-09-13T14:59:32.902 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.902 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.902 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:59:32.902 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.644121+0000", 2024-09-13T14:59:32.902 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00010556399999999999 2024-09-13T14:59:32.902 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.902 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.902 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:59:32.902 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.644223+0000", 2024-09-13T14:59:32.903 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00010186199999999999 2024-09-13T14:59:32.903 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:59:32.903 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:59:32.903 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:59:32.903 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.903 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.903 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:250009 8.f 8:f41ba2b5:::benchmark_data_smithi102_313275_object119422:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-13T14:59:32.903 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:50:13.408634+0000", 2024-09-13T14:59:32.903 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 559.40484184299999, 2024-09-13T14:59:32.904 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.235791938, 2024-09-13T14:59:32.904 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:59:32.904 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:59:32.904 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:32.904 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:59:32.904 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:59:32.904 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:59:32.904 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 250009 2024-09-13T14:59:32.904 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.905 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:59:32.905 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.905 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:59:32.905 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.408634+0000", 2024-09-13T14:59:32.905 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:59:32.905 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.905 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.905 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:59:32.905 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.408634+0000", 2024-09-13T14:59:32.905 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:59:32.905 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.906 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.906 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:59:32.906 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.408635+0000", 2024-09-13T14:59:32.906 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0610000000000001e-06 2024-09-13T14:59:32.906 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.906 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.907 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:59:32.907 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.408639+0000", 2024-09-13T14:59:32.907 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.5299999999999998e-06 2024-09-13T14:59:32.907 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.907 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.907 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:59:32.907 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.408654+0000", 2024-09-13T14:59:32.907 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.5174000000000001e-05 2024-09-13T14:59:32.908 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.908 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.908 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:59:32.908 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.408662+0000", 2024-09-13T14:59:32.908 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.3039999999999999e-06 2024-09-13T14:59:32.908 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.908 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.908 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:59:32.908 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.408995+0000", 2024-09-13T14:59:32.908 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00033359699999999998 2024-09-13T14:59:32.908 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.909 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.909 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:59:32.909 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.409004+0000", 2024-09-13T14:59:32.909 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.3969999999999994e-06 2024-09-13T14:59:32.909 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.909 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.909 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:59:32.909 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.632581+0000", 2024-09-13T14:59:32.909 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.223577205 2024-09-13T14:59:32.909 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.910 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.910 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:59:32.910 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.632695+0000", 2024-09-13T14:59:32.910 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000114461 2024-09-13T14:59:32.910 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.910 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.910 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:59:32.910 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.632781+0000", 2024-09-13T14:59:32.910 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.5303999999999995e-05 2024-09-13T14:59:32.910 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.910 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.910 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:59:32.910 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.640128+0000", 2024-09-13T14:59:32.910 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0073468739999999998 2024-09-13T14:59:32.911 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.911 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.911 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:32.911 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.643973+0000", 2024-09-13T14:59:32.911 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0038452460000000001 2024-09-13T14:59:32.911 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.911 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.911 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:32.911 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.644260+0000", 2024-09-13T14:59:32.912 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00028718999999999999 2024-09-13T14:59:32.912 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.912 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.912 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:59:32.912 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.644340+0000", 2024-09-13T14:59:32.912 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.9756e-05 2024-09-13T14:59:32.912 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.912 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.912 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:59:32.912 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.644426+0000", 2024-09-13T14:59:32.912 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.5839000000000003e-05 2024-09-13T14:59:32.912 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:59:32.913 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:59:32.913 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:59:32.913 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.913 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.913 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:250012 8.f 8:f74e2086:::benchmark_data_smithi102_313275_object119425:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-13T14:59:32.913 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:50:13.413092+0000", 2024-09-13T14:59:32.913 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 559.40038359499999, 2024-09-13T14:59:32.913 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2354481820000001, 2024-09-13T14:59:32.913 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:59:32.913 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:59:32.913 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:32.914 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:59:32.914 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:59:32.914 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:59:32.914 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 250012 2024-09-13T14:59:32.914 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.914 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:59:32.914 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.914 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:59:32.914 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.413092+0000", 2024-09-13T14:59:32.914 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:59:32.914 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.914 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.915 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:59:32.915 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.413092+0000", 2024-09-13T14:59:32.915 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:59:32.915 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.915 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.915 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:59:32.915 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.413094+0000", 2024-09-13T14:59:32.915 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.8819999999999999e-06 2024-09-13T14:59:32.915 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.915 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.915 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:59:32.915 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.413099+0000", 2024-09-13T14:59:32.915 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.7790000000000003e-06 2024-09-13T14:59:32.916 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.916 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.916 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:59:32.916 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.413109+0000", 2024-09-13T14:59:32.916 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0096e-05 2024-09-13T14:59:32.916 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.916 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.916 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:59:32.916 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.413117+0000", 2024-09-13T14:59:32.916 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.9030000000000004e-06 2024-09-13T14:59:32.916 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.916 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.916 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:59:32.917 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.413195+0000", 2024-09-13T14:59:32.917 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.8024999999999999e-05 2024-09-13T14:59:32.917 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.917 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.917 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:59:32.917 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.413203+0000", 2024-09-13T14:59:32.917 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.7530000000000007e-06 2024-09-13T14:59:32.917 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.917 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.917 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:59:32.917 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.636436+0000", 2024-09-13T14:59:32.917 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2232331329999999 2024-09-13T14:59:32.918 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.918 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.918 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:59:32.918 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.636568+0000", 2024-09-13T14:59:32.918 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000131144 2024-09-13T14:59:32.918 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.918 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.918 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:59:32.918 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.636679+0000", 2024-09-13T14:59:32.918 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000111508 2024-09-13T14:59:32.918 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.918 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.918 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:32.918 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.644461+0000", 2024-09-13T14:59:32.919 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0077816170000000002 2024-09-13T14:59:32.919 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.919 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.919 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:32.919 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.648104+0000", 2024-09-13T14:59:32.919 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0036430130000000001 2024-09-13T14:59:32.919 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.919 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.919 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:59:32.919 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.648133+0000", 2024-09-13T14:59:32.919 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.8815000000000001e-05 2024-09-13T14:59:32.919 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.919 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.919 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:59:32.920 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.648439+0000", 2024-09-13T14:59:32.920 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000306915 2024-09-13T14:59:32.920 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.920 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.920 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:59:32.920 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.648540+0000", 2024-09-13T14:59:32.920 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000100599 2024-09-13T14:59:32.920 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:59:32.920 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:59:32.920 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:59:32.920 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.920 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.921 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.46157.0:250019 8.f 8:f4c1da74:::benchmark_data_smithi102_313275_object119432:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-13T14:59:32.921 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:50:13.421510+0000", 2024-09-13T14:59:32.921 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 559.39196581800002, 2024-09-13T14:59:32.921 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.242516763, 2024-09-13T14:59:32.921 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:59:32.921 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:59:32.921 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:32.921 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:59:32.921 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.46157", 2024-09-13T14:59:32.921 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/7795259", 2024-09-13T14:59:32.921 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 250019 2024-09-13T14:59:32.921 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.921 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:59:32.921 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.922 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:59:32.922 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.421510+0000", 2024-09-13T14:59:32.922 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:59:32.922 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.922 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.922 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:59:32.922 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.421510+0000", 2024-09-13T14:59:32.922 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:59:32.922 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.922 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.922 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:59:32.922 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.421511+0000", 2024-09-13T14:59:32.922 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.8599999999999997e-07 2024-09-13T14:59:32.922 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.923 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.923 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:59:32.923 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.421515+0000", 2024-09-13T14:59:32.923 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.9609999999999999e-06 2024-09-13T14:59:32.923 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.923 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.923 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:59:32.923 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.421525+0000", 2024-09-13T14:59:32.923 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.0052e-05 2024-09-13T14:59:32.923 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.923 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.923 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:59:32.923 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.421530+0000", 2024-09-13T14:59:32.923 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.8560000000000003e-06 2024-09-13T14:59:32.924 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.924 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.924 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:59:32.924 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.421588+0000", 2024-09-13T14:59:32.924 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.7553999999999998e-05 2024-09-13T14:59:32.924 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.924 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.924 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for peered", 2024-09-13T14:59:32.924 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:13.421596+0000", 2024-09-13T14:59:32.924 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.6669999999999996e-06 2024-09-13T14:59:32.924 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.924 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.924 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:59:32.925 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.640161+0000", 2024-09-13T14:59:32.925 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2185656380000001 2024-09-13T14:59:32.925 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.925 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.925 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:59:32.925 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.640277+0000", 2024-09-13T14:59:32.925 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000116193 2024-09-13T14:59:32.925 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.925 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.925 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:59:32.925 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.640398+0000", 2024-09-13T14:59:32.925 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00012017 2024-09-13T14:59:32.926 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.926 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.926 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:59:32.926 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.652552+0000", 2024-09-13T14:59:32.926 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.012154094000000001 2024-09-13T14:59:32.926 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.926 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.926 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:32.926 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.652630+0000", 2024-09-13T14:59:32.926 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.8067999999999999e-05 2024-09-13T14:59:32.926 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.926 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.926 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:32.927 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.663828+0000", 2024-09-13T14:59:32.927 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.011198143000000001 2024-09-13T14:59:32.927 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.927 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.927 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:59:32.927 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.663934+0000", 2024-09-13T14:59:32.927 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00010572400000000001 2024-09-13T14:59:32.927 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.927 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.927 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:59:32.927 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:50:14.664026+0000", 2024-09-13T14:59:32.927 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.2757000000000007e-05 2024-09-13T14:59:32.927 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:59:32.927 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:59:32.928 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:59:32.928 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.928 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.928 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.36738.0:173474 10.d 10:b4f5f84b:::benchmark_data_smithi102_377304_object67632:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e480)", 2024-09-13T14:59:32.928 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:57:55.439121+0000", 2024-09-13T14:59:32.928 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 97.374354996999998, 2024-09-13T14:59:32.928 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.11163097, 2024-09-13T14:59:32.928 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:59:32.928 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:59:32.928 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:32.928 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:59:32.928 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.36738", 2024-09-13T14:59:32.928 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/2915890700", 2024-09-13T14:59:32.929 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 173474 2024-09-13T14:59:32.929 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.929 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:59:32.929 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.929 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:59:32.929 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:57:55.439121+0000", 2024-09-13T14:59:32.929 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:59:32.929 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.929 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.929 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:59:32.929 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:57:55.439121+0000", 2024-09-13T14:59:32.929 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:59:32.929 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.929 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.930 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:59:32.930 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:57:55.439122+0000", 2024-09-13T14:59:32.930 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.336e-06 2024-09-13T14:59:32.930 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.930 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.930 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:59:32.930 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:57:55.439127+0000", 2024-09-13T14:59:32.930 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.4049999999999999e-06 2024-09-13T14:59:32.930 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.930 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.930 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:59:32.930 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:57:55.439138+0000", 2024-09-13T14:59:32.930 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.075e-05 2024-09-13T14:59:32.931 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.931 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.931 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:59:32.931 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:57:55.439145+0000", 2024-09-13T14:59:32.931 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.0269999999999998e-06 2024-09-13T14:59:32.931 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.931 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.931 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:59:32.931 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:57:55.439703+0000", 2024-09-13T14:59:32.931 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00055806900000000001 2024-09-13T14:59:32.931 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.931 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.931 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:59:32.931 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:57:55.439804+0000", 2024-09-13T14:59:32.932 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0001011 2024-09-13T14:59:32.932 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.932 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.932 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 0,6", 2024-09-13T14:59:32.932 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:57:55.439916+0000", 2024-09-13T14:59:32.932 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0001119 2024-09-13T14:59:32.932 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.932 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.932 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:32.932 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:57:55.451030+0000", 2024-09-13T14:59:32.932 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.011114107999999999 2024-09-13T14:59:32.932 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.932 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.933 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:32.933 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:57:55.458351+0000", 2024-09-13T14:59:32.933 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0073207339999999997 2024-09-13T14:59:32.933 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.933 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.933 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:59:32.933 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:57:55.550651+0000", 2024-09-13T14:59:32.933 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.092300076999999994 2024-09-13T14:59:32.933 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.933 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.933 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:59:32.933 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:57:55.550707+0000", 2024-09-13T14:59:32.933 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.6047000000000001e-05 2024-09-13T14:59:32.933 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.934 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.934 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:59:32.934 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:57:55.550751+0000", 2024-09-13T14:59:32.934 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.4416999999999998e-05 2024-09-13T14:59:32.934 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:59:32.934 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:59:32.934 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:59:32.934 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.934 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.936 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.36738.0:173478 10.d 10:b5be5a28:::benchmark_data_smithi102_377304_object67636:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e480)", 2024-09-13T14:59:32.936 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:57:55.442185+0000", 2024-09-13T14:59:32.936 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 97.371290720000005, 2024-09-13T14:59:32.936 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.113307423, 2024-09-13T14:59:32.937 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:59:32.937 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:59:32.937 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:32.937 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:59:32.937 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.36738", 2024-09-13T14:59:32.937 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/2915890700", 2024-09-13T14:59:32.937 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 173478 2024-09-13T14:59:32.937 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.937 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:59:32.937 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.937 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:59:32.937 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:57:55.442185+0000", 2024-09-13T14:59:32.937 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:59:32.937 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.938 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.938 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:59:32.938 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:57:55.442185+0000", 2024-09-13T14:59:32.938 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:59:32.938 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.938 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.938 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:59:32.938 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:57:55.442186+0000", 2024-09-13T14:59:32.938 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.9599999999999998e-07 2024-09-13T14:59:32.938 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.938 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.938 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:59:32.938 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:57:55.442189+0000", 2024-09-13T14:59:32.938 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.4769999999999999e-06 2024-09-13T14:59:32.939 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.939 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.939 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:59:32.939 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:57:55.442198+0000", 2024-09-13T14:59:32.939 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.354e-06 2024-09-13T14:59:32.939 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.939 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.939 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:59:32.939 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:57:55.442203+0000", 2024-09-13T14:59:32.939 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.3770000000000004e-06 2024-09-13T14:59:32.939 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.939 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.939 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:59:32.939 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:57:55.453971+0000", 2024-09-13T14:59:32.940 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.011768420999999999 2024-09-13T14:59:32.940 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.940 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.940 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:59:32.940 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:57:55.454071+0000", 2024-09-13T14:59:32.940 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.9290000000000007e-05 2024-09-13T14:59:32.940 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.940 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.940 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 0,6", 2024-09-13T14:59:32.940 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:57:55.454174+0000", 2024-09-13T14:59:32.940 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000102929 2024-09-13T14:59:32.940 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.940 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.941 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:32.941 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:57:55.458318+0000", 2024-09-13T14:59:32.941 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0041440410000000002 2024-09-13T14:59:32.941 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.941 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.941 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:32.941 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:57:55.555369+0000", 2024-09-13T14:59:32.941 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.097051551999999999 2024-09-13T14:59:32.941 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.941 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.941 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:59:32.941 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:57:55.555392+0000", 2024-09-13T14:59:32.941 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.2708999999999999e-05 2024-09-13T14:59:32.941 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.942 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.942 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:59:32.942 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:57:55.555441+0000", 2024-09-13T14:59:32.942 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.8844000000000002e-05 2024-09-13T14:59:32.942 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.942 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.942 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:59:32.942 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:57:55.555492+0000", 2024-09-13T14:59:32.942 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.1533000000000003e-05 2024-09-13T14:59:32.942 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:59:32.942 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:59:32.942 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:59:32.942 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.942 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.943 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.36738.0:173476 10.5 e480/436 10:a3422701:::benchmark_data_smithi102_377304_object67634:head v 480'10579, mlcod=480'10576)", 2024-09-13T14:59:32.943 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:57:55.442866+0000", 2024-09-13T14:59:32.943 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 97.370609408000007, 2024-09-13T14:59:32.943 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.112416196, 2024-09-13T14:59:32.943 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:59:32.943 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:59:32.943 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:32.943 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:59:32.943 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.943 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:59:32.943 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:57:55.442866+0000", 2024-09-13T14:59:32.943 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:59:32.943 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.944 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.944 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:59:32.944 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:57:55.442866+0000", 2024-09-13T14:59:32.944 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:59:32.944 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.944 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.944 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:59:32.944 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:57:55.442867+0000", 2024-09-13T14:59:32.944 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.5300000000000004e-07 2024-09-13T14:59:32.944 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.944 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.944 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:59:32.944 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:57:55.442872+0000", 2024-09-13T14:59:32.944 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.7940000000000002e-06 2024-09-13T14:59:32.945 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.945 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.945 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:59:32.945 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:57:55.442879+0000", 2024-09-13T14:59:32.945 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.9689999999999997e-06 2024-09-13T14:59:32.945 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.945 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.945 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:59:32.945 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:57:55.442884+0000", 2024-09-13T14:59:32.945 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.7960000000000001e-06 2024-09-13T14:59:32.945 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.945 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.945 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:59:32.945 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:57:55.450122+0000", 2024-09-13T14:59:32.946 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0072379619999999997 2024-09-13T14:59:32.946 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.946 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.946 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:59:32.946 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:57:55.450147+0000", 2024-09-13T14:59:32.946 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.4749999999999999e-05 2024-09-13T14:59:32.946 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.946 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.946 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:59:32.946 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:57:55.555246+0000", 2024-09-13T14:59:32.946 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.105099205 2024-09-13T14:59:32.946 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.946 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.947 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:59:32.947 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:57:55.555282+0000", 2024-09-13T14:59:32.947 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.6066999999999999e-05 2024-09-13T14:59:32.947 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:59:32.947 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:59:32.947 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:59:32.947 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.947 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.947 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_op(client.36738.0:194168 10.4 10:2f6517e7:::benchmark_data_smithi102_377304_object88326:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e481)", 2024-09-13T14:59:32.947 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:58:17.098692+0000", 2024-09-13T14:59:32.947 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 75.714783789999998, 2024-09-13T14:59:32.947 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.12638518000000001, 2024-09-13T14:59:32.947 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:59:32.947 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:59:32.948 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:32.948 INFO:teuthology.orchestra.run.smithi102.stdout: "client_info": { 2024-09-13T14:59:32.948 INFO:teuthology.orchestra.run.smithi102.stdout: "client": "client.36738", 2024-09-13T14:59:32.948 INFO:teuthology.orchestra.run.smithi102.stdout: "client_addr": "172.21.15.102:0/2915890700", 2024-09-13T14:59:32.948 INFO:teuthology.orchestra.run.smithi102.stdout: "tid": 194168 2024-09-13T14:59:32.948 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.948 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:59:32.948 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.948 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:59:32.948 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.098692+0000", 2024-09-13T14:59:32.948 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:59:32.948 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.948 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.949 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:59:32.949 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.098692+0000", 2024-09-13T14:59:32.949 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:59:32.949 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.949 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.949 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:59:32.949 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.098693+0000", 2024-09-13T14:59:32.949 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.4500000000000001e-06 2024-09-13T14:59:32.949 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.949 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.949 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:59:32.949 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.098699+0000", 2024-09-13T14:59:32.949 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.5169999999999999e-06 2024-09-13T14:59:32.949 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.950 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.950 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:59:32.950 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.098711+0000", 2024-09-13T14:59:32.950 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.2099999999999999e-05 2024-09-13T14:59:32.950 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.950 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.950 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:59:32.950 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.098718+0000", 2024-09-13T14:59:32.950 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.306e-06 2024-09-13T14:59:32.950 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.950 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.950 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:59:32.950 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.099126+0000", 2024-09-13T14:59:32.950 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00040753099999999998 2024-09-13T14:59:32.951 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.951 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.951 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:59:32.951 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.099233+0000", 2024-09-13T14:59:32.951 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00010743699999999999 2024-09-13T14:59:32.951 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.951 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.951 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:59:32.951 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.099346+0000", 2024-09-13T14:59:32.951 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00011253599999999999 2024-09-13T14:59:32.951 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.951 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.951 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "op_commit", 2024-09-13T14:59:32.951 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.103658+0000", 2024-09-13T14:59:32.952 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0043120299999999997 2024-09-13T14:59:32.952 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.952 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.952 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:32.952 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.103722+0000", 2024-09-13T14:59:32.952 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.3937999999999997e-05 2024-09-13T14:59:32.952 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.952 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.952 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:32.952 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.106931+0000", 2024-09-13T14:59:32.952 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.003209702 2024-09-13T14:59:32.952 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.952 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.952 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:59:32.953 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.106976+0000", 2024-09-13T14:59:32.953 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.4622000000000001e-05 2024-09-13T14:59:32.953 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.953 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.953 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:59:32.953 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.225077+0000", 2024-09-13T14:59:32.953 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.11810101100000001 2024-09-13T14:59:32.953 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:59:32.953 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:59:32.953 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:59:32.953 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.953 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.953 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.36738.0:194163 10.5 e481/436 10:a9b4718d:::benchmark_data_smithi102_377304_object88321:head v 481'11897, mlcod=481'11894)", 2024-09-13T14:59:32.954 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:58:17.103314+0000", 2024-09-13T14:59:32.954 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 75.710161057999997, 2024-09-13T14:59:32.954 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.1402525, 2024-09-13T14:59:32.954 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:59:32.954 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:59:32.954 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:32.954 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:59:32.954 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.954 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:59:32.954 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.103314+0000", 2024-09-13T14:59:32.954 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:59:32.954 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.954 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.954 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:59:32.955 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.103314+0000", 2024-09-13T14:59:32.955 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:59:32.955 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.955 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.955 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:59:32.955 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.103315+0000", 2024-09-13T14:59:32.955 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 8.2500000000000004e-07 2024-09-13T14:59:32.955 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.955 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.955 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:59:32.955 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.103322+0000", 2024-09-13T14:59:32.955 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.2809999999999997e-06 2024-09-13T14:59:32.955 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.955 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.956 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:59:32.956 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.224833+0000", 2024-09-13T14:59:32.956 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.12151121400000001 2024-09-13T14:59:32.956 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.956 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.956 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:59:32.956 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.224861+0000", 2024-09-13T14:59:32.956 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.8138000000000001e-05 2024-09-13T14:59:32.956 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.956 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.956 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:59:32.956 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.226005+0000", 2024-09-13T14:59:32.956 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0011444510000000001 2024-09-13T14:59:32.957 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.957 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.957 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:59:32.957 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.226023+0000", 2024-09-13T14:59:32.958 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.8097e-05 2024-09-13T14:59:32.958 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.958 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.958 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:59:32.958 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.243492+0000", 2024-09-13T14:59:32.958 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.017468605000000002 2024-09-13T14:59:32.958 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.958 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.958 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:59:32.958 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.243567+0000", 2024-09-13T14:59:32.958 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.4888999999999994e-05 2024-09-13T14:59:32.958 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:59:32.958 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:59:32.959 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:59:32.959 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.959 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.959 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.36738.0:194174 10.2 e481/459 10:4273d6c4:::benchmark_data_smithi102_377304_object88332:head v 481'12080, mlcod=481'12077)", 2024-09-13T14:59:32.959 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:58:17.106220+0000", 2024-09-13T14:59:32.959 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 75.707255721999999, 2024-09-13T14:59:32.959 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.119079886, 2024-09-13T14:59:32.959 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:59:32.959 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:59:32.959 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:32.959 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:59:32.959 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.959 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:59:32.959 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.106220+0000", 2024-09-13T14:59:32.960 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:59:32.960 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.960 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.960 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:59:32.960 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.106220+0000", 2024-09-13T14:59:32.960 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:59:32.960 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.960 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.960 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:59:32.960 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.106220+0000", 2024-09-13T14:59:32.960 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.8199999999999999e-07 2024-09-13T14:59:32.960 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.960 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.961 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:59:32.961 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.106224+0000", 2024-09-13T14:59:32.961 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.8809999999999998e-06 2024-09-13T14:59:32.961 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.961 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.961 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:59:32.961 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.106236+0000", 2024-09-13T14:59:32.961 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.1557e-05 2024-09-13T14:59:32.961 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.961 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.961 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:59:32.961 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.106243+0000", 2024-09-13T14:59:32.961 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.5089999999999996e-06 2024-09-13T14:59:32.961 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.962 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.962 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:59:32.962 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.106340+0000", 2024-09-13T14:59:32.962 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 9.6983000000000003e-05 2024-09-13T14:59:32.962 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.962 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.962 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:59:32.962 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.106358+0000", 2024-09-13T14:59:32.962 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 1.8034999999999999e-05 2024-09-13T14:59:32.962 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.962 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.962 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:59:32.962 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.225256+0000", 2024-09-13T14:59:32.963 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.11889769 2024-09-13T14:59:32.963 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.963 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.963 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:59:32.963 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.225300+0000", 2024-09-13T14:59:32.963 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.3548999999999999e-05 2024-09-13T14:59:32.963 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:59:32.963 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:59:32.963 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:59:32.963 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.963 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.963 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop_reply(client.36738.0:194168 10.4 e481/462 ondisk, result = 0)", 2024-09-13T14:59:32.963 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:58:17.106262+0000", 2024-09-13T14:59:32.963 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 75.707213069000005, 2024-09-13T14:59:32.964 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.118822069, 2024-09-13T14:59:32.964 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:59:32.964 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:59:32.964 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "started", 2024-09-13T14:59:32.964 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:59:32.964 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.964 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:59:32.964 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.106262+0000", 2024-09-13T14:59:32.964 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:59:32.964 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.964 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.964 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:59:32.964 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.106262+0000", 2024-09-13T14:59:32.965 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:59:32.965 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.965 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.965 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:59:32.965 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.106263+0000", 2024-09-13T14:59:32.965 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.8999999999999998e-07 2024-09-13T14:59:32.965 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.965 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.965 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:59:32.965 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.106265+0000", 2024-09-13T14:59:32.965 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.5689999999999999e-06 2024-09-13T14:59:32.965 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.965 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.965 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:59:32.966 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.106271+0000", 2024-09-13T14:59:32.966 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.1190000000000002e-06 2024-09-13T14:59:32.966 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.966 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.966 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:59:32.966 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.106275+0000", 2024-09-13T14:59:32.966 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 4.0600000000000001e-06 2024-09-13T14:59:32.966 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.966 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.966 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:59:32.966 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.106918+0000", 2024-09-13T14:59:32.966 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.00064284800000000001 2024-09-13T14:59:32.966 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.967 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.967 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:59:32.967 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.106926+0000", 2024-09-13T14:59:32.967 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 7.4089999999999997e-06 2024-09-13T14:59:32.967 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.967 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.967 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:59:32.967 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.225085+0000", 2024-09-13T14:59:32.967 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.11815877399999999 2024-09-13T14:59:32.967 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:59:32.967 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:59:32.967 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:59:32.967 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.967 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.968 INFO:teuthology.orchestra.run.smithi102.stdout: "description": "osd_repop(client.36738.0:194176 10.2 e481/459 10:4102d8e2:::benchmark_data_smithi102_377304_object88334:head v 481'12081, mlcod=481'12078)", 2024-09-13T14:59:32.968 INFO:teuthology.orchestra.run.smithi102.stdout: "initiated_at": "2024-09-13T14:58:17.107723+0000", 2024-09-13T14:59:32.968 INFO:teuthology.orchestra.run.smithi102.stdout: "age": 75.705752844000003, 2024-09-13T14:59:32.968 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.12073995899999999, 2024-09-13T14:59:32.968 INFO:teuthology.orchestra.run.smithi102.stdout: "continuous": false, 2024-09-13T14:59:32.968 INFO:teuthology.orchestra.run.smithi102.stdout: "type_data": { 2024-09-13T14:59:32.968 INFO:teuthology.orchestra.run.smithi102.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:32.968 INFO:teuthology.orchestra.run.smithi102.stdout: "events": [ 2024-09-13T14:59:32.968 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.968 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "initiated", 2024-09-13T14:59:32.968 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.107723+0000", 2024-09-13T14:59:32.968 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:59:32.968 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.968 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.969 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "header_read", 2024-09-13T14:59:32.969 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.107723+0000", 2024-09-13T14:59:32.969 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0 2024-09-13T14:59:32.969 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.969 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.969 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "throttled", 2024-09-13T14:59:32.969 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.107723+0000", 2024-09-13T14:59:32.969 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 6.0699999999999997e-07 2024-09-13T14:59:32.969 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.969 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.969 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "all_read", 2024-09-13T14:59:32.969 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.107729+0000", 2024-09-13T14:59:32.969 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.7479999999999999e-06 2024-09-13T14:59:32.970 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.970 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.970 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "dispatched", 2024-09-13T14:59:32.970 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.225105+0000", 2024-09-13T14:59:32.970 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.117376296 2024-09-13T14:59:32.970 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.970 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.970 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "queued_for_pg", 2024-09-13T14:59:32.970 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.225111+0000", 2024-09-13T14:59:32.970 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 5.8440000000000003e-06 2024-09-13T14:59:32.970 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.970 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.970 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "reached_pg", 2024-09-13T14:59:32.970 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.225332+0000", 2024-09-13T14:59:32.971 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.000220836 2024-09-13T14:59:32.971 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.971 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.971 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "started", 2024-09-13T14:59:32.971 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.225353+0000", 2024-09-13T14:59:32.971 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 2.1036e-05 2024-09-13T14:59:32.971 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.971 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.971 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "commit_sent", 2024-09-13T14:59:32.971 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.228423+0000", 2024-09-13T14:59:32.971 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 0.0030699719999999998 2024-09-13T14:59:32.971 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-09-13T14:59:32.971 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-09-13T14:59:32.971 INFO:teuthology.orchestra.run.smithi102.stdout: "event": "done", 2024-09-13T14:59:32.972 INFO:teuthology.orchestra.run.smithi102.stdout: "time": "2024-09-13T14:58:17.228463+0000", 2024-09-13T14:59:32.972 INFO:teuthology.orchestra.run.smithi102.stdout: "duration": 3.9619999999999997e-05 2024-09-13T14:59:32.972 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:59:32.972 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:59:32.972 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:59:32.972 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-09-13T14:59:32.972 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-09-13T14:59:32.972 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-09-13T14:59:33.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:33 smithi102 ceph-mon[110183]: pgmap v1669: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:33.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:33 smithi102 ceph-mon[115269]: pgmap v1669: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:33.868 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.5 reset_purged_snaps_last 2024-09-13T14:59:33.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:33 smithi195 ceph-mon[104262]: pgmap v1669: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:34.349 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:34.370 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T14:59:34.585 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:59:35.557 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:35.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:35 smithi195 ceph-mon[104262]: pgmap v1670: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:35.788 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:35 smithi102 ceph-mon[115269]: pgmap v1670: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:35.789 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:35 smithi102 ceph-mon[110183]: pgmap v1670: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:36.349 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:59:36.349 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-13T14:59:36.349 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 0 2024-09-13T14:59:36.349 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:59:36.541 INFO:teuthology.orchestra.run.smithi102.stderr:nodeep-scrub is unset 2024-09-13T14:59:36.708 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:36 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/92362303' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:36.708 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:36 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:36.782 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:36 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/92362303' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:36.783 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:36 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:36.783 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:36 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/92362303' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:36.783 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:36 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:36.783 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:59:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:59:36] "GET /metrics HTTP/1.1" 200 34149 "" "Prometheus/2.51.0" 2024-09-13T14:59:37.039 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T14:59:37.243 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:59:37.621 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:37 smithi195 ceph-mon[104262]: pgmap v1671: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:37.621 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:37 smithi195 ceph-mon[104262]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:59:37.621 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:37 smithi195 ceph-mon[104262]: Cluster is now healthy 2024-09-13T14:59:37.621 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:37 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:59:37.621 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:37 smithi195 ceph-mon[104262]: osdmap e494: 8 total, 8 up, 7 in 2024-09-13T14:59:37.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:37 smithi102 ceph-mon[110183]: pgmap v1671: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:37.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:37 smithi102 ceph-mon[110183]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:59:37.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:37 smithi102 ceph-mon[110183]: Cluster is now healthy 2024-09-13T14:59:37.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:37 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:59:37.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:37 smithi102 ceph-mon[110183]: osdmap e494: 8 total, 8 up, 7 in 2024-09-13T14:59:37.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:37 smithi102 ceph-mon[115269]: pgmap v1671: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:37.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:37 smithi102 ceph-mon[115269]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:59:37.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:37 smithi102 ceph-mon[115269]: Cluster is now healthy 2024-09-13T14:59:37.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:37 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:59:37.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:37 smithi102 ceph-mon[115269]: osdmap e494: 8 total, 8 up, 7 in 2024-09-13T14:59:38.895 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:59:38.895 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-13T14:59:38.895 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-09-13T14:59:38.895 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-09-13T14:59:38.895 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:59:39.165 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.2 reset_purged_snaps_last 2024-09-13T14:59:39.365 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:39.549 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T14:59:39.753 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:59:39.794 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:39 smithi102 ceph-mon[110183]: pgmap v1673: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:39.794 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:39 smithi102 ceph-mon[115269]: pgmap v1673: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:39.874 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:39 smithi195 ceph-mon[104262]: pgmap v1673: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:41.405 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:59:41.406 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-09-13T14:59:41.406 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-09-13T14:59:41.406 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-13T14:59:41.406 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.406 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:191701 9.16 9:6edeef48:::benchmark_data_smithi102_348563_object70599:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e416)", 2024-09-13T14:59:41.406 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:53:45.825428+0000", 2024-09-13T14:59:41.406 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 355.57584119799998, 2024-09-13T14:59:41.406 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0089169039999999, 2024-09-13T14:59:41.406 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:41.406 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-13T14:59:41.407 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:41.407 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:59:41.407 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:59:41.407 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 191701 2024-09-13T14:59:41.407 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.407 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:41.407 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.407 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:41.407 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:45.825428+0000", 2024-09-13T14:59:41.407 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:41.407 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.407 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.407 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:41.408 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:45.825428+0000", 2024-09-13T14:59:41.408 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:41.408 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.408 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.408 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:41.408 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:45.825426+0000", 2024-09-13T14:59:41.408 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:59:41.408 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.408 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.408 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:41.408 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:45.825434+0000", 2024-09-13T14:59:41.408 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6939999999999995e-06 2024-09-13T14:59:41.408 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.409 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.409 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:41.409 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:45.825446+0000", 2024-09-13T14:59:41.409 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2466999999999999e-05 2024-09-13T14:59:41.409 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.409 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.409 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:41.409 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:45.825455+0000", 2024-09-13T14:59:41.409 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7909999999999993e-06 2024-09-13T14:59:41.409 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.410 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.410 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:41.410 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:45.828249+0000", 2024-09-13T14:59:41.410 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0027937119999999998 2024-09-13T14:59:41.410 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.410 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.410 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:41.410 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:45.828260+0000", 2024-09-13T14:59:41.410 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1411e-05 2024-09-13T14:59:41.410 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.410 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.410 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:41.410 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.834289+0000", 2024-09-13T14:59:41.410 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.006028682 2024-09-13T14:59:41.411 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.411 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.411 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:41.411 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.834345+0000", 2024-09-13T14:59:41.411 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5890999999999997e-05 2024-09-13T14:59:41.411 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:41.411 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:41.411 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:41.411 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.411 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.411 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:191711 9.16 9:6c8131eb:::benchmark_data_smithi102_348563_object70609:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e416)", 2024-09-13T14:59:41.411 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:53:45.825462+0000", 2024-09-13T14:59:41.411 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 355.57580712399999, 2024-09-13T14:59:41.412 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0181483579999999, 2024-09-13T14:59:41.412 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:41.412 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:41.412 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:41.412 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:59:41.412 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:59:41.412 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 191711 2024-09-13T14:59:41.412 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.412 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:41.412 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.413 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:41.413 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:45.825462+0000", 2024-09-13T14:59:41.413 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:41.413 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.413 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.413 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:41.413 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:45.825462+0000", 2024-09-13T14:59:41.413 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:41.413 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.413 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.413 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:41.413 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:45.825461+0000", 2024-09-13T14:59:41.413 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:59:41.414 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.414 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.414 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:41.414 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:45.825464+0000", 2024-09-13T14:59:41.414 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8619999999999999e-06 2024-09-13T14:59:41.414 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.414 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.414 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:41.414 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:45.825471+0000", 2024-09-13T14:59:41.414 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1029999999999996e-06 2024-09-13T14:59:41.414 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.414 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.414 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:41.414 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:45.825476+0000", 2024-09-13T14:59:41.415 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2340000000000002e-06 2024-09-13T14:59:41.415 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.415 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.415 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:41.415 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:45.828804+0000", 2024-09-13T14:59:41.415 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0033276669999999999 2024-09-13T14:59:41.415 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.415 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.415 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:41.415 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:45.828814+0000", 2024-09-13T14:59:41.415 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0182e-05 2024-09-13T14:59:41.415 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.415 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.415 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:41.416 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.834670+0000", 2024-09-13T14:59:41.416 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0058554689999999 2024-09-13T14:59:41.416 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.416 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.416 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:59:41.416 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.834786+0000", 2024-09-13T14:59:41.416 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000116667 2024-09-13T14:59:41.416 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.416 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.416 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:59:41.416 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.835209+0000", 2024-09-13T14:59:41.416 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00042288599999999998 2024-09-13T14:59:41.416 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.416 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.416 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:41.417 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.839559+0000", 2024-09-13T14:59:41.417 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0043492799999999996 2024-09-13T14:59:41.417 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.417 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.417 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:59:41.417 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.839616+0000", 2024-09-13T14:59:41.417 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7541999999999999e-05 2024-09-13T14:59:41.417 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.417 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.417 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:41.417 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.843527+0000", 2024-09-13T14:59:41.417 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0039107550000000001 2024-09-13T14:59:41.417 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.418 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.418 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:59:41.418 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.843575+0000", 2024-09-13T14:59:41.418 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8584000000000001e-05 2024-09-13T14:59:41.418 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.418 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.418 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:41.418 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.843610+0000", 2024-09-13T14:59:41.418 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4585999999999997e-05 2024-09-13T14:59:41.418 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:41.418 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:41.418 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:41.418 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.418 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.419 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:191724 9.16 9:6fed871c:::benchmark_data_smithi102_348563_object70622:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-13T14:59:41.419 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:53:45.840429+0000", 2024-09-13T14:59:41.419 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 355.56084006700002, 2024-09-13T14:59:41.419 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0107586420000001, 2024-09-13T14:59:41.419 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:41.419 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:41.419 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:41.419 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:59:41.419 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:59:41.420 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 191724 2024-09-13T14:59:41.420 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.420 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:41.420 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.420 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:41.420 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:45.840429+0000", 2024-09-13T14:59:41.420 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:41.420 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.420 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.421 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:41.421 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:45.840429+0000", 2024-09-13T14:59:41.421 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:41.421 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.421 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.421 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:41.422 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:45.840427+0000", 2024-09-13T14:59:41.422 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:59:41.422 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.422 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.422 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:41.423 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:45.840435+0000", 2024-09-13T14:59:41.423 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9889999999999992e-06 2024-09-13T14:59:41.423 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.423 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.423 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:41.423 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:45.840453+0000", 2024-09-13T14:59:41.424 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8539999999999999e-05 2024-09-13T14:59:41.424 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.424 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.424 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:41.424 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:45.840462+0000", 2024-09-13T14:59:41.424 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3010000000000007e-06 2024-09-13T14:59:41.424 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.425 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.425 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:41.425 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:45.840607+0000", 2024-09-13T14:59:41.425 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014534899999999999 2024-09-13T14:59:41.425 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.425 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.425 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:41.426 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:45.840620+0000", 2024-09-13T14:59:41.426 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3362e-05 2024-09-13T14:59:41.426 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.426 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.426 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:41.426 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.837357+0000", 2024-09-13T14:59:41.426 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.996737133 2024-09-13T14:59:41.426 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.426 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.426 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:59:41.426 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.837444+0000", 2024-09-13T14:59:41.426 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.6049000000000002e-05 2024-09-13T14:59:41.426 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.427 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.427 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:59:41.427 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.837738+0000", 2024-09-13T14:59:41.427 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00029419400000000001 2024-09-13T14:59:41.427 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.427 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.427 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:59:41.427 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.843473+0000", 2024-09-13T14:59:41.427 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0057355770000000004 2024-09-13T14:59:41.427 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.427 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.427 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:41.427 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.851041+0000", 2024-09-13T14:59:41.427 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0075677069999999999 2024-09-13T14:59:41.428 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.428 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.428 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:41.428 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.851107+0000", 2024-09-13T14:59:41.428 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6419000000000002e-05 2024-09-13T14:59:41.428 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.428 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.428 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:59:41.429 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.851152+0000", 2024-09-13T14:59:41.429 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4934e-05 2024-09-13T14:59:41.429 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.429 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.429 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:41.430 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.851187+0000", 2024-09-13T14:59:41.430 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5036000000000001e-05 2024-09-13T14:59:41.430 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:41.430 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:41.430 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:41.430 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.430 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.430 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:191802 9.16 9:6daa7e99:::benchmark_data_smithi102_348563_object70700:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-13T14:59:41.430 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:53:45.912611+0000", 2024-09-13T14:59:41.431 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 355.48865806200001, 2024-09-13T14:59:41.431 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.94674192099999999, 2024-09-13T14:59:41.431 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:41.431 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:41.431 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:41.431 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:59:41.432 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:59:41.432 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 191802 2024-09-13T14:59:41.432 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.432 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:41.432 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.432 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:41.433 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:45.912611+0000", 2024-09-13T14:59:41.433 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:41.433 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.433 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.433 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:41.433 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:45.912611+0000", 2024-09-13T14:59:41.433 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:41.434 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.434 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.434 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:41.434 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:45.912610+0000", 2024-09-13T14:59:41.434 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:59:41.434 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.435 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.435 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:41.435 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:45.912617+0000", 2024-09-13T14:59:41.435 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5039999999999996e-06 2024-09-13T14:59:41.435 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.435 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.435 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:41.436 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:45.912631+0000", 2024-09-13T14:59:41.436 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3767e-05 2024-09-13T14:59:41.436 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.436 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.436 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:41.436 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:45.912638+0000", 2024-09-13T14:59:41.436 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.793e-06 2024-09-13T14:59:41.437 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.437 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.437 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:41.437 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:45.912689+0000", 2024-09-13T14:59:41.437 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1051999999999998e-05 2024-09-13T14:59:41.437 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.437 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.437 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:41.438 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:45.912698+0000", 2024-09-13T14:59:41.438 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.2609999999999999e-06 2024-09-13T14:59:41.438 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.438 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.438 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:41.438 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.839648+0000", 2024-09-13T14:59:41.439 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.92694945200000001 2024-09-13T14:59:41.439 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.439 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.439 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:59:41.439 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.839760+0000", 2024-09-13T14:59:41.439 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000112087 2024-09-13T14:59:41.439 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.440 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.440 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:59:41.440 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.840060+0000", 2024-09-13T14:59:41.440 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00030009500000000001 2024-09-13T14:59:41.440 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.440 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.440 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:59:41.441 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.845048+0000", 2024-09-13T14:59:41.441 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0049877469999999998 2024-09-13T14:59:41.441 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.441 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.441 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:41.442 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.859205+0000", 2024-09-13T14:59:41.442 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014157148 2024-09-13T14:59:41.442 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.442 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.442 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:41.442 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.859277+0000", 2024-09-13T14:59:41.442 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2513000000000002e-05 2024-09-13T14:59:41.443 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.443 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.443 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:59:41.443 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.859314+0000", 2024-09-13T14:59:41.443 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7177e-05 2024-09-13T14:59:41.443 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.443 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.444 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:41.444 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.859353+0000", 2024-09-13T14:59:41.444 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8170000000000002e-05 2024-09-13T14:59:41.444 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:41.444 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:41.444 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:41.444 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.444 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.445 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:191843 9.16 9:6feb6594:::benchmark_data_smithi102_348563_object70741:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-13T14:59:41.445 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:53:45.953489+0000", 2024-09-13T14:59:41.445 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 355.44777938499999, 2024-09-13T14:59:41.445 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.90609784199999999, 2024-09-13T14:59:41.445 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:41.445 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:41.446 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:41.446 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:59:41.446 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:59:41.446 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 191843 2024-09-13T14:59:41.446 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.446 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:41.446 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.447 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:41.447 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:45.953489+0000", 2024-09-13T14:59:41.447 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:41.447 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.447 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.447 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:41.447 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:45.953489+0000", 2024-09-13T14:59:41.447 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:41.450 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.450 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.450 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:41.450 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:45.953488+0000", 2024-09-13T14:59:41.450 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:41.451 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.451 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.451 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:41.451 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:45.953495+0000", 2024-09-13T14:59:41.451 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4120000000000004e-06 2024-09-13T14:59:41.451 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.451 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.451 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:41.452 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:45.953507+0000", 2024-09-13T14:59:41.452 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1528999999999999e-05 2024-09-13T14:59:41.452 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.452 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.452 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:41.452 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:45.953513+0000", 2024-09-13T14:59:41.452 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.6099999999999997e-06 2024-09-13T14:59:41.453 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.453 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.453 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:41.453 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:45.953576+0000", 2024-09-13T14:59:41.453 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3157999999999994e-05 2024-09-13T14:59:41.453 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.453 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.454 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:41.454 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:45.953586+0000", 2024-09-13T14:59:41.454 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0152e-05 2024-09-13T14:59:41.454 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.454 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.454 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:41.454 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.841843+0000", 2024-09-13T14:59:41.454 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.88825671100000003 2024-09-13T14:59:41.455 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.455 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.455 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:59:41.455 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.841947+0000", 2024-09-13T14:59:41.455 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000104428 2024-09-13T14:59:41.455 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.455 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.456 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:59:41.456 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.842169+0000", 2024-09-13T14:59:41.456 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00022241700000000001 2024-09-13T14:59:41.456 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.456 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.456 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:59:41.456 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.848875+0000", 2024-09-13T14:59:41.456 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0067057660000000002 2024-09-13T14:59:41.457 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.457 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.457 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:41.457 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.859442+0000", 2024-09-13T14:59:41.457 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010567183000000001 2024-09-13T14:59:41.457 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.457 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.458 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:41.458 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.859500+0000", 2024-09-13T14:59:41.458 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7966e-05 2024-09-13T14:59:41.458 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.458 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.458 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:59:41.458 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.859549+0000", 2024-09-13T14:59:41.458 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8196999999999999e-05 2024-09-13T14:59:41.459 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.459 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.459 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:41.459 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.859587+0000", 2024-09-13T14:59:41.459 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8674000000000003e-05 2024-09-13T14:59:41.459 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:41.459 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:41.460 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:41.460 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.460 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.460 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:191880 9.16 9:6e1caa36:::benchmark_data_smithi102_348563_object70778:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-13T14:59:41.460 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:53:45.989056+0000", 2024-09-13T14:59:41.460 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 355.4122127, 2024-09-13T14:59:41.460 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.87072909399999998, 2024-09-13T14:59:41.461 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:41.461 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:41.461 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:41.461 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:59:41.461 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:59:41.461 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 191880 2024-09-13T14:59:41.461 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.462 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:41.462 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.462 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:41.462 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:45.989056+0000", 2024-09-13T14:59:41.462 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:41.462 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.462 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.462 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:41.463 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:45.989056+0000", 2024-09-13T14:59:41.463 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:41.463 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.463 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.463 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:41.463 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:45.989053+0000", 2024-09-13T14:59:41.463 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999971 2024-09-13T14:59:41.464 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.464 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.464 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:41.464 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:45.989064+0000", 2024-09-13T14:59:41.464 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0423e-05 2024-09-13T14:59:41.464 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.464 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.465 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:41.465 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:45.989087+0000", 2024-09-13T14:59:41.465 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2819999999999998e-05 2024-09-13T14:59:41.465 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.465 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.465 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:41.465 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:45.989095+0000", 2024-09-13T14:59:41.465 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.6680000000000007e-06 2024-09-13T14:59:41.466 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.466 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.466 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:41.466 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:45.989153+0000", 2024-09-13T14:59:41.466 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7819000000000003e-05 2024-09-13T14:59:41.466 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.466 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.466 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:41.466 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:45.989163+0000", 2024-09-13T14:59:41.466 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0071999999999999e-05 2024-09-13T14:59:41.467 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.467 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.467 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:41.467 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.843655+0000", 2024-09-13T14:59:41.467 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.85449162700000003 2024-09-13T14:59:41.467 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.467 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.467 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:59:41.468 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.843748+0000", 2024-09-13T14:59:41.468 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.3183000000000006e-05 2024-09-13T14:59:41.468 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.468 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.468 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:59:41.468 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.843952+0000", 2024-09-13T14:59:41.468 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00020431799999999999 2024-09-13T14:59:41.468 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.469 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.469 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:59:41.469 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.851050+0000", 2024-09-13T14:59:41.469 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0070972070000000003 2024-09-13T14:59:41.469 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.469 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.469 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:41.469 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.859640+0000", 2024-09-13T14:59:41.470 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0085908600000000005 2024-09-13T14:59:41.470 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.470 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.470 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:41.470 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.859694+0000", 2024-09-13T14:59:41.470 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4070000000000002e-05 2024-09-13T14:59:41.470 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.470 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.471 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:59:41.471 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.859750+0000", 2024-09-13T14:59:41.471 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5467999999999998e-05 2024-09-13T14:59:41.471 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.471 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.471 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:41.471 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.859785+0000", 2024-09-13T14:59:41.471 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5275999999999999e-05 2024-09-13T14:59:41.472 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:41.472 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:41.472 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:41.472 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.472 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.472 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:191906 9.16 9:6a14e7bb:::benchmark_data_smithi102_348563_object70804:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-13T14:59:41.472 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:53:46.018866+0000", 2024-09-13T14:59:41.472 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 355.382402845, 2024-09-13T14:59:41.473 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.84110149199999995, 2024-09-13T14:59:41.473 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:41.473 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:41.473 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:41.473 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:59:41.473 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:59:41.473 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 191906 2024-09-13T14:59:41.474 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.474 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:41.474 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.474 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:41.474 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.018866+0000", 2024-09-13T14:59:41.474 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:41.474 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.474 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.474 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:41.475 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.018866+0000", 2024-09-13T14:59:41.475 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:41.475 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.475 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.475 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:41.475 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.018864+0000", 2024-09-13T14:59:41.475 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:59:41.476 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.476 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.476 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:41.476 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.018870+0000", 2024-09-13T14:59:41.476 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1480000000000003e-06 2024-09-13T14:59:41.476 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.476 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.477 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:41.477 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.018880+0000", 2024-09-13T14:59:41.477 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.5599999999999999e-06 2024-09-13T14:59:41.477 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.477 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.477 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:41.477 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.018886+0000", 2024-09-13T14:59:41.478 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7119999999999997e-06 2024-09-13T14:59:41.478 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.478 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.478 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:41.478 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.018945+0000", 2024-09-13T14:59:41.478 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.9015000000000003e-05 2024-09-13T14:59:41.478 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.479 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.479 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:41.479 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.018955+0000", 2024-09-13T14:59:41.479 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0573e-05 2024-09-13T14:59:41.479 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.479 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.479 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:41.479 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.845078+0000", 2024-09-13T14:59:41.480 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.82612322400000004 2024-09-13T14:59:41.480 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.480 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.480 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:59:41.480 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.845150+0000", 2024-09-13T14:59:41.480 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1528999999999999e-05 2024-09-13T14:59:41.480 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.480 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.480 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:59:41.481 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.845393+0000", 2024-09-13T14:59:41.481 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000243024 2024-09-13T14:59:41.481 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.481 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.481 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:59:41.481 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.851055+0000", 2024-09-13T14:59:41.481 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.005662434 2024-09-13T14:59:41.482 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.482 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.482 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:41.482 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.859840+0000", 2024-09-13T14:59:41.482 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0087847840000000003 2024-09-13T14:59:41.482 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.482 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.482 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:41.483 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.859879+0000", 2024-09-13T14:59:41.483 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8384999999999999e-05 2024-09-13T14:59:41.483 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.483 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.483 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:59:41.483 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.859937+0000", 2024-09-13T14:59:41.483 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8291000000000001e-05 2024-09-13T14:59:41.483 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.484 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.484 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:41.484 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.859967+0000", 2024-09-13T14:59:41.484 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0510000000000001e-05 2024-09-13T14:59:41.484 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:41.484 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:41.484 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:41.485 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.485 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.485 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:191941 9.16 9:6d5cf8ff:::benchmark_data_smithi102_348563_object70839:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-13T14:59:41.485 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:53:46.050950+0000", 2024-09-13T14:59:41.485 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 355.35031915000002, 2024-09-13T14:59:41.485 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.81726218900000003, 2024-09-13T14:59:41.485 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:41.485 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:41.486 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:41.486 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:59:41.486 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:59:41.486 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 191941 2024-09-13T14:59:41.486 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.487 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:41.487 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.487 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:41.487 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.050950+0000", 2024-09-13T14:59:41.487 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:41.487 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.487 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.488 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:41.488 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.050950+0000", 2024-09-13T14:59:41.488 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:41.488 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.488 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.488 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:41.488 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.050948+0000", 2024-09-13T14:59:41.488 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:41.489 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.489 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.489 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:41.489 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.050955+0000", 2024-09-13T14:59:41.489 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2420000000000003e-06 2024-09-13T14:59:41.489 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.489 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.490 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:41.490 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.050969+0000", 2024-09-13T14:59:41.490 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4151000000000001e-05 2024-09-13T14:59:41.490 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.490 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.490 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:41.490 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.050977+0000", 2024-09-13T14:59:41.491 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3610000000000003e-06 2024-09-13T14:59:41.491 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.491 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.491 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:41.491 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.051037+0000", 2024-09-13T14:59:41.491 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0266999999999997e-05 2024-09-13T14:59:41.492 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.492 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.492 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:41.492 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.051048+0000", 2024-09-13T14:59:41.492 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0946e-05 2024-09-13T14:59:41.492 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.493 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.493 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:41.493 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.846707+0000", 2024-09-13T14:59:41.493 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.79565901800000005 2024-09-13T14:59:41.493 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.493 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.494 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:59:41.494 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.846824+0000", 2024-09-13T14:59:41.494 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011684399999999999 2024-09-13T14:59:41.494 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.494 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.494 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:59:41.495 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.847101+0000", 2024-09-13T14:59:41.495 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00027688799999999998 2024-09-13T14:59:41.495 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.495 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.495 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:59:41.495 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.854364+0000", 2024-09-13T14:59:41.495 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0072627639999999997 2024-09-13T14:59:41.495 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.496 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.496 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:41.496 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.865467+0000", 2024-09-13T14:59:41.496 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011103959 2024-09-13T14:59:41.496 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.496 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.496 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:41.497 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.868125+0000", 2024-09-13T14:59:41.497 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0026576270000000001 2024-09-13T14:59:41.497 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.497 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.497 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:59:41.497 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.868175+0000", 2024-09-13T14:59:41.498 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0268000000000001e-05 2024-09-13T14:59:41.498 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.498 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.498 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:41.498 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.868212+0000", 2024-09-13T14:59:41.498 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6463e-05 2024-09-13T14:59:41.498 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:41.498 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:41.498 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:41.498 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.498 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.498 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:192038 9.16 9:6aa38e62:::benchmark_data_smithi102_348563_object70936:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-13T14:59:41.498 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:53:46.147639+0000", 2024-09-13T14:59:41.498 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 355.25362982600001, 2024-09-13T14:59:41.499 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.72067591900000005, 2024-09-13T14:59:41.499 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:41.499 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:41.499 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:41.499 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:59:41.499 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:59:41.499 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 192038 2024-09-13T14:59:41.499 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.499 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:41.499 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.499 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:41.499 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.147639+0000", 2024-09-13T14:59:41.499 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:41.500 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.500 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.500 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:41.500 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.147639+0000", 2024-09-13T14:59:41.500 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:41.500 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.500 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.500 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:41.500 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.147637+0000", 2024-09-13T14:59:41.500 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:59:41.500 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.500 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.501 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:41.501 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.147645+0000", 2024-09-13T14:59:41.501 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0039999999999999e-06 2024-09-13T14:59:41.501 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.501 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.501 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:41.501 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.147664+0000", 2024-09-13T14:59:41.501 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8899999999999999e-05 2024-09-13T14:59:41.501 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.501 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.501 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:41.501 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.147673+0000", 2024-09-13T14:59:41.501 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.49e-06 2024-09-13T14:59:41.502 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.502 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.502 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:41.502 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.147727+0000", 2024-09-13T14:59:41.502 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4450999999999998e-05 2024-09-13T14:59:41.502 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.502 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.502 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:41.502 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.147737+0000", 2024-09-13T14:59:41.502 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0213e-05 2024-09-13T14:59:41.502 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.502 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.502 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:41.502 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.848927+0000", 2024-09-13T14:59:41.503 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.70118950899999999 2024-09-13T14:59:41.503 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.503 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.503 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:59:41.503 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.849037+0000", 2024-09-13T14:59:41.503 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011042799999999999 2024-09-13T14:59:41.503 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.503 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.503 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:59:41.503 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.849340+0000", 2024-09-13T14:59:41.503 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00030273799999999999 2024-09-13T14:59:41.503 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.503 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.503 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:59:41.504 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.857204+0000", 2024-09-13T14:59:41.504 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0078638319999999994 2024-09-13T14:59:41.504 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.504 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.504 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:41.504 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.865508+0000", 2024-09-13T14:59:41.504 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0083044620000000003 2024-09-13T14:59:41.504 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.504 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.504 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:41.504 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.868245+0000", 2024-09-13T14:59:41.504 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002736961 2024-09-13T14:59:41.504 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.505 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.505 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:59:41.505 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.868285+0000", 2024-09-13T14:59:41.505 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9731999999999998e-05 2024-09-13T14:59:41.505 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.505 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.505 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:41.505 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.868315+0000", 2024-09-13T14:59:41.505 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0046999999999999e-05 2024-09-13T14:59:41.505 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:41.505 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:41.505 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:41.505 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.505 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.506 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:192041 9.16 9:6814981f:::benchmark_data_smithi102_348563_object70939:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-13T14:59:41.506 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:53:46.150593+0000", 2024-09-13T14:59:41.506 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 355.25067534499999, 2024-09-13T14:59:41.506 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.72754937200000003, 2024-09-13T14:59:41.506 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:41.506 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:41.506 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:41.506 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:59:41.506 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:59:41.506 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 192041 2024-09-13T14:59:41.506 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.506 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:41.506 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.507 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:41.507 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.150593+0000", 2024-09-13T14:59:41.507 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:41.507 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.507 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.507 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:41.507 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.150593+0000", 2024-09-13T14:59:41.507 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:41.507 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.507 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.507 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:41.507 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.150592+0000", 2024-09-13T14:59:41.507 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:41.508 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.508 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.508 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:41.508 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.150601+0000", 2024-09-13T14:59:41.508 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.4810000000000006e-06 2024-09-13T14:59:41.508 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.508 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.508 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:41.508 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.150616+0000", 2024-09-13T14:59:41.508 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5451000000000002e-05 2024-09-13T14:59:41.508 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.508 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.508 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:41.509 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.150624+0000", 2024-09-13T14:59:41.509 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1470000000000001e-06 2024-09-13T14:59:41.509 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.509 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.509 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:41.509 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.150678+0000", 2024-09-13T14:59:41.509 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.427e-05 2024-09-13T14:59:41.509 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.509 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.509 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:41.510 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.150688+0000", 2024-09-13T14:59:41.510 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.7620000000000007e-06 2024-09-13T14:59:41.510 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.510 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.510 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:41.510 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.851230+0000", 2024-09-13T14:59:41.510 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.70054203299999995 2024-09-13T14:59:41.510 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.510 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.510 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:59:41.510 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.851314+0000", 2024-09-13T14:59:41.510 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3374000000000005e-05 2024-09-13T14:59:41.510 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.510 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.511 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:59:41.511 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.851559+0000", 2024-09-13T14:59:41.511 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00024558499999999999 2024-09-13T14:59:41.511 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.511 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.511 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:59:41.511 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.857210+0000", 2024-09-13T14:59:41.511 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0056510730000000004 2024-09-13T14:59:41.511 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.511 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.511 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:41.511 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.867557+0000", 2024-09-13T14:59:41.511 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010346981 2024-09-13T14:59:41.512 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.512 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.512 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:41.512 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.878080+0000", 2024-09-13T14:59:41.512 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010522557 2024-09-13T14:59:41.512 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.512 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.512 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:59:41.512 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.878112+0000", 2024-09-13T14:59:41.512 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.1967000000000001e-05 2024-09-13T14:59:41.512 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.512 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.512 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:41.512 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.878143+0000", 2024-09-13T14:59:41.513 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.1065999999999997e-05 2024-09-13T14:59:41.513 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:41.513 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:41.513 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:41.513 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.513 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.513 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:192075 9.16 9:6c96767a:::benchmark_data_smithi102_348563_object70973:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-13T14:59:41.513 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:53:46.189378+0000", 2024-09-13T14:59:41.513 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 355.21189062299999, 2024-09-13T14:59:41.513 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.68891249700000001, 2024-09-13T14:59:41.513 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:41.513 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:41.513 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:41.514 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:59:41.514 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:59:41.514 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 192075 2024-09-13T14:59:41.514 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.514 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:41.514 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.514 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:41.514 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.189378+0000", 2024-09-13T14:59:41.514 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:41.514 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.514 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.514 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:41.514 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.189378+0000", 2024-09-13T14:59:41.515 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:41.515 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.515 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.515 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:41.515 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.189376+0000", 2024-09-13T14:59:41.515 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-09-13T14:59:41.515 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.515 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.515 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:41.515 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.189385+0000", 2024-09-13T14:59:41.515 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.7459999999999995e-06 2024-09-13T14:59:41.515 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.515 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.515 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:41.516 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.189406+0000", 2024-09-13T14:59:41.516 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0407e-05 2024-09-13T14:59:41.516 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.516 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.516 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:41.516 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.189415+0000", 2024-09-13T14:59:41.516 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4979999999999995e-06 2024-09-13T14:59:41.516 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.516 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.516 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:41.516 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.189480+0000", 2024-09-13T14:59:41.516 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4231000000000002e-05 2024-09-13T14:59:41.516 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.516 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.517 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:41.517 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.189491+0000", 2024-09-13T14:59:41.517 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1412e-05 2024-09-13T14:59:41.517 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.517 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.517 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:41.517 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.853042+0000", 2024-09-13T14:59:41.517 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.66355140199999996 2024-09-13T14:59:41.517 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.517 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.517 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:59:41.517 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.853105+0000", 2024-09-13T14:59:41.517 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2198000000000003e-05 2024-09-13T14:59:41.518 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.518 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.518 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:59:41.518 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.853297+0000", 2024-09-13T14:59:41.518 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00019209500000000001 2024-09-13T14:59:41.518 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.518 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.518 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:59:41.518 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.859214+0000", 2024-09-13T14:59:41.518 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0059174060000000001 2024-09-13T14:59:41.518 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.518 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.518 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:41.518 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.867586+0000", 2024-09-13T14:59:41.519 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0083717089999999997 2024-09-13T14:59:41.519 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.519 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.519 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:41.519 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.878186+0000", 2024-09-13T14:59:41.519 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.01060065 2024-09-13T14:59:41.519 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.519 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.519 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:59:41.519 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.878256+0000", 2024-09-13T14:59:41.519 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9319999999999994e-05 2024-09-13T14:59:41.519 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.519 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.519 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:41.520 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.878291+0000", 2024-09-13T14:59:41.520 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4938000000000003e-05 2024-09-13T14:59:41.520 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:41.520 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:41.520 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:41.520 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.520 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.520 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:192082 9.16 9:6f1c9a5b:::benchmark_data_smithi102_348563_object70980:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-13T14:59:41.520 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:53:46.202778+0000", 2024-09-13T14:59:41.520 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 355.19849047000002, 2024-09-13T14:59:41.520 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.67569894699999999, 2024-09-13T14:59:41.520 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:41.520 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:41.520 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:41.521 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:59:41.521 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:59:41.521 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 192082 2024-09-13T14:59:41.521 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.521 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:41.521 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.521 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:41.521 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.202778+0000", 2024-09-13T14:59:41.521 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:41.521 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.521 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.521 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:41.521 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.202778+0000", 2024-09-13T14:59:41.521 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:41.522 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.522 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.522 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:41.522 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.202776+0000", 2024-09-13T14:59:41.522 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999976 2024-09-13T14:59:41.522 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.522 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.522 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:41.522 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.202786+0000", 2024-09-13T14:59:41.522 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0049e-05 2024-09-13T14:59:41.522 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.522 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.522 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:41.522 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.202806+0000", 2024-09-13T14:59:41.523 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0088000000000001e-05 2024-09-13T14:59:41.523 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.523 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.523 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:41.523 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.202815+0000", 2024-09-13T14:59:41.523 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7239999999999998e-06 2024-09-13T14:59:41.523 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.523 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.523 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:41.523 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.202872+0000", 2024-09-13T14:59:41.523 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7544000000000003e-05 2024-09-13T14:59:41.523 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.523 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.523 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:41.524 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.202885+0000", 2024-09-13T14:59:41.524 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2383e-05 2024-09-13T14:59:41.524 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.524 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.524 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:41.524 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.854397+0000", 2024-09-13T14:59:41.524 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.65151208800000004 2024-09-13T14:59:41.524 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.524 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.524 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:59:41.524 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.854493+0000", 2024-09-13T14:59:41.524 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.5933999999999993e-05 2024-09-13T14:59:41.524 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.525 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.525 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:59:41.525 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.854686+0000", 2024-09-13T14:59:41.525 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00019366699999999999 2024-09-13T14:59:41.525 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.525 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.525 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:59:41.525 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.859231+0000", 2024-09-13T14:59:41.525 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0045451329999999998 2024-09-13T14:59:41.525 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.525 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.525 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:41.525 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.878334+0000", 2024-09-13T14:59:41.525 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.019102266999999999 2024-09-13T14:59:41.525 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.526 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.526 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:41.526 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.878376+0000", 2024-09-13T14:59:41.526 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2589000000000002e-05 2024-09-13T14:59:41.526 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.526 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.526 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:59:41.526 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.878439+0000", 2024-09-13T14:59:41.526 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2638000000000006e-05 2024-09-13T14:59:41.526 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.526 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.526 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:41.526 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.878477+0000", 2024-09-13T14:59:41.526 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8284000000000001e-05 2024-09-13T14:59:41.527 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:41.527 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:41.527 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:41.527 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.527 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.527 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:192096 9.16 9:6e9f04c3:::benchmark_data_smithi102_348563_object70994:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-13T14:59:41.527 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:53:46.222519+0000", 2024-09-13T14:59:41.527 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 355.17875018799998, 2024-09-13T14:59:41.527 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.66234312699999998, 2024-09-13T14:59:41.527 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:41.527 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:41.527 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:41.527 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:59:41.527 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:59:41.528 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 192096 2024-09-13T14:59:41.528 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.528 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:41.528 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.528 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:41.528 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.222519+0000", 2024-09-13T14:59:41.528 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:41.528 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.528 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.528 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:41.528 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.222519+0000", 2024-09-13T14:59:41.528 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:41.528 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.529 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.529 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:41.529 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.222517+0000", 2024-09-13T14:59:41.529 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:59:41.529 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.529 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.529 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:41.529 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.222524+0000", 2024-09-13T14:59:41.529 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5950000000000001e-06 2024-09-13T14:59:41.529 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.529 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.529 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:41.529 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.222542+0000", 2024-09-13T14:59:41.529 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7557000000000001e-05 2024-09-13T14:59:41.530 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.530 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.530 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:41.530 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.222549+0000", 2024-09-13T14:59:41.530 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2139999999999999e-06 2024-09-13T14:59:41.530 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.530 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.530 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:41.530 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.222596+0000", 2024-09-13T14:59:41.530 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7139999999999999e-05 2024-09-13T14:59:41.530 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.530 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.530 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:41.530 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.222607+0000", 2024-09-13T14:59:41.531 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0909e-05 2024-09-13T14:59:41.531 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.531 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.531 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:41.531 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.855812+0000", 2024-09-13T14:59:41.531 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.63320483000000005 2024-09-13T14:59:41.531 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.531 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.531 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:59:41.531 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.855870+0000", 2024-09-13T14:59:41.531 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8505999999999999e-05 2024-09-13T14:59:41.531 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.531 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.531 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:59:41.532 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.856066+0000", 2024-09-13T14:59:41.532 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000195591 2024-09-13T14:59:41.532 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.532 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.532 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:59:41.532 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.859451+0000", 2024-09-13T14:59:41.532 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0033852190000000001 2024-09-13T14:59:41.532 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.532 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.532 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:41.532 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.884595+0000", 2024-09-13T14:59:41.532 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.025143714000000001 2024-09-13T14:59:41.532 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.532 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.533 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:41.533 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.884774+0000", 2024-09-13T14:59:41.533 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00017880500000000001 2024-09-13T14:59:41.533 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.533 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.533 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:59:41.533 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.884821+0000", 2024-09-13T14:59:41.533 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7274000000000002e-05 2024-09-13T14:59:41.533 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.533 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.533 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:41.533 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.884862+0000", 2024-09-13T14:59:41.533 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0661000000000003e-05 2024-09-13T14:59:41.533 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:41.534 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:41.534 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:41.534 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.534 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.534 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:192119 9.16 9:6e11a473:::benchmark_data_smithi102_348563_object71017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-13T14:59:41.534 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:53:46.252785+0000", 2024-09-13T14:59:41.534 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 355.14848340999998, 2024-09-13T14:59:41.534 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.63218929300000004, 2024-09-13T14:59:41.534 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:41.534 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:41.534 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:41.534 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:59:41.534 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:59:41.534 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 192119 2024-09-13T14:59:41.535 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.535 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:41.535 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.535 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:41.535 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.252785+0000", 2024-09-13T14:59:41.535 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:41.535 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.535 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.535 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:41.535 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.252785+0000", 2024-09-13T14:59:41.535 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:41.535 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.535 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.535 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:41.536 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.252784+0000", 2024-09-13T14:59:41.536 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:41.536 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.536 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.536 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:41.536 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.252791+0000", 2024-09-13T14:59:41.536 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5170000000000002e-06 2024-09-13T14:59:41.536 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.536 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.536 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:41.536 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.252806+0000", 2024-09-13T14:59:41.536 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5046e-05 2024-09-13T14:59:41.536 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.536 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.537 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:41.537 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.252814+0000", 2024-09-13T14:59:41.537 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1510000000000004e-06 2024-09-13T14:59:41.537 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.537 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.537 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:41.537 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.252862+0000", 2024-09-13T14:59:41.537 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7349999999999999e-05 2024-09-13T14:59:41.537 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.537 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.537 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:41.537 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.252873+0000", 2024-09-13T14:59:41.537 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1398e-05 2024-09-13T14:59:41.537 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.538 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.538 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:41.538 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.857249+0000", 2024-09-13T14:59:41.538 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.60437556800000003 2024-09-13T14:59:41.538 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.538 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.538 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:59:41.538 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.857351+0000", 2024-09-13T14:59:41.538 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000102181 2024-09-13T14:59:41.538 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.538 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.538 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:59:41.538 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.857673+0000", 2024-09-13T14:59:41.539 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00032154500000000001 2024-09-13T14:59:41.539 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.539 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.539 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:59:41.539 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.859649+0000", 2024-09-13T14:59:41.539 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0019765970000000001 2024-09-13T14:59:41.539 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.539 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.539 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:41.539 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.884627+0000", 2024-09-13T14:59:41.539 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024977727000000002 2024-09-13T14:59:41.539 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.539 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.539 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:41.540 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.884903+0000", 2024-09-13T14:59:41.540 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000276631 2024-09-13T14:59:41.540 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.540 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.540 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:59:41.540 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.884952+0000", 2024-09-13T14:59:41.540 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8925999999999999e-05 2024-09-13T14:59:41.540 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.540 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.540 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:41.540 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.884975+0000", 2024-09-13T14:59:41.540 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2293000000000002e-05 2024-09-13T14:59:41.540 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:41.540 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:41.541 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:41.541 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.541 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.541 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:192155 9.16 9:6e97ab74:::benchmark_data_smithi102_348563_object71053:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-13T14:59:41.541 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:53:46.320921+0000", 2024-09-13T14:59:41.541 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 355.08034813400002, 2024-09-13T14:59:41.541 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.56416036800000002, 2024-09-13T14:59:41.541 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:41.541 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:41.541 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:41.541 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:59:41.541 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:59:41.541 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 192155 2024-09-13T14:59:41.542 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.542 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:41.542 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.542 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:41.542 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.320921+0000", 2024-09-13T14:59:41.542 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:41.542 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.542 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.542 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:41.542 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.320921+0000", 2024-09-13T14:59:41.542 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:41.542 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.542 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.542 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:41.543 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.320911+0000", 2024-09-13T14:59:41.543 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999909 2024-09-13T14:59:41.543 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.543 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.543 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:41.543 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.320927+0000", 2024-09-13T14:59:41.543 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5787999999999999e-05 2024-09-13T14:59:41.543 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.543 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.543 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:41.543 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.320945+0000", 2024-09-13T14:59:41.543 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7314000000000001e-05 2024-09-13T14:59:41.543 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.543 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.544 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:41.544 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.320951+0000", 2024-09-13T14:59:41.544 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3799999999999999e-06 2024-09-13T14:59:41.544 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.544 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.544 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:41.544 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.321013+0000", 2024-09-13T14:59:41.544 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2081000000000004e-05 2024-09-13T14:59:41.544 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.544 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.544 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:41.544 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.321025+0000", 2024-09-13T14:59:41.544 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.205e-05 2024-09-13T14:59:41.545 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.545 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.545 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:41.545 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.860003+0000", 2024-09-13T14:59:41.545 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.53897767100000005 2024-09-13T14:59:41.545 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.545 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.545 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:59:41.545 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.860100+0000", 2024-09-13T14:59:41.545 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.7715999999999998e-05 2024-09-13T14:59:41.545 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.545 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.545 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:59:41.546 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.860420+0000", 2024-09-13T14:59:41.546 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00031967399999999999 2024-09-13T14:59:41.546 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.546 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.546 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:59:41.546 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.865475+0000", 2024-09-13T14:59:41.546 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0050548559999999999 2024-09-13T14:59:41.546 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.546 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.546 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:41.546 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.884729+0000", 2024-09-13T14:59:41.546 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.019254367000000001 2024-09-13T14:59:41.546 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.546 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.547 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:41.547 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.885027+0000", 2024-09-13T14:59:41.547 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000297585 2024-09-13T14:59:41.547 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.547 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.547 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:59:41.547 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.885058+0000", 2024-09-13T14:59:41.547 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.1040000000000001e-05 2024-09-13T14:59:41.547 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.547 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.547 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:41.547 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.885081+0000", 2024-09-13T14:59:41.547 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.302e-05 2024-09-13T14:59:41.547 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:41.548 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:41.548 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:41.548 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.548 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.548 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:192159 9.16 9:6b90a653:::benchmark_data_smithi102_348563_object71057:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-13T14:59:41.548 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:53:46.337845+0000", 2024-09-13T14:59:41.548 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 355.06342351400002, 2024-09-13T14:59:41.548 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.54736063000000001, 2024-09-13T14:59:41.548 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:41.548 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:41.548 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:41.548 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:59:41.548 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:59:41.548 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 192159 2024-09-13T14:59:41.549 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.549 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:41.549 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.549 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:41.549 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.337845+0000", 2024-09-13T14:59:41.549 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:41.549 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.549 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.549 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:41.549 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.337845+0000", 2024-09-13T14:59:41.549 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:41.549 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.549 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.550 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:41.550 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.337844+0000", 2024-09-13T14:59:41.550 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:41.550 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.550 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.550 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:41.550 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.337853+0000", 2024-09-13T14:59:41.550 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8750000000000006e-06 2024-09-13T14:59:41.550 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.550 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.550 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:41.550 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.337865+0000", 2024-09-13T14:59:41.550 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2527999999999999e-05 2024-09-13T14:59:41.550 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.550 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.551 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:41.551 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.337871+0000", 2024-09-13T14:59:41.551 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8819999999999998e-06 2024-09-13T14:59:41.551 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.551 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.551 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:41.551 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.337929+0000", 2024-09-13T14:59:41.551 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8037e-05 2024-09-13T14:59:41.551 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.551 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.551 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:41.551 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.337939+0000", 2024-09-13T14:59:41.551 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6870000000000006e-06 2024-09-13T14:59:41.551 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.551 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.552 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:41.552 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.862582+0000", 2024-09-13T14:59:41.552 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.52464357800000005 2024-09-13T14:59:41.552 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.552 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.552 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:59:41.552 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.862683+0000", 2024-09-13T14:59:41.552 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010106 2024-09-13T14:59:41.552 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.552 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.552 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:59:41.552 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.863080+0000", 2024-09-13T14:59:41.552 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00039613300000000001 2024-09-13T14:59:41.553 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.553 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.553 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:59:41.553 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.865567+0000", 2024-09-13T14:59:41.553 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0024869689999999999 2024-09-13T14:59:41.553 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.553 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.553 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:41.553 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.885115+0000", 2024-09-13T14:59:41.553 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.019548126999999998 2024-09-13T14:59:41.553 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.553 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.553 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:41.553 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.885146+0000", 2024-09-13T14:59:41.553 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.1612000000000002e-05 2024-09-13T14:59:41.554 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.554 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.554 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:59:41.554 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.885171+0000", 2024-09-13T14:59:41.554 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4944999999999999e-05 2024-09-13T14:59:41.554 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.554 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.554 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:41.554 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:46.885206+0000", 2024-09-13T14:59:41.554 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4669000000000003e-05 2024-09-13T14:59:41.554 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:41.554 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:41.554 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:41.554 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.554 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.555 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36528.0:196897 9.4 e419/381 9:253a0cb2:::benchmark_data_smithi102_348563_object75795:head v 419'7986, mlcod=419'7984)", 2024-09-13T14:59:41.555 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:53:52.508161+0000", 2024-09-13T14:59:41.555 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 348.89310731900002, 2024-09-13T14:59:41.555 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.12279667599999999, 2024-09-13T14:59:41.555 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:41.555 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:41.555 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:41.555 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.555 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:41.555 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:52.508161+0000", 2024-09-13T14:59:41.555 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:41.555 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.555 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.556 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:41.556 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:52.508161+0000", 2024-09-13T14:59:41.556 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:41.556 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.556 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.556 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:41.556 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:52.508160+0000", 2024-09-13T14:59:41.556 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:41.556 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.556 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.556 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:41.556 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:52.508172+0000", 2024-09-13T14:59:41.556 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2388000000000001e-05 2024-09-13T14:59:41.557 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.557 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.557 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:41.557 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:52.508188+0000", 2024-09-13T14:59:41.557 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.571e-05 2024-09-13T14:59:41.557 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.557 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.557 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:41.557 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:52.508196+0000", 2024-09-13T14:59:41.557 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1759999999999993e-06 2024-09-13T14:59:41.557 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.557 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.557 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:41.557 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:52.622741+0000", 2024-09-13T14:59:41.558 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.114544589 2024-09-13T14:59:41.558 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.558 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.558 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:59:41.558 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:52.622765+0000", 2024-09-13T14:59:41.558 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4426e-05 2024-09-13T14:59:41.558 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.558 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.558 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:59:41.558 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:52.630886+0000", 2024-09-13T14:59:41.558 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0081205289999999996 2024-09-13T14:59:41.558 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.558 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.558 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:41.559 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:52.630958+0000", 2024-09-13T14:59:41.559 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2286999999999994e-05 2024-09-13T14:59:41.559 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:41.559 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:41.559 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:41.559 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.559 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.559 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36528.0:196903 9.4 e419/381 9:24f241b5:::benchmark_data_smithi102_348563_object75801:head v 419'7987, mlcod=419'7984)", 2024-09-13T14:59:41.559 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:53:52.513007+0000", 2024-09-13T14:59:41.559 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 348.88826211499997, 2024-09-13T14:59:41.559 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.12359471700000001, 2024-09-13T14:59:41.559 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:41.559 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:41.560 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:41.560 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.560 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:41.560 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:52.513007+0000", 2024-09-13T14:59:41.560 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:41.560 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.560 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.560 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:41.560 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:52.513007+0000", 2024-09-13T14:59:41.560 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:41.560 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.560 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.560 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:41.561 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:52.513006+0000", 2024-09-13T14:59:41.561 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:59:41.561 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.561 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.561 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:41.561 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:52.513016+0000", 2024-09-13T14:59:41.561 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0614999999999999e-05 2024-09-13T14:59:41.561 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.561 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.561 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:41.561 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:52.622714+0000", 2024-09-13T14:59:41.561 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.10969787 2024-09-13T14:59:41.561 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.561 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.561 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:41.562 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:52.622739+0000", 2024-09-13T14:59:41.562 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5324e-05 2024-09-13T14:59:41.562 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.562 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.562 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:41.562 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:52.625711+0000", 2024-09-13T14:59:41.562 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0029719540000000002 2024-09-13T14:59:41.562 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.562 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.562 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:59:41.562 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:52.625764+0000", 2024-09-13T14:59:41.562 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2604e-05 2024-09-13T14:59:41.562 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.562 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.562 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:59:41.563 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:52.636557+0000", 2024-09-13T14:59:41.563 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010793285999999999 2024-09-13T14:59:41.563 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:41.563 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:41.563 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:41.563 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:52.636601+0000", 2024-09-13T14:59:41.563 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4215000000000003e-05 2024-09-13T14:59:41.563 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:41.563 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:41.563 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:41.563 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:41.563 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:41.563 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:59:41.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:41 smithi195 ceph-mon[104262]: pgmap v1674: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:41.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:41 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:59:41.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:41 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:59:41.824 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:41 smithi102 ceph-mon[115269]: pgmap v1674: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:41.824 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:41 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:59:41.825 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:41 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:59:41.825 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:41 smithi102 ceph-mon[110183]: pgmap v1674: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:41.825 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:41 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T14:59:41.825 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:41 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:59:41.845 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.1 reset_purged_snaps_last 2024-09-13T14:59:42.045 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:42.320 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T14:59:42.516 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:59:43.664 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:43 smithi102 ceph-mon[110183]: pgmap v1675: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:43.664 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:43 smithi102 ceph-mon[115269]: pgmap v1675: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:43.672 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:43 smithi195 ceph-mon[104262]: pgmap v1675: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:44.204 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:59:44.204 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-13T14:59:44.204 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 0 2024-09-13T14:59:44.204 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:59:44.879 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T14:59:45.076 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:59:45.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:45 smithi195 ceph-mon[104262]: pgmap v1676: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:45.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:45 smithi102 ceph-mon[110183]: pgmap v1676: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:45.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:45 smithi102 ceph-mon[115269]: pgmap v1676: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:46.919 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:59:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:59:46] "GET /metrics HTTP/1.1" 200 34147 "" "Prometheus/2.51.0" 2024-09-13T14:59:46.966 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:59:46.966 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-13T14:59:46.966 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-09-13T14:59:46.966 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-09-13T14:59:46.966 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:59:47.619 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:47 smithi195 ceph-mon[104262]: pgmap v1677: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:47.641 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T14:59:47.838 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:59:47.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:47 smithi102 ceph-mon[110183]: pgmap v1677: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:47.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:47 smithi102 ceph-mon[115269]: pgmap v1677: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:49.490 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:59:49.491 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-09-13T14:59:49.491 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-09-13T14:59:49.491 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-13T14:59:49.491 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.491 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:71809 9.12 9.d0c5fd32 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e388)", 2024-09-13T14:59:49.491 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:51:44.047745+0000", 2024-09-13T14:59:49.491 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 485.43964460799998, 2024-09-13T14:59:49.491 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0056764149999999, 2024-09-13T14:59:49.491 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:49.491 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-13T14:59:49.492 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:49.492 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:59:49.492 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:59:49.492 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 71809 2024-09-13T14:59:49.492 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.492 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:49.492 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.492 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:49.492 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.047745+0000", 2024-09-13T14:59:49.492 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:49.492 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.493 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.493 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:49.493 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.047745+0000", 2024-09-13T14:59:49.493 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:49.493 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.493 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.493 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:49.493 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.047744+0000", 2024-09-13T14:59:49.493 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:49.493 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.493 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.493 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:49.494 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.047778+0000", 2024-09-13T14:59:49.494 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4230000000000003e-05 2024-09-13T14:59:49.494 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.494 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.494 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:49.494 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.047791+0000", 2024-09-13T14:59:49.494 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2503e-05 2024-09-13T14:59:49.494 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.494 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.494 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:49.494 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.047798+0000", 2024-09-13T14:59:49.494 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5129999999999999e-06 2024-09-13T14:59:49.495 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.495 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.495 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:49.495 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.050041+0000", 2024-09-13T14:59:49.495 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0022429120000000001 2024-09-13T14:59:49.495 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.495 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.495 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:49.495 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.050049+0000", 2024-09-13T14:59:49.495 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5490000000000001e-06 2024-09-13T14:59:49.495 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.495 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.496 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:49.496 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:45.053408+0000", 2024-09-13T14:59:49.496 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0033590590000001 2024-09-13T14:59:49.496 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.496 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.496 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:49.496 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:45.053422+0000", 2024-09-13T14:59:49.496 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3913999999999999e-05 2024-09-13T14:59:49.496 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:49.496 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:49.496 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:49.496 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.497 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.497 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:71872 9.12 9.858a9f32 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-13T14:59:49.497 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:51:44.092200+0000", 2024-09-13T14:59:49.497 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 485.39518987500003, 2024-09-13T14:59:49.497 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.96131694700000003, 2024-09-13T14:59:49.497 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:49.497 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-13T14:59:49.497 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:49.497 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:59:49.497 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:59:49.497 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 71872 2024-09-13T14:59:49.497 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.497 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:49.498 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.498 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:49.498 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.092200+0000", 2024-09-13T14:59:49.498 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:49.498 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.498 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.498 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:49.498 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.092200+0000", 2024-09-13T14:59:49.498 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:49.498 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.498 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.498 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:49.498 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.092199+0000", 2024-09-13T14:59:49.499 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:49.499 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.499 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.499 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:49.499 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.092225+0000", 2024-09-13T14:59:49.499 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.605e-05 2024-09-13T14:59:49.499 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.499 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.499 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:49.499 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.092238+0000", 2024-09-13T14:59:49.499 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.375e-05 2024-09-13T14:59:49.499 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.499 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.500 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:49.500 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.092245+0000", 2024-09-13T14:59:49.500 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.781e-06 2024-09-13T14:59:49.500 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.500 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.500 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:49.500 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.092278+0000", 2024-09-13T14:59:49.500 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2747999999999999e-05 2024-09-13T14:59:49.500 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.500 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.500 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:49.500 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.092283+0000", 2024-09-13T14:59:49.501 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2560000000000002e-06 2024-09-13T14:59:49.501 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.501 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.501 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:49.501 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:45.053512+0000", 2024-09-13T14:59:49.501 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.96122935600000003 2024-09-13T14:59:49.501 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.501 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.501 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:49.501 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:45.053517+0000", 2024-09-13T14:59:49.501 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3200000000000001e-06 2024-09-13T14:59:49.502 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:49.502 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:49.502 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:49.502 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.502 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.502 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:71878 9.12 9.14765152 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-13T14:59:49.502 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:51:44.096503+0000", 2024-09-13T14:59:49.502 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 485.39088642799999, 2024-09-13T14:59:49.502 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.95702752599999996, 2024-09-13T14:59:49.502 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:49.502 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-13T14:59:49.502 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:49.502 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:59:49.502 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:59:49.503 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 71878 2024-09-13T14:59:49.503 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.503 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:49.503 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.503 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:49.503 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.096503+0000", 2024-09-13T14:59:49.503 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:49.503 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.503 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.503 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:49.503 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.096503+0000", 2024-09-13T14:59:49.503 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:49.503 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.504 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.504 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:49.504 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.096502+0000", 2024-09-13T14:59:49.504 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:59:49.504 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.504 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.504 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:49.504 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.096542+0000", 2024-09-13T14:59:49.504 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9483999999999997e-05 2024-09-13T14:59:49.504 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.504 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.504 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:49.504 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.096552+0000", 2024-09-13T14:59:49.505 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0515e-05 2024-09-13T14:59:49.505 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.505 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.505 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:49.505 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.096559+0000", 2024-09-13T14:59:49.505 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6359999999999997e-06 2024-09-13T14:59:49.505 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.505 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.505 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:49.505 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.096597+0000", 2024-09-13T14:59:49.505 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7833999999999997e-05 2024-09-13T14:59:49.505 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.506 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.506 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:49.506 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.096611+0000", 2024-09-13T14:59:49.506 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4005999999999999e-05 2024-09-13T14:59:49.506 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.506 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.506 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:49.506 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:45.053527+0000", 2024-09-13T14:59:49.506 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.95691620399999999 2024-09-13T14:59:49.506 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.506 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.506 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:49.506 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:45.053531+0000", 2024-09-13T14:59:49.507 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8060000000000001e-06 2024-09-13T14:59:49.507 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:49.507 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:49.507 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:49.507 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.507 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.507 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:71890 9.12 9.5bf03d92 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-13T14:59:49.507 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:51:44.102937+0000", 2024-09-13T14:59:49.507 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 485.38445272899997, 2024-09-13T14:59:49.507 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.95062656099999998, 2024-09-13T14:59:49.507 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:49.507 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-13T14:59:49.507 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:49.508 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:59:49.508 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:59:49.508 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 71890 2024-09-13T14:59:49.508 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.508 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:49.508 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.508 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:49.508 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.102937+0000", 2024-09-13T14:59:49.508 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:49.508 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.509 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.509 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:49.509 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.102937+0000", 2024-09-13T14:59:49.509 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:49.509 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.509 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.509 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:49.509 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.102936+0000", 2024-09-13T14:59:49.509 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:59:49.509 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.509 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.509 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:49.510 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.102989+0000", 2024-09-13T14:59:49.510 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2747000000000002e-05 2024-09-13T14:59:49.510 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.510 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.510 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:49.510 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.102997+0000", 2024-09-13T14:59:49.510 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2689999999999999e-06 2024-09-13T14:59:49.510 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.510 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.510 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:49.510 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.103002+0000", 2024-09-13T14:59:49.511 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8690000000000003e-06 2024-09-13T14:59:49.511 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.511 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.511 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:49.511 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.103044+0000", 2024-09-13T14:59:49.511 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2166000000000003e-05 2024-09-13T14:59:49.511 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.511 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.511 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:49.511 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.103051+0000", 2024-09-13T14:59:49.511 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.0060000000000003e-06 2024-09-13T14:59:49.512 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.512 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.512 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:49.512 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:45.053552+0000", 2024-09-13T14:59:49.512 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.95050080199999998 2024-09-13T14:59:49.512 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.512 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.512 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:49.512 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:45.053564+0000", 2024-09-13T14:59:49.512 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1391e-05 2024-09-13T14:59:49.512 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:49.512 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:49.513 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:49.513 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.513 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.513 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:71922 9.12 9.3bdb7492 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-13T14:59:49.513 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:51:44.122523+0000", 2024-09-13T14:59:49.513 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 485.36486720900001, 2024-09-13T14:59:49.513 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.93105367699999997, 2024-09-13T14:59:49.513 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:49.513 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-13T14:59:49.513 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:49.513 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:59:49.513 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:59:49.513 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 71922 2024-09-13T14:59:49.513 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.514 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:49.514 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.514 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:49.514 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.122523+0000", 2024-09-13T14:59:49.514 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:49.514 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.514 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.514 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:49.514 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.122523+0000", 2024-09-13T14:59:49.514 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:49.514 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.514 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.514 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:49.515 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.122522+0000", 2024-09-13T14:59:49.515 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:59:49.515 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.515 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.515 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:49.515 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.122551+0000", 2024-09-13T14:59:49.515 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9059000000000001e-05 2024-09-13T14:59:49.515 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.515 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.515 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:49.515 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.122562+0000", 2024-09-13T14:59:49.515 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1531e-05 2024-09-13T14:59:49.515 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.515 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.516 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:49.516 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.122569+0000", 2024-09-13T14:59:49.516 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9829999999999999e-06 2024-09-13T14:59:49.516 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.516 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.516 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:49.516 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.122614+0000", 2024-09-13T14:59:49.516 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5006000000000002e-05 2024-09-13T14:59:49.517 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.517 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.517 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:49.517 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.122622+0000", 2024-09-13T14:59:49.517 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4000000000000003e-06 2024-09-13T14:59:49.517 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.517 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.517 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:49.517 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:45.053571+0000", 2024-09-13T14:59:49.517 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.93094989299999997 2024-09-13T14:59:49.517 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.517 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.518 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:49.518 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:45.053576+0000", 2024-09-13T14:59:49.518 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7280000000000003e-06 2024-09-13T14:59:49.518 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:49.518 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:49.518 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:49.518 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.518 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.518 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:71953 9.12 9.ec3c91b2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-13T14:59:49.518 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:51:44.145473+0000", 2024-09-13T14:59:49.518 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 485.34191658100002, 2024-09-13T14:59:49.518 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.90813313799999995, 2024-09-13T14:59:49.519 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:49.519 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-13T14:59:49.519 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:49.519 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:59:49.519 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:59:49.519 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 71953 2024-09-13T14:59:49.519 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.519 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:49.519 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.519 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:49.519 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.145473+0000", 2024-09-13T14:59:49.519 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:49.519 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.520 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.520 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:49.520 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.145473+0000", 2024-09-13T14:59:49.520 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:49.520 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.520 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.520 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:49.520 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.145472+0000", 2024-09-13T14:59:49.520 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:59:49.520 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.520 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.520 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:49.520 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.145565+0000", 2024-09-13T14:59:49.521 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.2573999999999998e-05 2024-09-13T14:59:49.521 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.521 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.521 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:49.521 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.145582+0000", 2024-09-13T14:59:49.521 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6988999999999999e-05 2024-09-13T14:59:49.521 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.521 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.521 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:49.521 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.145588+0000", 2024-09-13T14:59:49.521 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8630000000000002e-06 2024-09-13T14:59:49.521 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.521 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.522 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:49.522 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.147950+0000", 2024-09-13T14:59:49.522 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002362032 2024-09-13T14:59:49.522 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.522 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.522 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:49.522 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.147957+0000", 2024-09-13T14:59:49.522 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.319e-06 2024-09-13T14:59:49.522 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.522 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.522 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:49.522 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:45.053598+0000", 2024-09-13T14:59:49.522 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.90564095600000005 2024-09-13T14:59:49.522 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.523 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.523 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:49.523 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:45.053606+0000", 2024-09-13T14:59:49.523 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5939999999999994e-06 2024-09-13T14:59:49.523 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:49.523 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:49.524 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:49.524 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.524 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.524 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:71961 9.12 9.70e3a1b2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-13T14:59:49.524 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:51:44.149647+0000", 2024-09-13T14:59:49.524 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 485.33774235200002, 2024-09-13T14:59:49.524 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.90397272299999998, 2024-09-13T14:59:49.524 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:49.524 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-13T14:59:49.524 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:49.524 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:59:49.524 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:59:49.525 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 71961 2024-09-13T14:59:49.525 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.525 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:49.525 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.525 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:49.525 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.149647+0000", 2024-09-13T14:59:49.525 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:49.525 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.525 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.525 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:49.525 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.149647+0000", 2024-09-13T14:59:49.525 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:49.526 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.526 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.526 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:49.526 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.149646+0000", 2024-09-13T14:59:49.526 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:49.526 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.526 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.526 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:49.526 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.149681+0000", 2024-09-13T14:59:49.526 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5284000000000003e-05 2024-09-13T14:59:49.526 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.526 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.526 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:49.527 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.149699+0000", 2024-09-13T14:59:49.527 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7442e-05 2024-09-13T14:59:49.527 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.527 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.527 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:49.527 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.149707+0000", 2024-09-13T14:59:49.527 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0960000000000001e-06 2024-09-13T14:59:49.527 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.527 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.527 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:49.527 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.151001+0000", 2024-09-13T14:59:49.527 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0012942609999999999 2024-09-13T14:59:49.527 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.528 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.528 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:49.528 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.151006+0000", 2024-09-13T14:59:49.528 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7999999999999998e-06 2024-09-13T14:59:49.528 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.528 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.528 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:49.528 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:45.053616+0000", 2024-09-13T14:59:49.528 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.90261008600000003 2024-09-13T14:59:49.528 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.528 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.528 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:49.529 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:45.053620+0000", 2024-09-13T14:59:49.529 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1080000000000003e-06 2024-09-13T14:59:49.529 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:49.529 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:49.529 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:49.529 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.529 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.529 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:72018 9.12 9.c68dcc52 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-13T14:59:49.529 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:51:44.190569+0000", 2024-09-13T14:59:49.529 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 485.29682039099998, 2024-09-13T14:59:49.529 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.863072167, 2024-09-13T14:59:49.529 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:49.530 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-13T14:59:49.530 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:49.530 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:59:49.530 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:59:49.530 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 72018 2024-09-13T14:59:49.530 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.530 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:49.530 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.530 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:49.530 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.190569+0000", 2024-09-13T14:59:49.530 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:49.530 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.530 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.531 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:49.531 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.190569+0000", 2024-09-13T14:59:49.531 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:49.531 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.531 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.531 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:49.531 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.190568+0000", 2024-09-13T14:59:49.531 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:49.531 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.531 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.531 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:49.531 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.190632+0000", 2024-09-13T14:59:49.531 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4212000000000001e-05 2024-09-13T14:59:49.531 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.532 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.532 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:49.532 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.190646+0000", 2024-09-13T14:59:49.532 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3796e-05 2024-09-13T14:59:49.532 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.532 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.532 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:49.532 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.190653+0000", 2024-09-13T14:59:49.532 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5699999999999998e-06 2024-09-13T14:59:49.532 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.532 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.532 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:49.532 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.190704+0000", 2024-09-13T14:59:49.533 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1348999999999999e-05 2024-09-13T14:59:49.533 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.533 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.533 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:49.533 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.190711+0000", 2024-09-13T14:59:49.533 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7120000000000003e-06 2024-09-13T14:59:49.533 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.533 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.533 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:49.533 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:45.053634+0000", 2024-09-13T14:59:49.533 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.86292300799999999 2024-09-13T14:59:49.533 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.533 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.534 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:49.534 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:45.053642+0000", 2024-09-13T14:59:49.534 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8769999999999993e-06 2024-09-13T14:59:49.534 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:49.534 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:49.534 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:49.534 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.534 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.534 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:72059 9.12 9.fcc48e32 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-13T14:59:49.534 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:51:44.222089+0000", 2024-09-13T14:59:49.534 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 485.26530090300002, 2024-09-13T14:59:49.534 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.83157225300000004, 2024-09-13T14:59:49.534 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:49.534 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-13T14:59:49.535 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:49.535 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:59:49.535 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:59:49.535 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 72059 2024-09-13T14:59:49.535 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.535 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:49.535 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.535 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:49.535 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.222089+0000", 2024-09-13T14:59:49.535 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:49.535 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.535 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.535 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:49.536 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.222089+0000", 2024-09-13T14:59:49.536 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:49.536 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.536 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.536 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:49.536 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.222088+0000", 2024-09-13T14:59:49.536 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:59:49.537 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.537 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.537 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:49.537 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.222204+0000", 2024-09-13T14:59:49.537 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000116346 2024-09-13T14:59:49.537 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.537 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.537 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:49.537 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.222216+0000", 2024-09-13T14:59:49.537 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1421e-05 2024-09-13T14:59:49.537 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.538 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.538 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:49.538 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.222223+0000", 2024-09-13T14:59:49.538 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4699999999999996e-06 2024-09-13T14:59:49.538 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.538 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.538 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:49.538 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.222270+0000", 2024-09-13T14:59:49.538 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6718000000000002e-05 2024-09-13T14:59:49.538 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.538 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.538 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:49.538 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.222278+0000", 2024-09-13T14:59:49.538 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.405e-06 2024-09-13T14:59:49.539 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.539 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.539 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:49.539 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:45.053651+0000", 2024-09-13T14:59:49.539 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.83137310499999995 2024-09-13T14:59:49.539 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.539 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.539 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:49 smithi195 ceph-mon[104262]: pgmap v1678: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:49.540 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:49.540 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:45.053661+0000", 2024-09-13T14:59:49.541 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.8689999999999999e-06 2024-09-13T14:59:49.541 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:49.541 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:49.541 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:49.541 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.541 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.541 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:72094 9.12 9.55d31a32 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-13T14:59:49.541 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:51:44.248433+0000", 2024-09-13T14:59:49.541 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 485.23895667099998, 2024-09-13T14:59:49.541 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.80525193299999998, 2024-09-13T14:59:49.541 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:49.541 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-13T14:59:49.541 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:49.542 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:59:49.542 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:59:49.542 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 72094 2024-09-13T14:59:49.542 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.542 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:49.542 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.542 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:49.542 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.248433+0000", 2024-09-13T14:59:49.542 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:49.542 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.542 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.542 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:49.542 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.248433+0000", 2024-09-13T14:59:49.543 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:49.543 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.543 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.543 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:49.543 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.248432+0000", 2024-09-13T14:59:49.543 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:49.543 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.543 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.543 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:49.543 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.248468+0000", 2024-09-13T14:59:49.543 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6715999999999999e-05 2024-09-13T14:59:49.543 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.543 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.544 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:49.544 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.248481+0000", 2024-09-13T14:59:49.544 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2531000000000001e-05 2024-09-13T14:59:49.544 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.544 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.544 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:49.544 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.248488+0000", 2024-09-13T14:59:49.544 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4379999999999998e-06 2024-09-13T14:59:49.544 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.544 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.544 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:49.544 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.248533+0000", 2024-09-13T14:59:49.544 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5158999999999999e-05 2024-09-13T14:59:49.545 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.545 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.545 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:49.545 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.248541+0000", 2024-09-13T14:59:49.545 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6550000000000004e-06 2024-09-13T14:59:49.545 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.545 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.545 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:49.545 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:45.053677+0000", 2024-09-13T14:59:49.545 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.805135991 2024-09-13T14:59:49.546 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.546 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.546 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:49.546 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:45.053685+0000", 2024-09-13T14:59:49.546 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9610000000000005e-06 2024-09-13T14:59:49.546 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:49.546 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:49.546 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:49.546 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.547 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.547 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:72114 9.12 9.bc8a4312 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-13T14:59:49.547 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:51:44.261553+0000", 2024-09-13T14:59:49.547 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 485.22583724700002, 2024-09-13T14:59:49.547 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.79214617899999995, 2024-09-13T14:59:49.547 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:49.547 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-13T14:59:49.547 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:49.547 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:59:49.548 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:59:49.548 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 72114 2024-09-13T14:59:49.548 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.548 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:49.548 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.549 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:49.549 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.261553+0000", 2024-09-13T14:59:49.549 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:49.549 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.549 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.549 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:49.549 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.261553+0000", 2024-09-13T14:59:49.549 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:49.550 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.550 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.550 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:49.550 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.261551+0000", 2024-09-13T14:59:49.550 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:49.550 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.550 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.550 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:49.550 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.261594+0000", 2024-09-13T14:59:49.550 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2469999999999998e-05 2024-09-13T14:59:49.550 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.551 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.551 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:49.551 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.261608+0000", 2024-09-13T14:59:49.551 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4551e-05 2024-09-13T14:59:49.551 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.551 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.552 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:49.552 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.261615+0000", 2024-09-13T14:59:49.552 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7170000000000002e-06 2024-09-13T14:59:49.552 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.552 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.552 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:49.552 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.261659+0000", 2024-09-13T14:59:49.552 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3937000000000003e-05 2024-09-13T14:59:49.552 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.552 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.552 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:49.552 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.261666+0000", 2024-09-13T14:59:49.553 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1330000000000001e-06 2024-09-13T14:59:49.553 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.553 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.553 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:49.553 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:45.053694+0000", 2024-09-13T14:59:49.553 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.79202844900000002 2024-09-13T14:59:49.553 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.553 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.553 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:49.553 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:45.053699+0000", 2024-09-13T14:59:49.553 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1799999999999998e-06 2024-09-13T14:59:49.554 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:49.554 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:49.554 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:49.554 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.554 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.554 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:72115 9.12 9.dce30712 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-13T14:59:49.554 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:51:44.261851+0000", 2024-09-13T14:59:49.554 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 485.22553903900001, 2024-09-13T14:59:49.554 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.791868566, 2024-09-13T14:59:49.554 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:49.554 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-13T14:59:49.554 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:49.555 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:59:49.555 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:59:49.555 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 72115 2024-09-13T14:59:49.555 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.555 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:49.555 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.555 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:49.555 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.261851+0000", 2024-09-13T14:59:49.555 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:49.555 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.556 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.556 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:49.556 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.261851+0000", 2024-09-13T14:59:49.556 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:49.556 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.556 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.556 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:49.556 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.261850+0000", 2024-09-13T14:59:49.556 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:59:49.556 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.556 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.556 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:49.557 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.261911+0000", 2024-09-13T14:59:49.557 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1268999999999999e-05 2024-09-13T14:59:49.557 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.557 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.557 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:49.557 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.261928+0000", 2024-09-13T14:59:49.557 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6424000000000001e-05 2024-09-13T14:59:49.557 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.558 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.558 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:49.558 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.261933+0000", 2024-09-13T14:59:49.558 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8400000000000002e-06 2024-09-13T14:59:49.558 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.558 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.558 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:49.558 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.261961+0000", 2024-09-13T14:59:49.558 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8187999999999998e-05 2024-09-13T14:59:49.558 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.558 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.558 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:49.558 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:44.261965+0000", 2024-09-13T14:59:49.559 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.019e-06 2024-09-13T14:59:49.559 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.559 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.559 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:49.559 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:45.053711+0000", 2024-09-13T14:59:49.559 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.79174673299999998 2024-09-13T14:59:49.559 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.559 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.559 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:49.559 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:45.053719+0000", 2024-09-13T14:59:49.559 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8059999999999995e-06 2024-09-13T14:59:49.559 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:49.560 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:49.560 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:49.560 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.560 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.560 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:131560 9.12 9:49960850:::benchmark_data_smithi102_348563_object10458:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e407)", 2024-09-13T14:59:49.560 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:52:46.885464+0000", 2024-09-13T14:59:49.560 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 422.60192538299998, 2024-09-13T14:59:49.560 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0001314290000001, 2024-09-13T14:59:49.560 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:49.560 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-13T14:59:49.560 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:49.560 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:59:49.560 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:59:49.561 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 131560 2024-09-13T14:59:49.561 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.561 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:49.561 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.561 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:49.561 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:46.885464+0000", 2024-09-13T14:59:49.561 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:49.561 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.561 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.561 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:49.561 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:46.885464+0000", 2024-09-13T14:59:49.561 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:49.561 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.562 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.562 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:49.562 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:46.885464+0000", 2024-09-13T14:59:49.562 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:59:49.562 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.562 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.562 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:49.562 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:46.885468+0000", 2024-09-13T14:59:49.562 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1620000000000001e-06 2024-09-13T14:59:49.562 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.562 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.562 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:49.562 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:46.885477+0000", 2024-09-13T14:59:49.562 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7250000000000003e-06 2024-09-13T14:59:49.563 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.563 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.563 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:49.563 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:46.885483+0000", 2024-09-13T14:59:49.563 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3500000000000002e-06 2024-09-13T14:59:49.563 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.563 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.563 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:49.563 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:46.885855+0000", 2024-09-13T14:59:49.563 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00037174199999999999 2024-09-13T14:59:49.563 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.563 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.563 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:49.564 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:46.885864+0000", 2024-09-13T14:59:49.564 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1540000000000008e-06 2024-09-13T14:59:49.564 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.564 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.564 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:49.564 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.885513+0000", 2024-09-13T14:59:49.564 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.99964860200000005 2024-09-13T14:59:49.564 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.564 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.564 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:49.564 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.885596+0000", 2024-09-13T14:59:49.564 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3250000000000004e-05 2024-09-13T14:59:49.565 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:49.565 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:49.565 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:49.565 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.565 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.565 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:131612 9.12 9:49279ff5:::benchmark_data_smithi102_348563_object10510:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e407)", 2024-09-13T14:59:49.565 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:52:46.923109+0000", 2024-09-13T14:59:49.565 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 422.56428046899998, 2024-09-13T14:59:49.565 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.97013644499999996, 2024-09-13T14:59:49.565 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:49.565 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:49.565 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:49.565 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:59:49.566 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:59:49.566 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 131612 2024-09-13T14:59:49.566 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.566 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:49.566 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.566 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:49.566 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:46.923109+0000", 2024-09-13T14:59:49.566 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:49.566 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.566 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.566 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:49.566 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:46.923109+0000", 2024-09-13T14:59:49.566 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:49.566 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.567 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.567 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:49.567 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:46.923108+0000", 2024-09-13T14:59:49.567 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:59:49.567 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.567 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.567 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:49.567 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:46.923113+0000", 2024-09-13T14:59:49.567 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0080000000000002e-06 2024-09-13T14:59:49.567 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.567 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.568 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:49.568 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:46.923124+0000", 2024-09-13T14:59:49.568 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0784999999999999e-05 2024-09-13T14:59:49.568 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.568 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.568 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:49.568 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:46.923131+0000", 2024-09-13T14:59:49.568 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.007e-06 2024-09-13T14:59:49.568 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.568 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.569 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:49.569 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:46.923225+0000", 2024-09-13T14:59:49.569 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.3826999999999995e-05 2024-09-13T14:59:49.569 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.569 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.569 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:49.569 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:46.923235+0000", 2024-09-13T14:59:49.569 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.9799999999999993e-06 2024-09-13T14:59:49.569 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.569 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.569 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:49.569 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.886008+0000", 2024-09-13T14:59:49.569 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.96277351300000003 2024-09-13T14:59:49.570 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.570 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.570 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:59:49.570 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.886129+0000", 2024-09-13T14:59:49.570 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000120246 2024-09-13T14:59:49.570 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.570 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.570 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:59:49.570 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.886252+0000", 2024-09-13T14:59:49.570 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000122948 2024-09-13T14:59:49.570 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.570 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.570 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:49.571 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.893105+0000", 2024-09-13T14:59:49.571 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0068538449999999999 2024-09-13T14:59:49.571 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.571 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.571 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:49.571 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.893149+0000", 2024-09-13T14:59:49.571 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4122999999999998e-05 2024-09-13T14:59:49.571 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.571 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.571 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:59:49.571 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.893161+0000", 2024-09-13T14:59:49.571 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1209999999999999e-05 2024-09-13T14:59:49.571 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.571 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.572 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:59:49.572 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.893209+0000", 2024-09-13T14:59:49.572 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8123e-05 2024-09-13T14:59:49.572 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.572 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.572 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:49.572 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.893246+0000", 2024-09-13T14:59:49.572 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6900999999999998e-05 2024-09-13T14:59:49.572 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:49.572 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:49.572 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:49.572 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.572 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.573 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:131615 9.12 9:4dbff602:::benchmark_data_smithi102_348563_object10513:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e407)", 2024-09-13T14:59:49.573 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:52:46.926274+0000", 2024-09-13T14:59:49.573 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 422.56111580499999, 2024-09-13T14:59:49.573 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.96716006899999996, 2024-09-13T14:59:49.573 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:49.573 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:49.573 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:49.573 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:59:49.573 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:59:49.573 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 131615 2024-09-13T14:59:49.573 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.573 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:49.573 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.573 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:49.574 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:46.926274+0000", 2024-09-13T14:59:49.574 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:49.574 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.574 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.574 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:49.574 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:46.926274+0000", 2024-09-13T14:59:49.574 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:49.574 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.574 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.574 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:49.574 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:46.926273+0000", 2024-09-13T14:59:49.574 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:49.575 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.575 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.575 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:49.575 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:46.926279+0000", 2024-09-13T14:59:49.575 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.579e-06 2024-09-13T14:59:49.575 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.575 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.575 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:49.575 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:46.926293+0000", 2024-09-13T14:59:49.575 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3558e-05 2024-09-13T14:59:49.575 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.575 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.575 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:49.576 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:46.926299+0000", 2024-09-13T14:59:49.576 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7139999999999996e-06 2024-09-13T14:59:49.576 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.576 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.576 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:49.576 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:46.926362+0000", 2024-09-13T14:59:49.576 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3009999999999995e-05 2024-09-13T14:59:49.576 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.576 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.576 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:49.576 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:46.926375+0000", 2024-09-13T14:59:49.576 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2796000000000001e-05 2024-09-13T14:59:49.576 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.576 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.577 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:49.577 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.888447+0000", 2024-09-13T14:59:49.577 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.96207200599999998 2024-09-13T14:59:49.577 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.577 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.577 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:59:49.577 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.888565+0000", 2024-09-13T14:59:49.577 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011762200000000001 2024-09-13T14:59:49.577 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.577 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.577 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:59:49.577 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.888680+0000", 2024-09-13T14:59:49.577 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000114695 2024-09-13T14:59:49.578 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.578 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.578 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:59:49.578 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.893255+0000", 2024-09-13T14:59:49.578 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0045749900000000001 2024-09-13T14:59:49.578 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.578 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.578 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:49.578 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.893315+0000", 2024-09-13T14:59:49.579 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0408000000000002e-05 2024-09-13T14:59:49.579 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.579 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.579 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:49.579 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.893359+0000", 2024-09-13T14:59:49.579 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4419999999999998e-05 2024-09-13T14:59:49.579 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.579 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.579 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:59:49.579 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.893399+0000", 2024-09-13T14:59:49.579 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9449999999999997e-05 2024-09-13T14:59:49.580 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.580 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.580 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:49.580 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.893434+0000", 2024-09-13T14:59:49.580 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5160000000000002e-05 2024-09-13T14:59:49.580 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:49.580 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:49.580 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:49.580 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.580 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.580 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:131666 9.12 9:4d72c1ae:::benchmark_data_smithi102_348563_object10564:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e407)", 2024-09-13T14:59:49.580 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:52:46.972133+0000", 2024-09-13T14:59:49.580 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 422.51525642199999, 2024-09-13T14:59:49.581 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.92147038299999995, 2024-09-13T14:59:49.581 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:49.581 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:49.581 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:49.581 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:59:49.581 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:59:49.581 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 131666 2024-09-13T14:59:49.581 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.581 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:49.581 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.581 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:49.581 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:46.972133+0000", 2024-09-13T14:59:49.581 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:49.582 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.582 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.582 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:49.582 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:46.972133+0000", 2024-09-13T14:59:49.582 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:49.582 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.582 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.582 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:49.582 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:46.972132+0000", 2024-09-13T14:59:49.582 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:49.582 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.582 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.582 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:49.582 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:46.972142+0000", 2024-09-13T14:59:49.583 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0105999999999999e-05 2024-09-13T14:59:49.583 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.583 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.583 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:49.583 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:46.972157+0000", 2024-09-13T14:59:49.583 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4577e-05 2024-09-13T14:59:49.583 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.583 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.583 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:49.583 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:46.972164+0000", 2024-09-13T14:59:49.583 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4250000000000001e-06 2024-09-13T14:59:49.583 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.583 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.583 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:49.584 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:46.972256+0000", 2024-09-13T14:59:49.584 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.2099e-05 2024-09-13T14:59:49.584 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.584 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.584 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:49.584 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:46.972268+0000", 2024-09-13T14:59:49.584 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2099999999999999e-05 2024-09-13T14:59:49.584 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.584 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.584 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:49.584 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.890951+0000", 2024-09-13T14:59:49.584 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.91868329299999996 2024-09-13T14:59:49.584 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.585 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.585 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:59:49.585 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.891098+0000", 2024-09-13T14:59:49.585 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014677399999999999 2024-09-13T14:59:49.585 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.585 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.585 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:59:49.585 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.891224+0000", 2024-09-13T14:59:49.585 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000125418 2024-09-13T14:59:49.585 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.585 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.585 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:59:49.585 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.893261+0000", 2024-09-13T14:59:49.586 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0020368740000000001 2024-09-13T14:59:49.586 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.586 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.586 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:49.586 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.893486+0000", 2024-09-13T14:59:49.586 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00022584999999999999 2024-09-13T14:59:49.586 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.586 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.586 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:49.587 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.893534+0000", 2024-09-13T14:59:49.587 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7904000000000001e-05 2024-09-13T14:59:49.587 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.587 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.587 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:59:49.587 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.893571+0000", 2024-09-13T14:59:49.587 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6745999999999997e-05 2024-09-13T14:59:49.587 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.587 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.587 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:49.587 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.893604+0000", 2024-09-13T14:59:49.587 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2654999999999998e-05 2024-09-13T14:59:49.587 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:49.588 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:49.588 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:49.588 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.588 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.588 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:131691 9.12 9:4fe6068c:::benchmark_data_smithi102_348563_object10589:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e407)", 2024-09-13T14:59:49.588 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:52:46.993586+0000", 2024-09-13T14:59:49.588 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 422.49380363799997, 2024-09-13T14:59:49.588 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.91107416200000002, 2024-09-13T14:59:49.588 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:49.588 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:49.588 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:49.588 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:59:49.588 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:59:49.589 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 131691 2024-09-13T14:59:49.589 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.589 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:49.589 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.589 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:49.589 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:46.993586+0000", 2024-09-13T14:59:49.589 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:49.589 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.589 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.590 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:49.590 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:46.993586+0000", 2024-09-13T14:59:49.590 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:49.590 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.590 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.590 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:49.590 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:46.993585+0000", 2024-09-13T14:59:49.590 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:59:49.590 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.590 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.590 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:49.590 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:46.993590+0000", 2024-09-13T14:59:49.590 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9370000000000003e-06 2024-09-13T14:59:49.590 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.591 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.591 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:49.591 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:46.993600+0000", 2024-09-13T14:59:49.591 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0171e-05 2024-09-13T14:59:49.591 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.591 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.591 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:49.591 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:46.993607+0000", 2024-09-13T14:59:49.591 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7390000000000002e-06 2024-09-13T14:59:49.591 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.591 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.592 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:49.592 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:46.993690+0000", 2024-09-13T14:59:49.592 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3232999999999994e-05 2024-09-13T14:59:49.592 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.592 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.592 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:49.592 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:46.993701+0000", 2024-09-13T14:59:49.592 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.064e-05 2024-09-13T14:59:49.592 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.592 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.592 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:49.592 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.893644+0000", 2024-09-13T14:59:49.592 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.89994333699999995 2024-09-13T14:59:49.593 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.593 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.593 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:59:49.593 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.893772+0000", 2024-09-13T14:59:49.593 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012715900000000001 2024-09-13T14:59:49.593 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.593 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.593 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:59:49.593 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.893896+0000", 2024-09-13T14:59:49.593 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012478399999999999 2024-09-13T14:59:49.593 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.593 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.593 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:49.593 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.898004+0000", 2024-09-13T14:59:49.594 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.004107622 2024-09-13T14:59:49.594 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.594 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.594 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:49.594 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.902484+0000", 2024-09-13T14:59:49.594 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.004480146 2024-09-13T14:59:49.594 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.594 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.594 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:59:49.594 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.904606+0000", 2024-09-13T14:59:49.594 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0021216730000000001 2024-09-13T14:59:49.594 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.594 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.595 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:59:49.595 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.904638+0000", 2024-09-13T14:59:49.595 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2335999999999997e-05 2024-09-13T14:59:49.595 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.595 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.595 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:49.595 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.904660+0000", 2024-09-13T14:59:49.595 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2172999999999999e-05 2024-09-13T14:59:49.595 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:49.595 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:49.595 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:49.595 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.595 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.595 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:131703 9.12 9:4c50bfa5:::benchmark_data_smithi102_348563_object10601:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e407)", 2024-09-13T14:59:49.596 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:52:47.007317+0000", 2024-09-13T14:59:49.596 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 422.48007277800002, 2024-09-13T14:59:49.596 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.89739671200000004, 2024-09-13T14:59:49.596 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:49.596 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:49.596 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:49.596 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:59:49.596 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:59:49.596 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 131703 2024-09-13T14:59:49.596 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.596 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:49.597 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.597 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:49.597 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.007317+0000", 2024-09-13T14:59:49.597 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:49.597 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.597 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.597 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:49.597 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.007317+0000", 2024-09-13T14:59:49.597 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:49.597 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.597 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.597 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:49.597 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.007316+0000", 2024-09-13T14:59:49.598 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:49.598 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.598 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.598 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:49.598 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.007322+0000", 2024-09-13T14:59:49.598 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3559999999999998e-06 2024-09-13T14:59:49.598 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.598 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.598 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:49.598 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.007332+0000", 2024-09-13T14:59:49.598 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0180000000000001e-05 2024-09-13T14:59:49.598 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.598 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.598 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:49.599 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.007340+0000", 2024-09-13T14:59:49.599 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5399999999999998e-06 2024-09-13T14:59:49.599 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.599 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.599 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:49.599 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.007412+0000", 2024-09-13T14:59:49.599 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2682999999999995e-05 2024-09-13T14:59:49.599 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.599 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.599 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:49.600 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.007421+0000", 2024-09-13T14:59:49.600 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2619999999999998e-06 2024-09-13T14:59:49.600 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.600 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.600 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:49.600 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.895781+0000", 2024-09-13T14:59:49.600 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.88836026199999996 2024-09-13T14:59:49.600 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.600 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.600 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:59:49.600 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.895896+0000", 2024-09-13T14:59:49.600 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000115466 2024-09-13T14:59:49.601 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.601 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.601 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:59:49.601 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.896018+0000", 2024-09-13T14:59:49.601 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000121612 2024-09-13T14:59:49.601 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.601 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.601 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:49.601 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.902414+0000", 2024-09-13T14:59:49.601 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0063956899999999999 2024-09-13T14:59:49.601 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.601 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.602 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:49.602 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.902531+0000", 2024-09-13T14:59:49.602 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000116846 2024-09-13T14:59:49.602 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.602 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.602 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:59:49.602 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.904668+0000", 2024-09-13T14:59:49.602 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0021372980000000001 2024-09-13T14:59:49.602 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.602 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.602 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:59:49.603 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.904693+0000", 2024-09-13T14:59:49.603 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5516000000000001e-05 2024-09-13T14:59:49.603 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.603 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.603 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:49.603 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.904714+0000", 2024-09-13T14:59:49.603 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0233000000000001e-05 2024-09-13T14:59:49.603 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:49.603 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:49.603 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:49.603 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.603 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.603 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:131756 9.12 9:48967c65:::benchmark_data_smithi102_348563_object10654:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e407)", 2024-09-13T14:59:49.604 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:52:47.071129+0000", 2024-09-13T14:59:49.604 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 422.41626027299998, 2024-09-13T14:59:49.604 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.83365577099999999, 2024-09-13T14:59:49.604 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:49.604 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:49.604 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:49.604 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:59:49.604 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:59:49.604 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 131756 2024-09-13T14:59:49.604 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.604 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:49.604 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.604 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:49.605 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.071129+0000", 2024-09-13T14:59:49.605 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:49.605 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.605 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.605 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:49.605 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.071129+0000", 2024-09-13T14:59:49.605 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:49.605 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.605 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.605 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:49.605 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.071127+0000", 2024-09-13T14:59:49.605 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:59:49.605 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.606 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.606 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:49.606 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.071135+0000", 2024-09-13T14:59:49.606 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5190000000000003e-06 2024-09-13T14:59:49.606 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.606 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.606 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:49.606 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.071149+0000", 2024-09-13T14:59:49.606 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3696000000000001e-05 2024-09-13T14:59:49.606 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.606 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.606 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:49.606 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.071155+0000", 2024-09-13T14:59:49.606 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.849e-06 2024-09-13T14:59:49.607 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.607 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.607 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:49.607 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.071210+0000", 2024-09-13T14:59:49.607 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4561999999999997e-05 2024-09-13T14:59:49.607 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.607 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.607 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:49.607 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.071226+0000", 2024-09-13T14:59:49.607 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5865000000000001e-05 2024-09-13T14:59:49.607 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.607 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.607 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:49.607 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.898036+0000", 2024-09-13T14:59:49.608 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.82681022599999998 2024-09-13T14:59:49.608 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.608 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.608 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:59:49.608 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.898162+0000", 2024-09-13T14:59:49.608 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000125797 2024-09-13T14:59:49.608 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.608 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.608 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:59:49.608 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.898275+0000", 2024-09-13T14:59:49.608 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000112787 2024-09-13T14:59:49.608 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.608 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.609 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:49.609 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.902449+0000", 2024-09-13T14:59:49.609 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0041747700000000004 2024-09-13T14:59:49.609 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.609 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.609 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:49.609 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.902562+0000", 2024-09-13T14:59:49.609 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000112453 2024-09-13T14:59:49.609 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.609 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.609 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:59:49.609 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.904723+0000", 2024-09-13T14:59:49.609 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0021608769999999998 2024-09-13T14:59:49.610 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.610 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.610 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:59:49.610 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.904757+0000", 2024-09-13T14:59:49.610 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.3782999999999998e-05 2024-09-13T14:59:49.610 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.610 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.610 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:49.611 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:47.904785+0000", 2024-09-13T14:59:49.611 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8654999999999999e-05 2024-09-13T14:59:49.611 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:49.611 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:49.611 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:49.611 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.611 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.611 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36738.0:9815 10.6 10:63ac9934:::benchmark_data_smithi102_377304_object9814:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e441)", 2024-09-13T14:59:49.611 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:55:16.935400+0000", 2024-09-13T14:59:49.611 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 272.55199014499999, 2024-09-13T14:59:49.611 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.036279811, 2024-09-13T14:59:49.611 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:49.611 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:49.611 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:49.612 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36738", 2024-09-13T14:59:49.612 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/2915890700", 2024-09-13T14:59:49.612 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 9815 2024-09-13T14:59:49.612 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.612 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:49.612 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.612 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:49.612 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.935400+0000", 2024-09-13T14:59:49.612 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:49.612 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.612 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.612 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:49.612 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.935400+0000", 2024-09-13T14:59:49.613 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:49.613 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.613 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.613 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:49.613 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.935399+0000", 2024-09-13T14:59:49.613 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:59:49.613 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.613 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.613 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:49.613 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.935457+0000", 2024-09-13T14:59:49.613 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8202999999999998e-05 2024-09-13T14:59:49.613 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.614 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.614 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:49.614 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.935465+0000", 2024-09-13T14:59:49.614 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1289999999999996e-06 2024-09-13T14:59:49.614 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.614 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.614 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:49.614 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.935469+0000", 2024-09-13T14:59:49.614 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1919999999999998e-06 2024-09-13T14:59:49.614 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.614 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.614 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:49.615 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.935512+0000", 2024-09-13T14:59:49.615 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2373999999999998e-05 2024-09-13T14:59:49.615 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.615 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.615 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:49.615 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.935519+0000", 2024-09-13T14:59:49.615 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8329999999999994e-06 2024-09-13T14:59:49.615 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.615 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.615 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:49.615 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:17.920405+0000", 2024-09-13T14:59:49.615 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.98488555499999997 2024-09-13T14:59:49.616 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.616 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.616 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:49.616 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:17.920420+0000", 2024-09-13T14:59:49.616 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4799e-05 2024-09-13T14:59:49.616 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.616 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.616 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:49.616 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:18.967493+0000", 2024-09-13T14:59:49.616 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.04707325 2024-09-13T14:59:49.617 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.617 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.617 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:59:49.617 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:18.967666+0000", 2024-09-13T14:59:49.617 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00017285099999999999 2024-09-13T14:59:49.617 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.617 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.617 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:59:49.617 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:18.967859+0000", 2024-09-13T14:59:49.617 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00019331099999999999 2024-09-13T14:59:49.617 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.617 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.617 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:49.618 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:18.970047+0000", 2024-09-13T14:59:49.618 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0021876320000000001 2024-09-13T14:59:49.618 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.618 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.618 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:59:49.618 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:18.970414+0000", 2024-09-13T14:59:49.618 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00036745700000000002 2024-09-13T14:59:49.618 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.618 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.618 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:49.618 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:18.971280+0000", 2024-09-13T14:59:49.618 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00086593 2024-09-13T14:59:49.618 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.619 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.619 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:59:49.619 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:18.971323+0000", 2024-09-13T14:59:49.619 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2929000000000002e-05 2024-09-13T14:59:49.619 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:49.619 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:49.619 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:49.619 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:18.971679+0000", 2024-09-13T14:59:49.619 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00035628899999999998 2024-09-13T14:59:49.619 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:49.619 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:49.619 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:49.619 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:49.620 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:49.620 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:59:49.620 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-13T14:59:49.620 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-09-13T14:59:49.620 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd in 4 2024-09-13T14:59:49.736 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:49 smithi102 ceph-mon[115269]: pgmap v1678: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:49.737 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:49 smithi102 ceph-mon[110183]: pgmap v1678: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:49.834 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:50.374 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T14:59:50.574 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:59:51.504 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:51 smithi102 ceph-mon[110183]: pgmap v1679: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:51.504 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:51 smithi102 ceph-mon[115269]: pgmap v1679: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:51.678 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:51 smithi195 ceph-mon[104262]: pgmap v1679: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:52.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:51 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-a[110159]: 2024-09-13T14:59:51.776+0000 7fa3e7f42640 -1 mon.a@0(leader).osd e494 definitely_dead 0 2024-09-13T14:59:52.234 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:59:52.235 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-13T14:59:52.235 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 0 2024-09-13T14:59:52.235 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:59:52.498 INFO:teuthology.orchestra.run.smithi102.stderr:marked in osd.4. 2024-09-13T14:59:52.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:52 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2934032678' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-13T14:59:52.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:52 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-13T14:59:52.833 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:52 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2934032678' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-13T14:59:52.833 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:52 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-13T14:59:52.833 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:52 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2934032678' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-13T14:59:52.833 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:52 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-13T14:59:52.914 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T14:59:53.112 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:59:53.235 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-09-13T14:59:53.236 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-13T14:59:53.439 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:53.500 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:53 smithi102 ceph-mon[110183]: pgmap v1680: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:53.500 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:53 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-13T14:59:53.500 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:53 smithi102 ceph-mon[110183]: osdmap e495: 8 total, 8 up, 8 in 2024-09-13T14:59:53.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:53 smithi195 ceph-mon[104262]: pgmap v1680: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:53.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:53 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-13T14:59:53.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:53 smithi195 ceph-mon[104262]: osdmap e495: 8 total, 8 up, 8 in 2024-09-13T14:59:53.863 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:53 smithi102 ceph-mon[115269]: pgmap v1680: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail 2024-09-13T14:59:53.863 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:53 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-13T14:59:53.863 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:53 smithi102 ceph-mon[115269]: osdmap e495: 8 total, 8 up, 8 in 2024-09-13T14:59:54.659 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:54 smithi102 ceph-mon[110183]: osdmap e496: 8 total, 8 up, 8 in 2024-09-13T14:59:54.659 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:54 smithi102 ceph-mon[115269]: osdmap e496: 8 total, 8 up, 8 in 2024-09-13T14:59:54.769 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:59:54.769 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-13T14:59:54.769 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-09-13T14:59:54.770 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-09-13T14:59:54.770 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:59:54.770 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:54 smithi195 ceph-mon[104262]: osdmap e496: 8 total, 8 up, 8 in 2024-09-13T14:59:55.196 INFO:teuthology.orchestra.run.smithi102.stdout:mon.a: {} 2024-09-13T14:59:55.196 INFO:teuthology.orchestra.run.smithi102.stdout:mon.b: {} 2024-09-13T14:59:55.197 INFO:teuthology.orchestra.run.smithi102.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T14:59:55.197 INFO:teuthology.orchestra.run.smithi102.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T14:59:55.198 INFO:teuthology.orchestra.run.smithi102.stdout:mon.c: {} 2024-09-13T14:59:55.198 INFO:teuthology.orchestra.run.smithi102.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T14:59:55.372 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T14:59:55.580 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:59:55.701 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:55 smithi195 ceph-mon[104262]: pgmap v1683: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-13T14:59:55.701 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:55 smithi195 ceph-mon[104262]: osdmap e497: 8 total, 8 up, 8 in 2024-09-13T14:59:55.701 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:55 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2994823955' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-13T14:59:55.701 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:55 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:59:55.701 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:55 smithi195 ceph-mon[104262]: osdmap e498: 8 total, 8 up, 8 in 2024-09-13T14:59:55.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:55 smithi102 ceph-mon[110183]: pgmap v1683: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-13T14:59:55.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:55 smithi102 ceph-mon[110183]: osdmap e497: 8 total, 8 up, 8 in 2024-09-13T14:59:55.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:55 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2994823955' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-13T14:59:55.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:55 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:59:55.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:55 smithi102 ceph-mon[110183]: osdmap e498: 8 total, 8 up, 8 in 2024-09-13T14:59:55.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:55 smithi102 ceph-mon[115269]: pgmap v1683: 103 pgs: 103 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-13T14:59:55.921 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:55 smithi102 ceph-mon[115269]: osdmap e497: 8 total, 8 up, 8 in 2024-09-13T14:59:55.921 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:55 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2994823955' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-13T14:59:55.921 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:55 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:59:55.921 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:55 smithi102 ceph-mon[115269]: osdmap e498: 8 total, 8 up, 8 in 2024-09-13T14:59:55.999 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-13T14:59:56.197 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:56.638 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 14:59:56 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:14:59:56] "GET /metrics HTTP/1.1" 200 34152 "" "Prometheus/2.51.0" 2024-09-13T14:59:57.258 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T14:59:57.258 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-09-13T14:59:57.258 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-09-13T14:59:57.258 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-13T14:59:57.259 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.259 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:72299 9.13 9:c8621b29:::benchmark_data_smithi102_348563_object72298:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-09-13T14:59:57.259 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:51:46.193942+0000", 2024-09-13T14:59:57.259 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 491.05983515899999, 2024-09-13T14:59:57.259 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1183224979999999, 2024-09-13T14:59:57.259 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:57.259 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:57.259 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:57.259 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:59:57.259 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:59:57.259 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 72299 2024-09-13T14:59:57.259 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.260 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:57.260 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.260 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:57.260 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.193942+0000", 2024-09-13T14:59:57.260 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:57.260 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.260 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.260 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:57.260 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.193942+0000", 2024-09-13T14:59:57.260 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:57.260 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.260 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.260 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:57.261 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.193940+0000", 2024-09-13T14:59:57.261 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:57.261 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.261 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.261 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:57.261 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.193990+0000", 2024-09-13T14:59:57.261 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9901999999999998e-05 2024-09-13T14:59:57.261 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.261 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.261 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:57.261 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.194003+0000", 2024-09-13T14:59:57.261 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3222e-05 2024-09-13T14:59:57.261 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.262 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.262 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:57.262 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.194009+0000", 2024-09-13T14:59:57.262 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7710000000000003e-06 2024-09-13T14:59:57.262 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.262 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.262 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.262 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.194063+0000", 2024-09-13T14:59:57.262 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3977999999999997e-05 2024-09-13T14:59:57.262 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.262 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.262 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:57.262 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.194076+0000", 2024-09-13T14:59:57.263 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3288000000000001e-05 2024-09-13T14:59:57.263 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.263 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.263 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.263 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.056600+0000", 2024-09-13T14:59:57.263 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.86252373199999999 2024-09-13T14:59:57.263 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.263 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.263 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:57.263 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.056614+0000", 2024-09-13T14:59:57.263 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3764999999999999e-05 2024-09-13T14:59:57.263 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.264 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.264 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.264 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.302885+0000", 2024-09-13T14:59:57.264 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.246271474 2024-09-13T14:59:57.264 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.264 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.264 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:59:57.264 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.303015+0000", 2024-09-13T14:59:57.264 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012967 2024-09-13T14:59:57.264 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.264 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.264 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:59:57.264 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.303117+0000", 2024-09-13T14:59:57.265 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010189800000000001 2024-09-13T14:59:57.265 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.265 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.265 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:59:57.265 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.307981+0000", 2024-09-13T14:59:57.265 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0048640360000000004 2024-09-13T14:59:57.265 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.265 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.265 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:57.265 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.308022+0000", 2024-09-13T14:59:57.265 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1356999999999997e-05 2024-09-13T14:59:57.265 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.265 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.266 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:57.266 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.310751+0000", 2024-09-13T14:59:57.266 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0027289879999999999 2024-09-13T14:59:57.266 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.266 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.266 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:59:57.266 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.310808+0000", 2024-09-13T14:59:57.266 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.6246000000000003e-05 2024-09-13T14:59:57.266 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.266 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.266 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:57.266 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.312264+0000", 2024-09-13T14:59:57.266 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001456511 2024-09-13T14:59:57.267 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:57.267 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:57.267 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:57.267 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.267 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.267 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:72416 9.13 9:ca62faaf:::benchmark_data_smithi102_348563_object72415:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-09-13T14:59:57.267 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:51:46.289815+0000", 2024-09-13T14:59:57.267 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 490.96396120200001, 2024-09-13T14:59:57.267 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0320571169999999, 2024-09-13T14:59:57.267 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:57.267 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:57.267 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:57.267 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:59:57.268 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:59:57.268 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 72416 2024-09-13T14:59:57.268 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.268 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:57.268 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.268 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:57.268 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.289815+0000", 2024-09-13T14:59:57.268 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:57.268 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.268 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.268 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:57.268 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.289815+0000", 2024-09-13T14:59:57.268 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:57.268 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.268 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.269 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:57.269 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.289809+0000", 2024-09-13T14:59:57.269 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999938 2024-09-13T14:59:57.269 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.269 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.269 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:57.269 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.289859+0000", 2024-09-13T14:59:57.269 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9295000000000002e-05 2024-09-13T14:59:57.269 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.269 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.269 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:57.269 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.289870+0000", 2024-09-13T14:59:57.269 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1575e-05 2024-09-13T14:59:57.270 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.270 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.270 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:57.270 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.289877+0000", 2024-09-13T14:59:57.270 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.0670000000000003e-06 2024-09-13T14:59:57.270 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.270 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.270 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.270 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.289939+0000", 2024-09-13T14:59:57.270 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1532999999999995e-05 2024-09-13T14:59:57.270 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.270 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.270 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:57.270 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.289956+0000", 2024-09-13T14:59:57.271 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7453999999999999e-05 2024-09-13T14:59:57.271 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.271 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.271 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.271 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.056739+0000", 2024-09-13T14:59:57.271 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.76678289799999999 2024-09-13T14:59:57.271 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.271 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.271 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:57.271 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.056751+0000", 2024-09-13T14:59:57.271 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1839e-05 2024-09-13T14:59:57.271 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.271 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.271 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.271 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.305540+0000", 2024-09-13T14:59:57.272 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2487891710000001 2024-09-13T14:59:57.272 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.272 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.272 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:59:57.272 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.305705+0000", 2024-09-13T14:59:57.272 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000164547 2024-09-13T14:59:57.272 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.272 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.272 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:59:57.272 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.305889+0000", 2024-09-13T14:59:57.272 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0001839 2024-09-13T14:59:57.272 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.273 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.273 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:59:57.273 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.308075+0000", 2024-09-13T14:59:57.273 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002185977 2024-09-13T14:59:57.273 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.274 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.274 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:57.274 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.312336+0000", 2024-09-13T14:59:57.274 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0042612129999999998 2024-09-13T14:59:57.274 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.274 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.274 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:57.274 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.320281+0000", 2024-09-13T14:59:57.274 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0079447949999999993 2024-09-13T14:59:57.274 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.274 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.274 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:59:57.274 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.320333+0000", 2024-09-13T14:59:57.274 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1956999999999997e-05 2024-09-13T14:59:57.275 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.275 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.275 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:57.275 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.321873+0000", 2024-09-13T14:59:57.275 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001540076 2024-09-13T14:59:57.275 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:57.275 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:57.275 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:57.275 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.275 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.275 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:72422 9.13 9:cc800bcd:::benchmark_data_smithi102_348563_object72421:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-09-13T14:59:57.275 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:51:46.291786+0000", 2024-09-13T14:59:57.275 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 490.96199027599999, 2024-09-13T14:59:57.275 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0314244129999999, 2024-09-13T14:59:57.275 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:57.275 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:57.276 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:57.276 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:59:57.276 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:59:57.276 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 72422 2024-09-13T14:59:57.276 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.276 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:57.276 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.276 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:57.276 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.291786+0000", 2024-09-13T14:59:57.276 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:57.276 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.276 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.276 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:57.276 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.291786+0000", 2024-09-13T14:59:57.277 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:57.277 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.277 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.277 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:57.277 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.291786+0000", 2024-09-13T14:59:57.277 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:59:57.277 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.277 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.277 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:57.277 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.291867+0000", 2024-09-13T14:59:57.277 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1588000000000006e-05 2024-09-13T14:59:57.278 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.278 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.278 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:57.278 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.291881+0000", 2024-09-13T14:59:57.278 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3638000000000001e-05 2024-09-13T14:59:57.278 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.278 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.278 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:57.278 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.291888+0000", 2024-09-13T14:59:57.278 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4610000000000002e-06 2024-09-13T14:59:57.278 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.278 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.279 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.279 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.291997+0000", 2024-09-13T14:59:57.279 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010846800000000001 2024-09-13T14:59:57.279 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.279 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.279 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:57.279 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.292005+0000", 2024-09-13T14:59:57.279 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9459999999999998e-06 2024-09-13T14:59:57.279 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.279 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.279 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.279 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.056783+0000", 2024-09-13T14:59:57.280 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.76477843000000001 2024-09-13T14:59:57.280 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.280 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.280 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:57.280 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.056792+0000", 2024-09-13T14:59:57.280 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8200000000000003e-06 2024-09-13T14:59:57.280 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.280 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.280 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.280 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.306918+0000", 2024-09-13T14:59:57.280 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2501263090000001 2024-09-13T14:59:57.280 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.280 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.281 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:59:57.281 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.307041+0000", 2024-09-13T14:59:57.281 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012237399999999999 2024-09-13T14:59:57.281 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.281 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.281 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:59:57.281 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.307146+0000", 2024-09-13T14:59:57.281 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000105288 2024-09-13T14:59:57.281 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.281 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.281 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:59:57.281 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.313885+0000", 2024-09-13T14:59:57.281 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0067393590000000003 2024-09-13T14:59:57.282 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.282 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.282 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:57.282 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.315634+0000", 2024-09-13T14:59:57.282 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001748318 2024-09-13T14:59:57.282 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.282 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.282 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:57.282 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.321938+0000", 2024-09-13T14:59:57.282 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0063039960000000001 2024-09-13T14:59:57.282 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.282 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.282 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:59:57.283 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.321991+0000", 2024-09-13T14:59:57.283 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3810000000000001e-05 2024-09-13T14:59:57.283 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.283 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.283 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:57.283 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.323211+0000", 2024-09-13T14:59:57.283 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0012194949999999999 2024-09-13T14:59:57.283 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:57.283 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:57.283 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:57.283 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.283 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.283 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:72445 9.13 9:c8139549:::benchmark_data_smithi102_348563_object72444:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-09-13T14:59:57.284 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:51:46.307279+0000", 2024-09-13T14:59:57.284 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 490.94649778100001, 2024-09-13T14:59:57.284 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0180606839999999, 2024-09-13T14:59:57.284 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:57.284 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:57.284 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:57.284 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:59:57.284 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:59:57.284 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 72445 2024-09-13T14:59:57.284 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.284 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:57.284 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.285 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:57.285 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.307279+0000", 2024-09-13T14:59:57.285 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:57.285 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.285 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.285 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:57.285 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.307279+0000", 2024-09-13T14:59:57.285 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:57.285 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.285 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.285 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:57.285 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.307278+0000", 2024-09-13T14:59:57.285 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:59:57.285 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.285 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.286 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:57.286 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.307327+0000", 2024-09-13T14:59:57.286 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8435e-05 2024-09-13T14:59:57.286 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.286 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.286 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:57.286 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.307333+0000", 2024-09-13T14:59:57.286 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5989999999999998e-06 2024-09-13T14:59:57.286 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.286 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.286 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:57.286 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.307338+0000", 2024-09-13T14:59:57.286 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0300000000000001e-06 2024-09-13T14:59:57.286 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.287 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.287 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.287 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.307385+0000", 2024-09-13T14:59:57.287 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6740000000000003e-05 2024-09-13T14:59:57.287 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.287 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.287 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:57.287 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.307397+0000", 2024-09-13T14:59:57.287 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1565e-05 2024-09-13T14:59:57.287 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.287 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.287 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.287 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.056816+0000", 2024-09-13T14:59:57.287 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.74941926000000003 2024-09-13T14:59:57.287 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.288 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.288 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:57.288 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.056825+0000", 2024-09-13T14:59:57.288 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.6719999999999993e-06 2024-09-13T14:59:57.288 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.288 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.288 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.288 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.308111+0000", 2024-09-13T14:59:57.288 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.251286127 2024-09-13T14:59:57.288 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.288 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.288 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:59:57.288 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.308256+0000", 2024-09-13T14:59:57.288 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014530000000000001 2024-09-13T14:59:57.288 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.289 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.289 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:59:57.289 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.308423+0000", 2024-09-13T14:59:57.289 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016677200000000001 2024-09-13T14:59:57.289 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.289 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.289 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:59:57.289 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.313898+0000", 2024-09-13T14:59:57.289 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0054755699999999999 2024-09-13T14:59:57.289 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.289 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.289 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:57.289 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.315748+0000", 2024-09-13T14:59:57.290 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0018497710000000001 2024-09-13T14:59:57.290 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.290 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.290 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:57.290 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.323271+0000", 2024-09-13T14:59:57.290 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0075229169999999996 2024-09-13T14:59:57.290 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.290 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.290 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:59:57.290 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.323324+0000", 2024-09-13T14:59:57.290 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2889000000000001e-05 2024-09-13T14:59:57.290 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.290 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.290 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:57.291 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.325340+0000", 2024-09-13T14:59:57.291 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0020155580000000002 2024-09-13T14:59:57.291 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:57.291 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:57.291 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:57.291 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.291 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.291 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:72478 9.13 9:cd39ee78:::benchmark_data_smithi102_348563_object72477:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-09-13T14:59:57.291 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:51:46.334058+0000", 2024-09-13T14:59:57.291 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 490.91971852500001, 2024-09-13T14:59:57.291 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.016141556, 2024-09-13T14:59:57.291 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:57.291 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:57.291 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:57.291 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:59:57.292 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:59:57.292 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 72478 2024-09-13T14:59:57.292 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.292 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:57.292 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.292 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:57.292 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.334058+0000", 2024-09-13T14:59:57.292 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:57.292 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.292 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.292 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:57.292 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.334058+0000", 2024-09-13T14:59:57.292 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:57.292 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.292 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.293 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:57.293 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.334057+0000", 2024-09-13T14:59:57.293 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:57.293 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.293 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.293 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:57.293 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.334087+0000", 2024-09-13T14:59:57.293 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0477999999999999e-05 2024-09-13T14:59:57.293 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.293 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.293 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:57.293 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.334098+0000", 2024-09-13T14:59:57.293 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0684e-05 2024-09-13T14:59:57.293 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.294 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.294 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:57.294 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.334105+0000", 2024-09-13T14:59:57.294 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6080000000000001e-06 2024-09-13T14:59:57.294 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.294 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.294 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.294 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.334197+0000", 2024-09-13T14:59:57.294 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.2443999999999998e-05 2024-09-13T14:59:57.294 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.294 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.294 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:57.294 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.334207+0000", 2024-09-13T14:59:57.295 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0013000000000001e-05 2024-09-13T14:59:57.295 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.295 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.295 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.295 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.056863+0000", 2024-09-13T14:59:57.295 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.72265576499999995 2024-09-13T14:59:57.295 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.295 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.295 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:57.295 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.056870+0000", 2024-09-13T14:59:57.295 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8569999999999997e-06 2024-09-13T14:59:57.295 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.295 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.296 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.296 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.309442+0000", 2024-09-13T14:59:57.296 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2525720090000001 2024-09-13T14:59:57.296 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.296 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.296 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:59:57.296 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.309592+0000", 2024-09-13T14:59:57.296 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015071600000000001 2024-09-13T14:59:57.296 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.296 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.296 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:59:57.296 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.309748+0000", 2024-09-13T14:59:57.296 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015559800000000001 2024-09-13T14:59:57.296 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.297 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.297 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:59:57.297 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.313905+0000", 2024-09-13T14:59:57.297 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0041569320000000003 2024-09-13T14:59:57.297 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.297 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.297 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:57.297 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.325405+0000", 2024-09-13T14:59:57.297 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011499578999999999 2024-09-13T14:59:57.297 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.297 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.297 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:57.297 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.348507+0000", 2024-09-13T14:59:57.298 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.023102192000000001 2024-09-13T14:59:57.298 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.298 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.298 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:59:57.298 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.348573+0000", 2024-09-13T14:59:57.298 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6265000000000003e-05 2024-09-13T14:59:57.298 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.298 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.298 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:57.298 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.350200+0000", 2024-09-13T14:59:57.298 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0016267650000000001 2024-09-13T14:59:57.298 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:57.298 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:57.299 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:57.299 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.299 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.299 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:72481 9.13 9:c872827f:::benchmark_data_smithi102_348563_object72480:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-09-13T14:59:57.299 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:51:46.336298+0000", 2024-09-13T14:59:57.299 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 490.91747822899998, 2024-09-13T14:59:57.299 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0149789880000002, 2024-09-13T14:59:57.299 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:57.299 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:57.299 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:57.299 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:59:57.299 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:59:57.299 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 72481 2024-09-13T14:59:57.299 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.300 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:57.300 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.300 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:57.300 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.336298+0000", 2024-09-13T14:59:57.300 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:57.300 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.300 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.300 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:57.301 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.336298+0000", 2024-09-13T14:59:57.301 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:57.301 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.301 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.301 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:57.301 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.336297+0000", 2024-09-13T14:59:57.301 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:57.301 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.301 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.301 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:57.301 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.336328+0000", 2024-09-13T14:59:57.301 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0834000000000003e-05 2024-09-13T14:59:57.301 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.302 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.302 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:57.302 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.336340+0000", 2024-09-13T14:59:57.302 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2451e-05 2024-09-13T14:59:57.302 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.302 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.302 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:57.302 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.336347+0000", 2024-09-13T14:59:57.302 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1339999999999998e-06 2024-09-13T14:59:57.302 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.302 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.302 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.302 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.336403+0000", 2024-09-13T14:59:57.302 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5198999999999998e-05 2024-09-13T14:59:57.303 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.303 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.303 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:57.303 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.336416+0000", 2024-09-13T14:59:57.303 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3451999999999999e-05 2024-09-13T14:59:57.303 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.303 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.303 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.303 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.056988+0000", 2024-09-13T14:59:57.303 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.72057172000000003 2024-09-13T14:59:57.303 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.303 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.303 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:57.303 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.056993+0000", 2024-09-13T14:59:57.304 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5690000000000004e-06 2024-09-13T14:59:57.304 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.304 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.304 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.304 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.312379+0000", 2024-09-13T14:59:57.304 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.255385564 2024-09-13T14:59:57.304 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.304 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.304 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:59:57.304 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.312524+0000", 2024-09-13T14:59:57.304 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014541200000000001 2024-09-13T14:59:57.304 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.304 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.304 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:59:57.305 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.312690+0000", 2024-09-13T14:59:57.305 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016532 2024-09-13T14:59:57.305 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.305 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.305 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:59:57.305 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.318889+0000", 2024-09-13T14:59:57.305 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.006199202 2024-09-13T14:59:57.305 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.305 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.305 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:57.305 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.325439+0000", 2024-09-13T14:59:57.305 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0065503610000000002 2024-09-13T14:59:57.305 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.305 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.305 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:57.306 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.350254+0000", 2024-09-13T14:59:57.306 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024814994999999999 2024-09-13T14:59:57.306 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.306 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.306 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:59:57.306 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.350321+0000", 2024-09-13T14:59:57.306 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7285000000000004e-05 2024-09-13T14:59:57.306 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.306 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.306 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:57.306 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.351277+0000", 2024-09-13T14:59:57.306 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000955977 2024-09-13T14:59:57.306 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:57.307 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:57.307 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:57.307 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.307 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.307 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:72521 9.13 9:cd0cc28e:::benchmark_data_smithi102_348563_object72520:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-09-13T14:59:57.307 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:51:46.367757+0000", 2024-09-13T14:59:57.307 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 490.88602003900002, 2024-09-13T14:59:57.307 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.984496971, 2024-09-13T14:59:57.307 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:57.307 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:57.307 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:57.307 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:59:57.307 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:59:57.307 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 72521 2024-09-13T14:59:57.308 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.308 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:57.308 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.308 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:57.308 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.367757+0000", 2024-09-13T14:59:57.308 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:57.308 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.308 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.308 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:57.308 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.367757+0000", 2024-09-13T14:59:57.308 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:57.308 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.308 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.309 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:57.309 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.367755+0000", 2024-09-13T14:59:57.309 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:57.309 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.309 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.309 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:57.309 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.367856+0000", 2024-09-13T14:59:57.309 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000100467 2024-09-13T14:59:57.309 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.309 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.309 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:57.309 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.367865+0000", 2024-09-13T14:59:57.309 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.833e-06 2024-09-13T14:59:57.309 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.310 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.310 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:57.310 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.367871+0000", 2024-09-13T14:59:57.310 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3180000000000003e-06 2024-09-13T14:59:57.310 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.310 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.310 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.310 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.368022+0000", 2024-09-13T14:59:57.310 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015103 2024-09-13T14:59:57.310 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.310 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.310 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:57.310 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.368032+0000", 2024-09-13T14:59:57.310 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.024e-05 2024-09-13T14:59:57.311 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.311 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.311 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.311 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.057022+0000", 2024-09-13T14:59:57.311 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.68899013099999995 2024-09-13T14:59:57.311 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.311 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.311 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:57.311 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.057033+0000", 2024-09-13T14:59:57.311 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0335e-05 2024-09-13T14:59:57.311 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.311 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.311 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.311 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.313958+0000", 2024-09-13T14:59:57.312 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.256924937 2024-09-13T14:59:57.312 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.312 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.312 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:59:57.312 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.314126+0000", 2024-09-13T14:59:57.312 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000167978 2024-09-13T14:59:57.312 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.312 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.312 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:59:57.312 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.314336+0000", 2024-09-13T14:59:57.312 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000209855 2024-09-13T14:59:57.312 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.312 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.313 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:59:57.313 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.320183+0000", 2024-09-13T14:59:57.313 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0058473869999999999 2024-09-13T14:59:57.313 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.313 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.313 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:57.313 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.325473+0000", 2024-09-13T14:59:57.313 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0052903059999999998 2024-09-13T14:59:57.313 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.313 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.313 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:57.313 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.351404+0000", 2024-09-13T14:59:57.313 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.025930947999999999 2024-09-13T14:59:57.313 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.313 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.314 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:59:57.314 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.351459+0000", 2024-09-13T14:59:57.314 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5000999999999997e-05 2024-09-13T14:59:57.314 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.314 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.314 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:57.314 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.352254+0000", 2024-09-13T14:59:57.314 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00079445299999999998 2024-09-13T14:59:57.314 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:57.314 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:57.314 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:57.314 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.314 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.315 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:72531 9.13 9:cd5ddef9:::benchmark_data_smithi102_348563_object72530:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-09-13T14:59:57.315 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:51:46.381783+0000", 2024-09-13T14:59:57.315 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 490.87199370799999, 2024-09-13T14:59:57.315 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.971096263, 2024-09-13T14:59:57.315 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:57.315 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:57.315 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:57.315 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:59:57.315 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:59:57.315 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 72531 2024-09-13T14:59:57.315 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.315 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:57.315 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.316 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:57.316 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.381783+0000", 2024-09-13T14:59:57.316 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:57.316 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.316 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.316 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:57.316 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.381783+0000", 2024-09-13T14:59:57.316 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:57.316 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.316 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.316 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:57.316 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.381781+0000", 2024-09-13T14:59:57.316 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:59:57.317 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.317 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.317 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:57.317 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.381839+0000", 2024-09-13T14:59:57.317 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8542999999999998e-05 2024-09-13T14:59:57.317 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.317 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.317 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:57.317 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.381853+0000", 2024-09-13T14:59:57.317 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3885e-05 2024-09-13T14:59:57.317 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.317 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.317 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:57.317 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.381861+0000", 2024-09-13T14:59:57.318 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7260000000000002e-06 2024-09-13T14:59:57.318 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.318 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.318 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.318 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.381922+0000", 2024-09-13T14:59:57.318 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0841000000000002e-05 2024-09-13T14:59:57.318 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.318 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.318 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:57.318 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.381931+0000", 2024-09-13T14:59:57.318 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.3510000000000008e-06 2024-09-13T14:59:57.318 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.319 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.319 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.319 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.057056+0000", 2024-09-13T14:59:57.319 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.67512471100000004 2024-09-13T14:59:57.319 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.319 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.319 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:57.319 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.057061+0000", 2024-09-13T14:59:57.319 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6009999999999997e-06 2024-09-13T14:59:57.319 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.319 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.319 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.319 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.315777+0000", 2024-09-13T14:59:57.320 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2587159479999999 2024-09-13T14:59:57.320 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.320 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.320 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:59:57.320 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.315929+0000", 2024-09-13T14:59:57.320 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015216700000000001 2024-09-13T14:59:57.320 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.320 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.320 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:59:57.320 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.316179+0000", 2024-09-13T14:59:57.320 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000250709 2024-09-13T14:59:57.320 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.320 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.320 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:59:57.321 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.320189+0000", 2024-09-13T14:59:57.321 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0040100420000000001 2024-09-13T14:59:57.321 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.321 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.321 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:57.321 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.333905+0000", 2024-09-13T14:59:57.321 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013715283999999999 2024-09-13T14:59:57.321 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.321 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.321 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:57.321 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.352333+0000", 2024-09-13T14:59:57.321 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018428402999999999 2024-09-13T14:59:57.321 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.322 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.322 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:59:57.322 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.352367+0000", 2024-09-13T14:59:57.322 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.3874999999999997e-05 2024-09-13T14:59:57.322 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.322 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.322 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:57.322 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.352879+0000", 2024-09-13T14:59:57.322 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00051223200000000005 2024-09-13T14:59:57.322 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:57.322 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:57.322 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:57.322 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.322 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.323 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:72556 9.13 9:cf932f14:::benchmark_data_smithi102_348563_object72555:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-09-13T14:59:57.323 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:51:46.403040+0000", 2024-09-13T14:59:57.323 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 490.85073648600002, 2024-09-13T14:59:57.323 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9505509889999999, 2024-09-13T14:59:57.323 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:57.323 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:57.323 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:57.323 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:59:57.323 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:59:57.323 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 72556 2024-09-13T14:59:57.323 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.323 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:57.323 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.323 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:57.324 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.403040+0000", 2024-09-13T14:59:57.324 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:57.324 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.324 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.324 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:57.324 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.403040+0000", 2024-09-13T14:59:57.324 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:57.324 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.324 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.324 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:57.324 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.403039+0000", 2024-09-13T14:59:57.324 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:59:57.324 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.325 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.325 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:57.325 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.403067+0000", 2024-09-13T14:59:57.325 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7789000000000001e-05 2024-09-13T14:59:57.325 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.325 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.325 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:57.325 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.403086+0000", 2024-09-13T14:59:57.325 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9351000000000002e-05 2024-09-13T14:59:57.325 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.325 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.325 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:57.325 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.403094+0000", 2024-09-13T14:59:57.325 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5050000000000001e-06 2024-09-13T14:59:57.325 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.326 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.326 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.326 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.403146+0000", 2024-09-13T14:59:57.326 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2153e-05 2024-09-13T14:59:57.326 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.326 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.326 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:57.326 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.403155+0000", 2024-09-13T14:59:57.326 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.3000000000000007e-06 2024-09-13T14:59:57.326 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.326 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.326 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.326 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.057087+0000", 2024-09-13T14:59:57.326 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.65393121300000001 2024-09-13T14:59:57.327 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.327 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.327 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:57.327 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.057095+0000", 2024-09-13T14:59:57.327 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8310000000000007e-06 2024-09-13T14:59:57.327 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.327 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.327 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.327 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.317544+0000", 2024-09-13T14:59:57.327 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.260448537 2024-09-13T14:59:57.327 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.327 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.327 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:59:57.327 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.317701+0000", 2024-09-13T14:59:57.327 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000157405 2024-09-13T14:59:57.328 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.328 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.328 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:59:57.328 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.317872+0000", 2024-09-13T14:59:57.328 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0001707 2024-09-13T14:59:57.328 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.328 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.328 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:59:57.328 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.320245+0000", 2024-09-13T14:59:57.328 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0023728360000000001 2024-09-13T14:59:57.328 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.328 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.328 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:57.328 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.351338+0000", 2024-09-13T14:59:57.330 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.031092813 2024-09-13T14:59:57.330 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.330 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.330 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:57.330 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.352986+0000", 2024-09-13T14:59:57.330 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0016485619999999999 2024-09-13T14:59:57.330 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.330 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.330 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:59:57.330 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.353017+0000", 2024-09-13T14:59:57.331 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0880999999999998e-05 2024-09-13T14:59:57.331 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.331 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.331 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:57.331 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.353591+0000", 2024-09-13T14:59:57.331 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00057408700000000003 2024-09-13T14:59:57.331 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:57.331 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:57.331 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:57.331 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.331 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.331 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:72572 9.13 9:ce9461e5:::benchmark_data_smithi102_348563_object72571:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-09-13T14:59:57.331 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:51:46.420582+0000", 2024-09-13T14:59:57.331 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 490.83319427700002, 2024-09-13T14:59:57.332 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9336864380000001, 2024-09-13T14:59:57.332 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:57.332 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:57.332 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:57.332 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:59:57.332 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:59:57.332 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 72572 2024-09-13T14:59:57.332 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.332 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:57.332 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.332 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:57.332 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.420582+0000", 2024-09-13T14:59:57.332 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:57.333 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.333 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.333 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:57.333 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.420582+0000", 2024-09-13T14:59:57.333 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:57.333 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.333 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.333 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:57.333 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.420581+0000", 2024-09-13T14:59:57.333 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:57.333 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.333 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.333 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:57.333 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.420633+0000", 2024-09-13T14:59:57.334 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2237999999999997e-05 2024-09-13T14:59:57.334 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.334 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.334 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:57.334 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.420644+0000", 2024-09-13T14:59:57.334 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1249e-05 2024-09-13T14:59:57.334 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.334 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.334 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:57.334 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.420650+0000", 2024-09-13T14:59:57.334 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4500000000000003e-06 2024-09-13T14:59:57.334 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.334 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.334 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.335 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.420696+0000", 2024-09-13T14:59:57.335 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5902999999999997e-05 2024-09-13T14:59:57.335 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.335 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.335 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:57.335 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:46.420704+0000", 2024-09-13T14:59:57.335 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.6559999999999998e-06 2024-09-13T14:59:57.335 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.335 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.335 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.335 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.057119+0000", 2024-09-13T14:59:57.335 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.63641496200000003 2024-09-13T14:59:57.335 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.335 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.336 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:57.336 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:47.057124+0000", 2024-09-13T14:59:57.336 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5920000000000002e-06 2024-09-13T14:59:57.336 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.336 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.336 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.336 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.318936+0000", 2024-09-13T14:59:57.336 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2618115219999999 2024-09-13T14:59:57.336 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.336 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.336 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:59:57.336 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.319072+0000", 2024-09-13T14:59:57.336 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013614700000000001 2024-09-13T14:59:57.337 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.337 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.337 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:59:57.337 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.319195+0000", 2024-09-13T14:59:57.337 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012294199999999999 2024-09-13T14:59:57.337 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.337 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.337 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:59:57.337 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.323229+0000", 2024-09-13T14:59:57.337 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0040348149999999998 2024-09-13T14:59:57.337 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.337 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.337 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:57.337 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.352299+0000", 2024-09-13T14:59:57.338 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.029069787 2024-09-13T14:59:57.338 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.338 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.338 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:57.338 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.353679+0000", 2024-09-13T14:59:57.338 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0013798129999999999 2024-09-13T14:59:57.338 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.338 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.338 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:59:57.338 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.353708+0000", 2024-09-13T14:59:57.338 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9454e-05 2024-09-13T14:59:57.338 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.338 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.338 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:57.339 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:51:48.354269+0000", 2024-09-13T14:59:57.339 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00056036699999999998 2024-09-13T14:59:57.339 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:57.339 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:57.339 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:57.339 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.339 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.339 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:113330 9.17 9:e8bce8ce:::benchmark_data_smithi102_348563_object113329:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e402)", 2024-09-13T14:59:57.339 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:52:26.477514+0000", 2024-09-13T14:59:57.339 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 450.77626276699999, 2024-09-13T14:59:57.339 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0168906770000001, 2024-09-13T14:59:57.339 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:57.339 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:57.339 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:57.340 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:59:57.340 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:59:57.340 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113330 2024-09-13T14:59:57.340 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.340 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:57.340 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.340 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:57.340 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.477514+0000", 2024-09-13T14:59:57.340 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:57.340 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.340 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.340 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:57.340 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.477514+0000", 2024-09-13T14:59:57.340 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:57.341 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.341 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.341 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:57.341 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.477512+0000", 2024-09-13T14:59:57.341 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:57.341 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.341 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.341 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:57.341 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.477630+0000", 2024-09-13T14:59:57.341 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011756000000000001 2024-09-13T14:59:57.341 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.341 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.341 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:57.342 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.477642+0000", 2024-09-13T14:59:57.342 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2492e-05 2024-09-13T14:59:57.342 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.342 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.342 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:57.342 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.477648+0000", 2024-09-13T14:59:57.342 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5230000000000003e-06 2024-09-13T14:59:57.342 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.342 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.342 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.342 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.477703+0000", 2024-09-13T14:59:57.342 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5500000000000001e-05 2024-09-13T14:59:57.342 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.342 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.342 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:57.343 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.477715+0000", 2024-09-13T14:59:57.343 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1083e-05 2024-09-13T14:59:57.343 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.343 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.343 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.343 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:27.337497+0000", 2024-09-13T14:59:57.343 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.85978237800000001 2024-09-13T14:59:57.343 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.343 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.343 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:57.343 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:27.337508+0000", 2024-09-13T14:59:57.343 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1004e-05 2024-09-13T14:59:57.343 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.344 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.344 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.344 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.485507+0000", 2024-09-13T14:59:57.344 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.147998759 2024-09-13T14:59:57.344 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.344 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.344 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:59:57.344 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.485653+0000", 2024-09-13T14:59:57.344 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014649300000000001 2024-09-13T14:59:57.344 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.344 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.344 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:59:57.344 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.485818+0000", 2024-09-13T14:59:57.345 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016457 2024-09-13T14:59:57.345 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.345 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.345 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:57.345 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.492578+0000", 2024-09-13T14:59:57.345 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0067599959999999999 2024-09-13T14:59:57.345 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.345 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.345 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:57.345 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.492640+0000", 2024-09-13T14:59:57.345 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2607999999999994e-05 2024-09-13T14:59:57.345 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.345 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.345 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:59:57.346 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.492652+0000", 2024-09-13T14:59:57.346 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2021000000000001e-05 2024-09-13T14:59:57.346 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.346 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.346 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:59:57.346 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.492691+0000", 2024-09-13T14:59:57.346 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8656999999999999e-05 2024-09-13T14:59:57.346 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.346 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.346 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:57.346 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.494405+0000", 2024-09-13T14:59:57.346 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0017135500000000001 2024-09-13T14:59:57.346 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:57.346 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:57.347 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:57.347 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.347 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.347 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:113341 9.17 9:ead480fa:::benchmark_data_smithi102_348563_object113340:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e402)", 2024-09-13T14:59:57.347 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:52:26.502483+0000", 2024-09-13T14:59:57.347 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 450.75129392899998, 2024-09-13T14:59:57.347 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0150210039999998, 2024-09-13T14:59:57.347 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:57.347 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:57.347 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:57.347 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:59:57.347 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:59:57.347 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113341 2024-09-13T14:59:57.348 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.348 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:57.348 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.348 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:57.348 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.502483+0000", 2024-09-13T14:59:57.348 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:57.348 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.348 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.348 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:57.348 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.502483+0000", 2024-09-13T14:59:57.348 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:57.348 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.348 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.349 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:57.349 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.502481+0000", 2024-09-13T14:59:57.349 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:59:57.349 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.349 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.349 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:57.349 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.502555+0000", 2024-09-13T14:59:57.349 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4099999999999999e-05 2024-09-13T14:59:57.349 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.349 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.349 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:57.349 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.502567+0000", 2024-09-13T14:59:57.349 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1698999999999999e-05 2024-09-13T14:59:57.350 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.350 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.350 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:57.350 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.502574+0000", 2024-09-13T14:59:57.350 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2570000000000001e-06 2024-09-13T14:59:57.350 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.350 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.350 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.350 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.502625+0000", 2024-09-13T14:59:57.350 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0603000000000003e-05 2024-09-13T14:59:57.350 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.350 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.350 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:57.351 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.502636+0000", 2024-09-13T14:59:57.351 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.082e-05 2024-09-13T14:59:57.351 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.351 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.351 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.351 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:27.337554+0000", 2024-09-13T14:59:57.351 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.83491843899999996 2024-09-13T14:59:57.351 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.351 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.351 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:57.351 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:27.337562+0000", 2024-09-13T14:59:57.351 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1960000000000008e-06 2024-09-13T14:59:57.351 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.352 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.352 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.352 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.489118+0000", 2024-09-13T14:59:57.352 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1515562509999999 2024-09-13T14:59:57.352 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.352 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.352 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:59:57.352 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.489230+0000", 2024-09-13T14:59:57.352 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011199100000000001 2024-09-13T14:59:57.352 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.352 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.352 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:59:57.352 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.489333+0000", 2024-09-13T14:59:57.353 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000102605 2024-09-13T14:59:57.353 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.353 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.353 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:59:57.353 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.498423+0000", 2024-09-13T14:59:57.353 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0090894700000000005 2024-09-13T14:59:57.353 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.353 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.353 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:57.353 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.498475+0000", 2024-09-13T14:59:57.353 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2296999999999998e-05 2024-09-13T14:59:57.353 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.353 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.353 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:57.354 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.515297+0000", 2024-09-13T14:59:57.354 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016822522999999999 2024-09-13T14:59:57.354 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.354 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.354 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:59:57.354 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.515353+0000", 2024-09-13T14:59:57.354 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5451000000000002e-05 2024-09-13T14:59:57.354 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.354 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.354 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:57.354 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.517504+0000", 2024-09-13T14:59:57.354 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0021509900000000002 2024-09-13T14:59:57.354 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:57.355 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:57.355 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:57.355 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.355 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.355 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:113388 9.17 9:ededae97:::benchmark_data_smithi102_348563_object113387:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e402)", 2024-09-13T14:59:57.355 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:52:26.574318+0000", 2024-09-13T14:59:57.355 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 450.67945844600001, 2024-09-13T14:59:57.355 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.944988825, 2024-09-13T14:59:57.355 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:57.355 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:57.355 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:57.355 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:59:57.355 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:59:57.355 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113388 2024-09-13T14:59:57.356 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.356 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:57.356 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.356 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:57.356 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.574318+0000", 2024-09-13T14:59:57.356 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:57.356 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.356 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.356 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:57.356 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.574318+0000", 2024-09-13T14:59:57.356 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:57.356 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.356 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.357 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:57.357 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.574316+0000", 2024-09-13T14:59:57.357 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999981 2024-09-13T14:59:57.357 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.357 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.357 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:57.357 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.574361+0000", 2024-09-13T14:59:57.357 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5012000000000002e-05 2024-09-13T14:59:57.357 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.357 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.357 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:57.357 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.574373+0000", 2024-09-13T14:59:57.358 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1810999999999999e-05 2024-09-13T14:59:57.358 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.358 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.358 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:57.358 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.574379+0000", 2024-09-13T14:59:57.358 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.348e-06 2024-09-13T14:59:57.358 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.358 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.358 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.358 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.574431+0000", 2024-09-13T14:59:57.358 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1205000000000002e-05 2024-09-13T14:59:57.358 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.359 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.359 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:57.359 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.574443+0000", 2024-09-13T14:59:57.359 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2842e-05 2024-09-13T14:59:57.359 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.359 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.359 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.359 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:27.337587+0000", 2024-09-13T14:59:57.359 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.76314326399999999 2024-09-13T14:59:57.359 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.359 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.359 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:57.359 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:27.337595+0000", 2024-09-13T14:59:57.359 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.4600000000000003e-06 2024-09-13T14:59:57.360 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.360 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.360 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.360 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.490785+0000", 2024-09-13T14:59:57.360 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1531900020000001 2024-09-13T14:59:57.360 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.360 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.360 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:59:57.360 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.490897+0000", 2024-09-13T14:59:57.360 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011162699999999999 2024-09-13T14:59:57.360 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.360 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.361 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:59:57.361 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.491023+0000", 2024-09-13T14:59:57.361 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012667900000000001 2024-09-13T14:59:57.361 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.361 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.361 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:59:57.361 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.498430+0000", 2024-09-13T14:59:57.361 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00740619 2024-09-13T14:59:57.361 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.361 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.361 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:57.361 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.498510+0000", 2024-09-13T14:59:57.362 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0636999999999994e-05 2024-09-13T14:59:57.362 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.362 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.362 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:57.362 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.517640+0000", 2024-09-13T14:59:57.362 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.019130004999999999 2024-09-13T14:59:57.362 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.362 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.362 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:59:57.362 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.517695+0000", 2024-09-13T14:59:57.362 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4718e-05 2024-09-13T14:59:57.362 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.363 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.363 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:57.363 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.519307+0000", 2024-09-13T14:59:57.363 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0016120279999999999 2024-09-13T14:59:57.363 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:57.363 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:57.363 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:57.363 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.363 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.363 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:113411 9.17 9:e864c0f1:::benchmark_data_smithi102_348563_object113410:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e402)", 2024-09-13T14:59:57.363 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:52:26.602093+0000", 2024-09-13T14:59:57.363 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 450.65168337699998, 2024-09-13T14:59:57.363 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.921966007, 2024-09-13T14:59:57.364 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:57.364 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:57.364 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:57.364 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:59:57.364 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:59:57.364 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113411 2024-09-13T14:59:57.364 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.364 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:57.364 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.364 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:57.364 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.602093+0000", 2024-09-13T14:59:57.364 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:57.364 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.364 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.365 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:57.365 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.602093+0000", 2024-09-13T14:59:57.365 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:57.365 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.365 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.365 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:57.365 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.602092+0000", 2024-09-13T14:59:57.365 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:59:57.365 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.365 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.365 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:57.365 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.602155+0000", 2024-09-13T14:59:57.365 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3108999999999995e-05 2024-09-13T14:59:57.366 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.366 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.366 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:57.366 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.602169+0000", 2024-09-13T14:59:57.366 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3198e-05 2024-09-13T14:59:57.366 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.366 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.366 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:57.366 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.602176+0000", 2024-09-13T14:59:57.366 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.238e-06 2024-09-13T14:59:57.366 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.366 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.366 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.366 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.602227+0000", 2024-09-13T14:59:57.367 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1411e-05 2024-09-13T14:59:57.367 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.367 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.367 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:57.367 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.602236+0000", 2024-09-13T14:59:57.367 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1579999999999994e-06 2024-09-13T14:59:57.367 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.367 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.367 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.367 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:27.337620+0000", 2024-09-13T14:59:57.367 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.73538361699999999 2024-09-13T14:59:57.367 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.367 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.367 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:57.367 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:27.337628+0000", 2024-09-13T14:59:57.368 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8730000000000007e-06 2024-09-13T14:59:57.368 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.368 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.368 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.368 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.494470+0000", 2024-09-13T14:59:57.368 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.15684242 2024-09-13T14:59:57.368 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.368 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.368 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:59:57.368 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.494630+0000", 2024-09-13T14:59:57.368 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015944 2024-09-13T14:59:57.368 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.368 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.369 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:59:57.369 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.494786+0000", 2024-09-13T14:59:57.369 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000155909 2024-09-13T14:59:57.369 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.369 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.369 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:59:57.369 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.498518+0000", 2024-09-13T14:59:57.369 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0037322029999999999 2024-09-13T14:59:57.369 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.369 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.369 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:57.369 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.509421+0000", 2024-09-13T14:59:57.369 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010903256 2024-09-13T14:59:57.370 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.370 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.370 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:57.370 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.521680+0000", 2024-09-13T14:59:57.370 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012258929 2024-09-13T14:59:57.370 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.370 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.370 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:59:57.370 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.521736+0000", 2024-09-13T14:59:57.370 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5813999999999998e-05 2024-09-13T14:59:57.370 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.370 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.370 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:57.371 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.524059+0000", 2024-09-13T14:59:57.371 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0023235220000000002 2024-09-13T14:59:57.371 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:57.371 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:57.371 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:57.371 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.371 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.371 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36528.0:113426 9.17 9:ee9b3220:::benchmark_data_smithi102_348563_object113425:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e402)", 2024-09-13T14:59:57.371 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:52:26.616755+0000", 2024-09-13T14:59:57.371 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 450.63702186799998, 2024-09-13T14:59:57.371 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.909422927, 2024-09-13T14:59:57.371 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:57.371 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:57.372 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:57.372 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36528", 2024-09-13T14:59:57.372 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/859174294", 2024-09-13T14:59:57.372 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 113426 2024-09-13T14:59:57.372 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.372 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:57.372 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.372 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:57.372 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.616755+0000", 2024-09-13T14:59:57.372 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:57.372 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.372 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.372 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:57.373 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.616755+0000", 2024-09-13T14:59:57.373 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:57.373 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.373 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.373 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:57.373 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.616754+0000", 2024-09-13T14:59:57.373 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T14:59:57.373 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.373 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.373 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:57.373 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.616878+0000", 2024-09-13T14:59:57.373 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000124043 2024-09-13T14:59:57.373 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.373 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.374 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:57.374 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.616888+0000", 2024-09-13T14:59:57.374 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0301e-05 2024-09-13T14:59:57.374 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.374 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.374 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:57.374 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.616895+0000", 2024-09-13T14:59:57.374 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6649999999999997e-06 2024-09-13T14:59:57.374 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.374 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.374 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.374 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.617105+0000", 2024-09-13T14:59:57.374 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00021071100000000001 2024-09-13T14:59:57.374 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.375 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.375 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:57.375 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:26.617115+0000", 2024-09-13T14:59:57.375 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.8849999999999994e-06 2024-09-13T14:59:57.375 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.375 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.375 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.375 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:27.337655+0000", 2024-09-13T14:59:57.375 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.72054000100000004 2024-09-13T14:59:57.375 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.375 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.375 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:57.375 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:27.337663+0000", 2024-09-13T14:59:57.375 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7230000000000004e-06 2024-09-13T14:59:57.376 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.376 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.376 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.376 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.496533+0000", 2024-09-13T14:59:57.376 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.158870515 2024-09-13T14:59:57.376 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.376 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.376 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:59:57.376 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.496848+0000", 2024-09-13T14:59:57.376 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00031459100000000002 2024-09-13T14:59:57.376 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.376 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.376 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:59:57.377 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.496952+0000", 2024-09-13T14:59:57.377 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000104147 2024-09-13T14:59:57.377 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.377 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.377 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:59:57.377 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.503972+0000", 2024-09-13T14:59:57.377 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0070199650000000004 2024-09-13T14:59:57.377 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.377 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.377 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:57.377 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.517572+0000", 2024-09-13T14:59:57.377 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013599546000000001 2024-09-13T14:59:57.377 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.377 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.378 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:57.378 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.524126+0000", 2024-09-13T14:59:57.378 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0065545020000000002 2024-09-13T14:59:57.378 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.378 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.378 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:59:57.378 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.524182+0000", 2024-09-13T14:59:57.378 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5952999999999998e-05 2024-09-13T14:59:57.378 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.378 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.378 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:57.378 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:28.526178+0000", 2024-09-13T14:59:57.378 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001995564 2024-09-13T14:59:57.378 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:57.379 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:57.379 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:57.379 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.379 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.379 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36738.0:9805 10.1 10:87853b1e:::benchmark_data_smithi102_377304_object9804:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e441)", 2024-09-13T14:59:57.379 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:55:16.922955+0000", 2024-09-13T14:59:57.379 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 280.33082210999999, 2024-09-13T14:59:57.379 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.271878337, 2024-09-13T14:59:57.379 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:57.379 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-13T14:59:57.379 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:57.379 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36738", 2024-09-13T14:59:57.379 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/2915890700", 2024-09-13T14:59:57.379 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 9805 2024-09-13T14:59:57.380 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.380 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:57.380 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.380 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:57.380 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.922955+0000", 2024-09-13T14:59:57.380 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:57.380 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.380 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.380 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:57.380 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.922955+0000", 2024-09-13T14:59:57.380 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:57.380 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.380 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.380 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:57.381 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.922952+0000", 2024-09-13T14:59:57.381 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999971 2024-09-13T14:59:57.381 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.381 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.381 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:57.381 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.922989+0000", 2024-09-13T14:59:57.381 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7221000000000002e-05 2024-09-13T14:59:57.381 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.381 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.381 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:57.381 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928405+0000", 2024-09-13T14:59:57.381 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0054158330000000001 2024-09-13T14:59:57.381 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.381 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.382 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:57.382 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928415+0000", 2024-09-13T14:59:57.382 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.7200000000000001e-06 2024-09-13T14:59:57.382 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.382 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.382 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.382 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928621+0000", 2024-09-13T14:59:57.382 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000206126 2024-09-13T14:59:57.382 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.382 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.382 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:57.382 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928636+0000", 2024-09-13T14:59:57.382 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5101e-05 2024-09-13T14:59:57.382 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.383 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.383 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.383 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:17.919116+0000", 2024-09-13T14:59:57.383 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.99048049599999999 2024-09-13T14:59:57.383 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.383 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.383 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:57.383 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:17.919126+0000", 2024-09-13T14:59:57.383 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6029999999999994e-06 2024-09-13T14:59:57.383 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.383 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.383 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.383 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:19.173075+0000", 2024-09-13T14:59:57.383 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.253948692 2024-09-13T14:59:57.384 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.384 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.384 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for missing object", 2024-09-13T14:59:57.384 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:19.174363+0000", 2024-09-13T14:59:57.384 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001287898 2024-09-13T14:59:57.384 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.384 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.384 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.384 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:19.178513+0000", 2024-09-13T14:59:57.384 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.004150089 2024-09-13T14:59:57.384 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.384 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.384 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for degraded object", 2024-09-13T14:59:57.385 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:19.178580+0000", 2024-09-13T14:59:57.385 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7539000000000005e-05 2024-09-13T14:59:57.385 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.385 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.385 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.385 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:19.194762+0000", 2024-09-13T14:59:57.385 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016181389000000001 2024-09-13T14:59:57.385 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.385 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.385 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:57.385 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:19.194833+0000", 2024-09-13T14:59:57.385 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1254999999999995e-05 2024-09-13T14:59:57.385 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:57.385 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:57.386 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:57.386 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.386 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.386 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36738.0:9808 10.c 10:318791bf:::benchmark_data_smithi102_377304_object9807:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e441)", 2024-09-13T14:59:57.386 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:55:16.928442+0000", 2024-09-13T14:59:57.386 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 280.325334903, 2024-09-13T14:59:57.386 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.037536926, 2024-09-13T14:59:57.386 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:57.386 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:57.386 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:57.386 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36738", 2024-09-13T14:59:57.386 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/2915890700", 2024-09-13T14:59:57.386 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 9808 2024-09-13T14:59:57.387 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.387 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:57.387 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.387 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:57.387 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928442+0000", 2024-09-13T14:59:57.387 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:57.387 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.387 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.387 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:57.387 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928442+0000", 2024-09-13T14:59:57.387 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:57.387 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.387 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.387 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:57.388 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928440+0000", 2024-09-13T14:59:57.388 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:57.388 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.388 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.388 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:57.388 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928475+0000", 2024-09-13T14:59:57.388 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4659999999999997e-05 2024-09-13T14:59:57.388 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.388 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.388 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:57.388 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928485+0000", 2024-09-13T14:59:57.388 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.5189999999999998e-06 2024-09-13T14:59:57.389 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.389 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.389 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:57.389 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928489+0000", 2024-09-13T14:59:57.389 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8339999999999998e-06 2024-09-13T14:59:57.389 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.389 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.389 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.389 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.929346+0000", 2024-09-13T14:59:57.389 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00085621199999999997 2024-09-13T14:59:57.389 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.389 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.389 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:57.389 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.929356+0000", 2024-09-13T14:59:57.390 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.049e-05 2024-09-13T14:59:57.390 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.390 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.390 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.390 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:17.918693+0000", 2024-09-13T14:59:57.390 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.98933645100000001 2024-09-13T14:59:57.390 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.390 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.390 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:57.390 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:17.918705+0000", 2024-09-13T14:59:57.390 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2299e-05 2024-09-13T14:59:57.390 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.391 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.391 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.391 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:18.957681+0000", 2024-09-13T14:59:57.391 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0389758419999999 2024-09-13T14:59:57.391 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.391 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.391 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:59:57.391 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:18.957841+0000", 2024-09-13T14:59:57.391 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016063199999999999 2024-09-13T14:59:57.391 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.391 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.391 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,3", 2024-09-13T14:59:57.391 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:18.958022+0000", 2024-09-13T14:59:57.391 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00018070299999999999 2024-09-13T14:59:57.392 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.392 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.392 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:59:57.392 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:18.959663+0000", 2024-09-13T14:59:57.392 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001641188 2024-09-13T14:59:57.392 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.392 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.392 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:57.392 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:18.963832+0000", 2024-09-13T14:59:57.392 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0041683049999999998 2024-09-13T14:59:57.392 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.392 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.392 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:57.393 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:18.965594+0000", 2024-09-13T14:59:57.393 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0017620800000000001 2024-09-13T14:59:57.393 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.393 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.393 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:59:57.393 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:18.965641+0000", 2024-09-13T14:59:57.393 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6956999999999998e-05 2024-09-13T14:59:57.393 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.393 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.393 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:57.393 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:18.965979+0000", 2024-09-13T14:59:57.393 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000338098 2024-09-13T14:59:57.393 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:57.393 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:57.394 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:57.394 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.394 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.394 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36738.0:9810 10.1 10:8c8c1ace:::benchmark_data_smithi102_377304_object9809:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e441)", 2024-09-13T14:59:57.394 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:55:16.928494+0000", 2024-09-13T14:59:57.394 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 280.32528277199998, 2024-09-13T14:59:57.394 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2640140639999999, 2024-09-13T14:59:57.394 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:57.394 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:57.394 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:57.394 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36738", 2024-09-13T14:59:57.394 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/2915890700", 2024-09-13T14:59:57.394 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 9810 2024-09-13T14:59:57.395 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.395 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:57.395 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.395 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:57.395 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928494+0000", 2024-09-13T14:59:57.395 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:57.395 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.395 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.395 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:57.395 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928494+0000", 2024-09-13T14:59:57.395 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:57.395 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.395 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.395 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:57.396 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928493+0000", 2024-09-13T14:59:57.396 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:59:57.396 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.396 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.396 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:57.396 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928527+0000", 2024-09-13T14:59:57.396 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.3821e-05 2024-09-13T14:59:57.396 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.396 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.396 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:57.396 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928533+0000", 2024-09-13T14:59:57.396 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0889999999999996e-06 2024-09-13T14:59:57.396 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.397 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.397 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:57.397 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928538+0000", 2024-09-13T14:59:57.397 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2989999999999998e-06 2024-09-13T14:59:57.397 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.397 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.397 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.397 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928660+0000", 2024-09-13T14:59:57.397 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000122012 2024-09-13T14:59:57.397 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.397 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.397 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:57.397 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928669+0000", 2024-09-13T14:59:57.397 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1339999999999993e-06 2024-09-13T14:59:57.398 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.398 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.398 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.398 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:17.919149+0000", 2024-09-13T14:59:57.398 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.99048049699999996 2024-09-13T14:59:57.398 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.398 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.398 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:57.398 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:17.919157+0000", 2024-09-13T14:59:57.398 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9629999999999998e-06 2024-09-13T14:59:57.398 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.398 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.398 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.398 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:19.178662+0000", 2024-09-13T14:59:57.399 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2595047319999999 2024-09-13T14:59:57.399 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.399 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.399 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:59:57.399 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:19.178786+0000", 2024-09-13T14:59:57.399 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000124051 2024-09-13T14:59:57.399 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.399 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.399 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:59:57.399 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:19.178891+0000", 2024-09-13T14:59:57.399 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000105219 2024-09-13T14:59:57.399 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.399 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.400 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:59:57.400 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:19.181396+0000", 2024-09-13T14:59:57.400 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0025047469999999999 2024-09-13T14:59:57.400 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.400 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.400 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:57.400 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:19.182895+0000", 2024-09-13T14:59:57.400 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001499055 2024-09-13T14:59:57.400 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.400 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.400 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:57.400 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:19.191955+0000", 2024-09-13T14:59:57.400 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0090602289999999995 2024-09-13T14:59:57.401 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.401 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.401 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:59:57.401 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:19.191991+0000", 2024-09-13T14:59:57.401 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6177999999999998e-05 2024-09-13T14:59:57.401 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.401 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.401 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:57.401 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:19.192508+0000", 2024-09-13T14:59:57.401 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00051648799999999995 2024-09-13T14:59:57.401 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:57.401 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:57.401 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:57.401 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.402 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.402 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36738.0:9811 10.1 10:8943192a:::benchmark_data_smithi102_377304_object9810:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e441)", 2024-09-13T14:59:57.402 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:55:16.928542+0000", 2024-09-13T14:59:57.402 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 280.32523465899999, 2024-09-13T14:59:57.402 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.264276787, 2024-09-13T14:59:57.402 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:57.402 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:57.402 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:57.402 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36738", 2024-09-13T14:59:57.402 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/2915890700", 2024-09-13T14:59:57.402 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 9811 2024-09-13T14:59:57.402 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.402 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:57.403 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.403 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:57.403 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928542+0000", 2024-09-13T14:59:57.403 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:57.403 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.403 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.403 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:57.403 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928542+0000", 2024-09-13T14:59:57.403 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:57.403 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.403 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.403 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:57.403 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928542+0000", 2024-09-13T14:59:57.403 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T14:59:57.404 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.404 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.404 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:57.404 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928568+0000", 2024-09-13T14:59:57.404 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5936999999999999e-05 2024-09-13T14:59:57.404 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.404 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.404 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:57.404 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928573+0000", 2024-09-13T14:59:57.404 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4210000000000003e-06 2024-09-13T14:59:57.404 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.404 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.404 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:57.405 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928577+0000", 2024-09-13T14:59:57.405 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.799e-06 2024-09-13T14:59:57.405 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.405 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.405 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.405 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928692+0000", 2024-09-13T14:59:57.405 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000115659 2024-09-13T14:59:57.405 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.405 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.405 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:57.405 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.928703+0000", 2024-09-13T14:59:57.405 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0062e-05 2024-09-13T14:59:57.405 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.406 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.406 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.406 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:17.919520+0000", 2024-09-13T14:59:57.406 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.99081732 2024-09-13T14:59:57.406 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.406 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.406 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:57.406 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:17.919531+0000", 2024-09-13T14:59:57.406 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1484999999999999e-05 2024-09-13T14:59:57.406 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.406 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.406 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.406 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:19.179311+0000", 2024-09-13T14:59:57.406 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2597794689999999 2024-09-13T14:59:57.407 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.407 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.407 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:59:57.407 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:19.179395+0000", 2024-09-13T14:59:57.407 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.4116000000000006e-05 2024-09-13T14:59:57.407 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.407 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.407 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:59:57.407 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:19.179496+0000", 2024-09-13T14:59:57.407 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000101041 2024-09-13T14:59:57.407 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.407 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.407 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:59:57.408 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:19.181404+0000", 2024-09-13T14:59:57.408 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001908332 2024-09-13T14:59:57.408 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.408 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.408 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:57.408 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:19.182937+0000", 2024-09-13T14:59:57.408 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001533005 2024-09-13T14:59:57.408 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.408 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.408 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:57.408 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:19.192548+0000", 2024-09-13T14:59:57.408 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0096108989999999991 2024-09-13T14:59:57.408 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.409 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.409 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:59:57.409 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:19.192581+0000", 2024-09-13T14:59:57.409 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.311e-05 2024-09-13T14:59:57.409 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.409 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.409 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:57.409 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:19.192819+0000", 2024-09-13T14:59:57.409 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00023754399999999999 2024-09-13T14:59:57.409 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:57.409 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:57.409 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:57.409 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.409 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.410 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.36738.0:9817 10.c 10:385a7ba4:::benchmark_data_smithi102_377304_object9816:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e441)", 2024-09-13T14:59:57.410 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:55:16.937249+0000", 2024-09-13T14:59:57.410 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 280.316528119, 2024-09-13T14:59:57.410 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0291470550000001, 2024-09-13T14:59:57.410 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T14:59:57.410 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:57.410 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-13T14:59:57.410 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.36738", 2024-09-13T14:59:57.410 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.102:0/2915890700", 2024-09-13T14:59:57.410 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 9817 2024-09-13T14:59:57.410 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.410 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T14:59:57.410 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.411 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T14:59:57.411 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.937249+0000", 2024-09-13T14:59:57.411 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:57.411 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.411 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.411 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T14:59:57.411 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.937249+0000", 2024-09-13T14:59:57.411 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T14:59:57.411 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.411 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.411 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T14:59:57.411 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.937247+0000", 2024-09-13T14:59:57.411 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:57.411 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.412 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.412 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T14:59:57.412 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.937301+0000", 2024-09-13T14:59:57.412 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3604999999999999e-05 2024-09-13T14:59:57.412 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.412 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.412 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T14:59:57.412 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.937312+0000", 2024-09-13T14:59:57.412 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1126e-05 2024-09-13T14:59:57.412 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.412 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.412 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T14:59:57.412 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.937317+0000", 2024-09-13T14:59:57.413 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5380000000000002e-06 2024-09-13T14:59:57.413 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.413 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.413 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.413 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.937352+0000", 2024-09-13T14:59:57.413 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5055000000000002e-05 2024-09-13T14:59:57.413 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.413 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.413 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:57.413 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:16.937360+0000", 2024-09-13T14:59:57.413 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8080000000000005e-06 2024-09-13T14:59:57.413 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.413 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.413 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.414 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:17.918729+0000", 2024-09-13T14:59:57.414 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.98136912700000001 2024-09-13T14:59:57.414 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.414 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.414 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for peered", 2024-09-13T14:59:57.414 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:17.918738+0000", 2024-09-13T14:59:57.414 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.4430000000000003e-06 2024-09-13T14:59:57.414 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.414 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.414 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T14:59:57.414 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:18.959065+0000", 2024-09-13T14:59:57.414 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0403277209999999 2024-09-13T14:59:57.414 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.414 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.415 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T14:59:57.415 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:18.959204+0000", 2024-09-13T14:59:57.415 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013865000000000001 2024-09-13T14:59:57.415 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.415 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.415 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for subops from 0,3", 2024-09-13T14:59:57.415 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:18.959318+0000", 2024-09-13T14:59:57.415 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000114129 2024-09-13T14:59:57.415 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.415 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.415 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op_commit", 2024-09-13T14:59:57.415 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:18.959805+0000", 2024-09-13T14:59:57.415 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00048650000000000001 2024-09-13T14:59:57.416 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.416 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.416 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:57.416 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:18.964080+0000", 2024-09-13T14:59:57.416 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0042753269999999998 2024-09-13T14:59:57.416 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.416 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.416 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:57.416 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:18.966033+0000", 2024-09-13T14:59:57.416 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0019524519999999999 2024-09-13T14:59:57.416 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.416 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.416 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T14:59:57.416 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:18.966088+0000", 2024-09-13T14:59:57.417 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5105999999999997e-05 2024-09-13T14:59:57.417 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T14:59:57.417 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T14:59:57.417 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T14:59:57.417 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:18.966396+0000", 2024-09-13T14:59:57.417 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000307972 2024-09-13T14:59:57.417 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:57.417 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:57.417 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:57.417 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T14:59:57.417 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T14:59:57.417 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T14:59:57.750 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:57 smithi102 ceph-mon[110183]: pgmap v1686: 103 pgs: 5 active+undersized+remapped, 98 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 9.2 KiB/s rd, 27 KiB/s wr, 6 op/s; 20 KiB/s, 3 objects/s recovering 2024-09-13T14:59:57.751 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:57 smithi102 ceph-mon[110183]: osdmap e499: 8 total, 8 up, 8 in 2024-09-13T14:59:57.751 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:57 smithi102 ceph-mon[115269]: pgmap v1686: 103 pgs: 5 active+undersized+remapped, 98 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 9.2 KiB/s rd, 27 KiB/s wr, 6 op/s; 20 KiB/s, 3 objects/s recovering 2024-09-13T14:59:57.751 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:57 smithi102 ceph-mon[115269]: osdmap e499: 8 total, 8 up, 8 in 2024-09-13T14:59:57.895 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:57 smithi195 ceph-mon[104262]: pgmap v1686: 103 pgs: 5 active+undersized+remapped, 98 active+clean; 4.6 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 9.2 KiB/s rd, 27 KiB/s wr, 6 op/s; 20 KiB/s, 3 objects/s recovering 2024-09-13T14:59:57.896 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:57 smithi195 ceph-mon[104262]: osdmap e499: 8 total, 8 up, 8 in 2024-09-13T14:59:58.187 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T14:59:58.389 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:59:58.508 INFO:teuthology.orchestra.run.smithi102.stderr:reweighted osd.0 to 1 (10000) 2024-09-13T14:59:58.777 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:58 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2575713238' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:59:58.885 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:58 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2575713238' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:59:58.885 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:58 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2575713238' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:59:59.310 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 0 1 2024-09-13T14:59:59.510 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:59.631 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:59 smithi102 ceph-mon[110183]: pgmap v1688: 103 pgs: 3 peering, 4 active+undersized+remapped, 96 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 8.5 KiB/s rd, 25 KiB/s wr, 6 op/s; 18 KiB/s, 2 keys/s, 6 objects/s recovering 2024-09-13T14:59:59.631 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:59 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2575713238' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T14:59:59.631 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:59 smithi102 ceph-mon[110183]: osdmap e500: 8 total, 8 up, 8 in 2024-09-13T14:59:59.631 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:59 smithi102 ceph-mon[110183]: 1.1a scrub starts 2024-09-13T14:59:59.631 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 14:59:59 smithi102 ceph-mon[110183]: 1.1a scrub ok 2024-09-13T14:59:59.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:59 smithi102 ceph-mon[115269]: pgmap v1688: 103 pgs: 3 peering, 4 active+undersized+remapped, 96 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 8.5 KiB/s rd, 25 KiB/s wr, 6 op/s; 18 KiB/s, 2 keys/s, 6 objects/s recovering 2024-09-13T14:59:59.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:59 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2575713238' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T14:59:59.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:59 smithi102 ceph-mon[115269]: osdmap e500: 8 total, 8 up, 8 in 2024-09-13T14:59:59.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:59 smithi102 ceph-mon[115269]: 1.1a scrub starts 2024-09-13T14:59:59.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 14:59:59 smithi102 ceph-mon[115269]: 1.1a scrub ok 2024-09-13T14:59:59.819 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:59 smithi195 ceph-mon[104262]: pgmap v1688: 103 pgs: 3 peering, 4 active+undersized+remapped, 96 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 8.5 KiB/s rd, 25 KiB/s wr, 6 op/s; 18 KiB/s, 2 keys/s, 6 objects/s recovering 2024-09-13T14:59:59.819 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:59 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2575713238' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T14:59:59.819 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:59 smithi195 ceph-mon[104262]: osdmap e500: 8 total, 8 up, 8 in 2024-09-13T14:59:59.819 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:59 smithi195 ceph-mon[104262]: 1.1a scrub starts 2024-09-13T14:59:59.819 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 14:59:59 smithi195 ceph-mon[104262]: 1.1a scrub ok 2024-09-13T15:00:00.070 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T15:00:00.070 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-13T15:00:00.070 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 0 2024-09-13T15:00:00.070 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T15:00:00.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:00 smithi102 ceph-mon[110183]: 1.e scrub starts 2024-09-13T15:00:00.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:00 smithi102 ceph-mon[110183]: 1.e scrub ok 2024-09-13T15:00:00.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:00 smithi102 ceph-mon[110183]: overall HEALTH_OK 2024-09-13T15:00:00.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:00 smithi102 ceph-mon[115269]: 1.e scrub starts 2024-09-13T15:00:00.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:00 smithi102 ceph-mon[115269]: 1.e scrub ok 2024-09-13T15:00:00.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:00 smithi102 ceph-mon[115269]: overall HEALTH_OK 2024-09-13T15:00:00.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:00 smithi195 ceph-mon[104262]: 1.e scrub starts 2024-09-13T15:00:00.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:00 smithi195 ceph-mon[104262]: 1.e scrub ok 2024-09-13T15:00:00.714 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:00 smithi195 ceph-mon[104262]: overall HEALTH_OK 2024-09-13T15:00:00.734 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T15:00:00.933 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:00:01.747 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:01 smithi195 ceph-mon[104262]: pgmap v1690: 103 pgs: 3 peering, 100 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 6.9 KiB/s rd, 21 KiB/s wr, 5 op/s; 15 KiB/s, 1 keys/s, 5 objects/s recovering 2024-09-13T15:00:01.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:01 smithi102 ceph-mon[110183]: pgmap v1690: 103 pgs: 3 peering, 100 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 6.9 KiB/s rd, 21 KiB/s wr, 5 op/s; 15 KiB/s, 1 keys/s, 5 objects/s recovering 2024-09-13T15:00:01.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:01 smithi102 ceph-mon[115269]: pgmap v1690: 103 pgs: 3 peering, 100 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 6.9 KiB/s rd, 21 KiB/s wr, 5 op/s; 15 KiB/s, 1 keys/s, 5 objects/s recovering 2024-09-13T15:00:02.522 INFO:teuthology.orchestra.run.smithi102.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-13T15:00:02.620 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:02 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2277724496' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-13T15:00:02.621 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:02 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-13T15:00:02.624 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T15:00:02.624 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-13T15:00:02.624 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-09-13T15:00:02.624 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-09-13T15:00:02.624 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T15:00:02.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:02 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2277724496' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-13T15:00:02.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:02 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-13T15:00:02.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:02 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2277724496' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-13T15:00:02.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:02 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-13T15:00:03.278 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T15:00:03.337 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-13T15:00:03.476 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:00:03.548 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:03.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:03 smithi102 ceph-mon[110183]: pgmap v1691: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 0 B/s, 1 keys/s, 2 objects/s recovering 2024-09-13T15:00:03.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:03 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-13T15:00:03.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:03 smithi102 ceph-mon[110183]: osdmap e501: 8 total, 8 up, 8 in 2024-09-13T15:00:03.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:03 smithi102 ceph-mon[115269]: pgmap v1691: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 0 B/s, 1 keys/s, 2 objects/s recovering 2024-09-13T15:00:03.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:03 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-13T15:00:03.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:03 smithi102 ceph-mon[115269]: osdmap e501: 8 total, 8 up, 8 in 2024-09-13T15:00:03.775 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:03 smithi195 ceph-mon[104262]: pgmap v1691: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 0 B/s, 1 keys/s, 2 objects/s recovering 2024-09-13T15:00:03.776 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:03 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-13T15:00:03.776 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:03 smithi195 ceph-mon[104262]: osdmap e501: 8 total, 8 up, 8 in 2024-09-13T15:00:04.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:04 smithi102 ceph-mon[110183]: osdmap e502: 8 total, 8 up, 8 in 2024-09-13T15:00:04.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:04 smithi102 ceph-mon[110183]: 1.6 scrub starts 2024-09-13T15:00:04.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:04 smithi102 ceph-mon[110183]: 1.6 scrub ok 2024-09-13T15:00:04.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:04 smithi102 ceph-mon[115269]: osdmap e502: 8 total, 8 up, 8 in 2024-09-13T15:00:04.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:04 smithi102 ceph-mon[115269]: 1.6 scrub starts 2024-09-13T15:00:04.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:04 smithi102 ceph-mon[115269]: 1.6 scrub ok 2024-09-13T15:00:04.702 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:04 smithi195 ceph-mon[104262]: osdmap e502: 8 total, 8 up, 8 in 2024-09-13T15:00:04.702 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:04 smithi195 ceph-mon[104262]: 1.6 scrub starts 2024-09-13T15:00:04.702 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:04 smithi195 ceph-mon[104262]: 1.6 scrub ok 2024-09-13T15:00:05.257 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-13T15:00:05.659 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-09-13T15:00:05.660 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-09-13T15:00:05.660 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-13T15:00:05.660 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.660 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDPGPush(9.13 392/391 [PushOp(9:cce1574f:::benchmark_data_smithi102_348563_object40483:head, version: 378'2457, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(9:cce1574f:::benchmark_data_smithi102_348563_object40483:head@378'2457, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T15:00:05.660 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:52:06.936874+0000", 2024-09-13T15:00:05.660 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 478.30869372299998, 2024-09-13T15:00:05.660 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.059475924999999999, 2024-09-13T15:00:05.660 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T15:00:05.660 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-13T15:00:05.660 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T15:00:05.660 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.661 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T15:00:05.661 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:06.936874+0000", 2024-09-13T15:00:05.661 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T15:00:05.661 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.661 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.661 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T15:00:05.661 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:06.936874+0000", 2024-09-13T15:00:05.661 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T15:00:05.661 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.661 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.661 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T15:00:05.661 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:06.936873+0000", 2024-09-13T15:00:05.661 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T15:00:05.662 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.662 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.662 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T15:00:05.662 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:06.936936+0000", 2024-09-13T15:00:05.662 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3254000000000001e-05 2024-09-13T15:00:05.662 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.662 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.662 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T15:00:05.662 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:06.936959+0000", 2024-09-13T15:00:05.662 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3076000000000001e-05 2024-09-13T15:00:05.662 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.662 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.662 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T15:00:05.663 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:06.936974+0000", 2024-09-13T15:00:05.663 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4769e-05 2024-09-13T15:00:05.663 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.663 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.663 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T15:00:05.663 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:06.994303+0000", 2024-09-13T15:00:05.663 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.057328955000000001 2024-09-13T15:00:05.663 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.663 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.663 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T15:00:05.663 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:06.994315+0000", 2024-09-13T15:00:05.663 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.201e-05 2024-09-13T15:00:05.663 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.664 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.664 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T15:00:05.664 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:06.996350+0000", 2024-09-13T15:00:05.664 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0020354599999999998 2024-09-13T15:00:05.664 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T15:00:05.664 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T15:00:05.664 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T15:00:05.664 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.664 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.664 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36528.0:95656 9.11 e392/386 9:892a0af5:::benchmark_data_smithi102_348563_object95655:head v 392'4898, mlcod=392'4896)", 2024-09-13T15:00:05.664 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:52:06.940813+0000", 2024-09-13T15:00:05.664 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 478.304754946, 2024-09-13T15:00:05.664 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.05923722, 2024-09-13T15:00:05.665 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T15:00:05.665 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:05.665 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T15:00:05.665 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.665 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T15:00:05.665 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:06.940813+0000", 2024-09-13T15:00:05.665 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T15:00:05.665 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.665 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.665 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T15:00:05.665 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:06.940813+0000", 2024-09-13T15:00:05.665 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T15:00:05.665 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.666 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.666 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T15:00:05.666 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:06.940812+0000", 2024-09-13T15:00:05.666 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T15:00:05.666 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.666 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.666 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T15:00:05.666 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:06.940861+0000", 2024-09-13T15:00:05.666 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8776999999999997e-05 2024-09-13T15:00:05.666 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.666 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.666 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T15:00:05.666 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:06.994165+0000", 2024-09-13T15:00:05.666 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.053304451000000003 2024-09-13T15:00:05.667 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.667 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.667 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T15:00:05.667 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:06.994181+0000", 2024-09-13T15:00:05.667 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6205000000000002e-05 2024-09-13T15:00:05.667 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.667 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.667 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T15:00:05.667 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:06.994231+0000", 2024-09-13T15:00:05.667 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9947999999999997e-05 2024-09-13T15:00:05.667 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.667 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.667 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T15:00:05.667 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:06.994254+0000", 2024-09-13T15:00:05.668 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2249e-05 2024-09-13T15:00:05.668 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.668 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.668 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T15:00:05.668 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:07.000018+0000", 2024-09-13T15:00:05.668 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.005764384 2024-09-13T15:00:05.668 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.668 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.668 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T15:00:05.668 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:07.000050+0000", 2024-09-13T15:00:05.668 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2515999999999998e-05 2024-09-13T15:00:05.668 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T15:00:05.668 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T15:00:05.668 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T15:00:05.669 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.669 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.669 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36528.0:95655 9.1 e392/381 9:8571fba6:::benchmark_data_smithi102_348563_object95654:head v 392'4899, mlcod=392'4897)", 2024-09-13T15:00:05.669 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:52:06.940990+0000", 2024-09-13T15:00:05.669 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 478.30457852799998, 2024-09-13T15:00:05.669 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.063504271000000001, 2024-09-13T15:00:05.669 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T15:00:05.669 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:05.669 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T15:00:05.669 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.669 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T15:00:05.669 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:06.940990+0000", 2024-09-13T15:00:05.669 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T15:00:05.669 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.670 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.670 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T15:00:05.670 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:06.940990+0000", 2024-09-13T15:00:05.670 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T15:00:05.670 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.670 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.670 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T15:00:05.670 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:06.940989+0000", 2024-09-13T15:00:05.670 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T15:00:05.670 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.670 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.670 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T15:00:05.670 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:06.941184+0000", 2024-09-13T15:00:05.670 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00019578 2024-09-13T15:00:05.671 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.671 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.671 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T15:00:05.671 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:06.994827+0000", 2024-09-13T15:00:05.671 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.053642626999999998 2024-09-13T15:00:05.671 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.671 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.671 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T15:00:05.671 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:06.994837+0000", 2024-09-13T15:00:05.671 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4299999999999995e-06 2024-09-13T15:00:05.671 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.671 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.671 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T15:00:05.671 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:06.995800+0000", 2024-09-13T15:00:05.672 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00096365299999999997 2024-09-13T15:00:05.672 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.672 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.672 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T15:00:05.672 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:06.995817+0000", 2024-09-13T15:00:05.672 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7218999999999999e-05 2024-09-13T15:00:05.672 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.672 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.672 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T15:00:05.672 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:07.004453+0000", 2024-09-13T15:00:05.672 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0086354520000000001 2024-09-13T15:00:05.672 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.672 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.672 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T15:00:05.673 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:52:07.004494+0000", 2024-09-13T15:00:05.673 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1103000000000003e-05 2024-09-13T15:00:05.673 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T15:00:05.673 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T15:00:05.673 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T15:00:05.673 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.673 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.673 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36528.0:188021 9.1 e415/381 9:804798bb:::benchmark_data_smithi102_348563_object66919:head v 415'7743, mlcod=415'7741)", 2024-09-13T15:00:05.673 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:53:41.828489+0000", 2024-09-13T15:00:05.673 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 383.41707953500003, 2024-09-13T15:00:05.673 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.20997781600000001, 2024-09-13T15:00:05.673 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T15:00:05.673 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:05.673 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T15:00:05.674 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.674 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T15:00:05.674 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:41.828489+0000", 2024-09-13T15:00:05.674 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T15:00:05.674 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.674 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.674 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T15:00:05.674 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:41.828489+0000", 2024-09-13T15:00:05.674 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T15:00:05.674 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.674 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.674 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T15:00:05.674 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:41.828488+0000", 2024-09-13T15:00:05.674 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T15:00:05.675 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.675 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.675 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T15:00:05.675 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:41.828498+0000", 2024-09-13T15:00:05.675 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0414999999999999e-05 2024-09-13T15:00:05.675 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.675 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.675 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T15:00:05.675 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:41.828512+0000", 2024-09-13T15:00:05.675 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3583e-05 2024-09-13T15:00:05.675 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.675 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.675 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T15:00:05.675 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:41.828519+0000", 2024-09-13T15:00:05.676 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3679999999999996e-06 2024-09-13T15:00:05.676 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.676 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.676 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T15:00:05.676 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:41.828579+0000", 2024-09-13T15:00:05.676 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0560999999999998e-05 2024-09-13T15:00:05.676 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.676 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.676 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T15:00:05.676 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:42.028489+0000", 2024-09-13T15:00:05.676 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.19990945600000001 2024-09-13T15:00:05.676 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.676 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.676 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T15:00:05.677 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:42.038410+0000", 2024-09-13T15:00:05.677 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0099208209999999998 2024-09-13T15:00:05.677 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.677 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.677 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T15:00:05.677 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:42.038466+0000", 2024-09-13T15:00:05.677 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.6771000000000002e-05 2024-09-13T15:00:05.677 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T15:00:05.677 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T15:00:05.677 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T15:00:05.677 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.677 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.677 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36528.0:188024 9.13 e415/396 9:cc7fd1e7:::benchmark_data_smithi102_348563_object66922:head v 415'7487, mlcod=415'7484)", 2024-09-13T15:00:05.678 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:53:41.830705+0000", 2024-09-13T15:00:05.678 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 383.414863309, 2024-09-13T15:00:05.678 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.21054535599999999, 2024-09-13T15:00:05.678 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T15:00:05.678 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:05.678 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T15:00:05.678 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.678 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T15:00:05.678 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:41.830705+0000", 2024-09-13T15:00:05.678 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T15:00:05.678 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.678 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.678 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T15:00:05.678 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:41.830705+0000", 2024-09-13T15:00:05.679 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T15:00:05.679 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.679 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.679 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T15:00:05.679 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:41.830704+0000", 2024-09-13T15:00:05.679 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T15:00:05.679 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.679 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.679 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T15:00:05.679 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:41.830713+0000", 2024-09-13T15:00:05.679 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.02e-06 2024-09-13T15:00:05.679 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.679 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.679 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T15:00:05.680 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:42.028380+0000", 2024-09-13T15:00:05.680 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.19766724199999999 2024-09-13T15:00:05.680 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.680 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.680 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T15:00:05.680 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:42.028396+0000", 2024-09-13T15:00:05.680 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6498999999999999e-05 2024-09-13T15:00:05.680 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.680 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.680 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T15:00:05.680 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:42.029221+0000", 2024-09-13T15:00:05.680 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00082415500000000003 2024-09-13T15:00:05.681 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.681 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.681 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T15:00:05.681 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:42.029239+0000", 2024-09-13T15:00:05.681 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8774999999999999e-05 2024-09-13T15:00:05.681 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.681 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.681 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T15:00:05.681 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:42.041023+0000", 2024-09-13T15:00:05.681 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.01178389 2024-09-13T15:00:05.681 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.681 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.681 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T15:00:05.681 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:42.041250+0000", 2024-09-13T15:00:05.682 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00022703600000000001 2024-09-13T15:00:05.682 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T15:00:05.682 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T15:00:05.682 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T15:00:05.682 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.682 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.682 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36528.0:188032 9.b e415/374 9:d64c5cf6:::benchmark_data_smithi102_348563_object66930:head v 415'11623, mlcod=415'11621)", 2024-09-13T15:00:05.682 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:53:41.837840+0000", 2024-09-13T15:00:05.682 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 383.40772849699999, 2024-09-13T15:00:05.682 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.210339263, 2024-09-13T15:00:05.682 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T15:00:05.682 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:05.682 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T15:00:05.682 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.683 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T15:00:05.683 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:41.837840+0000", 2024-09-13T15:00:05.683 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T15:00:05.683 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.683 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.683 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T15:00:05.683 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:41.837840+0000", 2024-09-13T15:00:05.683 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T15:00:05.683 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.683 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.683 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T15:00:05.683 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:41.837839+0000", 2024-09-13T15:00:05.683 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T15:00:05.683 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.684 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.684 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T15:00:05.684 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:41.837849+0000", 2024-09-13T15:00:05.684 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0169999999999999e-05 2024-09-13T15:00:05.684 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.684 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.684 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T15:00:05.684 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:42.028406+0000", 2024-09-13T15:00:05.684 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.190557227 2024-09-13T15:00:05.684 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.684 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.684 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T15:00:05.684 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:42.028417+0000", 2024-09-13T15:00:05.684 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1284000000000001e-05 2024-09-13T15:00:05.685 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.685 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.685 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T15:00:05.685 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:42.031718+0000", 2024-09-13T15:00:05.685 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0033011519999999999 2024-09-13T15:00:05.685 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.685 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.685 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T15:00:05.685 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:42.031740+0000", 2024-09-13T15:00:05.685 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1200999999999999e-05 2024-09-13T15:00:05.685 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.685 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.685 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T15:00:05.685 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:42.048103+0000", 2024-09-13T15:00:05.686 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016363773000000002 2024-09-13T15:00:05.686 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.686 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.686 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T15:00:05.686 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:42.048179+0000", 2024-09-13T15:00:05.686 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5655999999999995e-05 2024-09-13T15:00:05.686 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T15:00:05.686 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T15:00:05.686 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T15:00:05.686 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.686 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.686 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36528.0:188050 9.0 e415/381 9:03605f91:::benchmark_data_smithi102_348563_object66948:head v 415'7737, mlcod=415'7735)", 2024-09-13T15:00:05.686 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:53:41.856396+0000", 2024-09-13T15:00:05.686 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 383.38917230499999, 2024-09-13T15:00:05.687 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.17779988899999999, 2024-09-13T15:00:05.687 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T15:00:05.687 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:05.687 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T15:00:05.687 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.687 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T15:00:05.687 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:41.856396+0000", 2024-09-13T15:00:05.687 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T15:00:05.687 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.687 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.687 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T15:00:05.687 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:41.856396+0000", 2024-09-13T15:00:05.687 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T15:00:05.687 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.688 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.688 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T15:00:05.688 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:41.856395+0000", 2024-09-13T15:00:05.688 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T15:00:05.688 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.688 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.688 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T15:00:05.688 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:41.856403+0000", 2024-09-13T15:00:05.688 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.4330000000000004e-06 2024-09-13T15:00:05.688 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.688 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.688 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T15:00:05.688 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:42.028441+0000", 2024-09-13T15:00:05.689 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.17203776800000001 2024-09-13T15:00:05.689 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.689 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.689 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T15:00:05.689 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:42.028451+0000", 2024-09-13T15:00:05.689 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6989999999999998e-06 2024-09-13T15:00:05.689 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.689 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.689 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T15:00:05.689 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:42.029191+0000", 2024-09-13T15:00:05.689 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00074086200000000001 2024-09-13T15:00:05.689 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.689 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.689 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T15:00:05.690 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:42.029217+0000", 2024-09-13T15:00:05.690 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5358999999999999e-05 2024-09-13T15:00:05.690 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.690 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.690 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T15:00:05.690 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:42.034162+0000", 2024-09-13T15:00:05.690 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0049453759999999996 2024-09-13T15:00:05.690 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.690 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.690 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T15:00:05.690 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:53:42.034196+0000", 2024-09-13T15:00:05.690 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.3577000000000001e-05 2024-09-13T15:00:05.690 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T15:00:05.690 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T15:00:05.691 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T15:00:05.691 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.691 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.691 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36738.0:43668 10.e e460/447 10:7207ca7d:::benchmark_data_smithi102_377304_object43667:head v 460'2826, mlcod=460'2824)", 2024-09-13T15:00:05.691 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:55:45.045034+0000", 2024-09-13T15:00:05.691 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 260.20053427800002, 2024-09-13T15:00:05.691 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.104706725, 2024-09-13T15:00:05.691 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T15:00:05.691 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:05.691 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T15:00:05.691 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.691 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T15:00:05.691 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045034+0000", 2024-09-13T15:00:05.691 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T15:00:05.692 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.692 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.692 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T15:00:05.692 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045034+0000", 2024-09-13T15:00:05.692 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T15:00:05.692 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.692 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.692 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T15:00:05.692 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045033+0000", 2024-09-13T15:00:05.692 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T15:00:05.692 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.692 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.692 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T15:00:05.693 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045093+0000", 2024-09-13T15:00:05.693 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.9845e-05 2024-09-13T15:00:05.693 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.693 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.693 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T15:00:05.693 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045101+0000", 2024-09-13T15:00:05.693 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0730000000000005e-06 2024-09-13T15:00:05.693 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.693 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.693 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T15:00:05.693 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045106+0000", 2024-09-13T15:00:05.693 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3089999999999997e-06 2024-09-13T15:00:05.693 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.693 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.694 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T15:00:05.694 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045136+0000", 2024-09-13T15:00:05.694 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0877999999999998e-05 2024-09-13T15:00:05.694 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.694 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.694 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T15:00:05.694 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045149+0000", 2024-09-13T15:00:05.694 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2941e-05 2024-09-13T15:00:05.694 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.694 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.694 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T15:00:05.694 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.149711+0000", 2024-09-13T15:00:05.694 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.104562057 2024-09-13T15:00:05.694 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.694 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.695 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T15:00:05.695 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.149741+0000", 2024-09-13T15:00:05.695 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9220000000000001e-05 2024-09-13T15:00:05.695 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T15:00:05.695 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T15:00:05.695 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T15:00:05.695 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.695 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.695 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36738.0:43659 10.b e460/436 10:d26becbe:::benchmark_data_smithi102_377304_object43658:head v 460'2698, mlcod=460'2695)", 2024-09-13T15:00:05.695 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:55:45.045201+0000", 2024-09-13T15:00:05.695 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 260.20036680800001, 2024-09-13T15:00:05.695 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.10350997200000001, 2024-09-13T15:00:05.695 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T15:00:05.695 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:05.696 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T15:00:05.696 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.696 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T15:00:05.696 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045201+0000", 2024-09-13T15:00:05.696 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T15:00:05.696 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.696 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.696 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T15:00:05.696 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045201+0000", 2024-09-13T15:00:05.696 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T15:00:05.697 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.697 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.697 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T15:00:05.697 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045201+0000", 2024-09-13T15:00:05.697 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T15:00:05.697 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.697 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.697 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T15:00:05.697 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045242+0000", 2024-09-13T15:00:05.697 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1594999999999997e-05 2024-09-13T15:00:05.698 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.698 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.698 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T15:00:05.698 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045250+0000", 2024-09-13T15:00:05.698 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5310000000000004e-06 2024-09-13T15:00:05.698 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.698 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.698 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T15:00:05.698 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045254+0000", 2024-09-13T15:00:05.698 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.194e-06 2024-09-13T15:00:05.698 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.698 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.698 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T15:00:05.698 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045279+0000", 2024-09-13T15:00:05.699 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4769999999999998e-05 2024-09-13T15:00:05.699 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.699 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.699 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T15:00:05.699 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045291+0000", 2024-09-13T15:00:05.699 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2408999999999999e-05 2024-09-13T15:00:05.699 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.699 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.699 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T15:00:05.699 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.148688+0000", 2024-09-13T15:00:05.699 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.103396548 2024-09-13T15:00:05.699 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.699 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.699 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T15:00:05.700 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.148711+0000", 2024-09-13T15:00:05.700 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.374e-05 2024-09-13T15:00:05.700 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T15:00:05.700 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T15:00:05.700 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T15:00:05.700 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.700 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.700 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36738.0:43663 10.0 e460/448 10:0eb12d1f:::benchmark_data_smithi102_377304_object43662:head v 460'2781, mlcod=460'2779)", 2024-09-13T15:00:05.700 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:55:45.045654+0000", 2024-09-13T15:00:05.700 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 260.19991448899998, 2024-09-13T15:00:05.700 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.10277876499999999, 2024-09-13T15:00:05.700 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T15:00:05.700 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:05.701 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T15:00:05.701 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.701 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T15:00:05.701 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045654+0000", 2024-09-13T15:00:05.701 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T15:00:05.701 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.701 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.701 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T15:00:05.701 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045654+0000", 2024-09-13T15:00:05.701 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T15:00:05.701 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.701 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.701 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T15:00:05.701 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045653+0000", 2024-09-13T15:00:05.702 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T15:00:05.702 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.702 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.702 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T15:00:05.702 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045713+0000", 2024-09-13T15:00:05.702 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.9783e-05 2024-09-13T15:00:05.702 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.702 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.702 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T15:00:05.702 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045720+0000", 2024-09-13T15:00:05.702 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1459999999999999e-06 2024-09-13T15:00:05.702 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.702 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.703 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T15:00:05.703 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045723+0000", 2024-09-13T15:00:05.703 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.1930000000000002e-06 2024-09-13T15:00:05.703 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.703 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.703 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T15:00:05.703 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045752+0000", 2024-09-13T15:00:05.703 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8223999999999999e-05 2024-09-13T15:00:05.703 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.703 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.703 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T15:00:05.703 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045763+0000", 2024-09-13T15:00:05.703 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.181e-05 2024-09-13T15:00:05.703 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.703 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.704 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T15:00:05.704 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.148387+0000", 2024-09-13T15:00:05.704 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.102623805 2024-09-13T15:00:05.704 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.704 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.704 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T15:00:05.704 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.148432+0000", 2024-09-13T15:00:05.704 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5262000000000001e-05 2024-09-13T15:00:05.704 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T15:00:05.704 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T15:00:05.704 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T15:00:05.704 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.704 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.705 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36738.0:43667 10.a e460/436 10:51b79b08:::benchmark_data_smithi102_377304_object43666:head v 460'2804, mlcod=460'2802)", 2024-09-13T15:00:05.705 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:55:45.045813+0000", 2024-09-13T15:00:05.705 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 260.19975492700001, 2024-09-13T15:00:05.705 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.10402655199999999, 2024-09-13T15:00:05.705 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T15:00:05.705 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:05.705 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T15:00:05.705 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.705 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T15:00:05.705 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045813+0000", 2024-09-13T15:00:05.705 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T15:00:05.705 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.705 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.706 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T15:00:05.706 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045813+0000", 2024-09-13T15:00:05.706 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T15:00:05.706 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.706 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.706 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T15:00:05.706 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045813+0000", 2024-09-13T15:00:05.706 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T15:00:05.706 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.706 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.706 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T15:00:05.706 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045875+0000", 2024-09-13T15:00:05.706 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2755000000000005e-05 2024-09-13T15:00:05.707 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.707 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.707 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T15:00:05.707 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045886+0000", 2024-09-13T15:00:05.707 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0549e-05 2024-09-13T15:00:05.707 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.707 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.707 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T15:00:05.707 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045892+0000", 2024-09-13T15:00:05.707 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2589999999999997e-06 2024-09-13T15:00:05.707 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.707 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.707 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T15:00:05.707 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045931+0000", 2024-09-13T15:00:05.708 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8822999999999997e-05 2024-09-13T15:00:05.708 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.708 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.708 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T15:00:05.708 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.045947+0000", 2024-09-13T15:00:05.708 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6147e-05 2024-09-13T15:00:05.708 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.708 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.708 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T15:00:05.708 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.149817+0000", 2024-09-13T15:00:05.708 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.103869609 2024-09-13T15:00:05.708 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.708 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.709 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T15:00:05.709 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.149840+0000", 2024-09-13T15:00:05.709 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3149000000000001e-05 2024-09-13T15:00:05.709 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T15:00:05.709 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T15:00:05.709 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T15:00:05.709 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.709 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.709 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36738.0:43669 10.f e460/436 10:f48161ab:::benchmark_data_smithi102_377304_object43668:head v 460'2691, mlcod=460'2688)", 2024-09-13T15:00:05.709 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:55:45.046758+0000", 2024-09-13T15:00:05.709 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 260.19881036300001, 2024-09-13T15:00:05.709 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.103328541, 2024-09-13T15:00:05.709 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T15:00:05.709 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:05.710 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T15:00:05.710 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.710 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T15:00:05.710 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.046758+0000", 2024-09-13T15:00:05.710 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T15:00:05.710 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.710 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.710 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T15:00:05.710 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.046758+0000", 2024-09-13T15:00:05.710 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T15:00:05.710 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.710 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.710 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T15:00:05.710 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.046757+0000", 2024-09-13T15:00:05.711 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T15:00:05.711 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.711 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.711 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T15:00:05.711 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.046823+0000", 2024-09-13T15:00:05.711 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5493999999999993e-05 2024-09-13T15:00:05.711 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.711 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.711 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T15:00:05.711 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.148448+0000", 2024-09-13T15:00:05.711 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.101625698 2024-09-13T15:00:05.711 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.711 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.711 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T15:00:05.711 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.148455+0000", 2024-09-13T15:00:05.712 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0850000000000002e-06 2024-09-13T15:00:05.712 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.712 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.712 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T15:00:05.712 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.148496+0000", 2024-09-13T15:00:05.712 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1149999999999997e-05 2024-09-13T15:00:05.712 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.712 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.712 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T15:00:05.712 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.148514+0000", 2024-09-13T15:00:05.712 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8425000000000001e-05 2024-09-13T15:00:05.712 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.712 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.712 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T15:00:05.713 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.150063+0000", 2024-09-13T15:00:05.713 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001549337 2024-09-13T15:00:05.713 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.713 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.713 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T15:00:05.713 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.150086+0000", 2024-09-13T15:00:05.713 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2938e-05 2024-09-13T15:00:05.713 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T15:00:05.713 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T15:00:05.713 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T15:00:05.713 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.713 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.713 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36738.0:43666 10.b e460/436 10:d20c5cea:::benchmark_data_smithi102_377304_object43665:head v 460'2699, mlcod=460'2696)", 2024-09-13T15:00:05.713 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:55:45.047252+0000", 2024-09-13T15:00:05.714 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 260.19831651200002, 2024-09-13T15:00:05.714 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.104257054, 2024-09-13T15:00:05.714 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T15:00:05.714 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:05.714 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T15:00:05.714 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.714 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T15:00:05.714 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.047252+0000", 2024-09-13T15:00:05.714 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T15:00:05.714 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.714 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.714 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T15:00:05.714 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.047252+0000", 2024-09-13T15:00:05.714 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T15:00:05.715 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.715 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.715 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T15:00:05.715 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.047251+0000", 2024-09-13T15:00:05.715 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T15:00:05.715 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.715 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.715 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T15:00:05.715 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.047306+0000", 2024-09-13T15:00:05.715 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5194e-05 2024-09-13T15:00:05.715 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.715 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.715 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T15:00:05.715 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.149545+0000", 2024-09-13T15:00:05.716 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.10223884599999999 2024-09-13T15:00:05.716 INFO:teuthology.orchestra.run.smithi102.stderr:reweighted osd.1 to 1 (10000) 2024-09-13T15:00:05.717 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.717 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.717 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T15:00:05.717 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.149553+0000", 2024-09-13T15:00:05.717 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9060000000000002e-06 2024-09-13T15:00:05.717 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.717 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.717 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T15:00:05.717 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.149592+0000", 2024-09-13T15:00:05.717 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9258999999999998e-05 2024-09-13T15:00:05.718 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.718 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.718 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T15:00:05.718 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.149612+0000", 2024-09-13T15:00:05.718 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.925e-05 2024-09-13T15:00:05.718 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.718 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.718 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T15:00:05.718 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.151482+0000", 2024-09-13T15:00:05.718 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0018704520000000001 2024-09-13T15:00:05.718 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.718 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.718 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T15:00:05.719 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.151509+0000", 2024-09-13T15:00:05.719 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.6667000000000001e-05 2024-09-13T15:00:05.719 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T15:00:05.719 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T15:00:05.719 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T15:00:05.719 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.719 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.719 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36738.0:43670 10.e e460/447 10:7cbfdc04:::benchmark_data_smithi102_377304_object43669:head v 460'2827, mlcod=460'2824)", 2024-09-13T15:00:05.719 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:55:45.047844+0000", 2024-09-13T15:00:05.719 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 260.197724016, 2024-09-13T15:00:05.719 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.102086207, 2024-09-13T15:00:05.719 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T15:00:05.719 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:05.719 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T15:00:05.720 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.720 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T15:00:05.720 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.047844+0000", 2024-09-13T15:00:05.720 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T15:00:05.720 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.720 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.720 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T15:00:05.720 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.047844+0000", 2024-09-13T15:00:05.720 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T15:00:05.720 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.720 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.720 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T15:00:05.720 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.047844+0000", 2024-09-13T15:00:05.720 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999995 2024-09-13T15:00:05.721 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.721 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.721 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T15:00:05.721 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.047901+0000", 2024-09-13T15:00:05.721 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7136000000000003e-05 2024-09-13T15:00:05.721 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.721 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.721 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T15:00:05.721 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.148379+0000", 2024-09-13T15:00:05.721 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.100478291 2024-09-13T15:00:05.721 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.721 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.721 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T15:00:05.721 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.148389+0000", 2024-09-13T15:00:05.722 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.9699999999999994e-06 2024-09-13T15:00:05.722 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.722 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.722 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T15:00:05.722 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.148529+0000", 2024-09-13T15:00:05.722 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000139896 2024-09-13T15:00:05.722 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.722 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.722 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T15:00:05.722 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.148546+0000", 2024-09-13T15:00:05.722 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7204e-05 2024-09-13T15:00:05.722 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.722 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.722 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T15:00:05.723 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.149896+0000", 2024-09-13T15:00:05.723 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0013502379999999999 2024-09-13T15:00:05.723 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.723 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.723 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T15:00:05.723 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:55:45.149930+0000", 2024-09-13T15:00:05.723 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4066999999999998e-05 2024-09-13T15:00:05.723 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T15:00:05.723 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T15:00:05.723 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T15:00:05.723 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.723 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.723 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36738.0:145354 10.e e475/447 10:7108cbc4:::benchmark_data_smithi102_377304_object39512:head v 475'9225, mlcod=475'9223)", 2024-09-13T15:00:05.723 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:57:26.151218+0000", 2024-09-13T15:00:05.724 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 159.09435027699999, 2024-09-13T15:00:05.724 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.048923727, 2024-09-13T15:00:05.724 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T15:00:05.724 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:05.724 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T15:00:05.724 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.724 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T15:00:05.724 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.151218+0000", 2024-09-13T15:00:05.724 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T15:00:05.724 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.724 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.724 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T15:00:05.724 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.151218+0000", 2024-09-13T15:00:05.725 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T15:00:05.725 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.725 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.725 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T15:00:05.725 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.151217+0000", 2024-09-13T15:00:05.725 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T15:00:05.725 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.725 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.725 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T15:00:05.725 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.151225+0000", 2024-09-13T15:00:05.725 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0539999999999995e-06 2024-09-13T15:00:05.725 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.725 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.725 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T15:00:05.726 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.151237+0000", 2024-09-13T15:00:05.726 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1985e-05 2024-09-13T15:00:05.726 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.726 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.726 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T15:00:05.726 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.151245+0000", 2024-09-13T15:00:05.726 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8150000000000006e-06 2024-09-13T15:00:05.726 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.726 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.726 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T15:00:05.726 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.151289+0000", 2024-09-13T15:00:05.726 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3949999999999998e-05 2024-09-13T15:00:05.726 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.726 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.727 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T15:00:05.727 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.151308+0000", 2024-09-13T15:00:05.727 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8825999999999999e-05 2024-09-13T15:00:05.727 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.727 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.727 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T15:00:05.727 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.200101+0000", 2024-09-13T15:00:05.727 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.048793299999999998 2024-09-13T15:00:05.727 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.727 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.727 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T15:00:05.727 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.200142+0000", 2024-09-13T15:00:05.727 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0688000000000001e-05 2024-09-13T15:00:05.727 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T15:00:05.727 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T15:00:05.728 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T15:00:05.728 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.728 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.728 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36738.0:145355 10.a e475/462 10:58792695:::benchmark_data_smithi102_377304_object39513:head v 475'9346, mlcod=475'9343)", 2024-09-13T15:00:05.728 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:57:26.151597+0000", 2024-09-13T15:00:05.728 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 159.09397129600001, 2024-09-13T15:00:05.728 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.048505690999999997, 2024-09-13T15:00:05.728 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T15:00:05.728 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:05.728 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T15:00:05.728 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.728 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T15:00:05.728 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.151597+0000", 2024-09-13T15:00:05.729 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T15:00:05.729 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.729 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.729 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T15:00:05.729 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.151597+0000", 2024-09-13T15:00:05.729 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T15:00:05.729 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.729 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.729 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T15:00:05.729 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.151596+0000", 2024-09-13T15:00:05.729 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T15:00:05.729 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.729 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.729 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T15:00:05.730 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.151602+0000", 2024-09-13T15:00:05.730 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3180000000000003e-06 2024-09-13T15:00:05.730 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.730 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.730 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T15:00:05.730 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.193663+0000", 2024-09-13T15:00:05.730 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.042060814000000002 2024-09-13T15:00:05.730 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.730 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.730 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T15:00:05.730 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.193681+0000", 2024-09-13T15:00:05.730 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7869000000000001e-05 2024-09-13T15:00:05.730 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.730 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.731 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T15:00:05.731 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.193734+0000", 2024-09-13T15:00:05.731 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3084000000000002e-05 2024-09-13T15:00:05.731 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.731 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.731 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T15:00:05.731 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.194105+0000", 2024-09-13T15:00:05.731 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000370534 2024-09-13T15:00:05.731 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.731 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.731 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T15:00:05.731 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.200063+0000", 2024-09-13T15:00:05.731 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0059585419999999998 2024-09-13T15:00:05.731 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.732 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.732 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T15:00:05.732 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.200103+0000", 2024-09-13T15:00:05.732 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9269e-05 2024-09-13T15:00:05.732 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T15:00:05.732 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T15:00:05.732 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T15:00:05.732 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.732 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.732 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36738.0:145417 10.e e475/447 10:7361d1f4:::benchmark_data_smithi102_377304_object39575:head v 475'9228, mlcod=475'9225)", 2024-09-13T15:00:05.732 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:57:26.250079+0000", 2024-09-13T15:00:05.732 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 158.995489128, 2024-09-13T15:00:05.732 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.157585429, 2024-09-13T15:00:05.732 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T15:00:05.733 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:05.733 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T15:00:05.733 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.733 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T15:00:05.733 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.250079+0000", 2024-09-13T15:00:05.733 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T15:00:05.733 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.733 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.733 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T15:00:05.733 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.250079+0000", 2024-09-13T15:00:05.733 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T15:00:05.733 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.733 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.733 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T15:00:05.734 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.250078+0000", 2024-09-13T15:00:05.734 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T15:00:05.734 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.734 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.734 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T15:00:05.734 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.250085+0000", 2024-09-13T15:00:05.734 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3990000000000001e-06 2024-09-13T15:00:05.734 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.734 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.734 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T15:00:05.734 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.250097+0000", 2024-09-13T15:00:05.734 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2339e-05 2024-09-13T15:00:05.734 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.734 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.735 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T15:00:05.735 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.250102+0000", 2024-09-13T15:00:05.735 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2009999999999998e-06 2024-09-13T15:00:05.735 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.735 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.735 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T15:00:05.735 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.394130+0000", 2024-09-13T15:00:05.735 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.14402737900000001 2024-09-13T15:00:05.735 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.735 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.735 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T15:00:05.735 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.394159+0000", 2024-09-13T15:00:05.735 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9213999999999998e-05 2024-09-13T15:00:05.735 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.735 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.736 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T15:00:05.736 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.407589+0000", 2024-09-13T15:00:05.736 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013430121999999999 2024-09-13T15:00:05.736 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.736 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.736 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T15:00:05.736 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.407665+0000", 2024-09-13T15:00:05.736 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5642999999999994e-05 2024-09-13T15:00:05.736 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T15:00:05.736 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T15:00:05.736 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T15:00:05.736 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.736 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.736 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36738.0:145425 10.a e475/462 10:5debe653:::benchmark_data_smithi102_377304_object39583:head v 475'9349, mlcod=475'9347)", 2024-09-13T15:00:05.737 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:57:26.253715+0000", 2024-09-13T15:00:05.737 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 158.991852862, 2024-09-13T15:00:05.737 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.15074379800000001, 2024-09-13T15:00:05.737 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T15:00:05.737 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:05.737 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T15:00:05.737 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.737 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T15:00:05.737 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.253715+0000", 2024-09-13T15:00:05.737 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T15:00:05.737 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.737 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.737 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T15:00:05.737 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.253715+0000", 2024-09-13T15:00:05.738 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T15:00:05.738 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.738 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.738 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T15:00:05.738 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.253714+0000", 2024-09-13T15:00:05.738 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T15:00:05.738 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.738 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.738 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T15:00:05.738 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.253723+0000", 2024-09-13T15:00:05.738 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2640000000000008e-06 2024-09-13T15:00:05.738 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.738 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.738 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T15:00:05.739 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.394087+0000", 2024-09-13T15:00:05.739 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.14036406000000001 2024-09-13T15:00:05.739 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.739 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.739 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T15:00:05.739 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.394105+0000", 2024-09-13T15:00:05.739 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8471999999999999e-05 2024-09-13T15:00:05.739 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.739 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.739 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T15:00:05.739 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.394220+0000", 2024-09-13T15:00:05.739 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000115091 2024-09-13T15:00:05.739 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.739 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.740 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T15:00:05.740 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.394241+0000", 2024-09-13T15:00:05.740 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0735000000000001e-05 2024-09-13T15:00:05.740 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.740 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.740 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T15:00:05.740 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.402551+0000", 2024-09-13T15:00:05.740 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0083097919999999999 2024-09-13T15:00:05.740 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.740 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.740 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T15:00:05.740 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.404459+0000", 2024-09-13T15:00:05.740 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0019082890000000001 2024-09-13T15:00:05.740 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T15:00:05.741 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T15:00:05.741 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T15:00:05.741 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.741 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.741 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36738.0:145424 10.f e475/436 10:fdca46c7:::benchmark_data_smithi102_377304_object39582:head v 475'9044, mlcod=475'9042)", 2024-09-13T15:00:05.741 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:57:26.257606+0000", 2024-09-13T15:00:05.741 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 158.987962358, 2024-09-13T15:00:05.741 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.15080960600000001, 2024-09-13T15:00:05.741 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T15:00:05.741 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:05.741 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T15:00:05.741 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.741 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T15:00:05.741 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.257606+0000", 2024-09-13T15:00:05.742 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T15:00:05.742 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.742 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.742 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T15:00:05.742 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.257606+0000", 2024-09-13T15:00:05.742 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T15:00:05.742 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.742 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.742 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T15:00:05.742 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.257605+0000", 2024-09-13T15:00:05.742 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.999999 2024-09-13T15:00:05.742 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.742 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.742 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T15:00:05.742 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.257614+0000", 2024-09-13T15:00:05.743 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7790000000000001e-06 2024-09-13T15:00:05.743 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.743 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.743 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T15:00:05.743 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.394095+0000", 2024-09-13T15:00:05.743 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.136481562 2024-09-13T15:00:05.743 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.743 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.743 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T15:00:05.743 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.394126+0000", 2024-09-13T15:00:05.743 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0732000000000003e-05 2024-09-13T15:00:05.743 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.743 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.743 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T15:00:05.744 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.394861+0000", 2024-09-13T15:00:05.744 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00073477899999999997 2024-09-13T15:00:05.744 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.744 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.744 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T15:00:05.744 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.394882+0000", 2024-09-13T15:00:05.744 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1421999999999999e-05 2024-09-13T15:00:05.744 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.744 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.744 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T15:00:05.744 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.408393+0000", 2024-09-13T15:00:05.744 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013510787 2024-09-13T15:00:05.744 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.745 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.745 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T15:00:05.745 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.408415+0000", 2024-09-13T15:00:05.745 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2612e-05 2024-09-13T15:00:05.745 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T15:00:05.745 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T15:00:05.745 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T15:00:05.745 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.745 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.745 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_repop(client.36738.0:145446 10.e e475/447 10:7a686eeb:::benchmark_data_smithi102_377304_object39604:head v 475'9229, mlcod=475'9226)", 2024-09-13T15:00:05.745 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-13T14:57:26.281548+0000", 2024-09-13T15:00:05.745 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 158.96401994799999, 2024-09-13T15:00:05.745 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.12617741699999999, 2024-09-13T15:00:05.745 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-13T15:00:05.746 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:05.746 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-13T15:00:05.746 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.746 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-13T15:00:05.746 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.281548+0000", 2024-09-13T15:00:05.746 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T15:00:05.746 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.746 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.746 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-13T15:00:05.746 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.281548+0000", 2024-09-13T15:00:05.746 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-13T15:00:05.746 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.746 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.746 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-13T15:00:05.746 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.281547+0000", 2024-09-13T15:00:05.747 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4294967295.9999986 2024-09-13T15:00:05.747 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.747 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.747 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-13T15:00:05.747 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.281557+0000", 2024-09-13T15:00:05.747 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0023e-05 2024-09-13T15:00:05.747 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.747 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.747 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-13T15:00:05.747 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.394349+0000", 2024-09-13T15:00:05.747 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.11279236300000001 2024-09-13T15:00:05.747 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.747 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.747 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-13T15:00:05.748 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.394358+0000", 2024-09-13T15:00:05.748 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7080000000000003e-06 2024-09-13T15:00:05.748 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.748 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.748 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-13T15:00:05.748 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.397544+0000", 2024-09-13T15:00:05.748 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003186089 2024-09-13T15:00:05.748 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.748 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.748 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-13T15:00:05.748 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.397562+0000", 2024-09-13T15:00:05.748 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7972999999999999e-05 2024-09-13T15:00:05.748 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.748 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.749 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-13T15:00:05.749 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.407669+0000", 2024-09-13T15:00:05.749 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010106904999999999 2024-09-13T15:00:05.749 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-13T15:00:05.749 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-13T15:00:05.749 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-13T15:00:05.749 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-13T14:57:26.407726+0000", 2024-09-13T15:00:05.752 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.6839999999999998e-05 2024-09-13T15:00:05.752 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T15:00:05.752 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T15:00:05.752 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T15:00:05.752 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-13T15:00:05.752 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-13T15:00:05.752 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-13T15:00:05.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:05 smithi102 ceph-mon[110183]: pgmap v1694: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:05.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:05 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1633669437' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:00:05.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:05 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:00:05.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:05 smithi102 ceph-mon[115269]: pgmap v1694: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:05.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:05 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1633669437' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:00:05.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:05 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:00:05.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:05 smithi195 ceph-mon[104262]: pgmap v1694: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:05.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:05 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1633669437' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:00:05.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:05 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:00:06.442 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 1 1 2024-09-13T15:00:06.646 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:06.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:06 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T15:00:06.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:06 smithi102 ceph-mon[110183]: osdmap e503: 8 total, 8 up, 8 in 2024-09-13T15:00:06.669 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 15:00:06 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:15:00:06] "GET /metrics HTTP/1.1" 200 34152 "" "Prometheus/2.51.0" 2024-09-13T15:00:06.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:06 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T15:00:06.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:06 smithi102 ceph-mon[115269]: osdmap e503: 8 total, 8 up, 8 in 2024-09-13T15:00:06.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:06 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T15:00:06.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:06 smithi195 ceph-mon[104262]: osdmap e503: 8 total, 8 up, 8 in 2024-09-13T15:00:07.745 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:07 smithi102 ceph-mon[115269]: pgmap v1696: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:07.745 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:07 smithi102 ceph-mon[110183]: pgmap v1696: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:08.000 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:07 smithi195 ceph-mon[104262]: pgmap v1696: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:08.553 INFO:teuthology.orchestra.run.smithi102.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-13T15:00:08.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:08 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3791883443' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:00:08.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:08 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:00:08.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:08 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3791883443' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:00:08.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:08 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:00:08.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:08 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3791883443' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:00:08.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:08 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:00:09.338 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-13T15:00:09.555 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:09.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:09 smithi102 ceph-mon[110183]: pgmap v1697: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 96 B/s, 1 objects/s recovering 2024-09-13T15:00:09.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:09 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-13T15:00:09.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:09 smithi102 ceph-mon[110183]: osdmap e504: 8 total, 8 up, 8 in 2024-09-13T15:00:09.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:09 smithi102 ceph-mon[115269]: pgmap v1697: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 96 B/s, 1 objects/s recovering 2024-09-13T15:00:09.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:09 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-13T15:00:09.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:09 smithi102 ceph-mon[115269]: osdmap e504: 8 total, 8 up, 8 in 2024-09-13T15:00:09.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:09 smithi195 ceph-mon[104262]: pgmap v1697: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 96 B/s, 1 objects/s recovering 2024-09-13T15:00:09.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:09 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-13T15:00:09.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:09 smithi195 ceph-mon[104262]: osdmap e504: 8 total, 8 up, 8 in 2024-09-13T15:00:10.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:10 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:00:10.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:10 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:00:10.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:10 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:00:11.565 INFO:teuthology.orchestra.run.smithi102.stderr:reweighted osd.2 to 1 (10000) 2024-09-13T15:00:11.885 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:11 smithi102 ceph-mon[110183]: pgmap v1699: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 91 B/s, 1 objects/s recovering 2024-09-13T15:00:11.886 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:11 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:00:11.886 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:11 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3074262215' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:00:11.886 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:11 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:00:11.886 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:11 smithi102 ceph-mon[115269]: pgmap v1699: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 91 B/s, 1 objects/s recovering 2024-09-13T15:00:11.886 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:11 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:00:11.886 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:11 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3074262215' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:00:11.886 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:11 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:00:11.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:11 smithi195 ceph-mon[104262]: pgmap v1699: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 91 B/s, 1 objects/s recovering 2024-09-13T15:00:11.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:11 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:00:11.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:11 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3074262215' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:00:11.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:11 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:00:12.327 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 2 1 2024-09-13T15:00:12.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:12 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T15:00:12.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:12 smithi102 ceph-mon[110183]: osdmap e505: 8 total, 8 up, 8 in 2024-09-13T15:00:12.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:12 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T15:00:12.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:12 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T15:00:12.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:12 smithi102 ceph-mon[110183]: pgmap v1701: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 91 B/s, 1 objects/s recovering 2024-09-13T15:00:12.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:12 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T15:00:12.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:12 smithi102 ceph-mon[115269]: osdmap e505: 8 total, 8 up, 8 in 2024-09-13T15:00:12.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:12 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T15:00:12.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:12 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T15:00:12.920 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:12 smithi102 ceph-mon[115269]: pgmap v1701: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 91 B/s, 1 objects/s recovering 2024-09-13T15:00:12.992 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:12 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T15:00:12.992 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:12 smithi195 ceph-mon[104262]: osdmap e505: 8 total, 8 up, 8 in 2024-09-13T15:00:12.992 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:12 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T15:00:12.992 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:12 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T15:00:12.992 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:12 smithi195 ceph-mon[104262]: pgmap v1701: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 91 B/s, 1 objects/s recovering 2024-09-13T15:00:13.083 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:15.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:14 smithi102 ceph-mon[110183]: pgmap v1702: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 72 B/s, 0 objects/s recovering 2024-09-13T15:00:15.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:14 smithi102 ceph-mon[115269]: pgmap v1702: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 72 B/s, 0 objects/s recovering 2024-09-13T15:00:15.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:14 smithi195 ceph-mon[104262]: pgmap v1702: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 72 B/s, 0 objects/s recovering 2024-09-13T15:00:15.923 INFO:teuthology.orchestra.run.smithi102.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-13T15:00:16.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:15 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1484722563' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:00:16.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:15 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:00:16.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:15 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T15:00:16.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:15 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T15:00:16.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:15 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1484722563' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:00:16.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:15 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:00:16.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:15 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T15:00:16.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:15 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T15:00:16.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:15 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1484722563' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:00:16.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:15 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:00:16.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:15 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T15:00:16.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:15 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T15:00:16.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 15:00:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:15:00:16] "GET /metrics HTTP/1.1" 200 34152 "" "Prometheus/2.51.0" 2024-09-13T15:00:16.884 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-13T15:00:17.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:16 smithi102 ceph-mon[110183]: pgmap v1703: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:17.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:16 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-13T15:00:17.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:16 smithi102 ceph-mon[110183]: osdmap e506: 8 total, 8 up, 8 in 2024-09-13T15:00:17.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:16 smithi102 ceph-mon[115269]: pgmap v1703: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:17.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:16 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-13T15:00:17.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:16 smithi102 ceph-mon[115269]: osdmap e506: 8 total, 8 up, 8 in 2024-09-13T15:00:17.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:16 smithi195 ceph-mon[104262]: pgmap v1703: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:17.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:16 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-13T15:00:17.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:16 smithi195 ceph-mon[104262]: osdmap e506: 8 total, 8 up, 8 in 2024-09-13T15:00:17.323 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:19.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:18 smithi195 ceph-mon[104262]: pgmap v1705: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:19.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:18 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T15:00:19.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:18 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T15:00:19.323 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:18 smithi102 ceph-mon[115269]: pgmap v1705: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:19.324 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:18 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T15:00:19.324 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:18 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T15:00:19.324 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:18 smithi102 ceph-mon[110183]: pgmap v1705: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:19.324 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:18 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T15:00:19.324 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:18 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T15:00:20.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:19 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:00:20.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:19 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:00:20.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:19 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T15:00:20.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:19 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:00:20.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:19 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:00:20.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:19 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T15:00:20.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:19 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:00:20.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:19 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:00:20.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:19 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T15:00:20.932 INFO:teuthology.orchestra.run.smithi102.stderr:reweighted osd.3 to 1 (10000) 2024-09-13T15:00:21.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:20 smithi195 ceph-mon[104262]: pgmap v1706: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:21.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:20 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2119784875' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:00:21.315 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:20 smithi102 ceph-mon[110183]: pgmap v1706: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:21.315 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:20 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2119784875' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:00:21.315 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:20 smithi102 ceph-mon[115269]: pgmap v1706: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:21.316 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:20 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2119784875' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:00:21.738 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 3 1 2024-09-13T15:00:21.942 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:22.067 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:21 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2119784875' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T15:00:22.067 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:21 smithi102 ceph-mon[115269]: osdmap e507: 8 total, 8 up, 8 in 2024-09-13T15:00:22.067 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:21 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2119784875' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T15:00:22.067 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:21 smithi102 ceph-mon[110183]: osdmap e507: 8 total, 8 up, 8 in 2024-09-13T15:00:22.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:21 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2119784875' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T15:00:22.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:21 smithi195 ceph-mon[104262]: osdmap e507: 8 total, 8 up, 8 in 2024-09-13T15:00:23.116 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:22 smithi102 ceph-mon[115269]: pgmap v1708: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:23.116 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:22 smithi102 ceph-mon[110183]: pgmap v1708: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:23.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:22 smithi195 ceph-mon[104262]: pgmap v1708: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:23.951 INFO:teuthology.orchestra.run.smithi102.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-13T15:00:24.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:23 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3006720944' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:00:24.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:23 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:00:24.265 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:23 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3006720944' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:00:24.265 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:23 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:00:24.265 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:23 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3006720944' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:00:24.265 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:23 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:00:24.595 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-13T15:00:24.795 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:25.167 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:24 smithi102 ceph-mon[115269]: pgmap v1709: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:25.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:24 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-13T15:00:25.168 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:24 smithi102 ceph-mon[115269]: osdmap e508: 8 total, 8 up, 8 in 2024-09-13T15:00:25.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:24 smithi102 ceph-mon[110183]: pgmap v1709: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:25.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:24 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-13T15:00:25.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:24 smithi102 ceph-mon[110183]: osdmap e508: 8 total, 8 up, 8 in 2024-09-13T15:00:25.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:24 smithi195 ceph-mon[104262]: pgmap v1709: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:25.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:24 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-13T15:00:25.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:24 smithi195 ceph-mon[104262]: osdmap e508: 8 total, 8 up, 8 in 2024-09-13T15:00:26.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:25 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:00:26.297 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:25 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:00:26.297 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:25 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:00:26.668 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 15:00:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:15:00:26] "GET /metrics HTTP/1.1" 200 34145 "" "Prometheus/2.51.0" 2024-09-13T15:00:26.962 INFO:teuthology.orchestra.run.smithi102.stderr:reweighted osd.4 to 1 (10000) 2024-09-13T15:00:27.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:26 smithi195 ceph-mon[104262]: pgmap v1711: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:27.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:26 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2031909382' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:00:27.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:26 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:00:27.264 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:26 smithi102 ceph-mon[110183]: pgmap v1711: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:27.264 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:26 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2031909382' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:00:27.264 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:26 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:00:27.265 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:26 smithi102 ceph-mon[115269]: pgmap v1711: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:27.265 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:26 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2031909382' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:00:27.265 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:26 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:00:27.695 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 4 1 2024-09-13T15:00:27.894 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:28.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:27 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T15:00:28.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:27 smithi195 ceph-mon[104262]: osdmap e509: 8 total, 8 up, 8 in 2024-09-13T15:00:28.294 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:27 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T15:00:28.294 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:27 smithi102 ceph-mon[110183]: osdmap e509: 8 total, 8 up, 8 in 2024-09-13T15:00:28.294 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:27 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T15:00:28.294 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:27 smithi102 ceph-mon[115269]: osdmap e509: 8 total, 8 up, 8 in 2024-09-13T15:00:29.166 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:28 smithi102 ceph-mon[110183]: pgmap v1713: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:29.166 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:28 smithi102 ceph-mon[115269]: pgmap v1713: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:29.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:28 smithi195 ceph-mon[104262]: pgmap v1713: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:29.969 INFO:teuthology.orchestra.run.smithi102.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-13T15:00:30.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:29 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2046211618' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:00:30.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:29 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:00:30.289 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:29 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2046211618' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:00:30.289 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:29 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:00:30.290 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:29 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2046211618' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:00:30.290 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:29 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:00:30.739 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-13T15:00:30.940 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:31.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:30 smithi195 ceph-mon[104262]: pgmap v1714: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:31.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:30 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-13T15:00:31.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:30 smithi195 ceph-mon[104262]: osdmap e510: 8 total, 8 up, 8 in 2024-09-13T15:00:31.302 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:30 smithi102 ceph-mon[110183]: pgmap v1714: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:31.303 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:30 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-13T15:00:31.303 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:30 smithi102 ceph-mon[110183]: osdmap e510: 8 total, 8 up, 8 in 2024-09-13T15:00:31.303 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:30 smithi102 ceph-mon[115269]: pgmap v1714: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:31.303 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:30 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-13T15:00:31.303 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:30 smithi102 ceph-mon[115269]: osdmap e510: 8 total, 8 up, 8 in 2024-09-13T15:00:32.975 INFO:teuthology.orchestra.run.smithi102.stderr:reweighted osd.5 to 1 (10000) 2024-09-13T15:00:33.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:32 smithi195 ceph-mon[104262]: pgmap v1716: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:33.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:32 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/385987216' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:00:33.324 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:32 smithi102 ceph-mon[110183]: pgmap v1716: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:33.324 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:32 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/385987216' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:00:33.324 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:32 smithi102 ceph-mon[115269]: pgmap v1716: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:33.324 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:32 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/385987216' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:00:33.679 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 5 1 2024-09-13T15:00:33.877 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:34.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:33 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/385987216' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T15:00:34.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:33 smithi195 ceph-mon[104262]: osdmap e511: 8 total, 8 up, 8 in 2024-09-13T15:00:34.311 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:33 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/385987216' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T15:00:34.311 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:33 smithi102 ceph-mon[110183]: osdmap e511: 8 total, 8 up, 8 in 2024-09-13T15:00:34.311 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:33 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/385987216' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T15:00:34.311 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:33 smithi102 ceph-mon[115269]: osdmap e511: 8 total, 8 up, 8 in 2024-09-13T15:00:35.132 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:34 smithi102 ceph-mon[110183]: pgmap v1718: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:35.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:34 smithi102 ceph-mon[115269]: pgmap v1718: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:35.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:34 smithi195 ceph-mon[104262]: pgmap v1718: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:36.001 INFO:teuthology.orchestra.run.smithi102.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-13T15:00:36.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:35 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3179469565' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:00:36.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:35 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:00:36.298 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:35 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3179469565' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:00:36.298 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:35 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:00:36.298 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:35 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3179469565' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:00:36.299 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:35 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:00:36.669 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 15:00:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:15:00:36] "GET /metrics HTTP/1.1" 200 34145 "" "Prometheus/2.51.0" 2024-09-13T15:00:36.706 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-13T15:00:36.905 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:37.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:36 smithi195 ceph-mon[104262]: pgmap v1719: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:37.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:36 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-13T15:00:37.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:36 smithi195 ceph-mon[104262]: osdmap e512: 8 total, 8 up, 8 in 2024-09-13T15:00:37.336 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:36 smithi102 ceph-mon[110183]: pgmap v1719: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:37.336 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:36 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-13T15:00:37.336 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:36 smithi102 ceph-mon[110183]: osdmap e512: 8 total, 8 up, 8 in 2024-09-13T15:00:37.337 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:36 smithi102 ceph-mon[115269]: pgmap v1719: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:37.337 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:36 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-13T15:00:37.337 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:36 smithi102 ceph-mon[115269]: osdmap e512: 8 total, 8 up, 8 in 2024-09-13T15:00:39.002 INFO:teuthology.orchestra.run.smithi102.stderr:reweighted osd.7 to 1 (10000) 2024-09-13T15:00:39.298 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:39 smithi102 ceph-mon[110183]: pgmap v1721: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:39.298 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:39 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1628896795' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:00:39.298 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:39 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:00:39.299 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:39 smithi102 ceph-mon[115269]: pgmap v1721: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:39.299 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:39 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1628896795' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:00:39.299 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:39 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:00:39.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:39 smithi195 ceph-mon[104262]: pgmap v1721: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:39.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:39 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1628896795' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:00:39.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:39 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:00:39.656 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 7 1 2024-09-13T15:00:39.853 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:40.244 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:40 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T15:00:40.244 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:40 smithi102 ceph-mon[110183]: osdmap e513: 8 total, 8 up, 8 in 2024-09-13T15:00:40.245 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:40 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T15:00:40.245 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:40 smithi102 ceph-mon[115269]: osdmap e513: 8 total, 8 up, 8 in 2024-09-13T15:00:40.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:40 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T15:00:40.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:40 smithi195 ceph-mon[104262]: osdmap e513: 8 total, 8 up, 8 in 2024-09-13T15:00:41.296 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:41 smithi102 ceph-mon[110183]: pgmap v1723: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:41.297 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:41 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:00:41.297 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:41 smithi102 ceph-mon[115269]: pgmap v1723: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:41.297 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:41 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:00:41.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:41 smithi195 ceph-mon[104262]: pgmap v1723: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:41.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:41 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:00:42.021 INFO:teuthology.orchestra.run.smithi102.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-13T15:00:42.364 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:42 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3930738065' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:00:42.364 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:42 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3930738065' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:00:42.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:42 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3930738065' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:00:42.680 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-13T15:00:42.880 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:43.293 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:43 smithi102 ceph-mon[110183]: pgmap v1724: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:43.294 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:43 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3930738065' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-13T15:00:43.294 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:43 smithi102 ceph-mon[110183]: osdmap e514: 8 total, 8 up, 8 in 2024-09-13T15:00:43.294 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:43 smithi102 ceph-mon[115269]: pgmap v1724: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:43.294 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:43 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3930738065' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-13T15:00:43.294 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:43 smithi102 ceph-mon[115269]: osdmap e514: 8 total, 8 up, 8 in 2024-09-13T15:00:43.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:43 smithi195 ceph-mon[104262]: pgmap v1724: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:43.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:43 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3930738065' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-13T15:00:43.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:43 smithi195 ceph-mon[104262]: osdmap e514: 8 total, 8 up, 8 in 2024-09-13T15:00:45.027 INFO:teuthology.orchestra.run.smithi102.stderr:reweighted osd.6 to 1 (10000) 2024-09-13T15:00:45.339 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:45 smithi102 ceph-mon[110183]: pgmap v1726: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:45.339 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:45 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1235762917' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T15:00:45.340 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:45 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T15:00:45.340 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:45 smithi102 ceph-mon[115269]: pgmap v1726: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:45.340 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:45 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1235762917' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T15:00:45.340 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:45 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T15:00:45.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:45 smithi195 ceph-mon[104262]: pgmap v1726: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:45.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:45 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1235762917' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T15:00:45.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:45 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T15:00:45.662 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 6 1 2024-09-13T15:00:45.862 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:46.335 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:46 smithi102 ceph-mon[110183]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T15:00:46.335 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:46 smithi102 ceph-mon[110183]: osdmap e515: 8 total, 8 up, 8 in 2024-09-13T15:00:46.335 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:46 smithi102 ceph-mon[115269]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T15:00:46.335 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:46 smithi102 ceph-mon[115269]: osdmap e515: 8 total, 8 up, 8 in 2024-09-13T15:00:46.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:46 smithi195 ceph-mon[104262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T15:00:46.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:46 smithi195 ceph-mon[104262]: osdmap e515: 8 total, 8 up, 8 in 2024-09-13T15:00:46.669 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 15:00:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:15:00:46] "GET /metrics HTTP/1.1" 200 34153 "" "Prometheus/2.51.0" 2024-09-13T15:00:47.408 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:47 smithi102 ceph-mon[110183]: pgmap v1728: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:47.409 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:47 smithi102 ceph-mon[115269]: pgmap v1728: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:47.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:47 smithi195 ceph-mon[104262]: pgmap v1728: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:48.042 INFO:teuthology.orchestra.run.smithi102.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-13T15:00:48.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:48 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2618960362' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-13T15:00:48.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:48 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2618960362' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-13T15:00:48.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:48 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2618960362' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-13T15:00:48.770 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-13T15:00:48.770 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-13T15:00:48.771 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-13T15:00:48.771 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-13T15:00:48.771 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-13T15:00:48.771 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-13T15:00:48.972 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:49.350 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:49 smithi102 ceph-mon[110183]: pgmap v1729: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:49.351 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:49 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2618960362' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-13T15:00:49.351 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:49 smithi102 ceph-mon[110183]: osdmap e516: 8 total, 8 up, 8 in 2024-09-13T15:00:49.351 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:49 smithi102 ceph-mon[115269]: pgmap v1729: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:49.351 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:49 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2618960362' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-13T15:00:49.351 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:49 smithi102 ceph-mon[115269]: osdmap e516: 8 total, 8 up, 8 in 2024-09-13T15:00:49.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:49 smithi195 ceph-mon[104262]: pgmap v1729: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:49.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:49 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2618960362' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-13T15:00:49.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:49 smithi195 ceph-mon[104262]: osdmap e516: 8 total, 8 up, 8 in 2024-09-13T15:00:50.063 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:50 smithi102 ceph-mon[110183]: osdmap e517: 8 total, 8 up, 8 in 2024-09-13T15:00:50.063 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:50 smithi102 ceph-mon[115269]: osdmap e517: 8 total, 8 up, 8 in 2024-09-13T15:00:50.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:50 smithi195 ceph-mon[104262]: osdmap e517: 8 total, 8 up, 8 in 2024-09-13T15:00:50.718 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T15:00:50.718 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":517,"fsid":"08cee502-71d9-11ef-bceb-c7b262605968","created":"2024-09-13T14:04:48.179879+0000","modified":"2024-09-13T15:00:49.041594+0000","last_up_change":"2024-09-13T14:37:47.105103+0000","last_in_change":"2024-09-13T14:59:52.482950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T14:07:41.927240+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"474","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"471","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.059999942779541,"score_stable":2.059999942779541,"optimal_score":1,"raw_score_acting":2.059999942779541,"raw_score_stable":2.059999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T14:10:10.981300+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":85,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-13T14:11:27.935818+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-13T14:11:28.454262+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b43b49e0-1cf0-47b8-aa85-37108f321220","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":90,"up_from":94,"up_thru":501,"down_at":91,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":3285732759},{"type":"v1","addr":"172.21.15.102:6803","nonce":3285732759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":3285732759},{"type":"v1","addr":"172.21.15.102:6805","nonce":3285732759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3285732759},{"type":"v1","addr":"172.21.15.102:6809","nonce":3285732759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":3285732759},{"type":"v1","addr":"172.21.15.102:6807","nonce":3285732759}]},"public_addr":"172.21.15.102:6803/3285732759","cluster_addr":"172.21.15.102:6805/3285732759","heartbeat_back_addr":"172.21.15.102:6809/3285732759","heartbeat_front_addr":"172.21.15.102:6807/3285732759","state":["exists","up"]},{"osd":1,"uuid":"27e10d5e-c471-4132-9c73-83c5708c3b08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":95,"up_from":99,"up_thru":498,"down_at":96,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":2240964461},{"type":"v1","addr":"172.21.15.102:6811","nonce":2240964461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":2240964461},{"type":"v1","addr":"172.21.15.102:6813","nonce":2240964461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":2240964461},{"type":"v1","addr":"172.21.15.102:6817","nonce":2240964461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":2240964461},{"type":"v1","addr":"172.21.15.102:6815","nonce":2240964461}]},"public_addr":"172.21.15.102:6811/2240964461","cluster_addr":"172.21.15.102:6813/2240964461","heartbeat_back_addr":"172.21.15.102:6817/2240964461","heartbeat_front_addr":"172.21.15.102:6815/2240964461","state":["exists","up"]},{"osd":2,"uuid":"40a94b2b-2c24-4164-91be-0816a1acf3d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":100,"up_from":104,"up_thru":498,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":4066184234},{"type":"v1","addr":"172.21.15.102:6819","nonce":4066184234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":4066184234},{"type":"v1","addr":"172.21.15.102:6821","nonce":4066184234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":4066184234},{"type":"v1","addr":"172.21.15.102:6825","nonce":4066184234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":4066184234},{"type":"v1","addr":"172.21.15.102:6823","nonce":4066184234}]},"public_addr":"172.21.15.102:6819/4066184234","cluster_addr":"172.21.15.102:6821/4066184234","heartbeat_back_addr":"172.21.15.102:6825/4066184234","heartbeat_front_addr":"172.21.15.102:6823/4066184234","state":["exists","up"]},{"osd":3,"uuid":"2ed85faf-6610-4bcc-a09d-4a6f93f5c6a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":25,"last_clean_end":105,"up_from":109,"up_thru":495,"down_at":106,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1858757656},{"type":"v1","addr":"172.21.15.102:6827","nonce":1858757656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":1858757656},{"type":"v1","addr":"172.21.15.102:6829","nonce":1858757656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":1858757656},{"type":"v1","addr":"172.21.15.102:6833","nonce":1858757656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":1858757656},{"type":"v1","addr":"172.21.15.102:6831","nonce":1858757656}]},"public_addr":"172.21.15.102:6827/1858757656","cluster_addr":"172.21.15.102:6829/1858757656","heartbeat_back_addr":"172.21.15.102:6833/1858757656","heartbeat_front_addr":"172.21.15.102:6831/1858757656","state":["exists","up"]},{"osd":4,"uuid":"8a45f350-fb3e-4049-bcc4-532d3e20c238","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":498,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":2821651815},{"type":"v1","addr":"172.21.15.195:6801","nonce":2821651815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":2821651815},{"type":"v1","addr":"172.21.15.195:6803","nonce":2821651815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":2821651815},{"type":"v1","addr":"172.21.15.195:6807","nonce":2821651815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":2821651815},{"type":"v1","addr":"172.21.15.195:6805","nonce":2821651815}]},"public_addr":"172.21.15.195:6801/2821651815","cluster_addr":"172.21.15.195:6803/2821651815","heartbeat_back_addr":"172.21.15.195:6807/2821651815","heartbeat_front_addr":"172.21.15.195:6805/2821651815","state":["exists","up"]},{"osd":5,"uuid":"363cf638-db26-4c86-b2be-e075fd09ac16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":498,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":3074139237},{"type":"v1","addr":"172.21.15.195:6809","nonce":3074139237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":3074139237},{"type":"v1","addr":"172.21.15.195:6811","nonce":3074139237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":3074139237},{"type":"v1","addr":"172.21.15.195:6815","nonce":3074139237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":3074139237},{"type":"v1","addr":"172.21.15.195:6813","nonce":3074139237}]},"public_addr":"172.21.15.195:6809/3074139237","cluster_addr":"172.21.15.195:6811/3074139237","heartbeat_back_addr":"172.21.15.195:6815/3074139237","heartbeat_front_addr":"172.21.15.195:6813/3074139237","state":["exists","up"]},{"osd":6,"uuid":"f37416e8-9c25-4860-a32f-0d1c404a8bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":46,"last_clean_end":210,"up_from":239,"up_thru":516,"down_at":211,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":3802194762},{"type":"v1","addr":"172.21.15.195:6817","nonce":3802194762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":3802194762},{"type":"v1","addr":"172.21.15.195:6819","nonce":3802194762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":3802194762},{"type":"v1","addr":"172.21.15.195:6823","nonce":3802194762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":3802194762},{"type":"v1","addr":"172.21.15.195:6821","nonce":3802194762}]},"public_addr":"172.21.15.195:6817/3802194762","cluster_addr":"172.21.15.195:6819/3802194762","heartbeat_back_addr":"172.21.15.195:6823/3802194762","heartbeat_front_addr":"172.21.15.195:6821/3802194762","state":["exists","up"]},{"osd":7,"uuid":"9a384a4d-fddb-4fb3-843a-77c19cca0368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":498,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2059403787},{"type":"v1","addr":"172.21.15.195:6825","nonce":2059403787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2059403787},{"type":"v1","addr":"172.21.15.195:6827","nonce":2059403787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2059403787},{"type":"v1","addr":"172.21.15.195:6831","nonce":2059403787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2059403787},{"type":"v1","addr":"172.21.15.195:6829","nonce":2059403787}]},"public_addr":"172.21.15.195:6825/2059403787","cluster_addr":"172.21.15.195:6827/2059403787","heartbeat_back_addr":"172.21.15.195:6831/2059403787","heartbeat_front_addr":"172.21.15.195:6829/2059403787","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:06:53.502746+0000","dead_epoch":90},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:07:16.251669+0000","dead_epoch":95},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:07:38.848272+0000","dead_epoch":100},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:08:02.013035+0000","dead_epoch":105},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:08:23.043482+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:08:45.653159+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:09:10.187018+0000","dead_epoch":210},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:09:34.393753+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.c","mappings":[{"from":3,"to":2}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":3,"to":5},{"from":1,"to":0}]},{"pgid":"4.1d","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/3514811128":"2024-09-14T14:17:25.102165+0000","172.21.15.102:6801/3856547061":"2024-09-14T14:17:25.102165+0000","172.21.15.102:6800/3856547061":"2024-09-14T14:17:25.102165+0000","172.21.15.102:0/2872102326":"2024-09-14T14:17:25.102165+0000","172.21.15.102:0/3399546099":"2024-09-14T14:05:37.634624+0000","172.21.15.102:0/1430433627":"2024-09-14T14:14:08.048329+0000","172.21.15.102:0/2489373272":"2024-09-14T14:14:08.048329+0000","172.21.15.195:6835/883376241":"2024-09-14T14:14:48.059120+0000","172.21.15.102:6801/2761043787":"2024-09-14T14:05:15.286479+0000","172.21.15.102:0/1574792592":"2024-09-14T14:05:15.286479+0000","172.21.15.102:6801/63514412":"2024-09-14T14:10:09.037657+0000","172.21.15.102:6800/63514412":"2024-09-14T14:10:09.037657+0000","172.21.15.102:0/2083488703":"2024-09-14T14:14:08.048329+0000","172.21.15.102:0/2975343136":"2024-09-14T14:17:25.102165+0000","172.21.15.195:0/2524944732":"2024-09-14T14:14:48.059120+0000","172.21.15.102:6800/2761043787":"2024-09-14T14:05:15.286479+0000","172.21.15.102:6801/3480998245":"2024-09-14T14:05:37.634624+0000","172.21.15.102:6800/3480998245":"2024-09-14T14:05:37.634624+0000","172.21.15.195:0/1568198689":"2024-09-14T14:14:48.059120+0000","172.21.15.195:6834/883376241":"2024-09-14T14:14:48.059120+0000","172.21.15.102:0/1792178465":"2024-09-14T14:14:08.048329+0000","172.21.15.102:0/120875803":"2024-09-14T14:17:25.102165+0000","172.21.15.102:0/1160237252":"2024-09-14T14:10:09.037657+0000","172.21.15.102:0/1582001159":"2024-09-14T14:05:15.286479+0000","172.21.15.195:0/4292730726":"2024-09-14T14:14:48.059120+0000","172.21.15.102:6800/1522897266":"2024-09-14T14:14:08.048329+0000","172.21.15.102:0/1939570054":"2024-09-14T14:05:37.634624+0000","172.21.15.102:0/3189088157":"2024-09-14T14:05:15.286479+0000","172.21.15.195:0/3629133224":"2024-09-14T14:14:48.059120+0000","172.21.15.102:0/2308924552":"2024-09-14T14:10:09.037657+0000","172.21.15.102:0/2416257718":"2024-09-14T14:10:09.037657+0000","172.21.15.102:6801/1522897266":"2024-09-14T14:14:08.048329+0000","172.21.15.102:0/1765904351":"2024-09-14T14:10:09.037657+0000","172.21.15.102:0/2444394840":"2024-09-14T14:05:37.634624+0000","172.21.15.195:0/1729595567":"2024-09-14T14:14:48.059120+0000","172.21.15.195:0/495121375":"2024-09-14T14:14:48.059120+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":11,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T15:00:51.346 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:51 smithi102 ceph-mon[110183]: pgmap v1732: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:51.347 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:51 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1177834483' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T15:00:51.347 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:51 smithi102 ceph-mon[115269]: pgmap v1732: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:51.347 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:51 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1177834483' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T15:00:51.368 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-13T15:00:51.368 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-13T15:00:51.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:51 smithi195 ceph-mon[104262]: pgmap v1732: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:00:51.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:51 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1177834483' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T15:00:51.582 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:53.370 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:53 smithi102 ceph-mon[110183]: pgmap v1733: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 11 KiB/s rd, 32 KiB/s wr, 7 op/s; 23 KiB/s, 0 objects/s recovering 2024-09-13T15:00:53.370 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:53 smithi102 ceph-mon[115269]: pgmap v1733: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 11 KiB/s rd, 32 KiB/s wr, 7 op/s; 23 KiB/s, 0 objects/s recovering 2024-09-13T15:00:53.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:53 smithi195 ceph-mon[104262]: pgmap v1733: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 11 KiB/s rd, 32 KiB/s wr, 7 op/s; 23 KiB/s, 0 objects/s recovering 2024-09-13T15:00:53.871 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T15:00:53.871 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":517,"fsid":"08cee502-71d9-11ef-bceb-c7b262605968","created":"2024-09-13T14:04:48.179879+0000","modified":"2024-09-13T15:00:49.041594+0000","last_up_change":"2024-09-13T14:37:47.105103+0000","last_in_change":"2024-09-13T14:59:52.482950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T14:07:41.927240+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"474","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"471","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.059999942779541,"score_stable":2.059999942779541,"optimal_score":1,"raw_score_acting":2.059999942779541,"raw_score_stable":2.059999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T14:10:10.981300+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":85,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-13T14:11:27.935818+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-13T14:11:28.454262+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b43b49e0-1cf0-47b8-aa85-37108f321220","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":90,"up_from":94,"up_thru":501,"down_at":91,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":3285732759},{"type":"v1","addr":"172.21.15.102:6803","nonce":3285732759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":3285732759},{"type":"v1","addr":"172.21.15.102:6805","nonce":3285732759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3285732759},{"type":"v1","addr":"172.21.15.102:6809","nonce":3285732759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":3285732759},{"type":"v1","addr":"172.21.15.102:6807","nonce":3285732759}]},"public_addr":"172.21.15.102:6803/3285732759","cluster_addr":"172.21.15.102:6805/3285732759","heartbeat_back_addr":"172.21.15.102:6809/3285732759","heartbeat_front_addr":"172.21.15.102:6807/3285732759","state":["exists","up"]},{"osd":1,"uuid":"27e10d5e-c471-4132-9c73-83c5708c3b08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":95,"up_from":99,"up_thru":498,"down_at":96,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":2240964461},{"type":"v1","addr":"172.21.15.102:6811","nonce":2240964461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":2240964461},{"type":"v1","addr":"172.21.15.102:6813","nonce":2240964461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":2240964461},{"type":"v1","addr":"172.21.15.102:6817","nonce":2240964461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":2240964461},{"type":"v1","addr":"172.21.15.102:6815","nonce":2240964461}]},"public_addr":"172.21.15.102:6811/2240964461","cluster_addr":"172.21.15.102:6813/2240964461","heartbeat_back_addr":"172.21.15.102:6817/2240964461","heartbeat_front_addr":"172.21.15.102:6815/2240964461","state":["exists","up"]},{"osd":2,"uuid":"40a94b2b-2c24-4164-91be-0816a1acf3d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":100,"up_from":104,"up_thru":498,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":4066184234},{"type":"v1","addr":"172.21.15.102:6819","nonce":4066184234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":4066184234},{"type":"v1","addr":"172.21.15.102:6821","nonce":4066184234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":4066184234},{"type":"v1","addr":"172.21.15.102:6825","nonce":4066184234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":4066184234},{"type":"v1","addr":"172.21.15.102:6823","nonce":4066184234}]},"public_addr":"172.21.15.102:6819/4066184234","cluster_addr":"172.21.15.102:6821/4066184234","heartbeat_back_addr":"172.21.15.102:6825/4066184234","heartbeat_front_addr":"172.21.15.102:6823/4066184234","state":["exists","up"]},{"osd":3,"uuid":"2ed85faf-6610-4bcc-a09d-4a6f93f5c6a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":25,"last_clean_end":105,"up_from":109,"up_thru":495,"down_at":106,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1858757656},{"type":"v1","addr":"172.21.15.102:6827","nonce":1858757656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":1858757656},{"type":"v1","addr":"172.21.15.102:6829","nonce":1858757656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":1858757656},{"type":"v1","addr":"172.21.15.102:6833","nonce":1858757656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":1858757656},{"type":"v1","addr":"172.21.15.102:6831","nonce":1858757656}]},"public_addr":"172.21.15.102:6827/1858757656","cluster_addr":"172.21.15.102:6829/1858757656","heartbeat_back_addr":"172.21.15.102:6833/1858757656","heartbeat_front_addr":"172.21.15.102:6831/1858757656","state":["exists","up"]},{"osd":4,"uuid":"8a45f350-fb3e-4049-bcc4-532d3e20c238","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":498,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":2821651815},{"type":"v1","addr":"172.21.15.195:6801","nonce":2821651815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":2821651815},{"type":"v1","addr":"172.21.15.195:6803","nonce":2821651815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":2821651815},{"type":"v1","addr":"172.21.15.195:6807","nonce":2821651815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":2821651815},{"type":"v1","addr":"172.21.15.195:6805","nonce":2821651815}]},"public_addr":"172.21.15.195:6801/2821651815","cluster_addr":"172.21.15.195:6803/2821651815","heartbeat_back_addr":"172.21.15.195:6807/2821651815","heartbeat_front_addr":"172.21.15.195:6805/2821651815","state":["exists","up"]},{"osd":5,"uuid":"363cf638-db26-4c86-b2be-e075fd09ac16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":498,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":3074139237},{"type":"v1","addr":"172.21.15.195:6809","nonce":3074139237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":3074139237},{"type":"v1","addr":"172.21.15.195:6811","nonce":3074139237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":3074139237},{"type":"v1","addr":"172.21.15.195:6815","nonce":3074139237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":3074139237},{"type":"v1","addr":"172.21.15.195:6813","nonce":3074139237}]},"public_addr":"172.21.15.195:6809/3074139237","cluster_addr":"172.21.15.195:6811/3074139237","heartbeat_back_addr":"172.21.15.195:6815/3074139237","heartbeat_front_addr":"172.21.15.195:6813/3074139237","state":["exists","up"]},{"osd":6,"uuid":"f37416e8-9c25-4860-a32f-0d1c404a8bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":46,"last_clean_end":210,"up_from":239,"up_thru":516,"down_at":211,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":3802194762},{"type":"v1","addr":"172.21.15.195:6817","nonce":3802194762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":3802194762},{"type":"v1","addr":"172.21.15.195:6819","nonce":3802194762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":3802194762},{"type":"v1","addr":"172.21.15.195:6823","nonce":3802194762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":3802194762},{"type":"v1","addr":"172.21.15.195:6821","nonce":3802194762}]},"public_addr":"172.21.15.195:6817/3802194762","cluster_addr":"172.21.15.195:6819/3802194762","heartbeat_back_addr":"172.21.15.195:6823/3802194762","heartbeat_front_addr":"172.21.15.195:6821/3802194762","state":["exists","up"]},{"osd":7,"uuid":"9a384a4d-fddb-4fb3-843a-77c19cca0368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":498,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":2059403787},{"type":"v1","addr":"172.21.15.195:6825","nonce":2059403787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":2059403787},{"type":"v1","addr":"172.21.15.195:6827","nonce":2059403787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":2059403787},{"type":"v1","addr":"172.21.15.195:6831","nonce":2059403787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":2059403787},{"type":"v1","addr":"172.21.15.195:6829","nonce":2059403787}]},"public_addr":"172.21.15.195:6825/2059403787","cluster_addr":"172.21.15.195:6827/2059403787","heartbeat_back_addr":"172.21.15.195:6831/2059403787","heartbeat_front_addr":"172.21.15.195:6829/2059403787","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:06:53.502746+0000","dead_epoch":90},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:07:16.251669+0000","dead_epoch":95},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:07:38.848272+0000","dead_epoch":100},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:08:02.013035+0000","dead_epoch":105},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:08:23.043482+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:08:45.653159+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:09:10.187018+0000","dead_epoch":210},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:09:34.393753+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.c","mappings":[{"from":3,"to":2}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":3,"to":5},{"from":1,"to":0}]},{"pgid":"4.1d","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/3514811128":"2024-09-14T14:17:25.102165+0000","172.21.15.102:6801/3856547061":"2024-09-14T14:17:25.102165+0000","172.21.15.102:6800/3856547061":"2024-09-14T14:17:25.102165+0000","172.21.15.102:0/3399546099":"2024-09-14T14:05:37.634624+0000","172.21.15.102:0/2489373272":"2024-09-14T14:14:08.048329+0000","172.21.15.195:6835/883376241":"2024-09-14T14:14:48.059120+0000","172.21.15.102:6801/2761043787":"2024-09-14T14:05:15.286479+0000","172.21.15.102:0/1574792592":"2024-09-14T14:05:15.286479+0000","172.21.15.102:6801/63514412":"2024-09-14T14:10:09.037657+0000","172.21.15.102:6800/63514412":"2024-09-14T14:10:09.037657+0000","172.21.15.102:0/2083488703":"2024-09-14T14:14:08.048329+0000","172.21.15.102:6800/2761043787":"2024-09-14T14:05:15.286479+0000","172.21.15.102:6801/3480998245":"2024-09-14T14:05:37.634624+0000","172.21.15.195:6834/883376241":"2024-09-14T14:14:48.059120+0000","172.21.15.102:0/1792178465":"2024-09-14T14:14:08.048329+0000","172.21.15.102:0/1430433627":"2024-09-14T14:14:08.048329+0000","172.21.15.195:0/2524944732":"2024-09-14T14:14:48.059120+0000","172.21.15.102:0/2975343136":"2024-09-14T14:17:25.102165+0000","172.21.15.195:0/3629133224":"2024-09-14T14:14:48.059120+0000","172.21.15.102:0/1160237252":"2024-09-14T14:10:09.037657+0000","172.21.15.102:0/120875803":"2024-09-14T14:17:25.102165+0000","172.21.15.102:0/1582001159":"2024-09-14T14:05:15.286479+0000","172.21.15.195:0/1568198689":"2024-09-14T14:14:48.059120+0000","172.21.15.102:6800/3480998245":"2024-09-14T14:05:37.634624+0000","172.21.15.195:0/4292730726":"2024-09-14T14:14:48.059120+0000","172.21.15.102:6800/1522897266":"2024-09-14T14:14:08.048329+0000","172.21.15.102:0/1939570054":"2024-09-14T14:05:37.634624+0000","172.21.15.102:0/1765904351":"2024-09-14T14:10:09.037657+0000","172.21.15.102:6801/1522897266":"2024-09-14T14:14:08.048329+0000","172.21.15.102:0/3189088157":"2024-09-14T14:05:15.286479+0000","172.21.15.195:0/495121375":"2024-09-14T14:14:48.059120+0000","172.21.15.102:0/2444394840":"2024-09-14T14:05:37.634624+0000","172.21.15.195:0/1729595567":"2024-09-14T14:14:48.059120+0000","172.21.15.102:0/2308924552":"2024-09-14T14:10:09.037657+0000","172.21.15.102:0/2416257718":"2024-09-14T14:10:09.037657+0000","172.21.15.102:0/2872102326":"2024-09-14T14:17:25.102165+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":11,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T15:00:54.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:54 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3164393364' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T15:00:54.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:54 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3164393364' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T15:00:54.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:54 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3164393364' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T15:00:54.644 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-13T15:00:54.644 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-13T15:00:54.645 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-13T15:00:54.645 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-13T15:00:54.645 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-13T15:00:54.646 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-13T15:00:54.646 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-13T15:00:54.647 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-13T15:00:55.110 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:55.111 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:55.111 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:55.111 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:55.111 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:55.115 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:55.115 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:55.120 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:55.481 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:55 smithi102 ceph-mon[110183]: 1.f scrub starts 2024-09-13T15:00:55.481 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:55 smithi102 ceph-mon[110183]: 1.f scrub ok 2024-09-13T15:00:55.481 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:55 smithi102 ceph-mon[110183]: pgmap v1734: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 9.2 KiB/s rd, 27 KiB/s wr, 6 op/s; 20 KiB/s, 0 objects/s recovering 2024-09-13T15:00:55.481 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:55 smithi102 ceph-mon[115269]: 1.f scrub starts 2024-09-13T15:00:55.481 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:55 smithi102 ceph-mon[115269]: 1.f scrub ok 2024-09-13T15:00:55.481 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:55 smithi102 ceph-mon[115269]: pgmap v1734: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 9.2 KiB/s rd, 27 KiB/s wr, 6 op/s; 20 KiB/s, 0 objects/s recovering 2024-09-13T15:00:55.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:55 smithi195 ceph-mon[104262]: 1.f scrub starts 2024-09-13T15:00:55.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:55 smithi195 ceph-mon[104262]: 1.f scrub ok 2024-09-13T15:00:55.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:55 smithi195 ceph-mon[104262]: pgmap v1734: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 9.2 KiB/s rd, 27 KiB/s wr, 6 op/s; 20 KiB/s, 0 objects/s recovering 2024-09-13T15:00:56.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:56 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:00:56.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:56 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:00:56.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:56 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:00:56.770 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 15:00:56 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:15:00:56] "GET /metrics HTTP/1.1" 200 34171 "" "Prometheus/2.51.0" 2024-09-13T15:00:57.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:57 smithi102 ceph-mon[110183]: pgmap v1735: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 9.2 KiB/s rd, 27 KiB/s wr, 6 op/s; 20 KiB/s, 0 objects/s recovering 2024-09-13T15:00:57.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:57 smithi102 ceph-mon[115269]: pgmap v1735: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 9.2 KiB/s rd, 27 KiB/s wr, 6 op/s; 20 KiB/s, 0 objects/s recovering 2024-09-13T15:00:57.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:57 smithi195 ceph-mon[104262]: pgmap v1735: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 9.2 KiB/s rd, 27 KiB/s wr, 6 op/s; 20 KiB/s, 0 objects/s recovering 2024-09-13T15:00:58.457 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:58 smithi102 ceph-mon[110183]: 1.6 scrub starts 2024-09-13T15:00:58.457 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:58 smithi102 ceph-mon[110183]: 1.6 scrub ok 2024-09-13T15:00:58.457 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:58 smithi102 ceph-mon[115269]: 1.6 scrub starts 2024-09-13T15:00:58.457 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:58 smithi102 ceph-mon[115269]: 1.6 scrub ok 2024-09-13T15:00:58.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:58 smithi195 ceph-mon[104262]: 1.6 scrub starts 2024-09-13T15:00:58.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:58 smithi195 ceph-mon[104262]: 1.6 scrub ok 2024-09-13T15:00:59.412 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:00:59 smithi102 ceph-mon[110183]: pgmap v1736: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.8 KiB/s rd, 5.6 KiB/s wr, 1 op/s; 4.0 KiB/s, 0 objects/s recovering 2024-09-13T15:00:59.412 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:00:59 smithi102 ceph-mon[115269]: pgmap v1736: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.8 KiB/s rd, 5.6 KiB/s wr, 1 op/s; 4.0 KiB/s, 0 objects/s recovering 2024-09-13T15:00:59.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:00:59 smithi195 ceph-mon[104262]: pgmap v1736: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.8 KiB/s rd, 5.6 KiB/s wr, 1 op/s; 4.0 KiB/s, 0 objects/s recovering 2024-09-13T15:01:01.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:01:01 smithi102 ceph-mon[115269]: pgmap v1737: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.1 KiB/s wr, 1 op/s; 3.6 KiB/s, 0 objects/s recovering 2024-09-13T15:01:01.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:01:01 smithi102 ceph-mon[110183]: pgmap v1737: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.1 KiB/s wr, 1 op/s; 3.6 KiB/s, 0 objects/s recovering 2024-09-13T15:01:01.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:01 smithi195 ceph-mon[104262]: pgmap v1737: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.1 KiB/s wr, 1 op/s; 3.6 KiB/s, 0 objects/s recovering 2024-09-13T15:01:03.390 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:01:03 smithi102 ceph-mon[110183]: 1.7 scrub starts 2024-09-13T15:01:03.390 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:01:03 smithi102 ceph-mon[110183]: 1.7 scrub ok 2024-09-13T15:01:03.390 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:01:03 smithi102 ceph-mon[110183]: pgmap v1738: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:03.390 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:01:03 smithi102 ceph-mon[115269]: 1.7 scrub starts 2024-09-13T15:01:03.390 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:01:03 smithi102 ceph-mon[115269]: 1.7 scrub ok 2024-09-13T15:01:03.391 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:01:03 smithi102 ceph-mon[115269]: pgmap v1738: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:03.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:03 smithi195 ceph-mon[104262]: 1.7 scrub starts 2024-09-13T15:01:03.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:03 smithi195 ceph-mon[104262]: 1.7 scrub ok 2024-09-13T15:01:03.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:03 smithi195 ceph-mon[104262]: pgmap v1738: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:05.401 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:01:05 smithi102 ceph-mon[115269]: pgmap v1739: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:05.401 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:01:05 smithi102 ceph-mon[110183]: pgmap v1739: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:05.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:05 smithi195 ceph-mon[104262]: pgmap v1739: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:06.610 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 15:01:06 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:15:01:06] "GET /metrics HTTP/1.1" 200 34171 "" "Prometheus/2.51.0" 2024-09-13T15:01:07.290 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:01:07 smithi102 ceph-mon[110183]: pgmap v1740: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:07.290 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:01:07 smithi102 ceph-mon[115269]: pgmap v1740: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:07.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:07 smithi195 ceph-mon[104262]: pgmap v1740: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:07.593 INFO:teuthology.orchestra.run.smithi102.stdout:1026497184025 2024-09-13T15:01:07.593 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-13T15:01:09.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:09 smithi195 ceph-mon[104262]: pgmap v1741: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:09.605 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:01:09 smithi102 ceph-mon[115269]: pgmap v1741: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:09.605 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:01:09 smithi102 ceph-mon[110183]: pgmap v1741: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:09.630 INFO:teuthology.orchestra.run.smithi102.stdout:163208757872 2024-09-13T15:01:09.630 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-13T15:01:09.997 INFO:teuthology.orchestra.run.smithi102.stdout:425201762754 2024-09-13T15:01:09.997 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-13T15:01:10.366 INFO:teuthology.orchestra.run.smithi102.stdout:128849019516 2024-09-13T15:01:10.366 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-13T15:01:10.776 INFO:teuthology.orchestra.run.smithi102.stdout:403726926278 2024-09-13T15:01:10.776 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-13T15:01:11.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:01:11 smithi102 ceph-mon[110183]: pgmap v1742: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:11.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:01:11 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:01:11.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:01:11 smithi102 ceph-mon[115269]: pgmap v1742: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:11.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:01:11 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:01:11.465 INFO:teuthology.orchestra.run.smithi102.stdout:231928234607 2024-09-13T15:01:11.465 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-13T15:01:11.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:11 smithi195 ceph-mon[104262]: pgmap v1742: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:11.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:11 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:01:12.517 INFO:teuthology.orchestra.run.smithi102.stdout:446676599228 2024-09-13T15:01:12.518 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-13T15:01:12.916 INFO:teuthology.orchestra.run.smithi102.stdout:468151435700 2024-09-13T15:01:12.916 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-13T15:01:12.937 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:13.217 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:13.217 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:13.459 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:01:13 smithi102 ceph-mon[110183]: pgmap v1743: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:13.459 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:01:13 smithi102 ceph-mon[115269]: pgmap v1743: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:13.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:13 smithi195 ceph-mon[104262]: pgmap v1743: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:13.675 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:13.676 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:13.677 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:15.497 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:01:15 smithi102 ceph-mon[110183]: pgmap v1744: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:15.497 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:01:15 smithi102 ceph-mon[115269]: pgmap v1744: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:15.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:15 smithi195 ceph-mon[104262]: pgmap v1744: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:16.669 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 15:01:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:15:01:16] "GET /metrics HTTP/1.1" 200 34147 "" "Prometheus/2.51.0" 2024-09-13T15:01:17.447 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:17.448 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:17.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:17 smithi195 ceph-mon[104262]: pgmap v1745: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:17.537 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:01:17 smithi102 ceph-mon[110183]: pgmap v1745: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:17.537 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:01:17 smithi102 ceph-mon[115269]: pgmap v1745: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:18.257 INFO:teuthology.orchestra.run.smithi102.stdout:1026497184028 2024-09-13T15:01:19.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:19 smithi195 ceph-mon[104262]: pgmap v1746: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:19.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:19 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1885910336' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-13T15:01:19.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:01:19 smithi102 ceph-mon[110183]: pgmap v1746: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:19.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:01:19 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1885910336' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-13T15:01:19.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:01:19 smithi102 ceph-mon[115269]: pgmap v1746: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:19.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:01:19 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1885910336' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-13T15:01:20.498 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:20 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:01:20.504 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:01:20 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:01:20.504 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:01:20 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:01:21.251 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:21 smithi195 ceph-mon[104262]: pgmap v1747: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:21.305 INFO:teuthology.orchestra.run.smithi102.stdout:425201762757 2024-09-13T15:01:21.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:01:21 smithi102 ceph-mon[110183]: pgmap v1747: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:21.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:01:21 smithi102 ceph-mon[115269]: pgmap v1747: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:22.035 INFO:tasks.cephadm.ceph_manager.ceph:need seq 425201762754 got 425201762757 for osd.1 2024-09-13T15:01:22.036 DEBUG:teuthology.parallel:result is None 2024-09-13T15:01:22.358 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:01:22 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1372440245' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-13T15:01:22.358 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:01:22 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1372440245' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-13T15:01:22.497 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:22 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1372440245' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-13T15:01:23.123 INFO:teuthology.orchestra.run.smithi102.stdout:163208757876 2024-09-13T15:01:23.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:01:23 smithi102 ceph-mon[110183]: pgmap v1748: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:23.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:01:23 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/3736057048' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-13T15:01:23.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:01:23 smithi102 ceph-mon[115269]: pgmap v1748: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:23.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:01:23 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/3736057048' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-13T15:01:23.536 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:23 smithi195 ceph-mon[104262]: pgmap v1748: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:23.536 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:23 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/3736057048' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-13T15:01:23.951 INFO:teuthology.orchestra.run.smithi102.stdout:128849019520 2024-09-13T15:01:25.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:01:24 smithi102 ceph-mon[110183]: pgmap v1749: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:25.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:01:24 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T15:01:25.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:01:24 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T15:01:25.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:01:24 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/616243119' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-13T15:01:25.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:01:24 smithi102 ceph-mon[115269]: pgmap v1749: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:25.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:01:24 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T15:01:25.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:01:24 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T15:01:25.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:01:24 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/616243119' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-13T15:01:25.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:24 smithi195 ceph-mon[104262]: pgmap v1749: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:25.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:24 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T15:01:25.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:24 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T15:01:25.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:24 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/616243119' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-13T15:01:25.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:01:25 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:01:25.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:01:25 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2462607388' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-13T15:01:25.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:01:25 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:01:25.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:01:25 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2462607388' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-13T15:01:26.149 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:25 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:01:26.149 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:25 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2462607388' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-13T15:01:26.169 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849019516 got 128849019520 for osd.4 2024-09-13T15:01:26.169 DEBUG:teuthology.parallel:result is None 2024-09-13T15:01:26.625 INFO:teuthology.orchestra.run.smithi102.stdout:446676599231 2024-09-13T15:01:26.888 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 15:01:26 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:15:01:26] "GET /metrics HTTP/1.1" 200 34149 "" "Prometheus/2.51.0" 2024-09-13T15:01:27.163 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:01:26 smithi102 ceph-mon[110183]: pgmap v1750: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:27.164 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:01:26 smithi102 ceph-mon[115269]: pgmap v1750: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:27.203 INFO:teuthology.orchestra.run.smithi102.stdout:403726926281 2024-09-13T15:01:27.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:26 smithi195 ceph-mon[104262]: pgmap v1750: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:27.499 INFO:tasks.cephadm.ceph_manager.ceph:need seq 446676599228 got 446676599231 for osd.2 2024-09-13T15:01:27.499 DEBUG:teuthology.parallel:result is None 2024-09-13T15:01:28.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:01:27 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1317528889' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-13T15:01:28.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:01:27 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1317528889' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-13T15:01:28.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:27 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1317528889' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-13T15:01:28.323 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757872 got 163208757876 for osd.5 2024-09-13T15:01:28.323 DEBUG:teuthology.parallel:result is None 2024-09-13T15:01:28.759 INFO:tasks.cephadm.ceph_manager.ceph:need seq 1026497184025 got 1026497184028 for osd.6 2024-09-13T15:01:28.759 DEBUG:teuthology.parallel:result is None 2024-09-13T15:01:29.076 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:01:28 smithi102 ceph-mon[110183]: pgmap v1751: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:29.077 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:01:28 smithi102 ceph-mon[115269]: pgmap v1751: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:29.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:28 smithi195 ceph-mon[104262]: pgmap v1751: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:29.604 INFO:tasks.cephadm.ceph_manager.ceph:need seq 403726926278 got 403726926281 for osd.0 2024-09-13T15:01:29.604 DEBUG:teuthology.parallel:result is None 2024-09-13T15:01:30.191 INFO:teuthology.orchestra.run.smithi102.stdout:468151435704 2024-09-13T15:01:30.422 INFO:teuthology.orchestra.run.smithi102.stdout:231928234612 2024-09-13T15:01:30.877 INFO:tasks.cephadm.ceph_manager.ceph:need seq 468151435700 got 468151435704 for osd.3 2024-09-13T15:01:30.877 DEBUG:teuthology.parallel:result is None 2024-09-13T15:01:31.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:01:30 smithi102 ceph-mon[110183]: pgmap v1752: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:31.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:01:30 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/1942853011' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-13T15:01:31.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:01:30 smithi102 ceph-mon[110183]: from='client.? 172.21.15.102:0/2605949142' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-13T15:01:31.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:01:30 smithi102 ceph-mon[115269]: pgmap v1752: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:31.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:01:30 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/1942853011' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-13T15:01:31.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:01:30 smithi102 ceph-mon[115269]: from='client.? 172.21.15.102:0/2605949142' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-13T15:01:31.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:30 smithi195 ceph-mon[104262]: pgmap v1752: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:31.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:30 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/1942853011' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-13T15:01:31.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:30 smithi195 ceph-mon[104262]: from='client.? 172.21.15.102:0/2605949142' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-13T15:01:31.607 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234607 got 231928234612 for osd.7 2024-09-13T15:01:31.608 DEBUG:teuthology.parallel:result is None 2024-09-13T15:01:31.608 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-13T15:01:31.608 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T15:01:32.377 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:33.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:01:32 smithi102 ceph-mon[110183]: pgmap v1753: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:33.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:01:32 smithi102 ceph-mon[115269]: pgmap v1753: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:33.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:32 smithi195 ceph-mon[104262]: pgmap v1753: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:34.345 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T15:01:34.347 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T15:01:35.028 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:01:34 smithi102 ceph-mon[110183]: pgmap v1754: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:35.028 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:01:34 smithi102 ceph-mon[110183]: from='client.47207 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:01:35.028 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:01:34 smithi102 ceph-mon[115269]: pgmap v1754: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:35.028 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:01:34 smithi102 ceph-mon[115269]: from='client.47207 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:01:35.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:34 smithi195 ceph-mon[104262]: pgmap v1754: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:35.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:34 smithi195 ceph-mon[104262]: from='client.47207 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:01:35.497 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":1754,"stamp":"2024-09-13T15:01:33.880463+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790245,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":908,"num_write_kb":12625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":257,"num_bytes_recovered":8395848,"num_keys_recovered":40,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5652,"ondisk_log_size":5652,"up":309,"acting":309,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":309,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":7229196,"kb_used_data":66212,"kb_used_omap":41,"kb_used_meta":7162838,"kb_avail":742732020,"statfs":{"total":767960285184,"available":760557588480,"internally_reserved":0,"allocated":67801088,"data_stored":46977854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42913,"internal_metadata":7334746207},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003108"},"pg_stats":[{"pgid":"3.1f","version":"71'4","reported_seq":513,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.047466+0000","last_change":"2024-09-13T15:00:03.537248+0000","last_active":"2024-09-13T15:00:49.047466+0000","last_peered":"2024-09-13T15:00:49.047466+0000","last_clean":"2024-09-13T15:00:49.047466+0000","last_became_active":"2024-09-13T15:00:03.536943+0000","last_became_peered":"2024-09-13T15:00:03.536943+0000","last_unstale":"2024-09-13T15:00:49.047466+0000","last_undegraded":"2024-09-13T15:00:49.047466+0000","last_fullsized":"2024-09-13T15:00:49.047466+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:19.606950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1746,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":32,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.048713+0000","last_change":"2024-09-13T14:59:53.504853+0000","last_active":"2024-09-13T15:00:48.048713+0000","last_peered":"2024-09-13T15:00:48.048713+0000","last_clean":"2024-09-13T15:00:48.048713+0000","last_became_active":"2024-09-13T14:59:53.504556+0000","last_became_peered":"2024-09-13T14:59:53.504556+0000","last_unstale":"2024-09-13T15:00:48.048713+0000","last_undegraded":"2024-09-13T15:00:48.048713+0000","last_fullsized":"2024-09-13T15:00:48.048713+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:55.050435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1d","version":"90'149","reported_seq":530,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.220515+0000","last_change":"2024-09-13T14:59:56.507148+0000","last_active":"2024-09-13T15:00:49.220515+0000","last_peered":"2024-09-13T15:00:49.220515+0000","last_clean":"2024-09-13T15:00:49.220515+0000","last_became_active":"2024-09-13T14:59:56.506750+0000","last_became_peered":"2024-09-13T14:59:56.506750+0000","last_unstale":"2024-09-13T15:00:49.220515+0000","last_undegraded":"2024-09-13T15:00:49.220515+0000","last_fullsized":"2024-09-13T15:00:49.220515+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":471,"last_epoch_clean":499,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:38.760963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:38.760963+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:08:12.028324+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":938,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.050720+0000","last_change":"2024-09-13T14:46:01.769647+0000","last_active":"2024-09-13T15:00:49.050720+0000","last_peered":"2024-09-13T15:00:49.050720+0000","last_clean":"2024-09-13T15:00:49.050720+0000","last_became_active":"2024-09-13T14:46:01.769228+0000","last_became_peered":"2024-09-13T14:46:01.769228+0000","last_unstale":"2024-09-13T15:00:49.050720+0000","last_undegraded":"2024-09-13T15:00:49.050720+0000","last_fullsized":"2024-09-13T15:00:49.050720+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:58.696024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":973,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.049344+0000","last_change":"2024-09-13T14:46:01.766580+0000","last_active":"2024-09-13T15:00:49.049344+0000","last_peered":"2024-09-13T15:00:49.049344+0000","last_clean":"2024-09-13T15:00:49.049344+0000","last_became_active":"2024-09-13T14:46:01.766257+0000","last_became_peered":"2024-09-13T14:46:01.766257+0000","last_unstale":"2024-09-13T15:00:49.049344+0000","last_undegraded":"2024-09-13T15:00:49.049344+0000","last_fullsized":"2024-09-13T15:00:49.049344+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:01.126109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1c","version":"90'149","reported_seq":497,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.045051+0000","last_change":"2024-09-13T14:57:44.980830+0000","last_active":"2024-09-13T15:00:48.045051+0000","last_peered":"2024-09-13T15:00:48.045051+0000","last_clean":"2024-09-13T15:00:48.045051+0000","last_became_active":"2024-09-13T14:57:28.507948+0000","last_became_peered":"2024-09-13T14:57:28.507948+0000","last_unstale":"2024-09-13T15:00:48.045051+0000","last_undegraded":"2024-09-13T15:00:48.045051+0000","last_fullsized":"2024-09-13T15:00:48.045051+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":471,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:44.980740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:44.980740+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:52:20.504558+0000","scrub_duration":0.012428020999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":364,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.049165+0000","last_change":"2024-09-13T14:59:53.515721+0000","last_active":"2024-09-13T15:00:49.049165+0000","last_peered":"2024-09-13T15:00:49.049165+0000","last_clean":"2024-09-13T15:00:49.049165+0000","last_became_active":"2024-09-13T14:59:53.505349+0000","last_became_peered":"2024-09-13T14:59:53.505349+0000","last_unstale":"2024-09-13T15:00:49.049165+0000","last_undegraded":"2024-09-13T15:00:49.049165+0000","last_fullsized":"2024-09-13T15:00:49.049165+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:22:52.933174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":32,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.050410+0000","last_change":"2024-09-13T14:59:53.509385+0000","last_active":"2024-09-13T15:00:48.050410+0000","last_peered":"2024-09-13T15:00:48.050410+0000","last_clean":"2024-09-13T15:00:48.050410+0000","last_became_active":"2024-09-13T14:59:53.507008+0000","last_became_peered":"2024-09-13T14:59:53.507008+0000","last_unstale":"2024-09-13T15:00:48.050410+0000","last_undegraded":"2024-09-13T15:00:48.050410+0000","last_fullsized":"2024-09-13T15:00:48.050410+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:49:04.476461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":377,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.048270+0000","last_change":"2024-09-13T14:59:53.518804+0000","last_active":"2024-09-13T15:00:49.048270+0000","last_peered":"2024-09-13T15:00:49.048270+0000","last_clean":"2024-09-13T15:00:49.048270+0000","last_became_active":"2024-09-13T14:59:53.506622+0000","last_became_peered":"2024-09-13T14:59:53.506622+0000","last_unstale":"2024-09-13T15:00:49.048270+0000","last_undegraded":"2024-09-13T15:00:49.048270+0000","last_fullsized":"2024-09-13T15:00:49.048270+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:09:25.888980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":32,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.050435+0000","last_change":"2024-09-13T14:59:53.508269+0000","last_active":"2024-09-13T15:00:48.050435+0000","last_peered":"2024-09-13T15:00:48.050435+0000","last_clean":"2024-09-13T15:00:48.050435+0000","last_became_active":"2024-09-13T14:59:53.507928+0000","last_became_peered":"2024-09-13T14:59:53.507928+0000","last_unstale":"2024-09-13T15:00:48.050435+0000","last_undegraded":"2024-09-13T15:00:48.050435+0000","last_fullsized":"2024-09-13T15:00:48.050435+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:13.608699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"90'149","reported_seq":491,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.050558+0000","last_change":"2024-09-13T14:57:39.990337+0000","last_active":"2024-09-13T15:00:49.050558+0000","last_peered":"2024-09-13T15:00:49.050558+0000","last_clean":"2024-09-13T15:00:49.050558+0000","last_became_active":"2024-09-13T14:57:28.525160+0000","last_became_peered":"2024-09-13T14:57:28.525160+0000","last_unstale":"2024-09-13T15:00:49.050558+0000","last_undegraded":"2024-09-13T15:00:49.050558+0000","last_fullsized":"2024-09-13T15:00:49.050558+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":471,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:39.990271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:39.990271+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:55.404672+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":500,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.047193+0000","last_change":"2024-09-13T15:00:03.535707+0000","last_active":"2024-09-13T15:00:49.047193+0000","last_peered":"2024-09-13T15:00:49.047193+0000","last_clean":"2024-09-13T15:00:49.047193+0000","last_became_active":"2024-09-13T15:00:03.535394+0000","last_became_peered":"2024-09-13T15:00:03.535394+0000","last_unstale":"2024-09-13T15:00:49.047193+0000","last_undegraded":"2024-09-13T15:00:49.047193+0000","last_fullsized":"2024-09-13T15:00:49.047193+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:30:32.637273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":459,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.050764+0000","last_change":"2024-09-13T14:50:27.510133+0000","last_active":"2024-09-13T15:00:49.050764+0000","last_peered":"2024-09-13T15:00:49.050764+0000","last_clean":"2024-09-13T15:00:49.050764+0000","last_became_active":"2024-09-13T14:50:27.508774+0000","last_became_peered":"2024-09-13T14:50:27.508774+0000","last_unstale":"2024-09-13T15:00:49.050764+0000","last_undegraded":"2024-09-13T15:00:49.050764+0000","last_fullsized":"2024-09-13T15:00:49.050764+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:31:04.531746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.19","version":"90'149","reported_seq":579,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.049186+0000","last_change":"2024-09-13T14:57:38.050038+0000","last_active":"2024-09-13T15:00:49.049186+0000","last_peered":"2024-09-13T15:00:49.049186+0000","last_clean":"2024-09-13T15:00:49.049186+0000","last_became_active":"2024-09-13T14:57:28.530372+0000","last_became_peered":"2024-09-13T14:57:28.530372+0000","last_unstale":"2024-09-13T15:00:49.049186+0000","last_undegraded":"2024-09-13T15:00:49.049186+0000","last_fullsized":"2024-09-13T15:00:49.049186+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":471,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:38.049970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:38.049970+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:10.878382+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":32,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.050511+0000","last_change":"2024-09-13T14:59:53.504414+0000","last_active":"2024-09-13T15:00:48.050511+0000","last_peered":"2024-09-13T15:00:48.050511+0000","last_clean":"2024-09-13T15:00:48.050511+0000","last_became_active":"2024-09-13T14:59:53.503984+0000","last_became_peered":"2024-09-13T14:59:53.503984+0000","last_unstale":"2024-09-13T15:00:48.050511+0000","last_undegraded":"2024-09-13T15:00:48.050511+0000","last_fullsized":"2024-09-13T15:00:48.050511+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:22:35.693249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":961,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.050247+0000","last_change":"2024-09-13T14:59:53.510391+0000","last_active":"2024-09-13T15:00:49.050247+0000","last_peered":"2024-09-13T15:00:49.050247+0000","last_clean":"2024-09-13T15:00:49.050247+0000","last_became_active":"2024-09-13T14:59:53.510044+0000","last_became_peered":"2024-09-13T14:59:53.510044+0000","last_unstale":"2024-09-13T15:00:49.050247+0000","last_undegraded":"2024-09-13T15:00:49.050247+0000","last_fullsized":"2024-09-13T15:00:49.050247+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:25.871310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.18","version":"90'149","reported_seq":641,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.044754+0000","last_change":"2024-09-13T14:57:41.793060+0000","last_active":"2024-09-13T15:00:48.044754+0000","last_peered":"2024-09-13T15:00:48.044754+0000","last_clean":"2024-09-13T15:00:48.044754+0000","last_became_active":"2024-09-13T14:57:28.525726+0000","last_became_peered":"2024-09-13T14:57:28.525726+0000","last_unstale":"2024-09-13T15:00:48.044754+0000","last_undegraded":"2024-09-13T15:00:48.044754+0000","last_fullsized":"2024-09-13T15:00:48.044754+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":471,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:41.792934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:41.792934+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:18.870805+0000","scrub_duration":0.0079393269999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":391,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.220463+0000","last_change":"2024-09-13T14:59:53.503542+0000","last_active":"2024-09-13T15:00:49.220463+0000","last_peered":"2024-09-13T15:00:49.220463+0000","last_clean":"2024-09-13T15:00:49.220463+0000","last_became_active":"2024-09-13T14:59:53.503247+0000","last_became_peered":"2024-09-13T14:59:53.503247+0000","last_unstale":"2024-09-13T15:00:49.220463+0000","last_undegraded":"2024-09-13T15:00:49.220463+0000","last_fullsized":"2024-09-13T15:00:49.220463+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:23:30.238977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":501,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.047364+0000","last_change":"2024-09-13T15:00:03.538343+0000","last_active":"2024-09-13T15:00:49.047364+0000","last_peered":"2024-09-13T15:00:49.047364+0000","last_clean":"2024-09-13T15:00:49.047364+0000","last_became_active":"2024-09-13T15:00:03.537266+0000","last_became_peered":"2024-09-13T15:00:03.537266+0000","last_unstale":"2024-09-13T15:00:49.047364+0000","last_undegraded":"2024-09-13T15:00:49.047364+0000","last_fullsized":"2024-09-13T15:00:49.047364+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:51.421664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1b","version":"90'149","reported_seq":466,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.044262+0000","last_change":"2024-09-13T14:59:56.508684+0000","last_active":"2024-09-13T15:00:48.044262+0000","last_peered":"2024-09-13T15:00:48.044262+0000","last_clean":"2024-09-13T15:00:48.044262+0000","last_became_active":"2024-09-13T14:59:56.508302+0000","last_became_peered":"2024-09-13T14:59:56.508302+0000","last_unstale":"2024-09-13T15:00:48.044262+0000","last_undegraded":"2024-09-13T15:00:48.044262+0000","last_fullsized":"2024-09-13T15:00:48.044262+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":471,"last_epoch_clean":499,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:42.789102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:42.789102+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:17:57.495061+0000","scrub_duration":0.0083861109999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":959,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.048931+0000","last_change":"2024-09-13T14:47:47.868945+0000","last_active":"2024-09-13T15:00:49.048931+0000","last_peered":"2024-09-13T15:00:49.048931+0000","last_clean":"2024-09-13T15:00:49.048931+0000","last_became_active":"2024-09-13T14:47:47.868296+0000","last_became_peered":"2024-09-13T14:47:47.868296+0000","last_unstale":"2024-09-13T15:00:49.048931+0000","last_undegraded":"2024-09-13T15:00:49.048931+0000","last_fullsized":"2024-09-13T15:00:49.048931+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:22:24.778693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":182,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.221778+0000","last_change":"2024-09-13T15:00:49.221419+0000","last_active":"2024-09-13T15:00:49.221778+0000","last_peered":"2024-09-13T15:00:49.221778+0000","last_clean":"2024-09-13T15:00:49.221778+0000","last_became_active":"2024-09-13T15:00:49.221131+0000","last_became_peered":"2024-09-13T15:00:49.221131+0000","last_unstale":"2024-09-13T15:00:49.221778+0000","last_undegraded":"2024-09-13T15:00:49.221778+0000","last_fullsized":"2024-09-13T15:00:49.221778+0000","mapping_epoch":516,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:25:48.840688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1a","version":"90'149","reported_seq":421,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.050604+0000","last_change":"2024-09-13T14:59:58.822684+0000","last_active":"2024-09-13T15:00:48.050604+0000","last_peered":"2024-09-13T15:00:48.050604+0000","last_clean":"2024-09-13T15:00:48.050604+0000","last_became_active":"2024-09-13T14:59:56.506612+0000","last_became_peered":"2024-09-13T14:59:56.506612+0000","last_unstale":"2024-09-13T15:00:48.050604+0000","last_undegraded":"2024-09-13T15:00:48.050604+0000","last_fullsized":"2024-09-13T15:00:48.050604+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":471,"last_epoch_clean":499,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:59:58.822608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:59:58.822608+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:03:38.735268+0000","scrub_duration":0.0020743630000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":927,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.048706+0000","last_change":"2024-09-13T14:47:47.868624+0000","last_active":"2024-09-13T15:00:49.048706+0000","last_peered":"2024-09-13T15:00:49.048706+0000","last_clean":"2024-09-13T15:00:49.048706+0000","last_became_active":"2024-09-13T14:47:47.868031+0000","last_became_peered":"2024-09-13T14:47:47.868031+0000","last_unstale":"2024-09-13T15:00:49.048706+0000","last_undegraded":"2024-09-13T15:00:49.048706+0000","last_fullsized":"2024-09-13T15:00:49.048706+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:05:25.365023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":493,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.047504+0000","last_change":"2024-09-13T15:00:03.536415+0000","last_active":"2024-09-13T15:00:49.047504+0000","last_peered":"2024-09-13T15:00:49.047504+0000","last_clean":"2024-09-13T15:00:49.047504+0000","last_became_active":"2024-09-13T15:00:03.536117+0000","last_became_peered":"2024-09-13T15:00:03.536117+0000","last_unstale":"2024-09-13T15:00:49.047504+0000","last_undegraded":"2024-09-13T15:00:49.047504+0000","last_fullsized":"2024-09-13T15:00:49.047504+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:44:11.549146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.15","version":"90'149","reported_seq":543,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.045988+0000","last_change":"2024-09-13T14:57:46.923759+0000","last_active":"2024-09-13T15:00:48.045988+0000","last_peered":"2024-09-13T15:00:48.045988+0000","last_clean":"2024-09-13T15:00:48.045988+0000","last_became_active":"2024-09-13T14:57:28.519678+0000","last_became_peered":"2024-09-13T14:57:28.519678+0000","last_unstale":"2024-09-13T15:00:48.045988+0000","last_undegraded":"2024-09-13T15:00:48.045988+0000","last_fullsized":"2024-09-13T15:00:48.045988+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":471,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:46.923638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:46.923638+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:59.709420+0000","scrub_duration":0.002484039,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":891,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.049019+0000","last_change":"2024-09-13T14:38:53.899535+0000","last_active":"2024-09-13T15:00:49.049019+0000","last_peered":"2024-09-13T15:00:49.049019+0000","last_clean":"2024-09-13T15:00:49.049019+0000","last_became_active":"2024-09-13T14:38:53.899183+0000","last_became_peered":"2024-09-13T14:38:53.899183+0000","last_unstale":"2024-09-13T15:00:49.049019+0000","last_undegraded":"2024-09-13T15:00:49.049019+0000","last_fullsized":"2024-09-13T15:00:49.049019+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:33.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":350,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.044530+0000","last_change":"2024-09-13T14:47:47.861842+0000","last_active":"2024-09-13T15:00:48.044530+0000","last_peered":"2024-09-13T15:00:48.044530+0000","last_clean":"2024-09-13T15:00:48.044530+0000","last_became_active":"2024-09-13T14:47:47.861492+0000","last_became_peered":"2024-09-13T14:47:47.861492+0000","last_unstale":"2024-09-13T15:00:48.044530+0000","last_undegraded":"2024-09-13T15:00:48.044530+0000","last_fullsized":"2024-09-13T15:00:48.044530+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:17.279533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"90'149","reported_seq":833,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.050450+0000","last_change":"2024-09-13T14:59:56.508469+0000","last_active":"2024-09-13T15:00:49.050450+0000","last_peered":"2024-09-13T15:00:49.050450+0000","last_clean":"2024-09-13T15:00:49.050450+0000","last_became_active":"2024-09-13T14:59:56.508088+0000","last_became_peered":"2024-09-13T14:59:56.508088+0000","last_unstale":"2024-09-13T15:00:49.050450+0000","last_undegraded":"2024-09-13T15:00:49.050450+0000","last_fullsized":"2024-09-13T15:00:49.050450+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":499,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:31.013128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:31.013128+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:17:14.466888+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":319,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.045507+0000","last_change":"2024-09-13T14:59:53.526879+0000","last_active":"2024-09-13T15:00:48.045507+0000","last_peered":"2024-09-13T15:00:48.045507+0000","last_clean":"2024-09-13T15:00:48.045507+0000","last_became_active":"2024-09-13T14:59:53.514049+0000","last_became_peered":"2024-09-13T14:59:53.514049+0000","last_unstale":"2024-09-13T15:00:48.045507+0000","last_undegraded":"2024-09-13T15:00:48.045507+0000","last_fullsized":"2024-09-13T15:00:48.045507+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:31:53.727081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":270,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":377,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.220623+0000","last_change":"2024-09-13T14:47:47.867935+0000","last_active":"2024-09-13T15:00:49.220623+0000","last_peered":"2024-09-13T15:00:49.220623+0000","last_clean":"2024-09-13T15:00:49.220623+0000","last_became_active":"2024-09-13T14:47:47.867616+0000","last_became_peered":"2024-09-13T14:47:47.867616+0000","last_unstale":"2024-09-13T15:00:49.220623+0000","last_undegraded":"2024-09-13T15:00:49.220623+0000","last_fullsized":"2024-09-13T15:00:49.220623+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:52:38.619916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"90'149","reported_seq":797,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.044879+0000","last_change":"2024-09-13T14:57:47.798716+0000","last_active":"2024-09-13T15:00:48.044879+0000","last_peered":"2024-09-13T15:00:48.044879+0000","last_clean":"2024-09-13T15:00:48.044879+0000","last_became_active":"2024-09-13T14:57:28.526594+0000","last_became_peered":"2024-09-13T14:57:28.526594+0000","last_unstale":"2024-09-13T15:00:48.044879+0000","last_undegraded":"2024-09-13T15:00:48.044879+0000","last_fullsized":"2024-09-13T15:00:48.044879+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":471,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:47.798627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:47.798627+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:30:00.799937+0000","scrub_duration":0.0039598259999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":42,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.049246+0000","last_change":"2024-09-13T14:59:53.521057+0000","last_active":"2024-09-13T15:00:48.049246+0000","last_peered":"2024-09-13T15:00:48.049246+0000","last_clean":"2024-09-13T15:00:48.049246+0000","last_became_active":"2024-09-13T14:59:53.511100+0000","last_became_peered":"2024-09-13T14:59:53.511100+0000","last_unstale":"2024-09-13T15:00:48.049246+0000","last_undegraded":"2024-09-13T15:00:48.049246+0000","last_fullsized":"2024-09-13T15:00:48.049246+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:38:46.768037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4756,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":32,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.050519+0000","last_change":"2024-09-13T14:59:53.507755+0000","last_active":"2024-09-13T15:00:48.050519+0000","last_peered":"2024-09-13T15:00:48.050519+0000","last_clean":"2024-09-13T15:00:48.050519+0000","last_became_active":"2024-09-13T14:59:53.507484+0000","last_became_peered":"2024-09-13T14:59:53.507484+0000","last_unstale":"2024-09-13T15:00:48.050519+0000","last_undegraded":"2024-09-13T15:00:48.050519+0000","last_fullsized":"2024-09-13T15:00:48.050519+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:25.909660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.16","version":"90'149","reported_seq":810,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.221077+0000","last_change":"2024-09-13T14:57:40.798149+0000","last_active":"2024-09-13T15:00:49.221077+0000","last_peered":"2024-09-13T15:00:49.221077+0000","last_clean":"2024-09-13T15:00:49.221077+0000","last_became_active":"2024-09-13T14:57:28.528351+0000","last_became_peered":"2024-09-13T14:57:28.528351+0000","last_unstale":"2024-09-13T15:00:49.221077+0000","last_undegraded":"2024-09-13T15:00:49.221077+0000","last_fullsized":"2024-09-13T15:00:49.221077+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":471,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:40.797796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:40.797796+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:09.769513+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":32,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.050251+0000","last_change":"2024-09-13T14:59:53.507344+0000","last_active":"2024-09-13T15:00:48.050251+0000","last_peered":"2024-09-13T15:00:48.050251+0000","last_clean":"2024-09-13T15:00:48.050251+0000","last_became_active":"2024-09-13T14:59:53.507068+0000","last_became_peered":"2024-09-13T14:59:53.507068+0000","last_unstale":"2024-09-13T15:00:48.050251+0000","last_undegraded":"2024-09-13T15:00:48.050251+0000","last_fullsized":"2024-09-13T15:00:48.050251+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:47.526588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":655,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.054742+0000","last_change":"2024-09-13T15:00:49.054003+0000","last_active":"2024-09-13T15:00:49.054742+0000","last_peered":"2024-09-13T15:00:49.054742+0000","last_clean":"2024-09-13T15:00:49.054742+0000","last_became_active":"2024-09-13T15:00:49.053404+0000","last_became_peered":"2024-09-13T15:00:49.053404+0000","last_unstale":"2024-09-13T15:00:49.054742+0000","last_undegraded":"2024-09-13T15:00:49.054742+0000","last_fullsized":"2024-09-13T15:00:49.054742+0000","mapping_epoch":516,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:20.909606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":531,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.047439+0000","last_change":"2024-09-13T15:00:03.535864+0000","last_active":"2024-09-13T15:00:49.047439+0000","last_peered":"2024-09-13T15:00:49.047439+0000","last_clean":"2024-09-13T15:00:49.047439+0000","last_became_active":"2024-09-13T15:00:03.535481+0000","last_became_peered":"2024-09-13T15:00:03.535481+0000","last_unstale":"2024-09-13T15:00:49.047439+0000","last_undegraded":"2024-09-13T15:00:49.047439+0000","last_fullsized":"2024-09-13T15:00:49.047439+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:38:15.374371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"90'149","reported_seq":811,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.049029+0000","last_change":"2024-09-13T14:47:29.999643+0000","last_active":"2024-09-13T15:00:49.049029+0000","last_peered":"2024-09-13T15:00:49.049029+0000","last_clean":"2024-09-13T15:00:49.049029+0000","last_became_active":"2024-09-13T14:47:29.410951+0000","last_became_peered":"2024-09-13T14:47:29.410951+0000","last_unstale":"2024-09-13T15:00:49.049029+0000","last_undegraded":"2024-09-13T15:00:49.049029+0000","last_fullsized":"2024-09-13T15:00:49.049029+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:29.999519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:29.999519+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:45:08.824756+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":951,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.048806+0000","last_change":"2024-09-13T14:38:53.900107+0000","last_active":"2024-09-13T15:00:49.048806+0000","last_peered":"2024-09-13T15:00:49.048806+0000","last_clean":"2024-09-13T15:00:49.048806+0000","last_became_active":"2024-09-13T14:38:53.899556+0000","last_became_peered":"2024-09-13T14:38:53.899556+0000","last_unstale":"2024-09-13T15:00:49.048806+0000","last_undegraded":"2024-09-13T15:00:49.048806+0000","last_fullsized":"2024-09-13T15:00:49.048806+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:15:47.918163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":916,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.221217+0000","last_change":"2024-09-13T14:59:56.508453+0000","last_active":"2024-09-13T15:00:49.221217+0000","last_peered":"2024-09-13T15:00:49.221217+0000","last_clean":"2024-09-13T15:00:49.221217+0000","last_became_active":"2024-09-13T14:59:56.507789+0000","last_became_peered":"2024-09-13T14:59:56.507789+0000","last_unstale":"2024-09-13T15:00:49.221217+0000","last_undegraded":"2024-09-13T15:00:49.221217+0000","last_fullsized":"2024-09-13T15:00:49.221217+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":499,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:31:23.438892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":968,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.050971+0000","last_change":"2024-09-13T14:38:53.938227+0000","last_active":"2024-09-13T15:00:49.050971+0000","last_peered":"2024-09-13T15:00:49.050971+0000","last_clean":"2024-09-13T15:00:49.050971+0000","last_became_active":"2024-09-13T14:38:53.902867+0000","last_became_peered":"2024-09-13T14:38:53.902867+0000","last_unstale":"2024-09-13T15:00:49.050971+0000","last_undegraded":"2024-09-13T15:00:49.050971+0000","last_fullsized":"2024-09-13T15:00:49.050971+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:16.831756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"90'149","reported_seq":571,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.050034+0000","last_change":"2024-09-13T14:59:56.507724+0000","last_active":"2024-09-13T15:00:48.050034+0000","last_peered":"2024-09-13T15:00:48.050034+0000","last_clean":"2024-09-13T15:00:48.050034+0000","last_became_active":"2024-09-13T14:59:56.507194+0000","last_became_peered":"2024-09-13T14:59:56.507194+0000","last_unstale":"2024-09-13T15:00:48.050034+0000","last_undegraded":"2024-09-13T15:00:48.050034+0000","last_fullsized":"2024-09-13T15:00:48.050034+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":499,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:37.063494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:37.063494+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:36:50.707939+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":494,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.047366+0000","last_change":"2024-09-13T15:00:03.535624+0000","last_active":"2024-09-13T15:00:49.047366+0000","last_peered":"2024-09-13T15:00:49.047366+0000","last_clean":"2024-09-13T15:00:49.047366+0000","last_became_active":"2024-09-13T15:00:03.535351+0000","last_became_peered":"2024-09-13T15:00:03.535351+0000","last_unstale":"2024-09-13T15:00:49.047366+0000","last_undegraded":"2024-09-13T15:00:49.047366+0000","last_fullsized":"2024-09-13T15:00:49.047366+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:33:47.326298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":1121,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.046082+0000","last_change":"2024-09-13T14:59:56.510064+0000","last_active":"2024-09-13T15:00:48.046082+0000","last_peered":"2024-09-13T15:00:48.046082+0000","last_clean":"2024-09-13T15:00:48.046082+0000","last_became_active":"2024-09-13T14:59:56.508959+0000","last_became_peered":"2024-09-13T14:59:56.508959+0000","last_unstale":"2024-09-13T15:00:48.046082+0000","last_undegraded":"2024-09-13T15:00:48.046082+0000","last_fullsized":"2024-09-13T15:00:48.046082+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":499,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:26:49.194629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":378,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.220967+0000","last_change":"2024-09-13T14:47:47.876562+0000","last_active":"2024-09-13T15:00:49.220967+0000","last_peered":"2024-09-13T15:00:49.220967+0000","last_clean":"2024-09-13T15:00:49.220967+0000","last_became_active":"2024-09-13T14:47:47.876297+0000","last_became_peered":"2024-09-13T14:47:47.876297+0000","last_unstale":"2024-09-13T15:00:49.220967+0000","last_undegraded":"2024-09-13T15:00:49.220967+0000","last_fullsized":"2024-09-13T15:00:49.220967+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:03:27.761394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"90'149","reported_seq":386,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:56.559213+0000","last_change":"2024-09-13T15:00:56.559366+0000","last_active":"2024-09-13T15:00:56.559213+0000","last_peered":"2024-09-13T15:00:56.559213+0000","last_clean":"2024-09-13T15:00:56.559213+0000","last_became_active":"2024-09-13T15:00:49.050743+0000","last_became_peered":"2024-09-13T15:00:49.050743+0000","last_unstale":"2024-09-13T15:00:56.559213+0000","last_undegraded":"2024-09-13T15:00:56.559213+0000","last_fullsized":"2024-09-13T15:00:56.559213+0000","mapping_epoch":516,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T15:00:56.559183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T15:00:56.559183+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:25:50.528399+0000","scrub_duration":0.0016783429999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":392,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.220816+0000","last_change":"2024-09-13T14:59:53.503001+0000","last_active":"2024-09-13T15:00:49.220816+0000","last_peered":"2024-09-13T15:00:49.220816+0000","last_clean":"2024-09-13T15:00:49.220816+0000","last_became_active":"2024-09-13T14:59:53.502687+0000","last_became_peered":"2024-09-13T14:59:53.502687+0000","last_unstale":"2024-09-13T15:00:49.220816+0000","last_undegraded":"2024-09-13T15:00:49.220816+0000","last_fullsized":"2024-09-13T15:00:49.220816+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:02:23.981179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":1666,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.220848+0000","last_change":"2024-09-13T14:47:50.883365+0000","last_active":"2024-09-13T15:00:49.220848+0000","last_peered":"2024-09-13T15:00:49.220848+0000","last_clean":"2024-09-13T15:00:49.220848+0000","last_became_active":"2024-09-13T14:47:50.882609+0000","last_became_peered":"2024-09-13T14:47:50.882609+0000","last_unstale":"2024-09-13T15:00:49.220848+0000","last_undegraded":"2024-09-13T15:00:49.220848+0000","last_fullsized":"2024-09-13T15:00:49.220848+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:12:17.961575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":345,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.049061+0000","last_change":"2024-09-13T14:46:01.763906+0000","last_active":"2024-09-13T15:00:49.049061+0000","last_peered":"2024-09-13T15:00:49.049061+0000","last_clean":"2024-09-13T15:00:49.049061+0000","last_became_active":"2024-09-13T14:46:01.763542+0000","last_became_peered":"2024-09-13T14:46:01.763542+0000","last_unstale":"2024-09-13T15:00:49.049061+0000","last_undegraded":"2024-09-13T15:00:49.049061+0000","last_fullsized":"2024-09-13T15:00:49.049061+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:09:36.652767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"90'149","reported_seq":386,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:01:01.504353+0000","last_change":"2024-09-13T15:01:01.504440+0000","last_active":"2024-09-13T15:01:01.504353+0000","last_peered":"2024-09-13T15:01:01.504353+0000","last_clean":"2024-09-13T15:01:01.504353+0000","last_became_active":"2024-09-13T15:00:49.052038+0000","last_became_peered":"2024-09-13T15:00:49.052038+0000","last_unstale":"2024-09-13T15:01:01.504353+0000","last_undegraded":"2024-09-13T15:01:01.504353+0000","last_fullsized":"2024-09-13T15:01:01.504353+0000","mapping_epoch":516,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T15:01:01.504336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T15:01:01.504336+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:41:09.571281+0000","scrub_duration":0.0020290070000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":378,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.220748+0000","last_change":"2024-09-13T14:47:47.877272+0000","last_active":"2024-09-13T15:00:49.220748+0000","last_peered":"2024-09-13T15:00:49.220748+0000","last_clean":"2024-09-13T15:00:49.220748+0000","last_became_active":"2024-09-13T14:47:47.877003+0000","last_became_peered":"2024-09-13T14:47:47.877003+0000","last_unstale":"2024-09-13T15:00:49.220748+0000","last_undegraded":"2024-09-13T15:00:49.220748+0000","last_fullsized":"2024-09-13T15:00:49.220748+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:47:23.172452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":32,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.050662+0000","last_change":"2024-09-13T14:59:53.507662+0000","last_active":"2024-09-13T15:00:48.050662+0000","last_peered":"2024-09-13T15:00:48.050662+0000","last_clean":"2024-09-13T15:00:48.050662+0000","last_became_active":"2024-09-13T14:59:53.507388+0000","last_became_peered":"2024-09-13T14:59:53.507388+0000","last_unstale":"2024-09-13T15:00:48.050662+0000","last_undegraded":"2024-09-13T15:00:48.050662+0000","last_fullsized":"2024-09-13T15:00:48.050662+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:20:47.678025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":1276,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.048703+0000","last_change":"2024-09-13T14:47:50.892842+0000","last_active":"2024-09-13T15:00:49.048703+0000","last_peered":"2024-09-13T15:00:49.048703+0000","last_clean":"2024-09-13T15:00:49.048703+0000","last_became_active":"2024-09-13T14:47:50.892237+0000","last_became_peered":"2024-09-13T14:47:50.892237+0000","last_unstale":"2024-09-13T15:00:49.048703+0000","last_undegraded":"2024-09-13T15:00:49.048703+0000","last_fullsized":"2024-09-13T15:00:49.048703+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:20:05.796762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":8388646,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"90'149","reported_seq":621,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.047926+0000","last_change":"2024-09-13T14:47:32.275753+0000","last_active":"2024-09-13T15:00:49.047926+0000","last_peered":"2024-09-13T15:00:49.047926+0000","last_clean":"2024-09-13T15:00:49.047926+0000","last_became_active":"2024-09-13T14:47:29.401066+0000","last_became_peered":"2024-09-13T14:47:29.401066+0000","last_unstale":"2024-09-13T15:00:49.047926+0000","last_undegraded":"2024-09-13T15:00:49.047926+0000","last_fullsized":"2024-09-13T15:00:49.047926+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:32.275562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:32.275562+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:43:41.056775+0000","scrub_duration":0.0054674260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":378,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.220992+0000","last_change":"2024-09-13T14:47:47.870218+0000","last_active":"2024-09-13T15:00:49.220992+0000","last_peered":"2024-09-13T15:00:49.220992+0000","last_clean":"2024-09-13T15:00:49.220992+0000","last_became_active":"2024-09-13T14:47:47.869907+0000","last_became_peered":"2024-09-13T14:47:47.869907+0000","last_unstale":"2024-09-13T15:00:49.220992+0000","last_undegraded":"2024-09-13T15:00:49.220992+0000","last_fullsized":"2024-09-13T15:00:49.220992+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:41:49.141441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":378,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.220479+0000","last_change":"2024-09-13T14:47:47.866684+0000","last_active":"2024-09-13T15:00:49.220479+0000","last_peered":"2024-09-13T15:00:49.220479+0000","last_clean":"2024-09-13T15:00:49.220479+0000","last_became_active":"2024-09-13T14:47:47.866400+0000","last_became_peered":"2024-09-13T14:47:47.866400+0000","last_unstale":"2024-09-13T15:00:49.220479+0000","last_undegraded":"2024-09-13T15:00:49.220479+0000","last_fullsized":"2024-09-13T15:00:49.220479+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:08:34.289658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":861,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.050734+0000","last_change":"2024-09-13T14:46:04.782955+0000","last_active":"2024-09-13T15:00:49.050734+0000","last_peered":"2024-09-13T15:00:49.050734+0000","last_clean":"2024-09-13T15:00:49.050734+0000","last_became_active":"2024-09-13T14:46:04.782250+0000","last_became_peered":"2024-09-13T14:46:04.782250+0000","last_unstale":"2024-09-13T15:00:49.050734+0000","last_undegraded":"2024-09-13T15:00:49.050734+0000","last_fullsized":"2024-09-13T15:00:49.050734+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:24.596922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"90'149","reported_seq":753,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.044902+0000","last_change":"2024-09-13T14:59:56.509314+0000","last_active":"2024-09-13T15:00:49.044902+0000","last_peered":"2024-09-13T15:00:49.044902+0000","last_clean":"2024-09-13T15:00:49.044902+0000","last_became_active":"2024-09-13T14:59:56.508996+0000","last_became_peered":"2024-09-13T14:59:56.508996+0000","last_unstale":"2024-09-13T15:00:49.044902+0000","last_undegraded":"2024-09-13T15:00:49.044902+0000","last_fullsized":"2024-09-13T15:00:49.044902+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":499,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:29.629342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:29.629342+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:10:47.014236+0000","scrub_duration":0.0099290990000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":515,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.047552+0000","last_change":"2024-09-13T15:00:03.538661+0000","last_active":"2024-09-13T15:00:49.047552+0000","last_peered":"2024-09-13T15:00:49.047552+0000","last_clean":"2024-09-13T15:00:49.047552+0000","last_became_active":"2024-09-13T15:00:03.538114+0000","last_became_peered":"2024-09-13T15:00:03.538114+0000","last_unstale":"2024-09-13T15:00:49.047552+0000","last_undegraded":"2024-09-13T15:00:49.047552+0000","last_fullsized":"2024-09-13T15:00:49.047552+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:41:09.096859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":501,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.047070+0000","last_change":"2024-09-13T15:00:03.537417+0000","last_active":"2024-09-13T15:00:49.047070+0000","last_peered":"2024-09-13T15:00:49.047070+0000","last_clean":"2024-09-13T15:00:49.047070+0000","last_became_active":"2024-09-13T15:00:03.536451+0000","last_became_peered":"2024-09-13T15:00:03.536451+0000","last_unstale":"2024-09-13T15:00:49.047070+0000","last_undegraded":"2024-09-13T15:00:49.047070+0000","last_fullsized":"2024-09-13T15:00:49.047070+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:39:18.772336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":772,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.044260+0000","last_change":"2024-09-13T14:47:47.892973+0000","last_active":"2024-09-13T15:00:48.044260+0000","last_peered":"2024-09-13T15:00:48.044260+0000","last_clean":"2024-09-13T15:00:48.044260+0000","last_became_active":"2024-09-13T14:47:47.866986+0000","last_became_peered":"2024-09-13T14:47:47.866986+0000","last_unstale":"2024-09-13T15:00:48.044260+0000","last_undegraded":"2024-09-13T15:00:48.044260+0000","last_fullsized":"2024-09-13T15:00:48.044260+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:07.353290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"90'149","reported_seq":830,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.220754+0000","last_change":"2024-09-13T14:59:56.508952+0000","last_active":"2024-09-13T15:00:49.220754+0000","last_peered":"2024-09-13T15:00:49.220754+0000","last_clean":"2024-09-13T15:00:49.220754+0000","last_became_active":"2024-09-13T14:59:56.508586+0000","last_became_peered":"2024-09-13T14:59:56.508586+0000","last_unstale":"2024-09-13T15:00:49.220754+0000","last_undegraded":"2024-09-13T15:00:49.220754+0000","last_fullsized":"2024-09-13T15:00:49.220754+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":499,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:34.603316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:34.603316+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:00:46.717723+0000","scrub_duration":0.0057131120000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":149,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.052386+0000","last_change":"2024-09-13T15:00:49.051643+0000","last_active":"2024-09-13T15:00:49.052386+0000","last_peered":"2024-09-13T15:00:49.052386+0000","last_clean":"2024-09-13T15:00:49.052386+0000","last_became_active":"2024-09-13T15:00:49.051324+0000","last_became_peered":"2024-09-13T15:00:49.051324+0000","last_unstale":"2024-09-13T15:00:49.052386+0000","last_undegraded":"2024-09-13T15:00:49.052386+0000","last_fullsized":"2024-09-13T15:00:49.052386+0000","mapping_epoch":516,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:24:20.674659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":877,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.049759+0000","last_change":"2024-09-13T14:37:49.160719+0000","last_active":"2024-09-13T15:00:49.049759+0000","last_peered":"2024-09-13T15:00:49.049759+0000","last_clean":"2024-09-13T15:00:49.049759+0000","last_became_active":"2024-09-13T14:37:49.160401+0000","last_became_peered":"2024-09-13T14:37:49.160401+0000","last_unstale":"2024-09-13T15:00:49.049759+0000","last_undegraded":"2024-09-13T15:00:49.049759+0000","last_fullsized":"2024-09-13T15:00:49.049759+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:21.737945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":831,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.048638+0000","last_change":"2024-09-13T14:46:04.780718+0000","last_active":"2024-09-13T15:00:49.048638+0000","last_peered":"2024-09-13T15:00:49.048638+0000","last_clean":"2024-09-13T15:00:49.048638+0000","last_became_active":"2024-09-13T14:46:04.780120+0000","last_became_peered":"2024-09-13T14:46:04.780120+0000","last_unstale":"2024-09-13T15:00:49.048638+0000","last_undegraded":"2024-09-13T15:00:49.048638+0000","last_fullsized":"2024-09-13T15:00:49.048638+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:57:14.475888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":702,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.044355+0000","last_change":"2024-09-13T14:47:31.684319+0000","last_active":"2024-09-13T15:00:48.044355+0000","last_peered":"2024-09-13T15:00:48.044355+0000","last_clean":"2024-09-13T15:00:48.044355+0000","last_became_active":"2024-09-13T14:47:29.415658+0000","last_became_peered":"2024-09-13T14:47:29.415658+0000","last_unstale":"2024-09-13T15:00:48.044355+0000","last_undegraded":"2024-09-13T15:00:48.044355+0000","last_fullsized":"2024-09-13T15:00:48.044355+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:31.681433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:31.681433+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:29:34.663513+0000","scrub_duration":0.0059601139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":392,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.220883+0000","last_change":"2024-09-13T14:59:53.502354+0000","last_active":"2024-09-13T15:00:49.220883+0000","last_peered":"2024-09-13T15:00:49.220883+0000","last_clean":"2024-09-13T15:00:49.220883+0000","last_became_active":"2024-09-13T14:59:53.502017+0000","last_became_peered":"2024-09-13T14:59:53.502017+0000","last_unstale":"2024-09-13T15:00:49.220883+0000","last_undegraded":"2024-09-13T15:00:49.220883+0000","last_fullsized":"2024-09-13T15:00:49.220883+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:36:58.220200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":949,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.048895+0000","last_change":"2024-09-13T14:47:47.861249+0000","last_active":"2024-09-13T15:00:49.048895+0000","last_peered":"2024-09-13T15:00:49.048895+0000","last_clean":"2024-09-13T15:00:49.048895+0000","last_became_active":"2024-09-13T14:47:47.860873+0000","last_became_peered":"2024-09-13T14:47:47.860873+0000","last_unstale":"2024-09-13T15:00:49.048895+0000","last_undegraded":"2024-09-13T15:00:49.048895+0000","last_fullsized":"2024-09-13T15:00:49.048895+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:37:50.709437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"90'149","reported_seq":620,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.044011+0000","last_change":"2024-09-13T14:57:38.849773+0000","last_active":"2024-09-13T15:00:48.044011+0000","last_peered":"2024-09-13T15:00:48.044011+0000","last_clean":"2024-09-13T15:00:48.044011+0000","last_became_active":"2024-09-13T14:57:17.378580+0000","last_became_peered":"2024-09-13T14:57:17.378580+0000","last_unstale":"2024-09-13T15:00:48.044011+0000","last_undegraded":"2024-09-13T15:00:48.044011+0000","last_fullsized":"2024-09-13T15:00:48.044011+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:38.849610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:38.849610+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:22:24.723639+0000","scrub_duration":0.0034255890000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.049686+0000","last_change":"2024-09-13T14:59:53.505482+0000","last_active":"2024-09-13T15:00:48.049686+0000","last_peered":"2024-09-13T15:00:48.049686+0000","last_clean":"2024-09-13T15:00:48.049686+0000","last_became_active":"2024-09-13T14:59:53.505200+0000","last_became_peered":"2024-09-13T14:59:53.505200+0000","last_unstale":"2024-09-13T15:00:48.049686+0000","last_undegraded":"2024-09-13T15:00:48.049686+0000","last_fullsized":"2024-09-13T15:00:48.049686+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:50.788322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":32,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.050136+0000","last_change":"2024-09-13T14:59:53.511732+0000","last_active":"2024-09-13T15:00:48.050136+0000","last_peered":"2024-09-13T15:00:48.050136+0000","last_clean":"2024-09-13T15:00:48.050136+0000","last_became_active":"2024-09-13T14:59:53.511351+0000","last_became_peered":"2024-09-13T14:59:53.511351+0000","last_unstale":"2024-09-13T15:00:48.050136+0000","last_undegraded":"2024-09-13T15:00:48.050136+0000","last_fullsized":"2024-09-13T15:00:48.050136+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:48:55.490615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"90'149","reported_seq":845,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.220636+0000","last_change":"2024-09-13T14:57:39.766173+0000","last_active":"2024-09-13T15:00:49.220636+0000","last_peered":"2024-09-13T15:00:49.220636+0000","last_clean":"2024-09-13T15:00:49.220636+0000","last_became_active":"2024-09-13T14:57:17.389370+0000","last_became_peered":"2024-09-13T14:57:17.389370+0000","last_unstale":"2024-09-13T15:00:49.220636+0000","last_undegraded":"2024-09-13T15:00:49.220636+0000","last_fullsized":"2024-09-13T15:00:49.220636+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:39.765978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:39.765978+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:39.980756+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":32,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.050236+0000","last_change":"2024-09-13T14:59:53.507082+0000","last_active":"2024-09-13T15:00:48.050236+0000","last_peered":"2024-09-13T15:00:48.050236+0000","last_clean":"2024-09-13T15:00:48.050236+0000","last_became_active":"2024-09-13T14:59:53.506615+0000","last_became_peered":"2024-09-13T14:59:53.506615+0000","last_unstale":"2024-09-13T15:00:48.050236+0000","last_undegraded":"2024-09-13T15:00:48.050236+0000","last_fullsized":"2024-09-13T15:00:48.050236+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:50:49.491733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":270,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.222488+0000","last_change":"2024-09-13T15:00:49.221849+0000","last_active":"2024-09-13T15:00:49.222488+0000","last_peered":"2024-09-13T15:00:49.222488+0000","last_clean":"2024-09-13T15:00:49.222488+0000","last_became_active":"2024-09-13T15:00:49.221548+0000","last_became_peered":"2024-09-13T15:00:49.221548+0000","last_unstale":"2024-09-13T15:00:49.222488+0000","last_undegraded":"2024-09-13T15:00:49.222488+0000","last_fullsized":"2024-09-13T15:00:49.222488+0000","mapping_epoch":516,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:56:10.688533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"90'149","reported_seq":620,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.044660+0000","last_change":"2024-09-13T14:57:39.812785+0000","last_active":"2024-09-13T15:00:48.044660+0000","last_peered":"2024-09-13T15:00:48.044660+0000","last_clean":"2024-09-13T15:00:48.044660+0000","last_became_active":"2024-09-13T14:57:17.383307+0000","last_became_peered":"2024-09-13T14:57:17.383307+0000","last_unstale":"2024-09-13T15:00:48.044660+0000","last_undegraded":"2024-09-13T15:00:48.044660+0000","last_fullsized":"2024-09-13T15:00:48.044660+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:39.812688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:39.812688+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:18:59.726465+0000","scrub_duration":0.010496065000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":32,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.049158+0000","last_change":"2024-09-13T14:59:53.507167+0000","last_active":"2024-09-13T15:00:48.049158+0000","last_peered":"2024-09-13T15:00:48.049158+0000","last_clean":"2024-09-13T15:00:48.049158+0000","last_became_active":"2024-09-13T14:59:53.506358+0000","last_became_peered":"2024-09-13T14:59:53.506358+0000","last_unstale":"2024-09-13T15:00:48.049158+0000","last_undegraded":"2024-09-13T15:00:48.049158+0000","last_fullsized":"2024-09-13T15:00:48.049158+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:54:19.042442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":946,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.049165+0000","last_change":"2024-09-13T14:59:53.520257+0000","last_active":"2024-09-13T15:00:49.049165+0000","last_peered":"2024-09-13T15:00:49.049165+0000","last_clean":"2024-09-13T15:00:49.049165+0000","last_became_active":"2024-09-13T14:59:53.506185+0000","last_became_peered":"2024-09-13T14:59:53.506185+0000","last_unstale":"2024-09-13T15:00:49.049165+0000","last_undegraded":"2024-09-13T15:00:49.049165+0000","last_fullsized":"2024-09-13T15:00:49.049165+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:26.417709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"90'149","reported_seq":717,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.044013+0000","last_change":"2024-09-13T14:57:41.926732+0000","last_active":"2024-09-13T15:00:48.044013+0000","last_peered":"2024-09-13T15:00:48.044013+0000","last_clean":"2024-09-13T15:00:48.044013+0000","last_became_active":"2024-09-13T14:57:17.380843+0000","last_became_peered":"2024-09-13T14:57:17.380843+0000","last_unstale":"2024-09-13T15:00:48.044013+0000","last_undegraded":"2024-09-13T15:00:48.044013+0000","last_fullsized":"2024-09-13T15:00:48.044013+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:41.926626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:41.926626+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:08:57.137123+0000","scrub_duration":0.0019476179999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":537,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.047255+0000","last_change":"2024-09-13T15:00:03.537200+0000","last_active":"2024-09-13T15:00:49.047255+0000","last_peered":"2024-09-13T15:00:49.047255+0000","last_clean":"2024-09-13T15:00:49.047255+0000","last_became_active":"2024-09-13T15:00:03.536871+0000","last_became_peered":"2024-09-13T15:00:03.536871+0000","last_unstale":"2024-09-13T15:00:49.047255+0000","last_undegraded":"2024-09-13T15:00:49.047255+0000","last_fullsized":"2024-09-13T15:00:49.047255+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:02:58.780844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":335,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.048087+0000","last_change":"2024-09-13T14:37:49.168003+0000","last_active":"2024-09-13T15:00:49.048087+0000","last_peered":"2024-09-13T15:00:49.048087+0000","last_clean":"2024-09-13T15:00:49.048087+0000","last_became_active":"2024-09-13T14:37:49.167638+0000","last_became_peered":"2024-09-13T14:37:49.167638+0000","last_unstale":"2024-09-13T15:00:49.048087+0000","last_undegraded":"2024-09-13T15:00:49.048087+0000","last_fullsized":"2024-09-13T15:00:49.048087+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:29.051981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"90'149","reported_seq":485,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.049785+0000","last_change":"2024-09-13T14:59:59.848229+0000","last_active":"2024-09-13T15:00:48.049785+0000","last_peered":"2024-09-13T15:00:48.049785+0000","last_clean":"2024-09-13T15:00:48.049785+0000","last_became_active":"2024-09-13T14:59:56.506696+0000","last_became_peered":"2024-09-13T14:59:56.506696+0000","last_unstale":"2024-09-13T15:00:48.049785+0000","last_undegraded":"2024-09-13T15:00:48.049785+0000","last_fullsized":"2024-09-13T15:00:48.049785+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":499,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:59:59.848136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:59:59.848136+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:17:35.075355+0000","scrub_duration":0.0018058359999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":191,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.222458+0000","last_change":"2024-09-13T15:00:49.221793+0000","last_active":"2024-09-13T15:00:49.222458+0000","last_peered":"2024-09-13T15:00:49.222458+0000","last_clean":"2024-09-13T15:00:49.222458+0000","last_became_active":"2024-09-13T15:00:49.221474+0000","last_became_peered":"2024-09-13T15:00:49.221474+0000","last_unstale":"2024-09-13T15:00:49.222458+0000","last_undegraded":"2024-09-13T15:00:49.222458+0000","last_fullsized":"2024-09-13T15:00:49.222458+0000","mapping_epoch":516,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:18:18.489468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":525,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.046106+0000","last_change":"2024-09-13T14:37:49.611774+0000","last_active":"2024-09-13T15:00:49.046106+0000","last_peered":"2024-09-13T15:00:49.046106+0000","last_clean":"2024-09-13T15:00:49.046106+0000","last_became_active":"2024-09-13T14:37:49.611088+0000","last_became_peered":"2024-09-13T14:37:49.611088+0000","last_unstale":"2024-09-13T15:00:49.046106+0000","last_undegraded":"2024-09-13T15:00:49.046106+0000","last_fullsized":"2024-09-13T15:00:49.046106+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:08.933213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"90'149","reported_seq":386,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:53.542069+0000","last_change":"2024-09-13T15:00:53.542206+0000","last_active":"2024-09-13T15:00:53.542069+0000","last_peered":"2024-09-13T15:00:53.542069+0000","last_clean":"2024-09-13T15:00:53.542069+0000","last_became_active":"2024-09-13T15:00:49.054039+0000","last_became_peered":"2024-09-13T15:00:49.054039+0000","last_unstale":"2024-09-13T15:00:53.542069+0000","last_undegraded":"2024-09-13T15:00:53.542069+0000","last_fullsized":"2024-09-13T15:00:53.542069+0000","mapping_epoch":516,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T15:00:53.542043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T15:00:53.542043+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:52:36.487371+0000","scrub_duration":0.0016316410000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":32,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.049833+0000","last_change":"2024-09-13T14:59:53.511092+0000","last_active":"2024-09-13T15:00:48.049833+0000","last_peered":"2024-09-13T15:00:48.049833+0000","last_clean":"2024-09-13T15:00:48.049833+0000","last_became_active":"2024-09-13T14:59:53.509999+0000","last_became_peered":"2024-09-13T14:59:53.509999+0000","last_unstale":"2024-09-13T15:00:48.049833+0000","last_undegraded":"2024-09-13T15:00:48.049833+0000","last_fullsized":"2024-09-13T15:00:48.049833+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:22:05.834497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":327,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.046477+0000","last_change":"2024-09-13T14:59:53.521704+0000","last_active":"2024-09-13T15:00:49.046477+0000","last_peered":"2024-09-13T15:00:49.046477+0000","last_clean":"2024-09-13T15:00:49.046477+0000","last_became_active":"2024-09-13T14:59:53.512700+0000","last_became_peered":"2024-09-13T14:59:53.512700+0000","last_unstale":"2024-09-13T15:00:49.046477+0000","last_undegraded":"2024-09-13T15:00:49.046477+0000","last_fullsized":"2024-09-13T15:00:49.046477+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:46:39.342033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"90'149","reported_seq":852,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.049126+0000","last_change":"2024-09-13T14:57:40.996705+0000","last_active":"2024-09-13T15:00:49.049126+0000","last_peered":"2024-09-13T15:00:49.049126+0000","last_clean":"2024-09-13T15:00:49.049126+0000","last_became_active":"2024-09-13T14:57:17.404299+0000","last_became_peered":"2024-09-13T14:57:17.404299+0000","last_unstale":"2024-09-13T15:00:49.049126+0000","last_undegraded":"2024-09-13T15:00:49.049126+0000","last_fullsized":"2024-09-13T15:00:49.049126+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:40.996402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:40.996402+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:04.912532+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":344,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.044834+0000","last_change":"2024-09-13T14:38:53.898907+0000","last_active":"2024-09-13T15:00:48.044834+0000","last_peered":"2024-09-13T15:00:48.044834+0000","last_clean":"2024-09-13T15:00:48.044834+0000","last_became_active":"2024-09-13T14:38:53.898572+0000","last_became_peered":"2024-09-13T14:38:53.898572+0000","last_unstale":"2024-09-13T15:00:48.044834+0000","last_undegraded":"2024-09-13T15:00:48.044834+0000","last_fullsized":"2024-09-13T15:00:48.044834+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:29.222150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":314,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.046842+0000","last_change":"2024-09-13T14:59:53.517392+0000","last_active":"2024-09-13T15:00:49.046842+0000","last_peered":"2024-09-13T15:00:49.046842+0000","last_clean":"2024-09-13T15:00:49.046842+0000","last_became_active":"2024-09-13T14:59:53.505025+0000","last_became_peered":"2024-09-13T14:59:53.505025+0000","last_unstale":"2024-09-13T15:00:49.046842+0000","last_undegraded":"2024-09-13T15:00:49.046842+0000","last_fullsized":"2024-09-13T15:00:49.046842+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:51:58.294980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"90'149","reported_seq":661,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.045100+0000","last_change":"2024-09-13T14:59:56.510267+0000","last_active":"2024-09-13T15:00:49.045100+0000","last_peered":"2024-09-13T15:00:49.045100+0000","last_clean":"2024-09-13T15:00:49.045100+0000","last_became_active":"2024-09-13T14:59:56.509734+0000","last_became_peered":"2024-09-13T14:59:56.509734+0000","last_unstale":"2024-09-13T15:00:49.045100+0000","last_undegraded":"2024-09-13T15:00:49.045100+0000","last_fullsized":"2024-09-13T15:00:49.045100+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":499,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:39.030538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:39.030538+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:12:01.212385+0000","scrub_duration":0.014515093999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":191,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.052774+0000","last_change":"2024-09-13T15:00:49.050246+0000","last_active":"2024-09-13T15:00:49.052774+0000","last_peered":"2024-09-13T15:00:49.052774+0000","last_clean":"2024-09-13T15:00:49.052774+0000","last_became_active":"2024-09-13T15:00:49.049440+0000","last_became_peered":"2024-09-13T15:00:49.049440+0000","last_unstale":"2024-09-13T15:00:49.052774+0000","last_undegraded":"2024-09-13T15:00:49.052774+0000","last_fullsized":"2024-09-13T15:00:49.052774+0000","mapping_epoch":516,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:48:01.002498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":975,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.051134+0000","last_change":"2024-09-13T14:50:27.491166+0000","last_active":"2024-09-13T15:00:49.051134+0000","last_peered":"2024-09-13T15:00:49.051134+0000","last_clean":"2024-09-13T15:00:49.051134+0000","last_became_active":"2024-09-13T14:50:27.489905+0000","last_became_peered":"2024-09-13T14:50:27.489905+0000","last_unstale":"2024-09-13T15:00:49.051134+0000","last_undegraded":"2024-09-13T15:00:49.051134+0000","last_fullsized":"2024-09-13T15:00:49.051134+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:32.237904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"90'149","reported_seq":603,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.044285+0000","last_change":"2024-09-13T14:47:50.892667+0000","last_active":"2024-09-13T15:00:48.044285+0000","last_peered":"2024-09-13T15:00:48.044285+0000","last_clean":"2024-09-13T15:00:48.044285+0000","last_became_active":"2024-09-13T14:47:50.883312+0000","last_became_peered":"2024-09-13T14:47:50.883312+0000","last_unstale":"2024-09-13T15:00:48.044285+0000","last_undegraded":"2024-09-13T15:00:48.044285+0000","last_fullsized":"2024-09-13T15:00:48.044285+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:45.441819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:45.441819+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:39.138573+0000","scrub_duration":0.021368558999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":512,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.044238+0000","last_change":"2024-09-13T14:59:53.513320+0000","last_active":"2024-09-13T15:00:48.044238+0000","last_peered":"2024-09-13T15:00:48.044238+0000","last_clean":"2024-09-13T15:00:48.044238+0000","last_became_active":"2024-09-13T14:59:53.513034+0000","last_became_peered":"2024-09-13T14:59:53.513034+0000","last_unstale":"2024-09-13T15:00:48.044238+0000","last_undegraded":"2024-09-13T15:00:48.044238+0000","last_fullsized":"2024-09-13T15:00:48.044238+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:33:20.466928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":500,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.047547+0000","last_change":"2024-09-13T15:00:03.538450+0000","last_active":"2024-09-13T15:00:49.047547+0000","last_peered":"2024-09-13T15:00:49.047547+0000","last_clean":"2024-09-13T15:00:49.047547+0000","last_became_active":"2024-09-13T15:00:03.537992+0000","last_became_peered":"2024-09-13T15:00:03.537992+0000","last_unstale":"2024-09-13T15:00:49.047547+0000","last_undegraded":"2024-09-13T15:00:49.047547+0000","last_fullsized":"2024-09-13T15:00:49.047547+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:53:14.916796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"90'149","reported_seq":822,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.220923+0000","last_change":"2024-09-13T14:59:56.508891+0000","last_active":"2024-09-13T15:00:49.220923+0000","last_peered":"2024-09-13T15:00:49.220923+0000","last_clean":"2024-09-13T15:00:49.220923+0000","last_became_active":"2024-09-13T14:59:56.508530+0000","last_became_peered":"2024-09-13T14:59:56.508530+0000","last_unstale":"2024-09-13T15:00:49.220923+0000","last_undegraded":"2024-09-13T15:00:49.220923+0000","last_fullsized":"2024-09-13T15:00:49.220923+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":499,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:51.226748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:51.226748+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:16.784573+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":500,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.047018+0000","last_change":"2024-09-13T15:00:03.536672+0000","last_active":"2024-09-13T15:00:49.047018+0000","last_peered":"2024-09-13T15:00:49.047018+0000","last_clean":"2024-09-13T15:00:49.047018+0000","last_became_active":"2024-09-13T15:00:03.536379+0000","last_became_peered":"2024-09-13T15:00:03.536379+0000","last_unstale":"2024-09-13T15:00:49.047018+0000","last_undegraded":"2024-09-13T15:00:49.047018+0000","last_fullsized":"2024-09-13T15:00:49.047018+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:04:59.498496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":323,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.044586+0000","last_change":"2024-09-13T14:36:13.862181+0000","last_active":"2024-09-13T15:00:48.044586+0000","last_peered":"2024-09-13T15:00:48.044586+0000","last_clean":"2024-09-13T15:00:48.044586+0000","last_became_active":"2024-09-13T14:36:13.861873+0000","last_became_peered":"2024-09-13T14:36:13.861873+0000","last_unstale":"2024-09-13T15:00:48.044586+0000","last_undegraded":"2024-09-13T15:00:48.044586+0000","last_fullsized":"2024-09-13T15:00:48.044586+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:14.085700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"90'149","reported_seq":586,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.043939+0000","last_change":"2024-09-13T14:47:41.482552+0000","last_active":"2024-09-13T15:00:48.043939+0000","last_peered":"2024-09-13T15:00:48.043939+0000","last_clean":"2024-09-13T15:00:48.043939+0000","last_became_active":"2024-09-13T14:47:38.466659+0000","last_became_peered":"2024-09-13T14:47:38.466659+0000","last_unstale":"2024-09-13T15:00:48.043939+0000","last_undegraded":"2024-09-13T15:00:48.043939+0000","last_fullsized":"2024-09-13T15:00:48.043939+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:41.482364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:41.482364+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:50:49.239389+0000","scrub_duration":0.027843855000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":572,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.045360+0000","last_change":"2024-09-13T14:59:53.524423+0000","last_active":"2024-09-13T15:00:48.045360+0000","last_peered":"2024-09-13T15:00:48.045360+0000","last_clean":"2024-09-13T15:00:48.045360+0000","last_became_active":"2024-09-13T14:59:53.512737+0000","last_became_peered":"2024-09-13T14:59:53.512737+0000","last_unstale":"2024-09-13T15:00:48.045360+0000","last_undegraded":"2024-09-13T15:00:48.045360+0000","last_fullsized":"2024-09-13T15:00:48.045360+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:40:11.840478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":136,"num_keys_recovered":40,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":949,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.050261+0000","last_change":"2024-09-13T14:47:47.868783+0000","last_active":"2024-09-13T15:00:49.050261+0000","last_peered":"2024-09-13T15:00:49.050261+0000","last_clean":"2024-09-13T15:00:49.050261+0000","last_became_active":"2024-09-13T14:47:47.868478+0000","last_became_peered":"2024-09-13T14:47:47.868478+0000","last_unstale":"2024-09-13T15:00:49.050261+0000","last_undegraded":"2024-09-13T15:00:49.050261+0000","last_fullsized":"2024-09-13T15:00:49.050261+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:27:01.784142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"90'149","reported_seq":707,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.047035+0000","last_change":"2024-09-13T15:00:03.537967+0000","last_active":"2024-09-13T15:00:49.047035+0000","last_peered":"2024-09-13T15:00:49.047035+0000","last_clean":"2024-09-13T15:00:49.047035+0000","last_became_active":"2024-09-13T15:00:03.536110+0000","last_became_peered":"2024-09-13T15:00:03.536110+0000","last_unstale":"2024-09-13T15:00:49.047035+0000","last_undegraded":"2024-09-13T15:00:49.047035+0000","last_fullsized":"2024-09-13T15:00:49.047035+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:38.688341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.688341+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:46:23.002387+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":7202,"num_keys_recovered":40,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25132,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":8388646,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4619,"ondisk_log_size":4619,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234613,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":958364,"kb_used_data":6796,"kb_used_omap":1,"kb_used_meta":951550,"kb_avail":92786788,"statfs":{"total":95995035648,"available":95013670912,"internally_reserved":0,"allocated":6959104,"data_stored":4343398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":974387659},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":239,"seq":1026497184031,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":699208,"kb_used_data":10944,"kb_used_omap":8,"kb_used_meta":688247,"kb_avail":93045944,"statfs":{"total":95995035648,"available":95279046656,"internally_reserved":0,"allocated":11206656,"data_stored":8602625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8453,"internal_metadata":704765691},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757878,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1258440,"kb_used_data":10876,"kb_used_omap":1,"kb_used_meta":1247550,"kb_avail":92486712,"statfs":{"total":95995035648,"available":94706393088,"internally_reserved":0,"allocated":11137024,"data_stored":8533401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1277491654},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019522,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":908052,"kb_used_data":6532,"kb_used_omap":1,"kb_used_meta":901502,"kb_avail":92837100,"statfs":{"total":95995035648,"available":95065190400,"internally_reserved":0,"allocated":6688768,"data_stored":4080698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":923138502},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435705,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1167932,"kb_used_data":6836,"kb_used_omap":5,"kb_used_meta":1161082,"kb_avail":92577220,"statfs":{"total":95995035648,"available":94799073280,"internally_reserved":0,"allocated":7000064,"data_stored":4403851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":1188948094},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":104,"seq":446676599233,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":507720,"kb_used_data":6780,"kb_used_omap":7,"kb_used_meta":500920,"kb_avail":93237432,"statfs":{"total":95995035648,"available":95475130368,"internally_reserved":0,"allocated":6942720,"data_stored":4339385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8010,"internal_metadata":512942262},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":99,"seq":425201762759,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":519872,"kb_used_data":10608,"kb_used_omap":7,"kb_used_meta":509240,"kb_avail":93225280,"statfs":{"total":95995035648,"available":95462686720,"internally_reserved":0,"allocated":10862592,"data_stored":8270131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7759,"internal_metadata":521462193},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926283,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1209608,"kb_used_data":6840,"kb_used_omap":7,"kb_used_meta":1202744,"kb_avail":92535544,"statfs":{"total":95995035648,"available":94756397056,"internally_reserved":0,"allocated":7004160,"data_stored":4404365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":1231610152},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6860,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T15:01:35.502 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 08cee502-71d9-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T15:01:35.890 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:36.569 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 15:01:36 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: ::ffff:172.21.15.195 - - [13/Sep/2024:15:01:36] "GET /metrics HTTP/1.1" 200 34149 "" "Prometheus/2.51.0" 2024-09-13T15:01:37.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:01:36 smithi102 ceph-mon[110183]: pgmap v1755: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:37.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:01:36 smithi102 ceph-mon[115269]: pgmap v1755: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:37.247 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:36 smithi195 ceph-mon[104262]: pgmap v1755: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:38.625 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-09-13T15:01:38.626 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-09-13T15:01:39.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:01:38 smithi102 ceph-mon[110183]: pgmap v1756: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:39.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:01:38 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T15:01:39.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:01:38 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T15:01:39.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:01:38 smithi102 ceph-mon[115269]: pgmap v1756: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:39.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:01:38 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T15:01:39.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:01:38 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T15:01:39.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:38 smithi195 ceph-mon[104262]: pgmap v1756: 103 pgs: 103 active+clean; 4.6 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-13T15:01:39.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:38 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T15:01:39.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:38 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T15:01:39.436 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":1756,"stamp":"2024-09-13T15:01:37.881222+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790245,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":908,"num_write_kb":12625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":257,"num_bytes_recovered":8395848,"num_keys_recovered":40,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5652,"ondisk_log_size":5652,"up":309,"acting":309,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":309,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":7229196,"kb_used_data":66212,"kb_used_omap":41,"kb_used_meta":7162838,"kb_avail":742732020,"statfs":{"total":767960285184,"available":760557588480,"internally_reserved":0,"allocated":67801088,"data_stored":46977854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42913,"internal_metadata":7334746207},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002853"},"pg_stats":[{"pgid":"3.1f","version":"71'4","reported_seq":513,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.047466+0000","last_change":"2024-09-13T15:00:03.537248+0000","last_active":"2024-09-13T15:00:49.047466+0000","last_peered":"2024-09-13T15:00:49.047466+0000","last_clean":"2024-09-13T15:00:49.047466+0000","last_became_active":"2024-09-13T15:00:03.536943+0000","last_became_peered":"2024-09-13T15:00:03.536943+0000","last_unstale":"2024-09-13T15:00:49.047466+0000","last_undegraded":"2024-09-13T15:00:49.047466+0000","last_fullsized":"2024-09-13T15:00:49.047466+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:19.606950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1746,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":32,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.048713+0000","last_change":"2024-09-13T14:59:53.504853+0000","last_active":"2024-09-13T15:00:48.048713+0000","last_peered":"2024-09-13T15:00:48.048713+0000","last_clean":"2024-09-13T15:00:48.048713+0000","last_became_active":"2024-09-13T14:59:53.504556+0000","last_became_peered":"2024-09-13T14:59:53.504556+0000","last_unstale":"2024-09-13T15:00:48.048713+0000","last_undegraded":"2024-09-13T15:00:48.048713+0000","last_fullsized":"2024-09-13T15:00:48.048713+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:55.050435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1d","version":"90'149","reported_seq":530,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.220515+0000","last_change":"2024-09-13T14:59:56.507148+0000","last_active":"2024-09-13T15:00:49.220515+0000","last_peered":"2024-09-13T15:00:49.220515+0000","last_clean":"2024-09-13T15:00:49.220515+0000","last_became_active":"2024-09-13T14:59:56.506750+0000","last_became_peered":"2024-09-13T14:59:56.506750+0000","last_unstale":"2024-09-13T15:00:49.220515+0000","last_undegraded":"2024-09-13T15:00:49.220515+0000","last_fullsized":"2024-09-13T15:00:49.220515+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":471,"last_epoch_clean":499,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:38.760963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:38.760963+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:08:12.028324+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":938,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.050720+0000","last_change":"2024-09-13T14:46:01.769647+0000","last_active":"2024-09-13T15:00:49.050720+0000","last_peered":"2024-09-13T15:00:49.050720+0000","last_clean":"2024-09-13T15:00:49.050720+0000","last_became_active":"2024-09-13T14:46:01.769228+0000","last_became_peered":"2024-09-13T14:46:01.769228+0000","last_unstale":"2024-09-13T15:00:49.050720+0000","last_undegraded":"2024-09-13T15:00:49.050720+0000","last_fullsized":"2024-09-13T15:00:49.050720+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:58.696024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":973,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.049344+0000","last_change":"2024-09-13T14:46:01.766580+0000","last_active":"2024-09-13T15:00:49.049344+0000","last_peered":"2024-09-13T15:00:49.049344+0000","last_clean":"2024-09-13T15:00:49.049344+0000","last_became_active":"2024-09-13T14:46:01.766257+0000","last_became_peered":"2024-09-13T14:46:01.766257+0000","last_unstale":"2024-09-13T15:00:49.049344+0000","last_undegraded":"2024-09-13T15:00:49.049344+0000","last_fullsized":"2024-09-13T15:00:49.049344+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:01.126109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1c","version":"90'149","reported_seq":497,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.045051+0000","last_change":"2024-09-13T14:57:44.980830+0000","last_active":"2024-09-13T15:00:48.045051+0000","last_peered":"2024-09-13T15:00:48.045051+0000","last_clean":"2024-09-13T15:00:48.045051+0000","last_became_active":"2024-09-13T14:57:28.507948+0000","last_became_peered":"2024-09-13T14:57:28.507948+0000","last_unstale":"2024-09-13T15:00:48.045051+0000","last_undegraded":"2024-09-13T15:00:48.045051+0000","last_fullsized":"2024-09-13T15:00:48.045051+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":471,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:44.980740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:44.980740+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:52:20.504558+0000","scrub_duration":0.012428020999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":364,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.049165+0000","last_change":"2024-09-13T14:59:53.515721+0000","last_active":"2024-09-13T15:00:49.049165+0000","last_peered":"2024-09-13T15:00:49.049165+0000","last_clean":"2024-09-13T15:00:49.049165+0000","last_became_active":"2024-09-13T14:59:53.505349+0000","last_became_peered":"2024-09-13T14:59:53.505349+0000","last_unstale":"2024-09-13T15:00:49.049165+0000","last_undegraded":"2024-09-13T15:00:49.049165+0000","last_fullsized":"2024-09-13T15:00:49.049165+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:22:52.933174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":32,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.050410+0000","last_change":"2024-09-13T14:59:53.509385+0000","last_active":"2024-09-13T15:00:48.050410+0000","last_peered":"2024-09-13T15:00:48.050410+0000","last_clean":"2024-09-13T15:00:48.050410+0000","last_became_active":"2024-09-13T14:59:53.507008+0000","last_became_peered":"2024-09-13T14:59:53.507008+0000","last_unstale":"2024-09-13T15:00:48.050410+0000","last_undegraded":"2024-09-13T15:00:48.050410+0000","last_fullsized":"2024-09-13T15:00:48.050410+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:49:04.476461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":377,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.048270+0000","last_change":"2024-09-13T14:59:53.518804+0000","last_active":"2024-09-13T15:00:49.048270+0000","last_peered":"2024-09-13T15:00:49.048270+0000","last_clean":"2024-09-13T15:00:49.048270+0000","last_became_active":"2024-09-13T14:59:53.506622+0000","last_became_peered":"2024-09-13T14:59:53.506622+0000","last_unstale":"2024-09-13T15:00:49.048270+0000","last_undegraded":"2024-09-13T15:00:49.048270+0000","last_fullsized":"2024-09-13T15:00:49.048270+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:09:25.888980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":32,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.050435+0000","last_change":"2024-09-13T14:59:53.508269+0000","last_active":"2024-09-13T15:00:48.050435+0000","last_peered":"2024-09-13T15:00:48.050435+0000","last_clean":"2024-09-13T15:00:48.050435+0000","last_became_active":"2024-09-13T14:59:53.507928+0000","last_became_peered":"2024-09-13T14:59:53.507928+0000","last_unstale":"2024-09-13T15:00:48.050435+0000","last_undegraded":"2024-09-13T15:00:48.050435+0000","last_fullsized":"2024-09-13T15:00:48.050435+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:13.608699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"90'149","reported_seq":491,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.050558+0000","last_change":"2024-09-13T14:57:39.990337+0000","last_active":"2024-09-13T15:00:49.050558+0000","last_peered":"2024-09-13T15:00:49.050558+0000","last_clean":"2024-09-13T15:00:49.050558+0000","last_became_active":"2024-09-13T14:57:28.525160+0000","last_became_peered":"2024-09-13T14:57:28.525160+0000","last_unstale":"2024-09-13T15:00:49.050558+0000","last_undegraded":"2024-09-13T15:00:49.050558+0000","last_fullsized":"2024-09-13T15:00:49.050558+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":471,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:39.990271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:39.990271+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:55.404672+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":500,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.047193+0000","last_change":"2024-09-13T15:00:03.535707+0000","last_active":"2024-09-13T15:00:49.047193+0000","last_peered":"2024-09-13T15:00:49.047193+0000","last_clean":"2024-09-13T15:00:49.047193+0000","last_became_active":"2024-09-13T15:00:03.535394+0000","last_became_peered":"2024-09-13T15:00:03.535394+0000","last_unstale":"2024-09-13T15:00:49.047193+0000","last_undegraded":"2024-09-13T15:00:49.047193+0000","last_fullsized":"2024-09-13T15:00:49.047193+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:30:32.637273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":459,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.050764+0000","last_change":"2024-09-13T14:50:27.510133+0000","last_active":"2024-09-13T15:00:49.050764+0000","last_peered":"2024-09-13T15:00:49.050764+0000","last_clean":"2024-09-13T15:00:49.050764+0000","last_became_active":"2024-09-13T14:50:27.508774+0000","last_became_peered":"2024-09-13T14:50:27.508774+0000","last_unstale":"2024-09-13T15:00:49.050764+0000","last_undegraded":"2024-09-13T15:00:49.050764+0000","last_fullsized":"2024-09-13T15:00:49.050764+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:31:04.531746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.19","version":"90'149","reported_seq":579,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.049186+0000","last_change":"2024-09-13T14:57:38.050038+0000","last_active":"2024-09-13T15:00:49.049186+0000","last_peered":"2024-09-13T15:00:49.049186+0000","last_clean":"2024-09-13T15:00:49.049186+0000","last_became_active":"2024-09-13T14:57:28.530372+0000","last_became_peered":"2024-09-13T14:57:28.530372+0000","last_unstale":"2024-09-13T15:00:49.049186+0000","last_undegraded":"2024-09-13T15:00:49.049186+0000","last_fullsized":"2024-09-13T15:00:49.049186+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":471,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:38.049970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:38.049970+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:10.878382+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":32,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.050511+0000","last_change":"2024-09-13T14:59:53.504414+0000","last_active":"2024-09-13T15:00:48.050511+0000","last_peered":"2024-09-13T15:00:48.050511+0000","last_clean":"2024-09-13T15:00:48.050511+0000","last_became_active":"2024-09-13T14:59:53.503984+0000","last_became_peered":"2024-09-13T14:59:53.503984+0000","last_unstale":"2024-09-13T15:00:48.050511+0000","last_undegraded":"2024-09-13T15:00:48.050511+0000","last_fullsized":"2024-09-13T15:00:48.050511+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:22:35.693249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":961,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.050247+0000","last_change":"2024-09-13T14:59:53.510391+0000","last_active":"2024-09-13T15:00:49.050247+0000","last_peered":"2024-09-13T15:00:49.050247+0000","last_clean":"2024-09-13T15:00:49.050247+0000","last_became_active":"2024-09-13T14:59:53.510044+0000","last_became_peered":"2024-09-13T14:59:53.510044+0000","last_unstale":"2024-09-13T15:00:49.050247+0000","last_undegraded":"2024-09-13T15:00:49.050247+0000","last_fullsized":"2024-09-13T15:00:49.050247+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:25.871310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.18","version":"90'149","reported_seq":641,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.044754+0000","last_change":"2024-09-13T14:57:41.793060+0000","last_active":"2024-09-13T15:00:48.044754+0000","last_peered":"2024-09-13T15:00:48.044754+0000","last_clean":"2024-09-13T15:00:48.044754+0000","last_became_active":"2024-09-13T14:57:28.525726+0000","last_became_peered":"2024-09-13T14:57:28.525726+0000","last_unstale":"2024-09-13T15:00:48.044754+0000","last_undegraded":"2024-09-13T15:00:48.044754+0000","last_fullsized":"2024-09-13T15:00:48.044754+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":471,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:41.792934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:41.792934+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:18.870805+0000","scrub_duration":0.0079393269999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":391,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.220463+0000","last_change":"2024-09-13T14:59:53.503542+0000","last_active":"2024-09-13T15:00:49.220463+0000","last_peered":"2024-09-13T15:00:49.220463+0000","last_clean":"2024-09-13T15:00:49.220463+0000","last_became_active":"2024-09-13T14:59:53.503247+0000","last_became_peered":"2024-09-13T14:59:53.503247+0000","last_unstale":"2024-09-13T15:00:49.220463+0000","last_undegraded":"2024-09-13T15:00:49.220463+0000","last_fullsized":"2024-09-13T15:00:49.220463+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:23:30.238977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":501,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.047364+0000","last_change":"2024-09-13T15:00:03.538343+0000","last_active":"2024-09-13T15:00:49.047364+0000","last_peered":"2024-09-13T15:00:49.047364+0000","last_clean":"2024-09-13T15:00:49.047364+0000","last_became_active":"2024-09-13T15:00:03.537266+0000","last_became_peered":"2024-09-13T15:00:03.537266+0000","last_unstale":"2024-09-13T15:00:49.047364+0000","last_undegraded":"2024-09-13T15:00:49.047364+0000","last_fullsized":"2024-09-13T15:00:49.047364+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:51.421664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1b","version":"90'149","reported_seq":466,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.044262+0000","last_change":"2024-09-13T14:59:56.508684+0000","last_active":"2024-09-13T15:00:48.044262+0000","last_peered":"2024-09-13T15:00:48.044262+0000","last_clean":"2024-09-13T15:00:48.044262+0000","last_became_active":"2024-09-13T14:59:56.508302+0000","last_became_peered":"2024-09-13T14:59:56.508302+0000","last_unstale":"2024-09-13T15:00:48.044262+0000","last_undegraded":"2024-09-13T15:00:48.044262+0000","last_fullsized":"2024-09-13T15:00:48.044262+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":471,"last_epoch_clean":499,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:42.789102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:42.789102+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:17:57.495061+0000","scrub_duration":0.0083861109999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":959,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.048931+0000","last_change":"2024-09-13T14:47:47.868945+0000","last_active":"2024-09-13T15:00:49.048931+0000","last_peered":"2024-09-13T15:00:49.048931+0000","last_clean":"2024-09-13T15:00:49.048931+0000","last_became_active":"2024-09-13T14:47:47.868296+0000","last_became_peered":"2024-09-13T14:47:47.868296+0000","last_unstale":"2024-09-13T15:00:49.048931+0000","last_undegraded":"2024-09-13T15:00:49.048931+0000","last_fullsized":"2024-09-13T15:00:49.048931+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:22:24.778693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":182,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.221778+0000","last_change":"2024-09-13T15:00:49.221419+0000","last_active":"2024-09-13T15:00:49.221778+0000","last_peered":"2024-09-13T15:00:49.221778+0000","last_clean":"2024-09-13T15:00:49.221778+0000","last_became_active":"2024-09-13T15:00:49.221131+0000","last_became_peered":"2024-09-13T15:00:49.221131+0000","last_unstale":"2024-09-13T15:00:49.221778+0000","last_undegraded":"2024-09-13T15:00:49.221778+0000","last_fullsized":"2024-09-13T15:00:49.221778+0000","mapping_epoch":516,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:25:48.840688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1a","version":"90'149","reported_seq":421,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.050604+0000","last_change":"2024-09-13T14:59:58.822684+0000","last_active":"2024-09-13T15:00:48.050604+0000","last_peered":"2024-09-13T15:00:48.050604+0000","last_clean":"2024-09-13T15:00:48.050604+0000","last_became_active":"2024-09-13T14:59:56.506612+0000","last_became_peered":"2024-09-13T14:59:56.506612+0000","last_unstale":"2024-09-13T15:00:48.050604+0000","last_undegraded":"2024-09-13T15:00:48.050604+0000","last_fullsized":"2024-09-13T15:00:48.050604+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":471,"last_epoch_clean":499,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:59:58.822608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:59:58.822608+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:03:38.735268+0000","scrub_duration":0.0020743630000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":927,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.048706+0000","last_change":"2024-09-13T14:47:47.868624+0000","last_active":"2024-09-13T15:00:49.048706+0000","last_peered":"2024-09-13T15:00:49.048706+0000","last_clean":"2024-09-13T15:00:49.048706+0000","last_became_active":"2024-09-13T14:47:47.868031+0000","last_became_peered":"2024-09-13T14:47:47.868031+0000","last_unstale":"2024-09-13T15:00:49.048706+0000","last_undegraded":"2024-09-13T15:00:49.048706+0000","last_fullsized":"2024-09-13T15:00:49.048706+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:05:25.365023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":493,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.047504+0000","last_change":"2024-09-13T15:00:03.536415+0000","last_active":"2024-09-13T15:00:49.047504+0000","last_peered":"2024-09-13T15:00:49.047504+0000","last_clean":"2024-09-13T15:00:49.047504+0000","last_became_active":"2024-09-13T15:00:03.536117+0000","last_became_peered":"2024-09-13T15:00:03.536117+0000","last_unstale":"2024-09-13T15:00:49.047504+0000","last_undegraded":"2024-09-13T15:00:49.047504+0000","last_fullsized":"2024-09-13T15:00:49.047504+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:44:11.549146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.15","version":"90'149","reported_seq":543,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.045988+0000","last_change":"2024-09-13T14:57:46.923759+0000","last_active":"2024-09-13T15:00:48.045988+0000","last_peered":"2024-09-13T15:00:48.045988+0000","last_clean":"2024-09-13T15:00:48.045988+0000","last_became_active":"2024-09-13T14:57:28.519678+0000","last_became_peered":"2024-09-13T14:57:28.519678+0000","last_unstale":"2024-09-13T15:00:48.045988+0000","last_undegraded":"2024-09-13T15:00:48.045988+0000","last_fullsized":"2024-09-13T15:00:48.045988+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":471,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:46.923638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:46.923638+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:59.709420+0000","scrub_duration":0.002484039,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":891,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.049019+0000","last_change":"2024-09-13T14:38:53.899535+0000","last_active":"2024-09-13T15:00:49.049019+0000","last_peered":"2024-09-13T15:00:49.049019+0000","last_clean":"2024-09-13T15:00:49.049019+0000","last_became_active":"2024-09-13T14:38:53.899183+0000","last_became_peered":"2024-09-13T14:38:53.899183+0000","last_unstale":"2024-09-13T15:00:49.049019+0000","last_undegraded":"2024-09-13T15:00:49.049019+0000","last_fullsized":"2024-09-13T15:00:49.049019+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:33.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":350,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.044530+0000","last_change":"2024-09-13T14:47:47.861842+0000","last_active":"2024-09-13T15:00:48.044530+0000","last_peered":"2024-09-13T15:00:48.044530+0000","last_clean":"2024-09-13T15:00:48.044530+0000","last_became_active":"2024-09-13T14:47:47.861492+0000","last_became_peered":"2024-09-13T14:47:47.861492+0000","last_unstale":"2024-09-13T15:00:48.044530+0000","last_undegraded":"2024-09-13T15:00:48.044530+0000","last_fullsized":"2024-09-13T15:00:48.044530+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:17.279533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"90'149","reported_seq":833,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.050450+0000","last_change":"2024-09-13T14:59:56.508469+0000","last_active":"2024-09-13T15:00:49.050450+0000","last_peered":"2024-09-13T15:00:49.050450+0000","last_clean":"2024-09-13T15:00:49.050450+0000","last_became_active":"2024-09-13T14:59:56.508088+0000","last_became_peered":"2024-09-13T14:59:56.508088+0000","last_unstale":"2024-09-13T15:00:49.050450+0000","last_undegraded":"2024-09-13T15:00:49.050450+0000","last_fullsized":"2024-09-13T15:00:49.050450+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":499,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:31.013128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:31.013128+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:17:14.466888+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":319,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.045507+0000","last_change":"2024-09-13T14:59:53.526879+0000","last_active":"2024-09-13T15:00:48.045507+0000","last_peered":"2024-09-13T15:00:48.045507+0000","last_clean":"2024-09-13T15:00:48.045507+0000","last_became_active":"2024-09-13T14:59:53.514049+0000","last_became_peered":"2024-09-13T14:59:53.514049+0000","last_unstale":"2024-09-13T15:00:48.045507+0000","last_undegraded":"2024-09-13T15:00:48.045507+0000","last_fullsized":"2024-09-13T15:00:48.045507+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:31:53.727081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":270,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":377,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.220623+0000","last_change":"2024-09-13T14:47:47.867935+0000","last_active":"2024-09-13T15:00:49.220623+0000","last_peered":"2024-09-13T15:00:49.220623+0000","last_clean":"2024-09-13T15:00:49.220623+0000","last_became_active":"2024-09-13T14:47:47.867616+0000","last_became_peered":"2024-09-13T14:47:47.867616+0000","last_unstale":"2024-09-13T15:00:49.220623+0000","last_undegraded":"2024-09-13T15:00:49.220623+0000","last_fullsized":"2024-09-13T15:00:49.220623+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:52:38.619916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"90'149","reported_seq":797,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.044879+0000","last_change":"2024-09-13T14:57:47.798716+0000","last_active":"2024-09-13T15:00:48.044879+0000","last_peered":"2024-09-13T15:00:48.044879+0000","last_clean":"2024-09-13T15:00:48.044879+0000","last_became_active":"2024-09-13T14:57:28.526594+0000","last_became_peered":"2024-09-13T14:57:28.526594+0000","last_unstale":"2024-09-13T15:00:48.044879+0000","last_undegraded":"2024-09-13T15:00:48.044879+0000","last_fullsized":"2024-09-13T15:00:48.044879+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":471,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:47.798627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:47.798627+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:30:00.799937+0000","scrub_duration":0.0039598259999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'23","reported_seq":42,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.049246+0000","last_change":"2024-09-13T14:59:53.521057+0000","last_active":"2024-09-13T15:00:48.049246+0000","last_peered":"2024-09-13T15:00:48.049246+0000","last_clean":"2024-09-13T15:00:48.049246+0000","last_became_active":"2024-09-13T14:59:53.511100+0000","last_became_peered":"2024-09-13T14:59:53.511100+0000","last_unstale":"2024-09-13T15:00:48.049246+0000","last_undegraded":"2024-09-13T15:00:48.049246+0000","last_fullsized":"2024-09-13T15:00:48.049246+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:38:46.768037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4756,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":32,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.050519+0000","last_change":"2024-09-13T14:59:53.507755+0000","last_active":"2024-09-13T15:00:48.050519+0000","last_peered":"2024-09-13T15:00:48.050519+0000","last_clean":"2024-09-13T15:00:48.050519+0000","last_became_active":"2024-09-13T14:59:53.507484+0000","last_became_peered":"2024-09-13T14:59:53.507484+0000","last_unstale":"2024-09-13T15:00:48.050519+0000","last_undegraded":"2024-09-13T15:00:48.050519+0000","last_fullsized":"2024-09-13T15:00:48.050519+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:25.909660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.16","version":"90'149","reported_seq":810,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.221077+0000","last_change":"2024-09-13T14:57:40.798149+0000","last_active":"2024-09-13T15:00:49.221077+0000","last_peered":"2024-09-13T15:00:49.221077+0000","last_clean":"2024-09-13T15:00:49.221077+0000","last_became_active":"2024-09-13T14:57:28.528351+0000","last_became_peered":"2024-09-13T14:57:28.528351+0000","last_unstale":"2024-09-13T15:00:49.221077+0000","last_undegraded":"2024-09-13T15:00:49.221077+0000","last_fullsized":"2024-09-13T15:00:49.221077+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":471,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:40.797796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:40.797796+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:09.769513+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":32,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.050251+0000","last_change":"2024-09-13T14:59:53.507344+0000","last_active":"2024-09-13T15:00:48.050251+0000","last_peered":"2024-09-13T15:00:48.050251+0000","last_clean":"2024-09-13T15:00:48.050251+0000","last_became_active":"2024-09-13T14:59:53.507068+0000","last_became_peered":"2024-09-13T14:59:53.507068+0000","last_unstale":"2024-09-13T15:00:48.050251+0000","last_undegraded":"2024-09-13T15:00:48.050251+0000","last_fullsized":"2024-09-13T15:00:48.050251+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:47.526588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"85'95","reported_seq":655,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.054742+0000","last_change":"2024-09-13T15:00:49.054003+0000","last_active":"2024-09-13T15:00:49.054742+0000","last_peered":"2024-09-13T15:00:49.054742+0000","last_clean":"2024-09-13T15:00:49.054742+0000","last_became_active":"2024-09-13T15:00:49.053404+0000","last_became_peered":"2024-09-13T15:00:49.053404+0000","last_unstale":"2024-09-13T15:00:49.054742+0000","last_undegraded":"2024-09-13T15:00:49.054742+0000","last_fullsized":"2024-09-13T15:00:49.054742+0000","mapping_epoch":516,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:20.909606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082582100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":724,"num_read_kb":652,"num_write":68,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":531,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.047439+0000","last_change":"2024-09-13T15:00:03.535864+0000","last_active":"2024-09-13T15:00:49.047439+0000","last_peered":"2024-09-13T15:00:49.047439+0000","last_clean":"2024-09-13T15:00:49.047439+0000","last_became_active":"2024-09-13T15:00:03.535481+0000","last_became_peered":"2024-09-13T15:00:03.535481+0000","last_unstale":"2024-09-13T15:00:49.047439+0000","last_undegraded":"2024-09-13T15:00:49.047439+0000","last_fullsized":"2024-09-13T15:00:49.047439+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:38:15.374371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"90'149","reported_seq":811,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.049029+0000","last_change":"2024-09-13T14:47:29.999643+0000","last_active":"2024-09-13T15:00:49.049029+0000","last_peered":"2024-09-13T15:00:49.049029+0000","last_clean":"2024-09-13T15:00:49.049029+0000","last_became_active":"2024-09-13T14:47:29.410951+0000","last_became_peered":"2024-09-13T14:47:29.410951+0000","last_unstale":"2024-09-13T15:00:49.049029+0000","last_undegraded":"2024-09-13T15:00:49.049029+0000","last_fullsized":"2024-09-13T15:00:49.049029+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:29.999519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:29.999519+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:45:08.824756+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":951,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.048806+0000","last_change":"2024-09-13T14:38:53.900107+0000","last_active":"2024-09-13T15:00:49.048806+0000","last_peered":"2024-09-13T15:00:49.048806+0000","last_clean":"2024-09-13T15:00:49.048806+0000","last_became_active":"2024-09-13T14:38:53.899556+0000","last_became_peered":"2024-09-13T14:38:53.899556+0000","last_unstale":"2024-09-13T15:00:49.048806+0000","last_undegraded":"2024-09-13T15:00:49.048806+0000","last_fullsized":"2024-09-13T15:00:49.048806+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:15:47.918163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'140","reported_seq":916,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.221217+0000","last_change":"2024-09-13T14:59:56.508453+0000","last_active":"2024-09-13T15:00:49.221217+0000","last_peered":"2024-09-13T15:00:49.221217+0000","last_clean":"2024-09-13T15:00:49.221217+0000","last_became_active":"2024-09-13T14:59:56.507789+0000","last_became_peered":"2024-09-13T14:59:56.507789+0000","last_unstale":"2024-09-13T15:00:49.221217+0000","last_undegraded":"2024-09-13T15:00:49.221217+0000","last_fullsized":"2024-09-13T15:00:49.221217+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":499,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:31:23.438892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029715600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":676,"num_read_kb":2005,"num_write":107,"num_write_kb":1308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":968,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.050971+0000","last_change":"2024-09-13T14:38:53.938227+0000","last_active":"2024-09-13T15:00:49.050971+0000","last_peered":"2024-09-13T15:00:49.050971+0000","last_clean":"2024-09-13T15:00:49.050971+0000","last_became_active":"2024-09-13T14:38:53.902867+0000","last_became_peered":"2024-09-13T14:38:53.902867+0000","last_unstale":"2024-09-13T15:00:49.050971+0000","last_undegraded":"2024-09-13T15:00:49.050971+0000","last_fullsized":"2024-09-13T15:00:49.050971+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:16.831756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"90'149","reported_seq":571,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.050034+0000","last_change":"2024-09-13T14:59:56.507724+0000","last_active":"2024-09-13T15:00:48.050034+0000","last_peered":"2024-09-13T15:00:48.050034+0000","last_clean":"2024-09-13T15:00:48.050034+0000","last_became_active":"2024-09-13T14:59:56.507194+0000","last_became_peered":"2024-09-13T14:59:56.507194+0000","last_unstale":"2024-09-13T15:00:48.050034+0000","last_undegraded":"2024-09-13T15:00:48.050034+0000","last_fullsized":"2024-09-13T15:00:48.050034+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":499,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:37.063494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:37.063494+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:36:50.707939+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":494,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.047366+0000","last_change":"2024-09-13T15:00:03.535624+0000","last_active":"2024-09-13T15:00:49.047366+0000","last_peered":"2024-09-13T15:00:49.047366+0000","last_clean":"2024-09-13T15:00:49.047366+0000","last_became_active":"2024-09-13T15:00:03.535351+0000","last_became_peered":"2024-09-13T15:00:03.535351+0000","last_unstale":"2024-09-13T15:00:49.047366+0000","last_undegraded":"2024-09-13T15:00:49.047366+0000","last_fullsized":"2024-09-13T15:00:49.047366+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:33:47.326298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"85'115","reported_seq":1121,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.046082+0000","last_change":"2024-09-13T14:59:56.510064+0000","last_active":"2024-09-13T15:00:48.046082+0000","last_peered":"2024-09-13T15:00:48.046082+0000","last_clean":"2024-09-13T15:00:48.046082+0000","last_became_active":"2024-09-13T14:59:56.508959+0000","last_became_peered":"2024-09-13T14:59:56.508959+0000","last_unstale":"2024-09-13T15:00:48.046082+0000","last_undegraded":"2024-09-13T15:00:48.046082+0000","last_fullsized":"2024-09-13T15:00:48.046082+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":499,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:26:49.194629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":911,"num_read_kb":2116,"num_write":91,"num_write_kb":1226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":378,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.220967+0000","last_change":"2024-09-13T14:47:47.876562+0000","last_active":"2024-09-13T15:00:49.220967+0000","last_peered":"2024-09-13T15:00:49.220967+0000","last_clean":"2024-09-13T15:00:49.220967+0000","last_became_active":"2024-09-13T14:47:47.876297+0000","last_became_peered":"2024-09-13T14:47:47.876297+0000","last_unstale":"2024-09-13T15:00:49.220967+0000","last_undegraded":"2024-09-13T15:00:49.220967+0000","last_fullsized":"2024-09-13T15:00:49.220967+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:03:27.761394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"90'149","reported_seq":386,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:56.559213+0000","last_change":"2024-09-13T15:00:56.559366+0000","last_active":"2024-09-13T15:00:56.559213+0000","last_peered":"2024-09-13T15:00:56.559213+0000","last_clean":"2024-09-13T15:00:56.559213+0000","last_became_active":"2024-09-13T15:00:49.050743+0000","last_became_peered":"2024-09-13T15:00:49.050743+0000","last_unstale":"2024-09-13T15:00:56.559213+0000","last_undegraded":"2024-09-13T15:00:56.559213+0000","last_fullsized":"2024-09-13T15:00:56.559213+0000","mapping_epoch":516,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T15:00:56.559183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T15:00:56.559183+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:25:50.528399+0000","scrub_duration":0.0016783429999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":392,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.220816+0000","last_change":"2024-09-13T14:59:53.503001+0000","last_active":"2024-09-13T15:00:49.220816+0000","last_peered":"2024-09-13T15:00:49.220816+0000","last_clean":"2024-09-13T15:00:49.220816+0000","last_became_active":"2024-09-13T14:59:53.502687+0000","last_became_peered":"2024-09-13T14:59:53.502687+0000","last_unstale":"2024-09-13T15:00:49.220816+0000","last_undegraded":"2024-09-13T15:00:49.220816+0000","last_fullsized":"2024-09-13T15:00:49.220816+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:02:23.981179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"86'185","reported_seq":1666,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.220848+0000","last_change":"2024-09-13T14:47:50.883365+0000","last_active":"2024-09-13T15:00:49.220848+0000","last_peered":"2024-09-13T15:00:49.220848+0000","last_clean":"2024-09-13T15:00:49.220848+0000","last_became_active":"2024-09-13T14:47:50.882609+0000","last_became_peered":"2024-09-13T14:47:50.882609+0000","last_unstale":"2024-09-13T15:00:49.220848+0000","last_undegraded":"2024-09-13T15:00:49.220848+0000","last_fullsized":"2024-09-13T15:00:49.220848+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:12:17.961575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084963200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1230,"num_read_kb":2196,"num_write":189,"num_write_kb":1176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":345,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.049061+0000","last_change":"2024-09-13T14:46:01.763906+0000","last_active":"2024-09-13T15:00:49.049061+0000","last_peered":"2024-09-13T15:00:49.049061+0000","last_clean":"2024-09-13T15:00:49.049061+0000","last_became_active":"2024-09-13T14:46:01.763542+0000","last_became_peered":"2024-09-13T14:46:01.763542+0000","last_unstale":"2024-09-13T15:00:49.049061+0000","last_undegraded":"2024-09-13T15:00:49.049061+0000","last_fullsized":"2024-09-13T15:00:49.049061+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:09:36.652767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"90'149","reported_seq":386,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:01:01.504353+0000","last_change":"2024-09-13T15:01:01.504440+0000","last_active":"2024-09-13T15:01:01.504353+0000","last_peered":"2024-09-13T15:01:01.504353+0000","last_clean":"2024-09-13T15:01:01.504353+0000","last_became_active":"2024-09-13T15:00:49.052038+0000","last_became_peered":"2024-09-13T15:00:49.052038+0000","last_unstale":"2024-09-13T15:01:01.504353+0000","last_undegraded":"2024-09-13T15:01:01.504353+0000","last_fullsized":"2024-09-13T15:01:01.504353+0000","mapping_epoch":516,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T15:01:01.504336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T15:01:01.504336+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:41:09.571281+0000","scrub_duration":0.0020290070000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":378,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.220748+0000","last_change":"2024-09-13T14:47:47.877272+0000","last_active":"2024-09-13T15:00:49.220748+0000","last_peered":"2024-09-13T15:00:49.220748+0000","last_clean":"2024-09-13T15:00:49.220748+0000","last_became_active":"2024-09-13T14:47:47.877003+0000","last_became_peered":"2024-09-13T14:47:47.877003+0000","last_unstale":"2024-09-13T15:00:49.220748+0000","last_undegraded":"2024-09-13T15:00:49.220748+0000","last_fullsized":"2024-09-13T15:00:49.220748+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:47:23.172452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":32,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.050662+0000","last_change":"2024-09-13T14:59:53.507662+0000","last_active":"2024-09-13T15:00:48.050662+0000","last_peered":"2024-09-13T15:00:48.050662+0000","last_clean":"2024-09-13T15:00:48.050662+0000","last_became_active":"2024-09-13T14:59:53.507388+0000","last_became_peered":"2024-09-13T14:59:53.507388+0000","last_unstale":"2024-09-13T15:00:48.050662+0000","last_undegraded":"2024-09-13T15:00:48.050662+0000","last_fullsized":"2024-09-13T15:00:48.050662+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:20:47.678025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"86'117","reported_seq":1276,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.048703+0000","last_change":"2024-09-13T14:47:50.892842+0000","last_active":"2024-09-13T15:00:49.048703+0000","last_peered":"2024-09-13T15:00:49.048703+0000","last_clean":"2024-09-13T15:00:49.048703+0000","last_became_active":"2024-09-13T14:47:50.892237+0000","last_became_peered":"2024-09-13T14:47:50.892237+0000","last_unstale":"2024-09-13T15:00:49.048703+0000","last_undegraded":"2024-09-13T15:00:49.048703+0000","last_fullsized":"2024-09-13T15:00:49.048703+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:20:05.796762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054894099999999995,"stat_sum":{"num_bytes":4194323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1163,"num_read_kb":1058,"num_write":85,"num_write_kb":4135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":8388646,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"90'149","reported_seq":621,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.047926+0000","last_change":"2024-09-13T14:47:32.275753+0000","last_active":"2024-09-13T15:00:49.047926+0000","last_peered":"2024-09-13T15:00:49.047926+0000","last_clean":"2024-09-13T15:00:49.047926+0000","last_became_active":"2024-09-13T14:47:29.401066+0000","last_became_peered":"2024-09-13T14:47:29.401066+0000","last_unstale":"2024-09-13T15:00:49.047926+0000","last_undegraded":"2024-09-13T15:00:49.047926+0000","last_fullsized":"2024-09-13T15:00:49.047926+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:32.275562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:32.275562+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:43:41.056775+0000","scrub_duration":0.0054674260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":378,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.220992+0000","last_change":"2024-09-13T14:47:47.870218+0000","last_active":"2024-09-13T15:00:49.220992+0000","last_peered":"2024-09-13T15:00:49.220992+0000","last_clean":"2024-09-13T15:00:49.220992+0000","last_became_active":"2024-09-13T14:47:47.869907+0000","last_became_peered":"2024-09-13T14:47:47.869907+0000","last_unstale":"2024-09-13T15:00:49.220992+0000","last_undegraded":"2024-09-13T15:00:49.220992+0000","last_fullsized":"2024-09-13T15:00:49.220992+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:41:49.141441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":378,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.220479+0000","last_change":"2024-09-13T14:47:47.866684+0000","last_active":"2024-09-13T15:00:49.220479+0000","last_peered":"2024-09-13T15:00:49.220479+0000","last_clean":"2024-09-13T15:00:49.220479+0000","last_became_active":"2024-09-13T14:47:47.866400+0000","last_became_peered":"2024-09-13T14:47:47.866400+0000","last_unstale":"2024-09-13T15:00:49.220479+0000","last_undegraded":"2024-09-13T15:00:49.220479+0000","last_fullsized":"2024-09-13T15:00:49.220479+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:08:34.289658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"85'135","reported_seq":861,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.050734+0000","last_change":"2024-09-13T14:46:04.782955+0000","last_active":"2024-09-13T15:00:49.050734+0000","last_peered":"2024-09-13T15:00:49.050734+0000","last_clean":"2024-09-13T15:00:49.050734+0000","last_became_active":"2024-09-13T14:46:04.782250+0000","last_became_peered":"2024-09-13T14:46:04.782250+0000","last_unstale":"2024-09-13T15:00:49.050734+0000","last_undegraded":"2024-09-13T15:00:49.050734+0000","last_fullsized":"2024-09-13T15:00:49.050734+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:24.596922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061713399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":668,"num_read_kb":1803,"num_write":133,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"90'149","reported_seq":753,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.044902+0000","last_change":"2024-09-13T14:59:56.509314+0000","last_active":"2024-09-13T15:00:49.044902+0000","last_peered":"2024-09-13T15:00:49.044902+0000","last_clean":"2024-09-13T15:00:49.044902+0000","last_became_active":"2024-09-13T14:59:56.508996+0000","last_became_peered":"2024-09-13T14:59:56.508996+0000","last_unstale":"2024-09-13T15:00:49.044902+0000","last_undegraded":"2024-09-13T15:00:49.044902+0000","last_fullsized":"2024-09-13T15:00:49.044902+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":499,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:29.629342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:29.629342+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:10:47.014236+0000","scrub_duration":0.0099290990000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":515,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.047552+0000","last_change":"2024-09-13T15:00:03.538661+0000","last_active":"2024-09-13T15:00:49.047552+0000","last_peered":"2024-09-13T15:00:49.047552+0000","last_clean":"2024-09-13T15:00:49.047552+0000","last_became_active":"2024-09-13T15:00:03.538114+0000","last_became_peered":"2024-09-13T15:00:03.538114+0000","last_unstale":"2024-09-13T15:00:49.047552+0000","last_undegraded":"2024-09-13T15:00:49.047552+0000","last_fullsized":"2024-09-13T15:00:49.047552+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:41:09.096859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":501,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.047070+0000","last_change":"2024-09-13T15:00:03.537417+0000","last_active":"2024-09-13T15:00:49.047070+0000","last_peered":"2024-09-13T15:00:49.047070+0000","last_clean":"2024-09-13T15:00:49.047070+0000","last_became_active":"2024-09-13T15:00:03.536451+0000","last_became_peered":"2024-09-13T15:00:03.536451+0000","last_unstale":"2024-09-13T15:00:49.047070+0000","last_undegraded":"2024-09-13T15:00:49.047070+0000","last_fullsized":"2024-09-13T15:00:49.047070+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:39:18.772336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'56","reported_seq":772,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.044260+0000","last_change":"2024-09-13T14:47:47.892973+0000","last_active":"2024-09-13T15:00:48.044260+0000","last_peered":"2024-09-13T15:00:48.044260+0000","last_clean":"2024-09-13T15:00:48.044260+0000","last_became_active":"2024-09-13T14:47:47.866986+0000","last_became_peered":"2024-09-13T14:47:47.866986+0000","last_unstale":"2024-09-13T15:00:48.044260+0000","last_undegraded":"2024-09-13T15:00:48.044260+0000","last_fullsized":"2024-09-13T15:00:48.044260+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:07.353290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044476599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":251,"num_read_kb":2579,"num_write":44,"num_write_kb":2222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"90'149","reported_seq":830,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.220754+0000","last_change":"2024-09-13T14:59:56.508952+0000","last_active":"2024-09-13T15:00:49.220754+0000","last_peered":"2024-09-13T15:00:49.220754+0000","last_clean":"2024-09-13T15:00:49.220754+0000","last_became_active":"2024-09-13T14:59:56.508586+0000","last_became_peered":"2024-09-13T14:59:56.508586+0000","last_unstale":"2024-09-13T15:00:49.220754+0000","last_undegraded":"2024-09-13T15:00:49.220754+0000","last_fullsized":"2024-09-13T15:00:49.220754+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":499,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:34.603316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:34.603316+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:00:46.717723+0000","scrub_duration":0.0057131120000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":149,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.052386+0000","last_change":"2024-09-13T15:00:49.051643+0000","last_active":"2024-09-13T15:00:49.052386+0000","last_peered":"2024-09-13T15:00:49.052386+0000","last_clean":"2024-09-13T15:00:49.052386+0000","last_became_active":"2024-09-13T15:00:49.051324+0000","last_became_peered":"2024-09-13T15:00:49.051324+0000","last_unstale":"2024-09-13T15:00:49.052386+0000","last_undegraded":"2024-09-13T15:00:49.052386+0000","last_fullsized":"2024-09-13T15:00:49.052386+0000","mapping_epoch":516,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:24:20.674659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":877,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.049759+0000","last_change":"2024-09-13T14:37:49.160719+0000","last_active":"2024-09-13T15:00:49.049759+0000","last_peered":"2024-09-13T15:00:49.049759+0000","last_clean":"2024-09-13T15:00:49.049759+0000","last_became_active":"2024-09-13T14:37:49.160401+0000","last_became_peered":"2024-09-13T14:37:49.160401+0000","last_unstale":"2024-09-13T15:00:49.049759+0000","last_undegraded":"2024-09-13T15:00:49.049759+0000","last_fullsized":"2024-09-13T15:00:49.049759+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:21.737945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'134","reported_seq":831,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.048638+0000","last_change":"2024-09-13T14:46:04.780718+0000","last_active":"2024-09-13T15:00:49.048638+0000","last_peered":"2024-09-13T15:00:49.048638+0000","last_clean":"2024-09-13T15:00:49.048638+0000","last_became_active":"2024-09-13T14:46:04.780120+0000","last_became_peered":"2024-09-13T14:46:04.780120+0000","last_unstale":"2024-09-13T15:00:49.048638+0000","last_undegraded":"2024-09-13T15:00:49.048638+0000","last_fullsized":"2024-09-13T15:00:49.048638+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:10:11.096209+0000","last_clean_scrub_stamp":"2024-09-13T14:10:11.096209+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:57:14.475888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000330141,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":602,"num_write":129,"num_write_kb":226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":702,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.044355+0000","last_change":"2024-09-13T14:47:31.684319+0000","last_active":"2024-09-13T15:00:48.044355+0000","last_peered":"2024-09-13T15:00:48.044355+0000","last_clean":"2024-09-13T15:00:48.044355+0000","last_became_active":"2024-09-13T14:47:29.415658+0000","last_became_peered":"2024-09-13T14:47:29.415658+0000","last_unstale":"2024-09-13T15:00:48.044355+0000","last_undegraded":"2024-09-13T15:00:48.044355+0000","last_fullsized":"2024-09-13T15:00:48.044355+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:31.681433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:31.681433+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:29:34.663513+0000","scrub_duration":0.0059601139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":392,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.220883+0000","last_change":"2024-09-13T14:59:53.502354+0000","last_active":"2024-09-13T15:00:49.220883+0000","last_peered":"2024-09-13T15:00:49.220883+0000","last_clean":"2024-09-13T15:00:49.220883+0000","last_became_active":"2024-09-13T14:59:53.502017+0000","last_became_peered":"2024-09-13T14:59:53.502017+0000","last_unstale":"2024-09-13T15:00:49.220883+0000","last_undegraded":"2024-09-13T15:00:49.220883+0000","last_fullsized":"2024-09-13T15:00:49.220883+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:36:58.220200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":949,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.048895+0000","last_change":"2024-09-13T14:47:47.861249+0000","last_active":"2024-09-13T15:00:49.048895+0000","last_peered":"2024-09-13T15:00:49.048895+0000","last_clean":"2024-09-13T15:00:49.048895+0000","last_became_active":"2024-09-13T14:47:47.860873+0000","last_became_peered":"2024-09-13T14:47:47.860873+0000","last_unstale":"2024-09-13T15:00:49.048895+0000","last_undegraded":"2024-09-13T15:00:49.048895+0000","last_fullsized":"2024-09-13T15:00:49.048895+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:37:50.709437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"90'149","reported_seq":620,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.044011+0000","last_change":"2024-09-13T14:57:38.849773+0000","last_active":"2024-09-13T15:00:48.044011+0000","last_peered":"2024-09-13T15:00:48.044011+0000","last_clean":"2024-09-13T15:00:48.044011+0000","last_became_active":"2024-09-13T14:57:17.378580+0000","last_became_peered":"2024-09-13T14:57:17.378580+0000","last_unstale":"2024-09-13T15:00:48.044011+0000","last_undegraded":"2024-09-13T15:00:48.044011+0000","last_fullsized":"2024-09-13T15:00:48.044011+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:38.849610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:38.849610+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:22:24.723639+0000","scrub_duration":0.0034255890000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.049686+0000","last_change":"2024-09-13T14:59:53.505482+0000","last_active":"2024-09-13T15:00:48.049686+0000","last_peered":"2024-09-13T15:00:48.049686+0000","last_clean":"2024-09-13T15:00:48.049686+0000","last_became_active":"2024-09-13T14:59:53.505200+0000","last_became_peered":"2024-09-13T14:59:53.505200+0000","last_unstale":"2024-09-13T15:00:48.049686+0000","last_undegraded":"2024-09-13T15:00:48.049686+0000","last_fullsized":"2024-09-13T15:00:48.049686+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:50.788322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":32,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.050136+0000","last_change":"2024-09-13T14:59:53.511732+0000","last_active":"2024-09-13T15:00:48.050136+0000","last_peered":"2024-09-13T15:00:48.050136+0000","last_clean":"2024-09-13T15:00:48.050136+0000","last_became_active":"2024-09-13T14:59:53.511351+0000","last_became_peered":"2024-09-13T14:59:53.511351+0000","last_unstale":"2024-09-13T15:00:48.050136+0000","last_undegraded":"2024-09-13T15:00:48.050136+0000","last_fullsized":"2024-09-13T15:00:48.050136+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:48:55.490615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"90'149","reported_seq":845,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.220636+0000","last_change":"2024-09-13T14:57:39.766173+0000","last_active":"2024-09-13T15:00:49.220636+0000","last_peered":"2024-09-13T15:00:49.220636+0000","last_clean":"2024-09-13T15:00:49.220636+0000","last_became_active":"2024-09-13T14:57:17.389370+0000","last_became_peered":"2024-09-13T14:57:17.389370+0000","last_unstale":"2024-09-13T15:00:49.220636+0000","last_undegraded":"2024-09-13T15:00:49.220636+0000","last_fullsized":"2024-09-13T15:00:49.220636+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:39.765978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:39.765978+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:39.980756+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":32,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.050236+0000","last_change":"2024-09-13T14:59:53.507082+0000","last_active":"2024-09-13T15:00:48.050236+0000","last_peered":"2024-09-13T15:00:48.050236+0000","last_clean":"2024-09-13T15:00:48.050236+0000","last_became_active":"2024-09-13T14:59:53.506615+0000","last_became_peered":"2024-09-13T14:59:53.506615+0000","last_unstale":"2024-09-13T15:00:48.050236+0000","last_undegraded":"2024-09-13T15:00:48.050236+0000","last_fullsized":"2024-09-13T15:00:48.050236+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:50:49.491733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":270,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.222488+0000","last_change":"2024-09-13T15:00:49.221849+0000","last_active":"2024-09-13T15:00:49.222488+0000","last_peered":"2024-09-13T15:00:49.222488+0000","last_clean":"2024-09-13T15:00:49.222488+0000","last_became_active":"2024-09-13T15:00:49.221548+0000","last_became_peered":"2024-09-13T15:00:49.221548+0000","last_unstale":"2024-09-13T15:00:49.222488+0000","last_undegraded":"2024-09-13T15:00:49.222488+0000","last_fullsized":"2024-09-13T15:00:49.222488+0000","mapping_epoch":516,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:56:10.688533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"90'149","reported_seq":620,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.044660+0000","last_change":"2024-09-13T14:57:39.812785+0000","last_active":"2024-09-13T15:00:48.044660+0000","last_peered":"2024-09-13T15:00:48.044660+0000","last_clean":"2024-09-13T15:00:48.044660+0000","last_became_active":"2024-09-13T14:57:17.383307+0000","last_became_peered":"2024-09-13T14:57:17.383307+0000","last_unstale":"2024-09-13T15:00:48.044660+0000","last_undegraded":"2024-09-13T15:00:48.044660+0000","last_fullsized":"2024-09-13T15:00:48.044660+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:39.812688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:39.812688+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:18:59.726465+0000","scrub_duration":0.010496065000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":32,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.049158+0000","last_change":"2024-09-13T14:59:53.507167+0000","last_active":"2024-09-13T15:00:48.049158+0000","last_peered":"2024-09-13T15:00:48.049158+0000","last_clean":"2024-09-13T15:00:48.049158+0000","last_became_active":"2024-09-13T14:59:53.506358+0000","last_became_peered":"2024-09-13T14:59:53.506358+0000","last_unstale":"2024-09-13T15:00:48.049158+0000","last_undegraded":"2024-09-13T15:00:48.049158+0000","last_fullsized":"2024-09-13T15:00:48.049158+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:54:19.042442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":946,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.049165+0000","last_change":"2024-09-13T14:59:53.520257+0000","last_active":"2024-09-13T15:00:49.049165+0000","last_peered":"2024-09-13T15:00:49.049165+0000","last_clean":"2024-09-13T15:00:49.049165+0000","last_became_active":"2024-09-13T14:59:53.506185+0000","last_became_peered":"2024-09-13T14:59:53.506185+0000","last_unstale":"2024-09-13T15:00:49.049165+0000","last_undegraded":"2024-09-13T15:00:49.049165+0000","last_fullsized":"2024-09-13T15:00:49.049165+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:26.417709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"90'149","reported_seq":717,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.044013+0000","last_change":"2024-09-13T14:57:41.926732+0000","last_active":"2024-09-13T15:00:48.044013+0000","last_peered":"2024-09-13T15:00:48.044013+0000","last_clean":"2024-09-13T15:00:48.044013+0000","last_became_active":"2024-09-13T14:57:17.380843+0000","last_became_peered":"2024-09-13T14:57:17.380843+0000","last_unstale":"2024-09-13T15:00:48.044013+0000","last_undegraded":"2024-09-13T15:00:48.044013+0000","last_fullsized":"2024-09-13T15:00:48.044013+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:41.926626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:41.926626+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:08:57.137123+0000","scrub_duration":0.0019476179999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":537,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.047255+0000","last_change":"2024-09-13T15:00:03.537200+0000","last_active":"2024-09-13T15:00:49.047255+0000","last_peered":"2024-09-13T15:00:49.047255+0000","last_clean":"2024-09-13T15:00:49.047255+0000","last_became_active":"2024-09-13T15:00:03.536871+0000","last_became_peered":"2024-09-13T15:00:03.536871+0000","last_unstale":"2024-09-13T15:00:49.047255+0000","last_undegraded":"2024-09-13T15:00:49.047255+0000","last_fullsized":"2024-09-13T15:00:49.047255+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:02:58.780844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":335,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.048087+0000","last_change":"2024-09-13T14:37:49.168003+0000","last_active":"2024-09-13T15:00:49.048087+0000","last_peered":"2024-09-13T15:00:49.048087+0000","last_clean":"2024-09-13T15:00:49.048087+0000","last_became_active":"2024-09-13T14:37:49.167638+0000","last_became_peered":"2024-09-13T14:37:49.167638+0000","last_unstale":"2024-09-13T15:00:49.048087+0000","last_undegraded":"2024-09-13T15:00:49.048087+0000","last_fullsized":"2024-09-13T15:00:49.048087+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:29.051981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"90'149","reported_seq":485,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.049785+0000","last_change":"2024-09-13T14:59:59.848229+0000","last_active":"2024-09-13T15:00:48.049785+0000","last_peered":"2024-09-13T15:00:48.049785+0000","last_clean":"2024-09-13T15:00:48.049785+0000","last_became_active":"2024-09-13T14:59:56.506696+0000","last_became_peered":"2024-09-13T14:59:56.506696+0000","last_unstale":"2024-09-13T15:00:48.049785+0000","last_undegraded":"2024-09-13T15:00:48.049785+0000","last_fullsized":"2024-09-13T15:00:48.049785+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":499,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:59:59.848136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:59:59.848136+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:17:35.075355+0000","scrub_duration":0.0018058359999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":191,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.222458+0000","last_change":"2024-09-13T15:00:49.221793+0000","last_active":"2024-09-13T15:00:49.222458+0000","last_peered":"2024-09-13T15:00:49.222458+0000","last_clean":"2024-09-13T15:00:49.222458+0000","last_became_active":"2024-09-13T15:00:49.221474+0000","last_became_peered":"2024-09-13T15:00:49.221474+0000","last_unstale":"2024-09-13T15:00:49.222458+0000","last_undegraded":"2024-09-13T15:00:49.222458+0000","last_fullsized":"2024-09-13T15:00:49.222458+0000","mapping_epoch":516,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:18:18.489468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":525,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.046106+0000","last_change":"2024-09-13T14:37:49.611774+0000","last_active":"2024-09-13T15:00:49.046106+0000","last_peered":"2024-09-13T15:00:49.046106+0000","last_clean":"2024-09-13T15:00:49.046106+0000","last_became_active":"2024-09-13T14:37:49.611088+0000","last_became_peered":"2024-09-13T14:37:49.611088+0000","last_unstale":"2024-09-13T15:00:49.046106+0000","last_undegraded":"2024-09-13T15:00:49.046106+0000","last_fullsized":"2024-09-13T15:00:49.046106+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:08.933213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"90'149","reported_seq":386,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:53.542069+0000","last_change":"2024-09-13T15:00:53.542206+0000","last_active":"2024-09-13T15:00:53.542069+0000","last_peered":"2024-09-13T15:00:53.542069+0000","last_clean":"2024-09-13T15:00:53.542069+0000","last_became_active":"2024-09-13T15:00:49.054039+0000","last_became_peered":"2024-09-13T15:00:49.054039+0000","last_unstale":"2024-09-13T15:00:53.542069+0000","last_undegraded":"2024-09-13T15:00:53.542069+0000","last_fullsized":"2024-09-13T15:00:53.542069+0000","mapping_epoch":516,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T15:00:53.542043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T15:00:53.542043+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:52:36.487371+0000","scrub_duration":0.0016316410000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":32,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.049833+0000","last_change":"2024-09-13T14:59:53.511092+0000","last_active":"2024-09-13T15:00:48.049833+0000","last_peered":"2024-09-13T15:00:48.049833+0000","last_clean":"2024-09-13T15:00:48.049833+0000","last_became_active":"2024-09-13T14:59:53.509999+0000","last_became_peered":"2024-09-13T14:59:53.509999+0000","last_unstale":"2024-09-13T15:00:48.049833+0000","last_undegraded":"2024-09-13T15:00:48.049833+0000","last_fullsized":"2024-09-13T15:00:48.049833+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:22:05.834497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":327,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.046477+0000","last_change":"2024-09-13T14:59:53.521704+0000","last_active":"2024-09-13T15:00:49.046477+0000","last_peered":"2024-09-13T15:00:49.046477+0000","last_clean":"2024-09-13T15:00:49.046477+0000","last_became_active":"2024-09-13T14:59:53.512700+0000","last_became_peered":"2024-09-13T14:59:53.512700+0000","last_unstale":"2024-09-13T15:00:49.046477+0000","last_undegraded":"2024-09-13T15:00:49.046477+0000","last_fullsized":"2024-09-13T15:00:49.046477+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:46:39.342033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"90'149","reported_seq":852,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.049126+0000","last_change":"2024-09-13T14:57:40.996705+0000","last_active":"2024-09-13T15:00:49.049126+0000","last_peered":"2024-09-13T15:00:49.049126+0000","last_clean":"2024-09-13T15:00:49.049126+0000","last_became_active":"2024-09-13T14:57:17.404299+0000","last_became_peered":"2024-09-13T14:57:17.404299+0000","last_unstale":"2024-09-13T15:00:49.049126+0000","last_undegraded":"2024-09-13T15:00:49.049126+0000","last_fullsized":"2024-09-13T15:00:49.049126+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:40.996402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:40.996402+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:04.912532+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":344,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.044834+0000","last_change":"2024-09-13T14:38:53.898907+0000","last_active":"2024-09-13T15:00:48.044834+0000","last_peered":"2024-09-13T15:00:48.044834+0000","last_clean":"2024-09-13T15:00:48.044834+0000","last_became_active":"2024-09-13T14:38:53.898572+0000","last_became_peered":"2024-09-13T14:38:53.898572+0000","last_unstale":"2024-09-13T15:00:48.044834+0000","last_undegraded":"2024-09-13T15:00:48.044834+0000","last_fullsized":"2024-09-13T15:00:48.044834+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:29.222150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":314,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.046842+0000","last_change":"2024-09-13T14:59:53.517392+0000","last_active":"2024-09-13T15:00:49.046842+0000","last_peered":"2024-09-13T15:00:49.046842+0000","last_clean":"2024-09-13T15:00:49.046842+0000","last_became_active":"2024-09-13T14:59:53.505025+0000","last_became_peered":"2024-09-13T14:59:53.505025+0000","last_unstale":"2024-09-13T15:00:49.046842+0000","last_undegraded":"2024-09-13T15:00:49.046842+0000","last_fullsized":"2024-09-13T15:00:49.046842+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:51:58.294980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"90'149","reported_seq":661,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.045100+0000","last_change":"2024-09-13T14:59:56.510267+0000","last_active":"2024-09-13T15:00:49.045100+0000","last_peered":"2024-09-13T15:00:49.045100+0000","last_clean":"2024-09-13T15:00:49.045100+0000","last_became_active":"2024-09-13T14:59:56.509734+0000","last_became_peered":"2024-09-13T14:59:56.509734+0000","last_unstale":"2024-09-13T15:00:49.045100+0000","last_undegraded":"2024-09-13T15:00:49.045100+0000","last_fullsized":"2024-09-13T15:00:49.045100+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":499,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:57:39.030538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:57:39.030538+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:12:01.212385+0000","scrub_duration":0.014515093999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":191,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.052774+0000","last_change":"2024-09-13T15:00:49.050246+0000","last_active":"2024-09-13T15:00:49.052774+0000","last_peered":"2024-09-13T15:00:49.052774+0000","last_clean":"2024-09-13T15:00:49.052774+0000","last_became_active":"2024-09-13T15:00:49.049440+0000","last_became_peered":"2024-09-13T15:00:49.049440+0000","last_unstale":"2024-09-13T15:00:49.052774+0000","last_undegraded":"2024-09-13T15:00:49.052774+0000","last_fullsized":"2024-09-13T15:00:49.052774+0000","mapping_epoch":516,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:48:01.002498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":975,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.051134+0000","last_change":"2024-09-13T14:50:27.491166+0000","last_active":"2024-09-13T15:00:49.051134+0000","last_peered":"2024-09-13T15:00:49.051134+0000","last_clean":"2024-09-13T15:00:49.051134+0000","last_became_active":"2024-09-13T14:50:27.489905+0000","last_became_peered":"2024-09-13T14:50:27.489905+0000","last_unstale":"2024-09-13T15:00:49.051134+0000","last_undegraded":"2024-09-13T15:00:49.051134+0000","last_fullsized":"2024-09-13T15:00:49.051134+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:32.237904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"90'149","reported_seq":603,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.044285+0000","last_change":"2024-09-13T14:47:50.892667+0000","last_active":"2024-09-13T15:00:48.044285+0000","last_peered":"2024-09-13T15:00:48.044285+0000","last_clean":"2024-09-13T15:00:48.044285+0000","last_became_active":"2024-09-13T14:47:50.883312+0000","last_became_peered":"2024-09-13T14:47:50.883312+0000","last_unstale":"2024-09-13T15:00:48.044285+0000","last_undegraded":"2024-09-13T15:00:48.044285+0000","last_fullsized":"2024-09-13T15:00:48.044285+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:45.441819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:45.441819+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:39.138573+0000","scrub_duration":0.021368558999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":512,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.044238+0000","last_change":"2024-09-13T14:59:53.513320+0000","last_active":"2024-09-13T15:00:48.044238+0000","last_peered":"2024-09-13T15:00:48.044238+0000","last_clean":"2024-09-13T15:00:48.044238+0000","last_became_active":"2024-09-13T14:59:53.513034+0000","last_became_peered":"2024-09-13T14:59:53.513034+0000","last_unstale":"2024-09-13T15:00:48.044238+0000","last_undegraded":"2024-09-13T15:00:48.044238+0000","last_fullsized":"2024-09-13T15:00:48.044238+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:33:20.466928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":500,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.047547+0000","last_change":"2024-09-13T15:00:03.538450+0000","last_active":"2024-09-13T15:00:49.047547+0000","last_peered":"2024-09-13T15:00:49.047547+0000","last_clean":"2024-09-13T15:00:49.047547+0000","last_became_active":"2024-09-13T15:00:03.537992+0000","last_became_peered":"2024-09-13T15:00:03.537992+0000","last_unstale":"2024-09-13T15:00:49.047547+0000","last_undegraded":"2024-09-13T15:00:49.047547+0000","last_fullsized":"2024-09-13T15:00:49.047547+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:53:14.916796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"90'149","reported_seq":822,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.220923+0000","last_change":"2024-09-13T14:59:56.508891+0000","last_active":"2024-09-13T15:00:49.220923+0000","last_peered":"2024-09-13T15:00:49.220923+0000","last_clean":"2024-09-13T15:00:49.220923+0000","last_became_active":"2024-09-13T14:59:56.508530+0000","last_became_peered":"2024-09-13T14:59:56.508530+0000","last_unstale":"2024-09-13T15:00:49.220923+0000","last_undegraded":"2024-09-13T15:00:49.220923+0000","last_fullsized":"2024-09-13T15:00:49.220923+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":499,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:51.226748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:51.226748+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:16.784573+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":500,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.047018+0000","last_change":"2024-09-13T15:00:03.536672+0000","last_active":"2024-09-13T15:00:49.047018+0000","last_peered":"2024-09-13T15:00:49.047018+0000","last_clean":"2024-09-13T15:00:49.047018+0000","last_became_active":"2024-09-13T15:00:03.536379+0000","last_became_peered":"2024-09-13T15:00:03.536379+0000","last_unstale":"2024-09-13T15:00:49.047018+0000","last_undegraded":"2024-09-13T15:00:49.047018+0000","last_fullsized":"2024-09-13T15:00:49.047018+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:04:59.498496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":323,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.044586+0000","last_change":"2024-09-13T14:36:13.862181+0000","last_active":"2024-09-13T15:00:48.044586+0000","last_peered":"2024-09-13T15:00:48.044586+0000","last_clean":"2024-09-13T15:00:48.044586+0000","last_became_active":"2024-09-13T14:36:13.861873+0000","last_became_peered":"2024-09-13T14:36:13.861873+0000","last_unstale":"2024-09-13T15:00:48.044586+0000","last_undegraded":"2024-09-13T15:00:48.044586+0000","last_fullsized":"2024-09-13T15:00:48.044586+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:14.085700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"90'149","reported_seq":586,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.043939+0000","last_change":"2024-09-13T14:47:41.482552+0000","last_active":"2024-09-13T15:00:48.043939+0000","last_peered":"2024-09-13T15:00:48.043939+0000","last_clean":"2024-09-13T15:00:48.043939+0000","last_became_active":"2024-09-13T14:47:38.466659+0000","last_became_peered":"2024-09-13T14:47:38.466659+0000","last_unstale":"2024-09-13T15:00:48.043939+0000","last_undegraded":"2024-09-13T15:00:48.043939+0000","last_fullsized":"2024-09-13T15:00:48.043939+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:41.482364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:41.482364+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:50:49.239389+0000","scrub_duration":0.027843855000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":572,"reported_epoch":516,"state":"active+clean","last_fresh":"2024-09-13T15:00:48.045360+0000","last_change":"2024-09-13T14:59:53.524423+0000","last_active":"2024-09-13T15:00:48.045360+0000","last_peered":"2024-09-13T15:00:48.045360+0000","last_clean":"2024-09-13T15:00:48.045360+0000","last_became_active":"2024-09-13T14:59:53.512737+0000","last_became_peered":"2024-09-13T14:59:53.512737+0000","last_unstale":"2024-09-13T15:00:48.045360+0000","last_undegraded":"2024-09-13T15:00:48.045360+0000","last_fullsized":"2024-09-13T15:00:48.045360+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:28.209772+0000","last_clean_scrub_stamp":"2024-09-13T14:11:28.209772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:40:11.840478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":136,"num_keys_recovered":40,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":949,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.050261+0000","last_change":"2024-09-13T14:47:47.868783+0000","last_active":"2024-09-13T15:00:49.050261+0000","last_peered":"2024-09-13T15:00:49.050261+0000","last_clean":"2024-09-13T15:00:49.050261+0000","last_became_active":"2024-09-13T14:47:47.868478+0000","last_became_peered":"2024-09-13T14:47:47.868478+0000","last_unstale":"2024-09-13T15:00:49.050261+0000","last_undegraded":"2024-09-13T15:00:49.050261+0000","last_fullsized":"2024-09-13T15:00:49.050261+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:11:29.211560+0000","last_clean_scrub_stamp":"2024-09-13T14:11:29.211560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:27:01.784142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"90'149","reported_seq":707,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-09-13T15:00:49.047035+0000","last_change":"2024-09-13T15:00:03.537967+0000","last_active":"2024-09-13T15:00:49.047035+0000","last_peered":"2024-09-13T15:00:49.047035+0000","last_clean":"2024-09-13T15:00:49.047035+0000","last_became_active":"2024-09-13T15:00:03.536110+0000","last_became_peered":"2024-09-13T15:00:03.536110+0000","last_unstale":"2024-09-13T15:00:49.047035+0000","last_undegraded":"2024-09-13T15:00:49.047035+0000","last_fullsized":"2024-09-13T15:00:49.047035+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T14:47:38.688341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:07:41.927740+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.688341+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:46:23.002387+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":7202,"num_keys_recovered":40,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25132,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":8388646,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5038,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4619,"ondisk_log_size":4619,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234614,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":958364,"kb_used_data":6796,"kb_used_omap":1,"kb_used_meta":951550,"kb_avail":92786788,"statfs":{"total":95995035648,"available":95013670912,"internally_reserved":0,"allocated":6959104,"data_stored":4343398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":974387659},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":239,"seq":1026497184032,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":699208,"kb_used_data":10944,"kb_used_omap":8,"kb_used_meta":688247,"kb_avail":93045944,"statfs":{"total":95995035648,"available":95279046656,"internally_reserved":0,"allocated":11206656,"data_stored":8602625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8453,"internal_metadata":704765691},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757879,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1258440,"kb_used_data":10876,"kb_used_omap":1,"kb_used_meta":1247550,"kb_avail":92486712,"statfs":{"total":95995035648,"available":94706393088,"internally_reserved":0,"allocated":11137024,"data_stored":8533401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1277491654},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019523,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":908052,"kb_used_data":6532,"kb_used_omap":1,"kb_used_meta":901502,"kb_avail":92837100,"statfs":{"total":95995035648,"available":95065190400,"internally_reserved":0,"allocated":6688768,"data_stored":4080698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":923138502},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435705,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1167932,"kb_used_data":6836,"kb_used_omap":5,"kb_used_meta":1161082,"kb_avail":92577220,"statfs":{"total":95995035648,"available":94799073280,"internally_reserved":0,"allocated":7000064,"data_stored":4403851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6018,"internal_metadata":1188948094},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":104,"seq":446676599234,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":507720,"kb_used_data":6780,"kb_used_omap":7,"kb_used_meta":500920,"kb_avail":93237432,"statfs":{"total":95995035648,"available":95475130368,"internally_reserved":0,"allocated":6942720,"data_stored":4339385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8010,"internal_metadata":512942262},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":99,"seq":425201762760,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":519872,"kb_used_data":10608,"kb_used_omap":7,"kb_used_meta":509240,"kb_avail":93225280,"statfs":{"total":95995035648,"available":95462686720,"internally_reserved":0,"allocated":10862592,"data_stored":8270131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7759,"internal_metadata":521462193},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926284,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1209608,"kb_used_data":6840,"kb_used_omap":7,"kb_used_meta":1202744,"kb_avail":92535544,"statfs":{"total":95995035648,"available":94756397056,"internally_reserved":0,"allocated":7004160,"data_stored":4404365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7896,"internal_metadata":1231610152},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3640,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1398,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6302,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2776,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3028,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6860,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T15:01:39.440 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-13T15:01:39.441 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/parallel.py", line 48, in task with parallel.parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test cls/test_cls_rbd.sh) on smithi102 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=reef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' 2024-09-13T15:01:39.624 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=36fac01c397044609f02d01fc3e98cdf Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/parallel.py", line 48, in task with parallel.parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test cls/test_cls_rbd.sh) on smithi102 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=reef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' 2024-09-13T15:01:39.627 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-13T15:01:39.637 INFO:tasks.cephadm:Teardown begin 2024-09-13T15:01:39.637 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-13T15:01:39.692 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-13T15:01:39.735 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-13T15:01:39.736 DEBUG:teuthology.orchestra.run.smithi102:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-13T15:01:39.751 DEBUG:teuthology.orchestra.run.smithi195:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-13T15:01:39.771 INFO:tasks.cephadm:Stopping all daemons... 2024-09-13T15:01:39.772 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-13T15:01:39.772 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-08cee502-71d9-11ef-bceb-c7b262605968@mon.a 2024-09-13T15:01:39.997 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:39 smithi195 ceph-mon[104262]: from='client.47213 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:01:39.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:39 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:01:39.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:39 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:01:39.998 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:39 smithi195 ceph-mon[104262]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T15:01:40.136 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:01:39 smithi102 ceph-mon[115269]: from='client.47213 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:01:40.137 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:01:39 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:01:40.137 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:01:39 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:01:40.137 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:01:39 smithi102 ceph-mon[115269]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T15:01:40.137 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:01:39 smithi102 systemd[1]: Stopping Ceph mon.a for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T15:01:40.138 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:01:39 smithi102 ceph-mon[110183]: from='client.47213 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:01:40.138 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:01:39 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:01:40.138 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:01:39 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:01:40.138 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:01:39 smithi102 ceph-mon[110183]: from='mgr.34107 172.21.15.102:0/1920607954' entity='mgr.y' 2024-09-13T15:01:40.419 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 15:01:40 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: [13/Sep/2024:15:01:40] ENGINE Bus STOPPING 2024-09-13T15:01:40.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:01:40 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-a[110159]: 2024-09-13T15:01:40.134+0000 7fa3ed74d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:01:40.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:01:40 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-a[110159]: 2024-09-13T15:01:40.134+0000 7fa3ed74d640 -1 mon.a@0(leader) e4 *** Got Signal Terminated *** 2024-09-13T15:01:40.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:01:40 smithi102 podman[420596]: 2024-09-13 15:01:40.206134766 +0000 UTC m=+0.219538022 container died f3234b7bbdde9ed3592a602cc8a3bf41419784f4a968be100e922d6736358f59 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-aclamk-cbt-combined-d58f730, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-09-13T15:01:40.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:01:40 smithi102 podman[420596]: 2024-09-13 15:01:40.666777967 +0000 UTC m=+0.680181289 container cleanup f3234b7bbdde9ed3592a602cc8a3bf41419784f4a968be100e922d6736358f59 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-a, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-aclamk-cbt-combined-d58f730, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2) 2024-09-13T15:01:40.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Sep 13 15:01:40 smithi102 bash[420596]: ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-a 2024-09-13T15:01:40.919 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 15:01:40 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: [13/Sep/2024:15:01:40] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-13T15:01:40.919 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 15:01:40 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: [13/Sep/2024:15:01:40] ENGINE Bus STOPPED 2024-09-13T15:01:40.920 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 15:01:40 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: [13/Sep/2024:15:01:40] ENGINE Bus STARTING 2024-09-13T15:01:40.920 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 15:01:40 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: [13/Sep/2024:15:01:40] ENGINE Serving on http://:::9283 2024-09-13T15:01:40.920 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 15:01:40 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: [13/Sep/2024:15:01:40] ENGINE Bus STARTED 2024-09-13T15:01:41.139 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-08cee502-71d9-11ef-bceb-c7b262605968@mon.a.service' 2024-09-13T15:01:41.192 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:01:41.192 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-13T15:01:41.192 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-09-13T15:01:41.192 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-08cee502-71d9-11ef-bceb-c7b262605968@mon.c 2024-09-13T15:01:41.500 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:01:41 smithi102 systemd[1]: Stopping Ceph mon.c for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T15:01:41.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:01:41 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-c[115246]: 2024-09-13T15:01:41.497+0000 7f28c5cb6640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:01:41.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:01:41 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-c[115246]: 2024-09-13T15:01:41.498+0000 7f28c5cb6640 -1 mon.c@2(peon) e4 *** Got Signal Terminated *** 2024-09-13T15:01:41.919 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:01:41 smithi102 podman[420739]: 2024-09-13 15:01:41.703336029 +0000 UTC m=+0.330073916 container died dba6b3d55296917b6bbfcb26dc0d57810fa907f423fa552fca517d4a49931029 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-c, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-aclamk-cbt-combined-d58f730) 2024-09-13T15:01:41.919 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 15:01:41 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: [13/Sep/2024:15:01:41] ENGINE Bus STOPPING 2024-09-13T15:01:41.920 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 15:01:41 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: [13/Sep/2024:15:01:41] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-13T15:01:41.920 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 15:01:41 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: [13/Sep/2024:15:01:41] ENGINE Bus STOPPED 2024-09-13T15:01:41.920 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 15:01:41 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: [13/Sep/2024:15:01:41] ENGINE Bus STARTING 2024-09-13T15:01:41.920 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 15:01:41 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: [13/Sep/2024:15:01:41] ENGINE Serving on http://:::9283 2024-09-13T15:01:41.921 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 15:01:41 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: [13/Sep/2024:15:01:41] ENGINE Bus STARTED 2024-09-13T15:01:42.418 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:01:42 smithi102 podman[420739]: 2024-09-13 15:01:42.172450086 +0000 UTC m=+0.799187960 container cleanup dba6b3d55296917b6bbfcb26dc0d57810fa907f423fa552fca517d4a49931029 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-c, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True) 2024-09-13T15:01:42.419 INFO:journalctl@ceph.mon.c.smithi102.stdout:Sep 13 15:01:42 smithi102 bash[420739]: ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-c 2024-09-13T15:01:42.599 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-08cee502-71d9-11ef-bceb-c7b262605968@mon.c.service' 2024-09-13T15:01:42.634 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:01:42.634 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-09-13T15:01:42.635 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-09-13T15:01:42.635 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ceph-08cee502-71d9-11ef-bceb-c7b262605968@mon.b 2024-09-13T15:01:42.952 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:42 smithi195 systemd[1]: Stopping Ceph mon.b for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T15:01:43.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:42 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-b[104239]: 2024-09-13T15:01:42.949+0000 7f1f895a0640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:01:43.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:42 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-b[104239]: 2024-09-13T15:01:42.949+0000 7f1f895a0640 -1 mon.b@1(peon) e4 *** Got Signal Terminated *** 2024-09-13T15:01:43.248 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:43 smithi195 podman[285895]: 2024-09-13 15:01:43.07263911 +0000 UTC m=+0.214338230 container died 9921a2aa81682b9156cd45bc658f893cec490637ebe670d36db4a003bce58ba6 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-aclamk-cbt-combined-d58f730) 2024-09-13T15:01:43.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:43 smithi195 podman[285895]: 2024-09-13 15:01:43.550606272 +0000 UTC m=+0.692305395 container cleanup 9921a2aa81682b9156cd45bc658f893cec490637ebe670d36db4a003bce58ba6 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, GIT_BRANCH=HEAD) 2024-09-13T15:01:43.748 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:43 smithi195 bash[285895]: ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-b 2024-09-13T15:01:44.010 DEBUG:teuthology.orchestra.run.smithi195:> sudo pkill -f 'journalctl -f -n 0 -u ceph-08cee502-71d9-11ef-bceb-c7b262605968@mon.b.service' 2024-09-13T15:01:44.032 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:43 smithi195 podman[285913]: 2024-09-13 15:01:43.777421303 +0000 UTC m=+0.707079498 container remove 9921a2aa81682b9156cd45bc658f893cec490637ebe670d36db4a003bce58ba6 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mon-b, CEPH_POINT_RELEASE=, RELEASE=wip-aclamk-cbt-combined-d58f730, ceph=True, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T15:01:44.032 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:44 smithi195 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@mon.b.service: Deactivated successfully. 2024-09-13T15:01:44.032 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:44 smithi195 systemd[1]: Stopped Ceph mon.b for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T15:01:44.033 INFO:journalctl@ceph.mon.b.smithi195.stdout:Sep 13 15:01:44 smithi195 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@mon.b.service: Consumed 24.086s CPU time. 2024-09-13T15:01:44.047 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:01:44.047 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-09-13T15:01:44.048 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-13T15:01:44.048 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-08cee502-71d9-11ef-bceb-c7b262605968@mgr.y 2024-09-13T15:01:44.355 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 15:01:44 smithi102 systemd[1]: Stopping Ceph mgr.y for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T15:01:44.663 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 15:01:44 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: 2024-09-13T15:01:44.352+0000 7f0cf0aa1640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:01:44.663 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 15:01:44 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y[91219]: 2024-09-13T15:01:44.352+0000 7f0cf0aa1640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-09-13T15:01:44.663 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 15:01:44 smithi102 podman[420881]: 2024-09-13 15:01:44.39597518 +0000 UTC m=+0.150154700 container died 6b12b28dcde1bdb8dfb811817c1ccf703d65c60eccbbeca5b14d62d4020c8ae6 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-13T15:01:44.914 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 15:01:44 smithi102 podman[420881]: 2024-09-13 15:01:44.91133792 +0000 UTC m=+0.665517483 container cleanup 6b12b28dcde1bdb8dfb811817c1ccf703d65c60eccbbeca5b14d62d4020c8ae6 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, RELEASE=wip-aclamk-cbt-combined-d58f730, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T15:01:44.915 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 15:01:44 smithi102 bash[420881]: ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y 2024-09-13T15:01:45.385 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-08cee502-71d9-11ef-bceb-c7b262605968@mgr.y.service' 2024-09-13T15:01:45.407 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 15:01:45 smithi102 podman[420899]: 2024-09-13 15:01:45.154630574 +0000 UTC m=+0.761325391 container remove 6b12b28dcde1bdb8dfb811817c1ccf703d65c60eccbbeca5b14d62d4020c8ae6 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-y, org.label-schema.schema-version=1.0, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-09-13T15:01:45.407 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 15:01:45 smithi102 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@mgr.y.service: Deactivated successfully. 2024-09-13T15:01:45.407 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 15:01:45 smithi102 systemd[1]: Stopped Ceph mgr.y for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T15:01:45.408 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Sep 13 15:01:45 smithi102 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@mgr.y.service: Consumed 48.252s CPU time. 2024-09-13T15:01:45.420 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:01:45.420 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-13T15:01:45.421 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-13T15:01:45.421 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ceph-08cee502-71d9-11ef-bceb-c7b262605968@mgr.x 2024-09-13T15:01:45.747 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 15:01:45 smithi195 systemd[1]: Stopping Ceph mgr.x for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T15:01:46.048 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 15:01:45 smithi195 podman[286025]: 2024-09-13 15:01:45.763832135 +0000 UTC m=+0.149807130 container died 4a23b8f0ea8b5b747dff1959e85fcf3c6fcb4a43aa3c8f4ddecab77a898b6d4f (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-09-13T15:01:46.408 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 15:01:46 smithi195 podman[286025]: 2024-09-13 15:01:46.197443504 +0000 UTC m=+0.583418500 container cleanup 4a23b8f0ea8b5b747dff1959e85fcf3c6fcb4a43aa3c8f4ddecab77a898b6d4f (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-13T15:01:46.408 INFO:journalctl@ceph.mgr.x.smithi195.stdout:Sep 13 15:01:46 smithi195 bash[286025]: ceph-08cee502-71d9-11ef-bceb-c7b262605968-mgr-x 2024-09-13T15:01:46.529 DEBUG:teuthology.orchestra.run.smithi195:> sudo pkill -f 'journalctl -f -n 0 -u ceph-08cee502-71d9-11ef-bceb-c7b262605968@mgr.x.service' 2024-09-13T15:01:46.564 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:01:46.565 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-13T15:01:46.565 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-13T15:01:46.565 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-08cee502-71d9-11ef-bceb-c7b262605968@osd.0 2024-09-13T15:01:46.877 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 15:01:46 smithi102 systemd[1]: Stopping Ceph osd.0 for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T15:01:47.168 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 15:01:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0[127501]: 2024-09-13T15:01:46.874+0000 7fe9a3e43640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:01:47.169 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 15:01:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0[127501]: 2024-09-13T15:01:46.874+0000 7fe9a3e43640 -1 osd.0 517 *** Got signal Terminated *** 2024-09-13T15:01:47.169 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 15:01:46 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0[127501]: 2024-09-13T15:01:46.874+0000 7fe9a3e43640 -1 osd.0 517 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T15:01:51.498 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 15:01:51 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[89530]: ts=2024-09-13T15:01:51.202Z caller=refresh.go:90 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.102:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.102:8765: connect: connection refused" 2024-09-13T15:01:51.499 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 15:01:51 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[89530]: ts=2024-09-13T15:01:51.202Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.102:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.102:8765: connect: connection refused" 2024-09-13T15:01:51.499 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 15:01:51 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[89530]: ts=2024-09-13T15:01:51.202Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.102:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.102:8765: connect: connection refused" 2024-09-13T15:01:51.499 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 15:01:51 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[89530]: ts=2024-09-13T15:01:51.205Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.102:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.102:8765: connect: connection refused" 2024-09-13T15:01:51.499 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 15:01:51 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[89530]: ts=2024-09-13T15:01:51.205Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nfs msg="Unable to refresh target groups" err="Get \"http://172.21.15.102:8765/sd/prometheus/sd-config?service=nfs\": dial tcp 172.21.15.102:8765: connect: connection refused" 2024-09-13T15:01:51.499 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 15:01:51 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[89530]: ts=2024-09-13T15:01:51.205Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.102:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.102:8765: connect: connection refused" 2024-09-13T15:01:52.797 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 15:01:52 smithi102 podman[421013]: 2024-09-13 15:01:52.468485011 +0000 UTC m=+5.719232869 container stop d5c895f22b2a5ac04c6f0c37c5b546b2e36fd140969231235a504a9bfb77edd7 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.build-date=20240731) 2024-09-13T15:01:52.797 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 15:01:52 smithi102 podman[421013]: 2024-09-13 15:01:52.469073638 +0000 UTC m=+5.719821504 container died d5c895f22b2a5ac04c6f0c37c5b546b2e36fd140969231235a504a9bfb77edd7 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, RELEASE=wip-aclamk-cbt-combined-d58f730, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-13T15:01:53.057 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 15:01:53 smithi102 podman[421013]: 2024-09-13 15:01:53.041633152 +0000 UTC m=+6.292381009 container cleanup d5c895f22b2a5ac04c6f0c37c5b546b2e36fd140969231235a504a9bfb77edd7 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=wip-aclamk-cbt-combined-d58f730, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-09-13T15:01:53.058 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 15:01:53 smithi102 bash[421013]: ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0 2024-09-13T15:01:53.669 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 15:01:53 smithi102 podman[421035]: 2024-09-13 15:01:53.343125175 +0000 UTC m=+0.863733275 container remove d5c895f22b2a5ac04c6f0c37c5b546b2e36fd140969231235a504a9bfb77edd7 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0, io.buildah.version=1.37.2, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-13T15:01:54.039 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 15:01:53 smithi102 podman[421135]: 2024-09-13 15:01:53.710850516 +0000 UTC m=+0.110921330 container create f1e9966f0502aff34b8d1d3ad57d66113f25be73b793f2eccbf17b07e33827ae (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-aclamk-cbt-combined-d58f730, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-13T15:01:54.040 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 15:01:53 smithi102 podman[421135]: 2024-09-13 15:01:53.615939104 +0000 UTC m=+0.016009932 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T15:01:54.040 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 15:01:53 smithi102 podman[421135]: 2024-09-13 15:01:53.938877722 +0000 UTC m=+0.338948536 container init f1e9966f0502aff34b8d1d3ad57d66113f25be73b793f2eccbf17b07e33827ae (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, ceph=True) 2024-09-13T15:01:54.040 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 15:01:53 smithi102 podman[421135]: 2024-09-13 15:01:53.944081023 +0000 UTC m=+0.344151837 container start f1e9966f0502aff34b8d1d3ad57d66113f25be73b793f2eccbf17b07e33827ae (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0-deactivate, RELEASE=wip-aclamk-cbt-combined-d58f730, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-13T15:01:54.040 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 15:01:54 smithi102 podman[421135]: 2024-09-13 15:01:54.0387496 +0000 UTC m=+0.438820431 container attach f1e9966f0502aff34b8d1d3ad57d66113f25be73b793f2eccbf17b07e33827ae (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-aclamk-cbt-combined-d58f730, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-13T15:01:54.419 INFO:journalctl@ceph.osd.0.smithi102.stdout:Sep 13 15:01:54 smithi102 podman[421135]: 2024-09-13 15:01:54.209885899 +0000 UTC m=+0.609956715 container died f1e9966f0502aff34b8d1d3ad57d66113f25be73b793f2eccbf17b07e33827ae (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-0-deactivate, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True) 2024-09-13T15:01:55.083 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-08cee502-71d9-11ef-bceb-c7b262605968@osd.0.service' 2024-09-13T15:01:55.124 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:01:55.124 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-13T15:01:55.124 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-13T15:01:55.125 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-08cee502-71d9-11ef-bceb-c7b262605968@osd.1 2024-09-13T15:01:55.419 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 15:01:55 smithi102 systemd[1]: Stopping Ceph osd.1 for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T15:01:55.918 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 15:01:55 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1[132573]: 2024-09-13T15:01:55.480+0000 7f1d35a8e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:01:55.919 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 15:01:55 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1[132573]: 2024-09-13T15:01:55.480+0000 7f1d35a8e640 -1 osd.1 517 *** Got signal Terminated *** 2024-09-13T15:01:55.919 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 15:01:55 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1[132573]: 2024-09-13T15:01:55.480+0000 7f1d35a8e640 -1 osd.1 517 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T15:02:01.290 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 15:02:00 smithi102 podman[421265]: 2024-09-13 15:02:00.959525938 +0000 UTC m=+5.609593074 container died 5d264b086bc0b2c6d454c11958068b5a7787de1c28bb3b0aae62e9d74c35fab9 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1, RELEASE=wip-aclamk-cbt-combined-d58f730, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0) 2024-09-13T15:02:01.668 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 15:02:01 smithi102 podman[421265]: 2024-09-13 15:02:01.46274425 +0000 UTC m=+6.112811372 container cleanup 5d264b086bc0b2c6d454c11958068b5a7787de1c28bb3b0aae62e9d74c35fab9 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-aclamk-cbt-combined-d58f730, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-09-13T15:02:01.669 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 15:02:01 smithi102 bash[421265]: ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1 2024-09-13T15:02:02.085 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 15:02:01 smithi102 podman[421286]: 2024-09-13 15:02:01.722643934 +0000 UTC m=+0.756308329 container remove 5d264b086bc0b2c6d454c11958068b5a7787de1c28bb3b0aae62e9d74c35fab9 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T15:02:02.419 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 15:02:02 smithi102 podman[421381]: 2024-09-13 15:02:01.98656056 +0000 UTC m=+0.015943577 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T15:02:02.419 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 15:02:02 smithi102 podman[421381]: 2024-09-13 15:02:02.098456045 +0000 UTC m=+0.127839046 container create 7f4e70ee7999e8d39be484b4d3fe45551a93ba323476e43455b4149064266de4 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-13T15:02:02.419 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 15:02:02 smithi102 podman[421381]: 2024-09-13 15:02:02.317680153 +0000 UTC m=+0.347063163 container init 7f4e70ee7999e8d39be484b4d3fe45551a93ba323476e43455b4149064266de4 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1-deactivate, RELEASE=wip-aclamk-cbt-combined-d58f730, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-13T15:02:02.419 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 15:02:02 smithi102 podman[421381]: 2024-09-13 15:02:02.32265835 +0000 UTC m=+0.352041355 container start 7f4e70ee7999e8d39be484b4d3fe45551a93ba323476e43455b4149064266de4 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1-deactivate, ceph=True, GIT_CLEAN=True, RELEASE=wip-aclamk-cbt-combined-d58f730, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T15:02:02.797 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 15:02:02 smithi102 podman[421381]: 2024-09-13 15:02:02.417736894 +0000 UTC m=+0.447119905 container attach 7f4e70ee7999e8d39be484b4d3fe45551a93ba323476e43455b4149064266de4 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1-deactivate, RELEASE=wip-aclamk-cbt-combined-d58f730, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T15:02:02.797 INFO:journalctl@ceph.osd.1.smithi102.stdout:Sep 13 15:02:02 smithi102 podman[421381]: 2024-09-13 15:02:02.510634065 +0000 UTC m=+0.540017070 container died 7f4e70ee7999e8d39be484b4d3fe45551a93ba323476e43455b4149064266de4 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-1-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux ) 2024-09-13T15:02:03.088 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-08cee502-71d9-11ef-bceb-c7b262605968@osd.1.service' 2024-09-13T15:02:03.128 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:02:03.128 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-13T15:02:03.129 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-13T15:02:03.129 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-08cee502-71d9-11ef-bceb-c7b262605968@osd.2 2024-09-13T15:02:03.356 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 15:02:03 smithi102 systemd[1]: Stopping Ceph osd.2 for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T15:02:03.668 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 15:02:03 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2[137384]: 2024-09-13T15:02:03.467+0000 7fbcf8509640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:02:03.669 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 15:02:03 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2[137384]: 2024-09-13T15:02:03.467+0000 7fbcf8509640 -1 osd.2 517 *** Got signal Terminated *** 2024-09-13T15:02:03.669 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 15:02:03 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2[137384]: 2024-09-13T15:02:03.467+0000 7fbcf8509640 -1 osd.2 517 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T15:02:09.393 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 15:02:09 smithi102 podman[421507]: 2024-09-13 15:02:09.094872099 +0000 UTC m=+5.735835600 container died fba29968c6b9ca62e1714f20028939d268fd8f4d5be7ada86b12d86fec5c73d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-aclamk-cbt-combined-d58f730, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0) 2024-09-13T15:02:09.669 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 15:02:09 smithi102 podman[421507]: 2024-09-13 15:02:09.566002972 +0000 UTC m=+6.206966476 container cleanup fba29968c6b9ca62e1714f20028939d268fd8f4d5be7ada86b12d86fec5c73d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2, RELEASE=wip-aclamk-cbt-combined-d58f730, ceph=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-13T15:02:09.669 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 15:02:09 smithi102 bash[421507]: ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2 2024-09-13T15:02:10.169 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 15:02:09 smithi102 podman[421528]: 2024-09-13 15:02:09.759145486 +0000 UTC m=+0.661297740 container remove fba29968c6b9ca62e1714f20028939d268fd8f4d5be7ada86b12d86fec5c73d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True) 2024-09-13T15:02:10.447 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 15:02:10 smithi102 podman[421619]: 2024-09-13 15:02:10.078588375 +0000 UTC m=+0.016203726 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T15:02:10.447 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 15:02:10 smithi102 podman[421619]: 2024-09-13 15:02:10.193315026 +0000 UTC m=+0.130930374 container create 1e3192f96acd9fa74b02cad51161753872a56e411657bd7405b98611ba66af56 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=wip-aclamk-cbt-combined-d58f730, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T15:02:10.448 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 15:02:10 smithi102 podman[421619]: 2024-09-13 15:02:10.446120139 +0000 UTC m=+0.383735483 container init 1e3192f96acd9fa74b02cad51161753872a56e411657bd7405b98611ba66af56 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-aclamk-cbt-combined-d58f730, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T15:02:10.919 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 15:02:10 smithi102 podman[421619]: 2024-09-13 15:02:10.451621999 +0000 UTC m=+0.389237343 container start 1e3192f96acd9fa74b02cad51161753872a56e411657bd7405b98611ba66af56 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-aclamk-cbt-combined-d58f730) 2024-09-13T15:02:10.919 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 15:02:10 smithi102 podman[421619]: 2024-09-13 15:02:10.53772745 +0000 UTC m=+0.475342829 container attach 1e3192f96acd9fa74b02cad51161753872a56e411657bd7405b98611ba66af56 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=) 2024-09-13T15:02:10.919 INFO:journalctl@ceph.osd.2.smithi102.stdout:Sep 13 15:02:10 smithi102 podman[421619]: 2024-09-13 15:02:10.638309361 +0000 UTC m=+0.575924706 container died 1e3192f96acd9fa74b02cad51161753872a56e411657bd7405b98611ba66af56 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-2-deactivate, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-09-13T15:02:11.341 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-08cee502-71d9-11ef-bceb-c7b262605968@osd.2.service' 2024-09-13T15:02:11.378 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:02:11.378 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-13T15:02:11.379 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-13T15:02:11.379 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-08cee502-71d9-11ef-bceb-c7b262605968@osd.3 2024-09-13T15:02:11.669 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 15:02:11 smithi102 systemd[1]: Stopping Ceph osd.3 for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T15:02:12.168 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 15:02:11 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3[142208]: 2024-09-13T15:02:11.738+0000 7fd1db533640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:02:12.169 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 15:02:11 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3[142208]: 2024-09-13T15:02:11.738+0000 7fd1db533640 -1 osd.3 517 *** Got signal Terminated *** 2024-09-13T15:02:12.169 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 15:02:11 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3[142208]: 2024-09-13T15:02:11.738+0000 7fd1db533640 -1 osd.3 517 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T15:02:14.669 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 15:02:14 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3[142208]: 2024-09-13T15:02:14.329+0000 7fd1d3b40640 -1 osd.3 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:50.470303+0000 front 2024-09-13T15:01:50.470317+0000 (oldest deadline 2024-09-13T15:02:13.970392+0000) 2024-09-13T15:02:15.549 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:15 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:15.267+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:50.534049+0000 front 2024-09-13T15:01:50.534014+0000 (oldest deadline 2024-09-13T15:02:14.634064+0000) 2024-09-13T15:02:15.669 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 15:02:15 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3[142208]: 2024-09-13T15:02:15.331+0000 7fd1d3b40640 -1 osd.3 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:50.470303+0000 front 2024-09-13T15:01:50.470317+0000 (oldest deadline 2024-09-13T15:02:13.970392+0000) 2024-09-13T15:02:15.822 INFO:journalctl@ceph.osd.4.smithi195.stdout:Sep 13 15:02:15 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-4[38505]: 2024-09-13T15:02:15.647+0000 7fb6840a8640 -1 osd.4 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:50.052522+0000 front 2024-09-13T15:01:50.052460+0000 (oldest deadline 2024-09-13T15:02:14.752479+0000) 2024-09-13T15:02:15.822 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:15 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:15.547+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.963343+0000 front 2024-09-13T15:01:49.963380+0000 (oldest deadline 2024-09-13T15:02:14.663480+0000) 2024-09-13T15:02:16.566 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:16 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:16.543+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.963343+0000 front 2024-09-13T15:01:49.963380+0000 (oldest deadline 2024-09-13T15:02:14.663480+0000) 2024-09-13T15:02:16.566 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:16 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:16.304+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:50.534049+0000 front 2024-09-13T15:01:50.534014+0000 (oldest deadline 2024-09-13T15:02:14.634064+0000) 2024-09-13T15:02:16.669 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 15:02:16 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3[142208]: 2024-09-13T15:02:16.297+0000 7fd1d3b40640 -1 osd.3 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:50.470303+0000 front 2024-09-13T15:01:50.470317+0000 (oldest deadline 2024-09-13T15:02:13.970392+0000) 2024-09-13T15:02:16.998 INFO:journalctl@ceph.osd.4.smithi195.stdout:Sep 13 15:02:16 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-4[38505]: 2024-09-13T15:02:16.612+0000 7fb6840a8640 -1 osd.4 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:50.052522+0000 front 2024-09-13T15:01:50.052460+0000 (oldest deadline 2024-09-13T15:02:14.752479+0000) 2024-09-13T15:02:16.998 INFO:journalctl@ceph.osd.5.smithi195.stdout:Sep 13 15:02:16 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-5[44425]: 2024-09-13T15:02:16.564+0000 7fa743d86640 -1 osd.5 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.927574+0000 front 2024-09-13T15:01:49.927636+0000 (oldest deadline 2024-09-13T15:02:15.827343+0000) 2024-09-13T15:02:17.522 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 15:02:17 smithi102 podman[421746]: 2024-09-13 15:02:17.264139509 +0000 UTC m=+5.659909604 container died cb2596701036e089f0c6d8e67a427d5a298858b68305fa58f72e164db262dea8 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True) 2024-09-13T15:02:17.589 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:17 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:17.530+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.963343+0000 front 2024-09-13T15:01:49.963380+0000 (oldest deadline 2024-09-13T15:02:14.663480+0000) 2024-09-13T15:02:17.590 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:17 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:17.292+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:50.534049+0000 front 2024-09-13T15:01:50.534014+0000 (oldest deadline 2024-09-13T15:02:14.634064+0000) 2024-09-13T15:02:17.918 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 15:02:17 smithi102 podman[421746]: 2024-09-13 15:02:17.702978726 +0000 UTC m=+6.098748821 container cleanup cb2596701036e089f0c6d8e67a427d5a298858b68305fa58f72e164db262dea8 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-aclamk-cbt-combined-d58f730, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-09-13T15:02:17.919 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 15:02:17 smithi102 bash[421746]: ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3 2024-09-13T15:02:17.998 INFO:journalctl@ceph.osd.4.smithi195.stdout:Sep 13 15:02:17 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-4[38505]: 2024-09-13T15:02:17.587+0000 7fb6840a8640 -1 osd.4 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:50.052522+0000 front 2024-09-13T15:01:50.052460+0000 (oldest deadline 2024-09-13T15:02:14.752479+0000) 2024-09-13T15:02:17.998 INFO:journalctl@ceph.osd.5.smithi195.stdout:Sep 13 15:02:17 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-5[44425]: 2024-09-13T15:02:17.591+0000 7fa743d86640 -1 osd.5 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.927574+0000 front 2024-09-13T15:01:49.927636+0000 (oldest deadline 2024-09-13T15:02:15.827343+0000) 2024-09-13T15:02:18.364 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 15:02:18 smithi102 podman[421768]: 2024-09-13 15:02:18.020897446 +0000 UTC m=+0.761001877 container remove cb2596701036e089f0c6d8e67a427d5a298858b68305fa58f72e164db262dea8 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-13T15:02:18.556 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:18 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:18.488+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.963343+0000 front 2024-09-13T15:01:49.963380+0000 (oldest deadline 2024-09-13T15:02:14.663480+0000) 2024-09-13T15:02:18.556 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:18 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:18.259+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:50.534049+0000 front 2024-09-13T15:01:50.534014+0000 (oldest deadline 2024-09-13T15:02:14.634064+0000) 2024-09-13T15:02:18.641 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 15:02:18 smithi102 podman[421860]: 2024-09-13 15:02:18.363354944 +0000 UTC m=+0.109886746 container create ca06dc15c4c0bbd4aed81c4b918280b18f850ca9abecd2a3dced7e3b02fb0991 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T15:02:18.641 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 15:02:18 smithi102 podman[421860]: 2024-09-13 15:02:18.2696116 +0000 UTC m=+0.016143360 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T15:02:18.642 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 15:02:18 smithi102 podman[421860]: 2024-09-13 15:02:18.55716754 +0000 UTC m=+0.303699300 container init ca06dc15c4c0bbd4aed81c4b918280b18f850ca9abecd2a3dced7e3b02fb0991 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3-deactivate, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T15:02:18.642 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 15:02:18 smithi102 podman[421860]: 2024-09-13 15:02:18.562239717 +0000 UTC m=+0.308771480 container start ca06dc15c4c0bbd4aed81c4b918280b18f850ca9abecd2a3dced7e3b02fb0991 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , ceph=True) 2024-09-13T15:02:18.642 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 15:02:18 smithi102 podman[421860]: 2024-09-13 15:02:18.640463524 +0000 UTC m=+0.386995311 container attach ca06dc15c4c0bbd4aed81c4b918280b18f850ca9abecd2a3dced7e3b02fb0991 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T15:02:18.919 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 15:02:18 smithi102 podman[421860]: 2024-09-13 15:02:18.751184574 +0000 UTC m=+0.497716334 container died ca06dc15c4c0bbd4aed81c4b918280b18f850ca9abecd2a3dced7e3b02fb0991 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T15:02:18.997 INFO:journalctl@ceph.osd.4.smithi195.stdout:Sep 13 15:02:18 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-4[38505]: 2024-09-13T15:02:18.554+0000 7fb6840a8640 -1 osd.4 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:50.052522+0000 front 2024-09-13T15:01:50.052460+0000 (oldest deadline 2024-09-13T15:02:14.752479+0000) 2024-09-13T15:02:18.998 INFO:journalctl@ceph.osd.5.smithi195.stdout:Sep 13 15:02:18 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-5[44425]: 2024-09-13T15:02:18.622+0000 7fa743d86640 -1 osd.5 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.927574+0000 front 2024-09-13T15:01:49.927636+0000 (oldest deadline 2024-09-13T15:02:15.827343+0000) 2024-09-13T15:02:19.287 INFO:journalctl@ceph.osd.3.smithi102.stdout:Sep 13 15:02:19 smithi102 podman[421903]: 2024-09-13 15:02:19.268998005 +0000 UTC m=+0.511872200 container remove ca06dc15c4c0bbd4aed81c4b918280b18f850ca9abecd2a3dced7e3b02fb0991 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-13T15:02:19.292 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-08cee502-71d9-11ef-bceb-c7b262605968@osd.3.service' 2024-09-13T15:02:19.330 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:02:19.330 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-13T15:02:19.331 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-13T15:02:19.331 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ceph-08cee502-71d9-11ef-bceb-c7b262605968@osd.4 2024-09-13T15:02:19.571 INFO:journalctl@ceph.osd.4.smithi195.stdout:Sep 13 15:02:19 smithi195 systemd[1]: Stopping Ceph osd.4 for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T15:02:19.572 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:19 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:19.439+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.963343+0000 front 2024-09-13T15:01:49.963380+0000 (oldest deadline 2024-09-13T15:02:14.663480+0000) 2024-09-13T15:02:19.572 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:19 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:19.261+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:50.534049+0000 front 2024-09-13T15:01:50.534014+0000 (oldest deadline 2024-09-13T15:02:14.634064+0000) 2024-09-13T15:02:19.997 INFO:journalctl@ceph.osd.5.smithi195.stdout:Sep 13 15:02:19 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-5[44425]: 2024-09-13T15:02:19.670+0000 7fa743d86640 -1 osd.5 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.927574+0000 front 2024-09-13T15:01:49.927636+0000 (oldest deadline 2024-09-13T15:02:15.827343+0000) 2024-09-13T15:02:19.998 INFO:journalctl@ceph.osd.4.smithi195.stdout:Sep 13 15:02:19 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-4[38505]: 2024-09-13T15:02:19.569+0000 7fb6840a8640 -1 osd.4 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:50.052522+0000 front 2024-09-13T15:01:50.052460+0000 (oldest deadline 2024-09-13T15:02:14.752479+0000) 2024-09-13T15:02:19.998 INFO:journalctl@ceph.osd.4.smithi195.stdout:Sep 13 15:02:19 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-4[38505]: 2024-09-13T15:02:19.651+0000 7fb687a94640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:02:19.998 INFO:journalctl@ceph.osd.4.smithi195.stdout:Sep 13 15:02:19 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-4[38505]: 2024-09-13T15:02:19.651+0000 7fb687a94640 -1 osd.4 517 *** Got signal Terminated *** 2024-09-13T15:02:19.998 INFO:journalctl@ceph.osd.4.smithi195.stdout:Sep 13 15:02:19 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-4[38505]: 2024-09-13T15:02:19.651+0000 7fb687a94640 -1 osd.4 517 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T15:02:20.585 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:20 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:20.425+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.963343+0000 front 2024-09-13T15:01:49.963380+0000 (oldest deadline 2024-09-13T15:02:14.663480+0000) 2024-09-13T15:02:20.585 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:20 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:20.257+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:50.534049+0000 front 2024-09-13T15:01:50.534014+0000 (oldest deadline 2024-09-13T15:02:14.634064+0000) 2024-09-13T15:02:20.997 INFO:journalctl@ceph.osd.5.smithi195.stdout:Sep 13 15:02:20 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-5[44425]: 2024-09-13T15:02:20.688+0000 7fa743d86640 -1 osd.5 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.927574+0000 front 2024-09-13T15:01:49.927636+0000 (oldest deadline 2024-09-13T15:02:15.827343+0000) 2024-09-13T15:02:20.998 INFO:journalctl@ceph.osd.4.smithi195.stdout:Sep 13 15:02:20 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-4[38505]: 2024-09-13T15:02:20.583+0000 7fb6840a8640 -1 osd.4 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:50.052522+0000 front 2024-09-13T15:01:50.052460+0000 (oldest deadline 2024-09-13T15:02:14.752479+0000) 2024-09-13T15:02:21.610 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:21 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:21.300+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:50.534049+0000 front 2024-09-13T15:01:50.534014+0000 (oldest deadline 2024-09-13T15:02:14.634064+0000) 2024-09-13T15:02:21.610 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:21 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:21.439+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.963343+0000 front 2024-09-13T15:01:49.963380+0000 (oldest deadline 2024-09-13T15:02:14.663480+0000) 2024-09-13T15:02:21.998 INFO:journalctl@ceph.osd.4.smithi195.stdout:Sep 13 15:02:21 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-4[38505]: 2024-09-13T15:02:21.608+0000 7fb6840a8640 -1 osd.4 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:50.052522+0000 front 2024-09-13T15:01:50.052460+0000 (oldest deadline 2024-09-13T15:02:14.752479+0000) 2024-09-13T15:02:21.998 INFO:journalctl@ceph.osd.5.smithi195.stdout:Sep 13 15:02:21 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-5[44425]: 2024-09-13T15:02:21.650+0000 7fa743d86640 -1 osd.5 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.927574+0000 front 2024-09-13T15:01:49.927636+0000 (oldest deadline 2024-09-13T15:02:15.827343+0000) 2024-09-13T15:02:21.998 INFO:journalctl@ceph.osd.5.smithi195.stdout:Sep 13 15:02:21 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-5[44425]: 2024-09-13T15:02:21.650+0000 7fa743d86640 -1 osd.5 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:01:55.827811+0000 front 2024-09-13T15:01:55.827799+0000 (oldest deadline 2024-09-13T15:02:21.127718+0000) 2024-09-13T15:02:22.600 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:22 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:22.344+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:50.534049+0000 front 2024-09-13T15:01:50.534014+0000 (oldest deadline 2024-09-13T15:02:14.634064+0000) 2024-09-13T15:02:22.600 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:22 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:22.344+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:02:00.435721+0000 front 2024-09-13T15:02:00.435708+0000 (oldest deadline 2024-09-13T15:02:21.535538+0000) 2024-09-13T15:02:22.600 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:22 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:22.477+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.963343+0000 front 2024-09-13T15:01:49.963380+0000 (oldest deadline 2024-09-13T15:02:14.663480+0000) 2024-09-13T15:02:22.601 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:22 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:22.477+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:01:56.964520+0000 front 2024-09-13T15:01:56.964531+0000 (oldest deadline 2024-09-13T15:02:21.664327+0000) 2024-09-13T15:02:22.997 INFO:journalctl@ceph.osd.4.smithi195.stdout:Sep 13 15:02:22 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-4[38505]: 2024-09-13T15:02:22.598+0000 7fb6840a8640 -1 osd.4 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:50.052522+0000 front 2024-09-13T15:01:50.052460+0000 (oldest deadline 2024-09-13T15:02:14.752479+0000) 2024-09-13T15:02:22.998 INFO:journalctl@ceph.osd.5.smithi195.stdout:Sep 13 15:02:22 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-5[44425]: 2024-09-13T15:02:22.662+0000 7fa743d86640 -1 osd.5 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.927574+0000 front 2024-09-13T15:01:49.927636+0000 (oldest deadline 2024-09-13T15:02:15.827343+0000) 2024-09-13T15:02:22.998 INFO:journalctl@ceph.osd.5.smithi195.stdout:Sep 13 15:02:22 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-5[44425]: 2024-09-13T15:02:22.662+0000 7fa743d86640 -1 osd.5 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:01:55.827811+0000 front 2024-09-13T15:01:55.827799+0000 (oldest deadline 2024-09-13T15:02:21.127718+0000) 2024-09-13T15:02:23.625 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:23 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:23.469+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.963343+0000 front 2024-09-13T15:01:49.963380+0000 (oldest deadline 2024-09-13T15:02:14.663480+0000) 2024-09-13T15:02:23.625 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:23 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:23.469+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:01:56.964520+0000 front 2024-09-13T15:01:56.964531+0000 (oldest deadline 2024-09-13T15:02:21.664327+0000) 2024-09-13T15:02:23.625 INFO:journalctl@ceph.osd.5.smithi195.stdout:Sep 13 15:02:23 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-5[44425]: 2024-09-13T15:02:23.623+0000 7fa743d86640 -1 osd.5 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.927574+0000 front 2024-09-13T15:01:49.927636+0000 (oldest deadline 2024-09-13T15:02:15.827343+0000) 2024-09-13T15:02:23.625 INFO:journalctl@ceph.osd.5.smithi195.stdout:Sep 13 15:02:23 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-5[44425]: 2024-09-13T15:02:23.623+0000 7fa743d86640 -1 osd.5 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:01:55.827811+0000 front 2024-09-13T15:01:55.827799+0000 (oldest deadline 2024-09-13T15:02:21.127718+0000) 2024-09-13T15:02:23.626 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:23 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:23.318+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:50.534049+0000 front 2024-09-13T15:01:50.534014+0000 (oldest deadline 2024-09-13T15:02:14.634064+0000) 2024-09-13T15:02:23.626 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:23 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:23.318+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:02:00.435721+0000 front 2024-09-13T15:02:00.435708+0000 (oldest deadline 2024-09-13T15:02:21.535538+0000) 2024-09-13T15:02:24.001 INFO:journalctl@ceph.osd.4.smithi195.stdout:Sep 13 15:02:23 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-4[38505]: 2024-09-13T15:02:23.633+0000 7fb6840a8640 -1 osd.4 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:50.052522+0000 front 2024-09-13T15:01:50.052460+0000 (oldest deadline 2024-09-13T15:02:14.752479+0000) 2024-09-13T15:02:24.001 INFO:journalctl@ceph.osd.4.smithi195.stdout:Sep 13 15:02:23 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-4[38505]: 2024-09-13T15:02:23.633+0000 7fb6840a8640 -1 osd.4 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:01:58.253518+0000 front 2024-09-13T15:01:58.253448+0000 (oldest deadline 2024-09-13T15:02:22.953432+0000) 2024-09-13T15:02:24.637 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:24 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:24.447+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.963343+0000 front 2024-09-13T15:01:49.963380+0000 (oldest deadline 2024-09-13T15:02:14.663480+0000) 2024-09-13T15:02:24.637 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:24 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:24.447+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:01:56.964520+0000 front 2024-09-13T15:01:56.964531+0000 (oldest deadline 2024-09-13T15:02:21.664327+0000) 2024-09-13T15:02:24.637 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:24 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:24.332+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:50.534049+0000 front 2024-09-13T15:01:50.534014+0000 (oldest deadline 2024-09-13T15:02:14.634064+0000) 2024-09-13T15:02:24.638 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:24 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:24.332+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:02:00.435721+0000 front 2024-09-13T15:02:00.435708+0000 (oldest deadline 2024-09-13T15:02:21.535538+0000) 2024-09-13T15:02:24.998 INFO:journalctl@ceph.osd.4.smithi195.stdout:Sep 13 15:02:24 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-4[38505]: 2024-09-13T15:02:24.635+0000 7fb6840a8640 -1 osd.4 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:50.052522+0000 front 2024-09-13T15:01:50.052460+0000 (oldest deadline 2024-09-13T15:02:14.752479+0000) 2024-09-13T15:02:24.998 INFO:journalctl@ceph.osd.4.smithi195.stdout:Sep 13 15:02:24 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-4[38505]: 2024-09-13T15:02:24.635+0000 7fb6840a8640 -1 osd.4 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:01:58.253518+0000 front 2024-09-13T15:01:58.253448+0000 (oldest deadline 2024-09-13T15:02:22.953432+0000) 2024-09-13T15:02:24.998 INFO:journalctl@ceph.osd.5.smithi195.stdout:Sep 13 15:02:24 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-5[44425]: 2024-09-13T15:02:24.665+0000 7fa743d86640 -1 osd.5 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.927574+0000 front 2024-09-13T15:01:49.927636+0000 (oldest deadline 2024-09-13T15:02:15.827343+0000) 2024-09-13T15:02:24.998 INFO:journalctl@ceph.osd.5.smithi195.stdout:Sep 13 15:02:24 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-5[44425]: 2024-09-13T15:02:24.665+0000 7fa743d86640 -1 osd.5 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:01:55.827811+0000 front 2024-09-13T15:01:55.827799+0000 (oldest deadline 2024-09-13T15:02:21.127718+0000) 2024-09-13T15:02:25.625 INFO:journalctl@ceph.osd.4.smithi195.stdout:Sep 13 15:02:25 smithi195 podman[286171]: 2024-09-13 15:02:25.290376851 +0000 UTC m=+5.765777827 container died 515307701d73af562e32297b8bd73de8eab76c99d3d2d178a9317a8024bfe66e (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-4, io.buildah.version=1.37.2, RELEASE=reef-2c01458, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True) 2024-09-13T15:02:25.626 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:25 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:25.468+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.963343+0000 front 2024-09-13T15:01:49.963380+0000 (oldest deadline 2024-09-13T15:02:14.663480+0000) 2024-09-13T15:02:25.626 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:25 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:25.468+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:01:56.964520+0000 front 2024-09-13T15:01:56.964531+0000 (oldest deadline 2024-09-13T15:02:21.664327+0000) 2024-09-13T15:02:25.626 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:25 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:25.325+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:50.534049+0000 front 2024-09-13T15:01:50.534014+0000 (oldest deadline 2024-09-13T15:02:14.634064+0000) 2024-09-13T15:02:25.627 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:25 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:25.325+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:02:00.435721+0000 front 2024-09-13T15:02:00.435708+0000 (oldest deadline 2024-09-13T15:02:21.535538+0000) 2024-09-13T15:02:25.998 INFO:journalctl@ceph.osd.4.smithi195.stdout:Sep 13 15:02:25 smithi195 podman[286171]: 2024-09-13 15:02:25.82606723 +0000 UTC m=+6.301468213 container cleanup 515307701d73af562e32297b8bd73de8eab76c99d3d2d178a9317a8024bfe66e (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-4, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=reef-2c01458, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-09-13T15:02:25.998 INFO:journalctl@ceph.osd.4.smithi195.stdout:Sep 13 15:02:25 smithi195 bash[286171]: ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-4 2024-09-13T15:02:25.999 INFO:journalctl@ceph.osd.5.smithi195.stdout:Sep 13 15:02:25 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-5[44425]: 2024-09-13T15:02:25.657+0000 7fa743d86640 -1 osd.5 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.927574+0000 front 2024-09-13T15:01:49.927636+0000 (oldest deadline 2024-09-13T15:02:15.827343+0000) 2024-09-13T15:02:25.999 INFO:journalctl@ceph.osd.5.smithi195.stdout:Sep 13 15:02:25 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-5[44425]: 2024-09-13T15:02:25.657+0000 7fa743d86640 -1 osd.5 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:01:55.827811+0000 front 2024-09-13T15:01:55.827799+0000 (oldest deadline 2024-09-13T15:02:21.127718+0000) 2024-09-13T15:02:26.429 INFO:journalctl@ceph.osd.4.smithi195.stdout:Sep 13 15:02:26 smithi195 podman[286191]: 2024-09-13 15:02:26.06109572 +0000 UTC m=+0.755826415 container remove 515307701d73af562e32297b8bd73de8eab76c99d3d2d178a9317a8024bfe66e (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-4, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=reef-2c01458, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.build-date=20240731) 2024-09-13T15:02:26.429 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:26 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:26.287+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:50.534049+0000 front 2024-09-13T15:01:50.534014+0000 (oldest deadline 2024-09-13T15:02:14.634064+0000) 2024-09-13T15:02:26.429 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:26 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:26.287+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:02:00.435721+0000 front 2024-09-13T15:02:00.435708+0000 (oldest deadline 2024-09-13T15:02:21.535538+0000) 2024-09-13T15:02:26.747 INFO:journalctl@ceph.osd.5.smithi195.stdout:Sep 13 15:02:26 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-5[44425]: 2024-09-13T15:02:26.616+0000 7fa743d86640 -1 osd.5 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.927574+0000 front 2024-09-13T15:01:49.927636+0000 (oldest deadline 2024-09-13T15:02:15.827343+0000) 2024-09-13T15:02:26.748 INFO:journalctl@ceph.osd.5.smithi195.stdout:Sep 13 15:02:26 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-5[44425]: 2024-09-13T15:02:26.616+0000 7fa743d86640 -1 osd.5 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:01:55.827811+0000 front 2024-09-13T15:01:55.827799+0000 (oldest deadline 2024-09-13T15:02:21.127718+0000) 2024-09-13T15:02:26.748 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:26 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:26.427+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.963343+0000 front 2024-09-13T15:01:49.963380+0000 (oldest deadline 2024-09-13T15:02:14.663480+0000) 2024-09-13T15:02:26.748 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:26 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:26.427+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:01:56.964520+0000 front 2024-09-13T15:01:56.964531+0000 (oldest deadline 2024-09-13T15:02:21.664327+0000) 2024-09-13T15:02:26.749 INFO:journalctl@ceph.osd.4.smithi195.stdout:Sep 13 15:02:26 smithi195 podman[286284]: 2024-09-13 15:02:26.388284165 +0000 UTC m=+0.016280301 image pull c129f5a2fd4eccce3e6eec43830eff1929af98bc48f3b850cf144ce5b326e8b5 quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11 2024-09-13T15:02:26.749 INFO:journalctl@ceph.osd.4.smithi195.stdout:Sep 13 15:02:26 smithi195 podman[286284]: 2024-09-13 15:02:26.537363531 +0000 UTC m=+0.165359709 container create 2867eccd518ace0cd5c877c21bc46118c5e1a28b4d3c749371752072d6d088d6 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=reef-2c01458, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2) 2024-09-13T15:02:27.133 INFO:journalctl@ceph.osd.4.smithi195.stdout:Sep 13 15:02:26 smithi195 podman[286284]: 2024-09-13 15:02:26.781664723 +0000 UTC m=+0.409660853 container init 2867eccd518ace0cd5c877c21bc46118c5e1a28b4d3c749371752072d6d088d6 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-4-deactivate, RELEASE=reef-2c01458, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-13T15:02:27.133 INFO:journalctl@ceph.osd.4.smithi195.stdout:Sep 13 15:02:26 smithi195 podman[286284]: 2024-09-13 15:02:26.786667915 +0000 UTC m=+0.414664044 container start 2867eccd518ace0cd5c877c21bc46118c5e1a28b4d3c749371752072d6d088d6 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, RELEASE=reef-2c01458, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-09-13T15:02:27.133 INFO:journalctl@ceph.osd.4.smithi195.stdout:Sep 13 15:02:26 smithi195 podman[286284]: 2024-09-13 15:02:26.898293075 +0000 UTC m=+0.526289217 container attach 2867eccd518ace0cd5c877c21bc46118c5e1a28b4d3c749371752072d6d088d6 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-4-deactivate, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=reef-2c01458, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-09-13T15:02:27.419 INFO:journalctl@ceph.osd.4.smithi195.stdout:Sep 13 15:02:27 smithi195 podman[286284]: 2024-09-13 15:02:27.132648541 +0000 UTC m=+0.760644679 container died 2867eccd518ace0cd5c877c21bc46118c5e1a28b4d3c749371752072d6d088d6 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=reef-2c01458, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T15:02:27.420 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:27 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:27.287+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:50.534049+0000 front 2024-09-13T15:01:50.534014+0000 (oldest deadline 2024-09-13T15:02:14.634064+0000) 2024-09-13T15:02:27.420 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:27 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:27.287+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:02:00.435721+0000 front 2024-09-13T15:02:00.435708+0000 (oldest deadline 2024-09-13T15:02:21.535538+0000) 2024-09-13T15:02:27.686 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:27 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:27.476+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.963343+0000 front 2024-09-13T15:01:49.963380+0000 (oldest deadline 2024-09-13T15:02:14.663480+0000) 2024-09-13T15:02:27.686 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:27 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:27.476+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:01:56.964520+0000 front 2024-09-13T15:01:56.964531+0000 (oldest deadline 2024-09-13T15:02:21.664327+0000) 2024-09-13T15:02:27.687 INFO:journalctl@ceph.osd.5.smithi195.stdout:Sep 13 15:02:27 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-5[44425]: 2024-09-13T15:02:27.581+0000 7fa743d86640 -1 osd.5 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.927574+0000 front 2024-09-13T15:01:49.927636+0000 (oldest deadline 2024-09-13T15:02:15.827343+0000) 2024-09-13T15:02:27.687 INFO:journalctl@ceph.osd.5.smithi195.stdout:Sep 13 15:02:27 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-5[44425]: 2024-09-13T15:02:27.581+0000 7fa743d86640 -1 osd.5 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:01:55.827811+0000 front 2024-09-13T15:01:55.827799+0000 (oldest deadline 2024-09-13T15:02:21.127718+0000) 2024-09-13T15:02:27.706 DEBUG:teuthology.orchestra.run.smithi195:> sudo pkill -f 'journalctl -f -n 0 -u ceph-08cee502-71d9-11ef-bceb-c7b262605968@osd.4.service' 2024-09-13T15:02:27.749 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:02:27.749 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-13T15:02:27.749 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-13T15:02:27.750 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ceph-08cee502-71d9-11ef-bceb-c7b262605968@osd.5 2024-09-13T15:02:27.997 INFO:journalctl@ceph.osd.5.smithi195.stdout:Sep 13 15:02:27 smithi195 systemd[1]: Stopping Ceph osd.5 for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T15:02:28.478 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:28.243+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:50.534049+0000 front 2024-09-13T15:01:50.534014+0000 (oldest deadline 2024-09-13T15:02:14.634064+0000) 2024-09-13T15:02:28.479 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:28.243+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:02:00.435721+0000 front 2024-09-13T15:02:00.435708+0000 (oldest deadline 2024-09-13T15:02:21.535538+0000) 2024-09-13T15:02:28.479 INFO:journalctl@ceph.osd.5.smithi195.stdout:Sep 13 15:02:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-5[44425]: 2024-09-13T15:02:28.107+0000 7fa74b77a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:02:28.479 INFO:journalctl@ceph.osd.5.smithi195.stdout:Sep 13 15:02:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-5[44425]: 2024-09-13T15:02:28.107+0000 7fa74b77a640 -1 osd.5 517 *** Got signal Terminated *** 2024-09-13T15:02:28.479 INFO:journalctl@ceph.osd.5.smithi195.stdout:Sep 13 15:02:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-5[44425]: 2024-09-13T15:02:28.107+0000 7fa74b77a640 -1 osd.5 517 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T15:02:28.747 INFO:journalctl@ceph.osd.5.smithi195.stdout:Sep 13 15:02:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-5[44425]: 2024-09-13T15:02:28.576+0000 7fa743d86640 -1 osd.5 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.927574+0000 front 2024-09-13T15:01:49.927636+0000 (oldest deadline 2024-09-13T15:02:15.827343+0000) 2024-09-13T15:02:28.748 INFO:journalctl@ceph.osd.5.smithi195.stdout:Sep 13 15:02:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-5[44425]: 2024-09-13T15:02:28.576+0000 7fa743d86640 -1 osd.5 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:01:55.827811+0000 front 2024-09-13T15:01:55.827799+0000 (oldest deadline 2024-09-13T15:02:21.127718+0000) 2024-09-13T15:02:28.748 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:28.477+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.963343+0000 front 2024-09-13T15:01:49.963380+0000 (oldest deadline 2024-09-13T15:02:14.663480+0000) 2024-09-13T15:02:28.748 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:28 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:28.477+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:01:56.964520+0000 front 2024-09-13T15:01:56.964531+0000 (oldest deadline 2024-09-13T15:02:21.664327+0000) 2024-09-13T15:02:29.498 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:29 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:29.435+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.963343+0000 front 2024-09-13T15:01:49.963380+0000 (oldest deadline 2024-09-13T15:02:14.663480+0000) 2024-09-13T15:02:29.498 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:29 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:29.435+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:01:56.964520+0000 front 2024-09-13T15:01:56.964531+0000 (oldest deadline 2024-09-13T15:02:21.664327+0000) 2024-09-13T15:02:29.498 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:29 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:29.230+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:50.534049+0000 front 2024-09-13T15:01:50.534014+0000 (oldest deadline 2024-09-13T15:02:14.634064+0000) 2024-09-13T15:02:29.498 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:29 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:29.230+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:02:00.435721+0000 front 2024-09-13T15:02:00.435708+0000 (oldest deadline 2024-09-13T15:02:21.535538+0000) 2024-09-13T15:02:29.997 INFO:journalctl@ceph.osd.5.smithi195.stdout:Sep 13 15:02:29 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-5[44425]: 2024-09-13T15:02:29.539+0000 7fa743d86640 -1 osd.5 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.927574+0000 front 2024-09-13T15:01:49.927636+0000 (oldest deadline 2024-09-13T15:02:15.827343+0000) 2024-09-13T15:02:29.998 INFO:journalctl@ceph.osd.5.smithi195.stdout:Sep 13 15:02:29 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-5[44425]: 2024-09-13T15:02:29.539+0000 7fa743d86640 -1 osd.5 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:01:55.827811+0000 front 2024-09-13T15:01:55.827799+0000 (oldest deadline 2024-09-13T15:02:21.127718+0000) 2024-09-13T15:02:30.497 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:30 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:30.422+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.963343+0000 front 2024-09-13T15:01:49.963380+0000 (oldest deadline 2024-09-13T15:02:14.663480+0000) 2024-09-13T15:02:30.498 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:30 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:30.422+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:01:56.964520+0000 front 2024-09-13T15:01:56.964531+0000 (oldest deadline 2024-09-13T15:02:21.664327+0000) 2024-09-13T15:02:30.498 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:30 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:30.216+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:50.534049+0000 front 2024-09-13T15:01:50.534014+0000 (oldest deadline 2024-09-13T15:02:14.634064+0000) 2024-09-13T15:02:30.498 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:30 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:30.216+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:02:00.435721+0000 front 2024-09-13T15:02:00.435708+0000 (oldest deadline 2024-09-13T15:02:21.535538+0000) 2024-09-13T15:02:30.498 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:30 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:30.216+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-09-13T15:02:03.836587+0000 front 2024-09-13T15:02:03.836669+0000 (oldest deadline 2024-09-13T15:02:29.736636+0000) 2024-09-13T15:02:30.997 INFO:journalctl@ceph.osd.5.smithi195.stdout:Sep 13 15:02:30 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-5[44425]: 2024-09-13T15:02:30.566+0000 7fa743d86640 -1 osd.5 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.927574+0000 front 2024-09-13T15:01:49.927636+0000 (oldest deadline 2024-09-13T15:02:15.827343+0000) 2024-09-13T15:02:30.998 INFO:journalctl@ceph.osd.5.smithi195.stdout:Sep 13 15:02:30 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-5[44425]: 2024-09-13T15:02:30.566+0000 7fa743d86640 -1 osd.5 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:01:55.827811+0000 front 2024-09-13T15:01:55.827799+0000 (oldest deadline 2024-09-13T15:02:21.127718+0000) 2024-09-13T15:02:31.497 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:31.461+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.963343+0000 front 2024-09-13T15:01:49.963380+0000 (oldest deadline 2024-09-13T15:02:14.663480+0000) 2024-09-13T15:02:31.498 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:31.461+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:01:56.964520+0000 front 2024-09-13T15:01:56.964531+0000 (oldest deadline 2024-09-13T15:02:21.664327+0000) 2024-09-13T15:02:31.498 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:31.213+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:50.534049+0000 front 2024-09-13T15:01:50.534014+0000 (oldest deadline 2024-09-13T15:02:14.634064+0000) 2024-09-13T15:02:31.498 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:31.213+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:02:00.435721+0000 front 2024-09-13T15:02:00.435708+0000 (oldest deadline 2024-09-13T15:02:21.535538+0000) 2024-09-13T15:02:31.498 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:31.213+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-09-13T15:02:03.836587+0000 front 2024-09-13T15:02:03.836669+0000 (oldest deadline 2024-09-13T15:02:29.736636+0000) 2024-09-13T15:02:31.998 INFO:journalctl@ceph.osd.5.smithi195.stdout:Sep 13 15:02:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-5[44425]: 2024-09-13T15:02:31.599+0000 7fa743d86640 -1 osd.5 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.927574+0000 front 2024-09-13T15:01:49.927636+0000 (oldest deadline 2024-09-13T15:02:15.827343+0000) 2024-09-13T15:02:31.998 INFO:journalctl@ceph.osd.5.smithi195.stdout:Sep 13 15:02:31 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-5[44425]: 2024-09-13T15:02:31.599+0000 7fa743d86640 -1 osd.5 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:01:55.827811+0000 front 2024-09-13T15:01:55.827799+0000 (oldest deadline 2024-09-13T15:02:21.127718+0000) 2024-09-13T15:02:32.556 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:32.438+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.963343+0000 front 2024-09-13T15:01:49.963380+0000 (oldest deadline 2024-09-13T15:02:14.663480+0000) 2024-09-13T15:02:32.556 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:32.438+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:01:56.964520+0000 front 2024-09-13T15:01:56.964531+0000 (oldest deadline 2024-09-13T15:02:21.664327+0000) 2024-09-13T15:02:32.556 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:32.438+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-09-13T15:02:08.065814+0000 front 2024-09-13T15:02:08.065746+0000 (oldest deadline 2024-09-13T15:02:31.565719+0000) 2024-09-13T15:02:32.557 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:32.253+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:50.534049+0000 front 2024-09-13T15:01:50.534014+0000 (oldest deadline 2024-09-13T15:02:14.634064+0000) 2024-09-13T15:02:32.557 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:32.253+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:02:00.435721+0000 front 2024-09-13T15:02:00.435708+0000 (oldest deadline 2024-09-13T15:02:21.535538+0000) 2024-09-13T15:02:32.557 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:32.253+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-09-13T15:02:03.836587+0000 front 2024-09-13T15:02:03.836669+0000 (oldest deadline 2024-09-13T15:02:29.736636+0000) 2024-09-13T15:02:32.997 INFO:journalctl@ceph.osd.5.smithi195.stdout:Sep 13 15:02:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-5[44425]: 2024-09-13T15:02:32.554+0000 7fa743d86640 -1 osd.5 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.927574+0000 front 2024-09-13T15:01:49.927636+0000 (oldest deadline 2024-09-13T15:02:15.827343+0000) 2024-09-13T15:02:32.998 INFO:journalctl@ceph.osd.5.smithi195.stdout:Sep 13 15:02:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-5[44425]: 2024-09-13T15:02:32.554+0000 7fa743d86640 -1 osd.5 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:01:55.827811+0000 front 2024-09-13T15:01:55.827799+0000 (oldest deadline 2024-09-13T15:02:21.127718+0000) 2024-09-13T15:02:32.998 INFO:journalctl@ceph.osd.5.smithi195.stdout:Sep 13 15:02:32 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-5[44425]: 2024-09-13T15:02:32.554+0000 7fa743d86640 -1 osd.5 517 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-09-13T15:02:08.029767+0000 front 2024-09-13T15:02:08.029751+0000 (oldest deadline 2024-09-13T15:02:32.129614+0000) 2024-09-13T15:02:33.497 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:33 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:33.223+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:50.534049+0000 front 2024-09-13T15:01:50.534014+0000 (oldest deadline 2024-09-13T15:02:14.634064+0000) 2024-09-13T15:02:33.498 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:33 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:33.223+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:02:00.435721+0000 front 2024-09-13T15:02:00.435708+0000 (oldest deadline 2024-09-13T15:02:21.535538+0000) 2024-09-13T15:02:33.498 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:33 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:33.223+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-09-13T15:02:03.836587+0000 front 2024-09-13T15:02:03.836669+0000 (oldest deadline 2024-09-13T15:02:29.736636+0000) 2024-09-13T15:02:33.498 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:33 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:33.434+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.963343+0000 front 2024-09-13T15:01:49.963380+0000 (oldest deadline 2024-09-13T15:02:14.663480+0000) 2024-09-13T15:02:33.498 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:33 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:33.434+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:01:56.964520+0000 front 2024-09-13T15:01:56.964531+0000 (oldest deadline 2024-09-13T15:02:21.664327+0000) 2024-09-13T15:02:33.499 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:33 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:33.434+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-09-13T15:02:08.065814+0000 front 2024-09-13T15:02:08.065746+0000 (oldest deadline 2024-09-13T15:02:31.565719+0000) 2024-09-13T15:02:33.832 INFO:journalctl@ceph.osd.5.smithi195.stdout:Sep 13 15:02:33 smithi195 podman[286411]: 2024-09-13 15:02:33.560849778 +0000 UTC m=+5.585673984 container died 176ba2b3bd5984a0d1ab246714df2b0b0bbabc5717dbd30aea9df2ebe0928c93 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-5, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=reef-2c01458, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T15:02:34.190 INFO:journalctl@ceph.osd.5.smithi195.stdout:Sep 13 15:02:33 smithi195 podman[286411]: 2024-09-13 15:02:33.938388703 +0000 UTC m=+5.963212944 container cleanup 176ba2b3bd5984a0d1ab246714df2b0b0bbabc5717dbd30aea9df2ebe0928c93 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-5, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, RELEASE=reef-2c01458, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-09-13T15:02:34.190 INFO:journalctl@ceph.osd.5.smithi195.stdout:Sep 13 15:02:33 smithi195 bash[286411]: ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-5 2024-09-13T15:02:34.446 INFO:journalctl@ceph.osd.5.smithi195.stdout:Sep 13 15:02:34 smithi195 podman[286430]: 2024-09-13 15:02:34.189570957 +0000 UTC m=+0.627191023 container remove 176ba2b3bd5984a0d1ab246714df2b0b0bbabc5717dbd30aea9df2ebe0928c93 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-5, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=reef-2c01458, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T15:02:34.446 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:34 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:34.253+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:50.534049+0000 front 2024-09-13T15:01:50.534014+0000 (oldest deadline 2024-09-13T15:02:14.634064+0000) 2024-09-13T15:02:34.446 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:34 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:34.253+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:02:00.435721+0000 front 2024-09-13T15:02:00.435708+0000 (oldest deadline 2024-09-13T15:02:21.535538+0000) 2024-09-13T15:02:34.447 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:34 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:34.253+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-09-13T15:02:03.836587+0000 front 2024-09-13T15:02:03.836669+0000 (oldest deadline 2024-09-13T15:02:29.736636+0000) 2024-09-13T15:02:34.748 INFO:journalctl@ceph.osd.5.smithi195.stdout:Sep 13 15:02:34 smithi195 podman[286526]: 2024-09-13 15:02:34.452585432 +0000 UTC m=+0.016222701 image pull c129f5a2fd4eccce3e6eec43830eff1929af98bc48f3b850cf144ce5b326e8b5 quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11 2024-09-13T15:02:34.748 INFO:journalctl@ceph.osd.5.smithi195.stdout:Sep 13 15:02:34 smithi195 podman[286526]: 2024-09-13 15:02:34.565620413 +0000 UTC m=+0.129257677 container create e614bb803073255793302c2a67de85a84e76deb322331e4192e2d285996f4957 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, RELEASE=reef-2c01458, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-13T15:02:34.748 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:34 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:34.444+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.963343+0000 front 2024-09-13T15:01:49.963380+0000 (oldest deadline 2024-09-13T15:02:14.663480+0000) 2024-09-13T15:02:34.748 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:34 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:34.444+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:01:56.964520+0000 front 2024-09-13T15:01:56.964531+0000 (oldest deadline 2024-09-13T15:02:21.664327+0000) 2024-09-13T15:02:34.748 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:34 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:34.444+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-09-13T15:02:08.065814+0000 front 2024-09-13T15:02:08.065746+0000 (oldest deadline 2024-09-13T15:02:31.565719+0000) 2024-09-13T15:02:35.215 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:35 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:35.213+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:50.534049+0000 front 2024-09-13T15:01:50.534014+0000 (oldest deadline 2024-09-13T15:02:14.634064+0000) 2024-09-13T15:02:35.215 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:35 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:35.213+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:02:00.435721+0000 front 2024-09-13T15:02:00.435708+0000 (oldest deadline 2024-09-13T15:02:21.535538+0000) 2024-09-13T15:02:35.216 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:35 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:35.213+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-09-13T15:02:03.836587+0000 front 2024-09-13T15:02:03.836669+0000 (oldest deadline 2024-09-13T15:02:29.736636+0000) 2024-09-13T15:02:35.216 INFO:journalctl@ceph.osd.5.smithi195.stdout:Sep 13 15:02:34 smithi195 podman[286526]: 2024-09-13 15:02:34.768322032 +0000 UTC m=+0.331959293 container init e614bb803073255793302c2a67de85a84e76deb322331e4192e2d285996f4957 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, RELEASE=reef-2c01458, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-09-13T15:02:35.216 INFO:journalctl@ceph.osd.5.smithi195.stdout:Sep 13 15:02:34 smithi195 podman[286526]: 2024-09-13 15:02:34.773629087 +0000 UTC m=+0.337266349 container start e614bb803073255793302c2a67de85a84e76deb322331e4192e2d285996f4957 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-5-deactivate, RELEASE=reef-2c01458, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-09-13T15:02:35.216 INFO:journalctl@ceph.osd.5.smithi195.stdout:Sep 13 15:02:34 smithi195 podman[286526]: 2024-09-13 15:02:34.851460378 +0000 UTC m=+0.415097652 container attach e614bb803073255793302c2a67de85a84e76deb322331e4192e2d285996f4957 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=reef-2c01458, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-13T15:02:35.216 INFO:journalctl@ceph.osd.5.smithi195.stdout:Sep 13 15:02:34 smithi195 podman[286526]: 2024-09-13 15:02:34.961416534 +0000 UTC m=+0.525053804 container died e614bb803073255793302c2a67de85a84e76deb322331e4192e2d285996f4957 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, RELEASE=reef-2c01458, org.label-schema.build-date=20240731) 2024-09-13T15:02:35.497 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:35 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:35.448+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.963343+0000 front 2024-09-13T15:01:49.963380+0000 (oldest deadline 2024-09-13T15:02:14.663480+0000) 2024-09-13T15:02:35.497 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:35 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:35.448+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:01:56.964520+0000 front 2024-09-13T15:01:56.964531+0000 (oldest deadline 2024-09-13T15:02:21.664327+0000) 2024-09-13T15:02:35.497 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:35 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:35.448+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-09-13T15:02:08.065814+0000 front 2024-09-13T15:02:08.065746+0000 (oldest deadline 2024-09-13T15:02:31.565719+0000) 2024-09-13T15:02:35.526 DEBUG:teuthology.orchestra.run.smithi195:> sudo pkill -f 'journalctl -f -n 0 -u ceph-08cee502-71d9-11ef-bceb-c7b262605968@osd.5.service' 2024-09-13T15:02:35.565 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:02:35.565 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-13T15:02:35.565 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-13T15:02:35.565 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ceph-08cee502-71d9-11ef-bceb-c7b262605968@osd.6 2024-09-13T15:02:35.747 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:35 smithi195 systemd[1]: Stopping Ceph osd.6 for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T15:02:36.487 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:36 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:36.018+0000 7f810e3ae640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:02:36.487 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:36 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:36.018+0000 7f810e3ae640 -1 osd.6 517 *** Got signal Terminated *** 2024-09-13T15:02:36.488 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:36 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:36.018+0000 7f810e3ae640 -1 osd.6 517 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T15:02:36.488 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:36 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:36.207+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:50.534049+0000 front 2024-09-13T15:01:50.534014+0000 (oldest deadline 2024-09-13T15:02:14.634064+0000) 2024-09-13T15:02:36.488 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:36 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:36.207+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:02:00.435721+0000 front 2024-09-13T15:02:00.435708+0000 (oldest deadline 2024-09-13T15:02:21.535538+0000) 2024-09-13T15:02:36.488 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:36 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:36.207+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-09-13T15:02:03.836587+0000 front 2024-09-13T15:02:03.836669+0000 (oldest deadline 2024-09-13T15:02:29.736636+0000) 2024-09-13T15:02:36.747 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:36 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:36.485+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.963343+0000 front 2024-09-13T15:01:49.963380+0000 (oldest deadline 2024-09-13T15:02:14.663480+0000) 2024-09-13T15:02:36.748 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:36 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:36.485+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:01:56.964520+0000 front 2024-09-13T15:01:56.964531+0000 (oldest deadline 2024-09-13T15:02:21.664327+0000) 2024-09-13T15:02:36.748 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:36 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:36.485+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-09-13T15:02:08.065814+0000 front 2024-09-13T15:02:08.065746+0000 (oldest deadline 2024-09-13T15:02:31.565719+0000) 2024-09-13T15:02:37.497 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:37.233+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:50.534049+0000 front 2024-09-13T15:01:50.534014+0000 (oldest deadline 2024-09-13T15:02:14.634064+0000) 2024-09-13T15:02:37.498 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:37.233+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:02:00.435721+0000 front 2024-09-13T15:02:00.435708+0000 (oldest deadline 2024-09-13T15:02:21.535538+0000) 2024-09-13T15:02:37.498 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:37.233+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-09-13T15:02:03.836587+0000 front 2024-09-13T15:02:03.836669+0000 (oldest deadline 2024-09-13T15:02:29.736636+0000) 2024-09-13T15:02:37.998 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:37.524+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.963343+0000 front 2024-09-13T15:01:49.963380+0000 (oldest deadline 2024-09-13T15:02:14.663480+0000) 2024-09-13T15:02:37.998 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:37.524+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:01:56.964520+0000 front 2024-09-13T15:01:56.964531+0000 (oldest deadline 2024-09-13T15:02:21.664327+0000) 2024-09-13T15:02:37.998 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:37.524+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-09-13T15:02:08.065814+0000 front 2024-09-13T15:02:08.065746+0000 (oldest deadline 2024-09-13T15:02:31.565719+0000) 2024-09-13T15:02:37.998 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:37 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:37.524+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6830 osd.3 since back 2024-09-13T15:02:12.666695+0000 front 2024-09-13T15:02:12.666728+0000 (oldest deadline 2024-09-13T15:02:37.366616+0000) 2024-09-13T15:02:38.498 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:38 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:38.194+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:50.534049+0000 front 2024-09-13T15:01:50.534014+0000 (oldest deadline 2024-09-13T15:02:14.634064+0000) 2024-09-13T15:02:38.498 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:38 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:38.194+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:02:00.435721+0000 front 2024-09-13T15:02:00.435708+0000 (oldest deadline 2024-09-13T15:02:21.535538+0000) 2024-09-13T15:02:38.498 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:38 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:38.194+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-09-13T15:02:03.836587+0000 front 2024-09-13T15:02:03.836669+0000 (oldest deadline 2024-09-13T15:02:29.736636+0000) 2024-09-13T15:02:38.498 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:38 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:38.194+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6830 osd.3 since back 2024-09-13T15:02:14.338239+0000 front 2024-09-13T15:02:14.338291+0000 (oldest deadline 2024-09-13T15:02:37.838199+0000) 2024-09-13T15:02:38.998 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:38 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:38.549+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.963343+0000 front 2024-09-13T15:01:49.963380+0000 (oldest deadline 2024-09-13T15:02:14.663480+0000) 2024-09-13T15:02:38.998 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:38 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:38.549+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:01:56.964520+0000 front 2024-09-13T15:01:56.964531+0000 (oldest deadline 2024-09-13T15:02:21.664327+0000) 2024-09-13T15:02:38.998 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:38 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:38.549+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-09-13T15:02:08.065814+0000 front 2024-09-13T15:02:08.065746+0000 (oldest deadline 2024-09-13T15:02:31.565719+0000) 2024-09-13T15:02:38.998 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:38 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:38.549+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6830 osd.3 since back 2024-09-13T15:02:12.666695+0000 front 2024-09-13T15:02:12.666728+0000 (oldest deadline 2024-09-13T15:02:37.366616+0000) 2024-09-13T15:02:39.497 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:39 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:39.196+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:50.534049+0000 front 2024-09-13T15:01:50.534014+0000 (oldest deadline 2024-09-13T15:02:14.634064+0000) 2024-09-13T15:02:39.498 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:39 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:39.196+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:02:00.435721+0000 front 2024-09-13T15:02:00.435708+0000 (oldest deadline 2024-09-13T15:02:21.535538+0000) 2024-09-13T15:02:39.498 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:39 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:39.196+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-09-13T15:02:03.836587+0000 front 2024-09-13T15:02:03.836669+0000 (oldest deadline 2024-09-13T15:02:29.736636+0000) 2024-09-13T15:02:39.498 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:39 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:39.196+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6830 osd.3 since back 2024-09-13T15:02:14.338239+0000 front 2024-09-13T15:02:14.338291+0000 (oldest deadline 2024-09-13T15:02:37.838199+0000) 2024-09-13T15:02:39.997 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:39 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:39.585+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.963343+0000 front 2024-09-13T15:01:49.963380+0000 (oldest deadline 2024-09-13T15:02:14.663480+0000) 2024-09-13T15:02:39.998 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:39 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:39.585+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:01:56.964520+0000 front 2024-09-13T15:01:56.964531+0000 (oldest deadline 2024-09-13T15:02:21.664327+0000) 2024-09-13T15:02:39.998 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:39 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:39.585+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-09-13T15:02:08.065814+0000 front 2024-09-13T15:02:08.065746+0000 (oldest deadline 2024-09-13T15:02:31.565719+0000) 2024-09-13T15:02:39.998 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:39 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:39.585+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6830 osd.3 since back 2024-09-13T15:02:12.666695+0000 front 2024-09-13T15:02:12.666728+0000 (oldest deadline 2024-09-13T15:02:37.366616+0000) 2024-09-13T15:02:40.497 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:40 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:40.244+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:50.534049+0000 front 2024-09-13T15:01:50.534014+0000 (oldest deadline 2024-09-13T15:02:14.634064+0000) 2024-09-13T15:02:40.498 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:40 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:40.244+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:02:00.435721+0000 front 2024-09-13T15:02:00.435708+0000 (oldest deadline 2024-09-13T15:02:21.535538+0000) 2024-09-13T15:02:40.498 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:40 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:40.244+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-09-13T15:02:03.836587+0000 front 2024-09-13T15:02:03.836669+0000 (oldest deadline 2024-09-13T15:02:29.736636+0000) 2024-09-13T15:02:40.498 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:40 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6[168883]: 2024-09-13T15:02:40.244+0000 7f81069ba640 -1 osd.6 517 heartbeat_check: no reply from 172.21.15.102:6830 osd.3 since back 2024-09-13T15:02:14.338239+0000 front 2024-09-13T15:02:14.338291+0000 (oldest deadline 2024-09-13T15:02:37.838199+0000) 2024-09-13T15:02:40.997 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:40 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:40.588+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.963343+0000 front 2024-09-13T15:01:49.963380+0000 (oldest deadline 2024-09-13T15:02:14.663480+0000) 2024-09-13T15:02:40.998 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:40 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:40.588+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:01:56.964520+0000 front 2024-09-13T15:01:56.964531+0000 (oldest deadline 2024-09-13T15:02:21.664327+0000) 2024-09-13T15:02:40.998 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:40 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:40.588+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-09-13T15:02:08.065814+0000 front 2024-09-13T15:02:08.065746+0000 (oldest deadline 2024-09-13T15:02:31.565719+0000) 2024-09-13T15:02:40.998 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:40 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:40.588+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6830 osd.3 since back 2024-09-13T15:02:12.666695+0000 front 2024-09-13T15:02:12.666728+0000 (oldest deadline 2024-09-13T15:02:37.366616+0000) 2024-09-13T15:02:41.898 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:41 smithi195 podman[286654]: 2024-09-13 15:02:41.642576732 +0000 UTC m=+5.853802650 container died 6927a7bcecccb5b49ac842264f0b3b2312145c9eec7a599f20dac4e3aced5246 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6, GIT_BRANCH=HEAD, RELEASE=reef-2c01458, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-09-13T15:02:41.899 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:41 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:41.565+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.963343+0000 front 2024-09-13T15:01:49.963380+0000 (oldest deadline 2024-09-13T15:02:14.663480+0000) 2024-09-13T15:02:41.899 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:41 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:41.565+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:01:56.964520+0000 front 2024-09-13T15:01:56.964531+0000 (oldest deadline 2024-09-13T15:02:21.664327+0000) 2024-09-13T15:02:41.899 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:41 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:41.565+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-09-13T15:02:08.065814+0000 front 2024-09-13T15:02:08.065746+0000 (oldest deadline 2024-09-13T15:02:31.565719+0000) 2024-09-13T15:02:41.899 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:41 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:41.565+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6830 osd.3 since back 2024-09-13T15:02:12.666695+0000 front 2024-09-13T15:02:12.666728+0000 (oldest deadline 2024-09-13T15:02:37.366616+0000) 2024-09-13T15:02:42.248 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:41 smithi195 podman[286654]: 2024-09-13 15:02:41.999727945 +0000 UTC m=+6.210953861 container cleanup 6927a7bcecccb5b49ac842264f0b3b2312145c9eec7a599f20dac4e3aced5246 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, RELEASE=reef-2c01458, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-13T15:02:42.248 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:41 smithi195 bash[286654]: ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6 2024-09-13T15:02:42.584 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:42 smithi195 podman[286677]: 2024-09-13 15:02:42.286166144 +0000 UTC m=+0.642014620 container remove 6927a7bcecccb5b49ac842264f0b3b2312145c9eec7a599f20dac4e3aced5246 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=reef-2c01458, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-13T15:02:42.922 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:42 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:42.583+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.963343+0000 front 2024-09-13T15:01:49.963380+0000 (oldest deadline 2024-09-13T15:02:14.663480+0000) 2024-09-13T15:02:42.922 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:42 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:42.583+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:01:56.964520+0000 front 2024-09-13T15:01:56.964531+0000 (oldest deadline 2024-09-13T15:02:21.664327+0000) 2024-09-13T15:02:42.922 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:42 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:42.583+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-09-13T15:02:08.065814+0000 front 2024-09-13T15:02:08.065746+0000 (oldest deadline 2024-09-13T15:02:31.565719+0000) 2024-09-13T15:02:42.922 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:42 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:42.583+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6830 osd.3 since back 2024-09-13T15:02:12.666695+0000 front 2024-09-13T15:02:12.666728+0000 (oldest deadline 2024-09-13T15:02:37.366616+0000) 2024-09-13T15:02:42.923 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:42 smithi195 podman[286768]: 2024-09-13 15:02:42.514094512 +0000 UTC m=+0.016543817 image pull c129f5a2fd4eccce3e6eec43830eff1929af98bc48f3b850cf144ce5b326e8b5 quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11 2024-09-13T15:02:42.923 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:42 smithi195 podman[286768]: 2024-09-13 15:02:42.62690732 +0000 UTC m=+0.129356624 container create 35f5939f0cbabd715dd09d9324d1a76b541eccddf466eb8b121ae025ac62ae85 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-2c01458, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T15:02:42.923 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:42 smithi195 podman[286768]: 2024-09-13 15:02:42.820825068 +0000 UTC m=+0.323274372 container init 35f5939f0cbabd715dd09d9324d1a76b541eccddf466eb8b121ae025ac62ae85 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, RELEASE=reef-2c01458, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True) 2024-09-13T15:02:42.923 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:42 smithi195 podman[286768]: 2024-09-13 15:02:42.826104069 +0000 UTC m=+0.328553374 container start 35f5939f0cbabd715dd09d9324d1a76b541eccddf466eb8b121ae025ac62ae85 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, RELEASE=reef-2c01458, org.label-schema.license=GPLv2) 2024-09-13T15:02:42.923 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:42 smithi195 podman[286768]: 2024-09-13 15:02:42.920821246 +0000 UTC m=+0.423270556 container attach 35f5939f0cbabd715dd09d9324d1a76b541eccddf466eb8b121ae025ac62ae85 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-2c01458, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-09-13T15:02:43.247 INFO:journalctl@ceph.osd.6.smithi195.stdout:Sep 13 15:02:43 smithi195 podman[286768]: 2024-09-13 15:02:43.010561434 +0000 UTC m=+0.513010738 container died 35f5939f0cbabd715dd09d9324d1a76b541eccddf466eb8b121ae025ac62ae85 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, RELEASE=reef-2c01458, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T15:02:43.568 DEBUG:teuthology.orchestra.run.smithi195:> sudo pkill -f 'journalctl -f -n 0 -u ceph-08cee502-71d9-11ef-bceb-c7b262605968@osd.6.service' 2024-09-13T15:02:43.608 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:02:43.609 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-13T15:02:43.609 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-13T15:02:43.609 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ceph-08cee502-71d9-11ef-bceb-c7b262605968@osd.7 2024-09-13T15:02:43.931 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:43 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:43.558+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.963343+0000 front 2024-09-13T15:01:49.963380+0000 (oldest deadline 2024-09-13T15:02:14.663480+0000) 2024-09-13T15:02:43.932 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:43 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:43.558+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:01:56.964520+0000 front 2024-09-13T15:01:56.964531+0000 (oldest deadline 2024-09-13T15:02:21.664327+0000) 2024-09-13T15:02:43.932 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:43 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:43.558+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-09-13T15:02:08.065814+0000 front 2024-09-13T15:02:08.065746+0000 (oldest deadline 2024-09-13T15:02:31.565719+0000) 2024-09-13T15:02:43.932 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:43 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:43.558+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6830 osd.3 since back 2024-09-13T15:02:12.666695+0000 front 2024-09-13T15:02:12.666728+0000 (oldest deadline 2024-09-13T15:02:37.366616+0000) 2024-09-13T15:02:43.932 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:43 smithi195 systemd[1]: Stopping Ceph osd.7 for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T15:02:44.247 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:43 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:43.929+0000 7f6a08caf640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:02:44.248 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:43 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:43.929+0000 7f6a08caf640 -1 osd.7 517 *** Got signal Terminated *** 2024-09-13T15:02:44.248 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:43 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:43.929+0000 7f6a08caf640 -1 osd.7 517 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T15:02:44.998 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:44 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:44.581+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.963343+0000 front 2024-09-13T15:01:49.963380+0000 (oldest deadline 2024-09-13T15:02:14.663480+0000) 2024-09-13T15:02:44.998 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:44 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:44.581+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:01:56.964520+0000 front 2024-09-13T15:01:56.964531+0000 (oldest deadline 2024-09-13T15:02:21.664327+0000) 2024-09-13T15:02:44.998 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:44 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:44.581+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-09-13T15:02:08.065814+0000 front 2024-09-13T15:02:08.065746+0000 (oldest deadline 2024-09-13T15:02:31.565719+0000) 2024-09-13T15:02:44.998 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:44 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:44.581+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6830 osd.3 since back 2024-09-13T15:02:12.666695+0000 front 2024-09-13T15:02:12.666728+0000 (oldest deadline 2024-09-13T15:02:37.366616+0000) 2024-09-13T15:02:45.900 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:45 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:45.572+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.963343+0000 front 2024-09-13T15:01:49.963380+0000 (oldest deadline 2024-09-13T15:02:14.663480+0000) 2024-09-13T15:02:45.900 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:45 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:45.572+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:01:56.964520+0000 front 2024-09-13T15:01:56.964531+0000 (oldest deadline 2024-09-13T15:02:21.664327+0000) 2024-09-13T15:02:45.900 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:45 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:45.572+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-09-13T15:02:08.065814+0000 front 2024-09-13T15:02:08.065746+0000 (oldest deadline 2024-09-13T15:02:31.565719+0000) 2024-09-13T15:02:45.900 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:45 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:45.572+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6830 osd.3 since back 2024-09-13T15:02:12.666695+0000 front 2024-09-13T15:02:12.666728+0000 (oldest deadline 2024-09-13T15:02:37.366616+0000) 2024-09-13T15:02:46.912 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:46 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:46.581+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.963343+0000 front 2024-09-13T15:01:49.963380+0000 (oldest deadline 2024-09-13T15:02:14.663480+0000) 2024-09-13T15:02:46.913 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:46 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:46.581+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:01:56.964520+0000 front 2024-09-13T15:01:56.964531+0000 (oldest deadline 2024-09-13T15:02:21.664327+0000) 2024-09-13T15:02:46.913 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:46 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:46.581+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-09-13T15:02:08.065814+0000 front 2024-09-13T15:02:08.065746+0000 (oldest deadline 2024-09-13T15:02:31.565719+0000) 2024-09-13T15:02:46.913 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:46 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:46.581+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6830 osd.3 since back 2024-09-13T15:02:12.666695+0000 front 2024-09-13T15:02:12.666728+0000 (oldest deadline 2024-09-13T15:02:37.366616+0000) 2024-09-13T15:02:47.997 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:47 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:47.533+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.963343+0000 front 2024-09-13T15:01:49.963380+0000 (oldest deadline 2024-09-13T15:02:14.663480+0000) 2024-09-13T15:02:47.998 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:47 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:47.533+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:01:56.964520+0000 front 2024-09-13T15:01:56.964531+0000 (oldest deadline 2024-09-13T15:02:21.664327+0000) 2024-09-13T15:02:47.998 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:47 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:47.533+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-09-13T15:02:08.065814+0000 front 2024-09-13T15:02:08.065746+0000 (oldest deadline 2024-09-13T15:02:31.565719+0000) 2024-09-13T15:02:47.998 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:47 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:47.533+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6830 osd.3 since back 2024-09-13T15:02:12.666695+0000 front 2024-09-13T15:02:12.666728+0000 (oldest deadline 2024-09-13T15:02:37.366616+0000) 2024-09-13T15:02:47.998 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:47 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:47.533+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.195:6804 osd.4 since back 2024-09-13T15:02:23.969639+0000 front 2024-09-13T15:02:23.969683+0000 (oldest deadline 2024-09-13T15:02:47.469615+0000) 2024-09-13T15:02:48.998 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:48 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:48.549+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-09-13T15:01:49.963343+0000 front 2024-09-13T15:01:49.963380+0000 (oldest deadline 2024-09-13T15:02:14.663480+0000) 2024-09-13T15:02:48.998 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:48 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:48.549+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-09-13T15:01:56.964520+0000 front 2024-09-13T15:01:56.964531+0000 (oldest deadline 2024-09-13T15:02:21.664327+0000) 2024-09-13T15:02:48.998 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:48 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:48.549+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-09-13T15:02:08.065814+0000 front 2024-09-13T15:02:08.065746+0000 (oldest deadline 2024-09-13T15:02:31.565719+0000) 2024-09-13T15:02:48.998 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:48 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:48.549+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.102:6830 osd.3 since back 2024-09-13T15:02:12.666695+0000 front 2024-09-13T15:02:12.666728+0000 (oldest deadline 2024-09-13T15:02:37.366616+0000) 2024-09-13T15:02:48.998 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:48 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7[56761]: 2024-09-13T15:02:48.549+0000 7f6a052c3640 -1 osd.7 517 heartbeat_check: no reply from 172.21.15.195:6804 osd.4 since back 2024-09-13T15:02:23.969639+0000 front 2024-09-13T15:02:23.969683+0000 (oldest deadline 2024-09-13T15:02:47.469615+0000) 2024-09-13T15:02:49.889 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:49 smithi195 podman[286896]: 2024-09-13 15:02:49.617944539 +0000 UTC m=+5.791654893 container died a6b698c659cae7d2e0c4d3ce06914c67cc01f00e1c2e3a1e90b55334d94dfc28 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=reef-2c01458, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T15:02:50.247 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:49 smithi195 podman[286896]: 2024-09-13 15:02:49.985517933 +0000 UTC m=+6.159228275 container cleanup a6b698c659cae7d2e0c4d3ce06914c67cc01f00e1c2e3a1e90b55334d94dfc28 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, RELEASE=reef-2c01458) 2024-09-13T15:02:50.248 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:49 smithi195 bash[286896]: ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7 2024-09-13T15:02:50.572 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:50 smithi195 podman[286928]: 2024-09-13 15:02:50.261941749 +0000 UTC m=+0.647427087 container remove a6b698c659cae7d2e0c4d3ce06914c67cc01f00e1c2e3a1e90b55334d94dfc28 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, RELEASE=reef-2c01458, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True) 2024-09-13T15:02:50.882 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:50 smithi195 podman[287023]: 2024-09-13 15:02:50.473980779 +0000 UTC m=+0.015838806 image pull c129f5a2fd4eccce3e6eec43830eff1929af98bc48f3b850cf144ce5b326e8b5 quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11 2024-09-13T15:02:50.883 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:50 smithi195 podman[287023]: 2024-09-13 15:02:50.596124698 +0000 UTC m=+0.137982716 container create 77fd5892a03f9327144c2777d5bad2fb1f52dc70f37061b385c032458bf0a672 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=reef-2c01458, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True) 2024-09-13T15:02:50.883 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:50 smithi195 podman[287023]: 2024-09-13 15:02:50.798596916 +0000 UTC m=+0.340454934 container init 77fd5892a03f9327144c2777d5bad2fb1f52dc70f37061b385c032458bf0a672 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=reef-2c01458, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-13T15:02:50.883 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:50 smithi195 podman[287023]: 2024-09-13 15:02:50.80435982 +0000 UTC m=+0.346217842 container start 77fd5892a03f9327144c2777d5bad2fb1f52dc70f37061b385c032458bf0a672 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=reef-2c01458, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-13T15:02:50.883 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:50 smithi195 podman[287023]: 2024-09-13 15:02:50.881851437 +0000 UTC m=+0.423709468 container attach 77fd5892a03f9327144c2777d5bad2fb1f52dc70f37061b385c032458bf0a672 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, RELEASE=reef-2c01458, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-13T15:02:51.202 INFO:journalctl@ceph.osd.7.smithi195.stdout:Sep 13 15:02:50 smithi195 podman[287023]: 2024-09-13 15:02:50.985218808 +0000 UTC m=+0.527076827 container died 77fd5892a03f9327144c2777d5bad2fb1f52dc70f37061b385c032458bf0a672 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-2c01458, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T15:02:51.498 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 15:02:51 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[89530]: ts=2024-09-13T15:02:51.202Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.102:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.102:8765: connect: connection refused" 2024-09-13T15:02:51.498 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 15:02:51 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[89530]: ts=2024-09-13T15:02:51.202Z caller=refresh.go:90 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.102:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.102:8765: connect: connection refused" 2024-09-13T15:02:51.498 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 15:02:51 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[89530]: ts=2024-09-13T15:02:51.202Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.102:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.102:8765: connect: connection refused" 2024-09-13T15:02:51.498 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 15:02:51 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[89530]: ts=2024-09-13T15:02:51.205Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.102:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.102:8765: connect: connection refused" 2024-09-13T15:02:51.499 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 15:02:51 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[89530]: ts=2024-09-13T15:02:51.205Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.102:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.102:8765: connect: connection refused" 2024-09-13T15:02:51.499 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 15:02:51 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[89530]: ts=2024-09-13T15:02:51.205Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nfs msg="Unable to refresh target groups" err="Get \"http://172.21.15.102:8765/sd/prometheus/sd-config?service=nfs\": dial tcp 172.21.15.102:8765: connect: connection refused" 2024-09-13T15:02:51.579 DEBUG:teuthology.orchestra.run.smithi195:> sudo pkill -f 'journalctl -f -n 0 -u ceph-08cee502-71d9-11ef-bceb-c7b262605968@osd.7.service' 2024-09-13T15:02:51.618 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:02:51.619 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-13T15:02:51.619 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-09-13T15:02:51.619 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ceph-08cee502-71d9-11ef-bceb-c7b262605968@prometheus.a 2024-09-13T15:02:51.933 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 15:02:51 smithi195 systemd[1]: Stopping Ceph prometheus.a for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T15:02:52.210 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 15:02:51 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[89530]: ts=2024-09-13T15:02:51.931Z caller=main.go:964 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-13T15:02:52.211 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 15:02:51 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[89530]: ts=2024-09-13T15:02:51.932Z caller=main.go:988 level=info msg="Stopping scrape discovery manager..." 2024-09-13T15:02:52.211 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 15:02:51 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[89530]: ts=2024-09-13T15:02:51.932Z caller=main.go:1002 level=info msg="Stopping notify discovery manager..." 2024-09-13T15:02:52.211 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 15:02:51 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[89530]: ts=2024-09-13T15:02:51.932Z caller=manager.go:177 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-13T15:02:52.211 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 15:02:51 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[89530]: ts=2024-09-13T15:02:51.932Z caller=main.go:984 level=info msg="Scrape discovery manager stopped" 2024-09-13T15:02:52.211 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 15:02:51 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[89530]: ts=2024-09-13T15:02:51.932Z caller=main.go:998 level=info msg="Notify discovery manager stopped" 2024-09-13T15:02:52.211 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 15:02:51 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[89530]: ts=2024-09-13T15:02:51.932Z caller=manager.go:187 level=info component="rule manager" msg="Rule manager stopped" 2024-09-13T15:02:52.211 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 15:02:51 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[89530]: ts=2024-09-13T15:02:51.932Z caller=main.go:1039 level=info msg="Stopping scrape manager..." 2024-09-13T15:02:52.212 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 15:02:51 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[89530]: ts=2024-09-13T15:02:51.932Z caller=main.go:1031 level=info msg="Scrape manager stopped" 2024-09-13T15:02:52.212 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 15:02:51 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[89530]: ts=2024-09-13T15:02:51.934Z caller=notifier.go:618 level=info component=notifier msg="Stopping notification manager..." 2024-09-13T15:02:52.212 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 15:02:51 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[89530]: ts=2024-09-13T15:02:51.934Z caller=main.go:1261 level=info msg="Notifier manager stopped" 2024-09-13T15:02:52.212 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 15:02:51 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a[89530]: ts=2024-09-13T15:02:51.934Z caller=main.go:1273 level=info msg="See you next time!" 2024-09-13T15:02:52.212 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 15:02:51 smithi195 podman[287151]: 2024-09-13 15:02:51.942294473 +0000 UTC m=+0.107600794 container died d131e53f1ef9319ff3efc21e6bfc7a7e1bc630b1a34402d7507836f08a03326c (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T15:02:52.497 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 15:02:52 smithi195 podman[287151]: 2024-09-13 15:02:52.287794736 +0000 UTC m=+0.453101097 container cleanup d131e53f1ef9319ff3efc21e6bfc7a7e1bc630b1a34402d7507836f08a03326c (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T15:02:52.497 INFO:journalctl@ceph.prometheus.a.smithi195.stdout:Sep 13 15:02:52 smithi195 bash[287151]: ceph-08cee502-71d9-11ef-bceb-c7b262605968-prometheus-a 2024-09-13T15:02:52.725 DEBUG:teuthology.orchestra.run.smithi195:> sudo pkill -f 'journalctl -f -n 0 -u ceph-08cee502-71d9-11ef-bceb-c7b262605968@prometheus.a.service' 2024-09-13T15:02:52.759 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:02:52.759 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-09-13T15:02:52.759 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 08cee502-71d9-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-13T15:02:52.910 INFO:teuthology.orchestra.run.smithi102.stdout:Deleting cluster with fsid: 08cee502-71d9-11ef-bceb-c7b262605968 2024-09-13T15:02:54.565 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 15:02:54 smithi102 systemd[1]: Stopping Ceph alertmanager.a for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T15:02:54.841 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 15:02:54 smithi102 ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a[104597]: ts=2024-09-13T15:02:54.563Z caller=main.go:542 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-13T15:02:54.841 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 15:02:54 smithi102 podman[422286]: 2024-09-13 15:02:54.604807927 +0000 UTC m=+0.146142648 container died 0322b99ec767f5517953f4f2878ec0867ac99f60ce85590fb48052a82d694c02 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T15:02:55.169 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 15:02:55 smithi102 podman[422286]: 2024-09-13 15:02:55.020022772 +0000 UTC m=+0.561357484 container cleanup 0322b99ec767f5517953f4f2878ec0867ac99f60ce85590fb48052a82d694c02 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T15:02:55.169 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 15:02:55 smithi102 bash[422286]: ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a 2024-09-13T15:02:55.578 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 15:02:55 smithi102 podman[422304]: 2024-09-13 15:02:55.271426411 +0000 UTC m=+0.663295889 container remove 0322b99ec767f5517953f4f2878ec0867ac99f60ce85590fb48052a82d694c02 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T15:02:55.578 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 15:02:55 smithi102 podman[422304]: 2024-09-13 15:02:55.355734637 +0000 UTC m=+0.747604122 volume remove 28d455061ff69238c1c4f13fd6ba611ba16997265aa3b0324407e86cdafb6198 2024-09-13T15:02:55.918 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 15:02:55 smithi102 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-13T15:02:55.919 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 15:02:55 smithi102 systemd[1]: Stopped Ceph alertmanager.a for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T15:02:55.919 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Sep 13 15:02:55 smithi102 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@alertmanager.a.service: Consumed 3.957s CPU time. 2024-09-13T15:02:55.919 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 15:02:55 smithi102 systemd[1]: Stopping Ceph node-exporter.a for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T15:02:56.277 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 15:02:56 smithi102 podman[422418]: 2024-09-13 15:02:56.021373853 +0000 UTC m=+0.107407267 container died 0c92d0b39468100a77b6682d8279999de190faa4885fa01d6000970e4a5aedbd (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T15:02:56.669 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 15:02:56 smithi102 podman[422418]: 2024-09-13 15:02:56.492039339 +0000 UTC m=+0.578072796 container cleanup 0c92d0b39468100a77b6682d8279999de190faa4885fa01d6000970e4a5aedbd (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T15:02:56.669 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 15:02:56 smithi102 bash[422418]: ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a 2024-09-13T15:02:57.071 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 15:02:56 smithi102 podman[422436]: 2024-09-13 15:02:56.776661988 +0000 UTC m=+0.754896480 container remove 0c92d0b39468100a77b6682d8279999de190faa4885fa01d6000970e4a5aedbd (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T15:02:57.071 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 15:02:56 smithi102 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-13T15:02:57.071 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 15:02:56 smithi102 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-13T15:02:57.071 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 15:02:56 smithi102 systemd[1]: Stopped Ceph node-exporter.a for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T15:02:57.071 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Sep 13 15:02:56 smithi102 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@node-exporter.a.service: Consumed 8.267s CPU time. 2024-09-13T15:03:05.043 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 08cee502-71d9-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-13T15:03:05.197 INFO:teuthology.orchestra.run.smithi195.stdout:Deleting cluster with fsid: 08cee502-71d9-11ef-bceb-c7b262605968 2024-09-13T15:03:06.823 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 15:03:06 smithi195 systemd[1]: Stopping Ceph grafana.a for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T15:03:06.823 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 15:03:06 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=server t=2024-09-13T15:03:06.811238042Z level=info msg="Shutdown started" reason="System signal: terminated" 2024-09-13T15:03:06.823 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 15:03:06 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=ticker t=2024-09-13T15:03:06.811656645Z level=info msg=stopped last_tick=2024-09-13T15:03:00Z 2024-09-13T15:03:06.823 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 15:03:06 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=tracing t=2024-09-13T15:03:06.811670836Z level=info msg="Closing tracing" 2024-09-13T15:03:06.823 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 15:03:06 smithi195 ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a[94366]: logger=grafana-apiserver t=2024-09-13T15:03:06.811668513Z level=info msg="StorageObjectCountTracker pruner is exiting" 2024-09-13T15:03:06.823 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 15:03:06 smithi195 podman[287568]: 2024-09-13 15:03:06.821777648 +0000 UTC m=+0.112804890 container died 152e89bad2c83187ec1732fb75be2866fe0baed6be2b2e6353d87e2c4d70a1f2 (image=quay.io/ceph/grafana:10.4.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-09-13T15:03:07.485 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 15:03:07 smithi195 podman[287568]: 2024-09-13 15:03:07.200231374 +0000 UTC m=+0.491258571 container cleanup 152e89bad2c83187ec1732fb75be2866fe0baed6be2b2e6353d87e2c4d70a1f2 (image=quay.io/ceph/grafana:10.4.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-09-13T15:03:07.485 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 15:03:07 smithi195 bash[287568]: ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a 2024-09-13T15:03:07.747 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 15:03:07 smithi195 podman[287585]: 2024-09-13 15:03:07.484709262 +0000 UTC m=+0.648656975 container remove 152e89bad2c83187ec1732fb75be2866fe0baed6be2b2e6353d87e2c4d70a1f2 (image=quay.io/ceph/grafana:10.4.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-09-13T15:03:07.748 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 15:03:07 smithi195 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@grafana.a.service: Deactivated successfully. 2024-09-13T15:03:07.748 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 15:03:07 smithi195 systemd[1]: Stopped Ceph grafana.a for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T15:03:07.748 INFO:journalctl@ceph.grafana.a.smithi195.stdout:Sep 13 15:03:07 smithi195 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@grafana.a.service: Consumed 32.058s CPU time. 2024-09-13T15:03:08.051 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 15:03:07 smithi195 systemd[1]: Stopping Ceph node-exporter.b for 08cee502-71d9-11ef-bceb-c7b262605968... 2024-09-13T15:03:08.355 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 15:03:08 smithi195 podman[287698]: 2024-09-13 15:03:08.050900454 +0000 UTC m=+0.122712102 container died fde867a7dbf91652db7c868363d7d2cc2593716918a95075e2ea9cebc614ab0f (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T15:03:08.355 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 15:03:08 smithi195 podman[287698]: 2024-09-13 15:03:08.354635581 +0000 UTC m=+0.426447228 container cleanup fde867a7dbf91652db7c868363d7d2cc2593716918a95075e2ea9cebc614ab0f (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T15:03:08.673 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 15:03:08 smithi195 bash[287698]: ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b 2024-09-13T15:03:08.947 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 15:03:08 smithi195 podman[287715]: 2024-09-13 15:03:08.672414705 +0000 UTC m=+0.625728614 container remove fde867a7dbf91652db7c868363d7d2cc2593716918a95075e2ea9cebc614ab0f (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-08cee502-71d9-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T15:03:08.947 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 15:03:08 smithi195 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-13T15:03:08.947 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 15:03:08 smithi195 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-13T15:03:08.947 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 15:03:08 smithi195 systemd[1]: Stopped Ceph node-exporter.b for 08cee502-71d9-11ef-bceb-c7b262605968. 2024-09-13T15:03:08.947 INFO:journalctl@ceph.node-exporter.b.smithi195.stdout:Sep 13 15:03:08 smithi195 systemd[1]: ceph-08cee502-71d9-11ef-bceb-c7b262605968@node-exporter.b.service: Consumed 8.109s CPU time. 2024-09-13T15:03:13.596 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-13T15:03:13.629 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-13T15:03:13.658 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-13T15:03:13.660 DEBUG:teuthology.misc:Transferring archived files from smithi102:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/akupczyk-2024-09-13_10:00:18-upgrade-wip-aclamk-cbt-combined-distro-default-smithi/7903539/remote/smithi102/crash 2024-09-13T15:03:13.661 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar c -f - -C /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/crash -- . 2024-09-13T15:03:13.710 INFO:teuthology.orchestra.run.smithi102.stderr:tar: /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-13T15:03:13.710 INFO:teuthology.orchestra.run.smithi102.stderr:tar: Error is not recoverable: exiting now 2024-09-13T15:03:13.712 DEBUG:teuthology.misc:Transferring archived files from smithi195:/var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/akupczyk-2024-09-13_10:00:18-upgrade-wip-aclamk-cbt-combined-distro-default-smithi/7903539/remote/smithi195/crash 2024-09-13T15:03:13.713 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar c -f - -C /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/crash -- . 2024-09-13T15:03:13.746 INFO:teuthology.orchestra.run.smithi195.stderr:tar: /var/lib/ceph/08cee502-71d9-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-13T15:03:13.746 INFO:teuthology.orchestra.run.smithi195.stderr:tar: Error is not recoverable: exiting now 2024-09-13T15:03:13.747 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-13T15:03:13.748 DEBUG:teuthology.orchestra.run.smithi102:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v MON_DOWN | egrep -v 'out of quorum' | egrep -v PG_AVAILABILITY | head -n 1 2024-09-13T15:03:13.882 INFO:teuthology.orchestra.run.smithi102.stdout:2024-09-13T14:10:14.003907+0000 mon.a (mon.0) 693 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T15:03:13.895 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-09-13T15:03:13.895 INFO:tasks.cephadm:Compressing logs... 2024-09-13T15:03:13.895 DEBUG:teuthology.orchestra.run.smithi102:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-13T15:03:13.897 DEBUG:teuthology.orchestra.run.smithi195:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-13T15:03:13.921 INFO:teuthology.orchestra.run.smithi102.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-13T15:03:13.934 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-osd.0.log 2024-09-13T15:03:13.934 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-osd.1.log 2024-09-13T15:03:13.934 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-volume.log 2024-09-13T15:03:13.934 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph.audit.log 2024-09-13T15:03:13.934 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-mon.c.log 2024-09-13T15:03:13.934 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-mds.foo.smithi102.njetxr.log 2024-09-13T15:03:13.936 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph.cephadm.log 2024-09-13T15:03:13.936 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-mds.foo.smithi102.njetxr.log: gzip -5 --verbose -- /var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-mon.a.log 2024-09-13T15:03:13.937 INFO:teuthology.orchestra.run.smithi102.stderr: 86.0% -- replaced with /var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-mds.foo.smithi102.njetxr.log.gz 2024-09-13T15:03:13.937 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-osd.2.log 2024-09-13T15:03:13.937 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-mgr.y.log 2024-09-13T15:03:13.937 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph.log 2024-09-13T15:03:13.937 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-osd.3.log 2024-09-13T15:03:13.937 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-13T15:03:13.945 INFO:teuthology.orchestra.run.smithi195.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-13T15:03:13.945 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-osd.7.log 2024-09-13T15:03:13.946 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-volume.log 2024-09-13T15:03:13.946 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-osd.4.log 2024-09-13T15:03:13.946 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-osd.7.log: /var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph.audit.log 2024-09-13T15:03:13.947 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph.cephadm.log 2024-09-13T15:03:13.947 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-osd.6.log 2024-09-13T15:03:13.947 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph.audit.log: /var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-osd.5.log 2024-09-13T15:03:13.947 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph.log 2024-09-13T15:03:13.949 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-mgr.x.log 2024-09-13T15:03:13.949 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-mon.b.log 2024-09-13T15:03:13.949 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-mds.foo.smithi195.yqzrxi.log 2024-09-13T15:03:13.949 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-13T15:03:13.949 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-mds.foo.smithi195.yqzrxi.log: 83.7% -- replaced with /var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-mds.foo.smithi195.yqzrxi.log.gz 2024-09-13T15:03:14.138 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-osd.3.log: /var/log/ceph/cephadm.log: 93.4% -- replaced with /var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph.log.gz 2024-09-13T15:03:14.153 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cephadm.log: 85.8% -- replaced with /var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph.cephadm.log.gz 2024-09-13T15:03:14.213 INFO:teuthology.orchestra.run.smithi195.stderr: 88.1% -- replaced with /var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph.log.gz 2024-09-13T15:03:14.218 INFO:teuthology.orchestra.run.smithi102.stderr: 95.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-13T15:03:14.258 INFO:teuthology.orchestra.run.smithi102.stderr: 91.2% -- replaced with /var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-volume.log.gz 2024-09-13T15:03:14.291 INFO:teuthology.orchestra.run.smithi195.stderr: 90.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-13T15:03:14.296 INFO:teuthology.orchestra.run.smithi195.stderr: 90.5% -- replaced with /var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph.audit.log.gz 2024-09-13T15:03:14.317 INFO:teuthology.orchestra.run.smithi195.stderr: 91.3% -- replaced with /var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-volume.log.gz 2024-09-13T15:03:14.557 INFO:teuthology.orchestra.run.smithi102.stderr: 94.3% -- replaced with /var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph.audit.log.gz 2024-09-13T15:03:14.593 INFO:teuthology.orchestra.run.smithi195.stderr: 89.8% -- replaced with /var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-mgr.x.log.gz 2024-09-13T15:03:14.927 INFO:teuthology.orchestra.run.smithi102.stderr: 92.0% -- replaced with /var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph.cephadm.log.gz 2024-09-13T15:03:18.654 INFO:teuthology.orchestra.run.smithi102.stderr: 89.8% -- replaced with /var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-mgr.y.log.gz 2024-09-13T15:03:35.696 INFO:teuthology.orchestra.run.smithi102.stderr: 91.7% -- replaced with /var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-mon.c.log.gz 2024-09-13T15:03:42.135 INFO:teuthology.orchestra.run.smithi102.stderr: 90.4% -- replaced with /var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-mon.a.log.gz 2024-09-13T15:03:44.695 INFO:teuthology.orchestra.run.smithi195.stderr: 92.4% -- replaced with /var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-mon.b.log.gz 2024-09-13T15:05:51.855 INFO:teuthology.orchestra.run.smithi102.stderr: 93.4% -- replaced with /var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-osd.2.log.gz 2024-09-13T15:11:38.462 INFO:teuthology.orchestra.run.smithi102.stderr: 92.1% -- replaced with /var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-osd.1.log.gz 2024-09-13T15:17:32.402 INFO:teuthology.orchestra.run.smithi195.stderr: 93.7% -- replaced with /var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-osd.6.log.gz 2024-09-13T15:21:18.177 INFO:teuthology.orchestra.run.smithi102.stderr: 92.5% -- replaced with /var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-osd.0.log.gz 2024-09-13T15:21:29.366 INFO:teuthology.orchestra.run.smithi102.stderr: 92.2% -- replaced with /var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-osd.3.log.gz 2024-09-13T15:21:29.419 INFO:teuthology.orchestra.run.smithi102.stderr: 2024-09-13T15:21:29.419 INFO:teuthology.orchestra.run.smithi102.stderr:real 18m15.508s 2024-09-13T15:21:29.419 INFO:teuthology.orchestra.run.smithi102.stderr:user 8m58.829s 2024-09-13T15:21:29.419 INFO:teuthology.orchestra.run.smithi102.stderr:sys 0m24.512s 2024-09-13T15:22:55.814 INFO:teuthology.orchestra.run.smithi195.stderr: 92.5% -- replaced with /var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-osd.4.log.gz 2024-09-13T15:25:27.621 INFO:teuthology.orchestra.run.smithi195.stderr: 92.7% -- replaced with /var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-osd.5.log.gz 2024-09-13T15:25:42.785 INFO:teuthology.orchestra.run.smithi195.stderr: 91.7% -- replaced with /var/log/ceph/08cee502-71d9-11ef-bceb-c7b262605968/ceph-osd.7.log.gz 2024-09-13T15:25:42.850 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-09-13T15:25:42.850 INFO:teuthology.orchestra.run.smithi195.stderr:real 22m28.937s 2024-09-13T15:25:42.850 INFO:teuthology.orchestra.run.smithi195.stderr:user 7m52.729s 2024-09-13T15:25:42.850 INFO:teuthology.orchestra.run.smithi195.stderr:sys 0m23.539s 2024-09-13T15:25:42.851 INFO:tasks.cephadm:Archiving logs... 2024-09-13T15:25:42.851 DEBUG:teuthology.misc:Transferring archived files from smithi102:/var/log/ceph to /home/teuthworker/archive/akupczyk-2024-09-13_10:00:18-upgrade-wip-aclamk-cbt-combined-distro-default-smithi/7903539/remote/smithi102/log 2024-09-13T15:25:42.853 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-13T15:27:10.908 DEBUG:teuthology.misc:Transferring archived files from smithi195:/var/log/ceph to /home/teuthworker/archive/akupczyk-2024-09-13_10:00:18-upgrade-wip-aclamk-cbt-combined-distro-default-smithi/7903539/remote/smithi195/log 2024-09-13T15:27:10.910 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-13T15:28:30.668 INFO:tasks.cephadm:Removing cluster... 2024-09-13T15:28:30.690 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 08cee502-71d9-11ef-bceb-c7b262605968 --force 2024-09-13T15:28:31.248 INFO:teuthology.orchestra.run.smithi102.stdout:Deleting cluster with fsid: 08cee502-71d9-11ef-bceb-c7b262605968 2024-09-13T15:28:32.105 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 08cee502-71d9-11ef-bceb-c7b262605968 --force 2024-09-13T15:28:32.745 INFO:teuthology.orchestra.run.smithi195.stdout:Deleting cluster with fsid: 08cee502-71d9-11ef-bceb-c7b262605968 2024-09-13T15:28:33.545 INFO:tasks.cephadm:Removing cephadm ... 2024-09-13T15:28:33.545 DEBUG:teuthology.orchestra.run.smithi102:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-13T15:28:33.565 DEBUG:teuthology.orchestra.run.smithi195:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-13T15:28:33.580 INFO:tasks.cephadm:Teardown complete 2024-09-13T15:28:33.581 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-13T15:28:33.590 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-13T15:28:33.591 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T15:28:33.607 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T15:28:33.722 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-13T15:28:33.731 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-13T15:28:33.732 DEBUG:teuthology.orchestra.run.smithi102:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T15:28:33.734 DEBUG:teuthology.orchestra.run.smithi195:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T15:28:33.748 INFO:teuthology.orchestra.run.smithi102.stderr:bash: line 1: ntpq: command not found 2024-09-13T15:28:33.749 INFO:teuthology.orchestra.run.smithi195.stderr:bash: line 1: ntpq: command not found 2024-09-13T15:28:33.819 INFO:teuthology.orchestra.run.smithi195.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T15:28:33.820 INFO:teuthology.orchestra.run.smithi195.stdout:=============================================================================== 2024-09-13T15:28:33.820 INFO:teuthology.orchestra.run.smithi195.stdout:^* hv01.front.sepia.ceph.com 3 6 377 41 +302us[ +303us] +/- 58ms 2024-09-13T15:28:33.820 INFO:teuthology.orchestra.run.smithi195.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 43 -845us[ -843us] +/- 73ms 2024-09-13T15:28:33.820 INFO:teuthology.orchestra.run.smithi195.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 47 +493us[ +494us] +/- 73ms 2024-09-13T15:28:33.821 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T15:28:33.836 INFO:teuthology.orchestra.run.smithi102.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T15:28:33.837 INFO:teuthology.orchestra.run.smithi102.stdout:=============================================================================== 2024-09-13T15:28:33.837 INFO:teuthology.orchestra.run.smithi102.stdout:^* hv01.front.sepia.ceph.com 3 6 377 43 +297us[ +317us] +/- 58ms 2024-09-13T15:28:33.837 INFO:teuthology.orchestra.run.smithi102.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 46 -835us[ -815us] +/- 73ms 2024-09-13T15:28:33.837 INFO:teuthology.orchestra.run.smithi102.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 44 +507us[ +528us] +/- 73ms 2024-09-13T15:28:33.837 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T15:28:33.845 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-13T15:28:33.855 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-13T15:28:33.855 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-13T15:28:33.883 DEBUG:teuthology.orchestra.run.smithi102:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-13T15:28:34.065 DEBUG:teuthology.orchestra.run.smithi195:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-13T15:28:34.196 DEBUG:teuthology.orchestra.run.smithi102:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-13T15:28:34.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:28:34.227 DEBUG:teuthology.orchestra.run.smithi195:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-13T15:28:34.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:28:34.258 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-13T15:28:34.268 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-13T15:28:34.309 INFO:teuthology.task.internal:Duration was 5451.257909 seconds 2024-09-13T15:28:34.310 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-13T15:28:34.359 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-13T15:28:34.359 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T15:28:34.363 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T15:28:34.420 INFO:teuthology.orchestra.run.smithi195.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T15:28:34.429 INFO:teuthology.orchestra.run.smithi102.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T15:28:34.836 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-13T15:28:34.836 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi102.front.sepia.ceph.com 2024-09-13T15:28:34.836 DEBUG:teuthology.orchestra.run.smithi102:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T15:28:34.869 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi195.front.sepia.ceph.com 2024-09-13T15:28:34.870 DEBUG:teuthology.orchestra.run.smithi195:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T15:28:34.896 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-13T15:28:34.897 DEBUG:teuthology.orchestra.run.smithi102:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T15:28:34.913 DEBUG:teuthology.orchestra.run.smithi195:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T15:28:34.964 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-13T15:28:34.964 DEBUG:teuthology.orchestra.run.smithi102:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T15:28:34.966 DEBUG:teuthology.orchestra.run.smithi195:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T15:28:36.783 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-13T15:28:36.796 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-13T15:28:36.797 DEBUG:teuthology.orchestra.run.smithi102:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T15:28:36.823 DEBUG:teuthology.orchestra.run.smithi195:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T15:28:36.851 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-13T15:28:36.861 DEBUG:teuthology.orchestra.run.smithi102:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:28:36.865 DEBUG:teuthology.orchestra.run.smithi195:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:28:36.899 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern = core 2024-09-13T15:28:36.935 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern = core 2024-09-13T15:28:36.956 DEBUG:teuthology.orchestra.run.smithi102:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:28:37.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:28:37.002 DEBUG:teuthology.orchestra.run.smithi195:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:28:37.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:28:37.017 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-13T15:28:37.027 INFO:teuthology.task.internal:Transferring archived files... 2024-09-13T15:28:37.028 DEBUG:teuthology.misc:Transferring archived files from smithi102:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/akupczyk-2024-09-13_10:00:18-upgrade-wip-aclamk-cbt-combined-distro-default-smithi/7903539/remote/smithi102 2024-09-13T15:28:37.028 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T15:28:37.124 DEBUG:teuthology.misc:Transferring archived files from smithi195:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/akupczyk-2024-09-13_10:00:18-upgrade-wip-aclamk-cbt-combined-distro-default-smithi/7903539/remote/smithi195 2024-09-13T15:28:37.125 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T15:28:37.174 INFO:teuthology.task.internal:Removing archive directory... 2024-09-13T15:28:37.174 DEBUG:teuthology.orchestra.run.smithi102:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T15:28:37.176 DEBUG:teuthology.orchestra.run.smithi195:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T15:28:37.218 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-13T15:28:37.229 INFO:teuthology.task.internal:Not uploading archives. 2024-09-13T15:28:37.229 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-13T15:28:37.267 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-13T15:28:37.268 DEBUG:teuthology.orchestra.run.smithi102:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T15:28:37.270 DEBUG:teuthology.orchestra.run.smithi195:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T15:28:37.285 INFO:teuthology.orchestra.run.smithi195.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 15:28 /home/ubuntu/cephtest 2024-09-13T15:28:37.300 INFO:teuthology.orchestra.run.smithi102.stdout: 262169 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 13 15:28 /home/ubuntu/cephtest 2024-09-13T15:28:37.300 INFO:teuthology.orchestra.run.smithi102.stdout: 685118 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 13 14:59 /home/ubuntu/cephtest/mnt.0 2024-09-13T15:28:37.300 INFO:teuthology.orchestra.run.smithi102.stdout: 685124 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 13 14:59 /home/ubuntu/cephtest/mnt.0/client.0 2024-09-13T15:28:37.300 INFO:teuthology.orchestra.run.smithi102.stdout: 4729277 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 14:59 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-13T15:28:37.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:28:37.302 INFO:teuthology.orchestra.run.smithi102.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-09-13T15:28:37.302 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-13T15:28:37.309 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-13T15:28:37.321 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-13T15:28:37.368 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi102 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-13T15:28:37.372 INFO:teuthology.run:Summary data: description: upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} duration: 5451.2579090595245 failure_reason: 'Command failed (workunit test cls/test_cls_rbd.sh) on smithi102 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=reef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh''' flavor: default owner: scheduled_akupczyk@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=36fac01c397044609f02d01fc3e98cdf status: fail success: false 2024-09-13T15:28:37.373 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-13T15:28:37.504 INFO:teuthology.run:FAIL